2020-06-12T02:24:55.800 INFO:root:teuthology version: 1.0.0-6c966fe 2020-06-12T02:24:55.803 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_kchai@teuthology --name kchai-2020-06-11_10:13:29-rados-octopus-distro-basic-smithi --description rados/thrash-erasure-code/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/fast.yaml msgr-failures/few.yaml objectstore/bluestore-comp-zstd.yaml rados.yaml recovery-overrides/{more-async-partial-recovery.yaml} supported-random-distro$/{centos_latest.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/ec-small-objects-many-deletes.yaml} --verbose --archive /home/teuthworker/archive/kchai-2020-06-11_10:13:29-rados-octopus-distro-basic-smithi/5138043 --block --lock -- /tmp/teuthology-worker.s9ihg7so.tmp 2020-06-12T02:24:55.841 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-06-12T02:24:55.907 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/kchai-2020-06-11_10:13:29-rados-octopus-distro-basic-smithi/5138043 branch: octopus description: rados/thrash-erasure-code/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/fast.yaml msgr-failures/few.yaml objectstore/bluestore-comp-zstd.yaml rados.yaml recovery-overrides/{more-async-partial-recovery.yaml} supported-random-distro$/{centos_latest.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/ec-small-objects-many-deletes.yaml} email: tchaikov@gmail.com first_in_suite: false job_id: '5138043' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: kchai-2020-06-11_10:13:29-rados-octopus-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 30 os_type: centos os_version: '8.1' overrides: admin_socket: branch: octopus ceph: conf: global: mon client directed command retry: 5 ms inject socket failures: 5000 osd pool default ec fast read: true osd_async_recovery_min_cost: 1 osd_object_clean_region_max_num_intervals: 1000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug reject backfill probability: 0.1 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd heartbeat use min delay socket: true osd max backfills: 2 osd max markdown count: 1000 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true fs: xfs log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request sha1: 91d24f498556f86f6d1318d2efd5210c35b36762 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 91d24f498556f86f6d1318d2efd5210c35b36762 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: octopus-py3 sha1: 1e0d5d60071d5554f87a2be72ea0bcccb272aab3 owner: scheduled_kchai@teuthology priority: 90 repo: https://github.com/ceph/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b sha1: 91d24f498556f86f6d1318d2efd5210c35b36762 sleep_before_teardown: 0 suite: rados suite_branch: octopus-py3 suite_path: /home/teuthworker/src/github.com_tchaikov_ceph_octopus-py3/qa suite_relpath: qa suite_repo: https://github.com/tchaikov/ceph.git suite_sha1: 1e0d5d60071d5554f87a2be72ea0bcccb272aab3 tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 min_in: 4 timeout: 1200 - rados: clients: - client.0 ec_pool: true max_in_flight: 8 max_seconds: 600 objects: 20 op_weights: append: 10 delete: 20 read: 0 write: 0 ops: 400000 size: 16384 teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.5990 2020-06-12T02:24:55.908 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_tchaikov_ceph_octopus-py3/qa; will attempt to use it 2020-06-12T02:24:55.909 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_tchaikov_ceph_octopus-py3/qa/tasks 2020-06-12T02:24:55.909 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-06-12T02:24:55.911 INFO:teuthology.task.internal:Checking packages... 2020-06-12T02:24:55.942 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash '91d24f498556f86f6d1318d2efd5210c35b36762' 2020-06-12T02:24:55.942 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-06-12T02:24:55.943 INFO:teuthology.packaging:ref: None 2020-06-12T02:24:55.943 INFO:teuthology.packaging:tag: None 2020-06-12T02:24:55.944 INFO:teuthology.packaging:branch: octopus 2020-06-12T02:24:55.944 INFO:teuthology.packaging:sha1: 91d24f498556f86f6d1318d2efd5210c35b36762 2020-06-12T02:24:55.944 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=octopus 2020-06-12T02:24:56.236 INFO:teuthology.task.internal:Found packages for ceph version 15.2.3-298.g7488806bf1 2020-06-12T02:24:56.236 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-06-12T02:24:56.246 INFO:teuthology.task.internal:no buildpackages task found 2020-06-12T02:24:56.246 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-06-12T02:24:56.300 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-06-12T02:24:56.301 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-06-12T02:24:56.376 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-06-12T02:25:06.394 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-06-12T02:25:16.416 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-06-12T02:25:26.432 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-06-12T02:25:36.453 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_kchai@teuthology', 'count': 2, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/kchai-2020-06-11_10:13:29-rados-octopus-distro-basic-smithi/5138043'} 2020-06-12T02:25:36.482 DEBUG:teuthology.lock.ops:locked ubuntu@smithi107.front.sepia.ceph.com, ubuntu@smithi188.front.sepia.ceph.com 2020-06-12T02:25:36.574 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2a60180ae8>, signals=[15]) 2020-06-12T02:25:36.786 INFO:teuthology.provision.fog.smithi107:Scheduling deploy of centos 8.1 2020-06-12T02:25:36.793 INFO:teuthology.provision.fog.smithi188:Scheduling deploy of centos 8.1 2020-06-12T02:25:37.132 INFO:teuthology.orchestra.console:Power off smithi107 2020-06-12T02:25:37.133 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-06-12T02:25:37.160 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-12T02:25:37.195 INFO:teuthology.orchestra.console:Power off smithi188 2020-06-12T02:25:37.196 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-06-12T02:25:37.222 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-12T02:25:41.214 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-12T02:25:41.252 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-12T02:25:45.334 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-12T02:25:45.366 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-12T02:25:45.557 INFO:teuthology.orchestra.console:Power off for smithi107 completed 2020-06-12T02:25:45.659 INFO:teuthology.orchestra.console:Power on smithi107 2020-06-12T02:25:45.659 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-06-12T02:25:45.686 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-12T02:25:49.486 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-12T02:25:49.705 INFO:teuthology.orchestra.console:Power off for smithi188 completed 2020-06-12T02:25:49.721 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-12T02:25:49.810 INFO:teuthology.orchestra.console:Power on smithi188 2020-06-12T02:25:49.811 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-06-12T02:25:49.838 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-12T02:25:53.839 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-12T02:25:53.866 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-12T02:25:54.056 INFO:teuthology.orchestra.console:Power on for smithi107 completed 2020-06-12T02:25:54.159 INFO:teuthology.provision.fog.smithi107:Waiting for deploy to finish 2020-06-12T02:25:57.986 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-12T02:25:58.202 INFO:teuthology.orchestra.console:Power on for smithi188 completed 2020-06-12T02:25:58.303 INFO:teuthology.provision.fog.smithi188:Waiting for deploy to finish 2020-06-12T02:28:29.314 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:28:38.387 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:28:40.311 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:28:47.442 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:28:49.391 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:28:56.503 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:28:58.447 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:29:05.586 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:29:07.503 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:29:14.642 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:29:16.558 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:29:23.701 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:29:25.615 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:29:34.675 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:29:43.731 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:30:15.616 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-06-12T02:30:15.619 INFO:teuthology.orchestra.run.smithi107:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-06-12T02:30:29.705 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:30:30.054 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-06-12T02:30:30.055 INFO:teuthology.orchestra.run.smithi188:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-06-12T02:30:30.373 INFO:teuthology.provision.fog.smithi188:Node is ready 2020-06-12T02:30:30.374 INFO:teuthology.orchestra.run.smithi188:> hostname 2020-06-12T02:30:30.389 INFO:teuthology.orchestra.run.smithi188.stdout:smithi188.front.sepia.ceph.com 2020-06-12T02:30:30.390 INFO:teuthology.orchestra.run.smithi188:> grep smithi188.front.sepia.ceph.com /etc/hosts 2020-06-12T02:30:30.446 INFO:teuthology.orchestra.run.smithi188.stdout:172.21.15.188 smithi188.front.sepia.ceph.com smithi188 2020-06-12T02:30:30.447 INFO:teuthology.orchestra.run.smithi188:> sudo hostname smithi188 2020-06-12T02:30:30.570 INFO:teuthology.orchestra.run.smithi188:> sudo sed -i -e 's/smithi188.front.sepia.ceph.com/smithi188/g' /etc/hosts 2020-06-12T02:30:30.600 INFO:teuthology.orchestra.run.smithi188:> sudo sed -i -e 's/172.21.15.188/172.21.15.188/g' /etc/hosts 2020-06-12T02:30:30.670 INFO:teuthology.orchestra.run.smithi188:> sudo sed -i -e 's/smithi188.front.sepia.ceph.com/smithi188/g' /etc/hostname 2020-06-12T02:30:30.739 INFO:teuthology.orchestra.run.smithi188:> sudo hostname smithi188 2020-06-12T02:30:30.806 INFO:teuthology.provision.fog.smithi188:Deploy complete! 2020-06-12T02:30:30.808 DEBUG:teuthology.parallel:result is None 2020-06-12T02:30:31.250 INFO:teuthology.provision.fog.smithi107:Node is ready 2020-06-12T02:30:31.250 INFO:teuthology.orchestra.run.smithi107:> hostname 2020-06-12T02:30:31.276 INFO:teuthology.orchestra.run.smithi107.stdout:smithi107.front.sepia.ceph.com 2020-06-12T02:30:31.276 INFO:teuthology.orchestra.run.smithi107:> grep smithi107.front.sepia.ceph.com /etc/hosts 2020-06-12T02:30:31.333 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107 smithi107.front.sepia.ceph.com smithi107 2020-06-12T02:30:31.333 INFO:teuthology.orchestra.run.smithi107:> sudo hostname smithi107 2020-06-12T02:30:31.467 INFO:teuthology.orchestra.run.smithi107:> sudo sed -i -e 's/smithi107.front.sepia.ceph.com/smithi107/g' /etc/hosts 2020-06-12T02:30:31.499 INFO:teuthology.orchestra.run.smithi107:> sudo sed -i -e 's/172.21.15.107/172.21.15.107/g' /etc/hosts 2020-06-12T02:30:31.568 INFO:teuthology.orchestra.run.smithi107:> sudo sed -i -e 's/smithi107.front.sepia.ceph.com/smithi107/g' /etc/hostname 2020-06-12T02:30:31.637 INFO:teuthology.orchestra.run.smithi107:> sudo hostname smithi107 2020-06-12T02:30:31.705 INFO:teuthology.provision.fog.smithi107:Deploy complete! 2020-06-12T02:30:31.705 DEBUG:teuthology.parallel:result is None 2020-06-12T02:30:31.842 INFO:teuthology.lock.ops:Checking smithi107.front.sepia.ceph.com 2020-06-12T02:30:31.844 INFO:teuthology.lock.ops:New key found. Updating... 2020-06-12T02:30:31.866 INFO:teuthology.lock.ops:Checking smithi188.front.sepia.ceph.com 2020-06-12T02:30:31.867 INFO:teuthology.lock.ops:New key found. Updating... 2020-06-12T02:30:31.887 INFO:teuthology.lock.ops:Updating [smithi107.front.sepia.ceph.com]: set os type and version on server 2020-06-12T02:30:31.888 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-12T02:30:31.889 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:30:31.963 INFO:teuthology.orchestra.run.smithi107:> true 2020-06-12T02:30:32.132 INFO:teuthology.orchestra.run.smithi107:> uname -m 2020-06-12T02:30:32.185 INFO:teuthology.orchestra.run.smithi107.stdout:x86_64 2020-06-12T02:30:32.186 INFO:teuthology.orchestra.run.smithi107:> cat /etc/os-release 2020-06-12T02:30:32.239 INFO:teuthology.orchestra.run.smithi107.stdout:NAME="CentOS Linux" 2020-06-12T02:30:32.239 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION="8 (Core)" 2020-06-12T02:30:32.239 INFO:teuthology.orchestra.run.smithi107.stdout:ID="centos" 2020-06-12T02:30:32.240 INFO:teuthology.orchestra.run.smithi107.stdout:ID_LIKE="rhel fedora" 2020-06-12T02:30:32.240 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_ID="8" 2020-06-12T02:30:32.240 INFO:teuthology.orchestra.run.smithi107.stdout:PLATFORM_ID="platform:el8" 2020-06-12T02:30:32.240 INFO:teuthology.orchestra.run.smithi107.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-06-12T02:30:32.240 INFO:teuthology.orchestra.run.smithi107.stdout:ANSI_COLOR="0;31" 2020-06-12T02:30:32.241 INFO:teuthology.orchestra.run.smithi107.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-06-12T02:30:32.241 INFO:teuthology.orchestra.run.smithi107.stdout:HOME_URL="https://www.centos.org/" 2020-06-12T02:30:32.241 INFO:teuthology.orchestra.run.smithi107.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-06-12T02:30:32.241 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:30:32.241 INFO:teuthology.orchestra.run.smithi107.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-06-12T02:30:32.242 INFO:teuthology.orchestra.run.smithi107.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-06-12T02:30:32.242 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-06-12T02:30:32.242 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-06-12T02:30:32.242 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:30:32.244 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server 2020-06-12T02:30:32.261 INFO:teuthology.lock.ops:Updating [smithi188.front.sepia.ceph.com]: set os type and version on server 2020-06-12T02:30:32.261 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-12T02:30:32.262 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:30:32.341 INFO:teuthology.orchestra.run.smithi188:> true 2020-06-12T02:30:32.508 INFO:teuthology.orchestra.run.smithi188:> uname -m 2020-06-12T02:30:32.562 INFO:teuthology.orchestra.run.smithi188.stdout:x86_64 2020-06-12T02:30:32.563 INFO:teuthology.orchestra.run.smithi188:> cat /etc/os-release 2020-06-12T02:30:32.616 INFO:teuthology.orchestra.run.smithi188.stdout:NAME="CentOS Linux" 2020-06-12T02:30:32.616 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION="8 (Core)" 2020-06-12T02:30:32.617 INFO:teuthology.orchestra.run.smithi188.stdout:ID="centos" 2020-06-12T02:30:32.617 INFO:teuthology.orchestra.run.smithi188.stdout:ID_LIKE="rhel fedora" 2020-06-12T02:30:32.617 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION_ID="8" 2020-06-12T02:30:32.617 INFO:teuthology.orchestra.run.smithi188.stdout:PLATFORM_ID="platform:el8" 2020-06-12T02:30:32.618 INFO:teuthology.orchestra.run.smithi188.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-06-12T02:30:32.618 INFO:teuthology.orchestra.run.smithi188.stdout:ANSI_COLOR="0;31" 2020-06-12T02:30:32.618 INFO:teuthology.orchestra.run.smithi188.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-06-12T02:30:32.618 INFO:teuthology.orchestra.run.smithi188.stdout:HOME_URL="https://www.centos.org/" 2020-06-12T02:30:32.618 INFO:teuthology.orchestra.run.smithi188.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-06-12T02:30:32.619 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:30:32.620 INFO:teuthology.orchestra.run.smithi188.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-06-12T02:30:32.620 INFO:teuthology.orchestra.run.smithi188.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-06-12T02:30:32.620 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-06-12T02:30:32.620 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-06-12T02:30:32.620 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:30:32.621 INFO:teuthology.lock.ops:Updating smithi188.front.sepia.ceph.com on lock server 2020-06-12T02:30:32.644 INFO:teuthology.task.internal.lock_machines:2 smithi machines locked this try, 2/2 locked so far 2020-06-12T02:30:32.680 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi107.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDCX7fcNNixx1ENGx6xAbkp5L/s+PCWpsuZl3W/iLRXFzuXqOYn77VdRz1WRVag9Ve3HnpzwFAYD79a4d0zD3jNM/jI4SnkGl3koEVEe4DxDSq4ahaBdQw8Wv6HdPsDrs0gT4XhjcV7BL/7evMctHKg/KF8Qh7ZpuoffcMzlCh8XO+TK9a5noZUD1tbPqm+g53kw3tqZtuQpYUh/dLuI/d/dzbpo0A6k3UqXQDusAFLQG51NN3KpmxCSYKgv03X/5Jef1TdFeRKX2pEZDnw3lKt8tPAIo9598wagPCeL0Q3xkNvPmaH+0AxOo2GhPS8pRsafkwFt9PTi3vhKgdXlYhkWNM00/b5rIsSpXeuZP1KCq+fnmuxvyEoIHZkTAjioJ3NBLvgRut8A+3g9S6FFppbmdUzrBr4CO+eaB3Cx18EO+0IEc9el/XlgjIhNQnkKrg2oPNBlFkQAamqnQzpY3wusAaQXBkL/mtXwp6fDAsVLNZPB5FWN0XwzfaojtiSv3k= smithi188.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMj9WgFQM40so9VyWfiGhLrFUU1Ttz17IkSiY6OPQk+6CuHXLgcViAqQLjJNEyWSh7YmMmYwuzlBKA3ckVs2XDDqf67/eU7KN/go/Zxy/DZOjpuJg/X7RWeyr/VDmdP3eUi5i7iGu1/+xJowqhj5yjZq8b3PgkWfBjLw3YxUv8kTvfUEKf45z+6tGJYhDbr0BcWpMnDg3KX7mp2Zcy//p58JU0fNTRKB1LLmyViDDQWdPoK0hMKfjmy3BoX990kDJORhW7ak935HAhE0dzmZwwB3ZjjSjSVPvSjhX86YY/Aq9hShMhSxdzfqgalrmuJDZG2gF+/pDizPBrEMIF2YFj/A8331EEru73yOLLKVaA4h7tDjyk+i3RqxPjPyUMkX79DZwfoMGs6G6Zw8rnhnJq/NaFnxSS5rkNCoev1sEQZr1Tbb6qarNlVx+T9Z7XEP0/B8SdbX/GR+eyyAQmOf++RdfYgvUA9IutrDyK8gI3s3aCQFesO6ElnUmRD/FKYVc= 2020-06-12T02:30:32.681 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-06-12T02:30:32.756 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-06-12T02:30:32.770 INFO:teuthology.task.internal:Saving configuration 2020-06-12T02:30:32.794 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-06-12T02:30:32.805 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-06-12T02:30:32.822 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-06-12 02:25:36.463608', 'description': '/home/teuthworker/archive/kchai-2020-06-11_10:13:29-rados-octopus-distro-basic-smithi/5138043', 'locked_by': 'scheduled_kchai@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP1puh98thGfoaelNbTPmvOH6hDGearDsGlN+E2wP/cKsupa+YztqNlxTLph+O/s9LkX99bj7Qa2lkXD2+4qfpU=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi107.front.sepia.ceph.com'} 2020-06-12T02:30:32.839 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-06-12 02:25:36.463628', 'description': '/home/teuthworker/archive/kchai-2020-06-11_10:13:29-rados-octopus-distro-basic-smithi/5138043', 'locked_by': 'scheduled_kchai@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMYwSRh5fxUuR+Z3O5uqo9Ti5LztEFS4PPhBYhnGvzC3j6Fd7vQA7v0BrN03bKd4c3dOvZkFqKaRjEYfGE4vBYM=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi188.front.sepia.ceph.com'} 2020-06-12T02:30:32.840 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-06-12T02:30:32.850 INFO:teuthology.task.internal:roles: ubuntu@smithi107.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2020-06-12T02:30:32.851 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2020-06-12T02:30:32.851 INFO:teuthology.run_tasks:Running task console_log... 2020-06-12T02:30:32.958 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2a610fbd08>, signals=[15]) 2020-06-12T02:30:32.959 INFO:teuthology.run_tasks:Running task internal.connect... 2020-06-12T02:30:32.969 INFO:teuthology.task.internal:Opening connections... 2020-06-12T02:30:32.969 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi107.front.sepia.ceph.com 2020-06-12T02:30:32.979 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:30:33.055 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi188.front.sepia.ceph.com 2020-06-12T02:30:33.056 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:30:33.126 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-06-12T02:30:33.136 INFO:teuthology.orchestra.run.smithi107:> uname -m 2020-06-12T02:30:33.173 INFO:teuthology.orchestra.run.smithi107.stdout:x86_64 2020-06-12T02:30:33.174 INFO:teuthology.orchestra.run.smithi107:> cat /etc/os-release 2020-06-12T02:30:33.226 INFO:teuthology.orchestra.run.smithi107.stdout:NAME="CentOS Linux" 2020-06-12T02:30:33.227 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION="8 (Core)" 2020-06-12T02:30:33.227 INFO:teuthology.orchestra.run.smithi107.stdout:ID="centos" 2020-06-12T02:30:33.227 INFO:teuthology.orchestra.run.smithi107.stdout:ID_LIKE="rhel fedora" 2020-06-12T02:30:33.227 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_ID="8" 2020-06-12T02:30:33.228 INFO:teuthology.orchestra.run.smithi107.stdout:PLATFORM_ID="platform:el8" 2020-06-12T02:30:33.228 INFO:teuthology.orchestra.run.smithi107.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-06-12T02:30:33.228 INFO:teuthology.orchestra.run.smithi107.stdout:ANSI_COLOR="0;31" 2020-06-12T02:30:33.228 INFO:teuthology.orchestra.run.smithi107.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-06-12T02:30:33.228 INFO:teuthology.orchestra.run.smithi107.stdout:HOME_URL="https://www.centos.org/" 2020-06-12T02:30:33.229 INFO:teuthology.orchestra.run.smithi107.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-06-12T02:30:33.229 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:30:33.229 INFO:teuthology.orchestra.run.smithi107.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-06-12T02:30:33.229 INFO:teuthology.orchestra.run.smithi107.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-06-12T02:30:33.229 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-06-12T02:30:33.230 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-06-12T02:30:33.230 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:30:33.230 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server 2020-06-12T02:30:33.255 INFO:teuthology.orchestra.run.smithi188:> uname -m 2020-06-12T02:30:33.270 INFO:teuthology.orchestra.run.smithi188.stdout:x86_64 2020-06-12T02:30:33.270 INFO:teuthology.orchestra.run.smithi188:> cat /etc/os-release 2020-06-12T02:30:33.324 INFO:teuthology.orchestra.run.smithi188.stdout:NAME="CentOS Linux" 2020-06-12T02:30:33.324 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION="8 (Core)" 2020-06-12T02:30:33.325 INFO:teuthology.orchestra.run.smithi188.stdout:ID="centos" 2020-06-12T02:30:33.325 INFO:teuthology.orchestra.run.smithi188.stdout:ID_LIKE="rhel fedora" 2020-06-12T02:30:33.325 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION_ID="8" 2020-06-12T02:30:33.325 INFO:teuthology.orchestra.run.smithi188.stdout:PLATFORM_ID="platform:el8" 2020-06-12T02:30:33.326 INFO:teuthology.orchestra.run.smithi188.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-06-12T02:30:33.326 INFO:teuthology.orchestra.run.smithi188.stdout:ANSI_COLOR="0;31" 2020-06-12T02:30:33.326 INFO:teuthology.orchestra.run.smithi188.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-06-12T02:30:33.326 INFO:teuthology.orchestra.run.smithi188.stdout:HOME_URL="https://www.centos.org/" 2020-06-12T02:30:33.326 INFO:teuthology.orchestra.run.smithi188.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-06-12T02:30:33.327 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:30:33.327 INFO:teuthology.orchestra.run.smithi188.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-06-12T02:30:33.327 INFO:teuthology.orchestra.run.smithi188.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-06-12T02:30:33.327 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-06-12T02:30:33.327 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-06-12T02:30:33.328 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:30:33.328 INFO:teuthology.lock.ops:Updating smithi188.front.sepia.ceph.com on lock server 2020-06-12T02:30:33.348 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-06-12T02:30:33.381 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-06-12T02:30:33.496 INFO:teuthology.task.internal:Checking for old test directory... 2020-06-12T02:30:33.496 INFO:teuthology.orchestra.run.smithi107:> test '!' -e /home/ubuntu/cephtest 2020-06-12T02:30:33.499 INFO:teuthology.orchestra.run.smithi188:> test '!' -e /home/ubuntu/cephtest 2020-06-12T02:30:33.512 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-06-12T02:30:33.533 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-06-12T02:30:33.534 INFO:teuthology.orchestra.run.smithi107:> test -z $(ls -A /var/lib/ceph) 2020-06-12T02:30:33.554 INFO:teuthology.orchestra.run.smithi188:> test -z $(ls -A /var/lib/ceph) 2020-06-12T02:30:33.569 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-06-12T02:30:33.617 INFO:teuthology.run_tasks:Running task kernel... 2020-06-12T02:30:33.642 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}, 'mgr.y': {'kdb': True, 'sha1': 'distro'}, 'osd.0': {'kdb': True, 'sha1': 'distro'}, 'osd.1': {'kdb': True, 'sha1': 'distro'}, 'osd.2': {'kdb': True, 'sha1': 'distro'}, 'osd.3': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}, 'node-exporter.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'osd.4': {'kdb': True, 'sha1': 'distro'}, 'osd.5': {'kdb': True, 'sha1': 'distro'}, 'osd.6': {'kdb': True, 'sha1': 'distro'}, 'osd.7': {'kdb': True, 'sha1': 'distro'}, 'client.1': {'kdb': True, 'sha1': 'distro'}, 'prometheus.a': {'kdb': True, 'sha1': 'distro'}, 'node-exporter.b': {'kdb': True, 'sha1': 'distro'}} 2020-06-12T02:30:33.644 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2020-06-12T02:30:33.647 INFO:teuthology.orchestra.run.smithi107:> uname -r 2020-06-12T02:30:33.666 INFO:teuthology.orchestra.run.smithi107.stdout:4.18.0-147.8.1.el8_1.x86_64 2020-06-12T02:30:33.666 INFO:teuthology.task.kernel:Running kernel on smithi107: 4.18.0-147.8.1.el8_1.x86_64 2020-06-12T02:30:33.669 INFO:teuthology.orchestra.run.smithi107:> sudo yum install -y kernel 2020-06-12T02:30:36.178 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS-8 - AppStream 39 MB/s | 7.0 MB 00:00 2020-06-12T02:30:37.359 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS-8 - Base 26 MB/s | 2.2 MB 00:00 2020-06-12T02:30:37.775 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS-8 - Extras 88 kB/s | 5.5 kB 00:00 2020-06-12T02:30:37.922 INFO:teuthology.orchestra.run.smithi107.stdout:Copr repo for ceph-el8 owned by ktdreyer 460 kB/s | 47 kB 00:00 2020-06-12T02:30:38.295 INFO:teuthology.orchestra.run.smithi107.stdout:Extra Packages for Enterprise Linux 22 MB/s | 6.9 MB 00:00 2020-06-12T02:30:39.721 INFO:teuthology.orchestra.run.smithi107.stdout:lab-extras 3.9 MB/s | 20 kB 00:00 2020-06-12T02:30:40.689 INFO:teuthology.orchestra.run.smithi107.stdout:Package kernel-4.18.0-147.el8.x86_64 is already installed. 2020-06-12T02:30:40.689 INFO:teuthology.orchestra.run.smithi107.stdout:Package kernel-4.18.0-147.8.1.el8_1.x86_64 is already installed. 2020-06-12T02:30:40.831 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2020-06-12T02:30:40.831 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2020-06-12T02:30:40.832 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2020-06-12T02:30:40.959 INFO:teuthology.orchestra.run.smithi107:> echo no | sudo yum reinstall kernel || true 2020-06-12T02:30:41.398 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:02 ago on Fri 12 Jun 2020 02:30:39 AM UTC. 2020-06-12T02:30:41.749 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2020-06-12T02:30:41.759 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:30:41.760 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2020-06-12T02:30:41.760 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:30:41.760 INFO:teuthology.orchestra.run.smithi107.stdout:Reinstalling: 2020-06-12T02:30:41.760 INFO:teuthology.orchestra.run.smithi107.stdout: kernel x86_64 4.18.0-147.8.1.el8_1 CentOS-Base 1.5 M 2020-06-12T02:30:41.761 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:30:41.761 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2020-06-12T02:30:41.761 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:30:41.761 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:30:41.761 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 1.5 M 2020-06-12T02:30:41.762 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 0 2020-06-12T02:30:41.762 INFO:teuthology.orchestra.run.smithi107.stderr:Operation aborted. 2020-06-12T02:30:41.855 INFO:teuthology.orchestra.run.smithi107.stdout:Is this ok [y/N]: 2020-06-12T02:30:41.856 INFO:teuthology.orchestra.run.smithi107:> sudo yum reinstall -y kernel || true 2020-06-12T02:30:42.245 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:03 ago on Fri 12 Jun 2020 02:30:39 AM UTC. 2020-06-12T02:30:42.594 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2020-06-12T02:30:42.594 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:30:42.595 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2020-06-12T02:30:42.595 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:30:42.595 INFO:teuthology.orchestra.run.smithi107.stdout:Reinstalling: 2020-06-12T02:30:42.596 INFO:teuthology.orchestra.run.smithi107.stdout: kernel x86_64 4.18.0-147.8.1.el8_1 CentOS-Base 1.5 M 2020-06-12T02:30:42.596 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:30:42.596 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2020-06-12T02:30:42.597 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:30:42.597 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:30:42.597 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 1.5 M 2020-06-12T02:30:42.598 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 0 2020-06-12T02:30:42.598 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2020-06-12T02:30:42.667 INFO:teuthology.orchestra.run.smithi107.stdout:kernel-4.18.0-147.8.1.el8_1.x86_64.rpm 22 MB/s | 1.5 MB 00:00 2020-06-12T02:30:42.667 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2020-06-12T02:30:42.668 INFO:teuthology.orchestra.run.smithi107.stdout:Total 22 MB/s | 1.5 MB 00:00 2020-06-12T02:30:42.690 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2020-06-12T02:30:42.845 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2020-06-12T02:30:42.845 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2020-06-12T02:30:42.900 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2020-06-12T02:30:42.901 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2020-06-12T02:30:43.033 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2020-06-12T02:30:43.179 INFO:teuthology.orchestra.run.smithi107.stdout: Reinstalling : kernel-4.18.0-147.8.1.el8_1.x86_64 1/2 2020-06-12T02:30:43.179 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : kernel-4.18.0-147.8.1.el8_1.x86_64 2/2 2020-06-12T02:30:43.403 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-4.18.0-147.8.1.el8_1.x86_64 2/2 2020-06-12T02:30:43.404 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-4.18.0-147.8.1.el8_1.x86_64 1/2 2020-06-12T02:30:43.714 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-4.18.0-147.8.1.el8_1.x86_64 2/2 2020-06-12T02:30:43.715 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:30:43.715 INFO:teuthology.orchestra.run.smithi107.stdout:Reinstalled: 2020-06-12T02:30:43.716 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-4.18.0-147.8.1.el8_1.x86_64 2020-06-12T02:30:43.716 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:30:43.716 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2020-06-12T02:30:43.803 INFO:teuthology.orchestra.run.smithi107:> rpm -q kernel --last | head -n 1 2020-06-12T02:30:43.863 INFO:teuthology.orchestra.run.smithi107.stdout:kernel-4.18.0-147.8.1.el8_1.x86_64 Fri 12 Jun 2020 02:30:43 AM UTC 2020-06-12T02:30:43.864 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-147.8.1.el8_1.x86_64 2020-06-12T02:30:43.864 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2020-06-12T02:30:43.865 INFO:teuthology.orchestra.run.smithi188:> uname -r 2020-06-12T02:30:43.889 INFO:teuthology.orchestra.run.smithi188.stdout:4.18.0-147.8.1.el8_1.x86_64 2020-06-12T02:30:43.890 INFO:teuthology.task.kernel:Running kernel on smithi188: 4.18.0-147.8.1.el8_1.x86_64 2020-06-12T02:30:43.891 INFO:teuthology.orchestra.run.smithi188:> sudo yum install -y kernel 2020-06-12T02:30:46.609 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS-8 - AppStream 21 MB/s | 7.0 MB 00:00 2020-06-12T02:30:47.789 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS-8 - Base 24 MB/s | 2.2 MB 00:00 2020-06-12T02:30:48.221 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS-8 - Extras 88 kB/s | 5.5 kB 00:00 2020-06-12T02:30:48.373 INFO:teuthology.orchestra.run.smithi188.stdout:Copr repo for ceph-el8 owned by ktdreyer 442 kB/s | 47 kB 00:00 2020-06-12T02:30:48.728 INFO:teuthology.orchestra.run.smithi188.stdout:Extra Packages for Enterprise Linux 23 MB/s | 6.9 MB 00:00 2020-06-12T02:30:50.132 INFO:teuthology.orchestra.run.smithi188.stdout:lab-extras 4.3 MB/s | 20 kB 00:00 2020-06-12T02:30:51.079 INFO:teuthology.orchestra.run.smithi188.stdout:Package kernel-4.18.0-147.el8.x86_64 is already installed. 2020-06-12T02:30:51.080 INFO:teuthology.orchestra.run.smithi188.stdout:Package kernel-4.18.0-147.8.1.el8_1.x86_64 is already installed. 2020-06-12T02:30:51.218 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2020-06-12T02:30:51.219 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2020-06-12T02:30:51.219 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2020-06-12T02:30:51.352 INFO:teuthology.orchestra.run.smithi188:> echo no | sudo yum reinstall kernel || true 2020-06-12T02:30:51.789 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:00:01 ago on Fri 12 Jun 2020 02:30:50 AM UTC. 2020-06-12T02:30:52.121 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2020-06-12T02:30:52.131 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:30:52.131 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2020-06-12T02:30:52.131 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:30:52.132 INFO:teuthology.orchestra.run.smithi188.stdout:Reinstalling: 2020-06-12T02:30:52.132 INFO:teuthology.orchestra.run.smithi188.stdout: kernel x86_64 4.18.0-147.8.1.el8_1 CentOS-Base 1.5 M 2020-06-12T02:30:52.132 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:30:52.132 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2020-06-12T02:30:52.132 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:30:52.132 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:30:52.133 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 1.5 M 2020-06-12T02:30:52.133 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 0 2020-06-12T02:30:52.133 INFO:teuthology.orchestra.run.smithi188.stderr:Operation aborted. 2020-06-12T02:30:52.235 INFO:teuthology.orchestra.run.smithi188.stdout:Is this ok [y/N]: 2020-06-12T02:30:52.235 INFO:teuthology.orchestra.run.smithi188:> sudo yum reinstall -y kernel || true 2020-06-12T02:30:52.627 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:00:02 ago on Fri 12 Jun 2020 02:30:50 AM UTC. 2020-06-12T02:30:52.988 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2020-06-12T02:30:52.989 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:30:52.989 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2020-06-12T02:30:52.989 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:30:52.990 INFO:teuthology.orchestra.run.smithi188.stdout:Reinstalling: 2020-06-12T02:30:52.990 INFO:teuthology.orchestra.run.smithi188.stdout: kernel x86_64 4.18.0-147.8.1.el8_1 CentOS-Base 1.5 M 2020-06-12T02:30:52.990 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:30:52.990 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2020-06-12T02:30:52.990 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:30:52.991 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:30:52.991 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 1.5 M 2020-06-12T02:30:52.991 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 0 2020-06-12T02:30:52.991 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2020-06-12T02:30:53.038 INFO:teuthology.orchestra.run.smithi188.stdout:kernel-4.18.0-147.8.1.el8_1.x86_64.rpm 33 MB/s | 1.5 MB 00:00 2020-06-12T02:30:53.039 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2020-06-12T02:30:53.039 INFO:teuthology.orchestra.run.smithi188.stdout:Total 32 MB/s | 1.5 MB 00:00 2020-06-12T02:30:53.052 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2020-06-12T02:30:53.217 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2020-06-12T02:30:53.217 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2020-06-12T02:30:53.264 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2020-06-12T02:30:53.265 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2020-06-12T02:30:53.384 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2020-06-12T02:30:53.517 INFO:teuthology.orchestra.run.smithi188.stdout: Reinstalling : kernel-4.18.0-147.8.1.el8_1.x86_64 1/2 2020-06-12T02:30:53.518 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : kernel-4.18.0-147.8.1.el8_1.x86_64 2/2 2020-06-12T02:30:53.732 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: kernel-4.18.0-147.8.1.el8_1.x86_64 2/2 2020-06-12T02:30:53.732 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : kernel-4.18.0-147.8.1.el8_1.x86_64 1/2 2020-06-12T02:30:54.050 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : kernel-4.18.0-147.8.1.el8_1.x86_64 2/2 2020-06-12T02:30:54.051 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:30:54.051 INFO:teuthology.orchestra.run.smithi188.stdout:Reinstalled: 2020-06-12T02:30:54.052 INFO:teuthology.orchestra.run.smithi188.stdout: kernel-4.18.0-147.8.1.el8_1.x86_64 2020-06-12T02:30:54.052 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:30:54.052 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2020-06-12T02:30:54.134 INFO:teuthology.orchestra.run.smithi188:> rpm -q kernel --last | head -n 1 2020-06-12T02:30:54.230 INFO:teuthology.orchestra.run.smithi188.stdout:kernel-4.18.0-147.8.1.el8_1.x86_64 Fri 12 Jun 2020 02:30:53 AM UTC 2020-06-12T02:30:54.231 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-147.8.1.el8_1.x86_64 2020-06-12T02:30:54.231 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2020-06-12T02:30:54.232 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2020-06-12T02:30:54.232 INFO:teuthology.orchestra.run.smithi107:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-06-12T02:30:54.274 INFO:teuthology.orchestra.run.smithi107.stdout:ttyS1 2020-06-12T02:30:54.286 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2020-06-12T02:30:54.287 INFO:teuthology.orchestra.run.smithi188:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-06-12T02:30:54.325 INFO:teuthology.orchestra.run.smithi188.stdout:ttyS1 2020-06-12T02:30:54.336 INFO:teuthology.run_tasks:Running task internal.base... 2020-06-12T02:30:54.345 INFO:teuthology.task.internal:Creating test directory... 2020-06-12T02:30:54.346 INFO:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-06-12T02:30:54.350 INFO:teuthology.orchestra.run.smithi188:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-06-12T02:30:54.391 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2020-06-12T02:30:54.509 INFO:teuthology.run_tasks:Running task internal.archive... 2020-06-12T02:30:54.534 INFO:teuthology.task.internal:Creating archive directory... 2020-06-12T02:30:54.535 INFO:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-06-12T02:30:54.538 INFO:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-06-12T02:30:54.579 INFO:teuthology.run_tasks:Running task internal.coredump... 2020-06-12T02:30:54.597 INFO:teuthology.task.internal:Enabling coredump saving... 2020-06-12T02:30:54.598 INFO:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-06-12T02:30:54.607 INFO:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-06-12T02:30:54.673 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-06-12T02:30:54.680 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-06-12T02:30:54.688 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-06-12T02:30:54.690 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-06-12T02:30:54.691 INFO:teuthology.run_tasks:Running task internal.sudo... 2020-06-12T02:30:54.702 INFO:teuthology.task.internal:Configuring sudo... 2020-06-12T02:30:54.703 INFO:teuthology.orchestra.run.smithi107:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-06-12T02:30:54.707 INFO:teuthology.orchestra.run.smithi188:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-06-12T02:30:54.736 INFO:teuthology.run_tasks:Running task internal.syslog... 2020-06-12T02:30:54.751 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2020-06-12T02:30:54.752 INFO:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-06-12T02:30:54.781 INFO:teuthology.orchestra.run.smithi188:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-06-12T02:30:54.795 INFO:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-06-12T02:30:54.872 INFO:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2020-06-12T02:30:54.928 INFO:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-06-12T02:30:55.002 INFO:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2020-06-12T02:30:55.053 INFO:teuthology.orchestra.run.smithi107:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2020-06-12T02:30:55.141 INFO:teuthology.orchestra.run.smithi188:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-06-12T02:30:55.178 INFO:teuthology.orchestra.run.smithi188:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2020-06-12T02:30:55.235 INFO:teuthology.orchestra.run.smithi188:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-06-12T02:30:55.309 INFO:teuthology.orchestra.run.smithi188:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2020-06-12T02:30:55.360 INFO:teuthology.orchestra.run.smithi188:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2020-06-12T02:30:55.446 INFO:teuthology.orchestra.run.smithi107:> sudo service rsyslog restart 2020-06-12T02:30:55.448 INFO:teuthology.orchestra.run.smithi188:> sudo service rsyslog restart 2020-06-12T02:30:55.509 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2020-06-12T02:30:55.550 INFO:teuthology.orchestra.run.smithi188.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2020-06-12T02:30:55.930 INFO:teuthology.run_tasks:Running task internal.timer... 2020-06-12T02:30:55.953 INFO:teuthology.task.internal:Starting timer... 2020-06-12T02:30:55.954 INFO:teuthology.run_tasks:Running task pcp... 2020-06-12T02:30:56.089 INFO:teuthology.run_tasks:Running task selinux... 2020-06-12T02:30:56.120 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2020-06-12T02:30:56.121 INFO:teuthology.orchestra.run.smithi107:> sudo service auditd rotate 2020-06-12T02:30:56.197 INFO:teuthology.orchestra.run.smithi107.stdout:Rotating logs: [ OK ] 2020-06-12T02:30:56.199 INFO:teuthology.orchestra.run.smithi188:> sudo service auditd rotate 2020-06-12T02:30:56.257 INFO:teuthology.orchestra.run.smithi188.stdout:Rotating logs: [ OK ] 2020-06-12T02:30:56.259 DEBUG:teuthology.task.selinux:Getting current SELinux state 2020-06-12T02:30:56.259 INFO:teuthology.orchestra.run.smithi107:> /usr/sbin/getenforce 2020-06-12T02:30:56.284 INFO:teuthology.orchestra.run.smithi107.stdout:Permissive 2020-06-12T02:30:56.285 INFO:teuthology.orchestra.run.smithi188:> /usr/sbin/getenforce 2020-06-12T02:30:56.310 INFO:teuthology.orchestra.run.smithi188.stdout:Permissive 2020-06-12T02:30:56.310 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi107.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi188.front.sepia.ceph.com': 'permissive'} 2020-06-12T02:30:56.311 INFO:teuthology.orchestra.run.smithi107:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|scontext=system_u:system_r:logrotate_t:s0\)' 2020-06-12T02:30:56.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-06-12T02:30:56.354 INFO:teuthology.orchestra.run.smithi188:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|scontext=system_u:system_r:logrotate_t:s0\)' 2020-06-12T02:30:56.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-06-12T02:30:56.383 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2020-06-12T02:30:56.384 INFO:teuthology.orchestra.run.smithi107:> sudo /usr/sbin/setenforce permissive 2020-06-12T02:30:56.432 INFO:teuthology.orchestra.run.smithi188:> sudo /usr/sbin/setenforce permissive 2020-06-12T02:30:56.467 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2020-06-12T02:30:56.480 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2020-06-12T02:30:56.492 INFO:teuthology.repo_utils:Fetching master from origin 2020-06-12T02:30:56.574 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2020-06-12T02:30:56.601 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2020-06-12T02:30:56.604 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi107.front.sepia.ceph.com,smithi188.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2020-06-12T02:30:57.057 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2020-06-12T02:30:58.355 INFO:teuthology.task.ansible.out: [WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2020-06-12T02:31:01.104 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2020-06-12T02:31:01.141 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:31:01 +0000 (0:00:00.094) 0:00:00.094 *********** 2020-06-12T02:31:01.158 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:31:01 +0000 (0:00:00.017) 0:00:00.112 *********** 2020-06-12T02:31:02.863 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2020-06-12T02:31:02.864 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] 2020-06-12T02:31:02.879 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] 2020-06-12T02:31:02.991 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:31:02 +0000 (0:00:01.832) 0:00:01.944 *********** 2020-06-12T02:31:03.038 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:31:03 +0000 (0:00:00.047) 0:00:01.991 *********** 2020-06-12T02:31:03.615 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** ok: [smithi188.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-06-12T02:31:03.617 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-06-12T02:31:03.655 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:31:03 +0000 (0:00:00.617) 0:00:02.608 *********** 2020-06-12T02:31:03.704 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:31:03 +0000 (0:00:00.048) 0:00:02.657 *********** 2020-06-12T02:31:04.302 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2020-06-12T02:31:04.305 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} ok: [smithi188.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-06-12T02:31:04.350 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:31:04 +0000 (0:00:00.646) 0:00:03.303 *********** 2020-06-12T02:31:04.395 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:31:04 +0000 (0:00:00.040) 0:00:03.343 *********** 2020-06-12T02:31:04.415 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:31:04.470 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:31:04 +0000 (0:00:00.079) 0:00:03.423 *********** 2020-06-12T02:31:04.481 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:31:04.524 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:31:04 +0000 (0:00:00.053) 0:00:03.476 *********** 2020-06-12T02:31:05.091 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2020-06-12T02:31:05.101 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} ok: [smithi188.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-06-12T02:31:05.163 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:31:05 +0000 (0:00:00.639) 0:00:04.116 *********** 2020-06-12T02:31:05.215 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:31:05 +0000 (0:00:00.051) 0:00:04.168 *********** 2020-06-12T02:31:06.459 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2020-06-12T02:31:06.461 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1591929065.26481-42451454546237/source", "state": "file", "uid": 0} 2020-06-12T02:31:06.493 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:31:06 +0000 (0:00:01.278) 0:00:05.447 *********** 2020-06-12T02:31:06.506 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1591929065.1984246-107371179968566/source", "state": "file", "uid": 0} 2020-06-12T02:31:06.556 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:31:06 +0000 (0:00:00.062) 0:00:05.509 *********** 2020-06-12T02:31:07.192 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* ok: [smithi188.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtC 2020-06-12T02:31:07.193 INFO:teuthology.task.ansible.out:toHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-06-12T02:31:07.266 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2020-06-12T02:31:07.268 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-06-12T02:31:07.354 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2020-06-12T02:31:07.360 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2020-06-12T02:31:07.512 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:31:07 +0000 (0:00:00.955) 0:00:06.465 *********** 2020-06-12T02:31:07.556 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:31:07 +0000 (0:00:00.043) 0:00:06.509 *********** 2020-06-12T02:31:08.212 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi107.front.sepia.ceph.com] 2020-06-12T02:31:08.300 INFO:teuthology.task.ansible.out: [WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. Friday 12 June 2020 02:31:08 +0000 (0:00:00.743) 0:00:07.253 *********** 2020-06-12T02:31:08.314 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] 2020-06-12T02:31:08.560 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:31:08 +0000 (0:00:00.260) 0:00:07.513 *********** 2020-06-12T02:31:09.171 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** changed: [smithi107.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.222118", "end": "2020-06-12 02:31:09.116708", "rc": 0, "start": "2020-06-12 02:31:08.894590", "stderr": "", "stderr_lines": [], "stdout": "982b99592bd601b0475ee21e26576645605e4b52", "stdout_lines": ["982b99592bd601b0475ee21e26576645605e4b52"]} 2020-06-12T02:31:09.217 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:31:09 +0000 (0:00:00.656) 0:00:08.170 *********** 2020-06-12T02:31:09.276 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.251681", "end": "2020-06-12 02:31:09.213455", "rc": 0, "start": "2020-06-12 02:31:08.961774", "stderr": "", "stderr_lines": [], "stdout": "982b99592bd601b0475ee21e26576645605e4b52", "stdout_lines": ["982b99592bd601b0475ee21e26576645605e4b52"]} 2020-06-12T02:31:09.315 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:31:09 +0000 (0:00:00.098) 0:00:08.268 *********** 2020-06-12T02:31:09.565 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** changed: [smithi188.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.005518", "end": "2020-06-12 02:31:09.518020", "failed_when_result": false, "rc": 0, "start": "2020-06-12 02:31:09.512502", "stderr": "", "stderr_lines": [], "stdout": "743c659c40211acb851086b2a21beddfbfe09383", "stdout_lines": ["743c659c40211acb851086b2a21beddfbfe09383"]} 2020-06-12T02:31:09.616 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:31:09 +0000 (0:00:00.300) 0:00:08.569 *********** 2020-06-12T02:31:09.630 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.103579", "end": "2020-06-12 02:31:09.587452", "failed_when_result": false, "rc": 0, "start": "2020-06-12 02:31:09.483873", "stderr": "", "stderr_lines": [], "stdout": "743c659c40211acb851086b2a21beddfbfe09383", "stdout_lines": ["743c659c40211acb851086b2a21beddfbfe09383"]} 2020-06-12T02:31:09.670 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:31:09 +0000 (0:00:00.053) 0:00:08.623 *********** 2020-06-12T02:31:09.685 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2020-06-12T02:31:09.687 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:31:09.728 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:31:09 +0000 (0:00:00.058) 0:00:08.681 *********** 2020-06-12T02:31:09.738 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:31:09.794 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:31:09 +0000 (0:00:00.058) 0:00:08.740 *********** 2020-06-12T02:31:09.796 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2020-06-12T02:31:09.796 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:31:09.861 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:31:09 +0000 (0:00:00.074) 0:00:08.814 *********** 2020-06-12T02:31:09.870 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:31:09.927 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:31:09 +0000 (0:00:00.066) 0:00:08.881 *********** 2020-06-12T02:31:09.938 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2020-06-12T02:31:09.944 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:31:09.977 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:31:09 +0000 (0:00:00.049) 0:00:08.930 *********** 2020-06-12T02:31:09.990 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:31:10.042 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:31:10 +0000 (0:00:00.065) 0:00:08.995 *********** 2020-06-12T02:31:10.052 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2020-06-12T02:31:10.054 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:31:10.097 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:31:10 +0000 (0:00:00.054) 0:00:09.049 *********** 2020-06-12T02:31:10.106 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:31:10.156 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:31:10 +0000 (0:00:00.060) 0:00:09.109 *********** 2020-06-12T02:31:10.169 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:31:10.216 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:31:10 +0000 (0:00:00.059) 0:00:09.169 *********** 2020-06-12T02:31:10.226 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:31:10.263 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:31:10 +0000 (0:00:00.046) 0:00:09.217 *********** 2020-06-12T02:31:10.441 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2020-06-12T02:31:10.443 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-06-12T02:31:10.481 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:31:10 +0000 (0:00:00.217) 0:00:09.434 *********** 2020-06-12T02:31:10.502 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-06-12T02:31:10.542 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:31:10 +0000 (0:00:00.061) 0:00:09.495 *********** 2020-06-12T02:31:10.790 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2020-06-12T02:31:10.845 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2020-06-12T02:31:11.034 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2020-06-12T02:31:11.074 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2020-06-12T02:31:11.248 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2020-06-12T02:31:11.285 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2020-06-12T02:31:11.463 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2020-06-12T02:31:11.495 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2020-06-12T02:31:11.669 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2020-06-12T02:31:11.691 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2020-06-12T02:31:11.876 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2020-06-12T02:31:11.901 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2020-06-12T02:31:12.092 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2020-06-12T02:31:12.143 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2020-06-12T02:31:12.305 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2020-06-12T02:31:12.351 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2020-06-12T02:31:12.510 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2020-06-12T02:31:12.553 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2020-06-12T02:31:12.714 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2020-06-12T02:31:12.757 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2020-06-12T02:31:12.931 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2020-06-12T02:31:12.973 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2020-06-12T02:31:13.135 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'aemerson', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2020-06-12T02:31:13.169 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'aemerson', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2020-06-12T02:31:13.335 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2020-06-12T02:31:13.399 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2020-06-12T02:31:13.560 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2020-06-12T02:31:13.623 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2020-06-12T02:31:13.772 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2020-06-12T02:31:13.825 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2020-06-12T02:31:13.986 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2020-06-12T02:31:14.040 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2020-06-12T02:31:14.181 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2020-06-12T02:31:14.234 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2020-06-12T02:31:14.384 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2020-06-12T02:31:14.448 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2020-06-12T02:31:14.600 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2020-06-12T02:31:14.653 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2020-06-12T02:31:14.793 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2020-06-12T02:31:14.878 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2020-06-12T02:31:15.022 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2020-06-12T02:31:15.108 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2020-06-12T02:31:15.223 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1122} 2020-06-12T02:31:15.321 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1122} 2020-06-12T02:31:15.427 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2020-06-12T02:31:15.545 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2020-06-12T02:31:15.647 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2020-06-12T02:31:15.747 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2020-06-12T02:31:15.878 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2020-06-12T02:31:15.963 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2020-06-12T02:31:16.080 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2020-06-12T02:31:16.174 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2020-06-12T02:31:16.305 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2020-06-12T02:31:16.378 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2020-06-12T02:31:16.517 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2020-06-12T02:31:16.587 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2020-06-12T02:31:16.737 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2020-06-12T02:31:16.796 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2020-06-12T02:31:16.963 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2020-06-12T02:31:17.011 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2020-06-12T02:31:17.181 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2020-06-12T02:31:17.234 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2020-06-12T02:31:17.398 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2020-06-12T02:31:17.450 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2020-06-12T02:31:17.605 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2020-06-12T02:31:17.648 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2020-06-12T02:31:17.820 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2020-06-12T02:31:17.863 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2020-06-12T02:31:18.020 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2020-06-12T02:31:18.074 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2020-06-12T02:31:18.226 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2020-06-12T02:31:18.290 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2020-06-12T02:31:18.424 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2020-06-12T02:31:18.497 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2020-06-12T02:31:18.639 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2020-06-12T02:31:18.703 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2020-06-12T02:31:18.847 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2020-06-12T02:31:18.910 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2020-06-12T02:31:19.058 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-06-12T02:31:19.111 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-06-12T02:31:19.271 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2020-06-12T02:31:19.334 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2020-06-12T02:31:19.473 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2020-06-12T02:31:19.537 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2020-06-12T02:31:19.677 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2020-06-12T02:31:19.756 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2020-06-12T02:31:19.891 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2020-06-12T02:31:19.966 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2020-06-12T02:31:20.089 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2020-06-12T02:31:20.179 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2020-06-12T02:31:20.301 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2020-06-12T02:31:20.388 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2020-06-12T02:31:20.503 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2020-06-12T02:31:20.599 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2020-06-12T02:31:20.714 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2020-06-12T02:31:20.804 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2020-06-12T02:31:20.911 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2020-06-12T02:31:21.009 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2020-06-12T02:31:21.119 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2020-06-12T02:31:21.213 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2020-06-12T02:31:21.326 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2020-06-12T02:31:21.430 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2020-06-12T02:31:21.524 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2020-06-12T02:31:21.643 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2020-06-12T02:31:21.747 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2020-06-12T02:31:21.858 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2020-06-12T02:31:21.958 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2020-06-12T02:31:22.055 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2020-06-12T02:31:22.175 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2020-06-12T02:31:22.259 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2020-06-12T02:31:22.375 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2020-06-12T02:31:22.470 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2020-06-12T02:31:22.587 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2020-06-12T02:31:22.678 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2020-06-12T02:31:22.798 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2020-06-12T02:31:22.882 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2020-06-12T02:31:23.001 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2020-06-12T02:31:23.100 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2020-06-12T02:31:23.194 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2020-06-12T02:31:23.314 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2020-06-12T02:31:23.420 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2020-06-12T02:31:23.539 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2020-06-12T02:31:23.624 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2020-06-12T02:31:23.733 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2020-06-12T02:31:23.825 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2020-06-12T02:31:23.947 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2020-06-12T02:31:23.949 INFO:teuthology.task.ansible.out: 2020-06-12T02:31:24.045 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2020-06-12T02:31:24.163 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2020-06-12T02:31:24.268 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-06-12T02:31:24.359 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-06-12T02:31:24.463 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2020-06-12T02:31:24.563 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2020-06-12T02:31:24.675 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2020-06-12T02:31:24.790 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2020-06-12T02:31:24.885 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2020-06-12T02:31:24.991 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2020-06-12T02:31:25.087 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2020-06-12T02:31:25.195 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2020-06-12T02:31:25.292 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2020-06-12T02:31:25.394 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2020-06-12T02:31:25.487 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2020-06-12T02:31:25.604 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2020-06-12T02:31:25.702 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2020-06-12T02:31:25.797 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2020-06-12T02:31:25.911 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2020-06-12T02:31:26.004 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2020-06-12T02:31:26.132 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2020-06-12T02:31:26.204 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2020-06-12T02:31:26.329 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2020-06-12T02:31:26.412 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2020-06-12T02:31:26.537 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2020-06-12T02:31:26.618 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2020-06-12T02:31:26.756 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2020-06-12T02:31:26.825 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2020-06-12T02:31:26.963 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2020-06-12T02:31:27.023 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2020-06-12T02:31:27.194 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2020-06-12T02:31:27.262 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2020-06-12T02:31:27.419 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2020-06-12T02:31:27.482 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2020-06-12T02:31:27.632 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2020-06-12T02:31:27.684 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2020-06-12T02:31:27.877 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2020-06-12T02:31:27.899 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2020-06-12T02:31:28.097 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2020-06-12T02:31:28.119 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2020-06-12T02:31:28.315 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2020-06-12T02:31:28.336 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2020-06-12T02:31:28.521 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2020-06-12T02:31:28.543 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2020-06-12T02:31:28.722 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2020-06-12T02:31:28.754 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2020-06-12T02:31:28.930 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2020-06-12T02:31:28.962 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2020-06-12T02:31:29.132 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2020-06-12T02:31:29.152 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2020-06-12T02:31:29.351 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2020-06-12T02:31:29.363 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2020-06-12T02:31:29.551 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2020-06-12T02:31:29.583 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2020-06-12T02:31:29.752 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2020-06-12T02:31:29.773 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2020-06-12T02:31:29.979 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2020-06-12T02:31:30.001 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2020-06-12T02:31:30.171 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2020-06-12T02:31:30.214 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2020-06-12T02:31:30.390 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2020-06-12T02:31:30.420 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2020-06-12T02:31:30.590 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2020-06-12T02:31:30.628 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2020-06-12T02:31:30.774 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2020-06-12T02:31:30.840 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2020-06-12T02:31:31.002 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2020-06-12T02:31:31.045 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2020-06-12T02:31:31.202 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2020-06-12T02:31:31.683 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarva 2020-06-12T02:31:31.685 INFO:teuthology.task.ansible.out:lhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2020-06-12T02:31:31.848 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2020-06-12T02:31:31.900 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2020-06-12T02:31:32.048 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2020-06-12T02:31:32.114 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2020-06-12T02:31:32.251 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2020-06-12T02:31:32.314 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2020-06-12T02:31:32.460 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2020-06-12T02:31:32.513 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2020-06-12T02:31:32.679 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2020-06-12T02:31:32.722 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2020-06-12T02:31:32.885 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2020-06-12T02:31:32.917 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2020-06-12T02:31:33.091 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1205} 2020-06-12T02:31:33.134 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1205} 2020-06-12T02:31:33.307 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1206} 2020-06-12T02:31:33.344 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1206} 2020-06-12T02:31:33.511 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1207} 2020-06-12T02:31:33.549 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1207} 2020-06-12T02:31:33.712 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1208} 2020-06-12T02:31:33.754 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1208} 2020-06-12T02:31:33.913 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1209} 2020-06-12T02:31:33.965 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1209} 2020-06-12T02:31:34.130 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1210} 2020-06-12T02:31:34.195 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1210} 2020-06-12T02:31:34.323 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1211} 2020-06-12T02:31:34.411 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1211} 2020-06-12T02:31:34.526 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1212} 2020-06-12T02:31:34.631 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1212} 2020-06-12T02:31:34.741 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1213} 2020-06-12T02:31:34.836 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1213} 2020-06-12T02:31:34.959 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1214} 2020-06-12T02:31:35.026 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1214} 2020-06-12T02:31:35.163 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1215} 2020-06-12T02:31:35.237 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1215} 2020-06-12T02:31:35.371 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1216} 2020-06-12T02:31:35.478 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1216} 2020-06-12T02:31:35.588 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1217} 2020-06-12T02:31:35.672 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1217} 2020-06-12T02:31:35.803 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1218} 2020-06-12T02:31:35.880 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1218} 2020-06-12T02:31:37.255 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "name": "sunny", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2020-06-12T02:31:37.371 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "name": "sunny", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2020-06-12T02:31:37.478 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:31:37 +0000 (0:00:26.936) 0:00:36.432 *********** 2020-06-12T02:31:37.522 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:31:37 +0000 (0:00:00.036) 0:00:36.468 *********** 2020-06-12T02:31:37.525 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-06-12T02:31:37.569 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:31:37 +0000 (0:00:00.054) 0:00:36.522 *********** 2020-06-12T02:31:37.622 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:31:37 +0000 (0:00:00.053) 0:00:36.575 *********** 2020-06-12T02:31:37.635 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2020-06-12T02:31:37.639 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 2020-06-12T02:31:37.640 INFO:teuthology.task.ansible.out: 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAx 2020-06-12T02:31:37.642 INFO:teuthology.task.ansible.out:j+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8ea 2020-06-12T02:31:37.644 INFO:teuthology.task.ansible.out:c8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f261 2020-06-12T02:31:37.645 INFO:teuthology.task.ansible.out:02d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd" 2020-06-12T02:31:37.646 INFO:teuthology.task.ansible.out:, "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomai 2020-06-12T02:31:37.648 INFO:teuthology.task.ansible.out:wang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redha 2020-06-12T02:31:37.649 INFO:teuthology.task.ansible.out:t.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMiv 2020-06-12T02:31:37.651 INFO:teuthology.task.ansible.out:DNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, {"name": "bengland", "ovpn": "beng 2020-06-12T02:31:37.652 INFO:teuthology.task.ansible.out:land@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name" 2020-06-12T02:31:37.654 INFO:teuthology.task.ansible.out:: "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643 2020-06-12T02:31:37.656 INFO:teuthology.task.ansible.out:eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AA 2020-06-12T02:31:37.656 INFO:teuthology.task.ansible.out:jMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4 2020-06-12T02:31:37.658 INFO:teuthology.task.ansible.out:yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}]}, "changed": false} 2020-06-12T02:31:37.697 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:31:37 +0000 (0:00:00.074) 0:00:36.650 *********** 2020-06-12T02:31:37.710 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 2020-06-12T02:31:37.712 INFO:teuthology.task.ansible.out: 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAx 2020-06-12T02:31:37.714 INFO:teuthology.task.ansible.out:j+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8ea 2020-06-12T02:31:37.716 INFO:teuthology.task.ansible.out:c8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f261 2020-06-12T02:31:37.717 INFO:teuthology.task.ansible.out:02d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomai 2020-06-12T02:31:37.719 INFO:teuthology.task.ansible.out:wang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redha 2020-06-12T02:31:37.720 INFO:teuthology.task.ansible.out:t.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4Uk 2020-06-12T02:31:37.723 INFO:teuthology.task.ansible.out:ibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMiv 2020-06-12T02:31:37.725 INFO:teuthology.task.ansible.out:DNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, {"name": "bengland", "ovpn": "beng 2020-06-12T02:31:37.727 INFO:teuthology.task.ansible.out:land@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name" 2020-06-12T02:31:37.728 INFO:teuthology.task.ansible.out:: "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643 2020-06-12T02:31:37.730 INFO:teuthology.task.ansible.out:eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez 2020-06-12T02:31:37.730 INFO:teuthology.task.ansible.out:75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4 2020-06-12T02:31:37.732 INFO:teuthology.task.ansible.out:yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}]}, "changed": false} 2020-06-12T02:31:37.775 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:31:37 +0000 (0:00:00.078) 0:00:36.728 *********** 2020-06-12T02:31:38.549 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2020-06-12T02:31:38.551 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com -> localhost] => {"after": "982b99592bd601b0475ee21e26576645605e4b52", "attempts": 1, "before": "982b99592bd601b0475ee21e26576645605e4b52", "changed": false, "remote_url_changed": false} 2020-06-12T02:31:38.598 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:31:38 +0000 (0:00:00.822) 0:00:37.551 *********** 2020-06-12T02:31:38.618 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com -> localhost] => {"after": "982b99592bd601b0475ee21e26576645605e4b52", "attempts": 1, "before": "982b99592bd601b0475ee21e26576645605e4b52", "changed": false, "remote_url_changed": false} 2020-06-12T02:31:38.655 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:31:38 +0000 (0:00:00.057) 0:00:37.609 *********** 2020-06-12T02:31:38.875 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-06-12T02:31:38.877 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2020-06-12T02:31:38.969 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2020-06-12T02:31:39.095 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2020-06-12T02:31:39.211 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2020-06-12T02:31:39.342 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2020-06-12T02:31:39.418 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2020-06-12T02:31:39.575 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2020-06-12T02:31:39.577 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2020-06-12T02:31:39.658 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2020-06-12T02:31:39.660 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2020-06-12T02:31:39.794 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2020-06-12T02:31:39.908 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2020-06-12T02:31:40.036 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2020-06-12T02:31:40.038 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2020-06-12T02:31:40.133 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2020-06-12T02:31:40.135 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2020-06-12T02:31:40.274 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2020-06-12T02:31:40.358 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2020-06-12T02:31:40.496 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2020-06-12T02:31:40.606 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2020-06-12T02:31:40.722 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2020-06-12T02:31:40.733 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:31:40.831 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2020-06-12T02:31:40.842 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:31:40.956 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2020-06-12T02:31:40.959 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2020-06-12T02:31:40.968 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'aemerson', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:31:40.978 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:31:40.980 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:31:41.055 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2020-06-12T02:31:41.058 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2020-06-12T02:31:41.065 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'aemerson', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:31:41.076 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVN 2020-06-12T02:31:41.078 INFO:teuthology.task.ansible.out:TUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:31:41.212 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2020-06-12T02:31:41.295 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2020-06-12T02:31:41.433 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2020-06-12T02:31:41.435 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2020-06-12T02:31:41.540 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2020-06-12T02:31:41.542 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2020-06-12T02:31:41.662 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2020-06-12T02:31:41.756 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2020-06-12T02:31:41.881 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2020-06-12T02:31:41.974 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2020-06-12T02:31:42.120 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2020-06-12T02:31:42.213 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2020-06-12T02:31:42.358 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2020-06-12T02:31:42.479 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2020-06-12T02:31:42.601 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2020-06-12T02:31:42.702 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2020-06-12T02:31:42.817 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2020-06-12T02:31:42.921 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2020-06-12T02:31:43.047 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2020-06-12T02:31:43.049 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2020-06-12T02:31:43.141 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2020-06-12T02:31:43.143 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2020-06-12T02:31:43.266 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2020-06-12T02:31:43.269 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2020-06-12T02:31:43.371 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2020-06-12T02:31:43.372 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2020-06-12T02:31:43.485 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2020-06-12T02:31:43.602 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2020-06-12T02:31:43.716 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2020-06-12T02:31:43.830 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2020-06-12T02:31:43.940 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2020-06-12T02:31:43.942 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2020-06-12T02:31:44.054 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2020-06-12T02:31:44.056 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2020-06-12T02:31:44.169 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2020-06-12T02:31:44.281 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2020-06-12T02:31:44.400 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2020-06-12T02:31:44.492 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2020-06-12T02:31:44.622 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2020-06-12T02:31:44.728 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2020-06-12T02:31:44.865 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2020-06-12T02:31:44.969 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2020-06-12T02:31:45.084 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2020-06-12T02:31:45.188 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2020-06-12T02:31:45.314 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2020-06-12T02:31:45.316 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2020-06-12T02:31:45.424 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2020-06-12T02:31:45.426 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2020-06-12T02:31:45.548 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2020-06-12T02:31:45.644 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2020-06-12T02:31:45.778 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2020-06-12T02:31:45.867 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2020-06-12T02:31:45.992 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2020-06-12T02:31:46.085 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2020-06-12T02:31:46.231 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2020-06-12T02:31:46.309 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2020-06-12T02:31:46.462 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2020-06-12T02:31:46.535 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2020-06-12T02:31:46.690 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2020-06-12T02:31:46.762 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2020-06-12T02:31:46.908 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-06-12T02:31:47.017 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-06-12T02:31:47.142 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2020-06-12T02:31:47.261 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2020-06-12T02:31:47.370 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDH/YT1mGZg50qDdngGQ4Ea+jfJGZ582J6pQbYkrvuGsyATMzNwk73sAbn7mgbpw7nkW/ol/vwahXHZ5heC056BFnqiZ7tjoDeotDx/1XxFtkbyIaVflUeD1vBww38v6IslLHcVhPilauPVZSfVzEHI7JB0a+9OwNrEH3+dt1B6vR+WRy2gHG/aEJffPMv4yv/0FgRWy3cEiUvGlLq0KVVTbsqqLQhw/u5k8bBIoibBtZLFAMTrjLYp9Dv8dxJROvmey6O2TuX6gCY1rLGTAArYULpXH73OLIqaksJKc3srgoJrTSZPO6bo3BQeLz4ODJ7uvTpvQIhacBpp/dq1ln7p owasserm@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCiqChxNdielMPBchs0u6bxWstqU9oeY2K9qf1PNwEIrwDYTrpDjsdoVnS750tOBjWY0wr9Gf9ATOJXi89151oao9GaUueQ1rTgmu7GtPlKsuBUVNMFzLEa0mqneqDA4AfknisFt/LnL8fLM4BddT25tmQ4oUGS7F17p4kXqwboK/DsJciCMWWc/5AT4NVwdnYFzIA6ta25/bB94kpktaZ6HDw7G8q/mK62Yza3EZfq8SP7p0sPL7oVsInQ9AutcVonmALgzPa9t1/MPGaXUIqARaWtwe4MSlNEIs4afLFyteBVeq+dgU01yeyjFivwVmuS+aPFTm6sjGGe8/gOsFDIJiVertOqi663m7q1WYGlVANfif7uL2+uJ9yrvOmrFh5s/CJMJBRG1SLfK1j/ivFwvPAQc/uMqEIRzrVZOsuNLPO4zHPWzmBAPUteb+2BqSdUFYMlBxb3lmhFFffroDqnOK1nbdAfmyKiosUizQgX2Gs2wjjtejuhS9dpYkNu+sBOujsxa2mZUouyWf4fXyWGVjthrCZaM+FyeSDpJrgwgRiSKNB8aRxB10ZNo9WZEy6eH2adp0Xf6v2KsKvryDndqBEkBI8zLbGrPVHGSEqiSBWc+NeAB06uSscJd0pHU3hDg0qW0EaxOXaial9qrBiwcFVKEr0g7LDURQ6KZsZy9w== owasserm@owasserm.redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2020-06-12T02:31:47.498 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDH/YT1mGZg50qDdngGQ4Ea+jfJGZ582J6pQbYkrvuGsyATMzNwk73sAbn7mgbpw7nkW/ol/vwahXHZ5heC056BFnqiZ7tjoDeotDx/1XxFtkbyIaVflUeD1vBww38v6IslLHcVhPilauPVZSfVzEHI7JB0a+9OwNrEH3+dt1B6vR+WRy2gHG/aEJffPMv4yv/0FgRWy3cEiUvGlLq0KVVTbsqqLQhw/u5k8bBIoibBtZLFAMTrjLYp9Dv8dxJROvmey6O2TuX6gCY1rLGTAArYULpXH73OLIqaksJKc3srgoJrTSZPO6bo3BQeLz4ODJ7uvTpvQIhacBpp/dq1ln7p owasserm@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCiqChxNdielMPBchs0u6bxWstqU9oeY2K9qf1PNwEIrwDYTrpDjsdoVnS750tOBjWY0wr9Gf9ATOJXi89151oao9GaUueQ1rTgmu7GtPlKsuBUVNMFzLEa0mqneqDA4AfknisFt/LnL8fLM4BddT25tmQ4oUGS7F17p4kXqwboK/DsJciCMWWc/5AT4NVwdnYFzIA6ta25/bB94kpktaZ6HDw7G8q/mK62Yza3EZfq8SP7p0sPL7oVsInQ9AutcVonmALgzPa9t1/MPGaXUIqARaWtwe4MSlNEIs4afLFyteBVeq+dgU01yeyjFivwVmuS+aPFTm6sjGGe8/gOsFDIJiVertOqi663m7q1WYGlVANfif7uL2+uJ9yrvOmrFh5s/CJMJBRG1SLfK1j/ivFwvPAQc/uMqEIRzrVZOsuNLPO4zHPWzmBAPUteb+2BqSdUFYMlBxb3lmhFFffroDqnOK1nbdAfmyKiosUizQgX2Gs2wjjtejuhS9dpYkNu+sBOujsxa2mZUouyWf4fXyWGVjthrCZaM+FyeSDpJrgwgRiSKNB8aRxB10ZNo9WZEy6eH2adp0Xf6v2KsKvryDndqBEkBI8zLbGrPVHGSEqiSBWc+NeAB06uSscJd0pHU3hDg0qW0EaxOXaial9qrBiwcFVKEr0g7LDURQ6KZsZy9w== owasserm@owasserm.redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2020-06-12T02:31:47.598 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2020-06-12T02:31:47.600 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2020-06-12T02:31:47.726 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2020-06-12T02:31:47.728 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2020-06-12T02:31:47.839 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2020-06-12T02:31:47.841 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2020-06-12T02:31:47.950 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2020-06-12T02:31:47.953 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2020-06-12T02:31:48.102 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2020-06-12T02:31:48.189 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2020-06-12T02:31:48.320 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2020-06-12T02:31:48.416 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2020-06-12T02:31:48.551 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2020-06-12T02:31:48.660 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2020-06-12T02:31:48.780 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2020-06-12T02:31:48.782 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-06-12T02:31:48.879 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2020-06-12T02:31:48.881 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-06-12T02:31:48.999 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2020-06-12T02:31:49.001 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2020-06-12T02:31:49.002 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2020-06-12T02:31:49.113 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2020-06-12T02:31:49.115 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2020-06-12T02:31:49.117 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2020-06-12T02:31:49.238 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2020-06-12T02:31:49.337 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2020-06-12T02:31:49.453 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2020-06-12T02:31:49.560 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2020-06-12T02:31:49.685 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2020-06-12T02:31:49.779 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2020-06-12T02:31:49.920 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2020-06-12T02:31:50.024 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2020-06-12T02:31:50.150 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2020-06-12T02:31:50.244 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2020-06-12T02:31:50.384 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2020-06-12T02:31:50.388 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2020-06-12T02:31:50.486 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2020-06-12T02:31:50.488 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2020-06-12T02:31:50.623 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2020-06-12T02:31:50.689 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2020-06-12T02:31:50.856 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEudQMGBPqfZBV8bXpdQOgVTDP6GFsr5oL4gdbHFMuJCGe0Sfq8LTaSEBQx0MH7tXc7JJ1fU51U7DSt0kfy/PFeV1Y5XnNATiZu/SzdO3jHe06/8CCr3DLnlEmzXASq+tqDJbY1eV011d/TX1jiC8Qd1VNbpdZYExEDsDzwJXnWdh6rjTL4anGjIgQWcmU0XdbUJXvRP4dGjVvZVHZnflAkxLPBbAx/8BBOuBmv6vP4hNs4poi4VLwR6S6Ha1fgMJmsn6StsCVuOvKBJFuGvaulHgHgdoDJ1ryUFikkXXCWoNx4IbwtpQIkSxWMuAovlwwk9G8ozSs+RrFzCavE0f omw@omw-dev", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2020-06-12T02:31:50.919 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEudQMGBPqfZBV8bXpdQOgVTDP6GFsr5oL4gdbHFMuJCGe0Sfq8LTaSEBQx0MH7tXc7JJ1fU51U7DSt0kfy/PFeV1Y5XnNATiZu/SzdO3jHe06/8CCr3DLnlEmzXASq+tqDJbY1eV011d/TX1jiC8Qd1VNbpdZYExEDsDzwJXnWdh6rjTL4anGjIgQWcmU0XdbUJXvRP4dGjVvZVHZnflAkxLPBbAx/8BBOuBmv6vP4hNs4poi4VLwR6S6Ha1fgMJmsn6StsCVuOvKBJFuGvaulHgHgdoDJ1ryUFikkXXCWoNx4IbwtpQIkSxWMuAovlwwk9G8ozSs+RrFzCavE0f omw@omw-dev", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2020-06-12T02:31:51.077 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2020-06-12T02:31:51.079 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2020-06-12T02:31:51.140 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2020-06-12T02:31:51.142 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2020-06-12T02:31:51.299 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2020-06-12T02:31:51.367 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2020-06-12T02:31:51.535 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2020-06-12T02:31:51.581 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2020-06-12T02:31:51.775 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2020-06-12T02:31:51.777 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2020-06-12T02:31:51.814 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2020-06-12T02:31:51.815 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2020-06-12T02:31:51.993 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2020-06-12T02:31:52.035 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2020-06-12T02:31:52.213 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2020-06-12T02:31:52.245 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2020-06-12T02:31:52.434 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2020-06-12T02:31:52.461 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2020-06-12T02:31:52.665 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-06-12T02:31:52.692 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-06-12T02:31:52.891 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2020-06-12T02:31:52.916 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2020-06-12T02:31:53.121 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2020-06-12T02:31:53.144 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2020-06-12T02:31:53.345 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2020-06-12T02:31:53.355 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2020-06-12T02:31:53.563 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2020-06-12T02:31:53.585 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2020-06-12T02:31:53.774 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2020-06-12T02:31:53.826 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2020-06-12T02:31:54.005 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2020-06-12T02:31:54.050 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2020-06-12T02:31:54.236 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2020-06-12T02:31:54.268 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2020-06-12T02:31:54.476 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2020-06-12T02:31:54.488 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2020-06-12T02:31:54.714 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2020-06-12T02:31:54.716 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2020-06-12T02:31:54.945 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ 2020-06-12T02:31:54.947 INFO:teuthology.task.ansible.out:7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2020-06-12T02:31:55.172 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9/Io6M3q8yQqoPNPKAPJ6Fzq18DMfmNtxcD8GUWNo0xcMxd7XOlarvYc7INDQjkpzx6s+aqMme55pSUZuN5plW+nuFmRPqBaIxyeCnNjzCch0wbtEZe5dzT270FhzM1Tu4W0dw+WD6zrGjXayLNs2uxNp3UrWtMTBRzd0IfxJ1R2aJtiCE/HNxIZn+bhS/OmEPV1izz23MvKXD+nRaeJK6Aqrg1zbnpoNskz2fZ/AnmhnAMc4EHl/NUyW1mi4g81KcQMwE8poWmw1NpZje9gWFcISVClLvvhk/A6at/v90P7PisX+/ulTA/QYJYvGBjgbNtURYu/PWzYBOAEZQGfj shilpa@dhcp35-184.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqRSG3i8QY7t9sp6I8vDZ7LVdk+DDrMhI1QmvU2z+YQhUGDBOvvO6YP0gUXPDyalyj0sJXiwLWPmkrOsbnR96YCUcIt7Ba6R0d0w85JxcJ1WyPcsg2UyOBwq4A+s7VAFZm6FEWzlaubixsNp04gKDqBjSjaqA1Kwx8/Tg39byay8Hn3VJeZ1SkRukvhykz2e6gl7ydfuJQVmEyO3pTrd5157ZqgvQRABT10wc3vR2m4eC65hU/v0ci7HMY9oVtGEdWVBapWMuiocrOPCRWF5zrMJwW4vzPOCHa/47hBEIWfOWJidUVzAawnM3nOe7PykvvDK/iP8B6JpxKTE74eLeh shilpa@fedora.lab.eng.blr.redhat.com", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2020-06-12T02:31:55.202 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9/Io6M3q8yQqoPNPKAPJ6Fzq18DMfmNtxcD8GUWNo0xcMxd7XOlarvYc7INDQjkpzx6s+aqMme55pSUZuN5plW+nuFmRPqBaIxyeCnNjzCch0wbtEZe5dzT270FhzM1Tu4W0dw+WD6zrGjXayLNs2uxNp3UrWtMTBRzd0IfxJ1R2aJtiCE/HNxIZn+bhS/OmEPV1izz23MvKXD+nRaeJK6Aqrg1zbnpoNskz2fZ/AnmhnAMc4EHl/NUyW1mi4g81KcQMwE8poWmw1NpZje9gWFcISVClLvvhk/A6at/v90P7PisX+/ulTA/QYJYvGBjgbNtURYu/PWzYBOAEZQGfj shilpa@dhcp35-184.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqRSG3i8QY7t9sp6I8vDZ7LVdk+DDrMhI1QmvU2z+YQhUGDBOvvO6YP0gUXPDyalyj0sJXiwLWPmkrOsbnR96YCUcIt7Ba6R0d0w85JxcJ1WyPcsg2UyOBwq4A+s7VAFZm6FEWzlaubixsNp04gKDqBjSjaqA1Kwx8/Tg39byay8Hn3VJeZ1SkRukvhykz2e6gl7ydfuJQVmEyO3pTrd5157ZqgvQRABT10wc3vR2m4eC65hU/v0ci7HMY9oVtGEdWVBapWMuiocrOPCRWF5zrMJwW4vzPOCHa/47hBEIWfOWJidUVzAawnM3nOe7PykvvDK/iP8B6JpxKTE74eLeh shilpa@fedora.lab.eng.blr.redhat.com", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2020-06-12T02:31:55.384 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2020-06-12T02:31:55.428 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2020-06-12T02:31:55.617 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2020-06-12T02:31:55.672 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2020-06-12T02:31:55.843 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2020-06-12T02:31:55.895 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2020-06-12T02:31:56.083 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2020-06-12T02:31:56.127 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2020-06-12T02:31:56.319 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2020-06-12T02:31:56.358 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2020-06-12T02:31:56.544 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2020-06-12T02:31:56.597 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2020-06-12T02:31:56.772 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2020-06-12T02:31:56.814 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2020-06-12T02:31:56.992 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2020-06-12T02:31:57.023 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2020-06-12T02:31:57.212 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2020-06-12T02:31:57.256 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2020-06-12T02:31:57.437 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2020-06-12T02:31:57.469 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2020-06-12T02:31:57.663 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2020-06-12T02:31:57.665 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2020-06-12T02:31:57.722 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2020-06-12T02:31:57.724 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2020-06-12T02:31:57.893 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2020-06-12T02:31:57.895 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2020-06-12T02:31:57.929 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2020-06-12T02:31:57.931 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2020-06-12T02:31:58.127 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2020-06-12T02:31:58.148 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2020-06-12T02:31:58.368 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2020-06-12T02:31:58.369 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2020-06-12T02:31:58.583 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2020-06-12T02:31:58.598 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2020-06-12T02:31:58.807 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2020-06-12T02:31:58.829 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2020-06-12T02:31:59.036 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2020-06-12T02:31:59.047 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2020-06-12T02:31:59.271 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2020-06-12T02:31:59.286 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2020-06-12T02:31:59.498 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaY 2020-06-12T02:31:59.500 INFO:teuthology.task.ansible.out:kCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2020-06-12T02:31:59.735 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2020-06-12T02:31:59.759 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2020-06-12T02:31:59.988 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2020-06-12T02:32:00.010 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2020-06-12T02:32:00.234 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2020-06-12T02:32:00.243 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2020-06-12T02:32:00.456 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2020-06-12T02:32:00.467 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2020-06-12T02:32:00.679 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2020-06-12T02:32:00.714 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2020-06-12T02:32:00.916 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2020-06-12T02:32:00.939 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2020-06-12T02:32:01.137 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2020-06-12T02:32:01.161 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2020-06-12T02:32:01.398 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2020-06-12T02:32:01.400 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+k 2020-06-12T02:32:01.401 INFO:teuthology.task.ansible.out:nEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-06-12T02:32:01.618 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlnlBzRdWRqns/9VrYuF9tLILUjeGd9z8Tf/dPZmxAt3nnZqpTvI2xXemQpyt/2lsD0U24EuTc1Y/dBsgqvXtZOw51TZE8RK26hvCCaKGfcKONjvR7goHAmkYPHK9+F6m7fgYkRzA0+ZW4HcWNh/xxhhL2G1EGfz5bzIoGQJHSdzc6VeVIsY7HiYEMTxvxdsgV72Kjq97BdmHm56RBzoH24oN0UzNZS8Syx5SuWj9dA3IyFBPwjLlozSk6U2Ofd7qaXuQKl807eD9g6RhRldzHPS+i0EK3EM6Jvho5Y08k1KqV4gIEwMdzDbD3SH0xq+nUPFKcDTUFNp+tkg10GxqN zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2020-06-12T02:32:01.620 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlnlBzRdWRqns/9VrYuF9tLILUjeGd9z8Tf/dPZmxAt3nnZqpTvI2xXemQpyt/2lsD0U24EuTc1Y/dBsgqvXtZOw51TZE8RK26hvCCaKGfcKONjvR7goHAmkYPHK9+F6m7fgYkRzA0+ZW4HcWNh/xxhhL2G1EGfz5bzIoGQJHSdzc6VeVIsY7HiYEMTxvxdsgV72Kjq97BdmHm56RBzoH24oN0UzNZS8Syx5SuWj9dA3IyFBPwjLlozSk6U2Ofd7qaXuQKl807eD9g6RhRldzHPS+i0EK3EM6Jvho5Y08k1KqV4gIEwMdzDbD3SH0xq+nUPFKcDTUFNp+tkg10GxqN zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2020-06-12T02:32:01.841 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2020-06-12T02:32:01.843 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2020-06-12T02:32:02.070 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": 2020-06-12T02:32:02.072 INFO:teuthology.task.ansible.out: true} 2020-06-12T02:32:02.291 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2020-06-12T02:32:02.297 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2020-06-12T02:32:02.514 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2020-06-12T02:32:02.546 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2020-06-12T02:32:02.741 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2020-06-12T02:32:02.763 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2020-06-12T02:32:02.974 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2020-06-12T02:32:02.985 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2020-06-12T02:32:03.203 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2020-06-12T02:32:03.214 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2020-06-12T02:32:03.442 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2020-06-12T02:32:03.454 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2020-06-12T02:32:03.664 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2020-06-12T02:32:03.685 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2020-06-12T02:32:03.910 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2020-06-12T02:32:03.921 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2020-06-12T02:32:04.122 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2020-06-12T02:32:04.164 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2020-06-12T02:32:04.343 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2020-06-12T02:32:04.416 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2020-06-12T02:32:04.566 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2020-06-12T02:32:04.657 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2020-06-12T02:32:04.798 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2020-06-12T02:32:04.881 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2020-06-12T02:32:05.023 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2020-06-12T02:32:05.114 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2020-06-12T02:32:05.234 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2020-06-12T02:32:05.370 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2020-06-12T02:32:05.465 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2020-06-12T02:32:05.549 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:05 +0000 (0:00:26.893) 0:01:04.502 *********** 2020-06-12T02:32:05.607 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2020-06-12T02:32:05.679 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-06-12T02:32:05.681 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ 2020-06-12T02:32:05.683 INFO:teuthology.task.ansible.out:ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcN 2020-06-12T02:32:05.685 INFO:teuthology.task.ansible.out:E/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:05.715 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:05 +0000 (0:00:00.166) 0:01:04.669 *********** 2020-06-12T02:32:05.727 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:05.729 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:05.775 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:05.784 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:05.795 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:05.806 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:05.808 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:05.817 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:05.828 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:05.839 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:05.926 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2020-06-12T02:32:05.935 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.049 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2020-06-12T02:32:06.060 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.160 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'aemerson', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemerson", "validate_certs": true} 2020-06-12T02:32:06.327 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'aemerson', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemerson", "validate_certs": true} 2020-06-12T02:32:06.379 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2020-06-12T02:32:06.539 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2020-06-12T02:32:06.603 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2020-06-12T02:32:06.614 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.625 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.627 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.636 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.647 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.649 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.658 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.670 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.680 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.700 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.713 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.715 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.739 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.740 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.742 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.755 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.766 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} ok: [smithi188.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b2 2020-06-12T02:32:06.768 INFO:teuthology.task.ansible.out:68bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.778 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.780 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.791 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.802 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.814 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_ 2020-06-12T02:32:06.816 INFO:teuthology.task.ansible.out:var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.826 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.829 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.855 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.856 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.857 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.859 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.865 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.866 INFO:teuthology.task.ansible.out: skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.867 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.867 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.869 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.874 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.875 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.886 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.888 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.889 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.891 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.891 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.903 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.904 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.905 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.916 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.927 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.929 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.930 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.932 INFO:teuthology.task.ansible.out: 2020-06-12T02:32:06.933 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.935 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.935 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.937 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.949 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.949 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.950 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.950 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.961 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.963 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.973 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.974 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.990 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:06.997 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.009 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.011 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.020 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.022 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.032 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.034 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.035 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.046 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.047 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.057 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.059 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.068 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.087 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.098 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.109 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.111 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": { 2020-06-12T02:32:07.113 INFO:teuthology.task.ansible.out:"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.113 INFO:teuthology.task.ansible.out: 2020-06-12T02:32:07.114 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'al 2020-06-12T02:32:07.115 INFO:teuthology.task.ansible.out:fonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.124 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.126 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.137 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.138 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.149 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.151 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.161 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.163 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.173 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.173 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.185 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.187 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.198 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.200 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.211 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.213 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.223 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.224 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.235 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.246 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.249 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.258 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.260 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.270 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.272 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.283 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.293 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.309 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "c 2020-06-12T02:32:07.311 INFO:teuthology.task.ansible.out:hanged": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.318 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.320 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.320 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.331 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.332 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.342 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.344 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.366 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.368 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.383 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.384 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.385 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "ss 2020-06-12T02:32:07.387 INFO:teuthology.task.ansible.out:eshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.396 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.397 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.407 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.409 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.419 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.421 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.438 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.440 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.473 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:07 +0000 (0:00:01.758) 0:01:06.427 *********** 2020-06-12T02:32:07.486 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.488 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.489 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.493 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn 2020-06-12T02:32:07.494 INFO:teuthology.task.ansible.out:": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.510 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.515 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.518 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.553 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:07 +0000 (0:00:00.079) 0:01:06.506 *********** 2020-06-12T02:32:07.565 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.568 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyr 2020-06-12T02:32:07.572 INFO:teuthology.task.ansible.out:atsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.579 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.582 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.593 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.645 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:07 +0000 (0:00:00.092) 0:01:06.598 *********** 2020-06-12T02:32:07.676 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2020-06-12T02:32:07.678 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:07.712 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:07 +0000 (0:00:00.066) 0:01:06.665 *********** 2020-06-12T02:32:07.817 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-06-12T02:32:07.819 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2020-06-12T02:32:08.036 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2020-06-12T02:32:08.055 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2020-06-12T02:32:08.264 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2020-06-12T02:32:08.275 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2020-06-12T02:32:08.464 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2020-06-12T02:32:08.486 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2020-06-12T02:32:08.684 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2020-06-12T02:32:08.686 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2020-06-12T02:32:08.886 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2020-06-12T02:32:08.897 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2020-06-12T02:32:09.107 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2020-06-12T02:32:09.122 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2020-06-12T02:32:09.301 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2020-06-12T02:32:09.322 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2020-06-12T02:32:09.514 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} ok: [smithi107.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2020-06-12T02:32:09.726 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2020-06-12T02:32:09.748 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2020-06-12T02:32:09.937 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2020-06-12T02:32:09.948 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2020-06-12T02:32:10.145 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2020-06-12T02:32:10.156 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2020-06-12T02:32:10.342 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2020-06-12T02:32:10.354 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2020-06-12T02:32:10.543 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2020-06-12T02:32:10.575 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2020-06-12T02:32:10.742 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2020-06-12T02:32:10.774 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2020-06-12T02:32:10.950 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2020-06-12T02:32:10.982 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2020-06-12T02:32:11.141 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2020-06-12T02:32:11.173 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2020-06-12T02:32:11.346 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2020-06-12T02:32:11.393 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2020-06-12T02:32:11.537 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2020-06-12T02:32:11.600 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2020-06-12T02:32:11.737 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2020-06-12T02:32:11.802 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2020-06-12T02:32:11.937 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2020-06-12T02:32:12.000 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2020-06-12T02:32:12.135 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2020-06-12T02:32:12.219 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2020-06-12T02:32:12.344 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2020-06-12T02:32:12.422 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2020-06-12T02:32:12.546 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2020-06-12T02:32:12.650 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2020-06-12T02:32:12.753 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2020-06-12T02:32:12.837 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2020-06-12T02:32:12.949 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2020-06-12T02:32:13.046 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2020-06-12T02:32:13.146 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2020-06-12T02:32:13.252 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2020-06-12T02:32:13.371 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2020-06-12T02:32:13.449 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2020-06-12T02:32:13.580 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2020-06-12T02:32:13.656 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2020-06-12T02:32:13.784 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2020-06-12T02:32:13.895 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2020-06-12T02:32:13.976 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2020-06-12T02:32:14.090 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2020-06-12T02:32:14.175 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2020-06-12T02:32:14.311 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2020-06-12T02:32:14.384 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2020-06-12T02:32:14.522 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2020-06-12T02:32:14.581 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2020-06-12T02:32:14.728 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2020-06-12T02:32:14.791 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2020-06-12T02:32:14.926 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2020-06-12T02:32:15.001 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2020-06-12T02:32:15.128 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2020-06-12T02:32:15.211 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2020-06-12T02:32:15.334 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2020-06-12T02:32:15.410 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2020-06-12T02:32:15.541 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2020-06-12T02:32:15.605 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2020-06-12T02:32:15.751 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2020-06-12T02:32:15.825 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2020-06-12T02:32:15.950 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2020-06-12T02:32:16.033 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2020-06-12T02:32:16.149 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2020-06-12T02:32:16.238 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2020-06-12T02:32:16.350 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2020-06-12T02:32:16.446 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2020-06-12T02:32:16.503 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:16 +0000 (0:00:08.791) 0:01:15.457 *********** 2020-06-12T02:32:16.556 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2020-06-12T02:32:16.745 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2020-06-12T02:32:16.815 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:16 +0000 (0:00:00.310) 0:01:15.767 *********** 2020-06-12T02:32:17.070 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2020-06-12T02:32:17.072 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "checksum": "f166a1846201ae42f076bd651ad63f80375d8a1b", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "594f3164ffc654444ae430b4df4334d6", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1591929136.538742-61608108806457/source", "state": "file", "uid": 0} 2020-06-12T02:32:17.106 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:17 +0000 (0:00:00.292) 0:01:16.059 *********** 2020-06-12T02:32:17.135 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2020-06-12T02:32:17.137 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => { "msg": "Host smithi107.front.sepia.ceph.com is running CentOS 8.1 (Core)" } 2020-06-12T02:32:17.172 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:17 +0000 (0:00:00.066) 0:01:16.125 *********** 2020-06-12T02:32:17.213 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-06-12T02:32:17.215 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2020-06-12T02:32:17.259 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:17 +0000 (0:00:00.086) 0:01:16.212 *********** 2020-06-12T02:32:17.299 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-06-12T02:32:17.301 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:17.333 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:17 +0000 (0:00:00.073) 0:01:16.286 *********** 2020-06-12T02:32:17.430 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2020-06-12T02:32:17.433 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => {"changed": true, "checksum": "f166a1846201ae42f076bd651ad63f80375d8a1b", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "594f3164ffc654444ae430b4df4334d6", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1591929136.855342-280815395150710/source", "state": "file", "uid": 0} 2020-06-12T02:32:17.468 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:17 +0000 (0:00:00.134) 0:01:16.421 *********** 2020-06-12T02:32:17.507 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2020-06-12T02:32:17.509 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => { "msg": "Host smithi188.front.sepia.ceph.com is running CentOS 8.1 (Core)" } 2020-06-12T02:32:17.542 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:17 +0000 (0:00:00.074) 0:01:16.495 *********** 2020-06-12T02:32:17.582 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** ok: [smithi188.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2020-06-12T02:32:17.622 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:17 +0000 (0:00:00.076) 0:01:16.572 *********** 2020-06-12T02:32:17.669 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-06-12T02:32:17.670 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:17.702 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:17 +0000 (0:00:00.083) 0:01:16.655 *********** 2020-06-12T02:32:17.905 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-06-12T02:32:17.907 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.349648", "end": "2020-06-12 02:32:17.880646", "rc": 0, "start": "2020-06-12 02:32:17.530998", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-06-12T02:32:17.940 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:17 +0000 (0:00:00.238) 0:01:16.893 *********** 2020-06-12T02:32:17.985 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-06-12T02:32:17.987 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:18.026 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:18 +0000 (0:00:00.085) 0:01:16.979 *********** 2020-06-12T02:32:18.056 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-06-12T02:32:18.058 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:18.100 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:18 +0000 (0:00:00.074) 0:01:17.053 *********** 2020-06-12T02:32:18.139 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-06-12T02:32:18.141 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:18.175 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:18 +0000 (0:00:00.075) 0:01:17.128 *********** 2020-06-12T02:32:18.215 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-06-12T02:32:18.217 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:18.256 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:18 +0000 (0:00:00.080) 0:01:17.209 *********** 2020-06-12T02:32:18.265 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-06-12T02:32:18.267 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.312160", "end": "2020-06-12 02:32:18.228665", "rc": 0, "start": "2020-06-12 02:32:17.916505", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-06-12T02:32:18.302 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:18 +0000 (0:00:00.046) 0:01:17.255 *********** 2020-06-12T02:32:18.311 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2020-06-12T02:32:18.313 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:18.361 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:18 +0000 (0:00:00.058) 0:01:17.314 *********** 2020-06-12T02:32:18.370 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-06-12T02:32:18.373 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:18.406 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:18 +0000 (0:00:00.045) 0:01:17.360 *********** 2020-06-12T02:32:18.415 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-06-12T02:32:18.417 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:18.458 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:18 +0000 (0:00:00.043) 0:01:17.403 *********** 2020-06-12T02:32:18.461 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:18.493 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:18 +0000 (0:00:00.042) 0:01:17.446 *********** 2020-06-12T02:32:18.500 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-06-12T02:32:18.502 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi107.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-06-12T02:32:18.536 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:18 +0000 (0:00:00.043) 0:01:17.489 *********** 2020-06-12T02:32:18.544 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-06-12T02:32:18.546 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:18.578 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:18 +0000 (0:00:00.042) 0:01:17.531 *********** 2020-06-12T02:32:18.586 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2020-06-12T02:32:18.587 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:18.623 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:18 +0000 (0:00:00.043) 0:01:17.575 *********** 2020-06-12T02:32:18.629 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-06-12T02:32:18.632 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:18.664 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:18 +0000 (0:00:00.042) 0:01:17.617 *********** 2020-06-12T02:32:18.671 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2020-06-12T02:32:18.672 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:18.705 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:18 +0000 (0:00:00.041) 0:01:17.658 *********** 2020-06-12T02:32:18.724 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2020-06-12T02:32:18.725 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:18.762 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:18 +0000 (0:00:00.051) 0:01:17.710 *********** 2020-06-12T02:32:18.765 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-06-12T02:32:18.766 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-06-12T02:32:18.799 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:18 +0000 (0:00:00.043) 0:01:17.753 *********** 2020-06-12T02:32:18.809 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:18.843 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:18 +0000 (0:00:00.043) 0:01:17.796 *********** 2020-06-12T02:32:18.854 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-06-12T02:32:18.857 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:18.888 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:18 +0000 (0:00:00.044) 0:01:17.841 *********** 2020-06-12T02:32:18.897 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-06-12T02:32:18.899 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-06-12T02:32:18.910 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-06-12T02:32:18.942 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:18 +0000 (0:00:00.054) 0:01:17.895 *********** 2020-06-12T02:32:18.950 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2020-06-12T02:32:18.952 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-06-12T02:32:18.988 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:18 +0000 (0:00:00.045) 0:01:17.941 *********** 2020-06-12T02:32:18.997 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2020-06-12T02:32:18.999 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:19.032 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:19 +0000 (0:00:00.044) 0:01:17.985 *********** 2020-06-12T02:32:19.043 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:19.075 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:19 +0000 (0:00:00.042) 0:01:18.028 *********** 2020-06-12T02:32:19.083 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2020-06-12T02:32:19.085 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:19.125 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:19 +0000 (0:00:00.043) 0:01:18.072 *********** 2020-06-12T02:32:19.127 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-06-12T02:32:19.128 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:19.163 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:19 +0000 (0:00:00.044) 0:01:18.116 *********** 2020-06-12T02:32:19.182 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-06-12T02:32:19.184 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-06-12T02:32:19.218 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:19 +0000 (0:00:00.052) 0:01:18.169 *********** 2020-06-12T02:32:19.224 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-06-12T02:32:19.225 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:19.259 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:19 +0000 (0:00:00.043) 0:01:18.212 *********** 2020-06-12T02:32:19.267 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-06-12T02:32:19.269 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:19.302 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:19 +0000 (0:00:00.043) 0:01:18.256 *********** 2020-06-12T02:32:19.312 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-06-12T02:32:19.314 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:19.347 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:19 +0000 (0:00:00.045) 0:01:18.301 *********** 2020-06-12T02:32:19.357 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2020-06-12T02:32:19.358 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-06-12T02:32:19.395 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:19 +0000 (0:00:00.047) 0:01:18.348 *********** 2020-06-12T02:32:19.407 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* skipping: [smithi107.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-06-12T02:32:19.448 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:19 +0000 (0:00:00.048) 0:01:18.397 *********** 2020-06-12T02:32:19.453 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2020-06-12T02:32:19.455 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:19.490 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:19 +0000 (0:00:00.045) 0:01:18.443 *********** 2020-06-12T02:32:19.498 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-06-12T02:32:19.500 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:19.535 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:19 +0000 (0:00:00.044) 0:01:18.488 *********** 2020-06-12T02:32:19.544 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-06-12T02:32:19.546 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:19.584 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:19 +0000 (0:00:00.050) 0:01:18.538 *********** 2020-06-12T02:32:19.593 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-06-12T02:32:19.595 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:19.630 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:19 +0000 (0:00:00.045) 0:01:18.583 *********** 2020-06-12T02:32:19.640 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-06-12T02:32:19.642 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:19.682 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:19 +0000 (0:00:00.048) 0:01:18.632 *********** 2020-06-12T02:32:19.688 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2020-06-12T02:32:19.690 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:19.723 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:19 +0000 (0:00:00.044) 0:01:18.677 *********** 2020-06-12T02:32:19.732 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-06-12T02:32:19.737 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:19.767 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:19 +0000 (0:00:00.043) 0:01:18.720 *********** 2020-06-12T02:32:19.777 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2020-06-12T02:32:19.781 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:19.826 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:19 +0000 (0:00:00.058) 0:01:18.779 *********** 2020-06-12T02:32:19.835 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* skipping: [smithi188.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-06-12T02:32:19.871 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:19 +0000 (0:00:00.045) 0:01:18.825 *********** 2020-06-12T02:32:19.881 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:19.919 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:19 +0000 (0:00:00.047) 0:01:18.873 *********** 2020-06-12T02:32:19.928 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-06-12T02:32:19.930 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:19.967 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:19 +0000 (0:00:00.047) 0:01:18.920 *********** 2020-06-12T02:32:19.986 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2020-06-12T02:32:19.988 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:20.028 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:20 +0000 (0:00:00.058) 0:01:18.978 *********** 2020-06-12T02:32:20.034 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-06-12T02:32:20.036 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:20.076 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:20 +0000 (0:00:00.050) 0:01:19.029 *********** 2020-06-12T02:32:20.085 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2020-06-12T02:32:20.087 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-06-12T02:32:20.124 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:20 +0000 (0:00:00.048) 0:01:19.078 *********** 2020-06-12T02:32:20.134 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:20.172 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:20 +0000 (0:00:00.047) 0:01:19.125 *********** 2020-06-12T02:32:20.179 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-06-12T02:32:20.181 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:20.219 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:20 +0000 (0:00:00.047) 0:01:19.173 *********** 2020-06-12T02:32:20.229 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2020-06-12T02:32:20.230 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:20.271 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:20 +0000 (0:00:00.051) 0:01:19.225 *********** 2020-06-12T02:32:20.280 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-06-12T02:32:20.282 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:20.316 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:20 +0000 (0:00:00.044) 0:01:19.269 *********** 2020-06-12T02:32:20.325 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2020-06-12T02:32:20.327 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:20.360 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:20 +0000 (0:00:00.044) 0:01:19.314 *********** 2020-06-12T02:32:20.416 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:20.454 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:20 +0000 (0:00:00.093) 0:01:19.407 *********** 2020-06-12T02:32:20.497 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** skipping: [smithi188.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-06-12T02:32:20.533 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:20 +0000 (0:00:00.079) 0:01:19.486 *********** 2020-06-12T02:32:20.552 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-06-12T02:32:20.555 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-06-12T02:32:20.588 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:20 +0000 (0:00:00.054) 0:01:19.541 *********** 2020-06-12T02:32:20.599 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-06-12T02:32:20.601 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:20.640 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:20 +0000 (0:00:00.051) 0:01:19.593 *********** 2020-06-12T02:32:20.685 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-06-12T02:32:20.687 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:20.722 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:20 +0000 (0:00:00.081) 0:01:19.675 *********** 2020-06-12T02:32:20.952 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-06-12T02:32:20.954 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-06-12T02:32:20.986 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:20 +0000 (0:00:00.264) 0:01:19.940 *********** 2020-06-12T02:32:21.352 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-06-12T02:32:21.355 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2020-06-12T02:32:21.572 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2020-06-12T02:32:21.856 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2020-06-12T02:32:21.896 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:21 +0000 (0:00:00.909) 0:01:20.849 *********** 2020-06-12T02:32:22.112 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2020-06-12T02:32:22.147 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:22 +0000 (0:00:00.251) 0:01:21.101 *********** 2020-06-12T02:32:22.418 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* ok: [smithi107.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2020-06-12T02:32:22.693 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2020-06-12T02:32:22.915 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2020-06-12T02:32:22.951 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:22 +0000 (0:00:00.803) 0:01:21.904 *********** 2020-06-12T02:32:23.010 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-06-12T02:32:23.012 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:23.046 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:23 +0000 (0:00:00.095) 0:01:21.999 *********** 2020-06-12T02:32:23.087 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:23.120 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:23 +0000 (0:00:00.074) 0:01:22.074 *********** 2020-06-12T02:32:23.163 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:23.194 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:23 +0000 (0:00:00.073) 0:01:22.148 *********** 2020-06-12T02:32:23.213 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-06-12T02:32:23.215 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2020-06-12T02:32:23.253 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:23 +0000 (0:00:00.058) 0:01:22.206 *********** 2020-06-12T02:32:23.268 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-06-12T02:32:23.269 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:23.308 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:23 +0000 (0:00:00.055) 0:01:22.262 *********** 2020-06-12T02:32:23.318 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-06-12T02:32:23.320 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:23.355 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:23 +0000 (0:00:00.046) 0:01:22.308 *********** 2020-06-12T02:32:23.364 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-06-12T02:32:23.366 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:23.401 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:23 +0000 (0:00:00.046) 0:01:22.354 *********** 2020-06-12T02:32:23.410 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-06-12T02:32:23.412 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:23.450 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:23 +0000 (0:00:00.049) 0:01:22.404 *********** 2020-06-12T02:32:23.461 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-06-12T02:32:23.466 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:23.506 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:23 +0000 (0:00:00.055) 0:01:22.459 *********** 2020-06-12T02:32:23.516 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-06-12T02:32:23.518 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:23.567 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:23 +0000 (0:00:00.061) 0:01:22.521 *********** 2020-06-12T02:32:23.577 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-06-12T02:32:23.579 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:23.620 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:23 +0000 (0:00:00.052) 0:01:22.573 *********** 2020-06-12T02:32:23.629 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-06-12T02:32:23.630 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:23.670 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:23 +0000 (0:00:00.050) 0:01:22.623 *********** 2020-06-12T02:32:23.683 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:23.721 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:23 +0000 (0:00:00.050) 0:01:22.674 *********** 2020-06-12T02:32:23.731 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-06-12T02:32:23.733 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:23.772 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:23 +0000 (0:00:00.051) 0:01:22.726 *********** 2020-06-12T02:32:23.782 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-06-12T02:32:23.784 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:23.822 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:23 +0000 (0:00:00.050) 0:01:22.776 *********** 2020-06-12T02:32:23.836 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-06-12T02:32:23.838 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:23.880 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:23 +0000 (0:00:00.057) 0:01:22.834 *********** 2020-06-12T02:32:23.930 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-06-12T02:32:23.931 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:23.969 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:23 +0000 (0:00:00.088) 0:01:22.922 *********** 2020-06-12T02:32:24.005 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-06-12T02:32:24.006 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:24.041 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:24 +0000 (0:00:00.072) 0:01:22.994 *********** 2020-06-12T02:32:24.070 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-06-12T02:32:24.072 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:24.113 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:24 +0000 (0:00:00.071) 0:01:23.066 *********** 2020-06-12T02:32:24.528 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.489397", "end": "2020-06-12 02:32:24.500359", "rc": 0, "start": "2020-06-12 02:32:24.010962", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} changed: [smithi188.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.199257", "end": "2020-06-12 02:32:24.500334", "rc": 0, "start": "2020-06-12 02:32:24.301077", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-12T02:32:24.563 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:24 +0000 (0:00:00.450) 0:01:23.517 *********** 2020-06-12T02:32:24.599 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:24 +0000 (0:00:00.035) 0:01:23.552 *********** 2020-06-12T02:32:25.808 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: krb5-workstation"]} 2020-06-12T02:32:25.867 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:25 +0000 (0:00:01.264) 0:01:24.817 *********** 2020-06-12T02:32:25.876 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: krb5-workstation"]} 2020-06-12T02:32:25.916 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:25 +0000 (0:00:00.051) 0:01:24.869 *********** 2020-06-12T02:32:25.925 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-06-12T02:32:25.927 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:25.967 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:25 +0000 (0:00:00.051) 0:01:24.920 *********** 2020-06-12T02:32:25.978 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:26.014 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:26 +0000 (0:00:00.047) 0:01:24.968 *********** 2020-06-12T02:32:26.024 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2020-06-12T02:32:26.026 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:26.061 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:26 +0000 (0:00:00.046) 0:01:25.014 *********** 2020-06-12T02:32:26.072 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:26.114 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:26 +0000 (0:00:00.051) 0:01:25.066 *********** 2020-06-12T02:32:26.121 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-06-12T02:32:26.123 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:26.163 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:26 +0000 (0:00:00.050) 0:01:25.117 *********** 2020-06-12T02:32:26.176 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:26.211 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:26 +0000 (0:00:00.047) 0:01:25.164 *********** 2020-06-12T02:32:26.220 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-06-12T02:32:26.223 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:26.262 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:26 +0000 (0:00:00.051) 0:01:25.216 *********** 2020-06-12T02:32:26.271 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:26.311 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:26 +0000 (0:00:00.048) 0:01:25.265 *********** 2020-06-12T02:32:26.804 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2020-06-12T02:32:26.806 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2020-06-12T02:32:26.851 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:26 +0000 (0:00:00.539) 0:01:25.804 *********** 2020-06-12T02:32:26.872 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2020-06-12T02:32:26.874 INFO:teuthology.task.ansible.out: 2020-06-12T02:32:26.911 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:26 +0000 (0:00:00.060) 0:01:25.864 *********** 2020-06-12T02:32:29.822 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: smartmontools", "Installed: smartmontools-1:7.1-7.el8.x86_64", "Removed: smartmontools-1:6.6-3.el8.x86_64"]} 2020-06-12T02:32:29.870 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:29 +0000 (0:00:02.958) 0:01:28.823 *********** 2020-06-12T02:32:29.926 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: smartmontools", "Installed: smartmontools-1:7.1-7.el8.x86_64", "Removed: smartmontools-1:6.6-3.el8.x86_64"]} 2020-06-12T02:32:29.970 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:29 +0000 (0:00:00.096) 0:01:28.920 *********** 2020-06-12T02:32:30.489 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-06-12T02:32:30.491 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2020-06-12T02:32:30.591 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2020-06-12T02:32:31.031 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2020-06-12T02:32:31.033 INFO:teuthology.task.ansible.out: 2020-06-12T02:32:31.129 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2020-06-12T02:32:31.516 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2020-06-12T02:32:31.558 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:31 +0000 (0:00:01.590) 0:01:30.511 *********** 2020-06-12T02:32:31.560 INFO:teuthology.task.ansible.out: 2020-06-12T02:32:31.617 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2020-06-12T02:32:31.660 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:31 +0000 (0:00:00.102) 0:01:30.614 *********** 2020-06-12T02:32:31.778 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-06-12T02:32:31.813 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:31 +0000 (0:00:00.152) 0:01:30.766 *********** 2020-06-12T02:32:31.886 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-06-12T02:32:31.951 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:31 +0000 (0:00:00.138) 0:01:30.904 *********** 2020-06-12T02:32:32.365 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-06-12T02:32:32.367 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 12725, "state": "file", "uid": 0} 2020-06-12T02:32:32.479 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 12725, "state": "file", "uid": 0} 2020-06-12T02:32:32.835 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2020-06-12T02:32:32.960 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2020-06-12T02:32:33.334 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2020-06-12T02:32:33.373 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:33 +0000 (0:00:01.421) 0:01:32.326 *********** 2020-06-12T02:32:33.404 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2020-06-12T02:32:33.406 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-06-12T02:32:33.408 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:33.442 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:33 +0000 (0:00:00.068) 0:01:32.395 *********** 2020-06-12T02:32:33.480 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:33 +0000 (0:00:00.037) 0:01:32.432 *********** 2020-06-12T02:32:33.490 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-06-12T02:32:33.492 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2020-06-12T02:32:33.542 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:33 +0000 (0:00:00.062) 0:01:32.495 *********** 2020-06-12T02:32:33.553 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-06-12T02:32:33.555 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:33.701 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:33 +0000 (0:00:00.048) 0:01:32.543 *********** TASK [common : Include yum_systems vars] *************************************** ok: [smithi188.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} Friday 12 June 2020 02:32:33 +0000 (0:00:00.079) 0:01:32.623 *********** 2020-06-12T02:32:33.872 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-06-12T02:32:33.874 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2020-06-12T02:32:33.907 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:33 +0000 (0:00:00.237) 0:01:32.860 *********** 2020-06-12T02:32:33.965 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-06-12T02:32:33.965 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-06-12T02:32:33.966 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-06-12T02:32:33.987 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-06-12T02:32:33.998 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-06-12T02:32:34.035 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:34 +0000 (0:00:00.128) 0:01:32.988 *********** 2020-06-12T02:32:34.776 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* changed: [smithi107.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2020-06-12T02:32:34.804 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:34 +0000 (0:00:00.769) 0:01:33.758 *********** 2020-06-12T02:32:34.857 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-06-12T02:32:34.859 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-06-12T02:32:34.879 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-06-12T02:32:34.890 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-06-12T02:32:34.902 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-06-12T02:32:34.938 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:34 +0000 (0:00:00.134) 0:01:33.892 *********** 2020-06-12T02:32:36.406 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-06-12T02:32:36.408 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: nagios-common", "Installed: nrpe", "Installed: nagios-plugins", "Installed: nagios-plugins-load", "Installed: nagios-plugins-2.3.3-3.el8.x86_64", "Installed: nagios-plugins-load-2.3.3-3.el8.x86_64", "Removed: nagios-plugins-2.3.3-1.el8.x86_64", "Removed: nagios-plugins-load-2.3.3-1.el8.x86_64"]} 2020-06-12T02:32:36.456 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:36 +0000 (0:00:01.517) 0:01:35.409 *********** 2020-06-12T02:32:36.500 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2020-06-12T02:32:36.502 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:36.542 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:36 +0000 (0:00:00.086) 0:01:35.496 *********** 2020-06-12T02:32:37.061 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-06-12T02:32:37.063 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2020-06-12T02:32:37.102 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:37 +0000 (0:00:00.559) 0:01:36.055 *********** 2020-06-12T02:32:37.349 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-06-12T02:32:37.351 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: nagios-common", "Installed: nrpe", "Installed: nagios-plugins", "Installed: nagios-plugins-load", "Installed: nagios-plugins-2.3.3-3.el8.x86_64", "Installed: nagios-plugins-load-2.3.3-3.el8.x86_64", "Removed: nagios-plugins-2.3.3-1.el8.x86_64", "Removed: nagios-plugins-load-2.3.3-1.el8.x86_64"]} 2020-06-12T02:32:37.386 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:37 +0000 (0:00:00.283) 0:01:36.339 *********** 2020-06-12T02:32:37.438 INFO:teuthology.task.ansible.out: 2020-06-12T02:32:37.440 INFO:teuthology.task.ansible.out:TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:37.498 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:37 +0000 (0:00:00.111) 0:01:36.451 *********** 2020-06-12T02:32:37.654 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* ok: [smithi188.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2020-06-12T02:32:37.686 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:37 +0000 (0:00:00.188) 0:01:36.639 *********** 2020-06-12T02:32:37.727 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-06-12T02:32:37.729 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} 2020-06-12T02:32:37.738 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2020-06-12T02:32:37.773 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:37 +0000 (0:00:00.086) 0:01:36.726 *********** 2020-06-12T02:32:37.997 INFO:teuthology.task.ansible.out: 2020-06-12T02:32:37.999 INFO:teuthology.task.ansible.out:TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* ok: [smithi188.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-06-12T02:32:38.039 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:38 +0000 (0:00:00.266) 0:01:36.992 *********** 2020-06-12T02:32:38.057 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-06-12T02:32:38.058 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2020-06-12T02:32:38.095 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:38 +0000 (0:00:00.054) 0:01:37.047 *********** 2020-06-12T02:32:38.307 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-06-12T02:32:38.309 INFO:teuthology.task.ansible.out:fatal: [smithi188.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2020-06-12T02:32:38.343 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:38 +0000 (0:00:00.249) 0:01:37.296 *********** 2020-06-12T02:32:38.385 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-06-12T02:32:38.386 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:38.430 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:38 +0000 (0:00:00.082) 0:01:37.379 *********** 2020-06-12T02:32:38.651 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-06-12T02:32:38.653 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2020-06-12T02:32:38.722 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:38 +0000 (0:00:00.295) 0:01:37.675 *********** 2020-06-12T02:32:38.775 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-06-12T02:32:38.777 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} 2020-06-12T02:32:38.786 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2020-06-12T02:32:38.822 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:38 +0000 (0:00:00.099) 0:01:37.775 *********** 2020-06-12T02:32:38.970 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2020-06-12T02:32:38.972 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"changed": false, "checksum": "333141ba5e85aea94f51511a52d9b115d2e0112f", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1174, "state": "file", "uid": 0} 2020-06-12T02:32:39.025 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:39 +0000 (0:00:00.203) 0:01:37.979 *********** 2020-06-12T02:32:39.068 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-06-12T02:32:39.071 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-06-12T02:32:39.112 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:39 +0000 (0:00:00.086) 0:01:38.065 *********** 2020-06-12T02:32:39.353 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-06-12T02:32:39.355 INFO:teuthology.task.ansible.out:fatal: [smithi107.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2020-06-12T02:32:39.394 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:39 +0000 (0:00:00.281) 0:01:38.347 *********** 2020-06-12T02:32:39.429 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:32:39.472 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:39 +0000 (0:00:00.076) 0:01:38.424 *********** 2020-06-12T02:32:39.916 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-06-12T02:32:39.918 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2020-06-12 02:29:55 UTC", "ActiveEnterTimestampMonotonic": "13820925", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "time-sync.target system.slice network.target sysinit.target tmp.mount -.mount nss-lookup.target systemd-tmpfiles-setup.service basic.target var-run.mount local-fs.target systemd-journald.socket", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2020-06-12 02:29:55 UTC", "AssertTimestampMonotonic": "13820142", "Before": "multi-user.target getty@tty1.service xdm.service shutdown.target plymouth-quit.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2020-06-12 02:29:55 UTC", "ConditionTimestampMonotonic": "13820142", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "D 2020-06-12T02:32:39.920 INFO:teuthology.task.ansible.out:efaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "2489", "ExecMainStartTimestamp": "Fri 2020-06-12 02:29:55 UTC", "ExecMainStartTimestampMonotonic": "13820897", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "GID": "986", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2020-06-12 02:29:55 UTC", "InactiveExitTimestampMonotonic": "13820925", "InvocationID": "7a2977988db7445b88ab8d736d63be2a", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitD 2020-06-12T02:32:39.921 INFO:teuthology.task.ansible.out:ATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127436", "LimitNPROCSoft": "127436", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127436", "LimitSIGPENDINGSoft": "127436", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "2489", "MemoryAccounting": "yes", "MemoryCurrent": "1433600", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/var/tmp /run/nrpe", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode 2020-06-12T02:32:39.923 INFO:teuthology.task.ansible.out:": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2020-06-12 02:29:55 UTC", "StateChangeTimestampMonotonic": "13820925", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "990", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2020-06-12 02:29:55 UTC", "WatchdogTimestampMonotonic": "13820924", "WatchdogUSec": "0"}} 2020-06-12T02:32:39.953 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:39 +0000 (0:00:00.483) 0:01:38.907 *********** 2020-06-12T02:32:39.985 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2020-06-12T02:32:39.987 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "checksum": "333141ba5e85aea94f51511a52d9b115d2e0112f", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1174, "state": "file", "uid": 0} 2020-06-12T02:32:40.020 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:40 +0000 (0:00:00.066) 0:01:38.973 *********** 2020-06-12T02:32:40.191 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* changed: [smithi188.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001890", "end": "2020-06-12 02:32:40.168361", "rc": 0, "start": "2020-06-12 02:32:40.166471", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2020-06-12T02:32:40.227 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:40 +0000 (0:00:00.206) 0:01:39.180 *********** 2020-06-12T02:32:40.349 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-06-12T02:32:40.352 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2020-06-12 02:30:08 UTC", "ActiveEnterTimestampMonotonic": "13859273", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "tmp.mount systemd-tmpfiles-setup.service -.mount system.slice local-fs.target basic.target nss-lookup.target time-sync.target network.target var-run.mount systemd-journald.socket sysinit.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2020-06-12 02:30:08 UTC", "AssertTimestampMonotonic": "13858416", "Before": "shutdown.target plymouth-quit.service xdm.service getty@tty1.service multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2020-06-12 02:30:08 UTC", "ConditionTimestampMonotonic": "13858416", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "D 2020-06-12T02:32:40.353 INFO:teuthology.task.ansible.out:efaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "2495", "ExecMainStartTimestamp": "Fri 2020-06-12 02:30:08 UTC", "ExecMainStartTimestampMonotonic": "13859235", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "GID": "986", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2020-06-12 02:30:08 UTC", "InactiveExitTimestampMonotonic": "13859273", "InvocationID": "eac3ec3e738746f7a7022d97e6da5ce1", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitD 2020-06-12T02:32:40.355 INFO:teuthology.task.ansible.out:ATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127435", "LimitNPROCSoft": "127435", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127435", "LimitSIGPENDINGSoft": "127435", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "2495", "MemoryAccounting": "yes", "MemoryCurrent": "1404928", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/var/tmp /run/nrpe", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode 2020-06-12T02:32:40.357 INFO:teuthology.task.ansible.out:": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2020-06-12 02:30:08 UTC", "StateChangeTimestampMonotonic": "13859273", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "990", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2020-06-12 02:30:08 UTC", "WatchdogTimestampMonotonic": "13859273", "WatchdogUSec": "0"}} 2020-06-12T02:32:40.388 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:40 +0000 (0:00:00.161) 0:01:39.342 *********** 2020-06-12T02:32:40.390 INFO:teuthology.task.ansible.out: 2020-06-12T02:32:40.606 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2020-06-12T02:32:40.608 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001840", "end": "2020-06-12 02:32:40.576838", "rc": 0, "start": "2020-06-12 02:32:40.574998", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2020-06-12T02:32:40.643 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:40 +0000 (0:00:00.254) 0:01:39.597 *********** 2020-06-12T02:32:41.132 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2020-06-12T02:32:41.134 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: python3-libsemanage", "Installed: python3-policycoreutils"]} 2020-06-12T02:32:41.169 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:41 +0000 (0:00:00.525) 0:01:40.122 *********** 2020-06-12T02:32:41.569 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: python3-libsemanage", "Installed: python3-policycoreutils"]} 2020-06-12T02:32:41.603 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:41 +0000 (0:00:00.434) 0:01:40.556 *********** 2020-06-12T02:32:42.078 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* ok: [smithi188.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: selinux-policy-targeted"]} 2020-06-12T02:32:42.113 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:42 +0000 (0:00:00.509) 0:01:41.066 *********** 2020-06-12T02:32:42.508 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: selinux-policy-targeted"]} 2020-06-12T02:32:42.553 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:42 +0000 (0:00:00.440) 0:01:41.506 *********** 2020-06-12T02:32:43.513 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2020-06-12T02:32:43.515 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2020-06-12T02:32:43.550 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:43 +0000 (0:00:00.997) 0:01:42.503 *********** 2020-06-12T02:32:43.749 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2020-06-12T02:32:43.795 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:32:43 +0000 (0:00:00.245) 0:01:42.749 *********** 2020-06-12T02:33:00.004 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-06-12T02:33:00.006 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:16.236456", "end": "2020-06-12 02:32:59.974937", "failed_when_result": false, "rc": 0, "start": "2020-06-12 02:32:43.738481", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2020-06-12T02:33:00.039 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:00 +0000 (0:00:16.243) 0:01:58.992 *********** 2020-06-12T02:33:00.232 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:16.211283", "end": "2020-06-12 02:33:00.201678", "failed_when_result": false, "rc": 0, "start": "2020-06-12 02:32:43.990395", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2020-06-12T02:33:00.267 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:00 +0000 (0:00:00.227) 0:01:59.220 *********** 2020-06-12T02:33:00.548 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2020-06-12T02:33:00.550 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2020-06-12T02:33:00.592 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:00 +0000 (0:00:00.325) 0:01:59.545 *********** 2020-06-12T02:33:00.788 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2020-06-12T02:33:00.828 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:00 +0000 (0:00:00.235) 0:01:59.781 *********** 2020-06-12T02:33:00.848 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-06-12T02:33:00.850 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.011077", "end": "2020-06-12 02:33:00.793328", "rc": 0, "start": "2020-06-12 02:33:00.782251", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-12T02:33:00.884 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:00 +0000 (0:00:00.055) 0:01:59.837 *********** 2020-06-12T02:33:01.056 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.015793", "end": "2020-06-12 02:33:01.029409", "rc": 0, "start": "2020-06-12 02:33:01.013616", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-12T02:33:01.091 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:01 +0000 (0:00:00.206) 0:02:00.044 *********** 2020-06-12T02:33:01.117 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2020-06-12T02:33:01.119 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.006686", "end": "2020-06-12 02:33:01.082734", "rc": 0, "start": "2020-06-12 02:33:01.076048", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-12T02:33:01.153 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:01 +0000 (0:00:00.062) 0:02:00.106 *********** 2020-06-12T02:33:01.311 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.007921", "end": "2020-06-12 02:33:01.284153", "rc": 0, "start": "2020-06-12 02:33:01.276232", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-12T02:33:01.347 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:01 +0000 (0:00:00.192) 0:02:00.299 *********** 2020-06-12T02:33:17.540 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2020-06-12T02:33:17.543 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:16.174941", "end": "2020-06-12 02:33:17.513457", "rc": 0, "start": "2020-06-12 02:33:01.338516", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-12T02:33:17.577 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:17 +0000 (0:00:16.230) 0:02:16.530 *********** 2020-06-12T02:33:17.598 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:16.005394", "end": "2020-06-12 02:33:17.569833", "rc": 0, "start": "2020-06-12 02:33:01.564439", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-12T02:33:17.642 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:17 +0000 (0:00:00.064) 0:02:16.595 *********** 2020-06-12T02:33:17.810 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2020-06-12T02:33:17.812 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2020-06-12T02:33:17.858 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:17 +0000 (0:00:00.214) 0:02:16.810 *********** 2020-06-12T02:33:17.868 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2020-06-12T02:33:17.915 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:17 +0000 (0:00:00.058) 0:02:16.868 *********** 2020-06-12T02:33:17.925 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2020-06-12T02:33:17.929 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2020-06-12T02:33:17.965 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:17 +0000 (0:00:00.050) 0:02:16.919 *********** 2020-06-12T02:33:17.978 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2020-06-12T02:33:18.012 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:18 +0000 (0:00:00.047) 0:02:16.966 *********** 2020-06-12T02:33:18.033 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-06-12T02:33:18.035 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-06-12T02:33:18.083 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:18 +0000 (0:00:00.070) 0:02:17.037 *********** 2020-06-12T02:33:18.092 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-06-12T02:33:18.143 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:18 +0000 (0:00:00.048) 0:02:17.086 *********** 2020-06-12T02:33:18.149 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** ok: [smithi188.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ktdreyer/ceph-el8"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_ 2020-06-12T02:33:18.150 INFO:teuthology.task.ansible.out:master/roles/testnode/vars/centos_8.yml"} 2020-06-12T02:33:18.181 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:18 +0000 (0:00:00.049) 0:02:17.135 *********** 2020-06-12T02:33:18.193 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ktdreyer/ceph-el8"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2020-06-12T02:33:18.233 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:18 +0000 (0:00:00.051) 0:02:17.186 *********** 2020-06-12T02:33:18.242 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2020-06-12T02:33:18.244 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-06-12T02:33:18.282 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:18 +0000 (0:00:00.048) 0:02:17.235 *********** 2020-06-12T02:33:18.302 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-06-12T02:33:18.338 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:18 +0000 (0:00:00.055) 0:02:17.291 *********** 2020-06-12T02:33:18.494 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2020-06-12T02:33:18.496 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-06-12T02:33:18.530 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:18 +0000 (0:00:00.192) 0:02:17.484 *********** 2020-06-12T02:33:18.554 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-06-12T02:33:18.590 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:18 +0000 (0:00:00.059) 0:02:17.543 *********** 2020-06-12T02:33:18.745 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-06-12T02:33:18.747 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"changed": false, "gid": 1219, "name": "ubuntu", "state": "present", "system": false} 2020-06-12T02:33:18.780 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:18 +0000 (0:00:00.190) 0:02:17.733 *********** 2020-06-12T02:33:18.810 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "gid": 1219, "name": "ubuntu", "state": "present", "system": false} 2020-06-12T02:33:18.845 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:18 +0000 (0:00:00.064) 0:02:17.798 *********** 2020-06-12T02:33:18.855 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-06-12T02:33:18.857 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:18.892 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:18 +0000 (0:00:00.047) 0:02:17.846 *********** 2020-06-12T02:33:18.902 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:18.938 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:18 +0000 (0:00:00.045) 0:02:17.892 *********** 2020-06-12T02:33:18.950 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-06-12T02:33:18.952 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:18.992 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:18 +0000 (0:00:00.053) 0:02:17.945 *********** 2020-06-12T02:33:19.002 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:19.039 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:19 +0000 (0:00:00.046) 0:02:17.992 *********** 2020-06-12T02:33:19.214 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-06-12T02:33:19.215 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2020-06-12T02:33:19.253 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:19 +0000 (0:00:00.213) 0:02:18.206 *********** 2020-06-12T02:33:19.273 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2020-06-12T02:33:19.308 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:19 +0000 (0:00:00.055) 0:02:18.261 *********** 2020-06-12T02:33:19.846 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** ok: [smithi188.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2020-06-12T02:33:19.885 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:19 +0000 (0:00:00.576) 0:02:18.838 *********** 2020-06-12T02:33:19.896 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2020-06-12T02:33:19.932 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:19 +0000 (0:00:00.047) 0:02:18.885 *********** 2020-06-12T02:33:19.941 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-06-12T02:33:19.942 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi188"}, "changed": false} 2020-06-12T02:33:19.978 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:19 +0000 (0:00:00.046) 0:02:18.931 *********** 2020-06-12T02:33:19.990 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi107"}, "changed": false} 2020-06-12T02:33:20.023 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:20 +0000 (0:00:00.044) 0:02:18.976 *********** 2020-06-12T02:33:21.160 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2020-06-12T02:33:21.162 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi188", "ansible_hostname": "smithi188", "ansible_nodename": "smithi188"}, "changed": false, "name": "smithi188"} 2020-06-12T02:33:21.198 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:21 +0000 (0:00:01.175) 0:02:20.152 *********** 2020-06-12T02:33:21.208 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi107", "ansible_hostname": "smithi107", "ansible_nodename": "smithi107"}, "changed": false, "name": "smithi107"} 2020-06-12T02:33:21.244 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:21 +0000 (0:00:00.046) 0:02:20.198 *********** 2020-06-12T02:33:21.738 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* ok: [smithi188.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2020-06-12T02:33:21.789 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:21 +0000 (0:00:00.544) 0:02:20.742 *********** 2020-06-12T02:33:21.807 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2020-06-12T02:33:21.850 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:21 +0000 (0:00:00.056) 0:02:20.799 *********** 2020-06-12T02:33:22.320 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** ok: [smithi188.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2020-06-12T02:33:22.373 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:22 +0000 (0:00:00.527) 0:02:21.326 *********** 2020-06-12T02:33:22.389 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2020-06-12T02:33:22.427 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:22 +0000 (0:00:00.050) 0:02:21.377 *********** 2020-06-12T02:33:23.079 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-06-12T02:33:23.081 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} changed: [smithi188.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-06-12T02:33:23.120 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:23 +0000 (0:00:00.696) 0:02:22.074 *********** 2020-06-12T02:33:23.155 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:23 +0000 (0:00:00.034) 0:02:22.108 *********** 2020-06-12T02:33:23.652 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-06-12T02:33:23.654 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2020-06-12T02:33:23.690 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:23 +0000 (0:00:00.535) 0:02:22.643 *********** 2020-06-12T02:33:23.693 INFO:teuthology.task.ansible.out: 2020-06-12T02:33:23.704 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2020-06-12T02:33:23.741 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:23 +0000 (0:00:00.051) 0:02:22.694 *********** 2020-06-12T02:33:23.912 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-06-12T02:33:23.914 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2020-06-12T02:33:23.975 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:23 +0000 (0:00:00.234) 0:02:22.929 *********** 2020-06-12T02:33:23.988 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2020-06-12T02:33:24.028 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:24 +0000 (0:00:00.052) 0:02:22.982 *********** 2020-06-12T02:33:24.198 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-06-12T02:33:24.200 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2020-06-12T02:33:24.261 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:24 +0000 (0:00:00.233) 0:02:23.215 *********** 2020-06-12T02:33:24.271 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2020-06-12T02:33:24.309 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:24 +0000 (0:00:00.047) 0:02:23.262 *********** 2020-06-12T02:33:24.502 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* ok: [smithi107.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1219, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-06-12T02:33:24.550 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:24 +0000 (0:00:00.241) 0:02:23.504 *********** 2020-06-12T02:33:24.562 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1219, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-06-12T02:33:24.600 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:24 +0000 (0:00:00.049) 0:02:23.554 *********** 2020-06-12T02:33:25.056 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-06-12T02:33:25.058 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2020-06-12T02:33:25.090 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:25 +0000 (0:00:00.489) 0:02:24.043 *********** 2020-06-12T02:33:25.152 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2020-06-12T02:33:25.190 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:25 +0000 (0:00:00.099) 0:02:24.143 *********** 2020-06-12T02:33:25.651 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-06-12T02:33:25.653 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2020-06-12T02:33:25.689 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:25 +0000 (0:00:00.498) 0:02:24.642 *********** 2020-06-12T02:33:25.747 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2020-06-12T02:33:25.811 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:25 +0000 (0:00:00.122) 0:02:24.765 *********** 2020-06-12T02:33:27.610 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ [WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:01.688902", "end": "2020-06-12 02:33:27.574339", "rc": 0, "start": "2020-06-12 02:33:25.885437", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-12T02:33:27.655 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:27 +0000 (0:00:01.843) 0:02:26.608 *********** 2020-06-12T02:33:27.666 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:01.602591", "end": "2020-06-12 02:33:27.602366", "rc": 0, "start": "2020-06-12 02:33:25.999775", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-12T02:33:27.705 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:27 +0000 (0:00:00.050) 0:02:26.659 *********** 2020-06-12T02:33:28.196 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2020-06-12T02:33:28.199 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "aa5ad73629677ef17172417e50d2cdadccb7a3f2", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 397, "state": "file", "uid": 0} 2020-06-12T02:33:28.291 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "aa5ad73629677ef17172417e50d2cdadccb7a3f2", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 397, "state": "file", "uid": 0} 2020-06-12T02:33:28.293 INFO:teuthology.task.ansible.out: 2020-06-12T02:33:28.687 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "4ed3c0272e6c4ec99169cd68f0b43d40bdd4b296", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 380, "state": "file", "uid": 0} 2020-06-12T02:33:28.804 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "4ed3c0272e6c4ec99169cd68f0b43d40bdd4b296", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 380, "state": "file", "uid": 0} 2020-06-12T02:33:29.203 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "a862f43d3b3c7b8b592aa7dce23a6d013ab20bc3", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 382, "state": "file", "uid": 0} 2020-06-12T02:33:29.240 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:29 +0000 (0:00:01.534) 0:02:28.193 *********** 2020-06-12T02:33:29.290 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "a862f43d3b3c7b8b592aa7dce23a6d013ab20bc3", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 382, "state": "file", "uid": 0} 2020-06-12T02:33:29.328 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:29 +0000 (0:00:00.088) 0:02:28.282 *********** 2020-06-12T02:33:29.781 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-06-12T02:33:29.783 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2020-06-12T02:33:29.818 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:29 +0000 (0:00:00.489) 0:02:28.771 *********** 2020-06-12T02:33:29.878 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2020-06-12T02:33:29.916 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:29 +0000 (0:00:00.097) 0:02:28.869 *********** 2020-06-12T02:33:30.395 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-06-12T02:33:30.397 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'key': 'CentOS-AppStream', 'value': {'name': 'CentOS-$releasever - AppStream', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2020-06-12T02:33:30.476 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'key': 'CentOS-AppStream', 'value': {'name': 'CentOS-$releasever - AppStream', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2020-06-12T02:33:30.900 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'key': 'CentOS-Base', 'value': {'name': 'CentOS-$releasever - Base', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2020-06-12T02:33:30.973 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'key': 'CentOS-Base', 'value': {'name': 'CentOS-$releasever - Base', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2020-06-12T02:33:31.402 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'key': 'CentOS-Extras', 'value': {'name': 'CentOS-$releasever - Extras', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2020-06-12T02:33:31.462 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:31 +0000 (0:00:01.525) 0:02:30.395 *********** 2020-06-12T02:33:31.510 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'key': 'CentOS-Extras', 'value': {'name': 'CentOS-$releasever - Extras', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2020-06-12T02:33:31.548 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:31 +0000 (0:00:00.105) 0:02:30.501 *********** 2020-06-12T02:33:32.012 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-06-12T02:33:32.028 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item=ktdreyer/ceph-el8) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ktdreyer/ceph-el8"], "delta": "0:00:00.291016", "end": "2020-06-12 02:33:31.966152", "item": "ktdreyer/ceph-el8", "rc": 0, "start": "2020-06-12 02:33:31.675136", "stderr": "", "stderr_lines": [], "stdout": "Repository successfully enabled.", "stdout_lines": ["Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} [WARNING]: Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. 2020-06-12T02:33:32.049 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:32 +0000 (0:00:00.501) 0:02:31.002 *********** 2020-06-12T02:33:32.074 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => (item=ktdreyer/ceph-el8) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ktdreyer/ceph-el8"], "delta": "0:00:00.279951", "end": "2020-06-12 02:33:32.039667", "item": "ktdreyer/ceph-el8", "rc": 0, "start": "2020-06-12 02:33:31.759716", "stderr": "", "stderr_lines": [], "stdout": "Repository successfully enabled.", "stdout_lines": ["Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} 2020-06-12T02:33:32.111 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:32 +0000 (0:00:00.061) 0:02:31.064 *********** 2020-06-12T02:33:32.120 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2020-06-12T02:33:32.122 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:32.156 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:32 +0000 (0:00:00.045) 0:02:31.110 *********** 2020-06-12T02:33:32.168 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:32.204 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:32 +0000 (0:00:00.048) 0:02:31.158 *********** 2020-06-12T02:33:32.216 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2020-06-12T02:33:32.218 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-06-12T02:33:32.267 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:32 +0000 (0:00:00.062) 0:02:31.220 *********** 2020-06-12T02:33:32.279 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-06-12T02:33:32.280 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-06-12T02:33:32.324 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:32 +0000 (0:00:00.057) 0:02:31.277 *********** 2020-06-12T02:33:32.334 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-06-12T02:33:32.336 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:32.371 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:32 +0000 (0:00:00.046) 0:02:31.324 *********** 2020-06-12T02:33:32.386 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:32.422 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:32 +0000 (0:00:00.048) 0:02:31.372 *********** 2020-06-12T02:33:32.429 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-06-12T02:33:32.431 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2020-06-12T02:33:32.473 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:32 +0000 (0:00:00.053) 0:02:31.426 *********** 2020-06-12T02:33:32.482 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2020-06-12T02:33:32.517 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:32 +0000 (0:00:00.044) 0:02:31.470 *********** 2020-06-12T02:33:32.528 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2020-06-12T02:33:32.530 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:32.576 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:32 +0000 (0:00:00.059) 0:02:31.530 *********** 2020-06-12T02:33:32.587 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:32.623 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:32 +0000 (0:00:00.046) 0:02:31.576 *********** 2020-06-12T02:33:32.632 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:32.669 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:32 +0000 (0:00:00.046) 0:02:31.623 *********** 2020-06-12T02:33:32.679 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:32.714 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:32 +0000 (0:00:00.044) 0:02:31.667 *********** 2020-06-12T02:33:32.898 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-06-12T02:33:32.900 INFO:teuthology.task.ansible.out:fatal: [smithi107.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.006328", "end": "2020-06-12 02:33:32.866363", "msg": "non-zero return code", "rc": 1, "start": "2020-06-12 02:33:32.860035", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2020-06-12T02:33:32.948 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:32 +0000 (0:00:00.233) 0:02:31.901 *********** 2020-06-12T02:33:32.959 INFO:teuthology.task.ansible.out:fatal: [smithi188.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.006169", "end": "2020-06-12 02:33:32.913850", "msg": "non-zero return code", "rc": 1, "start": "2020-06-12 02:33:32.907681", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} 2020-06-12T02:33:32.961 INFO:teuthology.task.ansible.out:...ignoring 2020-06-12T02:33:32.999 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:32 +0000 (0:00:00.049) 0:02:31.952 *********** 2020-06-12T02:33:33.008 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-06-12T02:33:33.010 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:33.047 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:33 +0000 (0:00:00.048) 0:02:32.000 *********** 2020-06-12T02:33:33.066 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:33.106 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:33 +0000 (0:00:00.059) 0:02:32.060 *********** 2020-06-12T02:33:35.221 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-06-12T02:33:35.228 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+297+df420408.3.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64"]} 2020-06-12T02:33:35.282 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:35 +0000 (0:00:02.175) 0:02:34.235 *********** 2020-06-12T02:33:35.292 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+297+df420408.3.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64"]} 2020-06-12T02:33:35.338 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:35 +0000 (0:00:00.056) 0:02:34.292 *********** 2020-06-12T02:33:37.113 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-06-12T02:33:37.115 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-6.el8.x86_64", "Removed: boost-random-1.66.0-6.el8.x86_64"]} 2020-06-12T02:33:37.156 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:37 +0000 (0:00:01.817) 0:02:36.109 *********** 2020-06-12T02:33:37.304 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-6.el8.x86_64", "Removed: boost-random-1.66.0-6.el8.x86_64"]} 2020-06-12T02:33:37.339 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:37 +0000 (0:00:00.182) 0:02:36.292 *********** 2020-06-12T02:33:40.008 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-06-12T02:33:40.009 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: dnf-utils", "Installed: sysstat", "Installed: libedit", "Installed: boost-thread", "Installed: xfsprogs", "Installed: gdisk", "Installed: parted", "Installed: libgcrypt", "Installed: fuse-libs", "Installed: openssl", "Installed: libuuid", "Installed: podman", "Installed: podman-docker", "Installed: attr", "Installed: ant", "Installed: lsof", "Installed: gettext", "Installed: bc", "Installed: xfsdump", "Installed: blktrace", "Installed: usbredir", "Installed: libev-devel", "Installed: valgrind", "Installed: nfs-utils", "Installed: ncurses-devel", "Installed: gcc", "Installed: git", "Installed: python3-nose", "Installed: python3-virtualenv", "Installed: genisoimage", "Installed: qemu-img", "Installed: qemu-kvm-core", "Installed: qemu-kvm-block-rbd", "Installed: libacl-devel", "Installed: dbench", "Installed: autoconf", "Installed: gdb", "Installed: iozone", "Installed: boost-random-1.66.0-6.el8.x86_64", "Installed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+297+df420408.3.x86_64", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: boost-program-options-1.66.0-6.el8.x86_64"]} 2020-06-12T02:33:40.047 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:40 +0000 (0:00:02.708) 0:02:39.000 *********** 2020-06-12T02:33:40.098 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-06-12T02:33:40.100 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:40.132 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:40 +0000 (0:00:00.084) 0:02:39.085 *********** 2020-06-12T02:33:40.332 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-06-12T02:33:40.334 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: dnf-utils", "Installed: sysstat", "Installed: libedit", "Installed: boost-thread", "Installed: xfsprogs", "Installed: gdisk", "Installed: parted", "Installed: libgcrypt", "Installed: fuse-libs", "Installed: openssl", "Installed: libuuid", "Installed: podman", "Installed: podman-docker", "Installed: attr", "Installed: ant", "Installed: lsof", "Installed: gettext", "Installed: bc", "Installed: xfsdump", "Installed: blktrace", "Installed: usbredir", "Installed: libev-devel", "Installed: valgrind", "Installed: nfs-utils", "Installed: ncurses-devel", "Installed: gcc", "Installed: git", "Installed: python3-nose", "Installed: python3-virtualenv", "Installed: genisoimage", "Installed: qemu-img", "Installed: qemu-kvm-core", "Installed: qemu-kvm-block-rbd", "Installed: libacl-devel", "Installed: dbench", "Installed: autoconf", "Installed: gdb", "Installed: iozone", "Installed: boost-random-1.66.0-6.el8.x86_64", "Installed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+297+df420408.3.x86_64", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: boost-program-options-1.66.0-6.el8.x86_64"]} 2020-06-12T02:33:40.367 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:40 +0000 (0:00:00.234) 0:02:39.320 *********** 2020-06-12T02:33:40.407 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-06-12T02:33:40.409 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:40.441 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:40 +0000 (0:00:00.074) 0:02:39.395 *********** 2020-06-12T02:33:41.058 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-06-12T02:33:41.059 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2020-06-12T02:33:41.096 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:41 +0000 (0:00:00.654) 0:02:40.049 *********** 2020-06-12T02:33:41.136 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-06-12T02:33:41.138 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:41.173 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:41 +0000 (0:00:00.076) 0:02:40.126 *********** 2020-06-12T02:33:41.382 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-06-12T02:33:41.384 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2020-06-12T02:33:41.430 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:41 +0000 (0:00:00.257) 0:02:40.383 *********** 2020-06-12T02:33:41.473 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:41.508 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:41 +0000 (0:00:00.077) 0:02:40.461 *********** 2020-06-12T02:33:41.527 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-06-12T02:33:41.529 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "polkit.service basic.target sysinit.target system.slice dbus.service dbus.socket", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "network-pre.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "iptables.service ebtables.service ipset.service ip6tables.service shutdown.target", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTim 2020-06-12T02:33:41.531 INFO:teuthology.task.ansible.out:estampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127435", "LimitNPROCSoft": "127435", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127435", 2020-06-12T02:33:41.532 INFO:teuthology.task.ansible.out:"LimitSIGPENDINGSoft": "127435", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice dbus.socket", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3 2020-06-12T02:33:41.534 INFO:teuthology.task.ansible.out:", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2020-06-12T02:33:41.567 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:41 +0000 (0:00:00.059) 0:02:40.520 *********** 2020-06-12T02:33:41.844 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "polkit.service sysinit.target dbus.service dbus.socket basic.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "network-pre.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "ebtables.service ip6tables.service ipset.service shutdown.target iptables.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTim 2020-06-12T02:33:41.846 INFO:teuthology.task.ansible.out:estampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127436", "LimitNPROCSoft": "127436", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127436", 2020-06-12T02:33:41.848 INFO:teuthology.task.ansible.out:"LimitSIGPENDINGSoft": "127436", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target dbus.socket system.slice", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3 2020-06-12T02:33:41.850 INFO:teuthology.task.ansible.out:", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2020-06-12T02:33:41.882 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:41 +0000 (0:00:00.314) 0:02:40.835 *********** 2020-06-12T02:33:41.901 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ fatal: [smithi107.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2020-06-12T02:33:41.933 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:41 +0000 (0:00:00.050) 0:02:40.886 *********** 2020-06-12T02:33:42.195 INFO:teuthology.task.ansible.out:fatal: [smithi188.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2020-06-12T02:33:42.232 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:42 +0000 (0:00:00.299) 0:02:41.186 *********** 2020-06-12T02:33:42.746 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-06-12T02:33:42.748 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2020-06-12T02:33:42.780 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:42 +0000 (0:00:00.547) 0:02:41.733 *********** 2020-06-12T02:33:42.788 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2020-06-12T02:33:42.822 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:42 +0000 (0:00:00.043) 0:02:41.776 *********** 2020-06-12T02:33:42.833 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:42.867 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:42 +0000 (0:00:00.044) 0:02:41.821 *********** 2020-06-12T02:33:42.877 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:42.912 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:42 +0000 (0:00:00.044) 0:02:41.865 *********** 2020-06-12T02:33:42.923 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2020-06-12T02:33:42.925 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:42.961 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:42 +0000 (0:00:00.048) 0:02:41.914 *********** 2020-06-12T02:33:42.974 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:43.006 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:43 +0000 (0:00:00.045) 0:02:41.960 *********** 2020-06-12T02:33:43.016 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2020-06-12T02:33:43.018 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:43.059 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:43 +0000 (0:00:00.050) 0:02:42.010 *********** 2020-06-12T02:33:43.066 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:43.105 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:43 +0000 (0:00:00.047) 0:02:42.058 *********** 2020-06-12T02:33:43.113 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2020-06-12T02:33:43.114 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:43.155 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:43 +0000 (0:00:00.050) 0:02:42.108 *********** 2020-06-12T02:33:43.162 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:43.204 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:43 +0000 (0:00:00.048) 0:02:42.157 *********** 2020-06-12T02:33:43.213 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:43.249 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:43 +0000 (0:00:00.045) 0:02:42.202 *********** 2020-06-12T02:33:43.258 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:43.304 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:43 +0000 (0:00:00.053) 0:02:42.255 *********** 2020-06-12T02:33:43.312 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-06-12T02:33:43.314 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:43.350 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:43 +0000 (0:00:00.048) 0:02:42.304 *********** 2020-06-12T02:33:43.359 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:43.395 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:43 +0000 (0:00:00.044) 0:02:42.348 *********** 2020-06-12T02:33:43.403 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-06-12T02:33:43.439 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:43 +0000 (0:00:00.044) 0:02:42.393 *********** 2020-06-12T02:33:43.492 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:43 +0000 (0:00:00.052) 0:02:42.445 *********** 2020-06-12T02:33:43.502 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-06-12T02:33:43.504 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:43.543 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:43 +0000 (0:00:00.051) 0:02:42.496 *********** 2020-06-12T02:33:43.560 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:43.591 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:43 +0000 (0:00:00.047) 0:02:42.544 *********** 2020-06-12T02:33:43.600 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-06-12T02:33:43.602 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:43.636 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:43 +0000 (0:00:00.044) 0:02:42.589 *********** 2020-06-12T02:33:43.645 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:43.680 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:43 +0000 (0:00:00.044) 0:02:42.634 *********** 2020-06-12T02:33:43.700 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2020-06-12T02:33:43.710 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-06-12T02:33:43.735 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:43 +0000 (0:00:00.054) 0:02:42.688 *********** 2020-06-12T02:33:43.756 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-06-12T02:33:43.758 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-06-12T02:33:43.790 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:43 +0000 (0:00:00.055) 0:02:42.744 *********** 2020-06-12T02:33:43.800 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-06-12T02:33:43.801 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:43.835 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:43 +0000 (0:00:00.044) 0:02:42.788 *********** 2020-06-12T02:33:43.847 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:43.881 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:43 +0000 (0:00:00.045) 0:02:42.834 *********** 2020-06-12T02:33:43.891 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-06-12T02:33:43.927 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:43 +0000 (0:00:00.046) 0:02:42.880 *********** 2020-06-12T02:33:43.977 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:43 +0000 (0:00:00.050) 0:02:42.930 *********** 2020-06-12T02:33:43.990 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2020-06-12T02:33:43.991 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:44.039 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:44 +0000 (0:00:00.061) 0:02:42.992 *********** 2020-06-12T02:33:44.050 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:44.095 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:44 +0000 (0:00:00.055) 0:02:43.048 *********** 2020-06-12T02:33:44.104 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2020-06-12T02:33:44.106 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:44.139 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:44 +0000 (0:00:00.044) 0:02:43.092 *********** 2020-06-12T02:33:44.155 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:44.196 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:44 +0000 (0:00:00.056) 0:02:43.149 *********** 2020-06-12T02:33:44.205 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2020-06-12T02:33:44.207 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:44.241 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:44 +0000 (0:00:00.045) 0:02:43.194 *********** 2020-06-12T02:33:44.249 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:44.286 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:44 +0000 (0:00:00.044) 0:02:43.239 *********** 2020-06-12T02:33:44.299 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:44.332 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:44 +0000 (0:00:00.045) 0:02:43.285 *********** 2020-06-12T02:33:44.341 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:44.382 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:44 +0000 (0:00:00.050) 0:02:43.336 *********** 2020-06-12T02:33:44.390 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-06-12T02:33:44.392 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:44.427 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:44 +0000 (0:00:00.044) 0:02:43.380 *********** 2020-06-12T02:33:44.438 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:44.478 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:44 +0000 (0:00:00.051) 0:02:43.432 *********** 2020-06-12T02:33:44.487 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-06-12T02:33:44.490 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:44.524 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:44 +0000 (0:00:00.045) 0:02:43.477 *********** 2020-06-12T02:33:44.532 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:44.574 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:44 +0000 (0:00:00.049) 0:02:43.527 *********** 2020-06-12T02:33:44.584 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2020-06-12T02:33:44.586 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:44.623 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:44 +0000 (0:00:00.049) 0:02:43.576 *********** 2020-06-12T02:33:44.631 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:44.670 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:44 +0000 (0:00:00.043) 0:02:43.620 *********** 2020-06-12T02:33:44.676 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2020-06-12T02:33:44.677 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:44.715 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:44 +0000 (0:00:00.048) 0:02:43.669 *********** 2020-06-12T02:33:44.724 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:44.773 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:44 +0000 (0:00:00.055) 0:02:43.724 *********** 2020-06-12T02:33:44.781 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:44.820 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:44 +0000 (0:00:00.049) 0:02:43.773 *********** 2020-06-12T02:33:44.827 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:44.865 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:44 +0000 (0:00:00.045) 0:02:43.818 *********** 2020-06-12T02:33:44.875 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2020-06-12T02:33:44.877 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:44.918 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:44 +0000 (0:00:00.053) 0:02:43.872 *********** 2020-06-12T02:33:44.927 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:44.963 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:44 +0000 (0:00:00.044) 0:02:43.916 *********** 2020-06-12T02:33:44.973 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2020-06-12T02:33:44.975 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:45.012 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:45 +0000 (0:00:00.049) 0:02:43.965 *********** 2020-06-12T02:33:45.021 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:45.056 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:45 +0000 (0:00:00.044) 0:02:44.010 *********** 2020-06-12T02:33:45.064 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-06-12T02:33:45.066 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:45.099 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:45 +0000 (0:00:00.042) 0:02:44.052 *********** 2020-06-12T02:33:45.107 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:45.142 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:45 +0000 (0:00:00.043) 0:02:44.096 *********** 2020-06-12T02:33:45.152 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2020-06-12T02:33:45.154 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:45.190 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:45 +0000 (0:00:00.047) 0:02:44.143 *********** 2020-06-12T02:33:45.199 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:45.235 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:45 +0000 (0:00:00.043) 0:02:44.187 *********** 2020-06-12T02:33:45.242 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-06-12T02:33:45.244 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:45.278 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:45 +0000 (0:00:00.044) 0:02:44.232 *********** 2020-06-12T02:33:45.287 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:45.326 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:45 +0000 (0:00:00.043) 0:02:44.275 *********** 2020-06-12T02:33:45.331 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-06-12T02:33:45.333 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:45.367 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:45 +0000 (0:00:00.044) 0:02:44.320 *********** 2020-06-12T02:33:45.376 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:45.411 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:45 +0000 (0:00:00.044) 0:02:44.365 *********** 2020-06-12T02:33:45.421 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-06-12T02:33:45.423 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:45.456 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:45 +0000 (0:00:00.045) 0:02:44.410 *********** 2020-06-12T02:33:45.465 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:45.500 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:45 +0000 (0:00:00.043) 0:02:44.453 *********** 2020-06-12T02:33:45.509 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-06-12T02:33:45.510 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:45.544 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:45 +0000 (0:00:00.044) 0:02:44.498 *********** 2020-06-12T02:33:45.554 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:45.590 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:45 +0000 (0:00:00.045) 0:02:44.543 *********** 2020-06-12T02:33:45.599 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-06-12T02:33:45.602 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:45.634 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:45 +0000 (0:00:00.044) 0:02:44.588 *********** 2020-06-12T02:33:45.649 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:45.678 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:45 +0000 (0:00:00.043) 0:02:44.631 *********** 2020-06-12T02:33:45.686 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-06-12T02:33:45.688 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:45.720 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:45 +0000 (0:00:00.042) 0:02:44.674 *********** 2020-06-12T02:33:45.729 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:45.764 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:45 +0000 (0:00:00.044) 0:02:44.718 *********** 2020-06-12T02:33:45.773 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-06-12T02:33:45.775 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:45.821 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:45 +0000 (0:00:00.046) 0:02:44.764 *********** 2020-06-12T02:33:45.826 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:45.857 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:45 +0000 (0:00:00.045) 0:02:44.809 *********** 2020-06-12T02:33:45.867 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-06-12T02:33:45.869 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:45.903 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:45 +0000 (0:00:00.046) 0:02:44.856 *********** 2020-06-12T02:33:45.912 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:45.950 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:45 +0000 (0:00:00.047) 0:02:44.903 *********** 2020-06-12T02:33:45.966 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:46.003 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:45 +0000 (0:00:00.048) 0:02:44.951 *********** 2020-06-12T02:33:46.007 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:46.052 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:46 +0000 (0:00:00.053) 0:02:45.005 *********** 2020-06-12T02:33:46.063 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-06-12T02:33:46.065 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:46.101 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:46 +0000 (0:00:00.048) 0:02:45.054 *********** 2020-06-12T02:33:46.110 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:46.147 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:46 +0000 (0:00:00.046) 0:02:45.101 *********** 2020-06-12T02:33:46.156 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-06-12T02:33:46.158 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:46.192 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:46 +0000 (0:00:00.044) 0:02:45.146 *********** 2020-06-12T02:33:46.204 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:46.244 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:46 +0000 (0:00:00.051) 0:02:45.197 *********** 2020-06-12T02:33:46.259 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:46.290 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:46 +0000 (0:00:00.046) 0:02:45.243 *********** 2020-06-12T02:33:46.299 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:46.339 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:46 +0000 (0:00:00.049) 0:02:45.292 *********** 2020-06-12T02:33:46.349 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-06-12T02:33:46.351 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:46.390 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:46 +0000 (0:00:00.050) 0:02:45.343 *********** 2020-06-12T02:33:46.405 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:46.439 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:46 +0000 (0:00:00.048) 0:02:45.391 *********** 2020-06-12T02:33:46.449 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-06-12T02:33:46.450 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:46.493 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:46 +0000 (0:00:00.055) 0:02:45.446 *********** 2020-06-12T02:33:46.504 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:46.543 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:46 +0000 (0:00:00.049) 0:02:45.496 *********** 2020-06-12T02:33:46.717 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-06-12T02:33:46.719 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2020-06-12T02:33:46.754 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:46 +0000 (0:00:00.211) 0:02:45.708 *********** 2020-06-12T02:33:46.795 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2020-06-12T02:33:46.844 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:46 +0000 (0:00:00.088) 0:02:45.796 *********** 2020-06-12T02:33:46.987 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-06-12T02:33:46.989 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-06-12T02:33:47.022 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:47 +0000 (0:00:00.178) 0:02:45.975 *********** 2020-06-12T02:33:47.094 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-06-12T02:33:47.133 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:47 +0000 (0:00:00.110) 0:02:46.086 *********** 2020-06-12T02:33:47.254 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** [WARNING]: Cannot set fs attributes on a non-existent symlink target. follow should be set to False to avoid this. ok: [smithi188.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2020-06-12T02:33:47.287 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:47 +0000 (0:00:00.154) 0:02:46.241 *********** 2020-06-12T02:33:47.316 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-06-12T02:33:47.318 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:47.358 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:47 +0000 (0:00:00.070) 0:02:46.311 *********** 2020-06-12T02:33:47.367 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-06-12T02:33:47.368 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2020-06-12T02:33:47.408 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:47 +0000 (0:00:00.050) 0:02:46.361 *********** 2020-06-12T02:33:47.419 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-06-12T02:33:47.421 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:47.459 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:47 +0000 (0:00:00.048) 0:02:46.410 *********** 2020-06-12T02:33:47.474 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-06-12T02:33:47.476 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:47.511 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:47 +0000 (0:00:00.053) 0:02:46.464 *********** 2020-06-12T02:33:47.531 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:47.559 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:47 +0000 (0:00:00.049) 0:02:46.513 *********** 2020-06-12T02:33:47.571 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-06-12T02:33:47.573 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:47.612 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:47 +0000 (0:00:00.052) 0:02:46.565 *********** 2020-06-12T02:33:47.625 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:47.661 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:47 +0000 (0:00:00.049) 0:02:46.614 *********** 2020-06-12T02:33:47.669 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-06-12T02:33:47.677 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:47.706 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:47 +0000 (0:00:00.045) 0:02:46.659 *********** 2020-06-12T02:33:47.715 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-06-12T02:33:47.716 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:47.751 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:47 +0000 (0:00:00.045) 0:02:46.704 *********** 2020-06-12T02:33:47.760 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-06-12T02:33:47.762 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:47.797 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:47 +0000 (0:00:00.046) 0:02:46.751 *********** 2020-06-12T02:33:47.807 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-06-12T02:33:47.809 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:47.849 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:47 +0000 (0:00:00.051) 0:02:46.802 *********** 2020-06-12T02:33:47.858 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-06-12T02:33:47.859 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:47.895 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:47 +0000 (0:00:00.046) 0:02:46.849 *********** 2020-06-12T02:33:47.905 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-06-12T02:33:47.907 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:47.942 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:47 +0000 (0:00:00.046) 0:02:46.896 *********** 2020-06-12T02:33:47.952 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-06-12T02:33:47.954 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:47.988 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:47 +0000 (0:00:00.045) 0:02:46.942 *********** 2020-06-12T02:33:48.003 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:48.044 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:48 +0000 (0:00:00.055) 0:02:46.998 *********** 2020-06-12T02:33:48.056 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:48.095 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:48 +0000 (0:00:00.050) 0:02:47.048 *********** 2020-06-12T02:33:48.106 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-06-12T02:33:48.108 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:48.142 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:48 +0000 (0:00:00.047) 0:02:47.095 *********** 2020-06-12T02:33:48.151 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-06-12T02:33:48.152 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:48.186 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:48 +0000 (0:00:00.044) 0:02:47.140 *********** 2020-06-12T02:33:48.196 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-06-12T02:33:48.198 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:48.231 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:48 +0000 (0:00:00.045) 0:02:47.185 *********** 2020-06-12T02:33:48.240 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-06-12T02:33:48.242 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:48.277 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:48 +0000 (0:00:00.045) 0:02:47.230 *********** 2020-06-12T02:33:48.286 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-06-12T02:33:48.288 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:48.330 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:48 +0000 (0:00:00.053) 0:02:47.283 *********** 2020-06-12T02:33:48.342 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:48.373 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:48 +0000 (0:00:00.043) 0:02:47.327 *********** 2020-06-12T02:33:48.383 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-06-12T02:33:48.385 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:48.421 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:48 +0000 (0:00:00.047) 0:02:47.374 *********** 2020-06-12T02:33:48.429 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-06-12T02:33:48.431 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:48.467 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:48 +0000 (0:00:00.046) 0:02:47.421 *********** 2020-06-12T02:33:48.479 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-06-12T02:33:48.481 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:48.525 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:48 +0000 (0:00:00.057) 0:02:47.478 *********** 2020-06-12T02:33:48.542 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:48.575 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:48 +0000 (0:00:00.049) 0:02:47.528 *********** 2020-06-12T02:33:48.584 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2020-06-12T02:33:48.586 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:48.619 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:48 +0000 (0:00:00.044) 0:02:47.572 *********** 2020-06-12T02:33:48.628 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-06-12T02:33:48.629 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:48.671 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:48 +0000 (0:00:00.051) 0:02:47.624 *********** 2020-06-12T02:33:48.681 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2020-06-12T02:33:48.683 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:48.719 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:48 +0000 (0:00:00.048) 0:02:47.673 *********** 2020-06-12T02:33:48.729 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:48.773 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:48 +0000 (0:00:00.053) 0:02:47.726 *********** 2020-06-12T02:33:48.784 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-06-12T02:33:48.785 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:48.826 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:48 +0000 (0:00:00.053) 0:02:47.780 *********** 2020-06-12T02:33:48.836 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2020-06-12T02:33:48.840 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:48.877 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:48 +0000 (0:00:00.050) 0:02:47.830 *********** 2020-06-12T02:33:48.888 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-06-12T02:33:48.892 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:48.929 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:48 +0000 (0:00:00.052) 0:02:47.882 *********** 2020-06-12T02:33:48.939 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-06-12T02:33:48.940 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:48.976 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:48 +0000 (0:00:00.047) 0:02:47.930 *********** 2020-06-12T02:33:48.985 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-06-12T02:33:48.987 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:49.028 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:49 +0000 (0:00:00.051) 0:02:47.981 *********** 2020-06-12T02:33:49.037 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-06-12T02:33:49.039 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:49.081 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:49 +0000 (0:00:00.053) 0:02:48.034 *********** 2020-06-12T02:33:49.091 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-06-12T02:33:49.093 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:49.129 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:49 +0000 (0:00:00.047) 0:02:48.082 *********** 2020-06-12T02:33:49.137 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-06-12T02:33:49.139 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:49.178 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:49 +0000 (0:00:00.048) 0:02:48.131 *********** 2020-06-12T02:33:49.188 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-06-12T02:33:49.190 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:49.224 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:49 +0000 (0:00:00.045) 0:02:48.177 *********** 2020-06-12T02:33:49.233 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-06-12T02:33:49.235 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:49.272 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:49 +0000 (0:00:00.047) 0:02:48.225 *********** 2020-06-12T02:33:49.283 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:49.318 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:49 +0000 (0:00:00.046) 0:02:48.271 *********** 2020-06-12T02:33:49.334 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:49.370 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:49 +0000 (0:00:00.050) 0:02:48.322 *********** 2020-06-12T02:33:49.379 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-06-12T02:33:49.380 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:49.414 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:49 +0000 (0:00:00.045) 0:02:48.367 *********** 2020-06-12T02:33:49.423 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-06-12T02:33:49.425 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:49.462 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:49 +0000 (0:00:00.044) 0:02:48.412 *********** 2020-06-12T02:33:49.466 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-06-12T02:33:49.469 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-06-12T02:33:49.505 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:49 +0000 (0:00:00.045) 0:02:48.458 *********** 2020-06-12T02:33:49.513 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-06-12T02:33:49.515 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:49.556 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:49 +0000 (0:00:00.051) 0:02:48.509 *********** 2020-06-12T02:33:49.567 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-06-12T02:33:49.568 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-06-12T02:33:49.578 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-06-12T02:33:49.580 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-06-12T02:33:49.591 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-06-12T02:33:49.601 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-06-12T02:33:49.646 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:49 +0000 (0:00:00.089) 0:02:48.599 *********** 2020-06-12T02:33:49.654 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-06-12T02:33:49.656 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-06-12T02:33:49.698 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:49 +0000 (0:00:00.051) 0:02:48.651 *********** 2020-06-12T02:33:49.759 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-06-12T02:33:49.760 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} ok: [smithi107.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-06-12T02:33:49.770 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-06-12T02:33:49.782 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-06-12T02:33:49.784 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-06-12T02:33:49.794 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-06-12T02:33:49.796 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-06-12T02:33:49.843 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:49 +0000 (0:00:00.140) 0:02:48.791 *********** 2020-06-12T02:33:49.915 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-06-12T02:33:49.917 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.004029", "end": "2020-06-12 02:33:49.866618", "rc": 0, "start": "2020-06-12 02:33:49.862589", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-06-12T02:33:49.957 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:49 +0000 (0:00:00.118) 0:02:48.910 *********** 2020-06-12T02:33:49.998 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-06-12T02:33:50.000 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:50.036 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:50 +0000 (0:00:00.078) 0:02:48.989 *********** 2020-06-12T02:33:50.083 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:50.128 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:50 +0000 (0:00:00.092) 0:02:49.081 *********** 2020-06-12T02:33:50.142 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** changed: [smithi107.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.004106", "end": "2020-06-12 02:33:50.070593", "rc": 0, "start": "2020-06-12 02:33:50.066487", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-06-12T02:33:50.184 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:50 +0000 (0:00:00.056) 0:02:49.138 *********** 2020-06-12T02:33:50.194 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-06-12T02:33:50.196 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:50.233 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:50 +0000 (0:00:00.048) 0:02:49.186 *********** 2020-06-12T02:33:50.243 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:50.285 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:50 +0000 (0:00:00.052) 0:02:49.238 *********** 2020-06-12T02:33:50.295 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-06-12T02:33:50.296 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:50.329 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:50 +0000 (0:00:00.044) 0:02:49.283 *********** 2020-06-12T02:33:50.337 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-06-12T02:33:50.339 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:50.373 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:50 +0000 (0:00:00.043) 0:02:49.326 *********** 2020-06-12T02:33:50.412 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-06-12T02:33:50.414 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:50.448 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:50 +0000 (0:00:00.075) 0:02:49.401 *********** 2020-06-12T02:33:50.479 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-06-12T02:33:50.481 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:50.528 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:50 +0000 (0:00:00.079) 0:02:49.481 *********** 2020-06-12T02:33:51.250 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2020-06-12T02:33:51.253 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: gdisk", "Installed: device-mapper"]} 2020-06-12T02:33:51.289 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:51 +0000 (0:00:00.761) 0:02:50.243 *********** 2020-06-12T02:33:51.357 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-06-12T02:33:51.358 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930566139904, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227188999, 'block_used': 12906109, 'inode_total': 61054976, 'inode_available': 60990371, 'inode_used': 64605, 'uuid': '47f327f5-49a1-4d14-86a7-ebb9208a4b6e'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227188999, "block_size": 4096, "block_total": 240095108, "block_used": 12906109, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60990371, "inode_total": 61054976, "inode_used": 64605, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930566139904, "size_total": 983429562368, "uuid": "47f327f5-49a1-4d14-86a7-ebb9208a4b6e"}} 2020-06-12T02:33:51.405 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:51 +0000 (0:00:00.115) 0:02:50.358 *********** 2020-06-12T02:33:51.471 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2020-06-12T02:33:51.473 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: gdisk", "Installed: device-mapper"]} 2020-06-12T02:33:51.514 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:51 +0000 (0:00:00.103) 0:02:50.462 *********** 2020-06-12T02:33:51.583 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-06-12T02:33:51.585 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930566144000, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227189000, 'block_used': 12906108, 'inode_total': 61054976, 'inode_available': 60990371, 'inode_used': 64605, 'uuid': '47f327f5-49a1-4d14-86a7-ebb9208a4b6e'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227189000, "block_size": 4096, "block_total": 240095108, "block_used": 12906108, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60990371, "inode_total": 61054976, "inode_used": 64605, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930566144000, "size_total": 983429562368, "uuid": "47f327f5-49a1-4d14-86a7-ebb9208a4b6e"}} 2020-06-12T02:33:51.624 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:51 +0000 (0:00:00.109) 0:02:50.571 *********** 2020-06-12T02:33:51.637 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-06-12T02:33:51.638 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005214", "end": "2020-06-12 02:33:51.615589", "rc": 0, "start": "2020-06-12 02:33:51.610375", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-12T02:33:51.673 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:51 +0000 (0:00:00.055) 0:02:50.626 *********** 2020-06-12T02:33:51.756 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-06-12T02:33:51.758 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930566139904, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227188999, 'block_used': 12906109, 'inode_total': 61054976, 'inode_available': 60990371, 'inode_used': 64605, 'uuid': '47f327f5-49a1-4d14-86a7-ebb9208a4b6e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227188999, "block_size": 4096, "block_total": 240095108, "block_used": 12906109, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60990371, "inode_total": 61054976, "inode_used": 64605, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930566139904, "size_total": 983429562368, "uuid": "47f327f5-49a1-4d14-86a7-ebb9208a4b6e"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:51.794 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:51 +0000 (0:00:00.121) 0:02:50.748 *********** 2020-06-12T02:33:51.839 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-06-12T02:33:51.878 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:51 +0000 (0:00:00.083) 0:02:50.832 *********** 2020-06-12T02:33:51.891 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-06-12T02:33:51.893 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005032", "end": "2020-06-12 02:33:51.814575", "rc": 0, "start": "2020-06-12 02:33:51.809543", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-12T02:33:51.932 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:51 +0000 (0:00:00.053) 0:02:50.885 *********** 2020-06-12T02:33:51.986 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-06-12T02:33:51.988 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930566144000, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227189000, 'block_used': 12906108, 'inode_total': 61054976, 'inode_available': 60990371, 'inode_used': 64605, 'uuid': '47f327f5-49a1-4d14-86a7-ebb9208a4b6e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227189000, "block_size": 4096, "block_total": 240095108, "block_used": 12906108, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60990371, "inode_total": 61054976, "inode_used": 64605, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930566144000, "size_total": 983429562368, "uuid": "47f327f5-49a1-4d14-86a7-ebb9208a4b6e"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:33:52.030 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:52 +0000 (0:00:00.097) 0:02:50.983 *********** 2020-06-12T02:33:52.061 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-06-12T02:33:52.126 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:52 +0000 (0:00:00.096) 0:02:51.080 *********** 2020-06-12T02:33:53.390 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-06-12T02:33:53.391 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-kHYjCKPWJPeZMJB82BZgCO0MfYqFVNSpRRtmfYVFFpj9sf0cYBeCZ5sWTA180pNY'], 'uuids': ['00b53e3f-8587-4eb4-a6a9-9f1df9afef33'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.267096", "end": "2020-06-12 02:33:53.349714", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-kHYjCKPWJPeZMJB82BZgCO0MfYqFVNSpRRtmfYVFFpj9sf0cYBeCZ5sWTA180pNY"], "labels": [], "masters": [], "uuids": ["00b53e3f-8587-4eb4-a6a9-9f1df9afef33"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-12 02:33:52.082618", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-06-12T02:33:53.634 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-r9ETHmYWVhkpBjGQFx0Oh5iYvLOjBdiVcn3D8Ktrqs6l9zlr33X0MHG1MagmBJqL'], 'uuids': ['6949d338-161c-4097-b7e6-ec75e054e0aa'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.254542", "end": "2020-06-12 02:33:53.584375", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-r9ETHmYWVhkpBjGQFx0Oh5iYvLOjBdiVcn3D8Ktrqs6l9zlr33X0MHG1MagmBJqL"], "labels": [], "masters": [], "uuids": ["6949d338-161c-4097-b7e6-ec75e054e0aa"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-12 02:33:52.329833", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-06-12T02:33:54.655 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-WcPn3O-fo4W-0z1y-LoB8-8wrz-21lf-GxFXrl', 'nvme-INTEL_SSDPEDMD400G4_PHFT620500MP400BGN', 'nvme-nvme.8086-504846543632303530304d5034303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.067942", "end": "2020-06-12 02:33:54.621812", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-WcPn3O-fo4W-0z1y-LoB8-8wrz-21lf-GxFXrl", "nvme-INTEL_SSDPEDMD400G4_PHFT620500MP400BGN", "nvme-nvme.8086-504846543632303530304d5034303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-12 02:33:53.553870", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2020-06-12T02:33:54.656 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-06-12T02:33:54.925 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-cXdncS-U03T-rwlO-nH3Q-JEyh-YArQ-8Bt7vk', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400NK400BGN', 'nvme-nvme.8086-504846543632303430304e4b34303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.067592", "end": "2020-06-12 02:33:54.886161", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-cXdncS-U03T-rwlO-nH3Q-JEyh-YArQ-8Bt7vk", "nvme-INTEL_SSDPEDMD400G4_PHFT620400NK400BGN", "nvme-nvme.8086-504846543632303430304e4b34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-12 02:33:53.818569", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2020-06-12T02:33:54.927 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-06-12T02:33:55.923 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-kHYjCKPWJPeZMJB82BZgCO0MfYqFVNSpLnTVXmFux155Cx3E9bUofSiUE9ig1mdc'], 'uuids': ['b3ab9357-8819-4413-a5a9-c04077a91790'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.058672", "end": "2020-06-12 02:33:55.891942", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-kHYjCKPWJPeZMJB82BZgCO0MfYqFVNSpLnTVXmFux155Cx3E9bUofSiUE9ig1mdc"], "labels": [], "masters": [], "uuids": ["b3ab9357-8819-4413-a5a9-c04077a91790"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-12 02:33:54.833270", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-06-12T02:33:56.196 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-r9ETHmYWVhkpBjGQFx0Oh5iYvLOjBdiVBVEf0qSbdeHcVk3aN9Lq3s9ZkiQUYDfJ'], 'uuids': ['14ec6096-f035-40fe-a0f6-8e0f5c076a15'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.052800", "end": "2020-06-12 02:33:56.158225", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-r9ETHmYWVhkpBjGQFx0Oh5iYvLOjBdiVBVEf0qSbdeHcVk3aN9Lq3s9ZkiQUYDfJ"], "labels": [], "masters": [], "uuids": ["14ec6096-f035-40fe-a0f6-8e0f5c076a15"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-12 02:33:55.105425", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-06-12T02:33:57.198 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-kHYjCKPWJPeZMJB82BZgCO0MfYqFVNSprdXIAZSpcLVHtXA7PeHQlEs62SrgmMxn'], 'uuids': ['67d1c8a8-2a8c-4ba2-beae-e24b22733cb2'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.060883", "end": "2020-06-12 02:33:57.158972", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-kHYjCKPWJPeZMJB82BZgCO0MfYqFVNSprdXIAZSpcLVHtXA7PeHQlEs62SrgmMxn"], "labels": [], "masters": [], "uuids": ["67d1c8a8-2a8c-4ba2-beae-e24b22733cb2"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-12 02:33:56.098089", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-06-12T02:33:57.443 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-r9ETHmYWVhkpBjGQFx0Oh5iYvLOjBdiVAaiyJpX0zPcTb2m4CXFDYMABAuYoen2Q'], 'uuids': ['6609ea23-e114-4ae4-8b4d-3367e7849a84'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.045234", "end": "2020-06-12 02:33:57.408626", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-r9ETHmYWVhkpBjGQFx0Oh5iYvLOjBdiVAaiyJpX0zPcTb2m4CXFDYMABAuYoen2Q"], "labels": [], "masters": [], "uuids": ["6609ea23-e114-4ae4-8b4d-3367e7849a84"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-12 02:33:56.363392", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-06-12T02:33:58.434 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-kHYjCKPWJPeZMJB82BZgCO0MfYqFVNSp6opeVamz1ZJVGkWpE4N9BdRyvimkwvgF'], 'uuids': ['4e7d5cba-c940-40dc-b66c-dc4901eee08a'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.038836", "end": "2020-06-12 02:33:58.400842", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-kHYjCKPWJPeZMJB82BZgCO0MfYqFVNSp6opeVamz1ZJVGkWpE4N9BdRyvimkwvgF"], "labels": [], "masters": [], "uuids": ["4e7d5cba-c940-40dc-b66c-dc4901eee08a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-12 02:33:57.362006", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-06-12T02:33:58.446 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XTEX', 'wwn-0x5000c5009294703b'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5XTEX', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c5009294703b', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XTEX-part1', 'wwn-0x5000c5009294703b-part1'], 'uuids': ['47f327f5-49a1-4d14-86a7-ebb9208a4b6e'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '47f327f5-49a1-4d14-86a7-ebb9208a4b6e', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XTEX", "wwn-0x5000c5009294703b"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XTEX-part1", "wwn-0x5000c5009294703b-part1"], "labels": [], "masters": [], "uuids": ["47f327f5-49a1-4d14-86a7-ebb9208a4b6e"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "47f327f5-49a1-4d14-86a7-ebb9208a4b6e"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XTEX", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009294703b"}}, 2020-06-12T02:33:58.452 INFO:teuthology.task.ansible.out: "skip_reason": "Conditional result was False"} 2020-06-12T02:33:58.730 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-r9ETHmYWVhkpBjGQFx0Oh5iYvLOjBdiVJoNm0lIWeezdQVKnLafy3rGKGbWTyyC2'], 'uuids': ['aa3908b5-94fe-4327-89a3-4fc990016862'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.064575", "end": "2020-06-12 02:33:58.691822", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-r9ETHmYWVhkpBjGQFx0Oh5iYvLOjBdiVJoNm0lIWeezdQVKnLafy3rGKGbWTyyC2"], "labels": [], "masters": [], "uuids": ["aa3908b5-94fe-4327-89a3-4fc990016862"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-12 02:33:57.627247", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-06-12T02:33:58.747 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5R0J2', 'wwn-0x5000c50091e2f7f6'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5R0J2', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50091e2f7f6', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5R0J2-part1', 'wwn-0x5000c50091e2f7f6-part1'], 'uuids': ['47f327f5-49a1-4d14-86a7-ebb9208a4b6e'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '47f327f5-49a1-4d14-86a7-ebb9208a4b6e', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5R0J2", "wwn-0x5000c50091e2f7f6"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5R0J2-part1", "wwn-0x5000c50091e2f7f6-part1"], "labels": [], "masters": [], "uuids": ["47f327f5-49a1-4d14-86a7-ebb9208a4b6e"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "47f327f5-49a1-4d14-86a7-ebb9208a4b6e"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5R0J2", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091e2f7f6"}}, 2020-06-12T02:33:58.749 INFO:teuthology.task.ansible.out: "skip_reason": "Conditional result was False"} 2020-06-12T02:33:59.683 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-kHYjCKPWJPeZMJB82BZgCO0MfYqFVNSpJkujLKQM4LGLD92CTrogEXnAfIoyybGO'], 'uuids': ['e2e37519-a8d1-47eb-ad8a-6e5069853f9c'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.044281", "end": "2020-06-12 02:33:59.651500", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-kHYjCKPWJPeZMJB82BZgCO0MfYqFVNSpJkujLKQM4LGLD92CTrogEXnAfIoyybGO"], "labels": [], "masters": [], "uuids": ["e2e37519-a8d1-47eb-ad8a-6e5069853f9c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-12 02:33:58.607219", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-06-12T02:33:59.731 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:33:59 +0000 (0:00:07.604) 0:02:58.684 *********** 2020-06-12T02:34:00.015 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-r9ETHmYWVhkpBjGQFx0Oh5iYvLOjBdiV0XRWZZHQDJu042ad855PSJpyCCijpt4U'], 'uuids': ['b0153c6d-ebd1-49e3-b817-d915f44d31bd'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.080653", "end": "2020-06-12 02:33:59.982429", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-r9ETHmYWVhkpBjGQFx0Oh5iYvLOjBdiV0XRWZZHQDJu042ad855PSJpyCCijpt4U"], "labels": [], "masters": [], "uuids": ["b0153c6d-ebd1-49e3-b817-d915f44d31bd"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-12 02:33:58.901776", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-06-12T02:34:00.073 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:00 +0000 (0:00:00.339) 0:02:59.024 *********** 2020-06-12T02:34:00.132 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-06-12T02:34:00.134 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-kHYjCKPWJPeZMJB82BZgCO0MfYqFVNSpRRtmfYVFFpj9sf0cYBeCZ5sWTA180pNY'], 'uuids': ['00b53e3f-8587-4eb4-a6a9-9f1df9afef33'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.139541", "end": "2020-06-12 02:34:00.080683", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-kHYjCKPWJPeZMJB82BZgCO0MfYqFVNSpRRtmfYVFFpj9sf0cYBeCZ5sWTA180pNY"], "labels": [], "masters": [], "uuids": ["00b53e3f-8587-4eb4-a6a9-9f1df9afef33"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-12 02:33:59.941142", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.136895 s, 843 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.136895 s, 843 MB/s"], "stdout": "", "stdout_lines": []} 2020-06-12T02:34:00.343 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-WcPn3O-fo4W-0z1y-LoB8-8wrz-21lf-GxFXrl', 'nvme-INTEL_SSDPEDMD400G4_PHFT620500MP400BGN', 'nvme-nvme.8086-504846543632303530304d5034303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.037258", "end": "2020-06-12 02:34:00.304861", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-WcPn3O-fo4W-0z1y-LoB8-8wrz-21lf-GxFXrl", "nvme-INTEL_SSDPEDMD400G4_PHFT620500MP400BGN", "nvme-nvme.8086-504846543632303530304d5034303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-12 02:34:00.267603", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0346462 s, 3.3 GB/s", "std 2020-06-12T02:34:00.345 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0346462 s, 3.3 GB/s"], "stdout": "", "stdout_lines": []} 2020-06-12T02:34:00.447 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-r9ETHmYWVhkpBjGQFx0Oh5iYvLOjBdiVcn3D8Ktrqs6l9zlr33X0MHG1MagmBJqL'], 'uuids': ['6949d338-161c-4097-b7e6-ec75e054e0aa'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.138018", "end": "2020-06-12 02:34:00.417047", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-r9ETHmYWVhkpBjGQFx0Oh5iYvLOjBdiVcn3D8Ktrqs6l9zlr33X0MHG1MagmBJqL"], "labels": [], "masters": [], "uuids": ["6949d338-161c-4097-b7e6-ec75e054e0aa"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-12 02:34:00.279029", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.135255 s, 853 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.135255 s, 853 MB/s"], "stdout": "", "stdout_lines": []} 2020-06-12T02:34:00.684 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-kHYjCKPWJPeZMJB82BZgCO0MfYqFVNSpLnTVXmFux155Cx3E9bUofSiUE9ig1mdc'], 'uuids': ['b3ab9357-8819-4413-a5a9-c04077a91790'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.144779", "end": "2020-06-12 02:34:00.652574", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-kHYjCKPWJPeZMJB82BZgCO0MfYqFVNSpLnTVXmFux155Cx3E9bUofSiUE9ig1mdc"], "labels": [], "masters": [], "uuids": ["b3ab9357-8819-4413-a5a9-c04077a91790"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-12 02:34:00.507795", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.142129 s, 812 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.142129 s, 812 MB/s"], "stdout": "", "stdout_lines": []} 2020-06-12T02:34:00.712 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-cXdncS-U03T-rwlO-nH3Q-JEyh-YArQ-8Bt7vk', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400NK400BGN', 'nvme-nvme.8086-504846543632303430304e4b34303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.035850", "end": "2020-06-12 02:34:00.679789", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-cXdncS-U03T-rwlO-nH3Q-JEyh-YArQ-8Bt7vk", "nvme-INTEL_SSDPEDMD400G4_PHFT620400NK400BGN", "nvme-nvme.8086-504846543632303430304e4b34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-12 02:34:00.643939", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0334408 s, 3.4 GB/s", "std 2020-06-12T02:34:00.714 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0334408 s, 3.4 GB/s"], "stdout": "", "stdout_lines": []} 2020-06-12T02:34:01.013 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-kHYjCKPWJPeZMJB82BZgCO0MfYqFVNSprdXIAZSpcLVHtXA7PeHQlEs62SrgmMxn'], 'uuids': ['67d1c8a8-2a8c-4ba2-beae-e24b22733cb2'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.137657", "end": "2020-06-12 02:34:00.985344", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-kHYjCKPWJPeZMJB82BZgCO0MfYqFVNSprdXIAZSpcLVHtXA7PeHQlEs62SrgmMxn"], "labels": [], "masters": [], "uuids": ["67d1c8a8-2a8c-4ba2-beae-e24b22733cb2"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-12 02:34:00.847687", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.135161 s, 853 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.135161 s, 853 MB/s"], "stdout": "", "stdout_lines": []} 2020-06-12T02:34:01.077 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-r9ETHmYWVhkpBjGQFx0Oh5iYvLOjBdiVBVEf0qSbdeHcVk3aN9Lq3s9ZkiQUYDfJ'], 'uuids': ['14ec6096-f035-40fe-a0f6-8e0f5c076a15'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.140938", "end": "2020-06-12 02:34:01.039949", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-r9ETHmYWVhkpBjGQFx0Oh5iYvLOjBdiVBVEf0qSbdeHcVk3aN9Lq3s9ZkiQUYDfJ"], "labels": [], "masters": [], "uuids": ["14ec6096-f035-40fe-a0f6-8e0f5c076a15"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-12 02:34:00.899011", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.138362 s, 834 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.138362 s, 834 MB/s"], "stdout": "", "stdout_lines": []} 2020-06-12T02:34:01.354 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-kHYjCKPWJPeZMJB82BZgCO0MfYqFVNSp6opeVamz1ZJVGkWpE4N9BdRyvimkwvgF'], 'uuids': ['4e7d5cba-c940-40dc-b66c-dc4901eee08a'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.141523", "end": "2020-06-12 02:34:01.315789", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-kHYjCKPWJPeZMJB82BZgCO0MfYqFVNSp6opeVamz1ZJVGkWpE4N9BdRyvimkwvgF"], "labels": [], "masters": [], "uuids": ["4e7d5cba-c940-40dc-b66c-dc4901eee08a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-12 02:34:01.174266", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.138891 s, 830 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.138891 s, 830 MB/s"], "stdout": "", "stdout_lines": []} 2020-06-12T02:34:01.355 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XTEX', 'wwn-0x5000c5009294703b'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5XTEX', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c5009294703b', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XTEX-part1', 'wwn-0x5000c5009294703b-part1'], 'uuids': ['47f327f5-49a1-4d14-86a7-ebb9208a4b6e'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '47f327f5-49a1-4d14-86a7-ebb9208a4b6e', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XTEX", "wwn-0x5000c5009294703b"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XTEX-part1", "wwn-0x5000c5009294703b-part1"], "labels": [], "masters": [], "uuids": ["47f327f5-49a1-4d14-86a7-ebb9208a4b6e"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "47f327f5-49a1-4d14-86a7-ebb9208a4b6e"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XTEX", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009294703b"}}, 2020-06-12T02:34:01.357 INFO:teuthology.task.ansible.out: "skip_reason": "Conditional result was False"} 2020-06-12T02:34:01.429 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-r9ETHmYWVhkpBjGQFx0Oh5iYvLOjBdiVAaiyJpX0zPcTb2m4CXFDYMABAuYoen2Q'], 'uuids': ['6609ea23-e114-4ae4-8b4d-3367e7849a84'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.139606", "end": "2020-06-12 02:34:01.377676", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-r9ETHmYWVhkpBjGQFx0Oh5iYvLOjBdiVAaiyJpX0zPcTb2m4CXFDYMABAuYoen2Q"], "labels": [], "masters": [], "uuids": ["6609ea23-e114-4ae4-8b4d-3367e7849a84"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-12 02:34:01.238070", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.136961 s, 842 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.136961 s, 842 MB/s"], "stdout": "", "stdout_lines": []} 2020-06-12T02:34:01.703 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-kHYjCKPWJPeZMJB82BZgCO0MfYqFVNSpJkujLKQM4LGLD92CTrogEXnAfIoyybGO'], 'uuids': ['e2e37519-a8d1-47eb-ad8a-6e5069853f9c'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.148186", "end": "2020-06-12 02:34:01.670767", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-kHYjCKPWJPeZMJB82BZgCO0MfYqFVNSpJkujLKQM4LGLD92CTrogEXnAfIoyybGO"], "labels": [], "masters": [], "uuids": ["e2e37519-a8d1-47eb-ad8a-6e5069853f9c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-12 02:34:01.522581", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.145673 s, 792 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.145673 s, 792 MB/s"], "stdout": "", "stdout_lines": []} 2020-06-12T02:34:01.745 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:01 +0000 (0:00:01.674) 0:03:00.698 *********** 2020-06-12T02:34:01.759 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-r9ETHmYWVhkpBjGQFx0Oh5iYvLOjBdiVJoNm0lIWeezdQVKnLafy3rGKGbWTyyC2'], 'uuids': ['aa3908b5-94fe-4327-89a3-4fc990016862'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.142689", "end": "2020-06-12 02:34:01.713529", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-r9ETHmYWVhkpBjGQFx0Oh5iYvLOjBdiVJoNm0lIWeezdQVKnLafy3rGKGbWTyyC2"], "labels": [], "masters": [], "uuids": ["aa3908b5-94fe-4327-89a3-4fc990016862"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-12 02:34:01.570840", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.140115 s, 823 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.140115 s, 823 MB/s"], "stdout": "", "stdout_lines": []} 2020-06-12T02:34:01.779 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5R0J2', 'wwn-0x5000c50091e2f7f6'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5R0J2', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50091e2f7f6', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5R0J2-part1', 'wwn-0x5000c50091e2f7f6-part1'], 'uuids': ['47f327f5-49a1-4d14-86a7-ebb9208a4b6e'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '47f327f5-49a1-4d14-86a7-ebb9208a4b6e', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5R0J2", "wwn-0x5000c50091e2f7f6"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5R0J2-part1", "wwn-0x5000c50091e2f7f6-part1"], "labels": [], "masters": [], "uuids": ["47f327f5-49a1-4d14-86a7-ebb9208a4b6e"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "47f327f5-49a1-4d14-86a7-ebb9208a4b6e"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5R0J2", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091e2f7f6"}}, 2020-06-12T02:34:01.781 INFO:teuthology.task.ansible.out: "skip_reason": "Conditional result was False"} 2020-06-12T02:34:02.034 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-06-12T02:34:02.042 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.065395", "end": "2020-06-12 02:34:01.999366", "rc": 0, "start": "2020-06-12 02:34:01.933971", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-12T02:34:02.082 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:02 +0000 (0:00:00.330) 0:03:01.029 *********** 2020-06-12T02:34:02.115 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-06-12T02:34:02.116 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-r9ETHmYWVhkpBjGQFx0Oh5iYvLOjBdiV0XRWZZHQDJu042ad855PSJpyCCijpt4U'], 'uuids': ['b0153c6d-ebd1-49e3-b817-d915f44d31bd'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.140315", "end": "2020-06-12 02:34:02.082094", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-r9ETHmYWVhkpBjGQFx0Oh5iYvLOjBdiV0XRWZZHQDJu042ad855PSJpyCCijpt4U"], "labels": [], "masters": [], "uuids": ["b0153c6d-ebd1-49e3-b817-d915f44d31bd"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-12 02:34:01.941779", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.137822 s, 837 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.137822 s, 837 MB/s"], "stdout": "", "stdout_lines": []} 2020-06-12T02:34:02.156 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:02 +0000 (0:00:00.080) 0:03:01.110 *********** 2020-06-12T02:34:02.348 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-06-12T02:34:02.350 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.041900", "end": "2020-06-12 02:34:02.313179", "rc": 0, "start": "2020-06-12 02:34:02.271279", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-12T02:34:02.398 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:02 +0000 (0:00:00.241) 0:03:01.351 *********** 2020-06-12T02:34:02.428 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-06-12T02:34:02.475 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:02 +0000 (0:00:00.077) 0:03:01.429 *********** 2020-06-12T02:34:02.486 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-06-12T02:34:02.488 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.073855", "end": "2020-06-12 02:34:02.425820", "rc": 0, "start": "2020-06-12 02:34:02.351965", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-12T02:34:02.531 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:02 +0000 (0:00:00.055) 0:03:01.484 *********** 2020-06-12T02:34:02.540 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-06-12T02:34:02.542 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:02.585 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:02 +0000 (0:00:00.054) 0:03:01.539 *********** 2020-06-12T02:34:02.632 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-06-12T02:34:02.634 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:02.667 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:02 +0000 (0:00:00.081) 0:03:01.620 *********** 2020-06-12T02:34:02.695 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-06-12T02:34:02.696 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:02.754 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:02 +0000 (0:00:00.087) 0:03:01.707 *********** 2020-06-12T02:34:02.781 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-06-12T02:34:02.783 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.042705", "end": "2020-06-12 02:34:02.752019", "rc": 0, "start": "2020-06-12 02:34:02.709314", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-12T02:34:02.822 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:02 +0000 (0:00:00.067) 0:03:01.775 *********** 2020-06-12T02:34:02.834 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-06-12T02:34:02.836 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:02.876 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:02 +0000 (0:00:00.053) 0:03:01.829 *********** 2020-06-12T02:34:02.885 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-06-12T02:34:02.931 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:02 +0000 (0:00:00.055) 0:03:01.885 *********** 2020-06-12T02:34:02.976 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-06-12T02:34:02.979 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:03.029 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:03 +0000 (0:00:00.097) 0:03:01.982 *********** 2020-06-12T02:34:03.059 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-06-12T02:34:03.061 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:03.094 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:03 +0000 (0:00:00.064) 0:03:02.047 *********** 2020-06-12T02:34:03.103 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-06-12T02:34:03.104 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-06-12T02:34:03.143 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:03 +0000 (0:00:00.048) 0:03:02.096 *********** 2020-06-12T02:34:03.153 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:03.193 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:03 +0000 (0:00:00.049) 0:03:02.146 *********** 2020-06-12T02:34:03.201 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-06-12T02:34:03.208 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930566139904, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227188999, 'block_used': 12906109, 'inode_total': 61054976, 'inode_available': 60990371, 'inode_used': 64605, 'uuid': '47f327f5-49a1-4d14-86a7-ebb9208a4b6e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227188999, "block_size": 4096, "block_total": 240095108, "block_used": 12906109, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60990371, "inode_total": 61054976, "inode_used": 64605, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930566139904, "size_total": 983429562368, "uuid": "47f327f5-49a1-4d14-86a7-ebb9208a4b6e"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:03.238 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:03 +0000 (0:00:00.045) 0:03:02.192 *********** 2020-06-12T02:34:03.247 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-06-12T02:34:03.249 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:03.292 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:03 +0000 (0:00:00.053) 0:03:02.245 *********** 2020-06-12T02:34:03.301 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-06-12T02:34:03.303 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:03.336 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:03 +0000 (0:00:00.044) 0:03:02.290 *********** 2020-06-12T02:34:03.386 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-06-12T02:34:03.388 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:03.435 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:03 +0000 (0:00:00.098) 0:03:02.388 *********** 2020-06-12T02:34:03.476 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-06-12T02:34:03.478 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:03.528 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:03 +0000 (0:00:00.093) 0:03:02.482 *********** 2020-06-12T02:34:03.546 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-06-12T02:34:03.551 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-06-12T02:34:03.587 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:03 +0000 (0:00:00.058) 0:03:02.540 *********** 2020-06-12T02:34:03.598 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:03.638 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:03 +0000 (0:00:00.051) 0:03:02.592 *********** 2020-06-12T02:34:03.658 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-06-12T02:34:03.660 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930566144000, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227189000, 'block_used': 12906108, 'inode_total': 61054976, 'inode_available': 60990371, 'inode_used': 64605, 'uuid': '47f327f5-49a1-4d14-86a7-ebb9208a4b6e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227189000, "block_size": 4096, "block_total": 240095108, "block_used": 12906108, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60990371, "inode_total": 61054976, "inode_used": 64605, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930566144000, "size_total": 983429562368, "uuid": "47f327f5-49a1-4d14-86a7-ebb9208a4b6e"}, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:03.693 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:03 +0000 (0:00:00.055) 0:03:02.647 *********** 2020-06-12T02:34:03.705 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-06-12T02:34:03.707 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:03.741 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:03 +0000 (0:00:00.046) 0:03:02.693 *********** 2020-06-12T02:34:03.750 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-06-12T02:34:03.752 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:03.791 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:03 +0000 (0:00:00.050) 0:03:02.744 *********** 2020-06-12T02:34:03.800 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-06-12T02:34:03.802 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:03.845 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:03 +0000 (0:00:00.053) 0:03:02.798 *********** 2020-06-12T02:34:03.859 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-06-12T02:34:03.861 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:03.895 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:03 +0000 (0:00:00.050) 0:03:02.849 *********** 2020-06-12T02:34:03.937 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-06-12T02:34:03.938 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:03.972 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:03 +0000 (0:00:00.076) 0:03:02.926 *********** 2020-06-12T02:34:04.033 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-06-12T02:34:04.042 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:04.069 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:04 +0000 (0:00:00.096) 0:03:03.022 *********** 2020-06-12T02:34:04.110 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-06-12T02:34:04.112 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:04.150 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:04 +0000 (0:00:00.077) 0:03:03.100 *********** 2020-06-12T02:34:04.188 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:04.231 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:04 +0000 (0:00:00.084) 0:03:03.185 *********** 2020-06-12T02:34:04.661 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** changed: [smithi188.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-06-12T02:34:04.714 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:04 +0000 (0:00:00.482) 0:03:03.668 *********** 2020-06-12T02:34:04.733 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-06-12T02:34:04.782 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:04 +0000 (0:00:00.060) 0:03:03.728 *********** 2020-06-12T02:34:05.442 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-06-12T02:34:05.444 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-06-12T02:34:05.470 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-06-12T02:34:05.766 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-06-12T02:34:05.834 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-06-12T02:34:06.138 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-06-12T02:34:06.156 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-06-12T02:34:06.461 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-06-12T02:34:06.504 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-06-12T02:34:06.804 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-06-12T02:34:06.873 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:06 +0000 (0:00:02.091) 0:03:05.820 *********** 2020-06-12T02:34:06.880 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-06-12T02:34:06.930 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:06 +0000 (0:00:00.064) 0:03:05.884 *********** 2020-06-12T02:34:07.089 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-06-12T02:34:07.125 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:07 +0000 (0:00:00.194) 0:03:06.078 *********** 2020-06-12T02:34:07.149 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-06-12T02:34:07.187 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:07 +0000 (0:00:00.062) 0:03:06.140 *********** 2020-06-12T02:34:07.372 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-06-12T02:34:07.374 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-06-12T02:34:07.443 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-06-12T02:34:07.566 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-06-12T02:34:07.653 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-06-12T02:34:07.769 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-06-12T02:34:07.865 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-06-12T02:34:07.959 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-06-12T02:34:07.973 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:08.010 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:08 +0000 (0:00:00.823) 0:03:06.964 *********** 2020-06-12T02:34:08.013 INFO:teuthology.task.ansible.out: 2020-06-12T02:34:08.057 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-06-12T02:34:08.070 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:08.110 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:08 +0000 (0:00:00.099) 0:03:07.063 *********** 2020-06-12T02:34:08.240 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-06-12T02:34:08.242 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-06-12T02:34:08.280 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:08 +0000 (0:00:00.170) 0:03:07.233 *********** 2020-06-12T02:34:08.341 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2020-06-12T02:34:08.350 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} TASK [testnode : Create /var/lib/ceph] ***************************************** ok: [smithi188.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-06-12T02:34:08.384 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:08 +0000 (0:00:00.104) 0:03:07.337 *********** 2020-06-12T02:34:08.423 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:08 +0000 (0:00:00.038) 0:03:07.376 *********** 2020-06-12T02:34:08.471 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2020-06-12T02:34:08.473 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-06-12T02:34:08.511 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:08 +0000 (0:00:00.088) 0:03:07.465 *********** 2020-06-12T02:34:09.008 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2020-06-12T02:34:09.010 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"changed": true} 2020-06-12T02:34:09.069 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:09 +0000 (0:00:00.557) 0:03:08.023 *********** 2020-06-12T02:34:09.081 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => {"changed": true} 2020-06-12T02:34:09.117 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:09 +0000 (0:00:00.046) 0:03:08.069 *********** 2020-06-12T02:34:09.690 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2020-06-12T02:34:09.691 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-06-12T02:34:09.734 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:09 +0000 (0:00:00.617) 0:03:08.687 *********** 2020-06-12T02:34:09.743 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-06-12T02:34:09.790 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:09 +0000 (0:00:00.050) 0:03:08.737 *********** 2020-06-12T02:34:09.795 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2020-06-12T02:34:09.797 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:09.832 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:09 +0000 (0:00:00.048) 0:03:08.786 *********** 2020-06-12T02:34:09.843 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:09.881 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:09 +0000 (0:00:00.048) 0:03:08.834 *********** 2020-06-12T02:34:09.889 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2020-06-12T02:34:09.893 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:09.929 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:09 +0000 (0:00:00.047) 0:03:08.882 *********** 2020-06-12T02:34:09.939 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:09.986 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:09 +0000 (0:00:00.056) 0:03:08.939 *********** 2020-06-12T02:34:09.994 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2020-06-12T02:34:09.995 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:10.035 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:10 +0000 (0:00:00.049) 0:03:08.989 *********** 2020-06-12T02:34:10.044 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:10.084 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:10 +0000 (0:00:00.048) 0:03:09.037 *********** 2020-06-12T02:34:10.093 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-06-12T02:34:10.095 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:10.135 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:10 +0000 (0:00:00.051) 0:03:09.089 *********** 2020-06-12T02:34:10.144 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:10.185 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:10 +0000 (0:00:00.049) 0:03:09.139 *********** 2020-06-12T02:34:10.195 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2020-06-12T02:34:10.197 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:10.235 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:10 +0000 (0:00:00.049) 0:03:09.188 *********** 2020-06-12T02:34:10.244 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:10.281 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:10 +0000 (0:00:00.046) 0:03:09.234 *********** 2020-06-12T02:34:10.290 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-06-12T02:34:10.292 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:10.328 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:10 +0000 (0:00:00.046) 0:03:09.281 *********** 2020-06-12T02:34:10.337 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:10.377 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:10 +0000 (0:00:00.049) 0:03:09.331 *********** 2020-06-12T02:34:10.394 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:10.424 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:10 +0000 (0:00:00.046) 0:03:09.378 *********** 2020-06-12T02:34:10.434 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:10.472 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:10 +0000 (0:00:00.047) 0:03:09.425 *********** 2020-06-12T02:34:10.490 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:10.521 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:10 +0000 (0:00:00.049) 0:03:09.474 *********** 2020-06-12T02:34:10.531 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:10.568 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:10 +0000 (0:00:00.047) 0:03:09.522 *********** 2020-06-12T02:34:10.589 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-06-12T02:34:10.591 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:10.653 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:10 +0000 (0:00:00.073) 0:03:09.595 *********** skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:10.686 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:10 +0000 (0:00:00.044) 0:03:09.639 *********** 2020-06-12T02:34:10.696 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-06-12T02:34:10.697 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:10.735 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:10 +0000 (0:00:00.048) 0:03:09.688 *********** 2020-06-12T02:34:10.747 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:10.778 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:10 +0000 (0:00:00.043) 0:03:09.731 *********** 2020-06-12T02:34:10.817 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-06-12T02:34:10.818 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:10.854 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:10 +0000 (0:00:00.075) 0:03:09.807 *********** 2020-06-12T02:34:10.870 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:10.902 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:10 +0000 (0:00:00.044) 0:03:09.851 *********** 2020-06-12T02:34:11.430 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2020-06-12T02:34:11.466 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:11 +0000 (0:00:00.567) 0:03:10.419 *********** 2020-06-12T02:34:11.488 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2020-06-12T02:34:11.530 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:11 +0000 (0:00:00.061) 0:03:10.481 *********** 2020-06-12T02:34:11.787 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2020-06-12 02:30:08 UTC", "ActiveEnterTimestampMonotonic": "13830209", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "tmp.mount basic.target sntp.service -.mount system.slice systemd-tmpfiles-setup.service ntpd.service ntpdate.service systemd-journald.socket sysinit.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2020-06-12 02:30:07 UTC", "AssertTimestampMonotonic": "12731352", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2020-06-12 02:30:07 UTC", "ConditionTimestampMonotonic": "12731317", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target ntpd.service systemd-timesyncd.service", "ControlGroup": "/system.slice/chronyd.servic 2020-06-12T02:34:11.789 INFO:teuthology.task.ansible.out:e", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "2438", "ExecMainStartTimestamp": "Fri 2020-06-12 02:30:08 UTC", "ExecMainStartTimestampMonotonic": "13765953", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2020-06-12 02:30:07 UTC", "InactiveExitTimestampMonotonic": "12845756", "InvocationID": "669b4dcd2249456f89ead69b1bde13cc", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infini 2020-06-12T02:34:11.791 INFO:teuthology.task.ansible.out:ty", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127435", "LimitNPROCSoft": "127435", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127435", "LimitSIGPENDINGSoft": "127435", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "2438", "MemoryAccounting": "yes", "MemoryCurrent": "2560000", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits" 2020-06-12T02:34:11.792 INFO:teuthology.task.ansible.out:: "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2020-06-12 02:30:08 UTC", "StateChangeTimestampMonotonic": "13830209", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2020-06-12 02:30:08 UTC", "WatchdogTimestampMonotonic": "13765967", "WatchdogUSec": "0"}} 2020-06-12T02:34:11.819 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:11 +0000 (0:00:00.290) 0:03:10.772 *********** 2020-06-12T02:34:11.857 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-06-12T02:34:11.859 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2020-06-12T02:34:11.893 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:11 +0000 (0:00:00.074) 0:03:10.847 *********** 2020-06-12T02:34:11.905 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-06-12T02:34:11.907 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2020-06-12 02:29:55 UTC", "ActiveEnterTimestampMonotonic": "13850894", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sntp.service ntpd.service systemd-journald.socket systemd-tmpfiles-setup.service system.slice tmp.mount basic.target ntpdate.service sysinit.target -.mount", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2020-06-12 02:29:54 UTC", "AssertTimestampMonotonic": "12773920", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2020-06-12 02:29:54 UTC", "ConditionTimestampMonotonic": "12773895", "ConfigurationDirectoryMode": "0755", "Conflicts": "ntpd.service shutdown.target systemd-timesyncd.service", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description" 2020-06-12T02:34:11.908 INFO:teuthology.task.ansible.out:: "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "2428", "ExecMainStartTimestamp": "Fri 2020-06-12 02:29:55 UTC", "ExecMainStartTimestampMonotonic": "13775165", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2020-06-12 02:29:54 UTC", "InactiveExitTimestampMonotonic": "12805229", "InvocationID": "09c728df6baf4367b2c999a6eb0d2262", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "167772 2020-06-12T02:34:11.911 INFO:teuthology.task.ansible.out:16", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127436", "LimitNPROCSoft": "127436", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127436", "LimitSIGPENDINGSoft": "127436", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "2428", "MemoryAccounting": "yes", "MemoryCurrent": "2908160", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardE 2020-06-12T02:34:11.918 INFO:teuthology.task.ansible.out:rror": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2020-06-12 02:29:55 UTC", "StateChangeTimestampMonotonic": "13850894", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2020-06-12 02:29:55 UTC", "WatchdogTimestampMonotonic": "13775178", "WatchdogUSec": "0"}} 2020-06-12T02:34:11.947 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:11 +0000 (0:00:00.052) 0:03:10.899 *********** 2020-06-12T02:34:11.956 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-06-12T02:34:11.957 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2020-06-12T02:34:11.993 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:11 +0000 (0:00:00.046) 0:03:10.946 *********** 2020-06-12T02:34:12.004 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi188.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2020-06-12T02:34:12.039 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:12 +0000 (0:00:00.046) 0:03:10.993 *********** 2020-06-12T02:34:12.049 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-06-12T02:34:12.050 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:12.085 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:12 +0000 (0:00:00.045) 0:03:11.038 *********** 2020-06-12T02:34:12.093 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-06-12T02:34:12.095 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2020-06-12T02:34:12.129 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:12 +0000 (0:00:00.043) 0:03:11.082 *********** 2020-06-12T02:34:12.139 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-06-12T02:34:12.141 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:12.176 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:12 +0000 (0:00:00.047) 0:03:11.130 *********** 2020-06-12T02:34:12.185 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-06-12T02:34:12.186 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:12.230 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:12 +0000 (0:00:00.050) 0:03:11.180 *********** 2020-06-12T02:34:12.236 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-06-12T02:34:12.238 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:12.272 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:12 +0000 (0:00:00.045) 0:03:11.225 *********** 2020-06-12T02:34:12.282 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-06-12T02:34:12.284 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:12.316 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:12 +0000 (0:00:00.044) 0:03:11.269 *********** 2020-06-12T02:34:12.362 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:12.401 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:12 +0000 (0:00:00.084) 0:03:11.354 *********** 2020-06-12T02:34:12.489 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-06-12T02:34:12.492 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "gid": 1219, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2020-06-12T02:34:12.528 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:12 +0000 (0:00:00.125) 0:03:11.480 *********** 2020-06-12T02:34:12.635 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"changed": false, "gid": 1219, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2020-06-12T02:34:12.689 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:12 +0000 (0:00:00.162) 0:03:11.642 *********** 2020-06-12T02:34:13.088 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2020-06-12T02:34:13.090 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1219, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2020-06-12T02:34:13.130 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:13 +0000 (0:00:00.441) 0:03:12.083 *********** 2020-06-12T02:34:13.271 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1219, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2020-06-12T02:34:13.307 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:13 +0000 (0:00:00.177) 0:03:12.261 *********** 2020-06-12T02:34:14.478 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2020-06-12T02:34:14.480 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-06-12T02:34:14.513 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:14 +0000 (0:00:01.205) 0:03:13.466 *********** 2020-06-12T02:34:14.525 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-06-12T02:34:14.575 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:14 +0000 (0:00:00.062) 0:03:13.528 *********** 2020-06-12T02:34:14.585 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2020-06-12T02:34:14.587 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:14.622 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:14 +0000 (0:00:00.047) 0:03:13.575 *********** 2020-06-12T02:34:14.632 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:14.671 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:14 +0000 (0:00:00.048) 0:03:13.624 *********** 2020-06-12T02:34:14.685 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ ok: [smithi188.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2020-06-12T02:34:14.720 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:14 +0000 (0:00:00.049) 0:03:13.674 *********** 2020-06-12T02:34:14.730 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2020-06-12T02:34:14.768 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:14 +0000 (0:00:00.047) 0:03:13.721 *********** 2020-06-12T02:34:14.929 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2020-06-12T02:34:14.931 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2020-06-12T02:34:14.981 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:14 +0000 (0:00:00.213) 0:03:13.935 *********** 2020-06-12T02:34:15.002 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2020-06-12T02:34:15.050 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:15 +0000 (0:00:00.068) 0:03:14.003 *********** 2020-06-12T02:34:15.061 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2020-06-12T02:34:15.063 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:15.110 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:15 +0000 (0:00:00.058) 0:03:14.062 *********** 2020-06-12T02:34:15.120 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:15.155 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:15 +0000 (0:00:00.046) 0:03:14.109 *********** 2020-06-12T02:34:15.166 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2020-06-12T02:34:15.169 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-06-12T02:34:15.201 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:15 +0000 (0:00:00.045) 0:03:14.155 *********** 2020-06-12T02:34:15.211 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-06-12T02:34:15.248 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:15 +0000 (0:00:00.046) 0:03:14.201 *********** 2020-06-12T02:34:15.257 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2020-06-12T02:34:15.269 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:15.300 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:15 +0000 (0:00:00.051) 0:03:14.253 *********** 2020-06-12T02:34:15.309 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:15.345 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:15 +0000 (0:00:00.045) 0:03:14.298 *********** 2020-06-12T02:34:15.508 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** changed: [smithi188.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-06-12T02:34:15.563 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:15 +0000 (0:00:00.218) 0:03:14.517 *********** 2020-06-12T02:34:15.573 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-06-12T02:34:15.610 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:15 +0000 (0:00:00.046) 0:03:14.563 *********** 2020-06-12T02:34:15.775 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-06-12T02:34:15.778 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-06-12T02:34:15.811 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:15 +0000 (0:00:00.200) 0:03:14.764 *********** 2020-06-12T02:34:15.826 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-06-12T02:34:15.858 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:15 +0000 (0:00:00.047) 0:03:14.811 *********** 2020-06-12T02:34:15.868 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2020-06-12T02:34:15.870 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:15.910 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:15 +0000 (0:00:00.051) 0:03:14.863 *********** 2020-06-12T02:34:15.919 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:15.955 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:15 +0000 (0:00:00.045) 0:03:14.909 *********** 2020-06-12T02:34:15.976 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-06-12T02:34:15.978 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2020-06-12T02:34:16.013 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:16 +0000 (0:00:00.057) 0:03:14.966 *********** 2020-06-12T02:34:16.033 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2020-06-12T02:34:16.034 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2020-06-12T02:34:16.074 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:16 +0000 (0:00:00.061) 0:03:15.027 *********** 2020-06-12T02:34:16.083 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-06-12T02:34:16.085 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:16.087 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:16.087 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi188.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:16.134 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:16 +0000 (0:00:00.059) 0:03:15.087 *********** 2020-06-12T02:34:16.157 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:16.159 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:16.161 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:16.203 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:16 +0000 (0:00:00.069) 0:03:15.157 *********** 2020-06-12T02:34:16.217 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-06-12T02:34:16.219 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:16.268 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:16 +0000 (0:00:00.062) 0:03:15.219 *********** 2020-06-12T02:34:16.275 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:16.317 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:16 +0000 (0:00:00.050) 0:03:15.270 *********** 2020-06-12T02:34:16.326 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-06-12T02:34:16.328 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:16.362 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:16 +0000 (0:00:00.045) 0:03:15.315 *********** 2020-06-12T02:34:16.372 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:16.411 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:16 +0000 (0:00:00.048) 0:03:15.364 *********** 2020-06-12T02:34:16.420 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-06-12T02:34:16.422 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:16.460 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:16 +0000 (0:00:00.048) 0:03:15.413 *********** 2020-06-12T02:34:16.469 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:16.521 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:16 +0000 (0:00:00.061) 0:03:15.474 *********** 2020-06-12T02:34:16.531 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-06-12T02:34:16.532 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:16.573 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:16 +0000 (0:00:00.050) 0:03:15.525 *********** 2020-06-12T02:34:16.580 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:16.624 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:16 +0000 (0:00:00.047) 0:03:15.572 *********** 2020-06-12T02:34:16.627 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-06-12T02:34:16.629 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:16.665 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:16 +0000 (0:00:00.046) 0:03:15.619 *********** 2020-06-12T02:34:16.677 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:16.721 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:16 +0000 (0:00:00.055) 0:03:15.674 *********** 2020-06-12T02:34:16.731 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-06-12T02:34:16.732 INFO:teuthology.task.ansible.out:skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:16.771 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:16 +0000 (0:00:00.050) 0:03:15.725 *********** 2020-06-12T02:34:16.782 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:16.825 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:16 +0000 (0:00:00.053) 0:03:15.778 *********** 2020-06-12T02:34:16.842 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:16.886 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:16 +0000 (0:00:00.060) 0:03:15.839 *********** 2020-06-12T02:34:16.898 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:16.938 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:16 +0000 (0:00:00.052) 0:03:15.891 *********** 2020-06-12T02:34:16.952 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi188.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:17.003 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:16 +0000 (0:00:00.065) 0:03:15.956 *********** 2020-06-12T02:34:17.012 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-12T02:34:17.050 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:17 +0000 (0:00:00.046) 0:03:16.003 *********** 2020-06-12T02:34:17.251 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-06-12T02:34:17.253 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.004709", "end": "2020-06-12 02:34:17.202832", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-06-12 02:34:17.198123", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-12T02:34:17.319 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.004803", "end": "2020-06-12 02:34:17.252182", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-06-12 02:34:17.247379", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-12T02:34:17.464 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003043", "end": "2020-06-12 02:34:17.420674", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-06-12 02:34:17.417631", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-12T02:34:17.518 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.002937", "end": "2020-06-12 02:34:17.477762", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-06-12 02:34:17.474825", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-12T02:34:17.646 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002847", "end": "2020-06-12 02:34:17.615181", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-06-12 02:34:17.612334", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2020-06-12T02:34:17.699 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002703", "end": "2020-06-12 02:34:17.664789", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-06-12 02:34:17.662086", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2020-06-12T02:34:17.848 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.017275", "end": "2020-06-12 02:34:17.817081", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-06-12 02:34:17.799806", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-12T02:34:17.894 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:17 +0000 (0:00:00.844) 0:03:16.847 *********** 2020-06-12T02:34:17.915 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.014925", "end": "2020-06-12 02:34:17.860742", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-06-12 02:34:17.845817", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-12T02:34:17.952 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:17 +0000 (0:00:00.057) 0:03:16.905 *********** 2020-06-12T02:34:17.961 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-06-12T02:34:17.999 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:17 +0000 (0:00:00.045) 0:03:16.951 *********** 2020-06-12T02:34:18.043 INFO:teuthology.task.ansible.out:Friday 12 June 2020 02:34:18 +0000 (0:00:00.045) 0:03:16.996 *********** 2020-06-12T02:34:18.214 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-06-12T02:34:18.218 INFO:teuthology.task.ansible.out:changed: [smithi188.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2020-06-12T02:34:18.283 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2020-06-12T02:34:18.376 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2020-06-12T02:34:18.378 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2020-06-12T02:34:18.380 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched 2020-06-12T02:34:18.382 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* smithi107.front.sepia.ceph.com : ok=118 changed=37 unreachable=0 failed=0 skipped=171 rescued=0 ignored=3 smithi188.front.sepia.ceph.com : ok=118 changed=37 unreachable=0 failed=0 skipped=171 rescued=0 ignored=3 Friday 12 June 2020 02:34:18 +0000 (0:00:00.337) 0:03:17.334 *********** =============================================================================== users : Create all admin users with sudo access. ----------------------- 26.94s users : Update authorized_keys using the keys repo --------------------- 26.89s common : nrpe - Remove SELinux policy package -------------------------- 16.24s common : nrpe - Load SELinux policy package ---------------------------- 16.23s users : Remove revoked users -------------------------------------------- 8.79s testnode : Zap all non-root disks --------------------------------------- 7.60s common : Make sure smartmontools is installed --------------------------- 2.96s testnode : Install packages --------------------------------------------- 2.71s testnode : Ensure ceph packages are not present. ------------------------ 2.18s testnode : Create logical volume(s) ------------------------------------- 2.09s testnode : Rebuild rpmdb ------------------------------------------------ 1.84s Gathering Facts --------------------------------------------------------- 1.83s testnode : Ensure ceph dependency packages are not present. ------------- 1.82s users : Update authorized_keys for each user with literal keys ---------- 1.76s testnode : Blow away lingering OSD data and FSIDs ----------------------- 1.67s common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/. --- 1.59s testnode : Configure local mirrorlists ---------------------------------- 1.53s testnode : Configure version specific repos in /etc/yum.repos.d/ -------- 1.53s common : Install nrpe package and 2020-06-12T02:34:18.383 INFO:teuthology.task.ansible.out:dependencies (RHEL/CentOS) ------------ 1.52s common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/. --- 1.42s 2020-06-12T02:34:18.787 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi107.front.sepia.ceph.com'), Remote(name='ubuntu@smithi188.front.sepia.ceph.com')] 2020-06-12T02:34:18.788 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-12T02:34:18.790 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:34:18.872 INFO:teuthology.orchestra.run.smithi107:> true 2020-06-12T02:34:18.962 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-12T02:34:18.963 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:34:19.049 INFO:teuthology.orchestra.run.smithi188:> true 2020-06-12T02:34:19.139 INFO:teuthology.run_tasks:Running task clock... 2020-06-12T02:34:19.153 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2020-06-12T02:34:19.154 INFO:teuthology.orchestra.run:Running command with timeout 60 2020-06-12T02:34:19.154 INFO:teuthology.orchestra.run.smithi107:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-06-12T02:34:19.192 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2020-06-12T02:34:19.211 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2020-06-12T02:34:19.256 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: ntpd: command not found 2020-06-12T02:34:19.271 INFO:teuthology.orchestra.run.smithi107.stdout:506 Cannot talk to daemon 2020-06-12T02:34:19.290 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to start ntp.service: Unit ntp.service not found. 2020-06-12T02:34:19.308 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2020-06-12T02:34:19.362 INFO:teuthology.orchestra.run.smithi107.stderr:bash: ntpq: command not found 2020-06-12T02:34:19.365 INFO:teuthology.orchestra.run.smithi107.stdout:210 Number of sources = 4 2020-06-12T02:34:19.365 INFO:teuthology.orchestra.run.smithi107.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2020-06-12T02:34:19.365 INFO:teuthology.orchestra.run.smithi107.stdout:=============================================================================== 2020-06-12T02:34:19.365 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-06-12T02:34:19.366 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-06-12T02:34:19.366 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-06-12T02:34:19.366 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-06-12T02:34:19.366 INFO:teuthology.orchestra.run:Running command with timeout 60 2020-06-12T02:34:19.367 INFO:teuthology.orchestra.run.smithi188:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-06-12T02:34:19.402 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2020-06-12T02:34:19.420 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2020-06-12T02:34:19.459 INFO:teuthology.orchestra.run.smithi188.stderr:sudo: ntpd: command not found 2020-06-12T02:34:19.473 INFO:teuthology.orchestra.run.smithi188.stdout:506 Cannot talk to daemon 2020-06-12T02:34:19.491 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to start ntp.service: Unit ntp.service not found. 2020-06-12T02:34:19.508 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2020-06-12T02:34:19.575 INFO:teuthology.orchestra.run.smithi188.stderr:bash: ntpq: command not found 2020-06-12T02:34:19.577 INFO:teuthology.orchestra.run.smithi188.stdout:210 Number of sources = 4 2020-06-12T02:34:19.577 INFO:teuthology.orchestra.run.smithi188.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2020-06-12T02:34:19.578 INFO:teuthology.orchestra.run.smithi188.stdout:=============================================================================== 2020-06-12T02:34:19.578 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-06-12T02:34:19.578 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-06-12T02:34:19.578 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-06-12T02:34:19.579 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-06-12T02:34:19.580 INFO:teuthology.run_tasks:Running task install... 2020-06-12T02:34:19.593 DEBUG:teuthology.task.install:project ceph 2020-06-12T02:34:19.594 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '91d24f498556f86f6d1318d2efd5210c35b36762'}} 2020-06-12T02:34:19.594 DEBUG:teuthology.task.install:config {'sha1': '91d24f498556f86f6d1318d2efd5210c35b36762'} 2020-06-12T02:34:19.594 INFO:teuthology.task.install:Using flavor: basic 2020-06-12T02:34:19.601 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2020-06-12T02:34:19.602 INFO:teuthology.task.install:extra packages: [] 2020-06-12T02:34:19.602 INFO:teuthology.orchestra.run.smithi107:> sudo lsb_release -is 2020-06-12T02:34:19.642 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS 2020-06-12T02:34:19.643 DEBUG:teuthology.misc:System to be installed: CentOS 2020-06-12T02:34:19.643 INFO:teuthology.orchestra.run.smithi188:> sudo lsb_release -is 2020-06-12T02:34:19.644 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'tag': None, 'sha1': '91d24f498556f86f6d1318d2efd5210c35b36762', 'debuginfo': None, 'flavor': 'basic', 'downgrade_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'exclude_packages': [], 'extras': None, 'wait_for_package': False, 'project': 'ceph', 'packages': {}, 'install_ceph_packages': True, 'repos_only': False} 2020-06-12T02:34:19.645 INFO:teuthology.orchestra.run.smithi107:> sudo lsb_release -is 2020-06-12T02:34:19.682 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS 2020-06-12T02:34:19.683 DEBUG:teuthology.misc:System to be installed: CentOS 2020-06-12T02:34:19.684 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'tag': None, 'sha1': '91d24f498556f86f6d1318d2efd5210c35b36762', 'debuginfo': None, 'flavor': 'basic', 'downgrade_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'exclude_packages': [], 'extras': None, 'wait_for_package': False, 'project': 'ceph', 'packages': {}, 'install_ceph_packages': True, 'repos_only': False} 2020-06-12T02:34:19.685 INFO:teuthology.orchestra.run.smithi188:> sudo lsb_release -is 2020-06-12T02:34:19.720 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS 2020-06-12T02:34:19.721 DEBUG:teuthology.misc:System to be installed: CentOS 2020-06-12T02:34:19.722 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=91d24f498556f86f6d1318d2efd5210c35b36762 2020-06-12T02:34:19.763 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS 2020-06-12T02:34:19.764 DEBUG:teuthology.misc:System to be installed: CentOS 2020-06-12T02:34:19.765 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=91d24f498556f86f6d1318d2efd5210c35b36762 2020-06-12T02:34:19.917 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/octopus/91d24f498556f86f6d1318d2efd5210c35b36762/centos/8/flavors/default/ 2020-06-12T02:34:19.918 INFO:teuthology.task.install.rpm:Package version is 15.2.3-196.g91d24f4 2020-06-12T02:34:19.938 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/octopus/91d24f498556f86f6d1318d2efd5210c35b36762/centos/8/flavors/default/ 2020-06-12T02:34:19.939 INFO:teuthology.task.install.rpm:Package version is 15.2.3-196.g91d24f4 2020-06-12T02:34:20.060 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/octopus/91d24f498556f86f6d1318d2efd5210c35b36762/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/octopus/91d24f498556f86f6d1318d2efd5210c35b36762/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/octopus/91d24f498556f86f6d1318d2efd5210c35b36762/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2020-06-12T02:34:20.061 INFO:teuthology.orchestra.run.smithi107:> sudo sh -c 'cat > /etc/yum.repos.d/ceph.repo' 2020-06-12T02:34:20.070 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/octopus/91d24f498556f86f6d1318d2efd5210c35b36762/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/octopus/91d24f498556f86f6d1318d2efd5210c35b36762/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/octopus/91d24f498556f86f6d1318d2efd5210c35b36762/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2020-06-12T02:34:20.070 INFO:teuthology.orchestra.run.smithi188:> sudo sh -c 'cat > /etc/yum.repos.d/ceph.repo' 2020-06-12T02:34:20.106 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2020-06-12T02:34:20.107 INFO:teuthology.orchestra.run.smithi107:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/91d24f498556f86f6d1318d2efd5210c35b36762/;g' /etc/yum.repos.d/ceph.repo ; fi 2020-06-12T02:34:20.115 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2020-06-12T02:34:20.116 INFO:teuthology.orchestra.run.smithi188:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/91d24f498556f86f6d1318d2efd5210c35b36762/;g' /etc/yum.repos.d/ceph.repo ; fi 2020-06-12T02:34:20.178 INFO:teuthology.orchestra.run.smithi107:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2020-06-12T02:34:20.183 INFO:teuthology.orchestra.run.smithi188:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2020-06-12T02:34:20.260 INFO:teuthology.orchestra.run.smithi107:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2020-06-12T02:34:20.264 INFO:teuthology.orchestra.run.smithi188:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2020-06-12T02:34:20.327 INFO:teuthology.orchestra.run.smithi107.stdout:check_obsoletes = 1 2020-06-12T02:34:20.328 INFO:teuthology.orchestra.run.smithi107:> sudo yum clean all 2020-06-12T02:34:20.330 INFO:teuthology.orchestra.run.smithi188.stdout:check_obsoletes = 1 2020-06-12T02:34:20.331 INFO:teuthology.orchestra.run.smithi188:> sudo yum clean all 2020-06-12T02:34:20.608 INFO:teuthology.orchestra.run.smithi107.stdout:49 files removed 2020-06-12T02:34:20.609 INFO:teuthology.orchestra.run.smithi188.stdout:49 files removed 2020-06-12T02:34:20.636 INFO:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-radosgw 2020-06-12T02:34:20.638 INFO:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-radosgw 2020-06-12T02:34:21.221 INFO:teuthology.orchestra.run.smithi107.stdout:ceph packages for x86_64 280 kB/s | 69 kB 00:00 2020-06-12T02:34:21.223 INFO:teuthology.orchestra.run.smithi188.stdout:ceph packages for x86_64 280 kB/s | 69 kB 00:00 2020-06-12T02:34:21.453 INFO:teuthology.orchestra.run.smithi107.stdout:ceph noarch packages 69 kB/s | 14 kB 00:00 2020-06-12T02:34:21.454 INFO:teuthology.orchestra.run.smithi188.stdout:ceph noarch packages 69 kB/s | 14 kB 00:00 2020-06-12T02:34:21.677 INFO:teuthology.orchestra.run.smithi107.stdout:ceph source packages 6.9 kB/s | 1.4 kB 00:00 2020-06-12T02:34:21.678 INFO:teuthology.orchestra.run.smithi188.stdout:ceph source packages 6.9 kB/s | 1.4 kB 00:00 2020-06-12T02:34:21.907 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS-8 - AppStream 33 MB/s | 7.0 MB 00:00 2020-06-12T02:34:21.922 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS-8 - AppStream 31 MB/s | 7.0 MB 00:00 2020-06-12T02:34:23.105 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS-8 - Base 23 MB/s | 2.2 MB 00:00 2020-06-12T02:34:23.112 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS-8 - Base 22 MB/s | 2.2 MB 00:00 2020-06-12T02:34:23.529 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS-8 - Extras 78 kB/s | 5.5 kB 00:00 2020-06-12T02:34:23.540 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS-8 - Extras 87 kB/s | 5.5 kB 00:00 2020-06-12T02:34:23.672 INFO:teuthology.orchestra.run.smithi188.stdout:Copr repo for ceph-el8 owned by ktdreyer 543 kB/s | 47 kB 00:00 2020-06-12T02:34:23.699 INFO:teuthology.orchestra.run.smithi107.stdout:Copr repo for ceph-el8 owned by ktdreyer 380 kB/s | 47 kB 00:00 2020-06-12T02:34:24.090 INFO:teuthology.orchestra.run.smithi188.stdout:Extra Packages for Enterprise Linux 19 MB/s | 6.9 MB 00:00 2020-06-12T02:34:24.108 INFO:teuthology.orchestra.run.smithi107.stdout:Extra Packages for Enterprise Linux 19 MB/s | 6.9 MB 00:00 2020-06-12T02:34:25.541 INFO:teuthology.orchestra.run.smithi107.stdout:lab-extras 3.9 MB/s | 20 kB 00:00 2020-06-12T02:34:25.551 INFO:teuthology.orchestra.run.smithi188.stdout:lab-extras 3.9 MB/s | 20 kB 00:00 2020-06-12T02:34:26.545 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2020-06-12T02:34:26.563 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:34:26.564 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2020-06-12T02:34:26.564 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:34:26.564 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2020-06-12T02:34:26.564 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw x86_64 2:15.2.3-196.g91d24f4.el8 ceph 9.8 M 2020-06-12T02:34:26.564 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrading: 2020-06-12T02:34:26.565 INFO:teuthology.orchestra.run.smithi107.stdout: librados2 x86_64 2:15.2.3-196.g91d24f4.el8 ceph 3.5 M 2020-06-12T02:34:26.565 INFO:teuthology.orchestra.run.smithi107.stdout: librbd1 x86_64 2:15.2.3-196.g91d24f4.el8 ceph 1.9 M 2020-06-12T02:34:26.565 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2020-06-12T02:34:26.565 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base x86_64 2:15.2.3-196.g91d24f4.el8 ceph 5.5 M 2020-06-12T02:34:26.565 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common x86_64 2:15.2.3-196.g91d24f4.el8 ceph 21 M 2020-06-12T02:34:26.566 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-selinux x86_64 2:15.2.3-196.g91d24f4.el8 ceph 25 k 2020-06-12T02:34:26.566 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2 x86_64 2:15.2.3-196.g91d24f4.el8 ceph 509 k 2020-06-12T02:34:26.566 INFO:teuthology.orchestra.run.smithi107.stdout: libradosstriper1 x86_64 2:15.2.3-196.g91d24f4.el8 ceph 377 k 2020-06-12T02:34:26.566 INFO:teuthology.orchestra.run.smithi107.stdout: librgw2 x86_64 2:15.2.3-196.g91d24f4.el8 ceph 6.2 M 2020-06-12T02:34:26.566 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-argparse x86_64 2:15.2.3-196.g91d24f4.el8 ceph 40 k 2020-06-12T02:34:26.566 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-common x86_64 2:15.2.3-196.g91d24f4.el8 ceph 64 k 2020-06-12T02:34:26.567 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs x86_64 2:15.2.3-196.g91d24f4.el8 ceph 162 k 2020-06-12T02:34:26.567 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados x86_64 2:15.2.3-196.g91d24f4.el8 ceph 351 k 2020-06-12T02:34:26.567 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd x86_64 2:15.2.3-196.g91d24f4.el8 ceph 331 k 2020-06-12T02:34:26.567 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw x86_64 2:15.2.3-196.g91d24f4.el8 ceph 113 k 2020-06-12T02:34:26.567 INFO:teuthology.orchestra.run.smithi107.stdout: librdkafka x86_64 0.11.4-1.el8 CentOS-AppStream 353 k 2020-06-12T02:34:26.568 INFO:teuthology.orchestra.run.smithi107.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2020-06-12T02:34:26.568 INFO:teuthology.orchestra.run.smithi107.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2020-06-12T02:34:26.568 INFO:teuthology.orchestra.run.smithi107.stdout: librabbitmq x86_64 0.9.0-1.el8 CentOS-Base 47 k 2020-06-12T02:34:26.568 INFO:teuthology.orchestra.run.smithi107.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2020-06-12T02:34:26.568 INFO:teuthology.orchestra.run.smithi107.stdout: gperftools-libs x86_64 2.7-6.el8 epel 315 k 2020-06-12T02:34:26.568 INFO:teuthology.orchestra.run.smithi107.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2020-06-12T02:34:26.569 INFO:teuthology.orchestra.run.smithi107.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2020-06-12T02:34:26.569 INFO:teuthology.orchestra.run.smithi107.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2020-06-12T02:34:26.569 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:34:26.569 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2020-06-12T02:34:26.569 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:34:26.573 INFO:teuthology.orchestra.run.smithi107.stdout:Install 22 Packages 2020-06-12T02:34:26.574 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrade 2 Packages 2020-06-12T02:34:26.574 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:34:26.574 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 51 M 2020-06-12T02:34:26.574 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2020-06-12T02:34:26.575 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2020-06-12T02:34:26.576 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:34:26.576 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2020-06-12T02:34:26.576 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:34:26.576 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2020-06-12T02:34:26.576 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-radosgw x86_64 2:15.2.3-196.g91d24f4.el8 ceph 9.8 M 2020-06-12T02:34:26.577 INFO:teuthology.orchestra.run.smithi188.stdout:Upgrading: 2020-06-12T02:34:26.577 INFO:teuthology.orchestra.run.smithi188.stdout: librados2 x86_64 2:15.2.3-196.g91d24f4.el8 ceph 3.5 M 2020-06-12T02:34:26.577 INFO:teuthology.orchestra.run.smithi188.stdout: librbd1 x86_64 2:15.2.3-196.g91d24f4.el8 ceph 1.9 M 2020-06-12T02:34:26.577 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2020-06-12T02:34:26.577 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-base x86_64 2:15.2.3-196.g91d24f4.el8 ceph 5.5 M 2020-06-12T02:34:26.579 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-common x86_64 2:15.2.3-196.g91d24f4.el8 ceph 21 M 2020-06-12T02:34:26.579 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-selinux x86_64 2:15.2.3-196.g91d24f4.el8 ceph 25 k 2020-06-12T02:34:26.579 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs2 x86_64 2:15.2.3-196.g91d24f4.el8 ceph 509 k 2020-06-12T02:34:26.580 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1 x86_64 2:15.2.3-196.g91d24f4.el8 ceph 377 k 2020-06-12T02:34:26.580 INFO:teuthology.orchestra.run.smithi188.stdout: librgw2 x86_64 2:15.2.3-196.g91d24f4.el8 ceph 6.2 M 2020-06-12T02:34:26.580 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-argparse x86_64 2:15.2.3-196.g91d24f4.el8 ceph 40 k 2020-06-12T02:34:26.581 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-common x86_64 2:15.2.3-196.g91d24f4.el8 ceph 64 k 2020-06-12T02:34:26.581 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cephfs x86_64 2:15.2.3-196.g91d24f4.el8 ceph 162 k 2020-06-12T02:34:26.581 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rados x86_64 2:15.2.3-196.g91d24f4.el8 ceph 351 k 2020-06-12T02:34:26.581 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rbd x86_64 2:15.2.3-196.g91d24f4.el8 ceph 331 k 2020-06-12T02:34:26.581 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rgw x86_64 2:15.2.3-196.g91d24f4.el8 ceph 113 k 2020-06-12T02:34:26.581 INFO:teuthology.orchestra.run.smithi188.stdout: librdkafka x86_64 0.11.4-1.el8 CentOS-AppStream 353 k 2020-06-12T02:34:26.582 INFO:teuthology.orchestra.run.smithi188.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2020-06-12T02:34:26.582 INFO:teuthology.orchestra.run.smithi188.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2020-06-12T02:34:26.582 INFO:teuthology.orchestra.run.smithi188.stdout: librabbitmq x86_64 0.9.0-1.el8 CentOS-Base 47 k 2020-06-12T02:34:26.582 INFO:teuthology.orchestra.run.smithi188.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2020-06-12T02:34:26.582 INFO:teuthology.orchestra.run.smithi188.stdout: gperftools-libs x86_64 2.7-6.el8 epel 315 k 2020-06-12T02:34:26.583 INFO:teuthology.orchestra.run.smithi188.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2020-06-12T02:34:26.583 INFO:teuthology.orchestra.run.smithi188.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2020-06-12T02:34:26.583 INFO:teuthology.orchestra.run.smithi188.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2020-06-12T02:34:26.583 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:34:26.583 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2020-06-12T02:34:26.584 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:34:26.584 INFO:teuthology.orchestra.run.smithi188.stdout:Install 22 Packages 2020-06-12T02:34:26.584 INFO:teuthology.orchestra.run.smithi188.stdout:Upgrade 2 Packages 2020-06-12T02:34:26.584 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:34:26.584 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 51 M 2020-06-12T02:34:26.585 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2020-06-12T02:34:27.744 INFO:teuthology.orchestra.run.smithi107.stdout:(1/24): ceph-base-15.2.3-196.g91d24f4.el8.x86_6 4.6 MB/s | 5.5 MB 00:01 2020-06-12T02:34:27.792 INFO:teuthology.orchestra.run.smithi188.stdout:(1/24): ceph-base-15.2.3-196.g91d24f4.el8.x86_6 4.5 MB/s | 5.5 MB 00:01 2020-06-12T02:34:27.807 INFO:teuthology.orchestra.run.smithi107.stdout:(2/24): ceph-selinux-15.2.3-196.g91d24f4.el8.x8 407 kB/s | 25 kB 00:00 2020-06-12T02:34:27.838 INFO:teuthology.orchestra.run.smithi188.stdout:(2/24): ceph-selinux-15.2.3-196.g91d24f4.el8.x8 503 kB/s | 25 kB 00:00 2020-06-12T02:34:27.928 INFO:teuthology.orchestra.run.smithi107.stdout:(3/24): libcephfs2-15.2.3-196.g91d24f4.el8.x86_ 4.0 MB/s | 509 kB 00:00 2020-06-12T02:34:27.966 INFO:teuthology.orchestra.run.smithi188.stdout:(3/24): libcephfs2-15.2.3-196.g91d24f4.el8.x86_ 3.8 MB/s | 509 kB 00:00 2020-06-12T02:34:28.058 INFO:teuthology.orchestra.run.smithi107.stdout:(4/24): libradosstriper1-15.2.3-196.g91d24f4.el 2.8 MB/s | 377 kB 00:00 2020-06-12T02:34:28.091 INFO:teuthology.orchestra.run.smithi188.stdout:(4/24): libradosstriper1-15.2.3-196.g91d24f4.el 3.0 MB/s | 377 kB 00:00 2020-06-12T02:34:28.500 INFO:teuthology.orchestra.run.smithi107.stdout:(5/24): ceph-radosgw-15.2.3-196.g91d24f4.el8.x8 5.0 MB/s | 9.8 MB 00:01 2020-06-12T02:34:28.564 INFO:teuthology.orchestra.run.smithi107.stdout:(6/24): python3-ceph-argparse-15.2.3-196.g91d24 628 kB/s | 40 kB 00:00 2020-06-12T02:34:28.629 INFO:teuthology.orchestra.run.smithi107.stdout:(7/24): python3-ceph-common-15.2.3-196.g91d24f4 980 kB/s | 64 kB 00:00 2020-06-12T02:34:28.699 INFO:teuthology.orchestra.run.smithi107.stdout:(8/24): python3-cephfs-15.2.3-196.g91d24f4.el8. 2.3 MB/s | 162 kB 00:00 2020-06-12T02:34:28.781 INFO:teuthology.orchestra.run.smithi107.stdout:(9/24): python3-rados-15.2.3-196.g91d24f4.el8.x 4.2 MB/s | 351 kB 00:00 2020-06-12T02:34:28.858 INFO:teuthology.orchestra.run.smithi107.stdout:(10/24): python3-rbd-15.2.3-196.g91d24f4.el8.x8 4.2 MB/s | 331 kB 00:00 2020-06-12T02:34:28.927 INFO:teuthology.orchestra.run.smithi107.stdout:(11/24): python3-rgw-15.2.3-196.g91d24f4.el8.x8 1.6 MB/s | 113 kB 00:00 2020-06-12T02:34:28.961 INFO:teuthology.orchestra.run.smithi107.stdout:(12/24): librdkafka-0.11.4-1.el8.x86_64.rpm 9.9 MB/s | 353 kB 00:00 2020-06-12T02:34:28.966 INFO:teuthology.orchestra.run.smithi107.stdout:(13/24): lttng-ust-2.8.1-11.el8.x86_64.rpm 54 MB/s | 259 kB 00:00 2020-06-12T02:34:28.968 INFO:teuthology.orchestra.run.smithi107.stdout:(14/24): python3-prettytable-0.7.2-14.el8.noarc 31 MB/s | 44 kB 00:00 2020-06-12T02:34:28.969 INFO:teuthology.orchestra.run.smithi107.stdout:(15/24): librabbitmq-0.9.0-1.el8.x86_64.rpm 30 MB/s | 47 kB 00:00 2020-06-12T02:34:29.047 INFO:teuthology.orchestra.run.smithi107.stdout:(16/24): fmt-6.2.1-1.el8.x86_64.rpm 1.5 MB/s | 116 kB 00:00 2020-06-12T02:34:29.079 INFO:teuthology.orchestra.run.smithi107.stdout:(17/24): gperftools-libs-2.7-6.el8.x86_64.rpm 9.6 MB/s | 315 kB 00:00 2020-06-12T02:34:29.096 INFO:teuthology.orchestra.run.smithi107.stdout:(18/24): leveldb-1.22-1.el8.x86_64.rpm 11 MB/s | 181 kB 00:00 2020-06-12T02:34:29.111 INFO:teuthology.orchestra.run.smithi107.stdout:(19/24): liboath-2.6.2-3.el8.x86_64.rpm 3.6 MB/s | 59 kB 00:00 2020-06-12T02:34:29.127 INFO:teuthology.orchestra.run.smithi107.stdout:(20/24): libunwind-1.3.1-3.el8.x86_64.rpm 4.6 MB/s | 75 kB 00:00 2020-06-12T02:34:29.261 INFO:teuthology.orchestra.run.smithi107.stdout:(21/24): librgw2-15.2.3-196.g91d24f4.el8.x86_64 5.2 MB/s | 6.2 MB 00:01 2020-06-12T02:34:29.319 INFO:teuthology.orchestra.run.smithi188.stdout:(5/24): librgw2-15.2.3-196.g91d24f4.el8.x86_64. 5.1 MB/s | 6.2 MB 00:01 2020-06-12T02:34:29.389 INFO:teuthology.orchestra.run.smithi188.stdout:(6/24): python3-ceph-argparse-15.2.3-196.g91d24 576 kB/s | 40 kB 00:00 2020-06-12T02:34:29.459 INFO:teuthology.orchestra.run.smithi188.stdout:(7/24): python3-ceph-common-15.2.3-196.g91d24f4 910 kB/s | 64 kB 00:00 2020-06-12T02:34:29.532 INFO:teuthology.orchestra.run.smithi188.stdout:(8/24): python3-cephfs-15.2.3-196.g91d24f4.el8. 2.2 MB/s | 162 kB 00:00 2020-06-12T02:34:29.610 INFO:teuthology.orchestra.run.smithi188.stdout:(9/24): python3-rados-15.2.3-196.g91d24f4.el8.x 4.4 MB/s | 351 kB 00:00 2020-06-12T02:34:29.625 INFO:teuthology.orchestra.run.smithi107.stdout:(22/24): librbd1-15.2.3-196.g91d24f4.el8.x86_64 5.3 MB/s | 1.9 MB 00:00 2020-06-12T02:34:29.690 INFO:teuthology.orchestra.run.smithi188.stdout:(10/24): ceph-radosgw-15.2.3-196.g91d24f4.el8.x 3.1 MB/s | 9.8 MB 00:03 2020-06-12T02:34:29.691 INFO:teuthology.orchestra.run.smithi188.stdout:(11/24): python3-rbd-15.2.3-196.g91d24f4.el8.x8 4.2 MB/s | 331 kB 00:00 2020-06-12T02:34:29.742 INFO:teuthology.orchestra.run.smithi107.stdout:(23/24): librados2-15.2.3-196.g91d24f4.el8.x86_ 5.9 MB/s | 3.5 MB 00:00 2020-06-12T02:34:29.743 INFO:teuthology.orchestra.run.smithi188.stdout:(12/24): python3-rgw-15.2.3-196.g91d24f4.el8.x8 2.8 MB/s | 113 kB 00:00 2020-06-12T02:34:29.743 INFO:teuthology.orchestra.run.smithi188.stdout:(13/24): librdkafka-0.11.4-1.el8.x86_64.rpm 8.8 MB/s | 353 kB 00:00 2020-06-12T02:34:29.743 INFO:teuthology.orchestra.run.smithi188.stdout:(14/24): python3-prettytable-0.7.2-14.el8.noarc 28 MB/s | 44 kB 00:00 2020-06-12T02:34:29.743 INFO:teuthology.orchestra.run.smithi188.stdout:(15/24): librabbitmq-0.9.0-1.el8.x86_64.rpm 30 MB/s | 47 kB 00:00 2020-06-12T02:34:29.765 INFO:teuthology.orchestra.run.smithi188.stdout:(16/24): lttng-ust-2.8.1-11.el8.x86_64.rpm 6.5 MB/s | 259 kB 00:00 2020-06-12T02:34:29.814 INFO:teuthology.orchestra.run.smithi188.stdout:(17/24): fmt-6.2.1-1.el8.x86_64.rpm 1.4 MB/s | 116 kB 00:00 2020-06-12T02:34:29.845 INFO:teuthology.orchestra.run.smithi188.stdout:(18/24): leveldb-1.22-1.el8.x86_64.rpm 5.6 MB/s | 181 kB 00:00 2020-06-12T02:34:29.857 INFO:teuthology.orchestra.run.smithi188.stdout:(19/24): gperftools-libs-2.7-6.el8.x86_64.rpm 3.3 MB/s | 315 kB 00:00 2020-06-12T02:34:29.861 INFO:teuthology.orchestra.run.smithi188.stdout:(20/24): liboath-2.6.2-3.el8.x86_64.rpm 3.7 MB/s | 59 kB 00:00 2020-06-12T02:34:29.873 INFO:teuthology.orchestra.run.smithi188.stdout:(21/24): libunwind-1.3.1-3.el8.x86_64.rpm 4.6 MB/s | 75 kB 00:00 2020-06-12T02:34:29.989 INFO:teuthology.orchestra.run.smithi107.stdout:(24/24): ceph-common-15.2.3-196.g91d24f4.el8.x8 6.0 MB/s | 21 MB 00:03 2020-06-12T02:34:29.992 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2020-06-12T02:34:29.992 INFO:teuthology.orchestra.run.smithi107.stdout:Total 15 MB/s | 51 MB 00:03 2020-06-12T02:34:29.997 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2020-06-12T02:34:30.043 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2020-06-12T02:34:30.043 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2020-06-12T02:34:30.207 INFO:teuthology.orchestra.run.smithi188.stdout:(22/24): ceph-common-15.2.3-196.g91d24f4.el8.x8 5.7 MB/s | 21 MB 00:03 2020-06-12T02:34:30.216 INFO:teuthology.orchestra.run.smithi188.stdout:(23/24): librbd1-15.2.3-196.g91d24f4.el8.x86_64 5.6 MB/s | 1.9 MB 00:00 2020-06-12T02:34:30.229 INFO:teuthology.orchestra.run.smithi188.stdout:(24/24): librados2-15.2.3-196.g91d24f4.el8.x86_ 9.7 MB/s | 3.5 MB 00:00 2020-06-12T02:34:30.231 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2020-06-12T02:34:30.232 INFO:teuthology.orchestra.run.smithi188.stdout:Total 14 MB/s | 51 MB 00:03 2020-06-12T02:34:30.237 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2020-06-12T02:34:30.280 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2020-06-12T02:34:30.280 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2020-06-12T02:34:30.458 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2020-06-12T02:34:30.459 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2020-06-12T02:34:30.684 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2020-06-12T02:34:30.685 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2020-06-12T02:34:31.642 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2020-06-12T02:34:31.703 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2020-06-12T02:34:31.829 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/26 2020-06-12T02:34:31.891 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/26 2020-06-12T02:34:32.023 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/26 2020-06-12T02:34:32.102 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/26 2020-06-12T02:34:32.216 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/26 2020-06-12T02:34:32.299 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/26 2020-06-12T02:34:32.593 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/26 2020-06-12T02:34:32.671 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/26 2020-06-12T02:34:32.741 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : librados2-2:15.2.3-196.g91d24f4.el8.x86_64 4/26 2020-06-12T02:34:32.843 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : librados2-2:15.2.3-196.g91d24f4.el8.x86_64 4/26 2020-06-12T02:34:32.857 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados2-2:15.2.3-196.g91d24f4.el8.x86_64 4/26 2020-06-12T02:34:32.945 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librados2-2:15.2.3-196.g91d24f4.el8.x86_64 4/26 2020-06-12T02:34:33.050 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rados-2:15.2.3-196.g91d24f4.el8.x86_64 5/26 2020-06-12T02:34:33.116 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rados-2:15.2.3-196.g91d24f4.el8.x86_64 5/26 2020-06-12T02:34:33.183 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephfs2-2:15.2.3-196.g91d24f4.el8.x86_64 6/26 2020-06-12T02:34:33.263 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephfs2-2:15.2.3-196.g91d24f4.el8.x86_64 6/26 2020-06-12T02:34:33.422 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephfs2-2:15.2.3-196.g91d24f4.el8.x86_64 6/26 2020-06-12T02:34:33.504 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libcephfs2-2:15.2.3-196.g91d24f4.el8.x86_64 6/26 2020-06-12T02:34:33.641 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : librbd1-2:15.2.3-196.g91d24f4.el8.x86_64 7/26 2020-06-12T02:34:33.690 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librbd1-2:15.2.3-196.g91d24f4.el8.x86_64 7/26 2020-06-12T02:34:33.736 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : librbd1-2:15.2.3-196.g91d24f4.el8.x86_64 7/26 2020-06-12T02:34:33.787 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librbd1-2:15.2.3-196.g91d24f4.el8.x86_64 7/26 2020-06-12T02:34:33.874 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librabbitmq-0.9.0-1.el8.x86_64 8/26 2020-06-12T02:34:33.973 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librabbitmq-0.9.0-1.el8.x86_64 8/26 2020-06-12T02:34:34.025 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 9/26 2020-06-12T02:34:34.124 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 9/26 2020-06-12T02:34:34.626 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 9/26 2020-06-12T02:34:34.719 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 9/26 2020-06-12T02:34:34.791 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librgw2-2:15.2.3-196.g91d24f4.el8.x86_64 10/26 2020-06-12T02:34:34.857 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librgw2-2:15.2.3-196.g91d24f4.el8.x86_64 10/26 2020-06-12T02:34:34.874 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librgw2-2:15.2.3-196.g91d24f4.el8.x86_64 10/26 2020-06-12T02:34:34.951 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librgw2-2:15.2.3-196.g91d24f4.el8.x86_64 10/26 2020-06-12T02:34:35.012 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : leveldb-1.22-1.el8.x86_64 11/26 2020-06-12T02:34:35.128 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : leveldb-1.22-1.el8.x86_64 11/26 2020-06-12T02:34:35.141 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ceph-argparse-2:15.2.3-196.g91d24f4.el8.x8 12/26 2020-06-12T02:34:35.266 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-ceph-argparse-2:15.2.3-196.g91d24f4.el8.x8 12/26 2020-06-12T02:34:35.303 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cephfs-2:15.2.3-196.g91d24f4.el8.x86_64 13/26 2020-06-12T02:34:35.421 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cephfs-2:15.2.3-196.g91d24f4.el8.x86_64 13/26 2020-06-12T02:34:35.553 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rgw-2:15.2.3-196.g91d24f4.el8.x86_64 14/26 2020-06-12T02:34:35.654 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rgw-2:15.2.3-196.g91d24f4.el8.x86_64 14/26 2020-06-12T02:34:35.732 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rbd-2:15.2.3-196.g91d24f4.el8.x86_64 15/26 2020-06-12T02:34:35.835 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rbd-2:15.2.3-196.g91d24f4.el8.x86_64 15/26 2020-06-12T02:34:35.858 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libradosstriper1-2:15.2.3-196.g91d24f4.el8.x86_64 16/26 2020-06-12T02:34:35.957 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libradosstriper1-2:15.2.3-196.g91d24f4.el8.x86_64 16/26 2020-06-12T02:34:35.959 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libradosstriper1-2:15.2.3-196.g91d24f4.el8.x86_64 16/26 2020-06-12T02:34:36.035 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libradosstriper1-2:15.2.3-196.g91d24f4.el8.x86_64 16/26 2020-06-12T02:34:36.161 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/26 2020-06-12T02:34:36.235 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/26 2020-06-12T02:34:36.340 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : gperftools-libs-2.7-6.el8.x86_64 18/26 2020-06-12T02:34:36.415 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : gperftools-libs-2.7-6.el8.x86_64 18/26 2020-06-12T02:34:36.518 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 19/26 2020-06-12T02:34:36.626 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 19/26 2020-06-12T02:34:36.632 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ceph-common-2:15.2.3-196.g91d24f4.el8.x86_ 20/26 2020-06-12T02:34:36.765 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-ceph-common-2:15.2.3-196.g91d24f4.el8.x86_ 20/26 2020-06-12T02:34:39.730 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-common-2:15.2.3-196.g91d24f4.el8.x86_64 21/26 2020-06-12T02:34:39.914 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-common-2:15.2.3-196.g91d24f4.el8.x86_64 21/26 2020-06-12T02:34:39.991 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-common-2:15.2.3-196.g91d24f4.el8.x86_64 21/26 2020-06-12T02:34:40.090 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-common-2:15.2.3-196.g91d24f4.el8.x86_64 21/26 2020-06-12T02:34:40.658 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-common-2:15.2.3-196.g91d24f4.el8.x86_64 21/26 2020-06-12T02:34:40.758 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-common-2:15.2.3-196.g91d24f4.el8.x86_64 21/26 2020-06-12T02:34:40.915 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-base-2:15.2.3-196.g91d24f4.el8.x86_64 22/26 2020-06-12T02:34:41.004 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-base-2:15.2.3-196.g91d24f4.el8.x86_64 22/26 2020-06-12T02:34:41.022 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-base-2:15.2.3-196.g91d24f4.el8.x86_64 22/26 2020-06-12T02:34:41.111 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-base-2:15.2.3-196.g91d24f4.el8.x86_64 22/26 2020-06-12T02:34:41.148 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-selinux-2:15.2.3-196.g91d24f4.el8.x86_64 23/26 2020-06-12T02:34:41.247 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-selinux-2:15.2.3-196.g91d24f4.el8.x86_64 23/26 2020-06-12T02:34:57.337 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-selinux-2:15.2.3-196.g91d24f4.el8.x86_64 23/26 2020-06-12T02:34:57.337 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /sys 2020-06-12T02:34:57.338 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /proc 2020-06-12T02:34:57.338 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /dev 2020-06-12T02:34:57.345 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /run 2020-06-12T02:34:57.346 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /mnt 2020-06-12T02:34:57.346 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /var/tmp 2020-06-12T02:34:57.346 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /home 2020-06-12T02:34:57.347 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /tmp 2020-06-12T02:34:57.347 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /dev 2020-06-12T02:34:57.347 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:34:57.588 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-selinux-2:15.2.3-196.g91d24f4.el8.x86_64 23/26 2020-06-12T02:34:57.588 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /sys 2020-06-12T02:34:57.588 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /proc 2020-06-12T02:34:57.589 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /dev 2020-06-12T02:34:57.589 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /run 2020-06-12T02:34:57.589 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /mnt 2020-06-12T02:34:57.589 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /var/tmp 2020-06-12T02:34:57.589 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /home 2020-06-12T02:34:57.590 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /tmp 2020-06-12T02:34:57.590 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /dev 2020-06-12T02:34:57.590 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:34:58.403 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-radosgw-2:15.2.3-196.g91d24f4.el8.x86_64 24/26 2020-06-12T02:34:58.476 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-radosgw-2:15.2.3-196.g91d24f4.el8.x86_64 24/26 2020-06-12T02:34:58.477 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 25/26 2020-06-12T02:34:58.627 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-radosgw-2:15.2.3-196.g91d24f4.el8.x86_64 24/26 2020-06-12T02:34:58.663 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 25/26 2020-06-12T02:34:58.663 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 26/26 2020-06-12T02:34:58.703 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-radosgw-2:15.2.3-196.g91d24f4.el8.x86_64 24/26 2020-06-12T02:34:58.703 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 25/26 2020-06-12T02:34:58.920 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 25/26 2020-06-12T02:34:58.921 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 26/26 2020-06-12T02:34:59.312 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 26/26 2020-06-12T02:34:59.312 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-base-2:15.2.3-196.g91d24f4.el8.x86_64 1/26 2020-06-12T02:34:59.312 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-common-2:15.2.3-196.g91d24f4.el8.x86_64 2/26 2020-06-12T02:34:59.312 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-radosgw-2:15.2.3-196.g91d24f4.el8.x86_64 3/26 2020-06-12T02:34:59.313 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-selinux-2:15.2.3-196.g91d24f4.el8.x86_64 4/26 2020-06-12T02:34:59.313 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephfs2-2:15.2.3-196.g91d24f4.el8.x86_64 5/26 2020-06-12T02:34:59.313 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libradosstriper1-2:15.2.3-196.g91d24f4.el8.x86_64 6/26 2020-06-12T02:34:59.314 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librgw2-2:15.2.3-196.g91d24f4.el8.x86_64 7/26 2020-06-12T02:34:59.314 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ceph-argparse-2:15.2.3-196.g91d24f4.el8.x8 8/26 2020-06-12T02:34:59.314 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ceph-common-2:15.2.3-196.g91d24f4.el8.x86_ 9/26 2020-06-12T02:34:59.315 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cephfs-2:15.2.3-196.g91d24f4.el8.x86_64 10/26 2020-06-12T02:34:59.315 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rados-2:15.2.3-196.g91d24f4.el8.x86_64 11/26 2020-06-12T02:34:59.315 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rbd-2:15.2.3-196.g91d24f4.el8.x86_64 12/26 2020-06-12T02:34:59.316 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rgw-2:15.2.3-196.g91d24f4.el8.x86_64 13/26 2020-06-12T02:34:59.316 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 14/26 2020-06-12T02:34:59.316 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/26 2020-06-12T02:34:59.317 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/26 2020-06-12T02:34:59.317 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librabbitmq-0.9.0-1.el8.x86_64 17/26 2020-06-12T02:34:59.317 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 18/26 2020-06-12T02:34:59.318 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : gperftools-libs-2.7-6.el8.x86_64 19/26 2020-06-12T02:34:59.318 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : leveldb-1.22-1.el8.x86_64 20/26 2020-06-12T02:34:59.318 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 21/26 2020-06-12T02:34:59.320 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 22/26 2020-06-12T02:34:59.320 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados2-2:15.2.3-196.g91d24f4.el8.x86_64 23/26 2020-06-12T02:34:59.320 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 24/26 2020-06-12T02:34:59.321 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librbd1-2:15.2.3-196.g91d24f4.el8.x86_64 25/26 2020-06-12T02:34:59.579 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 26/26 2020-06-12T02:34:59.580 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-base-2:15.2.3-196.g91d24f4.el8.x86_64 1/26 2020-06-12T02:34:59.580 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-common-2:15.2.3-196.g91d24f4.el8.x86_64 2/26 2020-06-12T02:34:59.580 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-radosgw-2:15.2.3-196.g91d24f4.el8.x86_64 3/26 2020-06-12T02:34:59.581 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-selinux-2:15.2.3-196.g91d24f4.el8.x86_64 4/26 2020-06-12T02:34:59.581 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephfs2-2:15.2.3-196.g91d24f4.el8.x86_64 5/26 2020-06-12T02:34:59.582 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libradosstriper1-2:15.2.3-196.g91d24f4.el8.x86_64 6/26 2020-06-12T02:34:59.582 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librgw2-2:15.2.3-196.g91d24f4.el8.x86_64 7/26 2020-06-12T02:34:59.582 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-ceph-argparse-2:15.2.3-196.g91d24f4.el8.x8 8/26 2020-06-12T02:34:59.585 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-ceph-common-2:15.2.3-196.g91d24f4.el8.x86_ 9/26 2020-06-12T02:34:59.585 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cephfs-2:15.2.3-196.g91d24f4.el8.x86_64 10/26 2020-06-12T02:34:59.585 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rados-2:15.2.3-196.g91d24f4.el8.x86_64 11/26 2020-06-12T02:34:59.585 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rbd-2:15.2.3-196.g91d24f4.el8.x86_64 12/26 2020-06-12T02:34:59.585 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rgw-2:15.2.3-196.g91d24f4.el8.x86_64 13/26 2020-06-12T02:34:59.586 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 14/26 2020-06-12T02:34:59.586 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/26 2020-06-12T02:34:59.586 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/26 2020-06-12T02:34:59.586 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librabbitmq-0.9.0-1.el8.x86_64 17/26 2020-06-12T02:34:59.586 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 18/26 2020-06-12T02:34:59.587 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : gperftools-libs-2.7-6.el8.x86_64 19/26 2020-06-12T02:34:59.587 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : leveldb-1.22-1.el8.x86_64 20/26 2020-06-12T02:34:59.587 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 21/26 2020-06-12T02:34:59.587 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 22/26 2020-06-12T02:34:59.587 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados2-2:15.2.3-196.g91d24f4.el8.x86_64 23/26 2020-06-12T02:34:59.587 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 24/26 2020-06-12T02:34:59.588 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librbd1-2:15.2.3-196.g91d24f4.el8.x86_64 25/26 2020-06-12T02:35:00.225 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 26/26 2020-06-12T02:35:00.225 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:35:00.225 INFO:teuthology.orchestra.run.smithi107.stdout:Upgraded: 2020-06-12T02:35:00.226 INFO:teuthology.orchestra.run.smithi107.stdout: librados2-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:00.226 INFO:teuthology.orchestra.run.smithi107.stdout: librbd1-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:00.226 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:35:00.226 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2020-06-12T02:35:00.226 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:00.227 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:00.227 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:00.227 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-selinux-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:00.227 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:00.227 INFO:teuthology.orchestra.run.smithi107.stdout: libradosstriper1-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:00.228 INFO:teuthology.orchestra.run.smithi107.stdout: librgw2-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:00.228 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-argparse-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:00.228 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-common-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:00.229 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:00.229 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:00.229 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:00.229 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:00.229 INFO:teuthology.orchestra.run.smithi107.stdout: librdkafka-0.11.4-1.el8.x86_64 2020-06-12T02:35:00.230 INFO:teuthology.orchestra.run.smithi107.stdout: lttng-ust-2.8.1-11.el8.x86_64 2020-06-12T02:35:00.230 INFO:teuthology.orchestra.run.smithi107.stdout: python3-prettytable-0.7.2-14.el8.noarch 2020-06-12T02:35:00.230 INFO:teuthology.orchestra.run.smithi107.stdout: librabbitmq-0.9.0-1.el8.x86_64 2020-06-12T02:35:00.230 INFO:teuthology.orchestra.run.smithi107.stdout: fmt-6.2.1-1.el8.x86_64 2020-06-12T02:35:00.230 INFO:teuthology.orchestra.run.smithi107.stdout: gperftools-libs-2.7-6.el8.x86_64 2020-06-12T02:35:00.230 INFO:teuthology.orchestra.run.smithi107.stdout: leveldb-1.22-1.el8.x86_64 2020-06-12T02:35:00.231 INFO:teuthology.orchestra.run.smithi107.stdout: liboath-2.6.2-3.el8.x86_64 2020-06-12T02:35:00.231 INFO:teuthology.orchestra.run.smithi107.stdout: libunwind-1.3.1-3.el8.x86_64 2020-06-12T02:35:00.231 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:35:00.231 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2020-06-12T02:35:00.490 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 26/26 2020-06-12T02:35:00.491 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:35:00.491 INFO:teuthology.orchestra.run.smithi188.stdout:Upgraded: 2020-06-12T02:35:00.491 INFO:teuthology.orchestra.run.smithi188.stdout: librados2-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:00.491 INFO:teuthology.orchestra.run.smithi188.stdout: librbd1-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:00.491 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:35:00.492 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2020-06-12T02:35:00.492 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-radosgw-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:00.492 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-base-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:00.492 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-common-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:00.492 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-selinux-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:00.493 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs2-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:00.493 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:00.493 INFO:teuthology.orchestra.run.smithi188.stdout: librgw2-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:00.493 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-argparse-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:00.493 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-common-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:00.494 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cephfs-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:00.494 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rados-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:00.494 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rbd-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:00.494 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rgw-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:00.494 INFO:teuthology.orchestra.run.smithi188.stdout: librdkafka-0.11.4-1.el8.x86_64 2020-06-12T02:35:00.495 INFO:teuthology.orchestra.run.smithi188.stdout: lttng-ust-2.8.1-11.el8.x86_64 2020-06-12T02:35:00.495 INFO:teuthology.orchestra.run.smithi188.stdout: python3-prettytable-0.7.2-14.el8.noarch 2020-06-12T02:35:00.495 INFO:teuthology.orchestra.run.smithi188.stdout: librabbitmq-0.9.0-1.el8.x86_64 2020-06-12T02:35:00.495 INFO:teuthology.orchestra.run.smithi188.stdout: fmt-6.2.1-1.el8.x86_64 2020-06-12T02:35:00.495 INFO:teuthology.orchestra.run.smithi188.stdout: gperftools-libs-2.7-6.el8.x86_64 2020-06-12T02:35:00.495 INFO:teuthology.orchestra.run.smithi188.stdout: leveldb-1.22-1.el8.x86_64 2020-06-12T02:35:00.496 INFO:teuthology.orchestra.run.smithi188.stdout: liboath-2.6.2-3.el8.x86_64 2020-06-12T02:35:00.500 INFO:teuthology.orchestra.run.smithi188.stdout: libunwind-1.3.1-3.el8.x86_64 2020-06-12T02:35:00.500 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:35:00.501 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2020-06-12T02:35:00.573 INFO:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-test 2020-06-12T02:35:00.979 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:35 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:35:01.273 INFO:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-test 2020-06-12T02:35:01.335 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2020-06-12T02:35:01.336 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:35:01.336 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2020-06-12T02:35:01.337 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:35:01.337 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2020-06-12T02:35:01.337 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test x86_64 2:15.2.3-196.g91d24f4.el8 ceph 42 M 2020-06-12T02:35:01.337 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2020-06-12T02:35:01.337 INFO:teuthology.orchestra.run.smithi107.stdout: jq x86_64 1.5-12.el8 CentOS-AppStream 161 k 2020-06-12T02:35:01.338 INFO:teuthology.orchestra.run.smithi107.stdout: oniguruma x86_64 6.8.2-1.el8 CentOS-AppStream 188 k 2020-06-12T02:35:01.338 INFO:teuthology.orchestra.run.smithi107.stdout: socat x86_64 1.7.3.2-6.el8 CentOS-AppStream 298 k 2020-06-12T02:35:01.338 INFO:teuthology.orchestra.run.smithi107.stdout: libxslt x86_64 1.1.32-3.el8 CentOS-Base 249 k 2020-06-12T02:35:01.338 INFO:teuthology.orchestra.run.smithi107.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2020-06-12T02:35:01.338 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:35:01.339 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2020-06-12T02:35:01.339 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:35:01.339 INFO:teuthology.orchestra.run.smithi107.stdout:Install 6 Packages 2020-06-12T02:35:01.339 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:35:01.340 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 43 M 2020-06-12T02:35:01.340 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 185 M 2020-06-12T02:35:01.340 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2020-06-12T02:35:01.373 INFO:teuthology.orchestra.run.smithi107.stdout:(1/6): jq-1.5-12.el8.x86_64.rpm 4.7 MB/s | 161 kB 00:00 2020-06-12T02:35:01.377 INFO:teuthology.orchestra.run.smithi107.stdout:(2/6): oniguruma-6.8.2-1.el8.x86_64.rpm 4.8 MB/s | 188 kB 00:00 2020-06-12T02:35:01.380 INFO:teuthology.orchestra.run.smithi107.stdout:(3/6): socat-1.7.3.2-6.el8.x86_64.rpm 40 MB/s | 298 kB 00:00 2020-06-12T02:35:01.381 INFO:teuthology.orchestra.run.smithi107.stdout:(4/6): libxslt-1.1.32-3.el8.x86_64.rpm 55 MB/s | 249 kB 00:00 2020-06-12T02:35:01.443 INFO:teuthology.orchestra.run.smithi107.stdout:(5/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 1.1 MB/s | 69 kB 00:00 2020-06-12T02:35:01.689 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:00:36 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:35:02.050 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2020-06-12T02:35:02.052 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:35:02.052 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2020-06-12T02:35:02.052 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:35:02.053 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2020-06-12T02:35:02.053 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-test x86_64 2:15.2.3-196.g91d24f4.el8 ceph 42 M 2020-06-12T02:35:02.053 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2020-06-12T02:35:02.053 INFO:teuthology.orchestra.run.smithi188.stdout: jq x86_64 1.5-12.el8 CentOS-AppStream 161 k 2020-06-12T02:35:02.054 INFO:teuthology.orchestra.run.smithi188.stdout: oniguruma x86_64 6.8.2-1.el8 CentOS-AppStream 188 k 2020-06-12T02:35:02.054 INFO:teuthology.orchestra.run.smithi188.stdout: socat x86_64 1.7.3.2-6.el8 CentOS-AppStream 298 k 2020-06-12T02:35:02.054 INFO:teuthology.orchestra.run.smithi188.stdout: libxslt x86_64 1.1.32-3.el8 CentOS-Base 249 k 2020-06-12T02:35:02.054 INFO:teuthology.orchestra.run.smithi188.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2020-06-12T02:35:02.055 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:35:02.057 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2020-06-12T02:35:02.058 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:35:02.058 INFO:teuthology.orchestra.run.smithi188.stdout:Install 6 Packages 2020-06-12T02:35:02.058 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:35:02.058 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 43 M 2020-06-12T02:35:02.058 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 185 M 2020-06-12T02:35:02.059 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2020-06-12T02:35:02.089 INFO:teuthology.orchestra.run.smithi188.stdout:(1/6): jq-1.5-12.el8.x86_64.rpm 4.5 MB/s | 161 kB 00:00 2020-06-12T02:35:02.104 INFO:teuthology.orchestra.run.smithi188.stdout:(2/6): socat-1.7.3.2-6.el8.x86_64.rpm 54 MB/s | 298 kB 00:00 2020-06-12T02:35:02.105 INFO:teuthology.orchestra.run.smithi188.stdout:(3/6): oniguruma-6.8.2-1.el8.x86_64.rpm 4.4 MB/s | 188 kB 00:00 2020-06-12T02:35:02.107 INFO:teuthology.orchestra.run.smithi188.stdout:(4/6): libxslt-1.1.32-3.el8.x86_64.rpm 54 MB/s | 249 kB 00:00 2020-06-12T02:35:02.159 INFO:teuthology.orchestra.run.smithi188.stdout:(5/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 1.1 MB/s | 69 kB 00:00 2020-06-12T02:35:05.069 INFO:teuthology.orchestra.run.smithi107.stdout:(6/6): ceph-test-15.2.3-196.g91d24f4.el8.x86_64 11 MB/s | 42 MB 00:03 2020-06-12T02:35:05.072 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2020-06-12T02:35:05.073 INFO:teuthology.orchestra.run.smithi107.stdout:Total 11 MB/s | 43 MB 00:03 2020-06-12T02:35:05.083 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2020-06-12T02:35:05.101 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2020-06-12T02:35:05.101 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2020-06-12T02:35:05.371 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2020-06-12T02:35:05.372 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2020-06-12T02:35:05.808 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2020-06-12T02:35:06.052 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libxslt-1.1.32-3.el8.x86_64 1/6 2020-06-12T02:35:06.223 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 2/6 2020-06-12T02:35:06.450 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : socat-1.7.3.2-6.el8.x86_64 3/6 2020-06-12T02:35:07.309 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : oniguruma-6.8.2-1.el8.x86_64 4/6 2020-06-12T02:35:07.408 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: oniguruma-6.8.2-1.el8.x86_64 4/6 2020-06-12T02:35:07.613 INFO:teuthology.orchestra.run.smithi188.stdout:(6/6): ceph-test-15.2.3-196.g91d24f4.el8.x86_64 7.5 MB/s | 42 MB 00:05 2020-06-12T02:35:07.615 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2020-06-12T02:35:07.616 INFO:teuthology.orchestra.run.smithi188.stdout:Total 7.7 MB/s | 43 MB 00:05 2020-06-12T02:35:07.622 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2020-06-12T02:35:07.644 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2020-06-12T02:35:07.644 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2020-06-12T02:35:07.904 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2020-06-12T02:35:07.905 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2020-06-12T02:35:08.350 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2020-06-12T02:35:08.559 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libxslt-1.1.32-3.el8.x86_64 1/6 2020-06-12T02:35:08.740 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 2/6 2020-06-12T02:35:08.905 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : socat-1.7.3.2-6.el8.x86_64 3/6 2020-06-12T02:35:09.033 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : oniguruma-6.8.2-1.el8.x86_64 4/6 2020-06-12T02:35:09.097 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: oniguruma-6.8.2-1.el8.x86_64 4/6 2020-06-12T02:35:11.411 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : jq-1.5-12.el8.x86_64 5/6 2020-06-12T02:35:11.680 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-test-2:15.2.3-196.g91d24f4.el8.x86_64 6/6 2020-06-12T02:35:11.892 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-test-2:15.2.3-196.g91d24f4.el8.x86_64 6/6 2020-06-12T02:35:11.892 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-test-2:15.2.3-196.g91d24f4.el8.x86_64 1/6 2020-06-12T02:35:11.893 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : jq-1.5-12.el8.x86_64 2/6 2020-06-12T02:35:11.893 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : oniguruma-6.8.2-1.el8.x86_64 3/6 2020-06-12T02:35:11.894 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : socat-1.7.3.2-6.el8.x86_64 4/6 2020-06-12T02:35:11.894 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libxslt-1.1.32-3.el8.x86_64 5/6 2020-06-12T02:35:12.246 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2020-06-12T02:35:12.246 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:35:12.247 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2020-06-12T02:35:12.247 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test-2:15.2.3-196.g91d24f4.el8.x86_64 jq-1.5-12.el8.x86_64 2020-06-12T02:35:12.247 INFO:teuthology.orchestra.run.smithi107.stdout: oniguruma-6.8.2-1.el8.x86_64 socat-1.7.3.2-6.el8.x86_64 2020-06-12T02:35:12.247 INFO:teuthology.orchestra.run.smithi107.stdout: libxslt-1.1.32-3.el8.x86_64 xmlstarlet-1.6.1-11.el8.x86_64 2020-06-12T02:35:12.247 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:35:12.248 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2020-06-12T02:35:12.476 INFO:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph 2020-06-12T02:35:12.877 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:47 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:35:13.143 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : jq-1.5-12.el8.x86_64 5/6 2020-06-12T02:35:13.396 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-test-2:15.2.3-196.g91d24f4.el8.x86_64 6/6 2020-06-12T02:35:13.602 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-test-2:15.2.3-196.g91d24f4.el8.x86_64 6/6 2020-06-12T02:35:13.603 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-test-2:15.2.3-196.g91d24f4.el8.x86_64 1/6 2020-06-12T02:35:13.603 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : jq-1.5-12.el8.x86_64 2/6 2020-06-12T02:35:13.603 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : oniguruma-6.8.2-1.el8.x86_64 3/6 2020-06-12T02:35:13.604 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : socat-1.7.3.2-6.el8.x86_64 4/6 2020-06-12T02:35:13.604 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libxslt-1.1.32-3.el8.x86_64 5/6 2020-06-12T02:35:13.716 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2020-06-12T02:35:13.720 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:35:13.721 INFO:teuthology.orchestra.run.smithi107.stdout: Package ArchVersion Repository Size 2020-06-12T02:35:13.721 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:35:13.721 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2020-06-12T02:35:13.722 INFO:teuthology.orchestra.run.smithi107.stdout: ceph x86_642:15.2.3-196.g91d24f4.el8 ceph 6.4 k 2020-06-12T02:35:13.722 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2020-06-12T02:35:13.722 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds x86_642:15.2.3-196.g91d24f4.el8 ceph 2.0 M 2020-06-12T02:35:13.722 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr x86_642:15.2.3-196.g91d24f4.el8 ceph 1.4 M 2020-06-12T02:35:13.723 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon x86_642:15.2.3-196.g91d24f4.el8 ceph 4.4 M 2020-06-12T02:35:13.723 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd x86_642:15.2.3-196.g91d24f4.el8 ceph 17 M 2020-06-12T02:35:13.723 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-modules-core noarch2:15.2.3-196.g91d24f4.el8 ceph-noarch 165 k 2020-06-12T02:35:13.723 INFO:teuthology.orchestra.run.smithi107.stdout: python3-lxml x86_644.2.3-1.el8 CentOS-AppStream 1.5 M 2020-06-12T02:35:13.723 INFO:teuthology.orchestra.run.smithi107.stdout: python3-mako noarch1.0.6-13.el8 CentOS-AppStream 157 k 2020-06-12T02:35:13.724 INFO:teuthology.orchestra.run.smithi107.stdout: python3-markupsafe x86_640.23-19.el8 CentOS-AppStream 39 k 2020-06-12T02:35:13.724 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytz noarch2017.2-9.el8 CentOS-AppStream 54 k 2020-06-12T02:35:13.724 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webencodings noarch0.5.1-6.el8 CentOS-AppStream 27 k 2020-06-12T02:35:13.724 INFO:teuthology.orchestra.run.smithi107.stdout: python3-werkzeug noarch0.12.2-4.el8 CentOS-AppStream 457 k 2020-06-12T02:35:13.724 INFO:teuthology.orchestra.run.smithi107.stdout: python3-chardet noarch3.0.4-7.el8 CentOS-Base 195 k 2020-06-12T02:35:13.724 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pysocks noarch1.6.8-3.el8 CentOS-Base 34 k 2020-06-12T02:35:13.725 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests noarch2.20.0-2.1.el8_1 CentOS-Base 123 k 2020-06-12T02:35:13.725 INFO:teuthology.orchestra.run.smithi107.stdout: python3-urllib3 noarch1.24.2-2.el8 CentOS-Base 176 k 2020-06-12T02:35:13.725 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cheroot noarch8.2.1-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 145 k 2020-06-12T02:35:13.725 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cherrypy noarch18.4.0-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 385 k 2020-06-12T02:35:13.726 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco noarch6.2-6.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 11 k 2020-06-12T02:35:13.727 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-functools noarch2.0-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 18 k 2020-06-12T02:35:13.727 INFO:teuthology.orchestra.run.smithi107.stdout: python3-logutils noarch0.3.5-11.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 49 k 2020-06-12T02:35:13.727 INFO:teuthology.orchestra.run.smithi107.stdout: python3-more-itertools noarch7.2.0-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 59 k 2020-06-12T02:35:13.728 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pecan noarch1.3.2-9.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 283 k 2020-06-12T02:35:13.728 INFO:teuthology.orchestra.run.smithi107.stdout: python3-portend noarch2.6-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 16 k 2020-06-12T02:35:13.728 INFO:teuthology.orchestra.run.smithi107.stdout: python3-simplegeneric noarch0.8.1-17.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 19 k 2020-06-12T02:35:13.728 INFO:teuthology.orchestra.run.smithi107.stdout: python3-singledispatch noarch3.4.0.3-18.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 24 k 2020-06-12T02:35:13.729 INFO:teuthology.orchestra.run.smithi107.stdout: python3-tempora noarch1.14.1-5.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 30 k 2020-06-12T02:35:13.729 INFO:teuthology.orchestra.run.smithi107.stdout: python3-trustme noarch0.5.2-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 27 k 2020-06-12T02:35:13.729 INFO:teuthology.orchestra.run.smithi107.stdout: python3-zc-lockfile noarch2.0-2.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 23 k 2020-06-12T02:35:13.729 INFO:teuthology.orchestra.run.smithi107.stdout: python3-bcrypt x86_643.1.6-2.el8.1 epel 44 k 2020-06-12T02:35:13.729 INFO:teuthology.orchestra.run.smithi107.stdout: python3-beautifulsoup4 noarch4.6.3-2.el8.1 epel 185 k 2020-06-12T02:35:13.730 INFO:teuthology.orchestra.run.smithi107.stdout: python3-waitress noarch1.2.1-2.el8.1 epel 241 k 2020-06-12T02:35:13.730 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webob noarch1.8.5-1.el8.1 epel 251 k 2020-06-12T02:35:13.730 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webtest noarch2.0.33-1.el8 epel 85 k 2020-06-12T02:35:13.730 INFO:teuthology.orchestra.run.smithi107.stdout:Installing weak dependencies: 2020-06-12T02:35:13.730 INFO:teuthology.orchestra.run.smithi107.stdout: python3-html5lib noarch1:0.999999999-6.el8 CentOS-AppStream 214 k 2020-06-12T02:35:13.731 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cssselect noarch0.9.2-10.el8 epel 40 k 2020-06-12T02:35:13.731 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:35:13.731 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2020-06-12T02:35:13.731 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:35:13.731 INFO:teuthology.orchestra.run.smithi107.stdout:Install 36 Packages 2020-06-12T02:35:13.732 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:35:13.732 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 29 M 2020-06-12T02:35:13.733 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 105 M 2020-06-12T02:35:13.733 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2020-06-12T02:35:13.905 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2020-06-12T02:35:13.905 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:35:13.905 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2020-06-12T02:35:13.906 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-test-2:15.2.3-196.g91d24f4.el8.x86_64 jq-1.5-12.el8.x86_64 2020-06-12T02:35:13.906 INFO:teuthology.orchestra.run.smithi188.stdout: oniguruma-6.8.2-1.el8.x86_64 socat-1.7.3.2-6.el8.x86_64 2020-06-12T02:35:13.906 INFO:teuthology.orchestra.run.smithi188.stdout: libxslt-1.1.32-3.el8.x86_64 xmlstarlet-1.6.1-11.el8.x86_64 2020-06-12T02:35:13.907 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:35:13.907 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2020-06-12T02:35:13.909 INFO:teuthology.orchestra.run.smithi107.stdout:(1/36): ceph-15.2.3-196.g91d24f4.el8.x86_64.rpm 35 kB/s | 6.4 kB 00:00 2020-06-12T02:35:14.054 INFO:teuthology.orchestra.run.smithi107.stdout:(2/36): ceph-mgr-15.2.3-196.g91d24f4.el8.x86_64 4.4 MB/s | 1.4 MB 00:00 2020-06-12T02:35:14.089 INFO:teuthology.orchestra.run.smithi107.stdout:(3/36): ceph-mds-15.2.3-196.g91d24f4.el8.x86_64 5.6 MB/s | 2.0 MB 00:00 2020-06-12T02:35:14.124 INFO:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph 2020-06-12T02:35:14.159 INFO:teuthology.orchestra.run.smithi107.stdout:(4/36): ceph-mgr-modules-core-15.2.3-196.g91d24 2.3 MB/s | 165 kB 00:00 2020-06-12T02:35:14.210 INFO:teuthology.orchestra.run.smithi107.stdout:(5/36): python3-html5lib-0.999999999-6.el8.noar 4.1 MB/s | 214 kB 00:00 2020-06-12T02:35:14.287 INFO:teuthology.orchestra.run.smithi107.stdout:(6/36): python3-lxml-4.2.3-1.el8.x86_64.rpm 20 MB/s | 1.5 MB 00:00 2020-06-12T02:35:14.325 INFO:teuthology.orchestra.run.smithi107.stdout:(7/36): python3-mako-1.0.6-13.el8.noarch.rpm 4.1 MB/s | 157 kB 00:00 2020-06-12T02:35:14.360 INFO:teuthology.orchestra.run.smithi107.stdout:(8/36): python3-markupsafe-0.23-19.el8.x86_64.r 1.1 MB/s | 39 kB 00:00 2020-06-12T02:35:14.362 INFO:teuthology.orchestra.run.smithi107.stdout:(9/36): python3-pytz-2017.2-9.el8.noarch.rpm 35 MB/s | 54 kB 00:00 2020-06-12T02:35:14.363 INFO:teuthology.orchestra.run.smithi107.stdout:(10/36): python3-webencodings-0.5.1-6.el8.noarc 24 MB/s | 27 kB 00:00 2020-06-12T02:35:14.370 INFO:teuthology.orchestra.run.smithi107.stdout:(11/36): python3-werkzeug-0.12.2-4.el8.noarch.r 75 MB/s | 457 kB 00:00 2020-06-12T02:35:14.373 INFO:teuthology.orchestra.run.smithi107.stdout:(12/36): python3-chardet-3.0.4-7.el8.noarch.rpm 63 MB/s | 195 kB 00:00 2020-06-12T02:35:14.374 INFO:teuthology.orchestra.run.smithi107.stdout:(13/36): python3-pysocks-1.6.8-3.el8.noarch.rpm 27 MB/s | 34 kB 00:00 2020-06-12T02:35:14.376 INFO:teuthology.orchestra.run.smithi107.stdout:(14/36): python3-requests-2.20.0-2.1.el8_1.noar 57 MB/s | 123 kB 00:00 2020-06-12T02:35:14.379 INFO:teuthology.orchestra.run.smithi107.stdout:(15/36): python3-urllib3-1.24.2-2.el8.noarch.rp 63 MB/s | 176 kB 00:00 2020-06-12T02:35:14.440 INFO:teuthology.orchestra.run.smithi107.stdout:(16/36): python3-cheroot-8.2.1-1.el8.noarch.rpm 2.5 MB/s | 145 kB 00:00 2020-06-12T02:35:14.472 INFO:teuthology.orchestra.run.smithi107.stdout:(17/36): python3-cherrypy-18.4.0-1.el8.noarch.r 12 MB/s | 385 kB 00:00 2020-06-12T02:35:14.482 INFO:teuthology.orchestra.run.smithi107.stdout:(18/36): python3-jaraco-6.2-6.el8.noarch.rpm 979 kB/s | 11 kB 00:00 2020-06-12T02:35:14.489 INFO:teuthology.orchestra.run.smithi107.stdout:(19/36): python3-jaraco-functools-2.0-4.el8.noa 2.1 MB/s | 18 kB 00:00 2020-06-12T02:35:14.498 INFO:teuthology.orchestra.run.smithi107.stdout:(20/36): python3-logutils-0.3.5-11.el8.noarch.r 5.4 MB/s | 49 kB 00:00 2020-06-12T02:35:14.507 INFO:teuthology.orchestra.run.smithi107.stdout:(21/36): python3-more-itertools-7.2.0-4.el8.noa 6.6 MB/s | 59 kB 00:00 2020-06-12T02:35:14.522 INFO:teuthology.orchestra.run.smithi107.stdout:(22/36): python3-pecan-1.3.2-9.el8.noarch.rpm 23 MB/s | 283 kB 00:00 2020-06-12T02:35:14.528 INFO:teuthology.orchestra.run.smithi107.stdout:(23/36): python3-portend-2.6-1.el8.noarch.rpm 1.9 MB/s | 16 kB 00:00 2020-06-12T02:35:14.536 INFO:teuthology.orchestra.run.smithi107.stdout:(24/36): python3-simplegeneric-0.8.1-17.el8.noa 2.3 MB/s | 19 kB 00:00 2020-06-12T02:35:14.544 INFO:teuthology.orchestra.run.smithi107.stdout:(25/36): python3-singledispatch-3.4.0.3-18.el8. 2.8 MB/s | 24 kB 00:00 2020-06-12T02:35:14.548 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:00:49 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:35:14.567 INFO:teuthology.orchestra.run.smithi107.stdout:(26/36): python3-tempora-1.14.1-5.el8.noarch.rp 3.1 MB/s | 30 kB 00:00 2020-06-12T02:35:14.567 INFO:teuthology.orchestra.run.smithi107.stdout:(27/36): python3-trustme-0.5.2-4.el8.noarch.rpm 2.9 MB/s | 27 kB 00:00 2020-06-12T02:35:14.575 INFO:teuthology.orchestra.run.smithi107.stdout:(28/36): python3-zc-lockfile-2.0-2.el8.noarch.r 2.5 MB/s | 23 kB 00:00 2020-06-12T02:35:14.647 INFO:teuthology.orchestra.run.smithi107.stdout:(29/36): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 584 kB/s | 44 kB 00:00 2020-06-12T02:35:14.708 INFO:teuthology.orchestra.run.smithi107.stdout:(30/36): python3-beautifulsoup4-4.6.3-2.el8.1.n 3.2 MB/s | 185 kB 00:00 2020-06-12T02:35:14.722 INFO:teuthology.orchestra.run.smithi107.stdout:(31/36): python3-cssselect-0.9.2-10.el8.noarch. 2.1 MB/s | 40 kB 00:00 2020-06-12T02:35:14.845 INFO:teuthology.orchestra.run.smithi107.stdout:(32/36): ceph-osd-15.2.3-196.g91d24f4.el8.x86_6 21 MB/s | 17 MB 00:00 2020-06-12T02:35:14.847 INFO:teuthology.orchestra.run.smithi107.stdout:(33/36): python3-waitress-1.2.1-2.el8.1.noarch. 1.9 MB/s | 241 kB 00:00 2020-06-12T02:35:14.867 INFO:teuthology.orchestra.run.smithi107.stdout:(34/36): python3-webtest-2.0.33-1.el8.noarch.rp 4.3 MB/s | 85 kB 00:00 2020-06-12T02:35:14.918 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2020-06-12T02:35:14.922 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:35:14.922 INFO:teuthology.orchestra.run.smithi188.stdout: Package ArchVersion Repository Size 2020-06-12T02:35:14.923 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:35:14.923 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2020-06-12T02:35:14.923 INFO:teuthology.orchestra.run.smithi188.stdout: ceph x86_642:15.2.3-196.g91d24f4.el8 ceph 6.4 k 2020-06-12T02:35:14.923 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2020-06-12T02:35:14.923 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mds x86_642:15.2.3-196.g91d24f4.el8 ceph 2.0 M 2020-06-12T02:35:14.923 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr x86_642:15.2.3-196.g91d24f4.el8 ceph 1.4 M 2020-06-12T02:35:14.924 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mon x86_642:15.2.3-196.g91d24f4.el8 ceph 4.4 M 2020-06-12T02:35:14.924 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-osd x86_642:15.2.3-196.g91d24f4.el8 ceph 17 M 2020-06-12T02:35:14.924 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core noarch2:15.2.3-196.g91d24f4.el8 ceph-noarch 165 k 2020-06-12T02:35:14.924 INFO:teuthology.orchestra.run.smithi188.stdout: python3-lxml x86_644.2.3-1.el8 CentOS-AppStream 1.5 M 2020-06-12T02:35:14.924 INFO:teuthology.orchestra.run.smithi188.stdout: python3-mako noarch1.0.6-13.el8 CentOS-AppStream 157 k 2020-06-12T02:35:14.925 INFO:teuthology.orchestra.run.smithi188.stdout: python3-markupsafe x86_640.23-19.el8 CentOS-AppStream 39 k 2020-06-12T02:35:14.926 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytz noarch2017.2-9.el8 CentOS-AppStream 54 k 2020-06-12T02:35:14.926 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webencodings noarch0.5.1-6.el8 CentOS-AppStream 27 k 2020-06-12T02:35:14.926 INFO:teuthology.orchestra.run.smithi188.stdout: python3-werkzeug noarch0.12.2-4.el8 CentOS-AppStream 457 k 2020-06-12T02:35:14.927 INFO:teuthology.orchestra.run.smithi188.stdout: python3-chardet noarch3.0.4-7.el8 CentOS-Base 195 k 2020-06-12T02:35:14.927 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pysocks noarch1.6.8-3.el8 CentOS-Base 34 k 2020-06-12T02:35:14.928 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests noarch2.20.0-2.1.el8_1 CentOS-Base 123 k 2020-06-12T02:35:14.928 INFO:teuthology.orchestra.run.smithi188.stdout: python3-urllib3 noarch1.24.2-2.el8 CentOS-Base 176 k 2020-06-12T02:35:14.928 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cheroot noarch8.2.1-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 145 k 2020-06-12T02:35:14.928 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cherrypy noarch18.4.0-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 385 k 2020-06-12T02:35:14.929 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco noarch6.2-6.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 11 k 2020-06-12T02:35:14.929 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-functools noarch2.0-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 18 k 2020-06-12T02:35:14.929 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils noarch0.3.5-11.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 49 k 2020-06-12T02:35:14.929 INFO:teuthology.orchestra.run.smithi188.stdout: python3-more-itertools noarch7.2.0-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 59 k 2020-06-12T02:35:14.930 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan noarch1.3.2-9.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 283 k 2020-06-12T02:35:14.930 INFO:teuthology.orchestra.run.smithi188.stdout: python3-portend noarch2.6-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 16 k 2020-06-12T02:35:14.930 INFO:teuthology.orchestra.run.smithi188.stdout: python3-simplegeneric noarch0.8.1-17.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 19 k 2020-06-12T02:35:14.930 INFO:teuthology.orchestra.run.smithi188.stdout: python3-singledispatch noarch3.4.0.3-18.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 24 k 2020-06-12T02:35:14.930 INFO:teuthology.orchestra.run.smithi188.stdout: python3-tempora noarch1.14.1-5.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 30 k 2020-06-12T02:35:14.931 INFO:teuthology.orchestra.run.smithi188.stdout: python3-trustme noarch0.5.2-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 27 k 2020-06-12T02:35:14.931 INFO:teuthology.orchestra.run.smithi188.stdout: python3-zc-lockfile noarch2.0-2.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 23 k 2020-06-12T02:35:14.931 INFO:teuthology.orchestra.run.smithi188.stdout: python3-bcrypt x86_643.1.6-2.el8.1 epel 44 k 2020-06-12T02:35:14.931 INFO:teuthology.orchestra.run.smithi188.stdout: python3-beautifulsoup4 noarch4.6.3-2.el8.1 epel 185 k 2020-06-12T02:35:14.931 INFO:teuthology.orchestra.run.smithi188.stdout: python3-waitress noarch1.2.1-2.el8.1 epel 241 k 2020-06-12T02:35:14.932 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webob noarch1.8.5-1.el8.1 epel 251 k 2020-06-12T02:35:14.932 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webtest noarch2.0.33-1.el8 epel 85 k 2020-06-12T02:35:14.932 INFO:teuthology.orchestra.run.smithi188.stdout:Installing weak dependencies: 2020-06-12T02:35:14.932 INFO:teuthology.orchestra.run.smithi188.stdout: python3-html5lib noarch1:0.999999999-6.el8 CentOS-AppStream 214 k 2020-06-12T02:35:14.932 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cssselect noarch0.9.2-10.el8 epel 40 k 2020-06-12T02:35:14.932 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:35:14.933 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2020-06-12T02:35:14.933 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:35:14.933 INFO:teuthology.orchestra.run.smithi188.stdout:Install 36 Packages 2020-06-12T02:35:14.933 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:35:14.934 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 29 M 2020-06-12T02:35:14.934 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 105 M 2020-06-12T02:35:14.934 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2020-06-12T02:35:14.937 INFO:teuthology.orchestra.run.smithi107.stdout:(35/36): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.7 MB/s | 251 kB 00:00 2020-06-12T02:35:15.027 INFO:teuthology.orchestra.run.smithi188.stdout:(1/36): ceph-15.2.3-196.g91d24f4.el8.x86_64.rpm 65 kB/s | 6.4 kB 00:00 2020-06-12T02:35:15.051 INFO:teuthology.orchestra.run.smithi107.stdout:(36/36): ceph-mon-15.2.3-196.g91d24f4.el8.x86_6 3.8 MB/s | 4.4 MB 00:01 2020-06-12T02:35:15.054 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2020-06-12T02:35:15.055 INFO:teuthology.orchestra.run.smithi107.stdout:Total 22 MB/s | 29 MB 00:01 2020-06-12T02:35:15.063 INFO:teuthology.orchestra.run.smithi107.stderr:warning: /var/cache/dnf/copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8-0161c520b1c9fbf7/packages/python3-cheroot-8.2.1-1.el8.noarch.rpm: Header V3 RSA/SHA1 Signature, key ID 2a8a41ec: NOKEY 2020-06-12T02:35:15.104 INFO:teuthology.orchestra.run.smithi107.stdout:Copr repo for ceph-el8 owned by ktdreyer 25 kB/s | 1.0 kB 00:00 2020-06-12T02:35:15.201 INFO:teuthology.orchestra.run.smithi107.stderr:Importing GPG key 0x2A8A41EC: 2020-06-12T02:35:15.202 INFO:teuthology.orchestra.run.smithi107.stderr: Userid : "ktdreyer_ceph-el8 (None) " 2020-06-12T02:35:15.202 INFO:teuthology.orchestra.run.smithi107.stderr: Fingerprint: 64D3 346F FD63 5D2D B5D0 5873 1014 BDBE 2A8A 41EC 2020-06-12T02:35:15.202 INFO:teuthology.orchestra.run.smithi107.stderr: From : https://download.copr.fedorainfracloud.org/results/ktdreyer/ceph-el8/pubkey.gpg 2020-06-12T02:35:15.269 INFO:teuthology.orchestra.run.smithi188.stdout:(2/36): ceph-mds-15.2.3-196.g91d24f4.el8.x86_64 6.0 MB/s | 2.0 MB 00:00 2020-06-12T02:35:15.273 INFO:teuthology.orchestra.run.smithi188.stdout:(3/36): ceph-mgr-15.2.3-196.g91d24f4.el8.x86_64 4.1 MB/s | 1.4 MB 00:00 2020-06-12T02:35:15.298 INFO:teuthology.orchestra.run.smithi107.stdout:Key imported successfully 2020-06-12T02:35:15.311 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2020-06-12T02:35:15.337 INFO:teuthology.orchestra.run.smithi188.stdout:(4/36): ceph-mgr-modules-core-15.2.3-196.g91d24 2.6 MB/s | 165 kB 00:00 2020-06-12T02:35:15.354 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2020-06-12T02:35:15.354 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2020-06-12T02:35:15.369 INFO:teuthology.orchestra.run.smithi188.stdout:(5/36): python3-html5lib-0.999999999-6.el8.noar 6.4 MB/s | 214 kB 00:00 2020-06-12T02:35:15.389 INFO:teuthology.orchestra.run.smithi188.stdout:(6/36): python3-lxml-4.2.3-1.el8.x86_64.rpm 77 MB/s | 1.5 MB 00:00 2020-06-12T02:35:15.391 INFO:teuthology.orchestra.run.smithi188.stdout:(7/36): python3-mako-1.0.6-13.el8.noarch.rpm 60 MB/s | 157 kB 00:00 2020-06-12T02:35:15.393 INFO:teuthology.orchestra.run.smithi188.stdout:(8/36): python3-markupsafe-0.23-19.el8.x86_64.r 33 MB/s | 39 kB 00:00 2020-06-12T02:35:15.394 INFO:teuthology.orchestra.run.smithi188.stdout:(9/36): python3-pytz-2017.2-9.el8.noarch.rpm 36 MB/s | 54 kB 00:00 2020-06-12T02:35:15.396 INFO:teuthology.orchestra.run.smithi188.stdout:(10/36): python3-webencodings-0.5.1-6.el8.noarc 26 MB/s | 27 kB 00:00 2020-06-12T02:35:15.402 INFO:teuthology.orchestra.run.smithi188.stdout:(11/36): python3-werkzeug-0.12.2-4.el8.noarch.r 76 MB/s | 457 kB 00:00 2020-06-12T02:35:15.408 INFO:teuthology.orchestra.run.smithi188.stdout:(12/36): python3-chardet-3.0.4-7.el8.noarch.rpm 33 MB/s | 195 kB 00:00 2020-06-12T02:35:15.409 INFO:teuthology.orchestra.run.smithi188.stdout:(13/36): python3-pysocks-1.6.8-3.el8.noarch.rpm 30 MB/s | 34 kB 00:00 2020-06-12T02:35:15.411 INFO:teuthology.orchestra.run.smithi188.stdout:(14/36): python3-requests-2.20.0-2.1.el8_1.noar 56 MB/s | 123 kB 00:00 2020-06-12T02:35:15.414 INFO:teuthology.orchestra.run.smithi188.stdout:(15/36): python3-urllib3-1.24.2-2.el8.noarch.rp 64 MB/s | 176 kB 00:00 2020-06-12T02:35:15.475 INFO:teuthology.orchestra.run.smithi188.stdout:(16/36): ceph-mon-15.2.3-196.g91d24f4.el8.x86_6 9.7 MB/s | 4.4 MB 00:00 2020-06-12T02:35:15.476 INFO:teuthology.orchestra.run.smithi188.stdout:(17/36): python3-cheroot-8.2.1-1.el8.noarch.rpm 2.3 MB/s | 145 kB 00:00 2020-06-12T02:35:15.488 INFO:teuthology.orchestra.run.smithi188.stdout:(18/36): python3-jaraco-6.2-6.el8.noarch.rpm 1.0 MB/s | 11 kB 00:00 2020-06-12T02:35:15.498 INFO:teuthology.orchestra.run.smithi188.stdout:(19/36): python3-jaraco-functools-2.0-4.el8.noa 1.7 MB/s | 18 kB 00:00 2020-06-12T02:35:15.509 INFO:teuthology.orchestra.run.smithi188.stdout:(20/36): python3-logutils-0.3.5-11.el8.noarch.r 4.8 MB/s | 49 kB 00:00 2020-06-12T02:35:15.518 INFO:teuthology.orchestra.run.smithi188.stdout:(21/36): python3-more-itertools-7.2.0-4.el8.noa 6.4 MB/s | 59 kB 00:00 2020-06-12T02:35:15.536 INFO:teuthology.orchestra.run.smithi188.stdout:(22/36): python3-pecan-1.3.2-9.el8.noarch.rpm 15 MB/s | 283 kB 00:00 2020-06-12T02:35:15.544 INFO:teuthology.orchestra.run.smithi188.stdout:(23/36): python3-cherrypy-18.4.0-1.el8.noarch.r 5.5 MB/s | 385 kB 00:00 2020-06-12T02:35:15.545 INFO:teuthology.orchestra.run.smithi188.stdout:(24/36): python3-portend-2.6-1.el8.noarch.rpm 1.8 MB/s | 16 kB 00:00 2020-06-12T02:35:15.553 INFO:teuthology.orchestra.run.smithi188.stdout:(25/36): python3-simplegeneric-0.8.1-17.el8.noa 2.0 MB/s | 19 kB 00:00 2020-06-12T02:35:15.560 INFO:teuthology.orchestra.run.smithi188.stdout:(26/36): python3-singledispatch-3.4.0.3-18.el8. 1.6 MB/s | 24 kB 00:00 2020-06-12T02:35:15.562 INFO:teuthology.orchestra.run.smithi188.stdout:(27/36): python3-tempora-1.14.1-5.el8.noarch.rp 3.2 MB/s | 30 kB 00:00 2020-06-12T02:35:15.576 INFO:teuthology.orchestra.run.smithi188.stdout:(28/36): python3-trustme-0.5.2-4.el8.noarch.rpm 1.7 MB/s | 27 kB 00:00 2020-06-12T02:35:15.603 INFO:teuthology.orchestra.run.smithi188.stdout:(29/36): python3-zc-lockfile-2.0-2.el8.noarch.r 572 kB/s | 23 kB 00:00 2020-06-12T02:35:15.637 INFO:teuthology.orchestra.run.smithi188.stdout:(30/36): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 709 kB/s | 44 kB 00:00 2020-06-12T02:35:15.668 INFO:teuthology.orchestra.run.smithi188.stdout:(31/36): python3-cssselect-0.9.2-10.el8.noarch. 1.3 MB/s | 40 kB 00:00 2020-06-12T02:35:15.680 INFO:teuthology.orchestra.run.smithi188.stdout:(32/36): python3-beautifulsoup4-4.6.3-2.el8.1.n 2.4 MB/s | 185 kB 00:00 2020-06-12T02:35:15.692 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2020-06-12T02:35:15.692 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2020-06-12T02:35:15.717 INFO:teuthology.orchestra.run.smithi188.stdout:(33/36): python3-webob-1.8.5-1.el8.1.noarch.rpm 7.7 MB/s | 251 kB 00:00 2020-06-12T02:35:15.717 INFO:teuthology.orchestra.run.smithi188.stdout:(34/36): python3-waitress-1.2.1-2.el8.1.noarch. 5.1 MB/s | 241 kB 00:00 2020-06-12T02:35:15.733 INFO:teuthology.orchestra.run.smithi188.stdout:(35/36): python3-webtest-2.0.33-1.el8.noarch.rp 5.2 MB/s | 85 kB 00:00 2020-06-12T02:35:16.029 INFO:teuthology.orchestra.run.smithi188.stdout:(36/36): ceph-osd-15.2.3-196.g91d24f4.el8.x86_6 22 MB/s | 17 MB 00:00 2020-06-12T02:35:16.032 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2020-06-12T02:35:16.033 INFO:teuthology.orchestra.run.smithi188.stdout:Total 27 MB/s | 29 MB 00:01 2020-06-12T02:35:16.042 INFO:teuthology.orchestra.run.smithi188.stderr:warning: /var/cache/dnf/copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8-0161c520b1c9fbf7/packages/python3-cheroot-8.2.1-1.el8.noarch.rpm: Header V3 RSA/SHA1 Signature, key ID 2a8a41ec: NOKEY 2020-06-12T02:35:16.082 INFO:teuthology.orchestra.run.smithi188.stdout:Copr repo for ceph-el8 owned by ktdreyer 25 kB/s | 1.0 kB 00:00 2020-06-12T02:35:16.186 INFO:teuthology.orchestra.run.smithi188.stderr:Importing GPG key 0x2A8A41EC: 2020-06-12T02:35:16.186 INFO:teuthology.orchestra.run.smithi188.stderr: Userid : "ktdreyer_ceph-el8 (None) " 2020-06-12T02:35:16.187 INFO:teuthology.orchestra.run.smithi188.stderr: Fingerprint: 64D3 346F FD63 5D2D B5D0 5873 1014 BDBE 2A8A 41EC 2020-06-12T02:35:16.187 INFO:teuthology.orchestra.run.smithi188.stderr: From : https://download.copr.fedorainfracloud.org/results/ktdreyer/ceph-el8/pubkey.gpg 2020-06-12T02:35:16.281 INFO:teuthology.orchestra.run.smithi188.stdout:Key imported successfully 2020-06-12T02:35:16.307 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2020-06-12T02:35:16.345 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2020-06-12T02:35:16.346 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2020-06-12T02:35:16.656 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2020-06-12T02:35:16.657 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2020-06-12T02:35:17.004 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2020-06-12T02:35:17.234 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 1/36 2020-06-12T02:35:17.469 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-more-itertools-7.2.0-4.el8.noarch 2/36 2020-06-12T02:35:17.650 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 3/36 2020-06-12T02:35:17.824 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 4/36 2020-06-12T02:35:17.995 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2020-06-12T02:35:18.014 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 5/36 2020-06-12T02:35:18.190 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 6/36 2020-06-12T02:35:18.216 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 1/36 2020-06-12T02:35:18.347 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-trustme-0.5.2-4.el8.noarch 7/36 2020-06-12T02:35:18.437 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-more-itertools-7.2.0-4.el8.noarch 2/36 2020-06-12T02:35:18.511 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 8/36 2020-06-12T02:35:18.641 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 3/36 2020-06-12T02:35:18.708 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 9/36 2020-06-12T02:35:18.815 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 4/36 2020-06-12T02:35:18.859 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 10/36 2020-06-12T02:35:19.004 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 11/36 2020-06-12T02:35:19.038 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 5/36 2020-06-12T02:35:19.193 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 6/36 2020-06-12T02:35:19.194 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 12/36 2020-06-12T02:35:19.354 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-trustme-0.5.2-4.el8.noarch 7/36 2020-06-12T02:35:19.422 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cheroot-8.2.1-1.el8.noarch 13/36 2020-06-12T02:35:19.493 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 8/36 2020-06-12T02:35:19.628 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 14/36 2020-06-12T02:35:19.657 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 9/36 2020-06-12T02:35:19.808 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 10/36 2020-06-12T02:35:19.848 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-urllib3-1.24.2-2.el8.noarch 15/36 2020-06-12T02:35:19.954 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 11/36 2020-06-12T02:35:20.045 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 16/36 2020-06-12T02:35:20.178 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 12/36 2020-06-12T02:35:20.358 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 17/36 2020-06-12T02:35:20.373 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cheroot-8.2.1-1.el8.noarch 13/36 2020-06-12T02:35:20.567 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 18/36 2020-06-12T02:35:20.576 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 14/36 2020-06-12T02:35:20.797 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-urllib3-1.24.2-2.el8.noarch 15/36 2020-06-12T02:35:20.816 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 19/36 2020-06-12T02:35:20.992 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 20/36 2020-06-12T02:35:20.993 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 16/36 2020-06-12T02:35:21.299 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 17/36 2020-06-12T02:35:21.354 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 21/36 2020-06-12T02:35:21.491 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 18/36 2020-06-12T02:35:21.619 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-lxml-4.2.3-1.el8.x86_64 22/36 2020-06-12T02:35:21.725 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 19/36 2020-06-12T02:35:21.829 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 23/36 2020-06-12T02:35:21.917 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 20/36 2020-06-12T02:35:21.975 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/36 2020-06-12T02:35:22.127 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 25/36 2020-06-12T02:35:22.305 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 21/36 2020-06-12T02:35:22.348 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-portend-2.6-1.el8.noarch 26/36 2020-06-12T02:35:22.549 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-lxml-4.2.3-1.el8.x86_64 22/36 2020-06-12T02:35:22.568 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 27/36 2020-06-12T02:35:22.752 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 23/36 2020-06-12T02:35:22.809 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/36 2020-06-12T02:35:22.923 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/36 2020-06-12T02:35:23.109 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 25/36 2020-06-12T02:35:23.166 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 29/36 2020-06-12T02:35:23.355 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-portend-2.6-1.el8.noarch 26/36 2020-06-12T02:35:23.496 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 30/36 2020-06-12T02:35:23.566 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 27/36 2020-06-12T02:35:23.772 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/36 2020-06-12T02:35:23.885 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-modules-core-2:15.2.3-196.g91d24f4.el8.no 31/36 2020-06-12T02:35:24.056 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-2:15.2.3-196.g91d24f4.el8.x86_64 32/36 2020-06-12T02:35:24.137 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 29/36 2020-06-12T02:35:24.426 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 30/36 2020-06-12T02:35:24.786 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-modules-core-2:15.2.3-196.g91d24f4.el8.no 31/36 2020-06-12T02:35:24.945 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-2:15.2.3-196.g91d24f4.el8.x86_64 32/36 2020-06-12T02:35:25.463 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-2:15.2.3-196.g91d24f4.el8.x86_64 32/36 2020-06-12T02:35:25.706 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-osd-2:15.2.3-196.g91d24f4.el8.x86_64 33/36 2020-06-12T02:35:26.085 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-osd-2:15.2.3-196.g91d24f4.el8.x86_64 33/36 2020-06-12T02:35:26.246 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mon-2:15.2.3-196.g91d24f4.el8.x86_64 34/36 2020-06-12T02:35:26.356 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-2:15.2.3-196.g91d24f4.el8.x86_64 32/36 2020-06-12T02:35:26.441 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mon-2:15.2.3-196.g91d24f4.el8.x86_64 34/36 2020-06-12T02:35:26.563 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-osd-2:15.2.3-196.g91d24f4.el8.x86_64 33/36 2020-06-12T02:35:26.596 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mds-2:15.2.3-196.g91d24f4.el8.x86_64 35/36 2020-06-12T02:35:26.615 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mds-2:15.2.3-196.g91d24f4.el8.x86_64 35/36 2020-06-12T02:35:26.719 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-2:15.2.3-196.g91d24f4.el8.x86_64 36/36 2020-06-12T02:35:26.947 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-osd-2:15.2.3-196.g91d24f4.el8.x86_64 33/36 2020-06-12T02:35:27.028 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-2:15.2.3-196.g91d24f4.el8.x86_64 36/36 2020-06-12T02:35:27.029 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-2:15.2.3-196.g91d24f4.el8.x86_64 1/36 2020-06-12T02:35:27.029 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mds-2:15.2.3-196.g91d24f4.el8.x86_64 2/36 2020-06-12T02:35:27.029 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-2:15.2.3-196.g91d24f4.el8.x86_64 3/36 2020-06-12T02:35:27.029 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mon-2:15.2.3-196.g91d24f4.el8.x86_64 4/36 2020-06-12T02:35:27.029 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-osd-2:15.2.3-196.g91d24f4.el8.x86_64 5/36 2020-06-12T02:35:27.030 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-modules-core-2:15.2.3-196.g91d24f4.el8.no 6/36 2020-06-12T02:35:27.030 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 7/36 2020-06-12T02:35:27.030 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-lxml-4.2.3-1.el8.x86_64 8/36 2020-06-12T02:35:27.030 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 9/36 2020-06-12T02:35:27.031 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 10/36 2020-06-12T02:35:27.031 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 11/36 2020-06-12T02:35:27.031 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 12/36 2020-06-12T02:35:27.032 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 13/36 2020-06-12T02:35:27.032 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 14/36 2020-06-12T02:35:27.032 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 15/36 2020-06-12T02:35:27.032 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 16/36 2020-06-12T02:35:27.032 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-urllib3-1.24.2-2.el8.noarch 17/36 2020-06-12T02:35:27.033 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cheroot-8.2.1-1.el8.noarch 18/36 2020-06-12T02:35:27.033 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 19/36 2020-06-12T02:35:27.033 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 20/36 2020-06-12T02:35:27.033 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 21/36 2020-06-12T02:35:27.033 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 22/36 2020-06-12T02:35:27.034 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-more-itertools-7.2.0-4.el8.noarch 23/36 2020-06-12T02:35:27.034 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 24/36 2020-06-12T02:35:27.034 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-portend-2.6-1.el8.noarch 25/36 2020-06-12T02:35:27.034 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 26/36 2020-06-12T02:35:27.034 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 27/36 2020-06-12T02:35:27.034 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 28/36 2020-06-12T02:35:27.035 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-trustme-0.5.2-4.el8.noarch 29/36 2020-06-12T02:35:27.036 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 30/36 2020-06-12T02:35:27.036 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 31/36 2020-06-12T02:35:27.036 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 32/36 2020-06-12T02:35:27.037 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 33/36 2020-06-12T02:35:27.037 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 34/36 2020-06-12T02:35:27.037 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 35/36 2020-06-12T02:35:27.126 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mon-2:15.2.3-196.g91d24f4.el8.x86_64 34/36 2020-06-12T02:35:27.318 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mon-2:15.2.3-196.g91d24f4.el8.x86_64 34/36 2020-06-12T02:35:27.462 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mds-2:15.2.3-196.g91d24f4.el8.x86_64 35/36 2020-06-12T02:35:27.481 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mds-2:15.2.3-196.g91d24f4.el8.x86_64 35/36 2020-06-12T02:35:27.568 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-2:15.2.3-196.g91d24f4.el8.x86_64 36/36 2020-06-12T02:35:27.866 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 36/36 2020-06-12T02:35:27.867 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:35:27.867 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2020-06-12T02:35:27.867 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:27.867 INFO:teuthology.orchestra.run.smithi107.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2020-06-12T02:35:27.868 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cssselect-0.9.2-10.el8.noarch 2020-06-12T02:35:27.868 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:27.868 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:27.868 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:27.868 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:27.869 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-modules-core-2:15.2.3-196.g91d24f4.el8.noarch 2020-06-12T02:35:27.869 INFO:teuthology.orchestra.run.smithi107.stdout: python3-lxml-4.2.3-1.el8.x86_64 2020-06-12T02:35:27.869 INFO:teuthology.orchestra.run.smithi107.stdout: python3-mako-1.0.6-13.el8.noarch 2020-06-12T02:35:27.869 INFO:teuthology.orchestra.run.smithi107.stdout: python3-markupsafe-0.23-19.el8.x86_64 2020-06-12T02:35:27.869 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytz-2017.2-9.el8.noarch 2020-06-12T02:35:27.870 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webencodings-0.5.1-6.el8.noarch 2020-06-12T02:35:27.870 INFO:teuthology.orchestra.run.smithi107.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2020-06-12T02:35:27.870 INFO:teuthology.orchestra.run.smithi107.stdout: python3-chardet-3.0.4-7.el8.noarch 2020-06-12T02:35:27.870 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pysocks-1.6.8-3.el8.noarch 2020-06-12T02:35:27.870 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2020-06-12T02:35:27.871 INFO:teuthology.orchestra.run.smithi107.stdout: python3-urllib3-1.24.2-2.el8.noarch 2020-06-12T02:35:27.871 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cheroot-8.2.1-1.el8.noarch 2020-06-12T02:35:27.871 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2020-06-12T02:35:27.871 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-6.2-6.el8.noarch 2020-06-12T02:35:27.871 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2020-06-12T02:35:27.871 INFO:teuthology.orchestra.run.smithi107.stdout: python3-logutils-0.3.5-11.el8.noarch 2020-06-12T02:35:27.872 INFO:teuthology.orchestra.run.smithi107.stdout: python3-more-itertools-7.2.0-4.el8.noarch 2020-06-12T02:35:27.872 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pecan-1.3.2-9.el8.noarch 2020-06-12T02:35:27.872 INFO:teuthology.orchestra.run.smithi107.stdout: python3-portend-2.6-1.el8.noarch 2020-06-12T02:35:27.872 INFO:teuthology.orchestra.run.smithi107.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2020-06-12T02:35:27.872 INFO:teuthology.orchestra.run.smithi107.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2020-06-12T02:35:27.873 INFO:teuthology.orchestra.run.smithi107.stdout: python3-tempora-1.14.1-5.el8.noarch 2020-06-12T02:35:27.873 INFO:teuthology.orchestra.run.smithi107.stdout: python3-trustme-0.5.2-4.el8.noarch 2020-06-12T02:35:27.873 INFO:teuthology.orchestra.run.smithi107.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2020-06-12T02:35:27.873 INFO:teuthology.orchestra.run.smithi107.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2020-06-12T02:35:27.873 INFO:teuthology.orchestra.run.smithi107.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2020-06-12T02:35:27.873 INFO:teuthology.orchestra.run.smithi107.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2020-06-12T02:35:27.875 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webob-1.8.5-1.el8.1.noarch 2020-06-12T02:35:27.875 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webtest-2.0.33-1.el8.noarch 2020-06-12T02:35:27.875 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:35:27.876 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2020-06-12T02:35:27.882 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-2:15.2.3-196.g91d24f4.el8.x86_64 36/36 2020-06-12T02:35:27.882 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-2:15.2.3-196.g91d24f4.el8.x86_64 1/36 2020-06-12T02:35:27.882 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mds-2:15.2.3-196.g91d24f4.el8.x86_64 2/36 2020-06-12T02:35:27.883 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-2:15.2.3-196.g91d24f4.el8.x86_64 3/36 2020-06-12T02:35:27.883 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mon-2:15.2.3-196.g91d24f4.el8.x86_64 4/36 2020-06-12T02:35:27.883 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-osd-2:15.2.3-196.g91d24f4.el8.x86_64 5/36 2020-06-12T02:35:27.883 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-modules-core-2:15.2.3-196.g91d24f4.el8.no 6/36 2020-06-12T02:35:27.883 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 7/36 2020-06-12T02:35:27.884 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-lxml-4.2.3-1.el8.x86_64 8/36 2020-06-12T02:35:27.884 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 9/36 2020-06-12T02:35:27.884 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 10/36 2020-06-12T02:35:27.884 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 11/36 2020-06-12T02:35:27.884 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 12/36 2020-06-12T02:35:27.885 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 13/36 2020-06-12T02:35:27.885 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 14/36 2020-06-12T02:35:27.885 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 15/36 2020-06-12T02:35:27.887 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 16/36 2020-06-12T02:35:27.887 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-urllib3-1.24.2-2.el8.noarch 17/36 2020-06-12T02:35:27.887 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cheroot-8.2.1-1.el8.noarch 18/36 2020-06-12T02:35:27.887 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 19/36 2020-06-12T02:35:27.888 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 20/36 2020-06-12T02:35:27.888 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 21/36 2020-06-12T02:35:27.888 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 22/36 2020-06-12T02:35:27.888 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-more-itertools-7.2.0-4.el8.noarch 23/36 2020-06-12T02:35:27.888 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 24/36 2020-06-12T02:35:27.889 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-portend-2.6-1.el8.noarch 25/36 2020-06-12T02:35:27.890 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 26/36 2020-06-12T02:35:27.892 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 27/36 2020-06-12T02:35:27.892 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 28/36 2020-06-12T02:35:27.893 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-trustme-0.5.2-4.el8.noarch 29/36 2020-06-12T02:35:27.893 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 30/36 2020-06-12T02:35:27.894 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 31/36 2020-06-12T02:35:27.894 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 32/36 2020-06-12T02:35:27.894 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 33/36 2020-06-12T02:35:27.894 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 34/36 2020-06-12T02:35:27.894 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 35/36 2020-06-12T02:35:28.159 INFO:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-base 2020-06-12T02:35:28.567 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:03 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:35:28.817 INFO:teuthology.orchestra.run.smithi107.stdout:Package ceph-base-2:15.2.3-196.g91d24f4.el8.x86_64 is already installed. 2020-06-12T02:35:28.922 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2020-06-12T02:35:28.923 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2020-06-12T02:35:28.923 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2020-06-12T02:35:29.029 INFO:teuthology.orchestra.run.smithi107:> sudo yum -y install cephadm 2020-06-12T02:35:29.088 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 36/36 2020-06-12T02:35:29.089 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:35:29.089 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2020-06-12T02:35:29.089 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:29.089 INFO:teuthology.orchestra.run.smithi188.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2020-06-12T02:35:29.089 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cssselect-0.9.2-10.el8.noarch 2020-06-12T02:35:29.090 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mds-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:29.090 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:29.090 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mon-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:29.090 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-osd-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:29.090 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core-2:15.2.3-196.g91d24f4.el8.noarch 2020-06-12T02:35:29.091 INFO:teuthology.orchestra.run.smithi188.stdout: python3-lxml-4.2.3-1.el8.x86_64 2020-06-12T02:35:29.091 INFO:teuthology.orchestra.run.smithi188.stdout: python3-mako-1.0.6-13.el8.noarch 2020-06-12T02:35:29.091 INFO:teuthology.orchestra.run.smithi188.stdout: python3-markupsafe-0.23-19.el8.x86_64 2020-06-12T02:35:29.091 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytz-2017.2-9.el8.noarch 2020-06-12T02:35:29.091 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webencodings-0.5.1-6.el8.noarch 2020-06-12T02:35:29.092 INFO:teuthology.orchestra.run.smithi188.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2020-06-12T02:35:29.092 INFO:teuthology.orchestra.run.smithi188.stdout: python3-chardet-3.0.4-7.el8.noarch 2020-06-12T02:35:29.092 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pysocks-1.6.8-3.el8.noarch 2020-06-12T02:35:29.092 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2020-06-12T02:35:29.092 INFO:teuthology.orchestra.run.smithi188.stdout: python3-urllib3-1.24.2-2.el8.noarch 2020-06-12T02:35:29.093 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cheroot-8.2.1-1.el8.noarch 2020-06-12T02:35:29.093 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2020-06-12T02:35:29.093 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-6.2-6.el8.noarch 2020-06-12T02:35:29.093 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2020-06-12T02:35:29.093 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils-0.3.5-11.el8.noarch 2020-06-12T02:35:29.094 INFO:teuthology.orchestra.run.smithi188.stdout: python3-more-itertools-7.2.0-4.el8.noarch 2020-06-12T02:35:29.094 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan-1.3.2-9.el8.noarch 2020-06-12T02:35:29.094 INFO:teuthology.orchestra.run.smithi188.stdout: python3-portend-2.6-1.el8.noarch 2020-06-12T02:35:29.094 INFO:teuthology.orchestra.run.smithi188.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2020-06-12T02:35:29.094 INFO:teuthology.orchestra.run.smithi188.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2020-06-12T02:35:29.095 INFO:teuthology.orchestra.run.smithi188.stdout: python3-tempora-1.14.1-5.el8.noarch 2020-06-12T02:35:29.095 INFO:teuthology.orchestra.run.smithi188.stdout: python3-trustme-0.5.2-4.el8.noarch 2020-06-12T02:35:29.095 INFO:teuthology.orchestra.run.smithi188.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2020-06-12T02:35:29.095 INFO:teuthology.orchestra.run.smithi188.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2020-06-12T02:35:29.095 INFO:teuthology.orchestra.run.smithi188.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2020-06-12T02:35:29.096 INFO:teuthology.orchestra.run.smithi188.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2020-06-12T02:35:29.096 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webob-1.8.5-1.el8.1.noarch 2020-06-12T02:35:29.096 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webtest-2.0.33-1.el8.noarch 2020-06-12T02:35:29.096 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:35:29.096 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2020-06-12T02:35:29.395 INFO:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-base 2020-06-12T02:35:29.477 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:04 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:35:29.779 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2020-06-12T02:35:29.780 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:35:29.780 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2020-06-12T02:35:29.780 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:35:29.780 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2020-06-12T02:35:29.781 INFO:teuthology.orchestra.run.smithi107.stdout: cephadm x86_64 2:15.2.3-196.g91d24f4.el8 ceph 46 k 2020-06-12T02:35:29.781 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:35:29.781 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2020-06-12T02:35:29.781 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:35:29.781 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2020-06-12T02:35:29.782 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:35:29.782 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 46 k 2020-06-12T02:35:29.782 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 168 k 2020-06-12T02:35:29.782 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2020-06-12T02:35:29.889 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:04 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:35:29.936 INFO:teuthology.orchestra.run.smithi107.stdout:cephadm-15.2.3-196.g91d24f4.el8.x86_64.rpm 297 kB/s | 46 kB 00:00 2020-06-12T02:35:29.936 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2020-06-12T02:35:29.937 INFO:teuthology.orchestra.run.smithi107.stdout:Total 295 kB/s | 46 kB 00:00 2020-06-12T02:35:29.938 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2020-06-12T02:35:29.945 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2020-06-12T02:35:29.945 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2020-06-12T02:35:29.947 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2020-06-12T02:35:29.947 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2020-06-12T02:35:30.138 INFO:teuthology.orchestra.run.smithi188.stdout:Package ceph-base-2:15.2.3-196.g91d24f4.el8.x86_64 is already installed. 2020-06-12T02:35:30.242 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2020-06-12T02:35:30.247 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2020-06-12T02:35:30.248 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2020-06-12T02:35:30.290 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2020-06-12T02:35:30.337 INFO:teuthology.orchestra.run.smithi188:> sudo yum -y install cephadm 2020-06-12T02:35:30.738 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:05 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:35:31.083 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2020-06-12T02:35:31.084 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:35:31.085 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2020-06-12T02:35:31.085 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:35:31.085 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2020-06-12T02:35:31.085 INFO:teuthology.orchestra.run.smithi188.stdout: cephadm x86_64 2:15.2.3-196.g91d24f4.el8 ceph 46 k 2020-06-12T02:35:31.085 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:35:31.086 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2020-06-12T02:35:31.086 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:35:31.086 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2020-06-12T02:35:31.086 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:35:31.086 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 46 k 2020-06-12T02:35:31.087 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 168 k 2020-06-12T02:35:31.087 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2020-06-12T02:35:31.231 INFO:teuthology.orchestra.run.smithi188.stdout:cephadm-15.2.3-196.g91d24f4.el8.x86_64.rpm 315 kB/s | 46 kB 00:00 2020-06-12T02:35:31.232 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2020-06-12T02:35:31.232 INFO:teuthology.orchestra.run.smithi188.stdout:Total 313 kB/s | 46 kB 00:00 2020-06-12T02:35:31.234 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2020-06-12T02:35:31.241 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2020-06-12T02:35:31.241 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2020-06-12T02:35:31.243 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2020-06-12T02:35:31.243 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2020-06-12T02:35:31.307 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2020-06-12T02:35:31.868 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: cephadm-2:15.2.3-196.g91d24f4.el8.x86_64 1/1 2020-06-12T02:35:32.016 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : cephadm-2:15.2.3-196.g91d24f4.el8.x86_64 1/1 2020-06-12T02:35:32.158 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: cephadm-2:15.2.3-196.g91d24f4.el8.x86_64 1/1 2020-06-12T02:35:32.373 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : cephadm-2:15.2.3-196.g91d24f4.el8.x86_64 1/1 2020-06-12T02:35:32.374 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:35:32.374 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2020-06-12T02:35:32.374 INFO:teuthology.orchestra.run.smithi107.stdout: cephadm-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:32.374 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:35:32.375 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2020-06-12T02:35:32.599 INFO:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-immutable-object-cache 2020-06-12T02:35:32.716 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: cephadm-2:15.2.3-196.g91d24f4.el8.x86_64 1/1 2020-06-12T02:35:32.889 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : cephadm-2:15.2.3-196.g91d24f4.el8.x86_64 1/1 2020-06-12T02:35:33.011 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:08 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:35:33.030 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: cephadm-2:15.2.3-196.g91d24f4.el8.x86_64 1/1 2020-06-12T02:35:33.277 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : cephadm-2:15.2.3-196.g91d24f4.el8.x86_64 1/1 2020-06-12T02:35:33.277 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:35:33.277 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2020-06-12T02:35:33.278 INFO:teuthology.orchestra.run.smithi188.stdout: cephadm-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:33.278 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:35:33.278 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2020-06-12T02:35:33.390 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2020-06-12T02:35:33.392 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:35:33.393 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2020-06-12T02:35:33.393 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:35:33.394 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2020-06-12T02:35:33.394 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache x86_64 2:15.2.3-196.g91d24f4.el8 ceph 203 k 2020-06-12T02:35:33.394 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:35:33.395 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2020-06-12T02:35:33.395 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:35:33.395 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2020-06-12T02:35:33.396 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:35:33.397 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 203 k 2020-06-12T02:35:33.397 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 743 k 2020-06-12T02:35:33.399 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2020-06-12T02:35:33.541 INFO:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-immutable-object-cache 2020-06-12T02:35:33.583 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-immutable-object-cache-15.2.3-196.g91d24f4 1.0 MB/s | 203 kB 00:00 2020-06-12T02:35:33.584 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2020-06-12T02:35:33.584 INFO:teuthology.orchestra.run.smithi107.stdout:Total 1.0 MB/s | 203 kB 00:00 2020-06-12T02:35:33.586 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2020-06-12T02:35:33.598 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2020-06-12T02:35:33.598 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2020-06-12T02:35:33.644 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2020-06-12T02:35:33.645 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2020-06-12T02:35:33.772 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2020-06-12T02:35:33.952 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:08 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:35:33.953 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-immutable-object-cache-2:15.2.3-196.g91d24f4.el 1/1 2020-06-12T02:35:34.240 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-immutable-object-cache-2:15.2.3-196.g91d24f4.el 1/1 2020-06-12T02:35:34.381 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2020-06-12T02:35:34.382 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:35:34.382 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repo Size 2020-06-12T02:35:34.383 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:35:34.383 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2020-06-12T02:35:34.383 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-immutable-object-cache x86_64 2:15.2.3-196.g91d24f4.el8 ceph 203 k 2020-06-12T02:35:34.383 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:35:34.383 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2020-06-12T02:35:34.384 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:35:34.384 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2020-06-12T02:35:34.384 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:35:34.384 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 203 k 2020-06-12T02:35:34.384 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 743 k 2020-06-12T02:35:34.385 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2020-06-12T02:35:34.463 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-immutable-object-cache-2:15.2.3-196.g91d24f4.el 1/1 2020-06-12T02:35:34.463 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:35:34.463 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2020-06-12T02:35:34.463 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:34.464 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:35:34.464 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2020-06-12T02:35:34.575 INFO:teuthology.orchestra.run.smithi188.stdout:ceph-immutable-object-cache-15.2.3-196.g91d24f4 1.0 MB/s | 203 kB 00:00 2020-06-12T02:35:34.576 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2020-06-12T02:35:34.576 INFO:teuthology.orchestra.run.smithi188.stdout:Total 1.0 MB/s | 203 kB 00:00 2020-06-12T02:35:34.578 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2020-06-12T02:35:34.595 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2020-06-12T02:35:34.596 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2020-06-12T02:35:34.637 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2020-06-12T02:35:34.638 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2020-06-12T02:35:34.717 INFO:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr 2020-06-12T02:35:34.753 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2020-06-12T02:35:34.927 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-immutable-object-cache-2:15.2.3-196.g91d24f4.el 1/1 2020-06-12T02:35:35.122 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:10 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:35:35.240 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-immutable-object-cache-2:15.2.3-196.g91d24f4.el 1/1 2020-06-12T02:35:35.382 INFO:teuthology.orchestra.run.smithi107.stdout:Package ceph-mgr-2:15.2.3-196.g91d24f4.el8.x86_64 is already installed. 2020-06-12T02:35:35.488 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2020-06-12T02:35:35.489 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2020-06-12T02:35:35.489 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2020-06-12T02:35:35.509 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-immutable-object-cache-2:15.2.3-196.g91d24f4.el 1/1 2020-06-12T02:35:35.510 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:35:35.510 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2020-06-12T02:35:35.510 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-immutable-object-cache-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:35:35.511 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:35:35.511 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2020-06-12T02:35:35.583 INFO:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-dashboard 2020-06-12T02:35:35.711 INFO:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-mgr 2020-06-12T02:35:35.977 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:10 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:35:36.117 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:11 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:35:36.347 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2020-06-12T02:35:36.348 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:35:36.348 INFO:teuthology.orchestra.run.smithi107.stdout: Package ArchVersion Repository Size 2020-06-12T02:35:36.348 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:35:36.348 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2020-06-12T02:35:36.348 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-dashboard noarch2:15.2.3-196.g91d24f4.el8 ceph-noarch 3.4 M 2020-06-12T02:35:36.349 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2020-06-12T02:35:36.349 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-grafana-dashboards noarch2:15.2.3-196.g91d24f4.el8 ceph-noarch 22 k 2020-06-12T02:35:36.349 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-prometheus-alerts noarch2:15.2.3-196.g91d24f4.el8 ceph-noarch 8.7 k 2020-06-12T02:35:36.349 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jwt noarch1.6.1-2.el8 CentOS-Base 43 k 2020-06-12T02:35:36.349 INFO:teuthology.orchestra.run.smithi107.stdout: python3-repoze-lru noarch0.7-6.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 34 k 2020-06-12T02:35:36.350 INFO:teuthology.orchestra.run.smithi107.stdout: python3-routes noarch2.4.1-12.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 197 k 2020-06-12T02:35:36.350 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:35:36.350 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2020-06-12T02:35:36.350 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:35:36.350 INFO:teuthology.orchestra.run.smithi107.stdout:Install 6 Packages 2020-06-12T02:35:36.351 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:35:36.351 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 3.7 M 2020-06-12T02:35:36.351 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 19 M 2020-06-12T02:35:36.352 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2020-06-12T02:35:36.370 INFO:teuthology.orchestra.run.smithi188.stdout:Package ceph-mgr-2:15.2.3-196.g91d24f4.el8.x86_64 is already installed. 2020-06-12T02:35:36.459 INFO:teuthology.orchestra.run.smithi107.stdout:(1/6): ceph-prometheus-alerts-15.2.3-196.g91d24 81 kB/s | 8.7 kB 00:00 2020-06-12T02:35:36.473 INFO:teuthology.orchestra.run.smithi107.stdout:(2/6): ceph-grafana-dashboards-15.2.3-196.g91d2 183 kB/s | 22 kB 00:00 2020-06-12T02:35:36.476 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2020-06-12T02:35:36.477 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2020-06-12T02:35:36.477 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2020-06-12T02:35:36.490 INFO:teuthology.orchestra.run.smithi107.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.3 MB/s | 43 kB 00:00 2020-06-12T02:35:36.518 INFO:teuthology.orchestra.run.smithi107.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 747 kB/s | 34 kB 00:00 2020-06-12T02:35:36.564 INFO:teuthology.orchestra.run.smithi107.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 2.6 MB/s | 197 kB 00:00 2020-06-12T02:35:36.593 INFO:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-mgr-dashboard 2020-06-12T02:35:36.678 INFO:teuthology.orchestra.run.smithi107.stdout:(6/6): ceph-mgr-dashboard-15.2.3-196.g91d24f4.e 10 MB/s | 3.4 MB 00:00 2020-06-12T02:35:36.680 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2020-06-12T02:35:36.681 INFO:teuthology.orchestra.run.smithi107.stdout:Total 11 MB/s | 3.7 MB 00:00 2020-06-12T02:35:36.685 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2020-06-12T02:35:36.694 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2020-06-12T02:35:36.694 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2020-06-12T02:35:36.841 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2020-06-12T02:35:36.842 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2020-06-12T02:35:36.990 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:11 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:35:37.078 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2020-06-12T02:35:37.317 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2020-06-12T02:35:37.350 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2020-06-12T02:35:37.351 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:35:37.352 INFO:teuthology.orchestra.run.smithi188.stdout: Package ArchVersion Repository Size 2020-06-12T02:35:37.352 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:35:37.352 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2020-06-12T02:35:37.352 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-dashboard noarch2:15.2.3-196.g91d24f4.el8 ceph-noarch 3.4 M 2020-06-12T02:35:37.353 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2020-06-12T02:35:37.353 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-grafana-dashboards noarch2:15.2.3-196.g91d24f4.el8 ceph-noarch 22 k 2020-06-12T02:35:37.353 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-prometheus-alerts noarch2:15.2.3-196.g91d24f4.el8 ceph-noarch 8.7 k 2020-06-12T02:35:37.353 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jwt noarch1.6.1-2.el8 CentOS-Base 43 k 2020-06-12T02:35:37.354 INFO:teuthology.orchestra.run.smithi188.stdout: python3-repoze-lru noarch0.7-6.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 34 k 2020-06-12T02:35:37.354 INFO:teuthology.orchestra.run.smithi188.stdout: python3-routes noarch2.4.1-12.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 197 k 2020-06-12T02:35:37.355 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:35:37.355 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2020-06-12T02:35:37.355 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:35:37.355 INFO:teuthology.orchestra.run.smithi188.stdout:Install 6 Packages 2020-06-12T02:35:37.355 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:35:37.356 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 3.7 M 2020-06-12T02:35:37.356 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 19 M 2020-06-12T02:35:37.357 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2020-06-12T02:35:37.463 INFO:teuthology.orchestra.run.smithi188.stdout:(1/6): ceph-prometheus-alerts-15.2.3-196.g91d24 80 kB/s | 8.7 kB 00:00 2020-06-12T02:35:37.477 INFO:teuthology.orchestra.run.smithi188.stdout:(2/6): ceph-grafana-dashboards-15.2.3-196.g91d2 182 kB/s | 22 kB 00:00 2020-06-12T02:35:37.494 INFO:teuthology.orchestra.run.smithi188.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.3 MB/s | 43 kB 00:00 2020-06-12T02:35:37.510 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2020-06-12T02:35:37.527 INFO:teuthology.orchestra.run.smithi188.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 680 kB/s | 34 kB 00:00 2020-06-12T02:35:37.559 INFO:teuthology.orchestra.run.smithi188.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 3.0 MB/s | 197 kB 00:00 2020-06-12T02:35:37.681 INFO:teuthology.orchestra.run.smithi188.stdout:(6/6): ceph-mgr-dashboard-15.2.3-196.g91d24f4.e 10 MB/s | 3.4 MB 00:00 2020-06-12T02:35:37.684 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2020-06-12T02:35:37.685 INFO:teuthology.orchestra.run.smithi188.stdout:Total 11 MB/s | 3.7 MB 00:00 2020-06-12T02:35:37.689 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2020-06-12T02:35:37.698 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2020-06-12T02:35:37.698 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2020-06-12T02:35:37.704 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2020-06-12T02:35:37.824 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-prometheus-alerts-2:15.2.3-196.g91d24f4.el8.noa 4/6 2020-06-12T02:35:37.853 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2020-06-12T02:35:37.854 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2020-06-12T02:35:38.092 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2020-06-12T02:35:38.316 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2020-06-12T02:35:38.536 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2020-06-12T02:35:38.728 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2020-06-12T02:35:38.857 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-prometheus-alerts-2:15.2.3-196.g91d24f4.el8.noa 4/6 2020-06-12T02:35:38.894 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-grafana-dashboards-2:15.2.3-196.g91d24f4.el8.no 5/6 2020-06-12T02:35:39.152 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-dashboard-2:15.2.3-196.g91d24f4.el8.noarch 6/6 2020-06-12T02:35:39.342 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-dashboard-2:15.2.3-196.g91d24f4.el8.noarch 6/6 2020-06-12T02:35:39.343 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-grafana-dashboards-2:15.2.3-196.g91d24f4.el8.no 1/6 2020-06-12T02:35:39.343 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-dashboard-2:15.2.3-196.g91d24f4.el8.noarch 2/6 2020-06-12T02:35:39.343 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-prometheus-alerts-2:15.2.3-196.g91d24f4.el8.noa 3/6 2020-06-12T02:35:39.344 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2020-06-12T02:35:39.344 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2020-06-12T02:35:39.578 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2020-06-12T02:35:39.579 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:35:39.579 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2020-06-12T02:35:39.579 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-dashboard-2:15.2.3-196.g91d24f4.el8.noarch 2020-06-12T02:35:39.580 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-grafana-dashboards-2:15.2.3-196.g91d24f4.el8.noarch 2020-06-12T02:35:39.580 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-prometheus-alerts-2:15.2.3-196.g91d24f4.el8.noarch 2020-06-12T02:35:39.580 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jwt-1.6.1-2.el8.noarch 2020-06-12T02:35:39.580 INFO:teuthology.orchestra.run.smithi107.stdout: python3-repoze-lru-0.7-6.el8.noarch 2020-06-12T02:35:39.581 INFO:teuthology.orchestra.run.smithi107.stdout: python3-routes-2.4.1-12.el8.noarch 2020-06-12T02:35:39.581 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:35:39.581 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2020-06-12T02:35:39.731 INFO:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-diskprediction-local 2020-06-12T02:35:39.892 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-grafana-dashboards-2:15.2.3-196.g91d24f4.el8.no 5/6 2020-06-12T02:35:40.109 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-dashboard-2:15.2.3-196.g91d24f4.el8.noarch 6/6 2020-06-12T02:35:40.128 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:15 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:35:40.297 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-dashboard-2:15.2.3-196.g91d24f4.el8.noarch 6/6 2020-06-12T02:35:40.297 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-grafana-dashboards-2:15.2.3-196.g91d24f4.el8.no 1/6 2020-06-12T02:35:40.298 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-dashboard-2:15.2.3-196.g91d24f4.el8.noarch 2/6 2020-06-12T02:35:40.298 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-prometheus-alerts-2:15.2.3-196.g91d24f4.el8.noa 3/6 2020-06-12T02:35:40.298 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2020-06-12T02:35:40.298 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2020-06-12T02:35:40.521 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2020-06-12T02:35:40.522 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:35:40.522 INFO:teuthology.orchestra.run.smithi107.stdout: Package ArchVersion Repository Size 2020-06-12T02:35:40.523 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:35:40.523 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2020-06-12T02:35:40.523 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-diskprediction-local noarch2:15.2.3-196.g91d24f4.el8 ceph-noarch 8.6 M 2020-06-12T02:35:40.523 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2020-06-12T02:35:40.523 INFO:teuthology.orchestra.run.smithi107.stdout: openblas x86_640.3.3-2.el8 CentOS-AppStream 4.3 M 2020-06-12T02:35:40.524 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-threads x86_640.3.3-2.el8 CentOS-AppStream 4.4 M 2020-06-12T02:35:40.524 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy x86_641:1.14.3-9.el8 CentOS-AppStream 3.7 M 2020-06-12T02:35:40.524 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-f2py x86_641:1.14.3-9.el8 CentOS-AppStream 225 k 2020-06-12T02:35:40.524 INFO:teuthology.orchestra.run.smithi107.stdout: python3-scipy x86_641.0.0-20.module_el8.1.0+245+c39af44f CentOS-AppStream 15 M 2020-06-12T02:35:40.525 INFO:teuthology.orchestra.run.smithi107.stdout: libgfortran x86_648.3.1-4.5.el8 CentOS-Base 638 k 2020-06-12T02:35:40.525 INFO:teuthology.orchestra.run.smithi107.stdout: libquadmath x86_648.3.1-4.5.el8 CentOS-Base 169 k 2020-06-12T02:35:40.525 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:35:40.525 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2020-06-12T02:35:40.526 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:35:40.526 INFO:teuthology.orchestra.run.smithi107.stdout:Install 8 Packages 2020-06-12T02:35:40.526 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:35:40.527 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 37 M 2020-06-12T02:35:40.527 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 205 M 2020-06-12T02:35:40.527 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2020-06-12T02:35:40.559 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2020-06-12T02:35:40.559 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:35:40.560 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2020-06-12T02:35:40.560 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-dashboard-2:15.2.3-196.g91d24f4.el8.noarch 2020-06-12T02:35:40.560 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-grafana-dashboards-2:15.2.3-196.g91d24f4.el8.noarch 2020-06-12T02:35:40.560 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-prometheus-alerts-2:15.2.3-196.g91d24f4.el8.noarch 2020-06-12T02:35:40.561 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jwt-1.6.1-2.el8.noarch 2020-06-12T02:35:40.561 INFO:teuthology.orchestra.run.smithi188.stdout: python3-repoze-lru-0.7-6.el8.noarch 2020-06-12T02:35:40.562 INFO:teuthology.orchestra.run.smithi188.stdout: python3-routes-2.4.1-12.el8.noarch 2020-06-12T02:35:40.562 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:35:40.562 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2020-06-12T02:35:40.634 INFO:teuthology.orchestra.run.smithi107.stdout:(1/8): openblas-0.3.3-2.el8.x86_64.rpm 40 MB/s | 4.3 MB 00:00 2020-06-12T02:35:40.685 INFO:teuthology.orchestra.run.smithi107.stdout:(2/8): openblas-threads-0.3.3-2.el8.x86_64.rpm 27 MB/s | 4.4 MB 00:00 2020-06-12T02:35:40.702 INFO:teuthology.orchestra.run.smithi107.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 13 MB/s | 225 kB 00:00 2020-06-12T02:35:40.709 INFO:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-mgr-diskprediction-local 2020-06-12T02:35:40.779 INFO:teuthology.orchestra.run.smithi107.stdout:(4/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 26 MB/s | 3.7 MB 00:00 2020-06-12T02:35:40.849 INFO:teuthology.orchestra.run.smithi107.stdout:(5/8): libgfortran-8.3.1-4.5.el8.x86_64.rpm 9.0 MB/s | 638 kB 00:00 2020-06-12T02:35:40.893 INFO:teuthology.orchestra.run.smithi107.stdout:(6/8): libquadmath-8.3.1-4.5.el8.x86_64.rpm 3.8 MB/s | 169 kB 00:00 2020-06-12T02:35:41.109 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:16 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:35:41.183 INFO:teuthology.orchestra.run.smithi107.stdout:(7/8): python3-scipy-1.0.0-20.module_el8.1.0+24 30 MB/s | 15 MB 00:00 2020-06-12T02:35:41.309 INFO:teuthology.orchestra.run.smithi107.stdout:(8/8): ceph-mgr-diskprediction-local-15.2.3-196 11 MB/s | 8.6 MB 00:00 2020-06-12T02:35:41.311 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2020-06-12T02:35:41.312 INFO:teuthology.orchestra.run.smithi107.stdout:Total 46 MB/s | 37 MB 00:00 2020-06-12T02:35:41.323 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2020-06-12T02:35:41.351 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2020-06-12T02:35:41.352 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2020-06-12T02:35:41.476 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2020-06-12T02:35:41.480 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:35:41.480 INFO:teuthology.orchestra.run.smithi188.stdout: Package ArchVersion Repository Size 2020-06-12T02:35:41.481 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:35:41.481 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2020-06-12T02:35:41.481 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-diskprediction-local noarch2:15.2.3-196.g91d24f4.el8 ceph-noarch 8.6 M 2020-06-12T02:35:41.481 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2020-06-12T02:35:41.482 INFO:teuthology.orchestra.run.smithi188.stdout: openblas x86_640.3.3-2.el8 CentOS-AppStream 4.3 M 2020-06-12T02:35:41.483 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-threads x86_640.3.3-2.el8 CentOS-AppStream 4.4 M 2020-06-12T02:35:41.484 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy x86_641:1.14.3-9.el8 CentOS-AppStream 3.7 M 2020-06-12T02:35:41.485 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-f2py x86_641:1.14.3-9.el8 CentOS-AppStream 225 k 2020-06-12T02:35:41.485 INFO:teuthology.orchestra.run.smithi188.stdout: python3-scipy x86_641.0.0-20.module_el8.1.0+245+c39af44f CentOS-AppStream 15 M 2020-06-12T02:35:41.485 INFO:teuthology.orchestra.run.smithi188.stdout: libgfortran x86_648.3.1-4.5.el8 CentOS-Base 638 k 2020-06-12T02:35:41.485 INFO:teuthology.orchestra.run.smithi188.stdout: libquadmath x86_648.3.1-4.5.el8 CentOS-Base 169 k 2020-06-12T02:35:41.486 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:35:41.486 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2020-06-12T02:35:41.486 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:35:41.486 INFO:teuthology.orchestra.run.smithi188.stdout:Install 8 Packages 2020-06-12T02:35:41.487 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:35:41.487 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 37 M 2020-06-12T02:35:41.487 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 205 M 2020-06-12T02:35:41.487 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2020-06-12T02:35:41.598 INFO:teuthology.orchestra.run.smithi188.stdout:(1/8): openblas-threads-0.3.3-2.el8.x86_64.rpm 41 MB/s | 4.4 MB 00:00 2020-06-12T02:35:41.641 INFO:teuthology.orchestra.run.smithi188.stdout:(2/8): openblas-0.3.3-2.el8.x86_64.rpm 27 MB/s | 4.3 MB 00:00 2020-06-12T02:35:41.658 INFO:teuthology.orchestra.run.smithi188.stdout:(3/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 53 MB/s | 3.7 MB 00:00 2020-06-12T02:35:41.660 INFO:teuthology.orchestra.run.smithi188.stdout:(4/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 12 MB/s | 225 kB 00:00 2020-06-12T02:35:41.683 INFO:teuthology.orchestra.run.smithi188.stdout:(5/8): libgfortran-8.3.1-4.5.el8.x86_64.rpm 26 MB/s | 638 kB 00:00 2020-06-12T02:35:41.694 INFO:teuthology.orchestra.run.smithi188.stdout:(6/8): libquadmath-8.3.1-4.5.el8.x86_64.rpm 16 MB/s | 169 kB 00:00 2020-06-12T02:35:41.710 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2020-06-12T02:35:41.711 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2020-06-12T02:35:41.902 INFO:teuthology.orchestra.run.smithi188.stdout:(7/8): python3-scipy-1.0.0-20.module_el8.1.0+24 60 MB/s | 15 MB 00:00 2020-06-12T02:35:42.156 INFO:teuthology.orchestra.run.smithi188.stdout:(8/8): ceph-mgr-diskprediction-local-15.2.3-196 13 MB/s | 8.6 MB 00:00 2020-06-12T02:35:42.159 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2020-06-12T02:35:42.160 INFO:teuthology.orchestra.run.smithi188.stdout:Total 54 MB/s | 37 MB 00:00 2020-06-12T02:35:42.169 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2020-06-12T02:35:42.188 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2020-06-12T02:35:42.188 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2020-06-12T02:35:42.485 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2020-06-12T02:35:42.486 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2020-06-12T02:35:42.996 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2020-06-12T02:35:43.033 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2020-06-12T02:35:43.167 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libquadmath-8.3.1-4.5.el8.x86_64 1/8 2020-06-12T02:35:43.193 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libquadmath-8.3.1-4.5.el8.x86_64 1/8 2020-06-12T02:35:43.306 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libquadmath-8.3.1-4.5.el8.x86_64 1/8 2020-06-12T02:35:43.317 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libquadmath-8.3.1-4.5.el8.x86_64 1/8 2020-06-12T02:35:43.459 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libgfortran-8.3.1-4.5.el8.x86_64 2/8 2020-06-12T02:35:43.491 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libgfortran-8.3.1-4.5.el8.x86_64 2/8 2020-06-12T02:35:43.982 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libgfortran-8.3.1-4.5.el8.x86_64 2/8 2020-06-12T02:35:44.025 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libgfortran-8.3.1-4.5.el8.x86_64 2/8 2020-06-12T02:35:44.151 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openblas-0.3.3-2.el8.x86_64 3/8 2020-06-12T02:35:44.208 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : openblas-0.3.3-2.el8.x86_64 3/8 2020-06-12T02:35:44.690 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: openblas-0.3.3-2.el8.x86_64 3/8 2020-06-12T02:35:44.748 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: openblas-0.3.3-2.el8.x86_64 3/8 2020-06-12T02:35:44.834 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openblas-threads-0.3.3-2.el8.x86_64 4/8 2020-06-12T02:35:45.514 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: openblas-threads-0.3.3-2.el8.x86_64 4/8 2020-06-12T02:35:45.829 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2020-06-12T02:35:46.025 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2020-06-12T02:35:46.190 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : openblas-threads-0.3.3-2.el8.x86_64 4/8 2020-06-12T02:35:46.884 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: openblas-threads-0.3.3-2.el8.x86_64 4/8 2020-06-12T02:35:47.228 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2020-06-12T02:35:47.432 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2020-06-12T02:35:47.971 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2020-06-12T02:35:49.262 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x 7/8 2020-06-12T02:35:49.366 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2020-06-12T02:35:49.527 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-diskprediction-local-2:15.2.3-196.g91d24f4. 8/8 2020-06-12T02:35:49.751 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:15.2.3-196.g91d24f4. 8/8 2020-06-12T02:35:49.752 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-diskprediction-local-2:15.2.3-196.g91d24f4. 1/8 2020-06-12T02:35:49.752 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openblas-0.3.3-2.el8.x86_64 2/8 2020-06-12T02:35:49.752 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openblas-threads-0.3.3-2.el8.x86_64 3/8 2020-06-12T02:35:49.752 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2020-06-12T02:35:49.752 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2020-06-12T02:35:49.753 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x 6/8 2020-06-12T02:35:49.753 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libgfortran-8.3.1-4.5.el8.x86_64 7/8 2020-06-12T02:35:50.035 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libquadmath-8.3.1-4.5.el8.x86_64 8/8 2020-06-12T02:35:50.035 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:35:50.035 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2020-06-12T02:35:50.035 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-diskprediction-local-2:15.2.3-196.g91d24f4.el8.noarch 2020-06-12T02:35:50.036 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-0.3.3-2.el8.x86_64 2020-06-12T02:35:50.036 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-threads-0.3.3-2.el8.x86_64 2020-06-12T02:35:50.036 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2020-06-12T02:35:50.036 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2020-06-12T02:35:50.037 INFO:teuthology.orchestra.run.smithi107.stdout: python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x86_64 2020-06-12T02:35:50.037 INFO:teuthology.orchestra.run.smithi107.stdout: libgfortran-8.3.1-4.5.el8.x86_64 2020-06-12T02:35:50.037 INFO:teuthology.orchestra.run.smithi107.stdout: libquadmath-8.3.1-4.5.el8.x86_64 2020-06-12T02:35:50.037 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:35:50.038 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2020-06-12T02:35:50.180 INFO:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-rook 2020-06-12T02:35:50.579 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:25 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:35:50.647 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x 7/8 2020-06-12T02:35:50.839 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-diskprediction-local-2:15.2.3-196.g91d24f4. 8/8 2020-06-12T02:35:50.943 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2020-06-12T02:35:50.946 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:35:50.946 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2020-06-12T02:35:50.946 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:35:50.946 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2020-06-12T02:35:50.947 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-rook noarch 2:15.2.3-196.g91d24f4.el8 2020-06-12T02:35:50.947 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-noarch 27 k 2020-06-12T02:35:50.947 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2020-06-12T02:35:50.947 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2020-06-12T02:35:50.947 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2020-06-12T02:35:50.948 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2020-06-12T02:35:50.948 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2020-06-12T02:35:50.948 INFO:teuthology.orchestra.run.smithi107.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2020-06-12T02:35:50.948 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2020-06-12T02:35:50.948 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2020-06-12T02:35:50.949 INFO:teuthology.orchestra.run.smithi107.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2020-06-12T02:35:50.949 INFO:teuthology.orchestra.run.smithi107.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2020-06-12T02:35:50.949 INFO:teuthology.orchestra.run.smithi107.stdout: python3-kubernetes noarch 11.0.0-2.el8 epel 1.2 M 2020-06-12T02:35:50.949 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rsa noarch 3.4.2-12.el8 epel 73 k 2020-06-12T02:35:50.949 INFO:teuthology.orchestra.run.smithi107.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2020-06-12T02:35:50.954 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:35:50.954 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2020-06-12T02:35:50.954 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:35:50.955 INFO:teuthology.orchestra.run.smithi107.stdout:Install 13 Packages 2020-06-12T02:35:50.955 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:35:50.955 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 1.9 M 2020-06-12T02:35:50.956 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 24 M 2020-06-12T02:35:50.956 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2020-06-12T02:35:50.979 INFO:teuthology.orchestra.run.smithi107.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 845 kB/s | 27 kB 00:00 2020-06-12T02:35:50.981 INFO:teuthology.orchestra.run.smithi107.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 597 kB/s | 20 kB 00:00 2020-06-12T02:35:50.983 INFO:teuthology.orchestra.run.smithi107.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 40 MB/s | 126 kB 00:00 2020-06-12T02:35:50.984 INFO:teuthology.orchestra.run.smithi107.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 46 MB/s | 110 kB 00:00 2020-06-12T02:35:50.986 INFO:teuthology.orchestra.run.smithi107.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 28 MB/s | 43 kB 00:00 2020-06-12T02:35:50.987 INFO:teuthology.orchestra.run.smithi107.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 41 MB/s | 155 kB 00:00 2020-06-12T02:35:51.043 INFO:teuthology.orchestra.run.smithi107.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 289 kB/s | 16 kB 00:00 2020-06-12T02:35:51.044 INFO:teuthology.orchestra.run.smithi107.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 575 kB/s | 33 kB 00:00 2020-06-12T02:35:51.070 INFO:teuthology.orchestra.run.smithi107.stdout:(9/13): ceph-mgr-rook-15.2.3-196.g91d24f4.el8.n 222 kB/s | 27 kB 00:00 2020-06-12T02:35:51.073 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:15.2.3-196.g91d24f4. 8/8 2020-06-12T02:35:51.073 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-diskprediction-local-2:15.2.3-196.g91d24f4. 1/8 2020-06-12T02:35:51.073 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openblas-0.3.3-2.el8.x86_64 2/8 2020-06-12T02:35:51.074 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openblas-threads-0.3.3-2.el8.x86_64 3/8 2020-06-12T02:35:51.074 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2020-06-12T02:35:51.074 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2020-06-12T02:35:51.074 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x 6/8 2020-06-12T02:35:51.074 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libgfortran-8.3.1-4.5.el8.x86_64 7/8 2020-06-12T02:35:51.081 INFO:teuthology.orchestra.run.smithi107.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 2.1 MB/s | 81 kB 00:00 2020-06-12T02:35:51.100 INFO:teuthology.orchestra.run.smithi107.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 3.1 MB/s | 61 kB 00:00 2020-06-12T02:35:51.132 INFO:teuthology.orchestra.run.smithi107.stdout:(12/13): python3-rsa-3.4.2-12.el8.noarch.rpm 1.1 MB/s | 73 kB 00:00 2020-06-12T02:35:51.159 INFO:teuthology.orchestra.run.smithi107.stdout:(13/13): python3-kubernetes-11.0.0-2.el8.noarch 10 MB/s | 1.2 MB 00:00 2020-06-12T02:35:51.160 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2020-06-12T02:35:51.160 INFO:teuthology.orchestra.run.smithi107.stdout:Total 9.0 MB/s | 1.9 MB 00:00 2020-06-12T02:35:51.169 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2020-06-12T02:35:51.181 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2020-06-12T02:35:51.182 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2020-06-12T02:35:51.248 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2020-06-12T02:35:51.248 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2020-06-12T02:35:51.416 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libquadmath-8.3.1-4.5.el8.x86_64 8/8 2020-06-12T02:35:51.416 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:35:51.417 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2020-06-12T02:35:51.417 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-diskprediction-local-2:15.2.3-196.g91d24f4.el8.noarch 2020-06-12T02:35:51.418 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-0.3.3-2.el8.x86_64 2020-06-12T02:35:51.418 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-threads-0.3.3-2.el8.x86_64 2020-06-12T02:35:51.419 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2020-06-12T02:35:51.419 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2020-06-12T02:35:51.419 INFO:teuthology.orchestra.run.smithi188.stdout: python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x86_64 2020-06-12T02:35:51.419 INFO:teuthology.orchestra.run.smithi188.stdout: libgfortran-8.3.1-4.5.el8.x86_64 2020-06-12T02:35:51.419 INFO:teuthology.orchestra.run.smithi188.stdout: libquadmath-8.3.1-4.5.el8.x86_64 2020-06-12T02:35:51.420 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:35:51.420 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2020-06-12T02:35:51.556 INFO:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-mgr-rook 2020-06-12T02:35:51.742 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2020-06-12T02:35:51.957 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:26 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:35:51.984 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2020-06-12T02:35:52.179 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2020-06-12T02:35:52.331 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2020-06-12T02:35:52.333 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:35:52.333 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2020-06-12T02:35:52.333 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:35:52.334 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2020-06-12T02:35:52.334 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-rook noarch 2:15.2.3-196.g91d24f4.el8 2020-06-12T02:35:52.334 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-noarch 27 k 2020-06-12T02:35:52.334 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2020-06-12T02:35:52.334 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2020-06-12T02:35:52.335 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2020-06-12T02:35:52.335 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2020-06-12T02:35:52.335 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2020-06-12T02:35:52.335 INFO:teuthology.orchestra.run.smithi188.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2020-06-12T02:35:52.335 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2020-06-12T02:35:52.336 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2020-06-12T02:35:52.336 INFO:teuthology.orchestra.run.smithi188.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2020-06-12T02:35:52.336 INFO:teuthology.orchestra.run.smithi188.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2020-06-12T02:35:52.336 INFO:teuthology.orchestra.run.smithi188.stdout: python3-kubernetes noarch 11.0.0-2.el8 epel 1.2 M 2020-06-12T02:35:52.336 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rsa noarch 3.4.2-12.el8 epel 73 k 2020-06-12T02:35:52.337 INFO:teuthology.orchestra.run.smithi188.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2020-06-12T02:35:52.337 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:35:52.337 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2020-06-12T02:35:52.337 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:35:52.337 INFO:teuthology.orchestra.run.smithi188.stdout:Install 13 Packages 2020-06-12T02:35:52.337 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:35:52.338 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 1.9 M 2020-06-12T02:35:52.338 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 24 M 2020-06-12T02:35:52.339 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2020-06-12T02:35:52.373 INFO:teuthology.orchestra.run.smithi188.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 853 kB/s | 27 kB 00:00 2020-06-12T02:35:52.373 INFO:teuthology.orchestra.run.smithi188.stdout:(2/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 44 MB/s | 126 kB 00:00 2020-06-12T02:35:52.374 INFO:teuthology.orchestra.run.smithi188.stdout:(3/13): python3-pyasn1-modules-0.3.7-6.el8.noar 51 MB/s | 110 kB 00:00 2020-06-12T02:35:52.374 INFO:teuthology.orchestra.run.smithi188.stdout:(4/13): python3-jsonpointer-1.10-11.el8.noarch. 535 kB/s | 20 kB 00:00 2020-06-12T02:35:52.375 INFO:teuthology.orchestra.run.smithi188.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 25 MB/s | 43 kB 00:00 2020-06-12T02:35:52.376 INFO:teuthology.orchestra.run.smithi188.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 51 MB/s | 155 kB 00:00 2020-06-12T02:35:52.409 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rsa-3.4.2-12.el8.noarch 3/13 2020-06-12T02:35:52.423 INFO:teuthology.orchestra.run.smithi188.stdout:(7/13): python3-cachetools-3.1.1-4.el8.noarch.r 700 kB/s | 33 kB 00:00 2020-06-12T02:35:52.432 INFO:teuthology.orchestra.run.smithi188.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 292 kB/s | 16 kB 00:00 2020-06-12T02:35:52.454 INFO:teuthology.orchestra.run.smithi188.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 2.6 MB/s | 81 kB 00:00 2020-06-12T02:35:52.458 INFO:teuthology.orchestra.run.smithi188.stdout:(10/13): ceph-mgr-rook-15.2.3-196.g91d24f4.el8. 222 kB/s | 27 kB 00:00 2020-06-12T02:35:52.470 INFO:teuthology.orchestra.run.smithi188.stdout:(11/13): python3-rsa-3.4.2-12.el8.noarch.rpm 4.3 MB/s | 73 kB 00:00 2020-06-12T02:35:52.533 INFO:teuthology.orchestra.run.smithi188.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 820 kB/s | 61 kB 00:00 2020-06-12T02:35:52.547 INFO:teuthology.orchestra.run.smithi188.stdout:(13/13): python3-kubernetes-11.0.0-2.el8.noarch 10 MB/s | 1.2 MB 00:00 2020-06-12T02:35:52.548 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2020-06-12T02:35:52.549 INFO:teuthology.orchestra.run.smithi188.stdout:Total 8.9 MB/s | 1.9 MB 00:00 2020-06-12T02:35:52.558 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2020-06-12T02:35:52.570 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2020-06-12T02:35:52.570 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2020-06-12T02:35:52.589 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2020-06-12T02:35:52.643 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2020-06-12T02:35:52.644 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2020-06-12T02:35:52.796 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2020-06-12T02:35:53.096 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2020-06-12T02:35:53.113 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2020-06-12T02:35:53.365 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2020-06-12T02:35:53.552 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2020-06-12T02:35:53.570 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2020-06-12T02:35:53.746 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rsa-3.4.2-12.el8.noarch 3/13 2020-06-12T02:35:53.795 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2020-06-12T02:35:53.936 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2020-06-12T02:35:54.144 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2020-06-12T02:35:54.374 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2020-06-12T02:35:54.624 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2020-06-12T02:35:54.825 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2020-06-12T02:35:54.872 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2020-06-12T02:35:55.105 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-kubernetes-11.0.0-2.el8.noarch 10/13 2020-06-12T02:35:55.263 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2020-06-12T02:35:55.472 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2020-06-12T02:35:55.631 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-rook-2:15.2.3-196.g91d24f4.el8.noarch 13/13 2020-06-12T02:35:55.833 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-rook-2:15.2.3-196.g91d24f4.el8.noarch 13/13 2020-06-12T02:35:55.834 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-rook-2:15.2.3-196.g91d24f4.el8.noarch 1/13 2020-06-12T02:35:55.834 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2020-06-12T02:35:55.834 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2020-06-12T02:35:55.834 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2020-06-12T02:35:55.834 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2020-06-12T02:35:55.835 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2020-06-12T02:35:55.835 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2020-06-12T02:35:55.835 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2020-06-12T02:35:55.835 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2020-06-12T02:35:55.836 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2020-06-12T02:35:55.836 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-kubernetes-11.0.0-2.el8.noarch 11/13 2020-06-12T02:35:55.836 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rsa-3.4.2-12.el8.noarch 12/13 2020-06-12T02:35:55.872 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2020-06-12T02:35:56.120 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-kubernetes-11.0.0-2.el8.noarch 10/13 2020-06-12T02:35:56.142 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2020-06-12T02:35:56.142 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:35:56.143 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2020-06-12T02:35:56.143 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-rook-2:15.2.3-196.g91d24f4.el8.noarch 2020-06-12T02:35:56.143 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpatch-1.21-2.el8.noarch 2020-06-12T02:35:56.143 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpointer-1.10-11.el8.noarch 2020-06-12T02:35:56.143 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2020-06-12T02:35:56.144 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2020-06-12T02:35:56.144 INFO:teuthology.orchestra.run.smithi107.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2020-06-12T02:35:56.144 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2020-06-12T02:35:56.144 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cachetools-3.1.1-4.el8.noarch 2020-06-12T02:35:56.144 INFO:teuthology.orchestra.run.smithi107.stdout: python3-certifi-2018.10.15-7.el8.noarch 2020-06-12T02:35:56.145 INFO:teuthology.orchestra.run.smithi107.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2020-06-12T02:35:56.145 INFO:teuthology.orchestra.run.smithi107.stdout: python3-kubernetes-11.0.0-2.el8.noarch 2020-06-12T02:35:56.145 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rsa-3.4.2-12.el8.noarch 2020-06-12T02:35:56.145 INFO:teuthology.orchestra.run.smithi107.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2020-06-12T02:35:56.145 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:35:56.146 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2020-06-12T02:35:56.262 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2020-06-12T02:35:56.287 INFO:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-cephadm 2020-06-12T02:35:56.690 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:31 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:35:56.989 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2020-06-12T02:35:57.049 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2020-06-12T02:35:57.050 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:35:57.050 INFO:teuthology.orchestra.run.smithi107.stdout: Package ArchVersion Repository Size 2020-06-12T02:35:57.051 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:35:57.051 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2020-06-12T02:35:57.051 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-cephadm noarch2:15.2.3-196.g91d24f4.el8 ceph-noarch 45 k 2020-06-12T02:35:57.051 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2020-06-12T02:35:57.051 INFO:teuthology.orchestra.run.smithi107.stdout: python3-execnet noarch1.7.1-1.el8.2 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 356 k 2020-06-12T02:35:57.052 INFO:teuthology.orchestra.run.smithi107.stdout: python3-remoto noarch1.1.4-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 48 k 2020-06-12T02:35:57.052 INFO:teuthology.orchestra.run.smithi107.stdout: python3-apipkg noarch1.5-6.el8 epel 21 k 2020-06-12T02:35:57.052 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:35:57.052 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2020-06-12T02:35:57.053 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:35:57.053 INFO:teuthology.orchestra.run.smithi107.stdout:Install 4 Packages 2020-06-12T02:35:57.053 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:35:57.054 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 470 k 2020-06-12T02:35:57.054 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 1.5 M 2020-06-12T02:35:57.054 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2020-06-12T02:35:57.106 INFO:teuthology.orchestra.run.smithi107.stdout:(1/4): python3-remoto-1.1.4-4.el8.noarch.rpm 918 kB/s | 48 kB 00:00 2020-06-12T02:35:57.118 INFO:teuthology.orchestra.run.smithi107.stdout:(2/4): python3-execnet-1.7.1-1.el8.2.noarch.rpm 5.3 MB/s | 356 kB 00:00 2020-06-12T02:35:57.162 INFO:teuthology.orchestra.run.smithi107.stdout:(3/4): python3-apipkg-1.5-6.el8.noarch.rpm 374 kB/s | 21 kB 00:00 2020-06-12T02:35:57.197 INFO:teuthology.orchestra.run.smithi107.stdout:(4/4): ceph-mgr-cephadm-15.2.3-196.g91d24f4.el8 313 kB/s | 45 kB 00:00 2020-06-12T02:35:57.199 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2020-06-12T02:35:57.199 INFO:teuthology.orchestra.run.smithi107.stdout:Total 3.1 MB/s | 470 kB 00:00 2020-06-12T02:35:57.203 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2020-06-12T02:35:57.210 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2020-06-12T02:35:57.210 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2020-06-12T02:35:57.236 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2020-06-12T02:35:57.236 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2020-06-12T02:35:57.383 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2020-06-12T02:35:57.488 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-rook-2:15.2.3-196.g91d24f4.el8.noarch 13/13 2020-06-12T02:35:57.623 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/4 2020-06-12T02:35:57.688 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-rook-2:15.2.3-196.g91d24f4.el8.noarch 13/13 2020-06-12T02:35:57.688 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-rook-2:15.2.3-196.g91d24f4.el8.noarch 1/13 2020-06-12T02:35:57.688 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2020-06-12T02:35:57.688 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2020-06-12T02:35:57.689 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2020-06-12T02:35:57.689 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2020-06-12T02:35:57.689 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2020-06-12T02:35:57.689 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2020-06-12T02:35:57.690 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2020-06-12T02:35:57.690 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2020-06-12T02:35:57.690 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2020-06-12T02:35:57.690 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-kubernetes-11.0.0-2.el8.noarch 11/13 2020-06-12T02:35:57.690 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rsa-3.4.2-12.el8.noarch 12/13 2020-06-12T02:35:57.909 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-execnet-1.7.1-1.el8.2.noarch 2/4 2020-06-12T02:35:58.055 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2020-06-12T02:35:58.055 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:35:58.055 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2020-06-12T02:35:58.056 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-rook-2:15.2.3-196.g91d24f4.el8.noarch 2020-06-12T02:35:58.056 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpatch-1.21-2.el8.noarch 2020-06-12T02:35:58.056 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpointer-1.10-11.el8.noarch 2020-06-12T02:35:58.056 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2020-06-12T02:35:58.056 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2020-06-12T02:35:58.056 INFO:teuthology.orchestra.run.smithi188.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2020-06-12T02:35:58.057 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2020-06-12T02:35:58.057 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cachetools-3.1.1-4.el8.noarch 2020-06-12T02:35:58.057 INFO:teuthology.orchestra.run.smithi188.stdout: python3-certifi-2018.10.15-7.el8.noarch 2020-06-12T02:35:58.057 INFO:teuthology.orchestra.run.smithi188.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2020-06-12T02:35:58.060 INFO:teuthology.orchestra.run.smithi188.stdout: python3-kubernetes-11.0.0-2.el8.noarch 2020-06-12T02:35:58.060 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rsa-3.4.2-12.el8.noarch 2020-06-12T02:35:58.060 INFO:teuthology.orchestra.run.smithi188.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2020-06-12T02:35:58.061 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:35:58.061 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2020-06-12T02:35:58.099 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 3/4 2020-06-12T02:35:58.187 INFO:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-mgr-cephadm 2020-06-12T02:35:58.266 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-cephadm-2:15.2.3-196.g91d24f4.el8.noarch 4/4 2020-06-12T02:35:58.468 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-cephadm-2:15.2.3-196.g91d24f4.el8.noarch 4/4 2020-06-12T02:35:58.469 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-cephadm-2:15.2.3-196.g91d24f4.el8.noarch 1/4 2020-06-12T02:35:58.469 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-execnet-1.7.1-1.el8.2.noarch 2/4 2020-06-12T02:35:58.469 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 3/4 2020-06-12T02:35:58.590 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:33 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:35:58.954 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2020-06-12T02:35:58.955 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:35:58.955 INFO:teuthology.orchestra.run.smithi188.stdout: Package ArchVersion Repository Size 2020-06-12T02:35:58.955 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:35:58.955 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2020-06-12T02:35:58.956 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-cephadm noarch2:15.2.3-196.g91d24f4.el8 ceph-noarch 45 k 2020-06-12T02:35:58.956 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2020-06-12T02:35:58.956 INFO:teuthology.orchestra.run.smithi188.stdout: python3-execnet noarch1.7.1-1.el8.2 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 356 k 2020-06-12T02:35:58.956 INFO:teuthology.orchestra.run.smithi188.stdout: python3-remoto noarch1.1.4-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 48 k 2020-06-12T02:35:58.956 INFO:teuthology.orchestra.run.smithi188.stdout: python3-apipkg noarch1.5-6.el8 epel 21 k 2020-06-12T02:35:58.957 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:35:58.957 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2020-06-12T02:35:58.957 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:35:58.957 INFO:teuthology.orchestra.run.smithi188.stdout:Install 4 Packages 2020-06-12T02:35:58.957 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:35:58.958 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 470 k 2020-06-12T02:35:58.958 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 1.5 M 2020-06-12T02:35:58.958 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2020-06-12T02:35:59.026 INFO:teuthology.orchestra.run.smithi188.stdout:(1/4): python3-remoto-1.1.4-4.el8.noarch.rpm 889 kB/s | 48 kB 00:00 2020-06-12T02:35:59.027 INFO:teuthology.orchestra.run.smithi188.stdout:(2/4): python3-execnet-1.7.1-1.el8.2.noarch.rpm 5.2 MB/s | 356 kB 00:00 2020-06-12T02:35:59.068 INFO:teuthology.orchestra.run.smithi188.stdout:(3/4): python3-apipkg-1.5-6.el8.noarch.rpm 374 kB/s | 21 kB 00:00 2020-06-12T02:35:59.103 INFO:teuthology.orchestra.run.smithi188.stdout:(4/4): ceph-mgr-cephadm-15.2.3-196.g91d24f4.el8 310 kB/s | 45 kB 00:00 2020-06-12T02:35:59.104 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2020-06-12T02:35:59.105 INFO:teuthology.orchestra.run.smithi188.stdout:Total 3.1 MB/s | 470 kB 00:00 2020-06-12T02:35:59.114 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2020-06-12T02:35:59.119 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2020-06-12T02:35:59.120 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2020-06-12T02:35:59.140 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2020-06-12T02:35:59.140 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2020-06-12T02:35:59.267 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/4 2020-06-12T02:35:59.267 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:35:59.268 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2020-06-12T02:35:59.268 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-cephadm-2:15.2.3-196.g91d24f4.el8.noarch 2020-06-12T02:35:59.268 INFO:teuthology.orchestra.run.smithi107.stdout: python3-execnet-1.7.1-1.el8.2.noarch 2020-06-12T02:35:59.268 INFO:teuthology.orchestra.run.smithi107.stdout: python3-remoto-1.1.4-4.el8.noarch 2020-06-12T02:35:59.268 INFO:teuthology.orchestra.run.smithi107.stdout: python3-apipkg-1.5-6.el8.noarch 2020-06-12T02:35:59.269 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:35:59.269 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2020-06-12T02:35:59.286 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2020-06-12T02:35:59.439 INFO:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-fuse 2020-06-12T02:35:59.512 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/4 2020-06-12T02:35:59.756 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-execnet-1.7.1-1.el8.2.noarch 2/4 2020-06-12T02:35:59.845 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:34 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:35:59.930 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 3/4 2020-06-12T02:36:00.120 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-cephadm-2:15.2.3-196.g91d24f4.el8.noarch 4/4 2020-06-12T02:36:00.207 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2020-06-12T02:36:00.208 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:36:00.208 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2020-06-12T02:36:00.208 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:36:00.208 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2020-06-12T02:36:00.209 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse x86_64 2:15.2.3-196.g91d24f4.el8 ceph 574 k 2020-06-12T02:36:00.209 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2020-06-12T02:36:00.209 INFO:teuthology.orchestra.run.smithi107.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2020-06-12T02:36:00.209 INFO:teuthology.orchestra.run.smithi107.stdout: fuse-common x86_64 3.2.1-12.el8 CentOS-Base 21 k 2020-06-12T02:36:00.209 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:00.210 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2020-06-12T02:36:00.210 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:36:00.210 INFO:teuthology.orchestra.run.smithi107.stdout:Install 3 Packages 2020-06-12T02:36:00.210 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:00.211 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 677 k 2020-06-12T02:36:00.211 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 2.1 M 2020-06-12T02:36:00.211 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2020-06-12T02:36:00.244 INFO:teuthology.orchestra.run.smithi107.stdout:(1/3): fuse-2.9.7-12.el8.x86_64.rpm 2.4 MB/s | 82 kB 00:00 2020-06-12T02:36:00.248 INFO:teuthology.orchestra.run.smithi107.stdout:(2/3): fuse-common-3.2.1-12.el8.x86_64.rpm 557 kB/s | 21 kB 00:00 2020-06-12T02:36:00.323 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-cephadm-2:15.2.3-196.g91d24f4.el8.noarch 4/4 2020-06-12T02:36:00.324 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-cephadm-2:15.2.3-196.g91d24f4.el8.noarch 1/4 2020-06-12T02:36:00.324 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-execnet-1.7.1-1.el8.2.noarch 2/4 2020-06-12T02:36:00.324 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 3/4 2020-06-12T02:36:00.429 INFO:teuthology.orchestra.run.smithi107.stdout:(3/3): ceph-fuse-15.2.3-196.g91d24f4.el8.x86_64 2.6 MB/s | 574 kB 00:00 2020-06-12T02:36:00.500 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2020-06-12T02:36:00.500 INFO:teuthology.orchestra.run.smithi107.stdout:Total 3.0 MB/s | 677 kB 00:00 2020-06-12T02:36:00.501 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2020-06-12T02:36:00.501 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2020-06-12T02:36:00.501 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2020-06-12T02:36:00.514 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2020-06-12T02:36:00.514 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2020-06-12T02:36:00.579 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/4 2020-06-12T02:36:00.579 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:00.579 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2020-06-12T02:36:00.580 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-cephadm-2:15.2.3-196.g91d24f4.el8.noarch 2020-06-12T02:36:00.580 INFO:teuthology.orchestra.run.smithi188.stdout: python3-execnet-1.7.1-1.el8.2.noarch 2020-06-12T02:36:00.580 INFO:teuthology.orchestra.run.smithi188.stdout: python3-remoto-1.1.4-4.el8.noarch 2020-06-12T02:36:00.580 INFO:teuthology.orchestra.run.smithi188.stdout: python3-apipkg-1.5-6.el8.noarch 2020-06-12T02:36:00.580 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:00.581 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2020-06-12T02:36:00.672 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2020-06-12T02:36:00.673 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 1/3 2020-06-12T02:36:00.673 INFO:teuthology.orchestra.run.smithi107.stdout:warning: /etc/fuse.conf created as /etc/fuse.conf.rpmnew 2020-06-12T02:36:00.673 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:00.704 INFO:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-fuse 2020-06-12T02:36:01.096 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : fuse-2.9.7-12.el8.x86_64 2/3 2020-06-12T02:36:01.105 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:36 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:36:01.279 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-fuse-2:15.2.3-196.g91d24f4.el8.x86_64 3/3 2020-06-12T02:36:01.461 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2020-06-12T02:36:01.462 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:36:01.462 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2020-06-12T02:36:01.462 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:36:01.462 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2020-06-12T02:36:01.463 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-fuse x86_64 2:15.2.3-196.g91d24f4.el8 ceph 574 k 2020-06-12T02:36:01.463 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2020-06-12T02:36:01.463 INFO:teuthology.orchestra.run.smithi188.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2020-06-12T02:36:01.463 INFO:teuthology.orchestra.run.smithi188.stdout: fuse-common x86_64 3.2.1-12.el8 CentOS-Base 21 k 2020-06-12T02:36:01.463 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:01.464 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2020-06-12T02:36:01.464 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:36:01.464 INFO:teuthology.orchestra.run.smithi188.stdout:Install 3 Packages 2020-06-12T02:36:01.464 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:01.465 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 677 k 2020-06-12T02:36:01.465 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 2.1 M 2020-06-12T02:36:01.465 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2020-06-12T02:36:01.501 INFO:teuthology.orchestra.run.smithi188.stdout:(1/3): fuse-common-3.2.1-12.el8.x86_64.rpm 573 kB/s | 21 kB 00:00 2020-06-12T02:36:01.509 INFO:teuthology.orchestra.run.smithi188.stdout:(2/3): fuse-2.9.7-12.el8.x86_64.rpm 1.9 MB/s | 82 kB 00:00 2020-06-12T02:36:01.558 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-fuse-2:15.2.3-196.g91d24f4.el8.x86_64 3/3 2020-06-12T02:36:01.559 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-fuse-2:15.2.3-196.g91d24f4.el8.x86_64 1/3 2020-06-12T02:36:01.559 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/3 2020-06-12T02:36:01.681 INFO:teuthology.orchestra.run.smithi188.stdout:(3/3): ceph-fuse-15.2.3-196.g91d24f4.el8.x86_64 2.6 MB/s | 574 kB 00:00 2020-06-12T02:36:01.681 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2020-06-12T02:36:01.682 INFO:teuthology.orchestra.run.smithi188.stdout:Total 3.0 MB/s | 677 kB 00:00 2020-06-12T02:36:01.685 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2020-06-12T02:36:01.705 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2020-06-12T02:36:01.705 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2020-06-12T02:36:01.768 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2020-06-12T02:36:01.769 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2020-06-12T02:36:01.840 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/3 2020-06-12T02:36:01.840 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:01.841 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2020-06-12T02:36:01.841 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse-2:15.2.3-196.g91d24f4.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2020-06-12T02:36:01.841 INFO:teuthology.orchestra.run.smithi107.stdout: fuse-common-3.2.1-12.el8.x86_64 2020-06-12T02:36:01.841 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:01.842 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2020-06-12T02:36:01.926 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2020-06-12T02:36:01.927 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 1/3 2020-06-12T02:36:01.927 INFO:teuthology.orchestra.run.smithi188.stdout:warning: /etc/fuse.conf created as /etc/fuse.conf.rpmnew 2020-06-12T02:36:01.927 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:02.077 INFO:teuthology.orchestra.run.smithi107:> sudo yum -y install librados-devel 2020-06-12T02:36:02.361 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : fuse-2.9.7-12.el8.x86_64 2/3 2020-06-12T02:36:02.490 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:37 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:36:02.736 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-fuse-2:15.2.3-196.g91d24f4.el8.x86_64 3/3 2020-06-12T02:36:02.899 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2020-06-12T02:36:02.900 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:36:02.900 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2020-06-12T02:36:02.900 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:36:02.901 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2020-06-12T02:36:02.901 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel x86_64 2:15.2.3-196.g91d24f4.el8 ceph 141 k 2020-06-12T02:36:02.901 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:02.901 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2020-06-12T02:36:02.901 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:36:02.902 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2020-06-12T02:36:02.902 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:02.902 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 141 k 2020-06-12T02:36:02.902 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 487 k 2020-06-12T02:36:02.902 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2020-06-12T02:36:03.038 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-fuse-2:15.2.3-196.g91d24f4.el8.x86_64 3/3 2020-06-12T02:36:03.038 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-fuse-2:15.2.3-196.g91d24f4.el8.x86_64 1/3 2020-06-12T02:36:03.038 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/3 2020-06-12T02:36:03.070 INFO:teuthology.orchestra.run.smithi107.stdout:librados-devel-15.2.3-196.g91d24f4.el8.x86_64.r 828 kB/s | 141 kB 00:00 2020-06-12T02:36:03.071 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2020-06-12T02:36:03.072 INFO:teuthology.orchestra.run.smithi107.stdout:Total 823 kB/s | 141 kB 00:00 2020-06-12T02:36:03.073 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2020-06-12T02:36:03.080 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2020-06-12T02:36:03.080 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2020-06-12T02:36:03.129 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2020-06-12T02:36:03.130 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2020-06-12T02:36:03.236 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2020-06-12T02:36:03.309 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/3 2020-06-12T02:36:03.309 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:03.309 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2020-06-12T02:36:03.310 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-fuse-2:15.2.3-196.g91d24f4.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2020-06-12T02:36:03.310 INFO:teuthology.orchestra.run.smithi188.stdout: fuse-common-3.2.1-12.el8.x86_64 2020-06-12T02:36:03.310 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:03.310 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2020-06-12T02:36:03.428 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librados-devel-2:15.2.3-196.g91d24f4.el8.x86_64 1/1 2020-06-12T02:36:03.586 INFO:teuthology.orchestra.run.smithi188:> sudo yum -y install librados-devel 2020-06-12T02:36:03.618 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados-devel-2:15.2.3-196.g91d24f4.el8.x86_64 1/1 2020-06-12T02:36:03.847 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados-devel-2:15.2.3-196.g91d24f4.el8.x86_64 1/1 2020-06-12T02:36:03.847 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:03.848 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2020-06-12T02:36:03.848 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:36:03.848 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:03.848 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2020-06-12T02:36:03.996 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:38 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:36:04.142 INFO:teuthology.orchestra.run.smithi107:> sudo yum -y install libcephfs2 2020-06-12T02:36:04.383 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2020-06-12T02:36:04.383 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:36:04.383 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repo Size 2020-06-12T02:36:04.383 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:36:04.384 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2020-06-12T02:36:04.384 INFO:teuthology.orchestra.run.smithi188.stdout: librados-devel x86_64 2:15.2.3-196.g91d24f4.el8 ceph 141 k 2020-06-12T02:36:04.384 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:04.384 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2020-06-12T02:36:04.384 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:36:04.384 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2020-06-12T02:36:04.385 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:04.385 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 141 k 2020-06-12T02:36:04.385 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 487 k 2020-06-12T02:36:04.385 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2020-06-12T02:36:04.545 INFO:teuthology.orchestra.run.smithi188.stdout:librados-devel-15.2.3-196.g91d24f4.el8.x86_64.r 835 kB/s | 141 kB 00:00 2020-06-12T02:36:04.546 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2020-06-12T02:36:04.547 INFO:teuthology.orchestra.run.smithi188.stdout:Total 830 kB/s | 141 kB 00:00 2020-06-12T02:36:04.548 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:39 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:36:04.548 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2020-06-12T02:36:04.555 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2020-06-12T02:36:04.555 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2020-06-12T02:36:04.603 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2020-06-12T02:36:04.604 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2020-06-12T02:36:04.714 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2020-06-12T02:36:04.944 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librados-devel-2:15.2.3-196.g91d24f4.el8.x86_64 1/1 2020-06-12T02:36:05.124 INFO:teuthology.orchestra.run.smithi107.stdout:Package libcephfs2-2:15.2.3-196.g91d24f4.el8.x86_64 is already installed. 2020-06-12T02:36:05.144 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librados-devel-2:15.2.3-196.g91d24f4.el8.x86_64 1/1 2020-06-12T02:36:05.231 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2020-06-12T02:36:05.232 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2020-06-12T02:36:05.232 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2020-06-12T02:36:05.370 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados-devel-2:15.2.3-196.g91d24f4.el8.x86_64 1/1 2020-06-12T02:36:05.370 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:05.371 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2020-06-12T02:36:05.371 INFO:teuthology.orchestra.run.smithi188.stdout: librados-devel-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:36:05.371 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:05.371 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2020-06-12T02:36:05.407 INFO:teuthology.orchestra.run.smithi107:> sudo yum -y install libcephfs-devel 2020-06-12T02:36:05.647 INFO:teuthology.orchestra.run.smithi188:> sudo yum -y install libcephfs2 2020-06-12T02:36:05.812 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:40 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:36:06.058 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:41 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:36:06.164 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2020-06-12T02:36:06.165 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:36:06.166 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2020-06-12T02:36:06.166 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:36:06.166 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2020-06-12T02:36:06.166 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs-devel x86_64 2:15.2.3-196.g91d24f4.el8 ceph 22 k 2020-06-12T02:36:06.167 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:06.167 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2020-06-12T02:36:06.167 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:36:06.167 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2020-06-12T02:36:06.167 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:06.168 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 22 k 2020-06-12T02:36:06.168 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 77 k 2020-06-12T02:36:06.168 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2020-06-12T02:36:06.290 INFO:teuthology.orchestra.run.smithi107.stdout:libcephfs-devel-15.2.3-196.g91d24f4.el8.x86_64. 177 kB/s | 22 kB 00:00 2020-06-12T02:36:06.290 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2020-06-12T02:36:06.290 INFO:teuthology.orchestra.run.smithi107.stdout:Total 176 kB/s | 22 kB 00:00 2020-06-12T02:36:06.292 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2020-06-12T02:36:06.296 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2020-06-12T02:36:06.296 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2020-06-12T02:36:06.298 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2020-06-12T02:36:06.298 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2020-06-12T02:36:06.338 INFO:teuthology.orchestra.run.smithi188.stdout:Package libcephfs2-2:15.2.3-196.g91d24f4.el8.x86_64 is already installed. 2020-06-12T02:36:06.343 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2020-06-12T02:36:06.444 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2020-06-12T02:36:06.445 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2020-06-12T02:36:06.445 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2020-06-12T02:36:06.552 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephfs-devel-2:15.2.3-196.g91d24f4.el8.x86_64 1/1 2020-06-12T02:36:06.574 INFO:teuthology.orchestra.run.smithi188:> sudo yum -y install libcephfs-devel 2020-06-12T02:36:06.718 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephfs-devel-2:15.2.3-196.g91d24f4.el8.x86_64 1/1 2020-06-12T02:36:06.929 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephfs-devel-2:15.2.3-196.g91d24f4.el8.x86_64 1/1 2020-06-12T02:36:06.930 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:06.930 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2020-06-12T02:36:06.930 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs-devel-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:36:06.930 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:06.930 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2020-06-12T02:36:06.975 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:41 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:36:07.060 INFO:teuthology.orchestra.run.smithi107:> sudo yum -y install librados2 2020-06-12T02:36:07.351 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2020-06-12T02:36:07.352 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:36:07.352 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repo Size 2020-06-12T02:36:07.352 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:36:07.353 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2020-06-12T02:36:07.353 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs-devel x86_64 2:15.2.3-196.g91d24f4.el8 ceph 22 k 2020-06-12T02:36:07.353 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:07.353 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2020-06-12T02:36:07.353 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:36:07.354 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2020-06-12T02:36:07.354 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:07.354 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 22 k 2020-06-12T02:36:07.354 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 77 k 2020-06-12T02:36:07.354 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2020-06-12T02:36:07.461 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:42 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:36:07.498 INFO:teuthology.orchestra.run.smithi188.stdout:libcephfs-devel-15.2.3-196.g91d24f4.el8.x86_64. 149 kB/s | 22 kB 00:00 2020-06-12T02:36:07.499 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2020-06-12T02:36:07.499 INFO:teuthology.orchestra.run.smithi188.stdout:Total 148 kB/s | 22 kB 00:00 2020-06-12T02:36:07.501 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2020-06-12T02:36:07.505 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2020-06-12T02:36:07.505 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2020-06-12T02:36:07.507 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2020-06-12T02:36:07.508 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2020-06-12T02:36:07.555 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2020-06-12T02:36:07.724 INFO:teuthology.orchestra.run.smithi107.stdout:Package librados2-2:15.2.3-196.g91d24f4.el8.x86_64 is already installed. 2020-06-12T02:36:07.750 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephfs-devel-2:15.2.3-196.g91d24f4.el8.x86_64 1/1 2020-06-12T02:36:07.830 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2020-06-12T02:36:07.831 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2020-06-12T02:36:07.831 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2020-06-12T02:36:07.919 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libcephfs-devel-2:15.2.3-196.g91d24f4.el8.x86_64 1/1 2020-06-12T02:36:07.981 INFO:teuthology.orchestra.run.smithi107:> sudo yum -y install librbd1 2020-06-12T02:36:08.127 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephfs-devel-2:15.2.3-196.g91d24f4.el8.x86_64 1/1 2020-06-12T02:36:08.127 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:08.127 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2020-06-12T02:36:08.128 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs-devel-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:36:08.128 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:08.128 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2020-06-12T02:36:08.382 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:43 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:36:08.641 INFO:teuthology.orchestra.run.smithi107.stdout:Package librbd1-2:15.2.3-196.g91d24f4.el8.x86_64 is already installed. 2020-06-12T02:36:08.746 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2020-06-12T02:36:08.747 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2020-06-12T02:36:08.748 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2020-06-12T02:36:08.850 INFO:teuthology.orchestra.run.smithi188:> sudo yum -y install librados2 2020-06-12T02:36:08.867 INFO:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rados 2020-06-12T02:36:09.258 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:44 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:36:09.270 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:44 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:36:09.504 INFO:teuthology.orchestra.run.smithi188.stdout:Package librados2-2:15.2.3-196.g91d24f4.el8.x86_64 is already installed. 2020-06-12T02:36:09.524 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-rados-2:15.2.3-196.g91d24f4.el8.x86_64 is already installed. 2020-06-12T02:36:09.609 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2020-06-12T02:36:09.610 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2020-06-12T02:36:09.610 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2020-06-12T02:36:09.629 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2020-06-12T02:36:09.630 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2020-06-12T02:36:09.631 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2020-06-12T02:36:09.755 INFO:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rgw 2020-06-12T02:36:09.758 INFO:teuthology.orchestra.run.smithi188:> sudo yum -y install librbd1 2020-06-12T02:36:10.156 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:45 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:36:10.160 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:45 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:36:10.420 INFO:teuthology.orchestra.run.smithi188.stdout:Package librbd1-2:15.2.3-196.g91d24f4.el8.x86_64 is already installed. 2020-06-12T02:36:10.456 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-rgw-2:15.2.3-196.g91d24f4.el8.x86_64 is already installed. 2020-06-12T02:36:10.529 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2020-06-12T02:36:10.530 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2020-06-12T02:36:10.530 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2020-06-12T02:36:10.562 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2020-06-12T02:36:10.563 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2020-06-12T02:36:10.563 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2020-06-12T02:36:10.652 INFO:teuthology.orchestra.run.smithi188:> sudo yum -y install python3-rados 2020-06-12T02:36:10.806 INFO:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-cephfs 2020-06-12T02:36:11.054 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:46 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:36:11.209 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:46 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:36:11.312 INFO:teuthology.orchestra.run.smithi188.stdout:Package python3-rados-2:15.2.3-196.g91d24f4.el8.x86_64 is already installed. 2020-06-12T02:36:11.418 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2020-06-12T02:36:11.419 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2020-06-12T02:36:11.420 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2020-06-12T02:36:11.465 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-cephfs-2:15.2.3-196.g91d24f4.el8.x86_64 is already installed. 2020-06-12T02:36:11.544 INFO:teuthology.orchestra.run.smithi188:> sudo yum -y install python3-rgw 2020-06-12T02:36:11.571 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2020-06-12T02:36:11.573 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2020-06-12T02:36:11.573 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2020-06-12T02:36:11.688 INFO:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rbd 2020-06-12T02:36:11.948 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:46 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:36:12.091 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:47 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:36:12.203 INFO:teuthology.orchestra.run.smithi188.stdout:Package python3-rgw-2:15.2.3-196.g91d24f4.el8.x86_64 is already installed. 2020-06-12T02:36:12.312 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2020-06-12T02:36:12.313 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2020-06-12T02:36:12.314 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2020-06-12T02:36:12.357 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-rbd-2:15.2.3-196.g91d24f4.el8.x86_64 is already installed. 2020-06-12T02:36:12.458 INFO:teuthology.orchestra.run.smithi188:> sudo yum -y install python3-cephfs 2020-06-12T02:36:12.468 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2020-06-12T02:36:12.469 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2020-06-12T02:36:12.469 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2020-06-12T02:36:12.607 INFO:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-fuse 2020-06-12T02:36:12.871 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:47 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:36:13.009 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:48 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:36:13.137 INFO:teuthology.orchestra.run.smithi188.stdout:Package python3-cephfs-2:15.2.3-196.g91d24f4.el8.x86_64 is already installed. 2020-06-12T02:36:13.244 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2020-06-12T02:36:13.244 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2020-06-12T02:36:13.245 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2020-06-12T02:36:13.363 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2020-06-12T02:36:13.364 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:36:13.364 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2020-06-12T02:36:13.364 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:36:13.364 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2020-06-12T02:36:13.364 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse x86_64 2:15.2.3-196.g91d24f4.el8 ceph 72 k 2020-06-12T02:36:13.365 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:13.365 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2020-06-12T02:36:13.365 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:36:13.365 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2020-06-12T02:36:13.365 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:13.366 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 72 k 2020-06-12T02:36:13.366 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 230 k 2020-06-12T02:36:13.366 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2020-06-12T02:36:13.374 INFO:teuthology.orchestra.run.smithi188:> sudo yum -y install python3-rbd 2020-06-12T02:36:13.510 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-fuse-15.2.3-196.g91d24f4.el8.x86_64.rpm 497 kB/s | 72 kB 00:00 2020-06-12T02:36:13.511 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2020-06-12T02:36:13.511 INFO:teuthology.orchestra.run.smithi107.stdout:Total 494 kB/s | 72 kB 00:00 2020-06-12T02:36:13.513 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2020-06-12T02:36:13.523 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2020-06-12T02:36:13.524 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2020-06-12T02:36:13.571 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2020-06-12T02:36:13.572 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2020-06-12T02:36:13.685 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2020-06-12T02:36:13.781 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:48 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:36:13.858 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-fuse-2:15.2.3-196.g91d24f4.el8.x86_64 1/1 2020-06-12T02:36:14.056 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-fuse-2:15.2.3-196.g91d24f4.el8.x86_64 1/1 2020-06-12T02:36:14.228 INFO:teuthology.orchestra.run.smithi188.stdout:Package python3-rbd-2:15.2.3-196.g91d24f4.el8.x86_64 is already installed. 2020-06-12T02:36:14.321 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-fuse-2:15.2.3-196.g91d24f4.el8.x86_64 1/1 2020-06-12T02:36:14.321 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:14.321 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2020-06-12T02:36:14.322 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:36:14.322 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:14.322 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2020-06-12T02:36:14.333 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2020-06-12T02:36:14.334 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2020-06-12T02:36:14.334 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2020-06-12T02:36:14.456 INFO:teuthology.orchestra.run.smithi188:> sudo yum -y install rbd-fuse 2020-06-12T02:36:14.598 INFO:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-mirror 2020-06-12T02:36:14.858 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:49 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:36:15.004 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:50 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:36:15.258 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2020-06-12T02:36:15.259 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:36:15.260 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2020-06-12T02:36:15.260 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:36:15.260 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2020-06-12T02:36:15.260 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-fuse x86_64 2:15.2.3-196.g91d24f4.el8 ceph 72 k 2020-06-12T02:36:15.260 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:15.261 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2020-06-12T02:36:15.261 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:36:15.261 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2020-06-12T02:36:15.261 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:15.261 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 72 k 2020-06-12T02:36:15.262 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 230 k 2020-06-12T02:36:15.262 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2020-06-12T02:36:15.398 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2020-06-12T02:36:15.399 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:36:15.399 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2020-06-12T02:36:15.400 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:36:15.400 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2020-06-12T02:36:15.400 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror x86_64 2:15.2.3-196.g91d24f4.el8 ceph 2.7 M 2020-06-12T02:36:15.400 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:15.400 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2020-06-12T02:36:15.401 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:36:15.401 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2020-06-12T02:36:15.401 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:15.401 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 2.7 M 2020-06-12T02:36:15.401 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 13 M 2020-06-12T02:36:15.402 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2020-06-12T02:36:15.406 INFO:teuthology.orchestra.run.smithi188.stdout:rbd-fuse-15.2.3-196.g91d24f4.el8.x86_64.rpm 495 kB/s | 72 kB 00:00 2020-06-12T02:36:15.407 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2020-06-12T02:36:15.407 INFO:teuthology.orchestra.run.smithi188.stdout:Total 491 kB/s | 72 kB 00:00 2020-06-12T02:36:15.409 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2020-06-12T02:36:15.420 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2020-06-12T02:36:15.421 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2020-06-12T02:36:15.468 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2020-06-12T02:36:15.469 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2020-06-12T02:36:15.579 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2020-06-12T02:36:15.700 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-mirror-15.2.3-196.g91d24f4.el8.x86_64.rpm 9.1 MB/s | 2.7 MB 00:00 2020-06-12T02:36:15.701 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2020-06-12T02:36:15.701 INFO:teuthology.orchestra.run.smithi107.stdout:Total 9.0 MB/s | 2.7 MB 00:00 2020-06-12T02:36:15.703 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2020-06-12T02:36:15.716 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2020-06-12T02:36:15.716 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2020-06-12T02:36:15.765 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-fuse-2:15.2.3-196.g91d24f4.el8.x86_64 1/1 2020-06-12T02:36:15.774 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2020-06-12T02:36:15.774 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2020-06-12T02:36:15.966 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: rbd-fuse-2:15.2.3-196.g91d24f4.el8.x86_64 1/1 2020-06-12T02:36:16.134 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2020-06-12T02:36:16.208 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-fuse-2:15.2.3-196.g91d24f4.el8.x86_64 1/1 2020-06-12T02:36:16.209 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:16.209 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2020-06-12T02:36:16.209 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-fuse-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:36:16.209 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:16.210 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2020-06-12T02:36:16.491 INFO:teuthology.orchestra.run.smithi188:> sudo yum -y install rbd-mirror 2020-06-12T02:36:16.898 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:51 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:36:17.111 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-mirror-2:15.2.3-196.g91d24f4.el8.x86_64 1/1 2020-06-12T02:36:17.290 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2020-06-12T02:36:17.291 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:36:17.291 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2020-06-12T02:36:17.292 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:36:17.292 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2020-06-12T02:36:17.292 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-mirror x86_64 2:15.2.3-196.g91d24f4.el8 ceph 2.7 M 2020-06-12T02:36:17.292 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:17.292 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2020-06-12T02:36:17.293 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:36:17.293 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2020-06-12T02:36:17.293 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:17.294 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 2.7 M 2020-06-12T02:36:17.294 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 13 M 2020-06-12T02:36:17.294 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2020-06-12T02:36:17.441 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-mirror-2:15.2.3-196.g91d24f4.el8.x86_64 1/1 2020-06-12T02:36:17.591 INFO:teuthology.orchestra.run.smithi188.stdout:rbd-mirror-15.2.3-196.g91d24f4.el8.x86_64.rpm 9.1 MB/s | 2.7 MB 00:00 2020-06-12T02:36:17.592 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2020-06-12T02:36:17.592 INFO:teuthology.orchestra.run.smithi188.stdout:Total 9.1 MB/s | 2.7 MB 00:00 2020-06-12T02:36:17.594 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2020-06-12T02:36:17.608 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2020-06-12T02:36:17.608 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2020-06-12T02:36:17.665 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2020-06-12T02:36:17.665 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2020-06-12T02:36:17.686 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-mirror-2:15.2.3-196.g91d24f4.el8.x86_64 1/1 2020-06-12T02:36:17.686 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:17.686 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2020-06-12T02:36:17.686 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:36:17.687 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:17.687 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2020-06-12T02:36:17.923 INFO:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-nbd 2020-06-12T02:36:18.018 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2020-06-12T02:36:18.208 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-mirror-2:15.2.3-196.g91d24f4.el8.x86_64 1/1 2020-06-12T02:36:18.330 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:53 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:36:18.522 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: rbd-mirror-2:15.2.3-196.g91d24f4.el8.x86_64 1/1 2020-06-12T02:36:18.696 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2020-06-12T02:36:18.697 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:36:18.697 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2020-06-12T02:36:18.698 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:36:18.698 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2020-06-12T02:36:18.698 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd x86_64 2:15.2.3-196.g91d24f4.el8 ceph 141 k 2020-06-12T02:36:18.698 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:18.698 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2020-06-12T02:36:18.699 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2020-06-12T02:36:18.699 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2020-06-12T02:36:18.699 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:18.699 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 141 k 2020-06-12T02:36:18.699 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 526 k 2020-06-12T02:36:18.699 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2020-06-12T02:36:18.758 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-mirror-2:15.2.3-196.g91d24f4.el8.x86_64 1/1 2020-06-12T02:36:18.759 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:18.759 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2020-06-12T02:36:18.759 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-mirror-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:36:18.759 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:18.760 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2020-06-12T02:36:18.872 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-nbd-15.2.3-196.g91d24f4.el8.x86_64.rpm 825 kB/s | 141 kB 00:00 2020-06-12T02:36:18.872 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2020-06-12T02:36:18.872 INFO:teuthology.orchestra.run.smithi107.stdout:Total 820 kB/s | 141 kB 00:00 2020-06-12T02:36:18.872 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2020-06-12T02:36:18.882 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2020-06-12T02:36:18.883 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2020-06-12T02:36:18.931 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2020-06-12T02:36:18.931 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2020-06-12T02:36:19.043 INFO:teuthology.orchestra.run.smithi188:> sudo yum -y install rbd-nbd 2020-06-12T02:36:19.056 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2020-06-12T02:36:19.240 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-nbd-2:15.2.3-196.g91d24f4.el8.x86_64 1/1 2020-06-12T02:36:19.427 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-nbd-2:15.2.3-196.g91d24f4.el8.x86_64 1/1 2020-06-12T02:36:19.450 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:54 ago on Fri 12 Jun 2020 02:34:25 AM UTC. 2020-06-12T02:36:19.701 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-nbd-2:15.2.3-196.g91d24f4.el8.x86_64 1/1 2020-06-12T02:36:19.702 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:19.702 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2020-06-12T02:36:19.703 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:36:19.703 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:19.703 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2020-06-12T02:36:19.981 DEBUG:teuthology.parallel:result is None 2020-06-12T02:36:20.194 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2020-06-12T02:36:20.195 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:36:20.195 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2020-06-12T02:36:20.196 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:36:20.196 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2020-06-12T02:36:20.196 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-nbd x86_64 2:15.2.3-196.g91d24f4.el8 ceph 141 k 2020-06-12T02:36:20.196 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:20.196 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2020-06-12T02:36:20.197 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2020-06-12T02:36:20.197 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2020-06-12T02:36:20.197 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:20.197 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 141 k 2020-06-12T02:36:20.197 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 526 k 2020-06-12T02:36:20.198 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2020-06-12T02:36:20.390 INFO:teuthology.orchestra.run.smithi188.stdout:rbd-nbd-15.2.3-196.g91d24f4.el8.x86_64.rpm 730 kB/s | 141 kB 00:00 2020-06-12T02:36:20.390 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2020-06-12T02:36:20.391 INFO:teuthology.orchestra.run.smithi188.stdout:Total 725 kB/s | 141 kB 00:00 2020-06-12T02:36:20.392 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2020-06-12T02:36:20.404 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2020-06-12T02:36:20.404 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2020-06-12T02:36:20.455 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2020-06-12T02:36:20.456 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2020-06-12T02:36:21.027 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2020-06-12T02:36:21.273 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-nbd-2:15.2.3-196.g91d24f4.el8.x86_64 1/1 2020-06-12T02:36:21.482 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: rbd-nbd-2:15.2.3-196.g91d24f4.el8.x86_64 1/1 2020-06-12T02:36:21.721 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-nbd-2:15.2.3-196.g91d24f4.el8.x86_64 1/1 2020-06-12T02:36:21.721 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:21.722 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2020-06-12T02:36:21.722 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-nbd-2:15.2.3-196.g91d24f4.el8.x86_64 2020-06-12T02:36:21.722 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:21.722 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2020-06-12T02:36:21.982 DEBUG:teuthology.parallel:result is None 2020-06-12T02:36:21.982 INFO:teuthology.orchestra.run.smithi107:> sudo lsb_release -is 2020-06-12T02:36:22.028 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS 2020-06-12T02:36:22.029 DEBUG:teuthology.misc:System to be installed: CentOS 2020-06-12T02:36:22.030 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=91d24f498556f86f6d1318d2efd5210c35b36762 2020-06-12T02:36:22.214 INFO:teuthology.orchestra.run.smithi107:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2020-06-12T02:36:22.237 INFO:teuthology.orchestra.run.smithi107.stdout:15.2.3-196.g91d24f4.el8 2020-06-12T02:36:22.239 INFO:teuthology.packaging:The installed version of ceph is 15.2.3-196.g91d24f4.el8 2020-06-12T02:36:22.239 INFO:teuthology.task.install:The correct ceph version 15.2.3-196.g91d24f4 is installed. 2020-06-12T02:36:22.240 INFO:teuthology.orchestra.run.smithi188:> sudo lsb_release -is 2020-06-12T02:36:22.278 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS 2020-06-12T02:36:22.280 DEBUG:teuthology.misc:System to be installed: CentOS 2020-06-12T02:36:22.281 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=91d24f498556f86f6d1318d2efd5210c35b36762 2020-06-12T02:36:22.461 INFO:teuthology.orchestra.run.smithi188:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2020-06-12T02:36:22.486 INFO:teuthology.orchestra.run.smithi188.stdout:15.2.3-196.g91d24f4.el8 2020-06-12T02:36:22.486 INFO:teuthology.packaging:The installed version of ceph is 15.2.3-196.g91d24f4.el8 2020-06-12T02:36:22.487 INFO:teuthology.task.install:The correct ceph version 15.2.3-196.g91d24f4 is installed. 2020-06-12T02:36:22.487 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2020-06-12T02:36:22.488 INFO:teuthology.orchestra.run.smithi107:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2020-06-12T02:36:22.546 INFO:teuthology.orchestra.run.smithi188:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2020-06-12T02:36:22.575 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2020-06-12T02:36:22.576 INFO:teuthology.orchestra.run.smithi107:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2020-06-12T02:36:22.618 INFO:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-06-12T02:36:22.690 INFO:teuthology.orchestra.run.smithi188:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2020-06-12T02:36:22.719 INFO:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-06-12T02:36:22.786 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2020-06-12T02:36:22.787 INFO:teuthology.orchestra.run.smithi107:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2020-06-12T02:36:22.836 INFO:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-06-12T02:36:22.908 INFO:teuthology.orchestra.run.smithi188:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2020-06-12T02:36:22.954 INFO:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-06-12T02:36:23.020 INFO:teuthology.run_tasks:Running task ceph... 2020-06-12T02:36:23.089 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2020-06-12T02:36:23.090 INFO:teuthology.orchestra.run.smithi107:> sudo chmod 777 /var/log/ceph 2020-06-12T02:36:23.093 INFO:teuthology.orchestra.run.smithi188:> sudo chmod 777 /var/log/ceph 2020-06-12T02:36:23.123 INFO:tasks.ceph:Disabling ceph logrotate... 2020-06-12T02:36:23.123 INFO:teuthology.orchestra.run.smithi107:> sudo rm -f -- /etc/logrotate.d/ceph 2020-06-12T02:36:23.164 INFO:teuthology.orchestra.run.smithi188:> sudo rm -f -- /etc/logrotate.d/ceph 2020-06-12T02:36:23.192 INFO:tasks.ceph:Creating extra log directories... 2020-06-12T02:36:23.193 INFO:teuthology.orchestra.run.smithi107:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-06-12T02:36:23.234 INFO:teuthology.orchestra.run.smithi188:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-06-12T02:36:23.262 INFO:tasks.ceph:Creating ceph cluster ceph... 2020-06-12T02:36:23.263 INFO:tasks.ceph:config {'conf': {'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000, 'osd pool default ec fast read': True, 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd heartbeat use min delay socket': True, 'osd max backfills': 2, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2020-06-12T02:36:23.263 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/kchai-2020-06-11_10:13:29-rados-octopus-distro-basic-smithi/5138043', 'branch': 'octopus', 'description': 'rados/thrash-erasure-code/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/fast.yaml msgr-failures/few.yaml objectstore/bluestore-comp-zstd.yaml rados.yaml recovery-overrides/{more-async-partial-recovery.yaml} supported-random-distro$/{centos_latest.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/ec-small-objects-many-deletes.yaml}', 'email': 'tchaikov@gmail.com', 'first_in_suite': False, 'job_id': '5138043', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'kchai-2020-06-11_10:13:29-rados-octopus-distro-basic-smithi', 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 30}}], 'os_type': 'centos', 'os_version': '8.1', 'overrides': {'admin_socket': {'branch': 'octopus'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000, 'osd pool default ec fast read': True, 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd heartbeat use min delay socket': True, 'osd max backfills': 2, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request'], 'sha1': '91d24f498556f86f6d1318d2efd5210c35b36762'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '91d24f498556f86f6d1318d2efd5210c35b36762'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'octopus-py3', 'sha1': '1e0d5d60071d5554f87a2be72ea0bcccb272aab3'}}, 'owner': 'scheduled_kchai@teuthology', 'priority': 90, 'repo': 'https://github.com/ceph/ceph.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'sha1': '91d24f498556f86f6d1318d2efd5210c35b36762', 'sleep_before_teardown': 0, 'suite': 'rados', 'suite_branch': 'octopus-py3', 'suite_path': '/home/teuthworker/src/github.com_tchaikov_ceph_octopus-py3/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/tchaikov/ceph.git', 'suite_sha1': '1e0d5d60071d5554f87a2be72ea0bcccb272aab3', 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 4, 'timeout': 1200}}, {'rados': {'clients': ['client.0'], 'ec_pool': True, 'max_in_flight': 8, 'max_seconds': 600, 'objects': 20, 'op_weights': {'append': 10, 'delete': 20, 'read': 0, 'write': 0}, 'ops': 400000, 'size': 16384}}], 'teuthology_branch': 'master', 'tube': 'smithi', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.5990', 'targets': {'smithi107.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDCX7fcNNixx1ENGx6xAbkp5L/s+PCWpsuZl3W/iLRXFzuXqOYn77VdRz1WRVag9Ve3HnpzwFAYD79a4d0zD3jNM/jI4SnkGl3koEVEe4DxDSq4ahaBdQw8Wv6HdPsDrs0gT4XhjcV7BL/7evMctHKg/KF8Qh7ZpuoffcMzlCh8XO+TK9a5noZUD1tbPqm+g53kw3tqZtuQpYUh/dLuI/d/dzbpo0A6k3UqXQDusAFLQG51NN3KpmxCSYKgv03X/5Jef1TdFeRKX2pEZDnw3lKt8tPAIo9598wagPCeL0Q3xkNvPmaH+0AxOo2GhPS8pRsafkwFt9PTi3vhKgdXlYhkWNM00/b5rIsSpXeuZP1KCq+fnmuxvyEoIHZkTAjioJ3NBLvgRut8A+3g9S6FFppbmdUzrBr4CO+eaB3Cx18EO+0IEc9el/XlgjIhNQnkKrg2oPNBlFkQAamqnQzpY3wusAaQXBkL/mtXwp6fDAsVLNZPB5FWN0XwzfaojtiSv3k=', 'smithi188.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMj9WgFQM40so9VyWfiGhLrFUU1Ttz17IkSiY6OPQk+6CuHXLgcViAqQLjJNEyWSh7YmMmYwuzlBKA3ckVs2XDDqf67/eU7KN/go/Zxy/DZOjpuJg/X7RWeyr/VDmdP3eUi5i7iGu1/+xJowqhj5yjZq8b3PgkWfBjLw3YxUv8kTvfUEKf45z+6tGJYhDbr0BcWpMnDg3KX7mp2Zcy//p58JU0fNTRKB1LLmyViDDQWdPoK0hMKfjmy3BoX990kDJORhW7ak935HAhE0dzmZwwB3ZjjSjSVPvSjhX86YY/Aq9hShMhSxdzfqgalrmuJDZG2gF+/pDizPBrEMIF2YFj/A8331EEru73yOLLKVaA4h7tDjyk+i3RqxPjPyUMkX79DZwfoMGs6G6Zw8rnhnJq/NaFnxSS5rkNCoev1sEQZr1Tbb6qarNlVx+T9Z7XEP0/B8SdbX/GR+eyyAQmOf++RdfYgvUA9IutrDyK8gI3s3aCQFesO6ElnUmRD/FKYVc='}} 2020-06-12T02:36:23.264 INFO:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2020-06-12T02:36:23.304 INFO:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2020-06-12T02:36:23.320 INFO:teuthology.orchestra.run.smithi107:> sudo install -d -m0777 -- /var/run/ceph 2020-06-12T02:36:23.359 INFO:teuthology.orchestra.run.smithi188:> sudo install -d -m0777 -- /var/run/ceph 2020-06-12T02:36:23.444 DEBUG:teuthology.orchestra.remote:smithi107:/scratch_devs is 72B 2020-06-12T02:36:23.492 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2020-06-12T02:36:23.493 INFO:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-06-12T02:36:23.507 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2020-06-12T02:36:23.507 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-06-12T02:36:23.507 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 6h/6d Inode: 94048 Links: 1 2020-06-12T02:36:23.508 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-06-12T02:36:23.508 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2020-06-12T02:36:23.508 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2020-06-12 02:36:17.277822446 +0000 2020-06-12T02:36:23.508 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2020-06-12 02:34:05.407507029 +0000 2020-06-12T02:36:23.508 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2020-06-12 02:34:05.407507029 +0000 2020-06-12T02:36:23.509 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: - 2020-06-12T02:36:23.522 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2020-06-12T02:36:23.522 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2020-06-12T02:36:23.523 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000294045 s, 1.7 MB/s 2020-06-12T02:36:23.524 INFO:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-06-12T02:36:23.580 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2020-06-12T02:36:23.581 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-06-12T02:36:23.581 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 6h/6d Inode: 94639 Links: 1 2020-06-12T02:36:23.581 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-06-12T02:36:23.581 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2020-06-12T02:36:23.582 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2020-06-12 02:36:17.277822446 +0000 2020-06-12T02:36:23.582 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2020-06-12 02:34:05.745501591 +0000 2020-06-12T02:36:23.582 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2020-06-12 02:34:05.745501591 +0000 2020-06-12T02:36:23.582 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: - 2020-06-12T02:36:23.591 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2020-06-12T02:36:23.592 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2020-06-12T02:36:23.592 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000202518 s, 2.5 MB/s 2020-06-12T02:36:23.594 INFO:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-06-12T02:36:23.648 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-06-12T02:36:23.648 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-06-12T02:36:23.649 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 6h/6d Inode: 97346 Links: 1 2020-06-12T02:36:23.649 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-06-12T02:36:23.649 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2020-06-12T02:36:23.649 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2020-06-12 02:36:17.277822446 +0000 2020-06-12T02:36:23.649 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2020-06-12 02:34:06.101495863 +0000 2020-06-12T02:36:23.650 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2020-06-12 02:34:06.101495863 +0000 2020-06-12T02:36:23.650 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: - 2020-06-12T02:36:23.661 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2020-06-12T02:36:23.662 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2020-06-12T02:36:23.662 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.00020015 s, 2.6 MB/s 2020-06-12T02:36:23.664 INFO:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-06-12T02:36:23.718 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2020-06-12T02:36:23.718 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-06-12T02:36:23.719 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 6h/6d Inode: 95476 Links: 1 2020-06-12T02:36:23.719 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-06-12T02:36:23.719 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2020-06-12T02:36:23.719 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2020-06-12 02:36:17.278822431 +0000 2020-06-12T02:36:23.719 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2020-06-12 02:34:06.434490506 +0000 2020-06-12T02:36:23.720 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2020-06-12 02:34:06.434490506 +0000 2020-06-12T02:36:23.720 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: - 2020-06-12T02:36:23.731 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2020-06-12T02:36:23.732 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2020-06-12T02:36:23.732 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000224373 s, 2.3 MB/s 2020-06-12T02:36:23.733 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2020-06-12T02:36:23.751 DEBUG:teuthology.orchestra.remote:smithi188:/scratch_devs is 72B 2020-06-12T02:36:23.800 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2020-06-12T02:36:23.801 INFO:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-06-12T02:36:23.815 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2020-06-12T02:36:23.815 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-06-12T02:36:23.816 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 6h/6d Inode: 91841 Links: 1 2020-06-12T02:36:23.816 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-06-12T02:36:23.816 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2020-06-12T02:36:23.816 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2020-06-12 02:36:18.359333349 +0000 2020-06-12T02:36:23.816 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2020-06-12 02:34:05.409888434 +0000 2020-06-12T02:36:23.817 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2020-06-12 02:34:05.409888434 +0000 2020-06-12T02:36:23.817 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: - 2020-06-12T02:36:23.828 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2020-06-12T02:36:23.828 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2020-06-12T02:36:23.829 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000232619 s, 2.2 MB/s 2020-06-12T02:36:23.831 INFO:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-06-12T02:36:23.885 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2020-06-12T02:36:23.886 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-06-12T02:36:23.886 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 6h/6d Inode: 86908 Links: 1 2020-06-12T02:36:23.886 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-06-12T02:36:23.887 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2020-06-12T02:36:23.887 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2020-06-12 02:36:18.359333349 +0000 2020-06-12T02:36:23.887 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2020-06-12 02:34:05.775881810 +0000 2020-06-12T02:36:23.887 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2020-06-12 02:34:05.775881810 +0000 2020-06-12T02:36:23.887 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: - 2020-06-12T02:36:23.898 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2020-06-12T02:36:23.899 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2020-06-12T02:36:23.899 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000235341 s, 2.2 MB/s 2020-06-12T02:36:23.901 INFO:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-06-12T02:36:23.955 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-06-12T02:36:23.956 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-06-12T02:36:23.956 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 6h/6d Inode: 95529 Links: 1 2020-06-12T02:36:23.956 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-06-12T02:36:23.956 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2020-06-12T02:36:23.957 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2020-06-12 02:36:18.359333349 +0000 2020-06-12T02:36:23.957 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2020-06-12 02:34:06.123875511 +0000 2020-06-12T02:36:23.957 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2020-06-12 02:34:06.123875511 +0000 2020-06-12T02:36:23.957 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: - 2020-06-12T02:36:23.968 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2020-06-12T02:36:23.969 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2020-06-12T02:36:23.969 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000208973 s, 2.5 MB/s 2020-06-12T02:36:23.971 INFO:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-06-12T02:36:24.032 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2020-06-12T02:36:24.033 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-06-12T02:36:24.033 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 6h/6d Inode: 97352 Links: 1 2020-06-12T02:36:24.033 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-06-12T02:36:24.034 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2020-06-12T02:36:24.034 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2020-06-12 02:36:18.359333349 +0000 2020-06-12T02:36:24.034 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2020-06-12 02:34:06.460869411 +0000 2020-06-12T02:36:24.034 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2020-06-12 02:34:06.460869411 +0000 2020-06-12T02:36:24.034 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: - 2020-06-12T02:36:24.043 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2020-06-12T02:36:24.044 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2020-06-12T02:36:24.044 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000181449 s, 2.8 MB/s 2020-06-12T02:36:24.044 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2020-06-12T02:36:24.045 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi107.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi188.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2020-06-12T02:36:24.045 INFO:tasks.ceph:Generating config... 2020-06-12T02:36:24.047 INFO:tasks.ceph:[global] mon client directed command retry = 5 2020-06-12T02:36:24.047 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2020-06-12T02:36:24.048 INFO:tasks.ceph:[global] osd pool default ec fast read = True 2020-06-12T02:36:24.048 INFO:tasks.ceph:[global] osd_async_recovery_min_cost = 1 2020-06-12T02:36:24.048 INFO:tasks.ceph:[global] osd_object_clean_region_max_num_intervals = 1000 2020-06-12T02:36:24.049 INFO:tasks.ceph:[mgr] debug mgr = 20 2020-06-12T02:36:24.049 INFO:tasks.ceph:[mgr] debug ms = 1 2020-06-12T02:36:24.049 INFO:tasks.ceph:[mon] debug mon = 20 2020-06-12T02:36:24.050 INFO:tasks.ceph:[mon] debug ms = 1 2020-06-12T02:36:24.050 INFO:tasks.ceph:[mon] debug paxos = 20 2020-06-12T02:36:24.050 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2020-06-12T02:36:24.051 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2020-06-12T02:36:24.051 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2020-06-12T02:36:24.051 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2020-06-12T02:36:24.051 INFO:tasks.ceph:[osd] debug bluefs = 20 2020-06-12T02:36:24.052 INFO:tasks.ceph:[osd] debug bluestore = 20 2020-06-12T02:36:24.052 INFO:tasks.ceph:[osd] debug filestore = 20 2020-06-12T02:36:24.053 INFO:tasks.ceph:[osd] debug journal = 20 2020-06-12T02:36:24.053 INFO:tasks.ceph:[osd] debug ms = 20 2020-06-12T02:36:24.054 INFO:tasks.ceph:[osd] debug osd = 25 2020-06-12T02:36:24.054 INFO:tasks.ceph:[osd] debug rocksdb = 10 2020-06-12T02:36:24.054 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2020-06-12T02:36:24.055 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2020-06-12T02:36:24.055 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2020-06-12T02:36:24.055 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.1 2020-06-12T02:36:24.056 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2020-06-12T02:36:24.056 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2020-06-12T02:36:24.056 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2020-06-12T02:36:24.057 INFO:tasks.ceph:[osd] osd heartbeat use min delay socket = True 2020-06-12T02:36:24.057 INFO:tasks.ceph:[osd] osd max backfills = 2 2020-06-12T02:36:24.057 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2020-06-12T02:36:24.058 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2020-06-12T02:36:24.058 INFO:tasks.ceph:[osd] osd op queue = debug_random 2020-06-12T02:36:24.059 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2020-06-12T02:36:24.059 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2020-06-12T02:36:24.059 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2020-06-12T02:36:24.060 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2020-06-12T02:36:24.060 INFO:tasks.ceph:Setting up mon.a... 2020-06-12T02:36:24.061 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2020-06-12T02:36:24.109 INFO:teuthology.orchestra.run.smithi107.stdout:creating /etc/ceph/ceph.keyring 2020-06-12T02:36:24.111 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2020-06-12T02:36:24.153 INFO:teuthology.orchestra.run.smithi107:> sudo chmod 0644 /etc/ceph/ceph.keyring 2020-06-12T02:36:24.219 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.107'), ('mon.c', '[v2:172.21.15.107:3301,v1:172.21.15.107:6790]'), ('mon.b', '172.21.15.188')] 2020-06-12T02:36:24.220 INFO:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --add a 172.21.15.107 --addv c '[v2:172.21.15.107:3301,v1:172.21.15.107:6790]' --add b 172.21.15.188 --print /home/ubuntu/cephtest/ceph.monmap 2020-06-12T02:36:24.297 INFO:teuthology.orchestra.run.smithi107.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2020-06-12T02:36:24.297 INFO:teuthology.orchestra.run.smithi107.stdout:monmaptool: generated fsid 2f5030a1-3206-43c9-8471-d10429688d36 2020-06-12T02:36:24.297 INFO:teuthology.orchestra.run.smithi107.stdout:epoch 0 2020-06-12T02:36:24.298 INFO:teuthology.orchestra.run.smithi107.stdout:fsid 2f5030a1-3206-43c9-8471-d10429688d36 2020-06-12T02:36:24.298 INFO:teuthology.orchestra.run.smithi107.stdout:last_changed 2020-06-12T02:36:24.295197+0000 2020-06-12T02:36:24.298 INFO:teuthology.orchestra.run.smithi107.stdout:created 2020-06-12T02:36:24.295197+0000 2020-06-12T02:36:24.298 INFO:teuthology.orchestra.run.smithi107.stdout:min_mon_release 0 (unknown) 2020-06-12T02:36:24.298 INFO:teuthology.orchestra.run.smithi107.stdout:0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2020-06-12T02:36:24.299 INFO:teuthology.orchestra.run.smithi107.stdout:1: [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] mon.b 2020-06-12T02:36:24.299 INFO:teuthology.orchestra.run.smithi107.stdout:2: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.c 2020-06-12T02:36:24.299 INFO:teuthology.orchestra.run.smithi107.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2020-06-12T02:36:24.300 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 2f5030a1-3206-43c9-8471-d10429688d36... 2020-06-12T02:36:24.302 INFO:teuthology.orchestra.run.smithi107:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-06-12T02:36:24.343 INFO:teuthology.orchestra.run.smithi188:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-06-12T02:36:24.396 INFO:teuthology.orchestra.run.smithi107.stdout:[global] 2020-06-12T02:36:24.397 INFO:teuthology.orchestra.run.smithi107.stdout: chdir = "" 2020-06-12T02:36:24.397 INFO:teuthology.orchestra.run.smithi107.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-06-12T02:36:24.397 INFO:teuthology.orchestra.run.smithi107.stdout: auth supported = cephx 2020-06-12T02:36:24.397 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:24.398 INFO:teuthology.orchestra.run.smithi107.stdout: filestore xattr use omap = true 2020-06-12T02:36:24.398 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:24.398 INFO:teuthology.orchestra.run.smithi107.stdout: mon clock drift allowed = 1.000 2020-06-12T02:36:24.398 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:24.398 INFO:teuthology.orchestra.run.smithi107.stdout: osd crush chooseleaf type = 0 2020-06-12T02:36:24.399 INFO:teuthology.orchestra.run.smithi107.stdout: auth debug = true 2020-06-12T02:36:24.399 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:24.399 INFO:teuthology.orchestra.run.smithi107.stdout: ms die on old message = true 2020-06-12T02:36:24.400 INFO:teuthology.orchestra.run.smithi107.stdout: ms die on bug = true 2020-06-12T02:36:24.401 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:24.401 INFO:teuthology.orchestra.run.smithi107.stdout: mon max pg per osd = 10000 # >= luminous 2020-06-12T02:36:24.401 INFO:teuthology.orchestra.run.smithi107.stdout: mon pg warn max object skew = 0 2020-06-12T02:36:24.401 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:24.402 INFO:teuthology.orchestra.run.smithi107.stdout: # disable pg_autoscaler by default for new pools 2020-06-12T02:36:24.402 INFO:teuthology.orchestra.run.smithi107.stdout: osd_pool_default_pg_autoscale_mode = off 2020-06-12T02:36:24.402 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:24.402 INFO:teuthology.orchestra.run.smithi107.stdout: osd pool default size = 2 2020-06-12T02:36:24.402 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:24.403 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd allow primary affinity = true 2020-06-12T02:36:24.403 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd allow pg remap = true 2020-06-12T02:36:24.403 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on legacy crush tunables = false 2020-06-12T02:36:24.403 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on crush straw calc version zero = false 2020-06-12T02:36:24.403 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on no sortbitwise = false 2020-06-12T02:36:24.403 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on osd down out interval zero = false 2020-06-12T02:36:24.404 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on too few osds = false 2020-06-12T02:36:24.404 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-06-12T02:36:24.404 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_pool_no_redundancy = false 2020-06-12T02:36:24.404 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:24.404 INFO:teuthology.orchestra.run.smithi107.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-06-12T02:36:24.405 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:24.405 INFO:teuthology.orchestra.run.smithi107.stdout: osd default data pool replay window = 5 2020-06-12T02:36:24.405 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:24.405 INFO:teuthology.orchestra.run.smithi107.stdout: mon allow pool delete = true 2020-06-12T02:36:24.405 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:24.406 INFO:teuthology.orchestra.run.smithi107.stdout: mon cluster log file level = debug 2020-06-12T02:36:24.406 INFO:teuthology.orchestra.run.smithi107.stdout: debug asserts on shutdown = true 2020-06-12T02:36:24.406 INFO:teuthology.orchestra.run.smithi107.stdout: mon host = "172.21.15.107,[v2:172.21.15.107:3301,v1:172.21.15.107:6790],172.21.15.188" 2020-06-12T02:36:24.406 INFO:teuthology.orchestra.run.smithi107.stdout: mon client directed command retry = 5 2020-06-12T02:36:24.406 INFO:teuthology.orchestra.run.smithi107.stdout: ms inject socket failures = 5000 2020-06-12T02:36:24.407 INFO:teuthology.orchestra.run.smithi107.stdout: osd pool default ec fast read = True 2020-06-12T02:36:24.407 INFO:teuthology.orchestra.run.smithi107.stdout: osd_async_recovery_min_cost = 1 2020-06-12T02:36:24.407 INFO:teuthology.orchestra.run.smithi107.stdout: osd_object_clean_region_max_num_intervals = 1000 2020-06-12T02:36:24.407 INFO:teuthology.orchestra.run.smithi107.stdout: fsid = 2f5030a1-3206-43c9-8471-d10429688d36 2020-06-12T02:36:24.408 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:24.408 INFO:teuthology.orchestra.run.smithi107.stdout:[osd] 2020-06-12T02:36:24.408 INFO:teuthology.orchestra.run.smithi107.stdout: osd journal size = 100 2020-06-12T02:36:24.408 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:24.408 INFO:teuthology.orchestra.run.smithi107.stdout: osd scrub load threshold = 5.0 2020-06-12T02:36:24.408 INFO:teuthology.orchestra.run.smithi107.stdout: osd scrub max interval = 120 2020-06-12T02:36:24.409 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:24.409 INFO:teuthology.orchestra.run.smithi107.stdout: osd recover clone overlap = true 2020-06-12T02:36:24.409 INFO:teuthology.orchestra.run.smithi107.stdout: osd recovery max chunk = 1048576 2020-06-12T02:36:24.409 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:24.409 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug shutdown = true 2020-06-12T02:36:24.410 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug op order = true 2020-06-12T02:36:24.410 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug verify stray on activate = true 2020-06-12T02:36:24.410 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:24.410 INFO:teuthology.orchestra.run.smithi107.stdout: osd open classes on start = true 2020-06-12T02:36:24.410 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug pg log writeout = true 2020-06-12T02:36:24.411 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:24.411 INFO:teuthology.orchestra.run.smithi107.stdout: osd deep scrub update digest min age = 30 2020-06-12T02:36:24.411 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:24.411 INFO:teuthology.orchestra.run.smithi107.stdout: osd map max advance = 10 2020-06-12T02:36:24.412 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:24.412 INFO:teuthology.orchestra.run.smithi107.stdout: journal zero on create = true 2020-06-12T02:36:24.412 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:24.412 INFO:teuthology.orchestra.run.smithi107.stdout: filestore ondisk finisher threads = 3 2020-06-12T02:36:24.412 INFO:teuthology.orchestra.run.smithi107.stdout: filestore apply finisher threads = 3 2020-06-12T02:36:24.413 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:24.413 INFO:teuthology.orchestra.run.smithi107.stdout: bdev debug aio = true 2020-06-12T02:36:24.413 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug misdirected ops = true 2020-06-12T02:36:24.413 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore block size = 96636764160 2020-06-12T02:36:24.413 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore compression algorithm = zstd 2020-06-12T02:36:24.413 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore compression mode = aggressive 2020-06-12T02:36:24.414 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore fsck on mount = True 2020-06-12T02:36:24.414 INFO:teuthology.orchestra.run.smithi107.stdout: debug bluefs = 20 2020-06-12T02:36:24.414 INFO:teuthology.orchestra.run.smithi107.stdout: debug bluestore = 20 2020-06-12T02:36:24.414 INFO:teuthology.orchestra.run.smithi107.stdout: debug filestore = 20 2020-06-12T02:36:24.414 INFO:teuthology.orchestra.run.smithi107.stdout: debug journal = 20 2020-06-12T02:36:24.415 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 20 2020-06-12T02:36:24.415 INFO:teuthology.orchestra.run.smithi107.stdout: debug osd = 25 2020-06-12T02:36:24.415 INFO:teuthology.orchestra.run.smithi107.stdout: debug rocksdb = 10 2020-06-12T02:36:24.416 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd backfillfull_ratio = 0.85 2020-06-12T02:36:24.416 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd full ratio = 0.9 2020-06-12T02:36:24.416 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd nearfull ratio = 0.8 2020-06-12T02:36:24.416 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug reject backfill probability = 0.1 2020-06-12T02:36:24.416 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug verify cached snaps = True 2020-06-12T02:36:24.417 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug verify missing on start = True 2020-06-12T02:36:24.417 INFO:teuthology.orchestra.run.smithi107.stdout: osd failsafe full ratio = 0.95 2020-06-12T02:36:24.417 INFO:teuthology.orchestra.run.smithi107.stdout: osd heartbeat use min delay socket = True 2020-06-12T02:36:24.417 INFO:teuthology.orchestra.run.smithi107.stdout: osd max backfills = 2 2020-06-12T02:36:24.417 INFO:teuthology.orchestra.run.smithi107.stdout: osd max markdown count = 1000 2020-06-12T02:36:24.418 INFO:teuthology.orchestra.run.smithi107.stdout: osd objectstore = bluestore 2020-06-12T02:36:24.418 INFO:teuthology.orchestra.run.smithi107.stdout: osd op queue = debug_random 2020-06-12T02:36:24.418 INFO:teuthology.orchestra.run.smithi107.stdout: osd op queue cut off = debug_random 2020-06-12T02:36:24.418 INFO:teuthology.orchestra.run.smithi107.stdout: osd scrub min interval = 60 2020-06-12T02:36:24.418 INFO:teuthology.orchestra.run.smithi107.stdout: osd shutdown pgref assert = True 2020-06-12T02:36:24.419 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:24.419 INFO:teuthology.orchestra.run.smithi107.stdout:[mgr] 2020-06-12T02:36:24.419 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2020-06-12T02:36:24.419 INFO:teuthology.orchestra.run.smithi107.stdout: debug mgr = 20 2020-06-12T02:36:24.419 INFO:teuthology.orchestra.run.smithi107.stdout: debug mon = 20 2020-06-12T02:36:24.419 INFO:teuthology.orchestra.run.smithi107.stdout: debug auth = 20 2020-06-12T02:36:24.420 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min pgs per osd = 4 2020-06-12T02:36:24.420 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min bytes per osd = 10 2020-06-12T02:36:24.420 INFO:teuthology.orchestra.run.smithi107.stdout: mgr/telemetry/nag = false 2020-06-12T02:36:24.420 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:24.420 INFO:teuthology.orchestra.run.smithi107.stdout:[mon] 2020-06-12T02:36:24.421 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2020-06-12T02:36:24.421 INFO:teuthology.orchestra.run.smithi107.stdout: debug mon = 20 2020-06-12T02:36:24.421 INFO:teuthology.orchestra.run.smithi107.stdout: debug paxos = 20 2020-06-12T02:36:24.421 INFO:teuthology.orchestra.run.smithi107.stdout: debug auth = 20 2020-06-12T02:36:24.422 INFO:teuthology.orchestra.run.smithi107.stdout: mon data avail warn = 5 2020-06-12T02:36:24.422 INFO:teuthology.orchestra.run.smithi107.stdout: mon mgr mkfs grace = 240 2020-06-12T02:36:24.422 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min pgs per osd = 4 2020-06-12T02:36:24.422 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd reporter subtree level = osd 2020-06-12T02:36:24.422 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd prime pg temp = true 2020-06-12T02:36:24.423 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min bytes per osd = 10 2020-06-12T02:36:24.423 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:36:24.423 INFO:teuthology.orchestra.run.smithi107.stdout:[client] 2020-06-12T02:36:24.423 INFO:teuthology.orchestra.run.smithi107.stdout: rgw cache enabled = true 2020-06-12T02:36:24.424 INFO:teuthology.orchestra.run.smithi107.stdout: rgw enable ops log = true 2020-06-12T02:36:24.424 INFO:teuthology.orchestra.run.smithi107.stdout: rgw enable usage log = true 2020-06-12T02:36:24.424 INFO:teuthology.orchestra.run.smithi107.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-06-12T02:36:24.425 INFO:teuthology.orchestra.run.smithi107.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-06-12T02:36:24.425 INFO:teuthology.orchestra.run.smithi107.stdout:[mon.a] 2020-06-12T02:36:24.426 INFO:teuthology.orchestra.run.smithi107.stdout:[mon.c] 2020-06-12T02:36:24.426 INFO:teuthology.orchestra.run.smithi107.stdout:[mon.b] 2020-06-12T02:36:24.427 INFO:teuthology.orchestra.run.smithi188.stdout:[global] 2020-06-12T02:36:24.427 INFO:teuthology.orchestra.run.smithi188.stdout: chdir = "" 2020-06-12T02:36:24.428 INFO:teuthology.orchestra.run.smithi188.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-06-12T02:36:24.428 INFO:teuthology.orchestra.run.smithi188.stdout: auth supported = cephx 2020-06-12T02:36:24.428 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:24.428 INFO:teuthology.orchestra.run.smithi188.stdout: filestore xattr use omap = true 2020-06-12T02:36:24.428 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:24.429 INFO:teuthology.orchestra.run.smithi188.stdout: mon clock drift allowed = 1.000 2020-06-12T02:36:24.429 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:24.429 INFO:teuthology.orchestra.run.smithi188.stdout: osd crush chooseleaf type = 0 2020-06-12T02:36:24.429 INFO:teuthology.orchestra.run.smithi188.stdout: auth debug = true 2020-06-12T02:36:24.429 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:24.430 INFO:teuthology.orchestra.run.smithi188.stdout: ms die on old message = true 2020-06-12T02:36:24.430 INFO:teuthology.orchestra.run.smithi188.stdout: ms die on bug = true 2020-06-12T02:36:24.430 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:24.430 INFO:teuthology.orchestra.run.smithi188.stdout: mon max pg per osd = 10000 # >= luminous 2020-06-12T02:36:24.430 INFO:teuthology.orchestra.run.smithi188.stdout: mon pg warn max object skew = 0 2020-06-12T02:36:24.431 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:24.431 INFO:teuthology.orchestra.run.smithi188.stdout: # disable pg_autoscaler by default for new pools 2020-06-12T02:36:24.431 INFO:teuthology.orchestra.run.smithi188.stdout: osd_pool_default_pg_autoscale_mode = off 2020-06-12T02:36:24.433 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:24.433 INFO:teuthology.orchestra.run.smithi188.stdout: osd pool default size = 2 2020-06-12T02:36:24.433 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:24.433 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd allow primary affinity = true 2020-06-12T02:36:24.433 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd allow pg remap = true 2020-06-12T02:36:24.434 INFO:teuthology.orchestra.run.smithi188.stdout: mon warn on legacy crush tunables = false 2020-06-12T02:36:24.434 INFO:teuthology.orchestra.run.smithi188.stdout: mon warn on crush straw calc version zero = false 2020-06-12T02:36:24.434 INFO:teuthology.orchestra.run.smithi188.stdout: mon warn on no sortbitwise = false 2020-06-12T02:36:24.434 INFO:teuthology.orchestra.run.smithi188.stdout: mon warn on osd down out interval zero = false 2020-06-12T02:36:24.434 INFO:teuthology.orchestra.run.smithi188.stdout: mon warn on too few osds = false 2020-06-12T02:36:24.435 INFO:teuthology.orchestra.run.smithi188.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-06-12T02:36:24.435 INFO:teuthology.orchestra.run.smithi188.stdout: mon_warn_on_pool_no_redundancy = false 2020-06-12T02:36:24.435 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:24.435 INFO:teuthology.orchestra.run.smithi188.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-06-12T02:36:24.435 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:24.436 INFO:teuthology.orchestra.run.smithi188.stdout: osd default data pool replay window = 5 2020-06-12T02:36:24.436 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:24.436 INFO:teuthology.orchestra.run.smithi188.stdout: mon allow pool delete = true 2020-06-12T02:36:24.436 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:24.436 INFO:teuthology.orchestra.run.smithi188.stdout: mon cluster log file level = debug 2020-06-12T02:36:24.436 INFO:teuthology.orchestra.run.smithi188.stdout: debug asserts on shutdown = true 2020-06-12T02:36:24.437 INFO:teuthology.orchestra.run.smithi188.stdout: mon host = "172.21.15.107,[v2:172.21.15.107:3301,v1:172.21.15.107:6790],172.21.15.188" 2020-06-12T02:36:24.437 INFO:teuthology.orchestra.run.smithi188.stdout: mon client directed command retry = 5 2020-06-12T02:36:24.437 INFO:teuthology.orchestra.run.smithi188.stdout: ms inject socket failures = 5000 2020-06-12T02:36:24.437 INFO:teuthology.orchestra.run.smithi188.stdout: osd pool default ec fast read = True 2020-06-12T02:36:24.437 INFO:teuthology.orchestra.run.smithi188.stdout: osd_async_recovery_min_cost = 1 2020-06-12T02:36:24.438 INFO:teuthology.orchestra.run.smithi188.stdout: osd_object_clean_region_max_num_intervals = 1000 2020-06-12T02:36:24.438 INFO:teuthology.orchestra.run.smithi188.stdout: fsid = 2f5030a1-3206-43c9-8471-d10429688d36 2020-06-12T02:36:24.438 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:24.438 INFO:teuthology.orchestra.run.smithi188.stdout:[osd] 2020-06-12T02:36:24.438 INFO:teuthology.orchestra.run.smithi188.stdout: osd journal size = 100 2020-06-12T02:36:24.439 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:24.439 INFO:teuthology.orchestra.run.smithi188.stdout: osd scrub load threshold = 5.0 2020-06-12T02:36:24.439 INFO:teuthology.orchestra.run.smithi188.stdout: osd scrub max interval = 120 2020-06-12T02:36:24.439 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:24.439 INFO:teuthology.orchestra.run.smithi188.stdout: osd recover clone overlap = true 2020-06-12T02:36:24.439 INFO:teuthology.orchestra.run.smithi188.stdout: osd recovery max chunk = 1048576 2020-06-12T02:36:24.440 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:24.440 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug shutdown = true 2020-06-12T02:36:24.440 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug op order = true 2020-06-12T02:36:24.440 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug verify stray on activate = true 2020-06-12T02:36:24.440 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:24.441 INFO:teuthology.orchestra.run.smithi188.stdout: osd open classes on start = true 2020-06-12T02:36:24.441 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug pg log writeout = true 2020-06-12T02:36:24.441 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:24.441 INFO:teuthology.orchestra.run.smithi188.stdout: osd deep scrub update digest min age = 30 2020-06-12T02:36:24.441 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:24.442 INFO:teuthology.orchestra.run.smithi188.stdout: osd map max advance = 10 2020-06-12T02:36:24.442 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:24.442 INFO:teuthology.orchestra.run.smithi188.stdout: journal zero on create = true 2020-06-12T02:36:24.442 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:24.443 INFO:teuthology.orchestra.run.smithi188.stdout: filestore ondisk finisher threads = 3 2020-06-12T02:36:24.443 INFO:teuthology.orchestra.run.smithi188.stdout: filestore apply finisher threads = 3 2020-06-12T02:36:24.443 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:24.443 INFO:teuthology.orchestra.run.smithi188.stdout: bdev debug aio = true 2020-06-12T02:36:24.443 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug misdirected ops = true 2020-06-12T02:36:24.450 INFO:teuthology.orchestra.run.smithi188.stdout: bluestore block size = 96636764160 2020-06-12T02:36:24.457 INFO:teuthology.orchestra.run.smithi188.stdout: bluestore compression algorithm = zstd 2020-06-12T02:36:24.457 INFO:teuthology.orchestra.run.smithi188.stdout: bluestore compression mode = aggressive 2020-06-12T02:36:24.457 INFO:teuthology.orchestra.run.smithi188.stdout: bluestore fsck on mount = True 2020-06-12T02:36:24.457 INFO:teuthology.orchestra.run.smithi188.stdout: debug bluefs = 20 2020-06-12T02:36:24.458 INFO:teuthology.orchestra.run.smithi188.stdout: debug bluestore = 20 2020-06-12T02:36:24.458 INFO:teuthology.orchestra.run.smithi188.stdout: debug filestore = 20 2020-06-12T02:36:24.458 INFO:teuthology.orchestra.run.smithi188.stdout: debug journal = 20 2020-06-12T02:36:24.458 INFO:teuthology.orchestra.run.smithi188.stdout: debug ms = 20 2020-06-12T02:36:24.459 INFO:teuthology.orchestra.run.smithi188.stdout: debug osd = 25 2020-06-12T02:36:24.459 INFO:teuthology.orchestra.run.smithi188.stdout: debug rocksdb = 10 2020-06-12T02:36:24.459 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd backfillfull_ratio = 0.85 2020-06-12T02:36:24.459 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd full ratio = 0.9 2020-06-12T02:36:24.459 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd nearfull ratio = 0.8 2020-06-12T02:36:24.460 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug reject backfill probability = 0.1 2020-06-12T02:36:24.460 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug verify cached snaps = True 2020-06-12T02:36:24.460 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug verify missing on start = True 2020-06-12T02:36:24.460 INFO:teuthology.orchestra.run.smithi188.stdout: osd failsafe full ratio = 0.95 2020-06-12T02:36:24.460 INFO:teuthology.orchestra.run.smithi188.stdout: osd heartbeat use min delay socket = True 2020-06-12T02:36:24.461 INFO:teuthology.orchestra.run.smithi188.stdout: osd max backfills = 2 2020-06-12T02:36:24.461 INFO:teuthology.orchestra.run.smithi188.stdout: osd max markdown count = 1000 2020-06-12T02:36:24.461 INFO:teuthology.orchestra.run.smithi188.stdout: osd objectstore = bluestore 2020-06-12T02:36:24.461 INFO:teuthology.orchestra.run.smithi188.stdout: osd op queue = debug_random 2020-06-12T02:36:24.461 INFO:teuthology.orchestra.run.smithi188.stdout: osd op queue cut off = debug_random 2020-06-12T02:36:24.462 INFO:teuthology.orchestra.run.smithi188.stdout: osd scrub min interval = 60 2020-06-12T02:36:24.462 INFO:teuthology.orchestra.run.smithi188.stdout: osd shutdown pgref assert = True 2020-06-12T02:36:24.462 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:24.462 INFO:teuthology.orchestra.run.smithi188.stdout:[mgr] 2020-06-12T02:36:24.462 INFO:teuthology.orchestra.run.smithi188.stdout: debug ms = 1 2020-06-12T02:36:24.463 INFO:teuthology.orchestra.run.smithi188.stdout: debug mgr = 20 2020-06-12T02:36:24.463 INFO:teuthology.orchestra.run.smithi188.stdout: debug mon = 20 2020-06-12T02:36:24.463 INFO:teuthology.orchestra.run.smithi188.stdout: debug auth = 20 2020-06-12T02:36:24.463 INFO:teuthology.orchestra.run.smithi188.stdout: mon reweight min pgs per osd = 4 2020-06-12T02:36:24.464 INFO:teuthology.orchestra.run.smithi188.stdout: mon reweight min bytes per osd = 10 2020-06-12T02:36:24.464 INFO:teuthology.orchestra.run.smithi188.stdout: mgr/telemetry/nag = false 2020-06-12T02:36:24.464 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:24.464 INFO:teuthology.orchestra.run.smithi188.stdout:[mon] 2020-06-12T02:36:24.465 INFO:teuthology.orchestra.run.smithi188.stdout: debug ms = 1 2020-06-12T02:36:24.465 INFO:teuthology.orchestra.run.smithi188.stdout: debug mon = 20 2020-06-12T02:36:24.465 INFO:teuthology.orchestra.run.smithi188.stdout: debug paxos = 20 2020-06-12T02:36:24.465 INFO:teuthology.orchestra.run.smithi188.stdout: debug auth = 20 2020-06-12T02:36:24.465 INFO:teuthology.orchestra.run.smithi188.stdout: mon data avail warn = 5 2020-06-12T02:36:24.466 INFO:teuthology.orchestra.run.smithi188.stdout: mon mgr mkfs grace = 240 2020-06-12T02:36:24.466 INFO:teuthology.orchestra.run.smithi188.stdout: mon reweight min pgs per osd = 4 2020-06-12T02:36:24.466 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd reporter subtree level = osd 2020-06-12T02:36:24.466 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd prime pg temp = true 2020-06-12T02:36:24.467 INFO:teuthology.orchestra.run.smithi188.stdout: mon reweight min bytes per osd = 10 2020-06-12T02:36:24.467 INFO:teuthology.orchestra.run.smithi188.stdout: 2020-06-12T02:36:24.467 INFO:teuthology.orchestra.run.smithi188.stdout:[client] 2020-06-12T02:36:24.467 INFO:teuthology.orchestra.run.smithi188.stdout: rgw cache enabled = true 2020-06-12T02:36:24.467 INFO:teuthology.orchestra.run.smithi188.stdout: rgw enable ops log = true 2020-06-12T02:36:24.468 INFO:teuthology.orchestra.run.smithi188.stdout: rgw enable usage log = true 2020-06-12T02:36:24.468 INFO:teuthology.orchestra.run.smithi188.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-06-12T02:36:24.468 INFO:teuthology.orchestra.run.smithi188.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-06-12T02:36:24.468 INFO:teuthology.orchestra.run.smithi188.stdout:[mon.a] 2020-06-12T02:36:24.468 INFO:teuthology.orchestra.run.smithi188.stdout:[mon.c] 2020-06-12T02:36:24.469 INFO:teuthology.orchestra.run.smithi188.stdout:[mon.b] 2020-06-12T02:36:24.469 INFO:tasks.ceph:Creating admin key on mon.a... 2020-06-12T02:36:24.470 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2020-06-12T02:36:24.521 INFO:tasks.ceph:Copying monmap to all nodes... 2020-06-12T02:36:24.538 DEBUG:teuthology.orchestra.remote:smithi107:/etc/ceph/ceph.keyring is 206B 2020-06-12T02:36:24.645 DEBUG:teuthology.orchestra.remote:smithi107:/home/ubuntu/cephtest/ceph.monmap is 379B 2020-06-12T02:36:24.693 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi107.front.sepia.ceph.com 2020-06-12T02:36:24.694 INFO:teuthology.orchestra.run.smithi107:> sudo sh -c 'cat > /etc/ceph/ceph.keyring' && sudo chmod 0644 /etc/ceph/ceph.keyring 2020-06-12T02:36:24.753 INFO:teuthology.orchestra.run.smithi107:> cat > /home/ubuntu/cephtest/ceph.monmap 2020-06-12T02:36:24.839 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi188.front.sepia.ceph.com 2020-06-12T02:36:24.839 INFO:teuthology.orchestra.run.smithi188:> sudo sh -c 'cat > /etc/ceph/ceph.keyring' && sudo chmod 0644 /etc/ceph/ceph.keyring 2020-06-12T02:36:24.898 INFO:teuthology.orchestra.run.smithi188:> cat > /home/ubuntu/cephtest/ceph.monmap 2020-06-12T02:36:24.983 INFO:tasks.ceph:Setting up mon nodes... 2020-06-12T02:36:24.984 INFO:tasks.ceph:Setting up mgr nodes... 2020-06-12T02:36:24.984 INFO:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2020-06-12T02:36:25.041 INFO:teuthology.orchestra.run.smithi107.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2020-06-12T02:36:25.043 INFO:teuthology.orchestra.run.smithi188:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2020-06-12T02:36:25.115 INFO:teuthology.orchestra.run.smithi188.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2020-06-12T02:36:25.117 INFO:tasks.ceph:Setting up mds nodes... 2020-06-12T02:36:25.118 INFO:tasks.ceph_client:Setting up client nodes... 2020-06-12T02:36:25.118 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2020-06-12T02:36:25.161 INFO:teuthology.orchestra.run.smithi107.stdout:creating /etc/ceph/ceph.client.0.keyring 2020-06-12T02:36:25.176 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2020-06-12T02:36:25.225 INFO:teuthology.orchestra.run.smithi188.stdout:creating /etc/ceph/ceph.client.1.keyring 2020-06-12T02:36:25.241 INFO:tasks.ceph:Running mkfs on osd nodes... 2020-06-12T02:36:25.242 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi107.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi188.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2020-06-12T02:36:25.242 INFO:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2020-06-12T02:36:25.270 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2020-06-12T02:36:25.271 INFO:tasks.ceph:role: osd.0 2020-06-12T02:36:25.271 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi107.front.sepia.ceph.com 2020-06-12T02:36:25.271 INFO:teuthology.orchestra.run.smithi107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-06-12T02:36:25.794 INFO:teuthology.orchestra.run.smithi107.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-06-12T02:36:25.795 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 attr=2, projid32bit=1 2020-06-12T02:36:25.795 INFO:teuthology.orchestra.run.smithi107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-06-12T02:36:25.795 INFO:teuthology.orchestra.run.smithi107.stdout: = reflink=1 2020-06-12T02:36:25.795 INFO:teuthology.orchestra.run.smithi107.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-06-12T02:36:25.796 INFO:teuthology.orchestra.run.smithi107.stdout: = sunit=0 swidth=0 blks 2020-06-12T02:36:25.796 INFO:teuthology.orchestra.run.smithi107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-06-12T02:36:25.796 INFO:teuthology.orchestra.run.smithi107.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-06-12T02:36:25.796 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-06-12T02:36:25.796 INFO:teuthology.orchestra.run.smithi107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-06-12T02:36:25.797 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi107.front.sepia.ceph.com -o noatime 2020-06-12T02:36:25.798 INFO:teuthology.orchestra.run.smithi107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2020-06-12T02:36:25.911 INFO:teuthology.orchestra.run.smithi107:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2020-06-12T02:36:25.943 INFO:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2020-06-12T02:36:26.011 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2020-06-12T02:36:26.012 INFO:tasks.ceph:role: osd.1 2020-06-12T02:36:26.012 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi107.front.sepia.ceph.com 2020-06-12T02:36:26.013 INFO:teuthology.orchestra.run.smithi107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-06-12T02:36:26.525 INFO:teuthology.orchestra.run.smithi107.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-06-12T02:36:26.526 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 attr=2, projid32bit=1 2020-06-12T02:36:26.526 INFO:teuthology.orchestra.run.smithi107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-06-12T02:36:26.526 INFO:teuthology.orchestra.run.smithi107.stdout: = reflink=1 2020-06-12T02:36:26.527 INFO:teuthology.orchestra.run.smithi107.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-06-12T02:36:26.527 INFO:teuthology.orchestra.run.smithi107.stdout: = sunit=0 swidth=0 blks 2020-06-12T02:36:26.527 INFO:teuthology.orchestra.run.smithi107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-06-12T02:36:26.527 INFO:teuthology.orchestra.run.smithi107.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-06-12T02:36:26.527 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-06-12T02:36:26.527 INFO:teuthology.orchestra.run.smithi107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-06-12T02:36:26.528 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi107.front.sepia.ceph.com -o noatime 2020-06-12T02:36:26.529 INFO:teuthology.orchestra.run.smithi107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2020-06-12T02:36:26.582 INFO:teuthology.orchestra.run.smithi107:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2020-06-12T02:36:26.613 INFO:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2020-06-12T02:36:26.680 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2020-06-12T02:36:26.681 INFO:tasks.ceph:role: osd.2 2020-06-12T02:36:26.681 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi107.front.sepia.ceph.com 2020-06-12T02:36:26.682 INFO:teuthology.orchestra.run.smithi107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-06-12T02:36:27.207 INFO:teuthology.orchestra.run.smithi107.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-06-12T02:36:27.207 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 attr=2, projid32bit=1 2020-06-12T02:36:27.208 INFO:teuthology.orchestra.run.smithi107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-06-12T02:36:27.208 INFO:teuthology.orchestra.run.smithi107.stdout: = reflink=1 2020-06-12T02:36:27.208 INFO:teuthology.orchestra.run.smithi107.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-06-12T02:36:27.208 INFO:teuthology.orchestra.run.smithi107.stdout: = sunit=0 swidth=0 blks 2020-06-12T02:36:27.208 INFO:teuthology.orchestra.run.smithi107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-06-12T02:36:27.209 INFO:teuthology.orchestra.run.smithi107.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-06-12T02:36:27.209 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-06-12T02:36:27.209 INFO:teuthology.orchestra.run.smithi107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-06-12T02:36:27.210 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi107.front.sepia.ceph.com -o noatime 2020-06-12T02:36:27.210 INFO:teuthology.orchestra.run.smithi107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2020-06-12T02:36:27.278 INFO:teuthology.orchestra.run.smithi107:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2020-06-12T02:36:27.309 INFO:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2020-06-12T02:36:27.376 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2020-06-12T02:36:27.376 INFO:tasks.ceph:role: osd.3 2020-06-12T02:36:27.376 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi107.front.sepia.ceph.com 2020-06-12T02:36:27.377 INFO:teuthology.orchestra.run.smithi107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2020-06-12T02:36:27.905 INFO:teuthology.orchestra.run.smithi107.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2020-06-12T02:36:27.905 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 attr=2, projid32bit=1 2020-06-12T02:36:27.906 INFO:teuthology.orchestra.run.smithi107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-06-12T02:36:27.906 INFO:teuthology.orchestra.run.smithi107.stdout: = reflink=1 2020-06-12T02:36:27.906 INFO:teuthology.orchestra.run.smithi107.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-06-12T02:36:27.906 INFO:teuthology.orchestra.run.smithi107.stdout: = sunit=0 swidth=0 blks 2020-06-12T02:36:27.907 INFO:teuthology.orchestra.run.smithi107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-06-12T02:36:27.907 INFO:teuthology.orchestra.run.smithi107.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-06-12T02:36:27.907 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-06-12T02:36:27.907 INFO:teuthology.orchestra.run.smithi107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-06-12T02:36:27.908 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi107.front.sepia.ceph.com -o noatime 2020-06-12T02:36:27.909 INFO:teuthology.orchestra.run.smithi107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2020-06-12T02:36:28.028 INFO:teuthology.orchestra.run.smithi107:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2020-06-12T02:36:28.058 INFO:teuthology.orchestra.run.smithi107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-06-12T02:36:28.141 INFO:teuthology.orchestra.run.smithi107.stderr:2020-06-12T02:36:28.138+0000 7f9974309ec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2020-06-12T02:36:28.144 INFO:teuthology.orchestra.run.smithi107.stderr:2020-06-12T02:36:28.141+0000 7f9974309ec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2020-06-12T02:36:28.145 INFO:teuthology.orchestra.run.smithi107.stderr:2020-06-12T02:36:28.141+0000 7f9974309ec0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2020-06-12T02:36:28.145 INFO:teuthology.orchestra.run.smithi107.stderr:2020-06-12T02:36:28.141+0000 7f9974309ec0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2020-06-12T02:36:28.145 INFO:teuthology.orchestra.run.smithi107.stderr:2020-06-12T02:36:28.141+0000 7f9974309ec0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2020-06-12T02:36:28.153 INFO:teuthology.orchestra.run.smithi107.stderr:2020-06-12T02:36:28.149+0000 7f9974309ec0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2020-06-12T02:36:28.180 INFO:teuthology.orchestra.run.smithi107.stderr:2020-06-12T02:36:28.177+0000 7f9974309ec0 -1 freelist read_size_meta_from_db missing size meta in DB 2020-06-12T02:36:30.488 INFO:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2020-06-12T02:36:30.563 INFO:teuthology.orchestra.run.smithi107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-06-12T02:36:30.656 INFO:teuthology.orchestra.run.smithi107.stderr:2020-06-12T02:36:30.653+0000 7f7fc4761ec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2020-06-12T02:36:30.657 INFO:teuthology.orchestra.run.smithi107.stderr:2020-06-12T02:36:30.653+0000 7f7fc4761ec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2020-06-12T02:36:30.657 INFO:teuthology.orchestra.run.smithi107.stderr:2020-06-12T02:36:30.653+0000 7f7fc4761ec0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2020-06-12T02:36:30.657 INFO:teuthology.orchestra.run.smithi107.stderr:2020-06-12T02:36:30.653+0000 7f7fc4761ec0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2020-06-12T02:36:30.657 INFO:teuthology.orchestra.run.smithi107.stderr:2020-06-12T02:36:30.653+0000 7f7fc4761ec0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2020-06-12T02:36:30.658 INFO:teuthology.orchestra.run.smithi107.stderr:2020-06-12T02:36:30.654+0000 7f7fc4761ec0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2020-06-12T02:36:30.667 INFO:teuthology.orchestra.run.smithi107.stderr:2020-06-12T02:36:30.664+0000 7f7fc4761ec0 -1 freelist read_size_meta_from_db missing size meta in DB 2020-06-12T02:36:32.991 INFO:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2020-06-12T02:36:33.022 INFO:teuthology.orchestra.run.smithi107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-06-12T02:36:33.109 INFO:teuthology.orchestra.run.smithi107.stderr:2020-06-12T02:36:33.105+0000 7ff2319f9ec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2020-06-12T02:36:33.109 INFO:teuthology.orchestra.run.smithi107.stderr:2020-06-12T02:36:33.105+0000 7ff2319f9ec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2020-06-12T02:36:33.109 INFO:teuthology.orchestra.run.smithi107.stderr:2020-06-12T02:36:33.106+0000 7ff2319f9ec0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2020-06-12T02:36:33.110 INFO:teuthology.orchestra.run.smithi107.stderr:2020-06-12T02:36:33.106+0000 7ff2319f9ec0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2020-06-12T02:36:33.110 INFO:teuthology.orchestra.run.smithi107.stderr:2020-06-12T02:36:33.106+0000 7ff2319f9ec0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2020-06-12T02:36:33.110 INFO:teuthology.orchestra.run.smithi107.stderr:2020-06-12T02:36:33.106+0000 7ff2319f9ec0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2020-06-12T02:36:33.123 INFO:teuthology.orchestra.run.smithi107.stderr:2020-06-12T02:36:33.120+0000 7ff2319f9ec0 -1 freelist read_size_meta_from_db missing size meta in DB 2020-06-12T02:36:35.448 INFO:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2020-06-12T02:36:35.482 INFO:teuthology.orchestra.run.smithi107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-06-12T02:36:35.569 INFO:teuthology.orchestra.run.smithi107.stderr:2020-06-12T02:36:35.566+0000 7eff62e04ec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2020-06-12T02:36:35.570 INFO:teuthology.orchestra.run.smithi107.stderr:2020-06-12T02:36:35.566+0000 7eff62e04ec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2020-06-12T02:36:35.570 INFO:teuthology.orchestra.run.smithi107.stderr:2020-06-12T02:36:35.566+0000 7eff62e04ec0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2020-06-12T02:36:35.570 INFO:teuthology.orchestra.run.smithi107.stderr:2020-06-12T02:36:35.566+0000 7eff62e04ec0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2020-06-12T02:36:35.571 INFO:teuthology.orchestra.run.smithi107.stderr:2020-06-12T02:36:35.566+0000 7eff62e04ec0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2020-06-12T02:36:35.571 INFO:teuthology.orchestra.run.smithi107.stderr:2020-06-12T02:36:35.567+0000 7eff62e04ec0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2020-06-12T02:36:35.584 INFO:teuthology.orchestra.run.smithi107.stderr:2020-06-12T02:36:35.581+0000 7eff62e04ec0 -1 freelist read_size_meta_from_db missing size meta in DB 2020-06-12T02:36:37.912 INFO:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2020-06-12T02:36:37.945 INFO:teuthology.orchestra.run.smithi188:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2020-06-12T02:36:37.979 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2020-06-12T02:36:37.979 INFO:tasks.ceph:role: osd.4 2020-06-12T02:36:37.980 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi188.front.sepia.ceph.com 2020-06-12T02:36:37.980 INFO:teuthology.orchestra.run.smithi188:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-06-12T02:36:38.508 INFO:teuthology.orchestra.run.smithi188.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-06-12T02:36:38.508 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 attr=2, projid32bit=1 2020-06-12T02:36:38.508 INFO:teuthology.orchestra.run.smithi188.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-06-12T02:36:38.509 INFO:teuthology.orchestra.run.smithi188.stdout: = reflink=1 2020-06-12T02:36:38.509 INFO:teuthology.orchestra.run.smithi188.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-06-12T02:36:38.509 INFO:teuthology.orchestra.run.smithi188.stdout: = sunit=0 swidth=0 blks 2020-06-12T02:36:38.509 INFO:teuthology.orchestra.run.smithi188.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-06-12T02:36:38.509 INFO:teuthology.orchestra.run.smithi188.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-06-12T02:36:38.510 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-06-12T02:36:38.510 INFO:teuthology.orchestra.run.smithi188.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-06-12T02:36:38.511 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi188.front.sepia.ceph.com -o noatime 2020-06-12T02:36:38.511 INFO:teuthology.orchestra.run.smithi188:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2020-06-12T02:36:38.642 INFO:teuthology.orchestra.run.smithi188:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2020-06-12T02:36:38.676 INFO:teuthology.orchestra.run.smithi188:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2020-06-12T02:36:38.746 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2020-06-12T02:36:38.746 INFO:tasks.ceph:role: osd.5 2020-06-12T02:36:38.747 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi188.front.sepia.ceph.com 2020-06-12T02:36:38.747 INFO:teuthology.orchestra.run.smithi188:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-06-12T02:36:39.254 INFO:teuthology.orchestra.run.smithi188.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-06-12T02:36:39.254 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 attr=2, projid32bit=1 2020-06-12T02:36:39.254 INFO:teuthology.orchestra.run.smithi188.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-06-12T02:36:39.254 INFO:teuthology.orchestra.run.smithi188.stdout: = reflink=1 2020-06-12T02:36:39.255 INFO:teuthology.orchestra.run.smithi188.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-06-12T02:36:39.255 INFO:teuthology.orchestra.run.smithi188.stdout: = sunit=0 swidth=0 blks 2020-06-12T02:36:39.255 INFO:teuthology.orchestra.run.smithi188.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-06-12T02:36:39.255 INFO:teuthology.orchestra.run.smithi188.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-06-12T02:36:39.255 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-06-12T02:36:39.256 INFO:teuthology.orchestra.run.smithi188.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-06-12T02:36:39.256 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi188.front.sepia.ceph.com -o noatime 2020-06-12T02:36:39.256 INFO:teuthology.orchestra.run.smithi188:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2020-06-12T02:36:39.306 INFO:teuthology.orchestra.run.smithi188:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2020-06-12T02:36:39.338 INFO:teuthology.orchestra.run.smithi188:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2020-06-12T02:36:39.407 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2020-06-12T02:36:39.408 INFO:tasks.ceph:role: osd.6 2020-06-12T02:36:39.408 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi188.front.sepia.ceph.com 2020-06-12T02:36:39.408 INFO:teuthology.orchestra.run.smithi188:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-06-12T02:36:39.955 INFO:teuthology.orchestra.run.smithi188.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-06-12T02:36:39.956 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 attr=2, projid32bit=1 2020-06-12T02:36:39.956 INFO:teuthology.orchestra.run.smithi188.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-06-12T02:36:39.956 INFO:teuthology.orchestra.run.smithi188.stdout: = reflink=1 2020-06-12T02:36:39.957 INFO:teuthology.orchestra.run.smithi188.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-06-12T02:36:39.957 INFO:teuthology.orchestra.run.smithi188.stdout: = sunit=0 swidth=0 blks 2020-06-12T02:36:39.957 INFO:teuthology.orchestra.run.smithi188.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-06-12T02:36:39.957 INFO:teuthology.orchestra.run.smithi188.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-06-12T02:36:39.957 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-06-12T02:36:39.958 INFO:teuthology.orchestra.run.smithi188.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-06-12T02:36:39.958 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi188.front.sepia.ceph.com -o noatime 2020-06-12T02:36:39.959 INFO:teuthology.orchestra.run.smithi188:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2020-06-12T02:36:40.035 INFO:teuthology.orchestra.run.smithi188:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2020-06-12T02:36:40.070 INFO:teuthology.orchestra.run.smithi188:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2020-06-12T02:36:40.143 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2020-06-12T02:36:40.143 INFO:tasks.ceph:role: osd.7 2020-06-12T02:36:40.144 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi188.front.sepia.ceph.com 2020-06-12T02:36:40.144 INFO:teuthology.orchestra.run.smithi188:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2020-06-12T02:36:40.686 INFO:teuthology.orchestra.run.smithi188.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2020-06-12T02:36:40.686 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 attr=2, projid32bit=1 2020-06-12T02:36:40.686 INFO:teuthology.orchestra.run.smithi188.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-06-12T02:36:40.687 INFO:teuthology.orchestra.run.smithi188.stdout: = reflink=1 2020-06-12T02:36:40.687 INFO:teuthology.orchestra.run.smithi188.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-06-12T02:36:40.687 INFO:teuthology.orchestra.run.smithi188.stdout: = sunit=0 swidth=0 blks 2020-06-12T02:36:40.687 INFO:teuthology.orchestra.run.smithi188.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-06-12T02:36:40.687 INFO:teuthology.orchestra.run.smithi188.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-06-12T02:36:40.688 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-06-12T02:36:40.688 INFO:teuthology.orchestra.run.smithi188.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-06-12T02:36:40.689 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi188.front.sepia.ceph.com -o noatime 2020-06-12T02:36:40.689 INFO:teuthology.orchestra.run.smithi188:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2020-06-12T02:36:40.800 INFO:teuthology.orchestra.run.smithi188:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2020-06-12T02:36:40.832 INFO:teuthology.orchestra.run.smithi188:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-06-12T02:36:40.916 INFO:teuthology.orchestra.run.smithi188.stderr:2020-06-12T02:36:40.912+0000 7f5a00130ec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2020-06-12T02:36:40.919 INFO:teuthology.orchestra.run.smithi188.stderr:2020-06-12T02:36:40.914+0000 7f5a00130ec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2020-06-12T02:36:40.919 INFO:teuthology.orchestra.run.smithi188.stderr:2020-06-12T02:36:40.914+0000 7f5a00130ec0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2020-06-12T02:36:40.919 INFO:teuthology.orchestra.run.smithi188.stderr:2020-06-12T02:36:40.914+0000 7f5a00130ec0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2020-06-12T02:36:40.920 INFO:teuthology.orchestra.run.smithi188.stderr:2020-06-12T02:36:40.914+0000 7f5a00130ec0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2020-06-12T02:36:40.924 INFO:teuthology.orchestra.run.smithi188.stderr:2020-06-12T02:36:40.920+0000 7f5a00130ec0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2020-06-12T02:36:40.952 INFO:teuthology.orchestra.run.smithi188.stderr:2020-06-12T02:36:40.948+0000 7f5a00130ec0 -1 freelist read_size_meta_from_db missing size meta in DB 2020-06-12T02:36:43.263 INFO:teuthology.orchestra.run.smithi188:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2020-06-12T02:36:43.336 INFO:teuthology.orchestra.run.smithi188:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-06-12T02:36:43.425 INFO:teuthology.orchestra.run.smithi188.stderr:2020-06-12T02:36:43.421+0000 7f63b9051ec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2020-06-12T02:36:43.425 INFO:teuthology.orchestra.run.smithi188.stderr:2020-06-12T02:36:43.421+0000 7f63b9051ec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2020-06-12T02:36:43.426 INFO:teuthology.orchestra.run.smithi188.stderr:2020-06-12T02:36:43.421+0000 7f63b9051ec0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2020-06-12T02:36:43.426 INFO:teuthology.orchestra.run.smithi188.stderr:2020-06-12T02:36:43.421+0000 7f63b9051ec0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2020-06-12T02:36:43.426 INFO:teuthology.orchestra.run.smithi188.stderr:2020-06-12T02:36:43.421+0000 7f63b9051ec0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2020-06-12T02:36:43.427 INFO:teuthology.orchestra.run.smithi188.stderr:2020-06-12T02:36:43.422+0000 7f63b9051ec0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2020-06-12T02:36:43.440 INFO:teuthology.orchestra.run.smithi188.stderr:2020-06-12T02:36:43.437+0000 7f63b9051ec0 -1 freelist read_size_meta_from_db missing size meta in DB 2020-06-12T02:36:45.771 INFO:teuthology.orchestra.run.smithi188:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2020-06-12T02:36:45.803 INFO:teuthology.orchestra.run.smithi188:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-06-12T02:36:45.893 INFO:teuthology.orchestra.run.smithi188.stderr:2020-06-12T02:36:45.888+0000 7f07b21a1ec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2020-06-12T02:36:45.894 INFO:teuthology.orchestra.run.smithi188.stderr:2020-06-12T02:36:45.888+0000 7f07b21a1ec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2020-06-12T02:36:45.894 INFO:teuthology.orchestra.run.smithi188.stderr:2020-06-12T02:36:45.888+0000 7f07b21a1ec0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2020-06-12T02:36:45.894 INFO:teuthology.orchestra.run.smithi188.stderr:2020-06-12T02:36:45.888+0000 7f07b21a1ec0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2020-06-12T02:36:45.894 INFO:teuthology.orchestra.run.smithi188.stderr:2020-06-12T02:36:45.888+0000 7f07b21a1ec0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2020-06-12T02:36:45.895 INFO:teuthology.orchestra.run.smithi188.stderr:2020-06-12T02:36:45.889+0000 7f07b21a1ec0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2020-06-12T02:36:45.907 INFO:teuthology.orchestra.run.smithi188.stderr:2020-06-12T02:36:45.903+0000 7f07b21a1ec0 -1 freelist read_size_meta_from_db missing size meta in DB 2020-06-12T02:36:48.233 INFO:teuthology.orchestra.run.smithi188:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2020-06-12T02:36:48.266 INFO:teuthology.orchestra.run.smithi188:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-06-12T02:36:48.356 INFO:teuthology.orchestra.run.smithi188.stderr:2020-06-12T02:36:48.352+0000 7f454c763ec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2020-06-12T02:36:48.356 INFO:teuthology.orchestra.run.smithi188.stderr:2020-06-12T02:36:48.352+0000 7f454c763ec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2020-06-12T02:36:48.357 INFO:teuthology.orchestra.run.smithi188.stderr:2020-06-12T02:36:48.352+0000 7f454c763ec0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2020-06-12T02:36:48.357 INFO:teuthology.orchestra.run.smithi188.stderr:2020-06-12T02:36:48.352+0000 7f454c763ec0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2020-06-12T02:36:48.357 INFO:teuthology.orchestra.run.smithi188.stderr:2020-06-12T02:36:48.352+0000 7f454c763ec0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2020-06-12T02:36:48.358 INFO:teuthology.orchestra.run.smithi188.stderr:2020-06-12T02:36:48.353+0000 7f454c763ec0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2020-06-12T02:36:48.370 INFO:teuthology.orchestra.run.smithi188.stderr:2020-06-12T02:36:48.367+0000 7f454c763ec0 -1 freelist read_size_meta_from_db missing size meta in DB 2020-06-12T02:36:50.703 INFO:teuthology.orchestra.run.smithi188:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2020-06-12T02:36:50.736 INFO:tasks.ceph:Reading keys from all nodes... 2020-06-12T02:36:50.736 INFO:teuthology.orchestra.run.smithi107:> mktemp 2020-06-12T02:36:50.754 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.G7IWgY4kPV 2020-06-12T02:36:50.755 INFO:teuthology.orchestra.run.smithi107:> sudo cp /var/lib/ceph/mgr/ceph-y/keyring /tmp/tmp.G7IWgY4kPV 2020-06-12T02:36:50.834 INFO:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.G7IWgY4kPV 2020-06-12T02:36:50.962 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.G7IWgY4kPV is 56B 2020-06-12T02:36:51.011 INFO:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.G7IWgY4kPV 2020-06-12T02:36:51.043 INFO:teuthology.orchestra.run.smithi107:> mktemp 2020-06-12T02:36:51.081 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.VYwLcs2Ncq 2020-06-12T02:36:51.082 INFO:teuthology.orchestra.run.smithi107:> sudo cp /var/lib/ceph/osd/ceph-0/keyring /tmp/tmp.VYwLcs2Ncq 2020-06-12T02:36:51.150 INFO:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.VYwLcs2Ncq 2020-06-12T02:36:51.275 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.VYwLcs2Ncq is 56B 2020-06-12T02:36:51.322 INFO:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.VYwLcs2Ncq 2020-06-12T02:36:51.341 INFO:teuthology.orchestra.run.smithi107:> mktemp 2020-06-12T02:36:51.389 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.mZVtDQsuWx 2020-06-12T02:36:51.389 INFO:teuthology.orchestra.run.smithi107:> sudo cp /var/lib/ceph/osd/ceph-1/keyring /tmp/tmp.mZVtDQsuWx 2020-06-12T02:36:51.456 INFO:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.mZVtDQsuWx 2020-06-12T02:36:51.581 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.mZVtDQsuWx is 56B 2020-06-12T02:36:51.629 INFO:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.mZVtDQsuWx 2020-06-12T02:36:51.643 INFO:teuthology.orchestra.run.smithi107:> mktemp 2020-06-12T02:36:51.695 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.7aJi1gg5V7 2020-06-12T02:36:51.696 INFO:teuthology.orchestra.run.smithi107:> sudo cp /var/lib/ceph/osd/ceph-2/keyring /tmp/tmp.7aJi1gg5V7 2020-06-12T02:36:51.762 INFO:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.7aJi1gg5V7 2020-06-12T02:36:51.885 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.7aJi1gg5V7 is 56B 2020-06-12T02:36:51.934 INFO:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.7aJi1gg5V7 2020-06-12T02:36:51.948 INFO:teuthology.orchestra.run.smithi107:> mktemp 2020-06-12T02:36:52.001 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.ah2Nyt0pKy 2020-06-12T02:36:52.001 INFO:teuthology.orchestra.run.smithi107:> sudo cp /var/lib/ceph/osd/ceph-3/keyring /tmp/tmp.ah2Nyt0pKy 2020-06-12T02:36:52.068 INFO:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.ah2Nyt0pKy 2020-06-12T02:36:52.193 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.ah2Nyt0pKy is 56B 2020-06-12T02:36:52.241 INFO:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.ah2Nyt0pKy 2020-06-12T02:36:52.254 INFO:teuthology.orchestra.run.smithi188:> mktemp 2020-06-12T02:36:52.271 INFO:teuthology.orchestra.run.smithi188.stdout:/tmp/tmp.Sbcrxxpj1J 2020-06-12T02:36:52.272 INFO:teuthology.orchestra.run.smithi188:> sudo cp /var/lib/ceph/mgr/ceph-x/keyring /tmp/tmp.Sbcrxxpj1J 2020-06-12T02:36:52.343 INFO:teuthology.orchestra.run.smithi188:> sudo chmod 0666 /tmp/tmp.Sbcrxxpj1J 2020-06-12T02:36:52.513 DEBUG:teuthology.orchestra.remote:smithi188:/tmp/tmp.Sbcrxxpj1J is 56B 2020-06-12T02:36:52.548 INFO:teuthology.orchestra.run.smithi188:> rm -fr /tmp/tmp.Sbcrxxpj1J 2020-06-12T02:36:52.564 INFO:teuthology.orchestra.run.smithi188:> mktemp 2020-06-12T02:36:52.620 INFO:teuthology.orchestra.run.smithi188.stdout:/tmp/tmp.wZc5KvnVRP 2020-06-12T02:36:52.621 INFO:teuthology.orchestra.run.smithi188:> sudo cp /var/lib/ceph/osd/ceph-4/keyring /tmp/tmp.wZc5KvnVRP 2020-06-12T02:36:52.729 INFO:teuthology.orchestra.run.smithi188:> sudo chmod 0666 /tmp/tmp.wZc5KvnVRP 2020-06-12T02:36:52.816 DEBUG:teuthology.orchestra.remote:smithi188:/tmp/tmp.wZc5KvnVRP is 56B 2020-06-12T02:36:52.866 INFO:teuthology.orchestra.run.smithi188:> rm -fr /tmp/tmp.wZc5KvnVRP 2020-06-12T02:36:52.881 INFO:teuthology.orchestra.run.smithi188:> mktemp 2020-06-12T02:36:52.934 INFO:teuthology.orchestra.run.smithi188.stdout:/tmp/tmp.mOQ11lGtgD 2020-06-12T02:36:52.935 INFO:teuthology.orchestra.run.smithi188:> sudo cp /var/lib/ceph/osd/ceph-5/keyring /tmp/tmp.mOQ11lGtgD 2020-06-12T02:36:53.002 INFO:teuthology.orchestra.run.smithi188:> sudo chmod 0666 /tmp/tmp.mOQ11lGtgD 2020-06-12T02:36:53.133 DEBUG:teuthology.orchestra.remote:smithi188:/tmp/tmp.mOQ11lGtgD is 56B 2020-06-12T02:36:53.184 INFO:teuthology.orchestra.run.smithi188:> rm -fr /tmp/tmp.mOQ11lGtgD 2020-06-12T02:36:53.199 INFO:teuthology.orchestra.run.smithi188:> mktemp 2020-06-12T02:36:53.254 INFO:teuthology.orchestra.run.smithi188.stdout:/tmp/tmp.DZ6ZgnuL3h 2020-06-12T02:36:53.254 INFO:teuthology.orchestra.run.smithi188:> sudo cp /var/lib/ceph/osd/ceph-6/keyring /tmp/tmp.DZ6ZgnuL3h 2020-06-12T02:36:53.320 INFO:teuthology.orchestra.run.smithi188:> sudo chmod 0666 /tmp/tmp.DZ6ZgnuL3h 2020-06-12T02:36:53.444 DEBUG:teuthology.orchestra.remote:smithi188:/tmp/tmp.DZ6ZgnuL3h is 56B 2020-06-12T02:36:53.492 INFO:teuthology.orchestra.run.smithi188:> rm -fr /tmp/tmp.DZ6ZgnuL3h 2020-06-12T02:36:53.507 INFO:teuthology.orchestra.run.smithi188:> mktemp 2020-06-12T02:36:53.559 INFO:teuthology.orchestra.run.smithi188.stdout:/tmp/tmp.n1M7F4AyDk 2020-06-12T02:36:53.560 INFO:teuthology.orchestra.run.smithi188:> sudo cp /var/lib/ceph/osd/ceph-7/keyring /tmp/tmp.n1M7F4AyDk 2020-06-12T02:36:53.627 INFO:teuthology.orchestra.run.smithi188:> sudo chmod 0666 /tmp/tmp.n1M7F4AyDk 2020-06-12T02:36:53.751 DEBUG:teuthology.orchestra.remote:smithi188:/tmp/tmp.n1M7F4AyDk is 56B 2020-06-12T02:36:53.800 INFO:teuthology.orchestra.run.smithi188:> rm -fr /tmp/tmp.n1M7F4AyDk 2020-06-12T02:36:53.833 DEBUG:teuthology.orchestra.remote:smithi107:/etc/ceph/ceph.client.0.keyring is 59B 2020-06-12T02:36:53.903 DEBUG:teuthology.orchestra.remote:smithi188:/etc/ceph/ceph.client.1.keyring is 59B 2020-06-12T02:36:53.963 INFO:tasks.ceph:Adding keys to all mons... 2020-06-12T02:36:53.964 INFO:teuthology.orchestra.run.smithi107:> sudo tee -a /etc/ceph/ceph.keyring 2020-06-12T02:36:53.972 INFO:teuthology.orchestra.run.smithi188:> sudo tee -a /etc/ceph/ceph.keyring 2020-06-12T02:36:54.013 INFO:teuthology.orchestra.run.smithi107.stdout:[mgr.y] 2020-06-12T02:36:54.013 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQAp6uJe6fZaAhAAzGGOdH7+Fb7iK9V4ETLD6g== 2020-06-12T02:36:54.014 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.0] 2020-06-12T02:36:54.014 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQAs6uJeGCtZCBAAHdxY6wfrlEUKtbuDCzI+yQ== 2020-06-12T02:36:54.014 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.1] 2020-06-12T02:36:54.015 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQAu6uJee1cFJxAAcC81nsp8bOSsiKzTrWbnbg== 2020-06-12T02:36:54.015 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.2] 2020-06-12T02:36:54.015 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQAx6uJeLOBgBhAAYbnwaBadKqsZNw+JzLIwdw== 2020-06-12T02:36:54.015 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.3] 2020-06-12T02:36:54.016 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQAz6uJeqOXSIRAAOrDblKdnBB4rVLlfeC1o7Q== 2020-06-12T02:36:54.016 INFO:teuthology.orchestra.run.smithi107.stdout:[mgr.x] 2020-06-12T02:36:54.016 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQAp6uJep+bIBhAAD/wQgrDCg3sMB7dAq/yKKw== 2020-06-12T02:36:54.017 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.4] 2020-06-12T02:36:54.017 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQA46uJe3L+BNhAA5p2LPTdlRChRJGVH2swdjg== 2020-06-12T02:36:54.017 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.5] 2020-06-12T02:36:54.017 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQA76uJeros6GRAAkZCuGIxdViHE/aJVVRgq3Q== 2020-06-12T02:36:54.018 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.6] 2020-06-12T02:36:54.018 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQA96uJe7aoNNRAAZFSS9WZ1ukhB0GKCO6hg8g== 2020-06-12T02:36:54.018 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.7] 2020-06-12T02:36:54.018 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQBA6uJeFxQbFRAANP54jFYJD2nLTnwXCQEQ5w== 2020-06-12T02:36:54.019 INFO:teuthology.orchestra.run.smithi107.stdout:[client.0] 2020-06-12T02:36:54.019 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQAp6uJedMx3CRAAKjKnIK+qnISY8SUBgseOXg== 2020-06-12T02:36:54.019 INFO:teuthology.orchestra.run.smithi107.stdout:[client.1] 2020-06-12T02:36:54.019 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQAp6uJejLhPDRAAJQSPIsZlhfgrAg5/F7oO6g== 2020-06-12T02:36:54.022 INFO:teuthology.orchestra.run.smithi188.stdout:[mgr.y] 2020-06-12T02:36:54.022 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQAp6uJe6fZaAhAAzGGOdH7+Fb7iK9V4ETLD6g== 2020-06-12T02:36:54.023 INFO:teuthology.orchestra.run.smithi188.stdout:[osd.0] 2020-06-12T02:36:54.023 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQAs6uJeGCtZCBAAHdxY6wfrlEUKtbuDCzI+yQ== 2020-06-12T02:36:54.023 INFO:teuthology.orchestra.run.smithi188.stdout:[osd.1] 2020-06-12T02:36:54.023 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQAu6uJee1cFJxAAcC81nsp8bOSsiKzTrWbnbg== 2020-06-12T02:36:54.023 INFO:teuthology.orchestra.run.smithi188.stdout:[osd.2] 2020-06-12T02:36:54.024 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQAx6uJeLOBgBhAAYbnwaBadKqsZNw+JzLIwdw== 2020-06-12T02:36:54.024 INFO:teuthology.orchestra.run.smithi188.stdout:[osd.3] 2020-06-12T02:36:54.024 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQAz6uJeqOXSIRAAOrDblKdnBB4rVLlfeC1o7Q== 2020-06-12T02:36:54.024 INFO:teuthology.orchestra.run.smithi188.stdout:[mgr.x] 2020-06-12T02:36:54.024 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQAp6uJep+bIBhAAD/wQgrDCg3sMB7dAq/yKKw== 2020-06-12T02:36:54.025 INFO:teuthology.orchestra.run.smithi188.stdout:[osd.4] 2020-06-12T02:36:54.025 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQA46uJe3L+BNhAA5p2LPTdlRChRJGVH2swdjg== 2020-06-12T02:36:54.025 INFO:teuthology.orchestra.run.smithi188.stdout:[osd.5] 2020-06-12T02:36:54.025 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQA76uJeros6GRAAkZCuGIxdViHE/aJVVRgq3Q== 2020-06-12T02:36:54.025 INFO:teuthology.orchestra.run.smithi188.stdout:[osd.6] 2020-06-12T02:36:54.026 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQA96uJe7aoNNRAAZFSS9WZ1ukhB0GKCO6hg8g== 2020-06-12T02:36:54.026 INFO:teuthology.orchestra.run.smithi188.stdout:[osd.7] 2020-06-12T02:36:54.026 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQBA6uJeFxQbFRAANP54jFYJD2nLTnwXCQEQ5w== 2020-06-12T02:36:54.026 INFO:teuthology.orchestra.run.smithi188.stdout:[client.0] 2020-06-12T02:36:54.026 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQAp6uJedMx3CRAAKjKnIK+qnISY8SUBgseOXg== 2020-06-12T02:36:54.027 INFO:teuthology.orchestra.run.smithi188.stdout:[client.1] 2020-06-12T02:36:54.027 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQAp6uJejLhPDRAAJQSPIsZlhfgrAg5/F7oO6g== 2020-06-12T02:36:54.028 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-06-12T02:36:54.063 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-06-12T02:36:54.118 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-06-12T02:36:54.122 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-06-12T02:36:54.171 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-06-12T02:36:54.174 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-06-12T02:36:54.224 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-06-12T02:36:54.228 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-06-12T02:36:54.283 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-06-12T02:36:54.286 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-06-12T02:36:54.334 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-06-12T02:36:54.336 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-06-12T02:36:54.384 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-06-12T02:36:54.387 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-06-12T02:36:54.434 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-06-12T02:36:54.437 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-06-12T02:36:54.485 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-06-12T02:36:54.487 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-06-12T02:36:54.539 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-06-12T02:36:54.541 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-06-12T02:36:54.590 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-06-12T02:36:54.592 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-06-12T02:36:54.643 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-06-12T02:36:54.645 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-06-12T02:36:54.699 INFO:tasks.ceph:Running mkfs on mon nodes... 2020-06-12T02:36:54.700 INFO:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2020-06-12T02:36:54.727 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2020-06-12T02:36:54.819 INFO:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2020-06-12T02:36:54.846 INFO:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2020-06-12T02:36:54.913 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2020-06-12T02:36:55.007 INFO:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2020-06-12T02:36:55.049 INFO:teuthology.orchestra.run.smithi188:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2020-06-12T02:36:55.080 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2020-06-12T02:36:55.175 INFO:teuthology.orchestra.run.smithi188:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2020-06-12T02:36:55.203 INFO:teuthology.orchestra.run.smithi107:> rm -- /home/ubuntu/cephtest/ceph.monmap 2020-06-12T02:36:55.206 INFO:teuthology.orchestra.run.smithi188:> rm -- /home/ubuntu/cephtest/ceph.monmap 2020-06-12T02:36:55.257 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2020-06-12T02:36:55.258 INFO:tasks.ceph.mon.a:Restarting daemon 2020-06-12T02:36:55.258 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2020-06-12T02:36:55.261 INFO:tasks.ceph.mon.a:Started 2020-06-12T02:36:55.262 INFO:tasks.ceph.mon.c:Restarting daemon 2020-06-12T02:36:55.262 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2020-06-12T02:36:55.264 INFO:tasks.ceph.mon.c:Started 2020-06-12T02:36:55.264 INFO:tasks.ceph.mon.b:Restarting daemon 2020-06-12T02:36:55.265 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2020-06-12T02:36:55.301 INFO:tasks.ceph.mon.b:Started 2020-06-12T02:36:55.311 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2020-06-12T02:36:55.312 INFO:tasks.ceph.mgr.y:Restarting daemon 2020-06-12T02:36:55.312 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2020-06-12T02:36:55.315 INFO:tasks.ceph.mgr.y:Started 2020-06-12T02:36:55.315 INFO:tasks.ceph.mgr.x:Restarting daemon 2020-06-12T02:36:55.316 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2020-06-12T02:36:55.331 INFO:tasks.ceph.mgr.x:Started 2020-06-12T02:36:55.332 INFO:tasks.ceph:Setting crush tunables to default 2020-06-12T02:36:55.332 INFO:teuthology.orchestra.run.smithi107:> sudo ceph --cluster ceph osd crush tunables default 2020-06-12T02:37:00.698 INFO:teuthology.orchestra.run.smithi107.stderr:adjusted tunables profile to default 2020-06-12T02:37:00.708 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2020-06-12T02:37:00.709 INFO:teuthology.orchestra.run.smithi107:> mktemp 2020-06-12T02:37:00.723 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.z5uwuBowzQ 2020-06-12T02:37:00.723 INFO:teuthology.orchestra.run.smithi107:> sudo cp /var/lib/ceph/osd/ceph-0/fsid /tmp/tmp.z5uwuBowzQ 2020-06-12T02:37:00.791 INFO:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.z5uwuBowzQ 2020-06-12T02:37:00.915 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.z5uwuBowzQ is 37B 2020-06-12T02:37:00.962 INFO:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.z5uwuBowzQ 2020-06-12T02:37:00.977 INFO:teuthology.orchestra.run.smithi107:> mktemp 2020-06-12T02:37:01.033 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.fuKkvzFrLi 2020-06-12T02:37:01.034 INFO:teuthology.orchestra.run.smithi107:> sudo cp /var/lib/ceph/osd/ceph-1/fsid /tmp/tmp.fuKkvzFrLi 2020-06-12T02:37:01.101 INFO:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.fuKkvzFrLi 2020-06-12T02:37:01.228 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.fuKkvzFrLi is 37B 2020-06-12T02:37:01.283 INFO:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.fuKkvzFrLi 2020-06-12T02:37:01.299 INFO:teuthology.orchestra.run.smithi107:> mktemp 2020-06-12T02:37:01.352 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.OgLqBSAfjw 2020-06-12T02:37:01.352 INFO:teuthology.orchestra.run.smithi107:> sudo cp /var/lib/ceph/osd/ceph-2/fsid /tmp/tmp.OgLqBSAfjw 2020-06-12T02:37:01.420 INFO:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.OgLqBSAfjw 2020-06-12T02:37:01.553 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.OgLqBSAfjw is 37B 2020-06-12T02:37:01.602 INFO:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.OgLqBSAfjw 2020-06-12T02:37:01.625 INFO:teuthology.orchestra.run.smithi107:> mktemp 2020-06-12T02:37:01.686 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.kMg9RuqCjY 2020-06-12T02:37:01.686 INFO:teuthology.orchestra.run.smithi107:> sudo cp /var/lib/ceph/osd/ceph-3/fsid /tmp/tmp.kMg9RuqCjY 2020-06-12T02:37:01.754 INFO:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.kMg9RuqCjY 2020-06-12T02:37:01.879 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.kMg9RuqCjY is 37B 2020-06-12T02:37:01.932 INFO:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.kMg9RuqCjY 2020-06-12T02:37:01.946 INFO:teuthology.orchestra.run.smithi188:> mktemp 2020-06-12T02:37:01.960 INFO:teuthology.orchestra.run.smithi188.stdout:/tmp/tmp.fhuA7ZmtWn 2020-06-12T02:37:01.961 INFO:teuthology.orchestra.run.smithi188:> sudo cp /var/lib/ceph/osd/ceph-4/fsid /tmp/tmp.fhuA7ZmtWn 2020-06-12T02:37:02.029 INFO:teuthology.orchestra.run.smithi188:> sudo chmod 0666 /tmp/tmp.fhuA7ZmtWn 2020-06-12T02:37:02.154 DEBUG:teuthology.orchestra.remote:smithi188:/tmp/tmp.fhuA7ZmtWn is 37B 2020-06-12T02:37:02.202 INFO:teuthology.orchestra.run.smithi188:> rm -fr /tmp/tmp.fhuA7ZmtWn 2020-06-12T02:37:02.218 INFO:teuthology.orchestra.run.smithi188:> mktemp 2020-06-12T02:37:02.269 INFO:teuthology.orchestra.run.smithi188.stdout:/tmp/tmp.Ifcc6muGl2 2020-06-12T02:37:02.270 INFO:teuthology.orchestra.run.smithi188:> sudo cp /var/lib/ceph/osd/ceph-5/fsid /tmp/tmp.Ifcc6muGl2 2020-06-12T02:37:02.339 INFO:teuthology.orchestra.run.smithi188:> sudo chmod 0666 /tmp/tmp.Ifcc6muGl2 2020-06-12T02:37:02.472 DEBUG:teuthology.orchestra.remote:smithi188:/tmp/tmp.Ifcc6muGl2 is 37B 2020-06-12T02:37:02.520 INFO:teuthology.orchestra.run.smithi188:> rm -fr /tmp/tmp.Ifcc6muGl2 2020-06-12T02:37:02.534 INFO:teuthology.orchestra.run.smithi188:> mktemp 2020-06-12T02:37:02.587 INFO:teuthology.orchestra.run.smithi188.stdout:/tmp/tmp.l3YIffYW2G 2020-06-12T02:37:02.588 INFO:teuthology.orchestra.run.smithi188:> sudo cp /var/lib/ceph/osd/ceph-6/fsid /tmp/tmp.l3YIffYW2G 2020-06-12T02:37:02.655 INFO:teuthology.orchestra.run.smithi188:> sudo chmod 0666 /tmp/tmp.l3YIffYW2G 2020-06-12T02:37:02.780 DEBUG:teuthology.orchestra.remote:smithi188:/tmp/tmp.l3YIffYW2G is 37B 2020-06-12T02:37:02.798 INFO:teuthology.orchestra.run.smithi188:> rm -fr /tmp/tmp.l3YIffYW2G 2020-06-12T02:37:02.812 INFO:teuthology.orchestra.run.smithi188:> mktemp 2020-06-12T02:37:02.868 INFO:teuthology.orchestra.run.smithi188.stdout:/tmp/tmp.be3gMs2O79 2020-06-12T02:37:02.869 INFO:teuthology.orchestra.run.smithi188:> sudo cp /var/lib/ceph/osd/ceph-7/fsid /tmp/tmp.be3gMs2O79 2020-06-12T02:37:02.945 INFO:teuthology.orchestra.run.smithi188:> sudo chmod 0666 /tmp/tmp.be3gMs2O79 2020-06-12T02:37:03.152 DEBUG:teuthology.orchestra.remote:smithi188:/tmp/tmp.be3gMs2O79 is 37B 2020-06-12T02:37:04.172 INFO:teuthology.orchestra.run.smithi188:> rm -fr /tmp/tmp.be3gMs2O79 2020-06-12T02:37:04.186 INFO:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new ad4b4814-83e7-4a2e-9b1f-d10406174f7e 0 2020-06-12T02:37:04.423 INFO:teuthology.orchestra.run.smithi188.stdout:0 2020-06-12T02:37:04.434 INFO:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new 046adf0d-b9e9-41ad-a7c9-3841f8b8f790 1 2020-06-12T02:37:04.638 INFO:teuthology.orchestra.run.smithi188.stdout:1 2020-06-12T02:37:04.649 INFO:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new e15da6b7-dc79-4185-a5f9-0f443d8b52b6 2 2020-06-12T02:37:04.922 INFO:teuthology.orchestra.run.smithi188.stdout:2 2020-06-12T02:37:04.934 INFO:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new 843e0dd3-5ce4-46b9-9004-69849048d2bd 3 2020-06-12T02:37:05.206 INFO:teuthology.orchestra.run.smithi188.stdout:3 2020-06-12T02:37:05.218 INFO:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new c066ab09-e031-4b49-9df3-d54cd9f9b580 4 2020-06-12T02:37:05.492 INFO:teuthology.orchestra.run.smithi188.stdout:4 2020-06-12T02:37:05.504 INFO:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new 56a24e45-1b1b-4a65-b4b4-57cfab296f65 5 2020-06-12T02:37:05.781 INFO:teuthology.orchestra.run.smithi188.stdout:5 2020-06-12T02:37:05.793 INFO:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new f207a2a8-b3c7-4fd8-ae90-4c2d7a5c1c38 6 2020-06-12T02:37:06.065 INFO:teuthology.orchestra.run.smithi188.stdout:6 2020-06-12T02:37:06.077 INFO:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new 4e5cc44f-af17-47c9-973f-8e87d1bd98db 7 2020-06-12T02:37:06.352 INFO:teuthology.orchestra.run.smithi188.stdout:7 2020-06-12T02:37:06.361 INFO:tasks.ceph.osd.0:Restarting daemon 2020-06-12T02:37:06.362 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2020-06-12T02:37:06.365 INFO:tasks.ceph.osd.0:Started 2020-06-12T02:37:06.366 INFO:tasks.ceph.osd.1:Restarting daemon 2020-06-12T02:37:06.366 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2020-06-12T02:37:06.368 INFO:tasks.ceph.osd.1:Started 2020-06-12T02:37:06.369 INFO:tasks.ceph.osd.2:Restarting daemon 2020-06-12T02:37:06.369 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2020-06-12T02:37:06.371 INFO:tasks.ceph.osd.2:Started 2020-06-12T02:37:06.372 INFO:tasks.ceph.osd.3:Restarting daemon 2020-06-12T02:37:06.372 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2020-06-12T02:37:06.377 INFO:tasks.ceph.osd.3:Started 2020-06-12T02:37:06.377 INFO:tasks.ceph.osd.4:Restarting daemon 2020-06-12T02:37:06.378 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2020-06-12T02:37:06.380 INFO:tasks.ceph.osd.4:Started 2020-06-12T02:37:06.381 INFO:tasks.ceph.osd.5:Restarting daemon 2020-06-12T02:37:06.381 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2020-06-12T02:37:06.386 INFO:tasks.ceph.osd.5:Started 2020-06-12T02:37:06.387 INFO:tasks.ceph.osd.6:Restarting daemon 2020-06-12T02:37:06.387 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2020-06-12T02:37:06.389 INFO:tasks.ceph.osd.6:Started 2020-06-12T02:37:06.389 INFO:tasks.ceph.osd.7:Restarting daemon 2020-06-12T02:37:06.390 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2020-06-12T02:37:06.392 INFO:tasks.ceph.osd.7:Started 2020-06-12T02:37:06.392 INFO:tasks.ceph:Waiting for OSDs to come up 2020-06-12T02:37:06.976 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:37:06.973+0000 7f76af09fec0 -1 Falling back to public interface 2020-06-12T02:37:06.980 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:37:06.978+0000 7fa5f8444ec0 -1 Falling back to public interface 2020-06-12T02:37:06.984 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:37:06.981+0000 7f0c97189ec0 -1 Falling back to public interface 2020-06-12T02:37:06.988 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:06.986+0000 7fa0775a3ec0 -1 Falling back to public interface 2020-06-12T02:37:06.989 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:06.987+0000 7f2397e63ec0 -1 Falling back to public interface 2020-06-12T02:37:06.990 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:37:06.987+0000 7f87ca4f0ec0 -1 Falling back to public interface 2020-06-12T02:37:06.995 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:06.992+0000 7fd21d976ec0 -1 Falling back to public interface 2020-06-12T02:37:06.995 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:06.992+0000 7f464ea2aec0 -1 Falling back to public interface 2020-06-12T02:37:06.999 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:06.996+0000 7fd218194700 -1 PosixStack listen unable to listen on v1:0.0.0.0:6827/0: (98) Address already in use 2020-06-12T02:37:07.200 INFO:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2020-06-12T02:37:07.467 INFO:teuthology.misc.health.smithi107.stdout: 2020-06-12T02:37:07.468 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"2f5030a1-3206-43c9-8471-d10429688d36","created":"2020-06-12T02:37:00.578493+0000","modified":"2020-06-12T02:37:06.345617+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-06-12T02:37:03.610773+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"4","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"ad4b4814-83e7-4a2e-9b1f-d10406174f7e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"046adf0d-b9e9-41ad-a7c9-3841f8b8f790","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e15da6b7-dc79-4185-a5f9-0f443d8b52b6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"843e0dd3-5ce4-46b9-9004-69849048d2bd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c066ab09-e031-4b49-9df3-d54cd9f9b580","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"56a24e45-1b1b-4a65-b4b4-57cfab296f65","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f207a2a8-b3c7-4fd8-ae90-4c2d7a5c1c38","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e5cc44f-af17-47c9-973f-8e87d1bd98db","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-06-12T02:37:07.479 DEBUG:teuthology.misc:0 of 8 OSDs are up 2020-06-12T02:37:07.822 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:07.820+0000 7fd21d976ec0 -1 osd.4 0 log_to_monitors {default=true} 2020-06-12T02:37:08.045 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:37:08.043+0000 7f76af09fec0 -1 osd.1 0 log_to_monitors {default=true} 2020-06-12T02:37:08.056 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:37:08.052+0000 7fa5f8444ec0 -1 osd.3 0 log_to_monitors {default=true} 2020-06-12T02:37:08.056 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:37:08.046+0000 7f0c97189ec0 -1 osd.7 0 log_to_monitors {default=true} 2020-06-12T02:37:08.061 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:37:08.059+0000 7f87ca4f0ec0 -1 osd.6 0 log_to_monitors {default=true} 2020-06-12T02:37:08.067 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:08.065+0000 7fa0775a3ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-06-12T02:37:08.071 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:08.069+0000 7f2397e63ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-06-12T02:37:08.073 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:08.071+0000 7f464ea2aec0 -1 osd.5 0 log_to_monitors {default=true} 2020-06-12T02:37:09.711 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:37:09.708+0000 7f0c7f696700 -1 osd.7 0 waiting for initial osdmap 2020-06-12T02:37:09.712 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:37:09.707+0000 7f76975ac700 -1 osd.1 0 waiting for initial osdmap 2020-06-12T02:37:09.714 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:37:09.710+0000 7fa5e0951700 -1 osd.3 0 waiting for initial osdmap 2020-06-12T02:37:09.714 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:09.710+0000 7fa05fab0700 -1 osd.0 0 waiting for initial osdmap 2020-06-12T02:37:09.714 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:09.710+0000 7fd205e83700 -1 osd.4 0 waiting for initial osdmap 2020-06-12T02:37:09.714 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:09.711+0000 7f4636f37700 -1 osd.5 0 waiting for initial osdmap 2020-06-12T02:37:09.715 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:09.711+0000 7f2380370700 -1 osd.2 0 waiting for initial osdmap 2020-06-12T02:37:09.719 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:37:09.717+0000 7f87b29fd700 -1 osd.6 0 waiting for initial osdmap 2020-06-12T02:37:14.288 INFO:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2020-06-12T02:37:14.553 INFO:teuthology.misc.health.smithi107.stdout: 2020-06-12T02:37:14.554 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":15,"fsid":"2f5030a1-3206-43c9-8471-d10429688d36","created":"2020-06-12T02:37:00.578493+0000","modified":"2020-06-12T02:37:12.724594+0000","last_up_change":"2020-06-12T02:37:10.705891+0000","last_in_change":"2020-06-12T02:37:10.705891+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-06-12T02:37:03.610773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"ad4b4814-83e7-4a2e-9b1f-d10406174f7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":26614},{"type":"v1","addr":"172.21.15.107:6817","nonce":26614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6819","nonce":26614},{"type":"v1","addr":"172.21.15.107:6821","nonce":26614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6827","nonce":26614},{"type":"v1","addr":"172.21.15.107:6829","nonce":26614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":26614},{"type":"v1","addr":"172.21.15.107:6825","nonce":26614}]},"public_addr":"172.21.15.107:6817/26614","cluster_addr":"172.21.15.107:6821/26614","heartbeat_back_addr":"172.21.15.107:6829/26614","heartbeat_front_addr":"172.21.15.107:6825/26614","state":["exists","up"]},{"osd":1,"uuid":"046adf0d-b9e9-41ad-a7c9-3841f8b8f790","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":26611},{"type":"v1","addr":"172.21.15.107:6801","nonce":26611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":26611},{"type":"v1","addr":"172.21.15.107:6803","nonce":26611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":26611},{"type":"v1","addr":"172.21.15.107:6807","nonce":26611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":26611},{"type":"v1","addr":"172.21.15.107:6805","nonce":26611}]},"public_addr":"172.21.15.107:6801/26611","cluster_addr":"172.21.15.107:6803/26611","heartbeat_back_addr":"172.21.15.107:6807/26611","heartbeat_front_addr":"172.21.15.107:6805/26611","state":["exists","up"]},{"osd":2,"uuid":"e15da6b7-dc79-4185-a5f9-0f443d8b52b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":26613},{"type":"v1","addr":"172.21.15.107:6820","nonce":26613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":26613},{"type":"v1","addr":"172.21.15.107:6824","nonce":26613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":26613},{"type":"v1","addr":"172.21.15.107:6831","nonce":26613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":26613},{"type":"v1","addr":"172.21.15.107:6828","nonce":26613}]},"public_addr":"172.21.15.107:6820/26613","cluster_addr":"172.21.15.107:6824/26613","heartbeat_back_addr":"172.21.15.107:6831/26613","heartbeat_front_addr":"172.21.15.107:6828/26613","state":["exists","up"]},{"osd":3,"uuid":"843e0dd3-5ce4-46b9-9004-69849048d2bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":26612},{"type":"v1","addr":"172.21.15.107:6809","nonce":26612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":26612},{"type":"v1","addr":"172.21.15.107:6811","nonce":26612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":26612},{"type":"v1","addr":"172.21.15.107:6815","nonce":26612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":26612},{"type":"v1","addr":"172.21.15.107:6813","nonce":26612}]},"public_addr":"172.21.15.107:6809/26612","cluster_addr":"172.21.15.107:6811/26612","heartbeat_back_addr":"172.21.15.107:6815/26612","heartbeat_front_addr":"172.21.15.107:6813/26612","state":["exists","up"]},{"osd":4,"uuid":"c066ab09-e031-4b49-9df3-d54cd9f9b580","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":26685},{"type":"v1","addr":"172.21.15.188:6820","nonce":26685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":26685},{"type":"v1","addr":"172.21.15.188:6823","nonce":26685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":26685},{"type":"v1","addr":"172.21.15.188:6832","nonce":26685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6825","nonce":26685},{"type":"v1","addr":"172.21.15.188:6828","nonce":26685}]},"public_addr":"172.21.15.188:6820/26685","cluster_addr":"172.21.15.188:6823/26685","heartbeat_back_addr":"172.21.15.188:6832/26685","heartbeat_front_addr":"172.21.15.188:6828/26685","state":["exists","up"]},{"osd":5,"uuid":"56a24e45-1b1b-4a65-b4b4-57cfab296f65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6819","nonce":26684},{"type":"v1","addr":"172.21.15.188:6821","nonce":26684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":26684},{"type":"v1","addr":"172.21.15.188:6826","nonce":26684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6831","nonce":26684},{"type":"v1","addr":"172.21.15.188:6833","nonce":26684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6827","nonce":26684},{"type":"v1","addr":"172.21.15.188:6829","nonce":26684}]},"public_addr":"172.21.15.188:6821/26684","cluster_addr":"172.21.15.188:6826/26684","heartbeat_back_addr":"172.21.15.188:6833/26684","heartbeat_front_addr":"172.21.15.188:6829/26684","state":["exists","up"]},{"osd":6,"uuid":"f207a2a8-b3c7-4fd8-ae90-4c2d7a5c1c38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":26682},{"type":"v1","addr":"172.21.15.188:6811","nonce":26682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":26682},{"type":"v1","addr":"172.21.15.188:6813","nonce":26682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":26682},{"type":"v1","addr":"172.21.15.188:6817","nonce":26682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":26682},{"type":"v1","addr":"172.21.15.188:6815","nonce":26682}]},"public_addr":"172.21.15.188:6811/26682","cluster_addr":"172.21.15.188:6813/26682","heartbeat_back_addr":"172.21.15.188:6817/26682","heartbeat_front_addr":"172.21.15.188:6815/26682","state":["exists","up"]},{"osd":7,"uuid":"4e5cc44f-af17-47c9-973f-8e87d1bd98db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":26683},{"type":"v1","addr":"172.21.15.188:6803","nonce":26683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":26683},{"type":"v1","addr":"172.21.15.188:6805","nonce":26683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":26683},{"type":"v1","addr":"172.21.15.188:6809","nonce":26683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":26683},{"type":"v1","addr":"172.21.15.188:6807","nonce":26683}]},"public_addr":"172.21.15.188:6803/26683","cluster_addr":"172.21.15.188:6805/26683","heartbeat_back_addr":"172.21.15.188:6809/26683","heartbeat_front_addr":"172.21.15.188:6807/26683","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:37:09.019033+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:37:09.092469+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:37:09.056721+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:37:09.006828+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:37:08.775848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:37:09.087140+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:37:09.034074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:37:09.033458+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-06-12T02:37:14.562 DEBUG:teuthology.misc:8 of 8 OSDs are up 2020-06-12T02:37:14.562 INFO:tasks.ceph:Creating RBD pool 2020-06-12T02:37:14.563 INFO:teuthology.orchestra.run.smithi107:> sudo ceph --cluster ceph osd pool create rbd 8 2020-06-12T02:37:15.756 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'rbd' created 2020-06-12T02:37:15.768 INFO:teuthology.orchestra.run.smithi107:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2020-06-12T02:37:16.759 INFO:teuthology.orchestra.run.smithi107.stderr:enabled application 'rbd' on pool 'rbd' 2020-06-12T02:37:16.773 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2020-06-12T02:37:16.774 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-06-12T02:37:16.775 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2020-06-12T02:37:17.038 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:37:17.039 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":17,"fsid":"2f5030a1-3206-43c9-8471-d10429688d36","created":"2020-06-12T02:37:00.578493+0000","modified":"2020-06-12T02:37:16.746920+0000","last_up_change":"2020-06-12T02:37:10.705891+0000","last_in_change":"2020-06-12T02:37:10.705891+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-06-12T02:37:03.610773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-06-12T02:37:14.831188+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"ad4b4814-83e7-4a2e-9b1f-d10406174f7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":26614},{"type":"v1","addr":"172.21.15.107:6817","nonce":26614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6819","nonce":26614},{"type":"v1","addr":"172.21.15.107:6821","nonce":26614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6827","nonce":26614},{"type":"v1","addr":"172.21.15.107:6829","nonce":26614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":26614},{"type":"v1","addr":"172.21.15.107:6825","nonce":26614}]},"public_addr":"172.21.15.107:6817/26614","cluster_addr":"172.21.15.107:6821/26614","heartbeat_back_addr":"172.21.15.107:6829/26614","heartbeat_front_addr":"172.21.15.107:6825/26614","state":["exists","up"]},{"osd":1,"uuid":"046adf0d-b9e9-41ad-a7c9-3841f8b8f790","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":26611},{"type":"v1","addr":"172.21.15.107:6801","nonce":26611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":26611},{"type":"v1","addr":"172.21.15.107:6803","nonce":26611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":26611},{"type":"v1","addr":"172.21.15.107:6807","nonce":26611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":26611},{"type":"v1","addr":"172.21.15.107:6805","nonce":26611}]},"public_addr":"172.21.15.107:6801/26611","cluster_addr":"172.21.15.107:6803/26611","heartbeat_back_addr":"172.21.15.107:6807/26611","heartbeat_front_addr":"172.21.15.107:6805/26611","state":["exists","up"]},{"osd":2,"uuid":"e15da6b7-dc79-4185-a5f9-0f443d8b52b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":26613},{"type":"v1","addr":"172.21.15.107:6820","nonce":26613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":26613},{"type":"v1","addr":"172.21.15.107:6824","nonce":26613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":26613},{"type":"v1","addr":"172.21.15.107:6831","nonce":26613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":26613},{"type":"v1","addr":"172.21.15.107:6828","nonce":26613}]},"public_addr":"172.21.15.107:6820/26613","cluster_addr":"172.21.15.107:6824/26613","heartbeat_back_addr":"172.21.15.107:6831/26613","heartbeat_front_addr":"172.21.15.107:6828/26613","state":["exists","up"]},{"osd":3,"uuid":"843e0dd3-5ce4-46b9-9004-69849048d2bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":26612},{"type":"v1","addr":"172.21.15.107:6809","nonce":26612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":26612},{"type":"v1","addr":"172.21.15.107:6811","nonce":26612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":26612},{"type":"v1","addr":"172.21.15.107:6815","nonce":26612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":26612},{"type":"v1","addr":"172.21.15.107:6813","nonce":26612}]},"public_addr":"172.21.15.107:6809/26612","cluster_addr":"172.21.15.107:6811/26612","heartbeat_back_addr":"172.21.15.107:6815/26612","heartbeat_front_addr":"172.21.15.107:6813/26612","state":["exists","up"]},{"osd":4,"uuid":"c066ab09-e031-4b49-9df3-d54cd9f9b580","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":26685},{"type":"v1","addr":"172.21.15.188:6820","nonce":26685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":26685},{"type":"v1","addr":"172.21.15.188:6823","nonce":26685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":26685},{"type":"v1","addr":"172.21.15.188:6832","nonce":26685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6825","nonce":26685},{"type":"v1","addr":"172.21.15.188:6828","nonce":26685}]},"public_addr":"172.21.15.188:6820/26685","cluster_addr":"172.21.15.188:6823/26685","heartbeat_back_addr":"172.21.15.188:6832/26685","heartbeat_front_addr":"172.21.15.188:6828/26685","state":["exists","up"]},{"osd":5,"uuid":"56a24e45-1b1b-4a65-b4b4-57cfab296f65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6819","nonce":26684},{"type":"v1","addr":"172.21.15.188:6821","nonce":26684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":26684},{"type":"v1","addr":"172.21.15.188:6826","nonce":26684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6831","nonce":26684},{"type":"v1","addr":"172.21.15.188:6833","nonce":26684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6827","nonce":26684},{"type":"v1","addr":"172.21.15.188:6829","nonce":26684}]},"public_addr":"172.21.15.188:6821/26684","cluster_addr":"172.21.15.188:6826/26684","heartbeat_back_addr":"172.21.15.188:6833/26684","heartbeat_front_addr":"172.21.15.188:6829/26684","state":["exists","up"]},{"osd":6,"uuid":"f207a2a8-b3c7-4fd8-ae90-4c2d7a5c1c38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":26682},{"type":"v1","addr":"172.21.15.188:6811","nonce":26682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":26682},{"type":"v1","addr":"172.21.15.188:6813","nonce":26682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":26682},{"type":"v1","addr":"172.21.15.188:6817","nonce":26682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":26682},{"type":"v1","addr":"172.21.15.188:6815","nonce":26682}]},"public_addr":"172.21.15.188:6811/26682","cluster_addr":"172.21.15.188:6813/26682","heartbeat_back_addr":"172.21.15.188:6817/26682","heartbeat_front_addr":"172.21.15.188:6815/26682","state":["exists","up"]},{"osd":7,"uuid":"4e5cc44f-af17-47c9-973f-8e87d1bd98db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":26683},{"type":"v1","addr":"172.21.15.188:6803","nonce":26683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":26683},{"type":"v1","addr":"172.21.15.188:6805","nonce":26683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":26683},{"type":"v1","addr":"172.21.15.188:6809","nonce":26683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":26683},{"type":"v1","addr":"172.21.15.188:6807","nonce":26683}]},"public_addr":"172.21.15.188:6803/26683","cluster_addr":"172.21.15.188:6805/26683","heartbeat_back_addr":"172.21.15.188:6809/26683","heartbeat_front_addr":"172.21.15.188:6807/26683","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:37:09.019033+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:37:09.092469+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:37:09.056721+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:37:09.006828+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:37:08.775848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:37:09.087140+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:37:09.034074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:37:09.033458+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-06-12T02:37:17.049 INFO:tasks.ceph.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-06-12T02:37:03.610773+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-06-12T02:37:14.831188+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-06-12T02:37:17.050 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd pool get device_health_metrics pg_num 2020-06-12T02:37:17.306 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2020-06-12T02:37:17.317 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd pool get rbd pg_num 2020-06-12T02:37:17.574 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 8 2020-06-12T02:37:17.586 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2020-06-12T02:37:17.586 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2020-06-12T02:37:17.587 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early mgr dump --format=json 2020-06-12T02:37:17.885 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:37:17.895 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":4,"active_gid":4120,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":25743},{"type":"v1","addr":"172.21.15.188:6801","nonce":25743}]},"active_addr":"172.21.15.188:6801/25743","active_change":"2020-06-12T02:37:03.588112+0000","active_mgr_features":4540138292836696063,"available":true,"standbys":[{"gid":4100,"name":"y","mgr_features":4540138292836696063,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","osd_support","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.188:0","nonce":476746649}]},{"addrvec":[{"type":"v2","addr":"172.21.15.188:0","nonce":170529970}]},{"addrvec":[{"type":"v2","addr":"172.21.15.188:0","nonce":953232552}]}]}} 2020-06-12T02:37:17.899 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2020-06-12T02:37:17.900 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2020-06-12T02:37:17.900 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-06-12T02:37:18.158 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:37:18.159 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":18,"fsid":"2f5030a1-3206-43c9-8471-d10429688d36","created":"2020-06-12T02:37:00.578493+0000","modified":"2020-06-12T02:37:17.759876+0000","last_up_change":"2020-06-12T02:37:10.705891+0000","last_in_change":"2020-06-12T02:37:10.705891+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-06-12T02:37:03.610773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-06-12T02:37:14.831188+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"ad4b4814-83e7-4a2e-9b1f-d10406174f7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":26614},{"type":"v1","addr":"172.21.15.107:6817","nonce":26614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6819","nonce":26614},{"type":"v1","addr":"172.21.15.107:6821","nonce":26614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6827","nonce":26614},{"type":"v1","addr":"172.21.15.107:6829","nonce":26614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":26614},{"type":"v1","addr":"172.21.15.107:6825","nonce":26614}]},"public_addr":"172.21.15.107:6817/26614","cluster_addr":"172.21.15.107:6821/26614","heartbeat_back_addr":"172.21.15.107:6829/26614","heartbeat_front_addr":"172.21.15.107:6825/26614","state":["exists","up"]},{"osd":1,"uuid":"046adf0d-b9e9-41ad-a7c9-3841f8b8f790","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":26611},{"type":"v1","addr":"172.21.15.107:6801","nonce":26611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":26611},{"type":"v1","addr":"172.21.15.107:6803","nonce":26611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":26611},{"type":"v1","addr":"172.21.15.107:6807","nonce":26611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":26611},{"type":"v1","addr":"172.21.15.107:6805","nonce":26611}]},"public_addr":"172.21.15.107:6801/26611","cluster_addr":"172.21.15.107:6803/26611","heartbeat_back_addr":"172.21.15.107:6807/26611","heartbeat_front_addr":"172.21.15.107:6805/26611","state":["exists","up"]},{"osd":2,"uuid":"e15da6b7-dc79-4185-a5f9-0f443d8b52b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":26613},{"type":"v1","addr":"172.21.15.107:6820","nonce":26613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":26613},{"type":"v1","addr":"172.21.15.107:6824","nonce":26613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":26613},{"type":"v1","addr":"172.21.15.107:6831","nonce":26613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":26613},{"type":"v1","addr":"172.21.15.107:6828","nonce":26613}]},"public_addr":"172.21.15.107:6820/26613","cluster_addr":"172.21.15.107:6824/26613","heartbeat_back_addr":"172.21.15.107:6831/26613","heartbeat_front_addr":"172.21.15.107:6828/26613","state":["exists","up"]},{"osd":3,"uuid":"843e0dd3-5ce4-46b9-9004-69849048d2bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":26612},{"type":"v1","addr":"172.21.15.107:6809","nonce":26612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":26612},{"type":"v1","addr":"172.21.15.107:6811","nonce":26612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":26612},{"type":"v1","addr":"172.21.15.107:6815","nonce":26612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":26612},{"type":"v1","addr":"172.21.15.107:6813","nonce":26612}]},"public_addr":"172.21.15.107:6809/26612","cluster_addr":"172.21.15.107:6811/26612","heartbeat_back_addr":"172.21.15.107:6815/26612","heartbeat_front_addr":"172.21.15.107:6813/26612","state":["exists","up"]},{"osd":4,"uuid":"c066ab09-e031-4b49-9df3-d54cd9f9b580","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":26685},{"type":"v1","addr":"172.21.15.188:6820","nonce":26685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":26685},{"type":"v1","addr":"172.21.15.188:6823","nonce":26685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":26685},{"type":"v1","addr":"172.21.15.188:6832","nonce":26685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6825","nonce":26685},{"type":"v1","addr":"172.21.15.188:6828","nonce":26685}]},"public_addr":"172.21.15.188:6820/26685","cluster_addr":"172.21.15.188:6823/26685","heartbeat_back_addr":"172.21.15.188:6832/26685","heartbeat_front_addr":"172.21.15.188:6828/26685","state":["exists","up"]},{"osd":5,"uuid":"56a24e45-1b1b-4a65-b4b4-57cfab296f65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6819","nonce":26684},{"type":"v1","addr":"172.21.15.188:6821","nonce":26684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":26684},{"type":"v1","addr":"172.21.15.188:6826","nonce":26684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6831","nonce":26684},{"type":"v1","addr":"172.21.15.188:6833","nonce":26684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6827","nonce":26684},{"type":"v1","addr":"172.21.15.188:6829","nonce":26684}]},"public_addr":"172.21.15.188:6821/26684","cluster_addr":"172.21.15.188:6826/26684","heartbeat_back_addr":"172.21.15.188:6833/26684","heartbeat_front_addr":"172.21.15.188:6829/26684","state":["exists","up"]},{"osd":6,"uuid":"f207a2a8-b3c7-4fd8-ae90-4c2d7a5c1c38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":26682},{"type":"v1","addr":"172.21.15.188:6811","nonce":26682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":26682},{"type":"v1","addr":"172.21.15.188:6813","nonce":26682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":26682},{"type":"v1","addr":"172.21.15.188:6817","nonce":26682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":26682},{"type":"v1","addr":"172.21.15.188:6815","nonce":26682}]},"public_addr":"172.21.15.188:6811/26682","cluster_addr":"172.21.15.188:6813/26682","heartbeat_back_addr":"172.21.15.188:6817/26682","heartbeat_front_addr":"172.21.15.188:6815/26682","state":["exists","up"]},{"osd":7,"uuid":"4e5cc44f-af17-47c9-973f-8e87d1bd98db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":26683},{"type":"v1","addr":"172.21.15.188:6803","nonce":26683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":26683},{"type":"v1","addr":"172.21.15.188:6805","nonce":26683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":26683},{"type":"v1","addr":"172.21.15.188:6809","nonce":26683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":26683},{"type":"v1","addr":"172.21.15.188:6807","nonce":26683}]},"public_addr":"172.21.15.188:6803/26683","cluster_addr":"172.21.15.188:6805/26683","heartbeat_back_addr":"172.21.15.188:6809/26683","heartbeat_front_addr":"172.21.15.188:6807/26683","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:37:09.019033+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:37:09.092469+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:37:09.056721+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:37:09.006828+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:37:08.775848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:37:09.087140+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:37:09.034074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:37:09.033458+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-06-12T02:37:18.168 INFO:tasks.ceph.ceph_manager.ceph:all up! 2020-06-12T02:37:18.169 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-06-12T02:37:18.424 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:37:18.425 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":18,"fsid":"2f5030a1-3206-43c9-8471-d10429688d36","created":"2020-06-12T02:37:00.578493+0000","modified":"2020-06-12T02:37:17.759876+0000","last_up_change":"2020-06-12T02:37:10.705891+0000","last_in_change":"2020-06-12T02:37:10.705891+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-06-12T02:37:03.610773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-06-12T02:37:14.831188+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"ad4b4814-83e7-4a2e-9b1f-d10406174f7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":26614},{"type":"v1","addr":"172.21.15.107:6817","nonce":26614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6819","nonce":26614},{"type":"v1","addr":"172.21.15.107:6821","nonce":26614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6827","nonce":26614},{"type":"v1","addr":"172.21.15.107:6829","nonce":26614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":26614},{"type":"v1","addr":"172.21.15.107:6825","nonce":26614}]},"public_addr":"172.21.15.107:6817/26614","cluster_addr":"172.21.15.107:6821/26614","heartbeat_back_addr":"172.21.15.107:6829/26614","heartbeat_front_addr":"172.21.15.107:6825/26614","state":["exists","up"]},{"osd":1,"uuid":"046adf0d-b9e9-41ad-a7c9-3841f8b8f790","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":26611},{"type":"v1","addr":"172.21.15.107:6801","nonce":26611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":26611},{"type":"v1","addr":"172.21.15.107:6803","nonce":26611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":26611},{"type":"v1","addr":"172.21.15.107:6807","nonce":26611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":26611},{"type":"v1","addr":"172.21.15.107:6805","nonce":26611}]},"public_addr":"172.21.15.107:6801/26611","cluster_addr":"172.21.15.107:6803/26611","heartbeat_back_addr":"172.21.15.107:6807/26611","heartbeat_front_addr":"172.21.15.107:6805/26611","state":["exists","up"]},{"osd":2,"uuid":"e15da6b7-dc79-4185-a5f9-0f443d8b52b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":26613},{"type":"v1","addr":"172.21.15.107:6820","nonce":26613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":26613},{"type":"v1","addr":"172.21.15.107:6824","nonce":26613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":26613},{"type":"v1","addr":"172.21.15.107:6831","nonce":26613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":26613},{"type":"v1","addr":"172.21.15.107:6828","nonce":26613}]},"public_addr":"172.21.15.107:6820/26613","cluster_addr":"172.21.15.107:6824/26613","heartbeat_back_addr":"172.21.15.107:6831/26613","heartbeat_front_addr":"172.21.15.107:6828/26613","state":["exists","up"]},{"osd":3,"uuid":"843e0dd3-5ce4-46b9-9004-69849048d2bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":26612},{"type":"v1","addr":"172.21.15.107:6809","nonce":26612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":26612},{"type":"v1","addr":"172.21.15.107:6811","nonce":26612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":26612},{"type":"v1","addr":"172.21.15.107:6815","nonce":26612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":26612},{"type":"v1","addr":"172.21.15.107:6813","nonce":26612}]},"public_addr":"172.21.15.107:6809/26612","cluster_addr":"172.21.15.107:6811/26612","heartbeat_back_addr":"172.21.15.107:6815/26612","heartbeat_front_addr":"172.21.15.107:6813/26612","state":["exists","up"]},{"osd":4,"uuid":"c066ab09-e031-4b49-9df3-d54cd9f9b580","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":26685},{"type":"v1","addr":"172.21.15.188:6820","nonce":26685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":26685},{"type":"v1","addr":"172.21.15.188:6823","nonce":26685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":26685},{"type":"v1","addr":"172.21.15.188:6832","nonce":26685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6825","nonce":26685},{"type":"v1","addr":"172.21.15.188:6828","nonce":26685}]},"public_addr":"172.21.15.188:6820/26685","cluster_addr":"172.21.15.188:6823/26685","heartbeat_back_addr":"172.21.15.188:6832/26685","heartbeat_front_addr":"172.21.15.188:6828/26685","state":["exists","up"]},{"osd":5,"uuid":"56a24e45-1b1b-4a65-b4b4-57cfab296f65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6819","nonce":26684},{"type":"v1","addr":"172.21.15.188:6821","nonce":26684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":26684},{"type":"v1","addr":"172.21.15.188:6826","nonce":26684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6831","nonce":26684},{"type":"v1","addr":"172.21.15.188:6833","nonce":26684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6827","nonce":26684},{"type":"v1","addr":"172.21.15.188:6829","nonce":26684}]},"public_addr":"172.21.15.188:6821/26684","cluster_addr":"172.21.15.188:6826/26684","heartbeat_back_addr":"172.21.15.188:6833/26684","heartbeat_front_addr":"172.21.15.188:6829/26684","state":["exists","up"]},{"osd":6,"uuid":"f207a2a8-b3c7-4fd8-ae90-4c2d7a5c1c38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":26682},{"type":"v1","addr":"172.21.15.188:6811","nonce":26682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":26682},{"type":"v1","addr":"172.21.15.188:6813","nonce":26682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":26682},{"type":"v1","addr":"172.21.15.188:6817","nonce":26682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":26682},{"type":"v1","addr":"172.21.15.188:6815","nonce":26682}]},"public_addr":"172.21.15.188:6811/26682","cluster_addr":"172.21.15.188:6813/26682","heartbeat_back_addr":"172.21.15.188:6817/26682","heartbeat_front_addr":"172.21.15.188:6815/26682","state":["exists","up"]},{"osd":7,"uuid":"4e5cc44f-af17-47c9-973f-8e87d1bd98db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":26683},{"type":"v1","addr":"172.21.15.188:6803","nonce":26683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":26683},{"type":"v1","addr":"172.21.15.188:6805","nonce":26683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":26683},{"type":"v1","addr":"172.21.15.188:6809","nonce":26683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":26683},{"type":"v1","addr":"172.21.15.188:6807","nonce":26683}]},"public_addr":"172.21.15.188:6803/26683","cluster_addr":"172.21.15.188:6805/26683","heartbeat_back_addr":"172.21.15.188:6809/26683","heartbeat_front_addr":"172.21.15.188:6807/26683","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:37:09.019033+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:37:09.092469+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:37:09.056721+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:37:09.006828+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:37:08.775848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:37:09.087140+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:37:09.034074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:37:09.033458+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-06-12T02:37:18.443 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.0 flush_pg_stats 2020-06-12T02:37:18.601 INFO:teuthology.orchestra.run.smithi107.stdout:55834574853 2020-06-12T02:37:18.602 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.1 flush_pg_stats 2020-06-12T02:37:18.743 INFO:teuthology.orchestra.run.smithi107.stdout:55834574853 2020-06-12T02:37:18.743 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.2 flush_pg_stats 2020-06-12T02:37:18.884 INFO:teuthology.orchestra.run.smithi107.stdout:55834574852 2020-06-12T02:37:18.884 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.3 flush_pg_stats 2020-06-12T02:37:19.027 INFO:teuthology.orchestra.run.smithi107.stdout:55834574852 2020-06-12T02:37:19.028 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.4 flush_pg_stats 2020-06-12T02:37:19.170 INFO:teuthology.orchestra.run.smithi107.stdout:55834574851 2020-06-12T02:37:19.171 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.5 flush_pg_stats 2020-06-12T02:37:19.312 INFO:teuthology.orchestra.run.smithi107.stdout:55834574851 2020-06-12T02:37:19.312 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.6 flush_pg_stats 2020-06-12T02:37:19.454 INFO:teuthology.orchestra.run.smithi107.stdout:55834574852 2020-06-12T02:37:19.455 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.7 flush_pg_stats 2020-06-12T02:37:19.596 INFO:teuthology.orchestra.run.smithi107.stdout:55834574852 2020-06-12T02:37:19.597 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd last-stat-seq osd.0 2020-06-12T02:37:19.858 INFO:teuthology.orchestra.run.smithi107.stdout:55834574853 2020-06-12T02:37:19.869 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.0 2020-06-12T02:37:19.870 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd last-stat-seq osd.1 2020-06-12T02:37:20.124 INFO:teuthology.orchestra.run.smithi107.stdout:55834574853 2020-06-12T02:37:20.135 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.1 2020-06-12T02:37:20.136 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd last-stat-seq osd.2 2020-06-12T02:37:20.393 INFO:teuthology.orchestra.run.smithi107.stdout:55834574852 2020-06-12T02:37:20.404 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.2 2020-06-12T02:37:20.405 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd last-stat-seq osd.3 2020-06-12T02:37:20.661 INFO:teuthology.orchestra.run.smithi107.stdout:55834574852 2020-06-12T02:37:20.673 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.3 2020-06-12T02:37:20.673 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd last-stat-seq osd.4 2020-06-12T02:37:20.930 INFO:teuthology.orchestra.run.smithi107.stdout:55834574851 2020-06-12T02:37:20.942 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.4 2020-06-12T02:37:20.943 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd last-stat-seq osd.5 2020-06-12T02:37:21.201 INFO:teuthology.orchestra.run.smithi107.stdout:55834574851 2020-06-12T02:37:21.212 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.5 2020-06-12T02:37:21.213 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd last-stat-seq osd.6 2020-06-12T02:37:21.470 INFO:teuthology.orchestra.run.smithi107.stdout:55834574852 2020-06-12T02:37:21.481 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.6 2020-06-12T02:37:21.482 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd last-stat-seq osd.7 2020-06-12T02:37:21.739 INFO:teuthology.orchestra.run.smithi107.stdout:55834574852 2020-06-12T02:37:21.751 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.7 2020-06-12T02:37:21.751 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2020-06-12T02:37:21.752 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json 2020-06-12T02:37:22.006 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:37:22.007 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2020-06-12T02:37:22.016 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2020-06-12T02:37:21.594047+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":8,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":8390020,"kb_used_data":1348,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746584700,"statfs":{"total":773094113280,"available":764502732800,"internally_reserved":8589934592,"allocated":1380352,"data_stored":473181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.842527"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"17","state":"active+clean","last_fresh":"2020-06-12T02:37:16.760363+0000","last_change":"2020-06-12T02:37:16.758922+0000","last_active":"2020-06-12T02:37:16.760363+0000","last_peered":"2020-06-12T02:37:16.760363+0000","last_clean":"2020-06-12T02:37:16.760363+0000","last_became_active":"2020-06-12T02:37:16.758517+0000","last_became_peered":"2020-06-12T02:37:16.758517+0000","last_unstale":"2020-06-12T02:37:16.760363+0000","last_undegraded":"2020-06-12T02:37:16.760363+0000","last_fullsized":"2020-06-12T02:37:16.760363+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"8","reported_epoch":"17","state":"active+clean","last_fresh":"2020-06-12T02:37:16.763049+0000","last_change":"2020-06-12T02:37:16.763049+0000","last_active":"2020-06-12T02:37:16.763049+0000","last_peered":"2020-06-12T02:37:16.763049+0000","last_clean":"2020-06-12T02:37:16.763049+0000","last_became_active":"2020-06-12T02:37:16.762626+0000","last_became_peered":"2020-06-12T02:37:16.762626+0000","last_unstale":"2020-06-12T02:37:16.763049+0000","last_undegraded":"2020-06-12T02:37:16.763049+0000","last_fullsized":"2020-06-12T02:37:16.763049+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"17","state":"active+clean","last_fresh":"2020-06-12T02:37:16.869345+0000","last_change":"2020-06-12T02:37:16.868746+0000","last_active":"2020-06-12T02:37:16.869345+0000","last_peered":"2020-06-12T02:37:16.869345+0000","last_clean":"2020-06-12T02:37:16.869345+0000","last_became_active":"2020-06-12T02:37:16.868366+0000","last_became_peered":"2020-06-12T02:37:16.868366+0000","last_unstale":"2020-06-12T02:37:16.869345+0000","last_undegraded":"2020-06-12T02:37:16.869345+0000","last_fullsized":"2020-06-12T02:37:16.869345+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"17","state":"active+clean","last_fresh":"2020-06-12T02:37:16.763113+0000","last_change":"2020-06-12T02:37:16.762699+0000","last_active":"2020-06-12T02:37:16.763113+0000","last_peered":"2020-06-12T02:37:16.763113+0000","last_clean":"2020-06-12T02:37:16.763113+0000","last_became_active":"2020-06-12T02:37:16.762225+0000","last_became_peered":"2020-06-12T02:37:16.762225+0000","last_unstale":"2020-06-12T02:37:16.763113+0000","last_undegraded":"2020-06-12T02:37:16.763113+0000","last_fullsized":"2020-06-12T02:37:16.763113+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'2","reported_seq":"14","reported_epoch":"17","state":"active+clean","last_fresh":"2020-06-12T02:37:16.754944+0000","last_change":"2020-06-12T02:37:12.139089+0000","last_active":"2020-06-12T02:37:16.754944+0000","last_peered":"2020-06-12T02:37:16.754944+0000","last_clean":"2020-06-12T02:37:16.754944+0000","last_became_active":"2020-06-12T02:37:12.138712+0000","last_became_peered":"2020-06-12T02:37:12.138712+0000","last_unstale":"2020-06-12T02:37:16.754944+0000","last_undegraded":"2020-06-12T02:37:16.754944+0000","last_fullsized":"2020-06-12T02:37:16.754944+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":3,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:04.419204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:04.419204+0000","last_clean_scrub_stamp":"2020-06-12T02:37:04.419204+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"17","state":"active+clean","last_fresh":"2020-06-12T02:37:16.765561+0000","last_change":"2020-06-12T02:37:16.765125+0000","last_active":"2020-06-12T02:37:16.765561+0000","last_peered":"2020-06-12T02:37:16.765561+0000","last_clean":"2020-06-12T02:37:16.765561+0000","last_became_active":"2020-06-12T02:37:16.764803+0000","last_became_peered":"2020-06-12T02:37:16.764803+0000","last_unstale":"2020-06-12T02:37:16.765561+0000","last_undegraded":"2020-06-12T02:37:16.765561+0000","last_fullsized":"2020-06-12T02:37:16.765561+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"17","state":"active+clean","last_fresh":"2020-06-12T02:37:16.769544+0000","last_change":"2020-06-12T02:37:16.765460+0000","last_active":"2020-06-12T02:37:16.769544+0000","last_peered":"2020-06-12T02:37:16.769544+0000","last_clean":"2020-06-12T02:37:16.769544+0000","last_became_active":"2020-06-12T02:37:16.764958+0000","last_became_peered":"2020-06-12T02:37:16.764958+0000","last_unstale":"2020-06-12T02:37:16.769544+0000","last_undegraded":"2020-06-12T02:37:16.769544+0000","last_fullsized":"2020-06-12T02:37:16.769544+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"17","state":"active+clean","last_fresh":"2020-06-12T02:37:16.762500+0000","last_change":"2020-06-12T02:37:16.761008+0000","last_active":"2020-06-12T02:37:16.762500+0000","last_peered":"2020-06-12T02:37:16.762500+0000","last_clean":"2020-06-12T02:37:16.762500+0000","last_became_active":"2020-06-12T02:37:16.760694+0000","last_became_peered":"2020-06-12T02:37:16.760694+0000","last_unstale":"2020-06-12T02:37:16.762500+0000","last_undegraded":"2020-06-12T02:37:16.762500+0000","last_fullsized":"2020-06-12T02:37:16.762500+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"8","reported_epoch":"17","state":"active+clean","last_fresh":"2020-06-12T02:37:16.868466+0000","last_change":"2020-06-12T02:37:16.868466+0000","last_active":"2020-06-12T02:37:16.868466+0000","last_peered":"2020-06-12T02:37:16.868466+0000","last_clean":"2020-06-12T02:37:16.868466+0000","last_became_active":"2020-06-12T02:37:16.868098+0000","last_became_peered":"2020-06-12T02:37:16.868098+0000","last_unstale":"2020-06-12T02:37:16.868466+0000","last_undegraded":"2020-06-12T02:37:16.868466+0000","last_fullsized":"2020-06-12T02:37:16.868466+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048760,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323080,"statfs":{"total":96636764160,"available":95562833920,"internally_reserved":1073741824,"allocated":180224,"data_stored":62762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1910000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3660000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4339999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6459999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048760,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323080,"statfs":{"total":96636764160,"available":95562833920,"internally_reserved":1073741824,"allocated":180224,"data_stored":62762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.282}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2070000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1419999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1850000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7709999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048748,"kb_used_data":164,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323092,"statfs":{"total":96636764160,"available":95562846208,"internally_reserved":1073741824,"allocated":167936,"data_stored":56979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0620000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.026}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.216}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048748,"kb_used_data":164,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323092,"statfs":{"total":96636764160,"available":95562846208,"internally_reserved":1073741824,"allocated":167936,"data_stored":56979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3380000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.091}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6679999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0859999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.589}]}]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048760,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323080,"statfs":{"total":96636764160,"available":95562833920,"internally_reserved":1073741824,"allocated":180224,"data_stored":62762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79000000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1419999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4450000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3360000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3540000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5580000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048748,"kb_used_data":164,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323092,"statfs":{"total":96636764160,"available":95562846208,"internally_reserved":1073741824,"allocated":167936,"data_stored":56979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0600000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1048748,"kb_used_data":164,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323092,"statfs":{"total":96636764160,"available":95562846208,"internally_reserved":1073741824,"allocated":167936,"data_stored":56979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.629}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.056}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.964}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.034}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97899999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1909999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048748,"kb_used_data":164,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323092,"statfs":{"total":96636764160,"available":95562846208,"internally_reserved":1073741824,"allocated":167936,"data_stored":56979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80700000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5229999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.899}]}]}]}} 2020-06-12T02:37:22.018 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json 2020-06-12T02:37:22.271 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:37:22.273 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2020-06-12T02:37:22.282 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2020-06-12T02:37:21.594047+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":8,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":8390020,"kb_used_data":1348,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746584700,"statfs":{"total":773094113280,"available":764502732800,"internally_reserved":8589934592,"allocated":1380352,"data_stored":473181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.842527"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"17","state":"active+clean","last_fresh":"2020-06-12T02:37:16.760363+0000","last_change":"2020-06-12T02:37:16.758922+0000","last_active":"2020-06-12T02:37:16.760363+0000","last_peered":"2020-06-12T02:37:16.760363+0000","last_clean":"2020-06-12T02:37:16.760363+0000","last_became_active":"2020-06-12T02:37:16.758517+0000","last_became_peered":"2020-06-12T02:37:16.758517+0000","last_unstale":"2020-06-12T02:37:16.760363+0000","last_undegraded":"2020-06-12T02:37:16.760363+0000","last_fullsized":"2020-06-12T02:37:16.760363+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"8","reported_epoch":"17","state":"active+clean","last_fresh":"2020-06-12T02:37:16.763049+0000","last_change":"2020-06-12T02:37:16.763049+0000","last_active":"2020-06-12T02:37:16.763049+0000","last_peered":"2020-06-12T02:37:16.763049+0000","last_clean":"2020-06-12T02:37:16.763049+0000","last_became_active":"2020-06-12T02:37:16.762626+0000","last_became_peered":"2020-06-12T02:37:16.762626+0000","last_unstale":"2020-06-12T02:37:16.763049+0000","last_undegraded":"2020-06-12T02:37:16.763049+0000","last_fullsized":"2020-06-12T02:37:16.763049+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"17","state":"active+clean","last_fresh":"2020-06-12T02:37:16.869345+0000","last_change":"2020-06-12T02:37:16.868746+0000","last_active":"2020-06-12T02:37:16.869345+0000","last_peered":"2020-06-12T02:37:16.869345+0000","last_clean":"2020-06-12T02:37:16.869345+0000","last_became_active":"2020-06-12T02:37:16.868366+0000","last_became_peered":"2020-06-12T02:37:16.868366+0000","last_unstale":"2020-06-12T02:37:16.869345+0000","last_undegraded":"2020-06-12T02:37:16.869345+0000","last_fullsized":"2020-06-12T02:37:16.869345+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"17","state":"active+clean","last_fresh":"2020-06-12T02:37:16.763113+0000","last_change":"2020-06-12T02:37:16.762699+0000","last_active":"2020-06-12T02:37:16.763113+0000","last_peered":"2020-06-12T02:37:16.763113+0000","last_clean":"2020-06-12T02:37:16.763113+0000","last_became_active":"2020-06-12T02:37:16.762225+0000","last_became_peered":"2020-06-12T02:37:16.762225+0000","last_unstale":"2020-06-12T02:37:16.763113+0000","last_undegraded":"2020-06-12T02:37:16.763113+0000","last_fullsized":"2020-06-12T02:37:16.763113+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'2","reported_seq":"14","reported_epoch":"17","state":"active+clean","last_fresh":"2020-06-12T02:37:16.754944+0000","last_change":"2020-06-12T02:37:12.139089+0000","last_active":"2020-06-12T02:37:16.754944+0000","last_peered":"2020-06-12T02:37:16.754944+0000","last_clean":"2020-06-12T02:37:16.754944+0000","last_became_active":"2020-06-12T02:37:12.138712+0000","last_became_peered":"2020-06-12T02:37:12.138712+0000","last_unstale":"2020-06-12T02:37:16.754944+0000","last_undegraded":"2020-06-12T02:37:16.754944+0000","last_fullsized":"2020-06-12T02:37:16.754944+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":3,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:04.419204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:04.419204+0000","last_clean_scrub_stamp":"2020-06-12T02:37:04.419204+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"17","state":"active+clean","last_fresh":"2020-06-12T02:37:16.765561+0000","last_change":"2020-06-12T02:37:16.765125+0000","last_active":"2020-06-12T02:37:16.765561+0000","last_peered":"2020-06-12T02:37:16.765561+0000","last_clean":"2020-06-12T02:37:16.765561+0000","last_became_active":"2020-06-12T02:37:16.764803+0000","last_became_peered":"2020-06-12T02:37:16.764803+0000","last_unstale":"2020-06-12T02:37:16.765561+0000","last_undegraded":"2020-06-12T02:37:16.765561+0000","last_fullsized":"2020-06-12T02:37:16.765561+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"17","state":"active+clean","last_fresh":"2020-06-12T02:37:16.769544+0000","last_change":"2020-06-12T02:37:16.765460+0000","last_active":"2020-06-12T02:37:16.769544+0000","last_peered":"2020-06-12T02:37:16.769544+0000","last_clean":"2020-06-12T02:37:16.769544+0000","last_became_active":"2020-06-12T02:37:16.764958+0000","last_became_peered":"2020-06-12T02:37:16.764958+0000","last_unstale":"2020-06-12T02:37:16.769544+0000","last_undegraded":"2020-06-12T02:37:16.769544+0000","last_fullsized":"2020-06-12T02:37:16.769544+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"17","state":"active+clean","last_fresh":"2020-06-12T02:37:16.762500+0000","last_change":"2020-06-12T02:37:16.761008+0000","last_active":"2020-06-12T02:37:16.762500+0000","last_peered":"2020-06-12T02:37:16.762500+0000","last_clean":"2020-06-12T02:37:16.762500+0000","last_became_active":"2020-06-12T02:37:16.760694+0000","last_became_peered":"2020-06-12T02:37:16.760694+0000","last_unstale":"2020-06-12T02:37:16.762500+0000","last_undegraded":"2020-06-12T02:37:16.762500+0000","last_fullsized":"2020-06-12T02:37:16.762500+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"8","reported_epoch":"17","state":"active+clean","last_fresh":"2020-06-12T02:37:16.868466+0000","last_change":"2020-06-12T02:37:16.868466+0000","last_active":"2020-06-12T02:37:16.868466+0000","last_peered":"2020-06-12T02:37:16.868466+0000","last_clean":"2020-06-12T02:37:16.868466+0000","last_became_active":"2020-06-12T02:37:16.868098+0000","last_became_peered":"2020-06-12T02:37:16.868098+0000","last_unstale":"2020-06-12T02:37:16.868466+0000","last_undegraded":"2020-06-12T02:37:16.868466+0000","last_fullsized":"2020-06-12T02:37:16.868466+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048760,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323080,"statfs":{"total":96636764160,"available":95562833920,"internally_reserved":1073741824,"allocated":180224,"data_stored":62762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1910000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3660000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4339999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6459999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048760,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323080,"statfs":{"total":96636764160,"available":95562833920,"internally_reserved":1073741824,"allocated":180224,"data_stored":62762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.282}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2070000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1419999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1850000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7709999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048748,"kb_used_data":164,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323092,"statfs":{"total":96636764160,"available":95562846208,"internally_reserved":1073741824,"allocated":167936,"data_stored":56979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0620000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.026}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.216}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048748,"kb_used_data":164,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323092,"statfs":{"total":96636764160,"available":95562846208,"internally_reserved":1073741824,"allocated":167936,"data_stored":56979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3380000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.091}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6679999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0859999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.589}]}]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048760,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323080,"statfs":{"total":96636764160,"available":95562833920,"internally_reserved":1073741824,"allocated":180224,"data_stored":62762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79000000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1419999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4450000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3360000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3540000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5580000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048748,"kb_used_data":164,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323092,"statfs":{"total":96636764160,"available":95562846208,"internally_reserved":1073741824,"allocated":167936,"data_stored":56979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0600000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1048748,"kb_used_data":164,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323092,"statfs":{"total":96636764160,"available":95562846208,"internally_reserved":1073741824,"allocated":167936,"data_stored":56979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.629}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.056}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.964}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.034}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97899999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1909999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048748,"kb_used_data":164,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323092,"statfs":{"total":96636764160,"available":95562846208,"internally_reserved":1073741824,"allocated":167936,"data_stored":56979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80700000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5229999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.899}]}]}]}} 2020-06-12T02:37:22.284 INFO:tasks.ceph.ceph_manager.ceph:clean! 2020-06-12T02:37:22.284 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2020-06-12T02:37:22.285 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2020-06-12T02:37:22.285 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early health --format=json 2020-06-12T02:37:22.588 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:37:22.599 INFO:teuthology.orchestra.run.smithi107.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2020-06-12T02:37:22.600 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2020-06-12T02:37:22.600 INFO:teuthology.run_tasks:Running task thrashosds... 2020-06-12T02:37:22.618 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 4, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2020-06-12T02:37:22.619 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'octopus'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000, 'osd pool default ec fast read': True, 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd heartbeat use min delay socket': True, 'osd max backfills': 2, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request'], 'sha1': '91d24f498556f86f6d1318d2efd5210c35b36762'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '91d24f498556f86f6d1318d2efd5210c35b36762'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'octopus-py3', 'sha1': '1e0d5d60071d5554f87a2be72ea0bcccb272aab3'}} 2020-06-12T02:37:22.619 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 4, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2020-06-12T02:37:22.619 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early config set mgr mgr_debug_aggressive_pg_num_changes true 2020-06-12T02:37:22.894 INFO:tasks.thrashosds:Beginning thrashosds... 2020-06-12T02:37:22.894 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2020-06-12T02:37:22.895 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json 2020-06-12T02:37:23.146 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:37:23.148 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2020-06-12T02:37:23.158 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2020-06-12T02:37:21.594047+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":8,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":8390020,"kb_used_data":1348,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746584700,"statfs":{"total":773094113280,"available":764502732800,"internally_reserved":8589934592,"allocated":1380352,"data_stored":473181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.842527"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"17","state":"active+clean","last_fresh":"2020-06-12T02:37:16.760363+0000","last_change":"2020-06-12T02:37:16.758922+0000","last_active":"2020-06-12T02:37:16.760363+0000","last_peered":"2020-06-12T02:37:16.760363+0000","last_clean":"2020-06-12T02:37:16.760363+0000","last_became_active":"2020-06-12T02:37:16.758517+0000","last_became_peered":"2020-06-12T02:37:16.758517+0000","last_unstale":"2020-06-12T02:37:16.760363+0000","last_undegraded":"2020-06-12T02:37:16.760363+0000","last_fullsized":"2020-06-12T02:37:16.760363+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"8","reported_epoch":"17","state":"active+clean","last_fresh":"2020-06-12T02:37:16.763049+0000","last_change":"2020-06-12T02:37:16.763049+0000","last_active":"2020-06-12T02:37:16.763049+0000","last_peered":"2020-06-12T02:37:16.763049+0000","last_clean":"2020-06-12T02:37:16.763049+0000","last_became_active":"2020-06-12T02:37:16.762626+0000","last_became_peered":"2020-06-12T02:37:16.762626+0000","last_unstale":"2020-06-12T02:37:16.763049+0000","last_undegraded":"2020-06-12T02:37:16.763049+0000","last_fullsized":"2020-06-12T02:37:16.763049+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"17","state":"active+clean","last_fresh":"2020-06-12T02:37:16.869345+0000","last_change":"2020-06-12T02:37:16.868746+0000","last_active":"2020-06-12T02:37:16.869345+0000","last_peered":"2020-06-12T02:37:16.869345+0000","last_clean":"2020-06-12T02:37:16.869345+0000","last_became_active":"2020-06-12T02:37:16.868366+0000","last_became_peered":"2020-06-12T02:37:16.868366+0000","last_unstale":"2020-06-12T02:37:16.869345+0000","last_undegraded":"2020-06-12T02:37:16.869345+0000","last_fullsized":"2020-06-12T02:37:16.869345+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"17","state":"active+clean","last_fresh":"2020-06-12T02:37:16.763113+0000","last_change":"2020-06-12T02:37:16.762699+0000","last_active":"2020-06-12T02:37:16.763113+0000","last_peered":"2020-06-12T02:37:16.763113+0000","last_clean":"2020-06-12T02:37:16.763113+0000","last_became_active":"2020-06-12T02:37:16.762225+0000","last_became_peered":"2020-06-12T02:37:16.762225+0000","last_unstale":"2020-06-12T02:37:16.763113+0000","last_undegraded":"2020-06-12T02:37:16.763113+0000","last_fullsized":"2020-06-12T02:37:16.763113+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'2","reported_seq":"14","reported_epoch":"17","state":"active+clean","last_fresh":"2020-06-12T02:37:16.754944+0000","last_change":"2020-06-12T02:37:12.139089+0000","last_active":"2020-06-12T02:37:16.754944+0000","last_peered":"2020-06-12T02:37:16.754944+0000","last_clean":"2020-06-12T02:37:16.754944+0000","last_became_active":"2020-06-12T02:37:12.138712+0000","last_became_peered":"2020-06-12T02:37:12.138712+0000","last_unstale":"2020-06-12T02:37:16.754944+0000","last_undegraded":"2020-06-12T02:37:16.754944+0000","last_fullsized":"2020-06-12T02:37:16.754944+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":3,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:04.419204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:04.419204+0000","last_clean_scrub_stamp":"2020-06-12T02:37:04.419204+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"17","state":"active+clean","last_fresh":"2020-06-12T02:37:16.765561+0000","last_change":"2020-06-12T02:37:16.765125+0000","last_active":"2020-06-12T02:37:16.765561+0000","last_peered":"2020-06-12T02:37:16.765561+0000","last_clean":"2020-06-12T02:37:16.765561+0000","last_became_active":"2020-06-12T02:37:16.764803+0000","last_became_peered":"2020-06-12T02:37:16.764803+0000","last_unstale":"2020-06-12T02:37:16.765561+0000","last_undegraded":"2020-06-12T02:37:16.765561+0000","last_fullsized":"2020-06-12T02:37:16.765561+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"17","state":"active+clean","last_fresh":"2020-06-12T02:37:16.769544+0000","last_change":"2020-06-12T02:37:16.765460+0000","last_active":"2020-06-12T02:37:16.769544+0000","last_peered":"2020-06-12T02:37:16.769544+0000","last_clean":"2020-06-12T02:37:16.769544+0000","last_became_active":"2020-06-12T02:37:16.764958+0000","last_became_peered":"2020-06-12T02:37:16.764958+0000","last_unstale":"2020-06-12T02:37:16.769544+0000","last_undegraded":"2020-06-12T02:37:16.769544+0000","last_fullsized":"2020-06-12T02:37:16.769544+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"17","state":"active+clean","last_fresh":"2020-06-12T02:37:16.762500+0000","last_change":"2020-06-12T02:37:16.761008+0000","last_active":"2020-06-12T02:37:16.762500+0000","last_peered":"2020-06-12T02:37:16.762500+0000","last_clean":"2020-06-12T02:37:16.762500+0000","last_became_active":"2020-06-12T02:37:16.760694+0000","last_became_peered":"2020-06-12T02:37:16.760694+0000","last_unstale":"2020-06-12T02:37:16.762500+0000","last_undegraded":"2020-06-12T02:37:16.762500+0000","last_fullsized":"2020-06-12T02:37:16.762500+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"8","reported_epoch":"17","state":"active+clean","last_fresh":"2020-06-12T02:37:16.868466+0000","last_change":"2020-06-12T02:37:16.868466+0000","last_active":"2020-06-12T02:37:16.868466+0000","last_peered":"2020-06-12T02:37:16.868466+0000","last_clean":"2020-06-12T02:37:16.868466+0000","last_became_active":"2020-06-12T02:37:16.868098+0000","last_became_peered":"2020-06-12T02:37:16.868098+0000","last_unstale":"2020-06-12T02:37:16.868466+0000","last_undegraded":"2020-06-12T02:37:16.868466+0000","last_fullsized":"2020-06-12T02:37:16.868466+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048760,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323080,"statfs":{"total":96636764160,"available":95562833920,"internally_reserved":1073741824,"allocated":180224,"data_stored":62762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1910000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3660000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4339999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6459999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048760,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323080,"statfs":{"total":96636764160,"available":95562833920,"internally_reserved":1073741824,"allocated":180224,"data_stored":62762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.282}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2070000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1419999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1850000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7709999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048748,"kb_used_data":164,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323092,"statfs":{"total":96636764160,"available":95562846208,"internally_reserved":1073741824,"allocated":167936,"data_stored":56979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0620000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.026}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.216}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048748,"kb_used_data":164,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323092,"statfs":{"total":96636764160,"available":95562846208,"internally_reserved":1073741824,"allocated":167936,"data_stored":56979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3380000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.091}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6679999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0859999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.589}]}]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048760,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323080,"statfs":{"total":96636764160,"available":95562833920,"internally_reserved":1073741824,"allocated":180224,"data_stored":62762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79000000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1419999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4450000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3360000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3540000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5580000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048748,"kb_used_data":164,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323092,"statfs":{"total":96636764160,"available":95562846208,"internally_reserved":1073741824,"allocated":167936,"data_stored":56979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0600000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1048748,"kb_used_data":164,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323092,"statfs":{"total":96636764160,"available":95562846208,"internally_reserved":1073741824,"allocated":167936,"data_stored":56979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.629}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.056}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.964}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.034}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97899999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1909999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048748,"kb_used_data":164,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323092,"statfs":{"total":96636764160,"available":95562846208,"internally_reserved":1073741824,"allocated":167936,"data_stored":56979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80700000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5229999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.899}]}]}]}} 2020-06-12T02:37:23.160 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json 2020-06-12T02:37:23.413 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:37:23.414 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2020-06-12T02:37:23.424 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2020-06-12T02:37:21.594047+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":8,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":8390020,"kb_used_data":1348,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746584700,"statfs":{"total":773094113280,"available":764502732800,"internally_reserved":8589934592,"allocated":1380352,"data_stored":473181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.842527"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"17","state":"active+clean","last_fresh":"2020-06-12T02:37:16.760363+0000","last_change":"2020-06-12T02:37:16.758922+0000","last_active":"2020-06-12T02:37:16.760363+0000","last_peered":"2020-06-12T02:37:16.760363+0000","last_clean":"2020-06-12T02:37:16.760363+0000","last_became_active":"2020-06-12T02:37:16.758517+0000","last_became_peered":"2020-06-12T02:37:16.758517+0000","last_unstale":"2020-06-12T02:37:16.760363+0000","last_undegraded":"2020-06-12T02:37:16.760363+0000","last_fullsized":"2020-06-12T02:37:16.760363+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"8","reported_epoch":"17","state":"active+clean","last_fresh":"2020-06-12T02:37:16.763049+0000","last_change":"2020-06-12T02:37:16.763049+0000","last_active":"2020-06-12T02:37:16.763049+0000","last_peered":"2020-06-12T02:37:16.763049+0000","last_clean":"2020-06-12T02:37:16.763049+0000","last_became_active":"2020-06-12T02:37:16.762626+0000","last_became_peered":"2020-06-12T02:37:16.762626+0000","last_unstale":"2020-06-12T02:37:16.763049+0000","last_undegraded":"2020-06-12T02:37:16.763049+0000","last_fullsized":"2020-06-12T02:37:16.763049+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"17","state":"active+clean","last_fresh":"2020-06-12T02:37:16.869345+0000","last_change":"2020-06-12T02:37:16.868746+0000","last_active":"2020-06-12T02:37:16.869345+0000","last_peered":"2020-06-12T02:37:16.869345+0000","last_clean":"2020-06-12T02:37:16.869345+0000","last_became_active":"2020-06-12T02:37:16.868366+0000","last_became_peered":"2020-06-12T02:37:16.868366+0000","last_unstale":"2020-06-12T02:37:16.869345+0000","last_undegraded":"2020-06-12T02:37:16.869345+0000","last_fullsized":"2020-06-12T02:37:16.869345+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"17","state":"active+clean","last_fresh":"2020-06-12T02:37:16.763113+0000","last_change":"2020-06-12T02:37:16.762699+0000","last_active":"2020-06-12T02:37:16.763113+0000","last_peered":"2020-06-12T02:37:16.763113+0000","last_clean":"2020-06-12T02:37:16.763113+0000","last_became_active":"2020-06-12T02:37:16.762225+0000","last_became_peered":"2020-06-12T02:37:16.762225+0000","last_unstale":"2020-06-12T02:37:16.763113+0000","last_undegraded":"2020-06-12T02:37:16.763113+0000","last_fullsized":"2020-06-12T02:37:16.763113+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'2","reported_seq":"14","reported_epoch":"17","state":"active+clean","last_fresh":"2020-06-12T02:37:16.754944+0000","last_change":"2020-06-12T02:37:12.139089+0000","last_active":"2020-06-12T02:37:16.754944+0000","last_peered":"2020-06-12T02:37:16.754944+0000","last_clean":"2020-06-12T02:37:16.754944+0000","last_became_active":"2020-06-12T02:37:12.138712+0000","last_became_peered":"2020-06-12T02:37:12.138712+0000","last_unstale":"2020-06-12T02:37:16.754944+0000","last_undegraded":"2020-06-12T02:37:16.754944+0000","last_fullsized":"2020-06-12T02:37:16.754944+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":3,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:04.419204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:04.419204+0000","last_clean_scrub_stamp":"2020-06-12T02:37:04.419204+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"17","state":"active+clean","last_fresh":"2020-06-12T02:37:16.765561+0000","last_change":"2020-06-12T02:37:16.765125+0000","last_active":"2020-06-12T02:37:16.765561+0000","last_peered":"2020-06-12T02:37:16.765561+0000","last_clean":"2020-06-12T02:37:16.765561+0000","last_became_active":"2020-06-12T02:37:16.764803+0000","last_became_peered":"2020-06-12T02:37:16.764803+0000","last_unstale":"2020-06-12T02:37:16.765561+0000","last_undegraded":"2020-06-12T02:37:16.765561+0000","last_fullsized":"2020-06-12T02:37:16.765561+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"17","state":"active+clean","last_fresh":"2020-06-12T02:37:16.769544+0000","last_change":"2020-06-12T02:37:16.765460+0000","last_active":"2020-06-12T02:37:16.769544+0000","last_peered":"2020-06-12T02:37:16.769544+0000","last_clean":"2020-06-12T02:37:16.769544+0000","last_became_active":"2020-06-12T02:37:16.764958+0000","last_became_peered":"2020-06-12T02:37:16.764958+0000","last_unstale":"2020-06-12T02:37:16.769544+0000","last_undegraded":"2020-06-12T02:37:16.769544+0000","last_fullsized":"2020-06-12T02:37:16.769544+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"17","state":"active+clean","last_fresh":"2020-06-12T02:37:16.762500+0000","last_change":"2020-06-12T02:37:16.761008+0000","last_active":"2020-06-12T02:37:16.762500+0000","last_peered":"2020-06-12T02:37:16.762500+0000","last_clean":"2020-06-12T02:37:16.762500+0000","last_became_active":"2020-06-12T02:37:16.760694+0000","last_became_peered":"2020-06-12T02:37:16.760694+0000","last_unstale":"2020-06-12T02:37:16.762500+0000","last_undegraded":"2020-06-12T02:37:16.762500+0000","last_fullsized":"2020-06-12T02:37:16.762500+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"8","reported_epoch":"17","state":"active+clean","last_fresh":"2020-06-12T02:37:16.868466+0000","last_change":"2020-06-12T02:37:16.868466+0000","last_active":"2020-06-12T02:37:16.868466+0000","last_peered":"2020-06-12T02:37:16.868466+0000","last_clean":"2020-06-12T02:37:16.868466+0000","last_became_active":"2020-06-12T02:37:16.868098+0000","last_became_peered":"2020-06-12T02:37:16.868098+0000","last_unstale":"2020-06-12T02:37:16.868466+0000","last_undegraded":"2020-06-12T02:37:16.868466+0000","last_fullsized":"2020-06-12T02:37:16.868466+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048760,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323080,"statfs":{"total":96636764160,"available":95562833920,"internally_reserved":1073741824,"allocated":180224,"data_stored":62762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1910000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3660000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4339999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6459999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048760,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323080,"statfs":{"total":96636764160,"available":95562833920,"internally_reserved":1073741824,"allocated":180224,"data_stored":62762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.282}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2070000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1419999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1850000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7709999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048748,"kb_used_data":164,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323092,"statfs":{"total":96636764160,"available":95562846208,"internally_reserved":1073741824,"allocated":167936,"data_stored":56979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0620000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.026}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.216}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048748,"kb_used_data":164,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323092,"statfs":{"total":96636764160,"available":95562846208,"internally_reserved":1073741824,"allocated":167936,"data_stored":56979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3380000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.091}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6679999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0859999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.589}]}]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048760,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323080,"statfs":{"total":96636764160,"available":95562833920,"internally_reserved":1073741824,"allocated":180224,"data_stored":62762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79000000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1419999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4450000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3360000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3540000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5580000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048748,"kb_used_data":164,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323092,"statfs":{"total":96636764160,"available":95562846208,"internally_reserved":1073741824,"allocated":167936,"data_stored":56979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0600000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1048748,"kb_used_data":164,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323092,"statfs":{"total":96636764160,"available":95562846208,"internally_reserved":1073741824,"allocated":167936,"data_stored":56979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.629}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.056}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.964}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.034}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97899999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1909999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048748,"kb_used_data":164,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323092,"statfs":{"total":96636764160,"available":95562846208,"internally_reserved":1073741824,"allocated":167936,"data_stored":56979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80700000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5229999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.899}]}]}]}} 2020-06-12T02:37:23.425 INFO:tasks.ceph.ceph_manager.ceph:clean! 2020-06-12T02:37:23.426 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump 2020-06-12T02:37:23.684 INFO:teuthology.orchestra.run.smithi107.stdout:epoch 18 2020-06-12T02:37:23.684 INFO:teuthology.orchestra.run.smithi107.stdout:fsid 2f5030a1-3206-43c9-8471-d10429688d36 2020-06-12T02:37:23.684 INFO:teuthology.orchestra.run.smithi107.stdout:created 2020-06-12T02:37:00.578493+0000 2020-06-12T02:37:23.684 INFO:teuthology.orchestra.run.smithi107.stdout:modified 2020-06-12T02:37:17.759876+0000 2020-06-12T02:37:23.685 INFO:teuthology.orchestra.run.smithi107.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2020-06-12T02:37:23.685 INFO:teuthology.orchestra.run.smithi107.stdout:crush_version 4 2020-06-12T02:37:23.685 INFO:teuthology.orchestra.run.smithi107.stdout:full_ratio 0.95 2020-06-12T02:37:23.685 INFO:teuthology.orchestra.run.smithi107.stdout:backfillfull_ratio 0.9 2020-06-12T02:37:23.685 INFO:teuthology.orchestra.run.smithi107.stdout:nearfull_ratio 0.85 2020-06-12T02:37:23.686 INFO:teuthology.orchestra.run.smithi107.stdout:require_min_compat_client jewel 2020-06-12T02:37:23.686 INFO:teuthology.orchestra.run.smithi107.stdout:min_compat_client jewel 2020-06-12T02:37:23.686 INFO:teuthology.orchestra.run.smithi107.stdout:require_osd_release octopus 2020-06-12T02:37:23.686 INFO:teuthology.orchestra.run.smithi107.stdout:pool 1 'device_health_metrics' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 15 flags hashpspool stripe_width 0 pg_num_min 1 application mgr_devicehealth 2020-06-12T02:37:23.686 INFO:teuthology.orchestra.run.smithi107.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 18 flags hashpspool stripe_width 0 application rbd 2020-06-12T02:37:23.687 INFO:teuthology.orchestra.run.smithi107.stdout:max_osd 8 2020-06-12T02:37:23.687 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.107:6816/26614,v1:172.21.15.107:6817/26614] [v2:172.21.15.107:6819/26614,v1:172.21.15.107:6821/26614] exists,up ad4b4814-83e7-4a2e-9b1f-d10406174f7e 2020-06-12T02:37:23.687 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1 up in weight 1 up_from 13 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.107:6800/26611,v1:172.21.15.107:6801/26611] [v2:172.21.15.107:6802/26611,v1:172.21.15.107:6803/26611] exists,up 046adf0d-b9e9-41ad-a7c9-3841f8b8f790 2020-06-12T02:37:23.687 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2 up in weight 1 up_from 13 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.107:6818/26613,v1:172.21.15.107:6820/26613] [v2:172.21.15.107:6822/26613,v1:172.21.15.107:6824/26613] exists,up e15da6b7-dc79-4185-a5f9-0f443d8b52b6 2020-06-12T02:37:23.687 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.107:6808/26612,v1:172.21.15.107:6809/26612] [v2:172.21.15.107:6810/26612,v1:172.21.15.107:6811/26612] exists,up 843e0dd3-5ce4-46b9-9004-69849048d2bd 2020-06-12T02:37:23.687 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.188:6818/26685,v1:172.21.15.188:6820/26685] [v2:172.21.15.188:6822/26685,v1:172.21.15.188:6823/26685] exists,up c066ab09-e031-4b49-9df3-d54cd9f9b580 2020-06-12T02:37:23.688 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5 up in weight 1 up_from 13 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.188:6819/26684,v1:172.21.15.188:6821/26684] [v2:172.21.15.188:6824/26684,v1:172.21.15.188:6826/26684] exists,up 56a24e45-1b1b-4a65-b4b4-57cfab296f65 2020-06-12T02:37:23.688 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6 up in weight 1 up_from 13 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.188:6810/26682,v1:172.21.15.188:6811/26682] [v2:172.21.15.188:6812/26682,v1:172.21.15.188:6813/26682] exists,up f207a2a8-b3c7-4fd8-ae90-4c2d7a5c1c38 2020-06-12T02:37:23.688 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7 up in weight 1 up_from 13 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.188:6802/26683,v1:172.21.15.188:6803/26683] [v2:172.21.15.188:6804/26683,v1:172.21.15.188:6805/26683] exists,up 4e5cc44f-af17-47c9-973f-8e87d1bd98db 2020-06-12T02:37:23.703 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.107:6816/26614,v1:172.21.15.107:6817/26614] [v2:172.21.15.107:6819/26614,v1:172.21.15.107:6821/26614] exists,up ad4b4814-83e7-4a2e-9b1f-d10406174f7e', 'osd.1 up in weight 1 up_from 13 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.107:6800/26611,v1:172.21.15.107:6801/26611] [v2:172.21.15.107:6802/26611,v1:172.21.15.107:6803/26611] exists,up 046adf0d-b9e9-41ad-a7c9-3841f8b8f790', 'osd.2 up in weight 1 up_from 13 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.107:6818/26613,v1:172.21.15.107:6820/26613] [v2:172.21.15.107:6822/26613,v1:172.21.15.107:6824/26613] exists,up e15da6b7-dc79-4185-a5f9-0f443d8b52b6', 'osd.3 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.107:6808/26612,v1:172.21.15.107:6809/26612] [v2:172.21.15.107:6810/26612,v1:172.21.15.107:6811/26612] exists,up 843e0dd3-5ce4-46b9-9004-69849048d2bd', 'osd.4 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.188:6818/26685,v1:172.21.15.188:6820/26685] [v2:172.21.15.188:6822/26685,v1:172.21.15.188:6823/26685] exists,up c066ab09-e031-4b49-9df3-d54cd9f9b580', 'osd.5 up in weight 1 up_from 13 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.188:6819/26684,v1:172.21.15.188:6821/26684] [v2:172.21.15.188:6824/26684,v1:172.21.15.188:6826/26684] exists,up 56a24e45-1b1b-4a65-b4b4-57cfab296f65', 'osd.6 up in weight 1 up_from 13 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.188:6810/26682,v1:172.21.15.188:6811/26682] [v2:172.21.15.188:6812/26682,v1:172.21.15.188:6813/26682] exists,up f207a2a8-b3c7-4fd8-ae90-4c2d7a5c1c38', 'osd.7 up in weight 1 up_from 13 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.188:6802/26683,v1:172.21.15.188:6803/26683] [v2:172.21.15.188:6804/26683,v1:172.21.15.188:6805/26683] exists,up 4e5cc44f-af17-47c9-973f-8e87d1bd98db'] 2020-06-12T02:37:23.704 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2020-06-12T02:37:23.822 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:37:23.823 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "mon.a", 2020-06-12T02:37:23.823 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster": "ceph", 2020-06-12T02:37:23.823 INFO:teuthology.orchestra.run.smithi107.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2020-06-12T02:37:23.823 INFO:teuthology.orchestra.run.smithi107.stdout: "admin_socket_mode": "", 2020-06-12T02:37:23.823 INFO:teuthology.orchestra.run.smithi107.stdout: "allow_ansi": "Terminal", 2020-06-12T02:37:23.824 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_client_required": "cephx, none", 2020-06-12T02:37:23.824 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_cluster_required": "cephx", 2020-06-12T02:37:23.824 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_debug": "true", 2020-06-12T02:37:23.824 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_mon_ticket_ttl": "43200.000000", 2020-06-12T02:37:23.824 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_service_required": "cephx", 2020-06-12T02:37:23.825 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_service_ticket_ttl": "3600.000000", 2020-06-12T02:37:23.825 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_supported": "cephx", 2020-06-12T02:37:23.825 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_aio": "true", 2020-06-12T02:37:23.825 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_aio_max_queue_depth": "1024", 2020-06-12T02:37:23.825 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_aio_poll_ms": "250", 2020-06-12T02:37:23.826 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_aio_reap_max": "16", 2020-06-12T02:37:23.826 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_async_discard": "false", 2020-06-12T02:37:23.826 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_block_size": "4096", 2020-06-12T02:37:23.826 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_debug_aio": "false", 2020-06-12T02:37:23.826 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_debug_aio_log_age": "5.000000", 2020-06-12T02:37:23.826 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2020-06-12T02:37:23.827 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_debug_inflight_ios": "false", 2020-06-12T02:37:23.827 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_enable_discard": "false", 2020-06-12T02:37:23.827 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_inject_crash": "0", 2020-06-12T02:37:23.827 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_inject_crash_flush_delay": "2", 2020-06-12T02:37:23.827 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_nvme_retry_count": "-1", 2020-06-12T02:37:23.828 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_nvme_unbind_from_kernel": "false", 2020-06-12T02:37:23.828 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_alloc_size": "1048576", 2020-06-12T02:37:23.828 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_allocator": "bitmap", 2020-06-12T02:37:23.828 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_buffered_io": "false", 2020-06-12T02:37:23.828 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_compact_log_sync": "false", 2020-06-12T02:37:23.828 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2020-06-12T02:37:23.829 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_log_compact_min_size": "16777216", 2020-06-12T02:37:23.829 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_log_replay_check_allocations": "true", 2020-06-12T02:37:23.829 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_max_log_runway": "4194304", 2020-06-12T02:37:23.829 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_max_prefetch": "1048576", 2020-06-12T02:37:23.829 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_min_flush_size": "524288", 2020-06-12T02:37:23.830 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_min_log_runway": "1048576", 2020-06-12T02:37:23.830 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_preextend_wal_files": "false", 2020-06-12T02:37:23.830 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_shared_alloc_size": "65536", 2020-06-12T02:37:23.830 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_sync_write": "false", 2020-06-12T02:37:23.830 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2020-06-12T02:37:23.830 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2020-06-12T02:37:23.831 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2020-06-12T02:37:23.831 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_allocator": "bitmap", 2020-06-12T02:37:23.831 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2020-06-12T02:37:23.831 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2020-06-12T02:37:23.831 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2020-06-12T02:37:23.832 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_bitmapallocator_span_size": "1024", 2020-06-12T02:37:23.832 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_blobid_prealloc": "10240", 2020-06-12T02:37:23.832 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_block_create": "true", 2020-06-12T02:37:23.832 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_block_db_create": "false", 2020-06-12T02:37:23.832 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_block_db_path": "", 2020-06-12T02:37:23.832 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_block_db_size": "0", 2020-06-12T02:37:23.833 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_block_path": "", 2020-06-12T02:37:23.833 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_block_preallocate_file": "false", 2020-06-12T02:37:23.833 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_block_size": "107374182400", 2020-06-12T02:37:23.833 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_block_wal_create": "false", 2020-06-12T02:37:23.833 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_block_wal_path": "", 2020-06-12T02:37:23.834 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_block_wal_size": "100663296", 2020-06-12T02:37:23.834 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_bluefs": "true", 2020-06-12T02:37:23.834 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2020-06-12T02:37:23.834 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_bluefs_balance_interval": "1.000000", 2020-06-12T02:37:23.834 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_bluefs_db_compatibility": "true", 2020-06-12T02:37:23.834 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_bluefs_env_mirror": "false", 2020-06-12T02:37:23.835 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_bluefs_gift_ratio": "0.020000", 2020-06-12T02:37:23.835 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_bluefs_max_free": "10737418240", 2020-06-12T02:37:23.835 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_bluefs_max_ratio": "0.900000", 2020-06-12T02:37:23.835 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_bluefs_min": "1073741824", 2020-06-12T02:37:23.835 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_bluefs_min_free": "1073741824", 2020-06-12T02:37:23.835 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_bluefs_min_ratio": "0.020000", 2020-06-12T02:37:23.836 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_bluefs_reclaim_ratio": "0.200000", 2020-06-12T02:37:23.836 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_autotune": "true", 2020-06-12T02:37:23.836 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_autotune_interval": "5.000000", 2020-06-12T02:37:23.836 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_kv_ratio": "0.400000", 2020-06-12T02:37:23.836 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_meta_ratio": "0.400000", 2020-06-12T02:37:23.837 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_size": "0", 2020-06-12T02:37:23.837 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_size_hdd": "1073741824", 2020-06-12T02:37:23.837 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_size_ssd": "3221225472", 2020-06-12T02:37:23.837 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_trim_interval": "0.050000", 2020-06-12T02:37:23.837 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_trim_max_skip_pinned": "64", 2020-06-12T02:37:23.837 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_type": "2q", 2020-06-12T02:37:23.838 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_clone_cow": "true", 2020-06-12T02:37:23.838 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_compression_algorithm": "snappy", 2020-06-12T02:37:23.838 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_compression_max_blob_size": "0", 2020-06-12T02:37:23.838 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_compression_max_blob_size_hdd": "524288", 2020-06-12T02:37:23.838 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2020-06-12T02:37:23.839 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_compression_min_blob_size": "0", 2020-06-12T02:37:23.839 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_compression_min_blob_size_hdd": "131072", 2020-06-12T02:37:23.839 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_compression_min_blob_size_ssd": "8192", 2020-06-12T02:37:23.839 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_compression_mode": "none", 2020-06-12T02:37:23.839 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_compression_required_ratio": "0.875000", 2020-06-12T02:37:23.839 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_csum_type": "crc32c", 2020-06-12T02:37:23.840 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_enforce_settings": "default", 2020-06-12T02:37:23.840 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_freelist": "false", 2020-06-12T02:37:23.840 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_fsck_abort": "false", 2020-06-12T02:37:23.840 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_inject_bug21040": "false", 2020-06-12T02:37:23.840 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2020-06-12T02:37:23.841 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_inject_read_err": "false", 2020-06-12T02:37:23.841 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_misc": "false", 2020-06-12T02:37:23.841 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_no_reuse_blocks": "false", 2020-06-12T02:37:23.841 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_omit_block_device_write": "false", 2020-06-12T02:37:23.841 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_omit_kv_commit": "false", 2020-06-12T02:37:23.841 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2020-06-12T02:37:23.842 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_prefill": "0.000000", 2020-06-12T02:37:23.842 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_prefragment_max": "1048576", 2020-06-12T02:37:23.842 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_random_read_err": "0.000000", 2020-06-12T02:37:23.842 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2020-06-12T02:37:23.842 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_small_allocations": "0", 2020-06-12T02:37:23.843 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2020-06-12T02:37:23.843 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_default_buffered_read": "true", 2020-06-12T02:37:23.843 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_default_buffered_write": "false", 2020-06-12T02:37:23.843 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_deferred_batch_ops": "0", 2020-06-12T02:37:23.843 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2020-06-12T02:37:23.843 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2020-06-12T02:37:23.844 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2020-06-12T02:37:23.844 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_map_shard_max_size": "1200", 2020-06-12T02:37:23.844 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_map_shard_min_size": "150", 2020-06-12T02:37:23.844 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_map_shard_target_size": "500", 2020-06-12T02:37:23.845 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2020-06-12T02:37:23.845 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_freelist_blocks_per_key": "128", 2020-06-12T02:37:23.845 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2020-06-12T02:37:23.845 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2020-06-12T02:37:23.845 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_on_mkfs": "true", 2020-06-12T02:37:23.846 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2020-06-12T02:37:23.846 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_on_mount": "false", 2020-06-12T02:37:23.846 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_on_mount_deep": "false", 2020-06-12T02:37:23.846 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_on_umount": "false", 2020-06-12T02:37:23.846 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_on_umount_deep": "false", 2020-06-12T02:37:23.846 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_quick_fix_on_mount": "true", 2020-06-12T02:37:23.847 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_quick_fix_threads": "2", 2020-06-12T02:37:23.847 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2020-06-12T02:37:23.847 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_gc_enable_blob_threshold": "0", 2020-06-12T02:37:23.847 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_gc_enable_total_threshold": "0", 2020-06-12T02:37:23.847 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_ignore_data_csum": "false", 2020-06-12T02:37:23.848 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_ioring": "false", 2020-06-12T02:37:23.848 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_kvbackend": "rocksdb", 2020-06-12T02:37:23.848 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_log_collection_list_age": "60.000000", 2020-06-12T02:37:23.848 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2020-06-12T02:37:23.848 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_log_op_age": "5.000000", 2020-06-12T02:37:23.848 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_max_alloc_size": "0", 2020-06-12T02:37:23.849 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_max_blob_size": "0", 2020-06-12T02:37:23.849 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_max_blob_size_hdd": "524288", 2020-06-12T02:37:23.850 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_max_blob_size_ssd": "65536", 2020-06-12T02:37:23.850 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_max_defer_interval": "3.000000", 2020-06-12T02:37:23.850 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_max_deferred_txc": "32", 2020-06-12T02:37:23.850 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_min_alloc_size": "0", 2020-06-12T02:37:23.850 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_min_alloc_size_hdd": "65536", 2020-06-12T02:37:23.850 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_min_alloc_size_ssd": "4096", 2020-06-12T02:37:23.851 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_nid_prealloc": "1024", 2020-06-12T02:37:23.851 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_prefer_deferred_size": "0", 2020-06-12T02:37:23.851 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_prefer_deferred_size_hdd": "32768", 2020-06-12T02:37:23.851 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2020-06-12T02:37:23.851 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_retry_disk_reads": "3", 2020-06-12T02:37:23.852 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_rocksdb_cf": "false", 2020-06-12T02:37:23.852 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_rocksdb_cfs": "M= P= L=", 2020-06-12T02:37:23.852 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_rocksdb_options": "compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2", 2020-06-12T02:37:23.852 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_spdk_coremask": "0x1", 2020-06-12T02:37:23.852 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_spdk_io_sleep": "5", 2020-06-12T02:37:23.852 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_spdk_max_io_completion": "0", 2020-06-12T02:37:23.853 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_spdk_mem": "512", 2020-06-12T02:37:23.853 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_sync_submit_transaction": "false", 2020-06-12T02:37:23.853 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_throttle_bytes": "67108864", 2020-06-12T02:37:23.853 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_throttle_cost_per_io": "0", 2020-06-12T02:37:23.853 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2020-06-12T02:37:23.854 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2020-06-12T02:37:23.854 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2020-06-12T02:37:23.854 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_throttle_trace_rate": "0.000000", 2020-06-12T02:37:23.854 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_tracing": "false", 2020-06-12T02:37:23.854 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_volume_selection_policy": "rocksdb_original", 2020-06-12T02:37:23.854 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_volume_selection_reserved": "0", 2020-06-12T02:37:23.855 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2020-06-12T02:37:23.855 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2020-06-12T02:37:23.855 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_warn_on_legacy_statfs": "true", 2020-06-12T02:37:23.855 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2020-06-12T02:37:23.855 INFO:teuthology.orchestra.run.smithi107.stdout: "cephadm_path": "/usr/sbin/cephadm", 2020-06-12T02:37:23.856 INFO:teuthology.orchestra.run.smithi107.stdout: "cephx_cluster_require_signatures": "false", 2020-06-12T02:37:23.856 INFO:teuthology.orchestra.run.smithi107.stdout: "cephx_cluster_require_version": "1", 2020-06-12T02:37:23.856 INFO:teuthology.orchestra.run.smithi107.stdout: "cephx_require_signatures": "false", 2020-06-12T02:37:23.856 INFO:teuthology.orchestra.run.smithi107.stdout: "cephx_require_version": "1", 2020-06-12T02:37:23.856 INFO:teuthology.orchestra.run.smithi107.stdout: "cephx_service_require_signatures": "false", 2020-06-12T02:37:23.856 INFO:teuthology.orchestra.run.smithi107.stdout: "cephx_service_require_version": "1", 2020-06-12T02:37:23.857 INFO:teuthology.orchestra.run.smithi107.stdout: "cephx_sign_messages": "true", 2020-06-12T02:37:23.857 INFO:teuthology.orchestra.run.smithi107.stdout: "chdir": "", 2020-06-12T02:37:23.857 INFO:teuthology.orchestra.run.smithi107.stdout: "client_acl_type": "", 2020-06-12T02:37:23.857 INFO:teuthology.orchestra.run.smithi107.stdout: "client_cache_mid": "0.750000", 2020-06-12T02:37:23.857 INFO:teuthology.orchestra.run.smithi107.stdout: "client_cache_size": "16384", 2020-06-12T02:37:23.858 INFO:teuthology.orchestra.run.smithi107.stdout: "client_caps_release_delay": "5", 2020-06-12T02:37:23.858 INFO:teuthology.orchestra.run.smithi107.stdout: "client_check_pool_perm": "true", 2020-06-12T02:37:23.858 INFO:teuthology.orchestra.run.smithi107.stdout: "client_debug_force_sync_read": "false", 2020-06-12T02:37:23.858 INFO:teuthology.orchestra.run.smithi107.stdout: "client_debug_getattr_caps": "false", 2020-06-12T02:37:23.858 INFO:teuthology.orchestra.run.smithi107.stdout: "client_debug_inject_tick_delay": "0", 2020-06-12T02:37:23.858 INFO:teuthology.orchestra.run.smithi107.stdout: "client_die_on_failed_dentry_invalidate": "true", 2020-06-12T02:37:23.859 INFO:teuthology.orchestra.run.smithi107.stdout: "client_die_on_failed_remount": "false", 2020-06-12T02:37:23.859 INFO:teuthology.orchestra.run.smithi107.stdout: "client_dirsize_rbytes": "true", 2020-06-12T02:37:23.859 INFO:teuthology.orchestra.run.smithi107.stdout: "client_force_lazyio": "false", 2020-06-12T02:37:23.859 INFO:teuthology.orchestra.run.smithi107.stdout: "client_fs": "", 2020-06-12T02:37:23.859 INFO:teuthology.orchestra.run.smithi107.stdout: "client_inject_fixed_oldest_tid": "false", 2020-06-12T02:37:23.860 INFO:teuthology.orchestra.run.smithi107.stdout: "client_inject_release_failure": "false", 2020-06-12T02:37:23.860 INFO:teuthology.orchestra.run.smithi107.stdout: "client_max_inline_size": "4096", 2020-06-12T02:37:23.860 INFO:teuthology.orchestra.run.smithi107.stdout: "client_mds_namespace": "", 2020-06-12T02:37:23.860 INFO:teuthology.orchestra.run.smithi107.stdout: "client_metadata": "", 2020-06-12T02:37:23.860 INFO:teuthology.orchestra.run.smithi107.stdout: "client_mount_gid": "-1", 2020-06-12T02:37:23.860 INFO:teuthology.orchestra.run.smithi107.stdout: "client_mount_timeout": "300.000000", 2020-06-12T02:37:23.861 INFO:teuthology.orchestra.run.smithi107.stdout: "client_mount_uid": "-1", 2020-06-12T02:37:23.861 INFO:teuthology.orchestra.run.smithi107.stdout: "client_mountpoint": "/", 2020-06-12T02:37:23.861 INFO:teuthology.orchestra.run.smithi107.stdout: "client_notify_timeout": "10", 2020-06-12T02:37:23.861 INFO:teuthology.orchestra.run.smithi107.stdout: "client_oc": "true", 2020-06-12T02:37:23.861 INFO:teuthology.orchestra.run.smithi107.stdout: "client_oc_max_dirty": "104857600", 2020-06-12T02:37:23.861 INFO:teuthology.orchestra.run.smithi107.stdout: "client_oc_max_dirty_age": "5.000000", 2020-06-12T02:37:23.862 INFO:teuthology.orchestra.run.smithi107.stdout: "client_oc_max_objects": "1000", 2020-06-12T02:37:23.862 INFO:teuthology.orchestra.run.smithi107.stdout: "client_oc_size": "209715200", 2020-06-12T02:37:23.862 INFO:teuthology.orchestra.run.smithi107.stdout: "client_oc_target_dirty": "8388608", 2020-06-12T02:37:23.862 INFO:teuthology.orchestra.run.smithi107.stdout: "client_permissions": "true", 2020-06-12T02:37:23.862 INFO:teuthology.orchestra.run.smithi107.stdout: "client_quota_df": "true", 2020-06-12T02:37:23.863 INFO:teuthology.orchestra.run.smithi107.stdout: "client_readahead_max_bytes": "0", 2020-06-12T02:37:23.863 INFO:teuthology.orchestra.run.smithi107.stdout: "client_readahead_max_periods": "4", 2020-06-12T02:37:23.863 INFO:teuthology.orchestra.run.smithi107.stdout: "client_readahead_min": "131072", 2020-06-12T02:37:23.863 INFO:teuthology.orchestra.run.smithi107.stdout: "client_reconnect_stale": "false", 2020-06-12T02:37:23.863 INFO:teuthology.orchestra.run.smithi107.stdout: "client_snapdir": ".snap", 2020-06-12T02:37:23.863 INFO:teuthology.orchestra.run.smithi107.stdout: "client_tick_interval": "1.000000", 2020-06-12T02:37:23.864 INFO:teuthology.orchestra.run.smithi107.stdout: "client_trace": "", 2020-06-12T02:37:23.864 INFO:teuthology.orchestra.run.smithi107.stdout: "client_try_dentry_invalidate": "false", 2020-06-12T02:37:23.864 INFO:teuthology.orchestra.run.smithi107.stdout: "client_use_faked_inos": "false", 2020-06-12T02:37:23.864 INFO:teuthology.orchestra.run.smithi107.stdout: "client_use_random_mds": "false", 2020-06-12T02:37:23.864 INFO:teuthology.orchestra.run.smithi107.stdout: "clog_to_graylog": "false", 2020-06-12T02:37:23.865 INFO:teuthology.orchestra.run.smithi107.stdout: "clog_to_graylog_host": "127.0.0.1", 2020-06-12T02:37:23.865 INFO:teuthology.orchestra.run.smithi107.stdout: "clog_to_graylog_port": "12201", 2020-06-12T02:37:23.865 INFO:teuthology.orchestra.run.smithi107.stdout: "clog_to_monitors": "default=true", 2020-06-12T02:37:23.865 INFO:teuthology.orchestra.run.smithi107.stdout: "clog_to_syslog": "false", 2020-06-12T02:37:23.865 INFO:teuthology.orchestra.run.smithi107.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2020-06-12T02:37:23.865 INFO:teuthology.orchestra.run.smithi107.stdout: "clog_to_syslog_level": "info", 2020-06-12T02:37:23.866 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "-", 2020-06-12T02:37:23.866 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_network": "", 2020-06-12T02:37:23.866 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_network_interface": "", 2020-06-12T02:37:23.866 INFO:teuthology.orchestra.run.smithi107.stdout: "colors": "Terminal", 2020-06-12T02:37:23.866 INFO:teuthology.orchestra.run.smithi107.stdout: "compressor_zlib_isal": "false", 2020-06-12T02:37:23.866 INFO:teuthology.orchestra.run.smithi107.stdout: "compressor_zlib_level": "5", 2020-06-12T02:37:23.867 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image": "docker.io/ceph/ceph:v15", 2020-06-12T02:37:23.867 INFO:teuthology.orchestra.run.smithi107.stdout: "continuation_prompt": ">", 2020-06-12T02:37:23.867 INFO:teuthology.orchestra.run.smithi107.stdout: "crash_dir": "/var/lib/ceph/crash", 2020-06-12T02:37:23.867 INFO:teuthology.orchestra.run.smithi107.stdout: "crimson_osd_obc_lru_size": "10", 2020-06-12T02:37:23.867 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_location": "", 2020-06-12T02:37:23.868 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_location_hook": "", 2020-06-12T02:37:23.868 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_location_hook_timeout": "10", 2020-06-12T02:37:23.868 INFO:teuthology.orchestra.run.smithi107.stdout: "daemonize": "false", 2020-06-12T02:37:23.868 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_allow_any_pool_priority": "false", 2020-06-12T02:37:23.868 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_asok": "1/5", 2020-06-12T02:37:23.868 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_asok_assert_abort": "false", 2020-06-12T02:37:23.869 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_asserts_on_shutdown": "true", 2020-06-12T02:37:23.869 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_auth": "20/20", 2020-06-12T02:37:23.869 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_bdev": "1/3", 2020-06-12T02:37:23.869 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_bluefs": "1/5", 2020-06-12T02:37:23.869 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_bluestore": "1/5", 2020-06-12T02:37:23.870 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_buffer": "0/1", 2020-06-12T02:37:23.870 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_civetweb": "1/10", 2020-06-12T02:37:23.870 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_client": "0/5", 2020-06-12T02:37:23.870 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_compressor": "1/5", 2020-06-12T02:37:23.870 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_context": "0/1", 2020-06-12T02:37:23.870 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_crush": "1/1", 2020-06-12T02:37:23.871 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_crypto": "1/5", 2020-06-12T02:37:23.871 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_deliberately_leak_memory": "false", 2020-06-12T02:37:23.871 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_disable_randomized_ping": "false", 2020-06-12T02:37:23.871 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_dpdk": "1/5", 2020-06-12T02:37:23.871 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_eventtrace": "1/5", 2020-06-12T02:37:23.872 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_filer": "0/1", 2020-06-12T02:37:23.872 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_filestore": "1/3", 2020-06-12T02:37:23.872 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_finisher": "1/1", 2020-06-12T02:37:23.872 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_fuse": "1/5", 2020-06-12T02:37:23.872 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_heartbeat_testing_span": "0", 2020-06-12T02:37:23.872 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_heartbeatmap": "1/5", 2020-06-12T02:37:23.873 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_immutable_obj_cache": "0/5", 2020-06-12T02:37:23.873 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_javaclient": "1/5", 2020-06-12T02:37:23.873 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_journal": "1/3", 2020-06-12T02:37:23.873 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_journaler": "0/5", 2020-06-12T02:37:23.873 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_kstore": "1/5", 2020-06-12T02:37:23.874 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_leveldb": "4/5", 2020-06-12T02:37:23.874 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_lockdep": "0/1", 2020-06-12T02:37:23.874 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_mds": "1/5", 2020-06-12T02:37:23.874 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_mds_balancer": "1/5", 2020-06-12T02:37:23.874 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_mds_locker": "1/5", 2020-06-12T02:37:23.874 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_mds_log": "1/5", 2020-06-12T02:37:23.875 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_mds_log_expire": "1/5", 2020-06-12T02:37:23.875 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_mds_migrator": "1/5", 2020-06-12T02:37:23.875 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_memdb": "4/5", 2020-06-12T02:37:23.875 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_mgr": "1/5", 2020-06-12T02:37:23.875 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_mgrc": "1/5", 2020-06-12T02:37:23.875 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_mon": "20/20", 2020-06-12T02:37:23.876 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_monc": "0/10", 2020-06-12T02:37:23.876 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_ms": "1/1", 2020-06-12T02:37:23.876 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_none": "0/5", 2020-06-12T02:37:23.876 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_objclass": "0/5", 2020-06-12T02:37:23.876 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_objectcacher": "0/5", 2020-06-12T02:37:23.877 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_objecter": "0/1", 2020-06-12T02:37:23.877 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_optracker": "0/5", 2020-06-12T02:37:23.877 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_osd": "1/5", 2020-06-12T02:37:23.877 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_paxos": "20/20", 2020-06-12T02:37:23.877 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_perfcounter": "1/5", 2020-06-12T02:37:23.877 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_prioritycache": "1/5", 2020-06-12T02:37:23.878 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rados": "0/5", 2020-06-12T02:37:23.878 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rbd": "0/5", 2020-06-12T02:37:23.878 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rbd_mirror": "0/5", 2020-06-12T02:37:23.878 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rbd_replay": "0/5", 2020-06-12T02:37:23.878 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rbd_rwl": "0/5", 2020-06-12T02:37:23.879 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_refs": "0/0", 2020-06-12T02:37:23.879 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_reserver": "1/1", 2020-06-12T02:37:23.879 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rgw": "1/5", 2020-06-12T02:37:23.879 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rgw_sync": "1/5", 2020-06-12T02:37:23.879 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rocksdb": "4/5", 2020-06-12T02:37:23.879 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_shell": "false", 2020-06-12T02:37:23.880 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_striper": "0/1", 2020-06-12T02:37:23.880 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_test": "0/5", 2020-06-12T02:37:23.880 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_throttle": "1/1", 2020-06-12T02:37:23.880 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_timer": "0/1", 2020-06-12T02:37:23.880 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_tp": "0/5", 2020-06-12T02:37:23.880 INFO:teuthology.orchestra.run.smithi107.stdout: "device_failure_prediction_mode": "none", 2020-06-12T02:37:23.881 INFO:teuthology.orchestra.run.smithi107.stdout: "echo": "false", 2020-06-12T02:37:23.881 INFO:teuthology.orchestra.run.smithi107.stdout: "editor": "vim", 2020-06-12T02:37:23.881 INFO:teuthology.orchestra.run.smithi107.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2020-06-12T02:37:23.881 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2020-06-12T02:37:23.881 INFO:teuthology.orchestra.run.smithi107.stdout: "err_to_graylog": "false", 2020-06-12T02:37:23.882 INFO:teuthology.orchestra.run.smithi107.stdout: "err_to_stderr": "true", 2020-06-12T02:37:23.882 INFO:teuthology.orchestra.run.smithi107.stdout: "err_to_syslog": "false", 2020-06-12T02:37:23.882 INFO:teuthology.orchestra.run.smithi107.stdout: "event_tracing": "false", 2020-06-12T02:37:23.882 INFO:teuthology.orchestra.run.smithi107.stdout: "fake_statfs_for_testing": "0", 2020-06-12T02:37:23.882 INFO:teuthology.orchestra.run.smithi107.stdout: "fatal_signal_handlers": "true", 2020-06-12T02:37:23.882 INFO:teuthology.orchestra.run.smithi107.stdout: "feedback_to_output": "false", 2020-06-12T02:37:23.883 INFO:teuthology.orchestra.run.smithi107.stdout: "filer_max_purge_ops": "10", 2020-06-12T02:37:23.883 INFO:teuthology.orchestra.run.smithi107.stdout: "filer_max_truncate_ops": "128", 2020-06-12T02:37:23.883 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_apply_finisher_threads": "1", 2020-06-12T02:37:23.883 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_blackhole": "false", 2020-06-12T02:37:23.883 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_btrfs_clone_range": "true", 2020-06-12T02:37:23.884 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_btrfs_snap": "true", 2020-06-12T02:37:23.884 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_caller_concurrency": "10", 2020-06-12T02:37:23.884 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_collect_device_partition_information": "true", 2020-06-12T02:37:23.884 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_commit_timeout": "600.000000", 2020-06-12T02:37:23.884 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_debug_inject_read_err": "false", 2020-06-12T02:37:23.884 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_debug_omap_check": "false", 2020-06-12T02:37:23.885 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_debug_random_read_err": "0.000000", 2020-06-12T02:37:23.885 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_debug_verify_split": "false", 2020-06-12T02:37:23.885 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_dump_file": "", 2020-06-12T02:37:23.885 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2020-06-12T02:37:23.885 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_expected_throughput_ops": "200.000000", 2020-06-12T02:37:23.886 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_fadvise": "true", 2020-06-12T02:37:23.886 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_fail_eio": "true", 2020-06-12T02:37:23.886 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_fd_cache_shards": "16", 2020-06-12T02:37:23.886 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_fd_cache_size": "128", 2020-06-12T02:37:23.886 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_fiemap": "false", 2020-06-12T02:37:23.886 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_fiemap_threshold": "4096", 2020-06-12T02:37:23.887 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_fsync_flushes_journal_data": "false", 2020-06-12T02:37:23.887 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_index_retry_probability": "0.000000", 2020-06-12T02:37:23.887 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_inject_stall": "0", 2020-06-12T02:37:23.887 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_journal_parallel": "false", 2020-06-12T02:37:23.887 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_journal_trailing": "false", 2020-06-12T02:37:23.888 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_journal_writeahead": "false", 2020-06-12T02:37:23.888 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_kill_at": "0", 2020-06-12T02:37:23.888 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_alloc_hint_size": "1048576", 2020-06-12T02:37:23.888 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_inline_xattr_size": "0", 2020-06-12T02:37:23.888 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2020-06-12T02:37:23.888 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_inline_xattr_size_other": "512", 2020-06-12T02:37:23.889 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2020-06-12T02:37:23.889 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_inline_xattrs": "0", 2020-06-12T02:37:23.889 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2020-06-12T02:37:23.889 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_inline_xattrs_other": "2", 2020-06-12T02:37:23.889 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_inline_xattrs_xfs": "10", 2020-06-12T02:37:23.889 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_sync_interval": "5.000000", 2020-06-12T02:37:23.890 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_xattr_value_size": "0", 2020-06-12T02:37:23.890 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2020-06-12T02:37:23.890 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_xattr_value_size_other": "1024", 2020-06-12T02:37:23.890 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2020-06-12T02:37:23.890 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_merge_threshold": "-10", 2020-06-12T02:37:23.891 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_min_sync_interval": "0.010000", 2020-06-12T02:37:23.891 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_odsync_write": "false", 2020-06-12T02:37:23.891 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_omap_backend": "rocksdb", 2020-06-12T02:37:23.891 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_omap_backend_path": "", 2020-06-12T02:37:23.891 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_omap_header_cache_size": "1024", 2020-06-12T02:37:23.891 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_ondisk_finisher_threads": "1", 2020-06-12T02:37:23.892 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_op_thread_suicide_timeout": "180", 2020-06-12T02:37:23.892 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_op_thread_timeout": "60", 2020-06-12T02:37:23.892 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_op_threads": "2", 2020-06-12T02:37:23.892 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_punch_hole": "false", 2020-06-12T02:37:23.892 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2020-06-12T02:37:23.893 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2020-06-12T02:37:23.893 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2020-06-12T02:37:23.893 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_queue_high_threshhold": "0.900000", 2020-06-12T02:37:23.893 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_queue_low_threshhold": "0.300000", 2020-06-12T02:37:23.893 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_queue_max_bytes": "104857600", 2020-06-12T02:37:23.893 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2020-06-12T02:37:23.894 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2020-06-12T02:37:23.894 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2020-06-12T02:37:23.894 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_queue_max_ops": "50", 2020-06-12T02:37:23.894 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2020-06-12T02:37:23.894 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_seek_data_hole": "false", 2020-06-12T02:37:23.895 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_sloppy_crc": "false", 2020-06-12T02:37:23.895 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_sloppy_crc_block_size": "65536", 2020-06-12T02:37:23.895 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_splice": "false", 2020-06-12T02:37:23.895 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_split_multiple": "2", 2020-06-12T02:37:23.895 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_split_rand_factor": "20", 2020-06-12T02:37:23.895 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_update_to": "1000", 2020-06-12T02:37:23.897 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2020-06-12T02:37:23.897 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2020-06-12T02:37:23.898 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2020-06-12T02:37:23.898 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2020-06-12T02:37:23.898 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2020-06-12T02:37:23.898 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2020-06-12T02:37:23.898 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_enable": "true", 2020-06-12T02:37:23.899 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2020-06-12T02:37:23.899 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2020-06-12T02:37:23.899 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2020-06-12T02:37:23.899 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2020-06-12T02:37:23.899 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2020-06-12T02:37:23.900 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2020-06-12T02:37:23.900 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_xfs_extsize": "false", 2020-06-12T02:37:23.900 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_zfs_snap": "false", 2020-06-12T02:37:23.900 INFO:teuthology.orchestra.run.smithi107.stdout: "fio_dir": "/tmp/fio", 2020-06-12T02:37:23.900 INFO:teuthology.orchestra.run.smithi107.stdout: "fsid": "2f5030a1-3206-43c9-8471-d10429688d36", 2020-06-12T02:37:23.900 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_allow_other": "true", 2020-06-12T02:37:23.901 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_atomic_o_trunc": "true", 2020-06-12T02:37:23.901 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_big_writes": "true", 2020-06-12T02:37:23.901 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_debug": "false", 2020-06-12T02:37:23.901 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_default_permissions": "false", 2020-06-12T02:37:23.901 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_disable_pagecache": "false", 2020-06-12T02:37:23.902 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_max_write": "0", 2020-06-12T02:37:23.902 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_multithreaded": "true", 2020-06-12T02:37:23.902 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_require_active_mds": "true", 2020-06-12T02:37:23.902 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_set_user_groups": "true", 2020-06-12T02:37:23.902 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_syncfs_on_mksnap": "true", 2020-06-12T02:37:23.902 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_use_invalidate_cb": "true", 2020-06-12T02:37:23.903 INFO:teuthology.orchestra.run.smithi107.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2020-06-12T02:37:23.903 INFO:teuthology.orchestra.run.smithi107.stdout: "gss_target_name": "ceph", 2020-06-12T02:37:23.903 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_file": "", 2020-06-12T02:37:23.903 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_inject_failure": "0", 2020-06-12T02:37:23.903 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_interval": "5", 2020-06-12T02:37:23.904 INFO:teuthology.orchestra.run.smithi107.stdout: "host": "", 2020-06-12T02:37:23.904 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2020-06-12T02:37:23.904 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_max_inflight_ops": "128", 2020-06-12T02:37:23.904 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_max_size": "1073741824", 2020-06-12T02:37:23.904 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_path": "/tmp", 2020-06-12T02:37:23.904 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2020-06-12T02:37:23.905 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_watermark": "0.100000", 2020-06-12T02:37:23.905 INFO:teuthology.orchestra.run.smithi107.stdout: "inject_early_sigterm": "false", 2020-06-12T02:37:23.905 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_aio": "true", 2020-06-12T02:37:23.905 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_align_min_size": "65536", 2020-06-12T02:37:23.905 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_block_align": "true", 2020-06-12T02:37:23.906 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_block_size": "4096", 2020-06-12T02:37:23.906 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_dio": "true", 2020-06-12T02:37:23.906 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_discard": "false", 2020-06-12T02:37:23.906 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_force_aio": "false", 2020-06-12T02:37:23.906 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_ignore_corruption": "false", 2020-06-12T02:37:23.906 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_max_write_bytes": "10485760", 2020-06-12T02:37:23.907 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_max_write_entries": "100", 2020-06-12T02:37:23.907 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_replay_from": "0", 2020-06-12T02:37:23.907 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_throttle_high_multiple": "0.000000", 2020-06-12T02:37:23.907 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_throttle_high_threshhold": "0.900000", 2020-06-12T02:37:23.907 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_throttle_low_threshhold": "0.600000", 2020-06-12T02:37:23.908 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_throttle_max_multiple": "0.000000", 2020-06-12T02:37:23.908 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_write_header_frequency": "0", 2020-06-12T02:37:23.908 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_zero_on_create": "false", 2020-06-12T02:37:23.908 INFO:teuthology.orchestra.run.smithi107.stdout: "journaler_prefetch_periods": "10", 2020-06-12T02:37:23.908 INFO:teuthology.orchestra.run.smithi107.stdout: "journaler_prezero_periods": "5", 2020-06-12T02:37:23.908 INFO:teuthology.orchestra.run.smithi107.stdout: "journaler_write_head_interval": "15", 2020-06-12T02:37:23.909 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "", 2020-06-12T02:37:23.909 INFO:teuthology.orchestra.run.smithi107.stdout: "keyfile": "", 2020-06-12T02:37:23.909 INFO:teuthology.orchestra.run.smithi107.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2020-06-12T02:37:23.909 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_backend": "rocksdb", 2020-06-12T02:37:23.909 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_default_stripe_size": "65536", 2020-06-12T02:37:23.910 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_fsck_on_mount": "false", 2020-06-12T02:37:23.910 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_fsck_on_mount_deep": "true", 2020-06-12T02:37:23.910 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_max_bytes": "67108864", 2020-06-12T02:37:23.910 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_max_ops": "512", 2020-06-12T02:37:23.910 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_nid_prealloc": "1024", 2020-06-12T02:37:23.910 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_onode_map_size": "1024", 2020-06-12T02:37:23.911 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2020-06-12T02:37:23.911 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_sync_submit_transaction": "false", 2020-06-12T02:37:23.911 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_sync_transaction": "false", 2020-06-12T02:37:23.911 INFO:teuthology.orchestra.run.smithi107.stdout: "leveldb_block_size": "65536", 2020-06-12T02:37:23.911 INFO:teuthology.orchestra.run.smithi107.stdout: "leveldb_bloom_size": "0", 2020-06-12T02:37:23.912 INFO:teuthology.orchestra.run.smithi107.stdout: "leveldb_cache_size": "536870912", 2020-06-12T02:37:23.912 INFO:teuthology.orchestra.run.smithi107.stdout: "leveldb_compact_on_mount": "false", 2020-06-12T02:37:23.912 INFO:teuthology.orchestra.run.smithi107.stdout: "leveldb_compression": "false", 2020-06-12T02:37:23.912 INFO:teuthology.orchestra.run.smithi107.stdout: "leveldb_log": "", 2020-06-12T02:37:23.912 INFO:teuthology.orchestra.run.smithi107.stdout: "leveldb_log_to_ceph_log": "true", 2020-06-12T02:37:23.912 INFO:teuthology.orchestra.run.smithi107.stdout: "leveldb_max_open_files": "0", 2020-06-12T02:37:23.913 INFO:teuthology.orchestra.run.smithi107.stdout: "leveldb_paranoid": "false", 2020-06-12T02:37:23.913 INFO:teuthology.orchestra.run.smithi107.stdout: "leveldb_write_buffer_size": "33554432", 2020-06-12T02:37:23.913 INFO:teuthology.orchestra.run.smithi107.stdout: "lockdep": "false", 2020-06-12T02:37:23.913 INFO:teuthology.orchestra.run.smithi107.stdout: "lockdep_force_backtrace": "false", 2020-06-12T02:37:23.913 INFO:teuthology.orchestra.run.smithi107.stdout: "log_coarse_timestamps": "true", 2020-06-12T02:37:23.914 INFO:teuthology.orchestra.run.smithi107.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2020-06-12T02:37:23.914 INFO:teuthology.orchestra.run.smithi107.stdout: "log_flush_on_exit": "false", 2020-06-12T02:37:23.914 INFO:teuthology.orchestra.run.smithi107.stdout: "log_graylog_host": "127.0.0.1", 2020-06-12T02:37:23.914 INFO:teuthology.orchestra.run.smithi107.stdout: "log_graylog_port": "12201", 2020-06-12T02:37:23.914 INFO:teuthology.orchestra.run.smithi107.stdout: "log_max_new": "1000", 2020-06-12T02:37:23.914 INFO:teuthology.orchestra.run.smithi107.stdout: "log_max_recent": "10000", 2020-06-12T02:37:23.915 INFO:teuthology.orchestra.run.smithi107.stdout: "log_stderr_prefix": "", 2020-06-12T02:37:23.915 INFO:teuthology.orchestra.run.smithi107.stdout: "log_stop_at_utilization": "0.970000", 2020-06-12T02:37:23.915 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": "true", 2020-06-12T02:37:23.915 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_graylog": "false", 2020-06-12T02:37:23.915 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_stderr": "false", 2020-06-12T02:37:23.916 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_syslog": "false", 2020-06-12T02:37:23.916 INFO:teuthology.orchestra.run.smithi107.stdout: "max_completion_items": "50", 2020-06-12T02:37:23.916 INFO:teuthology.orchestra.run.smithi107.stdout: "max_rotating_auth_attempts": "10", 2020-06-12T02:37:23.916 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_action_on_write_error": "1", 2020-06-12T02:37:23.916 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_export_pin": "true", 2020-06-12T02:37:23.916 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_fragment_dirs": "true", 2020-06-12T02:37:23.917 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2020-06-12T02:37:23.917 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_fragment_interval": "5", 2020-06-12T02:37:23.917 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_fragment_size_max": "100000", 2020-06-12T02:37:23.917 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_idle_threshold": "0.000000", 2020-06-12T02:37:23.917 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_interval": "10", 2020-06-12T02:37:23.918 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_max": "-1", 2020-06-12T02:37:23.918 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_max_until": "-1", 2020-06-12T02:37:23.918 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_merge_size": "50", 2020-06-12T02:37:23.918 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_midchunk": "0.300000", 2020-06-12T02:37:23.918 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_min_rebalance": "0.100000", 2020-06-12T02:37:23.918 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_min_start": "0.200000", 2020-06-12T02:37:23.919 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_minchunk": "0.001000", 2020-06-12T02:37:23.919 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_mode": "0", 2020-06-12T02:37:23.919 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_need_max": "1.200000", 2020-06-12T02:37:23.919 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_need_min": "0.800000", 2020-06-12T02:37:23.919 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_replicate_threshold": "8000.000000", 2020-06-12T02:37:23.919 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_sample_interval": "3.000000", 2020-06-12T02:37:23.920 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_split_bits": "3", 2020-06-12T02:37:23.920 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_split_rd": "25000.000000", 2020-06-12T02:37:23.920 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_split_size": "10000", 2020-06-12T02:37:23.920 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_split_wr": "10000.000000", 2020-06-12T02:37:23.920 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_target_decay": "10.000000", 2020-06-12T02:37:23.921 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2020-06-12T02:37:23.921 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_beacon_grace": "15.000000", 2020-06-12T02:37:23.921 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_beacon_interval": "4.000000", 2020-06-12T02:37:23.921 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache_memory_limit": "4294967296", 2020-06-12T02:37:23.921 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache_mid": "0.700000", 2020-06-12T02:37:23.921 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache_release_free_interval": "10", 2020-06-12T02:37:23.922 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache_reservation": "0.050000", 2020-06-12T02:37:23.922 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache_trim_decay_rate": "1.000000", 2020-06-12T02:37:23.922 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache_trim_interval": "1", 2020-06-12T02:37:23.922 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache_trim_threshold": "65536", 2020-06-12T02:37:23.922 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2020-06-12T02:37:23.923 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_client_delegate_inos_pct": "50", 2020-06-12T02:37:23.923 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_client_prealloc_inos": "1000", 2020-06-12T02:37:23.923 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2020-06-12T02:37:23.923 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_damage_table_max_entries": "10000", 2020-06-12T02:37:23.923 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2020-06-12T02:37:23.923 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_debug_auth_pins": "false", 2020-06-12T02:37:23.924 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_debug_frag": "false", 2020-06-12T02:37:23.924 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_debug_scatterstat": "false", 2020-06-12T02:37:23.924 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_debug_subtrees": "false", 2020-06-12T02:37:23.924 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_decay_halflife": "5.000000", 2020-06-12T02:37:23.924 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_default_dir_hash": "2", 2020-06-12T02:37:23.924 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_defer_session_stale": "true", 2020-06-12T02:37:23.925 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_dir_keys_per_op": "16384", 2020-06-12T02:37:23.925 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_dir_max_commit_size": "10", 2020-06-12T02:37:23.925 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_dirstat_min_interval": "1.000000", 2020-06-12T02:37:23.925 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_dump_cache_after_rejoin": "false", 2020-06-12T02:37:23.925 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_dump_cache_on_map": "false", 2020-06-12T02:37:23.926 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_dump_cache_threshold_file": "0", 2020-06-12T02:37:23.926 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2020-06-12T02:37:23.926 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_early_reply": "true", 2020-06-12T02:37:23.926 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_enable_op_tracker": "true", 2020-06-12T02:37:23.926 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_enforce_unique_name": "true", 2020-06-12T02:37:23.926 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_forward_all_requests_to_auth": "false", 2020-06-12T02:37:23.927 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_freeze_tree_timeout": "30.000000", 2020-06-12T02:37:23.927 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2020-06-12T02:37:23.927 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_health_cache_threshold": "1.500000", 2020-06-12T02:37:23.927 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_health_summarize_threshold": "10", 2020-06-12T02:37:23.927 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_heartbeat_grace": "15.000000", 2020-06-12T02:37:23.928 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_inject_migrator_session_race": "false", 2020-06-12T02:37:23.928 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2020-06-12T02:37:23.928 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_join_fs": "", 2020-06-12T02:37:23.928 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_journal_format": "1", 2020-06-12T02:37:23.928 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_create_at": "0", 2020-06-12T02:37:23.928 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_export_at": "0", 2020-06-12T02:37:23.929 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_import_at": "0", 2020-06-12T02:37:23.929 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_journal_at": "0", 2020-06-12T02:37:23.929 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_journal_expire_at": "0", 2020-06-12T02:37:23.929 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_journal_replay_at": "0", 2020-06-12T02:37:23.929 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_link_at": "0", 2020-06-12T02:37:23.929 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_mdstable_at": "0", 2020-06-12T02:37:23.930 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_openc_at": "0", 2020-06-12T02:37:23.930 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_rename_at": "0", 2020-06-12T02:37:23.930 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log_events_per_segment": "1024", 2020-06-12T02:37:23.930 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log_max_events": "-1", 2020-06-12T02:37:23.930 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log_max_segments": "128", 2020-06-12T02:37:23.931 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log_pause": "false", 2020-06-12T02:37:23.931 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log_segment_size": "0", 2020-06-12T02:37:23.931 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log_skip_corrupt_events": "false", 2020-06-12T02:37:23.931 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_caps_per_client": "1048576", 2020-06-12T02:37:23.931 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_completed_flushes": "100000", 2020-06-12T02:37:23.931 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_completed_requests": "100000", 2020-06-12T02:37:23.932 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_export_size": "20971520", 2020-06-12T02:37:23.932 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_file_recover": "32", 2020-06-12T02:37:23.932 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_purge_files": "64", 2020-06-12T02:37:23.932 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_purge_ops": "8192", 2020-06-12T02:37:23.932 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2020-06-12T02:37:23.933 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_retries_on_remount_failure": "5", 2020-06-12T02:37:23.933 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_scrub_ops_in_progress": "5", 2020-06-12T02:37:23.933 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_snaps_per_dir": "100", 2020-06-12T02:37:23.933 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_xattr_pairs_size": "65536", 2020-06-12T02:37:23.933 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_min_caps_per_client": "100", 2020-06-12T02:37:23.933 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_mon_shutdown_timeout": "5.000000", 2020-06-12T02:37:23.934 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_numa_node": "-1", 2020-06-12T02:37:23.934 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_op_complaint_time": "30.000000", 2020-06-12T02:37:23.934 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_op_history_duration": "600", 2020-06-12T02:37:23.934 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_op_history_size": "20", 2020-06-12T02:37:23.934 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_op_log_threshold": "5", 2020-06-12T02:37:23.935 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2020-06-12T02:37:23.935 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_recall_global_max_decay_threshold": "65536", 2020-06-12T02:37:23.935 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_recall_max_caps": "5000", 2020-06-12T02:37:23.935 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_recall_max_decay_rate": "2.500000", 2020-06-12T02:37:23.935 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_recall_max_decay_threshold": "16384", 2020-06-12T02:37:23.935 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_recall_warning_decay_rate": "60.000000", 2020-06-12T02:37:23.936 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_recall_warning_threshold": "32768", 2020-06-12T02:37:23.936 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_reconnect_timeout": "45.000000", 2020-06-12T02:37:23.936 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_replay_interval": "1.000000", 2020-06-12T02:37:23.936 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_replay_unsafe_with_closed_session": "false", 2020-06-12T02:37:23.936 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_request_load_average_decay_rate": "60.000000", 2020-06-12T02:37:23.937 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_root_ino_gid": "0", 2020-06-12T02:37:23.937 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_root_ino_uid": "0", 2020-06-12T02:37:23.937 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_scatter_nudge_interval": "5.000000", 2020-06-12T02:37:23.937 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_session_blacklist_on_evict": "true", 2020-06-12T02:37:23.937 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_session_blacklist_on_timeout": "true", 2020-06-12T02:37:23.937 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2020-06-12T02:37:23.938 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_session_cache_liveness_magnitude": "10", 2020-06-12T02:37:23.938 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_sessionmap_keys_per_op": "1024", 2020-06-12T02:37:23.938 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_shutdown_check": "0", 2020-06-12T02:37:23.938 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_skip_ino": "0", 2020-06-12T02:37:23.938 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_snap_max_uid": "4294967294", 2020-06-12T02:37:23.939 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_snap_min_uid": "0", 2020-06-12T02:37:23.939 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_snap_rstat": "false", 2020-06-12T02:37:23.939 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_task_status_update_interval": "2.000000", 2020-06-12T02:37:23.939 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_thrash_exports": "0", 2020-06-12T02:37:23.939 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_thrash_fragments": "0", 2020-06-12T02:37:23.939 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_tick_interval": "5.000000", 2020-06-12T02:37:23.940 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_verify_backtrace": "1", 2020-06-12T02:37:23.940 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_verify_scatter": "false", 2020-06-12T02:37:23.940 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_wipe_ino_prealloc": "false", 2020-06-12T02:37:23.940 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_wipe_sessions": "false", 2020-06-12T02:37:23.940 INFO:teuthology.orchestra.run.smithi107.stdout: "mempool_debug": "false", 2020-06-12T02:37:23.941 INFO:teuthology.orchestra.run.smithi107.stdout: "memstore_debug_omit_block_device_write": "false", 2020-06-12T02:37:23.941 INFO:teuthology.orchestra.run.smithi107.stdout: "memstore_device_bytes": "1073741824", 2020-06-12T02:37:23.941 INFO:teuthology.orchestra.run.smithi107.stdout: "memstore_page_set": "false", 2020-06-12T02:37:23.941 INFO:teuthology.orchestra.run.smithi107.stdout: "memstore_page_size": "65536", 2020-06-12T02:37:23.941 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_client_bytes": "134217728", 2020-06-12T02:37:23.941 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_client_messages": "512", 2020-06-12T02:37:23.942 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2020-06-12T02:37:23.942 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_connect_retry_interval": "1.000000", 2020-06-12T02:37:23.942 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2020-06-12T02:37:23.942 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2020-06-12T02:37:23.942 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_initial_modules": "restful iostat", 2020-06-12T02:37:23.943 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_mds_bytes": "134217728", 2020-06-12T02:37:23.943 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_mds_messages": "128", 2020-06-12T02:37:23.943 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2020-06-12T02:37:23.943 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_mon_bytes": "134217728", 2020-06-12T02:37:23.943 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_mon_messages": "128", 2020-06-12T02:37:23.943 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_osd_bytes": "536870912", 2020-06-12T02:37:23.944 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_osd_messages": "8192", 2020-06-12T02:37:23.944 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_service_beacon_grace": "60.000000", 2020-06-12T02:37:23.944 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_stats_period": "5", 2020-06-12T02:37:23.944 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_stats_threshold": "5", 2020-06-12T02:37:23.945 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_tick_period": "2", 2020-06-12T02:37:23.945 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_accept_timeout_factor": "2.000000", 2020-06-12T02:37:23.945 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_allow_pool_delete": "true", 2020-06-12T02:37:23.945 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2020-06-12T02:37:23.945 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2020-06-12T02:37:23.946 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_bytes": "104857600", 2020-06-12T02:37:23.946 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_directed_command_retry": "5", 2020-06-12T02:37:23.946 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_hunt_interval": "3.000000", 2020-06-12T02:37:23.946 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2020-06-12T02:37:23.946 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2020-06-12T02:37:23.946 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2020-06-12T02:37:23.947 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_hunt_parallel": "3", 2020-06-12T02:37:23.947 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_log_interval": "1.000000", 2020-06-12T02:37:23.947 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_max_log_entries_per_message": "1000", 2020-06-12T02:37:23.947 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_ping_interval": "10.000000", 2020-06-12T02:37:23.947 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_ping_timeout": "30.000000", 2020-06-12T02:37:23.948 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_clock_drift_allowed": "1.000000", 2020-06-12T02:37:23.948 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2020-06-12T02:37:23.948 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2020-06-12T02:37:23.948 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cluster_log_file_level": "debug", 2020-06-12T02:37:23.948 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cluster_log_to_file": "true", 2020-06-12T02:37:23.948 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cluster_log_to_graylog": "false", 2020-06-12T02:37:23.949 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2020-06-12T02:37:23.949 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cluster_log_to_graylog_port": "12201", 2020-06-12T02:37:23.949 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cluster_log_to_stderr": "false", 2020-06-12T02:37:23.949 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cluster_log_to_syslog": "default=false", 2020-06-12T02:37:23.949 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2020-06-12T02:37:23.950 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cluster_log_to_syslog_level": "info", 2020-06-12T02:37:23.950 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_compact_on_bootstrap": "false", 2020-06-12T02:37:23.950 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_compact_on_start": "false", 2020-06-12T02:37:23.950 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_compact_on_trim": "true", 2020-06-12T02:37:23.950 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_config_key_max_entry_size": "65536", 2020-06-12T02:37:23.950 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cpu_threads": "4", 2020-06-12T02:37:23.951 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_crush_min_required_version": "hammer", 2020-06-12T02:37:23.951 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_daemon_bytes": "419430400", 2020-06-12T02:37:23.951 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2020-06-12T02:37:23.951 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_data_avail_crit": "5", 2020-06-12T02:37:23.951 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_data_avail_warn": "5", 2020-06-12T02:37:23.951 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_data_size_warn": "16106127360", 2020-06-12T02:37:23.952 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_block_osdmap_trim": "false", 2020-06-12T02:37:23.952 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_deprecated_as_obsolete": "false", 2020-06-12T02:37:23.952 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_dump_json": "false", 2020-06-12T02:37:23.952 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2020-06-12T02:37:23.952 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_dump_transactions": "false", 2020-06-12T02:37:23.953 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_extra_checks": "false", 2020-06-12T02:37:23.953 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_no_initial_persistent_features": "false", 2020-06-12T02:37:23.953 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2020-06-12T02:37:23.953 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_no_require_nautilus": "false", 2020-06-12T02:37:23.953 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_no_require_octopus": "false", 2020-06-12T02:37:23.953 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2020-06-12T02:37:23.954 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_delta_reset_interval": "10.000000", 2020-06-12T02:37:23.954 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_dns_srv_name": "ceph-mon", 2020-06-12T02:37:23.954 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_election_timeout": "5.000000", 2020-06-12T02:37:23.954 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_enable_op_tracker": "true", 2020-06-12T02:37:23.954 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_fake_pool_delete": "false", 2020-06-12T02:37:23.955 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_force_quorum_join": "false", 2020-06-12T02:37:23.955 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_globalid_prealloc": "10000", 2020-06-12T02:37:23.955 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_health_log_update_period": "5", 2020-06-12T02:37:23.955 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_health_max_detail": "50", 2020-06-12T02:37:23.955 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_health_to_clog": "true", 2020-06-12T02:37:23.955 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_health_to_clog_interval": "3600", 2020-06-12T02:37:23.956 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2020-06-12T02:37:23.956 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_host": "172.21.15.107,[v2:172.21.15.107:3301,v1:172.21.15.107:6790],172.21.15.188", 2020-06-12T02:37:23.956 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_initial_members": "", 2020-06-12T02:37:23.956 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2020-06-12T02:37:23.956 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2020-06-12T02:37:23.957 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_inject_transaction_delay_max": "10.000000", 2020-06-12T02:37:23.957 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2020-06-12T02:37:23.957 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_keyvaluedb": "rocksdb", 2020-06-12T02:37:23.957 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_lease": "5.000000", 2020-06-12T02:37:23.957 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2020-06-12T02:37:23.957 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_lease_renew_interval_factor": "0.600000", 2020-06-12T02:37:23.958 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_log_max_summary": "50", 2020-06-12T02:37:23.958 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_max_log_entries_per_event": "4096", 2020-06-12T02:37:23.958 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_max_log_epochs": "500", 2020-06-12T02:37:23.958 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_max_mdsmap_epochs": "500", 2020-06-12T02:37:23.958 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_max_mgrmap_epochs": "500", 2020-06-12T02:37:23.959 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_max_osd": "10000", 2020-06-12T02:37:23.963 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_max_pg_per_osd": "10000", 2020-06-12T02:37:23.963 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_max_pool_pg_num": "65536", 2020-06-12T02:37:23.964 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_max_snap_prune_per_epoch": "100", 2020-06-12T02:37:23.964 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_mds_blacklist_interval": "86400.000000", 2020-06-12T02:37:23.964 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_mds_force_trim_to": "0", 2020-06-12T02:37:23.964 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_mds_skip_sanity": "false", 2020-06-12T02:37:23.964 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_memory_autotune": "true", 2020-06-12T02:37:23.965 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_memory_target": "2147483648", 2020-06-12T02:37:23.965 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_mgr_beacon_grace": "30", 2020-06-12T02:37:23.965 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_mgr_blacklist_interval": "86400.000000", 2020-06-12T02:37:23.965 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_mgr_digest_period": "5", 2020-06-12T02:37:23.965 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_mgr_inactive_grace": "60", 2020-06-12T02:37:23.965 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_mgr_mkfs_grace": "240", 2020-06-12T02:37:23.966 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2020-06-12T02:37:23.966 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_min_osdmap_epochs": "500", 2020-06-12T02:37:23.966 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_op_complaint_time": "30", 2020-06-12T02:37:23.966 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_op_history_duration": "600", 2020-06-12T02:37:23.966 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_op_history_size": "20", 2020-06-12T02:37:23.967 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_op_history_slow_op_size": "20", 2020-06-12T02:37:23.967 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_op_history_slow_op_threshold": "10", 2020-06-12T02:37:23.967 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_op_log_threshold": "5", 2020-06-12T02:37:23.967 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_adjust_down_out_interval": "true", 2020-06-12T02:37:23.967 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2020-06-12T02:37:23.967 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2020-06-12T02:37:23.968 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_auto_mark_in": "false", 2020-06-12T02:37:23.968 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_auto_mark_new_in": "true", 2020-06-12T02:37:23.968 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2020-06-12T02:37:23.968 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_blacklist_default_expire": "3600.000000", 2020-06-12T02:37:23.968 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_cache_size": "500", 2020-06-12T02:37:23.969 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_cache_size_min": "134217728", 2020-06-12T02:37:23.969 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_crush_smoke_test": "true", 2020-06-12T02:37:23.969 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_destroyed_out_interval": "600", 2020-06-12T02:37:23.969 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_down_out_interval": "600", 2020-06-12T02:37:23.969 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_down_out_subtree_limit": "rack", 2020-06-12T02:37:23.969 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2020-06-12T02:37:23.970 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_force_trim_to": "0", 2020-06-12T02:37:23.970 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_full_ratio": "0.950000", 2020-06-12T02:37:23.970 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_initial_require_min_compat_client": "jewel", 2020-06-12T02:37:23.970 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_laggy_halflife": "3600", 2020-06-12T02:37:23.970 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_laggy_max_interval": "300", 2020-06-12T02:37:23.971 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_laggy_weight": "0.300000", 2020-06-12T02:37:23.971 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2020-06-12T02:37:23.971 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_max_creating_pgs": "1024", 2020-06-12T02:37:23.971 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_max_initial_pgs": "1024", 2020-06-12T02:37:23.971 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_min_down_reporters": "2", 2020-06-12T02:37:23.971 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_min_in_ratio": "0.750000", 2020-06-12T02:37:23.972 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_min_up_ratio": "0.300000", 2020-06-12T02:37:23.972 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_nearfull_ratio": "0.850000", 2020-06-12T02:37:23.972 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_prime_pg_temp": "true", 2020-06-12T02:37:23.972 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2020-06-12T02:37:23.972 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2020-06-12T02:37:23.972 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_report_timeout": "900", 2020-06-12T02:37:23.973 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_reporter_subtree_level": "osd", 2020-06-12T02:37:23.973 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2020-06-12T02:37:23.973 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_warn_op_age": "32.000000", 2020-06-12T02:37:23.973 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osdmap_full_prune_enabled": "true", 2020-06-12T02:37:23.973 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osdmap_full_prune_interval": "10", 2020-06-12T02:37:23.974 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osdmap_full_prune_min": "10000", 2020-06-12T02:37:23.974 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osdmap_full_prune_txsize": "100", 2020-06-12T02:37:23.974 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2020-06-12T02:37:23.974 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_pg_stuck_threshold": "60", 2020-06-12T02:37:23.974 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2020-06-12T02:37:23.974 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_pg_warn_min_objects": "10000", 2020-06-12T02:37:23.975 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_pg_warn_min_per_osd": "0", 2020-06-12T02:37:23.975 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_pg_warn_min_pool_objects": "1000", 2020-06-12T02:37:23.975 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_pool_quota_crit_threshold": "0", 2020-06-12T02:37:23.975 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_pool_quota_warn_threshold": "0", 2020-06-12T02:37:23.975 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_probe_timeout": "2.000000", 2020-06-12T02:37:23.976 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_reweight_max_change": "0.050000", 2020-06-12T02:37:23.976 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_reweight_max_osds": "4", 2020-06-12T02:37:23.976 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_reweight_min_bytes_per_osd": "10", 2020-06-12T02:37:23.976 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_reweight_min_pgs_per_osd": "4", 2020-06-12T02:37:23.976 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2020-06-12T02:37:23.976 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2020-06-12T02:37:23.977 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2020-06-12T02:37:23.977 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_scrub_interval": "86400", 2020-06-12T02:37:23.977 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_scrub_max_keys": "100", 2020-06-12T02:37:23.977 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_scrub_timeout": "300", 2020-06-12T02:37:23.977 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_session_timeout": "300", 2020-06-12T02:37:23.978 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_smart_report_timeout": "5", 2020-06-12T02:37:23.978 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_stat_smooth_intervals": "6", 2020-06-12T02:37:23.978 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_subscribe_interval": "86400.000000", 2020-06-12T02:37:23.978 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_sync_debug": "false", 2020-06-12T02:37:23.978 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_sync_max_payload_keys": "2000", 2020-06-12T02:37:23.979 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_sync_max_payload_size": "1048576", 2020-06-12T02:37:23.979 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_sync_provider_kill_at": "0", 2020-06-12T02:37:23.979 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_sync_requester_kill_at": "0", 2020-06-12T02:37:23.979 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_sync_timeout": "60.000000", 2020-06-12T02:37:23.979 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_target_pg_per_osd": "100", 2020-06-12T02:37:23.979 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_tick_interval": "5", 2020-06-12T02:37:23.980 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_timecheck_interval": "300.000000", 2020-06-12T02:37:23.980 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_timecheck_skew_interval": "30.000000", 2020-06-12T02:37:23.980 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2020-06-12T02:37:23.980 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2020-06-12T02:37:23.980 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2020-06-12T02:37:23.980 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_misplaced": "false", 2020-06-12T02:37:23.981 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2020-06-12T02:37:23.981 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2020-06-12T02:37:23.981 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_pool_no_app": "true", 2020-06-12T02:37:23.981 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_pool_no_redundancy": "false", 2020-06-12T02:37:23.981 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2020-06-12T02:37:23.982 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2020-06-12T02:37:23.982 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2020-06-12T02:37:23.982 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_too_few_osds": "false", 2020-06-12T02:37:23.982 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2020-06-12T02:37:23.982 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2020-06-12T02:37:23.982 INFO:teuthology.orchestra.run.smithi107.stdout: "monmap": "", 2020-06-12T02:37:23.983 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_max_op_threads": "5", 2020-06-12T02:37:23.983 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_op_threads": "3", 2020-06-12T02:37:23.983 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_buffer_size": "131072", 2020-06-12T02:37:23.983 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_cm": "false", 2020-06-12T02:37:23.983 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_device_name": "", 2020-06-12T02:37:23.984 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_dscp": "96", 2020-06-12T02:37:23.984 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_enable_hugepage": "false", 2020-06-12T02:37:23.984 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_gid_idx": "0", 2020-06-12T02:37:23.984 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_local_gid": "", 2020-06-12T02:37:23.984 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_polling_us": "1000", 2020-06-12T02:37:23.984 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_port_num": "1", 2020-06-12T02:37:23.985 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_receive_buffers": "32768", 2020-06-12T02:37:23.985 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_receive_queue_len": "4096", 2020-06-12T02:37:23.985 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_roce_ver": "1", 2020-06-12T02:37:23.985 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_send_buffers": "1024", 2020-06-12T02:37:23.985 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_sl": "3", 2020-06-12T02:37:23.986 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_support_srq": "true", 2020-06-12T02:37:23.986 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_type": "ib", 2020-06-12T02:37:23.986 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_bind_before_connect": "false", 2020-06-12T02:37:23.986 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_bind_ipv4": "true", 2020-06-12T02:37:23.986 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_bind_ipv6": "false", 2020-06-12T02:37:23.986 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_bind_msgr1": "true", 2020-06-12T02:37:23.987 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_bind_msgr2": "true", 2020-06-12T02:37:23.987 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_bind_port_max": "7300", 2020-06-12T02:37:23.987 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_bind_port_min": "6800", 2020-06-12T02:37:23.987 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_bind_prefer_ipv4": "false", 2020-06-12T02:37:23.987 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_bind_retry_count": "3", 2020-06-12T02:37:23.988 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_bind_retry_delay": "5", 2020-06-12T02:37:23.988 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_blackhole_client": "false", 2020-06-12T02:37:23.988 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_blackhole_mds": "false", 2020-06-12T02:37:23.988 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_blackhole_mgr": "false", 2020-06-12T02:37:23.988 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_blackhole_mon": "false", 2020-06-12T02:37:23.988 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_blackhole_osd": "false", 2020-06-12T02:37:23.989 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_client_mode": "crc secure", 2020-06-12T02:37:23.989 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_cluster_mode": "crc secure", 2020-06-12T02:37:23.989 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_cluster_type": "", 2020-06-12T02:37:23.989 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_connection_idle_timeout": "900", 2020-06-12T02:37:23.989 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_connection_ready_timeout": "10", 2020-06-12T02:37:23.990 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_crc_data": "true", 2020-06-12T02:37:23.990 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_crc_header": "true", 2020-06-12T02:37:23.990 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_die_on_bad_msg": "false", 2020-06-12T02:37:23.990 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_die_on_bug": "true", 2020-06-12T02:37:23.990 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_die_on_old_message": "true", 2020-06-12T02:37:23.990 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_die_on_skipped_message": "false", 2020-06-12T02:37:23.991 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_die_on_unhandled_msg": "false", 2020-06-12T02:37:23.991 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dispatch_throttle_bytes": "104857600", 2020-06-12T02:37:23.991 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_coremask": "0xF", 2020-06-12T02:37:23.991 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_debug_allow_loopback": "false", 2020-06-12T02:37:23.991 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2020-06-12T02:37:23.991 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_host_ipv4_addr": "", 2020-06-12T02:37:23.992 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_hugepages": "", 2020-06-12T02:37:23.992 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_hw_flow_control": "true", 2020-06-12T02:37:23.992 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2020-06-12T02:37:23.992 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_lro": "true", 2020-06-12T02:37:23.992 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_memory_channel": "4", 2020-06-12T02:37:23.993 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2020-06-12T02:37:23.993 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_pmd": "", 2020-06-12T02:37:23.993 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_port_id": "0", 2020-06-12T02:37:23.993 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2020-06-12T02:37:23.993 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dump_corrupt_message_level": "1", 2020-06-12T02:37:23.993 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dump_on_send": "false", 2020-06-12T02:37:23.994 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_initial_backoff": "0.200000", 2020-06-12T02:37:23.994 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_inject_delay_max": "1.000000", 2020-06-12T02:37:23.994 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_inject_delay_msg_type": "", 2020-06-12T02:37:23.994 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_inject_delay_probability": "0.000000", 2020-06-12T02:37:23.994 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_inject_delay_type": "", 2020-06-12T02:37:23.995 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_inject_internal_delays": "0.000000", 2020-06-12T02:37:23.995 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_inject_socket_failures": "5000", 2020-06-12T02:37:23.995 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_learn_addr_from_peer": "true", 2020-06-12T02:37:23.995 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_max_accept_failures": "4", 2020-06-12T02:37:23.995 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_max_backoff": "15.000000", 2020-06-12T02:37:23.995 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_mon_client_mode": "secure crc", 2020-06-12T02:37:23.996 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_mon_cluster_mode": "secure crc", 2020-06-12T02:37:23.996 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_mon_service_mode": "secure crc", 2020-06-12T02:37:23.996 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2020-06-12T02:37:23.996 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_pq_min_cost": "65536", 2020-06-12T02:37:23.996 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_public_type": "", 2020-06-12T02:37:23.996 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_service_mode": "crc secure", 2020-06-12T02:37:23.997 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_tcp_listen_backlog": "512", 2020-06-12T02:37:23.997 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_tcp_nodelay": "true", 2020-06-12T02:37:23.997 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_tcp_prefetch_max_size": "4096", 2020-06-12T02:37:23.997 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_tcp_rcvbuf": "0", 2020-06-12T02:37:23.997 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_type": "async+posix", 2020-06-12T02:37:23.998 INFO:teuthology.orchestra.run.smithi107.stdout: "no_config_file": "false", 2020-06-12T02:37:23.998 INFO:teuthology.orchestra.run.smithi107.stdout: "objecter_completion_locks_per_session": "32", 2020-06-12T02:37:23.998 INFO:teuthology.orchestra.run.smithi107.stdout: "objecter_debug_inject_relock_delay": "false", 2020-06-12T02:37:23.998 INFO:teuthology.orchestra.run.smithi107.stdout: "objecter_inflight_op_bytes": "104857600", 2020-06-12T02:37:23.998 INFO:teuthology.orchestra.run.smithi107.stdout: "objecter_inflight_ops": "1024", 2020-06-12T02:37:23.998 INFO:teuthology.orchestra.run.smithi107.stdout: "objecter_inject_no_watch_ping": "false", 2020-06-12T02:37:23.999 INFO:teuthology.orchestra.run.smithi107.stdout: "objecter_retry_writes_after_first_reply": "false", 2020-06-12T02:37:23.999 INFO:teuthology.orchestra.run.smithi107.stdout: "objecter_tick_interval": "5.000000", 2020-06-12T02:37:23.999 INFO:teuthology.orchestra.run.smithi107.stdout: "objecter_timeout": "10.000000", 2020-06-12T02:37:23.999 INFO:teuthology.orchestra.run.smithi107.stdout: "objectstore_blackhole": "false", 2020-06-12T02:37:23.999 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_agent_delay_time": "5.000000", 2020-06-12T02:37:24.000 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_agent_hist_halflife": "1000", 2020-06-12T02:37:24.000 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_agent_max_low_ops": "2", 2020-06-12T02:37:24.000 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_agent_max_ops": "4", 2020-06-12T02:37:24.000 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_agent_min_evict_effort": "0.100000", 2020-06-12T02:37:24.000 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_agent_quantize_effort": "0.100000", 2020-06-12T02:37:24.000 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_agent_slop": "0.020000", 2020-06-12T02:37:24.001 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_allow_recovery_below_min_size": "true", 2020-06-12T02:37:24.001 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_async_recovery_min_cost": "1", 2020-06-12T02:37:24.001 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_auto_mark_unfound_lost": "false", 2020-06-12T02:37:24.001 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_backfill_retry_interval": "30.000000", 2020-06-12T02:37:24.001 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_backfill_scan_max": "512", 2020-06-12T02:37:24.002 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_backfill_scan_min": "64", 2020-06-12T02:37:24.002 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_backoff_on_degraded": "false", 2020-06-12T02:37:24.002 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_backoff_on_peering": "false", 2020-06-12T02:37:24.002 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_backoff_on_unfound": "true", 2020-06-12T02:37:24.002 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_beacon_report_interval": "300", 2020-06-12T02:37:24.002 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_bench_duration": "30", 2020-06-12T02:37:24.003 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_bench_large_size_max_throughput": "104857600", 2020-06-12T02:37:24.003 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_bench_max_block_size": "67108864", 2020-06-12T02:37:24.003 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_bench_small_size_max_iops": "100", 2020-06-12T02:37:24.003 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_blkin_trace_all": "false", 2020-06-12T02:37:24.003 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2020-06-12T02:37:24.004 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2020-06-12T02:37:24.004 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_check_for_log_corruption": "false", 2020-06-12T02:37:24.004 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_check_max_object_name_len_on_startup": "true", 2020-06-12T02:37:24.004 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas", 2020-06-12T02:37:24.004 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2020-06-12T02:37:24.004 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas", 2020-06-12T02:37:24.005 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_class_update_on_start": "true", 2020-06-12T02:37:24.005 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_client_message_cap": "100", 2020-06-12T02:37:24.005 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_client_message_size_cap": "524288000", 2020-06-12T02:37:24.005 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_client_op_priority": "63", 2020-06-12T02:37:24.005 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_client_watch_timeout": "30", 2020-06-12T02:37:24.006 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_command_max_records": "256", 2020-06-12T02:37:24.006 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_command_thread_suicide_timeout": "900", 2020-06-12T02:37:24.006 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_command_thread_timeout": "600", 2020-06-12T02:37:24.006 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_copyfrom_max_chunk": "8388608", 2020-06-12T02:37:24.006 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_crush_chooseleaf_type": "0", 2020-06-12T02:37:24.006 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_crush_initial_weight": "-1.000000", 2020-06-12T02:37:24.007 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_crush_update_on_start": "true", 2020-06-12T02:37:24.007 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_crush_update_weight_set": "true", 2020-06-12T02:37:24.007 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2020-06-12T02:37:24.007 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2020-06-12T02:37:24.007 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2020-06-12T02:37:24.008 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_drop_ping_duration": "0", 2020-06-12T02:37:24.008 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_drop_ping_probability": "0.000000", 2020-06-12T02:37:24.008 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_feed_pullee": "-1", 2020-06-12T02:37:24.008 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_inject_copyfrom_error": "false", 2020-06-12T02:37:24.008 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2020-06-12T02:37:24.008 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2020-06-12T02:37:24.009 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_misdirected_ops": "false", 2020-06-12T02:37:24.009 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_no_acting_change": "false", 2020-06-12T02:37:24.009 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_no_purge_strays": "false", 2020-06-12T02:37:24.009 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_op_order": "false", 2020-06-12T02:37:24.009 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_pg_log_writeout": "false", 2020-06-12T02:37:24.010 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_pretend_recovery_active": "false", 2020-06-12T02:37:24.010 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_random_push_read_error": "0.000000", 2020-06-12T02:37:24.010 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2020-06-12T02:37:24.010 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_shutdown": "false", 2020-06-12T02:37:24.010 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2020-06-12T02:37:24.010 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2020-06-12T02:37:24.011 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_verify_cached_snaps": "false", 2020-06-12T02:37:24.011 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_verify_missing_on_start": "false", 2020-06-12T02:37:24.011 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_verify_snaps": "false", 2020-06-12T02:37:24.011 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_verify_stray_on_activate": "false", 2020-06-12T02:37:24.011 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_deep_scrub_interval": "604800.000000", 2020-06-12T02:37:24.012 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_deep_scrub_keys": "1024", 2020-06-12T02:37:24.012 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2020-06-12T02:37:24.012 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2020-06-12T02:37:24.012 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2020-06-12T02:37:24.012 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_deep_scrub_stride": "524288", 2020-06-12T02:37:24.012 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2020-06-12T02:37:24.013 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_default_data_pool_replay_window": "5", 2020-06-12T02:37:24.013 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_default_notify_timeout": "30", 2020-06-12T02:37:24.013 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_delete_sleep": "0.000000", 2020-06-12T02:37:24.013 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_delete_sleep_hdd": "5.000000", 2020-06-12T02:37:24.013 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_delete_sleep_hybrid": "2.000000", 2020-06-12T02:37:24.014 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_delete_sleep_ssd": "0.000000", 2020-06-12T02:37:24.014 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_discard_disconnected_ops": "true", 2020-06-12T02:37:24.014 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_enable_op_tracker": "true", 2020-06-12T02:37:24.014 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_erasure_code_plugins": "jerasure lrc isa", 2020-06-12T02:37:24.014 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_failsafe_full_ratio": "0.970000", 2020-06-12T02:37:24.014 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_fast_fail_on_connection_refused": "true", 2020-06-12T02:37:24.015 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_fast_info": "true", 2020-06-12T02:37:24.015 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_fast_shutdown": "true", 2020-06-12T02:37:24.015 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_find_best_info_ignore_history_les": "false", 2020-06-12T02:37:24.015 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_force_auth_primary_missing_objects": "100", 2020-06-12T02:37:24.015 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2020-06-12T02:37:24.015 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_function_tracing": "false", 2020-06-12T02:37:24.016 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_heartbeat_grace": "20", 2020-06-12T02:37:24.016 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_heartbeat_interval": "6", 2020-06-12T02:37:24.016 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2020-06-12T02:37:24.016 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_heartbeat_min_peers": "10", 2020-06-12T02:37:24.016 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_heartbeat_min_size": "2000", 2020-06-12T02:37:24.017 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_heartbeat_stale": "600", 2020-06-12T02:37:24.017 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2020-06-12T02:37:24.017 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_hit_set_max_size": "100000", 2020-06-12T02:37:24.017 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_hit_set_min_size": "1000", 2020-06-12T02:37:24.017 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_hit_set_namespace": ".ceph-internal", 2020-06-12T02:37:24.017 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_ignore_stale_divergent_priors": "false", 2020-06-12T02:37:24.018 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2020-06-12T02:37:24.018 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_inject_failure_on_pg_removal": "false", 2020-06-12T02:37:24.018 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2020-06-12T02:37:24.018 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_journal_flush_on_shutdown": "true", 2020-06-12T02:37:24.018 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_journal_size": "5120", 2020-06-12T02:37:24.019 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_kill_backfill_at": "0", 2020-06-12T02:37:24.019 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_loop_before_reset_tphandle": "64", 2020-06-12T02:37:24.019 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_map_cache_size": "50", 2020-06-12T02:37:24.019 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_map_dedup": "true", 2020-06-12T02:37:24.019 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_map_message_max": "40", 2020-06-12T02:37:24.019 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_map_message_max_bytes": "10485760", 2020-06-12T02:37:24.020 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_map_share_max_epochs": "40", 2020-06-12T02:37:24.020 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_attr_name_len": "100", 2020-06-12T02:37:24.020 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_attr_size": "0", 2020-06-12T02:37:24.020 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_backfills": "1", 2020-06-12T02:37:24.020 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_markdown_count": "5", 2020-06-12T02:37:24.021 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_markdown_period": "600", 2020-06-12T02:37:24.021 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_object_name_len": "2048", 2020-06-12T02:37:24.021 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_object_namespace_len": "256", 2020-06-12T02:37:24.021 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_object_size": "134217728", 2020-06-12T02:37:24.021 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2020-06-12T02:37:24.021 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_omap_entries_per_request": "1024", 2020-06-12T02:37:24.022 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_pg_blocked_by": "16", 2020-06-12T02:37:24.022 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_pg_log_entries": "10000", 2020-06-12T02:37:24.022 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2020-06-12T02:37:24.022 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_pgls": "1024", 2020-06-12T02:37:24.022 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_push_cost": "8388608", 2020-06-12T02:37:24.022 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_push_objects": "10", 2020-06-12T02:37:24.023 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_scrubs": "1", 2020-06-12T02:37:24.023 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2020-06-12T02:37:24.023 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_trimming_pgs": "2", 2020-06-12T02:37:24.023 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_write_op_reply_len": "32", 2020-06-12T02:37:24.023 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_write_size": "90", 2020-06-12T02:37:24.024 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2020-06-12T02:37:24.024 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_scheduler_background_best_effort_lim": "999999", 2020-06-12T02:37:24.024 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_scheduler_background_best_effort_res": "1", 2020-06-12T02:37:24.024 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2020-06-12T02:37:24.024 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_scheduler_background_recovery_lim": "999999", 2020-06-12T02:37:24.024 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_scheduler_background_recovery_res": "1", 2020-06-12T02:37:24.025 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2020-06-12T02:37:24.025 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_scheduler_client_lim": "999999", 2020-06-12T02:37:24.025 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_scheduler_client_res": "1", 2020-06-12T02:37:24.025 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_scheduler_client_wgt": "1", 2020-06-12T02:37:24.025 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_memory_base": "805306368", 2020-06-12T02:37:24.026 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_memory_cache_min": "134217728", 2020-06-12T02:37:24.026 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_memory_cache_resize_interval": "1.000000", 2020-06-12T02:37:24.026 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_memory_expected_fragmentation": "0.150000", 2020-06-12T02:37:24.026 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_memory_target": "4294967296", 2020-06-12T02:37:24.026 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2020-06-12T02:37:24.026 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_min_pg_log_entries": "250", 2020-06-12T02:37:24.027 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_min_recovery_priority": "0", 2020-06-12T02:37:24.027 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mon_ack_timeout": "30.000000", 2020-06-12T02:37:24.027 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mon_heartbeat_interval": "30", 2020-06-12T02:37:24.027 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2020-06-12T02:37:24.027 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mon_report_interval": "5", 2020-06-12T02:37:24.028 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mon_report_max_in_flight": "2", 2020-06-12T02:37:24.028 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mon_shutdown_timeout": "5.000000", 2020-06-12T02:37:24.028 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_num_cache_shards": "32", 2020-06-12T02:37:24.028 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_num_op_tracker_shard": "32", 2020-06-12T02:37:24.028 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_numa_auto_affinity": "true", 2020-06-12T02:37:24.028 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_numa_node": "-1", 2020-06-12T02:37:24.029 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_numa_prefer_iface": "true", 2020-06-12T02:37:24.029 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_object_clean_region_max_num_intervals": "1000", 2020-06-12T02:37:24.029 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_objecter_finishers": "1", 2020-06-12T02:37:24.029 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_objectstore": "bluestore", 2020-06-12T02:37:24.029 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_objectstore_fuse": "false", 2020-06-12T02:37:24.030 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_objectstore_tracing": "false", 2020-06-12T02:37:24.030 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_complaint_time": "30.000000", 2020-06-12T02:37:24.030 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_history_duration": "600", 2020-06-12T02:37:24.030 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_history_size": "20", 2020-06-12T02:37:24.030 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_history_slow_op_size": "20", 2020-06-12T02:37:24.030 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2020-06-12T02:37:24.031 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_log_threshold": "5", 2020-06-12T02:37:24.031 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_num_shards": "0", 2020-06-12T02:37:24.031 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_num_shards_hdd": "5", 2020-06-12T02:37:24.031 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_num_shards_ssd": "8", 2020-06-12T02:37:24.031 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_num_threads_per_shard": "0", 2020-06-12T02:37:24.031 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2020-06-12T02:37:24.032 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2020-06-12T02:37:24.032 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2020-06-12T02:37:24.032 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_pq_min_cost": "65536", 2020-06-12T02:37:24.032 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_queue": "wpq", 2020-06-12T02:37:24.032 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_queue_cut_off": "high", 2020-06-12T02:37:24.033 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_thread_suicide_timeout": "150", 2020-06-12T02:37:24.033 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_thread_timeout": "15", 2020-06-12T02:37:24.033 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_open_classes_on_start": "true", 2020-06-12T02:37:24.033 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_os_flags": "0", 2020-06-12T02:37:24.033 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_peering_op_priority": "255", 2020-06-12T02:37:24.033 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_delete_cost": "1048576", 2020-06-12T02:37:24.034 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_delete_priority": "5", 2020-06-12T02:37:24.034 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2020-06-12T02:37:24.034 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2020-06-12T02:37:24.034 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_log_dups_tracked": "3000", 2020-06-12T02:37:24.034 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_log_trim_max": "10000", 2020-06-12T02:37:24.035 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_log_trim_min": "100", 2020-06-12T02:37:24.035 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2020-06-12T02:37:24.035 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_object_context_cache_count": "64", 2020-06-12T02:37:24.035 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_stat_report_interval_max": "500", 2020-06-12T02:37:24.035 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2020-06-12T02:37:24.035 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_cache_min_evict_age": "0", 2020-06-12T02:37:24.036 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_cache_min_flush_age": "0", 2020-06-12T02:37:24.036 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2020-06-12T02:37:24.036 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2020-06-12T02:37:24.036 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2020-06-12T02:37:24.036 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_crush_rule": "-1", 2020-06-12T02:37:24.036 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_ec_fast_read": "true", 2020-06-12T02:37:24.037 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd", 2020-06-12T02:37:24.037 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_flag_hashpspool": "true", 2020-06-12T02:37:24.037 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_flag_nodelete": "false", 2020-06-12T02:37:24.037 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_flag_nopgchange": "false", 2020-06-12T02:37:24.037 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_flag_nosizechange": "false", 2020-06-12T02:37:24.038 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_flags": "0", 2020-06-12T02:37:24.038 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2020-06-12T02:37:24.038 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_min_size": "0", 2020-06-12T02:37:24.038 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2020-06-12T02:37:24.038 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_pg_num": "32", 2020-06-12T02:37:24.038 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_pgp_num": "0", 2020-06-12T02:37:24.039 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2020-06-12T02:37:24.039 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_size": "2", 2020-06-12T02:37:24.039 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_type": "replicated", 2020-06-12T02:37:24.039 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2020-06-12T02:37:24.039 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_use_gmt_hitset": "true", 2020-06-12T02:37:24.039 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_push_per_object_cost": "1000", 2020-06-12T02:37:24.040 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_read_ec_check_for_errors": "false", 2020-06-12T02:37:24.040 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recover_clone_overlap": "true", 2020-06-12T02:37:24.040 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recover_clone_overlap_limit": "10", 2020-06-12T02:37:24.040 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_cost": "20971520", 2020-06-12T02:37:24.040 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_delay_start": "0.000000", 2020-06-12T02:37:24.041 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_max_active": "0", 2020-06-12T02:37:24.041 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_max_active_hdd": "3", 2020-06-12T02:37:24.041 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_max_active_ssd": "10", 2020-06-12T02:37:24.041 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_max_chunk": "8388608", 2020-06-12T02:37:24.041 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2020-06-12T02:37:24.041 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_max_single_start": "1", 2020-06-12T02:37:24.042 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_op_priority": "3", 2020-06-12T02:37:24.042 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_op_warn_multiple": "16", 2020-06-12T02:37:24.042 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_priority": "5", 2020-06-12T02:37:24.042 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_retry_interval": "30.000000", 2020-06-12T02:37:24.042 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_sleep": "0.000000", 2020-06-12T02:37:24.042 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_sleep_hdd": "0.100000", 2020-06-12T02:37:24.043 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2020-06-12T02:37:24.043 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_sleep_ssd": "0.000000", 2020-06-12T02:37:24.043 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_repair_during_recovery": "false", 2020-06-12T02:37:24.043 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_requested_scrub_priority": "120", 2020-06-12T02:37:24.043 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_rollback_to_cluster_snap": "", 2020-06-12T02:37:24.044 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_auto_repair": "false", 2020-06-12T02:37:24.044 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_auto_repair_num_errors": "5", 2020-06-12T02:37:24.044 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_backoff_ratio": "0.660000", 2020-06-12T02:37:24.044 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_begin_hour": "0", 2020-06-12T02:37:24.044 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_begin_week_day": "0", 2020-06-12T02:37:24.045 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_chunk_max": "25", 2020-06-12T02:37:24.045 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_chunk_min": "5", 2020-06-12T02:37:24.045 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_cost": "52428800", 2020-06-12T02:37:24.045 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_during_recovery": "false", 2020-06-12T02:37:24.046 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_end_hour": "24", 2020-06-12T02:37:24.046 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_end_week_day": "7", 2020-06-12T02:37:24.046 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_extended_sleep": "0.000000", 2020-06-12T02:37:24.046 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2020-06-12T02:37:24.046 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_invalid_stats": "true", 2020-06-12T02:37:24.046 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_load_threshold": "0.500000", 2020-06-12T02:37:24.047 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_max_interval": "604800.000000", 2020-06-12T02:37:24.047 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_max_preemptions": "5", 2020-06-12T02:37:24.047 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_min_interval": "86400.000000", 2020-06-12T02:37:24.047 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_priority": "5", 2020-06-12T02:37:24.047 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_sleep": "0.000000", 2020-06-12T02:37:24.048 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_shutdown_pgref_assert": "false", 2020-06-12T02:37:24.048 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_skip_data_digest": "false", 2020-06-12T02:37:24.048 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_smart_report_timeout": "5", 2020-06-12T02:37:24.048 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_snap_trim_cost": "1048576", 2020-06-12T02:37:24.048 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_snap_trim_priority": "5", 2020-06-12T02:37:24.048 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_snap_trim_sleep": "0.000000", 2020-06-12T02:37:24.049 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2020-06-12T02:37:24.049 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2020-06-12T02:37:24.049 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2020-06-12T02:37:24.049 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_stats_ack_timeout_decay": "0.900000", 2020-06-12T02:37:24.049 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_stats_ack_timeout_factor": "2.000000", 2020-06-12T02:37:24.050 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2020-06-12T02:37:24.050 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_target_transaction_size": "30", 2020-06-12T02:37:24.050 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tier_default_cache_hit_set_count": "4", 2020-06-12T02:37:24.050 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2020-06-12T02:37:24.050 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2020-06-12T02:37:24.050 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2020-06-12T02:37:24.051 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2020-06-12T02:37:24.051 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2020-06-12T02:37:24.051 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2020-06-12T02:37:24.051 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tier_default_cache_mode": "writeback", 2020-06-12T02:37:24.051 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2020-06-12T02:37:24.052 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tier_promote_max_objects_sec": "25", 2020-06-12T02:37:24.052 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tracing": "false", 2020-06-12T02:37:24.052 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_use_stale_snap": "false", 2020-06-12T02:37:24.052 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2020-06-12T02:37:24.052 INFO:teuthology.orchestra.run.smithi107.stdout: "osdc_blkin_trace_all": "false", 2020-06-12T02:37:24.053 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_kill_at": "0", 2020-06-12T02:37:24.053 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_max_join_drift": "10", 2020-06-12T02:37:24.053 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_min": "500", 2020-06-12T02:37:24.053 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_min_wait": "0.050000", 2020-06-12T02:37:24.053 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_propose_interval": "1.000000", 2020-06-12T02:37:24.054 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_service_trim_max": "500", 2020-06-12T02:37:24.054 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_service_trim_min": "250", 2020-06-12T02:37:24.054 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_stash_full_interval": "25", 2020-06-12T02:37:24.054 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_trim_max": "500", 2020-06-12T02:37:24.054 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_trim_min": "250", 2020-06-12T02:37:24.054 INFO:teuthology.orchestra.run.smithi107.stdout: "perf": "true", 2020-06-12T02:37:24.055 INFO:teuthology.orchestra.run.smithi107.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2020-06-12T02:37:24.055 INFO:teuthology.orchestra.run.smithi107.stdout: "plugin_crypto_accelerator": "crypto_isal", 2020-06-12T02:37:24.055 INFO:teuthology.orchestra.run.smithi107.stdout: "plugin_dir": "/usr/lib64/ceph", 2020-06-12T02:37:24.055 INFO:teuthology.orchestra.run.smithi107.stdout: "prompt": "\u001b[01;33mCephFS:~\u001b[96m/\u001b[0m\u001b[01;33m>>>\u001b[00m ", 2020-06-12T02:37:24.055 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "-", 2020-06-12T02:37:24.056 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrv": "", 2020-06-12T02:37:24.056 INFO:teuthology.orchestra.run.smithi107.stdout: "public_bind_addr": "-", 2020-06-12T02:37:24.056 INFO:teuthology.orchestra.run.smithi107.stdout: "public_network": "", 2020-06-12T02:37:24.056 INFO:teuthology.orchestra.run.smithi107.stdout: "public_network_interface": "", 2020-06-12T02:37:24.056 INFO:teuthology.orchestra.run.smithi107.stdout: "qat_compressor_enabled": "false", 2020-06-12T02:37:24.056 INFO:teuthology.orchestra.run.smithi107.stdout: "quiet": "false", 2020-06-12T02:37:24.057 INFO:teuthology.orchestra.run.smithi107.stdout: "rados_mon_op_timeout": "0.000000", 2020-06-12T02:37:24.057 INFO:teuthology.orchestra.run.smithi107.stdout: "rados_osd_op_timeout": "0.000000", 2020-06-12T02:37:24.057 INFO:teuthology.orchestra.run.smithi107.stdout: "rados_tracing": "false", 2020-06-12T02:37:24.057 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_atime_update_interval": "60", 2020-06-12T02:37:24.057 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2020-06-12T02:37:24.058 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_balance_parent_reads": "false", 2020-06-12T02:37:24.058 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_balance_snap_reads": "false", 2020-06-12T02:37:24.058 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_blacklist_expire_seconds": "0", 2020-06-12T02:37:24.058 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_blacklist_on_break_lock": "true", 2020-06-12T02:37:24.058 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_blkin_trace_all": "false", 2020-06-12T02:37:24.058 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_cache": "true", 2020-06-12T02:37:24.059 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_cache_block_writes_upfront": "false", 2020-06-12T02:37:24.059 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_cache_max_dirty": "25165824", 2020-06-12T02:37:24.059 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_cache_max_dirty_age": "1.000000", 2020-06-12T02:37:24.059 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_cache_max_dirty_object": "0", 2020-06-12T02:37:24.059 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_cache_policy": "writearound", 2020-06-12T02:37:24.060 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_cache_size": "33554432", 2020-06-12T02:37:24.060 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_cache_target_dirty": "16777216", 2020-06-12T02:37:24.060 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_cache_writethrough_until_flush": "true", 2020-06-12T02:37:24.060 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_clone_copy_on_read": "false", 2020-06-12T02:37:24.060 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_compression_hint": "none", 2020-06-12T02:37:24.060 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_concurrent_management_ops": "10", 2020-06-12T02:37:24.061 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_default_clone_format": "auto", 2020-06-12T02:37:24.061 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_default_data_pool": "", 2020-06-12T02:37:24.061 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_default_features": "61", 2020-06-12T02:37:24.061 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_default_format": "2", 2020-06-12T02:37:24.061 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_default_map_options": "", 2020-06-12T02:37:24.062 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_default_order": "22", 2020-06-12T02:37:24.062 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_default_pool": "rbd", 2020-06-12T02:37:24.062 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_default_stripe_count": "0", 2020-06-12T02:37:24.062 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_default_stripe_unit": "0", 2020-06-12T02:37:24.062 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_disable_zero_copy_writes": "true", 2020-06-12T02:37:24.062 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_discard_granularity_bytes": "65536", 2020-06-12T02:37:24.063 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_discard_on_zeroed_write_same": "true", 2020-06-12T02:37:24.063 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_enable_alloc_hint": "true", 2020-06-12T02:37:24.063 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_io_scheduler": "simple", 2020-06-12T02:37:24.063 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2020-06-12T02:37:24.063 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_commit_age": "5.000000", 2020-06-12T02:37:24.063 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2020-06-12T02:37:24.064 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_max_payload_bytes": "16384", 2020-06-12T02:37:24.064 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_object_flush_age": "0.000000", 2020-06-12T02:37:24.064 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_object_flush_bytes": "1048576", 2020-06-12T02:37:24.064 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_object_flush_interval": "0", 2020-06-12T02:37:24.064 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2020-06-12T02:37:24.065 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2020-06-12T02:37:24.065 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_order": "24", 2020-06-12T02:37:24.065 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_pool": "", 2020-06-12T02:37:24.065 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_splay_width": "4", 2020-06-12T02:37:24.065 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_localize_parent_reads": "false", 2020-06-12T02:37:24.066 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_localize_snap_reads": "false", 2020-06-12T02:37:24.066 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2020-06-12T02:37:24.066 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2020-06-12T02:37:24.066 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2020-06-12T02:37:24.066 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2020-06-12T02:37:24.066 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2020-06-12T02:37:24.067 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2020-06-12T02:37:24.067 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_image_policy_type": "simple", 2020-06-12T02:37:24.067 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2020-06-12T02:37:24.067 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_image_state_check_interval": "30", 2020-06-12T02:37:24.067 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2020-06-12T02:37:24.068 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2020-06-12T02:37:24.068 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2020-06-12T02:37:24.068 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2020-06-12T02:37:24.068 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2020-06-12T02:37:24.068 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_memory_autotune": "true", 2020-06-12T02:37:24.068 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_memory_base": "805306368", 2020-06-12T02:37:24.069 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2020-06-12T02:37:24.069 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_memory_cache_min": "134217728", 2020-06-12T02:37:24.069 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2020-06-12T02:37:24.069 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2020-06-12T02:37:24.069 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_memory_target": "4294967296", 2020-06-12T02:37:24.069 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_perf_stats_prio": "5", 2020-06-12T02:37:24.070 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2020-06-12T02:37:24.070 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2020-06-12T02:37:24.070 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirroring_delete_delay": "0", 2020-06-12T02:37:24.070 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirroring_max_mirroring_snapshots": "3", 2020-06-12T02:37:24.070 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirroring_replay_delay": "0", 2020-06-12T02:37:24.071 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2020-06-12T02:37:24.071 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2020-06-12T02:37:24.071 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_move_to_trash_on_remove": "false", 2020-06-12T02:37:24.071 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2020-06-12T02:37:24.071 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mtime_update_interval": "60", 2020-06-12T02:37:24.071 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_non_blocking_aio": "true", 2020-06-12T02:37:24.072 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_op_thread_timeout": "60", 2020-06-12T02:37:24.072 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_op_threads": "1", 2020-06-12T02:37:24.072 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_parent_cache_enabled": "false", 2020-06-12T02:37:24.072 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_bps_burst": "0", 2020-06-12T02:37:24.072 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_bps_limit": "0", 2020-06-12T02:37:24.073 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_iops_burst": "0", 2020-06-12T02:37:24.073 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_iops_limit": "0", 2020-06-12T02:37:24.073 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_read_bps_burst": "0", 2020-06-12T02:37:24.073 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_read_bps_limit": "0", 2020-06-12T02:37:24.073 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_read_iops_burst": "0", 2020-06-12T02:37:24.073 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_read_iops_limit": "0", 2020-06-12T02:37:24.074 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_schedule_tick_min": "50", 2020-06-12T02:37:24.074 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_write_bps_burst": "0", 2020-06-12T02:37:24.074 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_write_bps_limit": "0", 2020-06-12T02:37:24.074 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_write_iops_burst": "0", 2020-06-12T02:37:24.074 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_write_iops_limit": "0", 2020-06-12T02:37:24.075 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_read_from_replica_policy": "default", 2020-06-12T02:37:24.075 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2020-06-12T02:37:24.075 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_readahead_max_bytes": "524288", 2020-06-12T02:37:24.075 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_readahead_trigger_requests": "10", 2020-06-12T02:37:24.075 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_request_timed_out_seconds": "30", 2020-06-12T02:37:24.076 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_rwl_enabled": "false", 2020-06-12T02:37:24.076 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_rwl_log_periodic_stats": "false", 2020-06-12T02:37:24.076 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_rwl_path": "/tmp", 2020-06-12T02:37:24.076 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_rwl_size": "1073741824", 2020-06-12T02:37:24.076 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_skip_partial_discard": "true", 2020-06-12T02:37:24.077 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2020-06-12T02:37:24.077 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_tracing": "false", 2020-06-12T02:37:24.077 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_validate_names": "true", 2020-06-12T02:37:24.077 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_validate_pool": "true", 2020-06-12T02:37:24.077 INFO:teuthology.orchestra.run.smithi107.stdout: "restapi_base_url": "", 2020-06-12T02:37:24.078 INFO:teuthology.orchestra.run.smithi107.stdout: "restapi_log_level": "", 2020-06-12T02:37:24.078 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_acl_grants_max_num": "100", 2020-06-12T02:37:24.078 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_admin_entry": "admin", 2020-06-12T02:37:24.078 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_barbican_url": "", 2020-06-12T02:37:24.078 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2020-06-12T02:37:24.079 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_bucket_default_quota_max_size": "-1", 2020-06-12T02:37:24.079 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_bucket_index_max_aio": "128", 2020-06-12T02:37:24.079 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_bucket_quota_cache_size": "10000", 2020-06-12T02:37:24.079 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_bucket_quota_soft_threshold": "0.950000", 2020-06-12T02:37:24.079 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_bucket_quota_ttl": "600", 2020-06-12T02:37:24.079 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_cache_enabled": "true", 2020-06-12T02:37:24.080 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_cache_expiry_interval": "900", 2020-06-12T02:37:24.080 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_cache_lru_size": "10000", 2020-06-12T02:37:24.080 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_content_length_compat": "false", 2020-06-12T02:37:24.080 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_copy_obj_progress": "true", 2020-06-12T02:37:24.081 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2020-06-12T02:37:24.081 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_cors_rules_max_num": "100", 2020-06-12T02:37:24.081 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_cross_domain_policy": "", 2020-06-12T02:37:24.081 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_default_encryption_key": "", 2020-06-12T02:37:24.081 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_require_ssl": "true", 2020-06-12T02:37:24.082 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2020-06-12T02:37:24.082 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2020-06-12T02:37:24.083 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_suppress_logs": "true", 2020-06-12T02:37:24.083 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_vault_addr": "", 2020-06-12T02:37:24.083 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_vault_auth": "token", 2020-06-12T02:37:24.083 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_vault_namespace": "", 2020-06-12T02:37:24.083 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_vault_prefix": "", 2020-06-12T02:37:24.084 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_vault_secret_engine": "transit", 2020-06-12T02:37:24.084 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_vault_token_file": "", 2020-06-12T02:37:24.084 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_curl_low_speed_limit": "1024", 2020-06-12T02:37:24.084 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_curl_low_speed_time": "300", 2020-06-12T02:37:24.084 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_curl_wait_timeout_ms": "1000", 2020-06-12T02:37:24.085 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2020-06-12T02:37:24.085 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_data_log_changes_size": "1000", 2020-06-12T02:37:24.085 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_data_log_num_shards": "128", 2020-06-12T02:37:24.085 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_data_log_obj_prefix": "data_log", 2020-06-12T02:37:24.085 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_data_log_window": "30", 2020-06-12T02:37:24.086 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_data_notify_interval_msec": "200", 2020-06-12T02:37:24.086 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_default_realm_info_oid": "default.realm", 2020-06-12T02:37:24.086 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_default_region_info_oid": "default.region", 2020-06-12T02:37:24.086 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_default_zone_info_oid": "default.zone", 2020-06-12T02:37:24.086 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2020-06-12T02:37:24.087 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_defer_to_bucket_acls": "", 2020-06-12T02:37:24.087 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_delete_multi_obj_max_num": "1000", 2020-06-12T02:37:24.087 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_admin_lim": "0.000000", 2020-06-12T02:37:24.087 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_admin_res": "100.000000", 2020-06-12T02:37:24.087 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2020-06-12T02:37:24.088 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_auth_lim": "0.000000", 2020-06-12T02:37:24.088 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_auth_res": "200.000000", 2020-06-12T02:37:24.088 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2020-06-12T02:37:24.088 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_data_lim": "0.000000", 2020-06-12T02:37:24.088 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_data_res": "500.000000", 2020-06-12T02:37:24.088 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_data_wgt": "500.000000", 2020-06-12T02:37:24.089 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2020-06-12T02:37:24.089 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_metadata_res": "500.000000", 2020-06-12T02:37:24.089 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2020-06-12T02:37:24.089 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dns_name": "", 2020-06-12T02:37:24.089 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dns_s3website_name": "", 2020-06-12T02:37:24.090 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dynamic_resharding": "true", 2020-06-12T02:37:24.090 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, pubsub", 2020-06-12T02:37:24.090 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_enable_gc_threads": "true", 2020-06-12T02:37:24.090 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_enable_lc_threads": "true", 2020-06-12T02:37:24.090 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_enable_ops_log": "false", 2020-06-12T02:37:24.090 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_enable_quota_threads": "true", 2020-06-12T02:37:24.091 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_enable_static_website": "false", 2020-06-12T02:37:24.091 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_enable_usage_log": "false", 2020-06-12T02:37:24.091 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_enforce_swift_acls": "true", 2020-06-12T02:37:24.091 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_exit_timeout_secs": "120", 2020-06-12T02:37:24.091 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_expose_bucket": "false", 2020-06-12T02:37:24.092 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_extended_http_attrs": "", 2020-06-12T02:37:24.092 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_fcgi_socket_backlog": "1024", 2020-06-12T02:37:24.092 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2020-06-12T02:37:24.092 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_frontends": "beast port=7480", 2020-06-12T02:37:24.092 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_gc_max_concurrent_io": "10", 2020-06-12T02:37:24.092 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_gc_max_deferred": "50", 2020-06-12T02:37:24.093 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2020-06-12T02:37:24.093 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_gc_max_objs": "32", 2020-06-12T02:37:24.093 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_gc_max_queue_size": "134213632", 2020-06-12T02:37:24.093 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_gc_max_trim_chunk": "16", 2020-06-12T02:37:24.093 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_gc_obj_min_wait": "7200", 2020-06-12T02:37:24.094 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_gc_processor_max_time": "3600", 2020-06-12T02:37:24.094 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_gc_processor_period": "3600", 2020-06-12T02:37:24.094 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_get_obj_max_req_size": "4194304", 2020-06-12T02:37:24.094 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_get_obj_window_size": "16777216", 2020-06-12T02:37:24.094 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_healthcheck_disabling_path": "", 2020-06-12T02:37:24.094 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_host": "", 2020-06-12T02:37:24.095 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ignore_get_invalid_range": "false", 2020-06-12T02:37:24.095 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_init_timeout": "300", 2020-06-12T02:37:24.095 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2020-06-12T02:37:24.095 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_accepted_admin_roles": "", 2020-06-12T02:37:24.095 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2020-06-12T02:37:24.096 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_admin_domain": "", 2020-06-12T02:37:24.096 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_admin_password": "", 2020-06-12T02:37:24.096 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_admin_password_path": "", 2020-06-12T02:37:24.096 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_admin_project": "", 2020-06-12T02:37:24.096 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_admin_tenant": "", 2020-06-12T02:37:24.096 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_admin_token": "", 2020-06-12T02:37:24.097 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_admin_token_path": "", 2020-06-12T02:37:24.097 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_admin_user": "", 2020-06-12T02:37:24.097 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_api_version": "2", 2020-06-12T02:37:24.097 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_barbican_domain": "", 2020-06-12T02:37:24.097 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_barbican_password": "", 2020-06-12T02:37:24.098 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_barbican_project": "", 2020-06-12T02:37:24.098 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_barbican_tenant": "", 2020-06-12T02:37:24.098 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_barbican_user": "", 2020-06-12T02:37:24.098 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_implicit_tenants": "false", 2020-06-12T02:37:24.098 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_token_cache_size": "10000", 2020-06-12T02:37:24.098 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_url": "", 2020-06-12T02:37:24.099 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_verify_ssl": "true", 2020-06-12T02:37:24.099 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_lc_debug_interval": "-1", 2020-06-12T02:37:24.099 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_lc_lock_max_time": "60", 2020-06-12T02:37:24.099 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_lc_max_objs": "32", 2020-06-12T02:37:24.099 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_lc_max_rules": "1000", 2020-06-12T02:37:24.100 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_lc_thread_delay": "0", 2020-06-12T02:37:24.100 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2020-06-12T02:37:24.100 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ldap_dnattr": "uid", 2020-06-12T02:37:24.100 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2020-06-12T02:37:24.100 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ldap_searchfilter": "", 2020-06-12T02:37:24.100 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2020-06-12T02:37:24.101 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ldap_uri": "ldaps://", 2020-06-12T02:37:24.101 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2020-06-12T02:37:24.101 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_list_bucket_min_readahead": "1000", 2020-06-12T02:37:24.101 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_list_buckets_max_chunk": "1000", 2020-06-12T02:37:24.101 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_log_http_headers": "", 2020-06-12T02:37:24.102 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_log_nonexistent_bucket": "false", 2020-06-12T02:37:24.102 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2020-06-12T02:37:24.102 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_log_object_name_utc": "false", 2020-06-12T02:37:24.102 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_attr_name_len": "0", 2020-06-12T02:37:24.102 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_attr_size": "0", 2020-06-12T02:37:24.102 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_attrs_num_in_req": "0", 2020-06-12T02:37:24.103 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_chunk_size": "4194304", 2020-06-12T02:37:24.103 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_concurrent_requests": "1024", 2020-06-12T02:37:24.103 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_dynamic_shards": "1999", 2020-06-12T02:37:24.103 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_listing_results": "1000", 2020-06-12T02:37:24.103 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_notify_retries": "3", 2020-06-12T02:37:24.103 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_objs_per_shard": "100000", 2020-06-12T02:37:24.104 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_put_param_size": "1048576", 2020-06-12T02:37:24.104 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_put_size": "5368709120", 2020-06-12T02:37:24.104 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_slo_entries": "1000", 2020-06-12T02:37:24.104 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_md_log_max_shards": "64", 2020-06-12T02:37:24.104 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_md_notify_interval_msec": "200", 2020-06-12T02:37:24.105 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_mime_types_file": "/etc/mime.types", 2020-06-12T02:37:24.105 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_mp_lock_max_time": "600", 2020-06-12T02:37:24.105 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_multipart_min_part_size": "5242880", 2020-06-12T02:37:24.105 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_multipart_part_upload_limit": "10000", 2020-06-12T02:37:24.105 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_fhcache_partitions": "3", 2020-06-12T02:37:24.105 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_fhcache_size": "2017", 2020-06-12T02:37:24.106 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2020-06-12T02:37:24.106 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_lru_lanes": "5", 2020-06-12T02:37:24.106 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_max_gc": "300", 2020-06-12T02:37:24.106 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_namespace_expire_secs": "300", 2020-06-12T02:37:24.106 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_s3_fast_attrs": "false", 2020-06-12T02:37:24.107 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_write_completion_interval_s": "10", 2020-06-12T02:37:24.107 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_num_async_rados_threads": "32", 2020-06-12T02:37:24.107 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_num_control_oids": "8", 2020-06-12T02:37:24.107 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_num_rados_handles": "1", 2020-06-12T02:37:24.107 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_numa_node": "-1", 2020-06-12T02:37:24.107 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_obj_stripe_size": "4194304", 2020-06-12T02:37:24.108 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_obj_tombstone_cache_size": "1000", 2020-06-12T02:37:24.108 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_objexp_chunk_size": "100", 2020-06-12T02:37:24.108 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_objexp_gc_interval": "600", 2020-06-12T02:37:24.108 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_objexp_hints_num_shards": "127", 2020-06-12T02:37:24.108 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_olh_pending_timeout_sec": "3600", 2020-06-12T02:37:24.108 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_op_thread_suicide_timeout": "0", 2020-06-12T02:37:24.109 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_op_thread_timeout": "600", 2020-06-12T02:37:24.109 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_op_tracing": "false", 2020-06-12T02:37:24.109 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_opa_token": "", 2020-06-12T02:37:24.109 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_opa_url": "", 2020-06-12T02:37:24.109 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_opa_verify_ssl": "true", 2020-06-12T02:37:24.110 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ops_log_data_backlog": "5242880", 2020-06-12T02:37:24.110 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ops_log_rados": "true", 2020-06-12T02:37:24.110 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ops_log_socket_path": "", 2020-06-12T02:37:24.110 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_override_bucket_index_max_shards": "0", 2020-06-12T02:37:24.110 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2020-06-12T02:37:24.110 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_period_push_interval": "2.000000", 2020-06-12T02:37:24.111 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_period_push_interval_max": "30.000000", 2020-06-12T02:37:24.111 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_period_root_pool": ".rgw.root", 2020-06-12T02:37:24.111 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_port": "", 2020-06-12T02:37:24.111 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_print_continue": "true", 2020-06-12T02:37:24.111 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_print_prohibited_content_length": "false", 2020-06-12T02:37:24.112 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_put_obj_max_window_size": "67108864", 2020-06-12T02:37:24.112 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_put_obj_min_window_size": "16777216", 2020-06-12T02:37:24.112 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2020-06-12T02:37:24.112 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_rados_pool_pg_num_min": "8", 2020-06-12T02:37:24.112 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_rados_pool_recovery_priority": "5", 2020-06-12T02:37:24.112 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_rados_tracing": "false", 2020-06-12T02:37:24.113 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_realm": "", 2020-06-12T02:37:24.113 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_realm_root_pool": ".rgw.root", 2020-06-12T02:37:24.113 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_region": "", 2020-06-12T02:37:24.113 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_region_root_pool": ".rgw.root", 2020-06-12T02:37:24.113 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_relaxed_region_enforcement": "false", 2020-06-12T02:37:24.113 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_relaxed_s3_bucket_names": "false", 2020-06-12T02:37:24.114 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2020-06-12T02:37:24.114 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_request_uri": "", 2020-06-12T02:37:24.114 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_reshard_batch_size": "64", 2020-06-12T02:37:24.114 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_reshard_bucket_lock_duration": "360", 2020-06-12T02:37:24.114 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_reshard_max_aio": "128", 2020-06-12T02:37:24.115 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_reshard_num_logs": "16", 2020-06-12T02:37:24.115 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_reshard_thread_interval": "600", 2020-06-12T02:37:24.115 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_resolve_cname": "false", 2020-06-12T02:37:24.115 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_rest_getusage_op_compat": "false", 2020-06-12T02:37:24.115 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_run_sync_thread": "true", 2020-06-12T02:37:24.115 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_s3_auth_order": "sts, external, local", 2020-06-12T02:37:24.116 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_s3_auth_use_keystone": "false", 2020-06-12T02:37:24.116 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_s3_auth_use_ldap": "false", 2020-06-12T02:37:24.116 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_s3_auth_use_rados": "true", 2020-06-12T02:37:24.116 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_s3_auth_use_sts": "false", 2020-06-12T02:37:24.116 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_s3_success_create_obj_status": "0", 2020-06-12T02:37:24.116 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_safe_max_objects_per_shard": "102400", 2020-06-12T02:37:24.117 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_scheduler_type": "throttler", 2020-06-12T02:37:24.117 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_script_uri": "", 2020-06-12T02:37:24.117 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_service_provider_name": "", 2020-06-12T02:37:24.117 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_shard_warning_threshold": "90.000000", 2020-06-12T02:37:24.117 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_socket_path": "", 2020-06-12T02:37:24.118 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sts_client_id": "", 2020-06-12T02:37:24.118 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sts_client_secret": "", 2020-06-12T02:37:24.118 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sts_entry": "sts", 2020-06-12T02:37:24.118 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sts_key": "sts", 2020-06-12T02:37:24.118 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sts_max_session_duration": "43200", 2020-06-12T02:37:24.118 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sts_token_introspection_url": "", 2020-06-12T02:37:24.119 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_account_in_url": "false", 2020-06-12T02:37:24.119 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_auth_entry": "auth", 2020-06-12T02:37:24.119 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_auth_url": "", 2020-06-12T02:37:24.119 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_custom_header": "", 2020-06-12T02:37:24.119 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_enforce_content_length": "false", 2020-06-12T02:37:24.120 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_need_stats": "true", 2020-06-12T02:37:24.120 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_tenant_name": "", 2020-06-12T02:37:24.120 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_token_expiration": "86400", 2020-06-12T02:37:24.120 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_url": "", 2020-06-12T02:37:24.120 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_url_prefix": "swift", 2020-06-12T02:37:24.120 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_versioning_enabled": "false", 2020-06-12T02:37:24.121 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2020-06-12T02:37:24.121 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_lease_period": "120", 2020-06-12T02:37:24.121 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2020-06-12T02:37:24.121 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_log_trim_interval": "1200", 2020-06-12T02:37:24.121 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_log_trim_max_buckets": "16", 2020-06-12T02:37:24.122 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2020-06-12T02:37:24.122 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2020-06-12T02:37:24.122 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_trace_history_size": "4096", 2020-06-12T02:37:24.122 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_trace_per_node_log_size": "32", 2020-06-12T02:37:24.122 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2020-06-12T02:37:24.122 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_thread_pool_size": "512", 2020-06-12T02:37:24.123 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_torrent_comment": "", 2020-06-12T02:37:24.123 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_torrent_createby": "", 2020-06-12T02:37:24.123 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_torrent_encoding": "", 2020-06-12T02:37:24.123 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_torrent_flag": "false", 2020-06-12T02:37:24.123 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_torrent_origin": "", 2020-06-12T02:37:24.123 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_torrent_sha_unit": "524288", 2020-06-12T02:37:24.124 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_torrent_tracker": "", 2020-06-12T02:37:24.124 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_trust_forwarded_https": "false", 2020-06-12T02:37:24.124 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_usage_log_flush_threshold": "1024", 2020-06-12T02:37:24.124 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_usage_log_tick_interval": "30", 2020-06-12T02:37:24.124 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_usage_max_shards": "32", 2020-06-12T02:37:24.125 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_usage_max_user_shards": "1", 2020-06-12T02:37:24.125 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_use_opa_authz": "false", 2020-06-12T02:37:24.125 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_user_default_quota_max_objects": "-1", 2020-06-12T02:37:24.125 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_user_default_quota_max_size": "-1", 2020-06-12T02:37:24.125 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_user_max_buckets": "1000", 2020-06-12T02:37:24.125 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2020-06-12T02:37:24.126 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_user_quota_sync_idle_users": "false", 2020-06-12T02:37:24.126 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_user_quota_sync_interval": "86400", 2020-06-12T02:37:24.126 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_user_quota_sync_wait_time": "86400", 2020-06-12T02:37:24.126 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_user_unique_email": "true", 2020-06-12T02:37:24.126 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_verify_ssl": "true", 2020-06-12T02:37:24.127 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_website_routing_rules_max_num": "50", 2020-06-12T02:37:24.127 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_zone": "", 2020-06-12T02:37:24.127 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_zone_root_pool": ".rgw.root", 2020-06-12T02:37:24.127 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_zonegroup": "", 2020-06-12T02:37:24.127 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2020-06-12T02:37:24.127 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_block_size": "4096", 2020-06-12T02:37:24.128 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_bloom_bits_per_key": "20", 2020-06-12T02:37:24.128 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2020-06-12T02:37:24.128 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "true", 2020-06-12T02:37:24.128 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_cache_row_ratio": "0.000000", 2020-06-12T02:37:24.128 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_cache_shard_bits": "4", 2020-06-12T02:37:24.128 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_cache_size": "536870912", 2020-06-12T02:37:24.129 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_cache_type": "binned_lru", 2020-06-12T02:37:24.129 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_collect_compaction_stats": "false", 2020-06-12T02:37:24.129 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_collect_extended_stats": "false", 2020-06-12T02:37:24.129 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_collect_memory_stats": "false", 2020-06-12T02:37:24.129 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_delete_range_threshold": "1048576", 2020-06-12T02:37:24.130 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_index_type": "binary_search", 2020-06-12T02:37:24.130 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_log_to_ceph_log": "true", 2020-06-12T02:37:24.130 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_metadata_block_size": "4096", 2020-06-12T02:37:24.130 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_partition_filters": "false", 2020-06-12T02:37:24.130 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_perf": "false", 2020-06-12T02:37:24.130 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2020-06-12T02:37:24.131 INFO:teuthology.orchestra.run.smithi107.stdout: "rotating_keys_bootstrap_timeout": "30", 2020-06-12T02:37:24.131 INFO:teuthology.orchestra.run.smithi107.stdout: "rotating_keys_renewal_timeout": "10", 2020-06-12T02:37:24.131 INFO:teuthology.orchestra.run.smithi107.stdout: "run_dir": "/var/run/ceph", 2020-06-12T02:37:24.131 INFO:teuthology.orchestra.run.smithi107.stdout: "setgroup": "", 2020-06-12T02:37:24.131 INFO:teuthology.orchestra.run.smithi107.stdout: "setuser": "", 2020-06-12T02:37:24.132 INFO:teuthology.orchestra.run.smithi107.stdout: "setuser_match_path": "", 2020-06-12T02:37:24.132 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_misplaced_ratio": "0.050000", 2020-06-12T02:37:24.132 INFO:teuthology.orchestra.run.smithi107.stdout: "thp": "false", 2020-06-12T02:37:24.132 INFO:teuthology.orchestra.run.smithi107.stdout: "threadpool_default_timeout": "60", 2020-06-12T02:37:24.132 INFO:teuthology.orchestra.run.smithi107.stdout: "threadpool_empty_queue_max_wait": "2", 2020-06-12T02:37:24.132 INFO:teuthology.orchestra.run.smithi107.stdout: "throttler_perf_counter": "true", 2020-06-12T02:37:24.133 INFO:teuthology.orchestra.run.smithi107.stdout: "timing": "false" 2020-06-12T02:37:24.133 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:37:24.134 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2020-06-12T02:37:24.256 INFO:teuthology.orchestra.run.smithi107.stdout:mon.a: {} 2020-06-12T02:37:24.266 INFO:teuthology.orchestra.run.smithi107.stdout:mon.b: {} 2020-06-12T02:37:24.276 INFO:teuthology.orchestra.run.smithi107.stdout:mon.c: {} 2020-06-12T02:37:24.277 INFO:teuthology.orchestra.run.smithi107.stderr:mon.a: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2020-06-12T02:37:24.277 INFO:teuthology.orchestra.run.smithi107.stderr:mon.b: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2020-06-12T02:37:24.277 INFO:teuthology.orchestra.run.smithi107.stderr:mon.c: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2020-06-12T02:37:24.286 INFO:teuthology.orchestra.run.smithi107:> type ceph-objectstore-tool 2020-06-12T02:37:24.300 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2020-06-12T02:37:24.301 INFO:teuthology.orchestra.run.smithi188:> type ceph-objectstore-tool 2020-06-12T02:37:24.319 INFO:teuthology.orchestra.run.smithi188.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2020-06-12T02:37:24.319 INFO:teuthology.run_tasks:Running task rados... 2020-06-12T02:37:24.331 INFO:tasks.rados:Beginning rados... 2020-06-12T02:37:24.331 DEBUG:teuthology.run_tasks:Unwinding manager rados 2020-06-12T02:37:24.343 INFO:tasks.rados:joining rados 2020-06-12T02:37:24.343 INFO:tasks.thrashosds.thrasher:starting do_thrash 2020-06-12T02:37:24.344 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2020-06-12T02:37:24.344 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2020-06-12T02:37:24.344 INFO:tasks.thrashosds.thrasher:Killing osd 7, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2020-06-12T02:37:24.345 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2020-06-12T02:37:24.345 INFO:teuthology.orchestra.run:waiting for 300 2020-06-12T02:37:24.346 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2020-06-12T02:37:24.346 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2020-06-12T02:37:24.347 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-06-12T02:37:24.347 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2020-06-12T02:37:24.348 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-06-12T02:37:24.348 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2020-06-12T02:37:24.349 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd set noscrub 2020-06-12T02:37:24.349 INFO:tasks.rados:clients are ['client.0', 'client.1'] 2020-06-12T02:37:24.350 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd erasure-code-profile set teuthologyprofile k=2 m=1 crush-failure-domain=osd 2020-06-12T02:37:24.361 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:24.358+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:24.386 INFO:tasks.ceph.osd.7:Stopped 2020-06-12T02:37:24.387 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.7 2020-06-12T02:37:24.387 INFO:tasks.thrashosds.thrasher:remote for osd 7 is ubuntu@smithi188.front.sepia.ceph.com 2020-06-12T02:37:24.388 INFO:teuthology.orchestra.run.smithi188:> type ceph-objectstore-tool 2020-06-12T02:37:24.442 INFO:teuthology.orchestra.run.smithi188.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2020-06-12T02:37:24.443 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-objectstore-tool --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op list-pgs 2020-06-12T02:37:24.483 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:24.481+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:24.493 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:37:24.494 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:37:24.494 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2020-06-12T02:37:24.494 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:37:24.499 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:37:24.503 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-06-12T02:37:24.540 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:37:24.551 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:24.546+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:37:24.579 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:37:24.621 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:37:24.640 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:24.640 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:24.649 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-06-12T02:37:24.656 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:37:24.657 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:24.653+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:24.679 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:37:24.712 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:37:24.752 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:24.750+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:24.783 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:24.784 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:24.792 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-06-12T02:37:24.876 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:24.873+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:24.889 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2020-06-12T02:37:24.901 INFO:tasks.rados:starting run 0 out of 1 2020-06-12T02:37:24.902 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2020-06-12T02:37:24.902 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd pool create unique_pool_0 16 16 erasure teuthologyprofile 2020-06-12T02:37:24.954 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:24.952+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:24.955 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:24.955 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:24.965 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-06-12T02:37:25.055 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:25.053+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:25.126 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:25.127 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:25.138 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-06-12T02:37:25.156 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:25.154+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:37:25.255 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:25.256 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:25.258 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:25.255+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:25.264 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-06-12T02:37:25.323 INFO:teuthology.orchestra.run.smithi188.stdout:2.5 2020-06-12T02:37:25.323 INFO:teuthology.orchestra.run.smithi188.stdout:2.0 2020-06-12T02:37:25.323 INFO:teuthology.orchestra.run.smithi188.stdout:2.7 2020-06-12T02:37:25.324 INFO:teuthology.orchestra.run.smithi188.stdout:1.0 2020-06-12T02:37:25.359 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:25.356+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:25.424 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:25.424 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:25.433 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-06-12T02:37:25.460 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:37:25.458+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:37:25.562 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:25.560+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:37:25.587 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:25.588 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:25.597 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-06-12T02:37:25.663 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:25.661+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:37:25.714 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:25.715 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:25.723 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-06-12T02:37:25.767 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:25.764+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:25.840 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-objectstore-tool --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op export --pgid 1.0 --file /var/log/ceph/exp.1.0.7 2020-06-12T02:37:25.841 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:25.842 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:25.849 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-06-12T02:37:25.890 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:37:25.887+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:37:25.965 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:25.961+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:26.008 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:26.008 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:26.023 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-06-12T02:37:26.065 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:26.062+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:26.167 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:26.165+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:26.180 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:26.180 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:26.189 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-06-12T02:37:26.267 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:26.265+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:37:26.306 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:26.307 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:26.315 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-06-12T02:37:26.369 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:26.366+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:26.470 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:26.467+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:37:26.472 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:26.472 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:26.481 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-06-12T02:37:26.571 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:26.568+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:26.638 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:26.638 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:26.646 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-06-12T02:37:26.690 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:26.688+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:26.717 INFO:teuthology.orchestra.run.smithi188.stderr:Exporting 1.0 info 1.0( v 15'2 (0'0,15'2] local-lis/les=13/14 n=2 ec=3/3 lis/c=13/13 les/c/f=14/14/0 sis=13) 2020-06-12T02:37:26.718 INFO:teuthology.orchestra.run.smithi188.stderr:Read #1:f0025334:::INTEL_SSDPEDMD400G4_PHFT620400NK400BGN:head# 2020-06-12T02:37:26.718 INFO:teuthology.orchestra.run.smithi188.stderr:Read #1:f4b6f209:::INTEL_SSDPEDMD400G4_PHFT620500MP400BGN:head# 2020-06-12T02:37:26.718 INFO:teuthology.orchestra.run.smithi188.stderr:Export successful 2020-06-12T02:37:26.767 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:26.767 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:26.772 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:37:26.770+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:37:26.775 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-06-12T02:37:26.873 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:37:26.870+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:37:26.897 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'unique_pool_0' created 2020-06-12T02:37:26.900 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd set nodeep-scrub 2020-06-12T02:37:26.911 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2020-06-12T02:37:26.935 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:26.935 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:26.943 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-06-12T02:37:26.994 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:26.991+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:27.060 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:27.061 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:27.069 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-06-12T02:37:27.112 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:27.110+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:27.175 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:27.172+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:27.185 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:27.185 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:27.194 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-06-12T02:37:27.234 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-objectstore-tool --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --force --op remove --pgid 1.0 2020-06-12T02:37:27.276 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:27.273+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:27.376 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:27.374+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:27.390 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:27.390 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:27.399 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-06-12T02:37:27.478 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:27.475+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:37:27.516 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:27.517 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:27.525 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-06-12T02:37:27.578 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:37:27.576+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:37:27.642 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:27.644 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:27.651 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-06-12T02:37:27.694 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:27.691+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:27.767 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:27.767 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:27.775 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-06-12T02:37:27.786 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:27.784+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:27.887 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:27.884+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:27.900 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2020-06-12T02:37:27.911 INFO:teuthology.orchestra.run.smithi107.stderr:enabled application 'rados' on pool 'unique_pool_0' 2020-06-12T02:37:27.946 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:27.946 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:27.954 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-06-12T02:37:27.995 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:37:27.992+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:37:28.089 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:28.086+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:28.103 INFO:teuthology.orchestra.run.smithi188.stdout: marking collection for removal 2020-06-12T02:37:28.103 INFO:teuthology.orchestra.run.smithi188.stdout:setting '_remove' omap key 2020-06-12T02:37:28.103 INFO:teuthology.orchestra.run.smithi188.stdout:finish_remove_pgs 1.0_head removing 1.0 2020-06-12T02:37:28.103 INFO:teuthology.orchestra.run.smithi188.stdout:Remove successful 2020-06-12T02:37:28.112 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:28.113 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:28.124 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-06-12T02:37:28.190 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:28.188+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:28.280 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:28.280 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:28.288 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-06-12T02:37:28.291 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:28.288+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:28.391 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:28.389+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:28.403 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:28.404 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:28.412 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-06-12T02:37:28.492 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:37:28.490+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:37:28.567 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:28.567 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:28.575 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-06-12T02:37:28.619 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.1.0.7 2020-06-12T02:37:28.620 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:37:28.617+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:37:28.691 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:28.692 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:28.694 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:37:28.691+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:37:28.700 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-06-12T02:37:28.795 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:28.793+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:28.856 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:28.856 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:28.865 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-06-12T02:37:28.896 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:28.894+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:28.924 INFO:teuthology.orchestra.run.smithi107:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 400000 --objects 20 --max-in-flight 8 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 0 --op write 0 --op delete 20 --op append 5 --op write_excl 0 --op append_excl 5 --pool unique_pool_0 2020-06-12T02:37:28.947 INFO:tasks.rados.rados.0.smithi107.stdout:adding op weight delete -> 20 2020-06-12T02:37:28.947 INFO:tasks.rados.rados.0.smithi107.stdout:adding op weight append -> 5 2020-06-12T02:37:28.947 INFO:tasks.rados.rados.0.smithi107.stdout:adding op weight append_excl -> 5 2020-06-12T02:37:28.947 INFO:tasks.rados.rados.0.smithi107.stdout:ceph version 15.2.3-196-g91d24f4 (91d24f498556f86f6d1318d2efd5210c35b36762) octopus (stable) 2020-06-12T02:37:28.948 INFO:tasks.rados.rados.0.smithi107.stdout:Configuration: 2020-06-12T02:37:28.948 INFO:tasks.rados.rados.0.smithi107.stdout: Number of operations: 400000 2020-06-12T02:37:28.948 INFO:tasks.rados.rados.0.smithi107.stdout: Number of objects: 20 2020-06-12T02:37:28.948 INFO:tasks.rados.rados.0.smithi107.stdout: Max in flight operations: 8 2020-06-12T02:37:28.948 INFO:tasks.rados.rados.0.smithi107.stdout: Object size (in bytes): 4000000 2020-06-12T02:37:28.949 INFO:tasks.rados.rados.0.smithi107.stdout: Write stride min: 400000 2020-06-12T02:37:28.949 INFO:tasks.rados.rados.0.smithi107.stdout: Write stride max: 800000 2020-06-12T02:37:28.987 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:28.987 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:28.995 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-06-12T02:37:29.037 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:29.034+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:29.098 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:37:29.096+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:37:29.152 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:29.153 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:29.161 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-06-12T02:37:29.200 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:29.198+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:29.300 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:29.297+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:29.317 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:29.317 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:29.325 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-06-12T02:37:29.401 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:29.399+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:29.447 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:29.447 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:29.455 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-06-12T02:37:29.469 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-06-12T02:37:29.470 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-06-12T02:37:29.470 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-06-12T02:37:29.470 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-06-12T02:37:29.470 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-06-12T02:37:29.470 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-06-12T02:37:29.471 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-06-12T02:37:29.471 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.7 2020-06-12T02:37:29.471 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: problem getting command descriptions from osd.7 2020-06-12T02:37:29.472 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-06-12T02:37:29.472 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-06-12T02:37:29.503 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:29.501+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:29.505 INFO:teuthology.orchestra.run.smithi188.stdout:get_pg_num_history pg_num_history pg_num_history(e18 pg_nums {1={3=1},2={16=8}} deleted_pools ) 2020-06-12T02:37:29.505 INFO:teuthology.orchestra.run.smithi188.stdout:Importing pgid 1.0 2020-06-12T02:37:29.506 INFO:teuthology.orchestra.run.smithi188.stdout:Write #1:f0025334:::INTEL_SSDPEDMD400G4_PHFT620400NK400BGN:head# 2020-06-12T02:37:29.506 INFO:teuthology.orchestra.run.smithi188.stdout:snapset 0=[]:{} 2020-06-12T02:37:29.506 INFO:teuthology.orchestra.run.smithi188.stdout:Write #1:f4b6f209:::INTEL_SSDPEDMD400G4_PHFT620500MP400BGN:head# 2020-06-12T02:37:29.507 INFO:teuthology.orchestra.run.smithi188.stdout:snapset 0=[]:{} 2020-06-12T02:37:29.509 INFO:teuthology.orchestra.run.smithi188.stdout:write_pg epoch 15 info 1.0( v 15'2 (0'0,15'2] local-lis/les=13/14 n=2 ec=3/3 lis/c=13/13 les/c/f=14/14/0 sis=13) 2020-06-12T02:37:29.511 INFO:teuthology.orchestra.run.smithi188.stdout:Import successful 2020-06-12T02:37:29.574 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:29.574 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:29.583 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-06-12T02:37:29.624 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:29.622+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:37:29.707 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:29.705+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:29.740 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:29.741 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:29.749 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-06-12T02:37:29.834 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:29.832+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:29.905 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:29.906 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:29.912 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:29.909+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:29.913 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-06-12T02:37:29.916 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd unset noscrub 2020-06-12T02:37:29.924 INFO:tasks.rados.rados.0.smithi107.stdout:1: write initial oid 1 2020-06-12T02:37:29.924 INFO:tasks.rados.rados.0.smithi107.stdout:1: seq_num 0 ranges {0=24576} 2020-06-12T02:37:29.925 INFO:tasks.rados.rados.0.smithi107.stdout:1: writing smithi10729407-1 from 0 to 24576 tid 1 2020-06-12T02:37:29.925 INFO:tasks.rados.rados.0.smithi107.stdout:2: write initial oid 2 2020-06-12T02:37:29.925 INFO:tasks.rados.rados.0.smithi107.stdout:2: seq_num 1 ranges {0=24576} 2020-06-12T02:37:29.925 INFO:tasks.rados.rados.0.smithi107.stdout:2: writing smithi10729407-2 from 0 to 24576 tid 1 2020-06-12T02:37:29.925 INFO:tasks.rados.rados.0.smithi107.stdout:3: write initial oid 3 2020-06-12T02:37:29.926 INFO:tasks.rados.rados.0.smithi107.stdout:3: seq_num 2 ranges {0=40960} 2020-06-12T02:37:29.927 INFO:tasks.rados.rados.0.smithi107.stdout:3: writing smithi10729407-3 from 0 to 40960 tid 1 2020-06-12T02:37:29.927 INFO:tasks.rados.rados.0.smithi107.stdout:4: write initial oid 4 2020-06-12T02:37:29.927 INFO:tasks.rados.rados.0.smithi107.stdout:4: seq_num 3 ranges {0=57344} 2020-06-12T02:37:29.928 INFO:tasks.rados.rados.0.smithi107.stdout:4: writing smithi10729407-4 from 0 to 57344 tid 1 2020-06-12T02:37:29.928 INFO:tasks.rados.rados.0.smithi107.stdout:5: write initial oid 5 2020-06-12T02:37:29.928 INFO:tasks.rados.rados.0.smithi107.stdout:5: seq_num 4 ranges {0=73728} 2020-06-12T02:37:29.928 INFO:tasks.rados.rados.0.smithi107.stdout:5: writing smithi10729407-5 from 0 to 73728 tid 1 2020-06-12T02:37:29.928 INFO:tasks.rados.rados.0.smithi107.stdout:6: write initial oid 6 2020-06-12T02:37:29.928 INFO:tasks.rados.rados.0.smithi107.stdout:6: seq_num 5 ranges {0=90112} 2020-06-12T02:37:29.929 INFO:tasks.rados.rados.0.smithi107.stdout:6: writing smithi10729407-6 from 0 to 90112 tid 1 2020-06-12T02:37:29.929 INFO:tasks.rados.rados.0.smithi107.stdout:7: write initial oid 7 2020-06-12T02:37:29.929 INFO:tasks.rados.rados.0.smithi107.stdout:7: seq_num 6 ranges {0=106496} 2020-06-12T02:37:29.930 INFO:tasks.rados.rados.0.smithi107.stdout:7: writing smithi10729407-7 from 0 to 106496 tid 1 2020-06-12T02:37:29.930 INFO:tasks.rados.rados.0.smithi107.stdout:8: write initial oid 8 2020-06-12T02:37:29.931 INFO:tasks.rados.rados.0.smithi107.stdout:8: seq_num 7 ranges {0=122880} 2020-06-12T02:37:29.931 INFO:tasks.rados.rados.0.smithi107.stdout:8: writing smithi10729407-8 from 0 to 122880 tid 1 2020-06-12T02:37:29.932 INFO:tasks.rados.rados.0.smithi107.stdout: waiting on 8 2020-06-12T02:37:29.932 INFO:tasks.rados.rados.0.smithi107.stdout:2: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:29.932 INFO:tasks.rados.rados.0.smithi107.stdout:4: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:29.932 INFO:tasks.rados.rados.0.smithi107.stdout:1: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:29.935 INFO:tasks.rados.rados.0.smithi107.stdout:2: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:29.936 INFO:tasks.rados.rados.0.smithi107.stdout:6: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:29.936 INFO:tasks.rados.rados.0.smithi107.stdout:7: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:29.936 INFO:tasks.rados.rados.0.smithi107.stdout:4: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:29.937 INFO:tasks.rados.rados.0.smithi107.stdout:5: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:29.937 INFO:tasks.rados.rados.0.smithi107.stdout:3: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:29.938 INFO:tasks.rados.rados.0.smithi107.stdout:4: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:29.938 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 2 (ObjNum 3 snap 0 seq_num 3) dirty exists 2020-06-12T02:37:29.939 INFO:tasks.rados.rados.0.smithi107.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2020-06-12T02:37:29.939 INFO:tasks.rados.rados.0.smithi107.stdout:2: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:29.939 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 2 (ObjNum 1 snap 0 seq_num 1) dirty exists 2020-06-12T02:37:29.939 INFO:tasks.rados.rados.0.smithi107.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2020-06-12T02:37:29.940 INFO:tasks.rados.rados.0.smithi107.stdout:2: done (7 left) 2020-06-12T02:37:29.940 INFO:tasks.rados.rados.0.smithi107.stdout:4: done (6 left) 2020-06-12T02:37:29.940 INFO:tasks.rados.rados.0.smithi107.stdout:9: write initial oid 9 2020-06-12T02:37:29.941 INFO:tasks.rados.rados.0.smithi107.stdout:9: seq_num 8 ranges {0=139264} 2020-06-12T02:37:29.941 INFO:tasks.rados.rados.0.smithi107.stdout:9: writing smithi10729407-9 from 0 to 139264 tid 1 2020-06-12T02:37:29.941 INFO:tasks.rados.rados.0.smithi107.stdout:10: write initial oid 10 2020-06-12T02:37:29.941 INFO:tasks.rados.rados.0.smithi107.stdout:10: seq_num 9 ranges {0=155648} 2020-06-12T02:37:29.941 INFO:tasks.rados.rados.0.smithi107.stdout:10: writing smithi10729407-10 from 0 to 155648 tid 1 2020-06-12T02:37:29.941 INFO:tasks.rados.rados.0.smithi107.stdout: waiting on 8 2020-06-12T02:37:29.942 INFO:tasks.rados.rados.0.smithi107.stdout:1: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:29.943 INFO:tasks.rados.rados.0.smithi107.stdout:1: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:29.943 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2020-06-12T02:37:29.943 INFO:tasks.rados.rados.0.smithi107.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2020-06-12T02:37:29.943 INFO:tasks.rados.rados.0.smithi107.stdout:1: done (7 left) 2020-06-12T02:37:29.944 INFO:tasks.rados.rados.0.smithi107.stdout:11: write initial oid 11 2020-06-12T02:37:29.944 INFO:tasks.rados.rados.0.smithi107.stdout:11: seq_num 10 ranges {0=172032} 2020-06-12T02:37:29.944 INFO:tasks.rados.rados.0.smithi107.stdout:11: writing smithi10729407-11 from 0 to 172032 tid 1 2020-06-12T02:37:29.944 INFO:tasks.rados.rados.0.smithi107.stdout: waiting on 8 2020-06-12T02:37:29.945 INFO:tasks.rados.rados.0.smithi107.stdout:5: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:29.945 INFO:tasks.rados.rados.0.smithi107.stdout:7: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:29.945 INFO:tasks.rados.rados.0.smithi107.stdout:6: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:29.946 INFO:tasks.rados.rados.0.smithi107.stdout:3: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:29.947 INFO:tasks.rados.rados.0.smithi107.stdout:8: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:29.947 INFO:tasks.rados.rados.0.smithi107.stdout:7: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:29.947 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 2 (ObjNum 6 snap 0 seq_num 6) dirty exists 2020-06-12T02:37:29.947 INFO:tasks.rados.rados.0.smithi107.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2020-06-12T02:37:29.947 INFO:tasks.rados.rados.0.smithi107.stdout:7: done (7 left) 2020-06-12T02:37:29.947 INFO:tasks.rados.rados.0.smithi107.stdout:12: write initial oid 12 2020-06-12T02:37:29.948 INFO:tasks.rados.rados.0.smithi107.stdout:12: seq_num 11 ranges {0=188416} 2020-06-12T02:37:29.948 INFO:tasks.rados.rados.0.smithi107.stdout:12: writing smithi10729407-12 from 0 to 188416 tid 1 2020-06-12T02:37:29.948 INFO:tasks.rados.rados.0.smithi107.stdout: waiting on 8 2020-06-12T02:37:29.948 INFO:tasks.rados.rados.0.smithi107.stdout:5: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:29.948 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 2 (ObjNum 4 snap 0 seq_num 4) dirty exists 2020-06-12T02:37:29.949 INFO:tasks.rados.rados.0.smithi107.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2020-06-12T02:37:29.949 INFO:tasks.rados.rados.0.smithi107.stdout:6: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:29.949 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 4 (ObjNum 5 snap 0 seq_num 5) dirty exists 2020-06-12T02:37:29.949 INFO:tasks.rados.rados.0.smithi107.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2020-06-12T02:37:29.949 INFO:tasks.rados.rados.0.smithi107.stdout:3: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:29.949 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 2 (ObjNum 2 snap 0 seq_num 2) dirty exists 2020-06-12T02:37:29.950 INFO:tasks.rados.rados.0.smithi107.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2020-06-12T02:37:29.950 INFO:tasks.rados.rados.0.smithi107.stdout:3: done (7 left) 2020-06-12T02:37:29.951 INFO:tasks.rados.rados.0.smithi107.stdout:5: done (6 left) 2020-06-12T02:37:29.951 INFO:tasks.rados.rados.0.smithi107.stdout:6: done (5 left) 2020-06-12T02:37:29.951 INFO:tasks.rados.rados.0.smithi107.stdout:13: write initial oid 13 2020-06-12T02:37:29.951 INFO:tasks.rados.rados.0.smithi107.stdout:13: seq_num 12 ranges {0=204800} 2020-06-12T02:37:29.952 INFO:tasks.rados.rados.0.smithi107.stdout:13: writing smithi10729407-13 from 0 to 204800 tid 1 2020-06-12T02:37:29.952 INFO:tasks.rados.rados.0.smithi107.stdout:14: write initial oid 14 2020-06-12T02:37:29.953 INFO:tasks.rados.rados.0.smithi107.stdout:14: seq_num 13 ranges {0=221184} 2020-06-12T02:37:29.953 INFO:tasks.rados.rados.0.smithi107.stdout:14: writing smithi10729407-14 from 0 to 221184 tid 1 2020-06-12T02:37:29.953 INFO:tasks.rados.rados.0.smithi107.stdout:9: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:29.953 INFO:tasks.rados.rados.0.smithi107.stdout:11: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:29.953 INFO:tasks.rados.rados.0.smithi107.stdout:9: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:29.954 INFO:tasks.rados.rados.0.smithi107.stdout:15: write initial oid 15 2020-06-12T02:37:29.954 INFO:tasks.rados.rados.0.smithi107.stdout:15: seq_num 14 ranges {0=237568} 2020-06-12T02:37:29.954 INFO:tasks.rados.rados.0.smithi107.stdout:15: writing smithi10729407-15 from 0 to 237568 tid 1 2020-06-12T02:37:29.954 INFO:tasks.rados.rados.0.smithi107.stdout:10: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:29.954 INFO:tasks.rados.rados.0.smithi107.stdout:8: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:29.955 INFO:tasks.rados.rados.0.smithi107.stdout: waiting on 8 2020-06-12T02:37:29.956 INFO:tasks.rados.rados.0.smithi107.stdout:8: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:29.956 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 2 (ObjNum 7 snap 0 seq_num 7) dirty exists 2020-06-12T02:37:29.956 INFO:tasks.rados.rados.0.smithi107.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2020-06-12T02:37:29.957 INFO:tasks.rados.rados.0.smithi107.stdout:8: done (7 left) 2020-06-12T02:37:29.957 INFO:tasks.rados.rados.0.smithi107.stdout:16: write initial oid 16 2020-06-12T02:37:29.957 INFO:tasks.rados.rados.0.smithi107.stdout:16: seq_num 15 ranges {0=253952} 2020-06-12T02:37:29.957 INFO:tasks.rados.rados.0.smithi107.stdout:16: writing smithi10729407-16 from 0 to 253952 tid 1 2020-06-12T02:37:29.957 INFO:tasks.rados.rados.0.smithi107.stdout:9: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:29.958 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 2 (ObjNum 8 snap 0 seq_num 8) dirty exists 2020-06-12T02:37:29.958 INFO:tasks.rados.rados.0.smithi107.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2020-06-12T02:37:29.959 INFO:tasks.rados.rados.0.smithi107.stdout:9: done (7 left) 2020-06-12T02:37:29.959 INFO:tasks.rados.rados.0.smithi107.stdout:17: write initial oid 17 2020-06-12T02:37:29.959 INFO:tasks.rados.rados.0.smithi107.stdout:17: seq_num 16 ranges {0=270336} 2020-06-12T02:37:29.961 INFO:tasks.rados.rados.0.smithi107.stdout:17: writing smithi10729407-17 from 0 to 270336 tid 1 2020-06-12T02:37:29.961 INFO:tasks.rados.rados.0.smithi107.stdout: waiting on 8 2020-06-12T02:37:29.962 INFO:tasks.rados.rados.0.smithi107.stdout:12: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:29.962 INFO:tasks.rados.rados.0.smithi107.stdout:11: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:29.962 INFO:tasks.rados.rados.0.smithi107.stdout:13: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:29.962 INFO:tasks.rados.rados.0.smithi107.stdout:10: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:29.962 INFO:tasks.rados.rados.0.smithi107.stdout:10: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:29.962 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 2 (ObjNum 9 snap 0 seq_num 9) dirty exists 2020-06-12T02:37:29.963 INFO:tasks.rados.rados.0.smithi107.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2020-06-12T02:37:29.963 INFO:tasks.rados.rados.0.smithi107.stdout:10: done (7 left) 2020-06-12T02:37:29.963 INFO:tasks.rados.rados.0.smithi107.stdout:18: write initial oid 18 2020-06-12T02:37:29.963 INFO:tasks.rados.rados.0.smithi107.stdout:18: seq_num 17 ranges {0=286720} 2020-06-12T02:37:29.963 INFO:tasks.rados.rados.0.smithi107.stdout:18: writing smithi10729407-18 from 0 to 286720 tid 1 2020-06-12T02:37:29.964 INFO:tasks.rados.rados.0.smithi107.stdout: waiting on 8 2020-06-12T02:37:29.964 INFO:tasks.rados.rados.0.smithi107.stdout:15: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:29.964 INFO:tasks.rados.rados.0.smithi107.stdout:12: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:29.964 INFO:tasks.rados.rados.0.smithi107.stdout:14: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:29.964 INFO:tasks.rados.rados.0.smithi107.stdout:11: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:29.964 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 2 (ObjNum 10 snap 0 seq_num 10) dirty exists 2020-06-12T02:37:29.965 INFO:tasks.rados.rados.0.smithi107.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2020-06-12T02:37:29.965 INFO:tasks.rados.rados.0.smithi107.stdout:11: done (7 left) 2020-06-12T02:37:29.965 INFO:tasks.rados.rados.0.smithi107.stdout:19: write initial oid 19 2020-06-12T02:37:29.965 INFO:tasks.rados.rados.0.smithi107.stdout:19: seq_num 18 ranges {0=303104} 2020-06-12T02:37:29.968 INFO:tasks.rados.rados.0.smithi107.stdout:19: writing smithi10729407-19 from 0 to 303104 tid 1 2020-06-12T02:37:29.969 INFO:tasks.rados.rados.0.smithi107.stdout: waiting on 8 2020-06-12T02:37:29.969 INFO:tasks.rados.rados.0.smithi107.stdout:13: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:29.969 INFO:tasks.rados.rados.0.smithi107.stdout:12: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:29.969 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 2 (ObjNum 11 snap 0 seq_num 11) dirty exists 2020-06-12T02:37:29.970 INFO:tasks.rados.rados.0.smithi107.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2020-06-12T02:37:29.970 INFO:tasks.rados.rados.0.smithi107.stdout:14: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:29.970 INFO:tasks.rados.rados.0.smithi107.stdout:12: done (7 left) 2020-06-12T02:37:29.970 INFO:tasks.rados.rados.0.smithi107.stdout:20: write initial oid 20 2020-06-12T02:37:29.970 INFO:tasks.rados.rados.0.smithi107.stdout:20: seq_num 19 ranges {0=319488} 2020-06-12T02:37:29.971 INFO:tasks.rados.rados.0.smithi107.stdout:20: writing smithi10729407-20 from 0 to 319488 tid 1 2020-06-12T02:37:29.971 INFO:tasks.rados.rados.0.smithi107.stdout: waiting on 8 2020-06-12T02:37:29.971 INFO:tasks.rados.rados.0.smithi107.stdout:17: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:29.971 INFO:tasks.rados.rados.0.smithi107.stdout:13: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:29.971 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 4 (ObjNum 12 snap 0 seq_num 12) dirty exists 2020-06-12T02:37:29.971 INFO:tasks.rados.rados.0.smithi107.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2020-06-12T02:37:29.972 INFO:tasks.rados.rados.0.smithi107.stdout:13: done (7 left) 2020-06-12T02:37:29.972 INFO:tasks.rados.rados.0.smithi107.stdout:21: delete oid 6 current snap is 0 2020-06-12T02:37:29.972 INFO:tasks.rados.rados.0.smithi107.stdout:14: finishing write tid 4 to smithi10729407-14 2020-06-12T02:37:29.972 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 4 (ObjNum 13 snap 0 seq_num 13) dirty exists 2020-06-12T02:37:29.972 INFO:tasks.rados.rados.0.smithi107.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2020-06-12T02:37:29.973 INFO:tasks.rados.rados.0.smithi107.stdout:15: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:29.973 INFO:tasks.rados.rados.0.smithi107.stdout:16: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:29.973 INFO:tasks.rados.rados.0.smithi107.stdout:15: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:29.973 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 4 (ObjNum 14 snap 0 seq_num 14) dirty exists 2020-06-12T02:37:29.973 INFO:tasks.rados.rados.0.smithi107.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2020-06-12T02:37:29.974 INFO:tasks.rados.rados.0.smithi107.stdout:17: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:29.976 INFO:tasks.rados.rados.0.smithi107.stdout:18: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:29.976 INFO:tasks.rados.rados.0.smithi107.stdout:14: done (7 left) 2020-06-12T02:37:29.976 INFO:tasks.rados.rados.0.smithi107.stdout:15: done (6 left) 2020-06-12T02:37:29.976 INFO:tasks.rados.rados.0.smithi107.stdout:21: done (5 left) 2020-06-12T02:37:29.977 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:37:29.977 INFO:tasks.rados.rados.0.smithi107.stdout:19: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:29.977 INFO:tasks.rados.rados.0.smithi107.stdout:22: seq_num 21 ranges {221184=360448} 2020-06-12T02:37:29.977 INFO:tasks.rados.rados.0.smithi107.stdout:22: writing smithi10729407-14 from 221184 to 581632 tid 1 2020-06-12T02:37:29.977 INFO:tasks.rados.rados.0.smithi107.stdout:23: delete oid 1 current snap is 0 2020-06-12T02:37:29.978 INFO:tasks.rados.rados.0.smithi107.stdout:17: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:29.978 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 4 (ObjNum 16 snap 0 seq_num 16) dirty exists 2020-06-12T02:37:29.978 INFO:tasks.rados.rados.0.smithi107.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2020-06-12T02:37:29.978 INFO:tasks.rados.rados.0.smithi107.stdout:16: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:29.978 INFO:tasks.rados.rados.0.smithi107.stdout:18: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:29.979 INFO:tasks.rados.rados.0.smithi107.stdout:16: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:29.979 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 4 (ObjNum 15 snap 0 seq_num 15) dirty exists 2020-06-12T02:37:29.979 INFO:tasks.rados.rados.0.smithi107.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2020-06-12T02:37:29.979 INFO:tasks.rados.rados.0.smithi107.stdout:20: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:29.979 INFO:tasks.rados.rados.0.smithi107.stdout:18: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:29.979 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 4 (ObjNum 17 snap 0 seq_num 17) dirty exists 2020-06-12T02:37:29.980 INFO:tasks.rados.rados.0.smithi107.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2020-06-12T02:37:29.980 INFO:tasks.rados.rados.0.smithi107.stdout:19: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:29.981 INFO:tasks.rados.rados.0.smithi107.stdout:19: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:29.981 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 4 (ObjNum 18 snap 0 seq_num 18) dirty exists 2020-06-12T02:37:29.981 INFO:tasks.rados.rados.0.smithi107.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2020-06-12T02:37:29.981 INFO:tasks.rados.rados.0.smithi107.stdout:20: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:29.981 INFO:tasks.rados.rados.0.smithi107.stdout:20: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:29.982 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 2 (ObjNum 19 snap 0 seq_num 19) dirty exists 2020-06-12T02:37:29.982 INFO:tasks.rados.rados.0.smithi107.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2020-06-12T02:37:29.982 INFO:tasks.rados.rados.0.smithi107.stdout:16: done (6 left) 2020-06-12T02:37:29.983 INFO:tasks.rados.rados.0.smithi107.stdout:17: done (5 left) 2020-06-12T02:37:29.983 INFO:tasks.rados.rados.0.smithi107.stdout:18: done (4 left) 2020-06-12T02:37:29.983 INFO:tasks.rados.rados.0.smithi107.stdout:19: done (3 left) 2020-06-12T02:37:29.983 INFO:tasks.rados.rados.0.smithi107.stdout:20: done (2 left) 2020-06-12T02:37:29.983 INFO:tasks.rados.rados.0.smithi107.stdout:23: done (1 left) 2020-06-12T02:37:29.984 INFO:tasks.rados.rados.0.smithi107.stdout:24: delete oid 5 current snap is 0 2020-06-12T02:37:29.984 INFO:tasks.rados.rados.0.smithi107.stdout:22: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:29.984 INFO:tasks.rados.rados.0.smithi107.stdout:22: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:29.985 INFO:tasks.rados.rados.0.smithi107.stdout:24: done (1 left) 2020-06-12T02:37:29.985 INFO:tasks.rados.rados.0.smithi107.stdout:25: delete oid 10 current snap is 0 2020-06-12T02:37:29.986 INFO:tasks.rados.rados.0.smithi107.stdout:22: finishing write tid 4 to smithi10729407-14 2020-06-12T02:37:29.986 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 6 (ObjNum 21 snap 0 seq_num 21) dirty exists 2020-06-12T02:37:29.986 INFO:tasks.rados.rados.0.smithi107.stdout:22: left oid 14 (ObjNum 21 snap 0 seq_num 21) 2020-06-12T02:37:29.988 INFO:tasks.rados.rados.0.smithi107.stdout:22: done (1 left) 2020-06-12T02:37:29.988 INFO:tasks.rados.rados.0.smithi107.stdout:25: done (0 left) 2020-06-12T02:37:29.988 INFO:tasks.rados.rados.0.smithi107.stdout:26: delete oid 4 current snap is 0 2020-06-12T02:37:29.989 INFO:tasks.rados.rados.0.smithi107.stdout:26: done (0 left) 2020-06-12T02:37:29.990 INFO:tasks.rados.rados.0.smithi107.stdout:27: delete oid 4 current snap is 0 2020-06-12T02:37:29.994 INFO:tasks.rados.rados.0.smithi107.stdout:27: done (0 left) 2020-06-12T02:37:29.995 INFO:tasks.rados.rados.0.smithi107.stdout:28: delete oid 12 current snap is 0 2020-06-12T02:37:29.996 INFO:tasks.rados.rados.0.smithi107.stdout:28: done (0 left) 2020-06-12T02:37:29.997 INFO:tasks.rados.rados.0.smithi107.stdout:29: delete oid 10 current snap is 0 2020-06-12T02:37:30.002 INFO:tasks.rados.rados.0.smithi107.stdout:29: done (0 left) 2020-06-12T02:37:30.002 INFO:tasks.rados.rados.0.smithi107.stdout:30: delete oid 12 current snap is 0 2020-06-12T02:37:30.005 INFO:tasks.rados.rados.0.smithi107.stdout:30: done (0 left) 2020-06-12T02:37:30.005 INFO:tasks.rados.rados.0.smithi107.stdout:31: delete oid 13 current snap is 0 2020-06-12T02:37:30.009 INFO:tasks.rados.rados.0.smithi107.stdout:31: done (0 left) 2020-06-12T02:37:30.009 INFO:tasks.rados.rados.0.smithi107.stdout:32: delete oid 2 current snap is 0 2020-06-12T02:37:30.012 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:30.010+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:30.013 INFO:tasks.rados.rados.0.smithi107.stdout:32: done (0 left) 2020-06-12T02:37:30.013 INFO:tasks.rados.rados.0.smithi107.stdout:33: delete oid 1 current snap is 0 2020-06-12T02:37:30.015 INFO:tasks.rados.rados.0.smithi107.stdout:33: done (0 left) 2020-06-12T02:37:30.016 INFO:tasks.rados.rados.0.smithi107.stdout:34: delete oid 16 current snap is 0 2020-06-12T02:37:30.018 INFO:tasks.rados.rados.0.smithi107.stdout:34: done (0 left) 2020-06-12T02:37:30.018 INFO:tasks.rados.rados.0.smithi107.stdout:35: delete oid 19 current snap is 0 2020-06-12T02:37:30.022 INFO:teuthology.orchestra.run.smithi188:> sudo rm -f /var/log/ceph/exp.1.0.7 2020-06-12T02:37:30.023 INFO:tasks.rados.rados.0.smithi107.stdout:35: done (0 left) 2020-06-12T02:37:30.023 INFO:tasks.rados.rados.0.smithi107.stdout:36: delete oid 10 current snap is 0 2020-06-12T02:37:30.027 INFO:tasks.rados.rados.0.smithi107.stdout:36: done (0 left) 2020-06-12T02:37:30.027 INFO:tasks.rados.rados.0.smithi107.stdout:37: delete oid 6 current snap is 0 2020-06-12T02:37:30.028 INFO:tasks.rados.rados.0.smithi107.stdout:37: done (0 left) 2020-06-12T02:37:30.028 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:37:30.029 INFO:tasks.rados.rados.0.smithi107.stdout:38: seq_num 37 ranges {0=622592} 2020-06-12T02:37:30.032 INFO:tasks.rados.rados.0.smithi107.stdout:38: writing smithi10729407-10 from 0 to 622592 tid 1 2020-06-12T02:37:30.032 INFO:tasks.rados.rados.0.smithi107.stdout:39: delete oid 13 current snap is 0 2020-06-12T02:37:30.034 INFO:tasks.rados.rados.0.smithi107.stdout:39: done (1 left) 2020-06-12T02:37:30.034 INFO:tasks.rados.rados.0.smithi107.stdout:40: delete oid 1 current snap is 0 2020-06-12T02:37:30.035 INFO:tasks.rados.rados.0.smithi107.stdout:40: done (1 left) 2020-06-12T02:37:30.036 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:37:30.036 INFO:tasks.rados.rados.0.smithi107.stdout:41: seq_num 40 ranges {0=679936} 2020-06-12T02:37:30.040 INFO:tasks.rados.rados.0.smithi107.stdout:41: writing smithi10729407-19 from 0 to 679936 tid 1 2020-06-12T02:37:30.040 INFO:tasks.rados.rados.0.smithi107.stdout:42: delete oid 4 current snap is 0 2020-06-12T02:37:30.040 INFO:tasks.rados.rados.0.smithi107.stdout:38: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:30.040 INFO:tasks.rados.rados.0.smithi107.stdout:38: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:30.041 INFO:tasks.rados.rados.0.smithi107.stdout:38: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:30.041 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 7 (ObjNum 37 snap 0 seq_num 37) dirty exists 2020-06-12T02:37:30.041 INFO:tasks.rados.rados.0.smithi107.stdout:38: left oid 10 (ObjNum 37 snap 0 seq_num 37) 2020-06-12T02:37:30.045 INFO:tasks.rados.rados.0.smithi107.stdout:38: done (2 left) 2020-06-12T02:37:30.045 INFO:tasks.rados.rados.0.smithi107.stdout:42: done (1 left) 2020-06-12T02:37:30.046 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:37:30.046 INFO:tasks.rados.rados.0.smithi107.stdout:43: seq_num 42 ranges {122880=712704} 2020-06-12T02:37:30.050 INFO:tasks.rados.rados.0.smithi107.stdout:43: writing smithi10729407-8 from 122880 to 835584 tid 1 2020-06-12T02:37:30.050 INFO:tasks.rados.rados.0.smithi107.stdout:44: delete oid 14 current snap is 0 2020-06-12T02:37:30.051 INFO:tasks.rados.rados.0.smithi107.stdout:41: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:30.051 INFO:tasks.rados.rados.0.smithi107.stdout:41: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:30.051 INFO:tasks.rados.rados.0.smithi107.stdout:41: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:30.051 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 9 (ObjNum 40 snap 0 seq_num 40) dirty exists 2020-06-12T02:37:30.051 INFO:tasks.rados.rados.0.smithi107.stdout:41: left oid 19 (ObjNum 40 snap 0 seq_num 40) 2020-06-12T02:37:30.052 INFO:tasks.rados.rados.0.smithi107.stdout:41: done (2 left) 2020-06-12T02:37:30.052 INFO:tasks.rados.rados.0.smithi107.stdout:44: done (1 left) 2020-06-12T02:37:30.053 INFO:tasks.rados.rados.0.smithi107.stdout:45: delete oid 3 current snap is 0 2020-06-12T02:37:30.054 INFO:tasks.rados.rados.0.smithi107.stdout:45: done (1 left) 2020-06-12T02:37:30.055 INFO:tasks.rados.rados.0.smithi107.stdout:46: delete oid 10 current snap is 0 2020-06-12T02:37:30.059 INFO:tasks.rados.rados.0.smithi107.stdout:46: done (1 left) 2020-06-12T02:37:30.059 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:37:30.060 INFO:tasks.rados.rados.0.smithi107.stdout:47: seq_num 46 ranges {106496=778240} 2020-06-12T02:37:30.065 INFO:tasks.rados.rados.0.smithi107.stdout:47: writing smithi10729407-7 from 106496 to 884736 tid 1 2020-06-12T02:37:30.065 INFO:tasks.rados.rados.0.smithi107.stdout:48: delete oid 4 current snap is 0 2020-06-12T02:37:30.065 INFO:tasks.rados.rados.0.smithi107.stdout:43: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:30.066 INFO:tasks.rados.rados.0.smithi107.stdout:43: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:30.066 INFO:tasks.rados.rados.0.smithi107.stdout:43: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:30.066 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 6 (ObjNum 42 snap 0 seq_num 42) dirty exists 2020-06-12T02:37:30.067 INFO:tasks.rados.rados.0.smithi107.stdout:43: left oid 8 (ObjNum 42 snap 0 seq_num 42) 2020-06-12T02:37:30.068 INFO:tasks.rados.rados.0.smithi107.stdout:43: done (2 left) 2020-06-12T02:37:30.068 INFO:tasks.rados.rados.0.smithi107.stdout:48: done (1 left) 2020-06-12T02:37:30.068 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:37:30.069 INFO:tasks.rados.rados.0.smithi107.stdout:49: seq_num 48 ranges {0=409600,409600=401408} 2020-06-12T02:37:30.072 INFO:tasks.rados.rados.0.smithi107.stdout:49: writing smithi10729407-6 from 0 to 409600 tid 1 2020-06-12T02:37:30.074 INFO:tasks.rados.rados.0.smithi107.stdout:49: writing smithi10729407-6 from 409600 to 811008 tid 2 2020-06-12T02:37:30.074 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:37:30.074 INFO:tasks.rados.rados.0.smithi107.stdout:50: seq_num 49 ranges {237568=425984,663552=401408} 2020-06-12T02:37:30.078 INFO:tasks.rados.rados.0.smithi107.stdout:50: writing smithi10729407-15 from 237568 to 663552 tid 1 2020-06-12T02:37:30.080 INFO:tasks.rados.rados.0.smithi107.stdout:50: writing smithi10729407-15 from 663552 to 1064960 tid 2 2020-06-12T02:37:30.081 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:37:30.081 INFO:tasks.rados.rados.0.smithi107.stdout:51: seq_num 50 ranges {270336=442368,712704=401408} 2020-06-12T02:37:30.084 INFO:tasks.rados.rados.0.smithi107.stdout:51: writing smithi10729407-17 from 270336 to 712704 tid 1 2020-06-12T02:37:30.087 INFO:tasks.rados.rados.0.smithi107.stdout:51: writing smithi10729407-17 from 712704 to 1114112 tid 2 2020-06-12T02:37:30.087 INFO:tasks.rados.rados.0.smithi107.stdout:52: delete oid 16 current snap is 0 2020-06-12T02:37:30.087 INFO:tasks.rados.rados.0.smithi107.stdout:47: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:30.088 INFO:tasks.rados.rados.0.smithi107.stdout:47: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:30.088 INFO:tasks.rados.rados.0.smithi107.stdout:47: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:30.088 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 8 (ObjNum 46 snap 0 seq_num 46) dirty exists 2020-06-12T02:37:30.088 INFO:tasks.rados.rados.0.smithi107.stdout:47: left oid 7 (ObjNum 46 snap 0 seq_num 46) 2020-06-12T02:37:30.088 INFO:tasks.rados.rados.0.smithi107.stdout:49: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:30.089 INFO:tasks.rados.rados.0.smithi107.stdout:49: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:30.089 INFO:tasks.rados.rados.0.smithi107.stdout:49: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:30.089 INFO:tasks.rados.rados.0.smithi107.stdout:49: finishing write tid 5 to smithi10729407-6 2020-06-12T02:37:30.089 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 12 (ObjNum 48 snap 0 seq_num 48) dirty exists 2020-06-12T02:37:30.089 INFO:tasks.rados.rados.0.smithi107.stdout:49: left oid 6 (ObjNum 48 snap 0 seq_num 48) 2020-06-12T02:37:30.089 INFO:tasks.rados.rados.0.smithi107.stdout:50: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:30.090 INFO:tasks.rados.rados.0.smithi107.stdout:47: done (4 left) 2020-06-12T02:37:30.091 INFO:tasks.rados.rados.0.smithi107.stdout:49: done (3 left) 2020-06-12T02:37:30.091 INFO:tasks.rados.rados.0.smithi107.stdout:52: done (2 left) 2020-06-12T02:37:30.091 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:37:30.091 INFO:tasks.rados.rados.0.smithi107.stdout:53: seq_num 52 ranges {0=475136,475136=401408} 2020-06-12T02:37:30.093 INFO:tasks.rados.rados.0.smithi107.stdout:53: writing smithi10729407-13 from 0 to 475136 tid 1 2020-06-12T02:37:30.096 INFO:tasks.rados.rados.0.smithi107.stdout:53: writing smithi10729407-13 from 475136 to 876544 tid 2 2020-06-12T02:37:30.096 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:37:30.097 INFO:tasks.rados.rados.0.smithi107.stdout:54: seq_num 53 ranges {139264=491520,630784=401408} 2020-06-12T02:37:30.098 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-06-12T02:37:30.104 INFO:tasks.rados.rados.0.smithi107.stdout:54: writing smithi10729407-9 from 139264 to 630784 tid 1 2020-06-12T02:37:30.105 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:30.105 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:30.106 INFO:tasks.rados.rados.0.smithi107.stdout:54: writing smithi10729407-9 from 630784 to 1032192 tid 2 2020-06-12T02:37:30.106 INFO:tasks.rados.rados.0.smithi107.stdout:55: delete oid 2 current snap is 0 2020-06-12T02:37:30.106 INFO:tasks.rados.rados.0.smithi107.stdout:50: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:30.106 INFO:tasks.rados.rados.0.smithi107.stdout:50: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:30.106 INFO:tasks.rados.rados.0.smithi107.stdout:51: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:30.107 INFO:tasks.rados.rados.0.smithi107.stdout:50: finishing write tid 5 to smithi10729407-15 2020-06-12T02:37:30.107 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 9 (ObjNum 49 snap 0 seq_num 49) dirty exists 2020-06-12T02:37:30.107 INFO:tasks.rados.rados.0.smithi107.stdout:50: left oid 15 (ObjNum 49 snap 0 seq_num 49) 2020-06-12T02:37:30.107 INFO:tasks.rados.rados.0.smithi107.stdout:51: finishing write tid 2 to smithi10729407-17 2020-06-12T02:37:30.107 INFO:tasks.rados.rados.0.smithi107.stdout:51: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:30.107 INFO:tasks.rados.rados.0.smithi107.stdout:51: finishing write tid 5 to smithi10729407-17 2020-06-12T02:37:30.108 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 11 (ObjNum 50 snap 0 seq_num 50) dirty exists 2020-06-12T02:37:30.108 INFO:tasks.rados.rados.0.smithi107.stdout:51: left oid 17 (ObjNum 50 snap 0 seq_num 50) 2020-06-12T02:37:30.108 INFO:tasks.rados.rados.0.smithi107.stdout:53: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:30.109 INFO:tasks.rados.rados.0.smithi107.stdout:53: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:30.109 INFO:tasks.rados.rados.0.smithi107.stdout:53: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:30.110 INFO:tasks.rados.rados.0.smithi107.stdout:50: done (4 left) 2020-06-12T02:37:30.110 INFO:tasks.rados.rados.0.smithi107.stdout:51: done (3 left) 2020-06-12T02:37:30.110 INFO:tasks.rados.rados.0.smithi107.stdout:55: done (2 left) 2020-06-12T02:37:30.110 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:37:30.111 INFO:tasks.rados.rados.0.smithi107.stdout:56: seq_num 55 ranges {0=524288,524288=401408} 2020-06-12T02:37:30.113 INFO:tasks.rados.rados.0.smithi107.stdout:56: writing smithi10729407-10 from 0 to 524288 tid 1 2020-06-12T02:37:30.114 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-06-12T02:37:30.114 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:37:30.110+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:37:30.115 INFO:tasks.rados.rados.0.smithi107.stdout:56: writing smithi10729407-10 from 524288 to 925696 tid 2 2020-06-12T02:37:30.116 INFO:tasks.rados.rados.0.smithi107.stdout:57: delete oid 3 current snap is 0 2020-06-12T02:37:30.116 INFO:tasks.rados.rados.0.smithi107.stdout:53: finishing write tid 5 to smithi10729407-13 2020-06-12T02:37:30.116 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 11 (ObjNum 52 snap 0 seq_num 52) dirty exists 2020-06-12T02:37:30.116 INFO:tasks.rados.rados.0.smithi107.stdout:53: left oid 13 (ObjNum 52 snap 0 seq_num 52) 2020-06-12T02:37:30.116 INFO:tasks.rados.rados.0.smithi107.stdout:54: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:30.117 INFO:tasks.rados.rados.0.smithi107.stdout:54: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:30.117 INFO:tasks.rados.rados.0.smithi107.stdout:54: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:30.117 INFO:tasks.rados.rados.0.smithi107.stdout:54: finishing write tid 5 to smithi10729407-9 2020-06-12T02:37:30.118 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 7 (ObjNum 53 snap 0 seq_num 53) dirty exists 2020-06-12T02:37:30.118 INFO:tasks.rados.rados.0.smithi107.stdout:54: left oid 9 (ObjNum 53 snap 0 seq_num 53) 2020-06-12T02:37:30.124 INFO:tasks.rados.rados.0.smithi107.stdout:53: done (3 left) 2020-06-12T02:37:30.124 INFO:tasks.rados.rados.0.smithi107.stdout:54: done (2 left) 2020-06-12T02:37:30.124 INFO:tasks.rados.rados.0.smithi107.stdout:57: done (1 left) 2020-06-12T02:37:30.124 INFO:tasks.rados.rados.0.smithi107.stdout:58: delete oid 9 current snap is 0 2020-06-12T02:37:30.126 INFO:tasks.rados.rados.0.smithi107.stdout:56: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:30.127 INFO:tasks.rados.rados.0.smithi107.stdout:56: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:30.127 INFO:tasks.rados.rados.0.smithi107.stdout:56: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:30.128 INFO:tasks.rados.rados.0.smithi107.stdout:58: done (1 left) 2020-06-12T02:37:30.128 INFO:tasks.rados.rados.0.smithi107.stdout:59: delete oid 16 current snap is 0 2020-06-12T02:37:30.130 INFO:tasks.rados.rados.0.smithi107.stdout:56: finishing write tid 5 to smithi10729407-10 2020-06-12T02:37:30.130 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 11 (ObjNum 55 snap 0 seq_num 55) dirty exists 2020-06-12T02:37:30.130 INFO:tasks.rados.rados.0.smithi107.stdout:56: left oid 10 (ObjNum 55 snap 0 seq_num 55) 2020-06-12T02:37:30.131 INFO:tasks.rados.rados.0.smithi107.stdout:56: done (1 left) 2020-06-12T02:37:30.131 INFO:tasks.rados.rados.0.smithi107.stdout:59: done (0 left) 2020-06-12T02:37:30.131 INFO:tasks.rados.rados.0.smithi107.stdout:60: delete oid 6 current snap is 0 2020-06-12T02:37:30.133 INFO:tasks.rados.rados.0.smithi107.stdout:60: done (0 left) 2020-06-12T02:37:30.133 INFO:tasks.rados.rados.0.smithi107.stdout:61: delete oid 8 current snap is 0 2020-06-12T02:37:30.134 INFO:tasks.rados.rados.0.smithi107.stdout:61: done (0 left) 2020-06-12T02:37:30.135 INFO:tasks.rados.rados.0.smithi107.stdout:62: delete oid 15 current snap is 0 2020-06-12T02:37:30.140 INFO:tasks.rados.rados.0.smithi107.stdout:62: done (0 left) 2020-06-12T02:37:30.141 INFO:tasks.rados.rados.0.smithi107.stdout:63: delete oid 7 current snap is 0 2020-06-12T02:37:30.144 INFO:tasks.rados.rados.0.smithi107.stdout:63: done (0 left) 2020-06-12T02:37:30.144 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:37:30.145 INFO:tasks.rados.rados.0.smithi107.stdout:64: seq_num 63 ranges {0=663552,663552=401408} 2020-06-12T02:37:30.148 INFO:tasks.rados.rados.0.smithi107.stdout:64: writing smithi10729407-7 from 0 to 663552 tid 1 2020-06-12T02:37:30.150 INFO:tasks.rados.rados.0.smithi107.stdout:64: writing smithi10729407-7 from 663552 to 1064960 tid 2 2020-06-12T02:37:30.151 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:37:30.151 INFO:tasks.rados.rados.0.smithi107.stdout:65: seq_num 64 ranges {0=679936,679936=401408} 2020-06-12T02:37:30.156 INFO:tasks.rados.rados.0.smithi107.stdout:65: writing smithi10729407-4 from 0 to 679936 tid 1 2020-06-12T02:37:30.158 INFO:tasks.rados.rados.0.smithi107.stdout:65: writing smithi10729407-4 from 679936 to 1081344 tid 2 2020-06-12T02:37:30.158 INFO:tasks.rados.rados.0.smithi107.stdout:66: delete oid 10 current snap is 0 2020-06-12T02:37:30.159 INFO:tasks.rados.rados.0.smithi107.stdout:64: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:30.159 INFO:tasks.rados.rados.0.smithi107.stdout:64: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:30.159 INFO:tasks.rados.rados.0.smithi107.stdout:64: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:30.161 INFO:tasks.rados.rados.0.smithi107.stdout:64: finishing write tid 5 to smithi10729407-7 2020-06-12T02:37:30.164 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 15 (ObjNum 63 snap 0 seq_num 63) dirty exists 2020-06-12T02:37:30.164 INFO:tasks.rados.rados.0.smithi107.stdout:64: left oid 7 (ObjNum 63 snap 0 seq_num 63) 2020-06-12T02:37:30.165 INFO:tasks.rados.rados.0.smithi107.stdout:64: done (2 left) 2020-06-12T02:37:30.165 INFO:tasks.rados.rados.0.smithi107.stdout:66: done (1 left) 2020-06-12T02:37:30.165 INFO:tasks.rados.rados.0.smithi107.stdout:67: delete oid 10 current snap is 0 2020-06-12T02:37:30.166 INFO:tasks.rados.rados.0.smithi107.stdout:67: done (1 left) 2020-06-12T02:37:30.166 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:37:30.166 INFO:tasks.rados.rados.0.smithi107.stdout:68: seq_num 67 ranges {0=729088,729088=401408} 2020-06-12T02:37:30.170 INFO:tasks.rados.rados.0.smithi107.stdout:68: writing smithi10729407-16 from 0 to 729088 tid 1 2020-06-12T02:37:30.173 INFO:tasks.rados.rados.0.smithi107.stdout:68: writing smithi10729407-16 from 729088 to 1130496 tid 2 2020-06-12T02:37:30.173 INFO:tasks.rados.rados.0.smithi107.stdout:69: delete oid 11 current snap is 0 2020-06-12T02:37:30.174 INFO:tasks.rados.rados.0.smithi107.stdout:65: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:30.174 INFO:tasks.rados.rados.0.smithi107.stdout:65: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:30.174 INFO:tasks.rados.rados.0.smithi107.stdout:65: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:30.175 INFO:tasks.rados.rados.0.smithi107.stdout:65: finishing write tid 5 to smithi10729407-4 2020-06-12T02:37:30.175 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 14 (ObjNum 64 snap 0 seq_num 64) dirty exists 2020-06-12T02:37:30.175 INFO:tasks.rados.rados.0.smithi107.stdout:65: left oid 4 (ObjNum 64 snap 0 seq_num 64) 2020-06-12T02:37:30.176 INFO:tasks.rados.rados.0.smithi107.stdout:65: done (2 left) 2020-06-12T02:37:30.176 INFO:tasks.rados.rados.0.smithi107.stdout:69: done (1 left) 2020-06-12T02:37:30.176 INFO:tasks.rados.rados.0.smithi107.stdout:70: delete oid 8 current snap is 0 2020-06-12T02:37:30.180 INFO:tasks.rados.rados.0.smithi107.stdout:70: done (1 left) 2020-06-12T02:37:30.180 INFO:tasks.rados.rados.0.smithi107.stdout:71: delete oid 1 current snap is 0 2020-06-12T02:37:30.181 INFO:tasks.rados.rados.0.smithi107.stdout:68: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:30.182 INFO:tasks.rados.rados.0.smithi107.stdout:71: done (1 left) 2020-06-12T02:37:30.182 INFO:tasks.rados.rados.0.smithi107.stdout:72: delete oid 9 current snap is 0 2020-06-12T02:37:30.183 INFO:tasks.rados.rados.0.smithi107.stdout:68: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:30.183 INFO:tasks.rados.rados.0.smithi107.stdout:68: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:30.184 INFO:tasks.rados.rados.0.smithi107.stdout:72: done (1 left) 2020-06-12T02:37:30.185 INFO:tasks.rados.rados.0.smithi107.stdout:73: delete oid 19 current snap is 0 2020-06-12T02:37:30.185 INFO:tasks.rados.rados.0.smithi107.stdout:68: finishing write tid 5 to smithi10729407-16 2020-06-12T02:37:30.185 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 12 (ObjNum 67 snap 0 seq_num 67) dirty exists 2020-06-12T02:37:30.185 INFO:tasks.rados.rados.0.smithi107.stdout:68: left oid 16 (ObjNum 67 snap 0 seq_num 67) 2020-06-12T02:37:30.186 INFO:tasks.rados.rados.0.smithi107.stdout:68: done (1 left) 2020-06-12T02:37:30.186 INFO:tasks.rados.rados.0.smithi107.stdout:73: done (0 left) 2020-06-12T02:37:30.187 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:37:30.187 INFO:tasks.rados.rados.0.smithi107.stdout:74: seq_num 73 ranges {1081344=425984,1507328=409600,1916928=393216} 2020-06-12T02:37:30.194 INFO:tasks.rados.rados.0.smithi107.stdout:74: writing smithi10729407-4 from 1081344 to 1507328 tid 1 2020-06-12T02:37:30.197 INFO:tasks.rados.rados.0.smithi107.stdout:74: writing smithi10729407-4 from 1507328 to 1916928 tid 2 2020-06-12T02:37:30.199 INFO:tasks.rados.rados.0.smithi107.stdout:74: writing smithi10729407-4 from 1916928 to 2310144 tid 3 2020-06-12T02:37:30.200 INFO:tasks.rados.rados.0.smithi107.stdout:75: delete oid 3 current snap is 0 2020-06-12T02:37:30.200 INFO:tasks.rados.rados.0.smithi107.stdout:74: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:30.201 INFO:tasks.rados.rados.0.smithi107.stdout:75: done (1 left) 2020-06-12T02:37:30.202 INFO:tasks.rados.rados.0.smithi107.stdout:76: delete oid 11 current snap is 0 2020-06-12T02:37:30.203 INFO:tasks.rados.rados.0.smithi107.stdout:76: done (1 left) 2020-06-12T02:37:30.204 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:37:30.204 INFO:tasks.rados.rados.0.smithi107.stdout:77: seq_num 76 ranges {0=475136,475136=458752,933888=344064} 2020-06-12T02:37:30.207 INFO:tasks.rados.rados.0.smithi107.stdout:77: writing smithi10729407-5 from 0 to 475136 tid 1 2020-06-12T02:37:30.210 INFO:tasks.rados.rados.0.smithi107.stdout:77: writing smithi10729407-5 from 475136 to 933888 tid 2 2020-06-12T02:37:30.213 INFO:tasks.rados.rados.0.smithi107.stdout:77: writing smithi10729407-5 from 933888 to 1277952 tid 3 2020-06-12T02:37:30.213 INFO:tasks.rados.rados.0.smithi107.stdout:78: delete oid 2 current snap is 0 2020-06-12T02:37:30.213 INFO:tasks.rados.rados.0.smithi107.stdout:74: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:30.214 INFO:tasks.rados.rados.0.smithi107.stdout:74: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:30.214 INFO:tasks.rados.rados.0.smithi107.stdout:74: finishing write tid 5 to smithi10729407-4 2020-06-12T02:37:30.214 INFO:tasks.rados.rados.0.smithi107.stdout:74: finishing write tid 6 to smithi10729407-4 2020-06-12T02:37:30.215 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 18 (ObjNum 73 snap 0 seq_num 73) dirty exists 2020-06-12T02:37:30.215 INFO:tasks.rados.rados.0.smithi107.stdout:74: left oid 4 (ObjNum 73 snap 0 seq_num 73) 2020-06-12T02:37:30.215 INFO:tasks.rados.rados.0.smithi107.stdout:77: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:30.217 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:30.214+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:30.228 INFO:tasks.rados.rados.0.smithi107.stdout:77: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:30.229 INFO:tasks.rados.rados.0.smithi107.stdout:74: done (2 left) 2020-06-12T02:37:30.229 INFO:tasks.rados.rados.0.smithi107.stdout:78: done (1 left) 2020-06-12T02:37:30.229 INFO:tasks.rados.rados.0.smithi107.stdout:79: delete oid 8 current snap is 0 2020-06-12T02:37:30.229 INFO:tasks.rados.rados.0.smithi107.stdout:77: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:30.230 INFO:tasks.rados.rados.0.smithi107.stdout:77: finishing write tid 5 to smithi10729407-5 2020-06-12T02:37:30.230 INFO:tasks.rados.rados.0.smithi107.stdout:79: done (1 left) 2020-06-12T02:37:30.230 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:37:30.230 INFO:tasks.rados.rados.0.smithi107.stdout:80: seq_num 79 ranges {0=532480,532480=565248,1097728=237568} 2020-06-12T02:37:30.230 INFO:tasks.rados.rados.0.smithi107.stdout:80: writing smithi10729407-1 from 0 to 532480 tid 1 2020-06-12T02:37:30.231 INFO:tasks.rados.rados.0.smithi107.stdout:80: writing smithi10729407-1 from 532480 to 1097728 tid 2 2020-06-12T02:37:30.231 INFO:tasks.rados.rados.0.smithi107.stdout:80: writing smithi10729407-1 from 1097728 to 1335296 tid 3 2020-06-12T02:37:30.231 INFO:tasks.rados.rados.0.smithi107.stdout:81: delete oid 2 current snap is 0 2020-06-12T02:37:30.231 INFO:tasks.rados.rados.0.smithi107.stdout:77: finishing write tid 6 to smithi10729407-5 2020-06-12T02:37:30.231 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 7 (ObjNum 76 snap 0 seq_num 76) dirty exists 2020-06-12T02:37:30.231 INFO:tasks.rados.rados.0.smithi107.stdout:77: left oid 5 (ObjNum 76 snap 0 seq_num 76) 2020-06-12T02:37:30.232 INFO:tasks.rados.rados.0.smithi107.stdout:80: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:30.232 INFO:tasks.rados.rados.0.smithi107.stdout:77: done (2 left) 2020-06-12T02:37:30.232 INFO:tasks.rados.rados.0.smithi107.stdout:81: done (1 left) 2020-06-12T02:37:30.233 INFO:tasks.rados.rados.0.smithi107.stdout:82: delete oid 7 current snap is 0 2020-06-12T02:37:30.233 INFO:tasks.rados.rados.0.smithi107.stdout:82: done (1 left) 2020-06-12T02:37:30.233 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:37:30.234 INFO:tasks.rados.rados.0.smithi107.stdout:83: seq_num 82 ranges {0=581632,581632=622592,1204224=180224} 2020-06-12T02:37:30.236 INFO:tasks.rados.rados.0.smithi107.stdout:83: writing smithi10729407-8 from 0 to 581632 tid 1 2020-06-12T02:37:30.240 INFO:tasks.rados.rados.0.smithi107.stdout:83: writing smithi10729407-8 from 581632 to 1204224 tid 2 2020-06-12T02:37:30.242 INFO:tasks.rados.rados.0.smithi107.stdout:83: writing smithi10729407-8 from 1204224 to 1384448 tid 3 2020-06-12T02:37:30.242 INFO:tasks.rados.rados.0.smithi107.stdout:84: delete oid 10 current snap is 0 2020-06-12T02:37:30.242 INFO:tasks.rados.rados.0.smithi107.stdout:80: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:30.243 INFO:tasks.rados.rados.0.smithi107.stdout:80: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:30.243 INFO:tasks.rados.rados.0.smithi107.stdout:80: finishing write tid 5 to smithi10729407-1 2020-06-12T02:37:30.243 INFO:tasks.rados.rados.0.smithi107.stdout:80: finishing write tid 6 to smithi10729407-1 2020-06-12T02:37:30.243 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 18 (ObjNum 79 snap 0 seq_num 79) dirty exists 2020-06-12T02:37:30.243 INFO:tasks.rados.rados.0.smithi107.stdout:80: left oid 1 (ObjNum 79 snap 0 seq_num 79) 2020-06-12T02:37:30.246 INFO:tasks.rados.rados.0.smithi107.stdout:80: done (2 left) 2020-06-12T02:37:30.246 INFO:tasks.rados.rados.0.smithi107.stdout:84: done (1 left) 2020-06-12T02:37:30.246 INFO:tasks.rados.rados.0.smithi107.stdout:85: delete oid 10 current snap is 0 2020-06-12T02:37:30.246 INFO:tasks.rados.rados.0.smithi107.stdout:83: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:30.248 INFO:tasks.rados.rados.0.smithi107.stdout:85: done (1 left) 2020-06-12T02:37:30.248 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:37:30.248 INFO:tasks.rados.rados.0.smithi107.stdout:86: seq_num 85 ranges {1114112=630784,1744896=720896,2465792=81920} 2020-06-12T02:37:30.258 INFO:tasks.rados.rados.0.smithi107.stdout:86: writing smithi10729407-17 from 1114112 to 1744896 tid 1 2020-06-12T02:37:30.262 INFO:tasks.rados.rados.0.smithi107.stdout:86: writing smithi10729407-17 from 1744896 to 2465792 tid 2 2020-06-12T02:37:30.263 INFO:tasks.rados.rados.0.smithi107.stdout:86: writing smithi10729407-17 from 2465792 to 2547712 tid 3 2020-06-12T02:37:30.263 INFO:tasks.rados.rados.0.smithi107.stdout:87: delete oid 19 current snap is 0 2020-06-12T02:37:30.263 INFO:tasks.rados.rados.0.smithi107.stdout:83: finishing write tid 2 to smithi10729407-8 2020-06-12T02:37:30.263 INFO:tasks.rados.rados.0.smithi107.stdout:83: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:30.264 INFO:tasks.rados.rados.0.smithi107.stdout:83: finishing write tid 5 to smithi10729407-8 2020-06-12T02:37:30.264 INFO:tasks.rados.rados.0.smithi107.stdout:83: finishing write tid 6 to smithi10729407-8 2020-06-12T02:37:30.264 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 17 (ObjNum 82 snap 0 seq_num 82) dirty exists 2020-06-12T02:37:30.264 INFO:tasks.rados.rados.0.smithi107.stdout:83: left oid 8 (ObjNum 82 snap 0 seq_num 82) 2020-06-12T02:37:30.265 INFO:tasks.rados.rados.0.smithi107.stdout:86: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:30.266 INFO:tasks.rados.rados.0.smithi107.stdout:83: done (2 left) 2020-06-12T02:37:30.266 INFO:tasks.rados.rados.0.smithi107.stdout:87: done (1 left) 2020-06-12T02:37:30.266 INFO:tasks.rados.rados.0.smithi107.stdout:88: delete oid 8 current snap is 0 2020-06-12T02:37:30.268 INFO:tasks.rados.rados.0.smithi107.stdout:88: done (1 left) 2020-06-12T02:37:30.268 INFO:tasks.rados.rados.0.smithi107.stdout:89: delete oid 1 current snap is 0 2020-06-12T02:37:30.270 INFO:tasks.rados.rados.0.smithi107.stdout:86: finishing write tid 2 to smithi10729407-17 2020-06-12T02:37:30.271 INFO:tasks.rados.rados.0.smithi107.stdout:89: done (1 left) 2020-06-12T02:37:30.271 INFO:tasks.rados.rados.0.smithi107.stdout:90: delete oid 13 current snap is 0 2020-06-12T02:37:30.272 INFO:tasks.rados.rados.0.smithi107.stdout:86: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:30.273 INFO:tasks.rados.rados.0.smithi107.stdout:86: finishing write tid 5 to smithi10729407-17 2020-06-12T02:37:30.273 INFO:tasks.rados.rados.0.smithi107.stdout:90: done (1 left) 2020-06-12T02:37:30.274 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:37:30.274 INFO:tasks.rados.rados.0.smithi107.stdout:91: seq_num 90 ranges {0=712704,712704=540672,1253376=262144} 2020-06-12T02:37:30.278 INFO:tasks.rados.rados.0.smithi107.stdout:91: writing smithi10729407-8 from 0 to 712704 tid 1 2020-06-12T02:37:30.280 INFO:tasks.rados.rados.0.smithi107.stdout:91: writing smithi10729407-8 from 712704 to 1253376 tid 2 2020-06-12T02:37:30.282 INFO:tasks.rados.rados.0.smithi107.stdout:91: writing smithi10729407-8 from 1253376 to 1515520 tid 3 2020-06-12T02:37:30.282 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:37:30.283 INFO:tasks.rados.rados.0.smithi107.stdout:92: seq_num 91 ranges {0=729088,729088=425984,1155072=376832} 2020-06-12T02:37:30.287 INFO:tasks.rados.rados.0.smithi107.stdout:92: writing smithi10729407-19 from 0 to 729088 tid 1 2020-06-12T02:37:30.289 INFO:tasks.rados.rados.0.smithi107.stdout:92: writing smithi10729407-19 from 729088 to 1155072 tid 2 2020-06-12T02:37:30.292 INFO:tasks.rados.rados.0.smithi107.stdout:92: writing smithi10729407-19 from 1155072 to 1531904 tid 3 2020-06-12T02:37:30.292 INFO:tasks.rados.rados.0.smithi107.stdout:93: delete oid 18 current snap is 0 2020-06-12T02:37:30.292 INFO:tasks.rados.rados.0.smithi107.stdout:86: finishing write tid 6 to smithi10729407-17 2020-06-12T02:37:30.293 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 22 (ObjNum 85 snap 0 seq_num 85) dirty exists 2020-06-12T02:37:30.293 INFO:tasks.rados.rados.0.smithi107.stdout:86: left oid 17 (ObjNum 85 snap 0 seq_num 85) 2020-06-12T02:37:30.293 INFO:tasks.rados.rados.0.smithi107.stdout:91: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:30.294 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:30.294 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:30.294 INFO:tasks.rados.rados.0.smithi107.stdout:91: finishing write tid 2 to smithi10729407-8 2020-06-12T02:37:30.294 INFO:tasks.rados.rados.0.smithi107.stdout:91: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:30.295 INFO:tasks.rados.rados.0.smithi107.stdout:91: finishing write tid 5 to smithi10729407-8 2020-06-12T02:37:30.295 INFO:tasks.rados.rados.0.smithi107.stdout:86: done (3 left) 2020-06-12T02:37:30.295 INFO:tasks.rados.rados.0.smithi107.stdout:93: done (2 left) 2020-06-12T02:37:30.296 INFO:tasks.rados.rados.0.smithi107.stdout:94: delete oid 17 current snap is 0 2020-06-12T02:37:30.296 INFO:tasks.rados.rados.0.smithi107.stdout:91: finishing write tid 6 to smithi10729407-8 2020-06-12T02:37:30.296 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 22 (ObjNum 90 snap 0 seq_num 90) dirty exists 2020-06-12T02:37:30.297 INFO:tasks.rados.rados.0.smithi107.stdout:91: left oid 8 (ObjNum 90 snap 0 seq_num 90) 2020-06-12T02:37:30.298 INFO:tasks.rados.rados.0.smithi107.stdout:91: done (2 left) 2020-06-12T02:37:30.299 INFO:tasks.rados.rados.0.smithi107.stdout:94: done (1 left) 2020-06-12T02:37:30.299 INFO:tasks.rados.rados.0.smithi107.stdout:95: delete oid 20 current snap is 0 2020-06-12T02:37:30.299 INFO:tasks.rados.rados.0.smithi107.stdout:92: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:30.300 INFO:tasks.rados.rados.0.smithi107.stdout:92: finishing write tid 2 to smithi10729407-19 2020-06-12T02:37:30.302 INFO:tasks.rados.rados.0.smithi107.stdout:95: done (1 left) 2020-06-12T02:37:30.302 INFO:tasks.rados.rados.0.smithi107.stdout:96: delete oid 16 current snap is 0 2020-06-12T02:37:30.302 INFO:tasks.rados.rados.0.smithi107.stdout:92: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:30.303 INFO:tasks.rados.rados.0.smithi107.stdout:92: finishing write tid 5 to smithi10729407-19 2020-06-12T02:37:30.304 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-06-12T02:37:30.305 INFO:tasks.rados.rados.0.smithi107.stdout:96: done (1 left) 2020-06-12T02:37:30.305 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:37:30.305 INFO:tasks.rados.rados.0.smithi107.stdout:97: seq_num 96 ranges {0=409600,409600=696320,1105920=507904} 2020-06-12T02:37:30.307 INFO:tasks.rados.rados.0.smithi107.stdout:97: writing smithi10729407-2 from 0 to 409600 tid 1 2020-06-12T02:37:30.311 INFO:tasks.rados.rados.0.smithi107.stdout:97: writing smithi10729407-2 from 409600 to 1105920 tid 2 2020-06-12T02:37:30.318 INFO:tasks.rados.rados.0.smithi107.stdout:97: writing smithi10729407-2 from 1105920 to 1613824 tid 3 2020-06-12T02:37:30.318 INFO:tasks.rados.rados.0.smithi107.stdout:98: delete oid 6 current snap is 0 2020-06-12T02:37:30.320 INFO:tasks.rados.rados.0.smithi107.stdout:92: finishing write tid 6 to smithi10729407-19 2020-06-12T02:37:30.320 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 15 (ObjNum 91 snap 0 seq_num 91) dirty exists 2020-06-12T02:37:30.321 INFO:tasks.rados.rados.0.smithi107.stdout:92: left oid 19 (ObjNum 91 snap 0 seq_num 91) 2020-06-12T02:37:30.321 INFO:tasks.rados.rados.0.smithi107.stdout:97: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:30.322 INFO:tasks.rados.rados.0.smithi107.stdout:92: done (2 left) 2020-06-12T02:37:30.322 INFO:tasks.rados.rados.0.smithi107.stdout:98: done (1 left) 2020-06-12T02:37:30.323 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:37:30.323 INFO:tasks.rados.rados.0.smithi107.stdout:99: seq_num 98 ranges {1515520=450560,1966080=466944,2433024=737280} 2020-06-12T02:37:30.327 INFO:tasks.rados.rados.0.smithi107.stdout:99: writing smithi10729407-8 from 1515520 to 1966080 tid 1 2020-06-12T02:37:30.330 INFO:tasks.rados.rados.0.smithi107.stdout:99: writing smithi10729407-8 from 1966080 to 2433024 tid 2 2020-06-12T02:37:30.334 INFO:tasks.rados.rados.0.smithi107.stdout:99: writing smithi10729407-8 from 2433024 to 3170304 tid 3 2020-06-12T02:37:30.335 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:37:30.335 INFO:tasks.rados.rados.0.smithi107.stdout:100: seq_num 99 ranges {0=466944,466944=802816,1269760=401408} 2020-06-12T02:37:30.337 INFO:tasks.rados.rados.0.smithi107.stdout:100: writing smithi10729407-7 from 0 to 466944 tid 1 2020-06-12T02:37:30.342 INFO:tasks.rados.rados.0.smithi107.stdout:100: writing smithi10729407-7 from 466944 to 1269760 tid 2 2020-06-12T02:37:30.345 INFO:tasks.rados.rados.0.smithi107.stdout:100: writing smithi10729407-7 from 1269760 to 1671168 tid 3 2020-06-12T02:37:30.346 INFO:tasks.rados.rados.0.smithi107.stdout:101: delete oid 5 current snap is 0 2020-06-12T02:37:30.346 INFO:tasks.rados.rados.0.smithi107.stdout:97: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:30.346 INFO:tasks.rados.rados.0.smithi107.stdout:97: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:30.346 INFO:tasks.rados.rados.0.smithi107.stdout:97: finishing write tid 5 to smithi10729407-2 2020-06-12T02:37:30.346 INFO:tasks.rados.rados.0.smithi107.stdout:97: finishing write tid 6 to smithi10729407-2 2020-06-12T02:37:30.347 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 10 (ObjNum 96 snap 0 seq_num 96) dirty exists 2020-06-12T02:37:30.347 INFO:tasks.rados.rados.0.smithi107.stdout:97: left oid 2 (ObjNum 96 snap 0 seq_num 96) 2020-06-12T02:37:30.347 INFO:tasks.rados.rados.0.smithi107.stdout:99: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:30.347 INFO:tasks.rados.rados.0.smithi107.stdout:99: finishing write tid 2 to smithi10729407-8 2020-06-12T02:37:30.347 INFO:tasks.rados.rados.0.smithi107.stdout:99: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:30.347 INFO:tasks.rados.rados.0.smithi107.stdout:99: finishing write tid 5 to smithi10729407-8 2020-06-12T02:37:30.348 INFO:tasks.rados.rados.0.smithi107.stdout:100: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:30.348 INFO:tasks.rados.rados.0.smithi107.stdout:99: finishing write tid 6 to smithi10729407-8 2020-06-12T02:37:30.348 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 26 (ObjNum 98 snap 0 seq_num 98) dirty exists 2020-06-12T02:37:30.348 INFO:tasks.rados.rados.0.smithi107.stdout:99: left oid 8 (ObjNum 98 snap 0 seq_num 98) 2020-06-12T02:37:30.349 INFO:tasks.rados.rados.0.smithi107.stdout:97: done (3 left) 2020-06-12T02:37:30.349 INFO:tasks.rados.rados.0.smithi107.stdout:99: done (2 left) 2020-06-12T02:37:30.349 INFO:tasks.rados.rados.0.smithi107.stdout:101: done (1 left) 2020-06-12T02:37:30.351 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:37:30.351 INFO:tasks.rados.rados.0.smithi107.stdout:102: seq_num 101 ranges {0=499712,499712=573440,1073152=524288,1597440=106496} 2020-06-12T02:37:30.351 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:30.347+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:30.352 INFO:tasks.rados.rados.0.smithi107.stdout:102: writing smithi10729407-3 from 0 to 499712 tid 1 2020-06-12T02:37:30.355 INFO:tasks.rados.rados.0.smithi107.stdout:102: writing smithi10729407-3 from 499712 to 1073152 tid 2 2020-06-12T02:37:30.359 INFO:tasks.rados.rados.0.smithi107.stdout:102: writing smithi10729407-3 from 1073152 to 1597440 tid 3 2020-06-12T02:37:30.360 INFO:tasks.rados.rados.0.smithi107.stdout:102: writing smithi10729407-3 from 1597440 to 1703936 tid 4 2020-06-12T02:37:30.361 INFO:tasks.rados.rados.0.smithi107.stdout:103: delete oid 13 current snap is 0 2020-06-12T02:37:30.361 INFO:tasks.rados.rados.0.smithi107.stdout:100: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:30.361 INFO:tasks.rados.rados.0.smithi107.stdout:100: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:30.361 INFO:tasks.rados.rados.0.smithi107.stdout:100: finishing write tid 5 to smithi10729407-7 2020-06-12T02:37:30.362 INFO:tasks.rados.rados.0.smithi107.stdout:100: finishing write tid 6 to smithi10729407-7 2020-06-12T02:37:30.362 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 21 (ObjNum 99 snap 0 seq_num 99) dirty exists 2020-06-12T02:37:30.362 INFO:tasks.rados.rados.0.smithi107.stdout:100: left oid 7 (ObjNum 99 snap 0 seq_num 99) 2020-06-12T02:37:30.363 INFO:tasks.rados.rados.0.smithi107.stdout:102: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:30.369 INFO:tasks.rados.rados.0.smithi107.stdout:100: done (2 left) 2020-06-12T02:37:30.370 INFO:tasks.rados.rados.0.smithi107.stdout:103: done (1 left) 2020-06-12T02:37:30.370 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:30.370 INFO:tasks.rados.rados.0.smithi107.stdout:104: seq_num 103 ranges {1613824=532480,2146304=737280,2883584=466944} 2020-06-12T02:37:30.381 INFO:tasks.rados.rados.0.smithi107.stdout:104: writing smithi10729407-2 from 1613824 to 2146304 tid 1 2020-06-12T02:37:30.385 INFO:tasks.rados.rados.0.smithi107.stdout:104: writing smithi10729407-2 from 2146304 to 2883584 tid 2 2020-06-12T02:37:30.388 INFO:tasks.rados.rados.0.smithi107.stdout:104: writing smithi10729407-2 from 2883584 to 3350528 tid 3 2020-06-12T02:37:30.389 INFO:tasks.rados.rados.0.smithi107.stdout:105: delete oid 8 current snap is 0 2020-06-12T02:37:30.389 INFO:tasks.rados.rados.0.smithi107.stdout:102: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:30.389 INFO:tasks.rados.rados.0.smithi107.stdout:102: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:30.389 INFO:tasks.rados.rados.0.smithi107.stdout:102: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:30.390 INFO:tasks.rados.rados.0.smithi107.stdout:102: finishing write tid 6 to smithi10729407-3 2020-06-12T02:37:30.390 INFO:tasks.rados.rados.0.smithi107.stdout:102: finishing write tid 7 to smithi10729407-3 2020-06-12T02:37:30.390 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 19 (ObjNum 101 snap 0 seq_num 101) dirty exists 2020-06-12T02:37:30.390 INFO:tasks.rados.rados.0.smithi107.stdout:102: left oid 3 (ObjNum 101 snap 0 seq_num 101) 2020-06-12T02:37:30.391 INFO:tasks.rados.rados.0.smithi107.stdout:104: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:30.392 INFO:tasks.rados.rados.0.smithi107.stdout:102: done (2 left) 2020-06-12T02:37:30.392 INFO:tasks.rados.rados.0.smithi107.stdout:105: done (1 left) 2020-06-12T02:37:30.392 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:37:30.392 INFO:tasks.rados.rados.0.smithi107.stdout:106: seq_num 105 ranges {0=565248,565248=507904,1073152=507904,1581056=188416} 2020-06-12T02:37:30.395 INFO:tasks.rados.rados.0.smithi107.stdout:106: writing smithi10729407-1 from 0 to 565248 tid 1 2020-06-12T02:37:30.399 INFO:tasks.rados.rados.0.smithi107.stdout:106: writing smithi10729407-1 from 565248 to 1073152 tid 2 2020-06-12T02:37:30.402 INFO:tasks.rados.rados.0.smithi107.stdout:106: writing smithi10729407-1 from 1073152 to 1581056 tid 3 2020-06-12T02:37:30.404 INFO:tasks.rados.rados.0.smithi107.stdout:106: writing smithi10729407-1 from 1581056 to 1769472 tid 4 2020-06-12T02:37:30.404 INFO:tasks.rados.rados.0.smithi107.stdout:107: delete oid 3 current snap is 0 2020-06-12T02:37:30.404 INFO:tasks.rados.rados.0.smithi107.stdout:104: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:30.404 INFO:tasks.rados.rados.0.smithi107.stdout:104: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:30.404 INFO:tasks.rados.rados.0.smithi107.stdout:104: finishing write tid 5 to smithi10729407-2 2020-06-12T02:37:30.405 INFO:tasks.rados.rados.0.smithi107.stdout:104: finishing write tid 6 to smithi10729407-2 2020-06-12T02:37:30.405 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 14 (ObjNum 103 snap 0 seq_num 103) dirty exists 2020-06-12T02:37:30.405 INFO:tasks.rados.rados.0.smithi107.stdout:104: left oid 2 (ObjNum 103 snap 0 seq_num 103) 2020-06-12T02:37:30.405 INFO:tasks.rados.rados.0.smithi107.stdout:106: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:30.407 INFO:tasks.rados.rados.0.smithi107.stdout:104: done (2 left) 2020-06-12T02:37:30.407 INFO:tasks.rados.rados.0.smithi107.stdout:107: done (1 left) 2020-06-12T02:37:30.407 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:37:30.407 INFO:tasks.rados.rados.0.smithi107.stdout:108: seq_num 107 ranges {0=598016,598016=729088,1327104=466944,1794048=8192} 2020-06-12T02:37:30.410 INFO:tasks.rados.rados.0.smithi107.stdout:108: writing smithi10729407-9 from 0 to 598016 tid 1 2020-06-12T02:37:30.414 INFO:tasks.rados.rados.0.smithi107.stdout:108: writing smithi10729407-9 from 598016 to 1327104 tid 2 2020-06-12T02:37:30.418 INFO:tasks.rados.rados.0.smithi107.stdout:108: writing smithi10729407-9 from 1327104 to 1794048 tid 3 2020-06-12T02:37:30.418 INFO:tasks.rados.rados.0.smithi107.stdout:108: writing smithi10729407-9 from 1794048 to 1802240 tid 4 2020-06-12T02:37:30.418 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:37:30.418 INFO:tasks.rados.rados.0.smithi107.stdout:109: seq_num 108 ranges {0=614400,614400=606208,1220608=598016} 2020-06-12T02:37:30.421 INFO:tasks.rados.rados.0.smithi107.stdout:109: writing smithi10729407-14 from 0 to 614400 tid 1 2020-06-12T02:37:30.423 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:30.420+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:30.425 INFO:tasks.rados.rados.0.smithi107.stdout:109: writing smithi10729407-14 from 614400 to 1220608 tid 2 2020-06-12T02:37:30.430 INFO:tasks.rados.rados.0.smithi107.stdout:109: writing smithi10729407-14 from 1220608 to 1818624 tid 3 2020-06-12T02:37:30.438 INFO:tasks.rados.rados.0.smithi107.stdout:110: delete oid 12 current snap is 0 2020-06-12T02:37:30.439 INFO:tasks.rados.rados.0.smithi107.stdout:106: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:30.439 INFO:tasks.rados.rados.0.smithi107.stdout:106: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:30.439 INFO:tasks.rados.rados.0.smithi107.stdout:106: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:30.439 INFO:tasks.rados.rados.0.smithi107.stdout:106: finishing write tid 6 to smithi10729407-1 2020-06-12T02:37:30.440 INFO:tasks.rados.rados.0.smithi107.stdout:106: finishing write tid 7 to smithi10729407-1 2020-06-12T02:37:30.440 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 25 (ObjNum 105 snap 0 seq_num 105) dirty exists 2020-06-12T02:37:30.440 INFO:tasks.rados.rados.0.smithi107.stdout:106: left oid 1 (ObjNum 105 snap 0 seq_num 105) 2020-06-12T02:37:30.440 INFO:tasks.rados.rados.0.smithi107.stdout:108: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:30.440 INFO:tasks.rados.rados.0.smithi107.stdout:108: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:30.440 INFO:tasks.rados.rados.0.smithi107.stdout:108: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:30.441 INFO:tasks.rados.rados.0.smithi107.stdout:108: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:30.441 INFO:tasks.rados.rados.0.smithi107.stdout:108: finishing write tid 6 to smithi10729407-9 2020-06-12T02:37:30.441 INFO:tasks.rados.rados.0.smithi107.stdout:106: done (3 left) 2020-06-12T02:37:30.441 INFO:tasks.rados.rados.0.smithi107.stdout:110: done (2 left) 2020-06-12T02:37:30.441 INFO:tasks.rados.rados.0.smithi107.stdout:111: delete oid 8 current snap is 0 2020-06-12T02:37:30.442 INFO:tasks.rados.rados.0.smithi107.stdout:111: done (2 left) 2020-06-12T02:37:30.442 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:37:30.442 INFO:tasks.rados.rados.0.smithi107.stdout:112: seq_num 111 ranges {1769472=663552,2433024=663552,3096576=450560,3547136=90112} 2020-06-12T02:37:30.444 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:30.444 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:30.447 INFO:tasks.rados.rados.0.smithi107.stdout:112: writing smithi10729407-1 from 1769472 to 2433024 tid 1 2020-06-12T02:37:30.450 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:37:30.450 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":24,"fsid":"2f5030a1-3206-43c9-8471-d10429688d36","created":"2020-06-12T02:37:00.578493+0000","modified":"2020-06-12T02:37:29.914980+0000","last_up_change":"2020-06-12T02:37:24.880579+0000","last_in_change":"2020-06-12T02:37:10.705891+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-06-12T02:37:03.610773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-06-12T02:37:14.831188+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2020-06-12T02:37:25.893619+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":true,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"ad4b4814-83e7-4a2e-9b1f-d10406174f7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":26614},{"type":"v1","addr":"172.21.15.107:6817","nonce":26614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6819","nonce":26614},{"type":"v1","addr":"172.21.15.107:6821","nonce":26614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6827","nonce":26614},{"type":"v1","addr":"172.21.15.107:6829","nonce":26614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":26614},{"type":"v1","addr":"172.21.15.107:6825","nonce":26614}]},"public_addr":"172.21.15.107:6817/26614","cluster_addr":"172.21.15.107:6821/26614","heartbeat_back_addr":"172.21.15.107:6829/26614","heartbeat_front_addr":"172.21.15.107:6825/26614","state":["exists","up"]},{"osd":1,"uuid":"046adf0d-b9e9-41ad-a7c9-3841f8b8f790","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":26611},{"type":"v1","addr":"172.21.15.107:6801","nonce":26611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":26611},{"type":"v1","addr":"172.21.15.107:6803","nonce":26611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":26611},{"type":"v1","addr":"172.21.15.107:6807","nonce":26611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":26611},{"type":"v1","addr":"172.21.15.107:6805","nonce":26611}]},"public_addr":"172.21.15.107:6801/26611","cluster_addr":"172.21.15.107:6803/26611","heartbeat_back_addr":"172.21.15.107:6807/26611","heartbeat_front_addr":"172.21.15.107:6805/26611","state":["exists","up"]},{"osd":2,"uuid":"e15da6b7-dc79-4185-a5f9-0f443d8b52b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":26613},{"type":"v1","addr":"172.21.15.107:6820","nonce":26613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":26613},{"type":"v1","addr":"172.21.15.107:6824","nonce":26613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":26613},{"type":"v1","addr":"172.21.15.107:6831","nonce":26613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":26613},{"type":"v1","addr":"172.21.15.107:6828","nonce":26613}]},"public_addr":"172.21.15.107:6820/26613","cluster_addr":"172.21.15.107:6824/26613","heartbeat_back_addr":"172.21.15.107:6831/26613","heartbeat_front_addr":"172.21.15.107:6828/26613","state":["exists","up"]},{"osd":3,"uuid":"843e0dd3-5ce4-46b9-9004-69849048d2bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":26612},{"type":"v1","addr":"172.21.15.107:6809","nonce":26612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":26612},{"type":"v1","addr":"172.21.15.107:6811","nonce":26612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":26612},{"type":"v1","addr":"172.21.15.107:6815","nonce":26612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":26612},{"type":"v1","addr":"172.21.15.107:6813","nonce":26612}]},"public_addr":"172.21.15.107:6809/26612","cluster_addr":"172.21.15.107:6811/26612","heartbeat_back_addr":"172.21.15.107:6815/26612","heartbeat_front_addr":"172.21.15.107:6813/26612","state":["exists","up"]},{"osd":4,"uuid":"c066ab09-e031-4b49-9df3-d54cd9f9b580","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":26685},{"type":"v1","addr":"172.21.15.188:6820","nonce":26685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":26685},{"type":"v1","addr":"172.21.15.188:6823","nonce":26685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":26685},{"type":"v1","addr":"172.21.15.188:6832","nonce":26685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6825","nonce":26685},{"type":"v1","addr":"172.21.15.188:6828","nonce":26685}]},"public_addr":"172.21.15.188:6820/26685","cluster_addr":"172.21.15.188:6823/26685","heartbeat_back_addr":"172.21.15.188:6832/26685","heartbeat_front_addr":"172.21.15.188:6828/26685","state":["exists","up"]},{"osd":5,"uuid":"56a24e45-1b1b-4a65-b4b4-57cfab296f65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6819","nonce":26684},{"type":"v1","addr":"172.21.15.188:6821","nonce":26684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":26684},{"type":"v1","addr":"172.21.15.188:6826","nonce":26684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6831","nonce":26684},{"type":"v1","addr":"172.21.15.188:6833","nonce":26684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6827","nonce":26684},{"type":"v1","addr":"172.21.15.188:6829","nonce":26684}]},"public_addr":"172.21.15.188:6821/26684","cluster_addr":"172.21.15.188:6826/26684","heartbeat_back_addr":"172.21.15.188:6833/26684","heartbeat_front_addr":"172.21.15.188:6829/26684","state":["exists","up"]},{"osd":6,"uuid":"f207a2a8-b3c7-4fd8-ae90-4c2d7a5c1c38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":26682},{"type":"v1","addr":"172.21.15.188:6811","nonce":26682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":26682},{"type":"v1","addr":"172.21.15.188:6813","nonce":26682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":26682},{"type":"v1","addr":"172.21.15.188:6817","nonce":26682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":26682},{"type":"v1","addr":"172.21.15.188:6815","nonce":26682}]},"public_addr":"172.21.15.188:6811/26682","cluster_addr":"172.21.15.188:6813/26682","heartbeat_back_addr":"172.21.15.188:6817/26682","heartbeat_front_addr":"172.21.15.188:6815/26682","state":["exists","up"]},{"osd":7,"uuid":"4e5cc44f-af17-47c9-973f-8e87d1bd98db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":19,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":26683},{"type":"v1","addr":"172.21.15.188:6803","nonce":26683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":26683},{"type":"v1","addr":"172.21.15.188:6805","nonce":26683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":26683},{"type":"v1","addr":"172.21.15.188:6809","nonce":26683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":26683},{"type":"v1","addr":"172.21.15.188:6807","nonce":26683}]},"public_addr":"172.21.15.188:6803/26683","cluster_addr":"172.21.15.188:6805/26683","heartbeat_back_addr":"172.21.15.188:6809/26683","heartbeat_front_addr":"172.21.15.188:6807/26683","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:37:09.019033+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:37:09.092469+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:37:09.056721+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:37:09.006828+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:37:08.775848+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:37:09.087140+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:37:09.034074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:37:09.033458+0000","dead_epoch":19}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-06-12T02:37:30.451 INFO:tasks.rados.rados.0.smithi107.stdout:112: writing smithi10729407-1 from 2433024 to 3096576 tid 2 2020-06-12T02:37:30.453 INFO:tasks.rados.rados.0.smithi107.stdout:112: writing smithi10729407-1 from 3096576 to 3547136 tid 3 2020-06-12T02:37:30.454 INFO:tasks.rados.rados.0.smithi107.stdout:112: writing smithi10729407-1 from 3547136 to 3637248 tid 4 2020-06-12T02:37:30.454 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:37:30.456 INFO:tasks.rados.rados.0.smithi107.stdout:113: seq_num 112 ranges {0=679936,679936=548864,1228800=655360} 2020-06-12T02:37:30.457 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-06-12T02:37:30.459 INFO:tasks.rados.rados.0.smithi107.stdout:113: writing smithi10729407-18 from 0 to 679936 tid 1 2020-06-12T02:37:30.462 INFO:tasks.rados.rados.0.smithi107.stdout:113: writing smithi10729407-18 from 679936 to 1228800 tid 2 2020-06-12T02:37:30.479 INFO:tasks.ceph.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-06-12T02:37:03.610773+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-06-12T02:37:14.831188+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'unique_pool_0', 'create_time': '2020-06-12T02:37:25.893619+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 3, 'size': 3, 'min_size': 2, 'crush_rule': 1, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'teuthologyprofile', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': True, 'options': {}, 'application_metadata': {'rados': {}}}] 2020-06-12T02:37:30.480 INFO:teuthology.orchestra.run.smithi188:> CEPH_ARGS='--filestore-merge-threshold 1 --filestore-split-multiple 1' sudo -E ceph-objectstore-tool --no-mon-config --log-file=/var/log/ceph/objectstore_tool.$pid.log --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op apply-layout-settings --pool device_health_metrics 2020-06-12T02:37:30.481 INFO:tasks.rados.rados.0.smithi107.stdout:113: writing smithi10729407-18 from 1228800 to 1884160 tid 3 2020-06-12T02:37:30.481 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:30.482 INFO:tasks.rados.rados.0.smithi107.stdout:114: seq_num 113 ranges {3350528=696320,4046848=434176,4481024=466944,4947968=303104} 2020-06-12T02:37:30.488 INFO:tasks.rados.rados.0.smithi107.stdout:114: writing smithi10729407-2 from 3350528 to 4046848 tid 1 2020-06-12T02:37:30.491 INFO:tasks.rados.rados.0.smithi107.stdout:114: writing smithi10729407-2 from 4046848 to 4481024 tid 2 2020-06-12T02:37:30.494 INFO:tasks.rados.rados.0.smithi107.stdout:114: writing smithi10729407-2 from 4481024 to 4947968 tid 3 2020-06-12T02:37:30.497 INFO:tasks.rados.rados.0.smithi107.stdout:114: writing smithi10729407-2 from 4947968 to 5251072 tid 4 2020-06-12T02:37:30.497 INFO:tasks.rados.rados.0.smithi107.stdout:115: delete oid 20 current snap is 0 2020-06-12T02:37:30.498 INFO:tasks.rados.rados.0.smithi107.stdout:109: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:30.498 INFO:tasks.rados.rados.0.smithi107.stdout:108: finishing write tid 7 to smithi10729407-9 2020-06-12T02:37:30.498 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 15 (ObjNum 107 snap 0 seq_num 107) dirty exists 2020-06-12T02:37:30.498 INFO:tasks.rados.rados.0.smithi107.stdout:108: left oid 9 (ObjNum 107 snap 0 seq_num 107) 2020-06-12T02:37:30.498 INFO:tasks.rados.rados.0.smithi107.stdout:109: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:30.499 INFO:tasks.rados.rados.0.smithi107.stdout:109: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:30.499 INFO:tasks.rados.rados.0.smithi107.stdout:109: finishing write tid 5 to smithi10729407-14 2020-06-12T02:37:30.499 INFO:tasks.rados.rados.0.smithi107.stdout:109: finishing write tid 6 to smithi10729407-14 2020-06-12T02:37:30.499 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 13 (ObjNum 108 snap 0 seq_num 108) dirty exists 2020-06-12T02:37:30.499 INFO:tasks.rados.rados.0.smithi107.stdout:109: left oid 14 (ObjNum 108 snap 0 seq_num 108) 2020-06-12T02:37:30.500 INFO:tasks.rados.rados.0.smithi107.stdout:112: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:30.500 INFO:tasks.rados.rados.0.smithi107.stdout:112: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:30.500 INFO:tasks.rados.rados.0.smithi107.stdout:113: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:30.500 INFO:tasks.rados.rados.0.smithi107.stdout:112: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:30.500 INFO:tasks.rados.rados.0.smithi107.stdout:112: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:30.500 INFO:tasks.rados.rados.0.smithi107.stdout:112: finishing write tid 6 to smithi10729407-1 2020-06-12T02:37:30.501 INFO:tasks.rados.rados.0.smithi107.stdout:112: finishing write tid 7 to smithi10729407-1 2020-06-12T02:37:30.501 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 30 (ObjNum 111 snap 0 seq_num 111) dirty exists 2020-06-12T02:37:30.501 INFO:tasks.rados.rados.0.smithi107.stdout:112: left oid 1 (ObjNum 111 snap 0 seq_num 111) 2020-06-12T02:37:30.501 INFO:tasks.rados.rados.0.smithi107.stdout:113: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:30.501 INFO:tasks.rados.rados.0.smithi107.stdout:113: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:30.502 INFO:tasks.rados.rados.0.smithi107.stdout:113: finishing write tid 5 to smithi10729407-18 2020-06-12T02:37:30.502 INFO:tasks.rados.rados.0.smithi107.stdout:113: finishing write tid 6 to smithi10729407-18 2020-06-12T02:37:30.502 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 19 (ObjNum 112 snap 0 seq_num 112) dirty exists 2020-06-12T02:37:30.502 INFO:tasks.rados.rados.0.smithi107.stdout:113: left oid 18 (ObjNum 112 snap 0 seq_num 112) 2020-06-12T02:37:30.502 INFO:tasks.rados.rados.0.smithi107.stdout:114: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:30.503 INFO:tasks.rados.rados.0.smithi107.stdout:108: done (5 left) 2020-06-12T02:37:30.504 INFO:tasks.rados.rados.0.smithi107.stdout:109: done (4 left) 2020-06-12T02:37:30.504 INFO:tasks.rados.rados.0.smithi107.stdout:112: done (3 left) 2020-06-12T02:37:30.504 INFO:tasks.rados.rados.0.smithi107.stdout:113: done (2 left) 2020-06-12T02:37:30.504 INFO:tasks.rados.rados.0.smithi107.stdout:115: done (1 left) 2020-06-12T02:37:30.505 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:37:30.505 INFO:tasks.rados.rados.0.smithi107.stdout:116: seq_num 115 ranges {0=729088,729088=647168,1376256=434176,1810432=122880} 2020-06-12T02:37:30.505 INFO:tasks.rados.rados.0.smithi107.stdout:116: writing smithi10729407-5 from 0 to 729088 tid 1 2020-06-12T02:37:30.510 INFO:tasks.rados.rados.0.smithi107.stdout:116: writing smithi10729407-5 from 729088 to 1376256 tid 2 2020-06-12T02:37:30.510 INFO:tasks.rados.rados.0.smithi107.stdout:116: writing smithi10729407-5 from 1376256 to 1810432 tid 3 2020-06-12T02:37:30.511 INFO:tasks.rados.rados.0.smithi107.stdout:116: writing smithi10729407-5 from 1810432 to 1933312 tid 4 2020-06-12T02:37:30.511 INFO:tasks.rados.rados.0.smithi107.stdout:117: delete oid 8 current snap is 0 2020-06-12T02:37:30.511 INFO:tasks.rados.rados.0.smithi107.stdout:114: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:30.511 INFO:tasks.rados.rados.0.smithi107.stdout:114: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:30.511 INFO:tasks.rados.rados.0.smithi107.stdout:114: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:30.512 INFO:tasks.rados.rados.0.smithi107.stdout:114: finishing write tid 6 to smithi10729407-2 2020-06-12T02:37:30.512 INFO:tasks.rados.rados.0.smithi107.stdout:114: finishing write tid 7 to smithi10729407-2 2020-06-12T02:37:30.512 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 19 (ObjNum 113 snap 0 seq_num 113) dirty exists 2020-06-12T02:37:30.512 INFO:tasks.rados.rados.0.smithi107.stdout:114: left oid 2 (ObjNum 113 snap 0 seq_num 113) 2020-06-12T02:37:30.513 INFO:tasks.rados.rados.0.smithi107.stdout:114: done (2 left) 2020-06-12T02:37:30.513 INFO:tasks.rados.rados.0.smithi107.stdout:117: done (1 left) 2020-06-12T02:37:30.514 INFO:tasks.rados.rados.0.smithi107.stdout:118: delete oid 2 current snap is 0 2020-06-12T02:37:30.514 INFO:tasks.rados.rados.0.smithi107.stdout:118: done (1 left) 2020-06-12T02:37:30.514 INFO:tasks.rados.rados.0.smithi107.stdout:119: delete oid 17 current snap is 0 2020-06-12T02:37:30.515 INFO:tasks.rados.rados.0.smithi107.stdout:116: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:30.515 INFO:tasks.rados.rados.0.smithi107.stdout:119: done (1 left) 2020-06-12T02:37:30.516 INFO:tasks.rados.rados.0.smithi107.stdout:120: delete oid 11 current snap is 0 2020-06-12T02:37:30.516 INFO:tasks.rados.rados.0.smithi107.stdout:116: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:30.517 INFO:tasks.rados.rados.0.smithi107.stdout:120: done (1 left) 2020-06-12T02:37:30.518 INFO:tasks.rados.rados.0.smithi107.stdout:121: delete oid 4 current snap is 0 2020-06-12T02:37:30.519 INFO:tasks.rados.rados.0.smithi107.stdout:116: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:30.519 INFO:tasks.rados.rados.0.smithi107.stdout:116: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:30.519 INFO:tasks.rados.rados.0.smithi107.stdout:116: finishing write tid 6 to smithi10729407-5 2020-06-12T02:37:30.520 INFO:tasks.rados.rados.0.smithi107.stdout:121: done (1 left) 2020-06-12T02:37:30.520 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:37:30.520 INFO:tasks.rados.rados.0.smithi107.stdout:122: seq_num 121 ranges {1884160=434176,2318336=753664,3072000=425984,3497984=425984} 2020-06-12T02:37:30.524 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:37:30.521+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:37:30.532 INFO:tasks.rados.rados.0.smithi107.stdout:122: writing smithi10729407-18 from 1884160 to 2318336 tid 1 2020-06-12T02:37:30.536 INFO:tasks.rados.rados.0.smithi107.stdout:122: writing smithi10729407-18 from 2318336 to 3072000 tid 2 2020-06-12T02:37:30.539 INFO:tasks.rados.rados.0.smithi107.stdout:122: writing smithi10729407-18 from 3072000 to 3497984 tid 3 2020-06-12T02:37:30.541 INFO:tasks.rados.rados.0.smithi107.stdout:122: writing smithi10729407-18 from 3497984 to 3923968 tid 4 2020-06-12T02:37:30.542 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:37:30.542 INFO:tasks.rados.rados.0.smithi107.stdout:123: seq_num 122 ranges {0=450560,450560=688128,1138688=630784,1769472=286720} 2020-06-12T02:37:30.544 INFO:tasks.rados.rados.0.smithi107.stdout:123: writing smithi10729407-13 from 0 to 450560 tid 1 2020-06-12T02:37:30.549 INFO:tasks.rados.rados.0.smithi107.stdout:123: writing smithi10729407-13 from 450560 to 1138688 tid 2 2020-06-12T02:37:30.553 INFO:tasks.rados.rados.0.smithi107.stdout:123: writing smithi10729407-13 from 1138688 to 1769472 tid 3 2020-06-12T02:37:30.555 INFO:tasks.rados.rados.0.smithi107.stdout:123: writing smithi10729407-13 from 1769472 to 2056192 tid 4 2020-06-12T02:37:30.556 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:37:30.556 INFO:tasks.rados.rados.0.smithi107.stdout:124: seq_num 123 ranges {0=466944,466944=573440,1040384=794624,1835008=237568} 2020-06-12T02:37:30.558 INFO:tasks.rados.rados.0.smithi107.stdout:124: writing smithi10729407-15 from 0 to 466944 tid 1 2020-06-12T02:37:30.562 INFO:tasks.rados.rados.0.smithi107.stdout:124: writing smithi10729407-15 from 466944 to 1040384 tid 2 2020-06-12T02:37:30.566 INFO:tasks.rados.rados.0.smithi107.stdout:124: writing smithi10729407-15 from 1040384 to 1835008 tid 3 2020-06-12T02:37:30.571 INFO:tasks.rados.rados.0.smithi107.stdout:124: writing smithi10729407-15 from 1835008 to 2072576 tid 4 2020-06-12T02:37:30.571 INFO:tasks.rados.rados.0.smithi107.stdout:125: delete oid 8 current snap is 0 2020-06-12T02:37:30.572 INFO:tasks.rados.rados.0.smithi107.stdout:116: finishing write tid 7 to smithi10729407-5 2020-06-12T02:37:30.572 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 13 (ObjNum 115 snap 0 seq_num 115) dirty exists 2020-06-12T02:37:30.572 INFO:tasks.rados.rados.0.smithi107.stdout:116: left oid 5 (ObjNum 115 snap 0 seq_num 115) 2020-06-12T02:37:30.572 INFO:tasks.rados.rados.0.smithi107.stdout:122: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:30.572 INFO:tasks.rados.rados.0.smithi107.stdout:122: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:30.573 INFO:tasks.rados.rados.0.smithi107.stdout:122: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:30.573 INFO:tasks.rados.rados.0.smithi107.stdout:122: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:30.573 INFO:tasks.rados.rados.0.smithi107.stdout:122: finishing write tid 6 to smithi10729407-18 2020-06-12T02:37:30.573 INFO:tasks.rados.rados.0.smithi107.stdout:122: finishing write tid 7 to smithi10729407-18 2020-06-12T02:37:30.573 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 24 (ObjNum 121 snap 0 seq_num 121) dirty exists 2020-06-12T02:37:30.574 INFO:tasks.rados.rados.0.smithi107.stdout:122: left oid 18 (ObjNum 121 snap 0 seq_num 121) 2020-06-12T02:37:30.574 INFO:tasks.rados.rados.0.smithi107.stdout:123: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:30.574 INFO:tasks.rados.rados.0.smithi107.stdout:123: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:30.574 INFO:tasks.rados.rados.0.smithi107.stdout:123: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:30.574 INFO:tasks.rados.rados.0.smithi107.stdout:123: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:30.575 INFO:tasks.rados.rados.0.smithi107.stdout:124: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:30.575 INFO:tasks.rados.rados.0.smithi107.stdout:123: finishing write tid 6 to smithi10729407-13 2020-06-12T02:37:30.575 INFO:tasks.rados.rados.0.smithi107.stdout:123: finishing write tid 7 to smithi10729407-13 2020-06-12T02:37:30.575 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 27 (ObjNum 122 snap 0 seq_num 122) dirty exists 2020-06-12T02:37:30.575 INFO:tasks.rados.rados.0.smithi107.stdout:123: left oid 13 (ObjNum 122 snap 0 seq_num 122) 2020-06-12T02:37:30.578 INFO:tasks.rados.rados.0.smithi107.stdout:124: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:30.579 INFO:tasks.rados.rados.0.smithi107.stdout:124: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:30.579 INFO:tasks.rados.rados.0.smithi107.stdout:124: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:30.579 INFO:tasks.rados.rados.0.smithi107.stdout:124: finishing write tid 6 to smithi10729407-15 2020-06-12T02:37:30.581 INFO:tasks.rados.rados.0.smithi107.stdout:124: finishing write tid 7 to smithi10729407-15 2020-06-12T02:37:30.581 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 34 (ObjNum 123 snap 0 seq_num 123) dirty exists 2020-06-12T02:37:30.581 INFO:tasks.rados.rados.0.smithi107.stdout:124: left oid 15 (ObjNum 123 snap 0 seq_num 123) 2020-06-12T02:37:30.582 INFO:tasks.rados.rados.0.smithi107.stdout:116: done (4 left) 2020-06-12T02:37:30.582 INFO:tasks.rados.rados.0.smithi107.stdout:122: done (3 left) 2020-06-12T02:37:30.582 INFO:tasks.rados.rados.0.smithi107.stdout:123: done (2 left) 2020-06-12T02:37:30.582 INFO:tasks.rados.rados.0.smithi107.stdout:124: done (1 left) 2020-06-12T02:37:30.582 INFO:tasks.rados.rados.0.smithi107.stdout:125: done (0 left) 2020-06-12T02:37:30.583 INFO:tasks.rados.rados.0.smithi107.stdout:126: delete oid 9 current snap is 0 2020-06-12T02:37:30.583 INFO:tasks.rados.rados.0.smithi107.stdout:126: done (0 left) 2020-06-12T02:37:30.583 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:37:30.584 INFO:tasks.rados.rados.0.smithi107.stdout:127: seq_num 126 ranges {0=516096,516096=630784,1146880=614400,1761280=360448} 2020-06-12T02:37:30.586 INFO:tasks.rados.rados.0.smithi107.stdout:127: writing smithi10729407-20 from 0 to 516096 tid 1 2020-06-12T02:37:30.590 INFO:tasks.rados.rados.0.smithi107.stdout:127: writing smithi10729407-20 from 516096 to 1146880 tid 2 2020-06-12T02:37:30.593 INFO:tasks.rados.rados.0.smithi107.stdout:127: writing smithi10729407-20 from 1146880 to 1761280 tid 3 2020-06-12T02:37:30.595 INFO:tasks.rados.rados.0.smithi107.stdout:127: writing smithi10729407-20 from 1761280 to 2121728 tid 4 2020-06-12T02:37:30.596 INFO:tasks.rados.rados.0.smithi107.stdout:128: delete oid 11 current snap is 0 2020-06-12T02:37:30.596 INFO:tasks.rados.rados.0.smithi107.stdout:127: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:30.597 INFO:tasks.rados.rados.0.smithi107.stdout:128: done (1 left) 2020-06-12T02:37:30.597 INFO:tasks.rados.rados.0.smithi107.stdout:129: delete oid 16 current snap is 0 2020-06-12T02:37:30.599 INFO:tasks.rados.rados.0.smithi107.stdout:129: done (1 left) 2020-06-12T02:37:30.599 INFO:tasks.rados.rados.0.smithi107.stdout:130: delete oid 1 current snap is 0 2020-06-12T02:37:30.603 INFO:tasks.rados.rados.0.smithi107.stdout:130: done (1 left) 2020-06-12T02:37:30.604 INFO:tasks.rados.rados.0.smithi107.stdout:131: delete oid 7 current snap is 0 2020-06-12T02:37:30.605 INFO:tasks.rados.rados.0.smithi107.stdout:131: done (1 left) 2020-06-12T02:37:30.605 INFO:tasks.rados.rados.0.smithi107.stdout:132: delete oid 5 current snap is 0 2020-06-12T02:37:30.607 INFO:tasks.rados.rados.0.smithi107.stdout:127: finishing write tid 2 to smithi10729407-20 2020-06-12T02:37:30.612 INFO:tasks.rados.rados.0.smithi107.stdout:132: done (1 left) 2020-06-12T02:37:30.612 INFO:tasks.rados.rados.0.smithi107.stdout:133: delete oid 17 current snap is 0 2020-06-12T02:37:30.612 INFO:tasks.rados.rados.0.smithi107.stdout:127: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:30.614 INFO:tasks.rados.rados.0.smithi107.stdout:133: done (1 left) 2020-06-12T02:37:30.614 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:37:30.614 INFO:tasks.rados.rados.0.smithi107.stdout:134: seq_num 133 ranges {0=630784,630784=663552,1294336=770048,2064384=172032} 2020-06-12T02:37:30.617 INFO:tasks.rados.rados.0.smithi107.stdout:134: writing smithi10729407-10 from 0 to 630784 tid 1 2020-06-12T02:37:30.621 INFO:tasks.rados.rados.0.smithi107.stdout:134: writing smithi10729407-10 from 630784 to 1294336 tid 2 2020-06-12T02:37:30.625 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:30.623+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:30.626 INFO:tasks.rados.rados.0.smithi107.stdout:134: writing smithi10729407-10 from 1294336 to 2064384 tid 3 2020-06-12T02:37:30.627 INFO:tasks.rados.rados.0.smithi107.stdout:134: writing smithi10729407-10 from 2064384 to 2236416 tid 4 2020-06-12T02:37:30.628 INFO:tasks.rados.rados.0.smithi107.stdout:135: delete oid 4 current snap is 0 2020-06-12T02:37:30.628 INFO:tasks.rados.rados.0.smithi107.stdout:127: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:30.628 INFO:tasks.rados.rados.0.smithi107.stdout:127: finishing write tid 6 to smithi10729407-20 2020-06-12T02:37:30.628 INFO:tasks.rados.rados.0.smithi107.stdout:127: finishing write tid 7 to smithi10729407-20 2020-06-12T02:37:30.628 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 9 (ObjNum 126 snap 0 seq_num 126) dirty exists 2020-06-12T02:37:30.629 INFO:tasks.rados.rados.0.smithi107.stdout:127: left oid 20 (ObjNum 126 snap 0 seq_num 126) 2020-06-12T02:37:30.629 INFO:tasks.rados.rados.0.smithi107.stdout:134: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:30.635 INFO:tasks.rados.rados.0.smithi107.stdout:127: done (2 left) 2020-06-12T02:37:30.635 INFO:tasks.rados.rados.0.smithi107.stdout:135: done (1 left) 2020-06-12T02:37:30.635 INFO:tasks.rados.rados.0.smithi107.stdout:136: delete oid 5 current snap is 0 2020-06-12T02:37:30.636 INFO:tasks.rados.rados.0.smithi107.stdout:136: done (1 left) 2020-06-12T02:37:30.636 INFO:tasks.rados.rados.0.smithi107.stdout:137: delete oid 13 current snap is 0 2020-06-12T02:37:30.636 INFO:tasks.rados.rados.0.smithi107.stdout:137: done (1 left) 2020-06-12T02:37:30.636 INFO:tasks.rados.rados.0.smithi107.stdout:138: delete oid 7 current snap is 0 2020-06-12T02:37:30.637 INFO:tasks.rados.rados.0.smithi107.stdout:134: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:30.637 INFO:tasks.rados.rados.0.smithi107.stdout:134: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:30.638 INFO:tasks.rados.rados.0.smithi107.stdout:138: done (1 left) 2020-06-12T02:37:30.639 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:37:30.639 INFO:tasks.rados.rados.0.smithi107.stdout:139: seq_num 138 ranges {0=720896,720896=540672,1261568=557056,1818624=507904} 2020-06-12T02:37:30.642 INFO:tasks.rados.rados.0.smithi107.stdout:139: writing smithi10729407-4 from 0 to 720896 tid 1 2020-06-12T02:37:30.646 INFO:tasks.rados.rados.0.smithi107.stdout:139: writing smithi10729407-4 from 720896 to 1261568 tid 2 2020-06-12T02:37:30.649 INFO:tasks.rados.rados.0.smithi107.stdout:139: writing smithi10729407-4 from 1261568 to 1818624 tid 3 2020-06-12T02:37:30.658 INFO:tasks.rados.rados.0.smithi107.stdout:139: writing smithi10729407-4 from 1818624 to 2326528 tid 4 2020-06-12T02:37:30.659 INFO:tasks.rados.rados.0.smithi107.stdout:140: delete oid 11 current snap is 0 2020-06-12T02:37:30.659 INFO:tasks.rados.rados.0.smithi107.stdout:134: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:30.659 INFO:tasks.rados.rados.0.smithi107.stdout:134: finishing write tid 6 to smithi10729407-10 2020-06-12T02:37:30.659 INFO:tasks.rados.rados.0.smithi107.stdout:134: finishing write tid 7 to smithi10729407-10 2020-06-12T02:37:30.660 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 20 (ObjNum 133 snap 0 seq_num 133) dirty exists 2020-06-12T02:37:30.660 INFO:tasks.rados.rados.0.smithi107.stdout:134: left oid 10 (ObjNum 133 snap 0 seq_num 133) 2020-06-12T02:37:30.660 INFO:tasks.rados.rados.0.smithi107.stdout:139: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:30.660 INFO:tasks.rados.rados.0.smithi107.stdout:134: done (2 left) 2020-06-12T02:37:30.660 INFO:tasks.rados.rados.0.smithi107.stdout:140: done (1 left) 2020-06-12T02:37:30.661 INFO:tasks.rados.rados.0.smithi107.stdout:141: delete oid 10 current snap is 0 2020-06-12T02:37:30.661 INFO:tasks.rados.rados.0.smithi107.stdout:141: done (1 left) 2020-06-12T02:37:30.661 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:37:30.661 INFO:tasks.rados.rados.0.smithi107.stdout:142: seq_num 141 ranges {1531904=770048,2301952=589824,2891776=729088,3620864=286720} 2020-06-12T02:37:30.662 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:30.662 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:30.667 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-06-12T02:37:30.668 INFO:tasks.rados.rados.0.smithi107.stdout:142: writing smithi10729407-19 from 1531904 to 2301952 tid 1 2020-06-12T02:37:30.672 INFO:tasks.rados.rados.0.smithi107.stdout:142: writing smithi10729407-19 from 2301952 to 2891776 tid 2 2020-06-12T02:37:30.676 INFO:tasks.rados.rados.0.smithi107.stdout:142: writing smithi10729407-19 from 2891776 to 3620864 tid 3 2020-06-12T02:37:30.678 INFO:tasks.rados.rados.0.smithi107.stdout:142: writing smithi10729407-19 from 3620864 to 3907584 tid 4 2020-06-12T02:37:30.678 INFO:tasks.rados.rados.0.smithi107.stdout:143: delete oid 13 current snap is 0 2020-06-12T02:37:30.678 INFO:tasks.rados.rados.0.smithi107.stdout:139: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:30.678 INFO:tasks.rados.rados.0.smithi107.stdout:139: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:30.679 INFO:tasks.rados.rados.0.smithi107.stdout:139: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:30.679 INFO:tasks.rados.rados.0.smithi107.stdout:139: finishing write tid 6 to smithi10729407-4 2020-06-12T02:37:30.679 INFO:tasks.rados.rados.0.smithi107.stdout:139: finishing write tid 7 to smithi10729407-4 2020-06-12T02:37:30.679 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 32 (ObjNum 138 snap 0 seq_num 138) dirty exists 2020-06-12T02:37:30.679 INFO:tasks.rados.rados.0.smithi107.stdout:139: left oid 4 (ObjNum 138 snap 0 seq_num 138) 2020-06-12T02:37:30.680 INFO:tasks.rados.rados.0.smithi107.stdout:142: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:30.680 INFO:tasks.rados.rados.0.smithi107.stdout:142: finishing write tid 2 to smithi10729407-19 2020-06-12T02:37:30.681 INFO:tasks.rados.rados.0.smithi107.stdout:139: done (2 left) 2020-06-12T02:37:30.681 INFO:tasks.rados.rados.0.smithi107.stdout:143: done (1 left) 2020-06-12T02:37:30.682 INFO:tasks.rados.rados.0.smithi107.stdout:144: delete oid 6 current snap is 0 2020-06-12T02:37:30.683 INFO:tasks.rados.rados.0.smithi107.stdout:144: done (1 left) 2020-06-12T02:37:30.683 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:37:30.683 INFO:tasks.rados.rados.0.smithi107.stdout:145: seq_num 144 ranges {0=16384} 2020-06-12T02:37:30.683 INFO:tasks.rados.rados.0.smithi107.stdout:145: writing smithi10729407-16 from 0 to 16384 tid 1 2020-06-12T02:37:30.683 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:37:30.684 INFO:tasks.rados.rados.0.smithi107.stdout:146: seq_num 145 ranges {0=32768} 2020-06-12T02:37:30.684 INFO:tasks.rados.rados.0.smithi107.stdout:146: writing smithi10729407-1 from 0 to 32768 tid 1 2020-06-12T02:37:30.684 INFO:tasks.rados.rados.0.smithi107.stdout:147: delete oid 2 current snap is 0 2020-06-12T02:37:30.684 INFO:tasks.rados.rados.0.smithi107.stdout:142: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:30.691 INFO:tasks.rados.rados.0.smithi107.stdout:147: done (3 left) 2020-06-12T02:37:30.691 INFO:tasks.rados.rados.0.smithi107.stdout:148: delete oid 7 current snap is 0 2020-06-12T02:37:30.692 INFO:tasks.rados.rados.0.smithi107.stdout:142: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:30.692 INFO:tasks.rados.rados.0.smithi107.stdout:142: finishing write tid 6 to smithi10729407-19 2020-06-12T02:37:30.692 INFO:tasks.rados.rados.0.smithi107.stdout:145: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:30.692 INFO:tasks.rados.rados.0.smithi107.stdout:142: finishing write tid 7 to smithi10729407-19 2020-06-12T02:37:30.692 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 21 (ObjNum 141 snap 0 seq_num 141) dirty exists 2020-06-12T02:37:30.693 INFO:tasks.rados.rados.0.smithi107.stdout:142: left oid 19 (ObjNum 141 snap 0 seq_num 141) 2020-06-12T02:37:30.693 INFO:tasks.rados.rados.0.smithi107.stdout:145: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:30.693 INFO:tasks.rados.rados.0.smithi107.stdout:146: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:30.693 INFO:tasks.rados.rados.0.smithi107.stdout:146: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:30.693 INFO:tasks.rados.rados.0.smithi107.stdout:142: done (3 left) 2020-06-12T02:37:30.694 INFO:tasks.rados.rados.0.smithi107.stdout:148: done (2 left) 2020-06-12T02:37:30.694 INFO:tasks.rados.rados.0.smithi107.stdout:149: delete oid 10 current snap is 0 2020-06-12T02:37:30.694 INFO:tasks.rados.rados.0.smithi107.stdout:145: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:30.694 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 23 (ObjNum 144 snap 0 seq_num 144) dirty exists 2020-06-12T02:37:30.694 INFO:tasks.rados.rados.0.smithi107.stdout:145: left oid 16 (ObjNum 144 snap 0 seq_num 144) 2020-06-12T02:37:30.694 INFO:tasks.rados.rados.0.smithi107.stdout:146: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:30.695 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 34 (ObjNum 145 snap 0 seq_num 145) dirty exists 2020-06-12T02:37:30.695 INFO:tasks.rados.rados.0.smithi107.stdout:146: left oid 1 (ObjNum 145 snap 0 seq_num 145) 2020-06-12T02:37:30.696 INFO:tasks.rados.rados.0.smithi107.stdout:145: done (2 left) 2020-06-12T02:37:30.696 INFO:tasks.rados.rados.0.smithi107.stdout:146: done (1 left) 2020-06-12T02:37:30.696 INFO:tasks.rados.rados.0.smithi107.stdout:149: done (0 left) 2020-06-12T02:37:30.699 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:37:30.699 INFO:tasks.rados.rados.0.smithi107.stdout:150: seq_num 149 ranges {0=98304} 2020-06-12T02:37:30.699 INFO:tasks.rados.rados.0.smithi107.stdout:150: writing smithi10729407-3 from 0 to 98304 tid 1 2020-06-12T02:37:30.700 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:37:30.701 INFO:tasks.rados.rados.0.smithi107.stdout:151: seq_num 150 ranges {0=114688} 2020-06-12T02:37:30.701 INFO:tasks.rados.rados.0.smithi107.stdout:151: writing smithi10729407-5 from 0 to 114688 tid 1 2020-06-12T02:37:30.701 INFO:tasks.rados.rados.0.smithi107.stdout:152: delete oid 4 current snap is 0 2020-06-12T02:37:30.702 INFO:tasks.rados.rados.0.smithi107.stdout:150: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:30.703 INFO:tasks.rados.rados.0.smithi107.stdout:151: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:30.703 INFO:tasks.rados.rados.0.smithi107.stdout:151: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:30.703 INFO:tasks.rados.rados.0.smithi107.stdout:152: done (2 left) 2020-06-12T02:37:30.703 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:37:30.704 INFO:tasks.rados.rados.0.smithi107.stdout:153: seq_num 152 ranges {3907584=147456} 2020-06-12T02:37:30.719 INFO:tasks.rados.rados.0.smithi107.stdout:153: writing smithi10729407-19 from 3907584 to 4055040 tid 1 2020-06-12T02:37:30.719 INFO:tasks.rados.rados.0.smithi107.stdout:154: delete oid 13 current snap is 0 2020-06-12T02:37:30.720 INFO:tasks.rados.rados.0.smithi107.stdout:151: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:30.720 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 17 (ObjNum 150 snap 0 seq_num 150) dirty exists 2020-06-12T02:37:30.720 INFO:tasks.rados.rados.0.smithi107.stdout:151: left oid 5 (ObjNum 150 snap 0 seq_num 150) 2020-06-12T02:37:30.720 INFO:tasks.rados.rados.0.smithi107.stdout:150: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:30.721 INFO:tasks.rados.rados.0.smithi107.stdout:150: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:30.721 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 25 (ObjNum 149 snap 0 seq_num 149) dirty exists 2020-06-12T02:37:30.721 INFO:tasks.rados.rados.0.smithi107.stdout:150: left oid 3 (ObjNum 149 snap 0 seq_num 149) 2020-06-12T02:37:30.722 INFO:tasks.rados.rados.0.smithi107.stdout:150: done (3 left) 2020-06-12T02:37:30.722 INFO:tasks.rados.rados.0.smithi107.stdout:151: done (2 left) 2020-06-12T02:37:30.722 INFO:tasks.rados.rados.0.smithi107.stdout:154: done (1 left) 2020-06-12T02:37:30.722 INFO:tasks.rados.rados.0.smithi107.stdout:155: delete oid 11 current snap is 0 2020-06-12T02:37:30.723 INFO:tasks.rados.rados.0.smithi107.stdout:153: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:30.723 INFO:tasks.rados.rados.0.smithi107.stdout:153: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:30.725 INFO:tasks.rados.rados.0.smithi107.stdout:153: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:30.725 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 23 (ObjNum 152 snap 0 seq_num 152) dirty exists 2020-06-12T02:37:30.725 INFO:tasks.rados.rados.0.smithi107.stdout:153: left oid 19 (ObjNum 152 snap 0 seq_num 152) 2020-06-12T02:37:30.726 INFO:tasks.rados.rados.0.smithi107.stdout:153: done (1 left) 2020-06-12T02:37:30.726 INFO:tasks.rados.rados.0.smithi107.stdout:155: done (0 left) 2020-06-12T02:37:30.727 INFO:tasks.rados.rados.0.smithi107.stdout:156: delete oid 8 current snap is 0 2020-06-12T02:37:30.728 INFO:tasks.rados.rados.0.smithi107.stdout:156: done (0 left) 2020-06-12T02:37:30.728 INFO:tasks.rados.rados.0.smithi107.stdout:157: delete oid 7 current snap is 0 2020-06-12T02:37:30.729 INFO:tasks.rados.rados.0.smithi107.stdout:157: done (0 left) 2020-06-12T02:37:30.729 INFO:tasks.rados.rados.0.smithi107.stdout:158: delete oid 7 current snap is 0 2020-06-12T02:37:30.731 INFO:tasks.rados.rados.0.smithi107.stdout:158: done (0 left) 2020-06-12T02:37:30.731 INFO:tasks.rados.rados.0.smithi107.stdout:159: delete oid 14 current snap is 0 2020-06-12T02:37:30.733 INFO:tasks.rados.rados.0.smithi107.stdout:159: done (0 left) 2020-06-12T02:37:30.733 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:37:30.733 INFO:tasks.rados.rados.0.smithi107.stdout:160: seq_num 159 ranges {0=270336} 2020-06-12T02:37:30.734 INFO:tasks.rados.rados.0.smithi107.stdout:160: writing smithi10729407-12 from 0 to 270336 tid 1 2020-06-12T02:37:30.735 INFO:tasks.rados.rados.0.smithi107.stdout:161: delete oid 2 current snap is 0 2020-06-12T02:37:30.736 INFO:tasks.rados.rados.0.smithi107.stdout:161: done (1 left) 2020-06-12T02:37:30.736 INFO:tasks.rados.rados.0.smithi107.stdout:162: delete oid 18 current snap is 0 2020-06-12T02:37:30.737 INFO:tasks.rados.rados.0.smithi107.stdout:160: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:30.740 INFO:tasks.rados.rados.0.smithi107.stdout:162: done (1 left) 2020-06-12T02:37:30.740 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:37:30.740 INFO:tasks.rados.rados.0.smithi107.stdout:163: seq_num 162 ranges {0=319488} 2020-06-12T02:37:30.742 INFO:tasks.rados.rados.0.smithi107.stdout:163: writing smithi10729407-4 from 0 to 319488 tid 1 2020-06-12T02:37:30.742 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:37:30.743 INFO:tasks.rados.rados.0.smithi107.stdout:164: seq_num 163 ranges {16384=335872} 2020-06-12T02:37:30.745 INFO:tasks.rados.rados.0.smithi107.stdout:164: writing smithi10729407-16 from 16384 to 352256 tid 1 2020-06-12T02:37:30.745 INFO:tasks.rados.rados.0.smithi107.stdout:165: delete oid 10 current snap is 0 2020-06-12T02:37:30.746 INFO:tasks.rados.rados.0.smithi107.stdout:160: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:30.746 INFO:tasks.rados.rados.0.smithi107.stdout:160: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:30.746 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 25 (ObjNum 159 snap 0 seq_num 159) dirty exists 2020-06-12T02:37:30.746 INFO:tasks.rados.rados.0.smithi107.stdout:160: left oid 12 (ObjNum 159 snap 0 seq_num 159) 2020-06-12T02:37:30.748 INFO:tasks.rados.rados.0.smithi107.stdout:163: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:30.748 INFO:tasks.rados.rados.0.smithi107.stdout:163: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:30.748 INFO:tasks.rados.rados.0.smithi107.stdout:160: done (3 left) 2020-06-12T02:37:30.748 INFO:tasks.rados.rados.0.smithi107.stdout:165: done (2 left) 2020-06-12T02:37:30.749 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:37:30.749 INFO:tasks.rados.rados.0.smithi107.stdout:166: seq_num 165 ranges {32768=368640} 2020-06-12T02:37:30.750 INFO:tasks.rados.rados.0.smithi107.stdout:166: writing smithi10729407-1 from 32768 to 401408 tid 1 2020-06-12T02:37:30.750 INFO:tasks.rados.rados.0.smithi107.stdout:167: delete oid 3 current snap is 0 2020-06-12T02:37:30.751 INFO:tasks.rados.rados.0.smithi107.stdout:163: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:30.751 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 35 (ObjNum 162 snap 0 seq_num 162) dirty exists 2020-06-12T02:37:30.751 INFO:tasks.rados.rados.0.smithi107.stdout:163: left oid 4 (ObjNum 162 snap 0 seq_num 162) 2020-06-12T02:37:30.751 INFO:tasks.rados.rados.0.smithi107.stdout:164: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:30.751 INFO:tasks.rados.rados.0.smithi107.stdout:164: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:30.752 INFO:tasks.rados.rados.0.smithi107.stdout:164: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:30.752 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 27 (ObjNum 163 snap 0 seq_num 163) dirty exists 2020-06-12T02:37:30.752 INFO:tasks.rados.rados.0.smithi107.stdout:164: left oid 16 (ObjNum 163 snap 0 seq_num 163) 2020-06-12T02:37:30.752 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:30.750+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:30.753 INFO:tasks.rados.rados.0.smithi107.stdout:163: done (3 left) 2020-06-12T02:37:30.753 INFO:tasks.rados.rados.0.smithi107.stdout:164: done (2 left) 2020-06-12T02:37:30.754 INFO:tasks.rados.rados.0.smithi107.stdout:167: done (1 left) 2020-06-12T02:37:30.754 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:37:30.754 INFO:tasks.rados.rados.0.smithi107.stdout:168: seq_num 167 ranges {0=401408} 2020-06-12T02:37:30.755 INFO:tasks.rados.rados.0.smithi107.stdout:168: writing smithi10729407-14 from 0 to 401408 tid 1 2020-06-12T02:37:30.756 INFO:tasks.rados.rados.0.smithi107.stdout:169: delete oid 10 current snap is 0 2020-06-12T02:37:30.756 INFO:tasks.rados.rados.0.smithi107.stdout:166: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:30.758 INFO:tasks.rados.rados.0.smithi107.stdout:169: done (2 left) 2020-06-12T02:37:30.759 INFO:tasks.rados.rados.0.smithi107.stdout:170: delete oid 12 current snap is 0 2020-06-12T02:37:30.761 INFO:tasks.rados.rados.0.smithi107.stdout:170: done (2 left) 2020-06-12T02:37:30.761 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:37:30.762 INFO:tasks.rados.rados.0.smithi107.stdout:171: seq_num 170 ranges {352256=450560} 2020-06-12T02:37:30.766 INFO:tasks.rados.rados.0.smithi107.stdout:171: writing smithi10729407-16 from 352256 to 802816 tid 1 2020-06-12T02:37:30.766 INFO:tasks.rados.rados.0.smithi107.stdout:172: delete oid 4 current snap is 0 2020-06-12T02:37:30.767 INFO:tasks.rados.rados.0.smithi107.stdout:166: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:30.767 INFO:tasks.rados.rados.0.smithi107.stdout:168: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:30.767 INFO:tasks.rados.rados.0.smithi107.stdout:168: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:30.767 INFO:tasks.rados.rados.0.smithi107.stdout:166: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:30.767 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 36 (ObjNum 165 snap 0 seq_num 165) dirty exists 2020-06-12T02:37:30.768 INFO:tasks.rados.rados.0.smithi107.stdout:166: left oid 1 (ObjNum 165 snap 0 seq_num 165) 2020-06-12T02:37:30.768 INFO:tasks.rados.rados.0.smithi107.stdout:168: finishing write tid 4 to smithi10729407-14 2020-06-12T02:37:30.768 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 20 (ObjNum 167 snap 0 seq_num 167) dirty exists 2020-06-12T02:37:30.768 INFO:tasks.rados.rados.0.smithi107.stdout:168: left oid 14 (ObjNum 167 snap 0 seq_num 167) 2020-06-12T02:37:30.769 INFO:tasks.rados.rados.0.smithi107.stdout:166: done (3 left) 2020-06-12T02:37:30.769 INFO:tasks.rados.rados.0.smithi107.stdout:168: done (2 left) 2020-06-12T02:37:30.769 INFO:tasks.rados.rados.0.smithi107.stdout:172: done (1 left) 2020-06-12T02:37:30.769 INFO:tasks.rados.rados.0.smithi107.stdout:173: delete oid 9 current snap is 0 2020-06-12T02:37:30.770 INFO:tasks.rados.rados.0.smithi107.stdout:173: done (1 left) 2020-06-12T02:37:30.771 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:37:30.771 INFO:tasks.rados.rados.0.smithi107.stdout:174: seq_num 173 ranges {2072576=499712} 2020-06-12T02:37:30.783 INFO:tasks.rados.rados.0.smithi107.stdout:174: writing smithi10729407-15 from 2072576 to 2572288 tid 1 2020-06-12T02:37:30.783 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:37:30.783 INFO:tasks.rados.rados.0.smithi107.stdout:175: seq_num 174 ranges {0=516096} 2020-06-12T02:37:30.786 INFO:tasks.rados.rados.0.smithi107.stdout:175: writing smithi10729407-13 from 0 to 516096 tid 1 2020-06-12T02:37:30.786 INFO:tasks.rados.rados.0.smithi107.stdout:176: delete oid 7 current snap is 0 2020-06-12T02:37:30.787 INFO:tasks.rados.rados.0.smithi107.stdout:171: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:30.787 INFO:tasks.rados.rados.0.smithi107.stdout:171: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:30.787 INFO:tasks.rados.rados.0.smithi107.stdout:171: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:30.787 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 30 (ObjNum 170 snap 0 seq_num 170) dirty exists 2020-06-12T02:37:30.787 INFO:tasks.rados.rados.0.smithi107.stdout:171: left oid 16 (ObjNum 170 snap 0 seq_num 170) 2020-06-12T02:37:30.790 INFO:tasks.rados.rados.0.smithi107.stdout:174: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:30.791 INFO:tasks.rados.rados.0.smithi107.stdout:175: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:30.792 INFO:tasks.rados.rados.0.smithi107.stdout:171: done (3 left) 2020-06-12T02:37:30.792 INFO:tasks.rados.rados.0.smithi107.stdout:176: done (2 left) 2020-06-12T02:37:30.793 INFO:tasks.rados.rados.0.smithi107.stdout:177: delete oid 9 current snap is 0 2020-06-12T02:37:30.793 INFO:tasks.rados.rados.0.smithi107.stdout:174: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:30.793 INFO:tasks.rados.rados.0.smithi107.stdout:175: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:30.794 INFO:tasks.rados.rados.0.smithi107.stdout:174: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:30.794 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 38 (ObjNum 173 snap 0 seq_num 173) dirty exists 2020-06-12T02:37:30.794 INFO:tasks.rados.rados.0.smithi107.stdout:174: left oid 15 (ObjNum 173 snap 0 seq_num 173) 2020-06-12T02:37:30.795 INFO:tasks.rados.rados.0.smithi107.stdout:175: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:30.795 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 38 (ObjNum 174 snap 0 seq_num 174) dirty exists 2020-06-12T02:37:30.795 INFO:tasks.rados.rados.0.smithi107.stdout:175: left oid 13 (ObjNum 174 snap 0 seq_num 174) 2020-06-12T02:37:30.797 INFO:tasks.rados.rados.0.smithi107.stdout:174: done (2 left) 2020-06-12T02:37:30.797 INFO:tasks.rados.rados.0.smithi107.stdout:175: done (1 left) 2020-06-12T02:37:30.797 INFO:tasks.rados.rados.0.smithi107.stdout:177: done (0 left) 2020-06-12T02:37:30.797 INFO:tasks.rados.rados.0.smithi107.stdout:178: delete oid 3 current snap is 0 2020-06-12T02:37:30.801 INFO:tasks.rados.rados.0.smithi107.stdout:178: done (0 left) 2020-06-12T02:37:30.802 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:37:30.802 INFO:tasks.rados.rados.0.smithi107.stdout:179: seq_num 178 ranges {4055040=589824} 2020-06-12T02:37:30.826 INFO:tasks.rados.rados.0.smithi107.stdout:179: writing smithi10729407-19 from 4055040 to 4644864 tid 1 2020-06-12T02:37:30.826 INFO:tasks.rados.rados.0.smithi107.stdout:180: delete oid 2 current snap is 0 2020-06-12T02:37:30.827 INFO:tasks.rados.rados.0.smithi107.stdout:180: done (1 left) 2020-06-12T02:37:30.827 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:37:30.828 INFO:tasks.rados.rados.0.smithi107.stdout:181: seq_num 180 ranges {516096=622592} 2020-06-12T02:37:30.828 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:30.825+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:30.833 INFO:tasks.rados.rados.0.smithi107.stdout:181: writing smithi10729407-13 from 516096 to 1138688 tid 1 2020-06-12T02:37:30.834 INFO:tasks.rados.rados.0.smithi107.stdout:182: delete oid 4 current snap is 0 2020-06-12T02:37:30.834 INFO:tasks.rados.rados.0.smithi107.stdout:179: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:30.834 INFO:tasks.rados.rados.0.smithi107.stdout:179: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:30.835 INFO:tasks.rados.rados.0.smithi107.stdout:179: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:30.836 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 28 (ObjNum 178 snap 0 seq_num 178) dirty exists 2020-06-12T02:37:30.836 INFO:tasks.rados.rados.0.smithi107.stdout:179: left oid 19 (ObjNum 178 snap 0 seq_num 178) 2020-06-12T02:37:30.837 INFO:tasks.rados.rados.0.smithi107.stdout:179: done (2 left) 2020-06-12T02:37:30.837 INFO:tasks.rados.rados.0.smithi107.stdout:182: done (1 left) 2020-06-12T02:37:30.837 INFO:tasks.rados.rados.0.smithi107.stdout:183: delete oid 3 current snap is 0 2020-06-12T02:37:30.839 INFO:tasks.rados.rados.0.smithi107.stdout:183: done (1 left) 2020-06-12T02:37:30.839 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:37:30.839 INFO:tasks.rados.rados.0.smithi107.stdout:184: seq_num 183 ranges {0=671744} 2020-06-12T02:37:30.840 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:30.840 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:30.843 INFO:tasks.rados.rados.0.smithi107.stdout:184: writing smithi10729407-10 from 0 to 671744 tid 1 2020-06-12T02:37:30.843 INFO:tasks.rados.rados.0.smithi107.stdout:185: delete oid 18 current snap is 0 2020-06-12T02:37:30.843 INFO:tasks.rados.rados.0.smithi107.stdout:181: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:30.843 INFO:tasks.rados.rados.0.smithi107.stdout:181: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:30.844 INFO:tasks.rados.rados.0.smithi107.stdout:181: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:30.844 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 40 (ObjNum 180 snap 0 seq_num 180) dirty exists 2020-06-12T02:37:30.844 INFO:tasks.rados.rados.0.smithi107.stdout:181: left oid 13 (ObjNum 180 snap 0 seq_num 180) 2020-06-12T02:37:30.845 INFO:tasks.rados.rados.0.smithi107.stdout:181: done (2 left) 2020-06-12T02:37:30.845 INFO:tasks.rados.rados.0.smithi107.stdout:185: done (1 left) 2020-06-12T02:37:30.845 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:37:30.846 INFO:tasks.rados.rados.0.smithi107.stdout:186: seq_num 185 ranges {0=704512} 2020-06-12T02:37:30.849 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-06-12T02:37:30.850 INFO:tasks.rados.rados.0.smithi107.stdout:186: writing smithi10729407-8 from 0 to 704512 tid 1 2020-06-12T02:37:30.850 INFO:tasks.rados.rados.0.smithi107.stdout:187: delete oid 11 current snap is 0 2020-06-12T02:37:30.850 INFO:tasks.rados.rados.0.smithi107.stdout:184: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:30.852 INFO:tasks.rados.rados.0.smithi107.stdout:184: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:30.853 INFO:tasks.rados.rados.0.smithi107.stdout:187: done (2 left) 2020-06-12T02:37:30.853 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:37:30.853 INFO:tasks.rados.rados.0.smithi107.stdout:188: seq_num 187 ranges {0=737280} 2020-06-12T02:37:30.858 INFO:tasks.rados.rados.0.smithi107.stdout:188: writing smithi10729407-12 from 0 to 737280 tid 1 2020-06-12T02:37:30.858 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:37:30.858 INFO:tasks.rados.rados.0.smithi107.stdout:189: seq_num 188 ranges {2572288=753664} 2020-06-12T02:37:30.877 INFO:tasks.rados.rados.0.smithi107.stdout:189: writing smithi10729407-15 from 2572288 to 3325952 tid 1 2020-06-12T02:37:30.877 INFO:tasks.rados.rados.0.smithi107.stdout:190: delete oid 17 current snap is 0 2020-06-12T02:37:30.877 INFO:tasks.rados.rados.0.smithi107.stdout:184: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:30.877 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 26 (ObjNum 183 snap 0 seq_num 183) dirty exists 2020-06-12T02:37:30.877 INFO:tasks.rados.rados.0.smithi107.stdout:184: left oid 10 (ObjNum 183 snap 0 seq_num 183) 2020-06-12T02:37:30.878 INFO:tasks.rados.rados.0.smithi107.stdout:186: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:30.878 INFO:tasks.rados.rados.0.smithi107.stdout:186: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:30.878 INFO:tasks.rados.rados.0.smithi107.stdout:186: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:30.878 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 40 (ObjNum 185 snap 0 seq_num 185) dirty exists 2020-06-12T02:37:30.879 INFO:tasks.rados.rados.0.smithi107.stdout:186: left oid 8 (ObjNum 185 snap 0 seq_num 185) 2020-06-12T02:37:30.879 INFO:tasks.rados.rados.0.smithi107.stdout:188: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:30.881 INFO:tasks.rados.rados.0.smithi107.stdout:188: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:30.881 INFO:tasks.rados.rados.0.smithi107.stdout:188: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:30.882 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 30 (ObjNum 187 snap 0 seq_num 187) dirty exists 2020-06-12T02:37:30.883 INFO:tasks.rados.rados.0.smithi107.stdout:188: left oid 12 (ObjNum 187 snap 0 seq_num 187) 2020-06-12T02:37:30.883 INFO:tasks.rados.rados.0.smithi107.stdout:184: done (4 left) 2020-06-12T02:37:30.884 INFO:tasks.rados.rados.0.smithi107.stdout:186: done (3 left) 2020-06-12T02:37:30.884 INFO:tasks.rados.rados.0.smithi107.stdout:188: done (2 left) 2020-06-12T02:37:30.884 INFO:tasks.rados.rados.0.smithi107.stdout:190: done (1 left) 2020-06-12T02:37:30.884 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:37:30.885 INFO:tasks.rados.rados.0.smithi107.stdout:191: seq_num 190 ranges {0=786432} 2020-06-12T02:37:30.885 INFO:tasks.rados.rados.0.smithi107.stdout:191: writing smithi10729407-7 from 0 to 786432 tid 1 2020-06-12T02:37:30.885 INFO:tasks.rados.rados.0.smithi107.stdout:192: delete oid 16 current snap is 0 2020-06-12T02:37:30.886 INFO:tasks.rados.rados.0.smithi107.stdout:189: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:30.887 INFO:tasks.rados.rados.0.smithi107.stdout:189: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:30.888 INFO:tasks.rados.rados.0.smithi107.stdout:189: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:30.888 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 42 (ObjNum 188 snap 0 seq_num 188) dirty exists 2020-06-12T02:37:30.888 INFO:tasks.rados.rados.0.smithi107.stdout:189: left oid 15 (ObjNum 188 snap 0 seq_num 188) 2020-06-12T02:37:30.891 INFO:tasks.rados.rados.0.smithi107.stdout:189: done (2 left) 2020-06-12T02:37:30.891 INFO:tasks.rados.rados.0.smithi107.stdout:192: done (1 left) 2020-06-12T02:37:30.892 INFO:tasks.rados.rados.0.smithi107.stdout:193: delete oid 3 current snap is 0 2020-06-12T02:37:30.892 INFO:tasks.rados.rados.0.smithi107.stdout:191: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:30.892 INFO:tasks.rados.rados.0.smithi107.stdout:191: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:30.893 INFO:tasks.rados.rados.0.smithi107.stdout:191: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:30.894 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 42 (ObjNum 190 snap 0 seq_num 190) dirty exists 2020-06-12T02:37:30.894 INFO:tasks.rados.rados.0.smithi107.stdout:191: left oid 7 (ObjNum 190 snap 0 seq_num 190) 2020-06-12T02:37:30.896 INFO:tasks.rados.rados.0.smithi107.stdout:191: done (1 left) 2020-06-12T02:37:30.897 INFO:tasks.rados.rados.0.smithi107.stdout:193: done (0 left) 2020-06-12T02:37:30.897 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:37:30.897 INFO:tasks.rados.rados.0.smithi107.stdout:194: seq_num 193 ranges {737280=434176,1171456=401408} 2020-06-12T02:37:30.904 INFO:tasks.rados.rados.0.smithi107.stdout:194: writing smithi10729407-12 from 737280 to 1171456 tid 1 2020-06-12T02:37:30.907 INFO:tasks.rados.rados.0.smithi107.stdout:194: writing smithi10729407-12 from 1171456 to 1572864 tid 2 2020-06-12T02:37:30.907 INFO:tasks.rados.rados.0.smithi107.stdout:195: delete oid 11 current snap is 0 2020-06-12T02:37:30.908 INFO:tasks.rados.rados.0.smithi107.stdout:195: done (1 left) 2020-06-12T02:37:30.909 INFO:tasks.rados.rados.0.smithi107.stdout:196: delete oid 15 current snap is 0 2020-06-12T02:37:30.910 INFO:tasks.rados.rados.0.smithi107.stdout:194: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:30.911 INFO:tasks.rados.rados.0.smithi107.stdout:194: finishing write tid 2 to smithi10729407-12 2020-06-12T02:37:30.911 INFO:tasks.rados.rados.0.smithi107.stdout:194: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:30.912 INFO:tasks.rados.rados.0.smithi107.stdout:196: done (1 left) 2020-06-12T02:37:30.912 INFO:tasks.rados.rados.0.smithi107.stdout:197: delete oid 4 current snap is 0 2020-06-12T02:37:30.913 INFO:tasks.rados.rados.0.smithi107.stdout:194: finishing write tid 5 to smithi10729407-12 2020-06-12T02:37:30.913 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 33 (ObjNum 193 snap 0 seq_num 193) dirty exists 2020-06-12T02:37:30.913 INFO:tasks.rados.rados.0.smithi107.stdout:194: left oid 12 (ObjNum 193 snap 0 seq_num 193) 2020-06-12T02:37:30.914 INFO:tasks.rados.rados.0.smithi107.stdout:194: done (1 left) 2020-06-12T02:37:30.914 INFO:tasks.rados.rados.0.smithi107.stdout:197: done (0 left) 2020-06-12T02:37:30.914 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:37:30.914 INFO:tasks.rados.rados.0.smithi107.stdout:198: seq_num 197 ranges {0=507904,507904=401408} 2020-06-12T02:37:30.917 INFO:tasks.rados.rados.0.smithi107.stdout:198: writing smithi10729407-4 from 0 to 507904 tid 1 2020-06-12T02:37:30.920 INFO:tasks.rados.rados.0.smithi107.stdout:198: writing smithi10729407-4 from 507904 to 909312 tid 2 2020-06-12T02:37:30.920 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:37:30.920 INFO:tasks.rados.rados.0.smithi107.stdout:199: seq_num 198 ranges {0=524288,524288=401408} 2020-06-12T02:37:30.924 INFO:tasks.rados.rados.0.smithi107.stdout:199: writing smithi10729407-6 from 0 to 524288 tid 1 2020-06-12T02:37:30.927 INFO:tasks.rados.rados.0.smithi107.stdout:199: writing smithi10729407-6 from 524288 to 925696 tid 2 2020-06-12T02:37:30.927 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:37:30.927 INFO:tasks.rados.rados.0.smithi107.stdout:200: seq_num 199 ranges {0=540672,540672=401408} 2020-06-12T02:37:30.930 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:37:30.927+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:37:30.931 INFO:tasks.rados.rados.0.smithi107.stdout:200: writing smithi10729407-18 from 0 to 540672 tid 1 2020-06-12T02:37:30.931 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2020-06-12T02:37:30.934 INFO:tasks.rados.rados.0.smithi107.stdout:200: writing smithi10729407-18 from 540672 to 942080 tid 2 2020-06-12T02:37:30.934 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:37:30.934 INFO:tasks.rados.rados.0.smithi107.stdout:201: seq_num 200 ranges {401408=557056,958464=401408} 2020-06-12T02:37:30.940 INFO:tasks.rados.rados.0.smithi107.stdout:201: writing smithi10729407-14 from 401408 to 958464 tid 1 2020-06-12T02:37:30.944 INFO:tasks.rados.rados.0.smithi107.stdout:201: writing smithi10729407-14 from 958464 to 1359872 tid 2 2020-06-12T02:37:30.944 INFO:tasks.rados.rados.0.smithi107.stdout:202: delete oid 10 current snap is 0 2020-06-12T02:37:30.944 INFO:tasks.rados.rados.0.smithi107.stdout:198: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:30.945 INFO:tasks.rados.rados.0.smithi107.stdout:198: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:30.945 INFO:tasks.rados.rados.0.smithi107.stdout:198: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:30.945 INFO:tasks.rados.rados.0.smithi107.stdout:198: finishing write tid 5 to smithi10729407-4 2020-06-12T02:37:30.945 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 42 (ObjNum 197 snap 0 seq_num 197) dirty exists 2020-06-12T02:37:30.945 INFO:tasks.rados.rados.0.smithi107.stdout:198: left oid 4 (ObjNum 197 snap 0 seq_num 197) 2020-06-12T02:37:30.946 INFO:tasks.rados.rados.0.smithi107.stdout:199: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:30.946 INFO:tasks.rados.rados.0.smithi107.stdout:199: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:30.946 INFO:tasks.rados.rados.0.smithi107.stdout:199: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:30.946 INFO:tasks.rados.rados.0.smithi107.stdout:199: finishing write tid 5 to smithi10729407-6 2020-06-12T02:37:30.946 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 39 (ObjNum 198 snap 0 seq_num 198) dirty exists 2020-06-12T02:37:30.947 INFO:tasks.rados.rados.0.smithi107.stdout:199: left oid 6 (ObjNum 198 snap 0 seq_num 198) 2020-06-12T02:37:30.947 INFO:tasks.rados.rados.0.smithi107.stdout:200: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:30.947 INFO:tasks.rados.rados.0.smithi107.stdout:200: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:30.947 INFO:tasks.rados.rados.0.smithi107.stdout:200: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:30.948 INFO:tasks.rados.rados.0.smithi107.stdout:200: finishing write tid 5 to smithi10729407-18 2020-06-12T02:37:30.948 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 32 (ObjNum 199 snap 0 seq_num 199) dirty exists 2020-06-12T02:37:30.948 INFO:tasks.rados.rados.0.smithi107.stdout:200: left oid 18 (ObjNum 199 snap 0 seq_num 199) 2020-06-12T02:37:30.948 INFO:tasks.rados.rados.0.smithi107.stdout:198: done (4 left) 2020-06-12T02:37:30.949 INFO:tasks.rados.rados.0.smithi107.stdout:199: done (3 left) 2020-06-12T02:37:30.949 INFO:tasks.rados.rados.0.smithi107.stdout:200: done (2 left) 2020-06-12T02:37:30.949 INFO:tasks.rados.rados.0.smithi107.stdout:202: done (1 left) 2020-06-12T02:37:30.949 INFO:tasks.rados.rados.0.smithi107.stdout:203: delete oid 12 current snap is 0 2020-06-12T02:37:30.952 INFO:tasks.rados.rados.0.smithi107.stdout:203: done (1 left) 2020-06-12T02:37:30.952 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:37:30.952 INFO:tasks.rados.rados.0.smithi107.stdout:204: seq_num 203 ranges {0=606208,606208=401408} 2020-06-12T02:37:30.955 INFO:tasks.rados.rados.0.smithi107.stdout:204: writing smithi10729407-3 from 0 to 606208 tid 1 2020-06-12T02:37:30.958 INFO:tasks.rados.rados.0.smithi107.stdout:204: writing smithi10729407-3 from 606208 to 1007616 tid 2 2020-06-12T02:37:30.959 INFO:tasks.rados.rados.0.smithi107.stdout:205: delete oid 12 current snap is 0 2020-06-12T02:37:30.959 INFO:tasks.rados.rados.0.smithi107.stdout:201: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:30.959 INFO:tasks.rados.rados.0.smithi107.stdout:201: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:30.959 INFO:tasks.rados.rados.0.smithi107.stdout:201: finishing write tid 4 to smithi10729407-14 2020-06-12T02:37:30.959 INFO:tasks.rados.rados.0.smithi107.stdout:201: finishing write tid 5 to smithi10729407-14 2020-06-12T02:37:30.960 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 25 (ObjNum 200 snap 0 seq_num 200) dirty exists 2020-06-12T02:37:30.960 INFO:tasks.rados.rados.0.smithi107.stdout:201: left oid 14 (ObjNum 200 snap 0 seq_num 200) 2020-06-12T02:37:30.960 INFO:tasks.rados.rados.0.smithi107.stdout:201: done (2 left) 2020-06-12T02:37:30.960 INFO:tasks.rados.rados.0.smithi107.stdout:205: done (1 left) 2020-06-12T02:37:30.961 INFO:tasks.rados.rados.0.smithi107.stdout:206: delete oid 12 current snap is 0 2020-06-12T02:37:30.963 INFO:tasks.rados.rados.0.smithi107.stdout:206: done (1 left) 2020-06-12T02:37:30.963 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:37:30.963 INFO:tasks.rados.rados.0.smithi107.stdout:207: seq_num 206 ranges {0=655360,655360=401408} 2020-06-12T02:37:30.967 INFO:tasks.rados.rados.0.smithi107.stdout:207: writing smithi10729407-15 from 0 to 655360 tid 1 2020-06-12T02:37:30.970 INFO:tasks.rados.rados.0.smithi107.stdout:207: writing smithi10729407-15 from 655360 to 1056768 tid 2 2020-06-12T02:37:30.970 INFO:tasks.rados.rados.0.smithi107.stdout:208: delete oid 19 current snap is 0 2020-06-12T02:37:30.970 INFO:tasks.rados.rados.0.smithi107.stdout:204: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:30.970 INFO:tasks.rados.rados.0.smithi107.stdout:204: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:30.970 INFO:tasks.rados.rados.0.smithi107.stdout:204: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:30.971 INFO:tasks.rados.rados.0.smithi107.stdout:204: finishing write tid 5 to smithi10729407-3 2020-06-12T02:37:30.971 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 37 (ObjNum 203 snap 0 seq_num 203) dirty exists 2020-06-12T02:37:30.971 INFO:tasks.rados.rados.0.smithi107.stdout:204: left oid 3 (ObjNum 203 snap 0 seq_num 203) 2020-06-12T02:37:30.974 INFO:tasks.rados.rados.0.smithi107.stdout:204: done (2 left) 2020-06-12T02:37:30.974 INFO:tasks.rados.rados.0.smithi107.stdout:208: done (1 left) 2020-06-12T02:37:30.974 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:37:30.975 INFO:tasks.rados.rados.0.smithi107.stdout:209: seq_num 208 ranges {0=688128,688128=401408} 2020-06-12T02:37:30.979 INFO:tasks.rados.rados.0.smithi107.stdout:209: writing smithi10729407-10 from 0 to 688128 tid 1 2020-06-12T02:37:30.982 INFO:tasks.rados.rados.0.smithi107.stdout:209: writing smithi10729407-10 from 688128 to 1089536 tid 2 2020-06-12T02:37:30.982 INFO:tasks.rados.rados.0.smithi107.stdout:210: delete oid 16 current snap is 0 2020-06-12T02:37:30.983 INFO:tasks.rados.rados.0.smithi107.stdout:207: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:30.983 INFO:tasks.rados.rados.0.smithi107.stdout:207: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:30.984 INFO:tasks.rados.rados.0.smithi107.stdout:207: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:30.985 INFO:tasks.rados.rados.0.smithi107.stdout:207: finishing write tid 5 to smithi10729407-15 2020-06-12T02:37:30.985 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 46 (ObjNum 206 snap 0 seq_num 206) dirty exists 2020-06-12T02:37:30.986 INFO:tasks.rados.rados.0.smithi107.stdout:207: left oid 15 (ObjNum 206 snap 0 seq_num 206) 2020-06-12T02:37:30.986 INFO:tasks.rados.rados.0.smithi107.stdout:207: done (2 left) 2020-06-12T02:37:30.987 INFO:tasks.rados.rados.0.smithi107.stdout:210: done (1 left) 2020-06-12T02:37:30.987 INFO:tasks.rados.rados.0.smithi107.stdout:211: delete oid 20 current snap is 0 2020-06-12T02:37:30.989 INFO:tasks.rados.rados.0.smithi107.stdout:209: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:30.992 INFO:tasks.rados.rados.0.smithi107.stdout:211: done (1 left) 2020-06-12T02:37:30.992 INFO:tasks.rados.rados.0.smithi107.stdout:212: delete oid 15 current snap is 0 2020-06-12T02:37:30.993 INFO:tasks.rados.rados.0.smithi107.stdout:209: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:30.993 INFO:tasks.rados.rados.0.smithi107.stdout:209: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:30.994 INFO:tasks.rados.rados.0.smithi107.stdout:209: finishing write tid 5 to smithi10729407-10 2020-06-12T02:37:30.994 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 30 (ObjNum 208 snap 0 seq_num 208) dirty exists 2020-06-12T02:37:30.995 INFO:tasks.rados.rados.0.smithi107.stdout:209: left oid 10 (ObjNum 208 snap 0 seq_num 208) 2020-06-12T02:37:30.996 INFO:tasks.rados.rados.0.smithi107.stdout:209: done (1 left) 2020-06-12T02:37:30.996 INFO:tasks.rados.rados.0.smithi107.stdout:212: done (0 left) 2020-06-12T02:37:30.997 INFO:tasks.rados.rados.0.smithi107.stdout:213: delete oid 5 current snap is 0 2020-06-12T02:37:30.999 INFO:tasks.rados.rados.0.smithi107.stdout:213: done (0 left) 2020-06-12T02:37:30.999 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:37:30.999 INFO:tasks.rados.rados.0.smithi107.stdout:214: seq_num 213 ranges {786432=770048,1556480=401408} 2020-06-12T02:37:31.006 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:31.006 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:31.007 INFO:tasks.rados.rados.0.smithi107.stdout:214: writing smithi10729407-7 from 786432 to 1556480 tid 1 2020-06-12T02:37:31.009 INFO:tasks.rados.rados.0.smithi107.stdout:214: writing smithi10729407-7 from 1556480 to 1957888 tid 2 2020-06-12T02:37:31.010 INFO:tasks.rados.rados.0.smithi107.stdout:215: delete oid 3 current snap is 0 2020-06-12T02:37:31.016 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-06-12T02:37:31.017 INFO:tasks.rados.rados.0.smithi107.stdout:215: done (1 left) 2020-06-12T02:37:31.017 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:37:31.017 INFO:tasks.rados.rados.0.smithi107.stdout:216: seq_num 215 ranges {0=409600,409600=532480,942080=270336} 2020-06-12T02:37:31.018 INFO:tasks.rados.rados.0.smithi107.stdout:216: writing smithi10729407-15 from 0 to 409600 tid 1 2020-06-12T02:37:31.022 INFO:tasks.rados.rados.0.smithi107.stdout:216: writing smithi10729407-15 from 409600 to 942080 tid 2 2020-06-12T02:37:31.024 INFO:tasks.rados.rados.0.smithi107.stdout:216: writing smithi10729407-15 from 942080 to 1212416 tid 3 2020-06-12T02:37:31.024 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:37:31.024 INFO:tasks.rados.rados.0.smithi107.stdout:217: seq_num 216 ranges {925696=425984,1351680=409600,1761280=393216} 2020-06-12T02:37:31.030 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:31.028+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:31.032 INFO:tasks.rados.rados.0.smithi107.stdout:217: writing smithi10729407-6 from 925696 to 1351680 tid 1 2020-06-12T02:37:31.036 INFO:tasks.rados.rados.0.smithi107.stdout:217: writing smithi10729407-6 from 1351680 to 1761280 tid 2 2020-06-12T02:37:31.038 INFO:tasks.rados.rados.0.smithi107.stdout:217: writing smithi10729407-6 from 1761280 to 2154496 tid 3 2020-06-12T02:37:31.038 INFO:tasks.rados.rados.0.smithi107.stdout:218: delete oid 20 current snap is 0 2020-06-12T02:37:31.039 INFO:tasks.rados.rados.0.smithi107.stdout:214: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:31.039 INFO:tasks.rados.rados.0.smithi107.stdout:214: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:31.039 INFO:tasks.rados.rados.0.smithi107.stdout:214: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:31.039 INFO:tasks.rados.rados.0.smithi107.stdout:214: finishing write tid 5 to smithi10729407-7 2020-06-12T02:37:31.039 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 45 (ObjNum 213 snap 0 seq_num 213) dirty exists 2020-06-12T02:37:31.040 INFO:tasks.rados.rados.0.smithi107.stdout:214: left oid 7 (ObjNum 213 snap 0 seq_num 213) 2020-06-12T02:37:31.040 INFO:tasks.rados.rados.0.smithi107.stdout:216: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:31.040 INFO:tasks.rados.rados.0.smithi107.stdout:216: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:31.040 INFO:tasks.rados.rados.0.smithi107.stdout:216: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:31.040 INFO:tasks.rados.rados.0.smithi107.stdout:216: finishing write tid 5 to smithi10729407-15 2020-06-12T02:37:31.040 INFO:tasks.rados.rados.0.smithi107.stdout:216: finishing write tid 6 to smithi10729407-15 2020-06-12T02:37:31.041 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 51 (ObjNum 215 snap 0 seq_num 215) dirty exists 2020-06-12T02:37:31.041 INFO:tasks.rados.rados.0.smithi107.stdout:216: left oid 15 (ObjNum 215 snap 0 seq_num 215) 2020-06-12T02:37:31.041 INFO:tasks.rados.rados.0.smithi107.stdout:214: done (3 left) 2020-06-12T02:37:31.042 INFO:tasks.rados.rados.0.smithi107.stdout:216: done (2 left) 2020-06-12T02:37:31.042 INFO:tasks.rados.rados.0.smithi107.stdout:218: done (1 left) 2020-06-12T02:37:31.042 INFO:tasks.rados.rados.0.smithi107.stdout:219: delete oid 15 current snap is 0 2020-06-12T02:37:31.042 INFO:tasks.rados.rados.0.smithi107.stdout:217: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:31.043 INFO:tasks.rados.rados.0.smithi107.stdout:217: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:31.043 INFO:tasks.rados.rados.0.smithi107.stdout:217: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:31.044 INFO:tasks.rados.rados.0.smithi107.stdout:217: finishing write tid 5 to smithi10729407-6 2020-06-12T02:37:31.044 INFO:tasks.rados.rados.0.smithi107.stdout:219: done (1 left) 2020-06-12T02:37:31.044 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:37:31.045 INFO:tasks.rados.rados.0.smithi107.stdout:220: seq_num 219 ranges {0=475136,475136=466944,942080=335872} 2020-06-12T02:37:31.046 INFO:tasks.rados.rados.0.smithi107.stdout:220: writing smithi10729407-19 from 0 to 475136 tid 1 2020-06-12T02:37:31.050 INFO:tasks.rados.rados.0.smithi107.stdout:220: writing smithi10729407-19 from 475136 to 942080 tid 2 2020-06-12T02:37:31.051 INFO:tasks.rados.rados.0.smithi107.stdout:220: writing smithi10729407-19 from 942080 to 1277952 tid 3 2020-06-12T02:37:31.051 INFO:tasks.rados.rados.0.smithi107.stdout:221: delete oid 2 current snap is 0 2020-06-12T02:37:31.052 INFO:tasks.rados.rados.0.smithi107.stdout:217: finishing write tid 6 to smithi10729407-6 2020-06-12T02:37:31.052 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 43 (ObjNum 216 snap 0 seq_num 216) dirty exists 2020-06-12T02:37:31.052 INFO:tasks.rados.rados.0.smithi107.stdout:217: left oid 6 (ObjNum 216 snap 0 seq_num 216) 2020-06-12T02:37:31.052 INFO:tasks.rados.rados.0.smithi107.stdout:220: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:31.053 INFO:tasks.rados.rados.0.smithi107.stdout:217: done (2 left) 2020-06-12T02:37:31.053 INFO:tasks.rados.rados.0.smithi107.stdout:221: done (1 left) 2020-06-12T02:37:31.053 INFO:tasks.rados.rados.0.smithi107.stdout:222: delete oid 15 current snap is 0 2020-06-12T02:37:31.055 INFO:tasks.rados.rados.0.smithi107.stdout:222: done (1 left) 2020-06-12T02:37:31.055 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:37:31.055 INFO:tasks.rados.rados.0.smithi107.stdout:223: seq_num 222 ranges {0=524288,524288=565248,1089536=237568} 2020-06-12T02:37:31.058 INFO:tasks.rados.rados.0.smithi107.stdout:223: writing smithi10729407-15 from 0 to 524288 tid 1 2020-06-12T02:37:31.061 INFO:tasks.rados.rados.0.smithi107.stdout:223: writing smithi10729407-15 from 524288 to 1089536 tid 2 2020-06-12T02:37:31.063 INFO:tasks.rados.rados.0.smithi107.stdout:223: writing smithi10729407-15 from 1089536 to 1327104 tid 3 2020-06-12T02:37:31.063 INFO:tasks.rados.rados.0.smithi107.stdout:224: delete oid 5 current snap is 0 2020-06-12T02:37:31.064 INFO:tasks.rados.rados.0.smithi107.stdout:220: finishing write tid 2 to smithi10729407-19 2020-06-12T02:37:31.064 INFO:tasks.rados.rados.0.smithi107.stdout:220: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:31.064 INFO:tasks.rados.rados.0.smithi107.stdout:220: finishing write tid 5 to smithi10729407-19 2020-06-12T02:37:31.064 INFO:tasks.rados.rados.0.smithi107.stdout:220: finishing write tid 6 to smithi10729407-19 2020-06-12T02:37:31.065 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 41 (ObjNum 219 snap 0 seq_num 219) dirty exists 2020-06-12T02:37:31.065 INFO:tasks.rados.rados.0.smithi107.stdout:220: left oid 19 (ObjNum 219 snap 0 seq_num 219) 2020-06-12T02:37:31.065 INFO:tasks.rados.rados.0.smithi107.stdout:223: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:31.065 INFO:tasks.rados.rados.0.smithi107.stdout:220: done (2 left) 2020-06-12T02:37:31.066 INFO:tasks.rados.rados.0.smithi107.stdout:224: done (1 left) 2020-06-12T02:37:31.066 INFO:tasks.rados.rados.0.smithi107.stdout:225: delete oid 8 current snap is 0 2020-06-12T02:37:31.069 INFO:tasks.rados.rados.0.smithi107.stdout:223: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:31.069 INFO:tasks.rados.rados.0.smithi107.stdout:223: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:31.070 INFO:tasks.rados.rados.0.smithi107.stdout:223: finishing write tid 5 to smithi10729407-15 2020-06-12T02:37:31.070 INFO:tasks.rados.rados.0.smithi107.stdout:225: done (1 left) 2020-06-12T02:37:31.070 INFO:tasks.rados.rados.0.smithi107.stdout:226: delete oid 8 current snap is 0 2020-06-12T02:37:31.072 INFO:tasks.rados.rados.0.smithi107.stdout:223: finishing write tid 6 to smithi10729407-15 2020-06-12T02:37:31.072 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 57 (ObjNum 222 snap 0 seq_num 222) dirty exists 2020-06-12T02:37:31.072 INFO:tasks.rados.rados.0.smithi107.stdout:223: left oid 15 (ObjNum 222 snap 0 seq_num 222) 2020-06-12T02:37:31.073 INFO:tasks.rados.rados.0.smithi107.stdout:223: done (1 left) 2020-06-12T02:37:31.073 INFO:tasks.rados.rados.0.smithi107.stdout:226: done (0 left) 2020-06-12T02:37:31.074 INFO:tasks.rados.rados.0.smithi107.stdout:227: delete oid 10 current snap is 0 2020-06-12T02:37:31.076 INFO:tasks.rados.rados.0.smithi107.stdout:227: done (0 left) 2020-06-12T02:37:31.076 INFO:tasks.rados.rados.0.smithi107.stdout:228: delete oid 8 current snap is 0 2020-06-12T02:37:31.079 INFO:tasks.rados.rados.0.smithi107.stdout:228: done (0 left) 2020-06-12T02:37:31.079 INFO:tasks.rados.rados.0.smithi107.stdout:229: delete oid 13 current snap is 0 2020-06-12T02:37:31.081 INFO:tasks.rados.rados.0.smithi107.stdout:229: done (0 left) 2020-06-12T02:37:31.082 INFO:tasks.rados.rados.0.smithi107.stdout:230: delete oid 13 current snap is 0 2020-06-12T02:37:31.087 INFO:tasks.rados.rados.0.smithi107.stdout:230: done (0 left) 2020-06-12T02:37:31.088 INFO:tasks.rados.rados.0.smithi107.stdout:231: delete oid 8 current snap is 0 2020-06-12T02:37:31.089 INFO:tasks.rados.rados.0.smithi107.stdout:231: done (0 left) 2020-06-12T02:37:31.090 INFO:tasks.rados.rados.0.smithi107.stdout:232: delete oid 2 current snap is 0 2020-06-12T02:37:31.092 INFO:tasks.rados.rados.0.smithi107.stdout:232: done (0 left) 2020-06-12T02:37:31.092 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:37:31.092 INFO:tasks.rados.rados.0.smithi107.stdout:233: seq_num 232 ranges {0=688128,688128=663552,1351680=139264} 2020-06-12T02:37:31.096 INFO:tasks.rados.rados.0.smithi107.stdout:233: writing smithi10729407-5 from 0 to 688128 tid 1 2020-06-12T02:37:31.100 INFO:tasks.rados.rados.0.smithi107.stdout:233: writing smithi10729407-5 from 688128 to 1351680 tid 2 2020-06-12T02:37:31.102 INFO:tasks.rados.rados.0.smithi107.stdout:233: writing smithi10729407-5 from 1351680 to 1490944 tid 3 2020-06-12T02:37:31.102 INFO:tasks.rados.rados.0.smithi107.stdout:234: delete oid 16 current snap is 0 2020-06-12T02:37:31.103 INFO:tasks.rados.rados.0.smithi107.stdout:234: done (1 left) 2020-06-12T02:37:31.104 INFO:tasks.rados.rados.0.smithi107.stdout:235: delete oid 8 current snap is 0 2020-06-12T02:37:31.107 INFO:tasks.rados.rados.0.smithi107.stdout:235: done (1 left) 2020-06-12T02:37:31.107 INFO:tasks.rados.rados.0.smithi107.stdout:236: delete oid 4 current snap is 0 2020-06-12T02:37:31.107 INFO:tasks.rados.rados.0.smithi107.stdout:233: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:31.108 INFO:tasks.rados.rados.0.smithi107.stdout:233: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:31.109 INFO:tasks.rados.rados.0.smithi107.stdout:233: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:31.109 INFO:tasks.rados.rados.0.smithi107.stdout:233: finishing write tid 5 to smithi10729407-5 2020-06-12T02:37:31.109 INFO:tasks.rados.rados.0.smithi107.stdout:236: done (1 left) 2020-06-12T02:37:31.109 INFO:tasks.rados.rados.0.smithi107.stdout:237: delete oid 1 current snap is 0 2020-06-12T02:37:31.110 INFO:tasks.rados.rados.0.smithi107.stdout:233: finishing write tid 6 to smithi10729407-5 2020-06-12T02:37:31.110 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 23 (ObjNum 232 snap 0 seq_num 232) dirty exists 2020-06-12T02:37:31.111 INFO:tasks.rados.rados.0.smithi107.stdout:233: left oid 5 (ObjNum 232 snap 0 seq_num 232) 2020-06-12T02:37:31.112 INFO:tasks.rados.rados.0.smithi107.stdout:233: done (1 left) 2020-06-12T02:37:31.112 INFO:tasks.rados.rados.0.smithi107.stdout:237: done (0 left) 2020-06-12T02:37:31.112 INFO:tasks.rados.rados.0.smithi107.stdout:238: delete oid 8 current snap is 0 2020-06-12T02:37:31.113 INFO:tasks.rados.rados.0.smithi107.stdout:238: done (0 left) 2020-06-12T02:37:31.114 INFO:tasks.rados.rados.0.smithi107.stdout:239: delete oid 8 current snap is 0 2020-06-12T02:37:31.116 INFO:tasks.rados.rados.0.smithi107.stdout:239: done (0 left) 2020-06-12T02:37:31.116 INFO:tasks.rados.rados.0.smithi107.stdout:240: delete oid 1 current snap is 0 2020-06-12T02:37:31.117 INFO:tasks.rados.rados.0.smithi107.stdout:240: done (0 left) 2020-06-12T02:37:31.118 INFO:tasks.rados.rados.0.smithi107.stdout:241: delete oid 18 current snap is 0 2020-06-12T02:37:31.119 INFO:tasks.rados.rados.0.smithi107.stdout:241: done (0 left) 2020-06-12T02:37:31.119 INFO:tasks.rados.rados.0.smithi107.stdout:242: delete oid 5 current snap is 0 2020-06-12T02:37:31.121 INFO:tasks.rados.rados.0.smithi107.stdout:242: done (0 left) 2020-06-12T02:37:31.121 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:37:31.122 INFO:tasks.rados.rados.0.smithi107.stdout:243: seq_num 242 ranges {0=458752,458752=753664,1212416=450560} 2020-06-12T02:37:31.124 INFO:tasks.rados.rados.0.smithi107.stdout:243: writing smithi10729407-2 from 0 to 458752 tid 1 2020-06-12T02:37:31.129 INFO:tasks.rados.rados.0.smithi107.stdout:243: writing smithi10729407-2 from 458752 to 1212416 tid 2 2020-06-12T02:37:31.134 INFO:tasks.rados.rados.0.smithi107.stdout:243: writing smithi10729407-2 from 1212416 to 1662976 tid 3 2020-06-12T02:37:31.134 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:37:31.134 INFO:tasks.rados.rados.0.smithi107.stdout:244: seq_num 243 ranges {0=475136,475136=638976,1114112=565248} 2020-06-12T02:37:31.134 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:37:31.131+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:37:31.139 INFO:tasks.rados.rados.0.smithi107.stdout:244: writing smithi10729407-20 from 0 to 475136 tid 1 2020-06-12T02:37:31.140 INFO:tasks.rados.rados.0.smithi107.stdout:244: writing smithi10729407-20 from 475136 to 1114112 tid 2 2020-06-12T02:37:31.144 INFO:tasks.rados.rados.0.smithi107.stdout:244: writing smithi10729407-20 from 1114112 to 1679360 tid 3 2020-06-12T02:37:31.144 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:37:31.144 INFO:tasks.rados.rados.0.smithi107.stdout:245: seq_num 244 ranges {0=491520,491520=573440,1064960=458752,1523712=172032} 2020-06-12T02:37:31.147 INFO:tasks.rados.rados.0.smithi107.stdout:245: writing smithi10729407-9 from 0 to 491520 tid 1 2020-06-12T02:37:31.151 INFO:tasks.rados.rados.0.smithi107.stdout:245: writing smithi10729407-9 from 491520 to 1064960 tid 2 2020-06-12T02:37:31.154 INFO:tasks.rados.rados.0.smithi107.stdout:245: writing smithi10729407-9 from 1064960 to 1523712 tid 3 2020-06-12T02:37:31.156 INFO:tasks.rados.rados.0.smithi107.stdout:245: writing smithi10729407-9 from 1523712 to 1695744 tid 4 2020-06-12T02:37:31.156 INFO:tasks.rados.rados.0.smithi107.stdout:246: delete oid 7 current snap is 0 2020-06-12T02:37:31.156 INFO:tasks.rados.rados.0.smithi107.stdout:243: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:31.156 INFO:tasks.rados.rados.0.smithi107.stdout:243: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:31.157 INFO:tasks.rados.rados.0.smithi107.stdout:243: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:31.157 INFO:tasks.rados.rados.0.smithi107.stdout:243: finishing write tid 5 to smithi10729407-2 2020-06-12T02:37:31.157 INFO:tasks.rados.rados.0.smithi107.stdout:244: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:31.157 INFO:tasks.rados.rados.0.smithi107.stdout:243: finishing write tid 6 to smithi10729407-2 2020-06-12T02:37:31.157 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 29 (ObjNum 242 snap 0 seq_num 242) dirty exists 2020-06-12T02:37:31.157 INFO:tasks.rados.rados.0.smithi107.stdout:243: left oid 2 (ObjNum 242 snap 0 seq_num 242) 2020-06-12T02:37:31.158 INFO:tasks.rados.rados.0.smithi107.stdout:244: finishing write tid 2 to smithi10729407-20 2020-06-12T02:37:31.158 INFO:tasks.rados.rados.0.smithi107.stdout:245: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:31.158 INFO:tasks.rados.rados.0.smithi107.stdout:244: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:31.158 INFO:tasks.rados.rados.0.smithi107.stdout:244: finishing write tid 5 to smithi10729407-20 2020-06-12T02:37:31.158 INFO:tasks.rados.rados.0.smithi107.stdout:244: finishing write tid 6 to smithi10729407-20 2020-06-12T02:37:31.159 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 15 (ObjNum 243 snap 0 seq_num 243) dirty exists 2020-06-12T02:37:31.159 INFO:tasks.rados.rados.0.smithi107.stdout:244: left oid 20 (ObjNum 243 snap 0 seq_num 243) 2020-06-12T02:37:31.162 INFO:tasks.rados.rados.0.smithi107.stdout:243: done (3 left) 2020-06-12T02:37:31.162 INFO:tasks.rados.rados.0.smithi107.stdout:244: done (2 left) 2020-06-12T02:37:31.163 INFO:tasks.rados.rados.0.smithi107.stdout:246: done (1 left) 2020-06-12T02:37:31.163 INFO:tasks.rados.rados.0.smithi107.stdout:247: delete oid 17 current snap is 0 2020-06-12T02:37:31.163 INFO:tasks.rados.rados.0.smithi107.stdout:247: done (1 left) 2020-06-12T02:37:31.163 INFO:tasks.rados.rados.0.smithi107.stdout:248: delete oid 18 current snap is 0 2020-06-12T02:37:31.163 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:31.164 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:31.164 INFO:tasks.rados.rados.0.smithi107.stdout:245: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:31.166 INFO:tasks.rados.rados.0.smithi107.stdout:245: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:31.167 INFO:tasks.rados.rados.0.smithi107.stdout:245: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:31.168 INFO:tasks.rados.rados.0.smithi107.stdout:245: finishing write tid 6 to smithi10729407-9 2020-06-12T02:37:31.169 INFO:tasks.rados.rados.0.smithi107.stdout:248: done (1 left) 2020-06-12T02:37:31.169 INFO:tasks.rados.rados.0.smithi107.stdout:249: delete oid 17 current snap is 0 2020-06-12T02:37:31.171 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-06-12T02:37:31.175 INFO:tasks.rados.rados.0.smithi107.stdout:245: finishing write tid 7 to smithi10729407-9 2020-06-12T02:37:31.175 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 38 (ObjNum 244 snap 0 seq_num 244) dirty exists 2020-06-12T02:37:31.176 INFO:tasks.rados.rados.0.smithi107.stdout:245: left oid 9 (ObjNum 244 snap 0 seq_num 244) 2020-06-12T02:37:31.176 INFO:tasks.rados.rados.0.smithi107.stdout:245: done (1 left) 2020-06-12T02:37:31.176 INFO:tasks.rados.rados.0.smithi107.stdout:249: done (0 left) 2020-06-12T02:37:31.176 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:37:31.176 INFO:tasks.rados.rados.0.smithi107.stdout:250: seq_num 249 ranges {2154496=573440,2727936=794624,3522560=409600} 2020-06-12T02:37:31.191 INFO:tasks.rados.rados.0.smithi107.stdout:250: writing smithi10729407-6 from 2154496 to 2727936 tid 1 2020-06-12T02:37:31.196 INFO:tasks.rados.rados.0.smithi107.stdout:250: writing smithi10729407-6 from 2727936 to 3522560 tid 2 2020-06-12T02:37:31.198 INFO:tasks.rados.rados.0.smithi107.stdout:250: writing smithi10729407-6 from 3522560 to 3932160 tid 3 2020-06-12T02:37:31.198 INFO:tasks.rados.rados.0.smithi107.stdout:251: delete oid 2 current snap is 0 2020-06-12T02:37:31.198 INFO:tasks.rados.rados.0.smithi107.stdout:250: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:31.200 INFO:tasks.rados.rados.0.smithi107.stdout:251: done (1 left) 2020-06-12T02:37:31.200 INFO:tasks.rados.rados.0.smithi107.stdout:252: delete oid 16 current snap is 0 2020-06-12T02:37:31.204 INFO:tasks.rados.rados.0.smithi107.stdout:252: done (1 left) 2020-06-12T02:37:31.204 INFO:tasks.rados.rados.0.smithi107.stdout:253: delete oid 9 current snap is 0 2020-06-12T02:37:31.204 INFO:tasks.rados.rados.0.smithi107.stdout:250: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:31.205 INFO:tasks.rados.rados.0.smithi107.stdout:250: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:31.206 INFO:tasks.rados.rados.0.smithi107.stdout:250: finishing write tid 5 to smithi10729407-6 2020-06-12T02:37:31.208 INFO:tasks.rados.rados.0.smithi107.stdout:250: finishing write tid 6 to smithi10729407-6 2020-06-12T02:37:31.208 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 49 (ObjNum 249 snap 0 seq_num 249) dirty exists 2020-06-12T02:37:31.208 INFO:tasks.rados.rados.0.smithi107.stdout:250: left oid 6 (ObjNum 249 snap 0 seq_num 249) 2020-06-12T02:37:31.209 INFO:tasks.rados.rados.0.smithi107.stdout:250: done (1 left) 2020-06-12T02:37:31.210 INFO:tasks.rados.rados.0.smithi107.stdout:253: done (0 left) 2020-06-12T02:37:31.210 INFO:tasks.rados.rados.0.smithi107.stdout:254: delete oid 5 current snap is 0 2020-06-12T02:37:31.212 INFO:tasks.rados.rados.0.smithi107.stdout:254: done (0 left) 2020-06-12T02:37:31.212 INFO:tasks.rados.rados.0.smithi107.stdout:255: delete oid 4 current snap is 0 2020-06-12T02:37:31.215 INFO:tasks.rados.rados.0.smithi107.stdout:255: done (0 left) 2020-06-12T02:37:31.215 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:37:31.215 INFO:tasks.rados.rados.0.smithi107.stdout:256: seq_num 255 ranges {0=679936,679936=548864,1228800=598016,1826816=57344} 2020-06-12T02:37:31.219 INFO:tasks.rados.rados.0.smithi107.stdout:256: writing smithi10729407-18 from 0 to 679936 tid 1 2020-06-12T02:37:31.222 INFO:tasks.rados.rados.0.smithi107.stdout:256: writing smithi10729407-18 from 679936 to 1228800 tid 2 2020-06-12T02:37:31.228 INFO:tasks.rados.rados.0.smithi107.stdout:256: writing smithi10729407-18 from 1228800 to 1826816 tid 3 2020-06-12T02:37:31.228 INFO:tasks.rados.rados.0.smithi107.stdout:256: writing smithi10729407-18 from 1826816 to 1884160 tid 4 2020-06-12T02:37:31.228 INFO:tasks.rados.rados.0.smithi107.stdout:257: delete oid 16 current snap is 0 2020-06-12T02:37:31.228 INFO:tasks.rados.rados.0.smithi107.stdout:256: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:31.229 INFO:tasks.rados.rados.0.smithi107.stdout:257: done (1 left) 2020-06-12T02:37:31.229 INFO:tasks.rados.rados.0.smithi107.stdout:258: delete oid 5 current snap is 0 2020-06-12T02:37:31.230 INFO:tasks.rados.rados.0.smithi107.stdout:258: done (1 left) 2020-06-12T02:37:31.231 INFO:tasks.rados.rados.0.smithi107.stdout:259: delete oid 3 current snap is 0 2020-06-12T02:37:31.232 INFO:tasks.rados.rados.0.smithi107.stdout:259: done (1 left) 2020-06-12T02:37:31.232 INFO:tasks.rados.rados.0.smithi107.stdout:260: delete oid 13 current snap is 0 2020-06-12T02:37:31.233 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:31.230+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:37:31.234 INFO:tasks.rados.rados.0.smithi107.stdout:260: done (1 left) 2020-06-12T02:37:31.234 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:37:31.235 INFO:tasks.rados.rados.0.smithi107.stdout:261: seq_num 260 ranges {0=761856,761856=417792,1179648=786432} 2020-06-12T02:37:31.239 INFO:tasks.rados.rados.0.smithi107.stdout:261: writing smithi10729407-17 from 0 to 761856 tid 1 2020-06-12T02:37:31.241 INFO:tasks.rados.rados.0.smithi107.stdout:261: writing smithi10729407-17 from 761856 to 1179648 tid 2 2020-06-12T02:37:31.246 INFO:tasks.rados.rados.0.smithi107.stdout:261: writing smithi10729407-17 from 1179648 to 1966080 tid 3 2020-06-12T02:37:31.246 INFO:tasks.rados.rados.0.smithi107.stdout:262: delete oid 6 current snap is 0 2020-06-12T02:37:31.247 INFO:tasks.rados.rados.0.smithi107.stdout:256: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:31.247 INFO:tasks.rados.rados.0.smithi107.stdout:256: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:31.247 INFO:tasks.rados.rados.0.smithi107.stdout:256: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:31.247 INFO:tasks.rados.rados.0.smithi107.stdout:256: finishing write tid 6 to smithi10729407-18 2020-06-12T02:37:31.247 INFO:tasks.rados.rados.0.smithi107.stdout:256: finishing write tid 7 to smithi10729407-18 2020-06-12T02:37:31.248 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 45 (ObjNum 255 snap 0 seq_num 255) dirty exists 2020-06-12T02:37:31.249 INFO:tasks.rados.rados.0.smithi107.stdout:256: left oid 18 (ObjNum 255 snap 0 seq_num 255) 2020-06-12T02:37:31.249 INFO:tasks.rados.rados.0.smithi107.stdout:261: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:31.250 INFO:tasks.rados.rados.0.smithi107.stdout:256: done (2 left) 2020-06-12T02:37:31.250 INFO:tasks.rados.rados.0.smithi107.stdout:262: done (1 left) 2020-06-12T02:37:31.250 INFO:tasks.rados.rados.0.smithi107.stdout:263: delete oid 6 current snap is 0 2020-06-12T02:37:31.251 INFO:tasks.rados.rados.0.smithi107.stdout:263: done (1 left) 2020-06-12T02:37:31.251 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:37:31.251 INFO:tasks.rados.rados.0.smithi107.stdout:264: seq_num 263 ranges {0=409600,409600=475136,884736=557056,1441792=573440} 2020-06-12T02:37:31.253 INFO:tasks.rados.rados.0.smithi107.stdout:264: writing smithi10729407-9 from 0 to 409600 tid 1 2020-06-12T02:37:31.256 INFO:tasks.rados.rados.0.smithi107.stdout:264: writing smithi10729407-9 from 409600 to 884736 tid 2 2020-06-12T02:37:31.260 INFO:tasks.rados.rados.0.smithi107.stdout:264: writing smithi10729407-9 from 884736 to 1441792 tid 3 2020-06-12T02:37:31.265 INFO:tasks.rados.rados.0.smithi107.stdout:264: writing smithi10729407-9 from 1441792 to 2015232 tid 4 2020-06-12T02:37:31.265 INFO:tasks.rados.rados.0.smithi107.stdout:265: delete oid 11 current snap is 0 2020-06-12T02:37:31.265 INFO:tasks.rados.rados.0.smithi107.stdout:261: finishing write tid 2 to smithi10729407-17 2020-06-12T02:37:31.265 INFO:tasks.rados.rados.0.smithi107.stdout:261: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:31.266 INFO:tasks.rados.rados.0.smithi107.stdout:261: finishing write tid 5 to smithi10729407-17 2020-06-12T02:37:31.266 INFO:tasks.rados.rados.0.smithi107.stdout:261: finishing write tid 6 to smithi10729407-17 2020-06-12T02:37:31.266 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 50 (ObjNum 260 snap 0 seq_num 260) dirty exists 2020-06-12T02:37:31.266 INFO:tasks.rados.rados.0.smithi107.stdout:261: left oid 17 (ObjNum 260 snap 0 seq_num 260) 2020-06-12T02:37:31.266 INFO:tasks.rados.rados.0.smithi107.stdout:264: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:31.267 INFO:tasks.rados.rados.0.smithi107.stdout:264: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:31.267 INFO:tasks.rados.rados.0.smithi107.stdout:261: done (2 left) 2020-06-12T02:37:31.268 INFO:tasks.rados.rados.0.smithi107.stdout:265: done (1 left) 2020-06-12T02:37:31.268 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:37:31.268 INFO:tasks.rados.rados.0.smithi107.stdout:266: seq_num 265 ranges {0=442368,442368=647168,1089536=573440,1662976=385024} 2020-06-12T02:37:31.269 INFO:tasks.rados.rados.0.smithi107.stdout:266: writing smithi10729407-4 from 0 to 442368 tid 1 2020-06-12T02:37:31.273 INFO:tasks.rados.rados.0.smithi107.stdout:266: writing smithi10729407-4 from 442368 to 1089536 tid 2 2020-06-12T02:37:31.276 INFO:tasks.rados.rados.0.smithi107.stdout:266: writing smithi10729407-4 from 1089536 to 1662976 tid 3 2020-06-12T02:37:31.279 INFO:tasks.rados.rados.0.smithi107.stdout:266: writing smithi10729407-4 from 1662976 to 2048000 tid 4 2020-06-12T02:37:31.279 INFO:tasks.rados.rados.0.smithi107.stdout:267: delete oid 2 current snap is 0 2020-06-12T02:37:31.279 INFO:tasks.rados.rados.0.smithi107.stdout:264: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:31.280 INFO:tasks.rados.rados.0.smithi107.stdout:264: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:31.280 INFO:tasks.rados.rados.0.smithi107.stdout:264: finishing write tid 6 to smithi10729407-9 2020-06-12T02:37:31.280 INFO:tasks.rados.rados.0.smithi107.stdout:264: finishing write tid 7 to smithi10729407-9 2020-06-12T02:37:31.280 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 50 (ObjNum 263 snap 0 seq_num 263) dirty exists 2020-06-12T02:37:31.280 INFO:tasks.rados.rados.0.smithi107.stdout:264: left oid 9 (ObjNum 263 snap 0 seq_num 263) 2020-06-12T02:37:31.281 INFO:tasks.rados.rados.0.smithi107.stdout:266: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:31.281 INFO:tasks.rados.rados.0.smithi107.stdout:266: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:31.282 INFO:tasks.rados.rados.0.smithi107.stdout:264: done (2 left) 2020-06-12T02:37:31.282 INFO:tasks.rados.rados.0.smithi107.stdout:267: done (1 left) 2020-06-12T02:37:31.282 INFO:tasks.rados.rados.0.smithi107.stdout:268: delete oid 2 current snap is 0 2020-06-12T02:37:31.283 INFO:tasks.rados.rados.0.smithi107.stdout:266: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:31.284 INFO:tasks.rados.rados.0.smithi107.stdout:268: done (1 left) 2020-06-12T02:37:31.284 INFO:tasks.rados.rados.0.smithi107.stdout:269: delete oid 3 current snap is 0 2020-06-12T02:37:31.285 INFO:tasks.rados.rados.0.smithi107.stdout:266: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:31.285 INFO:tasks.rados.rados.0.smithi107.stdout:266: finishing write tid 6 to smithi10729407-4 2020-06-12T02:37:31.287 INFO:tasks.rados.rados.0.smithi107.stdout:269: done (1 left) 2020-06-12T02:37:31.287 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:37:31.287 INFO:tasks.rados.rados.0.smithi107.stdout:270: seq_num 269 ranges {0=507904,507904=630784,1138688=557056,1695744=417792} 2020-06-12T02:37:31.290 INFO:tasks.rados.rados.0.smithi107.stdout:270: writing smithi10729407-13 from 0 to 507904 tid 1 2020-06-12T02:37:31.299 INFO:tasks.rados.rados.0.smithi107.stdout:270: writing smithi10729407-13 from 507904 to 1138688 tid 2 2020-06-12T02:37:31.300 INFO:tasks.rados.rados.0.smithi107.stdout:270: writing smithi10729407-13 from 1138688 to 1695744 tid 3 2020-06-12T02:37:31.301 INFO:tasks.rados.rados.0.smithi107.stdout:270: writing smithi10729407-13 from 1695744 to 2113536 tid 4 2020-06-12T02:37:31.302 INFO:tasks.rados.rados.0.smithi107.stdout:271: delete oid 2 current snap is 0 2020-06-12T02:37:31.302 INFO:tasks.rados.rados.0.smithi107.stdout:266: finishing write tid 7 to smithi10729407-4 2020-06-12T02:37:31.303 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 55 (ObjNum 265 snap 0 seq_num 265) dirty exists 2020-06-12T02:37:31.303 INFO:tasks.rados.rados.0.smithi107.stdout:266: left oid 4 (ObjNum 265 snap 0 seq_num 265) 2020-06-12T02:37:31.303 INFO:tasks.rados.rados.0.smithi107.stdout:270: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:31.305 INFO:tasks.rados.rados.0.smithi107.stdout:270: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:31.305 INFO:tasks.rados.rados.0.smithi107.stdout:266: done (2 left) 2020-06-12T02:37:31.305 INFO:tasks.rados.rados.0.smithi107.stdout:271: done (1 left) 2020-06-12T02:37:31.306 INFO:tasks.rados.rados.0.smithi107.stdout:272: delete oid 1 current snap is 0 2020-06-12T02:37:31.306 INFO:tasks.rados.rados.0.smithi107.stdout:272: done (1 left) 2020-06-12T02:37:31.307 INFO:tasks.rados.rados.0.smithi107.stdout:273: delete oid 4 current snap is 0 2020-06-12T02:37:31.307 INFO:tasks.rados.rados.0.smithi107.stdout:270: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:31.309 INFO:tasks.rados.rados.0.smithi107.stdout:270: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:31.309 INFO:tasks.rados.rados.0.smithi107.stdout:270: finishing write tid 6 to smithi10729407-13 2020-06-12T02:37:31.310 INFO:tasks.rados.rados.0.smithi107.stdout:270: finishing write tid 7 to smithi10729407-13 2020-06-12T02:37:31.311 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 54 (ObjNum 269 snap 0 seq_num 269) dirty exists 2020-06-12T02:37:31.311 INFO:tasks.rados.rados.0.smithi107.stdout:270: left oid 13 (ObjNum 269 snap 0 seq_num 269) 2020-06-12T02:37:31.312 INFO:tasks.rados.rados.0.smithi107.stdout:270: done (1 left) 2020-06-12T02:37:31.312 INFO:tasks.rados.rados.0.smithi107.stdout:273: done (0 left) 2020-06-12T02:37:31.312 INFO:tasks.rados.rados.0.smithi107.stdout:274: delete oid 14 current snap is 0 2020-06-12T02:37:31.314 INFO:tasks.rados.rados.0.smithi107.stdout:274: done (0 left) 2020-06-12T02:37:31.315 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:37:31.315 INFO:tasks.rados.rados.0.smithi107.stdout:275: seq_num 274 ranges {0=598016,598016=499712,1097728=745472,1843200=360448} 2020-06-12T02:37:31.318 INFO:tasks.rados.rados.0.smithi107.stdout:275: writing smithi10729407-5 from 0 to 598016 tid 1 2020-06-12T02:37:31.321 INFO:tasks.rados.rados.0.smithi107.stdout:275: writing smithi10729407-5 from 598016 to 1097728 tid 2 2020-06-12T02:37:31.325 INFO:tasks.rados.rados.0.smithi107.stdout:275: writing smithi10729407-5 from 1097728 to 1843200 tid 3 2020-06-12T02:37:31.327 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:31.327 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:31.328 INFO:tasks.rados.rados.0.smithi107.stdout:275: writing smithi10729407-5 from 1843200 to 2203648 tid 4 2020-06-12T02:37:31.329 INFO:tasks.rados.rados.0.smithi107.stdout:276: delete oid 20 current snap is 0 2020-06-12T02:37:31.329 INFO:tasks.rados.rados.0.smithi107.stdout:275: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:31.329 INFO:tasks.rados.rados.0.smithi107.stdout:275: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:31.330 INFO:tasks.rados.rados.0.smithi107.stdout:276: done (1 left) 2020-06-12T02:37:31.331 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:37:31.331 INFO:tasks.rados.rados.0.smithi107.stdout:277: seq_num 276 ranges {0=630784,630784=671744,1302528=704512,2007040=229376} 2020-06-12T02:37:31.333 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:37:31.331+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:37:31.333 INFO:tasks.rados.rados.0.smithi107.stdout:277: writing smithi10729407-14 from 0 to 630784 tid 1 2020-06-12T02:37:31.339 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-06-12T02:37:31.346 INFO:tasks.rados.rados.0.smithi107.stdout:277: writing smithi10729407-14 from 630784 to 1302528 tid 2 2020-06-12T02:37:31.348 INFO:tasks.rados.rados.0.smithi107.stdout:277: writing smithi10729407-14 from 1302528 to 2007040 tid 3 2020-06-12T02:37:31.349 INFO:tasks.rados.rados.0.smithi107.stdout:277: writing smithi10729407-14 from 2007040 to 2236416 tid 4 2020-06-12T02:37:31.349 INFO:tasks.rados.rados.0.smithi107.stdout:278: delete oid 9 current snap is 0 2020-06-12T02:37:31.349 INFO:tasks.rados.rados.0.smithi107.stdout:275: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:31.352 INFO:tasks.rados.rados.0.smithi107.stdout:275: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:31.353 INFO:tasks.rados.rados.0.smithi107.stdout:275: finishing write tid 6 to smithi10729407-5 2020-06-12T02:37:31.353 INFO:tasks.rados.rados.0.smithi107.stdout:275: finishing write tid 7 to smithi10729407-5 2020-06-12T02:37:31.353 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 31 (ObjNum 274 snap 0 seq_num 274) dirty exists 2020-06-12T02:37:31.354 INFO:tasks.rados.rados.0.smithi107.stdout:275: left oid 5 (ObjNum 274 snap 0 seq_num 274) 2020-06-12T02:37:31.354 INFO:tasks.rados.rados.0.smithi107.stdout:277: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:31.356 INFO:tasks.rados.rados.0.smithi107.stdout:275: done (2 left) 2020-06-12T02:37:31.356 INFO:tasks.rados.rados.0.smithi107.stdout:278: done (1 left) 2020-06-12T02:37:31.356 INFO:tasks.rados.rados.0.smithi107.stdout:279: delete oid 5 current snap is 0 2020-06-12T02:37:31.356 INFO:tasks.rados.rados.0.smithi107.stdout:277: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:31.358 INFO:tasks.rados.rados.0.smithi107.stdout:279: done (1 left) 2020-06-12T02:37:31.358 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:31.358 INFO:tasks.rados.rados.0.smithi107.stdout:280: seq_num 279 ranges {0=679936,679936=720896,1400832=483328,1884160=401408} 2020-06-12T02:37:31.359 INFO:tasks.rados.rados.0.smithi107.stdout:280: writing smithi10729407-2 from 0 to 679936 tid 1 2020-06-12T02:37:31.360 INFO:tasks.rados.rados.0.smithi107.stdout:280: writing smithi10729407-2 from 679936 to 1400832 tid 2 2020-06-12T02:37:31.361 INFO:tasks.rados.rados.0.smithi107.stdout:280: writing smithi10729407-2 from 1400832 to 1884160 tid 3 2020-06-12T02:37:31.362 INFO:teuthology.orchestra.run.smithi188.stderr:Nothing to do for non-filestore backend 2020-06-12T02:37:31.363 INFO:tasks.rados.rados.0.smithi107.stdout:280: writing smithi10729407-2 from 1884160 to 2285568 tid 4 2020-06-12T02:37:31.363 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:37:31.364 INFO:tasks.rados.rados.0.smithi107.stdout:281: seq_num 280 ranges {0=696320,696320=606208,1302528=688128,1990656=311296} 2020-06-12T02:37:31.366 INFO:tasks.rados.rados.0.smithi107.stdout:281: writing smithi10729407-5 from 0 to 696320 tid 1 2020-06-12T02:37:31.370 INFO:tasks.rados.rados.0.smithi107.stdout:281: writing smithi10729407-5 from 696320 to 1302528 tid 2 2020-06-12T02:37:31.375 INFO:tasks.rados.rados.0.smithi107.stdout:281: writing smithi10729407-5 from 1302528 to 1990656 tid 3 2020-06-12T02:37:31.376 INFO:tasks.rados.rados.0.smithi107.stdout:281: writing smithi10729407-5 from 1990656 to 2301952 tid 4 2020-06-12T02:37:31.377 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:37:31.378 INFO:tasks.rados.rados.0.smithi107.stdout:282: seq_num 281 ranges {0=712704,712704=491520,1204224=491520,1695744=622592} 2020-06-12T02:37:31.380 INFO:tasks.rados.rados.0.smithi107.stdout:282: writing smithi10729407-9 from 0 to 712704 tid 1 2020-06-12T02:37:31.386 INFO:tasks.rados.rados.0.smithi107.stdout:282: writing smithi10729407-9 from 712704 to 1204224 tid 2 2020-06-12T02:37:31.387 INFO:tasks.rados.rados.0.smithi107.stdout:282: writing smithi10729407-9 from 1204224 to 1695744 tid 3 2020-06-12T02:37:31.392 INFO:tasks.rados.rados.0.smithi107.stdout:282: writing smithi10729407-9 from 1695744 to 2318336 tid 4 2020-06-12T02:37:31.392 INFO:tasks.rados.rados.0.smithi107.stdout:283: delete oid 19 current snap is 0 2020-06-12T02:37:31.392 INFO:tasks.rados.rados.0.smithi107.stdout:277: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:31.393 INFO:tasks.rados.rados.0.smithi107.stdout:277: finishing write tid 4 to smithi10729407-14 2020-06-12T02:37:31.393 INFO:tasks.rados.rados.0.smithi107.stdout:277: finishing write tid 6 to smithi10729407-14 2020-06-12T02:37:31.393 INFO:tasks.rados.rados.0.smithi107.stdout:277: finishing write tid 7 to smithi10729407-14 2020-06-12T02:37:31.394 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 32 (ObjNum 276 snap 0 seq_num 276) dirty exists 2020-06-12T02:37:31.394 INFO:tasks.rados.rados.0.smithi107.stdout:277: left oid 14 (ObjNum 276 snap 0 seq_num 276) 2020-06-12T02:37:31.394 INFO:tasks.rados.rados.0.smithi107.stdout:280: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:31.395 INFO:tasks.rados.rados.0.smithi107.stdout:280: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:31.395 INFO:tasks.rados.rados.0.smithi107.stdout:280: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:31.395 INFO:tasks.rados.rados.0.smithi107.stdout:280: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:31.396 INFO:tasks.rados.rados.0.smithi107.stdout:280: finishing write tid 6 to smithi10729407-2 2020-06-12T02:37:31.396 INFO:tasks.rados.rados.0.smithi107.stdout:280: finishing write tid 7 to smithi10729407-2 2020-06-12T02:37:31.396 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 38 (ObjNum 279 snap 0 seq_num 279) dirty exists 2020-06-12T02:37:31.397 INFO:tasks.rados.rados.0.smithi107.stdout:280: left oid 2 (ObjNum 279 snap 0 seq_num 279) 2020-06-12T02:37:31.397 INFO:tasks.rados.rados.0.smithi107.stdout:281: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:31.397 INFO:tasks.rados.rados.0.smithi107.stdout:281: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:31.398 INFO:tasks.rados.rados.0.smithi107.stdout:281: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:31.398 INFO:tasks.rados.rados.0.smithi107.stdout:281: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:31.398 INFO:tasks.rados.rados.0.smithi107.stdout:281: finishing write tid 6 to smithi10729407-5 2020-06-12T02:37:31.399 INFO:tasks.rados.rados.0.smithi107.stdout:281: finishing write tid 7 to smithi10729407-5 2020-06-12T02:37:31.399 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 37 (ObjNum 280 snap 0 seq_num 280) dirty exists 2020-06-12T02:37:31.399 INFO:tasks.rados.rados.0.smithi107.stdout:281: left oid 5 (ObjNum 280 snap 0 seq_num 280) 2020-06-12T02:37:31.399 INFO:tasks.rados.rados.0.smithi107.stdout:282: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:31.401 INFO:tasks.rados.rados.0.smithi107.stdout:277: done (4 left) 2020-06-12T02:37:31.402 INFO:tasks.rados.rados.0.smithi107.stdout:280: done (3 left) 2020-06-12T02:37:31.402 INFO:tasks.rados.rados.0.smithi107.stdout:281: done (2 left) 2020-06-12T02:37:31.402 INFO:tasks.rados.rados.0.smithi107.stdout:283: done (1 left) 2020-06-12T02:37:31.402 INFO:tasks.rados.rados.0.smithi107.stdout:284: delete oid 17 current snap is 0 2020-06-12T02:37:31.403 INFO:tasks.rados.rados.0.smithi107.stdout:282: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:31.403 INFO:tasks.rados.rados.0.smithi107.stdout:284: done (1 left) 2020-06-12T02:37:31.403 INFO:tasks.rados.rados.0.smithi107.stdout:285: delete oid 12 current snap is 0 2020-06-12T02:37:31.404 INFO:tasks.rados.rados.0.smithi107.stdout:282: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:31.404 INFO:tasks.rados.rados.0.smithi107.stdout:282: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:31.404 INFO:tasks.rados.rados.0.smithi107.stdout:285: done (1 left) 2020-06-12T02:37:31.405 INFO:tasks.rados.rados.0.smithi107.stdout:286: delete oid 4 current snap is 0 2020-06-12T02:37:31.405 INFO:tasks.rados.rados.0.smithi107.stdout:282: finishing write tid 6 to smithi10729407-9 2020-06-12T02:37:31.406 INFO:tasks.rados.rados.0.smithi107.stdout:282: finishing write tid 7 to smithi10729407-9 2020-06-12T02:37:31.406 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 56 (ObjNum 281 snap 0 seq_num 281) dirty exists 2020-06-12T02:37:31.406 INFO:tasks.rados.rados.0.smithi107.stdout:282: left oid 9 (ObjNum 281 snap 0 seq_num 281) 2020-06-12T02:37:31.407 INFO:tasks.rados.rados.0.smithi107.stdout:282: done (1 left) 2020-06-12T02:37:31.407 INFO:tasks.rados.rados.0.smithi107.stdout:286: done (0 left) 2020-06-12T02:37:31.407 INFO:tasks.rados.rados.0.smithi107.stdout:287: delete oid 17 current snap is 0 2020-06-12T02:37:31.407 INFO:tasks.rados.rados.0.smithi107.stdout:287: done (0 left) 2020-06-12T02:37:31.407 INFO:tasks.rados.rados.0.smithi107.stdout:288: delete oid 9 current snap is 0 2020-06-12T02:37:31.407 INFO:tasks.rados.rados.0.smithi107.stdout:288: done (0 left) 2020-06-12T02:37:31.408 INFO:tasks.rados.rados.0.smithi107.stdout:289: delete oid 7 current snap is 0 2020-06-12T02:37:31.408 INFO:tasks.rados.rados.0.smithi107.stdout:289: done (0 left) 2020-06-12T02:37:31.408 INFO:tasks.rados.rados.0.smithi107.stdout:290: delete oid 7 current snap is 0 2020-06-12T02:37:31.410 INFO:tasks.rados.rados.0.smithi107.stdout:290: done (0 left) 2020-06-12T02:37:31.410 INFO:tasks.rados.rados.0.smithi107.stdout:291: delete oid 19 current snap is 0 2020-06-12T02:37:31.411 INFO:tasks.rados.rados.0.smithi107.stdout:291: done (0 left) 2020-06-12T02:37:31.411 INFO:tasks.rados.rados.0.smithi107.stdout:292: delete oid 14 current snap is 0 2020-06-12T02:37:31.414 INFO:tasks.rados.rados.0.smithi107.stdout:292: done (0 left) 2020-06-12T02:37:31.414 INFO:tasks.rados.rados.0.smithi107.stdout:293: delete oid 15 current snap is 0 2020-06-12T02:37:31.415 INFO:tasks.rados.rados.0.smithi107.stdout:293: done (0 left) 2020-06-12T02:37:31.416 INFO:tasks.rados.rados.0.smithi107.stdout:294: delete oid 18 current snap is 0 2020-06-12T02:37:31.418 INFO:tasks.rados.rados.0.smithi107.stdout:294: done (0 left) 2020-06-12T02:37:31.418 INFO:tasks.rados.rados.0.smithi107.stdout:295: delete oid 5 current snap is 0 2020-06-12T02:37:31.420 INFO:tasks.rados.rados.0.smithi107.stdout:295: done (0 left) 2020-06-12T02:37:31.420 INFO:tasks.rados.rados.0.smithi107.stdout:296: delete oid 15 current snap is 0 2020-06-12T02:37:31.422 INFO:tasks.rados.rados.0.smithi107.stdout:296: done (0 left) 2020-06-12T02:37:31.422 INFO:tasks.rados.rados.0.smithi107.stdout:297: delete oid 8 current snap is 0 2020-06-12T02:37:31.423 INFO:tasks.rados.rados.0.smithi107.stdout:297: done (0 left) 2020-06-12T02:37:31.423 INFO:tasks.rados.rados.0.smithi107.stdout:298: delete oid 15 current snap is 0 2020-06-12T02:37:31.425 INFO:tasks.rados.rados.0.smithi107.stdout:298: done (0 left) 2020-06-12T02:37:31.425 INFO:tasks.rados.rados.0.smithi107.stdout:299: delete oid 17 current snap is 0 2020-06-12T02:37:31.426 INFO:tasks.rados.rados.0.smithi107.stdout:299: done (0 left) 2020-06-12T02:37:31.426 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:37:31.427 INFO:tasks.rados.rados.0.smithi107.stdout:300: seq_num 299 ranges {2285568=212992} 2020-06-12T02:37:31.434 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:31.431+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:31.440 INFO:tasks.rados.rados.0.smithi107.stdout:300: writing smithi10729407-2 from 2285568 to 2498560 tid 1 2020-06-12T02:37:31.440 INFO:tasks.rados.rados.0.smithi107.stdout:301: delete oid 19 current snap is 0 2020-06-12T02:37:31.441 INFO:tasks.rados.rados.0.smithi107.stdout:301: done (1 left) 2020-06-12T02:37:31.442 INFO:tasks.rados.rados.0.smithi107.stdout:302: delete oid 19 current snap is 0 2020-06-12T02:37:31.443 INFO:tasks.rados.rados.0.smithi107.stdout:300: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:31.443 INFO:tasks.rados.rados.0.smithi107.stdout:302: done (1 left) 2020-06-12T02:37:31.444 INFO:tasks.rados.rados.0.smithi107.stdout:303: delete oid 20 current snap is 0 2020-06-12T02:37:31.444 INFO:tasks.rados.rados.0.smithi107.stdout:300: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:31.445 INFO:tasks.rados.rados.0.smithi107.stdout:300: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:31.445 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 40 (ObjNum 299 snap 0 seq_num 299) dirty exists 2020-06-12T02:37:31.446 INFO:tasks.rados.rados.0.smithi107.stdout:300: left oid 2 (ObjNum 299 snap 0 seq_num 299) 2020-06-12T02:37:31.446 INFO:tasks.rados.rados.0.smithi107.stdout:300: done (1 left) 2020-06-12T02:37:31.446 INFO:tasks.rados.rados.0.smithi107.stdout:303: done (0 left) 2020-06-12T02:37:31.446 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:37:31.446 INFO:tasks.rados.rados.0.smithi107.stdout:304: seq_num 303 ranges {0=278528} 2020-06-12T02:37:31.447 INFO:tasks.rados.rados.0.smithi107.stdout:304: writing smithi10729407-15 from 0 to 278528 tid 1 2020-06-12T02:37:31.447 INFO:tasks.rados.rados.0.smithi107.stdout:305: delete oid 12 current snap is 0 2020-06-12T02:37:31.448 INFO:tasks.rados.rados.0.smithi107.stdout:305: done (1 left) 2020-06-12T02:37:31.448 INFO:tasks.rados.rados.0.smithi107.stdout:306: delete oid 13 current snap is 0 2020-06-12T02:37:31.450 INFO:tasks.rados.rados.0.smithi107.stdout:306: done (1 left) 2020-06-12T02:37:31.450 INFO:tasks.rados.rados.0.smithi107.stdout:307: delete oid 12 current snap is 0 2020-06-12T02:37:31.451 INFO:tasks.rados.rados.0.smithi107.stdout:307: done (1 left) 2020-06-12T02:37:31.452 INFO:tasks.rados.rados.0.smithi107.stdout:308: delete oid 17 current snap is 0 2020-06-12T02:37:31.452 INFO:tasks.rados.rados.0.smithi107.stdout:304: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:31.453 INFO:tasks.rados.rados.0.smithi107.stdout:308: done (1 left) 2020-06-12T02:37:31.461 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:37:31.461 INFO:tasks.rados.rados.0.smithi107.stdout:309: seq_num 308 ranges {0=360448} 2020-06-12T02:37:31.462 INFO:tasks.rados.rados.0.smithi107.stdout:309: writing smithi10729407-18 from 0 to 360448 tid 1 2020-06-12T02:37:31.463 INFO:tasks.rados.rados.0.smithi107.stdout:310: delete oid 7 current snap is 0 2020-06-12T02:37:31.463 INFO:tasks.rados.rados.0.smithi107.stdout:304: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:31.463 INFO:tasks.rados.rados.0.smithi107.stdout:304: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:31.463 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 70 (ObjNum 303 snap 0 seq_num 303) dirty exists 2020-06-12T02:37:31.464 INFO:tasks.rados.rados.0.smithi107.stdout:304: left oid 15 (ObjNum 303 snap 0 seq_num 303) 2020-06-12T02:37:31.464 INFO:tasks.rados.rados.0.smithi107.stdout:304: done (2 left) 2020-06-12T02:37:31.464 INFO:tasks.rados.rados.0.smithi107.stdout:310: done (1 left) 2020-06-12T02:37:31.464 INFO:tasks.rados.rados.0.smithi107.stdout:311: delete oid 9 current snap is 0 2020-06-12T02:37:31.464 INFO:tasks.rados.rados.0.smithi107.stdout:309: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:31.465 INFO:tasks.rados.rados.0.smithi107.stdout:309: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:31.465 INFO:tasks.rados.rados.0.smithi107.stdout:311: done (1 left) 2020-06-12T02:37:31.465 INFO:tasks.rados.rados.0.smithi107.stdout:312: delete oid 1 current snap is 0 2020-06-12T02:37:31.465 INFO:tasks.rados.rados.0.smithi107.stdout:309: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:31.466 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 60 (ObjNum 308 snap 0 seq_num 308) dirty exists 2020-06-12T02:37:31.466 INFO:tasks.rados.rados.0.smithi107.stdout:309: left oid 18 (ObjNum 308 snap 0 seq_num 308) 2020-06-12T02:37:31.467 INFO:tasks.rados.rados.0.smithi107.stdout:309: done (1 left) 2020-06-12T02:37:31.467 INFO:tasks.rados.rados.0.smithi107.stdout:312: done (0 left) 2020-06-12T02:37:31.467 INFO:tasks.rados.rados.0.smithi107.stdout:313: delete oid 3 current snap is 0 2020-06-12T02:37:31.467 INFO:tasks.rados.rados.0.smithi107.stdout:313: done (0 left) 2020-06-12T02:37:31.467 INFO:tasks.rados.rados.0.smithi107.stdout:314: delete oid 1 current snap is 0 2020-06-12T02:37:31.468 INFO:tasks.rados.rados.0.smithi107.stdout:314: done (0 left) 2020-06-12T02:37:31.468 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:37:31.468 INFO:tasks.rados.rados.0.smithi107.stdout:315: seq_num 314 ranges {0=466944} 2020-06-12T02:37:31.469 INFO:tasks.rados.rados.0.smithi107.stdout:315: writing smithi10729407-1 from 0 to 466944 tid 1 2020-06-12T02:37:31.469 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:37:31.469 INFO:tasks.rados.rados.0.smithi107.stdout:316: seq_num 315 ranges {0=483328} 2020-06-12T02:37:31.472 INFO:tasks.rados.rados.0.smithi107.stdout:316: writing smithi10729407-8 from 0 to 483328 tid 1 2020-06-12T02:37:31.472 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:37:31.472 INFO:tasks.rados.rados.0.smithi107.stdout:317: seq_num 316 ranges {360448=499712} 2020-06-12T02:37:31.474 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-06-12T02:37:31.477 INFO:tasks.rados.rados.0.smithi107.stdout:317: writing smithi10729407-18 from 360448 to 860160 tid 1 2020-06-12T02:37:31.477 INFO:tasks.rados.rados.0.smithi107.stdout:318: delete oid 19 current snap is 0 2020-06-12T02:37:31.478 INFO:tasks.rados.rados.0.smithi107.stdout:315: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:31.478 INFO:tasks.rados.rados.0.smithi107.stdout:315: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:31.478 INFO:tasks.rados.rados.0.smithi107.stdout:316: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:31.478 INFO:tasks.rados.rados.0.smithi107.stdout:315: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:31.478 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 56 (ObjNum 314 snap 0 seq_num 314) dirty exists 2020-06-12T02:37:31.479 INFO:tasks.rados.rados.0.smithi107.stdout:315: left oid 1 (ObjNum 314 snap 0 seq_num 314) 2020-06-12T02:37:31.480 INFO:tasks.rados.rados.0.smithi107.stdout:315: done (3 left) 2020-06-12T02:37:31.480 INFO:tasks.rados.rados.0.smithi107.stdout:318: done (2 left) 2020-06-12T02:37:31.481 INFO:tasks.rados.rados.0.smithi107.stdout:319: delete oid 9 current snap is 0 2020-06-12T02:37:31.484 INFO:tasks.rados.rados.0.smithi107.stdout:316: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:31.484 INFO:tasks.rados.rados.0.smithi107.stdout:317: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:31.484 INFO:tasks.rados.rados.0.smithi107.stdout:317: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:31.485 INFO:tasks.rados.rados.0.smithi107.stdout:316: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:31.485 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 72 (ObjNum 315 snap 0 seq_num 315) dirty exists 2020-06-12T02:37:31.485 INFO:tasks.rados.rados.0.smithi107.stdout:316: left oid 8 (ObjNum 315 snap 0 seq_num 315) 2020-06-12T02:37:31.486 INFO:tasks.rados.rados.0.smithi107.stdout:316: done (2 left) 2020-06-12T02:37:31.486 INFO:tasks.rados.rados.0.smithi107.stdout:319: done (1 left) 2020-06-12T02:37:31.486 INFO:tasks.rados.rados.0.smithi107.stdout:320: delete oid 1 current snap is 0 2020-06-12T02:37:31.487 INFO:tasks.rados.rados.0.smithi107.stdout:317: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:31.487 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 63 (ObjNum 316 snap 0 seq_num 316) dirty exists 2020-06-12T02:37:31.487 INFO:tasks.rados.rados.0.smithi107.stdout:317: left oid 18 (ObjNum 316 snap 0 seq_num 316) 2020-06-12T02:37:31.490 INFO:tasks.rados.rados.0.smithi107.stdout:317: done (1 left) 2020-06-12T02:37:31.490 INFO:tasks.rados.rados.0.smithi107.stdout:320: done (0 left) 2020-06-12T02:37:31.491 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:31.491 INFO:tasks.rados.rados.0.smithi107.stdout:321: seq_num 320 ranges {2498560=565248} 2020-06-12T02:37:31.510 INFO:tasks.rados.rados.0.smithi107.stdout:321: writing smithi10729407-2 from 2498560 to 3063808 tid 1 2020-06-12T02:37:31.510 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:37:31.510 INFO:tasks.rados.rados.0.smithi107.stdout:322: seq_num 321 ranges {0=581632} 2020-06-12T02:37:31.511 INFO:tasks.rados.rados.0.smithi107.stdout:322: writing smithi10729407-6 from 0 to 581632 tid 1 2020-06-12T02:37:31.511 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:37:31.512 INFO:tasks.rados.rados.0.smithi107.stdout:323: seq_num 322 ranges {0=598016} 2020-06-12T02:37:31.522 INFO:tasks.rados.rados.0.smithi107.stdout:323: writing smithi10729407-1 from 0 to 598016 tid 1 2020-06-12T02:37:31.523 INFO:tasks.rados.rados.0.smithi107.stdout:324: delete oid 20 current snap is 0 2020-06-12T02:37:31.523 INFO:tasks.rados.rados.0.smithi107.stdout:321: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:31.523 INFO:tasks.rados.rados.0.smithi107.stdout:324: done (3 left) 2020-06-12T02:37:31.523 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:37:31.523 INFO:tasks.rados.rados.0.smithi107.stdout:325: seq_num 324 ranges {0=630784} 2020-06-12T02:37:31.524 INFO:tasks.rados.rados.0.smithi107.stdout:325: writing smithi10729407-9 from 0 to 630784 tid 1 2020-06-12T02:37:31.524 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:37:31.524 INFO:tasks.rados.rados.0.smithi107.stdout:326: seq_num 325 ranges {0=647168} 2020-06-12T02:37:31.524 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:31.524 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:31.532 INFO:tasks.rados.rados.0.smithi107.stdout:326: writing smithi10729407-17 from 0 to 647168 tid 1 2020-06-12T02:37:31.533 INFO:tasks.rados.rados.0.smithi107.stdout:327: delete oid 14 current snap is 0 2020-06-12T02:37:31.533 INFO:tasks.rados.rados.0.smithi107.stdout:321: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:31.533 INFO:tasks.rados.rados.0.smithi107.stdout:322: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:31.533 INFO:tasks.rados.rados.0.smithi107.stdout:323: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:31.534 INFO:tasks.rados.rados.0.smithi107.stdout:321: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:31.534 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 42 (ObjNum 320 snap 0 seq_num 320) dirty exists 2020-06-12T02:37:31.534 INFO:tasks.rados.rados.0.smithi107.stdout:321: left oid 2 (ObjNum 320 snap 0 seq_num 320) 2020-06-12T02:37:31.534 INFO:tasks.rados.rados.0.smithi107.stdout:323: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:31.534 INFO:tasks.rados.rados.0.smithi107.stdout:322: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:31.535 INFO:tasks.rados.rados.0.smithi107.stdout:321: done (5 left) 2020-06-12T02:37:31.535 INFO:tasks.rados.rados.0.smithi107.stdout:327: done (4 left) 2020-06-12T02:37:31.535 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:37:31.535 INFO:tasks.rados.rados.0.smithi107.stdout:328: seq_num 327 ranges {0=679936} 2020-06-12T02:37:31.536 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-06-12T02:37:31.538 INFO:tasks.rados.rados.0.smithi107.stdout:328: writing smithi10729407-4 from 0 to 679936 tid 1 2020-06-12T02:37:31.538 INFO:tasks.rados.rados.0.smithi107.stdout:329: delete oid 11 current snap is 0 2020-06-12T02:37:31.538 INFO:tasks.rados.rados.0.smithi107.stdout:325: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:31.539 INFO:tasks.rados.rados.0.smithi107.stdout:323: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:31.539 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 60 (ObjNum 322 snap 0 seq_num 322) dirty exists 2020-06-12T02:37:31.539 INFO:tasks.rados.rados.0.smithi107.stdout:323: left oid 1 (ObjNum 322 snap 0 seq_num 322) 2020-06-12T02:37:31.539 INFO:tasks.rados.rados.0.smithi107.stdout:322: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:31.539 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 61 (ObjNum 321 snap 0 seq_num 321) dirty exists 2020-06-12T02:37:31.540 INFO:tasks.rados.rados.0.smithi107.stdout:322: left oid 6 (ObjNum 321 snap 0 seq_num 321) 2020-06-12T02:37:31.540 INFO:tasks.rados.rados.0.smithi107.stdout:326: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:31.540 INFO:tasks.rados.rados.0.smithi107.stdout:326: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:31.540 INFO:tasks.rados.rados.0.smithi107.stdout:325: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:31.540 INFO:tasks.rados.rados.0.smithi107.stdout:325: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:31.540 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 66 (ObjNum 324 snap 0 seq_num 324) dirty exists 2020-06-12T02:37:31.541 INFO:tasks.rados.rados.0.smithi107.stdout:325: left oid 9 (ObjNum 324 snap 0 seq_num 324) 2020-06-12T02:37:31.541 INFO:tasks.rados.rados.0.smithi107.stdout:326: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:31.541 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 63 (ObjNum 325 snap 0 seq_num 325) dirty exists 2020-06-12T02:37:31.541 INFO:tasks.rados.rados.0.smithi107.stdout:326: left oid 17 (ObjNum 325 snap 0 seq_num 325) 2020-06-12T02:37:31.542 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:31.535+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:31.543 INFO:tasks.rados.rados.0.smithi107.stdout:322: done (5 left) 2020-06-12T02:37:31.544 INFO:tasks.rados.rados.0.smithi107.stdout:323: done (4 left) 2020-06-12T02:37:31.544 INFO:tasks.rados.rados.0.smithi107.stdout:325: done (3 left) 2020-06-12T02:37:31.544 INFO:tasks.rados.rados.0.smithi107.stdout:326: done (2 left) 2020-06-12T02:37:31.544 INFO:tasks.rados.rados.0.smithi107.stdout:329: done (1 left) 2020-06-12T02:37:31.544 INFO:tasks.rados.rados.0.smithi107.stdout:330: delete oid 1 current snap is 0 2020-06-12T02:37:31.545 INFO:tasks.rados.rados.0.smithi107.stdout:328: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:31.545 INFO:tasks.rados.rados.0.smithi107.stdout:330: done (1 left) 2020-06-12T02:37:31.545 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:37:31.545 INFO:tasks.rados.rados.0.smithi107.stdout:331: seq_num 330 ranges {0=737280} 2020-06-12T02:37:31.546 INFO:tasks.rados.rados.0.smithi107.stdout:331: writing smithi10729407-7 from 0 to 737280 tid 1 2020-06-12T02:37:31.547 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:37:31.547 INFO:tasks.rados.rados.0.smithi107.stdout:332: seq_num 331 ranges {0=753664} 2020-06-12T02:37:31.551 INFO:tasks.rados.rados.0.smithi107.stdout:332: writing smithi10729407-19 from 0 to 753664 tid 1 2020-06-12T02:37:31.551 INFO:tasks.rados.rados.0.smithi107.stdout:333: delete oid 14 current snap is 0 2020-06-12T02:37:31.551 INFO:tasks.rados.rados.0.smithi107.stdout:328: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:31.551 INFO:tasks.rados.rados.0.smithi107.stdout:328: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:31.552 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 65 (ObjNum 327 snap 0 seq_num 327) dirty exists 2020-06-12T02:37:31.552 INFO:tasks.rados.rados.0.smithi107.stdout:328: left oid 4 (ObjNum 327 snap 0 seq_num 327) 2020-06-12T02:37:31.552 INFO:tasks.rados.rados.0.smithi107.stdout:328: done (3 left) 2020-06-12T02:37:31.553 INFO:tasks.rados.rados.0.smithi107.stdout:333: done (2 left) 2020-06-12T02:37:31.553 INFO:tasks.rados.rados.0.smithi107.stdout:334: delete oid 10 current snap is 0 2020-06-12T02:37:31.554 INFO:tasks.rados.rados.0.smithi107.stdout:334: done (2 left) 2020-06-12T02:37:31.554 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:37:31.555 INFO:tasks.rados.rados.0.smithi107.stdout:335: seq_num 334 ranges {0=802816} 2020-06-12T02:37:31.560 INFO:tasks.rados.rados.0.smithi107.stdout:335: writing smithi10729407-1 from 0 to 802816 tid 1 2020-06-12T02:37:31.561 INFO:tasks.rados.rados.0.smithi107.stdout:336: delete oid 12 current snap is 0 2020-06-12T02:37:31.561 INFO:tasks.rados.rados.0.smithi107.stdout:331: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:31.561 INFO:tasks.rados.rados.0.smithi107.stdout:332: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:31.561 INFO:tasks.rados.rados.0.smithi107.stdout:331: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:31.561 INFO:tasks.rados.rados.0.smithi107.stdout:331: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:31.562 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 60 (ObjNum 330 snap 0 seq_num 330) dirty exists 2020-06-12T02:37:31.562 INFO:tasks.rados.rados.0.smithi107.stdout:331: left oid 7 (ObjNum 330 snap 0 seq_num 330) 2020-06-12T02:37:31.562 INFO:tasks.rados.rados.0.smithi107.stdout:332: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:31.563 INFO:tasks.rados.rados.0.smithi107.stdout:331: done (3 left) 2020-06-12T02:37:31.563 INFO:tasks.rados.rados.0.smithi107.stdout:336: done (2 left) 2020-06-12T02:37:31.563 INFO:tasks.rados.rados.0.smithi107.stdout:337: delete oid 2 current snap is 0 2020-06-12T02:37:31.563 INFO:tasks.rados.rados.0.smithi107.stdout:332: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:31.564 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 51 (ObjNum 331 snap 0 seq_num 331) dirty exists 2020-06-12T02:37:31.564 INFO:tasks.rados.rados.0.smithi107.stdout:332: left oid 19 (ObjNum 331 snap 0 seq_num 331) 2020-06-12T02:37:31.567 INFO:tasks.rados.rados.0.smithi107.stdout:332: done (2 left) 2020-06-12T02:37:31.567 INFO:tasks.rados.rados.0.smithi107.stdout:337: done (1 left) 2020-06-12T02:37:31.567 INFO:tasks.rados.rados.0.smithi107.stdout:338: delete oid 17 current snap is 0 2020-06-12T02:37:31.568 INFO:tasks.rados.rados.0.smithi107.stdout:335: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:31.569 INFO:tasks.rados.rados.0.smithi107.stdout:338: done (1 left) 2020-06-12T02:37:31.569 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:37:31.569 INFO:tasks.rados.rados.0.smithi107.stdout:339: seq_num 338 ranges {679936=466944,1146880=401408} 2020-06-12T02:37:31.575 INFO:tasks.rados.rados.0.smithi107.stdout:339: writing smithi10729407-4 from 679936 to 1146880 tid 1 2020-06-12T02:37:31.578 INFO:tasks.rados.rados.0.smithi107.stdout:339: writing smithi10729407-4 from 1146880 to 1548288 tid 2 2020-06-12T02:37:31.578 INFO:tasks.rados.rados.0.smithi107.stdout:340: delete oid 19 current snap is 0 2020-06-12T02:37:31.579 INFO:tasks.rados.rados.0.smithi107.stdout:335: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:31.579 INFO:tasks.rados.rados.0.smithi107.stdout:335: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:31.579 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 64 (ObjNum 334 snap 0 seq_num 334) dirty exists 2020-06-12T02:37:31.579 INFO:tasks.rados.rados.0.smithi107.stdout:335: left oid 1 (ObjNum 334 snap 0 seq_num 334) 2020-06-12T02:37:31.581 INFO:tasks.rados.rados.0.smithi107.stdout:335: done (2 left) 2020-06-12T02:37:31.581 INFO:tasks.rados.rados.0.smithi107.stdout:340: done (1 left) 2020-06-12T02:37:31.581 INFO:tasks.rados.rados.0.smithi107.stdout:341: delete oid 11 current snap is 0 2020-06-12T02:37:31.582 INFO:tasks.rados.rados.0.smithi107.stdout:339: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:31.583 INFO:tasks.rados.rados.0.smithi107.stdout:339: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:31.583 INFO:tasks.rados.rados.0.smithi107.stdout:341: done (1 left) 2020-06-12T02:37:31.584 INFO:tasks.rados.rados.0.smithi107.stdout:342: delete oid 19 current snap is 0 2020-06-12T02:37:31.584 INFO:tasks.rados.rados.0.smithi107.stdout:339: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:31.584 INFO:tasks.rados.rados.0.smithi107.stdout:339: finishing write tid 5 to smithi10729407-4 2020-06-12T02:37:31.585 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 69 (ObjNum 338 snap 0 seq_num 338) dirty exists 2020-06-12T02:37:31.585 INFO:tasks.rados.rados.0.smithi107.stdout:339: left oid 4 (ObjNum 338 snap 0 seq_num 338) 2020-06-12T02:37:31.585 INFO:tasks.rados.rados.0.smithi107.stdout:339: done (1 left) 2020-06-12T02:37:31.586 INFO:tasks.rados.rados.0.smithi107.stdout:342: done (0 left) 2020-06-12T02:37:31.586 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:37:31.586 INFO:tasks.rados.rados.0.smithi107.stdout:343: seq_num 342 ranges {0=532480,532480=401408} 2020-06-12T02:37:31.588 INFO:tasks.rados.rados.0.smithi107.stdout:343: writing smithi10729407-12 from 0 to 532480 tid 1 2020-06-12T02:37:31.590 INFO:tasks.rados.rados.0.smithi107.stdout:343: writing smithi10729407-12 from 532480 to 933888 tid 2 2020-06-12T02:37:31.590 INFO:tasks.rados.rados.0.smithi107.stdout:344: delete oid 3 current snap is 0 2020-06-12T02:37:31.592 INFO:tasks.rados.rados.0.smithi107.stdout:344: done (1 left) 2020-06-12T02:37:31.592 INFO:tasks.rados.rados.0.smithi107.stdout:345: delete oid 1 current snap is 0 2020-06-12T02:37:31.594 INFO:tasks.rados.rados.0.smithi107.stdout:345: done (1 left) 2020-06-12T02:37:31.594 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:37:31.594 INFO:tasks.rados.rados.0.smithi107.stdout:346: seq_num 345 ranges {0=581632,581632=401408} 2020-06-12T02:37:31.598 INFO:tasks.rados.rados.0.smithi107.stdout:346: writing smithi10729407-5 from 0 to 581632 tid 1 2020-06-12T02:37:31.601 INFO:tasks.rados.rados.0.smithi107.stdout:346: writing smithi10729407-5 from 581632 to 983040 tid 2 2020-06-12T02:37:31.602 INFO:tasks.rados.rados.0.smithi107.stdout:347: delete oid 10 current snap is 0 2020-06-12T02:37:31.602 INFO:tasks.rados.rados.0.smithi107.stdout:343: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:31.602 INFO:tasks.rados.rados.0.smithi107.stdout:343: finishing write tid 2 to smithi10729407-12 2020-06-12T02:37:31.602 INFO:tasks.rados.rados.0.smithi107.stdout:343: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:31.602 INFO:tasks.rados.rados.0.smithi107.stdout:343: finishing write tid 5 to smithi10729407-12 2020-06-12T02:37:31.603 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 57 (ObjNum 342 snap 0 seq_num 342) dirty exists 2020-06-12T02:37:31.603 INFO:tasks.rados.rados.0.smithi107.stdout:343: left oid 12 (ObjNum 342 snap 0 seq_num 342) 2020-06-12T02:37:31.603 INFO:tasks.rados.rados.0.smithi107.stdout:343: done (2 left) 2020-06-12T02:37:31.604 INFO:tasks.rados.rados.0.smithi107.stdout:347: done (1 left) 2020-06-12T02:37:31.604 INFO:tasks.rados.rados.0.smithi107.stdout:348: delete oid 4 current snap is 0 2020-06-12T02:37:31.606 INFO:tasks.rados.rados.0.smithi107.stdout:346: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:31.606 INFO:tasks.rados.rados.0.smithi107.stdout:348: done (1 left) 2020-06-12T02:37:31.606 INFO:tasks.rados.rados.0.smithi107.stdout:349: delete oid 20 current snap is 0 2020-06-12T02:37:31.607 INFO:tasks.rados.rados.0.smithi107.stdout:346: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:31.607 INFO:tasks.rados.rados.0.smithi107.stdout:346: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:31.609 INFO:tasks.rados.rados.0.smithi107.stdout:349: done (1 left) 2020-06-12T02:37:31.609 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:37:31.610 INFO:tasks.rados.rados.0.smithi107.stdout:350: seq_num 349 ranges {0=655360,655360=401408} 2020-06-12T02:37:31.612 INFO:tasks.rados.rados.0.smithi107.stdout:350: writing smithi10729407-16 from 0 to 655360 tid 1 2020-06-12T02:37:31.615 INFO:tasks.rados.rados.0.smithi107.stdout:350: writing smithi10729407-16 from 655360 to 1056768 tid 2 2020-06-12T02:37:31.615 INFO:tasks.rados.rados.0.smithi107.stdout:351: delete oid 19 current snap is 0 2020-06-12T02:37:31.615 INFO:tasks.rados.rados.0.smithi107.stdout:346: finishing write tid 5 to smithi10729407-5 2020-06-12T02:37:31.615 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 41 (ObjNum 345 snap 0 seq_num 345) dirty exists 2020-06-12T02:37:31.616 INFO:tasks.rados.rados.0.smithi107.stdout:346: left oid 5 (ObjNum 345 snap 0 seq_num 345) 2020-06-12T02:37:31.619 INFO:tasks.rados.rados.0.smithi107.stdout:346: done (2 left) 2020-06-12T02:37:31.620 INFO:tasks.rados.rados.0.smithi107.stdout:351: done (1 left) 2020-06-12T02:37:31.620 INFO:tasks.rados.rados.0.smithi107.stdout:352: delete oid 8 current snap is 0 2020-06-12T02:37:31.620 INFO:tasks.rados.rados.0.smithi107.stdout:350: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:31.621 INFO:tasks.rados.rados.0.smithi107.stdout:350: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:31.621 INFO:tasks.rados.rados.0.smithi107.stdout:350: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:31.622 INFO:tasks.rados.rados.0.smithi107.stdout:352: done (1 left) 2020-06-12T02:37:31.622 INFO:tasks.rados.rados.0.smithi107.stdout:353: delete oid 11 current snap is 0 2020-06-12T02:37:31.623 INFO:tasks.rados.rados.0.smithi107.stdout:350: finishing write tid 5 to smithi10729407-16 2020-06-12T02:37:31.623 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 49 (ObjNum 349 snap 0 seq_num 349) dirty exists 2020-06-12T02:37:31.623 INFO:tasks.rados.rados.0.smithi107.stdout:350: left oid 16 (ObjNum 349 snap 0 seq_num 349) 2020-06-12T02:37:31.624 INFO:tasks.rados.rados.0.smithi107.stdout:350: done (1 left) 2020-06-12T02:37:31.624 INFO:tasks.rados.rados.0.smithi107.stdout:353: done (0 left) 2020-06-12T02:37:31.625 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:37:31.625 INFO:tasks.rados.rados.0.smithi107.stdout:354: seq_num 353 ranges {0=720896,720896=401408} 2020-06-12T02:37:31.628 INFO:tasks.rados.rados.0.smithi107.stdout:354: writing smithi10729407-3 from 0 to 720896 tid 1 2020-06-12T02:37:31.631 INFO:tasks.rados.rados.0.smithi107.stdout:354: writing smithi10729407-3 from 720896 to 1122304 tid 2 2020-06-12T02:37:31.631 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:37:31.631 INFO:tasks.rados.rados.0.smithi107.stdout:355: seq_num 354 ranges {0=737280,737280=401408} 2020-06-12T02:37:31.636 INFO:tasks.rados.rados.0.smithi107.stdout:355: writing smithi10729407-19 from 0 to 737280 tid 1 2020-06-12T02:37:31.638 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:37:31.636+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:37:31.639 INFO:tasks.rados.rados.0.smithi107.stdout:355: writing smithi10729407-19 from 737280 to 1138688 tid 2 2020-06-12T02:37:31.639 INFO:tasks.rados.rados.0.smithi107.stdout:354: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:31.639 INFO:tasks.rados.rados.0.smithi107.stdout:354: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:31.639 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:37:31.640 INFO:tasks.rados.rados.0.smithi107.stdout:356: seq_num 355 ranges {0=753664,753664=401408} 2020-06-12T02:37:31.644 INFO:tasks.rados.rados.0.smithi107.stdout:356: writing smithi10729407-14 from 0 to 753664 tid 1 2020-06-12T02:37:31.647 INFO:tasks.rados.rados.0.smithi107.stdout:356: writing smithi10729407-14 from 753664 to 1155072 tid 2 2020-06-12T02:37:31.647 INFO:tasks.rados.rados.0.smithi107.stdout:357: delete oid 11 current snap is 0 2020-06-12T02:37:31.647 INFO:tasks.rados.rados.0.smithi107.stdout:354: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:31.647 INFO:tasks.rados.rados.0.smithi107.stdout:355: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:31.647 INFO:tasks.rados.rados.0.smithi107.stdout:354: finishing write tid 5 to smithi10729407-3 2020-06-12T02:37:31.648 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 52 (ObjNum 353 snap 0 seq_num 353) dirty exists 2020-06-12T02:37:31.648 INFO:tasks.rados.rados.0.smithi107.stdout:354: left oid 3 (ObjNum 353 snap 0 seq_num 353) 2020-06-12T02:37:31.648 INFO:tasks.rados.rados.0.smithi107.stdout:355: finishing write tid 2 to smithi10729407-19 2020-06-12T02:37:31.649 INFO:tasks.rados.rados.0.smithi107.stdout:355: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:31.649 INFO:tasks.rados.rados.0.smithi107.stdout:355: finishing write tid 5 to smithi10729407-19 2020-06-12T02:37:31.650 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 61 (ObjNum 354 snap 0 seq_num 354) dirty exists 2020-06-12T02:37:31.650 INFO:tasks.rados.rados.0.smithi107.stdout:355: left oid 19 (ObjNum 354 snap 0 seq_num 354) 2020-06-12T02:37:31.651 INFO:tasks.rados.rados.0.smithi107.stdout:356: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:31.653 INFO:tasks.rados.rados.0.smithi107.stdout:354: done (3 left) 2020-06-12T02:37:31.654 INFO:tasks.rados.rados.0.smithi107.stdout:355: done (2 left) 2020-06-12T02:37:31.654 INFO:tasks.rados.rados.0.smithi107.stdout:357: done (1 left) 2020-06-12T02:37:31.654 INFO:tasks.rados.rados.0.smithi107.stdout:358: delete oid 3 current snap is 0 2020-06-12T02:37:31.656 INFO:tasks.rados.rados.0.smithi107.stdout:356: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:31.657 INFO:tasks.rados.rados.0.smithi107.stdout:358: done (1 left) 2020-06-12T02:37:31.657 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:37:31.657 INFO:tasks.rados.rados.0.smithi107.stdout:359: seq_num 358 ranges {0=802816,802816=401408} 2020-06-12T02:37:31.662 INFO:tasks.rados.rados.0.smithi107.stdout:359: writing smithi10729407-17 from 0 to 802816 tid 1 2020-06-12T02:37:31.664 INFO:tasks.rados.rados.0.smithi107.stdout:359: writing smithi10729407-17 from 802816 to 1204224 tid 2 2020-06-12T02:37:31.665 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:37:31.665 INFO:tasks.rados.rados.0.smithi107.stdout:360: seq_num 359 ranges {581632=417792,999424=417792,1417216=385024} 2020-06-12T02:37:31.666 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:37:31.670 INFO:tasks.rados.rados.0.smithi107.stdout:360: writing smithi10729407-6 from 581632 to 999424 tid 1 2020-06-12T02:37:31.672 INFO:tasks.rados.rados.0.smithi107.stdout:360: writing smithi10729407-6 from 999424 to 1417216 tid 2 2020-06-12T02:37:31.680 INFO:tasks.rados.rados.0.smithi107.stdout:360: writing smithi10729407-6 from 1417216 to 1802240 tid 3 2020-06-12T02:37:31.681 INFO:tasks.rados.rados.0.smithi107.stdout:361: delete oid 18 current snap is 0 2020-06-12T02:37:31.681 INFO:tasks.rados.rados.0.smithi107.stdout:356: finishing write tid 4 to smithi10729407-14 2020-06-12T02:37:31.681 INFO:tasks.rados.rados.0.smithi107.stdout:356: finishing write tid 5 to smithi10729407-14 2020-06-12T02:37:31.682 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 42 (ObjNum 355 snap 0 seq_num 355) dirty exists 2020-06-12T02:37:31.682 INFO:tasks.rados.rados.0.smithi107.stdout:356: left oid 14 (ObjNum 355 snap 0 seq_num 355) 2020-06-12T02:37:31.682 INFO:tasks.rados.rados.0.smithi107.stdout:359: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:31.682 INFO:tasks.rados.rados.0.smithi107.stdout:359: finishing write tid 2 to smithi10729407-17 2020-06-12T02:37:31.682 INFO:tasks.rados.rados.0.smithi107.stdout:359: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:31.683 INFO:tasks.rados.rados.0.smithi107.stdout:359: finishing write tid 5 to smithi10729407-17 2020-06-12T02:37:31.683 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 73 (ObjNum 358 snap 0 seq_num 358) dirty exists 2020-06-12T02:37:31.683 INFO:tasks.rados.rados.0.smithi107.stdout:359: left oid 17 (ObjNum 358 snap 0 seq_num 358) 2020-06-12T02:37:31.683 INFO:tasks.rados.rados.0.smithi107.stdout:360: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:31.683 INFO:tasks.rados.rados.0.smithi107.stdout:356: done (3 left) 2020-06-12T02:37:31.684 INFO:tasks.rados.rados.0.smithi107.stdout:359: done (2 left) 2020-06-12T02:37:31.684 INFO:tasks.rados.rados.0.smithi107.stdout:361: done (1 left) 2020-06-12T02:37:31.684 INFO:tasks.rados.rados.0.smithi107.stdout:362: delete oid 14 current snap is 0 2020-06-12T02:37:31.685 INFO:tasks.rados.rados.0.smithi107.stdout:360: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:31.685 INFO:tasks.rados.rados.0.smithi107.stdout:360: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:31.685 INFO:tasks.rados.rados.0.smithi107.stdout:360: finishing write tid 5 to smithi10729407-6 2020-06-12T02:37:31.686 INFO:tasks.rados.rados.0.smithi107.stdout:360: finishing write tid 6 to smithi10729407-6 2020-06-12T02:37:31.686 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 69 (ObjNum 359 snap 0 seq_num 359) dirty exists 2020-06-12T02:37:31.686 INFO:tasks.rados.rados.0.smithi107.stdout:360: left oid 6 (ObjNum 359 snap 0 seq_num 359) 2020-06-12T02:37:31.687 INFO:tasks.rados.rados.0.smithi107.stdout:360: done (1 left) 2020-06-12T02:37:31.687 INFO:tasks.rados.rados.0.smithi107.stdout:362: done (0 left) 2020-06-12T02:37:31.687 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:37:31.687 INFO:tasks.rados.rados.0.smithi107.stdout:363: seq_num 362 ranges {1056768=466944,1523712=466944,1990656=335872} 2020-06-12T02:37:31.693 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:37:31.695 INFO:tasks.rados.rados.0.smithi107.stdout:363: writing smithi10729407-16 from 1056768 to 1523712 tid 1 2020-06-12T02:37:31.697 INFO:tasks.rados.rados.0.smithi107.stdout:363: writing smithi10729407-16 from 1523712 to 1990656 tid 2 2020-06-12T02:37:31.700 INFO:tasks.rados.rados.0.smithi107.stdout:363: writing smithi10729407-16 from 1990656 to 2326528 tid 3 2020-06-12T02:37:31.700 INFO:tasks.rados.rados.0.smithi107.stdout:364: delete oid 14 current snap is 0 2020-06-12T02:37:31.704 INFO:tasks.rados.rados.0.smithi107.stdout:364: done (1 left) 2020-06-12T02:37:31.704 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:37:31.704 INFO:tasks.rados.rados.0.smithi107.stdout:365: seq_num 364 ranges {933888=499712,1433600=638976,2072576=163840} 2020-06-12T02:37:31.712 INFO:tasks.rados.rados.0.smithi107.stdout:365: writing smithi10729407-12 from 933888 to 1433600 tid 1 2020-06-12T02:37:31.721 INFO:tasks.rados.rados.0.smithi107.stdout:365: writing smithi10729407-12 from 1433600 to 2072576 tid 2 2020-06-12T02:37:31.721 INFO:tasks.rados.rados.0.smithi107.stdout:365: writing smithi10729407-12 from 2072576 to 2236416 tid 3 2020-06-12T02:37:31.721 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:37:31.726 INFO:tasks.rados.rados.0.smithi107.stdout:366: seq_num 365 ranges {1802240=516096,2318336=573440,2891776=229376} 2020-06-12T02:37:31.726 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:37:31.726 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:37:31.727 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2020-06-12T02:37:31.730 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2020-06-12T02:37:31.730 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:37:31.731 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:37:31.732 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-06-12T02:37:31.732 INFO:tasks.rados.rados.0.smithi107.stdout:366: writing smithi10729407-6 from 1802240 to 2318336 tid 1 2020-06-12T02:37:31.734 INFO:tasks.rados.rados.0.smithi107.stdout:366: writing smithi10729407-6 from 2318336 to 2891776 tid 2 2020-06-12T02:37:31.735 INFO:tasks.rados.rados.0.smithi107.stdout:366: writing smithi10729407-6 from 2891776 to 3121152 tid 3 2020-06-12T02:37:31.735 INFO:tasks.rados.rados.0.smithi107.stdout:367: delete oid 5 current snap is 0 2020-06-12T02:37:31.735 INFO:tasks.rados.rados.0.smithi107.stdout:363: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:31.736 INFO:tasks.rados.rados.0.smithi107.stdout:363: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:31.736 INFO:tasks.rados.rados.0.smithi107.stdout:363: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:31.736 INFO:tasks.rados.rados.0.smithi107.stdout:363: finishing write tid 5 to smithi10729407-16 2020-06-12T02:37:31.736 INFO:tasks.rados.rados.0.smithi107.stdout:363: finishing write tid 6 to smithi10729407-16 2020-06-12T02:37:31.736 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 57 (ObjNum 362 snap 0 seq_num 362) dirty exists 2020-06-12T02:37:31.736 INFO:tasks.rados.rados.0.smithi107.stdout:363: left oid 16 (ObjNum 362 snap 0 seq_num 362) 2020-06-12T02:37:31.737 INFO:tasks.rados.rados.0.smithi107.stdout:365: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:31.737 INFO:tasks.rados.rados.0.smithi107.stdout:365: finishing write tid 2 to smithi10729407-12 2020-06-12T02:37:31.737 INFO:tasks.rados.rados.0.smithi107.stdout:365: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:31.737 INFO:tasks.rados.rados.0.smithi107.stdout:365: finishing write tid 5 to smithi10729407-12 2020-06-12T02:37:31.737 INFO:tasks.rados.rados.0.smithi107.stdout:365: finishing write tid 6 to smithi10729407-12 2020-06-12T02:37:31.738 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 65 (ObjNum 364 snap 0 seq_num 364) dirty exists 2020-06-12T02:37:31.738 INFO:tasks.rados.rados.0.smithi107.stdout:365: left oid 12 (ObjNum 364 snap 0 seq_num 364) 2020-06-12T02:37:31.739 INFO:tasks.rados.rados.0.smithi107.stdout:363: done (3 left) 2020-06-12T02:37:31.739 INFO:tasks.rados.rados.0.smithi107.stdout:365: done (2 left) 2020-06-12T02:37:31.739 INFO:tasks.rados.rados.0.smithi107.stdout:367: done (1 left) 2020-06-12T02:37:31.740 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:37:31.740 INFO:tasks.rados.rados.0.smithi107.stdout:368: seq_num 367 ranges {630784=548864,1179648=737280,1916928=65536} 2020-06-12T02:37:31.743 INFO:tasks.rados.rados.0.smithi107.stdout:368: writing smithi10729407-9 from 630784 to 1179648 tid 1 2020-06-12T02:37:31.743 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:37:31.740+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:37:31.747 INFO:tasks.rados.rados.0.smithi107.stdout:368: writing smithi10729407-9 from 1179648 to 1916928 tid 2 2020-06-12T02:37:31.748 INFO:tasks.rados.rados.0.smithi107.stdout:368: writing smithi10729407-9 from 1916928 to 1982464 tid 3 2020-06-12T02:37:31.748 INFO:tasks.rados.rados.0.smithi107.stdout:369: delete oid 12 current snap is 0 2020-06-12T02:37:31.749 INFO:tasks.rados.rados.0.smithi107.stdout:366: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:31.749 INFO:tasks.rados.rados.0.smithi107.stdout:366: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:31.749 INFO:tasks.rados.rados.0.smithi107.stdout:366: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:31.749 INFO:tasks.rados.rados.0.smithi107.stdout:366: finishing write tid 5 to smithi10729407-6 2020-06-12T02:37:31.749 INFO:tasks.rados.rados.0.smithi107.stdout:366: finishing write tid 6 to smithi10729407-6 2020-06-12T02:37:31.749 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 73 (ObjNum 365 snap 0 seq_num 365) dirty exists 2020-06-12T02:37:31.750 INFO:tasks.rados.rados.0.smithi107.stdout:366: left oid 6 (ObjNum 365 snap 0 seq_num 365) 2020-06-12T02:37:31.750 INFO:tasks.rados.rados.0.smithi107.stdout:368: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:31.751 INFO:tasks.rados.rados.0.smithi107.stdout:366: done (2 left) 2020-06-12T02:37:31.751 INFO:tasks.rados.rados.0.smithi107.stdout:369: done (1 left) 2020-06-12T02:37:31.751 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:37:31.751 INFO:tasks.rados.rados.0.smithi107.stdout:370: seq_num 369 ranges {0=589824,589824=507904,1097728=294912} 2020-06-12T02:37:31.752 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:37:31.755 INFO:tasks.rados.rados.0.smithi107.stdout:370: writing smithi10729407-8 from 0 to 589824 tid 1 2020-06-12T02:37:31.757 INFO:tasks.rados.rados.0.smithi107.stdout:370: writing smithi10729407-8 from 589824 to 1097728 tid 2 2020-06-12T02:37:31.759 INFO:tasks.rados.rados.0.smithi107.stdout:370: writing smithi10729407-8 from 1097728 to 1392640 tid 3 2020-06-12T02:37:31.760 INFO:tasks.rados.rados.0.smithi107.stdout:371: delete oid 4 current snap is 0 2020-06-12T02:37:31.760 INFO:tasks.rados.rados.0.smithi107.stdout:368: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:31.760 INFO:tasks.rados.rados.0.smithi107.stdout:368: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:31.761 INFO:tasks.rados.rados.0.smithi107.stdout:368: finishing write tid 5 to smithi10729407-9 2020-06-12T02:37:31.761 INFO:tasks.rados.rados.0.smithi107.stdout:370: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:31.762 INFO:tasks.rados.rados.0.smithi107.stdout:371: done (2 left) 2020-06-12T02:37:31.762 INFO:tasks.rados.rados.0.smithi107.stdout:372: delete oid 6 current snap is 0 2020-06-12T02:37:31.763 INFO:tasks.rados.rados.0.smithi107.stdout:368: finishing write tid 6 to smithi10729407-9 2020-06-12T02:37:31.763 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 71 (ObjNum 367 snap 0 seq_num 367) dirty exists 2020-06-12T02:37:31.763 INFO:tasks.rados.rados.0.smithi107.stdout:368: left oid 9 (ObjNum 367 snap 0 seq_num 367) 2020-06-12T02:37:31.764 INFO:tasks.rados.rados.0.smithi107.stdout:368: done (2 left) 2020-06-12T02:37:31.764 INFO:tasks.rados.rados.0.smithi107.stdout:372: done (1 left) 2020-06-12T02:37:31.764 INFO:tasks.rados.rados.0.smithi107.stdout:373: delete oid 14 current snap is 0 2020-06-12T02:37:31.765 INFO:tasks.rados.rados.0.smithi107.stdout:373: done (1 left) 2020-06-12T02:37:31.765 INFO:tasks.rados.rados.0.smithi107.stdout:374: delete oid 14 current snap is 0 2020-06-12T02:37:31.767 INFO:tasks.rados.rados.0.smithi107.stdout:370: finishing write tid 2 to smithi10729407-8 2020-06-12T02:37:31.768 INFO:tasks.rados.rados.0.smithi107.stdout:374: done (1 left) 2020-06-12T02:37:31.768 INFO:tasks.rados.rados.0.smithi107.stdout:375: delete oid 11 current snap is 0 2020-06-12T02:37:31.769 INFO:tasks.rados.rados.0.smithi107.stdout:370: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:31.769 INFO:tasks.rados.rados.0.smithi107.stdout:370: finishing write tid 5 to smithi10729407-8 2020-06-12T02:37:31.770 INFO:tasks.rados.rados.0.smithi107.stdout:375: done (1 left) 2020-06-12T02:37:31.770 INFO:tasks.rados.rados.0.smithi107.stdout:376: delete oid 6 current snap is 0 2020-06-12T02:37:31.770 INFO:tasks.rados.rados.0.smithi107.stdout:370: finishing write tid 6 to smithi10729407-8 2020-06-12T02:37:31.771 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 77 (ObjNum 369 snap 0 seq_num 369) dirty exists 2020-06-12T02:37:31.771 INFO:tasks.rados.rados.0.smithi107.stdout:370: left oid 8 (ObjNum 369 snap 0 seq_num 369) 2020-06-12T02:37:31.771 INFO:tasks.rados.rados.0.smithi107.stdout:370: done (1 left) 2020-06-12T02:37:31.772 INFO:tasks.rados.rados.0.smithi107.stdout:376: done (0 left) 2020-06-12T02:37:31.772 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:37:31.772 INFO:tasks.rados.rados.0.smithi107.stdout:377: seq_num 376 ranges {278528=704512,983040=548864,1531904=253952} 2020-06-12T02:37:31.777 INFO:tasks.rados.rados.0.smithi107.stdout:377: writing smithi10729407-15 from 278528 to 983040 tid 1 2020-06-12T02:37:31.786 INFO:tasks.rados.rados.0.smithi107.stdout:377: writing smithi10729407-15 from 983040 to 1531904 tid 2 2020-06-12T02:37:31.787 INFO:tasks.rados.rados.0.smithi107.stdout:377: writing smithi10729407-15 from 1531904 to 1785856 tid 3 2020-06-12T02:37:31.787 INFO:tasks.rados.rados.0.smithi107.stdout:378: delete oid 11 current snap is 0 2020-06-12T02:37:31.787 INFO:tasks.rados.rados.0.smithi107.stdout:377: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:31.787 INFO:tasks.rados.rados.0.smithi107.stdout:378: done (1 left) 2020-06-12T02:37:31.788 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:37:31.788 INFO:tasks.rados.rados.0.smithi107.stdout:379: seq_num 378 ranges {0=737280,737280=720896,1458176=81920} 2020-06-12T02:37:31.788 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:37:31.789 INFO:tasks.rados.rados.0.smithi107.stdout:379: writing smithi10729407-20 from 0 to 737280 tid 1 2020-06-12T02:37:31.794 INFO:tasks.rados.rados.0.smithi107.stdout:379: writing smithi10729407-20 from 737280 to 1458176 tid 2 2020-06-12T02:37:31.795 INFO:tasks.rados.rados.0.smithi107.stdout:379: writing smithi10729407-20 from 1458176 to 1540096 tid 3 2020-06-12T02:37:31.795 INFO:tasks.rados.rados.0.smithi107.stdout:380: delete oid 16 current snap is 0 2020-06-12T02:37:31.795 INFO:tasks.rados.rados.0.smithi107.stdout:377: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:31.796 INFO:tasks.rados.rados.0.smithi107.stdout:377: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:31.796 INFO:tasks.rados.rados.0.smithi107.stdout:377: finishing write tid 5 to smithi10729407-15 2020-06-12T02:37:31.796 INFO:tasks.rados.rados.0.smithi107.stdout:377: finishing write tid 6 to smithi10729407-15 2020-06-12T02:37:31.797 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 81 (ObjNum 376 snap 0 seq_num 376) dirty exists 2020-06-12T02:37:31.797 INFO:tasks.rados.rados.0.smithi107.stdout:377: left oid 15 (ObjNum 376 snap 0 seq_num 376) 2020-06-12T02:37:31.798 INFO:tasks.rados.rados.0.smithi107.stdout:377: done (2 left) 2020-06-12T02:37:31.799 INFO:tasks.rados.rados.0.smithi107.stdout:380: done (1 left) 2020-06-12T02:37:31.799 INFO:tasks.rados.rados.0.smithi107.stdout:381: delete oid 1 current snap is 0 2020-06-12T02:37:31.800 INFO:tasks.rados.rados.0.smithi107.stdout:379: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:31.801 INFO:tasks.rados.rados.0.smithi107.stdout:381: done (1 left) 2020-06-12T02:37:31.801 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:37:31.801 INFO:tasks.rados.rados.0.smithi107.stdout:382: seq_num 381 ranges {0=786432,786432=417792,1204224=385024} 2020-06-12T02:37:31.805 INFO:tasks.rados.rados.0.smithi107.stdout:382: writing smithi10729407-3 from 0 to 786432 tid 1 2020-06-12T02:37:31.808 INFO:tasks.rados.rados.0.smithi107.stdout:382: writing smithi10729407-3 from 786432 to 1204224 tid 2 2020-06-12T02:37:31.811 INFO:tasks.rados.rados.0.smithi107.stdout:382: writing smithi10729407-3 from 1204224 to 1589248 tid 3 2020-06-12T02:37:31.811 INFO:tasks.rados.rados.0.smithi107.stdout:383: delete oid 19 current snap is 0 2020-06-12T02:37:31.811 INFO:tasks.rados.rados.0.smithi107.stdout:379: finishing write tid 2 to smithi10729407-20 2020-06-12T02:37:31.811 INFO:tasks.rados.rados.0.smithi107.stdout:379: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:31.812 INFO:tasks.rados.rados.0.smithi107.stdout:379: finishing write tid 5 to smithi10729407-20 2020-06-12T02:37:31.812 INFO:tasks.rados.rados.0.smithi107.stdout:379: finishing write tid 6 to smithi10729407-20 2020-06-12T02:37:31.812 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 23 (ObjNum 378 snap 0 seq_num 378) dirty exists 2020-06-12T02:37:31.812 INFO:tasks.rados.rados.0.smithi107.stdout:379: left oid 20 (ObjNum 378 snap 0 seq_num 378) 2020-06-12T02:37:31.813 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:37:31.816 INFO:tasks.rados.rados.0.smithi107.stdout:379: done (2 left) 2020-06-12T02:37:31.816 INFO:tasks.rados.rados.0.smithi107.stdout:383: done (1 left) 2020-06-12T02:37:31.816 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:37:31.816 INFO:tasks.rados.rados.0.smithi107.stdout:384: seq_num 383 ranges {0=417792,417792=589824,1007616=417792,1425408=196608} 2020-06-12T02:37:31.817 INFO:tasks.rados.rados.0.smithi107.stdout:384: writing smithi10729407-6 from 0 to 417792 tid 1 2020-06-12T02:37:31.821 INFO:tasks.rados.rados.0.smithi107.stdout:384: writing smithi10729407-6 from 417792 to 1007616 tid 2 2020-06-12T02:37:31.824 INFO:tasks.rados.rados.0.smithi107.stdout:384: writing smithi10729407-6 from 1007616 to 1425408 tid 3 2020-06-12T02:37:31.825 INFO:tasks.rados.rados.0.smithi107.stdout:384: writing smithi10729407-6 from 1425408 to 1622016 tid 4 2020-06-12T02:37:31.825 INFO:tasks.rados.rados.0.smithi107.stdout:385: delete oid 10 current snap is 0 2020-06-12T02:37:31.825 INFO:tasks.rados.rados.0.smithi107.stdout:382: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:31.826 INFO:tasks.rados.rados.0.smithi107.stdout:382: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:31.826 INFO:tasks.rados.rados.0.smithi107.stdout:382: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:31.826 INFO:tasks.rados.rados.0.smithi107.stdout:382: finishing write tid 5 to smithi10729407-3 2020-06-12T02:37:31.826 INFO:tasks.rados.rados.0.smithi107.stdout:384: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:31.826 INFO:tasks.rados.rados.0.smithi107.stdout:382: finishing write tid 6 to smithi10729407-3 2020-06-12T02:37:31.827 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 62 (ObjNum 381 snap 0 seq_num 381) dirty exists 2020-06-12T02:37:31.827 INFO:tasks.rados.rados.0.smithi107.stdout:382: left oid 3 (ObjNum 381 snap 0 seq_num 381) 2020-06-12T02:37:31.829 INFO:tasks.rados.rados.0.smithi107.stdout:382: done (2 left) 2020-06-12T02:37:31.829 INFO:tasks.rados.rados.0.smithi107.stdout:385: done (1 left) 2020-06-12T02:37:31.829 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:37:31.829 INFO:tasks.rados.rados.0.smithi107.stdout:386: seq_num 385 ranges {737280=450560,1187840=761856,1949696=442368} 2020-06-12T02:37:31.836 INFO:tasks.rados.rados.0.smithi107.stdout:386: writing smithi10729407-7 from 737280 to 1187840 tid 1 2020-06-12T02:37:31.840 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:37:31.837+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:37:31.841 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:37:31.842 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-06-12T02:37:31.842 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-06-12T02:37:31.842 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-06-12T02:37:31.843 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-06-12T02:37:31.843 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-06-12T02:37:31.843 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-06-12T02:37:31.843 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-06-12T02:37:31.843 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.7 2020-06-12T02:37:31.843 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: problem getting command descriptions from osd.7 2020-06-12T02:37:31.844 INFO:tasks.rados.rados.0.smithi107.stdout:386: writing smithi10729407-7 from 1187840 to 1949696 tid 2 2020-06-12T02:37:31.845 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-06-12T02:37:31.845 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-06-12T02:37:31.846 INFO:tasks.rados.rados.0.smithi107.stdout:386: writing smithi10729407-7 from 1949696 to 2392064 tid 3 2020-06-12T02:37:31.846 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:37:31.846 INFO:tasks.rados.rados.0.smithi107.stdout:387: seq_num 386 ranges {0=466944,466944=638976,1105920=565248} 2020-06-12T02:37:31.848 INFO:tasks.rados.rados.0.smithi107.stdout:387: writing smithi10729407-1 from 0 to 466944 tid 1 2020-06-12T02:37:31.853 INFO:tasks.rados.rados.0.smithi107.stdout:387: writing smithi10729407-1 from 466944 to 1105920 tid 2 2020-06-12T02:37:31.857 INFO:tasks.rados.rados.0.smithi107.stdout:387: writing smithi10729407-1 from 1105920 to 1671168 tid 3 2020-06-12T02:37:31.858 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:37:31.858 INFO:tasks.rados.rados.0.smithi107.stdout:388: seq_num 387 ranges {0=483328,483328=524288,1007616=679936} 2020-06-12T02:37:31.860 INFO:tasks.rados.rados.0.smithi107.stdout:388: writing smithi10729407-10 from 0 to 483328 tid 1 2020-06-12T02:37:31.864 INFO:tasks.rados.rados.0.smithi107.stdout:388: writing smithi10729407-10 from 483328 to 1007616 tid 2 2020-06-12T02:37:31.869 INFO:tasks.rados.rados.0.smithi107.stdout:388: writing smithi10729407-10 from 1007616 to 1687552 tid 3 2020-06-12T02:37:31.869 INFO:tasks.rados.rados.0.smithi107.stdout:389: delete oid 4 current snap is 0 2020-06-12T02:37:31.870 INFO:tasks.rados.rados.0.smithi107.stdout:384: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:31.870 INFO:tasks.rados.rados.0.smithi107.stdout:384: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:31.870 INFO:tasks.rados.rados.0.smithi107.stdout:384: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:31.870 INFO:tasks.rados.rados.0.smithi107.stdout:384: finishing write tid 6 to smithi10729407-6 2020-06-12T02:37:31.870 INFO:tasks.rados.rados.0.smithi107.stdout:384: finishing write tid 7 to smithi10729407-6 2020-06-12T02:37:31.871 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 81 (ObjNum 383 snap 0 seq_num 383) dirty exists 2020-06-12T02:37:31.871 INFO:tasks.rados.rados.0.smithi107.stdout:384: left oid 6 (ObjNum 383 snap 0 seq_num 383) 2020-06-12T02:37:31.871 INFO:tasks.rados.rados.0.smithi107.stdout:386: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:31.871 INFO:tasks.rados.rados.0.smithi107.stdout:386: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:31.871 INFO:tasks.rados.rados.0.smithi107.stdout:386: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:31.871 INFO:tasks.rados.rados.0.smithi107.stdout:386: finishing write tid 5 to smithi10729407-7 2020-06-12T02:37:31.872 INFO:tasks.rados.rados.0.smithi107.stdout:387: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:31.872 INFO:tasks.rados.rados.0.smithi107.stdout:386: finishing write tid 6 to smithi10729407-7 2020-06-12T02:37:31.872 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 64 (ObjNum 385 snap 0 seq_num 385) dirty exists 2020-06-12T02:37:31.872 INFO:tasks.rados.rados.0.smithi107.stdout:386: left oid 7 (ObjNum 385 snap 0 seq_num 385) 2020-06-12T02:37:31.872 INFO:tasks.rados.rados.0.smithi107.stdout:387: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:31.873 INFO:tasks.rados.rados.0.smithi107.stdout:387: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:31.873 INFO:tasks.rados.rados.0.smithi107.stdout:387: finishing write tid 5 to smithi10729407-1 2020-06-12T02:37:31.873 INFO:tasks.rados.rados.0.smithi107.stdout:387: finishing write tid 6 to smithi10729407-1 2020-06-12T02:37:31.873 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 85 (ObjNum 386 snap 0 seq_num 386) dirty exists 2020-06-12T02:37:31.873 INFO:tasks.rados.rados.0.smithi107.stdout:387: left oid 1 (ObjNum 386 snap 0 seq_num 386) 2020-06-12T02:37:31.873 INFO:tasks.rados.rados.0.smithi107.stdout:388: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:31.874 INFO:tasks.rados.rados.0.smithi107.stdout:388: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:31.874 INFO:tasks.rados.rados.0.smithi107.stdout:384: done (4 left) 2020-06-12T02:37:31.874 INFO:tasks.rados.rados.0.smithi107.stdout:386: done (3 left) 2020-06-12T02:37:31.875 INFO:tasks.rados.rados.0.smithi107.stdout:387: done (2 left) 2020-06-12T02:37:31.875 INFO:tasks.rados.rados.0.smithi107.stdout:389: done (1 left) 2020-06-12T02:37:31.875 INFO:tasks.rados.rados.0.smithi107.stdout:390: delete oid 9 current snap is 0 2020-06-12T02:37:31.875 INFO:tasks.rados.rados.0.smithi107.stdout:390: done (1 left) 2020-06-12T02:37:31.875 INFO:tasks.rados.rados.0.smithi107.stdout:391: delete oid 19 current snap is 0 2020-06-12T02:37:31.876 INFO:tasks.rados.rados.0.smithi107.stdout:391: done (1 left) 2020-06-12T02:37:31.876 INFO:tasks.rados.rados.0.smithi107.stdout:392: delete oid 11 current snap is 0 2020-06-12T02:37:31.877 INFO:tasks.rados.rados.0.smithi107.stdout:388: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:31.877 INFO:tasks.rados.rados.0.smithi107.stdout:388: finishing write tid 5 to smithi10729407-10 2020-06-12T02:37:31.877 INFO:tasks.rados.rados.0.smithi107.stdout:392: done (1 left) 2020-06-12T02:37:31.878 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:37:31.878 INFO:tasks.rados.rados.0.smithi107.stdout:393: seq_num 392 ranges {0=573440,573440=802816,1376256=401408} 2020-06-12T02:37:31.880 INFO:teuthology.orchestra.run.smithi188:> CEPH_ARGS='--filestore-merge-threshold 1 --filestore-split-multiple 1' sudo -E ceph-objectstore-tool --no-mon-config --log-file=/var/log/ceph/objectstore_tool.$pid.log --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op apply-layout-settings --pool rbd 2020-06-12T02:37:31.881 INFO:tasks.rados.rados.0.smithi107.stdout:393: writing smithi10729407-19 from 0 to 573440 tid 1 2020-06-12T02:37:31.888 INFO:tasks.rados.rados.0.smithi107.stdout:393: writing smithi10729407-19 from 573440 to 1376256 tid 2 2020-06-12T02:37:31.889 INFO:tasks.rados.rados.0.smithi107.stdout:393: writing smithi10729407-19 from 1376256 to 1777664 tid 3 2020-06-12T02:37:31.889 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:37:31.889 INFO:tasks.rados.rados.0.smithi107.stdout:394: seq_num 393 ranges {1392640=589824,1982464=679936,2662400=524288} 2020-06-12T02:37:31.898 INFO:tasks.rados.rados.0.smithi107.stdout:394: writing smithi10729407-8 from 1392640 to 1982464 tid 1 2020-06-12T02:37:31.899 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:37:31.899 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2020-06-12T02:37:31.899 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2020-06-12T02:37:31.899 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [] 2020-06-12T02:37:31.900 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:37:31.902 INFO:tasks.rados.rados.0.smithi107.stdout:394: writing smithi10729407-8 from 1982464 to 2662400 tid 2 2020-06-12T02:37:31.905 INFO:tasks.rados.rados.0.smithi107.stdout:394: writing smithi10729407-8 from 2662400 to 3186688 tid 3 2020-06-12T02:37:31.906 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:37:31.906 INFO:tasks.rados.rados.0.smithi107.stdout:395: seq_num 394 ranges {2392064=606208,2998272=565248,3563520=557056,4120576=81920} 2020-06-12T02:37:31.909 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-06-12T02:37:31.922 INFO:tasks.rados.rados.0.smithi107.stdout:395: writing smithi10729407-7 from 2392064 to 2998272 tid 1 2020-06-12T02:37:31.925 INFO:tasks.rados.rados.0.smithi107.stdout:395: writing smithi10729407-7 from 2998272 to 3563520 tid 2 2020-06-12T02:37:31.929 INFO:tasks.rados.rados.0.smithi107.stdout:395: writing smithi10729407-7 from 3563520 to 4120576 tid 3 2020-06-12T02:37:31.929 INFO:tasks.rados.rados.0.smithi107.stdout:395: writing smithi10729407-7 from 4120576 to 4202496 tid 4 2020-06-12T02:37:31.930 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:37:31.930 INFO:tasks.rados.rados.0.smithi107.stdout:396: seq_num 395 ranges {1671168=622592,2293760=450560,2744320=753664} 2020-06-12T02:37:31.941 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:31.938+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:31.942 INFO:tasks.rados.rados.0.smithi107.stdout:396: writing smithi10729407-1 from 1671168 to 2293760 tid 1 2020-06-12T02:37:31.945 INFO:tasks.rados.rados.0.smithi107.stdout:396: writing smithi10729407-1 from 2293760 to 2744320 tid 2 2020-06-12T02:37:31.951 INFO:tasks.rados.rados.0.smithi107.stdout:396: writing smithi10729407-1 from 2744320 to 3497984 tid 3 2020-06-12T02:37:31.951 INFO:tasks.rados.rados.0.smithi107.stdout:397: delete oid 2 current snap is 0 2020-06-12T02:37:31.951 INFO:tasks.rados.rados.0.smithi107.stdout:388: finishing write tid 6 to smithi10729407-10 2020-06-12T02:37:31.951 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 38 (ObjNum 387 snap 0 seq_num 387) dirty exists 2020-06-12T02:37:31.952 INFO:tasks.rados.rados.0.smithi107.stdout:388: left oid 10 (ObjNum 387 snap 0 seq_num 387) 2020-06-12T02:37:31.952 INFO:tasks.rados.rados.0.smithi107.stdout:393: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:31.952 INFO:tasks.rados.rados.0.smithi107.stdout:393: finishing write tid 2 to smithi10729407-19 2020-06-12T02:37:31.952 INFO:tasks.rados.rados.0.smithi107.stdout:393: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:31.952 INFO:tasks.rados.rados.0.smithi107.stdout:393: finishing write tid 5 to smithi10729407-19 2020-06-12T02:37:31.953 INFO:tasks.rados.rados.0.smithi107.stdout:393: finishing write tid 6 to smithi10729407-19 2020-06-12T02:37:31.953 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 72 (ObjNum 392 snap 0 seq_num 392) dirty exists 2020-06-12T02:37:31.953 INFO:tasks.rados.rados.0.smithi107.stdout:393: left oid 19 (ObjNum 392 snap 0 seq_num 392) 2020-06-12T02:37:31.953 INFO:tasks.rados.rados.0.smithi107.stdout:394: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:31.953 INFO:tasks.rados.rados.0.smithi107.stdout:394: finishing write tid 2 to smithi10729407-8 2020-06-12T02:37:31.954 INFO:tasks.rados.rados.0.smithi107.stdout:394: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:31.954 INFO:tasks.rados.rados.0.smithi107.stdout:394: finishing write tid 5 to smithi10729407-8 2020-06-12T02:37:31.954 INFO:tasks.rados.rados.0.smithi107.stdout:394: finishing write tid 6 to smithi10729407-8 2020-06-12T02:37:31.954 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 85 (ObjNum 393 snap 0 seq_num 393) dirty exists 2020-06-12T02:37:31.954 INFO:tasks.rados.rados.0.smithi107.stdout:394: left oid 8 (ObjNum 393 snap 0 seq_num 393) 2020-06-12T02:37:31.954 INFO:tasks.rados.rados.0.smithi107.stdout:395: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:31.955 INFO:tasks.rados.rados.0.smithi107.stdout:395: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:31.955 INFO:tasks.rados.rados.0.smithi107.stdout:395: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:31.955 INFO:tasks.rados.rados.0.smithi107.stdout:395: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:31.955 INFO:tasks.rados.rados.0.smithi107.stdout:395: finishing write tid 6 to smithi10729407-7 2020-06-12T02:37:31.955 INFO:tasks.rados.rados.0.smithi107.stdout:396: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:31.956 INFO:tasks.rados.rados.0.smithi107.stdout:395: finishing write tid 7 to smithi10729407-7 2020-06-12T02:37:31.956 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 69 (ObjNum 394 snap 0 seq_num 394) dirty exists 2020-06-12T02:37:31.956 INFO:tasks.rados.rados.0.smithi107.stdout:395: left oid 7 (ObjNum 394 snap 0 seq_num 394) 2020-06-12T02:37:31.956 INFO:tasks.rados.rados.0.smithi107.stdout:388: done (5 left) 2020-06-12T02:37:31.957 INFO:tasks.rados.rados.0.smithi107.stdout:393: done (4 left) 2020-06-12T02:37:31.959 INFO:tasks.rados.rados.0.smithi107.stdout:394: done (3 left) 2020-06-12T02:37:31.959 INFO:tasks.rados.rados.0.smithi107.stdout:395: done (2 left) 2020-06-12T02:37:31.959 INFO:tasks.rados.rados.0.smithi107.stdout:397: done (1 left) 2020-06-12T02:37:31.959 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:37:31.962 INFO:tasks.rados.rados.0.smithi107.stdout:398: seq_num 397 ranges {1589248=655360,2244608=671744,2916352=532480} 2020-06-12T02:37:31.966 INFO:tasks.rados.rados.0.smithi107.stdout:398: writing smithi10729407-3 from 1589248 to 2244608 tid 1 2020-06-12T02:37:31.971 INFO:tasks.rados.rados.0.smithi107.stdout:398: writing smithi10729407-3 from 2244608 to 2916352 tid 2 2020-06-12T02:37:31.975 INFO:tasks.rados.rados.0.smithi107.stdout:398: writing smithi10729407-3 from 2916352 to 3448832 tid 3 2020-06-12T02:37:31.975 INFO:tasks.rados.rados.0.smithi107.stdout:399: delete oid 7 current snap is 0 2020-06-12T02:37:31.975 INFO:tasks.rados.rados.0.smithi107.stdout:396: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:31.976 INFO:tasks.rados.rados.0.smithi107.stdout:396: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:31.976 INFO:tasks.rados.rados.0.smithi107.stdout:396: finishing write tid 5 to smithi10729407-1 2020-06-12T02:37:31.976 INFO:tasks.rados.rados.0.smithi107.stdout:396: finishing write tid 6 to smithi10729407-1 2020-06-12T02:37:31.976 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 89 (ObjNum 395 snap 0 seq_num 395) dirty exists 2020-06-12T02:37:31.976 INFO:tasks.rados.rados.0.smithi107.stdout:396: left oid 1 (ObjNum 395 snap 0 seq_num 395) 2020-06-12T02:37:31.977 INFO:tasks.rados.rados.0.smithi107.stdout:398: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:31.979 INFO:tasks.rados.rados.0.smithi107.stdout:396: done (2 left) 2020-06-12T02:37:31.979 INFO:tasks.rados.rados.0.smithi107.stdout:399: done (1 left) 2020-06-12T02:37:31.979 INFO:tasks.rados.rados.0.smithi107.stdout:400: delete oid 20 current snap is 0 2020-06-12T02:37:31.980 INFO:tasks.rados.rados.0.smithi107.stdout:398: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:31.982 INFO:tasks.rados.rados.0.smithi107.stdout:398: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:31.983 INFO:tasks.rados.rados.0.smithi107.stdout:398: finishing write tid 5 to smithi10729407-3 2020-06-12T02:37:31.984 INFO:tasks.rados.rados.0.smithi107.stdout:400: done (1 left) 2020-06-12T02:37:31.984 INFO:tasks.rados.rados.0.smithi107.stdout:401: delete oid 10 current snap is 0 2020-06-12T02:37:31.985 INFO:tasks.rados.rados.0.smithi107.stdout:398: finishing write tid 6 to smithi10729407-3 2020-06-12T02:37:31.986 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 66 (ObjNum 397 snap 0 seq_num 397) dirty exists 2020-06-12T02:37:31.986 INFO:tasks.rados.rados.0.smithi107.stdout:398: left oid 3 (ObjNum 397 snap 0 seq_num 397) 2020-06-12T02:37:31.987 INFO:tasks.rados.rados.0.smithi107.stdout:398: done (1 left) 2020-06-12T02:37:31.987 INFO:tasks.rados.rados.0.smithi107.stdout:401: done (0 left) 2020-06-12T02:37:31.987 INFO:tasks.rados.rados.0.smithi107.stdout:402: delete oid 6 current snap is 0 2020-06-12T02:37:31.988 INFO:tasks.rados.rados.0.smithi107.stdout:402: done (0 left) 2020-06-12T02:37:31.988 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:37:31.989 INFO:tasks.rados.rados.0.smithi107.stdout:403: seq_num 402 ranges {1204224=737280,1941504=491520,2433024=516096,2949120=196608} 2020-06-12T02:37:31.999 INFO:tasks.rados.rados.0.smithi107.stdout:403: writing smithi10729407-17 from 1204224 to 1941504 tid 1 2020-06-12T02:37:32.002 INFO:tasks.rados.rados.0.smithi107.stdout:403: writing smithi10729407-17 from 1941504 to 2433024 tid 2 2020-06-12T02:37:32.006 INFO:tasks.rados.rados.0.smithi107.stdout:403: writing smithi10729407-17 from 2433024 to 2949120 tid 3 2020-06-12T02:37:32.007 INFO:tasks.rados.rados.0.smithi107.stdout:403: writing smithi10729407-17 from 2949120 to 3145728 tid 4 2020-06-12T02:37:32.007 INFO:tasks.rados.rados.0.smithi107.stdout:404: delete oid 3 current snap is 0 2020-06-12T02:37:32.007 INFO:tasks.rados.rados.0.smithi107.stdout:403: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:32.009 INFO:tasks.rados.rados.0.smithi107.stdout:404: done (1 left) 2020-06-12T02:37:32.009 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:37:32.009 INFO:tasks.rados.rados.0.smithi107.stdout:405: seq_num 404 ranges {0=770048,770048=712704,1482752=491520} 2020-06-12T02:37:32.014 INFO:tasks.rados.rados.0.smithi107.stdout:405: writing smithi10729407-13 from 0 to 770048 tid 1 2020-06-12T02:37:32.019 INFO:tasks.rados.rados.0.smithi107.stdout:405: writing smithi10729407-13 from 770048 to 1482752 tid 2 2020-06-12T02:37:32.022 INFO:tasks.rados.rados.0.smithi107.stdout:405: writing smithi10729407-13 from 1482752 to 1974272 tid 3 2020-06-12T02:37:32.022 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:37:32.023 INFO:tasks.rados.rados.0.smithi107.stdout:406: seq_num 405 ranges {1785856=786432,2572288=589824,3162112=614400} 2020-06-12T02:37:32.037 INFO:tasks.rados.rados.0.smithi107.stdout:406: writing smithi10729407-15 from 1785856 to 2572288 tid 1 2020-06-12T02:37:32.040 INFO:tasks.rados.rados.0.smithi107.stdout:406: writing smithi10729407-15 from 2572288 to 3162112 tid 2 2020-06-12T02:37:32.042 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:32.039+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:32.044 INFO:tasks.rados.rados.0.smithi107.stdout:406: writing smithi10729407-15 from 3162112 to 3776512 tid 3 2020-06-12T02:37:32.044 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:37:32.045 INFO:tasks.rados.rados.0.smithi107.stdout:407: seq_num 406 ranges {0=802816,802816=475136,1277952=499712,1777664=229376} 2020-06-12T02:37:32.049 INFO:tasks.rados.rados.0.smithi107.stdout:407: writing smithi10729407-6 from 0 to 802816 tid 1 2020-06-12T02:37:32.052 INFO:tasks.rados.rados.0.smithi107.stdout:407: writing smithi10729407-6 from 802816 to 1277952 tid 2 2020-06-12T02:37:32.055 INFO:tasks.rados.rados.0.smithi107.stdout:407: writing smithi10729407-6 from 1277952 to 1777664 tid 3 2020-06-12T02:37:32.057 INFO:tasks.rados.rados.0.smithi107.stdout:407: writing smithi10729407-6 from 1777664 to 2007040 tid 4 2020-06-12T02:37:32.058 INFO:tasks.rados.rados.0.smithi107.stdout:408: delete oid 10 current snap is 0 2020-06-12T02:37:32.058 INFO:tasks.rados.rados.0.smithi107.stdout:403: finishing write tid 2 to smithi10729407-17 2020-06-12T02:37:32.058 INFO:tasks.rados.rados.0.smithi107.stdout:403: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:32.058 INFO:tasks.rados.rados.0.smithi107.stdout:403: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:32.059 INFO:tasks.rados.rados.0.smithi107.stdout:403: finishing write tid 6 to smithi10729407-17 2020-06-12T02:37:32.059 INFO:tasks.rados.rados.0.smithi107.stdout:403: finishing write tid 7 to smithi10729407-17 2020-06-12T02:37:32.059 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 80 (ObjNum 402 snap 0 seq_num 402) dirty exists 2020-06-12T02:37:32.059 INFO:tasks.rados.rados.0.smithi107.stdout:403: left oid 17 (ObjNum 402 snap 0 seq_num 402) 2020-06-12T02:37:32.059 INFO:tasks.rados.rados.0.smithi107.stdout:405: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:32.059 INFO:tasks.rados.rados.0.smithi107.stdout:405: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:32.060 INFO:tasks.rados.rados.0.smithi107.stdout:405: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:32.060 INFO:tasks.rados.rados.0.smithi107.stdout:405: finishing write tid 5 to smithi10729407-13 2020-06-12T02:37:32.060 INFO:tasks.rados.rados.0.smithi107.stdout:405: finishing write tid 6 to smithi10729407-13 2020-06-12T02:37:32.060 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 74 (ObjNum 404 snap 0 seq_num 404) dirty exists 2020-06-12T02:37:32.060 INFO:tasks.rados.rados.0.smithi107.stdout:405: left oid 13 (ObjNum 404 snap 0 seq_num 404) 2020-06-12T02:37:32.061 INFO:tasks.rados.rados.0.smithi107.stdout:406: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:32.061 INFO:tasks.rados.rados.0.smithi107.stdout:406: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:32.061 INFO:tasks.rados.rados.0.smithi107.stdout:406: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:32.061 INFO:tasks.rados.rados.0.smithi107.stdout:406: finishing write tid 5 to smithi10729407-15 2020-06-12T02:37:32.062 INFO:tasks.rados.rados.0.smithi107.stdout:406: finishing write tid 6 to smithi10729407-15 2020-06-12T02:37:32.062 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 89 (ObjNum 405 snap 0 seq_num 405) dirty exists 2020-06-12T02:37:32.062 INFO:tasks.rados.rados.0.smithi107.stdout:406: left oid 15 (ObjNum 405 snap 0 seq_num 405) 2020-06-12T02:37:32.063 INFO:tasks.rados.rados.0.smithi107.stdout:407: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:32.063 INFO:tasks.rados.rados.0.smithi107.stdout:403: done (4 left) 2020-06-12T02:37:32.063 INFO:tasks.rados.rados.0.smithi107.stdout:405: done (3 left) 2020-06-12T02:37:32.064 INFO:tasks.rados.rados.0.smithi107.stdout:406: done (2 left) 2020-06-12T02:37:32.064 INFO:tasks.rados.rados.0.smithi107.stdout:408: done (1 left) 2020-06-12T02:37:32.064 INFO:tasks.rados.rados.0.smithi107.stdout:409: delete oid 19 current snap is 0 2020-06-12T02:37:32.064 INFO:tasks.rados.rados.0.smithi107.stdout:409: done (1 left) 2020-06-12T02:37:32.065 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:37:32.065 INFO:tasks.rados.rados.0.smithi107.stdout:410: seq_num 409 ranges {3776512=458752,4235264=532480,4767744=720896,5488640=352256} 2020-06-12T02:37:32.072 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:37:32.073 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:37:32.073 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2020-06-12T02:37:32.073 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:37:32.082 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-06-12T02:37:32.083 INFO:tasks.rados.rados.0.smithi107.stdout:410: writing smithi10729407-15 from 3776512 to 4235264 tid 1 2020-06-12T02:37:32.086 INFO:tasks.rados.rados.0.smithi107.stdout:410: writing smithi10729407-15 from 4235264 to 4767744 tid 2 2020-06-12T02:37:32.091 INFO:tasks.rados.rados.0.smithi107.stdout:410: writing smithi10729407-15 from 4767744 to 5488640 tid 3 2020-06-12T02:37:32.093 INFO:tasks.rados.rados.0.smithi107.stdout:410: writing smithi10729407-15 from 5488640 to 5840896 tid 4 2020-06-12T02:37:32.094 INFO:tasks.rados.rados.0.smithi107.stdout:411: delete oid 3 current snap is 0 2020-06-12T02:37:32.094 INFO:tasks.rados.rados.0.smithi107.stdout:407: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:32.094 INFO:tasks.rados.rados.0.smithi107.stdout:407: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:32.094 INFO:tasks.rados.rados.0.smithi107.stdout:407: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:32.094 INFO:tasks.rados.rados.0.smithi107.stdout:407: finishing write tid 6 to smithi10729407-6 2020-06-12T02:37:32.095 INFO:tasks.rados.rados.0.smithi107.stdout:407: finishing write tid 7 to smithi10729407-6 2020-06-12T02:37:32.095 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 95 (ObjNum 406 snap 0 seq_num 406) dirty exists 2020-06-12T02:37:32.095 INFO:tasks.rados.rados.0.smithi107.stdout:407: left oid 6 (ObjNum 406 snap 0 seq_num 406) 2020-06-12T02:37:32.095 INFO:tasks.rados.rados.0.smithi107.stdout:410: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:32.096 INFO:tasks.rados.rados.0.smithi107.stdout:407: done (2 left) 2020-06-12T02:37:32.096 INFO:tasks.rados.rados.0.smithi107.stdout:411: done (1 left) 2020-06-12T02:37:32.096 INFO:tasks.rados.rados.0.smithi107.stdout:412: delete oid 14 current snap is 0 2020-06-12T02:37:32.096 INFO:tasks.rados.rados.0.smithi107.stdout:410: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:32.098 INFO:tasks.rados.rados.0.smithi107.stdout:412: done (1 left) 2020-06-12T02:37:32.099 INFO:tasks.rados.rados.0.smithi107.stdout:413: delete oid 6 current snap is 0 2020-06-12T02:37:32.099 INFO:tasks.rados.rados.0.smithi107.stdout:413: done (1 left) 2020-06-12T02:37:32.099 INFO:tasks.rados.rados.0.smithi107.stdout:414: delete oid 11 current snap is 0 2020-06-12T02:37:32.100 INFO:tasks.rados.rados.0.smithi107.stdout:410: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:32.100 INFO:tasks.rados.rados.0.smithi107.stdout:414: done (1 left) 2020-06-12T02:37:32.101 INFO:tasks.rados.rados.0.smithi107.stdout:415: delete oid 16 current snap is 0 2020-06-12T02:37:32.101 INFO:tasks.rados.rados.0.smithi107.stdout:410: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:32.101 INFO:tasks.rados.rados.0.smithi107.stdout:410: finishing write tid 6 to smithi10729407-15 2020-06-12T02:37:32.102 INFO:tasks.rados.rados.0.smithi107.stdout:410: finishing write tid 7 to smithi10729407-15 2020-06-12T02:37:32.102 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 94 (ObjNum 409 snap 0 seq_num 409) dirty exists 2020-06-12T02:37:32.102 INFO:tasks.rados.rados.0.smithi107.stdout:410: left oid 15 (ObjNum 409 snap 0 seq_num 409) 2020-06-12T02:37:32.103 INFO:tasks.rados.rados.0.smithi107.stdout:410: done (1 left) 2020-06-12T02:37:32.103 INFO:tasks.rados.rados.0.smithi107.stdout:415: done (0 left) 2020-06-12T02:37:32.103 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:37:32.104 INFO:tasks.rados.rados.0.smithi107.stdout:416: seq_num 415 ranges {0=557056,557056=688128,1245184=663552,1908736=253952} 2020-06-12T02:37:32.105 INFO:tasks.rados.rados.0.smithi107.stdout:416: writing smithi10729407-10 from 0 to 557056 tid 1 2020-06-12T02:37:32.110 INFO:tasks.rados.rados.0.smithi107.stdout:416: writing smithi10729407-10 from 557056 to 1245184 tid 2 2020-06-12T02:37:32.114 INFO:tasks.rados.rados.0.smithi107.stdout:416: writing smithi10729407-10 from 1245184 to 1908736 tid 3 2020-06-12T02:37:32.117 INFO:tasks.rados.rados.0.smithi107.stdout:416: writing smithi10729407-10 from 1908736 to 2162688 tid 4 2020-06-12T02:37:32.117 INFO:tasks.rados.rados.0.smithi107.stdout:417: delete oid 2 current snap is 0 2020-06-12T02:37:32.117 INFO:tasks.rados.rados.0.smithi107.stdout:416: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:32.120 INFO:tasks.rados.rados.0.smithi107.stdout:416: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:32.122 INFO:tasks.rados.rados.0.smithi107.stdout:416: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:32.123 INFO:tasks.rados.rados.0.smithi107.stdout:417: done (1 left) 2020-06-12T02:37:32.123 INFO:tasks.rados.rados.0.smithi107.stdout:418: delete oid 18 current snap is 0 2020-06-12T02:37:32.124 INFO:tasks.rados.rados.0.smithi107.stdout:416: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:32.124 INFO:tasks.rados.rados.0.smithi107.stdout:416: finishing write tid 6 to smithi10729407-10 2020-06-12T02:37:32.126 INFO:tasks.rados.rados.0.smithi107.stdout:416: finishing write tid 7 to smithi10729407-10 2020-06-12T02:37:32.126 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 45 (ObjNum 415 snap 0 seq_num 415) dirty exists 2020-06-12T02:37:32.126 INFO:tasks.rados.rados.0.smithi107.stdout:416: left oid 10 (ObjNum 415 snap 0 seq_num 415) 2020-06-12T02:37:32.127 INFO:tasks.rados.rados.0.smithi107.stdout:416: done (1 left) 2020-06-12T02:37:32.127 INFO:tasks.rados.rados.0.smithi107.stdout:418: done (0 left) 2020-06-12T02:37:32.127 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:37:32.127 INFO:tasks.rados.rados.0.smithi107.stdout:419: seq_num 418 ranges {0=606208,606208=737280,1343488=434176,1777664=434176} 2020-06-12T02:37:32.129 INFO:tasks.rados.rados.0.smithi107.stdout:419: writing smithi10729407-2 from 0 to 606208 tid 1 2020-06-12T02:37:32.134 INFO:tasks.rados.rados.0.smithi107.stdout:419: writing smithi10729407-2 from 606208 to 1343488 tid 2 2020-06-12T02:37:32.137 INFO:tasks.rados.rados.0.smithi107.stdout:419: writing smithi10729407-2 from 1343488 to 1777664 tid 3 2020-06-12T02:37:32.140 INFO:tasks.rados.rados.0.smithi107.stdout:419: writing smithi10729407-2 from 1777664 to 2211840 tid 4 2020-06-12T02:37:32.140 INFO:tasks.rados.rados.0.smithi107.stdout:420: delete oid 3 current snap is 0 2020-06-12T02:37:32.140 INFO:tasks.rados.rados.0.smithi107.stdout:419: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:32.143 INFO:tasks.rados.rados.0.smithi107.stdout:420: done (1 left) 2020-06-12T02:37:32.143 INFO:tasks.rados.rados.0.smithi107.stdout:421: delete oid 3 current snap is 0 2020-06-12T02:37:32.144 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:32.140+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:32.145 INFO:tasks.rados.rados.0.smithi107.stdout:421: done (1 left) 2020-06-12T02:37:32.145 INFO:tasks.rados.rados.0.smithi107.stdout:422: delete oid 5 current snap is 0 2020-06-12T02:37:32.147 INFO:tasks.rados.rados.0.smithi107.stdout:419: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:32.147 INFO:tasks.rados.rados.0.smithi107.stdout:422: done (1 left) 2020-06-12T02:37:32.148 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:37:32.148 INFO:tasks.rados.rados.0.smithi107.stdout:423: seq_num 422 ranges {0=671744,671744=729088,1400832=417792,1818624=458752} 2020-06-12T02:37:32.151 INFO:tasks.rados.rados.0.smithi107.stdout:423: writing smithi10729407-5 from 0 to 671744 tid 1 2020-06-12T02:37:32.155 INFO:tasks.rados.rados.0.smithi107.stdout:423: writing smithi10729407-5 from 671744 to 1400832 tid 2 2020-06-12T02:37:32.158 INFO:tasks.rados.rados.0.smithi107.stdout:423: writing smithi10729407-5 from 1400832 to 1818624 tid 3 2020-06-12T02:37:32.160 INFO:tasks.rados.rados.0.smithi107.stdout:423: writing smithi10729407-5 from 1818624 to 2277376 tid 4 2020-06-12T02:37:32.161 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:37:32.161 INFO:tasks.rados.rados.0.smithi107.stdout:424: seq_num 423 ranges {0=688128,688128=614400,1302528=622592,1925120=368640} 2020-06-12T02:37:32.164 INFO:tasks.rados.rados.0.smithi107.stdout:424: writing smithi10729407-20 from 0 to 688128 tid 1 2020-06-12T02:37:32.171 INFO:tasks.rados.rados.0.smithi107.stdout:424: writing smithi10729407-20 from 688128 to 1302528 tid 2 2020-06-12T02:37:32.172 INFO:tasks.rados.rados.0.smithi107.stdout:424: writing smithi10729407-20 from 1302528 to 1925120 tid 3 2020-06-12T02:37:32.174 INFO:tasks.rados.rados.0.smithi107.stdout:424: writing smithi10729407-20 from 1925120 to 2293760 tid 4 2020-06-12T02:37:32.175 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:37:32.175 INFO:tasks.rados.rados.0.smithi107.stdout:425: seq_num 424 ranges {0=704512,704512=491520,1196032=434176,1630208=679936} 2020-06-12T02:37:32.178 INFO:tasks.rados.rados.0.smithi107.stdout:425: writing smithi10729407-7 from 0 to 704512 tid 1 2020-06-12T02:37:32.181 INFO:tasks.rados.rados.0.smithi107.stdout:425: writing smithi10729407-7 from 704512 to 1196032 tid 2 2020-06-12T02:37:32.184 INFO:tasks.rados.rados.0.smithi107.stdout:425: writing smithi10729407-7 from 1196032 to 1630208 tid 3 2020-06-12T02:37:32.189 INFO:tasks.rados.rados.0.smithi107.stdout:425: writing smithi10729407-7 from 1630208 to 2310144 tid 4 2020-06-12T02:37:32.189 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:37:32.189 INFO:tasks.rados.rados.0.smithi107.stdout:426: seq_num 425 ranges {1974272=720896,2695168=778240,3473408=638976,4112384=188416} 2020-06-12T02:37:32.209 INFO:tasks.rados.rados.0.smithi107.stdout:426: writing smithi10729407-13 from 1974272 to 2695168 tid 1 2020-06-12T02:37:32.210 INFO:tasks.rados.rados.0.smithi107.stdout:426: writing smithi10729407-13 from 2695168 to 3473408 tid 2 2020-06-12T02:37:32.212 INFO:tasks.rados.rados.0.smithi107.stdout:426: writing smithi10729407-13 from 3473408 to 4112384 tid 3 2020-06-12T02:37:32.214 INFO:tasks.rados.rados.0.smithi107.stdout:426: writing smithi10729407-13 from 4112384 to 4300800 tid 4 2020-06-12T02:37:32.214 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:37:32.214 INFO:tasks.rados.rados.0.smithi107.stdout:427: seq_num 426 ranges {0=737280,737280=712704,1449984=802816,2252800=90112} 2020-06-12T02:37:32.219 INFO:tasks.rados.rados.0.smithi107.stdout:427: writing smithi10729407-4 from 0 to 737280 tid 1 2020-06-12T02:37:32.223 INFO:tasks.rados.rados.0.smithi107.stdout:427: writing smithi10729407-4 from 737280 to 1449984 tid 2 2020-06-12T02:37:32.231 INFO:tasks.rados.rados.0.smithi107.stdout:427: writing smithi10729407-4 from 1449984 to 2252800 tid 3 2020-06-12T02:37:32.231 INFO:tasks.rados.rados.0.smithi107.stdout:427: writing smithi10729407-4 from 2252800 to 2342912 tid 4 2020-06-12T02:37:32.232 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:37:32.232 INFO:tasks.rados.rados.0.smithi107.stdout:428: seq_num 427 ranges {0=761856,761856=598016,1359872=606208,1966080=401408} 2020-06-12T02:37:32.233 INFO:tasks.rados.rados.0.smithi107.stdout:428: writing smithi10729407-9 from 0 to 761856 tid 1 2020-06-12T02:37:32.237 INFO:tasks.rados.rados.0.smithi107.stdout:428: writing smithi10729407-9 from 761856 to 1359872 tid 2 2020-06-12T02:37:32.241 INFO:tasks.rados.rados.0.smithi107.stdout:428: writing smithi10729407-9 from 1359872 to 1966080 tid 3 2020-06-12T02:37:32.244 INFO:tasks.rados.rados.0.smithi107.stdout:428: writing smithi10729407-9 from 1966080 to 2367488 tid 4 2020-06-12T02:37:32.244 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:37:32.244 INFO:tasks.rados.rados.0.smithi107.stdout:429: seq_num 428 ranges {0=778240,778240=483328,1261568=409600,1671168=417792,2088960=294912} 2020-06-12T02:37:32.245 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:37:32.245 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:37:32.245 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2020-06-12T02:37:32.246 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2020-06-12T02:37:32.246 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:37:32.247 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:37:32.241+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:37:32.248 INFO:tasks.rados.rados.0.smithi107.stdout:429: writing smithi10729407-6 from 0 to 778240 tid 1 2020-06-12T02:37:32.251 INFO:tasks.rados.rados.0.smithi107.stdout:429: writing smithi10729407-6 from 778240 to 1261568 tid 2 2020-06-12T02:37:32.254 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-06-12T02:37:32.255 INFO:tasks.rados.rados.0.smithi107.stdout:429: writing smithi10729407-6 from 1261568 to 1671168 tid 3 2020-06-12T02:37:32.257 INFO:tasks.rados.rados.0.smithi107.stdout:429: writing smithi10729407-6 from 1671168 to 2088960 tid 4 2020-06-12T02:37:32.260 INFO:tasks.rados.rados.0.smithi107.stdout:429: writing smithi10729407-6 from 2088960 to 2383872 tid 5 2020-06-12T02:37:32.261 INFO:tasks.rados.rados.0.smithi107.stdout: waiting on 8 2020-06-12T02:37:32.261 INFO:tasks.rados.rados.0.smithi107.stdout:419: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:32.261 INFO:tasks.rados.rados.0.smithi107.stdout:419: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:32.261 INFO:tasks.rados.rados.0.smithi107.stdout:419: finishing write tid 6 to smithi10729407-2 2020-06-12T02:37:32.262 INFO:tasks.rados.rados.0.smithi107.stdout:419: finishing write tid 7 to smithi10729407-2 2020-06-12T02:37:32.262 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 50 (ObjNum 418 snap 0 seq_num 418) dirty exists 2020-06-12T02:37:32.262 INFO:tasks.rados.rados.0.smithi107.stdout:419: left oid 2 (ObjNum 418 snap 0 seq_num 418) 2020-06-12T02:37:32.262 INFO:tasks.rados.rados.0.smithi107.stdout:423: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:32.263 INFO:tasks.rados.rados.0.smithi107.stdout:419: done (7 left) 2020-06-12T02:37:32.263 INFO:tasks.rados.rados.0.smithi107.stdout:430: delete oid 1 current snap is 0 2020-06-12T02:37:32.263 INFO:tasks.rados.rados.0.smithi107.stdout:423: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:32.263 INFO:tasks.rados.rados.0.smithi107.stdout:423: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:32.264 INFO:tasks.rados.rados.0.smithi107.stdout:423: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:32.264 INFO:tasks.rados.rados.0.smithi107.stdout:423: finishing write tid 6 to smithi10729407-5 2020-06-12T02:37:32.264 INFO:tasks.rados.rados.0.smithi107.stdout:424: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:32.264 INFO:tasks.rados.rados.0.smithi107.stdout:423: finishing write tid 7 to smithi10729407-5 2020-06-12T02:37:32.264 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 48 (ObjNum 422 snap 0 seq_num 422) dirty exists 2020-06-12T02:37:32.265 INFO:tasks.rados.rados.0.smithi107.stdout:423: left oid 5 (ObjNum 422 snap 0 seq_num 422) 2020-06-12T02:37:32.265 INFO:tasks.rados.rados.0.smithi107.stdout:424: finishing write tid 2 to smithi10729407-20 2020-06-12T02:37:32.265 INFO:tasks.rados.rados.0.smithi107.stdout:424: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:32.265 INFO:tasks.rados.rados.0.smithi107.stdout:424: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:32.266 INFO:tasks.rados.rados.0.smithi107.stdout:424: finishing write tid 6 to smithi10729407-20 2020-06-12T02:37:32.266 INFO:tasks.rados.rados.0.smithi107.stdout:424: finishing write tid 7 to smithi10729407-20 2020-06-12T02:37:32.267 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 29 (ObjNum 423 snap 0 seq_num 423) dirty exists 2020-06-12T02:37:32.267 INFO:tasks.rados.rados.0.smithi107.stdout:424: left oid 20 (ObjNum 423 snap 0 seq_num 423) 2020-06-12T02:37:32.267 INFO:tasks.rados.rados.0.smithi107.stdout:425: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:32.267 INFO:tasks.rados.rados.0.smithi107.stdout:425: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:32.268 INFO:tasks.rados.rados.0.smithi107.stdout:425: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:32.268 INFO:tasks.rados.rados.0.smithi107.stdout:425: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:32.268 INFO:tasks.rados.rados.0.smithi107.stdout:425: finishing write tid 6 to smithi10729407-7 2020-06-12T02:37:32.268 INFO:tasks.rados.rados.0.smithi107.stdout:425: finishing write tid 7 to smithi10729407-7 2020-06-12T02:37:32.268 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 79 (ObjNum 424 snap 0 seq_num 424) dirty exists 2020-06-12T02:37:32.269 INFO:tasks.rados.rados.0.smithi107.stdout:425: left oid 7 (ObjNum 424 snap 0 seq_num 424) 2020-06-12T02:37:32.269 INFO:tasks.rados.rados.0.smithi107.stdout:426: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:32.269 INFO:tasks.rados.rados.0.smithi107.stdout:426: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:32.269 INFO:tasks.rados.rados.0.smithi107.stdout:426: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:32.269 INFO:tasks.rados.rados.0.smithi107.stdout:426: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:32.269 INFO:tasks.rados.rados.0.smithi107.stdout:426: finishing write tid 6 to smithi10729407-13 2020-06-12T02:37:32.270 INFO:tasks.rados.rados.0.smithi107.stdout:427: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:32.270 INFO:tasks.rados.rados.0.smithi107.stdout:426: finishing write tid 7 to smithi10729407-13 2020-06-12T02:37:32.270 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 84 (ObjNum 425 snap 0 seq_num 425) dirty exists 2020-06-12T02:37:32.271 INFO:tasks.rados.rados.0.smithi107.stdout:426: left oid 13 (ObjNum 425 snap 0 seq_num 425) 2020-06-12T02:37:32.271 INFO:tasks.rados.rados.0.smithi107.stdout:427: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:32.271 INFO:tasks.rados.rados.0.smithi107.stdout:427: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:32.271 INFO:tasks.rados.rados.0.smithi107.stdout:427: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:32.271 INFO:tasks.rados.rados.0.smithi107.stdout:427: finishing write tid 6 to smithi10729407-4 2020-06-12T02:37:32.271 INFO:tasks.rados.rados.0.smithi107.stdout:427: finishing write tid 7 to smithi10729407-4 2020-06-12T02:37:32.272 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 85 (ObjNum 426 snap 0 seq_num 426) dirty exists 2020-06-12T02:37:32.272 INFO:tasks.rados.rados.0.smithi107.stdout:427: left oid 4 (ObjNum 426 snap 0 seq_num 426) 2020-06-12T02:37:32.272 INFO:tasks.rados.rados.0.smithi107.stdout:428: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:32.272 INFO:tasks.rados.rados.0.smithi107.stdout:429: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:32.272 INFO:tasks.rados.rados.0.smithi107.stdout:428: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:32.273 INFO:tasks.rados.rados.0.smithi107.stdout:428: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:32.273 INFO:tasks.rados.rados.0.smithi107.stdout:428: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:32.273 INFO:tasks.rados.rados.0.smithi107.stdout:428: finishing write tid 6 to smithi10729407-9 2020-06-12T02:37:32.273 INFO:tasks.rados.rados.0.smithi107.stdout:428: finishing write tid 7 to smithi10729407-9 2020-06-12T02:37:32.273 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 78 (ObjNum 427 snap 0 seq_num 427) dirty exists 2020-06-12T02:37:32.273 INFO:tasks.rados.rados.0.smithi107.stdout:428: left oid 9 (ObjNum 427 snap 0 seq_num 427) 2020-06-12T02:37:32.275 INFO:tasks.rados.rados.0.smithi107.stdout:429: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:32.276 INFO:tasks.rados.rados.0.smithi107.stdout:429: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:32.276 INFO:tasks.rados.rados.0.smithi107.stdout:429: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:32.276 INFO:tasks.rados.rados.0.smithi107.stdout:429: finishing write tid 5 to smithi10729407-6 2020-06-12T02:37:32.276 INFO:tasks.rados.rados.0.smithi107.stdout:429: finishing write tid 7 to smithi10729407-6 2020-06-12T02:37:32.276 INFO:tasks.rados.rados.0.smithi107.stdout:429: finishing write tid 8 to smithi10729407-6 2020-06-12T02:37:32.277 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 102 (ObjNum 428 snap 0 seq_num 428) dirty exists 2020-06-12T02:37:32.277 INFO:tasks.rados.rados.0.smithi107.stdout:429: left oid 6 (ObjNum 428 snap 0 seq_num 428) 2020-06-12T02:37:32.277 INFO:tasks.rados.rados.0.smithi107.stdout:423: done (7 left) 2020-06-12T02:37:32.277 INFO:tasks.rados.rados.0.smithi107.stdout:424: done (6 left) 2020-06-12T02:37:32.277 INFO:tasks.rados.rados.0.smithi107.stdout:425: done (5 left) 2020-06-12T02:37:32.278 INFO:tasks.rados.rados.0.smithi107.stdout:426: done (4 left) 2020-06-12T02:37:32.278 INFO:tasks.rados.rados.0.smithi107.stdout:427: done (3 left) 2020-06-12T02:37:32.278 INFO:tasks.rados.rados.0.smithi107.stdout:428: done (2 left) 2020-06-12T02:37:32.278 INFO:tasks.rados.rados.0.smithi107.stdout:429: done (1 left) 2020-06-12T02:37:32.278 INFO:tasks.rados.rados.0.smithi107.stdout:430: done (0 left) 2020-06-12T02:37:32.279 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:37:32.279 INFO:tasks.rados.rados.0.smithi107.stdout:431: seq_num 430 ranges {3186688=8192} 2020-06-12T02:37:32.286 INFO:tasks.rados.rados.0.smithi107.stdout:431: writing smithi10729407-8 from 3186688 to 3194880 tid 1 2020-06-12T02:37:32.287 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:37:32.287 INFO:tasks.rados.rados.0.smithi107.stdout:432: seq_num 431 ranges {2211840=24576} 2020-06-12T02:37:32.299 INFO:tasks.rados.rados.0.smithi107.stdout:432: writing smithi10729407-2 from 2211840 to 2236416 tid 1 2020-06-12T02:37:32.299 INFO:tasks.rados.rados.0.smithi107.stdout:433: delete oid 7 current snap is 0 2020-06-12T02:37:32.301 INFO:tasks.rados.rados.0.smithi107.stdout:431: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:32.301 INFO:tasks.rados.rados.0.smithi107.stdout:431: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:32.301 INFO:tasks.rados.rados.0.smithi107.stdout:431: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:32.302 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 96 (ObjNum 430 snap 0 seq_num 430) dirty exists 2020-06-12T02:37:32.302 INFO:tasks.rados.rados.0.smithi107.stdout:431: left oid 8 (ObjNum 430 snap 0 seq_num 430) 2020-06-12T02:37:32.302 INFO:tasks.rados.rados.0.smithi107.stdout:431: done (2 left) 2020-06-12T02:37:32.303 INFO:tasks.rados.rados.0.smithi107.stdout:433: done (1 left) 2020-06-12T02:37:32.303 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:37:32.303 INFO:tasks.rados.rados.0.smithi107.stdout:434: seq_num 433 ranges {2383872=57344} 2020-06-12T02:37:32.314 INFO:tasks.rados.rados.0.smithi107.stdout:434: writing smithi10729407-6 from 2383872 to 2441216 tid 1 2020-06-12T02:37:32.314 INFO:tasks.rados.rados.0.smithi107.stdout:435: delete oid 11 current snap is 0 2020-06-12T02:37:32.315 INFO:tasks.rados.rados.0.smithi107.stdout:432: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:32.315 INFO:tasks.rados.rados.0.smithi107.stdout:432: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:32.315 INFO:tasks.rados.rados.0.smithi107.stdout:432: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:32.315 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 52 (ObjNum 431 snap 0 seq_num 431) dirty exists 2020-06-12T02:37:32.316 INFO:tasks.rados.rados.0.smithi107.stdout:432: left oid 2 (ObjNum 431 snap 0 seq_num 431) 2020-06-12T02:37:32.316 INFO:tasks.rados.rados.0.smithi107.stdout:434: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:32.319 INFO:tasks.rados.rados.0.smithi107.stdout:434: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:32.320 INFO:tasks.rados.rados.0.smithi107.stdout:434: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:32.320 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 105 (ObjNum 433 snap 0 seq_num 433) dirty exists 2020-06-12T02:37:32.320 INFO:tasks.rados.rados.0.smithi107.stdout:434: left oid 6 (ObjNum 433 snap 0 seq_num 433) 2020-06-12T02:37:32.321 INFO:tasks.rados.rados.0.smithi107.stdout:432: done (2 left) 2020-06-12T02:37:32.322 INFO:tasks.rados.rados.0.smithi107.stdout:434: done (1 left) 2020-06-12T02:37:32.322 INFO:tasks.rados.rados.0.smithi107.stdout:435: done (0 left) 2020-06-12T02:37:32.322 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:37:32.322 INFO:tasks.rados.rados.0.smithi107.stdout:436: seq_num 435 ranges {3145728=90112} 2020-06-12T02:37:32.339 INFO:tasks.rados.rados.0.smithi107.stdout:436: writing smithi10729407-17 from 3145728 to 3235840 tid 1 2020-06-12T02:37:32.340 INFO:tasks.rados.rados.0.smithi107.stdout:437: delete oid 12 current snap is 0 2020-06-12T02:37:32.341 INFO:tasks.rados.rados.0.smithi107.stdout:437: done (1 left) 2020-06-12T02:37:32.341 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:37:32.342 INFO:tasks.rados.rados.0.smithi107.stdout:438: seq_num 437 ranges {0=122880} 2020-06-12T02:37:32.342 INFO:tasks.rados.rados.0.smithi107.stdout:438: writing smithi10729407-16 from 0 to 122880 tid 1 2020-06-12T02:37:32.343 INFO:tasks.rados.rados.0.smithi107.stdout:439: delete oid 9 current snap is 0 2020-06-12T02:37:32.343 INFO:tasks.rados.rados.0.smithi107.stdout:436: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:32.344 INFO:tasks.rados.rados.0.smithi107.stdout:436: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:32.344 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:32.342+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:32.345 INFO:tasks.rados.rados.0.smithi107.stdout:436: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:32.345 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 87 (ObjNum 435 snap 0 seq_num 435) dirty exists 2020-06-12T02:37:32.345 INFO:tasks.rados.rados.0.smithi107.stdout:436: left oid 17 (ObjNum 435 snap 0 seq_num 435) 2020-06-12T02:37:32.346 INFO:tasks.rados.rados.0.smithi107.stdout:438: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:32.346 INFO:tasks.rados.rados.0.smithi107.stdout:436: done (2 left) 2020-06-12T02:37:32.347 INFO:tasks.rados.rados.0.smithi107.stdout:439: done (1 left) 2020-06-12T02:37:32.347 INFO:tasks.rados.rados.0.smithi107.stdout:440: delete oid 8 current snap is 0 2020-06-12T02:37:32.352 INFO:tasks.rados.rados.0.smithi107.stdout:440: done (1 left) 2020-06-12T02:37:32.353 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:37:32.353 INFO:tasks.rados.rados.0.smithi107.stdout:441: seq_num 440 ranges {0=172032} 2020-06-12T02:37:32.354 INFO:tasks.rados.rados.0.smithi107.stdout:441: writing smithi10729407-1 from 0 to 172032 tid 1 2020-06-12T02:37:32.354 INFO:tasks.rados.rados.0.smithi107.stdout:442: delete oid 4 current snap is 0 2020-06-12T02:37:32.354 INFO:tasks.rados.rados.0.smithi107.stdout:438: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:32.355 INFO:tasks.rados.rados.0.smithi107.stdout:438: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:32.356 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 73 (ObjNum 437 snap 0 seq_num 437) dirty exists 2020-06-12T02:37:32.356 INFO:tasks.rados.rados.0.smithi107.stdout:438: left oid 16 (ObjNum 437 snap 0 seq_num 437) 2020-06-12T02:37:32.357 INFO:tasks.rados.rados.0.smithi107.stdout:438: done (2 left) 2020-06-12T02:37:32.358 INFO:tasks.rados.rados.0.smithi107.stdout:442: done (1 left) 2020-06-12T02:37:32.358 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:37:32.358 INFO:tasks.rados.rados.0.smithi107.stdout:443: seq_num 442 ranges {2293760=204800} 2020-06-12T02:37:32.372 INFO:tasks.rados.rados.0.smithi107.stdout:443: writing smithi10729407-20 from 2293760 to 2498560 tid 1 2020-06-12T02:37:32.375 INFO:tasks.rados.rados.0.smithi107.stdout:444: delete oid 11 current snap is 0 2020-06-12T02:37:32.375 INFO:tasks.rados.rados.0.smithi107.stdout:441: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:32.375 INFO:tasks.rados.rados.0.smithi107.stdout:441: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:32.375 INFO:tasks.rados.rados.0.smithi107.stdout:441: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:32.376 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 107 (ObjNum 440 snap 0 seq_num 440) dirty exists 2020-06-12T02:37:32.376 INFO:tasks.rados.rados.0.smithi107.stdout:441: left oid 1 (ObjNum 440 snap 0 seq_num 440) 2020-06-12T02:37:32.377 INFO:tasks.rados.rados.0.smithi107.stdout:441: done (2 left) 2020-06-12T02:37:32.377 INFO:tasks.rados.rados.0.smithi107.stdout:444: done (1 left) 2020-06-12T02:37:32.377 INFO:tasks.rados.rados.0.smithi107.stdout:445: delete oid 10 current snap is 0 2020-06-12T02:37:32.378 INFO:tasks.rados.rados.0.smithi107.stdout:443: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:32.378 INFO:tasks.rados.rados.0.smithi107.stdout:445: done (1 left) 2020-06-12T02:37:32.378 INFO:tasks.rados.rados.0.smithi107.stdout:446: delete oid 5 current snap is 0 2020-06-12T02:37:32.378 INFO:tasks.rados.rados.0.smithi107.stdout:443: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:32.379 INFO:tasks.rados.rados.0.smithi107.stdout:443: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:32.379 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 31 (ObjNum 442 snap 0 seq_num 442) dirty exists 2020-06-12T02:37:32.379 INFO:tasks.rados.rados.0.smithi107.stdout:443: left oid 20 (ObjNum 442 snap 0 seq_num 442) 2020-06-12T02:37:32.380 INFO:tasks.rados.rados.0.smithi107.stdout:443: done (1 left) 2020-06-12T02:37:32.380 INFO:tasks.rados.rados.0.smithi107.stdout:446: done (0 left) 2020-06-12T02:37:32.380 INFO:tasks.rados.rados.0.smithi107.stdout:447: delete oid 20 current snap is 0 2020-06-12T02:37:32.381 INFO:tasks.rados.rados.0.smithi107.stdout:447: done (0 left) 2020-06-12T02:37:32.381 INFO:tasks.rados.rados.0.smithi107.stdout:448: delete oid 13 current snap is 0 2020-06-12T02:37:32.384 INFO:tasks.rados.rados.0.smithi107.stdout:448: done (0 left) 2020-06-12T02:37:32.384 INFO:tasks.rados.rados.0.smithi107.stdout:449: delete oid 14 current snap is 0 2020-06-12T02:37:32.386 INFO:tasks.rados.rados.0.smithi107.stdout:449: done (0 left) 2020-06-12T02:37:32.387 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:37:32.387 INFO:tasks.rados.rados.0.smithi107.stdout:450: seq_num 449 ranges {0=327680} 2020-06-12T02:37:32.389 INFO:tasks.rados.rados.0.smithi107.stdout:450: writing smithi10729407-20 from 0 to 327680 tid 1 2020-06-12T02:37:32.389 INFO:tasks.rados.rados.0.smithi107.stdout:451: delete oid 1 current snap is 0 2020-06-12T02:37:32.390 INFO:tasks.rados.rados.0.smithi107.stdout:451: done (1 left) 2020-06-12T02:37:32.390 INFO:tasks.rados.rados.0.smithi107.stdout:452: delete oid 6 current snap is 0 2020-06-12T02:37:32.392 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:37:32.392 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2020-06-12T02:37:32.392 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2020-06-12T02:37:32.393 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2020-06-12T02:37:32.393 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.393 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.9s1 25/21 ECSubWrite(tid=408, reqid=client.4512.0:790, at_version=0'0, trim_to=0'0, roll_forward_to=25'66))", 2020-06-12T02:37:32.393 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:31.748926+0000", 2020-06-12T02:37:32.393 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.63962774600000005, 2020-06-12T02:37:32.394 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0018965430000000001, 2020-06-12T02:37:32.394 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:32.394 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:32.394 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:32.394 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.394 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:32.395 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.748926+0000", 2020-06-12T02:37:32.395 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:32.395 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.395 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.395 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:32.396 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.748926+0000", 2020-06-12T02:37:32.396 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2969000000000001e-05 2020-06-12T02:37:32.396 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.396 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.397 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:32.397 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.748979+0000", 2020-06-12T02:37:32.397 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5231999999999999e-05 2020-06-12T02:37:32.398 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.398 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.398 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:32.399 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.748995+0000", 2020-06-12T02:37:32.399 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.451e-05 2020-06-12T02:37:32.399 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.399 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.400 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:32.400 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.749009+0000", 2020-06-12T02:37:32.400 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2339999999999999e-06 2020-06-12T02:37:32.400 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.400 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.401 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:32.401 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.749015+0000", 2020-06-12T02:37:32.401 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3694000000000001e-05 2020-06-12T02:37:32.401 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.401 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.402 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:32.402 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.749109+0000", 2020-06-12T02:37:32.402 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7820000000000005e-06 2020-06-12T02:37:32.402 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.402 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.403 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:32.403 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.749119+0000", 2020-06-12T02:37:32.404 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0016549760000000001 2020-06-12T02:37:32.404 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.405 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.405 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:32.405 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.750774+0000", 2020-06-12T02:37:32.405 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9146e-05 2020-06-12T02:37:32.405 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.406 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.406 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:32.406 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.750823+0000", 2020-06-12T02:37:32.406 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0018965430000000001 2020-06-12T02:37:32.406 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.406 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:32.407 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.407 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.407 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.407 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.9s1 25/21 ECSubWrite(tid=411, reqid=client.4512.0:820, at_version=0'0, trim_to=0'0, roll_forward_to=25'67))", 2020-06-12T02:37:32.407 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:31.813466+0000", 2020-06-12T02:37:32.408 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.57508826700000004, 2020-06-12T02:37:32.408 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002949035, 2020-06-12T02:37:32.408 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:32.408 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:32.408 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:32.409 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.409 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:32.409 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.813466+0000", 2020-06-12T02:37:32.409 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:32.409 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.410 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.410 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:32.410 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.813466+0000", 2020-06-12T02:37:32.410 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7157000000000002e-05 2020-06-12T02:37:32.410 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.411 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.411 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:32.411 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.813523+0000", 2020-06-12T02:37:32.412 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6679000000000001e-05 2020-06-12T02:37:32.412 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.412 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.412 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:32.412 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.813540+0000", 2020-06-12T02:37:32.412 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6677999999999999e-05 2020-06-12T02:37:32.413 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.413 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.413 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:32.413 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.813556+0000", 2020-06-12T02:37:32.413 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6329999999999996e-06 2020-06-12T02:37:32.414 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.414 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.414 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:32.414 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.813564+0000", 2020-06-12T02:37:32.414 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010661 2020-06-12T02:37:32.415 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.415 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.415 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:32.415 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.813671+0000", 2020-06-12T02:37:32.415 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2058999999999999e-05 2020-06-12T02:37:32.416 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.416 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.416 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:32.416 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.813683+0000", 2020-06-12T02:37:32.416 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0026801279999999999 2020-06-12T02:37:32.416 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.417 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.417 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:32.417 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.816363+0000", 2020-06-12T02:37:32.417 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2091e-05 2020-06-12T02:37:32.417 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.418 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.418 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:32.418 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.816415+0000", 2020-06-12T02:37:32.418 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002949035 2020-06-12T02:37:32.419 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.419 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:32.419 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.419 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.419 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.420 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "pg_update_log_missing(3.0s1 epoch 25/21 rep_tid 54 entries 25'34 (0'0) error 3:dddb6900:::smithi10729407-10:head by client.4512.0:827 0.000000 -2 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 25'32)", 2020-06-12T02:37:32.420 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:31.824540+0000", 2020-06-12T02:37:32.420 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.56401414400000005, 2020-06-12T02:37:32.421 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0024324619999999998, 2020-06-12T02:37:32.421 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:32.421 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:32.421 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:32.422 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.422 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:32.422 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.824540+0000", 2020-06-12T02:37:32.422 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:32.422 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.422 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.423 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:32.423 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.824540+0000", 2020-06-12T02:37:32.423 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3236000000000003e-05 2020-06-12T02:37:32.423 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.423 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.424 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:32.424 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.824593+0000", 2020-06-12T02:37:32.424 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6909e-05 2020-06-12T02:37:32.424 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.424 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.424 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:32.425 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.824610+0000", 2020-06-12T02:37:32.425 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9262999999999998e-05 2020-06-12T02:37:32.425 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.425 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.425 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:32.426 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.824629+0000", 2020-06-12T02:37:32.426 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4659999999999999e-06 2020-06-12T02:37:32.426 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.426 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.426 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:32.427 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.824638+0000", 2020-06-12T02:37:32.427 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7886000000000002e-05 2020-06-12T02:37:32.427 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.427 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.428 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:32.428 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.824686+0000", 2020-06-12T02:37:32.428 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0022867019999999998 2020-06-12T02:37:32.428 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.428 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.428 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:32.429 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.826972+0000", 2020-06-12T02:37:32.429 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0024324619999999998 2020-06-12T02:37:32.429 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.429 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:32.429 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.430 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.430 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.430 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 25/21 ECSubWrite(tid=55, reqid=client.4512.0:838, at_version=25'35, trim_to=0'0, roll_forward_to=25'33))", 2020-06-12T02:37:32.430 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:31.860963+0000", 2020-06-12T02:37:32.430 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.52759073000000001, 2020-06-12T02:37:32.431 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027087320000000002, 2020-06-12T02:37:32.431 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:32.431 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:32.431 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:32.432 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.432 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:32.432 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.860963+0000", 2020-06-12T02:37:32.432 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:32.433 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.433 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.433 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:32.433 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.860963+0000", 2020-06-12T02:37:32.434 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015438600000000001 2020-06-12T02:37:32.434 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.434 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.434 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:32.434 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.861118+0000", 2020-06-12T02:37:32.434 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3998e-05 2020-06-12T02:37:32.435 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.435 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.435 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:32.435 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.861142+0000", 2020-06-12T02:37:32.435 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6690000000000001e-05 2020-06-12T02:37:32.436 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.436 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.436 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:32.436 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.861159+0000", 2020-06-12T02:37:32.436 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4909999999999999e-06 2020-06-12T02:37:32.437 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.437 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.437 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:32.437 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.861166+0000", 2020-06-12T02:37:32.438 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8192000000000001e-05 2020-06-12T02:37:32.438 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.438 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.438 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:32.438 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.861224+0000", 2020-06-12T02:37:32.439 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1871e-05 2020-06-12T02:37:32.439 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.439 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.440 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:32.440 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.861236+0000", 2020-06-12T02:37:32.440 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002383872 2020-06-12T02:37:32.440 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.440 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.441 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:32.441 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.863620+0000", 2020-06-12T02:37:32.441 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2231999999999997e-05 2020-06-12T02:37:32.441 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.442 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.442 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:32.442 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.863672+0000", 2020-06-12T02:37:32.442 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027087320000000002 2020-06-12T02:37:32.442 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.442 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:32.443 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.443 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.443 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.443 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 25/21 ECSubWrite(tid=56, reqid=client.4512.0:839, at_version=25'36, trim_to=0'0, roll_forward_to=25'33))", 2020-06-12T02:37:32.443 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:31.864514+0000", 2020-06-12T02:37:32.444 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.52404027200000003, 2020-06-12T02:37:32.444 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0034467529999999999, 2020-06-12T02:37:32.444 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:32.444 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:32.444 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:32.444 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.445 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:32.445 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.864514+0000", 2020-06-12T02:37:32.445 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:32.445 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.445 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.446 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:32.446 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.864514+0000", 2020-06-12T02:37:32.446 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000134296 2020-06-12T02:37:32.446 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.446 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.446 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:32.447 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.864648+0000", 2020-06-12T02:37:32.447 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2963999999999999e-05 2020-06-12T02:37:32.447 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.447 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.447 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:32.448 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.864671+0000", 2020-06-12T02:37:32.448 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8949000000000001e-05 2020-06-12T02:37:32.448 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.449 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.449 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:32.449 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.864690+0000", 2020-06-12T02:37:32.449 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8799999999999997e-06 2020-06-12T02:37:32.449 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.449 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.450 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:32.450 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.864699+0000", 2020-06-12T02:37:32.450 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000108707 2020-06-12T02:37:32.450 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.450 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.451 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:32.451 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.864808+0000", 2020-06-12T02:37:32.451 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3638000000000001e-05 2020-06-12T02:37:32.451 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.451 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.451 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:32.452 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.864821+0000", 2020-06-12T02:37:32.452 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0030785209999999999 2020-06-12T02:37:32.452 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.452 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.452 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:32.453 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.867900+0000", 2020-06-12T02:37:32.453 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0798000000000003e-05 2020-06-12T02:37:32.453 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.453 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.453 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:32.453 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.867961+0000", 2020-06-12T02:37:32.454 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0034467529999999999 2020-06-12T02:37:32.454 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.454 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:32.454 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.454 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.455 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.455 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 25/21 ECSubWrite(tid=58, reqid=client.4512.0:840, at_version=25'37, trim_to=0'0, roll_forward_to=25'36))", 2020-06-12T02:37:32.455 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:31.869639+0000", 2020-06-12T02:37:32.456 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.518914969, 2020-06-12T02:37:32.456 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0048985859999999999, 2020-06-12T02:37:32.456 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:32.456 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:32.456 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:32.457 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.457 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:32.457 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.869639+0000", 2020-06-12T02:37:32.457 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:32.457 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.458 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.458 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:32.458 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.869639+0000", 2020-06-12T02:37:32.458 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000136117 2020-06-12T02:37:32.459 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.459 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.459 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:32.460 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.869775+0000", 2020-06-12T02:37:32.460 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8303000000000001e-05 2020-06-12T02:37:32.460 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.461 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.461 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:32.461 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.869794+0000", 2020-06-12T02:37:32.462 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5994999999999998e-05 2020-06-12T02:37:32.462 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.462 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.463 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:32.463 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.869810+0000", 2020-06-12T02:37:32.463 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9789999999999998e-06 2020-06-12T02:37:32.464 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.464 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.464 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:32.465 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.869816+0000", 2020-06-12T02:37:32.465 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8430999999999994e-05 2020-06-12T02:37:32.465 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.465 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.465 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:32.466 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.869904+0000", 2020-06-12T02:37:32.466 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1843e-05 2020-06-12T02:37:32.466 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.467 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.467 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:32.467 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.869916+0000", 2020-06-12T02:37:32.467 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045684649999999999 2020-06-12T02:37:32.468 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.468 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.468 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:32.468 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.874484+0000", 2020-06-12T02:37:32.468 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3452999999999998e-05 2020-06-12T02:37:32.469 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.469 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.469 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:32.469 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.874538+0000", 2020-06-12T02:37:32.469 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0048985859999999999 2020-06-12T02:37:32.470 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.470 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:32.470 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.470 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.470 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.471 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "pg_update_log_missing(3.9s1 epoch 25/21 rep_tid 419 entries 25'68 (0'0) error 3:93ea0cad:::smithi10729407-19:head by client.4512.0:845 0.000000 -2 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 25'66)", 2020-06-12T02:37:32.471 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:31.872263+0000", 2020-06-12T02:37:32.471 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.51629143, 2020-06-12T02:37:32.471 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001363361, 2020-06-12T02:37:32.471 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:32.471 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:32.472 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:32.472 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.472 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:32.472 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.872263+0000", 2020-06-12T02:37:32.472 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:32.473 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.473 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.473 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:32.473 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.872263+0000", 2020-06-12T02:37:32.473 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6042000000000003e-05 2020-06-12T02:37:32.474 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.474 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.474 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:32.474 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.872319+0000", 2020-06-12T02:37:32.474 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6424000000000001e-05 2020-06-12T02:37:32.475 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.475 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.475 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:32.475 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.872335+0000", 2020-06-12T02:37:32.475 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1852e-05 2020-06-12T02:37:32.476 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.476 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.476 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:32.476 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.872357+0000", 2020-06-12T02:37:32.476 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6029999999999994e-06 2020-06-12T02:37:32.477 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.477 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.477 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:32.477 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.872367+0000", 2020-06-12T02:37:32.478 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000108275 2020-06-12T02:37:32.478 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.478 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.478 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:32.479 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.872475+0000", 2020-06-12T02:37:32.479 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001151165 2020-06-12T02:37:32.479 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.479 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.479 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:32.480 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.873626+0000", 2020-06-12T02:37:32.480 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001363361 2020-06-12T02:37:32.480 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.480 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:32.480 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.481 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.481 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.481 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 25/21 ECSubWrite(tid=60, reqid=client.4512.0:841, at_version=0'0, trim_to=0'0, roll_forward_to=25'38))", 2020-06-12T02:37:32.481 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:31.875391+0000", 2020-06-12T02:37:32.481 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.51316311599999997, 2020-06-12T02:37:32.481 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001638432, 2020-06-12T02:37:32.482 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:32.482 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:32.482 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:32.482 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.483 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:32.483 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.875391+0000", 2020-06-12T02:37:32.483 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:32.483 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.483 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.484 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:32.484 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.875391+0000", 2020-06-12T02:37:32.484 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2318000000000003e-05 2020-06-12T02:37:32.484 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.484 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.485 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:32.485 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.875443+0000", 2020-06-12T02:37:32.485 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6453e-05 2020-06-12T02:37:32.486 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.486 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.486 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:32.487 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.875460+0000", 2020-06-12T02:37:32.487 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5704000000000001e-05 2020-06-12T02:37:32.487 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.487 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.488 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:32.488 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.875476+0000", 2020-06-12T02:37:32.488 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1460000000000001e-06 2020-06-12T02:37:32.488 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.488 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.489 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:32.489 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.875482+0000", 2020-06-12T02:37:32.489 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9573000000000006e-05 2020-06-12T02:37:32.489 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.489 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.489 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:32.490 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.875551+0000", 2020-06-12T02:37:32.490 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.082e-05 2020-06-12T02:37:32.490 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.490 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.490 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:32.491 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.875562+0000", 2020-06-12T02:37:32.491 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014257300000000001 2020-06-12T02:37:32.491 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.491 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.491 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:32.491 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.876988+0000", 2020-06-12T02:37:32.492 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1687999999999998e-05 2020-06-12T02:37:32.492 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.492 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.492 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:32.492 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.877029+0000", 2020-06-12T02:37:32.493 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001638432 2020-06-12T02:37:32.493 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.493 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:32.493 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.493 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.494 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.494 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpRead(3.0s1 25/21 ECSubRead(tid=61, to_read={3:0dddb690:::smithi10729407-10:head=0,4096,0}, subchunks={3:0dddb690:::smithi10729407-10:head=[0,1]}, attrs_to_read=))", 2020-06-12T02:37:32.494 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:31.875865+0000", 2020-06-12T02:37:32.494 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.51268928400000002, 2020-06-12T02:37:32.494 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001120436, 2020-06-12T02:37:32.495 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:32.495 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:32.495 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:32.495 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.496 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:32.496 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.875865+0000", 2020-06-12T02:37:32.496 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:32.496 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.497 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.497 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:32.497 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.875865+0000", 2020-06-12T02:37:32.497 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.849e-05 2020-06-12T02:37:32.498 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.498 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.498 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:32.499 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.875893+0000", 2020-06-12T02:37:32.499 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2629999999999997e-06 2020-06-12T02:37:32.499 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.499 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.500 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:32.500 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.875901+0000", 2020-06-12T02:37:32.500 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2007e-05 2020-06-12T02:37:32.501 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.501 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.501 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:32.502 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.875913+0000", 2020-06-12T02:37:32.502 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5486999999999999e-05 2020-06-12T02:37:32.502 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.503 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.503 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:32.503 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.875928+0000", 2020-06-12T02:37:32.504 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4654999999999999e-05 2020-06-12T02:37:32.504 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.504 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.505 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:32.505 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.875963+0000", 2020-06-12T02:37:32.505 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0010225340000000001 2020-06-12T02:37:32.506 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.506 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.506 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:32.507 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.876985+0000", 2020-06-12T02:37:32.507 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001120436 2020-06-12T02:37:32.507 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.508 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:32.508 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.508 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.508 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.509 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.9s1 25/21 ECSubWrite(tid=420, reqid=client.4512.0:847, at_version=25'69, trim_to=0'0, roll_forward_to=25'67))", 2020-06-12T02:37:32.509 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:31.881061+0000", 2020-06-12T02:37:32.509 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.50749308900000001, 2020-06-12T02:37:32.510 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002773148, 2020-06-12T02:37:32.510 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:32.510 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:32.511 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:32.511 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.511 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:32.511 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.881061+0000", 2020-06-12T02:37:32.512 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:32.512 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.512 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.512 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:32.512 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.881061+0000", 2020-06-12T02:37:32.513 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000341252 2020-06-12T02:37:32.513 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.513 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.513 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:32.513 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.881402+0000", 2020-06-12T02:37:32.513 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.4189000000000002e-05 2020-06-12T02:37:32.514 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.514 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.514 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:32.514 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.881427+0000", 2020-06-12T02:37:32.514 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9117e-05 2020-06-12T02:37:32.515 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.515 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.515 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:32.515 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.881446+0000", 2020-06-12T02:37:32.515 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3520000000000007e-06 2020-06-12T02:37:32.516 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.516 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.516 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:32.516 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.881454+0000", 2020-06-12T02:37:32.516 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1829000000000002e-05 2020-06-12T02:37:32.516 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.517 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.517 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:32.517 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.881496+0000", 2020-06-12T02:37:32.517 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7510000000000008e-06 2020-06-12T02:37:32.517 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.518 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.518 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:32.518 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.881504+0000", 2020-06-12T02:37:32.518 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00229758 2020-06-12T02:37:32.518 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.518 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.519 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:32.519 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.883801+0000", 2020-06-12T02:37:32.519 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3077999999999997e-05 2020-06-12T02:37:32.519 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.519 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.520 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:32.520 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.883834+0000", 2020-06-12T02:37:32.520 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002773148 2020-06-12T02:37:32.520 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.520 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:32.521 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.521 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.521 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.521 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.9s1 25/21 ECSubWrite(tid=422, reqid=client.4512.0:848, at_version=25'70, trim_to=0'0, roll_forward_to=25'69))", 2020-06-12T02:37:32.522 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:31.885853+0000", 2020-06-12T02:37:32.522 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.50270084699999995, 2020-06-12T02:37:32.522 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.005286677, 2020-06-12T02:37:32.523 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:32.523 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:32.523 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:32.523 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.523 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:32.524 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.885853+0000", 2020-06-12T02:37:32.524 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:32.524 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.524 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.524 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:32.524 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.885853+0000", 2020-06-12T02:37:32.525 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047056399999999998 2020-06-12T02:37:32.525 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.525 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.525 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:32.525 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.886324+0000", 2020-06-12T02:37:32.526 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8995999999999999e-05 2020-06-12T02:37:32.526 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.526 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.526 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:32.526 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.886343+0000", 2020-06-12T02:37:32.527 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1165999999999999e-05 2020-06-12T02:37:32.527 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.527 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.527 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:32.527 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.886354+0000", 2020-06-12T02:37:32.527 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2329999999999997e-06 2020-06-12T02:37:32.528 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.528 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.528 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:32.528 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.886359+0000", 2020-06-12T02:37:32.528 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6804999999999997e-05 2020-06-12T02:37:32.529 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.529 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.529 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:32.529 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.886396+0000", 2020-06-12T02:37:32.529 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6530000000000006e-06 2020-06-12T02:37:32.530 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.530 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.530 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:32.530 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.886406+0000", 2020-06-12T02:37:32.530 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0047012759999999999 2020-06-12T02:37:32.531 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.531 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.531 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:32.531 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.891107+0000", 2020-06-12T02:37:32.531 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2984000000000001e-05 2020-06-12T02:37:32.532 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.532 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.532 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:32.532 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.891140+0000", 2020-06-12T02:37:32.532 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.005286677 2020-06-12T02:37:32.533 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.533 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:32.533 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.533 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.534 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.534 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.9s1 25/21 ECSubWrite(tid=423, reqid=client.4512.0:849, at_version=25'71, trim_to=0'0, roll_forward_to=25'69))", 2020-06-12T02:37:32.534 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:31.888974+0000", 2020-06-12T02:37:32.534 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.49958024699999998, 2020-06-12T02:37:32.535 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025031400000000001, 2020-06-12T02:37:32.535 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:32.535 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:32.535 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:32.535 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.536 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:32.536 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.888974+0000", 2020-06-12T02:37:32.536 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:32.536 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.536 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.536 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:32.537 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.888974+0000", 2020-06-12T02:37:32.537 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00020606699999999999 2020-06-12T02:37:32.537 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.537 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.537 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:32.538 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.889180+0000", 2020-06-12T02:37:32.538 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4266e-05 2020-06-12T02:37:32.538 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.538 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.538 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:32.539 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.889194+0000", 2020-06-12T02:37:32.539 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8220000000000002e-06 2020-06-12T02:37:32.539 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.539 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.539 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:32.539 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.889204+0000", 2020-06-12T02:37:32.540 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9389999999999996e-06 2020-06-12T02:37:32.540 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.540 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.540 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:32.540 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.889209+0000", 2020-06-12T02:37:32.541 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000336335 2020-06-12T02:37:32.541 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.541 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.541 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:32.542 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.889545+0000", 2020-06-12T02:37:32.542 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3820000000000001e-06 2020-06-12T02:37:32.542 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.542 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.542 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:32.543 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.889552+0000", 2020-06-12T02:37:32.543 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00187967 2020-06-12T02:37:32.543 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.543 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.543 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:32.543 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.891431+0000", 2020-06-12T02:37:32.544 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5658999999999998e-05 2020-06-12T02:37:32.544 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.544 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.544 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:32.544 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.891477+0000", 2020-06-12T02:37:32.545 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025031400000000001 2020-06-12T02:37:32.545 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.545 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:32.545 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.545 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.546 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.546 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 25/21 ECSubWrite(tid=63, reqid=client.4512.0:876, at_version=0'0, trim_to=0'0, roll_forward_to=25'39))", 2020-06-12T02:37:32.546 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:31.985146+0000", 2020-06-12T02:37:32.546 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.40340825200000002, 2020-06-12T02:37:32.546 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0015393990000000001, 2020-06-12T02:37:32.547 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:32.547 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:32.547 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:32.547 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.547 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:32.548 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.985146+0000", 2020-06-12T02:37:32.548 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:32.548 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.548 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.548 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:32.549 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.985146+0000", 2020-06-12T02:37:32.549 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2038999999999993e-05 2020-06-12T02:37:32.549 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.549 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.550 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:32.550 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.985208+0000", 2020-06-12T02:37:32.550 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6194000000000001e-05 2020-06-12T02:37:32.550 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.550 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.550 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:32.551 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.985224+0000", 2020-06-12T02:37:32.551 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7127000000000001e-05 2020-06-12T02:37:32.551 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.551 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.551 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:32.552 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.985241+0000", 2020-06-12T02:37:32.552 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7640000000000005e-06 2020-06-12T02:37:32.552 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.552 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.552 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:32.552 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.985249+0000", 2020-06-12T02:37:32.553 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4368e-05 2020-06-12T02:37:32.553 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.553 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.553 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:32.553 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.985283+0000", 2020-06-12T02:37:32.554 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5180000000000001e-06 2020-06-12T02:37:32.554 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.554 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.554 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:32.555 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.985290+0000", 2020-06-12T02:37:32.555 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001364538 2020-06-12T02:37:32.555 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.556 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.556 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:32.556 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.986654+0000", 2020-06-12T02:37:32.556 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0851e-05 2020-06-12T02:37:32.557 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.557 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.557 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:32.557 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:31.986685+0000", 2020-06-12T02:37:32.558 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0015393990000000001 2020-06-12T02:37:32.558 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.558 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:32.558 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.558 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.558 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.559 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.9s1 25/21 ECSubWrite(tid=453, reqid=client.4512.0:902, at_version=0'0, trim_to=0'0, roll_forward_to=25'73))", 2020-06-12T02:37:32.559 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:32.059223+0000", 2020-06-12T02:37:32.559 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.329331182, 2020-06-12T02:37:32.559 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0015781669999999999, 2020-06-12T02:37:32.560 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:32.560 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:32.560 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:32.560 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.560 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:32.560 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.059223+0000", 2020-06-12T02:37:32.561 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:32.561 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.561 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.562 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:32.562 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.059223+0000", 2020-06-12T02:37:32.563 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5198999999999998e-05 2020-06-12T02:37:32.563 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.563 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.563 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:32.564 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.059278+0000", 2020-06-12T02:37:32.564 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3984e-05 2020-06-12T02:37:32.564 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.564 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.564 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:32.565 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.059292+0000", 2020-06-12T02:37:32.565 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6288000000000001e-05 2020-06-12T02:37:32.565 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.565 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.565 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:32.565 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.059308+0000", 2020-06-12T02:37:32.566 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1380000000000004e-06 2020-06-12T02:37:32.566 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.566 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.566 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:32.567 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.059315+0000", 2020-06-12T02:37:32.567 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9738999999999999e-05 2020-06-12T02:37:32.567 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.567 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.567 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:32.567 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.059364+0000", 2020-06-12T02:37:32.568 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0752e-05 2020-06-12T02:37:32.568 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.568 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.568 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:32.568 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.059375+0000", 2020-06-12T02:37:32.568 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013801130000000001 2020-06-12T02:37:32.569 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.569 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.571 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:32.571 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.060755+0000", 2020-06-12T02:37:32.571 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5954000000000001e-05 2020-06-12T02:37:32.571 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.572 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.572 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:32.572 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.060801+0000", 2020-06-12T02:37:32.572 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0015781669999999999 2020-06-12T02:37:32.572 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.573 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:32.573 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.573 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.573 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.573 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 25/21 ECSubWrite(tid=65, reqid=client.4512.0:914, at_version=25'41, trim_to=0'0, roll_forward_to=25'39))", 2020-06-12T02:37:32.574 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:32.105659+0000", 2020-06-12T02:37:32.574 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.28289557399999998, 2020-06-12T02:37:32.574 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0030506600000000002, 2020-06-12T02:37:32.574 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:32.574 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:32.575 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:32.575 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.575 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:32.575 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.105659+0000", 2020-06-12T02:37:32.575 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:32.575 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.576 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.576 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:32.576 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.105659+0000", 2020-06-12T02:37:32.576 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013425000000000001 2020-06-12T02:37:32.576 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.577 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.577 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:32.577 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.105793+0000", 2020-06-12T02:37:32.577 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.4958000000000001e-05 2020-06-12T02:37:32.577 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.577 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.578 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:32.578 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.105818+0000", 2020-06-12T02:37:32.578 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0251e-05 2020-06-12T02:37:32.578 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.578 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.579 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:32.579 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.105838+0000", 2020-06-12T02:37:32.583 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3749999999999992e-06 2020-06-12T02:37:32.584 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.584 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.584 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:32.584 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.105847+0000", 2020-06-12T02:37:32.585 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0349999999999998e-05 2020-06-12T02:37:32.585 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.586 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.586 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:32.586 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.105888+0000", 2020-06-12T02:37:32.587 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2739999999999996e-06 2020-06-12T02:37:32.587 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.587 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.587 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:32.587 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.105897+0000", 2020-06-12T02:37:32.588 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027658040000000002 2020-06-12T02:37:32.588 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.588 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.588 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:32.588 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.108663+0000", 2020-06-12T02:37:32.589 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6397999999999999e-05 2020-06-12T02:37:32.589 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.589 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.589 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:32.589 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.108709+0000", 2020-06-12T02:37:32.590 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0030506600000000002 2020-06-12T02:37:32.590 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.590 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:32.590 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.590 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.591 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.591 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 25/21 ECSubWrite(tid=66, reqid=client.4512.0:915, at_version=25'42, trim_to=0'0, roll_forward_to=25'39))", 2020-06-12T02:37:32.591 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:32.110061+0000", 2020-06-12T02:37:32.591 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.27849334100000001, 2020-06-12T02:37:32.591 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0053112929999999999, 2020-06-12T02:37:32.592 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:32.592 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:32.592 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:32.592 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.592 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:32.592 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.110061+0000", 2020-06-12T02:37:32.593 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:32.593 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.593 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.593 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:32.593 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.110061+0000", 2020-06-12T02:37:32.594 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016627800000000001 2020-06-12T02:37:32.594 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.594 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.594 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:32.594 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.110227+0000", 2020-06-12T02:37:32.595 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.774e-05 2020-06-12T02:37:32.595 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.595 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.595 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:32.595 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.110245+0000", 2020-06-12T02:37:32.596 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0908e-05 2020-06-12T02:37:32.596 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.596 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.596 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:32.596 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.110256+0000", 2020-06-12T02:37:32.596 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.464e-06 2020-06-12T02:37:32.597 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.597 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.597 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:32.597 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.110260+0000", 2020-06-12T02:37:32.597 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2984000000000005e-05 2020-06-12T02:37:32.598 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.598 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.598 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:32.598 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.110333+0000", 2020-06-12T02:37:32.598 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1603999999999999e-05 2020-06-12T02:37:32.598 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.599 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.599 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:32.599 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.110345+0000", 2020-06-12T02:37:32.599 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0049692130000000001 2020-06-12T02:37:32.599 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.600 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.600 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:32.600 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.115314+0000", 2020-06-12T02:37:32.600 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8102e-05 2020-06-12T02:37:32.601 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.601 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.601 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:32.601 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.115372+0000", 2020-06-12T02:37:32.601 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0053112929999999999 2020-06-12T02:37:32.602 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.602 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:32.602 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.602 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.602 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.602 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 25/21 ECSubWrite(tid=67, reqid=client.4512.0:916, at_version=25'43, trim_to=0'0, roll_forward_to=25'41))", 2020-06-12T02:37:32.603 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:32.115403+0000", 2020-06-12T02:37:32.603 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.273151492, 2020-06-12T02:37:32.603 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0035170560000000002, 2020-06-12T02:37:32.603 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:32.603 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:32.604 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:32.604 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.604 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:32.604 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.115403+0000", 2020-06-12T02:37:32.605 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:32.605 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.605 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.605 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:32.606 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.115403+0000", 2020-06-12T02:37:32.606 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013603899999999999 2020-06-12T02:37:32.606 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.606 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.606 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:32.607 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.115539+0000", 2020-06-12T02:37:32.607 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3241e-05 2020-06-12T02:37:32.607 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.607 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.607 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:32.607 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.115562+0000", 2020-06-12T02:37:32.608 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6256000000000002e-05 2020-06-12T02:37:32.608 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.608 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.608 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:32.608 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.115578+0000", 2020-06-12T02:37:32.609 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3229999999999998e-06 2020-06-12T02:37:32.609 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.609 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.609 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:32.609 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.115587+0000", 2020-06-12T02:37:32.609 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1430000000000001e-05 2020-06-12T02:37:32.610 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.610 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.610 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:32.610 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.115628+0000", 2020-06-12T02:37:32.610 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4410000000000004e-06 2020-06-12T02:37:32.611 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.611 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.612 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:32.612 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.115635+0000", 2020-06-12T02:37:32.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.003228 2020-06-12T02:37:32.613 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.613 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.613 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:32.613 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.118863+0000", 2020-06-12T02:37:32.613 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6326000000000002e-05 2020-06-12T02:37:32.614 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.614 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.614 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:32.615 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.118920+0000", 2020-06-12T02:37:32.615 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0035170560000000002 2020-06-12T02:37:32.615 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.615 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:32.615 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.616 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.616 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.616 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 25/21 ECSubWrite(tid=68, reqid=client.4512.0:917, at_version=25'44, trim_to=0'0, roll_forward_to=25'42))", 2020-06-12T02:37:32.616 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:32.119674+0000", 2020-06-12T02:37:32.616 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.26887999499999998, 2020-06-12T02:37:32.617 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0021652939999999999, 2020-06-12T02:37:32.617 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:32.617 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:32.617 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:32.617 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.617 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:32.618 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.119674+0000", 2020-06-12T02:37:32.618 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:32.618 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.618 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.618 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:32.619 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.119674+0000", 2020-06-12T02:37:32.619 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5951000000000004e-05 2020-06-12T02:37:32.619 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.619 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.619 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:32.619 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.119760+0000", 2020-06-12T02:37:32.620 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5899000000000001e-05 2020-06-12T02:37:32.620 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.620 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.620 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:32.620 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.119776+0000", 2020-06-12T02:37:32.621 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1999e-05 2020-06-12T02:37:32.621 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.621 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.621 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:32.621 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.119788+0000", 2020-06-12T02:37:32.622 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0780000000000001e-06 2020-06-12T02:37:32.622 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.622 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.622 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:32.622 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.119794+0000", 2020-06-12T02:37:32.623 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1084000000000001e-05 2020-06-12T02:37:32.623 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.623 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.623 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:32.623 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.119845+0000", 2020-06-12T02:37:32.623 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1707e-05 2020-06-12T02:37:32.624 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.624 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.624 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:32.624 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.119857+0000", 2020-06-12T02:37:32.625 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001930677 2020-06-12T02:37:32.625 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.625 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.625 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:32.625 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.121788+0000", 2020-06-12T02:37:32.626 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1898999999999999e-05 2020-06-12T02:37:32.626 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.626 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.626 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:32.626 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.121839+0000", 2020-06-12T02:37:32.627 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0021652939999999999 2020-06-12T02:37:32.627 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.627 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:32.627 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.627 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.627 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.628 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 25/21 ECSubWrite(tid=69, reqid=client.4512.0:918, at_version=25'45, trim_to=0'0, roll_forward_to=25'42))", 2020-06-12T02:37:32.628 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:32.121196+0000", 2020-06-12T02:37:32.628 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.267358545, 2020-06-12T02:37:32.628 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00099058000000000002, 2020-06-12T02:37:32.628 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:32.629 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:32.629 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:32.629 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.629 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:32.629 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.121196+0000", 2020-06-12T02:37:32.629 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:32.630 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.630 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.630 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:32.630 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.121196+0000", 2020-06-12T02:37:32.630 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7006999999999998e-05 2020-06-12T02:37:32.631 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.631 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.631 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:32.631 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.121253+0000", 2020-06-12T02:37:32.631 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8321999999999998e-05 2020-06-12T02:37:32.631 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.632 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.632 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:32.632 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.121271+0000", 2020-06-12T02:37:32.632 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9755e-05 2020-06-12T02:37:32.632 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.633 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.633 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:32.633 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.121291+0000", 2020-06-12T02:37:32.633 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5910000000000001e-06 2020-06-12T02:37:32.633 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.633 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.634 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:32.634 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.121297+0000", 2020-06-12T02:37:32.634 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.462e-05 2020-06-12T02:37:32.634 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.634 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.635 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:32.635 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.121372+0000", 2020-06-12T02:37:32.635 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9026e-05 2020-06-12T02:37:32.635 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.636 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.636 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:32.636 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.121391+0000", 2020-06-12T02:37:32.636 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00075694900000000003 2020-06-12T02:37:32.636 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.636 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.637 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:32.637 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.122148+0000", 2020-06-12T02:37:32.637 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8309999999999997e-05 2020-06-12T02:37:32.637 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.637 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.638 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:32.638 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.122186+0000", 2020-06-12T02:37:32.638 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00099058000000000002 2020-06-12T02:37:32.638 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.638 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:32.638 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.639 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.639 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.639 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 25/21 ECSubWrite(tid=70, reqid=client.4512.0:918, at_version=0'0, trim_to=0'0, roll_forward_to=25'45))", 2020-06-12T02:37:32.639 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:32.123178+0000", 2020-06-12T02:37:32.639 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.26537629000000001, 2020-06-12T02:37:32.640 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014436830000000001, 2020-06-12T02:37:32.640 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:32.640 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:32.640 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:32.640 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.640 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:32.641 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.123178+0000", 2020-06-12T02:37:32.641 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:32.641 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.641 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.641 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:32.642 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.123178+0000", 2020-06-12T02:37:32.642 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5253000000000003e-05 2020-06-12T02:37:32.642 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.642 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.642 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:32.642 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.123213+0000", 2020-06-12T02:37:32.643 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4880000000000007e-06 2020-06-12T02:37:32.643 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.643 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.643 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:32.643 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.123222+0000", 2020-06-12T02:37:32.644 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9789999999999999e-06 2020-06-12T02:37:32.644 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.644 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.644 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:32.644 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.123231+0000", 2020-06-12T02:37:32.644 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0509999999999996e-06 2020-06-12T02:37:32.645 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.645 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.645 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:32.646 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.123236+0000", 2020-06-12T02:37:32.646 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4212000000000003e-05 2020-06-12T02:37:32.646 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.646 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.646 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:32.647 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.123280+0000", 2020-06-12T02:37:32.647 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3145e-05 2020-06-12T02:37:32.647 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.647 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.647 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:32.648 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.123293+0000", 2020-06-12T02:37:32.648 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012971300000000001 2020-06-12T02:37:32.648 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.648 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.648 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:32.649 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.124590+0000", 2020-06-12T02:37:32.649 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.1424999999999998e-05 2020-06-12T02:37:32.649 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.649 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.649 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:32.649 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.124622+0000", 2020-06-12T02:37:32.650 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014436830000000001 2020-06-12T02:37:32.650 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.650 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:32.650 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.650 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.651 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:32.651 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:37:32.666 INFO:tasks.rados.rados.0.smithi107.stdout:450: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:32.666 INFO:tasks.rados.rados.0.smithi107.stdout:452: done (1 left) 2020-06-12T02:37:32.666 INFO:tasks.rados.rados.0.smithi107.stdout:453: delete oid 14 current snap is 0 2020-06-12T02:37:32.667 INFO:tasks.rados.rados.0.smithi107.stdout:450: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:32.667 INFO:tasks.rados.rados.0.smithi107.stdout:450: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:32.667 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 34 (ObjNum 449 snap 0 seq_num 449) dirty exists 2020-06-12T02:37:32.667 INFO:tasks.rados.rados.0.smithi107.stdout:450: left oid 20 (ObjNum 449 snap 0 seq_num 449) 2020-06-12T02:37:32.667 INFO:tasks.rados.rados.0.smithi107.stdout:450: done (1 left) 2020-06-12T02:37:32.668 INFO:tasks.rados.rados.0.smithi107.stdout:453: done (0 left) 2020-06-12T02:37:32.668 INFO:tasks.rados.rados.0.smithi107.stdout:454: delete oid 18 current snap is 0 2020-06-12T02:37:32.668 INFO:tasks.rados.rados.0.smithi107.stdout:454: done (0 left) 2020-06-12T02:37:32.668 INFO:tasks.rados.rados.0.smithi107.stdout:455: delete oid 7 current snap is 0 2020-06-12T02:37:32.668 INFO:tasks.rados.rados.0.smithi107.stdout:455: done (0 left) 2020-06-12T02:37:32.669 INFO:tasks.rados.rados.0.smithi107.stdout:456: delete oid 9 current snap is 0 2020-06-12T02:37:32.669 INFO:tasks.rados.rados.0.smithi107.stdout:456: done (0 left) 2020-06-12T02:37:32.669 INFO:tasks.rados.rados.0.smithi107.stdout:457: delete oid 20 current snap is 0 2020-06-12T02:37:32.669 INFO:tasks.rados.rados.0.smithi107.stdout:457: done (0 left) 2020-06-12T02:37:32.669 INFO:tasks.rados.rados.0.smithi107.stdout:458: delete oid 16 current snap is 0 2020-06-12T02:37:32.669 INFO:tasks.rados.rados.0.smithi107.stdout:458: done (0 left) 2020-06-12T02:37:32.670 INFO:tasks.rados.rados.0.smithi107.stdout:459: delete oid 6 current snap is 0 2020-06-12T02:37:32.670 INFO:tasks.rados.rados.0.smithi107.stdout:459: done (0 left) 2020-06-12T02:37:32.670 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:37:32.670 INFO:tasks.rados.rados.0.smithi107.stdout:460: seq_num 459 ranges {0=491520} 2020-06-12T02:37:32.670 INFO:tasks.rados.rados.0.smithi107.stdout:460: writing smithi10729407-10 from 0 to 491520 tid 1 2020-06-12T02:37:32.671 INFO:tasks.rados.rados.0.smithi107.stdout:461: delete oid 15 current snap is 0 2020-06-12T02:37:32.671 INFO:tasks.rados.rados.0.smithi107.stdout:461: done (1 left) 2020-06-12T02:37:32.671 INFO:tasks.rados.rados.0.smithi107.stdout:462: delete oid 7 current snap is 0 2020-06-12T02:37:32.671 INFO:tasks.rados.rados.0.smithi107.stdout:462: done (1 left) 2020-06-12T02:37:32.671 INFO:tasks.rados.rados.0.smithi107.stdout:463: delete oid 6 current snap is 0 2020-06-12T02:37:32.671 INFO:tasks.rados.rados.0.smithi107.stdout:463: done (1 left) 2020-06-12T02:37:32.672 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:37:32.672 INFO:tasks.rados.rados.0.smithi107.stdout:464: seq_num 463 ranges {0=557056} 2020-06-12T02:37:32.672 INFO:tasks.rados.rados.0.smithi107.stdout:464: writing smithi10729407-3 from 0 to 557056 tid 1 2020-06-12T02:37:32.672 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:37:32.672 INFO:tasks.rados.rados.0.smithi107.stdout:465: seq_num 464 ranges {0=573440} 2020-06-12T02:37:32.673 INFO:tasks.rados.rados.0.smithi107.stdout:465: writing smithi10729407-15 from 0 to 573440 tid 1 2020-06-12T02:37:32.673 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:37:32.673 INFO:tasks.rados.rados.0.smithi107.stdout:466: seq_num 465 ranges {0=598016} 2020-06-12T02:37:32.673 INFO:tasks.rados.rados.0.smithi107.stdout:466: writing smithi10729407-13 from 0 to 598016 tid 1 2020-06-12T02:37:32.673 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:37:32.673 INFO:tasks.rados.rados.0.smithi107.stdout:467: seq_num 466 ranges {0=614400} 2020-06-12T02:37:32.674 INFO:tasks.rados.rados.0.smithi107.stdout:467: writing smithi10729407-14 from 0 to 614400 tid 1 2020-06-12T02:37:32.674 INFO:tasks.rados.rados.0.smithi107.stdout:468: delete oid 16 current snap is 0 2020-06-12T02:37:32.674 INFO:tasks.rados.rados.0.smithi107.stdout:460: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:32.674 INFO:tasks.rados.rados.0.smithi107.stdout:460: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:32.674 INFO:tasks.rados.rados.0.smithi107.stdout:460: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:32.675 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 48 (ObjNum 459 snap 0 seq_num 459) dirty exists 2020-06-12T02:37:32.675 INFO:tasks.rados.rados.0.smithi107.stdout:460: left oid 10 (ObjNum 459 snap 0 seq_num 459) 2020-06-12T02:37:32.675 INFO:tasks.rados.rados.0.smithi107.stdout:464: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:32.675 INFO:tasks.rados.rados.0.smithi107.stdout:464: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:32.675 INFO:tasks.rados.rados.0.smithi107.stdout:464: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:32.676 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 76 (ObjNum 463 snap 0 seq_num 463) dirty exists 2020-06-12T02:37:32.676 INFO:tasks.rados.rados.0.smithi107.stdout:464: left oid 3 (ObjNum 463 snap 0 seq_num 463) 2020-06-12T02:37:32.676 INFO:tasks.rados.rados.0.smithi107.stdout:465: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:32.676 INFO:tasks.rados.rados.0.smithi107.stdout:465: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:32.677 INFO:tasks.rados.rados.0.smithi107.stdout:466: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:32.677 INFO:tasks.rados.rados.0.smithi107.stdout:465: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:32.677 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 100 (ObjNum 464 snap 0 seq_num 464) dirty exists 2020-06-12T02:37:32.677 INFO:tasks.rados.rados.0.smithi107.stdout:465: left oid 15 (ObjNum 464 snap 0 seq_num 464) 2020-06-12T02:37:32.677 INFO:tasks.rados.rados.0.smithi107.stdout:466: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:32.678 INFO:tasks.rados.rados.0.smithi107.stdout:460: done (5 left) 2020-06-12T02:37:32.678 INFO:tasks.rados.rados.0.smithi107.stdout:464: done (4 left) 2020-06-12T02:37:32.678 INFO:tasks.rados.rados.0.smithi107.stdout:465: done (3 left) 2020-06-12T02:37:32.678 INFO:tasks.rados.rados.0.smithi107.stdout:468: done (2 left) 2020-06-12T02:37:32.678 INFO:tasks.rados.rados.0.smithi107.stdout:469: delete oid 11 current snap is 0 2020-06-12T02:37:32.679 INFO:tasks.rados.rados.0.smithi107.stdout:466: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:32.679 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 90 (ObjNum 465 snap 0 seq_num 465) dirty exists 2020-06-12T02:37:32.679 INFO:tasks.rados.rados.0.smithi107.stdout:466: left oid 13 (ObjNum 465 snap 0 seq_num 465) 2020-06-12T02:37:32.679 INFO:tasks.rados.rados.0.smithi107.stdout:467: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:32.679 INFO:tasks.rados.rados.0.smithi107.stdout:466: done (2 left) 2020-06-12T02:37:32.679 INFO:tasks.rados.rados.0.smithi107.stdout:469: done (1 left) 2020-06-12T02:37:32.680 INFO:tasks.rados.rados.0.smithi107.stdout:470: delete oid 8 current snap is 0 2020-06-12T02:37:32.680 INFO:tasks.rados.rados.0.smithi107.stdout:467: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:32.680 INFO:tasks.rados.rados.0.smithi107.stdout:470: done (1 left) 2020-06-12T02:37:32.680 INFO:tasks.rados.rados.0.smithi107.stdout:471: delete oid 12 current snap is 0 2020-06-12T02:37:32.680 INFO:tasks.rados.rados.0.smithi107.stdout:467: finishing write tid 4 to smithi10729407-14 2020-06-12T02:37:32.681 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 58 (ObjNum 466 snap 0 seq_num 466) dirty exists 2020-06-12T02:37:32.681 INFO:tasks.rados.rados.0.smithi107.stdout:467: left oid 14 (ObjNum 466 snap 0 seq_num 466) 2020-06-12T02:37:32.681 INFO:tasks.rados.rados.0.smithi107.stdout:467: done (1 left) 2020-06-12T02:37:32.681 INFO:tasks.rados.rados.0.smithi107.stdout:471: done (0 left) 2020-06-12T02:37:32.681 INFO:tasks.rados.rados.0.smithi107.stdout:472: delete oid 5 current snap is 0 2020-06-12T02:37:32.681 INFO:tasks.rados.rados.0.smithi107.stdout:472: done (0 left) 2020-06-12T02:37:32.682 INFO:tasks.rados.rados.0.smithi107.stdout:473: delete oid 18 current snap is 0 2020-06-12T02:37:32.682 INFO:tasks.rados.rados.0.smithi107.stdout:473: done (0 left) 2020-06-12T02:37:32.682 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:37:32.682 INFO:tasks.rados.rados.0.smithi107.stdout:474: seq_num 473 ranges {0=729088} 2020-06-12T02:37:32.682 INFO:tasks.rados.rados.0.smithi107.stdout:474: writing smithi10729407-4 from 0 to 729088 tid 1 2020-06-12T02:37:32.683 INFO:tasks.rados.rados.0.smithi107.stdout:475: delete oid 2 current snap is 0 2020-06-12T02:37:32.683 INFO:tasks.rados.rados.0.smithi107.stdout:475: done (1 left) 2020-06-12T02:37:32.683 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:37:32.683 INFO:tasks.rados.rados.0.smithi107.stdout:476: seq_num 475 ranges {0=761856} 2020-06-12T02:37:32.683 INFO:tasks.rados.rados.0.smithi107.stdout:476: writing smithi10729407-18 from 0 to 761856 tid 1 2020-06-12T02:37:32.683 INFO:tasks.rados.rados.0.smithi107.stdout:477: delete oid 16 current snap is 0 2020-06-12T02:37:32.684 INFO:tasks.rados.rados.0.smithi107.stdout:474: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:32.684 INFO:tasks.rados.rados.0.smithi107.stdout:474: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:32.684 INFO:tasks.rados.rados.0.smithi107.stdout:474: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:32.684 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 90 (ObjNum 473 snap 0 seq_num 473) dirty exists 2020-06-12T02:37:32.684 INFO:tasks.rados.rados.0.smithi107.stdout:474: left oid 4 (ObjNum 473 snap 0 seq_num 473) 2020-06-12T02:37:32.684 INFO:tasks.rados.rados.0.smithi107.stdout:476: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:32.685 INFO:tasks.rados.rados.0.smithi107.stdout:474: done (2 left) 2020-06-12T02:37:32.685 INFO:tasks.rados.rados.0.smithi107.stdout:477: done (1 left) 2020-06-12T02:37:32.685 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:37:32.685 INFO:tasks.rados.rados.0.smithi107.stdout:478: seq_num 477 ranges {0=794624} 2020-06-12T02:37:32.685 INFO:tasks.rados.rados.0.smithi107.stdout:478: writing smithi10729407-16 from 0 to 794624 tid 1 2020-06-12T02:37:32.686 INFO:tasks.rados.rados.0.smithi107.stdout:479: delete oid 1 current snap is 0 2020-06-12T02:37:32.686 INFO:tasks.rados.rados.0.smithi107.stdout:476: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:32.686 INFO:tasks.rados.rados.0.smithi107.stdout:476: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:32.686 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 84 (ObjNum 475 snap 0 seq_num 475) dirty exists 2020-06-12T02:37:32.687 INFO:tasks.rados.rados.0.smithi107.stdout:476: left oid 18 (ObjNum 475 snap 0 seq_num 475) 2020-06-12T02:37:32.687 INFO:tasks.rados.rados.0.smithi107.stdout:476: done (2 left) 2020-06-12T02:37:32.687 INFO:tasks.rados.rados.0.smithi107.stdout:479: done (1 left) 2020-06-12T02:37:32.687 INFO:tasks.rados.rados.0.smithi107.stdout:480: delete oid 11 current snap is 0 2020-06-12T02:37:32.687 INFO:tasks.rados.rados.0.smithi107.stdout:480: done (1 left) 2020-06-12T02:37:32.687 INFO:tasks.rados.rados.0.smithi107.stdout:481: delete oid 17 current snap is 0 2020-06-12T02:37:32.688 INFO:tasks.rados.rados.0.smithi107.stdout:478: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:32.688 INFO:tasks.rados.rados.0.smithi107.stdout:481: done (1 left) 2020-06-12T02:37:32.688 INFO:tasks.rados.rados.0.smithi107.stdout:482: delete oid 1 current snap is 0 2020-06-12T02:37:32.688 INFO:tasks.rados.rados.0.smithi107.stdout:478: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:32.688 INFO:tasks.rados.rados.0.smithi107.stdout:482: done (1 left) 2020-06-12T02:37:32.689 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:37:32.689 INFO:tasks.rados.rados.0.smithi107.stdout:483: seq_num 482 ranges {761856=475136,1236992=401408} 2020-06-12T02:37:32.689 INFO:tasks.rados.rados.0.smithi107.stdout:483: writing smithi10729407-18 from 761856 to 1236992 tid 1 2020-06-12T02:37:32.689 INFO:tasks.rados.rados.0.smithi107.stdout:483: writing smithi10729407-18 from 1236992 to 1638400 tid 2 2020-06-12T02:37:32.689 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:37:32.689 INFO:tasks.rados.rados.0.smithi107.stdout:484: seq_num 483 ranges {729088=491520,1220608=401408} 2020-06-12T02:37:32.690 INFO:tasks.rados.rados.0.smithi107.stdout:484: writing smithi10729407-4 from 729088 to 1220608 tid 1 2020-06-12T02:37:32.690 INFO:tasks.rados.rados.0.smithi107.stdout:484: writing smithi10729407-4 from 1220608 to 1622016 tid 2 2020-06-12T02:37:32.690 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:37:32.690 INFO:tasks.rados.rados.0.smithi107.stdout:485: seq_num 484 ranges {0=507904,507904=401408} 2020-06-12T02:37:32.690 INFO:tasks.rados.rados.0.smithi107.stdout:485: writing smithi10729407-7 from 0 to 507904 tid 1 2020-06-12T02:37:32.691 INFO:tasks.rados.rados.0.smithi107.stdout:485: writing smithi10729407-7 from 507904 to 909312 tid 2 2020-06-12T02:37:32.691 INFO:tasks.rados.rados.0.smithi107.stdout:486: delete oid 20 current snap is 0 2020-06-12T02:37:32.691 INFO:tasks.rados.rados.0.smithi107.stdout:478: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:32.691 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 80 (ObjNum 477 snap 0 seq_num 477) dirty exists 2020-06-12T02:37:32.691 INFO:tasks.rados.rados.0.smithi107.stdout:478: left oid 16 (ObjNum 477 snap 0 seq_num 477) 2020-06-12T02:37:32.691 INFO:tasks.rados.rados.0.smithi107.stdout:483: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:32.692 INFO:tasks.rados.rados.0.smithi107.stdout:483: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:32.692 INFO:tasks.rados.rados.0.smithi107.stdout:483: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:32.692 INFO:tasks.rados.rados.0.smithi107.stdout:483: finishing write tid 5 to smithi10729407-18 2020-06-12T02:37:32.692 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 87 (ObjNum 482 snap 0 seq_num 482) dirty exists 2020-06-12T02:37:32.692 INFO:tasks.rados.rados.0.smithi107.stdout:483: left oid 18 (ObjNum 482 snap 0 seq_num 482) 2020-06-12T02:37:32.692 INFO:tasks.rados.rados.0.smithi107.stdout:484: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:32.693 INFO:tasks.rados.rados.0.smithi107.stdout:484: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:32.693 INFO:tasks.rados.rados.0.smithi107.stdout:484: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:32.693 INFO:tasks.rados.rados.0.smithi107.stdout:478: done (4 left) 2020-06-12T02:37:32.693 INFO:tasks.rados.rados.0.smithi107.stdout:483: done (3 left) 2020-06-12T02:37:32.693 INFO:tasks.rados.rados.0.smithi107.stdout:486: done (2 left) 2020-06-12T02:37:32.694 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:32.694 INFO:tasks.rados.rados.0.smithi107.stdout:487: seq_num 486 ranges {0=548864,548864=401408} 2020-06-12T02:37:32.694 INFO:tasks.rados.rados.0.smithi107.stdout:487: writing smithi10729407-2 from 0 to 548864 tid 1 2020-06-12T02:37:32.694 INFO:tasks.rados.rados.0.smithi107.stdout:487: writing smithi10729407-2 from 548864 to 950272 tid 2 2020-06-12T02:37:32.694 INFO:tasks.rados.rados.0.smithi107.stdout:488: delete oid 14 current snap is 0 2020-06-12T02:37:32.694 INFO:tasks.rados.rados.0.smithi107.stdout:485: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:32.695 INFO:tasks.rados.rados.0.smithi107.stdout:485: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:32.695 INFO:tasks.rados.rados.0.smithi107.stdout:484: finishing write tid 5 to smithi10729407-4 2020-06-12T02:37:32.695 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 94 (ObjNum 483 snap 0 seq_num 483) dirty exists 2020-06-12T02:37:32.695 INFO:tasks.rados.rados.0.smithi107.stdout:484: left oid 4 (ObjNum 483 snap 0 seq_num 483) 2020-06-12T02:37:32.695 INFO:tasks.rados.rados.0.smithi107.stdout:485: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:32.696 INFO:tasks.rados.rados.0.smithi107.stdout:485: finishing write tid 5 to smithi10729407-7 2020-06-12T02:37:32.696 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 93 (ObjNum 484 snap 0 seq_num 484) dirty exists 2020-06-12T02:37:32.696 INFO:tasks.rados.rados.0.smithi107.stdout:485: left oid 7 (ObjNum 484 snap 0 seq_num 484) 2020-06-12T02:37:32.696 INFO:tasks.rados.rados.0.smithi107.stdout:487: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:32.696 INFO:tasks.rados.rados.0.smithi107.stdout:484: done (3 left) 2020-06-12T02:37:32.697 INFO:tasks.rados.rados.0.smithi107.stdout:485: done (2 left) 2020-06-12T02:37:32.697 INFO:tasks.rados.rados.0.smithi107.stdout:488: done (1 left) 2020-06-12T02:37:32.697 INFO:tasks.rados.rados.0.smithi107.stdout:489: delete oid 4 current snap is 0 2020-06-12T02:37:32.697 INFO:tasks.rados.rados.0.smithi107.stdout:487: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:32.697 INFO:tasks.rados.rados.0.smithi107.stdout:489: done (1 left) 2020-06-12T02:37:32.698 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:37:32.698 INFO:tasks.rados.rados.0.smithi107.stdout:490: seq_num 489 ranges {909312=598016,1507328=401408} 2020-06-12T02:37:32.698 INFO:tasks.rados.rados.0.smithi107.stdout:490: writing smithi10729407-7 from 909312 to 1507328 tid 1 2020-06-12T02:37:32.698 INFO:tasks.rados.rados.0.smithi107.stdout:490: writing smithi10729407-7 from 1507328 to 1908736 tid 2 2020-06-12T02:37:32.698 INFO:tasks.rados.rados.0.smithi107.stdout:491: delete oid 8 current snap is 0 2020-06-12T02:37:32.699 INFO:tasks.rados.rados.0.smithi107.stdout:487: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:32.699 INFO:tasks.rados.rados.0.smithi107.stdout:487: finishing write tid 5 to smithi10729407-2 2020-06-12T02:37:32.699 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 56 (ObjNum 486 snap 0 seq_num 486) dirty exists 2020-06-12T02:37:32.699 INFO:tasks.rados.rados.0.smithi107.stdout:487: left oid 2 (ObjNum 486 snap 0 seq_num 486) 2020-06-12T02:37:32.700 INFO:tasks.rados.rados.0.smithi107.stdout:487: done (2 left) 2020-06-12T02:37:32.700 INFO:tasks.rados.rados.0.smithi107.stdout:491: done (1 left) 2020-06-12T02:37:32.700 INFO:tasks.rados.rados.0.smithi107.stdout:492: delete oid 2 current snap is 0 2020-06-12T02:37:32.700 INFO:tasks.rados.rados.0.smithi107.stdout:490: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:32.700 INFO:tasks.rados.rados.0.smithi107.stdout:492: done (1 left) 2020-06-12T02:37:32.700 INFO:tasks.rados.rados.0.smithi107.stdout:493: delete oid 20 current snap is 0 2020-06-12T02:37:32.701 INFO:tasks.rados.rados.0.smithi107.stdout:493: done (1 left) 2020-06-12T02:37:32.701 INFO:tasks.rados.rados.0.smithi107.stdout:494: delete oid 13 current snap is 0 2020-06-12T02:37:32.701 INFO:tasks.rados.rados.0.smithi107.stdout:490: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:32.701 INFO:tasks.rados.rados.0.smithi107.stdout:490: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:32.701 INFO:tasks.rados.rados.0.smithi107.stdout:494: done (1 left) 2020-06-12T02:37:32.702 INFO:tasks.rados.rados.0.smithi107.stdout:495: delete oid 1 current snap is 0 2020-06-12T02:37:32.702 INFO:tasks.rados.rados.0.smithi107.stdout:490: finishing write tid 5 to smithi10729407-7 2020-06-12T02:37:32.702 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 96 (ObjNum 489 snap 0 seq_num 489) dirty exists 2020-06-12T02:37:32.702 INFO:tasks.rados.rados.0.smithi107.stdout:490: left oid 7 (ObjNum 489 snap 0 seq_num 489) 2020-06-12T02:37:32.702 INFO:tasks.rados.rados.0.smithi107.stdout:490: done (1 left) 2020-06-12T02:37:32.702 INFO:tasks.rados.rados.0.smithi107.stdout:495: done (0 left) 2020-06-12T02:37:32.703 INFO:tasks.rados.rados.0.smithi107.stdout:496: delete oid 19 current snap is 0 2020-06-12T02:37:32.703 INFO:tasks.rados.rados.0.smithi107.stdout:496: done (0 left) 2020-06-12T02:37:32.703 INFO:tasks.rados.rados.0.smithi107.stdout:497: delete oid 7 current snap is 0 2020-06-12T02:37:32.703 INFO:tasks.rados.rados.0.smithi107.stdout:497: done (0 left) 2020-06-12T02:37:32.703 INFO:tasks.rados.rados.0.smithi107.stdout:498: delete oid 7 current snap is 0 2020-06-12T02:37:32.707 INFO:tasks.rados.rados.0.smithi107.stdout:498: done (0 left) 2020-06-12T02:37:32.707 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:37:32.707 INFO:tasks.rados.rados.0.smithi107.stdout:499: seq_num 498 ranges {0=745472,745472=401408} 2020-06-12T02:37:32.707 INFO:tasks.rados.rados.0.smithi107.stdout:499: writing smithi10729407-4 from 0 to 745472 tid 1 2020-06-12T02:37:32.708 INFO:tasks.rados.rados.0.smithi107.stdout:499: writing smithi10729407-4 from 745472 to 1146880 tid 2 2020-06-12T02:37:32.708 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:37:32.708 INFO:tasks.rados.rados.0.smithi107.stdout:500: seq_num 499 ranges {0=761856,761856=401408} 2020-06-12T02:37:32.708 INFO:tasks.rados.rados.0.smithi107.stdout:500: writing smithi10729407-5 from 0 to 761856 tid 1 2020-06-12T02:37:32.708 INFO:tasks.rados.rados.0.smithi107.stdout:500: writing smithi10729407-5 from 761856 to 1163264 tid 2 2020-06-12T02:37:32.709 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:37:32.709 INFO:tasks.rados.rados.0.smithi107.stdout:501: seq_num 500 ranges {491520=778240,1269760=401408} 2020-06-12T02:37:32.709 INFO:tasks.rados.rados.0.smithi107.stdout:501: writing smithi10729407-10 from 491520 to 1269760 tid 1 2020-06-12T02:37:32.709 INFO:tasks.rados.rados.0.smithi107.stdout:501: writing smithi10729407-10 from 1269760 to 1671168 tid 2 2020-06-12T02:37:32.709 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:37:32.709 INFO:tasks.rados.rados.0.smithi107.stdout:502: seq_num 501 ranges {0=794624,794624=401408} 2020-06-12T02:37:32.710 INFO:tasks.rados.rados.0.smithi107.stdout:502: writing smithi10729407-14 from 0 to 794624 tid 1 2020-06-12T02:37:32.710 INFO:tasks.rados.rados.0.smithi107.stdout:502: writing smithi10729407-14 from 794624 to 1196032 tid 2 2020-06-12T02:37:32.710 INFO:tasks.rados.rados.0.smithi107.stdout:503: delete oid 8 current snap is 0 2020-06-12T02:37:32.710 INFO:tasks.rados.rados.0.smithi107.stdout:499: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:32.710 INFO:tasks.rados.rados.0.smithi107.stdout:499: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:32.711 INFO:tasks.rados.rados.0.smithi107.stdout:499: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:32.711 INFO:tasks.rados.rados.0.smithi107.stdout:499: finishing write tid 5 to smithi10729407-4 2020-06-12T02:37:32.711 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 98 (ObjNum 498 snap 0 seq_num 498) dirty exists 2020-06-12T02:37:32.711 INFO:tasks.rados.rados.0.smithi107.stdout:499: left oid 4 (ObjNum 498 snap 0 seq_num 498) 2020-06-12T02:37:32.711 INFO:tasks.rados.rados.0.smithi107.stdout:500: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:32.711 INFO:tasks.rados.rados.0.smithi107.stdout:500: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:32.712 INFO:tasks.rados.rados.0.smithi107.stdout:500: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:32.712 INFO:tasks.rados.rados.0.smithi107.stdout:501: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:32.712 INFO:tasks.rados.rados.0.smithi107.stdout:500: finishing write tid 5 to smithi10729407-5 2020-06-12T02:37:32.712 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 53 (ObjNum 499 snap 0 seq_num 499) dirty exists 2020-06-12T02:37:32.712 INFO:tasks.rados.rados.0.smithi107.stdout:500: left oid 5 (ObjNum 499 snap 0 seq_num 499) 2020-06-12T02:37:32.712 INFO:tasks.rados.rados.0.smithi107.stdout:501: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:32.713 INFO:tasks.rados.rados.0.smithi107.stdout:499: done (4 left) 2020-06-12T02:37:32.713 INFO:tasks.rados.rados.0.smithi107.stdout:500: done (3 left) 2020-06-12T02:37:32.713 INFO:tasks.rados.rados.0.smithi107.stdout:503: done (2 left) 2020-06-12T02:37:32.713 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:37:32.713 INFO:tasks.rados.rados.0.smithi107.stdout:504: seq_num 503 ranges {0=425984,425984=704512,1130496=98304} 2020-06-12T02:37:32.714 INFO:tasks.rados.rados.0.smithi107.stdout:504: writing smithi10729407-9 from 0 to 425984 tid 1 2020-06-12T02:37:32.714 INFO:tasks.rados.rados.0.smithi107.stdout:504: writing smithi10729407-9 from 425984 to 1130496 tid 2 2020-06-12T02:37:32.714 INFO:tasks.rados.rados.0.smithi107.stdout:504: writing smithi10729407-9 from 1130496 to 1228800 tid 3 2020-06-12T02:37:32.714 INFO:tasks.rados.rados.0.smithi107.stdout:505: delete oid 12 current snap is 0 2020-06-12T02:37:32.714 INFO:tasks.rados.rados.0.smithi107.stdout:501: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:32.714 INFO:tasks.rados.rados.0.smithi107.stdout:501: finishing write tid 5 to smithi10729407-10 2020-06-12T02:37:32.715 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 51 (ObjNum 500 snap 0 seq_num 500) dirty exists 2020-06-12T02:37:32.715 INFO:tasks.rados.rados.0.smithi107.stdout:501: left oid 10 (ObjNum 500 snap 0 seq_num 500) 2020-06-12T02:37:32.715 INFO:tasks.rados.rados.0.smithi107.stdout:502: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:32.715 INFO:tasks.rados.rados.0.smithi107.stdout:502: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:32.715 INFO:tasks.rados.rados.0.smithi107.stdout:502: finishing write tid 4 to smithi10729407-14 2020-06-12T02:37:32.715 INFO:tasks.rados.rados.0.smithi107.stdout:502: finishing write tid 5 to smithi10729407-14 2020-06-12T02:37:32.716 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 63 (ObjNum 501 snap 0 seq_num 501) dirty exists 2020-06-12T02:37:32.716 INFO:tasks.rados.rados.0.smithi107.stdout:502: left oid 14 (ObjNum 501 snap 0 seq_num 501) 2020-06-12T02:37:32.716 INFO:tasks.rados.rados.0.smithi107.stdout:501: done (3 left) 2020-06-12T02:37:32.716 INFO:tasks.rados.rados.0.smithi107.stdout:502: done (2 left) 2020-06-12T02:37:32.716 INFO:tasks.rados.rados.0.smithi107.stdout:505: done (1 left) 2020-06-12T02:37:32.717 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:37:32.717 INFO:tasks.rados.rados.0.smithi107.stdout:506: seq_num 505 ranges {1146880=466944,1613824=475136,2088960=327680} 2020-06-12T02:37:32.717 INFO:tasks.rados.rados.0.smithi107.stdout:506: writing smithi10729407-4 from 1146880 to 1613824 tid 1 2020-06-12T02:37:32.717 INFO:tasks.rados.rados.0.smithi107.stdout:506: writing smithi10729407-4 from 1613824 to 2088960 tid 2 2020-06-12T02:37:32.718 INFO:tasks.rados.rados.0.smithi107.stdout:506: writing smithi10729407-4 from 2088960 to 2416640 tid 3 2020-06-12T02:37:32.718 INFO:tasks.rados.rados.0.smithi107.stdout:507: delete oid 16 current snap is 0 2020-06-12T02:37:32.718 INFO:tasks.rados.rados.0.smithi107.stdout:504: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:32.718 INFO:tasks.rados.rados.0.smithi107.stdout:504: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:32.718 INFO:tasks.rados.rados.0.smithi107.stdout:504: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:32.718 INFO:tasks.rados.rados.0.smithi107.stdout:504: finishing write tid 5 to smithi10729407-9 2020-06-12T02:37:32.719 INFO:tasks.rados.rados.0.smithi107.stdout:504: finishing write tid 6 to smithi10729407-9 2020-06-12T02:37:32.719 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 91 (ObjNum 503 snap 0 seq_num 503) dirty exists 2020-06-12T02:37:32.719 INFO:tasks.rados.rados.0.smithi107.stdout:504: left oid 9 (ObjNum 503 snap 0 seq_num 503) 2020-06-12T02:37:32.719 INFO:tasks.rados.rados.0.smithi107.stdout:506: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:32.719 INFO:tasks.rados.rados.0.smithi107.stdout:504: done (2 left) 2020-06-12T02:37:32.720 INFO:tasks.rados.rados.0.smithi107.stdout:507: done (1 left) 2020-06-12T02:37:32.720 INFO:tasks.rados.rados.0.smithi107.stdout:508: delete oid 3 current snap is 0 2020-06-12T02:37:32.720 INFO:tasks.rados.rados.0.smithi107.stdout:508: done (1 left) 2020-06-12T02:37:32.720 INFO:tasks.rados.rados.0.smithi107.stdout:509: delete oid 16 current snap is 0 2020-06-12T02:37:32.720 INFO:tasks.rados.rados.0.smithi107.stdout:506: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:32.720 INFO:tasks.rados.rados.0.smithi107.stdout:509: done (1 left) 2020-06-12T02:37:32.721 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:37:32.721 INFO:tasks.rados.rados.0.smithi107.stdout:510: seq_num 509 ranges {0=532480,532480=409600,942080=393216} 2020-06-12T02:37:32.721 INFO:tasks.rados.rados.0.smithi107.stdout:510: writing smithi10729407-11 from 0 to 532480 tid 1 2020-06-12T02:37:32.721 INFO:tasks.rados.rados.0.smithi107.stdout:510: writing smithi10729407-11 from 532480 to 942080 tid 2 2020-06-12T02:37:32.721 INFO:tasks.rados.rados.0.smithi107.stdout:510: writing smithi10729407-11 from 942080 to 1335296 tid 3 2020-06-12T02:37:32.722 INFO:tasks.rados.rados.0.smithi107.stdout:511: delete oid 18 current snap is 0 2020-06-12T02:37:32.722 INFO:tasks.rados.rados.0.smithi107.stdout:506: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:32.722 INFO:tasks.rados.rados.0.smithi107.stdout:506: finishing write tid 5 to smithi10729407-4 2020-06-12T02:37:32.722 INFO:tasks.rados.rados.0.smithi107.stdout:506: finishing write tid 6 to smithi10729407-4 2020-06-12T02:37:32.722 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 102 (ObjNum 505 snap 0 seq_num 505) dirty exists 2020-06-12T02:37:32.722 INFO:tasks.rados.rados.0.smithi107.stdout:506: left oid 4 (ObjNum 505 snap 0 seq_num 505) 2020-06-12T02:37:32.723 INFO:tasks.rados.rados.0.smithi107.stdout:510: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:32.723 INFO:tasks.rados.rados.0.smithi107.stdout:510: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:32.723 INFO:tasks.rados.rados.0.smithi107.stdout:506: done (2 left) 2020-06-12T02:37:32.723 INFO:tasks.rados.rados.0.smithi107.stdout:511: done (1 left) 2020-06-12T02:37:32.723 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:37:32.723 INFO:tasks.rados.rados.0.smithi107.stdout:512: seq_num 511 ranges {2416640=565248,2981888=630784,3612672=172032} 2020-06-12T02:37:32.724 INFO:tasks.rados.rados.0.smithi107.stdout:512: writing smithi10729407-4 from 2416640 to 2981888 tid 1 2020-06-12T02:37:32.724 INFO:tasks.rados.rados.0.smithi107.stdout:512: writing smithi10729407-4 from 2981888 to 3612672 tid 2 2020-06-12T02:37:32.724 INFO:tasks.rados.rados.0.smithi107.stdout:512: writing smithi10729407-4 from 3612672 to 3784704 tid 3 2020-06-12T02:37:32.724 INFO:tasks.rados.rados.0.smithi107.stdout:513: delete oid 19 current snap is 0 2020-06-12T02:37:32.724 INFO:tasks.rados.rados.0.smithi107.stdout:510: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:32.725 INFO:tasks.rados.rados.0.smithi107.stdout:510: finishing write tid 5 to smithi10729407-11 2020-06-12T02:37:32.725 INFO:tasks.rados.rados.0.smithi107.stdout:510: finishing write tid 6 to smithi10729407-11 2020-06-12T02:37:32.725 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 67 (ObjNum 509 snap 0 seq_num 509) dirty exists 2020-06-12T02:37:32.725 INFO:tasks.rados.rados.0.smithi107.stdout:510: left oid 11 (ObjNum 509 snap 0 seq_num 509) 2020-06-12T02:37:32.725 INFO:tasks.rados.rados.0.smithi107.stdout:510: done (2 left) 2020-06-12T02:37:32.725 INFO:tasks.rados.rados.0.smithi107.stdout:513: done (1 left) 2020-06-12T02:37:32.726 INFO:tasks.rados.rados.0.smithi107.stdout:514: delete oid 13 current snap is 0 2020-06-12T02:37:32.726 INFO:tasks.rados.rados.0.smithi107.stdout:512: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:32.726 INFO:tasks.rados.rados.0.smithi107.stdout:514: done (1 left) 2020-06-12T02:37:32.726 INFO:tasks.rados.rados.0.smithi107.stdout:515: delete oid 3 current snap is 0 2020-06-12T02:37:32.727 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-06-12T02:37:32.728 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:32.650+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:32.732 INFO:tasks.rados.rados.0.smithi107.stdout:512: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:32.732 INFO:tasks.rados.rados.0.smithi107.stdout:512: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:32.732 INFO:tasks.rados.rados.0.smithi107.stdout:512: finishing write tid 5 to smithi10729407-4 2020-06-12T02:37:32.732 INFO:tasks.rados.rados.0.smithi107.stdout:515: done (1 left) 2020-06-12T02:37:32.733 INFO:tasks.rados.rados.0.smithi107.stdout:516: delete oid 13 current snap is 0 2020-06-12T02:37:32.733 INFO:tasks.rados.rados.0.smithi107.stdout:512: finishing write tid 6 to smithi10729407-4 2020-06-12T02:37:32.733 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 106 (ObjNum 511 snap 0 seq_num 511) dirty exists 2020-06-12T02:37:32.733 INFO:tasks.rados.rados.0.smithi107.stdout:512: left oid 4 (ObjNum 511 snap 0 seq_num 511) 2020-06-12T02:37:32.733 INFO:tasks.rados.rados.0.smithi107.stdout:512: done (1 left) 2020-06-12T02:37:32.733 INFO:tasks.rados.rados.0.smithi107.stdout:516: done (0 left) 2020-06-12T02:37:32.734 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:37:32.734 INFO:tasks.rados.rados.0.smithi107.stdout:517: seq_num 516 ranges {0=647168,647168=450560,1097728=352256} 2020-06-12T02:37:32.734 INFO:tasks.rados.rados.0.smithi107.stdout:517: writing smithi10729407-1 from 0 to 647168 tid 1 2020-06-12T02:37:32.734 INFO:tasks.rados.rados.0.smithi107.stdout:517: writing smithi10729407-1 from 647168 to 1097728 tid 2 2020-06-12T02:37:32.734 INFO:tasks.rados.rados.0.smithi107.stdout:517: writing smithi10729407-1 from 1097728 to 1449984 tid 3 2020-06-12T02:37:32.735 INFO:tasks.rados.rados.0.smithi107.stdout:518: delete oid 9 current snap is 0 2020-06-12T02:37:32.735 INFO:tasks.rados.rados.0.smithi107.stdout:517: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:32.735 INFO:tasks.rados.rados.0.smithi107.stdout:518: done (1 left) 2020-06-12T02:37:32.735 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:37:32.735 INFO:tasks.rados.rados.0.smithi107.stdout:519: seq_num 518 ranges {0=679936,679936=671744,1351680=131072} 2020-06-12T02:37:32.735 INFO:tasks.rados.rados.0.smithi107.stdout:519: writing smithi10729407-16 from 0 to 679936 tid 1 2020-06-12T02:37:32.736 INFO:tasks.rados.rados.0.smithi107.stdout:519: writing smithi10729407-16 from 679936 to 1351680 tid 2 2020-06-12T02:37:32.736 INFO:tasks.rados.rados.0.smithi107.stdout:519: writing smithi10729407-16 from 1351680 to 1482752 tid 3 2020-06-12T02:37:32.736 INFO:tasks.rados.rados.0.smithi107.stdout:520: delete oid 7 current snap is 0 2020-06-12T02:37:32.736 INFO:tasks.rados.rados.0.smithi107.stdout:517: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:32.736 INFO:tasks.rados.rados.0.smithi107.stdout:517: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:32.737 INFO:tasks.rados.rados.0.smithi107.stdout:517: finishing write tid 5 to smithi10729407-1 2020-06-12T02:37:32.737 INFO:tasks.rados.rados.0.smithi107.stdout:517: finishing write tid 6 to smithi10729407-1 2020-06-12T02:37:32.737 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 118 (ObjNum 516 snap 0 seq_num 516) dirty exists 2020-06-12T02:37:32.737 INFO:tasks.rados.rados.0.smithi107.stdout:517: left oid 1 (ObjNum 516 snap 0 seq_num 516) 2020-06-12T02:37:32.738 INFO:tasks.rados.rados.0.smithi107.stdout:517: done (2 left) 2020-06-12T02:37:32.738 INFO:tasks.rados.rados.0.smithi107.stdout:520: done (1 left) 2020-06-12T02:37:32.738 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:37:32.738 INFO:tasks.rados.rados.0.smithi107.stdout:521: seq_num 520 ranges {0=712704,712704=434176,1146880=368640} 2020-06-12T02:37:32.738 INFO:tasks.rados.rados.0.smithi107.stdout:521: writing smithi10729407-6 from 0 to 712704 tid 1 2020-06-12T02:37:32.738 INFO:tasks.rados.rados.0.smithi107.stdout:521: writing smithi10729407-6 from 712704 to 1146880 tid 2 2020-06-12T02:37:32.739 INFO:tasks.rados.rados.0.smithi107.stdout:521: writing smithi10729407-6 from 1146880 to 1515520 tid 3 2020-06-12T02:37:32.739 INFO:tasks.rados.rados.0.smithi107.stdout:522: delete oid 4 current snap is 0 2020-06-12T02:37:32.739 INFO:tasks.rados.rados.0.smithi107.stdout:519: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:32.739 INFO:tasks.rados.rados.0.smithi107.stdout:519: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:32.739 INFO:tasks.rados.rados.0.smithi107.stdout:519: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:32.740 INFO:tasks.rados.rados.0.smithi107.stdout:519: finishing write tid 5 to smithi10729407-16 2020-06-12T02:37:32.740 INFO:tasks.rados.rados.0.smithi107.stdout:519: finishing write tid 6 to smithi10729407-16 2020-06-12T02:37:32.740 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 88 (ObjNum 518 snap 0 seq_num 518) dirty exists 2020-06-12T02:37:32.740 INFO:tasks.rados.rados.0.smithi107.stdout:519: left oid 16 (ObjNum 518 snap 0 seq_num 518) 2020-06-12T02:37:32.740 INFO:tasks.rados.rados.0.smithi107.stdout:521: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:32.740 INFO:tasks.rados.rados.0.smithi107.stdout:519: done (2 left) 2020-06-12T02:37:32.741 INFO:tasks.rados.rados.0.smithi107.stdout:522: done (1 left) 2020-06-12T02:37:32.741 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:37:32.741 INFO:tasks.rados.rados.0.smithi107.stdout:523: seq_num 522 ranges {0=745472,745472=606208,1351680=196608} 2020-06-12T02:37:32.741 INFO:tasks.rados.rados.0.smithi107.stdout:523: writing smithi10729407-9 from 0 to 745472 tid 1 2020-06-12T02:37:32.741 INFO:tasks.rados.rados.0.smithi107.stdout:523: writing smithi10729407-9 from 745472 to 1351680 tid 2 2020-06-12T02:37:32.742 INFO:tasks.rados.rados.0.smithi107.stdout:523: writing smithi10729407-9 from 1351680 to 1548288 tid 3 2020-06-12T02:37:32.742 INFO:tasks.rados.rados.0.smithi107.stdout:524: delete oid 4 current snap is 0 2020-06-12T02:37:32.742 INFO:tasks.rados.rados.0.smithi107.stdout:521: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:32.742 INFO:tasks.rados.rados.0.smithi107.stdout:521: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:32.742 INFO:tasks.rados.rados.0.smithi107.stdout:521: finishing write tid 5 to smithi10729407-6 2020-06-12T02:37:32.742 INFO:tasks.rados.rados.0.smithi107.stdout:521: finishing write tid 6 to smithi10729407-6 2020-06-12T02:37:32.743 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 122 (ObjNum 520 snap 0 seq_num 520) dirty exists 2020-06-12T02:37:32.743 INFO:tasks.rados.rados.0.smithi107.stdout:521: left oid 6 (ObjNum 520 snap 0 seq_num 520) 2020-06-12T02:37:32.743 INFO:tasks.rados.rados.0.smithi107.stdout:523: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:32.743 INFO:tasks.rados.rados.0.smithi107.stdout:521: done (2 left) 2020-06-12T02:37:32.743 INFO:tasks.rados.rados.0.smithi107.stdout:524: done (1 left) 2020-06-12T02:37:32.744 INFO:tasks.rados.rados.0.smithi107.stdout:525: delete oid 18 current snap is 0 2020-06-12T02:37:32.744 INFO:tasks.rados.rados.0.smithi107.stdout:523: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:32.744 INFO:tasks.rados.rados.0.smithi107.stdout:523: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:32.744 INFO:tasks.rados.rados.0.smithi107.stdout:523: finishing write tid 5 to smithi10729407-9 2020-06-12T02:37:32.744 INFO:tasks.rados.rados.0.smithi107.stdout:525: done (1 left) 2020-06-12T02:37:32.744 INFO:tasks.rados.rados.0.smithi107.stdout:526: delete oid 17 current snap is 0 2020-06-12T02:37:32.745 INFO:tasks.rados.rados.0.smithi107.stdout:523: finishing write tid 6 to smithi10729407-9 2020-06-12T02:37:32.745 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 97 (ObjNum 522 snap 0 seq_num 522) dirty exists 2020-06-12T02:37:32.745 INFO:tasks.rados.rados.0.smithi107.stdout:523: left oid 9 (ObjNum 522 snap 0 seq_num 522) 2020-06-12T02:37:32.745 INFO:tasks.rados.rados.0.smithi107.stdout:523: done (1 left) 2020-06-12T02:37:32.745 INFO:tasks.rados.rados.0.smithi107.stdout:526: done (0 left) 2020-06-12T02:37:32.745 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:37:32.746 INFO:tasks.rados.rados.0.smithi107.stdout:527: seq_num 526 ranges {0=417792,417792=589824,1007616=614400} 2020-06-12T02:37:32.747 INFO:tasks.rados.rados.0.smithi107.stdout:527: writing smithi10729407-20 from 0 to 417792 tid 1 2020-06-12T02:37:32.747 INFO:tasks.rados.rados.0.smithi107.stdout:527: writing smithi10729407-20 from 417792 to 1007616 tid 2 2020-06-12T02:37:32.748 INFO:tasks.rados.rados.0.smithi107.stdout:527: writing smithi10729407-20 from 1007616 to 1622016 tid 3 2020-06-12T02:37:32.748 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:37:32.748 INFO:tasks.rados.rados.0.smithi107.stdout:528: seq_num 527 ranges {1548288=434176,1982464=475136,2457600=565248,3022848=163840} 2020-06-12T02:37:32.750 INFO:tasks.rados.rados.0.smithi107.stdout:528: writing smithi10729407-9 from 1548288 to 1982464 tid 1 2020-06-12T02:37:32.752 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:37:32.750+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:37:32.753 INFO:tasks.rados.rados.0.smithi107.stdout:528: writing smithi10729407-9 from 1982464 to 2457600 tid 2 2020-06-12T02:37:32.758 INFO:tasks.rados.rados.0.smithi107.stdout:528: writing smithi10729407-9 from 2457600 to 3022848 tid 3 2020-06-12T02:37:32.759 INFO:tasks.rados.rados.0.smithi107.stdout:528: writing smithi10729407-9 from 3022848 to 3186688 tid 4 2020-06-12T02:37:32.759 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:37:32.759 INFO:tasks.rados.rados.0.smithi107.stdout:529: seq_num 528 ranges {573440=450560,1024000=761856,1785856=442368} 2020-06-12T02:37:32.760 INFO:teuthology.orchestra.run.smithi188.stderr:Nothing to do for non-filestore backend 2020-06-12T02:37:32.764 INFO:tasks.rados.rados.0.smithi107.stdout:529: writing smithi10729407-15 from 573440 to 1024000 tid 1 2020-06-12T02:37:32.769 INFO:tasks.rados.rados.0.smithi107.stdout:529: writing smithi10729407-15 from 1024000 to 1785856 tid 2 2020-06-12T02:37:32.772 INFO:tasks.rados.rados.0.smithi107.stdout:529: writing smithi10729407-15 from 1785856 to 2228224 tid 3 2020-06-12T02:37:32.773 INFO:tasks.rados.rados.0.smithi107.stdout:530: delete oid 4 current snap is 0 2020-06-12T02:37:32.773 INFO:tasks.rados.rados.0.smithi107.stdout:527: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:32.773 INFO:tasks.rados.rados.0.smithi107.stdout:527: finishing write tid 2 to smithi10729407-20 2020-06-12T02:37:32.773 INFO:tasks.rados.rados.0.smithi107.stdout:527: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:32.773 INFO:tasks.rados.rados.0.smithi107.stdout:527: finishing write tid 5 to smithi10729407-20 2020-06-12T02:37:32.774 INFO:tasks.rados.rados.0.smithi107.stdout:527: finishing write tid 6 to smithi10729407-20 2020-06-12T02:37:32.774 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 41 (ObjNum 526 snap 0 seq_num 526) dirty exists 2020-06-12T02:37:32.774 INFO:tasks.rados.rados.0.smithi107.stdout:527: left oid 20 (ObjNum 526 snap 0 seq_num 526) 2020-06-12T02:37:32.774 INFO:tasks.rados.rados.0.smithi107.stdout:528: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:32.774 INFO:tasks.rados.rados.0.smithi107.stdout:528: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:32.775 INFO:tasks.rados.rados.0.smithi107.stdout:528: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:32.775 INFO:tasks.rados.rados.0.smithi107.stdout:528: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:32.775 INFO:tasks.rados.rados.0.smithi107.stdout:528: finishing write tid 6 to smithi10729407-9 2020-06-12T02:37:32.775 INFO:tasks.rados.rados.0.smithi107.stdout:529: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:32.775 INFO:tasks.rados.rados.0.smithi107.stdout:528: finishing write tid 7 to smithi10729407-9 2020-06-12T02:37:32.775 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 103 (ObjNum 527 snap 0 seq_num 527) dirty exists 2020-06-12T02:37:32.776 INFO:tasks.rados.rados.0.smithi107.stdout:528: left oid 9 (ObjNum 527 snap 0 seq_num 527) 2020-06-12T02:37:32.776 INFO:tasks.rados.rados.0.smithi107.stdout:527: done (3 left) 2020-06-12T02:37:32.776 INFO:tasks.rados.rados.0.smithi107.stdout:528: done (2 left) 2020-06-12T02:37:32.777 INFO:tasks.rados.rados.0.smithi107.stdout:530: done (1 left) 2020-06-12T02:37:32.777 INFO:tasks.rados.rados.0.smithi107.stdout:531: delete oid 8 current snap is 0 2020-06-12T02:37:32.779 INFO:tasks.rados.rados.0.smithi107.stdout:529: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:32.780 INFO:tasks.rados.rados.0.smithi107.stdout:529: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:32.781 INFO:tasks.rados.rados.0.smithi107.stdout:529: finishing write tid 5 to smithi10729407-15 2020-06-12T02:37:32.783 INFO:tasks.rados.rados.0.smithi107.stdout:529: finishing write tid 6 to smithi10729407-15 2020-06-12T02:37:32.783 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 107 (ObjNum 528 snap 0 seq_num 528) dirty exists 2020-06-12T02:37:32.783 INFO:tasks.rados.rados.0.smithi107.stdout:529: left oid 15 (ObjNum 528 snap 0 seq_num 528) 2020-06-12T02:37:32.784 INFO:tasks.rados.rados.0.smithi107.stdout:529: done (1 left) 2020-06-12T02:37:32.784 INFO:tasks.rados.rados.0.smithi107.stdout:531: done (0 left) 2020-06-12T02:37:32.784 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:37:32.784 INFO:tasks.rados.rados.0.smithi107.stdout:532: seq_num 531 ranges {0=499712,499712=417792,917504=548864,1466368=237568} 2020-06-12T02:37:32.787 INFO:tasks.rados.rados.0.smithi107.stdout:532: writing smithi10729407-19 from 0 to 499712 tid 1 2020-06-12T02:37:32.789 INFO:tasks.rados.rados.0.smithi107.stdout:532: writing smithi10729407-19 from 499712 to 917504 tid 2 2020-06-12T02:37:32.792 INFO:tasks.rados.rados.0.smithi107.stdout:532: writing smithi10729407-19 from 917504 to 1466368 tid 3 2020-06-12T02:37:32.794 INFO:tasks.rados.rados.0.smithi107.stdout:532: writing smithi10729407-19 from 1466368 to 1703936 tid 4 2020-06-12T02:37:32.794 INFO:tasks.rados.rados.0.smithi107.stdout:533: delete oid 3 current snap is 0 2020-06-12T02:37:32.795 INFO:tasks.rados.rados.0.smithi107.stdout:532: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:32.796 INFO:tasks.rados.rados.0.smithi107.stdout:532: finishing write tid 2 to smithi10729407-19 2020-06-12T02:37:32.796 INFO:tasks.rados.rados.0.smithi107.stdout:533: done (1 left) 2020-06-12T02:37:32.796 INFO:tasks.rados.rados.0.smithi107.stdout:534: delete oid 1 current snap is 0 2020-06-12T02:37:32.798 INFO:tasks.rados.rados.0.smithi107.stdout:534: done (1 left) 2020-06-12T02:37:32.798 INFO:tasks.rados.rados.0.smithi107.stdout:535: delete oid 18 current snap is 0 2020-06-12T02:37:32.799 INFO:tasks.rados.rados.0.smithi107.stdout:532: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:32.800 INFO:tasks.rados.rados.0.smithi107.stdout:532: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:32.800 INFO:tasks.rados.rados.0.smithi107.stdout:535: done (1 left) 2020-06-12T02:37:32.800 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:37:32.800 INFO:tasks.rados.rados.0.smithi107.stdout:536: seq_num 535 ranges {1163264=565248,1728512=802816,2531328=401408} 2020-06-12T02:37:32.809 INFO:tasks.rados.rados.0.smithi107.stdout:536: writing smithi10729407-5 from 1163264 to 1728512 tid 1 2020-06-12T02:37:32.814 INFO:tasks.rados.rados.0.smithi107.stdout:536: writing smithi10729407-5 from 1728512 to 2531328 tid 2 2020-06-12T02:37:32.816 INFO:tasks.rados.rados.0.smithi107.stdout:536: writing smithi10729407-5 from 2531328 to 2932736 tid 3 2020-06-12T02:37:32.817 INFO:tasks.rados.rados.0.smithi107.stdout:537: delete oid 1 current snap is 0 2020-06-12T02:37:32.817 INFO:tasks.rados.rados.0.smithi107.stdout:532: finishing write tid 6 to smithi10729407-19 2020-06-12T02:37:32.817 INFO:tasks.rados.rados.0.smithi107.stdout:532: finishing write tid 7 to smithi10729407-19 2020-06-12T02:37:32.817 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 83 (ObjNum 531 snap 0 seq_num 531) dirty exists 2020-06-12T02:37:32.817 INFO:tasks.rados.rados.0.smithi107.stdout:532: left oid 19 (ObjNum 531 snap 0 seq_num 531) 2020-06-12T02:37:32.818 INFO:tasks.rados.rados.0.smithi107.stdout:532: done (2 left) 2020-06-12T02:37:32.818 INFO:tasks.rados.rados.0.smithi107.stdout:537: done (1 left) 2020-06-12T02:37:32.819 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:37:32.819 INFO:tasks.rados.rados.0.smithi107.stdout:538: seq_num 537 ranges {1622016=598016,2220032=573440,2793472=491520,3284992=139264} 2020-06-12T02:37:32.830 INFO:tasks.rados.rados.0.smithi107.stdout:538: writing smithi10729407-20 from 1622016 to 2220032 tid 1 2020-06-12T02:37:32.833 INFO:tasks.rados.rados.0.smithi107.stdout:538: writing smithi10729407-20 from 2220032 to 2793472 tid 2 2020-06-12T02:37:32.837 INFO:tasks.rados.rados.0.smithi107.stdout:538: writing smithi10729407-20 from 2793472 to 3284992 tid 3 2020-06-12T02:37:32.838 INFO:tasks.rados.rados.0.smithi107.stdout:538: writing smithi10729407-20 from 3284992 to 3424256 tid 4 2020-06-12T02:37:32.838 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:37:32.838 INFO:tasks.rados.rados.0.smithi107.stdout:539: seq_num 538 ranges {1196032=614400,1810432=450560,2260992=704512,2965504=49152} 2020-06-12T02:37:32.848 INFO:tasks.rados.rados.0.smithi107.stdout:539: writing smithi10729407-14 from 1196032 to 1810432 tid 1 2020-06-12T02:37:32.851 INFO:tasks.rados.rados.0.smithi107.stdout:539: writing smithi10729407-14 from 1810432 to 2260992 tid 2 2020-06-12T02:37:32.854 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:37:32.850+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:37:32.856 INFO:tasks.rados.rados.0.smithi107.stdout:539: writing smithi10729407-14 from 2260992 to 2965504 tid 3 2020-06-12T02:37:32.857 INFO:tasks.rados.rados.0.smithi107.stdout:539: writing smithi10729407-14 from 2965504 to 3014656 tid 4 2020-06-12T02:37:32.857 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:37:32.857 INFO:tasks.rados.rados.0.smithi107.stdout:540: seq_num 539 ranges {0=630784,630784=737280,1368064=466944} 2020-06-12T02:37:32.860 INFO:tasks.rados.rados.0.smithi107.stdout:540: writing smithi10729407-13 from 0 to 630784 tid 1 2020-06-12T02:37:32.866 INFO:tasks.rados.rados.0.smithi107.stdout:540: writing smithi10729407-13 from 630784 to 1368064 tid 2 2020-06-12T02:37:32.869 INFO:tasks.rados.rados.0.smithi107.stdout:540: writing smithi10729407-13 from 1368064 to 1835008 tid 3 2020-06-12T02:37:32.869 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:37:32.869 INFO:tasks.rados.rados.0.smithi107.stdout:541: seq_num 540 ranges {3186688=647168,3833856=671744,4505600=532480} 2020-06-12T02:37:32.890 INFO:tasks.rados.rados.0.smithi107.stdout:541: writing smithi10729407-9 from 3186688 to 3833856 tid 1 2020-06-12T02:37:32.893 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:37:32.894 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2020-06-12T02:37:32.894 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.894 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1212 3.7s0 3:ea271007:::smithi10729407-9:head [append 0~647168 in=647168b,stat out=16b] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:32.894 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:32.889130+0000", 2020-06-12T02:37:32.895 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.002714943, 2020-06-12T02:37:32.895 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002733732, 2020-06-12T02:37:32.895 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:32.895 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:37:32.895 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:32.896 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:32.896 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:32.896 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1212 2020-06-12T02:37:32.896 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.896 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:32.897 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.897 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:32.897 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.889130+0000", 2020-06-12T02:37:32.897 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:32.897 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.898 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.898 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:32.898 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.889130+0000", 2020-06-12T02:37:32.898 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016558499999999999 2020-06-12T02:37:32.898 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.899 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.899 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:32.899 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.889296+0000", 2020-06-12T02:37:32.899 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3799999999999996e-06 2020-06-12T02:37:32.899 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.900 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.900 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:32.900 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.889303+0000", 2020-06-12T02:37:32.900 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1060000000000001e-05 2020-06-12T02:37:32.900 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.901 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.901 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:32.901 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.889314+0000", 2020-06-12T02:37:32.901 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6629999999999998e-06 2020-06-12T02:37:32.901 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.902 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.902 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:32.902 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.889320+0000", 2020-06-12T02:37:32.902 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3455999999999998e-05 2020-06-12T02:37:32.902 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.903 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.903 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:32.903 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.889373+0000", 2020-06-12T02:37:32.903 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011629 2020-06-12T02:37:32.903 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.904 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.904 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:32.904 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.889490+0000", 2020-06-12T02:37:32.904 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00070292200000000003 2020-06-12T02:37:32.905 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:32.905 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:32.905 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:32.905 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.890192+0000", 2020-06-12T02:37:32.905 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001062356 2020-06-12T02:37:32.906 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.906 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:32.906 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.906 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:32.906 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2020-06-12T02:37:32.907 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 1 2020-06-12T02:37:32.907 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:37:32.908 INFO:tasks.rados.rados.0.smithi107.stdout:541: writing smithi10729407-9 from 3833856 to 4505600 tid 2 2020-06-12T02:37:32.909 INFO:tasks.rados.rados.0.smithi107.stdout:541: writing smithi10729407-9 from 4505600 to 5038080 tid 3 2020-06-12T02:37:32.909 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:37:32.909 INFO:tasks.rados.rados.0.smithi107.stdout:542: seq_num 541 ranges {1335296=663552,1998848=557056,2555904=475136,3031040=172032} 2020-06-12T02:37:32.910 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-06-12T02:37:32.911 INFO:tasks.rados.rados.0.smithi107.stdout:542: writing smithi10729407-11 from 1335296 to 1998848 tid 1 2020-06-12T02:37:32.913 INFO:tasks.rados.rados.0.smithi107.stdout:542: writing smithi10729407-11 from 1998848 to 2555904 tid 2 2020-06-12T02:37:32.916 INFO:tasks.rados.rados.0.smithi107.stdout:542: writing smithi10729407-11 from 2555904 to 3031040 tid 3 2020-06-12T02:37:32.918 INFO:tasks.rados.rados.0.smithi107.stdout:542: writing smithi10729407-11 from 3031040 to 3203072 tid 4 2020-06-12T02:37:32.919 INFO:tasks.rados.rados.0.smithi107.stdout:543: delete oid 17 current snap is 0 2020-06-12T02:37:32.919 INFO:tasks.rados.rados.0.smithi107.stdout:536: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:32.919 INFO:tasks.rados.rados.0.smithi107.stdout:536: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:32.919 INFO:tasks.rados.rados.0.smithi107.stdout:536: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:32.920 INFO:tasks.rados.rados.0.smithi107.stdout:536: finishing write tid 5 to smithi10729407-5 2020-06-12T02:37:32.920 INFO:tasks.rados.rados.0.smithi107.stdout:536: finishing write tid 6 to smithi10729407-5 2020-06-12T02:37:32.920 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 57 (ObjNum 535 snap 0 seq_num 535) dirty exists 2020-06-12T02:37:32.920 INFO:tasks.rados.rados.0.smithi107.stdout:536: left oid 5 (ObjNum 535 snap 0 seq_num 535) 2020-06-12T02:37:32.920 INFO:tasks.rados.rados.0.smithi107.stdout:538: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:32.920 INFO:tasks.rados.rados.0.smithi107.stdout:538: finishing write tid 2 to smithi10729407-20 2020-06-12T02:37:32.921 INFO:tasks.rados.rados.0.smithi107.stdout:538: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:32.921 INFO:tasks.rados.rados.0.smithi107.stdout:538: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:32.921 INFO:tasks.rados.rados.0.smithi107.stdout:538: finishing write tid 6 to smithi10729407-20 2020-06-12T02:37:32.921 INFO:tasks.rados.rados.0.smithi107.stdout:538: finishing write tid 7 to smithi10729407-20 2020-06-12T02:37:32.921 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 46 (ObjNum 537 snap 0 seq_num 537) dirty exists 2020-06-12T02:37:32.922 INFO:tasks.rados.rados.0.smithi107.stdout:538: left oid 20 (ObjNum 537 snap 0 seq_num 537) 2020-06-12T02:37:32.923 INFO:tasks.rados.rados.0.smithi107.stdout:539: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:32.923 INFO:tasks.rados.rados.0.smithi107.stdout:539: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:32.923 INFO:tasks.rados.rados.0.smithi107.stdout:539: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:32.923 INFO:tasks.rados.rados.0.smithi107.stdout:540: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:32.923 INFO:tasks.rados.rados.0.smithi107.stdout:539: finishing write tid 4 to smithi10729407-14 2020-06-12T02:37:32.923 INFO:tasks.rados.rados.0.smithi107.stdout:540: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:32.924 INFO:tasks.rados.rados.0.smithi107.stdout:539: finishing write tid 6 to smithi10729407-14 2020-06-12T02:37:32.924 INFO:tasks.rados.rados.0.smithi107.stdout:539: finishing write tid 7 to smithi10729407-14 2020-06-12T02:37:32.924 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 72 (ObjNum 538 snap 0 seq_num 538) dirty exists 2020-06-12T02:37:32.924 INFO:tasks.rados.rados.0.smithi107.stdout:539: left oid 14 (ObjNum 538 snap 0 seq_num 538) 2020-06-12T02:37:32.924 INFO:tasks.rados.rados.0.smithi107.stdout:540: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:32.925 INFO:tasks.rados.rados.0.smithi107.stdout:540: finishing write tid 5 to smithi10729407-13 2020-06-12T02:37:32.925 INFO:tasks.rados.rados.0.smithi107.stdout:540: finishing write tid 6 to smithi10729407-13 2020-06-12T02:37:32.925 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 106 (ObjNum 539 snap 0 seq_num 539) dirty exists 2020-06-12T02:37:32.925 INFO:tasks.rados.rados.0.smithi107.stdout:540: left oid 13 (ObjNum 539 snap 0 seq_num 539) 2020-06-12T02:37:32.925 INFO:tasks.rados.rados.0.smithi107.stdout:541: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:32.925 INFO:tasks.rados.rados.0.smithi107.stdout:541: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:32.926 INFO:tasks.rados.rados.0.smithi107.stdout:541: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:32.926 INFO:tasks.rados.rados.0.smithi107.stdout:541: finishing write tid 5 to smithi10729407-9 2020-06-12T02:37:32.926 INFO:tasks.rados.rados.0.smithi107.stdout:541: finishing write tid 6 to smithi10729407-9 2020-06-12T02:37:32.926 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 108 (ObjNum 540 snap 0 seq_num 540) dirty exists 2020-06-12T02:37:32.927 INFO:tasks.rados.rados.0.smithi107.stdout:541: left oid 9 (ObjNum 540 snap 0 seq_num 540) 2020-06-12T02:37:32.927 INFO:tasks.rados.rados.0.smithi107.stdout:542: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:32.928 INFO:tasks.rados.rados.0.smithi107.stdout:536: done (6 left) 2020-06-12T02:37:32.928 INFO:tasks.rados.rados.0.smithi107.stdout:538: done (5 left) 2020-06-12T02:37:32.929 INFO:tasks.rados.rados.0.smithi107.stdout:539: done (4 left) 2020-06-12T02:37:32.929 INFO:tasks.rados.rados.0.smithi107.stdout:540: done (3 left) 2020-06-12T02:37:32.929 INFO:tasks.rados.rados.0.smithi107.stdout:541: done (2 left) 2020-06-12T02:37:32.929 INFO:tasks.rados.rados.0.smithi107.stdout:543: done (1 left) 2020-06-12T02:37:32.930 INFO:tasks.rados.rados.0.smithi107.stdout:544: delete oid 20 current snap is 0 2020-06-12T02:37:32.930 INFO:tasks.rados.rados.0.smithi107.stdout:544: done (1 left) 2020-06-12T02:37:32.930 INFO:tasks.rados.rados.0.smithi107.stdout:545: delete oid 18 current snap is 0 2020-06-12T02:37:32.930 INFO:tasks.rados.rados.0.smithi107.stdout:545: done (1 left) 2020-06-12T02:37:32.930 INFO:tasks.rados.rados.0.smithi107.stdout:546: delete oid 7 current snap is 0 2020-06-12T02:37:32.931 INFO:tasks.rados.rados.0.smithi107.stdout:546: done (1 left) 2020-06-12T02:37:32.931 INFO:tasks.rados.rados.0.smithi107.stdout:547: delete oid 6 current snap is 0 2020-06-12T02:37:32.932 INFO:tasks.rados.rados.0.smithi107.stdout:542: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:32.932 INFO:tasks.rados.rados.0.smithi107.stdout:542: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:32.932 INFO:tasks.rados.rados.0.smithi107.stdout:542: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:32.932 INFO:tasks.rados.rados.0.smithi107.stdout:542: finishing write tid 6 to smithi10729407-11 2020-06-12T02:37:32.933 INFO:tasks.rados.rados.0.smithi107.stdout:547: done (1 left) 2020-06-12T02:37:32.933 INFO:tasks.rados.rados.0.smithi107.stdout:548: delete oid 20 current snap is 0 2020-06-12T02:37:32.934 INFO:tasks.rados.rados.0.smithi107.stdout:542: finishing write tid 7 to smithi10729407-11 2020-06-12T02:37:32.934 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 77 (ObjNum 541 snap 0 seq_num 541) dirty exists 2020-06-12T02:37:32.934 INFO:tasks.rados.rados.0.smithi107.stdout:542: left oid 11 (ObjNum 541 snap 0 seq_num 541) 2020-06-12T02:37:32.934 INFO:tasks.rados.rados.0.smithi107.stdout:542: done (1 left) 2020-06-12T02:37:32.934 INFO:tasks.rados.rados.0.smithi107.stdout:548: done (0 left) 2020-06-12T02:37:32.935 INFO:tasks.rados.rados.0.smithi107.stdout:549: delete oid 4 current snap is 0 2020-06-12T02:37:32.936 INFO:tasks.rados.rados.0.smithi107.stdout:549: done (0 left) 2020-06-12T02:37:32.936 INFO:tasks.rados.rados.0.smithi107.stdout:550: delete oid 20 current snap is 0 2020-06-12T02:37:32.936 INFO:tasks.rados.rados.0.smithi107.stdout:550: done (0 left) 2020-06-12T02:37:32.936 INFO:tasks.rados.rados.0.smithi107.stdout:551: delete oid 8 current snap is 0 2020-06-12T02:37:32.937 INFO:tasks.rados.rados.0.smithi107.stdout:551: done (0 left) 2020-06-12T02:37:32.937 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:37:32.937 INFO:tasks.rados.rados.0.smithi107.stdout:552: seq_num 551 ranges {1835008=434176,2269184=647168,2916352=450560,3366912=507904} 2020-06-12T02:37:32.948 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd unset nodeep-scrub 2020-06-12T02:37:32.949 INFO:tasks.rados.rados.0.smithi107.stdout:552: writing smithi10729407-13 from 1835008 to 2269184 tid 1 2020-06-12T02:37:32.953 INFO:tasks.rados.rados.0.smithi107.stdout:552: writing smithi10729407-13 from 2269184 to 2916352 tid 2 2020-06-12T02:37:32.955 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:32.952+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:32.956 INFO:tasks.rados.rados.0.smithi107.stdout:552: writing smithi10729407-13 from 2916352 to 3366912 tid 3 2020-06-12T02:37:32.959 INFO:tasks.rados.rados.0.smithi107.stdout:552: writing smithi10729407-13 from 3366912 to 3874816 tid 4 2020-06-12T02:37:32.960 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:37:32.960 INFO:tasks.rados.rados.0.smithi107.stdout:553: seq_num 552 ranges {0=450560,450560=532480,983040=655360,1638400=417792} 2020-06-12T02:37:32.962 INFO:tasks.rados.rados.0.smithi107.stdout:553: writing smithi10729407-6 from 0 to 450560 tid 1 2020-06-12T02:37:32.966 INFO:tasks.rados.rados.0.smithi107.stdout:553: writing smithi10729407-6 from 450560 to 983040 tid 2 2020-06-12T02:37:32.970 INFO:tasks.rados.rados.0.smithi107.stdout:553: writing smithi10729407-6 from 983040 to 1638400 tid 3 2020-06-12T02:37:32.973 INFO:tasks.rados.rados.0.smithi107.stdout:553: writing smithi10729407-6 from 1638400 to 2056192 tid 4 2020-06-12T02:37:32.973 INFO:tasks.rados.rados.0.smithi107.stdout:554: delete oid 9 current snap is 0 2020-06-12T02:37:32.973 INFO:tasks.rados.rados.0.smithi107.stdout:552: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:32.974 INFO:tasks.rados.rados.0.smithi107.stdout:552: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:32.974 INFO:tasks.rados.rados.0.smithi107.stdout:552: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:32.974 INFO:tasks.rados.rados.0.smithi107.stdout:553: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:32.974 INFO:tasks.rados.rados.0.smithi107.stdout:552: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:32.974 INFO:tasks.rados.rados.0.smithi107.stdout:552: finishing write tid 6 to smithi10729407-13 2020-06-12T02:37:32.975 INFO:tasks.rados.rados.0.smithi107.stdout:552: finishing write tid 7 to smithi10729407-13 2020-06-12T02:37:32.975 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 112 (ObjNum 551 snap 0 seq_num 551) dirty exists 2020-06-12T02:37:32.975 INFO:tasks.rados.rados.0.smithi107.stdout:552: left oid 13 (ObjNum 551 snap 0 seq_num 551) 2020-06-12T02:37:32.975 INFO:tasks.rados.rados.0.smithi107.stdout:552: done (2 left) 2020-06-12T02:37:32.976 INFO:tasks.rados.rados.0.smithi107.stdout:554: done (1 left) 2020-06-12T02:37:32.976 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:37:32.976 INFO:tasks.rados.rados.0.smithi107.stdout:555: seq_num 554 ranges {0=483328,483328=704512,1187840=622592,1810432=278528} 2020-06-12T02:37:32.978 INFO:tasks.rados.rados.0.smithi107.stdout:555: writing smithi10729407-3 from 0 to 483328 tid 1 2020-06-12T02:37:32.983 INFO:tasks.rados.rados.0.smithi107.stdout:555: writing smithi10729407-3 from 483328 to 1187840 tid 2 2020-06-12T02:37:32.987 INFO:tasks.rados.rados.0.smithi107.stdout:555: writing smithi10729407-3 from 1187840 to 1810432 tid 3 2020-06-12T02:37:32.989 INFO:tasks.rados.rados.0.smithi107.stdout:555: writing smithi10729407-3 from 1810432 to 2088960 tid 4 2020-06-12T02:37:32.989 INFO:tasks.rados.rados.0.smithi107.stdout:556: delete oid 18 current snap is 0 2020-06-12T02:37:32.989 INFO:tasks.rados.rados.0.smithi107.stdout:553: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:32.989 INFO:tasks.rados.rados.0.smithi107.stdout:553: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:32.990 INFO:tasks.rados.rados.0.smithi107.stdout:553: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:32.990 INFO:tasks.rados.rados.0.smithi107.stdout:553: finishing write tid 6 to smithi10729407-6 2020-06-12T02:37:32.990 INFO:tasks.rados.rados.0.smithi107.stdout:555: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:32.991 INFO:tasks.rados.rados.0.smithi107.stdout:553: finishing write tid 7 to smithi10729407-6 2020-06-12T02:37:32.991 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 130 (ObjNum 552 snap 0 seq_num 552) dirty exists 2020-06-12T02:37:32.991 INFO:tasks.rados.rados.0.smithi107.stdout:553: left oid 6 (ObjNum 552 snap 0 seq_num 552) 2020-06-12T02:37:32.997 INFO:tasks.rados.rados.0.smithi107.stdout:553: done (2 left) 2020-06-12T02:37:32.997 INFO:tasks.rados.rados.0.smithi107.stdout:556: done (1 left) 2020-06-12T02:37:32.998 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:37:32.998 INFO:tasks.rados.rados.0.smithi107.stdout:557: seq_num 556 ranges {0=516096,516096=524288,1040384=638976,1679360=442368} 2020-06-12T02:37:33.000 INFO:tasks.rados.rados.0.smithi107.stdout:557: writing smithi10729407-2 from 0 to 516096 tid 1 2020-06-12T02:37:33.004 INFO:tasks.rados.rados.0.smithi107.stdout:557: writing smithi10729407-2 from 516096 to 1040384 tid 2 2020-06-12T02:37:33.008 INFO:tasks.rados.rados.0.smithi107.stdout:557: writing smithi10729407-2 from 1040384 to 1679360 tid 3 2020-06-12T02:37:33.011 INFO:tasks.rados.rados.0.smithi107.stdout:557: writing smithi10729407-2 from 1679360 to 2121728 tid 4 2020-06-12T02:37:33.011 INFO:tasks.rados.rados.0.smithi107.stdout:558: delete oid 6 current snap is 0 2020-06-12T02:37:33.011 INFO:tasks.rados.rados.0.smithi107.stdout:555: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:33.012 INFO:tasks.rados.rados.0.smithi107.stdout:555: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:33.012 INFO:tasks.rados.rados.0.smithi107.stdout:555: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:33.012 INFO:tasks.rados.rados.0.smithi107.stdout:555: finishing write tid 6 to smithi10729407-3 2020-06-12T02:37:33.012 INFO:tasks.rados.rados.0.smithi107.stdout:555: finishing write tid 7 to smithi10729407-3 2020-06-12T02:37:33.013 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 94 (ObjNum 554 snap 0 seq_num 554) dirty exists 2020-06-12T02:37:33.013 INFO:tasks.rados.rados.0.smithi107.stdout:555: left oid 3 (ObjNum 554 snap 0 seq_num 554) 2020-06-12T02:37:33.013 INFO:tasks.rados.rados.0.smithi107.stdout:555: done (2 left) 2020-06-12T02:37:33.014 INFO:tasks.rados.rados.0.smithi107.stdout:558: done (1 left) 2020-06-12T02:37:33.014 INFO:tasks.rados.rados.0.smithi107.stdout:559: delete oid 9 current snap is 0 2020-06-12T02:37:33.015 INFO:tasks.rados.rados.0.smithi107.stdout:559: done (1 left) 2020-06-12T02:37:33.016 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:37:33.016 INFO:tasks.rados.rados.0.smithi107.stdout:560: seq_num 559 ranges {0=565248,565248=573440,1138688=409600,1548288=622592} 2020-06-12T02:37:33.019 INFO:tasks.rados.rados.0.smithi107.stdout:560: writing smithi10729407-9 from 0 to 565248 tid 1 2020-06-12T02:37:33.022 INFO:tasks.rados.rados.0.smithi107.stdout:560: writing smithi10729407-9 from 565248 to 1138688 tid 2 2020-06-12T02:37:33.025 INFO:tasks.rados.rados.0.smithi107.stdout:560: writing smithi10729407-9 from 1138688 to 1548288 tid 3 2020-06-12T02:37:33.029 INFO:tasks.rados.rados.0.smithi107.stdout:560: writing smithi10729407-9 from 1548288 to 2170880 tid 4 2020-06-12T02:37:33.030 INFO:tasks.rados.rados.0.smithi107.stdout:561: delete oid 16 current snap is 0 2020-06-12T02:37:33.030 INFO:tasks.rados.rados.0.smithi107.stdout:557: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:33.030 INFO:tasks.rados.rados.0.smithi107.stdout:557: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:33.030 INFO:tasks.rados.rados.0.smithi107.stdout:557: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:33.030 INFO:tasks.rados.rados.0.smithi107.stdout:557: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:33.031 INFO:tasks.rados.rados.0.smithi107.stdout:557: finishing write tid 6 to smithi10729407-2 2020-06-12T02:37:33.031 INFO:tasks.rados.rados.0.smithi107.stdout:557: finishing write tid 7 to smithi10729407-2 2020-06-12T02:37:33.031 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 62 (ObjNum 556 snap 0 seq_num 556) dirty exists 2020-06-12T02:37:33.031 INFO:tasks.rados.rados.0.smithi107.stdout:557: left oid 2 (ObjNum 556 snap 0 seq_num 556) 2020-06-12T02:37:33.032 INFO:tasks.rados.rados.0.smithi107.stdout:560: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:33.033 INFO:tasks.rados.rados.0.smithi107.stdout:557: done (2 left) 2020-06-12T02:37:33.033 INFO:tasks.rados.rados.0.smithi107.stdout:561: done (1 left) 2020-06-12T02:37:33.033 INFO:tasks.rados.rados.0.smithi107.stdout:562: delete oid 19 current snap is 0 2020-06-12T02:37:33.036 INFO:tasks.rados.rados.0.smithi107.stdout:562: done (1 left) 2020-06-12T02:37:33.036 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:37:33.037 INFO:tasks.rados.rados.0.smithi107.stdout:563: seq_num 562 ranges {2932736=622592,3555328=630784,4186112=581632,4767744=393216} 2020-06-12T02:37:33.056 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:33.053+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:37:33.058 INFO:tasks.rados.rados.0.smithi107.stdout:563: writing smithi10729407-5 from 2932736 to 3555328 tid 1 2020-06-12T02:37:33.063 INFO:tasks.rados.rados.0.smithi107.stdout:563: writing smithi10729407-5 from 3555328 to 4186112 tid 2 2020-06-12T02:37:33.065 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:37:33.065 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:37:33.065 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2020-06-12T02:37:33.065 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2020-06-12T02:37:33.066 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:37:33.067 INFO:tasks.rados.rados.0.smithi107.stdout:563: writing smithi10729407-5 from 4186112 to 4767744 tid 3 2020-06-12T02:37:33.070 INFO:tasks.rados.rados.0.smithi107.stdout:563: writing smithi10729407-5 from 4767744 to 5160960 tid 4 2020-06-12T02:37:33.070 INFO:tasks.rados.rados.0.smithi107.stdout:564: delete oid 14 current snap is 0 2020-06-12T02:37:33.071 INFO:tasks.rados.rados.0.smithi107.stdout:560: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:33.071 INFO:tasks.rados.rados.0.smithi107.stdout:560: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:33.071 INFO:tasks.rados.rados.0.smithi107.stdout:560: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:33.071 INFO:tasks.rados.rados.0.smithi107.stdout:560: finishing write tid 6 to smithi10729407-9 2020-06-12T02:37:33.071 INFO:tasks.rados.rados.0.smithi107.stdout:560: finishing write tid 7 to smithi10729407-9 2020-06-12T02:37:33.072 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 117 (ObjNum 559 snap 0 seq_num 559) dirty exists 2020-06-12T02:37:33.072 INFO:tasks.rados.rados.0.smithi107.stdout:560: left oid 9 (ObjNum 559 snap 0 seq_num 559) 2020-06-12T02:37:33.073 INFO:tasks.rados.rados.0.smithi107.stdout:563: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:33.073 INFO:tasks.rados.rados.0.smithi107.stdout:563: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:33.073 INFO:tasks.rados.rados.0.smithi107.stdout:560: done (2 left) 2020-06-12T02:37:33.074 INFO:tasks.rados.rados.0.smithi107.stdout:564: done (1 left) 2020-06-12T02:37:33.074 INFO:tasks.rados.rados.0.smithi107.stdout:565: delete oid 16 current snap is 0 2020-06-12T02:37:33.074 INFO:tasks.rados.rados.0.smithi107.stdout:563: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:33.076 INFO:tasks.rados.rados.0.smithi107.stdout:563: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:33.077 INFO:tasks.rados.rados.0.smithi107.stdout:563: finishing write tid 6 to smithi10729407-5 2020-06-12T02:37:33.077 INFO:tasks.rados.rados.0.smithi107.stdout:565: done (1 left) 2020-06-12T02:37:33.077 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:37:33.077 INFO:tasks.rados.rados.0.smithi107.stdout:566: seq_num 565 ranges {1671168=671744,2342912=729088,3072000=753664,3825664=122880} 2020-06-12T02:37:33.078 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-06-12T02:37:33.091 INFO:tasks.rados.rados.0.smithi107.stdout:566: writing smithi10729407-10 from 1671168 to 2342912 tid 1 2020-06-12T02:37:33.096 INFO:tasks.rados.rados.0.smithi107.stdout:566: writing smithi10729407-10 from 2342912 to 3072000 tid 2 2020-06-12T02:37:33.102 INFO:tasks.rados.rados.0.smithi107.stdout:566: writing smithi10729407-10 from 3072000 to 3825664 tid 3 2020-06-12T02:37:33.103 INFO:tasks.rados.rados.0.smithi107.stdout:566: writing smithi10729407-10 from 3825664 to 3948544 tid 4 2020-06-12T02:37:33.103 INFO:tasks.rados.rados.0.smithi107.stdout:567: delete oid 20 current snap is 0 2020-06-12T02:37:33.104 INFO:tasks.rados.rados.0.smithi107.stdout:563: finishing write tid 7 to smithi10729407-5 2020-06-12T02:37:33.104 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 62 (ObjNum 562 snap 0 seq_num 562) dirty exists 2020-06-12T02:37:33.104 INFO:tasks.rados.rados.0.smithi107.stdout:563: left oid 5 (ObjNum 562 snap 0 seq_num 562) 2020-06-12T02:37:33.104 INFO:tasks.rados.rados.0.smithi107.stdout:566: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:33.105 INFO:tasks.rados.rados.0.smithi107.stdout:566: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:33.105 INFO:tasks.rados.rados.0.smithi107.stdout:563: done (2 left) 2020-06-12T02:37:33.105 INFO:tasks.rados.rados.0.smithi107.stdout:567: done (1 left) 2020-06-12T02:37:33.105 INFO:tasks.rados.rados.0.smithi107.stdout:568: delete oid 16 current snap is 0 2020-06-12T02:37:33.107 INFO:tasks.rados.rados.0.smithi107.stdout:568: done (1 left) 2020-06-12T02:37:33.107 INFO:tasks.rados.rados.0.smithi107.stdout:569: delete oid 15 current snap is 0 2020-06-12T02:37:33.111 INFO:tasks.rados.rados.0.smithi107.stdout:569: done (1 left) 2020-06-12T02:37:33.111 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:37:33.111 INFO:tasks.rados.rados.0.smithi107.stdout:570: seq_num 569 ranges {0=737280,737280=671744,1409024=737280,2146304=196608} 2020-06-12T02:37:33.115 INFO:tasks.rados.rados.0.smithi107.stdout:570: writing smithi10729407-12 from 0 to 737280 tid 1 2020-06-12T02:37:33.119 INFO:tasks.rados.rados.0.smithi107.stdout:570: writing smithi10729407-12 from 737280 to 1409024 tid 2 2020-06-12T02:37:33.124 INFO:tasks.rados.rados.0.smithi107.stdout:570: writing smithi10729407-12 from 1409024 to 2146304 tid 3 2020-06-12T02:37:33.125 INFO:tasks.rados.rados.0.smithi107.stdout:570: writing smithi10729407-12 from 2146304 to 2342912 tid 4 2020-06-12T02:37:33.125 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:37:33.125 INFO:tasks.rados.rados.0.smithi107.stdout:571: seq_num 570 ranges {0=753664,753664=548864,1302528=540672,1843200=516096} 2020-06-12T02:37:33.130 INFO:tasks.rados.rados.0.smithi107.stdout:571: writing smithi10729407-19 from 0 to 753664 tid 1 2020-06-12T02:37:33.133 INFO:tasks.rados.rados.0.smithi107.stdout:571: writing smithi10729407-19 from 753664 to 1302528 tid 2 2020-06-12T02:37:33.137 INFO:tasks.rados.rados.0.smithi107.stdout:571: writing smithi10729407-19 from 1302528 to 1843200 tid 3 2020-06-12T02:37:33.141 INFO:tasks.rados.rados.0.smithi107.stdout:571: writing smithi10729407-19 from 1843200 to 2359296 tid 4 2020-06-12T02:37:33.141 INFO:tasks.rados.rados.0.smithi107.stdout:572: delete oid 6 current snap is 0 2020-06-12T02:37:33.142 INFO:tasks.rados.rados.0.smithi107.stdout:566: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:33.142 INFO:tasks.rados.rados.0.smithi107.stdout:566: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:33.142 INFO:tasks.rados.rados.0.smithi107.stdout:566: finishing write tid 6 to smithi10729407-10 2020-06-12T02:37:33.142 INFO:tasks.rados.rados.0.smithi107.stdout:566: finishing write tid 7 to smithi10729407-10 2020-06-12T02:37:33.142 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 56 (ObjNum 565 snap 0 seq_num 565) dirty exists 2020-06-12T02:37:33.142 INFO:tasks.rados.rados.0.smithi107.stdout:566: left oid 10 (ObjNum 565 snap 0 seq_num 565) 2020-06-12T02:37:33.143 INFO:tasks.rados.rados.0.smithi107.stdout:570: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:33.143 INFO:tasks.rados.rados.0.smithi107.stdout:570: finishing write tid 2 to smithi10729407-12 2020-06-12T02:37:33.143 INFO:tasks.rados.rados.0.smithi107.stdout:570: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:33.145 INFO:tasks.rados.rados.0.smithi107.stdout:570: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:33.145 INFO:tasks.rados.rados.0.smithi107.stdout:570: finishing write tid 6 to smithi10729407-12 2020-06-12T02:37:33.145 INFO:tasks.rados.rados.0.smithi107.stdout:570: finishing write tid 7 to smithi10729407-12 2020-06-12T02:37:33.146 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 89 (ObjNum 569 snap 0 seq_num 569) dirty exists 2020-06-12T02:37:33.146 INFO:tasks.rados.rados.0.smithi107.stdout:570: left oid 12 (ObjNum 569 snap 0 seq_num 569) 2020-06-12T02:37:33.146 INFO:tasks.rados.rados.0.smithi107.stdout:571: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:33.147 INFO:tasks.rados.rados.0.smithi107.stdout:566: done (3 left) 2020-06-12T02:37:33.148 INFO:tasks.rados.rados.0.smithi107.stdout:570: done (2 left) 2020-06-12T02:37:33.148 INFO:tasks.rados.rados.0.smithi107.stdout:572: done (1 left) 2020-06-12T02:37:33.152 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:37:33.152 INFO:tasks.rados.rados.0.smithi107.stdout:573: seq_num 572 ranges {2088960=786432,2875392=770048,3645440=557056,4202496=278528} 2020-06-12T02:37:33.157 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:37:33.154+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:37:33.158 INFO:tasks.rados.rados.0.smithi107.stdout:573: writing smithi10729407-3 from 2088960 to 2875392 tid 1 2020-06-12T02:37:33.164 INFO:tasks.rados.rados.0.smithi107.stdout:573: writing smithi10729407-3 from 2875392 to 3645440 tid 2 2020-06-12T02:37:33.167 INFO:tasks.rados.rados.0.smithi107.stdout:573: writing smithi10729407-3 from 3645440 to 4202496 tid 3 2020-06-12T02:37:33.170 INFO:tasks.rados.rados.0.smithi107.stdout:573: writing smithi10729407-3 from 4202496 to 4481024 tid 4 2020-06-12T02:37:33.170 INFO:tasks.rados.rados.0.smithi107.stdout:574: delete oid 10 current snap is 0 2020-06-12T02:37:33.170 INFO:tasks.rados.rados.0.smithi107.stdout:571: finishing write tid 2 to smithi10729407-19 2020-06-12T02:37:33.170 INFO:tasks.rados.rados.0.smithi107.stdout:571: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:33.171 INFO:tasks.rados.rados.0.smithi107.stdout:571: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:33.171 INFO:tasks.rados.rados.0.smithi107.stdout:571: finishing write tid 6 to smithi10729407-19 2020-06-12T02:37:33.171 INFO:tasks.rados.rados.0.smithi107.stdout:571: finishing write tid 7 to smithi10729407-19 2020-06-12T02:37:33.171 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 94 (ObjNum 570 snap 0 seq_num 570) dirty exists 2020-06-12T02:37:33.172 INFO:tasks.rados.rados.0.smithi107.stdout:571: left oid 19 (ObjNum 570 snap 0 seq_num 570) 2020-06-12T02:37:33.172 INFO:tasks.rados.rados.0.smithi107.stdout:573: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:33.172 INFO:tasks.rados.rados.0.smithi107.stdout:571: done (2 left) 2020-06-12T02:37:33.173 INFO:tasks.rados.rados.0.smithi107.stdout:574: done (1 left) 2020-06-12T02:37:33.173 INFO:tasks.rados.rados.0.smithi107.stdout:575: delete oid 5 current snap is 0 2020-06-12T02:37:33.180 INFO:tasks.rados.rados.0.smithi107.stdout:575: done (1 left) 2020-06-12T02:37:33.181 INFO:tasks.rados.rados.0.smithi107.stdout:576: delete oid 17 current snap is 0 2020-06-12T02:37:33.181 INFO:tasks.rados.rados.0.smithi107.stdout:573: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:33.183 INFO:tasks.rados.rados.0.smithi107.stdout:576: done (1 left) 2020-06-12T02:37:33.183 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:37:33.183 INFO:tasks.rados.rados.0.smithi107.stdout:577: seq_num 576 ranges {0=49152} 2020-06-12T02:37:33.184 INFO:tasks.rados.rados.0.smithi107.stdout:577: writing smithi10729407-20 from 0 to 49152 tid 1 2020-06-12T02:37:33.184 INFO:tasks.rados.rados.0.smithi107.stdout:578: delete oid 7 current snap is 0 2020-06-12T02:37:33.185 INFO:tasks.rados.rados.0.smithi107.stdout:577: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:33.186 INFO:tasks.rados.rados.0.smithi107.stdout:577: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:33.186 INFO:tasks.rados.rados.0.smithi107.stdout:573: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:33.187 INFO:tasks.rados.rados.0.smithi107.stdout:573: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:33.187 INFO:tasks.rados.rados.0.smithi107.stdout:578: done (2 left) 2020-06-12T02:37:33.188 INFO:tasks.rados.rados.0.smithi107.stdout:579: delete oid 11 current snap is 0 2020-06-12T02:37:33.188 INFO:tasks.rados.rados.0.smithi107.stdout:573: finishing write tid 6 to smithi10729407-3 2020-06-12T02:37:33.189 INFO:tasks.rados.rados.0.smithi107.stdout:573: finishing write tid 7 to smithi10729407-3 2020-06-12T02:37:33.189 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 102 (ObjNum 572 snap 0 seq_num 572) dirty exists 2020-06-12T02:37:33.190 INFO:tasks.rados.rados.0.smithi107.stdout:573: left oid 3 (ObjNum 572 snap 0 seq_num 572) 2020-06-12T02:37:33.190 INFO:tasks.rados.rados.0.smithi107.stdout:577: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:33.190 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 52 (ObjNum 576 snap 0 seq_num 576) dirty exists 2020-06-12T02:37:33.190 INFO:tasks.rados.rados.0.smithi107.stdout:577: left oid 20 (ObjNum 576 snap 0 seq_num 576) 2020-06-12T02:37:33.192 INFO:tasks.rados.rados.0.smithi107.stdout:573: done (2 left) 2020-06-12T02:37:33.192 INFO:tasks.rados.rados.0.smithi107.stdout:577: done (1 left) 2020-06-12T02:37:33.192 INFO:tasks.rados.rados.0.smithi107.stdout:579: done (0 left) 2020-06-12T02:37:33.192 INFO:tasks.rados.rados.0.smithi107.stdout:580: delete oid 7 current snap is 0 2020-06-12T02:37:33.195 INFO:tasks.rados.rados.0.smithi107.stdout:580: done (0 left) 2020-06-12T02:37:33.195 INFO:tasks.rados.rados.0.smithi107.stdout:581: delete oid 7 current snap is 0 2020-06-12T02:37:33.197 INFO:tasks.rados.rados.0.smithi107.stdout:581: done (0 left) 2020-06-12T02:37:33.198 INFO:tasks.rados.rados.0.smithi107.stdout:582: delete oid 14 current snap is 0 2020-06-12T02:37:33.204 INFO:tasks.rados.rados.0.smithi107.stdout:582: done (0 left) 2020-06-12T02:37:33.204 INFO:tasks.rados.rados.0.smithi107.stdout:583: delete oid 20 current snap is 0 2020-06-12T02:37:33.205 INFO:tasks.rados.rados.0.smithi107.stdout:583: done (0 left) 2020-06-12T02:37:33.205 INFO:tasks.rados.rados.0.smithi107.stdout:584: delete oid 18 current snap is 0 2020-06-12T02:37:33.205 INFO:tasks.rados.rados.0.smithi107.stdout:584: done (0 left) 2020-06-12T02:37:33.210 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:37:33.210 INFO:tasks.rados.rados.0.smithi107.stdout:585: seq_num 584 ranges {0=188416} 2020-06-12T02:37:33.212 INFO:tasks.rados.rados.0.smithi107.stdout:585: writing smithi10729407-20 from 0 to 188416 tid 1 2020-06-12T02:37:33.212 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:37:33.212 INFO:tasks.rados.rados.0.smithi107.stdout:586: seq_num 585 ranges {0=204800} 2020-06-12T02:37:33.212 INFO:tasks.rados.rados.0.smithi107.stdout:586: writing smithi10729407-18 from 0 to 204800 tid 1 2020-06-12T02:37:33.213 INFO:tasks.rados.rados.0.smithi107.stdout:587: delete oid 10 current snap is 0 2020-06-12T02:37:33.213 INFO:tasks.rados.rados.0.smithi107.stdout:585: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:33.213 INFO:tasks.rados.rados.0.smithi107.stdout:587: done (2 left) 2020-06-12T02:37:33.213 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:37:33.214 INFO:tasks.rados.rados.0.smithi107.stdout:588: seq_num 587 ranges {0=237568} 2020-06-12T02:37:33.214 INFO:tasks.rados.rados.0.smithi107.stdout:588: writing smithi10729407-6 from 0 to 237568 tid 1 2020-06-12T02:37:33.215 INFO:tasks.rados.rados.0.smithi107.stdout:589: delete oid 13 current snap is 0 2020-06-12T02:37:33.215 INFO:tasks.rados.rados.0.smithi107.stdout:586: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:33.215 INFO:tasks.rados.rados.0.smithi107.stdout:586: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:33.215 INFO:tasks.rados.rados.0.smithi107.stdout:586: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:33.215 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 120 (ObjNum 585 snap 0 seq_num 585) dirty exists 2020-06-12T02:37:33.216 INFO:tasks.rados.rados.0.smithi107.stdout:586: left oid 18 (ObjNum 585 snap 0 seq_num 585) 2020-06-12T02:37:33.216 INFO:tasks.rados.rados.0.smithi107.stdout:585: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:33.216 INFO:tasks.rados.rados.0.smithi107.stdout:585: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:33.217 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 55 (ObjNum 584 snap 0 seq_num 584) dirty exists 2020-06-12T02:37:33.217 INFO:tasks.rados.rados.0.smithi107.stdout:585: left oid 20 (ObjNum 584 snap 0 seq_num 584) 2020-06-12T02:37:33.217 INFO:tasks.rados.rados.0.smithi107.stdout:585: done (3 left) 2020-06-12T02:37:33.218 INFO:tasks.rados.rados.0.smithi107.stdout:586: done (2 left) 2020-06-12T02:37:33.218 INFO:tasks.rados.rados.0.smithi107.stdout:589: done (1 left) 2020-06-12T02:37:33.218 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:37:33.218 INFO:tasks.rados.rados.0.smithi107.stdout:590: seq_num 589 ranges {0=270336} 2020-06-12T02:37:33.219 INFO:tasks.rados.rados.0.smithi107.stdout:590: writing smithi10729407-8 from 0 to 270336 tid 1 2020-06-12T02:37:33.219 INFO:tasks.rados.rados.0.smithi107.stdout:591: delete oid 15 current snap is 0 2020-06-12T02:37:33.220 INFO:tasks.rados.rados.0.smithi107.stdout:588: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:33.220 INFO:tasks.rados.rados.0.smithi107.stdout:588: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:33.221 INFO:tasks.rados.rados.0.smithi107.stdout:588: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:33.221 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 134 (ObjNum 587 snap 0 seq_num 587) dirty exists 2020-06-12T02:37:33.221 INFO:tasks.rados.rados.0.smithi107.stdout:588: left oid 6 (ObjNum 587 snap 0 seq_num 587) 2020-06-12T02:37:33.222 INFO:tasks.rados.rados.0.smithi107.stdout:590: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:33.224 INFO:tasks.rados.rados.0.smithi107.stdout:590: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:33.226 INFO:tasks.rados.rados.0.smithi107.stdout:588: done (2 left) 2020-06-12T02:37:33.226 INFO:tasks.rados.rados.0.smithi107.stdout:591: done (1 left) 2020-06-12T02:37:33.227 INFO:tasks.rados.rados.0.smithi107.stdout:592: delete oid 1 current snap is 0 2020-06-12T02:37:33.227 INFO:tasks.rados.rados.0.smithi107.stdout:590: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:33.228 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 112 (ObjNum 589 snap 0 seq_num 589) dirty exists 2020-06-12T02:37:33.228 INFO:tasks.rados.rados.0.smithi107.stdout:590: left oid 8 (ObjNum 589 snap 0 seq_num 589) 2020-06-12T02:37:33.229 INFO:tasks.rados.rados.0.smithi107.stdout:590: done (1 left) 2020-06-12T02:37:33.229 INFO:tasks.rados.rados.0.smithi107.stdout:592: done (0 left) 2020-06-12T02:37:33.229 INFO:tasks.rados.rados.0.smithi107.stdout:593: delete oid 19 current snap is 0 2020-06-12T02:37:33.230 INFO:tasks.rados.rados.0.smithi107.stdout:593: done (0 left) 2020-06-12T02:37:33.230 INFO:tasks.rados.rados.0.smithi107.stdout:594: delete oid 10 current snap is 0 2020-06-12T02:37:33.230 INFO:tasks.rados.rados.0.smithi107.stdout:594: done (0 left) 2020-06-12T02:37:33.231 INFO:tasks.rados.rados.0.smithi107.stdout:595: delete oid 13 current snap is 0 2020-06-12T02:37:33.232 INFO:tasks.rados.rados.0.smithi107.stdout:595: done (0 left) 2020-06-12T02:37:33.232 INFO:tasks.rados.rados.0.smithi107.stdout:596: delete oid 7 current snap is 0 2020-06-12T02:37:33.233 INFO:tasks.rados.rados.0.smithi107.stdout:596: done (0 left) 2020-06-12T02:37:33.234 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:37:33.234 INFO:tasks.rados.rados.0.smithi107.stdout:597: seq_num 596 ranges {0=385024} 2020-06-12T02:37:33.250 INFO:tasks.rados.rados.0.smithi107.stdout:597: writing smithi10729407-5 from 0 to 385024 tid 1 2020-06-12T02:37:33.250 INFO:tasks.rados.rados.0.smithi107.stdout:598: delete oid 7 current snap is 0 2020-06-12T02:37:33.250 INFO:tasks.rados.rados.0.smithi107.stdout:598: done (1 left) 2020-06-12T02:37:33.251 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:37:33.251 INFO:tasks.rados.rados.0.smithi107.stdout:599: seq_num 598 ranges {2170880=417792} 2020-06-12T02:37:33.252 INFO:tasks.rados.rados.0.smithi107.stdout:599: writing smithi10729407-9 from 2170880 to 2588672 tid 1 2020-06-12T02:37:33.252 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:37:33.252 INFO:tasks.rados.rados.0.smithi107.stdout:600: seq_num 599 ranges {0=434176} 2020-06-12T02:37:33.256 INFO:tasks.rados.rados.0.smithi107.stdout:600: writing smithi10729407-4 from 0 to 434176 tid 1 2020-06-12T02:37:33.256 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:37:33.257 INFO:tasks.rados.rados.0.smithi107.stdout:601: seq_num 600 ranges {0=450560} 2020-06-12T02:37:33.259 INFO:tasks.rados.rados.0.smithi107.stdout:601: writing smithi10729407-15 from 0 to 450560 tid 1 2020-06-12T02:37:33.259 INFO:tasks.rados.rados.0.smithi107.stdout:602: delete oid 18 current snap is 0 2020-06-12T02:37:33.260 INFO:tasks.rados.rados.0.smithi107.stdout:597: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:33.260 INFO:tasks.rados.rados.0.smithi107.stdout:597: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:33.260 INFO:tasks.rados.rados.0.smithi107.stdout:597: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:33.260 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 65 (ObjNum 596 snap 0 seq_num 596) dirty exists 2020-06-12T02:37:33.261 INFO:tasks.rados.rados.0.smithi107.stdout:597: left oid 5 (ObjNum 596 snap 0 seq_num 596) 2020-06-12T02:37:33.261 INFO:tasks.rados.rados.0.smithi107.stdout:599: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:33.261 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:37:33.256+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:37:33.263 INFO:tasks.rados.rados.0.smithi107.stdout:600: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:33.264 INFO:tasks.rados.rados.0.smithi107.stdout:600: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:33.264 INFO:tasks.rados.rados.0.smithi107.stdout:599: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:33.264 INFO:tasks.rados.rados.0.smithi107.stdout:600: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:33.265 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 115 (ObjNum 599 snap 0 seq_num 599) dirty exists 2020-06-12T02:37:33.265 INFO:tasks.rados.rados.0.smithi107.stdout:600: left oid 4 (ObjNum 599 snap 0 seq_num 599) 2020-06-12T02:37:33.265 INFO:tasks.rados.rados.0.smithi107.stdout:599: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:33.265 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 122 (ObjNum 598 snap 0 seq_num 598) dirty exists 2020-06-12T02:37:33.269 INFO:tasks.rados.rados.0.smithi107.stdout:599: left oid 9 (ObjNum 598 snap 0 seq_num 598) 2020-06-12T02:37:33.269 INFO:tasks.rados.rados.0.smithi107.stdout:601: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:33.269 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:37:33.269 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2020-06-12T02:37:33.270 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2020-06-12T02:37:33.270 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2020-06-12T02:37:33.270 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.270 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:938 3.2s0 3:4afffc6f:::smithi10729407-20:head [append 0~614400] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:33.271 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:32.167101+0000", 2020-06-12T02:37:33.271 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 1.0903914939999999, 2020-06-12T02:37:33.271 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013914031, 2020-06-12T02:37:33.273 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:33.273 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:33.273 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:33.273 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:33.273 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:33.274 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 938 2020-06-12T02:37:33.274 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.274 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:33.274 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.275 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:33.275 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.167101+0000", 2020-06-12T02:37:33.276 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:33.276 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.276 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.276 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:33.276 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.167101+0000", 2020-06-12T02:37:33.277 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037306900000000002 2020-06-12T02:37:33.277 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.277 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.277 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:33.277 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.167474+0000", 2020-06-12T02:37:33.277 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4400000000000005e-06 2020-06-12T02:37:33.278 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.278 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.278 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:33.278 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.167483+0000", 2020-06-12T02:37:33.278 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4449e-05 2020-06-12T02:37:33.279 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.279 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.279 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:33.279 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.167497+0000", 2020-06-12T02:37:33.279 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4960000000000001e-06 2020-06-12T02:37:33.280 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.280 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.280 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:33.280 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.167504+0000", 2020-06-12T02:37:33.280 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059738200000000003 2020-06-12T02:37:33.281 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.281 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.281 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:33.281 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.168101+0000", 2020-06-12T02:37:33.282 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012392800000000001 2020-06-12T02:37:33.283 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.283 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.285 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:33.285 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.168225+0000", 2020-06-12T02:37:33.286 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00089649499999999997 2020-06-12T02:37:33.286 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.286 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.286 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:33.287 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.169122+0000", 2020-06-12T02:37:33.287 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011682733000000001 2020-06-12T02:37:33.287 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.287 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.287 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:33.288 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.180804+0000", 2020-06-12T02:37:33.288 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012835900000000001 2020-06-12T02:37:33.288 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.288 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.288 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:33.289 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.180933+0000", 2020-06-12T02:37:33.289 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2680000000000001e-05 2020-06-12T02:37:33.289 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.289 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.289 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:33.289 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.181015+0000", 2020-06-12T02:37:33.290 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013914031 2020-06-12T02:37:33.290 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:33.290 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:33.291 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:33.292 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.292 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.292 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:962 3.7s0 3:ea271007:::smithi10729407-9:head [append 0~598016] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:33.292 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:32.236108+0000", 2020-06-12T02:37:33.292 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 1.0213846090000001, 2020-06-12T02:37:33.293 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018271632999999999, 2020-06-12T02:37:33.293 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:33.293 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:33.293 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:33.293 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:33.293 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:33.294 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 962 2020-06-12T02:37:33.294 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.294 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:33.294 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.294 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:33.295 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.236108+0000", 2020-06-12T02:37:33.295 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:33.295 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.295 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.295 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:33.296 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.236108+0000", 2020-06-12T02:37:33.296 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019838700000000001 2020-06-12T02:37:33.296 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.296 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.297 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:33.297 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.236307+0000", 2020-06-12T02:37:33.297 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4619999999999996e-06 2020-06-12T02:37:33.297 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.298 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.298 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:33.298 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.236315+0000", 2020-06-12T02:37:33.298 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1665999999999999e-05 2020-06-12T02:37:33.298 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.299 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.299 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:33.299 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.236337+0000", 2020-06-12T02:37:33.299 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5630000000000003e-06 2020-06-12T02:37:33.299 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.299 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.300 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:33.300 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.236344+0000", 2020-06-12T02:37:33.300 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023269440000000001 2020-06-12T02:37:33.300 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.300 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.301 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:33.301 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.238671+0000", 2020-06-12T02:37:33.301 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5969000000000003e-05 2020-06-12T02:37:33.301 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.301 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.302 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:33.302 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.238757+0000", 2020-06-12T02:37:33.302 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046101010000000001 2020-06-12T02:37:33.302 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.302 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.302 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:33.303 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.243367+0000", 2020-06-12T02:37:33.303 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7991000000000005e-05 2020-06-12T02:37:33.303 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.303 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.303 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:33.304 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.243455+0000", 2020-06-12T02:37:33.304 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00062561500000000005 2020-06-12T02:37:33.304 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.304 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.304 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:33.305 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.244081+0000", 2020-06-12T02:37:33.305 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010040778 2020-06-12T02:37:33.305 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.305 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.305 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:33.305 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.254122+0000", 2020-06-12T02:37:33.306 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010453100000000001 2020-06-12T02:37:33.306 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.306 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.306 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:33.306 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.254226+0000", 2020-06-12T02:37:33.307 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000153626 2020-06-12T02:37:33.307 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.307 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.307 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:33.307 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.254380+0000", 2020-06-12T02:37:33.307 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018271632999999999 2020-06-12T02:37:33.308 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:33.308 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:33.308 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:33.308 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.308 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.309 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:963 3.7s0 3:ea271007:::smithi10729407-9:head [append 0~606208] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:33.309 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:32.239688+0000", 2020-06-12T02:37:33.309 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 1.017804285, 2020-06-12T02:37:33.309 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016181383000000001, 2020-06-12T02:37:33.309 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:33.309 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:33.310 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:33.310 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:33.310 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:33.310 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 963 2020-06-12T02:37:33.310 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.311 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:33.311 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.311 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:33.311 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.239688+0000", 2020-06-12T02:37:33.311 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:33.311 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.312 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.312 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:33.312 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.239688+0000", 2020-06-12T02:37:33.312 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00022408000000000001 2020-06-12T02:37:33.312 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.313 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.313 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:33.313 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.239913+0000", 2020-06-12T02:37:33.313 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0399999999999993e-06 2020-06-12T02:37:33.313 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.314 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.314 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:33.314 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.239921+0000", 2020-06-12T02:37:33.314 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7499999999999998e-05 2020-06-12T02:37:33.314 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.314 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.315 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:33.315 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.239938+0000", 2020-06-12T02:37:33.315 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0739999999999998e-06 2020-06-12T02:37:33.315 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.315 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.316 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:33.316 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.239944+0000", 2020-06-12T02:37:33.316 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2867000000000002e-05 2020-06-12T02:37:33.316 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.316 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.316 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:33.317 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.239977+0000", 2020-06-12T02:37:33.317 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4425000000000007e-05 2020-06-12T02:37:33.317 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.317 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.317 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:33.318 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.240051+0000", 2020-06-12T02:37:33.318 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0064351139999999996 2020-06-12T02:37:33.318 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.318 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.319 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:33.319 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.246487+0000", 2020-06-12T02:37:33.319 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012310899999999999 2020-06-12T02:37:33.319 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.319 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.319 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:33.320 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.246610+0000", 2020-06-12T02:37:33.320 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00079552599999999996 2020-06-12T02:37:33.320 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.320 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.320 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:33.321 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.247405+0000", 2020-06-12T02:37:33.321 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0079562030000000002 2020-06-12T02:37:33.321 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.321 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.321 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:33.322 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.255361+0000", 2020-06-12T02:37:33.322 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036610400000000002 2020-06-12T02:37:33.322 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.322 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.322 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:33.322 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.255728+0000", 2020-06-12T02:37:33.323 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014234100000000001 2020-06-12T02:37:33.323 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.323 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.323 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:33.323 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.255870+0000", 2020-06-12T02:37:33.324 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016181383000000001 2020-06-12T02:37:33.324 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:33.324 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:33.324 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:33.324 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.324 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.325 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:964 3.7s0 3:ea271007:::smithi10729407-9:head [append 0~401408] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:33.325 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:32.242072+0000", 2020-06-12T02:37:33.325 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 1.015421256, 2020-06-12T02:37:33.325 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014317890999999999, 2020-06-12T02:37:33.325 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:33.326 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:33.326 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:33.326 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:33.326 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:33.326 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 964 2020-06-12T02:37:33.326 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.327 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:33.327 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.327 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:33.327 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242072+0000", 2020-06-12T02:37:33.327 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:33.328 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.328 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.328 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:33.328 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242072+0000", 2020-06-12T02:37:33.328 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000181743 2020-06-12T02:37:33.329 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.329 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.329 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:33.329 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242253+0000", 2020-06-12T02:37:33.329 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0350000000000001e-06 2020-06-12T02:37:33.330 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.330 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.331 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:33.331 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242261+0000", 2020-06-12T02:37:33.331 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6348999999999999e-05 2020-06-12T02:37:33.332 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.332 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.332 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:33.332 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242278+0000", 2020-06-12T02:37:33.332 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6979999999999998e-06 2020-06-12T02:37:33.333 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.333 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.333 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:33.333 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242285+0000", 2020-06-12T02:37:33.333 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5176000000000003e-05 2020-06-12T02:37:33.334 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.334 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.334 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:33.334 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242321+0000", 2020-06-12T02:37:33.335 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103311 2020-06-12T02:37:33.335 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.335 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.335 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:33.336 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242424+0000", 2020-06-12T02:37:33.336 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0087135170000000005 2020-06-12T02:37:33.336 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.336 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.336 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:33.337 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.251137+0000", 2020-06-12T02:37:33.337 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010788700000000001 2020-06-12T02:37:33.337 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.337 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.337 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:33.338 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.251245+0000", 2020-06-12T02:37:33.338 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056026800000000005 2020-06-12T02:37:33.338 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.338 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.338 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:33.339 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.251805+0000", 2020-06-12T02:37:33.339 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0035584219999999999 2020-06-12T02:37:33.339 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.340 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.340 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:33.340 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.255364+0000", 2020-06-12T02:37:33.340 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00085282999999999995 2020-06-12T02:37:33.340 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.341 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.341 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:33.341 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.256217+0000", 2020-06-12T02:37:33.341 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017265499999999999 2020-06-12T02:37:33.341 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.342 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.342 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:33.342 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.256389+0000", 2020-06-12T02:37:33.342 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014317890999999999 2020-06-12T02:37:33.342 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:33.343 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:33.343 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:33.343 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.344 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.344 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:965 3.7s0 3:ea271007:::smithi10729407-9:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:33.345 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:32.242343+0000", 2020-06-12T02:37:33.345 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 1.015149423, 2020-06-12T02:37:33.345 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014576832, 2020-06-12T02:37:33.345 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:33.346 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:33.346 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:33.346 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:33.346 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:33.346 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 965 2020-06-12T02:37:33.346 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.347 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:33.347 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.347 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:33.347 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242343+0000", 2020-06-12T02:37:33.347 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:33.348 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.348 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.348 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:33.348 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242343+0000", 2020-06-12T02:37:33.348 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5201e-05 2020-06-12T02:37:33.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.349 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.350 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:33.350 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242419+0000", 2020-06-12T02:37:33.350 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3489999999999997e-06 2020-06-12T02:37:33.350 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.351 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.351 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:33.351 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242425+0000", 2020-06-12T02:37:33.351 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1494999999999999e-05 2020-06-12T02:37:33.351 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.352 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.352 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:33.352 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242436+0000", 2020-06-12T02:37:33.352 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7410000000000004e-06 2020-06-12T02:37:33.352 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.353 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.353 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:33.353 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242446+0000", 2020-06-12T02:37:33.353 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2800999999999999e-05 2020-06-12T02:37:33.354 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.354 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.354 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:33.354 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242489+0000", 2020-06-12T02:37:33.354 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7392999999999996e-05 2020-06-12T02:37:33.355 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.355 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.355 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:33.355 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242566+0000", 2020-06-12T02:37:33.355 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011936647 2020-06-12T02:37:33.356 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.356 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.356 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:33.356 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.254503+0000", 2020-06-12T02:37:33.356 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114613 2020-06-12T02:37:33.357 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.357 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.357 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:33.357 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.254618+0000", 2020-06-12T02:37:33.357 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038691300000000001 2020-06-12T02:37:33.358 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.358 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.358 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:33.358 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.255004+0000", 2020-06-12T02:37:33.359 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014359850000000001 2020-06-12T02:37:33.359 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.359 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.359 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:33.359 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.256440+0000", 2020-06-12T02:37:33.360 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1623000000000001e-05 2020-06-12T02:37:33.360 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.360 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.360 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:33.360 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.256522+0000", 2020-06-12T02:37:33.361 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000398071 2020-06-12T02:37:33.361 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.361 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.361 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:33.361 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.256920+0000", 2020-06-12T02:37:33.362 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014576832 2020-06-12T02:37:33.362 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:33.362 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:33.362 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:33.362 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.363 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.363 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:966 3.7s0 3:ea271007:::smithi10729407-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:33.363 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:32.242493+0000", 2020-06-12T02:37:33.363 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 1.0149993740000001, 2020-06-12T02:37:33.363 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015989771, 2020-06-12T02:37:33.364 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:33.364 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:37:33.364 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:33.364 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:33.364 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:33.365 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 966 2020-06-12T02:37:33.365 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.365 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:33.365 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.365 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:33.366 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242493+0000", 2020-06-12T02:37:33.366 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:33.366 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.366 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.366 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:33.367 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242493+0000", 2020-06-12T02:37:33.367 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6276000000000002e-05 2020-06-12T02:37:33.367 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.367 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.368 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:33.368 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242540+0000", 2020-06-12T02:37:33.368 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4710000000000004e-06 2020-06-12T02:37:33.368 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.368 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.369 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:33.369 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242546+0000", 2020-06-12T02:37:33.369 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3157e-05 2020-06-12T02:37:33.370 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.370 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.370 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:33.371 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242559+0000", 2020-06-12T02:37:33.371 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7419999999999997e-06 2020-06-12T02:37:33.371 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.371 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.371 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:33.372 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242564+0000", 2020-06-12T02:37:33.372 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8620999999999999e-05 2020-06-12T02:37:33.372 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.372 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.372 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:33.373 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242593+0000", 2020-06-12T02:37:33.373 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0400000000000003e-05 2020-06-12T02:37:33.373 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.373 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.373 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:33.374 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242673+0000", 2020-06-12T02:37:33.374 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01258803 2020-06-12T02:37:33.374 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.374 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.375 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:33.375 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.255261+0000", 2020-06-12T02:37:33.375 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5599000000000004e-05 2020-06-12T02:37:33.375 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.375 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.376 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:33.376 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.255357+0000", 2020-06-12T02:37:33.376 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001947999 2020-06-12T02:37:33.376 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.376 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.377 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:33.377 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.257305+0000", 2020-06-12T02:37:33.377 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5921999999999994e-05 2020-06-12T02:37:33.377 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.377 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.378 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:33.378 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.257401+0000", 2020-06-12T02:37:33.378 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0010825539999999999 2020-06-12T02:37:33.378 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.379 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.379 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:33.379 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.258483+0000", 2020-06-12T02:37:33.379 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015989771 2020-06-12T02:37:33.379 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:33.380 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:33.380 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:33.380 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.380 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.381 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1050 3.7s0 3:e653d00f:::smithi10729407-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:33.381 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:32.464804+0000", 2020-06-12T02:37:33.381 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.79268921800000003, 2020-06-12T02:37:33.381 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01302733, 2020-06-12T02:37:33.381 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:33.382 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:37:33.382 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:33.382 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:33.382 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:33.382 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1050 2020-06-12T02:37:33.382 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.383 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:33.383 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.383 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:33.383 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.464804+0000", 2020-06-12T02:37:33.383 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:33.384 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.384 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.384 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:33.384 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.464804+0000", 2020-06-12T02:37:33.384 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3192000000000001e-05 2020-06-12T02:37:33.385 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.385 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.385 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:33.386 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.464827+0000", 2020-06-12T02:37:33.386 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.439e-06 2020-06-12T02:37:33.386 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.386 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.386 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:33.387 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.464830+0000", 2020-06-12T02:37:33.387 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7130000000000005e-06 2020-06-12T02:37:33.387 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.387 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.387 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:33.387 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.464838+0000", 2020-06-12T02:37:33.388 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.7259999999999998e-06 2020-06-12T02:37:33.388 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.388 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.388 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:33.388 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.464841+0000", 2020-06-12T02:37:33.389 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0034680309999999999 2020-06-12T02:37:33.389 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.389 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.389 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:33.389 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.468309+0000", 2020-06-12T02:37:33.389 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4370999999999997e-05 2020-06-12T02:37:33.390 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.390 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.390 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:33.390 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.468373+0000", 2020-06-12T02:37:33.391 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00162874 2020-06-12T02:37:33.391 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.391 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.391 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:33.391 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.470002+0000", 2020-06-12T02:37:33.392 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0284000000000006e-05 2020-06-12T02:37:33.392 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.392 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.392 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:33.392 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.470082+0000", 2020-06-12T02:37:33.393 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0050541960000000004 2020-06-12T02:37:33.393 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.393 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.393 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:33.393 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.475136+0000", 2020-06-12T02:37:33.393 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5369000000000002e-05 2020-06-12T02:37:33.394 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.394 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.394 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:33.394 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.475222+0000", 2020-06-12T02:37:33.394 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002609269 2020-06-12T02:37:33.395 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.395 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.395 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:33.395 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.477831+0000", 2020-06-12T02:37:33.395 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01302733 2020-06-12T02:37:33.396 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:33.396 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:33.396 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:33.396 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.396 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.396 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1054 3.bs0 3:d09543d2:::smithi10729407-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:33.397 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:32.475391+0000", 2020-06-12T02:37:33.397 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.78210205700000002, 2020-06-12T02:37:33.397 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013587917, 2020-06-12T02:37:33.397 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:33.397 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:37:33.398 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:33.398 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:33.398 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:33.398 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1054 2020-06-12T02:37:33.398 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.399 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:33.399 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.399 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:33.399 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.475391+0000", 2020-06-12T02:37:33.399 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:33.400 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.400 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.400 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:33.400 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.475391+0000", 2020-06-12T02:37:33.401 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4325e-05 2020-06-12T02:37:33.401 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.401 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.401 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:33.407 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.475435+0000", 2020-06-12T02:37:33.407 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9750000000000001e-06 2020-06-12T02:37:33.407 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.407 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.408 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:33.408 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.475442+0000", 2020-06-12T02:37:33.409 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2130000000000001e-05 2020-06-12T02:37:33.409 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.409 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.409 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:33.409 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.475454+0000", 2020-06-12T02:37:33.410 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5360000000000003e-06 2020-06-12T02:37:33.410 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.410 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.410 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:33.410 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.475459+0000", 2020-06-12T02:37:33.410 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0037654440000000002 2020-06-12T02:37:33.411 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.411 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.411 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:33.411 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.479224+0000", 2020-06-12T02:37:33.411 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011025499999999999 2020-06-12T02:37:33.412 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.412 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.412 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:33.412 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.479334+0000", 2020-06-12T02:37:33.412 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0039899269999999999 2020-06-12T02:37:33.413 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.413 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.413 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:33.413 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.483324+0000", 2020-06-12T02:37:33.413 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5917999999999999e-05 2020-06-12T02:37:33.413 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.414 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.414 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:33.414 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.483420+0000", 2020-06-12T02:37:33.414 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025636420000000001 2020-06-12T02:37:33.414 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.415 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.415 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:33.415 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.485984+0000", 2020-06-12T02:37:33.415 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2545000000000003e-05 2020-06-12T02:37:33.415 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.416 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.416 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:33.416 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.486066+0000", 2020-06-12T02:37:33.416 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0029122200000000001 2020-06-12T02:37:33.416 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.416 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.417 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:33.417 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.488979+0000", 2020-06-12T02:37:33.417 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013587917 2020-06-12T02:37:33.417 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:33.417 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:33.418 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:33.418 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.418 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.419 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1196 3.2s0 3:4afffc6f:::smithi10729407-20:head [append 0~573440] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:33.419 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:32.832608+0000", 2020-06-12T02:37:33.419 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.42488502099999997, 2020-06-12T02:37:33.419 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013278411, 2020-06-12T02:37:33.420 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:33.420 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:33.420 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:33.420 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:33.420 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:33.420 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1196 2020-06-12T02:37:33.421 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.421 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:33.421 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.421 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:33.422 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.832608+0000", 2020-06-12T02:37:33.422 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:33.422 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.422 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.422 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:33.423 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.832608+0000", 2020-06-12T02:37:33.423 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014747499999999999 2020-06-12T02:37:33.423 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.423 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.423 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:33.423 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.832755+0000", 2020-06-12T02:37:33.424 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8999999999999997e-06 2020-06-12T02:37:33.424 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.424 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.424 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:33.424 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.832760+0000", 2020-06-12T02:37:33.425 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1689999999999992e-06 2020-06-12T02:37:33.425 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.425 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.425 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:33.425 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.832768+0000", 2020-06-12T02:37:33.426 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1180000000000002e-06 2020-06-12T02:37:33.426 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.426 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.426 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:33.426 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.832772+0000", 2020-06-12T02:37:33.427 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013293109999999999 2020-06-12T02:37:33.427 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.427 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.427 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:33.427 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.834102+0000", 2020-06-12T02:37:33.427 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011059 2020-06-12T02:37:33.428 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.428 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.428 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:33.428 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.834212+0000", 2020-06-12T02:37:33.429 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073975099999999997 2020-06-12T02:37:33.429 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.429 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.430 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:33.430 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.834952+0000", 2020-06-12T02:37:33.430 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5048000000000003e-05 2020-06-12T02:37:33.430 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.430 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.430 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:33.431 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.835037+0000", 2020-06-12T02:37:33.431 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000754406 2020-06-12T02:37:33.431 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.431 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.431 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:33.432 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.835792+0000", 2020-06-12T02:37:33.432 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0098839050000000001 2020-06-12T02:37:33.432 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.432 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.432 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:33.433 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.845675+0000", 2020-06-12T02:37:33.433 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000133616 2020-06-12T02:37:33.433 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.433 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.433 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:33.434 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.845809+0000", 2020-06-12T02:37:33.434 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7121999999999998e-05 2020-06-12T02:37:33.434 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.435 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.435 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:33.435 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.845886+0000", 2020-06-12T02:37:33.435 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013278411 2020-06-12T02:37:33.435 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:33.436 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:33.436 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:33.436 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.436 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.436 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1199 3.2s0 3:4afffc6f:::smithi10729407-20:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:33.437 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:32.836999+0000", 2020-06-12T02:37:33.437 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.42049423600000002, 2020-06-12T02:37:33.437 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013240801, 2020-06-12T02:37:33.437 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:33.437 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:33.437 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:33.438 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:33.438 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:33.438 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1199 2020-06-12T02:37:33.438 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.438 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:33.439 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.439 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:33.439 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.836999+0000", 2020-06-12T02:37:33.440 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:33.440 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.440 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.440 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:33.441 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.836999+0000", 2020-06-12T02:37:33.441 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6937000000000002e-05 2020-06-12T02:37:33.441 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.442 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.442 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:33.442 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.837035+0000", 2020-06-12T02:37:33.442 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3459999999999998e-06 2020-06-12T02:37:33.443 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.443 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.443 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:33.443 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.837039+0000", 2020-06-12T02:37:33.443 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.682e-06 2020-06-12T02:37:33.444 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.444 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.444 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:33.444 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.837044+0000", 2020-06-12T02:37:33.444 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3410000000000001e-06 2020-06-12T02:37:33.445 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.445 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.445 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:33.445 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.837047+0000", 2020-06-12T02:37:33.445 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0077791300000000004 2020-06-12T02:37:33.446 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.446 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.446 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:33.446 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.844826+0000", 2020-06-12T02:37:33.446 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010998299999999999 2020-06-12T02:37:33.446 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.447 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.447 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:33.447 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.844936+0000", 2020-06-12T02:37:33.447 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00053193000000000001 2020-06-12T02:37:33.447 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.448 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.448 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:33.448 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.845468+0000", 2020-06-12T02:37:33.448 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00071538899999999998 2020-06-12T02:37:33.448 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.449 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.449 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:33.449 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.846183+0000", 2020-06-12T02:37:33.450 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0039640889999999996 2020-06-12T02:37:33.450 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.450 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.450 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:33.450 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.850147+0000", 2020-06-12T02:37:33.451 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1973999999999997e-05 2020-06-12T02:37:33.451 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.451 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.451 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:33.451 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.850239+0000", 2020-06-12T02:37:33.452 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013240801 2020-06-12T02:37:33.452 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:33.452 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:33.452 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:33.453 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.453 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.453 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1200 3.2s0 3:4afffc6f:::smithi10729407-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:33.453 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:32.837068+0000", 2020-06-12T02:37:33.453 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.42042475699999998, 2020-06-12T02:37:33.454 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014863888, 2020-06-12T02:37:33.454 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:33.454 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:37:33.454 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:33.454 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:33.454 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:33.455 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1200 2020-06-12T02:37:33.455 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.455 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:33.455 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.455 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:33.456 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.837068+0000", 2020-06-12T02:37:33.456 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:33.456 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.456 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.456 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:33.457 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.837068+0000", 2020-06-12T02:37:33.457 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2849000000000001e-05 2020-06-12T02:37:33.457 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.457 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.457 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:33.458 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.837091+0000", 2020-06-12T02:37:33.458 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0290000000000001e-06 2020-06-12T02:37:33.458 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.458 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.459 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:33.459 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.837094+0000", 2020-06-12T02:37:33.459 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1800000000000001e-06 2020-06-12T02:37:33.460 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.460 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.460 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:33.461 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.837100+0000", 2020-06-12T02:37:33.461 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2120000000000002e-06 2020-06-12T02:37:33.461 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.461 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.462 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:33.462 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.837102+0000", 2020-06-12T02:37:33.462 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090177509999999992 2020-06-12T02:37:33.462 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.462 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.463 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:33.463 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.846120+0000", 2020-06-12T02:37:33.463 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8610000000000003e-05 2020-06-12T02:37:33.463 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.463 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.464 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:33.464 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.846179+0000", 2020-06-12T02:37:33.464 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045638520000000002 2020-06-12T02:37:33.464 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.464 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.465 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:33.465 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.850742+0000", 2020-06-12T02:37:33.465 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0758000000000002e-05 2020-06-12T02:37:33.465 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.465 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.465 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:33.466 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.850823+0000", 2020-06-12T02:37:33.466 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011086469999999999 2020-06-12T02:37:33.466 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.466 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.466 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:33.467 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.851932+0000", 2020-06-12T02:37:33.467 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014863888 2020-06-12T02:37:33.467 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:33.467 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:33.467 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:33.468 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.468 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.468 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1246 3.bs0 3:d0ebae7d:::smithi10729407-3:head [append 0~704512] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:33.468 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:32.982196+0000", 2020-06-12T02:37:33.468 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.27529693500000002, 2020-06-12T02:37:33.468 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014304037, 2020-06-12T02:37:33.469 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:33.469 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:33.469 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:33.469 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:33.469 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:33.470 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1246 2020-06-12T02:37:33.470 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.471 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:33.471 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.471 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:33.471 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.982196+0000", 2020-06-12T02:37:33.471 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:33.472 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.472 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.472 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:33.472 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.982196+0000", 2020-06-12T02:37:33.472 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00023946899999999999 2020-06-12T02:37:33.473 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.473 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.473 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:33.473 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.982435+0000", 2020-06-12T02:37:33.473 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7560000000000005e-06 2020-06-12T02:37:33.474 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.474 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.474 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:33.474 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.982444+0000", 2020-06-12T02:37:33.474 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7720999999999999e-05 2020-06-12T02:37:33.475 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.475 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.475 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:33.475 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.982462+0000", 2020-06-12T02:37:33.475 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.841e-06 2020-06-12T02:37:33.476 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.476 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.476 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:33.476 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.982470+0000", 2020-06-12T02:37:33.476 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3967e-05 2020-06-12T02:37:33.477 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.477 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.477 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:33.477 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.982524+0000", 2020-06-12T02:37:33.478 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012391 2020-06-12T02:37:33.478 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.478 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.478 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:33.478 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.982647+0000", 2020-06-12T02:37:33.479 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001367037 2020-06-12T02:37:33.479 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.479 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.479 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:33.479 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.984015+0000", 2020-06-12T02:37:33.479 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011574300000000001 2020-06-12T02:37:33.480 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.480 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.480 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:33.481 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.984130+0000", 2020-06-12T02:37:33.481 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072667700000000001 2020-06-12T02:37:33.481 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.481 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.481 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:33.482 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.984857+0000", 2020-06-12T02:37:33.482 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.009453543 2020-06-12T02:37:33.482 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.482 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.482 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:33.483 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.994310+0000", 2020-06-12T02:37:33.483 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0020943250000000002 2020-06-12T02:37:33.483 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.483 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.483 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:33.484 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.996405+0000", 2020-06-12T02:37:33.484 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5048000000000002e-05 2020-06-12T02:37:33.484 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.484 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.484 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:33.484 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.996500+0000", 2020-06-12T02:37:33.485 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014304037 2020-06-12T02:37:33.485 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:33.485 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:33.485 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:33.485 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.486 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.486 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1252 3.8s0 3:14244450:::smithi10729407-2:head [append 0~516096] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:33.486 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:32.999793+0000", 2020-06-12T02:37:33.486 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.257700029, 2020-06-12T02:37:33.486 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017380335, 2020-06-12T02:37:33.487 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:33.487 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:33.487 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:33.487 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:33.487 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:33.488 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1252 2020-06-12T02:37:33.488 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.489 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:33.489 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.489 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:33.489 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.999793+0000", 2020-06-12T02:37:33.489 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:33.490 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.490 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.490 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:33.490 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.999793+0000", 2020-06-12T02:37:33.490 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000208051 2020-06-12T02:37:33.491 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.491 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.491 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:33.491 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.000001+0000", 2020-06-12T02:37:33.492 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9750000000000002e-06 2020-06-12T02:37:33.492 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.492 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.492 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:33.492 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.000011+0000", 2020-06-12T02:37:33.493 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7733000000000001e-05 2020-06-12T02:37:33.493 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.493 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.493 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:33.493 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.000028+0000", 2020-06-12T02:37:33.494 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7379999999999994e-06 2020-06-12T02:37:33.494 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.494 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.494 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:33.494 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.000036+0000", 2020-06-12T02:37:33.494 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.414e-05 2020-06-12T02:37:33.495 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.495 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.495 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:33.495 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.000090+0000", 2020-06-12T02:37:33.495 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012561699999999999 2020-06-12T02:37:33.496 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.496 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.496 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:33.496 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.000216+0000", 2020-06-12T02:37:33.496 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00079436199999999995 2020-06-12T02:37:33.497 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.497 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.497 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:33.497 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.001010+0000", 2020-06-12T02:37:33.497 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015996506000000001 2020-06-12T02:37:33.497 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.498 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.498 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:33.498 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.017007+0000", 2020-06-12T02:37:33.498 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0599999999999997e-05 2020-06-12T02:37:33.498 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.499 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.499 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:33.499 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.017057+0000", 2020-06-12T02:37:33.499 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011561300000000001 2020-06-12T02:37:33.499 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.499 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.500 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:33.500 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.017173+0000", 2020-06-12T02:37:33.500 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017380335 2020-06-12T02:37:33.500 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:33.500 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:33.501 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:33.501 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.501 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.501 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1253 3.8s0 3:14244450:::smithi10729407-2:head [append 0~524288] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:33.502 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.002800+0000", 2020-06-12T02:37:33.502 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.25469234099999999, 2020-06-12T02:37:33.502 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014566312999999999, 2020-06-12T02:37:33.502 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:33.502 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:33.503 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:33.503 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:33.503 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:33.503 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1253 2020-06-12T02:37:33.503 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.504 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:33.504 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.504 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:33.504 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.002800+0000", 2020-06-12T02:37:33.504 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:33.505 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.505 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.505 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:33.505 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.002800+0000", 2020-06-12T02:37:33.505 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00020157899999999999 2020-06-12T02:37:33.505 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.506 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.506 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:33.506 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.003002+0000", 2020-06-12T02:37:33.506 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6059999999999992e-06 2020-06-12T02:37:33.506 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.507 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.507 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:33.507 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.003012+0000", 2020-06-12T02:37:33.507 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5407e-05 2020-06-12T02:37:33.507 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.508 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.508 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:33.508 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.003027+0000", 2020-06-12T02:37:33.508 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.391e-06 2020-06-12T02:37:33.508 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.508 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.509 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:33.509 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.003034+0000", 2020-06-12T02:37:33.509 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0010026569999999999 2020-06-12T02:37:33.509 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.509 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.510 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:33.510 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.004037+0000", 2020-06-12T02:37:33.510 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4276000000000003e-05 2020-06-12T02:37:33.510 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.510 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.510 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:33.511 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.004131+0000", 2020-06-12T02:37:33.511 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00068651200000000002 2020-06-12T02:37:33.511 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.512 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.512 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:33.512 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.004818+0000", 2020-06-12T02:37:33.512 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012360715 2020-06-12T02:37:33.513 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.513 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.513 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:33.513 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.017179+0000", 2020-06-12T02:37:33.514 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6847000000000001e-05 2020-06-12T02:37:33.514 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.514 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.514 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:33.514 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.017225+0000", 2020-06-12T02:37:33.515 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000141323 2020-06-12T02:37:33.515 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.515 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.515 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:33.515 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.017367+0000", 2020-06-12T02:37:33.516 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014566312999999999 2020-06-12T02:37:33.516 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:33.516 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:33.516 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:33.516 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.516 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.517 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1261 3.7s0 3:ea271007:::smithi10729407-9:head [append 0~573440] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:33.517 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.021656+0000", 2020-06-12T02:37:33.517 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.23583643400000001, 2020-06-12T02:37:33.517 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013739365, 2020-06-12T02:37:33.517 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:33.518 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:33.518 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:33.518 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:33.518 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:33.518 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1261 2020-06-12T02:37:33.518 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.519 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:33.519 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.519 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:33.519 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.021656+0000", 2020-06-12T02:37:33.519 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:33.520 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.520 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.520 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:33.520 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.021656+0000", 2020-06-12T02:37:33.521 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000203637 2020-06-12T02:37:33.522 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.522 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.522 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:33.522 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.021860+0000", 2020-06-12T02:37:33.522 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1979e-05 2020-06-12T02:37:33.523 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.523 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.523 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:33.523 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.021872+0000", 2020-06-12T02:37:33.523 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5968e-05 2020-06-12T02:37:33.523 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.524 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.524 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:33.524 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.021888+0000", 2020-06-12T02:37:33.524 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8850000000000005e-06 2020-06-12T02:37:33.525 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.525 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.525 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:33.525 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.021897+0000", 2020-06-12T02:37:33.525 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072322 2020-06-12T02:37:33.525 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.526 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.526 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:33.526 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.022620+0000", 2020-06-12T02:37:33.526 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112147 2020-06-12T02:37:33.526 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.527 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.527 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:33.527 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.022732+0000", 2020-06-12T02:37:33.527 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049434200000000005 2020-06-12T02:37:33.527 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.528 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.528 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:33.529 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.023227+0000", 2020-06-12T02:37:33.529 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011917870000000001 2020-06-12T02:37:33.529 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.529 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.529 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:33.530 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.035144+0000", 2020-06-12T02:37:33.530 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0388000000000003e-05 2020-06-12T02:37:33.530 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.530 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.530 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:33.530 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.035235+0000", 2020-06-12T02:37:33.531 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000160929 2020-06-12T02:37:33.531 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.531 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.531 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:33.531 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.035396+0000", 2020-06-12T02:37:33.532 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013739365 2020-06-12T02:37:33.532 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:33.532 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:33.532 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:33.532 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.532 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.533 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1299 3.bs0 3:d0ebae7d:::smithi10729407-3:head [append 0~770048] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:33.533 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.162816+0000", 2020-06-12T02:37:33.533 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.094676890999999999, 2020-06-12T02:37:33.533 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016787209000000001, 2020-06-12T02:37:33.533 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:33.534 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:33.534 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:33.534 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:33.534 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:33.535 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1299 2020-06-12T02:37:33.535 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.535 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:33.536 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.536 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:33.536 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.162816+0000", 2020-06-12T02:37:33.536 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:33.537 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.537 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.537 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:33.539 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.162816+0000", 2020-06-12T02:37:33.539 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044992799999999999 2020-06-12T02:37:33.539 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.539 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.539 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:33.540 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.163266+0000", 2020-06-12T02:37:33.540 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6959999999999999e-06 2020-06-12T02:37:33.540 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.540 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.540 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:33.540 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.163272+0000", 2020-06-12T02:37:33.541 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3028e-05 2020-06-12T02:37:33.541 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.541 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.542 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:33.542 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.163286+0000", 2020-06-12T02:37:33.542 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2829999999999998e-06 2020-06-12T02:37:33.543 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.543 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.543 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:33.543 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.163292+0000", 2020-06-12T02:37:33.543 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1761000000000002e-05 2020-06-12T02:37:33.543 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.544 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.544 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:33.544 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.163334+0000", 2020-06-12T02:37:33.544 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6691000000000001e-05 2020-06-12T02:37:33.544 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.545 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.545 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:33.545 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.163410+0000", 2020-06-12T02:37:33.545 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004070176 2020-06-12T02:37:33.545 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.545 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.546 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:33.546 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.167480+0000", 2020-06-12T02:37:33.546 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7137000000000003e-05 2020-06-12T02:37:33.546 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.546 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.547 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:33.547 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.167568+0000", 2020-06-12T02:37:33.547 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000789648 2020-06-12T02:37:33.547 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.547 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.547 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:33.548 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168357+0000", 2020-06-12T02:37:33.548 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011078226 2020-06-12T02:37:33.548 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.548 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.548 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:33.549 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.179435+0000", 2020-06-12T02:37:33.549 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6649000000000003e-05 2020-06-12T02:37:33.549 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.549 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.549 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:33.549 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.179522+0000", 2020-06-12T02:37:33.550 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0986e-05 2020-06-12T02:37:33.550 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.550 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.550 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:33.550 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.179603+0000", 2020-06-12T02:37:33.551 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016787209000000001 2020-06-12T02:37:33.551 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:33.551 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:33.551 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:33.551 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.551 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.552 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1300 3.bs0 3:d0ebae7d:::smithi10729407-3:head [append 0~557056] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:33.552 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.166253+0000", 2020-06-12T02:37:33.552 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.091239382999999993, 2020-06-12T02:37:33.552 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019037724999999998, 2020-06-12T02:37:33.552 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:33.553 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:33.553 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:33.553 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:33.553 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:33.553 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1300 2020-06-12T02:37:33.554 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.554 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:33.554 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.554 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:33.554 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.166253+0000", 2020-06-12T02:37:33.554 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:33.555 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.555 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.555 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:33.555 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.166253+0000", 2020-06-12T02:37:33.555 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034533600000000002 2020-06-12T02:37:33.556 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.556 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.556 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:33.556 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.166599+0000", 2020-06-12T02:37:33.556 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7699999999999996e-06 2020-06-12T02:37:33.556 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.557 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.557 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:33.557 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.166608+0000", 2020-06-12T02:37:33.557 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7314999999999999e-05 2020-06-12T02:37:33.557 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.558 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.558 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:33.558 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.166626+0000", 2020-06-12T02:37:33.558 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9260000000000003e-06 2020-06-12T02:37:33.558 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.558 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.559 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:33.559 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.166633+0000", 2020-06-12T02:37:33.559 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0266000000000002e-05 2020-06-12T02:37:33.559 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.559 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.559 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:33.560 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.166693+0000", 2020-06-12T02:37:33.560 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6692000000000005e-05 2020-06-12T02:37:33.560 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.560 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.560 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:33.561 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.166760+0000", 2020-06-12T02:37:33.561 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0057626860000000004 2020-06-12T02:37:33.561 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.561 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.561 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:33.562 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.172522+0000", 2020-06-12T02:37:33.562 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000130493 2020-06-12T02:37:33.562 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.562 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.562 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:33.562 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.172653+0000", 2020-06-12T02:37:33.563 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00083376299999999995 2020-06-12T02:37:33.563 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.563 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.563 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:33.564 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.173487+0000", 2020-06-12T02:37:33.564 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0061197739999999997 2020-06-12T02:37:33.565 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.565 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.565 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:33.565 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.179606+0000", 2020-06-12T02:37:33.565 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0056160699999999999 2020-06-12T02:37:33.566 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.566 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.566 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:33.566 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.185222+0000", 2020-06-12T02:37:33.566 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8633999999999994e-05 2020-06-12T02:37:33.567 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.567 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.567 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:33.567 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.185291+0000", 2020-06-12T02:37:33.567 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019037724999999998 2020-06-12T02:37:33.567 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:33.568 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:33.568 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:33.568 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.568 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.568 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1301 3.bs0 3:d0ebae7d:::smithi10729407-3:head [append 0~278528] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:33.569 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.168146+0000", 2020-06-12T02:37:33.569 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.089346826000000004, 2020-06-12T02:37:33.569 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017691083999999999, 2020-06-12T02:37:33.569 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:33.569 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:33.570 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:33.570 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:33.570 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:33.570 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1301 2020-06-12T02:37:33.570 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.570 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:33.571 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.571 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:33.571 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168146+0000", 2020-06-12T02:37:33.571 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:33.571 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.572 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.572 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:33.572 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168146+0000", 2020-06-12T02:37:33.573 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000213666 2020-06-12T02:37:33.573 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.574 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.574 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:33.574 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168360+0000", 2020-06-12T02:37:33.574 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7009999999999996e-06 2020-06-12T02:37:33.574 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.575 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.575 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:33.575 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168367+0000", 2020-06-12T02:37:33.575 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5096000000000001e-05 2020-06-12T02:37:33.575 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.576 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.576 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:33.576 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168382+0000", 2020-06-12T02:37:33.576 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8739999999999997e-06 2020-06-12T02:37:33.577 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.577 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.577 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:33.577 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168389+0000", 2020-06-12T02:37:33.577 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0076120199999999997 2020-06-12T02:37:33.578 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.578 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.578 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:33.578 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.176001+0000", 2020-06-12T02:37:33.579 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9181000000000001e-05 2020-06-12T02:37:33.579 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.580 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.580 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:33.581 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.176060+0000", 2020-06-12T02:37:33.581 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00039587599999999998 2020-06-12T02:37:33.581 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.581 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.581 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:33.582 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.176456+0000", 2020-06-12T02:37:33.582 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0031522329999999999 2020-06-12T02:37:33.582 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.582 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.582 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:33.583 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.179609+0000", 2020-06-12T02:37:33.583 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0061683670000000001 2020-06-12T02:37:33.583 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.583 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.583 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:33.584 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.185777+0000", 2020-06-12T02:37:33.584 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0069999999999999e-05 2020-06-12T02:37:33.584 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.584 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.584 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:33.584 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.185837+0000", 2020-06-12T02:37:33.585 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017691083999999999 2020-06-12T02:37:33.585 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:33.585 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:33.585 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:33.585 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.586 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.586 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1302 3.bs0 3:d0ebae7d:::smithi10729407-3:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:33.586 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.168445+0000", 2020-06-12T02:37:33.586 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.089047546000000005, 2020-06-12T02:37:33.587 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017581657000000001, 2020-06-12T02:37:33.588 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:33.588 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:33.588 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:33.588 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:33.588 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:33.589 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1302 2020-06-12T02:37:33.589 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.589 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:33.589 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.589 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:33.590 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168445+0000", 2020-06-12T02:37:33.590 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:33.590 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.590 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.590 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:33.591 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168445+0000", 2020-06-12T02:37:33.591 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3532e-05 2020-06-12T02:37:33.591 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.591 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.591 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:33.592 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168519+0000", 2020-06-12T02:37:33.592 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2600000000000002e-06 2020-06-12T02:37:33.592 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.592 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.592 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:33.592 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168525+0000", 2020-06-12T02:37:33.593 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4338999999999999e-05 2020-06-12T02:37:33.593 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.593 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.593 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:33.594 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168539+0000", 2020-06-12T02:37:33.594 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7330000000000003e-06 2020-06-12T02:37:33.594 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.594 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.594 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:33.595 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168544+0000", 2020-06-12T02:37:33.595 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0095046090000000007 2020-06-12T02:37:33.595 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.595 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.595 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:33.596 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.178049+0000", 2020-06-12T02:37:33.596 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1524000000000003e-05 2020-06-12T02:37:33.596 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.596 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.596 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:33.597 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.178110+0000", 2020-06-12T02:37:33.597 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030514799999999999 2020-06-12T02:37:33.597 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.597 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.598 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:33.598 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.178415+0000", 2020-06-12T02:37:33.598 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011944480000000001 2020-06-12T02:37:33.598 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.598 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.599 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:33.599 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.179610+0000", 2020-06-12T02:37:33.599 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0063372849999999998 2020-06-12T02:37:33.599 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.599 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.600 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:33.600 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.185947+0000", 2020-06-12T02:37:33.600 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9778999999999996e-05 2020-06-12T02:37:33.600 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.601 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.601 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:33.601 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.186027+0000", 2020-06-12T02:37:33.601 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017581657000000001 2020-06-12T02:37:33.601 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:33.602 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:33.602 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:33.602 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.602 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.602 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1303 3.bs0 3:d0ebae7d:::smithi10729407-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:33.603 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.168587+0000", 2020-06-12T02:37:33.603 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.088905628, 2020-06-12T02:37:33.603 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019127155, 2020-06-12T02:37:33.603 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:33.603 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:37:33.604 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:33.604 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:33.604 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:33.604 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1303 2020-06-12T02:37:33.605 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.605 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:33.605 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.605 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:33.605 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168587+0000", 2020-06-12T02:37:33.606 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:33.606 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.606 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.606 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:33.606 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168587+0000", 2020-06-12T02:37:33.606 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3099999999999997e-05 2020-06-12T02:37:33.607 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.607 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.607 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:33.607 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168630+0000", 2020-06-12T02:37:33.607 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1600000000000003e-06 2020-06-12T02:37:33.608 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.608 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.608 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:33.608 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168636+0000", 2020-06-12T02:37:33.608 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3439e-05 2020-06-12T02:37:33.608 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.609 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.609 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:33.609 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168650+0000", 2020-06-12T02:37:33.609 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7850000000000002e-06 2020-06-12T02:37:33.609 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.610 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.610 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:33.610 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168654+0000", 2020-06-12T02:37:33.610 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010326317 2020-06-12T02:37:33.610 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.611 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.611 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:33.611 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.178980+0000", 2020-06-12T02:37:33.611 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6593000000000004e-05 2020-06-12T02:37:33.611 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.612 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:33.614 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.179047+0000", 2020-06-12T02:37:33.614 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0073410990000000002 2020-06-12T02:37:33.615 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.615 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.616 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:33.616 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.186388+0000", 2020-06-12T02:37:33.616 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9405000000000001e-05 2020-06-12T02:37:33.616 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.616 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.617 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:33.617 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.186487+0000", 2020-06-12T02:37:33.617 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001227257 2020-06-12T02:37:33.617 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:33.618 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:33.618 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:33.618 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.187714+0000", 2020-06-12T02:37:33.618 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019127155 2020-06-12T02:37:33.618 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:33.619 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:33.619 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:33.619 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:33.619 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:33.619 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:37:33.640 INFO:teuthology.orchestra.run.smithi188:> CEPH_ARGS='--filestore-merge-threshold 1 --filestore-split-multiple 1' sudo -E ceph-objectstore-tool --no-mon-config --log-file=/var/log/ceph/objectstore_tool.$pid.log --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op apply-layout-settings --pool unique_pool_0 2020-06-12T02:37:33.641 INFO:tasks.rados.rados.0.smithi107.stdout:597: done (4 left) 2020-06-12T02:37:33.641 INFO:tasks.rados.rados.0.smithi107.stdout:599: done (3 left) 2020-06-12T02:37:33.641 INFO:tasks.rados.rados.0.smithi107.stdout:600: done (2 left) 2020-06-12T02:37:33.642 INFO:tasks.rados.rados.0.smithi107.stdout:602: done (1 left) 2020-06-12T02:37:33.642 INFO:tasks.rados.rados.0.smithi107.stdout:603: delete oid 10 current snap is 0 2020-06-12T02:37:33.642 INFO:tasks.rados.rados.0.smithi107.stdout:601: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:33.642 INFO:tasks.rados.rados.0.smithi107.stdout:603: done (1 left) 2020-06-12T02:37:33.642 INFO:tasks.rados.rados.0.smithi107.stdout:604: delete oid 17 current snap is 0 2020-06-12T02:37:33.643 INFO:tasks.rados.rados.0.smithi107.stdout:601: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:33.643 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 115 (ObjNum 600 snap 0 seq_num 600) dirty exists 2020-06-12T02:37:33.643 INFO:tasks.rados.rados.0.smithi107.stdout:601: left oid 15 (ObjNum 600 snap 0 seq_num 600) 2020-06-12T02:37:33.643 INFO:tasks.rados.rados.0.smithi107.stdout:601: done (1 left) 2020-06-12T02:37:33.643 INFO:tasks.rados.rados.0.smithi107.stdout:604: done (0 left) 2020-06-12T02:37:33.644 INFO:tasks.rados.rados.0.smithi107.stdout:605: delete oid 2 current snap is 0 2020-06-12T02:37:33.644 INFO:tasks.rados.rados.0.smithi107.stdout:605: done (0 left) 2020-06-12T02:37:33.644 INFO:tasks.rados.rados.0.smithi107.stdout:606: delete oid 12 current snap is 0 2020-06-12T02:37:33.644 INFO:tasks.rados.rados.0.smithi107.stdout:606: done (0 left) 2020-06-12T02:37:33.644 INFO:tasks.rados.rados.0.smithi107.stdout:607: delete oid 9 current snap is 0 2020-06-12T02:37:33.644 INFO:tasks.rados.rados.0.smithi107.stdout:607: done (0 left) 2020-06-12T02:37:33.645 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:37:33.645 INFO:tasks.rados.rados.0.smithi107.stdout:608: seq_num 607 ranges {0=573440} 2020-06-12T02:37:33.645 INFO:tasks.rados.rados.0.smithi107.stdout:608: writing smithi10729407-9 from 0 to 573440 tid 1 2020-06-12T02:37:33.645 INFO:tasks.rados.rados.0.smithi107.stdout:609: delete oid 7 current snap is 0 2020-06-12T02:37:33.645 INFO:tasks.rados.rados.0.smithi107.stdout:609: done (1 left) 2020-06-12T02:37:33.646 INFO:tasks.rados.rados.0.smithi107.stdout:610: delete oid 8 current snap is 0 2020-06-12T02:37:33.646 INFO:tasks.rados.rados.0.smithi107.stdout:610: done (1 left) 2020-06-12T02:37:33.646 INFO:tasks.rados.rados.0.smithi107.stdout:611: delete oid 14 current snap is 0 2020-06-12T02:37:33.646 INFO:tasks.rados.rados.0.smithi107.stdout:608: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:33.646 INFO:tasks.rados.rados.0.smithi107.stdout:608: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:33.647 INFO:tasks.rados.rados.0.smithi107.stdout:611: done (1 left) 2020-06-12T02:37:33.647 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:37:33.647 INFO:tasks.rados.rados.0.smithi107.stdout:612: seq_num 611 ranges {434176=638976} 2020-06-12T02:37:33.647 INFO:tasks.rados.rados.0.smithi107.stdout:612: writing smithi10729407-4 from 434176 to 1073152 tid 1 2020-06-12T02:37:33.648 INFO:tasks.rados.rados.0.smithi107.stdout:613: delete oid 14 current snap is 0 2020-06-12T02:37:33.648 INFO:tasks.rados.rados.0.smithi107.stdout:608: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:33.648 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 126 (ObjNum 607 snap 0 seq_num 607) dirty exists 2020-06-12T02:37:33.648 INFO:tasks.rados.rados.0.smithi107.stdout:608: left oid 9 (ObjNum 607 snap 0 seq_num 607) 2020-06-12T02:37:33.648 INFO:tasks.rados.rados.0.smithi107.stdout:608: done (2 left) 2020-06-12T02:37:33.649 INFO:tasks.rados.rados.0.smithi107.stdout:613: done (1 left) 2020-06-12T02:37:33.649 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:37:33.649 INFO:tasks.rados.rados.0.smithi107.stdout:614: seq_num 613 ranges {0=671744} 2020-06-12T02:37:33.649 INFO:tasks.rados.rados.0.smithi107.stdout:614: writing smithi10729407-7 from 0 to 671744 tid 1 2020-06-12T02:37:33.649 INFO:tasks.rados.rados.0.smithi107.stdout:615: delete oid 9 current snap is 0 2020-06-12T02:37:33.650 INFO:tasks.rados.rados.0.smithi107.stdout:612: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:33.650 INFO:tasks.rados.rados.0.smithi107.stdout:612: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:33.650 INFO:tasks.rados.rados.0.smithi107.stdout:612: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:33.650 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 118 (ObjNum 611 snap 0 seq_num 611) dirty exists 2020-06-12T02:37:33.651 INFO:tasks.rados.rados.0.smithi107.stdout:612: left oid 4 (ObjNum 611 snap 0 seq_num 611) 2020-06-12T02:37:33.651 INFO:tasks.rados.rados.0.smithi107.stdout:612: done (2 left) 2020-06-12T02:37:33.651 INFO:tasks.rados.rados.0.smithi107.stdout:615: done (1 left) 2020-06-12T02:37:33.651 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:37:33.652 INFO:tasks.rados.rados.0.smithi107.stdout:616: seq_num 615 ranges {0=704512} 2020-06-12T02:37:33.652 INFO:tasks.rados.rados.0.smithi107.stdout:616: writing smithi10729407-9 from 0 to 704512 tid 1 2020-06-12T02:37:33.652 INFO:tasks.rados.rados.0.smithi107.stdout:617: delete oid 13 current snap is 0 2020-06-12T02:37:33.652 INFO:tasks.rados.rados.0.smithi107.stdout:614: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:33.652 INFO:tasks.rados.rados.0.smithi107.stdout:614: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:33.653 INFO:tasks.rados.rados.0.smithi107.stdout:616: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:33.653 INFO:tasks.rados.rados.0.smithi107.stdout:617: done (2 left) 2020-06-12T02:37:33.653 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:37:33.653 INFO:tasks.rados.rados.0.smithi107.stdout:618: seq_num 617 ranges {0=737280} 2020-06-12T02:37:33.653 INFO:tasks.rados.rados.0.smithi107.stdout:618: writing smithi10729407-12 from 0 to 737280 tid 1 2020-06-12T02:37:33.653 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:37:33.654 INFO:tasks.rados.rados.0.smithi107.stdout:619: seq_num 618 ranges {0=753664} 2020-06-12T02:37:33.654 INFO:tasks.rados.rados.0.smithi107.stdout:619: writing smithi10729407-8 from 0 to 753664 tid 1 2020-06-12T02:37:33.654 INFO:tasks.rados.rados.0.smithi107.stdout:620: delete oid 4 current snap is 0 2020-06-12T02:37:33.654 INFO:tasks.rados.rados.0.smithi107.stdout:614: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:33.654 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 122 (ObjNum 613 snap 0 seq_num 613) dirty exists 2020-06-12T02:37:33.655 INFO:tasks.rados.rados.0.smithi107.stdout:614: left oid 7 (ObjNum 613 snap 0 seq_num 613) 2020-06-12T02:37:33.655 INFO:tasks.rados.rados.0.smithi107.stdout:616: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:33.655 INFO:tasks.rados.rados.0.smithi107.stdout:616: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:33.655 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 129 (ObjNum 615 snap 0 seq_num 615) dirty exists 2020-06-12T02:37:33.655 INFO:tasks.rados.rados.0.smithi107.stdout:616: left oid 9 (ObjNum 615 snap 0 seq_num 615) 2020-06-12T02:37:33.656 INFO:tasks.rados.rados.0.smithi107.stdout:618: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:33.656 INFO:tasks.rados.rados.0.smithi107.stdout:614: done (4 left) 2020-06-12T02:37:33.656 INFO:tasks.rados.rados.0.smithi107.stdout:616: done (3 left) 2020-06-12T02:37:33.656 INFO:tasks.rados.rados.0.smithi107.stdout:620: done (2 left) 2020-06-12T02:37:33.657 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:37:33.657 INFO:tasks.rados.rados.0.smithi107.stdout:621: seq_num 620 ranges {0=794624} 2020-06-12T02:37:33.657 INFO:tasks.rados.rados.0.smithi107.stdout:621: writing smithi10729407-11 from 0 to 794624 tid 1 2020-06-12T02:37:33.657 INFO:tasks.rados.rados.0.smithi107.stdout:622: delete oid 9 current snap is 0 2020-06-12T02:37:33.657 INFO:tasks.rados.rados.0.smithi107.stdout:618: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:33.658 INFO:tasks.rados.rados.0.smithi107.stdout:618: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:33.658 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 98 (ObjNum 617 snap 0 seq_num 617) dirty exists 2020-06-12T02:37:33.658 INFO:tasks.rados.rados.0.smithi107.stdout:618: left oid 12 (ObjNum 617 snap 0 seq_num 617) 2020-06-12T02:37:33.658 INFO:tasks.rados.rados.0.smithi107.stdout:619: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:33.658 INFO:tasks.rados.rados.0.smithi107.stdout:619: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:33.658 INFO:tasks.rados.rados.0.smithi107.stdout:618: done (3 left) 2020-06-12T02:37:33.659 INFO:tasks.rados.rados.0.smithi107.stdout:622: done (2 left) 2020-06-12T02:37:33.659 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:37:33.659 INFO:tasks.rados.rados.0.smithi107.stdout:623: seq_num 622 ranges {737280=425984,1163264=401408} 2020-06-12T02:37:33.659 INFO:tasks.rados.rados.0.smithi107.stdout:623: writing smithi10729407-12 from 737280 to 1163264 tid 1 2020-06-12T02:37:33.659 INFO:tasks.rados.rados.0.smithi107.stdout:623: writing smithi10729407-12 from 1163264 to 1564672 tid 2 2020-06-12T02:37:33.660 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:37:33.660 INFO:tasks.rados.rados.0.smithi107.stdout:624: seq_num 623 ranges {0=442368,442368=401408} 2020-06-12T02:37:33.660 INFO:tasks.rados.rados.0.smithi107.stdout:624: writing smithi10729407-1 from 0 to 442368 tid 1 2020-06-12T02:37:33.660 INFO:tasks.rados.rados.0.smithi107.stdout:624: writing smithi10729407-1 from 442368 to 843776 tid 2 2020-06-12T02:37:33.660 INFO:tasks.rados.rados.0.smithi107.stdout:625: delete oid 13 current snap is 0 2020-06-12T02:37:33.661 INFO:tasks.rados.rados.0.smithi107.stdout:619: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:33.661 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 118 (ObjNum 618 snap 0 seq_num 618) dirty exists 2020-06-12T02:37:33.661 INFO:tasks.rados.rados.0.smithi107.stdout:619: left oid 8 (ObjNum 618 snap 0 seq_num 618) 2020-06-12T02:37:33.661 INFO:tasks.rados.rados.0.smithi107.stdout:621: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:33.661 INFO:tasks.rados.rados.0.smithi107.stdout:621: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:33.662 INFO:tasks.rados.rados.0.smithi107.stdout:623: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:33.662 INFO:tasks.rados.rados.0.smithi107.stdout:621: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:33.662 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 84 (ObjNum 620 snap 0 seq_num 620) dirty exists 2020-06-12T02:37:33.662 INFO:tasks.rados.rados.0.smithi107.stdout:621: left oid 11 (ObjNum 620 snap 0 seq_num 620) 2020-06-12T02:37:33.663 INFO:tasks.rados.rados.0.smithi107.stdout:623: finishing write tid 2 to smithi10729407-12 2020-06-12T02:37:33.663 INFO:tasks.rados.rados.0.smithi107.stdout:623: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:33.663 INFO:tasks.rados.rados.0.smithi107.stdout:619: done (4 left) 2020-06-12T02:37:33.663 INFO:tasks.rados.rados.0.smithi107.stdout:621: done (3 left) 2020-06-12T02:37:33.663 INFO:tasks.rados.rados.0.smithi107.stdout:625: done (2 left) 2020-06-12T02:37:33.664 INFO:tasks.rados.rados.0.smithi107.stdout:626: delete oid 8 current snap is 0 2020-06-12T02:37:33.664 INFO:tasks.rados.rados.0.smithi107.stdout:623: finishing write tid 5 to smithi10729407-12 2020-06-12T02:37:33.664 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 101 (ObjNum 622 snap 0 seq_num 622) dirty exists 2020-06-12T02:37:33.664 INFO:tasks.rados.rados.0.smithi107.stdout:623: left oid 12 (ObjNum 622 snap 0 seq_num 622) 2020-06-12T02:37:33.664 INFO:tasks.rados.rados.0.smithi107.stdout:623: done (2 left) 2020-06-12T02:37:33.665 INFO:tasks.rados.rados.0.smithi107.stdout:626: done (1 left) 2020-06-12T02:37:33.665 INFO:tasks.rados.rados.0.smithi107.stdout:627: delete oid 10 current snap is 0 2020-06-12T02:37:33.665 INFO:tasks.rados.rados.0.smithi107.stdout:624: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:33.666 INFO:tasks.rados.rados.0.smithi107.stdout:624: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:33.666 INFO:tasks.rados.rados.0.smithi107.stdout:624: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:33.666 INFO:tasks.rados.rados.0.smithi107.stdout:624: finishing write tid 5 to smithi10729407-1 2020-06-12T02:37:33.666 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 138 (ObjNum 623 snap 0 seq_num 623) dirty exists 2020-06-12T02:37:33.666 INFO:tasks.rados.rados.0.smithi107.stdout:624: left oid 1 (ObjNum 623 snap 0 seq_num 623) 2020-06-12T02:37:33.667 INFO:tasks.rados.rados.0.smithi107.stdout:624: done (1 left) 2020-06-12T02:37:33.667 INFO:tasks.rados.rados.0.smithi107.stdout:627: done (0 left) 2020-06-12T02:37:33.667 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:37:33.667 INFO:tasks.rados.rados.0.smithi107.stdout:628: seq_num 627 ranges {4481024=507904,4988928=401408} 2020-06-12T02:37:33.667 INFO:tasks.rados.rados.0.smithi107.stdout:628: writing smithi10729407-3 from 4481024 to 4988928 tid 1 2020-06-12T02:37:33.668 INFO:tasks.rados.rados.0.smithi107.stdout:628: writing smithi10729407-3 from 4988928 to 5390336 tid 2 2020-06-12T02:37:33.668 INFO:tasks.rados.rados.0.smithi107.stdout:629: delete oid 9 current snap is 0 2020-06-12T02:37:33.668 INFO:tasks.rados.rados.0.smithi107.stdout:628: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:33.668 INFO:tasks.rados.rados.0.smithi107.stdout:629: done (1 left) 2020-06-12T02:37:33.668 INFO:tasks.rados.rados.0.smithi107.stdout:630: delete oid 20 current snap is 0 2020-06-12T02:37:33.669 INFO:tasks.rados.rados.0.smithi107.stdout:630: done (1 left) 2020-06-12T02:37:33.669 INFO:tasks.rados.rados.0.smithi107.stdout:631: delete oid 9 current snap is 0 2020-06-12T02:37:33.669 INFO:tasks.rados.rados.0.smithi107.stdout:628: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:33.669 INFO:tasks.rados.rados.0.smithi107.stdout:631: done (1 left) 2020-06-12T02:37:33.669 INFO:tasks.rados.rados.0.smithi107.stdout:632: delete oid 8 current snap is 0 2020-06-12T02:37:33.670 INFO:tasks.rados.rados.0.smithi107.stdout:628: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:33.670 INFO:tasks.rados.rados.0.smithi107.stdout:632: done (1 left) 2020-06-12T02:37:33.670 INFO:tasks.rados.rados.0.smithi107.stdout:633: delete oid 8 current snap is 0 2020-06-12T02:37:33.670 INFO:tasks.rados.rados.0.smithi107.stdout:628: finishing write tid 5 to smithi10729407-3 2020-06-12T02:37:33.670 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 105 (ObjNum 627 snap 0 seq_num 627) dirty exists 2020-06-12T02:37:33.670 INFO:tasks.rados.rados.0.smithi107.stdout:628: left oid 3 (ObjNum 627 snap 0 seq_num 627) 2020-06-12T02:37:33.671 INFO:tasks.rados.rados.0.smithi107.stdout:628: done (1 left) 2020-06-12T02:37:33.671 INFO:tasks.rados.rados.0.smithi107.stdout:633: done (0 left) 2020-06-12T02:37:33.671 INFO:tasks.rados.rados.0.smithi107.stdout:634: delete oid 19 current snap is 0 2020-06-12T02:37:33.671 INFO:tasks.rados.rados.0.smithi107.stdout:634: done (0 left) 2020-06-12T02:37:33.671 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:37:33.672 INFO:tasks.rados.rados.0.smithi107.stdout:635: seq_num 634 ranges {5390336=622592,6012928=401408} 2020-06-12T02:37:33.672 INFO:tasks.rados.rados.0.smithi107.stdout:635: writing smithi10729407-3 from 5390336 to 6012928 tid 1 2020-06-12T02:37:33.672 INFO:tasks.rados.rados.0.smithi107.stdout:635: writing smithi10729407-3 from 6012928 to 6414336 tid 2 2020-06-12T02:37:33.672 INFO:tasks.rados.rados.0.smithi107.stdout:636: delete oid 16 current snap is 0 2020-06-12T02:37:33.672 INFO:tasks.rados.rados.0.smithi107.stdout:635: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:33.672 INFO:tasks.rados.rados.0.smithi107.stdout:636: done (1 left) 2020-06-12T02:37:33.673 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:37:33.673 INFO:tasks.rados.rados.0.smithi107.stdout:637: seq_num 636 ranges {0=655360,655360=401408} 2020-06-12T02:37:33.673 INFO:tasks.rados.rados.0.smithi107.stdout:637: writing smithi10729407-18 from 0 to 655360 tid 1 2020-06-12T02:37:33.673 INFO:tasks.rados.rados.0.smithi107.stdout:637: writing smithi10729407-18 from 655360 to 1056768 tid 2 2020-06-12T02:37:33.673 INFO:tasks.rados.rados.0.smithi107.stdout:638: delete oid 7 current snap is 0 2020-06-12T02:37:33.674 INFO:tasks.rados.rados.0.smithi107.stdout:635: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:33.674 INFO:tasks.rados.rados.0.smithi107.stdout:635: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:33.674 INFO:tasks.rados.rados.0.smithi107.stdout:635: finishing write tid 5 to smithi10729407-3 2020-06-12T02:37:33.674 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 109 (ObjNum 634 snap 0 seq_num 634) dirty exists 2020-06-12T02:37:33.675 INFO:tasks.rados.rados.0.smithi107.stdout:635: left oid 3 (ObjNum 634 snap 0 seq_num 634) 2020-06-12T02:37:33.675 INFO:tasks.rados.rados.0.smithi107.stdout:635: done (2 left) 2020-06-12T02:37:33.675 INFO:tasks.rados.rados.0.smithi107.stdout:638: done (1 left) 2020-06-12T02:37:33.675 INFO:tasks.rados.rados.0.smithi107.stdout:639: delete oid 16 current snap is 0 2020-06-12T02:37:33.675 INFO:tasks.rados.rados.0.smithi107.stdout:637: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:33.676 INFO:tasks.rados.rados.0.smithi107.stdout:637: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:33.676 INFO:tasks.rados.rados.0.smithi107.stdout:639: done (1 left) 2020-06-12T02:37:33.676 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:37:33.676 INFO:tasks.rados.rados.0.smithi107.stdout:640: seq_num 639 ranges {843776=704512,1548288=401408} 2020-06-12T02:37:33.676 INFO:tasks.rados.rados.0.smithi107.stdout:640: writing smithi10729407-1 from 843776 to 1548288 tid 1 2020-06-12T02:37:33.676 INFO:tasks.rados.rados.0.smithi107.stdout:640: writing smithi10729407-1 from 1548288 to 1949696 tid 2 2020-06-12T02:37:33.677 INFO:tasks.rados.rados.0.smithi107.stdout:641: delete oid 3 current snap is 0 2020-06-12T02:37:33.677 INFO:tasks.rados.rados.0.smithi107.stdout:637: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:33.677 INFO:tasks.rados.rados.0.smithi107.stdout:637: finishing write tid 5 to smithi10729407-18 2020-06-12T02:37:33.677 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 135 (ObjNum 636 snap 0 seq_num 636) dirty exists 2020-06-12T02:37:33.678 INFO:tasks.rados.rados.0.smithi107.stdout:637: left oid 18 (ObjNum 636 snap 0 seq_num 636) 2020-06-12T02:37:33.678 INFO:tasks.rados.rados.0.smithi107.stdout:637: done (2 left) 2020-06-12T02:37:33.678 INFO:tasks.rados.rados.0.smithi107.stdout:641: done (1 left) 2020-06-12T02:37:33.678 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:37:33.678 INFO:tasks.rados.rados.0.smithi107.stdout:642: seq_num 641 ranges {0=745472,745472=401408} 2020-06-12T02:37:33.679 INFO:tasks.rados.rados.0.smithi107.stdout:642: writing smithi10729407-20 from 0 to 745472 tid 1 2020-06-12T02:37:33.679 INFO:tasks.rados.rados.0.smithi107.stdout:642: writing smithi10729407-20 from 745472 to 1146880 tid 2 2020-06-12T02:37:33.679 INFO:tasks.rados.rados.0.smithi107.stdout:643: delete oid 14 current snap is 0 2020-06-12T02:37:33.679 INFO:tasks.rados.rados.0.smithi107.stdout:640: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:33.680 INFO:tasks.rados.rados.0.smithi107.stdout:640: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:33.680 INFO:tasks.rados.rados.0.smithi107.stdout:640: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:33.680 INFO:tasks.rados.rados.0.smithi107.stdout:640: finishing write tid 5 to smithi10729407-1 2020-06-12T02:37:33.680 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 141 (ObjNum 639 snap 0 seq_num 639) dirty exists 2020-06-12T02:37:33.681 INFO:tasks.rados.rados.0.smithi107.stdout:640: left oid 1 (ObjNum 639 snap 0 seq_num 639) 2020-06-12T02:37:33.681 INFO:tasks.rados.rados.0.smithi107.stdout:640: done (2 left) 2020-06-12T02:37:33.681 INFO:tasks.rados.rados.0.smithi107.stdout:643: done (1 left) 2020-06-12T02:37:33.681 INFO:tasks.rados.rados.0.smithi107.stdout:644: delete oid 7 current snap is 0 2020-06-12T02:37:33.681 INFO:tasks.rados.rados.0.smithi107.stdout:644: done (1 left) 2020-06-12T02:37:33.682 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:37:33.682 INFO:tasks.rados.rados.0.smithi107.stdout:645: seq_num 644 ranges {237568=794624,1032192=401408} 2020-06-12T02:37:33.682 INFO:tasks.rados.rados.0.smithi107.stdout:645: writing smithi10729407-6 from 237568 to 1032192 tid 1 2020-06-12T02:37:33.682 INFO:tasks.rados.rados.0.smithi107.stdout:645: writing smithi10729407-6 from 1032192 to 1433600 tid 2 2020-06-12T02:37:33.682 INFO:tasks.rados.rados.0.smithi107.stdout:646: delete oid 10 current snap is 0 2020-06-12T02:37:33.683 INFO:tasks.rados.rados.0.smithi107.stdout:642: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:33.683 INFO:tasks.rados.rados.0.smithi107.stdout:642: finishing write tid 2 to smithi10729407-20 2020-06-12T02:37:33.683 INFO:tasks.rados.rados.0.smithi107.stdout:642: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:33.683 INFO:tasks.rados.rados.0.smithi107.stdout:642: finishing write tid 5 to smithi10729407-20 2020-06-12T02:37:33.684 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 59 (ObjNum 641 snap 0 seq_num 641) dirty exists 2020-06-12T02:37:33.684 INFO:tasks.rados.rados.0.smithi107.stdout:642: left oid 20 (ObjNum 641 snap 0 seq_num 641) 2020-06-12T02:37:33.684 INFO:tasks.rados.rados.0.smithi107.stdout:642: done (2 left) 2020-06-12T02:37:33.684 INFO:tasks.rados.rados.0.smithi107.stdout:646: done (1 left) 2020-06-12T02:37:33.684 INFO:tasks.rados.rados.0.smithi107.stdout:647: delete oid 8 current snap is 0 2020-06-12T02:37:33.685 INFO:tasks.rados.rados.0.smithi107.stdout:647: done (1 left) 2020-06-12T02:37:33.685 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:37:33.685 INFO:tasks.rados.rados.0.smithi107.stdout:648: seq_num 647 ranges {0=442368,442368=589824,1032192=212992} 2020-06-12T02:37:33.685 INFO:tasks.rados.rados.0.smithi107.stdout:648: writing smithi10729407-14 from 0 to 442368 tid 1 2020-06-12T02:37:33.685 INFO:tasks.rados.rados.0.smithi107.stdout:648: writing smithi10729407-14 from 442368 to 1032192 tid 2 2020-06-12T02:37:33.686 INFO:tasks.rados.rados.0.smithi107.stdout:648: writing smithi10729407-14 from 1032192 to 1245184 tid 3 2020-06-12T02:37:33.686 INFO:tasks.rados.rados.0.smithi107.stdout:649: delete oid 18 current snap is 0 2020-06-12T02:37:33.686 INFO:tasks.rados.rados.0.smithi107.stdout:645: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:33.686 INFO:tasks.rados.rados.0.smithi107.stdout:645: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:33.687 INFO:tasks.rados.rados.0.smithi107.stdout:645: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:33.687 INFO:tasks.rados.rados.0.smithi107.stdout:648: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:33.687 INFO:tasks.rados.rados.0.smithi107.stdout:649: done (2 left) 2020-06-12T02:37:33.687 INFO:tasks.rados.rados.0.smithi107.stdout:650: delete oid 15 current snap is 0 2020-06-12T02:37:33.687 INFO:tasks.rados.rados.0.smithi107.stdout:645: finishing write tid 5 to smithi10729407-6 2020-06-12T02:37:33.688 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 144 (ObjNum 644 snap 0 seq_num 644) dirty exists 2020-06-12T02:37:33.688 INFO:tasks.rados.rados.0.smithi107.stdout:645: left oid 6 (ObjNum 644 snap 0 seq_num 644) 2020-06-12T02:37:33.688 INFO:tasks.rados.rados.0.smithi107.stdout:645: done (2 left) 2020-06-12T02:37:33.688 INFO:tasks.rados.rados.0.smithi107.stdout:650: done (1 left) 2020-06-12T02:37:33.688 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:33.689 INFO:tasks.rados.rados.0.smithi107.stdout:651: seq_num 650 ranges {0=491520,491520=647168,1138688=155648} 2020-06-12T02:37:33.689 INFO:tasks.rados.rados.0.smithi107.stdout:651: writing smithi10729407-2 from 0 to 491520 tid 1 2020-06-12T02:37:33.689 INFO:tasks.rados.rados.0.smithi107.stdout:651: writing smithi10729407-2 from 491520 to 1138688 tid 2 2020-06-12T02:37:33.689 INFO:tasks.rados.rados.0.smithi107.stdout:651: writing smithi10729407-2 from 1138688 to 1294336 tid 3 2020-06-12T02:37:33.689 INFO:tasks.rados.rados.0.smithi107.stdout:652: delete oid 12 current snap is 0 2020-06-12T02:37:33.690 INFO:tasks.rados.rados.0.smithi107.stdout:648: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:33.690 INFO:tasks.rados.rados.0.smithi107.stdout:648: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:33.690 INFO:tasks.rados.rados.0.smithi107.stdout:648: finishing write tid 5 to smithi10729407-14 2020-06-12T02:37:33.690 INFO:tasks.rados.rados.0.smithi107.stdout:648: finishing write tid 6 to smithi10729407-14 2020-06-12T02:37:33.690 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 89 (ObjNum 647 snap 0 seq_num 647) dirty exists 2020-06-12T02:37:33.691 INFO:tasks.rados.rados.0.smithi107.stdout:648: left oid 14 (ObjNum 647 snap 0 seq_num 647) 2020-06-12T02:37:33.691 INFO:tasks.rados.rados.0.smithi107.stdout:651: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:33.691 INFO:tasks.rados.rados.0.smithi107.stdout:648: done (2 left) 2020-06-12T02:37:33.692 INFO:tasks.rados.rados.0.smithi107.stdout:652: done (1 left) 2020-06-12T02:37:33.692 INFO:tasks.rados.rados.0.smithi107.stdout:653: delete oid 16 current snap is 0 2020-06-12T02:37:33.692 INFO:tasks.rados.rados.0.smithi107.stdout:653: done (1 left) 2020-06-12T02:37:33.692 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:37:33.693 INFO:tasks.rados.rados.0.smithi107.stdout:654: seq_num 653 ranges {1146880=540672,1687552=696320,2383872=106496} 2020-06-12T02:37:33.693 INFO:tasks.rados.rados.0.smithi107.stdout:654: writing smithi10729407-20 from 1146880 to 1687552 tid 1 2020-06-12T02:37:33.693 INFO:tasks.rados.rados.0.smithi107.stdout:654: writing smithi10729407-20 from 1687552 to 2383872 tid 2 2020-06-12T02:37:33.693 INFO:tasks.rados.rados.0.smithi107.stdout:654: writing smithi10729407-20 from 2383872 to 2490368 tid 3 2020-06-12T02:37:33.694 INFO:tasks.rados.rados.0.smithi107.stdout:655: delete oid 12 current snap is 0 2020-06-12T02:37:33.694 INFO:tasks.rados.rados.0.smithi107.stdout:651: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:33.694 INFO:tasks.rados.rados.0.smithi107.stdout:651: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:33.694 INFO:tasks.rados.rados.0.smithi107.stdout:651: finishing write tid 5 to smithi10729407-2 2020-06-12T02:37:33.694 INFO:tasks.rados.rados.0.smithi107.stdout:651: finishing write tid 6 to smithi10729407-2 2020-06-12T02:37:33.695 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 67 (ObjNum 650 snap 0 seq_num 650) dirty exists 2020-06-12T02:37:33.695 INFO:tasks.rados.rados.0.smithi107.stdout:651: left oid 2 (ObjNum 650 snap 0 seq_num 650) 2020-06-12T02:37:33.695 INFO:tasks.rados.rados.0.smithi107.stdout:651: done (2 left) 2020-06-12T02:37:33.695 INFO:tasks.rados.rados.0.smithi107.stdout:655: done (1 left) 2020-06-12T02:37:33.696 INFO:tasks.rados.rados.0.smithi107.stdout:656: delete oid 19 current snap is 0 2020-06-12T02:37:33.696 INFO:tasks.rados.rados.0.smithi107.stdout:654: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:33.696 INFO:tasks.rados.rados.0.smithi107.stdout:654: finishing write tid 2 to smithi10729407-20 2020-06-12T02:37:33.696 INFO:tasks.rados.rados.0.smithi107.stdout:656: done (1 left) 2020-06-12T02:37:33.696 INFO:tasks.rados.rados.0.smithi107.stdout:657: delete oid 3 current snap is 0 2020-06-12T02:37:33.697 INFO:tasks.rados.rados.0.smithi107.stdout:654: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:33.697 INFO:tasks.rados.rados.0.smithi107.stdout:654: finishing write tid 5 to smithi10729407-20 2020-06-12T02:37:33.697 INFO:tasks.rados.rados.0.smithi107.stdout:657: done (1 left) 2020-06-12T02:37:33.697 INFO:tasks.rados.rados.0.smithi107.stdout:658: delete oid 6 current snap is 0 2020-06-12T02:37:33.697 INFO:tasks.rados.rados.0.smithi107.stdout:654: finishing write tid 6 to smithi10729407-20 2020-06-12T02:37:33.698 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 63 (ObjNum 653 snap 0 seq_num 653) dirty exists 2020-06-12T02:37:33.698 INFO:tasks.rados.rados.0.smithi107.stdout:654: left oid 20 (ObjNum 653 snap 0 seq_num 653) 2020-06-12T02:37:33.698 INFO:tasks.rados.rados.0.smithi107.stdout:654: done (1 left) 2020-06-12T02:37:33.698 INFO:tasks.rados.rados.0.smithi107.stdout:658: done (0 left) 2020-06-12T02:37:33.698 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:37:33.699 INFO:tasks.rados.rados.0.smithi107.stdout:659: seq_num 658 ranges {0=622592,622592=565248,1187840=237568} 2020-06-12T02:37:33.699 INFO:tasks.rados.rados.0.smithi107.stdout:659: writing smithi10729407-16 from 0 to 622592 tid 1 2020-06-12T02:37:33.699 INFO:tasks.rados.rados.0.smithi107.stdout:659: writing smithi10729407-16 from 622592 to 1187840 tid 2 2020-06-12T02:37:33.699 INFO:tasks.rados.rados.0.smithi107.stdout:659: writing smithi10729407-16 from 1187840 to 1425408 tid 3 2020-06-12T02:37:33.699 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:37:33.700 INFO:tasks.rados.rados.0.smithi107.stdout:660: seq_num 659 ranges {1245184=647168,1892352=450560,2342912=352256} 2020-06-12T02:37:33.700 INFO:tasks.rados.rados.0.smithi107.stdout:660: writing smithi10729407-14 from 1245184 to 1892352 tid 1 2020-06-12T02:37:33.700 INFO:tasks.rados.rados.0.smithi107.stdout:660: writing smithi10729407-14 from 1892352 to 2342912 tid 2 2020-06-12T02:37:33.700 INFO:tasks.rados.rados.0.smithi107.stdout:660: writing smithi10729407-14 from 2342912 to 2695168 tid 3 2020-06-12T02:37:33.700 INFO:tasks.rados.rados.0.smithi107.stdout:661: delete oid 17 current snap is 0 2020-06-12T02:37:33.701 INFO:tasks.rados.rados.0.smithi107.stdout:659: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:33.701 INFO:tasks.rados.rados.0.smithi107.stdout:659: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:33.701 INFO:tasks.rados.rados.0.smithi107.stdout:659: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:33.701 INFO:tasks.rados.rados.0.smithi107.stdout:659: finishing write tid 5 to smithi10729407-16 2020-06-12T02:37:33.701 INFO:tasks.rados.rados.0.smithi107.stdout:659: finishing write tid 6 to smithi10729407-16 2020-06-12T02:37:33.702 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 117 (ObjNum 658 snap 0 seq_num 658) dirty exists 2020-06-12T02:37:33.702 INFO:tasks.rados.rados.0.smithi107.stdout:659: left oid 16 (ObjNum 658 snap 0 seq_num 658) 2020-06-12T02:37:33.702 INFO:tasks.rados.rados.0.smithi107.stdout:659: done (2 left) 2020-06-12T02:37:33.702 INFO:tasks.rados.rados.0.smithi107.stdout:661: done (1 left) 2020-06-12T02:37:33.702 INFO:tasks.rados.rados.0.smithi107.stdout:662: delete oid 18 current snap is 0 2020-06-12T02:37:33.703 INFO:tasks.rados.rados.0.smithi107.stdout:662: done (1 left) 2020-06-12T02:37:33.703 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:37:33.703 INFO:tasks.rados.rados.0.smithi107.stdout:663: seq_num 662 ranges {1425408=696320,2121728=557056,2678784=245760} 2020-06-12T02:37:33.703 INFO:tasks.rados.rados.0.smithi107.stdout:663: writing smithi10729407-16 from 1425408 to 2121728 tid 1 2020-06-12T02:37:33.704 INFO:tasks.rados.rados.0.smithi107.stdout:663: writing smithi10729407-16 from 2121728 to 2678784 tid 2 2020-06-12T02:37:33.704 INFO:tasks.rados.rados.0.smithi107.stdout:663: writing smithi10729407-16 from 2678784 to 2924544 tid 3 2020-06-12T02:37:33.704 INFO:tasks.rados.rados.0.smithi107.stdout:664: delete oid 13 current snap is 0 2020-06-12T02:37:33.704 INFO:tasks.rados.rados.0.smithi107.stdout:660: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:33.704 INFO:tasks.rados.rados.0.smithi107.stdout:660: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:33.705 INFO:tasks.rados.rados.0.smithi107.stdout:660: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:33.705 INFO:tasks.rados.rados.0.smithi107.stdout:660: finishing write tid 5 to smithi10729407-14 2020-06-12T02:37:33.705 INFO:tasks.rados.rados.0.smithi107.stdout:660: finishing write tid 6 to smithi10729407-14 2020-06-12T02:37:33.705 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 93 (ObjNum 659 snap 0 seq_num 659) dirty exists 2020-06-12T02:37:33.705 INFO:tasks.rados.rados.0.smithi107.stdout:660: left oid 14 (ObjNum 659 snap 0 seq_num 659) 2020-06-12T02:37:33.706 INFO:tasks.rados.rados.0.smithi107.stdout:663: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:33.706 INFO:tasks.rados.rados.0.smithi107.stdout:663: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:33.706 INFO:tasks.rados.rados.0.smithi107.stdout:660: done (2 left) 2020-06-12T02:37:33.706 INFO:tasks.rados.rados.0.smithi107.stdout:664: done (1 left) 2020-06-12T02:37:33.706 INFO:tasks.rados.rados.0.smithi107.stdout:665: delete oid 6 current snap is 0 2020-06-12T02:37:33.707 INFO:tasks.rados.rados.0.smithi107.stdout:663: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:33.707 INFO:tasks.rados.rados.0.smithi107.stdout:663: finishing write tid 5 to smithi10729407-16 2020-06-12T02:37:33.707 INFO:tasks.rados.rados.0.smithi107.stdout:665: done (1 left) 2020-06-12T02:37:33.707 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:37:33.707 INFO:tasks.rados.rados.0.smithi107.stdout:666: seq_num 665 ranges {0=745472,745472=606208,1351680=196608} 2020-06-12T02:37:33.708 INFO:tasks.rados.rados.0.smithi107.stdout:666: writing smithi10729407-18 from 0 to 745472 tid 1 2020-06-12T02:37:33.708 INFO:tasks.rados.rados.0.smithi107.stdout:666: writing smithi10729407-18 from 745472 to 1351680 tid 2 2020-06-12T02:37:33.708 INFO:tasks.rados.rados.0.smithi107.stdout:666: writing smithi10729407-18 from 1351680 to 1548288 tid 3 2020-06-12T02:37:33.708 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:37:33.708 INFO:tasks.rados.rados.0.smithi107.stdout:667: seq_num 666 ranges {794624=761856,1556480=491520,2048000=311296} 2020-06-12T02:37:33.709 INFO:tasks.rados.rados.0.smithi107.stdout:667: writing smithi10729407-11 from 794624 to 1556480 tid 1 2020-06-12T02:37:33.709 INFO:tasks.rados.rados.0.smithi107.stdout:667: writing smithi10729407-11 from 1556480 to 2048000 tid 2 2020-06-12T02:37:33.709 INFO:tasks.rados.rados.0.smithi107.stdout:667: writing smithi10729407-11 from 2048000 to 2359296 tid 3 2020-06-12T02:37:33.709 INFO:tasks.rados.rados.0.smithi107.stdout:668: delete oid 5 current snap is 0 2020-06-12T02:37:33.709 INFO:tasks.rados.rados.0.smithi107.stdout:663: finishing write tid 6 to smithi10729407-16 2020-06-12T02:37:33.710 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 121 (ObjNum 662 snap 0 seq_num 662) dirty exists 2020-06-12T02:37:33.710 INFO:tasks.rados.rados.0.smithi107.stdout:663: left oid 16 (ObjNum 662 snap 0 seq_num 662) 2020-06-12T02:37:33.710 INFO:tasks.rados.rados.0.smithi107.stdout:666: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:33.710 INFO:tasks.rados.rados.0.smithi107.stdout:666: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:33.710 INFO:tasks.rados.rados.0.smithi107.stdout:666: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:33.710 INFO:tasks.rados.rados.0.smithi107.stdout:666: finishing write tid 5 to smithi10729407-18 2020-06-12T02:37:33.711 INFO:tasks.rados.rados.0.smithi107.stdout:666: finishing write tid 6 to smithi10729407-18 2020-06-12T02:37:33.711 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 141 (ObjNum 665 snap 0 seq_num 665) dirty exists 2020-06-12T02:37:33.711 INFO:tasks.rados.rados.0.smithi107.stdout:666: left oid 18 (ObjNum 665 snap 0 seq_num 665) 2020-06-12T02:37:33.711 INFO:tasks.rados.rados.0.smithi107.stdout:663: done (3 left) 2020-06-12T02:37:33.711 INFO:tasks.rados.rados.0.smithi107.stdout:666: done (2 left) 2020-06-12T02:37:33.712 INFO:tasks.rados.rados.0.smithi107.stdout:668: done (1 left) 2020-06-12T02:37:33.712 INFO:tasks.rados.rados.0.smithi107.stdout:669: delete oid 2 current snap is 0 2020-06-12T02:37:33.712 INFO:tasks.rados.rados.0.smithi107.stdout:667: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:33.712 INFO:tasks.rados.rados.0.smithi107.stdout:669: done (1 left) 2020-06-12T02:37:33.712 INFO:tasks.rados.rados.0.smithi107.stdout:670: delete oid 4 current snap is 0 2020-06-12T02:37:33.712 INFO:tasks.rados.rados.0.smithi107.stdout:667: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:33.713 INFO:tasks.rados.rados.0.smithi107.stdout:670: done (1 left) 2020-06-12T02:37:33.713 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:37:33.713 INFO:tasks.rados.rados.0.smithi107.stdout:671: seq_num 670 ranges {1548288=425984,1974272=483328,2457600=507904,2965504=212992} 2020-06-12T02:37:33.713 INFO:tasks.rados.rados.0.smithi107.stdout:671: writing smithi10729407-18 from 1548288 to 1974272 tid 1 2020-06-12T02:37:33.713 INFO:tasks.rados.rados.0.smithi107.stdout:671: writing smithi10729407-18 from 1974272 to 2457600 tid 2 2020-06-12T02:37:33.714 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2020-06-12T02:37:33.714 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:37:33.618+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:37:33.715 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-06-12T02:37:33.721 INFO:tasks.rados.rados.0.smithi107.stdout:671: writing smithi10729407-18 from 2457600 to 2965504 tid 3 2020-06-12T02:37:33.721 INFO:tasks.rados.rados.0.smithi107.stdout:671: writing smithi10729407-18 from 2965504 to 3178496 tid 4 2020-06-12T02:37:33.721 INFO:tasks.rados.rados.0.smithi107.stdout:672: delete oid 3 current snap is 0 2020-06-12T02:37:33.721 INFO:tasks.rados.rados.0.smithi107.stdout:667: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:33.722 INFO:tasks.rados.rados.0.smithi107.stdout:667: finishing write tid 5 to smithi10729407-11 2020-06-12T02:37:33.722 INFO:tasks.rados.rados.0.smithi107.stdout:667: finishing write tid 6 to smithi10729407-11 2020-06-12T02:37:33.722 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 97 (ObjNum 666 snap 0 seq_num 666) dirty exists 2020-06-12T02:37:33.722 INFO:tasks.rados.rados.0.smithi107.stdout:667: left oid 11 (ObjNum 666 snap 0 seq_num 666) 2020-06-12T02:37:33.722 INFO:tasks.rados.rados.0.smithi107.stdout:671: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:33.723 INFO:tasks.rados.rados.0.smithi107.stdout:667: done (2 left) 2020-06-12T02:37:33.723 INFO:tasks.rados.rados.0.smithi107.stdout:672: done (1 left) 2020-06-12T02:37:33.723 INFO:tasks.rados.rados.0.smithi107.stdout:673: delete oid 10 current snap is 0 2020-06-12T02:37:33.723 INFO:tasks.rados.rados.0.smithi107.stdout:671: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:33.723 INFO:tasks.rados.rados.0.smithi107.stdout:673: done (1 left) 2020-06-12T02:37:33.724 INFO:tasks.rados.rados.0.smithi107.stdout:674: delete oid 4 current snap is 0 2020-06-12T02:37:33.724 INFO:tasks.rados.rados.0.smithi107.stdout:671: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:33.724 INFO:tasks.rados.rados.0.smithi107.stdout:671: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:33.724 INFO:tasks.rados.rados.0.smithi107.stdout:671: finishing write tid 6 to smithi10729407-18 2020-06-12T02:37:33.724 INFO:tasks.rados.rados.0.smithi107.stdout:674: done (1 left) 2020-06-12T02:37:33.725 INFO:tasks.rados.rados.0.smithi107.stdout:675: delete oid 6 current snap is 0 2020-06-12T02:37:33.725 INFO:tasks.rados.rados.0.smithi107.stdout:671: finishing write tid 7 to smithi10729407-18 2020-06-12T02:37:33.725 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 146 (ObjNum 670 snap 0 seq_num 670) dirty exists 2020-06-12T02:37:33.725 INFO:tasks.rados.rados.0.smithi107.stdout:671: left oid 18 (ObjNum 670 snap 0 seq_num 670) 2020-06-12T02:37:33.725 INFO:tasks.rados.rados.0.smithi107.stdout:671: done (1 left) 2020-06-12T02:37:33.725 INFO:tasks.rados.rados.0.smithi107.stdout:675: done (0 left) 2020-06-12T02:37:33.726 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:37:33.726 INFO:tasks.rados.rados.0.smithi107.stdout:676: seq_num 675 ranges {2490368=507904,2998272=704512,3702784=499712} 2020-06-12T02:37:33.726 INFO:tasks.rados.rados.0.smithi107.stdout:676: writing smithi10729407-20 from 2490368 to 2998272 tid 1 2020-06-12T02:37:33.726 INFO:tasks.rados.rados.0.smithi107.stdout:676: writing smithi10729407-20 from 2998272 to 3702784 tid 2 2020-06-12T02:37:33.726 INFO:tasks.rados.rados.0.smithi107.stdout:676: writing smithi10729407-20 from 3702784 to 4202496 tid 3 2020-06-12T02:37:33.727 INFO:tasks.rados.rados.0.smithi107.stdout:677: delete oid 13 current snap is 0 2020-06-12T02:37:33.727 INFO:tasks.rados.rados.0.smithi107.stdout:676: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:33.727 INFO:tasks.rados.rados.0.smithi107.stdout:677: done (1 left) 2020-06-12T02:37:33.727 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:37:33.727 INFO:tasks.rados.rados.0.smithi107.stdout:678: seq_num 677 ranges {0=540672,540672=516096,1056768=655360,1712128=32768} 2020-06-12T02:37:33.728 INFO:tasks.rados.rados.0.smithi107.stdout:678: writing smithi10729407-5 from 0 to 540672 tid 1 2020-06-12T02:37:33.728 INFO:tasks.rados.rados.0.smithi107.stdout:678: writing smithi10729407-5 from 540672 to 1056768 tid 2 2020-06-12T02:37:33.728 INFO:tasks.rados.rados.0.smithi107.stdout:678: writing smithi10729407-5 from 1056768 to 1712128 tid 3 2020-06-12T02:37:33.728 INFO:tasks.rados.rados.0.smithi107.stdout:678: writing smithi10729407-5 from 1712128 to 1744896 tid 4 2020-06-12T02:37:33.728 INFO:tasks.rados.rados.0.smithi107.stdout:679: delete oid 4 current snap is 0 2020-06-12T02:37:33.729 INFO:tasks.rados.rados.0.smithi107.stdout:676: finishing write tid 2 to smithi10729407-20 2020-06-12T02:37:33.729 INFO:tasks.rados.rados.0.smithi107.stdout:676: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:33.729 INFO:tasks.rados.rados.0.smithi107.stdout:676: finishing write tid 5 to smithi10729407-20 2020-06-12T02:37:33.729 INFO:tasks.rados.rados.0.smithi107.stdout:676: finishing write tid 6 to smithi10729407-20 2020-06-12T02:37:33.729 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 67 (ObjNum 675 snap 0 seq_num 675) dirty exists 2020-06-12T02:37:33.730 INFO:tasks.rados.rados.0.smithi107.stdout:676: left oid 20 (ObjNum 675 snap 0 seq_num 675) 2020-06-12T02:37:33.730 INFO:tasks.rados.rados.0.smithi107.stdout:678: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:33.730 INFO:tasks.rados.rados.0.smithi107.stdout:676: done (2 left) 2020-06-12T02:37:33.730 INFO:tasks.rados.rados.0.smithi107.stdout:679: done (1 left) 2020-06-12T02:37:33.730 INFO:tasks.rados.rados.0.smithi107.stdout:680: delete oid 2 current snap is 0 2020-06-12T02:37:33.731 INFO:tasks.rados.rados.0.smithi107.stdout:680: done (1 left) 2020-06-12T02:37:33.731 INFO:tasks.rados.rados.0.smithi107.stdout:681: delete oid 19 current snap is 0 2020-06-12T02:37:33.731 INFO:tasks.rados.rados.0.smithi107.stdout:678: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:33.731 INFO:tasks.rados.rados.0.smithi107.stdout:678: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:33.731 INFO:tasks.rados.rados.0.smithi107.stdout:678: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:33.731 INFO:tasks.rados.rados.0.smithi107.stdout:681: done (1 left) 2020-06-12T02:37:33.732 INFO:tasks.rados.rados.0.smithi107.stdout:682: delete oid 7 current snap is 0 2020-06-12T02:37:33.732 INFO:tasks.rados.rados.0.smithi107.stdout:678: finishing write tid 6 to smithi10729407-5 2020-06-12T02:37:33.732 INFO:tasks.rados.rados.0.smithi107.stdout:678: finishing write tid 7 to smithi10729407-5 2020-06-12T02:37:33.732 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 71 (ObjNum 677 snap 0 seq_num 677) dirty exists 2020-06-12T02:37:33.732 INFO:tasks.rados.rados.0.smithi107.stdout:678: left oid 5 (ObjNum 677 snap 0 seq_num 677) 2020-06-12T02:37:33.733 INFO:tasks.rados.rados.0.smithi107.stdout:678: done (1 left) 2020-06-12T02:37:33.733 INFO:tasks.rados.rados.0.smithi107.stdout:682: done (0 left) 2020-06-12T02:37:33.733 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:37:33.733 INFO:tasks.rados.rados.0.smithi107.stdout:683: seq_num 682 ranges {1949696=630784,2580480=745472,3325952=442368,3768320=16384} 2020-06-12T02:37:33.734 INFO:tasks.rados.rados.0.smithi107.stdout:683: writing smithi10729407-1 from 1949696 to 2580480 tid 1 2020-06-12T02:37:33.734 INFO:tasks.rados.rados.0.smithi107.stdout:683: writing smithi10729407-1 from 2580480 to 3325952 tid 2 2020-06-12T02:37:33.734 INFO:tasks.rados.rados.0.smithi107.stdout:683: writing smithi10729407-1 from 3325952 to 3768320 tid 3 2020-06-12T02:37:33.734 INFO:tasks.rados.rados.0.smithi107.stdout:683: writing smithi10729407-1 from 3768320 to 3784704 tid 4 2020-06-12T02:37:33.734 INFO:tasks.rados.rados.0.smithi107.stdout:684: delete oid 17 current snap is 0 2020-06-12T02:37:33.735 INFO:tasks.rados.rados.0.smithi107.stdout:683: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:33.735 INFO:tasks.rados.rados.0.smithi107.stdout:684: done (1 left) 2020-06-12T02:37:33.735 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:37:33.735 INFO:tasks.rados.rados.0.smithi107.stdout:685: seq_num 684 ranges {4202496=663552,4866048=507904,5373952=409600,5783552=286720} 2020-06-12T02:37:33.737 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:33.733+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:33.744 INFO:tasks.rados.rados.0.smithi107.stdout:685: writing smithi10729407-20 from 4202496 to 4866048 tid 1 2020-06-12T02:37:33.747 INFO:tasks.rados.rados.0.smithi107.stdout:685: writing smithi10729407-20 from 4866048 to 5373952 tid 2 2020-06-12T02:37:33.750 INFO:tasks.rados.rados.0.smithi107.stdout:685: writing smithi10729407-20 from 5373952 to 5783552 tid 3 2020-06-12T02:37:33.752 INFO:tasks.rados.rados.0.smithi107.stdout:685: writing smithi10729407-20 from 5783552 to 6070272 tid 4 2020-06-12T02:37:33.752 INFO:tasks.rados.rados.0.smithi107.stdout:686: delete oid 6 current snap is 0 2020-06-12T02:37:33.753 INFO:tasks.rados.rados.0.smithi107.stdout:683: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:33.753 INFO:tasks.rados.rados.0.smithi107.stdout:683: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:33.753 INFO:tasks.rados.rados.0.smithi107.stdout:683: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:33.753 INFO:tasks.rados.rados.0.smithi107.stdout:683: finishing write tid 6 to smithi10729407-1 2020-06-12T02:37:33.753 INFO:tasks.rados.rados.0.smithi107.stdout:683: finishing write tid 7 to smithi10729407-1 2020-06-12T02:37:33.754 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 152 (ObjNum 682 snap 0 seq_num 682) dirty exists 2020-06-12T02:37:33.754 INFO:tasks.rados.rados.0.smithi107.stdout:683: left oid 1 (ObjNum 682 snap 0 seq_num 682) 2020-06-12T02:37:33.755 INFO:tasks.rados.rados.0.smithi107.stdout:685: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:33.755 INFO:tasks.rados.rados.0.smithi107.stdout:683: done (2 left) 2020-06-12T02:37:33.755 INFO:tasks.rados.rados.0.smithi107.stdout:686: done (1 left) 2020-06-12T02:37:33.755 INFO:tasks.rados.rados.0.smithi107.stdout:687: delete oid 7 current snap is 0 2020-06-12T02:37:33.756 INFO:tasks.rados.rados.0.smithi107.stdout:687: done (1 left) 2020-06-12T02:37:33.756 INFO:tasks.rados.rados.0.smithi107.stdout:688: delete oid 10 current snap is 0 2020-06-12T02:37:33.758 INFO:tasks.rados.rados.0.smithi107.stdout:688: done (1 left) 2020-06-12T02:37:33.759 INFO:tasks.rados.rados.0.smithi107.stdout:689: delete oid 16 current snap is 0 2020-06-12T02:37:33.760 INFO:tasks.rados.rados.0.smithi107.stdout:689: done (1 left) 2020-06-12T02:37:33.761 INFO:tasks.rados.rados.0.smithi107.stdout:690: delete oid 19 current snap is 0 2020-06-12T02:37:33.763 INFO:tasks.rados.rados.0.smithi107.stdout:690: done (1 left) 2020-06-12T02:37:33.763 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:37:33.763 INFO:tasks.rados.rados.0.smithi107.stdout:691: seq_num 690 ranges {0=761856,761856=663552,1425408=409600,1835008=131072} 2020-06-12T02:37:33.767 INFO:tasks.rados.rados.0.smithi107.stdout:691: writing smithi10729407-9 from 0 to 761856 tid 1 2020-06-12T02:37:33.771 INFO:tasks.rados.rados.0.smithi107.stdout:691: writing smithi10729407-9 from 761856 to 1425408 tid 2 2020-06-12T02:37:33.774 INFO:tasks.rados.rados.0.smithi107.stdout:691: writing smithi10729407-9 from 1425408 to 1835008 tid 3 2020-06-12T02:37:33.775 INFO:tasks.rados.rados.0.smithi107.stdout:691: writing smithi10729407-9 from 1835008 to 1966080 tid 4 2020-06-12T02:37:33.775 INFO:tasks.rados.rados.0.smithi107.stdout:692: delete oid 13 current snap is 0 2020-06-12T02:37:33.775 INFO:tasks.rados.rados.0.smithi107.stdout:685: finishing write tid 2 to smithi10729407-20 2020-06-12T02:37:33.776 INFO:tasks.rados.rados.0.smithi107.stdout:685: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:33.776 INFO:tasks.rados.rados.0.smithi107.stdout:685: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:33.776 INFO:tasks.rados.rados.0.smithi107.stdout:685: finishing write tid 6 to smithi10729407-20 2020-06-12T02:37:33.776 INFO:tasks.rados.rados.0.smithi107.stdout:685: finishing write tid 7 to smithi10729407-20 2020-06-12T02:37:33.776 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 72 (ObjNum 684 snap 0 seq_num 684) dirty exists 2020-06-12T02:37:33.777 INFO:tasks.rados.rados.0.smithi107.stdout:685: left oid 20 (ObjNum 684 snap 0 seq_num 684) 2020-06-12T02:37:33.777 INFO:tasks.rados.rados.0.smithi107.stdout:691: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:33.779 INFO:tasks.rados.rados.0.smithi107.stdout:685: done (2 left) 2020-06-12T02:37:33.779 INFO:tasks.rados.rados.0.smithi107.stdout:692: done (1 left) 2020-06-12T02:37:33.780 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:37:33.780 INFO:tasks.rados.rados.0.smithi107.stdout:693: seq_num 692 ranges {3784704=794624,4579328=483328,5062656=720896} 2020-06-12T02:37:33.803 INFO:tasks.rados.rados.0.smithi107.stdout:693: writing smithi10729407-1 from 3784704 to 4579328 tid 1 2020-06-12T02:37:33.806 INFO:tasks.rados.rados.0.smithi107.stdout:693: writing smithi10729407-1 from 4579328 to 5062656 tid 2 2020-06-12T02:37:33.811 INFO:tasks.rados.rados.0.smithi107.stdout:693: writing smithi10729407-1 from 5062656 to 5783552 tid 3 2020-06-12T02:37:33.811 INFO:tasks.rados.rados.0.smithi107.stdout:694: delete oid 12 current snap is 0 2020-06-12T02:37:33.811 INFO:tasks.rados.rados.0.smithi107.stdout:691: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:33.811 INFO:tasks.rados.rados.0.smithi107.stdout:691: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:33.812 INFO:tasks.rados.rados.0.smithi107.stdout:691: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:33.812 INFO:tasks.rados.rados.0.smithi107.stdout:691: finishing write tid 6 to smithi10729407-9 2020-06-12T02:37:33.812 INFO:tasks.rados.rados.0.smithi107.stdout:691: finishing write tid 7 to smithi10729407-9 2020-06-12T02:37:33.812 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 151 (ObjNum 690 snap 0 seq_num 690) dirty exists 2020-06-12T02:37:33.813 INFO:tasks.rados.rados.0.smithi107.stdout:691: left oid 9 (ObjNum 690 snap 0 seq_num 690) 2020-06-12T02:37:33.813 INFO:tasks.rados.rados.0.smithi107.stdout:691: done (2 left) 2020-06-12T02:37:33.814 INFO:tasks.rados.rados.0.smithi107.stdout:694: done (1 left) 2020-06-12T02:37:33.814 INFO:tasks.rados.rados.0.smithi107.stdout:695: delete oid 15 current snap is 0 2020-06-12T02:37:33.814 INFO:tasks.rados.rados.0.smithi107.stdout:695: done (1 left) 2020-06-12T02:37:33.815 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:37:33.817 INFO:tasks.rados.rados.0.smithi107.stdout:696: seq_num 695 ranges {0=442368,442368=540672,983040=598016,1581056=466944} 2020-06-12T02:37:33.817 INFO:tasks.rados.rados.0.smithi107.stdout:696: writing smithi10729407-12 from 0 to 442368 tid 1 2020-06-12T02:37:33.822 INFO:tasks.rados.rados.0.smithi107.stdout:696: writing smithi10729407-12 from 442368 to 983040 tid 2 2020-06-12T02:37:33.824 INFO:tasks.rados.rados.0.smithi107.stdout:696: writing smithi10729407-12 from 983040 to 1581056 tid 3 2020-06-12T02:37:33.827 INFO:tasks.rados.rados.0.smithi107.stdout:696: writing smithi10729407-12 from 1581056 to 2048000 tid 4 2020-06-12T02:37:33.827 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:37:33.827 INFO:tasks.rados.rados.0.smithi107.stdout:697: seq_num 696 ranges {0=458752,458752=417792,876544=753664,1630208=434176} 2020-06-12T02:37:33.829 INFO:tasks.rados.rados.0.smithi107.stdout:697: writing smithi10729407-15 from 0 to 458752 tid 1 2020-06-12T02:37:33.832 INFO:tasks.rados.rados.0.smithi107.stdout:697: writing smithi10729407-15 from 458752 to 876544 tid 2 2020-06-12T02:37:33.837 INFO:tasks.rados.rados.0.smithi107.stdout:697: writing smithi10729407-15 from 876544 to 1630208 tid 3 2020-06-12T02:37:33.837 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:33.834+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:33.840 INFO:tasks.rados.rados.0.smithi107.stdout:697: writing smithi10729407-15 from 1630208 to 2064384 tid 4 2020-06-12T02:37:33.840 INFO:tasks.rados.rados.0.smithi107.stdout:698: delete oid 16 current snap is 0 2020-06-12T02:37:33.840 INFO:tasks.rados.rados.0.smithi107.stdout:693: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:33.840 INFO:tasks.rados.rados.0.smithi107.stdout:693: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:33.841 INFO:tasks.rados.rados.0.smithi107.stdout:693: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:33.841 INFO:tasks.rados.rados.0.smithi107.stdout:693: finishing write tid 5 to smithi10729407-1 2020-06-12T02:37:33.841 INFO:tasks.rados.rados.0.smithi107.stdout:693: finishing write tid 6 to smithi10729407-1 2020-06-12T02:37:33.841 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 157 (ObjNum 692 snap 0 seq_num 692) dirty exists 2020-06-12T02:37:33.841 INFO:tasks.rados.rados.0.smithi107.stdout:693: left oid 1 (ObjNum 692 snap 0 seq_num 692) 2020-06-12T02:37:33.842 INFO:tasks.rados.rados.0.smithi107.stdout:696: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:33.842 INFO:tasks.rados.rados.0.smithi107.stdout:696: finishing write tid 2 to smithi10729407-12 2020-06-12T02:37:33.842 INFO:tasks.rados.rados.0.smithi107.stdout:696: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:33.842 INFO:tasks.rados.rados.0.smithi107.stdout:696: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:33.842 INFO:tasks.rados.rados.0.smithi107.stdout:697: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:33.843 INFO:tasks.rados.rados.0.smithi107.stdout:696: finishing write tid 6 to smithi10729407-12 2020-06-12T02:37:33.843 INFO:tasks.rados.rados.0.smithi107.stdout:696: finishing write tid 7 to smithi10729407-12 2020-06-12T02:37:33.843 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 113 (ObjNum 695 snap 0 seq_num 695) dirty exists 2020-06-12T02:37:33.843 INFO:tasks.rados.rados.0.smithi107.stdout:696: left oid 12 (ObjNum 695 snap 0 seq_num 695) 2020-06-12T02:37:33.844 INFO:tasks.rados.rados.0.smithi107.stdout:693: done (3 left) 2020-06-12T02:37:33.844 INFO:tasks.rados.rados.0.smithi107.stdout:696: done (2 left) 2020-06-12T02:37:33.844 INFO:tasks.rados.rados.0.smithi107.stdout:698: done (1 left) 2020-06-12T02:37:33.844 INFO:tasks.rados.rados.0.smithi107.stdout:699: delete oid 2 current snap is 0 2020-06-12T02:37:33.845 INFO:tasks.rados.rados.0.smithi107.stdout:697: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:33.845 INFO:tasks.rados.rados.0.smithi107.stdout:699: done (1 left) 2020-06-12T02:37:33.845 INFO:tasks.rados.rados.0.smithi107.stdout:700: delete oid 19 current snap is 0 2020-06-12T02:37:33.846 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-06-12T02:37:33.847 INFO:tasks.rados.rados.0.smithi107.stdout:700: done (1 left) 2020-06-12T02:37:33.847 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:37:33.847 INFO:tasks.rados.rados.0.smithi107.stdout:701: seq_num 700 ranges {0=532480,532480=409600,942080=737280,1679360=458752} 2020-06-12T02:37:33.849 INFO:tasks.rados.rados.0.smithi107.stdout:701: writing smithi10729407-17 from 0 to 532480 tid 1 2020-06-12T02:37:33.851 INFO:tasks.rados.rados.0.smithi107.stdout:701: writing smithi10729407-17 from 532480 to 942080 tid 2 2020-06-12T02:37:33.856 INFO:tasks.rados.rados.0.smithi107.stdout:701: writing smithi10729407-17 from 942080 to 1679360 tid 3 2020-06-12T02:37:33.859 INFO:tasks.rados.rados.0.smithi107.stdout:701: writing smithi10729407-17 from 1679360 to 2138112 tid 4 2020-06-12T02:37:33.859 INFO:tasks.rados.rados.0.smithi107.stdout:702: delete oid 12 current snap is 0 2020-06-12T02:37:33.859 INFO:tasks.rados.rados.0.smithi107.stdout:697: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:33.860 INFO:tasks.rados.rados.0.smithi107.stdout:697: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:33.860 INFO:tasks.rados.rados.0.smithi107.stdout:697: finishing write tid 6 to smithi10729407-15 2020-06-12T02:37:33.860 INFO:tasks.rados.rados.0.smithi107.stdout:697: finishing write tid 7 to smithi10729407-15 2020-06-12T02:37:33.860 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 129 (ObjNum 696 snap 0 seq_num 696) dirty exists 2020-06-12T02:37:33.860 INFO:tasks.rados.rados.0.smithi107.stdout:697: left oid 15 (ObjNum 696 snap 0 seq_num 696) 2020-06-12T02:37:33.861 INFO:tasks.rados.rados.0.smithi107.stdout:701: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:33.861 INFO:tasks.rados.rados.0.smithi107.stdout:701: finishing write tid 2 to smithi10729407-17 2020-06-12T02:37:33.862 INFO:tasks.rados.rados.0.smithi107.stdout:697: done (2 left) 2020-06-12T02:37:33.862 INFO:tasks.rados.rados.0.smithi107.stdout:702: done (1 left) 2020-06-12T02:37:33.863 INFO:tasks.rados.rados.0.smithi107.stdout:703: delete oid 1 current snap is 0 2020-06-12T02:37:33.865 INFO:tasks.rados.rados.0.smithi107.stdout:703: done (1 left) 2020-06-12T02:37:33.865 INFO:tasks.rados.rados.0.smithi107.stdout:704: delete oid 2 current snap is 0 2020-06-12T02:37:33.867 INFO:tasks.rados.rados.0.smithi107.stdout:704: done (1 left) 2020-06-12T02:37:33.867 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:37:33.867 INFO:tasks.rados.rados.0.smithi107.stdout:705: seq_num 704 ranges {2359296=598016,2957312=745472,3702784=712704,4415488=147456} 2020-06-12T02:37:33.870 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:37:33.870 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:37:33.871 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:33.871 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1567 3.fs1 3:fc7e0bf7:::smithi10729407-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e26)", 2020-06-12T02:37:33.871 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.858586+0000", 2020-06-12T02:37:33.871 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.010148384, 2020-06-12T02:37:33.871 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010172884, 2020-06-12T02:37:33.872 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:33.872 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:33.872 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:33.872 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:33.872 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:33.872 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1567 2020-06-12T02:37:33.873 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:33.873 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:33.873 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:33.873 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:33.873 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.858586+0000", 2020-06-12T02:37:33.874 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:33.874 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:33.874 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:33.874 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:33.874 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.858586+0000", 2020-06-12T02:37:33.875 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2626999999999999e-05 2020-06-12T02:37:33.875 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:33.875 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:33.875 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:33.875 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.858609+0000", 2020-06-12T02:37:33.875 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4100000000000001e-06 2020-06-12T02:37:33.876 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:33.876 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:33.876 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:33.876 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.858613+0000", 2020-06-12T02:37:33.876 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5920000000000003e-06 2020-06-12T02:37:33.877 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:33.877 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:33.877 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:33.877 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.858621+0000", 2020-06-12T02:37:33.877 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.283e-06 2020-06-12T02:37:33.877 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:33.878 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:33.878 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:33.878 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.858623+0000", 2020-06-12T02:37:33.878 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0078603089999999994 2020-06-12T02:37:33.878 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:33.879 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:33.879 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:33.879 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.866483+0000", 2020-06-12T02:37:33.879 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000113687 2020-06-12T02:37:33.879 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:33.879 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:33.880 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:33.880 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.866597+0000", 2020-06-12T02:37:33.880 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00182632 2020-06-12T02:37:33.880 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:33.880 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:33.881 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:33.881 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.868423+0000", 2020-06-12T02:37:33.881 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7214999999999997e-05 2020-06-12T02:37:33.881 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:33.881 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:33.881 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:33.882 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.868521+0000", 2020-06-12T02:37:33.882 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0099344429999999994 2020-06-12T02:37:33.882 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:33.882 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:33.882 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:33.883 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:33.883 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:33.883 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpRead(3.fs1 26/21 ECSubRead(tid=726, to_read={3:fc7e0bf7:::smithi10729407-17:head=0,4096,0}, subchunks={3:fc7e0bf7:::smithi10729407-17:head=[0,1]}, attrs_to_read=))", 2020-06-12T02:37:33.883 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.868705+0000", 2020-06-12T02:37:33.883 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 2.9852000000000001e-05, 2020-06-12T02:37:33.883 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012115, 2020-06-12T02:37:33.884 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:33.884 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:37:33.884 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:33.884 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:33.884 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:33.885 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.868705+0000", 2020-06-12T02:37:33.885 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:33.885 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:33.885 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:33.885 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:33.885 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.868705+0000", 2020-06-12T02:37:33.886 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:33.886 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:33.886 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:33.886 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:33.886 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.868705+0000", 2020-06-12T02:37:33.887 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:33.887 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:33.887 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:33.887 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:33.887 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.868705+0000", 2020-06-12T02:37:33.887 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.9278000000000001e-05 2020-06-12T02:37:33.888 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:33.888 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:33.888 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:33.888 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.868724+0000", 2020-06-12T02:37:33.888 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6490000000000002e-06 2020-06-12T02:37:33.889 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:33.889 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:33.889 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:33.889 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.868731+0000", 2020-06-12T02:37:33.889 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3912999999999995e-05 2020-06-12T02:37:33.889 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:33.890 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:33.890 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:33.890 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.868804+0000", 2020-06-12T02:37:33.890 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9840000000000006e-05 2020-06-12T02:37:33.890 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:33.891 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:33.891 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:33.891 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:33.891 INFO:teuthology.orchestra.run.smithi188.stdout: ], 2020-06-12T02:37:33.891 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 2 2020-06-12T02:37:33.891 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:37:33.893 INFO:tasks.rados.rados.0.smithi107.stdout:705: writing smithi10729407-11 from 2359296 to 2957312 tid 1 2020-06-12T02:37:33.893 INFO:tasks.rados.rados.0.smithi107.stdout:705: writing smithi10729407-11 from 2957312 to 3702784 tid 2 2020-06-12T02:37:33.893 INFO:tasks.rados.rados.0.smithi107.stdout:705: writing smithi10729407-11 from 3702784 to 4415488 tid 3 2020-06-12T02:37:33.893 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-06-12T02:37:33.894 INFO:tasks.rados.rados.0.smithi107.stdout:705: writing smithi10729407-11 from 4415488 to 4562944 tid 4 2020-06-12T02:37:33.895 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:37:33.895 INFO:tasks.rados.rados.0.smithi107.stdout:706: seq_num 705 ranges {0=614400,614400=630784,1245184=524288,1769472=450560} 2020-06-12T02:37:33.897 INFO:tasks.rados.rados.0.smithi107.stdout:706: writing smithi10729407-16 from 0 to 614400 tid 1 2020-06-12T02:37:33.901 INFO:tasks.rados.rados.0.smithi107.stdout:706: writing smithi10729407-16 from 614400 to 1245184 tid 2 2020-06-12T02:37:33.904 INFO:tasks.rados.rados.0.smithi107.stdout:706: writing smithi10729407-16 from 1245184 to 1769472 tid 3 2020-06-12T02:37:33.907 INFO:tasks.rados.rados.0.smithi107.stdout:706: writing smithi10729407-16 from 1769472 to 2220032 tid 4 2020-06-12T02:37:33.907 INFO:tasks.rados.rados.0.smithi107.stdout:707: delete oid 2 current snap is 0 2020-06-12T02:37:33.908 INFO:tasks.rados.rados.0.smithi107.stdout:701: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:33.908 INFO:tasks.rados.rados.0.smithi107.stdout:701: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:33.908 INFO:tasks.rados.rados.0.smithi107.stdout:701: finishing write tid 6 to smithi10729407-17 2020-06-12T02:37:33.908 INFO:tasks.rados.rados.0.smithi107.stdout:701: finishing write tid 7 to smithi10729407-17 2020-06-12T02:37:33.908 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 129 (ObjNum 700 snap 0 seq_num 700) dirty exists 2020-06-12T02:37:33.908 INFO:tasks.rados.rados.0.smithi107.stdout:701: left oid 17 (ObjNum 700 snap 0 seq_num 700) 2020-06-12T02:37:33.909 INFO:tasks.rados.rados.0.smithi107.stdout:705: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:33.909 INFO:tasks.rados.rados.0.smithi107.stdout:705: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:33.911 INFO:tasks.rados.rados.0.smithi107.stdout:701: done (3 left) 2020-06-12T02:37:33.912 INFO:tasks.rados.rados.0.smithi107.stdout:707: done (2 left) 2020-06-12T02:37:33.912 INFO:tasks.rados.rados.0.smithi107.stdout:708: delete oid 9 current snap is 0 2020-06-12T02:37:33.912 INFO:tasks.rados.rados.0.smithi107.stdout:705: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:33.912 INFO:tasks.rados.rados.0.smithi107.stdout:705: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:33.912 INFO:tasks.rados.rados.0.smithi107.stdout:705: finishing write tid 6 to smithi10729407-11 2020-06-12T02:37:33.913 INFO:tasks.rados.rados.0.smithi107.stdout:706: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:33.913 INFO:tasks.rados.rados.0.smithi107.stdout:706: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:33.914 INFO:tasks.rados.rados.0.smithi107.stdout:705: finishing write tid 7 to smithi10729407-11 2020-06-12T02:37:33.914 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 102 (ObjNum 704 snap 0 seq_num 704) dirty exists 2020-06-12T02:37:33.914 INFO:tasks.rados.rados.0.smithi107.stdout:705: left oid 11 (ObjNum 704 snap 0 seq_num 704) 2020-06-12T02:37:33.915 INFO:tasks.rados.rados.0.smithi107.stdout:705: done (2 left) 2020-06-12T02:37:33.915 INFO:tasks.rados.rados.0.smithi107.stdout:708: done (1 left) 2020-06-12T02:37:33.915 INFO:tasks.rados.rados.0.smithi107.stdout:709: delete oid 14 current snap is 0 2020-06-12T02:37:33.915 INFO:tasks.rados.rados.0.smithi107.stdout:706: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:33.916 INFO:tasks.rados.rados.0.smithi107.stdout:706: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:33.916 INFO:tasks.rados.rados.0.smithi107.stdout:706: finishing write tid 6 to smithi10729407-16 2020-06-12T02:37:33.917 INFO:tasks.rados.rados.0.smithi107.stdout:709: done (1 left) 2020-06-12T02:37:33.917 INFO:tasks.rados.rados.0.smithi107.stdout:710: delete oid 15 current snap is 0 2020-06-12T02:37:33.918 INFO:tasks.rados.rados.0.smithi107.stdout:706: finishing write tid 7 to smithi10729407-16 2020-06-12T02:37:33.919 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 129 (ObjNum 705 snap 0 seq_num 705) dirty exists 2020-06-12T02:37:33.919 INFO:tasks.rados.rados.0.smithi107.stdout:706: left oid 16 (ObjNum 705 snap 0 seq_num 705) 2020-06-12T02:37:33.920 INFO:tasks.rados.rados.0.smithi107.stdout:706: done (1 left) 2020-06-12T02:37:33.920 INFO:tasks.rados.rados.0.smithi107.stdout:710: done (0 left) 2020-06-12T02:37:33.920 INFO:tasks.rados.rados.0.smithi107.stdout:711: delete oid 18 current snap is 0 2020-06-12T02:37:33.922 INFO:tasks.rados.rados.0.smithi107.stdout:711: done (0 left) 2020-06-12T02:37:33.922 INFO:tasks.rados.rados.0.smithi107.stdout:712: delete oid 10 current snap is 0 2020-06-12T02:37:33.924 INFO:tasks.rados.rados.0.smithi107.stdout:712: done (0 left) 2020-06-12T02:37:33.924 INFO:tasks.rados.rados.0.smithi107.stdout:713: delete oid 7 current snap is 0 2020-06-12T02:37:33.926 INFO:tasks.rados.rados.0.smithi107.stdout:713: done (0 left) 2020-06-12T02:37:33.926 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:37:33.926 INFO:tasks.rados.rados.0.smithi107.stdout:714: seq_num 713 ranges {0=745472,745472=557056,1302528=483328,1785856=565248} 2020-06-12T02:37:33.930 INFO:tasks.rados.rados.0.smithi107.stdout:714: writing smithi10729407-2 from 0 to 745472 tid 1 2020-06-12T02:37:33.934 INFO:tasks.rados.rados.0.smithi107.stdout:714: writing smithi10729407-2 from 745472 to 1302528 tid 2 2020-06-12T02:37:33.937 INFO:tasks.rados.rados.0.smithi107.stdout:714: writing smithi10729407-2 from 1302528 to 1785856 tid 3 2020-06-12T02:37:33.939 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:33.936+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:33.940 INFO:tasks.rados.rados.0.smithi107.stdout:714: writing smithi10729407-2 from 1785856 to 2351104 tid 4 2020-06-12T02:37:33.941 INFO:tasks.rados.rados.0.smithi107.stdout:715: delete oid 20 current snap is 0 2020-06-12T02:37:33.941 INFO:tasks.rados.rados.0.smithi107.stdout:714: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:33.943 INFO:tasks.rados.rados.0.smithi107.stdout:715: done (1 left) 2020-06-12T02:37:33.944 INFO:tasks.rados.rados.0.smithi107.stdout:716: delete oid 9 current snap is 0 2020-06-12T02:37:33.946 INFO:tasks.rados.rados.0.smithi107.stdout:716: done (1 left) 2020-06-12T02:37:33.946 INFO:tasks.rados.rados.0.smithi107.stdout:717: delete oid 19 current snap is 0 2020-06-12T02:37:33.947 INFO:tasks.rados.rados.0.smithi107.stdout:714: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:33.948 INFO:tasks.rados.rados.0.smithi107.stdout:717: done (1 left) 2020-06-12T02:37:33.949 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:37:33.949 INFO:tasks.rados.rados.0.smithi107.stdout:718: seq_num 717 ranges {0=16384} 2020-06-12T02:37:33.949 INFO:tasks.rados.rados.0.smithi107.stdout:718: writing smithi10729407-7 from 0 to 16384 tid 1 2020-06-12T02:37:33.949 INFO:tasks.rados.rados.0.smithi107.stdout:719: delete oid 17 current snap is 0 2020-06-12T02:37:33.950 INFO:tasks.rados.rados.0.smithi107.stdout:714: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:33.951 INFO:tasks.rados.rados.0.smithi107.stdout:719: done (2 left) 2020-06-12T02:37:33.951 INFO:tasks.rados.rados.0.smithi107.stdout:720: delete oid 6 current snap is 0 2020-06-12T02:37:33.953 INFO:tasks.rados.rados.0.smithi107.stdout:718: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:33.954 INFO:tasks.rados.rados.0.smithi107.stdout:714: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:33.954 INFO:tasks.rados.rados.0.smithi107.stdout:714: finishing write tid 6 to smithi10729407-2 2020-06-12T02:37:33.955 INFO:tasks.rados.rados.0.smithi107.stdout:714: finishing write tid 7 to smithi10729407-2 2020-06-12T02:37:33.956 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 77 (ObjNum 713 snap 0 seq_num 713) dirty exists 2020-06-12T02:37:33.956 INFO:tasks.rados.rados.0.smithi107.stdout:714: left oid 2 (ObjNum 713 snap 0 seq_num 713) 2020-06-12T02:37:33.956 INFO:tasks.rados.rados.0.smithi107.stdout:714: done (2 left) 2020-06-12T02:37:33.956 INFO:tasks.rados.rados.0.smithi107.stdout:720: done (1 left) 2020-06-12T02:37:33.957 INFO:tasks.rados.rados.0.smithi107.stdout:721: delete oid 13 current snap is 0 2020-06-12T02:37:33.959 INFO:tasks.rados.rados.0.smithi107.stdout:718: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:33.959 INFO:tasks.rados.rados.0.smithi107.stdout:721: done (1 left) 2020-06-12T02:37:33.960 INFO:tasks.rados.rados.0.smithi107.stdout:722: delete oid 19 current snap is 0 2020-06-12T02:37:33.961 INFO:tasks.rados.rados.0.smithi107.stdout:718: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:33.961 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 134 (ObjNum 717 snap 0 seq_num 717) dirty exists 2020-06-12T02:37:33.961 INFO:tasks.rados.rados.0.smithi107.stdout:718: left oid 7 (ObjNum 717 snap 0 seq_num 717) 2020-06-12T02:37:33.962 INFO:tasks.rados.rados.0.smithi107.stdout:718: done (1 left) 2020-06-12T02:37:33.962 INFO:tasks.rados.rados.0.smithi107.stdout:722: done (0 left) 2020-06-12T02:37:33.962 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:37:33.962 INFO:tasks.rados.rados.0.smithi107.stdout:723: seq_num 722 ranges {0=98304} 2020-06-12T02:37:33.963 INFO:tasks.rados.rados.0.smithi107.stdout:723: writing smithi10729407-15 from 0 to 98304 tid 1 2020-06-12T02:37:33.963 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:37:33.963 INFO:tasks.rados.rados.0.smithi107.stdout:724: seq_num 723 ranges {0=114688} 2020-06-12T02:37:33.963 INFO:tasks.rados.rados.0.smithi107.stdout:724: writing smithi10729407-4 from 0 to 114688 tid 1 2020-06-12T02:37:33.964 INFO:tasks.rados.rados.0.smithi107.stdout:725: delete oid 11 current snap is 0 2020-06-12T02:37:33.965 INFO:tasks.rados.rados.0.smithi107.stdout:723: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:33.966 INFO:tasks.rados.rados.0.smithi107.stdout:725: done (2 left) 2020-06-12T02:37:33.967 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:37:33.967 INFO:tasks.rados.rados.0.smithi107.stdout:726: seq_num 725 ranges {0=147456} 2020-06-12T02:37:33.967 INFO:tasks.rados.rados.0.smithi107.stdout:726: writing smithi10729407-12 from 0 to 147456 tid 1 2020-06-12T02:37:33.968 INFO:tasks.rados.rados.0.smithi107.stdout:727: delete oid 5 current snap is 0 2020-06-12T02:37:33.969 INFO:tasks.rados.rados.0.smithi107.stdout:724: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:33.969 INFO:tasks.rados.rados.0.smithi107.stdout:724: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:33.970 INFO:tasks.rados.rados.0.smithi107.stdout:723: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:33.970 INFO:tasks.rados.rados.0.smithi107.stdout:724: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:33.970 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 132 (ObjNum 723 snap 0 seq_num 723) dirty exists 2020-06-12T02:37:33.971 INFO:tasks.rados.rados.0.smithi107.stdout:724: left oid 4 (ObjNum 723 snap 0 seq_num 723) 2020-06-12T02:37:33.971 INFO:tasks.rados.rados.0.smithi107.stdout:724: done (3 left) 2020-06-12T02:37:33.972 INFO:tasks.rados.rados.0.smithi107.stdout:727: done (2 left) 2020-06-12T02:37:33.972 INFO:tasks.rados.rados.0.smithi107.stdout:728: delete oid 17 current snap is 0 2020-06-12T02:37:33.972 INFO:tasks.rados.rados.0.smithi107.stdout:726: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:33.972 INFO:tasks.rados.rados.0.smithi107.stdout:723: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:33.972 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 132 (ObjNum 722 snap 0 seq_num 722) dirty exists 2020-06-12T02:37:33.973 INFO:tasks.rados.rados.0.smithi107.stdout:723: left oid 15 (ObjNum 722 snap 0 seq_num 722) 2020-06-12T02:37:33.973 INFO:tasks.rados.rados.0.smithi107.stdout:726: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:33.975 INFO:tasks.rados.rados.0.smithi107.stdout:723: done (2 left) 2020-06-12T02:37:33.975 INFO:tasks.rados.rados.0.smithi107.stdout:728: done (1 left) 2020-06-12T02:37:33.975 INFO:tasks.rados.rados.0.smithi107.stdout:729: delete oid 17 current snap is 0 2020-06-12T02:37:33.975 INFO:tasks.rados.rados.0.smithi107.stdout:726: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:33.976 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 119 (ObjNum 725 snap 0 seq_num 725) dirty exists 2020-06-12T02:37:33.976 INFO:tasks.rados.rados.0.smithi107.stdout:726: left oid 12 (ObjNum 725 snap 0 seq_num 725) 2020-06-12T02:37:33.976 INFO:tasks.rados.rados.0.smithi107.stdout:726: done (1 left) 2020-06-12T02:37:33.977 INFO:tasks.rados.rados.0.smithi107.stdout:729: done (0 left) 2020-06-12T02:37:33.977 INFO:tasks.rados.rados.0.smithi107.stdout:730: delete oid 6 current snap is 0 2020-06-12T02:37:33.977 INFO:tasks.rados.rados.0.smithi107.stdout:730: done (0 left) 2020-06-12T02:37:33.978 INFO:tasks.rados.rados.0.smithi107.stdout:731: delete oid 8 current snap is 0 2020-06-12T02:37:33.979 INFO:tasks.rados.rados.0.smithi107.stdout:731: done (0 left) 2020-06-12T02:37:33.979 INFO:tasks.rados.rados.0.smithi107.stdout:732: delete oid 15 current snap is 0 2020-06-12T02:37:33.981 INFO:tasks.rados.rados.0.smithi107.stdout:732: done (0 left) 2020-06-12T02:37:33.981 INFO:tasks.rados.rados.0.smithi107.stdout:733: delete oid 16 current snap is 0 2020-06-12T02:37:33.983 INFO:tasks.rados.rados.0.smithi107.stdout:733: done (0 left) 2020-06-12T02:37:33.983 INFO:tasks.rados.rados.0.smithi107.stdout:734: delete oid 5 current snap is 0 2020-06-12T02:37:33.984 INFO:tasks.rados.rados.0.smithi107.stdout:734: done (0 left) 2020-06-12T02:37:33.985 INFO:tasks.rados.rados.0.smithi107.stdout:735: delete oid 14 current snap is 0 2020-06-12T02:37:33.987 INFO:tasks.rados.rados.0.smithi107.stdout:735: done (0 left) 2020-06-12T02:37:33.987 INFO:tasks.rados.rados.0.smithi107.stdout:736: delete oid 1 current snap is 0 2020-06-12T02:37:33.989 INFO:tasks.rados.rados.0.smithi107.stdout:736: done (0 left) 2020-06-12T02:37:33.989 INFO:tasks.rados.rados.0.smithi107.stdout:737: delete oid 2 current snap is 0 2020-06-12T02:37:33.992 INFO:tasks.rados.rados.0.smithi107.stdout:737: done (0 left) 2020-06-12T02:37:33.992 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:37:33.992 INFO:tasks.rados.rados.0.smithi107.stdout:738: seq_num 737 ranges {0=352256} 2020-06-12T02:37:33.994 INFO:tasks.rados.rados.0.smithi107.stdout:738: writing smithi10729407-18 from 0 to 352256 tid 1 2020-06-12T02:37:33.994 INFO:tasks.rados.rados.0.smithi107.stdout:739: delete oid 16 current snap is 0 2020-06-12T02:37:33.997 INFO:tasks.rados.rados.0.smithi107.stdout:739: done (1 left) 2020-06-12T02:37:33.997 INFO:tasks.rados.rados.0.smithi107.stdout:740: delete oid 4 current snap is 0 2020-06-12T02:37:33.999 INFO:tasks.rados.rados.0.smithi107.stdout:738: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:34.001 INFO:tasks.rados.rados.0.smithi107.stdout:740: done (1 left) 2020-06-12T02:37:34.002 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:37:34.002 INFO:tasks.rados.rados.0.smithi107.stdout:741: seq_num 740 ranges {0=401408} 2020-06-12T02:37:34.003 INFO:tasks.rados.rados.0.smithi107.stdout:741: writing smithi10729407-4 from 0 to 401408 tid 1 2020-06-12T02:37:34.003 INFO:tasks.rados.rados.0.smithi107.stdout:742: delete oid 12 current snap is 0 2020-06-12T02:37:34.003 INFO:tasks.rados.rados.0.smithi107.stdout:738: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:34.005 INFO:tasks.rados.rados.0.smithi107.stdout:738: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:34.005 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 156 (ObjNum 737 snap 0 seq_num 737) dirty exists 2020-06-12T02:37:34.005 INFO:tasks.rados.rados.0.smithi107.stdout:738: left oid 18 (ObjNum 737 snap 0 seq_num 737) 2020-06-12T02:37:34.006 INFO:tasks.rados.rados.0.smithi107.stdout:738: done (2 left) 2020-06-12T02:37:34.006 INFO:tasks.rados.rados.0.smithi107.stdout:742: done (1 left) 2020-06-12T02:37:34.006 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:37:34.006 INFO:tasks.rados.rados.0.smithi107.stdout:743: seq_num 742 ranges {0=434176} 2020-06-12T02:37:34.008 INFO:tasks.rados.rados.0.smithi107.stdout:743: writing smithi10729407-15 from 0 to 434176 tid 1 2020-06-12T02:37:34.008 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:37:34.009 INFO:tasks.rados.rados.0.smithi107.stdout:744: seq_num 743 ranges {0=450560} 2020-06-12T02:37:34.011 INFO:tasks.rados.rados.0.smithi107.stdout:744: writing smithi10729407-3 from 0 to 450560 tid 1 2020-06-12T02:37:34.011 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:37:34.012 INFO:tasks.rados.rados.0.smithi107.stdout:745: seq_num 744 ranges {0=466944} 2020-06-12T02:37:34.014 INFO:tasks.rados.rados.0.smithi107.stdout:745: writing smithi10729407-10 from 0 to 466944 tid 1 2020-06-12T02:37:34.015 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:37:34.015 INFO:tasks.rados.rados.0.smithi107.stdout:746: seq_num 745 ranges {0=483328} 2020-06-12T02:37:34.022 INFO:tasks.rados.rados.0.smithi107.stdout:746: writing smithi10729407-20 from 0 to 483328 tid 1 2020-06-12T02:37:34.023 INFO:tasks.rados.rados.0.smithi107.stdout:747: delete oid 17 current snap is 0 2020-06-12T02:37:34.023 INFO:tasks.rados.rados.0.smithi107.stdout:741: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:34.023 INFO:tasks.rados.rados.0.smithi107.stdout:741: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:34.023 INFO:tasks.rados.rados.0.smithi107.stdout:741: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:34.023 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 137 (ObjNum 740 snap 0 seq_num 740) dirty exists 2020-06-12T02:37:34.024 INFO:tasks.rados.rados.0.smithi107.stdout:741: left oid 4 (ObjNum 740 snap 0 seq_num 740) 2020-06-12T02:37:34.024 INFO:tasks.rados.rados.0.smithi107.stdout:743: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:34.024 INFO:tasks.rados.rados.0.smithi107.stdout:744: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:34.025 INFO:tasks.rados.rados.0.smithi107.stdout:743: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:34.026 INFO:tasks.rados.rados.0.smithi107.stdout:741: done (5 left) 2020-06-12T02:37:34.026 INFO:tasks.rados.rados.0.smithi107.stdout:747: done (4 left) 2020-06-12T02:37:34.026 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:37:34.026 INFO:tasks.rados.rados.0.smithi107.stdout:748: seq_num 747 ranges {0=516096} 2020-06-12T02:37:34.026 INFO:tasks.rados.rados.0.smithi107.stdout:748: writing smithi10729407-5 from 0 to 516096 tid 1 2020-06-12T02:37:34.027 INFO:tasks.rados.rados.0.smithi107.stdout:749: delete oid 16 current snap is 0 2020-06-12T02:37:34.027 INFO:tasks.rados.rados.0.smithi107.stdout:745: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:34.027 INFO:tasks.rados.rados.0.smithi107.stdout:745: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:34.027 INFO:tasks.rados.rados.0.smithi107.stdout:743: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:34.027 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 136 (ObjNum 742 snap 0 seq_num 742) dirty exists 2020-06-12T02:37:34.028 INFO:tasks.rados.rados.0.smithi107.stdout:743: left oid 15 (ObjNum 742 snap 0 seq_num 742) 2020-06-12T02:37:34.028 INFO:tasks.rados.rados.0.smithi107.stdout:745: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:34.028 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 67 (ObjNum 744 snap 0 seq_num 744) dirty exists 2020-06-12T02:37:34.028 INFO:tasks.rados.rados.0.smithi107.stdout:745: left oid 10 (ObjNum 744 snap 0 seq_num 744) 2020-06-12T02:37:34.028 INFO:tasks.rados.rados.0.smithi107.stdout:744: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:34.028 INFO:tasks.rados.rados.0.smithi107.stdout:746: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:34.029 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:37:34.030 INFO:tasks.rados.rados.0.smithi107.stdout:744: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:34.030 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 133 (ObjNum 743 snap 0 seq_num 743) dirty exists 2020-06-12T02:37:34.030 INFO:tasks.rados.rados.0.smithi107.stdout:744: left oid 3 (ObjNum 743 snap 0 seq_num 743) 2020-06-12T02:37:34.030 INFO:tasks.rados.rados.0.smithi107.stdout:746: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:34.031 INFO:tasks.rados.rados.0.smithi107.stdout:743: done (5 left) 2020-06-12T02:37:34.031 INFO:tasks.rados.rados.0.smithi107.stdout:744: done (4 left) 2020-06-12T02:37:34.031 INFO:tasks.rados.rados.0.smithi107.stdout:745: done (3 left) 2020-06-12T02:37:34.031 INFO:tasks.rados.rados.0.smithi107.stdout:749: done (2 left) 2020-06-12T02:37:34.031 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:37:34.032 INFO:tasks.rados.rados.0.smithi107.stdout:750: seq_num 749 ranges {0=548864} 2020-06-12T02:37:34.032 INFO:tasks.rados.rados.0.smithi107.stdout:750: writing smithi10729407-12 from 0 to 548864 tid 1 2020-06-12T02:37:34.033 INFO:tasks.rados.rados.0.smithi107.stdout:751: delete oid 7 current snap is 0 2020-06-12T02:37:34.033 INFO:tasks.rados.rados.0.smithi107.stdout:748: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:34.033 INFO:tasks.rados.rados.0.smithi107.stdout:746: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:34.033 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 75 (ObjNum 745 snap 0 seq_num 745) dirty exists 2020-06-12T02:37:34.033 INFO:tasks.rados.rados.0.smithi107.stdout:746: left oid 20 (ObjNum 745 snap 0 seq_num 745) 2020-06-12T02:37:34.034 INFO:tasks.rados.rados.0.smithi107.stdout:748: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:34.034 INFO:tasks.rados.rados.0.smithi107.stdout:748: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:34.035 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 75 (ObjNum 747 snap 0 seq_num 747) dirty exists 2020-06-12T02:37:34.035 INFO:tasks.rados.rados.0.smithi107.stdout:748: left oid 5 (ObjNum 747 snap 0 seq_num 747) 2020-06-12T02:37:34.035 INFO:tasks.rados.rados.0.smithi107.stdout:746: done (3 left) 2020-06-12T02:37:34.035 INFO:tasks.rados.rados.0.smithi107.stdout:748: done (2 left) 2020-06-12T02:37:34.036 INFO:tasks.rados.rados.0.smithi107.stdout:751: done (1 left) 2020-06-12T02:37:34.036 INFO:tasks.rados.rados.0.smithi107.stdout:752: delete oid 16 current snap is 0 2020-06-12T02:37:34.037 INFO:tasks.rados.rados.0.smithi107.stdout:752: done (1 left) 2020-06-12T02:37:34.037 INFO:tasks.rados.rados.0.smithi107.stdout:753: delete oid 9 current snap is 0 2020-06-12T02:37:34.038 INFO:tasks.rados.rados.0.smithi107.stdout:750: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:34.040 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:37:34.036+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:37:34.041 INFO:tasks.rados.rados.0.smithi107.stdout:753: done (1 left) 2020-06-12T02:37:34.041 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:37:34.041 INFO:tasks.rados.rados.0.smithi107.stdout:754: seq_num 753 ranges {0=614400} 2020-06-12T02:37:34.041 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:37:34.042 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:37:34.042 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:37:34.042 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:37:34.042 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:37:34.043 INFO:tasks.rados.rados.0.smithi107.stdout:754: writing smithi10729407-9 from 0 to 614400 tid 1 2020-06-12T02:37:34.043 INFO:tasks.rados.rados.0.smithi107.stdout:755: delete oid 6 current snap is 0 2020-06-12T02:37:34.044 INFO:tasks.rados.rados.0.smithi107.stdout:750: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:34.044 INFO:tasks.rados.rados.0.smithi107.stdout:750: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:34.044 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 122 (ObjNum 749 snap 0 seq_num 749) dirty exists 2020-06-12T02:37:34.044 INFO:tasks.rados.rados.0.smithi107.stdout:750: left oid 12 (ObjNum 749 snap 0 seq_num 749) 2020-06-12T02:37:34.045 INFO:tasks.rados.rados.0.smithi107.stdout:750: done (2 left) 2020-06-12T02:37:34.045 INFO:tasks.rados.rados.0.smithi107.stdout:755: done (1 left) 2020-06-12T02:37:34.045 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:37:34.045 INFO:tasks.rados.rados.0.smithi107.stdout:756: seq_num 755 ranges {0=647168} 2020-06-12T02:37:34.049 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:37:34.049 INFO:tasks.rados.rados.0.smithi107.stdout:756: writing smithi10729407-11 from 0 to 647168 tid 1 2020-06-12T02:37:34.250 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:37:34.250 INFO:tasks.rados.rados.0.smithi107.stdout:757: seq_num 756 ranges {0=671744} 2020-06-12T02:37:34.266 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-06-12T02:37:34.267 INFO:tasks.rados.rados.0.smithi107.stdout:757: writing smithi10729407-7 from 0 to 671744 tid 1 2020-06-12T02:37:34.267 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:37:34.267 INFO:tasks.rados.rados.0.smithi107.stdout:758: seq_num 757 ranges {483328=688128} 2020-06-12T02:37:34.267 INFO:tasks.rados.rados.0.smithi107.stdout:758: writing smithi10729407-20 from 483328 to 1171456 tid 1 2020-06-12T02:37:34.268 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:37:34.268 INFO:tasks.rados.rados.0.smithi107.stdout:759: seq_num 758 ranges {401408=704512} 2020-06-12T02:37:34.268 INFO:tasks.rados.rados.0.smithi107.stdout:759: writing smithi10729407-4 from 401408 to 1105920 tid 1 2020-06-12T02:37:34.268 INFO:tasks.rados.rados.0.smithi107.stdout:760: delete oid 19 current snap is 0 2020-06-12T02:37:34.268 INFO:tasks.rados.rados.0.smithi107.stdout:754: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:34.269 INFO:tasks.rados.rados.0.smithi107.stdout:754: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:34.269 INFO:tasks.rados.rados.0.smithi107.stdout:754: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:34.269 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 159 (ObjNum 753 snap 0 seq_num 753) dirty exists 2020-06-12T02:37:34.269 INFO:tasks.rados.rados.0.smithi107.stdout:754: left oid 9 (ObjNum 753 snap 0 seq_num 753) 2020-06-12T02:37:34.269 INFO:tasks.rados.rados.0.smithi107.stdout:756: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:34.269 INFO:tasks.rados.rados.0.smithi107.stdout:756: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:34.270 INFO:tasks.rados.rados.0.smithi107.stdout:756: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:34.270 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 107 (ObjNum 755 snap 0 seq_num 755) dirty exists 2020-06-12T02:37:34.270 INFO:tasks.rados.rados.0.smithi107.stdout:756: left oid 11 (ObjNum 755 snap 0 seq_num 755) 2020-06-12T02:37:34.270 INFO:tasks.rados.rados.0.smithi107.stdout:757: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:34.270 INFO:tasks.rados.rados.0.smithi107.stdout:757: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:34.271 INFO:tasks.rados.rados.0.smithi107.stdout:757: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:34.271 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 138 (ObjNum 756 snap 0 seq_num 756) dirty exists 2020-06-12T02:37:34.271 INFO:tasks.rados.rados.0.smithi107.stdout:757: left oid 7 (ObjNum 756 snap 0 seq_num 756) 2020-06-12T02:37:34.271 INFO:tasks.rados.rados.0.smithi107.stdout:758: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:34.271 INFO:tasks.rados.rados.0.smithi107.stdout:754: done (5 left) 2020-06-12T02:37:34.271 INFO:tasks.rados.rados.0.smithi107.stdout:756: done (4 left) 2020-06-12T02:37:34.272 INFO:tasks.rados.rados.0.smithi107.stdout:757: done (3 left) 2020-06-12T02:37:34.272 INFO:tasks.rados.rados.0.smithi107.stdout:760: done (2 left) 2020-06-12T02:37:34.272 INFO:tasks.rados.rados.0.smithi107.stdout:761: delete oid 8 current snap is 0 2020-06-12T02:37:34.273 INFO:tasks.rados.rados.0.smithi107.stdout:758: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:34.273 INFO:tasks.rados.rados.0.smithi107.stdout:758: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:34.273 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 77 (ObjNum 757 snap 0 seq_num 757) dirty exists 2020-06-12T02:37:34.273 INFO:tasks.rados.rados.0.smithi107.stdout:758: left oid 20 (ObjNum 757 snap 0 seq_num 757) 2020-06-12T02:37:34.274 INFO:tasks.rados.rados.0.smithi107.stdout:759: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:34.274 INFO:tasks.rados.rados.0.smithi107.stdout:758: done (2 left) 2020-06-12T02:37:34.274 INFO:tasks.rados.rados.0.smithi107.stdout:761: done (1 left) 2020-06-12T02:37:34.274 INFO:tasks.rados.rados.0.smithi107.stdout:762: delete oid 3 current snap is 0 2020-06-12T02:37:34.274 INFO:tasks.rados.rados.0.smithi107.stdout:759: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:34.274 INFO:tasks.rados.rados.0.smithi107.stdout:759: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:34.275 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 140 (ObjNum 758 snap 0 seq_num 758) dirty exists 2020-06-12T02:37:34.275 INFO:tasks.rados.rados.0.smithi107.stdout:759: left oid 4 (ObjNum 758 snap 0 seq_num 758) 2020-06-12T02:37:34.275 INFO:tasks.rados.rados.0.smithi107.stdout:759: done (1 left) 2020-06-12T02:37:34.275 INFO:tasks.rados.rados.0.smithi107.stdout:762: done (0 left) 2020-06-12T02:37:34.275 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:37:34.276 INFO:tasks.rados.rados.0.smithi107.stdout:763: seq_num 762 ranges {0=770048} 2020-06-12T02:37:34.276 INFO:tasks.rados.rados.0.smithi107.stdout:763: writing smithi10729407-2 from 0 to 770048 tid 1 2020-06-12T02:37:34.276 INFO:tasks.rados.rados.0.smithi107.stdout:764: delete oid 9 current snap is 0 2020-06-12T02:37:34.276 INFO:tasks.rados.rados.0.smithi107.stdout:764: done (1 left) 2020-06-12T02:37:34.276 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:37:34.276 INFO:tasks.rados.rados.0.smithi107.stdout:765: seq_num 764 ranges {0=802816} 2020-06-12T02:37:34.277 INFO:tasks.rados.rados.0.smithi107.stdout:765: writing smithi10729407-6 from 0 to 802816 tid 1 2020-06-12T02:37:34.277 INFO:tasks.rados.rados.0.smithi107.stdout:766: delete oid 19 current snap is 0 2020-06-12T02:37:34.277 INFO:tasks.rados.rados.0.smithi107.stdout:763: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:34.277 INFO:tasks.rados.rados.0.smithi107.stdout:763: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:34.277 INFO:tasks.rados.rados.0.smithi107.stdout:766: done (2 left) 2020-06-12T02:37:34.278 INFO:tasks.rados.rados.0.smithi107.stdout:767: delete oid 16 current snap is 0 2020-06-12T02:37:34.278 INFO:tasks.rados.rados.0.smithi107.stdout:763: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:34.278 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 80 (ObjNum 762 snap 0 seq_num 762) dirty exists 2020-06-12T02:37:34.278 INFO:tasks.rados.rados.0.smithi107.stdout:763: left oid 2 (ObjNum 762 snap 0 seq_num 762) 2020-06-12T02:37:34.278 INFO:tasks.rados.rados.0.smithi107.stdout:763: done (2 left) 2020-06-12T02:37:34.278 INFO:tasks.rados.rados.0.smithi107.stdout:767: done (1 left) 2020-06-12T02:37:34.279 INFO:tasks.rados.rados.0.smithi107.stdout:768: delete oid 4 current snap is 0 2020-06-12T02:37:34.279 INFO:tasks.rados.rados.0.smithi107.stdout:768: done (1 left) 2020-06-12T02:37:34.279 INFO:tasks.rados.rados.0.smithi107.stdout:769: delete oid 2 current snap is 0 2020-06-12T02:37:34.279 INFO:tasks.rados.rados.0.smithi107.stdout:765: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:34.279 INFO:tasks.rados.rados.0.smithi107.stdout:769: done (1 left) 2020-06-12T02:37:34.280 INFO:tasks.rados.rados.0.smithi107.stdout:770: delete oid 18 current snap is 0 2020-06-12T02:37:34.280 INFO:tasks.rados.rados.0.smithi107.stdout:770: done (1 left) 2020-06-12T02:37:34.280 INFO:tasks.rados.rados.0.smithi107.stdout:771: delete oid 14 current snap is 0 2020-06-12T02:37:34.280 INFO:tasks.rados.rados.0.smithi107.stdout:765: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:34.280 INFO:tasks.rados.rados.0.smithi107.stdout:765: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:34.280 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 164 (ObjNum 764 snap 0 seq_num 764) dirty exists 2020-06-12T02:37:34.281 INFO:tasks.rados.rados.0.smithi107.stdout:765: left oid 6 (ObjNum 764 snap 0 seq_num 764) 2020-06-12T02:37:34.281 INFO:tasks.rados.rados.0.smithi107.stdout:765: done (1 left) 2020-06-12T02:37:34.281 INFO:tasks.rados.rados.0.smithi107.stdout:771: done (0 left) 2020-06-12T02:37:34.281 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:37:34.281 INFO:tasks.rados.rados.0.smithi107.stdout:772: seq_num 771 ranges {671744=516096,1187840=401408} 2020-06-12T02:37:34.281 INFO:tasks.rados.rados.0.smithi107.stdout:772: writing smithi10729407-7 from 671744 to 1187840 tid 1 2020-06-12T02:37:34.282 INFO:tasks.rados.rados.0.smithi107.stdout:772: writing smithi10729407-7 from 1187840 to 1589248 tid 2 2020-06-12T02:37:34.282 INFO:tasks.rados.rados.0.smithi107.stdout:773: delete oid 20 current snap is 0 2020-06-12T02:37:34.282 INFO:tasks.rados.rados.0.smithi107.stdout:773: done (1 left) 2020-06-12T02:37:34.283 INFO:tasks.rados.rados.0.smithi107.stdout:774: delete oid 13 current snap is 0 2020-06-12T02:37:34.283 INFO:tasks.rados.rados.0.smithi107.stdout:772: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:34.283 INFO:tasks.rados.rados.0.smithi107.stdout:774: done (1 left) 2020-06-12T02:37:34.283 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:37:34.283 INFO:tasks.rados.rados.0.smithi107.stdout:775: seq_num 774 ranges {434176=565248,999424=401408} 2020-06-12T02:37:34.284 INFO:tasks.rados.rados.0.smithi107.stdout:775: writing smithi10729407-15 from 434176 to 999424 tid 1 2020-06-12T02:37:34.284 INFO:tasks.rados.rados.0.smithi107.stdout:775: writing smithi10729407-15 from 999424 to 1400832 tid 2 2020-06-12T02:37:34.284 INFO:tasks.rados.rados.0.smithi107.stdout:776: delete oid 2 current snap is 0 2020-06-12T02:37:34.284 INFO:tasks.rados.rados.0.smithi107.stdout:772: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:34.284 INFO:tasks.rados.rados.0.smithi107.stdout:772: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:34.285 INFO:tasks.rados.rados.0.smithi107.stdout:772: finishing write tid 5 to smithi10729407-7 2020-06-12T02:37:34.285 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 142 (ObjNum 771 snap 0 seq_num 771) dirty exists 2020-06-12T02:37:34.285 INFO:tasks.rados.rados.0.smithi107.stdout:772: left oid 7 (ObjNum 771 snap 0 seq_num 771) 2020-06-12T02:37:34.285 INFO:tasks.rados.rados.0.smithi107.stdout:772: done (2 left) 2020-06-12T02:37:34.285 INFO:tasks.rados.rados.0.smithi107.stdout:776: done (1 left) 2020-06-12T02:37:34.286 INFO:tasks.rados.rados.0.smithi107.stdout:777: delete oid 11 current snap is 0 2020-06-12T02:37:34.286 INFO:tasks.rados.rados.0.smithi107.stdout:775: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:34.286 INFO:tasks.rados.rados.0.smithi107.stdout:777: done (1 left) 2020-06-12T02:37:34.286 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:37:34.286 INFO:tasks.rados.rados.0.smithi107.stdout:778: seq_num 777 ranges {0=622592,622592=401408} 2020-06-12T02:37:34.287 INFO:tasks.rados.rados.0.smithi107.stdout:778: writing smithi10729407-8 from 0 to 622592 tid 1 2020-06-12T02:37:34.287 INFO:tasks.rados.rados.0.smithi107.stdout:778: writing smithi10729407-8 from 622592 to 1024000 tid 2 2020-06-12T02:37:34.287 INFO:tasks.rados.rados.0.smithi107.stdout:779: delete oid 7 current snap is 0 2020-06-12T02:37:34.287 INFO:tasks.rados.rados.0.smithi107.stdout:775: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:34.287 INFO:tasks.rados.rados.0.smithi107.stdout:775: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:34.287 INFO:tasks.rados.rados.0.smithi107.stdout:775: finishing write tid 5 to smithi10729407-15 2020-06-12T02:37:34.288 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 140 (ObjNum 774 snap 0 seq_num 774) dirty exists 2020-06-12T02:37:34.288 INFO:tasks.rados.rados.0.smithi107.stdout:775: left oid 15 (ObjNum 774 snap 0 seq_num 774) 2020-06-12T02:37:34.288 INFO:tasks.rados.rados.0.smithi107.stdout:775: done (2 left) 2020-06-12T02:37:34.288 INFO:tasks.rados.rados.0.smithi107.stdout:779: done (1 left) 2020-06-12T02:37:34.288 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:37:34.289 INFO:tasks.rados.rados.0.smithi107.stdout:780: seq_num 779 ranges {0=655360,655360=401408} 2020-06-12T02:37:34.289 INFO:tasks.rados.rados.0.smithi107.stdout:780: writing smithi10729407-7 from 0 to 655360 tid 1 2020-06-12T02:37:34.289 INFO:tasks.rados.rados.0.smithi107.stdout:780: writing smithi10729407-7 from 655360 to 1056768 tid 2 2020-06-12T02:37:34.289 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:37:34.289 INFO:tasks.rados.rados.0.smithi107.stdout:781: seq_num 780 ranges {0=671744,671744=401408} 2020-06-12T02:37:34.289 INFO:tasks.rados.rados.0.smithi107.stdout:781: writing smithi10729407-4 from 0 to 671744 tid 1 2020-06-12T02:37:34.290 INFO:tasks.rados.rados.0.smithi107.stdout:781: writing smithi10729407-4 from 671744 to 1073152 tid 2 2020-06-12T02:37:34.290 INFO:tasks.rados.rados.0.smithi107.stdout:782: delete oid 12 current snap is 0 2020-06-12T02:37:34.290 INFO:tasks.rados.rados.0.smithi107.stdout:778: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:34.290 INFO:tasks.rados.rados.0.smithi107.stdout:778: finishing write tid 2 to smithi10729407-8 2020-06-12T02:37:34.290 INFO:tasks.rados.rados.0.smithi107.stdout:778: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:34.291 INFO:tasks.rados.rados.0.smithi107.stdout:778: finishing write tid 5 to smithi10729407-8 2020-06-12T02:37:34.291 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 143 (ObjNum 777 snap 0 seq_num 777) dirty exists 2020-06-12T02:37:34.291 INFO:tasks.rados.rados.0.smithi107.stdout:778: left oid 8 (ObjNum 777 snap 0 seq_num 777) 2020-06-12T02:37:34.291 INFO:tasks.rados.rados.0.smithi107.stdout:780: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:34.291 INFO:tasks.rados.rados.0.smithi107.stdout:780: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:34.291 INFO:tasks.rados.rados.0.smithi107.stdout:780: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:34.292 INFO:tasks.rados.rados.0.smithi107.stdout:780: finishing write tid 5 to smithi10729407-7 2020-06-12T02:37:34.292 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 146 (ObjNum 779 snap 0 seq_num 779) dirty exists 2020-06-12T02:37:34.292 INFO:tasks.rados.rados.0.smithi107.stdout:780: left oid 7 (ObjNum 779 snap 0 seq_num 779) 2020-06-12T02:37:34.292 INFO:tasks.rados.rados.0.smithi107.stdout:778: done (3 left) 2020-06-12T02:37:34.293 INFO:tasks.rados.rados.0.smithi107.stdout:780: done (2 left) 2020-06-12T02:37:34.293 INFO:tasks.rados.rados.0.smithi107.stdout:782: done (1 left) 2020-06-12T02:37:34.293 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:37:34.293 INFO:tasks.rados.rados.0.smithi107.stdout:783: seq_num 782 ranges {0=704512,704512=401408} 2020-06-12T02:37:34.293 INFO:tasks.rados.rados.0.smithi107.stdout:783: writing smithi10729407-3 from 0 to 704512 tid 1 2020-06-12T02:37:34.294 INFO:tasks.rados.rados.0.smithi107.stdout:783: writing smithi10729407-3 from 704512 to 1105920 tid 2 2020-06-12T02:37:34.294 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:37:34.294 INFO:tasks.rados.rados.0.smithi107.stdout:784: seq_num 783 ranges {1400832=720896,2121728=401408} 2020-06-12T02:37:34.294 INFO:tasks.rados.rados.0.smithi107.stdout:784: writing smithi10729407-15 from 1400832 to 2121728 tid 1 2020-06-12T02:37:34.294 INFO:tasks.rados.rados.0.smithi107.stdout:784: writing smithi10729407-15 from 2121728 to 2523136 tid 2 2020-06-12T02:37:34.294 INFO:tasks.rados.rados.0.smithi107.stdout:785: delete oid 7 current snap is 0 2020-06-12T02:37:34.295 INFO:tasks.rados.rados.0.smithi107.stdout:781: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:34.295 INFO:tasks.rados.rados.0.smithi107.stdout:781: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:34.295 INFO:tasks.rados.rados.0.smithi107.stdout:781: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:34.295 INFO:tasks.rados.rados.0.smithi107.stdout:781: finishing write tid 5 to smithi10729407-4 2020-06-12T02:37:34.295 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 144 (ObjNum 780 snap 0 seq_num 780) dirty exists 2020-06-12T02:37:34.295 INFO:tasks.rados.rados.0.smithi107.stdout:781: left oid 4 (ObjNum 780 snap 0 seq_num 780) 2020-06-12T02:37:34.296 INFO:tasks.rados.rados.0.smithi107.stdout:783: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:34.296 INFO:tasks.rados.rados.0.smithi107.stdout:783: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:34.296 INFO:tasks.rados.rados.0.smithi107.stdout:783: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:34.296 INFO:tasks.rados.rados.0.smithi107.stdout:783: finishing write tid 5 to smithi10729407-3 2020-06-12T02:37:34.297 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 140 (ObjNum 782 snap 0 seq_num 782) dirty exists 2020-06-12T02:37:34.297 INFO:tasks.rados.rados.0.smithi107.stdout:783: left oid 3 (ObjNum 782 snap 0 seq_num 782) 2020-06-12T02:37:34.297 INFO:tasks.rados.rados.0.smithi107.stdout:781: done (3 left) 2020-06-12T02:37:34.297 INFO:tasks.rados.rados.0.smithi107.stdout:783: done (2 left) 2020-06-12T02:37:34.297 INFO:tasks.rados.rados.0.smithi107.stdout:785: done (1 left) 2020-06-12T02:37:34.298 INFO:tasks.rados.rados.0.smithi107.stdout:786: delete oid 4 current snap is 0 2020-06-12T02:37:34.298 INFO:tasks.rados.rados.0.smithi107.stdout:784: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:34.298 INFO:tasks.rados.rados.0.smithi107.stdout:786: done (1 left) 2020-06-12T02:37:34.298 INFO:tasks.rados.rados.0.smithi107.stdout:787: delete oid 12 current snap is 0 2020-06-12T02:37:34.298 INFO:tasks.rados.rados.0.smithi107.stdout:787: done (1 left) 2020-06-12T02:37:34.298 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:37:34.299 INFO:tasks.rados.rados.0.smithi107.stdout:788: seq_num 787 ranges {802816=786432,1589248=401408} 2020-06-12T02:37:34.299 INFO:tasks.rados.rados.0.smithi107.stdout:788: writing smithi10729407-6 from 802816 to 1589248 tid 1 2020-06-12T02:37:34.299 INFO:tasks.rados.rados.0.smithi107.stdout:788: writing smithi10729407-6 from 1589248 to 1990656 tid 2 2020-06-12T02:37:34.299 INFO:tasks.rados.rados.0.smithi107.stdout:789: delete oid 4 current snap is 0 2020-06-12T02:37:34.299 INFO:tasks.rados.rados.0.smithi107.stdout:784: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:34.300 INFO:tasks.rados.rados.0.smithi107.stdout:784: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:34.300 INFO:tasks.rados.rados.0.smithi107.stdout:784: finishing write tid 5 to smithi10729407-15 2020-06-12T02:37:34.300 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 146 (ObjNum 783 snap 0 seq_num 783) dirty exists 2020-06-12T02:37:34.300 INFO:tasks.rados.rados.0.smithi107.stdout:784: left oid 15 (ObjNum 783 snap 0 seq_num 783) 2020-06-12T02:37:34.300 INFO:tasks.rados.rados.0.smithi107.stdout:784: done (2 left) 2020-06-12T02:37:34.300 INFO:tasks.rados.rados.0.smithi107.stdout:789: done (1 left) 2020-06-12T02:37:34.301 INFO:tasks.rados.rados.0.smithi107.stdout:790: delete oid 8 current snap is 0 2020-06-12T02:37:34.301 INFO:tasks.rados.rados.0.smithi107.stdout:790: done (1 left) 2020-06-12T02:37:34.302 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:37:34.302 INFO:tasks.rados.rados.0.smithi107.stdout:791: seq_num 790 ranges {0=434176,434176=548864,983040=253952} 2020-06-12T02:37:34.302 INFO:tasks.rados.rados.0.smithi107.stdout:791: writing smithi10729407-13 from 0 to 434176 tid 1 2020-06-12T02:37:34.303 INFO:tasks.rados.rados.0.smithi107.stdout:791: writing smithi10729407-13 from 434176 to 983040 tid 2 2020-06-12T02:37:34.303 INFO:tasks.rados.rados.0.smithi107.stdout:791: writing smithi10729407-13 from 983040 to 1236992 tid 3 2020-06-12T02:37:34.303 INFO:tasks.rados.rados.0.smithi107.stdout:792: delete oid 17 current snap is 0 2020-06-12T02:37:34.303 INFO:tasks.rados.rados.0.smithi107.stdout:788: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:34.304 INFO:tasks.rados.rados.0.smithi107.stdout:788: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:34.304 INFO:tasks.rados.rados.0.smithi107.stdout:788: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:34.304 INFO:tasks.rados.rados.0.smithi107.stdout:788: finishing write tid 5 to smithi10729407-6 2020-06-12T02:37:34.304 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 167 (ObjNum 787 snap 0 seq_num 787) dirty exists 2020-06-12T02:37:34.304 INFO:tasks.rados.rados.0.smithi107.stdout:788: left oid 6 (ObjNum 787 snap 0 seq_num 787) 2020-06-12T02:37:34.305 INFO:tasks.rados.rados.0.smithi107.stdout:788: done (2 left) 2020-06-12T02:37:34.305 INFO:tasks.rados.rados.0.smithi107.stdout:792: done (1 left) 2020-06-12T02:37:34.305 INFO:tasks.rados.rados.0.smithi107.stdout:793: delete oid 4 current snap is 0 2020-06-12T02:37:34.305 INFO:tasks.rados.rados.0.smithi107.stdout:791: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:34.305 INFO:tasks.rados.rados.0.smithi107.stdout:793: done (1 left) 2020-06-12T02:37:34.305 INFO:tasks.rados.rados.0.smithi107.stdout:794: delete oid 18 current snap is 0 2020-06-12T02:37:34.306 INFO:tasks.rados.rados.0.smithi107.stdout:791: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:34.306 INFO:tasks.rados.rados.0.smithi107.stdout:791: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:34.306 INFO:tasks.rados.rados.0.smithi107.stdout:791: finishing write tid 5 to smithi10729407-13 2020-06-12T02:37:34.306 INFO:tasks.rados.rados.0.smithi107.stdout:794: done (1 left) 2020-06-12T02:37:34.306 INFO:tasks.rados.rados.0.smithi107.stdout:795: delete oid 16 current snap is 0 2020-06-12T02:37:34.307 INFO:tasks.rados.rados.0.smithi107.stdout:791: finishing write tid 6 to smithi10729407-13 2020-06-12T02:37:34.307 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 151 (ObjNum 790 snap 0 seq_num 790) dirty exists 2020-06-12T02:37:34.307 INFO:tasks.rados.rados.0.smithi107.stdout:791: left oid 13 (ObjNum 790 snap 0 seq_num 790) 2020-06-12T02:37:34.307 INFO:tasks.rados.rados.0.smithi107.stdout:791: done (1 left) 2020-06-12T02:37:34.307 INFO:tasks.rados.rados.0.smithi107.stdout:795: done (0 left) 2020-06-12T02:37:34.307 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:37:34.308 INFO:tasks.rados.rados.0.smithi107.stdout:796: seq_num 795 ranges {0=524288,524288=417792,942080=385024} 2020-06-12T02:37:34.308 INFO:tasks.rados.rados.0.smithi107.stdout:796: writing smithi10729407-18 from 0 to 524288 tid 1 2020-06-12T02:37:34.308 INFO:tasks.rados.rados.0.smithi107.stdout:796: writing smithi10729407-18 from 524288 to 942080 tid 2 2020-06-12T02:37:34.308 INFO:tasks.rados.rados.0.smithi107.stdout:796: writing smithi10729407-18 from 942080 to 1327104 tid 3 2020-06-12T02:37:34.308 INFO:tasks.rados.rados.0.smithi107.stdout:797: delete oid 16 current snap is 0 2020-06-12T02:37:34.308 INFO:tasks.rados.rados.0.smithi107.stdout:796: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:34.309 INFO:tasks.rados.rados.0.smithi107.stdout:797: done (1 left) 2020-06-12T02:37:34.309 INFO:tasks.rados.rados.0.smithi107.stdout:798: delete oid 11 current snap is 0 2020-06-12T02:37:34.309 INFO:tasks.rados.rados.0.smithi107.stdout:798: done (1 left) 2020-06-12T02:37:34.309 INFO:tasks.rados.rados.0.smithi107.stdout:799: delete oid 16 current snap is 0 2020-06-12T02:37:34.309 INFO:tasks.rados.rados.0.smithi107.stdout:796: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:34.310 INFO:tasks.rados.rados.0.smithi107.stdout:796: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:34.310 INFO:tasks.rados.rados.0.smithi107.stdout:796: finishing write tid 5 to smithi10729407-18 2020-06-12T02:37:34.310 INFO:tasks.rados.rados.0.smithi107.stdout:799: done (1 left) 2020-06-12T02:37:34.310 INFO:tasks.rados.rados.0.smithi107.stdout:800: delete oid 11 current snap is 0 2020-06-12T02:37:34.310 INFO:tasks.rados.rados.0.smithi107.stdout:796: finishing write tid 6 to smithi10729407-18 2020-06-12T02:37:34.310 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 166 (ObjNum 795 snap 0 seq_num 795) dirty exists 2020-06-12T02:37:34.311 INFO:tasks.rados.rados.0.smithi107.stdout:796: left oid 18 (ObjNum 795 snap 0 seq_num 795) 2020-06-12T02:37:34.311 INFO:tasks.rados.rados.0.smithi107.stdout:796: done (1 left) 2020-06-12T02:37:34.311 INFO:tasks.rados.rados.0.smithi107.stdout:800: done (0 left) 2020-06-12T02:37:34.311 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:37:34.311 INFO:tasks.rados.rados.0.smithi107.stdout:801: seq_num 800 ranges {0=606208,606208=688128,1294336=114688} 2020-06-12T02:37:34.312 INFO:tasks.rados.rados.0.smithi107.stdout:801: writing smithi10729407-16 from 0 to 606208 tid 1 2020-06-12T02:37:34.312 INFO:tasks.rados.rados.0.smithi107.stdout:801: writing smithi10729407-16 from 606208 to 1294336 tid 2 2020-06-12T02:37:34.312 INFO:tasks.rados.rados.0.smithi107.stdout:801: writing smithi10729407-16 from 1294336 to 1409024 tid 3 2020-06-12T02:37:34.312 INFO:tasks.rados.rados.0.smithi107.stdout:802: delete oid 4 current snap is 0 2020-06-12T02:37:34.312 INFO:tasks.rados.rados.0.smithi107.stdout:802: done (1 left) 2020-06-12T02:37:34.312 INFO:tasks.rados.rados.0.smithi107.stdout:803: delete oid 7 current snap is 0 2020-06-12T02:37:34.313 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:37:34.313 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:37:34.313 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:37:34.314 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:37:34.314 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:37:34.314 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-06-12T02:37:34.314 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-06-12T02:37:34.314 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-06-12T02:37:34.315 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-06-12T02:37:34.315 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-06-12T02:37:34.315 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-06-12T02:37:34.315 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-06-12T02:37:34.315 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.7 2020-06-12T02:37:34.315 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: problem getting command descriptions from osd.7 2020-06-12T02:37:34.316 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-06-12T02:37:34.316 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-06-12T02:37:34.319 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:37:34.252+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:37:34.320 INFO:tasks.rados.rados.0.smithi107.stdout:803: done (1 left) 2020-06-12T02:37:34.320 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:37:34.320 INFO:tasks.rados.rados.0.smithi107.stdout:804: seq_num 803 ranges {0=655360,655360=745472,1400832=57344} 2020-06-12T02:37:34.321 INFO:tasks.rados.rados.0.smithi107.stdout:804: writing smithi10729407-19 from 0 to 655360 tid 1 2020-06-12T02:37:34.321 INFO:tasks.rados.rados.0.smithi107.stdout:804: writing smithi10729407-19 from 655360 to 1400832 tid 2 2020-06-12T02:37:34.321 INFO:tasks.rados.rados.0.smithi107.stdout:804: writing smithi10729407-19 from 1400832 to 1458176 tid 3 2020-06-12T02:37:34.321 INFO:tasks.rados.rados.0.smithi107.stdout:805: delete oid 9 current snap is 0 2020-06-12T02:37:34.321 INFO:tasks.rados.rados.0.smithi107.stdout:801: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:34.322 INFO:tasks.rados.rados.0.smithi107.stdout:801: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:34.322 INFO:tasks.rados.rados.0.smithi107.stdout:801: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:34.322 INFO:tasks.rados.rados.0.smithi107.stdout:801: finishing write tid 5 to smithi10729407-16 2020-06-12T02:37:34.322 INFO:tasks.rados.rados.0.smithi107.stdout:801: finishing write tid 6 to smithi10729407-16 2020-06-12T02:37:34.322 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 147 (ObjNum 800 snap 0 seq_num 800) dirty exists 2020-06-12T02:37:34.322 INFO:tasks.rados.rados.0.smithi107.stdout:801: left oid 16 (ObjNum 800 snap 0 seq_num 800) 2020-06-12T02:37:34.323 INFO:tasks.rados.rados.0.smithi107.stdout:801: done (2 left) 2020-06-12T02:37:34.323 INFO:tasks.rados.rados.0.smithi107.stdout:805: done (1 left) 2020-06-12T02:37:34.323 INFO:tasks.rados.rados.0.smithi107.stdout:806: delete oid 6 current snap is 0 2020-06-12T02:37:34.323 INFO:tasks.rados.rados.0.smithi107.stdout:804: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:34.324 INFO:tasks.rados.rados.0.smithi107.stdout:806: done (1 left) 2020-06-12T02:37:34.324 INFO:tasks.rados.rados.0.smithi107.stdout:807: delete oid 4 current snap is 0 2020-06-12T02:37:34.324 INFO:tasks.rados.rados.0.smithi107.stdout:807: done (1 left) 2020-06-12T02:37:34.324 INFO:tasks.rados.rados.0.smithi107.stdout:808: delete oid 5 current snap is 0 2020-06-12T02:37:34.324 INFO:tasks.rados.rados.0.smithi107.stdout:808: done (1 left) 2020-06-12T02:37:34.325 INFO:tasks.rados.rados.0.smithi107.stdout:809: delete oid 3 current snap is 0 2020-06-12T02:37:34.325 INFO:tasks.rados.rados.0.smithi107.stdout:804: finishing write tid 2 to smithi10729407-19 2020-06-12T02:37:34.325 INFO:tasks.rados.rados.0.smithi107.stdout:804: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:34.325 INFO:tasks.rados.rados.0.smithi107.stdout:809: done (1 left) 2020-06-12T02:37:34.325 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:37:34.326 INFO:tasks.rados.rados.0.smithi107.stdout:810: seq_num 809 ranges {0=753664,753664=499712,1253376=303104} 2020-06-12T02:37:34.326 INFO:tasks.rados.rados.0.smithi107.stdout:810: writing smithi10729407-11 from 0 to 753664 tid 1 2020-06-12T02:37:34.326 INFO:tasks.rados.rados.0.smithi107.stdout:810: writing smithi10729407-11 from 753664 to 1253376 tid 2 2020-06-12T02:37:34.326 INFO:tasks.rados.rados.0.smithi107.stdout:810: writing smithi10729407-11 from 1253376 to 1556480 tid 3 2020-06-12T02:37:34.327 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:37:34.327 INFO:tasks.rados.rados.0.smithi107.stdout:811: seq_num 810 ranges {0=770048,770048=778240,1548288=24576} 2020-06-12T02:37:34.327 INFO:tasks.rados.rados.0.smithi107.stdout:811: writing smithi10729407-17 from 0 to 770048 tid 1 2020-06-12T02:37:34.327 INFO:tasks.rados.rados.0.smithi107.stdout:811: writing smithi10729407-17 from 770048 to 1548288 tid 2 2020-06-12T02:37:34.327 INFO:tasks.rados.rados.0.smithi107.stdout:811: writing smithi10729407-17 from 1548288 to 1572864 tid 3 2020-06-12T02:37:34.327 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:37:34.328 INFO:tasks.rados.rados.0.smithi107.stdout:812: seq_num 811 ranges {0=786432,786432=663552,1449984=139264} 2020-06-12T02:37:34.328 INFO:tasks.rados.rados.0.smithi107.stdout:812: writing smithi10729407-8 from 0 to 786432 tid 1 2020-06-12T02:37:34.328 INFO:tasks.rados.rados.0.smithi107.stdout:812: writing smithi10729407-8 from 786432 to 1449984 tid 2 2020-06-12T02:37:34.328 INFO:tasks.rados.rados.0.smithi107.stdout:812: writing smithi10729407-8 from 1449984 to 1589248 tid 3 2020-06-12T02:37:34.328 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:37:34.329 INFO:tasks.rados.rados.0.smithi107.stdout:813: seq_num 812 ranges {0=802816,802816=548864,1351680=253952} 2020-06-12T02:37:34.329 INFO:tasks.rados.rados.0.smithi107.stdout:813: writing smithi10729407-14 from 0 to 802816 tid 1 2020-06-12T02:37:34.330 INFO:tasks.rados.rados.0.smithi107.stdout:813: writing smithi10729407-14 from 802816 to 1351680 tid 2 2020-06-12T02:37:34.330 INFO:tasks.rados.rados.0.smithi107.stdout:813: writing smithi10729407-14 from 1351680 to 1605632 tid 3 2020-06-12T02:37:34.330 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:37:34.331 INFO:tasks.rados.rados.0.smithi107.stdout:814: seq_num 813 ranges {2523136=417792,2940928=434176,3375104=442368,3817472=327680} 2020-06-12T02:37:34.340 INFO:tasks.rados.rados.0.smithi107.stdout:814: writing smithi10729407-15 from 2523136 to 2940928 tid 1 2020-06-12T02:37:34.343 INFO:tasks.rados.rados.0.smithi107.stdout:814: writing smithi10729407-15 from 2940928 to 3375104 tid 2 2020-06-12T02:37:34.346 INFO:tasks.rados.rados.0.smithi107.stdout:814: writing smithi10729407-15 from 3375104 to 3817472 tid 3 2020-06-12T02:37:34.349 INFO:tasks.rados.rados.0.smithi107.stdout:814: writing smithi10729407-15 from 3817472 to 4145152 tid 4 2020-06-12T02:37:34.350 INFO:tasks.rados.rados.0.smithi107.stdout:815: delete oid 20 current snap is 0 2020-06-12T02:37:34.350 INFO:tasks.rados.rados.0.smithi107.stdout:804: finishing write tid 5 to smithi10729407-19 2020-06-12T02:37:34.350 INFO:tasks.rados.rados.0.smithi107.stdout:804: finishing write tid 6 to smithi10729407-19 2020-06-12T02:37:34.350 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 130 (ObjNum 803 snap 0 seq_num 803) dirty exists 2020-06-12T02:37:34.350 INFO:tasks.rados.rados.0.smithi107.stdout:804: left oid 19 (ObjNum 803 snap 0 seq_num 803) 2020-06-12T02:37:34.351 INFO:tasks.rados.rados.0.smithi107.stdout:810: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:34.351 INFO:tasks.rados.rados.0.smithi107.stdout:810: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:34.351 INFO:tasks.rados.rados.0.smithi107.stdout:810: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:34.351 INFO:tasks.rados.rados.0.smithi107.stdout:810: finishing write tid 5 to smithi10729407-11 2020-06-12T02:37:34.351 INFO:tasks.rados.rados.0.smithi107.stdout:810: finishing write tid 6 to smithi10729407-11 2020-06-12T02:37:34.352 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 115 (ObjNum 809 snap 0 seq_num 809) dirty exists 2020-06-12T02:37:34.352 INFO:tasks.rados.rados.0.smithi107.stdout:810: left oid 11 (ObjNum 809 snap 0 seq_num 809) 2020-06-12T02:37:34.352 INFO:tasks.rados.rados.0.smithi107.stdout:811: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:34.352 INFO:tasks.rados.rados.0.smithi107.stdout:811: finishing write tid 2 to smithi10729407-17 2020-06-12T02:37:34.352 INFO:tasks.rados.rados.0.smithi107.stdout:811: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:34.352 INFO:tasks.rados.rados.0.smithi107.stdout:811: finishing write tid 5 to smithi10729407-17 2020-06-12T02:37:34.353 INFO:tasks.rados.rados.0.smithi107.stdout:811: finishing write tid 6 to smithi10729407-17 2020-06-12T02:37:34.353 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 154 (ObjNum 810 snap 0 seq_num 810) dirty exists 2020-06-12T02:37:34.353 INFO:tasks.rados.rados.0.smithi107.stdout:811: left oid 17 (ObjNum 810 snap 0 seq_num 810) 2020-06-12T02:37:34.353 INFO:tasks.rados.rados.0.smithi107.stdout:812: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:34.353 INFO:tasks.rados.rados.0.smithi107.stdout:812: finishing write tid 2 to smithi10729407-8 2020-06-12T02:37:34.354 INFO:tasks.rados.rados.0.smithi107.stdout:812: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:34.355 INFO:tasks.rados.rados.0.smithi107.stdout:812: finishing write tid 5 to smithi10729407-8 2020-06-12T02:37:34.355 INFO:tasks.rados.rados.0.smithi107.stdout:812: finishing write tid 6 to smithi10729407-8 2020-06-12T02:37:34.355 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 151 (ObjNum 811 snap 0 seq_num 811) dirty exists 2020-06-12T02:37:34.356 INFO:tasks.rados.rados.0.smithi107.stdout:812: left oid 8 (ObjNum 811 snap 0 seq_num 811) 2020-06-12T02:37:34.356 INFO:tasks.rados.rados.0.smithi107.stdout:813: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:34.356 INFO:tasks.rados.rados.0.smithi107.stdout:813: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:34.356 INFO:tasks.rados.rados.0.smithi107.stdout:813: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:34.356 INFO:tasks.rados.rados.0.smithi107.stdout:813: finishing write tid 5 to smithi10729407-14 2020-06-12T02:37:34.356 INFO:tasks.rados.rados.0.smithi107.stdout:813: finishing write tid 6 to smithi10729407-14 2020-06-12T02:37:34.357 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 119 (ObjNum 812 snap 0 seq_num 812) dirty exists 2020-06-12T02:37:34.357 INFO:tasks.rados.rados.0.smithi107.stdout:813: left oid 14 (ObjNum 812 snap 0 seq_num 812) 2020-06-12T02:37:34.358 INFO:tasks.rados.rados.0.smithi107.stdout:814: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:34.358 INFO:tasks.rados.rados.0.smithi107.stdout:814: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:34.358 INFO:tasks.rados.rados.0.smithi107.stdout:814: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:34.359 INFO:tasks.rados.rados.0.smithi107.stdout:804: done (6 left) 2020-06-12T02:37:34.359 INFO:tasks.rados.rados.0.smithi107.stdout:810: done (5 left) 2020-06-12T02:37:34.359 INFO:tasks.rados.rados.0.smithi107.stdout:811: done (4 left) 2020-06-12T02:37:34.359 INFO:tasks.rados.rados.0.smithi107.stdout:812: done (3 left) 2020-06-12T02:37:34.359 INFO:tasks.rados.rados.0.smithi107.stdout:813: done (2 left) 2020-06-12T02:37:34.360 INFO:tasks.rados.rados.0.smithi107.stdout:815: done (1 left) 2020-06-12T02:37:34.360 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:37:34.360 INFO:tasks.rados.rados.0.smithi107.stdout:816: seq_num 815 ranges {0=458752,458752=655360,1114112=409600,1523712=139264} 2020-06-12T02:37:34.360 INFO:tasks.rados.rados.0.smithi107.stdout:816: writing smithi10729407-6 from 0 to 458752 tid 1 2020-06-12T02:37:34.361 INFO:tasks.rados.rados.0.smithi107.stdout:816: writing smithi10729407-6 from 458752 to 1114112 tid 2 2020-06-12T02:37:34.363 INFO:tasks.rados.rados.0.smithi107.stdout:816: writing smithi10729407-6 from 1114112 to 1523712 tid 3 2020-06-12T02:37:34.364 INFO:tasks.rados.rados.0.smithi107.stdout:816: writing smithi10729407-6 from 1523712 to 1662976 tid 4 2020-06-12T02:37:34.365 INFO:tasks.rados.rados.0.smithi107.stdout:817: delete oid 19 current snap is 0 2020-06-12T02:37:34.365 INFO:tasks.rados.rados.0.smithi107.stdout:814: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:34.365 INFO:tasks.rados.rados.0.smithi107.stdout:814: finishing write tid 6 to smithi10729407-15 2020-06-12T02:37:34.365 INFO:tasks.rados.rados.0.smithi107.stdout:814: finishing write tid 7 to smithi10729407-15 2020-06-12T02:37:34.366 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 156 (ObjNum 813 snap 0 seq_num 813) dirty exists 2020-06-12T02:37:34.366 INFO:tasks.rados.rados.0.smithi107.stdout:814: left oid 15 (ObjNum 813 snap 0 seq_num 813) 2020-06-12T02:37:34.366 INFO:tasks.rados.rados.0.smithi107.stdout:816: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:34.367 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:34.361+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:34.367 INFO:tasks.rados.rados.0.smithi107.stdout:814: done (2 left) 2020-06-12T02:37:34.367 INFO:tasks.rados.rados.0.smithi107.stdout:817: done (1 left) 2020-06-12T02:37:34.367 INFO:tasks.rados.rados.0.smithi107.stdout:818: delete oid 8 current snap is 0 2020-06-12T02:37:34.368 INFO:tasks.rados.rados.0.smithi107.stdout:818: done (1 left) 2020-06-12T02:37:34.369 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:37:34.369 INFO:tasks.rados.rados.0.smithi107.stdout:819: seq_num 818 ranges {1605632=507904,2113536=704512,2818048=499712} 2020-06-12T02:37:34.380 INFO:tasks.rados.rados.0.smithi107.stdout:819: writing smithi10729407-14 from 1605632 to 2113536 tid 1 2020-06-12T02:37:34.385 INFO:tasks.rados.rados.0.smithi107.stdout:819: writing smithi10729407-14 from 2113536 to 2818048 tid 2 2020-06-12T02:37:34.388 INFO:tasks.rados.rados.0.smithi107.stdout:819: writing smithi10729407-14 from 2818048 to 3317760 tid 3 2020-06-12T02:37:34.388 INFO:tasks.rados.rados.0.smithi107.stdout:820: delete oid 7 current snap is 0 2020-06-12T02:37:34.389 INFO:tasks.rados.rados.0.smithi107.stdout:816: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:34.389 INFO:tasks.rados.rados.0.smithi107.stdout:816: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:34.389 INFO:tasks.rados.rados.0.smithi107.stdout:816: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:34.389 INFO:tasks.rados.rados.0.smithi107.stdout:816: finishing write tid 6 to smithi10729407-6 2020-06-12T02:37:34.389 INFO:tasks.rados.rados.0.smithi107.stdout:816: finishing write tid 7 to smithi10729407-6 2020-06-12T02:37:34.390 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 173 (ObjNum 815 snap 0 seq_num 815) dirty exists 2020-06-12T02:37:34.390 INFO:tasks.rados.rados.0.smithi107.stdout:816: left oid 6 (ObjNum 815 snap 0 seq_num 815) 2020-06-12T02:37:34.390 INFO:tasks.rados.rados.0.smithi107.stdout:819: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:34.390 INFO:tasks.rados.rados.0.smithi107.stdout:816: done (2 left) 2020-06-12T02:37:34.391 INFO:tasks.rados.rados.0.smithi107.stdout:820: done (1 left) 2020-06-12T02:37:34.391 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:37:34.391 INFO:tasks.rados.rados.0.smithi107.stdout:821: seq_num 820 ranges {0=540672,540672=475136,1015808=598016,1613824=131072} 2020-06-12T02:37:34.393 INFO:tasks.rados.rados.0.smithi107.stdout:821: writing smithi10729407-7 from 0 to 540672 tid 1 2020-06-12T02:37:34.396 INFO:tasks.rados.rados.0.smithi107.stdout:821: writing smithi10729407-7 from 540672 to 1015808 tid 2 2020-06-12T02:37:34.400 INFO:tasks.rados.rados.0.smithi107.stdout:821: writing smithi10729407-7 from 1015808 to 1613824 tid 3 2020-06-12T02:37:34.401 INFO:tasks.rados.rados.0.smithi107.stdout:821: writing smithi10729407-7 from 1613824 to 1744896 tid 4 2020-06-12T02:37:34.401 INFO:tasks.rados.rados.0.smithi107.stdout:822: delete oid 18 current snap is 0 2020-06-12T02:37:34.402 INFO:tasks.rados.rados.0.smithi107.stdout:822: done (2 left) 2020-06-12T02:37:34.404 INFO:tasks.rados.rados.0.smithi107.stdout:823: delete oid 3 current snap is 0 2020-06-12T02:37:34.405 INFO:tasks.rados.rados.0.smithi107.stdout:819: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:34.405 INFO:tasks.rados.rados.0.smithi107.stdout:819: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:34.405 INFO:tasks.rados.rados.0.smithi107.stdout:819: finishing write tid 5 to smithi10729407-14 2020-06-12T02:37:34.405 INFO:tasks.rados.rados.0.smithi107.stdout:821: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:34.406 INFO:tasks.rados.rados.0.smithi107.stdout:819: finishing write tid 6 to smithi10729407-14 2020-06-12T02:37:34.406 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 123 (ObjNum 818 snap 0 seq_num 818) dirty exists 2020-06-12T02:37:34.406 INFO:tasks.rados.rados.0.smithi107.stdout:819: left oid 14 (ObjNum 818 snap 0 seq_num 818) 2020-06-12T02:37:34.407 INFO:tasks.rados.rados.0.smithi107.stdout:821: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:34.409 INFO:tasks.rados.rados.0.smithi107.stdout:821: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:34.410 INFO:tasks.rados.rados.0.smithi107.stdout:819: done (2 left) 2020-06-12T02:37:34.410 INFO:tasks.rados.rados.0.smithi107.stdout:823: done (1 left) 2020-06-12T02:37:34.410 INFO:tasks.rados.rados.0.smithi107.stdout:824: delete oid 12 current snap is 0 2020-06-12T02:37:34.411 INFO:tasks.rados.rados.0.smithi107.stdout:824: done (1 left) 2020-06-12T02:37:34.411 INFO:tasks.rados.rados.0.smithi107.stdout:825: delete oid 15 current snap is 0 2020-06-12T02:37:34.411 INFO:tasks.rados.rados.0.smithi107.stdout:821: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:34.411 INFO:tasks.rados.rados.0.smithi107.stdout:821: finishing write tid 6 to smithi10729407-7 2020-06-12T02:37:34.412 INFO:tasks.rados.rados.0.smithi107.stdout:825: done (1 left) 2020-06-12T02:37:34.412 INFO:tasks.rados.rados.0.smithi107.stdout:826: delete oid 14 current snap is 0 2020-06-12T02:37:34.412 INFO:tasks.rados.rados.0.smithi107.stdout:821: finishing write tid 7 to smithi10729407-7 2020-06-12T02:37:34.413 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 158 (ObjNum 820 snap 0 seq_num 820) dirty exists 2020-06-12T02:37:34.413 INFO:tasks.rados.rados.0.smithi107.stdout:821: left oid 7 (ObjNum 820 snap 0 seq_num 820) 2020-06-12T02:37:34.418 INFO:tasks.rados.rados.0.smithi107.stdout:821: done (1 left) 2020-06-12T02:37:34.418 INFO:tasks.rados.rados.0.smithi107.stdout:826: done (0 left) 2020-06-12T02:37:34.419 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:37:34.420 INFO:tasks.rados.rados.0.smithi107.stdout:827: seq_num 826 ranges {1556480=638976,2195456=630784,2826240=573440} 2020-06-12T02:37:34.429 INFO:tasks.rados.rados.0.smithi107.stdout:827: writing smithi10729407-11 from 1556480 to 2195456 tid 1 2020-06-12T02:37:34.436 INFO:tasks.rados.rados.0.smithi107.stdout:827: writing smithi10729407-11 from 2195456 to 2826240 tid 2 2020-06-12T02:37:34.437 INFO:tasks.rados.rados.0.smithi107.stdout:827: writing smithi10729407-11 from 2826240 to 3399680 tid 3 2020-06-12T02:37:34.437 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:37:34.438 INFO:tasks.rados.rados.0.smithi107.stdout:828: seq_num 827 ranges {1662976=655360,2318336=516096,2834432=688128} 2020-06-12T02:37:34.460 INFO:tasks.rados.rados.0.smithi107.stdout:828: writing smithi10729407-6 from 1662976 to 2318336 tid 1 2020-06-12T02:37:34.460 INFO:tasks.rados.rados.0.smithi107.stdout:828: writing smithi10729407-6 from 2318336 to 2834432 tid 2 2020-06-12T02:37:34.461 INFO:tasks.rados.rados.0.smithi107.stdout:828: writing smithi10729407-6 from 2834432 to 3522560 tid 3 2020-06-12T02:37:34.461 INFO:tasks.rados.rados.0.smithi107.stdout:829: delete oid 9 current snap is 0 2020-06-12T02:37:34.461 INFO:tasks.rados.rados.0.smithi107.stdout:827: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:34.461 INFO:tasks.rados.rados.0.smithi107.stdout:827: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:34.462 INFO:tasks.rados.rados.0.smithi107.stdout:827: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:34.462 INFO:tasks.rados.rados.0.smithi107.stdout:827: finishing write tid 5 to smithi10729407-11 2020-06-12T02:37:34.462 INFO:tasks.rados.rados.0.smithi107.stdout:827: finishing write tid 6 to smithi10729407-11 2020-06-12T02:37:34.462 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 128 (ObjNum 826 snap 0 seq_num 826) dirty exists 2020-06-12T02:37:34.462 INFO:tasks.rados.rados.0.smithi107.stdout:827: left oid 11 (ObjNum 826 snap 0 seq_num 826) 2020-06-12T02:37:34.463 INFO:tasks.rados.rados.0.smithi107.stdout:828: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:34.463 INFO:tasks.rados.rados.0.smithi107.stdout:827: done (2 left) 2020-06-12T02:37:34.463 INFO:tasks.rados.rados.0.smithi107.stdout:829: done (1 left) 2020-06-12T02:37:34.463 INFO:tasks.rados.rados.0.smithi107.stdout:830: delete oid 13 current snap is 0 2020-06-12T02:37:34.463 INFO:tasks.rados.rados.0.smithi107.stdout:830: done (1 left) 2020-06-12T02:37:34.464 INFO:tasks.rados.rados.0.smithi107.stdout:831: delete oid 16 current snap is 0 2020-06-12T02:37:34.464 INFO:tasks.rados.rados.0.smithi107.stdout:831: done (1 left) 2020-06-12T02:37:34.465 INFO:tasks.rados.rados.0.smithi107.stdout:832: delete oid 17 current snap is 0 2020-06-12T02:37:34.465 INFO:tasks.rados.rados.0.smithi107.stdout:832: done (1 left) 2020-06-12T02:37:34.465 INFO:tasks.rados.rados.0.smithi107.stdout:833: delete oid 19 current snap is 0 2020-06-12T02:37:34.466 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:34.461+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:34.472 INFO:tasks.rados.rados.0.smithi107.stdout:828: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:34.474 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:34.474 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:34.474 INFO:tasks.rados.rados.0.smithi107.stdout:833: done (1 left) 2020-06-12T02:37:34.475 INFO:tasks.rados.rados.0.smithi107.stdout:834: delete oid 15 current snap is 0 2020-06-12T02:37:34.475 INFO:tasks.rados.rados.0.smithi107.stdout:828: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:34.475 INFO:tasks.rados.rados.0.smithi107.stdout:828: finishing write tid 5 to smithi10729407-6 2020-06-12T02:37:34.475 INFO:tasks.rados.rados.0.smithi107.stdout:834: done (1 left) 2020-06-12T02:37:34.476 INFO:tasks.rados.rados.0.smithi107.stdout:835: delete oid 11 current snap is 0 2020-06-12T02:37:34.476 INFO:tasks.rados.rados.0.smithi107.stdout:828: finishing write tid 6 to smithi10729407-6 2020-06-12T02:37:34.476 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 177 (ObjNum 827 snap 0 seq_num 827) dirty exists 2020-06-12T02:37:34.476 INFO:tasks.rados.rados.0.smithi107.stdout:828: left oid 6 (ObjNum 827 snap 0 seq_num 827) 2020-06-12T02:37:34.477 INFO:tasks.rados.rados.0.smithi107.stdout:828: done (1 left) 2020-06-12T02:37:34.477 INFO:tasks.rados.rados.0.smithi107.stdout:835: done (0 left) 2020-06-12T02:37:34.477 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:37:34.479 INFO:tasks.rados.rados.0.smithi107.stdout:836: seq_num 835 ranges {0=794624,794624=442368,1236992=712704,1949696=49152} 2020-06-12T02:37:34.479 INFO:tasks.rados.rados.0.smithi107.stdout:836: writing smithi10729407-18 from 0 to 794624 tid 1 2020-06-12T02:37:34.480 INFO:tasks.rados.rados.0.smithi107.stdout:836: writing smithi10729407-18 from 794624 to 1236992 tid 2 2020-06-12T02:37:34.481 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-06-12T02:37:34.484 INFO:tasks.rados.rados.0.smithi107.stdout:836: writing smithi10729407-18 from 1236992 to 1949696 tid 3 2020-06-12T02:37:34.485 INFO:tasks.rados.rados.0.smithi107.stdout:836: writing smithi10729407-18 from 1949696 to 1998848 tid 4 2020-06-12T02:37:34.485 INFO:tasks.rados.rados.0.smithi107.stdout:837: delete oid 13 current snap is 0 2020-06-12T02:37:34.485 INFO:tasks.rados.rados.0.smithi107.stdout:836: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:34.488 INFO:tasks.rados.rados.0.smithi107.stdout:837: done (1 left) 2020-06-12T02:37:34.488 INFO:tasks.rados.rados.0.smithi107.stdout:838: delete oid 19 current snap is 0 2020-06-12T02:37:34.490 INFO:tasks.rados.rados.0.smithi107.stdout:836: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:34.490 INFO:tasks.rados.rados.0.smithi107.stdout:838: done (1 left) 2020-06-12T02:37:34.490 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:37:34.490 INFO:tasks.rados.rados.0.smithi107.stdout:839: seq_num 838 ranges {1744896=442368,2187264=540672,2727936=532480,3260416=442368,3702784=90112} 2020-06-12T02:37:34.501 INFO:tasks.rados.rados.0.smithi107.stdout:839: writing smithi10729407-7 from 1744896 to 2187264 tid 1 2020-06-12T02:37:34.505 INFO:tasks.rados.rados.0.smithi107.stdout:839: writing smithi10729407-7 from 2187264 to 2727936 tid 2 2020-06-12T02:37:34.508 INFO:tasks.rados.rados.0.smithi107.stdout:839: writing smithi10729407-7 from 2727936 to 3260416 tid 3 2020-06-12T02:37:34.512 INFO:tasks.rados.rados.0.smithi107.stdout:839: writing smithi10729407-7 from 3260416 to 3702784 tid 4 2020-06-12T02:37:34.513 INFO:tasks.rados.rados.0.smithi107.stdout:839: writing smithi10729407-7 from 3702784 to 3792896 tid 5 2020-06-12T02:37:34.513 INFO:tasks.rados.rados.0.smithi107.stdout:840: delete oid 10 current snap is 0 2020-06-12T02:37:34.513 INFO:tasks.rados.rados.0.smithi107.stdout:836: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:34.513 INFO:tasks.rados.rados.0.smithi107.stdout:836: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:34.513 INFO:tasks.rados.rados.0.smithi107.stdout:836: finishing write tid 6 to smithi10729407-18 2020-06-12T02:37:34.514 INFO:tasks.rados.rados.0.smithi107.stdout:836: finishing write tid 7 to smithi10729407-18 2020-06-12T02:37:34.514 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 174 (ObjNum 835 snap 0 seq_num 835) dirty exists 2020-06-12T02:37:34.514 INFO:tasks.rados.rados.0.smithi107.stdout:836: left oid 18 (ObjNum 835 snap 0 seq_num 835) 2020-06-12T02:37:34.514 INFO:tasks.rados.rados.0.smithi107.stdout:839: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:34.517 INFO:tasks.rados.rados.0.smithi107.stdout:839: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:34.517 INFO:tasks.rados.rados.0.smithi107.stdout:839: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:34.517 INFO:tasks.rados.rados.0.smithi107.stdout:836: done (2 left) 2020-06-12T02:37:34.518 INFO:tasks.rados.rados.0.smithi107.stdout:840: done (1 left) 2020-06-12T02:37:34.518 INFO:tasks.rados.rados.0.smithi107.stdout:841: delete oid 18 current snap is 0 2020-06-12T02:37:34.518 INFO:tasks.rados.rados.0.smithi107.stdout:839: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:34.519 INFO:tasks.rados.rados.0.smithi107.stdout:839: finishing write tid 5 to smithi10729407-7 2020-06-12T02:37:34.522 INFO:tasks.rados.rados.0.smithi107.stdout:841: done (1 left) 2020-06-12T02:37:34.522 INFO:tasks.rados.rados.0.smithi107.stdout:842: delete oid 19 current snap is 0 2020-06-12T02:37:34.523 INFO:tasks.rados.rados.0.smithi107.stdout:839: finishing write tid 7 to smithi10729407-7 2020-06-12T02:37:34.523 INFO:tasks.rados.rados.0.smithi107.stdout:839: finishing write tid 8 to smithi10729407-7 2020-06-12T02:37:34.523 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 166 (ObjNum 838 snap 0 seq_num 838) dirty exists 2020-06-12T02:37:34.523 INFO:tasks.rados.rados.0.smithi107.stdout:839: left oid 7 (ObjNum 838 snap 0 seq_num 838) 2020-06-12T02:37:34.524 INFO:tasks.rados.rados.0.smithi107.stdout:839: done (1 left) 2020-06-12T02:37:34.525 INFO:tasks.rados.rados.0.smithi107.stdout:842: done (0 left) 2020-06-12T02:37:34.525 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:37:34.525 INFO:tasks.rados.rados.0.smithi107.stdout:843: seq_num 842 ranges {0=507904,507904=475136,983040=516096,1499136=499712,1998848=114688} 2020-06-12T02:37:34.527 INFO:tasks.rados.rados.0.smithi107.stdout:843: writing smithi10729407-13 from 0 to 507904 tid 1 2020-06-12T02:37:34.531 INFO:tasks.rados.rados.0.smithi107.stdout:843: writing smithi10729407-13 from 507904 to 983040 tid 2 2020-06-12T02:37:34.534 INFO:tasks.rados.rados.0.smithi107.stdout:843: writing smithi10729407-13 from 983040 to 1499136 tid 3 2020-06-12T02:37:34.536 INFO:tasks.rados.rados.0.smithi107.stdout:843: writing smithi10729407-13 from 1499136 to 1998848 tid 4 2020-06-12T02:37:34.537 INFO:tasks.rados.rados.0.smithi107.stdout:843: writing smithi10729407-13 from 1998848 to 2113536 tid 5 2020-06-12T02:37:34.538 INFO:tasks.rados.rados.0.smithi107.stdout:844: delete oid 5 current snap is 0 2020-06-12T02:37:34.538 INFO:tasks.rados.rados.0.smithi107.stdout:843: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:34.539 INFO:tasks.rados.rados.0.smithi107.stdout:844: done (1 left) 2020-06-12T02:37:34.539 INFO:tasks.rados.rados.0.smithi107.stdout:845: delete oid 6 current snap is 0 2020-06-12T02:37:34.541 INFO:tasks.rados.rados.0.smithi107.stdout:845: done (1 left) 2020-06-12T02:37:34.541 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:37:34.541 INFO:tasks.rados.rados.0.smithi107.stdout:846: seq_num 845 ranges {0=557056,557056=581632,1138688=688128,1826816=335872} 2020-06-12T02:37:34.544 INFO:tasks.rados.rados.0.smithi107.stdout:846: writing smithi10729407-2 from 0 to 557056 tid 1 2020-06-12T02:37:34.549 INFO:tasks.rados.rados.0.smithi107.stdout:846: writing smithi10729407-2 from 557056 to 1138688 tid 2 2020-06-12T02:37:34.553 INFO:tasks.rados.rados.0.smithi107.stdout:846: writing smithi10729407-2 from 1138688 to 1826816 tid 3 2020-06-12T02:37:34.555 INFO:tasks.rados.rados.0.smithi107.stdout:846: writing smithi10729407-2 from 1826816 to 2162688 tid 4 2020-06-12T02:37:34.556 INFO:tasks.rados.rados.0.smithi107.stdout:847: delete oid 5 current snap is 0 2020-06-12T02:37:34.556 INFO:tasks.rados.rados.0.smithi107.stdout:843: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:34.556 INFO:tasks.rados.rados.0.smithi107.stdout:843: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:34.557 INFO:tasks.rados.rados.0.smithi107.stdout:843: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:34.557 INFO:tasks.rados.rados.0.smithi107.stdout:843: finishing write tid 5 to smithi10729407-13 2020-06-12T02:37:34.557 INFO:tasks.rados.rados.0.smithi107.stdout:843: finishing write tid 7 to smithi10729407-13 2020-06-12T02:37:34.558 INFO:tasks.rados.rados.0.smithi107.stdout:843: finishing write tid 8 to smithi10729407-13 2020-06-12T02:37:34.558 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 172 (ObjNum 842 snap 0 seq_num 842) dirty exists 2020-06-12T02:37:34.559 INFO:tasks.rados.rados.0.smithi107.stdout:843: left oid 13 (ObjNum 842 snap 0 seq_num 842) 2020-06-12T02:37:34.559 INFO:tasks.rados.rados.0.smithi107.stdout:846: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:34.560 INFO:tasks.rados.rados.0.smithi107.stdout:843: done (2 left) 2020-06-12T02:37:34.561 INFO:tasks.rados.rados.0.smithi107.stdout:847: done (1 left) 2020-06-12T02:37:34.561 INFO:tasks.rados.rados.0.smithi107.stdout:848: delete oid 3 current snap is 0 2020-06-12T02:37:34.561 INFO:tasks.rados.rados.0.smithi107.stdout:848: done (1 left) 2020-06-12T02:37:34.562 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:37:34.562 INFO:tasks.rados.rados.0.smithi107.stdout:849: seq_num 848 ranges {0=606208,606208=630784,1236992=458752,1695744=516096} 2020-06-12T02:37:34.573 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:34.559+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:34.573 INFO:tasks.rados.rados.0.smithi107.stdout:849: writing smithi10729407-16 from 0 to 606208 tid 1 2020-06-12T02:37:34.575 INFO:teuthology.orchestra.run.smithi188.stderr:Nothing to do for non-filestore backend 2020-06-12T02:37:34.576 INFO:tasks.rados.rados.0.smithi107.stdout:849: writing smithi10729407-16 from 606208 to 1236992 tid 2 2020-06-12T02:37:34.576 INFO:tasks.rados.rados.0.smithi107.stdout:849: writing smithi10729407-16 from 1236992 to 1695744 tid 3 2020-06-12T02:37:34.576 INFO:tasks.rados.rados.0.smithi107.stdout:849: writing smithi10729407-16 from 1695744 to 2211840 tid 4 2020-06-12T02:37:34.577 INFO:tasks.rados.rados.0.smithi107.stdout:850: delete oid 12 current snap is 0 2020-06-12T02:37:34.577 INFO:tasks.rados.rados.0.smithi107.stdout:846: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:34.577 INFO:tasks.rados.rados.0.smithi107.stdout:846: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:34.578 INFO:tasks.rados.rados.0.smithi107.stdout:846: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:34.578 INFO:tasks.rados.rados.0.smithi107.stdout:846: finishing write tid 6 to smithi10729407-2 2020-06-12T02:37:34.578 INFO:tasks.rados.rados.0.smithi107.stdout:846: finishing write tid 7 to smithi10729407-2 2020-06-12T02:37:34.579 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 87 (ObjNum 845 snap 0 seq_num 845) dirty exists 2020-06-12T02:37:34.579 INFO:tasks.rados.rados.0.smithi107.stdout:846: left oid 2 (ObjNum 845 snap 0 seq_num 845) 2020-06-12T02:37:34.580 INFO:tasks.rados.rados.0.smithi107.stdout:846: done (2 left) 2020-06-12T02:37:34.580 INFO:tasks.rados.rados.0.smithi107.stdout:850: done (1 left) 2020-06-12T02:37:34.580 INFO:tasks.rados.rados.0.smithi107.stdout:851: delete oid 9 current snap is 0 2020-06-12T02:37:34.582 INFO:tasks.rados.rados.0.smithi107.stdout:849: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:34.583 INFO:tasks.rados.rados.0.smithi107.stdout:849: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:34.583 INFO:tasks.rados.rados.0.smithi107.stdout:849: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:34.583 INFO:tasks.rados.rados.0.smithi107.stdout:851: done (1 left) 2020-06-12T02:37:34.584 INFO:tasks.rados.rados.0.smithi107.stdout:852: delete oid 6 current snap is 0 2020-06-12T02:37:34.584 INFO:tasks.rados.rados.0.smithi107.stdout:849: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:34.584 INFO:tasks.rados.rados.0.smithi107.stdout:852: done (1 left) 2020-06-12T02:37:34.584 INFO:tasks.rados.rados.0.smithi107.stdout:853: delete oid 19 current snap is 0 2020-06-12T02:37:34.584 INFO:tasks.rados.rados.0.smithi107.stdout:849: finishing write tid 6 to smithi10729407-16 2020-06-12T02:37:34.584 INFO:tasks.rados.rados.0.smithi107.stdout:853: done (1 left) 2020-06-12T02:37:34.585 INFO:tasks.rados.rados.0.smithi107.stdout:854: delete oid 9 current snap is 0 2020-06-12T02:37:34.585 INFO:tasks.rados.rados.0.smithi107.stdout:849: finishing write tid 7 to smithi10729407-16 2020-06-12T02:37:34.585 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 156 (ObjNum 848 snap 0 seq_num 848) dirty exists 2020-06-12T02:37:34.585 INFO:tasks.rados.rados.0.smithi107.stdout:849: left oid 16 (ObjNum 848 snap 0 seq_num 848) 2020-06-12T02:37:34.586 INFO:tasks.rados.rados.0.smithi107.stdout:849: done (1 left) 2020-06-12T02:37:34.586 INFO:tasks.rados.rados.0.smithi107.stdout:854: done (0 left) 2020-06-12T02:37:34.586 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:37:34.587 INFO:tasks.rados.rados.0.smithi107.stdout:855: seq_num 854 ranges {0=712704,712704=794624,1507328=450560,1957888=360448} 2020-06-12T02:37:34.587 INFO:tasks.rados.rados.0.smithi107.stdout:855: writing smithi10729407-10 from 0 to 712704 tid 1 2020-06-12T02:37:34.591 INFO:tasks.rados.rados.0.smithi107.stdout:855: writing smithi10729407-10 from 712704 to 1507328 tid 2 2020-06-12T02:37:34.594 INFO:tasks.rados.rados.0.smithi107.stdout:855: writing smithi10729407-10 from 1507328 to 1957888 tid 3 2020-06-12T02:37:34.596 INFO:tasks.rados.rados.0.smithi107.stdout:855: writing smithi10729407-10 from 1957888 to 2318336 tid 4 2020-06-12T02:37:34.597 INFO:tasks.rados.rados.0.smithi107.stdout:856: delete oid 8 current snap is 0 2020-06-12T02:37:34.597 INFO:tasks.rados.rados.0.smithi107.stdout:855: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:34.599 INFO:tasks.rados.rados.0.smithi107.stdout:856: done (1 left) 2020-06-12T02:37:34.599 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:37:34.599 INFO:tasks.rados.rados.0.smithi107.stdout:857: seq_num 856 ranges {0=745472,745472=557056,1302528=417792,1720320=434176,2154496=196608} 2020-06-12T02:37:34.604 INFO:tasks.rados.rados.0.smithi107.stdout:857: writing smithi10729407-14 from 0 to 745472 tid 1 2020-06-12T02:37:34.607 INFO:tasks.rados.rados.0.smithi107.stdout:857: writing smithi10729407-14 from 745472 to 1302528 tid 2 2020-06-12T02:37:34.610 INFO:tasks.rados.rados.0.smithi107.stdout:857: writing smithi10729407-14 from 1302528 to 1720320 tid 3 2020-06-12T02:37:34.613 INFO:tasks.rados.rados.0.smithi107.stdout:857: writing smithi10729407-14 from 1720320 to 2154496 tid 4 2020-06-12T02:37:34.614 INFO:tasks.rados.rados.0.smithi107.stdout:857: writing smithi10729407-14 from 2154496 to 2351104 tid 5 2020-06-12T02:37:34.615 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:37:34.616 INFO:tasks.rados.rados.0.smithi107.stdout:858: seq_num 857 ranges {0=761856,761856=442368,1204224=630784,1835008=532480} 2020-06-12T02:37:34.619 INFO:tasks.rados.rados.0.smithi107.stdout:858: writing smithi10729407-1 from 0 to 761856 tid 1 2020-06-12T02:37:34.622 INFO:tasks.rados.rados.0.smithi107.stdout:858: writing smithi10729407-1 from 761856 to 1204224 tid 2 2020-06-12T02:37:34.626 INFO:tasks.rados.rados.0.smithi107.stdout:858: writing smithi10729407-1 from 1204224 to 1835008 tid 3 2020-06-12T02:37:34.630 INFO:tasks.rados.rados.0.smithi107.stdout:858: writing smithi10729407-1 from 1835008 to 2367488 tid 4 2020-06-12T02:37:34.630 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:37:34.631 INFO:tasks.rados.rados.0.smithi107.stdout:859: seq_num 858 ranges {2211840=778240,2990080=729088,3719168=434176,4153344=442368} 2020-06-12T02:37:34.633 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:34.633 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:34.642 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-06-12T02:37:34.646 INFO:tasks.rados.rados.0.smithi107.stdout:859: writing smithi10729407-16 from 2211840 to 2990080 tid 1 2020-06-12T02:37:34.650 INFO:tasks.rados.rados.0.smithi107.stdout:859: writing smithi10729407-16 from 2990080 to 3719168 tid 2 2020-06-12T02:37:34.653 INFO:tasks.rados.rados.0.smithi107.stdout:859: writing smithi10729407-16 from 3719168 to 4153344 tid 3 2020-06-12T02:37:34.656 INFO:tasks.rados.rados.0.smithi107.stdout:859: writing smithi10729407-16 from 4153344 to 4595712 tid 4 2020-06-12T02:37:34.657 INFO:tasks.rados.rados.0.smithi107.stdout:860: delete oid 11 current snap is 0 2020-06-12T02:37:34.657 INFO:tasks.rados.rados.0.smithi107.stdout:855: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:34.657 INFO:tasks.rados.rados.0.smithi107.stdout:855: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:34.657 INFO:tasks.rados.rados.0.smithi107.stdout:855: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:34.657 INFO:tasks.rados.rados.0.smithi107.stdout:855: finishing write tid 6 to smithi10729407-10 2020-06-12T02:37:34.658 INFO:tasks.rados.rados.0.smithi107.stdout:855: finishing write tid 7 to smithi10729407-10 2020-06-12T02:37:34.658 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 73 (ObjNum 854 snap 0 seq_num 854) dirty exists 2020-06-12T02:37:34.658 INFO:tasks.rados.rados.0.smithi107.stdout:855: left oid 10 (ObjNum 854 snap 0 seq_num 854) 2020-06-12T02:37:34.661 INFO:tasks.rados.rados.0.smithi107.stdout:857: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:34.661 INFO:tasks.rados.rados.0.smithi107.stdout:857: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:34.661 INFO:tasks.rados.rados.0.smithi107.stdout:857: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:34.661 INFO:tasks.rados.rados.0.smithi107.stdout:857: finishing write tid 4 to smithi10729407-14 2020-06-12T02:37:34.662 INFO:tasks.rados.rados.0.smithi107.stdout:857: finishing write tid 5 to smithi10729407-14 2020-06-12T02:37:34.662 INFO:tasks.rados.rados.0.smithi107.stdout:857: finishing write tid 7 to smithi10729407-14 2020-06-12T02:37:34.662 INFO:tasks.rados.rados.0.smithi107.stdout:857: finishing write tid 8 to smithi10729407-14 2020-06-12T02:37:34.662 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 135 (ObjNum 856 snap 0 seq_num 856) dirty exists 2020-06-12T02:37:34.662 INFO:tasks.rados.rados.0.smithi107.stdout:857: left oid 14 (ObjNum 856 snap 0 seq_num 856) 2020-06-12T02:37:34.662 INFO:tasks.rados.rados.0.smithi107.stdout:858: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:34.663 INFO:tasks.rados.rados.0.smithi107.stdout:858: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:34.663 INFO:tasks.rados.rados.0.smithi107.stdout:858: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:34.663 INFO:tasks.rados.rados.0.smithi107.stdout:858: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:34.663 INFO:tasks.rados.rados.0.smithi107.stdout:858: finishing write tid 6 to smithi10729407-1 2020-06-12T02:37:34.663 INFO:tasks.rados.rados.0.smithi107.stdout:858: finishing write tid 7 to smithi10729407-1 2020-06-12T02:37:34.664 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 184 (ObjNum 857 snap 0 seq_num 857) dirty exists 2020-06-12T02:37:34.664 INFO:tasks.rados.rados.0.smithi107.stdout:858: left oid 1 (ObjNum 857 snap 0 seq_num 857) 2020-06-12T02:37:34.665 INFO:tasks.rados.rados.0.smithi107.stdout:859: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:34.665 INFO:tasks.rados.rados.0.smithi107.stdout:855: done (4 left) 2020-06-12T02:37:34.665 INFO:tasks.rados.rados.0.smithi107.stdout:857: done (3 left) 2020-06-12T02:37:34.666 INFO:tasks.rados.rados.0.smithi107.stdout:858: done (2 left) 2020-06-12T02:37:34.666 INFO:tasks.rados.rados.0.smithi107.stdout:860: done (1 left) 2020-06-12T02:37:34.666 INFO:tasks.rados.rados.0.smithi107.stdout:861: delete oid 12 current snap is 0 2020-06-12T02:37:34.666 INFO:tasks.rados.rados.0.smithi107.stdout:861: done (1 left) 2020-06-12T02:37:34.666 INFO:tasks.rados.rados.0.smithi107.stdout:862: delete oid 10 current snap is 0 2020-06-12T02:37:34.667 INFO:tasks.rados.rados.0.smithi107.stdout:862: done (1 left) 2020-06-12T02:37:34.667 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:37:34.667 INFO:tasks.rados.rados.0.smithi107.stdout:863: seq_num 862 ranges {0=40960} 2020-06-12T02:37:34.667 INFO:tasks.rados.rados.0.smithi107.stdout:863: writing smithi10729407-11 from 0 to 40960 tid 1 2020-06-12T02:37:34.667 INFO:tasks.rados.rados.0.smithi107.stdout:864: delete oid 7 current snap is 0 2020-06-12T02:37:34.668 INFO:tasks.rados.rados.0.smithi107.stdout:859: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:34.668 INFO:tasks.rados.rados.0.smithi107.stdout:859: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:34.668 INFO:tasks.rados.rados.0.smithi107.stdout:859: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:34.670 INFO:tasks.rados.rados.0.smithi107.stdout:864: done (2 left) 2020-06-12T02:37:34.673 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:37:34.673 INFO:tasks.rados.rados.0.smithi107.stdout:865: seq_num 864 ranges {0=73728} 2020-06-12T02:37:34.674 INFO:tasks.rados.rados.0.smithi107.stdout:865: writing smithi10729407-3 from 0 to 73728 tid 1 2020-06-12T02:37:34.674 INFO:tasks.rados.rados.0.smithi107.stdout:866: delete oid 15 current snap is 0 2020-06-12T02:37:34.674 INFO:tasks.rados.rados.0.smithi107.stdout:863: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:34.675 INFO:tasks.rados.rados.0.smithi107.stdout:859: finishing write tid 6 to smithi10729407-16 2020-06-12T02:37:34.675 INFO:tasks.rados.rados.0.smithi107.stdout:866: done (3 left) 2020-06-12T02:37:34.675 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:37:34.675 INFO:tasks.rados.rados.0.smithi107.stdout:867: seq_num 866 ranges {0=106496} 2020-06-12T02:37:34.675 INFO:tasks.rados.rados.0.smithi107.stdout:867: writing smithi10729407-17 from 0 to 106496 tid 1 2020-06-12T02:37:34.676 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:37:34.676 INFO:tasks.rados.rados.0.smithi107.stdout:868: seq_num 867 ranges {0=122880} 2020-06-12T02:37:34.676 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:34.663+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:34.677 INFO:tasks.rados.rados.0.smithi107.stdout:868: writing smithi10729407-20 from 0 to 122880 tid 1 2020-06-12T02:37:34.677 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:37:34.678 INFO:tasks.rados.rados.0.smithi107.stdout:869: seq_num 868 ranges {0=139264} 2020-06-12T02:37:34.678 INFO:tasks.rados.rados.0.smithi107.stdout:869: writing smithi10729407-10 from 0 to 139264 tid 1 2020-06-12T02:37:34.678 INFO:tasks.rados.rados.0.smithi107.stdout:870: delete oid 12 current snap is 0 2020-06-12T02:37:34.678 INFO:tasks.rados.rados.0.smithi107.stdout:863: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:34.678 INFO:tasks.rados.rados.0.smithi107.stdout:859: finishing write tid 7 to smithi10729407-16 2020-06-12T02:37:34.680 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 161 (ObjNum 858 snap 0 seq_num 858) dirty exists 2020-06-12T02:37:34.680 INFO:tasks.rados.rados.0.smithi107.stdout:859: left oid 16 (ObjNum 858 snap 0 seq_num 858) 2020-06-12T02:37:34.681 INFO:tasks.rados.rados.0.smithi107.stdout:865: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:34.681 INFO:tasks.rados.rados.0.smithi107.stdout:867: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:34.681 INFO:tasks.rados.rados.0.smithi107.stdout:859: done (6 left) 2020-06-12T02:37:34.681 INFO:tasks.rados.rados.0.smithi107.stdout:870: done (5 left) 2020-06-12T02:37:34.681 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:37:34.682 INFO:tasks.rados.rados.0.smithi107.stdout:871: seq_num 870 ranges {2113536=172032} 2020-06-12T02:37:34.687 INFO:tasks.rados.rados.0.smithi107.stdout:871: writing smithi10729407-13 from 2113536 to 2285568 tid 1 2020-06-12T02:37:34.688 INFO:tasks.rados.rados.0.smithi107.stdout:872: delete oid 8 current snap is 0 2020-06-12T02:37:34.688 INFO:tasks.rados.rados.0.smithi107.stdout:863: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:34.688 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 138 (ObjNum 862 snap 0 seq_num 862) dirty exists 2020-06-12T02:37:34.688 INFO:tasks.rados.rados.0.smithi107.stdout:863: left oid 11 (ObjNum 862 snap 0 seq_num 862) 2020-06-12T02:37:34.688 INFO:tasks.rados.rados.0.smithi107.stdout:865: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:34.689 INFO:tasks.rados.rados.0.smithi107.stdout:867: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:34.689 INFO:tasks.rados.rados.0.smithi107.stdout:868: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:34.689 INFO:tasks.rados.rados.0.smithi107.stdout:868: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:34.689 INFO:tasks.rados.rados.0.smithi107.stdout:868: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:34.690 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 81 (ObjNum 867 snap 0 seq_num 867) dirty exists 2020-06-12T02:37:34.690 INFO:tasks.rados.rados.0.smithi107.stdout:868: left oid 20 (ObjNum 867 snap 0 seq_num 867) 2020-06-12T02:37:34.690 INFO:tasks.rados.rados.0.smithi107.stdout:867: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:34.691 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 157 (ObjNum 866 snap 0 seq_num 866) dirty exists 2020-06-12T02:37:34.691 INFO:tasks.rados.rados.0.smithi107.stdout:867: left oid 17 (ObjNum 866 snap 0 seq_num 866) 2020-06-12T02:37:34.691 INFO:tasks.rados.rados.0.smithi107.stdout:865: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:34.691 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 163 (ObjNum 864 snap 0 seq_num 864) dirty exists 2020-06-12T02:37:34.691 INFO:tasks.rados.rados.0.smithi107.stdout:865: left oid 3 (ObjNum 864 snap 0 seq_num 864) 2020-06-12T02:37:34.693 INFO:tasks.rados.rados.0.smithi107.stdout:869: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:34.693 INFO:tasks.rados.rados.0.smithi107.stdout:869: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:34.693 INFO:tasks.rados.rados.0.smithi107.stdout:869: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:34.694 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 76 (ObjNum 868 snap 0 seq_num 868) dirty exists 2020-06-12T02:37:34.694 INFO:tasks.rados.rados.0.smithi107.stdout:869: left oid 10 (ObjNum 868 snap 0 seq_num 868) 2020-06-12T02:37:34.695 INFO:tasks.rados.rados.0.smithi107.stdout:863: done (6 left) 2020-06-12T02:37:34.695 INFO:tasks.rados.rados.0.smithi107.stdout:865: done (5 left) 2020-06-12T02:37:34.696 INFO:tasks.rados.rados.0.smithi107.stdout:867: done (4 left) 2020-06-12T02:37:34.696 INFO:tasks.rados.rados.0.smithi107.stdout:868: done (3 left) 2020-06-12T02:37:34.696 INFO:tasks.rados.rados.0.smithi107.stdout:869: done (2 left) 2020-06-12T02:37:34.696 INFO:tasks.rados.rados.0.smithi107.stdout:872: done (1 left) 2020-06-12T02:37:34.696 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:37:34.697 INFO:tasks.rados.rados.0.smithi107.stdout:873: seq_num 872 ranges {0=212992} 2020-06-12T02:37:34.697 INFO:tasks.rados.rados.0.smithi107.stdout:873: writing smithi10729407-8 from 0 to 212992 tid 1 2020-06-12T02:37:34.697 INFO:tasks.rados.rados.0.smithi107.stdout:874: delete oid 10 current snap is 0 2020-06-12T02:37:34.697 INFO:tasks.rados.rados.0.smithi107.stdout:874: done (2 left) 2020-06-12T02:37:34.697 INFO:tasks.rados.rados.0.smithi107.stdout:875: delete oid 12 current snap is 0 2020-06-12T02:37:34.698 INFO:tasks.rados.rados.0.smithi107.stdout:871: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:34.698 INFO:tasks.rados.rados.0.smithi107.stdout:873: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:34.699 INFO:tasks.rados.rados.0.smithi107.stdout:875: done (2 left) 2020-06-12T02:37:34.699 INFO:tasks.rados.rados.0.smithi107.stdout:876: delete oid 14 current snap is 0 2020-06-12T02:37:34.700 INFO:tasks.rados.rados.0.smithi107.stdout:873: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:34.700 INFO:tasks.rados.rados.0.smithi107.stdout:871: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:34.700 INFO:tasks.rados.rados.0.smithi107.stdout:873: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:34.700 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 164 (ObjNum 872 snap 0 seq_num 872) dirty exists 2020-06-12T02:37:34.700 INFO:tasks.rados.rados.0.smithi107.stdout:873: left oid 8 (ObjNum 872 snap 0 seq_num 872) 2020-06-12T02:37:34.700 INFO:tasks.rados.rados.0.smithi107.stdout:871: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:34.701 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 175 (ObjNum 870 snap 0 seq_num 870) dirty exists 2020-06-12T02:37:34.701 INFO:tasks.rados.rados.0.smithi107.stdout:871: left oid 13 (ObjNum 870 snap 0 seq_num 870) 2020-06-12T02:37:34.701 INFO:tasks.rados.rados.0.smithi107.stdout:871: done (2 left) 2020-06-12T02:37:34.701 INFO:tasks.rados.rados.0.smithi107.stdout:873: done (1 left) 2020-06-12T02:37:34.701 INFO:tasks.rados.rados.0.smithi107.stdout:876: done (0 left) 2020-06-12T02:37:34.702 INFO:tasks.rados.rados.0.smithi107.stdout:877: delete oid 10 current snap is 0 2020-06-12T02:37:34.704 INFO:tasks.rados.rados.0.smithi107.stdout:877: done (0 left) 2020-06-12T02:37:34.704 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:37:34.704 INFO:tasks.rados.rados.0.smithi107.stdout:878: seq_num 877 ranges {0=294912} 2020-06-12T02:37:34.707 INFO:tasks.rados.rados.0.smithi107.stdout:878: writing smithi10729407-4 from 0 to 294912 tid 1 2020-06-12T02:37:34.707 INFO:tasks.rados.rados.0.smithi107.stdout:879: delete oid 16 current snap is 0 2020-06-12T02:37:34.708 INFO:tasks.rados.rados.0.smithi107.stdout:879: done (1 left) 2020-06-12T02:37:34.708 INFO:tasks.rados.rados.0.smithi107.stdout:880: delete oid 10 current snap is 0 2020-06-12T02:37:34.710 INFO:tasks.rados.rados.0.smithi107.stdout:878: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:34.711 INFO:tasks.rados.rados.0.smithi107.stdout:878: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:34.711 INFO:tasks.rados.rados.0.smithi107.stdout:880: done (1 left) 2020-06-12T02:37:34.711 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:37:34.712 INFO:tasks.rados.rados.0.smithi107.stdout:881: seq_num 880 ranges {212992=344064} 2020-06-12T02:37:34.714 INFO:tasks.rados.rados.0.smithi107.stdout:881: writing smithi10729407-8 from 212992 to 557056 tid 1 2020-06-12T02:37:34.715 INFO:tasks.rados.rados.0.smithi107.stdout:882: delete oid 2 current snap is 0 2020-06-12T02:37:34.715 INFO:tasks.rados.rados.0.smithi107.stdout:878: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:34.716 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 159 (ObjNum 877 snap 0 seq_num 877) dirty exists 2020-06-12T02:37:34.716 INFO:tasks.rados.rados.0.smithi107.stdout:878: left oid 4 (ObjNum 877 snap 0 seq_num 877) 2020-06-12T02:37:34.717 INFO:tasks.rados.rados.0.smithi107.stdout:878: done (2 left) 2020-06-12T02:37:34.717 INFO:tasks.rados.rados.0.smithi107.stdout:882: done (1 left) 2020-06-12T02:37:34.718 INFO:tasks.rados.rados.0.smithi107.stdout:883: delete oid 20 current snap is 0 2020-06-12T02:37:34.719 INFO:tasks.rados.rados.0.smithi107.stdout:883: done (1 left) 2020-06-12T02:37:34.719 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:37:34.720 INFO:tasks.rados.rados.0.smithi107.stdout:884: seq_num 883 ranges {2367488=393216} 2020-06-12T02:37:34.734 INFO:tasks.rados.rados.0.smithi107.stdout:884: writing smithi10729407-1 from 2367488 to 2760704 tid 1 2020-06-12T02:37:34.734 INFO:tasks.rados.rados.0.smithi107.stdout:885: delete oid 14 current snap is 0 2020-06-12T02:37:34.734 INFO:tasks.rados.rados.0.smithi107.stdout:881: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:34.735 INFO:tasks.rados.rados.0.smithi107.stdout:881: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:34.735 INFO:tasks.rados.rados.0.smithi107.stdout:881: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:34.735 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 166 (ObjNum 880 snap 0 seq_num 880) dirty exists 2020-06-12T02:37:34.736 INFO:tasks.rados.rados.0.smithi107.stdout:881: left oid 8 (ObjNum 880 snap 0 seq_num 880) 2020-06-12T02:37:34.737 INFO:tasks.rados.rados.0.smithi107.stdout:884: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:34.739 INFO:tasks.rados.rados.0.smithi107.stdout:881: done (2 left) 2020-06-12T02:37:34.739 INFO:tasks.rados.rados.0.smithi107.stdout:885: done (1 left) 2020-06-12T02:37:34.740 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:37:34.740 INFO:tasks.rados.rados.0.smithi107.stdout:886: seq_num 885 ranges {40960=425984} 2020-06-12T02:37:34.743 INFO:tasks.rados.rados.0.smithi107.stdout:886: writing smithi10729407-11 from 40960 to 466944 tid 1 2020-06-12T02:37:34.743 INFO:tasks.rados.rados.0.smithi107.stdout:887: delete oid 14 current snap is 0 2020-06-12T02:37:34.744 INFO:tasks.rados.rados.0.smithi107.stdout:884: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:34.744 INFO:tasks.rados.rados.0.smithi107.stdout:884: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:34.744 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 186 (ObjNum 883 snap 0 seq_num 883) dirty exists 2020-06-12T02:37:34.745 INFO:tasks.rados.rados.0.smithi107.stdout:884: left oid 1 (ObjNum 883 snap 0 seq_num 883) 2020-06-12T02:37:34.746 INFO:tasks.rados.rados.0.smithi107.stdout:886: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:34.750 INFO:tasks.rados.rados.0.smithi107.stdout:884: done (2 left) 2020-06-12T02:37:34.750 INFO:tasks.rados.rados.0.smithi107.stdout:887: done (1 left) 2020-06-12T02:37:34.750 INFO:tasks.rados.rados.0.smithi107.stdout:888: delete oid 3 current snap is 0 2020-06-12T02:37:34.752 INFO:tasks.rados.rados.0.smithi107.stdout:886: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:34.752 INFO:tasks.rados.rados.0.smithi107.stdout:886: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:34.753 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 143 (ObjNum 885 snap 0 seq_num 885) dirty exists 2020-06-12T02:37:34.753 INFO:tasks.rados.rados.0.smithi107.stdout:886: left oid 11 (ObjNum 885 snap 0 seq_num 885) 2020-06-12T02:37:34.754 INFO:tasks.rados.rados.0.smithi107.stdout:886: done (1 left) 2020-06-12T02:37:34.754 INFO:tasks.rados.rados.0.smithi107.stdout:888: done (0 left) 2020-06-12T02:37:34.755 INFO:tasks.rados.rados.0.smithi107.stdout:889: delete oid 2 current snap is 0 2020-06-12T02:37:34.756 INFO:tasks.rados.rados.0.smithi107.stdout:889: done (0 left) 2020-06-12T02:37:34.756 INFO:tasks.rados.rados.0.smithi107.stdout:890: delete oid 10 current snap is 0 2020-06-12T02:37:34.758 INFO:tasks.rados.rados.0.smithi107.stdout:890: done (0 left) 2020-06-12T02:37:34.758 INFO:tasks.rados.rados.0.smithi107.stdout:891: delete oid 19 current snap is 0 2020-06-12T02:37:34.759 INFO:tasks.rados.rados.0.smithi107.stdout:891: done (0 left) 2020-06-12T02:37:34.760 INFO:tasks.rados.rados.0.smithi107.stdout:892: delete oid 1 current snap is 0 2020-06-12T02:37:34.761 INFO:tasks.rados.rados.0.smithi107.stdout:892: done (0 left) 2020-06-12T02:37:34.761 INFO:tasks.rados.rados.0.smithi107.stdout:893: delete oid 12 current snap is 0 2020-06-12T02:37:34.762 INFO:tasks.rados.rados.0.smithi107.stdout:893: done (0 left) 2020-06-12T02:37:34.762 INFO:tasks.rados.rados.0.smithi107.stdout:894: delete oid 20 current snap is 0 2020-06-12T02:37:34.764 INFO:tasks.rados.rados.0.smithi107.stdout:894: done (0 left) 2020-06-12T02:37:34.764 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:37:34.765 INFO:tasks.rados.rados.0.smithi107.stdout:895: seq_num 894 ranges {0=581632} 2020-06-12T02:37:34.773 INFO:tasks.rados.rados.0.smithi107.stdout:895: writing smithi10729407-6 from 0 to 581632 tid 1 2020-06-12T02:37:34.774 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:37:34.775 INFO:tasks.rados.rados.0.smithi107.stdout:896: seq_num 895 ranges {0=598016} 2020-06-12T02:37:34.775 INFO:tasks.rados.rados.0.smithi107.stdout:896: writing smithi10729407-10 from 0 to 598016 tid 1 2020-06-12T02:37:34.776 INFO:tasks.rados.rados.0.smithi107.stdout:897: delete oid 11 current snap is 0 2020-06-12T02:37:34.776 INFO:tasks.rados.rados.0.smithi107.stdout:897: done (2 left) 2020-06-12T02:37:34.777 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:37:34.777 INFO:tasks.rados.rados.0.smithi107.stdout:898: seq_num 897 ranges {0=630784} 2020-06-12T02:37:34.779 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:34.765+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:37:34.780 INFO:tasks.rados.rados.0.smithi107.stdout:898: writing smithi10729407-18 from 0 to 630784 tid 1 2020-06-12T02:37:34.781 INFO:tasks.rados.rados.0.smithi107.stdout:899: delete oid 1 current snap is 0 2020-06-12T02:37:34.781 INFO:tasks.rados.rados.0.smithi107.stdout:895: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:34.782 INFO:tasks.rados.rados.0.smithi107.stdout:895: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:34.782 INFO:tasks.rados.rados.0.smithi107.stdout:896: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:34.783 INFO:tasks.rados.rados.0.smithi107.stdout:895: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:34.784 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 189 (ObjNum 894 snap 0 seq_num 894) dirty exists 2020-06-12T02:37:34.784 INFO:tasks.rados.rados.0.smithi107.stdout:895: left oid 6 (ObjNum 894 snap 0 seq_num 894) 2020-06-12T02:37:34.785 INFO:tasks.rados.rados.0.smithi107.stdout:895: done (3 left) 2020-06-12T02:37:34.786 INFO:tasks.rados.rados.0.smithi107.stdout:899: done (2 left) 2020-06-12T02:37:34.786 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:37:34.787 INFO:tasks.rados.rados.0.smithi107.stdout:900: seq_num 899 ranges {0=663552} 2020-06-12T02:37:34.788 INFO:tasks.rados.rados.0.smithi107.stdout:900: writing smithi10729407-2 from 0 to 663552 tid 1 2020-06-12T02:37:34.789 INFO:tasks.rados.rados.0.smithi107.stdout:901: delete oid 19 current snap is 0 2020-06-12T02:37:34.789 INFO:tasks.rados.rados.0.smithi107.stdout:896: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:34.789 INFO:tasks.rados.rados.0.smithi107.stdout:896: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:34.790 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 82 (ObjNum 895 snap 0 seq_num 895) dirty exists 2020-06-12T02:37:34.790 INFO:tasks.rados.rados.0.smithi107.stdout:896: left oid 10 (ObjNum 895 snap 0 seq_num 895) 2020-06-12T02:37:34.790 INFO:tasks.rados.rados.0.smithi107.stdout:898: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:34.791 INFO:tasks.rados.rados.0.smithi107.stdout:898: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:34.791 INFO:tasks.rados.rados.0.smithi107.stdout:896: done (3 left) 2020-06-12T02:37:34.791 INFO:tasks.rados.rados.0.smithi107.stdout:901: done (2 left) 2020-06-12T02:37:34.792 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:37:34.792 INFO:tasks.rados.rados.0.smithi107.stdout:902: seq_num 901 ranges {0=696320} 2020-06-12T02:37:34.793 INFO:tasks.rados.rados.0.smithi107.stdout:902: writing smithi10729407-11 from 0 to 696320 tid 1 2020-06-12T02:37:34.793 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:37:34.794 INFO:tasks.rados.rados.0.smithi107.stdout:903: seq_num 902 ranges {0=712704} 2020-06-12T02:37:34.795 INFO:tasks.rados.rados.0.smithi107.stdout:903: writing smithi10729407-12 from 0 to 712704 tid 1 2020-06-12T02:37:34.796 INFO:tasks.rados.rados.0.smithi107.stdout:904: delete oid 9 current snap is 0 2020-06-12T02:37:34.796 INFO:tasks.rados.rados.0.smithi107.stdout:898: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:34.796 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 179 (ObjNum 897 snap 0 seq_num 897) dirty exists 2020-06-12T02:37:34.797 INFO:tasks.rados.rados.0.smithi107.stdout:898: left oid 18 (ObjNum 897 snap 0 seq_num 897) 2020-06-12T02:37:34.797 INFO:tasks.rados.rados.0.smithi107.stdout:900: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:34.797 INFO:tasks.rados.rados.0.smithi107.stdout:900: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:34.798 INFO:tasks.rados.rados.0.smithi107.stdout:900: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:34.798 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 91 (ObjNum 899 snap 0 seq_num 899) dirty exists 2020-06-12T02:37:34.798 INFO:tasks.rados.rados.0.smithi107.stdout:900: left oid 2 (ObjNum 899 snap 0 seq_num 899) 2020-06-12T02:37:34.799 INFO:tasks.rados.rados.0.smithi107.stdout:898: done (4 left) 2020-06-12T02:37:34.800 INFO:tasks.rados.rados.0.smithi107.stdout:900: done (3 left) 2020-06-12T02:37:34.800 INFO:tasks.rados.rados.0.smithi107.stdout:904: done (2 left) 2020-06-12T02:37:34.800 INFO:tasks.rados.rados.0.smithi107.stdout:905: delete oid 10 current snap is 0 2020-06-12T02:37:34.801 INFO:tasks.rados.rados.0.smithi107.stdout:902: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:34.801 INFO:tasks.rados.rados.0.smithi107.stdout:902: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:34.801 INFO:tasks.rados.rados.0.smithi107.stdout:902: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:34.802 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 146 (ObjNum 901 snap 0 seq_num 901) dirty exists 2020-06-12T02:37:34.802 INFO:tasks.rados.rados.0.smithi107.stdout:902: left oid 11 (ObjNum 901 snap 0 seq_num 901) 2020-06-12T02:37:34.802 INFO:tasks.rados.rados.0.smithi107.stdout:902: done (2 left) 2020-06-12T02:37:34.803 INFO:tasks.rados.rados.0.smithi107.stdout:905: done (1 left) 2020-06-12T02:37:34.803 INFO:tasks.rados.rados.0.smithi107.stdout:906: delete oid 5 current snap is 0 2020-06-12T02:37:34.804 INFO:tasks.rados.rados.0.smithi107.stdout:903: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:34.804 INFO:tasks.rados.rados.0.smithi107.stdout:906: done (1 left) 2020-06-12T02:37:34.805 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:37:34.805 INFO:tasks.rados.rados.0.smithi107.stdout:907: seq_num 906 ranges {0=778240} 2020-06-12T02:37:34.807 INFO:tasks.rados.rados.0.smithi107.stdout:907: writing smithi10729407-19 from 0 to 778240 tid 1 2020-06-12T02:37:34.808 INFO:tasks.rados.rados.0.smithi107.stdout:908: delete oid 18 current snap is 0 2020-06-12T02:37:34.808 INFO:tasks.rados.rados.0.smithi107.stdout:903: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:34.808 INFO:tasks.rados.rados.0.smithi107.stdout:903: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:34.809 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 145 (ObjNum 902 snap 0 seq_num 902) dirty exists 2020-06-12T02:37:34.809 INFO:tasks.rados.rados.0.smithi107.stdout:903: left oid 12 (ObjNum 902 snap 0 seq_num 902) 2020-06-12T02:37:34.810 INFO:tasks.rados.rados.0.smithi107.stdout:903: done (2 left) 2020-06-12T02:37:34.811 INFO:tasks.rados.rados.0.smithi107.stdout:908: done (1 left) 2020-06-12T02:37:34.812 INFO:tasks.rados.rados.0.smithi107.stdout:909: delete oid 13 current snap is 0 2020-06-12T02:37:34.813 INFO:tasks.rados.rados.0.smithi107.stdout:909: done (1 left) 2020-06-12T02:37:34.814 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:37:34.814 INFO:tasks.rados.rados.0.smithi107.stdout:910: seq_num 909 ranges {0=425984,425984=401408} 2020-06-12T02:37:34.815 INFO:tasks.rados.rados.0.smithi107.stdout:910: writing smithi10729407-7 from 0 to 425984 tid 1 2020-06-12T02:37:34.817 INFO:tasks.rados.rados.0.smithi107.stdout:910: writing smithi10729407-7 from 425984 to 827392 tid 2 2020-06-12T02:37:34.817 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:37:34.817 INFO:tasks.rados.rados.0.smithi107.stdout:911: seq_num 910 ranges {581632=442368,1024000=401408} 2020-06-12T02:37:34.820 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:34.821 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:34.822 INFO:tasks.rados.rados.0.smithi107.stdout:911: writing smithi10729407-6 from 581632 to 1024000 tid 1 2020-06-12T02:37:34.825 INFO:tasks.rados.rados.0.smithi107.stdout:911: writing smithi10729407-6 from 1024000 to 1425408 tid 2 2020-06-12T02:37:34.826 INFO:tasks.rados.rados.0.smithi107.stdout:912: delete oid 13 current snap is 0 2020-06-12T02:37:34.826 INFO:tasks.rados.rados.0.smithi107.stdout:907: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:34.826 INFO:tasks.rados.rados.0.smithi107.stdout:907: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:34.826 INFO:tasks.rados.rados.0.smithi107.stdout:907: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:34.827 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 147 (ObjNum 906 snap 0 seq_num 906) dirty exists 2020-06-12T02:37:34.827 INFO:tasks.rados.rados.0.smithi107.stdout:907: left oid 19 (ObjNum 906 snap 0 seq_num 906) 2020-06-12T02:37:34.827 INFO:tasks.rados.rados.0.smithi107.stdout:910: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:34.827 INFO:tasks.rados.rados.0.smithi107.stdout:910: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:34.827 INFO:tasks.rados.rados.0.smithi107.stdout:910: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:34.827 INFO:tasks.rados.rados.0.smithi107.stdout:910: finishing write tid 5 to smithi10729407-7 2020-06-12T02:37:34.828 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 179 (ObjNum 909 snap 0 seq_num 909) dirty exists 2020-06-12T02:37:34.828 INFO:tasks.rados.rados.0.smithi107.stdout:910: left oid 7 (ObjNum 909 snap 0 seq_num 909) 2020-06-12T02:37:34.828 INFO:tasks.rados.rados.0.smithi107.stdout:907: done (3 left) 2020-06-12T02:37:34.828 INFO:tasks.rados.rados.0.smithi107.stdout:910: done (2 left) 2020-06-12T02:37:34.829 INFO:tasks.rados.rados.0.smithi107.stdout:912: done (1 left) 2020-06-12T02:37:34.829 INFO:tasks.rados.rados.0.smithi107.stdout:913: delete oid 19 current snap is 0 2020-06-12T02:37:34.829 INFO:tasks.rados.rados.0.smithi107.stdout:913: done (1 left) 2020-06-12T02:37:34.830 INFO:tasks.rados.rados.0.smithi107.stdout:914: delete oid 14 current snap is 0 2020-06-12T02:37:34.831 INFO:tasks.rados.rados.0.smithi107.stdout:911: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:34.831 INFO:tasks.rados.rados.0.smithi107.stdout:911: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:34.831 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-06-12T02:37:34.832 INFO:tasks.rados.rados.0.smithi107.stdout:911: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:34.833 INFO:tasks.rados.rados.0.smithi107.stdout:914: done (1 left) 2020-06-12T02:37:34.833 INFO:tasks.rados.rados.0.smithi107.stdout:915: delete oid 10 current snap is 0 2020-06-12T02:37:34.833 INFO:tasks.rados.rados.0.smithi107.stdout:911: finishing write tid 5 to smithi10729407-6 2020-06-12T02:37:34.834 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 193 (ObjNum 910 snap 0 seq_num 910) dirty exists 2020-06-12T02:37:34.834 INFO:tasks.rados.rados.0.smithi107.stdout:911: left oid 6 (ObjNum 910 snap 0 seq_num 910) 2020-06-12T02:37:34.835 INFO:tasks.rados.rados.0.smithi107.stdout:911: done (1 left) 2020-06-12T02:37:34.835 INFO:tasks.rados.rados.0.smithi107.stdout:915: done (0 left) 2020-06-12T02:37:34.835 INFO:tasks.rados.rados.0.smithi107.stdout:916: delete oid 2 current snap is 0 2020-06-12T02:37:34.837 INFO:tasks.rados.rados.0.smithi107.stdout:916: done (0 left) 2020-06-12T02:37:34.838 INFO:tasks.rados.rados.0.smithi107.stdout:917: delete oid 18 current snap is 0 2020-06-12T02:37:34.844 INFO:tasks.rados.rados.0.smithi107.stdout:917: done (0 left) 2020-06-12T02:37:34.844 INFO:tasks.rados.rados.0.smithi107.stdout:918: delete oid 17 current snap is 0 2020-06-12T02:37:34.845 INFO:tasks.rados.rados.0.smithi107.stdout:918: done (0 left) 2020-06-12T02:37:34.846 INFO:tasks.rados.rados.0.smithi107.stdout:919: delete oid 5 current snap is 0 2020-06-12T02:37:34.846 INFO:tasks.rados.rados.0.smithi107.stdout:919: done (0 left) 2020-06-12T02:37:34.846 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:37:34.846 INFO:tasks.rados.rados.0.smithi107.stdout:920: seq_num 919 ranges {294912=598016,892928=401408} 2020-06-12T02:37:34.850 INFO:tasks.rados.rados.0.smithi107.stdout:920: writing smithi10729407-4 from 294912 to 892928 tid 1 2020-06-12T02:37:34.852 INFO:tasks.rados.rados.0.smithi107.stdout:920: writing smithi10729407-4 from 892928 to 1294336 tid 2 2020-06-12T02:37:34.853 INFO:tasks.rados.rados.0.smithi107.stdout:921: delete oid 3 current snap is 0 2020-06-12T02:37:34.854 INFO:tasks.rados.rados.0.smithi107.stdout:921: done (1 left) 2020-06-12T02:37:34.854 INFO:tasks.rados.rados.0.smithi107.stdout:922: delete oid 1 current snap is 0 2020-06-12T02:37:34.857 INFO:tasks.rados.rados.0.smithi107.stdout:920: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:34.857 INFO:tasks.rados.rados.0.smithi107.stdout:922: done (1 left) 2020-06-12T02:37:34.857 INFO:tasks.rados.rados.0.smithi107.stdout:923: delete oid 6 current snap is 0 2020-06-12T02:37:34.859 INFO:tasks.rados.rados.0.smithi107.stdout:923: done (1 left) 2020-06-12T02:37:34.859 INFO:tasks.rados.rados.0.smithi107.stdout:924: delete oid 14 current snap is 0 2020-06-12T02:37:34.861 INFO:tasks.rados.rados.0.smithi107.stdout:920: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:34.861 INFO:tasks.rados.rados.0.smithi107.stdout:924: done (1 left) 2020-06-12T02:37:34.861 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:37:34.861 INFO:tasks.rados.rados.0.smithi107.stdout:925: seq_num 924 ranges {0=679936,679936=401408} 2020-06-12T02:37:34.864 INFO:tasks.rados.rados.0.smithi107.stdout:925: writing smithi10729407-9 from 0 to 679936 tid 1 2020-06-12T02:37:34.868 INFO:tasks.rados.rados.0.smithi107.stdout:925: writing smithi10729407-9 from 679936 to 1081344 tid 2 2020-06-12T02:37:34.868 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:37:34.868 INFO:tasks.rados.rados.0.smithi107.stdout:920: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:34.868 INFO:tasks.rados.rados.0.smithi107.stdout:926: seq_num 925 ranges {0=696320,696320=401408} 2020-06-12T02:37:34.871 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:34.868+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:37:34.872 INFO:tasks.rados.rados.0.smithi107.stdout:926: writing smithi10729407-14 from 0 to 696320 tid 1 2020-06-12T02:37:34.875 INFO:tasks.rados.rados.0.smithi107.stdout:926: writing smithi10729407-14 from 696320 to 1097728 tid 2 2020-06-12T02:37:34.875 INFO:tasks.rados.rados.0.smithi107.stdout:927: delete oid 6 current snap is 0 2020-06-12T02:37:34.876 INFO:tasks.rados.rados.0.smithi107.stdout:920: finishing write tid 5 to smithi10729407-4 2020-06-12T02:37:34.876 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 163 (ObjNum 919 snap 0 seq_num 919) dirty exists 2020-06-12T02:37:34.876 INFO:tasks.rados.rados.0.smithi107.stdout:920: left oid 4 (ObjNum 919 snap 0 seq_num 919) 2020-06-12T02:37:34.876 INFO:tasks.rados.rados.0.smithi107.stdout:925: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:34.876 INFO:tasks.rados.rados.0.smithi107.stdout:925: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:34.877 INFO:tasks.rados.rados.0.smithi107.stdout:925: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:34.877 INFO:tasks.rados.rados.0.smithi107.stdout:925: finishing write tid 5 to smithi10729407-9 2020-06-12T02:37:34.877 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 185 (ObjNum 924 snap 0 seq_num 924) dirty exists 2020-06-12T02:37:34.877 INFO:tasks.rados.rados.0.smithi107.stdout:925: left oid 9 (ObjNum 924 snap 0 seq_num 924) 2020-06-12T02:37:34.880 INFO:tasks.rados.rados.0.smithi107.stdout:926: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:34.881 INFO:tasks.rados.rados.0.smithi107.stdout:926: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:34.883 INFO:tasks.rados.rados.0.smithi107.stdout:926: finishing write tid 4 to smithi10729407-14 2020-06-12T02:37:34.884 INFO:tasks.rados.rados.0.smithi107.stdout:920: done (3 left) 2020-06-12T02:37:34.884 INFO:tasks.rados.rados.0.smithi107.stdout:925: done (2 left) 2020-06-12T02:37:34.884 INFO:tasks.rados.rados.0.smithi107.stdout:927: done (1 left) 2020-06-12T02:37:34.884 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:37:34.885 INFO:tasks.rados.rados.0.smithi107.stdout:928: seq_num 927 ranges {1081344=729088,1810432=401408} 2020-06-12T02:37:34.893 INFO:tasks.rados.rados.0.smithi107.stdout:928: writing smithi10729407-9 from 1081344 to 1810432 tid 1 2020-06-12T02:37:34.896 INFO:tasks.rados.rados.0.smithi107.stdout:928: writing smithi10729407-9 from 1810432 to 2211840 tid 2 2020-06-12T02:37:34.896 INFO:tasks.rados.rados.0.smithi107.stdout:929: delete oid 7 current snap is 0 2020-06-12T02:37:34.896 INFO:tasks.rados.rados.0.smithi107.stdout:926: finishing write tid 5 to smithi10729407-14 2020-06-12T02:37:34.897 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 151 (ObjNum 925 snap 0 seq_num 925) dirty exists 2020-06-12T02:37:34.897 INFO:tasks.rados.rados.0.smithi107.stdout:926: left oid 14 (ObjNum 925 snap 0 seq_num 925) 2020-06-12T02:37:34.898 INFO:tasks.rados.rados.0.smithi107.stdout:926: done (2 left) 2020-06-12T02:37:34.898 INFO:tasks.rados.rados.0.smithi107.stdout:929: done (1 left) 2020-06-12T02:37:34.899 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:37:34.899 INFO:tasks.rados.rados.0.smithi107.stdout:930: seq_num 929 ranges {0=761856,761856=401408} 2020-06-12T02:37:34.903 INFO:tasks.rados.rados.0.smithi107.stdout:930: writing smithi10729407-13 from 0 to 761856 tid 1 2020-06-12T02:37:34.906 INFO:tasks.rados.rados.0.smithi107.stdout:930: writing smithi10729407-13 from 761856 to 1163264 tid 2 2020-06-12T02:37:34.906 INFO:tasks.rados.rados.0.smithi107.stdout:931: delete oid 15 current snap is 0 2020-06-12T02:37:34.907 INFO:tasks.rados.rados.0.smithi107.stdout:928: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:34.907 INFO:tasks.rados.rados.0.smithi107.stdout:928: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:34.907 INFO:tasks.rados.rados.0.smithi107.stdout:928: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:34.907 INFO:tasks.rados.rados.0.smithi107.stdout:928: finishing write tid 5 to smithi10729407-9 2020-06-12T02:37:34.908 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 188 (ObjNum 927 snap 0 seq_num 927) dirty exists 2020-06-12T02:37:34.908 INFO:tasks.rados.rados.0.smithi107.stdout:928: left oid 9 (ObjNum 927 snap 0 seq_num 927) 2020-06-12T02:37:34.908 INFO:tasks.rados.rados.0.smithi107.stdout:928: done (2 left) 2020-06-12T02:37:34.908 INFO:tasks.rados.rados.0.smithi107.stdout:931: done (1 left) 2020-06-12T02:37:34.909 INFO:tasks.rados.rados.0.smithi107.stdout:932: delete oid 5 current snap is 0 2020-06-12T02:37:34.909 INFO:tasks.rados.rados.0.smithi107.stdout:932: done (1 left) 2020-06-12T02:37:34.909 INFO:tasks.rados.rados.0.smithi107.stdout:933: delete oid 11 current snap is 0 2020-06-12T02:37:34.910 INFO:tasks.rados.rados.0.smithi107.stdout:930: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:34.914 INFO:tasks.rados.rados.0.smithi107.stdout:933: done (1 left) 2020-06-12T02:37:34.914 INFO:tasks.rados.rados.0.smithi107.stdout:934: delete oid 17 current snap is 0 2020-06-12T02:37:34.916 INFO:tasks.rados.rados.0.smithi107.stdout:930: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:34.919 INFO:tasks.rados.rados.0.smithi107.stdout:934: done (1 left) 2020-06-12T02:37:34.919 INFO:tasks.rados.rados.0.smithi107.stdout:935: delete oid 18 current snap is 0 2020-06-12T02:37:34.921 INFO:tasks.rados.rados.0.smithi107.stdout:930: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:34.922 INFO:tasks.rados.rados.0.smithi107.stdout:935: done (1 left) 2020-06-12T02:37:34.922 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:37:34.922 INFO:tasks.rados.rados.0.smithi107.stdout:936: seq_num 935 ranges {0=466944,466944=720896,1187840=81920} 2020-06-12T02:37:34.924 INFO:tasks.rados.rados.0.smithi107.stdout:936: writing smithi10729407-11 from 0 to 466944 tid 1 2020-06-12T02:37:34.930 INFO:tasks.rados.rados.0.smithi107.stdout:936: writing smithi10729407-11 from 466944 to 1187840 tid 2 2020-06-12T02:37:34.930 INFO:tasks.rados.rados.0.smithi107.stdout:936: writing smithi10729407-11 from 1187840 to 1269760 tid 3 2020-06-12T02:37:34.930 INFO:tasks.rados.rados.0.smithi107.stdout:937: delete oid 16 current snap is 0 2020-06-12T02:37:34.931 INFO:tasks.rados.rados.0.smithi107.stdout:930: finishing write tid 5 to smithi10729407-13 2020-06-12T02:37:34.931 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 184 (ObjNum 929 snap 0 seq_num 929) dirty exists 2020-06-12T02:37:34.931 INFO:tasks.rados.rados.0.smithi107.stdout:930: left oid 13 (ObjNum 929 snap 0 seq_num 929) 2020-06-12T02:37:34.931 INFO:tasks.rados.rados.0.smithi107.stdout:936: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:34.934 INFO:tasks.rados.rados.0.smithi107.stdout:930: done (2 left) 2020-06-12T02:37:34.934 INFO:tasks.rados.rados.0.smithi107.stdout:937: done (1 left) 2020-06-12T02:37:34.934 INFO:tasks.rados.rados.0.smithi107.stdout:938: delete oid 10 current snap is 0 2020-06-12T02:37:34.935 INFO:tasks.rados.rados.0.smithi107.stdout:936: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:34.936 INFO:tasks.rados.rados.0.smithi107.stdout:936: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:34.938 INFO:tasks.rados.rados.0.smithi107.stdout:936: finishing write tid 5 to smithi10729407-11 2020-06-12T02:37:34.939 INFO:tasks.rados.rados.0.smithi107.stdout:936: finishing write tid 6 to smithi10729407-11 2020-06-12T02:37:34.940 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 156 (ObjNum 935 snap 0 seq_num 935) dirty exists 2020-06-12T02:37:34.940 INFO:tasks.rados.rados.0.smithi107.stdout:936: left oid 11 (ObjNum 935 snap 0 seq_num 935) 2020-06-12T02:37:34.940 INFO:tasks.rados.rados.0.smithi107.stdout:936: done (1 left) 2020-06-12T02:37:34.940 INFO:tasks.rados.rados.0.smithi107.stdout:938: done (0 left) 2020-06-12T02:37:34.940 INFO:tasks.rados.rados.0.smithi107.stdout:939: delete oid 13 current snap is 0 2020-06-12T02:37:34.941 INFO:tasks.rados.rados.0.smithi107.stdout:939: done (0 left) 2020-06-12T02:37:34.941 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:37:34.941 INFO:tasks.rados.rados.0.smithi107.stdout:940: seq_num 939 ranges {0=532480,532480=704512,1236992=98304} 2020-06-12T02:37:34.944 INFO:tasks.rados.rados.0.smithi107.stdout:940: writing smithi10729407-19 from 0 to 532480 tid 1 2020-06-12T02:37:34.948 INFO:tasks.rados.rados.0.smithi107.stdout:940: writing smithi10729407-19 from 532480 to 1236992 tid 2 2020-06-12T02:37:34.949 INFO:tasks.rados.rados.0.smithi107.stdout:940: writing smithi10729407-19 from 1236992 to 1335296 tid 3 2020-06-12T02:37:34.949 INFO:tasks.rados.rados.0.smithi107.stdout:941: delete oid 3 current snap is 0 2020-06-12T02:37:34.950 INFO:tasks.rados.rados.0.smithi107.stdout:941: done (1 left) 2020-06-12T02:37:34.951 INFO:tasks.rados.rados.0.smithi107.stdout:942: delete oid 2 current snap is 0 2020-06-12T02:37:34.953 INFO:tasks.rados.rados.0.smithi107.stdout:942: done (1 left) 2020-06-12T02:37:34.953 INFO:tasks.rados.rados.0.smithi107.stdout:943: delete oid 14 current snap is 0 2020-06-12T02:37:34.954 INFO:tasks.rados.rados.0.smithi107.stdout:940: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:34.954 INFO:tasks.rados.rados.0.smithi107.stdout:943: done (1 left) 2020-06-12T02:37:34.955 INFO:tasks.rados.rados.0.smithi107.stdout:944: delete oid 9 current snap is 0 2020-06-12T02:37:34.955 INFO:tasks.rados.rados.0.smithi107.stdout:940: finishing write tid 2 to smithi10729407-19 2020-06-12T02:37:34.956 INFO:tasks.rados.rados.0.smithi107.stdout:940: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:34.957 INFO:tasks.rados.rados.0.smithi107.stdout:940: finishing write tid 5 to smithi10729407-19 2020-06-12T02:37:34.958 INFO:tasks.rados.rados.0.smithi107.stdout:940: finishing write tid 6 to smithi10729407-19 2020-06-12T02:37:34.958 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 152 (ObjNum 939 snap 0 seq_num 939) dirty exists 2020-06-12T02:37:34.959 INFO:tasks.rados.rados.0.smithi107.stdout:940: left oid 19 (ObjNum 939 snap 0 seq_num 939) 2020-06-12T02:37:34.959 INFO:tasks.rados.rados.0.smithi107.stdout:940: done (1 left) 2020-06-12T02:37:34.959 INFO:tasks.rados.rados.0.smithi107.stdout:944: done (0 left) 2020-06-12T02:37:34.959 INFO:tasks.rados.rados.0.smithi107.stdout:945: delete oid 9 current snap is 0 2020-06-12T02:37:34.960 INFO:tasks.rados.rados.0.smithi107.stdout:945: done (0 left) 2020-06-12T02:37:34.960 INFO:tasks.rados.rados.0.smithi107.stdout:946: delete oid 2 current snap is 0 2020-06-12T02:37:34.961 INFO:tasks.rados.rados.0.smithi107.stdout:946: done (0 left) 2020-06-12T02:37:34.961 INFO:tasks.rados.rados.0.smithi107.stdout:947: delete oid 6 current snap is 0 2020-06-12T02:37:34.962 INFO:tasks.rados.rados.0.smithi107.stdout:947: done (0 left) 2020-06-12T02:37:34.962 INFO:tasks.rados.rados.0.smithi107.stdout:948: delete oid 11 current snap is 0 2020-06-12T02:37:34.964 INFO:tasks.rados.rados.0.smithi107.stdout:948: done (0 left) 2020-06-12T02:37:34.964 INFO:tasks.rados.rados.0.smithi107.stdout:949: delete oid 13 current snap is 0 2020-06-12T02:37:34.966 INFO:tasks.rados.rados.0.smithi107.stdout:949: done (0 left) 2020-06-12T02:37:34.967 INFO:tasks.rados.rados.0.smithi107.stdout:950: delete oid 17 current snap is 0 2020-06-12T02:37:34.968 INFO:tasks.rados.rados.0.smithi107.stdout:950: done (0 left) 2020-06-12T02:37:34.968 INFO:tasks.rados.rados.0.smithi107.stdout:951: delete oid 17 current snap is 0 2020-06-12T02:37:34.969 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:37:34.965+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:37:34.970 INFO:tasks.rados.rados.0.smithi107.stdout:951: done (0 left) 2020-06-12T02:37:34.970 INFO:tasks.rados.rados.0.smithi107.stdout:952: delete oid 4 current snap is 0 2020-06-12T02:37:34.971 INFO:tasks.rados.rados.0.smithi107.stdout:952: done (0 left) 2020-06-12T02:37:34.972 INFO:tasks.rados.rados.0.smithi107.stdout:953: delete oid 3 current snap is 0 2020-06-12T02:37:34.973 INFO:tasks.rados.rados.0.smithi107.stdout:953: done (0 left) 2020-06-12T02:37:34.973 INFO:tasks.rados.rados.0.smithi107.stdout:954: delete oid 4 current snap is 0 2020-06-12T02:37:34.975 INFO:tasks.rados.rados.0.smithi107.stdout:954: done (0 left) 2020-06-12T02:37:34.975 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:37:34.975 INFO:tasks.rados.rados.0.smithi107.stdout:955: seq_num 954 ranges {0=786432,786432=671744,1458176=131072} 2020-06-12T02:37:34.979 INFO:tasks.rados.rados.0.smithi107.stdout:955: writing smithi10729407-13 from 0 to 786432 tid 1 2020-06-12T02:37:34.988 INFO:tasks.rados.rados.0.smithi107.stdout:955: writing smithi10729407-13 from 786432 to 1458176 tid 2 2020-06-12T02:37:34.988 INFO:tasks.rados.rados.0.smithi107.stdout:955: writing smithi10729407-13 from 1458176 to 1589248 tid 3 2020-06-12T02:37:34.989 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:37:34.991 INFO:tasks.rados.rados.0.smithi107.stdout:956: seq_num 955 ranges {0=802816,802816=557056,1359872=245760} 2020-06-12T02:37:34.991 INFO:tasks.rados.rados.0.smithi107.stdout:956: writing smithi10729407-1 from 0 to 802816 tid 1 2020-06-12T02:37:34.993 INFO:tasks.rados.rados.0.smithi107.stdout:956: writing smithi10729407-1 from 802816 to 1359872 tid 2 2020-06-12T02:37:34.994 INFO:tasks.rados.rados.0.smithi107.stdout:956: writing smithi10729407-1 from 1359872 to 1605632 tid 3 2020-06-12T02:37:34.994 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:34.995 INFO:tasks.rados.rados.0.smithi107.stdout:957: seq_num 956 ranges {0=417792,417792=434176,851968=770048} 2020-06-12T02:37:34.996 INFO:tasks.rados.rados.0.smithi107.stdout:957: writing smithi10729407-2 from 0 to 417792 tid 1 2020-06-12T02:37:34.999 INFO:tasks.rados.rados.0.smithi107.stdout:957: writing smithi10729407-2 from 417792 to 851968 tid 2 2020-06-12T02:37:35.004 INFO:tasks.rados.rados.0.smithi107.stdout:957: writing smithi10729407-2 from 851968 to 1622016 tid 3 2020-06-12T02:37:35.004 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:37:35.004 INFO:tasks.rados.rados.0.smithi107.stdout:958: seq_num 957 ranges {0=434176,434176=720896,1155072=483328} 2020-06-12T02:37:35.007 INFO:tasks.rados.rados.0.smithi107.stdout:958: writing smithi10729407-14 from 0 to 434176 tid 1 2020-06-12T02:37:35.008 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:35.008 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:35.012 INFO:tasks.rados.rados.0.smithi107.stdout:958: writing smithi10729407-14 from 434176 to 1155072 tid 2 2020-06-12T02:37:35.015 INFO:tasks.rados.rados.0.smithi107.stdout:958: writing smithi10729407-14 from 1155072 to 1638400 tid 3 2020-06-12T02:37:35.016 INFO:tasks.rados.rados.0.smithi107.stdout:959: delete oid 3 current snap is 0 2020-06-12T02:37:35.016 INFO:tasks.rados.rados.0.smithi107.stdout:955: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:35.016 INFO:tasks.rados.rados.0.smithi107.stdout:955: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:35.016 INFO:tasks.rados.rados.0.smithi107.stdout:955: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:35.016 INFO:tasks.rados.rados.0.smithi107.stdout:955: finishing write tid 5 to smithi10729407-13 2020-06-12T02:37:35.017 INFO:tasks.rados.rados.0.smithi107.stdout:955: finishing write tid 6 to smithi10729407-13 2020-06-12T02:37:35.017 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 190 (ObjNum 954 snap 0 seq_num 954) dirty exists 2020-06-12T02:37:35.017 INFO:tasks.rados.rados.0.smithi107.stdout:955: left oid 13 (ObjNum 954 snap 0 seq_num 954) 2020-06-12T02:37:35.017 INFO:tasks.rados.rados.0.smithi107.stdout:956: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:35.017 INFO:tasks.rados.rados.0.smithi107.stdout:956: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:35.018 INFO:tasks.rados.rados.0.smithi107.stdout:956: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:35.018 INFO:tasks.rados.rados.0.smithi107.stdout:957: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:35.018 INFO:tasks.rados.rados.0.smithi107.stdout:956: finishing write tid 5 to smithi10729407-1 2020-06-12T02:37:35.018 INFO:tasks.rados.rados.0.smithi107.stdout:956: finishing write tid 6 to smithi10729407-1 2020-06-12T02:37:35.018 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 201 (ObjNum 955 snap 0 seq_num 955) dirty exists 2020-06-12T02:37:35.019 INFO:tasks.rados.rados.0.smithi107.stdout:956: left oid 1 (ObjNum 955 snap 0 seq_num 955) 2020-06-12T02:37:35.019 INFO:tasks.rados.rados.0.smithi107.stdout:957: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:35.019 INFO:tasks.rados.rados.0.smithi107.stdout:957: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:35.019 INFO:tasks.rados.rados.0.smithi107.stdout:957: finishing write tid 5 to smithi10729407-2 2020-06-12T02:37:35.019 INFO:tasks.rados.rados.0.smithi107.stdout:958: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:35.020 INFO:tasks.rados.rados.0.smithi107.stdout:957: finishing write tid 6 to smithi10729407-2 2020-06-12T02:37:35.020 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 98 (ObjNum 956 snap 0 seq_num 956) dirty exists 2020-06-12T02:37:35.020 INFO:tasks.rados.rados.0.smithi107.stdout:957: left oid 2 (ObjNum 956 snap 0 seq_num 956) 2020-06-12T02:37:35.022 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-06-12T02:37:35.022 INFO:tasks.rados.rados.0.smithi107.stdout:955: done (4 left) 2020-06-12T02:37:35.023 INFO:tasks.rados.rados.0.smithi107.stdout:956: done (3 left) 2020-06-12T02:37:35.023 INFO:tasks.rados.rados.0.smithi107.stdout:957: done (2 left) 2020-06-12T02:37:35.023 INFO:tasks.rados.rados.0.smithi107.stdout:959: done (1 left) 2020-06-12T02:37:35.023 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:37:35.023 INFO:tasks.rados.rados.0.smithi107.stdout:960: seq_num 959 ranges {0=466944,466944=540672,1007616=557056,1564672=106496} 2020-06-12T02:37:35.024 INFO:tasks.rados.rados.0.smithi107.stdout:960: writing smithi10729407-6 from 0 to 466944 tid 1 2020-06-12T02:37:35.024 INFO:tasks.rados.rados.0.smithi107.stdout:960: writing smithi10729407-6 from 466944 to 1007616 tid 2 2020-06-12T02:37:35.027 INFO:tasks.rados.rados.0.smithi107.stdout:960: writing smithi10729407-6 from 1007616 to 1564672 tid 3 2020-06-12T02:37:35.028 INFO:tasks.rados.rados.0.smithi107.stdout:960: writing smithi10729407-6 from 1564672 to 1671168 tid 4 2020-06-12T02:37:35.028 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:37:35.029 INFO:tasks.rados.rados.0.smithi107.stdout:961: seq_num 960 ranges {0=483328,483328=425984,909312=761856,1671168=16384} 2020-06-12T02:37:35.030 INFO:tasks.rados.rados.0.smithi107.stdout:961: writing smithi10729407-9 from 0 to 483328 tid 1 2020-06-12T02:37:35.033 INFO:tasks.rados.rados.0.smithi107.stdout:961: writing smithi10729407-9 from 483328 to 909312 tid 2 2020-06-12T02:37:35.039 INFO:tasks.rados.rados.0.smithi107.stdout:961: writing smithi10729407-9 from 909312 to 1671168 tid 3 2020-06-12T02:37:35.039 INFO:tasks.rados.rados.0.smithi107.stdout:961: writing smithi10729407-9 from 1671168 to 1687552 tid 4 2020-06-12T02:37:35.039 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:37:35.040 INFO:tasks.rados.rados.0.smithi107.stdout:962: seq_num 961 ranges {0=499712,499712=712704,1212416=491520} 2020-06-12T02:37:35.042 INFO:tasks.rados.rados.0.smithi107.stdout:962: writing smithi10729407-7 from 0 to 499712 tid 1 2020-06-12T02:37:35.046 INFO:tasks.rados.rados.0.smithi107.stdout:962: writing smithi10729407-7 from 499712 to 1212416 tid 2 2020-06-12T02:37:35.051 INFO:tasks.rados.rados.0.smithi107.stdout:962: writing smithi10729407-7 from 1212416 to 1703936 tid 3 2020-06-12T02:37:35.052 INFO:tasks.rados.rados.0.smithi107.stdout:963: delete oid 18 current snap is 0 2020-06-12T02:37:35.052 INFO:tasks.rados.rados.0.smithi107.stdout:958: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:35.052 INFO:tasks.rados.rados.0.smithi107.stdout:958: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:35.052 INFO:tasks.rados.rados.0.smithi107.stdout:958: finishing write tid 5 to smithi10729407-14 2020-06-12T02:37:35.053 INFO:tasks.rados.rados.0.smithi107.stdout:958: finishing write tid 6 to smithi10729407-14 2020-06-12T02:37:35.053 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 162 (ObjNum 957 snap 0 seq_num 957) dirty exists 2020-06-12T02:37:35.053 INFO:tasks.rados.rados.0.smithi107.stdout:958: left oid 14 (ObjNum 957 snap 0 seq_num 957) 2020-06-12T02:37:35.053 INFO:tasks.rados.rados.0.smithi107.stdout:960: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:35.053 INFO:tasks.rados.rados.0.smithi107.stdout:960: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:35.054 INFO:tasks.rados.rados.0.smithi107.stdout:960: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:35.054 INFO:tasks.rados.rados.0.smithi107.stdout:960: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:35.054 INFO:tasks.rados.rados.0.smithi107.stdout:960: finishing write tid 6 to smithi10729407-6 2020-06-12T02:37:35.054 INFO:tasks.rados.rados.0.smithi107.stdout:961: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:35.054 INFO:tasks.rados.rados.0.smithi107.stdout:960: finishing write tid 7 to smithi10729407-6 2020-06-12T02:37:35.055 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 206 (ObjNum 959 snap 0 seq_num 959) dirty exists 2020-06-12T02:37:35.055 INFO:tasks.rados.rados.0.smithi107.stdout:960: left oid 6 (ObjNum 959 snap 0 seq_num 959) 2020-06-12T02:37:35.055 INFO:tasks.rados.rados.0.smithi107.stdout:961: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:35.055 INFO:tasks.rados.rados.0.smithi107.stdout:961: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:35.055 INFO:tasks.rados.rados.0.smithi107.stdout:961: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:35.056 INFO:tasks.rados.rados.0.smithi107.stdout:962: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:35.056 INFO:tasks.rados.rados.0.smithi107.stdout:961: finishing write tid 6 to smithi10729407-9 2020-06-12T02:37:35.057 INFO:tasks.rados.rados.0.smithi107.stdout:961: finishing write tid 7 to smithi10729407-9 2020-06-12T02:37:35.057 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 196 (ObjNum 960 snap 0 seq_num 960) dirty exists 2020-06-12T02:37:35.057 INFO:tasks.rados.rados.0.smithi107.stdout:961: left oid 9 (ObjNum 960 snap 0 seq_num 960) 2020-06-12T02:37:35.057 INFO:tasks.rados.rados.0.smithi107.stdout:958: done (4 left) 2020-06-12T02:37:35.058 INFO:tasks.rados.rados.0.smithi107.stdout:960: done (3 left) 2020-06-12T02:37:35.059 INFO:tasks.rados.rados.0.smithi107.stdout:961: done (2 left) 2020-06-12T02:37:35.060 INFO:tasks.rados.rados.0.smithi107.stdout:963: done (1 left) 2020-06-12T02:37:35.060 INFO:tasks.rados.rados.0.smithi107.stdout:964: delete oid 15 current snap is 0 2020-06-12T02:37:35.060 INFO:tasks.rados.rados.0.smithi107.stdout:962: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:35.060 INFO:tasks.rados.rados.0.smithi107.stdout:964: done (1 left) 2020-06-12T02:37:35.061 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:37:35.061 INFO:tasks.rados.rados.0.smithi107.stdout:965: seq_num 964 ranges {0=548864,548864=761856,1310720=442368} 2020-06-12T02:37:35.062 INFO:tasks.rados.rados.0.smithi107.stdout:965: writing smithi10729407-17 from 0 to 548864 tid 1 2020-06-12T02:37:35.064 INFO:tasks.rados.rados.0.smithi107.stdout:965: writing smithi10729407-17 from 548864 to 1310720 tid 2 2020-06-12T02:37:35.067 INFO:tasks.rados.rados.0.smithi107.stdout:965: writing smithi10729407-17 from 1310720 to 1753088 tid 3 2020-06-12T02:37:35.067 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:37:35.067 INFO:tasks.rados.rados.0.smithi107.stdout:966: seq_num 965 ranges {1687552=565248,2252800=647168,2899968=548864,3448832=8192} 2020-06-12T02:37:35.070 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:35.067+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:35.079 INFO:tasks.rados.rados.0.smithi107.stdout:966: writing smithi10729407-9 from 1687552 to 2252800 tid 1 2020-06-12T02:37:35.083 INFO:tasks.rados.rados.0.smithi107.stdout:966: writing smithi10729407-9 from 2252800 to 2899968 tid 2 2020-06-12T02:37:35.087 INFO:tasks.rados.rados.0.smithi107.stdout:966: writing smithi10729407-9 from 2899968 to 3448832 tid 3 2020-06-12T02:37:35.087 INFO:tasks.rados.rados.0.smithi107.stdout:966: writing smithi10729407-9 from 3448832 to 3457024 tid 4 2020-06-12T02:37:35.087 INFO:tasks.rados.rados.0.smithi107.stdout:967: delete oid 19 current snap is 0 2020-06-12T02:37:35.088 INFO:tasks.rados.rados.0.smithi107.stdout:962: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:35.088 INFO:tasks.rados.rados.0.smithi107.stdout:962: finishing write tid 5 to smithi10729407-7 2020-06-12T02:37:35.088 INFO:tasks.rados.rados.0.smithi107.stdout:962: finishing write tid 6 to smithi10729407-7 2020-06-12T02:37:35.088 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 194 (ObjNum 961 snap 0 seq_num 961) dirty exists 2020-06-12T02:37:35.089 INFO:tasks.rados.rados.0.smithi107.stdout:962: left oid 7 (ObjNum 961 snap 0 seq_num 961) 2020-06-12T02:37:35.089 INFO:tasks.rados.rados.0.smithi107.stdout:965: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:35.089 INFO:tasks.rados.rados.0.smithi107.stdout:965: finishing write tid 2 to smithi10729407-17 2020-06-12T02:37:35.089 INFO:tasks.rados.rados.0.smithi107.stdout:965: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:35.089 INFO:tasks.rados.rados.0.smithi107.stdout:965: finishing write tid 5 to smithi10729407-17 2020-06-12T02:37:35.090 INFO:tasks.rados.rados.0.smithi107.stdout:965: finishing write tid 6 to smithi10729407-17 2020-06-12T02:37:35.091 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 172 (ObjNum 964 snap 0 seq_num 964) dirty exists 2020-06-12T02:37:35.091 INFO:tasks.rados.rados.0.smithi107.stdout:965: left oid 17 (ObjNum 964 snap 0 seq_num 964) 2020-06-12T02:37:35.091 INFO:tasks.rados.rados.0.smithi107.stdout:966: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:35.094 INFO:tasks.rados.rados.0.smithi107.stdout:962: done (3 left) 2020-06-12T02:37:35.094 INFO:tasks.rados.rados.0.smithi107.stdout:965: done (2 left) 2020-06-12T02:37:35.094 INFO:tasks.rados.rados.0.smithi107.stdout:967: done (1 left) 2020-06-12T02:37:35.095 INFO:tasks.rados.rados.0.smithi107.stdout:968: delete oid 1 current snap is 0 2020-06-12T02:37:35.096 INFO:tasks.rados.rados.0.smithi107.stdout:968: done (1 left) 2020-06-12T02:37:35.096 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:37:35.096 INFO:tasks.rados.rados.0.smithi107.stdout:969: seq_num 968 ranges {0=614400,614400=745472,1359872=458752} 2020-06-12T02:37:35.096 INFO:tasks.rados.rados.0.smithi107.stdout:969: writing smithi10729407-3 from 0 to 614400 tid 1 2020-06-12T02:37:35.100 INFO:tasks.rados.rados.0.smithi107.stdout:969: writing smithi10729407-3 from 614400 to 1359872 tid 2 2020-06-12T02:37:35.103 INFO:tasks.rados.rados.0.smithi107.stdout:969: writing smithi10729407-3 from 1359872 to 1818624 tid 3 2020-06-12T02:37:35.104 INFO:tasks.rados.rados.0.smithi107.stdout:970: delete oid 17 current snap is 0 2020-06-12T02:37:35.104 INFO:tasks.rados.rados.0.smithi107.stdout:966: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:35.104 INFO:tasks.rados.rados.0.smithi107.stdout:966: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:35.104 INFO:tasks.rados.rados.0.smithi107.stdout:966: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:35.104 INFO:tasks.rados.rados.0.smithi107.stdout:966: finishing write tid 6 to smithi10729407-9 2020-06-12T02:37:35.105 INFO:tasks.rados.rados.0.smithi107.stdout:966: finishing write tid 7 to smithi10729407-9 2020-06-12T02:37:35.106 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 202 (ObjNum 965 snap 0 seq_num 965) dirty exists 2020-06-12T02:37:35.106 INFO:tasks.rados.rados.0.smithi107.stdout:966: left oid 9 (ObjNum 965 snap 0 seq_num 965) 2020-06-12T02:37:35.106 INFO:tasks.rados.rados.0.smithi107.stdout:969: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:35.107 INFO:tasks.rados.rados.0.smithi107.stdout:966: done (2 left) 2020-06-12T02:37:35.107 INFO:tasks.rados.rados.0.smithi107.stdout:970: done (1 left) 2020-06-12T02:37:35.108 INFO:tasks.rados.rados.0.smithi107.stdout:971: delete oid 18 current snap is 0 2020-06-12T02:37:35.117 INFO:tasks.rados.rados.0.smithi107.stdout:971: done (1 left) 2020-06-12T02:37:35.118 INFO:tasks.rados.rados.0.smithi107.stdout:972: delete oid 15 current snap is 0 2020-06-12T02:37:35.118 INFO:tasks.rados.rados.0.smithi107.stdout:969: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:35.118 INFO:tasks.rados.rados.0.smithi107.stdout:969: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:35.118 INFO:tasks.rados.rados.0.smithi107.stdout:969: finishing write tid 5 to smithi10729407-3 2020-06-12T02:37:35.119 INFO:tasks.rados.rados.0.smithi107.stdout:972: done (1 left) 2020-06-12T02:37:35.119 INFO:tasks.rados.rados.0.smithi107.stdout:973: delete oid 5 current snap is 0 2020-06-12T02:37:35.119 INFO:tasks.rados.rados.0.smithi107.stdout:973: done (1 left) 2020-06-12T02:37:35.119 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:37:35.119 INFO:tasks.rados.rados.0.smithi107.stdout:974: seq_num 973 ranges {0=704512,704512=573440,1277952=507904,1785856=122880} 2020-06-12T02:37:35.120 INFO:tasks.rados.rados.0.smithi107.stdout:974: writing smithi10729407-10 from 0 to 704512 tid 1 2020-06-12T02:37:35.120 INFO:tasks.rados.rados.0.smithi107.stdout:974: writing smithi10729407-10 from 704512 to 1277952 tid 2 2020-06-12T02:37:35.122 INFO:tasks.rados.rados.0.smithi107.stdout:974: writing smithi10729407-10 from 1277952 to 1785856 tid 3 2020-06-12T02:37:35.123 INFO:tasks.rados.rados.0.smithi107.stdout:974: writing smithi10729407-10 from 1785856 to 1908736 tid 4 2020-06-12T02:37:35.123 INFO:tasks.rados.rados.0.smithi107.stdout:975: delete oid 19 current snap is 0 2020-06-12T02:37:35.124 INFO:tasks.rados.rados.0.smithi107.stdout:969: finishing write tid 6 to smithi10729407-3 2020-06-12T02:37:35.124 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 174 (ObjNum 968 snap 0 seq_num 968) dirty exists 2020-06-12T02:37:35.124 INFO:tasks.rados.rados.0.smithi107.stdout:969: left oid 3 (ObjNum 968 snap 0 seq_num 968) 2020-06-12T02:37:35.125 INFO:tasks.rados.rados.0.smithi107.stdout:969: done (2 left) 2020-06-12T02:37:35.125 INFO:tasks.rados.rados.0.smithi107.stdout:975: done (1 left) 2020-06-12T02:37:35.125 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:37:35.126 INFO:tasks.rados.rados.0.smithi107.stdout:976: seq_num 975 ranges {0=737280,737280=786432,1523712=417792} 2020-06-12T02:37:35.130 INFO:tasks.rados.rados.0.smithi107.stdout:976: writing smithi10729407-11 from 0 to 737280 tid 1 2020-06-12T02:37:35.136 INFO:tasks.rados.rados.0.smithi107.stdout:976: writing smithi10729407-11 from 737280 to 1523712 tid 2 2020-06-12T02:37:35.137 INFO:tasks.rados.rados.0.smithi107.stdout:976: writing smithi10729407-11 from 1523712 to 1941504 tid 3 2020-06-12T02:37:35.137 INFO:tasks.rados.rados.0.smithi107.stdout:977: delete oid 12 current snap is 0 2020-06-12T02:37:35.137 INFO:tasks.rados.rados.0.smithi107.stdout:974: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:35.138 INFO:tasks.rados.rados.0.smithi107.stdout:974: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:35.138 INFO:tasks.rados.rados.0.smithi107.stdout:974: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:35.138 INFO:tasks.rados.rados.0.smithi107.stdout:974: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:35.138 INFO:tasks.rados.rados.0.smithi107.stdout:974: finishing write tid 6 to smithi10729407-10 2020-06-12T02:37:35.138 INFO:tasks.rados.rados.0.smithi107.stdout:974: finishing write tid 7 to smithi10729407-10 2020-06-12T02:37:35.139 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 90 (ObjNum 973 snap 0 seq_num 973) dirty exists 2020-06-12T02:37:35.139 INFO:tasks.rados.rados.0.smithi107.stdout:974: left oid 10 (ObjNum 973 snap 0 seq_num 973) 2020-06-12T02:37:35.139 INFO:tasks.rados.rados.0.smithi107.stdout:974: done (2 left) 2020-06-12T02:37:35.139 INFO:tasks.rados.rados.0.smithi107.stdout:977: done (1 left) 2020-06-12T02:37:35.140 INFO:tasks.rados.rados.0.smithi107.stdout:978: delete oid 17 current snap is 0 2020-06-12T02:37:35.140 INFO:tasks.rados.rados.0.smithi107.stdout:976: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:35.148 INFO:tasks.rados.rados.0.smithi107.stdout:978: done (1 left) 2020-06-12T02:37:35.148 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:37:35.149 INFO:tasks.rados.rados.0.smithi107.stdout:979: seq_num 978 ranges {1671168=786432,2457600=442368,2899968=647168,3547136=114688} 2020-06-12T02:37:35.157 INFO:tasks.rados.rados.0.smithi107.stdout:979: writing smithi10729407-6 from 1671168 to 2457600 tid 1 2020-06-12T02:37:35.158 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:35.158 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:35.159 INFO:tasks.rados.rados.0.smithi107.stdout:979: writing smithi10729407-6 from 2457600 to 2899968 tid 2 2020-06-12T02:37:35.162 INFO:tasks.rados.rados.0.smithi107.stdout:979: writing smithi10729407-6 from 2899968 to 3547136 tid 3 2020-06-12T02:37:35.163 INFO:tasks.rados.rados.0.smithi107.stdout:979: writing smithi10729407-6 from 3547136 to 3661824 tid 4 2020-06-12T02:37:35.164 INFO:tasks.rados.rados.0.smithi107.stdout:980: delete oid 1 current snap is 0 2020-06-12T02:37:35.164 INFO:tasks.rados.rados.0.smithi107.stdout:976: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:35.164 INFO:tasks.rados.rados.0.smithi107.stdout:976: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:35.164 INFO:tasks.rados.rados.0.smithi107.stdout:976: finishing write tid 5 to smithi10729407-11 2020-06-12T02:37:35.164 INFO:tasks.rados.rados.0.smithi107.stdout:976: finishing write tid 6 to smithi10729407-11 2020-06-12T02:37:35.165 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 166 (ObjNum 975 snap 0 seq_num 975) dirty exists 2020-06-12T02:37:35.165 INFO:tasks.rados.rados.0.smithi107.stdout:976: left oid 11 (ObjNum 975 snap 0 seq_num 975) 2020-06-12T02:37:35.166 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-06-12T02:37:35.167 INFO:tasks.rados.rados.0.smithi107.stdout:979: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:35.168 INFO:tasks.rados.rados.0.smithi107.stdout:979: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:35.169 INFO:tasks.rados.rados.0.smithi107.stdout:979: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:35.169 INFO:tasks.rados.rados.0.smithi107.stdout:979: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:35.169 INFO:tasks.rados.rados.0.smithi107.stdout:979: finishing write tid 6 to smithi10729407-6 2020-06-12T02:37:35.170 INFO:tasks.rados.rados.0.smithi107.stdout:976: done (2 left) 2020-06-12T02:37:35.171 INFO:tasks.rados.rados.0.smithi107.stdout:980: done (1 left) 2020-06-12T02:37:35.171 INFO:tasks.rados.rados.0.smithi107.stdout:981: delete oid 1 current snap is 0 2020-06-12T02:37:35.171 INFO:tasks.rados.rados.0.smithi107.stdout:979: finishing write tid 7 to smithi10729407-6 2020-06-12T02:37:35.171 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 212 (ObjNum 978 snap 0 seq_num 978) dirty exists 2020-06-12T02:37:35.172 INFO:tasks.rados.rados.0.smithi107.stdout:979: left oid 6 (ObjNum 978 snap 0 seq_num 978) 2020-06-12T02:37:35.172 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:35.167+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:35.172 INFO:tasks.rados.rados.0.smithi107.stdout:979: done (1 left) 2020-06-12T02:37:35.172 INFO:tasks.rados.rados.0.smithi107.stdout:981: done (0 left) 2020-06-12T02:37:35.173 INFO:tasks.rados.rados.0.smithi107.stdout:982: delete oid 20 current snap is 0 2020-06-12T02:37:35.173 INFO:tasks.rados.rados.0.smithi107.stdout:982: done (0 left) 2020-06-12T02:37:35.174 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:37:35.174 INFO:tasks.rados.rados.0.smithi107.stdout:983: seq_num 982 ranges {1589248=450560,2039808=425984,2465792=679936,3145728=466944,3612672=32768} 2020-06-12T02:37:35.184 INFO:tasks.rados.rados.0.smithi107.stdout:983: writing smithi10729407-13 from 1589248 to 2039808 tid 1 2020-06-12T02:37:35.187 INFO:tasks.rados.rados.0.smithi107.stdout:983: writing smithi10729407-13 from 2039808 to 2465792 tid 2 2020-06-12T02:37:35.191 INFO:tasks.rados.rados.0.smithi107.stdout:983: writing smithi10729407-13 from 2465792 to 3145728 tid 3 2020-06-12T02:37:35.206 INFO:tasks.rados.rados.0.smithi107.stdout:983: writing smithi10729407-13 from 3145728 to 3612672 tid 4 2020-06-12T02:37:35.207 INFO:tasks.rados.rados.0.smithi107.stdout:983: writing smithi10729407-13 from 3612672 to 3645440 tid 5 2020-06-12T02:37:35.207 INFO:tasks.rados.rados.0.smithi107.stdout:984: delete oid 16 current snap is 0 2020-06-12T02:37:35.207 INFO:tasks.rados.rados.0.smithi107.stdout:983: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:35.207 INFO:tasks.rados.rados.0.smithi107.stdout:983: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:35.208 INFO:tasks.rados.rados.0.smithi107.stdout:984: done (1 left) 2020-06-12T02:37:35.208 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:37:35.208 INFO:tasks.rados.rados.0.smithi107.stdout:985: seq_num 984 ranges {1703936=483328,2187264=598016,2785280=647168,3432448=360448} 2020-06-12T02:37:35.212 INFO:tasks.rados.rados.0.smithi107.stdout:985: writing smithi10729407-7 from 1703936 to 2187264 tid 1 2020-06-12T02:37:35.216 INFO:tasks.rados.rados.0.smithi107.stdout:985: writing smithi10729407-7 from 2187264 to 2785280 tid 2 2020-06-12T02:37:35.221 INFO:tasks.rados.rados.0.smithi107.stdout:985: writing smithi10729407-7 from 2785280 to 3432448 tid 3 2020-06-12T02:37:35.224 INFO:tasks.rados.rados.0.smithi107.stdout:985: writing smithi10729407-7 from 3432448 to 3792896 tid 4 2020-06-12T02:37:35.224 INFO:tasks.rados.rados.0.smithi107.stdout:986: delete oid 20 current snap is 0 2020-06-12T02:37:35.224 INFO:tasks.rados.rados.0.smithi107.stdout:983: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:35.225 INFO:tasks.rados.rados.0.smithi107.stdout:983: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:35.225 INFO:tasks.rados.rados.0.smithi107.stdout:983: finishing write tid 5 to smithi10729407-13 2020-06-12T02:37:35.225 INFO:tasks.rados.rados.0.smithi107.stdout:983: finishing write tid 7 to smithi10729407-13 2020-06-12T02:37:35.225 INFO:tasks.rados.rados.0.smithi107.stdout:983: finishing write tid 8 to smithi10729407-13 2020-06-12T02:37:35.225 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 200 (ObjNum 982 snap 0 seq_num 982) dirty exists 2020-06-12T02:37:35.226 INFO:tasks.rados.rados.0.smithi107.stdout:983: left oid 13 (ObjNum 982 snap 0 seq_num 982) 2020-06-12T02:37:35.226 INFO:tasks.rados.rados.0.smithi107.stdout:985: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:35.226 INFO:tasks.rados.rados.0.smithi107.stdout:983: done (2 left) 2020-06-12T02:37:35.227 INFO:tasks.rados.rados.0.smithi107.stdout:986: done (1 left) 2020-06-12T02:37:35.227 INFO:tasks.rados.rados.0.smithi107.stdout:987: delete oid 1 current snap is 0 2020-06-12T02:37:35.227 INFO:tasks.rados.rados.0.smithi107.stdout:985: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:35.228 INFO:tasks.rados.rados.0.smithi107.stdout:987: done (1 left) 2020-06-12T02:37:35.228 INFO:tasks.rados.rados.0.smithi107.stdout:988: delete oid 9 current snap is 0 2020-06-12T02:37:35.230 INFO:tasks.rados.rados.0.smithi107.stdout:988: done (1 left) 2020-06-12T02:37:35.230 INFO:tasks.rados.rados.0.smithi107.stdout:989: delete oid 3 current snap is 0 2020-06-12T02:37:35.231 INFO:tasks.rados.rados.0.smithi107.stdout:985: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:35.231 INFO:tasks.rados.rados.0.smithi107.stdout:985: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:35.232 INFO:tasks.rados.rados.0.smithi107.stdout:985: finishing write tid 6 to smithi10729407-7 2020-06-12T02:37:35.233 INFO:tasks.rados.rados.0.smithi107.stdout:985: finishing write tid 7 to smithi10729407-7 2020-06-12T02:37:35.234 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 205 (ObjNum 984 snap 0 seq_num 984) dirty exists 2020-06-12T02:37:35.234 INFO:tasks.rados.rados.0.smithi107.stdout:985: left oid 7 (ObjNum 984 snap 0 seq_num 984) 2020-06-12T02:37:35.234 INFO:tasks.rados.rados.0.smithi107.stdout:985: done (1 left) 2020-06-12T02:37:35.235 INFO:tasks.rados.rados.0.smithi107.stdout:989: done (0 left) 2020-06-12T02:37:35.235 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:37:35.235 INFO:tasks.rados.rados.0.smithi107.stdout:990: seq_num 989 ranges {3792896=573440,4366336=466944,4833280=434176,5267456=638976,5906432=65536} 2020-06-12T02:37:35.260 INFO:tasks.rados.rados.0.smithi107.stdout:990: writing smithi10729407-7 from 3792896 to 4366336 tid 1 2020-06-12T02:37:35.260 INFO:tasks.rados.rados.0.smithi107.stdout:990: writing smithi10729407-7 from 4366336 to 4833280 tid 2 2020-06-12T02:37:35.262 INFO:tasks.rados.rados.0.smithi107.stdout:990: writing smithi10729407-7 from 4833280 to 5267456 tid 3 2020-06-12T02:37:35.266 INFO:tasks.rados.rados.0.smithi107.stdout:990: writing smithi10729407-7 from 5267456 to 5906432 tid 4 2020-06-12T02:37:35.267 INFO:tasks.rados.rados.0.smithi107.stdout:990: writing smithi10729407-7 from 5906432 to 5971968 tid 5 2020-06-12T02:37:35.267 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:37:35.268 INFO:tasks.rados.rados.0.smithi107.stdout:991: seq_num 990 ranges {1908736=589824,2498560=753664,3252224=638976,3891200=212992} 2020-06-12T02:37:35.271 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:37:35.268+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:37:35.281 INFO:tasks.rados.rados.0.smithi107.stdout:991: writing smithi10729407-10 from 1908736 to 2498560 tid 1 2020-06-12T02:37:35.286 INFO:tasks.rados.rados.0.smithi107.stdout:991: writing smithi10729407-10 from 2498560 to 3252224 tid 2 2020-06-12T02:37:35.290 INFO:tasks.rados.rados.0.smithi107.stdout:991: writing smithi10729407-10 from 3252224 to 3891200 tid 3 2020-06-12T02:37:35.292 INFO:tasks.rados.rados.0.smithi107.stdout:991: writing smithi10729407-10 from 3891200 to 4104192 tid 4 2020-06-12T02:37:35.292 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:37:35.292 INFO:tasks.rados.rados.0.smithi107.stdout:992: seq_num 991 ranges {1638400=606208,2244608=638976,2883584=794624,3678208=172032} 2020-06-12T02:37:35.299 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:35.300 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:35.306 INFO:tasks.rados.rados.0.smithi107.stdout:992: writing smithi10729407-14 from 1638400 to 2244608 tid 1 2020-06-12T02:37:35.308 INFO:tasks.rados.rados.0.smithi107.stdout:992: writing smithi10729407-14 from 2244608 to 2883584 tid 2 2020-06-12T02:37:35.311 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-06-12T02:37:35.313 INFO:tasks.rados.rados.0.smithi107.stdout:992: writing smithi10729407-14 from 2883584 to 3678208 tid 3 2020-06-12T02:37:35.314 INFO:tasks.rados.rados.0.smithi107.stdout:992: writing smithi10729407-14 from 3678208 to 3850240 tid 4 2020-06-12T02:37:35.315 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:37:35.315 INFO:tasks.rados.rados.0.smithi107.stdout:993: seq_num 992 ranges {1941504=622592,2564096=524288,3088384=606208,3694592=475136} 2020-06-12T02:37:35.328 INFO:tasks.rados.rados.0.smithi107.stdout:993: writing smithi10729407-11 from 1941504 to 2564096 tid 1 2020-06-12T02:37:35.339 INFO:tasks.rados.rados.0.smithi107.stdout:993: writing smithi10729407-11 from 2564096 to 3088384 tid 2 2020-06-12T02:37:35.339 INFO:tasks.rados.rados.0.smithi107.stdout:993: writing smithi10729407-11 from 3088384 to 3694592 tid 3 2020-06-12T02:37:35.339 INFO:tasks.rados.rados.0.smithi107.stdout:993: writing smithi10729407-11 from 3694592 to 4169728 tid 4 2020-06-12T02:37:35.340 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:37:35.340 INFO:tasks.rados.rados.0.smithi107.stdout:994: seq_num 993 ranges {0=638976,638976=409600,1048576=409600,1458176=688128,2146304=98304} 2020-06-12T02:37:35.342 INFO:tasks.rados.rados.0.smithi107.stdout:994: writing smithi10729407-5 from 0 to 638976 tid 1 2020-06-12T02:37:35.344 INFO:tasks.rados.rados.0.smithi107.stdout:994: writing smithi10729407-5 from 638976 to 1048576 tid 2 2020-06-12T02:37:35.347 INFO:tasks.rados.rados.0.smithi107.stdout:994: writing smithi10729407-5 from 1048576 to 1458176 tid 3 2020-06-12T02:37:35.351 INFO:tasks.rados.rados.0.smithi107.stdout:994: writing smithi10729407-5 from 1458176 to 2146304 tid 4 2020-06-12T02:37:35.352 INFO:tasks.rados.rados.0.smithi107.stdout:994: writing smithi10729407-5 from 2146304 to 2244608 tid 5 2020-06-12T02:37:35.353 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:37:35.353 INFO:tasks.rados.rados.0.smithi107.stdout:995: seq_num 994 ranges {0=655360,655360=688128,1343488=622592,1966080=294912} 2020-06-12T02:37:35.357 INFO:tasks.rados.rados.0.smithi107.stdout:995: writing smithi10729407-16 from 0 to 655360 tid 1 2020-06-12T02:37:35.361 INFO:tasks.rados.rados.0.smithi107.stdout:995: writing smithi10729407-16 from 655360 to 1343488 tid 2 2020-06-12T02:37:35.364 INFO:tasks.rados.rados.0.smithi107.stdout:995: writing smithi10729407-16 from 1343488 to 1966080 tid 3 2020-06-12T02:37:35.367 INFO:tasks.rados.rados.0.smithi107.stdout:995: writing smithi10729407-16 from 1966080 to 2260992 tid 4 2020-06-12T02:37:35.367 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:37:35.367 INFO:tasks.rados.rados.0.smithi107.stdout:996: seq_num 995 ranges {1622016=671744,2293760=573440,2867200=778240,3645440=253952} 2020-06-12T02:37:35.371 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:37:35.368+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:37:35.379 INFO:tasks.rados.rados.0.smithi107.stdout:996: writing smithi10729407-2 from 1622016 to 2293760 tid 1 2020-06-12T02:37:35.383 INFO:tasks.rados.rados.0.smithi107.stdout:996: writing smithi10729407-2 from 2293760 to 2867200 tid 2 2020-06-12T02:37:35.387 INFO:tasks.rados.rados.0.smithi107.stdout:996: writing smithi10729407-2 from 2867200 to 3645440 tid 3 2020-06-12T02:37:35.390 INFO:tasks.rados.rados.0.smithi107.stdout:996: writing smithi10729407-2 from 3645440 to 3899392 tid 4 2020-06-12T02:37:35.391 INFO:tasks.rados.rados.0.smithi107.stdout:997: delete oid 18 current snap is 0 2020-06-12T02:37:35.391 INFO:tasks.rados.rados.0.smithi107.stdout:990: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:35.391 INFO:tasks.rados.rados.0.smithi107.stdout:990: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:35.391 INFO:tasks.rados.rados.0.smithi107.stdout:990: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:35.392 INFO:tasks.rados.rados.0.smithi107.stdout:990: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:35.392 INFO:tasks.rados.rados.0.smithi107.stdout:990: finishing write tid 5 to smithi10729407-7 2020-06-12T02:37:35.392 INFO:tasks.rados.rados.0.smithi107.stdout:990: finishing write tid 7 to smithi10729407-7 2020-06-12T02:37:35.393 INFO:tasks.rados.rados.0.smithi107.stdout:990: finishing write tid 8 to smithi10729407-7 2020-06-12T02:37:35.393 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 211 (ObjNum 989 snap 0 seq_num 989) dirty exists 2020-06-12T02:37:35.393 INFO:tasks.rados.rados.0.smithi107.stdout:990: left oid 7 (ObjNum 989 snap 0 seq_num 989) 2020-06-12T02:37:35.393 INFO:tasks.rados.rados.0.smithi107.stdout:991: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:35.394 INFO:tasks.rados.rados.0.smithi107.stdout:991: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:35.394 INFO:tasks.rados.rados.0.smithi107.stdout:991: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:35.394 INFO:tasks.rados.rados.0.smithi107.stdout:991: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:35.395 INFO:tasks.rados.rados.0.smithi107.stdout:991: finishing write tid 6 to smithi10729407-10 2020-06-12T02:37:35.395 INFO:tasks.rados.rados.0.smithi107.stdout:991: finishing write tid 7 to smithi10729407-10 2020-06-12T02:37:35.396 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 95 (ObjNum 990 snap 0 seq_num 990) dirty exists 2020-06-12T02:37:35.396 INFO:tasks.rados.rados.0.smithi107.stdout:991: left oid 10 (ObjNum 990 snap 0 seq_num 990) 2020-06-12T02:37:35.397 INFO:tasks.rados.rados.0.smithi107.stdout:992: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:35.397 INFO:tasks.rados.rados.0.smithi107.stdout:992: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:35.397 INFO:tasks.rados.rados.0.smithi107.stdout:992: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:35.398 INFO:tasks.rados.rados.0.smithi107.stdout:992: finishing write tid 4 to smithi10729407-14 2020-06-12T02:37:35.398 INFO:tasks.rados.rados.0.smithi107.stdout:992: finishing write tid 6 to smithi10729407-14 2020-06-12T02:37:35.398 INFO:tasks.rados.rados.0.smithi107.stdout:992: finishing write tid 7 to smithi10729407-14 2020-06-12T02:37:35.398 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 171 (ObjNum 991 snap 0 seq_num 991) dirty exists 2020-06-12T02:37:35.400 INFO:tasks.rados.rados.0.smithi107.stdout:992: left oid 14 (ObjNum 991 snap 0 seq_num 991) 2020-06-12T02:37:35.400 INFO:tasks.rados.rados.0.smithi107.stdout:993: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:35.400 INFO:tasks.rados.rados.0.smithi107.stdout:993: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:35.404 INFO:tasks.rados.rados.0.smithi107.stdout:993: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:35.404 INFO:tasks.rados.rados.0.smithi107.stdout:993: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:35.404 INFO:tasks.rados.rados.0.smithi107.stdout:993: finishing write tid 6 to smithi10729407-11 2020-06-12T02:37:35.405 INFO:tasks.rados.rados.0.smithi107.stdout:993: finishing write tid 7 to smithi10729407-11 2020-06-12T02:37:35.405 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 176 (ObjNum 992 snap 0 seq_num 992) dirty exists 2020-06-12T02:37:35.405 INFO:tasks.rados.rados.0.smithi107.stdout:993: left oid 11 (ObjNum 992 snap 0 seq_num 992) 2020-06-12T02:37:35.406 INFO:tasks.rados.rados.0.smithi107.stdout:994: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:35.406 INFO:tasks.rados.rados.0.smithi107.stdout:994: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:35.406 INFO:tasks.rados.rados.0.smithi107.stdout:994: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:35.407 INFO:tasks.rados.rados.0.smithi107.stdout:994: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:35.407 INFO:tasks.rados.rados.0.smithi107.stdout:994: finishing write tid 5 to smithi10729407-5 2020-06-12T02:37:35.407 INFO:tasks.rados.rados.0.smithi107.stdout:994: finishing write tid 7 to smithi10729407-5 2020-06-12T02:37:35.407 INFO:tasks.rados.rados.0.smithi107.stdout:994: finishing write tid 8 to smithi10729407-5 2020-06-12T02:37:35.408 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 88 (ObjNum 993 snap 0 seq_num 993) dirty exists 2020-06-12T02:37:35.408 INFO:tasks.rados.rados.0.smithi107.stdout:994: left oid 5 (ObjNum 993 snap 0 seq_num 993) 2020-06-12T02:37:35.408 INFO:tasks.rados.rados.0.smithi107.stdout:995: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:35.409 INFO:tasks.rados.rados.0.smithi107.stdout:995: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:35.410 INFO:tasks.rados.rados.0.smithi107.stdout:995: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:35.410 INFO:tasks.rados.rados.0.smithi107.stdout:995: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:35.411 INFO:tasks.rados.rados.0.smithi107.stdout:995: finishing write tid 6 to smithi10729407-16 2020-06-12T02:37:35.411 INFO:tasks.rados.rados.0.smithi107.stdout:995: finishing write tid 7 to smithi10729407-16 2020-06-12T02:37:35.412 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 181 (ObjNum 994 snap 0 seq_num 994) dirty exists 2020-06-12T02:37:35.412 INFO:tasks.rados.rados.0.smithi107.stdout:995: left oid 16 (ObjNum 994 snap 0 seq_num 994) 2020-06-12T02:37:35.412 INFO:tasks.rados.rados.0.smithi107.stdout:996: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:35.415 INFO:tasks.rados.rados.0.smithi107.stdout:990: done (7 left) 2020-06-12T02:37:35.416 INFO:tasks.rados.rados.0.smithi107.stdout:991: done (6 left) 2020-06-12T02:37:35.418 INFO:tasks.rados.rados.0.smithi107.stdout:992: done (5 left) 2020-06-12T02:37:35.418 INFO:tasks.rados.rados.0.smithi107.stdout:993: done (4 left) 2020-06-12T02:37:35.419 INFO:tasks.rados.rados.0.smithi107.stdout:994: done (3 left) 2020-06-12T02:37:35.419 INFO:tasks.rados.rados.0.smithi107.stdout:995: done (2 left) 2020-06-12T02:37:35.420 INFO:tasks.rados.rados.0.smithi107.stdout:997: done (1 left) 2020-06-12T02:37:35.420 INFO:tasks.rados.rados.0.smithi107.stdout:998: delete oid 4 current snap is 0 2020-06-12T02:37:35.421 INFO:tasks.rados.rados.0.smithi107.stdout:998: done (1 left) 2020-06-12T02:37:35.421 INFO:tasks.rados.rados.0.smithi107.stdout:999: delete oid 18 current snap is 0 2020-06-12T02:37:35.421 INFO:tasks.rados.rados.0.smithi107.stdout:996: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:35.422 INFO:tasks.rados.rados.0.smithi107.stdout:996: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:35.422 INFO:tasks.rados.rados.0.smithi107.stdout:996: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:35.423 INFO:tasks.rados.rados.0.smithi107.stdout:999: done (1 left) 2020-06-12T02:37:35.423 INFO:tasks.rados.rados.0.smithi107.stdout:1000: delete oid 15 current snap is 0 2020-06-12T02:37:35.423 INFO:tasks.rados.rados.0.smithi107.stdout:996: finishing write tid 6 to smithi10729407-2 2020-06-12T02:37:35.424 INFO:tasks.rados.rados.0.smithi107.stdout:996: finishing write tid 7 to smithi10729407-2 2020-06-12T02:37:35.424 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 103 (ObjNum 995 snap 0 seq_num 995) dirty exists 2020-06-12T02:37:35.424 INFO:tasks.rados.rados.0.smithi107.stdout:996: left oid 2 (ObjNum 995 snap 0 seq_num 995) 2020-06-12T02:37:35.424 INFO:tasks.rados.rados.0.smithi107.stdout:996: done (1 left) 2020-06-12T02:37:35.425 INFO:tasks.rados.rados.0.smithi107.stdout:1000: done (0 left) 2020-06-12T02:37:35.425 INFO:tasks.rados.rados.0.smithi107.stdout:1001: delete oid 18 current snap is 0 2020-06-12T02:37:35.425 INFO:tasks.rados.rados.0.smithi107.stdout:1001: done (0 left) 2020-06-12T02:37:35.425 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:37:35.426 INFO:tasks.rados.rados.0.smithi107.stdout:1002: seq_num 1001 ranges {0=770048,770048=729088,1499136=770048,2269184=106496} 2020-06-12T02:37:35.426 INFO:tasks.rados.rados.0.smithi107.stdout:1002: writing smithi10729407-19 from 0 to 770048 tid 1 2020-06-12T02:37:35.426 INFO:tasks.rados.rados.0.smithi107.stdout:1002: writing smithi10729407-19 from 770048 to 1499136 tid 2 2020-06-12T02:37:35.426 INFO:tasks.rados.rados.0.smithi107.stdout:1002: writing smithi10729407-19 from 1499136 to 2269184 tid 3 2020-06-12T02:37:35.427 INFO:tasks.rados.rados.0.smithi107.stdout:1002: writing smithi10729407-19 from 2269184 to 2375680 tid 4 2020-06-12T02:37:35.427 INFO:tasks.rados.rados.0.smithi107.stdout:1003: delete oid 8 current snap is 0 2020-06-12T02:37:35.427 INFO:tasks.rados.rados.0.smithi107.stdout:1002: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:35.432 INFO:tasks.rados.rados.0.smithi107.stdout:1003: done (1 left) 2020-06-12T02:37:35.432 INFO:tasks.rados.rados.0.smithi107.stdout:1004: delete oid 15 current snap is 0 2020-06-12T02:37:35.433 INFO:tasks.rados.rados.0.smithi107.stdout:1002: finishing write tid 2 to smithi10729407-19 2020-06-12T02:37:35.433 INFO:tasks.rados.rados.0.smithi107.stdout:1004: done (1 left) 2020-06-12T02:37:35.434 INFO:tasks.rados.rados.0.smithi107.stdout:1005: delete oid 1 current snap is 0 2020-06-12T02:37:35.434 INFO:tasks.rados.rados.0.smithi107.stdout:1002: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:35.434 INFO:tasks.rados.rados.0.smithi107.stdout:1005: done (1 left) 2020-06-12T02:37:35.435 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:37:35.435 INFO:tasks.rados.rados.0.smithi107.stdout:1006: seq_num 1005 ranges {3661824=32768} 2020-06-12T02:37:35.443 INFO:tasks.rados.rados.0.smithi107.stdout:1006: writing smithi10729407-6 from 3661824 to 3694592 tid 1 2020-06-12T02:37:35.444 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:37:35.445 INFO:tasks.rados.rados.0.smithi107.stdout:1007: seq_num 1006 ranges {0=49152} 2020-06-12T02:37:35.445 INFO:tasks.rados.rados.0.smithi107.stdout:1007: writing smithi10729407-4 from 0 to 49152 tid 1 2020-06-12T02:37:35.445 INFO:tasks.rados.rados.0.smithi107.stdout:1008: delete oid 17 current snap is 0 2020-06-12T02:37:35.446 INFO:tasks.rados.rados.0.smithi107.stdout:1002: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:35.446 INFO:tasks.rados.rados.0.smithi107.stdout:1002: finishing write tid 6 to smithi10729407-19 2020-06-12T02:37:35.446 INFO:tasks.rados.rados.0.smithi107.stdout:1002: finishing write tid 7 to smithi10729407-19 2020-06-12T02:37:35.446 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 160 (ObjNum 1001 snap 0 seq_num 1001) dirty exists 2020-06-12T02:37:35.447 INFO:tasks.rados.rados.0.smithi107.stdout:1002: left oid 19 (ObjNum 1001 snap 0 seq_num 1001) 2020-06-12T02:37:35.448 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:35.448 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:35.449 INFO:tasks.rados.rados.0.smithi107.stdout:1006: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:35.449 INFO:tasks.rados.rados.0.smithi107.stdout:1007: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:35.449 INFO:tasks.rados.rados.0.smithi107.stdout:1007: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:35.451 INFO:tasks.rados.rados.0.smithi107.stdout:1002: done (3 left) 2020-06-12T02:37:35.452 INFO:tasks.rados.rados.0.smithi107.stdout:1008: done (2 left) 2020-06-12T02:37:35.452 INFO:tasks.rados.rados.0.smithi107.stdout:1009: delete oid 2 current snap is 0 2020-06-12T02:37:35.452 INFO:tasks.rados.rados.0.smithi107.stdout:1006: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:35.453 INFO:tasks.rados.rados.0.smithi107.stdout:1007: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:35.453 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 177 (ObjNum 1006 snap 0 seq_num 1006) dirty exists 2020-06-12T02:37:35.453 INFO:tasks.rados.rados.0.smithi107.stdout:1007: left oid 4 (ObjNum 1006 snap 0 seq_num 1006) 2020-06-12T02:37:35.453 INFO:tasks.rados.rados.0.smithi107.stdout:1006: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:35.454 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 218 (ObjNum 1005 snap 0 seq_num 1005) dirty exists 2020-06-12T02:37:35.454 INFO:tasks.rados.rados.0.smithi107.stdout:1006: left oid 6 (ObjNum 1005 snap 0 seq_num 1005) 2020-06-12T02:37:35.455 INFO:tasks.rados.rados.0.smithi107.stdout:1006: done (2 left) 2020-06-12T02:37:35.455 INFO:tasks.rados.rados.0.smithi107.stdout:1007: done (1 left) 2020-06-12T02:37:35.455 INFO:tasks.rados.rados.0.smithi107.stdout:1009: done (0 left) 2020-06-12T02:37:35.455 INFO:tasks.rados.rados.0.smithi107.stdout:1010: delete oid 11 current snap is 0 2020-06-12T02:37:35.457 INFO:tasks.rados.rados.0.smithi107.stdout:1010: done (0 left) 2020-06-12T02:37:35.457 INFO:tasks.rados.rados.0.smithi107.stdout:1011: delete oid 4 current snap is 0 2020-06-12T02:37:35.457 INFO:tasks.rados.rados.0.smithi107.stdout:1011: done (0 left) 2020-06-12T02:37:35.458 INFO:tasks.rados.rados.0.smithi107.stdout:1012: delete oid 8 current snap is 0 2020-06-12T02:37:35.458 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-06-12T02:37:35.475 INFO:tasks.rados.rados.0.smithi107.stdout:1012: done (0 left) 2020-06-12T02:37:35.476 INFO:tasks.rados.rados.0.smithi107.stdout:1013: delete oid 6 current snap is 0 2020-06-12T02:37:35.476 INFO:tasks.rados.rados.0.smithi107.stdout:1013: done (0 left) 2020-06-12T02:37:35.477 INFO:tasks.rados.rados.0.smithi107.stdout:1014: delete oid 19 current snap is 0 2020-06-12T02:37:35.477 INFO:tasks.rados.rados.0.smithi107.stdout:1014: done (0 left) 2020-06-12T02:37:35.477 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:37:35.478 INFO:tasks.rados.rados.0.smithi107.stdout:1015: seq_num 1014 ranges {0=188416} 2020-06-12T02:37:35.479 INFO:tasks.rados.rados.0.smithi107.stdout:1015: writing smithi10729407-12 from 0 to 188416 tid 1 2020-06-12T02:37:35.479 INFO:tasks.rados.rados.0.smithi107.stdout:1016: delete oid 11 current snap is 0 2020-06-12T02:37:35.479 INFO:tasks.rados.rados.0.smithi107.stdout:1016: done (1 left) 2020-06-12T02:37:35.479 INFO:tasks.rados.rados.0.smithi107.stdout:1017: delete oid 5 current snap is 0 2020-06-12T02:37:35.480 INFO:tasks.rados.rados.0.smithi107.stdout:1015: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:35.480 INFO:tasks.rados.rados.0.smithi107.stdout:1015: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:35.480 INFO:tasks.rados.rados.0.smithi107.stdout:1017: done (1 left) 2020-06-12T02:37:35.481 INFO:tasks.rados.rados.0.smithi107.stdout:1018: delete oid 11 current snap is 0 2020-06-12T02:37:35.482 INFO:tasks.rados.rados.0.smithi107.stdout:1018: done (1 left) 2020-06-12T02:37:35.482 INFO:tasks.rados.rados.0.smithi107.stdout:1019: delete oid 16 current snap is 0 2020-06-12T02:37:35.482 INFO:tasks.rados.rados.0.smithi107.stdout:1015: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:35.483 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 163 (ObjNum 1014 snap 0 seq_num 1014) dirty exists 2020-06-12T02:37:35.483 INFO:tasks.rados.rados.0.smithi107.stdout:1015: left oid 12 (ObjNum 1014 snap 0 seq_num 1014) 2020-06-12T02:37:35.483 INFO:tasks.rados.rados.0.smithi107.stdout:1015: done (1 left) 2020-06-12T02:37:35.484 INFO:tasks.rados.rados.0.smithi107.stdout:1019: done (0 left) 2020-06-12T02:37:35.485 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:37:35.485 INFO:tasks.rados.rados.0.smithi107.stdout:1020: seq_num 1019 ranges {3850240=270336} 2020-06-12T02:37:35.488 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:35.485+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:35.498 INFO:tasks.rados.rados.0.smithi107.stdout:1020: writing smithi10729407-14 from 3850240 to 4120576 tid 1 2020-06-12T02:37:35.498 INFO:tasks.rados.rados.0.smithi107.stdout:1021: delete oid 12 current snap is 0 2020-06-12T02:37:35.500 INFO:tasks.rados.rados.0.smithi107.stdout:1021: done (1 left) 2020-06-12T02:37:35.500 INFO:tasks.rados.rados.0.smithi107.stdout:1022: delete oid 12 current snap is 0 2020-06-12T02:37:35.502 INFO:tasks.rados.rados.0.smithi107.stdout:1022: done (1 left) 2020-06-12T02:37:35.502 INFO:tasks.rados.rados.0.smithi107.stdout:1023: delete oid 6 current snap is 0 2020-06-12T02:37:35.504 INFO:tasks.rados.rados.0.smithi107.stdout:1020: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:35.505 INFO:tasks.rados.rados.0.smithi107.stdout:1020: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:35.505 INFO:tasks.rados.rados.0.smithi107.stdout:1023: done (1 left) 2020-06-12T02:37:35.505 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:37:35.506 INFO:tasks.rados.rados.0.smithi107.stdout:1024: seq_num 1023 ranges {0=335872} 2020-06-12T02:37:35.507 INFO:tasks.rados.rados.0.smithi107.stdout:1024: writing smithi10729407-8 from 0 to 335872 tid 1 2020-06-12T02:37:35.508 INFO:tasks.rados.rados.0.smithi107.stdout:1025: delete oid 7 current snap is 0 2020-06-12T02:37:35.510 INFO:tasks.rados.rados.0.smithi107.stdout:1020: finishing write tid 4 to smithi10729407-14 2020-06-12T02:37:35.511 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 181 (ObjNum 1019 snap 0 seq_num 1019) dirty exists 2020-06-12T02:37:35.511 INFO:tasks.rados.rados.0.smithi107.stdout:1020: left oid 14 (ObjNum 1019 snap 0 seq_num 1019) 2020-06-12T02:37:35.512 INFO:tasks.rados.rados.0.smithi107.stdout:1020: done (2 left) 2020-06-12T02:37:35.512 INFO:tasks.rados.rados.0.smithi107.stdout:1025: done (1 left) 2020-06-12T02:37:35.512 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:37:35.513 INFO:tasks.rados.rados.0.smithi107.stdout:1026: seq_num 1025 ranges {4104192=368640} 2020-06-12T02:37:35.534 INFO:tasks.rados.rados.0.smithi107.stdout:1026: writing smithi10729407-10 from 4104192 to 4472832 tid 1 2020-06-12T02:37:35.535 INFO:tasks.rados.rados.0.smithi107.stdout:1027: delete oid 9 current snap is 0 2020-06-12T02:37:35.535 INFO:tasks.rados.rados.0.smithi107.stdout:1024: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:35.535 INFO:tasks.rados.rados.0.smithi107.stdout:1024: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:35.536 INFO:tasks.rados.rados.0.smithi107.stdout:1024: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:35.536 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 175 (ObjNum 1023 snap 0 seq_num 1023) dirty exists 2020-06-12T02:37:35.536 INFO:tasks.rados.rados.0.smithi107.stdout:1024: left oid 8 (ObjNum 1023 snap 0 seq_num 1023) 2020-06-12T02:37:35.537 INFO:tasks.rados.rados.0.smithi107.stdout:1024: done (2 left) 2020-06-12T02:37:35.537 INFO:tasks.rados.rados.0.smithi107.stdout:1027: done (1 left) 2020-06-12T02:37:35.537 INFO:tasks.rados.rados.0.smithi107.stdout:1028: delete oid 20 current snap is 0 2020-06-12T02:37:35.538 INFO:tasks.rados.rados.0.smithi107.stdout:1026: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:35.539 INFO:tasks.rados.rados.0.smithi107.stdout:1028: done (1 left) 2020-06-12T02:37:35.540 INFO:tasks.rados.rados.0.smithi107.stdout:1029: delete oid 5 current snap is 0 2020-06-12T02:37:35.541 INFO:tasks.rados.rados.0.smithi107.stdout:1026: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:35.541 INFO:tasks.rados.rados.0.smithi107.stdout:1029: done (1 left) 2020-06-12T02:37:35.541 INFO:tasks.rados.rados.0.smithi107.stdout:1030: delete oid 14 current snap is 0 2020-06-12T02:37:35.542 INFO:tasks.rados.rados.0.smithi107.stdout:1026: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:35.543 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 97 (ObjNum 1025 snap 0 seq_num 1025) dirty exists 2020-06-12T02:37:35.543 INFO:tasks.rados.rados.0.smithi107.stdout:1026: left oid 10 (ObjNum 1025 snap 0 seq_num 1025) 2020-06-12T02:37:35.543 INFO:tasks.rados.rados.0.smithi107.stdout:1026: done (1 left) 2020-06-12T02:37:35.544 INFO:tasks.rados.rados.0.smithi107.stdout:1030: done (0 left) 2020-06-12T02:37:35.544 INFO:tasks.rados.rados.0.smithi107.stdout:1031: delete oid 4 current snap is 0 2020-06-12T02:37:35.545 INFO:tasks.rados.rados.0.smithi107.stdout:1031: done (0 left) 2020-06-12T02:37:35.545 INFO:tasks.rados.rados.0.smithi107.stdout:1032: delete oid 4 current snap is 0 2020-06-12T02:37:35.546 INFO:tasks.rados.rados.0.smithi107.stdout:1032: done (0 left) 2020-06-12T02:37:35.546 INFO:tasks.rados.rados.0.smithi107.stdout:1033: delete oid 4 current snap is 0 2020-06-12T02:37:35.548 INFO:tasks.rados.rados.0.smithi107.stdout:1033: done (0 left) 2020-06-12T02:37:35.548 INFO:tasks.rados.rados.0.smithi107.stdout:1034: delete oid 20 current snap is 0 2020-06-12T02:37:35.549 INFO:tasks.rados.rados.0.smithi107.stdout:1034: done (0 left) 2020-06-12T02:37:35.550 INFO:tasks.rados.rados.0.smithi107.stdout:1035: delete oid 14 current snap is 0 2020-06-12T02:37:35.551 INFO:tasks.rados.rados.0.smithi107.stdout:1035: done (0 left) 2020-06-12T02:37:35.552 INFO:tasks.rados.rados.0.smithi107.stdout:1036: delete oid 17 current snap is 0 2020-06-12T02:37:35.553 INFO:tasks.rados.rados.0.smithi107.stdout:1036: done (0 left) 2020-06-12T02:37:35.553 INFO:tasks.rados.rados.0.smithi107.stdout:1037: delete oid 10 current snap is 0 2020-06-12T02:37:35.554 INFO:tasks.rados.rados.0.smithi107.stdout:1037: done (0 left) 2020-06-12T02:37:35.555 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:37:35.555 INFO:tasks.rados.rados.0.smithi107.stdout:1038: seq_num 1037 ranges {0=573440} 2020-06-12T02:37:35.558 INFO:tasks.rados.rados.0.smithi107.stdout:1038: writing smithi10729407-18 from 0 to 573440 tid 1 2020-06-12T02:37:35.558 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:37:35.559 INFO:tasks.rados.rados.0.smithi107.stdout:1039: seq_num 1038 ranges {0=589824} 2020-06-12T02:37:35.561 INFO:tasks.rados.rados.0.smithi107.stdout:1039: writing smithi10729407-9 from 0 to 589824 tid 1 2020-06-12T02:37:35.562 INFO:tasks.rados.rados.0.smithi107.stdout:1040: delete oid 7 current snap is 0 2020-06-12T02:37:35.564 INFO:tasks.rados.rados.0.smithi107.stdout:1040: done (2 left) 2020-06-12T02:37:35.565 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:37:35.565 INFO:tasks.rados.rados.0.smithi107.stdout:1041: seq_num 1040 ranges {0=622592} 2020-06-12T02:37:35.568 INFO:tasks.rados.rados.0.smithi107.stdout:1041: writing smithi10729407-17 from 0 to 622592 tid 1 2020-06-12T02:37:35.569 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:37:35.569 INFO:tasks.rados.rados.0.smithi107.stdout:1042: seq_num 1041 ranges {335872=638976} 2020-06-12T02:37:35.574 INFO:tasks.rados.rados.0.smithi107.stdout:1042: writing smithi10729407-8 from 335872 to 974848 tid 1 2020-06-12T02:37:35.574 INFO:tasks.rados.rados.0.smithi107.stdout:1043: delete oid 11 current snap is 0 2020-06-12T02:37:35.575 INFO:tasks.rados.rados.0.smithi107.stdout:1038: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:35.576 INFO:tasks.rados.rados.0.smithi107.stdout:1038: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:35.577 INFO:tasks.rados.rados.0.smithi107.stdout:1039: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:35.577 INFO:tasks.rados.rados.0.smithi107.stdout:1038: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:35.577 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 210 (ObjNum 1037 snap 0 seq_num 1037) dirty exists 2020-06-12T02:37:35.577 INFO:tasks.rados.rados.0.smithi107.stdout:1038: left oid 18 (ObjNum 1037 snap 0 seq_num 1037) 2020-06-12T02:37:35.578 INFO:tasks.rados.rados.0.smithi107.stdout:1039: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:35.579 INFO:tasks.rados.rados.0.smithi107.stdout:1039: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:35.579 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 212 (ObjNum 1038 snap 0 seq_num 1038) dirty exists 2020-06-12T02:37:35.579 INFO:tasks.rados.rados.0.smithi107.stdout:1039: left oid 9 (ObjNum 1038 snap 0 seq_num 1038) 2020-06-12T02:37:35.580 INFO:tasks.rados.rados.0.smithi107.stdout:1041: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:35.581 INFO:tasks.rados.rados.0.smithi107.stdout:1041: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:35.581 INFO:tasks.rados.rados.0.smithi107.stdout:1038: done (4 left) 2020-06-12T02:37:35.581 INFO:tasks.rados.rados.0.smithi107.stdout:1039: done (3 left) 2020-06-12T02:37:35.582 INFO:tasks.rados.rados.0.smithi107.stdout:1043: done (2 left) 2020-06-12T02:37:35.582 INFO:tasks.rados.rados.0.smithi107.stdout:1044: delete oid 4 current snap is 0 2020-06-12T02:37:35.582 INFO:tasks.rados.rados.0.smithi107.stdout:1041: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:35.582 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 185 (ObjNum 1040 snap 0 seq_num 1040) dirty exists 2020-06-12T02:37:35.583 INFO:tasks.rados.rados.0.smithi107.stdout:1041: left oid 17 (ObjNum 1040 snap 0 seq_num 1040) 2020-06-12T02:37:35.583 INFO:tasks.rados.rados.0.smithi107.stdout:1042: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:35.584 INFO:tasks.rados.rados.0.smithi107.stdout:1041: done (2 left) 2020-06-12T02:37:35.584 INFO:tasks.rados.rados.0.smithi107.stdout:1044: done (1 left) 2020-06-12T02:37:35.585 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:37:35.585 INFO:tasks.rados.rados.0.smithi107.stdout:1045: seq_num 1044 ranges {589824=688128} 2020-06-12T02:37:35.588 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:35.585+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:35.590 INFO:tasks.rados.rados.0.smithi107.stdout:1045: writing smithi10729407-9 from 589824 to 1277952 tid 1 2020-06-12T02:37:35.590 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:37:35.590 INFO:tasks.rados.rados.0.smithi107.stdout:1046: seq_num 1045 ranges {0=704512} 2020-06-12T02:37:35.594 INFO:tasks.rados.rados.0.smithi107.stdout:1046: writing smithi10729407-16 from 0 to 704512 tid 1 2020-06-12T02:37:35.595 INFO:tasks.rados.rados.0.smithi107.stdout:1047: delete oid 4 current snap is 0 2020-06-12T02:37:35.596 INFO:tasks.rados.rados.0.smithi107.stdout:1042: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:35.596 INFO:tasks.rados.rados.0.smithi107.stdout:1042: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:35.596 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 177 (ObjNum 1041 snap 0 seq_num 1041) dirty exists 2020-06-12T02:37:35.596 INFO:tasks.rados.rados.0.smithi107.stdout:1042: left oid 8 (ObjNum 1041 snap 0 seq_num 1041) 2020-06-12T02:37:35.597 INFO:tasks.rados.rados.0.smithi107.stdout:1042: done (3 left) 2020-06-12T02:37:35.598 INFO:tasks.rados.rados.0.smithi107.stdout:1047: done (2 left) 2020-06-12T02:37:35.598 INFO:tasks.rados.rados.0.smithi107.stdout:1048: delete oid 19 current snap is 0 2020-06-12T02:37:35.599 INFO:tasks.rados.rados.0.smithi107.stdout:1048: done (2 left) 2020-06-12T02:37:35.600 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:37:35.600 INFO:tasks.rados.rados.0.smithi107.stdout:1049: seq_num 1048 ranges {622592=761856} 2020-06-12T02:37:35.606 INFO:tasks.rados.rados.0.smithi107.stdout:1049: writing smithi10729407-17 from 622592 to 1384448 tid 1 2020-06-12T02:37:35.607 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:37:35.607 INFO:tasks.rados.rados.0.smithi107.stdout:1050: seq_num 1049 ranges {0=778240} 2020-06-12T02:37:35.611 INFO:tasks.rados.rados.0.smithi107.stdout:1050: writing smithi10729407-3 from 0 to 778240 tid 1 2020-06-12T02:37:35.612 INFO:tasks.rados.rados.0.smithi107.stdout:1051: delete oid 11 current snap is 0 2020-06-12T02:37:35.612 INFO:tasks.rados.rados.0.smithi107.stdout:1045: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:35.612 INFO:tasks.rados.rados.0.smithi107.stdout:1045: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:35.612 INFO:tasks.rados.rados.0.smithi107.stdout:1045: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:35.613 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 214 (ObjNum 1044 snap 0 seq_num 1044) dirty exists 2020-06-12T02:37:35.613 INFO:tasks.rados.rados.0.smithi107.stdout:1045: left oid 9 (ObjNum 1044 snap 0 seq_num 1044) 2020-06-12T02:37:35.613 INFO:tasks.rados.rados.0.smithi107.stdout:1046: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:35.613 INFO:tasks.rados.rados.0.smithi107.stdout:1046: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:35.614 INFO:tasks.rados.rados.0.smithi107.stdout:1046: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:35.614 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 184 (ObjNum 1045 snap 0 seq_num 1045) dirty exists 2020-06-12T02:37:35.614 INFO:tasks.rados.rados.0.smithi107.stdout:1046: left oid 16 (ObjNum 1045 snap 0 seq_num 1045) 2020-06-12T02:37:35.615 INFO:tasks.rados.rados.0.smithi107.stdout:1045: done (4 left) 2020-06-12T02:37:35.616 INFO:tasks.rados.rados.0.smithi107.stdout:1046: done (3 left) 2020-06-12T02:37:35.617 INFO:tasks.rados.rados.0.smithi107.stdout:1051: done (2 left) 2020-06-12T02:37:35.617 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:37:35.617 INFO:tasks.rados.rados.0.smithi107.stdout:1052: seq_num 1051 ranges {0=409600,409600=401408} 2020-06-12T02:37:35.617 INFO:tasks.rados.rados.0.smithi107.stdout:1052: writing smithi10729407-19 from 0 to 409600 tid 1 2020-06-12T02:37:35.618 INFO:tasks.rados.rados.0.smithi107.stdout:1052: writing smithi10729407-19 from 409600 to 811008 tid 2 2020-06-12T02:37:35.619 INFO:tasks.rados.rados.0.smithi107.stdout:1053: delete oid 6 current snap is 0 2020-06-12T02:37:35.619 INFO:tasks.rados.rados.0.smithi107.stdout:1049: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:35.619 INFO:tasks.rados.rados.0.smithi107.stdout:1049: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:35.619 INFO:tasks.rados.rados.0.smithi107.stdout:1049: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:35.620 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 189 (ObjNum 1048 snap 0 seq_num 1048) dirty exists 2020-06-12T02:37:35.620 INFO:tasks.rados.rados.0.smithi107.stdout:1049: left oid 17 (ObjNum 1048 snap 0 seq_num 1048) 2020-06-12T02:37:35.621 INFO:tasks.rados.rados.0.smithi107.stdout:1050: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:35.622 INFO:tasks.rados.rados.0.smithi107.stdout:1049: done (3 left) 2020-06-12T02:37:35.622 INFO:tasks.rados.rados.0.smithi107.stdout:1053: done (2 left) 2020-06-12T02:37:35.622 INFO:tasks.rados.rados.0.smithi107.stdout:1054: delete oid 14 current snap is 0 2020-06-12T02:37:35.623 INFO:tasks.rados.rados.0.smithi107.stdout:1052: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:35.623 INFO:tasks.rados.rados.0.smithi107.stdout:1050: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:35.624 INFO:tasks.rados.rados.0.smithi107.stdout:1050: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:35.624 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 186 (ObjNum 1049 snap 0 seq_num 1049) dirty exists 2020-06-12T02:37:35.625 INFO:tasks.rados.rados.0.smithi107.stdout:1050: left oid 3 (ObjNum 1049 snap 0 seq_num 1049) 2020-06-12T02:37:35.625 INFO:tasks.rados.rados.0.smithi107.stdout:1050: done (2 left) 2020-06-12T02:37:35.626 INFO:tasks.rados.rados.0.smithi107.stdout:1054: done (1 left) 2020-06-12T02:37:35.626 INFO:tasks.rados.rados.0.smithi107.stdout:1055: delete oid 20 current snap is 0 2020-06-12T02:37:35.627 INFO:tasks.rados.rados.0.smithi107.stdout:1052: finishing write tid 2 to smithi10729407-19 2020-06-12T02:37:35.627 INFO:tasks.rados.rados.0.smithi107.stdout:1052: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:35.627 INFO:tasks.rados.rados.0.smithi107.stdout:1055: done (1 left) 2020-06-12T02:37:35.628 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:37:35.628 INFO:tasks.rados.rados.0.smithi107.stdout:1056: seq_num 1055 ranges {0=475136,475136=401408} 2020-06-12T02:37:35.629 INFO:tasks.rados.rados.0.smithi107.stdout:1056: writing smithi10729407-11 from 0 to 475136 tid 1 2020-06-12T02:37:35.632 INFO:tasks.rados.rados.0.smithi107.stdout:1056: writing smithi10729407-11 from 475136 to 876544 tid 2 2020-06-12T02:37:35.632 INFO:tasks.rados.rados.0.smithi107.stdout:1057: delete oid 13 current snap is 0 2020-06-12T02:37:35.632 INFO:tasks.rados.rados.0.smithi107.stdout:1052: finishing write tid 5 to smithi10729407-19 2020-06-12T02:37:35.633 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 169 (ObjNum 1051 snap 0 seq_num 1051) dirty exists 2020-06-12T02:37:35.633 INFO:tasks.rados.rados.0.smithi107.stdout:1052: left oid 19 (ObjNum 1051 snap 0 seq_num 1051) 2020-06-12T02:37:35.634 INFO:tasks.rados.rados.0.smithi107.stdout:1052: done (2 left) 2020-06-12T02:37:35.634 INFO:tasks.rados.rados.0.smithi107.stdout:1057: done (1 left) 2020-06-12T02:37:35.634 INFO:tasks.rados.rados.0.smithi107.stdout:1058: delete oid 18 current snap is 0 2020-06-12T02:37:35.637 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:35.637 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:35.637 INFO:tasks.rados.rados.0.smithi107.stdout:1056: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:35.638 INFO:tasks.rados.rados.0.smithi107.stdout:1058: done (1 left) 2020-06-12T02:37:35.639 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:37:35.639 INFO:tasks.rados.rados.0.smithi107.stdout:1059: seq_num 1058 ranges {0=524288,524288=401408} 2020-06-12T02:37:35.642 INFO:tasks.rados.rados.0.smithi107.stdout:1059: writing smithi10729407-13 from 0 to 524288 tid 1 2020-06-12T02:37:35.644 INFO:tasks.rados.rados.0.smithi107.stdout:1059: writing smithi10729407-13 from 524288 to 925696 tid 2 2020-06-12T02:37:35.644 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:35.645 INFO:tasks.rados.rados.0.smithi107.stdout:1060: seq_num 1059 ranges {0=540672,540672=401408} 2020-06-12T02:37:35.647 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-06-12T02:37:35.649 INFO:tasks.rados.rados.0.smithi107.stdout:1060: writing smithi10729407-2 from 0 to 540672 tid 1 2020-06-12T02:37:35.656 INFO:tasks.rados.rados.0.smithi107.stdout:1060: writing smithi10729407-2 from 540672 to 942080 tid 2 2020-06-12T02:37:35.657 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:37:35.657 INFO:tasks.rados.rados.0.smithi107.stdout:1061: seq_num 1060 ranges {0=557056,557056=401408} 2020-06-12T02:37:35.657 INFO:tasks.rados.rados.0.smithi107.stdout:1061: writing smithi10729407-4 from 0 to 557056 tid 1 2020-06-12T02:37:35.658 INFO:tasks.rados.rados.0.smithi107.stdout:1061: writing smithi10729407-4 from 557056 to 958464 tid 2 2020-06-12T02:37:35.659 INFO:tasks.rados.rados.0.smithi107.stdout:1062: delete oid 15 current snap is 0 2020-06-12T02:37:35.659 INFO:tasks.rados.rados.0.smithi107.stdout:1056: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:35.659 INFO:tasks.rados.rados.0.smithi107.stdout:1056: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:35.659 INFO:tasks.rados.rados.0.smithi107.stdout:1056: finishing write tid 5 to smithi10729407-11 2020-06-12T02:37:35.659 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 189 (ObjNum 1055 snap 0 seq_num 1055) dirty exists 2020-06-12T02:37:35.660 INFO:tasks.rados.rados.0.smithi107.stdout:1056: left oid 11 (ObjNum 1055 snap 0 seq_num 1055) 2020-06-12T02:37:35.660 INFO:tasks.rados.rados.0.smithi107.stdout:1059: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:35.660 INFO:tasks.rados.rados.0.smithi107.stdout:1059: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:35.660 INFO:tasks.rados.rados.0.smithi107.stdout:1059: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:35.660 INFO:tasks.rados.rados.0.smithi107.stdout:1059: finishing write tid 5 to smithi10729407-13 2020-06-12T02:37:35.661 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 217 (ObjNum 1058 snap 0 seq_num 1058) dirty exists 2020-06-12T02:37:35.661 INFO:tasks.rados.rados.0.smithi107.stdout:1059: left oid 13 (ObjNum 1058 snap 0 seq_num 1058) 2020-06-12T02:37:35.661 INFO:tasks.rados.rados.0.smithi107.stdout:1060: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:35.662 INFO:tasks.rados.rados.0.smithi107.stdout:1056: done (4 left) 2020-06-12T02:37:35.662 INFO:tasks.rados.rados.0.smithi107.stdout:1059: done (3 left) 2020-06-12T02:37:35.662 INFO:tasks.rados.rados.0.smithi107.stdout:1062: done (2 left) 2020-06-12T02:37:35.662 INFO:tasks.rados.rados.0.smithi107.stdout:1063: delete oid 19 current snap is 0 2020-06-12T02:37:35.663 INFO:tasks.rados.rados.0.smithi107.stdout:1060: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:35.663 INFO:tasks.rados.rados.0.smithi107.stdout:1061: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:35.664 INFO:tasks.rados.rados.0.smithi107.stdout:1060: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:35.664 INFO:tasks.rados.rados.0.smithi107.stdout:1063: done (2 left) 2020-06-12T02:37:35.664 INFO:tasks.rados.rados.0.smithi107.stdout:1064: delete oid 7 current snap is 0 2020-06-12T02:37:35.665 INFO:tasks.rados.rados.0.smithi107.stdout:1061: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:35.666 INFO:tasks.rados.rados.0.smithi107.stdout:1060: finishing write tid 5 to smithi10729407-2 2020-06-12T02:37:35.666 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 107 (ObjNum 1059 snap 0 seq_num 1059) dirty exists 2020-06-12T02:37:35.666 INFO:tasks.rados.rados.0.smithi107.stdout:1060: left oid 2 (ObjNum 1059 snap 0 seq_num 1059) 2020-06-12T02:37:35.666 INFO:tasks.rados.rados.0.smithi107.stdout:1061: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:35.671 INFO:tasks.rados.rados.0.smithi107.stdout:1061: finishing write tid 5 to smithi10729407-4 2020-06-12T02:37:35.671 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 192 (ObjNum 1060 snap 0 seq_num 1060) dirty exists 2020-06-12T02:37:35.671 INFO:tasks.rados.rados.0.smithi107.stdout:1061: left oid 4 (ObjNum 1060 snap 0 seq_num 1060) 2020-06-12T02:37:35.672 INFO:tasks.rados.rados.0.smithi107.stdout:1060: done (2 left) 2020-06-12T02:37:35.672 INFO:tasks.rados.rados.0.smithi107.stdout:1061: done (1 left) 2020-06-12T02:37:35.672 INFO:tasks.rados.rados.0.smithi107.stdout:1064: done (0 left) 2020-06-12T02:37:35.672 INFO:tasks.rados.rados.0.smithi107.stdout:1065: delete oid 13 current snap is 0 2020-06-12T02:37:35.673 INFO:tasks.rados.rados.0.smithi107.stdout:1065: done (0 left) 2020-06-12T02:37:35.673 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:37:35.673 INFO:tasks.rados.rados.0.smithi107.stdout:1066: seq_num 1065 ranges {704512=638976,1343488=401408} 2020-06-12T02:37:35.679 INFO:tasks.rados.rados.0.smithi107.stdout:1066: writing smithi10729407-16 from 704512 to 1343488 tid 1 2020-06-12T02:37:35.682 INFO:tasks.rados.rados.0.smithi107.stdout:1066: writing smithi10729407-16 from 1343488 to 1744896 tid 2 2020-06-12T02:37:35.682 INFO:tasks.rados.rados.0.smithi107.stdout:1067: delete oid 19 current snap is 0 2020-06-12T02:37:35.683 INFO:tasks.rados.rados.0.smithi107.stdout:1067: done (1 left) 2020-06-12T02:37:35.684 INFO:tasks.rados.rados.0.smithi107.stdout:1068: delete oid 7 current snap is 0 2020-06-12T02:37:35.686 INFO:tasks.rados.rados.0.smithi107.stdout:1068: done (1 left) 2020-06-12T02:37:35.687 INFO:tasks.rados.rados.0.smithi107.stdout:1069: delete oid 4 current snap is 0 2020-06-12T02:37:35.687 INFO:tasks.rados.rados.0.smithi107.stdout:1066: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:35.688 INFO:tasks.rados.rados.0.smithi107.stdout:1069: done (1 left) 2020-06-12T02:37:35.688 INFO:tasks.rados.rados.0.smithi107.stdout:1070: delete oid 17 current snap is 0 2020-06-12T02:37:35.689 INFO:tasks.rados.rados.0.smithi107.stdout:1066: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:35.689 INFO:tasks.rados.rados.0.smithi107.stdout:1066: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:35.690 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:35.686+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:35.690 INFO:tasks.rados.rados.0.smithi107.stdout:1070: done (1 left) 2020-06-12T02:37:35.690 INFO:tasks.rados.rados.0.smithi107.stdout:1071: delete oid 20 current snap is 0 2020-06-12T02:37:35.691 INFO:tasks.rados.rados.0.smithi107.stdout:1066: finishing write tid 5 to smithi10729407-16 2020-06-12T02:37:35.692 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 189 (ObjNum 1065 snap 0 seq_num 1065) dirty exists 2020-06-12T02:37:35.692 INFO:tasks.rados.rados.0.smithi107.stdout:1066: left oid 16 (ObjNum 1065 snap 0 seq_num 1065) 2020-06-12T02:37:35.692 INFO:tasks.rados.rados.0.smithi107.stdout:1066: done (1 left) 2020-06-12T02:37:35.693 INFO:tasks.rados.rados.0.smithi107.stdout:1071: done (0 left) 2020-06-12T02:37:35.693 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:37:35.693 INFO:tasks.rados.rados.0.smithi107.stdout:1072: seq_num 1071 ranges {0=745472,745472=401408} 2020-06-12T02:37:35.696 INFO:tasks.rados.rados.0.smithi107.stdout:1072: writing smithi10729407-19 from 0 to 745472 tid 1 2020-06-12T02:37:35.699 INFO:tasks.rados.rados.0.smithi107.stdout:1072: writing smithi10729407-19 from 745472 to 1146880 tid 2 2020-06-12T02:37:35.699 INFO:tasks.rados.rados.0.smithi107.stdout:1073: delete oid 18 current snap is 0 2020-06-12T02:37:35.700 INFO:tasks.rados.rados.0.smithi107.stdout:1073: done (1 left) 2020-06-12T02:37:35.701 INFO:tasks.rados.rados.0.smithi107.stdout:1074: delete oid 8 current snap is 0 2020-06-12T02:37:35.703 INFO:tasks.rados.rados.0.smithi107.stdout:1074: done (1 left) 2020-06-12T02:37:35.704 INFO:tasks.rados.rados.0.smithi107.stdout:1075: delete oid 10 current snap is 0 2020-06-12T02:37:35.704 INFO:tasks.rados.rados.0.smithi107.stdout:1072: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:35.704 INFO:tasks.rados.rados.0.smithi107.stdout:1072: finishing write tid 2 to smithi10729407-19 2020-06-12T02:37:35.705 INFO:tasks.rados.rados.0.smithi107.stdout:1072: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:35.706 INFO:tasks.rados.rados.0.smithi107.stdout:1072: finishing write tid 5 to smithi10729407-19 2020-06-12T02:37:35.707 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 174 (ObjNum 1071 snap 0 seq_num 1071) dirty exists 2020-06-12T02:37:35.707 INFO:tasks.rados.rados.0.smithi107.stdout:1072: left oid 19 (ObjNum 1071 snap 0 seq_num 1071) 2020-06-12T02:37:35.707 INFO:tasks.rados.rados.0.smithi107.stdout:1072: done (1 left) 2020-06-12T02:37:35.707 INFO:tasks.rados.rados.0.smithi107.stdout:1075: done (0 left) 2020-06-12T02:37:35.707 INFO:tasks.rados.rados.0.smithi107.stdout:1076: delete oid 10 current snap is 0 2020-06-12T02:37:35.709 INFO:tasks.rados.rados.0.smithi107.stdout:1076: done (0 left) 2020-06-12T02:37:35.709 INFO:tasks.rados.rados.0.smithi107.stdout:1077: delete oid 5 current snap is 0 2020-06-12T02:37:35.710 INFO:tasks.rados.rados.0.smithi107.stdout:1077: done (0 left) 2020-06-12T02:37:35.711 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:37:35.711 INFO:tasks.rados.rados.0.smithi107.stdout:1078: seq_num 1077 ranges {0=442368,442368=434176,876544=368640} 2020-06-12T02:37:35.713 INFO:tasks.rados.rados.0.smithi107.stdout:1078: writing smithi10729407-13 from 0 to 442368 tid 1 2020-06-12T02:37:35.715 INFO:tasks.rados.rados.0.smithi107.stdout:1078: writing smithi10729407-13 from 442368 to 876544 tid 2 2020-06-12T02:37:35.717 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd set noscrub 2020-06-12T02:37:35.718 INFO:tasks.rados.rados.0.smithi107.stdout:1078: writing smithi10729407-13 from 876544 to 1245184 tid 3 2020-06-12T02:37:35.718 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:37:35.718 INFO:tasks.rados.rados.0.smithi107.stdout:1079: seq_num 1078 ranges {0=458752,458752=720896,1179648=81920} 2020-06-12T02:37:35.721 INFO:tasks.rados.rados.0.smithi107.stdout:1079: writing smithi10729407-14 from 0 to 458752 tid 1 2020-06-12T02:37:35.725 INFO:tasks.rados.rados.0.smithi107.stdout:1079: writing smithi10729407-14 from 458752 to 1179648 tid 2 2020-06-12T02:37:35.726 INFO:tasks.rados.rados.0.smithi107.stdout:1079: writing smithi10729407-14 from 1179648 to 1261568 tid 3 2020-06-12T02:37:35.726 INFO:tasks.rados.rados.0.smithi107.stdout:1080: delete oid 12 current snap is 0 2020-06-12T02:37:35.726 INFO:tasks.rados.rados.0.smithi107.stdout:1078: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:35.727 INFO:tasks.rados.rados.0.smithi107.stdout:1078: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:35.729 INFO:tasks.rados.rados.0.smithi107.stdout:1080: done (2 left) 2020-06-12T02:37:35.729 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:37:35.729 INFO:tasks.rados.rados.0.smithi107.stdout:1081: seq_num 1080 ranges {0=491520,491520=540672,1032192=262144} 2020-06-12T02:37:35.732 INFO:tasks.rados.rados.0.smithi107.stdout:1081: writing smithi10729407-12 from 0 to 491520 tid 1 2020-06-12T02:37:35.735 INFO:tasks.rados.rados.0.smithi107.stdout:1081: writing smithi10729407-12 from 491520 to 1032192 tid 2 2020-06-12T02:37:35.737 INFO:tasks.rados.rados.0.smithi107.stdout:1081: writing smithi10729407-12 from 1032192 to 1294336 tid 3 2020-06-12T02:37:35.737 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:37:35.737 INFO:tasks.rados.rados.0.smithi107.stdout:1082: seq_num 1081 ranges {0=507904,507904=425984,933888=376832} 2020-06-12T02:37:35.740 INFO:tasks.rados.rados.0.smithi107.stdout:1082: writing smithi10729407-5 from 0 to 507904 tid 1 2020-06-12T02:37:35.743 INFO:tasks.rados.rados.0.smithi107.stdout:1082: writing smithi10729407-5 from 507904 to 933888 tid 2 2020-06-12T02:37:35.745 INFO:tasks.rados.rados.0.smithi107.stdout:1082: writing smithi10729407-5 from 933888 to 1310720 tid 3 2020-06-12T02:37:35.746 INFO:tasks.rados.rados.0.smithi107.stdout:1083: delete oid 8 current snap is 0 2020-06-12T02:37:35.746 INFO:tasks.rados.rados.0.smithi107.stdout:1078: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:35.746 INFO:tasks.rados.rados.0.smithi107.stdout:1079: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:35.746 INFO:tasks.rados.rados.0.smithi107.stdout:1078: finishing write tid 5 to smithi10729407-13 2020-06-12T02:37:35.746 INFO:tasks.rados.rados.0.smithi107.stdout:1078: finishing write tid 6 to smithi10729407-13 2020-06-12T02:37:35.747 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 224 (ObjNum 1077 snap 0 seq_num 1077) dirty exists 2020-06-12T02:37:35.747 INFO:tasks.rados.rados.0.smithi107.stdout:1078: left oid 13 (ObjNum 1077 snap 0 seq_num 1077) 2020-06-12T02:37:35.747 INFO:tasks.rados.rados.0.smithi107.stdout:1081: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:35.747 INFO:tasks.rados.rados.0.smithi107.stdout:1079: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:35.747 INFO:tasks.rados.rados.0.smithi107.stdout:1079: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:35.748 INFO:tasks.rados.rados.0.smithi107.stdout:1079: finishing write tid 5 to smithi10729407-14 2020-06-12T02:37:35.748 INFO:tasks.rados.rados.0.smithi107.stdout:1079: finishing write tid 6 to smithi10729407-14 2020-06-12T02:37:35.748 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 193 (ObjNum 1078 snap 0 seq_num 1078) dirty exists 2020-06-12T02:37:35.748 INFO:tasks.rados.rados.0.smithi107.stdout:1079: left oid 14 (ObjNum 1078 snap 0 seq_num 1078) 2020-06-12T02:37:35.748 INFO:tasks.rados.rados.0.smithi107.stdout:1081: finishing write tid 2 to smithi10729407-12 2020-06-12T02:37:35.749 INFO:tasks.rados.rados.0.smithi107.stdout:1081: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:35.749 INFO:tasks.rados.rados.0.smithi107.stdout:1081: finishing write tid 5 to smithi10729407-12 2020-06-12T02:37:35.750 INFO:tasks.rados.rados.0.smithi107.stdout:1082: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:35.750 INFO:tasks.rados.rados.0.smithi107.stdout:1078: done (4 left) 2020-06-12T02:37:35.750 INFO:tasks.rados.rados.0.smithi107.stdout:1079: done (3 left) 2020-06-12T02:37:35.750 INFO:tasks.rados.rados.0.smithi107.stdout:1083: done (2 left) 2020-06-12T02:37:35.751 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:35.751 INFO:tasks.rados.rados.0.smithi107.stdout:1084: seq_num 1083 ranges {942080=540672,1482752=589824,2072576=212992} 2020-06-12T02:37:35.758 INFO:tasks.rados.rados.0.smithi107.stdout:1084: writing smithi10729407-2 from 942080 to 1482752 tid 1 2020-06-12T02:37:35.762 INFO:tasks.rados.rados.0.smithi107.stdout:1084: writing smithi10729407-2 from 1482752 to 2072576 tid 2 2020-06-12T02:37:35.763 INFO:tasks.rados.rados.0.smithi107.stdout:1084: writing smithi10729407-2 from 2072576 to 2285568 tid 3 2020-06-12T02:37:35.764 INFO:tasks.rados.rados.0.smithi107.stdout:1085: delete oid 18 current snap is 0 2020-06-12T02:37:35.764 INFO:tasks.rados.rados.0.smithi107.stdout:1082: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:35.764 INFO:tasks.rados.rados.0.smithi107.stdout:1081: finishing write tid 6 to smithi10729407-12 2020-06-12T02:37:35.764 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 179 (ObjNum 1080 snap 0 seq_num 1080) dirty exists 2020-06-12T02:37:35.764 INFO:tasks.rados.rados.0.smithi107.stdout:1081: left oid 12 (ObjNum 1080 snap 0 seq_num 1080) 2020-06-12T02:37:35.765 INFO:tasks.rados.rados.0.smithi107.stdout:1082: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:35.765 INFO:tasks.rados.rados.0.smithi107.stdout:1082: finishing write tid 5 to smithi10729407-5 2020-06-12T02:37:35.765 INFO:tasks.rados.rados.0.smithi107.stdout:1082: finishing write tid 6 to smithi10729407-5 2020-06-12T02:37:35.765 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 95 (ObjNum 1081 snap 0 seq_num 1081) dirty exists 2020-06-12T02:37:35.765 INFO:tasks.rados.rados.0.smithi107.stdout:1082: left oid 5 (ObjNum 1081 snap 0 seq_num 1081) 2020-06-12T02:37:35.766 INFO:tasks.rados.rados.0.smithi107.stdout:1084: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:35.766 INFO:tasks.rados.rados.0.smithi107.stdout:1081: done (3 left) 2020-06-12T02:37:35.766 INFO:tasks.rados.rados.0.smithi107.stdout:1082: done (2 left) 2020-06-12T02:37:35.767 INFO:tasks.rados.rados.0.smithi107.stdout:1085: done (1 left) 2020-06-12T02:37:35.767 INFO:tasks.rados.rados.0.smithi107.stdout:1086: delete oid 13 current snap is 0 2020-06-12T02:37:35.771 INFO:tasks.rados.rados.0.smithi107.stdout:1084: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:35.772 INFO:tasks.rados.rados.0.smithi107.stdout:1084: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:35.772 INFO:tasks.rados.rados.0.smithi107.stdout:1084: finishing write tid 5 to smithi10729407-2 2020-06-12T02:37:35.772 INFO:tasks.rados.rados.0.smithi107.stdout:1086: done (1 left) 2020-06-12T02:37:35.773 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:37:35.773 INFO:tasks.rados.rados.0.smithi107.stdout:1087: seq_num 1086 ranges {1744896=598016,2342912=647168,2990080=155648} 2020-06-12T02:37:35.785 INFO:tasks.rados.rados.0.smithi107.stdout:1087: writing smithi10729407-16 from 1744896 to 2342912 tid 1 2020-06-12T02:37:35.789 INFO:tasks.rados.rados.0.smithi107.stdout:1087: writing smithi10729407-16 from 2342912 to 2990080 tid 2 2020-06-12T02:37:35.790 INFO:tasks.rados.rados.0.smithi107.stdout:1087: writing smithi10729407-16 from 2990080 to 3145728 tid 3 2020-06-12T02:37:35.791 INFO:tasks.rados.rados.0.smithi107.stdout:1088: delete oid 3 current snap is 0 2020-06-12T02:37:35.791 INFO:tasks.rados.rados.0.smithi107.stdout:1084: finishing write tid 6 to smithi10729407-2 2020-06-12T02:37:35.791 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 111 (ObjNum 1083 snap 0 seq_num 1083) dirty exists 2020-06-12T02:37:35.791 INFO:tasks.rados.rados.0.smithi107.stdout:1084: left oid 2 (ObjNum 1083 snap 0 seq_num 1083) 2020-06-12T02:37:35.792 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:37:35.788+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:37:35.792 INFO:tasks.rados.rados.0.smithi107.stdout:1087: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:35.796 INFO:tasks.rados.rados.0.smithi107.stdout:1084: done (2 left) 2020-06-12T02:37:35.797 INFO:tasks.rados.rados.0.smithi107.stdout:1088: done (1 left) 2020-06-12T02:37:35.797 INFO:tasks.rados.rados.0.smithi107.stdout:1089: delete oid 3 current snap is 0 2020-06-12T02:37:35.798 INFO:tasks.rados.rados.0.smithi107.stdout:1087: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:35.799 INFO:tasks.rados.rados.0.smithi107.stdout:1087: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:35.800 INFO:tasks.rados.rados.0.smithi107.stdout:1087: finishing write tid 5 to smithi10729407-16 2020-06-12T02:37:35.801 INFO:tasks.rados.rados.0.smithi107.stdout:1089: done (1 left) 2020-06-12T02:37:35.801 INFO:tasks.rados.rados.0.smithi107.stdout:1090: delete oid 7 current snap is 0 2020-06-12T02:37:35.802 INFO:tasks.rados.rados.0.smithi107.stdout:1087: finishing write tid 6 to smithi10729407-16 2020-06-12T02:37:35.802 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 194 (ObjNum 1086 snap 0 seq_num 1086) dirty exists 2020-06-12T02:37:35.802 INFO:tasks.rados.rados.0.smithi107.stdout:1087: left oid 16 (ObjNum 1086 snap 0 seq_num 1086) 2020-06-12T02:37:35.803 INFO:tasks.rados.rados.0.smithi107.stdout:1087: done (1 left) 2020-06-12T02:37:35.803 INFO:tasks.rados.rados.0.smithi107.stdout:1090: done (0 left) 2020-06-12T02:37:35.804 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:37:35.804 INFO:tasks.rados.rados.0.smithi107.stdout:1091: seq_num 1090 ranges {0=663552,663552=630784,1294336=172032} 2020-06-12T02:37:35.805 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:35.806 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:35.806 INFO:tasks.rados.rados.0.smithi107.stdout:1091: writing smithi10729407-4 from 0 to 663552 tid 1 2020-06-12T02:37:35.810 INFO:tasks.rados.rados.0.smithi107.stdout:1091: writing smithi10729407-4 from 663552 to 1294336 tid 2 2020-06-12T02:37:35.812 INFO:tasks.rados.rados.0.smithi107.stdout:1091: writing smithi10729407-4 from 1294336 to 1466368 tid 3 2020-06-12T02:37:35.812 INFO:tasks.rados.rados.0.smithi107.stdout:1092: delete oid 2 current snap is 0 2020-06-12T02:37:35.814 INFO:tasks.rados.rados.0.smithi107.stdout:1091: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:35.814 INFO:tasks.rados.rados.0.smithi107.stdout:1092: done (1 left) 2020-06-12T02:37:35.814 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:37:35.815 INFO:tasks.rados.rados.0.smithi107.stdout:1093: seq_num 1092 ranges {1310720=696320,2007040=802816} 2020-06-12T02:37:35.816 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-06-12T02:37:35.825 INFO:tasks.rados.rados.0.smithi107.stdout:1093: writing smithi10729407-5 from 1310720 to 2007040 tid 1 2020-06-12T02:37:35.830 INFO:tasks.rados.rados.0.smithi107.stdout:1093: writing smithi10729407-5 from 2007040 to 2809856 tid 2 2020-06-12T02:37:35.830 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:37:35.830 INFO:tasks.rados.rados.0.smithi107.stdout:1094: seq_num 1093 ranges {1261568=712704,1974272=688128,2662400=114688} 2020-06-12T02:37:35.841 INFO:tasks.rados.rados.0.smithi107.stdout:1094: writing smithi10729407-14 from 1261568 to 1974272 tid 1 2020-06-12T02:37:35.845 INFO:tasks.rados.rados.0.smithi107.stdout:1094: writing smithi10729407-14 from 1974272 to 2662400 tid 2 2020-06-12T02:37:35.846 INFO:tasks.rados.rados.0.smithi107.stdout:1094: writing smithi10729407-14 from 2662400 to 2777088 tid 3 2020-06-12T02:37:35.846 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:37:35.846 INFO:tasks.rados.rados.0.smithi107.stdout:1095: seq_num 1094 ranges {1294336=729088,2023424=573440,2596864=229376} 2020-06-12T02:37:35.857 INFO:tasks.rados.rados.0.smithi107.stdout:1095: writing smithi10729407-12 from 1294336 to 2023424 tid 1 2020-06-12T02:37:35.861 INFO:tasks.rados.rados.0.smithi107.stdout:1095: writing smithi10729407-12 from 2023424 to 2596864 tid 2 2020-06-12T02:37:35.862 INFO:tasks.rados.rados.0.smithi107.stdout:1095: writing smithi10729407-12 from 2596864 to 2826240 tid 3 2020-06-12T02:37:35.863 INFO:tasks.rados.rados.0.smithi107.stdout:1096: delete oid 7 current snap is 0 2020-06-12T02:37:35.863 INFO:tasks.rados.rados.0.smithi107.stdout:1091: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:35.863 INFO:tasks.rados.rados.0.smithi107.stdout:1091: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:35.863 INFO:tasks.rados.rados.0.smithi107.stdout:1091: finishing write tid 5 to smithi10729407-4 2020-06-12T02:37:35.863 INFO:tasks.rados.rados.0.smithi107.stdout:1091: finishing write tid 6 to smithi10729407-4 2020-06-12T02:37:35.864 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 198 (ObjNum 1090 snap 0 seq_num 1090) dirty exists 2020-06-12T02:37:35.864 INFO:tasks.rados.rados.0.smithi107.stdout:1091: left oid 4 (ObjNum 1090 snap 0 seq_num 1090) 2020-06-12T02:37:35.864 INFO:tasks.rados.rados.0.smithi107.stdout:1093: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:35.864 INFO:tasks.rados.rados.0.smithi107.stdout:1093: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:35.864 INFO:tasks.rados.rados.0.smithi107.stdout:1093: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:35.865 INFO:tasks.rados.rados.0.smithi107.stdout:1093: finishing write tid 5 to smithi10729407-5 2020-06-12T02:37:35.865 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 98 (ObjNum 1092 snap 0 seq_num 1092) dirty exists 2020-06-12T02:37:35.865 INFO:tasks.rados.rados.0.smithi107.stdout:1093: left oid 5 (ObjNum 1092 snap 0 seq_num 1092) 2020-06-12T02:37:35.865 INFO:tasks.rados.rados.0.smithi107.stdout:1094: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:35.865 INFO:tasks.rados.rados.0.smithi107.stdout:1094: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:35.866 INFO:tasks.rados.rados.0.smithi107.stdout:1094: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:35.866 INFO:tasks.rados.rados.0.smithi107.stdout:1094: finishing write tid 5 to smithi10729407-14 2020-06-12T02:37:35.866 INFO:tasks.rados.rados.0.smithi107.stdout:1094: finishing write tid 6 to smithi10729407-14 2020-06-12T02:37:35.866 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 197 (ObjNum 1093 snap 0 seq_num 1093) dirty exists 2020-06-12T02:37:35.866 INFO:tasks.rados.rados.0.smithi107.stdout:1094: left oid 14 (ObjNum 1093 snap 0 seq_num 1093) 2020-06-12T02:37:35.872 INFO:tasks.rados.rados.0.smithi107.stdout:1095: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:35.872 INFO:tasks.rados.rados.0.smithi107.stdout:1091: done (4 left) 2020-06-12T02:37:35.872 INFO:tasks.rados.rados.0.smithi107.stdout:1093: done (3 left) 2020-06-12T02:37:35.872 INFO:tasks.rados.rados.0.smithi107.stdout:1094: done (2 left) 2020-06-12T02:37:35.872 INFO:tasks.rados.rados.0.smithi107.stdout:1096: done (1 left) 2020-06-12T02:37:35.873 INFO:tasks.rados.rados.0.smithi107.stdout:1097: delete oid 2 current snap is 0 2020-06-12T02:37:35.873 INFO:tasks.rados.rados.0.smithi107.stdout:1097: done (1 left) 2020-06-12T02:37:35.873 INFO:tasks.rados.rados.0.smithi107.stdout:1098: delete oid 9 current snap is 0 2020-06-12T02:37:35.873 INFO:tasks.rados.rados.0.smithi107.stdout:1098: done (1 left) 2020-06-12T02:37:35.873 INFO:tasks.rados.rados.0.smithi107.stdout:1099: delete oid 10 current snap is 0 2020-06-12T02:37:35.874 INFO:tasks.rados.rados.0.smithi107.stdout:1095: finishing write tid 2 to smithi10729407-12 2020-06-12T02:37:35.874 INFO:tasks.rados.rados.0.smithi107.stdout:1095: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:35.874 INFO:tasks.rados.rados.0.smithi107.stdout:1099: done (1 left) 2020-06-12T02:37:35.875 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:37:35.875 INFO:tasks.rados.rados.0.smithi107.stdout:1100: seq_num 1099 ranges {0=409600,409600=442368,851968=720896,1572864=40960} 2020-06-12T02:37:35.875 INFO:tasks.rados.rados.0.smithi107.stdout:1100: writing smithi10729407-15 from 0 to 409600 tid 1 2020-06-12T02:37:35.877 INFO:tasks.rados.rados.0.smithi107.stdout:1100: writing smithi10729407-15 from 409600 to 851968 tid 2 2020-06-12T02:37:35.882 INFO:tasks.rados.rados.0.smithi107.stdout:1100: writing smithi10729407-15 from 851968 to 1572864 tid 3 2020-06-12T02:37:35.883 INFO:tasks.rados.rados.0.smithi107.stdout:1100: writing smithi10729407-15 from 1572864 to 1613824 tid 4 2020-06-12T02:37:35.883 INFO:tasks.rados.rados.0.smithi107.stdout:1101: delete oid 6 current snap is 0 2020-06-12T02:37:35.883 INFO:tasks.rados.rados.0.smithi107.stdout:1095: finishing write tid 5 to smithi10729407-12 2020-06-12T02:37:35.883 INFO:tasks.rados.rados.0.smithi107.stdout:1095: finishing write tid 6 to smithi10729407-12 2020-06-12T02:37:35.883 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 183 (ObjNum 1094 snap 0 seq_num 1094) dirty exists 2020-06-12T02:37:35.884 INFO:tasks.rados.rados.0.smithi107.stdout:1095: left oid 12 (ObjNum 1094 snap 0 seq_num 1094) 2020-06-12T02:37:35.884 INFO:tasks.rados.rados.0.smithi107.stdout:1100: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:35.886 INFO:tasks.rados.rados.0.smithi107.stdout:1095: done (2 left) 2020-06-12T02:37:35.886 INFO:tasks.rados.rados.0.smithi107.stdout:1101: done (1 left) 2020-06-12T02:37:35.886 INFO:tasks.rados.rados.0.smithi107.stdout:1102: delete oid 7 current snap is 0 2020-06-12T02:37:35.888 INFO:tasks.rados.rados.0.smithi107.stdout:1100: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:35.888 INFO:tasks.rados.rados.0.smithi107.stdout:1102: done (1 left) 2020-06-12T02:37:35.889 INFO:tasks.rados.rados.0.smithi107.stdout:1103: delete oid 3 current snap is 0 2020-06-12T02:37:35.889 INFO:tasks.rados.rados.0.smithi107.stdout:1100: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:35.889 INFO:tasks.rados.rados.0.smithi107.stdout:1100: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:35.890 INFO:tasks.rados.rados.0.smithi107.stdout:1100: finishing write tid 6 to smithi10729407-15 2020-06-12T02:37:35.891 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:35.888+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:35.892 INFO:tasks.rados.rados.0.smithi107.stdout:1103: done (1 left) 2020-06-12T02:37:35.893 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:37:35.893 INFO:tasks.rados.rados.0.smithi107.stdout:1104: seq_num 1103 ranges {0=475136,475136=425984,901120=696320,1597440=81920} 2020-06-12T02:37:35.894 INFO:tasks.rados.rados.0.smithi107.stdout:1104: writing smithi10729407-7 from 0 to 475136 tid 1 2020-06-12T02:37:35.897 INFO:tasks.rados.rados.0.smithi107.stdout:1104: writing smithi10729407-7 from 475136 to 901120 tid 2 2020-06-12T02:37:35.901 INFO:tasks.rados.rados.0.smithi107.stdout:1104: writing smithi10729407-7 from 901120 to 1597440 tid 3 2020-06-12T02:37:35.904 INFO:tasks.rados.rados.0.smithi107.stdout:1104: writing smithi10729407-7 from 1597440 to 1679360 tid 4 2020-06-12T02:37:35.905 INFO:tasks.rados.rados.0.smithi107.stdout:1105: delete oid 20 current snap is 0 2020-06-12T02:37:35.905 INFO:tasks.rados.rados.0.smithi107.stdout:1100: finishing write tid 7 to smithi10729407-15 2020-06-12T02:37:35.905 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 185 (ObjNum 1099 snap 0 seq_num 1099) dirty exists 2020-06-12T02:37:35.906 INFO:tasks.rados.rados.0.smithi107.stdout:1100: left oid 15 (ObjNum 1099 snap 0 seq_num 1099) 2020-06-12T02:37:35.906 INFO:tasks.rados.rados.0.smithi107.stdout:1104: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:35.906 INFO:tasks.rados.rados.0.smithi107.stdout:1100: done (2 left) 2020-06-12T02:37:35.907 INFO:tasks.rados.rados.0.smithi107.stdout:1105: done (1 left) 2020-06-12T02:37:35.907 INFO:tasks.rados.rados.0.smithi107.stdout:1106: delete oid 17 current snap is 0 2020-06-12T02:37:35.910 INFO:tasks.rados.rados.0.smithi107.stdout:1106: done (1 left) 2020-06-12T02:37:35.911 INFO:tasks.rados.rados.0.smithi107.stdout:1107: delete oid 14 current snap is 0 2020-06-12T02:37:35.911 INFO:tasks.rados.rados.0.smithi107.stdout:1104: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:35.912 INFO:tasks.rados.rados.0.smithi107.stdout:1104: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:35.912 INFO:tasks.rados.rados.0.smithi107.stdout:1104: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:35.913 INFO:tasks.rados.rados.0.smithi107.stdout:1104: finishing write tid 6 to smithi10729407-7 2020-06-12T02:37:35.913 INFO:tasks.rados.rados.0.smithi107.stdout:1107: done (1 left) 2020-06-12T02:37:35.913 INFO:tasks.rados.rados.0.smithi107.stdout:1108: delete oid 17 current snap is 0 2020-06-12T02:37:35.914 INFO:tasks.rados.rados.0.smithi107.stdout:1104: finishing write tid 7 to smithi10729407-7 2020-06-12T02:37:35.915 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 233 (ObjNum 1103 snap 0 seq_num 1103) dirty exists 2020-06-12T02:37:35.915 INFO:tasks.rados.rados.0.smithi107.stdout:1104: left oid 7 (ObjNum 1103 snap 0 seq_num 1103) 2020-06-12T02:37:35.915 INFO:tasks.rados.rados.0.smithi107.stdout:1104: done (1 left) 2020-06-12T02:37:35.916 INFO:tasks.rados.rados.0.smithi107.stdout:1108: done (0 left) 2020-06-12T02:37:35.916 INFO:tasks.rados.rados.0.smithi107.stdout:1109: delete oid 19 current snap is 0 2020-06-12T02:37:35.917 INFO:tasks.rados.rados.0.smithi107.stdout:1109: done (0 left) 2020-06-12T02:37:35.918 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:37:35.918 INFO:tasks.rados.rados.0.smithi107.stdout:1110: seq_num 1109 ranges {0=581632,581632=532480,1114112=647168,1761280=24576} 2020-06-12T02:37:35.921 INFO:tasks.rados.rados.0.smithi107.stdout:1110: writing smithi10729407-17 from 0 to 581632 tid 1 2020-06-12T02:37:35.924 INFO:tasks.rados.rados.0.smithi107.stdout:1110: writing smithi10729407-17 from 581632 to 1114112 tid 2 2020-06-12T02:37:35.928 INFO:tasks.rados.rados.0.smithi107.stdout:1110: writing smithi10729407-17 from 1114112 to 1761280 tid 3 2020-06-12T02:37:35.929 INFO:tasks.rados.rados.0.smithi107.stdout:1110: writing smithi10729407-17 from 1761280 to 1785856 tid 4 2020-06-12T02:37:35.929 INFO:tasks.rados.rados.0.smithi107.stdout:1111: delete oid 14 current snap is 0 2020-06-12T02:37:35.929 INFO:tasks.rados.rados.0.smithi107.stdout:1110: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:35.929 INFO:tasks.rados.rados.0.smithi107.stdout:1110: finishing write tid 2 to smithi10729407-17 2020-06-12T02:37:35.930 INFO:tasks.rados.rados.0.smithi107.stdout:1111: done (1 left) 2020-06-12T02:37:35.931 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:37:35.931 INFO:tasks.rados.rados.0.smithi107.stdout:1112: seq_num 1111 ranges {0=614400,614400=753664,1368064=450560} 2020-06-12T02:37:35.933 INFO:tasks.rados.rados.0.smithi107.stdout:1112: writing smithi10729407-6 from 0 to 614400 tid 1 2020-06-12T02:37:35.939 INFO:tasks.rados.rados.0.smithi107.stdout:1112: writing smithi10729407-6 from 614400 to 1368064 tid 2 2020-06-12T02:37:35.942 INFO:tasks.rados.rados.0.smithi107.stdout:1112: writing smithi10729407-6 from 1368064 to 1818624 tid 3 2020-06-12T02:37:35.942 INFO:tasks.rados.rados.0.smithi107.stdout:1113: delete oid 3 current snap is 0 2020-06-12T02:37:35.943 INFO:tasks.rados.rados.0.smithi107.stdout:1110: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:35.943 INFO:tasks.rados.rados.0.smithi107.stdout:1110: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:35.943 INFO:tasks.rados.rados.0.smithi107.stdout:1110: finishing write tid 6 to smithi10729407-17 2020-06-12T02:37:35.944 INFO:tasks.rados.rados.0.smithi107.stdout:1110: finishing write tid 7 to smithi10729407-17 2020-06-12T02:37:35.944 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 205 (ObjNum 1109 snap 0 seq_num 1109) dirty exists 2020-06-12T02:37:35.945 INFO:tasks.rados.rados.0.smithi107.stdout:1110: left oid 17 (ObjNum 1109 snap 0 seq_num 1109) 2020-06-12T02:37:35.946 INFO:tasks.rados.rados.0.smithi107.stdout:1112: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:35.947 INFO:tasks.rados.rados.0.smithi107.stdout:1110: done (2 left) 2020-06-12T02:37:35.947 INFO:tasks.rados.rados.0.smithi107.stdout:1113: done (1 left) 2020-06-12T02:37:35.947 INFO:tasks.rados.rados.0.smithi107.stdout:1114: delete oid 5 current snap is 0 2020-06-12T02:37:35.948 INFO:tasks.rados.rados.0.smithi107.stdout:1114: done (1 left) 2020-06-12T02:37:35.948 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:37:35.948 INFO:tasks.rados.rados.0.smithi107.stdout:1115: seq_num 1114 ranges {1466368=663552,2129920=802816,2932736=401408} 2020-06-12T02:37:35.960 INFO:tasks.rados.rados.0.smithi107.stdout:1115: writing smithi10729407-4 from 1466368 to 2129920 tid 1 2020-06-12T02:37:35.966 INFO:tasks.rados.rados.0.smithi107.stdout:1115: writing smithi10729407-4 from 2129920 to 2932736 tid 2 2020-06-12T02:37:35.969 INFO:tasks.rados.rados.0.smithi107.stdout:1115: writing smithi10729407-4 from 2932736 to 3334144 tid 3 2020-06-12T02:37:35.969 INFO:tasks.rados.rados.0.smithi107.stdout:1116: delete oid 3 current snap is 0 2020-06-12T02:37:35.970 INFO:tasks.rados.rados.0.smithi107.stdout:1112: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:35.970 INFO:tasks.rados.rados.0.smithi107.stdout:1112: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:35.970 INFO:tasks.rados.rados.0.smithi107.stdout:1112: finishing write tid 5 to smithi10729407-6 2020-06-12T02:37:35.970 INFO:tasks.rados.rados.0.smithi107.stdout:1112: finishing write tid 6 to smithi10729407-6 2020-06-12T02:37:35.971 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 226 (ObjNum 1111 snap 0 seq_num 1111) dirty exists 2020-06-12T02:37:35.971 INFO:tasks.rados.rados.0.smithi107.stdout:1112: left oid 6 (ObjNum 1111 snap 0 seq_num 1111) 2020-06-12T02:37:35.971 INFO:tasks.rados.rados.0.smithi107.stdout:1115: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:35.972 INFO:tasks.rados.rados.0.smithi107.stdout:1112: done (2 left) 2020-06-12T02:37:35.972 INFO:tasks.rados.rados.0.smithi107.stdout:1116: done (1 left) 2020-06-12T02:37:35.972 INFO:tasks.rados.rados.0.smithi107.stdout:1117: delete oid 2 current snap is 0 2020-06-12T02:37:35.973 INFO:tasks.rados.rados.0.smithi107.stdout:1117: done (1 left) 2020-06-12T02:37:35.973 INFO:tasks.rados.rados.0.smithi107.stdout:1118: delete oid 5 current snap is 0 2020-06-12T02:37:35.974 INFO:tasks.rados.rados.0.smithi107.stdout:1115: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:35.974 INFO:tasks.rados.rados.0.smithi107.stdout:1115: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:35.978 INFO:tasks.rados.rados.0.smithi107.stdout:1115: finishing write tid 5 to smithi10729407-4 2020-06-12T02:37:35.979 INFO:tasks.rados.rados.0.smithi107.stdout:1118: done (1 left) 2020-06-12T02:37:35.979 INFO:tasks.rados.rados.0.smithi107.stdout:1119: delete oid 9 current snap is 0 2020-06-12T02:37:35.979 INFO:tasks.rados.rados.0.smithi107.stdout:1115: finishing write tid 6 to smithi10729407-4 2020-06-12T02:37:35.980 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 209 (ObjNum 1114 snap 0 seq_num 1114) dirty exists 2020-06-12T02:37:35.980 INFO:tasks.rados.rados.0.smithi107.stdout:1115: left oid 4 (ObjNum 1114 snap 0 seq_num 1114) 2020-06-12T02:37:35.982 INFO:tasks.rados.rados.0.smithi107.stdout:1115: done (1 left) 2020-06-12T02:37:35.982 INFO:tasks.rados.rados.0.smithi107.stdout:1119: done (0 left) 2020-06-12T02:37:35.983 INFO:tasks.rados.rados.0.smithi107.stdout:1120: delete oid 20 current snap is 0 2020-06-12T02:37:35.983 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:35.983 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:35.984 INFO:tasks.rados.rados.0.smithi107.stdout:1120: done (0 left) 2020-06-12T02:37:35.984 INFO:tasks.rados.rados.0.smithi107.stdout:1121: delete oid 6 current snap is 0 2020-06-12T02:37:35.985 INFO:tasks.rados.rados.0.smithi107.stdout:1121: done (0 left) 2020-06-12T02:37:35.986 INFO:tasks.rados.rados.0.smithi107.stdout:1122: delete oid 3 current snap is 0 2020-06-12T02:37:35.987 INFO:tasks.rados.rados.0.smithi107.stdout:1122: done (0 left) 2020-06-12T02:37:35.987 INFO:tasks.rados.rados.0.smithi107.stdout:1123: delete oid 15 current snap is 0 2020-06-12T02:37:35.990 INFO:tasks.rados.rados.0.smithi107.stdout:1123: done (0 left) 2020-06-12T02:37:35.990 INFO:tasks.rados.rados.0.smithi107.stdout:1124: delete oid 9 current snap is 0 2020-06-12T02:37:35.992 INFO:tasks.rados.rados.0.smithi107.stdout:1124: done (0 left) 2020-06-12T02:37:35.992 INFO:tasks.rados.rados.0.smithi107.stdout:1125: delete oid 4 current snap is 0 2020-06-12T02:37:35.993 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:37:35.989+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:37:35.994 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-06-12T02:37:35.995 INFO:tasks.rados.rados.0.smithi107.stdout:1125: done (0 left) 2020-06-12T02:37:35.995 INFO:tasks.rados.rados.0.smithi107.stdout:1126: delete oid 14 current snap is 0 2020-06-12T02:37:35.996 INFO:tasks.rados.rados.0.smithi107.stdout:1126: done (0 left) 2020-06-12T02:37:35.996 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:37:35.996 INFO:tasks.rados.rados.0.smithi107.stdout:1127: seq_num 1126 ranges {0=466944,466944=720896,1187840=778240,1966080=106496} 2020-06-12T02:37:35.998 INFO:tasks.rados.rados.0.smithi107.stdout:1127: writing smithi10729407-13 from 0 to 466944 tid 1 2020-06-12T02:37:36.002 INFO:tasks.rados.rados.0.smithi107.stdout:1127: writing smithi10729407-13 from 466944 to 1187840 tid 2 2020-06-12T02:37:36.007 INFO:tasks.rados.rados.0.smithi107.stdout:1127: writing smithi10729407-13 from 1187840 to 1966080 tid 3 2020-06-12T02:37:36.008 INFO:tasks.rados.rados.0.smithi107.stdout:1127: writing smithi10729407-13 from 1966080 to 2072576 tid 4 2020-06-12T02:37:36.009 INFO:tasks.rados.rados.0.smithi107.stdout:1128: delete oid 5 current snap is 0 2020-06-12T02:37:36.009 INFO:tasks.rados.rados.0.smithi107.stdout:1127: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:36.011 INFO:tasks.rados.rados.0.smithi107.stdout:1128: done (1 left) 2020-06-12T02:37:36.011 INFO:tasks.rados.rados.0.smithi107.stdout:1129: delete oid 8 current snap is 0 2020-06-12T02:37:36.012 INFO:tasks.rados.rados.0.smithi107.stdout:1127: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:36.013 INFO:tasks.rados.rados.0.smithi107.stdout:1129: done (1 left) 2020-06-12T02:37:36.013 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:37:36.014 INFO:tasks.rados.rados.0.smithi107.stdout:1130: seq_num 1129 ranges {1679360=516096,2195456=770048,2965504=598016,3563520=237568} 2020-06-12T02:37:36.026 INFO:tasks.rados.rados.0.smithi107.stdout:1130: writing smithi10729407-7 from 1679360 to 2195456 tid 1 2020-06-12T02:37:36.031 INFO:tasks.rados.rados.0.smithi107.stdout:1130: writing smithi10729407-7 from 2195456 to 2965504 tid 2 2020-06-12T02:37:36.035 INFO:tasks.rados.rados.0.smithi107.stdout:1130: writing smithi10729407-7 from 2965504 to 3563520 tid 3 2020-06-12T02:37:36.037 INFO:tasks.rados.rados.0.smithi107.stdout:1130: writing smithi10729407-7 from 3563520 to 3801088 tid 4 2020-06-12T02:37:36.037 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:37:36.037 INFO:tasks.rados.rados.0.smithi107.stdout:1131: seq_num 1130 ranges {876544=532480,1409024=655360,2064384=753664,2818048=196608} 2020-06-12T02:37:36.046 INFO:tasks.rados.rados.0.smithi107.stdout:1131: writing smithi10729407-11 from 876544 to 1409024 tid 1 2020-06-12T02:37:36.049 INFO:tasks.rados.rados.0.smithi107.stdout:1131: writing smithi10729407-11 from 1409024 to 2064384 tid 2 2020-06-12T02:37:36.055 INFO:tasks.rados.rados.0.smithi107.stdout:1131: writing smithi10729407-11 from 2064384 to 2818048 tid 3 2020-06-12T02:37:36.056 INFO:tasks.rados.rados.0.smithi107.stdout:1131: writing smithi10729407-11 from 2818048 to 3014656 tid 4 2020-06-12T02:37:36.056 INFO:tasks.rados.rados.0.smithi107.stdout:1132: delete oid 6 current snap is 0 2020-06-12T02:37:36.059 INFO:tasks.rados.rados.0.smithi107.stdout:1127: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:36.060 INFO:tasks.rados.rados.0.smithi107.stdout:1127: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:36.061 INFO:tasks.rados.rados.0.smithi107.stdout:1127: finishing write tid 6 to smithi10729407-13 2020-06-12T02:37:36.061 INFO:tasks.rados.rados.0.smithi107.stdout:1127: finishing write tid 7 to smithi10729407-13 2020-06-12T02:37:36.061 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 238 (ObjNum 1126 snap 0 seq_num 1126) dirty exists 2020-06-12T02:37:36.061 INFO:tasks.rados.rados.0.smithi107.stdout:1127: left oid 13 (ObjNum 1126 snap 0 seq_num 1126) 2020-06-12T02:37:36.061 INFO:tasks.rados.rados.0.smithi107.stdout:1130: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:36.062 INFO:tasks.rados.rados.0.smithi107.stdout:1130: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:36.062 INFO:tasks.rados.rados.0.smithi107.stdout:1130: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:36.062 INFO:tasks.rados.rados.0.smithi107.stdout:1130: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:36.062 INFO:tasks.rados.rados.0.smithi107.stdout:1130: finishing write tid 6 to smithi10729407-7 2020-06-12T02:37:36.062 INFO:tasks.rados.rados.0.smithi107.stdout:1131: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:36.063 INFO:tasks.rados.rados.0.smithi107.stdout:1130: finishing write tid 7 to smithi10729407-7 2020-06-12T02:37:36.063 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 243 (ObjNum 1129 snap 0 seq_num 1129) dirty exists 2020-06-12T02:37:36.063 INFO:tasks.rados.rados.0.smithi107.stdout:1130: left oid 7 (ObjNum 1129 snap 0 seq_num 1129) 2020-06-12T02:37:36.064 INFO:tasks.rados.rados.0.smithi107.stdout:1131: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:36.064 INFO:tasks.rados.rados.0.smithi107.stdout:1131: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:36.065 INFO:tasks.rados.rados.0.smithi107.stdout:1131: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:36.065 INFO:tasks.rados.rados.0.smithi107.stdout:1131: finishing write tid 6 to smithi10729407-11 2020-06-12T02:37:36.065 INFO:tasks.rados.rados.0.smithi107.stdout:1127: done (3 left) 2020-06-12T02:37:36.065 INFO:tasks.rados.rados.0.smithi107.stdout:1130: done (2 left) 2020-06-12T02:37:36.066 INFO:tasks.rados.rados.0.smithi107.stdout:1132: done (1 left) 2020-06-12T02:37:36.066 INFO:tasks.rados.rados.0.smithi107.stdout:1133: delete oid 5 current snap is 0 2020-06-12T02:37:36.068 INFO:tasks.rados.rados.0.smithi107.stdout:1131: finishing write tid 7 to smithi10729407-11 2020-06-12T02:37:36.068 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 205 (ObjNum 1130 snap 0 seq_num 1130) dirty exists 2020-06-12T02:37:36.068 INFO:tasks.rados.rados.0.smithi107.stdout:1131: left oid 11 (ObjNum 1130 snap 0 seq_num 1130) 2020-06-12T02:37:36.068 INFO:tasks.rados.rados.0.smithi107.stdout:1131: done (1 left) 2020-06-12T02:37:36.068 INFO:tasks.rados.rados.0.smithi107.stdout:1133: done (0 left) 2020-06-12T02:37:36.069 INFO:tasks.rados.rados.0.smithi107.stdout:1134: delete oid 16 current snap is 0 2020-06-12T02:37:36.069 INFO:tasks.rados.rados.0.smithi107.stdout:1134: done (0 left) 2020-06-12T02:37:36.069 INFO:tasks.rados.rados.0.smithi107.stdout:1135: delete oid 17 current snap is 0 2020-06-12T02:37:36.074 INFO:tasks.rados.rados.0.smithi107.stdout:1135: done (0 left) 2020-06-12T02:37:36.075 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:37:36.075 INFO:tasks.rados.rados.0.smithi107.stdout:1136: seq_num 1135 ranges {0=614400,614400=524288,1138688=540672,1679360=483328,2162688=57344} 2020-06-12T02:37:36.075 INFO:tasks.rados.rados.0.smithi107.stdout:1136: writing smithi10729407-17 from 0 to 614400 tid 1 2020-06-12T02:37:36.078 INFO:tasks.rados.rados.0.smithi107.stdout:1136: writing smithi10729407-17 from 614400 to 1138688 tid 2 2020-06-12T02:37:36.081 INFO:tasks.rados.rados.0.smithi107.stdout:1136: writing smithi10729407-17 from 1138688 to 1679360 tid 3 2020-06-12T02:37:36.084 INFO:tasks.rados.rados.0.smithi107.stdout:1136: writing smithi10729407-17 from 1679360 to 2162688 tid 4 2020-06-12T02:37:36.085 INFO:tasks.rados.rados.0.smithi107.stdout:1136: writing smithi10729407-17 from 2162688 to 2220032 tid 5 2020-06-12T02:37:36.085 INFO:tasks.rados.rados.0.smithi107.stdout:1137: delete oid 12 current snap is 0 2020-06-12T02:37:36.085 INFO:tasks.rados.rados.0.smithi107.stdout:1136: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:36.086 INFO:tasks.rados.rados.0.smithi107.stdout:1136: finishing write tid 2 to smithi10729407-17 2020-06-12T02:37:36.088 INFO:tasks.rados.rados.0.smithi107.stdout:1137: done (1 left) 2020-06-12T02:37:36.089 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:37:36.089 INFO:tasks.rados.rados.0.smithi107.stdout:1138: seq_num 1137 ranges {2072576=647168,2719744=696320,3416064=557056,3973120=352256} 2020-06-12T02:37:36.093 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:36.090+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:37:36.103 INFO:tasks.rados.rados.0.smithi107.stdout:1138: writing smithi10729407-13 from 2072576 to 2719744 tid 1 2020-06-12T02:37:36.108 INFO:tasks.rados.rados.0.smithi107.stdout:1138: writing smithi10729407-13 from 2719744 to 3416064 tid 2 2020-06-12T02:37:36.113 INFO:tasks.rados.rados.0.smithi107.stdout:1138: writing smithi10729407-13 from 3416064 to 3973120 tid 3 2020-06-12T02:37:36.116 INFO:tasks.rados.rados.0.smithi107.stdout:1138: writing smithi10729407-13 from 3973120 to 4325376 tid 4 2020-06-12T02:37:36.116 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:37:36.116 INFO:tasks.rados.rados.0.smithi107.stdout:1139: seq_num 1138 ranges {3801088=663552,4464640=581632,5046272=712704,5758976=311296} 2020-06-12T02:37:36.141 INFO:tasks.rados.rados.0.smithi107.stdout:1139: writing smithi10729407-7 from 3801088 to 4464640 tid 1 2020-06-12T02:37:36.144 INFO:tasks.rados.rados.0.smithi107.stdout:1139: writing smithi10729407-7 from 4464640 to 5046272 tid 2 2020-06-12T02:37:36.149 INFO:tasks.rados.rados.0.smithi107.stdout:1139: writing smithi10729407-7 from 5046272 to 5758976 tid 3 2020-06-12T02:37:36.151 INFO:tasks.rados.rados.0.smithi107.stdout:1139: writing smithi10729407-7 from 5758976 to 6070272 tid 4 2020-06-12T02:37:36.152 INFO:tasks.rados.rados.0.smithi107.stdout:1140: delete oid 2 current snap is 0 2020-06-12T02:37:36.152 INFO:tasks.rados.rados.0.smithi107.stdout:1136: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:36.152 INFO:tasks.rados.rados.0.smithi107.stdout:1136: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:36.153 INFO:tasks.rados.rados.0.smithi107.stdout:1136: finishing write tid 5 to smithi10729407-17 2020-06-12T02:37:36.153 INFO:tasks.rados.rados.0.smithi107.stdout:1136: finishing write tid 7 to smithi10729407-17 2020-06-12T02:37:36.153 INFO:tasks.rados.rados.0.smithi107.stdout:1136: finishing write tid 8 to smithi10729407-17 2020-06-12T02:37:36.154 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 217 (ObjNum 1135 snap 0 seq_num 1135) dirty exists 2020-06-12T02:37:36.154 INFO:tasks.rados.rados.0.smithi107.stdout:1136: left oid 17 (ObjNum 1135 snap 0 seq_num 1135) 2020-06-12T02:37:36.155 INFO:tasks.rados.rados.0.smithi107.stdout:1138: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:36.155 INFO:tasks.rados.rados.0.smithi107.stdout:1138: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:36.155 INFO:tasks.rados.rados.0.smithi107.stdout:1138: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:36.155 INFO:tasks.rados.rados.0.smithi107.stdout:1138: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:36.156 INFO:tasks.rados.rados.0.smithi107.stdout:1138: finishing write tid 6 to smithi10729407-13 2020-06-12T02:37:36.156 INFO:tasks.rados.rados.0.smithi107.stdout:1138: finishing write tid 7 to smithi10729407-13 2020-06-12T02:37:36.156 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 248 (ObjNum 1137 snap 0 seq_num 1137) dirty exists 2020-06-12T02:37:36.157 INFO:tasks.rados.rados.0.smithi107.stdout:1138: left oid 13 (ObjNum 1137 snap 0 seq_num 1137) 2020-06-12T02:37:36.158 INFO:tasks.rados.rados.0.smithi107.stdout:1136: done (3 left) 2020-06-12T02:37:36.158 INFO:tasks.rados.rados.0.smithi107.stdout:1138: done (2 left) 2020-06-12T02:37:36.159 INFO:tasks.rados.rados.0.smithi107.stdout:1140: done (1 left) 2020-06-12T02:37:36.159 INFO:tasks.rados.rados.0.smithi107.stdout:1141: delete oid 10 current snap is 0 2020-06-12T02:37:36.159 INFO:tasks.rados.rados.0.smithi107.stdout:1139: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:36.160 INFO:tasks.rados.rados.0.smithi107.stdout:1141: done (1 left) 2020-06-12T02:37:36.160 INFO:tasks.rados.rados.0.smithi107.stdout:1142: delete oid 18 current snap is 0 2020-06-12T02:37:36.160 INFO:tasks.rados.rados.0.smithi107.stdout:1139: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:36.161 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:36.161 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:36.163 INFO:tasks.rados.rados.0.smithi107.stdout:1139: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:36.164 INFO:tasks.rados.rados.0.smithi107.stdout:1142: done (1 left) 2020-06-12T02:37:36.164 INFO:tasks.rados.rados.0.smithi107.stdout:1143: delete oid 10 current snap is 0 2020-06-12T02:37:36.164 INFO:tasks.rados.rados.0.smithi107.stdout:1139: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:36.164 INFO:tasks.rados.rados.0.smithi107.stdout:1139: finishing write tid 6 to smithi10729407-7 2020-06-12T02:37:36.165 INFO:tasks.rados.rados.0.smithi107.stdout:1139: finishing write tid 7 to smithi10729407-7 2020-06-12T02:37:36.165 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 253 (ObjNum 1138 snap 0 seq_num 1138) dirty exists 2020-06-12T02:37:36.165 INFO:tasks.rados.rados.0.smithi107.stdout:1139: left oid 7 (ObjNum 1138 snap 0 seq_num 1138) 2020-06-12T02:37:36.166 INFO:tasks.rados.rados.0.smithi107.stdout:1139: done (1 left) 2020-06-12T02:37:36.166 INFO:tasks.rados.rados.0.smithi107.stdout:1143: done (0 left) 2020-06-12T02:37:36.166 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:37:36.167 INFO:tasks.rados.rados.0.smithi107.stdout:1144: seq_num 1143 ranges {0=753664,753664=450560,1204224=507904,1712128=589824,2301952=57344} 2020-06-12T02:37:36.168 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-06-12T02:37:36.169 INFO:tasks.rados.rados.0.smithi107.stdout:1144: writing smithi10729407-5 from 0 to 753664 tid 1 2020-06-12T02:37:36.170 INFO:tasks.rados.rados.0.smithi107.stdout:1144: writing smithi10729407-5 from 753664 to 1204224 tid 2 2020-06-12T02:37:36.172 INFO:tasks.rados.rados.0.smithi107.stdout:1144: writing smithi10729407-5 from 1204224 to 1712128 tid 3 2020-06-12T02:37:36.176 INFO:tasks.rados.rados.0.smithi107.stdout:1144: writing smithi10729407-5 from 1712128 to 2301952 tid 4 2020-06-12T02:37:36.177 INFO:tasks.rados.rados.0.smithi107.stdout:1144: writing smithi10729407-5 from 2301952 to 2359296 tid 5 2020-06-12T02:37:36.178 INFO:tasks.rados.rados.0.smithi107.stdout:1145: delete oid 18 current snap is 0 2020-06-12T02:37:36.178 INFO:tasks.rados.rados.0.smithi107.stdout:1144: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:36.178 INFO:tasks.rados.rados.0.smithi107.stdout:1144: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:36.179 INFO:tasks.rados.rados.0.smithi107.stdout:1145: done (1 left) 2020-06-12T02:37:36.179 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:37:36.180 INFO:tasks.rados.rados.0.smithi107.stdout:1146: seq_num 1145 ranges {0=786432,786432=622592,1409024=516096,1925120=417792,2342912=49152} 2020-06-12T02:37:36.184 INFO:tasks.rados.rados.0.smithi107.stdout:1146: writing smithi10729407-12 from 0 to 786432 tid 1 2020-06-12T02:37:36.187 INFO:tasks.rados.rados.0.smithi107.stdout:1146: writing smithi10729407-12 from 786432 to 1409024 tid 2 2020-06-12T02:37:36.190 INFO:tasks.rados.rados.0.smithi107.stdout:1146: writing smithi10729407-12 from 1409024 to 1925120 tid 3 2020-06-12T02:37:36.194 INFO:tasks.rados.rados.0.smithi107.stdout:1146: writing smithi10729407-12 from 1925120 to 2342912 tid 4 2020-06-12T02:37:36.195 INFO:tasks.rados.rados.0.smithi107.stdout:1146: writing smithi10729407-12 from 2342912 to 2392064 tid 5 2020-06-12T02:37:36.195 INFO:tasks.rados.rados.0.smithi107.stdout:1147: delete oid 14 current snap is 0 2020-06-12T02:37:36.195 INFO:tasks.rados.rados.0.smithi107.stdout:1144: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:36.196 INFO:tasks.rados.rados.0.smithi107.stdout:1144: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:36.196 INFO:tasks.rados.rados.0.smithi107.stdout:1144: finishing write tid 5 to smithi10729407-5 2020-06-12T02:37:36.196 INFO:tasks.rados.rados.0.smithi107.stdout:1144: finishing write tid 7 to smithi10729407-5 2020-06-12T02:37:36.197 INFO:tasks.rados.rados.0.smithi107.stdout:1144: finishing write tid 8 to smithi10729407-5 2020-06-12T02:37:36.197 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 108 (ObjNum 1143 snap 0 seq_num 1143) dirty exists 2020-06-12T02:37:36.197 INFO:tasks.rados.rados.0.smithi107.stdout:1144: left oid 5 (ObjNum 1143 snap 0 seq_num 1143) 2020-06-12T02:37:36.198 INFO:tasks.rados.rados.0.smithi107.stdout:1146: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:36.198 INFO:tasks.rados.rados.0.smithi107.stdout:1146: finishing write tid 2 to smithi10729407-12 2020-06-12T02:37:36.198 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:36.191+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:36.200 INFO:tasks.rados.rados.0.smithi107.stdout:1144: done (2 left) 2020-06-12T02:37:36.200 INFO:tasks.rados.rados.0.smithi107.stdout:1147: done (1 left) 2020-06-12T02:37:36.200 INFO:tasks.rados.rados.0.smithi107.stdout:1148: delete oid 4 current snap is 0 2020-06-12T02:37:36.201 INFO:tasks.rados.rados.0.smithi107.stdout:1148: done (1 left) 2020-06-12T02:37:36.201 INFO:tasks.rados.rados.0.smithi107.stdout:1149: delete oid 6 current snap is 0 2020-06-12T02:37:36.201 INFO:tasks.rados.rados.0.smithi107.stdout:1146: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:36.201 INFO:tasks.rados.rados.0.smithi107.stdout:1146: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:36.201 INFO:tasks.rados.rados.0.smithi107.stdout:1146: finishing write tid 5 to smithi10729407-12 2020-06-12T02:37:36.202 INFO:tasks.rados.rados.0.smithi107.stdout:1146: finishing write tid 7 to smithi10729407-12 2020-06-12T02:37:36.202 INFO:tasks.rados.rados.0.smithi107.stdout:1149: done (1 left) 2020-06-12T02:37:36.203 INFO:tasks.rados.rados.0.smithi107.stdout:1150: delete oid 15 current snap is 0 2020-06-12T02:37:36.203 INFO:tasks.rados.rados.0.smithi107.stdout:1146: finishing write tid 8 to smithi10729407-12 2020-06-12T02:37:36.204 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 191 (ObjNum 1145 snap 0 seq_num 1145) dirty exists 2020-06-12T02:37:36.204 INFO:tasks.rados.rados.0.smithi107.stdout:1146: left oid 12 (ObjNum 1145 snap 0 seq_num 1145) 2020-06-12T02:37:36.204 INFO:tasks.rados.rados.0.smithi107.stdout:1146: done (1 left) 2020-06-12T02:37:36.205 INFO:tasks.rados.rados.0.smithi107.stdout:1150: done (0 left) 2020-06-12T02:37:36.205 INFO:tasks.rados.rados.0.smithi107.stdout:1151: delete oid 20 current snap is 0 2020-06-12T02:37:36.205 INFO:tasks.rados.rados.0.smithi107.stdout:1151: done (0 left) 2020-06-12T02:37:36.205 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:37:36.205 INFO:tasks.rados.rados.0.smithi107.stdout:1152: seq_num 1151 ranges {0=81920} 2020-06-12T02:37:36.206 INFO:tasks.rados.rados.0.smithi107.stdout:1152: writing smithi10729407-14 from 0 to 81920 tid 1 2020-06-12T02:37:36.206 INFO:tasks.rados.rados.0.smithi107.stdout:1153: delete oid 3 current snap is 0 2020-06-12T02:37:36.207 INFO:tasks.rados.rados.0.smithi107.stdout:1153: done (1 left) 2020-06-12T02:37:36.207 INFO:tasks.rados.rados.0.smithi107.stdout:1154: delete oid 17 current snap is 0 2020-06-12T02:37:36.207 INFO:tasks.rados.rados.0.smithi107.stdout:1152: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:36.207 INFO:tasks.rados.rados.0.smithi107.stdout:1154: done (1 left) 2020-06-12T02:37:36.208 INFO:tasks.rados.rados.0.smithi107.stdout:1155: delete oid 10 current snap is 0 2020-06-12T02:37:36.210 INFO:tasks.rados.rados.0.smithi107.stdout:1152: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:36.210 INFO:tasks.rados.rados.0.smithi107.stdout:1155: done (1 left) 2020-06-12T02:37:36.210 INFO:tasks.rados.rados.0.smithi107.stdout:1156: delete oid 6 current snap is 0 2020-06-12T02:37:36.211 INFO:tasks.rados.rados.0.smithi107.stdout:1152: finishing write tid 4 to smithi10729407-14 2020-06-12T02:37:36.211 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 208 (ObjNum 1151 snap 0 seq_num 1151) dirty exists 2020-06-12T02:37:36.211 INFO:tasks.rados.rados.0.smithi107.stdout:1152: left oid 14 (ObjNum 1151 snap 0 seq_num 1151) 2020-06-12T02:37:36.215 INFO:tasks.rados.rados.0.smithi107.stdout:1152: done (1 left) 2020-06-12T02:37:36.215 INFO:tasks.rados.rados.0.smithi107.stdout:1156: done (0 left) 2020-06-12T02:37:36.215 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:37:36.216 INFO:tasks.rados.rados.0.smithi107.stdout:1157: seq_num 1156 ranges {0=163840} 2020-06-12T02:37:36.216 INFO:tasks.rados.rados.0.smithi107.stdout:1157: writing smithi10729407-15 from 0 to 163840 tid 1 2020-06-12T02:37:36.216 INFO:tasks.rados.rados.0.smithi107.stdout:1158: delete oid 14 current snap is 0 2020-06-12T02:37:36.218 INFO:tasks.rados.rados.0.smithi107.stdout:1158: done (1 left) 2020-06-12T02:37:36.218 INFO:tasks.rados.rados.0.smithi107.stdout:1159: delete oid 19 current snap is 0 2020-06-12T02:37:36.219 INFO:tasks.rados.rados.0.smithi107.stdout:1157: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:36.220 INFO:tasks.rados.rados.0.smithi107.stdout:1159: done (1 left) 2020-06-12T02:37:36.221 INFO:tasks.rados.rados.0.smithi107.stdout:1160: delete oid 7 current snap is 0 2020-06-12T02:37:36.226 INFO:tasks.rados.rados.0.smithi107.stdout:1160: done (1 left) 2020-06-12T02:37:36.226 INFO:tasks.rados.rados.0.smithi107.stdout:1161: delete oid 20 current snap is 0 2020-06-12T02:37:36.227 INFO:tasks.rados.rados.0.smithi107.stdout:1157: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:36.228 INFO:tasks.rados.rados.0.smithi107.stdout:1157: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:36.228 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 190 (ObjNum 1156 snap 0 seq_num 1156) dirty exists 2020-06-12T02:37:36.228 INFO:tasks.rados.rados.0.smithi107.stdout:1157: left oid 15 (ObjNum 1156 snap 0 seq_num 1156) 2020-06-12T02:37:36.229 INFO:tasks.rados.rados.0.smithi107.stdout:1157: done (1 left) 2020-06-12T02:37:36.229 INFO:tasks.rados.rados.0.smithi107.stdout:1161: done (0 left) 2020-06-12T02:37:36.229 INFO:tasks.rados.rados.0.smithi107.stdout:1162: delete oid 19 current snap is 0 2020-06-12T02:37:36.230 INFO:tasks.rados.rados.0.smithi107.stdout:1162: done (0 left) 2020-06-12T02:37:36.231 INFO:tasks.rados.rados.0.smithi107.stdout:1163: delete oid 11 current snap is 0 2020-06-12T02:37:36.232 INFO:tasks.rados.rados.0.smithi107.stdout:1163: done (0 left) 2020-06-12T02:37:36.233 INFO:tasks.rados.rados.0.smithi107.stdout:1164: delete oid 7 current snap is 0 2020-06-12T02:37:36.234 INFO:tasks.rados.rados.0.smithi107.stdout:1164: done (0 left) 2020-06-12T02:37:36.234 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:36.235 INFO:tasks.rados.rados.0.smithi107.stdout:1165: seq_num 1164 ranges {0=303104} 2020-06-12T02:37:36.236 INFO:tasks.rados.rados.0.smithi107.stdout:1165: writing smithi10729407-2 from 0 to 303104 tid 1 2020-06-12T02:37:36.237 INFO:tasks.rados.rados.0.smithi107.stdout:1166: delete oid 8 current snap is 0 2020-06-12T02:37:36.238 INFO:tasks.rados.rados.0.smithi107.stdout:1166: done (1 left) 2020-06-12T02:37:36.239 INFO:tasks.rados.rados.0.smithi107.stdout:1167: delete oid 16 current snap is 0 2020-06-12T02:37:36.241 INFO:tasks.rados.rados.0.smithi107.stdout:1167: done (1 left) 2020-06-12T02:37:36.241 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:37:36.242 INFO:tasks.rados.rados.0.smithi107.stdout:1168: seq_num 1167 ranges {163840=352256} 2020-06-12T02:37:36.244 INFO:tasks.rados.rados.0.smithi107.stdout:1168: writing smithi10729407-15 from 163840 to 516096 tid 1 2020-06-12T02:37:36.244 INFO:tasks.rados.rados.0.smithi107.stdout:1169: delete oid 14 current snap is 0 2020-06-12T02:37:36.245 INFO:tasks.rados.rados.0.smithi107.stdout:1165: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:36.247 INFO:tasks.rados.rados.0.smithi107.stdout:1165: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:36.248 INFO:tasks.rados.rados.0.smithi107.stdout:1169: done (2 left) 2020-06-12T02:37:36.248 INFO:tasks.rados.rados.0.smithi107.stdout:1170: delete oid 20 current snap is 0 2020-06-12T02:37:36.249 INFO:tasks.rados.rados.0.smithi107.stdout:1168: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:36.249 INFO:tasks.rados.rados.0.smithi107.stdout:1165: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:36.250 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 117 (ObjNum 1164 snap 0 seq_num 1164) dirty exists 2020-06-12T02:37:36.250 INFO:tasks.rados.rados.0.smithi107.stdout:1165: left oid 2 (ObjNum 1164 snap 0 seq_num 1164) 2020-06-12T02:37:36.251 INFO:tasks.rados.rados.0.smithi107.stdout:1165: done (2 left) 2020-06-12T02:37:36.252 INFO:tasks.rados.rados.0.smithi107.stdout:1170: done (1 left) 2020-06-12T02:37:36.252 INFO:tasks.rados.rados.0.smithi107.stdout:1171: delete oid 6 current snap is 0 2020-06-12T02:37:36.255 INFO:tasks.rados.rados.0.smithi107.stdout:1171: done (1 left) 2020-06-12T02:37:36.255 INFO:tasks.rados.rados.0.smithi107.stdout:1172: delete oid 9 current snap is 0 2020-06-12T02:37:36.255 INFO:tasks.rados.rados.0.smithi107.stdout:1168: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:36.256 INFO:tasks.rados.rados.0.smithi107.stdout:1168: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:36.256 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 193 (ObjNum 1167 snap 0 seq_num 1167) dirty exists 2020-06-12T02:37:36.256 INFO:tasks.rados.rados.0.smithi107.stdout:1168: left oid 15 (ObjNum 1167 snap 0 seq_num 1167) 2020-06-12T02:37:36.257 INFO:tasks.rados.rados.0.smithi107.stdout:1168: done (1 left) 2020-06-12T02:37:36.257 INFO:tasks.rados.rados.0.smithi107.stdout:1172: done (0 left) 2020-06-12T02:37:36.257 INFO:tasks.rados.rados.0.smithi107.stdout:1173: delete oid 10 current snap is 0 2020-06-12T02:37:36.260 INFO:tasks.rados.rados.0.smithi107.stdout:1173: done (0 left) 2020-06-12T02:37:36.260 INFO:tasks.rados.rados.0.smithi107.stdout:1174: delete oid 13 current snap is 0 2020-06-12T02:37:36.262 INFO:tasks.rados.rados.0.smithi107.stdout:1174: done (0 left) 2020-06-12T02:37:36.262 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:37:36.263 INFO:tasks.rados.rados.0.smithi107.stdout:1175: seq_num 1174 ranges {0=466944} 2020-06-12T02:37:36.265 INFO:tasks.rados.rados.0.smithi107.stdout:1175: writing smithi10729407-4 from 0 to 466944 tid 1 2020-06-12T02:37:36.266 INFO:tasks.rados.rados.0.smithi107.stdout:1176: delete oid 5 current snap is 0 2020-06-12T02:37:36.268 INFO:tasks.rados.rados.0.smithi107.stdout:1176: done (1 left) 2020-06-12T02:37:36.268 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:37:36.269 INFO:tasks.rados.rados.0.smithi107.stdout:1177: seq_num 1176 ranges {0=499712} 2020-06-12T02:37:36.272 INFO:tasks.rados.rados.0.smithi107.stdout:1177: writing smithi10729407-9 from 0 to 499712 tid 1 2020-06-12T02:37:36.272 INFO:tasks.rados.rados.0.smithi107.stdout:1178: delete oid 15 current snap is 0 2020-06-12T02:37:36.272 INFO:tasks.rados.rados.0.smithi107.stdout:1175: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:36.273 INFO:tasks.rados.rados.0.smithi107.stdout:1175: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:36.274 INFO:tasks.rados.rados.0.smithi107.stdout:1178: done (2 left) 2020-06-12T02:37:36.279 INFO:tasks.rados.rados.0.smithi107.stdout:1179: delete oid 1 current snap is 0 2020-06-12T02:37:36.280 INFO:tasks.rados.rados.0.smithi107.stdout:1175: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:36.287 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 221 (ObjNum 1174 snap 0 seq_num 1174) dirty exists 2020-06-12T02:37:36.299 INFO:tasks.rados.rados.0.smithi107.stdout:1175: left oid 4 (ObjNum 1174 snap 0 seq_num 1174) 2020-06-12T02:37:36.300 INFO:tasks.rados.rados.0.smithi107.stdout:1177: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:36.300 INFO:tasks.rados.rados.0.smithi107.stdout:1175: done (2 left) 2020-06-12T02:37:36.300 INFO:tasks.rados.rados.0.smithi107.stdout:1179: done (1 left) 2020-06-12T02:37:36.300 INFO:tasks.rados.rados.0.smithi107.stdout:1180: delete oid 17 current snap is 0 2020-06-12T02:37:36.301 INFO:tasks.rados.rados.0.smithi107.stdout:1180: done (1 left) 2020-06-12T02:37:36.301 INFO:tasks.rados.rados.0.smithi107.stdout:1181: delete oid 6 current snap is 0 2020-06-12T02:37:36.303 INFO:tasks.rados.rados.0.smithi107.stdout:1177: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:36.303 INFO:tasks.rados.rados.0.smithi107.stdout:1181: done (1 left) 2020-06-12T02:37:36.303 INFO:tasks.rados.rados.0.smithi107.stdout:1182: delete oid 15 current snap is 0 2020-06-12T02:37:36.304 INFO:tasks.rados.rados.0.smithi107.stdout:1177: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:36.304 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 225 (ObjNum 1176 snap 0 seq_num 1176) dirty exists 2020-06-12T02:37:36.304 INFO:tasks.rados.rados.0.smithi107.stdout:1177: left oid 9 (ObjNum 1176 snap 0 seq_num 1176) 2020-06-12T02:37:36.304 INFO:tasks.rados.rados.0.smithi107.stdout:1177: done (1 left) 2020-06-12T02:37:36.304 INFO:tasks.rados.rados.0.smithi107.stdout:1182: done (0 left) 2020-06-12T02:37:36.305 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:37:36.305 INFO:tasks.rados.rados.0.smithi107.stdout:1183: seq_num 1182 ranges {0=606208} 2020-06-12T02:37:36.305 INFO:tasks.rados.rados.0.smithi107.stdout:1183: writing smithi10729407-13 from 0 to 606208 tid 1 2020-06-12T02:37:36.305 INFO:tasks.rados.rados.0.smithi107.stdout:1184: delete oid 20 current snap is 0 2020-06-12T02:37:36.305 INFO:tasks.rados.rados.0.smithi107.stdout:1184: done (1 left) 2020-06-12T02:37:36.306 INFO:tasks.rados.rados.0.smithi107.stdout:1185: delete oid 17 current snap is 0 2020-06-12T02:37:36.306 INFO:tasks.rados.rados.0.smithi107.stdout:1185: done (1 left) 2020-06-12T02:37:36.306 INFO:tasks.rados.rados.0.smithi107.stdout:1186: delete oid 19 current snap is 0 2020-06-12T02:37:36.306 INFO:tasks.rados.rados.0.smithi107.stdout:1183: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:36.307 INFO:tasks.rados.rados.0.smithi107.stdout:1183: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:36.307 INFO:tasks.rados.rados.0.smithi107.stdout:1186: done (1 left) 2020-06-12T02:37:36.307 INFO:tasks.rados.rados.0.smithi107.stdout:1187: delete oid 20 current snap is 0 2020-06-12T02:37:36.307 INFO:tasks.rados.rados.0.smithi107.stdout:1183: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:36.307 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 258 (ObjNum 1182 snap 0 seq_num 1182) dirty exists 2020-06-12T02:37:36.308 INFO:tasks.rados.rados.0.smithi107.stdout:1183: left oid 13 (ObjNum 1182 snap 0 seq_num 1182) 2020-06-12T02:37:36.308 INFO:tasks.rados.rados.0.smithi107.stdout:1183: done (1 left) 2020-06-12T02:37:36.308 INFO:tasks.rados.rados.0.smithi107.stdout:1187: done (0 left) 2020-06-12T02:37:36.308 INFO:tasks.rados.rados.0.smithi107.stdout:1188: delete oid 12 current snap is 0 2020-06-12T02:37:36.308 INFO:tasks.rados.rados.0.smithi107.stdout:1188: done (0 left) 2020-06-12T02:37:36.308 INFO:tasks.rados.rados.0.smithi107.stdout:1189: delete oid 5 current snap is 0 2020-06-12T02:37:36.309 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:37:36.299+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:37:36.309 INFO:tasks.rados.rados.0.smithi107.stdout:1189: done (0 left) 2020-06-12T02:37:36.310 INFO:tasks.rados.rados.0.smithi107.stdout:1190: delete oid 20 current snap is 0 2020-06-12T02:37:36.310 INFO:tasks.rados.rados.0.smithi107.stdout:1190: done (0 left) 2020-06-12T02:37:36.310 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:37:36.310 INFO:tasks.rados.rados.0.smithi107.stdout:1191: seq_num 1190 ranges {0=737280} 2020-06-12T02:37:36.311 INFO:tasks.rados.rados.0.smithi107.stdout:1191: writing smithi10729407-8 from 0 to 737280 tid 1 2020-06-12T02:37:36.311 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:37:36.311 INFO:tasks.rados.rados.0.smithi107.stdout:1192: seq_num 1191 ranges {0=753664} 2020-06-12T02:37:36.314 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:36.314 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:36.315 INFO:tasks.rados.rados.0.smithi107.stdout:1192: writing smithi10729407-6 from 0 to 753664 tid 1 2020-06-12T02:37:36.315 INFO:tasks.rados.rados.0.smithi107.stdout:1193: delete oid 15 current snap is 0 2020-06-12T02:37:36.317 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-06-12T02:37:36.319 INFO:tasks.rados.rados.0.smithi107.stdout:1191: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:36.320 INFO:tasks.rados.rados.0.smithi107.stdout:1191: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:36.321 INFO:tasks.rados.rados.0.smithi107.stdout:1193: done (2 left) 2020-06-12T02:37:36.321 INFO:tasks.rados.rados.0.smithi107.stdout:1194: delete oid 1 current snap is 0 2020-06-12T02:37:36.322 INFO:tasks.rados.rados.0.smithi107.stdout:1192: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:36.323 INFO:tasks.rados.rados.0.smithi107.stdout:1191: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:36.323 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 197 (ObjNum 1190 snap 0 seq_num 1190) dirty exists 2020-06-12T02:37:36.323 INFO:tasks.rados.rados.0.smithi107.stdout:1191: left oid 8 (ObjNum 1190 snap 0 seq_num 1190) 2020-06-12T02:37:36.324 INFO:tasks.rados.rados.0.smithi107.stdout:1192: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:36.325 INFO:tasks.rados.rados.0.smithi107.stdout:1191: done (2 left) 2020-06-12T02:37:36.325 INFO:tasks.rados.rados.0.smithi107.stdout:1194: done (1 left) 2020-06-12T02:37:36.325 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:37:36.325 INFO:tasks.rados.rados.0.smithi107.stdout:1195: seq_num 1194 ranges {0=802816} 2020-06-12T02:37:36.326 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-06-12T02:37:36.331 INFO:tasks.rados.rados.0.smithi107.stdout:1195: writing smithi10729407-15 from 0 to 802816 tid 1 2020-06-12T02:37:36.331 INFO:tasks.rados.rados.0.smithi107.stdout:1196: delete oid 13 current snap is 0 2020-06-12T02:37:36.332 INFO:tasks.rados.rados.0.smithi107.stdout:1192: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:36.332 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 235 (ObjNum 1191 snap 0 seq_num 1191) dirty exists 2020-06-12T02:37:36.332 INFO:tasks.rados.rados.0.smithi107.stdout:1192: left oid 6 (ObjNum 1191 snap 0 seq_num 1191) 2020-06-12T02:37:36.333 INFO:tasks.rados.rados.0.smithi107.stdout:1192: done (2 left) 2020-06-12T02:37:36.333 INFO:tasks.rados.rados.0.smithi107.stdout:1196: done (1 left) 2020-06-12T02:37:36.333 INFO:tasks.rados.rados.0.smithi107.stdout:1197: delete oid 12 current snap is 0 2020-06-12T02:37:36.334 INFO:tasks.rados.rados.0.smithi107.stdout:1197: done (1 left) 2020-06-12T02:37:36.335 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:37:36.336 INFO:tasks.rados.rados.0.smithi107.stdout:1198: seq_num 1197 ranges {0=450560,450560=401408} 2020-06-12T02:37:36.338 INFO:tasks.rados.rados.0.smithi107.stdout:1198: writing smithi10729407-14 from 0 to 450560 tid 1 2020-06-12T02:37:36.340 INFO:tasks.rados.rados.0.smithi107.stdout:1198: writing smithi10729407-14 from 450560 to 851968 tid 2 2020-06-12T02:37:36.341 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:36.341 INFO:tasks.rados.rados.0.smithi107.stdout:1199: seq_num 1198 ranges {303104=466944,770048=401408} 2020-06-12T02:37:36.346 INFO:tasks.rados.rados.0.smithi107.stdout:1199: writing smithi10729407-2 from 303104 to 770048 tid 1 2020-06-12T02:37:36.350 INFO:tasks.rados.rados.0.smithi107.stdout:1199: writing smithi10729407-2 from 770048 to 1171456 tid 2 2020-06-12T02:37:36.350 INFO:tasks.rados.rados.0.smithi107.stdout:1200: delete oid 19 current snap is 0 2020-06-12T02:37:36.350 INFO:tasks.rados.rados.0.smithi107.stdout:1195: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:36.350 INFO:tasks.rados.rados.0.smithi107.stdout:1195: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:36.351 INFO:tasks.rados.rados.0.smithi107.stdout:1195: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:36.351 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 200 (ObjNum 1194 snap 0 seq_num 1194) dirty exists 2020-06-12T02:37:36.351 INFO:tasks.rados.rados.0.smithi107.stdout:1195: left oid 15 (ObjNum 1194 snap 0 seq_num 1194) 2020-06-12T02:37:36.351 INFO:tasks.rados.rados.0.smithi107.stdout:1198: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:36.351 INFO:tasks.rados.rados.0.smithi107.stdout:1198: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:36.352 INFO:tasks.rados.rados.0.smithi107.stdout:1198: finishing write tid 4 to smithi10729407-14 2020-06-12T02:37:36.352 INFO:tasks.rados.rados.0.smithi107.stdout:1198: finishing write tid 5 to smithi10729407-14 2020-06-12T02:37:36.352 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 214 (ObjNum 1197 snap 0 seq_num 1197) dirty exists 2020-06-12T02:37:36.352 INFO:tasks.rados.rados.0.smithi107.stdout:1198: left oid 14 (ObjNum 1197 snap 0 seq_num 1197) 2020-06-12T02:37:36.353 INFO:tasks.rados.rados.0.smithi107.stdout:1195: done (3 left) 2020-06-12T02:37:36.353 INFO:tasks.rados.rados.0.smithi107.stdout:1198: done (2 left) 2020-06-12T02:37:36.353 INFO:tasks.rados.rados.0.smithi107.stdout:1200: done (1 left) 2020-06-12T02:37:36.354 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:37:36.354 INFO:tasks.rados.rados.0.smithi107.stdout:1201: seq_num 1200 ranges {0=499712,499712=401408} 2020-06-12T02:37:36.354 INFO:tasks.rados.rados.0.smithi107.stdout:1201: writing smithi10729407-20 from 0 to 499712 tid 1 2020-06-12T02:37:36.356 INFO:tasks.rados.rados.0.smithi107.stdout:1201: writing smithi10729407-20 from 499712 to 901120 tid 2 2020-06-12T02:37:36.356 INFO:tasks.rados.rados.0.smithi107.stdout:1202: delete oid 9 current snap is 0 2020-06-12T02:37:36.359 INFO:tasks.rados.rados.0.smithi107.stdout:1199: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:36.360 INFO:tasks.rados.rados.0.smithi107.stdout:1199: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:36.360 INFO:tasks.rados.rados.0.smithi107.stdout:1199: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:36.361 INFO:tasks.rados.rados.0.smithi107.stdout:1201: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:36.362 INFO:tasks.rados.rados.0.smithi107.stdout:1199: finishing write tid 5 to smithi10729407-2 2020-06-12T02:37:36.362 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 120 (ObjNum 1198 snap 0 seq_num 1198) dirty exists 2020-06-12T02:37:36.362 INFO:tasks.rados.rados.0.smithi107.stdout:1199: left oid 2 (ObjNum 1198 snap 0 seq_num 1198) 2020-06-12T02:37:36.362 INFO:tasks.rados.rados.0.smithi107.stdout:1199: done (2 left) 2020-06-12T02:37:36.363 INFO:tasks.rados.rados.0.smithi107.stdout:1202: done (1 left) 2020-06-12T02:37:36.363 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:37:36.363 INFO:tasks.rados.rados.0.smithi107.stdout:1203: seq_num 1202 ranges {0=540672,540672=401408} 2020-06-12T02:37:36.364 INFO:tasks.rados.rados.0.smithi107.stdout:1203: writing smithi10729407-9 from 0 to 540672 tid 1 2020-06-12T02:37:36.369 INFO:tasks.rados.rados.0.smithi107.stdout:1203: writing smithi10729407-9 from 540672 to 942080 tid 2 2020-06-12T02:37:36.369 INFO:tasks.rados.rados.0.smithi107.stdout:1204: delete oid 17 current snap is 0 2020-06-12T02:37:36.370 INFO:tasks.rados.rados.0.smithi107.stdout:1201: finishing write tid 2 to smithi10729407-20 2020-06-12T02:37:36.371 INFO:tasks.rados.rados.0.smithi107.stdout:1201: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:36.372 INFO:tasks.rados.rados.0.smithi107.stdout:1201: finishing write tid 5 to smithi10729407-20 2020-06-12T02:37:36.372 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 100 (ObjNum 1200 snap 0 seq_num 1200) dirty exists 2020-06-12T02:37:36.372 INFO:tasks.rados.rados.0.smithi107.stdout:1201: left oid 20 (ObjNum 1200 snap 0 seq_num 1200) 2020-06-12T02:37:36.373 INFO:tasks.rados.rados.0.smithi107.stdout:1201: done (2 left) 2020-06-12T02:37:36.374 INFO:tasks.rados.rados.0.smithi107.stdout:1204: done (1 left) 2020-06-12T02:37:36.374 INFO:tasks.rados.rados.0.smithi107.stdout:1205: delete oid 7 current snap is 0 2020-06-12T02:37:36.374 INFO:tasks.rados.rados.0.smithi107.stdout:1203: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:36.375 INFO:tasks.rados.rados.0.smithi107.stdout:1203: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:36.375 INFO:tasks.rados.rados.0.smithi107.stdout:1205: done (1 left) 2020-06-12T02:37:36.376 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:37:36.376 INFO:tasks.rados.rados.0.smithi107.stdout:1206: seq_num 1205 ranges {0=589824,589824=401408} 2020-06-12T02:37:36.378 INFO:tasks.rados.rados.0.smithi107.stdout:1206: writing smithi10729407-17 from 0 to 589824 tid 1 2020-06-12T02:37:36.381 INFO:tasks.rados.rados.0.smithi107.stdout:1206: writing smithi10729407-17 from 589824 to 991232 tid 2 2020-06-12T02:37:36.381 INFO:tasks.rados.rados.0.smithi107.stdout:1207: delete oid 8 current snap is 0 2020-06-12T02:37:36.383 INFO:tasks.rados.rados.0.smithi107.stdout:1207: done (2 left) 2020-06-12T02:37:36.383 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:37:36.384 INFO:tasks.rados.rados.0.smithi107.stdout:1208: seq_num 1207 ranges {0=622592,622592=401408} 2020-06-12T02:37:36.387 INFO:tasks.rados.rados.0.smithi107.stdout:1208: writing smithi10729407-16 from 0 to 622592 tid 1 2020-06-12T02:37:36.390 INFO:tasks.rados.rados.0.smithi107.stdout:1208: writing smithi10729407-16 from 622592 to 1024000 tid 2 2020-06-12T02:37:36.390 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:37:36.390 INFO:tasks.rados.rados.0.smithi107.stdout:1209: seq_num 1208 ranges {0=638976,638976=401408} 2020-06-12T02:37:36.395 INFO:tasks.rados.rados.0.smithi107.stdout:1209: writing smithi10729407-7 from 0 to 638976 tid 1 2020-06-12T02:37:36.398 INFO:tasks.rados.rados.0.smithi107.stdout:1209: writing smithi10729407-7 from 638976 to 1040384 tid 2 2020-06-12T02:37:36.398 INFO:tasks.rados.rados.0.smithi107.stdout:1210: delete oid 10 current snap is 0 2020-06-12T02:37:36.398 INFO:tasks.rados.rados.0.smithi107.stdout:1203: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:36.398 INFO:tasks.rados.rados.0.smithi107.stdout:1203: finishing write tid 5 to smithi10729407-9 2020-06-12T02:37:36.398 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 229 (ObjNum 1202 snap 0 seq_num 1202) dirty exists 2020-06-12T02:37:36.399 INFO:tasks.rados.rados.0.smithi107.stdout:1203: left oid 9 (ObjNum 1202 snap 0 seq_num 1202) 2020-06-12T02:37:36.399 INFO:tasks.rados.rados.0.smithi107.stdout:1206: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:36.399 INFO:tasks.rados.rados.0.smithi107.stdout:1206: finishing write tid 2 to smithi10729407-17 2020-06-12T02:37:36.399 INFO:tasks.rados.rados.0.smithi107.stdout:1206: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:36.399 INFO:tasks.rados.rados.0.smithi107.stdout:1208: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:36.400 INFO:tasks.rados.rados.0.smithi107.stdout:1206: finishing write tid 5 to smithi10729407-17 2020-06-12T02:37:36.400 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 227 (ObjNum 1205 snap 0 seq_num 1205) dirty exists 2020-06-12T02:37:36.400 INFO:tasks.rados.rados.0.smithi107.stdout:1206: left oid 17 (ObjNum 1205 snap 0 seq_num 1205) 2020-06-12T02:37:36.401 INFO:tasks.rados.rados.0.smithi107.stdout:1203: done (4 left) 2020-06-12T02:37:36.401 INFO:tasks.rados.rados.0.smithi107.stdout:1206: done (3 left) 2020-06-12T02:37:36.401 INFO:tasks.rados.rados.0.smithi107.stdout:1210: done (2 left) 2020-06-12T02:37:36.401 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:37:36.401 INFO:tasks.rados.rados.0.smithi107.stdout:1211: seq_num 1210 ranges {942080=671744,1613824=401408} 2020-06-12T02:37:36.404 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:36.401+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:36.410 INFO:tasks.rados.rados.0.smithi107.stdout:1211: writing smithi10729407-9 from 942080 to 1613824 tid 1 2020-06-12T02:37:36.412 INFO:tasks.rados.rados.0.smithi107.stdout:1211: writing smithi10729407-9 from 1613824 to 2015232 tid 2 2020-06-12T02:37:36.413 INFO:tasks.rados.rados.0.smithi107.stdout:1212: delete oid 15 current snap is 0 2020-06-12T02:37:36.414 INFO:tasks.rados.rados.0.smithi107.stdout:1212: done (3 left) 2020-06-12T02:37:36.414 INFO:tasks.rados.rados.0.smithi107.stdout:1213: delete oid 1 current snap is 0 2020-06-12T02:37:36.415 INFO:tasks.rados.rados.0.smithi107.stdout:1209: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:36.416 INFO:tasks.rados.rados.0.smithi107.stdout:1208: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:36.416 INFO:tasks.rados.rados.0.smithi107.stdout:1208: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:36.416 INFO:tasks.rados.rados.0.smithi107.stdout:1209: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:36.416 INFO:tasks.rados.rados.0.smithi107.stdout:1208: finishing write tid 5 to smithi10729407-16 2020-06-12T02:37:36.417 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 205 (ObjNum 1207 snap 0 seq_num 1207) dirty exists 2020-06-12T02:37:36.417 INFO:tasks.rados.rados.0.smithi107.stdout:1208: left oid 16 (ObjNum 1207 snap 0 seq_num 1207) 2020-06-12T02:37:36.417 INFO:tasks.rados.rados.0.smithi107.stdout:1209: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:36.417 INFO:tasks.rados.rados.0.smithi107.stdout:1209: finishing write tid 5 to smithi10729407-7 2020-06-12T02:37:36.417 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 263 (ObjNum 1208 snap 0 seq_num 1208) dirty exists 2020-06-12T02:37:36.417 INFO:tasks.rados.rados.0.smithi107.stdout:1209: left oid 7 (ObjNum 1208 snap 0 seq_num 1208) 2020-06-12T02:37:36.418 INFO:tasks.rados.rados.0.smithi107.stdout:1208: done (3 left) 2020-06-12T02:37:36.418 INFO:tasks.rados.rados.0.smithi107.stdout:1209: done (2 left) 2020-06-12T02:37:36.418 INFO:tasks.rados.rados.0.smithi107.stdout:1213: done (1 left) 2020-06-12T02:37:36.419 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:37:36.419 INFO:tasks.rados.rados.0.smithi107.stdout:1214: seq_num 1213 ranges {0=720896,720896=401408} 2020-06-12T02:37:36.421 INFO:tasks.rados.rados.0.smithi107.stdout:1214: writing smithi10729407-12 from 0 to 720896 tid 1 2020-06-12T02:37:36.424 INFO:tasks.rados.rados.0.smithi107.stdout:1214: writing smithi10729407-12 from 720896 to 1122304 tid 2 2020-06-12T02:37:36.424 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:37:36.425 INFO:tasks.rados.rados.0.smithi107.stdout:1215: seq_num 1214 ranges {753664=737280,1490944=401408} 2020-06-12T02:37:36.433 INFO:tasks.rados.rados.0.smithi107.stdout:1215: writing smithi10729407-6 from 753664 to 1490944 tid 1 2020-06-12T02:37:36.436 INFO:tasks.rados.rados.0.smithi107.stdout:1215: writing smithi10729407-6 from 1490944 to 1892352 tid 2 2020-06-12T02:37:36.436 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:37:36.436 INFO:tasks.rados.rados.0.smithi107.stdout:1216: seq_num 1215 ranges {0=753664,753664=401408} 2020-06-12T02:37:36.440 INFO:tasks.rados.rados.0.smithi107.stdout:1216: writing smithi10729407-3 from 0 to 753664 tid 1 2020-06-12T02:37:36.443 INFO:tasks.rados.rados.0.smithi107.stdout:1216: writing smithi10729407-3 from 753664 to 1155072 tid 2 2020-06-12T02:37:36.443 INFO:tasks.rados.rados.0.smithi107.stdout:1217: delete oid 16 current snap is 0 2020-06-12T02:37:36.443 INFO:tasks.rados.rados.0.smithi107.stdout:1211: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:36.445 INFO:tasks.rados.rados.0.smithi107.stdout:1211: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:36.445 INFO:tasks.rados.rados.0.smithi107.stdout:1211: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:36.445 INFO:tasks.rados.rados.0.smithi107.stdout:1211: finishing write tid 5 to smithi10729407-9 2020-06-12T02:37:36.445 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 232 (ObjNum 1210 snap 0 seq_num 1210) dirty exists 2020-06-12T02:37:36.446 INFO:tasks.rados.rados.0.smithi107.stdout:1211: left oid 9 (ObjNum 1210 snap 0 seq_num 1210) 2020-06-12T02:37:36.446 INFO:tasks.rados.rados.0.smithi107.stdout:1214: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:36.446 INFO:tasks.rados.rados.0.smithi107.stdout:1214: finishing write tid 2 to smithi10729407-12 2020-06-12T02:37:36.446 INFO:tasks.rados.rados.0.smithi107.stdout:1214: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:36.446 INFO:tasks.rados.rados.0.smithi107.stdout:1214: finishing write tid 5 to smithi10729407-12 2020-06-12T02:37:36.447 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 200 (ObjNum 1213 snap 0 seq_num 1213) dirty exists 2020-06-12T02:37:36.447 INFO:tasks.rados.rados.0.smithi107.stdout:1214: left oid 12 (ObjNum 1213 snap 0 seq_num 1213) 2020-06-12T02:37:36.447 INFO:tasks.rados.rados.0.smithi107.stdout:1215: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:36.454 INFO:tasks.rados.rados.0.smithi107.stdout:1215: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:36.455 INFO:tasks.rados.rados.0.smithi107.stdout:1215: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:36.457 INFO:tasks.rados.rados.0.smithi107.stdout:1216: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:36.457 INFO:tasks.rados.rados.0.smithi107.stdout:1216: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:36.457 INFO:tasks.rados.rados.0.smithi107.stdout:1215: finishing write tid 5 to smithi10729407-6 2020-06-12T02:37:36.457 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 240 (ObjNum 1214 snap 0 seq_num 1214) dirty exists 2020-06-12T02:37:36.458 INFO:tasks.rados.rados.0.smithi107.stdout:1215: left oid 6 (ObjNum 1214 snap 0 seq_num 1214) 2020-06-12T02:37:36.459 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2020-06-12T02:37:36.459 INFO:tasks.rados.rados.0.smithi107.stdout:1216: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:36.459 INFO:tasks.rados.rados.0.smithi107.stdout:1216: finishing write tid 5 to smithi10729407-3 2020-06-12T02:37:36.459 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 208 (ObjNum 1215 snap 0 seq_num 1215) dirty exists 2020-06-12T02:37:36.459 INFO:tasks.rados.rados.0.smithi107.stdout:1216: left oid 3 (ObjNum 1215 snap 0 seq_num 1215) 2020-06-12T02:37:36.460 INFO:tasks.rados.rados.0.smithi107.stdout:1211: done (4 left) 2020-06-12T02:37:36.460 INFO:tasks.rados.rados.0.smithi107.stdout:1214: done (3 left) 2020-06-12T02:37:36.460 INFO:tasks.rados.rados.0.smithi107.stdout:1215: done (2 left) 2020-06-12T02:37:36.460 INFO:tasks.rados.rados.0.smithi107.stdout:1216: done (1 left) 2020-06-12T02:37:36.460 INFO:tasks.rados.rados.0.smithi107.stdout:1217: done (0 left) 2020-06-12T02:37:36.461 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:37:36.461 INFO:tasks.rados.rados.0.smithi107.stdout:1218: seq_num 1217 ranges {991232=786432,1777664=401408} 2020-06-12T02:37:36.468 INFO:tasks.rados.rados.0.smithi107.stdout:1218: writing smithi10729407-17 from 991232 to 1777664 tid 1 2020-06-12T02:37:36.472 INFO:tasks.rados.rados.0.smithi107.stdout:1218: writing smithi10729407-17 from 1777664 to 2179072 tid 2 2020-06-12T02:37:36.472 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:37:36.472 INFO:tasks.rados.rados.0.smithi107.stdout:1219: seq_num 1218 ranges {1155072=802816,1957888=401408} 2020-06-12T02:37:36.484 INFO:tasks.rados.rados.0.smithi107.stdout:1219: writing smithi10729407-3 from 1155072 to 1957888 tid 1 2020-06-12T02:37:36.485 INFO:tasks.rados.rados.0.smithi107.stdout:1219: writing smithi10729407-3 from 1957888 to 2359296 tid 2 2020-06-12T02:37:36.486 INFO:tasks.rados.rados.0.smithi107.stdout:1220: delete oid 16 current snap is 0 2020-06-12T02:37:36.486 INFO:tasks.rados.rados.0.smithi107.stdout:1218: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:36.486 INFO:tasks.rados.rados.0.smithi107.stdout:1218: finishing write tid 2 to smithi10729407-17 2020-06-12T02:37:36.486 INFO:tasks.rados.rados.0.smithi107.stdout:1218: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:36.486 INFO:tasks.rados.rados.0.smithi107.stdout:1218: finishing write tid 5 to smithi10729407-17 2020-06-12T02:37:36.487 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 230 (ObjNum 1217 snap 0 seq_num 1217) dirty exists 2020-06-12T02:37:36.487 INFO:tasks.rados.rados.0.smithi107.stdout:1218: left oid 17 (ObjNum 1217 snap 0 seq_num 1217) 2020-06-12T02:37:36.491 INFO:tasks.rados.rados.0.smithi107.stdout:1219: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:36.492 INFO:tasks.rados.rados.0.smithi107.stdout:1219: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:36.493 INFO:tasks.rados.rados.0.smithi107.stdout:1219: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:36.493 INFO:tasks.rados.rados.0.smithi107.stdout:1218: done (2 left) 2020-06-12T02:37:36.494 INFO:tasks.rados.rados.0.smithi107.stdout:1220: done (1 left) 2020-06-12T02:37:36.494 INFO:tasks.rados.rados.0.smithi107.stdout:1221: delete oid 13 current snap is 0 2020-06-12T02:37:36.495 INFO:tasks.rados.rados.0.smithi107.stdout:1219: finishing write tid 5 to smithi10729407-3 2020-06-12T02:37:36.496 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 212 (ObjNum 1218 snap 0 seq_num 1218) dirty exists 2020-06-12T02:37:36.496 INFO:tasks.rados.rados.0.smithi107.stdout:1219: left oid 3 (ObjNum 1218 snap 0 seq_num 1218) 2020-06-12T02:37:36.496 INFO:tasks.rados.rados.0.smithi107.stdout:1219: done (1 left) 2020-06-12T02:37:36.498 INFO:tasks.rados.rados.0.smithi107.stdout:1221: done (0 left) 2020-06-12T02:37:36.500 INFO:tasks.rados.rados.0.smithi107.stdout:1222: delete oid 15 current snap is 0 2020-06-12T02:37:36.500 INFO:tasks.rados.rados.0.smithi107.stdout:1222: done (0 left) 2020-06-12T02:37:36.501 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:37:36.501 INFO:tasks.rados.rados.0.smithi107.stdout:1223: seq_num 1222 ranges {1040384=475136,1515520=606208,2121728=196608} 2020-06-12T02:37:36.510 INFO:tasks.rados.rados.0.smithi107.stdout:1223: writing smithi10729407-7 from 1040384 to 1515520 tid 1 2020-06-12T02:37:36.511 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:36.502+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:36.512 INFO:tasks.rados.rados.0.smithi107.stdout:1223: writing smithi10729407-7 from 1515520 to 2121728 tid 2 2020-06-12T02:37:36.513 INFO:tasks.rados.rados.0.smithi107.stdout:1223: writing smithi10729407-7 from 2121728 to 2318336 tid 3 2020-06-12T02:37:36.514 INFO:tasks.rados.rados.0.smithi107.stdout:1224: delete oid 18 current snap is 0 2020-06-12T02:37:36.514 INFO:tasks.rados.rados.0.smithi107.stdout:1223: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:36.514 INFO:tasks.rados.rados.0.smithi107.stdout:1224: done (1 left) 2020-06-12T02:37:36.515 INFO:tasks.rados.rados.0.smithi107.stdout:1225: delete oid 18 current snap is 0 2020-06-12T02:37:36.517 INFO:tasks.rados.rados.0.smithi107.stdout:1225: done (1 left) 2020-06-12T02:37:36.517 INFO:tasks.rados.rados.0.smithi107.stdout:1226: delete oid 11 current snap is 0 2020-06-12T02:37:36.517 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:36.518 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:36.518 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:37:36.519 INFO:tasks.rados.rados.0.smithi107.stdout:1226: done (1 left) 2020-06-12T02:37:36.520 INFO:tasks.rados.rados.0.smithi107.stdout:1227: delete oid 16 current snap is 0 2020-06-12T02:37:36.520 INFO:tasks.rados.rados.0.smithi107.stdout:1223: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:36.521 INFO:tasks.rados.rados.0.smithi107.stdout:1223: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:36.521 INFO:tasks.rados.rados.0.smithi107.stdout:1223: finishing write tid 5 to smithi10729407-7 2020-06-12T02:37:36.522 INFO:tasks.rados.rados.0.smithi107.stdout:1223: finishing write tid 6 to smithi10729407-7 2020-06-12T02:37:36.522 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 268 (ObjNum 1222 snap 0 seq_num 1222) dirty exists 2020-06-12T02:37:36.522 INFO:tasks.rados.rados.0.smithi107.stdout:1223: left oid 7 (ObjNum 1222 snap 0 seq_num 1222) 2020-06-12T02:37:36.523 INFO:tasks.rados.rados.0.smithi107.stdout:1223: done (1 left) 2020-06-12T02:37:36.523 INFO:tasks.rados.rados.0.smithi107.stdout:1227: done (0 left) 2020-06-12T02:37:36.523 INFO:tasks.rados.rados.0.smithi107.stdout:1228: delete oid 6 current snap is 0 2020-06-12T02:37:36.524 INFO:tasks.rados.rados.0.smithi107.stdout:1228: done (0 left) 2020-06-12T02:37:36.524 INFO:tasks.rados.rados.0.smithi107.stdout:1229: delete oid 15 current snap is 0 2020-06-12T02:37:36.526 INFO:tasks.rados.rados.0.smithi107.stdout:1229: done (0 left) 2020-06-12T02:37:36.526 INFO:tasks.rados.rados.0.smithi107.stdout:1230: delete oid 3 current snap is 0 2020-06-12T02:37:36.529 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-06-12T02:37:36.531 INFO:tasks.rados.rados.0.smithi107.stdout:1230: done (0 left) 2020-06-12T02:37:36.531 INFO:tasks.rados.rados.0.smithi107.stdout:1231: delete oid 4 current snap is 0 2020-06-12T02:37:36.533 INFO:tasks.rados.rados.0.smithi107.stdout:1231: done (0 left) 2020-06-12T02:37:36.534 INFO:tasks.rados.rados.0.smithi107.stdout:1232: delete oid 5 current snap is 0 2020-06-12T02:37:36.535 INFO:tasks.rados.rados.0.smithi107.stdout:1232: done (0 left) 2020-06-12T02:37:36.536 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:36.536 INFO:tasks.rados.rados.0.smithi107.stdout:1233: seq_num 1232 ranges {1171456=638976,1810432=753664,2564096=49152} 2020-06-12T02:37:36.544 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:37:36.545 INFO:tasks.rados.rados.0.smithi107.stdout:1233: writing smithi10729407-2 from 1171456 to 1810432 tid 1 2020-06-12T02:37:36.550 INFO:tasks.rados.rados.0.smithi107.stdout:1233: writing smithi10729407-2 from 1810432 to 2564096 tid 2 2020-06-12T02:37:36.551 INFO:tasks.rados.rados.0.smithi107.stdout:1233: writing smithi10729407-2 from 2564096 to 2613248 tid 3 2020-06-12T02:37:36.551 INFO:tasks.rados.rados.0.smithi107.stdout:1234: delete oid 5 current snap is 0 2020-06-12T02:37:36.552 INFO:tasks.rados.rados.0.smithi107.stdout:1233: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:36.553 INFO:tasks.rados.rados.0.smithi107.stdout:1234: done (1 left) 2020-06-12T02:37:36.553 INFO:tasks.rados.rados.0.smithi107.stdout:1235: delete oid 13 current snap is 0 2020-06-12T02:37:36.558 INFO:tasks.rados.rados.0.smithi107.stdout:1235: done (1 left) 2020-06-12T02:37:36.559 INFO:tasks.rados.rados.0.smithi107.stdout:1236: delete oid 10 current snap is 0 2020-06-12T02:37:36.560 INFO:tasks.rados.rados.0.smithi107.stdout:1233: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:36.560 INFO:tasks.rados.rados.0.smithi107.stdout:1233: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:36.561 INFO:tasks.rados.rados.0.smithi107.stdout:1233: finishing write tid 5 to smithi10729407-2 2020-06-12T02:37:36.561 INFO:tasks.rados.rados.0.smithi107.stdout:1236: done (1 left) 2020-06-12T02:37:36.561 INFO:tasks.rados.rados.0.smithi107.stdout:1237: delete oid 3 current snap is 0 2020-06-12T02:37:36.563 INFO:tasks.rados.rados.0.smithi107.stdout:1233: finishing write tid 6 to smithi10729407-2 2020-06-12T02:37:36.564 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 124 (ObjNum 1232 snap 0 seq_num 1232) dirty exists 2020-06-12T02:37:36.564 INFO:tasks.rados.rados.0.smithi107.stdout:1233: left oid 2 (ObjNum 1232 snap 0 seq_num 1232) 2020-06-12T02:37:36.564 INFO:tasks.rados.rados.0.smithi107.stdout:1233: done (1 left) 2020-06-12T02:37:36.564 INFO:tasks.rados.rados.0.smithi107.stdout:1237: done (0 left) 2020-06-12T02:37:36.564 INFO:tasks.rados.rados.0.smithi107.stdout:1238: delete oid 4 current snap is 0 2020-06-12T02:37:36.565 INFO:tasks.rados.rados.0.smithi107.stdout:1238: done (0 left) 2020-06-12T02:37:36.565 INFO:tasks.rados.rados.0.smithi107.stdout:1239: delete oid 7 current snap is 0 2020-06-12T02:37:36.567 INFO:tasks.rados.rados.0.smithi107.stdout:1239: done (0 left) 2020-06-12T02:37:36.567 INFO:tasks.rados.rados.0.smithi107.stdout:1240: delete oid 12 current snap is 0 2020-06-12T02:37:36.569 INFO:tasks.rados.rados.0.smithi107.stdout:1240: done (0 left) 2020-06-12T02:37:36.569 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:37:36.569 INFO:tasks.rados.rados.0.smithi107.stdout:1241: seq_num 1240 ranges {0=778240,778240=671744,1449984=131072} 2020-06-12T02:37:36.574 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:37:36.574 INFO:tasks.rados.rados.0.smithi107.stdout:1241: writing smithi10729407-3 from 0 to 778240 tid 1 2020-06-12T02:37:36.579 INFO:tasks.rados.rados.0.smithi107.stdout:1241: writing smithi10729407-3 from 778240 to 1449984 tid 2 2020-06-12T02:37:36.580 INFO:tasks.rados.rados.0.smithi107.stdout:1241: writing smithi10729407-3 from 1449984 to 1581056 tid 3 2020-06-12T02:37:36.580 INFO:tasks.rados.rados.0.smithi107.stdout:1242: delete oid 14 current snap is 0 2020-06-12T02:37:36.581 INFO:tasks.rados.rados.0.smithi107.stdout:1241: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:36.586 INFO:tasks.rados.rados.0.smithi107.stdout:1242: done (1 left) 2020-06-12T02:37:36.587 INFO:tasks.rados.rados.0.smithi107.stdout:1243: delete oid 8 current snap is 0 2020-06-12T02:37:36.587 INFO:tasks.rados.rados.0.smithi107.stdout:1243: done (1 left) 2020-06-12T02:37:36.588 INFO:tasks.rados.rados.0.smithi107.stdout:1244: delete oid 18 current snap is 0 2020-06-12T02:37:36.590 INFO:tasks.rados.rados.0.smithi107.stdout:1241: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:36.591 INFO:tasks.rados.rados.0.smithi107.stdout:1241: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:36.591 INFO:tasks.rados.rados.0.smithi107.stdout:1244: done (1 left) 2020-06-12T02:37:36.591 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:37:36.591 INFO:tasks.rados.rados.0.smithi107.stdout:1245: seq_num 1244 ranges {0=442368,442368=614400,1056768=589824} 2020-06-12T02:37:36.592 INFO:tasks.rados.rados.0.smithi107.stdout:1245: writing smithi10729407-18 from 0 to 442368 tid 1 2020-06-12T02:37:36.597 INFO:tasks.rados.rados.0.smithi107.stdout:1245: writing smithi10729407-18 from 442368 to 1056768 tid 2 2020-06-12T02:37:36.600 INFO:tasks.rados.rados.0.smithi107.stdout:1245: writing smithi10729407-18 from 1056768 to 1646592 tid 3 2020-06-12T02:37:36.600 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:37:36.601 INFO:tasks.rados.rados.0.smithi107.stdout:1246: seq_num 1245 ranges {0=458752,458752=499712,958464=425984,1384448=278528} 2020-06-12T02:37:36.603 INFO:tasks.rados.rados.0.smithi107.stdout:1246: writing smithi10729407-13 from 0 to 458752 tid 1 2020-06-12T02:37:36.605 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:36.602+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:36.606 INFO:tasks.rados.rados.0.smithi107.stdout:1246: writing smithi10729407-13 from 458752 to 958464 tid 2 2020-06-12T02:37:36.607 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:37:36.609 INFO:tasks.rados.rados.0.smithi107.stdout:1246: writing smithi10729407-13 from 958464 to 1384448 tid 3 2020-06-12T02:37:36.612 INFO:tasks.rados.rados.0.smithi107.stdout:1246: writing smithi10729407-13 from 1384448 to 1662976 tid 4 2020-06-12T02:37:36.612 INFO:tasks.rados.rados.0.smithi107.stdout:1247: delete oid 17 current snap is 0 2020-06-12T02:37:36.613 INFO:tasks.rados.rados.0.smithi107.stdout:1241: finishing write tid 5 to smithi10729407-3 2020-06-12T02:37:36.613 INFO:tasks.rados.rados.0.smithi107.stdout:1241: finishing write tid 6 to smithi10729407-3 2020-06-12T02:37:36.613 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 220 (ObjNum 1240 snap 0 seq_num 1240) dirty exists 2020-06-12T02:37:36.613 INFO:tasks.rados.rados.0.smithi107.stdout:1241: left oid 3 (ObjNum 1240 snap 0 seq_num 1240) 2020-06-12T02:37:36.613 INFO:tasks.rados.rados.0.smithi107.stdout:1245: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:36.614 INFO:tasks.rados.rados.0.smithi107.stdout:1245: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:36.614 INFO:tasks.rados.rados.0.smithi107.stdout:1246: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:36.615 INFO:tasks.rados.rados.0.smithi107.stdout:1245: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:36.615 INFO:tasks.rados.rados.0.smithi107.stdout:1245: finishing write tid 5 to smithi10729407-18 2020-06-12T02:37:36.618 INFO:tasks.rados.rados.0.smithi107.stdout:1241: done (3 left) 2020-06-12T02:37:36.618 INFO:tasks.rados.rados.0.smithi107.stdout:1247: done (2 left) 2020-06-12T02:37:36.618 INFO:tasks.rados.rados.0.smithi107.stdout:1248: delete oid 10 current snap is 0 2020-06-12T02:37:36.619 INFO:tasks.rados.rados.0.smithi107.stdout:1245: finishing write tid 6 to smithi10729407-18 2020-06-12T02:37:36.619 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 239 (ObjNum 1244 snap 0 seq_num 1244) dirty exists 2020-06-12T02:37:36.619 INFO:tasks.rados.rados.0.smithi107.stdout:1245: left oid 18 (ObjNum 1244 snap 0 seq_num 1244) 2020-06-12T02:37:36.619 INFO:tasks.rados.rados.0.smithi107.stdout:1246: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:36.620 INFO:tasks.rados.rados.0.smithi107.stdout:1246: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:36.620 INFO:tasks.rados.rados.0.smithi107.stdout:1246: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:36.621 INFO:tasks.rados.rados.0.smithi107.stdout:1245: done (2 left) 2020-06-12T02:37:36.621 INFO:tasks.rados.rados.0.smithi107.stdout:1248: done (1 left) 2020-06-12T02:37:36.621 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:37:36.622 INFO:tasks.rados.rados.0.smithi107.stdout:1249: seq_num 1248 ranges {0=507904,507904=598016,1105920=606208} 2020-06-12T02:37:36.624 INFO:tasks.rados.rados.0.smithi107.stdout:1249: writing smithi10729407-1 from 0 to 507904 tid 1 2020-06-12T02:37:36.635 INFO:tasks.rados.rados.0.smithi107.stdout:1249: writing smithi10729407-1 from 507904 to 1105920 tid 2 2020-06-12T02:37:36.635 INFO:tasks.rados.rados.0.smithi107.stdout:1249: writing smithi10729407-1 from 1105920 to 1712128 tid 3 2020-06-12T02:37:36.635 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:37:36.636 INFO:tasks.rados.rados.0.smithi107.stdout:1250: seq_num 1249 ranges {0=524288,524288=483328,1007616=409600,1417216=311296} 2020-06-12T02:37:36.636 INFO:tasks.rados.rados.0.smithi107.stdout:1250: writing smithi10729407-4 from 0 to 524288 tid 1 2020-06-12T02:37:36.637 INFO:tasks.rados.rados.0.smithi107.stdout:1250: writing smithi10729407-4 from 524288 to 1007616 tid 2 2020-06-12T02:37:36.638 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:37:36.639 INFO:tasks.rados.rados.0.smithi107.stdout:1250: writing smithi10729407-4 from 1007616 to 1417216 tid 3 2020-06-12T02:37:36.641 INFO:tasks.rados.rados.0.smithi107.stdout:1250: writing smithi10729407-4 from 1417216 to 1728512 tid 4 2020-06-12T02:37:36.642 INFO:tasks.rados.rados.0.smithi107.stdout:1251: delete oid 2 current snap is 0 2020-06-12T02:37:36.642 INFO:tasks.rados.rados.0.smithi107.stdout:1246: finishing write tid 6 to smithi10729407-13 2020-06-12T02:37:36.642 INFO:tasks.rados.rados.0.smithi107.stdout:1246: finishing write tid 7 to smithi10729407-13 2020-06-12T02:37:36.642 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 275 (ObjNum 1245 snap 0 seq_num 1245) dirty exists 2020-06-12T02:37:36.642 INFO:tasks.rados.rados.0.smithi107.stdout:1246: left oid 13 (ObjNum 1245 snap 0 seq_num 1245) 2020-06-12T02:37:36.643 INFO:tasks.rados.rados.0.smithi107.stdout:1249: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:36.643 INFO:tasks.rados.rados.0.smithi107.stdout:1249: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:36.643 INFO:tasks.rados.rados.0.smithi107.stdout:1249: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:36.643 INFO:tasks.rados.rados.0.smithi107.stdout:1249: finishing write tid 5 to smithi10729407-1 2020-06-12T02:37:36.643 INFO:tasks.rados.rados.0.smithi107.stdout:1249: finishing write tid 6 to smithi10729407-1 2020-06-12T02:37:36.644 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 245 (ObjNum 1248 snap 0 seq_num 1248) dirty exists 2020-06-12T02:37:36.644 INFO:tasks.rados.rados.0.smithi107.stdout:1249: left oid 1 (ObjNum 1248 snap 0 seq_num 1248) 2020-06-12T02:37:36.644 INFO:tasks.rados.rados.0.smithi107.stdout:1250: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:36.646 INFO:tasks.rados.rados.0.smithi107.stdout:1246: done (3 left) 2020-06-12T02:37:36.646 INFO:tasks.rados.rados.0.smithi107.stdout:1249: done (2 left) 2020-06-12T02:37:36.647 INFO:tasks.rados.rados.0.smithi107.stdout:1251: done (1 left) 2020-06-12T02:37:36.647 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:37:36.647 INFO:tasks.rados.rados.0.smithi107.stdout:1252: seq_num 1251 ranges {0=557056,557056=655360,1212416=425984,1638400=122880} 2020-06-12T02:37:36.650 INFO:tasks.rados.rados.0.smithi107.stdout:1252: writing smithi10729407-10 from 0 to 557056 tid 1 2020-06-12T02:37:36.653 INFO:tasks.rados.rados.0.smithi107.stdout:1252: writing smithi10729407-10 from 557056 to 1212416 tid 2 2020-06-12T02:37:36.656 INFO:tasks.rados.rados.0.smithi107.stdout:1252: writing smithi10729407-10 from 1212416 to 1638400 tid 3 2020-06-12T02:37:36.657 INFO:tasks.rados.rados.0.smithi107.stdout:1252: writing smithi10729407-10 from 1638400 to 1761280 tid 4 2020-06-12T02:37:36.658 INFO:tasks.rados.rados.0.smithi107.stdout:1253: delete oid 18 current snap is 0 2020-06-12T02:37:36.658 INFO:tasks.rados.rados.0.smithi107.stdout:1250: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:36.658 INFO:tasks.rados.rados.0.smithi107.stdout:1250: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:36.658 INFO:tasks.rados.rados.0.smithi107.stdout:1250: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:36.658 INFO:tasks.rados.rados.0.smithi107.stdout:1250: finishing write tid 6 to smithi10729407-4 2020-06-12T02:37:36.659 INFO:tasks.rados.rados.0.smithi107.stdout:1250: finishing write tid 7 to smithi10729407-4 2020-06-12T02:37:36.659 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 238 (ObjNum 1249 snap 0 seq_num 1249) dirty exists 2020-06-12T02:37:36.659 INFO:tasks.rados.rados.0.smithi107.stdout:1250: left oid 4 (ObjNum 1249 snap 0 seq_num 1249) 2020-06-12T02:37:36.662 INFO:tasks.rados.rados.0.smithi107.stdout:1252: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:36.662 INFO:tasks.rados.rados.0.smithi107.stdout:1250: done (2 left) 2020-06-12T02:37:36.662 INFO:tasks.rados.rados.0.smithi107.stdout:1253: done (1 left) 2020-06-12T02:37:36.662 INFO:tasks.rados.rados.0.smithi107.stdout:1254: delete oid 11 current snap is 0 2020-06-12T02:37:36.666 INFO:tasks.rados.rados.0.smithi107.stdout:1252: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:36.667 INFO:tasks.rados.rados.0.smithi107.stdout:1252: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:36.667 INFO:tasks.rados.rados.0.smithi107.stdout:1254: done (1 left) 2020-06-12T02:37:36.668 INFO:tasks.rados.rados.0.smithi107.stdout:1255: delete oid 14 current snap is 0 2020-06-12T02:37:36.669 INFO:tasks.rados.rados.0.smithi107.stdout:1252: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:36.669 INFO:tasks.rados.rados.0.smithi107.stdout:1252: finishing write tid 6 to smithi10729407-10 2020-06-12T02:37:36.670 INFO:tasks.rados.rados.0.smithi107.stdout:1255: done (1 left) 2020-06-12T02:37:36.670 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:37:36.670 INFO:tasks.rados.rados.0.smithi107.stdout:1256: seq_num 1255 ranges {0=622592,622592=638976,1261568=565248} 2020-06-12T02:37:36.671 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:37:36.672 INFO:tasks.rados.rados.0.smithi107.stdout:1256: writing smithi10729407-14 from 0 to 622592 tid 1 2020-06-12T02:37:36.676 INFO:tasks.rados.rados.0.smithi107.stdout:1256: writing smithi10729407-14 from 622592 to 1261568 tid 2 2020-06-12T02:37:36.681 INFO:tasks.rados.rados.0.smithi107.stdout:1256: writing smithi10729407-14 from 1261568 to 1826816 tid 3 2020-06-12T02:37:36.682 INFO:tasks.rados.rados.0.smithi107.stdout:1257: delete oid 8 current snap is 0 2020-06-12T02:37:36.687 INFO:tasks.rados.rados.0.smithi107.stdout:1252: finishing write tid 7 to smithi10729407-10 2020-06-12T02:37:36.688 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 113 (ObjNum 1251 snap 0 seq_num 1251) dirty exists 2020-06-12T02:37:36.688 INFO:tasks.rados.rados.0.smithi107.stdout:1252: left oid 10 (ObjNum 1251 snap 0 seq_num 1251) 2020-06-12T02:37:36.689 INFO:tasks.rados.rados.0.smithi107.stdout:1252: done (2 left) 2020-06-12T02:37:36.690 INFO:tasks.rados.rados.0.smithi107.stdout:1257: done (1 left) 2020-06-12T02:37:36.690 INFO:tasks.rados.rados.0.smithi107.stdout:1258: delete oid 5 current snap is 0 2020-06-12T02:37:36.690 INFO:tasks.rados.rados.0.smithi107.stdout:1258: done (1 left) 2020-06-12T02:37:36.690 INFO:tasks.rados.rados.0.smithi107.stdout:1259: delete oid 20 current snap is 0 2020-06-12T02:37:36.690 INFO:tasks.rados.rados.0.smithi107.stdout:1256: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:36.691 INFO:tasks.rados.rados.0.smithi107.stdout:1259: done (1 left) 2020-06-12T02:37:36.691 INFO:tasks.rados.rados.0.smithi107.stdout:1260: delete oid 18 current snap is 0 2020-06-12T02:37:36.691 INFO:tasks.rados.rados.0.smithi107.stdout:1256: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:36.691 INFO:tasks.rados.rados.0.smithi107.stdout:1260: done (1 left) 2020-06-12T02:37:36.691 INFO:tasks.rados.rados.0.smithi107.stdout:1261: delete oid 8 current snap is 0 2020-06-12T02:37:36.692 INFO:tasks.rados.rados.0.smithi107.stdout:1256: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:36.692 INFO:tasks.rados.rados.0.smithi107.stdout:1256: finishing write tid 5 to smithi10729407-14 2020-06-12T02:37:36.696 INFO:tasks.rados.rados.0.smithi107.stdout:1256: finishing write tid 6 to smithi10729407-14 2020-06-12T02:37:36.697 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 222 (ObjNum 1255 snap 0 seq_num 1255) dirty exists 2020-06-12T02:37:36.697 INFO:tasks.rados.rados.0.smithi107.stdout:1256: left oid 14 (ObjNum 1255 snap 0 seq_num 1255) 2020-06-12T02:37:36.697 INFO:tasks.rados.rados.0.smithi107.stdout:1256: done (1 left) 2020-06-12T02:37:36.697 INFO:tasks.rados.rados.0.smithi107.stdout:1261: done (0 left) 2020-06-12T02:37:36.697 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:37:36.698 INFO:tasks.rados.rados.0.smithi107.stdout:1262: seq_num 1261 ranges {1662976=729088,2392064=745472,3137536=458752} 2020-06-12T02:37:36.698 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:37:36.698 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:37:36.698 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2020-06-12T02:37:36.698 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:37:36.699 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:37:36.700 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-06-12T02:37:36.700 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-06-12T02:37:36.701 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-06-12T02:37:36.701 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-06-12T02:37:36.701 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-06-12T02:37:36.702 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-06-12T02:37:36.702 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-06-12T02:37:36.702 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.7 2020-06-12T02:37:36.705 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: problem getting command descriptions from osd.7 2020-06-12T02:37:36.708 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-06-12T02:37:36.708 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-06-12T02:37:36.709 INFO:tasks.rados.rados.0.smithi107.stdout:1262: writing smithi10729407-13 from 1662976 to 2392064 tid 1 2020-06-12T02:37:36.709 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:36.703+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:36.710 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-06-12T02:37:36.711 INFO:tasks.rados.rados.0.smithi107.stdout:1262: writing smithi10729407-13 from 2392064 to 3137536 tid 2 2020-06-12T02:37:36.712 INFO:tasks.rados.rados.0.smithi107.stdout:1262: writing smithi10729407-13 from 3137536 to 3596288 tid 3 2020-06-12T02:37:36.713 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:37:36.713 INFO:tasks.rados.rados.0.smithi107.stdout:1263: seq_num 1262 ranges {0=745472,745472=679936,1425408=524288} 2020-06-12T02:37:36.717 INFO:tasks.rados.rados.0.smithi107.stdout:1263: writing smithi10729407-15 from 0 to 745472 tid 1 2020-06-12T02:37:36.721 INFO:tasks.rados.rados.0.smithi107.stdout:1263: writing smithi10729407-15 from 745472 to 1425408 tid 2 2020-06-12T02:37:36.724 INFO:tasks.rados.rados.0.smithi107.stdout:1263: writing smithi10729407-15 from 1425408 to 1949696 tid 3 2020-06-12T02:37:36.724 INFO:tasks.rados.rados.0.smithi107.stdout:1264: delete oid 14 current snap is 0 2020-06-12T02:37:36.725 INFO:tasks.rados.rados.0.smithi107.stdout:1262: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:36.725 INFO:tasks.rados.rados.0.smithi107.stdout:1262: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:36.725 INFO:tasks.rados.rados.0.smithi107.stdout:1262: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:36.725 INFO:tasks.rados.rados.0.smithi107.stdout:1262: finishing write tid 5 to smithi10729407-13 2020-06-12T02:37:36.725 INFO:tasks.rados.rados.0.smithi107.stdout:1262: finishing write tid 6 to smithi10729407-13 2020-06-12T02:37:36.726 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 279 (ObjNum 1261 snap 0 seq_num 1261) dirty exists 2020-06-12T02:37:36.726 INFO:tasks.rados.rados.0.smithi107.stdout:1262: left oid 13 (ObjNum 1261 snap 0 seq_num 1261) 2020-06-12T02:37:36.726 INFO:tasks.rados.rados.0.smithi107.stdout:1263: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:36.727 INFO:tasks.rados.rados.0.smithi107.stdout:1262: done (2 left) 2020-06-12T02:37:36.728 INFO:tasks.rados.rados.0.smithi107.stdout:1264: done (1 left) 2020-06-12T02:37:36.728 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:37:36.728 INFO:tasks.rados.rados.0.smithi107.stdout:1265: seq_num 1264 ranges {0=778240,778240=450560,1228800=573440,1802240=180224} 2020-06-12T02:37:36.732 INFO:tasks.rados.rados.0.smithi107.stdout:1265: writing smithi10729407-8 from 0 to 778240 tid 1 2020-06-12T02:37:36.735 INFO:tasks.rados.rados.0.smithi107.stdout:1265: writing smithi10729407-8 from 778240 to 1228800 tid 2 2020-06-12T02:37:36.738 INFO:tasks.rados.rados.0.smithi107.stdout:1265: writing smithi10729407-8 from 1228800 to 1802240 tid 3 2020-06-12T02:37:36.739 INFO:tasks.rados.rados.0.smithi107.stdout:1265: writing smithi10729407-8 from 1802240 to 1982464 tid 4 2020-06-12T02:37:36.740 INFO:tasks.rados.rados.0.smithi107.stdout:1266: delete oid 7 current snap is 0 2020-06-12T02:37:36.740 INFO:tasks.rados.rados.0.smithi107.stdout:1263: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:36.740 INFO:tasks.rados.rados.0.smithi107.stdout:1263: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:36.740 INFO:tasks.rados.rados.0.smithi107.stdout:1263: finishing write tid 5 to smithi10729407-15 2020-06-12T02:37:36.741 INFO:tasks.rados.rados.0.smithi107.stdout:1265: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:36.742 INFO:tasks.rados.rados.0.smithi107.stdout:1266: done (2 left) 2020-06-12T02:37:36.742 INFO:tasks.rados.rados.0.smithi107.stdout:1267: delete oid 6 current snap is 0 2020-06-12T02:37:36.743 INFO:tasks.rados.rados.0.smithi107.stdout:1267: done (2 left) 2020-06-12T02:37:36.744 INFO:tasks.rados.rados.0.smithi107.stdout:1268: delete oid 11 current snap is 0 2020-06-12T02:37:36.746 INFO:tasks.rados.rados.0.smithi107.stdout:1268: done (2 left) 2020-06-12T02:37:36.746 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:37:36.746 INFO:tasks.rados.rados.0.smithi107.stdout:1269: seq_num 1268 ranges {0=442368,442368=786432,1228800=557056,1785856=262144} 2020-06-12T02:37:36.748 INFO:tasks.rados.rados.0.smithi107.stdout:1269: writing smithi10729407-16 from 0 to 442368 tid 1 2020-06-12T02:37:36.754 INFO:tasks.rados.rados.0.smithi107.stdout:1269: writing smithi10729407-16 from 442368 to 1228800 tid 2 2020-06-12T02:37:36.758 INFO:tasks.rados.rados.0.smithi107.stdout:1269: writing smithi10729407-16 from 1228800 to 1785856 tid 3 2020-06-12T02:37:36.760 INFO:tasks.rados.rados.0.smithi107.stdout:1269: writing smithi10729407-16 from 1785856 to 2048000 tid 4 2020-06-12T02:37:36.760 INFO:tasks.rados.rados.0.smithi107.stdout:1270: delete oid 20 current snap is 0 2020-06-12T02:37:36.761 INFO:tasks.rados.rados.0.smithi107.stdout:1265: finishing write tid 2 to smithi10729407-8 2020-06-12T02:37:36.761 INFO:tasks.rados.rados.0.smithi107.stdout:1263: finishing write tid 6 to smithi10729407-15 2020-06-12T02:37:36.761 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 211 (ObjNum 1262 snap 0 seq_num 1262) dirty exists 2020-06-12T02:37:36.761 INFO:tasks.rados.rados.0.smithi107.stdout:1263: left oid 15 (ObjNum 1262 snap 0 seq_num 1262) 2020-06-12T02:37:36.761 INFO:tasks.rados.rados.0.smithi107.stdout:1265: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:36.762 INFO:tasks.rados.rados.0.smithi107.stdout:1265: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:36.762 INFO:tasks.rados.rados.0.smithi107.stdout:1265: finishing write tid 6 to smithi10729407-8 2020-06-12T02:37:36.762 INFO:tasks.rados.rados.0.smithi107.stdout:1265: finishing write tid 7 to smithi10729407-8 2020-06-12T02:37:36.762 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 216 (ObjNum 1264 snap 0 seq_num 1264) dirty exists 2020-06-12T02:37:36.762 INFO:tasks.rados.rados.0.smithi107.stdout:1265: left oid 8 (ObjNum 1264 snap 0 seq_num 1264) 2020-06-12T02:37:36.762 INFO:tasks.rados.rados.0.smithi107.stdout:1269: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:36.763 INFO:tasks.rados.rados.0.smithi107.stdout:1269: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:36.764 INFO:tasks.rados.rados.0.smithi107.stdout:1263: done (3 left) 2020-06-12T02:37:36.764 INFO:tasks.rados.rados.0.smithi107.stdout:1265: done (2 left) 2020-06-12T02:37:36.764 INFO:tasks.rados.rados.0.smithi107.stdout:1270: done (1 left) 2020-06-12T02:37:36.764 INFO:tasks.rados.rados.0.smithi107.stdout:1271: delete oid 6 current snap is 0 2020-06-12T02:37:36.766 INFO:tasks.rados.rados.0.smithi107.stdout:1269: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:36.766 INFO:tasks.rados.rados.0.smithi107.stdout:1271: done (1 left) 2020-06-12T02:37:36.766 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:37:36.767 INFO:tasks.rados.rados.0.smithi107.stdout:1272: seq_num 1271 ranges {0=491520,491520=491520,983040=729088,1712128=385024} 2020-06-12T02:37:36.770 INFO:tasks.rados.rados.0.smithi107.stdout:1272: writing smithi10729407-6 from 0 to 491520 tid 1 2020-06-12T02:37:36.772 INFO:tasks.rados.rados.0.smithi107.stdout:1272: writing smithi10729407-6 from 491520 to 983040 tid 2 2020-06-12T02:37:36.776 INFO:tasks.rados.rados.0.smithi107.stdout:1272: writing smithi10729407-6 from 983040 to 1712128 tid 3 2020-06-12T02:37:36.779 INFO:tasks.rados.rados.0.smithi107.stdout:1272: writing smithi10729407-6 from 1712128 to 2097152 tid 4 2020-06-12T02:37:36.779 INFO:tasks.rados.rados.0.smithi107.stdout:1273: delete oid 19 current snap is 0 2020-06-12T02:37:36.779 INFO:tasks.rados.rados.0.smithi107.stdout:1269: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:36.780 INFO:tasks.rados.rados.0.smithi107.stdout:1269: finishing write tid 6 to smithi10729407-16 2020-06-12T02:37:36.780 INFO:tasks.rados.rados.0.smithi107.stdout:1269: finishing write tid 7 to smithi10729407-16 2020-06-12T02:37:36.780 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 225 (ObjNum 1268 snap 0 seq_num 1268) dirty exists 2020-06-12T02:37:36.780 INFO:tasks.rados.rados.0.smithi107.stdout:1269: left oid 16 (ObjNum 1268 snap 0 seq_num 1268) 2020-06-12T02:37:36.781 INFO:tasks.rados.rados.0.smithi107.stdout:1272: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:36.781 INFO:tasks.rados.rados.0.smithi107.stdout:1272: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:36.782 INFO:tasks.rados.rados.0.smithi107.stdout:1269: done (2 left) 2020-06-12T02:37:36.782 INFO:tasks.rados.rados.0.smithi107.stdout:1273: done (1 left) 2020-06-12T02:37:36.782 INFO:tasks.rados.rados.0.smithi107.stdout:1274: delete oid 18 current snap is 0 2020-06-12T02:37:36.782 INFO:tasks.rados.rados.0.smithi107.stdout:1274: done (1 left) 2020-06-12T02:37:36.783 INFO:tasks.rados.rados.0.smithi107.stdout:1275: delete oid 8 current snap is 0 2020-06-12T02:37:36.784 INFO:tasks.rados.rados.0.smithi107.stdout:1272: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:36.784 INFO:tasks.rados.rados.0.smithi107.stdout:1275: done (1 left) 2020-06-12T02:37:36.785 INFO:tasks.rados.rados.0.smithi107.stdout:1276: delete oid 4 current snap is 0 2020-06-12T02:37:36.785 INFO:tasks.rados.rados.0.smithi107.stdout:1272: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:36.786 INFO:tasks.rados.rados.0.smithi107.stdout:1272: finishing write tid 6 to smithi10729407-6 2020-06-12T02:37:36.786 INFO:tasks.rados.rados.0.smithi107.stdout:1276: done (1 left) 2020-06-12T02:37:36.787 INFO:tasks.rados.rados.0.smithi107.stdout:1277: delete oid 8 current snap is 0 2020-06-12T02:37:36.788 INFO:tasks.rados.rados.0.smithi107.stdout:1272: finishing write tid 7 to smithi10729407-6 2020-06-12T02:37:36.788 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 252 (ObjNum 1271 snap 0 seq_num 1271) dirty exists 2020-06-12T02:37:36.789 INFO:tasks.rados.rados.0.smithi107.stdout:1272: left oid 6 (ObjNum 1271 snap 0 seq_num 1271) 2020-06-12T02:37:36.792 INFO:tasks.rados.rados.0.smithi107.stdout:1272: done (1 left) 2020-06-12T02:37:36.792 INFO:tasks.rados.rados.0.smithi107.stdout:1277: done (0 left) 2020-06-12T02:37:36.792 INFO:tasks.rados.rados.0.smithi107.stdout:1278: delete oid 10 current snap is 0 2020-06-12T02:37:36.795 INFO:tasks.rados.rados.0.smithi107.stdout:1278: done (0 left) 2020-06-12T02:37:36.795 INFO:tasks.rados.rados.0.smithi107.stdout:1279: delete oid 16 current snap is 0 2020-06-12T02:37:36.797 INFO:tasks.rados.rados.0.smithi107.stdout:1279: done (0 left) 2020-06-12T02:37:36.797 INFO:tasks.rados.rados.0.smithi107.stdout:1280: delete oid 8 current snap is 0 2020-06-12T02:37:36.801 INFO:tasks.rados.rados.0.smithi107.stdout:1280: done (0 left) 2020-06-12T02:37:36.801 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:37:36.802 INFO:tasks.rados.rados.0.smithi107.stdout:1281: seq_num 1280 ranges {0=647168,647168=696320,1343488=499712,1843200=409600} 2020-06-12T02:37:36.805 INFO:tasks.rados.rados.0.smithi107.stdout:1281: writing smithi10729407-10 from 0 to 647168 tid 1 2020-06-12T02:37:36.807 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:36.804+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:36.809 INFO:tasks.rados.rados.0.smithi107.stdout:1281: writing smithi10729407-10 from 647168 to 1343488 tid 2 2020-06-12T02:37:36.813 INFO:tasks.rados.rados.0.smithi107.stdout:1281: writing smithi10729407-10 from 1343488 to 1843200 tid 3 2020-06-12T02:37:36.815 INFO:tasks.rados.rados.0.smithi107.stdout:1281: writing smithi10729407-10 from 1843200 to 2252800 tid 4 2020-06-12T02:37:36.816 INFO:tasks.rados.rados.0.smithi107.stdout:1282: delete oid 13 current snap is 0 2020-06-12T02:37:36.816 INFO:tasks.rados.rados.0.smithi107.stdout:1281: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:36.817 INFO:tasks.rados.rados.0.smithi107.stdout:1282: done (1 left) 2020-06-12T02:37:36.817 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:37:36.818 INFO:tasks.rados.rados.0.smithi107.stdout:1283: seq_num 1282 ranges {0=679936,679936=466944,1146880=458752,1605632=622592,2228224=57344} 2020-06-12T02:37:36.823 INFO:tasks.rados.rados.0.smithi107.stdout:1283: writing smithi10729407-17 from 0 to 679936 tid 1 2020-06-12T02:37:36.825 INFO:tasks.rados.rados.0.smithi107.stdout:1283: writing smithi10729407-17 from 679936 to 1146880 tid 2 2020-06-12T02:37:36.827 INFO:tasks.rados.rados.0.smithi107.stdout:1283: writing smithi10729407-17 from 1146880 to 1605632 tid 3 2020-06-12T02:37:36.831 INFO:tasks.rados.rados.0.smithi107.stdout:1283: writing smithi10729407-17 from 1605632 to 2228224 tid 4 2020-06-12T02:37:36.833 INFO:tasks.rados.rados.0.smithi107.stdout:1283: writing smithi10729407-17 from 2228224 to 2285568 tid 5 2020-06-12T02:37:36.833 INFO:tasks.rados.rados.0.smithi107.stdout:1284: delete oid 4 current snap is 0 2020-06-12T02:37:36.833 INFO:tasks.rados.rados.0.smithi107.stdout:1281: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:36.834 INFO:tasks.rados.rados.0.smithi107.stdout:1281: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:36.834 INFO:tasks.rados.rados.0.smithi107.stdout:1281: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:36.834 INFO:tasks.rados.rados.0.smithi107.stdout:1281: finishing write tid 6 to smithi10729407-10 2020-06-12T02:37:36.834 INFO:tasks.rados.rados.0.smithi107.stdout:1281: finishing write tid 7 to smithi10729407-10 2020-06-12T02:37:36.835 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 119 (ObjNum 1280 snap 0 seq_num 1280) dirty exists 2020-06-12T02:37:36.835 INFO:tasks.rados.rados.0.smithi107.stdout:1281: left oid 10 (ObjNum 1280 snap 0 seq_num 1280) 2020-06-12T02:37:36.836 INFO:tasks.rados.rados.0.smithi107.stdout:1283: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:36.837 INFO:tasks.rados.rados.0.smithi107.stdout:1283: finishing write tid 2 to smithi10729407-17 2020-06-12T02:37:36.837 INFO:tasks.rados.rados.0.smithi107.stdout:1283: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:36.838 INFO:tasks.rados.rados.0.smithi107.stdout:1283: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:36.839 INFO:tasks.rados.rados.0.smithi107.stdout:1283: finishing write tid 5 to smithi10729407-17 2020-06-12T02:37:36.839 INFO:tasks.rados.rados.0.smithi107.stdout:1283: finishing write tid 7 to smithi10729407-17 2020-06-12T02:37:36.840 INFO:tasks.rados.rados.0.smithi107.stdout:1281: done (2 left) 2020-06-12T02:37:36.840 INFO:tasks.rados.rados.0.smithi107.stdout:1284: done (1 left) 2020-06-12T02:37:36.840 INFO:tasks.rados.rados.0.smithi107.stdout:1285: delete oid 8 current snap is 0 2020-06-12T02:37:36.841 INFO:tasks.rados.rados.0.smithi107.stdout:1283: finishing write tid 8 to smithi10729407-17 2020-06-12T02:37:36.841 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 245 (ObjNum 1282 snap 0 seq_num 1282) dirty exists 2020-06-12T02:37:36.842 INFO:tasks.rados.rados.0.smithi107.stdout:1283: left oid 17 (ObjNum 1282 snap 0 seq_num 1282) 2020-06-12T02:37:36.843 INFO:tasks.rados.rados.0.smithi107.stdout:1283: done (1 left) 2020-06-12T02:37:36.843 INFO:tasks.rados.rados.0.smithi107.stdout:1285: done (0 left) 2020-06-12T02:37:36.843 INFO:tasks.rados.rados.0.smithi107.stdout:1286: delete oid 10 current snap is 0 2020-06-12T02:37:36.845 INFO:tasks.rados.rados.0.smithi107.stdout:1286: done (0 left) 2020-06-12T02:37:36.845 INFO:tasks.rados.rados.0.smithi107.stdout:1287: delete oid 1 current snap is 0 2020-06-12T02:37:36.846 INFO:tasks.rados.rados.0.smithi107.stdout:1287: done (0 left) 2020-06-12T02:37:36.847 INFO:tasks.rados.rados.0.smithi107.stdout:1288: delete oid 16 current snap is 0 2020-06-12T02:37:36.849 INFO:tasks.rados.rados.0.smithi107.stdout:1288: done (0 left) 2020-06-12T02:37:36.849 INFO:tasks.rados.rados.0.smithi107.stdout:1289: delete oid 7 current snap is 0 2020-06-12T02:37:36.852 INFO:tasks.rados.rados.0.smithi107.stdout:1289: done (0 left) 2020-06-12T02:37:36.852 INFO:tasks.rados.rados.0.smithi107.stdout:1290: delete oid 14 current snap is 0 2020-06-12T02:37:36.854 INFO:tasks.rados.rados.0.smithi107.stdout:1290: done (0 left) 2020-06-12T02:37:36.854 INFO:tasks.rados.rados.0.smithi107.stdout:1291: delete oid 6 current snap is 0 2020-06-12T02:37:36.857 INFO:tasks.rados.rados.0.smithi107.stdout:1291: done (0 left) 2020-06-12T02:37:36.857 INFO:tasks.rados.rados.0.smithi107.stdout:1292: delete oid 19 current snap is 0 2020-06-12T02:37:36.860 INFO:tasks.rados.rados.0.smithi107.stdout:1292: done (0 left) 2020-06-12T02:37:36.860 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:37:36.861 INFO:tasks.rados.rados.0.smithi107.stdout:1293: seq_num 1292 ranges {0=40960} 2020-06-12T02:37:36.861 INFO:tasks.rados.rados.0.smithi107.stdout:1293: writing smithi10729407-6 from 0 to 40960 tid 1 2020-06-12T02:37:36.861 INFO:tasks.rados.rados.0.smithi107.stdout:1294: delete oid 17 current snap is 0 2020-06-12T02:37:36.862 INFO:tasks.rados.rados.0.smithi107.stdout:1293: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:36.865 INFO:tasks.rados.rados.0.smithi107.stdout:1294: done (1 left) 2020-06-12T02:37:36.865 INFO:tasks.rados.rados.0.smithi107.stdout:1295: delete oid 8 current snap is 0 2020-06-12T02:37:36.865 INFO:tasks.rados.rados.0.smithi107.stdout:1293: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:36.867 INFO:tasks.rados.rados.0.smithi107.stdout:1293: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:36.867 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 256 (ObjNum 1292 snap 0 seq_num 1292) dirty exists 2020-06-12T02:37:36.867 INFO:tasks.rados.rados.0.smithi107.stdout:1293: left oid 6 (ObjNum 1292 snap 0 seq_num 1292) 2020-06-12T02:37:36.867 INFO:tasks.rados.rados.0.smithi107.stdout:1293: done (1 left) 2020-06-12T02:37:36.867 INFO:tasks.rados.rados.0.smithi107.stdout:1295: done (0 left) 2020-06-12T02:37:36.868 INFO:tasks.rados.rados.0.smithi107.stdout:1296: delete oid 4 current snap is 0 2020-06-12T02:37:36.869 INFO:tasks.rados.rados.0.smithi107.stdout:1296: done (0 left) 2020-06-12T02:37:36.869 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:37:36.869 INFO:tasks.rados.rados.0.smithi107.stdout:1297: seq_num 1296 ranges {0=106496} 2020-06-12T02:37:36.869 INFO:tasks.rados.rados.0.smithi107.stdout:1297: writing smithi10729407-14 from 0 to 106496 tid 1 2020-06-12T02:37:36.870 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:37:36.870 INFO:tasks.rados.rados.0.smithi107.stdout:1298: seq_num 1297 ranges {0=122880} 2020-06-12T02:37:36.870 INFO:tasks.rados.rados.0.smithi107.stdout:1298: writing smithi10729407-1 from 0 to 122880 tid 1 2020-06-12T02:37:36.871 INFO:tasks.rados.rados.0.smithi107.stdout:1299: delete oid 4 current snap is 0 2020-06-12T02:37:36.871 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:37:36.872 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:37:36.872 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2020-06-12T02:37:36.872 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2020-06-12T02:37:36.872 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:37:36.872 INFO:tasks.rados.rados.0.smithi107.stdout:1299: done (2 left) 2020-06-12T02:37:36.873 INFO:tasks.rados.rados.0.smithi107.stdout:1300: delete oid 12 current snap is 0 2020-06-12T02:37:36.873 INFO:tasks.rados.rados.0.smithi107.stdout:1297: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:36.874 INFO:tasks.rados.rados.0.smithi107.stdout:1297: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:36.874 INFO:tasks.rados.rados.0.smithi107.stdout:1300: done (2 left) 2020-06-12T02:37:36.875 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:36.875 INFO:tasks.rados.rados.0.smithi107.stdout:1301: seq_num 1300 ranges {0=180224} 2020-06-12T02:37:36.875 INFO:tasks.rados.rados.0.smithi107.stdout:1301: writing smithi10729407-2 from 0 to 180224 tid 1 2020-06-12T02:37:36.876 INFO:tasks.rados.rados.0.smithi107.stdout:1302: delete oid 3 current snap is 0 2020-06-12T02:37:36.876 INFO:tasks.rados.rados.0.smithi107.stdout:1298: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:36.876 INFO:tasks.rados.rados.0.smithi107.stdout:1298: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:36.876 INFO:tasks.rados.rados.0.smithi107.stdout:1297: finishing write tid 4 to smithi10729407-14 2020-06-12T02:37:36.876 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 227 (ObjNum 1296 snap 0 seq_num 1296) dirty exists 2020-06-12T02:37:36.877 INFO:tasks.rados.rados.0.smithi107.stdout:1297: left oid 14 (ObjNum 1296 snap 0 seq_num 1296) 2020-06-12T02:37:36.877 INFO:tasks.rados.rados.0.smithi107.stdout:1298: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:36.877 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 258 (ObjNum 1297 snap 0 seq_num 1297) dirty exists 2020-06-12T02:37:36.877 INFO:tasks.rados.rados.0.smithi107.stdout:1298: left oid 1 (ObjNum 1297 snap 0 seq_num 1297) 2020-06-12T02:37:36.879 INFO:tasks.rados.rados.0.smithi107.stdout:1297: done (3 left) 2020-06-12T02:37:36.880 INFO:tasks.rados.rados.0.smithi107.stdout:1298: done (2 left) 2020-06-12T02:37:36.880 INFO:tasks.rados.rados.0.smithi107.stdout:1302: done (1 left) 2020-06-12T02:37:36.880 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:37:36.880 INFO:tasks.rados.rados.0.smithi107.stdout:1303: seq_num 1302 ranges {0=212992} 2020-06-12T02:37:36.881 INFO:tasks.rados.rados.0.smithi107.stdout:1303: writing smithi10729407-10 from 0 to 212992 tid 1 2020-06-12T02:37:36.881 INFO:tasks.rados.rados.0.smithi107.stdout:1304: delete oid 18 current snap is 0 2020-06-12T02:37:36.881 INFO:tasks.rados.rados.0.smithi107.stdout:1301: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:36.884 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-06-12T02:37:36.884 INFO:tasks.rados.rados.0.smithi107.stdout:1303: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:36.885 INFO:tasks.rados.rados.0.smithi107.stdout:1304: done (2 left) 2020-06-12T02:37:36.885 INFO:tasks.rados.rados.0.smithi107.stdout:1305: delete oid 16 current snap is 0 2020-06-12T02:37:36.886 INFO:tasks.rados.rados.0.smithi107.stdout:1301: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:36.886 INFO:tasks.rados.rados.0.smithi107.stdout:1303: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:36.901 INFO:tasks.rados.rados.0.smithi107.stdout:1305: done (2 left) 2020-06-12T02:37:36.901 INFO:tasks.rados.rados.0.smithi107.stdout:1306: delete oid 13 current snap is 0 2020-06-12T02:37:36.901 INFO:tasks.rados.rados.0.smithi107.stdout:1301: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:36.902 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 127 (ObjNum 1300 snap 0 seq_num 1300) dirty exists 2020-06-12T02:37:36.902 INFO:tasks.rados.rados.0.smithi107.stdout:1301: left oid 2 (ObjNum 1300 snap 0 seq_num 1300) 2020-06-12T02:37:36.902 INFO:tasks.rados.rados.0.smithi107.stdout:1303: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:36.902 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 122 (ObjNum 1302 snap 0 seq_num 1302) dirty exists 2020-06-12T02:37:36.902 INFO:tasks.rados.rados.0.smithi107.stdout:1303: left oid 10 (ObjNum 1302 snap 0 seq_num 1302) 2020-06-12T02:37:36.903 INFO:tasks.rados.rados.0.smithi107.stdout:1301: done (2 left) 2020-06-12T02:37:36.903 INFO:tasks.rados.rados.0.smithi107.stdout:1303: done (1 left) 2020-06-12T02:37:36.903 INFO:tasks.rados.rados.0.smithi107.stdout:1306: done (0 left) 2020-06-12T02:37:36.903 INFO:tasks.rados.rados.0.smithi107.stdout:1307: delete oid 1 current snap is 0 2020-06-12T02:37:36.903 INFO:tasks.rados.rados.0.smithi107.stdout:1307: done (0 left) 2020-06-12T02:37:36.904 INFO:tasks.rados.rados.0.smithi107.stdout:1308: delete oid 4 current snap is 0 2020-06-12T02:37:36.904 INFO:tasks.rados.rados.0.smithi107.stdout:1308: done (0 left) 2020-06-12T02:37:36.904 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:37:36.904 INFO:tasks.rados.rados.0.smithi107.stdout:1309: seq_num 1308 ranges {0=311296} 2020-06-12T02:37:36.904 INFO:tasks.rados.rados.0.smithi107.stdout:1309: writing smithi10729407-8 from 0 to 311296 tid 1 2020-06-12T02:37:36.904 INFO:tasks.rados.rados.0.smithi107.stdout:1310: delete oid 17 current snap is 0 2020-06-12T02:37:36.905 INFO:tasks.rados.rados.0.smithi107.stdout:1310: done (1 left) 2020-06-12T02:37:36.905 INFO:tasks.rados.rados.0.smithi107.stdout:1311: delete oid 19 current snap is 0 2020-06-12T02:37:36.905 INFO:tasks.rados.rados.0.smithi107.stdout:1311: done (1 left) 2020-06-12T02:37:36.905 INFO:tasks.rados.rados.0.smithi107.stdout:1312: delete oid 9 current snap is 0 2020-06-12T02:37:36.905 INFO:tasks.rados.rados.0.smithi107.stdout:1312: done (1 left) 2020-06-12T02:37:36.906 INFO:tasks.rados.rados.0.smithi107.stdout:1313: delete oid 2 current snap is 0 2020-06-12T02:37:36.906 INFO:tasks.rados.rados.0.smithi107.stdout:1309: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:36.906 INFO:tasks.rados.rados.0.smithi107.stdout:1313: done (1 left) 2020-06-12T02:37:36.907 INFO:tasks.rados.rados.0.smithi107.stdout:1314: delete oid 5 current snap is 0 2020-06-12T02:37:36.907 INFO:tasks.rados.rados.0.smithi107.stdout:1309: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:36.907 INFO:tasks.rados.rados.0.smithi107.stdout:1309: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:36.907 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 223 (ObjNum 1308 snap 0 seq_num 1308) dirty exists 2020-06-12T02:37:36.907 INFO:tasks.rados.rados.0.smithi107.stdout:1309: left oid 8 (ObjNum 1308 snap 0 seq_num 1308) 2020-06-12T02:37:36.908 INFO:tasks.rados.rados.0.smithi107.stdout:1309: done (1 left) 2020-06-12T02:37:36.908 INFO:tasks.rados.rados.0.smithi107.stdout:1314: done (0 left) 2020-06-12T02:37:36.911 INFO:tasks.rados.rados.0.smithi107.stdout:1315: delete oid 19 current snap is 0 2020-06-12T02:37:36.912 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:36.905+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:36.913 INFO:tasks.rados.rados.0.smithi107.stdout:1315: done (0 left) 2020-06-12T02:37:36.914 INFO:tasks.rados.rados.0.smithi107.stdout:1316: delete oid 13 current snap is 0 2020-06-12T02:37:36.916 INFO:tasks.rados.rados.0.smithi107.stdout:1316: done (0 left) 2020-06-12T02:37:36.916 INFO:tasks.rados.rados.0.smithi107.stdout:1317: delete oid 4 current snap is 0 2020-06-12T02:37:36.919 INFO:tasks.rados.rados.0.smithi107.stdout:1317: done (0 left) 2020-06-12T02:37:36.919 INFO:tasks.rados.rados.0.smithi107.stdout:1318: delete oid 4 current snap is 0 2020-06-12T02:37:36.920 INFO:tasks.rados.rados.0.smithi107.stdout:1318: done (0 left) 2020-06-12T02:37:36.920 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:37:36.920 INFO:tasks.rados.rados.0.smithi107.stdout:1319: seq_num 1318 ranges {0=483328} 2020-06-12T02:37:36.923 INFO:tasks.rados.rados.0.smithi107.stdout:1319: writing smithi10729407-12 from 0 to 483328 tid 1 2020-06-12T02:37:36.923 INFO:tasks.rados.rados.0.smithi107.stdout:1320: delete oid 10 current snap is 0 2020-06-12T02:37:36.924 INFO:tasks.rados.rados.0.smithi107.stdout:1320: done (1 left) 2020-06-12T02:37:36.925 INFO:tasks.rados.rados.0.smithi107.stdout:1321: delete oid 4 current snap is 0 2020-06-12T02:37:36.926 INFO:tasks.rados.rados.0.smithi107.stdout:1321: done (1 left) 2020-06-12T02:37:36.926 INFO:tasks.rados.rados.0.smithi107.stdout:1322: delete oid 17 current snap is 0 2020-06-12T02:37:36.928 INFO:tasks.rados.rados.0.smithi107.stdout:1322: done (1 left) 2020-06-12T02:37:36.929 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:37:36.929 INFO:tasks.rados.rados.0.smithi107.stdout:1323: seq_num 1322 ranges {0=548864} 2020-06-12T02:37:36.932 INFO:tasks.rados.rados.0.smithi107.stdout:1323: writing smithi10729407-2 from 0 to 548864 tid 1 2020-06-12T02:37:36.933 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:37:36.933 INFO:tasks.rados.rados.0.smithi107.stdout:1324: seq_num 1323 ranges {0=565248} 2020-06-12T02:37:36.937 INFO:tasks.rados.rados.0.smithi107.stdout:1324: writing smithi10729407-10 from 0 to 565248 tid 1 2020-06-12T02:37:36.938 INFO:tasks.rados.rados.0.smithi107.stdout:1325: delete oid 13 current snap is 0 2020-06-12T02:37:36.938 INFO:tasks.rados.rados.0.smithi107.stdout:1319: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:36.938 INFO:tasks.rados.rados.0.smithi107.stdout:1319: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:36.938 INFO:tasks.rados.rados.0.smithi107.stdout:1319: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:36.939 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 208 (ObjNum 1318 snap 0 seq_num 1318) dirty exists 2020-06-12T02:37:36.939 INFO:tasks.rados.rados.0.smithi107.stdout:1319: left oid 12 (ObjNum 1318 snap 0 seq_num 1318) 2020-06-12T02:37:36.939 INFO:tasks.rados.rados.0.smithi107.stdout:1323: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:36.940 INFO:tasks.rados.rados.0.smithi107.stdout:1323: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:36.940 INFO:tasks.rados.rados.0.smithi107.stdout:1319: done (3 left) 2020-06-12T02:37:36.941 INFO:tasks.rados.rados.0.smithi107.stdout:1325: done (2 left) 2020-06-12T02:37:36.941 INFO:tasks.rados.rados.0.smithi107.stdout:1326: delete oid 12 current snap is 0 2020-06-12T02:37:36.942 INFO:tasks.rados.rados.0.smithi107.stdout:1323: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:36.942 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 130 (ObjNum 1322 snap 0 seq_num 1322) dirty exists 2020-06-12T02:37:36.942 INFO:tasks.rados.rados.0.smithi107.stdout:1323: left oid 2 (ObjNum 1322 snap 0 seq_num 1322) 2020-06-12T02:37:36.943 INFO:tasks.rados.rados.0.smithi107.stdout:1323: done (2 left) 2020-06-12T02:37:36.943 INFO:tasks.rados.rados.0.smithi107.stdout:1326: done (1 left) 2020-06-12T02:37:36.943 INFO:tasks.rados.rados.0.smithi107.stdout:1327: delete oid 5 current snap is 0 2020-06-12T02:37:36.943 INFO:tasks.rados.rados.0.smithi107.stdout:1324: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:36.946 INFO:tasks.rados.rados.0.smithi107.stdout:1327: done (1 left) 2020-06-12T02:37:36.946 INFO:tasks.rados.rados.0.smithi107.stdout:1328: delete oid 4 current snap is 0 2020-06-12T02:37:36.948 INFO:tasks.rados.rados.0.smithi107.stdout:1328: done (1 left) 2020-06-12T02:37:36.948 INFO:tasks.rados.rados.0.smithi107.stdout:1329: delete oid 9 current snap is 0 2020-06-12T02:37:36.948 INFO:tasks.rados.rados.0.smithi107.stdout:1324: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:36.951 INFO:tasks.rados.rados.0.smithi107.stdout:1329: done (1 left) 2020-06-12T02:37:36.951 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:37:36.951 INFO:tasks.rados.rados.0.smithi107.stdout:1330: seq_num 1329 ranges {0=663552} 2020-06-12T02:37:36.955 INFO:tasks.rados.rados.0.smithi107.stdout:1330: writing smithi10729407-19 from 0 to 663552 tid 1 2020-06-12T02:37:36.955 INFO:tasks.rados.rados.0.smithi107.stdout:1331: delete oid 7 current snap is 0 2020-06-12T02:37:36.955 INFO:tasks.rados.rados.0.smithi107.stdout:1324: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:36.956 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 125 (ObjNum 1323 snap 0 seq_num 1323) dirty exists 2020-06-12T02:37:36.956 INFO:tasks.rados.rados.0.smithi107.stdout:1324: left oid 10 (ObjNum 1323 snap 0 seq_num 1323) 2020-06-12T02:37:36.958 INFO:tasks.rados.rados.0.smithi107.stdout:1324: done (2 left) 2020-06-12T02:37:36.958 INFO:tasks.rados.rados.0.smithi107.stdout:1331: done (1 left) 2020-06-12T02:37:36.958 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:37:36.958 INFO:tasks.rados.rados.0.smithi107.stdout:1332: seq_num 1331 ranges {0=696320} 2020-06-12T02:37:36.962 INFO:tasks.rados.rados.0.smithi107.stdout:1332: writing smithi10729407-1 from 0 to 696320 tid 1 2020-06-12T02:37:36.962 INFO:tasks.rados.rados.0.smithi107.stdout:1333: delete oid 14 current snap is 0 2020-06-12T02:37:36.962 INFO:tasks.rados.rados.0.smithi107.stdout:1330: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:36.963 INFO:tasks.rados.rados.0.smithi107.stdout:1330: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:36.964 INFO:tasks.rados.rados.0.smithi107.stdout:1330: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:36.964 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 211 (ObjNum 1329 snap 0 seq_num 1329) dirty exists 2020-06-12T02:37:36.964 INFO:tasks.rados.rados.0.smithi107.stdout:1330: left oid 19 (ObjNum 1329 snap 0 seq_num 1329) 2020-06-12T02:37:36.967 INFO:tasks.rados.rados.0.smithi107.stdout:1332: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:36.968 INFO:tasks.rados.rados.0.smithi107.stdout:1330: done (2 left) 2020-06-12T02:37:36.968 INFO:tasks.rados.rados.0.smithi107.stdout:1333: done (1 left) 2020-06-12T02:37:36.968 INFO:tasks.rados.rados.0.smithi107.stdout:1334: delete oid 16 current snap is 0 2020-06-12T02:37:36.973 INFO:tasks.rados.rados.0.smithi107.stdout:1334: done (1 left) 2020-06-12T02:37:36.973 INFO:tasks.rados.rados.0.smithi107.stdout:1335: delete oid 15 current snap is 0 2020-06-12T02:37:36.973 INFO:tasks.rados.rados.0.smithi107.stdout:1332: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:36.974 INFO:tasks.rados.rados.0.smithi107.stdout:1332: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:36.975 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 261 (ObjNum 1331 snap 0 seq_num 1331) dirty exists 2020-06-12T02:37:36.975 INFO:tasks.rados.rados.0.smithi107.stdout:1332: left oid 1 (ObjNum 1331 snap 0 seq_num 1331) 2020-06-12T02:37:36.975 INFO:tasks.rados.rados.0.smithi107.stdout:1332: done (1 left) 2020-06-12T02:37:36.976 INFO:tasks.rados.rados.0.smithi107.stdout:1335: done (0 left) 2020-06-12T02:37:36.976 INFO:tasks.rados.rados.0.smithi107.stdout:1336: delete oid 17 current snap is 0 2020-06-12T02:37:36.977 INFO:tasks.rados.rados.0.smithi107.stdout:1336: done (0 left) 2020-06-12T02:37:36.977 INFO:tasks.rados.rados.0.smithi107.stdout:1337: delete oid 4 current snap is 0 2020-06-12T02:37:36.978 INFO:tasks.rados.rados.0.smithi107.stdout:1337: done (0 left) 2020-06-12T02:37:36.979 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:37:36.979 INFO:tasks.rados.rados.0.smithi107.stdout:1338: seq_num 1337 ranges {0=802816} 2020-06-12T02:37:36.983 INFO:tasks.rados.rados.0.smithi107.stdout:1338: writing smithi10729407-18 from 0 to 802816 tid 1 2020-06-12T02:37:36.983 INFO:tasks.rados.rados.0.smithi107.stdout:1339: delete oid 9 current snap is 0 2020-06-12T02:37:36.988 INFO:tasks.rados.rados.0.smithi107.stdout:1338: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:36.991 INFO:tasks.rados.rados.0.smithi107.stdout:1339: done (1 left) 2020-06-12T02:37:36.991 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:37:36.991 INFO:tasks.rados.rados.0.smithi107.stdout:1340: seq_num 1339 ranges {663552=434176,1097728=401408} 2020-06-12T02:37:37.010 INFO:tasks.rados.rados.0.smithi107.stdout:1340: writing smithi10729407-19 from 663552 to 1097728 tid 1 2020-06-12T02:37:37.011 INFO:tasks.rados.rados.0.smithi107.stdout:1340: writing smithi10729407-19 from 1097728 to 1499136 tid 2 2020-06-12T02:37:37.011 INFO:tasks.rados.rados.0.smithi107.stdout:1341: delete oid 16 current snap is 0 2020-06-12T02:37:37.011 INFO:tasks.rados.rados.0.smithi107.stdout:1338: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:37.011 INFO:tasks.rados.rados.0.smithi107.stdout:1338: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:37.012 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 248 (ObjNum 1337 snap 0 seq_num 1337) dirty exists 2020-06-12T02:37:37.012 INFO:tasks.rados.rados.0.smithi107.stdout:1338: left oid 18 (ObjNum 1337 snap 0 seq_num 1337) 2020-06-12T02:37:37.012 INFO:tasks.rados.rados.0.smithi107.stdout:1338: done (2 left) 2020-06-12T02:37:37.012 INFO:tasks.rados.rados.0.smithi107.stdout:1341: done (1 left) 2020-06-12T02:37:37.013 INFO:tasks.rados.rados.0.smithi107.stdout:1342: delete oid 9 current snap is 0 2020-06-12T02:37:37.013 INFO:tasks.rados.rados.0.smithi107.stdout:1340: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:37.013 INFO:tasks.rados.rados.0.smithi107.stdout:1342: done (1 left) 2020-06-12T02:37:37.013 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:37:37.013 INFO:tasks.rados.rados.0.smithi107.stdout:1343: seq_num 1342 ranges {0=483328,483328=401408} 2020-06-12T02:37:37.013 INFO:tasks.rados.rados.0.smithi107.stdout:1343: writing smithi10729407-12 from 0 to 483328 tid 1 2020-06-12T02:37:37.017 INFO:tasks.rados.rados.0.smithi107.stdout:1343: writing smithi10729407-12 from 483328 to 884736 tid 2 2020-06-12T02:37:37.017 INFO:tasks.rados.rados.0.smithi107.stdout:1344: delete oid 16 current snap is 0 2020-06-12T02:37:37.018 INFO:tasks.rados.rados.0.smithi107.stdout:1340: finishing write tid 2 to smithi10729407-19 2020-06-12T02:37:37.018 INFO:tasks.rados.rados.0.smithi107.stdout:1340: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:37.018 INFO:tasks.rados.rados.0.smithi107.stdout:1340: finishing write tid 5 to smithi10729407-19 2020-06-12T02:37:37.018 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 214 (ObjNum 1339 snap 0 seq_num 1339) dirty exists 2020-06-12T02:37:37.018 INFO:tasks.rados.rados.0.smithi107.stdout:1340: left oid 19 (ObjNum 1339 snap 0 seq_num 1339) 2020-06-12T02:37:37.020 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:37.006+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:37.020 INFO:tasks.rados.rados.0.smithi107.stdout:1340: done (2 left) 2020-06-12T02:37:37.020 INFO:tasks.rados.rados.0.smithi107.stdout:1344: done (1 left) 2020-06-12T02:37:37.020 INFO:tasks.rados.rados.0.smithi107.stdout:1345: delete oid 16 current snap is 0 2020-06-12T02:37:37.021 INFO:tasks.rados.rados.0.smithi107.stdout:1343: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:37.021 INFO:tasks.rados.rados.0.smithi107.stdout:1345: done (1 left) 2020-06-12T02:37:37.021 INFO:tasks.rados.rados.0.smithi107.stdout:1346: delete oid 11 current snap is 0 2020-06-12T02:37:37.021 INFO:tasks.rados.rados.0.smithi107.stdout:1346: done (1 left) 2020-06-12T02:37:37.021 INFO:tasks.rados.rados.0.smithi107.stdout:1347: delete oid 2 current snap is 0 2020-06-12T02:37:37.022 INFO:tasks.rados.rados.0.smithi107.stdout:1347: done (1 left) 2020-06-12T02:37:37.022 INFO:tasks.rados.rados.0.smithi107.stdout:1348: delete oid 15 current snap is 0 2020-06-12T02:37:37.022 INFO:tasks.rados.rados.0.smithi107.stdout:1343: finishing write tid 2 to smithi10729407-12 2020-06-12T02:37:37.022 INFO:tasks.rados.rados.0.smithi107.stdout:1343: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:37.022 INFO:tasks.rados.rados.0.smithi107.stdout:1348: done (1 left) 2020-06-12T02:37:37.022 INFO:tasks.rados.rados.0.smithi107.stdout:1349: delete oid 3 current snap is 0 2020-06-12T02:37:37.023 INFO:tasks.rados.rados.0.smithi107.stdout:1343: finishing write tid 5 to smithi10729407-12 2020-06-12T02:37:37.023 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 217 (ObjNum 1342 snap 0 seq_num 1342) dirty exists 2020-06-12T02:37:37.023 INFO:tasks.rados.rados.0.smithi107.stdout:1343: left oid 12 (ObjNum 1342 snap 0 seq_num 1342) 2020-06-12T02:37:37.024 INFO:tasks.rados.rados.0.smithi107.stdout:1343: done (1 left) 2020-06-12T02:37:37.024 INFO:tasks.rados.rados.0.smithi107.stdout:1349: done (0 left) 2020-06-12T02:37:37.024 INFO:tasks.rados.rados.0.smithi107.stdout:1350: delete oid 10 current snap is 0 2020-06-12T02:37:37.024 INFO:tasks.rados.rados.0.smithi107.stdout:1350: done (0 left) 2020-06-12T02:37:37.024 INFO:tasks.rados.rados.0.smithi107.stdout:1351: delete oid 13 current snap is 0 2020-06-12T02:37:37.025 INFO:tasks.rados.rados.0.smithi107.stdout:1351: done (0 left) 2020-06-12T02:37:37.036 INFO:tasks.rados.rados.0.smithi107.stdout:1352: delete oid 10 current snap is 0 2020-06-12T02:37:37.038 INFO:tasks.rados.rados.0.smithi107.stdout:1352: done (0 left) 2020-06-12T02:37:37.038 INFO:tasks.rados.rados.0.smithi107.stdout:1353: delete oid 14 current snap is 0 2020-06-12T02:37:37.038 INFO:tasks.rados.rados.0.smithi107.stdout:1353: done (0 left) 2020-06-12T02:37:37.038 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:37:37.039 INFO:tasks.rados.rados.0.smithi107.stdout:1354: seq_num 1353 ranges {0=663552,663552=401408} 2020-06-12T02:37:37.039 INFO:tasks.rados.rados.0.smithi107.stdout:1354: writing smithi10729407-16 from 0 to 663552 tid 1 2020-06-12T02:37:37.039 INFO:tasks.rados.rados.0.smithi107.stdout:1354: writing smithi10729407-16 from 663552 to 1064960 tid 2 2020-06-12T02:37:37.039 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:37:37.041 INFO:tasks.rados.rados.0.smithi107.stdout:1355: seq_num 1354 ranges {696320=679936,1376256=401408} 2020-06-12T02:37:37.041 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:37:37.042 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2020-06-12T02:37:37.042 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2020-06-12T02:37:37.042 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2020-06-12T02:37:37.042 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.042 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s2 25/21 ECSubWrite(tid=277, reqid=client.4512.0:1102, at_version=25'61, trim_to=0'0, roll_forward_to=25'60))", 2020-06-12T02:37:37.043 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:32.593596+0000", 2020-06-12T02:37:37.043 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 4.4328641040000001, 2020-06-12T02:37:37.043 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0072991619999999997, 2020-06-12T02:37:37.043 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:37.043 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:37.043 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:37.044 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.044 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:37.044 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.593596+0000", 2020-06-12T02:37:37.044 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:37.044 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.045 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.045 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:37.045 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.593596+0000", 2020-06-12T02:37:37.045 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054472400000000001 2020-06-12T02:37:37.046 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.046 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.046 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:37.047 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.594140+0000", 2020-06-12T02:37:37.047 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5156e-05 2020-06-12T02:37:37.047 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.047 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.047 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:37.047 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.594155+0000", 2020-06-12T02:37:37.048 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8709999999999992e-06 2020-06-12T02:37:37.048 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.048 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.048 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:37.048 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.594165+0000", 2020-06-12T02:37:37.049 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3549999999999998e-06 2020-06-12T02:37:37.049 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.049 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.049 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:37.049 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.594170+0000", 2020-06-12T02:37:37.049 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6307999999999998e-05 2020-06-12T02:37:37.050 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.050 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.050 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.050 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.594216+0000", 2020-06-12T02:37:37.050 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1255e-05 2020-06-12T02:37:37.051 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.051 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.051 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:37.051 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.594227+0000", 2020-06-12T02:37:37.051 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0066139500000000004 2020-06-12T02:37:37.051 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.052 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.052 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:37.052 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.600841+0000", 2020-06-12T02:37:37.052 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3542999999999999e-05 2020-06-12T02:37:37.052 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.053 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.053 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:37.053 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.600895+0000", 2020-06-12T02:37:37.053 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0072991619999999997 2020-06-12T02:37:37.053 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.054 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:37.054 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.054 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.054 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.054 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s2 25/21 ECSubWrite(tid=314, reqid=client.4512.0:1218, at_version=25'74, trim_to=0'0, roll_forward_to=25'73))", 2020-06-12T02:37:37.056 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:32.917299+0000", 2020-06-12T02:37:37.056 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 4.109160718, 2020-06-12T02:37:37.056 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0066074560000000003, 2020-06-12T02:37:37.057 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:37.057 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:37.057 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:37.057 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.057 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:37.058 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.917299+0000", 2020-06-12T02:37:37.058 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:37.058 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.058 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.058 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:37.059 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.917299+0000", 2020-06-12T02:37:37.059 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037899299999999999 2020-06-12T02:37:37.059 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.059 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.059 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:37.060 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.917678+0000", 2020-06-12T02:37:37.060 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6007e-05 2020-06-12T02:37:37.060 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.060 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.060 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:37.061 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.917704+0000", 2020-06-12T02:37:37.061 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6501e-05 2020-06-12T02:37:37.061 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.061 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.061 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:37.062 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.917720+0000", 2020-06-12T02:37:37.062 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9559e-05 2020-06-12T02:37:37.062 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.062 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.063 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:37.063 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.917740+0000", 2020-06-12T02:37:37.063 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000151211 2020-06-12T02:37:37.063 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.063 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.064 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.064 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.917891+0000", 2020-06-12T02:37:37.064 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6090000000000003e-06 2020-06-12T02:37:37.064 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.064 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.064 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:37.065 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.917899+0000", 2020-06-12T02:37:37.065 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0059626610000000002 2020-06-12T02:37:37.065 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.065 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.065 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:37.066 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.923861+0000", 2020-06-12T02:37:37.066 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4914999999999999e-05 2020-06-12T02:37:37.066 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.066 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.067 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:37.067 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.923906+0000", 2020-06-12T02:37:37.067 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0066074560000000003 2020-06-12T02:37:37.067 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.067 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:37.068 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.068 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.068 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.068 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.9s1 25/21 ECSubWrite(tid=615, reqid=client.4512.0:1285, at_version=25'85, trim_to=0'0, roll_forward_to=25'84))", 2020-06-12T02:37:37.068 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.117566+0000", 2020-06-12T02:37:37.068 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.908894037, 2020-06-12T02:37:37.069 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010635947999999999, 2020-06-12T02:37:37.069 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:37.069 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:37.069 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:37.069 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.070 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:37.070 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.117566+0000", 2020-06-12T02:37:37.070 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:37.070 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.070 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.071 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:37.071 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.117566+0000", 2020-06-12T02:37:37.071 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000451781 2020-06-12T02:37:37.071 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.071 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.071 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:37.072 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.118017+0000", 2020-06-12T02:37:37.072 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9958999999999999e-05 2020-06-12T02:37:37.072 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.072 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.072 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:37.073 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.118047+0000", 2020-06-12T02:37:37.073 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.694e-05 2020-06-12T02:37:37.073 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.073 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.073 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:37.073 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.118064+0000", 2020-06-12T02:37:37.074 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7629999999999995e-06 2020-06-12T02:37:37.074 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.074 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.074 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:37.074 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.118074+0000", 2020-06-12T02:37:37.075 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1209000000000003e-05 2020-06-12T02:37:37.075 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.075 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.075 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.075 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.118135+0000", 2020-06-12T02:37:37.075 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6164999999999999e-05 2020-06-12T02:37:37.076 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.076 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.076 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:37.076 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.118151+0000", 2020-06-12T02:37:37.076 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0099907599999999996 2020-06-12T02:37:37.077 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.077 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.077 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:37.077 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.128142+0000", 2020-06-12T02:37:37.078 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9370999999999998e-05 2020-06-12T02:37:37.078 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.078 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.078 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:37.078 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.128202+0000", 2020-06-12T02:37:37.079 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010635947999999999 2020-06-12T02:37:37.079 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.079 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:37.079 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.079 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.079 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.080 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.9s1 25/21 ECSubWrite(tid=616, reqid=client.4512.0:1286, at_version=25'86, trim_to=0'0, roll_forward_to=25'84))", 2020-06-12T02:37:37.080 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.120870+0000", 2020-06-12T02:37:37.080 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.9055898679999999, 2020-06-12T02:37:37.080 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0088027680000000007, 2020-06-12T02:37:37.080 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:37.081 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:37.081 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:37.081 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.081 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:37.081 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.120870+0000", 2020-06-12T02:37:37.082 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:37.082 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.082 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.082 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:37.082 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.120870+0000", 2020-06-12T02:37:37.082 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041078300000000001 2020-06-12T02:37:37.083 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.083 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.083 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:37.083 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.121281+0000", 2020-06-12T02:37:37.083 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3048e-05 2020-06-12T02:37:37.084 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.084 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.084 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:37.084 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.121304+0000", 2020-06-12T02:37:37.084 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9318e-05 2020-06-12T02:37:37.085 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.085 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.085 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:37.085 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.121323+0000", 2020-06-12T02:37:37.086 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2228999999999999e-05 2020-06-12T02:37:37.086 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.086 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.086 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:37.086 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.121335+0000", 2020-06-12T02:37:37.086 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0010068500000000001 2020-06-12T02:37:37.087 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.087 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.087 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.088 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.122342+0000", 2020-06-12T02:37:37.088 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2475e-05 2020-06-12T02:37:37.088 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.088 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.088 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:37.088 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.122354+0000", 2020-06-12T02:37:37.089 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.007283884 2020-06-12T02:37:37.089 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.089 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.089 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:37.089 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.129638+0000", 2020-06-12T02:37:37.095 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4180999999999997e-05 2020-06-12T02:37:37.095 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.096 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.096 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:37.096 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.129673+0000", 2020-06-12T02:37:37.096 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0088027680000000007 2020-06-12T02:37:37.096 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.096 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:37.097 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.097 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.097 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.098 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.9s1 25/21 ECSubWrite(tid=623, reqid=client.4512.0:1292, at_version=25'91, trim_to=0'0, roll_forward_to=25'90))", 2020-06-12T02:37:37.098 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.136725+0000", 2020-06-12T02:37:37.098 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.8897343000000002, 2020-06-12T02:37:37.098 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0061581780000000003, 2020-06-12T02:37:37.098 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:37.099 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:37.099 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:37.099 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.099 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:37.099 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.136725+0000", 2020-06-12T02:37:37.099 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:37.100 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.100 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.100 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:37.100 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.136725+0000", 2020-06-12T02:37:37.100 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030685599999999999 2020-06-12T02:37:37.101 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.101 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.101 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:37.101 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.137032+0000", 2020-06-12T02:37:37.102 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7299000000000002e-05 2020-06-12T02:37:37.102 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.102 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.102 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:37.102 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.137049+0000", 2020-06-12T02:37:37.102 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0912e-05 2020-06-12T02:37:37.103 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.103 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.103 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:37.103 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.137060+0000", 2020-06-12T02:37:37.103 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9330000000000001e-06 2020-06-12T02:37:37.104 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.104 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.104 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:37.104 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.137068+0000", 2020-06-12T02:37:37.104 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2169000000000002e-05 2020-06-12T02:37:37.104 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.105 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.105 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.105 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.137110+0000", 2020-06-12T02:37:37.105 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2078e-05 2020-06-12T02:37:37.105 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.106 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.106 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:37.106 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.137123+0000", 2020-06-12T02:37:37.106 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0057026129999999996 2020-06-12T02:37:37.106 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.106 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.107 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:37.107 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.142825+0000", 2020-06-12T02:37:37.107 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8318e-05 2020-06-12T02:37:37.107 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.107 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.108 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:37.108 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.142883+0000", 2020-06-12T02:37:37.108 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0061581780000000003 2020-06-12T02:37:37.108 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.109 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:37.109 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.109 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.109 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.109 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.9s1 25/21 ECSubWrite(tid=624, reqid=client.4512.0:1293, at_version=25'92, trim_to=0'0, roll_forward_to=25'90))", 2020-06-12T02:37:37.110 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.139378+0000", 2020-06-12T02:37:37.110 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.8870811700000001, 2020-06-12T02:37:37.110 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0060310509999999999, 2020-06-12T02:37:37.110 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:37.110 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:37.111 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:37.111 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.111 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:37.111 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.139378+0000", 2020-06-12T02:37:37.111 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:37.111 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.112 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.112 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:37.112 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.139378+0000", 2020-06-12T02:37:37.112 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00026027 2020-06-12T02:37:37.112 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.113 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.113 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:37.113 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.139639+0000", 2020-06-12T02:37:37.113 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3728999999999999e-05 2020-06-12T02:37:37.113 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.114 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.114 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:37.114 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.139662+0000", 2020-06-12T02:37:37.114 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4942e-05 2020-06-12T02:37:37.114 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.115 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.115 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:37.115 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.139677+0000", 2020-06-12T02:37:37.115 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0480000000000001e-06 2020-06-12T02:37:37.115 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.116 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.116 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:37.116 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.139684+0000", 2020-06-12T02:37:37.116 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6037e-05 2020-06-12T02:37:37.116 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.116 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.117 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.117 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.139730+0000", 2020-06-12T02:37:37.117 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2379e-05 2020-06-12T02:37:37.117 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.117 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.118 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:37.120 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.139743+0000", 2020-06-12T02:37:37.120 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0056246910000000002 2020-06-12T02:37:37.120 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.121 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.121 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:37.122 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.145368+0000", 2020-06-12T02:37:37.123 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1955000000000001e-05 2020-06-12T02:37:37.123 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.123 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.123 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:37.124 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.145409+0000", 2020-06-12T02:37:37.124 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0060310509999999999 2020-06-12T02:37:37.124 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.124 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:37.125 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.125 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.126 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.126 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s2 26/21 ECSubWrite(tid=383, reqid=client.4512.0:1481, at_version=26'94, trim_to=0'0, roll_forward_to=26'93))", 2020-06-12T02:37:37.126 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.611544+0000", 2020-06-12T02:37:37.126 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.4149157109999999, 2020-06-12T02:37:37.127 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0062564580000000003, 2020-06-12T02:37:37.127 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:37.127 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:37.128 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:37.128 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.128 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:37.129 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.611544+0000", 2020-06-12T02:37:37.129 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:37.129 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.130 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.130 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:37.130 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.611544+0000", 2020-06-12T02:37:37.131 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00045840699999999998 2020-06-12T02:37:37.131 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.131 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.131 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:37.132 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.612002+0000", 2020-06-12T02:37:37.132 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6559000000000001e-05 2020-06-12T02:37:37.132 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.132 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.133 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:37.133 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.612029+0000", 2020-06-12T02:37:37.133 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9573e-05 2020-06-12T02:37:37.134 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.134 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.135 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:37.135 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.612048+0000", 2020-06-12T02:37:37.136 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7880000000000007e-06 2020-06-12T02:37:37.136 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.136 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.136 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:37.136 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.612056+0000", 2020-06-12T02:37:37.137 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4539999999999997e-05 2020-06-12T02:37:37.138 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.138 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.138 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.139 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.612101+0000", 2020-06-12T02:37:37.139 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2279999999999998e-06 2020-06-12T02:37:37.139 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.139 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.139 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:37.140 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.612109+0000", 2020-06-12T02:37:37.140 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0056091930000000002 2020-06-12T02:37:37.140 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.140 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.141 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:37.141 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.617718+0000", 2020-06-12T02:37:37.141 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2169999999999994e-05 2020-06-12T02:37:37.141 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.141 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.141 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:37.142 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.617800+0000", 2020-06-12T02:37:37.142 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0062564580000000003 2020-06-12T02:37:37.142 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.142 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:37.142 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.143 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.143 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.143 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s2 26/21 ECSubWrite(tid=408, reqid=client.4512.0:1572, at_version=26'99, trim_to=0'0, roll_forward_to=26'97))", 2020-06-12T02:37:37.143 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.887897+0000", 2020-06-12T02:37:37.144 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.1385626759999998, 2020-06-12T02:37:37.144 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0098940590000000002, 2020-06-12T02:37:37.144 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:37.144 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:37.144 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:37.145 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.145 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:37.145 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.887897+0000", 2020-06-12T02:37:37.145 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:37.145 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.145 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.146 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:37.146 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.887897+0000", 2020-06-12T02:37:37.146 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052976499999999997 2020-06-12T02:37:37.147 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.147 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.147 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:37.147 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888427+0000", 2020-06-12T02:37:37.148 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1276000000000001e-05 2020-06-12T02:37:37.148 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.148 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.148 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:37.148 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888448+0000", 2020-06-12T02:37:37.149 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7121000000000001e-05 2020-06-12T02:37:37.149 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.149 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.150 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:37.150 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888465+0000", 2020-06-12T02:37:37.150 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0490000000000004e-06 2020-06-12T02:37:37.151 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.151 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.151 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:37.151 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888473+0000", 2020-06-12T02:37:37.151 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6770999999999997e-05 2020-06-12T02:37:37.152 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.152 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.152 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.152 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888510+0000", 2020-06-12T02:37:37.152 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7649999999999993e-06 2020-06-12T02:37:37.152 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.153 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.153 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:37.153 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888518+0000", 2020-06-12T02:37:37.153 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0092201669999999996 2020-06-12T02:37:37.153 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.154 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.154 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:37.154 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.897738+0000", 2020-06-12T02:37:37.154 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3145e-05 2020-06-12T02:37:37.154 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.155 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.155 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:37.155 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.897791+0000", 2020-06-12T02:37:37.155 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0098940590000000002 2020-06-12T02:37:37.155 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.155 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:37.156 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.156 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.156 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.156 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s2 26/21 ECSubWrite(tid=409, reqid=client.4512.0:1573, at_version=26'100, trim_to=0'0, roll_forward_to=26'98))", 2020-06-12T02:37:37.156 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.892374+0000", 2020-06-12T02:37:37.157 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.1340856750000001, 2020-06-12T02:37:37.157 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014071322000000001, 2020-06-12T02:37:37.157 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:37.157 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:37.157 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:37.157 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.158 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:37.158 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.892374+0000", 2020-06-12T02:37:37.158 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:37.158 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.158 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.159 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:37.159 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.892374+0000", 2020-06-12T02:37:37.159 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047864199999999999 2020-06-12T02:37:37.159 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.159 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.160 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:37.160 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.892853+0000", 2020-06-12T02:37:37.160 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5139000000000001e-05 2020-06-12T02:37:37.160 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.160 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.160 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:37.161 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.892878+0000", 2020-06-12T02:37:37.161 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1543999999999999e-05 2020-06-12T02:37:37.161 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.161 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.161 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:37.162 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.892899+0000", 2020-06-12T02:37:37.162 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8989999999999996e-06 2020-06-12T02:37:37.162 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.162 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.162 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:37.162 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.892909+0000", 2020-06-12T02:37:37.163 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00023009199999999999 2020-06-12T02:37:37.163 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.163 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.163 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.163 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.893139+0000", 2020-06-12T02:37:37.164 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0044000000000001e-05 2020-06-12T02:37:37.164 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.164 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.164 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:37.164 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.893149+0000", 2020-06-12T02:37:37.164 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013233405 2020-06-12T02:37:37.165 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.165 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.165 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:37.165 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.906383+0000", 2020-06-12T02:37:37.165 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2557000000000004e-05 2020-06-12T02:37:37.166 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.166 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.166 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:37.166 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.906445+0000", 2020-06-12T02:37:37.166 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014071322000000001 2020-06-12T02:37:37.167 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.167 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:37.167 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.167 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.167 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.167 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s2 26/21 ECSubWrite(tid=410, reqid=client.4512.0:1574, at_version=26'101, trim_to=0'0, roll_forward_to=26'98))", 2020-06-12T02:37:37.168 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.895487+0000", 2020-06-12T02:37:37.168 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.1309728799999998, 2020-06-12T02:37:37.168 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012904807000000001, 2020-06-12T02:37:37.168 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:37.169 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:37.169 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:37.169 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.169 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:37.169 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895487+0000", 2020-06-12T02:37:37.170 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:37.170 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.170 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.170 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:37.170 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895487+0000", 2020-06-12T02:37:37.171 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8478000000000002e-05 2020-06-12T02:37:37.171 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.171 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.171 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:37.171 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895575+0000", 2020-06-12T02:37:37.172 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8672e-05 2020-06-12T02:37:37.172 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.172 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.172 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:37.172 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895594+0000", 2020-06-12T02:37:37.173 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8028000000000001e-05 2020-06-12T02:37:37.173 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.173 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.173 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:37.174 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895612+0000", 2020-06-12T02:37:37.174 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2689999999999999e-06 2020-06-12T02:37:37.174 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.174 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.174 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:37.174 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895620+0000", 2020-06-12T02:37:37.175 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0038940289999999998 2020-06-12T02:37:37.175 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.175 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.175 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.175 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.899514+0000", 2020-06-12T02:37:37.176 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4370000000000001e-06 2020-06-12T02:37:37.176 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.176 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.176 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:37.176 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.899522+0000", 2020-06-12T02:37:37.176 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0088184019999999995 2020-06-12T02:37:37.177 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.177 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.177 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:37.177 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.908340+0000", 2020-06-12T02:37:37.177 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1492000000000001e-05 2020-06-12T02:37:37.178 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.178 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.178 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:37.178 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.908392+0000", 2020-06-12T02:37:37.178 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012904807000000001 2020-06-12T02:37:37.178 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.179 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:37.179 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.179 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.179 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.179 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s2 26/21 ECSubWrite(tid=411, reqid=client.4512.0:1575, at_version=26'102, trim_to=0'0, roll_forward_to=26'98))", 2020-06-12T02:37:37.180 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.899359+0000", 2020-06-12T02:37:37.180 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.1271006890000002, 2020-06-12T02:37:37.180 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.009071862, 2020-06-12T02:37:37.180 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:37.180 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:37.180 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:37.181 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.181 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:37.181 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.899359+0000", 2020-06-12T02:37:37.181 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:37.181 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.182 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.182 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:37.182 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.899359+0000", 2020-06-12T02:37:37.182 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3011999999999999e-05 2020-06-12T02:37:37.182 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.183 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.183 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:37.183 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.899422+0000", 2020-06-12T02:37:37.183 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3204999999999999e-05 2020-06-12T02:37:37.183 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.183 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.184 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:37.184 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.899445+0000", 2020-06-12T02:37:37.184 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8564999999999999e-05 2020-06-12T02:37:37.184 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.184 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.185 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:37.185 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.899464+0000", 2020-06-12T02:37:37.185 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6259999999999997e-06 2020-06-12T02:37:37.185 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.185 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.185 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:37.186 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.899470+0000", 2020-06-12T02:37:37.186 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00096442999999999995 2020-06-12T02:37:37.186 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.186 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.186 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.187 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.900435+0000", 2020-06-12T02:37:37.187 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1058e-05 2020-06-12T02:37:37.187 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.187 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.187 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:37.188 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.900446+0000", 2020-06-12T02:37:37.188 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0079506830000000001 2020-06-12T02:37:37.188 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.188 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.188 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:37.189 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.908397+0000", 2020-06-12T02:37:37.189 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4282999999999997e-05 2020-06-12T02:37:37.189 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.189 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.189 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:37.190 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.908431+0000", 2020-06-12T02:37:37.190 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.009071862 2020-06-12T02:37:37.190 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.190 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:37.190 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.190 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.191 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.191 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 27/21 ECSubWrite(tid=169, reqid=client.4512.0:2720, at_version=27'109, trim_to=0'0, roll_forward_to=27'107))", 2020-06-12T02:37:37.191 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:36.649813+0000", 2020-06-12T02:37:37.191 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.37664709800000001, 2020-06-12T02:37:37.191 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0086910270000000005, 2020-06-12T02:37:37.192 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:37.192 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:37.192 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:37.192 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.192 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:37.192 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.649813+0000", 2020-06-12T02:37:37.193 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:37.193 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.193 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.193 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:37.193 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.649813+0000", 2020-06-12T02:37:37.194 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018473400000000001 2020-06-12T02:37:37.194 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.194 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.194 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:37.194 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.649997+0000", 2020-06-12T02:37:37.194 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0846999999999998e-05 2020-06-12T02:37:37.195 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.195 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.195 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:37.195 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.650028+0000", 2020-06-12T02:37:37.195 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9769999999999999e-05 2020-06-12T02:37:37.196 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.196 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.196 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:37.196 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.650048+0000", 2020-06-12T02:37:37.196 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8e-05 2020-06-12T02:37:37.196 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.197 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.197 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:37.197 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.650066+0000", 2020-06-12T02:37:37.197 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9342999999999997e-05 2020-06-12T02:37:37.197 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.197 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.198 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.198 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.650125+0000", 2020-06-12T02:37:37.198 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3171e-05 2020-06-12T02:37:37.198 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.198 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.199 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:37.199 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.650138+0000", 2020-06-12T02:37:37.199 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0082900739999999997 2020-06-12T02:37:37.199 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.199 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.200 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:37.200 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.658428+0000", 2020-06-12T02:37:37.200 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5087999999999997e-05 2020-06-12T02:37:37.200 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.200 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.200 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:37.201 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.658504+0000", 2020-06-12T02:37:37.201 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0086910270000000005 2020-06-12T02:37:37.201 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.202 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:37.202 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.202 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.203 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.203 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 27/21 ECSubWrite(tid=170, reqid=client.4512.0:2721, at_version=27'110, trim_to=0'0, roll_forward_to=27'107))", 2020-06-12T02:37:37.203 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:36.654148+0000", 2020-06-12T02:37:37.203 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.37231154700000002, 2020-06-12T02:37:37.203 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0096132549999999994, 2020-06-12T02:37:37.204 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:37.204 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:37.204 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:37.204 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.204 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:37.204 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.654148+0000", 2020-06-12T02:37:37.205 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:37.205 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.205 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.205 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:37.205 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.654148+0000", 2020-06-12T02:37:37.206 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00027898700000000002 2020-06-12T02:37:37.206 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.206 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.206 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:37.206 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.654427+0000", 2020-06-12T02:37:37.207 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.7942000000000001e-05 2020-06-12T02:37:37.207 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.207 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.207 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:37.207 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.654455+0000", 2020-06-12T02:37:37.207 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8269000000000001e-05 2020-06-12T02:37:37.208 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.208 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.208 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:37.208 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.654473+0000", 2020-06-12T02:37:37.208 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0450000000000003e-06 2020-06-12T02:37:37.209 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.209 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.209 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:37.209 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.654480+0000", 2020-06-12T02:37:37.209 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013121000000000001 2020-06-12T02:37:37.210 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.210 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.210 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.210 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.654612+0000", 2020-06-12T02:37:37.210 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.167e-05 2020-06-12T02:37:37.210 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.211 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.211 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:37.211 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.654623+0000", 2020-06-12T02:37:37.211 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090820680000000004 2020-06-12T02:37:37.211 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.212 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.212 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:37.212 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.663705+0000", 2020-06-12T02:37:37.212 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6063999999999997e-05 2020-06-12T02:37:37.212 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.212 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.213 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:37.213 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.663761+0000", 2020-06-12T02:37:37.213 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0096132549999999994 2020-06-12T02:37:37.213 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.213 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:37.214 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.214 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.214 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.214 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 27/21 ECSubWrite(tid=171, reqid=client.4512.0:2722, at_version=27'111, trim_to=0'0, roll_forward_to=27'107))", 2020-06-12T02:37:37.214 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:36.658174+0000", 2020-06-12T02:37:37.215 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.36828523699999999, 2020-06-12T02:37:37.215 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0056268059999999998, 2020-06-12T02:37:37.215 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:37.215 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:37.215 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:37.215 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.216 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:37.216 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.658174+0000", 2020-06-12T02:37:37.216 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:37.216 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.216 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.217 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:37.217 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.658174+0000", 2020-06-12T02:37:37.217 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8320999999999996e-05 2020-06-12T02:37:37.217 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.217 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.218 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:37.218 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.658273+0000", 2020-06-12T02:37:37.218 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.073e-05 2020-06-12T02:37:37.218 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.218 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.219 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:37.219 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.658293+0000", 2020-06-12T02:37:37.219 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6487e-05 2020-06-12T02:37:37.219 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.219 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.219 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:37.220 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.658310+0000", 2020-06-12T02:37:37.220 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8839999999999997e-06 2020-06-12T02:37:37.220 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.220 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.220 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:37.221 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.658317+0000", 2020-06-12T02:37:37.221 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000221301 2020-06-12T02:37:37.221 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.221 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.221 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.221 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.658538+0000", 2020-06-12T02:37:37.222 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0936e-05 2020-06-12T02:37:37.222 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.222 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.222 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:37.222 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.658549+0000", 2020-06-12T02:37:37.223 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.005217419 2020-06-12T02:37:37.223 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.223 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.223 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:37.223 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.663766+0000", 2020-06-12T02:37:37.224 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4727999999999997e-05 2020-06-12T02:37:37.224 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.224 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.224 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:37.224 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.663801+0000", 2020-06-12T02:37:37.224 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0056268059999999998 2020-06-12T02:37:37.225 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.225 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:37.225 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.225 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.225 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.226 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s2 27/21 ECSubWrite(tid=740, reqid=client.4512.0:2730, at_version=27'220, trim_to=0'0, roll_forward_to=27'217))", 2020-06-12T02:37:37.226 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:36.676961+0000", 2020-06-12T02:37:37.226 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.349498949, 2020-06-12T02:37:37.226 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0088096930000000004, 2020-06-12T02:37:37.226 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:37.227 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:37.227 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:37.227 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.227 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:37.227 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.676961+0000", 2020-06-12T02:37:37.227 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:37.228 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.228 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.228 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:37.228 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.676961+0000", 2020-06-12T02:37:37.228 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00068605000000000001 2020-06-12T02:37:37.229 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.229 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.229 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:37.229 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.677647+0000", 2020-06-12T02:37:37.229 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0792e-05 2020-06-12T02:37:37.229 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.230 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.230 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:37.230 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.677668+0000", 2020-06-12T02:37:37.230 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0032999999999999e-05 2020-06-12T02:37:37.230 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.231 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.231 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:37.231 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.677688+0000", 2020-06-12T02:37:37.231 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8369999999999999e-06 2020-06-12T02:37:37.231 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.231 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.232 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:37.232 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.677694+0000", 2020-06-12T02:37:37.232 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001677563 2020-06-12T02:37:37.232 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.232 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.233 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.233 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.679372+0000", 2020-06-12T02:37:37.233 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0089999999999996e-06 2020-06-12T02:37:37.233 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.233 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.233 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:37.234 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.679381+0000", 2020-06-12T02:37:37.234 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0063373370000000002 2020-06-12T02:37:37.234 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.234 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.234 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:37.235 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.685718+0000", 2020-06-12T02:37:37.235 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2071999999999999e-05 2020-06-12T02:37:37.235 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.235 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.235 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:37.235 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.685770+0000", 2020-06-12T02:37:37.236 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0088096930000000004 2020-06-12T02:37:37.236 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.236 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:37.236 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.236 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.236 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.237 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s2 27/21 ECSubWrite(tid=741, reqid=client.4512.0:2731, at_version=27'221, trim_to=0'0, roll_forward_to=27'217))", 2020-06-12T02:37:37.237 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:36.680670+0000", 2020-06-12T02:37:37.237 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.34578937399999998, 2020-06-12T02:37:37.237 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0060158629999999998, 2020-06-12T02:37:37.237 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:37.238 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:37.238 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:37.238 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.238 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:37.238 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.680670+0000", 2020-06-12T02:37:37.238 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:37.239 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.239 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.239 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:37.239 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.680670+0000", 2020-06-12T02:37:37.239 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00062323599999999999 2020-06-12T02:37:37.240 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.240 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.240 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:37.240 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.681293+0000", 2020-06-12T02:37:37.240 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8008000000000001e-05 2020-06-12T02:37:37.240 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.241 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.241 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:37.241 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.681311+0000", 2020-06-12T02:37:37.241 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8484000000000001e-05 2020-06-12T02:37:37.241 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.242 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.242 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:37.242 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.681330+0000", 2020-06-12T02:37:37.242 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1670000000000004e-06 2020-06-12T02:37:37.242 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.243 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.243 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:37.243 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.681336+0000", 2020-06-12T02:37:37.243 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00078741800000000002 2020-06-12T02:37:37.243 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.243 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.244 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.244 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.682124+0000", 2020-06-12T02:37:37.244 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1929999999999999e-05 2020-06-12T02:37:37.244 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.244 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.244 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:37.245 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.682135+0000", 2020-06-12T02:37:37.245 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045003320000000001 2020-06-12T02:37:37.245 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.245 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.246 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:37.246 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.686636+0000", 2020-06-12T02:37:37.246 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0287999999999997e-05 2020-06-12T02:37:37.246 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.246 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.246 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:37.247 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.686686+0000", 2020-06-12T02:37:37.247 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0060158629999999998 2020-06-12T02:37:37.247 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.247 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:37.247 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.248 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.248 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.248 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 27/21 ECSubWrite(tid=179, reqid=client.4512.0:2781, at_version=0'0, trim_to=0'0, roll_forward_to=27'115))", 2020-06-12T02:37:37.248 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:36.810022+0000", 2020-06-12T02:37:37.248 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.21643714, 2020-06-12T02:37:37.248 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093088630000000006, 2020-06-12T02:37:37.249 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:37.249 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:37.249 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:37.249 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.249 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:37.250 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.810022+0000", 2020-06-12T02:37:37.250 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:37.250 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.250 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.250 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:37.251 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.810022+0000", 2020-06-12T02:37:37.251 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0468000000000004e-05 2020-06-12T02:37:37.251 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.251 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.251 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:37.251 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.810093+0000", 2020-06-12T02:37:37.252 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6742999999999999e-05 2020-06-12T02:37:37.252 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.252 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.252 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:37.252 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.810110+0000", 2020-06-12T02:37:37.252 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8359000000000002e-05 2020-06-12T02:37:37.253 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.253 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.253 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:37.253 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.810128+0000", 2020-06-12T02:37:37.253 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8390000000000007e-06 2020-06-12T02:37:37.254 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.254 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.254 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:37.254 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.810136+0000", 2020-06-12T02:37:37.254 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004899558 2020-06-12T02:37:37.255 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.255 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.255 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.255 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.815035+0000", 2020-06-12T02:37:37.255 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7008e-05 2020-06-12T02:37:37.255 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.256 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.256 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:37.256 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.815052+0000", 2020-06-12T02:37:37.256 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0042246180000000003 2020-06-12T02:37:37.256 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.257 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.257 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:37.257 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.819277+0000", 2020-06-12T02:37:37.257 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.427e-05 2020-06-12T02:37:37.257 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.257 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.258 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:37.258 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.819331+0000", 2020-06-12T02:37:37.258 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093088630000000006 2020-06-12T02:37:37.258 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.258 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:37.259 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.259 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.259 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.259 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 27/21 ECSubWrite(tid=180, reqid=client.4512.0:2782, at_version=27'116, trim_to=0'0, roll_forward_to=27'115))", 2020-06-12T02:37:37.259 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:36.811166+0000", 2020-06-12T02:37:37.259 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.215293175, 2020-06-12T02:37:37.260 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013435703, 2020-06-12T02:37:37.260 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:37.260 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:37.260 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:37.260 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.261 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:37.261 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.811166+0000", 2020-06-12T02:37:37.261 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:37.261 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.261 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.261 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:37.262 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.811166+0000", 2020-06-12T02:37:37.262 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015779300000000001 2020-06-12T02:37:37.262 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.262 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.262 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:37.263 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.811324+0000", 2020-06-12T02:37:37.263 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8149000000000001e-05 2020-06-12T02:37:37.263 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.263 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.263 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:37.263 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.811352+0000", 2020-06-12T02:37:37.264 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6880000000000001e-05 2020-06-12T02:37:37.264 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.264 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.264 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:37.264 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.811369+0000", 2020-06-12T02:37:37.265 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1099999999999999e-06 2020-06-12T02:37:37.265 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.265 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.265 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:37.265 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.811375+0000", 2020-06-12T02:37:37.265 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0038906349999999999 2020-06-12T02:37:37.266 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.266 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.266 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.266 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.815266+0000", 2020-06-12T02:37:37.266 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0536e-05 2020-06-12T02:37:37.267 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.267 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.267 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:37.267 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.815277+0000", 2020-06-12T02:37:37.267 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0092955610000000008 2020-06-12T02:37:37.268 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.268 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.268 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:37.268 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.824572+0000", 2020-06-12T02:37:37.268 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0039000000000002e-05 2020-06-12T02:37:37.268 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.269 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.269 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:37.269 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.824602+0000", 2020-06-12T02:37:37.269 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013435703 2020-06-12T02:37:37.269 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.270 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:37.270 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.270 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.270 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.270 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 27/21 ECSubWrite(tid=181, reqid=client.4512.0:2783, at_version=27'117, trim_to=0'0, roll_forward_to=27'115))", 2020-06-12T02:37:37.270 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:36.816115+0000", 2020-06-12T02:37:37.271 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.210344844, 2020-06-12T02:37:37.271 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085240260000000005, 2020-06-12T02:37:37.271 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:37.271 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:37.271 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:37.272 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.272 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:37.272 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.816115+0000", 2020-06-12T02:37:37.272 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:37.272 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.272 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.273 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:37.273 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.816115+0000", 2020-06-12T02:37:37.273 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011663900000000001 2020-06-12T02:37:37.273 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.273 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.274 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:37.274 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.816231+0000", 2020-06-12T02:37:37.274 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2373000000000001e-05 2020-06-12T02:37:37.274 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.274 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.274 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:37.275 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.816254+0000", 2020-06-12T02:37:37.275 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0010999999999999e-05 2020-06-12T02:37:37.275 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.275 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.275 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:37.276 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.816274+0000", 2020-06-12T02:37:37.276 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6769999999999998e-06 2020-06-12T02:37:37.276 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.276 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.276 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:37.276 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.816280+0000", 2020-06-12T02:37:37.277 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0031573920000000002 2020-06-12T02:37:37.277 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.277 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.277 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.277 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.819438+0000", 2020-06-12T02:37:37.278 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0868e-05 2020-06-12T02:37:37.278 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.278 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.278 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:37.278 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.819449+0000", 2020-06-12T02:37:37.278 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0051582829999999996 2020-06-12T02:37:37.279 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.279 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.279 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:37.279 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.824607+0000", 2020-06-12T02:37:37.279 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.1782999999999997e-05 2020-06-12T02:37:37.280 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.280 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.280 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:37.280 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.824639+0000", 2020-06-12T02:37:37.280 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085240260000000005 2020-06-12T02:37:37.280 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.281 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:37.281 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.281 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.281 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.281 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 27/21 ECSubWrite(tid=182, reqid=client.4512.0:2784, at_version=27'118, trim_to=0'0, roll_forward_to=27'115))", 2020-06-12T02:37:37.282 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:36.819948+0000", 2020-06-12T02:37:37.282 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.20651123199999999, 2020-06-12T02:37:37.282 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006449125, 2020-06-12T02:37:37.282 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:37.282 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:37.282 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:37.283 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.283 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:37.283 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.819948+0000", 2020-06-12T02:37:37.283 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:37.283 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.284 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.284 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:37.284 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.819948+0000", 2020-06-12T02:37:37.284 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9300000000000004e-05 2020-06-12T02:37:37.284 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.284 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.285 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:37.285 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.820018+0000", 2020-06-12T02:37:37.285 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2920000000000001e-05 2020-06-12T02:37:37.285 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.285 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.286 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:37.286 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.820041+0000", 2020-06-12T02:37:37.286 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0113e-05 2020-06-12T02:37:37.286 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.286 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.287 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:37.287 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.820051+0000", 2020-06-12T02:37:37.287 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0689999999999998e-06 2020-06-12T02:37:37.287 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.287 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.287 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:37.288 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.820057+0000", 2020-06-12T02:37:37.288 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0024862009999999999 2020-06-12T02:37:37.288 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.288 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.288 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.288 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.822543+0000", 2020-06-12T02:37:37.289 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2531000000000001e-05 2020-06-12T02:37:37.289 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.289 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.289 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:37.289 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.822556+0000", 2020-06-12T02:37:37.290 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0037946019999999998 2020-06-12T02:37:37.290 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.290 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.290 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:37.290 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.826350+0000", 2020-06-12T02:37:37.290 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7389000000000003e-05 2020-06-12T02:37:37.291 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.291 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.291 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:37.291 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.826398+0000", 2020-06-12T02:37:37.291 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006449125 2020-06-12T02:37:37.292 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.292 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:37.292 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.292 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.292 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:37.292 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:37:37.302 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-06-12T02:37:37.303 INFO:tasks.rados.rados.0.smithi107.stdout:1355: writing smithi10729407-1 from 696320 to 1376256 tid 1 2020-06-12T02:37:37.303 INFO:tasks.rados.rados.0.smithi107.stdout:1355: writing smithi10729407-1 from 1376256 to 1777664 tid 2 2020-06-12T02:37:37.303 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:37:37.304 INFO:tasks.rados.rados.0.smithi107.stdout:1356: seq_num 1355 ranges {0=696320,696320=401408} 2020-06-12T02:37:37.304 INFO:tasks.rados.rados.0.smithi107.stdout:1356: writing smithi10729407-3 from 0 to 696320 tid 1 2020-06-12T02:37:37.304 INFO:tasks.rados.rados.0.smithi107.stdout:1356: writing smithi10729407-3 from 696320 to 1097728 tid 2 2020-06-12T02:37:37.304 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:37:37.304 INFO:tasks.rados.rados.0.smithi107.stdout:1357: seq_num 1356 ranges {0=720896,720896=401408} 2020-06-12T02:37:37.305 INFO:tasks.rados.rados.0.smithi107.stdout:1357: writing smithi10729407-11 from 0 to 720896 tid 1 2020-06-12T02:37:37.305 INFO:tasks.rados.rados.0.smithi107.stdout:1357: writing smithi10729407-11 from 720896 to 1122304 tid 2 2020-06-12T02:37:37.305 INFO:tasks.rados.rados.0.smithi107.stdout:1358: delete oid 15 current snap is 0 2020-06-12T02:37:37.305 INFO:tasks.rados.rados.0.smithi107.stdout:1354: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:37.305 INFO:tasks.rados.rados.0.smithi107.stdout:1354: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:37.306 INFO:tasks.rados.rados.0.smithi107.stdout:1354: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:37.306 INFO:tasks.rados.rados.0.smithi107.stdout:1354: finishing write tid 5 to smithi10729407-16 2020-06-12T02:37:37.306 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 237 (ObjNum 1353 snap 0 seq_num 1353) dirty exists 2020-06-12T02:37:37.306 INFO:tasks.rados.rados.0.smithi107.stdout:1354: left oid 16 (ObjNum 1353 snap 0 seq_num 1353) 2020-06-12T02:37:37.306 INFO:tasks.rados.rados.0.smithi107.stdout:1355: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:37.306 INFO:tasks.rados.rados.0.smithi107.stdout:1355: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:37.307 INFO:tasks.rados.rados.0.smithi107.stdout:1355: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:37.307 INFO:tasks.rados.rados.0.smithi107.stdout:1355: finishing write tid 5 to smithi10729407-1 2020-06-12T02:37:37.307 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 264 (ObjNum 1354 snap 0 seq_num 1354) dirty exists 2020-06-12T02:37:37.307 INFO:tasks.rados.rados.0.smithi107.stdout:1355: left oid 1 (ObjNum 1354 snap 0 seq_num 1354) 2020-06-12T02:37:37.307 INFO:tasks.rados.rados.0.smithi107.stdout:1356: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:37.307 INFO:tasks.rados.rados.0.smithi107.stdout:1356: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:37.308 INFO:tasks.rados.rados.0.smithi107.stdout:1356: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:37.308 INFO:tasks.rados.rados.0.smithi107.stdout:1356: finishing write tid 5 to smithi10729407-3 2020-06-12T02:37:37.308 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 240 (ObjNum 1355 snap 0 seq_num 1355) dirty exists 2020-06-12T02:37:37.308 INFO:tasks.rados.rados.0.smithi107.stdout:1356: left oid 3 (ObjNum 1355 snap 0 seq_num 1355) 2020-06-12T02:37:37.308 INFO:tasks.rados.rados.0.smithi107.stdout:1354: done (4 left) 2020-06-12T02:37:37.309 INFO:tasks.rados.rados.0.smithi107.stdout:1355: done (3 left) 2020-06-12T02:37:37.309 INFO:tasks.rados.rados.0.smithi107.stdout:1356: done (2 left) 2020-06-12T02:37:37.309 INFO:tasks.rados.rados.0.smithi107.stdout:1358: done (1 left) 2020-06-12T02:37:37.309 INFO:tasks.rados.rados.0.smithi107.stdout:1359: delete oid 2 current snap is 0 2020-06-12T02:37:37.309 INFO:tasks.rados.rados.0.smithi107.stdout:1357: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:37.309 INFO:tasks.rados.rados.0.smithi107.stdout:1359: done (1 left) 2020-06-12T02:37:37.310 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:37:37.310 INFO:tasks.rados.rados.0.smithi107.stdout:1360: seq_num 1359 ranges {0=770048,770048=401408} 2020-06-12T02:37:37.310 INFO:tasks.rados.rados.0.smithi107.stdout:1360: writing smithi10729407-14 from 0 to 770048 tid 1 2020-06-12T02:37:37.310 INFO:tasks.rados.rados.0.smithi107.stdout:1360: writing smithi10729407-14 from 770048 to 1171456 tid 2 2020-06-12T02:37:37.310 INFO:tasks.rados.rados.0.smithi107.stdout:1361: delete oid 15 current snap is 0 2020-06-12T02:37:37.311 INFO:tasks.rados.rados.0.smithi107.stdout:1357: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:37.311 INFO:tasks.rados.rados.0.smithi107.stdout:1357: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:37.311 INFO:tasks.rados.rados.0.smithi107.stdout:1357: finishing write tid 5 to smithi10729407-11 2020-06-12T02:37:37.311 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 233 (ObjNum 1356 snap 0 seq_num 1356) dirty exists 2020-06-12T02:37:37.311 INFO:tasks.rados.rados.0.smithi107.stdout:1357: left oid 11 (ObjNum 1356 snap 0 seq_num 1356) 2020-06-12T02:37:37.311 INFO:tasks.rados.rados.0.smithi107.stdout:1357: done (2 left) 2020-06-12T02:37:37.312 INFO:tasks.rados.rados.0.smithi107.stdout:1361: done (1 left) 2020-06-12T02:37:37.312 INFO:tasks.rados.rados.0.smithi107.stdout:1362: delete oid 10 current snap is 0 2020-06-12T02:37:37.312 INFO:tasks.rados.rados.0.smithi107.stdout:1362: done (1 left) 2020-06-12T02:37:37.312 INFO:tasks.rados.rados.0.smithi107.stdout:1363: delete oid 2 current snap is 0 2020-06-12T02:37:37.312 INFO:tasks.rados.rados.0.smithi107.stdout:1363: done (1 left) 2020-06-12T02:37:37.313 INFO:tasks.rados.rados.0.smithi107.stdout:1364: delete oid 5 current snap is 0 2020-06-12T02:37:37.313 INFO:tasks.rados.rados.0.smithi107.stdout:1364: done (1 left) 2020-06-12T02:37:37.313 INFO:tasks.rados.rados.0.smithi107.stdout:1365: delete oid 19 current snap is 0 2020-06-12T02:37:37.313 INFO:tasks.rados.rados.0.smithi107.stdout:1365: done (1 left) 2020-06-12T02:37:37.313 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:37:37.313 INFO:tasks.rados.rados.0.smithi107.stdout:1366: seq_num 1365 ranges {1777664=466944,2244608=614400,2859008=188416} 2020-06-12T02:37:37.314 INFO:tasks.rados.rados.0.smithi107.stdout:1366: writing smithi10729407-1 from 1777664 to 2244608 tid 1 2020-06-12T02:37:37.314 INFO:tasks.rados.rados.0.smithi107.stdout:1366: writing smithi10729407-1 from 2244608 to 2859008 tid 2 2020-06-12T02:37:37.314 INFO:tasks.rados.rados.0.smithi107.stdout:1366: writing smithi10729407-1 from 2859008 to 3047424 tid 3 2020-06-12T02:37:37.314 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:37:37.314 INFO:tasks.rados.rados.0.smithi107.stdout:1367: seq_num 1366 ranges {0=483328,483328=499712,983040=303104} 2020-06-12T02:37:37.315 INFO:tasks.rados.rados.0.smithi107.stdout:1367: writing smithi10729407-5 from 0 to 483328 tid 1 2020-06-12T02:37:37.315 INFO:tasks.rados.rados.0.smithi107.stdout:1367: writing smithi10729407-5 from 483328 to 983040 tid 2 2020-06-12T02:37:37.315 INFO:tasks.rados.rados.0.smithi107.stdout:1367: writing smithi10729407-5 from 983040 to 1286144 tid 3 2020-06-12T02:37:37.315 INFO:tasks.rados.rados.0.smithi107.stdout:1368: delete oid 13 current snap is 0 2020-06-12T02:37:37.315 INFO:tasks.rados.rados.0.smithi107.stdout:1367: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:37.315 INFO:tasks.rados.rados.0.smithi107.stdout:1368: done (3 left) 2020-06-12T02:37:37.316 INFO:tasks.rados.rados.0.smithi107.stdout:1369: delete oid 17 current snap is 0 2020-06-12T02:37:37.316 INFO:tasks.rados.rados.0.smithi107.stdout:1367: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:37.316 INFO:tasks.rados.rados.0.smithi107.stdout:1369: done (3 left) 2020-06-12T02:37:37.316 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:37:37.316 INFO:tasks.rados.rados.0.smithi107.stdout:1370: seq_num 1369 ranges {0=532480,532480=598016,1130496=204800} 2020-06-12T02:37:37.316 INFO:tasks.rados.rados.0.smithi107.stdout:1370: writing smithi10729407-17 from 0 to 532480 tid 1 2020-06-12T02:37:37.317 INFO:tasks.rados.rados.0.smithi107.stdout:1370: writing smithi10729407-17 from 532480 to 1130496 tid 2 2020-06-12T02:37:37.317 INFO:tasks.rados.rados.0.smithi107.stdout:1370: writing smithi10729407-17 from 1130496 to 1335296 tid 3 2020-06-12T02:37:37.317 INFO:tasks.rados.rados.0.smithi107.stdout:1371: delete oid 4 current snap is 0 2020-06-12T02:37:37.317 INFO:tasks.rados.rados.0.smithi107.stdout:1367: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:37.317 INFO:tasks.rados.rados.0.smithi107.stdout:1367: finishing write tid 5 to smithi10729407-5 2020-06-12T02:37:37.318 INFO:tasks.rados.rados.0.smithi107.stdout:1367: finishing write tid 6 to smithi10729407-5 2020-06-12T02:37:37.318 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 120 (ObjNum 1366 snap 0 seq_num 1366) dirty exists 2020-06-12T02:37:37.318 INFO:tasks.rados.rados.0.smithi107.stdout:1367: left oid 5 (ObjNum 1366 snap 0 seq_num 1366) 2020-06-12T02:37:37.318 INFO:tasks.rados.rados.0.smithi107.stdout:1370: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:37.318 INFO:tasks.rados.rados.0.smithi107.stdout:1370: finishing write tid 2 to smithi10729407-17 2020-06-12T02:37:37.318 INFO:tasks.rados.rados.0.smithi107.stdout:1370: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:37.319 INFO:tasks.rados.rados.0.smithi107.stdout:1370: finishing write tid 5 to smithi10729407-17 2020-06-12T02:37:37.319 INFO:tasks.rados.rados.0.smithi107.stdout:1367: done (4 left) 2020-06-12T02:37:37.319 INFO:tasks.rados.rados.0.smithi107.stdout:1371: done (3 left) 2020-06-12T02:37:37.319 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:37:37.319 INFO:tasks.rados.rados.0.smithi107.stdout:1372: seq_num 1371 ranges {0=565248,565248=770048,1335296=32768} 2020-06-12T02:37:37.320 INFO:tasks.rados.rados.0.smithi107.stdout:1372: writing smithi10729407-15 from 0 to 565248 tid 1 2020-06-12T02:37:37.320 INFO:tasks.rados.rados.0.smithi107.stdout:1372: writing smithi10729407-15 from 565248 to 1335296 tid 2 2020-06-12T02:37:37.320 INFO:tasks.rados.rados.0.smithi107.stdout:1372: writing smithi10729407-15 from 1335296 to 1368064 tid 3 2020-06-12T02:37:37.320 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:37:37.320 INFO:tasks.rados.rados.0.smithi107.stdout:1373: seq_num 1372 ranges {1097728=581632,1679360=655360,2334720=147456} 2020-06-12T02:37:37.320 INFO:tasks.rados.rados.0.smithi107.stdout:1373: writing smithi10729407-3 from 1097728 to 1679360 tid 1 2020-06-12T02:37:37.321 INFO:tasks.rados.rados.0.smithi107.stdout:1373: writing smithi10729407-3 from 1679360 to 2334720 tid 2 2020-06-12T02:37:37.321 INFO:tasks.rados.rados.0.smithi107.stdout:1373: writing smithi10729407-3 from 2334720 to 2482176 tid 3 2020-06-12T02:37:37.321 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:37:37.321 INFO:tasks.rados.rados.0.smithi107.stdout:1374: seq_num 1373 ranges {1122304=598016,1720320=532480,2252800=270336} 2020-06-12T02:37:37.321 INFO:tasks.rados.rados.0.smithi107.stdout:1374: writing smithi10729407-11 from 1122304 to 1720320 tid 1 2020-06-12T02:37:37.322 INFO:tasks.rados.rados.0.smithi107.stdout:1374: writing smithi10729407-11 from 1720320 to 2252800 tid 2 2020-06-12T02:37:37.322 INFO:tasks.rados.rados.0.smithi107.stdout:1374: writing smithi10729407-11 from 2252800 to 2523136 tid 3 2020-06-12T02:37:37.322 INFO:tasks.rados.rados.0.smithi107.stdout:1375: delete oid 16 current snap is 0 2020-06-12T02:37:37.322 INFO:tasks.rados.rados.0.smithi107.stdout:1370: finishing write tid 6 to smithi10729407-17 2020-06-12T02:37:37.322 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 263 (ObjNum 1369 snap 0 seq_num 1369) dirty exists 2020-06-12T02:37:37.322 INFO:tasks.rados.rados.0.smithi107.stdout:1370: left oid 17 (ObjNum 1369 snap 0 seq_num 1369) 2020-06-12T02:37:37.323 INFO:tasks.rados.rados.0.smithi107.stdout:1372: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:37.323 INFO:tasks.rados.rados.0.smithi107.stdout:1372: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:37.323 INFO:tasks.rados.rados.0.smithi107.stdout:1372: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:37.323 INFO:tasks.rados.rados.0.smithi107.stdout:1372: finishing write tid 5 to smithi10729407-15 2020-06-12T02:37:37.323 INFO:tasks.rados.rados.0.smithi107.stdout:1372: finishing write tid 6 to smithi10729407-15 2020-06-12T02:37:37.323 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 231 (ObjNum 1371 snap 0 seq_num 1371) dirty exists 2020-06-12T02:37:37.324 INFO:tasks.rados.rados.0.smithi107.stdout:1372: left oid 15 (ObjNum 1371 snap 0 seq_num 1371) 2020-06-12T02:37:37.324 INFO:tasks.rados.rados.0.smithi107.stdout:1373: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:37.324 INFO:tasks.rados.rados.0.smithi107.stdout:1373: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:37.324 INFO:tasks.rados.rados.0.smithi107.stdout:1373: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:37.324 INFO:tasks.rados.rados.0.smithi107.stdout:1373: finishing write tid 5 to smithi10729407-3 2020-06-12T02:37:37.325 INFO:tasks.rados.rados.0.smithi107.stdout:1373: finishing write tid 6 to smithi10729407-3 2020-06-12T02:37:37.325 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 244 (ObjNum 1372 snap 0 seq_num 1372) dirty exists 2020-06-12T02:37:37.325 INFO:tasks.rados.rados.0.smithi107.stdout:1373: left oid 3 (ObjNum 1372 snap 0 seq_num 1372) 2020-06-12T02:37:37.325 INFO:tasks.rados.rados.0.smithi107.stdout:1370: done (6 left) 2020-06-12T02:37:37.325 INFO:tasks.rados.rados.0.smithi107.stdout:1372: done (5 left) 2020-06-12T02:37:37.325 INFO:tasks.rados.rados.0.smithi107.stdout:1373: done (4 left) 2020-06-12T02:37:37.326 INFO:tasks.rados.rados.0.smithi107.stdout:1375: done (3 left) 2020-06-12T02:37:37.326 INFO:tasks.rados.rados.0.smithi107.stdout:1376: delete oid 15 current snap is 0 2020-06-12T02:37:37.326 INFO:tasks.rados.rados.0.smithi107.stdout:1376: done (3 left) 2020-06-12T02:37:37.326 INFO:tasks.rados.rados.0.smithi107.stdout:1377: delete oid 2 current snap is 0 2020-06-12T02:37:37.326 INFO:tasks.rados.rados.0.smithi107.stdout:1377: done (3 left) 2020-06-12T02:37:37.326 INFO:tasks.rados.rados.0.smithi107.stdout:1378: delete oid 19 current snap is 0 2020-06-12T02:37:37.327 INFO:tasks.rados.rados.0.smithi107.stdout:1378: done (3 left) 2020-06-12T02:37:37.327 INFO:tasks.rados.rados.0.smithi107.stdout:1379: delete oid 6 current snap is 0 2020-06-12T02:37:37.327 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:37:37.291+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:37:37.394 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:37:37.392+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:37:37.453 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:37:37.453 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:37:37.453 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2020-06-12T02:37:37.453 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:37:37.461 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-06-12T02:37:37.505 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:37:37.502+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:37:37.597 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:37.594+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:37:37.619 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:37:37.620 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:37:37.620 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2020-06-12T02:37:37.620 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2020-06-12T02:37:37.620 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:37:37.627 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-06-12T02:37:37.679 INFO:tasks.rados.rados.0.smithi107.stdout:1360: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:37.685 INFO:tasks.rados.rados.0.smithi107.stdout:1366: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:37.690 INFO:tasks.rados.rados.0.smithi107.stdout:1366: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:37.691 INFO:tasks.rados.rados.0.smithi107.stdout:1366: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:37.691 INFO:tasks.rados.rados.0.smithi107.stdout:1366: finishing write tid 5 to smithi10729407-1 2020-06-12T02:37:37.692 INFO:tasks.rados.rados.0.smithi107.stdout:1374: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:37.692 INFO:tasks.rados.rados.0.smithi107.stdout:1374: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:37.692 INFO:tasks.rados.rados.0.smithi107.stdout:1366: finishing write tid 6 to smithi10729407-1 2020-06-12T02:37:37.693 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 268 (ObjNum 1365 snap 0 seq_num 1365) dirty exists 2020-06-12T02:37:37.693 INFO:tasks.rados.rados.0.smithi107.stdout:1366: left oid 1 (ObjNum 1365 snap 0 seq_num 1365) 2020-06-12T02:37:37.693 INFO:tasks.rados.rados.0.smithi107.stdout:1374: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:37.694 INFO:tasks.rados.rados.0.smithi107.stdout:1374: finishing write tid 5 to smithi10729407-11 2020-06-12T02:37:37.694 INFO:tasks.rados.rados.0.smithi107.stdout:1366: done (3 left) 2020-06-12T02:37:37.694 INFO:tasks.rados.rados.0.smithi107.stdout:1379: done (2 left) 2020-06-12T02:37:37.694 INFO:tasks.rados.rados.0.smithi107.stdout:1380: delete oid 17 current snap is 0 2020-06-12T02:37:37.694 INFO:tasks.rados.rados.0.smithi107.stdout:1360: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:37.694 INFO:tasks.rados.rados.0.smithi107.stdout:1360: finishing write tid 4 to smithi10729407-14 2020-06-12T02:37:37.695 INFO:tasks.rados.rados.0.smithi107.stdout:1374: finishing write tid 6 to smithi10729407-11 2020-06-12T02:37:37.695 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 238 (ObjNum 1373 snap 0 seq_num 1373) dirty exists 2020-06-12T02:37:37.695 INFO:tasks.rados.rados.0.smithi107.stdout:1374: left oid 11 (ObjNum 1373 snap 0 seq_num 1373) 2020-06-12T02:37:37.696 INFO:tasks.rados.rados.0.smithi107.stdout:1360: finishing write tid 5 to smithi10729407-14 2020-06-12T02:37:37.696 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 240 (ObjNum 1359 snap 0 seq_num 1359) dirty exists 2020-06-12T02:37:37.696 INFO:tasks.rados.rados.0.smithi107.stdout:1360: left oid 14 (ObjNum 1359 snap 0 seq_num 1359) 2020-06-12T02:37:37.698 INFO:tasks.rados.rados.0.smithi107.stdout:1360: done (2 left) 2020-06-12T02:37:37.698 INFO:tasks.rados.rados.0.smithi107.stdout:1374: done (1 left) 2020-06-12T02:37:37.698 INFO:tasks.rados.rados.0.smithi107.stdout:1380: done (0 left) 2020-06-12T02:37:37.699 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:37:37.699 INFO:tasks.rados.rados.0.smithi107.stdout:1381: seq_num 1380 ranges {0=720896,720896=573440,1294336=229376} 2020-06-12T02:37:37.699 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:37.695+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:37.701 INFO:tasks.rados.rados.0.smithi107.stdout:1381: writing smithi10729407-10 from 0 to 720896 tid 1 2020-06-12T02:37:37.709 INFO:tasks.rados.rados.0.smithi107.stdout:1381: writing smithi10729407-10 from 720896 to 1294336 tid 2 2020-06-12T02:37:37.709 INFO:tasks.rados.rados.0.smithi107.stdout:1381: writing smithi10729407-10 from 1294336 to 1523712 tid 3 2020-06-12T02:37:37.710 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:37:37.710 INFO:tasks.rados.rados.0.smithi107.stdout:1382: seq_num 1381 ranges {0=737280,737280=458752,1196032=344064} 2020-06-12T02:37:37.712 INFO:tasks.rados.rados.0.smithi107.stdout:1382: writing smithi10729407-19 from 0 to 737280 tid 1 2020-06-12T02:37:37.715 INFO:tasks.rados.rados.0.smithi107.stdout:1382: writing smithi10729407-19 from 737280 to 1196032 tid 2 2020-06-12T02:37:37.717 INFO:tasks.rados.rados.0.smithi107.stdout:1382: writing smithi10729407-19 from 1196032 to 1540096 tid 3 2020-06-12T02:37:37.718 INFO:tasks.rados.rados.0.smithi107.stdout:1383: delete oid 8 current snap is 0 2020-06-12T02:37:37.718 INFO:tasks.rados.rados.0.smithi107.stdout:1381: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:37.718 INFO:tasks.rados.rados.0.smithi107.stdout:1381: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:37.718 INFO:tasks.rados.rados.0.smithi107.stdout:1381: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:37.719 INFO:tasks.rados.rados.0.smithi107.stdout:1381: finishing write tid 5 to smithi10729407-10 2020-06-12T02:37:37.721 INFO:tasks.rados.rados.0.smithi107.stdout:1381: finishing write tid 6 to smithi10729407-10 2020-06-12T02:37:37.724 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 132 (ObjNum 1380 snap 0 seq_num 1380) dirty exists 2020-06-12T02:37:37.724 INFO:tasks.rados.rados.0.smithi107.stdout:1381: left oid 10 (ObjNum 1380 snap 0 seq_num 1380) 2020-06-12T02:37:37.724 INFO:tasks.rados.rados.0.smithi107.stdout:1381: done (2 left) 2020-06-12T02:37:37.724 INFO:tasks.rados.rados.0.smithi107.stdout:1383: done (1 left) 2020-06-12T02:37:37.724 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:37:37.725 INFO:tasks.rados.rados.0.smithi107.stdout:1384: seq_num 1383 ranges {0=770048,770048=630784,1400832=172032} 2020-06-12T02:37:37.725 INFO:tasks.rados.rados.0.smithi107.stdout:1384: writing smithi10729407-8 from 0 to 770048 tid 1 2020-06-12T02:37:37.729 INFO:tasks.rados.rados.0.smithi107.stdout:1384: writing smithi10729407-8 from 770048 to 1400832 tid 2 2020-06-12T02:37:37.730 INFO:tasks.rados.rados.0.smithi107.stdout:1384: writing smithi10729407-8 from 1400832 to 1572864 tid 3 2020-06-12T02:37:37.730 INFO:tasks.rados.rados.0.smithi107.stdout:1385: delete oid 9 current snap is 0 2020-06-12T02:37:37.730 INFO:tasks.rados.rados.0.smithi107.stdout:1382: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:37.731 INFO:tasks.rados.rados.0.smithi107.stdout:1382: finishing write tid 2 to smithi10729407-19 2020-06-12T02:37:37.731 INFO:tasks.rados.rados.0.smithi107.stdout:1382: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:37.731 INFO:tasks.rados.rados.0.smithi107.stdout:1382: finishing write tid 5 to smithi10729407-19 2020-06-12T02:37:37.731 INFO:tasks.rados.rados.0.smithi107.stdout:1382: finishing write tid 6 to smithi10729407-19 2020-06-12T02:37:37.731 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 223 (ObjNum 1381 snap 0 seq_num 1381) dirty exists 2020-06-12T02:37:37.731 INFO:tasks.rados.rados.0.smithi107.stdout:1382: left oid 19 (ObjNum 1381 snap 0 seq_num 1381) 2020-06-12T02:37:37.732 INFO:tasks.rados.rados.0.smithi107.stdout:1382: done (2 left) 2020-06-12T02:37:37.732 INFO:tasks.rados.rados.0.smithi107.stdout:1385: done (1 left) 2020-06-12T02:37:37.732 INFO:tasks.rados.rados.0.smithi107.stdout:1386: delete oid 11 current snap is 0 2020-06-12T02:37:37.734 INFO:tasks.rados.rados.0.smithi107.stdout:1386: done (1 left) 2020-06-12T02:37:37.734 INFO:tasks.rados.rados.0.smithi107.stdout:1387: delete oid 6 current snap is 0 2020-06-12T02:37:37.737 INFO:tasks.rados.rados.0.smithi107.stdout:1384: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:37.737 INFO:tasks.rados.rados.0.smithi107.stdout:1384: finishing write tid 2 to smithi10729407-8 2020-06-12T02:37:37.737 INFO:tasks.rados.rados.0.smithi107.stdout:1387: done (1 left) 2020-06-12T02:37:37.741 INFO:tasks.rados.rados.0.smithi107.stdout:1388: delete oid 2 current snap is 0 2020-06-12T02:37:37.742 INFO:tasks.rados.rados.0.smithi107.stdout:1384: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:37.742 INFO:tasks.rados.rados.0.smithi107.stdout:1384: finishing write tid 5 to smithi10729407-8 2020-06-12T02:37:37.742 INFO:tasks.rados.rados.0.smithi107.stdout:1388: done (1 left) 2020-06-12T02:37:37.742 INFO:tasks.rados.rados.0.smithi107.stdout:1389: delete oid 3 current snap is 0 2020-06-12T02:37:37.743 INFO:tasks.rados.rados.0.smithi107.stdout:1384: finishing write tid 6 to smithi10729407-8 2020-06-12T02:37:37.743 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 237 (ObjNum 1383 snap 0 seq_num 1383) dirty exists 2020-06-12T02:37:37.743 INFO:tasks.rados.rados.0.smithi107.stdout:1384: left oid 8 (ObjNum 1383 snap 0 seq_num 1383) 2020-06-12T02:37:37.743 INFO:tasks.rados.rados.0.smithi107.stdout:1384: done (1 left) 2020-06-12T02:37:37.743 INFO:tasks.rados.rados.0.smithi107.stdout:1389: done (0 left) 2020-06-12T02:37:37.743 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:37:37.744 INFO:tasks.rados.rados.0.smithi107.stdout:1390: seq_num 1389 ranges {0=466944,466944=786432,1253376=417792} 2020-06-12T02:37:37.744 INFO:tasks.rados.rados.0.smithi107.stdout:1390: writing smithi10729407-20 from 0 to 466944 tid 1 2020-06-12T02:37:37.750 INFO:tasks.rados.rados.0.smithi107.stdout:1390: writing smithi10729407-20 from 466944 to 1253376 tid 2 2020-06-12T02:37:37.752 INFO:tasks.rados.rados.0.smithi107.stdout:1390: writing smithi10729407-20 from 1253376 to 1671168 tid 3 2020-06-12T02:37:37.753 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:37:37.753 INFO:tasks.rados.rados.0.smithi107.stdout:1391: seq_num 1390 ranges {0=483328,483328=671744,1155072=532480} 2020-06-12T02:37:37.755 INFO:tasks.rados.rados.0.smithi107.stdout:1391: writing smithi10729407-6 from 0 to 483328 tid 1 2020-06-12T02:37:37.759 INFO:tasks.rados.rados.0.smithi107.stdout:1391: writing smithi10729407-6 from 483328 to 1155072 tid 2 2020-06-12T02:37:37.763 INFO:tasks.rados.rados.0.smithi107.stdout:1391: writing smithi10729407-6 from 1155072 to 1687552 tid 3 2020-06-12T02:37:37.763 INFO:tasks.rados.rados.0.smithi107.stdout:1392: delete oid 4 current snap is 0 2020-06-12T02:37:37.763 INFO:tasks.rados.rados.0.smithi107.stdout:1390: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:37.763 INFO:tasks.rados.rados.0.smithi107.stdout:1390: finishing write tid 2 to smithi10729407-20 2020-06-12T02:37:37.764 INFO:tasks.rados.rados.0.smithi107.stdout:1390: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:37.764 INFO:tasks.rados.rados.0.smithi107.stdout:1390: finishing write tid 5 to smithi10729407-20 2020-06-12T02:37:37.764 INFO:tasks.rados.rados.0.smithi107.stdout:1390: finishing write tid 6 to smithi10729407-20 2020-06-12T02:37:37.764 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 106 (ObjNum 1389 snap 0 seq_num 1389) dirty exists 2020-06-12T02:37:37.764 INFO:tasks.rados.rados.0.smithi107.stdout:1390: left oid 20 (ObjNum 1389 snap 0 seq_num 1389) 2020-06-12T02:37:37.765 INFO:tasks.rados.rados.0.smithi107.stdout:1391: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:37.766 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:37:37.766 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2020-06-12T02:37:37.766 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2020-06-12T02:37:37.767 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2020-06-12T02:37:37.767 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.767 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:962 3.7s0 3:ea271007:::smithi10729407-9:head [append 0~598016] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:37.767 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:32.236108+0000", 2020-06-12T02:37:37.767 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 5.5252260939999998, 2020-06-12T02:37:37.768 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018271632999999999, 2020-06-12T02:37:37.768 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:37.768 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:37.768 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:37.768 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:37.769 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:37.769 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 962 2020-06-12T02:37:37.769 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.769 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:37.769 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.770 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:37.770 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.236108+0000", 2020-06-12T02:37:37.770 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:37.770 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.770 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.770 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:37.771 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.236108+0000", 2020-06-12T02:37:37.771 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019838700000000001 2020-06-12T02:37:37.771 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.771 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.771 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:37.772 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.236307+0000", 2020-06-12T02:37:37.772 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4619999999999996e-06 2020-06-12T02:37:37.772 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.772 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.772 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:37.773 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.236315+0000", 2020-06-12T02:37:37.773 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1665999999999999e-05 2020-06-12T02:37:37.773 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.773 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.773 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:37.774 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.236337+0000", 2020-06-12T02:37:37.774 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5630000000000003e-06 2020-06-12T02:37:37.774 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.774 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.774 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:37.775 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.236344+0000", 2020-06-12T02:37:37.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023269440000000001 2020-06-12T02:37:37.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.775 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.775 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.238671+0000", 2020-06-12T02:37:37.776 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5969000000000003e-05 2020-06-12T02:37:37.776 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.776 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.776 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:37.776 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.238757+0000", 2020-06-12T02:37:37.777 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046101010000000001 2020-06-12T02:37:37.777 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.777 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.777 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.777 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.243367+0000", 2020-06-12T02:37:37.778 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7991000000000005e-05 2020-06-12T02:37:37.778 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.778 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.778 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:37.778 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.243455+0000", 2020-06-12T02:37:37.779 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00062561500000000005 2020-06-12T02:37:37.779 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.779 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.779 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:37.780 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.244081+0000", 2020-06-12T02:37:37.780 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010040778 2020-06-12T02:37:37.780 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.780 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.780 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:37.780 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.254122+0000", 2020-06-12T02:37:37.781 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010453100000000001 2020-06-12T02:37:37.781 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.781 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.781 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:37.781 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.254226+0000", 2020-06-12T02:37:37.782 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000153626 2020-06-12T02:37:37.782 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.782 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.782 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:37.783 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.254380+0000", 2020-06-12T02:37:37.783 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018271632999999999 2020-06-12T02:37:37.783 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.783 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:37.783 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.784 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.784 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.784 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:963 3.7s0 3:ea271007:::smithi10729407-9:head [append 0~606208] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:37.784 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:32.239688+0000", 2020-06-12T02:37:37.784 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 5.5216457700000001, 2020-06-12T02:37:37.784 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016181383000000001, 2020-06-12T02:37:37.785 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:37.785 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:37.785 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:37.785 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:37.785 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:37.786 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 963 2020-06-12T02:37:37.786 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.786 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:37.786 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.786 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:37.786 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.239688+0000", 2020-06-12T02:37:37.787 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:37.787 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.787 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.788 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:37.788 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.239688+0000", 2020-06-12T02:37:37.788 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00022408000000000001 2020-06-12T02:37:37.788 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.788 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.789 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:37.789 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.239913+0000", 2020-06-12T02:37:37.789 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0399999999999993e-06 2020-06-12T02:37:37.789 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.789 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.790 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:37.790 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.239921+0000", 2020-06-12T02:37:37.790 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7499999999999998e-05 2020-06-12T02:37:37.790 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.790 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.791 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:37.791 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.239938+0000", 2020-06-12T02:37:37.791 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0739999999999998e-06 2020-06-12T02:37:37.791 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.791 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.791 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:37.792 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.239944+0000", 2020-06-12T02:37:37.792 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2867000000000002e-05 2020-06-12T02:37:37.792 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.792 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.792 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.793 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.239977+0000", 2020-06-12T02:37:37.793 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4425000000000007e-05 2020-06-12T02:37:37.793 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.793 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.794 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:37.794 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.240051+0000", 2020-06-12T02:37:37.794 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0064351139999999996 2020-06-12T02:37:37.794 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.794 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.794 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.795 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.246487+0000", 2020-06-12T02:37:37.795 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012310899999999999 2020-06-12T02:37:37.795 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.795 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.795 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:37.796 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.246610+0000", 2020-06-12T02:37:37.796 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00079552599999999996 2020-06-12T02:37:37.796 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.796 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.796 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:37.796 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.247405+0000", 2020-06-12T02:37:37.797 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0079562030000000002 2020-06-12T02:37:37.797 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.797 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.797 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:37.797 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.255361+0000", 2020-06-12T02:37:37.798 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036610400000000002 2020-06-12T02:37:37.798 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.798 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.798 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:37.798 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.255728+0000", 2020-06-12T02:37:37.798 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014234100000000001 2020-06-12T02:37:37.799 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.799 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.799 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:37.799 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.255870+0000", 2020-06-12T02:37:37.799 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016181383000000001 2020-06-12T02:37:37.800 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.800 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:37.800 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.800 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.800 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.800 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:965 3.7s0 3:ea271007:::smithi10729407-9:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:37.801 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:32.242343+0000", 2020-06-12T02:37:37.801 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 5.5189909080000001, 2020-06-12T02:37:37.801 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014576832, 2020-06-12T02:37:37.801 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:37.801 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:37.802 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:37.802 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:37.802 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:37.802 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 965 2020-06-12T02:37:37.802 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.802 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:37.803 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.803 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:37.803 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242343+0000", 2020-06-12T02:37:37.804 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:37.804 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.804 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.804 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:37.804 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242343+0000", 2020-06-12T02:37:37.804 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5201e-05 2020-06-12T02:37:37.805 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.805 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.805 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:37.805 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242419+0000", 2020-06-12T02:37:37.805 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3489999999999997e-06 2020-06-12T02:37:37.806 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.806 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.806 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:37.806 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242425+0000", 2020-06-12T02:37:37.806 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1494999999999999e-05 2020-06-12T02:37:37.806 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.807 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.807 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:37.807 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242436+0000", 2020-06-12T02:37:37.807 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7410000000000004e-06 2020-06-12T02:37:37.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.808 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.808 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:37.808 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242446+0000", 2020-06-12T02:37:37.808 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2800999999999999e-05 2020-06-12T02:37:37.809 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.809 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.809 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.809 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242489+0000", 2020-06-12T02:37:37.810 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7392999999999996e-05 2020-06-12T02:37:37.810 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.810 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.810 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:37.810 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242566+0000", 2020-06-12T02:37:37.811 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011936647 2020-06-12T02:37:37.811 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.811 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.811 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.812 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.254503+0000", 2020-06-12T02:37:37.812 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114613 2020-06-12T02:37:37.812 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.813 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.813 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:37.813 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.254618+0000", 2020-06-12T02:37:37.813 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038691300000000001 2020-06-12T02:37:37.814 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.814 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.814 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:37.814 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.255004+0000", 2020-06-12T02:37:37.814 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014359850000000001 2020-06-12T02:37:37.814 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.815 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.815 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:37.815 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.256440+0000", 2020-06-12T02:37:37.815 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1623000000000001e-05 2020-06-12T02:37:37.815 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.816 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.816 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:37.816 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.256522+0000", 2020-06-12T02:37:37.816 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000398071 2020-06-12T02:37:37.816 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.817 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.817 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:37.817 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.256920+0000", 2020-06-12T02:37:37.817 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014576832 2020-06-12T02:37:37.817 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.817 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:37.818 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.818 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.818 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.818 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:966 3.7s0 3:ea271007:::smithi10729407-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:37.819 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:32.242493+0000", 2020-06-12T02:37:37.819 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 5.518840859, 2020-06-12T02:37:37.819 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015989771, 2020-06-12T02:37:37.819 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:37.819 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:37:37.820 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:37.820 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:37.820 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:37.820 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 966 2020-06-12T02:37:37.820 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.820 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:37.821 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.821 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:37.821 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242493+0000", 2020-06-12T02:37:37.821 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:37.821 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.822 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.822 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:37.822 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242493+0000", 2020-06-12T02:37:37.822 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6276000000000002e-05 2020-06-12T02:37:37.822 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.822 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.823 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:37.823 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242540+0000", 2020-06-12T02:37:37.823 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4710000000000004e-06 2020-06-12T02:37:37.823 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.824 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.824 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:37.824 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242546+0000", 2020-06-12T02:37:37.824 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3157e-05 2020-06-12T02:37:37.824 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.825 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.825 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:37.825 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242559+0000", 2020-06-12T02:37:37.825 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7419999999999997e-06 2020-06-12T02:37:37.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.825 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.826 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:37.826 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242564+0000", 2020-06-12T02:37:37.826 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8620999999999999e-05 2020-06-12T02:37:37.826 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.826 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.827 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.827 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242593+0000", 2020-06-12T02:37:37.827 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0400000000000003e-05 2020-06-12T02:37:37.827 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.827 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.827 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:37.828 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242673+0000", 2020-06-12T02:37:37.828 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01258803 2020-06-12T02:37:37.828 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.828 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.828 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.829 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.255261+0000", 2020-06-12T02:37:37.829 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5599000000000004e-05 2020-06-12T02:37:37.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.829 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.829 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:37.829 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.255357+0000", 2020-06-12T02:37:37.830 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001947999 2020-06-12T02:37:37.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.830 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.830 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.830 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.257305+0000", 2020-06-12T02:37:37.831 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5921999999999994e-05 2020-06-12T02:37:37.831 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.831 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.831 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:37.831 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.257401+0000", 2020-06-12T02:37:37.831 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0010825539999999999 2020-06-12T02:37:37.832 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.832 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.832 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:37.832 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.258483+0000", 2020-06-12T02:37:37.832 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015989771 2020-06-12T02:37:37.833 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.833 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:37.833 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.833 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.833 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.833 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1200 3.2s0 3:4afffc6f:::smithi10729407-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:37.834 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:32.837068+0000", 2020-06-12T02:37:37.834 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 4.9242662419999998, 2020-06-12T02:37:37.834 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014863888, 2020-06-12T02:37:37.834 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:37.835 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:37:37.835 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:37.835 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:37.835 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:37.835 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1200 2020-06-12T02:37:37.836 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.836 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:37.836 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.836 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:37.836 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.837068+0000", 2020-06-12T02:37:37.836 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:37.837 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.837 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.837 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:37.837 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.837068+0000", 2020-06-12T02:37:37.837 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2849000000000001e-05 2020-06-12T02:37:37.838 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.838 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.838 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:37.838 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.837091+0000", 2020-06-12T02:37:37.838 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0290000000000001e-06 2020-06-12T02:37:37.838 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.839 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.839 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:37.839 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.837094+0000", 2020-06-12T02:37:37.839 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1800000000000001e-06 2020-06-12T02:37:37.839 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.839 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.840 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:37.840 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.837100+0000", 2020-06-12T02:37:37.840 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2120000000000002e-06 2020-06-12T02:37:37.840 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.840 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.841 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:37.841 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.837102+0000", 2020-06-12T02:37:37.841 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090177509999999992 2020-06-12T02:37:37.841 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.841 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.842 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.842 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.846120+0000", 2020-06-12T02:37:37.842 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8610000000000003e-05 2020-06-12T02:37:37.842 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.842 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.842 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:37.843 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.846179+0000", 2020-06-12T02:37:37.843 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045638520000000002 2020-06-12T02:37:37.843 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.843 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.843 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.844 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.850742+0000", 2020-06-12T02:37:37.844 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0758000000000002e-05 2020-06-12T02:37:37.844 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.844 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.844 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:37.845 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.850823+0000", 2020-06-12T02:37:37.845 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011086469999999999 2020-06-12T02:37:37.845 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.845 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.845 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:37.846 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.851932+0000", 2020-06-12T02:37:37.846 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014863888 2020-06-12T02:37:37.846 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.846 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:37.846 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.846 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.847 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.847 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1252 3.8s0 3:14244450:::smithi10729407-2:head [append 0~516096] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:37.847 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:32.999793+0000", 2020-06-12T02:37:37.847 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 4.7615415140000001, 2020-06-12T02:37:37.847 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017380335, 2020-06-12T02:37:37.848 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:37.848 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:37.848 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:37.848 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:37.848 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:37.848 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1252 2020-06-12T02:37:37.849 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.849 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:37.849 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.849 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:37.849 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.999793+0000", 2020-06-12T02:37:37.850 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:37.850 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.850 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.850 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:37.850 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.999793+0000", 2020-06-12T02:37:37.850 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000208051 2020-06-12T02:37:37.851 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.851 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.851 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:37.851 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.000001+0000", 2020-06-12T02:37:37.851 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9750000000000002e-06 2020-06-12T02:37:37.852 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.852 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.852 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:37.852 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.000011+0000", 2020-06-12T02:37:37.852 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7733000000000001e-05 2020-06-12T02:37:37.852 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.853 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.853 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:37.853 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.000028+0000", 2020-06-12T02:37:37.853 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7379999999999994e-06 2020-06-12T02:37:37.853 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.853 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.854 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:37.854 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.000036+0000", 2020-06-12T02:37:37.854 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.414e-05 2020-06-12T02:37:37.854 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.855 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.855 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.855 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.000090+0000", 2020-06-12T02:37:37.855 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012561699999999999 2020-06-12T02:37:37.855 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.856 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.856 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:37.856 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.000216+0000", 2020-06-12T02:37:37.856 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00079436199999999995 2020-06-12T02:37:37.856 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.856 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.857 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:37.857 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.001010+0000", 2020-06-12T02:37:37.857 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015996506000000001 2020-06-12T02:37:37.857 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.857 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.858 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:37.858 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.017007+0000", 2020-06-12T02:37:37.858 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0599999999999997e-05 2020-06-12T02:37:37.859 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.859 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.859 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:37.859 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.017057+0000", 2020-06-12T02:37:37.859 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011561300000000001 2020-06-12T02:37:37.859 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.860 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.860 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:37.860 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.017173+0000", 2020-06-12T02:37:37.860 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017380335 2020-06-12T02:37:37.860 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.861 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:37.861 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.861 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.861 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.861 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1253 3.8s0 3:14244450:::smithi10729407-2:head [append 0~524288] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:37.861 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.002800+0000", 2020-06-12T02:37:37.862 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 4.7585338259999999, 2020-06-12T02:37:37.862 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014566312999999999, 2020-06-12T02:37:37.862 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:37.862 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:37.862 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:37.863 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:37.863 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:37.863 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1253 2020-06-12T02:37:37.863 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.863 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:37.863 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.864 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:37.864 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.002800+0000", 2020-06-12T02:37:37.864 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:37.864 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.865 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.865 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:37.865 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.002800+0000", 2020-06-12T02:37:37.865 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00020157899999999999 2020-06-12T02:37:37.865 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.866 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.866 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:37.866 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.003002+0000", 2020-06-12T02:37:37.866 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6059999999999992e-06 2020-06-12T02:37:37.866 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.866 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.867 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:37.867 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.003012+0000", 2020-06-12T02:37:37.867 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5407e-05 2020-06-12T02:37:37.867 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.867 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.868 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:37.868 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.003027+0000", 2020-06-12T02:37:37.868 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.391e-06 2020-06-12T02:37:37.868 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.868 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.868 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:37.869 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.003034+0000", 2020-06-12T02:37:37.869 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0010026569999999999 2020-06-12T02:37:37.869 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.869 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.869 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.870 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.004037+0000", 2020-06-12T02:37:37.870 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4276000000000003e-05 2020-06-12T02:37:37.870 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.870 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.870 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:37.870 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.004131+0000", 2020-06-12T02:37:37.871 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00068651200000000002 2020-06-12T02:37:37.871 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.871 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.871 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:37.871 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.004818+0000", 2020-06-12T02:37:37.871 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012360715 2020-06-12T02:37:37.872 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.872 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.872 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:37.872 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.017179+0000", 2020-06-12T02:37:37.872 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6847000000000001e-05 2020-06-12T02:37:37.873 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.873 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.873 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:37.873 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.017225+0000", 2020-06-12T02:37:37.873 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000141323 2020-06-12T02:37:37.873 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.874 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.874 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:37.874 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.017367+0000", 2020-06-12T02:37:37.874 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014566312999999999 2020-06-12T02:37:37.874 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.875 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:37.875 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.875 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.875 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.876 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1299 3.bs0 3:d0ebae7d:::smithi10729407-3:head [append 0~770048] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:37.876 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.162816+0000", 2020-06-12T02:37:37.876 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 4.5985183760000004, 2020-06-12T02:37:37.876 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016787209000000001, 2020-06-12T02:37:37.876 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:37.876 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:37.877 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:37.877 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:37.877 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:37.877 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1299 2020-06-12T02:37:37.877 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.878 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:37.878 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.878 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:37.878 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.162816+0000", 2020-06-12T02:37:37.878 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:37.878 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.879 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.879 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:37.879 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.162816+0000", 2020-06-12T02:37:37.879 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044992799999999999 2020-06-12T02:37:37.879 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.880 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.880 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:37.880 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.163266+0000", 2020-06-12T02:37:37.880 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6959999999999999e-06 2020-06-12T02:37:37.881 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.881 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.881 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:37.881 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.163272+0000", 2020-06-12T02:37:37.881 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3028e-05 2020-06-12T02:37:37.882 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.882 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.882 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:37.882 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.163286+0000", 2020-06-12T02:37:37.882 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2829999999999998e-06 2020-06-12T02:37:37.882 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.883 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:37.883 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.163292+0000", 2020-06-12T02:37:37.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1761000000000002e-05 2020-06-12T02:37:37.883 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.884 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.884 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.163334+0000", 2020-06-12T02:37:37.884 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6691000000000001e-05 2020-06-12T02:37:37.884 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.885 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:37.885 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.163410+0000", 2020-06-12T02:37:37.885 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004070176 2020-06-12T02:37:37.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.886 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.886 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.886 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.167480+0000", 2020-06-12T02:37:37.886 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7137000000000003e-05 2020-06-12T02:37:37.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:37.887 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.167568+0000", 2020-06-12T02:37:37.887 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000789648 2020-06-12T02:37:37.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.888 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:37.888 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168357+0000", 2020-06-12T02:37:37.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011078226 2020-06-12T02:37:37.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.889 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:37.889 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.179435+0000", 2020-06-12T02:37:37.889 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6649000000000003e-05 2020-06-12T02:37:37.889 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.889 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.889 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:37.890 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.179522+0000", 2020-06-12T02:37:37.890 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0986e-05 2020-06-12T02:37:37.890 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.890 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.890 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:37.891 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.179603+0000", 2020-06-12T02:37:37.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016787209000000001 2020-06-12T02:37:37.891 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.891 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:37.891 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.891 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.892 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.892 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1300 3.bs0 3:d0ebae7d:::smithi10729407-3:head [append 0~557056] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:37.892 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.166253+0000", 2020-06-12T02:37:37.892 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 4.5950808680000002, 2020-06-12T02:37:37.892 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019037724999999998, 2020-06-12T02:37:37.893 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:37.893 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:37.893 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:37.893 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:37.893 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:37.893 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1300 2020-06-12T02:37:37.894 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.894 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:37.894 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.894 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:37.894 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.166253+0000", 2020-06-12T02:37:37.895 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:37.895 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.895 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.895 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:37.896 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.166253+0000", 2020-06-12T02:37:37.897 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034533600000000002 2020-06-12T02:37:37.897 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.897 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.897 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:37.897 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.166599+0000", 2020-06-12T02:37:37.898 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7699999999999996e-06 2020-06-12T02:37:37.898 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.898 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.898 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:37.898 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.166608+0000", 2020-06-12T02:37:37.899 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7314999999999999e-05 2020-06-12T02:37:37.899 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.899 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.899 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:37.899 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.166626+0000", 2020-06-12T02:37:37.899 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9260000000000003e-06 2020-06-12T02:37:37.900 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.900 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.900 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:37.900 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.166633+0000", 2020-06-12T02:37:37.900 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0266000000000002e-05 2020-06-12T02:37:37.901 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.901 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.901 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.903 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.166693+0000", 2020-06-12T02:37:37.903 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6692000000000005e-05 2020-06-12T02:37:37.903 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.903 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.904 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:37.904 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.166760+0000", 2020-06-12T02:37:37.904 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0057626860000000004 2020-06-12T02:37:37.904 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.904 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.905 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.906 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.172522+0000", 2020-06-12T02:37:37.906 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000130493 2020-06-12T02:37:37.906 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.906 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.906 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:37.911 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.172653+0000", 2020-06-12T02:37:37.911 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00083376299999999995 2020-06-12T02:37:37.911 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.912 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.912 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:37.912 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.173487+0000", 2020-06-12T02:37:37.912 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0061197739999999997 2020-06-12T02:37:37.912 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.913 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.913 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:37.913 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.179606+0000", 2020-06-12T02:37:37.913 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0056160699999999999 2020-06-12T02:37:37.913 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.914 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.914 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:37.914 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.185222+0000", 2020-06-12T02:37:37.914 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8633999999999994e-05 2020-06-12T02:37:37.914 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.914 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.915 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:37.915 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.185291+0000", 2020-06-12T02:37:37.915 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019037724999999998 2020-06-12T02:37:37.915 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.915 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:37.916 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.916 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.916 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.916 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1301 3.bs0 3:d0ebae7d:::smithi10729407-3:head [append 0~278528] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:37.916 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.168146+0000", 2020-06-12T02:37:37.916 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 4.5931883109999996, 2020-06-12T02:37:37.917 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017691083999999999, 2020-06-12T02:37:37.917 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:37.917 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:37.917 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:37.917 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:37.918 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:37.918 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1301 2020-06-12T02:37:37.918 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.918 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:37.918 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.918 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:37.919 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168146+0000", 2020-06-12T02:37:37.919 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:37.919 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.919 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.919 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:37.920 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168146+0000", 2020-06-12T02:37:37.920 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000213666 2020-06-12T02:37:37.920 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.920 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.920 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:37.920 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168360+0000", 2020-06-12T02:37:37.921 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7009999999999996e-06 2020-06-12T02:37:37.921 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.921 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.921 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:37.921 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168367+0000", 2020-06-12T02:37:37.922 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5096000000000001e-05 2020-06-12T02:37:37.922 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.922 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.922 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:37.922 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168382+0000", 2020-06-12T02:37:37.923 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8739999999999997e-06 2020-06-12T02:37:37.923 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.923 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.923 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:37.924 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168389+0000", 2020-06-12T02:37:37.924 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0076120199999999997 2020-06-12T02:37:37.924 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.924 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.924 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.925 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.176001+0000", 2020-06-12T02:37:37.925 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9181000000000001e-05 2020-06-12T02:37:37.925 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.925 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.925 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:37.926 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.176060+0000", 2020-06-12T02:37:37.926 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00039587599999999998 2020-06-12T02:37:37.926 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.926 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.926 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:37.926 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.176456+0000", 2020-06-12T02:37:37.927 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0031522329999999999 2020-06-12T02:37:37.927 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.927 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.927 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:37.927 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.179609+0000", 2020-06-12T02:37:37.928 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0061683670000000001 2020-06-12T02:37:37.928 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.928 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.928 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:37.928 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.185777+0000", 2020-06-12T02:37:37.928 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0069999999999999e-05 2020-06-12T02:37:37.929 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.929 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.929 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:37.929 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.185837+0000", 2020-06-12T02:37:37.929 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017691083999999999 2020-06-12T02:37:37.930 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.930 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:37.930 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.930 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.930 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.930 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1302 3.bs0 3:d0ebae7d:::smithi10729407-3:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:37.931 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.168445+0000", 2020-06-12T02:37:37.931 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 4.5928890310000003, 2020-06-12T02:37:37.931 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017581657000000001, 2020-06-12T02:37:37.931 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:37.931 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:37.932 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:37.932 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:37.932 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:37.932 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1302 2020-06-12T02:37:37.932 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.932 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:37.933 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.933 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:37.933 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168445+0000", 2020-06-12T02:37:37.933 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:37.933 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.934 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.934 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:37.934 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168445+0000", 2020-06-12T02:37:37.934 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3532e-05 2020-06-12T02:37:37.934 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.934 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.935 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:37.935 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168519+0000", 2020-06-12T02:37:37.935 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2600000000000002e-06 2020-06-12T02:37:37.935 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.935 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.936 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:37.936 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168525+0000", 2020-06-12T02:37:37.936 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4338999999999999e-05 2020-06-12T02:37:37.936 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.936 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.937 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:37.937 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168539+0000", 2020-06-12T02:37:37.938 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7330000000000003e-06 2020-06-12T02:37:37.938 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.939 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.939 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:37.939 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168544+0000", 2020-06-12T02:37:37.939 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0095046090000000007 2020-06-12T02:37:37.939 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.940 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.940 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.940 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.178049+0000", 2020-06-12T02:37:37.940 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1524000000000003e-05 2020-06-12T02:37:37.940 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.941 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.941 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:37.941 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.178110+0000", 2020-06-12T02:37:37.941 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030514799999999999 2020-06-12T02:37:37.941 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.942 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.942 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:37.942 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.178415+0000", 2020-06-12T02:37:37.942 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011944480000000001 2020-06-12T02:37:37.942 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.942 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.943 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:37.943 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.179610+0000", 2020-06-12T02:37:37.943 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0063372849999999998 2020-06-12T02:37:37.943 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.943 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.944 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:37.944 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.185947+0000", 2020-06-12T02:37:37.944 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9778999999999996e-05 2020-06-12T02:37:37.944 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.944 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.944 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:37.945 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.186027+0000", 2020-06-12T02:37:37.945 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017581657000000001 2020-06-12T02:37:37.945 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.945 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:37.945 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.946 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.946 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.946 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1303 3.bs0 3:d0ebae7d:::smithi10729407-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:37.946 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.168587+0000", 2020-06-12T02:37:37.946 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 4.5927471129999997, 2020-06-12T02:37:37.946 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019127155, 2020-06-12T02:37:37.947 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:37.947 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:37:37.947 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:37.947 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:37.947 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:37.948 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1303 2020-06-12T02:37:37.948 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.948 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:37.948 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.948 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:37.948 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168587+0000", 2020-06-12T02:37:37.949 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:37.949 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.949 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.949 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:37.949 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168587+0000", 2020-06-12T02:37:37.950 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3099999999999997e-05 2020-06-12T02:37:37.950 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.950 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.950 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:37.951 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168630+0000", 2020-06-12T02:37:37.951 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1600000000000003e-06 2020-06-12T02:37:37.951 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.951 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.951 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:37.951 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168636+0000", 2020-06-12T02:37:37.952 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3439e-05 2020-06-12T02:37:37.952 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.952 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.952 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:37.952 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168650+0000", 2020-06-12T02:37:37.953 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7850000000000002e-06 2020-06-12T02:37:37.953 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.953 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.953 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:37.953 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168654+0000", 2020-06-12T02:37:37.954 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010326317 2020-06-12T02:37:37.954 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.954 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.954 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.954 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.178980+0000", 2020-06-12T02:37:37.954 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6593000000000004e-05 2020-06-12T02:37:37.955 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.955 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.955 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:37.955 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.179047+0000", 2020-06-12T02:37:37.955 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0073410990000000002 2020-06-12T02:37:37.956 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.956 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.956 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.956 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.186388+0000", 2020-06-12T02:37:37.956 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9405000000000001e-05 2020-06-12T02:37:37.956 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.957 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.957 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:37.957 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.186487+0000", 2020-06-12T02:37:37.957 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001227257 2020-06-12T02:37:37.957 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.958 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.958 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:37.958 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.187714+0000", 2020-06-12T02:37:37.958 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019127155 2020-06-12T02:37:37.958 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.959 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:37.959 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.959 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.962 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.962 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1523 3.2s0 3:4afffc6f:::smithi10729407-20:head [append 0~507904] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-06-12T02:37:37.962 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.746564+0000", 2020-06-12T02:37:37.962 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 4.0147704080000004, 2020-06-12T02:37:37.963 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016382178000000001, 2020-06-12T02:37:37.963 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:37.963 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:37.963 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:37.963 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:37.964 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:37.964 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1523 2020-06-12T02:37:37.964 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.964 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:37.964 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.965 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:37.965 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.746564+0000", 2020-06-12T02:37:37.965 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:37.965 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.965 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.966 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:37.966 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.746564+0000", 2020-06-12T02:37:37.966 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013497300000000001 2020-06-12T02:37:37.966 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.966 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.966 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:37.967 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.746699+0000", 2020-06-12T02:37:37.967 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0549999999999999e-06 2020-06-12T02:37:37.967 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.967 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.969 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:37.970 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.746704+0000", 2020-06-12T02:37:37.970 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7439999999999996e-06 2020-06-12T02:37:37.970 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.971 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.971 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:37.971 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.746713+0000", 2020-06-12T02:37:37.972 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5009999999999998e-06 2020-06-12T02:37:37.972 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.972 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.973 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:37.973 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.746717+0000", 2020-06-12T02:37:37.973 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012975580000000001 2020-06-12T02:37:37.974 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.974 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.975 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.975 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.748015+0000", 2020-06-12T02:37:37.976 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5599000000000007e-05 2020-06-12T02:37:37.976 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.976 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.977 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:37.977 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.748080+0000", 2020-06-12T02:37:37.977 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044923640000000004 2020-06-12T02:37:37.978 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.978 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.978 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:37.979 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.752573+0000", 2020-06-12T02:37:37.979 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2525999999999998e-05 2020-06-12T02:37:37.980 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.980 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.980 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:37.980 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.752625+0000", 2020-06-12T02:37:37.981 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044074400000000002 2020-06-12T02:37:37.981 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.981 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.981 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:37.982 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.753066+0000", 2020-06-12T02:37:37.982 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0051370260000000003 2020-06-12T02:37:37.982 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.982 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.982 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:37.983 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.758203+0000", 2020-06-12T02:37:37.983 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046689590000000003 2020-06-12T02:37:37.983 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.983 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.983 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:37.984 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.762872+0000", 2020-06-12T02:37:37.984 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4128999999999994e-05 2020-06-12T02:37:37.984 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.984 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.984 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:37.985 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.762946+0000", 2020-06-12T02:37:37.985 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016382178000000001 2020-06-12T02:37:37.985 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.985 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:37.985 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:37.986 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.986 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.987 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1525 3.2s0 3:4afffc6f:::smithi10729407-20:head [append 0~286720] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-06-12T02:37:37.987 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.750576+0000", 2020-06-12T02:37:37.987 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 4.0107586529999999, 2020-06-12T02:37:37.988 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015530921, 2020-06-12T02:37:37.988 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:37.988 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:37.989 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:37.989 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:37.989 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:37.990 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1525 2020-06-12T02:37:37.990 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.991 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:37.991 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.991 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:37.992 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750576+0000", 2020-06-12T02:37:37.992 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:37.992 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.992 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.993 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:37.993 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750576+0000", 2020-06-12T02:37:37.993 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7447000000000004e-05 2020-06-12T02:37:37.994 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.994 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.994 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:37.995 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750673+0000", 2020-06-12T02:37:37.995 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4089999999999996e-06 2020-06-12T02:37:37.995 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.995 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.996 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:37.996 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750677+0000", 2020-06-12T02:37:37.996 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.789e-06 2020-06-12T02:37:37.996 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.997 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.997 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:37.997 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750686+0000", 2020-06-12T02:37:37.998 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.967e-06 2020-06-12T02:37:37.998 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.998 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:37.998 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:37.999 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750691+0000", 2020-06-12T02:37:37.999 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4802000000000003e-05 2020-06-12T02:37:37.999 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:37.999 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.000 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:38.000 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750726+0000", 2020-06-12T02:37:38.001 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0620999999999999e-05 2020-06-12T02:37:38.001 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.001 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.001 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:38.002 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750817+0000", 2020-06-12T02:37:38.002 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0074052679999999996 2020-06-12T02:37:38.002 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.002 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.003 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:38.003 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.758222+0000", 2020-06-12T02:37:38.003 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5697000000000004e-05 2020-06-12T02:37:38.004 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.004 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.004 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:38.004 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.758298+0000", 2020-06-12T02:37:38.005 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00046273100000000003 2020-06-12T02:37:38.005 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.005 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.005 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:38.005 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.758760+0000", 2020-06-12T02:37:38.006 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0029510069999999998 2020-06-12T02:37:38.006 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.006 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.006 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:38.006 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.761711+0000", 2020-06-12T02:37:38.007 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043243439999999999 2020-06-12T02:37:38.007 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.007 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.007 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:38.007 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.766036+0000", 2020-06-12T02:37:38.008 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0839000000000004e-05 2020-06-12T02:37:38.008 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.008 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.008 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:38.008 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.766107+0000", 2020-06-12T02:37:38.009 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015530921 2020-06-12T02:37:38.009 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:38.009 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:38.009 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:38.009 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.009 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.010 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1526 3.2s0 3:4afffc6f:::smithi10729407-20:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-06-12T02:37:38.010 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.750720+0000", 2020-06-12T02:37:38.010 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 4.0106142470000004, 2020-06-12T02:37:38.010 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015683592999999999, 2020-06-12T02:37:38.011 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:38.011 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:38.011 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:38.011 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:38.011 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:38.012 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1526 2020-06-12T02:37:38.012 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.012 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:38.012 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.012 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:38.013 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750720+0000", 2020-06-12T02:37:38.013 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:38.013 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.013 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.014 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:38.014 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750720+0000", 2020-06-12T02:37:38.014 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8281999999999997e-05 2020-06-12T02:37:38.014 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.014 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.015 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:38.015 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750758+0000", 2020-06-12T02:37:38.015 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3189999999999999e-06 2020-06-12T02:37:38.015 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.015 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.016 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:38.016 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750762+0000", 2020-06-12T02:37:38.016 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8490000000000002e-06 2020-06-12T02:37:38.016 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.016 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.016 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:38.017 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750767+0000", 2020-06-12T02:37:38.017 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.4080000000000001e-06 2020-06-12T02:37:38.017 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.017 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.017 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:38.018 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750770+0000", 2020-06-12T02:37:38.018 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5030999999999994e-05 2020-06-12T02:37:38.018 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.018 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.018 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:38.019 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750845+0000", 2020-06-12T02:37:38.019 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8128000000000003e-05 2020-06-12T02:37:38.019 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.019 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.020 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:38.020 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750903+0000", 2020-06-12T02:37:38.020 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097174800000000006 2020-06-12T02:37:38.020 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.020 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.020 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:38.021 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.760620+0000", 2020-06-12T02:37:38.021 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044212800000000002 2020-06-12T02:37:38.021 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.022 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.022 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:38.022 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.761063+0000", 2020-06-12T02:37:38.022 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00033657099999999997 2020-06-12T02:37:38.022 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.022 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.023 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:38.023 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.761399+0000", 2020-06-12T02:37:38.023 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00046871599999999999 2020-06-12T02:37:38.023 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.023 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.024 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:38.024 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.761868+0000", 2020-06-12T02:37:38.024 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044489129999999997 2020-06-12T02:37:38.024 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.024 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.025 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:38.025 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.766317+0000", 2020-06-12T02:37:38.025 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6767999999999993e-05 2020-06-12T02:37:38.025 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.025 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.025 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:38.026 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.766404+0000", 2020-06-12T02:37:38.026 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015683592999999999 2020-06-12T02:37:38.026 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:38.026 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:38.026 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:38.027 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.027 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.027 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1527 3.2s0 3:4afffc6f:::smithi10729407-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e26)", 2020-06-12T02:37:38.027 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.750793+0000", 2020-06-12T02:37:38.027 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 4.0105409590000001, 2020-06-12T02:37:38.028 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016832799999999998, 2020-06-12T02:37:38.028 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:38.028 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:37:38.028 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:38.028 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:38.028 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:38.029 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1527 2020-06-12T02:37:38.029 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.029 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:38.029 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.029 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:38.030 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750793+0000", 2020-06-12T02:37:38.030 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:38.030 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.030 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.030 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:38.031 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750793+0000", 2020-06-12T02:37:38.031 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3218e-05 2020-06-12T02:37:38.031 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.031 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.032 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:38.032 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750817+0000", 2020-06-12T02:37:38.032 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0939999999999999e-06 2020-06-12T02:37:38.032 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.032 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.033 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:38.033 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750820+0000", 2020-06-12T02:37:38.033 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4810000000000003e-06 2020-06-12T02:37:38.033 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.033 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.033 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:38.034 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750826+0000", 2020-06-12T02:37:38.034 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.277e-06 2020-06-12T02:37:38.034 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.034 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.034 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:38.035 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750828+0000", 2020-06-12T02:37:38.035 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7738000000000005e-05 2020-06-12T02:37:38.035 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.035 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.035 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:38.036 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750926+0000", 2020-06-12T02:37:38.036 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6310000000000005e-05 2020-06-12T02:37:38.036 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.036 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.036 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:38.036 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.751012+0000", 2020-06-12T02:37:38.037 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010605622 2020-06-12T02:37:38.037 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.037 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.037 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:38.037 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.761618+0000", 2020-06-12T02:37:38.038 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2778000000000002e-05 2020-06-12T02:37:38.038 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.038 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.038 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:38.038 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.761671+0000", 2020-06-12T02:37:38.039 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.005069044 2020-06-12T02:37:38.039 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.039 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.039 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:38.039 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.766740+0000", 2020-06-12T02:37:38.039 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6335999999999997e-05 2020-06-12T02:37:38.040 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.040 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.040 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:38.040 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.766796+0000", 2020-06-12T02:37:38.040 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00082990199999999998 2020-06-12T02:37:38.041 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.041 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.041 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:38.042 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.767626+0000", 2020-06-12T02:37:38.042 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016832799999999998 2020-06-12T02:37:38.042 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:38.042 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:38.042 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:38.043 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.043 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.043 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s1 26/21 ECSubWrite(tid=408, reqid=client.4512.0:1572, at_version=26'99, trim_to=0'0, roll_forward_to=26'97))", 2020-06-12T02:37:38.043 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.887944+0000", 2020-06-12T02:37:38.043 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.8733904799999999, 2020-06-12T02:37:38.044 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019136641999999999, 2020-06-12T02:37:38.044 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:38.044 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:38.046 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:38.046 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.046 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:38.046 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.887944+0000", 2020-06-12T02:37:38.046 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:38.047 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.047 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.047 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:38.047 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.887944+0000", 2020-06-12T02:37:38.047 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00063708500000000002 2020-06-12T02:37:38.047 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.048 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.048 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:38.048 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888581+0000", 2020-06-12T02:37:38.048 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9644e-05 2020-06-12T02:37:38.048 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.049 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.049 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:38.049 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888600+0000", 2020-06-12T02:37:38.049 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.721e-05 2020-06-12T02:37:38.049 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.050 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.050 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:38.050 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888618+0000", 2020-06-12T02:37:38.050 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7500000000000003e-06 2020-06-12T02:37:38.051 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.051 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.051 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:38.051 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888625+0000", 2020-06-12T02:37:38.051 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2707999999999998e-05 2020-06-12T02:37:38.051 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.052 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.052 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:38.052 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888668+0000", 2020-06-12T02:37:38.052 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0491e-05 2020-06-12T02:37:38.052 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.053 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.053 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:38.053 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888679+0000", 2020-06-12T02:37:38.054 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018351985000000001 2020-06-12T02:37:38.054 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.054 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.054 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:38.054 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.907031+0000", 2020-06-12T02:37:38.054 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9768999999999997e-05 2020-06-12T02:37:38.055 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.055 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.055 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:38.055 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.907080+0000", 2020-06-12T02:37:38.056 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019136641999999999 2020-06-12T02:37:38.056 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:38.056 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:38.056 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:38.057 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.057 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.057 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s1 26/21 ECSubWrite(tid=409, reqid=client.4512.0:1573, at_version=26'100, trim_to=0'0, roll_forward_to=26'98))", 2020-06-12T02:37:38.057 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.892432+0000", 2020-06-12T02:37:38.057 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.8689018900000001, 2020-06-12T02:37:38.058 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01685733, 2020-06-12T02:37:38.058 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:38.058 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:38.058 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:38.059 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.059 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:38.059 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.892432+0000", 2020-06-12T02:37:38.059 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:38.059 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.060 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.060 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:38.060 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.892432+0000", 2020-06-12T02:37:38.061 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059245900000000004 2020-06-12T02:37:38.061 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.061 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.061 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:38.061 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.893025+0000", 2020-06-12T02:37:38.062 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1591e-05 2020-06-12T02:37:38.062 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.062 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.062 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:38.062 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.893046+0000", 2020-06-12T02:37:38.062 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2429000000000002e-05 2020-06-12T02:37:38.063 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.063 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.063 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:38.063 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.893069+0000", 2020-06-12T02:37:38.064 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0389999999999993e-06 2020-06-12T02:37:38.064 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.064 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.064 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:38.064 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.893078+0000", 2020-06-12T02:37:38.065 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012883099999999999 2020-06-12T02:37:38.065 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.065 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.065 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:38.065 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.893207+0000", 2020-06-12T02:37:38.066 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2407e-05 2020-06-12T02:37:38.066 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.066 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.066 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:38.066 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.893219+0000", 2020-06-12T02:37:38.066 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016029015000000001 2020-06-12T02:37:38.067 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.067 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.067 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:38.067 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.909248+0000", 2020-06-12T02:37:38.067 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1559e-05 2020-06-12T02:37:38.068 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.068 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.068 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:38.068 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.909290+0000", 2020-06-12T02:37:38.068 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01685733 2020-06-12T02:37:38.068 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:38.069 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:38.069 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:38.069 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.069 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.069 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1577 3.bs0 3:d09543d2:::smithi10729407-16:head [append 0~614400] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-06-12T02:37:38.070 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.895667+0000", 2020-06-12T02:37:38.070 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.8656668060000001, 2020-06-12T02:37:38.070 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015968431000000002, 2020-06-12T02:37:38.070 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:38.070 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:38.070 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:38.071 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:38.071 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:38.071 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1577 2020-06-12T02:37:38.071 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.071 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:38.072 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.072 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:38.072 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895667+0000", 2020-06-12T02:37:38.072 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:38.072 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.072 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.073 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:38.073 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895667+0000", 2020-06-12T02:37:38.073 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019894699999999999 2020-06-12T02:37:38.073 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.073 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.074 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:38.074 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895866+0000", 2020-06-12T02:37:38.074 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0057e-05 2020-06-12T02:37:38.075 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.075 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.076 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:38.076 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895876+0000", 2020-06-12T02:37:38.076 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5666000000000001e-05 2020-06-12T02:37:38.076 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.076 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.077 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:38.077 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895902+0000", 2020-06-12T02:37:38.077 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6510000000000001e-06 2020-06-12T02:37:38.077 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.077 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.078 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:38.078 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895910+0000", 2020-06-12T02:37:38.078 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5513000000000003e-05 2020-06-12T02:37:38.078 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.078 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.079 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:38.079 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895955+0000", 2020-06-12T02:37:38.079 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000106716 2020-06-12T02:37:38.079 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.079 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.080 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:38.080 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.896062+0000", 2020-06-12T02:37:38.080 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00063197000000000004 2020-06-12T02:37:38.080 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.080 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.080 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:38.081 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.896694+0000", 2020-06-12T02:37:38.081 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014775 2020-06-12T02:37:38.081 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.081 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.081 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:38.082 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.911469+0000", 2020-06-12T02:37:38.082 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4872999999999999e-05 2020-06-12T02:37:38.082 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.082 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.083 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:38.083 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.911554+0000", 2020-06-12T02:37:38.083 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2038000000000003e-05 2020-06-12T02:37:38.083 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.084 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.084 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:38.084 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.911636+0000", 2020-06-12T02:37:38.084 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015968431000000002 2020-06-12T02:37:38.084 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:38.084 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:38.085 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:38.085 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.085 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.085 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1641 3.bs0 3:d0ebae7d:::smithi10729407-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e26)", 2020-06-12T02:37:38.085 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:34.010321+0000", 2020-06-12T02:37:38.086 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.7510129540000001, 2020-06-12T02:37:38.087 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014458459999999999, 2020-06-12T02:37:38.087 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:38.087 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:37:38.088 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:38.088 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:38.088 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:38.088 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1641 2020-06-12T02:37:38.088 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.089 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:38.089 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.089 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:38.089 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:34.010321+0000", 2020-06-12T02:37:38.089 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:38.090 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.090 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.090 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:38.090 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:34.010321+0000", 2020-06-12T02:37:38.090 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9689e-05 2020-06-12T02:37:38.091 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.091 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.091 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:38.091 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:34.010351+0000", 2020-06-12T02:37:38.091 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5370000000000002e-06 2020-06-12T02:37:38.092 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.092 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.092 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:38.092 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:34.010355+0000", 2020-06-12T02:37:38.092 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9539999999999998e-06 2020-06-12T02:37:38.092 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.093 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.093 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:38.093 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:34.010361+0000", 2020-06-12T02:37:38.093 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6759999999999999e-06 2020-06-12T02:37:38.093 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.094 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.094 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:38.094 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:34.010364+0000", 2020-06-12T02:37:38.094 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0026005450000000001 2020-06-12T02:37:38.094 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.095 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.095 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:38.095 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:34.012965+0000", 2020-06-12T02:37:38.095 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1150000000000003e-05 2020-06-12T02:37:38.095 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.096 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.096 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:38.096 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:34.013016+0000", 2020-06-12T02:37:38.096 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0040653460000000001 2020-06-12T02:37:38.097 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.097 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.097 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:38.097 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:34.017081+0000", 2020-06-12T02:37:38.097 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000100735 2020-06-12T02:37:38.098 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.098 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.098 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:38.098 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:34.017182+0000", 2020-06-12T02:37:38.098 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0055364780000000001 2020-06-12T02:37:38.098 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.099 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.099 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:38.099 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:34.022718+0000", 2020-06-12T02:37:38.099 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2418999999999997e-05 2020-06-12T02:37:38.099 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.100 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.100 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:38.100 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:34.022811+0000", 2020-06-12T02:37:38.100 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0019689310000000002 2020-06-12T02:37:38.100 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:38.100 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:38.101 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:38.101 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:34.024780+0000", 2020-06-12T02:37:38.101 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014458459999999999 2020-06-12T02:37:38.101 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:38.101 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:38.102 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:38.102 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:38.102 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:38.102 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:37:38.102 INFO:tasks.rados.rados.0.smithi107.stdout:1390: done (2 left) 2020-06-12T02:37:38.103 INFO:tasks.rados.rados.0.smithi107.stdout:1392: done (1 left) 2020-06-12T02:37:38.103 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:37:38.103 INFO:tasks.rados.rados.0.smithi107.stdout:1393: seq_num 1392 ranges {1540096=516096,2056192=483328,2539520=720896} 2020-06-12T02:37:38.126 INFO:tasks.rados.rados.0.smithi107.stdout:1393: writing smithi10729407-19 from 1540096 to 2056192 tid 1 2020-06-12T02:37:38.127 INFO:tasks.rados.rados.0.smithi107.stdout:1393: writing smithi10729407-19 from 2056192 to 2539520 tid 2 2020-06-12T02:37:38.127 INFO:tasks.rados.rados.0.smithi107.stdout:1393: writing smithi10729407-19 from 2539520 to 3260416 tid 3 2020-06-12T02:37:38.127 INFO:tasks.rados.rados.0.smithi107.stdout:1394: delete oid 1 current snap is 0 2020-06-12T02:37:38.127 INFO:tasks.rados.rados.0.smithi107.stdout:1391: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:38.127 INFO:tasks.rados.rados.0.smithi107.stdout:1391: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:38.128 INFO:tasks.rados.rados.0.smithi107.stdout:1391: finishing write tid 5 to smithi10729407-6 2020-06-12T02:37:38.128 INFO:tasks.rados.rados.0.smithi107.stdout:1391: finishing write tid 6 to smithi10729407-6 2020-06-12T02:37:38.128 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 274 (ObjNum 1390 snap 0 seq_num 1390) dirty exists 2020-06-12T02:37:38.128 INFO:tasks.rados.rados.0.smithi107.stdout:1391: left oid 6 (ObjNum 1390 snap 0 seq_num 1390) 2020-06-12T02:37:38.128 INFO:tasks.rados.rados.0.smithi107.stdout:1393: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:38.129 INFO:tasks.rados.rados.0.smithi107.stdout:1393: finishing write tid 2 to smithi10729407-19 2020-06-12T02:37:38.129 INFO:tasks.rados.rados.0.smithi107.stdout:1391: done (2 left) 2020-06-12T02:37:38.129 INFO:tasks.rados.rados.0.smithi107.stdout:1394: done (1 left) 2020-06-12T02:37:38.129 INFO:tasks.rados.rados.0.smithi107.stdout:1395: delete oid 3 current snap is 0 2020-06-12T02:37:38.130 INFO:tasks.rados.rados.0.smithi107.stdout:1395: done (1 left) 2020-06-12T02:37:38.130 INFO:tasks.rados.rados.0.smithi107.stdout:1396: delete oid 4 current snap is 0 2020-06-12T02:37:38.130 INFO:tasks.rados.rados.0.smithi107.stdout:1393: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:38.130 INFO:tasks.rados.rados.0.smithi107.stdout:1393: finishing write tid 5 to smithi10729407-19 2020-06-12T02:37:38.130 INFO:tasks.rados.rados.0.smithi107.stdout:1393: finishing write tid 6 to smithi10729407-19 2020-06-12T02:37:38.131 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 227 (ObjNum 1392 snap 0 seq_num 1392) dirty exists 2020-06-12T02:37:38.131 INFO:tasks.rados.rados.0.smithi107.stdout:1393: left oid 19 (ObjNum 1392 snap 0 seq_num 1392) 2020-06-12T02:37:38.131 INFO:tasks.rados.rados.0.smithi107.stdout:1393: done (1 left) 2020-06-12T02:37:38.131 INFO:tasks.rados.rados.0.smithi107.stdout:1396: done (0 left) 2020-06-12T02:37:38.131 INFO:tasks.rados.rados.0.smithi107.stdout:1397: delete oid 2 current snap is 0 2020-06-12T02:37:38.131 INFO:tasks.rados.rados.0.smithi107.stdout:1397: done (0 left) 2020-06-12T02:37:38.132 INFO:tasks.rados.rados.0.smithi107.stdout:1398: delete oid 5 current snap is 0 2020-06-12T02:37:38.132 INFO:tasks.rados.rados.0.smithi107.stdout:1398: done (0 left) 2020-06-12T02:37:38.132 INFO:tasks.rados.rados.0.smithi107.stdout:1399: delete oid 14 current snap is 0 2020-06-12T02:37:38.132 INFO:tasks.rados.rados.0.smithi107.stdout:1399: done (0 left) 2020-06-12T02:37:38.132 INFO:tasks.rados.rados.0.smithi107.stdout:1400: delete oid 20 current snap is 0 2020-06-12T02:37:38.133 INFO:tasks.rados.rados.0.smithi107.stdout:1400: done (0 left) 2020-06-12T02:37:38.133 INFO:tasks.rados.rados.0.smithi107.stdout:1401: delete oid 9 current snap is 0 2020-06-12T02:37:38.133 INFO:tasks.rados.rados.0.smithi107.stdout:1401: done (0 left) 2020-06-12T02:37:38.133 INFO:tasks.rados.rados.0.smithi107.stdout:1402: delete oid 8 current snap is 0 2020-06-12T02:37:38.133 INFO:tasks.rados.rados.0.smithi107.stdout:1402: done (0 left) 2020-06-12T02:37:38.133 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:37:38.134 INFO:tasks.rados.rados.0.smithi107.stdout:1403: seq_num 1402 ranges {0=688128,688128=581632,1269760=622592} 2020-06-12T02:37:38.134 INFO:tasks.rados.rados.0.smithi107.stdout:1403: writing smithi10729407-15 from 0 to 688128 tid 1 2020-06-12T02:37:38.134 INFO:tasks.rados.rados.0.smithi107.stdout:1403: writing smithi10729407-15 from 688128 to 1269760 tid 2 2020-06-12T02:37:38.134 INFO:tasks.rados.rados.0.smithi107.stdout:1403: writing smithi10729407-15 from 1269760 to 1892352 tid 3 2020-06-12T02:37:38.134 INFO:tasks.rados.rados.0.smithi107.stdout:1404: delete oid 10 current snap is 0 2020-06-12T02:37:38.135 INFO:tasks.rados.rados.0.smithi107.stdout:1403: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:38.135 INFO:tasks.rados.rados.0.smithi107.stdout:1404: done (1 left) 2020-06-12T02:37:38.135 INFO:tasks.rados.rados.0.smithi107.stdout:1405: delete oid 5 current snap is 0 2020-06-12T02:37:38.135 INFO:tasks.rados.rados.0.smithi107.stdout:1405: done (1 left) 2020-06-12T02:37:38.135 INFO:tasks.rados.rados.0.smithi107.stdout:1406: delete oid 4 current snap is 0 2020-06-12T02:37:38.136 INFO:tasks.rados.rados.0.smithi107.stdout:1403: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:38.136 INFO:tasks.rados.rados.0.smithi107.stdout:1403: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:38.136 INFO:tasks.rados.rados.0.smithi107.stdout:1403: finishing write tid 5 to smithi10729407-15 2020-06-12T02:37:38.136 INFO:tasks.rados.rados.0.smithi107.stdout:1406: done (1 left) 2020-06-12T02:37:38.136 INFO:tasks.rados.rados.0.smithi107.stdout:1407: delete oid 19 current snap is 0 2020-06-12T02:37:38.136 INFO:tasks.rados.rados.0.smithi107.stdout:1403: finishing write tid 6 to smithi10729407-15 2020-06-12T02:37:38.137 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 242 (ObjNum 1402 snap 0 seq_num 1402) dirty exists 2020-06-12T02:37:38.137 INFO:tasks.rados.rados.0.smithi107.stdout:1403: left oid 15 (ObjNum 1402 snap 0 seq_num 1402) 2020-06-12T02:37:38.137 INFO:tasks.rados.rados.0.smithi107.stdout:1403: done (1 left) 2020-06-12T02:37:38.137 INFO:tasks.rados.rados.0.smithi107.stdout:1407: done (0 left) 2020-06-12T02:37:38.137 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:37:38.138 INFO:tasks.rados.rados.0.smithi107.stdout:1408: seq_num 1407 ranges {0=770048,770048=450560,1220608=516096,1736704=237568} 2020-06-12T02:37:38.138 INFO:tasks.rados.rados.0.smithi107.stdout:1408: writing smithi10729407-17 from 0 to 770048 tid 1 2020-06-12T02:37:38.138 INFO:tasks.rados.rados.0.smithi107.stdout:1408: writing smithi10729407-17 from 770048 to 1220608 tid 2 2020-06-12T02:37:38.138 INFO:tasks.rados.rados.0.smithi107.stdout:1408: writing smithi10729407-17 from 1220608 to 1736704 tid 3 2020-06-12T02:37:38.138 INFO:tasks.rados.rados.0.smithi107.stdout:1408: writing smithi10729407-17 from 1736704 to 1974272 tid 4 2020-06-12T02:37:38.138 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:37:38.139 INFO:tasks.rados.rados.0.smithi107.stdout:1409: seq_num 1408 ranges {0=786432,786432=737280,1523712=466944} 2020-06-12T02:37:38.139 INFO:tasks.rados.rados.0.smithi107.stdout:1409: writing smithi10729407-14 from 0 to 786432 tid 1 2020-06-12T02:37:38.139 INFO:tasks.rados.rados.0.smithi107.stdout:1409: writing smithi10729407-14 from 786432 to 1523712 tid 2 2020-06-12T02:37:38.139 INFO:tasks.rados.rados.0.smithi107.stdout:1409: writing smithi10729407-14 from 1523712 to 1990656 tid 3 2020-06-12T02:37:38.140 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:37:38.140 INFO:tasks.rados.rados.0.smithi107.stdout:1410: seq_num 1409 ranges {0=802816,802816=622592,1425408=475136,1900544=106496} 2020-06-12T02:37:38.140 INFO:tasks.rados.rados.0.smithi107.stdout:1410: writing smithi10729407-1 from 0 to 802816 tid 1 2020-06-12T02:37:38.140 INFO:tasks.rados.rados.0.smithi107.stdout:1410: writing smithi10729407-1 from 802816 to 1425408 tid 2 2020-06-12T02:37:38.140 INFO:tasks.rados.rados.0.smithi107.stdout:1410: writing smithi10729407-1 from 1425408 to 1900544 tid 3 2020-06-12T02:37:38.141 INFO:tasks.rados.rados.0.smithi107.stdout:1410: writing smithi10729407-1 from 1900544 to 2007040 tid 4 2020-06-12T02:37:38.141 INFO:tasks.rados.rados.0.smithi107.stdout:1411: delete oid 4 current snap is 0 2020-06-12T02:37:38.141 INFO:tasks.rados.rados.0.smithi107.stdout:1408: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:38.141 INFO:tasks.rados.rados.0.smithi107.stdout:1408: finishing write tid 2 to smithi10729407-17 2020-06-12T02:37:38.141 INFO:tasks.rados.rados.0.smithi107.stdout:1408: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:38.141 INFO:tasks.rados.rados.0.smithi107.stdout:1408: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:38.142 INFO:tasks.rados.rados.0.smithi107.stdout:1408: finishing write tid 6 to smithi10729407-17 2020-06-12T02:37:38.142 INFO:tasks.rados.rados.0.smithi107.stdout:1408: finishing write tid 7 to smithi10729407-17 2020-06-12T02:37:38.142 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 273 (ObjNum 1407 snap 0 seq_num 1407) dirty exists 2020-06-12T02:37:38.142 INFO:tasks.rados.rados.0.smithi107.stdout:1408: left oid 17 (ObjNum 1407 snap 0 seq_num 1407) 2020-06-12T02:37:38.142 INFO:tasks.rados.rados.0.smithi107.stdout:1409: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:38.143 INFO:tasks.rados.rados.0.smithi107.stdout:1409: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:38.143 INFO:tasks.rados.rados.0.smithi107.stdout:1409: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:38.143 INFO:tasks.rados.rados.0.smithi107.stdout:1409: finishing write tid 5 to smithi10729407-14 2020-06-12T02:37:38.143 INFO:tasks.rados.rados.0.smithi107.stdout:1408: done (3 left) 2020-06-12T02:37:38.143 INFO:tasks.rados.rados.0.smithi107.stdout:1411: done (2 left) 2020-06-12T02:37:38.143 INFO:tasks.rados.rados.0.smithi107.stdout:1412: delete oid 7 current snap is 0 2020-06-12T02:37:38.144 INFO:tasks.rados.rados.0.smithi107.stdout:1409: finishing write tid 6 to smithi10729407-14 2020-06-12T02:37:38.144 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 246 (ObjNum 1408 snap 0 seq_num 1408) dirty exists 2020-06-12T02:37:38.144 INFO:tasks.rados.rados.0.smithi107.stdout:1409: left oid 14 (ObjNum 1408 snap 0 seq_num 1408) 2020-06-12T02:37:38.144 INFO:tasks.rados.rados.0.smithi107.stdout:1410: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:38.144 INFO:tasks.rados.rados.0.smithi107.stdout:1409: done (2 left) 2020-06-12T02:37:38.145 INFO:tasks.rados.rados.0.smithi107.stdout:1412: done (1 left) 2020-06-12T02:37:38.145 INFO:tasks.rados.rados.0.smithi107.stdout:1413: delete oid 19 current snap is 0 2020-06-12T02:37:38.145 INFO:tasks.rados.rados.0.smithi107.stdout:1413: done (1 left) 2020-06-12T02:37:38.145 INFO:tasks.rados.rados.0.smithi107.stdout:1414: delete oid 9 current snap is 0 2020-06-12T02:37:38.145 INFO:tasks.rados.rados.0.smithi107.stdout:1414: done (1 left) 2020-06-12T02:37:38.145 INFO:tasks.rados.rados.0.smithi107.stdout:1415: delete oid 5 current snap is 0 2020-06-12T02:37:38.146 INFO:tasks.rados.rados.0.smithi107.stdout:1415: done (1 left) 2020-06-12T02:37:38.146 INFO:tasks.rados.rados.0.smithi107.stdout:1416: delete oid 15 current snap is 0 2020-06-12T02:37:38.146 INFO:tasks.rados.rados.0.smithi107.stdout:1410: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:38.146 INFO:tasks.rados.rados.0.smithi107.stdout:1410: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:38.146 INFO:tasks.rados.rados.0.smithi107.stdout:1410: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:38.147 INFO:tasks.rados.rados.0.smithi107.stdout:1416: done (1 left) 2020-06-12T02:37:38.147 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:37:38.147 INFO:tasks.rados.rados.0.smithi107.stdout:1417: seq_num 1416 ranges {0=524288,524288=663552,1187840=630784,1818624=311296} 2020-06-12T02:37:38.147 INFO:tasks.rados.rados.0.smithi107.stdout:1417: writing smithi10729407-7 from 0 to 524288 tid 1 2020-06-12T02:37:38.147 INFO:tasks.rados.rados.0.smithi107.stdout:1417: writing smithi10729407-7 from 524288 to 1187840 tid 2 2020-06-12T02:37:38.147 INFO:tasks.rados.rados.0.smithi107.stdout:1417: writing smithi10729407-7 from 1187840 to 1818624 tid 3 2020-06-12T02:37:38.148 INFO:tasks.rados.rados.0.smithi107.stdout:1417: writing smithi10729407-7 from 1818624 to 2129920 tid 4 2020-06-12T02:37:38.148 INFO:tasks.rados.rados.0.smithi107.stdout:1418: delete oid 4 current snap is 0 2020-06-12T02:37:38.148 INFO:tasks.rados.rados.0.smithi107.stdout:1410: finishing write tid 6 to smithi10729407-1 2020-06-12T02:37:38.149 INFO:tasks.rados.rados.0.smithi107.stdout:1410: finishing write tid 7 to smithi10729407-1 2020-06-12T02:37:38.149 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 280 (ObjNum 1409 snap 0 seq_num 1409) dirty exists 2020-06-12T02:37:38.149 INFO:tasks.rados.rados.0.smithi107.stdout:1410: left oid 1 (ObjNum 1409 snap 0 seq_num 1409) 2020-06-12T02:37:38.150 INFO:tasks.rados.rados.0.smithi107.stdout:1417: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:38.151 INFO:tasks.rados.rados.0.smithi107.stdout:1410: done (2 left) 2020-06-12T02:37:38.151 INFO:tasks.rados.rados.0.smithi107.stdout:1418: done (1 left) 2020-06-12T02:37:38.151 INFO:tasks.rados.rados.0.smithi107.stdout:1419: delete oid 17 current snap is 0 2020-06-12T02:37:38.151 INFO:tasks.rados.rados.0.smithi107.stdout:1419: done (1 left) 2020-06-12T02:37:38.151 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:37:38.152 INFO:tasks.rados.rados.0.smithi107.stdout:1420: seq_num 1419 ranges {0=573440,573440=712704,1286144=450560,1736704=442368} 2020-06-12T02:37:38.152 INFO:tasks.rados.rados.0.smithi107.stdout:1420: writing smithi10729407-9 from 0 to 573440 tid 1 2020-06-12T02:37:38.152 INFO:tasks.rados.rados.0.smithi107.stdout:1420: writing smithi10729407-9 from 573440 to 1286144 tid 2 2020-06-12T02:37:38.152 INFO:tasks.rados.rados.0.smithi107.stdout:1420: writing smithi10729407-9 from 1286144 to 1736704 tid 3 2020-06-12T02:37:38.152 INFO:tasks.rados.rados.0.smithi107.stdout:1420: writing smithi10729407-9 from 1736704 to 2179072 tid 4 2020-06-12T02:37:38.153 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:37:38.153 INFO:tasks.rados.rados.0.smithi107.stdout:1421: seq_num 1420 ranges {0=589824,589824=598016,1187840=614400,1802240=393216} 2020-06-12T02:37:38.153 INFO:tasks.rados.rados.0.smithi107.stdout:1421: writing smithi10729407-15 from 0 to 589824 tid 1 2020-06-12T02:37:38.153 INFO:tasks.rados.rados.0.smithi107.stdout:1421: writing smithi10729407-15 from 589824 to 1187840 tid 2 2020-06-12T02:37:38.153 INFO:tasks.rados.rados.0.smithi107.stdout:1421: writing smithi10729407-15 from 1187840 to 1802240 tid 3 2020-06-12T02:37:38.154 INFO:tasks.rados.rados.0.smithi107.stdout:1421: writing smithi10729407-15 from 1802240 to 2195456 tid 4 2020-06-12T02:37:38.154 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:37:38.154 INFO:tasks.rados.rados.0.smithi107.stdout:1422: seq_num 1421 ranges {0=606208,606208=483328,1089536=417792,1507328=704512} 2020-06-12T02:37:38.154 INFO:tasks.rados.rados.0.smithi107.stdout:1422: writing smithi10729407-5 from 0 to 606208 tid 1 2020-06-12T02:37:38.154 INFO:tasks.rados.rados.0.smithi107.stdout:1422: writing smithi10729407-5 from 606208 to 1089536 tid 2 2020-06-12T02:37:38.155 INFO:tasks.rados.rados.0.smithi107.stdout:1422: writing smithi10729407-5 from 1089536 to 1507328 tid 3 2020-06-12T02:37:38.155 INFO:tasks.rados.rados.0.smithi107.stdout:1422: writing smithi10729407-5 from 1507328 to 2211840 tid 4 2020-06-12T02:37:38.155 INFO:tasks.rados.rados.0.smithi107.stdout:1423: delete oid 11 current snap is 0 2020-06-12T02:37:38.155 INFO:tasks.rados.rados.0.smithi107.stdout:1417: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:38.155 INFO:tasks.rados.rados.0.smithi107.stdout:1417: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:38.155 INFO:tasks.rados.rados.0.smithi107.stdout:1417: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:38.156 INFO:tasks.rados.rados.0.smithi107.stdout:1417: finishing write tid 6 to smithi10729407-7 2020-06-12T02:37:38.156 INFO:tasks.rados.rados.0.smithi107.stdout:1417: finishing write tid 7 to smithi10729407-7 2020-06-12T02:37:38.156 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 294 (ObjNum 1416 snap 0 seq_num 1416) dirty exists 2020-06-12T02:37:38.156 INFO:tasks.rados.rados.0.smithi107.stdout:1417: left oid 7 (ObjNum 1416 snap 0 seq_num 1416) 2020-06-12T02:37:38.156 INFO:tasks.rados.rados.0.smithi107.stdout:1420: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:38.157 INFO:tasks.rados.rados.0.smithi107.stdout:1420: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:38.157 INFO:tasks.rados.rados.0.smithi107.stdout:1420: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:38.157 INFO:tasks.rados.rados.0.smithi107.stdout:1420: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:38.157 INFO:tasks.rados.rados.0.smithi107.stdout:1420: finishing write tid 6 to smithi10729407-9 2020-06-12T02:37:38.157 INFO:tasks.rados.rados.0.smithi107.stdout:1421: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:38.157 INFO:tasks.rados.rados.0.smithi107.stdout:1420: finishing write tid 7 to smithi10729407-9 2020-06-12T02:37:38.158 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 257 (ObjNum 1419 snap 0 seq_num 1419) dirty exists 2020-06-12T02:37:38.158 INFO:tasks.rados.rados.0.smithi107.stdout:1420: left oid 9 (ObjNum 1419 snap 0 seq_num 1419) 2020-06-12T02:37:38.158 INFO:tasks.rados.rados.0.smithi107.stdout:1421: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:38.158 INFO:tasks.rados.rados.0.smithi107.stdout:1422: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:38.158 INFO:tasks.rados.rados.0.smithi107.stdout:1421: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:38.159 INFO:tasks.rados.rados.0.smithi107.stdout:1421: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:38.159 INFO:tasks.rados.rados.0.smithi107.stdout:1421: finishing write tid 6 to smithi10729407-15 2020-06-12T02:37:38.160 INFO:tasks.rados.rados.0.smithi107.stdout:1421: finishing write tid 7 to smithi10729407-15 2020-06-12T02:37:38.160 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 248 (ObjNum 1420 snap 0 seq_num 1420) dirty exists 2020-06-12T02:37:38.160 INFO:tasks.rados.rados.0.smithi107.stdout:1421: left oid 15 (ObjNum 1420 snap 0 seq_num 1420) 2020-06-12T02:37:38.160 INFO:tasks.rados.rados.0.smithi107.stdout:1417: done (4 left) 2020-06-12T02:37:38.161 INFO:tasks.rados.rados.0.smithi107.stdout:1420: done (3 left) 2020-06-12T02:37:38.161 INFO:tasks.rados.rados.0.smithi107.stdout:1421: done (2 left) 2020-06-12T02:37:38.161 INFO:tasks.rados.rados.0.smithi107.stdout:1423: done (1 left) 2020-06-12T02:37:38.161 INFO:tasks.rados.rados.0.smithi107.stdout:1424: delete oid 18 current snap is 0 2020-06-12T02:37:38.162 INFO:tasks.rados.rados.0.smithi107.stdout:1424: done (1 left) 2020-06-12T02:37:38.162 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:37:38.162 INFO:tasks.rados.rados.0.smithi107.stdout:1425: seq_num 1424 ranges {1687552=655360,2342912=581632,2924544=589824,3514368=434176} 2020-06-12T02:37:38.162 INFO:tasks.rados.rados.0.smithi107.stdout:1425: writing smithi10729407-6 from 1687552 to 2342912 tid 1 2020-06-12T02:37:38.163 INFO:tasks.rados.rados.0.smithi107.stdout:1425: writing smithi10729407-6 from 2342912 to 2924544 tid 2 2020-06-12T02:37:38.163 INFO:tasks.rados.rados.0.smithi107.stdout:1425: writing smithi10729407-6 from 2924544 to 3514368 tid 3 2020-06-12T02:37:38.163 INFO:tasks.rados.rados.0.smithi107.stdout:1425: writing smithi10729407-6 from 3514368 to 3948544 tid 4 2020-06-12T02:37:38.163 INFO:tasks.rados.rados.0.smithi107.stdout:1426: delete oid 12 current snap is 0 2020-06-12T02:37:38.163 INFO:tasks.rados.rados.0.smithi107.stdout:1422: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:38.163 INFO:tasks.rados.rados.0.smithi107.stdout:1422: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:38.164 INFO:tasks.rados.rados.0.smithi107.stdout:1422: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:38.164 INFO:tasks.rados.rados.0.smithi107.stdout:1422: finishing write tid 6 to smithi10729407-5 2020-06-12T02:37:38.164 INFO:tasks.rados.rados.0.smithi107.stdout:1422: finishing write tid 7 to smithi10729407-5 2020-06-12T02:37:38.164 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 128 (ObjNum 1421 snap 0 seq_num 1421) dirty exists 2020-06-12T02:37:38.164 INFO:tasks.rados.rados.0.smithi107.stdout:1422: left oid 5 (ObjNum 1421 snap 0 seq_num 1421) 2020-06-12T02:37:38.165 INFO:tasks.rados.rados.0.smithi107.stdout:1425: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:38.165 INFO:tasks.rados.rados.0.smithi107.stdout:1422: done (2 left) 2020-06-12T02:37:38.165 INFO:tasks.rados.rados.0.smithi107.stdout:1426: done (1 left) 2020-06-12T02:37:38.165 INFO:tasks.rados.rados.0.smithi107.stdout:1427: delete oid 1 current snap is 0 2020-06-12T02:37:38.165 INFO:tasks.rados.rados.0.smithi107.stdout:1425: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:38.166 INFO:tasks.rados.rados.0.smithi107.stdout:1425: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:38.166 INFO:tasks.rados.rados.0.smithi107.stdout:1425: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:38.166 INFO:tasks.rados.rados.0.smithi107.stdout:1425: finishing write tid 6 to smithi10729407-6 2020-06-12T02:37:38.166 INFO:tasks.rados.rados.0.smithi107.stdout:1427: done (1 left) 2020-06-12T02:37:38.166 INFO:tasks.rados.rados.0.smithi107.stdout:1428: delete oid 19 current snap is 0 2020-06-12T02:37:38.166 INFO:tasks.rados.rados.0.smithi107.stdout:1428: done (1 left) 2020-06-12T02:37:38.167 INFO:tasks.rados.rados.0.smithi107.stdout:1429: delete oid 17 current snap is 0 2020-06-12T02:37:38.167 INFO:tasks.rados.rados.0.smithi107.stdout:1425: finishing write tid 7 to smithi10729407-6 2020-06-12T02:37:38.167 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 285 (ObjNum 1424 snap 0 seq_num 1424) dirty exists 2020-06-12T02:37:38.167 INFO:tasks.rados.rados.0.smithi107.stdout:1425: left oid 6 (ObjNum 1424 snap 0 seq_num 1424) 2020-06-12T02:37:38.167 INFO:tasks.rados.rados.0.smithi107.stdout:1425: done (1 left) 2020-06-12T02:37:38.168 INFO:tasks.rados.rados.0.smithi107.stdout:1429: done (0 left) 2020-06-12T02:37:38.168 INFO:tasks.rados.rados.0.smithi107.stdout:1430: delete oid 6 current snap is 0 2020-06-12T02:37:38.168 INFO:tasks.rados.rados.0.smithi107.stdout:1430: done (0 left) 2020-06-12T02:37:38.168 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:38.168 INFO:tasks.rados.rados.0.smithi107.stdout:1431: seq_num 1430 ranges {0=753664,753664=737280,1490944=589824,2080768=278528} 2020-06-12T02:37:38.169 INFO:tasks.rados.rados.0.smithi107.stdout:1431: writing smithi10729407-2 from 0 to 753664 tid 1 2020-06-12T02:37:38.169 INFO:tasks.rados.rados.0.smithi107.stdout:1431: writing smithi10729407-2 from 753664 to 1490944 tid 2 2020-06-12T02:37:38.169 INFO:tasks.rados.rados.0.smithi107.stdout:1431: writing smithi10729407-2 from 1490944 to 2080768 tid 3 2020-06-12T02:37:38.169 INFO:tasks.rados.rados.0.smithi107.stdout:1431: writing smithi10729407-2 from 2080768 to 2359296 tid 4 2020-06-12T02:37:38.169 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:37:38.170 INFO:tasks.rados.rados.0.smithi107.stdout:1432: seq_num 1431 ranges {2195456=770048,2965504=622592,3588096=794624,4382720=188416} 2020-06-12T02:37:38.170 INFO:tasks.rados.rados.0.smithi107.stdout:1432: writing smithi10729407-15 from 2195456 to 2965504 tid 1 2020-06-12T02:37:38.170 INFO:tasks.rados.rados.0.smithi107.stdout:1432: writing smithi10729407-15 from 2965504 to 3588096 tid 2 2020-06-12T02:37:38.170 INFO:tasks.rados.rados.0.smithi107.stdout:1432: writing smithi10729407-15 from 3588096 to 4382720 tid 3 2020-06-12T02:37:38.170 INFO:tasks.rados.rados.0.smithi107.stdout:1432: writing smithi10729407-15 from 4382720 to 4571136 tid 4 2020-06-12T02:37:38.171 INFO:tasks.rados.rados.0.smithi107.stdout:1433: delete oid 3 current snap is 0 2020-06-12T02:37:38.171 INFO:tasks.rados.rados.0.smithi107.stdout:1431: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:38.171 INFO:tasks.rados.rados.0.smithi107.stdout:1431: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:38.171 INFO:tasks.rados.rados.0.smithi107.stdout:1431: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:38.172 INFO:tasks.rados.rados.0.smithi107.stdout:1431: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:38.172 INFO:tasks.rados.rados.0.smithi107.stdout:1431: finishing write tid 6 to smithi10729407-2 2020-06-12T02:37:38.172 INFO:tasks.rados.rados.0.smithi107.stdout:1431: finishing write tid 7 to smithi10729407-2 2020-06-12T02:37:38.172 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 141 (ObjNum 1430 snap 0 seq_num 1430) dirty exists 2020-06-12T02:37:38.172 INFO:tasks.rados.rados.0.smithi107.stdout:1431: left oid 2 (ObjNum 1430 snap 0 seq_num 1430) 2020-06-12T02:37:38.172 INFO:tasks.rados.rados.0.smithi107.stdout:1432: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:38.173 INFO:tasks.rados.rados.0.smithi107.stdout:1431: done (2 left) 2020-06-12T02:37:38.173 INFO:tasks.rados.rados.0.smithi107.stdout:1433: done (1 left) 2020-06-12T02:37:38.173 INFO:tasks.rados.rados.0.smithi107.stdout:1434: delete oid 13 current snap is 0 2020-06-12T02:37:38.173 INFO:tasks.rados.rados.0.smithi107.stdout:1432: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:38.173 INFO:tasks.rados.rados.0.smithi107.stdout:1434: done (1 left) 2020-06-12T02:37:38.174 INFO:tasks.rados.rados.0.smithi107.stdout:1435: delete oid 19 current snap is 0 2020-06-12T02:37:38.174 INFO:tasks.rados.rados.0.smithi107.stdout:1432: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:38.174 INFO:tasks.rados.rados.0.smithi107.stdout:1432: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:38.174 INFO:tasks.rados.rados.0.smithi107.stdout:1432: finishing write tid 6 to smithi10729407-15 2020-06-12T02:37:38.174 INFO:tasks.rados.rados.0.smithi107.stdout:1435: done (1 left) 2020-06-12T02:37:38.175 INFO:tasks.rados.rados.0.smithi107.stdout:1436: delete oid 17 current snap is 0 2020-06-12T02:37:38.175 INFO:tasks.rados.rados.0.smithi107.stdout:1432: finishing write tid 7 to smithi10729407-15 2020-06-12T02:37:38.175 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 253 (ObjNum 1431 snap 0 seq_num 1431) dirty exists 2020-06-12T02:37:38.175 INFO:tasks.rados.rados.0.smithi107.stdout:1432: left oid 15 (ObjNum 1431 snap 0 seq_num 1431) 2020-06-12T02:37:38.175 INFO:tasks.rados.rados.0.smithi107.stdout:1432: done (1 left) 2020-06-12T02:37:38.175 INFO:tasks.rados.rados.0.smithi107.stdout:1436: done (0 left) 2020-06-12T02:37:38.176 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:37:38.176 INFO:tasks.rados.rados.0.smithi107.stdout:1437: seq_num 1436 ranges {0=57344} 2020-06-12T02:37:38.176 INFO:tasks.rados.rados.0.smithi107.stdout:1437: writing smithi10729407-17 from 0 to 57344 tid 1 2020-06-12T02:37:38.176 INFO:tasks.rados.rados.0.smithi107.stdout:1438: delete oid 11 current snap is 0 2020-06-12T02:37:38.176 INFO:tasks.rados.rados.0.smithi107.stdout:1438: done (1 left) 2020-06-12T02:37:38.177 INFO:tasks.rados.rados.0.smithi107.stdout:1439: delete oid 16 current snap is 0 2020-06-12T02:37:38.177 INFO:tasks.rados.rados.0.smithi107.stdout:1437: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:38.177 INFO:tasks.rados.rados.0.smithi107.stdout:1439: done (1 left) 2020-06-12T02:37:38.177 INFO:tasks.rados.rados.0.smithi107.stdout:1440: delete oid 5 current snap is 0 2020-06-12T02:37:38.177 INFO:tasks.rados.rados.0.smithi107.stdout:1437: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:38.178 INFO:tasks.rados.rados.0.smithi107.stdout:1437: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:38.178 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 280 (ObjNum 1436 snap 0 seq_num 1436) dirty exists 2020-06-12T02:37:38.179 INFO:tasks.rados.rados.0.smithi107.stdout:1437: left oid 17 (ObjNum 1436 snap 0 seq_num 1436) 2020-06-12T02:37:38.179 INFO:tasks.rados.rados.0.smithi107.stdout:1437: done (1 left) 2020-06-12T02:37:38.179 INFO:tasks.rados.rados.0.smithi107.stdout:1440: done (0 left) 2020-06-12T02:37:38.179 INFO:tasks.rados.rados.0.smithi107.stdout:1441: delete oid 10 current snap is 0 2020-06-12T02:37:38.180 INFO:tasks.rados.rados.0.smithi107.stdout:1441: done (0 left) 2020-06-12T02:37:38.180 INFO:tasks.rados.rados.0.smithi107.stdout:1442: delete oid 19 current snap is 0 2020-06-12T02:37:38.180 INFO:tasks.rados.rados.0.smithi107.stdout:1442: done (0 left) 2020-06-12T02:37:38.181 INFO:tasks.rados.rados.0.smithi107.stdout:1443: delete oid 10 current snap is 0 2020-06-12T02:37:38.181 INFO:tasks.rados.rados.0.smithi107.stdout:1443: done (0 left) 2020-06-12T02:37:38.181 INFO:tasks.rados.rados.0.smithi107.stdout:1444: delete oid 8 current snap is 0 2020-06-12T02:37:38.181 INFO:tasks.rados.rados.0.smithi107.stdout:1444: done (0 left) 2020-06-12T02:37:38.181 INFO:tasks.rados.rados.0.smithi107.stdout:1445: delete oid 20 current snap is 0 2020-06-12T02:37:38.181 INFO:tasks.rados.rados.0.smithi107.stdout:1445: done (0 left) 2020-06-12T02:37:38.182 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:37:38.182 INFO:tasks.rados.rados.0.smithi107.stdout:1446: seq_num 1445 ranges {0=204800} 2020-06-12T02:37:38.182 INFO:tasks.rados.rados.0.smithi107.stdout:1446: writing smithi10729407-12 from 0 to 204800 tid 1 2020-06-12T02:37:38.182 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:37:38.182 INFO:tasks.rados.rados.0.smithi107.stdout:1447: seq_num 1446 ranges {0=221184} 2020-06-12T02:37:38.183 INFO:tasks.rados.rados.0.smithi107.stdout:1447: writing smithi10729407-3 from 0 to 221184 tid 1 2020-06-12T02:37:38.183 INFO:tasks.rados.rados.0.smithi107.stdout:1448: delete oid 20 current snap is 0 2020-06-12T02:37:38.183 INFO:tasks.rados.rados.0.smithi107.stdout:1446: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:38.183 INFO:tasks.rados.rados.0.smithi107.stdout:1448: done (2 left) 2020-06-12T02:37:38.183 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:37:38.184 INFO:tasks.rados.rados.0.smithi107.stdout:1449: seq_num 1448 ranges {0=253952} 2020-06-12T02:37:38.184 INFO:tasks.rados.rados.0.smithi107.stdout:1449: writing smithi10729407-11 from 0 to 253952 tid 1 2020-06-12T02:37:38.184 INFO:tasks.rados.rados.0.smithi107.stdout:1450: delete oid 9 current snap is 0 2020-06-12T02:37:38.184 INFO:tasks.rados.rados.0.smithi107.stdout:1446: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:38.184 INFO:tasks.rados.rados.0.smithi107.stdout:1447: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:38.184 INFO:tasks.rados.rados.0.smithi107.stdout:1447: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:38.185 INFO:tasks.rados.rados.0.smithi107.stdout:1446: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:38.185 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 235 (ObjNum 1445 snap 0 seq_num 1445) dirty exists 2020-06-12T02:37:38.185 INFO:tasks.rados.rados.0.smithi107.stdout:1446: left oid 12 (ObjNum 1445 snap 0 seq_num 1445) 2020-06-12T02:37:38.185 INFO:tasks.rados.rados.0.smithi107.stdout:1447: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:38.185 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 251 (ObjNum 1446 snap 0 seq_num 1446) dirty exists 2020-06-12T02:37:38.186 INFO:tasks.rados.rados.0.smithi107.stdout:1447: left oid 3 (ObjNum 1446 snap 0 seq_num 1446) 2020-06-12T02:37:38.186 INFO:tasks.rados.rados.0.smithi107.stdout:1446: done (3 left) 2020-06-12T02:37:38.186 INFO:tasks.rados.rados.0.smithi107.stdout:1447: done (2 left) 2020-06-12T02:37:38.186 INFO:tasks.rados.rados.0.smithi107.stdout:1450: done (1 left) 2020-06-12T02:37:38.186 INFO:tasks.rados.rados.0.smithi107.stdout:1451: delete oid 1 current snap is 0 2020-06-12T02:37:38.187 INFO:tasks.rados.rados.0.smithi107.stdout:1449: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:38.187 INFO:tasks.rados.rados.0.smithi107.stdout:1449: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:38.187 INFO:tasks.rados.rados.0.smithi107.stdout:1449: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:38.187 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 250 (ObjNum 1448 snap 0 seq_num 1448) dirty exists 2020-06-12T02:37:38.187 INFO:tasks.rados.rados.0.smithi107.stdout:1449: left oid 11 (ObjNum 1448 snap 0 seq_num 1448) 2020-06-12T02:37:38.187 INFO:tasks.rados.rados.0.smithi107.stdout:1449: done (1 left) 2020-06-12T02:37:38.188 INFO:tasks.rados.rados.0.smithi107.stdout:1451: done (0 left) 2020-06-12T02:37:38.188 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:37:38.188 INFO:tasks.rados.rados.0.smithi107.stdout:1452: seq_num 1451 ranges {0=303104} 2020-06-12T02:37:38.188 INFO:tasks.rados.rados.0.smithi107.stdout:1452: writing smithi10729407-18 from 0 to 303104 tid 1 2020-06-12T02:37:38.188 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:38.189 INFO:tasks.rados.rados.0.smithi107.stdout:1453: seq_num 1452 ranges {2359296=319488} 2020-06-12T02:37:38.189 INFO:tasks.rados.rados.0.smithi107.stdout:1453: writing smithi10729407-2 from 2359296 to 2678784 tid 1 2020-06-12T02:37:38.189 INFO:tasks.rados.rados.0.smithi107.stdout:1454: delete oid 20 current snap is 0 2020-06-12T02:37:38.189 INFO:tasks.rados.rados.0.smithi107.stdout:1452: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:38.189 INFO:tasks.rados.rados.0.smithi107.stdout:1452: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:38.189 INFO:tasks.rados.rados.0.smithi107.stdout:1452: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:38.190 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 261 (ObjNum 1451 snap 0 seq_num 1451) dirty exists 2020-06-12T02:37:38.190 INFO:tasks.rados.rados.0.smithi107.stdout:1452: left oid 18 (ObjNum 1451 snap 0 seq_num 1451) 2020-06-12T02:37:38.190 INFO:tasks.rados.rados.0.smithi107.stdout:1452: done (2 left) 2020-06-12T02:37:38.190 INFO:tasks.rados.rados.0.smithi107.stdout:1454: done (1 left) 2020-06-12T02:37:38.190 INFO:tasks.rados.rados.0.smithi107.stdout:1455: delete oid 16 current snap is 0 2020-06-12T02:37:38.191 INFO:tasks.rados.rados.0.smithi107.stdout:1453: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:38.191 INFO:tasks.rados.rados.0.smithi107.stdout:1455: done (1 left) 2020-06-12T02:37:38.191 INFO:tasks.rados.rados.0.smithi107.stdout:1456: delete oid 12 current snap is 0 2020-06-12T02:37:38.191 INFO:tasks.rados.rados.0.smithi107.stdout:1453: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:38.191 INFO:tasks.rados.rados.0.smithi107.stdout:1456: done (1 left) 2020-06-12T02:37:38.191 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:37:38.192 INFO:tasks.rados.rados.0.smithi107.stdout:1457: seq_num 1456 ranges {57344=393216} 2020-06-12T02:37:38.192 INFO:tasks.rados.rados.0.smithi107.stdout:1457: writing smithi10729407-17 from 57344 to 450560 tid 1 2020-06-12T02:37:38.192 INFO:tasks.rados.rados.0.smithi107.stdout:1458: delete oid 13 current snap is 0 2020-06-12T02:37:38.192 INFO:tasks.rados.rados.0.smithi107.stdout:1453: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:38.192 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 143 (ObjNum 1452 snap 0 seq_num 1452) dirty exists 2020-06-12T02:37:38.193 INFO:tasks.rados.rados.0.smithi107.stdout:1453: left oid 2 (ObjNum 1452 snap 0 seq_num 1452) 2020-06-12T02:37:38.193 INFO:tasks.rados.rados.0.smithi107.stdout:1453: done (2 left) 2020-06-12T02:37:38.193 INFO:tasks.rados.rados.0.smithi107.stdout:1458: done (1 left) 2020-06-12T02:37:38.193 INFO:tasks.rados.rados.0.smithi107.stdout:1459: delete oid 6 current snap is 0 2020-06-12T02:37:38.193 INFO:tasks.rados.rados.0.smithi107.stdout:1457: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:38.194 INFO:tasks.rados.rados.0.smithi107.stdout:1457: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:38.194 INFO:tasks.rados.rados.0.smithi107.stdout:1459: done (1 left) 2020-06-12T02:37:38.194 INFO:tasks.rados.rados.0.smithi107.stdout:1460: delete oid 2 current snap is 0 2020-06-12T02:37:38.194 INFO:tasks.rados.rados.0.smithi107.stdout:1457: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:38.194 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 282 (ObjNum 1456 snap 0 seq_num 1456) dirty exists 2020-06-12T02:37:38.194 INFO:tasks.rados.rados.0.smithi107.stdout:1457: left oid 17 (ObjNum 1456 snap 0 seq_num 1456) 2020-06-12T02:37:38.195 INFO:tasks.rados.rados.0.smithi107.stdout:1457: done (1 left) 2020-06-12T02:37:38.195 INFO:tasks.rados.rados.0.smithi107.stdout:1460: done (0 left) 2020-06-12T02:37:38.195 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:37:38.195 INFO:tasks.rados.rados.0.smithi107.stdout:1461: seq_num 1460 ranges {0=458752} 2020-06-12T02:37:38.195 INFO:tasks.rados.rados.0.smithi107.stdout:1461: writing smithi10729407-8 from 0 to 458752 tid 1 2020-06-12T02:37:38.196 INFO:tasks.rados.rados.0.smithi107.stdout:1462: delete oid 7 current snap is 0 2020-06-12T02:37:38.196 INFO:tasks.rados.rados.0.smithi107.stdout:1462: done (1 left) 2020-06-12T02:37:38.196 INFO:tasks.rados.rados.0.smithi107.stdout:1463: delete oid 7 current snap is 0 2020-06-12T02:37:38.196 INFO:tasks.rados.rados.0.smithi107.stdout:1461: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:38.196 INFO:tasks.rados.rados.0.smithi107.stdout:1463: done (1 left) 2020-06-12T02:37:38.196 INFO:tasks.rados.rados.0.smithi107.stdout:1464: delete oid 4 current snap is 0 2020-06-12T02:37:38.197 INFO:tasks.rados.rados.0.smithi107.stdout:1464: done (1 left) 2020-06-12T02:37:38.197 INFO:tasks.rados.rados.0.smithi107.stdout:1465: delete oid 10 current snap is 0 2020-06-12T02:37:38.197 INFO:tasks.rados.rados.0.smithi107.stdout:1461: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:38.197 INFO:tasks.rados.rados.0.smithi107.stdout:1461: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:38.197 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 256 (ObjNum 1460 snap 0 seq_num 1460) dirty exists 2020-06-12T02:37:38.198 INFO:tasks.rados.rados.0.smithi107.stdout:1461: left oid 8 (ObjNum 1460 snap 0 seq_num 1460) 2020-06-12T02:37:38.198 INFO:tasks.rados.rados.0.smithi107.stdout:1461: done (1 left) 2020-06-12T02:37:38.198 INFO:tasks.rados.rados.0.smithi107.stdout:1465: done (0 left) 2020-06-12T02:37:38.198 INFO:tasks.rados.rados.0.smithi107.stdout:1466: delete oid 11 current snap is 0 2020-06-12T02:37:38.198 INFO:tasks.rados.rados.0.smithi107.stdout:1466: done (0 left) 2020-06-12T02:37:38.198 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:37:38.199 INFO:tasks.rados.rados.0.smithi107.stdout:1467: seq_num 1466 ranges {450560=557056} 2020-06-12T02:37:38.199 INFO:tasks.rados.rados.0.smithi107.stdout:1467: writing smithi10729407-17 from 450560 to 1007616 tid 1 2020-06-12T02:37:38.199 INFO:tasks.rados.rados.0.smithi107.stdout:1468: delete oid 16 current snap is 0 2020-06-12T02:37:38.199 INFO:tasks.rados.rados.0.smithi107.stdout:1468: done (1 left) 2020-06-12T02:37:38.199 INFO:tasks.rados.rados.0.smithi107.stdout:1469: delete oid 1 current snap is 0 2020-06-12T02:37:38.200 INFO:tasks.rados.rados.0.smithi107.stdout:1469: done (1 left) 2020-06-12T02:37:38.200 INFO:tasks.rados.rados.0.smithi107.stdout:1470: delete oid 18 current snap is 0 2020-06-12T02:37:38.200 INFO:tasks.rados.rados.0.smithi107.stdout:1467: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:38.200 INFO:tasks.rados.rados.0.smithi107.stdout:1470: done (1 left) 2020-06-12T02:37:38.200 INFO:tasks.rados.rados.0.smithi107.stdout:1471: delete oid 6 current snap is 0 2020-06-12T02:37:38.201 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:38.101+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:37:38.201 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-06-12T02:37:38.205 INFO:tasks.rados.rados.0.smithi107.stdout:1467: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:38.205 INFO:tasks.rados.rados.0.smithi107.stdout:1467: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:38.206 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 285 (ObjNum 1466 snap 0 seq_num 1466) dirty exists 2020-06-12T02:37:38.206 INFO:tasks.rados.rados.0.smithi107.stdout:1467: left oid 17 (ObjNum 1466 snap 0 seq_num 1466) 2020-06-12T02:37:38.206 INFO:tasks.rados.rados.0.smithi107.stdout:1467: done (1 left) 2020-06-12T02:37:38.206 INFO:tasks.rados.rados.0.smithi107.stdout:1471: done (0 left) 2020-06-12T02:37:38.206 INFO:tasks.rados.rados.0.smithi107.stdout:1472: delete oid 18 current snap is 0 2020-06-12T02:37:38.207 INFO:tasks.rados.rados.0.smithi107.stdout:1472: done (0 left) 2020-06-12T02:37:38.207 INFO:tasks.rados.rados.0.smithi107.stdout:1473: delete oid 9 current snap is 0 2020-06-12T02:37:38.207 INFO:tasks.rados.rados.0.smithi107.stdout:1473: done (0 left) 2020-06-12T02:37:38.207 INFO:tasks.rados.rados.0.smithi107.stdout:1474: delete oid 9 current snap is 0 2020-06-12T02:37:38.207 INFO:tasks.rados.rados.0.smithi107.stdout:1474: done (0 left) 2020-06-12T02:37:38.208 INFO:tasks.rados.rados.0.smithi107.stdout:1475: delete oid 2 current snap is 0 2020-06-12T02:37:38.208 INFO:tasks.rados.rados.0.smithi107.stdout:1475: done (0 left) 2020-06-12T02:37:38.208 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:37:38.208 INFO:tasks.rados.rados.0.smithi107.stdout:1476: seq_num 1475 ranges {0=712704} 2020-06-12T02:37:38.208 INFO:tasks.rados.rados.0.smithi107.stdout:1476: writing smithi10729407-20 from 0 to 712704 tid 1 2020-06-12T02:37:38.208 INFO:tasks.rados.rados.0.smithi107.stdout:1477: delete oid 6 current snap is 0 2020-06-12T02:37:38.209 INFO:tasks.rados.rados.0.smithi107.stdout:1477: done (1 left) 2020-06-12T02:37:38.209 INFO:tasks.rados.rados.0.smithi107.stdout:1478: delete oid 19 current snap is 0 2020-06-12T02:37:38.209 INFO:tasks.rados.rados.0.smithi107.stdout:1478: done (1 left) 2020-06-12T02:37:38.209 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:37:38.209 INFO:tasks.rados.rados.0.smithi107.stdout:1479: seq_num 1478 ranges {0=761856} 2020-06-12T02:37:38.210 INFO:tasks.rados.rados.0.smithi107.stdout:1479: writing smithi10729407-11 from 0 to 761856 tid 1 2020-06-12T02:37:38.210 INFO:tasks.rados.rados.0.smithi107.stdout:1480: delete oid 6 current snap is 0 2020-06-12T02:37:38.210 INFO:tasks.rados.rados.0.smithi107.stdout:1476: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:38.210 INFO:tasks.rados.rados.0.smithi107.stdout:1476: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:38.210 INFO:tasks.rados.rados.0.smithi107.stdout:1476: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:38.210 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 112 (ObjNum 1475 snap 0 seq_num 1475) dirty exists 2020-06-12T02:37:38.211 INFO:tasks.rados.rados.0.smithi107.stdout:1476: left oid 20 (ObjNum 1475 snap 0 seq_num 1475) 2020-06-12T02:37:38.211 INFO:tasks.rados.rados.0.smithi107.stdout:1479: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:38.211 INFO:tasks.rados.rados.0.smithi107.stdout:1479: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:38.211 INFO:tasks.rados.rados.0.smithi107.stdout:1476: done (2 left) 2020-06-12T02:37:38.211 INFO:tasks.rados.rados.0.smithi107.stdout:1480: done (1 left) 2020-06-12T02:37:38.212 INFO:tasks.rados.rados.0.smithi107.stdout:1481: delete oid 19 current snap is 0 2020-06-12T02:37:38.212 INFO:tasks.rados.rados.0.smithi107.stdout:1479: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:38.212 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 253 (ObjNum 1478 snap 0 seq_num 1478) dirty exists 2020-06-12T02:37:38.212 INFO:tasks.rados.rados.0.smithi107.stdout:1479: left oid 11 (ObjNum 1478 snap 0 seq_num 1478) 2020-06-12T02:37:38.212 INFO:tasks.rados.rados.0.smithi107.stdout:1479: done (1 left) 2020-06-12T02:37:38.213 INFO:tasks.rados.rados.0.smithi107.stdout:1481: done (0 left) 2020-06-12T02:37:38.213 INFO:tasks.rados.rados.0.smithi107.stdout:1482: delete oid 14 current snap is 0 2020-06-12T02:37:38.213 INFO:tasks.rados.rados.0.smithi107.stdout:1482: done (0 left) 2020-06-12T02:37:38.213 INFO:tasks.rados.rados.0.smithi107.stdout:1483: delete oid 12 current snap is 0 2020-06-12T02:37:38.214 INFO:tasks.rados.rados.0.smithi107.stdout:1483: done (0 left) 2020-06-12T02:37:38.214 INFO:tasks.rados.rados.0.smithi107.stdout:1484: delete oid 13 current snap is 0 2020-06-12T02:37:38.214 INFO:tasks.rados.rados.0.smithi107.stdout:1484: done (0 left) 2020-06-12T02:37:38.214 INFO:tasks.rados.rados.0.smithi107.stdout:1485: delete oid 14 current snap is 0 2020-06-12T02:37:38.214 INFO:tasks.rados.rados.0.smithi107.stdout:1485: done (0 left) 2020-06-12T02:37:38.214 INFO:tasks.rados.rados.0.smithi107.stdout:1486: delete oid 3 current snap is 0 2020-06-12T02:37:38.215 INFO:tasks.rados.rados.0.smithi107.stdout:1486: done (0 left) 2020-06-12T02:37:38.215 INFO:tasks.rados.rados.0.smithi107.stdout:1487: delete oid 15 current snap is 0 2020-06-12T02:37:38.215 INFO:tasks.rados.rados.0.smithi107.stdout:1487: done (0 left) 2020-06-12T02:37:38.215 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:37:38.215 INFO:tasks.rados.rados.0.smithi107.stdout:1488: seq_num 1487 ranges {0=507904,507904=401408} 2020-06-12T02:37:38.216 INFO:tasks.rados.rados.0.smithi107.stdout:1488: writing smithi10729407-6 from 0 to 507904 tid 1 2020-06-12T02:37:38.216 INFO:tasks.rados.rados.0.smithi107.stdout:1488: writing smithi10729407-6 from 507904 to 909312 tid 2 2020-06-12T02:37:38.216 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:37:38.216 INFO:tasks.rados.rados.0.smithi107.stdout:1489: seq_num 1488 ranges {0=524288,524288=401408} 2020-06-12T02:37:38.216 INFO:tasks.rados.rados.0.smithi107.stdout:1489: writing smithi10729407-15 from 0 to 524288 tid 1 2020-06-12T02:37:38.217 INFO:tasks.rados.rados.0.smithi107.stdout:1489: writing smithi10729407-15 from 524288 to 925696 tid 2 2020-06-12T02:37:38.217 INFO:tasks.rados.rados.0.smithi107.stdout:1490: delete oid 3 current snap is 0 2020-06-12T02:37:38.217 INFO:tasks.rados.rados.0.smithi107.stdout:1488: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:38.218 INFO:tasks.rados.rados.0.smithi107.stdout:1488: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:38.218 INFO:tasks.rados.rados.0.smithi107.stdout:1488: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:38.218 INFO:tasks.rados.rados.0.smithi107.stdout:1490: done (2 left) 2020-06-12T02:37:38.218 INFO:tasks.rados.rados.0.smithi107.stdout:1491: delete oid 1 current snap is 0 2020-06-12T02:37:38.218 INFO:tasks.rados.rados.0.smithi107.stdout:1489: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:38.221 INFO:tasks.rados.rados.0.smithi107.stdout:1489: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:38.221 INFO:tasks.rados.rados.0.smithi107.stdout:1488: finishing write tid 5 to smithi10729407-6 2020-06-12T02:37:38.221 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 296 (ObjNum 1487 snap 0 seq_num 1487) dirty exists 2020-06-12T02:37:38.221 INFO:tasks.rados.rados.0.smithi107.stdout:1488: left oid 6 (ObjNum 1487 snap 0 seq_num 1487) 2020-06-12T02:37:38.221 INFO:tasks.rados.rados.0.smithi107.stdout:1489: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:38.222 INFO:tasks.rados.rados.0.smithi107.stdout:1488: done (2 left) 2020-06-12T02:37:38.222 INFO:tasks.rados.rados.0.smithi107.stdout:1491: done (1 left) 2020-06-12T02:37:38.222 INFO:tasks.rados.rados.0.smithi107.stdout:1492: delete oid 8 current snap is 0 2020-06-12T02:37:38.222 INFO:tasks.rados.rados.0.smithi107.stdout:1489: finishing write tid 5 to smithi10729407-15 2020-06-12T02:37:38.222 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 260 (ObjNum 1488 snap 0 seq_num 1488) dirty exists 2020-06-12T02:37:38.223 INFO:tasks.rados.rados.0.smithi107.stdout:1489: left oid 15 (ObjNum 1488 snap 0 seq_num 1488) 2020-06-12T02:37:38.223 INFO:tasks.rados.rados.0.smithi107.stdout:1489: done (1 left) 2020-06-12T02:37:38.223 INFO:tasks.rados.rados.0.smithi107.stdout:1492: done (0 left) 2020-06-12T02:37:38.223 INFO:tasks.rados.rados.0.smithi107.stdout:1493: delete oid 13 current snap is 0 2020-06-12T02:37:38.223 INFO:tasks.rados.rados.0.smithi107.stdout:1493: done (0 left) 2020-06-12T02:37:38.223 INFO:tasks.rados.rados.0.smithi107.stdout:1494: delete oid 7 current snap is 0 2020-06-12T02:37:38.224 INFO:tasks.rados.rados.0.smithi107.stdout:1494: done (0 left) 2020-06-12T02:37:38.224 INFO:tasks.rados.rados.0.smithi107.stdout:1495: delete oid 7 current snap is 0 2020-06-12T02:37:38.224 INFO:tasks.rados.rados.0.smithi107.stdout:1495: done (0 left) 2020-06-12T02:37:38.224 INFO:tasks.rados.rados.0.smithi107.stdout:1496: delete oid 11 current snap is 0 2020-06-12T02:37:38.224 INFO:tasks.rados.rados.0.smithi107.stdout:1496: done (0 left) 2020-06-12T02:37:38.224 INFO:tasks.rados.rados.0.smithi107.stdout:1497: delete oid 3 current snap is 0 2020-06-12T02:37:38.225 INFO:tasks.rados.rados.0.smithi107.stdout:1497: done (0 left) 2020-06-12T02:37:38.225 INFO:tasks.rados.rados.0.smithi107.stdout:1498: delete oid 6 current snap is 0 2020-06-12T02:37:38.225 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:38.217+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:38.226 INFO:tasks.rados.rados.0.smithi107.stdout:1498: done (0 left) 2020-06-12T02:37:38.226 INFO:tasks.rados.rados.0.smithi107.stdout:1499: delete oid 16 current snap is 0 2020-06-12T02:37:38.227 INFO:tasks.rados.rados.0.smithi107.stdout:1499: done (0 left) 2020-06-12T02:37:38.227 INFO:tasks.rados.rados.0.smithi107.stdout:1500: delete oid 13 current snap is 0 2020-06-12T02:37:38.227 INFO:tasks.rados.rados.0.smithi107.stdout:1500: done (0 left) 2020-06-12T02:37:38.228 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:37:38.228 INFO:tasks.rados.rados.0.smithi107.stdout:1501: seq_num 1500 ranges {0=729088,729088=401408} 2020-06-12T02:37:38.231 INFO:tasks.rados.rados.0.smithi107.stdout:1501: writing smithi10729407-1 from 0 to 729088 tid 1 2020-06-12T02:37:38.234 INFO:tasks.rados.rados.0.smithi107.stdout:1501: writing smithi10729407-1 from 729088 to 1130496 tid 2 2020-06-12T02:37:38.234 INFO:tasks.rados.rados.0.smithi107.stdout:1502: delete oid 10 current snap is 0 2020-06-12T02:37:38.235 INFO:tasks.rados.rados.0.smithi107.stdout:1502: done (1 left) 2020-06-12T02:37:38.235 INFO:tasks.rados.rados.0.smithi107.stdout:1503: delete oid 7 current snap is 0 2020-06-12T02:37:38.237 INFO:tasks.rados.rados.0.smithi107.stdout:1503: done (1 left) 2020-06-12T02:37:38.238 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:37:38.238 INFO:tasks.rados.rados.0.smithi107.stdout:1504: seq_num 1503 ranges {712704=778240,1490944=401408} 2020-06-12T02:37:38.246 INFO:tasks.rados.rados.0.smithi107.stdout:1504: writing smithi10729407-20 from 712704 to 1490944 tid 1 2020-06-12T02:37:38.249 INFO:tasks.rados.rados.0.smithi107.stdout:1504: writing smithi10729407-20 from 1490944 to 1892352 tid 2 2020-06-12T02:37:38.249 INFO:tasks.rados.rados.0.smithi107.stdout:1505: delete oid 10 current snap is 0 2020-06-12T02:37:38.249 INFO:tasks.rados.rados.0.smithi107.stdout:1501: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:38.250 INFO:tasks.rados.rados.0.smithi107.stdout:1501: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:38.250 INFO:tasks.rados.rados.0.smithi107.stdout:1501: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:38.250 INFO:tasks.rados.rados.0.smithi107.stdout:1501: finishing write tid 5 to smithi10729407-1 2020-06-12T02:37:38.250 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 301 (ObjNum 1500 snap 0 seq_num 1500) dirty exists 2020-06-12T02:37:38.250 INFO:tasks.rados.rados.0.smithi107.stdout:1501: left oid 1 (ObjNum 1500 snap 0 seq_num 1500) 2020-06-12T02:37:38.251 INFO:tasks.rados.rados.0.smithi107.stdout:1501: done (2 left) 2020-06-12T02:37:38.251 INFO:tasks.rados.rados.0.smithi107.stdout:1505: done (1 left) 2020-06-12T02:37:38.251 INFO:tasks.rados.rados.0.smithi107.stdout:1506: delete oid 2 current snap is 0 2020-06-12T02:37:38.252 INFO:tasks.rados.rados.0.smithi107.stdout:1506: done (1 left) 2020-06-12T02:37:38.252 INFO:tasks.rados.rados.0.smithi107.stdout:1507: delete oid 18 current snap is 0 2020-06-12T02:37:38.253 INFO:tasks.rados.rados.0.smithi107.stdout:1504: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:38.253 INFO:tasks.rados.rados.0.smithi107.stdout:1507: done (1 left) 2020-06-12T02:37:38.254 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:37:38.254 INFO:tasks.rados.rados.0.smithi107.stdout:1508: seq_num 1507 ranges {0=442368,442368=729088,1171456=73728} 2020-06-12T02:37:38.256 INFO:tasks.rados.rados.0.smithi107.stdout:1508: writing smithi10729407-4 from 0 to 442368 tid 1 2020-06-12T02:37:38.261 INFO:tasks.rados.rados.0.smithi107.stdout:1508: writing smithi10729407-4 from 442368 to 1171456 tid 2 2020-06-12T02:37:38.262 INFO:tasks.rados.rados.0.smithi107.stdout:1508: writing smithi10729407-4 from 1171456 to 1245184 tid 3 2020-06-12T02:37:38.262 INFO:tasks.rados.rados.0.smithi107.stdout:1509: delete oid 17 current snap is 0 2020-06-12T02:37:38.262 INFO:tasks.rados.rados.0.smithi107.stdout:1504: finishing write tid 2 to smithi10729407-20 2020-06-12T02:37:38.262 INFO:tasks.rados.rados.0.smithi107.stdout:1504: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:38.263 INFO:tasks.rados.rados.0.smithi107.stdout:1504: finishing write tid 5 to smithi10729407-20 2020-06-12T02:37:38.263 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 115 (ObjNum 1503 snap 0 seq_num 1503) dirty exists 2020-06-12T02:37:38.263 INFO:tasks.rados.rados.0.smithi107.stdout:1504: left oid 20 (ObjNum 1503 snap 0 seq_num 1503) 2020-06-12T02:37:38.263 INFO:tasks.rados.rados.0.smithi107.stdout:1508: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:38.267 INFO:tasks.rados.rados.0.smithi107.stdout:1508: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:38.267 INFO:tasks.rados.rados.0.smithi107.stdout:1508: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:38.267 INFO:tasks.rados.rados.0.smithi107.stdout:1508: finishing write tid 5 to smithi10729407-4 2020-06-12T02:37:38.268 INFO:tasks.rados.rados.0.smithi107.stdout:1504: done (2 left) 2020-06-12T02:37:38.268 INFO:tasks.rados.rados.0.smithi107.stdout:1509: done (1 left) 2020-06-12T02:37:38.268 INFO:tasks.rados.rados.0.smithi107.stdout:1510: delete oid 15 current snap is 0 2020-06-12T02:37:38.269 INFO:tasks.rados.rados.0.smithi107.stdout:1508: finishing write tid 6 to smithi10729407-4 2020-06-12T02:37:38.269 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 289 (ObjNum 1507 snap 0 seq_num 1507) dirty exists 2020-06-12T02:37:38.269 INFO:tasks.rados.rados.0.smithi107.stdout:1508: left oid 4 (ObjNum 1507 snap 0 seq_num 1507) 2020-06-12T02:37:38.270 INFO:tasks.rados.rados.0.smithi107.stdout:1508: done (1 left) 2020-06-12T02:37:38.270 INFO:tasks.rados.rados.0.smithi107.stdout:1510: done (0 left) 2020-06-12T02:37:38.270 INFO:tasks.rados.rados.0.smithi107.stdout:1511: delete oid 5 current snap is 0 2020-06-12T02:37:38.271 INFO:tasks.rados.rados.0.smithi107.stdout:1511: done (0 left) 2020-06-12T02:37:38.272 INFO:tasks.rados.rados.0.smithi107.stdout:1512: delete oid 14 current snap is 0 2020-06-12T02:37:38.273 INFO:tasks.rados.rados.0.smithi107.stdout:1512: done (0 left) 2020-06-12T02:37:38.274 INFO:tasks.rados.rados.0.smithi107.stdout:1513: delete oid 20 current snap is 0 2020-06-12T02:37:38.284 INFO:tasks.rados.rados.0.smithi107.stdout:1513: done (0 left) 2020-06-12T02:37:38.284 INFO:tasks.rados.rados.0.smithi107.stdout:1514: delete oid 13 current snap is 0 2020-06-12T02:37:38.285 INFO:tasks.rados.rados.0.smithi107.stdout:1514: done (0 left) 2020-06-12T02:37:38.285 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:37:38.285 INFO:tasks.rados.rados.0.smithi107.stdout:1515: seq_num 1514 ranges {1245184=565248,1810432=770048,2580480=32768} 2020-06-12T02:37:38.288 INFO:tasks.rados.rados.0.smithi107.stdout:1515: writing smithi10729407-4 from 1245184 to 1810432 tid 1 2020-06-12T02:37:38.292 INFO:tasks.rados.rados.0.smithi107.stdout:1515: writing smithi10729407-4 from 1810432 to 2580480 tid 2 2020-06-12T02:37:38.293 INFO:tasks.rados.rados.0.smithi107.stdout:1515: writing smithi10729407-4 from 2580480 to 2613248 tid 3 2020-06-12T02:37:38.293 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:37:38.293 INFO:tasks.rados.rados.0.smithi107.stdout:1516: seq_num 1515 ranges {0=581632,581632=655360,1236992=147456} 2020-06-12T02:37:38.296 INFO:tasks.rados.rados.0.smithi107.stdout:1516: writing smithi10729407-8 from 0 to 581632 tid 1 2020-06-12T02:37:38.302 INFO:tasks.rados.rados.0.smithi107.stdout:1516: writing smithi10729407-8 from 581632 to 1236992 tid 2 2020-06-12T02:37:38.303 INFO:tasks.rados.rados.0.smithi107.stdout:1516: writing smithi10729407-8 from 1236992 to 1384448 tid 3 2020-06-12T02:37:38.303 INFO:tasks.rados.rados.0.smithi107.stdout:1517: delete oid 15 current snap is 0 2020-06-12T02:37:38.303 INFO:tasks.rados.rados.0.smithi107.stdout:1515: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:38.303 INFO:tasks.rados.rados.0.smithi107.stdout:1515: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:38.303 INFO:tasks.rados.rados.0.smithi107.stdout:1515: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:38.304 INFO:tasks.rados.rados.0.smithi107.stdout:1515: finishing write tid 5 to smithi10729407-4 2020-06-12T02:37:38.304 INFO:tasks.rados.rados.0.smithi107.stdout:1516: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:38.305 INFO:tasks.rados.rados.0.smithi107.stdout:1515: finishing write tid 6 to smithi10729407-4 2020-06-12T02:37:38.305 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 294 (ObjNum 1514 snap 0 seq_num 1514) dirty exists 2020-06-12T02:37:38.305 INFO:tasks.rados.rados.0.smithi107.stdout:1515: left oid 4 (ObjNum 1514 snap 0 seq_num 1514) 2020-06-12T02:37:38.310 INFO:tasks.rados.rados.0.smithi107.stdout:1515: done (2 left) 2020-06-12T02:37:38.310 INFO:tasks.rados.rados.0.smithi107.stdout:1517: done (1 left) 2020-06-12T02:37:38.310 INFO:tasks.rados.rados.0.smithi107.stdout:1518: delete oid 13 current snap is 0 2020-06-12T02:37:38.311 INFO:tasks.rados.rados.0.smithi107.stdout:1516: finishing write tid 2 to smithi10729407-8 2020-06-12T02:37:38.311 INFO:tasks.rados.rados.0.smithi107.stdout:1518: done (1 left) 2020-06-12T02:37:38.312 INFO:tasks.rados.rados.0.smithi107.stdout:1519: delete oid 4 current snap is 0 2020-06-12T02:37:38.314 INFO:tasks.rados.rados.0.smithi107.stdout:1516: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:38.315 INFO:tasks.rados.rados.0.smithi107.stdout:1516: finishing write tid 5 to smithi10729407-8 2020-06-12T02:37:38.315 INFO:tasks.rados.rados.0.smithi107.stdout:1519: done (1 left) 2020-06-12T02:37:38.315 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:37:38.315 INFO:tasks.rados.rados.0.smithi107.stdout:1520: seq_num 1519 ranges {1130496=647168,1777664=589824,2367488=212992} 2020-06-12T02:37:38.323 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:37:38.320+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:37:38.324 INFO:tasks.rados.rados.0.smithi107.stdout:1520: writing smithi10729407-1 from 1130496 to 1777664 tid 1 2020-06-12T02:37:38.328 INFO:tasks.rados.rados.0.smithi107.stdout:1520: writing smithi10729407-1 from 1777664 to 2367488 tid 2 2020-06-12T02:37:38.329 INFO:tasks.rados.rados.0.smithi107.stdout:1520: writing smithi10729407-1 from 2367488 to 2580480 tid 3 2020-06-12T02:37:38.329 INFO:tasks.rados.rados.0.smithi107.stdout:1521: delete oid 18 current snap is 0 2020-06-12T02:37:38.330 INFO:tasks.rados.rados.0.smithi107.stdout:1516: finishing write tid 6 to smithi10729407-8 2020-06-12T02:37:38.330 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 267 (ObjNum 1515 snap 0 seq_num 1515) dirty exists 2020-06-12T02:37:38.330 INFO:tasks.rados.rados.0.smithi107.stdout:1516: left oid 8 (ObjNum 1515 snap 0 seq_num 1515) 2020-06-12T02:37:38.330 INFO:tasks.rados.rados.0.smithi107.stdout:1516: done (2 left) 2020-06-12T02:37:38.331 INFO:tasks.rados.rados.0.smithi107.stdout:1521: done (1 left) 2020-06-12T02:37:38.331 INFO:tasks.rados.rados.0.smithi107.stdout:1522: delete oid 4 current snap is 0 2020-06-12T02:37:38.332 INFO:tasks.rados.rados.0.smithi107.stdout:1522: done (1 left) 2020-06-12T02:37:38.333 INFO:tasks.rados.rados.0.smithi107.stdout:1523: delete oid 12 current snap is 0 2020-06-12T02:37:38.333 INFO:tasks.rados.rados.0.smithi107.stdout:1520: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:38.334 INFO:tasks.rados.rados.0.smithi107.stdout:1520: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:38.335 INFO:tasks.rados.rados.0.smithi107.stdout:1520: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:38.336 INFO:tasks.rados.rados.0.smithi107.stdout:1523: done (1 left) 2020-06-12T02:37:38.336 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:37:38.336 INFO:tasks.rados.rados.0.smithi107.stdout:1524: seq_num 1523 ranges {0=712704,712704=581632,1294336=221184} 2020-06-12T02:37:38.339 INFO:tasks.rados.rados.0.smithi107.stdout:1524: writing smithi10729407-3 from 0 to 712704 tid 1 2020-06-12T02:37:38.343 INFO:tasks.rados.rados.0.smithi107.stdout:1524: writing smithi10729407-3 from 712704 to 1294336 tid 2 2020-06-12T02:37:38.345 INFO:tasks.rados.rados.0.smithi107.stdout:1524: writing smithi10729407-3 from 1294336 to 1515520 tid 3 2020-06-12T02:37:38.345 INFO:tasks.rados.rados.0.smithi107.stdout:1520: finishing write tid 5 to smithi10729407-1 2020-06-12T02:37:38.345 INFO:tasks.rados.rados.0.smithi107.stdout:1525: delete oid 10 current snap is 0 2020-06-12T02:37:38.345 INFO:tasks.rados.rados.0.smithi107.stdout:1520: finishing write tid 6 to smithi10729407-1 2020-06-12T02:37:38.345 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 305 (ObjNum 1519 snap 0 seq_num 1519) dirty exists 2020-06-12T02:37:38.346 INFO:tasks.rados.rados.0.smithi107.stdout:1520: left oid 1 (ObjNum 1519 snap 0 seq_num 1519) 2020-06-12T02:37:38.346 INFO:tasks.rados.rados.0.smithi107.stdout:1524: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:38.347 INFO:tasks.rados.rados.0.smithi107.stdout:1520: done (2 left) 2020-06-12T02:37:38.347 INFO:tasks.rados.rados.0.smithi107.stdout:1525: done (1 left) 2020-06-12T02:37:38.347 INFO:tasks.rados.rados.0.smithi107.stdout:1526: delete oid 13 current snap is 0 2020-06-12T02:37:38.348 INFO:tasks.rados.rados.0.smithi107.stdout:1526: done (1 left) 2020-06-12T02:37:38.349 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:37:38.349 INFO:tasks.rados.rados.0.smithi107.stdout:1527: seq_num 1526 ranges {0=761856,761856=630784,1392640=172032} 2020-06-12T02:37:38.353 INFO:tasks.rados.rados.0.smithi107.stdout:1527: writing smithi10729407-14 from 0 to 761856 tid 1 2020-06-12T02:37:38.357 INFO:tasks.rados.rados.0.smithi107.stdout:1527: writing smithi10729407-14 from 761856 to 1392640 tid 2 2020-06-12T02:37:38.359 INFO:tasks.rados.rados.0.smithi107.stdout:1527: writing smithi10729407-14 from 1392640 to 1564672 tid 3 2020-06-12T02:37:38.359 INFO:tasks.rados.rados.0.smithi107.stdout:1528: delete oid 16 current snap is 0 2020-06-12T02:37:38.359 INFO:tasks.rados.rados.0.smithi107.stdout:1524: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:38.359 INFO:tasks.rados.rados.0.smithi107.stdout:1524: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:38.360 INFO:tasks.rados.rados.0.smithi107.stdout:1524: finishing write tid 5 to smithi10729407-3 2020-06-12T02:37:38.360 INFO:tasks.rados.rados.0.smithi107.stdout:1524: finishing write tid 6 to smithi10729407-3 2020-06-12T02:37:38.361 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 261 (ObjNum 1523 snap 0 seq_num 1523) dirty exists 2020-06-12T02:37:38.361 INFO:tasks.rados.rados.0.smithi107.stdout:1524: left oid 3 (ObjNum 1523 snap 0 seq_num 1523) 2020-06-12T02:37:38.362 INFO:tasks.rados.rados.0.smithi107.stdout:1524: done (2 left) 2020-06-12T02:37:38.362 INFO:tasks.rados.rados.0.smithi107.stdout:1528: done (1 left) 2020-06-12T02:37:38.363 INFO:tasks.rados.rados.0.smithi107.stdout:1529: delete oid 5 current snap is 0 2020-06-12T02:37:38.363 INFO:tasks.rados.rados.0.smithi107.stdout:1527: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:38.363 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:37:38.364 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:37:38.364 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.364 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:3244 3.es1 3:78f01648:::smithi10729407-5:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e27)", 2020-06-12T02:37:38.364 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:38.360867+0000", 2020-06-12T02:37:38.364 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.00034905899999999998, 2020-06-12T02:37:38.365 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00036887200000000001, 2020-06-12T02:37:38.365 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:38.365 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:38.365 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:38.365 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:38.366 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:38.366 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 3244 2020-06-12T02:37:38.366 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.366 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:38.366 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.367 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:38.367 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.360867+0000", 2020-06-12T02:37:38.367 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:38.367 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.367 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.368 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:38.368 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.360867+0000", 2020-06-12T02:37:38.368 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4620000000000003e-05 2020-06-12T02:37:38.368 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.368 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.369 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:38.369 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.360911+0000", 2020-06-12T02:37:38.370 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2930000000000003e-06 2020-06-12T02:37:38.370 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.370 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.370 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:38.371 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.360918+0000", 2020-06-12T02:37:38.371 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6626e-05 2020-06-12T02:37:38.371 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.371 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.371 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:38.372 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.360935+0000", 2020-06-12T02:37:38.372 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5650000000000001e-06 2020-06-12T02:37:38.372 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.372 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.372 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:38.373 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.360941+0000", 2020-06-12T02:37:38.373 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1841000000000001e-05 2020-06-12T02:37:38.373 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.373 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.373 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:38.373 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.360992+0000", 2020-06-12T02:37:38.374 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7527000000000006e-05 2020-06-12T02:37:38.374 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.374 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.374 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:38.374 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.361060+0000", 2020-06-12T02:37:38.375 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00019347200000000001 2020-06-12T02:37:38.375 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:38.375 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:38.375 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:38.375 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:38.375 INFO:teuthology.orchestra.run.smithi188.stdout: ], 2020-06-12T02:37:38.376 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 1 2020-06-12T02:37:38.376 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:37:38.377 INFO:tasks.rados.rados.0.smithi107.stdout:1529: done (1 left) 2020-06-12T02:37:38.378 INFO:tasks.rados.rados.0.smithi107.stdout:1530: delete oid 10 current snap is 0 2020-06-12T02:37:38.378 INFO:tasks.rados.rados.0.smithi107.stdout:1530: done (1 left) 2020-06-12T02:37:38.378 INFO:tasks.rados.rados.0.smithi107.stdout:1531: delete oid 10 current snap is 0 2020-06-12T02:37:38.378 INFO:tasks.rados.rados.0.smithi107.stdout:1531: done (1 left) 2020-06-12T02:37:38.379 INFO:tasks.rados.rados.0.smithi107.stdout:1532: delete oid 13 current snap is 0 2020-06-12T02:37:38.379 INFO:tasks.rados.rados.0.smithi107.stdout:1527: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:38.379 INFO:tasks.rados.rados.0.smithi107.stdout:1527: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:38.379 INFO:tasks.rados.rados.0.smithi107.stdout:1527: finishing write tid 5 to smithi10729407-14 2020-06-12T02:37:38.379 INFO:tasks.rados.rados.0.smithi107.stdout:1532: done (1 left) 2020-06-12T02:37:38.379 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:37:38.380 INFO:tasks.rados.rados.0.smithi107.stdout:1533: seq_num 1532 ranges {0=466944,466944=786432,1253376=417792} 2020-06-12T02:37:38.380 INFO:tasks.rados.rados.0.smithi107.stdout:1533: writing smithi10729407-6 from 0 to 466944 tid 1 2020-06-12T02:37:38.380 INFO:tasks.rados.rados.0.smithi107.stdout:1533: writing smithi10729407-6 from 466944 to 1253376 tid 2 2020-06-12T02:37:38.381 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-06-12T02:37:38.382 INFO:tasks.rados.rados.0.smithi107.stdout:1533: writing smithi10729407-6 from 1253376 to 1671168 tid 3 2020-06-12T02:37:38.382 INFO:tasks.rados.rados.0.smithi107.stdout:1534: delete oid 17 current snap is 0 2020-06-12T02:37:38.382 INFO:tasks.rados.rados.0.smithi107.stdout:1527: finishing write tid 6 to smithi10729407-14 2020-06-12T02:37:38.383 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 261 (ObjNum 1526 snap 0 seq_num 1526) dirty exists 2020-06-12T02:37:38.383 INFO:tasks.rados.rados.0.smithi107.stdout:1527: left oid 14 (ObjNum 1526 snap 0 seq_num 1526) 2020-06-12T02:37:38.383 INFO:tasks.rados.rados.0.smithi107.stdout:1533: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:38.383 INFO:tasks.rados.rados.0.smithi107.stdout:1527: done (2 left) 2020-06-12T02:37:38.383 INFO:tasks.rados.rados.0.smithi107.stdout:1534: done (1 left) 2020-06-12T02:37:38.383 INFO:tasks.rados.rados.0.smithi107.stdout:1535: delete oid 19 current snap is 0 2020-06-12T02:37:38.384 INFO:tasks.rados.rados.0.smithi107.stdout:1535: done (1 left) 2020-06-12T02:37:38.385 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:37:38.385 INFO:tasks.rados.rados.0.smithi107.stdout:1536: seq_num 1535 ranges {0=516096,516096=442368,958464=688128,1646592=73728} 2020-06-12T02:37:38.386 INFO:tasks.rados.rados.0.smithi107.stdout:1536: writing smithi10729407-7 from 0 to 516096 tid 1 2020-06-12T02:37:38.389 INFO:tasks.rados.rados.0.smithi107.stdout:1536: writing smithi10729407-7 from 516096 to 958464 tid 2 2020-06-12T02:37:38.393 INFO:tasks.rados.rados.0.smithi107.stdout:1536: writing smithi10729407-7 from 958464 to 1646592 tid 3 2020-06-12T02:37:38.394 INFO:tasks.rados.rados.0.smithi107.stdout:1536: writing smithi10729407-7 from 1646592 to 1720320 tid 4 2020-06-12T02:37:38.394 INFO:tasks.rados.rados.0.smithi107.stdout:1537: delete oid 16 current snap is 0 2020-06-12T02:37:38.394 INFO:tasks.rados.rados.0.smithi107.stdout:1533: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:38.395 INFO:tasks.rados.rados.0.smithi107.stdout:1533: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:38.395 INFO:tasks.rados.rados.0.smithi107.stdout:1533: finishing write tid 5 to smithi10729407-6 2020-06-12T02:37:38.395 INFO:tasks.rados.rados.0.smithi107.stdout:1533: finishing write tid 6 to smithi10729407-6 2020-06-12T02:37:38.395 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 309 (ObjNum 1532 snap 0 seq_num 1532) dirty exists 2020-06-12T02:37:38.395 INFO:tasks.rados.rados.0.smithi107.stdout:1533: left oid 6 (ObjNum 1532 snap 0 seq_num 1532) 2020-06-12T02:37:38.395 INFO:tasks.rados.rados.0.smithi107.stdout:1536: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:38.397 INFO:tasks.rados.rados.0.smithi107.stdout:1536: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:38.400 INFO:tasks.rados.rados.0.smithi107.stdout:1536: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:38.400 INFO:tasks.rados.rados.0.smithi107.stdout:1536: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:38.400 INFO:tasks.rados.rados.0.smithi107.stdout:1533: done (2 left) 2020-06-12T02:37:38.401 INFO:tasks.rados.rados.0.smithi107.stdout:1537: done (1 left) 2020-06-12T02:37:38.401 INFO:tasks.rados.rados.0.smithi107.stdout:1538: delete oid 18 current snap is 0 2020-06-12T02:37:38.401 INFO:tasks.rados.rados.0.smithi107.stdout:1536: finishing write tid 6 to smithi10729407-7 2020-06-12T02:37:38.402 INFO:tasks.rados.rados.0.smithi107.stdout:1536: finishing write tid 7 to smithi10729407-7 2020-06-12T02:37:38.402 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 313 (ObjNum 1535 snap 0 seq_num 1535) dirty exists 2020-06-12T02:37:38.402 INFO:tasks.rados.rados.0.smithi107.stdout:1536: left oid 7 (ObjNum 1535 snap 0 seq_num 1535) 2020-06-12T02:37:38.403 INFO:tasks.rados.rados.0.smithi107.stdout:1536: done (1 left) 2020-06-12T02:37:38.403 INFO:tasks.rados.rados.0.smithi107.stdout:1538: done (0 left) 2020-06-12T02:37:38.403 INFO:tasks.rados.rados.0.smithi107.stdout:1539: delete oid 9 current snap is 0 2020-06-12T02:37:38.405 INFO:tasks.rados.rados.0.smithi107.stdout:1539: done (0 left) 2020-06-12T02:37:38.405 INFO:tasks.rados.rados.0.smithi107.stdout:1540: delete oid 5 current snap is 0 2020-06-12T02:37:38.406 INFO:tasks.rados.rados.0.smithi107.stdout:1540: done (0 left) 2020-06-12T02:37:38.407 INFO:tasks.rados.rados.0.smithi107.stdout:1541: delete oid 14 current snap is 0 2020-06-12T02:37:38.409 INFO:tasks.rados.rados.0.smithi107.stdout:1541: done (0 left) 2020-06-12T02:37:38.409 INFO:tasks.rados.rados.0.smithi107.stdout:1542: delete oid 2 current snap is 0 2020-06-12T02:37:38.410 INFO:tasks.rados.rados.0.smithi107.stdout:1542: done (0 left) 2020-06-12T02:37:38.410 INFO:tasks.rados.rados.0.smithi107.stdout:1543: delete oid 10 current snap is 0 2020-06-12T02:37:38.414 INFO:tasks.rados.rados.0.smithi107.stdout:1543: done (0 left) 2020-06-12T02:37:38.414 INFO:tasks.rados.rados.0.smithi107.stdout:1544: delete oid 2 current snap is 0 2020-06-12T02:37:38.416 INFO:tasks.rados.rados.0.smithi107.stdout:1544: done (0 left) 2020-06-12T02:37:38.416 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:37:38.417 INFO:tasks.rados.rados.0.smithi107.stdout:1545: seq_num 1544 ranges {1720320=663552,2383872=696320,3080192=458752,3538944=49152} 2020-06-12T02:37:38.427 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:37:38.424+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:37:38.428 INFO:tasks.rados.rados.0.smithi107.stdout:1545: writing smithi10729407-7 from 1720320 to 2383872 tid 1 2020-06-12T02:37:38.433 INFO:tasks.rados.rados.0.smithi107.stdout:1545: writing smithi10729407-7 from 2383872 to 3080192 tid 2 2020-06-12T02:37:38.436 INFO:tasks.rados.rados.0.smithi107.stdout:1545: writing smithi10729407-7 from 3080192 to 3538944 tid 3 2020-06-12T02:37:38.436 INFO:tasks.rados.rados.0.smithi107.stdout:1545: writing smithi10729407-7 from 3538944 to 3588096 tid 4 2020-06-12T02:37:38.437 INFO:tasks.rados.rados.0.smithi107.stdout:1546: delete oid 14 current snap is 0 2020-06-12T02:37:38.437 INFO:tasks.rados.rados.0.smithi107.stdout:1545: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:38.438 INFO:tasks.rados.rados.0.smithi107.stdout:1546: done (1 left) 2020-06-12T02:37:38.439 INFO:tasks.rados.rados.0.smithi107.stdout:1547: delete oid 14 current snap is 0 2020-06-12T02:37:38.443 INFO:tasks.rados.rados.0.smithi107.stdout:1547: done (1 left) 2020-06-12T02:37:38.443 INFO:tasks.rados.rados.0.smithi107.stdout:1548: delete oid 12 current snap is 0 2020-06-12T02:37:38.444 INFO:tasks.rados.rados.0.smithi107.stdout:1545: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:38.444 INFO:tasks.rados.rados.0.smithi107.stdout:1545: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:38.445 INFO:tasks.rados.rados.0.smithi107.stdout:1545: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:38.445 INFO:tasks.rados.rados.0.smithi107.stdout:1548: done (1 left) 2020-06-12T02:37:38.446 INFO:tasks.rados.rados.0.smithi107.stdout:1549: delete oid 12 current snap is 0 2020-06-12T02:37:38.446 INFO:tasks.rados.rados.0.smithi107.stdout:1545: finishing write tid 6 to smithi10729407-7 2020-06-12T02:37:38.446 INFO:tasks.rados.rados.0.smithi107.stdout:1545: finishing write tid 7 to smithi10729407-7 2020-06-12T02:37:38.447 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 318 (ObjNum 1544 snap 0 seq_num 1544) dirty exists 2020-06-12T02:37:38.447 INFO:tasks.rados.rados.0.smithi107.stdout:1545: left oid 7 (ObjNum 1544 snap 0 seq_num 1544) 2020-06-12T02:37:38.450 INFO:tasks.rados.rados.0.smithi107.stdout:1545: done (1 left) 2020-06-12T02:37:38.451 INFO:tasks.rados.rados.0.smithi107.stdout:1549: done (0 left) 2020-06-12T02:37:38.451 INFO:tasks.rados.rados.0.smithi107.stdout:1550: delete oid 2 current snap is 0 2020-06-12T02:37:38.452 INFO:tasks.rados.rados.0.smithi107.stdout:1550: done (0 left) 2020-06-12T02:37:38.452 INFO:tasks.rados.rados.0.smithi107.stdout:1551: delete oid 16 current snap is 0 2020-06-12T02:37:38.454 INFO:tasks.rados.rados.0.smithi107.stdout:1551: done (0 left) 2020-06-12T02:37:38.454 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:37:38.454 INFO:tasks.rados.rados.0.smithi107.stdout:1552: seq_num 1551 ranges {1515520=786432,2301952=737280,3039232=466944} 2020-06-12T02:37:38.466 INFO:tasks.rados.rados.0.smithi107.stdout:1552: writing smithi10729407-3 from 1515520 to 2301952 tid 1 2020-06-12T02:37:38.468 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd set nodeep-scrub 2020-06-12T02:37:38.470 INFO:tasks.rados.rados.0.smithi107.stdout:1552: writing smithi10729407-3 from 2301952 to 3039232 tid 2 2020-06-12T02:37:38.473 INFO:tasks.rados.rados.0.smithi107.stdout:1552: writing smithi10729407-3 from 3039232 to 3506176 tid 3 2020-06-12T02:37:38.474 INFO:tasks.rados.rados.0.smithi107.stdout:1553: delete oid 11 current snap is 0 2020-06-12T02:37:38.475 INFO:tasks.rados.rados.0.smithi107.stdout:1552: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:38.476 INFO:tasks.rados.rados.0.smithi107.stdout:1553: done (1 left) 2020-06-12T02:37:38.476 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:37:38.476 INFO:tasks.rados.rados.0.smithi107.stdout:1554: seq_num 1553 ranges {0=417792,417792=507904,925696=622592,1548288=475136} 2020-06-12T02:37:38.478 INFO:tasks.rados.rados.0.smithi107.stdout:1554: writing smithi10729407-4 from 0 to 417792 tid 1 2020-06-12T02:37:38.482 INFO:tasks.rados.rados.0.smithi107.stdout:1554: writing smithi10729407-4 from 417792 to 925696 tid 2 2020-06-12T02:37:38.485 INFO:tasks.rados.rados.0.smithi107.stdout:1554: writing smithi10729407-4 from 925696 to 1548288 tid 3 2020-06-12T02:37:38.488 INFO:tasks.rados.rados.0.smithi107.stdout:1554: writing smithi10729407-4 from 1548288 to 2023424 tid 4 2020-06-12T02:37:38.489 INFO:tasks.rados.rados.0.smithi107.stdout:1555: delete oid 6 current snap is 0 2020-06-12T02:37:38.489 INFO:tasks.rados.rados.0.smithi107.stdout:1552: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:38.489 INFO:tasks.rados.rados.0.smithi107.stdout:1552: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:38.489 INFO:tasks.rados.rados.0.smithi107.stdout:1552: finishing write tid 5 to smithi10729407-3 2020-06-12T02:37:38.489 INFO:tasks.rados.rados.0.smithi107.stdout:1552: finishing write tid 6 to smithi10729407-3 2020-06-12T02:37:38.490 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 268 (ObjNum 1551 snap 0 seq_num 1551) dirty exists 2020-06-12T02:37:38.490 INFO:tasks.rados.rados.0.smithi107.stdout:1552: left oid 3 (ObjNum 1551 snap 0 seq_num 1551) 2020-06-12T02:37:38.490 INFO:tasks.rados.rados.0.smithi107.stdout:1554: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:38.490 INFO:tasks.rados.rados.0.smithi107.stdout:1552: done (2 left) 2020-06-12T02:37:38.491 INFO:tasks.rados.rados.0.smithi107.stdout:1555: done (1 left) 2020-06-12T02:37:38.491 INFO:tasks.rados.rados.0.smithi107.stdout:1556: delete oid 16 current snap is 0 2020-06-12T02:37:38.492 INFO:tasks.rados.rados.0.smithi107.stdout:1556: done (1 left) 2020-06-12T02:37:38.492 INFO:tasks.rados.rados.0.smithi107.stdout:1557: delete oid 13 current snap is 0 2020-06-12T02:37:38.494 INFO:tasks.rados.rados.0.smithi107.stdout:1554: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:38.496 INFO:tasks.rados.rados.0.smithi107.stdout:1557: done (1 left) 2020-06-12T02:37:38.497 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:37:38.497 INFO:tasks.rados.rados.0.smithi107.stdout:1558: seq_num 1557 ranges {0=483328,483328=442368,925696=606208,1531904=557056} 2020-06-12T02:37:38.499 INFO:tasks.rados.rados.0.smithi107.stdout:1558: writing smithi10729407-14 from 0 to 483328 tid 1 2020-06-12T02:37:38.513 INFO:tasks.rados.rados.0.smithi107.stdout:1558: writing smithi10729407-14 from 483328 to 925696 tid 2 2020-06-12T02:37:38.513 INFO:tasks.rados.rados.0.smithi107.stdout:1558: writing smithi10729407-14 from 925696 to 1531904 tid 3 2020-06-12T02:37:38.514 INFO:tasks.rados.rados.0.smithi107.stdout:1558: writing smithi10729407-14 from 1531904 to 2088960 tid 4 2020-06-12T02:37:38.514 INFO:tasks.rados.rados.0.smithi107.stdout:1559: delete oid 5 current snap is 0 2020-06-12T02:37:38.514 INFO:tasks.rados.rados.0.smithi107.stdout:1554: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:38.514 INFO:tasks.rados.rados.0.smithi107.stdout:1554: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:38.515 INFO:tasks.rados.rados.0.smithi107.stdout:1554: finishing write tid 6 to smithi10729407-4 2020-06-12T02:37:38.515 INFO:tasks.rados.rados.0.smithi107.stdout:1558: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:38.515 INFO:tasks.rados.rados.0.smithi107.stdout:1554: finishing write tid 7 to smithi10729407-4 2020-06-12T02:37:38.515 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 302 (ObjNum 1553 snap 0 seq_num 1553) dirty exists 2020-06-12T02:37:38.515 INFO:tasks.rados.rados.0.smithi107.stdout:1554: left oid 4 (ObjNum 1553 snap 0 seq_num 1553) 2020-06-12T02:37:38.516 INFO:tasks.rados.rados.0.smithi107.stdout:1554: done (2 left) 2020-06-12T02:37:38.516 INFO:tasks.rados.rados.0.smithi107.stdout:1559: done (1 left) 2020-06-12T02:37:38.516 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:37:38.516 INFO:tasks.rados.rados.0.smithi107.stdout:1560: seq_num 1559 ranges {0=516096,516096=663552,1179648=573440,1753088=368640} 2020-06-12T02:37:38.517 INFO:tasks.rados.rados.0.smithi107.stdout:1560: writing smithi10729407-18 from 0 to 516096 tid 1 2020-06-12T02:37:38.519 INFO:tasks.rados.rados.0.smithi107.stdout:1560: writing smithi10729407-18 from 516096 to 1179648 tid 2 2020-06-12T02:37:38.519 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:37:38.520 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:37:38.520 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:37:38.520 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:37:38.520 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:37:38.522 INFO:tasks.rados.rados.0.smithi107.stdout:1560: writing smithi10729407-18 from 1179648 to 1753088 tid 3 2020-06-12T02:37:38.524 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:37:38.521+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:37:38.525 INFO:tasks.rados.rados.0.smithi107.stdout:1560: writing smithi10729407-18 from 1753088 to 2121728 tid 4 2020-06-12T02:37:38.525 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:37:38.531 INFO:tasks.rados.rados.0.smithi107.stdout:1561: seq_num 1560 ranges {3588096=532480,4120576=548864,4669440=778240,5447680=278528} 2020-06-12T02:37:38.532 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-06-12T02:37:38.547 INFO:tasks.rados.rados.0.smithi107.stdout:1561: writing smithi10729407-7 from 3588096 to 4120576 tid 1 2020-06-12T02:37:38.550 INFO:tasks.rados.rados.0.smithi107.stdout:1561: writing smithi10729407-7 from 4120576 to 4669440 tid 2 2020-06-12T02:37:38.556 INFO:tasks.rados.rados.0.smithi107.stdout:1561: writing smithi10729407-7 from 4669440 to 5447680 tid 3 2020-06-12T02:37:38.558 INFO:tasks.rados.rados.0.smithi107.stdout:1561: writing smithi10729407-7 from 5447680 to 5726208 tid 4 2020-06-12T02:37:38.558 INFO:tasks.rados.rados.0.smithi107.stdout:1562: delete oid 4 current snap is 0 2020-06-12T02:37:38.558 INFO:tasks.rados.rados.0.smithi107.stdout:1558: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:38.558 INFO:tasks.rados.rados.0.smithi107.stdout:1558: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:38.559 INFO:tasks.rados.rados.0.smithi107.stdout:1558: finishing write tid 4 to smithi10729407-14 2020-06-12T02:37:38.559 INFO:tasks.rados.rados.0.smithi107.stdout:1558: finishing write tid 6 to smithi10729407-14 2020-06-12T02:37:38.559 INFO:tasks.rados.rados.0.smithi107.stdout:1560: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:38.559 INFO:tasks.rados.rados.0.smithi107.stdout:1558: finishing write tid 7 to smithi10729407-14 2020-06-12T02:37:38.559 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 270 (ObjNum 1557 snap 0 seq_num 1557) dirty exists 2020-06-12T02:37:38.559 INFO:tasks.rados.rados.0.smithi107.stdout:1558: left oid 14 (ObjNum 1557 snap 0 seq_num 1557) 2020-06-12T02:37:38.560 INFO:tasks.rados.rados.0.smithi107.stdout:1560: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:38.560 INFO:tasks.rados.rados.0.smithi107.stdout:1560: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:38.560 INFO:tasks.rados.rados.0.smithi107.stdout:1560: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:38.560 INFO:tasks.rados.rados.0.smithi107.stdout:1560: finishing write tid 6 to smithi10729407-18 2020-06-12T02:37:38.561 INFO:tasks.rados.rados.0.smithi107.stdout:1560: finishing write tid 7 to smithi10729407-18 2020-06-12T02:37:38.561 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 274 (ObjNum 1559 snap 0 seq_num 1559) dirty exists 2020-06-12T02:37:38.561 INFO:tasks.rados.rados.0.smithi107.stdout:1560: left oid 18 (ObjNum 1559 snap 0 seq_num 1559) 2020-06-12T02:37:38.561 INFO:tasks.rados.rados.0.smithi107.stdout:1561: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:38.562 INFO:tasks.rados.rados.0.smithi107.stdout:1558: done (3 left) 2020-06-12T02:37:38.562 INFO:tasks.rados.rados.0.smithi107.stdout:1560: done (2 left) 2020-06-12T02:37:38.562 INFO:tasks.rados.rados.0.smithi107.stdout:1562: done (1 left) 2020-06-12T02:37:38.563 INFO:tasks.rados.rados.0.smithi107.stdout:1563: delete oid 16 current snap is 0 2020-06-12T02:37:38.564 INFO:tasks.rados.rados.0.smithi107.stdout:1561: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:38.565 INFO:tasks.rados.rados.0.smithi107.stdout:1561: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:38.566 INFO:tasks.rados.rados.0.smithi107.stdout:1561: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:38.566 INFO:tasks.rados.rados.0.smithi107.stdout:1561: finishing write tid 6 to smithi10729407-7 2020-06-12T02:37:38.566 INFO:tasks.rados.rados.0.smithi107.stdout:1563: done (1 left) 2020-06-12T02:37:38.566 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:37:38.566 INFO:tasks.rados.rados.0.smithi107.stdout:1564: seq_num 1563 ranges {1384448=581632,1966080=598016,2564096=598016,3162112=409600} 2020-06-12T02:37:38.576 INFO:tasks.rados.rados.0.smithi107.stdout:1564: writing smithi10729407-8 from 1384448 to 1966080 tid 1 2020-06-12T02:37:38.579 INFO:tasks.rados.rados.0.smithi107.stdout:1564: writing smithi10729407-8 from 1966080 to 2564096 tid 2 2020-06-12T02:37:38.583 INFO:tasks.rados.rados.0.smithi107.stdout:1564: writing smithi10729407-8 from 2564096 to 3162112 tid 3 2020-06-12T02:37:38.586 INFO:tasks.rados.rados.0.smithi107.stdout:1564: writing smithi10729407-8 from 3162112 to 3571712 tid 4 2020-06-12T02:37:38.586 INFO:tasks.rados.rados.0.smithi107.stdout:1565: delete oid 9 current snap is 0 2020-06-12T02:37:38.586 INFO:tasks.rados.rados.0.smithi107.stdout:1561: finishing write tid 7 to smithi10729407-7 2020-06-12T02:37:38.586 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 324 (ObjNum 1560 snap 0 seq_num 1560) dirty exists 2020-06-12T02:37:38.587 INFO:tasks.rados.rados.0.smithi107.stdout:1561: left oid 7 (ObjNum 1560 snap 0 seq_num 1560) 2020-06-12T02:37:38.587 INFO:tasks.rados.rados.0.smithi107.stdout:1564: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:38.587 INFO:tasks.rados.rados.0.smithi107.stdout:1561: done (2 left) 2020-06-12T02:37:38.588 INFO:tasks.rados.rados.0.smithi107.stdout:1565: done (1 left) 2020-06-12T02:37:38.588 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:37:38.588 INFO:tasks.rados.rados.0.smithi107.stdout:1566: seq_num 1565 ranges {0=614400,614400=770048,1384448=565248,1949696=270336} 2020-06-12T02:37:38.591 INFO:tasks.rados.rados.0.smithi107.stdout:1566: writing smithi10729407-16 from 0 to 614400 tid 1 2020-06-12T02:37:38.596 INFO:tasks.rados.rados.0.smithi107.stdout:1566: writing smithi10729407-16 from 614400 to 1384448 tid 2 2020-06-12T02:37:38.599 INFO:tasks.rados.rados.0.smithi107.stdout:1566: writing smithi10729407-16 from 1384448 to 1949696 tid 3 2020-06-12T02:37:38.602 INFO:tasks.rados.rados.0.smithi107.stdout:1566: writing smithi10729407-16 from 1949696 to 2220032 tid 4 2020-06-12T02:37:38.603 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:37:38.603 INFO:tasks.rados.rados.0.smithi107.stdout:1567: seq_num 1566 ranges {0=630784,630784=655360,1286144=770048,2056192=180224} 2020-06-12T02:37:38.605 INFO:tasks.rados.rados.0.smithi107.stdout:1567: writing smithi10729407-10 from 0 to 630784 tid 1 2020-06-12T02:37:38.609 INFO:tasks.rados.rados.0.smithi107.stdout:1567: writing smithi10729407-10 from 630784 to 1286144 tid 2 2020-06-12T02:37:38.614 INFO:tasks.rados.rados.0.smithi107.stdout:1567: writing smithi10729407-10 from 1286144 to 2056192 tid 3 2020-06-12T02:37:38.616 INFO:tasks.rados.rados.0.smithi107.stdout:1567: writing smithi10729407-10 from 2056192 to 2236416 tid 4 2020-06-12T02:37:38.616 INFO:tasks.rados.rados.0.smithi107.stdout:1568: delete oid 1 current snap is 0 2020-06-12T02:37:38.616 INFO:tasks.rados.rados.0.smithi107.stdout:1564: finishing write tid 2 to smithi10729407-8 2020-06-12T02:37:38.617 INFO:tasks.rados.rados.0.smithi107.stdout:1564: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:38.617 INFO:tasks.rados.rados.0.smithi107.stdout:1564: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:38.617 INFO:tasks.rados.rados.0.smithi107.stdout:1564: finishing write tid 6 to smithi10729407-8 2020-06-12T02:37:38.618 INFO:tasks.rados.rados.0.smithi107.stdout:1564: finishing write tid 7 to smithi10729407-8 2020-06-12T02:37:38.618 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 272 (ObjNum 1563 snap 0 seq_num 1563) dirty exists 2020-06-12T02:37:38.618 INFO:tasks.rados.rados.0.smithi107.stdout:1564: left oid 8 (ObjNum 1563 snap 0 seq_num 1563) 2020-06-12T02:37:38.618 INFO:tasks.rados.rados.0.smithi107.stdout:1566: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:38.618 INFO:tasks.rados.rados.0.smithi107.stdout:1566: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:38.618 INFO:tasks.rados.rados.0.smithi107.stdout:1566: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:38.619 INFO:tasks.rados.rados.0.smithi107.stdout:1566: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:38.619 INFO:tasks.rados.rados.0.smithi107.stdout:1566: finishing write tid 6 to smithi10729407-16 2020-06-12T02:37:38.619 INFO:tasks.rados.rados.0.smithi107.stdout:1566: finishing write tid 7 to smithi10729407-16 2020-06-12T02:37:38.620 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 275 (ObjNum 1565 snap 0 seq_num 1565) dirty exists 2020-06-12T02:37:38.620 INFO:tasks.rados.rados.0.smithi107.stdout:1566: left oid 16 (ObjNum 1565 snap 0 seq_num 1565) 2020-06-12T02:37:38.620 INFO:tasks.rados.rados.0.smithi107.stdout:1564: done (3 left) 2020-06-12T02:37:38.620 INFO:tasks.rados.rados.0.smithi107.stdout:1566: done (2 left) 2020-06-12T02:37:38.620 INFO:tasks.rados.rados.0.smithi107.stdout:1568: done (1 left) 2020-06-12T02:37:38.621 INFO:tasks.rados.rados.0.smithi107.stdout:1569: delete oid 6 current snap is 0 2020-06-12T02:37:38.621 INFO:tasks.rados.rados.0.smithi107.stdout:1567: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:38.623 INFO:tasks.rados.rados.0.smithi107.stdout:1567: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:38.627 INFO:tasks.rados.rados.0.smithi107.stdout:1569: done (1 left) 2020-06-12T02:37:38.628 INFO:tasks.rados.rados.0.smithi107.stdout:1570: delete oid 4 current snap is 0 2020-06-12T02:37:38.628 INFO:tasks.rados.rados.0.smithi107.stdout:1567: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:38.628 INFO:tasks.rados.rados.0.smithi107.stdout:1570: done (1 left) 2020-06-12T02:37:38.628 INFO:tasks.rados.rados.0.smithi107.stdout:1571: delete oid 3 current snap is 0 2020-06-12T02:37:38.629 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:37:38.625+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:37:38.629 INFO:tasks.rados.rados.0.smithi107.stdout:1567: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:38.629 INFO:tasks.rados.rados.0.smithi107.stdout:1567: finishing write tid 6 to smithi10729407-10 2020-06-12T02:37:38.630 INFO:tasks.rados.rados.0.smithi107.stdout:1571: done (1 left) 2020-06-12T02:37:38.630 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:37:38.630 INFO:tasks.rados.rados.0.smithi107.stdout:1572: seq_num 1571 ranges {2220032=720896,2940928=524288,3465216=557056,4022272=524288} 2020-06-12T02:37:38.646 INFO:tasks.rados.rados.0.smithi107.stdout:1572: writing smithi10729407-16 from 2220032 to 2940928 tid 1 2020-06-12T02:37:38.649 INFO:tasks.rados.rados.0.smithi107.stdout:1572: writing smithi10729407-16 from 2940928 to 3465216 tid 2 2020-06-12T02:37:38.664 INFO:tasks.rados.rados.0.smithi107.stdout:1572: writing smithi10729407-16 from 3465216 to 4022272 tid 3 2020-06-12T02:37:38.664 INFO:tasks.rados.rados.0.smithi107.stdout:1572: writing smithi10729407-16 from 4022272 to 4546560 tid 4 2020-06-12T02:37:38.664 INFO:tasks.rados.rados.0.smithi107.stdout:1573: delete oid 15 current snap is 0 2020-06-12T02:37:38.665 INFO:tasks.rados.rados.0.smithi107.stdout:1573: done (2 left) 2020-06-12T02:37:38.665 INFO:tasks.rados.rados.0.smithi107.stdout:1574: delete oid 1 current snap is 0 2020-06-12T02:37:38.665 INFO:tasks.rados.rados.0.smithi107.stdout:1567: finishing write tid 7 to smithi10729407-10 2020-06-12T02:37:38.665 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 147 (ObjNum 1566 snap 0 seq_num 1566) dirty exists 2020-06-12T02:37:38.665 INFO:tasks.rados.rados.0.smithi107.stdout:1567: left oid 10 (ObjNum 1566 snap 0 seq_num 1566) 2020-06-12T02:37:38.665 INFO:tasks.rados.rados.0.smithi107.stdout:1572: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:38.666 INFO:tasks.rados.rados.0.smithi107.stdout:1572: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:38.666 INFO:tasks.rados.rados.0.smithi107.stdout:1567: done (2 left) 2020-06-12T02:37:38.666 INFO:tasks.rados.rados.0.smithi107.stdout:1574: done (1 left) 2020-06-12T02:37:38.666 INFO:tasks.rados.rados.0.smithi107.stdout:1575: delete oid 11 current snap is 0 2020-06-12T02:37:38.667 INFO:tasks.rados.rados.0.smithi107.stdout:1575: done (1 left) 2020-06-12T02:37:38.667 INFO:tasks.rados.rados.0.smithi107.stdout:1576: delete oid 7 current snap is 0 2020-06-12T02:37:38.667 INFO:tasks.rados.rados.0.smithi107.stdout:1572: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:38.668 INFO:tasks.rados.rados.0.smithi107.stdout:1572: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:38.668 INFO:tasks.rados.rados.0.smithi107.stdout:1572: finishing write tid 6 to smithi10729407-16 2020-06-12T02:37:38.668 INFO:tasks.rados.rados.0.smithi107.stdout:1572: finishing write tid 7 to smithi10729407-16 2020-06-12T02:37:38.669 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 281 (ObjNum 1571 snap 0 seq_num 1571) dirty exists 2020-06-12T02:37:38.669 INFO:tasks.rados.rados.0.smithi107.stdout:1572: left oid 16 (ObjNum 1571 snap 0 seq_num 1571) 2020-06-12T02:37:38.669 INFO:tasks.rados.rados.0.smithi107.stdout:1572: done (1 left) 2020-06-12T02:37:38.669 INFO:tasks.rados.rados.0.smithi107.stdout:1576: done (0 left) 2020-06-12T02:37:38.670 INFO:tasks.rados.rados.0.smithi107.stdout:1577: delete oid 5 current snap is 0 2020-06-12T02:37:38.674 INFO:tasks.rados.rados.0.smithi107.stdout:1577: done (0 left) 2020-06-12T02:37:38.675 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:37:38.675 INFO:tasks.rados.rados.0.smithi107.stdout:1578: seq_num 1577 ranges {2121728=16384} 2020-06-12T02:37:38.686 INFO:tasks.rados.rados.0.smithi107.stdout:1578: writing smithi10729407-18 from 2121728 to 2138112 tid 1 2020-06-12T02:37:38.686 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:37:38.686 INFO:tasks.rados.rados.0.smithi107.stdout:1579: seq_num 1578 ranges {0=32768} 2020-06-12T02:37:38.687 INFO:tasks.rados.rados.0.smithi107.stdout:1579: writing smithi10729407-1 from 0 to 32768 tid 1 2020-06-12T02:37:38.687 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:37:38.687 INFO:tasks.rados.rados.0.smithi107.stdout:1580: seq_num 1579 ranges {0=49152} 2020-06-12T02:37:38.687 INFO:tasks.rados.rados.0.smithi107.stdout:1580: writing smithi10729407-17 from 0 to 49152 tid 1 2020-06-12T02:37:38.687 INFO:tasks.rados.rados.0.smithi107.stdout:1581: delete oid 2 current snap is 0 2020-06-12T02:37:38.688 INFO:tasks.rados.rados.0.smithi107.stdout:1581: done (3 left) 2020-06-12T02:37:38.690 INFO:tasks.rados.rados.0.smithi107.stdout:1582: delete oid 3 current snap is 0 2020-06-12T02:37:38.699 INFO:tasks.rados.rados.0.smithi107.stdout:1578: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:38.699 INFO:tasks.rados.rados.0.smithi107.stdout:1580: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:38.699 INFO:tasks.rados.rados.0.smithi107.stdout:1579: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:38.699 INFO:tasks.rados.rados.0.smithi107.stdout:1582: done (3 left) 2020-06-12T02:37:38.700 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:37:38.700 INFO:tasks.rados.rados.0.smithi107.stdout:1583: seq_num 1582 ranges {0=98304} 2020-06-12T02:37:38.700 INFO:tasks.rados.rados.0.smithi107.stdout:1583: writing smithi10729407-5 from 0 to 98304 tid 1 2020-06-12T02:37:38.700 INFO:tasks.rados.rados.0.smithi107.stdout:1584: delete oid 16 current snap is 0 2020-06-12T02:37:38.700 INFO:tasks.rados.rados.0.smithi107.stdout:1579: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:38.701 INFO:tasks.rados.rados.0.smithi107.stdout:1578: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:38.701 INFO:tasks.rados.rados.0.smithi107.stdout:1583: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:38.701 INFO:tasks.rados.rados.0.smithi107.stdout:1580: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:38.701 INFO:tasks.rados.rados.0.smithi107.stdout:1578: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:38.701 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 277 (ObjNum 1577 snap 0 seq_num 1577) dirty exists 2020-06-12T02:37:38.702 INFO:tasks.rados.rados.0.smithi107.stdout:1578: left oid 18 (ObjNum 1577 snap 0 seq_num 1577) 2020-06-12T02:37:38.703 INFO:tasks.rados.rados.0.smithi107.stdout:1579: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:38.703 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 315 (ObjNum 1578 snap 0 seq_num 1578) dirty exists 2020-06-12T02:37:38.703 INFO:tasks.rados.rados.0.smithi107.stdout:1579: left oid 1 (ObjNum 1578 snap 0 seq_num 1578) 2020-06-12T02:37:38.703 INFO:tasks.rados.rados.0.smithi107.stdout:1580: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:38.703 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 306 (ObjNum 1579 snap 0 seq_num 1579) dirty exists 2020-06-12T02:37:38.704 INFO:tasks.rados.rados.0.smithi107.stdout:1580: left oid 17 (ObjNum 1579 snap 0 seq_num 1579) 2020-06-12T02:37:38.704 INFO:tasks.rados.rados.0.smithi107.stdout:1578: done (4 left) 2020-06-12T02:37:38.705 INFO:tasks.rados.rados.0.smithi107.stdout:1579: done (3 left) 2020-06-12T02:37:38.705 INFO:tasks.rados.rados.0.smithi107.stdout:1580: done (2 left) 2020-06-12T02:37:38.705 INFO:tasks.rados.rados.0.smithi107.stdout:1584: done (1 left) 2020-06-12T02:37:38.705 INFO:tasks.rados.rados.0.smithi107.stdout:1585: delete oid 1 current snap is 0 2020-06-12T02:37:38.705 INFO:tasks.rados.rados.0.smithi107.stdout:1583: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:38.706 INFO:tasks.rados.rados.0.smithi107.stdout:1583: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:38.707 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 136 (ObjNum 1582 snap 0 seq_num 1582) dirty exists 2020-06-12T02:37:38.707 INFO:tasks.rados.rados.0.smithi107.stdout:1583: left oid 5 (ObjNum 1582 snap 0 seq_num 1582) 2020-06-12T02:37:38.707 INFO:tasks.rados.rados.0.smithi107.stdout:1583: done (1 left) 2020-06-12T02:37:38.707 INFO:tasks.rados.rados.0.smithi107.stdout:1585: done (0 left) 2020-06-12T02:37:38.708 INFO:tasks.rados.rados.0.smithi107.stdout:1586: delete oid 9 current snap is 0 2020-06-12T02:37:38.709 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:37:38.709 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:37:38.710 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:37:38.710 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:37:38.710 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.710 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:858 3.3s0 3:ce45c70a:::smithi10729407-7:head [append 0~565248] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:38.710 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.924643+0000", 2020-06-12T02:37:38.711 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 6.7771957470000004, 2020-06-12T02:37:38.711 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015332551999999999, 2020-06-12T02:37:38.711 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:38.711 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:38.711 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:38.712 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:38.712 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:38.712 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 858 2020-06-12T02:37:38.712 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.712 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:38.712 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.713 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:38.713 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.924643+0000", 2020-06-12T02:37:38.713 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:38.713 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.713 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.714 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:38.714 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.924643+0000", 2020-06-12T02:37:38.714 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00067535299999999996 2020-06-12T02:37:38.714 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.714 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.715 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:38.715 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925318+0000", 2020-06-12T02:37:38.715 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9629999999999993e-06 2020-06-12T02:37:38.715 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.715 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.715 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:38.716 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925328+0000", 2020-06-12T02:37:38.716 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7357999999999999e-05 2020-06-12T02:37:38.716 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.716 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.716 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:38.717 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925345+0000", 2020-06-12T02:37:38.717 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7019999999999996e-06 2020-06-12T02:37:38.717 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.717 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.718 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:38.718 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925355+0000", 2020-06-12T02:37:38.718 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6289999999999998e-05 2020-06-12T02:37:38.718 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.719 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.719 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:38.719 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925411+0000", 2020-06-12T02:37:38.719 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8855000000000004e-05 2020-06-12T02:37:38.719 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.720 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.721 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:38.721 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925480+0000", 2020-06-12T02:37:38.721 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0074521149999999996 2020-06-12T02:37:38.721 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.723 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.723 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:38.723 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.932932+0000", 2020-06-12T02:37:38.723 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3647999999999995e-05 2020-06-12T02:37:38.723 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.724 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.724 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:38.724 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.933026+0000", 2020-06-12T02:37:38.724 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00078753199999999997 2020-06-12T02:37:38.724 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.725 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.725 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:38.725 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.933813+0000", 2020-06-12T02:37:38.725 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.006013582 2020-06-12T02:37:38.725 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.726 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.731 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:38.731 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.939827+0000", 2020-06-12T02:37:38.731 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0111999999999996e-05 2020-06-12T02:37:38.731 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.731 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.732 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:38.732 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.939897+0000", 2020-06-12T02:37:38.732 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8041999999999996e-05 2020-06-12T02:37:38.732 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.732 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.733 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:38.733 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.939975+0000", 2020-06-12T02:37:38.733 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015332551999999999 2020-06-12T02:37:38.733 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:38.733 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:38.734 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:38.734 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.734 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.734 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:859 3.3s0 3:ce45c70a:::smithi10729407-7:head [append 0~557056] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:38.734 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.927973+0000", 2020-06-12T02:37:38.734 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 6.7738655359999997, 2020-06-12T02:37:38.735 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014425955000000001, 2020-06-12T02:37:38.735 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:38.735 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:38.735 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:38.735 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:38.736 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:38.736 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 859 2020-06-12T02:37:38.736 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.736 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:38.736 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.736 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:38.737 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.927973+0000", 2020-06-12T02:37:38.737 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:38.737 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.737 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.737 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:38.739 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.927973+0000", 2020-06-12T02:37:38.739 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00058871900000000003 2020-06-12T02:37:38.740 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.740 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.740 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:38.740 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928562+0000", 2020-06-12T02:37:38.740 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5679999999999999e-06 2020-06-12T02:37:38.741 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.741 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.741 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:38.741 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928567+0000", 2020-06-12T02:37:38.741 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1883e-05 2020-06-12T02:37:38.741 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.742 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.742 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:38.742 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928579+0000", 2020-06-12T02:37:38.742 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.097e-06 2020-06-12T02:37:38.742 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.743 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.743 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:38.743 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928586+0000", 2020-06-12T02:37:38.743 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6851999999999998e-05 2020-06-12T02:37:38.744 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.744 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.744 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:38.744 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928633+0000", 2020-06-12T02:37:38.744 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6867999999999997e-05 2020-06-12T02:37:38.744 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.745 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.745 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:38.745 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928700+0000", 2020-06-12T02:37:38.745 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0079226580000000008 2020-06-12T02:37:38.745 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.746 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.746 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:38.746 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.936623+0000", 2020-06-12T02:37:38.746 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000108597 2020-06-12T02:37:38.746 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.746 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.747 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:38.747 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.936731+0000", 2020-06-12T02:37:38.747 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054516799999999995 2020-06-12T02:37:38.747 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.747 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.748 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:38.748 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.937276+0000", 2020-06-12T02:37:38.748 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0043134339999999997 2020-06-12T02:37:38.748 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.748 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.748 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:38.749 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941590+0000", 2020-06-12T02:37:38.749 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00074530099999999997 2020-06-12T02:37:38.749 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.749 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.749 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:38.751 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.942335+0000", 2020-06-12T02:37:38.751 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3810000000000001e-05 2020-06-12T02:37:38.751 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.752 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.752 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:38.752 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.942399+0000", 2020-06-12T02:37:38.752 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014425955000000001 2020-06-12T02:37:38.752 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:38.753 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:38.753 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:38.753 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.753 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.754 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:860 3.3s0 3:ce45c70a:::smithi10729407-7:head [append 0~81920] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:38.754 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.928619+0000", 2020-06-12T02:37:38.754 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 6.7732198610000003, 2020-06-12T02:37:38.754 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017431941999999999, 2020-06-12T02:37:38.755 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:38.755 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:38.755 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:38.755 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:38.755 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:38.756 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 860 2020-06-12T02:37:38.756 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.756 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:38.756 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.756 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:38.757 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928619+0000", 2020-06-12T02:37:38.757 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:38.757 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.757 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.757 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:38.758 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928619+0000", 2020-06-12T02:37:38.758 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000105809 2020-06-12T02:37:38.758 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.758 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.758 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:38.758 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928724+0000", 2020-06-12T02:37:38.759 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2339999999999994e-06 2020-06-12T02:37:38.759 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.759 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.759 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:38.759 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928733+0000", 2020-06-12T02:37:38.760 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0268000000000001e-05 2020-06-12T02:37:38.760 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.760 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.760 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:38.760 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928743+0000", 2020-06-12T02:37:38.761 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5190000000000003e-06 2020-06-12T02:37:38.761 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.761 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.761 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:38.762 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928747+0000", 2020-06-12T02:37:38.762 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0103999999999998e-05 2020-06-12T02:37:38.762 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.762 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.762 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:38.763 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928777+0000", 2020-06-12T02:37:38.763 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8249000000000002e-05 2020-06-12T02:37:38.763 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.763 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.764 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:38.764 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928866+0000", 2020-06-12T02:37:38.764 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011151629 2020-06-12T02:37:38.764 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.765 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.765 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:38.765 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.940017+0000", 2020-06-12T02:37:38.765 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8281e-05 2020-06-12T02:37:38.765 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.766 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.766 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:38.766 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.940076+0000", 2020-06-12T02:37:38.766 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000346736 2020-06-12T02:37:38.766 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.766 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.767 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:38.767 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.940422+0000", 2020-06-12T02:37:38.767 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024135049999999998 2020-06-12T02:37:38.767 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.767 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.768 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:38.768 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.942836+0000", 2020-06-12T02:37:38.768 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003170258 2020-06-12T02:37:38.768 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.768 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.769 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:38.769 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946006+0000", 2020-06-12T02:37:38.769 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4350000000000001e-05 2020-06-12T02:37:38.770 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.770 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.770 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:38.770 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946050+0000", 2020-06-12T02:37:38.771 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017431941999999999 2020-06-12T02:37:38.771 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:38.771 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:38.771 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:38.772 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.772 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.772 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:861 3.3s0 3:ce45c70a:::smithi10729407-7:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:38.772 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.928778+0000", 2020-06-12T02:37:38.772 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 6.7730602480000002, 2020-06-12T02:37:38.773 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017522738, 2020-06-12T02:37:38.773 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:38.773 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:38.773 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:38.774 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:38.774 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:38.774 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 861 2020-06-12T02:37:38.774 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.774 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:38.775 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.775 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:38.775 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928778+0000", 2020-06-12T02:37:38.775 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:38.775 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.775 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.776 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:38.776 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928778+0000", 2020-06-12T02:37:38.776 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8186000000000002e-05 2020-06-12T02:37:38.776 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.776 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.777 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:38.777 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928826+0000", 2020-06-12T02:37:38.777 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6689999999999997e-06 2020-06-12T02:37:38.777 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.777 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.777 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:38.778 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928831+0000", 2020-06-12T02:37:38.778 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.686e-06 2020-06-12T02:37:38.778 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.778 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.778 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:38.779 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928838+0000", 2020-06-12T02:37:38.779 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7159999999999999e-06 2020-06-12T02:37:38.779 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.779 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.779 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:38.779 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928840+0000", 2020-06-12T02:37:38.780 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4212999999999997e-05 2020-06-12T02:37:38.780 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.780 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.780 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:38.780 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928895+0000", 2020-06-12T02:37:38.781 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5853999999999994e-05 2020-06-12T02:37:38.781 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.781 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.781 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:38.782 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928990+0000", 2020-06-12T02:37:38.782 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012023982000000001 2020-06-12T02:37:38.782 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.782 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.782 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:38.783 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941014+0000", 2020-06-12T02:37:38.783 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5104e-05 2020-06-12T02:37:38.783 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.783 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.783 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:38.783 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941070+0000", 2020-06-12T02:37:38.784 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00026444499999999998 2020-06-12T02:37:38.784 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.784 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.784 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:38.784 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941334+0000", 2020-06-12T02:37:38.785 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001704075 2020-06-12T02:37:38.785 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.785 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.785 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:38.785 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.943038+0000", 2020-06-12T02:37:38.785 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031891570000000002 2020-06-12T02:37:38.786 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.786 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.786 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:38.786 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946227+0000", 2020-06-12T02:37:38.786 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3651000000000003e-05 2020-06-12T02:37:38.787 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.787 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.787 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:38.787 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946301+0000", 2020-06-12T02:37:38.787 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017522738 2020-06-12T02:37:38.787 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:38.788 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:38.788 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:38.788 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.788 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.788 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:862 3.3s0 3:ce45c70a:::smithi10729407-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:38.789 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.928876+0000", 2020-06-12T02:37:38.789 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 6.7729627519999998, 2020-06-12T02:37:38.789 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019206890000000001, 2020-06-12T02:37:38.789 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:38.789 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:38.789 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:38.790 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:38.790 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:38.790 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 862 2020-06-12T02:37:38.790 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.790 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:38.791 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.791 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:38.791 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928876+0000", 2020-06-12T02:37:38.791 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:38.791 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.792 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.792 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:38.792 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928876+0000", 2020-06-12T02:37:38.792 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9268999999999999e-05 2020-06-12T02:37:38.792 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.793 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.793 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:38.793 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928925+0000", 2020-06-12T02:37:38.793 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8120000000000002e-06 2020-06-12T02:37:38.793 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.794 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.794 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:38.794 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928932+0000", 2020-06-12T02:37:38.794 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3471e-05 2020-06-12T02:37:38.794 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.795 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.795 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:38.795 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928945+0000", 2020-06-12T02:37:38.795 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4279999999999998e-06 2020-06-12T02:37:38.795 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.796 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.796 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:38.796 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928950+0000", 2020-06-12T02:37:38.796 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6358000000000004e-05 2020-06-12T02:37:38.796 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.796 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.797 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:38.797 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.929016+0000", 2020-06-12T02:37:38.797 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9697999999999996e-05 2020-06-12T02:37:38.797 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.797 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.798 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:38.798 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.929086+0000", 2020-06-12T02:37:38.798 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012409205 2020-06-12T02:37:38.798 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.799 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.799 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:38.799 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941495+0000", 2020-06-12T02:37:38.799 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9660000000000006e-05 2020-06-12T02:37:38.799 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.800 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.800 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:38.800 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941585+0000", 2020-06-12T02:37:38.800 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050329099999999998 2020-06-12T02:37:38.800 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.800 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.801 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:38.801 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946617+0000", 2020-06-12T02:37:38.801 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8437999999999996e-05 2020-06-12T02:37:38.801 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.801 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.802 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:38.802 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946706+0000", 2020-06-12T02:37:38.802 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0013766410000000001 2020-06-12T02:37:38.802 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.803 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.803 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:38.803 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.948083+0000", 2020-06-12T02:37:38.803 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019206890000000001 2020-06-12T02:37:38.803 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:38.803 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:38.804 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:38.804 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.804 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.804 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:879 3.fs1 3:fc7e0bf7:::smithi10729407-17:head [append 0~491520] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:38.804 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.000881+0000", 2020-06-12T02:37:38.805 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 6.700957571, 2020-06-12T02:37:38.805 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017891845, 2020-06-12T02:37:38.805 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:38.805 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:38.805 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:38.805 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:38.806 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:38.806 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 879 2020-06-12T02:37:38.806 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.806 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:38.806 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.807 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:38.807 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.000881+0000", 2020-06-12T02:37:38.807 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:38.807 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.807 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.807 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:38.808 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.000881+0000", 2020-06-12T02:37:38.808 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00044151599999999998 2020-06-12T02:37:38.808 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.808 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.808 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:38.809 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001322+0000", 2020-06-12T02:37:38.809 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0210000000000001e-06 2020-06-12T02:37:38.809 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.809 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.809 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:38.809 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001326+0000", 2020-06-12T02:37:38.810 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6659999999999997e-06 2020-06-12T02:37:38.810 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.810 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.810 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:38.810 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001335+0000", 2020-06-12T02:37:38.811 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4299999999999999e-06 2020-06-12T02:37:38.811 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.811 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.811 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:38.811 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001339+0000", 2020-06-12T02:37:38.811 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0028741510000000001 2020-06-12T02:37:38.812 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.812 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.812 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:38.812 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.004214+0000", 2020-06-12T02:37:38.813 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010480999999999999 2020-06-12T02:37:38.813 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.813 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.813 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:38.813 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.004318+0000", 2020-06-12T02:37:38.814 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021112359999999998 2020-06-12T02:37:38.814 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.814 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.814 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:38.814 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006430+0000", 2020-06-12T02:37:38.814 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2935000000000001e-05 2020-06-12T02:37:38.815 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.815 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.815 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:38.815 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006483+0000", 2020-06-12T02:37:38.815 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00056411099999999995 2020-06-12T02:37:38.816 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.816 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.816 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:38.816 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.007047+0000", 2020-06-12T02:37:38.816 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0076749369999999997 2020-06-12T02:37:38.816 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.817 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.817 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:38.817 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.014722+0000", 2020-06-12T02:37:38.817 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00395693 2020-06-12T02:37:38.817 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.818 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.818 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:38.818 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.018679+0000", 2020-06-12T02:37:38.818 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4102000000000001e-05 2020-06-12T02:37:38.818 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.819 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.819 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:38.819 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.018773+0000", 2020-06-12T02:37:38.819 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017891845 2020-06-12T02:37:38.820 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:38.820 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:38.820 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:38.820 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.821 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.821 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:880 3.fs1 3:fc7e0bf7:::smithi10729407-17:head [append 0~516096] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:38.821 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.004750+0000", 2020-06-12T02:37:38.821 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 6.6970881550000003, 2020-06-12T02:37:38.822 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014302860000000001, 2020-06-12T02:37:38.822 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:38.822 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:38.822 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:38.822 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:38.823 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:38.823 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 880 2020-06-12T02:37:38.823 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.823 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:38.823 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.824 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:38.824 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.004750+0000", 2020-06-12T02:37:38.824 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:38.825 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.825 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.825 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:38.825 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.004750+0000", 2020-06-12T02:37:38.826 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00048597100000000003 2020-06-12T02:37:38.826 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.826 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.826 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:38.826 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.005236+0000", 2020-06-12T02:37:38.827 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2290000000000002e-06 2020-06-12T02:37:38.827 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.827 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.827 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:38.827 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.005241+0000", 2020-06-12T02:37:38.828 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1155000000000001e-05 2020-06-12T02:37:38.828 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.828 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.828 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:38.828 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.005253+0000", 2020-06-12T02:37:38.829 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2570000000000001e-06 2020-06-12T02:37:38.830 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.830 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.830 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:38.830 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.005257+0000", 2020-06-12T02:37:38.830 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8241e-05 2020-06-12T02:37:38.831 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.832 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.833 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:38.833 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.005315+0000", 2020-06-12T02:37:38.833 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6916999999999996e-05 2020-06-12T02:37:38.833 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.834 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.834 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:38.834 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.005382+0000", 2020-06-12T02:37:38.834 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0039171969999999999 2020-06-12T02:37:38.834 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.835 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.835 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:38.835 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.009299+0000", 2020-06-12T02:37:38.835 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000102623 2020-06-12T02:37:38.835 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.835 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.836 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:38.836 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.009402+0000", 2020-06-12T02:37:38.836 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00070054899999999996 2020-06-12T02:37:38.836 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.837 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.837 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:38.837 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.010102+0000", 2020-06-12T02:37:38.838 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.005383891 2020-06-12T02:37:38.838 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.838 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.838 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:38.838 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.015486+0000", 2020-06-12T02:37:38.839 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0034234310000000002 2020-06-12T02:37:38.839 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.839 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.839 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:38.839 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.018910+0000", 2020-06-12T02:37:38.839 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000143399 2020-06-12T02:37:38.840 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.840 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.840 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:38.840 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.019053+0000", 2020-06-12T02:37:38.840 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014302860000000001 2020-06-12T02:37:38.841 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:38.841 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:38.841 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:38.841 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.841 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.842 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:883 3.fs1 3:fc7e0bf7:::smithi10729407-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:38.842 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.006322+0000", 2020-06-12T02:37:38.842 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 6.6955164099999998, 2020-06-12T02:37:38.842 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01444995, 2020-06-12T02:37:38.842 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:38.843 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:38.843 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:38.843 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:38.843 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:38.843 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 883 2020-06-12T02:37:38.844 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.844 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:38.844 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.844 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:38.844 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006322+0000", 2020-06-12T02:37:38.845 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:38.845 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.845 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.845 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:38.845 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006322+0000", 2020-06-12T02:37:38.846 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1342999999999999e-05 2020-06-12T02:37:38.846 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.846 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.846 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:38.846 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006343+0000", 2020-06-12T02:37:38.846 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.021e-06 2020-06-12T02:37:38.847 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.848 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.848 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:38.848 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006346+0000", 2020-06-12T02:37:38.848 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2990000000000002e-06 2020-06-12T02:37:38.848 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.849 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.849 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:38.849 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006353+0000", 2020-06-12T02:37:38.849 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3240000000000001e-06 2020-06-12T02:37:38.849 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.850 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.850 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:38.850 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006355+0000", 2020-06-12T02:37:38.850 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0090315139999999992 2020-06-12T02:37:38.850 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.851 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.851 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:38.851 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.015386+0000", 2020-06-12T02:37:38.851 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5129000000000003e-05 2020-06-12T02:37:38.851 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.851 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.852 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:38.852 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.015482+0000", 2020-06-12T02:37:38.852 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0043414839999999996 2020-06-12T02:37:38.852 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.852 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.853 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:38.853 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.019823+0000", 2020-06-12T02:37:38.853 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6991999999999997e-05 2020-06-12T02:37:38.853 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.853 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.853 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:38.854 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.019910+0000", 2020-06-12T02:37:38.854 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00086184399999999996 2020-06-12T02:37:38.854 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.854 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.854 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:38.855 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.020772+0000", 2020-06-12T02:37:38.855 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01444995 2020-06-12T02:37:38.855 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:38.855 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:38.855 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:38.855 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.856 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.856 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:886 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~712704] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:38.856 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.017916+0000", 2020-06-12T02:37:38.856 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 6.6839224799999997, 2020-06-12T02:37:38.856 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014030252999999999, 2020-06-12T02:37:38.857 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:38.857 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:38.857 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:38.858 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:38.858 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:38.858 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 886 2020-06-12T02:37:38.858 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.858 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:38.859 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.859 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:38.859 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.017916+0000", 2020-06-12T02:37:38.859 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:38.859 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.860 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.860 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:38.860 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.017916+0000", 2020-06-12T02:37:38.860 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00067910699999999997 2020-06-12T02:37:38.860 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.860 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.861 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:38.861 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.018595+0000", 2020-06-12T02:37:38.861 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.0949999999999996e-06 2020-06-12T02:37:38.861 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.861 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.862 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:38.862 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.018603+0000", 2020-06-12T02:37:38.862 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5002e-05 2020-06-12T02:37:38.862 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.862 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.862 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:38.863 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.018618+0000", 2020-06-12T02:37:38.863 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3300000000000004e-06 2020-06-12T02:37:38.863 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.863 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.863 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:38.864 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.018624+0000", 2020-06-12T02:37:38.864 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8525999999999997e-05 2020-06-12T02:37:38.864 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.864 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.864 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:38.864 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.018663+0000", 2020-06-12T02:37:38.865 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000104404 2020-06-12T02:37:38.865 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.865 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.865 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:38.865 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.018767+0000", 2020-06-12T02:37:38.867 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0055157289999999996 2020-06-12T02:37:38.867 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.867 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.868 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:38.868 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.024283+0000", 2020-06-12T02:37:38.868 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1125999999999998e-05 2020-06-12T02:37:38.868 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.868 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.869 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:38.869 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.024334+0000", 2020-06-12T02:37:38.869 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000482585 2020-06-12T02:37:38.869 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.869 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.870 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:38.870 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.024817+0000", 2020-06-12T02:37:38.870 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0047718500000000002 2020-06-12T02:37:38.870 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.871 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.871 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:38.871 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.029589+0000", 2020-06-12T02:37:38.871 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023106070000000001 2020-06-12T02:37:38.871 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.871 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.872 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:38.872 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.031899+0000", 2020-06-12T02:37:38.872 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6891999999999998e-05 2020-06-12T02:37:38.872 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.872 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.873 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:38.873 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.031946+0000", 2020-06-12T02:37:38.873 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014030252999999999 2020-06-12T02:37:38.873 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:38.873 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:38.874 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:38.874 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.874 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.874 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:889 3.3s0 3:c2ca77da:::smithi10729407-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:38.874 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.022069+0000", 2020-06-12T02:37:38.874 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 6.6797697740000004, 2020-06-12T02:37:38.875 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01386604, 2020-06-12T02:37:38.875 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:38.875 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:38.875 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:38.876 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:38.876 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:38.876 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 889 2020-06-12T02:37:38.876 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.876 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:38.877 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.877 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:38.877 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.022069+0000", 2020-06-12T02:37:38.877 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:38.877 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.878 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.878 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:38.878 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.022069+0000", 2020-06-12T02:37:38.878 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4539000000000001e-05 2020-06-12T02:37:38.878 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.879 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.879 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:38.879 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.022123+0000", 2020-06-12T02:37:38.879 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.888e-06 2020-06-12T02:37:38.879 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.880 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.880 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:38.880 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.022130+0000", 2020-06-12T02:37:38.880 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4331999999999999e-05 2020-06-12T02:37:38.880 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.880 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.881 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:38.881 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.022144+0000", 2020-06-12T02:37:38.881 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4870000000000004e-06 2020-06-12T02:37:38.881 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.881 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.882 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:38.882 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.022149+0000", 2020-06-12T02:37:38.882 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4615999999999999e-05 2020-06-12T02:37:38.882 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.882 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.882 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:38.883 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.022173+0000", 2020-06-12T02:37:38.883 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1434999999999998e-05 2020-06-12T02:37:38.883 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.883 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.883 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:38.884 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.022225+0000", 2020-06-12T02:37:38.884 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0079399250000000005 2020-06-12T02:37:38.884 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.884 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.884 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:38.884 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.030165+0000", 2020-06-12T02:37:38.885 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8032000000000002e-05 2020-06-12T02:37:38.885 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.885 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.885 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:38.886 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.030233+0000", 2020-06-12T02:37:38.886 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004515453 2020-06-12T02:37:38.886 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.887 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.887 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:38.887 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.034748+0000", 2020-06-12T02:37:38.887 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9342000000000002e-05 2020-06-12T02:37:38.887 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.887 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.888 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:38.888 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.034808+0000", 2020-06-12T02:37:38.888 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0011269909999999999 2020-06-12T02:37:38.888 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.888 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.889 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:38.889 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.035935+0000", 2020-06-12T02:37:38.889 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01386604 2020-06-12T02:37:38.889 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:38.889 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:38.889 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:38.890 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.890 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.890 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:933 3.es1 3:78f01648:::smithi10729407-5:head [append 0~417792] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:38.890 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.156677+0000", 2020-06-12T02:37:38.890 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 6.5451615890000001, 2020-06-12T02:37:38.891 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016833311, 2020-06-12T02:37:38.891 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:38.891 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:38.891 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:38.891 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:38.892 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:38.892 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 933 2020-06-12T02:37:38.892 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.892 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:38.892 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.892 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:38.893 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.156677+0000", 2020-06-12T02:37:38.893 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:38.893 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.893 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.893 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:38.894 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.156677+0000", 2020-06-12T02:37:38.894 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00062571700000000003 2020-06-12T02:37:38.894 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.894 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.894 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:38.894 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157303+0000", 2020-06-12T02:37:38.895 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0740000000000003e-06 2020-06-12T02:37:38.895 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.895 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.895 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:38.895 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157307+0000", 2020-06-12T02:37:38.896 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4419999999999998e-06 2020-06-12T02:37:38.896 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.896 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.896 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:38.897 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157315+0000", 2020-06-12T02:37:38.897 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5300000000000001e-06 2020-06-12T02:37:38.897 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.897 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.897 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:38.898 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157319+0000", 2020-06-12T02:37:38.898 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4995e-05 2020-06-12T02:37:38.898 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.898 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.898 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:38.898 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157354+0000", 2020-06-12T02:37:38.899 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5883000000000006e-05 2020-06-12T02:37:38.899 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.899 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.899 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:38.899 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157419+0000", 2020-06-12T02:37:38.900 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045316280000000002 2020-06-12T02:37:38.900 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.900 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.900 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:38.900 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.161951+0000", 2020-06-12T02:37:38.900 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3300999999999996e-05 2020-06-12T02:37:38.901 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.901 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.901 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:38.901 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.162014+0000", 2020-06-12T02:37:38.901 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00039050100000000002 2020-06-12T02:37:38.902 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.902 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.902 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:38.902 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.162405+0000", 2020-06-12T02:37:38.902 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050686250000000002 2020-06-12T02:37:38.902 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.903 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.903 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:38.903 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.167474+0000", 2020-06-12T02:37:38.903 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.005942653 2020-06-12T02:37:38.903 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.904 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.904 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:38.904 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.173416+0000", 2020-06-12T02:37:38.904 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3962000000000006e-05 2020-06-12T02:37:38.904 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.904 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.905 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:38.905 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.173510+0000", 2020-06-12T02:37:38.905 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016833311 2020-06-12T02:37:38.905 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:38.905 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:38.906 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:38.906 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.906 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.906 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:934 3.es1 3:78f01648:::smithi10729407-5:head [append 0~458752] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:38.907 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.159379+0000", 2020-06-12T02:37:38.907 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 6.5424589360000001, 2020-06-12T02:37:38.907 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014294441, 2020-06-12T02:37:38.907 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:38.907 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:38.908 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:38.908 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:38.908 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:38.908 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 934 2020-06-12T02:37:38.908 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.908 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:38.909 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.909 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:38.909 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.159379+0000", 2020-06-12T02:37:38.909 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:38.909 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.910 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.910 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:38.910 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.159379+0000", 2020-06-12T02:37:38.910 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000706666 2020-06-12T02:37:38.910 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.910 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.911 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:38.911 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160086+0000", 2020-06-12T02:37:38.911 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0941000000000001e-05 2020-06-12T02:37:38.911 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.911 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.912 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:38.912 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160097+0000", 2020-06-12T02:37:38.912 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6980999999999999e-05 2020-06-12T02:37:38.912 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.912 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.912 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:38.913 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160114+0000", 2020-06-12T02:37:38.913 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5089999999999996e-06 2020-06-12T02:37:38.913 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.913 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.913 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:38.914 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160122+0000", 2020-06-12T02:37:38.914 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00389617 2020-06-12T02:37:38.914 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.914 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.914 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:38.914 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.164018+0000", 2020-06-12T02:37:38.915 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9804999999999998e-05 2020-06-12T02:37:38.915 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.915 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.915 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:38.916 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.164108+0000", 2020-06-12T02:37:38.916 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000378942 2020-06-12T02:37:38.916 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.916 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.917 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:38.917 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.164486+0000", 2020-06-12T02:37:38.917 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031318969999999998 2020-06-12T02:37:38.918 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.918 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.918 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:38.918 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.167618+0000", 2020-06-12T02:37:38.918 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0059775799999999997 2020-06-12T02:37:38.919 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.919 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.919 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:38.919 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.173596+0000", 2020-06-12T02:37:38.919 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7949999999999997e-05 2020-06-12T02:37:38.920 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.920 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.920 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:38.920 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.173674+0000", 2020-06-12T02:37:38.920 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014294441 2020-06-12T02:37:38.921 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:38.921 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:38.921 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:38.921 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.921 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.922 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:935 3.es1 3:78f01648:::smithi10729407-5:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:38.922 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.160186+0000", 2020-06-12T02:37:38.922 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 6.5416519949999996, 2020-06-12T02:37:38.922 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013696494, 2020-06-12T02:37:38.923 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:38.923 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:38.923 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:38.923 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:38.923 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:38.924 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 935 2020-06-12T02:37:38.924 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.924 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:38.924 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.924 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:38.924 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160186+0000", 2020-06-12T02:37:38.925 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:38.925 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.925 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.925 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:38.925 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160186+0000", 2020-06-12T02:37:38.926 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0474000000000001e-05 2020-06-12T02:37:38.926 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.926 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.926 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:38.927 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160277+0000", 2020-06-12T02:37:38.927 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9480000000000008e-06 2020-06-12T02:37:38.927 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.927 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.927 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:38.928 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160285+0000", 2020-06-12T02:37:38.928 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2921e-05 2020-06-12T02:37:38.928 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.928 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.928 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:38.928 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160298+0000", 2020-06-12T02:37:38.929 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0549999999999996e-06 2020-06-12T02:37:38.929 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.929 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.929 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:38.929 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160304+0000", 2020-06-12T02:37:38.935 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.006421698 2020-06-12T02:37:38.936 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.936 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.936 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:38.936 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.166725+0000", 2020-06-12T02:37:38.937 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000108103 2020-06-12T02:37:38.937 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.937 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.937 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:38.938 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.166834+0000", 2020-06-12T02:37:38.938 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00031828500000000002 2020-06-12T02:37:38.938 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.938 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.938 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:38.939 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.167152+0000", 2020-06-12T02:37:38.939 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025414019999999999 2020-06-12T02:37:38.939 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.939 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.940 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:38.940 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.169693+0000", 2020-06-12T02:37:38.940 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0040771100000000001 2020-06-12T02:37:38.940 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.940 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.941 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:38.941 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.173770+0000", 2020-06-12T02:37:38.941 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011249800000000001 2020-06-12T02:37:38.941 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.942 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.942 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:38.942 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.173883+0000", 2020-06-12T02:37:38.942 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013696494 2020-06-12T02:37:38.942 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:38.943 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:38.943 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:38.943 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.943 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.943 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:936 3.es1 3:78f01648:::smithi10729407-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:38.944 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.160357+0000", 2020-06-12T02:37:38.944 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 6.5414818779999999, 2020-06-12T02:37:38.944 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015435590000000001, 2020-06-12T02:37:38.944 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:38.944 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:38.945 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:38.945 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:38.945 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:38.945 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 936 2020-06-12T02:37:38.945 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.945 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:38.946 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.946 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:38.946 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160357+0000", 2020-06-12T02:37:38.946 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:38.946 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.947 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.947 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:38.947 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160357+0000", 2020-06-12T02:37:38.947 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4435999999999999e-05 2020-06-12T02:37:38.947 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.947 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.948 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:38.948 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160411+0000", 2020-06-12T02:37:38.948 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2329999999999995e-06 2020-06-12T02:37:38.948 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.948 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.949 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:38.949 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160420+0000", 2020-06-12T02:37:38.949 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7003e-05 2020-06-12T02:37:38.949 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.949 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.950 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:38.950 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160437+0000", 2020-06-12T02:37:38.950 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8370000000000002e-06 2020-06-12T02:37:38.950 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.950 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.951 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:38.951 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160443+0000", 2020-06-12T02:37:38.951 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0070635489999999997 2020-06-12T02:37:38.951 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.951 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.951 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:38.952 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.167507+0000", 2020-06-12T02:37:38.952 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010589700000000001 2020-06-12T02:37:38.952 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.952 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.952 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:38.953 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.167612+0000", 2020-06-12T02:37:38.953 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0065032299999999996 2020-06-12T02:37:38.953 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.953 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.953 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:38.953 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.174116+0000", 2020-06-12T02:37:38.954 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3128999999999999e-05 2020-06-12T02:37:38.954 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.954 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.954 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:38.954 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.174169+0000", 2020-06-12T02:37:38.955 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0016232760000000001 2020-06-12T02:37:38.955 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.955 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.955 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:38.955 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.175792+0000", 2020-06-12T02:37:38.956 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015435590000000001 2020-06-12T02:37:38.956 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:38.956 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:38.956 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:38.956 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.956 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.957 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:950 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~778240] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:38.957 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.207643+0000", 2020-06-12T02:37:38.957 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 6.4941952949999999, 2020-06-12T02:37:38.957 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014844661, 2020-06-12T02:37:38.957 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:38.958 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:38.958 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:38.958 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:38.959 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:38.959 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 950 2020-06-12T02:37:38.959 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.960 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:38.960 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.960 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:38.960 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.207643+0000", 2020-06-12T02:37:38.961 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:38.961 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.961 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.961 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:38.961 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.207643+0000", 2020-06-12T02:37:38.962 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00072843099999999998 2020-06-12T02:37:38.962 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.962 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.962 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:38.963 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208372+0000", 2020-06-12T02:37:38.963 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9120000000000001e-06 2020-06-12T02:37:38.963 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.963 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.963 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:38.964 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208378+0000", 2020-06-12T02:37:38.964 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9660000000000008e-06 2020-06-12T02:37:38.964 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.964 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.964 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:38.965 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208388+0000", 2020-06-12T02:37:38.965 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1279999999999999e-06 2020-06-12T02:37:38.965 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.965 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.965 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:38.965 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208394+0000", 2020-06-12T02:37:38.966 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0439999999999998e-05 2020-06-12T02:37:38.966 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.966 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.966 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:38.966 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208444+0000", 2020-06-12T02:37:38.967 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2611000000000001e-05 2020-06-12T02:37:38.967 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.967 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.967 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:38.967 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208507+0000", 2020-06-12T02:37:38.967 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001228301 2020-06-12T02:37:38.968 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.968 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.968 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:38.968 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.209735+0000", 2020-06-12T02:37:38.968 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8875000000000001e-05 2020-06-12T02:37:38.969 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.969 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.969 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:38.969 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.209794+0000", 2020-06-12T02:37:38.969 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053218899999999997 2020-06-12T02:37:38.969 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.970 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.970 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:38.970 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.210326+0000", 2020-06-12T02:37:38.970 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0078198439999999994 2020-06-12T02:37:38.970 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.971 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.971 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:38.971 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.218146+0000", 2020-06-12T02:37:38.971 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004300496 2020-06-12T02:37:38.971 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.971 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.972 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:38.972 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.222446+0000", 2020-06-12T02:37:38.972 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1467999999999997e-05 2020-06-12T02:37:38.972 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.972 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.973 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:38.973 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.222488+0000", 2020-06-12T02:37:38.973 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014844661 2020-06-12T02:37:38.974 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:38.974 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:38.975 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:38.975 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.975 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.975 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:953 3.3s0 3:c2ca77da:::smithi10729407-13:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:38.975 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.212848+0000", 2020-06-12T02:37:38.976 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 6.4889900589999998, 2020-06-12T02:37:38.976 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013849036, 2020-06-12T02:37:38.976 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:38.976 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:38.976 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:38.977 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:38.977 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:38.977 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 953 2020-06-12T02:37:38.977 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.977 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:38.977 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.978 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:38.978 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212848+0000", 2020-06-12T02:37:38.978 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:38.978 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.978 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.979 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:38.979 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212848+0000", 2020-06-12T02:37:38.979 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7985000000000003e-05 2020-06-12T02:37:38.979 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.979 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.979 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:38.980 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212886+0000", 2020-06-12T02:37:38.980 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1439999999999999e-06 2020-06-12T02:37:38.980 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.980 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.980 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:38.981 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212889+0000", 2020-06-12T02:37:38.981 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4360000000000002e-06 2020-06-12T02:37:38.981 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.981 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.981 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:38.981 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212898+0000", 2020-06-12T02:37:38.982 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3489999999999999e-06 2020-06-12T02:37:38.982 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.982 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.982 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:38.982 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212900+0000", 2020-06-12T02:37:38.984 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0047082119999999998 2020-06-12T02:37:38.984 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.984 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.985 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:38.985 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.217608+0000", 2020-06-12T02:37:38.985 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3783999999999997e-05 2020-06-12T02:37:38.985 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.985 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.986 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:38.986 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.217682+0000", 2020-06-12T02:37:38.986 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00026563599999999998 2020-06-12T02:37:38.986 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.986 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.987 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:38.987 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.217948+0000", 2020-06-12T02:37:38.987 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000355399 2020-06-12T02:37:38.987 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.987 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.987 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:38.988 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.218303+0000", 2020-06-12T02:37:38.988 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0083342989999999999 2020-06-12T02:37:38.988 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.988 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.988 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:38.989 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.226638+0000", 2020-06-12T02:37:38.989 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9791999999999999e-05 2020-06-12T02:37:38.989 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.989 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.989 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:38.989 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.226697+0000", 2020-06-12T02:37:38.990 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013849036 2020-06-12T02:37:38.990 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:38.990 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:38.990 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:38.990 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.991 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.991 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:954 3.3s0 3:c2ca77da:::smithi10729407-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:38.991 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.212923+0000", 2020-06-12T02:37:38.991 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 6.4889155189999999, 2020-06-12T02:37:38.991 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015411546999999999, 2020-06-12T02:37:38.992 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:38.992 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:38.992 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:38.992 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:38.992 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:38.993 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 954 2020-06-12T02:37:38.993 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.993 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:38.993 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.993 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:38.993 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212923+0000", 2020-06-12T02:37:38.994 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:38.994 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.994 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.994 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:38.994 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212923+0000", 2020-06-12T02:37:38.995 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3598e-05 2020-06-12T02:37:38.995 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.995 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.995 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:38.995 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212946+0000", 2020-06-12T02:37:38.995 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.061e-06 2020-06-12T02:37:38.996 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.996 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.996 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:38.996 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212950+0000", 2020-06-12T02:37:38.996 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8700000000000003e-06 2020-06-12T02:37:38.997 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.997 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.997 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:38.997 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212956+0000", 2020-06-12T02:37:38.998 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0770000000000002e-06 2020-06-12T02:37:38.998 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.998 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.998 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:38.998 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212958+0000", 2020-06-12T02:37:38.999 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051235170000000002 2020-06-12T02:37:38.999 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:38.999 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:38.999 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:38.999 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.218082+0000", 2020-06-12T02:37:38.999 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9246999999999997e-05 2020-06-12T02:37:39.000 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.000 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.000 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:39.000 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.218141+0000", 2020-06-12T02:37:39.000 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0087981910000000003 2020-06-12T02:37:39.001 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.001 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.001 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:39.001 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.226939+0000", 2020-06-12T02:37:39.001 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4119999999999997e-05 2020-06-12T02:37:39.002 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.002 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.002 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:39.002 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.226994+0000", 2020-06-12T02:37:39.002 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001340866 2020-06-12T02:37:39.002 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.003 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.003 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:39.003 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.228334+0000", 2020-06-12T02:37:39.003 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015411546999999999 2020-06-12T02:37:39.003 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:39.004 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:39.004 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:39.004 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.004 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.004 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1097 3.es1 3:78f01648:::smithi10729407-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:39.004 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.576496+0000", 2020-06-12T02:37:39.005 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 6.1253420370000002, 2020-06-12T02:37:39.005 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014992914, 2020-06-12T02:37:39.005 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:39.005 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:39.006 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:39.006 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:39.006 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:39.006 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1097 2020-06-12T02:37:39.007 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.007 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:39.007 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.007 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:39.007 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576496+0000", 2020-06-12T02:37:39.008 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:39.008 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.008 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.008 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:39.008 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576496+0000", 2020-06-12T02:37:39.009 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3221999999999999e-05 2020-06-12T02:37:39.009 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.009 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.009 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:39.009 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576520+0000", 2020-06-12T02:37:39.010 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.988e-06 2020-06-12T02:37:39.010 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.010 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.010 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:39.010 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576523+0000", 2020-06-12T02:37:39.011 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2940000000000002e-06 2020-06-12T02:37:39.011 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.011 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.011 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:39.011 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576529+0000", 2020-06-12T02:37:39.011 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6060000000000002e-06 2020-06-12T02:37:39.012 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.012 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.012 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:39.012 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576534+0000", 2020-06-12T02:37:39.012 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001254554 2020-06-12T02:37:39.013 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.013 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.013 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:39.013 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.577789+0000", 2020-06-12T02:37:39.013 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3177999999999998e-05 2020-06-12T02:37:39.013 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.014 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.014 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:39.014 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.577842+0000", 2020-06-12T02:37:39.014 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0063091220000000003 2020-06-12T02:37:39.014 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.015 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.015 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:39.015 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.584151+0000", 2020-06-12T02:37:39.015 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.783e-05 2020-06-12T02:37:39.015 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.016 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.016 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:39.016 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.584209+0000", 2020-06-12T02:37:39.016 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045863650000000002 2020-06-12T02:37:39.017 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.017 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.017 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:39.017 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.588796+0000", 2020-06-12T02:37:39.017 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7614000000000006e-05 2020-06-12T02:37:39.018 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.018 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.018 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:39.018 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.588873+0000", 2020-06-12T02:37:39.018 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0026161410000000002 2020-06-12T02:37:39.019 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.019 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.019 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:39.019 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.591489+0000", 2020-06-12T02:37:39.019 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014992914 2020-06-12T02:37:39.019 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:39.020 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:39.020 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:39.020 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.020 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.021 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1207 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~630784] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:39.021 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.859578+0000", 2020-06-12T02:37:39.021 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 5.8422605010000002, 2020-06-12T02:37:39.021 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014710433, 2020-06-12T02:37:39.021 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:39.022 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:39.022 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:39.022 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:39.022 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:39.022 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1207 2020-06-12T02:37:39.023 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.023 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:39.023 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.023 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:39.023 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.859578+0000", 2020-06-12T02:37:39.023 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:39.024 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.024 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.024 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:39.024 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.859578+0000", 2020-06-12T02:37:39.024 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000775511 2020-06-12T02:37:39.025 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.025 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.025 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:39.025 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.860353+0000", 2020-06-12T02:37:39.025 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5325999999999999e-05 2020-06-12T02:37:39.026 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.026 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.026 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:39.026 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.860369+0000", 2020-06-12T02:37:39.026 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7430000000000001e-05 2020-06-12T02:37:39.026 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.027 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.027 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:39.027 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.860386+0000", 2020-06-12T02:37:39.027 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.426e-06 2020-06-12T02:37:39.027 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.028 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.028 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:39.028 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.860393+0000", 2020-06-12T02:37:39.028 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3585999999999999e-05 2020-06-12T02:37:39.028 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.028 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.029 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:39.029 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.860436+0000", 2020-06-12T02:37:39.029 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2559999999999996e-05 2020-06-12T02:37:39.029 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.029 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.030 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:39.030 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.860509+0000", 2020-06-12T02:37:39.030 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00063177700000000003 2020-06-12T02:37:39.030 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.030 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.030 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:39.031 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.861141+0000", 2020-06-12T02:37:39.031 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0026307570000000001 2020-06-12T02:37:39.031 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.031 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.031 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:39.032 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.863771+0000", 2020-06-12T02:37:39.032 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010445545000000001 2020-06-12T02:37:39.032 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.032 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.033 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:39.033 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.874217+0000", 2020-06-12T02:37:39.034 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1514999999999995e-05 2020-06-12T02:37:39.034 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.034 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.034 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:39.034 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.874288+0000", 2020-06-12T02:37:39.035 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014710433 2020-06-12T02:37:39.035 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:39.035 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:39.035 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:39.035 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.035 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.036 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2788 3.fs1 3:fc7e0bf7:::smithi10729407-17:head [append 0~679936] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:39.036 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:36.820585+0000", 2020-06-12T02:37:39.036 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 1.8812534379999999, 2020-06-12T02:37:39.036 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01440234, 2020-06-12T02:37:39.036 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:39.037 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:39.037 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:39.037 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:39.037 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:39.037 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2788 2020-06-12T02:37:39.038 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.041 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:39.041 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.041 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:39.041 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.820585+0000", 2020-06-12T02:37:39.042 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:39.042 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.042 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.042 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:39.042 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.820585+0000", 2020-06-12T02:37:39.043 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001119386 2020-06-12T02:37:39.043 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.043 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.043 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:39.043 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.821704+0000", 2020-06-12T02:37:39.043 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2069000000000001e-05 2020-06-12T02:37:39.044 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.044 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.044 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:39.044 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.821716+0000", 2020-06-12T02:37:39.044 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2909e-05 2020-06-12T02:37:39.045 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.045 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.045 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:39.045 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.821729+0000", 2020-06-12T02:37:39.045 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4559999999999997e-06 2020-06-12T02:37:39.045 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.046 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.046 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:39.046 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.821736+0000", 2020-06-12T02:37:39.046 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6715000000000003e-05 2020-06-12T02:37:39.046 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.047 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.047 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:39.047 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.821772+0000", 2020-06-12T02:37:39.047 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7078999999999998e-05 2020-06-12T02:37:39.047 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.047 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.048 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:39.048 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.821850+0000", 2020-06-12T02:37:39.048 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00049280600000000004 2020-06-12T02:37:39.048 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.048 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.049 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:39.049 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.822342+0000", 2020-06-12T02:37:39.049 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051665629999999999 2020-06-12T02:37:39.049 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.049 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.049 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:39.050 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.827509+0000", 2020-06-12T02:37:39.050 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0073895280000000002 2020-06-12T02:37:39.050 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.050 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.051 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:39.051 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.834898+0000", 2020-06-12T02:37:39.051 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8828999999999999e-05 2020-06-12T02:37:39.051 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.052 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.052 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:39.052 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.834987+0000", 2020-06-12T02:37:39.052 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01440234 2020-06-12T02:37:39.052 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:39.053 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:39.053 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:39.053 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:39.053 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:39.053 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:37:39.054 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-06-12T02:37:39.074 INFO:tasks.rados.rados.0.smithi107.stdout:1586: done (0 left) 2020-06-12T02:37:39.075 INFO:tasks.rados.rados.0.smithi107.stdout:1587: delete oid 5 current snap is 0 2020-06-12T02:37:39.075 INFO:tasks.rados.rados.0.smithi107.stdout:1587: done (0 left) 2020-06-12T02:37:39.075 INFO:tasks.rados.rados.0.smithi107.stdout:1588: delete oid 5 current snap is 0 2020-06-12T02:37:39.075 INFO:tasks.rados.rados.0.smithi107.stdout:1588: done (0 left) 2020-06-12T02:37:39.076 INFO:tasks.rados.rados.0.smithi107.stdout:1589: delete oid 10 current snap is 0 2020-06-12T02:37:39.076 INFO:tasks.rados.rados.0.smithi107.stdout:1589: done (0 left) 2020-06-12T02:37:39.076 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:37:39.076 INFO:tasks.rados.rados.0.smithi107.stdout:1590: seq_num 1589 ranges {0=221184} 2020-06-12T02:37:39.076 INFO:tasks.rados.rados.0.smithi107.stdout:1590: writing smithi10729407-3 from 0 to 221184 tid 1 2020-06-12T02:37:39.077 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:37:39.077 INFO:tasks.rados.rados.0.smithi107.stdout:1591: seq_num 1590 ranges {0=237568} 2020-06-12T02:37:39.077 INFO:tasks.rados.rados.0.smithi107.stdout:1591: writing smithi10729407-12 from 0 to 237568 tid 1 2020-06-12T02:37:39.077 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:37:39.077 INFO:tasks.rados.rados.0.smithi107.stdout:1592: seq_num 1591 ranges {0=253952} 2020-06-12T02:37:39.077 INFO:tasks.rados.rados.0.smithi107.stdout:1592: writing smithi10729407-5 from 0 to 253952 tid 1 2020-06-12T02:37:39.078 INFO:tasks.rados.rados.0.smithi107.stdout:1593: delete oid 11 current snap is 0 2020-06-12T02:37:39.078 INFO:tasks.rados.rados.0.smithi107.stdout:1590: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:39.078 INFO:tasks.rados.rados.0.smithi107.stdout:1591: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:39.078 INFO:tasks.rados.rados.0.smithi107.stdout:1592: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:39.078 INFO:tasks.rados.rados.0.smithi107.stdout:1593: done (3 left) 2020-06-12T02:37:39.079 INFO:tasks.rados.rados.0.smithi107.stdout:1594: delete oid 11 current snap is 0 2020-06-12T02:37:39.079 INFO:tasks.rados.rados.0.smithi107.stdout:1590: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:39.079 INFO:tasks.rados.rados.0.smithi107.stdout:1592: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:39.079 INFO:tasks.rados.rados.0.smithi107.stdout:1590: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:39.079 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 285 (ObjNum 1589 snap 0 seq_num 1589) dirty exists 2020-06-12T02:37:39.079 INFO:tasks.rados.rados.0.smithi107.stdout:1590: left oid 3 (ObjNum 1589 snap 0 seq_num 1589) 2020-06-12T02:37:39.080 INFO:tasks.rados.rados.0.smithi107.stdout:1591: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:39.080 INFO:tasks.rados.rados.0.smithi107.stdout:1590: done (3 left) 2020-06-12T02:37:39.080 INFO:tasks.rados.rados.0.smithi107.stdout:1594: done (2 left) 2020-06-12T02:37:39.080 INFO:tasks.rados.rados.0.smithi107.stdout:1595: delete oid 7 current snap is 0 2020-06-12T02:37:39.080 INFO:tasks.rados.rados.0.smithi107.stdout:1591: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:39.081 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 245 (ObjNum 1590 snap 0 seq_num 1590) dirty exists 2020-06-12T02:37:39.081 INFO:tasks.rados.rados.0.smithi107.stdout:1591: left oid 12 (ObjNum 1590 snap 0 seq_num 1590) 2020-06-12T02:37:39.081 INFO:tasks.rados.rados.0.smithi107.stdout:1592: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:39.081 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 140 (ObjNum 1591 snap 0 seq_num 1591) dirty exists 2020-06-12T02:37:39.081 INFO:tasks.rados.rados.0.smithi107.stdout:1592: left oid 5 (ObjNum 1591 snap 0 seq_num 1591) 2020-06-12T02:37:39.081 INFO:tasks.rados.rados.0.smithi107.stdout:1591: done (2 left) 2020-06-12T02:37:39.082 INFO:tasks.rados.rados.0.smithi107.stdout:1592: done (1 left) 2020-06-12T02:37:39.082 INFO:tasks.rados.rados.0.smithi107.stdout:1595: done (0 left) 2020-06-12T02:37:39.082 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:37:39.082 INFO:tasks.rados.rados.0.smithi107.stdout:1596: seq_num 1595 ranges {0=319488} 2020-06-12T02:37:39.082 INFO:tasks.rados.rados.0.smithi107.stdout:1596: writing smithi10729407-9 from 0 to 319488 tid 1 2020-06-12T02:37:39.083 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:37:39.083 INFO:tasks.rados.rados.0.smithi107.stdout:1597: seq_num 1596 ranges {2088960=335872} 2020-06-12T02:37:39.083 INFO:tasks.rados.rados.0.smithi107.stdout:1597: writing smithi10729407-14 from 2088960 to 2424832 tid 1 2020-06-12T02:37:39.083 INFO:tasks.rados.rados.0.smithi107.stdout:1598: delete oid 3 current snap is 0 2020-06-12T02:37:39.083 INFO:tasks.rados.rados.0.smithi107.stdout:1596: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:39.083 INFO:tasks.rados.rados.0.smithi107.stdout:1596: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:39.084 INFO:tasks.rados.rados.0.smithi107.stdout:1596: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:39.084 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 280 (ObjNum 1595 snap 0 seq_num 1595) dirty exists 2020-06-12T02:37:39.084 INFO:tasks.rados.rados.0.smithi107.stdout:1596: left oid 9 (ObjNum 1595 snap 0 seq_num 1595) 2020-06-12T02:37:39.084 INFO:tasks.rados.rados.0.smithi107.stdout:1596: done (2 left) 2020-06-12T02:37:39.084 INFO:tasks.rados.rados.0.smithi107.stdout:1598: done (1 left) 2020-06-12T02:37:39.085 INFO:tasks.rados.rados.0.smithi107.stdout:1599: delete oid 17 current snap is 0 2020-06-12T02:37:39.085 INFO:tasks.rados.rados.0.smithi107.stdout:1599: done (1 left) 2020-06-12T02:37:39.085 INFO:tasks.rados.rados.0.smithi107.stdout:1600: delete oid 17 current snap is 0 2020-06-12T02:37:39.085 INFO:tasks.rados.rados.0.smithi107.stdout:1597: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:39.085 INFO:tasks.rados.rados.0.smithi107.stdout:1597: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:39.085 INFO:tasks.rados.rados.0.smithi107.stdout:1600: done (1 left) 2020-06-12T02:37:39.086 INFO:tasks.rados.rados.0.smithi107.stdout:1601: delete oid 10 current snap is 0 2020-06-12T02:37:39.086 INFO:tasks.rados.rados.0.smithi107.stdout:1597: finishing write tid 4 to smithi10729407-14 2020-06-12T02:37:39.086 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 275 (ObjNum 1596 snap 0 seq_num 1596) dirty exists 2020-06-12T02:37:39.086 INFO:tasks.rados.rados.0.smithi107.stdout:1597: left oid 14 (ObjNum 1596 snap 0 seq_num 1596) 2020-06-12T02:37:39.086 INFO:tasks.rados.rados.0.smithi107.stdout:1597: done (1 left) 2020-06-12T02:37:39.087 INFO:tasks.rados.rados.0.smithi107.stdout:1601: done (0 left) 2020-06-12T02:37:39.087 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:37:39.088 INFO:tasks.rados.rados.0.smithi107.stdout:1602: seq_num 1601 ranges {0=417792} 2020-06-12T02:37:39.088 INFO:tasks.rados.rados.0.smithi107.stdout:1602: writing smithi10729407-19 from 0 to 417792 tid 1 2020-06-12T02:37:39.088 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:37:39.088 INFO:tasks.rados.rados.0.smithi107.stdout:1603: seq_num 1602 ranges {2138112=434176} 2020-06-12T02:37:39.088 INFO:tasks.rados.rados.0.smithi107.stdout:1603: writing smithi10729407-18 from 2138112 to 2572288 tid 1 2020-06-12T02:37:39.089 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:37:39.089 INFO:tasks.rados.rados.0.smithi107.stdout:1604: seq_num 1603 ranges {253952=450560} 2020-06-12T02:37:39.089 INFO:tasks.rados.rados.0.smithi107.stdout:1604: writing smithi10729407-5 from 253952 to 704512 tid 1 2020-06-12T02:37:39.089 INFO:tasks.rados.rados.0.smithi107.stdout:1605: delete oid 6 current snap is 0 2020-06-12T02:37:39.089 INFO:tasks.rados.rados.0.smithi107.stdout:1602: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:39.089 INFO:tasks.rados.rados.0.smithi107.stdout:1602: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:39.090 INFO:tasks.rados.rados.0.smithi107.stdout:1602: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:39.090 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 247 (ObjNum 1601 snap 0 seq_num 1601) dirty exists 2020-06-12T02:37:39.090 INFO:tasks.rados.rados.0.smithi107.stdout:1602: left oid 19 (ObjNum 1601 snap 0 seq_num 1601) 2020-06-12T02:37:39.090 INFO:tasks.rados.rados.0.smithi107.stdout:1603: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:39.090 INFO:tasks.rados.rados.0.smithi107.stdout:1603: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:39.091 INFO:tasks.rados.rados.0.smithi107.stdout:1602: done (3 left) 2020-06-12T02:37:39.091 INFO:tasks.rados.rados.0.smithi107.stdout:1605: done (2 left) 2020-06-12T02:37:39.091 INFO:tasks.rados.rados.0.smithi107.stdout:1606: delete oid 9 current snap is 0 2020-06-12T02:37:39.092 INFO:tasks.rados.rados.0.smithi107.stdout:1603: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:39.092 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 282 (ObjNum 1602 snap 0 seq_num 1602) dirty exists 2020-06-12T02:37:39.092 INFO:tasks.rados.rados.0.smithi107.stdout:1603: left oid 18 (ObjNum 1602 snap 0 seq_num 1602) 2020-06-12T02:37:39.092 INFO:tasks.rados.rados.0.smithi107.stdout:1603: done (2 left) 2020-06-12T02:37:39.092 INFO:tasks.rados.rados.0.smithi107.stdout:1606: done (1 left) 2020-06-12T02:37:39.093 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:37:39.093 INFO:tasks.rados.rados.0.smithi107.stdout:1607: seq_num 1606 ranges {0=499712} 2020-06-12T02:37:39.093 INFO:tasks.rados.rados.0.smithi107.stdout:1607: writing smithi10729407-10 from 0 to 499712 tid 1 2020-06-12T02:37:39.093 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:37:39.093 INFO:tasks.rados.rados.0.smithi107.stdout:1608: seq_num 1607 ranges {2572288=516096} 2020-06-12T02:37:39.093 INFO:tasks.rados.rados.0.smithi107.stdout:1608: writing smithi10729407-18 from 2572288 to 3088384 tid 1 2020-06-12T02:37:39.094 INFO:tasks.rados.rados.0.smithi107.stdout:1609: delete oid 6 current snap is 0 2020-06-12T02:37:39.094 INFO:tasks.rados.rados.0.smithi107.stdout:1604: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:39.094 INFO:tasks.rados.rados.0.smithi107.stdout:1604: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:39.094 INFO:tasks.rados.rados.0.smithi107.stdout:1604: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:39.094 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 142 (ObjNum 1603 snap 0 seq_num 1603) dirty exists 2020-06-12T02:37:39.095 INFO:tasks.rados.rados.0.smithi107.stdout:1604: left oid 5 (ObjNum 1603 snap 0 seq_num 1603) 2020-06-12T02:37:39.095 INFO:tasks.rados.rados.0.smithi107.stdout:1607: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:39.095 INFO:tasks.rados.rados.0.smithi107.stdout:1607: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:39.095 INFO:tasks.rados.rados.0.smithi107.stdout:1607: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:39.095 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 151 (ObjNum 1606 snap 0 seq_num 1606) dirty exists 2020-06-12T02:37:39.095 INFO:tasks.rados.rados.0.smithi107.stdout:1607: left oid 10 (ObjNum 1606 snap 0 seq_num 1606) 2020-06-12T02:37:39.096 INFO:tasks.rados.rados.0.smithi107.stdout:1604: done (3 left) 2020-06-12T02:37:39.096 INFO:tasks.rados.rados.0.smithi107.stdout:1607: done (2 left) 2020-06-12T02:37:39.096 INFO:tasks.rados.rados.0.smithi107.stdout:1609: done (1 left) 2020-06-12T02:37:39.096 INFO:tasks.rados.rados.0.smithi107.stdout:1610: delete oid 12 current snap is 0 2020-06-12T02:37:39.096 INFO:tasks.rados.rados.0.smithi107.stdout:1610: done (1 left) 2020-06-12T02:37:39.096 INFO:tasks.rados.rados.0.smithi107.stdout:1611: delete oid 15 current snap is 0 2020-06-12T02:37:39.097 INFO:tasks.rados.rados.0.smithi107.stdout:1608: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:39.097 INFO:tasks.rados.rados.0.smithi107.stdout:1611: done (1 left) 2020-06-12T02:37:39.097 INFO:tasks.rados.rados.0.smithi107.stdout:1612: delete oid 8 current snap is 0 2020-06-12T02:37:39.097 INFO:tasks.rados.rados.0.smithi107.stdout:1608: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:39.097 INFO:tasks.rados.rados.0.smithi107.stdout:1608: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:39.098 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 285 (ObjNum 1607 snap 0 seq_num 1607) dirty exists 2020-06-12T02:37:39.098 INFO:tasks.rados.rados.0.smithi107.stdout:1608: left oid 18 (ObjNum 1607 snap 0 seq_num 1607) 2020-06-12T02:37:39.098 INFO:tasks.rados.rados.0.smithi107.stdout:1608: done (1 left) 2020-06-12T02:37:39.098 INFO:tasks.rados.rados.0.smithi107.stdout:1612: done (0 left) 2020-06-12T02:37:39.098 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:37:39.098 INFO:tasks.rados.rados.0.smithi107.stdout:1613: seq_num 1612 ranges {0=606208} 2020-06-12T02:37:39.099 INFO:tasks.rados.rados.0.smithi107.stdout:1613: writing smithi10729407-2 from 0 to 606208 tid 1 2020-06-12T02:37:39.099 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:37:39.099 INFO:tasks.rados.rados.0.smithi107.stdout:1614: seq_num 1613 ranges {0=622592} 2020-06-12T02:37:39.099 INFO:tasks.rados.rados.0.smithi107.stdout:1614: writing smithi10729407-11 from 0 to 622592 tid 1 2020-06-12T02:37:39.099 INFO:tasks.rados.rados.0.smithi107.stdout:1615: delete oid 7 current snap is 0 2020-06-12T02:37:39.100 INFO:tasks.rados.rados.0.smithi107.stdout:1613: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:39.100 INFO:tasks.rados.rados.0.smithi107.stdout:1613: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:39.100 INFO:tasks.rados.rados.0.smithi107.stdout:1613: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:39.100 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 152 (ObjNum 1612 snap 0 seq_num 1612) dirty exists 2020-06-12T02:37:39.100 INFO:tasks.rados.rados.0.smithi107.stdout:1613: left oid 2 (ObjNum 1612 snap 0 seq_num 1612) 2020-06-12T02:37:39.100 INFO:tasks.rados.rados.0.smithi107.stdout:1613: done (2 left) 2020-06-12T02:37:39.101 INFO:tasks.rados.rados.0.smithi107.stdout:1615: done (1 left) 2020-06-12T02:37:39.101 INFO:tasks.rados.rados.0.smithi107.stdout:1616: delete oid 17 current snap is 0 2020-06-12T02:37:39.101 INFO:tasks.rados.rados.0.smithi107.stdout:1616: done (1 left) 2020-06-12T02:37:39.101 INFO:tasks.rados.rados.0.smithi107.stdout:1617: delete oid 16 current snap is 0 2020-06-12T02:37:39.101 INFO:tasks.rados.rados.0.smithi107.stdout:1614: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:39.102 INFO:tasks.rados.rados.0.smithi107.stdout:1617: done (1 left) 2020-06-12T02:37:39.102 INFO:tasks.rados.rados.0.smithi107.stdout:1618: delete oid 17 current snap is 0 2020-06-12T02:37:39.102 INFO:tasks.rados.rados.0.smithi107.stdout:1614: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:39.102 INFO:tasks.rados.rados.0.smithi107.stdout:1614: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:39.102 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 277 (ObjNum 1613 snap 0 seq_num 1613) dirty exists 2020-06-12T02:37:39.102 INFO:tasks.rados.rados.0.smithi107.stdout:1614: left oid 11 (ObjNum 1613 snap 0 seq_num 1613) 2020-06-12T02:37:39.103 INFO:tasks.rados.rados.0.smithi107.stdout:1614: done (1 left) 2020-06-12T02:37:39.103 INFO:tasks.rados.rados.0.smithi107.stdout:1618: done (0 left) 2020-06-12T02:37:39.103 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:37:39.103 INFO:tasks.rados.rados.0.smithi107.stdout:1619: seq_num 1618 ranges {0=704512} 2020-06-12T02:37:39.103 INFO:tasks.rados.rados.0.smithi107.stdout:1619: writing smithi10729407-3 from 0 to 704512 tid 1 2020-06-12T02:37:39.103 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:37:39.104 INFO:tasks.rados.rados.0.smithi107.stdout:1620: seq_num 1619 ranges {0=720896} 2020-06-12T02:37:39.104 INFO:tasks.rados.rados.0.smithi107.stdout:1620: writing smithi10729407-9 from 0 to 720896 tid 1 2020-06-12T02:37:39.104 INFO:tasks.rados.rados.0.smithi107.stdout:1621: delete oid 2 current snap is 0 2020-06-12T02:37:39.104 INFO:tasks.rados.rados.0.smithi107.stdout:1619: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:39.104 INFO:tasks.rados.rados.0.smithi107.stdout:1619: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:39.105 INFO:tasks.rados.rados.0.smithi107.stdout:1621: done (2 left) 2020-06-12T02:37:39.105 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:37:39.105 INFO:tasks.rados.rados.0.smithi107.stdout:1622: seq_num 1621 ranges {0=753664} 2020-06-12T02:37:39.105 INFO:tasks.rados.rados.0.smithi107.stdout:1622: writing smithi10729407-12 from 0 to 753664 tid 1 2020-06-12T02:37:39.105 INFO:tasks.rados.rados.0.smithi107.stdout:1623: delete oid 10 current snap is 0 2020-06-12T02:37:39.106 INFO:tasks.rados.rados.0.smithi107.stdout:1619: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:39.106 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 289 (ObjNum 1618 snap 0 seq_num 1618) dirty exists 2020-06-12T02:37:39.106 INFO:tasks.rados.rados.0.smithi107.stdout:1619: left oid 3 (ObjNum 1618 snap 0 seq_num 1618) 2020-06-12T02:37:39.106 INFO:tasks.rados.rados.0.smithi107.stdout:1620: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:39.106 INFO:tasks.rados.rados.0.smithi107.stdout:1620: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:39.106 INFO:tasks.rados.rados.0.smithi107.stdout:1620: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:39.107 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 287 (ObjNum 1619 snap 0 seq_num 1619) dirty exists 2020-06-12T02:37:39.107 INFO:tasks.rados.rados.0.smithi107.stdout:1620: left oid 9 (ObjNum 1619 snap 0 seq_num 1619) 2020-06-12T02:37:39.107 INFO:tasks.rados.rados.0.smithi107.stdout:1619: done (3 left) 2020-06-12T02:37:39.107 INFO:tasks.rados.rados.0.smithi107.stdout:1620: done (2 left) 2020-06-12T02:37:39.107 INFO:tasks.rados.rados.0.smithi107.stdout:1623: done (1 left) 2020-06-12T02:37:39.107 INFO:tasks.rados.rados.0.smithi107.stdout:1624: delete oid 2 current snap is 0 2020-06-12T02:37:39.108 INFO:tasks.rados.rados.0.smithi107.stdout:1624: done (1 left) 2020-06-12T02:37:39.108 INFO:tasks.rados.rados.0.smithi107.stdout:1625: delete oid 5 current snap is 0 2020-06-12T02:37:39.108 INFO:tasks.rados.rados.0.smithi107.stdout:1625: done (1 left) 2020-06-12T02:37:39.108 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:37:39.108 INFO:tasks.rados.rados.0.smithi107.stdout:1626: seq_num 1625 ranges {622592=417792,1040384=401408} 2020-06-12T02:37:39.109 INFO:tasks.rados.rados.0.smithi107.stdout:1626: writing smithi10729407-11 from 622592 to 1040384 tid 1 2020-06-12T02:37:39.109 INFO:tasks.rados.rados.0.smithi107.stdout:1626: writing smithi10729407-11 from 1040384 to 1441792 tid 2 2020-06-12T02:37:39.109 INFO:tasks.rados.rados.0.smithi107.stdout:1627: delete oid 4 current snap is 0 2020-06-12T02:37:39.109 INFO:tasks.rados.rados.0.smithi107.stdout:1622: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:39.109 INFO:tasks.rados.rados.0.smithi107.stdout:1622: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:39.109 INFO:tasks.rados.rados.0.smithi107.stdout:1622: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:39.110 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 250 (ObjNum 1621 snap 0 seq_num 1621) dirty exists 2020-06-12T02:37:39.110 INFO:tasks.rados.rados.0.smithi107.stdout:1622: left oid 12 (ObjNum 1621 snap 0 seq_num 1621) 2020-06-12T02:37:39.110 INFO:tasks.rados.rados.0.smithi107.stdout:1626: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:39.110 INFO:tasks.rados.rados.0.smithi107.stdout:1622: done (2 left) 2020-06-12T02:37:39.110 INFO:tasks.rados.rados.0.smithi107.stdout:1627: done (1 left) 2020-06-12T02:37:39.111 INFO:tasks.rados.rados.0.smithi107.stdout:1628: delete oid 10 current snap is 0 2020-06-12T02:37:39.111 INFO:tasks.rados.rados.0.smithi107.stdout:1628: done (1 left) 2020-06-12T02:37:39.111 INFO:tasks.rados.rados.0.smithi107.stdout:1629: delete oid 9 current snap is 0 2020-06-12T02:37:39.111 INFO:tasks.rados.rados.0.smithi107.stdout:1626: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:39.111 INFO:tasks.rados.rados.0.smithi107.stdout:1626: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:39.111 INFO:tasks.rados.rados.0.smithi107.stdout:1626: finishing write tid 5 to smithi10729407-11 2020-06-12T02:37:39.112 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 280 (ObjNum 1625 snap 0 seq_num 1625) dirty exists 2020-06-12T02:37:39.112 INFO:tasks.rados.rados.0.smithi107.stdout:1626: left oid 11 (ObjNum 1625 snap 0 seq_num 1625) 2020-06-12T02:37:39.112 INFO:tasks.rados.rados.0.smithi107.stdout:1626: done (1 left) 2020-06-12T02:37:39.112 INFO:tasks.rados.rados.0.smithi107.stdout:1629: done (0 left) 2020-06-12T02:37:39.112 INFO:tasks.rados.rados.0.smithi107.stdout:1630: delete oid 1 current snap is 0 2020-06-12T02:37:39.112 INFO:tasks.rados.rados.0.smithi107.stdout:1630: done (0 left) 2020-06-12T02:37:39.113 INFO:tasks.rados.rados.0.smithi107.stdout:1631: delete oid 12 current snap is 0 2020-06-12T02:37:39.113 INFO:tasks.rados.rados.0.smithi107.stdout:1631: done (0 left) 2020-06-12T02:37:39.113 INFO:tasks.rados.rados.0.smithi107.stdout:1632: delete oid 14 current snap is 0 2020-06-12T02:37:39.113 INFO:tasks.rados.rados.0.smithi107.stdout:1632: done (0 left) 2020-06-12T02:37:39.113 INFO:tasks.rados.rados.0.smithi107.stdout:1633: delete oid 15 current snap is 0 2020-06-12T02:37:39.114 INFO:tasks.rados.rados.0.smithi107.stdout:1633: done (0 left) 2020-06-12T02:37:39.114 INFO:tasks.rados.rados.0.smithi107.stdout:1634: delete oid 2 current snap is 0 2020-06-12T02:37:39.114 INFO:tasks.rados.rados.0.smithi107.stdout:1634: done (0 left) 2020-06-12T02:37:39.114 INFO:tasks.rados.rados.0.smithi107.stdout:1635: delete oid 15 current snap is 0 2020-06-12T02:37:39.114 INFO:tasks.rados.rados.0.smithi107.stdout:1635: done (0 left) 2020-06-12T02:37:39.114 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:37:39.115 INFO:tasks.rados.rados.0.smithi107.stdout:1636: seq_num 1635 ranges {0=589824,589824=401408} 2020-06-12T02:37:39.115 INFO:tasks.rados.rados.0.smithi107.stdout:1636: writing smithi10729407-17 from 0 to 589824 tid 1 2020-06-12T02:37:39.115 INFO:tasks.rados.rados.0.smithi107.stdout:1636: writing smithi10729407-17 from 589824 to 991232 tid 2 2020-06-12T02:37:39.115 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:37:39.115 INFO:tasks.rados.rados.0.smithi107.stdout:1637: seq_num 1636 ranges {0=606208,606208=401408} 2020-06-12T02:37:39.116 INFO:tasks.rados.rados.0.smithi107.stdout:1637: writing smithi10729407-14 from 0 to 606208 tid 1 2020-06-12T02:37:39.116 INFO:tasks.rados.rados.0.smithi107.stdout:1637: writing smithi10729407-14 from 606208 to 1007616 tid 2 2020-06-12T02:37:39.116 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:37:39.116 INFO:tasks.rados.rados.0.smithi107.stdout:1638: seq_num 1637 ranges {3088384=622592,3710976=401408} 2020-06-12T02:37:39.116 INFO:tasks.rados.rados.0.smithi107.stdout:1638: writing smithi10729407-18 from 3088384 to 3710976 tid 1 2020-06-12T02:37:39.116 INFO:tasks.rados.rados.0.smithi107.stdout:1638: writing smithi10729407-18 from 3710976 to 4112384 tid 2 2020-06-12T02:37:39.117 INFO:tasks.rados.rados.0.smithi107.stdout:1639: delete oid 6 current snap is 0 2020-06-12T02:37:39.118 INFO:tasks.rados.rados.0.smithi107.stdout:1636: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:39.118 INFO:tasks.rados.rados.0.smithi107.stdout:1636: finishing write tid 2 to smithi10729407-17 2020-06-12T02:37:39.118 INFO:tasks.rados.rados.0.smithi107.stdout:1636: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:39.118 INFO:tasks.rados.rados.0.smithi107.stdout:1636: finishing write tid 5 to smithi10729407-17 2020-06-12T02:37:39.118 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 314 (ObjNum 1635 snap 0 seq_num 1635) dirty exists 2020-06-12T02:37:39.119 INFO:tasks.rados.rados.0.smithi107.stdout:1636: left oid 17 (ObjNum 1635 snap 0 seq_num 1635) 2020-06-12T02:37:39.119 INFO:tasks.rados.rados.0.smithi107.stdout:1637: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:39.119 INFO:tasks.rados.rados.0.smithi107.stdout:1637: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:39.119 INFO:tasks.rados.rados.0.smithi107.stdout:1637: finishing write tid 4 to smithi10729407-14 2020-06-12T02:37:39.120 INFO:tasks.rados.rados.0.smithi107.stdout:1637: finishing write tid 5 to smithi10729407-14 2020-06-12T02:37:39.120 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 284 (ObjNum 1636 snap 0 seq_num 1636) dirty exists 2020-06-12T02:37:39.120 INFO:tasks.rados.rados.0.smithi107.stdout:1637: left oid 14 (ObjNum 1636 snap 0 seq_num 1636) 2020-06-12T02:37:39.120 INFO:tasks.rados.rados.0.smithi107.stdout:1636: done (3 left) 2020-06-12T02:37:39.121 INFO:tasks.rados.rados.0.smithi107.stdout:1637: done (2 left) 2020-06-12T02:37:39.121 INFO:tasks.rados.rados.0.smithi107.stdout:1639: done (1 left) 2020-06-12T02:37:39.121 INFO:tasks.rados.rados.0.smithi107.stdout:1640: delete oid 8 current snap is 0 2020-06-12T02:37:39.121 INFO:tasks.rados.rados.0.smithi107.stdout:1640: done (1 left) 2020-06-12T02:37:39.121 INFO:tasks.rados.rados.0.smithi107.stdout:1641: delete oid 11 current snap is 0 2020-06-12T02:37:39.122 INFO:tasks.rados.rados.0.smithi107.stdout:1638: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:39.122 INFO:tasks.rados.rados.0.smithi107.stdout:1638: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:39.122 INFO:tasks.rados.rados.0.smithi107.stdout:1641: done (1 left) 2020-06-12T02:37:39.122 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:37:39.122 INFO:tasks.rados.rados.0.smithi107.stdout:1642: seq_num 1641 ranges {0=688128,688128=401408} 2020-06-12T02:37:39.123 INFO:tasks.rados.rados.0.smithi107.stdout:1642: writing smithi10729407-4 from 0 to 688128 tid 1 2020-06-12T02:37:39.123 INFO:tasks.rados.rados.0.smithi107.stdout:1642: writing smithi10729407-4 from 688128 to 1089536 tid 2 2020-06-12T02:37:39.123 INFO:tasks.rados.rados.0.smithi107.stdout:1643: delete oid 19 current snap is 0 2020-06-12T02:37:39.123 INFO:tasks.rados.rados.0.smithi107.stdout:1638: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:39.123 INFO:tasks.rados.rados.0.smithi107.stdout:1638: finishing write tid 5 to smithi10729407-18 2020-06-12T02:37:39.124 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 291 (ObjNum 1637 snap 0 seq_num 1637) dirty exists 2020-06-12T02:37:39.124 INFO:tasks.rados.rados.0.smithi107.stdout:1638: left oid 18 (ObjNum 1637 snap 0 seq_num 1637) 2020-06-12T02:37:39.124 INFO:tasks.rados.rados.0.smithi107.stdout:1638: done (2 left) 2020-06-12T02:37:39.124 INFO:tasks.rados.rados.0.smithi107.stdout:1643: done (1 left) 2020-06-12T02:37:39.124 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:37:39.125 INFO:tasks.rados.rados.0.smithi107.stdout:1644: seq_num 1643 ranges {0=720896,720896=401408} 2020-06-12T02:37:39.125 INFO:tasks.rados.rados.0.smithi107.stdout:1644: writing smithi10729407-8 from 0 to 720896 tid 1 2020-06-12T02:37:39.125 INFO:tasks.rados.rados.0.smithi107.stdout:1644: writing smithi10729407-8 from 720896 to 1122304 tid 2 2020-06-12T02:37:39.125 INFO:tasks.rados.rados.0.smithi107.stdout:1645: delete oid 7 current snap is 0 2020-06-12T02:37:39.126 INFO:tasks.rados.rados.0.smithi107.stdout:1642: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:39.126 INFO:tasks.rados.rados.0.smithi107.stdout:1642: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:39.126 INFO:tasks.rados.rados.0.smithi107.stdout:1642: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:39.126 INFO:tasks.rados.rados.0.smithi107.stdout:1642: finishing write tid 5 to smithi10729407-4 2020-06-12T02:37:39.126 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 317 (ObjNum 1641 snap 0 seq_num 1641) dirty exists 2020-06-12T02:37:39.126 INFO:tasks.rados.rados.0.smithi107.stdout:1642: left oid 4 (ObjNum 1641 snap 0 seq_num 1641) 2020-06-12T02:37:39.127 INFO:tasks.rados.rados.0.smithi107.stdout:1642: done (2 left) 2020-06-12T02:37:39.127 INFO:tasks.rados.rados.0.smithi107.stdout:1645: done (1 left) 2020-06-12T02:37:39.127 INFO:tasks.rados.rados.0.smithi107.stdout:1646: delete oid 17 current snap is 0 2020-06-12T02:37:39.127 INFO:tasks.rados.rados.0.smithi107.stdout:1646: done (1 left) 2020-06-12T02:37:39.127 INFO:tasks.rados.rados.0.smithi107.stdout:1647: delete oid 16 current snap is 0 2020-06-12T02:37:39.128 INFO:tasks.rados.rados.0.smithi107.stdout:1644: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:39.128 INFO:tasks.rados.rados.0.smithi107.stdout:1647: done (1 left) 2020-06-12T02:37:39.128 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:37:39.128 INFO:tasks.rados.rados.0.smithi107.stdout:1648: seq_num 1647 ranges {0=794624,794624=401408} 2020-06-12T02:37:39.128 INFO:tasks.rados.rados.0.smithi107.stdout:1648: writing smithi10729407-5 from 0 to 794624 tid 1 2020-06-12T02:37:39.129 INFO:tasks.rados.rados.0.smithi107.stdout:1648: writing smithi10729407-5 from 794624 to 1196032 tid 2 2020-06-12T02:37:39.129 INFO:tasks.rados.rados.0.smithi107.stdout:1649: delete oid 11 current snap is 0 2020-06-12T02:37:39.129 INFO:tasks.rados.rados.0.smithi107.stdout:1644: finishing write tid 2 to smithi10729407-8 2020-06-12T02:37:39.129 INFO:tasks.rados.rados.0.smithi107.stdout:1644: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:39.129 INFO:tasks.rados.rados.0.smithi107.stdout:1644: finishing write tid 5 to smithi10729407-8 2020-06-12T02:37:39.129 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 281 (ObjNum 1643 snap 0 seq_num 1643) dirty exists 2020-06-12T02:37:39.130 INFO:tasks.rados.rados.0.smithi107.stdout:1644: left oid 8 (ObjNum 1643 snap 0 seq_num 1643) 2020-06-12T02:37:39.130 INFO:tasks.rados.rados.0.smithi107.stdout:1644: done (2 left) 2020-06-12T02:37:39.130 INFO:tasks.rados.rados.0.smithi107.stdout:1649: done (1 left) 2020-06-12T02:37:39.130 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:37:39.130 INFO:tasks.rados.rados.0.smithi107.stdout:1650: seq_num 1649 ranges {1007616=425984,1433600=802816} 2020-06-12T02:37:39.131 INFO:tasks.rados.rados.0.smithi107.stdout:1650: writing smithi10729407-14 from 1007616 to 1433600 tid 1 2020-06-12T02:37:39.131 INFO:tasks.rados.rados.0.smithi107.stdout:1650: writing smithi10729407-14 from 1433600 to 2236416 tid 2 2020-06-12T02:37:39.131 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:37:39.131 INFO:tasks.rados.rados.0.smithi107.stdout:1651: seq_num 1650 ranges {0=442368,442368=737280,1179648=65536} 2020-06-12T02:37:39.131 INFO:tasks.rados.rados.0.smithi107.stdout:1651: writing smithi10729407-1 from 0 to 442368 tid 1 2020-06-12T02:37:39.131 INFO:tasks.rados.rados.0.smithi107.stdout:1651: writing smithi10729407-1 from 442368 to 1179648 tid 2 2020-06-12T02:37:39.132 INFO:tasks.rados.rados.0.smithi107.stdout:1651: writing smithi10729407-1 from 1179648 to 1245184 tid 3 2020-06-12T02:37:39.132 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:37:39.132 INFO:tasks.rados.rados.0.smithi107.stdout:1652: seq_num 1651 ranges {0=458752,458752=622592,1081344=180224} 2020-06-12T02:37:39.132 INFO:tasks.rados.rados.0.smithi107.stdout:1652: writing smithi10729407-11 from 0 to 458752 tid 1 2020-06-12T02:37:39.132 INFO:tasks.rados.rados.0.smithi107.stdout:1652: writing smithi10729407-11 from 458752 to 1081344 tid 2 2020-06-12T02:37:39.133 INFO:tasks.rados.rados.0.smithi107.stdout:1652: writing smithi10729407-11 from 1081344 to 1261568 tid 3 2020-06-12T02:37:39.133 INFO:tasks.rados.rados.0.smithi107.stdout:1653: delete oid 15 current snap is 0 2020-06-12T02:37:39.133 INFO:tasks.rados.rados.0.smithi107.stdout:1648: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:39.133 INFO:tasks.rados.rados.0.smithi107.stdout:1648: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:39.133 INFO:tasks.rados.rados.0.smithi107.stdout:1648: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:39.133 INFO:tasks.rados.rados.0.smithi107.stdout:1648: finishing write tid 5 to smithi10729407-5 2020-06-12T02:37:39.139 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 146 (ObjNum 1647 snap 0 seq_num 1647) dirty exists 2020-06-12T02:37:39.139 INFO:tasks.rados.rados.0.smithi107.stdout:1648: left oid 5 (ObjNum 1647 snap 0 seq_num 1647) 2020-06-12T02:37:39.140 INFO:tasks.rados.rados.0.smithi107.stdout:1650: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:39.140 INFO:tasks.rados.rados.0.smithi107.stdout:1650: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:39.140 INFO:tasks.rados.rados.0.smithi107.stdout:1650: finishing write tid 4 to smithi10729407-14 2020-06-12T02:37:39.141 INFO:tasks.rados.rados.0.smithi107.stdout:1651: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:39.141 INFO:tasks.rados.rados.0.smithi107.stdout:1650: finishing write tid 5 to smithi10729407-14 2020-06-12T02:37:39.141 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 289 (ObjNum 1649 snap 0 seq_num 1649) dirty exists 2020-06-12T02:37:39.142 INFO:tasks.rados.rados.0.smithi107.stdout:1650: left oid 14 (ObjNum 1649 snap 0 seq_num 1649) 2020-06-12T02:37:39.142 INFO:tasks.rados.rados.0.smithi107.stdout:1651: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:39.142 INFO:tasks.rados.rados.0.smithi107.stdout:1651: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:39.142 INFO:tasks.rados.rados.0.smithi107.stdout:1648: done (4 left) 2020-06-12T02:37:39.143 INFO:tasks.rados.rados.0.smithi107.stdout:1650: done (3 left) 2020-06-12T02:37:39.143 INFO:tasks.rados.rados.0.smithi107.stdout:1653: done (2 left) 2020-06-12T02:37:39.143 INFO:tasks.rados.rados.0.smithi107.stdout:1654: delete oid 4 current snap is 0 2020-06-12T02:37:39.143 INFO:tasks.rados.rados.0.smithi107.stdout:1651: finishing write tid 5 to smithi10729407-1 2020-06-12T02:37:39.143 INFO:tasks.rados.rados.0.smithi107.stdout:1652: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:39.144 INFO:tasks.rados.rados.0.smithi107.stdout:1654: done (2 left) 2020-06-12T02:37:39.144 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:37:39.144 INFO:tasks.rados.rados.0.smithi107.stdout:1655: seq_num 1654 ranges {0=507904,507904=671744,1179648=131072} 2020-06-12T02:37:39.144 INFO:tasks.rados.rados.0.smithi107.stdout:1655: writing smithi10729407-16 from 0 to 507904 tid 1 2020-06-12T02:37:39.145 INFO:tasks.rados.rados.0.smithi107.stdout:1655: writing smithi10729407-16 from 507904 to 1179648 tid 2 2020-06-12T02:37:39.145 INFO:tasks.rados.rados.0.smithi107.stdout:1655: writing smithi10729407-16 from 1179648 to 1310720 tid 3 2020-06-12T02:37:39.145 INFO:tasks.rados.rados.0.smithi107.stdout:1656: delete oid 4 current snap is 0 2020-06-12T02:37:39.145 INFO:tasks.rados.rados.0.smithi107.stdout:1651: finishing write tid 6 to smithi10729407-1 2020-06-12T02:37:39.146 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 324 (ObjNum 1650 snap 0 seq_num 1650) dirty exists 2020-06-12T02:37:39.146 INFO:tasks.rados.rados.0.smithi107.stdout:1651: left oid 1 (ObjNum 1650 snap 0 seq_num 1650) 2020-06-12T02:37:39.146 INFO:tasks.rados.rados.0.smithi107.stdout:1652: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:39.147 INFO:tasks.rados.rados.0.smithi107.stdout:1652: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:39.147 INFO:tasks.rados.rados.0.smithi107.stdout:1652: finishing write tid 5 to smithi10729407-11 2020-06-12T02:37:39.147 INFO:tasks.rados.rados.0.smithi107.stdout:1652: finishing write tid 6 to smithi10729407-11 2020-06-12T02:37:39.147 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 293 (ObjNum 1651 snap 0 seq_num 1651) dirty exists 2020-06-12T02:37:39.147 INFO:tasks.rados.rados.0.smithi107.stdout:1652: left oid 11 (ObjNum 1651 snap 0 seq_num 1651) 2020-06-12T02:37:39.148 INFO:tasks.rados.rados.0.smithi107.stdout:1655: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:39.148 INFO:tasks.rados.rados.0.smithi107.stdout:1651: done (3 left) 2020-06-12T02:37:39.148 INFO:tasks.rados.rados.0.smithi107.stdout:1652: done (2 left) 2020-06-12T02:37:39.148 INFO:tasks.rados.rados.0.smithi107.stdout:1656: done (1 left) 2020-06-12T02:37:39.148 INFO:tasks.rados.rados.0.smithi107.stdout:1657: delete oid 10 current snap is 0 2020-06-12T02:37:39.149 INFO:tasks.rados.rados.0.smithi107.stdout:1657: done (1 left) 2020-06-12T02:37:39.149 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:37:39.149 INFO:tasks.rados.rados.0.smithi107.stdout:1658: seq_num 1657 ranges {1122304=557056,1679360=729088,2408448=73728} 2020-06-12T02:37:39.149 INFO:tasks.rados.rados.0.smithi107.stdout:1658: writing smithi10729407-8 from 1122304 to 1679360 tid 1 2020-06-12T02:37:39.149 INFO:tasks.rados.rados.0.smithi107.stdout:1658: writing smithi10729407-8 from 1679360 to 2408448 tid 2 2020-06-12T02:37:39.150 INFO:tasks.rados.rados.0.smithi107.stdout:1658: writing smithi10729407-8 from 2408448 to 2482176 tid 3 2020-06-12T02:37:39.150 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:37:39.150 INFO:tasks.rados.rados.0.smithi107.stdout:1659: seq_num 1658 ranges {0=573440,573440=655360,1228800=147456} 2020-06-12T02:37:39.150 INFO:tasks.rados.rados.0.smithi107.stdout:1659: writing smithi10729407-20 from 0 to 573440 tid 1 2020-06-12T02:37:39.151 INFO:tasks.rados.rados.0.smithi107.stdout:1659: writing smithi10729407-20 from 573440 to 1228800 tid 2 2020-06-12T02:37:39.151 INFO:tasks.rados.rados.0.smithi107.stdout:1659: writing smithi10729407-20 from 1228800 to 1376256 tid 3 2020-06-12T02:37:39.151 INFO:tasks.rados.rados.0.smithi107.stdout:1660: delete oid 15 current snap is 0 2020-06-12T02:37:39.151 INFO:tasks.rados.rados.0.smithi107.stdout:1655: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:39.151 INFO:tasks.rados.rados.0.smithi107.stdout:1655: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:39.152 INFO:tasks.rados.rados.0.smithi107.stdout:1655: finishing write tid 5 to smithi10729407-16 2020-06-12T02:37:39.152 INFO:tasks.rados.rados.0.smithi107.stdout:1655: finishing write tid 6 to smithi10729407-16 2020-06-12T02:37:39.152 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 294 (ObjNum 1654 snap 0 seq_num 1654) dirty exists 2020-06-12T02:37:39.152 INFO:tasks.rados.rados.0.smithi107.stdout:1655: left oid 16 (ObjNum 1654 snap 0 seq_num 1654) 2020-06-12T02:37:39.153 INFO:tasks.rados.rados.0.smithi107.stdout:1658: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:39.153 INFO:tasks.rados.rados.0.smithi107.stdout:1658: finishing write tid 2 to smithi10729407-8 2020-06-12T02:37:39.153 INFO:tasks.rados.rados.0.smithi107.stdout:1658: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:39.153 INFO:tasks.rados.rados.0.smithi107.stdout:1658: finishing write tid 5 to smithi10729407-8 2020-06-12T02:37:39.154 INFO:tasks.rados.rados.0.smithi107.stdout:1658: finishing write tid 6 to smithi10729407-8 2020-06-12T02:37:39.154 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 286 (ObjNum 1657 snap 0 seq_num 1657) dirty exists 2020-06-12T02:37:39.154 INFO:tasks.rados.rados.0.smithi107.stdout:1658: left oid 8 (ObjNum 1657 snap 0 seq_num 1657) 2020-06-12T02:37:39.154 INFO:tasks.rados.rados.0.smithi107.stdout:1659: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:39.154 INFO:tasks.rados.rados.0.smithi107.stdout:1655: done (3 left) 2020-06-12T02:37:39.155 INFO:tasks.rados.rados.0.smithi107.stdout:1658: done (2 left) 2020-06-12T02:37:39.155 INFO:tasks.rados.rados.0.smithi107.stdout:1660: done (1 left) 2020-06-12T02:37:39.155 INFO:tasks.rados.rados.0.smithi107.stdout:1661: delete oid 12 current snap is 0 2020-06-12T02:37:39.156 INFO:tasks.rados.rados.0.smithi107.stdout:1659: finishing write tid 2 to smithi10729407-20 2020-06-12T02:37:39.156 INFO:tasks.rados.rados.0.smithi107.stdout:1661: done (1 left) 2020-06-12T02:37:39.157 INFO:tasks.rados.rados.0.smithi107.stdout:1662: delete oid 6 current snap is 0 2020-06-12T02:37:39.157 INFO:tasks.rados.rados.0.smithi107.stdout:1659: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:39.157 INFO:tasks.rados.rados.0.smithi107.stdout:1659: finishing write tid 5 to smithi10729407-20 2020-06-12T02:37:39.158 INFO:tasks.rados.rados.0.smithi107.stdout:1659: finishing write tid 6 to smithi10729407-20 2020-06-12T02:37:39.158 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 120 (ObjNum 1658 snap 0 seq_num 1658) dirty exists 2020-06-12T02:37:39.158 INFO:tasks.rados.rados.0.smithi107.stdout:1659: left oid 20 (ObjNum 1658 snap 0 seq_num 1658) 2020-06-12T02:37:39.159 INFO:tasks.rados.rados.0.smithi107.stdout:1659: done (1 left) 2020-06-12T02:37:39.159 INFO:tasks.rados.rados.0.smithi107.stdout:1662: done (0 left) 2020-06-12T02:37:39.159 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:37:39.160 INFO:tasks.rados.rados.0.smithi107.stdout:1663: seq_num 1662 ranges {0=638976,638976=598016,1236992=204800} 2020-06-12T02:37:39.160 INFO:tasks.rados.rados.0.smithi107.stdout:1663: writing smithi10729407-15 from 0 to 638976 tid 1 2020-06-12T02:37:39.160 INFO:tasks.rados.rados.0.smithi107.stdout:1663: writing smithi10729407-15 from 638976 to 1236992 tid 2 2020-06-12T02:37:39.160 INFO:tasks.rados.rados.0.smithi107.stdout:1663: writing smithi10729407-15 from 1236992 to 1441792 tid 3 2020-06-12T02:37:39.161 INFO:tasks.rados.rados.0.smithi107.stdout:1664: delete oid 19 current snap is 0 2020-06-12T02:37:39.161 INFO:tasks.rados.rados.0.smithi107.stdout:1663: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:39.161 INFO:tasks.rados.rados.0.smithi107.stdout:1664: done (1 left) 2020-06-12T02:37:39.162 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:37:39.162 INFO:tasks.rados.rados.0.smithi107.stdout:1665: seq_num 1664 ranges {0=671744,671744=761856,1433600=40960} 2020-06-12T02:37:39.162 INFO:tasks.rados.rados.0.smithi107.stdout:1665: writing smithi10729407-19 from 0 to 671744 tid 1 2020-06-12T02:37:39.163 INFO:tasks.rados.rados.0.smithi107.stdout:1665: writing smithi10729407-19 from 671744 to 1433600 tid 2 2020-06-12T02:37:39.163 INFO:tasks.rados.rados.0.smithi107.stdout:1665: writing smithi10729407-19 from 1433600 to 1474560 tid 3 2020-06-12T02:37:39.163 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:37:39.164 INFO:tasks.rados.rados.0.smithi107.stdout:1666: seq_num 1665 ranges {1245184=688128,1933312=696320,2629632=106496} 2020-06-12T02:37:39.164 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:39.052+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:39.165 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-06-12T02:37:39.173 INFO:tasks.rados.rados.0.smithi107.stdout:1666: writing smithi10729407-1 from 1245184 to 1933312 tid 1 2020-06-12T02:37:39.173 INFO:tasks.rados.rados.0.smithi107.stdout:1666: writing smithi10729407-1 from 1933312 to 2629632 tid 2 2020-06-12T02:37:39.173 INFO:tasks.rados.rados.0.smithi107.stdout:1666: writing smithi10729407-1 from 2629632 to 2736128 tid 3 2020-06-12T02:37:39.174 INFO:tasks.rados.rados.0.smithi107.stdout:1667: delete oid 18 current snap is 0 2020-06-12T02:37:39.174 INFO:tasks.rados.rados.0.smithi107.stdout:1663: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:39.174 INFO:tasks.rados.rados.0.smithi107.stdout:1663: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:39.175 INFO:tasks.rados.rados.0.smithi107.stdout:1663: finishing write tid 5 to smithi10729407-15 2020-06-12T02:37:39.175 INFO:tasks.rados.rados.0.smithi107.stdout:1663: finishing write tid 6 to smithi10729407-15 2020-06-12T02:37:39.175 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 291 (ObjNum 1662 snap 0 seq_num 1662) dirty exists 2020-06-12T02:37:39.176 INFO:tasks.rados.rados.0.smithi107.stdout:1663: left oid 15 (ObjNum 1662 snap 0 seq_num 1662) 2020-06-12T02:37:39.176 INFO:tasks.rados.rados.0.smithi107.stdout:1665: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:39.176 INFO:tasks.rados.rados.0.smithi107.stdout:1665: finishing write tid 2 to smithi10729407-19 2020-06-12T02:37:39.176 INFO:tasks.rados.rados.0.smithi107.stdout:1665: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:39.177 INFO:tasks.rados.rados.0.smithi107.stdout:1665: finishing write tid 5 to smithi10729407-19 2020-06-12T02:37:39.177 INFO:tasks.rados.rados.0.smithi107.stdout:1665: finishing write tid 6 to smithi10729407-19 2020-06-12T02:37:39.177 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 258 (ObjNum 1664 snap 0 seq_num 1664) dirty exists 2020-06-12T02:37:39.178 INFO:tasks.rados.rados.0.smithi107.stdout:1665: left oid 19 (ObjNum 1664 snap 0 seq_num 1664) 2020-06-12T02:37:39.178 INFO:tasks.rados.rados.0.smithi107.stdout:1666: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:39.178 INFO:tasks.rados.rados.0.smithi107.stdout:1663: done (3 left) 2020-06-12T02:37:39.179 INFO:tasks.rados.rados.0.smithi107.stdout:1665: done (2 left) 2020-06-12T02:37:39.179 INFO:tasks.rados.rados.0.smithi107.stdout:1667: done (1 left) 2020-06-12T02:37:39.179 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:37:39.180 INFO:tasks.rados.rados.0.smithi107.stdout:1668: seq_num 1667 ranges {1261568=729088,1990656=466944,2457600=335872} 2020-06-12T02:37:39.180 INFO:tasks.rados.rados.0.smithi107.stdout:1668: writing smithi10729407-11 from 1261568 to 1990656 tid 1 2020-06-12T02:37:39.180 INFO:tasks.rados.rados.0.smithi107.stdout:1668: writing smithi10729407-11 from 1990656 to 2457600 tid 2 2020-06-12T02:37:39.181 INFO:tasks.rados.rados.0.smithi107.stdout:1668: writing smithi10729407-11 from 2457600 to 2793472 tid 3 2020-06-12T02:37:39.181 INFO:tasks.rados.rados.0.smithi107.stdout:1669: delete oid 17 current snap is 0 2020-06-12T02:37:39.181 INFO:tasks.rados.rados.0.smithi107.stdout:1666: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:39.182 INFO:tasks.rados.rados.0.smithi107.stdout:1666: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:39.182 INFO:tasks.rados.rados.0.smithi107.stdout:1666: finishing write tid 5 to smithi10729407-1 2020-06-12T02:37:39.182 INFO:tasks.rados.rados.0.smithi107.stdout:1666: finishing write tid 6 to smithi10729407-1 2020-06-12T02:37:39.183 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 329 (ObjNum 1665 snap 0 seq_num 1665) dirty exists 2020-06-12T02:37:39.183 INFO:tasks.rados.rados.0.smithi107.stdout:1666: left oid 1 (ObjNum 1665 snap 0 seq_num 1665) 2020-06-12T02:37:39.183 INFO:tasks.rados.rados.0.smithi107.stdout:1666: done (2 left) 2020-06-12T02:37:39.184 INFO:tasks.rados.rados.0.smithi107.stdout:1669: done (1 left) 2020-06-12T02:37:39.184 INFO:tasks.rados.rados.0.smithi107.stdout:1670: delete oid 4 current snap is 0 2020-06-12T02:37:39.184 INFO:tasks.rados.rados.0.smithi107.stdout:1668: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:39.185 INFO:tasks.rados.rados.0.smithi107.stdout:1670: done (1 left) 2020-06-12T02:37:39.185 INFO:tasks.rados.rados.0.smithi107.stdout:1671: delete oid 10 current snap is 0 2020-06-12T02:37:39.185 INFO:tasks.rados.rados.0.smithi107.stdout:1671: done (1 left) 2020-06-12T02:37:39.186 INFO:tasks.rados.rados.0.smithi107.stdout:1672: delete oid 5 current snap is 0 2020-06-12T02:37:39.186 INFO:tasks.rados.rados.0.smithi107.stdout:1672: done (1 left) 2020-06-12T02:37:39.187 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:37:39.187 INFO:tasks.rados.rados.0.smithi107.stdout:1673: seq_num 1672 ranges {1474560=409600,1884160=688128,2572288=516096} 2020-06-12T02:37:39.187 INFO:tasks.rados.rados.0.smithi107.stdout:1673: writing smithi10729407-19 from 1474560 to 1884160 tid 1 2020-06-12T02:37:39.188 INFO:tasks.rados.rados.0.smithi107.stdout:1673: writing smithi10729407-19 from 1884160 to 2572288 tid 2 2020-06-12T02:37:39.188 INFO:tasks.rados.rados.0.smithi107.stdout:1673: writing smithi10729407-19 from 2572288 to 3088384 tid 3 2020-06-12T02:37:39.188 INFO:tasks.rados.rados.0.smithi107.stdout:1674: delete oid 12 current snap is 0 2020-06-12T02:37:39.189 INFO:tasks.rados.rados.0.smithi107.stdout:1668: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:39.189 INFO:tasks.rados.rados.0.smithi107.stdout:1668: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:39.189 INFO:tasks.rados.rados.0.smithi107.stdout:1668: finishing write tid 5 to smithi10729407-11 2020-06-12T02:37:39.190 INFO:tasks.rados.rados.0.smithi107.stdout:1668: finishing write tid 6 to smithi10729407-11 2020-06-12T02:37:39.190 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 297 (ObjNum 1667 snap 0 seq_num 1667) dirty exists 2020-06-12T02:37:39.190 INFO:tasks.rados.rados.0.smithi107.stdout:1668: left oid 11 (ObjNum 1667 snap 0 seq_num 1667) 2020-06-12T02:37:39.191 INFO:tasks.rados.rados.0.smithi107.stdout:1673: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:39.191 INFO:tasks.rados.rados.0.smithi107.stdout:1673: finishing write tid 2 to smithi10729407-19 2020-06-12T02:37:39.191 INFO:tasks.rados.rados.0.smithi107.stdout:1673: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:39.192 INFO:tasks.rados.rados.0.smithi107.stdout:1673: finishing write tid 5 to smithi10729407-19 2020-06-12T02:37:39.192 INFO:tasks.rados.rados.0.smithi107.stdout:1673: finishing write tid 6 to smithi10729407-19 2020-06-12T02:37:39.192 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 262 (ObjNum 1672 snap 0 seq_num 1672) dirty exists 2020-06-12T02:37:39.192 INFO:tasks.rados.rados.0.smithi107.stdout:1673: left oid 19 (ObjNum 1672 snap 0 seq_num 1672) 2020-06-12T02:37:39.193 INFO:tasks.rados.rados.0.smithi107.stdout:1668: done (2 left) 2020-06-12T02:37:39.193 INFO:tasks.rados.rados.0.smithi107.stdout:1673: done (1 left) 2020-06-12T02:37:39.193 INFO:tasks.rados.rados.0.smithi107.stdout:1674: done (0 left) 2020-06-12T02:37:39.194 INFO:tasks.rados.rados.0.smithi107.stdout:1675: delete oid 6 current snap is 0 2020-06-12T02:37:39.194 INFO:tasks.rados.rados.0.smithi107.stdout:1675: done (0 left) 2020-06-12T02:37:39.194 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:37:39.195 INFO:tasks.rados.rados.0.smithi107.stdout:1676: seq_num 1675 ranges {0=458752,458752=794624,1253376=409600} 2020-06-12T02:37:39.195 INFO:tasks.rados.rados.0.smithi107.stdout:1676: writing smithi10729407-7 from 0 to 458752 tid 1 2020-06-12T02:37:39.195 INFO:tasks.rados.rados.0.smithi107.stdout:1676: writing smithi10729407-7 from 458752 to 1253376 tid 2 2020-06-12T02:37:39.196 INFO:tasks.rados.rados.0.smithi107.stdout:1676: writing smithi10729407-7 from 1253376 to 1662976 tid 3 2020-06-12T02:37:39.196 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:37:39.196 INFO:tasks.rados.rados.0.smithi107.stdout:1677: seq_num 1676 ranges {0=475136,475136=679936,1155072=524288} 2020-06-12T02:37:39.197 INFO:tasks.rados.rados.0.smithi107.stdout:1677: writing smithi10729407-17 from 0 to 475136 tid 1 2020-06-12T02:37:39.197 INFO:tasks.rados.rados.0.smithi107.stdout:1677: writing smithi10729407-17 from 475136 to 1155072 tid 2 2020-06-12T02:37:39.197 INFO:tasks.rados.rados.0.smithi107.stdout:1677: writing smithi10729407-17 from 1155072 to 1679360 tid 3 2020-06-12T02:37:39.198 INFO:tasks.rados.rados.0.smithi107.stdout:1678: delete oid 1 current snap is 0 2020-06-12T02:37:39.198 INFO:tasks.rados.rados.0.smithi107.stdout:1676: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:39.198 INFO:tasks.rados.rados.0.smithi107.stdout:1676: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:39.198 INFO:tasks.rados.rados.0.smithi107.stdout:1676: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:39.199 INFO:tasks.rados.rados.0.smithi107.stdout:1676: finishing write tid 5 to smithi10729407-7 2020-06-12T02:37:39.199 INFO:tasks.rados.rados.0.smithi107.stdout:1677: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:39.199 INFO:tasks.rados.rados.0.smithi107.stdout:1676: finishing write tid 6 to smithi10729407-7 2020-06-12T02:37:39.199 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 332 (ObjNum 1675 snap 0 seq_num 1675) dirty exists 2020-06-12T02:37:39.199 INFO:tasks.rados.rados.0.smithi107.stdout:1676: left oid 7 (ObjNum 1675 snap 0 seq_num 1675) 2020-06-12T02:37:39.200 INFO:tasks.rados.rados.0.smithi107.stdout:1676: done (2 left) 2020-06-12T02:37:39.200 INFO:tasks.rados.rados.0.smithi107.stdout:1678: done (1 left) 2020-06-12T02:37:39.200 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:37:39.200 INFO:tasks.rados.rados.0.smithi107.stdout:1679: seq_num 1678 ranges {2793472=507904,3301376=442368,3743744=622592,4366336=139264} 2020-06-12T02:37:39.201 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:37:39.164+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:37:39.203 INFO:tasks.rados.rados.0.smithi107.stdout:1679: writing smithi10729407-11 from 2793472 to 3301376 tid 1 2020-06-12T02:37:39.203 INFO:tasks.rados.rados.0.smithi107.stdout:1679: writing smithi10729407-11 from 3301376 to 3743744 tid 2 2020-06-12T02:37:39.204 INFO:tasks.rados.rados.0.smithi107.stdout:1679: writing smithi10729407-11 from 3743744 to 4366336 tid 3 2020-06-12T02:37:39.204 INFO:tasks.rados.rados.0.smithi107.stdout:1679: writing smithi10729407-11 from 4366336 to 4505600 tid 4 2020-06-12T02:37:39.204 INFO:tasks.rados.rados.0.smithi107.stdout:1680: delete oid 8 current snap is 0 2020-06-12T02:37:39.204 INFO:tasks.rados.rados.0.smithi107.stdout:1677: finishing write tid 2 to smithi10729407-17 2020-06-12T02:37:39.204 INFO:tasks.rados.rados.0.smithi107.stdout:1677: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:39.205 INFO:tasks.rados.rados.0.smithi107.stdout:1677: finishing write tid 5 to smithi10729407-17 2020-06-12T02:37:39.205 INFO:tasks.rados.rados.0.smithi107.stdout:1677: finishing write tid 6 to smithi10729407-17 2020-06-12T02:37:39.205 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 326 (ObjNum 1676 snap 0 seq_num 1676) dirty exists 2020-06-12T02:37:39.205 INFO:tasks.rados.rados.0.smithi107.stdout:1677: left oid 17 (ObjNum 1676 snap 0 seq_num 1676) 2020-06-12T02:37:39.205 INFO:tasks.rados.rados.0.smithi107.stdout:1679: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:39.206 INFO:tasks.rados.rados.0.smithi107.stdout:1677: done (2 left) 2020-06-12T02:37:39.206 INFO:tasks.rados.rados.0.smithi107.stdout:1680: done (1 left) 2020-06-12T02:37:39.206 INFO:tasks.rados.rados.0.smithi107.stdout:1681: delete oid 9 current snap is 0 2020-06-12T02:37:39.206 INFO:tasks.rados.rados.0.smithi107.stdout:1679: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:39.206 INFO:tasks.rados.rados.0.smithi107.stdout:1681: done (1 left) 2020-06-12T02:37:39.206 INFO:tasks.rados.rados.0.smithi107.stdout:1682: delete oid 19 current snap is 0 2020-06-12T02:37:39.207 INFO:tasks.rados.rados.0.smithi107.stdout:1679: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:39.207 INFO:tasks.rados.rados.0.smithi107.stdout:1679: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:39.207 INFO:tasks.rados.rados.0.smithi107.stdout:1679: finishing write tid 6 to smithi10729407-11 2020-06-12T02:37:39.207 INFO:tasks.rados.rados.0.smithi107.stdout:1679: finishing write tid 7 to smithi10729407-11 2020-06-12T02:37:39.207 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 302 (ObjNum 1678 snap 0 seq_num 1678) dirty exists 2020-06-12T02:37:39.208 INFO:tasks.rados.rados.0.smithi107.stdout:1679: left oid 11 (ObjNum 1678 snap 0 seq_num 1678) 2020-06-12T02:37:39.208 INFO:tasks.rados.rados.0.smithi107.stdout:1679: done (1 left) 2020-06-12T02:37:39.208 INFO:tasks.rados.rados.0.smithi107.stdout:1682: done (0 left) 2020-06-12T02:37:39.208 INFO:tasks.rados.rados.0.smithi107.stdout:1683: delete oid 2 current snap is 0 2020-06-12T02:37:39.208 INFO:tasks.rados.rados.0.smithi107.stdout:1683: done (0 left) 2020-06-12T02:37:39.208 INFO:tasks.rados.rados.0.smithi107.stdout:1684: delete oid 1 current snap is 0 2020-06-12T02:37:39.209 INFO:tasks.rados.rados.0.smithi107.stdout:1684: done (0 left) 2020-06-12T02:37:39.209 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:37:39.209 INFO:tasks.rados.rados.0.smithi107.stdout:1685: seq_num 1684 ranges {0=606208,606208=598016,1204224=606208} 2020-06-12T02:37:39.210 INFO:tasks.rados.rados.0.smithi107.stdout:1685: writing smithi10729407-10 from 0 to 606208 tid 1 2020-06-12T02:37:39.210 INFO:tasks.rados.rados.0.smithi107.stdout:1685: writing smithi10729407-10 from 606208 to 1204224 tid 2 2020-06-12T02:37:39.213 INFO:tasks.rados.rados.0.smithi107.stdout:1685: writing smithi10729407-10 from 1204224 to 1810432 tid 3 2020-06-12T02:37:39.214 INFO:tasks.rados.rados.0.smithi107.stdout:1686: delete oid 7 current snap is 0 2020-06-12T02:37:39.214 INFO:tasks.rados.rados.0.smithi107.stdout:1685: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:39.216 INFO:tasks.rados.rados.0.smithi107.stdout:1686: done (1 left) 2020-06-12T02:37:39.216 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:37:39.217 INFO:tasks.rados.rados.0.smithi107.stdout:1687: seq_num 1686 ranges {0=647168,647168=770048,1417216=434176} 2020-06-12T02:37:39.222 INFO:tasks.rados.rados.0.smithi107.stdout:1687: writing smithi10729407-9 from 0 to 647168 tid 1 2020-06-12T02:37:39.230 INFO:tasks.rados.rados.0.smithi107.stdout:1687: writing smithi10729407-9 from 647168 to 1417216 tid 2 2020-06-12T02:37:39.231 INFO:tasks.rados.rados.0.smithi107.stdout:1687: writing smithi10729407-9 from 1417216 to 1851392 tid 3 2020-06-12T02:37:39.231 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:39.231 INFO:tasks.rados.rados.0.smithi107.stdout:1688: seq_num 1687 ranges {0=663552,663552=655360,1318912=548864} 2020-06-12T02:37:39.233 INFO:tasks.rados.rados.0.smithi107.stdout:1688: writing smithi10729407-2 from 0 to 663552 tid 1 2020-06-12T02:37:39.237 INFO:tasks.rados.rados.0.smithi107.stdout:1688: writing smithi10729407-2 from 663552 to 1318912 tid 2 2020-06-12T02:37:39.242 INFO:tasks.rados.rados.0.smithi107.stdout:1688: writing smithi10729407-2 from 1318912 to 1867776 tid 3 2020-06-12T02:37:39.242 INFO:tasks.rados.rados.0.smithi107.stdout:1689: delete oid 17 current snap is 0 2020-06-12T02:37:39.243 INFO:tasks.rados.rados.0.smithi107.stdout:1685: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:39.243 INFO:tasks.rados.rados.0.smithi107.stdout:1685: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:39.243 INFO:tasks.rados.rados.0.smithi107.stdout:1685: finishing write tid 5 to smithi10729407-10 2020-06-12T02:37:39.243 INFO:tasks.rados.rados.0.smithi107.stdout:1685: finishing write tid 6 to smithi10729407-10 2020-06-12T02:37:39.244 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 159 (ObjNum 1684 snap 0 seq_num 1684) dirty exists 2020-06-12T02:37:39.244 INFO:tasks.rados.rados.0.smithi107.stdout:1685: left oid 10 (ObjNum 1684 snap 0 seq_num 1684) 2020-06-12T02:37:39.244 INFO:tasks.rados.rados.0.smithi107.stdout:1687: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:39.244 INFO:tasks.rados.rados.0.smithi107.stdout:1688: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:39.244 INFO:tasks.rados.rados.0.smithi107.stdout:1687: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:39.245 INFO:tasks.rados.rados.0.smithi107.stdout:1687: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:39.246 INFO:tasks.rados.rados.0.smithi107.stdout:1687: finishing write tid 5 to smithi10729407-9 2020-06-12T02:37:39.246 INFO:tasks.rados.rados.0.smithi107.stdout:1685: done (3 left) 2020-06-12T02:37:39.246 INFO:tasks.rados.rados.0.smithi107.stdout:1689: done (2 left) 2020-06-12T02:37:39.246 INFO:tasks.rados.rados.0.smithi107.stdout:1690: delete oid 16 current snap is 0 2020-06-12T02:37:39.246 INFO:tasks.rados.rados.0.smithi107.stdout:1687: finishing write tid 6 to smithi10729407-9 2020-06-12T02:37:39.247 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 297 (ObjNum 1686 snap 0 seq_num 1686) dirty exists 2020-06-12T02:37:39.247 INFO:tasks.rados.rados.0.smithi107.stdout:1687: left oid 9 (ObjNum 1686 snap 0 seq_num 1686) 2020-06-12T02:37:39.249 INFO:tasks.rados.rados.0.smithi107.stdout:1687: done (2 left) 2020-06-12T02:37:39.249 INFO:tasks.rados.rados.0.smithi107.stdout:1690: done (1 left) 2020-06-12T02:37:39.249 INFO:tasks.rados.rados.0.smithi107.stdout:1691: delete oid 12 current snap is 0 2020-06-12T02:37:39.251 INFO:tasks.rados.rados.0.smithi107.stdout:1688: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:39.252 INFO:tasks.rados.rados.0.smithi107.stdout:1688: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:39.254 INFO:tasks.rados.rados.0.smithi107.stdout:1691: done (1 left) 2020-06-12T02:37:39.254 INFO:tasks.rados.rados.0.smithi107.stdout:1692: delete oid 19 current snap is 0 2020-06-12T02:37:39.255 INFO:tasks.rados.rados.0.smithi107.stdout:1688: finishing write tid 5 to smithi10729407-2 2020-06-12T02:37:39.257 INFO:tasks.rados.rados.0.smithi107.stdout:1692: done (1 left) 2020-06-12T02:37:39.257 INFO:tasks.rados.rados.0.smithi107.stdout:1693: delete oid 14 current snap is 0 2020-06-12T02:37:39.257 INFO:tasks.rados.rados.0.smithi107.stdout:1688: finishing write tid 6 to smithi10729407-2 2020-06-12T02:37:39.258 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 160 (ObjNum 1687 snap 0 seq_num 1687) dirty exists 2020-06-12T02:37:39.258 INFO:tasks.rados.rados.0.smithi107.stdout:1688: left oid 2 (ObjNum 1687 snap 0 seq_num 1687) 2020-06-12T02:37:39.261 INFO:tasks.rados.rados.0.smithi107.stdout:1688: done (1 left) 2020-06-12T02:37:39.261 INFO:tasks.rados.rados.0.smithi107.stdout:1693: done (0 left) 2020-06-12T02:37:39.262 INFO:tasks.rados.rados.0.smithi107.stdout:1694: delete oid 18 current snap is 0 2020-06-12T02:37:39.262 INFO:tasks.rados.rados.0.smithi107.stdout:1694: done (0 left) 2020-06-12T02:37:39.263 INFO:tasks.rados.rados.0.smithi107.stdout:1695: delete oid 20 current snap is 0 2020-06-12T02:37:39.268 INFO:tasks.rados.rados.0.smithi107.stdout:1695: done (0 left) 2020-06-12T02:37:39.268 INFO:tasks.rados.rados.0.smithi107.stdout:1696: delete oid 4 current snap is 0 2020-06-12T02:37:39.268 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:39.265+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:39.271 INFO:tasks.rados.rados.0.smithi107.stdout:1696: done (0 left) 2020-06-12T02:37:39.271 INFO:tasks.rados.rados.0.smithi107.stdout:1697: delete oid 4 current snap is 0 2020-06-12T02:37:39.273 INFO:tasks.rados.rados.0.smithi107.stdout:1697: done (0 left) 2020-06-12T02:37:39.273 INFO:tasks.rados.rados.0.smithi107.stdout:1698: delete oid 12 current snap is 0 2020-06-12T02:37:39.275 INFO:tasks.rados.rados.0.smithi107.stdout:1698: done (0 left) 2020-06-12T02:37:39.275 INFO:tasks.rados.rados.0.smithi107.stdout:1699: delete oid 7 current snap is 0 2020-06-12T02:37:39.277 INFO:tasks.rados.rados.0.smithi107.stdout:1699: done (0 left) 2020-06-12T02:37:39.277 INFO:tasks.rados.rados.0.smithi107.stdout:1700: delete oid 7 current snap is 0 2020-06-12T02:37:39.278 INFO:tasks.rados.rados.0.smithi107.stdout:1700: done (0 left) 2020-06-12T02:37:39.279 INFO:tasks.rados.rados.0.smithi107.stdout:1701: delete oid 13 current snap is 0 2020-06-12T02:37:39.280 INFO:tasks.rados.rados.0.smithi107.stdout:1701: done (0 left) 2020-06-12T02:37:39.280 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:37:39.281 INFO:tasks.rados.rados.0.smithi107.stdout:1702: seq_num 1701 ranges {0=491520,491520=737280,1228800=753664,1982464=114688} 2020-06-12T02:37:39.283 INFO:tasks.rados.rados.0.smithi107.stdout:1702: writing smithi10729407-18 from 0 to 491520 tid 1 2020-06-12T02:37:39.287 INFO:tasks.rados.rados.0.smithi107.stdout:1702: writing smithi10729407-18 from 491520 to 1228800 tid 2 2020-06-12T02:37:39.292 INFO:tasks.rados.rados.0.smithi107.stdout:1702: writing smithi10729407-18 from 1228800 to 1982464 tid 3 2020-06-12T02:37:39.293 INFO:tasks.rados.rados.0.smithi107.stdout:1702: writing smithi10729407-18 from 1982464 to 2097152 tid 4 2020-06-12T02:37:39.293 INFO:tasks.rados.rados.0.smithi107.stdout:1703: delete oid 2 current snap is 0 2020-06-12T02:37:39.293 INFO:tasks.rados.rados.0.smithi107.stdout:1702: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:39.294 INFO:tasks.rados.rados.0.smithi107.stdout:1702: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:39.298 INFO:tasks.rados.rados.0.smithi107.stdout:1703: done (1 left) 2020-06-12T02:37:39.299 INFO:tasks.rados.rados.0.smithi107.stdout:1704: delete oid 16 current snap is 0 2020-06-12T02:37:39.302 INFO:tasks.rados.rados.0.smithi107.stdout:1704: done (1 left) 2020-06-12T02:37:39.302 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:37:39.303 INFO:tasks.rados.rados.0.smithi107.stdout:1705: seq_num 1704 ranges {0=540672,540672=434176,974848=524288,1499136=647168} 2020-06-12T02:37:39.305 INFO:tasks.rados.rados.0.smithi107.stdout:1705: writing smithi10729407-7 from 0 to 540672 tid 1 2020-06-12T02:37:39.308 INFO:tasks.rados.rados.0.smithi107.stdout:1705: writing smithi10729407-7 from 540672 to 974848 tid 2 2020-06-12T02:37:39.311 INFO:tasks.rados.rados.0.smithi107.stdout:1705: writing smithi10729407-7 from 974848 to 1499136 tid 3 2020-06-12T02:37:39.316 INFO:tasks.rados.rados.0.smithi107.stdout:1705: writing smithi10729407-7 from 1499136 to 2146304 tid 4 2020-06-12T02:37:39.316 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:37:39.316 INFO:tasks.rados.rados.0.smithi107.stdout:1706: seq_num 1705 ranges {0=565248,565248=720896,1286144=729088,2015232=155648} 2020-06-12T02:37:39.320 INFO:tasks.rados.rados.0.smithi107.stdout:1706: writing smithi10729407-1 from 0 to 565248 tid 1 2020-06-12T02:37:39.325 INFO:tasks.rados.rados.0.smithi107.stdout:1706: writing smithi10729407-1 from 565248 to 1286144 tid 2 2020-06-12T02:37:39.328 INFO:tasks.rados.rados.0.smithi107.stdout:1706: writing smithi10729407-1 from 1286144 to 2015232 tid 3 2020-06-12T02:37:39.330 INFO:tasks.rados.rados.0.smithi107.stdout:1706: writing smithi10729407-1 from 2015232 to 2170880 tid 4 2020-06-12T02:37:39.331 INFO:tasks.rados.rados.0.smithi107.stdout:1707: delete oid 16 current snap is 0 2020-06-12T02:37:39.331 INFO:tasks.rados.rados.0.smithi107.stdout:1702: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:39.331 INFO:tasks.rados.rados.0.smithi107.stdout:1702: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:39.331 INFO:tasks.rados.rados.0.smithi107.stdout:1702: finishing write tid 6 to smithi10729407-18 2020-06-12T02:37:39.331 INFO:tasks.rados.rados.0.smithi107.stdout:1702: finishing write tid 7 to smithi10729407-18 2020-06-12T02:37:39.332 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 303 (ObjNum 1701 snap 0 seq_num 1701) dirty exists 2020-06-12T02:37:39.332 INFO:tasks.rados.rados.0.smithi107.stdout:1702: left oid 18 (ObjNum 1701 snap 0 seq_num 1701) 2020-06-12T02:37:39.332 INFO:tasks.rados.rados.0.smithi107.stdout:1705: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:39.332 INFO:tasks.rados.rados.0.smithi107.stdout:1705: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:39.332 INFO:tasks.rados.rados.0.smithi107.stdout:1705: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:39.332 INFO:tasks.rados.rados.0.smithi107.stdout:1706: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:39.333 INFO:tasks.rados.rados.0.smithi107.stdout:1705: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:39.333 INFO:tasks.rados.rados.0.smithi107.stdout:1705: finishing write tid 6 to smithi10729407-7 2020-06-12T02:37:39.333 INFO:tasks.rados.rados.0.smithi107.stdout:1705: finishing write tid 7 to smithi10729407-7 2020-06-12T02:37:39.333 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 341 (ObjNum 1704 snap 0 seq_num 1704) dirty exists 2020-06-12T02:37:39.334 INFO:tasks.rados.rados.0.smithi107.stdout:1705: left oid 7 (ObjNum 1704 snap 0 seq_num 1704) 2020-06-12T02:37:39.334 INFO:tasks.rados.rados.0.smithi107.stdout:1702: done (3 left) 2020-06-12T02:37:39.334 INFO:tasks.rados.rados.0.smithi107.stdout:1705: done (2 left) 2020-06-12T02:37:39.335 INFO:tasks.rados.rados.0.smithi107.stdout:1707: done (1 left) 2020-06-12T02:37:39.335 INFO:tasks.rados.rados.0.smithi107.stdout:1708: delete oid 13 current snap is 0 2020-06-12T02:37:39.336 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:37:39.336 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:37:39.336 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.336 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:3617 3.ds1 3:b983fd28:::smithi10729407-1:head [append 0~720896 in=720896b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:39.337 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:39.323526+0000", 2020-06-12T02:37:39.337 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.0095547400000000008, 2020-06-12T02:37:39.337 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0095801819999999996, 2020-06-12T02:37:39.337 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:39.337 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:39.338 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:39.343 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:39.344 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:39.344 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 3617 2020-06-12T02:37:39.344 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.345 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:39.345 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.345 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:39.346 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.323526+0000", 2020-06-12T02:37:39.346 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:39.346 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.347 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.347 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:39.347 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.323526+0000", 2020-06-12T02:37:39.348 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00097405899999999999 2020-06-12T02:37:39.348 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.348 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.349 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:39.349 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.324500+0000", 2020-06-12T02:37:39.349 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5340000000000002e-06 2020-06-12T02:37:39.350 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.350 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.350 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:39.351 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.324507+0000", 2020-06-12T02:37:39.351 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6049000000000002e-05 2020-06-12T02:37:39.351 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.352 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.352 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:39.352 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.324523+0000", 2020-06-12T02:37:39.352 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3159999999999998e-06 2020-06-12T02:37:39.353 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.353 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.353 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:39.354 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.324528+0000", 2020-06-12T02:37:39.354 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4907000000000003e-05 2020-06-12T02:37:39.354 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.355 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.355 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:39.355 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.324563+0000", 2020-06-12T02:37:39.356 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9836e-05 2020-06-12T02:37:39.356 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.356 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.357 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:39.357 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.324632+0000", 2020-06-12T02:37:39.357 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053242600000000001 2020-06-12T02:37:39.357 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.358 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.358 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:39.358 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.325165+0000", 2020-06-12T02:37:39.359 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001639127 2020-06-12T02:37:39.360 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:39.360 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:39.361 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:39.361 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.361 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.362 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:3618 3.ds1 3:b983fd28:::smithi10729407-1:head [append 0~729088 in=729088b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:39.362 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:39.327647+0000", 2020-06-12T02:37:39.362 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.0054337559999999997, 2020-06-12T02:37:39.362 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0055166989999999999, 2020-06-12T02:37:39.363 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:39.363 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:39.363 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:39.364 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:39.365 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:39.365 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 3618 2020-06-12T02:37:39.365 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.365 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:39.365 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.366 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:39.366 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.327647+0000", 2020-06-12T02:37:39.366 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:39.366 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.367 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.367 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:39.367 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.327647+0000", 2020-06-12T02:37:39.367 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00083516299999999999 2020-06-12T02:37:39.368 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.368 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.368 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:39.368 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.328482+0000", 2020-06-12T02:37:39.368 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1780000000000002e-06 2020-06-12T02:37:39.368 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.369 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.369 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:39.369 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.328487+0000", 2020-06-12T02:37:39.369 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2063e-05 2020-06-12T02:37:39.369 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.370 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.370 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:39.370 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.328499+0000", 2020-06-12T02:37:39.370 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1030000000000001e-06 2020-06-12T02:37:39.370 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.371 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.371 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:39.371 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.328504+0000", 2020-06-12T02:37:39.371 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7909999999999999e-05 2020-06-12T02:37:39.371 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.371 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.372 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:39.372 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.328532+0000", 2020-06-12T02:37:39.372 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3495000000000001e-05 2020-06-12T02:37:39.372 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.372 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.373 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:39.373 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.328596+0000", 2020-06-12T02:37:39.373 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00044369 2020-06-12T02:37:39.373 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.373 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.374 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:39.374 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.329039+0000", 2020-06-12T02:37:39.374 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0013926019999999999 2020-06-12T02:37:39.375 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:39.375 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:39.375 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:39.375 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.375 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.376 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:3619 3.ds1 3:b983fd28:::smithi10729407-1:head [append 0~155648 in=155648b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:39.376 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:39.328542+0000", 2020-06-12T02:37:39.376 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.0045387910000000004, 2020-06-12T02:37:39.376 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0046843199999999996, 2020-06-12T02:37:39.376 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:39.376 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:39.377 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:39.377 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:39.377 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:39.377 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 3619 2020-06-12T02:37:39.377 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.378 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:39.378 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.378 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:39.378 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.328542+0000", 2020-06-12T02:37:39.378 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:39.379 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.379 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.379 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:39.380 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.328542+0000", 2020-06-12T02:37:39.380 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014349299999999999 2020-06-12T02:37:39.380 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.380 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.380 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:39.381 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.328685+0000", 2020-06-12T02:37:39.381 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9250000000000003e-06 2020-06-12T02:37:39.381 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.381 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.381 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:39.382 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.328690+0000", 2020-06-12T02:37:39.382 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0436e-05 2020-06-12T02:37:39.382 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.382 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.382 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:39.383 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.328701+0000", 2020-06-12T02:37:39.383 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3639999999999999e-06 2020-06-12T02:37:39.383 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.383 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.383 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:39.384 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.328704+0000", 2020-06-12T02:37:39.384 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0029847200000000002 2020-06-12T02:37:39.384 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.385 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.385 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:39.385 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.331689+0000", 2020-06-12T02:37:39.385 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8566000000000006e-05 2020-06-12T02:37:39.385 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.386 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.386 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:39.386 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.331767+0000", 2020-06-12T02:37:39.386 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000440061 2020-06-12T02:37:39.386 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.386 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.387 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:39.387 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.332207+0000", 2020-06-12T02:37:39.387 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003665565 2020-06-12T02:37:39.387 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:39.387 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:39.388 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:39.388 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.388 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.388 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:3620 3.ds1 3:b983fd28:::smithi10729407-1:head [setxattr _header (55) in=62b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:39.388 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:39.328739+0000", 2020-06-12T02:37:39.389 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.0043416089999999997, 2020-06-12T02:37:39.389 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004583345, 2020-06-12T02:37:39.389 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:39.389 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:39.389 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:39.390 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:39.390 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:39.390 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 3620 2020-06-12T02:37:39.390 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.390 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:39.390 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.391 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:39.391 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.328739+0000", 2020-06-12T02:37:39.391 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:39.391 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.391 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.392 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:39.392 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.328739+0000", 2020-06-12T02:37:39.392 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8093999999999998e-05 2020-06-12T02:37:39.392 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.392 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.392 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:39.393 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.328777+0000", 2020-06-12T02:37:39.393 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.0250000000000002e-06 2020-06-12T02:37:39.393 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.393 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.393 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:39.394 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.328785+0000", 2020-06-12T02:37:39.394 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4330000000000001e-06 2020-06-12T02:37:39.394 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.395 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.395 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:39.395 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.328791+0000", 2020-06-12T02:37:39.395 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.818e-06 2020-06-12T02:37:39.395 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.396 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.396 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:39.396 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.328794+0000", 2020-06-12T02:37:39.396 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0044415879999999998 2020-06-12T02:37:39.396 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.397 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.397 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:39.397 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.333236+0000", 2020-06-12T02:37:39.397 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8232000000000007e-05 2020-06-12T02:37:39.397 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.398 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.398 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:39.398 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.333304+0000", 2020-06-12T02:37:39.398 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045651900000000002 2020-06-12T02:37:39.398 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:39.398 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:39.399 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:39.399 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.399 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.399 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:3621 3.ds1 3.14bfc19d (undecoded) ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:37:39.399 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:39.328817+0000", 2020-06-12T02:37:39.400 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.0042634250000000004, 2020-06-12T02:37:39.400 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045857240000000002, 2020-06-12T02:37:39.400 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:39.400 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "queued for pg", 2020-06-12T02:37:39.400 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:39.400 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:39.401 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:39.401 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 3621 2020-06-12T02:37:39.401 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.401 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:39.401 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.402 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:39.402 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.328817+0000", 2020-06-12T02:37:39.402 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:39.402 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.402 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.402 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:39.403 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.328817+0000", 2020-06-12T02:37:39.403 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2396e-05 2020-06-12T02:37:39.403 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.403 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.403 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:39.404 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.328839+0000", 2020-06-12T02:37:39.404 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.18e-06 2020-06-12T02:37:39.404 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.404 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.405 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:39.405 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.328843+0000", 2020-06-12T02:37:39.405 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2199999999999997e-06 2020-06-12T02:37:39.405 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.406 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.406 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:39.406 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.328849+0000", 2020-06-12T02:37:39.406 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.305e-06 2020-06-12T02:37:39.406 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.407 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.407 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:39.407 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.328851+0000", 2020-06-12T02:37:39.407 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4100999999999998e-05 2020-06-12T02:37:39.407 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:39.408 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:39.408 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:39.408 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.408 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.408 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWriteReply(3.ds1 27/21 ECSubWriteReply(tid=1020, last_complete=27'334, committed=1, applied=1))", 2020-06-12T02:37:39.408 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:39.332590+0000", 2020-06-12T02:37:39.409 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.000490442, 2020-06-12T02:37:39.409 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000847965, 2020-06-12T02:37:39.409 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:39.409 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "queued for pg", 2020-06-12T02:37:39.409 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:39.410 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.410 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:39.410 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.332590+0000", 2020-06-12T02:37:39.410 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:39.410 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.410 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.411 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:39.411 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.332590+0000", 2020-06-12T02:37:39.411 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9873e-05 2020-06-12T02:37:39.411 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.411 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.412 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:39.412 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.332660+0000", 2020-06-12T02:37:39.412 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9480000000000003e-06 2020-06-12T02:37:39.412 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.412 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.412 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:39.413 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.332670+0000", 2020-06-12T02:37:39.413 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6651e-05 2020-06-12T02:37:39.413 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.413 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.413 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:39.414 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.332687+0000", 2020-06-12T02:37:39.414 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0600000000000002e-06 2020-06-12T02:37:39.414 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:39.414 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:39.414 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:39.414 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:39.332694+0000", 2020-06-12T02:37:39.415 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000103532 2020-06-12T02:37:39.415 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:39.415 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:39.416 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:39.416 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:39.416 INFO:teuthology.orchestra.run.smithi188.stdout: ], 2020-06-12T02:37:39.416 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 6 2020-06-12T02:37:39.416 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:37:39.417 INFO:tasks.rados.rados.0.smithi107.stdout:1708: done (1 left) 2020-06-12T02:37:39.417 INFO:tasks.rados.rados.0.smithi107.stdout:1709: delete oid 11 current snap is 0 2020-06-12T02:37:39.417 INFO:tasks.rados.rados.0.smithi107.stdout:1706: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:39.417 INFO:tasks.rados.rados.0.smithi107.stdout:1706: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:39.424 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-06-12T02:37:39.425 INFO:tasks.rados.rados.0.smithi107.stdout:1706: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:39.425 INFO:tasks.rados.rados.0.smithi107.stdout:1706: finishing write tid 6 to smithi10729407-1 2020-06-12T02:37:39.425 INFO:tasks.rados.rados.0.smithi107.stdout:1709: done (1 left) 2020-06-12T02:37:39.425 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:37:39.426 INFO:tasks.rados.rados.0.smithi107.stdout:1710: seq_num 1709 ranges {0=630784,630784=655360,1286144=712704,1998848=237568} 2020-06-12T02:37:39.426 INFO:tasks.rados.rados.0.smithi107.stdout:1710: writing smithi10729407-20 from 0 to 630784 tid 1 2020-06-12T02:37:39.426 INFO:tasks.rados.rados.0.smithi107.stdout:1710: writing smithi10729407-20 from 630784 to 1286144 tid 2 2020-06-12T02:37:39.426 INFO:tasks.rados.rados.0.smithi107.stdout:1710: writing smithi10729407-20 from 1286144 to 1998848 tid 3 2020-06-12T02:37:39.426 INFO:tasks.rados.rados.0.smithi107.stdout:1710: writing smithi10729407-20 from 1998848 to 2236416 tid 4 2020-06-12T02:37:39.427 INFO:tasks.rados.rados.0.smithi107.stdout:1711: delete oid 8 current snap is 0 2020-06-12T02:37:39.427 INFO:tasks.rados.rados.0.smithi107.stdout:1706: finishing write tid 7 to smithi10729407-1 2020-06-12T02:37:39.427 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 337 (ObjNum 1705 snap 0 seq_num 1705) dirty exists 2020-06-12T02:37:39.427 INFO:tasks.rados.rados.0.smithi107.stdout:1706: left oid 1 (ObjNum 1705 snap 0 seq_num 1705) 2020-06-12T02:37:39.427 INFO:tasks.rados.rados.0.smithi107.stdout:1710: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:39.427 INFO:tasks.rados.rados.0.smithi107.stdout:1706: done (2 left) 2020-06-12T02:37:39.428 INFO:tasks.rados.rados.0.smithi107.stdout:1711: done (1 left) 2020-06-12T02:37:39.428 INFO:tasks.rados.rados.0.smithi107.stdout:1712: delete oid 3 current snap is 0 2020-06-12T02:37:39.428 INFO:tasks.rados.rados.0.smithi107.stdout:1710: finishing write tid 2 to smithi10729407-20 2020-06-12T02:37:39.428 INFO:tasks.rados.rados.0.smithi107.stdout:1710: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:39.428 INFO:tasks.rados.rados.0.smithi107.stdout:1712: done (1 left) 2020-06-12T02:37:39.428 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:37:39.429 INFO:tasks.rados.rados.0.smithi107.stdout:1713: seq_num 1712 ranges {2170880=679936,2850816=761856,3612672=483328,4096000=360448} 2020-06-12T02:37:39.429 INFO:tasks.rados.rados.0.smithi107.stdout:1713: writing smithi10729407-1 from 2170880 to 2850816 tid 1 2020-06-12T02:37:39.429 INFO:tasks.rados.rados.0.smithi107.stdout:1713: writing smithi10729407-1 from 2850816 to 3612672 tid 2 2020-06-12T02:37:39.429 INFO:tasks.rados.rados.0.smithi107.stdout:1713: writing smithi10729407-1 from 3612672 to 4096000 tid 3 2020-06-12T02:37:39.429 INFO:tasks.rados.rados.0.smithi107.stdout:1713: writing smithi10729407-1 from 4096000 to 4456448 tid 4 2020-06-12T02:37:39.430 INFO:tasks.rados.rados.0.smithi107.stdout:1714: delete oid 5 current snap is 0 2020-06-12T02:37:39.430 INFO:tasks.rados.rados.0.smithi107.stdout:1710: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:39.430 INFO:tasks.rados.rados.0.smithi107.stdout:1710: finishing write tid 6 to smithi10729407-20 2020-06-12T02:37:39.430 INFO:tasks.rados.rados.0.smithi107.stdout:1710: finishing write tid 7 to smithi10729407-20 2020-06-12T02:37:39.430 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 126 (ObjNum 1709 snap 0 seq_num 1709) dirty exists 2020-06-12T02:37:39.431 INFO:tasks.rados.rados.0.smithi107.stdout:1710: left oid 20 (ObjNum 1709 snap 0 seq_num 1709) 2020-06-12T02:37:39.431 INFO:tasks.rados.rados.0.smithi107.stdout:1713: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:39.431 INFO:tasks.rados.rados.0.smithi107.stdout:1710: done (2 left) 2020-06-12T02:37:39.432 INFO:tasks.rados.rados.0.smithi107.stdout:1714: done (1 left) 2020-06-12T02:37:39.432 INFO:tasks.rados.rados.0.smithi107.stdout:1715: delete oid 3 current snap is 0 2020-06-12T02:37:39.432 INFO:tasks.rados.rados.0.smithi107.stdout:1715: done (1 left) 2020-06-12T02:37:39.433 INFO:tasks.rados.rados.0.smithi107.stdout:1716: delete oid 16 current snap is 0 2020-06-12T02:37:39.433 INFO:tasks.rados.rados.0.smithi107.stdout:1713: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:39.433 INFO:tasks.rados.rados.0.smithi107.stdout:1716: done (1 left) 2020-06-12T02:37:39.433 INFO:tasks.rados.rados.0.smithi107.stdout:1717: delete oid 8 current snap is 0 2020-06-12T02:37:39.433 INFO:tasks.rados.rados.0.smithi107.stdout:1713: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:39.434 INFO:tasks.rados.rados.0.smithi107.stdout:1713: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:39.434 INFO:tasks.rados.rados.0.smithi107.stdout:1713: finishing write tid 6 to smithi10729407-1 2020-06-12T02:37:39.434 INFO:tasks.rados.rados.0.smithi107.stdout:1717: done (1 left) 2020-06-12T02:37:39.434 INFO:tasks.rados.rados.0.smithi107.stdout:1718: delete oid 9 current snap is 0 2020-06-12T02:37:39.434 INFO:tasks.rados.rados.0.smithi107.stdout:1713: finishing write tid 7 to smithi10729407-1 2020-06-12T02:37:39.435 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 342 (ObjNum 1712 snap 0 seq_num 1712) dirty exists 2020-06-12T02:37:39.435 INFO:tasks.rados.rados.0.smithi107.stdout:1713: left oid 1 (ObjNum 1712 snap 0 seq_num 1712) 2020-06-12T02:37:39.435 INFO:tasks.rados.rados.0.smithi107.stdout:1713: done (1 left) 2020-06-12T02:37:39.435 INFO:tasks.rados.rados.0.smithi107.stdout:1718: done (0 left) 2020-06-12T02:37:39.436 INFO:tasks.rados.rados.0.smithi107.stdout:1719: delete oid 1 current snap is 0 2020-06-12T02:37:39.436 INFO:tasks.rados.rados.0.smithi107.stdout:1719: done (0 left) 2020-06-12T02:37:39.436 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:37:39.436 INFO:tasks.rados.rados.0.smithi107.stdout:1720: seq_num 1719 ranges {0=794624,794624=802816,1597440=638976,2236416=163840} 2020-06-12T02:37:39.437 INFO:tasks.rados.rados.0.smithi107.stdout:1720: writing smithi10729407-16 from 0 to 794624 tid 1 2020-06-12T02:37:39.437 INFO:tasks.rados.rados.0.smithi107.stdout:1720: writing smithi10729407-16 from 794624 to 1597440 tid 2 2020-06-12T02:37:39.437 INFO:tasks.rados.rados.0.smithi107.stdout:1720: writing smithi10729407-16 from 1597440 to 2236416 tid 3 2020-06-12T02:37:39.437 INFO:tasks.rados.rados.0.smithi107.stdout:1720: writing smithi10729407-16 from 2236416 to 2400256 tid 4 2020-06-12T02:37:39.437 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:37:39.438 INFO:tasks.rados.rados.0.smithi107.stdout:1721: seq_num 1720 ranges {0=8192} 2020-06-12T02:37:39.438 INFO:tasks.rados.rados.0.smithi107.stdout:1721: writing smithi10729407-3 from 0 to 8192 tid 1 2020-06-12T02:37:39.438 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:37:39.438 INFO:tasks.rados.rados.0.smithi107.stdout:1722: seq_num 1721 ranges {0=24576} 2020-06-12T02:37:39.438 INFO:tasks.rados.rados.0.smithi107.stdout:1722: writing smithi10729407-14 from 0 to 24576 tid 1 2020-06-12T02:37:39.438 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:37:39.439 INFO:tasks.rados.rados.0.smithi107.stdout:1723: seq_num 1722 ranges {0=40960} 2020-06-12T02:37:39.439 INFO:tasks.rados.rados.0.smithi107.stdout:1723: writing smithi10729407-1 from 0 to 40960 tid 1 2020-06-12T02:37:39.439 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:37:39.439 INFO:tasks.rados.rados.0.smithi107.stdout:1724: seq_num 1723 ranges {0=57344} 2020-06-12T02:37:39.439 INFO:tasks.rados.rados.0.smithi107.stdout:1724: writing smithi10729407-9 from 0 to 57344 tid 1 2020-06-12T02:37:39.439 INFO:tasks.rados.rados.0.smithi107.stdout:1725: delete oid 6 current snap is 0 2020-06-12T02:37:39.440 INFO:tasks.rados.rados.0.smithi107.stdout:1720: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:39.440 INFO:tasks.rados.rados.0.smithi107.stdout:1724: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:39.440 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:37:39.440 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:37:39.441 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:37:39.441 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:37:39.441 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:37:39.415+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:37:39.445 INFO:tasks.rados.rados.0.smithi107.stdout:1720: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:39.445 INFO:tasks.rados.rados.0.smithi107.stdout:1720: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:39.445 INFO:tasks.rados.rados.0.smithi107.stdout:1722: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:39.446 INFO:tasks.rados.rados.0.smithi107.stdout:1720: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:39.446 INFO:tasks.rados.rados.0.smithi107.stdout:1723: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:39.446 INFO:tasks.rados.rados.0.smithi107.stdout:1724: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:39.447 INFO:tasks.rados.rados.0.smithi107.stdout:1723: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:39.447 INFO:tasks.rados.rados.0.smithi107.stdout:1720: finishing write tid 6 to smithi10729407-16 2020-06-12T02:37:39.447 INFO:tasks.rados.rados.0.smithi107.stdout:1725: done (5 left) 2020-06-12T02:37:39.447 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:37:39.447 INFO:tasks.rados.rados.0.smithi107.stdout:1726: seq_num 1725 ranges {0=98304} 2020-06-12T02:37:39.448 INFO:tasks.rados.rados.0.smithi107.stdout:1726: writing smithi10729407-13 from 0 to 98304 tid 1 2020-06-12T02:37:39.448 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:37:39.448 INFO:tasks.rados.rados.0.smithi107.stdout:1727: seq_num 1726 ranges {0=114688} 2020-06-12T02:37:39.448 INFO:tasks.rados.rados.0.smithi107.stdout:1727: writing smithi10729407-11 from 0 to 114688 tid 1 2020-06-12T02:37:39.448 INFO:tasks.rados.rados.0.smithi107.stdout:1728: delete oid 6 current snap is 0 2020-06-12T02:37:39.449 INFO:tasks.rados.rados.0.smithi107.stdout:1723: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:39.449 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 345 (ObjNum 1722 snap 0 seq_num 1722) dirty exists 2020-06-12T02:37:39.449 INFO:tasks.rados.rados.0.smithi107.stdout:1723: left oid 1 (ObjNum 1722 snap 0 seq_num 1722) 2020-06-12T02:37:39.449 INFO:tasks.rados.rados.0.smithi107.stdout:1720: finishing write tid 7 to smithi10729407-16 2020-06-12T02:37:39.449 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 305 (ObjNum 1719 snap 0 seq_num 1719) dirty exists 2020-06-12T02:37:39.450 INFO:tasks.rados.rados.0.smithi107.stdout:1720: left oid 16 (ObjNum 1719 snap 0 seq_num 1719) 2020-06-12T02:37:39.450 INFO:tasks.rados.rados.0.smithi107.stdout:1724: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:39.450 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 306 (ObjNum 1723 snap 0 seq_num 1723) dirty exists 2020-06-12T02:37:39.450 INFO:tasks.rados.rados.0.smithi107.stdout:1724: left oid 9 (ObjNum 1723 snap 0 seq_num 1723) 2020-06-12T02:37:39.450 INFO:tasks.rados.rados.0.smithi107.stdout:1722: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:39.451 INFO:tasks.rados.rados.0.smithi107.stdout:1722: finishing write tid 4 to smithi10729407-14 2020-06-12T02:37:39.451 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 306 (ObjNum 1721 snap 0 seq_num 1721) dirty exists 2020-06-12T02:37:39.451 INFO:tasks.rados.rados.0.smithi107.stdout:1722: left oid 14 (ObjNum 1721 snap 0 seq_num 1721) 2020-06-12T02:37:39.451 INFO:tasks.rados.rados.0.smithi107.stdout:1721: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:39.451 INFO:tasks.rados.rados.0.smithi107.stdout:1721: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:39.452 INFO:tasks.rados.rados.0.smithi107.stdout:1727: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:39.452 INFO:tasks.rados.rados.0.smithi107.stdout:1726: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:39.452 INFO:tasks.rados.rados.0.smithi107.stdout:1721: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:39.452 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 307 (ObjNum 1720 snap 0 seq_num 1720) dirty exists 2020-06-12T02:37:39.452 INFO:tasks.rados.rados.0.smithi107.stdout:1721: left oid 3 (ObjNum 1720 snap 0 seq_num 1720) 2020-06-12T02:37:39.452 INFO:tasks.rados.rados.0.smithi107.stdout:1726: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:39.453 INFO:tasks.rados.rados.0.smithi107.stdout:1720: done (7 left) 2020-06-12T02:37:39.453 INFO:tasks.rados.rados.0.smithi107.stdout:1721: done (6 left) 2020-06-12T02:37:39.453 INFO:tasks.rados.rados.0.smithi107.stdout:1722: done (5 left) 2020-06-12T02:37:39.453 INFO:tasks.rados.rados.0.smithi107.stdout:1723: done (4 left) 2020-06-12T02:37:39.453 INFO:tasks.rados.rados.0.smithi107.stdout:1724: done (3 left) 2020-06-12T02:37:39.454 INFO:tasks.rados.rados.0.smithi107.stdout:1728: done (2 left) 2020-06-12T02:37:39.454 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:37:39.454 INFO:tasks.rados.rados.0.smithi107.stdout:1729: seq_num 1728 ranges {0=147456} 2020-06-12T02:37:39.454 INFO:tasks.rados.rados.0.smithi107.stdout:1729: writing smithi10729407-6 from 0 to 147456 tid 1 2020-06-12T02:37:39.454 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:37:39.454 INFO:tasks.rados.rados.0.smithi107.stdout:1730: seq_num 1729 ranges {0=163840} 2020-06-12T02:37:39.455 INFO:tasks.rados.rados.0.smithi107.stdout:1730: writing smithi10729407-19 from 0 to 163840 tid 1 2020-06-12T02:37:39.455 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:37:39.455 INFO:tasks.rados.rados.0.smithi107.stdout:1731: seq_num 1730 ranges {24576=180224} 2020-06-12T02:37:39.455 INFO:tasks.rados.rados.0.smithi107.stdout:1731: writing smithi10729407-14 from 24576 to 204800 tid 1 2020-06-12T02:37:39.455 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:37:39.456 INFO:tasks.rados.rados.0.smithi107.stdout:1732: seq_num 1731 ranges {8192=196608} 2020-06-12T02:37:39.456 INFO:tasks.rados.rados.0.smithi107.stdout:1732: writing smithi10729407-3 from 8192 to 204800 tid 1 2020-06-12T02:37:39.456 INFO:tasks.rados.rados.0.smithi107.stdout:1733: delete oid 15 current snap is 0 2020-06-12T02:37:39.457 INFO:tasks.rados.rados.0.smithi107.stdout:1726: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:39.457 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 344 (ObjNum 1725 snap 0 seq_num 1725) dirty exists 2020-06-12T02:37:39.457 INFO:tasks.rados.rados.0.smithi107.stdout:1726: left oid 13 (ObjNum 1725 snap 0 seq_num 1725) 2020-06-12T02:37:39.457 INFO:tasks.rados.rados.0.smithi107.stdout:1727: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:39.457 INFO:tasks.rados.rados.0.smithi107.stdout:1729: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:39.458 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:37:39.460 INFO:tasks.rados.rados.0.smithi107.stdout:1727: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:39.460 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 308 (ObjNum 1726 snap 0 seq_num 1726) dirty exists 2020-06-12T02:37:39.461 INFO:tasks.rados.rados.0.smithi107.stdout:1727: left oid 11 (ObjNum 1726 snap 0 seq_num 1726) 2020-06-12T02:37:39.461 INFO:tasks.rados.rados.0.smithi107.stdout:1726: done (6 left) 2020-06-12T02:37:39.461 INFO:tasks.rados.rados.0.smithi107.stdout:1727: done (5 left) 2020-06-12T02:37:39.461 INFO:tasks.rados.rados.0.smithi107.stdout:1733: done (4 left) 2020-06-12T02:37:39.461 INFO:tasks.rados.rados.0.smithi107.stdout:1734: delete oid 15 current snap is 0 2020-06-12T02:37:39.462 INFO:tasks.rados.rados.0.smithi107.stdout:1730: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:39.462 INFO:tasks.rados.rados.0.smithi107.stdout:1730: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:39.462 INFO:tasks.rados.rados.0.smithi107.stdout:1729: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:39.462 INFO:tasks.rados.rados.0.smithi107.stdout:1729: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:39.462 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 349 (ObjNum 1728 snap 0 seq_num 1728) dirty exists 2020-06-12T02:37:39.462 INFO:tasks.rados.rados.0.smithi107.stdout:1729: left oid 6 (ObjNum 1728 snap 0 seq_num 1728) 2020-06-12T02:37:39.463 INFO:tasks.rados.rados.0.smithi107.stdout:1730: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:39.463 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 269 (ObjNum 1729 snap 0 seq_num 1729) dirty exists 2020-06-12T02:37:39.463 INFO:tasks.rados.rados.0.smithi107.stdout:1730: left oid 19 (ObjNum 1729 snap 0 seq_num 1729) 2020-06-12T02:37:39.463 INFO:tasks.rados.rados.0.smithi107.stdout:1729: done (4 left) 2020-06-12T02:37:39.463 INFO:tasks.rados.rados.0.smithi107.stdout:1730: done (3 left) 2020-06-12T02:37:39.464 INFO:tasks.rados.rados.0.smithi107.stdout:1734: done (2 left) 2020-06-12T02:37:39.464 INFO:tasks.rados.rados.0.smithi107.stdout:1735: delete oid 20 current snap is 0 2020-06-12T02:37:39.464 INFO:tasks.rados.rados.0.smithi107.stdout:1732: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:39.464 INFO:tasks.rados.rados.0.smithi107.stdout:1735: done (2 left) 2020-06-12T02:37:39.464 INFO:tasks.rados.rados.0.smithi107.stdout:1736: delete oid 16 current snap is 0 2020-06-12T02:37:39.465 INFO:tasks.rados.rados.0.smithi107.stdout:1732: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:39.465 INFO:tasks.rados.rados.0.smithi107.stdout:1731: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:39.465 INFO:tasks.rados.rados.0.smithi107.stdout:1731: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:39.467 INFO:tasks.rados.rados.0.smithi107.stdout:1732: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:39.467 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 309 (ObjNum 1731 snap 0 seq_num 1731) dirty exists 2020-06-12T02:37:39.467 INFO:tasks.rados.rados.0.smithi107.stdout:1732: left oid 3 (ObjNum 1731 snap 0 seq_num 1731) 2020-06-12T02:37:39.467 INFO:tasks.rados.rados.0.smithi107.stdout:1731: finishing write tid 4 to smithi10729407-14 2020-06-12T02:37:39.468 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 310 (ObjNum 1730 snap 0 seq_num 1730) dirty exists 2020-06-12T02:37:39.468 INFO:tasks.rados.rados.0.smithi107.stdout:1731: left oid 14 (ObjNum 1730 snap 0 seq_num 1730) 2020-06-12T02:37:39.468 INFO:tasks.rados.rados.0.smithi107.stdout:1731: done (2 left) 2020-06-12T02:37:39.468 INFO:tasks.rados.rados.0.smithi107.stdout:1732: done (1 left) 2020-06-12T02:37:39.468 INFO:tasks.rados.rados.0.smithi107.stdout:1736: done (0 left) 2020-06-12T02:37:39.468 INFO:tasks.rados.rados.0.smithi107.stdout:1737: delete oid 10 current snap is 0 2020-06-12T02:37:39.469 INFO:tasks.rados.rados.0.smithi107.stdout:1737: done (0 left) 2020-06-12T02:37:39.469 INFO:tasks.rados.rados.0.smithi107.stdout:1738: delete oid 3 current snap is 0 2020-06-12T02:37:39.470 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2020-06-12T02:37:39.473 INFO:tasks.rados.rados.0.smithi107.stdout:1738: done (0 left) 2020-06-12T02:37:39.473 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:37:39.473 INFO:tasks.rados.rados.0.smithi107.stdout:1739: seq_num 1738 ranges {0=311296} 2020-06-12T02:37:39.476 INFO:tasks.rados.rados.0.smithi107.stdout:1739: writing smithi10729407-17 from 0 to 311296 tid 1 2020-06-12T02:37:39.476 INFO:tasks.rados.rados.0.smithi107.stdout:1740: delete oid 14 current snap is 0 2020-06-12T02:37:39.477 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:37:39.481 INFO:tasks.rados.rados.0.smithi107.stdout:1739: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:39.481 INFO:tasks.rados.rados.0.smithi107.stdout:1740: done (1 left) 2020-06-12T02:37:39.481 INFO:tasks.rados.rados.0.smithi107.stdout:1741: delete oid 3 current snap is 0 2020-06-12T02:37:39.482 INFO:tasks.rados.rados.0.smithi107.stdout:1739: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:39.483 INFO:tasks.rados.rados.0.smithi107.stdout:1739: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:39.484 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 331 (ObjNum 1738 snap 0 seq_num 1738) dirty exists 2020-06-12T02:37:39.484 INFO:tasks.rados.rados.0.smithi107.stdout:1739: left oid 17 (ObjNum 1738 snap 0 seq_num 1738) 2020-06-12T02:37:39.485 INFO:tasks.rados.rados.0.smithi107.stdout:1739: done (1 left) 2020-06-12T02:37:39.486 INFO:tasks.rados.rados.0.smithi107.stdout:1741: done (0 left) 2020-06-12T02:37:39.486 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:37:39.486 INFO:tasks.rados.rados.0.smithi107.stdout:1742: seq_num 1741 ranges {0=360448} 2020-06-12T02:37:39.487 INFO:tasks.rados.rados.0.smithi107.stdout:1742: writing smithi10729407-15 from 0 to 360448 tid 1 2020-06-12T02:37:39.487 INFO:tasks.rados.rados.0.smithi107.stdout:1743: delete oid 3 current snap is 0 2020-06-12T02:37:39.489 INFO:tasks.rados.rados.0.smithi107.stdout:1743: done (1 left) 2020-06-12T02:37:39.490 INFO:tasks.rados.rados.0.smithi107.stdout:1744: delete oid 10 current snap is 0 2020-06-12T02:37:39.491 INFO:tasks.rados.rados.0.smithi107.stdout:1744: done (1 left) 2020-06-12T02:37:39.491 INFO:tasks.rados.rados.0.smithi107.stdout:1745: delete oid 4 current snap is 0 2020-06-12T02:37:39.492 INFO:tasks.rados.rados.0.smithi107.stdout:1742: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:39.493 INFO:tasks.rados.rados.0.smithi107.stdout:1745: done (1 left) 2020-06-12T02:37:39.495 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:37:39.495 INFO:tasks.rados.rados.0.smithi107.stdout:1746: seq_num 1745 ranges {0=434176} 2020-06-12T02:37:39.496 INFO:tasks.rados.rados.0.smithi107.stdout:1746: writing smithi10729407-4 from 0 to 434176 tid 1 2020-06-12T02:37:39.497 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:37:39.497 INFO:tasks.rados.rados.0.smithi107.stdout:1747: seq_num 1746 ranges {0=450560} 2020-06-12T02:37:39.499 INFO:tasks.rados.rados.0.smithi107.stdout:1747: writing smithi10729407-3 from 0 to 450560 tid 1 2020-06-12T02:37:39.499 INFO:tasks.rados.rados.0.smithi107.stdout:1748: delete oid 10 current snap is 0 2020-06-12T02:37:39.499 INFO:tasks.rados.rados.0.smithi107.stdout:1742: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:39.500 INFO:tasks.rados.rados.0.smithi107.stdout:1742: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:39.500 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 298 (ObjNum 1741 snap 0 seq_num 1741) dirty exists 2020-06-12T02:37:39.501 INFO:tasks.rados.rados.0.smithi107.stdout:1742: left oid 15 (ObjNum 1741 snap 0 seq_num 1741) 2020-06-12T02:37:39.502 INFO:tasks.rados.rados.0.smithi107.stdout:1746: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:39.502 INFO:tasks.rados.rados.0.smithi107.stdout:1746: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:39.502 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:37:39.503 INFO:tasks.rados.rados.0.smithi107.stdout:1742: done (3 left) 2020-06-12T02:37:39.504 INFO:tasks.rados.rados.0.smithi107.stdout:1748: done (2 left) 2020-06-12T02:37:39.504 INFO:tasks.rados.rados.0.smithi107.stdout:1749: delete oid 5 current snap is 0 2020-06-12T02:37:39.504 INFO:tasks.rados.rados.0.smithi107.stdout:1746: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:39.505 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 334 (ObjNum 1745 snap 0 seq_num 1745) dirty exists 2020-06-12T02:37:39.505 INFO:tasks.rados.rados.0.smithi107.stdout:1746: left oid 4 (ObjNum 1745 snap 0 seq_num 1745) 2020-06-12T02:37:39.505 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-06-12T02:37:39.506 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-06-12T02:37:39.506 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-06-12T02:37:39.506 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-06-12T02:37:39.507 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-06-12T02:37:39.507 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-06-12T02:37:39.507 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-06-12T02:37:39.508 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.7 2020-06-12T02:37:39.508 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: problem getting command descriptions from osd.7 2020-06-12T02:37:39.511 INFO:tasks.rados.rados.0.smithi107.stdout:1747: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:39.511 INFO:tasks.rados.rados.0.smithi107.stdout:1746: done (2 left) 2020-06-12T02:37:39.511 INFO:tasks.rados.rados.0.smithi107.stdout:1749: done (1 left) 2020-06-12T02:37:39.512 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:37:39.512 INFO:tasks.rados.rados.0.smithi107.stdout:1750: seq_num 1749 ranges {0=499712} 2020-06-12T02:37:39.512 INFO:tasks.rados.rados.0.smithi107.stdout:1750: writing smithi10729407-2 from 0 to 499712 tid 1 2020-06-12T02:37:39.513 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:37:39.513 INFO:tasks.rados.rados.0.smithi107.stdout:1751: seq_num 1750 ranges {311296=516096} 2020-06-12T02:37:39.513 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-06-12T02:37:39.514 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-06-12T02:37:39.516 INFO:tasks.rados.rados.0.smithi107.stdout:1751: writing smithi10729407-17 from 311296 to 827392 tid 1 2020-06-12T02:37:39.516 INFO:tasks.rados.rados.0.smithi107.stdout:1752: delete oid 1 current snap is 0 2020-06-12T02:37:39.516 INFO:tasks.rados.rados.0.smithi107.stdout:1747: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:39.517 INFO:tasks.rados.rados.0.smithi107.stdout:1747: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:39.517 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 315 (ObjNum 1746 snap 0 seq_num 1746) dirty exists 2020-06-12T02:37:39.517 INFO:tasks.rados.rados.0.smithi107.stdout:1747: left oid 3 (ObjNum 1746 snap 0 seq_num 1746) 2020-06-12T02:37:39.518 INFO:tasks.rados.rados.0.smithi107.stdout:1750: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:39.518 INFO:tasks.rados.rados.0.smithi107.stdout:1750: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:39.518 INFO:tasks.rados.rados.0.smithi107.stdout:1750: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:39.519 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 163 (ObjNum 1749 snap 0 seq_num 1749) dirty exists 2020-06-12T02:37:39.519 INFO:tasks.rados.rados.0.smithi107.stdout:1750: left oid 2 (ObjNum 1749 snap 0 seq_num 1749) 2020-06-12T02:37:39.520 INFO:tasks.rados.rados.0.smithi107.stdout:1747: done (3 left) 2020-06-12T02:37:39.520 INFO:tasks.rados.rados.0.smithi107.stdout:1750: done (2 left) 2020-06-12T02:37:39.521 INFO:tasks.rados.rados.0.smithi107.stdout:1752: done (1 left) 2020-06-12T02:37:39.521 INFO:tasks.rados.rados.0.smithi107.stdout:1753: delete oid 8 current snap is 0 2020-06-12T02:37:39.521 INFO:tasks.rados.rados.0.smithi107.stdout:1753: done (1 left) 2020-06-12T02:37:39.521 INFO:tasks.rados.rados.0.smithi107.stdout:1754: delete oid 9 current snap is 0 2020-06-12T02:37:39.522 INFO:tasks.rados.rados.0.smithi107.stdout:1751: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:39.522 INFO:tasks.rados.rados.0.smithi107.stdout:1751: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:39.523 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:37:39.517+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:37:39.523 INFO:tasks.rados.rados.0.smithi107.stdout:1754: done (1 left) 2020-06-12T02:37:39.523 INFO:tasks.rados.rados.0.smithi107.stdout:1755: delete oid 10 current snap is 0 2020-06-12T02:37:39.523 INFO:tasks.rados.rados.0.smithi107.stdout:1751: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:39.524 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 336 (ObjNum 1750 snap 0 seq_num 1750) dirty exists 2020-06-12T02:37:39.524 INFO:tasks.rados.rados.0.smithi107.stdout:1751: left oid 17 (ObjNum 1750 snap 0 seq_num 1750) 2020-06-12T02:37:39.524 INFO:tasks.rados.rados.0.smithi107.stdout:1751: done (1 left) 2020-06-12T02:37:39.524 INFO:tasks.rados.rados.0.smithi107.stdout:1755: done (0 left) 2020-06-12T02:37:39.524 INFO:tasks.rados.rados.0.smithi107.stdout:1756: delete oid 14 current snap is 0 2020-06-12T02:37:39.525 INFO:tasks.rados.rados.0.smithi107.stdout:1756: done (0 left) 2020-06-12T02:37:39.525 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:37:39.525 INFO:tasks.rados.rados.0.smithi107.stdout:1757: seq_num 1756 ranges {0=614400} 2020-06-12T02:37:39.527 INFO:tasks.rados.rados.0.smithi107.stdout:1757: writing smithi10729407-10 from 0 to 614400 tid 1 2020-06-12T02:37:39.528 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:37:39.528 INFO:tasks.rados.rados.0.smithi107.stdout:1758: seq_num 1757 ranges {434176=630784} 2020-06-12T02:37:39.535 INFO:tasks.rados.rados.0.smithi107.stdout:1758: writing smithi10729407-4 from 434176 to 1064960 tid 1 2020-06-12T02:37:39.535 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:37:39.535 INFO:tasks.rados.rados.0.smithi107.stdout:1759: seq_num 1758 ranges {0=647168} 2020-06-12T02:37:39.538 INFO:tasks.rados.rados.0.smithi107.stdout:1759: writing smithi10729407-14 from 0 to 647168 tid 1 2020-06-12T02:37:39.539 INFO:tasks.rados.rados.0.smithi107.stdout:1760: delete oid 20 current snap is 0 2020-06-12T02:37:39.539 INFO:tasks.rados.rados.0.smithi107.stdout:1757: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:39.539 INFO:tasks.rados.rados.0.smithi107.stdout:1757: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:39.539 INFO:tasks.rados.rados.0.smithi107.stdout:1757: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:39.539 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 165 (ObjNum 1756 snap 0 seq_num 1756) dirty exists 2020-06-12T02:37:39.540 INFO:tasks.rados.rados.0.smithi107.stdout:1757: left oid 10 (ObjNum 1756 snap 0 seq_num 1756) 2020-06-12T02:37:39.540 INFO:tasks.rados.rados.0.smithi107.stdout:1758: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:39.541 INFO:tasks.rados.rados.0.smithi107.stdout:1757: done (3 left) 2020-06-12T02:37:39.541 INFO:tasks.rados.rados.0.smithi107.stdout:1760: done (2 left) 2020-06-12T02:37:39.541 INFO:tasks.rados.rados.0.smithi107.stdout:1761: delete oid 8 current snap is 0 2020-06-12T02:37:39.548 INFO:tasks.rados.rados.0.smithi107.stdout:1758: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:39.549 INFO:tasks.rados.rados.0.smithi107.stdout:1761: done (2 left) 2020-06-12T02:37:39.549 INFO:tasks.rados.rados.0.smithi107.stdout:1762: delete oid 11 current snap is 0 2020-06-12T02:37:39.549 INFO:tasks.rados.rados.0.smithi107.stdout:1758: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:39.549 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 338 (ObjNum 1757 snap 0 seq_num 1757) dirty exists 2020-06-12T02:37:39.549 INFO:tasks.rados.rados.0.smithi107.stdout:1758: left oid 4 (ObjNum 1757 snap 0 seq_num 1757) 2020-06-12T02:37:39.550 INFO:tasks.rados.rados.0.smithi107.stdout:1759: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:39.550 INFO:tasks.rados.rados.0.smithi107.stdout:1759: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:39.550 INFO:tasks.rados.rados.0.smithi107.stdout:1758: done (2 left) 2020-06-12T02:37:39.550 INFO:tasks.rados.rados.0.smithi107.stdout:1762: done (1 left) 2020-06-12T02:37:39.550 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:37:39.551 INFO:tasks.rados.rados.0.smithi107.stdout:1763: seq_num 1762 ranges {614400=712704} 2020-06-12T02:37:39.555 INFO:tasks.rados.rados.0.smithi107.stdout:1763: writing smithi10729407-10 from 614400 to 1327104 tid 1 2020-06-12T02:37:39.555 INFO:tasks.rados.rados.0.smithi107.stdout:1764: delete oid 9 current snap is 0 2020-06-12T02:37:39.555 INFO:tasks.rados.rados.0.smithi107.stdout:1759: finishing write tid 4 to smithi10729407-14 2020-06-12T02:37:39.555 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 314 (ObjNum 1758 snap 0 seq_num 1758) dirty exists 2020-06-12T02:37:39.556 INFO:tasks.rados.rados.0.smithi107.stdout:1759: left oid 14 (ObjNum 1758 snap 0 seq_num 1758) 2020-06-12T02:37:39.557 INFO:tasks.rados.rados.0.smithi107.stdout:1759: done (2 left) 2020-06-12T02:37:39.557 INFO:tasks.rados.rados.0.smithi107.stdout:1764: done (1 left) 2020-06-12T02:37:39.557 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:37:39.558 INFO:tasks.rados.rados.0.smithi107.stdout:1765: seq_num 1764 ranges {0=753664} 2020-06-12T02:37:39.562 INFO:tasks.rados.rados.0.smithi107.stdout:1765: writing smithi10729407-8 from 0 to 753664 tid 1 2020-06-12T02:37:39.562 INFO:tasks.rados.rados.0.smithi107.stdout:1766: delete oid 19 current snap is 0 2020-06-12T02:37:39.562 INFO:tasks.rados.rados.0.smithi107.stdout:1763: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:39.562 INFO:tasks.rados.rados.0.smithi107.stdout:1763: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:39.563 INFO:tasks.rados.rados.0.smithi107.stdout:1763: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:39.563 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 167 (ObjNum 1762 snap 0 seq_num 1762) dirty exists 2020-06-12T02:37:39.563 INFO:tasks.rados.rados.0.smithi107.stdout:1763: left oid 10 (ObjNum 1762 snap 0 seq_num 1762) 2020-06-12T02:37:39.564 INFO:tasks.rados.rados.0.smithi107.stdout:1763: done (2 left) 2020-06-12T02:37:39.564 INFO:tasks.rados.rados.0.smithi107.stdout:1766: done (1 left) 2020-06-12T02:37:39.564 INFO:tasks.rados.rados.0.smithi107.stdout:1767: delete oid 20 current snap is 0 2020-06-12T02:37:39.570 INFO:tasks.rados.rados.0.smithi107.stdout:1765: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:39.570 INFO:tasks.rados.rados.0.smithi107.stdout:1765: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:39.570 INFO:tasks.rados.rados.0.smithi107.stdout:1767: done (1 left) 2020-06-12T02:37:39.570 INFO:tasks.rados.rados.0.smithi107.stdout:1768: delete oid 2 current snap is 0 2020-06-12T02:37:39.572 INFO:tasks.rados.rados.0.smithi107.stdout:1768: done (1 left) 2020-06-12T02:37:39.572 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:37:39.572 INFO:tasks.rados.rados.0.smithi107.stdout:1769: seq_num 1768 ranges {1064960=417792,1482752=401408} 2020-06-12T02:37:39.580 INFO:tasks.rados.rados.0.smithi107.stdout:1769: writing smithi10729407-4 from 1064960 to 1482752 tid 1 2020-06-12T02:37:39.582 INFO:tasks.rados.rados.0.smithi107.stdout:1769: writing smithi10729407-4 from 1482752 to 1884160 tid 2 2020-06-12T02:37:39.583 INFO:tasks.rados.rados.0.smithi107.stdout:1770: delete oid 19 current snap is 0 2020-06-12T02:37:39.583 INFO:tasks.rados.rados.0.smithi107.stdout:1765: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:39.583 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 302 (ObjNum 1764 snap 0 seq_num 1764) dirty exists 2020-06-12T02:37:39.583 INFO:tasks.rados.rados.0.smithi107.stdout:1765: left oid 8 (ObjNum 1764 snap 0 seq_num 1764) 2020-06-12T02:37:39.586 INFO:tasks.rados.rados.0.smithi107.stdout:1769: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:39.586 INFO:tasks.rados.rados.0.smithi107.stdout:1765: done (2 left) 2020-06-12T02:37:39.586 INFO:tasks.rados.rados.0.smithi107.stdout:1770: done (1 left) 2020-06-12T02:37:39.587 INFO:tasks.rados.rados.0.smithi107.stdout:1771: delete oid 14 current snap is 0 2020-06-12T02:37:39.587 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:39.587 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:39.588 INFO:tasks.rados.rados.0.smithi107.stdout:1771: done (1 left) 2020-06-12T02:37:39.589 INFO:tasks.rados.rados.0.smithi107.stdout:1772: delete oid 3 current snap is 0 2020-06-12T02:37:39.594 INFO:tasks.rados.rados.0.smithi107.stdout:1769: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:39.595 INFO:tasks.rados.rados.0.smithi107.stdout:1769: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:39.595 INFO:tasks.rados.rados.0.smithi107.stdout:1772: done (1 left) 2020-06-12T02:37:39.595 INFO:tasks.rados.rados.0.smithi107.stdout:1773: delete oid 14 current snap is 0 2020-06-12T02:37:39.596 INFO:tasks.rados.rados.0.smithi107.stdout:1769: finishing write tid 5 to smithi10729407-4 2020-06-12T02:37:39.596 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 341 (ObjNum 1768 snap 0 seq_num 1768) dirty exists 2020-06-12T02:37:39.596 INFO:tasks.rados.rados.0.smithi107.stdout:1769: left oid 4 (ObjNum 1768 snap 0 seq_num 1768) 2020-06-12T02:37:39.598 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-06-12T02:37:39.599 INFO:tasks.rados.rados.0.smithi107.stdout:1769: done (1 left) 2020-06-12T02:37:39.599 INFO:tasks.rados.rados.0.smithi107.stdout:1773: done (0 left) 2020-06-12T02:37:39.599 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:37:39.599 INFO:tasks.rados.rados.0.smithi107.stdout:1774: seq_num 1773 ranges {0=499712,499712=401408} 2020-06-12T02:37:39.600 INFO:tasks.rados.rados.0.smithi107.stdout:1774: writing smithi10729407-12 from 0 to 499712 tid 1 2020-06-12T02:37:39.602 INFO:tasks.rados.rados.0.smithi107.stdout:1774: writing smithi10729407-12 from 499712 to 901120 tid 2 2020-06-12T02:37:39.602 INFO:tasks.rados.rados.0.smithi107.stdout:1775: delete oid 14 current snap is 0 2020-06-12T02:37:39.604 INFO:tasks.rados.rados.0.smithi107.stdout:1775: done (1 left) 2020-06-12T02:37:39.604 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:37:39.604 INFO:tasks.rados.rados.0.smithi107.stdout:1776: seq_num 1775 ranges {1884160=532480,2416640=401408} 2020-06-12T02:37:39.617 INFO:tasks.rados.rados.0.smithi107.stdout:1776: writing smithi10729407-4 from 1884160 to 2416640 tid 1 2020-06-12T02:37:39.619 INFO:tasks.rados.rados.0.smithi107.stdout:1776: writing smithi10729407-4 from 2416640 to 2818048 tid 2 2020-06-12T02:37:39.620 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:39.620 INFO:tasks.rados.rados.0.smithi107.stdout:1777: seq_num 1776 ranges {0=548864,548864=401408} 2020-06-12T02:37:39.622 INFO:tasks.rados.rados.0.smithi107.stdout:1777: writing smithi10729407-2 from 0 to 548864 tid 1 2020-06-12T02:37:39.631 INFO:tasks.rados.rados.0.smithi107.stdout:1777: writing smithi10729407-2 from 548864 to 950272 tid 2 2020-06-12T02:37:39.631 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:37:39.632 INFO:tasks.rados.rados.0.smithi107.stdout:1778: seq_num 1777 ranges {753664=565248,1318912=401408} 2020-06-12T02:37:39.638 INFO:tasks.rados.rados.0.smithi107.stdout:1778: writing smithi10729407-8 from 753664 to 1318912 tid 1 2020-06-12T02:37:39.638 INFO:tasks.rados.rados.0.smithi107.stdout:1778: writing smithi10729407-8 from 1318912 to 1720320 tid 2 2020-06-12T02:37:39.639 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:37:39.639 INFO:tasks.rados.rados.0.smithi107.stdout:1779: seq_num 1778 ranges {360448=581632,942080=401408} 2020-06-12T02:37:39.640 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:39.636+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:37:39.641 INFO:tasks.rados.rados.0.smithi107.stdout:1779: writing smithi10729407-15 from 360448 to 942080 tid 1 2020-06-12T02:37:39.646 INFO:tasks.rados.rados.0.smithi107.stdout:1779: writing smithi10729407-15 from 942080 to 1343488 tid 2 2020-06-12T02:37:39.647 INFO:tasks.rados.rados.0.smithi107.stdout:1780: delete oid 17 current snap is 0 2020-06-12T02:37:39.647 INFO:tasks.rados.rados.0.smithi107.stdout:1774: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:39.647 INFO:tasks.rados.rados.0.smithi107.stdout:1774: finishing write tid 2 to smithi10729407-12 2020-06-12T02:37:39.647 INFO:tasks.rados.rados.0.smithi107.stdout:1774: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:39.648 INFO:tasks.rados.rados.0.smithi107.stdout:1774: finishing write tid 5 to smithi10729407-12 2020-06-12T02:37:39.648 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 274 (ObjNum 1773 snap 0 seq_num 1773) dirty exists 2020-06-12T02:37:39.648 INFO:tasks.rados.rados.0.smithi107.stdout:1774: left oid 12 (ObjNum 1773 snap 0 seq_num 1773) 2020-06-12T02:37:39.649 INFO:tasks.rados.rados.0.smithi107.stdout:1776: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:39.649 INFO:tasks.rados.rados.0.smithi107.stdout:1776: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:39.649 INFO:tasks.rados.rados.0.smithi107.stdout:1776: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:39.649 INFO:tasks.rados.rados.0.smithi107.stdout:1776: finishing write tid 5 to smithi10729407-4 2020-06-12T02:37:39.650 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 344 (ObjNum 1775 snap 0 seq_num 1775) dirty exists 2020-06-12T02:37:39.650 INFO:tasks.rados.rados.0.smithi107.stdout:1776: left oid 4 (ObjNum 1775 snap 0 seq_num 1775) 2020-06-12T02:37:39.650 INFO:tasks.rados.rados.0.smithi107.stdout:1777: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:39.650 INFO:tasks.rados.rados.0.smithi107.stdout:1777: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:39.650 INFO:tasks.rados.rados.0.smithi107.stdout:1777: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:39.650 INFO:tasks.rados.rados.0.smithi107.stdout:1777: finishing write tid 5 to smithi10729407-2 2020-06-12T02:37:39.651 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 167 (ObjNum 1776 snap 0 seq_num 1776) dirty exists 2020-06-12T02:37:39.651 INFO:tasks.rados.rados.0.smithi107.stdout:1777: left oid 2 (ObjNum 1776 snap 0 seq_num 1776) 2020-06-12T02:37:39.651 INFO:tasks.rados.rados.0.smithi107.stdout:1778: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:39.652 INFO:tasks.rados.rados.0.smithi107.stdout:1778: finishing write tid 2 to smithi10729407-8 2020-06-12T02:37:39.652 INFO:tasks.rados.rados.0.smithi107.stdout:1778: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:39.653 INFO:tasks.rados.rados.0.smithi107.stdout:1774: done (5 left) 2020-06-12T02:37:39.659 INFO:tasks.rados.rados.0.smithi107.stdout:1776: done (4 left) 2020-06-12T02:37:39.659 INFO:tasks.rados.rados.0.smithi107.stdout:1777: done (3 left) 2020-06-12T02:37:39.659 INFO:tasks.rados.rados.0.smithi107.stdout:1780: done (2 left) 2020-06-12T02:37:39.660 INFO:tasks.rados.rados.0.smithi107.stdout:1781: delete oid 20 current snap is 0 2020-06-12T02:37:39.660 INFO:tasks.rados.rados.0.smithi107.stdout:1778: finishing write tid 5 to smithi10729407-8 2020-06-12T02:37:39.660 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 305 (ObjNum 1777 snap 0 seq_num 1777) dirty exists 2020-06-12T02:37:39.661 INFO:tasks.rados.rados.0.smithi107.stdout:1778: left oid 8 (ObjNum 1777 snap 0 seq_num 1777) 2020-06-12T02:37:39.661 INFO:tasks.rados.rados.0.smithi107.stdout:1778: done (2 left) 2020-06-12T02:37:39.661 INFO:tasks.rados.rados.0.smithi107.stdout:1781: done (1 left) 2020-06-12T02:37:39.661 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:37:39.661 INFO:tasks.rados.rados.0.smithi107.stdout:1782: seq_num 1781 ranges {2097152=630784,2727936=401408} 2020-06-12T02:37:39.666 INFO:tasks.rados.rados.0.smithi107.stdout:1782: writing smithi10729407-18 from 2097152 to 2727936 tid 1 2020-06-12T02:37:39.669 INFO:tasks.rados.rados.0.smithi107.stdout:1782: writing smithi10729407-18 from 2727936 to 3129344 tid 2 2020-06-12T02:37:39.669 INFO:tasks.rados.rados.0.smithi107.stdout:1783: delete oid 7 current snap is 0 2020-06-12T02:37:39.669 INFO:tasks.rados.rados.0.smithi107.stdout:1779: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:39.670 INFO:tasks.rados.rados.0.smithi107.stdout:1779: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:39.670 INFO:tasks.rados.rados.0.smithi107.stdout:1779: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:39.670 INFO:tasks.rados.rados.0.smithi107.stdout:1779: finishing write tid 5 to smithi10729407-15 2020-06-12T02:37:39.670 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 308 (ObjNum 1778 snap 0 seq_num 1778) dirty exists 2020-06-12T02:37:39.670 INFO:tasks.rados.rados.0.smithi107.stdout:1779: left oid 15 (ObjNum 1778 snap 0 seq_num 1778) 2020-06-12T02:37:39.671 INFO:tasks.rados.rados.0.smithi107.stdout:1779: done (2 left) 2020-06-12T02:37:39.671 INFO:tasks.rados.rados.0.smithi107.stdout:1783: done (1 left) 2020-06-12T02:37:39.671 INFO:tasks.rados.rados.0.smithi107.stdout:1784: delete oid 10 current snap is 0 2020-06-12T02:37:39.674 INFO:tasks.rados.rados.0.smithi107.stdout:1782: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:39.674 INFO:tasks.rados.rados.0.smithi107.stdout:1784: done (1 left) 2020-06-12T02:37:39.674 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:37:39.675 INFO:tasks.rados.rados.0.smithi107.stdout:1785: seq_num 1784 ranges {0=688128,688128=401408} 2020-06-12T02:37:39.679 INFO:tasks.rados.rados.0.smithi107.stdout:1785: writing smithi10729407-19 from 0 to 688128 tid 1 2020-06-12T02:37:39.681 INFO:tasks.rados.rados.0.smithi107.stdout:1785: writing smithi10729407-19 from 688128 to 1089536 tid 2 2020-06-12T02:37:39.682 INFO:tasks.rados.rados.0.smithi107.stdout:1786: delete oid 3 current snap is 0 2020-06-12T02:37:39.683 INFO:tasks.rados.rados.0.smithi107.stdout:1782: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:39.684 INFO:tasks.rados.rados.0.smithi107.stdout:1782: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:39.684 INFO:tasks.rados.rados.0.smithi107.stdout:1782: finishing write tid 5 to smithi10729407-18 2020-06-12T02:37:39.684 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 311 (ObjNum 1781 snap 0 seq_num 1781) dirty exists 2020-06-12T02:37:39.685 INFO:tasks.rados.rados.0.smithi107.stdout:1782: left oid 18 (ObjNum 1781 snap 0 seq_num 1781) 2020-06-12T02:37:39.685 INFO:tasks.rados.rados.0.smithi107.stdout:1785: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:39.686 INFO:tasks.rados.rados.0.smithi107.stdout:1782: done (2 left) 2020-06-12T02:37:39.686 INFO:tasks.rados.rados.0.smithi107.stdout:1786: done (1 left) 2020-06-12T02:37:39.686 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:37:39.687 INFO:tasks.rados.rados.0.smithi107.stdout:1787: seq_num 1786 ranges {0=720896,720896=401408} 2020-06-12T02:37:39.690 INFO:tasks.rados.rados.0.smithi107.stdout:1787: writing smithi10729407-17 from 0 to 720896 tid 1 2020-06-12T02:37:39.693 INFO:tasks.rados.rados.0.smithi107.stdout:1787: writing smithi10729407-17 from 720896 to 1122304 tid 2 2020-06-12T02:37:39.693 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:37:39.693 INFO:tasks.rados.rados.0.smithi107.stdout:1788: seq_num 1787 ranges {2818048=737280,3555328=401408} 2020-06-12T02:37:39.723 INFO:tasks.rados.rados.0.smithi107.stdout:1788: writing smithi10729407-4 from 2818048 to 3555328 tid 1 2020-06-12T02:37:39.724 INFO:tasks.rados.rados.0.smithi107.stdout:1788: writing smithi10729407-4 from 3555328 to 3956736 tid 2 2020-06-12T02:37:39.724 INFO:tasks.rados.rados.0.smithi107.stdout:1789: delete oid 14 current snap is 0 2020-06-12T02:37:39.724 INFO:tasks.rados.rados.0.smithi107.stdout:1785: finishing write tid 2 to smithi10729407-19 2020-06-12T02:37:39.724 INFO:tasks.rados.rados.0.smithi107.stdout:1785: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:39.725 INFO:tasks.rados.rados.0.smithi107.stdout:1785: finishing write tid 5 to smithi10729407-19 2020-06-12T02:37:39.725 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 277 (ObjNum 1784 snap 0 seq_num 1784) dirty exists 2020-06-12T02:37:39.725 INFO:tasks.rados.rados.0.smithi107.stdout:1785: left oid 19 (ObjNum 1784 snap 0 seq_num 1784) 2020-06-12T02:37:39.725 INFO:tasks.rados.rados.0.smithi107.stdout:1787: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:39.725 INFO:tasks.rados.rados.0.smithi107.stdout:1787: finishing write tid 2 to smithi10729407-17 2020-06-12T02:37:39.725 INFO:tasks.rados.rados.0.smithi107.stdout:1787: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:39.726 INFO:tasks.rados.rados.0.smithi107.stdout:1787: finishing write tid 5 to smithi10729407-17 2020-06-12T02:37:39.726 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 348 (ObjNum 1786 snap 0 seq_num 1786) dirty exists 2020-06-12T02:37:39.726 INFO:tasks.rados.rados.0.smithi107.stdout:1787: left oid 17 (ObjNum 1786 snap 0 seq_num 1786) 2020-06-12T02:37:39.726 INFO:tasks.rados.rados.0.smithi107.stdout:1785: done (3 left) 2020-06-12T02:37:39.726 INFO:tasks.rados.rados.0.smithi107.stdout:1787: done (2 left) 2020-06-12T02:37:39.727 INFO:tasks.rados.rados.0.smithi107.stdout:1789: done (1 left) 2020-06-12T02:37:39.727 INFO:tasks.rados.rados.0.smithi107.stdout:1790: delete oid 17 current snap is 0 2020-06-12T02:37:39.727 INFO:tasks.rados.rados.0.smithi107.stdout:1788: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:39.727 INFO:tasks.rados.rados.0.smithi107.stdout:1790: done (1 left) 2020-06-12T02:37:39.727 INFO:tasks.rados.rados.0.smithi107.stdout:1791: delete oid 19 current snap is 0 2020-06-12T02:37:39.727 INFO:tasks.rados.rados.0.smithi107.stdout:1788: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:39.730 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:39.720+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:37:39.730 INFO:tasks.rados.rados.0.smithi107.stdout:1788: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:39.731 INFO:tasks.rados.rados.0.smithi107.stdout:1791: done (1 left) 2020-06-12T02:37:39.731 INFO:tasks.rados.rados.0.smithi107.stdout:1792: delete oid 11 current snap is 0 2020-06-12T02:37:39.731 INFO:tasks.rados.rados.0.smithi107.stdout:1788: finishing write tid 5 to smithi10729407-4 2020-06-12T02:37:39.731 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 352 (ObjNum 1787 snap 0 seq_num 1787) dirty exists 2020-06-12T02:37:39.732 INFO:tasks.rados.rados.0.smithi107.stdout:1788: left oid 4 (ObjNum 1787 snap 0 seq_num 1787) 2020-06-12T02:37:39.732 INFO:tasks.rados.rados.0.smithi107.stdout:1788: done (1 left) 2020-06-12T02:37:39.732 INFO:tasks.rados.rados.0.smithi107.stdout:1792: done (0 left) 2020-06-12T02:37:39.732 INFO:tasks.rados.rados.0.smithi107.stdout:1793: delete oid 14 current snap is 0 2020-06-12T02:37:39.733 INFO:tasks.rados.rados.0.smithi107.stdout:1793: done (0 left) 2020-06-12T02:37:39.733 INFO:tasks.rados.rados.0.smithi107.stdout:1794: delete oid 19 current snap is 0 2020-06-12T02:37:39.734 INFO:tasks.rados.rados.0.smithi107.stdout:1794: done (0 left) 2020-06-12T02:37:39.735 INFO:tasks.rados.rados.0.smithi107.stdout:1795: delete oid 17 current snap is 0 2020-06-12T02:37:39.736 INFO:tasks.rados.rados.0.smithi107.stdout:1795: done (0 left) 2020-06-12T02:37:39.736 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:37:39.736 INFO:tasks.rados.rados.0.smithi107.stdout:1796: seq_num 1795 ranges {1343488=466944,1810432=507904,2318336=294912} 2020-06-12T02:37:39.753 INFO:tasks.rados.rados.0.smithi107.stdout:1796: writing smithi10729407-15 from 1343488 to 1810432 tid 1 2020-06-12T02:37:39.753 INFO:tasks.rados.rados.0.smithi107.stdout:1796: writing smithi10729407-15 from 1810432 to 2318336 tid 2 2020-06-12T02:37:39.753 INFO:tasks.rados.rados.0.smithi107.stdout:1796: writing smithi10729407-15 from 2318336 to 2613248 tid 3 2020-06-12T02:37:39.754 INFO:tasks.rados.rados.0.smithi107.stdout:1797: delete oid 13 current snap is 0 2020-06-12T02:37:39.754 INFO:tasks.rados.rados.0.smithi107.stdout:1796: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:39.754 INFO:tasks.rados.rados.0.smithi107.stdout:1797: done (1 left) 2020-06-12T02:37:39.754 INFO:tasks.rados.rados.0.smithi107.stdout:1798: delete oid 6 current snap is 0 2020-06-12T02:37:39.764 INFO:tasks.rados.rados.0.smithi107.stdout:1798: done (1 left) 2020-06-12T02:37:39.764 INFO:tasks.rados.rados.0.smithi107.stdout:1799: delete oid 11 current snap is 0 2020-06-12T02:37:39.764 INFO:tasks.rados.rados.0.smithi107.stdout:1799: done (1 left) 2020-06-12T02:37:39.764 INFO:tasks.rados.rados.0.smithi107.stdout:1800: delete oid 14 current snap is 0 2020-06-12T02:37:39.765 INFO:tasks.rados.rados.0.smithi107.stdout:1796: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:39.765 INFO:tasks.rados.rados.0.smithi107.stdout:1796: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:39.765 INFO:tasks.rados.rados.0.smithi107.stdout:1800: done (1 left) 2020-06-12T02:37:39.765 INFO:tasks.rados.rados.0.smithi107.stdout:1801: delete oid 16 current snap is 0 2020-06-12T02:37:39.765 INFO:tasks.rados.rados.0.smithi107.stdout:1796: finishing write tid 5 to smithi10729407-15 2020-06-12T02:37:39.766 INFO:tasks.rados.rados.0.smithi107.stdout:1796: finishing write tid 6 to smithi10729407-15 2020-06-12T02:37:39.766 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 312 (ObjNum 1795 snap 0 seq_num 1795) dirty exists 2020-06-12T02:37:39.766 INFO:tasks.rados.rados.0.smithi107.stdout:1796: left oid 15 (ObjNum 1795 snap 0 seq_num 1795) 2020-06-12T02:37:39.766 INFO:tasks.rados.rados.0.smithi107.stdout:1796: done (1 left) 2020-06-12T02:37:39.766 INFO:tasks.rados.rados.0.smithi107.stdout:1801: done (0 left) 2020-06-12T02:37:39.767 INFO:tasks.rados.rados.0.smithi107.stdout:1802: delete oid 19 current snap is 0 2020-06-12T02:37:39.767 INFO:tasks.rados.rados.0.smithi107.stdout:1802: done (0 left) 2020-06-12T02:37:39.767 INFO:tasks.rados.rados.0.smithi107.stdout:1803: delete oid 19 current snap is 0 2020-06-12T02:37:39.768 INFO:tasks.rados.rados.0.smithi107.stdout:1803: done (0 left) 2020-06-12T02:37:39.768 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:37:39.768 INFO:tasks.rados.rados.0.smithi107.stdout:1804: seq_num 1803 ranges {0=606208,606208=434176,1040384=368640} 2020-06-12T02:37:39.769 INFO:tasks.rados.rados.0.smithi107.stdout:1804: writing smithi10729407-7 from 0 to 606208 tid 1 2020-06-12T02:37:39.771 INFO:tasks.rados.rados.0.smithi107.stdout:1804: writing smithi10729407-7 from 606208 to 1040384 tid 2 2020-06-12T02:37:39.774 INFO:tasks.rados.rados.0.smithi107.stdout:1804: writing smithi10729407-7 from 1040384 to 1409024 tid 3 2020-06-12T02:37:39.775 INFO:tasks.rados.rados.0.smithi107.stdout:1805: delete oid 8 current snap is 0 2020-06-12T02:37:39.777 INFO:tasks.rados.rados.0.smithi107.stdout:1805: done (1 left) 2020-06-12T02:37:39.777 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:37:39.778 INFO:tasks.rados.rados.0.smithi107.stdout:1806: seq_num 1805 ranges {2613248=638976,3252224=598016,3850240=204800} 2020-06-12T02:37:39.780 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:39.781 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:39.790 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-06-12T02:37:39.795 INFO:tasks.rados.rados.0.smithi107.stdout:1806: writing smithi10729407-15 from 2613248 to 3252224 tid 1 2020-06-12T02:37:39.798 INFO:tasks.rados.rados.0.smithi107.stdout:1806: writing smithi10729407-15 from 3252224 to 3850240 tid 2 2020-06-12T02:37:39.800 INFO:tasks.rados.rados.0.smithi107.stdout:1806: writing smithi10729407-15 from 3850240 to 4055040 tid 3 2020-06-12T02:37:39.800 INFO:tasks.rados.rados.0.smithi107.stdout:1807: delete oid 2 current snap is 0 2020-06-12T02:37:39.800 INFO:tasks.rados.rados.0.smithi107.stdout:1804: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:39.800 INFO:tasks.rados.rados.0.smithi107.stdout:1804: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:39.800 INFO:tasks.rados.rados.0.smithi107.stdout:1804: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:39.801 INFO:tasks.rados.rados.0.smithi107.stdout:1804: finishing write tid 5 to smithi10729407-7 2020-06-12T02:37:39.801 INFO:tasks.rados.rados.0.smithi107.stdout:1804: finishing write tid 6 to smithi10729407-7 2020-06-12T02:37:39.801 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 350 (ObjNum 1803 snap 0 seq_num 1803) dirty exists 2020-06-12T02:37:39.801 INFO:tasks.rados.rados.0.smithi107.stdout:1804: left oid 7 (ObjNum 1803 snap 0 seq_num 1803) 2020-06-12T02:37:39.802 INFO:tasks.rados.rados.0.smithi107.stdout:1804: done (2 left) 2020-06-12T02:37:39.802 INFO:tasks.rados.rados.0.smithi107.stdout:1807: done (1 left) 2020-06-12T02:37:39.802 INFO:tasks.rados.rados.0.smithi107.stdout:1808: delete oid 19 current snap is 0 2020-06-12T02:37:39.803 INFO:tasks.rados.rados.0.smithi107.stdout:1806: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:39.803 INFO:tasks.rados.rados.0.smithi107.stdout:1808: done (1 left) 2020-06-12T02:37:39.803 INFO:tasks.rados.rados.0.smithi107.stdout:1809: delete oid 1 current snap is 0 2020-06-12T02:37:39.804 INFO:tasks.rados.rados.0.smithi107.stdout:1809: done (1 left) 2020-06-12T02:37:39.805 INFO:tasks.rados.rados.0.smithi107.stdout:1810: delete oid 19 current snap is 0 2020-06-12T02:37:39.806 INFO:tasks.rados.rados.0.smithi107.stdout:1810: done (1 left) 2020-06-12T02:37:39.807 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:37:39.807 INFO:tasks.rados.rados.0.smithi107.stdout:1811: seq_num 1810 ranges {0=720896,720896=466944,1187840=335872} 2020-06-12T02:37:39.810 INFO:tasks.rados.rados.0.smithi107.stdout:1811: writing smithi10729407-20 from 0 to 720896 tid 1 2020-06-12T02:37:39.813 INFO:tasks.rados.rados.0.smithi107.stdout:1811: writing smithi10729407-20 from 720896 to 1187840 tid 2 2020-06-12T02:37:39.816 INFO:tasks.rados.rados.0.smithi107.stdout:1811: writing smithi10729407-20 from 1187840 to 1523712 tid 3 2020-06-12T02:37:39.816 INFO:tasks.rados.rados.0.smithi107.stdout:1812: delete oid 2 current snap is 0 2020-06-12T02:37:39.816 INFO:tasks.rados.rados.0.smithi107.stdout:1806: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:39.816 INFO:tasks.rados.rados.0.smithi107.stdout:1806: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:39.816 INFO:tasks.rados.rados.0.smithi107.stdout:1806: finishing write tid 5 to smithi10729407-15 2020-06-12T02:37:39.817 INFO:tasks.rados.rados.0.smithi107.stdout:1806: finishing write tid 6 to smithi10729407-15 2020-06-12T02:37:39.817 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 317 (ObjNum 1805 snap 0 seq_num 1805) dirty exists 2020-06-12T02:37:39.817 INFO:tasks.rados.rados.0.smithi107.stdout:1806: left oid 15 (ObjNum 1805 snap 0 seq_num 1805) 2020-06-12T02:37:39.818 INFO:tasks.rados.rados.0.smithi107.stdout:1806: done (2 left) 2020-06-12T02:37:39.819 INFO:tasks.rados.rados.0.smithi107.stdout:1812: done (1 left) 2020-06-12T02:37:39.819 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:37:39.819 INFO:tasks.rados.rados.0.smithi107.stdout:1813: seq_num 1812 ranges {0=753664,753664=638976,1392640=163840} 2020-06-12T02:37:39.822 INFO:tasks.rados.rados.0.smithi107.stdout:1813: writing smithi10729407-1 from 0 to 753664 tid 1 2020-06-12T02:37:39.824 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:37:39.822+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:37:39.826 INFO:tasks.rados.rados.0.smithi107.stdout:1813: writing smithi10729407-1 from 753664 to 1392640 tid 2 2020-06-12T02:37:39.827 INFO:tasks.rados.rados.0.smithi107.stdout:1813: writing smithi10729407-1 from 1392640 to 1556480 tid 3 2020-06-12T02:37:39.828 INFO:tasks.rados.rados.0.smithi107.stdout:1814: delete oid 11 current snap is 0 2020-06-12T02:37:39.828 INFO:tasks.rados.rados.0.smithi107.stdout:1811: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:39.828 INFO:tasks.rados.rados.0.smithi107.stdout:1811: finishing write tid 2 to smithi10729407-20 2020-06-12T02:37:39.828 INFO:tasks.rados.rados.0.smithi107.stdout:1811: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:39.828 INFO:tasks.rados.rados.0.smithi107.stdout:1811: finishing write tid 5 to smithi10729407-20 2020-06-12T02:37:39.829 INFO:tasks.rados.rados.0.smithi107.stdout:1811: finishing write tid 6 to smithi10729407-20 2020-06-12T02:37:39.829 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 134 (ObjNum 1810 snap 0 seq_num 1810) dirty exists 2020-06-12T02:37:39.829 INFO:tasks.rados.rados.0.smithi107.stdout:1811: left oid 20 (ObjNum 1810 snap 0 seq_num 1810) 2020-06-12T02:37:39.831 INFO:tasks.rados.rados.0.smithi107.stdout:1813: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:39.833 INFO:tasks.rados.rados.0.smithi107.stdout:1813: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:39.833 INFO:tasks.rados.rados.0.smithi107.stdout:1813: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:39.833 INFO:tasks.rados.rados.0.smithi107.stdout:1813: finishing write tid 5 to smithi10729407-1 2020-06-12T02:37:39.834 INFO:tasks.rados.rados.0.smithi107.stdout:1813: finishing write tid 6 to smithi10729407-1 2020-06-12T02:37:39.834 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 356 (ObjNum 1812 snap 0 seq_num 1812) dirty exists 2020-06-12T02:37:39.835 INFO:tasks.rados.rados.0.smithi107.stdout:1813: left oid 1 (ObjNum 1812 snap 0 seq_num 1812) 2020-06-12T02:37:39.835 INFO:tasks.rados.rados.0.smithi107.stdout:1811: done (2 left) 2020-06-12T02:37:39.835 INFO:tasks.rados.rados.0.smithi107.stdout:1813: done (1 left) 2020-06-12T02:37:39.835 INFO:tasks.rados.rados.0.smithi107.stdout:1814: done (0 left) 2020-06-12T02:37:39.835 INFO:tasks.rados.rados.0.smithi107.stdout:1815: delete oid 11 current snap is 0 2020-06-12T02:37:39.836 INFO:tasks.rados.rados.0.smithi107.stdout:1815: done (0 left) 2020-06-12T02:37:39.836 INFO:tasks.rados.rados.0.smithi107.stdout:1816: delete oid 11 current snap is 0 2020-06-12T02:37:39.837 INFO:tasks.rados.rados.0.smithi107.stdout:1816: done (0 left) 2020-06-12T02:37:39.837 INFO:tasks.rados.rados.0.smithi107.stdout:1817: delete oid 17 current snap is 0 2020-06-12T02:37:39.839 INFO:tasks.rados.rados.0.smithi107.stdout:1817: done (0 left) 2020-06-12T02:37:39.839 INFO:tasks.rados.rados.0.smithi107.stdout:1818: delete oid 7 current snap is 0 2020-06-12T02:37:39.841 INFO:tasks.rados.rados.0.smithi107.stdout:1818: done (0 left) 2020-06-12T02:37:39.841 INFO:tasks.rados.rados.0.smithi107.stdout:1819: delete oid 9 current snap is 0 2020-06-12T02:37:39.843 INFO:tasks.rados.rados.0.smithi107.stdout:1819: done (0 left) 2020-06-12T02:37:39.843 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:37:39.843 INFO:tasks.rados.rados.0.smithi107.stdout:1820: seq_num 1819 ranges {0=466944,466944=679936,1146880=524288} 2020-06-12T02:37:39.845 INFO:tasks.rados.rados.0.smithi107.stdout:1820: writing smithi10729407-10 from 0 to 466944 tid 1 2020-06-12T02:37:39.849 INFO:tasks.rados.rados.0.smithi107.stdout:1820: writing smithi10729407-10 from 466944 to 1146880 tid 2 2020-06-12T02:37:39.852 INFO:tasks.rados.rados.0.smithi107.stdout:1820: writing smithi10729407-10 from 1146880 to 1671168 tid 3 2020-06-12T02:37:39.853 INFO:tasks.rados.rados.0.smithi107.stdout:1821: delete oid 7 current snap is 0 2020-06-12T02:37:39.853 INFO:tasks.rados.rados.0.smithi107.stdout:1820: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:39.855 INFO:tasks.rados.rados.0.smithi107.stdout:1821: done (1 left) 2020-06-12T02:37:39.855 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:37:39.855 INFO:tasks.rados.rados.0.smithi107.stdout:1822: seq_num 1821 ranges {0=507904,507904=450560,958464=565248,1523712=188416} 2020-06-12T02:37:39.858 INFO:tasks.rados.rados.0.smithi107.stdout:1822: writing smithi10729407-13 from 0 to 507904 tid 1 2020-06-12T02:37:39.860 INFO:tasks.rados.rados.0.smithi107.stdout:1822: writing smithi10729407-13 from 507904 to 958464 tid 2 2020-06-12T02:37:39.864 INFO:tasks.rados.rados.0.smithi107.stdout:1822: writing smithi10729407-13 from 958464 to 1523712 tid 3 2020-06-12T02:37:39.866 INFO:tasks.rados.rados.0.smithi107.stdout:1822: writing smithi10729407-13 from 1523712 to 1712128 tid 4 2020-06-12T02:37:39.866 INFO:tasks.rados.rados.0.smithi107.stdout:1823: delete oid 6 current snap is 0 2020-06-12T02:37:39.866 INFO:tasks.rados.rados.0.smithi107.stdout:1820: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:39.866 INFO:tasks.rados.rados.0.smithi107.stdout:1820: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:39.866 INFO:tasks.rados.rados.0.smithi107.stdout:1820: finishing write tid 5 to smithi10729407-10 2020-06-12T02:37:39.867 INFO:tasks.rados.rados.0.smithi107.stdout:1820: finishing write tid 6 to smithi10729407-10 2020-06-12T02:37:39.867 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 172 (ObjNum 1819 snap 0 seq_num 1819) dirty exists 2020-06-12T02:37:39.867 INFO:tasks.rados.rados.0.smithi107.stdout:1820: left oid 10 (ObjNum 1819 snap 0 seq_num 1819) 2020-06-12T02:37:39.868 INFO:tasks.rados.rados.0.smithi107.stdout:1822: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:39.868 INFO:tasks.rados.rados.0.smithi107.stdout:1820: done (2 left) 2020-06-12T02:37:39.868 INFO:tasks.rados.rados.0.smithi107.stdout:1823: done (1 left) 2020-06-12T02:37:39.868 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:37:39.868 INFO:tasks.rados.rados.0.smithi107.stdout:1824: seq_num 1823 ranges {0=540672,540672=614400,1155072=581632,1736704=8192} 2020-06-12T02:37:39.871 INFO:tasks.rados.rados.0.smithi107.stdout:1824: writing smithi10729407-16 from 0 to 540672 tid 1 2020-06-12T02:37:39.875 INFO:tasks.rados.rados.0.smithi107.stdout:1824: writing smithi10729407-16 from 540672 to 1155072 tid 2 2020-06-12T02:37:39.879 INFO:tasks.rados.rados.0.smithi107.stdout:1824: writing smithi10729407-16 from 1155072 to 1736704 tid 3 2020-06-12T02:37:39.879 INFO:tasks.rados.rados.0.smithi107.stdout:1824: writing smithi10729407-16 from 1736704 to 1744896 tid 4 2020-06-12T02:37:39.879 INFO:tasks.rados.rados.0.smithi107.stdout:1825: delete oid 20 current snap is 0 2020-06-12T02:37:39.880 INFO:tasks.rados.rados.0.smithi107.stdout:1822: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:39.880 INFO:tasks.rados.rados.0.smithi107.stdout:1822: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:39.880 INFO:tasks.rados.rados.0.smithi107.stdout:1822: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:39.880 INFO:tasks.rados.rados.0.smithi107.stdout:1822: finishing write tid 6 to smithi10729407-13 2020-06-12T02:37:39.880 INFO:tasks.rados.rados.0.smithi107.stdout:1822: finishing write tid 7 to smithi10729407-13 2020-06-12T02:37:39.880 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 357 (ObjNum 1821 snap 0 seq_num 1821) dirty exists 2020-06-12T02:37:39.881 INFO:tasks.rados.rados.0.smithi107.stdout:1822: left oid 13 (ObjNum 1821 snap 0 seq_num 1821) 2020-06-12T02:37:39.881 INFO:tasks.rados.rados.0.smithi107.stdout:1824: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:39.882 INFO:tasks.rados.rados.0.smithi107.stdout:1822: done (2 left) 2020-06-12T02:37:39.882 INFO:tasks.rados.rados.0.smithi107.stdout:1825: done (1 left) 2020-06-12T02:37:39.882 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:37:39.882 INFO:tasks.rados.rados.0.smithi107.stdout:1826: seq_num 1825 ranges {0=573440,573440=434176,1007616=540672,1548288=229376} 2020-06-12T02:37:39.885 INFO:tasks.rados.rados.0.smithi107.stdout:1826: writing smithi10729407-14 from 0 to 573440 tid 1 2020-06-12T02:37:39.889 INFO:tasks.rados.rados.0.smithi107.stdout:1826: writing smithi10729407-14 from 573440 to 1007616 tid 2 2020-06-12T02:37:39.892 INFO:tasks.rados.rados.0.smithi107.stdout:1826: writing smithi10729407-14 from 1007616 to 1548288 tid 3 2020-06-12T02:37:39.894 INFO:tasks.rados.rados.0.smithi107.stdout:1826: writing smithi10729407-14 from 1548288 to 1777664 tid 4 2020-06-12T02:37:39.894 INFO:tasks.rados.rados.0.smithi107.stdout:1827: delete oid 5 current snap is 0 2020-06-12T02:37:39.894 INFO:tasks.rados.rados.0.smithi107.stdout:1824: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:39.895 INFO:tasks.rados.rados.0.smithi107.stdout:1824: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:39.895 INFO:tasks.rados.rados.0.smithi107.stdout:1824: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:39.895 INFO:tasks.rados.rados.0.smithi107.stdout:1824: finishing write tid 6 to smithi10729407-16 2020-06-12T02:37:39.895 INFO:tasks.rados.rados.0.smithi107.stdout:1824: finishing write tid 7 to smithi10729407-16 2020-06-12T02:37:39.895 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 323 (ObjNum 1823 snap 0 seq_num 1823) dirty exists 2020-06-12T02:37:39.895 INFO:tasks.rados.rados.0.smithi107.stdout:1824: left oid 16 (ObjNum 1823 snap 0 seq_num 1823) 2020-06-12T02:37:39.896 INFO:tasks.rados.rados.0.smithi107.stdout:1826: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:39.896 INFO:tasks.rados.rados.0.smithi107.stdout:1824: done (2 left) 2020-06-12T02:37:39.896 INFO:tasks.rados.rados.0.smithi107.stdout:1827: done (1 left) 2020-06-12T02:37:39.897 INFO:tasks.rados.rados.0.smithi107.stdout:1828: delete oid 16 current snap is 0 2020-06-12T02:37:39.900 INFO:tasks.rados.rados.0.smithi107.stdout:1828: done (1 left) 2020-06-12T02:37:39.900 INFO:tasks.rados.rados.0.smithi107.stdout:1829: delete oid 11 current snap is 0 2020-06-12T02:37:39.903 INFO:tasks.rados.rados.0.smithi107.stdout:1826: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:39.903 INFO:tasks.rados.rados.0.smithi107.stdout:1826: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:39.906 INFO:tasks.rados.rados.0.smithi107.stdout:1826: finishing write tid 4 to smithi10729407-14 2020-06-12T02:37:39.909 INFO:tasks.rados.rados.0.smithi107.stdout:1826: finishing write tid 6 to smithi10729407-14 2020-06-12T02:37:39.909 INFO:tasks.rados.rados.0.smithi107.stdout:1829: done (1 left) 2020-06-12T02:37:39.910 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:37:39.910 INFO:tasks.rados.rados.0.smithi107.stdout:1830: seq_num 1829 ranges {1671168=638976,2310144=770048,3080192=434176} 2020-06-12T02:37:39.922 INFO:tasks.rados.rados.0.smithi107.stdout:1830: writing smithi10729407-10 from 1671168 to 2310144 tid 1 2020-06-12T02:37:39.925 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:39.922+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:39.926 INFO:tasks.rados.rados.0.smithi107.stdout:1830: writing smithi10729407-10 from 2310144 to 3080192 tid 2 2020-06-12T02:37:39.930 INFO:tasks.rados.rados.0.smithi107.stdout:1830: writing smithi10729407-10 from 3080192 to 3514368 tid 3 2020-06-12T02:37:39.930 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:37:39.930 INFO:tasks.rados.rados.0.smithi107.stdout:1831: seq_num 1830 ranges {0=655360,655360=655360,1310720=548864} 2020-06-12T02:37:39.933 INFO:tasks.rados.rados.0.smithi107.stdout:1831: writing smithi10729407-8 from 0 to 655360 tid 1 2020-06-12T02:37:39.938 INFO:tasks.rados.rados.0.smithi107.stdout:1831: writing smithi10729407-8 from 655360 to 1310720 tid 2 2020-06-12T02:37:39.942 INFO:tasks.rados.rados.0.smithi107.stdout:1831: writing smithi10729407-8 from 1310720 to 1859584 tid 3 2020-06-12T02:37:39.942 INFO:tasks.rados.rados.0.smithi107.stdout:1832: delete oid 3 current snap is 0 2020-06-12T02:37:39.943 INFO:tasks.rados.rados.0.smithi107.stdout:1826: finishing write tid 7 to smithi10729407-14 2020-06-12T02:37:39.943 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 331 (ObjNum 1825 snap 0 seq_num 1825) dirty exists 2020-06-12T02:37:39.943 INFO:tasks.rados.rados.0.smithi107.stdout:1826: left oid 14 (ObjNum 1825 snap 0 seq_num 1825) 2020-06-12T02:37:39.943 INFO:tasks.rados.rados.0.smithi107.stdout:1830: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:39.943 INFO:tasks.rados.rados.0.smithi107.stdout:1830: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:39.943 INFO:tasks.rados.rados.0.smithi107.stdout:1830: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:39.944 INFO:tasks.rados.rados.0.smithi107.stdout:1830: finishing write tid 5 to smithi10729407-10 2020-06-12T02:37:39.944 INFO:tasks.rados.rados.0.smithi107.stdout:1830: finishing write tid 6 to smithi10729407-10 2020-06-12T02:37:39.944 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 176 (ObjNum 1829 snap 0 seq_num 1829) dirty exists 2020-06-12T02:37:39.944 INFO:tasks.rados.rados.0.smithi107.stdout:1830: left oid 10 (ObjNum 1829 snap 0 seq_num 1829) 2020-06-12T02:37:39.945 INFO:tasks.rados.rados.0.smithi107.stdout:1826: done (3 left) 2020-06-12T02:37:39.946 INFO:tasks.rados.rados.0.smithi107.stdout:1830: done (2 left) 2020-06-12T02:37:39.946 INFO:tasks.rados.rados.0.smithi107.stdout:1832: done (1 left) 2020-06-12T02:37:39.946 INFO:tasks.rados.rados.0.smithi107.stdout:1833: delete oid 17 current snap is 0 2020-06-12T02:37:39.946 INFO:tasks.rados.rados.0.smithi107.stdout:1831: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:39.947 INFO:tasks.rados.rados.0.smithi107.stdout:1831: finishing write tid 2 to smithi10729407-8 2020-06-12T02:37:39.948 INFO:tasks.rados.rados.0.smithi107.stdout:1833: done (1 left) 2020-06-12T02:37:39.948 INFO:tasks.rados.rados.0.smithi107.stdout:1834: delete oid 14 current snap is 0 2020-06-12T02:37:39.948 INFO:tasks.rados.rados.0.smithi107.stdout:1831: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:39.950 INFO:tasks.rados.rados.0.smithi107.stdout:1831: finishing write tid 5 to smithi10729407-8 2020-06-12T02:37:39.950 INFO:tasks.rados.rados.0.smithi107.stdout:1834: done (1 left) 2020-06-12T02:37:39.951 INFO:tasks.rados.rados.0.smithi107.stdout:1835: delete oid 3 current snap is 0 2020-06-12T02:37:39.951 INFO:tasks.rados.rados.0.smithi107.stdout:1831: finishing write tid 6 to smithi10729407-8 2020-06-12T02:37:39.951 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 321 (ObjNum 1830 snap 0 seq_num 1830) dirty exists 2020-06-12T02:37:39.952 INFO:tasks.rados.rados.0.smithi107.stdout:1831: left oid 8 (ObjNum 1830 snap 0 seq_num 1830) 2020-06-12T02:37:39.952 INFO:tasks.rados.rados.0.smithi107.stdout:1831: done (1 left) 2020-06-12T02:37:39.952 INFO:tasks.rados.rados.0.smithi107.stdout:1835: done (0 left) 2020-06-12T02:37:39.953 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:37:39.953 INFO:tasks.rados.rados.0.smithi107.stdout:1836: seq_num 1835 ranges {1556480=737280,2293760=524288,2818048=516096,3334144=163840} 2020-06-12T02:37:39.964 INFO:tasks.rados.rados.0.smithi107.stdout:1836: writing smithi10729407-1 from 1556480 to 2293760 tid 1 2020-06-12T02:37:39.967 INFO:tasks.rados.rados.0.smithi107.stdout:1836: writing smithi10729407-1 from 2293760 to 2818048 tid 2 2020-06-12T02:37:39.970 INFO:tasks.rados.rados.0.smithi107.stdout:1836: writing smithi10729407-1 from 2818048 to 3334144 tid 3 2020-06-12T02:37:39.971 INFO:tasks.rados.rados.0.smithi107.stdout:1836: writing smithi10729407-1 from 3334144 to 3497984 tid 4 2020-06-12T02:37:39.972 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:37:39.972 INFO:tasks.rados.rados.0.smithi107.stdout:1837: seq_num 1836 ranges {0=753664,753664=409600,1163264=729088,1892352=65536} 2020-06-12T02:37:39.973 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:39.974 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:39.976 INFO:tasks.rados.rados.0.smithi107.stdout:1837: writing smithi10729407-6 from 0 to 753664 tid 1 2020-06-12T02:37:39.978 INFO:tasks.rados.rados.0.smithi107.stdout:1837: writing smithi10729407-6 from 753664 to 1163264 tid 2 2020-06-12T02:37:39.982 INFO:tasks.rados.rados.0.smithi107.stdout:1837: writing smithi10729407-6 from 1163264 to 1892352 tid 3 2020-06-12T02:37:39.983 INFO:tasks.rados.rados.0.smithi107.stdout:1837: writing smithi10729407-6 from 1892352 to 1957888 tid 4 2020-06-12T02:37:39.983 INFO:tasks.rados.rados.0.smithi107.stdout:1838: delete oid 7 current snap is 0 2020-06-12T02:37:39.983 INFO:tasks.rados.rados.0.smithi107.stdout:1836: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:39.983 INFO:tasks.rados.rados.0.smithi107.stdout:1836: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:39.984 INFO:tasks.rados.rados.0.smithi107.stdout:1836: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:39.984 INFO:tasks.rados.rados.0.smithi107.stdout:1836: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:39.985 INFO:tasks.rados.rados.0.smithi107.stdout:1838: done (2 left) 2020-06-12T02:37:39.985 INFO:tasks.rados.rados.0.smithi107.stdout:1839: delete oid 2 current snap is 0 2020-06-12T02:37:39.986 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-06-12T02:37:39.987 INFO:tasks.rados.rados.0.smithi107.stdout:1839: done (2 left) 2020-06-12T02:37:39.987 INFO:tasks.rados.rados.0.smithi107.stdout:1840: delete oid 13 current snap is 0 2020-06-12T02:37:39.988 INFO:tasks.rados.rados.0.smithi107.stdout:1840: done (2 left) 2020-06-12T02:37:39.989 INFO:tasks.rados.rados.0.smithi107.stdout:1841: delete oid 20 current snap is 0 2020-06-12T02:37:39.989 INFO:tasks.rados.rados.0.smithi107.stdout:1836: finishing write tid 6 to smithi10729407-1 2020-06-12T02:37:39.990 INFO:tasks.rados.rados.0.smithi107.stdout:1841: done (2 left) 2020-06-12T02:37:39.991 INFO:tasks.rados.rados.0.smithi107.stdout:1842: delete oid 10 current snap is 0 2020-06-12T02:37:39.992 INFO:tasks.rados.rados.0.smithi107.stdout:1842: done (2 left) 2020-06-12T02:37:39.992 INFO:tasks.rados.rados.0.smithi107.stdout:1843: delete oid 7 current snap is 0 2020-06-12T02:37:39.993 INFO:tasks.rados.rados.0.smithi107.stdout:1837: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:39.994 INFO:tasks.rados.rados.0.smithi107.stdout:1837: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:39.995 INFO:tasks.rados.rados.0.smithi107.stdout:1836: finishing write tid 7 to smithi10729407-1 2020-06-12T02:37:39.995 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 362 (ObjNum 1835 snap 0 seq_num 1835) dirty exists 2020-06-12T02:37:39.995 INFO:tasks.rados.rados.0.smithi107.stdout:1836: left oid 1 (ObjNum 1835 snap 0 seq_num 1835) 2020-06-12T02:37:39.996 INFO:tasks.rados.rados.0.smithi107.stdout:1837: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:39.996 INFO:tasks.rados.rados.0.smithi107.stdout:1836: done (2 left) 2020-06-12T02:37:39.997 INFO:tasks.rados.rados.0.smithi107.stdout:1843: done (1 left) 2020-06-12T02:37:39.997 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:37:39.997 INFO:tasks.rados.rados.0.smithi107.stdout:1844: seq_num 1843 ranges {0=475136,475136=450560,925696=483328,1409024=483328,1892352=188416} 2020-06-12T02:37:39.999 INFO:tasks.rados.rados.0.smithi107.stdout:1844: writing smithi10729407-17 from 0 to 475136 tid 1 2020-06-12T02:37:40.002 INFO:tasks.rados.rados.0.smithi107.stdout:1844: writing smithi10729407-17 from 475136 to 925696 tid 2 2020-06-12T02:37:40.005 INFO:tasks.rados.rados.0.smithi107.stdout:1844: writing smithi10729407-17 from 925696 to 1409024 tid 3 2020-06-12T02:37:40.008 INFO:tasks.rados.rados.0.smithi107.stdout:1844: writing smithi10729407-17 from 1409024 to 1892352 tid 4 2020-06-12T02:37:40.010 INFO:tasks.rados.rados.0.smithi107.stdout:1844: writing smithi10729407-17 from 1892352 to 2080768 tid 5 2020-06-12T02:37:40.010 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:37:40.010 INFO:tasks.rados.rados.0.smithi107.stdout:1845: seq_num 1844 ranges {0=491520,491520=737280,1228800=688128,1916928=180224} 2020-06-12T02:37:40.013 INFO:tasks.rados.rados.0.smithi107.stdout:1845: writing smithi10729407-7 from 0 to 491520 tid 1 2020-06-12T02:37:40.018 INFO:tasks.rados.rados.0.smithi107.stdout:1845: writing smithi10729407-7 from 491520 to 1228800 tid 2 2020-06-12T02:37:40.022 INFO:tasks.rados.rados.0.smithi107.stdout:1845: writing smithi10729407-7 from 1228800 to 1916928 tid 3 2020-06-12T02:37:40.023 INFO:tasks.rados.rados.0.smithi107.stdout:1845: writing smithi10729407-7 from 1916928 to 2097152 tid 4 2020-06-12T02:37:40.023 INFO:tasks.rados.rados.0.smithi107.stdout:1846: delete oid 20 current snap is 0 2020-06-12T02:37:40.024 INFO:tasks.rados.rados.0.smithi107.stdout:1837: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:40.024 INFO:tasks.rados.rados.0.smithi107.stdout:1837: finishing write tid 6 to smithi10729407-6 2020-06-12T02:37:40.024 INFO:tasks.rados.rados.0.smithi107.stdout:1837: finishing write tid 7 to smithi10729407-6 2020-06-12T02:37:40.024 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 367 (ObjNum 1836 snap 0 seq_num 1836) dirty exists 2020-06-12T02:37:40.024 INFO:tasks.rados.rados.0.smithi107.stdout:1837: left oid 6 (ObjNum 1836 snap 0 seq_num 1836) 2020-06-12T02:37:40.024 INFO:tasks.rados.rados.0.smithi107.stdout:1844: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:40.025 INFO:tasks.rados.rados.0.smithi107.stdout:1844: finishing write tid 2 to smithi10729407-17 2020-06-12T02:37:40.025 INFO:tasks.rados.rados.0.smithi107.stdout:1844: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:40.025 INFO:tasks.rados.rados.0.smithi107.stdout:1844: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:40.025 INFO:tasks.rados.rados.0.smithi107.stdout:1844: finishing write tid 5 to smithi10729407-17 2020-06-12T02:37:40.025 INFO:tasks.rados.rados.0.smithi107.stdout:1844: finishing write tid 7 to smithi10729407-17 2020-06-12T02:37:40.026 INFO:tasks.rados.rados.0.smithi107.stdout:1844: finishing write tid 8 to smithi10729407-17 2020-06-12T02:37:40.026 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 361 (ObjNum 1843 snap 0 seq_num 1843) dirty exists 2020-06-12T02:37:40.026 INFO:tasks.rados.rados.0.smithi107.stdout:1844: left oid 17 (ObjNum 1843 snap 0 seq_num 1843) 2020-06-12T02:37:40.026 INFO:tasks.rados.rados.0.smithi107.stdout:1845: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:40.027 INFO:tasks.rados.rados.0.smithi107.stdout:1837: done (3 left) 2020-06-12T02:37:40.027 INFO:tasks.rados.rados.0.smithi107.stdout:1844: done (2 left) 2020-06-12T02:37:40.027 INFO:tasks.rados.rados.0.smithi107.stdout:1846: done (1 left) 2020-06-12T02:37:40.027 INFO:tasks.rados.rados.0.smithi107.stdout:1847: delete oid 12 current snap is 0 2020-06-12T02:37:40.028 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:40.024+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:40.028 INFO:tasks.rados.rados.0.smithi107.stdout:1847: done (1 left) 2020-06-12T02:37:40.028 INFO:tasks.rados.rados.0.smithi107.stdout:1848: delete oid 4 current snap is 0 2020-06-12T02:37:40.030 INFO:tasks.rados.rados.0.smithi107.stdout:1845: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:40.031 INFO:tasks.rados.rados.0.smithi107.stdout:1845: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:40.033 INFO:tasks.rados.rados.0.smithi107.stdout:1848: done (1 left) 2020-06-12T02:37:40.033 INFO:tasks.rados.rados.0.smithi107.stdout:1849: delete oid 4 current snap is 0 2020-06-12T02:37:40.034 INFO:tasks.rados.rados.0.smithi107.stdout:1845: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:40.036 INFO:tasks.rados.rados.0.smithi107.stdout:1845: finishing write tid 6 to smithi10729407-7 2020-06-12T02:37:40.037 INFO:tasks.rados.rados.0.smithi107.stdout:1845: finishing write tid 7 to smithi10729407-7 2020-06-12T02:37:40.038 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 365 (ObjNum 1844 snap 0 seq_num 1844) dirty exists 2020-06-12T02:37:40.038 INFO:tasks.rados.rados.0.smithi107.stdout:1845: left oid 7 (ObjNum 1844 snap 0 seq_num 1844) 2020-06-12T02:37:40.039 INFO:tasks.rados.rados.0.smithi107.stdout:1845: done (1 left) 2020-06-12T02:37:40.039 INFO:tasks.rados.rados.0.smithi107.stdout:1849: done (0 left) 2020-06-12T02:37:40.039 INFO:tasks.rados.rados.0.smithi107.stdout:1850: delete oid 15 current snap is 0 2020-06-12T02:37:40.041 INFO:tasks.rados.rados.0.smithi107.stdout:1850: done (0 left) 2020-06-12T02:37:40.042 INFO:tasks.rados.rados.0.smithi107.stdout:1851: delete oid 15 current snap is 0 2020-06-12T02:37:40.048 INFO:tasks.rados.rados.0.smithi107.stdout:1851: done (0 left) 2020-06-12T02:37:40.048 INFO:tasks.rados.rados.0.smithi107.stdout:1852: delete oid 5 current snap is 0 2020-06-12T02:37:40.050 INFO:tasks.rados.rados.0.smithi107.stdout:1852: done (0 left) 2020-06-12T02:37:40.050 INFO:tasks.rados.rados.0.smithi107.stdout:1853: delete oid 14 current snap is 0 2020-06-12T02:37:40.052 INFO:tasks.rados.rados.0.smithi107.stdout:1853: done (0 left) 2020-06-12T02:37:40.052 INFO:tasks.rados.rados.0.smithi107.stdout:1854: delete oid 2 current snap is 0 2020-06-12T02:37:40.053 INFO:tasks.rados.rados.0.smithi107.stdout:1854: done (0 left) 2020-06-12T02:37:40.053 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:37:40.054 INFO:tasks.rados.rados.0.smithi107.stdout:1855: seq_num 1854 ranges {3129344=655360,3784704=425984,4210688=614400,4825088=565248} 2020-06-12T02:37:40.073 INFO:tasks.rados.rados.0.smithi107.stdout:1855: writing smithi10729407-18 from 3129344 to 3784704 tid 1 2020-06-12T02:37:40.076 INFO:tasks.rados.rados.0.smithi107.stdout:1855: writing smithi10729407-18 from 3784704 to 4210688 tid 2 2020-06-12T02:37:40.079 INFO:tasks.rados.rados.0.smithi107.stdout:1855: writing smithi10729407-18 from 4210688 to 4825088 tid 3 2020-06-12T02:37:40.083 INFO:tasks.rados.rados.0.smithi107.stdout:1855: writing smithi10729407-18 from 4825088 to 5390336 tid 4 2020-06-12T02:37:40.083 INFO:tasks.rados.rados.0.smithi107.stdout:1856: delete oid 13 current snap is 0 2020-06-12T02:37:40.083 INFO:tasks.rados.rados.0.smithi107.stdout:1855: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:40.085 INFO:tasks.rados.rados.0.smithi107.stdout:1856: done (1 left) 2020-06-12T02:37:40.085 INFO:tasks.rados.rados.0.smithi107.stdout:1857: delete oid 2 current snap is 0 2020-06-12T02:37:40.086 INFO:tasks.rados.rados.0.smithi107.stdout:1857: done (1 left) 2020-06-12T02:37:40.087 INFO:tasks.rados.rados.0.smithi107.stdout:1858: delete oid 1 current snap is 0 2020-06-12T02:37:40.090 INFO:tasks.rados.rados.0.smithi107.stdout:1855: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:40.090 INFO:tasks.rados.rados.0.smithi107.stdout:1855: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:40.090 INFO:tasks.rados.rados.0.smithi107.stdout:1855: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:40.091 INFO:tasks.rados.rados.0.smithi107.stdout:1855: finishing write tid 6 to smithi10729407-18 2020-06-12T02:37:40.091 INFO:tasks.rados.rados.0.smithi107.stdout:1858: done (1 left) 2020-06-12T02:37:40.092 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:37:40.092 INFO:tasks.rados.rados.0.smithi107.stdout:1859: seq_num 1858 ranges {1957888=720896,2678784=417792,3096576=598016,3694592=589824} 2020-06-12T02:37:40.093 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [7] live_osds: [0, 1, 2, 3, 4, 5, 6] 2020-06-12T02:37:40.093 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2020-06-12T02:37:40.094 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.000000 2020-06-12T02:37:40.094 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd primary-affinity 5 0 2020-06-12T02:37:40.106 INFO:tasks.rados.rados.0.smithi107.stdout:1859: writing smithi10729407-6 from 1957888 to 2678784 tid 1 2020-06-12T02:37:40.108 INFO:tasks.rados.rados.0.smithi107.stdout:1859: writing smithi10729407-6 from 2678784 to 3096576 tid 2 2020-06-12T02:37:40.112 INFO:tasks.rados.rados.0.smithi107.stdout:1859: writing smithi10729407-6 from 3096576 to 3694592 tid 3 2020-06-12T02:37:40.116 INFO:tasks.rados.rados.0.smithi107.stdout:1859: writing smithi10729407-6 from 3694592 to 4284416 tid 4 2020-06-12T02:37:40.116 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:37:40.116 INFO:tasks.rados.rados.0.smithi107.stdout:1860: seq_num 1859 ranges {0=737280,737280=704512,1441792=802816,2244608=98304} 2020-06-12T02:37:40.120 INFO:tasks.rados.rados.0.smithi107.stdout:1860: writing smithi10729407-19 from 0 to 737280 tid 1 2020-06-12T02:37:40.124 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:40.124 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:40.125 INFO:tasks.rados.rados.0.smithi107.stdout:1860: writing smithi10729407-19 from 737280 to 1441792 tid 2 2020-06-12T02:37:40.128 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:37:40.125+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:37:40.129 INFO:tasks.rados.rados.0.smithi107.stdout:1860: writing smithi10729407-19 from 1441792 to 2244608 tid 3 2020-06-12T02:37:40.130 INFO:tasks.rados.rados.0.smithi107.stdout:1860: writing smithi10729407-19 from 2244608 to 2342912 tid 4 2020-06-12T02:37:40.130 INFO:tasks.rados.rados.0.smithi107.stdout:1861: delete oid 2 current snap is 0 2020-06-12T02:37:40.130 INFO:tasks.rados.rados.0.smithi107.stdout:1855: finishing write tid 7 to smithi10729407-18 2020-06-12T02:37:40.130 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 317 (ObjNum 1854 snap 0 seq_num 1854) dirty exists 2020-06-12T02:37:40.131 INFO:tasks.rados.rados.0.smithi107.stdout:1855: left oid 18 (ObjNum 1854 snap 0 seq_num 1854) 2020-06-12T02:37:40.131 INFO:tasks.rados.rados.0.smithi107.stdout:1859: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:40.131 INFO:tasks.rados.rados.0.smithi107.stdout:1859: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:40.131 INFO:tasks.rados.rados.0.smithi107.stdout:1859: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:40.131 INFO:tasks.rados.rados.0.smithi107.stdout:1859: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:40.131 INFO:tasks.rados.rados.0.smithi107.stdout:1859: finishing write tid 6 to smithi10729407-6 2020-06-12T02:37:40.132 INFO:tasks.rados.rados.0.smithi107.stdout:1859: finishing write tid 7 to smithi10729407-6 2020-06-12T02:37:40.132 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 373 (ObjNum 1858 snap 0 seq_num 1858) dirty exists 2020-06-12T02:37:40.132 INFO:tasks.rados.rados.0.smithi107.stdout:1859: left oid 6 (ObjNum 1858 snap 0 seq_num 1858) 2020-06-12T02:37:40.133 INFO:tasks.rados.rados.0.smithi107.stdout:1860: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:40.133 INFO:tasks.rados.rados.0.smithi107.stdout:1855: done (3 left) 2020-06-12T02:37:40.133 INFO:tasks.rados.rados.0.smithi107.stdout:1859: done (2 left) 2020-06-12T02:37:40.133 INFO:tasks.rados.rados.0.smithi107.stdout:1861: done (1 left) 2020-06-12T02:37:40.133 INFO:tasks.rados.rados.0.smithi107.stdout:1862: delete oid 15 current snap is 0 2020-06-12T02:37:40.134 INFO:tasks.rados.rados.0.smithi107.stdout:1862: done (1 left) 2020-06-12T02:37:40.134 INFO:tasks.rados.rados.0.smithi107.stdout:1863: delete oid 14 current snap is 0 2020-06-12T02:37:40.136 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-06-12T02:37:40.137 INFO:tasks.rados.rados.0.smithi107.stdout:1863: done (1 left) 2020-06-12T02:37:40.137 INFO:tasks.rados.rados.0.smithi107.stdout:1864: delete oid 7 current snap is 0 2020-06-12T02:37:40.139 INFO:tasks.rados.rados.0.smithi107.stdout:1860: finishing write tid 2 to smithi10729407-19 2020-06-12T02:37:40.140 INFO:tasks.rados.rados.0.smithi107.stdout:1860: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:40.141 INFO:tasks.rados.rados.0.smithi107.stdout:1864: done (1 left) 2020-06-12T02:37:40.142 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:37:40.142 INFO:tasks.rados.rados.0.smithi107.stdout:1865: seq_num 1864 ranges {0=24576} 2020-06-12T02:37:40.142 INFO:tasks.rados.rados.0.smithi107.stdout:1865: writing smithi10729407-15 from 0 to 24576 tid 1 2020-06-12T02:37:40.142 INFO:tasks.rados.rados.0.smithi107.stdout:1866: delete oid 17 current snap is 0 2020-06-12T02:37:40.144 INFO:tasks.rados.rados.0.smithi107.stdout:1860: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:40.144 INFO:tasks.rados.rados.0.smithi107.stdout:1860: finishing write tid 6 to smithi10729407-19 2020-06-12T02:37:40.146 INFO:tasks.rados.rados.0.smithi107.stdout:1860: finishing write tid 7 to smithi10729407-19 2020-06-12T02:37:40.146 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 289 (ObjNum 1859 snap 0 seq_num 1859) dirty exists 2020-06-12T02:37:40.146 INFO:tasks.rados.rados.0.smithi107.stdout:1860: left oid 19 (ObjNum 1859 snap 0 seq_num 1859) 2020-06-12T02:37:40.147 INFO:tasks.rados.rados.0.smithi107.stdout:1860: done (2 left) 2020-06-12T02:37:40.147 INFO:tasks.rados.rados.0.smithi107.stdout:1866: done (1 left) 2020-06-12T02:37:40.147 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:37:40.147 INFO:tasks.rados.rados.0.smithi107.stdout:1867: seq_num 1866 ranges {0=57344} 2020-06-12T02:37:40.148 INFO:tasks.rados.rados.0.smithi107.stdout:1867: writing smithi10729407-9 from 0 to 57344 tid 1 2020-06-12T02:37:40.148 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:37:40.148 INFO:tasks.rados.rados.0.smithi107.stdout:1868: seq_num 1867 ranges {4284416=73728} 2020-06-12T02:37:40.171 INFO:tasks.rados.rados.0.smithi107.stdout:1868: writing smithi10729407-6 from 4284416 to 4358144 tid 1 2020-06-12T02:37:40.172 INFO:tasks.rados.rados.0.smithi107.stdout:1869: delete oid 14 current snap is 0 2020-06-12T02:37:40.172 INFO:tasks.rados.rados.0.smithi107.stdout:1865: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:40.172 INFO:tasks.rados.rados.0.smithi107.stdout:1865: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:40.172 INFO:tasks.rados.rados.0.smithi107.stdout:1865: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:40.172 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 326 (ObjNum 1864 snap 0 seq_num 1864) dirty exists 2020-06-12T02:37:40.172 INFO:tasks.rados.rados.0.smithi107.stdout:1865: left oid 15 (ObjNum 1864 snap 0 seq_num 1864) 2020-06-12T02:37:40.173 INFO:tasks.rados.rados.0.smithi107.stdout:1867: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:40.173 INFO:tasks.rados.rados.0.smithi107.stdout:1867: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:40.173 INFO:tasks.rados.rados.0.smithi107.stdout:1867: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:40.173 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 319 (ObjNum 1866 snap 0 seq_num 1866) dirty exists 2020-06-12T02:37:40.173 INFO:tasks.rados.rados.0.smithi107.stdout:1867: left oid 9 (ObjNum 1866 snap 0 seq_num 1866) 2020-06-12T02:37:40.174 INFO:tasks.rados.rados.0.smithi107.stdout:1868: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:40.174 INFO:tasks.rados.rados.0.smithi107.stdout:1868: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:40.175 INFO:tasks.rados.rados.0.smithi107.stdout:1865: done (3 left) 2020-06-12T02:37:40.175 INFO:tasks.rados.rados.0.smithi107.stdout:1867: done (2 left) 2020-06-12T02:37:40.175 INFO:tasks.rados.rados.0.smithi107.stdout:1869: done (1 left) 2020-06-12T02:37:40.175 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:37:40.176 INFO:tasks.rados.rados.0.smithi107.stdout:1870: seq_num 1869 ranges {0=106496} 2020-06-12T02:37:40.176 INFO:tasks.rados.rados.0.smithi107.stdout:1870: writing smithi10729407-10 from 0 to 106496 tid 1 2020-06-12T02:37:40.177 INFO:tasks.rados.rados.0.smithi107.stdout:1871: delete oid 18 current snap is 0 2020-06-12T02:37:40.177 INFO:tasks.rados.rados.0.smithi107.stdout:1868: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:40.177 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 375 (ObjNum 1867 snap 0 seq_num 1867) dirty exists 2020-06-12T02:37:40.177 INFO:tasks.rados.rados.0.smithi107.stdout:1868: left oid 6 (ObjNum 1867 snap 0 seq_num 1867) 2020-06-12T02:37:40.178 INFO:tasks.rados.rados.0.smithi107.stdout:1868: done (2 left) 2020-06-12T02:37:40.179 INFO:tasks.rados.rados.0.smithi107.stdout:1871: done (1 left) 2020-06-12T02:37:40.179 INFO:tasks.rados.rados.0.smithi107.stdout:1872: delete oid 2 current snap is 0 2020-06-12T02:37:40.179 INFO:tasks.rados.rados.0.smithi107.stdout:1870: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:40.182 INFO:tasks.rados.rados.0.smithi107.stdout:1870: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:40.183 INFO:tasks.rados.rados.0.smithi107.stdout:1870: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:40.184 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 179 (ObjNum 1869 snap 0 seq_num 1869) dirty exists 2020-06-12T02:37:40.184 INFO:tasks.rados.rados.0.smithi107.stdout:1870: left oid 10 (ObjNum 1869 snap 0 seq_num 1869) 2020-06-12T02:37:40.185 INFO:tasks.rados.rados.0.smithi107.stdout:1870: done (1 left) 2020-06-12T02:37:40.185 INFO:tasks.rados.rados.0.smithi107.stdout:1872: done (0 left) 2020-06-12T02:37:40.185 INFO:tasks.rados.rados.0.smithi107.stdout:1873: delete oid 5 current snap is 0 2020-06-12T02:37:40.186 INFO:tasks.rados.rados.0.smithi107.stdout:1873: done (0 left) 2020-06-12T02:37:40.186 INFO:tasks.rados.rados.0.smithi107.stdout:1874: delete oid 15 current snap is 0 2020-06-12T02:37:40.188 INFO:tasks.rados.rados.0.smithi107.stdout:1874: done (0 left) 2020-06-12T02:37:40.189 INFO:tasks.rados.rados.0.smithi107.stdout:1875: delete oid 12 current snap is 0 2020-06-12T02:37:40.191 INFO:tasks.rados.rados.0.smithi107.stdout:1875: done (0 left) 2020-06-12T02:37:40.192 INFO:tasks.rados.rados.0.smithi107.stdout:1876: delete oid 16 current snap is 0 2020-06-12T02:37:40.193 INFO:tasks.rados.rados.0.smithi107.stdout:1876: done (0 left) 2020-06-12T02:37:40.193 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:37:40.193 INFO:tasks.rados.rados.0.smithi107.stdout:1877: seq_num 1876 ranges {0=221184} 2020-06-12T02:37:40.194 INFO:tasks.rados.rados.0.smithi107.stdout:1877: writing smithi10729407-11 from 0 to 221184 tid 1 2020-06-12T02:37:40.194 INFO:tasks.rados.rados.0.smithi107.stdout:1878: delete oid 2 current snap is 0 2020-06-12T02:37:40.196 INFO:tasks.rados.rados.0.smithi107.stdout:1878: done (1 left) 2020-06-12T02:37:40.196 INFO:tasks.rados.rados.0.smithi107.stdout:1879: delete oid 5 current snap is 0 2020-06-12T02:37:40.197 INFO:tasks.rados.rados.0.smithi107.stdout:1879: done (1 left) 2020-06-12T02:37:40.198 INFO:tasks.rados.rados.0.smithi107.stdout:1880: delete oid 17 current snap is 0 2020-06-12T02:37:40.198 INFO:tasks.rados.rados.0.smithi107.stdout:1877: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:40.199 INFO:tasks.rados.rados.0.smithi107.stdout:1877: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:40.199 INFO:tasks.rados.rados.0.smithi107.stdout:1880: done (1 left) 2020-06-12T02:37:40.199 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:37:40.200 INFO:tasks.rados.rados.0.smithi107.stdout:1881: seq_num 1880 ranges {0=294912} 2020-06-12T02:37:40.201 INFO:tasks.rados.rados.0.smithi107.stdout:1881: writing smithi10729407-1 from 0 to 294912 tid 1 2020-06-12T02:37:40.201 INFO:tasks.rados.rados.0.smithi107.stdout:1882: delete oid 19 current snap is 0 2020-06-12T02:37:40.202 INFO:tasks.rados.rados.0.smithi107.stdout:1877: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:40.202 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 338 (ObjNum 1876 snap 0 seq_num 1876) dirty exists 2020-06-12T02:37:40.202 INFO:tasks.rados.rados.0.smithi107.stdout:1877: left oid 11 (ObjNum 1876 snap 0 seq_num 1876) 2020-06-12T02:37:40.203 INFO:tasks.rados.rados.0.smithi107.stdout:1877: done (2 left) 2020-06-12T02:37:40.203 INFO:tasks.rados.rados.0.smithi107.stdout:1882: done (1 left) 2020-06-12T02:37:40.203 INFO:tasks.rados.rados.0.smithi107.stdout:1883: delete oid 4 current snap is 0 2020-06-12T02:37:40.205 INFO:tasks.rados.rados.0.smithi107.stdout:1883: done (1 left) 2020-06-12T02:37:40.205 INFO:tasks.rados.rados.0.smithi107.stdout:1884: delete oid 5 current snap is 0 2020-06-12T02:37:40.205 INFO:tasks.rados.rados.0.smithi107.stdout:1881: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:40.207 INFO:tasks.rados.rados.0.smithi107.stdout:1881: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:40.208 INFO:tasks.rados.rados.0.smithi107.stdout:1884: done (1 left) 2020-06-12T02:37:40.208 INFO:tasks.rados.rados.0.smithi107.stdout:1885: delete oid 14 current snap is 0 2020-06-12T02:37:40.209 INFO:tasks.rados.rados.0.smithi107.stdout:1881: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:40.209 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 377 (ObjNum 1880 snap 0 seq_num 1880) dirty exists 2020-06-12T02:37:40.209 INFO:tasks.rados.rados.0.smithi107.stdout:1881: left oid 1 (ObjNum 1880 snap 0 seq_num 1880) 2020-06-12T02:37:40.209 INFO:tasks.rados.rados.0.smithi107.stdout:1881: done (1 left) 2020-06-12T02:37:40.209 INFO:tasks.rados.rados.0.smithi107.stdout:1885: done (0 left) 2020-06-12T02:37:40.210 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:37:40.210 INFO:tasks.rados.rados.0.smithi107.stdout:1886: seq_num 1885 ranges {0=376832} 2020-06-12T02:37:40.210 INFO:tasks.rados.rados.0.smithi107.stdout:1886: writing smithi10729407-17 from 0 to 376832 tid 1 2020-06-12T02:37:40.211 INFO:tasks.rados.rados.0.smithi107.stdout:1887: delete oid 18 current snap is 0 2020-06-12T02:37:40.212 INFO:tasks.rados.rados.0.smithi107.stdout:1887: done (1 left) 2020-06-12T02:37:40.212 INFO:tasks.rados.rados.0.smithi107.stdout:1888: delete oid 18 current snap is 0 2020-06-12T02:37:40.214 INFO:tasks.rados.rados.0.smithi107.stdout:1888: done (1 left) 2020-06-12T02:37:40.215 INFO:tasks.rados.rados.0.smithi107.stdout:1889: delete oid 8 current snap is 0 2020-06-12T02:37:40.216 INFO:tasks.rados.rados.0.smithi107.stdout:1889: done (1 left) 2020-06-12T02:37:40.216 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:37:40.217 INFO:tasks.rados.rados.0.smithi107.stdout:1890: seq_num 1889 ranges {294912=442368} 2020-06-12T02:37:40.221 INFO:tasks.rados.rados.0.smithi107.stdout:1890: writing smithi10729407-1 from 294912 to 737280 tid 1 2020-06-12T02:37:40.221 INFO:tasks.rados.rados.0.smithi107.stdout:1891: delete oid 7 current snap is 0 2020-06-12T02:37:40.221 INFO:tasks.rados.rados.0.smithi107.stdout:1886: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:40.221 INFO:tasks.rados.rados.0.smithi107.stdout:1886: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:40.222 INFO:tasks.rados.rados.0.smithi107.stdout:1886: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:40.222 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 368 (ObjNum 1885 snap 0 seq_num 1885) dirty exists 2020-06-12T02:37:40.222 INFO:tasks.rados.rados.0.smithi107.stdout:1886: left oid 17 (ObjNum 1885 snap 0 seq_num 1885) 2020-06-12T02:37:40.223 INFO:tasks.rados.rados.0.smithi107.stdout:1886: done (2 left) 2020-06-12T02:37:40.223 INFO:tasks.rados.rados.0.smithi107.stdout:1891: done (1 left) 2020-06-12T02:37:40.223 INFO:tasks.rados.rados.0.smithi107.stdout:1892: delete oid 6 current snap is 0 2020-06-12T02:37:40.226 INFO:tasks.rados.rados.0.smithi107.stdout:1890: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:40.226 INFO:tasks.rados.rados.0.smithi107.stdout:1890: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:40.227 INFO:tasks.rados.rados.0.smithi107.stdout:1890: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:40.227 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 379 (ObjNum 1889 snap 0 seq_num 1889) dirty exists 2020-06-12T02:37:40.228 INFO:tasks.rados.rados.0.smithi107.stdout:1890: left oid 1 (ObjNum 1889 snap 0 seq_num 1889) 2020-06-12T02:37:40.230 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:37:40.227+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:37:40.231 INFO:tasks.rados.rados.0.smithi107.stdout:1890: done (1 left) 2020-06-12T02:37:40.232 INFO:tasks.rados.rados.0.smithi107.stdout:1892: done (0 left) 2020-06-12T02:37:40.232 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:37:40.232 INFO:tasks.rados.rados.0.smithi107.stdout:1893: seq_num 1892 ranges {0=491520} 2020-06-12T02:37:40.234 INFO:tasks.rados.rados.0.smithi107.stdout:1893: writing smithi10729407-7 from 0 to 491520 tid 1 2020-06-12T02:37:40.234 INFO:tasks.rados.rados.0.smithi107.stdout:1894: delete oid 1 current snap is 0 2020-06-12T02:37:40.236 INFO:tasks.rados.rados.0.smithi107.stdout:1894: done (1 left) 2020-06-12T02:37:40.236 INFO:tasks.rados.rados.0.smithi107.stdout:1895: delete oid 9 current snap is 0 2020-06-12T02:37:40.238 INFO:tasks.rados.rados.0.smithi107.stdout:1895: done (1 left) 2020-06-12T02:37:40.238 INFO:tasks.rados.rados.0.smithi107.stdout:1896: delete oid 10 current snap is 0 2020-06-12T02:37:40.242 INFO:tasks.rados.rados.0.smithi107.stdout:1896: done (1 left) 2020-06-12T02:37:40.243 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:37:40.243 INFO:tasks.rados.rados.0.smithi107.stdout:1897: seq_num 1896 ranges {0=557056} 2020-06-12T02:37:40.247 INFO:tasks.rados.rados.0.smithi107.stdout:1897: writing smithi10729407-6 from 0 to 557056 tid 1 2020-06-12T02:37:40.247 INFO:tasks.rados.rados.0.smithi107.stdout:1898: delete oid 14 current snap is 0 2020-06-12T02:37:40.247 INFO:tasks.rados.rados.0.smithi107.stdout:1893: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:40.247 INFO:tasks.rados.rados.0.smithi107.stdout:1893: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:40.247 INFO:tasks.rados.rados.0.smithi107.stdout:1893: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:40.248 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 370 (ObjNum 1892 snap 0 seq_num 1892) dirty exists 2020-06-12T02:37:40.248 INFO:tasks.rados.rados.0.smithi107.stdout:1893: left oid 7 (ObjNum 1892 snap 0 seq_num 1892) 2020-06-12T02:37:40.252 INFO:tasks.rados.rados.0.smithi107.stdout:1897: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:40.252 INFO:tasks.rados.rados.0.smithi107.stdout:1893: done (2 left) 2020-06-12T02:37:40.253 INFO:tasks.rados.rados.0.smithi107.stdout:1898: done (1 left) 2020-06-12T02:37:40.253 INFO:tasks.rados.rados.0.smithi107.stdout:1899: delete oid 12 current snap is 0 2020-06-12T02:37:40.253 INFO:tasks.rados.rados.0.smithi107.stdout:1897: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:40.254 INFO:tasks.rados.rados.0.smithi107.stdout:1899: done (1 left) 2020-06-12T02:37:40.254 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:37:40.255 INFO:tasks.rados.rados.0.smithi107.stdout:1900: seq_num 1899 ranges {0=614400} 2020-06-12T02:37:40.259 INFO:tasks.rados.rados.0.smithi107.stdout:1900: writing smithi10729407-13 from 0 to 614400 tid 1 2020-06-12T02:37:40.259 INFO:tasks.rados.rados.0.smithi107.stdout:1901: delete oid 14 current snap is 0 2020-06-12T02:37:40.259 INFO:tasks.rados.rados.0.smithi107.stdout:1897: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:40.260 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 383 (ObjNum 1896 snap 0 seq_num 1896) dirty exists 2020-06-12T02:37:40.260 INFO:tasks.rados.rados.0.smithi107.stdout:1897: left oid 6 (ObjNum 1896 snap 0 seq_num 1896) 2020-06-12T02:37:40.261 INFO:tasks.rados.rados.0.smithi107.stdout:1897: done (2 left) 2020-06-12T02:37:40.261 INFO:tasks.rados.rados.0.smithi107.stdout:1901: done (1 left) 2020-06-12T02:37:40.261 INFO:tasks.rados.rados.0.smithi107.stdout:1902: delete oid 15 current snap is 0 2020-06-12T02:37:40.266 INFO:tasks.rados.rados.0.smithi107.stdout:1902: done (1 left) 2020-06-12T02:37:40.266 INFO:tasks.rados.rados.0.smithi107.stdout:1903: delete oid 20 current snap is 0 2020-06-12T02:37:40.268 INFO:tasks.rados.rados.0.smithi107.stdout:1900: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:40.268 INFO:tasks.rados.rados.0.smithi107.stdout:1903: done (1 left) 2020-06-12T02:37:40.269 INFO:tasks.rados.rados.0.smithi107.stdout:1904: delete oid 17 current snap is 0 2020-06-12T02:37:40.274 INFO:tasks.rados.rados.0.smithi107.stdout:1900: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:40.274 INFO:tasks.rados.rados.0.smithi107.stdout:1904: done (1 left) 2020-06-12T02:37:40.274 INFO:tasks.rados.rados.0.smithi107.stdout:1905: delete oid 11 current snap is 0 2020-06-12T02:37:40.275 INFO:tasks.rados.rados.0.smithi107.stdout:1900: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:40.275 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 372 (ObjNum 1899 snap 0 seq_num 1899) dirty exists 2020-06-12T02:37:40.275 INFO:tasks.rados.rados.0.smithi107.stdout:1900: left oid 13 (ObjNum 1899 snap 0 seq_num 1899) 2020-06-12T02:37:40.276 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:40.276 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:40.276 INFO:tasks.rados.rados.0.smithi107.stdout:1900: done (1 left) 2020-06-12T02:37:40.276 INFO:tasks.rados.rados.0.smithi107.stdout:1905: done (0 left) 2020-06-12T02:37:40.276 INFO:tasks.rados.rados.0.smithi107.stdout:1906: delete oid 6 current snap is 0 2020-06-12T02:37:40.278 INFO:tasks.rados.rados.0.smithi107.stdout:1906: done (0 left) 2020-06-12T02:37:40.278 INFO:tasks.rados.rados.0.smithi107.stdout:1907: delete oid 2 current snap is 0 2020-06-12T02:37:40.280 INFO:tasks.rados.rados.0.smithi107.stdout:1907: done (0 left) 2020-06-12T02:37:40.280 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:37:40.280 INFO:tasks.rados.rados.0.smithi107.stdout:1908: seq_num 1907 ranges {0=745472} 2020-06-12T02:37:40.284 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-06-12T02:37:40.285 INFO:tasks.rados.rados.0.smithi107.stdout:1908: writing smithi10729407-15 from 0 to 745472 tid 1 2020-06-12T02:37:40.285 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:37:40.286 INFO:tasks.rados.rados.0.smithi107.stdout:1909: seq_num 1908 ranges {0=761856} 2020-06-12T02:37:40.289 INFO:tasks.rados.rados.0.smithi107.stdout:1909: writing smithi10729407-5 from 0 to 761856 tid 1 2020-06-12T02:37:40.289 INFO:tasks.rados.rados.0.smithi107.stdout:1910: delete oid 18 current snap is 0 2020-06-12T02:37:40.291 INFO:tasks.rados.rados.0.smithi107.stdout:1910: done (2 left) 2020-06-12T02:37:40.291 INFO:tasks.rados.rados.0.smithi107.stdout:1911: delete oid 14 current snap is 0 2020-06-12T02:37:40.293 INFO:tasks.rados.rados.0.smithi107.stdout:1911: done (2 left) 2020-06-12T02:37:40.293 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:37:40.293 INFO:tasks.rados.rados.0.smithi107.stdout:1912: seq_num 1911 ranges {0=409600,409600=401408} 2020-06-12T02:37:40.296 INFO:tasks.rados.rados.0.smithi107.stdout:1912: writing smithi10729407-12 from 0 to 409600 tid 1 2020-06-12T02:37:40.299 INFO:tasks.rados.rados.0.smithi107.stdout:1912: writing smithi10729407-12 from 409600 to 811008 tid 2 2020-06-12T02:37:40.299 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:37:40.299 INFO:tasks.rados.rados.0.smithi107.stdout:1913: seq_num 1912 ranges {0=425984,425984=401408} 2020-06-12T02:37:40.301 INFO:tasks.rados.rados.0.smithi107.stdout:1913: writing smithi10729407-2 from 0 to 425984 tid 1 2020-06-12T02:37:40.304 INFO:tasks.rados.rados.0.smithi107.stdout:1913: writing smithi10729407-2 from 425984 to 827392 tid 2 2020-06-12T02:37:40.304 INFO:tasks.rados.rados.0.smithi107.stdout:1914: delete oid 1 current snap is 0 2020-06-12T02:37:40.304 INFO:tasks.rados.rados.0.smithi107.stdout:1908: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:40.305 INFO:tasks.rados.rados.0.smithi107.stdout:1909: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:40.305 INFO:tasks.rados.rados.0.smithi107.stdout:1909: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:40.305 INFO:tasks.rados.rados.0.smithi107.stdout:1909: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:40.305 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 156 (ObjNum 1908 snap 0 seq_num 1908) dirty exists 2020-06-12T02:37:40.306 INFO:tasks.rados.rados.0.smithi107.stdout:1909: left oid 5 (ObjNum 1908 snap 0 seq_num 1908) 2020-06-12T02:37:40.306 INFO:tasks.rados.rados.0.smithi107.stdout:1908: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:40.306 INFO:tasks.rados.rados.0.smithi107.stdout:1908: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:40.306 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 331 (ObjNum 1907 snap 0 seq_num 1907) dirty exists 2020-06-12T02:37:40.306 INFO:tasks.rados.rados.0.smithi107.stdout:1908: left oid 15 (ObjNum 1907 snap 0 seq_num 1907) 2020-06-12T02:37:40.307 INFO:tasks.rados.rados.0.smithi107.stdout:1912: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:40.307 INFO:tasks.rados.rados.0.smithi107.stdout:1908: done (4 left) 2020-06-12T02:37:40.307 INFO:tasks.rados.rados.0.smithi107.stdout:1909: done (3 left) 2020-06-12T02:37:40.307 INFO:tasks.rados.rados.0.smithi107.stdout:1914: done (2 left) 2020-06-12T02:37:40.308 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:37:40.308 INFO:tasks.rados.rados.0.smithi107.stdout:1915: seq_num 1914 ranges {745472=458752,1204224=401408} 2020-06-12T02:37:40.313 INFO:tasks.rados.rados.0.smithi107.stdout:1915: writing smithi10729407-15 from 745472 to 1204224 tid 1 2020-06-12T02:37:40.316 INFO:tasks.rados.rados.0.smithi107.stdout:1915: writing smithi10729407-15 from 1204224 to 1605632 tid 2 2020-06-12T02:37:40.317 INFO:tasks.rados.rados.0.smithi107.stdout:1916: delete oid 3 current snap is 0 2020-06-12T02:37:40.317 INFO:tasks.rados.rados.0.smithi107.stdout:1912: finishing write tid 2 to smithi10729407-12 2020-06-12T02:37:40.317 INFO:tasks.rados.rados.0.smithi107.stdout:1912: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:40.317 INFO:tasks.rados.rados.0.smithi107.stdout:1912: finishing write tid 5 to smithi10729407-12 2020-06-12T02:37:40.317 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 295 (ObjNum 1911 snap 0 seq_num 1911) dirty exists 2020-06-12T02:37:40.318 INFO:tasks.rados.rados.0.smithi107.stdout:1912: left oid 12 (ObjNum 1911 snap 0 seq_num 1911) 2020-06-12T02:37:40.318 INFO:tasks.rados.rados.0.smithi107.stdout:1913: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:40.318 INFO:tasks.rados.rados.0.smithi107.stdout:1913: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:40.318 INFO:tasks.rados.rados.0.smithi107.stdout:1913: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:40.318 INFO:tasks.rados.rados.0.smithi107.stdout:1913: finishing write tid 5 to smithi10729407-2 2020-06-12T02:37:40.318 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 179 (ObjNum 1912 snap 0 seq_num 1912) dirty exists 2020-06-12T02:37:40.319 INFO:tasks.rados.rados.0.smithi107.stdout:1913: left oid 2 (ObjNum 1912 snap 0 seq_num 1912) 2020-06-12T02:37:40.319 INFO:tasks.rados.rados.0.smithi107.stdout:1915: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:40.319 INFO:tasks.rados.rados.0.smithi107.stdout:1912: done (3 left) 2020-06-12T02:37:40.320 INFO:tasks.rados.rados.0.smithi107.stdout:1913: done (2 left) 2020-06-12T02:37:40.320 INFO:tasks.rados.rados.0.smithi107.stdout:1916: done (1 left) 2020-06-12T02:37:40.320 INFO:tasks.rados.rados.0.smithi107.stdout:1917: delete oid 4 current snap is 0 2020-06-12T02:37:40.321 INFO:tasks.rados.rados.0.smithi107.stdout:1917: done (1 left) 2020-06-12T02:37:40.321 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:37:40.322 INFO:tasks.rados.rados.0.smithi107.stdout:1918: seq_num 1917 ranges {761856=507904,1269760=401408} 2020-06-12T02:37:40.329 INFO:tasks.rados.rados.0.smithi107.stdout:1918: writing smithi10729407-5 from 761856 to 1269760 tid 1 2020-06-12T02:37:40.331 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:37:40.328+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:37:40.331 INFO:tasks.rados.rados.0.smithi107.stdout:1918: writing smithi10729407-5 from 1269760 to 1671168 tid 2 2020-06-12T02:37:40.332 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:37:40.332 INFO:tasks.rados.rados.0.smithi107.stdout:1919: seq_num 1918 ranges {0=532480,532480=401408} 2020-06-12T02:37:40.334 INFO:tasks.rados.rados.0.smithi107.stdout:1919: writing smithi10729407-18 from 0 to 532480 tid 1 2020-06-12T02:37:40.337 INFO:tasks.rados.rados.0.smithi107.stdout:1919: writing smithi10729407-18 from 532480 to 933888 tid 2 2020-06-12T02:37:40.337 INFO:tasks.rados.rados.0.smithi107.stdout:1920: delete oid 4 current snap is 0 2020-06-12T02:37:40.337 INFO:tasks.rados.rados.0.smithi107.stdout:1915: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:40.338 INFO:tasks.rados.rados.0.smithi107.stdout:1915: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:40.338 INFO:tasks.rados.rados.0.smithi107.stdout:1915: finishing write tid 5 to smithi10729407-15 2020-06-12T02:37:40.339 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 334 (ObjNum 1914 snap 0 seq_num 1914) dirty exists 2020-06-12T02:37:40.339 INFO:tasks.rados.rados.0.smithi107.stdout:1915: left oid 15 (ObjNum 1914 snap 0 seq_num 1914) 2020-06-12T02:37:40.339 INFO:tasks.rados.rados.0.smithi107.stdout:1918: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:40.340 INFO:tasks.rados.rados.0.smithi107.stdout:1915: done (3 left) 2020-06-12T02:37:40.340 INFO:tasks.rados.rados.0.smithi107.stdout:1920: done (2 left) 2020-06-12T02:37:40.340 INFO:tasks.rados.rados.0.smithi107.stdout:1921: delete oid 2 current snap is 0 2020-06-12T02:37:40.341 INFO:tasks.rados.rados.0.smithi107.stdout:1918: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:40.344 INFO:tasks.rados.rados.0.smithi107.stdout:1919: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:40.344 INFO:tasks.rados.rados.0.smithi107.stdout:1918: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:40.345 INFO:tasks.rados.rados.0.smithi107.stdout:1918: finishing write tid 5 to smithi10729407-5 2020-06-12T02:37:40.345 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 159 (ObjNum 1917 snap 0 seq_num 1917) dirty exists 2020-06-12T02:37:40.346 INFO:tasks.rados.rados.0.smithi107.stdout:1918: left oid 5 (ObjNum 1917 snap 0 seq_num 1917) 2020-06-12T02:37:40.346 INFO:tasks.rados.rados.0.smithi107.stdout:1919: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:40.347 INFO:tasks.rados.rados.0.smithi107.stdout:1919: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:40.347 INFO:tasks.rados.rados.0.smithi107.stdout:1918: done (2 left) 2020-06-12T02:37:40.347 INFO:tasks.rados.rados.0.smithi107.stdout:1921: done (1 left) 2020-06-12T02:37:40.348 INFO:tasks.rados.rados.0.smithi107.stdout:1922: delete oid 2 current snap is 0 2020-06-12T02:37:40.349 INFO:tasks.rados.rados.0.smithi107.stdout:1919: finishing write tid 5 to smithi10729407-18 2020-06-12T02:37:40.350 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 327 (ObjNum 1918 snap 0 seq_num 1918) dirty exists 2020-06-12T02:37:40.350 INFO:tasks.rados.rados.0.smithi107.stdout:1919: left oid 18 (ObjNum 1918 snap 0 seq_num 1918) 2020-06-12T02:37:40.352 INFO:tasks.rados.rados.0.smithi107.stdout:1919: done (1 left) 2020-06-12T02:37:40.352 INFO:tasks.rados.rados.0.smithi107.stdout:1922: done (0 left) 2020-06-12T02:37:40.352 INFO:tasks.rados.rados.0.smithi107.stdout:1923: delete oid 14 current snap is 0 2020-06-12T02:37:40.355 INFO:tasks.rados.rados.0.smithi107.stdout:1923: done (0 left) 2020-06-12T02:37:40.355 INFO:tasks.rados.rados.0.smithi107.stdout:1924: delete oid 12 current snap is 0 2020-06-12T02:37:40.357 INFO:tasks.rados.rados.0.smithi107.stdout:1924: done (0 left) 2020-06-12T02:37:40.358 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:37:40.358 INFO:tasks.rados.rados.0.smithi107.stdout:1925: seq_num 1924 ranges {0=630784,630784=401408} 2020-06-12T02:37:40.362 INFO:tasks.rados.rados.0.smithi107.stdout:1925: writing smithi10729407-2 from 0 to 630784 tid 1 2020-06-12T02:37:40.365 INFO:tasks.rados.rados.0.smithi107.stdout:1925: writing smithi10729407-2 from 630784 to 1032192 tid 2 2020-06-12T02:37:40.366 INFO:tasks.rados.rados.0.smithi107.stdout:1926: delete oid 14 current snap is 0 2020-06-12T02:37:40.369 INFO:tasks.rados.rados.0.smithi107.stdout:1926: done (1 left) 2020-06-12T02:37:40.370 INFO:tasks.rados.rados.0.smithi107.stdout:1927: delete oid 11 current snap is 0 2020-06-12T02:37:40.372 INFO:tasks.rados.rados.0.smithi107.stdout:1925: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:40.374 INFO:tasks.rados.rados.0.smithi107.stdout:1925: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:40.374 INFO:tasks.rados.rados.0.smithi107.stdout:1927: done (1 left) 2020-06-12T02:37:40.374 INFO:tasks.rados.rados.0.smithi107.stdout:1928: delete oid 6 current snap is 0 2020-06-12T02:37:40.374 INFO:tasks.rados.rados.0.smithi107.stdout:1925: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:40.376 INFO:tasks.rados.rados.0.smithi107.stdout:1925: finishing write tid 5 to smithi10729407-2 2020-06-12T02:37:40.376 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 184 (ObjNum 1924 snap 0 seq_num 1924) dirty exists 2020-06-12T02:37:40.376 INFO:tasks.rados.rados.0.smithi107.stdout:1925: left oid 2 (ObjNum 1924 snap 0 seq_num 1924) 2020-06-12T02:37:40.376 INFO:tasks.rados.rados.0.smithi107.stdout:1925: done (1 left) 2020-06-12T02:37:40.377 INFO:tasks.rados.rados.0.smithi107.stdout:1928: done (0 left) 2020-06-12T02:37:40.377 INFO:tasks.rados.rados.0.smithi107.stdout:1929: delete oid 7 current snap is 0 2020-06-12T02:37:40.385 INFO:tasks.rados.rados.0.smithi107.stdout:1929: done (0 left) 2020-06-12T02:37:40.385 INFO:tasks.rados.rados.0.smithi107.stdout:1930: delete oid 10 current snap is 0 2020-06-12T02:37:40.386 INFO:tasks.rados.rados.0.smithi107.stdout:1930: done (0 left) 2020-06-12T02:37:40.387 INFO:tasks.rados.rados.0.smithi107.stdout:1931: delete oid 1 current snap is 0 2020-06-12T02:37:40.387 INFO:tasks.rados.rados.0.smithi107.stdout:1931: done (0 left) 2020-06-12T02:37:40.387 INFO:tasks.rados.rados.0.smithi107.stdout:1932: delete oid 6 current snap is 0 2020-06-12T02:37:40.387 INFO:tasks.rados.rados.0.smithi107.stdout:1932: done (0 left) 2020-06-12T02:37:40.387 INFO:tasks.rados.rados.0.smithi107.stdout:1933: delete oid 14 current snap is 0 2020-06-12T02:37:40.388 INFO:tasks.rados.rados.0.smithi107.stdout:1933: done (0 left) 2020-06-12T02:37:40.388 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:37:40.388 INFO:tasks.rados.rados.0.smithi107.stdout:1934: seq_num 1933 ranges {933888=778240,1712128=401408} 2020-06-12T02:37:40.396 INFO:tasks.rados.rados.0.smithi107.stdout:1934: writing smithi10729407-18 from 933888 to 1712128 tid 1 2020-06-12T02:37:40.399 INFO:tasks.rados.rados.0.smithi107.stdout:1934: writing smithi10729407-18 from 1712128 to 2113536 tid 2 2020-06-12T02:37:40.400 INFO:tasks.rados.rados.0.smithi107.stdout:1935: delete oid 14 current snap is 0 2020-06-12T02:37:40.402 INFO:tasks.rados.rados.0.smithi107.stdout:1935: done (1 left) 2020-06-12T02:37:40.403 INFO:tasks.rados.rados.0.smithi107.stdout:1936: delete oid 9 current snap is 0 2020-06-12T02:37:40.404 INFO:tasks.rados.rados.0.smithi107.stdout:1934: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:40.407 INFO:tasks.rados.rados.0.smithi107.stdout:1936: done (1 left) 2020-06-12T02:37:40.407 INFO:tasks.rados.rados.0.smithi107.stdout:1937: delete oid 3 current snap is 0 2020-06-12T02:37:40.408 INFO:tasks.rados.rados.0.smithi107.stdout:1934: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:40.409 INFO:tasks.rados.rados.0.smithi107.stdout:1934: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:40.410 INFO:tasks.rados.rados.0.smithi107.stdout:1937: done (1 left) 2020-06-12T02:37:40.410 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:37:40.410 INFO:tasks.rados.rados.0.smithi107.stdout:1938: seq_num 1937 ranges {0=450560,450560=573440,1024000=229376} 2020-06-12T02:37:40.412 INFO:tasks.rados.rados.0.smithi107.stdout:1938: writing smithi10729407-3 from 0 to 450560 tid 1 2020-06-12T02:37:40.415 INFO:tasks.rados.rados.0.smithi107.stdout:1938: writing smithi10729407-3 from 450560 to 1024000 tid 2 2020-06-12T02:37:40.417 INFO:tasks.rados.rados.0.smithi107.stdout:1938: writing smithi10729407-3 from 1024000 to 1253376 tid 3 2020-06-12T02:37:40.417 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:37:40.417 INFO:tasks.rados.rados.0.smithi107.stdout:1939: seq_num 1938 ranges {0=466944,466944=458752,925696=344064} 2020-06-12T02:37:40.421 INFO:tasks.rados.rados.0.smithi107.stdout:1939: writing smithi10729407-1 from 0 to 466944 tid 1 2020-06-12T02:37:40.423 INFO:tasks.rados.rados.0.smithi107.stdout:1939: writing smithi10729407-1 from 466944 to 925696 tid 2 2020-06-12T02:37:40.425 INFO:tasks.rados.rados.0.smithi107.stdout:1939: writing smithi10729407-1 from 925696 to 1269760 tid 3 2020-06-12T02:37:40.425 INFO:tasks.rados.rados.0.smithi107.stdout:1940: delete oid 16 current snap is 0 2020-06-12T02:37:40.426 INFO:tasks.rados.rados.0.smithi107.stdout:1934: finishing write tid 5 to smithi10729407-18 2020-06-12T02:37:40.426 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 331 (ObjNum 1933 snap 0 seq_num 1933) dirty exists 2020-06-12T02:37:40.426 INFO:tasks.rados.rados.0.smithi107.stdout:1934: left oid 18 (ObjNum 1933 snap 0 seq_num 1933) 2020-06-12T02:37:40.426 INFO:tasks.rados.rados.0.smithi107.stdout:1938: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:40.426 INFO:tasks.rados.rados.0.smithi107.stdout:1938: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:40.427 INFO:tasks.rados.rados.0.smithi107.stdout:1938: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:40.427 INFO:tasks.rados.rados.0.smithi107.stdout:1938: finishing write tid 5 to smithi10729407-3 2020-06-12T02:37:40.429 INFO:tasks.rados.rados.0.smithi107.stdout:1938: finishing write tid 6 to smithi10729407-3 2020-06-12T02:37:40.429 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 333 (ObjNum 1937 snap 0 seq_num 1937) dirty exists 2020-06-12T02:37:40.429 INFO:tasks.rados.rados.0.smithi107.stdout:1938: left oid 3 (ObjNum 1937 snap 0 seq_num 1937) 2020-06-12T02:37:40.430 INFO:tasks.rados.rados.0.smithi107.stdout:1934: done (3 left) 2020-06-12T02:37:40.430 INFO:tasks.rados.rados.0.smithi107.stdout:1938: done (2 left) 2020-06-12T02:37:40.430 INFO:tasks.rados.rados.0.smithi107.stdout:1940: done (1 left) 2020-06-12T02:37:40.430 INFO:tasks.rados.rados.0.smithi107.stdout:1941: delete oid 3 current snap is 0 2020-06-12T02:37:40.432 INFO:tasks.rados.rados.0.smithi107.stdout:1939: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:40.432 INFO:tasks.rados.rados.0.smithi107.stdout:1941: done (1 left) 2020-06-12T02:37:40.432 INFO:tasks.rados.rados.0.smithi107.stdout:1942: delete oid 10 current snap is 0 2020-06-12T02:37:40.432 INFO:tasks.rados.rados.0.smithi107.stdout:1939: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:40.432 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:37:40.429+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:37:40.434 INFO:tasks.rados.rados.0.smithi107.stdout:1942: done (1 left) 2020-06-12T02:37:40.434 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:37:40.435 INFO:tasks.rados.rados.0.smithi107.stdout:1943: seq_num 1942 ranges {0=532480,532480=450560,983040=352256} 2020-06-12T02:37:40.437 INFO:tasks.rados.rados.0.smithi107.stdout:1943: writing smithi10729407-17 from 0 to 532480 tid 1 2020-06-12T02:37:40.440 INFO:tasks.rados.rados.0.smithi107.stdout:1943: writing smithi10729407-17 from 532480 to 983040 tid 2 2020-06-12T02:37:40.442 INFO:tasks.rados.rados.0.smithi107.stdout:1943: writing smithi10729407-17 from 983040 to 1335296 tid 3 2020-06-12T02:37:40.443 INFO:tasks.rados.rados.0.smithi107.stdout:1944: delete oid 12 current snap is 0 2020-06-12T02:37:40.443 INFO:tasks.rados.rados.0.smithi107.stdout:1939: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:40.443 INFO:tasks.rados.rados.0.smithi107.stdout:1939: finishing write tid 5 to smithi10729407-1 2020-06-12T02:37:40.443 INFO:tasks.rados.rados.0.smithi107.stdout:1939: finishing write tid 6 to smithi10729407-1 2020-06-12T02:37:40.443 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 392 (ObjNum 1938 snap 0 seq_num 1938) dirty exists 2020-06-12T02:37:40.444 INFO:tasks.rados.rados.0.smithi107.stdout:1939: left oid 1 (ObjNum 1938 snap 0 seq_num 1938) 2020-06-12T02:37:40.445 INFO:tasks.rados.rados.0.smithi107.stdout:1939: done (2 left) 2020-06-12T02:37:40.445 INFO:tasks.rados.rados.0.smithi107.stdout:1944: done (1 left) 2020-06-12T02:37:40.445 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:37:40.445 INFO:tasks.rados.rados.0.smithi107.stdout:1945: seq_num 1944 ranges {0=565248,565248=614400,1179648=188416} 2020-06-12T02:37:40.448 INFO:tasks.rados.rados.0.smithi107.stdout:1945: writing smithi10729407-7 from 0 to 565248 tid 1 2020-06-12T02:37:40.449 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:40.449 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:40.453 INFO:tasks.rados.rados.0.smithi107.stdout:1945: writing smithi10729407-7 from 565248 to 1179648 tid 2 2020-06-12T02:37:40.454 INFO:tasks.rados.rados.0.smithi107.stdout:1945: writing smithi10729407-7 from 1179648 to 1368064 tid 3 2020-06-12T02:37:40.454 INFO:tasks.rados.rados.0.smithi107.stdout:1946: delete oid 16 current snap is 0 2020-06-12T02:37:40.454 INFO:tasks.rados.rados.0.smithi107.stdout:1943: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:40.455 INFO:tasks.rados.rados.0.smithi107.stdout:1943: finishing write tid 2 to smithi10729407-17 2020-06-12T02:37:40.455 INFO:tasks.rados.rados.0.smithi107.stdout:1943: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:40.455 INFO:tasks.rados.rados.0.smithi107.stdout:1943: finishing write tid 5 to smithi10729407-17 2020-06-12T02:37:40.455 INFO:tasks.rados.rados.0.smithi107.stdout:1943: finishing write tid 6 to smithi10729407-17 2020-06-12T02:37:40.455 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 375 (ObjNum 1942 snap 0 seq_num 1942) dirty exists 2020-06-12T02:37:40.455 INFO:tasks.rados.rados.0.smithi107.stdout:1943: left oid 17 (ObjNum 1942 snap 0 seq_num 1942) 2020-06-12T02:37:40.459 INFO:tasks.rados.rados.0.smithi107.stdout:1945: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:40.460 INFO:tasks.rados.rados.0.smithi107.stdout:1945: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:40.460 INFO:tasks.rados.rados.0.smithi107.stdout:1943: done (2 left) 2020-06-12T02:37:40.461 INFO:tasks.rados.rados.0.smithi107.stdout:1946: done (1 left) 2020-06-12T02:37:40.461 INFO:tasks.rados.rados.0.smithi107.stdout:1947: delete oid 1 current snap is 0 2020-06-12T02:37:40.462 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-06-12T02:37:40.463 INFO:tasks.rados.rados.0.smithi107.stdout:1945: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:40.463 INFO:tasks.rados.rados.0.smithi107.stdout:1945: finishing write tid 5 to smithi10729407-7 2020-06-12T02:37:40.463 INFO:tasks.rados.rados.0.smithi107.stdout:1947: done (1 left) 2020-06-12T02:37:40.464 INFO:tasks.rados.rados.0.smithi107.stdout:1948: delete oid 2 current snap is 0 2020-06-12T02:37:40.465 INFO:tasks.rados.rados.0.smithi107.stdout:1945: finishing write tid 6 to smithi10729407-7 2020-06-12T02:37:40.465 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 377 (ObjNum 1944 snap 0 seq_num 1944) dirty exists 2020-06-12T02:37:40.465 INFO:tasks.rados.rados.0.smithi107.stdout:1945: left oid 7 (ObjNum 1944 snap 0 seq_num 1944) 2020-06-12T02:37:40.466 INFO:tasks.rados.rados.0.smithi107.stdout:1945: done (1 left) 2020-06-12T02:37:40.466 INFO:tasks.rados.rados.0.smithi107.stdout:1948: done (0 left) 2020-06-12T02:37:40.466 INFO:tasks.rados.rados.0.smithi107.stdout:1949: delete oid 11 current snap is 0 2020-06-12T02:37:40.469 INFO:tasks.rados.rados.0.smithi107.stdout:1949: done (0 left) 2020-06-12T02:37:40.470 INFO:tasks.rados.rados.0.smithi107.stdout:1950: delete oid 4 current snap is 0 2020-06-12T02:37:40.471 INFO:tasks.rados.rados.0.smithi107.stdout:1950: done (0 left) 2020-06-12T02:37:40.471 INFO:tasks.rados.rados.0.smithi107.stdout:1951: delete oid 6 current snap is 0 2020-06-12T02:37:40.473 INFO:tasks.rados.rados.0.smithi107.stdout:1951: done (0 left) 2020-06-12T02:37:40.473 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:37:40.474 INFO:tasks.rados.rados.0.smithi107.stdout:1952: seq_num 1951 ranges {0=679936,679936=655360,1335296=147456} 2020-06-12T02:37:40.477 INFO:tasks.rados.rados.0.smithi107.stdout:1952: writing smithi10729407-9 from 0 to 679936 tid 1 2020-06-12T02:37:40.481 INFO:tasks.rados.rados.0.smithi107.stdout:1952: writing smithi10729407-9 from 679936 to 1335296 tid 2 2020-06-12T02:37:40.483 INFO:tasks.rados.rados.0.smithi107.stdout:1952: writing smithi10729407-9 from 1335296 to 1482752 tid 3 2020-06-12T02:37:40.483 INFO:tasks.rados.rados.0.smithi107.stdout:1953: delete oid 17 current snap is 0 2020-06-12T02:37:40.489 INFO:tasks.rados.rados.0.smithi107.stdout:1953: done (1 left) 2020-06-12T02:37:40.489 INFO:tasks.rados.rados.0.smithi107.stdout:1954: delete oid 6 current snap is 0 2020-06-12T02:37:40.490 INFO:tasks.rados.rados.0.smithi107.stdout:1952: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:40.491 INFO:tasks.rados.rados.0.smithi107.stdout:1952: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:40.492 INFO:tasks.rados.rados.0.smithi107.stdout:1952: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:40.492 INFO:tasks.rados.rados.0.smithi107.stdout:1954: done (1 left) 2020-06-12T02:37:40.492 INFO:tasks.rados.rados.0.smithi107.stdout:1955: delete oid 19 current snap is 0 2020-06-12T02:37:40.493 INFO:tasks.rados.rados.0.smithi107.stdout:1952: finishing write tid 5 to smithi10729407-9 2020-06-12T02:37:40.494 INFO:tasks.rados.rados.0.smithi107.stdout:1955: done (1 left) 2020-06-12T02:37:40.494 INFO:tasks.rados.rados.0.smithi107.stdout:1956: delete oid 8 current snap is 0 2020-06-12T02:37:40.495 INFO:tasks.rados.rados.0.smithi107.stdout:1952: finishing write tid 6 to smithi10729407-9 2020-06-12T02:37:40.495 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 335 (ObjNum 1951 snap 0 seq_num 1951) dirty exists 2020-06-12T02:37:40.495 INFO:tasks.rados.rados.0.smithi107.stdout:1952: left oid 9 (ObjNum 1951 snap 0 seq_num 1951) 2020-06-12T02:37:40.496 INFO:tasks.rados.rados.0.smithi107.stdout:1952: done (1 left) 2020-06-12T02:37:40.496 INFO:tasks.rados.rados.0.smithi107.stdout:1956: done (0 left) 2020-06-12T02:37:40.496 INFO:tasks.rados.rados.0.smithi107.stdout:1957: delete oid 6 current snap is 0 2020-06-12T02:37:40.498 INFO:tasks.rados.rados.0.smithi107.stdout:1957: done (0 left) 2020-06-12T02:37:40.498 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:37:40.498 INFO:tasks.rados.rados.0.smithi107.stdout:1958: seq_num 1957 ranges {0=786432,786432=409600,1196032=393216} 2020-06-12T02:37:40.502 INFO:tasks.rados.rados.0.smithi107.stdout:1958: writing smithi10729407-17 from 0 to 786432 tid 1 2020-06-12T02:37:40.504 INFO:tasks.rados.rados.0.smithi107.stdout:1958: writing smithi10729407-17 from 786432 to 1196032 tid 2 2020-06-12T02:37:40.507 INFO:tasks.rados.rados.0.smithi107.stdout:1958: writing smithi10729407-17 from 1196032 to 1589248 tid 3 2020-06-12T02:37:40.507 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:37:40.507 INFO:tasks.rados.rados.0.smithi107.stdout:1959: seq_num 1958 ranges {0=802816,802816=696320,1499136=106496} 2020-06-12T02:37:40.512 INFO:tasks.rados.rados.0.smithi107.stdout:1959: writing smithi10729407-2 from 0 to 802816 tid 1 2020-06-12T02:37:40.516 INFO:tasks.rados.rados.0.smithi107.stdout:1959: writing smithi10729407-2 from 802816 to 1499136 tid 2 2020-06-12T02:37:40.517 INFO:tasks.rados.rados.0.smithi107.stdout:1959: writing smithi10729407-2 from 1499136 to 1605632 tid 3 2020-06-12T02:37:40.517 INFO:tasks.rados.rados.0.smithi107.stdout:1960: delete oid 19 current snap is 0 2020-06-12T02:37:40.517 INFO:tasks.rados.rados.0.smithi107.stdout:1958: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:40.518 INFO:tasks.rados.rados.0.smithi107.stdout:1958: finishing write tid 2 to smithi10729407-17 2020-06-12T02:37:40.518 INFO:tasks.rados.rados.0.smithi107.stdout:1958: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:40.519 INFO:tasks.rados.rados.0.smithi107.stdout:1958: finishing write tid 5 to smithi10729407-17 2020-06-12T02:37:40.519 INFO:tasks.rados.rados.0.smithi107.stdout:1960: done (2 left) 2020-06-12T02:37:40.519 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:37:40.520 INFO:tasks.rados.rados.0.smithi107.stdout:1961: seq_num 1960 ranges {0=434176,434176=466944,901120=524288,1425408=212992} 2020-06-12T02:37:40.522 INFO:tasks.rados.rados.0.smithi107.stdout:1961: writing smithi10729407-11 from 0 to 434176 tid 1 2020-06-12T02:37:40.525 INFO:tasks.rados.rados.0.smithi107.stdout:1961: writing smithi10729407-11 from 434176 to 901120 tid 2 2020-06-12T02:37:40.528 INFO:tasks.rados.rados.0.smithi107.stdout:1961: writing smithi10729407-11 from 901120 to 1425408 tid 3 2020-06-12T02:37:40.530 INFO:tasks.rados.rados.0.smithi107.stdout:1961: writing smithi10729407-11 from 1425408 to 1638400 tid 4 2020-06-12T02:37:40.530 INFO:tasks.rados.rados.0.smithi107.stdout:1962: delete oid 9 current snap is 0 2020-06-12T02:37:40.530 INFO:tasks.rados.rados.0.smithi107.stdout:1959: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:40.531 INFO:tasks.rados.rados.0.smithi107.stdout:1958: finishing write tid 6 to smithi10729407-17 2020-06-12T02:37:40.531 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 381 (ObjNum 1957 snap 0 seq_num 1957) dirty exists 2020-06-12T02:37:40.531 INFO:tasks.rados.rados.0.smithi107.stdout:1958: left oid 17 (ObjNum 1957 snap 0 seq_num 1957) 2020-06-12T02:37:40.531 INFO:tasks.rados.rados.0.smithi107.stdout:1959: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:40.531 INFO:tasks.rados.rados.0.smithi107.stdout:1959: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:40.532 INFO:tasks.rados.rados.0.smithi107.stdout:1959: finishing write tid 5 to smithi10729407-2 2020-06-12T02:37:40.532 INFO:tasks.rados.rados.0.smithi107.stdout:1959: finishing write tid 6 to smithi10729407-2 2020-06-12T02:37:40.532 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 189 (ObjNum 1958 snap 0 seq_num 1958) dirty exists 2020-06-12T02:37:40.532 INFO:tasks.rados.rados.0.smithi107.stdout:1959: left oid 2 (ObjNum 1958 snap 0 seq_num 1958) 2020-06-12T02:37:40.532 INFO:tasks.rados.rados.0.smithi107.stdout:1961: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:40.533 INFO:tasks.rados.rados.0.smithi107.stdout:1958: done (3 left) 2020-06-12T02:37:40.533 INFO:tasks.rados.rados.0.smithi107.stdout:1959: done (2 left) 2020-06-12T02:37:40.533 INFO:tasks.rados.rados.0.smithi107.stdout:1962: done (1 left) 2020-06-12T02:37:40.534 INFO:tasks.rados.rados.0.smithi107.stdout:1963: delete oid 14 current snap is 0 2020-06-12T02:37:40.534 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:40.530+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:40.537 INFO:tasks.rados.rados.0.smithi107.stdout:1961: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:40.538 INFO:tasks.rados.rados.0.smithi107.stdout:1961: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:40.538 INFO:tasks.rados.rados.0.smithi107.stdout:1961: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:40.539 INFO:tasks.rados.rados.0.smithi107.stdout:1961: finishing write tid 6 to smithi10729407-11 2020-06-12T02:37:40.539 INFO:tasks.rados.rados.0.smithi107.stdout:1963: done (1 left) 2020-06-12T02:37:40.539 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:37:40.540 INFO:tasks.rados.rados.0.smithi107.stdout:1964: seq_num 1963 ranges {2113536=483328,2596864=565248,3162112=638976} 2020-06-12T02:37:40.553 INFO:tasks.rados.rados.0.smithi107.stdout:1964: writing smithi10729407-18 from 2113536 to 2596864 tid 1 2020-06-12T02:37:40.557 INFO:tasks.rados.rados.0.smithi107.stdout:1964: writing smithi10729407-18 from 2596864 to 3162112 tid 2 2020-06-12T02:37:40.567 INFO:tasks.rados.rados.0.smithi107.stdout:1964: writing smithi10729407-18 from 3162112 to 3801088 tid 3 2020-06-12T02:37:40.567 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:37:40.567 INFO:tasks.rados.rados.0.smithi107.stdout:1961: finishing write tid 7 to smithi10729407-11 2020-06-12T02:37:40.568 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 354 (ObjNum 1960 snap 0 seq_num 1960) dirty exists 2020-06-12T02:37:40.568 INFO:tasks.rados.rados.0.smithi107.stdout:1961: left oid 11 (ObjNum 1960 snap 0 seq_num 1960) 2020-06-12T02:37:40.568 INFO:tasks.rados.rados.0.smithi107.stdout:1964: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:40.568 INFO:tasks.rados.rados.0.smithi107.stdout:1965: seq_num 1964 ranges {0=499712,499712=450560,950272=548864,1499136=204800} 2020-06-12T02:37:40.569 INFO:tasks.rados.rados.0.smithi107.stdout:1965: writing smithi10729407-14 from 0 to 499712 tid 1 2020-06-12T02:37:40.569 INFO:tasks.rados.rados.0.smithi107.stdout:1965: writing smithi10729407-14 from 499712 to 950272 tid 2 2020-06-12T02:37:40.571 INFO:tasks.rados.rados.0.smithi107.stdout:1965: writing smithi10729407-14 from 950272 to 1499136 tid 3 2020-06-12T02:37:40.572 INFO:tasks.rados.rados.0.smithi107.stdout:1965: writing smithi10729407-14 from 1499136 to 1703936 tid 4 2020-06-12T02:37:40.573 INFO:tasks.rados.rados.0.smithi107.stdout:1961: done (2 left) 2020-06-12T02:37:40.573 INFO:tasks.rados.rados.0.smithi107.stdout:1966: delete oid 2 current snap is 0 2020-06-12T02:37:40.573 INFO:tasks.rados.rados.0.smithi107.stdout:1964: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:40.573 INFO:tasks.rados.rados.0.smithi107.stdout:1964: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:40.573 INFO:tasks.rados.rados.0.smithi107.stdout:1965: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:40.574 INFO:tasks.rados.rados.0.smithi107.stdout:1964: finishing write tid 5 to smithi10729407-18 2020-06-12T02:37:40.574 INFO:tasks.rados.rados.0.smithi107.stdout:1964: finishing write tid 6 to smithi10729407-18 2020-06-12T02:37:40.575 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 340 (ObjNum 1963 snap 0 seq_num 1963) dirty exists 2020-06-12T02:37:40.575 INFO:tasks.rados.rados.0.smithi107.stdout:1964: left oid 18 (ObjNum 1963 snap 0 seq_num 1963) 2020-06-12T02:37:40.577 INFO:tasks.rados.rados.0.smithi107.stdout:1964: done (2 left) 2020-06-12T02:37:40.577 INFO:tasks.rados.rados.0.smithi107.stdout:1966: done (1 left) 2020-06-12T02:37:40.577 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:37:40.578 INFO:tasks.rados.rados.0.smithi107.stdout:1967: seq_num 1966 ranges {0=532480,532480=622592,1155072=516096,1671168=65536} 2020-06-12T02:37:40.581 INFO:tasks.rados.rados.0.smithi107.stdout:1967: writing smithi10729407-1 from 0 to 532480 tid 1 2020-06-12T02:37:40.590 INFO:tasks.rados.rados.0.smithi107.stdout:1967: writing smithi10729407-1 from 532480 to 1155072 tid 2 2020-06-12T02:37:40.590 INFO:tasks.rados.rados.0.smithi107.stdout:1967: writing smithi10729407-1 from 1155072 to 1671168 tid 3 2020-06-12T02:37:40.590 INFO:tasks.rados.rados.0.smithi107.stdout:1967: writing smithi10729407-1 from 1671168 to 1736704 tid 4 2020-06-12T02:37:40.591 INFO:tasks.rados.rados.0.smithi107.stdout:1968: delete oid 12 current snap is 0 2020-06-12T02:37:40.591 INFO:tasks.rados.rados.0.smithi107.stdout:1965: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:40.591 INFO:tasks.rados.rados.0.smithi107.stdout:1965: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:40.591 INFO:tasks.rados.rados.0.smithi107.stdout:1965: finishing write tid 4 to smithi10729407-14 2020-06-12T02:37:40.591 INFO:tasks.rados.rados.0.smithi107.stdout:1965: finishing write tid 6 to smithi10729407-14 2020-06-12T02:37:40.592 INFO:tasks.rados.rados.0.smithi107.stdout:1965: finishing write tid 7 to smithi10729407-14 2020-06-12T02:37:40.592 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 360 (ObjNum 1964 snap 0 seq_num 1964) dirty exists 2020-06-12T02:37:40.592 INFO:tasks.rados.rados.0.smithi107.stdout:1965: left oid 14 (ObjNum 1964 snap 0 seq_num 1964) 2020-06-12T02:37:40.592 INFO:tasks.rados.rados.0.smithi107.stdout:1967: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:40.592 INFO:tasks.rados.rados.0.smithi107.stdout:1965: done (2 left) 2020-06-12T02:37:40.592 INFO:tasks.rados.rados.0.smithi107.stdout:1968: done (1 left) 2020-06-12T02:37:40.593 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:40.593 INFO:tasks.rados.rados.0.smithi107.stdout:1969: seq_num 1968 ranges {0=565248,565248=786432,1351680=417792} 2020-06-12T02:37:40.594 INFO:tasks.rados.rados.0.smithi107.stdout:1969: writing smithi10729407-2 from 0 to 565248 tid 1 2020-06-12T02:37:40.598 INFO:tasks.rados.rados.0.smithi107.stdout:1969: writing smithi10729407-2 from 565248 to 1351680 tid 2 2020-06-12T02:37:40.601 INFO:tasks.rados.rados.0.smithi107.stdout:1969: writing smithi10729407-2 from 1351680 to 1769472 tid 3 2020-06-12T02:37:40.601 INFO:tasks.rados.rados.0.smithi107.stdout:1970: delete oid 7 current snap is 0 2020-06-12T02:37:40.601 INFO:tasks.rados.rados.0.smithi107.stdout:1967: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:40.602 INFO:tasks.rados.rados.0.smithi107.stdout:1967: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:40.602 INFO:tasks.rados.rados.0.smithi107.stdout:1967: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:40.602 INFO:tasks.rados.rados.0.smithi107.stdout:1967: finishing write tid 6 to smithi10729407-1 2020-06-12T02:37:40.602 INFO:tasks.rados.rados.0.smithi107.stdout:1969: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:40.602 INFO:tasks.rados.rados.0.smithi107.stdout:1967: finishing write tid 7 to smithi10729407-1 2020-06-12T02:37:40.603 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 401 (ObjNum 1966 snap 0 seq_num 1966) dirty exists 2020-06-12T02:37:40.603 INFO:tasks.rados.rados.0.smithi107.stdout:1967: left oid 1 (ObjNum 1966 snap 0 seq_num 1966) 2020-06-12T02:37:40.603 INFO:tasks.rados.rados.0.smithi107.stdout:1967: done (2 left) 2020-06-12T02:37:40.604 INFO:tasks.rados.rados.0.smithi107.stdout:1970: done (1 left) 2020-06-12T02:37:40.604 INFO:tasks.rados.rados.0.smithi107.stdout:1971: delete oid 7 current snap is 0 2020-06-12T02:37:40.605 INFO:tasks.rados.rados.0.smithi107.stdout:1971: done (1 left) 2020-06-12T02:37:40.606 INFO:tasks.rados.rados.0.smithi107.stdout:1972: delete oid 15 current snap is 0 2020-06-12T02:37:40.608 INFO:tasks.rados.rados.0.smithi107.stdout:1972: done (1 left) 2020-06-12T02:37:40.609 INFO:tasks.rados.rados.0.smithi107.stdout:1973: delete oid 15 current snap is 0 2020-06-12T02:37:40.610 INFO:tasks.rados.rados.0.smithi107.stdout:1969: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:40.611 INFO:tasks.rados.rados.0.smithi107.stdout:1969: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:40.611 INFO:tasks.rados.rados.0.smithi107.stdout:1969: finishing write tid 5 to smithi10729407-2 2020-06-12T02:37:40.612 INFO:tasks.rados.rados.0.smithi107.stdout:1973: done (1 left) 2020-06-12T02:37:40.612 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:37:40.612 INFO:tasks.rados.rados.0.smithi107.stdout:1974: seq_num 1973 ranges {0=647168,647168=663552,1310720=540672} 2020-06-12T02:37:40.614 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:40.614 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:40.616 INFO:tasks.rados.rados.0.smithi107.stdout:1974: writing smithi10729407-3 from 0 to 647168 tid 1 2020-06-12T02:37:40.620 INFO:tasks.rados.rados.0.smithi107.stdout:1974: writing smithi10729407-3 from 647168 to 1310720 tid 2 2020-06-12T02:37:40.624 INFO:tasks.rados.rados.0.smithi107.stdout:1974: writing smithi10729407-3 from 1310720 to 1851392 tid 3 2020-06-12T02:37:40.624 INFO:tasks.rados.rados.0.smithi107.stdout:1975: delete oid 6 current snap is 0 2020-06-12T02:37:40.625 INFO:tasks.rados.rados.0.smithi107.stdout:1969: finishing write tid 6 to smithi10729407-2 2020-06-12T02:37:40.625 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 194 (ObjNum 1968 snap 0 seq_num 1968) dirty exists 2020-06-12T02:37:40.625 INFO:tasks.rados.rados.0.smithi107.stdout:1969: left oid 2 (ObjNum 1968 snap 0 seq_num 1968) 2020-06-12T02:37:40.626 INFO:tasks.rados.rados.0.smithi107.stdout:1974: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:40.626 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-06-12T02:37:40.627 INFO:tasks.rados.rados.0.smithi107.stdout:1969: done (2 left) 2020-06-12T02:37:40.627 INFO:tasks.rados.rados.0.smithi107.stdout:1975: done (1 left) 2020-06-12T02:37:40.627 INFO:tasks.rados.rados.0.smithi107.stdout:1976: delete oid 17 current snap is 0 2020-06-12T02:37:40.628 INFO:tasks.rados.rados.0.smithi107.stdout:1976: done (1 left) 2020-06-12T02:37:40.628 INFO:tasks.rados.rados.0.smithi107.stdout:1977: delete oid 1 current snap is 0 2020-06-12T02:37:40.629 INFO:tasks.rados.rados.0.smithi107.stdout:1974: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:40.630 INFO:tasks.rados.rados.0.smithi107.stdout:1974: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:40.631 INFO:tasks.rados.rados.0.smithi107.stdout:1974: finishing write tid 5 to smithi10729407-3 2020-06-12T02:37:40.632 INFO:tasks.rados.rados.0.smithi107.stdout:1977: done (1 left) 2020-06-12T02:37:40.632 INFO:tasks.rados.rados.0.smithi107.stdout:1978: delete oid 20 current snap is 0 2020-06-12T02:37:40.634 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:40.631+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:40.635 INFO:tasks.rados.rados.0.smithi107.stdout:1974: finishing write tid 6 to smithi10729407-3 2020-06-12T02:37:40.635 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 340 (ObjNum 1973 snap 0 seq_num 1973) dirty exists 2020-06-12T02:37:40.636 INFO:tasks.rados.rados.0.smithi107.stdout:1974: left oid 3 (ObjNum 1973 snap 0 seq_num 1973) 2020-06-12T02:37:40.637 INFO:tasks.rados.rados.0.smithi107.stdout:1974: done (1 left) 2020-06-12T02:37:40.637 INFO:tasks.rados.rados.0.smithi107.stdout:1978: done (0 left) 2020-06-12T02:37:40.637 INFO:tasks.rados.rados.0.smithi107.stdout:1979: delete oid 3 current snap is 0 2020-06-12T02:37:40.639 INFO:tasks.rados.rados.0.smithi107.stdout:1979: done (0 left) 2020-06-12T02:37:40.639 INFO:tasks.rados.rados.0.smithi107.stdout:1980: delete oid 3 current snap is 0 2020-06-12T02:37:40.643 INFO:tasks.rados.rados.0.smithi107.stdout:1980: done (0 left) 2020-06-12T02:37:40.644 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:37:40.644 INFO:tasks.rados.rados.0.smithi107.stdout:1981: seq_num 1980 ranges {0=770048,770048=704512,1474560=475136,1949696=24576} 2020-06-12T02:37:40.648 INFO:tasks.rados.rados.0.smithi107.stdout:1981: writing smithi10729407-15 from 0 to 770048 tid 1 2020-06-12T02:37:40.652 INFO:tasks.rados.rados.0.smithi107.stdout:1981: writing smithi10729407-15 from 770048 to 1474560 tid 2 2020-06-12T02:37:40.655 INFO:tasks.rados.rados.0.smithi107.stdout:1981: writing smithi10729407-15 from 1474560 to 1949696 tid 3 2020-06-12T02:37:40.656 INFO:tasks.rados.rados.0.smithi107.stdout:1981: writing smithi10729407-15 from 1949696 to 1974272 tid 4 2020-06-12T02:37:40.656 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:37:40.656 INFO:tasks.rados.rados.0.smithi107.stdout:1982: seq_num 1981 ranges {0=786432,786432=581632,1368064=622592} 2020-06-12T02:37:40.660 INFO:tasks.rados.rados.0.smithi107.stdout:1982: writing smithi10729407-10 from 0 to 786432 tid 1 2020-06-12T02:37:40.663 INFO:tasks.rados.rados.0.smithi107.stdout:1982: writing smithi10729407-10 from 786432 to 1368064 tid 2 2020-06-12T02:37:40.667 INFO:tasks.rados.rados.0.smithi107.stdout:1982: writing smithi10729407-10 from 1368064 to 1990656 tid 3 2020-06-12T02:37:40.668 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:37:40.668 INFO:tasks.rados.rados.0.smithi107.stdout:1983: seq_num 1982 ranges {0=802816,802816=466944,1269760=434176,1703936=303104} 2020-06-12T02:37:40.673 INFO:tasks.rados.rados.0.smithi107.stdout:1983: writing smithi10729407-7 from 0 to 802816 tid 1 2020-06-12T02:37:40.675 INFO:tasks.rados.rados.0.smithi107.stdout:1983: writing smithi10729407-7 from 802816 to 1269760 tid 2 2020-06-12T02:37:40.678 INFO:tasks.rados.rados.0.smithi107.stdout:1983: writing smithi10729407-7 from 1269760 to 1703936 tid 3 2020-06-12T02:37:40.682 INFO:tasks.rados.rados.0.smithi107.stdout:1983: writing smithi10729407-7 from 1703936 to 2007040 tid 4 2020-06-12T02:37:40.682 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:37:40.683 INFO:tasks.rados.rados.0.smithi107.stdout:1984: seq_num 1983 ranges {1769472=417792,2187264=753664,2940928=647168,3588096=204800} 2020-06-12T02:37:40.692 INFO:tasks.rados.rados.0.smithi107.stdout:1984: writing smithi10729407-2 from 1769472 to 2187264 tid 1 2020-06-12T02:37:40.697 INFO:tasks.rados.rados.0.smithi107.stdout:1984: writing smithi10729407-2 from 2187264 to 2940928 tid 2 2020-06-12T02:37:40.701 INFO:tasks.rados.rados.0.smithi107.stdout:1984: writing smithi10729407-2 from 2940928 to 3588096 tid 3 2020-06-12T02:37:40.702 INFO:tasks.rados.rados.0.smithi107.stdout:1984: writing smithi10729407-2 from 3588096 to 3792896 tid 4 2020-06-12T02:37:40.703 INFO:tasks.rados.rados.0.smithi107.stdout:1985: delete oid 1 current snap is 0 2020-06-12T02:37:40.703 INFO:tasks.rados.rados.0.smithi107.stdout:1981: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:40.703 INFO:tasks.rados.rados.0.smithi107.stdout:1981: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:40.703 INFO:tasks.rados.rados.0.smithi107.stdout:1981: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:40.703 INFO:tasks.rados.rados.0.smithi107.stdout:1981: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:40.704 INFO:tasks.rados.rados.0.smithi107.stdout:1981: finishing write tid 6 to smithi10729407-15 2020-06-12T02:37:40.704 INFO:tasks.rados.rados.0.smithi107.stdout:1981: finishing write tid 7 to smithi10729407-15 2020-06-12T02:37:40.704 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 342 (ObjNum 1980 snap 0 seq_num 1980) dirty exists 2020-06-12T02:37:40.704 INFO:tasks.rados.rados.0.smithi107.stdout:1981: left oid 15 (ObjNum 1980 snap 0 seq_num 1980) 2020-06-12T02:37:40.704 INFO:tasks.rados.rados.0.smithi107.stdout:1982: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:40.705 INFO:tasks.rados.rados.0.smithi107.stdout:1982: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:40.705 INFO:tasks.rados.rados.0.smithi107.stdout:1982: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:40.705 INFO:tasks.rados.rados.0.smithi107.stdout:1982: finishing write tid 5 to smithi10729407-10 2020-06-12T02:37:40.705 INFO:tasks.rados.rados.0.smithi107.stdout:1982: finishing write tid 6 to smithi10729407-10 2020-06-12T02:37:40.705 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 186 (ObjNum 1981 snap 0 seq_num 1981) dirty exists 2020-06-12T02:37:40.705 INFO:tasks.rados.rados.0.smithi107.stdout:1982: left oid 10 (ObjNum 1981 snap 0 seq_num 1981) 2020-06-12T02:37:40.706 INFO:tasks.rados.rados.0.smithi107.stdout:1983: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:40.706 INFO:tasks.rados.rados.0.smithi107.stdout:1983: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:40.706 INFO:tasks.rados.rados.0.smithi107.stdout:1983: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:40.706 INFO:tasks.rados.rados.0.smithi107.stdout:1983: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:40.706 INFO:tasks.rados.rados.0.smithi107.stdout:1983: finishing write tid 6 to smithi10729407-7 2020-06-12T02:37:40.707 INFO:tasks.rados.rados.0.smithi107.stdout:1983: finishing write tid 7 to smithi10729407-7 2020-06-12T02:37:40.707 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 384 (ObjNum 1982 snap 0 seq_num 1982) dirty exists 2020-06-12T02:37:40.707 INFO:tasks.rados.rados.0.smithi107.stdout:1983: left oid 7 (ObjNum 1982 snap 0 seq_num 1982) 2020-06-12T02:37:40.707 INFO:tasks.rados.rados.0.smithi107.stdout:1984: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:40.708 INFO:tasks.rados.rados.0.smithi107.stdout:1981: done (4 left) 2020-06-12T02:37:40.708 INFO:tasks.rados.rados.0.smithi107.stdout:1982: done (3 left) 2020-06-12T02:37:40.708 INFO:tasks.rados.rados.0.smithi107.stdout:1983: done (2 left) 2020-06-12T02:37:40.709 INFO:tasks.rados.rados.0.smithi107.stdout:1985: done (1 left) 2020-06-12T02:37:40.709 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:37:40.709 INFO:tasks.rados.rados.0.smithi107.stdout:1986: seq_num 1985 ranges {0=450560,450560=573440,1024000=614400,1638400=417792} 2020-06-12T02:37:40.709 INFO:tasks.rados.rados.0.smithi107.stdout:1986: writing smithi10729407-3 from 0 to 450560 tid 1 2020-06-12T02:37:40.711 INFO:tasks.rados.rados.0.smithi107.stdout:1986: writing smithi10729407-3 from 450560 to 1024000 tid 2 2020-06-12T02:37:40.714 INFO:tasks.rados.rados.0.smithi107.stdout:1986: writing smithi10729407-3 from 1024000 to 1638400 tid 3 2020-06-12T02:37:40.717 INFO:tasks.rados.rados.0.smithi107.stdout:1986: writing smithi10729407-3 from 1638400 to 2056192 tid 4 2020-06-12T02:37:40.718 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:37:40.718 INFO:tasks.rados.rados.0.smithi107.stdout:1987: seq_num 1986 ranges {1703936=466944,2170880=458752,2629632=417792,3047424=573440,3620864=155648} 2020-06-12T02:37:40.730 INFO:tasks.rados.rados.0.smithi107.stdout:1987: writing smithi10729407-14 from 1703936 to 2170880 tid 1 2020-06-12T02:37:40.733 INFO:tasks.rados.rados.0.smithi107.stdout:1987: writing smithi10729407-14 from 2170880 to 2629632 tid 2 2020-06-12T02:37:40.735 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:37:40.732+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:37:40.736 INFO:tasks.rados.rados.0.smithi107.stdout:1987: writing smithi10729407-14 from 2629632 to 3047424 tid 3 2020-06-12T02:37:40.739 INFO:tasks.rados.rados.0.smithi107.stdout:1987: writing smithi10729407-14 from 3047424 to 3620864 tid 4 2020-06-12T02:37:40.741 INFO:tasks.rados.rados.0.smithi107.stdout:1987: writing smithi10729407-14 from 3620864 to 3776512 tid 5 2020-06-12T02:37:40.741 INFO:tasks.rados.rados.0.smithi107.stdout:1988: delete oid 1 current snap is 0 2020-06-12T02:37:40.741 INFO:tasks.rados.rados.0.smithi107.stdout:1984: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:40.741 INFO:tasks.rados.rados.0.smithi107.stdout:1984: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:40.742 INFO:tasks.rados.rados.0.smithi107.stdout:1984: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:40.742 INFO:tasks.rados.rados.0.smithi107.stdout:1984: finishing write tid 6 to smithi10729407-2 2020-06-12T02:37:40.742 INFO:tasks.rados.rados.0.smithi107.stdout:1984: finishing write tid 7 to smithi10729407-2 2020-06-12T02:37:40.742 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 199 (ObjNum 1983 snap 0 seq_num 1983) dirty exists 2020-06-12T02:37:40.742 INFO:tasks.rados.rados.0.smithi107.stdout:1984: left oid 2 (ObjNum 1983 snap 0 seq_num 1983) 2020-06-12T02:37:40.742 INFO:tasks.rados.rados.0.smithi107.stdout:1986: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:40.743 INFO:tasks.rados.rados.0.smithi107.stdout:1986: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:40.743 INFO:tasks.rados.rados.0.smithi107.stdout:1986: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:40.743 INFO:tasks.rados.rados.0.smithi107.stdout:1986: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:40.743 INFO:tasks.rados.rados.0.smithi107.stdout:1986: finishing write tid 6 to smithi10729407-3 2020-06-12T02:37:40.743 INFO:tasks.rados.rados.0.smithi107.stdout:1986: finishing write tid 7 to smithi10729407-3 2020-06-12T02:37:40.744 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 347 (ObjNum 1985 snap 0 seq_num 1985) dirty exists 2020-06-12T02:37:40.744 INFO:tasks.rados.rados.0.smithi107.stdout:1986: left oid 3 (ObjNum 1985 snap 0 seq_num 1985) 2020-06-12T02:37:40.744 INFO:tasks.rados.rados.0.smithi107.stdout:1987: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:40.744 INFO:tasks.rados.rados.0.smithi107.stdout:1987: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:40.744 INFO:tasks.rados.rados.0.smithi107.stdout:1987: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:40.747 INFO:tasks.rados.rados.0.smithi107.stdout:1987: finishing write tid 4 to smithi10729407-14 2020-06-12T02:37:40.749 INFO:tasks.rados.rados.0.smithi107.stdout:1987: finishing write tid 5 to smithi10729407-14 2020-06-12T02:37:40.750 INFO:tasks.rados.rados.0.smithi107.stdout:1987: finishing write tid 7 to smithi10729407-14 2020-06-12T02:37:40.750 INFO:tasks.rados.rados.0.smithi107.stdout:1984: done (3 left) 2020-06-12T02:37:40.751 INFO:tasks.rados.rados.0.smithi107.stdout:1986: done (2 left) 2020-06-12T02:37:40.751 INFO:tasks.rados.rados.0.smithi107.stdout:1988: done (1 left) 2020-06-12T02:37:40.751 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:37:40.751 INFO:tasks.rados.rados.0.smithi107.stdout:1989: seq_num 1988 ranges {0=499712,499712=622592,1122304=434176,1556480=548864} 2020-06-12T02:37:40.753 INFO:tasks.rados.rados.0.smithi107.stdout:1989: writing smithi10729407-12 from 0 to 499712 tid 1 2020-06-12T02:37:40.757 INFO:tasks.rados.rados.0.smithi107.stdout:1989: writing smithi10729407-12 from 499712 to 1122304 tid 2 2020-06-12T02:37:40.760 INFO:tasks.rados.rados.0.smithi107.stdout:1989: writing smithi10729407-12 from 1122304 to 1556480 tid 3 2020-06-12T02:37:40.763 INFO:tasks.rados.rados.0.smithi107.stdout:1989: writing smithi10729407-12 from 1556480 to 2105344 tid 4 2020-06-12T02:37:40.764 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:37:40.764 INFO:tasks.rados.rados.0.smithi107.stdout:1990: seq_num 1989 ranges {0=516096,516096=507904,1024000=589824,1613824=507904} 2020-06-12T02:37:40.771 INFO:tasks.rados.rados.0.smithi107.stdout:1990: writing smithi10729407-17 from 0 to 516096 tid 1 2020-06-12T02:37:40.772 INFO:tasks.rados.rados.0.smithi107.stdout:1990: writing smithi10729407-17 from 516096 to 1024000 tid 2 2020-06-12T02:37:40.773 INFO:tasks.rados.rados.0.smithi107.stdout:1990: writing smithi10729407-17 from 1024000 to 1613824 tid 3 2020-06-12T02:37:40.777 INFO:tasks.rados.rados.0.smithi107.stdout:1990: writing smithi10729407-17 from 1613824 to 2121728 tid 4 2020-06-12T02:37:40.779 INFO:tasks.rados.rados.0.smithi107.stdout:1991: delete oid 18 current snap is 0 2020-06-12T02:37:40.780 INFO:tasks.rados.rados.0.smithi107.stdout:1987: finishing write tid 8 to smithi10729407-14 2020-06-12T02:37:40.780 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 366 (ObjNum 1986 snap 0 seq_num 1986) dirty exists 2020-06-12T02:37:40.780 INFO:tasks.rados.rados.0.smithi107.stdout:1987: left oid 14 (ObjNum 1986 snap 0 seq_num 1986) 2020-06-12T02:37:40.780 INFO:tasks.rados.rados.0.smithi107.stdout:1989: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:40.781 INFO:tasks.rados.rados.0.smithi107.stdout:1989: finishing write tid 2 to smithi10729407-12 2020-06-12T02:37:40.782 INFO:tasks.rados.rados.0.smithi107.stdout:1989: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:40.782 INFO:tasks.rados.rados.0.smithi107.stdout:1989: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:40.782 INFO:tasks.rados.rados.0.smithi107.stdout:1989: finishing write tid 6 to smithi10729407-12 2020-06-12T02:37:40.782 INFO:tasks.rados.rados.0.smithi107.stdout:1989: finishing write tid 7 to smithi10729407-12 2020-06-12T02:37:40.782 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 305 (ObjNum 1988 snap 0 seq_num 1988) dirty exists 2020-06-12T02:37:40.783 INFO:tasks.rados.rados.0.smithi107.stdout:1989: left oid 12 (ObjNum 1988 snap 0 seq_num 1988) 2020-06-12T02:37:40.783 INFO:tasks.rados.rados.0.smithi107.stdout:1990: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:40.784 INFO:tasks.rados.rados.0.smithi107.stdout:1990: finishing write tid 2 to smithi10729407-17 2020-06-12T02:37:40.791 INFO:tasks.rados.rados.0.smithi107.stdout:1990: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:40.792 INFO:tasks.rados.rados.0.smithi107.stdout:1987: done (3 left) 2020-06-12T02:37:40.792 INFO:tasks.rados.rados.0.smithi107.stdout:1989: done (2 left) 2020-06-12T02:37:40.792 INFO:tasks.rados.rados.0.smithi107.stdout:1991: done (1 left) 2020-06-12T02:37:40.792 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:37:40.793 INFO:tasks.rados.rados.0.smithi107.stdout:1992: seq_num 1991 ranges {1638400=548864,2187264=679936,2867200=606208,3473408=319488} 2020-06-12T02:37:40.793 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:40.793 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:40.795 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-06-12T02:37:40.795 INFO:tasks.rados.rados.0.smithi107.stdout:1992: writing smithi10729407-11 from 1638400 to 2187264 tid 1 2020-06-12T02:37:40.797 INFO:tasks.rados.rados.0.smithi107.stdout:1992: writing smithi10729407-11 from 2187264 to 2867200 tid 2 2020-06-12T02:37:40.801 INFO:tasks.rados.rados.0.smithi107.stdout:1992: writing smithi10729407-11 from 2867200 to 3473408 tid 3 2020-06-12T02:37:40.803 INFO:tasks.rados.rados.0.smithi107.stdout:1992: writing smithi10729407-11 from 3473408 to 3792896 tid 4 2020-06-12T02:37:40.803 INFO:tasks.rados.rados.0.smithi107.stdout:1993: delete oid 15 current snap is 0 2020-06-12T02:37:40.804 INFO:tasks.rados.rados.0.smithi107.stdout:1990: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:40.804 INFO:tasks.rados.rados.0.smithi107.stdout:1990: finishing write tid 6 to smithi10729407-17 2020-06-12T02:37:40.804 INFO:tasks.rados.rados.0.smithi107.stdout:1990: finishing write tid 7 to smithi10729407-17 2020-06-12T02:37:40.804 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 387 (ObjNum 1989 snap 0 seq_num 1989) dirty exists 2020-06-12T02:37:40.804 INFO:tasks.rados.rados.0.smithi107.stdout:1990: left oid 17 (ObjNum 1989 snap 0 seq_num 1989) 2020-06-12T02:37:40.805 INFO:tasks.rados.rados.0.smithi107.stdout:1992: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:40.806 INFO:tasks.rados.rados.0.smithi107.stdout:1990: done (2 left) 2020-06-12T02:37:40.806 INFO:tasks.rados.rados.0.smithi107.stdout:1993: done (1 left) 2020-06-12T02:37:40.806 INFO:tasks.rados.rados.0.smithi107.stdout:1994: delete oid 3 current snap is 0 2020-06-12T02:37:40.811 INFO:tasks.rados.rados.0.smithi107.stdout:1992: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:40.812 INFO:tasks.rados.rados.0.smithi107.stdout:1994: done (1 left) 2020-06-12T02:37:40.812 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:37:40.813 INFO:tasks.rados.rados.0.smithi107.stdout:1995: seq_num 1994 ranges {0=598016,598016=778240,1376256=778240,2154496=49152} 2020-06-12T02:37:40.816 INFO:tasks.rados.rados.0.smithi107.stdout:1995: writing smithi10729407-16 from 0 to 598016 tid 1 2020-06-12T02:37:40.822 INFO:tasks.rados.rados.0.smithi107.stdout:1995: writing smithi10729407-16 from 598016 to 1376256 tid 2 2020-06-12T02:37:40.826 INFO:tasks.rados.rados.0.smithi107.stdout:1995: writing smithi10729407-16 from 1376256 to 2154496 tid 3 2020-06-12T02:37:40.827 INFO:tasks.rados.rados.0.smithi107.stdout:1995: writing smithi10729407-16 from 2154496 to 2203648 tid 4 2020-06-12T02:37:40.827 INFO:tasks.rados.rados.0.smithi107.stdout:1992: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:40.828 INFO:tasks.rados.rados.0.smithi107.stdout:1992: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:40.828 INFO:tasks.rados.rados.0.smithi107.stdout:1992: finishing write tid 6 to smithi10729407-11 2020-06-12T02:37:40.828 INFO:tasks.rados.rados.0.smithi107.stdout:1992: finishing write tid 7 to smithi10729407-11 2020-06-12T02:37:40.828 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 371 (ObjNum 1991 snap 0 seq_num 1991) dirty exists 2020-06-12T02:37:40.830 INFO:tasks.rados.rados.0.smithi107.stdout:1992: left oid 11 (ObjNum 1991 snap 0 seq_num 1991) 2020-06-12T02:37:40.830 INFO:tasks.rados.rados.0.smithi107.stdout:1995: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:40.830 INFO:tasks.rados.rados.0.smithi107.stdout:1992: done (1 left) 2020-06-12T02:37:40.830 INFO:tasks.rados.rados.0.smithi107.stdout:1996: delete oid 8 current snap is 0 2020-06-12T02:37:40.831 INFO:tasks.rados.rados.0.smithi107.stdout:1996: done (1 left) 2020-06-12T02:37:40.831 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:37:40.831 INFO:tasks.rados.rados.0.smithi107.stdout:1997: seq_num 1996 ranges {0=638976,638976=548864,1187840=794624,1982464=262144} 2020-06-12T02:37:40.833 INFO:tasks.rados.rados.0.smithi107.stdout:1997: writing smithi10729407-6 from 0 to 638976 tid 1 2020-06-12T02:37:40.836 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:40.832+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:40.836 INFO:tasks.rados.rados.0.smithi107.stdout:1997: writing smithi10729407-6 from 638976 to 1187840 tid 2 2020-06-12T02:37:40.841 INFO:tasks.rados.rados.0.smithi107.stdout:1997: writing smithi10729407-6 from 1187840 to 1982464 tid 3 2020-06-12T02:37:40.843 INFO:tasks.rados.rados.0.smithi107.stdout:1997: writing smithi10729407-6 from 1982464 to 2244608 tid 4 2020-06-12T02:37:40.843 INFO:tasks.rados.rados.0.smithi107.stdout:1998: delete oid 3 current snap is 0 2020-06-12T02:37:40.844 INFO:tasks.rados.rados.0.smithi107.stdout:1995: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:40.844 INFO:tasks.rados.rados.0.smithi107.stdout:1995: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:40.844 INFO:tasks.rados.rados.0.smithi107.stdout:1995: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:40.844 INFO:tasks.rados.rados.0.smithi107.stdout:1995: finishing write tid 6 to smithi10729407-16 2020-06-12T02:37:40.844 INFO:tasks.rados.rados.0.smithi107.stdout:1995: finishing write tid 7 to smithi10729407-16 2020-06-12T02:37:40.845 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 353 (ObjNum 1994 snap 0 seq_num 1994) dirty exists 2020-06-12T02:37:40.845 INFO:tasks.rados.rados.0.smithi107.stdout:1995: left oid 16 (ObjNum 1994 snap 0 seq_num 1994) 2020-06-12T02:37:40.845 INFO:tasks.rados.rados.0.smithi107.stdout:1997: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:40.845 INFO:tasks.rados.rados.0.smithi107.stdout:1995: done (2 left) 2020-06-12T02:37:40.846 INFO:tasks.rados.rados.0.smithi107.stdout:1998: done (1 left) 2020-06-12T02:37:40.846 INFO:tasks.rados.rados.0.smithi107.stdout:1999: delete oid 7 current snap is 0 2020-06-12T02:37:40.848 INFO:tasks.rados.rados.0.smithi107.stdout:1997: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:40.849 INFO:tasks.rados.rados.0.smithi107.stdout:1997: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:40.849 INFO:tasks.rados.rados.0.smithi107.stdout:1999: done (1 left) 2020-06-12T02:37:40.849 INFO:tasks.rados.rados.0.smithi107.stdout:2000: delete oid 7 current snap is 0 2020-06-12T02:37:40.850 INFO:tasks.rados.rados.0.smithi107.stdout:1997: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:40.850 INFO:tasks.rados.rados.0.smithi107.stdout:1997: finishing write tid 6 to smithi10729407-6 2020-06-12T02:37:40.851 INFO:tasks.rados.rados.0.smithi107.stdout:1997: finishing write tid 7 to smithi10729407-6 2020-06-12T02:37:40.851 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 410 (ObjNum 1996 snap 0 seq_num 1996) dirty exists 2020-06-12T02:37:40.852 INFO:tasks.rados.rados.0.smithi107.stdout:1997: left oid 6 (ObjNum 1996 snap 0 seq_num 1996) 2020-06-12T02:37:40.853 INFO:tasks.rados.rados.0.smithi107.stdout:1997: done (1 left) 2020-06-12T02:37:40.853 INFO:tasks.rados.rados.0.smithi107.stdout:2000: done (0 left) 2020-06-12T02:37:40.853 INFO:tasks.rados.rados.0.smithi107.stdout:2001: delete oid 9 current snap is 0 2020-06-12T02:37:40.856 INFO:tasks.rados.rados.0.smithi107.stdout:2001: done (0 left) 2020-06-12T02:37:40.856 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:37:40.856 INFO:tasks.rados.rados.0.smithi107.stdout:2002: seq_num 2001 ranges {1671168=720896,2392064=417792,2809856=532480,3342336=442368,3784704=212992} 2020-06-12T02:37:40.868 INFO:tasks.rados.rados.0.smithi107.stdout:2002: writing smithi10729407-5 from 1671168 to 2392064 tid 1 2020-06-12T02:37:40.870 INFO:tasks.rados.rados.0.smithi107.stdout:2002: writing smithi10729407-5 from 2392064 to 2809856 tid 2 2020-06-12T02:37:40.873 INFO:tasks.rados.rados.0.smithi107.stdout:2002: writing smithi10729407-5 from 2809856 to 3342336 tid 3 2020-06-12T02:37:40.876 INFO:tasks.rados.rados.0.smithi107.stdout:2002: writing smithi10729407-5 from 3342336 to 3784704 tid 4 2020-06-12T02:37:40.877 INFO:tasks.rados.rados.0.smithi107.stdout:2002: writing smithi10729407-5 from 3784704 to 3997696 tid 5 2020-06-12T02:37:40.878 INFO:tasks.rados.rados.0.smithi107.stdout:2003: delete oid 11 current snap is 0 2020-06-12T02:37:40.878 INFO:tasks.rados.rados.0.smithi107.stdout:2002: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:40.879 INFO:tasks.rados.rados.0.smithi107.stdout:2003: done (1 left) 2020-06-12T02:37:40.879 INFO:tasks.rados.rados.0.smithi107.stdout:2004: delete oid 6 current snap is 0 2020-06-12T02:37:40.883 INFO:tasks.rados.rados.0.smithi107.stdout:2002: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:40.884 INFO:tasks.rados.rados.0.smithi107.stdout:2004: done (1 left) 2020-06-12T02:37:40.884 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:37:40.884 INFO:tasks.rados.rados.0.smithi107.stdout:2005: seq_num 2004 ranges {0=770048,770048=475136,1245184=753664,1998848=376832} 2020-06-12T02:37:40.888 INFO:tasks.rados.rados.0.smithi107.stdout:2005: writing smithi10729407-1 from 0 to 770048 tid 1 2020-06-12T02:37:40.891 INFO:tasks.rados.rados.0.smithi107.stdout:2005: writing smithi10729407-1 from 770048 to 1245184 tid 2 2020-06-12T02:37:40.895 INFO:tasks.rados.rados.0.smithi107.stdout:2005: writing smithi10729407-1 from 1245184 to 1998848 tid 3 2020-06-12T02:37:40.898 INFO:tasks.rados.rados.0.smithi107.stdout:2005: writing smithi10729407-1 from 1998848 to 2375680 tid 4 2020-06-12T02:37:40.898 INFO:tasks.rados.rados.0.smithi107.stdout:2006: delete oid 4 current snap is 0 2020-06-12T02:37:40.898 INFO:tasks.rados.rados.0.smithi107.stdout:2002: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:40.899 INFO:tasks.rados.rados.0.smithi107.stdout:2002: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:40.899 INFO:tasks.rados.rados.0.smithi107.stdout:2002: finishing write tid 5 to smithi10729407-5 2020-06-12T02:37:40.899 INFO:tasks.rados.rados.0.smithi107.stdout:2002: finishing write tid 7 to smithi10729407-5 2020-06-12T02:37:40.899 INFO:tasks.rados.rados.0.smithi107.stdout:2002: finishing write tid 8 to smithi10729407-5 2020-06-12T02:37:40.899 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 165 (ObjNum 2001 snap 0 seq_num 2001) dirty exists 2020-06-12T02:37:40.900 INFO:tasks.rados.rados.0.smithi107.stdout:2002: left oid 5 (ObjNum 2001 snap 0 seq_num 2001) 2020-06-12T02:37:40.900 INFO:tasks.rados.rados.0.smithi107.stdout:2005: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:40.900 INFO:tasks.rados.rados.0.smithi107.stdout:2005: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:40.901 INFO:tasks.rados.rados.0.smithi107.stdout:2002: done (2 left) 2020-06-12T02:37:40.901 INFO:tasks.rados.rados.0.smithi107.stdout:2006: done (1 left) 2020-06-12T02:37:40.901 INFO:tasks.rados.rados.0.smithi107.stdout:2007: delete oid 2 current snap is 0 2020-06-12T02:37:40.904 INFO:tasks.rados.rados.0.smithi107.stdout:2007: done (1 left) 2020-06-12T02:37:40.904 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:37:40.904 INFO:tasks.rados.rados.0.smithi107.stdout:2008: seq_num 2007 ranges {0=16384} 2020-06-12T02:37:40.905 INFO:tasks.rados.rados.0.smithi107.stdout:2008: writing smithi10729407-18 from 0 to 16384 tid 1 2020-06-12T02:37:40.905 INFO:tasks.rados.rados.0.smithi107.stdout:2009: delete oid 16 current snap is 0 2020-06-12T02:37:40.905 INFO:tasks.rados.rados.0.smithi107.stdout:2005: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:40.907 INFO:tasks.rados.rados.0.smithi107.stdout:2005: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:40.907 INFO:tasks.rados.rados.0.smithi107.stdout:2005: finishing write tid 6 to smithi10729407-1 2020-06-12T02:37:40.908 INFO:tasks.rados.rados.0.smithi107.stdout:2005: finishing write tid 7 to smithi10729407-1 2020-06-12T02:37:40.909 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 416 (ObjNum 2004 snap 0 seq_num 2004) dirty exists 2020-06-12T02:37:40.909 INFO:tasks.rados.rados.0.smithi107.stdout:2005: left oid 1 (ObjNum 2004 snap 0 seq_num 2004) 2020-06-12T02:37:40.909 INFO:tasks.rados.rados.0.smithi107.stdout:2008: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:40.911 INFO:tasks.rados.rados.0.smithi107.stdout:2005: done (2 left) 2020-06-12T02:37:40.912 INFO:tasks.rados.rados.0.smithi107.stdout:2009: done (1 left) 2020-06-12T02:37:40.912 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:37:40.912 INFO:tasks.rados.rados.0.smithi107.stdout:2010: seq_num 2009 ranges {0=49152} 2020-06-12T02:37:40.912 INFO:tasks.rados.rados.0.smithi107.stdout:2010: writing smithi10729407-16 from 0 to 49152 tid 1 2020-06-12T02:37:40.913 INFO:tasks.rados.rados.0.smithi107.stdout:2011: delete oid 15 current snap is 0 2020-06-12T02:37:40.917 INFO:tasks.rados.rados.0.smithi107.stdout:2011: done (2 left) 2020-06-12T02:37:40.917 INFO:tasks.rados.rados.0.smithi107.stdout:2012: delete oid 6 current snap is 0 2020-06-12T02:37:40.917 INFO:tasks.rados.rados.0.smithi107.stdout:2008: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:40.918 INFO:tasks.rados.rados.0.smithi107.stdout:2008: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:40.918 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 344 (ObjNum 2007 snap 0 seq_num 2007) dirty exists 2020-06-12T02:37:40.919 INFO:tasks.rados.rados.0.smithi107.stdout:2008: left oid 18 (ObjNum 2007 snap 0 seq_num 2007) 2020-06-12T02:37:40.919 INFO:tasks.rados.rados.0.smithi107.stdout:2010: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:40.919 INFO:tasks.rados.rados.0.smithi107.stdout:2010: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:40.920 INFO:tasks.rados.rados.0.smithi107.stdout:2008: done (2 left) 2020-06-12T02:37:40.920 INFO:tasks.rados.rados.0.smithi107.stdout:2012: done (1 left) 2020-06-12T02:37:40.920 INFO:tasks.rados.rados.0.smithi107.stdout:2013: delete oid 19 current snap is 0 2020-06-12T02:37:40.921 INFO:tasks.rados.rados.0.smithi107.stdout:2010: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:40.921 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 357 (ObjNum 2009 snap 0 seq_num 2009) dirty exists 2020-06-12T02:37:40.921 INFO:tasks.rados.rados.0.smithi107.stdout:2010: left oid 16 (ObjNum 2009 snap 0 seq_num 2009) 2020-06-12T02:37:40.923 INFO:tasks.rados.rados.0.smithi107.stdout:2010: done (1 left) 2020-06-12T02:37:40.923 INFO:tasks.rados.rados.0.smithi107.stdout:2013: done (0 left) 2020-06-12T02:37:40.923 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:37:40.923 INFO:tasks.rados.rados.0.smithi107.stdout:2014: seq_num 2013 ranges {0=114688} 2020-06-12T02:37:40.923 INFO:tasks.rados.rados.0.smithi107.stdout:2014: writing smithi10729407-2 from 0 to 114688 tid 1 2020-06-12T02:37:40.924 INFO:tasks.rados.rados.0.smithi107.stdout:2015: delete oid 3 current snap is 0 2020-06-12T02:37:40.924 INFO:tasks.rados.rados.0.smithi107.stdout:2015: done (1 left) 2020-06-12T02:37:40.924 INFO:tasks.rados.rados.0.smithi107.stdout:2016: delete oid 15 current snap is 0 2020-06-12T02:37:40.925 INFO:tasks.rados.rados.0.smithi107.stdout:2014: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:40.927 INFO:tasks.rados.rados.0.smithi107.stdout:2014: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:40.927 INFO:tasks.rados.rados.0.smithi107.stdout:2016: done (1 left) 2020-06-12T02:37:40.928 INFO:tasks.rados.rados.0.smithi107.stdout:2017: delete oid 7 current snap is 0 2020-06-12T02:37:40.930 INFO:tasks.rados.rados.0.smithi107.stdout:2014: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:40.931 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 202 (ObjNum 2013 snap 0 seq_num 2013) dirty exists 2020-06-12T02:37:40.931 INFO:tasks.rados.rados.0.smithi107.stdout:2014: left oid 2 (ObjNum 2013 snap 0 seq_num 2013) 2020-06-12T02:37:40.931 INFO:tasks.rados.rados.0.smithi107.stdout:2014: done (1 left) 2020-06-12T02:37:40.931 INFO:tasks.rados.rados.0.smithi107.stdout:2017: done (0 left) 2020-06-12T02:37:40.931 INFO:tasks.rados.rados.0.smithi107.stdout:2018: delete oid 19 current snap is 0 2020-06-12T02:37:40.932 INFO:tasks.rados.rados.0.smithi107.stdout:2018: done (0 left) 2020-06-12T02:37:40.932 INFO:tasks.rados.rados.0.smithi107.stdout:2019: delete oid 13 current snap is 0 2020-06-12T02:37:40.934 INFO:tasks.rados.rados.0.smithi107.stdout:2019: done (0 left) 2020-06-12T02:37:40.934 INFO:tasks.rados.rados.0.smithi107.stdout:2020: delete oid 2 current snap is 0 2020-06-12T02:37:40.936 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:40.933+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:40.939 INFO:tasks.rados.rados.0.smithi107.stdout:2020: done (0 left) 2020-06-12T02:37:40.939 INFO:tasks.rados.rados.0.smithi107.stdout:2021: delete oid 14 current snap is 0 2020-06-12T02:37:40.942 INFO:tasks.rados.rados.0.smithi107.stdout:2021: done (0 left) 2020-06-12T02:37:40.942 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:37:40.942 INFO:tasks.rados.rados.0.smithi107.stdout:2022: seq_num 2021 ranges {3997696=253952} 2020-06-12T02:37:40.949 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:40.949 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:40.958 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-06-12T02:37:40.965 INFO:tasks.rados.rados.0.smithi107.stdout:2022: writing smithi10729407-5 from 3997696 to 4251648 tid 1 2020-06-12T02:37:40.965 INFO:tasks.rados.rados.0.smithi107.stdout:2023: delete oid 10 current snap is 0 2020-06-12T02:37:40.969 INFO:tasks.rados.rados.0.smithi107.stdout:2023: done (1 left) 2020-06-12T02:37:40.969 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:37:40.970 INFO:tasks.rados.rados.0.smithi107.stdout:2024: seq_num 2023 ranges {2375680=286720} 2020-06-12T02:37:40.985 INFO:tasks.rados.rados.0.smithi107.stdout:2024: writing smithi10729407-1 from 2375680 to 2662400 tid 1 2020-06-12T02:37:40.985 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:37:40.985 INFO:tasks.rados.rados.0.smithi107.stdout:2025: seq_num 2024 ranges {0=303104} 2020-06-12T02:37:40.987 INFO:tasks.rados.rados.0.smithi107.stdout:2025: writing smithi10729407-3 from 0 to 303104 tid 1 2020-06-12T02:37:40.987 INFO:tasks.rados.rados.0.smithi107.stdout:2026: delete oid 7 current snap is 0 2020-06-12T02:37:40.987 INFO:tasks.rados.rados.0.smithi107.stdout:2022: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:40.988 INFO:tasks.rados.rados.0.smithi107.stdout:2022: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:40.988 INFO:tasks.rados.rados.0.smithi107.stdout:2022: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:40.988 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 167 (ObjNum 2021 snap 0 seq_num 2021) dirty exists 2020-06-12T02:37:40.988 INFO:tasks.rados.rados.0.smithi107.stdout:2022: left oid 5 (ObjNum 2021 snap 0 seq_num 2021) 2020-06-12T02:37:40.989 INFO:tasks.rados.rados.0.smithi107.stdout:2024: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:40.989 INFO:tasks.rados.rados.0.smithi107.stdout:2022: done (3 left) 2020-06-12T02:37:40.990 INFO:tasks.rados.rados.0.smithi107.stdout:2026: done (2 left) 2020-06-12T02:37:40.990 INFO:tasks.rados.rados.0.smithi107.stdout:2027: delete oid 14 current snap is 0 2020-06-12T02:37:40.990 INFO:tasks.rados.rados.0.smithi107.stdout:2024: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:40.991 INFO:tasks.rados.rados.0.smithi107.stdout:2025: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:40.992 INFO:tasks.rados.rados.0.smithi107.stdout:2024: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:40.992 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 419 (ObjNum 2023 snap 0 seq_num 2023) dirty exists 2020-06-12T02:37:40.992 INFO:tasks.rados.rados.0.smithi107.stdout:2024: left oid 1 (ObjNum 2023 snap 0 seq_num 2023) 2020-06-12T02:37:40.992 INFO:tasks.rados.rados.0.smithi107.stdout:2024: done (2 left) 2020-06-12T02:37:40.993 INFO:tasks.rados.rados.0.smithi107.stdout:2027: done (1 left) 2020-06-12T02:37:40.993 INFO:tasks.rados.rados.0.smithi107.stdout:2028: delete oid 15 current snap is 0 2020-06-12T02:37:40.994 INFO:tasks.rados.rados.0.smithi107.stdout:2025: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:40.994 INFO:tasks.rados.rados.0.smithi107.stdout:2028: done (1 left) 2020-06-12T02:37:40.994 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:37:40.995 INFO:tasks.rados.rados.0.smithi107.stdout:2029: seq_num 2028 ranges {0=368640} 2020-06-12T02:37:41.003 INFO:tasks.rados.rados.0.smithi107.stdout:2029: writing smithi10729407-10 from 0 to 368640 tid 1 2020-06-12T02:37:41.003 INFO:tasks.rados.rados.0.smithi107.stdout:2030: delete oid 16 current snap is 0 2020-06-12T02:37:41.004 INFO:tasks.rados.rados.0.smithi107.stdout:2025: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:41.004 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 360 (ObjNum 2024 snap 0 seq_num 2024) dirty exists 2020-06-12T02:37:41.004 INFO:tasks.rados.rados.0.smithi107.stdout:2025: left oid 3 (ObjNum 2024 snap 0 seq_num 2024) 2020-06-12T02:37:41.004 INFO:tasks.rados.rados.0.smithi107.stdout:2025: done (2 left) 2020-06-12T02:37:41.004 INFO:tasks.rados.rados.0.smithi107.stdout:2030: done (1 left) 2020-06-12T02:37:41.005 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:37:41.005 INFO:tasks.rados.rados.0.smithi107.stdout:2031: seq_num 2030 ranges {0=401408} 2020-06-12T02:37:41.006 INFO:tasks.rados.rados.0.smithi107.stdout:2031: writing smithi10729407-7 from 0 to 401408 tid 1 2020-06-12T02:37:41.006 INFO:tasks.rados.rados.0.smithi107.stdout:2032: delete oid 5 current snap is 0 2020-06-12T02:37:41.006 INFO:tasks.rados.rados.0.smithi107.stdout:2029: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:41.006 INFO:tasks.rados.rados.0.smithi107.stdout:2029: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:41.006 INFO:tasks.rados.rados.0.smithi107.stdout:2029: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:41.007 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 189 (ObjNum 2028 snap 0 seq_num 2028) dirty exists 2020-06-12T02:37:41.007 INFO:tasks.rados.rados.0.smithi107.stdout:2029: left oid 10 (ObjNum 2028 snap 0 seq_num 2028) 2020-06-12T02:37:41.008 INFO:tasks.rados.rados.0.smithi107.stdout:2029: done (2 left) 2020-06-12T02:37:41.009 INFO:tasks.rados.rados.0.smithi107.stdout:2032: done (1 left) 2020-06-12T02:37:41.009 INFO:tasks.rados.rados.0.smithi107.stdout:2033: delete oid 18 current snap is 0 2020-06-12T02:37:41.009 INFO:tasks.rados.rados.0.smithi107.stdout:2031: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:41.010 INFO:tasks.rados.rados.0.smithi107.stdout:2033: done (1 left) 2020-06-12T02:37:41.011 INFO:tasks.rados.rados.0.smithi107.stdout:2034: delete oid 8 current snap is 0 2020-06-12T02:37:41.012 INFO:tasks.rados.rados.0.smithi107.stdout:2031: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:41.013 INFO:tasks.rados.rados.0.smithi107.stdout:2031: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:41.013 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 391 (ObjNum 2030 snap 0 seq_num 2030) dirty exists 2020-06-12T02:37:41.013 INFO:tasks.rados.rados.0.smithi107.stdout:2031: left oid 7 (ObjNum 2030 snap 0 seq_num 2030) 2020-06-12T02:37:41.015 INFO:tasks.rados.rados.0.smithi107.stdout:2031: done (1 left) 2020-06-12T02:37:41.015 INFO:tasks.rados.rados.0.smithi107.stdout:2034: done (0 left) 2020-06-12T02:37:41.015 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:37:41.015 INFO:tasks.rados.rados.0.smithi107.stdout:2035: seq_num 2034 ranges {401408=475136} 2020-06-12T02:37:41.019 INFO:tasks.rados.rados.0.smithi107.stdout:2035: writing smithi10729407-7 from 401408 to 876544 tid 1 2020-06-12T02:37:41.020 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:37:41.020 INFO:tasks.rados.rados.0.smithi107.stdout:2036: seq_num 2035 ranges {0=491520} 2020-06-12T02:37:41.023 INFO:tasks.rados.rados.0.smithi107.stdout:2036: writing smithi10729407-6 from 0 to 491520 tid 1 2020-06-12T02:37:41.023 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:37:41.023 INFO:tasks.rados.rados.0.smithi107.stdout:2037: seq_num 2036 ranges {0=507904} 2020-06-12T02:37:41.026 INFO:tasks.rados.rados.0.smithi107.stdout:2037: writing smithi10729407-4 from 0 to 507904 tid 1 2020-06-12T02:37:41.027 INFO:tasks.rados.rados.0.smithi107.stdout:2038: delete oid 12 current snap is 0 2020-06-12T02:37:41.027 INFO:tasks.rados.rados.0.smithi107.stdout:2035: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:41.029 INFO:tasks.rados.rados.0.smithi107.stdout:2036: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:41.029 INFO:tasks.rados.rados.0.smithi107.stdout:2038: done (3 left) 2020-06-12T02:37:41.030 INFO:tasks.rados.rados.0.smithi107.stdout:2039: delete oid 9 current snap is 0 2020-06-12T02:37:41.030 INFO:tasks.rados.rados.0.smithi107.stdout:2036: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:41.032 INFO:tasks.rados.rados.0.smithi107.stdout:2035: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:41.032 INFO:tasks.rados.rados.0.smithi107.stdout:2036: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:41.032 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 421 (ObjNum 2035 snap 0 seq_num 2035) dirty exists 2020-06-12T02:37:41.033 INFO:tasks.rados.rados.0.smithi107.stdout:2036: left oid 6 (ObjNum 2035 snap 0 seq_num 2035) 2020-06-12T02:37:41.033 INFO:tasks.rados.rados.0.smithi107.stdout:2037: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:41.033 INFO:tasks.rados.rados.0.smithi107.stdout:2036: done (3 left) 2020-06-12T02:37:41.034 INFO:tasks.rados.rados.0.smithi107.stdout:2039: done (2 left) 2020-06-12T02:37:41.034 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:37:41.034 INFO:tasks.rados.rados.0.smithi107.stdout:2040: seq_num 2039 ranges {0=557056} 2020-06-12T02:37:41.036 INFO:tasks.rados.rados.0.smithi107.stdout:2040: writing smithi10729407-12 from 0 to 557056 tid 1 2020-06-12T02:37:41.036 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:37:41.036 INFO:tasks.rados.rados.0.smithi107.stdout:2041: seq_num 2040 ranges {2662400=573440} 2020-06-12T02:37:41.038 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:37:41.035+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:37:41.054 INFO:tasks.rados.rados.0.smithi107.stdout:2041: writing smithi10729407-1 from 2662400 to 3235840 tid 1 2020-06-12T02:37:41.056 INFO:tasks.rados.rados.0.smithi107.stdout:2042: delete oid 9 current snap is 0 2020-06-12T02:37:41.056 INFO:tasks.rados.rados.0.smithi107.stdout:2035: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:41.056 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 393 (ObjNum 2034 snap 0 seq_num 2034) dirty exists 2020-06-12T02:37:41.056 INFO:tasks.rados.rados.0.smithi107.stdout:2035: left oid 7 (ObjNum 2034 snap 0 seq_num 2034) 2020-06-12T02:37:41.056 INFO:tasks.rados.rados.0.smithi107.stdout:2037: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:41.057 INFO:tasks.rados.rados.0.smithi107.stdout:2040: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:41.057 INFO:tasks.rados.rados.0.smithi107.stdout:2037: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:41.057 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 390 (ObjNum 2036 snap 0 seq_num 2036) dirty exists 2020-06-12T02:37:41.057 INFO:tasks.rados.rados.0.smithi107.stdout:2037: left oid 4 (ObjNum 2036 snap 0 seq_num 2036) 2020-06-12T02:37:41.057 INFO:tasks.rados.rados.0.smithi107.stdout:2040: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:41.057 INFO:tasks.rados.rados.0.smithi107.stdout:2040: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:41.058 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 310 (ObjNum 2039 snap 0 seq_num 2039) dirty exists 2020-06-12T02:37:41.058 INFO:tasks.rados.rados.0.smithi107.stdout:2040: left oid 12 (ObjNum 2039 snap 0 seq_num 2039) 2020-06-12T02:37:41.059 INFO:tasks.rados.rados.0.smithi107.stdout:2035: done (4 left) 2020-06-12T02:37:41.059 INFO:tasks.rados.rados.0.smithi107.stdout:2037: done (3 left) 2020-06-12T02:37:41.059 INFO:tasks.rados.rados.0.smithi107.stdout:2040: done (2 left) 2020-06-12T02:37:41.059 INFO:tasks.rados.rados.0.smithi107.stdout:2042: done (1 left) 2020-06-12T02:37:41.059 INFO:tasks.rados.rados.0.smithi107.stdout:2043: delete oid 4 current snap is 0 2020-06-12T02:37:41.060 INFO:tasks.rados.rados.0.smithi107.stdout:2041: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:41.060 INFO:tasks.rados.rados.0.smithi107.stdout:2043: done (1 left) 2020-06-12T02:37:41.060 INFO:tasks.rados.rados.0.smithi107.stdout:2044: delete oid 9 current snap is 0 2020-06-12T02:37:41.060 INFO:tasks.rados.rados.0.smithi107.stdout:2041: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:41.061 INFO:tasks.rados.rados.0.smithi107.stdout:2041: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:41.062 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 423 (ObjNum 2040 snap 0 seq_num 2040) dirty exists 2020-06-12T02:37:41.062 INFO:tasks.rados.rados.0.smithi107.stdout:2041: left oid 1 (ObjNum 2040 snap 0 seq_num 2040) 2020-06-12T02:37:41.062 INFO:tasks.rados.rados.0.smithi107.stdout:2041: done (1 left) 2020-06-12T02:37:41.063 INFO:tasks.rados.rados.0.smithi107.stdout:2044: done (0 left) 2020-06-12T02:37:41.063 INFO:tasks.rados.rados.0.smithi107.stdout:2045: delete oid 2 current snap is 0 2020-06-12T02:37:41.068 INFO:tasks.rados.rados.0.smithi107.stdout:2045: done (0 left) 2020-06-12T02:37:41.068 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:37:41.068 INFO:tasks.rados.rados.0.smithi107.stdout:2046: seq_num 2045 ranges {0=655360} 2020-06-12T02:37:41.071 INFO:tasks.rados.rados.0.smithi107.stdout:2046: writing smithi10729407-9 from 0 to 655360 tid 1 2020-06-12T02:37:41.071 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:41.072 INFO:tasks.rados.rados.0.smithi107.stdout:2047: seq_num 2046 ranges {0=671744} 2020-06-12T02:37:41.075 INFO:tasks.rados.rados.0.smithi107.stdout:2047: writing smithi10729407-2 from 0 to 671744 tid 1 2020-06-12T02:37:41.075 INFO:tasks.rados.rados.0.smithi107.stdout:2048: delete oid 3 current snap is 0 2020-06-12T02:37:41.080 INFO:tasks.rados.rados.0.smithi107.stdout:2048: done (2 left) 2020-06-12T02:37:41.080 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:37:41.080 INFO:tasks.rados.rados.0.smithi107.stdout:2049: seq_num 2048 ranges {0=704512} 2020-06-12T02:37:41.085 INFO:tasks.rados.rados.0.smithi107.stdout:2049: writing smithi10729407-16 from 0 to 704512 tid 1 2020-06-12T02:37:41.085 INFO:tasks.rados.rados.0.smithi107.stdout:2050: delete oid 14 current snap is 0 2020-06-12T02:37:41.086 INFO:tasks.rados.rados.0.smithi107.stdout:2046: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:41.086 INFO:tasks.rados.rados.0.smithi107.stdout:2046: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:41.086 INFO:tasks.rados.rados.0.smithi107.stdout:2047: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:41.086 INFO:tasks.rados.rados.0.smithi107.stdout:2046: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:41.086 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 350 (ObjNum 2045 snap 0 seq_num 2045) dirty exists 2020-06-12T02:37:41.087 INFO:tasks.rados.rados.0.smithi107.stdout:2046: left oid 9 (ObjNum 2045 snap 0 seq_num 2045) 2020-06-12T02:37:41.087 INFO:tasks.rados.rados.0.smithi107.stdout:2047: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:41.088 INFO:tasks.rados.rados.0.smithi107.stdout:2046: done (3 left) 2020-06-12T02:37:41.088 INFO:tasks.rados.rados.0.smithi107.stdout:2050: done (2 left) 2020-06-12T02:37:41.088 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:37:41.088 INFO:tasks.rados.rados.0.smithi107.stdout:2051: seq_num 2050 ranges {655360=737280} 2020-06-12T02:37:41.097 INFO:tasks.rados.rados.0.smithi107.stdout:2051: writing smithi10729407-9 from 655360 to 1392640 tid 1 2020-06-12T02:37:41.097 INFO:tasks.rados.rados.0.smithi107.stdout:2052: delete oid 15 current snap is 0 2020-06-12T02:37:41.097 INFO:tasks.rados.rados.0.smithi107.stdout:2047: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:41.097 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 206 (ObjNum 2046 snap 0 seq_num 2046) dirty exists 2020-06-12T02:37:41.097 INFO:tasks.rados.rados.0.smithi107.stdout:2047: left oid 2 (ObjNum 2046 snap 0 seq_num 2046) 2020-06-12T02:37:41.098 INFO:tasks.rados.rados.0.smithi107.stdout:2049: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:41.098 INFO:tasks.rados.rados.0.smithi107.stdout:2049: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:41.099 INFO:tasks.rados.rados.0.smithi107.stdout:2049: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:41.099 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 364 (ObjNum 2048 snap 0 seq_num 2048) dirty exists 2020-06-12T02:37:41.099 INFO:tasks.rados.rados.0.smithi107.stdout:2049: left oid 16 (ObjNum 2048 snap 0 seq_num 2048) 2020-06-12T02:37:41.100 INFO:tasks.rados.rados.0.smithi107.stdout:2047: done (3 left) 2020-06-12T02:37:41.100 INFO:tasks.rados.rados.0.smithi107.stdout:2049: done (2 left) 2020-06-12T02:37:41.101 INFO:tasks.rados.rados.0.smithi107.stdout:2052: done (1 left) 2020-06-12T02:37:41.101 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:37:41.101 INFO:tasks.rados.rados.0.smithi107.stdout:2053: seq_num 2052 ranges {491520=770048} 2020-06-12T02:37:41.108 INFO:tasks.rados.rados.0.smithi107.stdout:2053: writing smithi10729407-6 from 491520 to 1261568 tid 1 2020-06-12T02:37:41.108 INFO:tasks.rados.rados.0.smithi107.stdout:2054: delete oid 12 current snap is 0 2020-06-12T02:37:41.108 INFO:tasks.rados.rados.0.smithi107.stdout:2051: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:41.108 INFO:tasks.rados.rados.0.smithi107.stdout:2051: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:41.109 INFO:tasks.rados.rados.0.smithi107.stdout:2051: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:41.109 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 352 (ObjNum 2050 snap 0 seq_num 2050) dirty exists 2020-06-12T02:37:41.109 INFO:tasks.rados.rados.0.smithi107.stdout:2051: left oid 9 (ObjNum 2050 snap 0 seq_num 2050) 2020-06-12T02:37:41.111 INFO:tasks.rados.rados.0.smithi107.stdout:2051: done (2 left) 2020-06-12T02:37:41.112 INFO:tasks.rados.rados.0.smithi107.stdout:2054: done (1 left) 2020-06-12T02:37:41.112 INFO:tasks.rados.rados.0.smithi107.stdout:2055: delete oid 11 current snap is 0 2020-06-12T02:37:41.114 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:41.115 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:41.115 INFO:tasks.rados.rados.0.smithi107.stdout:2053: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:41.116 INFO:tasks.rados.rados.0.smithi107.stdout:2055: done (1 left) 2020-06-12T02:37:41.116 INFO:tasks.rados.rados.0.smithi107.stdout:2056: delete oid 11 current snap is 0 2020-06-12T02:37:41.117 INFO:tasks.rados.rados.0.smithi107.stdout:2053: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:41.119 INFO:tasks.rados.rados.0.smithi107.stdout:2056: done (1 left) 2020-06-12T02:37:41.119 INFO:tasks.rados.rados.0.smithi107.stdout:2057: delete oid 20 current snap is 0 2020-06-12T02:37:41.119 INFO:tasks.rados.rados.0.smithi107.stdout:2053: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:41.120 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 425 (ObjNum 2052 snap 0 seq_num 2052) dirty exists 2020-06-12T02:37:41.120 INFO:tasks.rados.rados.0.smithi107.stdout:2053: left oid 6 (ObjNum 2052 snap 0 seq_num 2052) 2020-06-12T02:37:41.120 INFO:tasks.rados.rados.0.smithi107.stdout:2053: done (1 left) 2020-06-12T02:37:41.121 INFO:tasks.rados.rados.0.smithi107.stdout:2057: done (0 left) 2020-06-12T02:37:41.121 INFO:tasks.rados.rados.0.smithi107.stdout:2058: delete oid 11 current snap is 0 2020-06-12T02:37:41.124 INFO:tasks.rados.rados.0.smithi107.stdout:2058: done (0 left) 2020-06-12T02:37:41.124 INFO:tasks.rados.rados.0.smithi107.stdout:2059: delete oid 2 current snap is 0 2020-06-12T02:37:41.126 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-06-12T02:37:41.126 INFO:tasks.rados.rados.0.smithi107.stdout:2059: done (0 left) 2020-06-12T02:37:41.126 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:37:41.128 INFO:tasks.rados.rados.0.smithi107.stdout:2060: seq_num 2059 ranges {1261568=491520,1753088=401408} 2020-06-12T02:37:41.133 INFO:tasks.rados.rados.0.smithi107.stdout:2060: writing smithi10729407-6 from 1261568 to 1753088 tid 1 2020-06-12T02:37:41.154 INFO:tasks.rados.rados.0.smithi107.stdout:2060: writing smithi10729407-6 from 1753088 to 2154496 tid 2 2020-06-12T02:37:41.154 INFO:tasks.rados.rados.0.smithi107.stdout:2061: delete oid 19 current snap is 0 2020-06-12T02:37:41.155 INFO:tasks.rados.rados.0.smithi107.stdout:2061: done (1 left) 2020-06-12T02:37:41.155 INFO:tasks.rados.rados.0.smithi107.stdout:2062: delete oid 5 current snap is 0 2020-06-12T02:37:41.155 INFO:tasks.rados.rados.0.smithi107.stdout:2062: done (1 left) 2020-06-12T02:37:41.155 INFO:tasks.rados.rados.0.smithi107.stdout:2063: delete oid 7 current snap is 0 2020-06-12T02:37:41.155 INFO:tasks.rados.rados.0.smithi107.stdout:2060: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:41.156 INFO:tasks.rados.rados.0.smithi107.stdout:2060: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:41.156 INFO:tasks.rados.rados.0.smithi107.stdout:2063: done (1 left) 2020-06-12T02:37:41.156 INFO:tasks.rados.rados.0.smithi107.stdout:2064: delete oid 11 current snap is 0 2020-06-12T02:37:41.156 INFO:tasks.rados.rados.0.smithi107.stdout:2060: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:41.156 INFO:tasks.rados.rados.0.smithi107.stdout:2060: finishing write tid 5 to smithi10729407-6 2020-06-12T02:37:41.156 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 428 (ObjNum 2059 snap 0 seq_num 2059) dirty exists 2020-06-12T02:37:41.157 INFO:tasks.rados.rados.0.smithi107.stdout:2060: left oid 6 (ObjNum 2059 snap 0 seq_num 2059) 2020-06-12T02:37:41.157 INFO:tasks.rados.rados.0.smithi107.stdout:2060: done (1 left) 2020-06-12T02:37:41.157 INFO:tasks.rados.rados.0.smithi107.stdout:2064: done (0 left) 2020-06-12T02:37:41.157 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:37:41.157 INFO:tasks.rados.rados.0.smithi107.stdout:2065: seq_num 2064 ranges {3235840=573440,3809280=401408} 2020-06-12T02:37:41.158 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:37:41.150+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:37:41.170 INFO:tasks.rados.rados.0.smithi107.stdout:2065: writing smithi10729407-1 from 3235840 to 3809280 tid 1 2020-06-12T02:37:41.172 INFO:tasks.rados.rados.0.smithi107.stdout:2065: writing smithi10729407-1 from 3809280 to 4210688 tid 2 2020-06-12T02:37:41.172 INFO:tasks.rados.rados.0.smithi107.stdout:2066: delete oid 12 current snap is 0 2020-06-12T02:37:41.179 INFO:tasks.rados.rados.0.smithi107.stdout:2066: done (1 left) 2020-06-12T02:37:41.180 INFO:tasks.rados.rados.0.smithi107.stdout:2067: delete oid 19 current snap is 0 2020-06-12T02:37:41.180 INFO:tasks.rados.rados.0.smithi107.stdout:2067: done (1 left) 2020-06-12T02:37:41.180 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:37:41.180 INFO:tasks.rados.rados.0.smithi107.stdout:2068: seq_num 2067 ranges {368640=622592,991232=401408} 2020-06-12T02:37:41.181 INFO:tasks.rados.rados.0.smithi107.stdout:2068: writing smithi10729407-10 from 368640 to 991232 tid 1 2020-06-12T02:37:41.184 INFO:tasks.rados.rados.0.smithi107.stdout:2068: writing smithi10729407-10 from 991232 to 1392640 tid 2 2020-06-12T02:37:41.184 INFO:tasks.rados.rados.0.smithi107.stdout:2069: delete oid 12 current snap is 0 2020-06-12T02:37:41.185 INFO:tasks.rados.rados.0.smithi107.stdout:2065: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:41.185 INFO:tasks.rados.rados.0.smithi107.stdout:2065: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:41.185 INFO:tasks.rados.rados.0.smithi107.stdout:2065: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:41.185 INFO:tasks.rados.rados.0.smithi107.stdout:2065: finishing write tid 5 to smithi10729407-1 2020-06-12T02:37:41.185 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 431 (ObjNum 2064 snap 0 seq_num 2064) dirty exists 2020-06-12T02:37:41.186 INFO:tasks.rados.rados.0.smithi107.stdout:2065: left oid 1 (ObjNum 2064 snap 0 seq_num 2064) 2020-06-12T02:37:41.186 INFO:tasks.rados.rados.0.smithi107.stdout:2065: done (2 left) 2020-06-12T02:37:41.186 INFO:tasks.rados.rados.0.smithi107.stdout:2069: done (1 left) 2020-06-12T02:37:41.187 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:41.187 INFO:tasks.rados.rados.0.smithi107.stdout:2070: seq_num 2069 ranges {0=655360,655360=401408} 2020-06-12T02:37:41.190 INFO:tasks.rados.rados.0.smithi107.stdout:2070: writing smithi10729407-2 from 0 to 655360 tid 1 2020-06-12T02:37:41.193 INFO:tasks.rados.rados.0.smithi107.stdout:2070: writing smithi10729407-2 from 655360 to 1056768 tid 2 2020-06-12T02:37:41.194 INFO:tasks.rados.rados.0.smithi107.stdout:2071: delete oid 7 current snap is 0 2020-06-12T02:37:41.194 INFO:tasks.rados.rados.0.smithi107.stdout:2068: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:41.194 INFO:tasks.rados.rados.0.smithi107.stdout:2068: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:41.194 INFO:tasks.rados.rados.0.smithi107.stdout:2068: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:41.194 INFO:tasks.rados.rados.0.smithi107.stdout:2068: finishing write tid 5 to smithi10729407-10 2020-06-12T02:37:41.195 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 192 (ObjNum 2067 snap 0 seq_num 2067) dirty exists 2020-06-12T02:37:41.195 INFO:tasks.rados.rados.0.smithi107.stdout:2068: left oid 10 (ObjNum 2067 snap 0 seq_num 2067) 2020-06-12T02:37:41.199 INFO:tasks.rados.rados.0.smithi107.stdout:2070: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:41.199 INFO:tasks.rados.rados.0.smithi107.stdout:2068: done (2 left) 2020-06-12T02:37:41.200 INFO:tasks.rados.rados.0.smithi107.stdout:2071: done (1 left) 2020-06-12T02:37:41.200 INFO:tasks.rados.rados.0.smithi107.stdout:2072: delete oid 7 current snap is 0 2020-06-12T02:37:41.204 INFO:tasks.rados.rados.0.smithi107.stdout:2072: done (1 left) 2020-06-12T02:37:41.205 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:37:41.205 INFO:tasks.rados.rados.0.smithi107.stdout:2073: seq_num 2072 ranges {1392640=704512,2097152=401408} 2020-06-12T02:37:41.214 INFO:tasks.rados.rados.0.smithi107.stdout:2073: writing smithi10729407-9 from 1392640 to 2097152 tid 1 2020-06-12T02:37:41.216 INFO:tasks.rados.rados.0.smithi107.stdout:2073: writing smithi10729407-9 from 2097152 to 2498560 tid 2 2020-06-12T02:37:41.217 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:37:41.217 INFO:tasks.rados.rados.0.smithi107.stdout:2074: seq_num 2073 ranges {0=729088,729088=401408} 2020-06-12T02:37:41.221 INFO:tasks.rados.rados.0.smithi107.stdout:2074: writing smithi10729407-20 from 0 to 729088 tid 1 2020-06-12T02:37:41.224 INFO:tasks.rados.rados.0.smithi107.stdout:2074: writing smithi10729407-20 from 729088 to 1130496 tid 2 2020-06-12T02:37:41.224 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:37:41.225 INFO:tasks.rados.rados.0.smithi107.stdout:2075: seq_num 2074 ranges {0=745472,745472=401408} 2020-06-12T02:37:41.229 INFO:tasks.rados.rados.0.smithi107.stdout:2075: writing smithi10729407-3 from 0 to 745472 tid 1 2020-06-12T02:37:41.232 INFO:tasks.rados.rados.0.smithi107.stdout:2075: writing smithi10729407-3 from 745472 to 1146880 tid 2 2020-06-12T02:37:41.232 INFO:tasks.rados.rados.0.smithi107.stdout:2076: delete oid 4 current snap is 0 2020-06-12T02:37:41.233 INFO:tasks.rados.rados.0.smithi107.stdout:2070: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:41.233 INFO:tasks.rados.rados.0.smithi107.stdout:2070: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:41.233 INFO:tasks.rados.rados.0.smithi107.stdout:2070: finishing write tid 5 to smithi10729407-2 2020-06-12T02:37:41.233 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 210 (ObjNum 2069 snap 0 seq_num 2069) dirty exists 2020-06-12T02:37:41.233 INFO:tasks.rados.rados.0.smithi107.stdout:2070: left oid 2 (ObjNum 2069 snap 0 seq_num 2069) 2020-06-12T02:37:41.234 INFO:tasks.rados.rados.0.smithi107.stdout:2073: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:41.234 INFO:tasks.rados.rados.0.smithi107.stdout:2073: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:41.234 INFO:tasks.rados.rados.0.smithi107.stdout:2073: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:41.234 INFO:tasks.rados.rados.0.smithi107.stdout:2074: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:41.235 INFO:tasks.rados.rados.0.smithi107.stdout:2073: finishing write tid 5 to smithi10729407-9 2020-06-12T02:37:41.235 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 355 (ObjNum 2072 snap 0 seq_num 2072) dirty exists 2020-06-12T02:37:41.235 INFO:tasks.rados.rados.0.smithi107.stdout:2073: left oid 9 (ObjNum 2072 snap 0 seq_num 2072) 2020-06-12T02:37:41.235 INFO:tasks.rados.rados.0.smithi107.stdout:2074: finishing write tid 2 to smithi10729407-20 2020-06-12T02:37:41.236 INFO:tasks.rados.rados.0.smithi107.stdout:2074: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:41.236 INFO:tasks.rados.rados.0.smithi107.stdout:2070: done (4 left) 2020-06-12T02:37:41.237 INFO:tasks.rados.rados.0.smithi107.stdout:2073: done (3 left) 2020-06-12T02:37:41.237 INFO:tasks.rados.rados.0.smithi107.stdout:2076: done (2 left) 2020-06-12T02:37:41.237 INFO:tasks.rados.rados.0.smithi107.stdout:2077: delete oid 14 current snap is 0 2020-06-12T02:37:41.237 INFO:tasks.rados.rados.0.smithi107.stdout:2075: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:41.238 INFO:tasks.rados.rados.0.smithi107.stdout:2074: finishing write tid 5 to smithi10729407-20 2020-06-12T02:37:41.238 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 143 (ObjNum 2073 snap 0 seq_num 2073) dirty exists 2020-06-12T02:37:41.238 INFO:tasks.rados.rados.0.smithi107.stdout:2074: left oid 20 (ObjNum 2073 snap 0 seq_num 2073) 2020-06-12T02:37:41.240 INFO:tasks.rados.rados.0.smithi107.stdout:2074: done (2 left) 2020-06-12T02:37:41.240 INFO:tasks.rados.rados.0.smithi107.stdout:2077: done (1 left) 2020-06-12T02:37:41.240 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:37:41.240 INFO:tasks.rados.rados.0.smithi107.stdout:2078: seq_num 2077 ranges {2121728=794624,2916352=401408} 2020-06-12T02:37:41.254 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:41.252+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:37:41.255 INFO:tasks.rados.rados.0.smithi107.stdout:2078: writing smithi10729407-17 from 2121728 to 2916352 tid 1 2020-06-12T02:37:41.260 INFO:tasks.rados.rados.0.smithi107.stdout:2078: writing smithi10729407-17 from 2916352 to 3317760 tid 2 2020-06-12T02:37:41.260 INFO:tasks.rados.rados.0.smithi107.stdout:2079: delete oid 11 current snap is 0 2020-06-12T02:37:41.261 INFO:tasks.rados.rados.0.smithi107.stdout:2075: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:41.261 INFO:tasks.rados.rados.0.smithi107.stdout:2075: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:41.261 INFO:tasks.rados.rados.0.smithi107.stdout:2075: finishing write tid 5 to smithi10729407-3 2020-06-12T02:37:41.261 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 367 (ObjNum 2074 snap 0 seq_num 2074) dirty exists 2020-06-12T02:37:41.261 INFO:tasks.rados.rados.0.smithi107.stdout:2075: left oid 3 (ObjNum 2074 snap 0 seq_num 2074) 2020-06-12T02:37:41.262 INFO:tasks.rados.rados.0.smithi107.stdout:2075: done (2 left) 2020-06-12T02:37:41.262 INFO:tasks.rados.rados.0.smithi107.stdout:2079: done (1 left) 2020-06-12T02:37:41.262 INFO:tasks.rados.rados.0.smithi107.stdout:2080: delete oid 5 current snap is 0 2020-06-12T02:37:41.263 INFO:tasks.rados.rados.0.smithi107.stdout:2080: done (1 left) 2020-06-12T02:37:41.263 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:37:41.263 INFO:tasks.rados.rados.0.smithi107.stdout:2081: seq_num 2080 ranges {0=442368,442368=581632,1024000=221184} 2020-06-12T02:37:41.264 INFO:tasks.rados.rados.0.smithi107.stdout:2081: writing smithi10729407-18 from 0 to 442368 tid 1 2020-06-12T02:37:41.268 INFO:tasks.rados.rados.0.smithi107.stdout:2081: writing smithi10729407-18 from 442368 to 1024000 tid 2 2020-06-12T02:37:41.270 INFO:tasks.rados.rados.0.smithi107.stdout:2081: writing smithi10729407-18 from 1024000 to 1245184 tid 3 2020-06-12T02:37:41.271 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:37:41.271 INFO:tasks.rados.rados.0.smithi107.stdout:2082: seq_num 2081 ranges {0=458752,458752=466944,925696=335872} 2020-06-12T02:37:41.273 INFO:tasks.rados.rados.0.smithi107.stdout:2082: writing smithi10729407-4 from 0 to 458752 tid 1 2020-06-12T02:37:41.281 INFO:tasks.rados.rados.0.smithi107.stdout:2082: writing smithi10729407-4 from 458752 to 925696 tid 2 2020-06-12T02:37:41.281 INFO:tasks.rados.rados.0.smithi107.stdout:2082: writing smithi10729407-4 from 925696 to 1261568 tid 3 2020-06-12T02:37:41.282 INFO:tasks.rados.rados.0.smithi107.stdout:2083: delete oid 13 current snap is 0 2020-06-12T02:37:41.282 INFO:tasks.rados.rados.0.smithi107.stdout:2078: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:41.283 INFO:tasks.rados.rados.0.smithi107.stdout:2078: finishing write tid 2 to smithi10729407-17 2020-06-12T02:37:41.283 INFO:tasks.rados.rados.0.smithi107.stdout:2078: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:41.283 INFO:tasks.rados.rados.0.smithi107.stdout:2081: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:41.284 INFO:tasks.rados.rados.0.smithi107.stdout:2078: finishing write tid 5 to smithi10729407-17 2020-06-12T02:37:41.284 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 395 (ObjNum 2077 snap 0 seq_num 2077) dirty exists 2020-06-12T02:37:41.284 INFO:tasks.rados.rados.0.smithi107.stdout:2078: left oid 17 (ObjNum 2077 snap 0 seq_num 2077) 2020-06-12T02:37:41.285 INFO:tasks.rados.rados.0.smithi107.stdout:2081: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:41.285 INFO:tasks.rados.rados.0.smithi107.stdout:2081: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:41.285 INFO:tasks.rados.rados.0.smithi107.stdout:2081: finishing write tid 5 to smithi10729407-18 2020-06-12T02:37:41.286 INFO:tasks.rados.rados.0.smithi107.stdout:2082: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:41.286 INFO:tasks.rados.rados.0.smithi107.stdout:2078: done (3 left) 2020-06-12T02:37:41.286 INFO:tasks.rados.rados.0.smithi107.stdout:2083: done (2 left) 2020-06-12T02:37:41.287 INFO:tasks.rados.rados.0.smithi107.stdout:2084: delete oid 8 current snap is 0 2020-06-12T02:37:41.288 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:41.289 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:41.289 INFO:tasks.rados.rados.0.smithi107.stdout:2081: finishing write tid 6 to smithi10729407-18 2020-06-12T02:37:41.290 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 359 (ObjNum 2080 snap 0 seq_num 2080) dirty exists 2020-06-12T02:37:41.290 INFO:tasks.rados.rados.0.smithi107.stdout:2081: left oid 18 (ObjNum 2080 snap 0 seq_num 2080) 2020-06-12T02:37:41.291 INFO:tasks.rados.rados.0.smithi107.stdout:2081: done (2 left) 2020-06-12T02:37:41.292 INFO:tasks.rados.rados.0.smithi107.stdout:2084: done (1 left) 2020-06-12T02:37:41.292 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:37:41.292 INFO:tasks.rados.rados.0.smithi107.stdout:2085: seq_num 2084 ranges {4210688=507904,4718592=565248,5283840=237568} 2020-06-12T02:37:41.294 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-06-12T02:37:41.309 INFO:tasks.rados.rados.0.smithi107.stdout:2085: writing smithi10729407-1 from 4210688 to 4718592 tid 1 2020-06-12T02:37:41.316 INFO:tasks.rados.rados.0.smithi107.stdout:2085: writing smithi10729407-1 from 4718592 to 5283840 tid 2 2020-06-12T02:37:41.317 INFO:tasks.rados.rados.0.smithi107.stdout:2085: writing smithi10729407-1 from 5283840 to 5521408 tid 3 2020-06-12T02:37:41.317 INFO:tasks.rados.rados.0.smithi107.stdout:2086: delete oid 7 current snap is 0 2020-06-12T02:37:41.318 INFO:tasks.rados.rados.0.smithi107.stdout:2082: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:41.318 INFO:tasks.rados.rados.0.smithi107.stdout:2082: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:41.318 INFO:tasks.rados.rados.0.smithi107.stdout:2082: finishing write tid 5 to smithi10729407-4 2020-06-12T02:37:41.322 INFO:tasks.rados.rados.0.smithi107.stdout:2082: finishing write tid 6 to smithi10729407-4 2020-06-12T02:37:41.322 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 399 (ObjNum 2081 snap 0 seq_num 2081) dirty exists 2020-06-12T02:37:41.322 INFO:tasks.rados.rados.0.smithi107.stdout:2082: left oid 4 (ObjNum 2081 snap 0 seq_num 2081) 2020-06-12T02:37:41.322 INFO:tasks.rados.rados.0.smithi107.stdout:2085: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:41.323 INFO:tasks.rados.rados.0.smithi107.stdout:2082: done (2 left) 2020-06-12T02:37:41.323 INFO:tasks.rados.rados.0.smithi107.stdout:2086: done (1 left) 2020-06-12T02:37:41.323 INFO:tasks.rados.rados.0.smithi107.stdout:2087: delete oid 6 current snap is 0 2020-06-12T02:37:41.324 INFO:tasks.rados.rados.0.smithi107.stdout:2085: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:41.325 INFO:tasks.rados.rados.0.smithi107.stdout:2085: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:41.328 INFO:tasks.rados.rados.0.smithi107.stdout:2085: finishing write tid 5 to smithi10729407-1 2020-06-12T02:37:41.329 INFO:tasks.rados.rados.0.smithi107.stdout:2085: finishing write tid 6 to smithi10729407-1 2020-06-12T02:37:41.330 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 435 (ObjNum 2084 snap 0 seq_num 2084) dirty exists 2020-06-12T02:37:41.330 INFO:tasks.rados.rados.0.smithi107.stdout:2085: left oid 1 (ObjNum 2084 snap 0 seq_num 2084) 2020-06-12T02:37:41.330 INFO:tasks.rados.rados.0.smithi107.stdout:2085: done (1 left) 2020-06-12T02:37:41.330 INFO:tasks.rados.rados.0.smithi107.stdout:2087: done (0 left) 2020-06-12T02:37:41.330 INFO:tasks.rados.rados.0.smithi107.stdout:2088: delete oid 12 current snap is 0 2020-06-12T02:37:41.331 INFO:tasks.rados.rados.0.smithi107.stdout:2088: done (0 left) 2020-06-12T02:37:41.331 INFO:tasks.rados.rados.0.smithi107.stdout:2089: delete oid 5 current snap is 0 2020-06-12T02:37:41.332 INFO:tasks.rados.rados.0.smithi107.stdout:2089: done (0 left) 2020-06-12T02:37:41.333 INFO:tasks.rados.rados.0.smithi107.stdout:2090: delete oid 15 current snap is 0 2020-06-12T02:37:41.334 INFO:tasks.rados.rados.0.smithi107.stdout:2090: done (0 left) 2020-06-12T02:37:41.334 INFO:tasks.rados.rados.0.smithi107.stdout:2091: delete oid 7 current snap is 0 2020-06-12T02:37:41.337 INFO:tasks.rados.rados.0.smithi107.stdout:2091: done (0 left) 2020-06-12T02:37:41.337 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:37:41.338 INFO:tasks.rados.rados.0.smithi107.stdout:2092: seq_num 2091 ranges {0=622592,622592=606208,1228800=196608} 2020-06-12T02:37:41.340 INFO:tasks.rados.rados.0.smithi107.stdout:2092: writing smithi10729407-15 from 0 to 622592 tid 1 2020-06-12T02:37:41.344 INFO:tasks.rados.rados.0.smithi107.stdout:2092: writing smithi10729407-15 from 622592 to 1228800 tid 2 2020-06-12T02:37:41.345 INFO:tasks.rados.rados.0.smithi107.stdout:2092: writing smithi10729407-15 from 1228800 to 1425408 tid 3 2020-06-12T02:37:41.345 INFO:tasks.rados.rados.0.smithi107.stdout:2093: delete oid 10 current snap is 0 2020-06-12T02:37:41.348 INFO:tasks.rados.rados.0.smithi107.stdout:2093: done (1 left) 2020-06-12T02:37:41.348 INFO:tasks.rados.rados.0.smithi107.stdout:2094: delete oid 16 current snap is 0 2020-06-12T02:37:41.350 INFO:tasks.rados.rados.0.smithi107.stdout:2092: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:41.351 INFO:tasks.rados.rados.0.smithi107.stdout:2094: done (1 left) 2020-06-12T02:37:41.351 INFO:tasks.rados.rados.0.smithi107.stdout:2095: delete oid 2 current snap is 0 2020-06-12T02:37:41.352 INFO:tasks.rados.rados.0.smithi107.stdout:2092: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:41.352 INFO:tasks.rados.rados.0.smithi107.stdout:2092: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:41.353 INFO:tasks.rados.rados.0.smithi107.stdout:2092: finishing write tid 5 to smithi10729407-15 2020-06-12T02:37:41.355 INFO:tasks.rados.rados.0.smithi107.stdout:2092: finishing write tid 6 to smithi10729407-15 2020-06-12T02:37:41.355 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 355 (ObjNum 2091 snap 0 seq_num 2091) dirty exists 2020-06-12T02:37:41.355 INFO:tasks.rados.rados.0.smithi107.stdout:2092: left oid 15 (ObjNum 2091 snap 0 seq_num 2091) 2020-06-12T02:37:41.356 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:37:41.352+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:37:41.356 INFO:tasks.rados.rados.0.smithi107.stdout:2092: done (1 left) 2020-06-12T02:37:41.356 INFO:tasks.rados.rados.0.smithi107.stdout:2095: done (0 left) 2020-06-12T02:37:41.356 INFO:tasks.rados.rados.0.smithi107.stdout:2096: delete oid 8 current snap is 0 2020-06-12T02:37:41.361 INFO:tasks.rados.rados.0.smithi107.stdout:2096: done (0 left) 2020-06-12T02:37:41.362 INFO:tasks.rados.rados.0.smithi107.stdout:2097: delete oid 3 current snap is 0 2020-06-12T02:37:41.372 INFO:tasks.rados.rados.0.smithi107.stdout:2097: done (0 left) 2020-06-12T02:37:41.372 INFO:tasks.rados.rados.0.smithi107.stdout:2098: delete oid 1 current snap is 0 2020-06-12T02:37:41.373 INFO:teuthology.orchestra.run.smithi107.stderr:set osd.5 primary-affinity to 0 (802) 2020-06-12T02:37:41.437 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:41.437 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:41.445 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-06-12T02:37:41.456 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:37:41.453+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:37:41.484 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd unset noscrub 2020-06-12T02:37:41.518 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-06-12T02:37:41.574 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:37:41.571+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:37:41.577 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:41.577 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:41.589 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-06-12T02:37:41.673 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:37:41.670+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:37:41.684 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:37:41.707 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:37:41.727 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:37:41.747 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:37:41.760 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:37:41.757+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:37:41.768 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:37:41.772 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:37:41.772 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2020-06-12T02:37:41.773 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2020-06-12T02:37:41.773 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2020-06-12T02:37:41.773 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.773 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s2 25/21 ECSubWrite(tid=277, reqid=client.4512.0:1102, at_version=25'61, trim_to=0'0, roll_forward_to=25'60))", 2020-06-12T02:37:41.778 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:32.593596+0000", 2020-06-12T02:37:41.778 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 9.1745225670000004, 2020-06-12T02:37:41.778 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0072991619999999997, 2020-06-12T02:37:41.778 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:41.779 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:41.779 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:41.779 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.779 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:41.779 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.593596+0000", 2020-06-12T02:37:41.792 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:41.792 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.793 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.793 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:41.793 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.593596+0000", 2020-06-12T02:37:41.793 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054472400000000001 2020-06-12T02:37:41.793 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.794 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.794 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:41.794 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.594140+0000", 2020-06-12T02:37:41.794 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5156e-05 2020-06-12T02:37:41.795 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.795 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.795 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:41.795 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.594155+0000", 2020-06-12T02:37:41.795 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8709999999999992e-06 2020-06-12T02:37:41.795 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.796 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.796 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:41.796 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.594165+0000", 2020-06-12T02:37:41.796 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3549999999999998e-06 2020-06-12T02:37:41.796 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.797 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.797 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:41.797 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.594170+0000", 2020-06-12T02:37:41.797 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6307999999999998e-05 2020-06-12T02:37:41.797 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.797 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.798 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:41.798 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.594216+0000", 2020-06-12T02:37:41.798 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1255e-05 2020-06-12T02:37:41.798 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.798 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.799 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:41.799 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.594227+0000", 2020-06-12T02:37:41.799 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0066139500000000004 2020-06-12T02:37:41.799 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.799 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.800 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:41.800 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.600841+0000", 2020-06-12T02:37:41.800 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3542999999999999e-05 2020-06-12T02:37:41.800 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.802 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.803 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:41.803 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.600895+0000", 2020-06-12T02:37:41.803 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0072991619999999997 2020-06-12T02:37:41.804 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:41.804 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:41.804 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:41.805 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.805 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.805 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.9s1 25/21 ECSubWrite(tid=615, reqid=client.4512.0:1285, at_version=25'85, trim_to=0'0, roll_forward_to=25'84))", 2020-06-12T02:37:41.808 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.117566+0000", 2020-06-12T02:37:41.808 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 8.6505524999999999, 2020-06-12T02:37:41.808 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010635947999999999, 2020-06-12T02:37:41.809 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:41.809 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:41.809 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:41.810 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.810 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:41.810 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.117566+0000", 2020-06-12T02:37:41.810 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:41.811 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.811 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.812 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:41.812 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.117566+0000", 2020-06-12T02:37:41.812 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000451781 2020-06-12T02:37:41.813 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.813 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.813 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:41.813 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.118017+0000", 2020-06-12T02:37:41.814 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9958999999999999e-05 2020-06-12T02:37:41.814 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.814 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.815 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:41.815 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.118047+0000", 2020-06-12T02:37:41.815 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.694e-05 2020-06-12T02:37:41.815 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.816 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.816 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:41.816 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.118064+0000", 2020-06-12T02:37:41.816 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7629999999999995e-06 2020-06-12T02:37:41.817 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.817 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.817 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:41.818 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.118074+0000", 2020-06-12T02:37:41.818 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1209000000000003e-05 2020-06-12T02:37:41.818 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.819 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.820 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:41.820 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.118135+0000", 2020-06-12T02:37:41.820 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6164999999999999e-05 2020-06-12T02:37:41.820 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.821 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.821 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:41.821 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.118151+0000", 2020-06-12T02:37:41.821 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0099907599999999996 2020-06-12T02:37:41.822 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.822 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.822 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:41.823 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.128142+0000", 2020-06-12T02:37:41.823 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9370999999999998e-05 2020-06-12T02:37:41.823 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.823 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.823 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:41.823 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.128202+0000", 2020-06-12T02:37:41.824 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010635947999999999 2020-06-12T02:37:41.824 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:41.824 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:41.824 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:41.824 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.825 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.825 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.9s1 25/21 ECSubWrite(tid=616, reqid=client.4512.0:1286, at_version=25'86, trim_to=0'0, roll_forward_to=25'84))", 2020-06-12T02:37:41.825 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.120870+0000", 2020-06-12T02:37:41.825 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 8.6472483310000001, 2020-06-12T02:37:41.825 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0088027680000000007, 2020-06-12T02:37:41.826 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:41.826 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:41.826 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:41.826 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.826 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:41.827 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.120870+0000", 2020-06-12T02:37:41.827 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:41.827 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.827 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.827 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:41.827 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.120870+0000", 2020-06-12T02:37:41.828 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041078300000000001 2020-06-12T02:37:41.828 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.828 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.828 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:41.828 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.121281+0000", 2020-06-12T02:37:41.829 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3048e-05 2020-06-12T02:37:41.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.829 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.829 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:41.829 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.121304+0000", 2020-06-12T02:37:41.830 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9318e-05 2020-06-12T02:37:41.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.830 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.830 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:41.831 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.121323+0000", 2020-06-12T02:37:41.831 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2228999999999999e-05 2020-06-12T02:37:41.831 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.831 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.831 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:41.832 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.121335+0000", 2020-06-12T02:37:41.832 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0010068500000000001 2020-06-12T02:37:41.832 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.836 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.837 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:41.837 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.122342+0000", 2020-06-12T02:37:41.837 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2475e-05 2020-06-12T02:37:41.838 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.838 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.838 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:41.838 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.122354+0000", 2020-06-12T02:37:41.839 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.007283884 2020-06-12T02:37:41.839 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.839 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.840 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:41.840 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.129638+0000", 2020-06-12T02:37:41.840 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4180999999999997e-05 2020-06-12T02:37:41.841 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.841 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.841 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:41.841 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.129673+0000", 2020-06-12T02:37:41.842 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0088027680000000007 2020-06-12T02:37:41.842 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:41.842 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:41.843 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:41.843 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.843 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.844 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s2 26/21 ECSubWrite(tid=408, reqid=client.4512.0:1572, at_version=26'99, trim_to=0'0, roll_forward_to=26'97))", 2020-06-12T02:37:41.844 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.887897+0000", 2020-06-12T02:37:41.844 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 7.8802211389999997, 2020-06-12T02:37:41.845 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0098940590000000002, 2020-06-12T02:37:41.845 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:41.846 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:41.846 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:41.846 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.847 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:41.847 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.887897+0000", 2020-06-12T02:37:41.847 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:41.848 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.848 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.848 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:41.849 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.887897+0000", 2020-06-12T02:37:41.849 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052976499999999997 2020-06-12T02:37:41.849 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.850 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.850 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:41.850 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888427+0000", 2020-06-12T02:37:41.850 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1276000000000001e-05 2020-06-12T02:37:41.851 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.851 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.851 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:41.852 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888448+0000", 2020-06-12T02:37:41.852 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7121000000000001e-05 2020-06-12T02:37:41.852 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.853 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.853 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:41.853 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888465+0000", 2020-06-12T02:37:41.853 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0490000000000004e-06 2020-06-12T02:37:41.854 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.854 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.854 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:41.854 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888473+0000", 2020-06-12T02:37:41.854 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6770999999999997e-05 2020-06-12T02:37:41.855 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.855 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.855 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:41.855 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888510+0000", 2020-06-12T02:37:41.855 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7649999999999993e-06 2020-06-12T02:37:41.855 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.856 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.856 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:41.856 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888518+0000", 2020-06-12T02:37:41.856 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0092201669999999996 2020-06-12T02:37:41.856 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.857 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.857 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:41.857 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.897738+0000", 2020-06-12T02:37:41.857 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3145e-05 2020-06-12T02:37:41.857 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.858 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.858 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:41.858 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.897791+0000", 2020-06-12T02:37:41.858 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0098940590000000002 2020-06-12T02:37:41.858 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:41.858 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:41.859 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:41.859 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.859 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.859 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s2 26/21 ECSubWrite(tid=409, reqid=client.4512.0:1573, at_version=26'100, trim_to=0'0, roll_forward_to=26'98))", 2020-06-12T02:37:41.859 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.892374+0000", 2020-06-12T02:37:41.860 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 7.8757441379999999, 2020-06-12T02:37:41.860 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014071322000000001, 2020-06-12T02:37:41.860 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:41.860 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:41.860 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:41.861 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.865 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:41.865 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.892374+0000", 2020-06-12T02:37:41.866 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:41.866 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.866 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.866 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:41.866 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.892374+0000", 2020-06-12T02:37:41.867 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047864199999999999 2020-06-12T02:37:41.867 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.867 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.867 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:41.867 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.892853+0000", 2020-06-12T02:37:41.868 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5139000000000001e-05 2020-06-12T02:37:41.868 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.868 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.868 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:41.868 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.892878+0000", 2020-06-12T02:37:41.869 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1543999999999999e-05 2020-06-12T02:37:41.869 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.869 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.869 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:41.869 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.892899+0000", 2020-06-12T02:37:41.870 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8989999999999996e-06 2020-06-12T02:37:41.870 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.870 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.870 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:41.871 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.892909+0000", 2020-06-12T02:37:41.871 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00023009199999999999 2020-06-12T02:37:41.871 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.871 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.871 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:41.871 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.893139+0000", 2020-06-12T02:37:41.872 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0044000000000001e-05 2020-06-12T02:37:41.872 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.872 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.872 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:41.872 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.893149+0000", 2020-06-12T02:37:41.873 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013233405 2020-06-12T02:37:41.873 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.873 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.873 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:41.873 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.906383+0000", 2020-06-12T02:37:41.874 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2557000000000004e-05 2020-06-12T02:37:41.874 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.874 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.874 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:41.874 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.906445+0000", 2020-06-12T02:37:41.876 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014071322000000001 2020-06-12T02:37:41.877 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:41.877 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:41.877 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:41.877 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.877 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.878 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s2 26/21 ECSubWrite(tid=410, reqid=client.4512.0:1574, at_version=26'101, trim_to=0'0, roll_forward_to=26'98))", 2020-06-12T02:37:41.878 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.895487+0000", 2020-06-12T02:37:41.878 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 7.8726313430000001, 2020-06-12T02:37:41.878 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012904807000000001, 2020-06-12T02:37:41.878 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:41.879 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:41.879 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:41.879 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.879 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:41.880 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895487+0000", 2020-06-12T02:37:41.880 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:41.880 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.880 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.881 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:41.881 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895487+0000", 2020-06-12T02:37:41.881 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8478000000000002e-05 2020-06-12T02:37:41.881 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.882 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.882 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:41.882 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895575+0000", 2020-06-12T02:37:41.882 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8672e-05 2020-06-12T02:37:41.883 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.883 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:41.883 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895594+0000", 2020-06-12T02:37:41.884 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8028000000000001e-05 2020-06-12T02:37:41.884 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.885 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:41.885 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895612+0000", 2020-06-12T02:37:41.885 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2689999999999999e-06 2020-06-12T02:37:41.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:41.887 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895620+0000", 2020-06-12T02:37:41.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0038940289999999998 2020-06-12T02:37:41.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.888 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:41.888 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.899514+0000", 2020-06-12T02:37:41.889 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4370000000000001e-06 2020-06-12T02:37:41.889 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.889 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.889 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:41.889 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.899522+0000", 2020-06-12T02:37:41.890 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0088184019999999995 2020-06-12T02:37:41.890 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.890 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.890 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:41.890 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.908340+0000", 2020-06-12T02:37:41.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1492000000000001e-05 2020-06-12T02:37:41.891 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.891 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.891 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:41.891 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.908392+0000", 2020-06-12T02:37:41.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012904807000000001 2020-06-12T02:37:41.892 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:41.892 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:41.892 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:41.892 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.892 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.893 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s2 26/21 ECSubWrite(tid=411, reqid=client.4512.0:1575, at_version=26'102, trim_to=0'0, roll_forward_to=26'98))", 2020-06-12T02:37:41.893 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.899359+0000", 2020-06-12T02:37:41.893 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 7.868759152, 2020-06-12T02:37:41.894 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.009071862, 2020-06-12T02:37:41.894 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:41.894 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:41.894 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:41.894 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.894 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:41.895 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.899359+0000", 2020-06-12T02:37:41.895 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:41.895 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.895 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.896 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:41.896 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.899359+0000", 2020-06-12T02:37:41.896 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3011999999999999e-05 2020-06-12T02:37:41.896 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.896 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.897 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:41.897 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.899422+0000", 2020-06-12T02:37:41.897 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3204999999999999e-05 2020-06-12T02:37:41.897 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.897 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.898 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:41.898 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.899445+0000", 2020-06-12T02:37:41.898 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8564999999999999e-05 2020-06-12T02:37:41.898 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.898 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.899 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:41.899 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.899464+0000", 2020-06-12T02:37:41.899 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6259999999999997e-06 2020-06-12T02:37:41.899 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.899 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.900 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:41.900 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.899470+0000", 2020-06-12T02:37:41.900 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00096442999999999995 2020-06-12T02:37:41.900 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.900 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.900 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:41.901 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.900435+0000", 2020-06-12T02:37:41.901 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1058e-05 2020-06-12T02:37:41.901 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.901 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.901 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:41.902 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.900446+0000", 2020-06-12T02:37:41.902 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0079506830000000001 2020-06-12T02:37:41.902 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.902 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.902 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:41.903 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.908397+0000", 2020-06-12T02:37:41.903 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4282999999999997e-05 2020-06-12T02:37:41.903 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.903 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.904 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:41.904 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.908431+0000", 2020-06-12T02:37:41.904 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.009071862 2020-06-12T02:37:41.904 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:41.904 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:41.905 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:41.905 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.905 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.906 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 27/21 ECSubWrite(tid=169, reqid=client.4512.0:2720, at_version=27'109, trim_to=0'0, roll_forward_to=27'107))", 2020-06-12T02:37:41.906 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:36.649813+0000", 2020-06-12T02:37:41.906 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 5.1183055609999997, 2020-06-12T02:37:41.906 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0086910270000000005, 2020-06-12T02:37:41.906 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:41.907 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:41.907 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:41.907 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.907 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:41.907 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.649813+0000", 2020-06-12T02:37:41.907 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:41.908 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.908 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.908 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:41.908 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.649813+0000", 2020-06-12T02:37:41.908 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018473400000000001 2020-06-12T02:37:41.909 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.909 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.909 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:41.909 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.649997+0000", 2020-06-12T02:37:41.909 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0846999999999998e-05 2020-06-12T02:37:41.910 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.911 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.911 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:41.911 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.650028+0000", 2020-06-12T02:37:41.911 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9769999999999999e-05 2020-06-12T02:37:41.911 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.912 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.912 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:41.912 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.650048+0000", 2020-06-12T02:37:41.912 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8e-05 2020-06-12T02:37:41.912 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.912 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.913 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:41.913 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.650066+0000", 2020-06-12T02:37:41.913 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9342999999999997e-05 2020-06-12T02:37:41.913 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.914 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.914 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:41.914 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.650125+0000", 2020-06-12T02:37:41.914 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3171e-05 2020-06-12T02:37:41.914 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.915 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.915 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:41.915 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.650138+0000", 2020-06-12T02:37:41.915 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0082900739999999997 2020-06-12T02:37:41.916 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.916 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.916 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:41.916 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.658428+0000", 2020-06-12T02:37:41.917 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5087999999999997e-05 2020-06-12T02:37:41.917 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.917 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.917 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:41.917 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.658504+0000", 2020-06-12T02:37:41.918 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0086910270000000005 2020-06-12T02:37:41.918 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:41.918 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:41.918 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:41.918 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.919 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.919 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 27/21 ECSubWrite(tid=170, reqid=client.4512.0:2721, at_version=27'110, trim_to=0'0, roll_forward_to=27'107))", 2020-06-12T02:37:41.920 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:36.654148+0000", 2020-06-12T02:37:41.920 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 5.1139700100000001, 2020-06-12T02:37:41.921 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0096132549999999994, 2020-06-12T02:37:41.921 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:41.921 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:41.921 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:41.921 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.922 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:41.922 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.654148+0000", 2020-06-12T02:37:41.922 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:41.922 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.922 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.923 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:41.923 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.654148+0000", 2020-06-12T02:37:41.923 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00027898700000000002 2020-06-12T02:37:41.923 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.923 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.924 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:41.924 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.654427+0000", 2020-06-12T02:37:41.924 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.7942000000000001e-05 2020-06-12T02:37:41.924 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.924 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.925 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:41.925 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.654455+0000", 2020-06-12T02:37:41.925 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8269000000000001e-05 2020-06-12T02:37:41.925 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.925 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.926 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:41.926 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.654473+0000", 2020-06-12T02:37:41.926 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0450000000000003e-06 2020-06-12T02:37:41.926 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.926 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.927 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:41.927 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.654480+0000", 2020-06-12T02:37:41.927 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013121000000000001 2020-06-12T02:37:41.927 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.927 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.928 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:41.928 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.654612+0000", 2020-06-12T02:37:41.928 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.167e-05 2020-06-12T02:37:41.929 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.929 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.929 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:41.929 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.654623+0000", 2020-06-12T02:37:41.930 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090820680000000004 2020-06-12T02:37:41.930 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.930 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.930 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:41.931 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.663705+0000", 2020-06-12T02:37:41.931 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6063999999999997e-05 2020-06-12T02:37:41.931 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.932 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.932 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:41.932 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.663761+0000", 2020-06-12T02:37:41.932 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0096132549999999994 2020-06-12T02:37:41.933 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:41.933 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:41.933 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:41.933 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.934 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.934 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s2 27/21 ECSubWrite(tid=740, reqid=client.4512.0:2730, at_version=27'220, trim_to=0'0, roll_forward_to=27'217))", 2020-06-12T02:37:41.934 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:36.676961+0000", 2020-06-12T02:37:41.934 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 5.0911574120000003, 2020-06-12T02:37:41.934 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0088096930000000004, 2020-06-12T02:37:41.935 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:41.935 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:41.935 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:41.935 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.935 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:41.936 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.676961+0000", 2020-06-12T02:37:41.936 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:41.936 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.936 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.936 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:41.937 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.676961+0000", 2020-06-12T02:37:41.937 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00068605000000000001 2020-06-12T02:37:41.937 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.937 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.937 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:41.938 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.677647+0000", 2020-06-12T02:37:41.938 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0792e-05 2020-06-12T02:37:41.938 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.938 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.938 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:41.939 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.677668+0000", 2020-06-12T02:37:41.939 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0032999999999999e-05 2020-06-12T02:37:41.940 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.940 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.940 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:41.941 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.677688+0000", 2020-06-12T02:37:41.941 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8369999999999999e-06 2020-06-12T02:37:41.941 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.942 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.942 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:41.942 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.677694+0000", 2020-06-12T02:37:41.943 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001677563 2020-06-12T02:37:41.943 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.943 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.943 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:41.944 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.679372+0000", 2020-06-12T02:37:41.944 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0089999999999996e-06 2020-06-12T02:37:41.944 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.945 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.945 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:41.945 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.679381+0000", 2020-06-12T02:37:41.945 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0063373370000000002 2020-06-12T02:37:41.946 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.946 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.946 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:41.946 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.685718+0000", 2020-06-12T02:37:41.946 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2071999999999999e-05 2020-06-12T02:37:41.946 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.947 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.947 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:41.947 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.685770+0000", 2020-06-12T02:37:41.947 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0088096930000000004 2020-06-12T02:37:41.947 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:41.948 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:41.948 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:41.948 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.948 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.949 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 27/21 ECSubWrite(tid=179, reqid=client.4512.0:2781, at_version=0'0, trim_to=0'0, roll_forward_to=27'115))", 2020-06-12T02:37:41.950 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:36.810022+0000", 2020-06-12T02:37:41.950 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 4.9580956030000003, 2020-06-12T02:37:41.950 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093088630000000006, 2020-06-12T02:37:41.950 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:41.950 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:41.951 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:41.951 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.951 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:41.951 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.810022+0000", 2020-06-12T02:37:41.951 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:41.951 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.952 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.952 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:41.952 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.810022+0000", 2020-06-12T02:37:41.952 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0468000000000004e-05 2020-06-12T02:37:41.952 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.953 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.953 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:41.953 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.810093+0000", 2020-06-12T02:37:41.953 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6742999999999999e-05 2020-06-12T02:37:41.953 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.954 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.954 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:41.954 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.810110+0000", 2020-06-12T02:37:41.954 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8359000000000002e-05 2020-06-12T02:37:41.954 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.954 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.955 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:41.955 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.810128+0000", 2020-06-12T02:37:41.955 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8390000000000007e-06 2020-06-12T02:37:41.955 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.955 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.956 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:41.956 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.810136+0000", 2020-06-12T02:37:41.956 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004899558 2020-06-12T02:37:41.956 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.956 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.956 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:41.957 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.815035+0000", 2020-06-12T02:37:41.957 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7008e-05 2020-06-12T02:37:41.957 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.957 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.957 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:41.958 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.815052+0000", 2020-06-12T02:37:41.958 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0042246180000000003 2020-06-12T02:37:41.958 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.958 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.958 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:41.958 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.819277+0000", 2020-06-12T02:37:41.959 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.427e-05 2020-06-12T02:37:41.959 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.959 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.960 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:41.960 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.819331+0000", 2020-06-12T02:37:41.960 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093088630000000006 2020-06-12T02:37:41.960 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:41.961 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:41.961 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:41.961 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.961 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.961 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 27/21 ECSubWrite(tid=180, reqid=client.4512.0:2782, at_version=27'116, trim_to=0'0, roll_forward_to=27'115))", 2020-06-12T02:37:41.962 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:36.811166+0000", 2020-06-12T02:37:41.962 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 4.9569516379999996, 2020-06-12T02:37:41.962 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013435703, 2020-06-12T02:37:41.962 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:41.962 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:41.963 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:41.963 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.963 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:41.963 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.811166+0000", 2020-06-12T02:37:41.963 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:41.963 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.964 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.964 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:41.964 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.811166+0000", 2020-06-12T02:37:41.964 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015779300000000001 2020-06-12T02:37:41.964 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.965 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.965 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:41.965 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.811324+0000", 2020-06-12T02:37:41.965 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8149000000000001e-05 2020-06-12T02:37:41.965 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.965 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.966 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:41.966 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.811352+0000", 2020-06-12T02:37:41.966 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6880000000000001e-05 2020-06-12T02:37:41.966 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.966 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.967 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:41.967 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.811369+0000", 2020-06-12T02:37:41.967 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1099999999999999e-06 2020-06-12T02:37:41.967 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.967 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.967 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:41.968 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.811375+0000", 2020-06-12T02:37:41.968 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0038906349999999999 2020-06-12T02:37:41.968 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.968 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.968 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:41.969 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.815266+0000", 2020-06-12T02:37:41.969 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0536e-05 2020-06-12T02:37:41.969 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.969 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.969 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:41.970 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.815277+0000", 2020-06-12T02:37:41.970 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0092955610000000008 2020-06-12T02:37:41.971 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.971 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.971 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:41.971 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.824572+0000", 2020-06-12T02:37:41.971 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0039000000000002e-05 2020-06-12T02:37:41.972 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.972 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.972 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:41.972 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.824602+0000", 2020-06-12T02:37:41.972 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013435703 2020-06-12T02:37:41.972 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:41.973 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:41.973 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:41.973 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.973 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.973 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 27/21 ECSubWrite(tid=181, reqid=client.4512.0:2783, at_version=27'117, trim_to=0'0, roll_forward_to=27'115))", 2020-06-12T02:37:41.974 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:36.816115+0000", 2020-06-12T02:37:41.974 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 4.952003307, 2020-06-12T02:37:41.974 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085240260000000005, 2020-06-12T02:37:41.974 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:41.974 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:41.975 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:41.975 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.975 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:41.975 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.816115+0000", 2020-06-12T02:37:41.975 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:41.975 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.976 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.976 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:41.976 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.816115+0000", 2020-06-12T02:37:41.976 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011663900000000001 2020-06-12T02:37:41.976 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.976 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.977 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:41.977 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.816231+0000", 2020-06-12T02:37:41.977 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2373000000000001e-05 2020-06-12T02:37:41.977 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.977 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.978 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:41.978 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.816254+0000", 2020-06-12T02:37:41.978 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0010999999999999e-05 2020-06-12T02:37:41.978 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.978 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.979 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:41.979 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.816274+0000", 2020-06-12T02:37:41.979 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6769999999999998e-06 2020-06-12T02:37:41.979 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.979 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.979 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:41.980 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.816280+0000", 2020-06-12T02:37:41.980 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0031573920000000002 2020-06-12T02:37:41.980 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.981 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.981 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:41.981 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.819438+0000", 2020-06-12T02:37:41.981 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0868e-05 2020-06-12T02:37:41.981 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.982 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.982 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:41.982 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.819449+0000", 2020-06-12T02:37:41.982 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0051582829999999996 2020-06-12T02:37:41.982 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.983 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.983 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:41.983 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.824607+0000", 2020-06-12T02:37:41.983 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.1782999999999997e-05 2020-06-12T02:37:41.983 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.983 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.984 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:41.984 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.824639+0000", 2020-06-12T02:37:41.984 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085240260000000005 2020-06-12T02:37:41.984 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:41.984 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:41.985 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:41.985 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.985 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.985 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s2 27/21 ECSubWrite(tid=794, reqid=client.4512.0:2907, at_version=27'231, trim_to=0'0, roll_forward_to=27'229))", 2020-06-12T02:37:41.985 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:37.056587+0000", 2020-06-12T02:37:41.985 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 4.711530744, 2020-06-12T02:37:41.986 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0066184590000000001, 2020-06-12T02:37:41.986 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:41.986 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:41.986 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:41.986 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.987 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:41.987 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:37.056587+0000", 2020-06-12T02:37:41.987 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:41.987 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.987 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.987 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:41.988 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:37.056587+0000", 2020-06-12T02:37:41.988 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00064928200000000005 2020-06-12T02:37:41.988 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.988 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.988 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:41.989 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:37.057237+0000", 2020-06-12T02:37:41.989 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6503000000000001e-05 2020-06-12T02:37:41.989 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.989 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.996 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:41.996 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:37.057253+0000", 2020-06-12T02:37:41.996 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2261e-05 2020-06-12T02:37:41.996 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.996 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.997 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:41.997 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:37.057265+0000", 2020-06-12T02:37:41.997 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2429999999999996e-06 2020-06-12T02:37:41.997 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.997 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.997 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:41.998 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:37.057271+0000", 2020-06-12T02:37:41.998 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0887999999999999e-05 2020-06-12T02:37:41.998 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.998 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.998 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:41.999 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:37.057312+0000", 2020-06-12T02:37:41.999 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7590000000000003e-06 2020-06-12T02:37:41.999 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:41.999 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:41.999 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:42.000 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:37.057320+0000", 2020-06-12T02:37:42.000 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0058473309999999999 2020-06-12T02:37:42.000 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.000 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.000 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:42.000 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:37.063168+0000", 2020-06-12T02:37:42.001 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8192000000000003e-05 2020-06-12T02:37:42.001 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.001 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.001 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:42.001 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:37.063206+0000", 2020-06-12T02:37:42.002 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0066184590000000001 2020-06-12T02:37:42.002 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.002 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:42.002 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.003 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.003 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.003 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.9s1 27/21 ECSubWrite(tid=1423, reqid=client.4512.0:2966, at_version=27'220, trim_to=0'0, roll_forward_to=27'218))", 2020-06-12T02:37:42.003 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:37.712779+0000", 2020-06-12T02:37:42.003 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 4.0553390629999999, 2020-06-12T02:37:42.004 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0083213780000000008, 2020-06-12T02:37:42.004 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:42.004 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:42.004 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:42.004 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.004 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:42.005 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:37.712779+0000", 2020-06-12T02:37:42.005 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:42.005 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.005 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.006 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:42.006 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:37.712779+0000", 2020-06-12T02:37:42.006 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040324600000000001 2020-06-12T02:37:42.006 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.007 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.007 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:42.007 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:37.713182+0000", 2020-06-12T02:37:42.007 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3363999999999999e-05 2020-06-12T02:37:42.007 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.008 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.008 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:42.008 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:37.713206+0000", 2020-06-12T02:37:42.008 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7955e-05 2020-06-12T02:37:42.008 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.008 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.009 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:42.009 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:37.713224+0000", 2020-06-12T02:37:42.009 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9650000000000008e-06 2020-06-12T02:37:42.009 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.009 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.010 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:42.010 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:37.713232+0000", 2020-06-12T02:37:42.010 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010466 2020-06-12T02:37:42.010 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.010 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.010 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.011 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:37.713336+0000", 2020-06-12T02:37:42.011 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1079e-05 2020-06-12T02:37:42.011 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.011 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.011 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:42.012 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:37.713347+0000", 2020-06-12T02:37:42.012 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0076994960000000001 2020-06-12T02:37:42.012 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.012 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.012 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:42.012 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:37.721047+0000", 2020-06-12T02:37:42.013 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3613000000000003e-05 2020-06-12T02:37:42.013 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.013 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.013 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:42.013 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:37.721100+0000", 2020-06-12T02:37:42.014 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0083213780000000008 2020-06-12T02:37:42.014 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.014 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:42.014 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.014 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.015 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.015 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 27/21 ECSubWrite(tid=221, reqid=client.4512.0:3330, at_version=27'143, trim_to=0'0, roll_forward_to=27'141))", 2020-06-12T02:37:42.015 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:38.608388+0000", 2020-06-12T02:37:42.015 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.1597298700000001, 2020-06-12T02:37:42.015 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0067944019999999997, 2020-06-12T02:37:42.016 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:42.016 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:42.017 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:42.017 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.017 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:42.017 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.608388+0000", 2020-06-12T02:37:42.017 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:42.018 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.018 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.018 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:42.018 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.608388+0000", 2020-06-12T02:37:42.018 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012563799999999999 2020-06-12T02:37:42.018 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.019 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.019 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:42.019 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.608514+0000", 2020-06-12T02:37:42.019 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6772000000000001e-05 2020-06-12T02:37:42.019 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.020 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.020 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:42.020 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.608531+0000", 2020-06-12T02:37:42.020 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1313e-05 2020-06-12T02:37:42.020 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.021 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.021 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:42.021 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.608542+0000", 2020-06-12T02:37:42.021 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0049999999999998e-06 2020-06-12T02:37:42.022 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.022 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.022 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:42.022 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.608549+0000", 2020-06-12T02:37:42.022 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2689999999999999e-05 2020-06-12T02:37:42.022 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.023 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.023 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.023 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.608602+0000", 2020-06-12T02:37:42.023 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3369e-05 2020-06-12T02:37:42.023 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.024 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.024 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:42.024 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.608615+0000", 2020-06-12T02:37:42.024 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0065104819999999997 2020-06-12T02:37:42.024 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.024 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.025 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:42.025 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.615125+0000", 2020-06-12T02:37:42.025 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7132999999999997e-05 2020-06-12T02:37:42.025 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.025 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.026 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:42.026 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.615183+0000", 2020-06-12T02:37:42.026 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0067944019999999997 2020-06-12T02:37:42.026 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.027 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:42.027 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.027 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.027 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.028 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 27/21 ECSubWrite(tid=222, reqid=client.4512.0:3331, at_version=27'144, trim_to=0'0, roll_forward_to=27'141))", 2020-06-12T02:37:42.028 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:38.609919+0000", 2020-06-12T02:37:42.028 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.1581986729999998, 2020-06-12T02:37:42.028 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0096801079999999998, 2020-06-12T02:37:42.028 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:42.028 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:42.029 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:42.029 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.029 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:42.029 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.609919+0000", 2020-06-12T02:37:42.029 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:42.030 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.030 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.030 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:42.030 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.609919+0000", 2020-06-12T02:37:42.030 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00024651799999999999 2020-06-12T02:37:42.031 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.031 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.031 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:42.031 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.610166+0000", 2020-06-12T02:37:42.031 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5100999999999999e-05 2020-06-12T02:37:42.031 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.032 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.032 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:42.032 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.610191+0000", 2020-06-12T02:37:42.032 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3227e-05 2020-06-12T02:37:42.032 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.033 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.033 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:42.033 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.610214+0000", 2020-06-12T02:37:42.033 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8339999999999999e-06 2020-06-12T02:37:42.033 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.033 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.034 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:42.034 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.610222+0000", 2020-06-12T02:37:42.034 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013659169999999999 2020-06-12T02:37:42.034 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.034 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.035 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.035 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.611588+0000", 2020-06-12T02:37:42.035 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2160999999999999e-05 2020-06-12T02:37:42.035 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.035 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.036 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:42.036 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.611600+0000", 2020-06-12T02:37:42.036 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.007943795 2020-06-12T02:37:42.036 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.036 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.036 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:42.037 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.619544+0000", 2020-06-12T02:37:42.037 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5554999999999999e-05 2020-06-12T02:37:42.038 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.038 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.038 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:42.038 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.619600+0000", 2020-06-12T02:37:42.038 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0096801079999999998 2020-06-12T02:37:42.039 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.039 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:42.039 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.039 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.039 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.039 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 27/21 ECSubWrite(tid=223, reqid=client.4512.0:3332, at_version=27'145, trim_to=0'0, roll_forward_to=27'141))", 2020-06-12T02:37:42.040 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:38.615521+0000", 2020-06-12T02:37:42.040 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.15259746, 2020-06-12T02:37:42.040 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0072940510000000002, 2020-06-12T02:37:42.040 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:42.040 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:42.041 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:42.041 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.041 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:42.041 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.615521+0000", 2020-06-12T02:37:42.042 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:42.042 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.042 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.042 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:42.042 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.615521+0000", 2020-06-12T02:37:42.043 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000242845 2020-06-12T02:37:42.043 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.043 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.043 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:42.044 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.615763+0000", 2020-06-12T02:37:42.044 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4327e-05 2020-06-12T02:37:42.044 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.044 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.045 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:42.045 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.615778+0000", 2020-06-12T02:37:42.045 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3340000000000008e-06 2020-06-12T02:37:42.045 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.046 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.046 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:42.046 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.615787+0000", 2020-06-12T02:37:42.046 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0250000000000008e-06 2020-06-12T02:37:42.047 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.047 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.047 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:42.047 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.615796+0000", 2020-06-12T02:37:42.047 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9427e-05 2020-06-12T02:37:42.048 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.048 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.048 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.048 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.615846+0000", 2020-06-12T02:37:42.048 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5616000000000001e-05 2020-06-12T02:37:42.049 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.049 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.049 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:42.049 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.615861+0000", 2020-06-12T02:37:42.049 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069158700000000002 2020-06-12T02:37:42.049 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.050 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.050 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:42.050 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.622777+0000", 2020-06-12T02:37:42.051 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7607000000000001e-05 2020-06-12T02:37:42.051 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.051 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.051 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:42.052 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.622815+0000", 2020-06-12T02:37:42.052 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0072940510000000002 2020-06-12T02:37:42.052 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.052 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:42.052 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.053 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.053 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.053 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 27/21 ECSubWrite(tid=226, reqid=client.4512.0:3334, at_version=0'0, trim_to=0'0, roll_forward_to=27'147))", 2020-06-12T02:37:42.053 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:38.627654+0000", 2020-06-12T02:37:42.054 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.140463886, 2020-06-12T02:37:42.054 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01294059, 2020-06-12T02:37:42.054 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:42.055 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:42.055 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:42.055 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.056 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:42.056 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.627654+0000", 2020-06-12T02:37:42.056 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:42.057 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.057 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.057 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:42.057 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.627654+0000", 2020-06-12T02:37:42.058 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6134000000000003e-05 2020-06-12T02:37:42.058 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.059 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.059 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:42.059 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.627690+0000", 2020-06-12T02:37:42.059 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1439999999999992e-06 2020-06-12T02:37:42.059 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.059 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.060 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:42.060 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.627699+0000", 2020-06-12T02:37:42.060 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.59e-05 2020-06-12T02:37:42.060 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.060 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.061 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:42.061 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.627715+0000", 2020-06-12T02:37:42.061 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.602e-06 2020-06-12T02:37:42.061 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.062 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.062 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:42.062 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.627724+0000", 2020-06-12T02:37:42.062 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0059999999999998e-05 2020-06-12T02:37:42.062 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.063 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.063 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.063 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.627784+0000", 2020-06-12T02:37:42.063 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1885999999999999e-05 2020-06-12T02:37:42.063 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.064 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.064 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:42.064 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.627796+0000", 2020-06-12T02:37:42.064 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012765789 2020-06-12T02:37:42.065 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.065 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.065 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:42.065 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.640562+0000", 2020-06-12T02:37:42.065 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3074999999999998e-05 2020-06-12T02:37:42.066 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.066 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.066 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:42.066 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.640595+0000", 2020-06-12T02:37:42.066 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01294059 2020-06-12T02:37:42.067 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.067 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:42.067 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.067 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.067 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.068 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s2 27/21 ECSubWrite(tid=950, reqid=client.4512.0:3417, at_version=27'276, trim_to=0'0, roll_forward_to=27'275))", 2020-06-12T02:37:42.068 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:38.825335+0000", 2020-06-12T02:37:42.068 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 2.9427835569999998, 2020-06-12T02:37:42.068 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0066797250000000001, 2020-06-12T02:37:42.068 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:42.069 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:42.069 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:42.069 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.069 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:42.069 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.825335+0000", 2020-06-12T02:37:42.070 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:42.070 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.072 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.072 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:42.072 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.825335+0000", 2020-06-12T02:37:42.072 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000415895 2020-06-12T02:37:42.072 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.073 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.073 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:42.073 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.825750+0000", 2020-06-12T02:37:42.073 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6141999999999999e-05 2020-06-12T02:37:42.073 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.074 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.074 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:42.074 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.825767+0000", 2020-06-12T02:37:42.074 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1514e-05 2020-06-12T02:37:42.074 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.075 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.075 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:42.075 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.825778+0000", 2020-06-12T02:37:42.075 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4809999999999997e-06 2020-06-12T02:37:42.075 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.075 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.076 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:42.076 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.825784+0000", 2020-06-12T02:37:42.076 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2658999999999998e-05 2020-06-12T02:37:42.076 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.076 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.077 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.077 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.825836+0000", 2020-06-12T02:37:42.077 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7759999999999992e-06 2020-06-12T02:37:42.077 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.078 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.078 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:42.078 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.825846+0000", 2020-06-12T02:37:42.078 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0061351560000000001 2020-06-12T02:37:42.078 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.079 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.079 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:42.079 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.831981+0000", 2020-06-12T02:37:42.079 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3102000000000003e-05 2020-06-12T02:37:42.079 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.079 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.080 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:42.080 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.832014+0000", 2020-06-12T02:37:42.080 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0066797250000000001 2020-06-12T02:37:42.080 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.080 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:42.080 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.081 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.081 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:42.081 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:37:42.083 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-06-12T02:37:42.084 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:37:42.084 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:37:42.085 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-06-12T02:37:42.085 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-06-12T02:37:42.085 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-06-12T02:37:42.085 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-06-12T02:37:42.085 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-06-12T02:37:42.086 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-06-12T02:37:42.086 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-06-12T02:37:42.086 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.7 2020-06-12T02:37:42.086 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: problem getting command descriptions from osd.7 2020-06-12T02:37:42.086 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-06-12T02:37:42.087 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-06-12T02:37:42.087 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:42.080+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:42.184 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:37:42.181+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:37:42.241 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:37:42.241 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:37:42.241 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2020-06-12T02:37:42.242 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:37:42.249 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-06-12T02:37:42.294 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:42.291+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:42.365 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:37:42.365 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:37:42.365 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2020-06-12T02:37:42.366 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2020-06-12T02:37:42.366 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:37:42.370 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2020-06-12T02:37:42.384 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-06-12T02:37:42.388 INFO:tasks.rados.rados.0.smithi107.stdout:2098: done (0 left) 2020-06-12T02:37:42.389 INFO:tasks.rados.rados.0.smithi107.stdout:2099: delete oid 10 current snap is 0 2020-06-12T02:37:42.393 INFO:tasks.rados.rados.0.smithi107.stdout:2099: done (0 left) 2020-06-12T02:37:42.394 INFO:tasks.rados.rados.0.smithi107.stdout:2100: delete oid 7 current snap is 0 2020-06-12T02:37:42.395 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:37:42.387+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:37:42.396 INFO:tasks.rados.rados.0.smithi107.stdout:2100: done (0 left) 2020-06-12T02:37:42.407 INFO:tasks.rados.rados.0.smithi107.stdout:2101: delete oid 2 current snap is 0 2020-06-12T02:37:42.408 INFO:tasks.rados.rados.0.smithi107.stdout:2101: done (0 left) 2020-06-12T02:37:42.408 INFO:tasks.rados.rados.0.smithi107.stdout:2102: delete oid 17 current snap is 0 2020-06-12T02:37:42.408 INFO:tasks.rados.rados.0.smithi107.stdout:2102: done (0 left) 2020-06-12T02:37:42.409 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:37:42.417 INFO:tasks.rados.rados.0.smithi107.stdout:2103: seq_num 2102 ranges {0=409600,409600=581632,991232=622592} 2020-06-12T02:37:42.421 INFO:tasks.rados.rados.0.smithi107.stdout:2103: writing smithi10729407-12 from 0 to 409600 tid 1 2020-06-12T02:37:42.421 INFO:tasks.rados.rados.0.smithi107.stdout:2103: writing smithi10729407-12 from 409600 to 991232 tid 2 2020-06-12T02:37:42.421 INFO:tasks.rados.rados.0.smithi107.stdout:2103: writing smithi10729407-12 from 991232 to 1613824 tid 3 2020-06-12T02:37:42.421 INFO:tasks.rados.rados.0.smithi107.stdout:2104: delete oid 18 current snap is 0 2020-06-12T02:37:42.422 INFO:tasks.rados.rados.0.smithi107.stdout:2103: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:42.422 INFO:tasks.rados.rados.0.smithi107.stdout:2104: done (1 left) 2020-06-12T02:37:42.422 INFO:tasks.rados.rados.0.smithi107.stdout:2105: delete oid 17 current snap is 0 2020-06-12T02:37:42.422 INFO:tasks.rados.rados.0.smithi107.stdout:2105: done (1 left) 2020-06-12T02:37:42.422 INFO:tasks.rados.rados.0.smithi107.stdout:2106: delete oid 11 current snap is 0 2020-06-12T02:37:42.423 INFO:tasks.rados.rados.0.smithi107.stdout:2103: finishing write tid 2 to smithi10729407-12 2020-06-12T02:37:42.423 INFO:tasks.rados.rados.0.smithi107.stdout:2103: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:42.423 INFO:tasks.rados.rados.0.smithi107.stdout:2106: done (1 left) 2020-06-12T02:37:42.424 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:37:42.424 INFO:tasks.rados.rados.0.smithi107.stdout:2107: seq_num 2106 ranges {0=475136,475136=573440,1048576=630784} 2020-06-12T02:37:42.424 INFO:tasks.rados.rados.0.smithi107.stdout:2107: writing smithi10729407-6 from 0 to 475136 tid 1 2020-06-12T02:37:42.424 INFO:tasks.rados.rados.0.smithi107.stdout:2107: writing smithi10729407-6 from 475136 to 1048576 tid 2 2020-06-12T02:37:42.426 INFO:tasks.rados.rados.0.smithi107.stdout:2107: writing smithi10729407-6 from 1048576 to 1679360 tid 3 2020-06-12T02:37:42.426 INFO:tasks.rados.rados.0.smithi107.stdout:2108: delete oid 17 current snap is 0 2020-06-12T02:37:42.426 INFO:tasks.rados.rados.0.smithi107.stdout:2103: finishing write tid 5 to smithi10729407-12 2020-06-12T02:37:42.426 INFO:tasks.rados.rados.0.smithi107.stdout:2103: finishing write tid 6 to smithi10729407-12 2020-06-12T02:37:42.426 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 320 (ObjNum 2102 snap 0 seq_num 2102) dirty exists 2020-06-12T02:37:42.427 INFO:tasks.rados.rados.0.smithi107.stdout:2103: left oid 12 (ObjNum 2102 snap 0 seq_num 2102) 2020-06-12T02:37:42.427 INFO:tasks.rados.rados.0.smithi107.stdout:2107: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:42.427 INFO:tasks.rados.rados.0.smithi107.stdout:2103: done (2 left) 2020-06-12T02:37:42.427 INFO:tasks.rados.rados.0.smithi107.stdout:2108: done (1 left) 2020-06-12T02:37:42.427 INFO:tasks.rados.rados.0.smithi107.stdout:2109: delete oid 5 current snap is 0 2020-06-12T02:37:42.428 INFO:tasks.rados.rados.0.smithi107.stdout:2109: done (1 left) 2020-06-12T02:37:42.428 INFO:tasks.rados.rados.0.smithi107.stdout:2110: delete oid 19 current snap is 0 2020-06-12T02:37:42.431 INFO:tasks.rados.rados.0.smithi107.stdout:2107: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:42.431 INFO:tasks.rados.rados.0.smithi107.stdout:2110: done (1 left) 2020-06-12T02:37:42.431 INFO:tasks.rados.rados.0.smithi107.stdout:2111: delete oid 3 current snap is 0 2020-06-12T02:37:42.432 INFO:tasks.rados.rados.0.smithi107.stdout:2107: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:42.432 INFO:tasks.rados.rados.0.smithi107.stdout:2107: finishing write tid 5 to smithi10729407-6 2020-06-12T02:37:42.432 INFO:tasks.rados.rados.0.smithi107.stdout:2111: done (1 left) 2020-06-12T02:37:42.432 INFO:tasks.rados.rados.0.smithi107.stdout:2112: delete oid 9 current snap is 0 2020-06-12T02:37:42.433 INFO:tasks.rados.rados.0.smithi107.stdout:2107: finishing write tid 6 to smithi10729407-6 2020-06-12T02:37:42.433 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 441 (ObjNum 2106 snap 0 seq_num 2106) dirty exists 2020-06-12T02:37:42.434 INFO:tasks.rados.rados.0.smithi107.stdout:2107: left oid 6 (ObjNum 2106 snap 0 seq_num 2106) 2020-06-12T02:37:42.436 INFO:tasks.rados.rados.0.smithi107.stdout:2107: done (1 left) 2020-06-12T02:37:42.437 INFO:tasks.rados.rados.0.smithi107.stdout:2112: done (0 left) 2020-06-12T02:37:42.437 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:37:42.437 INFO:tasks.rados.rados.0.smithi107.stdout:2113: seq_num 2112 ranges {0=581632,581632=679936,1261568=524288} 2020-06-12T02:37:42.441 INFO:tasks.rados.rados.0.smithi107.stdout:2113: writing smithi10729407-16 from 0 to 581632 tid 1 2020-06-12T02:37:42.444 INFO:tasks.rados.rados.0.smithi107.stdout:2113: writing smithi10729407-16 from 581632 to 1261568 tid 2 2020-06-12T02:37:42.447 INFO:tasks.rados.rados.0.smithi107.stdout:2113: writing smithi10729407-16 from 1261568 to 1785856 tid 3 2020-06-12T02:37:42.448 INFO:tasks.rados.rados.0.smithi107.stdout:2114: delete oid 8 current snap is 0 2020-06-12T02:37:42.449 INFO:tasks.rados.rados.0.smithi107.stdout:2113: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:42.450 INFO:tasks.rados.rados.0.smithi107.stdout:2114: done (1 left) 2020-06-12T02:37:42.450 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:37:42.450 INFO:tasks.rados.rados.0.smithi107.stdout:2115: seq_num 2114 ranges {1261568=614400,1875968=491520,2367488=638976,3006464=73728} 2020-06-12T02:37:42.460 INFO:tasks.rados.rados.0.smithi107.stdout:2115: writing smithi10729407-4 from 1261568 to 1875968 tid 1 2020-06-12T02:37:42.463 INFO:tasks.rados.rados.0.smithi107.stdout:2115: writing smithi10729407-4 from 1875968 to 2367488 tid 2 2020-06-12T02:37:42.468 INFO:tasks.rados.rados.0.smithi107.stdout:2115: writing smithi10729407-4 from 2367488 to 3006464 tid 3 2020-06-12T02:37:42.468 INFO:tasks.rados.rados.0.smithi107.stdout:2115: writing smithi10729407-4 from 3006464 to 3080192 tid 4 2020-06-12T02:37:42.469 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:37:42.469 INFO:tasks.rados.rados.0.smithi107.stdout:2116: seq_num 2115 ranges {0=630784,630784=778240,1409024=425984} 2020-06-12T02:37:42.473 INFO:tasks.rados.rados.0.smithi107.stdout:2116: writing smithi10729407-13 from 0 to 630784 tid 1 2020-06-12T02:37:42.496 INFO:tasks.rados.rados.0.smithi107.stdout:2116: writing smithi10729407-13 from 630784 to 1409024 tid 2 2020-06-12T02:37:42.496 INFO:tasks.rados.rados.0.smithi107.stdout:2116: writing smithi10729407-13 from 1409024 to 1835008 tid 3 2020-06-12T02:37:42.496 INFO:tasks.rados.rados.0.smithi107.stdout:2117: delete oid 3 current snap is 0 2020-06-12T02:37:42.496 INFO:tasks.rados.rados.0.smithi107.stdout:2113: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:42.497 INFO:tasks.rados.rados.0.smithi107.stdout:2113: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:42.497 INFO:tasks.rados.rados.0.smithi107.stdout:2113: finishing write tid 5 to smithi10729407-16 2020-06-12T02:37:42.497 INFO:tasks.rados.rados.0.smithi107.stdout:2113: finishing write tid 6 to smithi10729407-16 2020-06-12T02:37:42.497 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 374 (ObjNum 2112 snap 0 seq_num 2112) dirty exists 2020-06-12T02:37:42.497 INFO:tasks.rados.rados.0.smithi107.stdout:2113: left oid 16 (ObjNum 2112 snap 0 seq_num 2112) 2020-06-12T02:37:42.498 INFO:tasks.rados.rados.0.smithi107.stdout:2115: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:42.498 INFO:tasks.rados.rados.0.smithi107.stdout:2115: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:42.498 INFO:tasks.rados.rados.0.smithi107.stdout:2115: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:42.498 INFO:tasks.rados.rados.0.smithi107.stdout:2115: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:42.498 INFO:tasks.rados.rados.0.smithi107.stdout:2115: finishing write tid 6 to smithi10729407-4 2020-06-12T02:37:42.499 INFO:tasks.rados.rados.0.smithi107.stdout:2116: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:42.499 INFO:tasks.rados.rados.0.smithi107.stdout:2115: finishing write tid 7 to smithi10729407-4 2020-06-12T02:37:42.499 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 407 (ObjNum 2114 snap 0 seq_num 2114) dirty exists 2020-06-12T02:37:42.499 INFO:tasks.rados.rados.0.smithi107.stdout:2115: left oid 4 (ObjNum 2114 snap 0 seq_num 2114) 2020-06-12T02:37:42.499 INFO:tasks.rados.rados.0.smithi107.stdout:2113: done (3 left) 2020-06-12T02:37:42.500 INFO:tasks.rados.rados.0.smithi107.stdout:2115: done (2 left) 2020-06-12T02:37:42.500 INFO:tasks.rados.rados.0.smithi107.stdout:2117: done (1 left) 2020-06-12T02:37:42.500 INFO:tasks.rados.rados.0.smithi107.stdout:2118: delete oid 14 current snap is 0 2020-06-12T02:37:42.500 INFO:tasks.rados.rados.0.smithi107.stdout:2116: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:42.500 INFO:tasks.rados.rados.0.smithi107.stdout:2116: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:42.501 INFO:tasks.rados.rados.0.smithi107.stdout:2116: finishing write tid 5 to smithi10729407-13 2020-06-12T02:37:42.501 INFO:tasks.rados.rados.0.smithi107.stdout:2118: done (1 left) 2020-06-12T02:37:42.501 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:42.501 INFO:tasks.rados.rados.0.smithi107.stdout:2119: seq_num 2118 ranges {0=679936,679936=434176,1114112=622592,1736704=147456} 2020-06-12T02:37:42.502 INFO:tasks.rados.rados.0.smithi107.stdout:2119: writing smithi10729407-2 from 0 to 679936 tid 1 2020-06-12T02:37:42.502 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:42.491+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:42.503 INFO:tasks.rados.rados.0.smithi107.stdout:2119: writing smithi10729407-2 from 679936 to 1114112 tid 2 2020-06-12T02:37:42.503 INFO:tasks.rados.rados.0.smithi107.stdout:2119: writing smithi10729407-2 from 1114112 to 1736704 tid 3 2020-06-12T02:37:42.504 INFO:tasks.rados.rados.0.smithi107.stdout:2119: writing smithi10729407-2 from 1736704 to 1884160 tid 4 2020-06-12T02:37:42.504 INFO:tasks.rados.rados.0.smithi107.stdout:2120: delete oid 11 current snap is 0 2020-06-12T02:37:42.504 INFO:tasks.rados.rados.0.smithi107.stdout:2116: finishing write tid 6 to smithi10729407-13 2020-06-12T02:37:42.504 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 404 (ObjNum 2115 snap 0 seq_num 2115) dirty exists 2020-06-12T02:37:42.504 INFO:tasks.rados.rados.0.smithi107.stdout:2116: left oid 13 (ObjNum 2115 snap 0 seq_num 2115) 2020-06-12T02:37:42.505 INFO:tasks.rados.rados.0.smithi107.stdout:2119: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:42.506 INFO:tasks.rados.rados.0.smithi107.stdout:2116: done (2 left) 2020-06-12T02:37:42.506 INFO:tasks.rados.rados.0.smithi107.stdout:2120: done (1 left) 2020-06-12T02:37:42.506 INFO:tasks.rados.rados.0.smithi107.stdout:2121: delete oid 3 current snap is 0 2020-06-12T02:37:42.508 INFO:tasks.rados.rados.0.smithi107.stdout:2121: done (1 left) 2020-06-12T02:37:42.509 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:37:42.509 INFO:tasks.rados.rados.0.smithi107.stdout:2122: seq_num 2121 ranges {1425408=729088,2154496=483328,2637824=720896} 2020-06-12T02:37:42.520 INFO:tasks.rados.rados.0.smithi107.stdout:2122: writing smithi10729407-15 from 1425408 to 2154496 tid 1 2020-06-12T02:37:42.523 INFO:tasks.rados.rados.0.smithi107.stdout:2122: writing smithi10729407-15 from 2154496 to 2637824 tid 2 2020-06-12T02:37:42.528 INFO:tasks.rados.rados.0.smithi107.stdout:2122: writing smithi10729407-15 from 2637824 to 3358720 tid 3 2020-06-12T02:37:42.529 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:37:42.530 INFO:tasks.rados.rados.0.smithi107.stdout:2123: seq_num 2122 ranges {0=745472,745472=417792,1163264=606208,1769472=180224} 2020-06-12T02:37:42.532 INFO:tasks.rados.rados.0.smithi107.stdout:2123: writing smithi10729407-18 from 0 to 745472 tid 1 2020-06-12T02:37:42.535 INFO:tasks.rados.rados.0.smithi107.stdout:2123: writing smithi10729407-18 from 745472 to 1163264 tid 2 2020-06-12T02:37:42.539 INFO:tasks.rados.rados.0.smithi107.stdout:2123: writing smithi10729407-18 from 1163264 to 1769472 tid 3 2020-06-12T02:37:42.540 INFO:tasks.rados.rados.0.smithi107.stdout:2123: writing smithi10729407-18 from 1769472 to 1949696 tid 4 2020-06-12T02:37:42.540 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:37:42.541 INFO:tasks.rados.rados.0.smithi107.stdout:2124: seq_num 2123 ranges {0=761856,761856=704512,1466368=409600,1875968=90112} 2020-06-12T02:37:42.545 INFO:tasks.rados.rados.0.smithi107.stdout:2124: writing smithi10729407-11 from 0 to 761856 tid 1 2020-06-12T02:37:42.549 INFO:tasks.rados.rados.0.smithi107.stdout:2124: writing smithi10729407-11 from 761856 to 1466368 tid 2 2020-06-12T02:37:42.554 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:37:42.555 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2020-06-12T02:37:42.555 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2020-06-12T02:37:42.555 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2020-06-12T02:37:42.556 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.556 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:962 3.7s0 3:ea271007:::smithi10729407-9:head [append 0~598016] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:42.556 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:32.236108+0000", 2020-06-12T02:37:42.557 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 10.311101580000001, 2020-06-12T02:37:42.557 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018271632999999999, 2020-06-12T02:37:42.557 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:42.557 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:42.558 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:42.558 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:42.558 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:42.559 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 962 2020-06-12T02:37:42.559 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.559 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:42.560 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.560 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:42.560 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.236108+0000", 2020-06-12T02:37:42.561 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:42.561 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.561 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.562 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:42.562 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.236108+0000", 2020-06-12T02:37:42.562 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019838700000000001 2020-06-12T02:37:42.562 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.563 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.563 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:42.563 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.236307+0000", 2020-06-12T02:37:42.564 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4619999999999996e-06 2020-06-12T02:37:42.564 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.565 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.565 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:42.565 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.236315+0000", 2020-06-12T02:37:42.566 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1665999999999999e-05 2020-06-12T02:37:42.566 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.566 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.567 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:42.567 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.236337+0000", 2020-06-12T02:37:42.567 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5630000000000003e-06 2020-06-12T02:37:42.567 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.568 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.568 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:42.568 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.236344+0000", 2020-06-12T02:37:42.569 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023269440000000001 2020-06-12T02:37:42.569 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.569 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.570 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.570 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.238671+0000", 2020-06-12T02:37:42.570 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5969000000000003e-05 2020-06-12T02:37:42.571 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.572 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.572 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:42.572 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.238757+0000", 2020-06-12T02:37:42.572 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046101010000000001 2020-06-12T02:37:42.573 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.573 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.573 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.574 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.243367+0000", 2020-06-12T02:37:42.574 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7991000000000005e-05 2020-06-12T02:37:42.574 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.575 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.575 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:42.576 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.243455+0000", 2020-06-12T02:37:42.576 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00062561500000000005 2020-06-12T02:37:42.576 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.577 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.577 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:42.577 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.244081+0000", 2020-06-12T02:37:42.578 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010040778 2020-06-12T02:37:42.578 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.578 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.578 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:42.579 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.254122+0000", 2020-06-12T02:37:42.579 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010453100000000001 2020-06-12T02:37:42.579 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.580 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.580 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:42.580 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.254226+0000", 2020-06-12T02:37:42.580 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000153626 2020-06-12T02:37:42.581 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.582 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.582 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:42.582 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.254380+0000", 2020-06-12T02:37:42.583 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018271632999999999 2020-06-12T02:37:42.583 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.583 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:42.584 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.584 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.584 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.584 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:963 3.7s0 3:ea271007:::smithi10729407-9:head [append 0~606208] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:42.585 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:32.239688+0000", 2020-06-12T02:37:42.586 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 10.307521255999999, 2020-06-12T02:37:42.586 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016181383000000001, 2020-06-12T02:37:42.586 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:42.586 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:42.586 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:42.587 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:42.587 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:42.587 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 963 2020-06-12T02:37:42.587 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.587 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:42.588 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.588 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:42.588 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.239688+0000", 2020-06-12T02:37:42.588 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:42.588 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.589 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.589 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:42.589 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.239688+0000", 2020-06-12T02:37:42.589 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00022408000000000001 2020-06-12T02:37:42.589 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.590 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.590 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:42.590 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.239913+0000", 2020-06-12T02:37:42.590 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0399999999999993e-06 2020-06-12T02:37:42.590 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.591 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.591 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:42.591 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.239921+0000", 2020-06-12T02:37:42.591 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7499999999999998e-05 2020-06-12T02:37:42.591 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.591 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.592 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:42.592 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.239938+0000", 2020-06-12T02:37:42.592 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0739999999999998e-06 2020-06-12T02:37:42.592 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.592 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.593 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:42.593 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.239944+0000", 2020-06-12T02:37:42.593 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2867000000000002e-05 2020-06-12T02:37:42.593 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.593 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.594 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.594 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.239977+0000", 2020-06-12T02:37:42.594 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4425000000000007e-05 2020-06-12T02:37:42.594 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.594 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.594 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:42.595 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.240051+0000", 2020-06-12T02:37:42.595 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0064351139999999996 2020-06-12T02:37:42.595 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.595 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.596 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.596 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.246487+0000", 2020-06-12T02:37:42.596 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012310899999999999 2020-06-12T02:37:42.597 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.597 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.597 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:42.597 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.246610+0000", 2020-06-12T02:37:42.598 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00079552599999999996 2020-06-12T02:37:42.598 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.598 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.598 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:42.599 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.247405+0000", 2020-06-12T02:37:42.599 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0079562030000000002 2020-06-12T02:37:42.599 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.600 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.600 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:42.600 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.255361+0000", 2020-06-12T02:37:42.600 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036610400000000002 2020-06-12T02:37:42.600 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.601 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.601 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:42.601 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.255728+0000", 2020-06-12T02:37:42.601 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014234100000000001 2020-06-12T02:37:42.602 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.602 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.602 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:42.603 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.255870+0000", 2020-06-12T02:37:42.603 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016181383000000001 2020-06-12T02:37:42.603 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.604 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:42.604 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.604 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.604 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.605 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:966 3.7s0 3:ea271007:::smithi10729407-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:42.605 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:32.242493+0000", 2020-06-12T02:37:42.605 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 10.304716344999999, 2020-06-12T02:37:42.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015989771, 2020-06-12T02:37:42.612 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:42.612 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:37:42.613 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:42.613 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:42.613 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:42.613 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 966 2020-06-12T02:37:42.613 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.614 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:42.614 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.614 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:42.614 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242493+0000", 2020-06-12T02:37:42.615 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:42.615 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.615 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.615 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:42.616 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242493+0000", 2020-06-12T02:37:42.616 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6276000000000002e-05 2020-06-12T02:37:42.616 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.616 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.617 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:42.617 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242540+0000", 2020-06-12T02:37:42.617 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4710000000000004e-06 2020-06-12T02:37:42.617 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.618 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.618 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:42.618 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242546+0000", 2020-06-12T02:37:42.619 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3157e-05 2020-06-12T02:37:42.619 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.619 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.619 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:42.619 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242559+0000", 2020-06-12T02:37:42.620 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7419999999999997e-06 2020-06-12T02:37:42.620 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.620 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.620 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:42.621 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242564+0000", 2020-06-12T02:37:42.621 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8620999999999999e-05 2020-06-12T02:37:42.621 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.621 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.622 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.622 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242593+0000", 2020-06-12T02:37:42.622 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0400000000000003e-05 2020-06-12T02:37:42.622 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.622 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.623 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:42.623 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.242673+0000", 2020-06-12T02:37:42.623 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01258803 2020-06-12T02:37:42.623 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.623 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.624 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.624 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.255261+0000", 2020-06-12T02:37:42.624 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5599000000000004e-05 2020-06-12T02:37:42.624 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.625 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.625 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:42.625 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.255357+0000", 2020-06-12T02:37:42.625 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001947999 2020-06-12T02:37:42.626 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.626 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.626 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.626 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.257305+0000", 2020-06-12T02:37:42.626 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5921999999999994e-05 2020-06-12T02:37:42.627 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.627 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.627 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:42.627 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.257401+0000", 2020-06-12T02:37:42.627 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0010825539999999999 2020-06-12T02:37:42.628 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.628 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.628 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:42.628 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.258483+0000", 2020-06-12T02:37:42.628 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015989771 2020-06-12T02:37:42.629 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.629 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:42.629 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.629 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.630 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.630 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1200 3.2s0 3:4afffc6f:::smithi10729407-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:42.630 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:32.837068+0000", 2020-06-12T02:37:42.630 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 9.710141728, 2020-06-12T02:37:42.631 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014863888, 2020-06-12T02:37:42.631 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:42.631 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:37:42.631 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:42.631 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:42.632 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:42.632 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1200 2020-06-12T02:37:42.632 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.632 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:42.633 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.633 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:42.633 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.837068+0000", 2020-06-12T02:37:42.633 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:42.634 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.634 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.634 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:42.634 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.837068+0000", 2020-06-12T02:37:42.635 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2849000000000001e-05 2020-06-12T02:37:42.635 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.635 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.636 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:42.636 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.837091+0000", 2020-06-12T02:37:42.636 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0290000000000001e-06 2020-06-12T02:37:42.636 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.636 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.637 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:42.637 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.837094+0000", 2020-06-12T02:37:42.637 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1800000000000001e-06 2020-06-12T02:37:42.637 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.638 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.638 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:42.638 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.837100+0000", 2020-06-12T02:37:42.638 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2120000000000002e-06 2020-06-12T02:37:42.638 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.639 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.639 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:42.639 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.837102+0000", 2020-06-12T02:37:42.639 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090177509999999992 2020-06-12T02:37:42.640 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.640 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.640 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.640 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.846120+0000", 2020-06-12T02:37:42.640 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8610000000000003e-05 2020-06-12T02:37:42.641 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.641 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.641 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:42.641 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.846179+0000", 2020-06-12T02:37:42.642 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045638520000000002 2020-06-12T02:37:42.642 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.642 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.642 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.643 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.850742+0000", 2020-06-12T02:37:42.643 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0758000000000002e-05 2020-06-12T02:37:42.643 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.643 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.643 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:42.644 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.850823+0000", 2020-06-12T02:37:42.644 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011086469999999999 2020-06-12T02:37:42.644 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.644 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.644 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:42.645 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.851932+0000", 2020-06-12T02:37:42.645 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014863888 2020-06-12T02:37:42.645 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.645 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:42.645 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.646 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.646 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.646 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1252 3.8s0 3:14244450:::smithi10729407-2:head [append 0~516096] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:42.646 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:32.999793+0000", 2020-06-12T02:37:42.647 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 9.5474169999999994, 2020-06-12T02:37:42.647 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017380335, 2020-06-12T02:37:42.647 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:42.647 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:42.648 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:42.648 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:42.648 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:42.648 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1252 2020-06-12T02:37:42.648 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.649 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:42.649 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.649 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:42.649 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.999793+0000", 2020-06-12T02:37:42.649 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:42.650 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.650 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.650 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:42.650 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.999793+0000", 2020-06-12T02:37:42.651 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000208051 2020-06-12T02:37:42.651 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.651 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.651 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:42.651 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.000001+0000", 2020-06-12T02:37:42.652 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9750000000000002e-06 2020-06-12T02:37:42.652 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.652 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.652 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:42.653 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.000011+0000", 2020-06-12T02:37:42.653 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7733000000000001e-05 2020-06-12T02:37:42.653 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.653 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.653 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:42.654 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.000028+0000", 2020-06-12T02:37:42.654 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7379999999999994e-06 2020-06-12T02:37:42.654 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.654 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.654 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:42.655 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.000036+0000", 2020-06-12T02:37:42.655 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.414e-05 2020-06-12T02:37:42.655 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.655 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.655 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.656 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.000090+0000", 2020-06-12T02:37:42.656 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012561699999999999 2020-06-12T02:37:42.656 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.656 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.656 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:42.657 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.000216+0000", 2020-06-12T02:37:42.658 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00079436199999999995 2020-06-12T02:37:42.658 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.658 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.658 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:42.659 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.001010+0000", 2020-06-12T02:37:42.659 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015996506000000001 2020-06-12T02:37:42.659 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.659 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.659 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:42.660 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.017007+0000", 2020-06-12T02:37:42.660 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0599999999999997e-05 2020-06-12T02:37:42.660 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.660 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.660 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:42.661 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.017057+0000", 2020-06-12T02:37:42.661 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011561300000000001 2020-06-12T02:37:42.661 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.661 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.661 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:42.662 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.017173+0000", 2020-06-12T02:37:42.662 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017380335 2020-06-12T02:37:42.662 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.662 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:42.662 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.663 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.663 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.663 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1299 3.bs0 3:d0ebae7d:::smithi10729407-3:head [append 0~770048] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:42.663 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.162816+0000", 2020-06-12T02:37:42.663 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 9.3843938619999996, 2020-06-12T02:37:42.663 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016787209000000001, 2020-06-12T02:37:42.664 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:42.664 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:42.664 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:42.664 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:42.664 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:42.665 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1299 2020-06-12T02:37:42.665 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.665 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:42.665 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.665 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:42.666 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.162816+0000", 2020-06-12T02:37:42.666 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:42.666 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.666 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.666 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:42.666 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.162816+0000", 2020-06-12T02:37:42.667 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044992799999999999 2020-06-12T02:37:42.667 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.667 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.667 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:42.667 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.163266+0000", 2020-06-12T02:37:42.668 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6959999999999999e-06 2020-06-12T02:37:42.668 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.668 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.668 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:42.668 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.163272+0000", 2020-06-12T02:37:42.668 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3028e-05 2020-06-12T02:37:42.669 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.669 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.669 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:42.669 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.163286+0000", 2020-06-12T02:37:42.669 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2829999999999998e-06 2020-06-12T02:37:42.670 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.670 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.670 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:42.670 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.163292+0000", 2020-06-12T02:37:42.670 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1761000000000002e-05 2020-06-12T02:37:42.670 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.671 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.671 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.671 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.163334+0000", 2020-06-12T02:37:42.671 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6691000000000001e-05 2020-06-12T02:37:42.671 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.672 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.672 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:42.672 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.163410+0000", 2020-06-12T02:37:42.672 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004070176 2020-06-12T02:37:42.672 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.672 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.673 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.674 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.167480+0000", 2020-06-12T02:37:42.674 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7137000000000003e-05 2020-06-12T02:37:42.674 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.675 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.675 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:42.675 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.167568+0000", 2020-06-12T02:37:42.676 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000789648 2020-06-12T02:37:42.676 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.676 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.676 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:42.677 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168357+0000", 2020-06-12T02:37:42.677 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011078226 2020-06-12T02:37:42.677 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.678 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.678 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:42.678 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.179435+0000", 2020-06-12T02:37:42.679 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6649000000000003e-05 2020-06-12T02:37:42.679 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.679 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.679 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:42.680 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.179522+0000", 2020-06-12T02:37:42.680 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0986e-05 2020-06-12T02:37:42.680 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.683 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.684 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:42.684 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.179603+0000", 2020-06-12T02:37:42.684 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016787209000000001 2020-06-12T02:37:42.685 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.685 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:42.685 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.686 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.686 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.686 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1300 3.bs0 3:d0ebae7d:::smithi10729407-3:head [append 0~557056] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:42.687 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.166253+0000", 2020-06-12T02:37:42.687 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 9.3809563540000003, 2020-06-12T02:37:42.687 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019037724999999998, 2020-06-12T02:37:42.688 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:42.688 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:42.688 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:42.688 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:42.689 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:42.689 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1300 2020-06-12T02:37:42.689 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.690 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:42.690 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.691 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:42.691 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.166253+0000", 2020-06-12T02:37:42.691 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:42.692 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.692 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.692 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:42.692 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.166253+0000", 2020-06-12T02:37:42.693 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034533600000000002 2020-06-12T02:37:42.693 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.693 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.694 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:42.694 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.166599+0000", 2020-06-12T02:37:42.694 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7699999999999996e-06 2020-06-12T02:37:42.695 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.695 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.695 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:42.696 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.166608+0000", 2020-06-12T02:37:42.696 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7314999999999999e-05 2020-06-12T02:37:42.696 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.697 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.697 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:42.697 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.166626+0000", 2020-06-12T02:37:42.697 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9260000000000003e-06 2020-06-12T02:37:42.697 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.698 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.698 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:42.698 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.166633+0000", 2020-06-12T02:37:42.698 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0266000000000002e-05 2020-06-12T02:37:42.698 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.699 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.699 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.699 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.166693+0000", 2020-06-12T02:37:42.699 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6692000000000005e-05 2020-06-12T02:37:42.699 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.700 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.700 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:42.700 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.166760+0000", 2020-06-12T02:37:42.700 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0057626860000000004 2020-06-12T02:37:42.700 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.700 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.701 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.701 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.172522+0000", 2020-06-12T02:37:42.701 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000130493 2020-06-12T02:37:42.701 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.701 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.702 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:42.702 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.172653+0000", 2020-06-12T02:37:42.702 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00083376299999999995 2020-06-12T02:37:42.703 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.703 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.703 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:42.704 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.173487+0000", 2020-06-12T02:37:42.704 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0061197739999999997 2020-06-12T02:37:42.704 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.705 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.705 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:42.706 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.179606+0000", 2020-06-12T02:37:42.706 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0056160699999999999 2020-06-12T02:37:42.707 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.707 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.707 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:42.707 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.185222+0000", 2020-06-12T02:37:42.707 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8633999999999994e-05 2020-06-12T02:37:42.708 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.708 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.708 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:42.708 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.185291+0000", 2020-06-12T02:37:42.708 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019037724999999998 2020-06-12T02:37:42.709 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.709 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:42.709 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.709 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.709 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.710 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1301 3.bs0 3:d0ebae7d:::smithi10729407-3:head [append 0~278528] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:42.710 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.168146+0000", 2020-06-12T02:37:42.710 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 9.3790637970000006, 2020-06-12T02:37:42.710 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017691083999999999, 2020-06-12T02:37:42.710 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:42.711 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:42.711 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:42.711 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:42.711 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:42.711 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1301 2020-06-12T02:37:42.712 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.712 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:42.712 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.712 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:42.712 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168146+0000", 2020-06-12T02:37:42.712 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:42.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.713 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.713 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:42.713 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168146+0000", 2020-06-12T02:37:42.713 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000213666 2020-06-12T02:37:42.714 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.714 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:42.714 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168360+0000", 2020-06-12T02:37:42.714 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7009999999999996e-06 2020-06-12T02:37:42.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.715 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.715 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:42.715 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168367+0000", 2020-06-12T02:37:42.715 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5096000000000001e-05 2020-06-12T02:37:42.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.716 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.716 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:42.716 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168382+0000", 2020-06-12T02:37:42.716 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8739999999999997e-06 2020-06-12T02:37:42.716 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.717 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.717 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:42.717 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168389+0000", 2020-06-12T02:37:42.717 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0076120199999999997 2020-06-12T02:37:42.717 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.718 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.718 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.718 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.176001+0000", 2020-06-12T02:37:42.718 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9181000000000001e-05 2020-06-12T02:37:42.718 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.718 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.719 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:42.719 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.176060+0000", 2020-06-12T02:37:42.719 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00039587599999999998 2020-06-12T02:37:42.719 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.719 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.720 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:42.720 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.176456+0000", 2020-06-12T02:37:42.720 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0031522329999999999 2020-06-12T02:37:42.720 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.720 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.721 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:42.721 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.179609+0000", 2020-06-12T02:37:42.721 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0061683670000000001 2020-06-12T02:37:42.721 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.721 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.722 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:42.722 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.185777+0000", 2020-06-12T02:37:42.722 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0069999999999999e-05 2020-06-12T02:37:42.722 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.723 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.723 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:42.723 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.185837+0000", 2020-06-12T02:37:42.723 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017691083999999999 2020-06-12T02:37:42.723 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.723 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:42.724 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.724 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.724 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.724 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1302 3.bs0 3:d0ebae7d:::smithi10729407-3:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:42.724 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.168445+0000", 2020-06-12T02:37:42.725 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 9.3787645170000005, 2020-06-12T02:37:42.725 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017581657000000001, 2020-06-12T02:37:42.725 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:42.725 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:42.725 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:42.725 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:42.726 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:42.726 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1302 2020-06-12T02:37:42.726 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.726 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:42.726 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.727 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:42.727 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168445+0000", 2020-06-12T02:37:42.727 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:42.727 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.727 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.727 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:42.728 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168445+0000", 2020-06-12T02:37:42.728 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3532e-05 2020-06-12T02:37:42.728 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.728 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.728 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:42.729 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168519+0000", 2020-06-12T02:37:42.729 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2600000000000002e-06 2020-06-12T02:37:42.729 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.729 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.729 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:42.729 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168525+0000", 2020-06-12T02:37:42.730 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4338999999999999e-05 2020-06-12T02:37:42.730 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.730 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.730 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:42.730 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168539+0000", 2020-06-12T02:37:42.731 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7330000000000003e-06 2020-06-12T02:37:42.731 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.731 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.731 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:42.731 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168544+0000", 2020-06-12T02:37:42.732 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0095046090000000007 2020-06-12T02:37:42.732 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.732 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.732 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.732 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.178049+0000", 2020-06-12T02:37:42.732 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1524000000000003e-05 2020-06-12T02:37:42.733 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.733 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.733 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:42.733 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.178110+0000", 2020-06-12T02:37:42.733 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030514799999999999 2020-06-12T02:37:42.734 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.734 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.734 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:42.734 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.178415+0000", 2020-06-12T02:37:42.734 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011944480000000001 2020-06-12T02:37:42.735 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.735 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.735 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:42.735 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.179610+0000", 2020-06-12T02:37:42.735 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0063372849999999998 2020-06-12T02:37:42.735 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.736 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.736 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:42.736 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.185947+0000", 2020-06-12T02:37:42.736 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9778999999999996e-05 2020-06-12T02:37:42.736 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.737 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.738 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:42.738 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.186027+0000", 2020-06-12T02:37:42.739 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017581657000000001 2020-06-12T02:37:42.739 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.739 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:42.739 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.740 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.740 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.740 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1303 3.bs0 3:d0ebae7d:::smithi10729407-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:42.740 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.168587+0000", 2020-06-12T02:37:42.740 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 9.3786225989999998, 2020-06-12T02:37:42.741 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019127155, 2020-06-12T02:37:42.741 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:42.741 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:37:42.741 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:42.741 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:42.742 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:42.742 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1303 2020-06-12T02:37:42.742 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.742 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:42.743 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.743 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:42.743 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168587+0000", 2020-06-12T02:37:42.743 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:42.743 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.744 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.744 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:42.744 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168587+0000", 2020-06-12T02:37:42.744 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3099999999999997e-05 2020-06-12T02:37:42.744 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.744 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.745 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:42.745 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168630+0000", 2020-06-12T02:37:42.745 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1600000000000003e-06 2020-06-12T02:37:42.745 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.745 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.746 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:42.746 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168636+0000", 2020-06-12T02:37:42.746 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3439e-05 2020-06-12T02:37:42.746 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.747 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.747 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:42.747 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168650+0000", 2020-06-12T02:37:42.747 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7850000000000002e-06 2020-06-12T02:37:42.747 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.747 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.748 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:42.748 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168654+0000", 2020-06-12T02:37:42.748 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010326317 2020-06-12T02:37:42.748 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.748 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.749 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.749 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.178980+0000", 2020-06-12T02:37:42.749 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6593000000000004e-05 2020-06-12T02:37:42.749 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.749 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.750 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:42.750 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.179047+0000", 2020-06-12T02:37:42.750 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0073410990000000002 2020-06-12T02:37:42.750 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.750 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.751 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.751 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.186388+0000", 2020-06-12T02:37:42.751 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9405000000000001e-05 2020-06-12T02:37:42.751 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.751 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.751 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:42.752 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.186487+0000", 2020-06-12T02:37:42.752 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001227257 2020-06-12T02:37:42.752 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.752 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.752 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:42.753 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.187714+0000", 2020-06-12T02:37:42.753 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019127155 2020-06-12T02:37:42.753 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.753 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:42.754 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.754 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.754 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.754 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1523 3.2s0 3:4afffc6f:::smithi10729407-20:head [append 0~507904] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-06-12T02:37:42.755 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.746564+0000", 2020-06-12T02:37:42.755 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 8.8006458940000005, 2020-06-12T02:37:42.755 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016382178000000001, 2020-06-12T02:37:42.755 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:42.755 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:42.756 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:42.756 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:42.756 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:42.757 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1523 2020-06-12T02:37:42.757 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.757 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:42.757 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.757 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:42.757 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.746564+0000", 2020-06-12T02:37:42.758 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:42.758 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.758 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.758 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:42.758 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.746564+0000", 2020-06-12T02:37:42.759 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013497300000000001 2020-06-12T02:37:42.759 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.759 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.759 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:42.759 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.746699+0000", 2020-06-12T02:37:42.759 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0549999999999999e-06 2020-06-12T02:37:42.760 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.760 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.760 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:42.760 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.746704+0000", 2020-06-12T02:37:42.760 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7439999999999996e-06 2020-06-12T02:37:42.761 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.761 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.761 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:42.761 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.746713+0000", 2020-06-12T02:37:42.761 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5009999999999998e-06 2020-06-12T02:37:42.762 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.762 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.762 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:42.762 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.746717+0000", 2020-06-12T02:37:42.762 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012975580000000001 2020-06-12T02:37:42.762 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.763 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.763 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.763 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.748015+0000", 2020-06-12T02:37:42.763 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5599000000000007e-05 2020-06-12T02:37:42.763 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.763 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.764 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:42.764 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.748080+0000", 2020-06-12T02:37:42.764 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044923640000000004 2020-06-12T02:37:42.764 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.764 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.765 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.765 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.752573+0000", 2020-06-12T02:37:42.765 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2525999999999998e-05 2020-06-12T02:37:42.765 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.765 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.765 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:42.766 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.752625+0000", 2020-06-12T02:37:42.766 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044074400000000002 2020-06-12T02:37:42.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.766 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.766 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:42.767 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.753066+0000", 2020-06-12T02:37:42.767 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0051370260000000003 2020-06-12T02:37:42.767 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.767 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.767 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:42.767 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.758203+0000", 2020-06-12T02:37:42.768 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046689590000000003 2020-06-12T02:37:42.768 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.768 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.768 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:42.768 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.762872+0000", 2020-06-12T02:37:42.769 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4128999999999994e-05 2020-06-12T02:37:42.769 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.769 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.769 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:42.769 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.762946+0000", 2020-06-12T02:37:42.769 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016382178000000001 2020-06-12T02:37:42.770 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.770 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:42.770 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.770 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.770 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.771 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1525 3.2s0 3:4afffc6f:::smithi10729407-20:head [append 0~286720] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-06-12T02:37:42.771 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.750576+0000", 2020-06-12T02:37:42.771 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 8.796634139, 2020-06-12T02:37:42.771 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015530921, 2020-06-12T02:37:42.772 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:42.772 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:42.772 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:42.772 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:42.772 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:42.773 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1525 2020-06-12T02:37:42.773 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.773 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:42.773 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.773 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:42.773 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750576+0000", 2020-06-12T02:37:42.774 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:42.774 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.774 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.774 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:42.774 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750576+0000", 2020-06-12T02:37:42.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7447000000000004e-05 2020-06-12T02:37:42.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.775 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:42.775 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750673+0000", 2020-06-12T02:37:42.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4089999999999996e-06 2020-06-12T02:37:42.776 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.776 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.776 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:42.776 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750677+0000", 2020-06-12T02:37:42.776 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.789e-06 2020-06-12T02:37:42.777 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.777 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.777 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:42.777 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750686+0000", 2020-06-12T02:37:42.777 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.967e-06 2020-06-12T02:37:42.777 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.778 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.778 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:42.778 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750691+0000", 2020-06-12T02:37:42.778 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4802000000000003e-05 2020-06-12T02:37:42.778 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.779 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.779 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.779 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750726+0000", 2020-06-12T02:37:42.779 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0620999999999999e-05 2020-06-12T02:37:42.779 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.779 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.780 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:42.780 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750817+0000", 2020-06-12T02:37:42.780 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0074052679999999996 2020-06-12T02:37:42.780 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.780 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.780 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.781 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.758222+0000", 2020-06-12T02:37:42.781 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5697000000000004e-05 2020-06-12T02:37:42.781 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.781 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.781 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:42.782 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.758298+0000", 2020-06-12T02:37:42.782 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00046273100000000003 2020-06-12T02:37:42.782 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.782 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.782 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:42.783 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.758760+0000", 2020-06-12T02:37:42.783 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0029510069999999998 2020-06-12T02:37:42.783 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.783 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.783 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:42.784 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.761711+0000", 2020-06-12T02:37:42.784 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043243439999999999 2020-06-12T02:37:42.784 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.784 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.784 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:42.785 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.766036+0000", 2020-06-12T02:37:42.785 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0839000000000004e-05 2020-06-12T02:37:42.785 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.785 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.785 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:42.790 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.766107+0000", 2020-06-12T02:37:42.791 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015530921 2020-06-12T02:37:42.791 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.791 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:42.791 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.791 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.792 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.792 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1526 3.2s0 3:4afffc6f:::smithi10729407-20:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-06-12T02:37:42.792 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.750720+0000", 2020-06-12T02:37:42.792 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 8.7964897329999996, 2020-06-12T02:37:42.792 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015683592999999999, 2020-06-12T02:37:42.793 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:42.793 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:42.793 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:42.793 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:42.793 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:42.793 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1526 2020-06-12T02:37:42.794 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.794 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:42.794 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.794 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:42.794 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750720+0000", 2020-06-12T02:37:42.795 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:42.795 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.795 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.795 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:42.795 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750720+0000", 2020-06-12T02:37:42.795 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8281999999999997e-05 2020-06-12T02:37:42.796 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.796 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.796 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:42.796 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750758+0000", 2020-06-12T02:37:42.796 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3189999999999999e-06 2020-06-12T02:37:42.797 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.797 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.797 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:42.797 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750762+0000", 2020-06-12T02:37:42.797 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8490000000000002e-06 2020-06-12T02:37:42.797 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.798 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.798 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:42.798 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750767+0000", 2020-06-12T02:37:42.798 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.4080000000000001e-06 2020-06-12T02:37:42.798 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.799 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.799 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:42.799 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750770+0000", 2020-06-12T02:37:42.799 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5030999999999994e-05 2020-06-12T02:37:42.799 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.799 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.800 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.800 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750845+0000", 2020-06-12T02:37:42.800 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8128000000000003e-05 2020-06-12T02:37:42.800 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.800 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.801 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:42.801 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750903+0000", 2020-06-12T02:37:42.801 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097174800000000006 2020-06-12T02:37:42.801 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.801 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.801 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.802 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.760620+0000", 2020-06-12T02:37:42.802 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044212800000000002 2020-06-12T02:37:42.802 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.802 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.802 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:42.803 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.761063+0000", 2020-06-12T02:37:42.803 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00033657099999999997 2020-06-12T02:37:42.803 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.803 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.803 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:42.803 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.761399+0000", 2020-06-12T02:37:42.804 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00046871599999999999 2020-06-12T02:37:42.804 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.804 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.804 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:42.804 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.761868+0000", 2020-06-12T02:37:42.805 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044489129999999997 2020-06-12T02:37:42.805 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.805 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.805 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:42.805 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.766317+0000", 2020-06-12T02:37:42.805 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6767999999999993e-05 2020-06-12T02:37:42.806 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.806 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.806 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:42.806 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.766404+0000", 2020-06-12T02:37:42.806 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015683592999999999 2020-06-12T02:37:42.807 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.807 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:42.807 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.807 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.807 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1527 3.2s0 3:4afffc6f:::smithi10729407-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e26)", 2020-06-12T02:37:42.808 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.750793+0000", 2020-06-12T02:37:42.808 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 8.7964164450000002, 2020-06-12T02:37:42.808 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016832799999999998, 2020-06-12T02:37:42.808 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:42.808 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:37:42.808 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:42.809 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:42.809 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:42.809 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1527 2020-06-12T02:37:42.809 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.809 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:42.810 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.810 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:42.810 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750793+0000", 2020-06-12T02:37:42.810 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:42.810 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.810 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.811 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:42.811 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750793+0000", 2020-06-12T02:37:42.811 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3218e-05 2020-06-12T02:37:42.811 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.811 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.812 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:42.812 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750817+0000", 2020-06-12T02:37:42.812 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0939999999999999e-06 2020-06-12T02:37:42.812 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.812 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.812 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:42.813 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750820+0000", 2020-06-12T02:37:42.813 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4810000000000003e-06 2020-06-12T02:37:42.813 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.813 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.813 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:42.814 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750826+0000", 2020-06-12T02:37:42.814 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.277e-06 2020-06-12T02:37:42.814 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.814 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.814 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:42.814 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750828+0000", 2020-06-12T02:37:42.815 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7738000000000005e-05 2020-06-12T02:37:42.815 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.815 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.815 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.815 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750926+0000", 2020-06-12T02:37:42.816 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6310000000000005e-05 2020-06-12T02:37:42.816 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.816 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.816 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:42.816 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.751012+0000", 2020-06-12T02:37:42.816 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010605622 2020-06-12T02:37:42.817 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.817 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.817 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.817 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.761618+0000", 2020-06-12T02:37:42.817 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2778000000000002e-05 2020-06-12T02:37:42.818 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.818 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.818 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:42.818 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.761671+0000", 2020-06-12T02:37:42.818 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.005069044 2020-06-12T02:37:42.818 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.819 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.819 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.819 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.766740+0000", 2020-06-12T02:37:42.819 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6335999999999997e-05 2020-06-12T02:37:42.819 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.820 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.820 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:42.820 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.766796+0000", 2020-06-12T02:37:42.820 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00082990199999999998 2020-06-12T02:37:42.820 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.820 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.821 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:42.821 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.767626+0000", 2020-06-12T02:37:42.821 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016832799999999998 2020-06-12T02:37:42.821 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.821 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:42.822 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.822 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.822 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.822 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s1 26/21 ECSubWrite(tid=408, reqid=client.4512.0:1572, at_version=26'99, trim_to=0'0, roll_forward_to=26'97))", 2020-06-12T02:37:42.822 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.887944+0000", 2020-06-12T02:37:42.822 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 8.6592659659999995, 2020-06-12T02:37:42.823 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019136641999999999, 2020-06-12T02:37:42.823 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:42.823 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:42.823 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:42.823 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.823 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:42.824 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.887944+0000", 2020-06-12T02:37:42.824 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:42.824 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.824 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.824 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:42.825 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.887944+0000", 2020-06-12T02:37:42.825 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00063708500000000002 2020-06-12T02:37:42.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.825 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.825 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:42.826 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888581+0000", 2020-06-12T02:37:42.826 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9644e-05 2020-06-12T02:37:42.826 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.826 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.826 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:42.826 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888600+0000", 2020-06-12T02:37:42.827 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.721e-05 2020-06-12T02:37:42.827 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.827 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.827 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:42.827 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888618+0000", 2020-06-12T02:37:42.828 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7500000000000003e-06 2020-06-12T02:37:42.828 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.828 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.828 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:42.828 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888625+0000", 2020-06-12T02:37:42.828 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2707999999999998e-05 2020-06-12T02:37:42.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.829 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.829 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.829 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888668+0000", 2020-06-12T02:37:42.829 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0491e-05 2020-06-12T02:37:42.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.830 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.830 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:42.830 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888679+0000", 2020-06-12T02:37:42.830 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018351985000000001 2020-06-12T02:37:42.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.831 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.831 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:42.831 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.907031+0000", 2020-06-12T02:37:42.831 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9768999999999997e-05 2020-06-12T02:37:42.831 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.831 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.832 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:42.832 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.907080+0000", 2020-06-12T02:37:42.832 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019136641999999999 2020-06-12T02:37:42.832 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.832 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:42.833 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.833 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.833 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.833 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s1 26/21 ECSubWrite(tid=409, reqid=client.4512.0:1573, at_version=26'100, trim_to=0'0, roll_forward_to=26'98))", 2020-06-12T02:37:42.833 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.892432+0000", 2020-06-12T02:37:42.834 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 8.6547773760000002, 2020-06-12T02:37:42.834 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01685733, 2020-06-12T02:37:42.834 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:42.834 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:42.834 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:42.834 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.835 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:42.835 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.892432+0000", 2020-06-12T02:37:42.835 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:42.835 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.835 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.835 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:42.836 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.892432+0000", 2020-06-12T02:37:42.836 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059245900000000004 2020-06-12T02:37:42.836 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.836 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.836 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:42.837 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.893025+0000", 2020-06-12T02:37:42.837 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1591e-05 2020-06-12T02:37:42.837 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.837 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.837 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:42.837 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.893046+0000", 2020-06-12T02:37:42.838 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2429000000000002e-05 2020-06-12T02:37:42.838 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.838 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.838 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:42.838 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.893069+0000", 2020-06-12T02:37:42.839 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0389999999999993e-06 2020-06-12T02:37:42.839 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.839 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.839 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:42.839 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.893078+0000", 2020-06-12T02:37:42.839 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012883099999999999 2020-06-12T02:37:42.840 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.840 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.840 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.840 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.893207+0000", 2020-06-12T02:37:42.840 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2407e-05 2020-06-12T02:37:42.841 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.841 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.841 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:42.841 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.893219+0000", 2020-06-12T02:37:42.841 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016029015000000001 2020-06-12T02:37:42.841 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.842 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.842 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:42.842 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.909248+0000", 2020-06-12T02:37:42.842 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1559e-05 2020-06-12T02:37:42.842 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.843 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.843 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:42.843 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.909290+0000", 2020-06-12T02:37:42.843 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01685733 2020-06-12T02:37:42.843 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.843 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:42.844 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.844 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.845 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.845 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1577 3.bs0 3:d09543d2:::smithi10729407-16:head [append 0~614400] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-06-12T02:37:42.845 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.895667+0000", 2020-06-12T02:37:42.845 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 8.6515422920000002, 2020-06-12T02:37:42.846 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015968431000000002, 2020-06-12T02:37:42.846 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:42.846 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:42.846 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:42.846 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:42.846 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:42.847 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1577 2020-06-12T02:37:42.847 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.847 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:42.847 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.847 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:42.848 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895667+0000", 2020-06-12T02:37:42.848 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:42.848 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.848 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.848 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:42.848 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895667+0000", 2020-06-12T02:37:42.849 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019894699999999999 2020-06-12T02:37:42.849 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.849 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.849 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:42.849 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895866+0000", 2020-06-12T02:37:42.850 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0057e-05 2020-06-12T02:37:42.850 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.850 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.850 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:42.850 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895876+0000", 2020-06-12T02:37:42.850 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5666000000000001e-05 2020-06-12T02:37:42.851 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.851 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.851 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:42.851 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895902+0000", 2020-06-12T02:37:42.851 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6510000000000001e-06 2020-06-12T02:37:42.852 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.852 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.852 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:42.852 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895910+0000", 2020-06-12T02:37:42.852 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5513000000000003e-05 2020-06-12T02:37:42.852 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.853 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.853 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.853 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895955+0000", 2020-06-12T02:37:42.853 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000106716 2020-06-12T02:37:42.853 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.854 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.854 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:42.854 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.896062+0000", 2020-06-12T02:37:42.854 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00063197000000000004 2020-06-12T02:37:42.854 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.854 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.855 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:42.855 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.896694+0000", 2020-06-12T02:37:42.855 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014775 2020-06-12T02:37:42.855 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.855 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.855 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:42.856 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.911469+0000", 2020-06-12T02:37:42.856 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4872999999999999e-05 2020-06-12T02:37:42.856 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.856 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.856 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:42.857 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.911554+0000", 2020-06-12T02:37:42.857 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2038000000000003e-05 2020-06-12T02:37:42.857 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.857 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.857 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:42.857 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.911636+0000", 2020-06-12T02:37:42.858 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015968431000000002 2020-06-12T02:37:42.858 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.858 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:42.858 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.858 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.859 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.859 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:3329 3.bs0 3:d09543d2:::smithi10729407-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:37:42.859 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:38.600071+0000", 2020-06-12T02:37:42.859 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.9471383229999999, 2020-06-12T02:37:42.859 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016512401999999999, 2020-06-12T02:37:42.859 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:42.860 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:37:42.860 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:42.860 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:42.860 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:42.860 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 3329 2020-06-12T02:37:42.861 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.861 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:42.861 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.861 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:42.861 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.600071+0000", 2020-06-12T02:37:42.861 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:42.862 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.862 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.862 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:42.862 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.600071+0000", 2020-06-12T02:37:42.862 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5961999999999998e-05 2020-06-12T02:37:42.863 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.863 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.863 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:42.863 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.600117+0000", 2020-06-12T02:37:42.863 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6890000000000001e-06 2020-06-12T02:37:42.863 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.864 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.864 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:42.864 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.600123+0000", 2020-06-12T02:37:42.864 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2027e-05 2020-06-12T02:37:42.864 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.865 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.865 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:42.865 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.600135+0000", 2020-06-12T02:37:42.865 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.9840000000000003e-06 2020-06-12T02:37:42.865 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.865 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.866 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:42.866 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.600139+0000", 2020-06-12T02:37:42.866 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0073161570000000002 2020-06-12T02:37:42.866 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.866 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.867 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.867 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.607455+0000", 2020-06-12T02:37:42.867 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8794000000000004e-05 2020-06-12T02:37:42.867 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.867 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.867 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:42.868 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.607544+0000", 2020-06-12T02:37:42.868 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0068056540000000004 2020-06-12T02:37:42.868 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.868 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.868 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.869 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.614350+0000", 2020-06-12T02:37:42.869 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6471999999999994e-05 2020-06-12T02:37:42.869 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.869 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.869 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:42.869 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.614446+0000", 2020-06-12T02:37:42.870 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0021376630000000001 2020-06-12T02:37:42.870 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.870 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.870 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:42.870 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.616584+0000", 2020-06-12T02:37:42.871 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016512401999999999 2020-06-12T02:37:42.871 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.871 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:42.871 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.871 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.872 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.872 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:3588 3.8s0 3:14244450:::smithi10729407-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:37:42.872 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:39.240636+0000", 2020-06-12T02:37:42.872 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.3065736010000002, 2020-06-12T02:37:42.872 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0153145, 2020-06-12T02:37:42.873 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:42.873 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:37:42.873 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:42.873 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:42.873 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:42.874 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 3588 2020-06-12T02:37:42.874 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.874 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:42.874 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.874 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:42.874 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.240636+0000", 2020-06-12T02:37:42.875 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:42.875 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.875 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.875 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:42.875 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.240636+0000", 2020-06-12T02:37:42.876 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1634e-05 2020-06-12T02:37:42.876 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.876 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.876 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:42.876 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.240658+0000", 2020-06-12T02:37:42.876 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9689999999999999e-06 2020-06-12T02:37:42.877 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.877 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.877 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:42.877 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.240661+0000", 2020-06-12T02:37:42.877 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3459999999999999e-06 2020-06-12T02:37:42.878 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.878 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.878 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:42.878 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.240667+0000", 2020-06-12T02:37:42.878 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3690000000000001e-06 2020-06-12T02:37:42.878 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.879 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.879 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:42.879 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.240669+0000", 2020-06-12T02:37:42.879 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010317392999999999 2020-06-12T02:37:42.879 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.880 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.880 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.880 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.250987+0000", 2020-06-12T02:37:42.880 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9679999999999997e-05 2020-06-12T02:37:42.880 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.880 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.881 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:42.881 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.251057+0000", 2020-06-12T02:37:42.881 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0026745390000000001 2020-06-12T02:37:42.881 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.881 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.882 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.882 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.253731+0000", 2020-06-12T02:37:42.882 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011061999999999999 2020-06-12T02:37:42.882 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.882 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.882 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:42.883 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.253842+0000", 2020-06-12T02:37:42.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0021089500000000001 2020-06-12T02:37:42.883 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.883 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:42.884 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.255951+0000", 2020-06-12T02:37:42.884 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0153145 2020-06-12T02:37:42.884 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.884 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:42.885 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.885 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.885 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:3607 3.7s0 3:e653d00f:::smithi10729407-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:37:42.885 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:39.292176+0000", 2020-06-12T02:37:42.885 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.2550338860000001, 2020-06-12T02:37:42.886 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016904665999999999, 2020-06-12T02:37:42.886 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:42.886 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:37:42.886 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:42.886 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:42.887 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:42.887 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 3607 2020-06-12T02:37:42.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.887 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:42.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:42.888 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.292176+0000", 2020-06-12T02:37:42.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:42.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.888 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:42.889 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.292176+0000", 2020-06-12T02:37:42.889 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4786000000000001e-05 2020-06-12T02:37:42.889 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.889 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.889 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:42.889 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.292221+0000", 2020-06-12T02:37:42.890 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0159999999999997e-06 2020-06-12T02:37:42.890 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.890 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.890 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:42.890 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.292227+0000", 2020-06-12T02:37:42.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2269999999999999e-05 2020-06-12T02:37:42.891 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.891 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.891 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:42.891 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.292239+0000", 2020-06-12T02:37:42.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1849999999999997e-06 2020-06-12T02:37:42.892 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.892 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.892 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:42.892 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.292243+0000", 2020-06-12T02:37:42.892 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0077395820000000001 2020-06-12T02:37:42.892 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.893 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.893 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.893 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.299983+0000", 2020-06-12T02:37:42.893 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7809999999999997e-05 2020-06-12T02:37:42.893 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.894 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.894 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:42.894 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.300040+0000", 2020-06-12T02:37:42.894 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0049104680000000003 2020-06-12T02:37:42.894 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.894 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.895 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:42.895 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.304951+0000", 2020-06-12T02:37:42.895 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2177999999999997e-05 2020-06-12T02:37:42.895 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.895 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.896 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:42.896 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.305043+0000", 2020-06-12T02:37:42.896 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0040373709999999997 2020-06-12T02:37:42.896 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:42.896 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:42.896 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:42.897 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.309081+0000", 2020-06-12T02:37:42.897 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016904665999999999 2020-06-12T02:37:42.897 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.897 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:42.897 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.898 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:42.898 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:42.898 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:37:42.898 INFO:tasks.rados.rados.0.smithi107.stdout:2124: writing smithi10729407-11 from 1466368 to 1875968 tid 3 2020-06-12T02:37:42.898 INFO:tasks.rados.rados.0.smithi107.stdout:2124: writing smithi10729407-11 from 1875968 to 1966080 tid 4 2020-06-12T02:37:42.899 INFO:tasks.rados.rados.0.smithi107.stdout:2125: delete oid 12 current snap is 0 2020-06-12T02:37:42.899 INFO:tasks.rados.rados.0.smithi107.stdout:2119: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:42.899 INFO:tasks.rados.rados.0.smithi107.stdout:2119: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:42.899 INFO:tasks.rados.rados.0.smithi107.stdout:2119: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:42.899 INFO:tasks.rados.rados.0.smithi107.stdout:2119: finishing write tid 6 to smithi10729407-2 2020-06-12T02:37:42.899 INFO:tasks.rados.rados.0.smithi107.stdout:2119: finishing write tid 7 to smithi10729407-2 2020-06-12T02:37:42.900 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 217 (ObjNum 2118 snap 0 seq_num 2118) dirty exists 2020-06-12T02:37:42.900 INFO:tasks.rados.rados.0.smithi107.stdout:2119: left oid 2 (ObjNum 2118 snap 0 seq_num 2118) 2020-06-12T02:37:42.900 INFO:tasks.rados.rados.0.smithi107.stdout:2122: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:42.900 INFO:tasks.rados.rados.0.smithi107.stdout:2122: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:42.900 INFO:tasks.rados.rados.0.smithi107.stdout:2122: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:42.901 INFO:tasks.rados.rados.0.smithi107.stdout:2122: finishing write tid 5 to smithi10729407-15 2020-06-12T02:37:42.901 INFO:tasks.rados.rados.0.smithi107.stdout:2123: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:42.901 INFO:tasks.rados.rados.0.smithi107.stdout:2122: finishing write tid 6 to smithi10729407-15 2020-06-12T02:37:42.901 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 361 (ObjNum 2121 snap 0 seq_num 2121) dirty exists 2020-06-12T02:37:42.901 INFO:tasks.rados.rados.0.smithi107.stdout:2122: left oid 15 (ObjNum 2121 snap 0 seq_num 2121) 2020-06-12T02:37:42.901 INFO:tasks.rados.rados.0.smithi107.stdout:2123: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:42.902 INFO:tasks.rados.rados.0.smithi107.stdout:2123: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:42.920 INFO:tasks.rados.rados.0.smithi107.stdout:2119: done (4 left) 2020-06-12T02:37:42.920 INFO:tasks.rados.rados.0.smithi107.stdout:2122: done (3 left) 2020-06-12T02:37:42.920 INFO:tasks.rados.rados.0.smithi107.stdout:2125: done (2 left) 2020-06-12T02:37:42.921 INFO:tasks.rados.rados.0.smithi107.stdout:2126: delete oid 15 current snap is 0 2020-06-12T02:37:42.921 INFO:tasks.rados.rados.0.smithi107.stdout:2124: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:42.921 INFO:tasks.rados.rados.0.smithi107.stdout:2123: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:42.921 INFO:tasks.rados.rados.0.smithi107.stdout:2123: finishing write tid 6 to smithi10729407-18 2020-06-12T02:37:42.921 INFO:tasks.rados.rados.0.smithi107.stdout:2126: done (2 left) 2020-06-12T02:37:42.922 INFO:tasks.rados.rados.0.smithi107.stdout:2127: delete oid 7 current snap is 0 2020-06-12T02:37:42.922 INFO:tasks.rados.rados.0.smithi107.stdout:2123: finishing write tid 7 to smithi10729407-18 2020-06-12T02:37:42.922 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 366 (ObjNum 2122 snap 0 seq_num 2122) dirty exists 2020-06-12T02:37:42.922 INFO:tasks.rados.rados.0.smithi107.stdout:2123: left oid 18 (ObjNum 2122 snap 0 seq_num 2122) 2020-06-12T02:37:42.922 INFO:tasks.rados.rados.0.smithi107.stdout:2124: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:42.923 INFO:tasks.rados.rados.0.smithi107.stdout:2124: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:42.923 INFO:tasks.rados.rados.0.smithi107.stdout:2123: done (2 left) 2020-06-12T02:37:42.923 INFO:tasks.rados.rados.0.smithi107.stdout:2127: done (1 left) 2020-06-12T02:37:42.923 INFO:tasks.rados.rados.0.smithi107.stdout:2128: delete oid 19 current snap is 0 2020-06-12T02:37:42.923 INFO:tasks.rados.rados.0.smithi107.stdout:2124: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:42.923 INFO:tasks.rados.rados.0.smithi107.stdout:2124: finishing write tid 6 to smithi10729407-11 2020-06-12T02:37:42.924 INFO:tasks.rados.rados.0.smithi107.stdout:2128: done (1 left) 2020-06-12T02:37:42.924 INFO:tasks.rados.rados.0.smithi107.stdout:2129: delete oid 1 current snap is 0 2020-06-12T02:37:42.924 INFO:tasks.rados.rados.0.smithi107.stdout:2124: finishing write tid 7 to smithi10729407-11 2020-06-12T02:37:42.924 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 389 (ObjNum 2123 snap 0 seq_num 2123) dirty exists 2020-06-12T02:37:42.924 INFO:tasks.rados.rados.0.smithi107.stdout:2124: left oid 11 (ObjNum 2123 snap 0 seq_num 2123) 2020-06-12T02:37:42.925 INFO:tasks.rados.rados.0.smithi107.stdout:2124: done (1 left) 2020-06-12T02:37:42.925 INFO:tasks.rados.rados.0.smithi107.stdout:2129: done (0 left) 2020-06-12T02:37:42.925 INFO:tasks.rados.rados.0.smithi107.stdout:2130: delete oid 11 current snap is 0 2020-06-12T02:37:42.925 INFO:tasks.rados.rados.0.smithi107.stdout:2130: done (0 left) 2020-06-12T02:37:42.925 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:42.925 INFO:tasks.rados.rados.0.smithi107.stdout:2131: seq_num 2130 ranges {1884160=475136,2359296=745472,3104768=565248,3670016=294912} 2020-06-12T02:37:42.926 INFO:tasks.rados.rados.0.smithi107.stdout:2131: writing smithi10729407-2 from 1884160 to 2359296 tid 1 2020-06-12T02:37:42.926 INFO:tasks.rados.rados.0.smithi107.stdout:2131: writing smithi10729407-2 from 2359296 to 3104768 tid 2 2020-06-12T02:37:42.926 INFO:tasks.rados.rados.0.smithi107.stdout:2131: writing smithi10729407-2 from 3104768 to 3670016 tid 3 2020-06-12T02:37:42.926 INFO:tasks.rados.rados.0.smithi107.stdout:2131: writing smithi10729407-2 from 3670016 to 3964928 tid 4 2020-06-12T02:37:42.926 INFO:tasks.rados.rados.0.smithi107.stdout:2132: delete oid 5 current snap is 0 2020-06-12T02:37:42.927 INFO:tasks.rados.rados.0.smithi107.stdout:2131: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:42.927 INFO:tasks.rados.rados.0.smithi107.stdout:2132: done (1 left) 2020-06-12T02:37:42.927 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:37:42.927 INFO:tasks.rados.rados.0.smithi107.stdout:2133: seq_num 2132 ranges {0=516096,516096=516096,1032192=532480,1564672=417792,1982464=139264} 2020-06-12T02:37:42.927 INFO:tasks.rados.rados.0.smithi107.stdout:2133: writing smithi10729407-11 from 0 to 516096 tid 1 2020-06-12T02:37:42.927 INFO:tasks.rados.rados.0.smithi107.stdout:2133: writing smithi10729407-11 from 516096 to 1032192 tid 2 2020-06-12T02:37:42.928 INFO:tasks.rados.rados.0.smithi107.stdout:2133: writing smithi10729407-11 from 1032192 to 1564672 tid 3 2020-06-12T02:37:42.928 INFO:tasks.rados.rados.0.smithi107.stdout:2133: writing smithi10729407-11 from 1564672 to 1982464 tid 4 2020-06-12T02:37:42.928 INFO:tasks.rados.rados.0.smithi107.stdout:2133: writing smithi10729407-11 from 1982464 to 2121728 tid 5 2020-06-12T02:37:42.928 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:37:42.928 INFO:tasks.rados.rados.0.smithi107.stdout:2134: seq_num 2133 ranges {1835008=532480,2367488=794624,3162112=737280,3899392=73728} 2020-06-12T02:37:42.929 INFO:tasks.rados.rados.0.smithi107.stdout:2134: writing smithi10729407-13 from 1835008 to 2367488 tid 1 2020-06-12T02:37:42.929 INFO:tasks.rados.rados.0.smithi107.stdout:2134: writing smithi10729407-13 from 2367488 to 3162112 tid 2 2020-06-12T02:37:42.929 INFO:tasks.rados.rados.0.smithi107.stdout:2134: writing smithi10729407-13 from 3162112 to 3899392 tid 3 2020-06-12T02:37:42.929 INFO:tasks.rados.rados.0.smithi107.stdout:2134: writing smithi10729407-13 from 3899392 to 3973120 tid 4 2020-06-12T02:37:42.929 INFO:tasks.rados.rados.0.smithi107.stdout:2135: delete oid 19 current snap is 0 2020-06-12T02:37:42.929 INFO:tasks.rados.rados.0.smithi107.stdout:2131: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:42.930 INFO:tasks.rados.rados.0.smithi107.stdout:2131: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:42.930 INFO:tasks.rados.rados.0.smithi107.stdout:2131: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:42.930 INFO:tasks.rados.rados.0.smithi107.stdout:2131: finishing write tid 6 to smithi10729407-2 2020-06-12T02:37:42.930 INFO:tasks.rados.rados.0.smithi107.stdout:2131: finishing write tid 7 to smithi10729407-2 2020-06-12T02:37:42.930 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 222 (ObjNum 2130 snap 0 seq_num 2130) dirty exists 2020-06-12T02:37:42.931 INFO:tasks.rados.rados.0.smithi107.stdout:2131: left oid 2 (ObjNum 2130 snap 0 seq_num 2130) 2020-06-12T02:37:42.931 INFO:tasks.rados.rados.0.smithi107.stdout:2133: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:42.931 INFO:tasks.rados.rados.0.smithi107.stdout:2133: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:42.931 INFO:tasks.rados.rados.0.smithi107.stdout:2133: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:42.931 INFO:tasks.rados.rados.0.smithi107.stdout:2133: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:42.931 INFO:tasks.rados.rados.0.smithi107.stdout:2133: finishing write tid 5 to smithi10729407-11 2020-06-12T02:37:42.932 INFO:tasks.rados.rados.0.smithi107.stdout:2133: finishing write tid 7 to smithi10729407-11 2020-06-12T02:37:42.932 INFO:tasks.rados.rados.0.smithi107.stdout:2133: finishing write tid 8 to smithi10729407-11 2020-06-12T02:37:42.932 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 396 (ObjNum 2132 snap 0 seq_num 2132) dirty exists 2020-06-12T02:37:42.932 INFO:tasks.rados.rados.0.smithi107.stdout:2133: left oid 11 (ObjNum 2132 snap 0 seq_num 2132) 2020-06-12T02:37:42.932 INFO:tasks.rados.rados.0.smithi107.stdout:2134: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:42.932 INFO:tasks.rados.rados.0.smithi107.stdout:2131: done (3 left) 2020-06-12T02:37:42.933 INFO:tasks.rados.rados.0.smithi107.stdout:2133: done (2 left) 2020-06-12T02:37:42.933 INFO:tasks.rados.rados.0.smithi107.stdout:2135: done (1 left) 2020-06-12T02:37:42.933 INFO:tasks.rados.rados.0.smithi107.stdout:2136: delete oid 5 current snap is 0 2020-06-12T02:37:42.933 INFO:tasks.rados.rados.0.smithi107.stdout:2136: done (1 left) 2020-06-12T02:37:42.933 INFO:tasks.rados.rados.0.smithi107.stdout:2137: delete oid 10 current snap is 0 2020-06-12T02:37:42.934 INFO:tasks.rados.rados.0.smithi107.stdout:2134: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:42.934 INFO:tasks.rados.rados.0.smithi107.stdout:2137: done (1 left) 2020-06-12T02:37:42.934 INFO:tasks.rados.rados.0.smithi107.stdout:2138: delete oid 17 current snap is 0 2020-06-12T02:37:42.934 INFO:tasks.rados.rados.0.smithi107.stdout:2134: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:42.934 INFO:tasks.rados.rados.0.smithi107.stdout:2134: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:42.934 INFO:tasks.rados.rados.0.smithi107.stdout:2134: finishing write tid 6 to smithi10729407-13 2020-06-12T02:37:42.935 INFO:tasks.rados.rados.0.smithi107.stdout:2138: done (1 left) 2020-06-12T02:37:42.935 INFO:tasks.rados.rados.0.smithi107.stdout:2139: delete oid 16 current snap is 0 2020-06-12T02:37:42.935 INFO:tasks.rados.rados.0.smithi107.stdout:2134: finishing write tid 7 to smithi10729407-13 2020-06-12T02:37:42.935 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 410 (ObjNum 2133 snap 0 seq_num 2133) dirty exists 2020-06-12T02:37:42.935 INFO:tasks.rados.rados.0.smithi107.stdout:2134: left oid 13 (ObjNum 2133 snap 0 seq_num 2133) 2020-06-12T02:37:42.936 INFO:tasks.rados.rados.0.smithi107.stdout:2134: done (1 left) 2020-06-12T02:37:42.936 INFO:tasks.rados.rados.0.smithi107.stdout:2139: done (0 left) 2020-06-12T02:37:42.936 INFO:tasks.rados.rados.0.smithi107.stdout:2140: delete oid 19 current snap is 0 2020-06-12T02:37:42.936 INFO:tasks.rados.rados.0.smithi107.stdout:2140: done (0 left) 2020-06-12T02:37:42.936 INFO:tasks.rados.rados.0.smithi107.stdout:2141: delete oid 4 current snap is 0 2020-06-12T02:37:42.936 INFO:tasks.rados.rados.0.smithi107.stdout:2141: done (0 left) 2020-06-12T02:37:42.937 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:37:42.937 INFO:tasks.rados.rados.0.smithi107.stdout:2142: seq_num 2141 ranges {0=663552,663552=720896,1384448=696320,2080768=188416} 2020-06-12T02:37:42.937 INFO:tasks.rados.rados.0.smithi107.stdout:2142: writing smithi10729407-17 from 0 to 663552 tid 1 2020-06-12T02:37:42.937 INFO:tasks.rados.rados.0.smithi107.stdout:2142: writing smithi10729407-17 from 663552 to 1384448 tid 2 2020-06-12T02:37:42.937 INFO:tasks.rados.rados.0.smithi107.stdout:2142: writing smithi10729407-17 from 1384448 to 2080768 tid 3 2020-06-12T02:37:42.938 INFO:tasks.rados.rados.0.smithi107.stdout:2142: writing smithi10729407-17 from 2080768 to 2269184 tid 4 2020-06-12T02:37:42.938 INFO:tasks.rados.rados.0.smithi107.stdout:2143: delete oid 3 current snap is 0 2020-06-12T02:37:42.938 INFO:tasks.rados.rados.0.smithi107.stdout:2142: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:42.938 INFO:tasks.rados.rados.0.smithi107.stdout:2143: done (1 left) 2020-06-12T02:37:42.938 INFO:tasks.rados.rados.0.smithi107.stdout:2144: delete oid 6 current snap is 0 2020-06-12T02:37:42.938 INFO:tasks.rados.rados.0.smithi107.stdout:2144: done (1 left) 2020-06-12T02:37:42.939 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:37:42.939 INFO:tasks.rados.rados.0.smithi107.stdout:2145: seq_num 2144 ranges {0=712704,712704=425984,1138688=516096,1654784=581632,2236416=81920} 2020-06-12T02:37:42.939 INFO:tasks.rados.rados.0.smithi107.stdout:2145: writing smithi10729407-8 from 0 to 712704 tid 1 2020-06-12T02:37:42.939 INFO:tasks.rados.rados.0.smithi107.stdout:2145: writing smithi10729407-8 from 712704 to 1138688 tid 2 2020-06-12T02:37:42.939 INFO:tasks.rados.rados.0.smithi107.stdout:2145: writing smithi10729407-8 from 1138688 to 1654784 tid 3 2020-06-12T02:37:42.940 INFO:tasks.rados.rados.0.smithi107.stdout:2145: writing smithi10729407-8 from 1654784 to 2236416 tid 4 2020-06-12T02:37:42.940 INFO:tasks.rados.rados.0.smithi107.stdout:2145: writing smithi10729407-8 from 2236416 to 2318336 tid 5 2020-06-12T02:37:42.940 INFO:tasks.rados.rados.0.smithi107.stdout:2146: delete oid 7 current snap is 0 2020-06-12T02:37:42.940 INFO:tasks.rados.rados.0.smithi107.stdout:2142: finishing write tid 2 to smithi10729407-17 2020-06-12T02:37:42.940 INFO:tasks.rados.rados.0.smithi107.stdout:2142: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:42.940 INFO:tasks.rados.rados.0.smithi107.stdout:2142: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:42.941 INFO:tasks.rados.rados.0.smithi107.stdout:2142: finishing write tid 6 to smithi10729407-17 2020-06-12T02:37:42.941 INFO:tasks.rados.rados.0.smithi107.stdout:2142: finishing write tid 7 to smithi10729407-17 2020-06-12T02:37:42.941 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 414 (ObjNum 2141 snap 0 seq_num 2141) dirty exists 2020-06-12T02:37:42.941 INFO:tasks.rados.rados.0.smithi107.stdout:2142: left oid 17 (ObjNum 2141 snap 0 seq_num 2141) 2020-06-12T02:37:42.941 INFO:tasks.rados.rados.0.smithi107.stdout:2142: done (2 left) 2020-06-12T02:37:42.942 INFO:tasks.rados.rados.0.smithi107.stdout:2146: done (1 left) 2020-06-12T02:37:42.942 INFO:tasks.rados.rados.0.smithi107.stdout:2147: delete oid 9 current snap is 0 2020-06-12T02:37:42.942 INFO:tasks.rados.rados.0.smithi107.stdout:2147: done (1 left) 2020-06-12T02:37:42.942 INFO:tasks.rados.rados.0.smithi107.stdout:2148: delete oid 14 current snap is 0 2020-06-12T02:37:42.942 INFO:tasks.rados.rados.0.smithi107.stdout:2145: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:42.942 INFO:tasks.rados.rados.0.smithi107.stdout:2145: finishing write tid 2 to smithi10729407-8 2020-06-12T02:37:42.943 INFO:tasks.rados.rados.0.smithi107.stdout:2145: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:42.943 INFO:tasks.rados.rados.0.smithi107.stdout:2148: done (1 left) 2020-06-12T02:37:42.943 INFO:tasks.rados.rados.0.smithi107.stdout:2149: delete oid 4 current snap is 0 2020-06-12T02:37:42.943 INFO:tasks.rados.rados.0.smithi107.stdout:2145: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:42.943 INFO:tasks.rados.rados.0.smithi107.stdout:2145: finishing write tid 5 to smithi10729407-8 2020-06-12T02:37:42.943 INFO:tasks.rados.rados.0.smithi107.stdout:2145: finishing write tid 7 to smithi10729407-8 2020-06-12T02:37:42.944 INFO:tasks.rados.rados.0.smithi107.stdout:2149: done (1 left) 2020-06-12T02:37:42.944 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:37:42.944 INFO:tasks.rados.rados.0.smithi107.stdout:2150: seq_num 2149 ranges {3964928=794624,4759552=647168,5406720=655360,6062080=303104} 2020-06-12T02:37:42.944 INFO:tasks.rados.rados.0.smithi107.stdout:2150: writing smithi10729407-2 from 3964928 to 4759552 tid 1 2020-06-12T02:37:42.944 INFO:tasks.rados.rados.0.smithi107.stdout:2150: writing smithi10729407-2 from 4759552 to 5406720 tid 2 2020-06-12T02:37:42.945 INFO:tasks.rados.rados.0.smithi107.stdout:2150: writing smithi10729407-2 from 5406720 to 6062080 tid 3 2020-06-12T02:37:42.945 INFO:tasks.rados.rados.0.smithi107.stdout:2150: writing smithi10729407-2 from 6062080 to 6365184 tid 4 2020-06-12T02:37:42.945 INFO:tasks.rados.rados.0.smithi107.stdout:2151: delete oid 5 current snap is 0 2020-06-12T02:37:42.945 INFO:tasks.rados.rados.0.smithi107.stdout:2145: finishing write tid 8 to smithi10729407-8 2020-06-12T02:37:42.945 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 368 (ObjNum 2144 snap 0 seq_num 2144) dirty exists 2020-06-12T02:37:42.945 INFO:tasks.rados.rados.0.smithi107.stdout:2145: left oid 8 (ObjNum 2144 snap 0 seq_num 2144) 2020-06-12T02:37:42.946 INFO:tasks.rados.rados.0.smithi107.stdout:2150: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:42.946 INFO:tasks.rados.rados.0.smithi107.stdout:2145: done (2 left) 2020-06-12T02:37:42.946 INFO:tasks.rados.rados.0.smithi107.stdout:2151: done (1 left) 2020-06-12T02:37:42.946 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:37:42.946 INFO:tasks.rados.rados.0.smithi107.stdout:2152: seq_num 2151 ranges {0=32768} 2020-06-12T02:37:42.947 INFO:tasks.rados.rados.0.smithi107.stdout:2152: writing smithi10729407-12 from 0 to 32768 tid 1 2020-06-12T02:37:42.947 INFO:tasks.rados.rados.0.smithi107.stdout:2153: delete oid 6 current snap is 0 2020-06-12T02:37:42.947 INFO:tasks.rados.rados.0.smithi107.stdout:2153: done (2 left) 2020-06-12T02:37:42.947 INFO:tasks.rados.rados.0.smithi107.stdout:2154: delete oid 18 current snap is 0 2020-06-12T02:37:42.947 INFO:tasks.rados.rados.0.smithi107.stdout:2152: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:42.947 INFO:tasks.rados.rados.0.smithi107.stdout:2152: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:42.948 INFO:tasks.rados.rados.0.smithi107.stdout:2150: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:42.948 INFO:tasks.rados.rados.0.smithi107.stdout:2152: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:42.948 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 327 (ObjNum 2151 snap 0 seq_num 2151) dirty exists 2020-06-12T02:37:42.948 INFO:tasks.rados.rados.0.smithi107.stdout:2152: left oid 12 (ObjNum 2151 snap 0 seq_num 2151) 2020-06-12T02:37:42.948 INFO:tasks.rados.rados.0.smithi107.stdout:2150: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:42.949 INFO:tasks.rados.rados.0.smithi107.stdout:2152: done (2 left) 2020-06-12T02:37:42.949 INFO:tasks.rados.rados.0.smithi107.stdout:2154: done (1 left) 2020-06-12T02:37:42.949 INFO:tasks.rados.rados.0.smithi107.stdout:2155: delete oid 3 current snap is 0 2020-06-12T02:37:42.949 INFO:tasks.rados.rados.0.smithi107.stdout:2150: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:42.949 INFO:tasks.rados.rados.0.smithi107.stdout:2150: finishing write tid 6 to smithi10729407-2 2020-06-12T02:37:42.949 INFO:tasks.rados.rados.0.smithi107.stdout:2155: done (1 left) 2020-06-12T02:37:42.950 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:37:42.950 INFO:tasks.rados.rados.0.smithi107.stdout:2156: seq_num 2155 ranges {0=98304} 2020-06-12T02:37:42.950 INFO:tasks.rados.rados.0.smithi107.stdout:2156: writing smithi10729407-1 from 0 to 98304 tid 1 2020-06-12T02:37:42.950 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:37:42.950 INFO:tasks.rados.rados.0.smithi107.stdout:2157: seq_num 2156 ranges {0=114688} 2020-06-12T02:37:42.951 INFO:tasks.rados.rados.0.smithi107.stdout:2157: writing smithi10729407-18 from 0 to 114688 tid 1 2020-06-12T02:37:42.951 INFO:tasks.rados.rados.0.smithi107.stdout:2158: delete oid 14 current snap is 0 2020-06-12T02:37:42.951 INFO:tasks.rados.rados.0.smithi107.stdout:2150: finishing write tid 7 to smithi10729407-2 2020-06-12T02:37:42.951 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 227 (ObjNum 2149 snap 0 seq_num 2149) dirty exists 2020-06-12T02:37:42.951 INFO:tasks.rados.rados.0.smithi107.stdout:2150: left oid 2 (ObjNum 2149 snap 0 seq_num 2149) 2020-06-12T02:37:42.951 INFO:tasks.rados.rados.0.smithi107.stdout:2156: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:42.952 INFO:tasks.rados.rados.0.smithi107.stdout:2157: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:42.952 INFO:tasks.rados.rados.0.smithi107.stdout:2150: done (3 left) 2020-06-12T02:37:42.952 INFO:tasks.rados.rados.0.smithi107.stdout:2158: done (2 left) 2020-06-12T02:37:42.952 INFO:tasks.rados.rados.0.smithi107.stdout:2159: delete oid 7 current snap is 0 2020-06-12T02:37:42.952 INFO:tasks.rados.rados.0.smithi107.stdout:2157: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:42.953 INFO:tasks.rados.rados.0.smithi107.stdout:2156: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:42.953 INFO:tasks.rados.rados.0.smithi107.stdout:2159: done (2 left) 2020-06-12T02:37:42.953 INFO:tasks.rados.rados.0.smithi107.stdout:2160: delete oid 4 current snap is 0 2020-06-12T02:37:42.953 INFO:tasks.rados.rados.0.smithi107.stdout:2157: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:42.953 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 370 (ObjNum 2156 snap 0 seq_num 2156) dirty exists 2020-06-12T02:37:42.954 INFO:tasks.rados.rados.0.smithi107.stdout:2157: left oid 18 (ObjNum 2156 snap 0 seq_num 2156) 2020-06-12T02:37:42.954 INFO:tasks.rados.rados.0.smithi107.stdout:2156: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:42.954 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 446 (ObjNum 2155 snap 0 seq_num 2155) dirty exists 2020-06-12T02:37:42.954 INFO:tasks.rados.rados.0.smithi107.stdout:2156: left oid 1 (ObjNum 2155 snap 0 seq_num 2155) 2020-06-12T02:37:42.954 INFO:tasks.rados.rados.0.smithi107.stdout:2156: done (2 left) 2020-06-12T02:37:42.954 INFO:tasks.rados.rados.0.smithi107.stdout:2157: done (1 left) 2020-06-12T02:37:42.955 INFO:tasks.rados.rados.0.smithi107.stdout:2160: done (0 left) 2020-06-12T02:37:42.955 INFO:tasks.rados.rados.0.smithi107.stdout:2161: delete oid 10 current snap is 0 2020-06-12T02:37:42.955 INFO:tasks.rados.rados.0.smithi107.stdout:2161: done (0 left) 2020-06-12T02:37:42.955 INFO:tasks.rados.rados.0.smithi107.stdout:2162: delete oid 17 current snap is 0 2020-06-12T02:37:42.955 INFO:tasks.rados.rados.0.smithi107.stdout:2162: done (0 left) 2020-06-12T02:37:42.956 INFO:tasks.rados.rados.0.smithi107.stdout:2163: delete oid 15 current snap is 0 2020-06-12T02:37:42.956 INFO:tasks.rados.rados.0.smithi107.stdout:2163: done (0 left) 2020-06-12T02:37:42.956 INFO:tasks.rados.rados.0.smithi107.stdout:2164: delete oid 5 current snap is 0 2020-06-12T02:37:42.956 INFO:tasks.rados.rados.0.smithi107.stdout:2164: done (0 left) 2020-06-12T02:37:42.956 INFO:tasks.rados.rados.0.smithi107.stdout:2165: delete oid 18 current snap is 0 2020-06-12T02:37:42.956 INFO:tasks.rados.rados.0.smithi107.stdout:2165: done (0 left) 2020-06-12T02:37:42.957 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:37:42.957 INFO:tasks.rados.rados.0.smithi107.stdout:2166: seq_num 2165 ranges {2318336=262144} 2020-06-12T02:37:42.957 INFO:tasks.rados.rados.0.smithi107.stdout:2166: writing smithi10729407-8 from 2318336 to 2580480 tid 1 2020-06-12T02:37:42.957 INFO:tasks.rados.rados.0.smithi107.stdout:2167: delete oid 11 current snap is 0 2020-06-12T02:37:42.957 INFO:tasks.rados.rados.0.smithi107.stdout:2167: done (1 left) 2020-06-12T02:37:42.958 INFO:tasks.rados.rados.0.smithi107.stdout:2168: delete oid 13 current snap is 0 2020-06-12T02:37:42.958 INFO:tasks.rados.rados.0.smithi107.stdout:2168: done (1 left) 2020-06-12T02:37:42.958 INFO:tasks.rados.rados.0.smithi107.stdout:2169: delete oid 17 current snap is 0 2020-06-12T02:37:42.958 INFO:tasks.rados.rados.0.smithi107.stdout:2166: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:42.958 INFO:tasks.rados.rados.0.smithi107.stdout:2166: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:42.958 INFO:tasks.rados.rados.0.smithi107.stdout:2169: done (1 left) 2020-06-12T02:37:42.959 INFO:tasks.rados.rados.0.smithi107.stdout:2170: delete oid 11 current snap is 0 2020-06-12T02:37:42.959 INFO:tasks.rados.rados.0.smithi107.stdout:2166: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:42.959 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 371 (ObjNum 2165 snap 0 seq_num 2165) dirty exists 2020-06-12T02:37:42.959 INFO:tasks.rados.rados.0.smithi107.stdout:2166: left oid 8 (ObjNum 2165 snap 0 seq_num 2165) 2020-06-12T02:37:42.959 INFO:tasks.rados.rados.0.smithi107.stdout:2166: done (1 left) 2020-06-12T02:37:42.960 INFO:tasks.rados.rados.0.smithi107.stdout:2170: done (0 left) 2020-06-12T02:37:42.960 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:37:42.960 INFO:tasks.rados.rados.0.smithi107.stdout:2171: seq_num 2170 ranges {0=352256} 2020-06-12T02:37:42.960 INFO:tasks.rados.rados.0.smithi107.stdout:2171: writing smithi10729407-9 from 0 to 352256 tid 1 2020-06-12T02:37:42.960 INFO:tasks.rados.rados.0.smithi107.stdout:2172: delete oid 2 current snap is 0 2020-06-12T02:37:42.960 INFO:tasks.rados.rados.0.smithi107.stdout:2172: done (1 left) 2020-06-12T02:37:42.961 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:37:42.961 INFO:tasks.rados.rados.0.smithi107.stdout:2173: seq_num 2172 ranges {2580480=385024} 2020-06-12T02:37:42.961 INFO:tasks.rados.rados.0.smithi107.stdout:2173: writing smithi10729407-8 from 2580480 to 2965504 tid 1 2020-06-12T02:37:42.961 INFO:tasks.rados.rados.0.smithi107.stdout:2174: delete oid 4 current snap is 0 2020-06-12T02:37:42.961 INFO:tasks.rados.rados.0.smithi107.stdout:2171: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:42.961 INFO:tasks.rados.rados.0.smithi107.stdout:2171: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:42.962 INFO:tasks.rados.rados.0.smithi107.stdout:2171: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:42.962 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 373 (ObjNum 2170 snap 0 seq_num 2170) dirty exists 2020-06-12T02:37:42.962 INFO:tasks.rados.rados.0.smithi107.stdout:2171: left oid 9 (ObjNum 2170 snap 0 seq_num 2170) 2020-06-12T02:37:42.962 INFO:tasks.rados.rados.0.smithi107.stdout:2171: done (2 left) 2020-06-12T02:37:42.962 INFO:tasks.rados.rados.0.smithi107.stdout:2174: done (1 left) 2020-06-12T02:37:42.963 INFO:tasks.rados.rados.0.smithi107.stdout:2175: delete oid 11 current snap is 0 2020-06-12T02:37:42.963 INFO:tasks.rados.rados.0.smithi107.stdout:2175: done (1 left) 2020-06-12T02:37:42.963 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:37:42.963 INFO:tasks.rados.rados.0.smithi107.stdout:2176: seq_num 2175 ranges {0=434176} 2020-06-12T02:37:42.963 INFO:tasks.rados.rados.0.smithi107.stdout:2176: writing smithi10729407-17 from 0 to 434176 tid 1 2020-06-12T02:37:42.963 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:37:42.964 INFO:tasks.rados.rados.0.smithi107.stdout:2177: seq_num 2176 ranges {32768=450560} 2020-06-12T02:37:42.964 INFO:tasks.rados.rados.0.smithi107.stdout:2177: writing smithi10729407-12 from 32768 to 483328 tid 1 2020-06-12T02:37:42.964 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:37:42.964 INFO:tasks.rados.rados.0.smithi107.stdout:2178: seq_num 2177 ranges {0=466944} 2020-06-12T02:37:42.964 INFO:tasks.rados.rados.0.smithi107.stdout:2178: writing smithi10729407-19 from 0 to 466944 tid 1 2020-06-12T02:37:42.965 INFO:tasks.rados.rados.0.smithi107.stdout:2179: delete oid 9 current snap is 0 2020-06-12T02:37:42.965 INFO:tasks.rados.rados.0.smithi107.stdout:2173: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:42.965 INFO:tasks.rados.rados.0.smithi107.stdout:2173: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:42.965 INFO:tasks.rados.rados.0.smithi107.stdout:2173: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:42.965 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 373 (ObjNum 2172 snap 0 seq_num 2172) dirty exists 2020-06-12T02:37:42.965 INFO:tasks.rados.rados.0.smithi107.stdout:2173: left oid 8 (ObjNum 2172 snap 0 seq_num 2172) 2020-06-12T02:37:42.966 INFO:tasks.rados.rados.0.smithi107.stdout:2176: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:42.966 INFO:tasks.rados.rados.0.smithi107.stdout:2173: done (4 left) 2020-06-12T02:37:42.966 INFO:tasks.rados.rados.0.smithi107.stdout:2179: done (3 left) 2020-06-12T02:37:42.966 INFO:tasks.rados.rados.0.smithi107.stdout:2180: delete oid 4 current snap is 0 2020-06-12T02:37:42.966 INFO:tasks.rados.rados.0.smithi107.stdout:2177: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:42.967 INFO:tasks.rados.rados.0.smithi107.stdout:2176: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:42.967 INFO:tasks.rados.rados.0.smithi107.stdout:2178: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:42.967 INFO:tasks.rados.rados.0.smithi107.stdout:2180: done (3 left) 2020-06-12T02:37:42.967 INFO:tasks.rados.rados.0.smithi107.stdout:2181: delete oid 9 current snap is 0 2020-06-12T02:37:42.967 INFO:tasks.rados.rados.0.smithi107.stdout:2177: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:42.967 INFO:tasks.rados.rados.0.smithi107.stdout:2176: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:42.968 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 421 (ObjNum 2175 snap 0 seq_num 2175) dirty exists 2020-06-12T02:37:42.968 INFO:tasks.rados.rados.0.smithi107.stdout:2176: left oid 17 (ObjNum 2175 snap 0 seq_num 2175) 2020-06-12T02:37:42.968 INFO:tasks.rados.rados.0.smithi107.stdout:2176: done (3 left) 2020-06-12T02:37:42.968 INFO:tasks.rados.rados.0.smithi107.stdout:2181: done (2 left) 2020-06-12T02:37:42.968 INFO:tasks.rados.rados.0.smithi107.stdout:2182: delete oid 5 current snap is 0 2020-06-12T02:37:42.968 INFO:tasks.rados.rados.0.smithi107.stdout:2177: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:42.969 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 330 (ObjNum 2176 snap 0 seq_num 2176) dirty exists 2020-06-12T02:37:42.969 INFO:tasks.rados.rados.0.smithi107.stdout:2177: left oid 12 (ObjNum 2176 snap 0 seq_num 2176) 2020-06-12T02:37:42.969 INFO:tasks.rados.rados.0.smithi107.stdout:2178: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:42.969 INFO:tasks.rados.rados.0.smithi107.stdout:2177: done (2 left) 2020-06-12T02:37:42.969 INFO:tasks.rados.rados.0.smithi107.stdout:2182: done (1 left) 2020-06-12T02:37:42.970 INFO:tasks.rados.rados.0.smithi107.stdout:2183: delete oid 2 current snap is 0 2020-06-12T02:37:42.970 INFO:tasks.rados.rados.0.smithi107.stdout:2178: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:42.970 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 331 (ObjNum 2177 snap 0 seq_num 2177) dirty exists 2020-06-12T02:37:42.970 INFO:tasks.rados.rados.0.smithi107.stdout:2178: left oid 19 (ObjNum 2177 snap 0 seq_num 2177) 2020-06-12T02:37:42.970 INFO:tasks.rados.rados.0.smithi107.stdout:2178: done (1 left) 2020-06-12T02:37:42.970 INFO:tasks.rados.rados.0.smithi107.stdout:2183: done (0 left) 2020-06-12T02:37:42.971 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:37:42.971 INFO:tasks.rados.rados.0.smithi107.stdout:2184: seq_num 2183 ranges {98304=565248} 2020-06-12T02:37:42.971 INFO:tasks.rados.rados.0.smithi107.stdout:2184: writing smithi10729407-1 from 98304 to 663552 tid 1 2020-06-12T02:37:42.971 INFO:tasks.rados.rados.0.smithi107.stdout:2185: delete oid 12 current snap is 0 2020-06-12T02:37:42.971 INFO:tasks.rados.rados.0.smithi107.stdout:2185: done (1 left) 2020-06-12T02:37:42.972 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:37:42.972 INFO:tasks.rados.rados.0.smithi107.stdout:2186: seq_num 2185 ranges {434176=598016} 2020-06-12T02:37:42.972 INFO:tasks.rados.rados.0.smithi107.stdout:2186: writing smithi10729407-17 from 434176 to 1032192 tid 1 2020-06-12T02:37:42.972 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:37:42.973 INFO:tasks.rados.rados.0.smithi107.stdout:2187: seq_num 2186 ranges {466944=614400} 2020-06-12T02:37:42.973 INFO:tasks.rados.rados.0.smithi107.stdout:2187: writing smithi10729407-19 from 466944 to 1081344 tid 1 2020-06-12T02:37:42.973 INFO:tasks.rados.rados.0.smithi107.stdout:2188: delete oid 12 current snap is 0 2020-06-12T02:37:42.973 INFO:tasks.rados.rados.0.smithi107.stdout:2184: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:42.973 INFO:tasks.rados.rados.0.smithi107.stdout:2184: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:42.974 INFO:tasks.rados.rados.0.smithi107.stdout:2184: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:42.974 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 448 (ObjNum 2183 snap 0 seq_num 2183) dirty exists 2020-06-12T02:37:42.974 INFO:tasks.rados.rados.0.smithi107.stdout:2184: left oid 1 (ObjNum 2183 snap 0 seq_num 2183) 2020-06-12T02:37:42.974 INFO:tasks.rados.rados.0.smithi107.stdout:2186: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:42.974 INFO:tasks.rados.rados.0.smithi107.stdout:2186: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:42.974 INFO:tasks.rados.rados.0.smithi107.stdout:2186: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:42.975 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 424 (ObjNum 2185 snap 0 seq_num 2185) dirty exists 2020-06-12T02:37:42.975 INFO:tasks.rados.rados.0.smithi107.stdout:2186: left oid 17 (ObjNum 2185 snap 0 seq_num 2185) 2020-06-12T02:37:42.975 INFO:tasks.rados.rados.0.smithi107.stdout:2187: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:42.975 INFO:tasks.rados.rados.0.smithi107.stdout:2187: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:42.975 INFO:tasks.rados.rados.0.smithi107.stdout:2184: done (3 left) 2020-06-12T02:37:42.975 INFO:tasks.rados.rados.0.smithi107.stdout:2186: done (2 left) 2020-06-12T02:37:42.976 INFO:tasks.rados.rados.0.smithi107.stdout:2188: done (1 left) 2020-06-12T02:37:42.976 INFO:tasks.rados.rados.0.smithi107.stdout:2189: delete oid 16 current snap is 0 2020-06-12T02:37:42.976 INFO:tasks.rados.rados.0.smithi107.stdout:2187: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:42.976 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 334 (ObjNum 2186 snap 0 seq_num 2186) dirty exists 2020-06-12T02:37:42.976 INFO:tasks.rados.rados.0.smithi107.stdout:2187: left oid 19 (ObjNum 2186 snap 0 seq_num 2186) 2020-06-12T02:37:42.977 INFO:tasks.rados.rados.0.smithi107.stdout:2187: done (1 left) 2020-06-12T02:37:42.977 INFO:tasks.rados.rados.0.smithi107.stdout:2189: done (0 left) 2020-06-12T02:37:42.977 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:37:42.977 INFO:tasks.rados.rados.0.smithi107.stdout:2190: seq_num 2189 ranges {1081344=671744} 2020-06-12T02:37:42.977 INFO:tasks.rados.rados.0.smithi107.stdout:2190: writing smithi10729407-19 from 1081344 to 1753088 tid 1 2020-06-12T02:37:42.977 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:37:42.978 INFO:tasks.rados.rados.0.smithi107.stdout:2191: seq_num 2190 ranges {0=688128} 2020-06-12T02:37:42.978 INFO:tasks.rados.rados.0.smithi107.stdout:2191: writing smithi10729407-5 from 0 to 688128 tid 1 2020-06-12T02:37:42.978 INFO:tasks.rados.rados.0.smithi107.stdout:2192: delete oid 4 current snap is 0 2020-06-12T02:37:42.978 INFO:tasks.rados.rados.0.smithi107.stdout:2190: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:42.978 INFO:tasks.rados.rados.0.smithi107.stdout:2190: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:42.979 INFO:tasks.rados.rados.0.smithi107.stdout:2192: done (2 left) 2020-06-12T02:37:42.979 INFO:tasks.rados.rados.0.smithi107.stdout:2193: delete oid 3 current snap is 0 2020-06-12T02:37:42.979 INFO:tasks.rados.rados.0.smithi107.stdout:2190: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:42.979 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 337 (ObjNum 2189 snap 0 seq_num 2189) dirty exists 2020-06-12T02:37:42.979 INFO:tasks.rados.rados.0.smithi107.stdout:2190: left oid 19 (ObjNum 2189 snap 0 seq_num 2189) 2020-06-12T02:37:42.979 INFO:tasks.rados.rados.0.smithi107.stdout:2191: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:42.980 INFO:tasks.rados.rados.0.smithi107.stdout:2191: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:42.980 INFO:tasks.rados.rados.0.smithi107.stdout:2190: done (2 left) 2020-06-12T02:37:42.980 INFO:tasks.rados.rados.0.smithi107.stdout:2193: done (1 left) 2020-06-12T02:37:42.980 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:37:42.980 INFO:tasks.rados.rados.0.smithi107.stdout:2194: seq_num 2193 ranges {0=737280} 2020-06-12T02:37:42.981 INFO:tasks.rados.rados.0.smithi107.stdout:2194: writing smithi10729407-12 from 0 to 737280 tid 1 2020-06-12T02:37:42.981 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:37:42.981 INFO:tasks.rados.rados.0.smithi107.stdout:2191: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:42.981 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 179 (ObjNum 2190 snap 0 seq_num 2190) dirty exists 2020-06-12T02:37:42.981 INFO:tasks.rados.rados.0.smithi107.stdout:2191: left oid 5 (ObjNum 2190 snap 0 seq_num 2190) 2020-06-12T02:37:42.981 INFO:tasks.rados.rados.0.smithi107.stdout:2195: seq_num 2194 ranges {0=753664} 2020-06-12T02:37:42.982 INFO:tasks.rados.rados.0.smithi107.stdout:2195: writing smithi10729407-9 from 0 to 753664 tid 1 2020-06-12T02:37:42.982 INFO:tasks.rados.rados.0.smithi107.stdout:2191: done (2 left) 2020-06-12T02:37:42.982 INFO:tasks.rados.rados.0.smithi107.stdout:2196: delete oid 5 current snap is 0 2020-06-12T02:37:42.982 INFO:tasks.rados.rados.0.smithi107.stdout:2196: done (2 left) 2020-06-12T02:37:42.982 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:37:42.983 INFO:tasks.rados.rados.0.smithi107.stdout:2197: seq_num 2196 ranges {0=786432} 2020-06-12T02:37:42.983 INFO:tasks.rados.rados.0.smithi107.stdout:2197: writing smithi10729407-13 from 0 to 786432 tid 1 2020-06-12T02:37:42.983 INFO:tasks.rados.rados.0.smithi107.stdout:2198: delete oid 17 current snap is 0 2020-06-12T02:37:42.983 INFO:tasks.rados.rados.0.smithi107.stdout:2194: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:42.983 INFO:tasks.rados.rados.0.smithi107.stdout:2194: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:42.983 INFO:tasks.rados.rados.0.smithi107.stdout:2194: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:42.984 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 339 (ObjNum 2193 snap 0 seq_num 2193) dirty exists 2020-06-12T02:37:42.984 INFO:tasks.rados.rados.0.smithi107.stdout:2194: left oid 12 (ObjNum 2193 snap 0 seq_num 2193) 2020-06-12T02:37:42.984 INFO:tasks.rados.rados.0.smithi107.stdout:2195: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:42.984 INFO:tasks.rados.rados.0.smithi107.stdout:2195: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:42.984 INFO:tasks.rados.rados.0.smithi107.stdout:2195: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:42.985 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 377 (ObjNum 2194 snap 0 seq_num 2194) dirty exists 2020-06-12T02:37:42.985 INFO:tasks.rados.rados.0.smithi107.stdout:2195: left oid 9 (ObjNum 2194 snap 0 seq_num 2194) 2020-06-12T02:37:42.985 INFO:tasks.rados.rados.0.smithi107.stdout:2194: done (3 left) 2020-06-12T02:37:42.985 INFO:tasks.rados.rados.0.smithi107.stdout:2195: done (2 left) 2020-06-12T02:37:42.986 INFO:tasks.rados.rados.0.smithi107.stdout:2198: done (1 left) 2020-06-12T02:37:42.986 INFO:tasks.rados.rados.0.smithi107.stdout:2199: delete oid 6 current snap is 0 2020-06-12T02:37:42.986 INFO:tasks.rados.rados.0.smithi107.stdout:2199: done (1 left) 2020-06-12T02:37:42.986 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:37:42.986 INFO:tasks.rados.rados.0.smithi107.stdout:2200: seq_num 2199 ranges {0=434176,434176=401408} 2020-06-12T02:37:42.986 INFO:tasks.rados.rados.0.smithi107.stdout:2200: writing smithi10729407-2 from 0 to 434176 tid 1 2020-06-12T02:37:42.987 INFO:tasks.rados.rados.0.smithi107.stdout:2200: writing smithi10729407-2 from 434176 to 835584 tid 2 2020-06-12T02:37:42.987 INFO:tasks.rados.rados.0.smithi107.stdout:2201: delete oid 19 current snap is 0 2020-06-12T02:37:42.987 INFO:tasks.rados.rados.0.smithi107.stdout:2197: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:42.987 INFO:tasks.rados.rados.0.smithi107.stdout:2197: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:42.987 INFO:tasks.rados.rados.0.smithi107.stdout:2197: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:42.988 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 415 (ObjNum 2196 snap 0 seq_num 2196) dirty exists 2020-06-12T02:37:42.988 INFO:tasks.rados.rados.0.smithi107.stdout:2197: left oid 13 (ObjNum 2196 snap 0 seq_num 2196) 2020-06-12T02:37:42.988 INFO:tasks.rados.rados.0.smithi107.stdout:2197: done (2 left) 2020-06-12T02:37:42.988 INFO:tasks.rados.rados.0.smithi107.stdout:2201: done (1 left) 2020-06-12T02:37:42.988 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:37:42.988 INFO:tasks.rados.rados.0.smithi107.stdout:2202: seq_num 2201 ranges {0=466944,466944=401408} 2020-06-12T02:37:42.989 INFO:tasks.rados.rados.0.smithi107.stdout:2202: writing smithi10729407-11 from 0 to 466944 tid 1 2020-06-12T02:37:42.989 INFO:tasks.rados.rados.0.smithi107.stdout:2202: writing smithi10729407-11 from 466944 to 868352 tid 2 2020-06-12T02:37:42.989 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:37:42.989 INFO:tasks.rados.rados.0.smithi107.stdout:2203: seq_num 2202 ranges {0=483328,483328=401408} 2020-06-12T02:37:42.990 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-06-12T02:37:42.990 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:42.899+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:42.995 INFO:tasks.rados.rados.0.smithi107.stdout:2203: writing smithi10729407-6 from 0 to 483328 tid 1 2020-06-12T02:37:42.995 INFO:tasks.rados.rados.0.smithi107.stdout:2203: writing smithi10729407-6 from 483328 to 884736 tid 2 2020-06-12T02:37:42.995 INFO:tasks.rados.rados.0.smithi107.stdout:2204: delete oid 5 current snap is 0 2020-06-12T02:37:42.996 INFO:tasks.rados.rados.0.smithi107.stdout:2200: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:42.996 INFO:tasks.rados.rados.0.smithi107.stdout:2200: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:42.996 INFO:tasks.rados.rados.0.smithi107.stdout:2200: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:42.996 INFO:tasks.rados.rados.0.smithi107.stdout:2200: finishing write tid 5 to smithi10729407-2 2020-06-12T02:37:42.996 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 232 (ObjNum 2199 snap 0 seq_num 2199) dirty exists 2020-06-12T02:37:42.996 INFO:tasks.rados.rados.0.smithi107.stdout:2200: left oid 2 (ObjNum 2199 snap 0 seq_num 2199) 2020-06-12T02:37:42.997 INFO:tasks.rados.rados.0.smithi107.stdout:2202: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:42.997 INFO:tasks.rados.rados.0.smithi107.stdout:2200: done (3 left) 2020-06-12T02:37:42.997 INFO:tasks.rados.rados.0.smithi107.stdout:2204: done (2 left) 2020-06-12T02:37:42.997 INFO:tasks.rados.rados.0.smithi107.stdout:2205: delete oid 18 current snap is 0 2020-06-12T02:37:42.997 INFO:tasks.rados.rados.0.smithi107.stdout:2205: done (2 left) 2020-06-12T02:37:42.998 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:37:42.998 INFO:tasks.rados.rados.0.smithi107.stdout:2206: seq_num 2205 ranges {0=532480,532480=401408} 2020-06-12T02:37:42.998 INFO:tasks.rados.rados.0.smithi107.stdout:2206: writing smithi10729407-15 from 0 to 532480 tid 1 2020-06-12T02:37:42.998 INFO:tasks.rados.rados.0.smithi107.stdout:2206: writing smithi10729407-15 from 532480 to 933888 tid 2 2020-06-12T02:37:42.998 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:37:42.999 INFO:tasks.rados.rados.0.smithi107.stdout:2207: seq_num 2206 ranges {663552=548864,1212416=401408} 2020-06-12T02:37:42.999 INFO:tasks.rados.rados.0.smithi107.stdout:2207: writing smithi10729407-1 from 663552 to 1212416 tid 1 2020-06-12T02:37:42.999 INFO:tasks.rados.rados.0.smithi107.stdout:2207: writing smithi10729407-1 from 1212416 to 1613824 tid 2 2020-06-12T02:37:42.999 INFO:tasks.rados.rados.0.smithi107.stdout:2208: delete oid 9 current snap is 0 2020-06-12T02:37:42.999 INFO:tasks.rados.rados.0.smithi107.stdout:2202: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:42.999 INFO:tasks.rados.rados.0.smithi107.stdout:2202: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:43.000 INFO:tasks.rados.rados.0.smithi107.stdout:2203: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:43.000 INFO:tasks.rados.rados.0.smithi107.stdout:2203: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:43.000 INFO:tasks.rados.rados.0.smithi107.stdout:2203: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:43.000 INFO:tasks.rados.rados.0.smithi107.stdout:2202: finishing write tid 5 to smithi10729407-11 2020-06-12T02:37:43.000 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 404 (ObjNum 2201 snap 0 seq_num 2201) dirty exists 2020-06-12T02:37:43.001 INFO:tasks.rados.rados.0.smithi107.stdout:2202: left oid 11 (ObjNum 2201 snap 0 seq_num 2201) 2020-06-12T02:37:43.001 INFO:tasks.rados.rados.0.smithi107.stdout:2203: finishing write tid 5 to smithi10729407-6 2020-06-12T02:37:43.001 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 452 (ObjNum 2202 snap 0 seq_num 2202) dirty exists 2020-06-12T02:37:43.001 INFO:tasks.rados.rados.0.smithi107.stdout:2203: left oid 6 (ObjNum 2202 snap 0 seq_num 2202) 2020-06-12T02:37:43.001 INFO:tasks.rados.rados.0.smithi107.stdout:2206: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:43.001 INFO:tasks.rados.rados.0.smithi107.stdout:2206: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:43.002 INFO:tasks.rados.rados.0.smithi107.stdout:2206: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:43.002 INFO:tasks.rados.rados.0.smithi107.stdout:2206: finishing write tid 5 to smithi10729407-15 2020-06-12T02:37:43.002 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 376 (ObjNum 2205 snap 0 seq_num 2205) dirty exists 2020-06-12T02:37:43.002 INFO:tasks.rados.rados.0.smithi107.stdout:2206: left oid 15 (ObjNum 2205 snap 0 seq_num 2205) 2020-06-12T02:37:43.002 INFO:tasks.rados.rados.0.smithi107.stdout:2202: done (4 left) 2020-06-12T02:37:43.003 INFO:tasks.rados.rados.0.smithi107.stdout:2203: done (3 left) 2020-06-12T02:37:43.003 INFO:tasks.rados.rados.0.smithi107.stdout:2206: done (2 left) 2020-06-12T02:37:43.003 INFO:tasks.rados.rados.0.smithi107.stdout:2208: done (1 left) 2020-06-12T02:37:43.003 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:37:43.003 INFO:tasks.rados.rados.0.smithi107.stdout:2209: seq_num 2208 ranges {868352=589824,1458176=401408} 2020-06-12T02:37:43.003 INFO:tasks.rados.rados.0.smithi107.stdout:2209: writing smithi10729407-11 from 868352 to 1458176 tid 1 2020-06-12T02:37:43.004 INFO:tasks.rados.rados.0.smithi107.stdout:2209: writing smithi10729407-11 from 1458176 to 1859584 tid 2 2020-06-12T02:37:43.004 INFO:tasks.rados.rados.0.smithi107.stdout:2210: delete oid 12 current snap is 0 2020-06-12T02:37:43.004 INFO:tasks.rados.rados.0.smithi107.stdout:2207: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:43.004 INFO:tasks.rados.rados.0.smithi107.stdout:2207: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:43.004 INFO:tasks.rados.rados.0.smithi107.stdout:2207: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:43.005 INFO:tasks.rados.rados.0.smithi107.stdout:2207: finishing write tid 5 to smithi10729407-1 2020-06-12T02:37:43.005 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 455 (ObjNum 2206 snap 0 seq_num 2206) dirty exists 2020-06-12T02:37:43.005 INFO:tasks.rados.rados.0.smithi107.stdout:2207: left oid 1 (ObjNum 2206 snap 0 seq_num 2206) 2020-06-12T02:37:43.005 INFO:tasks.rados.rados.0.smithi107.stdout:2207: done (2 left) 2020-06-12T02:37:43.005 INFO:tasks.rados.rados.0.smithi107.stdout:2210: done (1 left) 2020-06-12T02:37:43.005 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:37:43.006 INFO:tasks.rados.rados.0.smithi107.stdout:2211: seq_num 2210 ranges {0=622592,622592=401408} 2020-06-12T02:37:43.006 INFO:tasks.rados.rados.0.smithi107.stdout:2211: writing smithi10729407-9 from 0 to 622592 tid 1 2020-06-12T02:37:43.006 INFO:tasks.rados.rados.0.smithi107.stdout:2211: writing smithi10729407-9 from 622592 to 1024000 tid 2 2020-06-12T02:37:43.006 INFO:tasks.rados.rados.0.smithi107.stdout:2212: delete oid 4 current snap is 0 2020-06-12T02:37:43.006 INFO:tasks.rados.rados.0.smithi107.stdout:2209: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:43.007 INFO:tasks.rados.rados.0.smithi107.stdout:2209: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:43.007 INFO:tasks.rados.rados.0.smithi107.stdout:2211: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:43.007 INFO:tasks.rados.rados.0.smithi107.stdout:2209: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:43.007 INFO:tasks.rados.rados.0.smithi107.stdout:2212: done (2 left) 2020-06-12T02:37:43.007 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:37:43.007 INFO:tasks.rados.rados.0.smithi107.stdout:2213: seq_num 2212 ranges {0=655360,655360=401408} 2020-06-12T02:37:43.008 INFO:tasks.rados.rados.0.smithi107.stdout:2213: writing smithi10729407-10 from 0 to 655360 tid 1 2020-06-12T02:37:43.008 INFO:tasks.rados.rados.0.smithi107.stdout:2213: writing smithi10729407-10 from 655360 to 1056768 tid 2 2020-06-12T02:37:43.008 INFO:tasks.rados.rados.0.smithi107.stdout:2214: delete oid 13 current snap is 0 2020-06-12T02:37:43.008 INFO:tasks.rados.rados.0.smithi107.stdout:2211: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:43.008 INFO:tasks.rados.rados.0.smithi107.stdout:2211: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:43.008 INFO:tasks.rados.rados.0.smithi107.stdout:2209: finishing write tid 5 to smithi10729407-11 2020-06-12T02:37:43.009 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 407 (ObjNum 2208 snap 0 seq_num 2208) dirty exists 2020-06-12T02:37:43.009 INFO:tasks.rados.rados.0.smithi107.stdout:2209: left oid 11 (ObjNum 2208 snap 0 seq_num 2208) 2020-06-12T02:37:43.009 INFO:tasks.rados.rados.0.smithi107.stdout:2211: finishing write tid 5 to smithi10729407-9 2020-06-12T02:37:43.009 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 382 (ObjNum 2210 snap 0 seq_num 2210) dirty exists 2020-06-12T02:37:43.009 INFO:tasks.rados.rados.0.smithi107.stdout:2211: left oid 9 (ObjNum 2210 snap 0 seq_num 2210) 2020-06-12T02:37:43.010 INFO:tasks.rados.rados.0.smithi107.stdout:2209: done (3 left) 2020-06-12T02:37:43.010 INFO:tasks.rados.rados.0.smithi107.stdout:2211: done (2 left) 2020-06-12T02:37:43.010 INFO:tasks.rados.rados.0.smithi107.stdout:2214: done (1 left) 2020-06-12T02:37:43.010 INFO:tasks.rados.rados.0.smithi107.stdout:2215: delete oid 16 current snap is 0 2020-06-12T02:37:43.010 INFO:tasks.rados.rados.0.smithi107.stdout:2213: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:43.010 INFO:tasks.rados.rados.0.smithi107.stdout:2215: done (1 left) 2020-06-12T02:37:43.011 INFO:tasks.rados.rados.0.smithi107.stdout:2216: delete oid 17 current snap is 0 2020-06-12T02:37:43.011 INFO:tasks.rados.rados.0.smithi107.stdout:2216: done (1 left) 2020-06-12T02:37:43.011 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:43.011 INFO:tasks.rados.rados.0.smithi107.stdout:2217: seq_num 2216 ranges {835584=720896,1556480=401408} 2020-06-12T02:37:43.011 INFO:tasks.rados.rados.0.smithi107.stdout:2217: writing smithi10729407-2 from 835584 to 1556480 tid 1 2020-06-12T02:37:43.014 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:43.009+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:37:43.014 INFO:tasks.rados.rados.0.smithi107.stdout:2217: writing smithi10729407-2 from 1556480 to 1957888 tid 2 2020-06-12T02:37:43.014 INFO:tasks.rados.rados.0.smithi107.stdout:2218: delete oid 12 current snap is 0 2020-06-12T02:37:43.015 INFO:tasks.rados.rados.0.smithi107.stdout:2213: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:43.015 INFO:tasks.rados.rados.0.smithi107.stdout:2213: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:43.015 INFO:tasks.rados.rados.0.smithi107.stdout:2213: finishing write tid 5 to smithi10729407-10 2020-06-12T02:37:43.015 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 199 (ObjNum 2212 snap 0 seq_num 2212) dirty exists 2020-06-12T02:37:43.015 INFO:tasks.rados.rados.0.smithi107.stdout:2213: left oid 10 (ObjNum 2212 snap 0 seq_num 2212) 2020-06-12T02:37:43.016 INFO:tasks.rados.rados.0.smithi107.stdout:2213: done (2 left) 2020-06-12T02:37:43.016 INFO:tasks.rados.rados.0.smithi107.stdout:2218: done (1 left) 2020-06-12T02:37:43.016 INFO:tasks.rados.rados.0.smithi107.stdout:2219: delete oid 13 current snap is 0 2020-06-12T02:37:43.016 INFO:tasks.rados.rados.0.smithi107.stdout:2217: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:43.016 INFO:tasks.rados.rados.0.smithi107.stdout:2219: done (1 left) 2020-06-12T02:37:43.016 INFO:tasks.rados.rados.0.smithi107.stdout:2220: delete oid 9 current snap is 0 2020-06-12T02:37:43.017 INFO:tasks.rados.rados.0.smithi107.stdout:2220: done (1 left) 2020-06-12T02:37:43.017 INFO:tasks.rados.rados.0.smithi107.stdout:2221: delete oid 3 current snap is 0 2020-06-12T02:37:43.017 INFO:tasks.rados.rados.0.smithi107.stdout:2217: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:43.017 INFO:tasks.rados.rados.0.smithi107.stdout:2217: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:43.017 INFO:tasks.rados.rados.0.smithi107.stdout:2217: finishing write tid 5 to smithi10729407-2 2020-06-12T02:37:43.018 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 235 (ObjNum 2216 snap 0 seq_num 2216) dirty exists 2020-06-12T02:37:43.018 INFO:tasks.rados.rados.0.smithi107.stdout:2217: left oid 2 (ObjNum 2216 snap 0 seq_num 2216) 2020-06-12T02:37:43.018 INFO:tasks.rados.rados.0.smithi107.stdout:2217: done (1 left) 2020-06-12T02:37:43.018 INFO:tasks.rados.rados.0.smithi107.stdout:2221: done (0 left) 2020-06-12T02:37:43.018 INFO:tasks.rados.rados.0.smithi107.stdout:2222: delete oid 16 current snap is 0 2020-06-12T02:37:43.018 INFO:tasks.rados.rados.0.smithi107.stdout:2222: done (0 left) 2020-06-12T02:37:43.019 INFO:tasks.rados.rados.0.smithi107.stdout:2223: delete oid 9 current snap is 0 2020-06-12T02:37:43.019 INFO:tasks.rados.rados.0.smithi107.stdout:2223: done (0 left) 2020-06-12T02:37:43.023 INFO:tasks.rados.rados.0.smithi107.stdout:2224: delete oid 11 current snap is 0 2020-06-12T02:37:43.023 INFO:tasks.rados.rados.0.smithi107.stdout:2224: done (0 left) 2020-06-12T02:37:43.024 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:37:43.024 INFO:tasks.rados.rados.0.smithi107.stdout:2225: seq_num 2224 ranges {0=450560,450560=466944,917504=335872} 2020-06-12T02:37:43.025 INFO:tasks.rados.rados.0.smithi107.stdout:2225: writing smithi10729407-17 from 0 to 450560 tid 1 2020-06-12T02:37:43.025 INFO:tasks.rados.rados.0.smithi107.stdout:2225: writing smithi10729407-17 from 450560 to 917504 tid 2 2020-06-12T02:37:43.026 INFO:tasks.rados.rados.0.smithi107.stdout:2225: writing smithi10729407-17 from 917504 to 1253376 tid 3 2020-06-12T02:37:43.026 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:37:43.026 INFO:tasks.rados.rados.0.smithi107.stdout:2226: seq_num 2225 ranges {0=466944,466944=753664,1220608=49152} 2020-06-12T02:37:43.026 INFO:tasks.rados.rados.0.smithi107.stdout:2226: writing smithi10729407-11 from 0 to 466944 tid 1 2020-06-12T02:37:43.027 INFO:tasks.rados.rados.0.smithi107.stdout:2226: writing smithi10729407-11 from 466944 to 1220608 tid 2 2020-06-12T02:37:43.028 INFO:tasks.rados.rados.0.smithi107.stdout:2226: writing smithi10729407-11 from 1220608 to 1269760 tid 3 2020-06-12T02:37:43.028 INFO:tasks.rados.rados.0.smithi107.stdout:2227: delete oid 7 current snap is 0 2020-06-12T02:37:43.029 INFO:tasks.rados.rados.0.smithi107.stdout:2225: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:43.029 INFO:tasks.rados.rados.0.smithi107.stdout:2225: finishing write tid 2 to smithi10729407-17 2020-06-12T02:37:43.029 INFO:tasks.rados.rados.0.smithi107.stdout:2225: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:43.029 INFO:tasks.rados.rados.0.smithi107.stdout:2225: finishing write tid 5 to smithi10729407-17 2020-06-12T02:37:43.029 INFO:tasks.rados.rados.0.smithi107.stdout:2225: finishing write tid 6 to smithi10729407-17 2020-06-12T02:37:43.030 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 432 (ObjNum 2224 snap 0 seq_num 2224) dirty exists 2020-06-12T02:37:43.030 INFO:tasks.rados.rados.0.smithi107.stdout:2225: left oid 17 (ObjNum 2224 snap 0 seq_num 2224) 2020-06-12T02:37:43.030 INFO:tasks.rados.rados.0.smithi107.stdout:2225: done (2 left) 2020-06-12T02:37:43.031 INFO:tasks.rados.rados.0.smithi107.stdout:2227: done (1 left) 2020-06-12T02:37:43.031 INFO:tasks.rados.rados.0.smithi107.stdout:2228: delete oid 10 current snap is 0 2020-06-12T02:37:43.033 INFO:tasks.rados.rados.0.smithi107.stdout:2226: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:43.035 INFO:tasks.rados.rados.0.smithi107.stdout:2226: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:43.035 INFO:tasks.rados.rados.0.smithi107.stdout:2228: done (1 left) 2020-06-12T02:37:43.036 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:37:43.036 INFO:tasks.rados.rados.0.smithi107.stdout:2229: seq_num 2228 ranges {0=524288,524288=450560,974848=352256} 2020-06-12T02:37:43.039 INFO:tasks.rados.rados.0.smithi107.stdout:2229: writing smithi10729407-5 from 0 to 524288 tid 1 2020-06-12T02:37:43.042 INFO:tasks.rados.rados.0.smithi107.stdout:2229: writing smithi10729407-5 from 524288 to 974848 tid 2 2020-06-12T02:37:43.047 INFO:tasks.rados.rados.0.smithi107.stdout:2229: writing smithi10729407-5 from 974848 to 1327104 tid 3 2020-06-12T02:37:43.047 INFO:tasks.rados.rados.0.smithi107.stdout:2230: delete oid 19 current snap is 0 2020-06-12T02:37:43.047 INFO:tasks.rados.rados.0.smithi107.stdout:2226: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:43.048 INFO:tasks.rados.rados.0.smithi107.stdout:2226: finishing write tid 5 to smithi10729407-11 2020-06-12T02:37:43.048 INFO:tasks.rados.rados.0.smithi107.stdout:2226: finishing write tid 6 to smithi10729407-11 2020-06-12T02:37:43.048 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 412 (ObjNum 2225 snap 0 seq_num 2225) dirty exists 2020-06-12T02:37:43.048 INFO:tasks.rados.rados.0.smithi107.stdout:2226: left oid 11 (ObjNum 2225 snap 0 seq_num 2225) 2020-06-12T02:37:43.048 INFO:tasks.rados.rados.0.smithi107.stdout:2229: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:43.049 INFO:tasks.rados.rados.0.smithi107.stdout:2226: done (2 left) 2020-06-12T02:37:43.049 INFO:tasks.rados.rados.0.smithi107.stdout:2230: done (1 left) 2020-06-12T02:37:43.049 INFO:tasks.rados.rados.0.smithi107.stdout:2231: delete oid 12 current snap is 0 2020-06-12T02:37:43.050 INFO:tasks.rados.rados.0.smithi107.stdout:2229: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:43.053 INFO:tasks.rados.rados.0.smithi107.stdout:2231: done (1 left) 2020-06-12T02:37:43.053 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:37:43.053 INFO:tasks.rados.rados.0.smithi107.stdout:2232: seq_num 2231 ranges {1253376=573440,1826816=507904,2334720=294912} 2020-06-12T02:37:43.063 INFO:tasks.rados.rados.0.smithi107.stdout:2232: writing smithi10729407-17 from 1253376 to 1826816 tid 1 2020-06-12T02:37:43.065 INFO:tasks.rados.rados.0.smithi107.stdout:2232: writing smithi10729407-17 from 1826816 to 2334720 tid 2 2020-06-12T02:37:43.067 INFO:tasks.rados.rados.0.smithi107.stdout:2232: writing smithi10729407-17 from 2334720 to 2629632 tid 3 2020-06-12T02:37:43.068 INFO:tasks.rados.rados.0.smithi107.stdout:2233: delete oid 6 current snap is 0 2020-06-12T02:37:43.068 INFO:tasks.rados.rados.0.smithi107.stdout:2229: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:43.068 INFO:tasks.rados.rados.0.smithi107.stdout:2229: finishing write tid 5 to smithi10729407-5 2020-06-12T02:37:43.068 INFO:tasks.rados.rados.0.smithi107.stdout:2229: finishing write tid 6 to smithi10729407-5 2020-06-12T02:37:43.068 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 185 (ObjNum 2228 snap 0 seq_num 2228) dirty exists 2020-06-12T02:37:43.069 INFO:tasks.rados.rados.0.smithi107.stdout:2229: left oid 5 (ObjNum 2228 snap 0 seq_num 2228) 2020-06-12T02:37:43.069 INFO:tasks.rados.rados.0.smithi107.stdout:2232: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:43.069 INFO:tasks.rados.rados.0.smithi107.stdout:2229: done (2 left) 2020-06-12T02:37:43.070 INFO:tasks.rados.rados.0.smithi107.stdout:2233: done (1 left) 2020-06-12T02:37:43.070 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:37:43.070 INFO:tasks.rados.rados.0.smithi107.stdout:2234: seq_num 2233 ranges {0=606208,606208=679936,1286144=122880} 2020-06-12T02:37:43.073 INFO:tasks.rados.rados.0.smithi107.stdout:2234: writing smithi10729407-13 from 0 to 606208 tid 1 2020-06-12T02:37:43.077 INFO:tasks.rados.rados.0.smithi107.stdout:2234: writing smithi10729407-13 from 606208 to 1286144 tid 2 2020-06-12T02:37:43.080 INFO:tasks.rados.rados.0.smithi107.stdout:2234: writing smithi10729407-13 from 1286144 to 1409024 tid 3 2020-06-12T02:37:43.080 INFO:tasks.rados.rados.0.smithi107.stdout:2235: delete oid 19 current snap is 0 2020-06-12T02:37:43.080 INFO:tasks.rados.rados.0.smithi107.stdout:2232: finishing write tid 2 to smithi10729407-17 2020-06-12T02:37:43.080 INFO:tasks.rados.rados.0.smithi107.stdout:2232: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:43.080 INFO:tasks.rados.rados.0.smithi107.stdout:2232: finishing write tid 5 to smithi10729407-17 2020-06-12T02:37:43.081 INFO:tasks.rados.rados.0.smithi107.stdout:2232: finishing write tid 6 to smithi10729407-17 2020-06-12T02:37:43.081 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 436 (ObjNum 2231 snap 0 seq_num 2231) dirty exists 2020-06-12T02:37:43.081 INFO:tasks.rados.rados.0.smithi107.stdout:2232: left oid 17 (ObjNum 2231 snap 0 seq_num 2231) 2020-06-12T02:37:43.082 INFO:tasks.rados.rados.0.smithi107.stdout:2232: done (2 left) 2020-06-12T02:37:43.082 INFO:tasks.rados.rados.0.smithi107.stdout:2235: done (1 left) 2020-06-12T02:37:43.082 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:43.082 INFO:tasks.rados.rados.0.smithi107.stdout:2236: seq_num 2235 ranges {1957888=638976,2596864=442368,3039232=360448} 2020-06-12T02:37:43.095 INFO:tasks.rados.rados.0.smithi107.stdout:2236: writing smithi10729407-2 from 1957888 to 2596864 tid 1 2020-06-12T02:37:43.098 INFO:tasks.rados.rados.0.smithi107.stdout:2236: writing smithi10729407-2 from 2596864 to 3039232 tid 2 2020-06-12T02:37:43.100 INFO:tasks.rados.rados.0.smithi107.stdout:2236: writing smithi10729407-2 from 3039232 to 3399680 tid 3 2020-06-12T02:37:43.101 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:37:43.102 INFO:tasks.rados.rados.0.smithi107.stdout:2237: seq_num 2236 ranges {0=655360,655360=778240,1433600=24576} 2020-06-12T02:37:43.104 INFO:tasks.rados.rados.0.smithi107.stdout:2237: writing smithi10729407-6 from 0 to 655360 tid 1 2020-06-12T02:37:43.110 INFO:tasks.rados.rados.0.smithi107.stdout:2237: writing smithi10729407-6 from 655360 to 1433600 tid 2 2020-06-12T02:37:43.110 INFO:tasks.rados.rados.0.smithi107.stdout:2237: writing smithi10729407-6 from 1433600 to 1458176 tid 3 2020-06-12T02:37:43.110 INFO:tasks.rados.rados.0.smithi107.stdout:2238: delete oid 16 current snap is 0 2020-06-12T02:37:43.110 INFO:tasks.rados.rados.0.smithi107.stdout:2234: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:43.111 INFO:tasks.rados.rados.0.smithi107.stdout:2234: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:43.111 INFO:tasks.rados.rados.0.smithi107.stdout:2234: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:43.111 INFO:tasks.rados.rados.0.smithi107.stdout:2234: finishing write tid 5 to smithi10729407-13 2020-06-12T02:37:43.111 INFO:tasks.rados.rados.0.smithi107.stdout:2234: finishing write tid 6 to smithi10729407-13 2020-06-12T02:37:43.111 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 422 (ObjNum 2233 snap 0 seq_num 2233) dirty exists 2020-06-12T02:37:43.112 INFO:tasks.rados.rados.0.smithi107.stdout:2234: left oid 13 (ObjNum 2233 snap 0 seq_num 2233) 2020-06-12T02:37:43.112 INFO:tasks.rados.rados.0.smithi107.stdout:2236: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:43.112 INFO:tasks.rados.rados.0.smithi107.stdout:2236: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:43.112 INFO:tasks.rados.rados.0.smithi107.stdout:2236: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:43.113 INFO:tasks.rados.rados.0.smithi107.stdout:2236: finishing write tid 5 to smithi10729407-2 2020-06-12T02:37:43.113 INFO:tasks.rados.rados.0.smithi107.stdout:2237: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:43.114 INFO:tasks.rados.rados.0.smithi107.stdout:2236: finishing write tid 6 to smithi10729407-2 2020-06-12T02:37:43.114 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 239 (ObjNum 2235 snap 0 seq_num 2235) dirty exists 2020-06-12T02:37:43.114 INFO:tasks.rados.rados.0.smithi107.stdout:2236: left oid 2 (ObjNum 2235 snap 0 seq_num 2235) 2020-06-12T02:37:43.114 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:37:43.110+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:37:43.115 INFO:tasks.rados.rados.0.smithi107.stdout:2234: done (3 left) 2020-06-12T02:37:43.115 INFO:tasks.rados.rados.0.smithi107.stdout:2236: done (2 left) 2020-06-12T02:37:43.116 INFO:tasks.rados.rados.0.smithi107.stdout:2238: done (1 left) 2020-06-12T02:37:43.116 INFO:tasks.rados.rados.0.smithi107.stdout:2239: delete oid 19 current snap is 0 2020-06-12T02:37:43.116 INFO:tasks.rados.rados.0.smithi107.stdout:2237: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:43.117 INFO:tasks.rados.rados.0.smithi107.stdout:2237: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:43.117 INFO:tasks.rados.rados.0.smithi107.stdout:2237: finishing write tid 5 to smithi10729407-6 2020-06-12T02:37:43.117 INFO:tasks.rados.rados.0.smithi107.stdout:2239: done (1 left) 2020-06-12T02:37:43.118 INFO:tasks.rados.rados.0.smithi107.stdout:2240: delete oid 9 current snap is 0 2020-06-12T02:37:43.118 INFO:tasks.rados.rados.0.smithi107.stdout:2237: finishing write tid 6 to smithi10729407-6 2020-06-12T02:37:43.118 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 460 (ObjNum 2236 snap 0 seq_num 2236) dirty exists 2020-06-12T02:37:43.118 INFO:tasks.rados.rados.0.smithi107.stdout:2237: left oid 6 (ObjNum 2236 snap 0 seq_num 2236) 2020-06-12T02:37:43.121 INFO:tasks.rados.rados.0.smithi107.stdout:2237: done (1 left) 2020-06-12T02:37:43.122 INFO:tasks.rados.rados.0.smithi107.stdout:2240: done (0 left) 2020-06-12T02:37:43.122 INFO:tasks.rados.rados.0.smithi107.stdout:2241: delete oid 1 current snap is 0 2020-06-12T02:37:43.124 INFO:tasks.rados.rados.0.smithi107.stdout:2241: done (0 left) 2020-06-12T02:37:43.124 INFO:tasks.rados.rados.0.smithi107.stdout:2242: delete oid 17 current snap is 0 2020-06-12T02:37:43.127 INFO:tasks.rados.rados.0.smithi107.stdout:2242: done (0 left) 2020-06-12T02:37:43.128 INFO:tasks.rados.rados.0.smithi107.stdout:2243: delete oid 13 current snap is 0 2020-06-12T02:37:43.130 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:37:43.130 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:37:43.130 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.130 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:4733 3.3s0 3:c2ca77da:::smithi10729407-13:head [stat out=16b,delete] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-06-12T02:37:43.131 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:43.126469+0000", 2020-06-12T02:37:43.131 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.001675908, 2020-06-12T02:37:43.131 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0017042509999999999, 2020-06-12T02:37:43.131 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:43.131 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:43.131 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:43.132 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:43.132 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:43.132 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 4733 2020-06-12T02:37:43.132 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.132 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:43.133 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.133 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:43.133 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.126469+0000", 2020-06-12T02:37:43.133 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:43.133 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.134 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.134 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:43.134 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.126469+0000", 2020-06-12T02:37:43.134 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6007999999999999e-05 2020-06-12T02:37:43.134 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.134 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.135 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:43.135 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.126495+0000", 2020-06-12T02:37:43.135 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4379999999999997e-06 2020-06-12T02:37:43.135 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.135 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.136 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:43.136 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.126499+0000", 2020-06-12T02:37:43.136 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2723e-05 2020-06-12T02:37:43.136 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.136 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.136 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:43.137 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.126512+0000", 2020-06-12T02:37:43.137 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4059999999999998e-06 2020-06-12T02:37:43.137 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.137 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.137 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:43.138 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.126516+0000", 2020-06-12T02:37:43.138 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0097999999999999e-05 2020-06-12T02:37:43.138 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.138 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.138 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:43.139 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.126546+0000", 2020-06-12T02:37:43.139 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0201999999999997e-05 2020-06-12T02:37:43.139 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.139 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.140 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:43.140 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.126607+0000", 2020-06-12T02:37:43.140 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00028199300000000002 2020-06-12T02:37:43.140 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.140 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.141 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:43.141 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.126889+0000", 2020-06-12T02:37:43.141 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054423999999999998 2020-06-12T02:37:43.141 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.141 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.141 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:43.142 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.127433+0000", 2020-06-12T02:37:43.142 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00096410799999999998 2020-06-12T02:37:43.142 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.142 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:43.142 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.143 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.143 INFO:teuthology.orchestra.run.smithi188.stdout: ], 2020-06-12T02:37:43.143 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 1 2020-06-12T02:37:43.143 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:37:43.146 INFO:tasks.rados.rados.0.smithi107.stdout:2243: done (0 left) 2020-06-12T02:37:43.146 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:37:43.147 INFO:tasks.rados.rados.0.smithi107.stdout:2244: seq_num 2243 ranges {0=770048,770048=417792,1187840=385024} 2020-06-12T02:37:43.147 INFO:tasks.rados.rados.0.smithi107.stdout:2244: writing smithi10729407-9 from 0 to 770048 tid 1 2020-06-12T02:37:43.147 INFO:tasks.rados.rados.0.smithi107.stdout:2244: writing smithi10729407-9 from 770048 to 1187840 tid 2 2020-06-12T02:37:43.147 INFO:tasks.rados.rados.0.smithi107.stdout:2244: writing smithi10729407-9 from 1187840 to 1572864 tid 3 2020-06-12T02:37:43.147 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:37:43.148 INFO:tasks.rados.rados.0.smithi107.stdout:2245: seq_num 2244 ranges {0=786432,786432=704512,1490944=98304} 2020-06-12T02:37:43.148 INFO:tasks.rados.rados.0.smithi107.stdout:2245: writing smithi10729407-3 from 0 to 786432 tid 1 2020-06-12T02:37:43.148 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-06-12T02:37:43.149 INFO:tasks.rados.rados.0.smithi107.stdout:2245: writing smithi10729407-3 from 786432 to 1490944 tid 2 2020-06-12T02:37:43.150 INFO:tasks.rados.rados.0.smithi107.stdout:2245: writing smithi10729407-3 from 1490944 to 1589248 tid 3 2020-06-12T02:37:43.150 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:43.150 INFO:tasks.rados.rados.0.smithi107.stdout:2246: seq_num 2245 ranges {3399680=802816,4202496=589824,4792320=212992} 2020-06-12T02:37:43.173 INFO:tasks.rados.rados.0.smithi107.stdout:2246: writing smithi10729407-2 from 3399680 to 4202496 tid 1 2020-06-12T02:37:43.176 INFO:tasks.rados.rados.0.smithi107.stdout:2246: writing smithi10729407-2 from 4202496 to 4792320 tid 2 2020-06-12T02:37:43.178 INFO:tasks.rados.rados.0.smithi107.stdout:2246: writing smithi10729407-2 from 4792320 to 5005312 tid 3 2020-06-12T02:37:43.178 INFO:tasks.rados.rados.0.smithi107.stdout:2247: delete oid 5 current snap is 0 2020-06-12T02:37:43.178 INFO:tasks.rados.rados.0.smithi107.stdout:2244: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:43.178 INFO:tasks.rados.rados.0.smithi107.stdout:2244: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:43.178 INFO:tasks.rados.rados.0.smithi107.stdout:2244: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:43.179 INFO:tasks.rados.rados.0.smithi107.stdout:2244: finishing write tid 5 to smithi10729407-9 2020-06-12T02:37:43.179 INFO:tasks.rados.rados.0.smithi107.stdout:2244: finishing write tid 6 to smithi10729407-9 2020-06-12T02:37:43.179 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 389 (ObjNum 2243 snap 0 seq_num 2243) dirty exists 2020-06-12T02:37:43.179 INFO:tasks.rados.rados.0.smithi107.stdout:2244: left oid 9 (ObjNum 2243 snap 0 seq_num 2243) 2020-06-12T02:37:43.179 INFO:tasks.rados.rados.0.smithi107.stdout:2245: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:43.180 INFO:tasks.rados.rados.0.smithi107.stdout:2245: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:43.180 INFO:tasks.rados.rados.0.smithi107.stdout:2245: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:43.180 INFO:tasks.rados.rados.0.smithi107.stdout:2245: finishing write tid 5 to smithi10729407-3 2020-06-12T02:37:43.180 INFO:tasks.rados.rados.0.smithi107.stdout:2245: finishing write tid 6 to smithi10729407-3 2020-06-12T02:37:43.180 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 389 (ObjNum 2244 snap 0 seq_num 2244) dirty exists 2020-06-12T02:37:43.181 INFO:tasks.rados.rados.0.smithi107.stdout:2245: left oid 3 (ObjNum 2244 snap 0 seq_num 2244) 2020-06-12T02:37:43.181 INFO:tasks.rados.rados.0.smithi107.stdout:2246: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:43.182 INFO:tasks.rados.rados.0.smithi107.stdout:2244: done (3 left) 2020-06-12T02:37:43.182 INFO:tasks.rados.rados.0.smithi107.stdout:2245: done (2 left) 2020-06-12T02:37:43.183 INFO:tasks.rados.rados.0.smithi107.stdout:2247: done (1 left) 2020-06-12T02:37:43.183 INFO:tasks.rados.rados.0.smithi107.stdout:2248: delete oid 1 current snap is 0 2020-06-12T02:37:43.183 INFO:tasks.rados.rados.0.smithi107.stdout:2248: done (1 left) 2020-06-12T02:37:43.184 INFO:tasks.rados.rados.0.smithi107.stdout:2249: delete oid 8 current snap is 0 2020-06-12T02:37:43.188 INFO:tasks.rados.rados.0.smithi107.stdout:2249: done (1 left) 2020-06-12T02:37:43.188 INFO:tasks.rados.rados.0.smithi107.stdout:2250: delete oid 16 current snap is 0 2020-06-12T02:37:43.189 INFO:tasks.rados.rados.0.smithi107.stdout:2246: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:43.189 INFO:tasks.rados.rados.0.smithi107.stdout:2246: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:43.190 INFO:tasks.rados.rados.0.smithi107.stdout:2246: finishing write tid 5 to smithi10729407-2 2020-06-12T02:37:43.191 INFO:tasks.rados.rados.0.smithi107.stdout:2250: done (1 left) 2020-06-12T02:37:43.191 INFO:tasks.rados.rados.0.smithi107.stdout:2251: delete oid 10 current snap is 0 2020-06-12T02:37:43.191 INFO:tasks.rados.rados.0.smithi107.stdout:2246: finishing write tid 6 to smithi10729407-2 2020-06-12T02:37:43.191 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 243 (ObjNum 2245 snap 0 seq_num 2245) dirty exists 2020-06-12T02:37:43.191 INFO:tasks.rados.rados.0.smithi107.stdout:2246: left oid 2 (ObjNum 2245 snap 0 seq_num 2245) 2020-06-12T02:37:43.194 INFO:tasks.rados.rados.0.smithi107.stdout:2246: done (1 left) 2020-06-12T02:37:43.194 INFO:tasks.rados.rados.0.smithi107.stdout:2251: done (0 left) 2020-06-12T02:37:43.194 INFO:tasks.rados.rados.0.smithi107.stdout:2252: delete oid 17 current snap is 0 2020-06-12T02:37:43.195 INFO:tasks.rados.rados.0.smithi107.stdout:2252: done (0 left) 2020-06-12T02:37:43.195 INFO:tasks.rados.rados.0.smithi107.stdout:2253: delete oid 17 current snap is 0 2020-06-12T02:37:43.197 INFO:tasks.rados.rados.0.smithi107.stdout:2253: done (0 left) 2020-06-12T02:37:43.198 INFO:tasks.rados.rados.0.smithi107.stdout:2254: delete oid 4 current snap is 0 2020-06-12T02:37:43.200 INFO:tasks.rados.rados.0.smithi107.stdout:2254: done (0 left) 2020-06-12T02:37:43.200 INFO:tasks.rados.rados.0.smithi107.stdout:2255: delete oid 10 current snap is 0 2020-06-12T02:37:43.201 INFO:tasks.rados.rados.0.smithi107.stdout:2255: done (0 left) 2020-06-12T02:37:43.202 INFO:tasks.rados.rados.0.smithi107.stdout:2256: delete oid 10 current snap is 0 2020-06-12T02:37:43.203 INFO:tasks.rados.rados.0.smithi107.stdout:2256: done (0 left) 2020-06-12T02:37:43.203 INFO:tasks.rados.rados.0.smithi107.stdout:2257: delete oid 15 current snap is 0 2020-06-12T02:37:43.206 INFO:tasks.rados.rados.0.smithi107.stdout:2257: done (0 left) 2020-06-12T02:37:43.206 INFO:tasks.rados.rados.0.smithi107.stdout:2258: delete oid 2 current snap is 0 2020-06-12T02:37:43.210 INFO:tasks.rados.rados.0.smithi107.stdout:2258: done (0 left) 2020-06-12T02:37:43.210 INFO:tasks.rados.rados.0.smithi107.stdout:2259: delete oid 10 current snap is 0 2020-06-12T02:37:43.213 INFO:tasks.rados.rados.0.smithi107.stdout:2259: done (0 left) 2020-06-12T02:37:43.214 INFO:tasks.rados.rados.0.smithi107.stdout:2260: delete oid 6 current snap is 0 2020-06-12T02:37:43.214 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:37:43.211+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:37:43.216 INFO:tasks.rados.rados.0.smithi107.stdout:2260: done (0 left) 2020-06-12T02:37:43.216 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:37:43.217 INFO:tasks.rados.rados.0.smithi107.stdout:2261: seq_num 2260 ranges {1269760=655360,1925120=548864,2473984=655360} 2020-06-12T02:37:43.227 INFO:tasks.rados.rados.0.smithi107.stdout:2261: writing smithi10729407-11 from 1269760 to 1925120 tid 1 2020-06-12T02:37:43.229 INFO:tasks.rados.rados.0.smithi107.stdout:2261: writing smithi10729407-11 from 1925120 to 2473984 tid 2 2020-06-12T02:37:43.234 INFO:tasks.rados.rados.0.smithi107.stdout:2261: writing smithi10729407-11 from 2473984 to 3129344 tid 3 2020-06-12T02:37:43.234 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:37:43.234 INFO:tasks.rados.rados.0.smithi107.stdout:2262: seq_num 2261 ranges {1572864=671744,2244608=434176,2678784=557056,3235840=212992} 2020-06-12T02:37:43.246 INFO:tasks.rados.rados.0.smithi107.stdout:2262: writing smithi10729407-9 from 1572864 to 2244608 tid 1 2020-06-12T02:37:43.249 INFO:tasks.rados.rados.0.smithi107.stdout:2262: writing smithi10729407-9 from 2244608 to 2678784 tid 2 2020-06-12T02:37:43.253 INFO:tasks.rados.rados.0.smithi107.stdout:2262: writing smithi10729407-9 from 2678784 to 3235840 tid 3 2020-06-12T02:37:43.255 INFO:tasks.rados.rados.0.smithi107.stdout:2262: writing smithi10729407-9 from 3235840 to 3448832 tid 4 2020-06-12T02:37:43.255 INFO:tasks.rados.rados.0.smithi107.stdout:2263: delete oid 7 current snap is 0 2020-06-12T02:37:43.255 INFO:tasks.rados.rados.0.smithi107.stdout:2261: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:43.255 INFO:tasks.rados.rados.0.smithi107.stdout:2261: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:43.256 INFO:tasks.rados.rados.0.smithi107.stdout:2261: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:43.256 INFO:tasks.rados.rados.0.smithi107.stdout:2261: finishing write tid 5 to smithi10729407-11 2020-06-12T02:37:43.256 INFO:tasks.rados.rados.0.smithi107.stdout:2261: finishing write tid 6 to smithi10729407-11 2020-06-12T02:37:43.256 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 416 (ObjNum 2260 snap 0 seq_num 2260) dirty exists 2020-06-12T02:37:43.256 INFO:tasks.rados.rados.0.smithi107.stdout:2261: left oid 11 (ObjNum 2260 snap 0 seq_num 2260) 2020-06-12T02:37:43.258 INFO:tasks.rados.rados.0.smithi107.stdout:2261: done (2 left) 2020-06-12T02:37:43.258 INFO:tasks.rados.rados.0.smithi107.stdout:2263: done (1 left) 2020-06-12T02:37:43.258 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:37:43.259 INFO:tasks.rados.rados.0.smithi107.stdout:2264: seq_num 2263 ranges {0=704512,704512=606208,1310720=524288,1835008=73728} 2020-06-12T02:37:43.263 INFO:tasks.rados.rados.0.smithi107.stdout:2264: writing smithi10729407-10 from 0 to 704512 tid 1 2020-06-12T02:37:43.267 INFO:tasks.rados.rados.0.smithi107.stdout:2264: writing smithi10729407-10 from 704512 to 1310720 tid 2 2020-06-12T02:37:43.270 INFO:tasks.rados.rados.0.smithi107.stdout:2264: writing smithi10729407-10 from 1310720 to 1835008 tid 3 2020-06-12T02:37:43.271 INFO:tasks.rados.rados.0.smithi107.stdout:2264: writing smithi10729407-10 from 1835008 to 1908736 tid 4 2020-06-12T02:37:43.271 INFO:tasks.rados.rados.0.smithi107.stdout:2265: delete oid 1 current snap is 0 2020-06-12T02:37:43.271 INFO:tasks.rados.rados.0.smithi107.stdout:2262: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:43.271 INFO:tasks.rados.rados.0.smithi107.stdout:2262: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:43.272 INFO:tasks.rados.rados.0.smithi107.stdout:2262: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:43.272 INFO:tasks.rados.rados.0.smithi107.stdout:2262: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:43.272 INFO:tasks.rados.rados.0.smithi107.stdout:2262: finishing write tid 6 to smithi10729407-9 2020-06-12T02:37:43.272 INFO:tasks.rados.rados.0.smithi107.stdout:2262: finishing write tid 7 to smithi10729407-9 2020-06-12T02:37:43.272 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 394 (ObjNum 2261 snap 0 seq_num 2261) dirty exists 2020-06-12T02:37:43.273 INFO:tasks.rados.rados.0.smithi107.stdout:2262: left oid 9 (ObjNum 2261 snap 0 seq_num 2261) 2020-06-12T02:37:43.273 INFO:tasks.rados.rados.0.smithi107.stdout:2264: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:43.273 INFO:tasks.rados.rados.0.smithi107.stdout:2262: done (2 left) 2020-06-12T02:37:43.274 INFO:tasks.rados.rados.0.smithi107.stdout:2265: done (1 left) 2020-06-12T02:37:43.274 INFO:tasks.rados.rados.0.smithi107.stdout:2266: delete oid 9 current snap is 0 2020-06-12T02:37:43.277 INFO:tasks.rados.rados.0.smithi107.stdout:2266: done (1 left) 2020-06-12T02:37:43.277 INFO:tasks.rados.rados.0.smithi107.stdout:2267: delete oid 4 current snap is 0 2020-06-12T02:37:43.278 INFO:tasks.rados.rados.0.smithi107.stdout:2264: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:43.278 INFO:tasks.rados.rados.0.smithi107.stdout:2267: done (1 left) 2020-06-12T02:37:43.278 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:37:43.278 INFO:tasks.rados.rados.0.smithi107.stdout:2268: seq_num 2267 ranges {0=778240,778240=589824,1368064=507904,1875968=106496} 2020-06-12T02:37:43.282 INFO:tasks.rados.rados.0.smithi107.stdout:2268: writing smithi10729407-15 from 0 to 778240 tid 1 2020-06-12T02:37:43.286 INFO:tasks.rados.rados.0.smithi107.stdout:2268: writing smithi10729407-15 from 778240 to 1368064 tid 2 2020-06-12T02:37:43.290 INFO:tasks.rados.rados.0.smithi107.stdout:2268: writing smithi10729407-15 from 1368064 to 1875968 tid 3 2020-06-12T02:37:43.291 INFO:tasks.rados.rados.0.smithi107.stdout:2268: writing smithi10729407-15 from 1875968 to 1982464 tid 4 2020-06-12T02:37:43.291 INFO:tasks.rados.rados.0.smithi107.stdout:2269: delete oid 6 current snap is 0 2020-06-12T02:37:43.291 INFO:tasks.rados.rados.0.smithi107.stdout:2264: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:43.291 INFO:tasks.rados.rados.0.smithi107.stdout:2264: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:43.291 INFO:tasks.rados.rados.0.smithi107.stdout:2264: finishing write tid 6 to smithi10729407-10 2020-06-12T02:37:43.292 INFO:tasks.rados.rados.0.smithi107.stdout:2264: finishing write tid 7 to smithi10729407-10 2020-06-12T02:37:43.292 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 209 (ObjNum 2263 snap 0 seq_num 2263) dirty exists 2020-06-12T02:37:43.292 INFO:tasks.rados.rados.0.smithi107.stdout:2264: left oid 10 (ObjNum 2263 snap 0 seq_num 2263) 2020-06-12T02:37:43.293 INFO:tasks.rados.rados.0.smithi107.stdout:2264: done (2 left) 2020-06-12T02:37:43.293 INFO:tasks.rados.rados.0.smithi107.stdout:2269: done (1 left) 2020-06-12T02:37:43.293 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:37:43.293 INFO:tasks.rados.rados.0.smithi107.stdout:2270: seq_num 2269 ranges {0=409600,409600=761856,1171456=524288,1695744=319488} 2020-06-12T02:37:43.294 INFO:tasks.rados.rados.0.smithi107.stdout:2270: writing smithi10729407-6 from 0 to 409600 tid 1 2020-06-12T02:37:43.298 INFO:tasks.rados.rados.0.smithi107.stdout:2270: writing smithi10729407-6 from 409600 to 1171456 tid 2 2020-06-12T02:37:43.301 INFO:tasks.rados.rados.0.smithi107.stdout:2270: writing smithi10729407-6 from 1171456 to 1695744 tid 3 2020-06-12T02:37:43.303 INFO:tasks.rados.rados.0.smithi107.stdout:2270: writing smithi10729407-6 from 1695744 to 2015232 tid 4 2020-06-12T02:37:43.304 INFO:tasks.rados.rados.0.smithi107.stdout:2271: delete oid 17 current snap is 0 2020-06-12T02:37:43.304 INFO:tasks.rados.rados.0.smithi107.stdout:2268: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:43.304 INFO:tasks.rados.rados.0.smithi107.stdout:2268: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:43.304 INFO:tasks.rados.rados.0.smithi107.stdout:2268: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:43.304 INFO:tasks.rados.rados.0.smithi107.stdout:2268: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:43.305 INFO:tasks.rados.rados.0.smithi107.stdout:2268: finishing write tid 6 to smithi10729407-15 2020-06-12T02:37:43.305 INFO:tasks.rados.rados.0.smithi107.stdout:2270: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:43.305 INFO:tasks.rados.rados.0.smithi107.stdout:2268: finishing write tid 7 to smithi10729407-15 2020-06-12T02:37:43.305 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 383 (ObjNum 2267 snap 0 seq_num 2267) dirty exists 2020-06-12T02:37:43.305 INFO:tasks.rados.rados.0.smithi107.stdout:2268: left oid 15 (ObjNum 2267 snap 0 seq_num 2267) 2020-06-12T02:37:43.306 INFO:tasks.rados.rados.0.smithi107.stdout:2268: done (2 left) 2020-06-12T02:37:43.306 INFO:tasks.rados.rados.0.smithi107.stdout:2271: done (1 left) 2020-06-12T02:37:43.306 INFO:tasks.rados.rados.0.smithi107.stdout:2272: delete oid 4 current snap is 0 2020-06-12T02:37:43.308 INFO:tasks.rados.rados.0.smithi107.stdout:2272: done (1 left) 2020-06-12T02:37:43.309 INFO:tasks.rados.rados.0.smithi107.stdout:2273: delete oid 14 current snap is 0 2020-06-12T02:37:43.309 INFO:tasks.rados.rados.0.smithi107.stdout:2270: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:43.309 INFO:tasks.rados.rados.0.smithi107.stdout:2270: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:43.310 INFO:tasks.rados.rados.0.smithi107.stdout:2270: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:43.310 INFO:tasks.rados.rados.0.smithi107.stdout:2270: finishing write tid 6 to smithi10729407-6 2020-06-12T02:37:43.311 INFO:tasks.rados.rados.0.smithi107.stdout:2273: done (1 left) 2020-06-12T02:37:43.312 INFO:tasks.rados.rados.0.smithi107.stdout:2274: delete oid 15 current snap is 0 2020-06-12T02:37:43.312 INFO:tasks.rados.rados.0.smithi107.stdout:2270: finishing write tid 7 to smithi10729407-6 2020-06-12T02:37:43.312 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 470 (ObjNum 2269 snap 0 seq_num 2269) dirty exists 2020-06-12T02:37:43.312 INFO:tasks.rados.rados.0.smithi107.stdout:2270: left oid 6 (ObjNum 2269 snap 0 seq_num 2269) 2020-06-12T02:37:43.313 INFO:tasks.rados.rados.0.smithi107.stdout:2270: done (1 left) 2020-06-12T02:37:43.313 INFO:tasks.rados.rados.0.smithi107.stdout:2274: done (0 left) 2020-06-12T02:37:43.313 INFO:tasks.rados.rados.0.smithi107.stdout:2275: delete oid 2 current snap is 0 2020-06-12T02:37:43.315 INFO:tasks.rados.rados.0.smithi107.stdout:2275: done (0 left) 2020-06-12T02:37:43.315 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:37:43.315 INFO:tasks.rados.rados.0.smithi107.stdout:2276: seq_num 2275 ranges {0=507904,507904=516096,1024000=516096,1540096=548864,2088960=24576} 2020-06-12T02:37:43.316 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:37:43.312+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:37:43.317 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:37:43.317 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:37:43.317 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:37:43.318 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:37:43.318 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:37:43.318 INFO:tasks.rados.rados.0.smithi107.stdout:2276: writing smithi10729407-14 from 0 to 507904 tid 1 2020-06-12T02:37:43.320 INFO:tasks.rados.rados.0.smithi107.stdout:2276: writing smithi10729407-14 from 507904 to 1024000 tid 2 2020-06-12T02:37:43.324 INFO:tasks.rados.rados.0.smithi107.stdout:2276: writing smithi10729407-14 from 1024000 to 1540096 tid 3 2020-06-12T02:37:43.328 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-06-12T02:37:43.329 INFO:tasks.rados.rados.0.smithi107.stdout:2276: writing smithi10729407-14 from 1540096 to 2088960 tid 4 2020-06-12T02:37:43.330 INFO:tasks.rados.rados.0.smithi107.stdout:2276: writing smithi10729407-14 from 2088960 to 2113536 tid 5 2020-06-12T02:37:43.330 INFO:tasks.rados.rados.0.smithi107.stdout:2277: delete oid 18 current snap is 0 2020-06-12T02:37:43.330 INFO:tasks.rados.rados.0.smithi107.stdout:2276: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:43.330 INFO:tasks.rados.rados.0.smithi107.stdout:2276: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:43.332 INFO:tasks.rados.rados.0.smithi107.stdout:2277: done (1 left) 2020-06-12T02:37:43.332 INFO:tasks.rados.rados.0.smithi107.stdout:2278: delete oid 16 current snap is 0 2020-06-12T02:37:43.333 INFO:tasks.rados.rados.0.smithi107.stdout:2276: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:43.336 INFO:tasks.rados.rados.0.smithi107.stdout:2276: finishing write tid 4 to smithi10729407-14 2020-06-12T02:37:43.336 INFO:tasks.rados.rados.0.smithi107.stdout:2278: done (1 left) 2020-06-12T02:37:43.336 INFO:tasks.rados.rados.0.smithi107.stdout:2279: delete oid 17 current snap is 0 2020-06-12T02:37:43.337 INFO:tasks.rados.rados.0.smithi107.stdout:2276: finishing write tid 5 to smithi10729407-14 2020-06-12T02:37:43.337 INFO:tasks.rados.rados.0.smithi107.stdout:2276: finishing write tid 7 to smithi10729407-14 2020-06-12T02:37:43.338 INFO:tasks.rados.rados.0.smithi107.stdout:2279: done (1 left) 2020-06-12T02:37:43.338 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:37:43.338 INFO:tasks.rados.rados.0.smithi107.stdout:2280: seq_num 2279 ranges {0=573440,573440=450560,1024000=499712,1523712=606208,2129920=49152} 2020-06-12T02:37:43.341 INFO:tasks.rados.rados.0.smithi107.stdout:2280: writing smithi10729407-13 from 0 to 573440 tid 1 2020-06-12T02:37:43.344 INFO:tasks.rados.rados.0.smithi107.stdout:2280: writing smithi10729407-13 from 573440 to 1024000 tid 2 2020-06-12T02:37:43.348 INFO:tasks.rados.rados.0.smithi107.stdout:2280: writing smithi10729407-13 from 1024000 to 1523712 tid 3 2020-06-12T02:37:43.352 INFO:tasks.rados.rados.0.smithi107.stdout:2280: writing smithi10729407-13 from 1523712 to 2129920 tid 4 2020-06-12T02:37:43.352 INFO:tasks.rados.rados.0.smithi107.stdout:2280: writing smithi10729407-13 from 2129920 to 2179072 tid 5 2020-06-12T02:37:43.352 INFO:tasks.rados.rados.0.smithi107.stdout:2281: delete oid 8 current snap is 0 2020-06-12T02:37:43.353 INFO:tasks.rados.rados.0.smithi107.stdout:2276: finishing write tid 8 to smithi10729407-14 2020-06-12T02:37:43.353 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 423 (ObjNum 2275 snap 0 seq_num 2275) dirty exists 2020-06-12T02:37:43.353 INFO:tasks.rados.rados.0.smithi107.stdout:2276: left oid 14 (ObjNum 2275 snap 0 seq_num 2275) 2020-06-12T02:37:43.353 INFO:tasks.rados.rados.0.smithi107.stdout:2280: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:43.354 INFO:tasks.rados.rados.0.smithi107.stdout:2276: done (2 left) 2020-06-12T02:37:43.355 INFO:tasks.rados.rados.0.smithi107.stdout:2281: done (1 left) 2020-06-12T02:37:43.355 INFO:tasks.rados.rados.0.smithi107.stdout:2282: delete oid 3 current snap is 0 2020-06-12T02:37:43.359 INFO:tasks.rados.rados.0.smithi107.stdout:2280: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:43.359 INFO:tasks.rados.rados.0.smithi107.stdout:2280: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:43.360 INFO:tasks.rados.rados.0.smithi107.stdout:2280: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:43.361 INFO:tasks.rados.rados.0.smithi107.stdout:2280: finishing write tid 5 to smithi10729407-13 2020-06-12T02:37:43.361 INFO:tasks.rados.rados.0.smithi107.stdout:2282: done (1 left) 2020-06-12T02:37:43.361 INFO:tasks.rados.rados.0.smithi107.stdout:2283: delete oid 18 current snap is 0 2020-06-12T02:37:43.363 INFO:tasks.rados.rados.0.smithi107.stdout:2280: finishing write tid 7 to smithi10729407-13 2020-06-12T02:37:43.365 INFO:tasks.rados.rados.0.smithi107.stdout:2280: finishing write tid 8 to smithi10729407-13 2020-06-12T02:37:43.365 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 430 (ObjNum 2279 snap 0 seq_num 2279) dirty exists 2020-06-12T02:37:43.365 INFO:tasks.rados.rados.0.smithi107.stdout:2280: left oid 13 (ObjNum 2279 snap 0 seq_num 2279) 2020-06-12T02:37:43.366 INFO:tasks.rados.rados.0.smithi107.stdout:2280: done (1 left) 2020-06-12T02:37:43.366 INFO:tasks.rados.rados.0.smithi107.stdout:2283: done (0 left) 2020-06-12T02:37:43.366 INFO:tasks.rados.rados.0.smithi107.stdout:2284: delete oid 4 current snap is 0 2020-06-12T02:37:43.370 INFO:tasks.rados.rados.0.smithi107.stdout:2284: done (0 left) 2020-06-12T02:37:43.370 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:37:43.370 INFO:tasks.rados.rados.0.smithi107.stdout:2285: seq_num 2284 ranges {0=655360,655360=720896,1376256=638976,2015232=245760} 2020-06-12T02:37:43.374 INFO:tasks.rados.rados.0.smithi107.stdout:2285: writing smithi10729407-19 from 0 to 655360 tid 1 2020-06-12T02:37:43.378 INFO:tasks.rados.rados.0.smithi107.stdout:2285: writing smithi10729407-19 from 655360 to 1376256 tid 2 2020-06-12T02:37:43.382 INFO:tasks.rados.rados.0.smithi107.stdout:2285: writing smithi10729407-19 from 1376256 to 2015232 tid 3 2020-06-12T02:37:43.383 INFO:tasks.rados.rados.0.smithi107.stdout:2285: writing smithi10729407-19 from 2015232 to 2260992 tid 4 2020-06-12T02:37:43.384 INFO:tasks.rados.rados.0.smithi107.stdout:2286: delete oid 10 current snap is 0 2020-06-12T02:37:43.384 INFO:tasks.rados.rados.0.smithi107.stdout:2285: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:43.386 INFO:tasks.rados.rados.0.smithi107.stdout:2286: done (1 left) 2020-06-12T02:37:43.386 INFO:tasks.rados.rados.0.smithi107.stdout:2287: delete oid 3 current snap is 0 2020-06-12T02:37:43.389 INFO:tasks.rados.rados.0.smithi107.stdout:2287: done (1 left) 2020-06-12T02:37:43.390 INFO:tasks.rados.rados.0.smithi107.stdout:2288: delete oid 7 current snap is 0 2020-06-12T02:37:43.390 INFO:tasks.rados.rados.0.smithi107.stdout:2285: finishing write tid 2 to smithi10729407-19 2020-06-12T02:37:43.391 INFO:tasks.rados.rados.0.smithi107.stdout:2285: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:43.391 INFO:tasks.rados.rados.0.smithi107.stdout:2285: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:43.392 INFO:tasks.rados.rados.0.smithi107.stdout:2288: done (1 left) 2020-06-12T02:37:43.392 INFO:tasks.rados.rados.0.smithi107.stdout:2289: delete oid 16 current snap is 0 2020-06-12T02:37:43.393 INFO:tasks.rados.rados.0.smithi107.stdout:2285: finishing write tid 6 to smithi10729407-19 2020-06-12T02:37:43.398 INFO:tasks.rados.rados.0.smithi107.stdout:2285: finishing write tid 7 to smithi10729407-19 2020-06-12T02:37:43.398 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 351 (ObjNum 2284 snap 0 seq_num 2284) dirty exists 2020-06-12T02:37:43.398 INFO:tasks.rados.rados.0.smithi107.stdout:2285: left oid 19 (ObjNum 2284 snap 0 seq_num 2284) 2020-06-12T02:37:43.399 INFO:tasks.rados.rados.0.smithi107.stdout:2285: done (1 left) 2020-06-12T02:37:43.399 INFO:tasks.rados.rados.0.smithi107.stdout:2289: done (0 left) 2020-06-12T02:37:43.400 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:37:43.400 INFO:tasks.rados.rados.0.smithi107.stdout:2290: seq_num 2289 ranges {0=745472,745472=598016,1343488=425984,1769472=491520,2260992=90112} 2020-06-12T02:37:43.403 INFO:tasks.rados.rados.0.smithi107.stdout:2290: writing smithi10729407-10 from 0 to 745472 tid 1 2020-06-12T02:37:43.407 INFO:tasks.rados.rados.0.smithi107.stdout:2290: writing smithi10729407-10 from 745472 to 1343488 tid 2 2020-06-12T02:37:43.410 INFO:tasks.rados.rados.0.smithi107.stdout:2290: writing smithi10729407-10 from 1343488 to 1769472 tid 3 2020-06-12T02:37:43.413 INFO:tasks.rados.rados.0.smithi107.stdout:2290: writing smithi10729407-10 from 1769472 to 2260992 tid 4 2020-06-12T02:37:43.414 INFO:tasks.rados.rados.0.smithi107.stdout:2290: writing smithi10729407-10 from 2260992 to 2351104 tid 5 2020-06-12T02:37:43.414 INFO:tasks.rados.rados.0.smithi107.stdout:2291: delete oid 2 current snap is 0 2020-06-12T02:37:43.415 INFO:tasks.rados.rados.0.smithi107.stdout:2290: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:43.416 INFO:tasks.rados.rados.0.smithi107.stdout:2290: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:43.416 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:43.413+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:43.418 INFO:tasks.rados.rados.0.smithi107.stdout:2290: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:43.419 INFO:tasks.rados.rados.0.smithi107.stdout:2290: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:43.421 INFO:tasks.rados.rados.0.smithi107.stdout:2291: done (1 left) 2020-06-12T02:37:43.421 INFO:tasks.rados.rados.0.smithi107.stdout:2292: delete oid 2 current snap is 0 2020-06-12T02:37:43.421 INFO:tasks.rados.rados.0.smithi107.stdout:2290: finishing write tid 5 to smithi10729407-10 2020-06-12T02:37:43.422 INFO:tasks.rados.rados.0.smithi107.stdout:2290: finishing write tid 7 to smithi10729407-10 2020-06-12T02:37:43.423 INFO:tasks.rados.rados.0.smithi107.stdout:2290: finishing write tid 8 to smithi10729407-10 2020-06-12T02:37:43.438 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 216 (ObjNum 2289 snap 0 seq_num 2289) dirty exists 2020-06-12T02:37:43.438 INFO:tasks.rados.rados.0.smithi107.stdout:2290: left oid 10 (ObjNum 2289 snap 0 seq_num 2289) 2020-06-12T02:37:43.439 INFO:tasks.rados.rados.0.smithi107.stdout:2290: done (1 left) 2020-06-12T02:37:43.440 INFO:tasks.rados.rados.0.smithi107.stdout:2292: done (0 left) 2020-06-12T02:37:43.440 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:37:43.440 INFO:tasks.rados.rados.0.smithi107.stdout:2293: seq_num 2292 ranges {0=794624,794624=647168,1441792=598016,2039808=360448} 2020-06-12T02:37:43.440 INFO:tasks.rados.rados.0.smithi107.stdout:2293: writing smithi10729407-1 from 0 to 794624 tid 1 2020-06-12T02:37:43.441 INFO:tasks.rados.rados.0.smithi107.stdout:2293: writing smithi10729407-1 from 794624 to 1441792 tid 2 2020-06-12T02:37:43.441 INFO:tasks.rados.rados.0.smithi107.stdout:2293: writing smithi10729407-1 from 1441792 to 2039808 tid 3 2020-06-12T02:37:43.441 INFO:tasks.rados.rados.0.smithi107.stdout:2293: writing smithi10729407-1 from 2039808 to 2400256 tid 4 2020-06-12T02:37:43.441 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:37:43.441 INFO:tasks.rados.rados.0.smithi107.stdout:2294: seq_num 2293 ranges {2015232=8192} 2020-06-12T02:37:43.455 INFO:tasks.rados.rados.0.smithi107.stdout:2294: writing smithi10729407-6 from 2015232 to 2023424 tid 1 2020-06-12T02:37:43.455 INFO:tasks.rados.rados.0.smithi107.stdout:2295: delete oid 7 current snap is 0 2020-06-12T02:37:43.455 INFO:tasks.rados.rados.0.smithi107.stdout:2293: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:43.456 INFO:tasks.rados.rados.0.smithi107.stdout:2293: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:43.456 INFO:tasks.rados.rados.0.smithi107.stdout:2293: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:43.456 INFO:tasks.rados.rados.0.smithi107.stdout:2293: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:43.456 INFO:tasks.rados.rados.0.smithi107.stdout:2293: finishing write tid 6 to smithi10729407-1 2020-06-12T02:37:43.456 INFO:tasks.rados.rados.0.smithi107.stdout:2293: finishing write tid 7 to smithi10729407-1 2020-06-12T02:37:43.457 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 475 (ObjNum 2292 snap 0 seq_num 2292) dirty exists 2020-06-12T02:37:43.457 INFO:tasks.rados.rados.0.smithi107.stdout:2293: left oid 1 (ObjNum 2292 snap 0 seq_num 2292) 2020-06-12T02:37:43.457 INFO:tasks.rados.rados.0.smithi107.stdout:2294: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:43.457 INFO:tasks.rados.rados.0.smithi107.stdout:2293: done (2 left) 2020-06-12T02:37:43.457 INFO:tasks.rados.rados.0.smithi107.stdout:2295: done (1 left) 2020-06-12T02:37:43.458 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:37:43.458 INFO:tasks.rados.rados.0.smithi107.stdout:2296: seq_num 2295 ranges {0=40960} 2020-06-12T02:37:43.458 INFO:tasks.rados.rados.0.smithi107.stdout:2296: writing smithi10729407-16 from 0 to 40960 tid 1 2020-06-12T02:37:43.458 INFO:tasks.rados.rados.0.smithi107.stdout:2297: delete oid 7 current snap is 0 2020-06-12T02:37:43.458 INFO:tasks.rados.rados.0.smithi107.stdout:2294: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:43.458 INFO:tasks.rados.rados.0.smithi107.stdout:2297: done (2 left) 2020-06-12T02:37:43.459 INFO:tasks.rados.rados.0.smithi107.stdout:2298: delete oid 3 current snap is 0 2020-06-12T02:37:43.459 INFO:tasks.rados.rados.0.smithi107.stdout:2294: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:43.459 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 477 (ObjNum 2293 snap 0 seq_num 2293) dirty exists 2020-06-12T02:37:43.459 INFO:tasks.rados.rados.0.smithi107.stdout:2294: left oid 6 (ObjNum 2293 snap 0 seq_num 2293) 2020-06-12T02:37:43.460 INFO:tasks.rados.rados.0.smithi107.stdout:2296: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:43.460 INFO:tasks.rados.rados.0.smithi107.stdout:2296: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:43.461 INFO:tasks.rados.rados.0.smithi107.stdout:2294: done (2 left) 2020-06-12T02:37:43.461 INFO:tasks.rados.rados.0.smithi107.stdout:2298: done (1 left) 2020-06-12T02:37:43.461 INFO:tasks.rados.rados.0.smithi107.stdout:2299: delete oid 19 current snap is 0 2020-06-12T02:37:43.461 INFO:tasks.rados.rados.0.smithi107.stdout:2296: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:43.461 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 396 (ObjNum 2295 snap 0 seq_num 2295) dirty exists 2020-06-12T02:37:43.461 INFO:tasks.rados.rados.0.smithi107.stdout:2296: left oid 16 (ObjNum 2295 snap 0 seq_num 2295) 2020-06-12T02:37:43.463 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:37:43.464 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:37:43.464 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:37:43.464 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:37:43.464 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.464 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:858 3.3s0 3:ce45c70a:::smithi10729407-7:head [append 0~565248] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:43.465 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.924643+0000", 2020-06-12T02:37:43.465 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.535175431000001, 2020-06-12T02:37:43.465 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015332551999999999, 2020-06-12T02:37:43.465 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:43.465 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:43.466 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:43.466 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:43.466 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:43.466 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 858 2020-06-12T02:37:43.466 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.467 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:43.467 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.467 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:43.467 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.924643+0000", 2020-06-12T02:37:43.467 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:43.468 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.468 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.468 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:43.468 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.924643+0000", 2020-06-12T02:37:43.468 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00067535299999999996 2020-06-12T02:37:43.468 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.469 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.469 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:43.469 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925318+0000", 2020-06-12T02:37:43.469 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9629999999999993e-06 2020-06-12T02:37:43.469 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.470 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.470 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:43.470 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925328+0000", 2020-06-12T02:37:43.470 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7357999999999999e-05 2020-06-12T02:37:43.470 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.470 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.471 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:43.471 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925345+0000", 2020-06-12T02:37:43.471 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7019999999999996e-06 2020-06-12T02:37:43.471 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.471 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.472 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:43.472 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925355+0000", 2020-06-12T02:37:43.472 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6289999999999998e-05 2020-06-12T02:37:43.472 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.472 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.472 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:43.473 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925411+0000", 2020-06-12T02:37:43.473 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8855000000000004e-05 2020-06-12T02:37:43.473 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.473 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.473 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:43.474 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925480+0000", 2020-06-12T02:37:43.474 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0074521149999999996 2020-06-12T02:37:43.474 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.474 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.474 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:43.475 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.932932+0000", 2020-06-12T02:37:43.475 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3647999999999995e-05 2020-06-12T02:37:43.475 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.475 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.475 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:43.475 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.933026+0000", 2020-06-12T02:37:43.476 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00078753199999999997 2020-06-12T02:37:43.476 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.476 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.476 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:43.476 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.933813+0000", 2020-06-12T02:37:43.477 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.006013582 2020-06-12T02:37:43.477 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.477 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.477 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:43.477 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.939827+0000", 2020-06-12T02:37:43.477 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0111999999999996e-05 2020-06-12T02:37:43.478 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.478 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.478 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:43.478 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.939897+0000", 2020-06-12T02:37:43.478 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8041999999999996e-05 2020-06-12T02:37:43.479 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.479 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.479 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:43.479 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.939975+0000", 2020-06-12T02:37:43.479 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015332551999999999 2020-06-12T02:37:43.479 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.480 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:43.480 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.480 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.480 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.480 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:859 3.3s0 3:ce45c70a:::smithi10729407-7:head [append 0~557056] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:43.480 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.927973+0000", 2020-06-12T02:37:43.481 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.531845219999999, 2020-06-12T02:37:43.481 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014425955000000001, 2020-06-12T02:37:43.481 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:43.481 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:43.481 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:43.482 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:43.482 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:43.482 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 859 2020-06-12T02:37:43.482 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.482 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:43.482 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.483 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:43.483 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.927973+0000", 2020-06-12T02:37:43.483 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:43.483 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.483 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.484 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:43.484 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.927973+0000", 2020-06-12T02:37:43.484 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00058871900000000003 2020-06-12T02:37:43.484 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.484 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.484 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:43.485 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928562+0000", 2020-06-12T02:37:43.485 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5679999999999999e-06 2020-06-12T02:37:43.485 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.485 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.485 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:43.486 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928567+0000", 2020-06-12T02:37:43.486 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1883e-05 2020-06-12T02:37:43.486 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.486 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.486 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:43.486 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928579+0000", 2020-06-12T02:37:43.487 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.097e-06 2020-06-12T02:37:43.487 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.487 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.487 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:43.487 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928586+0000", 2020-06-12T02:37:43.488 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6851999999999998e-05 2020-06-12T02:37:43.488 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.488 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.488 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:43.488 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928633+0000", 2020-06-12T02:37:43.488 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6867999999999997e-05 2020-06-12T02:37:43.489 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.489 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.489 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:43.489 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928700+0000", 2020-06-12T02:37:43.489 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0079226580000000008 2020-06-12T02:37:43.490 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.490 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.490 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:43.490 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.936623+0000", 2020-06-12T02:37:43.490 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000108597 2020-06-12T02:37:43.490 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.491 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.491 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:43.491 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.936731+0000", 2020-06-12T02:37:43.491 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054516799999999995 2020-06-12T02:37:43.491 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.492 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.492 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:43.492 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.937276+0000", 2020-06-12T02:37:43.492 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0043134339999999997 2020-06-12T02:37:43.492 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.492 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.493 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:43.493 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941590+0000", 2020-06-12T02:37:43.493 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00074530099999999997 2020-06-12T02:37:43.493 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.493 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.493 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:43.494 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.942335+0000", 2020-06-12T02:37:43.494 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3810000000000001e-05 2020-06-12T02:37:43.494 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.494 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.494 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:43.495 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.942399+0000", 2020-06-12T02:37:43.495 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014425955000000001 2020-06-12T02:37:43.495 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.495 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:43.495 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.495 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.496 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.496 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:860 3.3s0 3:ce45c70a:::smithi10729407-7:head [append 0~81920] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:43.496 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.928619+0000", 2020-06-12T02:37:43.496 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.531199545, 2020-06-12T02:37:43.496 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017431941999999999, 2020-06-12T02:37:43.497 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:43.497 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:43.497 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:43.497 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:43.497 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:43.497 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 860 2020-06-12T02:37:43.498 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.498 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:43.498 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.498 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:43.498 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928619+0000", 2020-06-12T02:37:43.499 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:43.499 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.499 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.499 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:43.499 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928619+0000", 2020-06-12T02:37:43.500 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000105809 2020-06-12T02:37:43.500 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.500 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.500 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:43.500 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928724+0000", 2020-06-12T02:37:43.500 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2339999999999994e-06 2020-06-12T02:37:43.501 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.501 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.501 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:43.501 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928733+0000", 2020-06-12T02:37:43.501 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0268000000000001e-05 2020-06-12T02:37:43.502 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.502 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.502 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:43.502 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928743+0000", 2020-06-12T02:37:43.502 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5190000000000003e-06 2020-06-12T02:37:43.502 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.503 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.503 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:43.503 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928747+0000", 2020-06-12T02:37:43.503 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0103999999999998e-05 2020-06-12T02:37:43.503 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.504 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.504 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:43.504 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928777+0000", 2020-06-12T02:37:43.504 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8249000000000002e-05 2020-06-12T02:37:43.504 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.505 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.505 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:43.505 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928866+0000", 2020-06-12T02:37:43.505 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011151629 2020-06-12T02:37:43.506 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.506 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.506 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:43.506 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.940017+0000", 2020-06-12T02:37:43.506 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8281e-05 2020-06-12T02:37:43.507 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.507 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.507 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:43.507 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.940076+0000", 2020-06-12T02:37:43.507 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000346736 2020-06-12T02:37:43.507 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.508 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.508 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:43.508 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.940422+0000", 2020-06-12T02:37:43.508 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024135049999999998 2020-06-12T02:37:43.508 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.509 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.509 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:43.509 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.942836+0000", 2020-06-12T02:37:43.509 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003170258 2020-06-12T02:37:43.509 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.509 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.510 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:43.510 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946006+0000", 2020-06-12T02:37:43.510 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4350000000000001e-05 2020-06-12T02:37:43.510 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.510 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.511 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:43.511 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946050+0000", 2020-06-12T02:37:43.511 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017431941999999999 2020-06-12T02:37:43.511 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.511 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:43.511 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.512 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.512 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.512 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:861 3.3s0 3:ce45c70a:::smithi10729407-7:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:43.512 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.928778+0000", 2020-06-12T02:37:43.512 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.531039932000001, 2020-06-12T02:37:43.513 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017522738, 2020-06-12T02:37:43.513 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:43.513 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:43.513 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:43.513 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:43.514 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:43.514 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 861 2020-06-12T02:37:43.514 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.514 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:43.514 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.514 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:43.515 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928778+0000", 2020-06-12T02:37:43.515 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:43.515 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.515 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.515 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:43.515 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928778+0000", 2020-06-12T02:37:43.516 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8186000000000002e-05 2020-06-12T02:37:43.516 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.516 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.516 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:43.516 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928826+0000", 2020-06-12T02:37:43.517 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6689999999999997e-06 2020-06-12T02:37:43.517 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.517 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.517 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:43.517 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928831+0000", 2020-06-12T02:37:43.518 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.686e-06 2020-06-12T02:37:43.518 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.518 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.518 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:43.518 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928838+0000", 2020-06-12T02:37:43.518 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7159999999999999e-06 2020-06-12T02:37:43.519 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.519 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.519 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:43.519 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928840+0000", 2020-06-12T02:37:43.519 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4212999999999997e-05 2020-06-12T02:37:43.520 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.520 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.520 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:43.520 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928895+0000", 2020-06-12T02:37:43.520 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5853999999999994e-05 2020-06-12T02:37:43.520 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.521 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.521 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:43.521 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928990+0000", 2020-06-12T02:37:43.521 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012023982000000001 2020-06-12T02:37:43.521 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.522 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.522 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:43.522 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941014+0000", 2020-06-12T02:37:43.522 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5104e-05 2020-06-12T02:37:43.522 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.522 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.523 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:43.523 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941070+0000", 2020-06-12T02:37:43.523 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00026444499999999998 2020-06-12T02:37:43.523 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.523 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.523 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:43.524 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941334+0000", 2020-06-12T02:37:43.524 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001704075 2020-06-12T02:37:43.524 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.524 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.524 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:43.525 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.943038+0000", 2020-06-12T02:37:43.525 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031891570000000002 2020-06-12T02:37:43.525 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.525 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.525 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:43.525 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946227+0000", 2020-06-12T02:37:43.526 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3651000000000003e-05 2020-06-12T02:37:43.526 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.526 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.526 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:43.526 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946301+0000", 2020-06-12T02:37:43.527 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017522738 2020-06-12T02:37:43.527 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.527 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:43.527 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.527 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.527 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.528 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:862 3.3s0 3:ce45c70a:::smithi10729407-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:43.528 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.928876+0000", 2020-06-12T02:37:43.528 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.530942436, 2020-06-12T02:37:43.528 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019206890000000001, 2020-06-12T02:37:43.528 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:43.529 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:43.529 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:43.529 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:43.529 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:43.529 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 862 2020-06-12T02:37:43.529 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.530 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:43.530 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.530 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:43.530 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928876+0000", 2020-06-12T02:37:43.530 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:43.531 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.531 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.531 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:43.531 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928876+0000", 2020-06-12T02:37:43.531 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9268999999999999e-05 2020-06-12T02:37:43.531 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.532 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.532 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:43.532 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928925+0000", 2020-06-12T02:37:43.532 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8120000000000002e-06 2020-06-12T02:37:43.532 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.533 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.533 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:43.533 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928932+0000", 2020-06-12T02:37:43.533 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3471e-05 2020-06-12T02:37:43.533 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.534 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.534 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:43.534 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928945+0000", 2020-06-12T02:37:43.534 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4279999999999998e-06 2020-06-12T02:37:43.534 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.534 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.535 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:43.535 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928950+0000", 2020-06-12T02:37:43.535 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6358000000000004e-05 2020-06-12T02:37:43.535 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.535 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.536 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:43.536 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.929016+0000", 2020-06-12T02:37:43.536 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9697999999999996e-05 2020-06-12T02:37:43.536 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.536 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.537 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:43.537 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.929086+0000", 2020-06-12T02:37:43.537 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012409205 2020-06-12T02:37:43.537 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.537 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.538 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:43.538 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941495+0000", 2020-06-12T02:37:43.538 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9660000000000006e-05 2020-06-12T02:37:43.538 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.538 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.539 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:43.539 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941585+0000", 2020-06-12T02:37:43.539 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050329099999999998 2020-06-12T02:37:43.539 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.539 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.540 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:43.540 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946617+0000", 2020-06-12T02:37:43.540 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8437999999999996e-05 2020-06-12T02:37:43.540 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.540 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.541 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:43.541 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946706+0000", 2020-06-12T02:37:43.541 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0013766410000000001 2020-06-12T02:37:43.541 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.541 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.541 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:43.542 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.948083+0000", 2020-06-12T02:37:43.542 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019206890000000001 2020-06-12T02:37:43.542 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.542 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:43.542 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.543 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.543 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.543 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:879 3.fs1 3:fc7e0bf7:::smithi10729407-17:head [append 0~491520] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:43.543 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.000881+0000", 2020-06-12T02:37:43.544 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.458937255, 2020-06-12T02:37:43.544 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017891845, 2020-06-12T02:37:43.544 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:43.544 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:43.544 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:43.545 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:43.545 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:43.545 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 879 2020-06-12T02:37:43.545 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.545 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:43.546 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.546 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:43.546 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.000881+0000", 2020-06-12T02:37:43.546 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:43.546 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.546 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.547 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:43.547 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.000881+0000", 2020-06-12T02:37:43.547 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00044151599999999998 2020-06-12T02:37:43.547 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.547 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.548 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:43.548 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001322+0000", 2020-06-12T02:37:43.548 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0210000000000001e-06 2020-06-12T02:37:43.548 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.548 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.548 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:43.549 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001326+0000", 2020-06-12T02:37:43.549 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6659999999999997e-06 2020-06-12T02:37:43.549 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.549 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.549 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:43.550 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001335+0000", 2020-06-12T02:37:43.550 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4299999999999999e-06 2020-06-12T02:37:43.550 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.550 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.550 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:43.550 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001339+0000", 2020-06-12T02:37:43.551 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0028741510000000001 2020-06-12T02:37:43.551 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.551 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.551 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:43.551 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.004214+0000", 2020-06-12T02:37:43.552 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010480999999999999 2020-06-12T02:37:43.552 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.552 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.552 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:43.552 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.004318+0000", 2020-06-12T02:37:43.552 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021112359999999998 2020-06-12T02:37:43.553 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.553 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.553 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:43.553 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006430+0000", 2020-06-12T02:37:43.553 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2935000000000001e-05 2020-06-12T02:37:43.554 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.554 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.554 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:43.554 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006483+0000", 2020-06-12T02:37:43.554 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00056411099999999995 2020-06-12T02:37:43.554 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.555 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.555 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:43.555 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.007047+0000", 2020-06-12T02:37:43.555 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0076749369999999997 2020-06-12T02:37:43.555 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.556 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.556 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:43.556 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.014722+0000", 2020-06-12T02:37:43.556 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00395693 2020-06-12T02:37:43.556 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.556 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.557 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:43.557 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.018679+0000", 2020-06-12T02:37:43.557 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4102000000000001e-05 2020-06-12T02:37:43.557 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.557 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.558 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:43.558 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.018773+0000", 2020-06-12T02:37:43.558 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017891845 2020-06-12T02:37:43.558 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.558 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:43.558 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.559 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.559 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.559 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:880 3.fs1 3:fc7e0bf7:::smithi10729407-17:head [append 0~516096] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:43.559 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.004750+0000", 2020-06-12T02:37:43.559 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.455067839, 2020-06-12T02:37:43.560 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014302860000000001, 2020-06-12T02:37:43.560 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:43.560 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:43.560 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:43.560 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:43.560 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:43.561 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 880 2020-06-12T02:37:43.561 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.561 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:43.561 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.561 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:43.562 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.004750+0000", 2020-06-12T02:37:43.562 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:43.562 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.562 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.562 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:43.562 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.004750+0000", 2020-06-12T02:37:43.563 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00048597100000000003 2020-06-12T02:37:43.563 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.563 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.563 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:43.563 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.005236+0000", 2020-06-12T02:37:43.564 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2290000000000002e-06 2020-06-12T02:37:43.564 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.564 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.564 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:43.564 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.005241+0000", 2020-06-12T02:37:43.564 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1155000000000001e-05 2020-06-12T02:37:43.565 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.565 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.565 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:43.565 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.005253+0000", 2020-06-12T02:37:43.565 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2570000000000001e-06 2020-06-12T02:37:43.566 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.566 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.566 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:43.566 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.005257+0000", 2020-06-12T02:37:43.566 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8241e-05 2020-06-12T02:37:43.566 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.567 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.567 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:43.567 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.005315+0000", 2020-06-12T02:37:43.567 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6916999999999996e-05 2020-06-12T02:37:43.567 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.568 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.568 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:43.568 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.005382+0000", 2020-06-12T02:37:43.568 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0039171969999999999 2020-06-12T02:37:43.568 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.568 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.569 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:43.569 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.009299+0000", 2020-06-12T02:37:43.569 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000102623 2020-06-12T02:37:43.569 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.569 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.569 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:43.570 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.009402+0000", 2020-06-12T02:37:43.570 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00070054899999999996 2020-06-12T02:37:43.570 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.570 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.570 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:43.571 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.010102+0000", 2020-06-12T02:37:43.571 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.005383891 2020-06-12T02:37:43.571 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.571 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.571 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:43.572 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.015486+0000", 2020-06-12T02:37:43.572 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0034234310000000002 2020-06-12T02:37:43.572 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.572 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.572 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:43.572 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.018910+0000", 2020-06-12T02:37:43.573 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000143399 2020-06-12T02:37:43.573 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.573 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.573 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:43.573 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.019053+0000", 2020-06-12T02:37:43.574 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014302860000000001 2020-06-12T02:37:43.574 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.574 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:43.574 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.574 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.574 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.575 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:883 3.fs1 3:fc7e0bf7:::smithi10729407-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:43.575 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.006322+0000", 2020-06-12T02:37:43.575 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.453496094, 2020-06-12T02:37:43.575 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01444995, 2020-06-12T02:37:43.575 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:43.576 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:43.576 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:43.576 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:43.576 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:43.576 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 883 2020-06-12T02:37:43.576 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.577 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:43.577 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.577 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:43.577 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006322+0000", 2020-06-12T02:37:43.577 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:43.578 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.578 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.578 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:43.578 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006322+0000", 2020-06-12T02:37:43.578 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1342999999999999e-05 2020-06-12T02:37:43.578 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.579 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.579 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:43.579 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006343+0000", 2020-06-12T02:37:43.579 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.021e-06 2020-06-12T02:37:43.579 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.580 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.580 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:43.580 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006346+0000", 2020-06-12T02:37:43.580 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2990000000000002e-06 2020-06-12T02:37:43.580 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.580 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.581 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:43.581 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006353+0000", 2020-06-12T02:37:43.581 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3240000000000001e-06 2020-06-12T02:37:43.581 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.581 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.582 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:43.582 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006355+0000", 2020-06-12T02:37:43.582 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0090315139999999992 2020-06-12T02:37:43.582 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.582 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.582 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:43.583 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.015386+0000", 2020-06-12T02:37:43.583 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5129000000000003e-05 2020-06-12T02:37:43.583 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.583 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.583 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:43.584 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.015482+0000", 2020-06-12T02:37:43.584 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0043414839999999996 2020-06-12T02:37:43.584 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.584 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.584 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:43.584 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.019823+0000", 2020-06-12T02:37:43.585 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6991999999999997e-05 2020-06-12T02:37:43.585 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.585 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.585 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:43.585 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.019910+0000", 2020-06-12T02:37:43.586 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00086184399999999996 2020-06-12T02:37:43.586 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.586 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.586 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:43.586 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.020772+0000", 2020-06-12T02:37:43.586 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01444995 2020-06-12T02:37:43.587 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.587 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:43.587 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.587 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.587 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.588 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:886 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~712704] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:43.588 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.017916+0000", 2020-06-12T02:37:43.588 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.441902164, 2020-06-12T02:37:43.588 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014030252999999999, 2020-06-12T02:37:43.588 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:43.588 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:43.589 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:43.589 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:43.589 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:43.589 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 886 2020-06-12T02:37:43.589 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.590 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:43.590 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.590 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:43.590 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.017916+0000", 2020-06-12T02:37:43.590 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:43.590 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.591 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.591 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:43.591 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.017916+0000", 2020-06-12T02:37:43.591 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00067910699999999997 2020-06-12T02:37:43.591 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.592 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.592 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:43.592 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.018595+0000", 2020-06-12T02:37:43.592 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.0949999999999996e-06 2020-06-12T02:37:43.592 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.592 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.593 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:43.593 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.018603+0000", 2020-06-12T02:37:43.593 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5002e-05 2020-06-12T02:37:43.593 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.593 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.594 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:43.594 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.018618+0000", 2020-06-12T02:37:43.594 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3300000000000004e-06 2020-06-12T02:37:43.594 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.594 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.594 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:43.595 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.018624+0000", 2020-06-12T02:37:43.595 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8525999999999997e-05 2020-06-12T02:37:43.595 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.595 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.595 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:43.596 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.018663+0000", 2020-06-12T02:37:43.596 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000104404 2020-06-12T02:37:43.596 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.596 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.596 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:43.596 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.018767+0000", 2020-06-12T02:37:43.597 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0055157289999999996 2020-06-12T02:37:43.597 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.597 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.597 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:43.597 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.024283+0000", 2020-06-12T02:37:43.598 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1125999999999998e-05 2020-06-12T02:37:43.598 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.598 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.598 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:43.598 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.024334+0000", 2020-06-12T02:37:43.598 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000482585 2020-06-12T02:37:43.599 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.599 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.600 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:43.600 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.024817+0000", 2020-06-12T02:37:43.600 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0047718500000000002 2020-06-12T02:37:43.600 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.600 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.601 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:43.601 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.029589+0000", 2020-06-12T02:37:43.601 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023106070000000001 2020-06-12T02:37:43.601 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.601 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.602 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:43.602 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.031899+0000", 2020-06-12T02:37:43.602 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6891999999999998e-05 2020-06-12T02:37:43.602 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.602 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.603 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:43.603 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.031946+0000", 2020-06-12T02:37:43.603 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014030252999999999 2020-06-12T02:37:43.603 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.603 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:43.603 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.604 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.604 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.604 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:889 3.3s0 3:c2ca77da:::smithi10729407-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:43.604 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.022069+0000", 2020-06-12T02:37:43.604 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.437749458000001, 2020-06-12T02:37:43.604 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01386604, 2020-06-12T02:37:43.605 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:43.605 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:43.605 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:43.605 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:43.606 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:43.606 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 889 2020-06-12T02:37:43.606 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.607 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:43.607 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.607 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:43.608 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.022069+0000", 2020-06-12T02:37:43.608 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:43.608 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.608 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.608 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:43.609 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.022069+0000", 2020-06-12T02:37:43.609 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4539000000000001e-05 2020-06-12T02:37:43.609 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.609 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.609 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:43.610 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.022123+0000", 2020-06-12T02:37:43.610 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.888e-06 2020-06-12T02:37:43.610 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.610 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.610 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:43.610 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.022130+0000", 2020-06-12T02:37:43.611 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4331999999999999e-05 2020-06-12T02:37:43.611 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.611 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.611 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:43.611 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.022144+0000", 2020-06-12T02:37:43.612 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4870000000000004e-06 2020-06-12T02:37:43.612 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.612 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.612 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:43.612 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.022149+0000", 2020-06-12T02:37:43.612 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4615999999999999e-05 2020-06-12T02:37:43.613 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.613 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.613 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:43.613 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.022173+0000", 2020-06-12T02:37:43.613 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1434999999999998e-05 2020-06-12T02:37:43.614 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.614 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.614 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:43.614 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.022225+0000", 2020-06-12T02:37:43.614 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0079399250000000005 2020-06-12T02:37:43.614 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.615 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.615 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:43.615 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.030165+0000", 2020-06-12T02:37:43.615 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8032000000000002e-05 2020-06-12T02:37:43.615 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.616 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.616 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:43.616 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.030233+0000", 2020-06-12T02:37:43.616 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004515453 2020-06-12T02:37:43.616 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.616 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.617 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:43.617 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.034748+0000", 2020-06-12T02:37:43.617 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9342000000000002e-05 2020-06-12T02:37:43.617 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.617 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.618 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:43.618 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.034808+0000", 2020-06-12T02:37:43.618 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0011269909999999999 2020-06-12T02:37:43.618 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.618 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.618 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:43.619 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.035935+0000", 2020-06-12T02:37:43.619 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01386604 2020-06-12T02:37:43.619 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.619 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:43.619 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.620 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.620 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.620 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:933 3.es1 3:78f01648:::smithi10729407-5:head [append 0~417792] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:43.620 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.156677+0000", 2020-06-12T02:37:43.620 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.303141273, 2020-06-12T02:37:43.620 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016833311, 2020-06-12T02:37:43.621 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:43.621 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:43.621 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:43.621 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:43.621 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:43.622 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 933 2020-06-12T02:37:43.622 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.622 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:43.622 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.622 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:43.622 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.156677+0000", 2020-06-12T02:37:43.623 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:43.623 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.623 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.623 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:43.623 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.156677+0000", 2020-06-12T02:37:43.624 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00062571700000000003 2020-06-12T02:37:43.624 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.624 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.624 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:43.624 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157303+0000", 2020-06-12T02:37:43.624 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0740000000000003e-06 2020-06-12T02:37:43.625 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.625 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.625 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:43.625 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157307+0000", 2020-06-12T02:37:43.625 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4419999999999998e-06 2020-06-12T02:37:43.626 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.626 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.626 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:43.626 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157315+0000", 2020-06-12T02:37:43.626 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5300000000000001e-06 2020-06-12T02:37:43.626 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.627 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.627 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:43.627 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157319+0000", 2020-06-12T02:37:43.627 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4995e-05 2020-06-12T02:37:43.627 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.628 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.628 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:43.628 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157354+0000", 2020-06-12T02:37:43.628 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5883000000000006e-05 2020-06-12T02:37:43.628 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.628 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.629 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:43.629 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157419+0000", 2020-06-12T02:37:43.629 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045316280000000002 2020-06-12T02:37:43.629 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.629 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.630 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:43.630 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.161951+0000", 2020-06-12T02:37:43.630 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3300999999999996e-05 2020-06-12T02:37:43.630 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.630 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.635 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:43.635 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.162014+0000", 2020-06-12T02:37:43.635 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00039050100000000002 2020-06-12T02:37:43.635 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.635 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.636 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:43.636 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.162405+0000", 2020-06-12T02:37:43.636 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050686250000000002 2020-06-12T02:37:43.636 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.636 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.636 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:43.637 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.167474+0000", 2020-06-12T02:37:43.637 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.005942653 2020-06-12T02:37:43.637 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.637 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.637 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:43.638 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.173416+0000", 2020-06-12T02:37:43.638 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3962000000000006e-05 2020-06-12T02:37:43.638 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.638 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.639 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:43.639 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.173510+0000", 2020-06-12T02:37:43.639 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016833311 2020-06-12T02:37:43.639 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.639 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:43.639 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.640 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.640 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.640 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:934 3.es1 3:78f01648:::smithi10729407-5:head [append 0~458752] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:43.641 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.159379+0000", 2020-06-12T02:37:43.641 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.30043862, 2020-06-12T02:37:43.641 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014294441, 2020-06-12T02:37:43.641 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:43.641 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:43.641 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:43.642 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:43.642 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:43.642 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 934 2020-06-12T02:37:43.642 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.642 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:43.643 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.643 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:43.643 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.159379+0000", 2020-06-12T02:37:43.643 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:43.644 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.644 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.644 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:43.644 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.159379+0000", 2020-06-12T02:37:43.644 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000706666 2020-06-12T02:37:43.645 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.645 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.645 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:43.645 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160086+0000", 2020-06-12T02:37:43.645 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0941000000000001e-05 2020-06-12T02:37:43.646 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.646 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.646 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:43.646 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160097+0000", 2020-06-12T02:37:43.646 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6980999999999999e-05 2020-06-12T02:37:43.647 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.647 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.647 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:43.647 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160114+0000", 2020-06-12T02:37:43.647 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5089999999999996e-06 2020-06-12T02:37:43.647 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.648 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.648 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:43.648 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160122+0000", 2020-06-12T02:37:43.648 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00389617 2020-06-12T02:37:43.648 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.649 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.649 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:43.649 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.164018+0000", 2020-06-12T02:37:43.649 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9804999999999998e-05 2020-06-12T02:37:43.649 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.650 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.650 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:43.650 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.164108+0000", 2020-06-12T02:37:43.650 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000378942 2020-06-12T02:37:43.650 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.650 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.651 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:43.651 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.164486+0000", 2020-06-12T02:37:43.651 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031318969999999998 2020-06-12T02:37:43.651 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.651 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.652 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:43.652 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.167618+0000", 2020-06-12T02:37:43.652 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0059775799999999997 2020-06-12T02:37:43.652 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.652 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.652 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:43.653 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.173596+0000", 2020-06-12T02:37:43.653 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7949999999999997e-05 2020-06-12T02:37:43.653 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.653 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.653 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:43.654 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.173674+0000", 2020-06-12T02:37:43.654 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014294441 2020-06-12T02:37:43.654 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.654 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:43.654 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.654 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.655 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.655 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:936 3.es1 3:78f01648:::smithi10729407-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:43.655 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.160357+0000", 2020-06-12T02:37:43.655 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.299461561999999, 2020-06-12T02:37:43.655 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015435590000000001, 2020-06-12T02:37:43.656 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:43.656 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:43.656 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:43.656 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:43.656 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:43.656 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 936 2020-06-12T02:37:43.657 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.657 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:43.657 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.657 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:43.657 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160357+0000", 2020-06-12T02:37:43.658 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:43.658 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.658 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.658 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:43.658 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160357+0000", 2020-06-12T02:37:43.658 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4435999999999999e-05 2020-06-12T02:37:43.659 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.659 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.659 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:43.659 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160411+0000", 2020-06-12T02:37:43.659 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2329999999999995e-06 2020-06-12T02:37:43.660 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.660 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.660 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:43.660 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160420+0000", 2020-06-12T02:37:43.660 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7003e-05 2020-06-12T02:37:43.660 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.661 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.661 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:43.661 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160437+0000", 2020-06-12T02:37:43.661 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8370000000000002e-06 2020-06-12T02:37:43.661 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.662 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.662 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:43.662 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160443+0000", 2020-06-12T02:37:43.662 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0070635489999999997 2020-06-12T02:37:43.662 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.662 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.663 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:43.663 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.167507+0000", 2020-06-12T02:37:43.663 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010589700000000001 2020-06-12T02:37:43.663 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.663 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.664 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:43.664 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.167612+0000", 2020-06-12T02:37:43.664 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0065032299999999996 2020-06-12T02:37:43.664 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.664 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.664 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:43.665 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.174116+0000", 2020-06-12T02:37:43.665 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3128999999999999e-05 2020-06-12T02:37:43.665 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.665 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.665 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:43.666 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.174169+0000", 2020-06-12T02:37:43.666 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0016232760000000001 2020-06-12T02:37:43.666 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.666 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.666 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:43.666 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.175792+0000", 2020-06-12T02:37:43.667 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015435590000000001 2020-06-12T02:37:43.667 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.667 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:43.667 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.667 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.668 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.668 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:950 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~778240] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:43.668 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.207643+0000", 2020-06-12T02:37:43.668 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.252174978999999, 2020-06-12T02:37:43.668 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014844661, 2020-06-12T02:37:43.668 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:43.669 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:43.669 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:43.669 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:43.669 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:43.669 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 950 2020-06-12T02:37:43.670 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.670 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:43.670 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.670 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:43.670 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.207643+0000", 2020-06-12T02:37:43.671 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:43.671 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.671 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.671 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:43.671 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.207643+0000", 2020-06-12T02:37:43.671 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00072843099999999998 2020-06-12T02:37:43.672 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.672 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.672 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:43.672 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208372+0000", 2020-06-12T02:37:43.672 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9120000000000001e-06 2020-06-12T02:37:43.673 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.673 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.673 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:43.673 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208378+0000", 2020-06-12T02:37:43.673 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9660000000000008e-06 2020-06-12T02:37:43.673 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.674 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.674 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:43.674 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208388+0000", 2020-06-12T02:37:43.674 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1279999999999999e-06 2020-06-12T02:37:43.674 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.675 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.675 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:43.675 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208394+0000", 2020-06-12T02:37:43.675 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0439999999999998e-05 2020-06-12T02:37:43.675 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.675 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.676 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:43.676 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208444+0000", 2020-06-12T02:37:43.676 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2611000000000001e-05 2020-06-12T02:37:43.676 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.676 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.676 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:43.677 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208507+0000", 2020-06-12T02:37:43.677 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001228301 2020-06-12T02:37:43.677 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.677 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.677 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:43.678 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.209735+0000", 2020-06-12T02:37:43.678 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8875000000000001e-05 2020-06-12T02:37:43.678 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.678 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.678 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:43.678 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.209794+0000", 2020-06-12T02:37:43.679 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053218899999999997 2020-06-12T02:37:43.679 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.679 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.679 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:43.679 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.210326+0000", 2020-06-12T02:37:43.680 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0078198439999999994 2020-06-12T02:37:43.680 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.680 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.680 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:43.680 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.218146+0000", 2020-06-12T02:37:43.680 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004300496 2020-06-12T02:37:43.681 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.681 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.681 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:43.681 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.222446+0000", 2020-06-12T02:37:43.681 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1467999999999997e-05 2020-06-12T02:37:43.682 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.682 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.682 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:43.682 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.222488+0000", 2020-06-12T02:37:43.682 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014844661 2020-06-12T02:37:43.682 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.683 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:43.683 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.683 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.683 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.683 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:953 3.3s0 3:c2ca77da:::smithi10729407-13:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:43.684 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.212848+0000", 2020-06-12T02:37:43.684 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.246969742999999, 2020-06-12T02:37:43.684 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013849036, 2020-06-12T02:37:43.684 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:43.684 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:43.684 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:43.685 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:43.685 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:43.685 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 953 2020-06-12T02:37:43.685 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.685 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:43.686 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.686 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:43.686 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212848+0000", 2020-06-12T02:37:43.686 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:43.686 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.686 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.687 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:43.687 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212848+0000", 2020-06-12T02:37:43.687 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7985000000000003e-05 2020-06-12T02:37:43.687 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.687 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.688 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:43.688 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212886+0000", 2020-06-12T02:37:43.688 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1439999999999999e-06 2020-06-12T02:37:43.688 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.688 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.688 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:43.689 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212889+0000", 2020-06-12T02:37:43.689 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4360000000000002e-06 2020-06-12T02:37:43.689 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.689 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.689 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:43.690 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212898+0000", 2020-06-12T02:37:43.690 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3489999999999999e-06 2020-06-12T02:37:43.690 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.690 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.690 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:43.690 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212900+0000", 2020-06-12T02:37:43.691 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0047082119999999998 2020-06-12T02:37:43.691 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.691 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.691 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:43.691 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.217608+0000", 2020-06-12T02:37:43.692 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3783999999999997e-05 2020-06-12T02:37:43.692 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.692 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.692 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:43.692 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.217682+0000", 2020-06-12T02:37:43.692 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00026563599999999998 2020-06-12T02:37:43.693 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.693 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.693 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:43.693 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.217948+0000", 2020-06-12T02:37:43.693 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000355399 2020-06-12T02:37:43.694 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.694 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.694 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:43.694 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.218303+0000", 2020-06-12T02:37:43.694 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0083342989999999999 2020-06-12T02:37:43.694 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.695 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.695 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:43.695 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.226638+0000", 2020-06-12T02:37:43.695 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9791999999999999e-05 2020-06-12T02:37:43.695 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.696 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.696 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:43.696 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.226697+0000", 2020-06-12T02:37:43.696 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013849036 2020-06-12T02:37:43.696 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.696 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:43.697 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.697 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.697 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.697 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:954 3.3s0 3:c2ca77da:::smithi10729407-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:43.697 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.212923+0000", 2020-06-12T02:37:43.698 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.246895202999999, 2020-06-12T02:37:43.698 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015411546999999999, 2020-06-12T02:37:43.698 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:43.698 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:43.698 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:43.698 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:43.699 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:43.699 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 954 2020-06-12T02:37:43.699 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.699 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:43.699 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.700 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:43.700 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212923+0000", 2020-06-12T02:37:43.700 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:43.700 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.700 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.700 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:43.701 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212923+0000", 2020-06-12T02:37:43.701 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3598e-05 2020-06-12T02:37:43.701 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.701 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.701 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:43.702 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212946+0000", 2020-06-12T02:37:43.702 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.061e-06 2020-06-12T02:37:43.702 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.702 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.702 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:43.702 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212950+0000", 2020-06-12T02:37:43.703 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8700000000000003e-06 2020-06-12T02:37:43.703 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.703 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.703 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:43.703 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212956+0000", 2020-06-12T02:37:43.703 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0770000000000002e-06 2020-06-12T02:37:43.704 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.704 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.704 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:43.704 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212958+0000", 2020-06-12T02:37:43.704 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051235170000000002 2020-06-12T02:37:43.705 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.705 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.705 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:43.705 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.218082+0000", 2020-06-12T02:37:43.705 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9246999999999997e-05 2020-06-12T02:37:43.705 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.706 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.706 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:43.706 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.218141+0000", 2020-06-12T02:37:43.706 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0087981910000000003 2020-06-12T02:37:43.707 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.707 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.707 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:43.707 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.226939+0000", 2020-06-12T02:37:43.707 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4119999999999997e-05 2020-06-12T02:37:43.708 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.708 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.708 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:43.708 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.226994+0000", 2020-06-12T02:37:43.708 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001340866 2020-06-12T02:37:43.709 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.709 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.709 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:43.709 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.228334+0000", 2020-06-12T02:37:43.709 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015411546999999999 2020-06-12T02:37:43.709 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.710 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:43.710 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.710 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.710 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.710 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1097 3.es1 3:78f01648:::smithi10729407-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:43.711 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.576496+0000", 2020-06-12T02:37:43.711 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 10.883321721, 2020-06-12T02:37:43.711 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014992914, 2020-06-12T02:37:43.711 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:43.711 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:43.711 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:43.712 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:43.712 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:43.712 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1097 2020-06-12T02:37:43.712 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.712 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:43.712 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.713 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:43.713 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576496+0000", 2020-06-12T02:37:43.713 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:43.713 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.713 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.714 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:43.714 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576496+0000", 2020-06-12T02:37:43.714 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3221999999999999e-05 2020-06-12T02:37:43.714 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.714 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.714 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:43.715 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576520+0000", 2020-06-12T02:37:43.715 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.988e-06 2020-06-12T02:37:43.715 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.715 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.715 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:43.716 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576523+0000", 2020-06-12T02:37:43.716 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2940000000000002e-06 2020-06-12T02:37:43.716 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.716 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.716 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:43.716 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576529+0000", 2020-06-12T02:37:43.717 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6060000000000002e-06 2020-06-12T02:37:43.717 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.717 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.717 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:43.717 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576534+0000", 2020-06-12T02:37:43.718 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001254554 2020-06-12T02:37:43.718 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.718 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.718 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:43.718 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.577789+0000", 2020-06-12T02:37:43.718 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3177999999999998e-05 2020-06-12T02:37:43.719 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.719 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.719 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:43.719 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.577842+0000", 2020-06-12T02:37:43.719 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0063091220000000003 2020-06-12T02:37:43.720 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.720 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.720 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:43.720 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.584151+0000", 2020-06-12T02:37:43.720 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.783e-05 2020-06-12T02:37:43.720 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.721 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.721 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:43.721 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.584209+0000", 2020-06-12T02:37:43.721 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045863650000000002 2020-06-12T02:37:43.721 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.722 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.722 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:43.722 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.588796+0000", 2020-06-12T02:37:43.722 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7614000000000006e-05 2020-06-12T02:37:43.722 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.722 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.723 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:43.723 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.588873+0000", 2020-06-12T02:37:43.723 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0026161410000000002 2020-06-12T02:37:43.723 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.723 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.724 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:43.724 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.591489+0000", 2020-06-12T02:37:43.724 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014992914 2020-06-12T02:37:43.724 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.724 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:43.724 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.725 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.725 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.725 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1207 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~630784] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:43.725 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.859578+0000", 2020-06-12T02:37:43.725 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 10.600240185000001, 2020-06-12T02:37:43.726 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014710433, 2020-06-12T02:37:43.726 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:43.726 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:43.726 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:43.726 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:43.726 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:43.727 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1207 2020-06-12T02:37:43.727 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.727 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:43.727 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.727 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:43.728 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.859578+0000", 2020-06-12T02:37:43.728 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:43.728 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.728 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.728 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:43.728 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.859578+0000", 2020-06-12T02:37:43.729 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000775511 2020-06-12T02:37:43.729 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.729 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.729 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:43.729 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.860353+0000", 2020-06-12T02:37:43.729 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5325999999999999e-05 2020-06-12T02:37:43.730 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.730 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.730 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:43.730 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.860369+0000", 2020-06-12T02:37:43.730 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7430000000000001e-05 2020-06-12T02:37:43.731 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.731 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.731 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:43.731 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.860386+0000", 2020-06-12T02:37:43.731 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.426e-06 2020-06-12T02:37:43.731 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.732 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.732 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:43.732 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.860393+0000", 2020-06-12T02:37:43.732 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3585999999999999e-05 2020-06-12T02:37:43.732 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.733 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.733 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:43.733 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.860436+0000", 2020-06-12T02:37:43.733 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2559999999999996e-05 2020-06-12T02:37:43.733 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.733 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.734 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:43.734 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.860509+0000", 2020-06-12T02:37:43.734 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00063177700000000003 2020-06-12T02:37:43.734 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.734 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.735 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:43.735 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.861141+0000", 2020-06-12T02:37:43.735 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0026307570000000001 2020-06-12T02:37:43.735 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.735 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.735 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:43.736 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.863771+0000", 2020-06-12T02:37:43.736 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010445545000000001 2020-06-12T02:37:43.736 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.736 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.736 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:43.736 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.874217+0000", 2020-06-12T02:37:43.737 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1514999999999995e-05 2020-06-12T02:37:43.737 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.737 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.737 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:43.737 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.874288+0000", 2020-06-12T02:37:43.738 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014710433 2020-06-12T02:37:43.738 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.738 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:43.738 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.738 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.738 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.739 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2788 3.fs1 3:fc7e0bf7:::smithi10729407-17:head [append 0~679936] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:43.739 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:36.820585+0000", 2020-06-12T02:37:43.739 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 6.6392331220000003, 2020-06-12T02:37:43.739 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01440234, 2020-06-12T02:37:43.739 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:43.740 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:43.740 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:43.740 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:43.740 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:43.740 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2788 2020-06-12T02:37:43.741 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.741 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:43.741 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.741 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:43.742 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.820585+0000", 2020-06-12T02:37:43.742 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:43.742 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.742 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.742 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:43.742 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.820585+0000", 2020-06-12T02:37:43.743 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001119386 2020-06-12T02:37:43.743 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.743 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.743 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:43.743 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.821704+0000", 2020-06-12T02:37:43.744 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2069000000000001e-05 2020-06-12T02:37:43.744 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.744 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.744 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:43.745 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.821716+0000", 2020-06-12T02:37:43.745 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2909e-05 2020-06-12T02:37:43.745 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.745 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.745 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:43.746 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.821729+0000", 2020-06-12T02:37:43.746 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4559999999999997e-06 2020-06-12T02:37:43.746 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.746 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.746 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:43.747 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.821736+0000", 2020-06-12T02:37:43.747 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6715000000000003e-05 2020-06-12T02:37:43.747 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.747 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.747 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:43.747 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.821772+0000", 2020-06-12T02:37:43.748 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7078999999999998e-05 2020-06-12T02:37:43.748 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.748 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.748 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:43.748 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.821850+0000", 2020-06-12T02:37:43.749 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00049280600000000004 2020-06-12T02:37:43.749 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.749 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.749 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:43.749 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.822342+0000", 2020-06-12T02:37:43.749 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051665629999999999 2020-06-12T02:37:43.750 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.750 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.750 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:43.750 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.827509+0000", 2020-06-12T02:37:43.750 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0073895280000000002 2020-06-12T02:37:43.751 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.751 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.751 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:43.751 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.834898+0000", 2020-06-12T02:37:43.751 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8828999999999999e-05 2020-06-12T02:37:43.751 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.752 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.752 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:43.752 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.834987+0000", 2020-06-12T02:37:43.752 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01440234 2020-06-12T02:37:43.752 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.752 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:43.753 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.753 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.753 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.753 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:4813 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~450560] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-06-12T02:37:43.753 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:43.343096+0000", 2020-06-12T02:37:43.754 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.116722378, 2020-06-12T02:37:43.754 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014820242000000001, 2020-06-12T02:37:43.754 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:43.754 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:43.754 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:43.755 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:43.755 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:43.755 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 4813 2020-06-12T02:37:43.755 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.755 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:43.755 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.756 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:43.756 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343096+0000", 2020-06-12T02:37:43.756 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:43.756 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.756 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.756 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:43.757 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343096+0000", 2020-06-12T02:37:43.757 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00068963 2020-06-12T02:37:43.757 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.757 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.757 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:43.758 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343785+0000", 2020-06-12T02:37:43.758 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1479999999999997e-06 2020-06-12T02:37:43.758 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.758 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.758 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:43.758 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343790+0000", 2020-06-12T02:37:43.759 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4060000000000005e-06 2020-06-12T02:37:43.759 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.759 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.759 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:43.759 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343799+0000", 2020-06-12T02:37:43.760 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0970000000000002e-06 2020-06-12T02:37:43.760 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.760 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.760 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:43.760 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343805+0000", 2020-06-12T02:37:43.760 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00023736499999999999 2020-06-12T02:37:43.761 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.761 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.761 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:43.761 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.344042+0000", 2020-06-12T02:37:43.761 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000113866 2020-06-12T02:37:43.762 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.762 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.762 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:43.762 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.344156+0000", 2020-06-12T02:37:43.762 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036114760000000002 2020-06-12T02:37:43.762 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.763 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.763 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:43.763 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.347768+0000", 2020-06-12T02:37:43.763 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1842999999999996e-05 2020-06-12T02:37:43.763 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.763 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.764 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:43.764 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.347850+0000", 2020-06-12T02:37:43.764 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00048835999999999997 2020-06-12T02:37:43.764 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.764 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.765 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:43.765 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.348338+0000", 2020-06-12T02:37:43.765 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0049373139999999999 2020-06-12T02:37:43.765 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.765 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.765 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:43.766 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.353275+0000", 2020-06-12T02:37:43.766 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045727119999999996 2020-06-12T02:37:43.766 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.766 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.766 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:43.767 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.357848+0000", 2020-06-12T02:37:43.767 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8025e-05 2020-06-12T02:37:43.767 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:43.767 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:43.767 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:43.767 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.357916+0000", 2020-06-12T02:37:43.768 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014820242000000001 2020-06-12T02:37:43.768 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.768 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:43.768 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.768 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:43.768 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:43.769 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:37:43.782 INFO:tasks.rados.rados.0.smithi107.stdout:2296: done (1 left) 2020-06-12T02:37:43.783 INFO:tasks.rados.rados.0.smithi107.stdout:2299: done (0 left) 2020-06-12T02:37:43.783 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:37:43.783 INFO:tasks.rados.rados.0.smithi107.stdout:2300: seq_num 2299 ranges {2179072=106496} 2020-06-12T02:37:43.783 INFO:tasks.rados.rados.0.smithi107.stdout:2300: writing smithi10729407-13 from 2179072 to 2285568 tid 1 2020-06-12T02:37:43.783 INFO:tasks.rados.rados.0.smithi107.stdout:2301: delete oid 12 current snap is 0 2020-06-12T02:37:43.784 INFO:tasks.rados.rados.0.smithi107.stdout:2301: done (1 left) 2020-06-12T02:37:43.784 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:37:43.784 INFO:tasks.rados.rados.0.smithi107.stdout:2302: seq_num 2301 ranges {1130496=139264} 2020-06-12T02:37:43.784 INFO:tasks.rados.rados.0.smithi107.stdout:2302: writing smithi10729407-20 from 1130496 to 1269760 tid 1 2020-06-12T02:37:43.784 INFO:tasks.rados.rados.0.smithi107.stdout:2303: delete oid 19 current snap is 0 2020-06-12T02:37:43.784 INFO:tasks.rados.rados.0.smithi107.stdout:2300: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:43.785 INFO:tasks.rados.rados.0.smithi107.stdout:2300: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:43.785 INFO:tasks.rados.rados.0.smithi107.stdout:2300: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:43.785 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 435 (ObjNum 2299 snap 0 seq_num 2299) dirty exists 2020-06-12T02:37:43.785 INFO:tasks.rados.rados.0.smithi107.stdout:2300: left oid 13 (ObjNum 2299 snap 0 seq_num 2299) 2020-06-12T02:37:43.785 INFO:tasks.rados.rados.0.smithi107.stdout:2300: done (2 left) 2020-06-12T02:37:43.786 INFO:tasks.rados.rados.0.smithi107.stdout:2303: done (1 left) 2020-06-12T02:37:43.786 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:37:43.786 INFO:tasks.rados.rados.0.smithi107.stdout:2304: seq_num 2303 ranges {0=172032} 2020-06-12T02:37:43.786 INFO:tasks.rados.rados.0.smithi107.stdout:2304: writing smithi10729407-19 from 0 to 172032 tid 1 2020-06-12T02:37:43.786 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:37:43.787 INFO:tasks.rados.rados.0.smithi107.stdout:2305: seq_num 2304 ranges {0=188416} 2020-06-12T02:37:43.787 INFO:tasks.rados.rados.0.smithi107.stdout:2305: writing smithi10729407-7 from 0 to 188416 tid 1 2020-06-12T02:37:43.787 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:43.787 INFO:tasks.rados.rados.0.smithi107.stdout:2306: seq_num 2305 ranges {0=212992} 2020-06-12T02:37:43.787 INFO:tasks.rados.rados.0.smithi107.stdout:2306: writing smithi10729407-2 from 0 to 212992 tid 1 2020-06-12T02:37:43.787 INFO:tasks.rados.rados.0.smithi107.stdout:2307: delete oid 5 current snap is 0 2020-06-12T02:37:43.788 INFO:tasks.rados.rados.0.smithi107.stdout:2302: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:43.788 INFO:tasks.rados.rados.0.smithi107.stdout:2302: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:43.788 INFO:tasks.rados.rados.0.smithi107.stdout:2302: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:43.788 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 145 (ObjNum 2301 snap 0 seq_num 2301) dirty exists 2020-06-12T02:37:43.788 INFO:tasks.rados.rados.0.smithi107.stdout:2302: left oid 20 (ObjNum 2301 snap 0 seq_num 2301) 2020-06-12T02:37:43.789 INFO:tasks.rados.rados.0.smithi107.stdout:2304: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:43.789 INFO:tasks.rados.rados.0.smithi107.stdout:2302: done (4 left) 2020-06-12T02:37:43.789 INFO:tasks.rados.rados.0.smithi107.stdout:2307: done (3 left) 2020-06-12T02:37:43.789 INFO:tasks.rados.rados.0.smithi107.stdout:2308: delete oid 5 current snap is 0 2020-06-12T02:37:43.789 INFO:tasks.rados.rados.0.smithi107.stdout:2305: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:43.789 INFO:tasks.rados.rados.0.smithi107.stdout:2304: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:43.790 INFO:tasks.rados.rados.0.smithi107.stdout:2308: done (3 left) 2020-06-12T02:37:43.790 INFO:tasks.rados.rados.0.smithi107.stdout:2309: delete oid 9 current snap is 0 2020-06-12T02:37:43.790 INFO:tasks.rados.rados.0.smithi107.stdout:2306: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:43.790 INFO:tasks.rados.rados.0.smithi107.stdout:2304: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:43.790 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 356 (ObjNum 2303 snap 0 seq_num 2303) dirty exists 2020-06-12T02:37:43.791 INFO:tasks.rados.rados.0.smithi107.stdout:2304: left oid 19 (ObjNum 2303 snap 0 seq_num 2303) 2020-06-12T02:37:43.791 INFO:tasks.rados.rados.0.smithi107.stdout:2305: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:43.791 INFO:tasks.rados.rados.0.smithi107.stdout:2305: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:43.791 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 437 (ObjNum 2304 snap 0 seq_num 2304) dirty exists 2020-06-12T02:37:43.791 INFO:tasks.rados.rados.0.smithi107.stdout:2305: left oid 7 (ObjNum 2304 snap 0 seq_num 2304) 2020-06-12T02:37:43.791 INFO:tasks.rados.rados.0.smithi107.stdout:2304: done (3 left) 2020-06-12T02:37:43.792 INFO:tasks.rados.rados.0.smithi107.stdout:2305: done (2 left) 2020-06-12T02:37:43.792 INFO:tasks.rados.rados.0.smithi107.stdout:2309: done (1 left) 2020-06-12T02:37:43.792 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:37:43.792 INFO:tasks.rados.rados.0.smithi107.stdout:2310: seq_num 2309 ranges {0=278528} 2020-06-12T02:37:43.792 INFO:tasks.rados.rados.0.smithi107.stdout:2310: writing smithi10729407-15 from 0 to 278528 tid 1 2020-06-12T02:37:43.793 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:37:43.793 INFO:tasks.rados.rados.0.smithi107.stdout:2311: seq_num 2310 ranges {0=294912} 2020-06-12T02:37:43.793 INFO:tasks.rados.rados.0.smithi107.stdout:2311: writing smithi10729407-5 from 0 to 294912 tid 1 2020-06-12T02:37:43.793 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:37:43.793 INFO:tasks.rados.rados.0.smithi107.stdout:2312: seq_num 2311 ranges {2285568=311296} 2020-06-12T02:37:43.793 INFO:tasks.rados.rados.0.smithi107.stdout:2312: writing smithi10729407-13 from 2285568 to 2596864 tid 1 2020-06-12T02:37:43.794 INFO:tasks.rados.rados.0.smithi107.stdout:2313: delete oid 16 current snap is 0 2020-06-12T02:37:43.794 INFO:tasks.rados.rados.0.smithi107.stdout:2306: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:43.794 INFO:tasks.rados.rados.0.smithi107.stdout:2310: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:43.794 INFO:tasks.rados.rados.0.smithi107.stdout:2306: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:43.794 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 249 (ObjNum 2305 snap 0 seq_num 2305) dirty exists 2020-06-12T02:37:43.795 INFO:tasks.rados.rados.0.smithi107.stdout:2306: left oid 2 (ObjNum 2305 snap 0 seq_num 2305) 2020-06-12T02:37:43.795 INFO:tasks.rados.rados.0.smithi107.stdout:2310: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:43.795 INFO:tasks.rados.rados.0.smithi107.stdout:2310: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:43.795 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 387 (ObjNum 2309 snap 0 seq_num 2309) dirty exists 2020-06-12T02:37:43.795 INFO:tasks.rados.rados.0.smithi107.stdout:2310: left oid 15 (ObjNum 2309 snap 0 seq_num 2309) 2020-06-12T02:37:43.795 INFO:tasks.rados.rados.0.smithi107.stdout:2311: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:43.796 INFO:tasks.rados.rados.0.smithi107.stdout:2311: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:43.796 INFO:tasks.rados.rados.0.smithi107.stdout:2311: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:43.796 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 190 (ObjNum 2310 snap 0 seq_num 2310) dirty exists 2020-06-12T02:37:43.796 INFO:tasks.rados.rados.0.smithi107.stdout:2311: left oid 5 (ObjNum 2310 snap 0 seq_num 2310) 2020-06-12T02:37:43.796 INFO:tasks.rados.rados.0.smithi107.stdout:2306: done (4 left) 2020-06-12T02:37:43.797 INFO:tasks.rados.rados.0.smithi107.stdout:2310: done (3 left) 2020-06-12T02:37:43.797 INFO:tasks.rados.rados.0.smithi107.stdout:2311: done (2 left) 2020-06-12T02:37:43.797 INFO:tasks.rados.rados.0.smithi107.stdout:2313: done (1 left) 2020-06-12T02:37:43.797 INFO:tasks.rados.rados.0.smithi107.stdout:2314: delete oid 14 current snap is 0 2020-06-12T02:37:43.797 INFO:tasks.rados.rados.0.smithi107.stdout:2312: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:43.797 INFO:tasks.rados.rados.0.smithi107.stdout:2312: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:43.798 INFO:tasks.rados.rados.0.smithi107.stdout:2312: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:43.798 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 439 (ObjNum 2311 snap 0 seq_num 2311) dirty exists 2020-06-12T02:37:43.798 INFO:tasks.rados.rados.0.smithi107.stdout:2312: left oid 13 (ObjNum 2311 snap 0 seq_num 2311) 2020-06-12T02:37:43.798 INFO:tasks.rados.rados.0.smithi107.stdout:2312: done (1 left) 2020-06-12T02:37:43.798 INFO:tasks.rados.rados.0.smithi107.stdout:2314: done (0 left) 2020-06-12T02:37:43.799 INFO:tasks.rados.rados.0.smithi107.stdout:2315: delete oid 12 current snap is 0 2020-06-12T02:37:43.799 INFO:tasks.rados.rados.0.smithi107.stdout:2315: done (0 left) 2020-06-12T02:37:43.799 INFO:tasks.rados.rados.0.smithi107.stdout:2316: delete oid 1 current snap is 0 2020-06-12T02:37:43.799 INFO:tasks.rados.rados.0.smithi107.stdout:2316: done (0 left) 2020-06-12T02:37:43.799 INFO:tasks.rados.rados.0.smithi107.stdout:2317: delete oid 4 current snap is 0 2020-06-12T02:37:43.799 INFO:tasks.rados.rados.0.smithi107.stdout:2317: done (0 left) 2020-06-12T02:37:43.800 INFO:tasks.rados.rados.0.smithi107.stdout:2318: delete oid 20 current snap is 0 2020-06-12T02:37:43.800 INFO:tasks.rados.rados.0.smithi107.stdout:2318: done (0 left) 2020-06-12T02:37:43.800 INFO:tasks.rados.rados.0.smithi107.stdout:2319: delete oid 8 current snap is 0 2020-06-12T02:37:43.800 INFO:tasks.rados.rados.0.smithi107.stdout:2319: done (0 left) 2020-06-12T02:37:43.800 INFO:tasks.rados.rados.0.smithi107.stdout:2320: delete oid 3 current snap is 0 2020-06-12T02:37:43.801 INFO:tasks.rados.rados.0.smithi107.stdout:2320: done (0 left) 2020-06-12T02:37:43.801 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:37:43.801 INFO:tasks.rados.rados.0.smithi107.stdout:2321: seq_num 2320 ranges {0=458752} 2020-06-12T02:37:43.801 INFO:tasks.rados.rados.0.smithi107.stdout:2321: writing smithi10729407-1 from 0 to 458752 tid 1 2020-06-12T02:37:43.801 INFO:tasks.rados.rados.0.smithi107.stdout:2322: delete oid 19 current snap is 0 2020-06-12T02:37:43.801 INFO:tasks.rados.rados.0.smithi107.stdout:2322: done (1 left) 2020-06-12T02:37:43.802 INFO:tasks.rados.rados.0.smithi107.stdout:2323: delete oid 16 current snap is 0 2020-06-12T02:37:43.802 INFO:tasks.rados.rados.0.smithi107.stdout:2323: done (1 left) 2020-06-12T02:37:43.802 INFO:tasks.rados.rados.0.smithi107.stdout:2324: delete oid 12 current snap is 0 2020-06-12T02:37:43.802 INFO:tasks.rados.rados.0.smithi107.stdout:2321: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:43.802 INFO:tasks.rados.rados.0.smithi107.stdout:2324: done (1 left) 2020-06-12T02:37:43.803 INFO:tasks.rados.rados.0.smithi107.stdout:2325: delete oid 6 current snap is 0 2020-06-12T02:37:43.803 INFO:tasks.rados.rados.0.smithi107.stdout:2321: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:43.803 INFO:tasks.rados.rados.0.smithi107.stdout:2321: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:43.803 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 480 (ObjNum 2320 snap 0 seq_num 2320) dirty exists 2020-06-12T02:37:43.803 INFO:tasks.rados.rados.0.smithi107.stdout:2321: left oid 1 (ObjNum 2320 snap 0 seq_num 2320) 2020-06-12T02:37:43.803 INFO:tasks.rados.rados.0.smithi107.stdout:2321: done (1 left) 2020-06-12T02:37:43.804 INFO:tasks.rados.rados.0.smithi107.stdout:2325: done (0 left) 2020-06-12T02:37:43.804 INFO:tasks.rados.rados.0.smithi107.stdout:2326: delete oid 18 current snap is 0 2020-06-12T02:37:43.804 INFO:tasks.rados.rados.0.smithi107.stdout:2326: done (0 left) 2020-06-12T02:37:43.804 INFO:tasks.rados.rados.0.smithi107.stdout:2327: delete oid 14 current snap is 0 2020-06-12T02:37:43.804 INFO:tasks.rados.rados.0.smithi107.stdout:2327: done (0 left) 2020-06-12T02:37:43.805 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:37:43.805 INFO:tasks.rados.rados.0.smithi107.stdout:2328: seq_num 2327 ranges {0=581632} 2020-06-12T02:37:43.805 INFO:tasks.rados.rados.0.smithi107.stdout:2328: writing smithi10729407-4 from 0 to 581632 tid 1 2020-06-12T02:37:43.805 INFO:tasks.rados.rados.0.smithi107.stdout:2329: delete oid 9 current snap is 0 2020-06-12T02:37:43.805 INFO:tasks.rados.rados.0.smithi107.stdout:2329: done (1 left) 2020-06-12T02:37:43.805 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:37:43.806 INFO:tasks.rados.rados.0.smithi107.stdout:2330: seq_num 2329 ranges {188416=614400} 2020-06-12T02:37:43.806 INFO:tasks.rados.rados.0.smithi107.stdout:2330: writing smithi10729407-7 from 188416 to 802816 tid 1 2020-06-12T02:37:43.806 INFO:tasks.rados.rados.0.smithi107.stdout:2331: delete oid 2 current snap is 0 2020-06-12T02:37:43.806 INFO:tasks.rados.rados.0.smithi107.stdout:2328: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:43.806 INFO:tasks.rados.rados.0.smithi107.stdout:2328: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:43.807 INFO:tasks.rados.rados.0.smithi107.stdout:2328: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:43.807 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 448 (ObjNum 2327 snap 0 seq_num 2327) dirty exists 2020-06-12T02:37:43.807 INFO:tasks.rados.rados.0.smithi107.stdout:2328: left oid 4 (ObjNum 2327 snap 0 seq_num 2327) 2020-06-12T02:37:43.807 INFO:tasks.rados.rados.0.smithi107.stdout:2328: done (2 left) 2020-06-12T02:37:43.807 INFO:tasks.rados.rados.0.smithi107.stdout:2331: done (1 left) 2020-06-12T02:37:43.808 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:37:43.808 INFO:tasks.rados.rados.0.smithi107.stdout:2332: seq_num 2331 ranges {2596864=647168} 2020-06-12T02:37:43.808 INFO:tasks.rados.rados.0.smithi107.stdout:2332: writing smithi10729407-13 from 2596864 to 3244032 tid 1 2020-06-12T02:37:43.808 INFO:tasks.rados.rados.0.smithi107.stdout:2333: delete oid 6 current snap is 0 2020-06-12T02:37:43.809 INFO:tasks.rados.rados.0.smithi107.stdout:2330: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:43.809 INFO:tasks.rados.rados.0.smithi107.stdout:2330: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:43.809 INFO:tasks.rados.rados.0.smithi107.stdout:2330: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:43.809 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 441 (ObjNum 2329 snap 0 seq_num 2329) dirty exists 2020-06-12T02:37:43.809 INFO:tasks.rados.rados.0.smithi107.stdout:2330: left oid 7 (ObjNum 2329 snap 0 seq_num 2329) 2020-06-12T02:37:43.810 INFO:tasks.rados.rados.0.smithi107.stdout:2330: done (2 left) 2020-06-12T02:37:43.810 INFO:tasks.rados.rados.0.smithi107.stdout:2333: done (1 left) 2020-06-12T02:37:43.810 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:37:43.810 INFO:tasks.rados.rados.0.smithi107.stdout:2334: seq_num 2333 ranges {0=679936} 2020-06-12T02:37:43.810 INFO:tasks.rados.rados.0.smithi107.stdout:2334: writing smithi10729407-16 from 0 to 679936 tid 1 2020-06-12T02:37:43.810 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:37:43.811 INFO:tasks.rados.rados.0.smithi107.stdout:2335: seq_num 2334 ranges {0=696320} 2020-06-12T02:37:43.811 INFO:tasks.rados.rados.0.smithi107.stdout:2335: writing smithi10729407-19 from 0 to 696320 tid 1 2020-06-12T02:37:43.811 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:37:43.811 INFO:tasks.rados.rados.0.smithi107.stdout:2336: seq_num 2335 ranges {802816=712704} 2020-06-12T02:37:43.811 INFO:tasks.rados.rados.0.smithi107.stdout:2336: writing smithi10729407-7 from 802816 to 1515520 tid 1 2020-06-12T02:37:43.811 INFO:tasks.rados.rados.0.smithi107.stdout:2337: delete oid 1 current snap is 0 2020-06-12T02:37:43.812 INFO:tasks.rados.rados.0.smithi107.stdout:2332: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:43.812 INFO:tasks.rados.rados.0.smithi107.stdout:2332: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:43.812 INFO:tasks.rados.rados.0.smithi107.stdout:2334: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:43.812 INFO:tasks.rados.rados.0.smithi107.stdout:2332: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:43.812 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 443 (ObjNum 2331 snap 0 seq_num 2331) dirty exists 2020-06-12T02:37:43.813 INFO:tasks.rados.rados.0.smithi107.stdout:2332: left oid 13 (ObjNum 2331 snap 0 seq_num 2331) 2020-06-12T02:37:43.813 INFO:tasks.rados.rados.0.smithi107.stdout:2334: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:43.813 INFO:tasks.rados.rados.0.smithi107.stdout:2335: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:43.813 INFO:tasks.rados.rados.0.smithi107.stdout:2334: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:43.813 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 402 (ObjNum 2333 snap 0 seq_num 2333) dirty exists 2020-06-12T02:37:43.813 INFO:tasks.rados.rados.0.smithi107.stdout:2334: left oid 16 (ObjNum 2333 snap 0 seq_num 2333) 2020-06-12T02:37:43.814 INFO:tasks.rados.rados.0.smithi107.stdout:2332: done (4 left) 2020-06-12T02:37:43.814 INFO:tasks.rados.rados.0.smithi107.stdout:2334: done (3 left) 2020-06-12T02:37:43.814 INFO:tasks.rados.rados.0.smithi107.stdout:2337: done (2 left) 2020-06-12T02:37:43.814 INFO:tasks.rados.rados.0.smithi107.stdout:2338: delete oid 20 current snap is 0 2020-06-12T02:37:43.814 INFO:tasks.rados.rados.0.smithi107.stdout:2335: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:43.815 INFO:tasks.rados.rados.0.smithi107.stdout:2336: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:43.815 INFO:tasks.rados.rados.0.smithi107.stdout:2335: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:43.815 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 361 (ObjNum 2334 snap 0 seq_num 2334) dirty exists 2020-06-12T02:37:43.815 INFO:tasks.rados.rados.0.smithi107.stdout:2335: left oid 19 (ObjNum 2334 snap 0 seq_num 2334) 2020-06-12T02:37:43.815 INFO:tasks.rados.rados.0.smithi107.stdout:2335: done (2 left) 2020-06-12T02:37:43.815 INFO:tasks.rados.rados.0.smithi107.stdout:2338: done (1 left) 2020-06-12T02:37:43.816 INFO:tasks.rados.rados.0.smithi107.stdout:2339: delete oid 14 current snap is 0 2020-06-12T02:37:43.816 INFO:tasks.rados.rados.0.smithi107.stdout:2336: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:43.816 INFO:tasks.rados.rados.0.smithi107.stdout:2339: done (1 left) 2020-06-12T02:37:43.816 INFO:tasks.rados.rados.0.smithi107.stdout:2340: delete oid 12 current snap is 0 2020-06-12T02:37:43.816 INFO:tasks.rados.rados.0.smithi107.stdout:2336: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:43.817 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 445 (ObjNum 2335 snap 0 seq_num 2335) dirty exists 2020-06-12T02:37:43.817 INFO:tasks.rados.rados.0.smithi107.stdout:2336: left oid 7 (ObjNum 2335 snap 0 seq_num 2335) 2020-06-12T02:37:43.817 INFO:tasks.rados.rados.0.smithi107.stdout:2336: done (1 left) 2020-06-12T02:37:43.817 INFO:tasks.rados.rados.0.smithi107.stdout:2340: done (0 left) 2020-06-12T02:37:43.817 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:37:43.817 INFO:tasks.rados.rados.0.smithi107.stdout:2341: seq_num 2340 ranges {3129344=794624} 2020-06-12T02:37:43.818 INFO:tasks.rados.rados.0.smithi107.stdout:2341: writing smithi10729407-11 from 3129344 to 3923968 tid 1 2020-06-12T02:37:43.818 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:37:43.818 INFO:tasks.rados.rados.0.smithi107.stdout:2342: seq_num 2341 ranges {0=409600,409600=401408} 2020-06-12T02:37:43.818 INFO:tasks.rados.rados.0.smithi107.stdout:2342: writing smithi10729407-6 from 0 to 409600 tid 1 2020-06-12T02:37:43.818 INFO:tasks.rados.rados.0.smithi107.stdout:2342: writing smithi10729407-6 from 409600 to 811008 tid 2 2020-06-12T02:37:43.819 INFO:tasks.rados.rados.0.smithi107.stdout:2343: delete oid 5 current snap is 0 2020-06-12T02:37:43.819 INFO:tasks.rados.rados.0.smithi107.stdout:2343: done (2 left) 2020-06-12T02:37:43.819 INFO:tasks.rados.rados.0.smithi107.stdout:2344: delete oid 20 current snap is 0 2020-06-12T02:37:43.819 INFO:tasks.rados.rados.0.smithi107.stdout:2344: done (2 left) 2020-06-12T02:37:43.819 INFO:tasks.rados.rados.0.smithi107.stdout:2345: delete oid 15 current snap is 0 2020-06-12T02:37:43.820 INFO:tasks.rados.rados.0.smithi107.stdout:2341: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:43.820 INFO:tasks.rados.rados.0.smithi107.stdout:2345: done (2 left) 2020-06-12T02:37:43.820 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:37:43.820 INFO:tasks.rados.rados.0.smithi107.stdout:2346: seq_num 2345 ranges {0=483328,483328=401408} 2020-06-12T02:37:43.820 INFO:tasks.rados.rados.0.smithi107.stdout:2346: writing smithi10729407-2 from 0 to 483328 tid 1 2020-06-12T02:37:43.820 INFO:tasks.rados.rados.0.smithi107.stdout:2346: writing smithi10729407-2 from 483328 to 884736 tid 2 2020-06-12T02:37:43.821 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:37:43.821 INFO:tasks.rados.rados.0.smithi107.stdout:2347: seq_num 2346 ranges {0=499712,499712=401408} 2020-06-12T02:37:43.821 INFO:tasks.rados.rados.0.smithi107.stdout:2347: writing smithi10729407-9 from 0 to 499712 tid 1 2020-06-12T02:37:43.821 INFO:tasks.rados.rados.0.smithi107.stdout:2347: writing smithi10729407-9 from 499712 to 901120 tid 2 2020-06-12T02:37:43.821 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:37:43.822 INFO:tasks.rados.rados.0.smithi107.stdout:2348: seq_num 2347 ranges {679936=516096,1196032=401408} 2020-06-12T02:37:43.822 INFO:tasks.rados.rados.0.smithi107.stdout:2348: writing smithi10729407-16 from 679936 to 1196032 tid 1 2020-06-12T02:37:43.822 INFO:tasks.rados.rados.0.smithi107.stdout:2348: writing smithi10729407-16 from 1196032 to 1597440 tid 2 2020-06-12T02:37:43.822 INFO:tasks.rados.rados.0.smithi107.stdout:2349: delete oid 12 current snap is 0 2020-06-12T02:37:43.822 INFO:tasks.rados.rados.0.smithi107.stdout:2341: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:43.822 INFO:tasks.rados.rados.0.smithi107.stdout:2342: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:43.823 INFO:tasks.rados.rados.0.smithi107.stdout:2342: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:43.823 INFO:tasks.rados.rados.0.smithi107.stdout:2341: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:43.823 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 428 (ObjNum 2340 snap 0 seq_num 2340) dirty exists 2020-06-12T02:37:43.823 INFO:tasks.rados.rados.0.smithi107.stdout:2341: left oid 11 (ObjNum 2340 snap 0 seq_num 2340) 2020-06-12T02:37:43.823 INFO:tasks.rados.rados.0.smithi107.stdout:2342: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:43.824 INFO:tasks.rados.rados.0.smithi107.stdout:2342: finishing write tid 5 to smithi10729407-6 2020-06-12T02:37:43.824 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 486 (ObjNum 2341 snap 0 seq_num 2341) dirty exists 2020-06-12T02:37:43.824 INFO:tasks.rados.rados.0.smithi107.stdout:2342: left oid 6 (ObjNum 2341 snap 0 seq_num 2341) 2020-06-12T02:37:43.824 INFO:tasks.rados.rados.0.smithi107.stdout:2346: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:43.824 INFO:tasks.rados.rados.0.smithi107.stdout:2346: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:43.824 INFO:tasks.rados.rados.0.smithi107.stdout:2346: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:43.825 INFO:tasks.rados.rados.0.smithi107.stdout:2346: finishing write tid 5 to smithi10729407-2 2020-06-12T02:37:43.825 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 253 (ObjNum 2345 snap 0 seq_num 2345) dirty exists 2020-06-12T02:37:43.825 INFO:tasks.rados.rados.0.smithi107.stdout:2346: left oid 2 (ObjNum 2345 snap 0 seq_num 2345) 2020-06-12T02:37:43.825 INFO:tasks.rados.rados.0.smithi107.stdout:2347: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:43.825 INFO:tasks.rados.rados.0.smithi107.stdout:2347: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:43.826 INFO:tasks.rados.rados.0.smithi107.stdout:2347: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:43.826 INFO:tasks.rados.rados.0.smithi107.stdout:2347: finishing write tid 5 to smithi10729407-9 2020-06-12T02:37:43.826 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 403 (ObjNum 2346 snap 0 seq_num 2346) dirty exists 2020-06-12T02:37:43.826 INFO:tasks.rados.rados.0.smithi107.stdout:2347: left oid 9 (ObjNum 2346 snap 0 seq_num 2346) 2020-06-12T02:37:43.826 INFO:tasks.rados.rados.0.smithi107.stdout:2341: done (5 left) 2020-06-12T02:37:43.826 INFO:tasks.rados.rados.0.smithi107.stdout:2342: done (4 left) 2020-06-12T02:37:43.827 INFO:tasks.rados.rados.0.smithi107.stdout:2346: done (3 left) 2020-06-12T02:37:43.827 INFO:tasks.rados.rados.0.smithi107.stdout:2347: done (2 left) 2020-06-12T02:37:43.827 INFO:tasks.rados.rados.0.smithi107.stdout:2349: done (1 left) 2020-06-12T02:37:43.827 INFO:tasks.rados.rados.0.smithi107.stdout:2350: delete oid 7 current snap is 0 2020-06-12T02:37:43.827 INFO:tasks.rados.rados.0.smithi107.stdout:2350: done (1 left) 2020-06-12T02:37:43.828 INFO:tasks.rados.rados.0.smithi107.stdout:2351: delete oid 8 current snap is 0 2020-06-12T02:37:43.828 INFO:tasks.rados.rados.0.smithi107.stdout:2348: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:43.828 INFO:tasks.rados.rados.0.smithi107.stdout:2351: done (1 left) 2020-06-12T02:37:43.828 INFO:tasks.rados.rados.0.smithi107.stdout:2352: delete oid 15 current snap is 0 2020-06-12T02:37:43.828 INFO:tasks.rados.rados.0.smithi107.stdout:2348: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:43.828 INFO:tasks.rados.rados.0.smithi107.stdout:2348: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:43.829 INFO:tasks.rados.rados.0.smithi107.stdout:2348: finishing write tid 5 to smithi10729407-16 2020-06-12T02:37:43.829 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 405 (ObjNum 2347 snap 0 seq_num 2347) dirty exists 2020-06-12T02:37:43.829 INFO:tasks.rados.rados.0.smithi107.stdout:2348: left oid 16 (ObjNum 2347 snap 0 seq_num 2347) 2020-06-12T02:37:43.829 INFO:tasks.rados.rados.0.smithi107.stdout:2348: done (1 left) 2020-06-12T02:37:43.829 INFO:tasks.rados.rados.0.smithi107.stdout:2352: done (0 left) 2020-06-12T02:37:43.830 INFO:tasks.rados.rados.0.smithi107.stdout:2353: delete oid 10 current snap is 0 2020-06-12T02:37:43.830 INFO:tasks.rados.rados.0.smithi107.stdout:2353: done (0 left) 2020-06-12T02:37:43.830 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:37:43.830 INFO:tasks.rados.rados.0.smithi107.stdout:2354: seq_num 2353 ranges {696320=614400,1310720=401408} 2020-06-12T02:37:43.830 INFO:tasks.rados.rados.0.smithi107.stdout:2354: writing smithi10729407-19 from 696320 to 1310720 tid 1 2020-06-12T02:37:43.830 INFO:tasks.rados.rados.0.smithi107.stdout:2354: writing smithi10729407-19 from 1310720 to 1712128 tid 2 2020-06-12T02:37:43.831 INFO:tasks.rados.rados.0.smithi107.stdout:2355: delete oid 15 current snap is 0 2020-06-12T02:37:43.831 INFO:tasks.rados.rados.0.smithi107.stdout:2355: done (1 left) 2020-06-12T02:37:43.831 INFO:tasks.rados.rados.0.smithi107.stdout:2356: delete oid 6 current snap is 0 2020-06-12T02:37:43.831 INFO:tasks.rados.rados.0.smithi107.stdout:2356: done (1 left) 2020-06-12T02:37:43.831 INFO:tasks.rados.rados.0.smithi107.stdout:2357: delete oid 6 current snap is 0 2020-06-12T02:37:43.832 INFO:tasks.rados.rados.0.smithi107.stdout:2354: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:43.832 INFO:tasks.rados.rados.0.smithi107.stdout:2354: finishing write tid 2 to smithi10729407-19 2020-06-12T02:37:43.832 INFO:tasks.rados.rados.0.smithi107.stdout:2357: done (1 left) 2020-06-12T02:37:43.832 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:37:43.832 INFO:tasks.rados.rados.0.smithi107.stdout:2358: seq_num 2357 ranges {0=679936,679936=401408} 2020-06-12T02:37:43.832 INFO:tasks.rados.rados.0.smithi107.stdout:2358: writing smithi10729407-1 from 0 to 679936 tid 1 2020-06-12T02:37:43.833 INFO:tasks.rados.rados.0.smithi107.stdout:2358: writing smithi10729407-1 from 679936 to 1081344 tid 2 2020-06-12T02:37:43.833 INFO:tasks.rados.rados.0.smithi107.stdout:2359: delete oid 7 current snap is 0 2020-06-12T02:37:43.833 INFO:tasks.rados.rados.0.smithi107.stdout:2354: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:43.833 INFO:tasks.rados.rados.0.smithi107.stdout:2354: finishing write tid 5 to smithi10729407-19 2020-06-12T02:37:43.833 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 366 (ObjNum 2353 snap 0 seq_num 2353) dirty exists 2020-06-12T02:37:43.834 INFO:tasks.rados.rados.0.smithi107.stdout:2354: left oid 19 (ObjNum 2353 snap 0 seq_num 2353) 2020-06-12T02:37:43.839 INFO:tasks.rados.rados.0.smithi107.stdout:2354: done (2 left) 2020-06-12T02:37:43.839 INFO:tasks.rados.rados.0.smithi107.stdout:2359: done (1 left) 2020-06-12T02:37:43.839 INFO:tasks.rados.rados.0.smithi107.stdout:2360: delete oid 5 current snap is 0 2020-06-12T02:37:43.840 INFO:tasks.rados.rados.0.smithi107.stdout:2360: done (1 left) 2020-06-12T02:37:43.840 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:37:43.840 INFO:tasks.rados.rados.0.smithi107.stdout:2361: seq_num 2360 ranges {0=729088,729088=401408} 2020-06-12T02:37:43.840 INFO:tasks.rados.rados.0.smithi107.stdout:2361: writing smithi10729407-10 from 0 to 729088 tid 1 2020-06-12T02:37:43.840 INFO:tasks.rados.rados.0.smithi107.stdout:2361: writing smithi10729407-10 from 729088 to 1130496 tid 2 2020-06-12T02:37:43.841 INFO:tasks.rados.rados.0.smithi107.stdout:2362: delete oid 18 current snap is 0 2020-06-12T02:37:43.841 INFO:tasks.rados.rados.0.smithi107.stdout:2358: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:43.841 INFO:tasks.rados.rados.0.smithi107.stdout:2358: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:43.841 INFO:tasks.rados.rados.0.smithi107.stdout:2358: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:43.841 INFO:tasks.rados.rados.0.smithi107.stdout:2358: finishing write tid 5 to smithi10729407-1 2020-06-12T02:37:43.841 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 491 (ObjNum 2357 snap 0 seq_num 2357) dirty exists 2020-06-12T02:37:43.842 INFO:tasks.rados.rados.0.smithi107.stdout:2358: left oid 1 (ObjNum 2357 snap 0 seq_num 2357) 2020-06-12T02:37:43.842 INFO:tasks.rados.rados.0.smithi107.stdout:2358: done (2 left) 2020-06-12T02:37:43.842 INFO:tasks.rados.rados.0.smithi107.stdout:2362: done (1 left) 2020-06-12T02:37:43.843 INFO:tasks.rados.rados.0.smithi107.stdout:2363: delete oid 14 current snap is 0 2020-06-12T02:37:43.843 INFO:tasks.rados.rados.0.smithi107.stdout:2363: done (1 left) 2020-06-12T02:37:43.843 INFO:tasks.rados.rados.0.smithi107.stdout:2364: delete oid 8 current snap is 0 2020-06-12T02:37:43.843 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:43.768+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:43.844 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-06-12T02:37:43.870 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:43.868+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:43.906 INFO:tasks.rados.rados.0.smithi107.stdout:2361: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:43.907 INFO:tasks.rados.rados.0.smithi107.stdout:2364: done (1 left) 2020-06-12T02:37:43.907 INFO:tasks.rados.rados.0.smithi107.stdout:2365: delete oid 4 current snap is 0 2020-06-12T02:37:43.908 INFO:tasks.rados.rados.0.smithi107.stdout:2365: done (1 left) 2020-06-12T02:37:43.908 INFO:tasks.rados.rados.0.smithi107.stdout:2366: delete oid 18 current snap is 0 2020-06-12T02:37:43.909 INFO:tasks.rados.rados.0.smithi107.stdout:2361: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:43.910 INFO:tasks.rados.rados.0.smithi107.stdout:2361: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:43.911 INFO:tasks.rados.rados.0.smithi107.stdout:2366: done (1 left) 2020-06-12T02:37:43.911 INFO:tasks.rados.rados.0.smithi107.stdout:2367: delete oid 2 current snap is 0 2020-06-12T02:37:43.912 INFO:tasks.rados.rados.0.smithi107.stdout:2361: finishing write tid 5 to smithi10729407-10 2020-06-12T02:37:43.912 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 220 (ObjNum 2360 snap 0 seq_num 2360) dirty exists 2020-06-12T02:37:43.912 INFO:tasks.rados.rados.0.smithi107.stdout:2361: left oid 10 (ObjNum 2360 snap 0 seq_num 2360) 2020-06-12T02:37:43.914 INFO:tasks.rados.rados.0.smithi107.stdout:2361: done (1 left) 2020-06-12T02:37:43.914 INFO:tasks.rados.rados.0.smithi107.stdout:2367: done (0 left) 2020-06-12T02:37:43.914 INFO:tasks.rados.rados.0.smithi107.stdout:2368: delete oid 18 current snap is 0 2020-06-12T02:37:43.916 INFO:tasks.rados.rados.0.smithi107.stdout:2368: done (0 left) 2020-06-12T02:37:43.916 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:37:43.917 INFO:tasks.rados.rados.0.smithi107.stdout:2369: seq_num 2368 ranges {0=466944,466944=753664,1220608=49152} 2020-06-12T02:37:43.919 INFO:tasks.rados.rados.0.smithi107.stdout:2369: writing smithi10729407-4 from 0 to 466944 tid 1 2020-06-12T02:37:43.923 INFO:tasks.rados.rados.0.smithi107.stdout:2369: writing smithi10729407-4 from 466944 to 1220608 tid 2 2020-06-12T02:37:43.924 INFO:tasks.rados.rados.0.smithi107.stdout:2369: writing smithi10729407-4 from 1220608 to 1269760 tid 3 2020-06-12T02:37:43.924 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:37:43.924 INFO:tasks.rados.rados.0.smithi107.stdout:2370: seq_num 2369 ranges {0=483328,483328=638976,1122304=163840} 2020-06-12T02:37:43.926 INFO:tasks.rados.rados.0.smithi107.stdout:2370: writing smithi10729407-5 from 0 to 483328 tid 1 2020-06-12T02:37:43.931 INFO:tasks.rados.rados.0.smithi107.stdout:2370: writing smithi10729407-5 from 483328 to 1122304 tid 2 2020-06-12T02:37:43.932 INFO:tasks.rados.rados.0.smithi107.stdout:2370: writing smithi10729407-5 from 1122304 to 1286144 tid 3 2020-06-12T02:37:43.932 INFO:tasks.rados.rados.0.smithi107.stdout:2371: delete oid 13 current snap is 0 2020-06-12T02:37:43.932 INFO:tasks.rados.rados.0.smithi107.stdout:2369: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:43.933 INFO:tasks.rados.rados.0.smithi107.stdout:2369: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:43.933 INFO:tasks.rados.rados.0.smithi107.stdout:2369: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:43.933 INFO:tasks.rados.rados.0.smithi107.stdout:2369: finishing write tid 5 to smithi10729407-4 2020-06-12T02:37:43.934 INFO:tasks.rados.rados.0.smithi107.stdout:2369: finishing write tid 6 to smithi10729407-4 2020-06-12T02:37:43.934 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 453 (ObjNum 2368 snap 0 seq_num 2368) dirty exists 2020-06-12T02:37:43.934 INFO:tasks.rados.rados.0.smithi107.stdout:2369: left oid 4 (ObjNum 2368 snap 0 seq_num 2368) 2020-06-12T02:37:43.935 INFO:tasks.rados.rados.0.smithi107.stdout:2369: done (2 left) 2020-06-12T02:37:43.935 INFO:tasks.rados.rados.0.smithi107.stdout:2371: done (1 left) 2020-06-12T02:37:43.935 INFO:tasks.rados.rados.0.smithi107.stdout:2372: delete oid 7 current snap is 0 2020-06-12T02:37:43.939 INFO:tasks.rados.rados.0.smithi107.stdout:2370: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:43.940 INFO:tasks.rados.rados.0.smithi107.stdout:2370: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:43.942 INFO:tasks.rados.rados.0.smithi107.stdout:2372: done (1 left) 2020-06-12T02:37:43.942 INFO:tasks.rados.rados.0.smithi107.stdout:2373: delete oid 14 current snap is 0 2020-06-12T02:37:43.942 INFO:tasks.rados.rados.0.smithi107.stdout:2370: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:43.943 INFO:tasks.rados.rados.0.smithi107.stdout:2370: finishing write tid 5 to smithi10729407-5 2020-06-12T02:37:43.946 INFO:tasks.rados.rados.0.smithi107.stdout:2373: done (1 left) 2020-06-12T02:37:43.946 INFO:tasks.rados.rados.0.smithi107.stdout:2374: delete oid 2 current snap is 0 2020-06-12T02:37:43.946 INFO:tasks.rados.rados.0.smithi107.stdout:2370: finishing write tid 6 to smithi10729407-5 2020-06-12T02:37:43.946 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 196 (ObjNum 2369 snap 0 seq_num 2369) dirty exists 2020-06-12T02:37:43.946 INFO:tasks.rados.rados.0.smithi107.stdout:2370: left oid 5 (ObjNum 2369 snap 0 seq_num 2369) 2020-06-12T02:37:43.947 INFO:tasks.rados.rados.0.smithi107.stdout:2370: done (1 left) 2020-06-12T02:37:43.947 INFO:tasks.rados.rados.0.smithi107.stdout:2374: done (0 left) 2020-06-12T02:37:43.947 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:37:43.948 INFO:tasks.rados.rados.0.smithi107.stdout:2375: seq_num 2374 ranges {0=565248,565248=507904,1073152=294912} 2020-06-12T02:37:43.948 INFO:tasks.rados.rados.0.smithi107.stdout:2375: writing smithi10729407-7 from 0 to 565248 tid 1 2020-06-12T02:37:43.951 INFO:tasks.rados.rados.0.smithi107.stdout:2375: writing smithi10729407-7 from 565248 to 1073152 tid 2 2020-06-12T02:37:43.953 INFO:tasks.rados.rados.0.smithi107.stdout:2375: writing smithi10729407-7 from 1073152 to 1368064 tid 3 2020-06-12T02:37:43.954 INFO:tasks.rados.rados.0.smithi107.stdout:2376: delete oid 4 current snap is 0 2020-06-12T02:37:43.954 INFO:tasks.rados.rados.0.smithi107.stdout:2375: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:43.956 INFO:tasks.rados.rados.0.smithi107.stdout:2376: done (1 left) 2020-06-12T02:37:43.956 INFO:tasks.rados.rados.0.smithi107.stdout:2377: delete oid 4 current snap is 0 2020-06-12T02:37:43.959 INFO:tasks.rados.rados.0.smithi107.stdout:2377: done (1 left) 2020-06-12T02:37:43.959 INFO:tasks.rados.rados.0.smithi107.stdout:2378: delete oid 20 current snap is 0 2020-06-12T02:37:43.961 INFO:tasks.rados.rados.0.smithi107.stdout:2375: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:43.961 INFO:tasks.rados.rados.0.smithi107.stdout:2375: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:43.962 INFO:tasks.rados.rados.0.smithi107.stdout:2375: finishing write tid 5 to smithi10729407-7 2020-06-12T02:37:43.962 INFO:tasks.rados.rados.0.smithi107.stdout:2378: done (1 left) 2020-06-12T02:37:43.962 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:37:43.963 INFO:tasks.rados.rados.0.smithi107.stdout:2379: seq_num 2378 ranges {1130496=630784,1761280=450560,2211840=352256} 2020-06-12T02:37:43.971 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:43.968+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:43.972 INFO:tasks.rados.rados.0.smithi107.stdout:2379: writing smithi10729407-10 from 1130496 to 1761280 tid 1 2020-06-12T02:37:43.974 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:37:43.974 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:37:43.974 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2020-06-12T02:37:43.974 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:37:43.975 INFO:tasks.rados.rados.0.smithi107.stdout:2379: writing smithi10729407-10 from 1761280 to 2211840 tid 2 2020-06-12T02:37:43.976 INFO:tasks.rados.rados.0.smithi107.stdout:2379: writing smithi10729407-10 from 2211840 to 2564096 tid 3 2020-06-12T02:37:43.976 INFO:tasks.rados.rados.0.smithi107.stdout:2380: delete oid 5 current snap is 0 2020-06-12T02:37:43.977 INFO:tasks.rados.rados.0.smithi107.stdout:2375: finishing write tid 6 to smithi10729407-7 2020-06-12T02:37:43.977 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 453 (ObjNum 2374 snap 0 seq_num 2374) dirty exists 2020-06-12T02:37:43.977 INFO:tasks.rados.rados.0.smithi107.stdout:2375: left oid 7 (ObjNum 2374 snap 0 seq_num 2374) 2020-06-12T02:37:43.978 INFO:tasks.rados.rados.0.smithi107.stdout:2379: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:43.983 INFO:tasks.rados.rados.0.smithi107.stdout:2375: done (2 left) 2020-06-12T02:37:43.983 INFO:tasks.rados.rados.0.smithi107.stdout:2380: done (1 left) 2020-06-12T02:37:43.983 INFO:tasks.rados.rados.0.smithi107.stdout:2381: delete oid 1 current snap is 0 2020-06-12T02:37:43.984 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-06-12T02:37:43.985 INFO:tasks.rados.rados.0.smithi107.stdout:2381: done (1 left) 2020-06-12T02:37:43.986 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:37:43.986 INFO:tasks.rados.rados.0.smithi107.stdout:2382: seq_num 2381 ranges {0=679936,679936=548864,1228800=253952} 2020-06-12T02:37:43.990 INFO:tasks.rados.rados.0.smithi107.stdout:2382: writing smithi10729407-1 from 0 to 679936 tid 1 2020-06-12T02:37:43.993 INFO:tasks.rados.rados.0.smithi107.stdout:2382: writing smithi10729407-1 from 679936 to 1228800 tid 2 2020-06-12T02:37:43.995 INFO:tasks.rados.rados.0.smithi107.stdout:2382: writing smithi10729407-1 from 1228800 to 1482752 tid 3 2020-06-12T02:37:43.995 INFO:tasks.rados.rados.0.smithi107.stdout:2383: delete oid 9 current snap is 0 2020-06-12T02:37:43.995 INFO:tasks.rados.rados.0.smithi107.stdout:2379: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:43.995 INFO:tasks.rados.rados.0.smithi107.stdout:2379: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:43.996 INFO:tasks.rados.rados.0.smithi107.stdout:2379: finishing write tid 5 to smithi10729407-10 2020-06-12T02:37:43.996 INFO:tasks.rados.rados.0.smithi107.stdout:2379: finishing write tid 6 to smithi10729407-10 2020-06-12T02:37:43.996 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 224 (ObjNum 2378 snap 0 seq_num 2378) dirty exists 2020-06-12T02:37:43.997 INFO:tasks.rados.rados.0.smithi107.stdout:2379: left oid 10 (ObjNum 2378 snap 0 seq_num 2378) 2020-06-12T02:37:43.999 INFO:tasks.rados.rados.0.smithi107.stdout:2382: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:43.999 INFO:tasks.rados.rados.0.smithi107.stdout:2379: done (2 left) 2020-06-12T02:37:43.999 INFO:tasks.rados.rados.0.smithi107.stdout:2383: done (1 left) 2020-06-12T02:37:43.999 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:44.000 INFO:tasks.rados.rados.0.smithi107.stdout:2384: seq_num 2383 ranges {0=720896,720896=720896,1441792=81920} 2020-06-12T02:37:44.001 INFO:tasks.rados.rados.0.smithi107.stdout:2384: writing smithi10729407-2 from 0 to 720896 tid 1 2020-06-12T02:37:44.006 INFO:tasks.rados.rados.0.smithi107.stdout:2384: writing smithi10729407-2 from 720896 to 1441792 tid 2 2020-06-12T02:37:44.007 INFO:tasks.rados.rados.0.smithi107.stdout:2384: writing smithi10729407-2 from 1441792 to 1523712 tid 3 2020-06-12T02:37:44.008 INFO:tasks.rados.rados.0.smithi107.stdout:2385: delete oid 12 current snap is 0 2020-06-12T02:37:44.008 INFO:tasks.rados.rados.0.smithi107.stdout:2382: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:44.008 INFO:tasks.rados.rados.0.smithi107.stdout:2382: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:44.008 INFO:tasks.rados.rados.0.smithi107.stdout:2382: finishing write tid 5 to smithi10729407-1 2020-06-12T02:37:44.008 INFO:tasks.rados.rados.0.smithi107.stdout:2382: finishing write tid 6 to smithi10729407-1 2020-06-12T02:37:44.009 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 496 (ObjNum 2381 snap 0 seq_num 2381) dirty exists 2020-06-12T02:37:44.009 INFO:tasks.rados.rados.0.smithi107.stdout:2382: left oid 1 (ObjNum 2381 snap 0 seq_num 2381) 2020-06-12T02:37:44.009 INFO:tasks.rados.rados.0.smithi107.stdout:2384: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:44.012 INFO:tasks.rados.rados.0.smithi107.stdout:2382: done (2 left) 2020-06-12T02:37:44.012 INFO:tasks.rados.rados.0.smithi107.stdout:2385: done (1 left) 2020-06-12T02:37:44.012 INFO:tasks.rados.rados.0.smithi107.stdout:2386: delete oid 5 current snap is 0 2020-06-12T02:37:44.012 INFO:tasks.rados.rados.0.smithi107.stdout:2386: done (1 left) 2020-06-12T02:37:44.012 INFO:tasks.rados.rados.0.smithi107.stdout:2387: delete oid 7 current snap is 0 2020-06-12T02:37:44.015 INFO:tasks.rados.rados.0.smithi107.stdout:2387: done (1 left) 2020-06-12T02:37:44.015 INFO:tasks.rados.rados.0.smithi107.stdout:2388: delete oid 15 current snap is 0 2020-06-12T02:37:44.016 INFO:tasks.rados.rados.0.smithi107.stdout:2384: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:44.016 INFO:tasks.rados.rados.0.smithi107.stdout:2388: done (1 left) 2020-06-12T02:37:44.017 INFO:tasks.rados.rados.0.smithi107.stdout:2389: delete oid 6 current snap is 0 2020-06-12T02:37:44.017 INFO:tasks.rados.rados.0.smithi107.stdout:2384: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:44.017 INFO:tasks.rados.rados.0.smithi107.stdout:2384: finishing write tid 5 to smithi10729407-2 2020-06-12T02:37:44.018 INFO:tasks.rados.rados.0.smithi107.stdout:2384: finishing write tid 6 to smithi10729407-2 2020-06-12T02:37:44.018 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 259 (ObjNum 2383 snap 0 seq_num 2383) dirty exists 2020-06-12T02:37:44.018 INFO:tasks.rados.rados.0.smithi107.stdout:2384: left oid 2 (ObjNum 2383 snap 0 seq_num 2383) 2020-06-12T02:37:44.018 INFO:tasks.rados.rados.0.smithi107.stdout:2384: done (1 left) 2020-06-12T02:37:44.019 INFO:tasks.rados.rados.0.smithi107.stdout:2389: done (0 left) 2020-06-12T02:37:44.019 INFO:tasks.rados.rados.0.smithi107.stdout:2390: delete oid 17 current snap is 0 2020-06-12T02:37:44.020 INFO:tasks.rados.rados.0.smithi107.stdout:2390: done (0 left) 2020-06-12T02:37:44.020 INFO:tasks.rados.rados.0.smithi107.stdout:2391: delete oid 7 current snap is 0 2020-06-12T02:37:44.023 INFO:tasks.rados.rados.0.smithi107.stdout:2391: done (0 left) 2020-06-12T02:37:44.023 INFO:tasks.rados.rados.0.smithi107.stdout:2392: delete oid 3 current snap is 0 2020-06-12T02:37:44.026 INFO:tasks.rados.rados.0.smithi107.stdout:2392: done (0 left) 2020-06-12T02:37:44.026 INFO:tasks.rados.rados.0.smithi107.stdout:2393: delete oid 11 current snap is 0 2020-06-12T02:37:44.028 INFO:tasks.rados.rados.0.smithi107.stdout:2393: done (0 left) 2020-06-12T02:37:44.029 INFO:tasks.rados.rados.0.smithi107.stdout:2394: delete oid 20 current snap is 0 2020-06-12T02:37:44.030 INFO:tasks.rados.rados.0.smithi107.stdout:2394: done (0 left) 2020-06-12T02:37:44.030 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:37:44.030 INFO:tasks.rados.rados.0.smithi107.stdout:2395: seq_num 2394 ranges {0=499712,499712=696320,1196032=507904} 2020-06-12T02:37:44.032 INFO:tasks.rados.rados.0.smithi107.stdout:2395: writing smithi10729407-18 from 0 to 499712 tid 1 2020-06-12T02:37:44.036 INFO:tasks.rados.rados.0.smithi107.stdout:2395: writing smithi10729407-18 from 499712 to 1196032 tid 2 2020-06-12T02:37:44.044 INFO:tasks.rados.rados.0.smithi107.stdout:2395: writing smithi10729407-18 from 1196032 to 1703936 tid 3 2020-06-12T02:37:44.044 INFO:tasks.rados.rados.0.smithi107.stdout:2396: delete oid 10 current snap is 0 2020-06-12T02:37:44.044 INFO:tasks.rados.rados.0.smithi107.stdout:2395: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:44.045 INFO:tasks.rados.rados.0.smithi107.stdout:2396: done (1 left) 2020-06-12T02:37:44.045 INFO:tasks.rados.rados.0.smithi107.stdout:2397: delete oid 7 current snap is 0 2020-06-12T02:37:44.046 INFO:tasks.rados.rados.0.smithi107.stdout:2395: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:44.047 INFO:tasks.rados.rados.0.smithi107.stdout:2397: done (1 left) 2020-06-12T02:37:44.047 INFO:tasks.rados.rados.0.smithi107.stdout:2398: delete oid 20 current snap is 0 2020-06-12T02:37:44.048 INFO:tasks.rados.rados.0.smithi107.stdout:2395: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:44.050 INFO:tasks.rados.rados.0.smithi107.stdout:2395: finishing write tid 5 to smithi10729407-18 2020-06-12T02:37:44.051 INFO:tasks.rados.rados.0.smithi107.stdout:2398: done (1 left) 2020-06-12T02:37:44.051 INFO:tasks.rados.rados.0.smithi107.stdout:2399: delete oid 5 current snap is 0 2020-06-12T02:37:44.051 INFO:tasks.rados.rados.0.smithi107.stdout:2395: finishing write tid 6 to smithi10729407-18 2020-06-12T02:37:44.052 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 411 (ObjNum 2394 snap 0 seq_num 2394) dirty exists 2020-06-12T02:37:44.052 INFO:tasks.rados.rados.0.smithi107.stdout:2395: left oid 18 (ObjNum 2394 snap 0 seq_num 2394) 2020-06-12T02:37:44.052 INFO:tasks.rados.rados.0.smithi107.stdout:2395: done (1 left) 2020-06-12T02:37:44.053 INFO:tasks.rados.rados.0.smithi107.stdout:2399: done (0 left) 2020-06-12T02:37:44.053 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:37:44.053 INFO:tasks.rados.rados.0.smithi107.stdout:2400: seq_num 2399 ranges {1482752=581632,2064384=565248,2629632=638976} 2020-06-12T02:37:44.062 INFO:tasks.rados.rados.0.smithi107.stdout:2400: writing smithi10729407-1 from 1482752 to 2064384 tid 1 2020-06-12T02:37:44.065 INFO:tasks.rados.rados.0.smithi107.stdout:2400: writing smithi10729407-1 from 2064384 to 2629632 tid 2 2020-06-12T02:37:44.069 INFO:tasks.rados.rados.0.smithi107.stdout:2400: writing smithi10729407-1 from 2629632 to 3268608 tid 3 2020-06-12T02:37:44.070 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:37:44.070 INFO:tasks.rados.rados.0.smithi107.stdout:2401: seq_num 2400 ranges {1597440=598016,2195456=450560,2646016=516096,3162112=237568} 2020-06-12T02:37:44.072 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:44.069+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:44.081 INFO:tasks.rados.rados.0.smithi107.stdout:2401: writing smithi10729407-16 from 1597440 to 2195456 tid 1 2020-06-12T02:37:44.084 INFO:tasks.rados.rados.0.smithi107.stdout:2401: writing smithi10729407-16 from 2195456 to 2646016 tid 2 2020-06-12T02:37:44.089 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-06-12T02:37:44.090 INFO:tasks.rados.rados.0.smithi107.stdout:2401: writing smithi10729407-16 from 2646016 to 3162112 tid 3 2020-06-12T02:37:44.091 INFO:tasks.rados.rados.0.smithi107.stdout:2401: writing smithi10729407-16 from 3162112 to 3399680 tid 4 2020-06-12T02:37:44.091 INFO:tasks.rados.rados.0.smithi107.stdout:2402: delete oid 17 current snap is 0 2020-06-12T02:37:44.091 INFO:tasks.rados.rados.0.smithi107.stdout:2400: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:44.091 INFO:tasks.rados.rados.0.smithi107.stdout:2400: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:44.092 INFO:tasks.rados.rados.0.smithi107.stdout:2400: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:44.092 INFO:tasks.rados.rados.0.smithi107.stdout:2400: finishing write tid 5 to smithi10729407-1 2020-06-12T02:37:44.092 INFO:tasks.rados.rados.0.smithi107.stdout:2400: finishing write tid 6 to smithi10729407-1 2020-06-12T02:37:44.092 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 501 (ObjNum 2399 snap 0 seq_num 2399) dirty exists 2020-06-12T02:37:44.092 INFO:tasks.rados.rados.0.smithi107.stdout:2400: left oid 1 (ObjNum 2399 snap 0 seq_num 2399) 2020-06-12T02:37:44.093 INFO:tasks.rados.rados.0.smithi107.stdout:2400: done (2 left) 2020-06-12T02:37:44.093 INFO:tasks.rados.rados.0.smithi107.stdout:2402: done (1 left) 2020-06-12T02:37:44.094 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:37:44.094 INFO:tasks.rados.rados.0.smithi107.stdout:2403: seq_num 2402 ranges {0=638976,638976=622592,1261568=483328,1744896=98304} 2020-06-12T02:37:44.095 INFO:tasks.rados.rados.0.smithi107.stdout:2403: writing smithi10729407-10 from 0 to 638976 tid 1 2020-06-12T02:37:44.099 INFO:tasks.rados.rados.0.smithi107.stdout:2403: writing smithi10729407-10 from 638976 to 1261568 tid 2 2020-06-12T02:37:44.102 INFO:tasks.rados.rados.0.smithi107.stdout:2403: writing smithi10729407-10 from 1261568 to 1744896 tid 3 2020-06-12T02:37:44.102 INFO:tasks.rados.rados.0.smithi107.stdout:2403: writing smithi10729407-10 from 1744896 to 1843200 tid 4 2020-06-12T02:37:44.103 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:37:44.103 INFO:tasks.rados.rados.0.smithi107.stdout:2404: seq_num 2403 ranges {1712128=655360,2367488=557056,2924544=647168} 2020-06-12T02:37:44.115 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:37:44.115 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:37:44.115 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:37:44.116 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:37:44.116 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:37:44.116 INFO:tasks.rados.rados.0.smithi107.stdout:2404: writing smithi10729407-19 from 1712128 to 2367488 tid 1 2020-06-12T02:37:44.119 INFO:tasks.rados.rados.0.smithi107.stdout:2404: writing smithi10729407-19 from 2367488 to 2924544 tid 2 2020-06-12T02:37:44.124 INFO:tasks.rados.rados.0.smithi107.stdout:2404: writing smithi10729407-19 from 2924544 to 3571712 tid 3 2020-06-12T02:37:44.124 INFO:tasks.rados.rados.0.smithi107.stdout:2405: delete oid 12 current snap is 0 2020-06-12T02:37:44.124 INFO:tasks.rados.rados.0.smithi107.stdout:2401: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:44.124 INFO:tasks.rados.rados.0.smithi107.stdout:2401: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:44.124 INFO:tasks.rados.rados.0.smithi107.stdout:2401: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:44.125 INFO:tasks.rados.rados.0.smithi107.stdout:2401: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:44.125 INFO:tasks.rados.rados.0.smithi107.stdout:2401: finishing write tid 6 to smithi10729407-16 2020-06-12T02:37:44.125 INFO:tasks.rados.rados.0.smithi107.stdout:2401: finishing write tid 7 to smithi10729407-16 2020-06-12T02:37:44.125 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 411 (ObjNum 2400 snap 0 seq_num 2400) dirty exists 2020-06-12T02:37:44.125 INFO:tasks.rados.rados.0.smithi107.stdout:2401: left oid 16 (ObjNum 2400 snap 0 seq_num 2400) 2020-06-12T02:37:44.126 INFO:tasks.rados.rados.0.smithi107.stdout:2403: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:44.126 INFO:tasks.rados.rados.0.smithi107.stdout:2403: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:44.126 INFO:tasks.rados.rados.0.smithi107.stdout:2403: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:44.126 INFO:tasks.rados.rados.0.smithi107.stdout:2403: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:44.126 INFO:tasks.rados.rados.0.smithi107.stdout:2403: finishing write tid 6 to smithi10729407-10 2020-06-12T02:37:44.126 INFO:tasks.rados.rados.0.smithi107.stdout:2403: finishing write tid 7 to smithi10729407-10 2020-06-12T02:37:44.127 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 230 (ObjNum 2402 snap 0 seq_num 2402) dirty exists 2020-06-12T02:37:44.127 INFO:tasks.rados.rados.0.smithi107.stdout:2403: left oid 10 (ObjNum 2402 snap 0 seq_num 2402) 2020-06-12T02:37:44.127 INFO:tasks.rados.rados.0.smithi107.stdout:2404: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:44.128 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-06-12T02:37:44.132 INFO:tasks.rados.rados.0.smithi107.stdout:2404: finishing write tid 2 to smithi10729407-19 2020-06-12T02:37:44.133 INFO:tasks.rados.rados.0.smithi107.stdout:2404: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:44.133 INFO:tasks.rados.rados.0.smithi107.stdout:2404: finishing write tid 5 to smithi10729407-19 2020-06-12T02:37:44.135 INFO:tasks.rados.rados.0.smithi107.stdout:2404: finishing write tid 6 to smithi10729407-19 2020-06-12T02:37:44.135 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 371 (ObjNum 2403 snap 0 seq_num 2403) dirty exists 2020-06-12T02:37:44.136 INFO:tasks.rados.rados.0.smithi107.stdout:2404: left oid 19 (ObjNum 2403 snap 0 seq_num 2403) 2020-06-12T02:37:44.136 INFO:tasks.rados.rados.0.smithi107.stdout:2401: done (3 left) 2020-06-12T02:37:44.137 INFO:tasks.rados.rados.0.smithi107.stdout:2403: done (2 left) 2020-06-12T02:37:44.137 INFO:tasks.rados.rados.0.smithi107.stdout:2404: done (1 left) 2020-06-12T02:37:44.137 INFO:tasks.rados.rados.0.smithi107.stdout:2405: done (0 left) 2020-06-12T02:37:44.138 INFO:tasks.rados.rados.0.smithi107.stdout:2406: delete oid 1 current snap is 0 2020-06-12T02:37:44.138 INFO:tasks.rados.rados.0.smithi107.stdout:2406: done (0 left) 2020-06-12T02:37:44.139 INFO:tasks.rados.rados.0.smithi107.stdout:2407: delete oid 14 current snap is 0 2020-06-12T02:37:44.141 INFO:tasks.rados.rados.0.smithi107.stdout:2407: done (0 left) 2020-06-12T02:37:44.142 INFO:tasks.rados.rados.0.smithi107.stdout:2408: delete oid 20 current snap is 0 2020-06-12T02:37:44.146 INFO:tasks.rados.rados.0.smithi107.stdout:2408: done (0 left) 2020-06-12T02:37:44.146 INFO:tasks.rados.rados.0.smithi107.stdout:2409: delete oid 16 current snap is 0 2020-06-12T02:37:44.148 INFO:tasks.rados.rados.0.smithi107.stdout:2409: done (0 left) 2020-06-12T02:37:44.149 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:37:44.149 INFO:tasks.rados.rados.0.smithi107.stdout:2410: seq_num 2409 ranges {0=753664,753664=663552,1417216=540672} 2020-06-12T02:37:44.153 INFO:tasks.rados.rados.0.smithi107.stdout:2410: writing smithi10729407-13 from 0 to 753664 tid 1 2020-06-12T02:37:44.156 INFO:tasks.rados.rados.0.smithi107.stdout:2410: writing smithi10729407-13 from 753664 to 1417216 tid 2 2020-06-12T02:37:44.160 INFO:tasks.rados.rados.0.smithi107.stdout:2410: writing smithi10729407-13 from 1417216 to 1957888 tid 3 2020-06-12T02:37:44.160 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:37:44.160 INFO:tasks.rados.rados.0.smithi107.stdout:2411: seq_num 2410 ranges {0=770048,770048=598016,1368064=491520,1859584=114688} 2020-06-12T02:37:44.164 INFO:tasks.rados.rados.0.smithi107.stdout:2411: writing smithi10729407-14 from 0 to 770048 tid 1 2020-06-12T02:37:44.168 INFO:tasks.rados.rados.0.smithi107.stdout:2411: writing smithi10729407-14 from 770048 to 1368064 tid 2 2020-06-12T02:37:44.173 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:37:44.169+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:37:44.173 INFO:tasks.rados.rados.0.smithi107.stdout:2411: writing smithi10729407-14 from 1368064 to 1859584 tid 3 2020-06-12T02:37:44.174 INFO:tasks.rados.rados.0.smithi107.stdout:2411: writing smithi10729407-14 from 1859584 to 1974272 tid 4 2020-06-12T02:37:44.174 INFO:tasks.rados.rados.0.smithi107.stdout:2412: delete oid 7 current snap is 0 2020-06-12T02:37:44.174 INFO:tasks.rados.rados.0.smithi107.stdout:2410: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:44.175 INFO:tasks.rados.rados.0.smithi107.stdout:2410: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:44.175 INFO:tasks.rados.rados.0.smithi107.stdout:2410: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:44.176 INFO:tasks.rados.rados.0.smithi107.stdout:2410: finishing write tid 5 to smithi10729407-13 2020-06-12T02:37:44.176 INFO:tasks.rados.rados.0.smithi107.stdout:2412: done (2 left) 2020-06-12T02:37:44.176 INFO:tasks.rados.rados.0.smithi107.stdout:2413: delete oid 4 current snap is 0 2020-06-12T02:37:44.178 INFO:tasks.rados.rados.0.smithi107.stdout:2410: finishing write tid 6 to smithi10729407-13 2020-06-12T02:37:44.178 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 460 (ObjNum 2409 snap 0 seq_num 2409) dirty exists 2020-06-12T02:37:44.178 INFO:tasks.rados.rados.0.smithi107.stdout:2410: left oid 13 (ObjNum 2409 snap 0 seq_num 2409) 2020-06-12T02:37:44.183 INFO:tasks.rados.rados.0.smithi107.stdout:2410: done (2 left) 2020-06-12T02:37:44.183 INFO:tasks.rados.rados.0.smithi107.stdout:2413: done (1 left) 2020-06-12T02:37:44.184 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:37:44.184 INFO:tasks.rados.rados.0.smithi107.stdout:2414: seq_num 2413 ranges {0=417792,417792=647168,1064960=671744,1736704=286720} 2020-06-12T02:37:44.184 INFO:tasks.rados.rados.0.smithi107.stdout:2414: writing smithi10729407-16 from 0 to 417792 tid 1 2020-06-12T02:37:44.189 INFO:tasks.rados.rados.0.smithi107.stdout:2414: writing smithi10729407-16 from 417792 to 1064960 tid 2 2020-06-12T02:37:44.193 INFO:tasks.rados.rados.0.smithi107.stdout:2414: writing smithi10729407-16 from 1064960 to 1736704 tid 3 2020-06-12T02:37:44.195 INFO:tasks.rados.rados.0.smithi107.stdout:2414: writing smithi10729407-16 from 1736704 to 2023424 tid 4 2020-06-12T02:37:44.195 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:37:44.195 INFO:tasks.rados.rados.0.smithi107.stdout:2415: seq_num 2414 ranges {0=434176,434176=532480,966656=475136,1441792=598016} 2020-06-12T02:37:44.198 INFO:tasks.rados.rados.0.smithi107.stdout:2415: writing smithi10729407-12 from 0 to 434176 tid 1 2020-06-12T02:37:44.203 INFO:tasks.rados.rados.0.smithi107.stdout:2415: writing smithi10729407-12 from 434176 to 966656 tid 2 2020-06-12T02:37:44.204 INFO:tasks.rados.rados.0.smithi107.stdout:2415: writing smithi10729407-12 from 966656 to 1441792 tid 3 2020-06-12T02:37:44.208 INFO:tasks.rados.rados.0.smithi107.stdout:2415: writing smithi10729407-12 from 1441792 to 2039808 tid 4 2020-06-12T02:37:44.208 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:37:44.209 INFO:tasks.rados.rados.0.smithi107.stdout:2416: seq_num 2415 ranges {0=450560,450560=417792,868352=630784,1499136=524288,2023424=32768} 2020-06-12T02:37:44.210 INFO:tasks.rados.rados.0.smithi107.stdout:2416: writing smithi10729407-11 from 0 to 450560 tid 1 2020-06-12T02:37:44.213 INFO:tasks.rados.rados.0.smithi107.stdout:2416: writing smithi10729407-11 from 450560 to 868352 tid 2 2020-06-12T02:37:44.217 INFO:tasks.rados.rados.0.smithi107.stdout:2416: writing smithi10729407-11 from 868352 to 1499136 tid 3 2020-06-12T02:37:44.220 INFO:tasks.rados.rados.0.smithi107.stdout:2416: writing smithi10729407-11 from 1499136 to 2023424 tid 4 2020-06-12T02:37:44.221 INFO:tasks.rados.rados.0.smithi107.stdout:2416: writing smithi10729407-11 from 2023424 to 2056192 tid 5 2020-06-12T02:37:44.221 INFO:tasks.rados.rados.0.smithi107.stdout:2417: delete oid 8 current snap is 0 2020-06-12T02:37:44.221 INFO:tasks.rados.rados.0.smithi107.stdout:2411: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:44.221 INFO:tasks.rados.rados.0.smithi107.stdout:2411: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:44.222 INFO:tasks.rados.rados.0.smithi107.stdout:2411: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:44.222 INFO:tasks.rados.rados.0.smithi107.stdout:2411: finishing write tid 4 to smithi10729407-14 2020-06-12T02:37:44.222 INFO:tasks.rados.rados.0.smithi107.stdout:2411: finishing write tid 6 to smithi10729407-14 2020-06-12T02:37:44.222 INFO:tasks.rados.rados.0.smithi107.stdout:2411: finishing write tid 7 to smithi10729407-14 2020-06-12T02:37:44.223 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 437 (ObjNum 2410 snap 0 seq_num 2410) dirty exists 2020-06-12T02:37:44.223 INFO:tasks.rados.rados.0.smithi107.stdout:2411: left oid 14 (ObjNum 2410 snap 0 seq_num 2410) 2020-06-12T02:37:44.223 INFO:tasks.rados.rados.0.smithi107.stdout:2414: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:44.223 INFO:tasks.rados.rados.0.smithi107.stdout:2414: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:44.224 INFO:tasks.rados.rados.0.smithi107.stdout:2414: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:44.224 INFO:tasks.rados.rados.0.smithi107.stdout:2414: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:44.224 INFO:tasks.rados.rados.0.smithi107.stdout:2414: finishing write tid 6 to smithi10729407-16 2020-06-12T02:37:44.224 INFO:tasks.rados.rados.0.smithi107.stdout:2415: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:44.225 INFO:tasks.rados.rados.0.smithi107.stdout:2414: finishing write tid 7 to smithi10729407-16 2020-06-12T02:37:44.225 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 417 (ObjNum 2413 snap 0 seq_num 2413) dirty exists 2020-06-12T02:37:44.225 INFO:tasks.rados.rados.0.smithi107.stdout:2414: left oid 16 (ObjNum 2413 snap 0 seq_num 2413) 2020-06-12T02:37:44.225 INFO:tasks.rados.rados.0.smithi107.stdout:2415: finishing write tid 2 to smithi10729407-12 2020-06-12T02:37:44.226 INFO:tasks.rados.rados.0.smithi107.stdout:2415: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:44.226 INFO:tasks.rados.rados.0.smithi107.stdout:2415: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:44.226 INFO:tasks.rados.rados.0.smithi107.stdout:2415: finishing write tid 6 to smithi10729407-12 2020-06-12T02:37:44.226 INFO:tasks.rados.rados.0.smithi107.stdout:2416: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:44.227 INFO:tasks.rados.rados.0.smithi107.stdout:2415: finishing write tid 7 to smithi10729407-12 2020-06-12T02:37:44.227 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 377 (ObjNum 2414 snap 0 seq_num 2414) dirty exists 2020-06-12T02:37:44.227 INFO:tasks.rados.rados.0.smithi107.stdout:2415: left oid 12 (ObjNum 2414 snap 0 seq_num 2414) 2020-06-12T02:37:44.231 INFO:tasks.rados.rados.0.smithi107.stdout:2411: done (4 left) 2020-06-12T02:37:44.231 INFO:tasks.rados.rados.0.smithi107.stdout:2414: done (3 left) 2020-06-12T02:37:44.231 INFO:tasks.rados.rados.0.smithi107.stdout:2415: done (2 left) 2020-06-12T02:37:44.232 INFO:tasks.rados.rados.0.smithi107.stdout:2417: done (1 left) 2020-06-12T02:37:44.232 INFO:tasks.rados.rados.0.smithi107.stdout:2418: delete oid 12 current snap is 0 2020-06-12T02:37:44.232 INFO:tasks.rados.rados.0.smithi107.stdout:2418: done (1 left) 2020-06-12T02:37:44.233 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:37:44.233 INFO:tasks.rados.rados.0.smithi107.stdout:2419: seq_num 2418 ranges {1843200=499712,2342912=516096,2859008=458752,3317760=630784} 2020-06-12T02:37:44.241 INFO:tasks.rados.rados.0.smithi107.stdout:2419: writing smithi10729407-10 from 1843200 to 2342912 tid 1 2020-06-12T02:37:44.247 INFO:tasks.rados.rados.0.smithi107.stdout:2419: writing smithi10729407-10 from 2342912 to 2859008 tid 2 2020-06-12T02:37:44.247 INFO:tasks.rados.rados.0.smithi107.stdout:2419: writing smithi10729407-10 from 2859008 to 3317760 tid 3 2020-06-12T02:37:44.249 INFO:tasks.rados.rados.0.smithi107.stdout:2419: writing smithi10729407-10 from 3317760 to 3948544 tid 4 2020-06-12T02:37:44.250 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:37:44.250 INFO:tasks.rados.rados.0.smithi107.stdout:2420: seq_num 2419 ranges {0=516096,516096=802816,1318912=614400,1933312=188416} 2020-06-12T02:37:44.252 INFO:tasks.rados.rados.0.smithi107.stdout:2420: writing smithi10729407-7 from 0 to 516096 tid 1 2020-06-12T02:37:44.258 INFO:tasks.rados.rados.0.smithi107.stdout:2420: writing smithi10729407-7 from 516096 to 1318912 tid 2 2020-06-12T02:37:44.263 INFO:tasks.rados.rados.0.smithi107.stdout:2420: writing smithi10729407-7 from 1318912 to 1933312 tid 3 2020-06-12T02:37:44.265 INFO:tasks.rados.rados.0.smithi107.stdout:2420: writing smithi10729407-7 from 1933312 to 2121728 tid 4 2020-06-12T02:37:44.266 INFO:tasks.rados.rados.0.smithi107.stdout:2421: delete oid 8 current snap is 0 2020-06-12T02:37:44.266 INFO:tasks.rados.rados.0.smithi107.stdout:2416: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:44.267 INFO:tasks.rados.rados.0.smithi107.stdout:2416: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:44.267 INFO:tasks.rados.rados.0.smithi107.stdout:2416: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:44.268 INFO:tasks.rados.rados.0.smithi107.stdout:2416: finishing write tid 5 to smithi10729407-11 2020-06-12T02:37:44.268 INFO:tasks.rados.rados.0.smithi107.stdout:2416: finishing write tid 7 to smithi10729407-11 2020-06-12T02:37:44.268 INFO:tasks.rados.rados.0.smithi107.stdout:2416: finishing write tid 8 to smithi10729407-11 2020-06-12T02:37:44.268 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 443 (ObjNum 2415 snap 0 seq_num 2415) dirty exists 2020-06-12T02:37:44.268 INFO:tasks.rados.rados.0.smithi107.stdout:2416: left oid 11 (ObjNum 2415 snap 0 seq_num 2415) 2020-06-12T02:37:44.269 INFO:tasks.rados.rados.0.smithi107.stdout:2419: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:44.269 INFO:tasks.rados.rados.0.smithi107.stdout:2419: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:44.269 INFO:tasks.rados.rados.0.smithi107.stdout:2419: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:44.269 INFO:tasks.rados.rados.0.smithi107.stdout:2419: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:44.269 INFO:tasks.rados.rados.0.smithi107.stdout:2419: finishing write tid 6 to smithi10729407-10 2020-06-12T02:37:44.270 INFO:tasks.rados.rados.0.smithi107.stdout:2420: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:44.270 INFO:tasks.rados.rados.0.smithi107.stdout:2419: finishing write tid 7 to smithi10729407-10 2020-06-12T02:37:44.270 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 235 (ObjNum 2418 snap 0 seq_num 2418) dirty exists 2020-06-12T02:37:44.270 INFO:tasks.rados.rados.0.smithi107.stdout:2419: left oid 10 (ObjNum 2418 snap 0 seq_num 2418) 2020-06-12T02:37:44.272 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:37:44.272 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:37:44.272 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:37:44.273 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:37:44.273 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.273 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1206 3.5s0 3:a9510bda:::smithi10729407-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:44.273 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.856688+0000", 2020-06-12T02:37:44.273 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.406225376, 2020-06-12T02:37:44.274 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019116942000000001, 2020-06-12T02:37:44.274 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:44.274 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:44.274 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:44.274 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:44.275 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:44.275 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1206 2020-06-12T02:37:44.275 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.275 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:44.275 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.275 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:44.276 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856688+0000", 2020-06-12T02:37:44.276 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:44.276 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.276 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.276 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:44.277 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856688+0000", 2020-06-12T02:37:44.277 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4355e-05 2020-06-12T02:37:44.277 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.277 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.277 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:44.278 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856713+0000", 2020-06-12T02:37:44.278 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0850000000000001e-06 2020-06-12T02:37:44.278 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.278 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.278 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:44.279 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856716+0000", 2020-06-12T02:37:44.279 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4509999999999997e-06 2020-06-12T02:37:44.279 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.279 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.279 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:44.280 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856722+0000", 2020-06-12T02:37:44.280 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2879999999999999e-06 2020-06-12T02:37:44.280 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.280 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.280 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:44.280 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856724+0000", 2020-06-12T02:37:44.281 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0057507449999999998 2020-06-12T02:37:44.281 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.281 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.281 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:44.281 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.862475+0000", 2020-06-12T02:37:44.282 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5759000000000003e-05 2020-06-12T02:37:44.282 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.282 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.282 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:44.282 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.862571+0000", 2020-06-12T02:37:44.282 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012199018000000001 2020-06-12T02:37:44.283 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.283 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.283 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:44.283 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.874770+0000", 2020-06-12T02:37:44.283 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7260999999999999e-05 2020-06-12T02:37:44.284 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.284 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.284 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:44.284 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.874827+0000", 2020-06-12T02:37:44.284 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00097798000000000004 2020-06-12T02:37:44.284 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.285 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.285 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:44.285 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.875805+0000", 2020-06-12T02:37:44.285 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019116942000000001 2020-06-12T02:37:44.285 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:44.286 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:44.286 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:44.286 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.286 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.286 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1572 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~745472] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-06-12T02:37:44.287 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.886439+0000", 2020-06-12T02:37:44.287 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 10.376474657999999, 2020-06-12T02:37:44.287 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021442571000000001, 2020-06-12T02:37:44.287 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:44.287 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:44.288 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:44.288 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:44.288 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:44.288 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1572 2020-06-12T02:37:44.288 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.289 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:44.289 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.289 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:44.289 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.886439+0000", 2020-06-12T02:37:44.289 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:44.289 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.290 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.290 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:44.290 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.886439+0000", 2020-06-12T02:37:44.290 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00073027599999999997 2020-06-12T02:37:44.290 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.291 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.291 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:44.291 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887169+0000", 2020-06-12T02:37:44.291 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.49e-06 2020-06-12T02:37:44.291 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.292 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.292 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:44.292 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887175+0000", 2020-06-12T02:37:44.292 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7899999999999994e-06 2020-06-12T02:37:44.292 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.292 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.293 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:44.293 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887185+0000", 2020-06-12T02:37:44.293 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.42e-06 2020-06-12T02:37:44.293 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.293 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.294 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:44.294 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887189+0000", 2020-06-12T02:37:44.294 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1619999999999999e-05 2020-06-12T02:37:44.294 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.294 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.294 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:44.295 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887221+0000", 2020-06-12T02:37:44.295 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9720000000000002e-05 2020-06-12T02:37:44.295 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.295 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.295 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:44.296 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887300+0000", 2020-06-12T02:37:44.296 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047354100000000001 2020-06-12T02:37:44.296 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.296 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.296 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:44.296 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887774+0000", 2020-06-12T02:37:44.297 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0030649750000000002 2020-06-12T02:37:44.297 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.297 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.297 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:44.297 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.890839+0000", 2020-06-12T02:37:44.298 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016995321000000001 2020-06-12T02:37:44.298 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.298 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.298 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:44.298 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.907834+0000", 2020-06-12T02:37:44.299 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7417999999999999e-05 2020-06-12T02:37:44.299 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.299 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.299 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:44.299 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.907882+0000", 2020-06-12T02:37:44.300 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021442571000000001 2020-06-12T02:37:44.300 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:44.300 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:44.300 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:44.300 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.300 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.301 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1573 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~712704] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-06-12T02:37:44.301 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.890888+0000", 2020-06-12T02:37:44.301 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 10.372025923000001, 2020-06-12T02:37:44.301 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018868001999999998, 2020-06-12T02:37:44.301 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:44.302 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:44.302 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:44.302 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:44.302 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:44.302 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1573 2020-06-12T02:37:44.302 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.303 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:44.303 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.303 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:44.303 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.890888+0000", 2020-06-12T02:37:44.303 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:44.304 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.304 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.304 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:44.304 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.890888+0000", 2020-06-12T02:37:44.304 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000683605 2020-06-12T02:37:44.304 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.305 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.305 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:44.305 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891571+0000", 2020-06-12T02:37:44.305 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4999999999999999e-06 2020-06-12T02:37:44.305 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.306 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.306 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:44.306 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891577+0000", 2020-06-12T02:37:44.306 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.171e-05 2020-06-12T02:37:44.306 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.306 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.307 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:44.307 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891589+0000", 2020-06-12T02:37:44.307 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7650000000000001e-06 2020-06-12T02:37:44.307 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.307 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.308 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:44.308 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891593+0000", 2020-06-12T02:37:44.308 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6689000000000001e-05 2020-06-12T02:37:44.308 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.309 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.309 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:44.309 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891630+0000", 2020-06-12T02:37:44.309 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9877999999999997e-05 2020-06-12T02:37:44.309 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.309 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.310 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:44.310 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891710+0000", 2020-06-12T02:37:44.310 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053367200000000001 2020-06-12T02:37:44.310 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.310 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.311 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:44.311 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.892244+0000", 2020-06-12T02:37:44.311 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004927196 2020-06-12T02:37:44.311 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.312 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.312 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:44.312 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.897171+0000", 2020-06-12T02:37:44.312 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012496419 2020-06-12T02:37:44.312 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.312 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.313 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:44.313 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.909667+0000", 2020-06-12T02:37:44.313 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8567999999999996e-05 2020-06-12T02:37:44.313 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.313 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.314 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:44.314 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.909756+0000", 2020-06-12T02:37:44.314 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018868001999999998 2020-06-12T02:37:44.314 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:44.314 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:44.315 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:44.315 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.315 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.315 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1576 3.5s0 3:a446366d:::smithi10729407-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e26)", 2020-06-12T02:37:44.315 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.891955+0000", 2020-06-12T02:37:44.316 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 10.370958683, 2020-06-12T02:37:44.316 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019893246999999999, 2020-06-12T02:37:44.316 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:44.316 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:44.316 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:44.317 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:44.317 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:44.317 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1576 2020-06-12T02:37:44.317 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.317 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:44.318 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.318 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:44.318 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891955+0000", 2020-06-12T02:37:44.318 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:44.318 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.319 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.319 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:44.319 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891955+0000", 2020-06-12T02:37:44.319 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2067999999999999e-05 2020-06-12T02:37:44.319 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.320 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.320 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:44.320 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891977+0000", 2020-06-12T02:37:44.320 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.18e-06 2020-06-12T02:37:44.320 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.320 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.321 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:44.321 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891980+0000", 2020-06-12T02:37:44.321 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3629999999999999e-06 2020-06-12T02:37:44.321 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.321 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.322 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:44.322 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891987+0000", 2020-06-12T02:37:44.322 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.181e-06 2020-06-12T02:37:44.322 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.322 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.322 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:44.323 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891989+0000", 2020-06-12T02:37:44.324 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050959120000000002 2020-06-12T02:37:44.324 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.324 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.325 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:44.325 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.897085+0000", 2020-06-12T02:37:44.325 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1089999999999998e-05 2020-06-12T02:37:44.325 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.325 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.326 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:44.326 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.897166+0000", 2020-06-12T02:37:44.326 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013400235 2020-06-12T02:37:44.326 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.326 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.326 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:44.327 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.910566+0000", 2020-06-12T02:37:44.327 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000104981 2020-06-12T02:37:44.327 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.327 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.327 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:44.328 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.910671+0000", 2020-06-12T02:37:44.328 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001177237 2020-06-12T02:37:44.328 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.328 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.328 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:44.328 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.911848+0000", 2020-06-12T02:37:44.329 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019893246999999999 2020-06-12T02:37:44.329 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:44.329 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:44.329 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:44.330 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.330 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.331 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2913 3.5s0 3:a9510bda:::smithi10729407-14:head [append 0~770048,stat] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:44.331 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.070354+0000", 2020-06-12T02:37:44.331 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 7.1925598820000003, 2020-06-12T02:37:44.331 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60690798599999995, 2020-06-12T02:37:44.331 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:44.332 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:44.332 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:44.332 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:44.332 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:44.333 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2913 2020-06-12T02:37:44.333 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.333 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:44.333 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.333 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:44.334 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.070354+0000", 2020-06-12T02:37:44.334 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:44.334 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.334 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.335 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:44.335 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.070354+0000", 2020-06-12T02:37:44.335 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00098478900000000002 2020-06-12T02:37:44.335 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.335 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.336 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:44.336 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071339+0000", 2020-06-12T02:37:44.336 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8349999999999993e-06 2020-06-12T02:37:44.336 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.336 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.337 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:44.337 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071347+0000", 2020-06-12T02:37:44.337 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0749e-05 2020-06-12T02:37:44.337 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.337 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.338 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:44.338 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071358+0000", 2020-06-12T02:37:44.338 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9369999999999997e-06 2020-06-12T02:37:44.338 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.338 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.338 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:44.339 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071362+0000", 2020-06-12T02:37:44.339 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0426e-05 2020-06-12T02:37:44.339 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.339 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.339 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:44.340 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071423+0000", 2020-06-12T02:37:44.340 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4129999999999998e-05 2020-06-12T02:37:44.340 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.341 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.341 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:44.341 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071487+0000", 2020-06-12T02:37:44.341 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053752599999999998 2020-06-12T02:37:44.341 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.342 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.342 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:44.342 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072024+0000", 2020-06-12T02:37:44.342 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036162579999999998 2020-06-12T02:37:44.342 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.343 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.343 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:44.343 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075640+0000", 2020-06-12T02:37:44.343 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60152201999999999 2020-06-12T02:37:44.343 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.343 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.344 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:44.344 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677162+0000", 2020-06-12T02:37:44.344 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9315999999999996e-05 2020-06-12T02:37:44.344 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.344 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.345 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:44.345 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677262+0000", 2020-06-12T02:37:44.345 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60690798599999995 2020-06-12T02:37:44.345 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:44.345 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:44.345 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:44.346 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.346 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.346 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2914 3.5s0 3:a9510bda:::smithi10729407-14:head [append 0~401408] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:44.346 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.072409+0000", 2020-06-12T02:37:44.346 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 7.1905044379999996, 2020-06-12T02:37:44.347 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.61956801100000003, 2020-06-12T02:37:44.347 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:44.347 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:44.347 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:44.347 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:44.347 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:44.348 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2914 2020-06-12T02:37:44.348 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.348 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:44.348 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.348 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:44.349 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072409+0000", 2020-06-12T02:37:44.349 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:44.349 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.349 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.349 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:44.350 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072409+0000", 2020-06-12T02:37:44.350 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000446317 2020-06-12T02:37:44.350 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.351 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.351 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:44.351 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072856+0000", 2020-06-12T02:37:44.351 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2140000000000004e-06 2020-06-12T02:37:44.352 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.352 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.352 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:44.352 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072861+0000", 2020-06-12T02:37:44.352 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8710000000000003e-06 2020-06-12T02:37:44.353 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.353 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.353 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:44.353 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072870+0000", 2020-06-12T02:37:44.353 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2270000000000004e-06 2020-06-12T02:37:44.353 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.354 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.354 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:44.354 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072874+0000", 2020-06-12T02:37:44.354 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024649960000000001 2020-06-12T02:37:44.354 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.355 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.355 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:44.355 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075339+0000", 2020-06-12T02:37:44.355 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9179999999999996e-05 2020-06-12T02:37:44.355 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.355 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.356 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:44.356 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075438+0000", 2020-06-12T02:37:44.356 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60189281400000005 2020-06-12T02:37:44.356 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.356 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.357 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:44.357 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677331+0000", 2020-06-12T02:37:44.357 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012602800000000001 2020-06-12T02:37:44.357 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.357 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.358 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:44.358 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677457+0000", 2020-06-12T02:37:44.358 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00088167699999999998 2020-06-12T02:37:44.358 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.358 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.358 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:44.359 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.678339+0000", 2020-06-12T02:37:44.359 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004312995 2020-06-12T02:37:44.359 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.359 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.359 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:44.359 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.682652+0000", 2020-06-12T02:37:44.360 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0092869140000000003 2020-06-12T02:37:44.360 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.360 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.360 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:44.361 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691938+0000", 2020-06-12T02:37:44.361 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8778e-05 2020-06-12T02:37:44.361 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.362 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.362 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:44.362 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691977+0000", 2020-06-12T02:37:44.362 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.61956801100000003 2020-06-12T02:37:44.362 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:44.363 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:44.363 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:44.363 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.363 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.363 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2915 3.5s0 3:a9510bda:::smithi10729407-14:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:44.364 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.072912+0000", 2020-06-12T02:37:44.364 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 7.1900016979999997, 2020-06-12T02:37:44.364 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.6192204, 2020-06-12T02:37:44.364 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:44.364 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:44.364 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:44.365 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:44.365 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:44.365 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2915 2020-06-12T02:37:44.365 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.366 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:44.366 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.366 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:44.366 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072912+0000", 2020-06-12T02:37:44.366 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:44.367 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.367 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.367 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:44.367 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072912+0000", 2020-06-12T02:37:44.367 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9201e-05 2020-06-12T02:37:44.367 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.368 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.368 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:44.368 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072951+0000", 2020-06-12T02:37:44.368 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5480000000000002e-06 2020-06-12T02:37:44.368 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.369 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.369 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:44.369 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072955+0000", 2020-06-12T02:37:44.369 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3879999999999997e-06 2020-06-12T02:37:44.369 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.369 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.370 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:44.370 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072961+0000", 2020-06-12T02:37:44.370 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6589999999999999e-06 2020-06-12T02:37:44.370 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.370 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.371 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:44.371 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072964+0000", 2020-06-12T02:37:44.372 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025221039999999998 2020-06-12T02:37:44.372 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.372 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.372 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:44.372 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075486+0000", 2020-06-12T02:37:44.373 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5038999999999998e-05 2020-06-12T02:37:44.373 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.373 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.373 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:44.373 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075561+0000", 2020-06-12T02:37:44.373 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60452397499999999 2020-06-12T02:37:44.374 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.374 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.374 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:44.374 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680085+0000", 2020-06-12T02:37:44.374 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1398999999999999e-05 2020-06-12T02:37:44.375 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.375 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.375 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:44.375 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680156+0000", 2020-06-12T02:37:44.375 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000274465 2020-06-12T02:37:44.375 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.376 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.376 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:44.376 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680431+0000", 2020-06-12T02:37:44.376 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025071999999999998 2020-06-12T02:37:44.376 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.377 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.377 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:44.377 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.682938+0000", 2020-06-12T02:37:44.377 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0091116449999999998 2020-06-12T02:37:44.377 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.377 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.378 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:44.378 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692050+0000", 2020-06-12T02:37:44.378 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2776999999999997e-05 2020-06-12T02:37:44.378 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.378 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.379 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:44.379 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692132+0000", 2020-06-12T02:37:44.379 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.6192204 2020-06-12T02:37:44.379 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:44.380 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:44.380 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:44.380 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.380 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.381 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2916 3.5s0 3:a9510bda:::smithi10729407-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:37:44.381 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.072986+0000", 2020-06-12T02:37:44.381 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 7.1899274589999997, 2020-06-12T02:37:44.381 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.62063941099999997, 2020-06-12T02:37:44.382 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:44.382 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:44.382 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:44.383 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:44.383 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:44.383 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2916 2020-06-12T02:37:44.383 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.383 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:44.383 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.384 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:44.384 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072986+0000", 2020-06-12T02:37:44.384 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:44.384 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.384 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.385 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:44.385 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072986+0000", 2020-06-12T02:37:44.385 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2101000000000001e-05 2020-06-12T02:37:44.385 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.385 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.385 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:44.386 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073008+0000", 2020-06-12T02:37:44.386 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1549999999999999e-06 2020-06-12T02:37:44.386 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.386 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.386 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:44.387 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073011+0000", 2020-06-12T02:37:44.387 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2360000000000001e-06 2020-06-12T02:37:44.387 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.387 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.387 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:44.388 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073018+0000", 2020-06-12T02:37:44.388 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2790000000000001e-06 2020-06-12T02:37:44.388 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.388 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.388 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:44.388 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073020+0000", 2020-06-12T02:37:44.389 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025637670000000002 2020-06-12T02:37:44.389 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.389 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.389 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:44.389 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075584+0000", 2020-06-12T02:37:44.390 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1814999999999998e-05 2020-06-12T02:37:44.390 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.390 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.390 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:44.390 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075636+0000", 2020-06-12T02:37:44.390 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60499943199999995 2020-06-12T02:37:44.391 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.391 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.391 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:44.391 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680635+0000", 2020-06-12T02:37:44.392 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0968000000000002e-05 2020-06-12T02:37:44.392 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.392 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.393 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:44.393 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680706+0000", 2020-06-12T02:37:44.393 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011777101999999999 2020-06-12T02:37:44.393 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.393 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.393 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:44.394 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692483+0000", 2020-06-12T02:37:44.394 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5890999999999995e-05 2020-06-12T02:37:44.394 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.394 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.394 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:44.395 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692569+0000", 2020-06-12T02:37:44.395 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001056665 2020-06-12T02:37:44.395 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.395 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.395 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:44.396 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.693626+0000", 2020-06-12T02:37:44.396 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.62063941099999997 2020-06-12T02:37:44.396 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:44.396 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:44.396 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:44.396 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.397 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.397 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2922 3.ds1 3:b983fd28:::smithi10729407-1:head [append 0~466944] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:44.397 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.097192+0000", 2020-06-12T02:37:44.397 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 7.1657220920000002, 2020-06-12T02:37:44.397 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58672644699999998, 2020-06-12T02:37:44.398 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:44.398 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:44.398 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:44.398 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:44.398 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:44.398 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2922 2020-06-12T02:37:44.399 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.399 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:44.399 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.399 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:44.399 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097192+0000", 2020-06-12T02:37:44.400 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:44.400 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.400 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.400 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:44.400 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097192+0000", 2020-06-12T02:37:44.401 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047206100000000002 2020-06-12T02:37:44.401 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.401 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.402 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:44.402 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097664+0000", 2020-06-12T02:37:44.402 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8359999999999999e-06 2020-06-12T02:37:44.403 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.403 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.403 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:44.403 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097668+0000", 2020-06-12T02:37:44.404 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0815999999999999e-05 2020-06-12T02:37:44.404 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.404 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.404 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:44.404 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097679+0000", 2020-06-12T02:37:44.404 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.028e-06 2020-06-12T02:37:44.405 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.405 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.405 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:44.405 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097684+0000", 2020-06-12T02:37:44.405 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3932999999999999e-05 2020-06-12T02:37:44.406 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.406 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.406 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:44.406 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097748+0000", 2020-06-12T02:37:44.406 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2214999999999999e-05 2020-06-12T02:37:44.407 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.407 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.407 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:44.407 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097820+0000", 2020-06-12T02:37:44.407 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00041583800000000003 2020-06-12T02:37:44.407 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.408 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.408 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:44.408 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.098236+0000", 2020-06-12T02:37:44.408 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0020339960000000002 2020-06-12T02:37:44.408 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.409 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.409 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:44.409 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.100270+0000", 2020-06-12T02:37:44.409 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58355657299999997 2020-06-12T02:37:44.409 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.409 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.410 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:44.410 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.683827+0000", 2020-06-12T02:37:44.410 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1150999999999995e-05 2020-06-12T02:37:44.410 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.410 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.411 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:44.411 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.683918+0000", 2020-06-12T02:37:44.411 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58672644699999998 2020-06-12T02:37:44.411 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:44.412 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:44.412 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:44.412 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.413 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.413 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2923 3.ds1 3:b983fd28:::smithi10729407-1:head [append 0~614400] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:44.413 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.100504+0000", 2020-06-12T02:37:44.413 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 7.1624094510000003, 2020-06-12T02:37:44.413 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58853946999999995, 2020-06-12T02:37:44.414 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:44.414 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:44.414 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:44.414 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:44.414 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:44.415 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2923 2020-06-12T02:37:44.415 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.415 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:44.415 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.415 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:44.415 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.100504+0000", 2020-06-12T02:37:44.416 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:44.416 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.416 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.416 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:44.416 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.100504+0000", 2020-06-12T02:37:44.417 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00068395399999999996 2020-06-12T02:37:44.417 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.417 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.417 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:44.417 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101188+0000", 2020-06-12T02:37:44.418 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4050000000000003e-06 2020-06-12T02:37:44.418 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.418 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.418 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:44.418 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101196+0000", 2020-06-12T02:37:44.418 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.328e-05 2020-06-12T02:37:44.419 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.419 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.419 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:44.419 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101209+0000", 2020-06-12T02:37:44.419 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3419999999999996e-06 2020-06-12T02:37:44.419 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.420 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.420 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:44.420 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101215+0000", 2020-06-12T02:37:44.420 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1389000000000005e-05 2020-06-12T02:37:44.420 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.421 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.421 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:44.421 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101277+0000", 2020-06-12T02:37:44.421 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7724000000000004e-05 2020-06-12T02:37:44.422 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.422 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.422 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:44.422 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101344+0000", 2020-06-12T02:37:44.423 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047852499999999999 2020-06-12T02:37:44.423 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.423 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.423 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:44.424 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101823+0000", 2020-06-12T02:37:44.424 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0039255510000000002 2020-06-12T02:37:44.424 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.424 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.424 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:44.425 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105748+0000", 2020-06-12T02:37:44.425 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58321340099999996 2020-06-12T02:37:44.425 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.425 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.425 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:44.425 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.688962+0000", 2020-06-12T02:37:44.426 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1898999999999996e-05 2020-06-12T02:37:44.426 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.426 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.426 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:44.426 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689044+0000", 2020-06-12T02:37:44.427 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58853946999999995 2020-06-12T02:37:44.427 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:44.427 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:44.427 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:44.427 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.428 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.428 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2924 3.ds1 3:b983fd28:::smithi10729407-1:head [append 0~188416] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:44.428 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.101633+0000", 2020-06-12T02:37:44.428 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 7.1612804570000002, 2020-06-12T02:37:44.428 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58758259199999996, 2020-06-12T02:37:44.428 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:44.429 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:44.429 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:44.429 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:44.429 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:44.429 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2924 2020-06-12T02:37:44.430 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.430 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:44.430 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.430 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:44.430 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101633+0000", 2020-06-12T02:37:44.430 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:44.431 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.431 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.431 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:44.431 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101633+0000", 2020-06-12T02:37:44.431 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00018775399999999999 2020-06-12T02:37:44.432 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.432 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.432 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:44.432 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101821+0000", 2020-06-12T02:37:44.433 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7669999999999998e-06 2020-06-12T02:37:44.433 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.433 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.434 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:44.434 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101828+0000", 2020-06-12T02:37:44.434 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2716e-05 2020-06-12T02:37:44.434 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.435 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.435 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:44.435 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101840+0000", 2020-06-12T02:37:44.435 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.215e-06 2020-06-12T02:37:44.435 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.436 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.436 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:44.436 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101846+0000", 2020-06-12T02:37:44.436 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021214250000000001 2020-06-12T02:37:44.436 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.437 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.437 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:44.437 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.103967+0000", 2020-06-12T02:37:44.437 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1018000000000003e-05 2020-06-12T02:37:44.437 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.438 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.438 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:44.438 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.104038+0000", 2020-06-12T02:37:44.438 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000333364 2020-06-12T02:37:44.438 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.439 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.439 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:44.439 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.104371+0000", 2020-06-12T02:37:44.439 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001378546 2020-06-12T02:37:44.439 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.439 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.440 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:44.440 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105750+0000", 2020-06-12T02:37:44.440 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58338272899999999 2020-06-12T02:37:44.440 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.440 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.441 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:44.441 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689133+0000", 2020-06-12T02:37:44.441 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3058000000000003e-05 2020-06-12T02:37:44.441 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.441 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.441 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:44.442 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689216+0000", 2020-06-12T02:37:44.442 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58758259199999996 2020-06-12T02:37:44.442 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:44.442 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:44.442 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:44.443 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.443 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.443 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2925 3.ds1 3:b983fd28:::smithi10729407-1:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:44.444 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.101899+0000", 2020-06-12T02:37:44.444 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 7.1610147919999996, 2020-06-12T02:37:44.444 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58747810899999997, 2020-06-12T02:37:44.444 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:44.444 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:44.445 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:44.445 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:44.445 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:44.445 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2925 2020-06-12T02:37:44.445 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.446 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:44.446 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.446 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:44.446 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101899+0000", 2020-06-12T02:37:44.446 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:44.446 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.447 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.447 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:44.447 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101899+0000", 2020-06-12T02:37:44.447 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2906999999999998e-05 2020-06-12T02:37:44.447 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.448 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.448 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:44.448 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101972+0000", 2020-06-12T02:37:44.448 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1789999999999996e-06 2020-06-12T02:37:44.448 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.448 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.449 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:44.449 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101978+0000", 2020-06-12T02:37:44.449 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1358e-05 2020-06-12T02:37:44.449 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.449 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.450 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:44.450 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101989+0000", 2020-06-12T02:37:44.455 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.031e-06 2020-06-12T02:37:44.455 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.455 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.456 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:44.456 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101993+0000", 2020-06-12T02:37:44.456 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033218990000000001 2020-06-12T02:37:44.456 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.456 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.457 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:44.457 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105315+0000", 2020-06-12T02:37:44.457 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6324999999999999e-05 2020-06-12T02:37:44.457 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.457 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.458 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:44.458 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105382+0000", 2020-06-12T02:37:44.458 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000216624 2020-06-12T02:37:44.458 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.458 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.458 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:44.459 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105598+0000", 2020-06-12T02:37:44.459 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001236586 2020-06-12T02:37:44.459 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.459 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.459 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:44.460 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.106835+0000", 2020-06-12T02:37:44.460 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58245509200000001 2020-06-12T02:37:44.460 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.460 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.460 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:44.460 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689290+0000", 2020-06-12T02:37:44.461 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7107999999999993e-05 2020-06-12T02:37:44.461 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.461 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.462 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:44.462 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689377+0000", 2020-06-12T02:37:44.462 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58747810899999997 2020-06-12T02:37:44.462 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:44.463 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:44.463 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:44.463 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.463 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.463 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2926 3.ds1 3:b983fd28:::smithi10729407-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:37:44.463 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.102036+0000", 2020-06-12T02:37:44.464 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 7.1608778080000004, 2020-06-12T02:37:44.464 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58868048699999997, 2020-06-12T02:37:44.464 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:44.464 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:44.464 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:44.465 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:44.465 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:44.465 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2926 2020-06-12T02:37:44.465 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.466 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:44.466 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.466 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:44.466 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102036+0000", 2020-06-12T02:37:44.466 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:44.466 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.467 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.467 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:44.467 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102036+0000", 2020-06-12T02:37:44.467 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3786999999999999e-05 2020-06-12T02:37:44.467 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.468 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.468 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:44.468 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102080+0000", 2020-06-12T02:37:44.468 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7660000000000004e-06 2020-06-12T02:37:44.468 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.468 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.469 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:44.469 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102085+0000", 2020-06-12T02:37:44.469 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3145e-05 2020-06-12T02:37:44.469 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.469 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.470 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:44.470 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102099+0000", 2020-06-12T02:37:44.470 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2440000000000004e-06 2020-06-12T02:37:44.470 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.470 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.471 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:44.471 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102103+0000", 2020-06-12T02:37:44.471 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036826990000000002 2020-06-12T02:37:44.471 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.471 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.471 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:44.472 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105786+0000", 2020-06-12T02:37:44.472 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.6774000000000006e-05 2020-06-12T02:37:44.472 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.472 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.472 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:44.473 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105882+0000", 2020-06-12T02:37:44.473 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58352084900000001 2020-06-12T02:37:44.473 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.473 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.473 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:44.473 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689403+0000", 2020-06-12T02:37:44.474 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3788999999999995e-05 2020-06-12T02:37:44.474 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.474 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.474 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:44.474 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689467+0000", 2020-06-12T02:37:44.475 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001249434 2020-06-12T02:37:44.475 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.475 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.475 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:44.475 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690716+0000", 2020-06-12T02:37:44.476 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58868048699999997 2020-06-12T02:37:44.476 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:44.476 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:44.476 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:44.476 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.477 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.477 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2950 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~598016] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:44.477 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.162562+0000", 2020-06-12T02:37:44.477 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 7.1003514340000002, 2020-06-12T02:37:44.477 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52780611399999999, 2020-06-12T02:37:44.478 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:44.478 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:44.478 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:44.478 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:44.478 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:44.478 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2950 2020-06-12T02:37:44.479 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.479 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:44.479 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.479 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:44.479 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.162562+0000", 2020-06-12T02:37:44.480 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:44.480 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.480 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.480 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:44.480 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.162562+0000", 2020-06-12T02:37:44.480 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00070718099999999998 2020-06-12T02:37:44.481 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.481 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.481 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:44.481 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163269+0000", 2020-06-12T02:37:44.481 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2029999999999997e-06 2020-06-12T02:37:44.482 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.482 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.482 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:44.482 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163276+0000", 2020-06-12T02:37:44.482 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2734999999999999e-05 2020-06-12T02:37:44.482 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.483 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.483 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:44.483 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163288+0000", 2020-06-12T02:37:44.483 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.404e-06 2020-06-12T02:37:44.483 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.484 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.484 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:44.484 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163295+0000", 2020-06-12T02:37:44.484 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5784000000000006e-05 2020-06-12T02:37:44.484 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.484 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.485 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:44.485 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163361+0000", 2020-06-12T02:37:44.485 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5318e-05 2020-06-12T02:37:44.485 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.485 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.486 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:44.486 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163436+0000", 2020-06-12T02:37:44.487 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054801100000000005 2020-06-12T02:37:44.487 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.487 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.487 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:44.487 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163984+0000", 2020-06-12T02:37:44.488 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.007023077 2020-06-12T02:37:44.488 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.488 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.488 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:44.488 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171007+0000", 2020-06-12T02:37:44.489 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51932051199999996 2020-06-12T02:37:44.489 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.489 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.489 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:44.489 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690327+0000", 2020-06-12T02:37:44.489 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0889000000000001e-05 2020-06-12T02:37:44.490 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.490 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.490 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:44.490 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690368+0000", 2020-06-12T02:37:44.490 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52780611399999999 2020-06-12T02:37:44.491 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:44.491 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:44.491 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:44.491 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.491 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.492 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2951 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~532480] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:44.492 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.165470+0000", 2020-06-12T02:37:44.492 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 7.0974438260000001, 2020-06-12T02:37:44.492 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52507917900000001, 2020-06-12T02:37:44.492 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:44.492 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:44.493 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:44.493 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:44.493 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:44.493 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2951 2020-06-12T02:37:44.493 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.494 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:44.494 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.494 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:44.494 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.165470+0000", 2020-06-12T02:37:44.494 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:44.495 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.495 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.495 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:44.495 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.165470+0000", 2020-06-12T02:37:44.495 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00062231600000000004 2020-06-12T02:37:44.495 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.496 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.496 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:44.496 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166092+0000", 2020-06-12T02:37:44.496 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1610000000000002e-06 2020-06-12T02:37:44.496 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.497 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.497 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:44.497 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166097+0000", 2020-06-12T02:37:44.497 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7780000000000002e-06 2020-06-12T02:37:44.497 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.497 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.498 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:44.498 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166107+0000", 2020-06-12T02:37:44.498 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7169999999999999e-06 2020-06-12T02:37:44.498 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.498 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.499 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:44.499 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166112+0000", 2020-06-12T02:37:44.499 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4792000000000002e-05 2020-06-12T02:37:44.499 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.499 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.499 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:44.500 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166147+0000", 2020-06-12T02:37:44.500 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9336e-05 2020-06-12T02:37:44.500 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.500 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.500 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:44.501 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166236+0000", 2020-06-12T02:37:44.501 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00067747200000000004 2020-06-12T02:37:44.501 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.501 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.501 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:44.502 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166913+0000", 2020-06-12T02:37:44.502 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0047718669999999999 2020-06-12T02:37:44.502 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.502 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.502 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:44.502 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171685+0000", 2020-06-12T02:37:44.503 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51881990700000002 2020-06-12T02:37:44.503 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.503 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.503 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:44.503 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690505+0000", 2020-06-12T02:37:44.503 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3832999999999999e-05 2020-06-12T02:37:44.504 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.504 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.504 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:44.504 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690549+0000", 2020-06-12T02:37:44.504 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52507917900000001 2020-06-12T02:37:44.505 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:44.505 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:44.505 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:44.505 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.505 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.506 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2952 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~270336] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:44.506 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.166891+0000", 2020-06-12T02:37:44.506 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 7.096022177, 2020-06-12T02:37:44.506 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52426417000000003, 2020-06-12T02:37:44.506 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:44.506 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:44.507 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:44.507 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:44.507 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:44.507 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2952 2020-06-12T02:37:44.507 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.508 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:44.508 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.508 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:44.508 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166891+0000", 2020-06-12T02:37:44.508 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:44.508 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.509 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.509 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:44.509 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166891+0000", 2020-06-12T02:37:44.509 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00031878599999999999 2020-06-12T02:37:44.509 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.510 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.510 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:44.510 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167210+0000", 2020-06-12T02:37:44.510 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2360000000000004e-06 2020-06-12T02:37:44.510 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.511 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.511 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:44.511 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167219+0000", 2020-06-12T02:37:44.511 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3298e-05 2020-06-12T02:37:44.511 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.511 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.512 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:44.512 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167232+0000", 2020-06-12T02:37:44.512 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5849999999999999e-06 2020-06-12T02:37:44.512 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.512 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.512 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:44.513 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167237+0000", 2020-06-12T02:37:44.513 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0018755740000000001 2020-06-12T02:37:44.513 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.513 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.513 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:44.514 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.169113+0000", 2020-06-12T02:37:44.514 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000104768 2020-06-12T02:37:44.514 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.514 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.514 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:44.514 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.169218+0000", 2020-06-12T02:37:44.515 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00057389799999999999 2020-06-12T02:37:44.515 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.515 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.515 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:44.515 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.169792+0000", 2020-06-12T02:37:44.516 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0019333989999999999 2020-06-12T02:37:44.516 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.516 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.516 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:44.516 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171725+0000", 2020-06-12T02:37:44.516 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51938508699999997 2020-06-12T02:37:44.517 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.517 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.517 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:44.517 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691110+0000", 2020-06-12T02:37:44.517 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5538999999999999e-05 2020-06-12T02:37:44.518 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.518 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.518 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:44.518 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691156+0000", 2020-06-12T02:37:44.518 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52426417000000003 2020-06-12T02:37:44.518 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:44.519 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:44.519 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:44.519 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.519 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.519 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2953 3.5s0 3:a446366d:::smithi10729407-11:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:44.520 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.167293+0000", 2020-06-12T02:37:44.520 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 7.0956207300000003, 2020-06-12T02:37:44.520 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52428350999999995, 2020-06-12T02:37:44.520 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:44.520 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:44.520 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:44.521 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:44.521 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:44.521 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2953 2020-06-12T02:37:44.521 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.521 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:44.522 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.522 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:44.522 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167293+0000", 2020-06-12T02:37:44.522 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:44.522 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.522 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.523 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:44.523 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167293+0000", 2020-06-12T02:37:44.523 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3880000000000004e-05 2020-06-12T02:37:44.523 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.523 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.524 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:44.524 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167367+0000", 2020-06-12T02:37:44.524 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.319e-06 2020-06-12T02:37:44.524 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.524 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.524 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:44.525 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167373+0000", 2020-06-12T02:37:44.525 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1719000000000001e-05 2020-06-12T02:37:44.525 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.525 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.525 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:44.526 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167385+0000", 2020-06-12T02:37:44.526 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4580000000000003e-06 2020-06-12T02:37:44.526 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.526 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.526 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:44.526 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167389+0000", 2020-06-12T02:37:44.527 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036341110000000002 2020-06-12T02:37:44.527 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.527 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.527 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:44.527 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171023+0000", 2020-06-12T02:37:44.527 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9418000000000005e-05 2020-06-12T02:37:44.528 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.528 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.528 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:44.528 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171093+0000", 2020-06-12T02:37:44.528 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00029087900000000002 2020-06-12T02:37:44.529 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.529 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.530 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:44.530 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171384+0000", 2020-06-12T02:37:44.530 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000489347 2020-06-12T02:37:44.530 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.530 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.531 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:44.531 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171873+0000", 2020-06-12T02:37:44.531 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51963691300000003 2020-06-12T02:37:44.531 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.531 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.531 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:44.532 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691510+0000", 2020-06-12T02:37:44.532 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6465999999999997e-05 2020-06-12T02:37:44.532 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.532 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.532 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:44.533 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691576+0000", 2020-06-12T02:37:44.533 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52428350999999995 2020-06-12T02:37:44.533 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:44.533 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:44.533 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:44.533 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.534 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.534 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2954 3.5s0 3:a446366d:::smithi10729407-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:37:44.534 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.167432+0000", 2020-06-12T02:37:44.534 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 7.0954819809999998, 2020-06-12T02:37:44.534 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52568652699999996, 2020-06-12T02:37:44.535 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:44.535 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:44.535 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:44.535 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:44.535 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:44.535 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2954 2020-06-12T02:37:44.536 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.536 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:44.536 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.536 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:44.536 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167432+0000", 2020-06-12T02:37:44.537 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:44.537 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.537 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.537 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:44.537 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167432+0000", 2020-06-12T02:37:44.537 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3556000000000001e-05 2020-06-12T02:37:44.538 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.538 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.538 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:44.538 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167475+0000", 2020-06-12T02:37:44.538 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7350000000000001e-06 2020-06-12T02:37:44.539 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.539 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.539 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:44.539 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167481+0000", 2020-06-12T02:37:44.539 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3128e-05 2020-06-12T02:37:44.539 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.540 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.540 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:44.540 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167494+0000", 2020-06-12T02:37:44.540 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1579999999999998e-06 2020-06-12T02:37:44.540 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.541 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.541 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:44.541 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167498+0000", 2020-06-12T02:37:44.541 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004082174 2020-06-12T02:37:44.541 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.541 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.542 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:44.542 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171580+0000", 2020-06-12T02:37:44.543 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8954000000000002e-05 2020-06-12T02:37:44.543 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.543 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.543 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:44.544 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171679+0000", 2020-06-12T02:37:44.544 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51993235100000001 2020-06-12T02:37:44.544 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.544 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.544 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:44.545 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691612+0000", 2020-06-12T02:37:44.545 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4329000000000001e-05 2020-06-12T02:37:44.545 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.545 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.545 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:44.546 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691666+0000", 2020-06-12T02:37:44.546 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001452142 2020-06-12T02:37:44.546 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.546 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.546 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:44.547 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.693118+0000", 2020-06-12T02:37:44.547 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52568652699999996 2020-06-12T02:37:44.547 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:44.547 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:44.547 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:44.547 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.548 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.548 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2959 3.ds1 3:b3445631:::smithi10729407-6:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e27)", 2020-06-12T02:37:44.548 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.176248+0000", 2020-06-12T02:37:44.548 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 7.0866652400000003, 2020-06-12T02:37:44.548 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51549877099999997, 2020-06-12T02:37:44.549 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:44.549 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:44.549 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:44.549 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:44.549 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:44.550 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2959 2020-06-12T02:37:44.550 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.550 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:44.550 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.550 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:44.550 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176248+0000", 2020-06-12T02:37:44.551 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:44.551 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.551 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.551 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:44.551 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176248+0000", 2020-06-12T02:37:44.552 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5120999999999999e-05 2020-06-12T02:37:44.552 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.552 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.552 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:44.552 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176274+0000", 2020-06-12T02:37:44.552 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9189999999999998e-06 2020-06-12T02:37:44.553 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.553 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.553 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:44.553 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176278+0000", 2020-06-12T02:37:44.553 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6339e-05 2020-06-12T02:37:44.554 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.554 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.554 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:44.554 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176305+0000", 2020-06-12T02:37:44.554 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9089999999999999e-06 2020-06-12T02:37:44.555 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.555 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.555 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:44.555 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176310+0000", 2020-06-12T02:37:44.555 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6095999999999999e-05 2020-06-12T02:37:44.556 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.556 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.556 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:44.556 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176376+0000", 2020-06-12T02:37:44.556 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8590999999999994e-05 2020-06-12T02:37:44.557 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.557 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.557 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:44.557 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176444+0000", 2020-06-12T02:37:44.557 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00024346400000000001 2020-06-12T02:37:44.558 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.558 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.558 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:44.558 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176688+0000", 2020-06-12T02:37:44.558 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00044220899999999999 2020-06-12T02:37:44.559 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.559 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.559 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:44.559 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.177130+0000", 2020-06-12T02:37:44.559 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51452984400000001 2020-06-12T02:37:44.559 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.560 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.560 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:44.560 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691660+0000", 2020-06-12T02:37:44.560 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7279000000000002e-05 2020-06-12T02:37:44.560 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.561 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.561 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:44.561 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691747+0000", 2020-06-12T02:37:44.561 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51549877099999997 2020-06-12T02:37:44.561 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:44.561 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:44.562 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:44.562 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.562 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.562 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:3032 3.ds1 3:b983fd28:::smithi10729407-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:37:44.562 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.864289+0000", 2020-06-12T02:37:44.563 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 6.3986243219999999, 2020-06-12T02:37:44.563 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019572216999999999, 2020-06-12T02:37:44.563 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:44.563 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:44.563 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:44.563 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:44.564 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:44.564 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 3032 2020-06-12T02:37:44.564 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.564 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:44.564 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.565 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:44.565 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864289+0000", 2020-06-12T02:37:44.565 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:44.565 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.565 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.565 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:44.566 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864289+0000", 2020-06-12T02:37:44.566 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5701999999999997e-05 2020-06-12T02:37:44.566 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.566 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.566 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:44.567 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864335+0000", 2020-06-12T02:37:44.567 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0910000000000004e-06 2020-06-12T02:37:44.567 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.567 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.567 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:44.567 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864342+0000", 2020-06-12T02:37:44.568 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2301e-05 2020-06-12T02:37:44.568 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.568 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.568 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:44.568 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864354+0000", 2020-06-12T02:37:44.569 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5199999999999997e-06 2020-06-12T02:37:44.569 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.569 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.569 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:44.569 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864360+0000", 2020-06-12T02:37:44.569 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00049180999999999997 2020-06-12T02:37:44.570 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.570 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.570 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:44.570 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864852+0000", 2020-06-12T02:37:44.570 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6725000000000003e-05 2020-06-12T02:37:44.571 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.571 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.571 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:44.571 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864908+0000", 2020-06-12T02:37:44.571 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0089687010000000008 2020-06-12T02:37:44.572 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.572 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.572 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:44.572 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.873877+0000", 2020-06-12T02:37:44.572 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3382999999999999e-05 2020-06-12T02:37:44.572 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.573 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.573 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:44.573 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.873941+0000", 2020-06-12T02:37:44.573 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.006408927 2020-06-12T02:37:44.573 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.574 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.574 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:44.574 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.880350+0000", 2020-06-12T02:37:44.574 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3905e-05 2020-06-12T02:37:44.574 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.575 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.575 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:44.575 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.880403+0000", 2020-06-12T02:37:44.575 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0034581519999999999 2020-06-12T02:37:44.575 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:44.576 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:44.576 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:44.576 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.883862+0000", 2020-06-12T02:37:44.576 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019572216999999999 2020-06-12T02:37:44.576 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:44.577 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:44.577 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:44.577 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:44.577 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:44.577 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:37:44.578 INFO:tasks.rados.rados.0.smithi107.stdout:2416: done (3 left) 2020-06-12T02:37:44.578 INFO:tasks.rados.rados.0.smithi107.stdout:2419: done (2 left) 2020-06-12T02:37:44.578 INFO:tasks.rados.rados.0.smithi107.stdout:2421: done (1 left) 2020-06-12T02:37:44.578 INFO:tasks.rados.rados.0.smithi107.stdout:2422: delete oid 8 current snap is 0 2020-06-12T02:37:44.578 INFO:tasks.rados.rados.0.smithi107.stdout:2420: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:44.579 INFO:tasks.rados.rados.0.smithi107.stdout:2420: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:44.579 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd unset nodeep-scrub 2020-06-12T02:37:44.596 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-06-12T02:37:44.597 INFO:tasks.rados.rados.0.smithi107.stdout:2422: done (1 left) 2020-06-12T02:37:44.597 INFO:tasks.rados.rados.0.smithi107.stdout:2423: delete oid 11 current snap is 0 2020-06-12T02:37:44.597 INFO:tasks.rados.rados.0.smithi107.stdout:2423: done (1 left) 2020-06-12T02:37:44.598 INFO:tasks.rados.rados.0.smithi107.stdout:2424: delete oid 3 current snap is 0 2020-06-12T02:37:44.598 INFO:tasks.rados.rados.0.smithi107.stdout:2420: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:44.598 INFO:tasks.rados.rados.0.smithi107.stdout:2420: finishing write tid 6 to smithi10729407-7 2020-06-12T02:37:44.598 INFO:tasks.rados.rados.0.smithi107.stdout:2424: done (1 left) 2020-06-12T02:37:44.598 INFO:tasks.rados.rados.0.smithi107.stdout:2425: delete oid 12 current snap is 0 2020-06-12T02:37:44.599 INFO:tasks.rados.rados.0.smithi107.stdout:2420: finishing write tid 7 to smithi10729407-7 2020-06-12T02:37:44.599 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 466 (ObjNum 2419 snap 0 seq_num 2419) dirty exists 2020-06-12T02:37:44.599 INFO:tasks.rados.rados.0.smithi107.stdout:2420: left oid 7 (ObjNum 2419 snap 0 seq_num 2419) 2020-06-12T02:37:44.599 INFO:tasks.rados.rados.0.smithi107.stdout:2420: done (1 left) 2020-06-12T02:37:44.600 INFO:tasks.rados.rados.0.smithi107.stdout:2425: done (0 left) 2020-06-12T02:37:44.600 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:37:44.600 INFO:tasks.rados.rados.0.smithi107.stdout:2426: seq_num 2425 ranges {0=622592,622592=507904,1130496=606208,1736704=450560,2187264=40960} 2020-06-12T02:37:44.600 INFO:tasks.rados.rados.0.smithi107.stdout:2426: writing smithi10729407-4 from 0 to 622592 tid 1 2020-06-12T02:37:44.600 INFO:tasks.rados.rados.0.smithi107.stdout:2426: writing smithi10729407-4 from 622592 to 1130496 tid 2 2020-06-12T02:37:44.600 INFO:tasks.rados.rados.0.smithi107.stdout:2426: writing smithi10729407-4 from 1130496 to 1736704 tid 3 2020-06-12T02:37:44.601 INFO:tasks.rados.rados.0.smithi107.stdout:2426: writing smithi10729407-4 from 1736704 to 2187264 tid 4 2020-06-12T02:37:44.601 INFO:tasks.rados.rados.0.smithi107.stdout:2426: writing smithi10729407-4 from 2187264 to 2228224 tid 5 2020-06-12T02:37:44.601 INFO:tasks.rados.rados.0.smithi107.stdout:2427: delete oid 1 current snap is 0 2020-06-12T02:37:44.601 INFO:tasks.rados.rados.0.smithi107.stdout:2426: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:44.601 INFO:tasks.rados.rados.0.smithi107.stdout:2427: done (1 left) 2020-06-12T02:37:44.602 INFO:tasks.rados.rados.0.smithi107.stdout:2428: delete oid 16 current snap is 0 2020-06-12T02:37:44.602 INFO:tasks.rados.rados.0.smithi107.stdout:2426: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:44.602 INFO:tasks.rados.rados.0.smithi107.stdout:2426: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:44.602 INFO:tasks.rados.rados.0.smithi107.stdout:2426: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:44.602 INFO:tasks.rados.rados.0.smithi107.stdout:2428: done (1 left) 2020-06-12T02:37:44.602 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:37:44.603 INFO:tasks.rados.rados.0.smithi107.stdout:2429: seq_num 2428 ranges {0=671744,671744=614400,1286144=786432,2072576=204800} 2020-06-12T02:37:44.603 INFO:tasks.rados.rados.0.smithi107.stdout:2429: writing smithi10729407-8 from 0 to 671744 tid 1 2020-06-12T02:37:44.603 INFO:tasks.rados.rados.0.smithi107.stdout:2429: writing smithi10729407-8 from 671744 to 1286144 tid 2 2020-06-12T02:37:44.603 INFO:tasks.rados.rados.0.smithi107.stdout:2429: writing smithi10729407-8 from 1286144 to 2072576 tid 3 2020-06-12T02:37:44.603 INFO:tasks.rados.rados.0.smithi107.stdout:2429: writing smithi10729407-8 from 2072576 to 2277376 tid 4 2020-06-12T02:37:44.604 INFO:tasks.rados.rados.0.smithi107.stdout:2430: delete oid 12 current snap is 0 2020-06-12T02:37:44.604 INFO:tasks.rados.rados.0.smithi107.stdout:2426: finishing write tid 5 to smithi10729407-4 2020-06-12T02:37:44.604 INFO:tasks.rados.rados.0.smithi107.stdout:2426: finishing write tid 7 to smithi10729407-4 2020-06-12T02:37:44.604 INFO:tasks.rados.rados.0.smithi107.stdout:2426: finishing write tid 8 to smithi10729407-4 2020-06-12T02:37:44.604 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 464 (ObjNum 2425 snap 0 seq_num 2425) dirty exists 2020-06-12T02:37:44.605 INFO:tasks.rados.rados.0.smithi107.stdout:2426: left oid 4 (ObjNum 2425 snap 0 seq_num 2425) 2020-06-12T02:37:44.605 INFO:tasks.rados.rados.0.smithi107.stdout:2429: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:44.605 INFO:tasks.rados.rados.0.smithi107.stdout:2429: finishing write tid 2 to smithi10729407-8 2020-06-12T02:37:44.605 INFO:tasks.rados.rados.0.smithi107.stdout:2426: done (2 left) 2020-06-12T02:37:44.605 INFO:tasks.rados.rados.0.smithi107.stdout:2430: done (1 left) 2020-06-12T02:37:44.606 INFO:tasks.rados.rados.0.smithi107.stdout:2431: delete oid 1 current snap is 0 2020-06-12T02:37:44.606 INFO:tasks.rados.rados.0.smithi107.stdout:2431: done (1 left) 2020-06-12T02:37:44.606 INFO:tasks.rados.rados.0.smithi107.stdout:2432: delete oid 7 current snap is 0 2020-06-12T02:37:44.606 INFO:tasks.rados.rados.0.smithi107.stdout:2429: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:44.606 INFO:tasks.rados.rados.0.smithi107.stdout:2432: done (1 left) 2020-06-12T02:37:44.607 INFO:tasks.rados.rados.0.smithi107.stdout:2433: delete oid 7 current snap is 0 2020-06-12T02:37:44.607 INFO:tasks.rados.rados.0.smithi107.stdout:2429: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:44.607 INFO:tasks.rados.rados.0.smithi107.stdout:2429: finishing write tid 6 to smithi10729407-8 2020-06-12T02:37:44.607 INFO:tasks.rados.rados.0.smithi107.stdout:2429: finishing write tid 7 to smithi10729407-8 2020-06-12T02:37:44.607 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 402 (ObjNum 2428 snap 0 seq_num 2428) dirty exists 2020-06-12T02:37:44.607 INFO:tasks.rados.rados.0.smithi107.stdout:2429: left oid 8 (ObjNum 2428 snap 0 seq_num 2428) 2020-06-12T02:37:44.608 INFO:tasks.rados.rados.0.smithi107.stdout:2429: done (1 left) 2020-06-12T02:37:44.608 INFO:tasks.rados.rados.0.smithi107.stdout:2433: done (0 left) 2020-06-12T02:37:44.608 INFO:tasks.rados.rados.0.smithi107.stdout:2434: delete oid 4 current snap is 0 2020-06-12T02:37:44.608 INFO:tasks.rados.rados.0.smithi107.stdout:2434: done (0 left) 2020-06-12T02:37:44.608 INFO:tasks.rados.rados.0.smithi107.stdout:2435: delete oid 5 current snap is 0 2020-06-12T02:37:44.609 INFO:tasks.rados.rados.0.smithi107.stdout:2435: done (0 left) 2020-06-12T02:37:44.609 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:37:44.609 INFO:tasks.rados.rados.0.smithi107.stdout:2436: seq_num 2435 ranges {3948544=786432,4734976=655360,5390336=532480,5922816=417792} 2020-06-12T02:37:44.609 INFO:tasks.rados.rados.0.smithi107.stdout:2436: writing smithi10729407-10 from 3948544 to 4734976 tid 1 2020-06-12T02:37:44.609 INFO:tasks.rados.rados.0.smithi107.stdout:2436: writing smithi10729407-10 from 4734976 to 5390336 tid 2 2020-06-12T02:37:44.609 INFO:tasks.rados.rados.0.smithi107.stdout:2436: writing smithi10729407-10 from 5390336 to 5922816 tid 3 2020-06-12T02:37:44.610 INFO:tasks.rados.rados.0.smithi107.stdout:2436: writing smithi10729407-10 from 5922816 to 6340608 tid 4 2020-06-12T02:37:44.610 INFO:tasks.rados.rados.0.smithi107.stdout:2437: delete oid 13 current snap is 0 2020-06-12T02:37:44.610 INFO:tasks.rados.rados.0.smithi107.stdout:2436: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:44.610 INFO:tasks.rados.rados.0.smithi107.stdout:2436: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:44.610 INFO:tasks.rados.rados.0.smithi107.stdout:2437: done (1 left) 2020-06-12T02:37:44.611 INFO:tasks.rados.rados.0.smithi107.stdout:2438: delete oid 8 current snap is 0 2020-06-12T02:37:44.611 INFO:tasks.rados.rados.0.smithi107.stdout:2436: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:44.611 INFO:tasks.rados.rados.0.smithi107.stdout:2438: done (1 left) 2020-06-12T02:37:44.611 INFO:tasks.rados.rados.0.smithi107.stdout:2439: delete oid 18 current snap is 0 2020-06-12T02:37:44.611 INFO:tasks.rados.rados.0.smithi107.stdout:2436: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:44.611 INFO:tasks.rados.rados.0.smithi107.stdout:2436: finishing write tid 6 to smithi10729407-10 2020-06-12T02:37:44.612 INFO:tasks.rados.rados.0.smithi107.stdout:2436: finishing write tid 7 to smithi10729407-10 2020-06-12T02:37:44.612 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 240 (ObjNum 2435 snap 0 seq_num 2435) dirty exists 2020-06-12T02:37:44.612 INFO:tasks.rados.rados.0.smithi107.stdout:2436: left oid 10 (ObjNum 2435 snap 0 seq_num 2435) 2020-06-12T02:37:44.612 INFO:tasks.rados.rados.0.smithi107.stdout:2436: done (1 left) 2020-06-12T02:37:44.612 INFO:tasks.rados.rados.0.smithi107.stdout:2439: done (0 left) 2020-06-12T02:37:44.613 INFO:tasks.rados.rados.0.smithi107.stdout:2440: delete oid 7 current snap is 0 2020-06-12T02:37:44.613 INFO:tasks.rados.rados.0.smithi107.stdout:2440: done (0 left) 2020-06-12T02:37:44.613 INFO:tasks.rados.rados.0.smithi107.stdout:2441: delete oid 5 current snap is 0 2020-06-12T02:37:44.613 INFO:tasks.rados.rados.0.smithi107.stdout:2441: done (0 left) 2020-06-12T02:37:44.613 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:37:44.613 INFO:tasks.rados.rados.0.smithi107.stdout:2442: seq_num 2441 ranges {1523712=90112} 2020-06-12T02:37:44.614 INFO:tasks.rados.rados.0.smithi107.stdout:2442: writing smithi10729407-2 from 1523712 to 1613824 tid 1 2020-06-12T02:37:44.614 INFO:tasks.rados.rados.0.smithi107.stdout:2443: delete oid 5 current snap is 0 2020-06-12T02:37:44.614 INFO:tasks.rados.rados.0.smithi107.stdout:2443: done (1 left) 2020-06-12T02:37:44.614 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:37:44.614 INFO:tasks.rados.rados.0.smithi107.stdout:2444: seq_num 2443 ranges {0=122880} 2020-06-12T02:37:44.615 INFO:tasks.rados.rados.0.smithi107.stdout:2444: writing smithi10729407-13 from 0 to 122880 tid 1 2020-06-12T02:37:44.615 INFO:tasks.rados.rados.0.smithi107.stdout:2445: delete oid 17 current snap is 0 2020-06-12T02:37:44.615 INFO:tasks.rados.rados.0.smithi107.stdout:2442: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:44.615 INFO:tasks.rados.rados.0.smithi107.stdout:2445: done (2 left) 2020-06-12T02:37:44.615 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:37:44.615 INFO:tasks.rados.rados.0.smithi107.stdout:2446: seq_num 2445 ranges {0=155648} 2020-06-12T02:37:44.616 INFO:tasks.rados.rados.0.smithi107.stdout:2446: writing smithi10729407-20 from 0 to 155648 tid 1 2020-06-12T02:37:44.616 INFO:tasks.rados.rados.0.smithi107.stdout:2447: delete oid 14 current snap is 0 2020-06-12T02:37:44.616 INFO:tasks.rados.rados.0.smithi107.stdout:2442: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:44.616 INFO:tasks.rados.rados.0.smithi107.stdout:2442: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:44.616 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 261 (ObjNum 2441 snap 0 seq_num 2441) dirty exists 2020-06-12T02:37:44.616 INFO:tasks.rados.rados.0.smithi107.stdout:2442: left oid 2 (ObjNum 2441 snap 0 seq_num 2441) 2020-06-12T02:37:44.617 INFO:tasks.rados.rados.0.smithi107.stdout:2444: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:44.617 INFO:tasks.rados.rados.0.smithi107.stdout:2442: done (3 left) 2020-06-12T02:37:44.617 INFO:tasks.rados.rados.0.smithi107.stdout:2447: done (2 left) 2020-06-12T02:37:44.617 INFO:tasks.rados.rados.0.smithi107.stdout:2448: delete oid 15 current snap is 0 2020-06-12T02:37:44.617 INFO:tasks.rados.rados.0.smithi107.stdout:2444: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:44.618 INFO:tasks.rados.rados.0.smithi107.stdout:2446: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:44.618 INFO:tasks.rados.rados.0.smithi107.stdout:2444: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:44.618 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 472 (ObjNum 2443 snap 0 seq_num 2443) dirty exists 2020-06-12T02:37:44.618 INFO:tasks.rados.rados.0.smithi107.stdout:2444: left oid 13 (ObjNum 2443 snap 0 seq_num 2443) 2020-06-12T02:37:44.618 INFO:tasks.rados.rados.0.smithi107.stdout:2444: done (2 left) 2020-06-12T02:37:44.618 INFO:tasks.rados.rados.0.smithi107.stdout:2448: done (1 left) 2020-06-12T02:37:44.619 INFO:tasks.rados.rados.0.smithi107.stdout:2449: delete oid 3 current snap is 0 2020-06-12T02:37:44.619 INFO:tasks.rados.rados.0.smithi107.stdout:2446: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:44.619 INFO:tasks.rados.rados.0.smithi107.stdout:2446: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:44.619 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 154 (ObjNum 2445 snap 0 seq_num 2445) dirty exists 2020-06-12T02:37:44.619 INFO:tasks.rados.rados.0.smithi107.stdout:2446: left oid 20 (ObjNum 2445 snap 0 seq_num 2445) 2020-06-12T02:37:44.620 INFO:tasks.rados.rados.0.smithi107.stdout:2446: done (1 left) 2020-06-12T02:37:44.620 INFO:tasks.rados.rados.0.smithi107.stdout:2449: done (0 left) 2020-06-12T02:37:44.620 INFO:tasks.rados.rados.0.smithi107.stdout:2450: delete oid 11 current snap is 0 2020-06-12T02:37:44.620 INFO:tasks.rados.rados.0.smithi107.stdout:2450: done (0 left) 2020-06-12T02:37:44.620 INFO:tasks.rados.rados.0.smithi107.stdout:2451: delete oid 12 current snap is 0 2020-06-12T02:37:44.620 INFO:tasks.rados.rados.0.smithi107.stdout:2451: done (0 left) 2020-06-12T02:37:44.621 INFO:tasks.rados.rados.0.smithi107.stdout:2452: delete oid 3 current snap is 0 2020-06-12T02:37:44.621 INFO:tasks.rados.rados.0.smithi107.stdout:2452: done (0 left) 2020-06-12T02:37:44.621 INFO:tasks.rados.rados.0.smithi107.stdout:2453: delete oid 4 current snap is 0 2020-06-12T02:37:44.621 INFO:tasks.rados.rados.0.smithi107.stdout:2453: done (0 left) 2020-06-12T02:37:44.621 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:37:44.622 INFO:tasks.rados.rados.0.smithi107.stdout:2454: seq_num 2453 ranges {0=286720} 2020-06-12T02:37:44.622 INFO:tasks.rados.rados.0.smithi107.stdout:2454: writing smithi10729407-17 from 0 to 286720 tid 1 2020-06-12T02:37:44.622 INFO:tasks.rados.rados.0.smithi107.stdout:2455: delete oid 15 current snap is 0 2020-06-12T02:37:44.622 INFO:tasks.rados.rados.0.smithi107.stdout:2455: done (1 left) 2020-06-12T02:37:44.622 INFO:tasks.rados.rados.0.smithi107.stdout:2456: delete oid 12 current snap is 0 2020-06-12T02:37:44.622 INFO:tasks.rados.rados.0.smithi107.stdout:2454: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:44.623 INFO:tasks.rados.rados.0.smithi107.stdout:2456: done (1 left) 2020-06-12T02:37:44.623 INFO:tasks.rados.rados.0.smithi107.stdout:2457: delete oid 13 current snap is 0 2020-06-12T02:37:44.623 INFO:tasks.rados.rados.0.smithi107.stdout:2457: done (1 left) 2020-06-12T02:37:44.623 INFO:tasks.rados.rados.0.smithi107.stdout:2458: delete oid 15 current snap is 0 2020-06-12T02:37:44.623 INFO:tasks.rados.rados.0.smithi107.stdout:2454: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:44.623 INFO:tasks.rados.rados.0.smithi107.stdout:2454: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:44.624 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 469 (ObjNum 2453 snap 0 seq_num 2453) dirty exists 2020-06-12T02:37:44.624 INFO:tasks.rados.rados.0.smithi107.stdout:2454: left oid 17 (ObjNum 2453 snap 0 seq_num 2453) 2020-06-12T02:37:44.624 INFO:tasks.rados.rados.0.smithi107.stdout:2454: done (1 left) 2020-06-12T02:37:44.624 INFO:tasks.rados.rados.0.smithi107.stdout:2458: done (0 left) 2020-06-12T02:37:44.624 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:37:44.625 INFO:tasks.rados.rados.0.smithi107.stdout:2459: seq_num 2458 ranges {0=368640} 2020-06-12T02:37:44.625 INFO:tasks.rados.rados.0.smithi107.stdout:2459: writing smithi10729407-6 from 0 to 368640 tid 1 2020-06-12T02:37:44.625 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:37:44.625 INFO:tasks.rados.rados.0.smithi107.stdout:2460: seq_num 2459 ranges {0=385024} 2020-06-12T02:37:44.625 INFO:tasks.rados.rados.0.smithi107.stdout:2460: writing smithi10729407-11 from 0 to 385024 tid 1 2020-06-12T02:37:44.625 INFO:tasks.rados.rados.0.smithi107.stdout:2461: delete oid 2 current snap is 0 2020-06-12T02:37:44.626 INFO:tasks.rados.rados.0.smithi107.stdout:2461: done (2 left) 2020-06-12T02:37:44.626 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:37:44.626 INFO:tasks.rados.rados.0.smithi107.stdout:2462: seq_num 2461 ranges {0=425984} 2020-06-12T02:37:44.626 INFO:tasks.rados.rados.0.smithi107.stdout:2462: writing smithi10729407-16 from 0 to 425984 tid 1 2020-06-12T02:37:44.626 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:37:44.627 INFO:tasks.rados.rados.0.smithi107.stdout:2463: seq_num 2462 ranges {0=442368} 2020-06-12T02:37:44.627 INFO:tasks.rados.rados.0.smithi107.stdout:2463: writing smithi10729407-3 from 0 to 442368 tid 1 2020-06-12T02:37:44.627 INFO:tasks.rados.rados.0.smithi107.stdout:2464: delete oid 17 current snap is 0 2020-06-12T02:37:44.627 INFO:tasks.rados.rados.0.smithi107.stdout:2459: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:44.627 INFO:tasks.rados.rados.0.smithi107.stdout:2460: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:44.628 INFO:tasks.rados.rados.0.smithi107.stdout:2459: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:44.628 INFO:tasks.rados.rados.0.smithi107.stdout:2459: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:44.628 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 506 (ObjNum 2458 snap 0 seq_num 2458) dirty exists 2020-06-12T02:37:44.628 INFO:tasks.rados.rados.0.smithi107.stdout:2459: left oid 6 (ObjNum 2458 snap 0 seq_num 2458) 2020-06-12T02:37:44.629 INFO:tasks.rados.rados.0.smithi107.stdout:2460: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:44.629 INFO:tasks.rados.rados.0.smithi107.stdout:2460: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:44.629 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 448 (ObjNum 2459 snap 0 seq_num 2459) dirty exists 2020-06-12T02:37:44.629 INFO:tasks.rados.rados.0.smithi107.stdout:2460: left oid 11 (ObjNum 2459 snap 0 seq_num 2459) 2020-06-12T02:37:44.629 INFO:tasks.rados.rados.0.smithi107.stdout:2459: done (4 left) 2020-06-12T02:37:44.630 INFO:tasks.rados.rados.0.smithi107.stdout:2460: done (3 left) 2020-06-12T02:37:44.630 INFO:tasks.rados.rados.0.smithi107.stdout:2464: done (2 left) 2020-06-12T02:37:44.630 INFO:tasks.rados.rados.0.smithi107.stdout:2465: delete oid 4 current snap is 0 2020-06-12T02:37:44.630 INFO:tasks.rados.rados.0.smithi107.stdout:2462: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:44.630 INFO:tasks.rados.rados.0.smithi107.stdout:2463: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:44.630 INFO:tasks.rados.rados.0.smithi107.stdout:2465: done (2 left) 2020-06-12T02:37:44.631 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:44.631 INFO:tasks.rados.rados.0.smithi107.stdout:2466: seq_num 2465 ranges {0=491520} 2020-06-12T02:37:44.631 INFO:tasks.rados.rados.0.smithi107.stdout:2466: writing smithi10729407-2 from 0 to 491520 tid 1 2020-06-12T02:37:44.631 INFO:tasks.rados.rados.0.smithi107.stdout:2467: delete oid 7 current snap is 0 2020-06-12T02:37:44.631 INFO:tasks.rados.rados.0.smithi107.stdout:2463: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:44.632 INFO:tasks.rados.rados.0.smithi107.stdout:2462: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:44.632 INFO:tasks.rados.rados.0.smithi107.stdout:2463: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:44.632 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 424 (ObjNum 2462 snap 0 seq_num 2462) dirty exists 2020-06-12T02:37:44.632 INFO:tasks.rados.rados.0.smithi107.stdout:2463: left oid 3 (ObjNum 2462 snap 0 seq_num 2462) 2020-06-12T02:37:44.632 INFO:tasks.rados.rados.0.smithi107.stdout:2462: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:44.633 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 425 (ObjNum 2461 snap 0 seq_num 2461) dirty exists 2020-06-12T02:37:44.633 INFO:tasks.rados.rados.0.smithi107.stdout:2462: left oid 16 (ObjNum 2461 snap 0 seq_num 2461) 2020-06-12T02:37:44.633 INFO:tasks.rados.rados.0.smithi107.stdout:2462: done (3 left) 2020-06-12T02:37:44.633 INFO:tasks.rados.rados.0.smithi107.stdout:2463: done (2 left) 2020-06-12T02:37:44.633 INFO:tasks.rados.rados.0.smithi107.stdout:2467: done (1 left) 2020-06-12T02:37:44.633 INFO:tasks.rados.rados.0.smithi107.stdout:2468: delete oid 15 current snap is 0 2020-06-12T02:37:44.634 INFO:tasks.rados.rados.0.smithi107.stdout:2466: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:44.634 INFO:tasks.rados.rados.0.smithi107.stdout:2468: done (1 left) 2020-06-12T02:37:44.634 INFO:tasks.rados.rados.0.smithi107.stdout:2469: delete oid 16 current snap is 0 2020-06-12T02:37:44.634 INFO:tasks.rados.rados.0.smithi107.stdout:2466: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:44.634 INFO:tasks.rados.rados.0.smithi107.stdout:2466: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:44.635 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 264 (ObjNum 2465 snap 0 seq_num 2465) dirty exists 2020-06-12T02:37:44.635 INFO:tasks.rados.rados.0.smithi107.stdout:2466: left oid 2 (ObjNum 2465 snap 0 seq_num 2465) 2020-06-12T02:37:44.635 INFO:tasks.rados.rados.0.smithi107.stdout:2466: done (1 left) 2020-06-12T02:37:44.635 INFO:tasks.rados.rados.0.smithi107.stdout:2469: done (0 left) 2020-06-12T02:37:44.635 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:37:44.636 INFO:tasks.rados.rados.0.smithi107.stdout:2470: seq_num 2469 ranges {0=557056} 2020-06-12T02:37:44.636 INFO:tasks.rados.rados.0.smithi107.stdout:2470: writing smithi10729407-9 from 0 to 557056 tid 1 2020-06-12T02:37:44.636 INFO:tasks.rados.rados.0.smithi107.stdout:2471: delete oid 7 current snap is 0 2020-06-12T02:37:44.636 INFO:tasks.rados.rados.0.smithi107.stdout:2471: done (1 left) 2020-06-12T02:37:44.636 INFO:tasks.rados.rados.0.smithi107.stdout:2472: delete oid 3 current snap is 0 2020-06-12T02:37:44.636 INFO:tasks.rados.rados.0.smithi107.stdout:2472: done (1 left) 2020-06-12T02:37:44.637 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:37:44.637 INFO:tasks.rados.rados.0.smithi107.stdout:2473: seq_num 2472 ranges {0=606208} 2020-06-12T02:37:44.637 INFO:tasks.rados.rados.0.smithi107.stdout:2473: writing smithi10729407-12 from 0 to 606208 tid 1 2020-06-12T02:37:44.637 INFO:tasks.rados.rados.0.smithi107.stdout:2474: delete oid 4 current snap is 0 2020-06-12T02:37:44.637 INFO:tasks.rados.rados.0.smithi107.stdout:2470: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:44.638 INFO:tasks.rados.rados.0.smithi107.stdout:2470: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:44.638 INFO:tasks.rados.rados.0.smithi107.stdout:2470: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:44.638 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 414 (ObjNum 2469 snap 0 seq_num 2469) dirty exists 2020-06-12T02:37:44.638 INFO:tasks.rados.rados.0.smithi107.stdout:2470: left oid 9 (ObjNum 2469 snap 0 seq_num 2469) 2020-06-12T02:37:44.638 INFO:tasks.rados.rados.0.smithi107.stdout:2470: done (2 left) 2020-06-12T02:37:44.638 INFO:tasks.rados.rados.0.smithi107.stdout:2474: done (1 left) 2020-06-12T02:37:44.639 INFO:tasks.rados.rados.0.smithi107.stdout:2475: delete oid 1 current snap is 0 2020-06-12T02:37:44.639 INFO:tasks.rados.rados.0.smithi107.stdout:2475: done (1 left) 2020-06-12T02:37:44.639 INFO:tasks.rados.rados.0.smithi107.stdout:2476: delete oid 10 current snap is 0 2020-06-12T02:37:44.639 INFO:tasks.rados.rados.0.smithi107.stdout:2473: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:44.639 INFO:tasks.rados.rados.0.smithi107.stdout:2473: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:44.640 INFO:tasks.rados.rados.0.smithi107.stdout:2476: done (1 left) 2020-06-12T02:37:44.640 INFO:tasks.rados.rados.0.smithi107.stdout:2477: delete oid 5 current snap is 0 2020-06-12T02:37:44.640 INFO:tasks.rados.rados.0.smithi107.stdout:2473: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:44.640 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 384 (ObjNum 2472 snap 0 seq_num 2472) dirty exists 2020-06-12T02:37:44.640 INFO:tasks.rados.rados.0.smithi107.stdout:2473: left oid 12 (ObjNum 2472 snap 0 seq_num 2472) 2020-06-12T02:37:44.640 INFO:tasks.rados.rados.0.smithi107.stdout:2473: done (1 left) 2020-06-12T02:37:44.641 INFO:tasks.rados.rados.0.smithi107.stdout:2477: done (0 left) 2020-06-12T02:37:44.641 INFO:tasks.rados.rados.0.smithi107.stdout:2478: delete oid 14 current snap is 0 2020-06-12T02:37:44.641 INFO:tasks.rados.rados.0.smithi107.stdout:2478: done (0 left) 2020-06-12T02:37:44.641 INFO:tasks.rados.rados.0.smithi107.stdout:2479: delete oid 6 current snap is 0 2020-06-12T02:37:44.641 INFO:tasks.rados.rados.0.smithi107.stdout:2479: done (0 left) 2020-06-12T02:37:44.642 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:37:44.642 INFO:tasks.rados.rados.0.smithi107.stdout:2480: seq_num 2479 ranges {0=729088} 2020-06-12T02:37:44.642 INFO:tasks.rados.rados.0.smithi107.stdout:2480: writing smithi10729407-16 from 0 to 729088 tid 1 2020-06-12T02:37:44.642 INFO:tasks.rados.rados.0.smithi107.stdout:2481: delete oid 12 current snap is 0 2020-06-12T02:37:44.642 INFO:tasks.rados.rados.0.smithi107.stdout:2481: done (1 left) 2020-06-12T02:37:44.642 INFO:tasks.rados.rados.0.smithi107.stdout:2482: delete oid 1 current snap is 0 2020-06-12T02:37:44.643 INFO:tasks.rados.rados.0.smithi107.stdout:2482: done (1 left) 2020-06-12T02:37:44.643 INFO:tasks.rados.rados.0.smithi107.stdout:2483: delete oid 7 current snap is 0 2020-06-12T02:37:44.643 INFO:tasks.rados.rados.0.smithi107.stdout:2480: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:44.643 INFO:tasks.rados.rados.0.smithi107.stdout:2483: done (1 left) 2020-06-12T02:37:44.643 INFO:tasks.rados.rados.0.smithi107.stdout:2484: delete oid 20 current snap is 0 2020-06-12T02:37:44.644 INFO:tasks.rados.rados.0.smithi107.stdout:2480: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:44.644 INFO:tasks.rados.rados.0.smithi107.stdout:2480: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:44.644 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 429 (ObjNum 2479 snap 0 seq_num 2479) dirty exists 2020-06-12T02:37:44.644 INFO:tasks.rados.rados.0.smithi107.stdout:2480: left oid 16 (ObjNum 2479 snap 0 seq_num 2479) 2020-06-12T02:37:44.644 INFO:tasks.rados.rados.0.smithi107.stdout:2480: done (1 left) 2020-06-12T02:37:44.644 INFO:tasks.rados.rados.0.smithi107.stdout:2484: done (0 left) 2020-06-12T02:37:44.645 INFO:tasks.rados.rados.0.smithi107.stdout:2485: delete oid 16 current snap is 0 2020-06-12T02:37:44.645 INFO:tasks.rados.rados.0.smithi107.stdout:2485: done (0 left) 2020-06-12T02:37:44.645 INFO:tasks.rados.rados.0.smithi107.stdout:2486: delete oid 10 current snap is 0 2020-06-12T02:37:44.645 INFO:tasks.rados.rados.0.smithi107.stdout:2486: done (0 left) 2020-06-12T02:37:44.645 INFO:tasks.rados.rados.0.smithi107.stdout:2487: delete oid 12 current snap is 0 2020-06-12T02:37:44.646 INFO:tasks.rados.rados.0.smithi107.stdout:2487: done (0 left) 2020-06-12T02:37:44.646 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:37:44.646 INFO:tasks.rados.rados.0.smithi107.stdout:2488: seq_num 2487 ranges {0=458752,458752=401408} 2020-06-12T02:37:44.646 INFO:tasks.rados.rados.0.smithi107.stdout:2488: writing smithi10729407-7 from 0 to 458752 tid 1 2020-06-12T02:37:44.646 INFO:tasks.rados.rados.0.smithi107.stdout:2488: writing smithi10729407-7 from 458752 to 860160 tid 2 2020-06-12T02:37:44.646 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:37:44.647 INFO:tasks.rados.rados.0.smithi107.stdout:2489: seq_num 2488 ranges {0=475136,475136=401408} 2020-06-12T02:37:44.647 INFO:tasks.rados.rados.0.smithi107.stdout:2489: writing smithi10729407-5 from 0 to 475136 tid 1 2020-06-12T02:37:44.647 INFO:tasks.rados.rados.0.smithi107.stdout:2489: writing smithi10729407-5 from 475136 to 876544 tid 2 2020-06-12T02:37:44.647 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:37:44.647 INFO:tasks.rados.rados.0.smithi107.stdout:2490: seq_num 2489 ranges {0=491520,491520=401408} 2020-06-12T02:37:44.648 INFO:tasks.rados.rados.0.smithi107.stdout:2490: writing smithi10729407-3 from 0 to 491520 tid 1 2020-06-12T02:37:44.648 INFO:tasks.rados.rados.0.smithi107.stdout:2490: writing smithi10729407-3 from 491520 to 892928 tid 2 2020-06-12T02:37:44.648 INFO:tasks.rados.rados.0.smithi107.stdout:2491: delete oid 1 current snap is 0 2020-06-12T02:37:44.648 INFO:tasks.rados.rados.0.smithi107.stdout:2488: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:44.648 INFO:tasks.rados.rados.0.smithi107.stdout:2488: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:44.648 INFO:tasks.rados.rados.0.smithi107.stdout:2488: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:44.649 INFO:tasks.rados.rados.0.smithi107.stdout:2488: finishing write tid 5 to smithi10729407-7 2020-06-12T02:37:44.649 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 479 (ObjNum 2487 snap 0 seq_num 2487) dirty exists 2020-06-12T02:37:44.649 INFO:tasks.rados.rados.0.smithi107.stdout:2488: left oid 7 (ObjNum 2487 snap 0 seq_num 2487) 2020-06-12T02:37:44.649 INFO:tasks.rados.rados.0.smithi107.stdout:2489: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:44.649 INFO:tasks.rados.rados.0.smithi107.stdout:2489: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:44.650 INFO:tasks.rados.rados.0.smithi107.stdout:2488: done (3 left) 2020-06-12T02:37:44.650 INFO:tasks.rados.rados.0.smithi107.stdout:2491: done (2 left) 2020-06-12T02:37:44.650 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:37:44.650 INFO:tasks.rados.rados.0.smithi107.stdout:2492: seq_num 2491 ranges {0=524288,524288=401408} 2020-06-12T02:37:44.650 INFO:tasks.rados.rados.0.smithi107.stdout:2492: writing smithi10729407-16 from 0 to 524288 tid 1 2020-06-12T02:37:44.650 INFO:tasks.rados.rados.0.smithi107.stdout:2492: writing smithi10729407-16 from 524288 to 925696 tid 2 2020-06-12T02:37:44.651 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:37:44.651 INFO:tasks.rados.rados.0.smithi107.stdout:2493: seq_num 2492 ranges {0=540672,540672=401408} 2020-06-12T02:37:44.651 INFO:tasks.rados.rados.0.smithi107.stdout:2493: writing smithi10729407-1 from 0 to 540672 tid 1 2020-06-12T02:37:44.651 INFO:tasks.rados.rados.0.smithi107.stdout:2493: writing smithi10729407-1 from 540672 to 942080 tid 2 2020-06-12T02:37:44.651 INFO:tasks.rados.rados.0.smithi107.stdout:2494: delete oid 15 current snap is 0 2020-06-12T02:37:44.652 INFO:tasks.rados.rados.0.smithi107.stdout:2489: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:44.652 INFO:tasks.rados.rados.0.smithi107.stdout:2490: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:44.652 INFO:tasks.rados.rados.0.smithi107.stdout:2489: finishing write tid 5 to smithi10729407-5 2020-06-12T02:37:44.652 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 206 (ObjNum 2488 snap 0 seq_num 2488) dirty exists 2020-06-12T02:37:44.652 INFO:tasks.rados.rados.0.smithi107.stdout:2489: left oid 5 (ObjNum 2488 snap 0 seq_num 2488) 2020-06-12T02:37:44.652 INFO:tasks.rados.rados.0.smithi107.stdout:2490: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:44.653 INFO:tasks.rados.rados.0.smithi107.stdout:2490: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:44.653 INFO:tasks.rados.rados.0.smithi107.stdout:2490: finishing write tid 5 to smithi10729407-3 2020-06-12T02:37:44.653 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 433 (ObjNum 2489 snap 0 seq_num 2489) dirty exists 2020-06-12T02:37:44.653 INFO:tasks.rados.rados.0.smithi107.stdout:2490: left oid 3 (ObjNum 2489 snap 0 seq_num 2489) 2020-06-12T02:37:44.653 INFO:tasks.rados.rados.0.smithi107.stdout:2492: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:44.654 INFO:tasks.rados.rados.0.smithi107.stdout:2489: done (4 left) 2020-06-12T02:37:44.654 INFO:tasks.rados.rados.0.smithi107.stdout:2490: done (3 left) 2020-06-12T02:37:44.659 INFO:tasks.rados.rados.0.smithi107.stdout:2494: done (2 left) 2020-06-12T02:37:44.660 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:37:44.660 INFO:tasks.rados.rados.0.smithi107.stdout:2495: seq_num 2494 ranges {0=573440,573440=401408} 2020-06-12T02:37:44.660 INFO:tasks.rados.rados.0.smithi107.stdout:2495: writing smithi10729407-15 from 0 to 573440 tid 1 2020-06-12T02:37:44.660 INFO:tasks.rados.rados.0.smithi107.stdout:2495: writing smithi10729407-15 from 573440 to 974848 tid 2 2020-06-12T02:37:44.660 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:37:44.660 INFO:tasks.rados.rados.0.smithi107.stdout:2496: seq_num 2495 ranges {3571712=589824,4161536=401408} 2020-06-12T02:37:44.661 INFO:tasks.rados.rados.0.smithi107.stdout:2496: writing smithi10729407-19 from 3571712 to 4161536 tid 1 2020-06-12T02:37:44.661 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:37:44.661 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:37:44.661 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:37:44.662 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:37:44.662 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:37:44.662 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:37:44.662 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:37:44.662 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-06-12T02:37:44.663 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-06-12T02:37:44.663 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-06-12T02:37:44.663 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-06-12T02:37:44.663 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-06-12T02:37:44.663 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-06-12T02:37:44.664 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-06-12T02:37:44.664 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.7 2020-06-12T02:37:44.664 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: problem getting command descriptions from osd.7 2020-06-12T02:37:44.664 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-06-12T02:37:44.664 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-06-12T02:37:44.670 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:37:44.578+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:37:44.670 INFO:tasks.rados.rados.0.smithi107.stdout:2496: writing smithi10729407-19 from 4161536 to 4562944 tid 2 2020-06-12T02:37:44.671 INFO:tasks.rados.rados.0.smithi107.stdout:2497: delete oid 13 current snap is 0 2020-06-12T02:37:44.671 INFO:tasks.rados.rados.0.smithi107.stdout:2493: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:44.671 INFO:tasks.rados.rados.0.smithi107.stdout:2492: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:44.671 INFO:tasks.rados.rados.0.smithi107.stdout:2493: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:44.671 INFO:tasks.rados.rados.0.smithi107.stdout:2493: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:44.672 INFO:tasks.rados.rados.0.smithi107.stdout:2492: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:44.672 INFO:tasks.rados.rados.0.smithi107.stdout:2495: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:44.672 INFO:tasks.rados.rados.0.smithi107.stdout:2493: finishing write tid 5 to smithi10729407-1 2020-06-12T02:37:44.672 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 513 (ObjNum 2492 snap 0 seq_num 2492) dirty exists 2020-06-12T02:37:44.672 INFO:tasks.rados.rados.0.smithi107.stdout:2493: left oid 1 (ObjNum 2492 snap 0 seq_num 2492) 2020-06-12T02:37:44.673 INFO:tasks.rados.rados.0.smithi107.stdout:2492: finishing write tid 5 to smithi10729407-16 2020-06-12T02:37:44.673 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 436 (ObjNum 2491 snap 0 seq_num 2491) dirty exists 2020-06-12T02:37:44.673 INFO:tasks.rados.rados.0.smithi107.stdout:2492: left oid 16 (ObjNum 2491 snap 0 seq_num 2491) 2020-06-12T02:37:44.673 INFO:tasks.rados.rados.0.smithi107.stdout:2495: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:44.673 INFO:tasks.rados.rados.0.smithi107.stdout:2495: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:44.673 INFO:tasks.rados.rados.0.smithi107.stdout:2495: finishing write tid 5 to smithi10729407-15 2020-06-12T02:37:44.674 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 411 (ObjNum 2494 snap 0 seq_num 2494) dirty exists 2020-06-12T02:37:44.674 INFO:tasks.rados.rados.0.smithi107.stdout:2495: left oid 15 (ObjNum 2494 snap 0 seq_num 2494) 2020-06-12T02:37:44.674 INFO:tasks.rados.rados.0.smithi107.stdout:2492: done (4 left) 2020-06-12T02:37:44.674 INFO:tasks.rados.rados.0.smithi107.stdout:2493: done (3 left) 2020-06-12T02:37:44.674 INFO:tasks.rados.rados.0.smithi107.stdout:2495: done (2 left) 2020-06-12T02:37:44.675 INFO:tasks.rados.rados.0.smithi107.stdout:2497: done (1 left) 2020-06-12T02:37:44.675 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:37:44.675 INFO:tasks.rados.rados.0.smithi107.stdout:2498: seq_num 2497 ranges {557056=622592,1179648=401408} 2020-06-12T02:37:44.675 INFO:tasks.rados.rados.0.smithi107.stdout:2498: writing smithi10729407-9 from 557056 to 1179648 tid 1 2020-06-12T02:37:44.675 INFO:tasks.rados.rados.0.smithi107.stdout:2498: writing smithi10729407-9 from 1179648 to 1581056 tid 2 2020-06-12T02:37:44.676 INFO:tasks.rados.rados.0.smithi107.stdout:2499: delete oid 5 current snap is 0 2020-06-12T02:37:44.676 INFO:tasks.rados.rados.0.smithi107.stdout:2496: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:44.676 INFO:tasks.rados.rados.0.smithi107.stdout:2496: finishing write tid 2 to smithi10729407-19 2020-06-12T02:37:44.676 INFO:tasks.rados.rados.0.smithi107.stdout:2496: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:44.677 INFO:tasks.rados.rados.0.smithi107.stdout:2496: finishing write tid 5 to smithi10729407-19 2020-06-12T02:37:44.677 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 389 (ObjNum 2495 snap 0 seq_num 2495) dirty exists 2020-06-12T02:37:44.677 INFO:tasks.rados.rados.0.smithi107.stdout:2496: left oid 19 (ObjNum 2495 snap 0 seq_num 2495) 2020-06-12T02:37:44.677 INFO:tasks.rados.rados.0.smithi107.stdout:2496: done (2 left) 2020-06-12T02:37:44.677 INFO:tasks.rados.rados.0.smithi107.stdout:2499: done (1 left) 2020-06-12T02:37:44.677 INFO:tasks.rados.rados.0.smithi107.stdout:2500: delete oid 6 current snap is 0 2020-06-12T02:37:44.678 INFO:tasks.rados.rados.0.smithi107.stdout:2500: done (1 left) 2020-06-12T02:37:44.678 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:37:44.678 INFO:tasks.rados.rados.0.smithi107.stdout:2501: seq_num 2500 ranges {0=679936,679936=401408} 2020-06-12T02:37:44.678 INFO:tasks.rados.rados.0.smithi107.stdout:2501: writing smithi10729407-17 from 0 to 679936 tid 1 2020-06-12T02:37:44.678 INFO:tasks.rados.rados.0.smithi107.stdout:2501: writing smithi10729407-17 from 679936 to 1081344 tid 2 2020-06-12T02:37:44.679 INFO:tasks.rados.rados.0.smithi107.stdout:2502: delete oid 5 current snap is 0 2020-06-12T02:37:44.679 INFO:tasks.rados.rados.0.smithi107.stdout:2498: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:44.679 INFO:tasks.rados.rados.0.smithi107.stdout:2498: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:44.679 INFO:tasks.rados.rados.0.smithi107.stdout:2498: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:44.679 INFO:tasks.rados.rados.0.smithi107.stdout:2498: finishing write tid 5 to smithi10729407-9 2020-06-12T02:37:44.679 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 417 (ObjNum 2497 snap 0 seq_num 2497) dirty exists 2020-06-12T02:37:44.680 INFO:tasks.rados.rados.0.smithi107.stdout:2498: left oid 9 (ObjNum 2497 snap 0 seq_num 2497) 2020-06-12T02:37:44.680 INFO:tasks.rados.rados.0.smithi107.stdout:2498: done (2 left) 2020-06-12T02:37:44.680 INFO:tasks.rados.rados.0.smithi107.stdout:2502: done (1 left) 2020-06-12T02:37:44.680 INFO:tasks.rados.rados.0.smithi107.stdout:2503: delete oid 2 current snap is 0 2020-06-12T02:37:44.680 INFO:tasks.rados.rados.0.smithi107.stdout:2501: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:44.681 INFO:tasks.rados.rados.0.smithi107.stdout:2503: done (1 left) 2020-06-12T02:37:44.681 INFO:tasks.rados.rados.0.smithi107.stdout:2504: delete oid 9 current snap is 0 2020-06-12T02:37:44.681 INFO:tasks.rados.rados.0.smithi107.stdout:2504: done (1 left) 2020-06-12T02:37:44.681 INFO:tasks.rados.rados.0.smithi107.stdout:2505: delete oid 14 current snap is 0 2020-06-12T02:37:44.681 INFO:tasks.rados.rados.0.smithi107.stdout:2505: done (1 left) 2020-06-12T02:37:44.681 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:37:44.682 INFO:tasks.rados.rados.0.smithi107.stdout:2506: seq_num 2505 ranges {0=761856,761856=401408} 2020-06-12T02:37:44.682 INFO:tasks.rados.rados.0.smithi107.stdout:2506: writing smithi10729407-5 from 0 to 761856 tid 1 2020-06-12T02:37:44.682 INFO:tasks.rados.rados.0.smithi107.stdout:2506: writing smithi10729407-5 from 761856 to 1163264 tid 2 2020-06-12T02:37:44.682 INFO:tasks.rados.rados.0.smithi107.stdout:2507: delete oid 9 current snap is 0 2020-06-12T02:37:44.682 INFO:tasks.rados.rados.0.smithi107.stdout:2501: finishing write tid 2 to smithi10729407-17 2020-06-12T02:37:44.683 INFO:tasks.rados.rados.0.smithi107.stdout:2501: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:44.683 INFO:tasks.rados.rados.0.smithi107.stdout:2501: finishing write tid 5 to smithi10729407-17 2020-06-12T02:37:44.683 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 475 (ObjNum 2500 snap 0 seq_num 2500) dirty exists 2020-06-12T02:37:44.683 INFO:tasks.rados.rados.0.smithi107.stdout:2501: left oid 17 (ObjNum 2500 snap 0 seq_num 2500) 2020-06-12T02:37:44.683 INFO:tasks.rados.rados.0.smithi107.stdout:2501: done (2 left) 2020-06-12T02:37:44.683 INFO:tasks.rados.rados.0.smithi107.stdout:2507: done (1 left) 2020-06-12T02:37:44.684 INFO:tasks.rados.rados.0.smithi107.stdout:2508: delete oid 12 current snap is 0 2020-06-12T02:37:44.684 INFO:tasks.rados.rados.0.smithi107.stdout:2508: done (1 left) 2020-06-12T02:37:44.684 INFO:tasks.rados.rados.0.smithi107.stdout:2509: delete oid 16 current snap is 0 2020-06-12T02:37:44.684 INFO:tasks.rados.rados.0.smithi107.stdout:2506: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:44.684 INFO:tasks.rados.rados.0.smithi107.stdout:2509: done (1 left) 2020-06-12T02:37:44.685 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:37:44.685 INFO:tasks.rados.rados.0.smithi107.stdout:2510: seq_num 2509 ranges {0=425984,425984=589824,1015808=212992} 2020-06-12T02:37:44.685 INFO:tasks.rados.rados.0.smithi107.stdout:2510: writing smithi10729407-6 from 0 to 425984 tid 1 2020-06-12T02:37:44.685 INFO:tasks.rados.rados.0.smithi107.stdout:2510: writing smithi10729407-6 from 425984 to 1015808 tid 2 2020-06-12T02:37:44.685 INFO:tasks.rados.rados.0.smithi107.stdout:2510: writing smithi10729407-6 from 1015808 to 1228800 tid 3 2020-06-12T02:37:44.685 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:37:44.686 INFO:tasks.rados.rados.0.smithi107.stdout:2511: seq_num 2510 ranges {860160=442368,1302528=475136,1777664=327680} 2020-06-12T02:37:44.686 INFO:tasks.rados.rados.0.smithi107.stdout:2511: writing smithi10729407-7 from 860160 to 1302528 tid 1 2020-06-12T02:37:44.686 INFO:tasks.rados.rados.0.smithi107.stdout:2511: writing smithi10729407-7 from 1302528 to 1777664 tid 2 2020-06-12T02:37:44.686 INFO:tasks.rados.rados.0.smithi107.stdout:2511: writing smithi10729407-7 from 1777664 to 2105344 tid 3 2020-06-12T02:37:44.686 INFO:tasks.rados.rados.0.smithi107.stdout:2512: delete oid 12 current snap is 0 2020-06-12T02:37:44.687 INFO:tasks.rados.rados.0.smithi107.stdout:2506: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:44.687 INFO:tasks.rados.rados.0.smithi107.stdout:2506: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:44.687 INFO:tasks.rados.rados.0.smithi107.stdout:2506: finishing write tid 5 to smithi10729407-5 2020-06-12T02:37:44.687 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 211 (ObjNum 2505 snap 0 seq_num 2505) dirty exists 2020-06-12T02:37:44.687 INFO:tasks.rados.rados.0.smithi107.stdout:2506: left oid 5 (ObjNum 2505 snap 0 seq_num 2505) 2020-06-12T02:37:44.687 INFO:tasks.rados.rados.0.smithi107.stdout:2510: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:44.688 INFO:tasks.rados.rados.0.smithi107.stdout:2510: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:44.688 INFO:tasks.rados.rados.0.smithi107.stdout:2510: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:44.688 INFO:tasks.rados.rados.0.smithi107.stdout:2510: finishing write tid 5 to smithi10729407-6 2020-06-12T02:37:44.688 INFO:tasks.rados.rados.0.smithi107.stdout:2510: finishing write tid 6 to smithi10729407-6 2020-06-12T02:37:44.688 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 518 (ObjNum 2509 snap 0 seq_num 2509) dirty exists 2020-06-12T02:37:44.688 INFO:tasks.rados.rados.0.smithi107.stdout:2510: left oid 6 (ObjNum 2509 snap 0 seq_num 2509) 2020-06-12T02:37:44.689 INFO:tasks.rados.rados.0.smithi107.stdout:2506: done (3 left) 2020-06-12T02:37:44.689 INFO:tasks.rados.rados.0.smithi107.stdout:2510: done (2 left) 2020-06-12T02:37:44.689 INFO:tasks.rados.rados.0.smithi107.stdout:2512: done (1 left) 2020-06-12T02:37:44.689 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:37:44.689 INFO:tasks.rados.rados.0.smithi107.stdout:2513: seq_num 2512 ranges {974848=475136,1449984=638976,2088960=163840} 2020-06-12T02:37:44.692 INFO:tasks.rados.rados.0.smithi107.stdout:2513: writing smithi10729407-15 from 974848 to 1449984 tid 1 2020-06-12T02:37:44.692 INFO:tasks.rados.rados.0.smithi107.stdout:2513: writing smithi10729407-15 from 1449984 to 2088960 tid 2 2020-06-12T02:37:44.693 INFO:tasks.rados.rados.0.smithi107.stdout:2513: writing smithi10729407-15 from 2088960 to 2252800 tid 3 2020-06-12T02:37:44.693 INFO:tasks.rados.rados.0.smithi107.stdout:2514: delete oid 6 current snap is 0 2020-06-12T02:37:44.693 INFO:tasks.rados.rados.0.smithi107.stdout:2511: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:44.693 INFO:tasks.rados.rados.0.smithi107.stdout:2511: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:44.693 INFO:tasks.rados.rados.0.smithi107.stdout:2511: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:44.694 INFO:tasks.rados.rados.0.smithi107.stdout:2511: finishing write tid 5 to smithi10729407-7 2020-06-12T02:37:44.694 INFO:tasks.rados.rados.0.smithi107.stdout:2511: finishing write tid 6 to smithi10729407-7 2020-06-12T02:37:44.694 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 484 (ObjNum 2510 snap 0 seq_num 2510) dirty exists 2020-06-12T02:37:44.694 INFO:tasks.rados.rados.0.smithi107.stdout:2511: left oid 7 (ObjNum 2510 snap 0 seq_num 2510) 2020-06-12T02:37:44.694 INFO:tasks.rados.rados.0.smithi107.stdout:2511: done (2 left) 2020-06-12T02:37:44.695 INFO:tasks.rados.rados.0.smithi107.stdout:2514: done (1 left) 2020-06-12T02:37:44.695 INFO:tasks.rados.rados.0.smithi107.stdout:2515: delete oid 8 current snap is 0 2020-06-12T02:37:44.695 INFO:tasks.rados.rados.0.smithi107.stdout:2513: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:44.695 INFO:tasks.rados.rados.0.smithi107.stdout:2513: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:44.695 INFO:tasks.rados.rados.0.smithi107.stdout:2513: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:44.695 INFO:tasks.rados.rados.0.smithi107.stdout:2513: finishing write tid 5 to smithi10729407-15 2020-06-12T02:37:44.696 INFO:tasks.rados.rados.0.smithi107.stdout:2515: done (1 left) 2020-06-12T02:37:44.696 INFO:tasks.rados.rados.0.smithi107.stdout:2516: delete oid 13 current snap is 0 2020-06-12T02:37:44.696 INFO:tasks.rados.rados.0.smithi107.stdout:2513: finishing write tid 6 to smithi10729407-15 2020-06-12T02:37:44.696 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 415 (ObjNum 2512 snap 0 seq_num 2512) dirty exists 2020-06-12T02:37:44.696 INFO:tasks.rados.rados.0.smithi107.stdout:2513: left oid 15 (ObjNum 2512 snap 0 seq_num 2512) 2020-06-12T02:37:44.697 INFO:tasks.rados.rados.0.smithi107.stdout:2513: done (1 left) 2020-06-12T02:37:44.697 INFO:tasks.rados.rados.0.smithi107.stdout:2516: done (0 left) 2020-06-12T02:37:44.697 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:37:44.697 INFO:tasks.rados.rados.0.smithi107.stdout:2517: seq_num 2516 ranges {0=540672,540672=581632,1122304=221184} 2020-06-12T02:37:44.697 INFO:tasks.rados.rados.0.smithi107.stdout:2517: writing smithi10729407-13 from 0 to 540672 tid 1 2020-06-12T02:37:44.699 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:44.692+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:44.699 INFO:tasks.rados.rados.0.smithi107.stdout:2517: writing smithi10729407-13 from 540672 to 1122304 tid 2 2020-06-12T02:37:44.699 INFO:tasks.rados.rados.0.smithi107.stdout:2517: writing smithi10729407-13 from 1122304 to 1343488 tid 3 2020-06-12T02:37:44.700 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:37:44.700 INFO:tasks.rados.rados.0.smithi107.stdout:2518: seq_num 2517 ranges {0=557056,557056=516096,1073152=286720} 2020-06-12T02:37:44.700 INFO:tasks.rados.rados.0.smithi107.stdout:2518: writing smithi10729407-20 from 0 to 557056 tid 1 2020-06-12T02:37:44.704 INFO:tasks.rados.rados.0.smithi107.stdout:2518: writing smithi10729407-20 from 557056 to 1073152 tid 2 2020-06-12T02:37:44.706 INFO:tasks.rados.rados.0.smithi107.stdout:2518: writing smithi10729407-20 from 1073152 to 1359872 tid 3 2020-06-12T02:37:44.706 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:37:44.706 INFO:tasks.rados.rados.0.smithi107.stdout:2519: seq_num 2518 ranges {892928=581632,1474560=794624,2269184=8192} 2020-06-12T02:37:44.715 INFO:tasks.rados.rados.0.smithi107.stdout:2519: writing smithi10729407-3 from 892928 to 1474560 tid 1 2020-06-12T02:37:44.720 INFO:tasks.rados.rados.0.smithi107.stdout:2519: writing smithi10729407-3 from 1474560 to 2269184 tid 2 2020-06-12T02:37:44.720 INFO:tasks.rados.rados.0.smithi107.stdout:2519: writing smithi10729407-3 from 2269184 to 2277376 tid 3 2020-06-12T02:37:44.720 INFO:tasks.rados.rados.0.smithi107.stdout:2520: delete oid 14 current snap is 0 2020-06-12T02:37:44.720 INFO:tasks.rados.rados.0.smithi107.stdout:2517: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:44.721 INFO:tasks.rados.rados.0.smithi107.stdout:2517: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:44.721 INFO:tasks.rados.rados.0.smithi107.stdout:2517: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:44.721 INFO:tasks.rados.rados.0.smithi107.stdout:2517: finishing write tid 5 to smithi10729407-13 2020-06-12T02:37:44.721 INFO:tasks.rados.rados.0.smithi107.stdout:2517: finishing write tid 6 to smithi10729407-13 2020-06-12T02:37:44.721 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 489 (ObjNum 2516 snap 0 seq_num 2516) dirty exists 2020-06-12T02:37:44.722 INFO:tasks.rados.rados.0.smithi107.stdout:2517: left oid 13 (ObjNum 2516 snap 0 seq_num 2516) 2020-06-12T02:37:44.722 INFO:tasks.rados.rados.0.smithi107.stdout:2518: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:44.722 INFO:tasks.rados.rados.0.smithi107.stdout:2518: finishing write tid 2 to smithi10729407-20 2020-06-12T02:37:44.722 INFO:tasks.rados.rados.0.smithi107.stdout:2518: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:44.722 INFO:tasks.rados.rados.0.smithi107.stdout:2518: finishing write tid 5 to smithi10729407-20 2020-06-12T02:37:44.722 INFO:tasks.rados.rados.0.smithi107.stdout:2518: finishing write tid 6 to smithi10729407-20 2020-06-12T02:37:44.723 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 159 (ObjNum 2517 snap 0 seq_num 2517) dirty exists 2020-06-12T02:37:44.723 INFO:tasks.rados.rados.0.smithi107.stdout:2518: left oid 20 (ObjNum 2517 snap 0 seq_num 2517) 2020-06-12T02:37:44.723 INFO:tasks.rados.rados.0.smithi107.stdout:2519: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:44.723 INFO:tasks.rados.rados.0.smithi107.stdout:2517: done (3 left) 2020-06-12T02:37:44.724 INFO:tasks.rados.rados.0.smithi107.stdout:2518: done (2 left) 2020-06-12T02:37:44.724 INFO:tasks.rados.rados.0.smithi107.stdout:2520: done (1 left) 2020-06-12T02:37:44.724 INFO:tasks.rados.rados.0.smithi107.stdout:2521: delete oid 15 current snap is 0 2020-06-12T02:37:44.728 INFO:tasks.rados.rados.0.smithi107.stdout:2519: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:44.730 INFO:tasks.rados.rados.0.smithi107.stdout:2521: done (1 left) 2020-06-12T02:37:44.731 INFO:tasks.rados.rados.0.smithi107.stdout:2522: delete oid 9 current snap is 0 2020-06-12T02:37:44.732 INFO:tasks.rados.rados.0.smithi107.stdout:2519: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:44.732 INFO:tasks.rados.rados.0.smithi107.stdout:2519: finishing write tid 5 to smithi10729407-3 2020-06-12T02:37:44.733 INFO:tasks.rados.rados.0.smithi107.stdout:2519: finishing write tid 6 to smithi10729407-3 2020-06-12T02:37:44.733 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 441 (ObjNum 2518 snap 0 seq_num 2518) dirty exists 2020-06-12T02:37:44.733 INFO:tasks.rados.rados.0.smithi107.stdout:2519: left oid 3 (ObjNum 2518 snap 0 seq_num 2518) 2020-06-12T02:37:44.735 INFO:tasks.rados.rados.0.smithi107.stdout:2519: done (1 left) 2020-06-12T02:37:44.735 INFO:tasks.rados.rados.0.smithi107.stdout:2522: done (0 left) 2020-06-12T02:37:44.735 INFO:tasks.rados.rados.0.smithi107.stdout:2523: delete oid 12 current snap is 0 2020-06-12T02:37:44.737 INFO:tasks.rados.rados.0.smithi107.stdout:2523: done (0 left) 2020-06-12T02:37:44.737 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:37:44.737 INFO:tasks.rados.rados.0.smithi107.stdout:2524: seq_num 2523 ranges {1359872=663552,2023424=622592,2646016=180224} 2020-06-12T02:37:44.750 INFO:tasks.rados.rados.0.smithi107.stdout:2524: writing smithi10729407-20 from 1359872 to 2023424 tid 1 2020-06-12T02:37:44.754 INFO:tasks.rados.rados.0.smithi107.stdout:2524: writing smithi10729407-20 from 2023424 to 2646016 tid 2 2020-06-12T02:37:44.756 INFO:tasks.rados.rados.0.smithi107.stdout:2524: writing smithi10729407-20 from 2646016 to 2826240 tid 3 2020-06-12T02:37:44.756 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:37:44.756 INFO:tasks.rados.rados.0.smithi107.stdout:2525: seq_num 2524 ranges {2277376=679936,2957312=557056,3514368=245760} 2020-06-12T02:37:44.772 INFO:tasks.rados.rados.0.smithi107.stdout:2525: writing smithi10729407-3 from 2277376 to 2957312 tid 1 2020-06-12T02:37:44.776 INFO:tasks.rados.rados.0.smithi107.stdout:2525: writing smithi10729407-3 from 2957312 to 3514368 tid 2 2020-06-12T02:37:44.777 INFO:tasks.rados.rados.0.smithi107.stdout:2525: writing smithi10729407-3 from 3514368 to 3760128 tid 3 2020-06-12T02:37:44.777 INFO:tasks.rados.rados.0.smithi107.stdout:2526: delete oid 4 current snap is 0 2020-06-12T02:37:44.778 INFO:tasks.rados.rados.0.smithi107.stdout:2524: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:44.778 INFO:tasks.rados.rados.0.smithi107.stdout:2524: finishing write tid 2 to smithi10729407-20 2020-06-12T02:37:44.778 INFO:tasks.rados.rados.0.smithi107.stdout:2524: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:44.780 INFO:tasks.rados.rados.0.smithi107.stdout:2524: finishing write tid 5 to smithi10729407-20 2020-06-12T02:37:44.781 INFO:tasks.rados.rados.0.smithi107.stdout:2524: finishing write tid 6 to smithi10729407-20 2020-06-12T02:37:44.781 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 163 (ObjNum 2523 snap 0 seq_num 2523) dirty exists 2020-06-12T02:37:44.781 INFO:tasks.rados.rados.0.smithi107.stdout:2524: left oid 20 (ObjNum 2523 snap 0 seq_num 2523) 2020-06-12T02:37:44.783 INFO:tasks.rados.rados.0.smithi107.stdout:2524: done (2 left) 2020-06-12T02:37:44.783 INFO:tasks.rados.rados.0.smithi107.stdout:2526: done (1 left) 2020-06-12T02:37:44.783 INFO:tasks.rados.rados.0.smithi107.stdout:2527: delete oid 20 current snap is 0 2020-06-12T02:37:44.783 INFO:tasks.rados.rados.0.smithi107.stdout:2525: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:44.787 INFO:tasks.rados.rados.0.smithi107.stdout:2525: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:44.787 INFO:tasks.rados.rados.0.smithi107.stdout:2525: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:44.787 INFO:tasks.rados.rados.0.smithi107.stdout:2527: done (1 left) 2020-06-12T02:37:44.787 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:37:44.787 INFO:tasks.rados.rados.0.smithi107.stdout:2528: seq_num 2527 ranges {0=729088,729088=606208,1335296=196608} 2020-06-12T02:37:44.790 INFO:tasks.rados.rados.0.smithi107.stdout:2528: writing smithi10729407-9 from 0 to 729088 tid 1 2020-06-12T02:37:44.796 INFO:tasks.rados.rados.0.smithi107.stdout:2528: writing smithi10729407-9 from 729088 to 1335296 tid 2 2020-06-12T02:37:44.797 INFO:tasks.rados.rados.0.smithi107.stdout:2528: writing smithi10729407-9 from 1335296 to 1531904 tid 3 2020-06-12T02:37:44.797 INFO:tasks.rados.rados.0.smithi107.stdout:2529: delete oid 19 current snap is 0 2020-06-12T02:37:44.797 INFO:tasks.rados.rados.0.smithi107.stdout:2525: finishing write tid 5 to smithi10729407-3 2020-06-12T02:37:44.798 INFO:tasks.rados.rados.0.smithi107.stdout:2525: finishing write tid 6 to smithi10729407-3 2020-06-12T02:37:44.803 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 445 (ObjNum 2524 snap 0 seq_num 2524) dirty exists 2020-06-12T02:37:44.803 INFO:tasks.rados.rados.0.smithi107.stdout:2525: left oid 3 (ObjNum 2524 snap 0 seq_num 2524) 2020-06-12T02:37:44.804 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:44.796+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:37:44.805 INFO:tasks.rados.rados.0.smithi107.stdout:2525: done (2 left) 2020-06-12T02:37:44.805 INFO:tasks.rados.rados.0.smithi107.stdout:2529: done (1 left) 2020-06-12T02:37:44.805 INFO:tasks.rados.rados.0.smithi107.stdout:2530: delete oid 1 current snap is 0 2020-06-12T02:37:44.806 INFO:tasks.rados.rados.0.smithi107.stdout:2530: done (1 left) 2020-06-12T02:37:44.807 INFO:tasks.rados.rados.0.smithi107.stdout:2531: delete oid 2 current snap is 0 2020-06-12T02:37:44.807 INFO:tasks.rados.rados.0.smithi107.stdout:2528: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:44.807 INFO:tasks.rados.rados.0.smithi107.stdout:2528: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:44.807 INFO:tasks.rados.rados.0.smithi107.stdout:2528: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:44.807 INFO:tasks.rados.rados.0.smithi107.stdout:2528: finishing write tid 5 to smithi10729407-9 2020-06-12T02:37:44.808 INFO:tasks.rados.rados.0.smithi107.stdout:2531: done (1 left) 2020-06-12T02:37:44.808 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:37:44.808 INFO:tasks.rados.rados.0.smithi107.stdout:2532: seq_num 2531 ranges {0=794624,794624=540672,1335296=262144} 2020-06-12T02:37:44.811 INFO:tasks.rados.rados.0.smithi107.stdout:2532: writing smithi10729407-8 from 0 to 794624 tid 1 2020-06-12T02:37:44.813 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:44.813 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:44.814 INFO:tasks.rados.rados.0.smithi107.stdout:2532: writing smithi10729407-8 from 794624 to 1335296 tid 2 2020-06-12T02:37:44.816 INFO:tasks.rados.rados.0.smithi107.stdout:2532: writing smithi10729407-8 from 1335296 to 1597440 tid 3 2020-06-12T02:37:44.816 INFO:tasks.rados.rados.0.smithi107.stdout:2533: delete oid 13 current snap is 0 2020-06-12T02:37:44.817 INFO:tasks.rados.rados.0.smithi107.stdout:2528: finishing write tid 6 to smithi10729407-9 2020-06-12T02:37:44.817 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 424 (ObjNum 2527 snap 0 seq_num 2527) dirty exists 2020-06-12T02:37:44.817 INFO:tasks.rados.rados.0.smithi107.stdout:2528: left oid 9 (ObjNum 2527 snap 0 seq_num 2527) 2020-06-12T02:37:44.818 INFO:tasks.rados.rados.0.smithi107.stdout:2528: done (2 left) 2020-06-12T02:37:44.818 INFO:tasks.rados.rados.0.smithi107.stdout:2533: done (1 left) 2020-06-12T02:37:44.819 INFO:tasks.rados.rados.0.smithi107.stdout:2534: delete oid 11 current snap is 0 2020-06-12T02:37:44.823 INFO:tasks.rados.rados.0.smithi107.stdout:2532: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:44.824 INFO:tasks.rados.rados.0.smithi107.stdout:2534: done (1 left) 2020-06-12T02:37:44.824 INFO:tasks.rados.rados.0.smithi107.stdout:2535: delete oid 12 current snap is 0 2020-06-12T02:37:44.826 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-06-12T02:37:44.827 INFO:tasks.rados.rados.0.smithi107.stdout:2532: finishing write tid 2 to smithi10729407-8 2020-06-12T02:37:44.827 INFO:tasks.rados.rados.0.smithi107.stdout:2535: done (1 left) 2020-06-12T02:37:44.827 INFO:tasks.rados.rados.0.smithi107.stdout:2536: delete oid 2 current snap is 0 2020-06-12T02:37:44.828 INFO:tasks.rados.rados.0.smithi107.stdout:2532: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:44.828 INFO:tasks.rados.rados.0.smithi107.stdout:2532: finishing write tid 5 to smithi10729407-8 2020-06-12T02:37:44.829 INFO:tasks.rados.rados.0.smithi107.stdout:2536: done (1 left) 2020-06-12T02:37:44.829 INFO:tasks.rados.rados.0.smithi107.stdout:2537: delete oid 9 current snap is 0 2020-06-12T02:37:44.829 INFO:tasks.rados.rados.0.smithi107.stdout:2532: finishing write tid 6 to smithi10729407-8 2020-06-12T02:37:44.829 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 421 (ObjNum 2531 snap 0 seq_num 2531) dirty exists 2020-06-12T02:37:44.829 INFO:tasks.rados.rados.0.smithi107.stdout:2532: left oid 8 (ObjNum 2531 snap 0 seq_num 2531) 2020-06-12T02:37:44.830 INFO:tasks.rados.rados.0.smithi107.stdout:2532: done (1 left) 2020-06-12T02:37:44.830 INFO:tasks.rados.rados.0.smithi107.stdout:2537: done (0 left) 2020-06-12T02:37:44.830 INFO:tasks.rados.rados.0.smithi107.stdout:2538: delete oid 7 current snap is 0 2020-06-12T02:37:44.833 INFO:tasks.rados.rados.0.smithi107.stdout:2538: done (0 left) 2020-06-12T02:37:44.833 INFO:tasks.rados.rados.0.smithi107.stdout:2539: delete oid 1 current snap is 0 2020-06-12T02:37:44.835 INFO:tasks.rados.rados.0.smithi107.stdout:2539: done (0 left) 2020-06-12T02:37:44.837 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:37:44.837 INFO:tasks.rados.rados.0.smithi107.stdout:2540: seq_num 2539 ranges {0=532480,532480=466944,999424=475136,1474560=262144} 2020-06-12T02:37:44.838 INFO:tasks.rados.rados.0.smithi107.stdout:2540: writing smithi10729407-13 from 0 to 532480 tid 1 2020-06-12T02:37:44.841 INFO:tasks.rados.rados.0.smithi107.stdout:2540: writing smithi10729407-13 from 532480 to 999424 tid 2 2020-06-12T02:37:44.843 INFO:tasks.rados.rados.0.smithi107.stdout:2540: writing smithi10729407-13 from 999424 to 1474560 tid 3 2020-06-12T02:37:44.846 INFO:tasks.rados.rados.0.smithi107.stdout:2540: writing smithi10729407-13 from 1474560 to 1736704 tid 4 2020-06-12T02:37:44.846 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:37:44.846 INFO:tasks.rados.rados.0.smithi107.stdout:2541: seq_num 2540 ranges {0=548864,548864=802816,1351680=401408} 2020-06-12T02:37:44.849 INFO:tasks.rados.rados.0.smithi107.stdout:2541: writing smithi10729407-12 from 0 to 548864 tid 1 2020-06-12T02:37:44.854 INFO:tasks.rados.rados.0.smithi107.stdout:2541: writing smithi10729407-12 from 548864 to 1351680 tid 2 2020-06-12T02:37:44.857 INFO:tasks.rados.rados.0.smithi107.stdout:2541: writing smithi10729407-12 from 1351680 to 1753088 tid 3 2020-06-12T02:37:44.858 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:37:44.858 INFO:tasks.rados.rados.0.smithi107.stdout:2542: seq_num 2541 ranges {0=565248,565248=688128,1253376=442368,1695744=73728} 2020-06-12T02:37:44.861 INFO:tasks.rados.rados.0.smithi107.stdout:2542: writing smithi10729407-10 from 0 to 565248 tid 1 2020-06-12T02:37:44.865 INFO:tasks.rados.rados.0.smithi107.stdout:2542: writing smithi10729407-10 from 565248 to 1253376 tid 2 2020-06-12T02:37:44.868 INFO:tasks.rados.rados.0.smithi107.stdout:2542: writing smithi10729407-10 from 1253376 to 1695744 tid 3 2020-06-12T02:37:44.871 INFO:tasks.rados.rados.0.smithi107.stdout:2542: writing smithi10729407-10 from 1695744 to 1769472 tid 4 2020-06-12T02:37:44.871 INFO:tasks.rados.rados.0.smithi107.stdout:2543: delete oid 17 current snap is 0 2020-06-12T02:37:44.871 INFO:tasks.rados.rados.0.smithi107.stdout:2540: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:44.872 INFO:tasks.rados.rados.0.smithi107.stdout:2540: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:44.872 INFO:tasks.rados.rados.0.smithi107.stdout:2540: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:44.872 INFO:tasks.rados.rados.0.smithi107.stdout:2540: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:44.872 INFO:tasks.rados.rados.0.smithi107.stdout:2540: finishing write tid 6 to smithi10729407-13 2020-06-12T02:37:44.872 INFO:tasks.rados.rados.0.smithi107.stdout:2541: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:44.873 INFO:tasks.rados.rados.0.smithi107.stdout:2540: finishing write tid 7 to smithi10729407-13 2020-06-12T02:37:44.873 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 496 (ObjNum 2539 snap 0 seq_num 2539) dirty exists 2020-06-12T02:37:44.873 INFO:tasks.rados.rados.0.smithi107.stdout:2540: left oid 13 (ObjNum 2539 snap 0 seq_num 2539) 2020-06-12T02:37:44.873 INFO:tasks.rados.rados.0.smithi107.stdout:2541: finishing write tid 2 to smithi10729407-12 2020-06-12T02:37:44.873 INFO:tasks.rados.rados.0.smithi107.stdout:2541: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:44.874 INFO:tasks.rados.rados.0.smithi107.stdout:2542: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:44.874 INFO:tasks.rados.rados.0.smithi107.stdout:2541: finishing write tid 5 to smithi10729407-12 2020-06-12T02:37:44.874 INFO:tasks.rados.rados.0.smithi107.stdout:2541: finishing write tid 6 to smithi10729407-12 2020-06-12T02:37:44.874 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 398 (ObjNum 2540 snap 0 seq_num 2540) dirty exists 2020-06-12T02:37:44.874 INFO:tasks.rados.rados.0.smithi107.stdout:2541: left oid 12 (ObjNum 2540 snap 0 seq_num 2540) 2020-06-12T02:37:44.875 INFO:tasks.rados.rados.0.smithi107.stdout:2540: done (3 left) 2020-06-12T02:37:44.875 INFO:tasks.rados.rados.0.smithi107.stdout:2541: done (2 left) 2020-06-12T02:37:44.875 INFO:tasks.rados.rados.0.smithi107.stdout:2543: done (1 left) 2020-06-12T02:37:44.876 INFO:tasks.rados.rados.0.smithi107.stdout:2544: delete oid 9 current snap is 0 2020-06-12T02:37:44.876 INFO:tasks.rados.rados.0.smithi107.stdout:2544: done (1 left) 2020-06-12T02:37:44.876 INFO:tasks.rados.rados.0.smithi107.stdout:2545: delete oid 2 current snap is 0 2020-06-12T02:37:44.881 INFO:tasks.rados.rados.0.smithi107.stdout:2542: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:44.881 INFO:tasks.rados.rados.0.smithi107.stdout:2542: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:44.881 INFO:tasks.rados.rados.0.smithi107.stdout:2542: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:44.882 INFO:tasks.rados.rados.0.smithi107.stdout:2545: done (1 left) 2020-06-12T02:37:44.882 INFO:tasks.rados.rados.0.smithi107.stdout:2546: delete oid 2 current snap is 0 2020-06-12T02:37:44.883 INFO:tasks.rados.rados.0.smithi107.stdout:2542: finishing write tid 6 to smithi10729407-10 2020-06-12T02:37:44.884 INFO:tasks.rados.rados.0.smithi107.stdout:2542: finishing write tid 7 to smithi10729407-10 2020-06-12T02:37:44.884 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 247 (ObjNum 2541 snap 0 seq_num 2541) dirty exists 2020-06-12T02:37:44.884 INFO:tasks.rados.rados.0.smithi107.stdout:2542: left oid 10 (ObjNum 2541 snap 0 seq_num 2541) 2020-06-12T02:37:44.885 INFO:tasks.rados.rados.0.smithi107.stdout:2542: done (1 left) 2020-06-12T02:37:44.885 INFO:tasks.rados.rados.0.smithi107.stdout:2546: done (0 left) 2020-06-12T02:37:44.885 INFO:tasks.rados.rados.0.smithi107.stdout:2547: delete oid 9 current snap is 0 2020-06-12T02:37:44.886 INFO:tasks.rados.rados.0.smithi107.stdout:2547: done (0 left) 2020-06-12T02:37:44.886 INFO:tasks.rados.rados.0.smithi107.stdout:2548: delete oid 4 current snap is 0 2020-06-12T02:37:44.889 INFO:tasks.rados.rados.0.smithi107.stdout:2548: done (0 left) 2020-06-12T02:37:44.889 INFO:tasks.rados.rados.0.smithi107.stdout:2549: delete oid 18 current snap is 0 2020-06-12T02:37:44.889 INFO:tasks.rados.rados.0.smithi107.stdout:2549: done (0 left) 2020-06-12T02:37:44.890 INFO:tasks.rados.rados.0.smithi107.stdout:2550: delete oid 7 current snap is 0 2020-06-12T02:37:44.892 INFO:tasks.rados.rados.0.smithi107.stdout:2550: done (0 left) 2020-06-12T02:37:44.892 INFO:tasks.rados.rados.0.smithi107.stdout:2551: delete oid 5 current snap is 0 2020-06-12T02:37:44.893 INFO:tasks.rados.rados.0.smithi107.stdout:2551: done (0 left) 2020-06-12T02:37:44.894 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:37:44.894 INFO:tasks.rados.rados.0.smithi107.stdout:2552: seq_num 2551 ranges {0=729088,729088=778240,1507328=417792,1925120=8192} 2020-06-12T02:37:44.898 INFO:tasks.rados.rados.0.smithi107.stdout:2552: writing smithi10729407-5 from 0 to 729088 tid 1 2020-06-12T02:37:44.899 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:37:44.896+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:37:44.903 INFO:tasks.rados.rados.0.smithi107.stdout:2552: writing smithi10729407-5 from 729088 to 1507328 tid 2 2020-06-12T02:37:44.904 INFO:tasks.rados.rados.0.smithi107.stdout:2552: writing smithi10729407-5 from 1507328 to 1925120 tid 3 2020-06-12T02:37:44.905 INFO:tasks.rados.rados.0.smithi107.stdout:2552: writing smithi10729407-5 from 1925120 to 1933312 tid 4 2020-06-12T02:37:44.905 INFO:tasks.rados.rados.0.smithi107.stdout:2553: delete oid 13 current snap is 0 2020-06-12T02:37:44.907 INFO:tasks.rados.rados.0.smithi107.stdout:2553: done (1 left) 2020-06-12T02:37:44.908 INFO:tasks.rados.rados.0.smithi107.stdout:2554: delete oid 10 current snap is 0 2020-06-12T02:37:44.908 INFO:tasks.rados.rados.0.smithi107.stdout:2552: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:44.911 INFO:tasks.rados.rados.0.smithi107.stdout:2554: done (1 left) 2020-06-12T02:37:44.911 INFO:tasks.rados.rados.0.smithi107.stdout:2555: delete oid 2 current snap is 0 2020-06-12T02:37:44.913 INFO:tasks.rados.rados.0.smithi107.stdout:2552: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:44.914 INFO:tasks.rados.rados.0.smithi107.stdout:2552: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:44.914 INFO:tasks.rados.rados.0.smithi107.stdout:2555: done (1 left) 2020-06-12T02:37:44.914 INFO:tasks.rados.rados.0.smithi107.stdout:2556: delete oid 18 current snap is 0 2020-06-12T02:37:44.915 INFO:tasks.rados.rados.0.smithi107.stdout:2552: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:44.915 INFO:tasks.rados.rados.0.smithi107.stdout:2552: finishing write tid 6 to smithi10729407-5 2020-06-12T02:37:44.915 INFO:tasks.rados.rados.0.smithi107.stdout:2556: done (1 left) 2020-06-12T02:37:44.915 INFO:tasks.rados.rados.0.smithi107.stdout:2557: delete oid 2 current snap is 0 2020-06-12T02:37:44.916 INFO:tasks.rados.rados.0.smithi107.stdout:2552: finishing write tid 7 to smithi10729407-5 2020-06-12T02:37:44.916 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 217 (ObjNum 2551 snap 0 seq_num 2551) dirty exists 2020-06-12T02:37:44.917 INFO:tasks.rados.rados.0.smithi107.stdout:2552: left oid 5 (ObjNum 2551 snap 0 seq_num 2551) 2020-06-12T02:37:44.917 INFO:tasks.rados.rados.0.smithi107.stdout:2552: done (1 left) 2020-06-12T02:37:44.917 INFO:tasks.rados.rados.0.smithi107.stdout:2557: done (0 left) 2020-06-12T02:37:44.917 INFO:tasks.rados.rados.0.smithi107.stdout:2558: delete oid 15 current snap is 0 2020-06-12T02:37:44.919 INFO:tasks.rados.rados.0.smithi107.stdout:2558: done (0 left) 2020-06-12T02:37:44.919 INFO:tasks.rados.rados.0.smithi107.stdout:2559: delete oid 14 current snap is 0 2020-06-12T02:37:44.920 INFO:tasks.rados.rados.0.smithi107.stdout:2559: done (0 left) 2020-06-12T02:37:44.921 INFO:tasks.rados.rados.0.smithi107.stdout:2560: delete oid 18 current snap is 0 2020-06-12T02:37:44.922 INFO:tasks.rados.rados.0.smithi107.stdout:2560: done (0 left) 2020-06-12T02:37:44.922 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:37:44.922 INFO:tasks.rados.rados.0.smithi107.stdout:2561: seq_num 2560 ranges {1753088=483328,2236416=589824,2826240=589824,3416064=425984} 2020-06-12T02:37:44.934 INFO:tasks.rados.rados.0.smithi107.stdout:2561: writing smithi10729407-12 from 1753088 to 2236416 tid 1 2020-06-12T02:37:44.938 INFO:tasks.rados.rados.0.smithi107.stdout:2561: writing smithi10729407-12 from 2236416 to 2826240 tid 2 2020-06-12T02:37:44.941 INFO:tasks.rados.rados.0.smithi107.stdout:2561: writing smithi10729407-12 from 2826240 to 3416064 tid 3 2020-06-12T02:37:44.945 INFO:tasks.rados.rados.0.smithi107.stdout:2561: writing smithi10729407-12 from 3416064 to 3842048 tid 4 2020-06-12T02:37:44.945 INFO:tasks.rados.rados.0.smithi107.stdout:2562: delete oid 16 current snap is 0 2020-06-12T02:37:44.945 INFO:tasks.rados.rados.0.smithi107.stdout:2561: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:44.948 INFO:tasks.rados.rados.0.smithi107.stdout:2562: done (1 left) 2020-06-12T02:37:44.948 INFO:tasks.rados.rados.0.smithi107.stdout:2563: delete oid 8 current snap is 0 2020-06-12T02:37:44.950 INFO:tasks.rados.rados.0.smithi107.stdout:2563: done (1 left) 2020-06-12T02:37:44.951 INFO:tasks.rados.rados.0.smithi107.stdout:2564: delete oid 2 current snap is 0 2020-06-12T02:37:44.952 INFO:tasks.rados.rados.0.smithi107.stdout:2561: finishing write tid 2 to smithi10729407-12 2020-06-12T02:37:44.953 INFO:tasks.rados.rados.0.smithi107.stdout:2561: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:44.953 INFO:tasks.rados.rados.0.smithi107.stdout:2564: done (1 left) 2020-06-12T02:37:44.953 INFO:tasks.rados.rados.0.smithi107.stdout:2565: delete oid 7 current snap is 0 2020-06-12T02:37:44.954 INFO:tasks.rados.rados.0.smithi107.stdout:2561: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:44.956 INFO:tasks.rados.rados.0.smithi107.stdout:2565: done (1 left) 2020-06-12T02:37:44.956 INFO:tasks.rados.rados.0.smithi107.stdout:2566: delete oid 17 current snap is 0 2020-06-12T02:37:44.957 INFO:tasks.rados.rados.0.smithi107.stdout:2561: finishing write tid 6 to smithi10729407-12 2020-06-12T02:37:44.958 INFO:tasks.rados.rados.0.smithi107.stdout:2561: finishing write tid 7 to smithi10729407-12 2020-06-12T02:37:44.958 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 403 (ObjNum 2560 snap 0 seq_num 2560) dirty exists 2020-06-12T02:37:44.958 INFO:tasks.rados.rados.0.smithi107.stdout:2561: left oid 12 (ObjNum 2560 snap 0 seq_num 2560) 2020-06-12T02:37:44.958 INFO:tasks.rados.rados.0.smithi107.stdout:2561: done (1 left) 2020-06-12T02:37:44.958 INFO:tasks.rados.rados.0.smithi107.stdout:2566: done (0 left) 2020-06-12T02:37:44.959 INFO:tasks.rados.rados.0.smithi107.stdout:2567: delete oid 18 current snap is 0 2020-06-12T02:37:44.959 INFO:tasks.rados.rados.0.smithi107.stdout:2567: done (0 left) 2020-06-12T02:37:44.960 INFO:tasks.rados.rados.0.smithi107.stdout:2568: delete oid 12 current snap is 0 2020-06-12T02:37:44.962 INFO:tasks.rados.rados.0.smithi107.stdout:2568: done (0 left) 2020-06-12T02:37:44.962 INFO:tasks.rados.rados.0.smithi107.stdout:2569: delete oid 18 current snap is 0 2020-06-12T02:37:44.965 INFO:tasks.rados.rados.0.smithi107.stdout:2569: done (0 left) 2020-06-12T02:37:44.965 INFO:tasks.rados.rados.0.smithi107.stdout:2570: delete oid 10 current snap is 0 2020-06-12T02:37:44.967 INFO:tasks.rados.rados.0.smithi107.stdout:2570: done (0 left) 2020-06-12T02:37:44.967 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:37:44.967 INFO:tasks.rados.rados.0.smithi107.stdout:2571: seq_num 2570 ranges {0=647168,647168=729088,1376256=516096,1892352=360448} 2020-06-12T02:37:44.970 INFO:tasks.rados.rados.0.smithi107.stdout:2571: writing smithi10729407-10 from 0 to 647168 tid 1 2020-06-12T02:37:44.974 INFO:tasks.rados.rados.0.smithi107.stdout:2571: writing smithi10729407-10 from 647168 to 1376256 tid 2 2020-06-12T02:37:44.978 INFO:tasks.rados.rados.0.smithi107.stdout:2571: writing smithi10729407-10 from 1376256 to 1892352 tid 3 2020-06-12T02:37:44.980 INFO:tasks.rados.rados.0.smithi107.stdout:2571: writing smithi10729407-10 from 1892352 to 2252800 tid 4 2020-06-12T02:37:44.980 INFO:tasks.rados.rados.0.smithi107.stdout:2572: delete oid 1 current snap is 0 2020-06-12T02:37:44.981 INFO:tasks.rados.rados.0.smithi107.stdout:2571: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:44.983 INFO:tasks.rados.rados.0.smithi107.stdout:2572: done (1 left) 2020-06-12T02:37:44.983 INFO:tasks.rados.rados.0.smithi107.stdout:2573: delete oid 18 current snap is 0 2020-06-12T02:37:44.986 INFO:tasks.rados.rados.0.smithi107.stdout:2573: done (1 left) 2020-06-12T02:37:44.986 INFO:tasks.rados.rados.0.smithi107.stdout:2574: delete oid 17 current snap is 0 2020-06-12T02:37:44.987 INFO:tasks.rados.rados.0.smithi107.stdout:2571: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:44.987 INFO:tasks.rados.rados.0.smithi107.stdout:2571: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:44.988 INFO:tasks.rados.rados.0.smithi107.stdout:2571: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:44.988 INFO:tasks.rados.rados.0.smithi107.stdout:2574: done (1 left) 2020-06-12T02:37:44.989 INFO:tasks.rados.rados.0.smithi107.stdout:2575: delete oid 9 current snap is 0 2020-06-12T02:37:44.989 INFO:tasks.rados.rados.0.smithi107.stdout:2571: finishing write tid 6 to smithi10729407-10 2020-06-12T02:37:44.991 INFO:tasks.rados.rados.0.smithi107.stdout:2571: finishing write tid 7 to smithi10729407-10 2020-06-12T02:37:44.991 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 254 (ObjNum 2570 snap 0 seq_num 2570) dirty exists 2020-06-12T02:37:44.991 INFO:tasks.rados.rados.0.smithi107.stdout:2571: left oid 10 (ObjNum 2570 snap 0 seq_num 2570) 2020-06-12T02:37:44.992 INFO:tasks.rados.rados.0.smithi107.stdout:2571: done (1 left) 2020-06-12T02:37:44.992 INFO:tasks.rados.rados.0.smithi107.stdout:2575: done (0 left) 2020-06-12T02:37:44.992 INFO:tasks.rados.rados.0.smithi107.stdout:2576: delete oid 6 current snap is 0 2020-06-12T02:37:44.995 INFO:tasks.rados.rados.0.smithi107.stdout:2576: done (0 left) 2020-06-12T02:37:44.995 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:37:44.995 INFO:tasks.rados.rados.0.smithi107.stdout:2577: seq_num 2576 ranges {0=753664,753664=434176,1187840=507904,1695744=647168,2342912=16384} 2020-06-12T02:37:44.996 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:37:44.992+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:37:45.000 INFO:tasks.rados.rados.0.smithi107.stdout:2577: writing smithi10729407-8 from 0 to 753664 tid 1 2020-06-12T02:37:45.003 INFO:tasks.rados.rados.0.smithi107.stdout:2577: writing smithi10729407-8 from 753664 to 1187840 tid 2 2020-06-12T02:37:45.007 INFO:tasks.rados.rados.0.smithi107.stdout:2577: writing smithi10729407-8 from 1187840 to 1695744 tid 3 2020-06-12T02:37:45.013 INFO:tasks.rados.rados.0.smithi107.stdout:2577: writing smithi10729407-8 from 1695744 to 2342912 tid 4 2020-06-12T02:37:45.014 INFO:tasks.rados.rados.0.smithi107.stdout:2577: writing smithi10729407-8 from 2342912 to 2359296 tid 5 2020-06-12T02:37:45.014 INFO:tasks.rados.rados.0.smithi107.stdout:2578: delete oid 19 current snap is 0 2020-06-12T02:37:45.016 INFO:tasks.rados.rados.0.smithi107.stdout:2577: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:45.016 INFO:tasks.rados.rados.0.smithi107.stdout:2578: done (1 left) 2020-06-12T02:37:45.017 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:37:45.017 INFO:tasks.rados.rados.0.smithi107.stdout:2579: seq_num 2578 ranges {0=786432,786432=655360,1441792=475136,1916928=475136} 2020-06-12T02:37:45.021 INFO:tasks.rados.rados.0.smithi107.stdout:2579: writing smithi10729407-18 from 0 to 786432 tid 1 2020-06-12T02:37:45.025 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:45.025 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:45.027 INFO:tasks.rados.rados.0.smithi107.stdout:2579: writing smithi10729407-18 from 786432 to 1441792 tid 2 2020-06-12T02:37:45.029 INFO:tasks.rados.rados.0.smithi107.stdout:2579: writing smithi10729407-18 from 1441792 to 1916928 tid 3 2020-06-12T02:37:45.033 INFO:tasks.rados.rados.0.smithi107.stdout:2579: writing smithi10729407-18 from 1916928 to 2392064 tid 4 2020-06-12T02:37:45.033 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:37:45.033 INFO:tasks.rados.rados.0.smithi107.stdout:2580: seq_num 2579 ranges {0=802816,802816=540672,1343488=679936,2023424=385024} 2020-06-12T02:37:45.036 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-06-12T02:37:45.039 INFO:tasks.rados.rados.0.smithi107.stdout:2580: writing smithi10729407-1 from 0 to 802816 tid 1 2020-06-12T02:37:45.046 INFO:tasks.rados.rados.0.smithi107.stdout:2580: writing smithi10729407-1 from 802816 to 1343488 tid 2 2020-06-12T02:37:45.048 INFO:tasks.rados.rados.0.smithi107.stdout:2580: writing smithi10729407-1 from 1343488 to 2023424 tid 3 2020-06-12T02:37:45.051 INFO:tasks.rados.rados.0.smithi107.stdout:2580: writing smithi10729407-1 from 2023424 to 2408448 tid 4 2020-06-12T02:37:45.051 INFO:tasks.rados.rados.0.smithi107.stdout:2581: delete oid 10 current snap is 0 2020-06-12T02:37:45.051 INFO:tasks.rados.rados.0.smithi107.stdout:2577: finishing write tid 2 to smithi10729407-8 2020-06-12T02:37:45.051 INFO:tasks.rados.rados.0.smithi107.stdout:2577: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:45.052 INFO:tasks.rados.rados.0.smithi107.stdout:2577: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:45.052 INFO:tasks.rados.rados.0.smithi107.stdout:2577: finishing write tid 5 to smithi10729407-8 2020-06-12T02:37:45.052 INFO:tasks.rados.rados.0.smithi107.stdout:2577: finishing write tid 7 to smithi10729407-8 2020-06-12T02:37:45.052 INFO:tasks.rados.rados.0.smithi107.stdout:2577: finishing write tid 8 to smithi10729407-8 2020-06-12T02:37:45.052 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 429 (ObjNum 2576 snap 0 seq_num 2576) dirty exists 2020-06-12T02:37:45.053 INFO:tasks.rados.rados.0.smithi107.stdout:2577: left oid 8 (ObjNum 2576 snap 0 seq_num 2576) 2020-06-12T02:37:45.053 INFO:tasks.rados.rados.0.smithi107.stdout:2579: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:45.053 INFO:tasks.rados.rados.0.smithi107.stdout:2579: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:45.053 INFO:tasks.rados.rados.0.smithi107.stdout:2579: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:45.053 INFO:tasks.rados.rados.0.smithi107.stdout:2579: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:45.053 INFO:tasks.rados.rados.0.smithi107.stdout:2579: finishing write tid 6 to smithi10729407-18 2020-06-12T02:37:45.054 INFO:tasks.rados.rados.0.smithi107.stdout:2579: finishing write tid 7 to smithi10729407-18 2020-06-12T02:37:45.054 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 439 (ObjNum 2578 snap 0 seq_num 2578) dirty exists 2020-06-12T02:37:45.054 INFO:tasks.rados.rados.0.smithi107.stdout:2579: left oid 18 (ObjNum 2578 snap 0 seq_num 2578) 2020-06-12T02:37:45.054 INFO:tasks.rados.rados.0.smithi107.stdout:2580: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:45.055 INFO:tasks.rados.rados.0.smithi107.stdout:2580: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:45.055 INFO:tasks.rados.rados.0.smithi107.stdout:2577: done (3 left) 2020-06-12T02:37:45.055 INFO:tasks.rados.rados.0.smithi107.stdout:2579: done (2 left) 2020-06-12T02:37:45.055 INFO:tasks.rados.rados.0.smithi107.stdout:2581: done (1 left) 2020-06-12T02:37:45.056 INFO:tasks.rados.rados.0.smithi107.stdout:2582: delete oid 17 current snap is 0 2020-06-12T02:37:45.056 INFO:tasks.rados.rados.0.smithi107.stdout:2582: done (1 left) 2020-06-12T02:37:45.056 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:37:45.057 INFO:tasks.rados.rados.0.smithi107.stdout:2583: seq_num 2582 ranges {2359296=49152} 2020-06-12T02:37:45.070 INFO:tasks.rados.rados.0.smithi107.stdout:2583: writing smithi10729407-8 from 2359296 to 2408448 tid 1 2020-06-12T02:37:45.070 INFO:tasks.rados.rados.0.smithi107.stdout:2584: delete oid 2 current snap is 0 2020-06-12T02:37:45.071 INFO:tasks.rados.rados.0.smithi107.stdout:2580: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:45.071 INFO:tasks.rados.rados.0.smithi107.stdout:2580: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:45.071 INFO:tasks.rados.rados.0.smithi107.stdout:2580: finishing write tid 6 to smithi10729407-1 2020-06-12T02:37:45.071 INFO:tasks.rados.rados.0.smithi107.stdout:2580: finishing write tid 7 to smithi10729407-1 2020-06-12T02:37:45.071 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 528 (ObjNum 2579 snap 0 seq_num 2579) dirty exists 2020-06-12T02:37:45.072 INFO:tasks.rados.rados.0.smithi107.stdout:2580: left oid 1 (ObjNum 2579 snap 0 seq_num 2579) 2020-06-12T02:37:45.072 INFO:tasks.rados.rados.0.smithi107.stdout:2580: done (2 left) 2020-06-12T02:37:45.073 INFO:tasks.rados.rados.0.smithi107.stdout:2584: done (1 left) 2020-06-12T02:37:45.073 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:37:45.073 INFO:tasks.rados.rados.0.smithi107.stdout:2585: seq_num 2584 ranges {0=81920} 2020-06-12T02:37:45.074 INFO:tasks.rados.rados.0.smithi107.stdout:2585: writing smithi10729407-4 from 0 to 81920 tid 1 2020-06-12T02:37:45.074 INFO:tasks.rados.rados.0.smithi107.stdout:2586: delete oid 3 current snap is 0 2020-06-12T02:37:45.074 INFO:tasks.rados.rados.0.smithi107.stdout:2583: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:45.077 INFO:tasks.rados.rados.0.smithi107.stdout:2585: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:45.078 INFO:tasks.rados.rados.0.smithi107.stdout:2583: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:45.079 INFO:tasks.rados.rados.0.smithi107.stdout:2586: done (2 left) 2020-06-12T02:37:45.079 INFO:tasks.rados.rados.0.smithi107.stdout:2587: delete oid 9 current snap is 0 2020-06-12T02:37:45.079 INFO:tasks.rados.rados.0.smithi107.stdout:2583: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:45.080 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 431 (ObjNum 2582 snap 0 seq_num 2582) dirty exists 2020-06-12T02:37:45.080 INFO:tasks.rados.rados.0.smithi107.stdout:2583: left oid 8 (ObjNum 2582 snap 0 seq_num 2582) 2020-06-12T02:37:45.082 INFO:tasks.rados.rados.0.smithi107.stdout:2583: done (2 left) 2020-06-12T02:37:45.084 INFO:tasks.rados.rados.0.smithi107.stdout:2587: done (1 left) 2020-06-12T02:37:45.084 INFO:tasks.rados.rados.0.smithi107.stdout:2588: delete oid 12 current snap is 0 2020-06-12T02:37:45.085 INFO:tasks.rados.rados.0.smithi107.stdout:2585: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:45.086 INFO:tasks.rados.rados.0.smithi107.stdout:2585: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:45.086 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 483 (ObjNum 2584 snap 0 seq_num 2584) dirty exists 2020-06-12T02:37:45.086 INFO:tasks.rados.rados.0.smithi107.stdout:2585: left oid 4 (ObjNum 2584 snap 0 seq_num 2584) 2020-06-12T02:37:45.087 INFO:tasks.rados.rados.0.smithi107.stdout:2585: done (1 left) 2020-06-12T02:37:45.088 INFO:tasks.rados.rados.0.smithi107.stdout:2588: done (0 left) 2020-06-12T02:37:45.088 INFO:tasks.rados.rados.0.smithi107.stdout:2589: delete oid 12 current snap is 0 2020-06-12T02:37:45.088 INFO:tasks.rados.rados.0.smithi107.stdout:2589: done (0 left) 2020-06-12T02:37:45.088 INFO:tasks.rados.rados.0.smithi107.stdout:2590: delete oid 12 current snap is 0 2020-06-12T02:37:45.091 INFO:tasks.rados.rados.0.smithi107.stdout:2590: done (0 left) 2020-06-12T02:37:45.091 INFO:tasks.rados.rados.0.smithi107.stdout:2591: delete oid 20 current snap is 0 2020-06-12T02:37:45.091 INFO:tasks.rados.rados.0.smithi107.stdout:2591: done (0 left) 2020-06-12T02:37:45.091 INFO:tasks.rados.rados.0.smithi107.stdout:2592: delete oid 5 current snap is 0 2020-06-12T02:37:45.091 INFO:tasks.rados.rados.0.smithi107.stdout:2592: done (0 left) 2020-06-12T02:37:45.092 INFO:tasks.rados.rados.0.smithi107.stdout:2593: delete oid 10 current snap is 0 2020-06-12T02:37:45.092 INFO:tasks.rados.rados.0.smithi107.stdout:2593: done (0 left) 2020-06-12T02:37:45.092 INFO:tasks.rados.rados.0.smithi107.stdout:2594: delete oid 1 current snap is 0 2020-06-12T02:37:45.094 INFO:tasks.rados.rados.0.smithi107.stdout:2594: done (0 left) 2020-06-12T02:37:45.094 INFO:tasks.rados.rados.0.smithi107.stdout:2595: delete oid 4 current snap is 0 2020-06-12T02:37:45.096 INFO:tasks.rados.rados.0.smithi107.stdout:2595: done (0 left) 2020-06-12T02:37:45.096 INFO:tasks.rados.rados.0.smithi107.stdout:2596: delete oid 13 current snap is 0 2020-06-12T02:37:45.097 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:37:45.093+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:37:45.098 INFO:tasks.rados.rados.0.smithi107.stdout:2596: done (0 left) 2020-06-12T02:37:45.098 INFO:tasks.rados.rados.0.smithi107.stdout:2597: delete oid 3 current snap is 0 2020-06-12T02:37:45.100 INFO:tasks.rados.rados.0.smithi107.stdout:2597: done (0 left) 2020-06-12T02:37:45.100 INFO:tasks.rados.rados.0.smithi107.stdout:2598: delete oid 18 current snap is 0 2020-06-12T02:37:45.101 INFO:tasks.rados.rados.0.smithi107.stdout:2598: done (0 left) 2020-06-12T02:37:45.102 INFO:tasks.rados.rados.0.smithi107.stdout:2599: delete oid 6 current snap is 0 2020-06-12T02:37:45.104 INFO:tasks.rados.rados.0.smithi107.stdout:2599: done (0 left) 2020-06-12T02:37:45.104 INFO:tasks.rados.rados.0.smithi107.stdout:2600: delete oid 5 current snap is 0 2020-06-12T02:37:45.107 INFO:tasks.rados.rados.0.smithi107.stdout:2600: done (0 left) 2020-06-12T02:37:45.107 INFO:tasks.rados.rados.0.smithi107.stdout:2601: delete oid 11 current snap is 0 2020-06-12T02:37:45.109 INFO:tasks.rados.rados.0.smithi107.stdout:2601: done (0 left) 2020-06-12T02:37:45.109 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:37:45.109 INFO:tasks.rados.rados.0.smithi107.stdout:2602: seq_num 2601 ranges {0=368640} 2020-06-12T02:37:45.111 INFO:tasks.rados.rados.0.smithi107.stdout:2602: writing smithi10729407-13 from 0 to 368640 tid 1 2020-06-12T02:37:45.111 INFO:tasks.rados.rados.0.smithi107.stdout:2603: delete oid 15 current snap is 0 2020-06-12T02:37:45.113 INFO:tasks.rados.rados.0.smithi107.stdout:2603: done (1 left) 2020-06-12T02:37:45.114 INFO:tasks.rados.rados.0.smithi107.stdout:2604: delete oid 18 current snap is 0 2020-06-12T02:37:45.116 INFO:tasks.rados.rados.0.smithi107.stdout:2604: done (1 left) 2020-06-12T02:37:45.117 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:37:45.117 INFO:tasks.rados.rados.0.smithi107.stdout:2605: seq_num 2604 ranges {0=417792} 2020-06-12T02:37:45.118 INFO:tasks.rados.rados.0.smithi107.stdout:2605: writing smithi10729407-7 from 0 to 417792 tid 1 2020-06-12T02:37:45.118 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:37:45.119 INFO:tasks.rados.rados.0.smithi107.stdout:2606: seq_num 2605 ranges {0=434176} 2020-06-12T02:37:45.121 INFO:tasks.rados.rados.0.smithi107.stdout:2606: writing smithi10729407-10 from 0 to 434176 tid 1 2020-06-12T02:37:45.121 INFO:tasks.rados.rados.0.smithi107.stdout:2607: delete oid 14 current snap is 0 2020-06-12T02:37:45.121 INFO:tasks.rados.rados.0.smithi107.stdout:2602: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:45.121 INFO:tasks.rados.rados.0.smithi107.stdout:2602: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:45.122 INFO:tasks.rados.rados.0.smithi107.stdout:2602: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:45.122 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 502 (ObjNum 2601 snap 0 seq_num 2601) dirty exists 2020-06-12T02:37:45.122 INFO:tasks.rados.rados.0.smithi107.stdout:2602: left oid 13 (ObjNum 2601 snap 0 seq_num 2601) 2020-06-12T02:37:45.125 INFO:tasks.rados.rados.0.smithi107.stdout:2602: done (3 left) 2020-06-12T02:37:45.125 INFO:tasks.rados.rados.0.smithi107.stdout:2607: done (2 left) 2020-06-12T02:37:45.125 INFO:tasks.rados.rados.0.smithi107.stdout:2608: delete oid 19 current snap is 0 2020-06-12T02:37:45.126 INFO:tasks.rados.rados.0.smithi107.stdout:2605: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:45.126 INFO:tasks.rados.rados.0.smithi107.stdout:2605: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:45.127 INFO:tasks.rados.rados.0.smithi107.stdout:2606: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:45.127 INFO:tasks.rados.rados.0.smithi107.stdout:2608: done (2 left) 2020-06-12T02:37:45.127 INFO:tasks.rados.rados.0.smithi107.stdout:2609: delete oid 6 current snap is 0 2020-06-12T02:37:45.127 INFO:tasks.rados.rados.0.smithi107.stdout:2605: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:45.128 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 504 (ObjNum 2604 snap 0 seq_num 2604) dirty exists 2020-06-12T02:37:45.128 INFO:tasks.rados.rados.0.smithi107.stdout:2605: left oid 7 (ObjNum 2604 snap 0 seq_num 2604) 2020-06-12T02:37:45.129 INFO:tasks.rados.rados.0.smithi107.stdout:2605: done (2 left) 2020-06-12T02:37:45.129 INFO:tasks.rados.rados.0.smithi107.stdout:2609: done (1 left) 2020-06-12T02:37:45.129 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:37:45.129 INFO:tasks.rados.rados.0.smithi107.stdout:2610: seq_num 2609 ranges {0=499712} 2020-06-12T02:37:45.133 INFO:tasks.rados.rados.0.smithi107.stdout:2610: writing smithi10729407-4 from 0 to 499712 tid 1 2020-06-12T02:37:45.134 INFO:tasks.rados.rados.0.smithi107.stdout:2611: delete oid 12 current snap is 0 2020-06-12T02:37:45.134 INFO:tasks.rados.rados.0.smithi107.stdout:2606: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:45.134 INFO:tasks.rados.rados.0.smithi107.stdout:2606: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:45.134 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 258 (ObjNum 2605 snap 0 seq_num 2605) dirty exists 2020-06-12T02:37:45.134 INFO:tasks.rados.rados.0.smithi107.stdout:2606: left oid 10 (ObjNum 2605 snap 0 seq_num 2605) 2020-06-12T02:37:45.135 INFO:tasks.rados.rados.0.smithi107.stdout:2606: done (2 left) 2020-06-12T02:37:45.135 INFO:tasks.rados.rados.0.smithi107.stdout:2611: done (1 left) 2020-06-12T02:37:45.135 INFO:tasks.rados.rados.0.smithi107.stdout:2612: delete oid 13 current snap is 0 2020-06-12T02:37:45.198 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:45.195+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:45.203 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:45.203 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:45.211 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-06-12T02:37:45.299 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:45.296+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:45.328 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:45.329 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:45.337 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-06-12T02:37:45.338 INFO:tasks.rados.rados.0.smithi107.stdout:2612: done (1 left) 2020-06-12T02:37:45.339 INFO:tasks.rados.rados.0.smithi107.stdout:2613: delete oid 2 current snap is 0 2020-06-12T02:37:45.341 INFO:tasks.rados.rados.0.smithi107.stdout:2610: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:45.341 INFO:tasks.rados.rados.0.smithi107.stdout:2613: done (1 left) 2020-06-12T02:37:45.341 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:37:45.342 INFO:tasks.rados.rados.0.smithi107.stdout:2614: seq_num 2613 ranges {0=565248} 2020-06-12T02:37:45.344 INFO:tasks.rados.rados.0.smithi107.stdout:2614: writing smithi10729407-13 from 0 to 565248 tid 1 2020-06-12T02:37:45.344 INFO:tasks.rados.rados.0.smithi107.stdout:2615: delete oid 12 current snap is 0 2020-06-12T02:37:45.345 INFO:tasks.rados.rados.0.smithi107.stdout:2610: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:45.347 INFO:tasks.rados.rados.0.smithi107.stdout:2610: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:45.347 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 486 (ObjNum 2609 snap 0 seq_num 2609) dirty exists 2020-06-12T02:37:45.347 INFO:tasks.rados.rados.0.smithi107.stdout:2610: left oid 4 (ObjNum 2609 snap 0 seq_num 2609) 2020-06-12T02:37:45.350 INFO:tasks.rados.rados.0.smithi107.stdout:2610: done (2 left) 2020-06-12T02:37:45.351 INFO:tasks.rados.rados.0.smithi107.stdout:2615: done (1 left) 2020-06-12T02:37:45.351 INFO:tasks.rados.rados.0.smithi107.stdout:2616: delete oid 8 current snap is 0 2020-06-12T02:37:45.351 INFO:tasks.rados.rados.0.smithi107.stdout:2614: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:45.351 INFO:tasks.rados.rados.0.smithi107.stdout:2614: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:45.353 INFO:tasks.rados.rados.0.smithi107.stdout:2614: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:45.353 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 507 (ObjNum 2613 snap 0 seq_num 2613) dirty exists 2020-06-12T02:37:45.353 INFO:tasks.rados.rados.0.smithi107.stdout:2614: left oid 13 (ObjNum 2613 snap 0 seq_num 2613) 2020-06-12T02:37:45.354 INFO:tasks.rados.rados.0.smithi107.stdout:2614: done (1 left) 2020-06-12T02:37:45.354 INFO:tasks.rados.rados.0.smithi107.stdout:2616: done (0 left) 2020-06-12T02:37:45.354 INFO:tasks.rados.rados.0.smithi107.stdout:2617: delete oid 4 current snap is 0 2020-06-12T02:37:45.358 INFO:tasks.rados.rados.0.smithi107.stdout:2617: done (0 left) 2020-06-12T02:37:45.358 INFO:tasks.rados.rados.0.smithi107.stdout:2618: delete oid 14 current snap is 0 2020-06-12T02:37:45.361 INFO:tasks.rados.rados.0.smithi107.stdout:2618: done (0 left) 2020-06-12T02:37:45.362 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:37:45.362 INFO:tasks.rados.rados.0.smithi107.stdout:2619: seq_num 2618 ranges {0=655360} 2020-06-12T02:37:45.365 INFO:tasks.rados.rados.0.smithi107.stdout:2619: writing smithi10729407-5 from 0 to 655360 tid 1 2020-06-12T02:37:45.365 INFO:tasks.rados.rados.0.smithi107.stdout:2620: delete oid 11 current snap is 0 2020-06-12T02:37:45.367 INFO:tasks.rados.rados.0.smithi107.stdout:2620: done (1 left) 2020-06-12T02:37:45.367 INFO:tasks.rados.rados.0.smithi107.stdout:2621: delete oid 17 current snap is 0 2020-06-12T02:37:45.369 INFO:tasks.rados.rados.0.smithi107.stdout:2621: done (1 left) 2020-06-12T02:37:45.369 INFO:tasks.rados.rados.0.smithi107.stdout:2622: delete oid 19 current snap is 0 2020-06-12T02:37:45.371 INFO:tasks.rados.rados.0.smithi107.stdout:2619: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:45.374 INFO:tasks.rados.rados.0.smithi107.stdout:2622: done (1 left) 2020-06-12T02:37:45.375 INFO:tasks.rados.rados.0.smithi107.stdout:2623: delete oid 12 current snap is 0 2020-06-12T02:37:45.375 INFO:tasks.rados.rados.0.smithi107.stdout:2619: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:45.375 INFO:tasks.rados.rados.0.smithi107.stdout:2623: done (1 left) 2020-06-12T02:37:45.375 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:37:45.376 INFO:tasks.rados.rados.0.smithi107.stdout:2624: seq_num 2623 ranges {0=737280} 2020-06-12T02:37:45.379 INFO:tasks.rados.rados.0.smithi107.stdout:2624: writing smithi10729407-12 from 0 to 737280 tid 1 2020-06-12T02:37:45.379 INFO:tasks.rados.rados.0.smithi107.stdout:2625: delete oid 14 current snap is 0 2020-06-12T02:37:45.380 INFO:tasks.rados.rados.0.smithi107.stdout:2619: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:45.380 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 221 (ObjNum 2618 snap 0 seq_num 2618) dirty exists 2020-06-12T02:37:45.380 INFO:tasks.rados.rados.0.smithi107.stdout:2619: left oid 5 (ObjNum 2618 snap 0 seq_num 2618) 2020-06-12T02:37:45.381 INFO:tasks.rados.rados.0.smithi107.stdout:2619: done (2 left) 2020-06-12T02:37:45.381 INFO:tasks.rados.rados.0.smithi107.stdout:2625: done (1 left) 2020-06-12T02:37:45.381 INFO:tasks.rados.rados.0.smithi107.stdout:2626: delete oid 1 current snap is 0 2020-06-12T02:37:45.384 INFO:tasks.rados.rados.0.smithi107.stdout:2626: done (1 left) 2020-06-12T02:37:45.384 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:37:45.384 INFO:tasks.rados.rados.0.smithi107.stdout:2627: seq_num 2626 ranges {0=786432} 2020-06-12T02:37:45.389 INFO:tasks.rados.rados.0.smithi107.stdout:2627: writing smithi10729407-8 from 0 to 786432 tid 1 2020-06-12T02:37:45.402 INFO:tasks.rados.rados.0.smithi107.stdout:2628: delete oid 9 current snap is 0 2020-06-12T02:37:45.402 INFO:tasks.rados.rados.0.smithi107.stdout:2624: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:45.402 INFO:tasks.rados.rados.0.smithi107.stdout:2624: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:45.402 INFO:tasks.rados.rados.0.smithi107.stdout:2624: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:45.402 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 415 (ObjNum 2623 snap 0 seq_num 2623) dirty exists 2020-06-12T02:37:45.403 INFO:tasks.rados.rados.0.smithi107.stdout:2624: left oid 12 (ObjNum 2623 snap 0 seq_num 2623) 2020-06-12T02:37:45.405 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2020-06-12T02:37:45.405 INFO:tasks.rados.rados.0.smithi107.stdout:2624: done (2 left) 2020-06-12T02:37:45.405 INFO:tasks.rados.rados.0.smithi107.stdout:2628: done (1 left) 2020-06-12T02:37:45.406 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:37:45.406 INFO:tasks.rados.rados.0.smithi107.stdout:2629: seq_num 2628 ranges {0=417792,417792=401408} 2020-06-12T02:37:45.406 INFO:tasks.rados.rados.0.smithi107.stdout:2629: writing smithi10729407-18 from 0 to 417792 tid 1 2020-06-12T02:37:45.406 INFO:tasks.rados.rados.0.smithi107.stdout:2629: writing smithi10729407-18 from 417792 to 819200 tid 2 2020-06-12T02:37:45.406 INFO:tasks.rados.rados.0.smithi107.stdout:2630: delete oid 20 current snap is 0 2020-06-12T02:37:45.407 INFO:tasks.rados.rados.0.smithi107.stdout:2627: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:45.407 INFO:tasks.rados.rados.0.smithi107.stdout:2627: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:45.407 INFO:tasks.rados.rados.0.smithi107.stdout:2629: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:45.420 INFO:tasks.rados.rados.0.smithi107.stdout:2627: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:45.420 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 435 (ObjNum 2626 snap 0 seq_num 2626) dirty exists 2020-06-12T02:37:45.420 INFO:tasks.rados.rados.0.smithi107.stdout:2627: left oid 8 (ObjNum 2626 snap 0 seq_num 2626) 2020-06-12T02:37:45.420 INFO:tasks.rados.rados.0.smithi107.stdout:2627: done (2 left) 2020-06-12T02:37:45.421 INFO:tasks.rados.rados.0.smithi107.stdout:2630: done (1 left) 2020-06-12T02:37:45.421 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:37:45.421 INFO:tasks.rados.rados.0.smithi107.stdout:2631: seq_num 2630 ranges {0=450560,450560=401408} 2020-06-12T02:37:45.422 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:45.401+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:45.423 INFO:tasks.rados.rados.0.smithi107.stdout:2631: writing smithi10729407-3 from 0 to 450560 tid 1 2020-06-12T02:37:45.423 INFO:tasks.rados.rados.0.smithi107.stdout:2631: writing smithi10729407-3 from 450560 to 851968 tid 2 2020-06-12T02:37:45.424 INFO:tasks.rados.rados.0.smithi107.stdout:2632: delete oid 9 current snap is 0 2020-06-12T02:37:45.424 INFO:tasks.rados.rados.0.smithi107.stdout:2629: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:45.424 INFO:tasks.rados.rados.0.smithi107.stdout:2629: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:45.424 INFO:tasks.rados.rados.0.smithi107.stdout:2632: done (2 left) 2020-06-12T02:37:45.424 INFO:tasks.rados.rados.0.smithi107.stdout:2633: delete oid 1 current snap is 0 2020-06-12T02:37:45.425 INFO:tasks.rados.rados.0.smithi107.stdout:2631: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:45.425 INFO:tasks.rados.rados.0.smithi107.stdout:2631: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:45.425 INFO:tasks.rados.rados.0.smithi107.stdout:2631: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:45.425 INFO:tasks.rados.rados.0.smithi107.stdout:2629: finishing write tid 5 to smithi10729407-18 2020-06-12T02:37:45.425 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 446 (ObjNum 2628 snap 0 seq_num 2628) dirty exists 2020-06-12T02:37:45.425 INFO:tasks.rados.rados.0.smithi107.stdout:2629: left oid 18 (ObjNum 2628 snap 0 seq_num 2628) 2020-06-12T02:37:45.426 INFO:tasks.rados.rados.0.smithi107.stdout:2629: done (2 left) 2020-06-12T02:37:45.426 INFO:tasks.rados.rados.0.smithi107.stdout:2633: done (1 left) 2020-06-12T02:37:45.426 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:37:45.426 INFO:tasks.rados.rados.0.smithi107.stdout:2634: seq_num 2633 ranges {0=499712,499712=401408} 2020-06-12T02:37:45.426 INFO:tasks.rados.rados.0.smithi107.stdout:2634: writing smithi10729407-1 from 0 to 499712 tid 1 2020-06-12T02:37:45.428 INFO:tasks.rados.rados.0.smithi107.stdout:2634: writing smithi10729407-1 from 499712 to 901120 tid 2 2020-06-12T02:37:45.428 INFO:tasks.rados.rados.0.smithi107.stdout:2635: delete oid 20 current snap is 0 2020-06-12T02:37:45.429 INFO:tasks.rados.rados.0.smithi107.stdout:2631: finishing write tid 5 to smithi10729407-3 2020-06-12T02:37:45.429 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 451 (ObjNum 2630 snap 0 seq_num 2630) dirty exists 2020-06-12T02:37:45.430 INFO:tasks.rados.rados.0.smithi107.stdout:2631: left oid 3 (ObjNum 2630 snap 0 seq_num 2630) 2020-06-12T02:37:45.430 INFO:tasks.rados.rados.0.smithi107.stdout:2634: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:45.430 INFO:tasks.rados.rados.0.smithi107.stdout:2631: done (2 left) 2020-06-12T02:37:45.430 INFO:tasks.rados.rados.0.smithi107.stdout:2635: done (1 left) 2020-06-12T02:37:45.431 INFO:tasks.rados.rados.0.smithi107.stdout:2636: delete oid 19 current snap is 0 2020-06-12T02:37:45.431 INFO:tasks.rados.rados.0.smithi107.stdout:2636: done (1 left) 2020-06-12T02:37:45.431 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:37:45.432 INFO:tasks.rados.rados.0.smithi107.stdout:2637: seq_num 2636 ranges {786432=557056,1343488=401408} 2020-06-12T02:37:45.438 INFO:tasks.rados.rados.0.smithi107.stdout:2637: writing smithi10729407-8 from 786432 to 1343488 tid 1 2020-06-12T02:37:45.438 INFO:tasks.rados.rados.0.smithi107.stdout:2637: writing smithi10729407-8 from 1343488 to 1744896 tid 2 2020-06-12T02:37:45.438 INFO:tasks.rados.rados.0.smithi107.stdout:2638: delete oid 15 current snap is 0 2020-06-12T02:37:45.439 INFO:tasks.rados.rados.0.smithi107.stdout:2634: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:45.439 INFO:tasks.rados.rados.0.smithi107.stdout:2634: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:45.439 INFO:tasks.rados.rados.0.smithi107.stdout:2634: finishing write tid 5 to smithi10729407-1 2020-06-12T02:37:45.439 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 536 (ObjNum 2633 snap 0 seq_num 2633) dirty exists 2020-06-12T02:37:45.440 INFO:tasks.rados.rados.0.smithi107.stdout:2634: left oid 1 (ObjNum 2633 snap 0 seq_num 2633) 2020-06-12T02:37:45.442 INFO:tasks.rados.rados.0.smithi107.stdout:2637: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:45.445 INFO:tasks.rados.rados.0.smithi107.stdout:2637: finishing write tid 2 to smithi10729407-8 2020-06-12T02:37:45.446 INFO:tasks.rados.rados.0.smithi107.stdout:2637: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:45.447 INFO:tasks.rados.rados.0.smithi107.stdout:2634: done (2 left) 2020-06-12T02:37:45.447 INFO:tasks.rados.rados.0.smithi107.stdout:2638: done (1 left) 2020-06-12T02:37:45.447 INFO:tasks.rados.rados.0.smithi107.stdout:2639: delete oid 19 current snap is 0 2020-06-12T02:37:45.448 INFO:tasks.rados.rados.0.smithi107.stdout:2637: finishing write tid 5 to smithi10729407-8 2020-06-12T02:37:45.448 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 438 (ObjNum 2636 snap 0 seq_num 2636) dirty exists 2020-06-12T02:37:45.448 INFO:tasks.rados.rados.0.smithi107.stdout:2637: left oid 8 (ObjNum 2636 snap 0 seq_num 2636) 2020-06-12T02:37:45.449 INFO:tasks.rados.rados.0.smithi107.stdout:2637: done (1 left) 2020-06-12T02:37:45.449 INFO:tasks.rados.rados.0.smithi107.stdout:2639: done (0 left) 2020-06-12T02:37:45.449 INFO:tasks.rados.rados.0.smithi107.stdout:2640: delete oid 16 current snap is 0 2020-06-12T02:37:45.451 INFO:tasks.rados.rados.0.smithi107.stdout:2640: done (0 left) 2020-06-12T02:37:45.452 INFO:tasks.rados.rados.0.smithi107.stdout:2641: delete oid 2 current snap is 0 2020-06-12T02:37:45.453 INFO:tasks.rados.rados.0.smithi107.stdout:2641: done (0 left) 2020-06-12T02:37:45.453 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:37:45.453 INFO:tasks.rados.rados.0.smithi107.stdout:2642: seq_num 2641 ranges {901120=638976,1540096=401408} 2020-06-12T02:37:45.462 INFO:tasks.rados.rados.0.smithi107.stdout:2642: writing smithi10729407-1 from 901120 to 1540096 tid 1 2020-06-12T02:37:45.464 INFO:tasks.rados.rados.0.smithi107.stdout:2642: writing smithi10729407-1 from 1540096 to 1941504 tid 2 2020-06-12T02:37:45.464 INFO:tasks.rados.rados.0.smithi107.stdout:2643: delete oid 12 current snap is 0 2020-06-12T02:37:45.466 INFO:tasks.rados.rados.0.smithi107.stdout:2643: done (1 left) 2020-06-12T02:37:45.468 INFO:tasks.rados.rados.0.smithi107.stdout:2644: delete oid 4 current snap is 0 2020-06-12T02:37:45.468 INFO:tasks.rados.rados.0.smithi107.stdout:2642: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:45.469 INFO:tasks.rados.rados.0.smithi107.stdout:2644: done (1 left) 2020-06-12T02:37:45.470 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:37:45.470 INFO:tasks.rados.rados.0.smithi107.stdout:2645: seq_num 2644 ranges {1744896=688128,2433024=401408} 2020-06-12T02:37:45.480 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:45.481 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:45.482 INFO:tasks.rados.rados.0.smithi107.stdout:2645: writing smithi10729407-8 from 1744896 to 2433024 tid 1 2020-06-12T02:37:45.485 INFO:tasks.rados.rados.0.smithi107.stdout:2645: writing smithi10729407-8 from 2433024 to 2834432 tid 2 2020-06-12T02:37:45.485 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:37:45.485 INFO:tasks.rados.rados.0.smithi107.stdout:2646: seq_num 2645 ranges {0=704512,704512=401408} 2020-06-12T02:37:45.496 INFO:tasks.rados.rados.0.smithi107.stdout:2646: writing smithi10729407-19 from 0 to 704512 tid 1 2020-06-12T02:37:45.496 INFO:tasks.rados.rados.0.smithi107.stdout:2646: writing smithi10729407-19 from 704512 to 1105920 tid 2 2020-06-12T02:37:45.496 INFO:tasks.rados.rados.0.smithi107.stdout:2647: delete oid 17 current snap is 0 2020-06-12T02:37:45.496 INFO:tasks.rados.rados.0.smithi107.stdout:2642: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:45.497 INFO:tasks.rados.rados.0.smithi107.stdout:2642: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:45.497 INFO:tasks.rados.rados.0.smithi107.stdout:2642: finishing write tid 5 to smithi10729407-1 2020-06-12T02:37:45.497 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 539 (ObjNum 2641 snap 0 seq_num 2641) dirty exists 2020-06-12T02:37:45.497 INFO:tasks.rados.rados.0.smithi107.stdout:2642: left oid 1 (ObjNum 2641 snap 0 seq_num 2641) 2020-06-12T02:37:45.497 INFO:tasks.rados.rados.0.smithi107.stdout:2645: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:45.498 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-06-12T02:37:45.500 INFO:tasks.rados.rados.0.smithi107.stdout:2642: done (3 left) 2020-06-12T02:37:45.501 INFO:tasks.rados.rados.0.smithi107.stdout:2647: done (2 left) 2020-06-12T02:37:45.501 INFO:tasks.rados.rados.0.smithi107.stdout:2648: delete oid 1 current snap is 0 2020-06-12T02:37:45.501 INFO:tasks.rados.rados.0.smithi107.stdout:2645: finishing write tid 2 to smithi10729407-8 2020-06-12T02:37:45.501 INFO:tasks.rados.rados.0.smithi107.stdout:2648: done (2 left) 2020-06-12T02:37:45.501 INFO:tasks.rados.rados.0.smithi107.stdout:2649: delete oid 13 current snap is 0 2020-06-12T02:37:45.502 INFO:tasks.rados.rados.0.smithi107.stdout:2645: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:45.502 INFO:tasks.rados.rados.0.smithi107.stdout:2646: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:45.503 INFO:tasks.rados.rados.0.smithi107.stdout:2645: finishing write tid 5 to smithi10729407-8 2020-06-12T02:37:45.503 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 442 (ObjNum 2644 snap 0 seq_num 2644) dirty exists 2020-06-12T02:37:45.503 INFO:tasks.rados.rados.0.smithi107.stdout:2645: left oid 8 (ObjNum 2644 snap 0 seq_num 2644) 2020-06-12T02:37:45.503 INFO:tasks.rados.rados.0.smithi107.stdout:2646: finishing write tid 2 to smithi10729407-19 2020-06-12T02:37:45.504 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:37:45.501+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:37:45.505 INFO:tasks.rados.rados.0.smithi107.stdout:2645: done (2 left) 2020-06-12T02:37:45.505 INFO:tasks.rados.rados.0.smithi107.stdout:2649: done (1 left) 2020-06-12T02:37:45.505 INFO:tasks.rados.rados.0.smithi107.stdout:2650: delete oid 7 current snap is 0 2020-06-12T02:37:45.506 INFO:tasks.rados.rados.0.smithi107.stdout:2646: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:45.506 INFO:tasks.rados.rados.0.smithi107.stdout:2646: finishing write tid 5 to smithi10729407-19 2020-06-12T02:37:45.507 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 421 (ObjNum 2645 snap 0 seq_num 2645) dirty exists 2020-06-12T02:37:45.507 INFO:tasks.rados.rados.0.smithi107.stdout:2646: left oid 19 (ObjNum 2645 snap 0 seq_num 2645) 2020-06-12T02:37:45.509 INFO:tasks.rados.rados.0.smithi107.stdout:2646: done (1 left) 2020-06-12T02:37:45.510 INFO:tasks.rados.rados.0.smithi107.stdout:2650: done (0 left) 2020-06-12T02:37:45.510 INFO:tasks.rados.rados.0.smithi107.stdout:2651: delete oid 7 current snap is 0 2020-06-12T02:37:45.514 INFO:tasks.rados.rados.0.smithi107.stdout:2651: done (0 left) 2020-06-12T02:37:45.515 INFO:tasks.rados.rados.0.smithi107.stdout:2652: delete oid 15 current snap is 0 2020-06-12T02:37:45.517 INFO:tasks.rados.rados.0.smithi107.stdout:2652: done (0 left) 2020-06-12T02:37:45.517 INFO:tasks.rados.rados.0.smithi107.stdout:2653: delete oid 8 current snap is 0 2020-06-12T02:37:45.519 INFO:tasks.rados.rados.0.smithi107.stdout:2653: done (0 left) 2020-06-12T02:37:45.519 INFO:tasks.rados.rados.0.smithi107.stdout:2654: delete oid 8 current snap is 0 2020-06-12T02:37:45.522 INFO:tasks.rados.rados.0.smithi107.stdout:2654: done (0 left) 2020-06-12T02:37:45.522 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:37:45.523 INFO:tasks.rados.rados.0.smithi107.stdout:2655: seq_num 2654 ranges {0=458752,458752=761856,1220608=40960} 2020-06-12T02:37:45.525 INFO:tasks.rados.rados.0.smithi107.stdout:2655: writing smithi10729407-20 from 0 to 458752 tid 1 2020-06-12T02:37:45.530 INFO:tasks.rados.rados.0.smithi107.stdout:2655: writing smithi10729407-20 from 458752 to 1220608 tid 2 2020-06-12T02:37:45.530 INFO:tasks.rados.rados.0.smithi107.stdout:2655: writing smithi10729407-20 from 1220608 to 1261568 tid 3 2020-06-12T02:37:45.531 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:37:45.531 INFO:tasks.rados.rados.0.smithi107.stdout:2656: seq_num 2655 ranges {0=475136,475136=647168,1122304=155648} 2020-06-12T02:37:45.533 INFO:tasks.rados.rados.0.smithi107.stdout:2656: writing smithi10729407-4 from 0 to 475136 tid 1 2020-06-12T02:37:45.537 INFO:tasks.rados.rados.0.smithi107.stdout:2656: writing smithi10729407-4 from 475136 to 1122304 tid 2 2020-06-12T02:37:45.538 INFO:tasks.rados.rados.0.smithi107.stdout:2656: writing smithi10729407-4 from 1122304 to 1277952 tid 3 2020-06-12T02:37:45.539 INFO:tasks.rados.rados.0.smithi107.stdout:2657: delete oid 2 current snap is 0 2020-06-12T02:37:45.539 INFO:tasks.rados.rados.0.smithi107.stdout:2655: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:45.539 INFO:tasks.rados.rados.0.smithi107.stdout:2655: finishing write tid 2 to smithi10729407-20 2020-06-12T02:37:45.539 INFO:tasks.rados.rados.0.smithi107.stdout:2655: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:45.540 INFO:tasks.rados.rados.0.smithi107.stdout:2657: done (2 left) 2020-06-12T02:37:45.540 INFO:tasks.rados.rados.0.smithi107.stdout:2658: delete oid 9 current snap is 0 2020-06-12T02:37:45.542 INFO:tasks.rados.rados.0.smithi107.stdout:2656: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:45.542 INFO:tasks.rados.rados.0.smithi107.stdout:2658: done (2 left) 2020-06-12T02:37:45.543 INFO:tasks.rados.rados.0.smithi107.stdout:2659: delete oid 1 current snap is 0 2020-06-12T02:37:45.543 INFO:tasks.rados.rados.0.smithi107.stdout:2655: finishing write tid 5 to smithi10729407-20 2020-06-12T02:37:45.544 INFO:tasks.rados.rados.0.smithi107.stdout:2655: finishing write tid 6 to smithi10729407-20 2020-06-12T02:37:45.544 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 171 (ObjNum 2654 snap 0 seq_num 2654) dirty exists 2020-06-12T02:37:45.544 INFO:tasks.rados.rados.0.smithi107.stdout:2655: left oid 20 (ObjNum 2654 snap 0 seq_num 2654) 2020-06-12T02:37:45.545 INFO:tasks.rados.rados.0.smithi107.stdout:2655: done (2 left) 2020-06-12T02:37:45.545 INFO:tasks.rados.rados.0.smithi107.stdout:2659: done (1 left) 2020-06-12T02:37:45.545 INFO:tasks.rados.rados.0.smithi107.stdout:2660: delete oid 18 current snap is 0 2020-06-12T02:37:45.546 INFO:tasks.rados.rados.0.smithi107.stdout:2656: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:45.546 INFO:tasks.rados.rados.0.smithi107.stdout:2656: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:45.546 INFO:tasks.rados.rados.0.smithi107.stdout:2656: finishing write tid 5 to smithi10729407-4 2020-06-12T02:37:45.547 INFO:tasks.rados.rados.0.smithi107.stdout:2660: done (1 left) 2020-06-12T02:37:45.547 INFO:tasks.rados.rados.0.smithi107.stdout:2661: delete oid 12 current snap is 0 2020-06-12T02:37:45.548 INFO:tasks.rados.rados.0.smithi107.stdout:2656: finishing write tid 6 to smithi10729407-4 2020-06-12T02:37:45.548 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 494 (ObjNum 2655 snap 0 seq_num 2655) dirty exists 2020-06-12T02:37:45.549 INFO:tasks.rados.rados.0.smithi107.stdout:2656: left oid 4 (ObjNum 2655 snap 0 seq_num 2655) 2020-06-12T02:37:45.549 INFO:tasks.rados.rados.0.smithi107.stdout:2656: done (1 left) 2020-06-12T02:37:45.549 INFO:tasks.rados.rados.0.smithi107.stdout:2661: done (0 left) 2020-06-12T02:37:45.549 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:37:45.549 INFO:tasks.rados.rados.0.smithi107.stdout:2662: seq_num 2661 ranges {0=573440,573440=802816} 2020-06-12T02:37:45.551 INFO:tasks.rados.rados.0.smithi107.stdout:2662: writing smithi10729407-9 from 0 to 573440 tid 1 2020-06-12T02:37:45.557 INFO:tasks.rados.rados.0.smithi107.stdout:2662: writing smithi10729407-9 from 573440 to 1376256 tid 2 2020-06-12T02:37:45.558 INFO:tasks.rados.rados.0.smithi107.stdout:2663: delete oid 20 current snap is 0 2020-06-12T02:37:45.558 INFO:tasks.rados.rados.0.smithi107.stdout:2662: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:45.562 INFO:tasks.rados.rados.0.smithi107.stdout:2663: done (1 left) 2020-06-12T02:37:45.562 INFO:tasks.rados.rados.0.smithi107.stdout:2664: delete oid 5 current snap is 0 2020-06-12T02:37:45.564 INFO:tasks.rados.rados.0.smithi107.stdout:2664: done (1 left) 2020-06-12T02:37:45.564 INFO:tasks.rados.rados.0.smithi107.stdout:2665: delete oid 4 current snap is 0 2020-06-12T02:37:45.564 INFO:tasks.rados.rados.0.smithi107.stdout:2662: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:45.564 INFO:tasks.rados.rados.0.smithi107.stdout:2662: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:45.566 INFO:tasks.rados.rados.0.smithi107.stdout:2662: finishing write tid 5 to smithi10729407-9 2020-06-12T02:37:45.566 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 452 (ObjNum 2661 snap 0 seq_num 2661) dirty exists 2020-06-12T02:37:45.566 INFO:tasks.rados.rados.0.smithi107.stdout:2662: left oid 9 (ObjNum 2661 snap 0 seq_num 2661) 2020-06-12T02:37:45.567 INFO:tasks.rados.rados.0.smithi107.stdout:2662: done (1 left) 2020-06-12T02:37:45.567 INFO:tasks.rados.rados.0.smithi107.stdout:2665: done (0 left) 2020-06-12T02:37:45.567 INFO:tasks.rados.rados.0.smithi107.stdout:2666: delete oid 13 current snap is 0 2020-06-12T02:37:45.606 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:45.603+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:37:45.608 INFO:tasks.rados.rados.0.smithi107.stdout:2666: done (0 left) 2020-06-12T02:37:45.608 INFO:tasks.rados.rados.0.smithi107.stdout:2667: delete oid 18 current snap is 0 2020-06-12T02:37:45.610 INFO:tasks.rados.rados.0.smithi107.stdout:2667: done (0 left) 2020-06-12T02:37:45.610 INFO:tasks.rados.rados.0.smithi107.stdout:2668: delete oid 20 current snap is 0 2020-06-12T02:37:45.611 INFO:tasks.rados.rados.0.smithi107.stdout:2668: done (0 left) 2020-06-12T02:37:45.611 INFO:tasks.rados.rados.0.smithi107.stdout:2669: delete oid 5 current snap is 0 2020-06-12T02:37:45.615 INFO:tasks.rados.rados.0.smithi107.stdout:2669: done (0 left) 2020-06-12T02:37:45.615 INFO:tasks.rados.rados.0.smithi107.stdout:2670: delete oid 10 current snap is 0 2020-06-12T02:37:45.616 INFO:tasks.rados.rados.0.smithi107.stdout:2670: done (0 left) 2020-06-12T02:37:45.616 INFO:tasks.rados.rados.0.smithi107.stdout:2671: delete oid 17 current snap is 0 2020-06-12T02:37:45.617 INFO:tasks.rados.rados.0.smithi107.stdout:2671: done (0 left) 2020-06-12T02:37:45.618 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:37:45.619 INFO:tasks.rados.rados.0.smithi107.stdout:2672: seq_num 2671 ranges {0=737280,737280=491520,1228800=311296} 2020-06-12T02:37:45.622 INFO:tasks.rados.rados.0.smithi107.stdout:2672: writing smithi10729407-4 from 0 to 737280 tid 1 2020-06-12T02:37:45.624 INFO:tasks.rados.rados.0.smithi107.stdout:2672: writing smithi10729407-4 from 737280 to 1228800 tid 2 2020-06-12T02:37:45.626 INFO:tasks.rados.rados.0.smithi107.stdout:2672: writing smithi10729407-4 from 1228800 to 1540096 tid 3 2020-06-12T02:37:45.627 INFO:tasks.rados.rados.0.smithi107.stdout:2673: delete oid 11 current snap is 0 2020-06-12T02:37:45.628 INFO:tasks.rados.rados.0.smithi107.stdout:2672: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:45.629 INFO:tasks.rados.rados.0.smithi107.stdout:2673: done (1 left) 2020-06-12T02:37:45.629 INFO:tasks.rados.rados.0.smithi107.stdout:2674: delete oid 8 current snap is 0 2020-06-12T02:37:45.632 INFO:tasks.rados.rados.0.smithi107.stdout:2674: done (1 left) 2020-06-12T02:37:45.632 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:37:45.632 INFO:tasks.rados.rados.0.smithi107.stdout:2675: seq_num 2674 ranges {0=794624,794624=548864,1343488=253952} 2020-06-12T02:37:45.635 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:45.635 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:45.636 INFO:tasks.rados.rados.0.smithi107.stdout:2675: writing smithi10729407-13 from 0 to 794624 tid 1 2020-06-12T02:37:45.639 INFO:tasks.rados.rados.0.smithi107.stdout:2675: writing smithi10729407-13 from 794624 to 1343488 tid 2 2020-06-12T02:37:45.641 INFO:tasks.rados.rados.0.smithi107.stdout:2675: writing smithi10729407-13 from 1343488 to 1597440 tid 3 2020-06-12T02:37:45.641 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:37:45.641 INFO:tasks.rados.rados.0.smithi107.stdout:2676: seq_num 2675 ranges {0=409600,409600=434176,843776=663552,1507328=106496} 2020-06-12T02:37:45.643 INFO:tasks.rados.rados.0.smithi107.stdout:2676: writing smithi10729407-12 from 0 to 409600 tid 1 2020-06-12T02:37:45.646 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-06-12T02:37:45.647 INFO:tasks.rados.rados.0.smithi107.stdout:2676: writing smithi10729407-12 from 409600 to 843776 tid 2 2020-06-12T02:37:45.651 INFO:tasks.rados.rados.0.smithi107.stdout:2676: writing smithi10729407-12 from 843776 to 1507328 tid 3 2020-06-12T02:37:45.651 INFO:tasks.rados.rados.0.smithi107.stdout:2676: writing smithi10729407-12 from 1507328 to 1613824 tid 4 2020-06-12T02:37:45.652 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:37:45.652 INFO:tasks.rados.rados.0.smithi107.stdout:2677: seq_num 2676 ranges {0=425984,425984=720896,1146880=417792,1564672=65536} 2020-06-12T02:37:45.654 INFO:tasks.rados.rados.0.smithi107.stdout:2677: writing smithi10729407-8 from 0 to 425984 tid 1 2020-06-12T02:37:45.658 INFO:tasks.rados.rados.0.smithi107.stdout:2677: writing smithi10729407-8 from 425984 to 1146880 tid 2 2020-06-12T02:37:45.661 INFO:tasks.rados.rados.0.smithi107.stdout:2677: writing smithi10729407-8 from 1146880 to 1564672 tid 3 2020-06-12T02:37:45.661 INFO:tasks.rados.rados.0.smithi107.stdout:2677: writing smithi10729407-8 from 1564672 to 1630208 tid 4 2020-06-12T02:37:45.662 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:37:45.662 INFO:tasks.rados.rados.0.smithi107.stdout:2678: seq_num 2677 ranges {0=442368,442368=647168,1089536=557056} 2020-06-12T02:37:45.664 INFO:tasks.rados.rados.0.smithi107.stdout:2678: writing smithi10729407-6 from 0 to 442368 tid 1 2020-06-12T02:37:45.669 INFO:tasks.rados.rados.0.smithi107.stdout:2678: writing smithi10729407-6 from 442368 to 1089536 tid 2 2020-06-12T02:37:45.671 INFO:tasks.rados.rados.0.smithi107.stdout:2678: writing smithi10729407-6 from 1089536 to 1646592 tid 3 2020-06-12T02:37:45.672 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:37:45.672 INFO:tasks.rados.rados.0.smithi107.stdout:2679: seq_num 2678 ranges {0=458752,458752=532480,991232=434176,1425408=237568} 2020-06-12T02:37:45.680 INFO:tasks.rados.rados.0.smithi107.stdout:2679: writing smithi10729407-7 from 0 to 458752 tid 1 2020-06-12T02:37:45.680 INFO:tasks.rados.rados.0.smithi107.stdout:2679: writing smithi10729407-7 from 458752 to 991232 tid 2 2020-06-12T02:37:45.680 INFO:tasks.rados.rados.0.smithi107.stdout:2679: writing smithi10729407-7 from 991232 to 1425408 tid 3 2020-06-12T02:37:45.682 INFO:tasks.rados.rados.0.smithi107.stdout:2679: writing smithi10729407-7 from 1425408 to 1662976 tid 4 2020-06-12T02:37:45.682 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:37:45.682 INFO:tasks.rados.rados.0.smithi107.stdout:2680: seq_num 2679 ranges {851968=475136,1327104=417792,1744896=638976,2383872=147456} 2020-06-12T02:37:45.689 INFO:tasks.rados.rados.0.smithi107.stdout:2680: writing smithi10729407-3 from 851968 to 1327104 tid 1 2020-06-12T02:37:45.692 INFO:tasks.rados.rados.0.smithi107.stdout:2680: writing smithi10729407-3 from 1327104 to 1744896 tid 2 2020-06-12T02:37:45.696 INFO:tasks.rados.rados.0.smithi107.stdout:2680: writing smithi10729407-3 from 1744896 to 2383872 tid 3 2020-06-12T02:37:45.697 INFO:tasks.rados.rados.0.smithi107.stdout:2680: writing smithi10729407-3 from 2383872 to 2531328 tid 4 2020-06-12T02:37:45.697 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:45.697 INFO:tasks.rados.rados.0.smithi107.stdout:2681: seq_num 2680 ranges {0=491520,491520=704512,1196032=450560,1646592=49152} 2020-06-12T02:37:45.699 INFO:tasks.rados.rados.0.smithi107.stdout:2681: writing smithi10729407-2 from 0 to 491520 tid 1 2020-06-12T02:37:45.706 INFO:tasks.rados.rados.0.smithi107.stdout:2681: writing smithi10729407-2 from 491520 to 1196032 tid 2 2020-06-12T02:37:45.706 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:45.704+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:45.707 INFO:tasks.rados.rados.0.smithi107.stdout:2681: writing smithi10729407-2 from 1196032 to 1646592 tid 3 2020-06-12T02:37:45.708 INFO:tasks.rados.rados.0.smithi107.stdout:2681: writing smithi10729407-2 from 1646592 to 1695744 tid 4 2020-06-12T02:37:45.709 INFO:tasks.rados.rados.0.smithi107.stdout: waiting on 8 2020-06-12T02:37:45.709 INFO:tasks.rados.rados.0.smithi107.stdout:2672: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:45.709 INFO:tasks.rados.rados.0.smithi107.stdout:2672: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:45.709 INFO:tasks.rados.rados.0.smithi107.stdout:2672: finishing write tid 5 to smithi10729407-4 2020-06-12T02:37:45.709 INFO:tasks.rados.rados.0.smithi107.stdout:2672: finishing write tid 6 to smithi10729407-4 2020-06-12T02:37:45.710 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 500 (ObjNum 2671 snap 0 seq_num 2671) dirty exists 2020-06-12T02:37:45.710 INFO:tasks.rados.rados.0.smithi107.stdout:2672: left oid 4 (ObjNum 2671 snap 0 seq_num 2671) 2020-06-12T02:37:45.710 INFO:tasks.rados.rados.0.smithi107.stdout:2675: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:45.710 INFO:tasks.rados.rados.0.smithi107.stdout:2675: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:45.710 INFO:tasks.rados.rados.0.smithi107.stdout:2675: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:45.711 INFO:tasks.rados.rados.0.smithi107.stdout:2675: finishing write tid 5 to smithi10729407-13 2020-06-12T02:37:45.711 INFO:tasks.rados.rados.0.smithi107.stdout:2675: finishing write tid 6 to smithi10729407-13 2020-06-12T02:37:45.711 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 515 (ObjNum 2674 snap 0 seq_num 2674) dirty exists 2020-06-12T02:37:45.711 INFO:tasks.rados.rados.0.smithi107.stdout:2675: left oid 13 (ObjNum 2674 snap 0 seq_num 2674) 2020-06-12T02:37:45.711 INFO:tasks.rados.rados.0.smithi107.stdout:2676: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:45.711 INFO:tasks.rados.rados.0.smithi107.stdout:2676: finishing write tid 2 to smithi10729407-12 2020-06-12T02:37:45.712 INFO:tasks.rados.rados.0.smithi107.stdout:2676: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:45.712 INFO:tasks.rados.rados.0.smithi107.stdout:2676: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:45.712 INFO:tasks.rados.rados.0.smithi107.stdout:2676: finishing write tid 6 to smithi10729407-12 2020-06-12T02:37:45.712 INFO:tasks.rados.rados.0.smithi107.stdout:2676: finishing write tid 7 to smithi10729407-12 2020-06-12T02:37:45.712 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 427 (ObjNum 2675 snap 0 seq_num 2675) dirty exists 2020-06-12T02:37:45.713 INFO:tasks.rados.rados.0.smithi107.stdout:2676: left oid 12 (ObjNum 2675 snap 0 seq_num 2675) 2020-06-12T02:37:45.713 INFO:tasks.rados.rados.0.smithi107.stdout:2677: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:45.713 INFO:tasks.rados.rados.0.smithi107.stdout:2677: finishing write tid 2 to smithi10729407-8 2020-06-12T02:37:45.713 INFO:tasks.rados.rados.0.smithi107.stdout:2677: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:45.714 INFO:tasks.rados.rados.0.smithi107.stdout:2677: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:45.714 INFO:tasks.rados.rados.0.smithi107.stdout:2677: finishing write tid 6 to smithi10729407-8 2020-06-12T02:37:45.714 INFO:tasks.rados.rados.0.smithi107.stdout:2678: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:45.714 INFO:tasks.rados.rados.0.smithi107.stdout:2677: finishing write tid 7 to smithi10729407-8 2020-06-12T02:37:45.715 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 451 (ObjNum 2676 snap 0 seq_num 2676) dirty exists 2020-06-12T02:37:45.715 INFO:tasks.rados.rados.0.smithi107.stdout:2677: left oid 8 (ObjNum 2676 snap 0 seq_num 2676) 2020-06-12T02:37:45.715 INFO:tasks.rados.rados.0.smithi107.stdout:2678: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:45.715 INFO:tasks.rados.rados.0.smithi107.stdout:2678: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:45.716 INFO:tasks.rados.rados.0.smithi107.stdout:2678: finishing write tid 5 to smithi10729407-6 2020-06-12T02:37:45.716 INFO:tasks.rados.rados.0.smithi107.stdout:2678: finishing write tid 6 to smithi10729407-6 2020-06-12T02:37:45.716 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 545 (ObjNum 2677 snap 0 seq_num 2677) dirty exists 2020-06-12T02:37:45.716 INFO:tasks.rados.rados.0.smithi107.stdout:2678: left oid 6 (ObjNum 2677 snap 0 seq_num 2677) 2020-06-12T02:37:45.716 INFO:tasks.rados.rados.0.smithi107.stdout:2679: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:45.717 INFO:tasks.rados.rados.0.smithi107.stdout:2679: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:45.717 INFO:tasks.rados.rados.0.smithi107.stdout:2679: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:45.717 INFO:tasks.rados.rados.0.smithi107.stdout:2679: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:45.717 INFO:tasks.rados.rados.0.smithi107.stdout:2679: finishing write tid 6 to smithi10729407-7 2020-06-12T02:37:45.717 INFO:tasks.rados.rados.0.smithi107.stdout:2679: finishing write tid 7 to smithi10729407-7 2020-06-12T02:37:45.718 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 520 (ObjNum 2678 snap 0 seq_num 2678) dirty exists 2020-06-12T02:37:45.721 INFO:tasks.rados.rados.0.smithi107.stdout:2679: left oid 7 (ObjNum 2678 snap 0 seq_num 2678) 2020-06-12T02:37:45.722 INFO:tasks.rados.rados.0.smithi107.stdout:2680: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:45.722 INFO:tasks.rados.rados.0.smithi107.stdout:2680: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:45.722 INFO:tasks.rados.rados.0.smithi107.stdout:2681: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:45.722 INFO:tasks.rados.rados.0.smithi107.stdout:2672: done (7 left) 2020-06-12T02:37:45.722 INFO:tasks.rados.rados.0.smithi107.stdout:2675: done (6 left) 2020-06-12T02:37:45.723 INFO:tasks.rados.rados.0.smithi107.stdout:2676: done (5 left) 2020-06-12T02:37:45.723 INFO:tasks.rados.rados.0.smithi107.stdout:2677: done (4 left) 2020-06-12T02:37:45.723 INFO:tasks.rados.rados.0.smithi107.stdout:2678: done (3 left) 2020-06-12T02:37:45.723 INFO:tasks.rados.rados.0.smithi107.stdout:2679: done (2 left) 2020-06-12T02:37:45.723 INFO:tasks.rados.rados.0.smithi107.stdout:2682: delete oid 5 current snap is 0 2020-06-12T02:37:45.725 INFO:tasks.rados.rados.0.smithi107.stdout:2680: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:45.725 INFO:tasks.rados.rados.0.smithi107.stdout:2680: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:45.725 INFO:tasks.rados.rados.0.smithi107.stdout:2680: finishing write tid 6 to smithi10729407-3 2020-06-12T02:37:45.726 INFO:tasks.rados.rados.0.smithi107.stdout:2682: done (2 left) 2020-06-12T02:37:45.726 INFO:tasks.rados.rados.0.smithi107.stdout:2683: delete oid 18 current snap is 0 2020-06-12T02:37:45.726 INFO:tasks.rados.rados.0.smithi107.stdout:2683: done (2 left) 2020-06-12T02:37:45.726 INFO:tasks.rados.rados.0.smithi107.stdout:2684: delete oid 11 current snap is 0 2020-06-12T02:37:45.727 INFO:tasks.rados.rados.0.smithi107.stdout:2680: finishing write tid 7 to smithi10729407-3 2020-06-12T02:37:45.727 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 457 (ObjNum 2679 snap 0 seq_num 2679) dirty exists 2020-06-12T02:37:45.727 INFO:tasks.rados.rados.0.smithi107.stdout:2680: left oid 3 (ObjNum 2679 snap 0 seq_num 2679) 2020-06-12T02:37:45.727 INFO:tasks.rados.rados.0.smithi107.stdout:2680: done (2 left) 2020-06-12T02:37:45.727 INFO:tasks.rados.rados.0.smithi107.stdout:2684: done (1 left) 2020-06-12T02:37:45.727 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:37:45.728 INFO:tasks.rados.rados.0.smithi107.stdout:2685: seq_num 2684 ranges {0=557056,557056=688128,1245184=425984,1671168=90112} 2020-06-12T02:37:45.728 INFO:tasks.rados.rados.0.smithi107.stdout:2685: writing smithi10729407-18 from 0 to 557056 tid 1 2020-06-12T02:37:45.728 INFO:tasks.rados.rados.0.smithi107.stdout:2685: writing smithi10729407-18 from 557056 to 1245184 tid 2 2020-06-12T02:37:45.729 INFO:tasks.rados.rados.0.smithi107.stdout:2685: writing smithi10729407-18 from 1245184 to 1671168 tid 3 2020-06-12T02:37:45.729 INFO:tasks.rados.rados.0.smithi107.stdout:2685: writing smithi10729407-18 from 1671168 to 1761280 tid 4 2020-06-12T02:37:45.730 INFO:tasks.rados.rados.0.smithi107.stdout:2686: delete oid 3 current snap is 0 2020-06-12T02:37:45.730 INFO:tasks.rados.rados.0.smithi107.stdout:2681: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:45.730 INFO:tasks.rados.rados.0.smithi107.stdout:2681: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:45.730 INFO:tasks.rados.rados.0.smithi107.stdout:2681: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:45.730 INFO:tasks.rados.rados.0.smithi107.stdout:2681: finishing write tid 6 to smithi10729407-2 2020-06-12T02:37:45.730 INFO:tasks.rados.rados.0.smithi107.stdout:2681: finishing write tid 7 to smithi10729407-2 2020-06-12T02:37:45.731 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 281 (ObjNum 2680 snap 0 seq_num 2680) dirty exists 2020-06-12T02:37:45.731 INFO:tasks.rados.rados.0.smithi107.stdout:2681: left oid 2 (ObjNum 2680 snap 0 seq_num 2680) 2020-06-12T02:37:45.731 INFO:tasks.rados.rados.0.smithi107.stdout:2685: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:45.731 INFO:tasks.rados.rados.0.smithi107.stdout:2681: done (2 left) 2020-06-12T02:37:45.731 INFO:tasks.rados.rados.0.smithi107.stdout:2686: done (1 left) 2020-06-12T02:37:45.732 INFO:tasks.rados.rados.0.smithi107.stdout:2687: delete oid 17 current snap is 0 2020-06-12T02:37:45.733 INFO:tasks.rados.rados.0.smithi107.stdout:2685: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:45.733 INFO:tasks.rados.rados.0.smithi107.stdout:2685: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:45.734 INFO:tasks.rados.rados.0.smithi107.stdout:2687: done (1 left) 2020-06-12T02:37:45.735 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:37:45.735 INFO:tasks.rados.rados.0.smithi107.stdout:2688: seq_num 2687 ranges {0=606208,606208=745472,1351680=458752} 2020-06-12T02:37:45.737 INFO:tasks.rados.rados.0.smithi107.stdout:2688: writing smithi10729407-5 from 0 to 606208 tid 1 2020-06-12T02:37:45.741 INFO:tasks.rados.rados.0.smithi107.stdout:2688: writing smithi10729407-5 from 606208 to 1351680 tid 2 2020-06-12T02:37:45.744 INFO:tasks.rados.rados.0.smithi107.stdout:2688: writing smithi10729407-5 from 1351680 to 1810432 tid 3 2020-06-12T02:37:45.745 INFO:tasks.rados.rados.0.smithi107.stdout:2689: delete oid 7 current snap is 0 2020-06-12T02:37:45.745 INFO:tasks.rados.rados.0.smithi107.stdout:2685: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:45.745 INFO:tasks.rados.rados.0.smithi107.stdout:2685: finishing write tid 6 to smithi10729407-18 2020-06-12T02:37:45.745 INFO:tasks.rados.rados.0.smithi107.stdout:2685: finishing write tid 7 to smithi10729407-18 2020-06-12T02:37:45.746 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 459 (ObjNum 2684 snap 0 seq_num 2684) dirty exists 2020-06-12T02:37:45.746 INFO:tasks.rados.rados.0.smithi107.stdout:2685: left oid 18 (ObjNum 2684 snap 0 seq_num 2684) 2020-06-12T02:37:45.746 INFO:tasks.rados.rados.0.smithi107.stdout:2688: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:45.747 INFO:tasks.rados.rados.0.smithi107.stdout:2685: done (2 left) 2020-06-12T02:37:45.747 INFO:tasks.rados.rados.0.smithi107.stdout:2689: done (1 left) 2020-06-12T02:37:45.747 INFO:tasks.rados.rados.0.smithi107.stdout:2690: delete oid 9 current snap is 0 2020-06-12T02:37:45.750 INFO:tasks.rados.rados.0.smithi107.stdout:2688: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:45.751 INFO:tasks.rados.rados.0.smithi107.stdout:2688: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:45.752 INFO:tasks.rados.rados.0.smithi107.stdout:2688: finishing write tid 5 to smithi10729407-5 2020-06-12T02:37:45.752 INFO:tasks.rados.rados.0.smithi107.stdout:2690: done (1 left) 2020-06-12T02:37:45.752 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:37:45.752 INFO:tasks.rados.rados.0.smithi107.stdout:2691: seq_num 2690 ranges {0=655360,655360=794624,1449984=409600} 2020-06-12T02:37:45.756 INFO:tasks.rados.rados.0.smithi107.stdout:2691: writing smithi10729407-11 from 0 to 655360 tid 1 2020-06-12T02:37:45.761 INFO:tasks.rados.rados.0.smithi107.stdout:2691: writing smithi10729407-11 from 655360 to 1449984 tid 2 2020-06-12T02:37:45.764 INFO:tasks.rados.rados.0.smithi107.stdout:2691: writing smithi10729407-11 from 1449984 to 1859584 tid 3 2020-06-12T02:37:45.764 INFO:tasks.rados.rados.0.smithi107.stdout:2692: delete oid 16 current snap is 0 2020-06-12T02:37:45.764 INFO:tasks.rados.rados.0.smithi107.stdout:2688: finishing write tid 6 to smithi10729407-5 2020-06-12T02:37:45.765 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 228 (ObjNum 2687 snap 0 seq_num 2687) dirty exists 2020-06-12T02:37:45.765 INFO:tasks.rados.rados.0.smithi107.stdout:2688: left oid 5 (ObjNum 2687 snap 0 seq_num 2687) 2020-06-12T02:37:45.765 INFO:tasks.rados.rados.0.smithi107.stdout:2691: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:45.766 INFO:tasks.rados.rados.0.smithi107.stdout:2688: done (2 left) 2020-06-12T02:37:45.766 INFO:tasks.rados.rados.0.smithi107.stdout:2692: done (1 left) 2020-06-12T02:37:45.766 INFO:tasks.rados.rados.0.smithi107.stdout:2693: delete oid 10 current snap is 0 2020-06-12T02:37:45.769 INFO:tasks.rados.rados.0.smithi107.stdout:2693: done (1 left) 2020-06-12T02:37:45.769 INFO:tasks.rados.rados.0.smithi107.stdout:2694: delete oid 1 current snap is 0 2020-06-12T02:37:45.771 INFO:tasks.rados.rados.0.smithi107.stdout:2691: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:45.772 INFO:tasks.rados.rados.0.smithi107.stdout:2691: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:45.772 INFO:tasks.rados.rados.0.smithi107.stdout:2691: finishing write tid 5 to smithi10729407-11 2020-06-12T02:37:45.773 INFO:tasks.rados.rados.0.smithi107.stdout:2694: done (1 left) 2020-06-12T02:37:45.774 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:37:45.774 INFO:tasks.rados.rados.0.smithi107.stdout:2695: seq_num 2694 ranges {1761280=729088,2490368=786432,3276800=417792} 2020-06-12T02:37:45.789 INFO:tasks.rados.rados.0.smithi107.stdout:2695: writing smithi10729407-18 from 1761280 to 2490368 tid 1 2020-06-12T02:37:45.797 INFO:tasks.rados.rados.0.smithi107.stdout:2695: writing smithi10729407-18 from 2490368 to 3276800 tid 2 2020-06-12T02:37:45.798 INFO:tasks.rados.rados.0.smithi107.stdout:2695: writing smithi10729407-18 from 3276800 to 3694592 tid 3 2020-06-12T02:37:45.798 INFO:tasks.rados.rados.0.smithi107.stdout:2696: delete oid 6 current snap is 0 2020-06-12T02:37:45.799 INFO:tasks.rados.rados.0.smithi107.stdout:2691: finishing write tid 6 to smithi10729407-11 2020-06-12T02:37:45.799 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 464 (ObjNum 2690 snap 0 seq_num 2690) dirty exists 2020-06-12T02:37:45.799 INFO:tasks.rados.rados.0.smithi107.stdout:2691: left oid 11 (ObjNum 2690 snap 0 seq_num 2690) 2020-06-12T02:37:45.799 INFO:tasks.rados.rados.0.smithi107.stdout:2695: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:45.801 INFO:tasks.rados.rados.0.smithi107.stdout:2691: done (2 left) 2020-06-12T02:37:45.801 INFO:tasks.rados.rados.0.smithi107.stdout:2696: done (1 left) 2020-06-12T02:37:45.801 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:37:45.802 INFO:tasks.rados.rados.0.smithi107.stdout:2697: seq_num 2696 ranges {1859584=761856,2621440=548864,3170304=655360} 2020-06-12T02:37:45.807 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:45.804+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:45.815 INFO:tasks.rados.rados.0.smithi107.stdout:2697: writing smithi10729407-11 from 1859584 to 2621440 tid 1 2020-06-12T02:37:45.818 INFO:tasks.rados.rados.0.smithi107.stdout:2697: writing smithi10729407-11 from 2621440 to 3170304 tid 2 2020-06-12T02:37:45.819 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:45.819 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:45.824 INFO:tasks.rados.rados.0.smithi107.stdout:2697: writing smithi10729407-11 from 3170304 to 3825664 tid 3 2020-06-12T02:37:45.824 INFO:tasks.rados.rados.0.smithi107.stdout:2698: delete oid 13 current snap is 0 2020-06-12T02:37:45.824 INFO:tasks.rados.rados.0.smithi107.stdout:2695: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:45.824 INFO:tasks.rados.rados.0.smithi107.stdout:2695: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:45.825 INFO:tasks.rados.rados.0.smithi107.stdout:2695: finishing write tid 5 to smithi10729407-18 2020-06-12T02:37:45.825 INFO:tasks.rados.rados.0.smithi107.stdout:2695: finishing write tid 6 to smithi10729407-18 2020-06-12T02:37:45.825 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 464 (ObjNum 2694 snap 0 seq_num 2694) dirty exists 2020-06-12T02:37:45.825 INFO:tasks.rados.rados.0.smithi107.stdout:2695: left oid 18 (ObjNum 2694 snap 0 seq_num 2694) 2020-06-12T02:37:45.826 INFO:tasks.rados.rados.0.smithi107.stdout:2697: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:45.826 INFO:tasks.rados.rados.0.smithi107.stdout:2695: done (2 left) 2020-06-12T02:37:45.827 INFO:tasks.rados.rados.0.smithi107.stdout:2698: done (1 left) 2020-06-12T02:37:45.827 INFO:tasks.rados.rados.0.smithi107.stdout:2699: delete oid 1 current snap is 0 2020-06-12T02:37:45.832 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-06-12T02:37:45.833 INFO:tasks.rados.rados.0.smithi107.stdout:2697: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:45.834 INFO:tasks.rados.rados.0.smithi107.stdout:2697: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:45.837 INFO:tasks.rados.rados.0.smithi107.stdout:2697: finishing write tid 5 to smithi10729407-11 2020-06-12T02:37:45.837 INFO:tasks.rados.rados.0.smithi107.stdout:2699: done (1 left) 2020-06-12T02:37:45.837 INFO:tasks.rados.rados.0.smithi107.stdout:2700: delete oid 8 current snap is 0 2020-06-12T02:37:45.837 INFO:tasks.rados.rados.0.smithi107.stdout:2697: finishing write tid 6 to smithi10729407-11 2020-06-12T02:37:45.838 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 468 (ObjNum 2696 snap 0 seq_num 2696) dirty exists 2020-06-12T02:37:45.838 INFO:tasks.rados.rados.0.smithi107.stdout:2697: left oid 11 (ObjNum 2696 snap 0 seq_num 2696) 2020-06-12T02:37:45.838 INFO:tasks.rados.rados.0.smithi107.stdout:2697: done (1 left) 2020-06-12T02:37:45.838 INFO:tasks.rados.rados.0.smithi107.stdout:2700: done (0 left) 2020-06-12T02:37:45.838 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:45.839 INFO:tasks.rados.rados.0.smithi107.stdout:2701: seq_num 2700 ranges {1695744=425984,2121728=540672,2662400=753664,3416064=311296} 2020-06-12T02:37:45.849 INFO:tasks.rados.rados.0.smithi107.stdout:2701: writing smithi10729407-2 from 1695744 to 2121728 tid 1 2020-06-12T02:37:45.853 INFO:tasks.rados.rados.0.smithi107.stdout:2701: writing smithi10729407-2 from 2121728 to 2662400 tid 2 2020-06-12T02:37:45.858 INFO:tasks.rados.rados.0.smithi107.stdout:2701: writing smithi10729407-2 from 2662400 to 3416064 tid 3 2020-06-12T02:37:45.859 INFO:tasks.rados.rados.0.smithi107.stdout:2701: writing smithi10729407-2 from 3416064 to 3727360 tid 4 2020-06-12T02:37:45.860 INFO:tasks.rados.rados.0.smithi107.stdout:2702: delete oid 7 current snap is 0 2020-06-12T02:37:45.860 INFO:tasks.rados.rados.0.smithi107.stdout:2701: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:45.862 INFO:tasks.rados.rados.0.smithi107.stdout:2702: done (1 left) 2020-06-12T02:37:45.862 INFO:tasks.rados.rados.0.smithi107.stdout:2703: delete oid 5 current snap is 0 2020-06-12T02:37:45.863 INFO:tasks.rados.rados.0.smithi107.stdout:2703: done (1 left) 2020-06-12T02:37:45.864 INFO:tasks.rados.rados.0.smithi107.stdout:2704: delete oid 4 current snap is 0 2020-06-12T02:37:45.866 INFO:tasks.rados.rados.0.smithi107.stdout:2701: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:45.867 INFO:tasks.rados.rados.0.smithi107.stdout:2701: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:45.867 INFO:tasks.rados.rados.0.smithi107.stdout:2701: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:45.868 INFO:tasks.rados.rados.0.smithi107.stdout:2701: finishing write tid 6 to smithi10729407-2 2020-06-12T02:37:45.868 INFO:tasks.rados.rados.0.smithi107.stdout:2701: finishing write tid 7 to smithi10729407-2 2020-06-12T02:37:45.868 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 286 (ObjNum 2700 snap 0 seq_num 2700) dirty exists 2020-06-12T02:37:45.869 INFO:tasks.rados.rados.0.smithi107.stdout:2701: left oid 2 (ObjNum 2700 snap 0 seq_num 2700) 2020-06-12T02:37:45.871 INFO:tasks.rados.rados.0.smithi107.stdout:2701: done (1 left) 2020-06-12T02:37:45.872 INFO:tasks.rados.rados.0.smithi107.stdout:2704: done (0 left) 2020-06-12T02:37:45.872 INFO:tasks.rados.rados.0.smithi107.stdout:2705: delete oid 13 current snap is 0 2020-06-12T02:37:45.875 INFO:tasks.rados.rados.0.smithi107.stdout:2705: done (0 left) 2020-06-12T02:37:45.876 INFO:tasks.rados.rados.0.smithi107.stdout:2706: delete oid 2 current snap is 0 2020-06-12T02:37:45.877 INFO:tasks.rados.rados.0.smithi107.stdout:2706: done (0 left) 2020-06-12T02:37:45.877 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:37:45.878 INFO:tasks.rados.rados.0.smithi107.stdout:2707: seq_num 2706 ranges {0=524288,524288=647168,1171456=696320,1867776=262144} 2020-06-12T02:37:45.883 INFO:tasks.rados.rados.0.smithi107.stdout:2707: writing smithi10729407-6 from 0 to 524288 tid 1 2020-06-12T02:37:45.885 INFO:tasks.rados.rados.0.smithi107.stdout:2707: writing smithi10729407-6 from 524288 to 1171456 tid 2 2020-06-12T02:37:45.889 INFO:tasks.rados.rados.0.smithi107.stdout:2707: writing smithi10729407-6 from 1171456 to 1867776 tid 3 2020-06-12T02:37:45.891 INFO:tasks.rados.rados.0.smithi107.stdout:2707: writing smithi10729407-6 from 1867776 to 2129920 tid 4 2020-06-12T02:37:45.891 INFO:tasks.rados.rados.0.smithi107.stdout:2708: delete oid 4 current snap is 0 2020-06-12T02:37:45.892 INFO:tasks.rados.rados.0.smithi107.stdout:2707: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:45.893 INFO:tasks.rados.rados.0.smithi107.stdout:2708: done (1 left) 2020-06-12T02:37:45.893 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:37:45.893 INFO:tasks.rados.rados.0.smithi107.stdout:2709: seq_num 2708 ranges {0=557056,557056=458752,1015808=712704,1728512=434176} 2020-06-12T02:37:45.896 INFO:tasks.rados.rados.0.smithi107.stdout:2709: writing smithi10729407-13 from 0 to 557056 tid 1 2020-06-12T02:37:45.899 INFO:tasks.rados.rados.0.smithi107.stdout:2709: writing smithi10729407-13 from 557056 to 1015808 tid 2 2020-06-12T02:37:45.903 INFO:tasks.rados.rados.0.smithi107.stdout:2709: writing smithi10729407-13 from 1015808 to 1728512 tid 3 2020-06-12T02:37:45.906 INFO:tasks.rados.rados.0.smithi107.stdout:2709: writing smithi10729407-13 from 1728512 to 2162688 tid 4 2020-06-12T02:37:45.906 INFO:tasks.rados.rados.0.smithi107.stdout:2710: delete oid 7 current snap is 0 2020-06-12T02:37:45.907 INFO:tasks.rados.rados.0.smithi107.stdout:2707: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:45.907 INFO:tasks.rados.rados.0.smithi107.stdout:2707: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:45.907 INFO:tasks.rados.rados.0.smithi107.stdout:2707: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:45.908 INFO:tasks.rados.rados.0.smithi107.stdout:2707: finishing write tid 6 to smithi10729407-6 2020-06-12T02:37:45.908 INFO:tasks.rados.rados.0.smithi107.stdout:2707: finishing write tid 7 to smithi10729407-6 2020-06-12T02:37:45.908 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 553 (ObjNum 2706 snap 0 seq_num 2706) dirty exists 2020-06-12T02:37:45.909 INFO:tasks.rados.rados.0.smithi107.stdout:2707: left oid 6 (ObjNum 2706 snap 0 seq_num 2706) 2020-06-12T02:37:45.909 INFO:tasks.rados.rados.0.smithi107.stdout:2709: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:45.911 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:45.908+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:45.912 INFO:tasks.rados.rados.0.smithi107.stdout:2709: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:45.916 INFO:tasks.rados.rados.0.smithi107.stdout:2709: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:45.916 INFO:tasks.rados.rados.0.smithi107.stdout:2709: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:45.917 INFO:tasks.rados.rados.0.smithi107.stdout:2709: finishing write tid 6 to smithi10729407-13 2020-06-12T02:37:45.917 INFO:tasks.rados.rados.0.smithi107.stdout:2707: done (2 left) 2020-06-12T02:37:45.918 INFO:tasks.rados.rados.0.smithi107.stdout:2710: done (1 left) 2020-06-12T02:37:45.918 INFO:tasks.rados.rados.0.smithi107.stdout:2711: delete oid 20 current snap is 0 2020-06-12T02:37:45.918 INFO:tasks.rados.rados.0.smithi107.stdout:2709: finishing write tid 7 to smithi10729407-13 2020-06-12T02:37:45.918 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 529 (ObjNum 2708 snap 0 seq_num 2708) dirty exists 2020-06-12T02:37:45.919 INFO:tasks.rados.rados.0.smithi107.stdout:2709: left oid 13 (ObjNum 2708 snap 0 seq_num 2708) 2020-06-12T02:37:45.919 INFO:tasks.rados.rados.0.smithi107.stdout:2709: done (1 left) 2020-06-12T02:37:45.919 INFO:tasks.rados.rados.0.smithi107.stdout:2711: done (0 left) 2020-06-12T02:37:45.919 INFO:tasks.rados.rados.0.smithi107.stdout:2712: delete oid 8 current snap is 0 2020-06-12T02:37:45.921 INFO:tasks.rados.rados.0.smithi107.stdout:2712: done (0 left) 2020-06-12T02:37:45.921 INFO:tasks.rados.rados.0.smithi107.stdout:2713: delete oid 13 current snap is 0 2020-06-12T02:37:45.923 INFO:tasks.rados.rados.0.smithi107.stdout:2713: done (0 left) 2020-06-12T02:37:45.924 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:37:45.924 INFO:tasks.rados.rados.0.smithi107.stdout:2714: seq_num 2713 ranges {0=647168,647168=688128,1335296=450560,1785856=466944} 2020-06-12T02:37:45.928 INFO:tasks.rados.rados.0.smithi107.stdout:2714: writing smithi10729407-15 from 0 to 647168 tid 1 2020-06-12T02:37:45.934 INFO:tasks.rados.rados.0.smithi107.stdout:2714: writing smithi10729407-15 from 647168 to 1335296 tid 2 2020-06-12T02:37:45.934 INFO:tasks.rados.rados.0.smithi107.stdout:2714: writing smithi10729407-15 from 1335296 to 1785856 tid 3 2020-06-12T02:37:45.937 INFO:tasks.rados.rados.0.smithi107.stdout:2714: writing smithi10729407-15 from 1785856 to 2252800 tid 4 2020-06-12T02:37:45.937 INFO:tasks.rados.rados.0.smithi107.stdout:2715: delete oid 5 current snap is 0 2020-06-12T02:37:45.938 INFO:tasks.rados.rados.0.smithi107.stdout:2714: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:45.939 INFO:tasks.rados.rados.0.smithi107.stdout:2715: done (1 left) 2020-06-12T02:37:45.939 INFO:tasks.rados.rados.0.smithi107.stdout:2716: delete oid 12 current snap is 0 2020-06-12T02:37:45.940 INFO:tasks.rados.rados.0.smithi107.stdout:2714: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:45.943 INFO:tasks.rados.rados.0.smithi107.stdout:2716: done (1 left) 2020-06-12T02:37:45.943 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:37:45.944 INFO:tasks.rados.rados.0.smithi107.stdout:2717: seq_num 2716 ranges {0=696320,696320=786432,1482752=671744,2154496=147456} 2020-06-12T02:37:45.946 INFO:tasks.rados.rados.0.smithi107.stdout:2717: writing smithi10729407-8 from 0 to 696320 tid 1 2020-06-12T02:37:45.951 INFO:tasks.rados.rados.0.smithi107.stdout:2717: writing smithi10729407-8 from 696320 to 1482752 tid 2 2020-06-12T02:37:45.955 INFO:tasks.rados.rados.0.smithi107.stdout:2717: writing smithi10729407-8 from 1482752 to 2154496 tid 3 2020-06-12T02:37:45.956 INFO:tasks.rados.rados.0.smithi107.stdout:2717: writing smithi10729407-8 from 2154496 to 2301952 tid 4 2020-06-12T02:37:45.957 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:37:45.957 INFO:tasks.rados.rados.0.smithi107.stdout:2718: seq_num 2717 ranges {0=712704,712704=671744,1384448=434176,1818624=499712} 2020-06-12T02:37:45.962 INFO:tasks.rados.rados.0.smithi107.stdout:2718: writing smithi10729407-2 from 0 to 712704 tid 1 2020-06-12T02:37:45.966 INFO:tasks.rados.rados.0.smithi107.stdout:2718: writing smithi10729407-2 from 712704 to 1384448 tid 2 2020-06-12T02:37:45.969 INFO:tasks.rados.rados.0.smithi107.stdout:2718: writing smithi10729407-2 from 1384448 to 1818624 tid 3 2020-06-12T02:37:45.972 INFO:tasks.rados.rados.0.smithi107.stdout:2718: writing smithi10729407-2 from 1818624 to 2318336 tid 4 2020-06-12T02:37:45.972 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:37:45.972 INFO:tasks.rados.rados.0.smithi107.stdout:2719: seq_num 2718 ranges {1105920=729088,1835008=557056,2392064=638976,3031040=409600} 2020-06-12T02:37:45.980 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:45.980 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:45.982 INFO:tasks.rados.rados.0.smithi107.stdout:2719: writing smithi10729407-19 from 1105920 to 1835008 tid 1 2020-06-12T02:37:45.992 INFO:tasks.rados.rados.0.smithi107.stdout:2719: writing smithi10729407-19 from 1835008 to 2392064 tid 2 2020-06-12T02:37:45.993 INFO:tasks.rados.rados.0.smithi107.stdout:2719: writing smithi10729407-19 from 2392064 to 3031040 tid 3 2020-06-12T02:37:45.993 INFO:tasks.rados.rados.0.smithi107.stdout:2719: writing smithi10729407-19 from 3031040 to 3440640 tid 4 2020-06-12T02:37:45.993 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:37:45.993 INFO:tasks.rados.rados.0.smithi107.stdout:2720: seq_num 2719 ranges {0=745472,745472=442368,1187840=442368,1630208=720896} 2020-06-12T02:37:45.994 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-06-12T02:37:45.997 INFO:tasks.rados.rados.0.smithi107.stdout:2720: writing smithi10729407-4 from 0 to 745472 tid 1 2020-06-12T02:37:45.999 INFO:tasks.rados.rados.0.smithi107.stdout:2720: writing smithi10729407-4 from 745472 to 1187840 tid 2 2020-06-12T02:37:46.002 INFO:tasks.rados.rados.0.smithi107.stdout:2720: writing smithi10729407-4 from 1187840 to 1630208 tid 3 2020-06-12T02:37:46.007 INFO:tasks.rados.rados.0.smithi107.stdout:2720: writing smithi10729407-4 from 1630208 to 2351104 tid 4 2020-06-12T02:37:46.007 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:37:46.007 INFO:tasks.rados.rados.0.smithi107.stdout:2714: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:46.007 INFO:tasks.rados.rados.0.smithi107.stdout:2721: seq_num 2720 ranges {0=761856,761856=729088,1490944=655360,2146304=221184} 2020-06-12T02:37:46.012 INFO:tasks.rados.rados.0.smithi107.stdout:2721: writing smithi10729407-12 from 0 to 761856 tid 1 2020-06-12T02:37:46.012 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:37:46.009+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:37:46.015 INFO:tasks.rados.rados.0.smithi107.stdout:2721: writing smithi10729407-12 from 761856 to 1490944 tid 2 2020-06-12T02:37:46.019 INFO:tasks.rados.rados.0.smithi107.stdout:2721: writing smithi10729407-12 from 1490944 to 2146304 tid 3 2020-06-12T02:37:46.021 INFO:tasks.rados.rados.0.smithi107.stdout:2721: writing smithi10729407-12 from 2146304 to 2367488 tid 4 2020-06-12T02:37:46.021 INFO:tasks.rados.rados.0.smithi107.stdout:2722: delete oid 10 current snap is 0 2020-06-12T02:37:46.021 INFO:tasks.rados.rados.0.smithi107.stdout:2714: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:46.021 INFO:tasks.rados.rados.0.smithi107.stdout:2714: finishing write tid 6 to smithi10729407-15 2020-06-12T02:37:46.022 INFO:tasks.rados.rados.0.smithi107.stdout:2714: finishing write tid 7 to smithi10729407-15 2020-06-12T02:37:46.022 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 458 (ObjNum 2713 snap 0 seq_num 2713) dirty exists 2020-06-12T02:37:46.022 INFO:tasks.rados.rados.0.smithi107.stdout:2714: left oid 15 (ObjNum 2713 snap 0 seq_num 2713) 2020-06-12T02:37:46.022 INFO:tasks.rados.rados.0.smithi107.stdout:2717: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:46.022 INFO:tasks.rados.rados.0.smithi107.stdout:2717: finishing write tid 2 to smithi10729407-8 2020-06-12T02:37:46.023 INFO:tasks.rados.rados.0.smithi107.stdout:2717: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:46.023 INFO:tasks.rados.rados.0.smithi107.stdout:2717: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:46.023 INFO:tasks.rados.rados.0.smithi107.stdout:2717: finishing write tid 6 to smithi10729407-8 2020-06-12T02:37:46.023 INFO:tasks.rados.rados.0.smithi107.stdout:2717: finishing write tid 7 to smithi10729407-8 2020-06-12T02:37:46.023 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 463 (ObjNum 2716 snap 0 seq_num 2716) dirty exists 2020-06-12T02:37:46.023 INFO:tasks.rados.rados.0.smithi107.stdout:2717: left oid 8 (ObjNum 2716 snap 0 seq_num 2716) 2020-06-12T02:37:46.024 INFO:tasks.rados.rados.0.smithi107.stdout:2718: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:46.024 INFO:tasks.rados.rados.0.smithi107.stdout:2718: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:46.024 INFO:tasks.rados.rados.0.smithi107.stdout:2718: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:46.024 INFO:tasks.rados.rados.0.smithi107.stdout:2718: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:46.024 INFO:tasks.rados.rados.0.smithi107.stdout:2718: finishing write tid 6 to smithi10729407-2 2020-06-12T02:37:46.024 INFO:tasks.rados.rados.0.smithi107.stdout:2718: finishing write tid 7 to smithi10729407-2 2020-06-12T02:37:46.025 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 292 (ObjNum 2717 snap 0 seq_num 2717) dirty exists 2020-06-12T02:37:46.025 INFO:tasks.rados.rados.0.smithi107.stdout:2718: left oid 2 (ObjNum 2717 snap 0 seq_num 2717) 2020-06-12T02:37:46.025 INFO:tasks.rados.rados.0.smithi107.stdout:2719: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:46.025 INFO:tasks.rados.rados.0.smithi107.stdout:2719: finishing write tid 2 to smithi10729407-19 2020-06-12T02:37:46.025 INFO:tasks.rados.rados.0.smithi107.stdout:2719: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:46.026 INFO:tasks.rados.rados.0.smithi107.stdout:2719: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:46.026 INFO:tasks.rados.rados.0.smithi107.stdout:2719: finishing write tid 6 to smithi10729407-19 2020-06-12T02:37:46.026 INFO:tasks.rados.rados.0.smithi107.stdout:2720: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:46.026 INFO:tasks.rados.rados.0.smithi107.stdout:2719: finishing write tid 7 to smithi10729407-19 2020-06-12T02:37:46.026 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 433 (ObjNum 2718 snap 0 seq_num 2718) dirty exists 2020-06-12T02:37:46.026 INFO:tasks.rados.rados.0.smithi107.stdout:2719: left oid 19 (ObjNum 2718 snap 0 seq_num 2718) 2020-06-12T02:37:46.027 INFO:tasks.rados.rados.0.smithi107.stdout:2720: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:46.027 INFO:tasks.rados.rados.0.smithi107.stdout:2720: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:46.027 INFO:tasks.rados.rados.0.smithi107.stdout:2720: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:46.027 INFO:tasks.rados.rados.0.smithi107.stdout:2720: finishing write tid 6 to smithi10729407-4 2020-06-12T02:37:46.027 INFO:tasks.rados.rados.0.smithi107.stdout:2720: finishing write tid 7 to smithi10729407-4 2020-06-12T02:37:46.028 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 508 (ObjNum 2719 snap 0 seq_num 2719) dirty exists 2020-06-12T02:37:46.028 INFO:tasks.rados.rados.0.smithi107.stdout:2720: left oid 4 (ObjNum 2719 snap 0 seq_num 2719) 2020-06-12T02:37:46.028 INFO:tasks.rados.rados.0.smithi107.stdout:2721: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:46.032 INFO:tasks.rados.rados.0.smithi107.stdout:2714: done (6 left) 2020-06-12T02:37:46.032 INFO:tasks.rados.rados.0.smithi107.stdout:2717: done (5 left) 2020-06-12T02:37:46.032 INFO:tasks.rados.rados.0.smithi107.stdout:2718: done (4 left) 2020-06-12T02:37:46.033 INFO:tasks.rados.rados.0.smithi107.stdout:2719: done (3 left) 2020-06-12T02:37:46.033 INFO:tasks.rados.rados.0.smithi107.stdout:2720: done (2 left) 2020-06-12T02:37:46.033 INFO:tasks.rados.rados.0.smithi107.stdout:2722: done (1 left) 2020-06-12T02:37:46.033 INFO:tasks.rados.rados.0.smithi107.stdout:2723: delete oid 4 current snap is 0 2020-06-12T02:37:46.033 INFO:tasks.rados.rados.0.smithi107.stdout:2721: finishing write tid 2 to smithi10729407-12 2020-06-12T02:37:46.033 INFO:tasks.rados.rados.0.smithi107.stdout:2721: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:46.034 INFO:tasks.rados.rados.0.smithi107.stdout:2721: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:46.034 INFO:tasks.rados.rados.0.smithi107.stdout:2723: done (1 left) 2020-06-12T02:37:46.034 INFO:tasks.rados.rados.0.smithi107.stdout:2724: delete oid 17 current snap is 0 2020-06-12T02:37:46.034 INFO:tasks.rados.rados.0.smithi107.stdout:2721: finishing write tid 6 to smithi10729407-12 2020-06-12T02:37:46.035 INFO:tasks.rados.rados.0.smithi107.stdout:2721: finishing write tid 7 to smithi10729407-12 2020-06-12T02:37:46.035 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 438 (ObjNum 2720 snap 0 seq_num 2720) dirty exists 2020-06-12T02:37:46.035 INFO:tasks.rados.rados.0.smithi107.stdout:2721: left oid 12 (ObjNum 2720 snap 0 seq_num 2720) 2020-06-12T02:37:46.036 INFO:tasks.rados.rados.0.smithi107.stdout:2721: done (1 left) 2020-06-12T02:37:46.036 INFO:tasks.rados.rados.0.smithi107.stdout:2724: done (0 left) 2020-06-12T02:37:46.036 INFO:tasks.rados.rados.0.smithi107.stdout:2725: delete oid 17 current snap is 0 2020-06-12T02:37:46.038 INFO:tasks.rados.rados.0.smithi107.stdout:2725: done (0 left) 2020-06-12T02:37:46.039 INFO:tasks.rados.rados.0.smithi107.stdout:2726: delete oid 7 current snap is 0 2020-06-12T02:37:46.041 INFO:tasks.rados.rados.0.smithi107.stdout:2726: done (0 left) 2020-06-12T02:37:46.041 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:37:46.042 INFO:tasks.rados.rados.0.smithi107.stdout:2727: seq_num 2726 ranges {0=57344} 2020-06-12T02:37:46.042 INFO:tasks.rados.rados.0.smithi107.stdout:2727: writing smithi10729407-9 from 0 to 57344 tid 1 2020-06-12T02:37:46.042 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:37:46.042 INFO:tasks.rados.rados.0.smithi107.stdout:2728: seq_num 2727 ranges {0=73728} 2020-06-12T02:37:46.042 INFO:tasks.rados.rados.0.smithi107.stdout:2728: writing smithi10729407-10 from 0 to 73728 tid 1 2020-06-12T02:37:46.042 INFO:tasks.rados.rados.0.smithi107.stdout:2729: delete oid 8 current snap is 0 2020-06-12T02:37:46.045 INFO:tasks.rados.rados.0.smithi107.stdout:2729: done (2 left) 2020-06-12T02:37:46.045 INFO:tasks.rados.rados.0.smithi107.stdout:2730: delete oid 11 current snap is 0 2020-06-12T02:37:46.045 INFO:tasks.rados.rados.0.smithi107.stdout:2727: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:46.047 INFO:tasks.rados.rados.0.smithi107.stdout:2728: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:46.048 INFO:tasks.rados.rados.0.smithi107.stdout:2728: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:46.049 INFO:tasks.rados.rados.0.smithi107.stdout:2728: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:46.050 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 263 (ObjNum 2727 snap 0 seq_num 2727) dirty exists 2020-06-12T02:37:46.050 INFO:tasks.rados.rados.0.smithi107.stdout:2728: left oid 10 (ObjNum 2727 snap 0 seq_num 2727) 2020-06-12T02:37:46.051 INFO:tasks.rados.rados.0.smithi107.stdout:2728: done (2 left) 2020-06-12T02:37:46.051 INFO:tasks.rados.rados.0.smithi107.stdout:2730: done (1 left) 2020-06-12T02:37:46.051 INFO:tasks.rados.rados.0.smithi107.stdout:2731: delete oid 13 current snap is 0 2020-06-12T02:37:46.054 INFO:tasks.rados.rados.0.smithi107.stdout:2727: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:46.061 INFO:tasks.rados.rados.0.smithi107.stdout:2727: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:46.062 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 466 (ObjNum 2726 snap 0 seq_num 2726) dirty exists 2020-06-12T02:37:46.062 INFO:tasks.rados.rados.0.smithi107.stdout:2727: left oid 9 (ObjNum 2726 snap 0 seq_num 2726) 2020-06-12T02:37:46.062 INFO:tasks.rados.rados.0.smithi107.stdout:2727: done (1 left) 2020-06-12T02:37:46.062 INFO:tasks.rados.rados.0.smithi107.stdout:2731: done (0 left) 2020-06-12T02:37:46.063 INFO:tasks.rados.rados.0.smithi107.stdout:2732: delete oid 2 current snap is 0 2020-06-12T02:37:46.063 INFO:tasks.rados.rados.0.smithi107.stdout:2732: done (0 left) 2020-06-12T02:37:46.063 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:37:46.063 INFO:tasks.rados.rados.0.smithi107.stdout:2733: seq_num 2732 ranges {3440640=163840} 2020-06-12T02:37:46.078 INFO:tasks.rados.rados.0.smithi107.stdout:2733: writing smithi10729407-19 from 3440640 to 3604480 tid 1 2020-06-12T02:37:46.078 INFO:tasks.rados.rados.0.smithi107.stdout:2734: delete oid 18 current snap is 0 2020-06-12T02:37:46.080 INFO:tasks.rados.rados.0.smithi107.stdout:2734: done (1 left) 2020-06-12T02:37:46.080 INFO:tasks.rados.rados.0.smithi107.stdout:2735: delete oid 17 current snap is 0 2020-06-12T02:37:46.085 INFO:tasks.rados.rados.0.smithi107.stdout:2733: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:46.085 INFO:tasks.rados.rados.0.smithi107.stdout:2735: done (1 left) 2020-06-12T02:37:46.085 INFO:tasks.rados.rados.0.smithi107.stdout:2736: delete oid 20 current snap is 0 2020-06-12T02:37:46.085 INFO:tasks.rados.rados.0.smithi107.stdout:2733: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:46.085 INFO:tasks.rados.rados.0.smithi107.stdout:2733: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:46.086 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 440 (ObjNum 2732 snap 0 seq_num 2732) dirty exists 2020-06-12T02:37:46.086 INFO:tasks.rados.rados.0.smithi107.stdout:2733: left oid 19 (ObjNum 2732 snap 0 seq_num 2732) 2020-06-12T02:37:46.086 INFO:tasks.rados.rados.0.smithi107.stdout:2733: done (1 left) 2020-06-12T02:37:46.086 INFO:tasks.rados.rados.0.smithi107.stdout:2736: done (0 left) 2020-06-12T02:37:46.086 INFO:tasks.rados.rados.0.smithi107.stdout:2737: delete oid 1 current snap is 0 2020-06-12T02:37:46.087 INFO:tasks.rados.rados.0.smithi107.stdout:2737: done (0 left) 2020-06-12T02:37:46.087 INFO:tasks.rados.rados.0.smithi107.stdout:2738: delete oid 6 current snap is 0 2020-06-12T02:37:46.088 INFO:tasks.rados.rados.0.smithi107.stdout:2738: done (0 left) 2020-06-12T02:37:46.088 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:37:46.088 INFO:tasks.rados.rados.0.smithi107.stdout:2739: seq_num 2738 ranges {0=262144} 2020-06-12T02:37:46.090 INFO:tasks.rados.rados.0.smithi107.stdout:2739: writing smithi10729407-17 from 0 to 262144 tid 1 2020-06-12T02:37:46.090 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:37:46.090 INFO:tasks.rados.rados.0.smithi107.stdout:2740: seq_num 2739 ranges {0=278528} 2020-06-12T02:37:46.091 INFO:tasks.rados.rados.0.smithi107.stdout:2740: writing smithi10729407-16 from 0 to 278528 tid 1 2020-06-12T02:37:46.092 INFO:tasks.rados.rados.0.smithi107.stdout:2741: delete oid 10 current snap is 0 2020-06-12T02:37:46.094 INFO:tasks.rados.rados.0.smithi107.stdout:2739: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:46.094 INFO:tasks.rados.rados.0.smithi107.stdout:2739: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:46.095 INFO:tasks.rados.rados.0.smithi107.stdout:2741: done (2 left) 2020-06-12T02:37:46.095 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:37:46.095 INFO:tasks.rados.rados.0.smithi107.stdout:2742: seq_num 2741 ranges {0=311296} 2020-06-12T02:37:46.097 INFO:tasks.rados.rados.0.smithi107.stdout:2742: writing smithi10729407-3 from 0 to 311296 tid 1 2020-06-12T02:37:46.097 INFO:tasks.rados.rados.0.smithi107.stdout:2743: delete oid 4 current snap is 0 2020-06-12T02:37:46.098 INFO:tasks.rados.rados.0.smithi107.stdout:2739: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:46.098 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 514 (ObjNum 2738 snap 0 seq_num 2738) dirty exists 2020-06-12T02:37:46.098 INFO:tasks.rados.rados.0.smithi107.stdout:2739: left oid 17 (ObjNum 2738 snap 0 seq_num 2738) 2020-06-12T02:37:46.099 INFO:tasks.rados.rados.0.smithi107.stdout:2740: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:46.099 INFO:tasks.rados.rados.0.smithi107.stdout:2740: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:46.099 INFO:tasks.rados.rados.0.smithi107.stdout:2739: done (3 left) 2020-06-12T02:37:46.100 INFO:tasks.rados.rados.0.smithi107.stdout:2743: done (2 left) 2020-06-12T02:37:46.100 INFO:tasks.rados.rados.0.smithi107.stdout:2744: delete oid 8 current snap is 0 2020-06-12T02:37:46.101 INFO:tasks.rados.rados.0.smithi107.stdout:2744: done (2 left) 2020-06-12T02:37:46.101 INFO:tasks.rados.rados.0.smithi107.stdout:2745: delete oid 4 current snap is 0 2020-06-12T02:37:46.102 INFO:tasks.rados.rados.0.smithi107.stdout:2740: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:46.103 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 461 (ObjNum 2739 snap 0 seq_num 2739) dirty exists 2020-06-12T02:37:46.103 INFO:tasks.rados.rados.0.smithi107.stdout:2740: left oid 16 (ObjNum 2739 snap 0 seq_num 2739) 2020-06-12T02:37:46.104 INFO:tasks.rados.rados.0.smithi107.stdout:2742: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:46.104 INFO:tasks.rados.rados.0.smithi107.stdout:2740: done (2 left) 2020-06-12T02:37:46.104 INFO:tasks.rados.rados.0.smithi107.stdout:2745: done (1 left) 2020-06-12T02:37:46.104 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:37:46.105 INFO:tasks.rados.rados.0.smithi107.stdout:2746: seq_num 2745 ranges {0=376832} 2020-06-12T02:37:46.106 INFO:tasks.rados.rados.0.smithi107.stdout:2746: writing smithi10729407-1 from 0 to 376832 tid 1 2020-06-12T02:37:46.106 INFO:tasks.rados.rados.0.smithi107.stdout:2747: delete oid 8 current snap is 0 2020-06-12T02:37:46.106 INFO:tasks.rados.rados.0.smithi107.stdout:2742: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:46.108 INFO:tasks.rados.rados.0.smithi107.stdout:2742: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:46.108 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 463 (ObjNum 2741 snap 0 seq_num 2741) dirty exists 2020-06-12T02:37:46.108 INFO:tasks.rados.rados.0.smithi107.stdout:2742: left oid 3 (ObjNum 2741 snap 0 seq_num 2741) 2020-06-12T02:37:46.108 INFO:tasks.rados.rados.0.smithi107.stdout:2742: done (2 left) 2020-06-12T02:37:46.109 INFO:tasks.rados.rados.0.smithi107.stdout:2747: done (1 left) 2020-06-12T02:37:46.109 INFO:tasks.rados.rados.0.smithi107.stdout:2748: delete oid 6 current snap is 0 2020-06-12T02:37:46.110 INFO:tasks.rados.rados.0.smithi107.stdout:2746: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:46.111 INFO:tasks.rados.rados.0.smithi107.stdout:2746: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:46.113 INFO:tasks.rados.rados.0.smithi107.stdout:2746: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:46.113 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 557 (ObjNum 2745 snap 0 seq_num 2745) dirty exists 2020-06-12T02:37:46.113 INFO:tasks.rados.rados.0.smithi107.stdout:2746: left oid 1 (ObjNum 2745 snap 0 seq_num 2745) 2020-06-12T02:37:46.113 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:46.110+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:46.114 INFO:tasks.rados.rados.0.smithi107.stdout:2746: done (1 left) 2020-06-12T02:37:46.114 INFO:tasks.rados.rados.0.smithi107.stdout:2748: done (0 left) 2020-06-12T02:37:46.114 INFO:tasks.rados.rados.0.smithi107.stdout:2749: delete oid 5 current snap is 0 2020-06-12T02:37:46.116 INFO:tasks.rados.rados.0.smithi107.stdout:2749: done (0 left) 2020-06-12T02:37:46.123 INFO:tasks.rados.rados.0.smithi107.stdout:2750: delete oid 4 current snap is 0 2020-06-12T02:37:46.124 INFO:tasks.rados.rados.0.smithi107.stdout:2750: done (0 left) 2020-06-12T02:37:46.124 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:37:46.124 INFO:tasks.rados.rados.0.smithi107.stdout:2751: seq_num 2750 ranges {0=458752} 2020-06-12T02:37:46.125 INFO:tasks.rados.rados.0.smithi107.stdout:2751: writing smithi10729407-7 from 0 to 458752 tid 1 2020-06-12T02:37:46.125 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:37:46.125 INFO:tasks.rados.rados.0.smithi107.stdout:2752: seq_num 2751 ranges {0=483328} 2020-06-12T02:37:46.125 INFO:tasks.rados.rados.0.smithi107.stdout:2752: writing smithi10729407-2 from 0 to 483328 tid 1 2020-06-12T02:37:46.125 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:37:46.126 INFO:tasks.rados.rados.0.smithi107.stdout:2753: seq_num 2752 ranges {57344=499712} 2020-06-12T02:37:46.127 INFO:tasks.rados.rados.0.smithi107.stdout:2753: writing smithi10729407-9 from 57344 to 557056 tid 1 2020-06-12T02:37:46.127 INFO:tasks.rados.rados.0.smithi107.stdout:2754: delete oid 16 current snap is 0 2020-06-12T02:37:46.129 INFO:tasks.rados.rados.0.smithi107.stdout:2751: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:46.129 INFO:tasks.rados.rados.0.smithi107.stdout:2751: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:46.130 INFO:tasks.rados.rados.0.smithi107.stdout:2752: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:46.130 INFO:tasks.rados.rados.0.smithi107.stdout:2752: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:46.131 INFO:tasks.rados.rados.0.smithi107.stdout:2751: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:46.131 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 535 (ObjNum 2750 snap 0 seq_num 2750) dirty exists 2020-06-12T02:37:46.131 INFO:tasks.rados.rados.0.smithi107.stdout:2751: left oid 7 (ObjNum 2750 snap 0 seq_num 2750) 2020-06-12T02:37:46.132 INFO:tasks.rados.rados.0.smithi107.stdout:2752: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:46.132 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 295 (ObjNum 2751 snap 0 seq_num 2751) dirty exists 2020-06-12T02:37:46.132 INFO:tasks.rados.rados.0.smithi107.stdout:2752: left oid 2 (ObjNum 2751 snap 0 seq_num 2751) 2020-06-12T02:37:46.134 INFO:tasks.rados.rados.0.smithi107.stdout:2751: done (3 left) 2020-06-12T02:37:46.134 INFO:tasks.rados.rados.0.smithi107.stdout:2752: done (2 left) 2020-06-12T02:37:46.134 INFO:tasks.rados.rados.0.smithi107.stdout:2754: done (1 left) 2020-06-12T02:37:46.135 INFO:tasks.rados.rados.0.smithi107.stdout:2755: delete oid 18 current snap is 0 2020-06-12T02:37:46.136 INFO:tasks.rados.rados.0.smithi107.stdout:2753: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:46.140 INFO:tasks.rados.rados.0.smithi107.stdout:2755: done (1 left) 2020-06-12T02:37:46.141 INFO:tasks.rados.rados.0.smithi107.stdout:2756: delete oid 5 current snap is 0 2020-06-12T02:37:46.141 INFO:tasks.rados.rados.0.smithi107.stdout:2753: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:46.141 INFO:tasks.rados.rados.0.smithi107.stdout:2753: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:46.142 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 470 (ObjNum 2752 snap 0 seq_num 2752) dirty exists 2020-06-12T02:37:46.142 INFO:tasks.rados.rados.0.smithi107.stdout:2753: left oid 9 (ObjNum 2752 snap 0 seq_num 2752) 2020-06-12T02:37:46.143 INFO:tasks.rados.rados.0.smithi107.stdout:2753: done (1 left) 2020-06-12T02:37:46.143 INFO:tasks.rados.rados.0.smithi107.stdout:2756: done (0 left) 2020-06-12T02:37:46.143 INFO:tasks.rados.rados.0.smithi107.stdout:2757: delete oid 13 current snap is 0 2020-06-12T02:37:46.146 INFO:tasks.rados.rados.0.smithi107.stdout:2757: done (0 left) 2020-06-12T02:37:46.146 INFO:tasks.rados.rados.0.smithi107.stdout:2758: delete oid 14 current snap is 0 2020-06-12T02:37:46.147 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:46.147 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:46.148 INFO:tasks.rados.rados.0.smithi107.stdout:2758: done (0 left) 2020-06-12T02:37:46.148 INFO:tasks.rados.rados.0.smithi107.stdout:2759: delete oid 10 current snap is 0 2020-06-12T02:37:46.149 INFO:tasks.rados.rados.0.smithi107.stdout:2759: done (0 left) 2020-06-12T02:37:46.150 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:37:46.150 INFO:tasks.rados.rados.0.smithi107.stdout:2760: seq_num 2759 ranges {3604480=614400} 2020-06-12T02:37:46.157 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-06-12T02:37:46.173 INFO:tasks.rados.rados.0.smithi107.stdout:2760: writing smithi10729407-19 from 3604480 to 4218880 tid 1 2020-06-12T02:37:46.173 INFO:tasks.rados.rados.0.smithi107.stdout:2761: delete oid 4 current snap is 0 2020-06-12T02:37:46.175 INFO:tasks.rados.rados.0.smithi107.stdout:2761: done (1 left) 2020-06-12T02:37:46.175 INFO:tasks.rados.rados.0.smithi107.stdout:2762: delete oid 16 current snap is 0 2020-06-12T02:37:46.177 INFO:tasks.rados.rados.0.smithi107.stdout:2762: done (1 left) 2020-06-12T02:37:46.178 INFO:tasks.rados.rados.0.smithi107.stdout:2763: delete oid 18 current snap is 0 2020-06-12T02:37:46.178 INFO:tasks.rados.rados.0.smithi107.stdout:2760: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:46.178 INFO:tasks.rados.rados.0.smithi107.stdout:2760: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:46.179 INFO:tasks.rados.rados.0.smithi107.stdout:2760: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:46.179 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 442 (ObjNum 2759 snap 0 seq_num 2759) dirty exists 2020-06-12T02:37:46.179 INFO:tasks.rados.rados.0.smithi107.stdout:2760: left oid 19 (ObjNum 2759 snap 0 seq_num 2759) 2020-06-12T02:37:46.180 INFO:tasks.rados.rados.0.smithi107.stdout:2760: done (1 left) 2020-06-12T02:37:46.180 INFO:tasks.rados.rados.0.smithi107.stdout:2763: done (0 left) 2020-06-12T02:37:46.180 INFO:tasks.rados.rados.0.smithi107.stdout:2764: delete oid 6 current snap is 0 2020-06-12T02:37:46.182 INFO:tasks.rados.rados.0.smithi107.stdout:2764: done (0 left) 2020-06-12T02:37:46.182 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:37:46.182 INFO:tasks.rados.rados.0.smithi107.stdout:2765: seq_num 2764 ranges {376832=696320} 2020-06-12T02:37:46.188 INFO:tasks.rados.rados.0.smithi107.stdout:2765: writing smithi10729407-1 from 376832 to 1073152 tid 1 2020-06-12T02:37:46.188 INFO:tasks.rados.rados.0.smithi107.stdout:2766: delete oid 17 current snap is 0 2020-06-12T02:37:46.190 INFO:tasks.rados.rados.0.smithi107.stdout:2766: done (1 left) 2020-06-12T02:37:46.190 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:37:46.190 INFO:tasks.rados.rados.0.smithi107.stdout:2767: seq_num 2766 ranges {0=729088} 2020-06-12T02:37:46.194 INFO:tasks.rados.rados.0.smithi107.stdout:2767: writing smithi10729407-4 from 0 to 729088 tid 1 2020-06-12T02:37:46.195 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:37:46.195 INFO:tasks.rados.rados.0.smithi107.stdout:2768: seq_num 2767 ranges {0=745472} 2020-06-12T02:37:46.199 INFO:tasks.rados.rados.0.smithi107.stdout:2768: writing smithi10729407-5 from 0 to 745472 tid 1 2020-06-12T02:37:46.199 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:37:46.200 INFO:tasks.rados.rados.0.smithi107.stdout:2765: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:46.200 INFO:tasks.rados.rados.0.smithi107.stdout:2769: seq_num 2768 ranges {2252800=761856} 2020-06-12T02:37:46.213 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:46.210+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:46.216 INFO:tasks.rados.rados.0.smithi107.stdout:2769: writing smithi10729407-15 from 2252800 to 3014656 tid 1 2020-06-12T02:37:46.216 INFO:tasks.rados.rados.0.smithi107.stdout:2770: delete oid 19 current snap is 0 2020-06-12T02:37:46.217 INFO:tasks.rados.rados.0.smithi107.stdout:2765: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:46.217 INFO:tasks.rados.rados.0.smithi107.stdout:2765: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:46.217 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 561 (ObjNum 2764 snap 0 seq_num 2764) dirty exists 2020-06-12T02:37:46.217 INFO:tasks.rados.rados.0.smithi107.stdout:2765: left oid 1 (ObjNum 2764 snap 0 seq_num 2764) 2020-06-12T02:37:46.218 INFO:tasks.rados.rados.0.smithi107.stdout:2767: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:46.218 INFO:tasks.rados.rados.0.smithi107.stdout:2767: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:46.218 INFO:tasks.rados.rados.0.smithi107.stdout:2768: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:46.218 INFO:tasks.rados.rados.0.smithi107.stdout:2767: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:46.218 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 521 (ObjNum 2766 snap 0 seq_num 2766) dirty exists 2020-06-12T02:37:46.219 INFO:tasks.rados.rados.0.smithi107.stdout:2767: left oid 4 (ObjNum 2766 snap 0 seq_num 2766) 2020-06-12T02:37:46.219 INFO:tasks.rados.rados.0.smithi107.stdout:2768: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:46.219 INFO:tasks.rados.rados.0.smithi107.stdout:2768: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:46.219 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 234 (ObjNum 2767 snap 0 seq_num 2767) dirty exists 2020-06-12T02:37:46.219 INFO:tasks.rados.rados.0.smithi107.stdout:2768: left oid 5 (ObjNum 2767 snap 0 seq_num 2767) 2020-06-12T02:37:46.220 INFO:tasks.rados.rados.0.smithi107.stdout:2765: done (4 left) 2020-06-12T02:37:46.220 INFO:tasks.rados.rados.0.smithi107.stdout:2767: done (3 left) 2020-06-12T02:37:46.220 INFO:tasks.rados.rados.0.smithi107.stdout:2768: done (2 left) 2020-06-12T02:37:46.220 INFO:tasks.rados.rados.0.smithi107.stdout:2770: done (1 left) 2020-06-12T02:37:46.221 INFO:tasks.rados.rados.0.smithi107.stdout:2771: delete oid 10 current snap is 0 2020-06-12T02:37:46.221 INFO:tasks.rados.rados.0.smithi107.stdout:2771: done (1 left) 2020-06-12T02:37:46.222 INFO:tasks.rados.rados.0.smithi107.stdout:2772: delete oid 7 current snap is 0 2020-06-12T02:37:46.223 INFO:tasks.rados.rados.0.smithi107.stdout:2772: done (1 left) 2020-06-12T02:37:46.223 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:37:46.223 INFO:tasks.rados.rados.0.smithi107.stdout:2773: seq_num 2772 ranges {0=434176,434176=401408} 2020-06-12T02:37:46.226 INFO:tasks.rados.rados.0.smithi107.stdout:2773: writing smithi10729407-10 from 0 to 434176 tid 1 2020-06-12T02:37:46.228 INFO:tasks.rados.rados.0.smithi107.stdout:2773: writing smithi10729407-10 from 434176 to 835584 tid 2 2020-06-12T02:37:46.228 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:37:46.229 INFO:tasks.rados.rados.0.smithi107.stdout:2774: seq_num 2773 ranges {1073152=450560,1523712=401408} 2020-06-12T02:37:46.237 INFO:tasks.rados.rados.0.smithi107.stdout:2774: writing smithi10729407-1 from 1073152 to 1523712 tid 1 2020-06-12T02:37:46.239 INFO:tasks.rados.rados.0.smithi107.stdout:2774: writing smithi10729407-1 from 1523712 to 1925120 tid 2 2020-06-12T02:37:46.240 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:37:46.240 INFO:tasks.rados.rados.0.smithi107.stdout:2775: seq_num 2774 ranges {729088=466944,1196032=401408} 2020-06-12T02:37:46.250 INFO:tasks.rados.rados.0.smithi107.stdout:2775: writing smithi10729407-4 from 729088 to 1196032 tid 1 2020-06-12T02:37:46.250 INFO:tasks.rados.rados.0.smithi107.stdout:2775: writing smithi10729407-4 from 1196032 to 1597440 tid 2 2020-06-12T02:37:46.250 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:37:46.250 INFO:tasks.rados.rados.0.smithi107.stdout:2776: seq_num 2775 ranges {2367488=483328,2850816=401408} 2020-06-12T02:37:46.264 INFO:tasks.rados.rados.0.smithi107.stdout:2776: writing smithi10729407-12 from 2367488 to 2850816 tid 1 2020-06-12T02:37:46.267 INFO:tasks.rados.rados.0.smithi107.stdout:2776: writing smithi10729407-12 from 2850816 to 3252224 tid 2 2020-06-12T02:37:46.267 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:37:46.267 INFO:tasks.rados.rados.0.smithi107.stdout:2777: seq_num 2776 ranges {311296=499712,811008=401408} 2020-06-12T02:37:46.271 INFO:tasks.rados.rados.0.smithi107.stdout:2777: writing smithi10729407-3 from 311296 to 811008 tid 1 2020-06-12T02:37:46.274 INFO:tasks.rados.rados.0.smithi107.stdout:2777: writing smithi10729407-3 from 811008 to 1212416 tid 2 2020-06-12T02:37:46.275 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:37:46.275 INFO:tasks.rados.rados.0.smithi107.stdout:2778: seq_num 2777 ranges {0=516096,516096=401408} 2020-06-12T02:37:46.277 INFO:tasks.rados.rados.0.smithi107.stdout:2778: writing smithi10729407-20 from 0 to 516096 tid 1 2020-06-12T02:37:46.280 INFO:tasks.rados.rados.0.smithi107.stdout:2778: writing smithi10729407-20 from 516096 to 917504 tid 2 2020-06-12T02:37:46.280 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:37:46.280 INFO:tasks.rados.rados.0.smithi107.stdout:2779: seq_num 2778 ranges {0=532480,532480=401408} 2020-06-12T02:37:46.283 INFO:tasks.rados.rados.0.smithi107.stdout:2779: writing smithi10729407-19 from 0 to 532480 tid 1 2020-06-12T02:37:46.285 INFO:tasks.rados.rados.0.smithi107.stdout:2779: writing smithi10729407-19 from 532480 to 933888 tid 2 2020-06-12T02:37:46.286 INFO:tasks.rados.rados.0.smithi107.stdout: waiting on 8 2020-06-12T02:37:46.292 INFO:tasks.rados.rados.0.smithi107.stdout:2769: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:46.293 INFO:tasks.rados.rados.0.smithi107.stdout:2769: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:46.293 INFO:tasks.rados.rados.0.smithi107.stdout:2769: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:46.293 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 468 (ObjNum 2768 snap 0 seq_num 2768) dirty exists 2020-06-12T02:37:46.293 INFO:tasks.rados.rados.0.smithi107.stdout:2769: left oid 15 (ObjNum 2768 snap 0 seq_num 2768) 2020-06-12T02:37:46.293 INFO:tasks.rados.rados.0.smithi107.stdout:2773: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:46.294 INFO:tasks.rados.rados.0.smithi107.stdout:2773: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:46.294 INFO:tasks.rados.rados.0.smithi107.stdout:2773: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:46.294 INFO:tasks.rados.rados.0.smithi107.stdout:2773: finishing write tid 5 to smithi10729407-10 2020-06-12T02:37:46.294 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 269 (ObjNum 2772 snap 0 seq_num 2772) dirty exists 2020-06-12T02:37:46.294 INFO:tasks.rados.rados.0.smithi107.stdout:2773: left oid 10 (ObjNum 2772 snap 0 seq_num 2772) 2020-06-12T02:37:46.295 INFO:tasks.rados.rados.0.smithi107.stdout:2774: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:46.295 INFO:tasks.rados.rados.0.smithi107.stdout:2774: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:46.295 INFO:tasks.rados.rados.0.smithi107.stdout:2774: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:46.295 INFO:tasks.rados.rados.0.smithi107.stdout:2774: finishing write tid 5 to smithi10729407-1 2020-06-12T02:37:46.295 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 564 (ObjNum 2773 snap 0 seq_num 2773) dirty exists 2020-06-12T02:37:46.296 INFO:tasks.rados.rados.0.smithi107.stdout:2774: left oid 1 (ObjNum 2773 snap 0 seq_num 2773) 2020-06-12T02:37:46.296 INFO:tasks.rados.rados.0.smithi107.stdout:2775: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:46.296 INFO:tasks.rados.rados.0.smithi107.stdout:2775: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:46.296 INFO:tasks.rados.rados.0.smithi107.stdout:2775: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:46.296 INFO:tasks.rados.rados.0.smithi107.stdout:2775: finishing write tid 5 to smithi10729407-4 2020-06-12T02:37:46.297 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 524 (ObjNum 2774 snap 0 seq_num 2774) dirty exists 2020-06-12T02:37:46.297 INFO:tasks.rados.rados.0.smithi107.stdout:2775: left oid 4 (ObjNum 2774 snap 0 seq_num 2774) 2020-06-12T02:37:46.297 INFO:tasks.rados.rados.0.smithi107.stdout:2776: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:46.297 INFO:tasks.rados.rados.0.smithi107.stdout:2776: finishing write tid 2 to smithi10729407-12 2020-06-12T02:37:46.297 INFO:tasks.rados.rados.0.smithi107.stdout:2776: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:46.298 INFO:tasks.rados.rados.0.smithi107.stdout:2776: finishing write tid 5 to smithi10729407-12 2020-06-12T02:37:46.298 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 446 (ObjNum 2775 snap 0 seq_num 2775) dirty exists 2020-06-12T02:37:46.298 INFO:tasks.rados.rados.0.smithi107.stdout:2776: left oid 12 (ObjNum 2775 snap 0 seq_num 2775) 2020-06-12T02:37:46.298 INFO:tasks.rados.rados.0.smithi107.stdout:2777: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:46.299 INFO:tasks.rados.rados.0.smithi107.stdout:2777: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:46.299 INFO:tasks.rados.rados.0.smithi107.stdout:2777: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:46.299 INFO:tasks.rados.rados.0.smithi107.stdout:2778: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:46.299 INFO:tasks.rados.rados.0.smithi107.stdout:2769: done (7 left) 2020-06-12T02:37:46.299 INFO:tasks.rados.rados.0.smithi107.stdout:2773: done (6 left) 2020-06-12T02:37:46.300 INFO:tasks.rados.rados.0.smithi107.stdout:2774: done (5 left) 2020-06-12T02:37:46.300 INFO:tasks.rados.rados.0.smithi107.stdout:2775: done (4 left) 2020-06-12T02:37:46.300 INFO:tasks.rados.rados.0.smithi107.stdout:2776: done (3 left) 2020-06-12T02:37:46.300 INFO:tasks.rados.rados.0.smithi107.stdout:2780: delete oid 11 current snap is 0 2020-06-12T02:37:46.300 INFO:tasks.rados.rados.0.smithi107.stdout:2778: finishing write tid 2 to smithi10729407-20 2020-06-12T02:37:46.301 INFO:tasks.rados.rados.0.smithi107.stdout:2778: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:46.301 INFO:tasks.rados.rados.0.smithi107.stdout:2777: finishing write tid 5 to smithi10729407-3 2020-06-12T02:37:46.301 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 468 (ObjNum 2776 snap 0 seq_num 2776) dirty exists 2020-06-12T02:37:46.301 INFO:tasks.rados.rados.0.smithi107.stdout:2777: left oid 3 (ObjNum 2776 snap 0 seq_num 2776) 2020-06-12T02:37:46.302 INFO:tasks.rados.rados.0.smithi107.stdout:2777: done (3 left) 2020-06-12T02:37:46.302 INFO:tasks.rados.rados.0.smithi107.stdout:2780: done (2 left) 2020-06-12T02:37:46.302 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:37:46.302 INFO:tasks.rados.rados.0.smithi107.stdout:2781: seq_num 2780 ranges {3252224=565248,3817472=401408} 2020-06-12T02:37:46.310 INFO:tasks.rados.rados.0.smithi107.stdout:2781: writing smithi10729407-12 from 3252224 to 3817472 tid 1 2020-06-12T02:37:46.315 INFO:tasks.rados.rados.0.smithi107.stdout:2781: writing smithi10729407-12 from 3817472 to 4218880 tid 2 2020-06-12T02:37:46.316 INFO:tasks.rados.rados.0.smithi107.stdout:2782: delete oid 17 current snap is 0 2020-06-12T02:37:46.316 INFO:tasks.rados.rados.0.smithi107.stdout:2778: finishing write tid 5 to smithi10729407-20 2020-06-12T02:37:46.316 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 178 (ObjNum 2777 snap 0 seq_num 2777) dirty exists 2020-06-12T02:37:46.316 INFO:tasks.rados.rados.0.smithi107.stdout:2778: left oid 20 (ObjNum 2777 snap 0 seq_num 2777) 2020-06-12T02:37:46.316 INFO:tasks.rados.rados.0.smithi107.stdout:2779: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:46.317 INFO:tasks.rados.rados.0.smithi107.stdout:2779: finishing write tid 2 to smithi10729407-19 2020-06-12T02:37:46.317 INFO:tasks.rados.rados.0.smithi107.stdout:2779: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:46.317 INFO:tasks.rados.rados.0.smithi107.stdout:2779: finishing write tid 5 to smithi10729407-19 2020-06-12T02:37:46.317 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 449 (ObjNum 2778 snap 0 seq_num 2778) dirty exists 2020-06-12T02:37:46.317 INFO:tasks.rados.rados.0.smithi107.stdout:2779: left oid 19 (ObjNum 2778 snap 0 seq_num 2778) 2020-06-12T02:37:46.319 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:46.319 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:46.319 INFO:tasks.rados.rados.0.smithi107.stdout:2778: done (3 left) 2020-06-12T02:37:46.319 INFO:tasks.rados.rados.0.smithi107.stdout:2779: done (2 left) 2020-06-12T02:37:46.320 INFO:tasks.rados.rados.0.smithi107.stdout:2782: done (1 left) 2020-06-12T02:37:46.320 INFO:tasks.rados.rados.0.smithi107.stdout:2783: delete oid 14 current snap is 0 2020-06-12T02:37:46.320 INFO:tasks.rados.rados.0.smithi107.stdout:2781: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:46.320 INFO:tasks.rados.rados.0.smithi107.stdout:2783: done (1 left) 2020-06-12T02:37:46.320 INFO:tasks.rados.rados.0.smithi107.stdout:2784: delete oid 1 current snap is 0 2020-06-12T02:37:46.321 INFO:tasks.rados.rados.0.smithi107.stdout:2781: finishing write tid 2 to smithi10729407-12 2020-06-12T02:37:46.321 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:37:46.316+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:37:46.323 INFO:tasks.rados.rados.0.smithi107.stdout:2784: done (1 left) 2020-06-12T02:37:46.323 INFO:tasks.rados.rados.0.smithi107.stdout:2785: delete oid 15 current snap is 0 2020-06-12T02:37:46.323 INFO:tasks.rados.rados.0.smithi107.stdout:2781: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:46.324 INFO:tasks.rados.rados.0.smithi107.stdout:2781: finishing write tid 5 to smithi10729407-12 2020-06-12T02:37:46.324 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 452 (ObjNum 2780 snap 0 seq_num 2780) dirty exists 2020-06-12T02:37:46.325 INFO:tasks.rados.rados.0.smithi107.stdout:2781: left oid 12 (ObjNum 2780 snap 0 seq_num 2780) 2020-06-12T02:37:46.325 INFO:tasks.rados.rados.0.smithi107.stdout:2781: done (1 left) 2020-06-12T02:37:46.326 INFO:tasks.rados.rados.0.smithi107.stdout:2785: done (0 left) 2020-06-12T02:37:46.326 INFO:tasks.rados.rados.0.smithi107.stdout:2786: delete oid 15 current snap is 0 2020-06-12T02:37:46.327 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-06-12T02:37:46.327 INFO:tasks.rados.rados.0.smithi107.stdout:2786: done (0 left) 2020-06-12T02:37:46.328 INFO:tasks.rados.rados.0.smithi107.stdout:2787: delete oid 9 current snap is 0 2020-06-12T02:37:46.328 INFO:tasks.rados.rados.0.smithi107.stdout:2787: done (0 left) 2020-06-12T02:37:46.334 INFO:tasks.rados.rados.0.smithi107.stdout:2788: delete oid 6 current snap is 0 2020-06-12T02:37:46.334 INFO:tasks.rados.rados.0.smithi107.stdout:2788: done (0 left) 2020-06-12T02:37:46.335 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:37:46.338 INFO:tasks.rados.rados.0.smithi107.stdout:2789: seq_num 2788 ranges {0=696320,696320=401408} 2020-06-12T02:37:46.338 INFO:tasks.rados.rados.0.smithi107.stdout:2789: writing smithi10729407-13 from 0 to 696320 tid 1 2020-06-12T02:37:46.339 INFO:tasks.rados.rados.0.smithi107.stdout:2789: writing smithi10729407-13 from 696320 to 1097728 tid 2 2020-06-12T02:37:46.340 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:37:46.340 INFO:tasks.rados.rados.0.smithi107.stdout:2790: seq_num 2789 ranges {4218880=720896,4939776=401408} 2020-06-12T02:37:46.363 INFO:tasks.rados.rados.0.smithi107.stdout:2790: writing smithi10729407-12 from 4218880 to 4939776 tid 1 2020-06-12T02:37:46.366 INFO:tasks.rados.rados.0.smithi107.stdout:2790: writing smithi10729407-12 from 4939776 to 5341184 tid 2 2020-06-12T02:37:46.367 INFO:tasks.rados.rados.0.smithi107.stdout:2791: delete oid 7 current snap is 0 2020-06-12T02:37:46.367 INFO:tasks.rados.rados.0.smithi107.stdout:2789: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:46.367 INFO:tasks.rados.rados.0.smithi107.stdout:2789: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:46.367 INFO:tasks.rados.rados.0.smithi107.stdout:2789: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:46.367 INFO:tasks.rados.rados.0.smithi107.stdout:2789: finishing write tid 5 to smithi10729407-13 2020-06-12T02:37:46.368 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 540 (ObjNum 2788 snap 0 seq_num 2788) dirty exists 2020-06-12T02:37:46.368 INFO:tasks.rados.rados.0.smithi107.stdout:2789: left oid 13 (ObjNum 2788 snap 0 seq_num 2788) 2020-06-12T02:37:46.368 INFO:tasks.rados.rados.0.smithi107.stdout:2789: done (2 left) 2020-06-12T02:37:46.369 INFO:tasks.rados.rados.0.smithi107.stdout:2791: done (1 left) 2020-06-12T02:37:46.369 INFO:tasks.rados.rados.0.smithi107.stdout:2792: delete oid 7 current snap is 0 2020-06-12T02:37:46.370 INFO:tasks.rados.rados.0.smithi107.stdout:2790: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:46.370 INFO:tasks.rados.rados.0.smithi107.stdout:2792: done (1 left) 2020-06-12T02:37:46.370 INFO:tasks.rados.rados.0.smithi107.stdout:2793: delete oid 20 current snap is 0 2020-06-12T02:37:46.375 INFO:tasks.rados.rados.0.smithi107.stdout:2790: finishing write tid 2 to smithi10729407-12 2020-06-12T02:37:46.375 INFO:tasks.rados.rados.0.smithi107.stdout:2790: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:46.377 INFO:tasks.rados.rados.0.smithi107.stdout:2790: finishing write tid 5 to smithi10729407-12 2020-06-12T02:37:46.377 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 455 (ObjNum 2789 snap 0 seq_num 2789) dirty exists 2020-06-12T02:37:46.377 INFO:tasks.rados.rados.0.smithi107.stdout:2790: left oid 12 (ObjNum 2789 snap 0 seq_num 2789) 2020-06-12T02:37:46.378 INFO:tasks.rados.rados.0.smithi107.stdout:2790: done (1 left) 2020-06-12T02:37:46.379 INFO:tasks.rados.rados.0.smithi107.stdout:2793: done (0 left) 2020-06-12T02:37:46.379 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:37:46.379 INFO:tasks.rados.rados.0.smithi107.stdout:2794: seq_num 2793 ranges {0=786432,786432=401408} 2020-06-12T02:37:46.383 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [7] live_osds: [0, 1, 2, 3, 4, 5, 6] 2020-06-12T02:37:46.383 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2020-06-12T02:37:46.385 INFO:tasks.thrashosds.thrasher:Shrinking pool device_health_metrics 2020-06-12T02:37:46.385 INFO:tasks.ceph.ceph_manager.ceph:contract_pool device_health_metrics by 10 min 8 2020-06-12T02:37:46.385 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json 2020-06-12T02:37:46.386 INFO:tasks.rados.rados.0.smithi107.stdout:2794: writing smithi10729407-9 from 0 to 786432 tid 1 2020-06-12T02:37:46.387 INFO:tasks.rados.rados.0.smithi107.stdout:2794: writing smithi10729407-9 from 786432 to 1187840 tid 2 2020-06-12T02:37:46.387 INFO:tasks.rados.rados.0.smithi107.stdout:2795: delete oid 13 current snap is 0 2020-06-12T02:37:46.391 INFO:tasks.rados.rados.0.smithi107.stdout:2794: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:46.392 INFO:tasks.rados.rados.0.smithi107.stdout:2795: done (1 left) 2020-06-12T02:37:46.392 INFO:tasks.rados.rados.0.smithi107.stdout:2796: delete oid 1 current snap is 0 2020-06-12T02:37:46.395 INFO:tasks.rados.rados.0.smithi107.stdout:2796: done (1 left) 2020-06-12T02:37:46.395 INFO:tasks.rados.rados.0.smithi107.stdout:2797: delete oid 10 current snap is 0 2020-06-12T02:37:46.397 INFO:tasks.rados.rados.0.smithi107.stdout:2794: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:46.397 INFO:tasks.rados.rados.0.smithi107.stdout:2794: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:46.398 INFO:tasks.rados.rados.0.smithi107.stdout:2797: done (1 left) 2020-06-12T02:37:46.398 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:37:46.398 INFO:tasks.rados.rados.0.smithi107.stdout:2798: seq_num 2797 ranges {0=450560,450560=712704,1163264=90112} 2020-06-12T02:37:46.401 INFO:tasks.rados.rados.0.smithi107.stdout:2798: writing smithi10729407-1 from 0 to 450560 tid 1 2020-06-12T02:37:46.405 INFO:tasks.rados.rados.0.smithi107.stdout:2798: writing smithi10729407-1 from 450560 to 1163264 tid 2 2020-06-12T02:37:46.406 INFO:tasks.rados.rados.0.smithi107.stdout:2798: writing smithi10729407-1 from 1163264 to 1253376 tid 3 2020-06-12T02:37:46.406 INFO:tasks.rados.rados.0.smithi107.stdout:2799: delete oid 10 current snap is 0 2020-06-12T02:37:46.407 INFO:tasks.rados.rados.0.smithi107.stdout:2794: finishing write tid 5 to smithi10729407-9 2020-06-12T02:37:46.407 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 475 (ObjNum 2793 snap 0 seq_num 2793) dirty exists 2020-06-12T02:37:46.407 INFO:tasks.rados.rados.0.smithi107.stdout:2794: left oid 9 (ObjNum 2793 snap 0 seq_num 2793) 2020-06-12T02:37:46.407 INFO:tasks.rados.rados.0.smithi107.stdout:2798: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:46.408 INFO:tasks.rados.rados.0.smithi107.stdout:2794: done (2 left) 2020-06-12T02:37:46.408 INFO:tasks.rados.rados.0.smithi107.stdout:2799: done (1 left) 2020-06-12T02:37:46.409 INFO:tasks.rados.rados.0.smithi107.stdout:2800: delete oid 10 current snap is 0 2020-06-12T02:37:46.410 INFO:tasks.rados.rados.0.smithi107.stdout:2800: done (1 left) 2020-06-12T02:37:46.410 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:37:46.411 INFO:tasks.rados.rados.0.smithi107.stdout:2801: seq_num 2800 ranges {933888=499712,1433600=417792,1851392=385024} 2020-06-12T02:37:46.418 INFO:tasks.rados.rados.0.smithi107.stdout:2801: writing smithi10729407-19 from 933888 to 1433600 tid 1 2020-06-12T02:37:46.420 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:46.417+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:46.421 INFO:tasks.rados.rados.0.smithi107.stdout:2801: writing smithi10729407-19 from 1433600 to 1851392 tid 2 2020-06-12T02:37:46.423 INFO:tasks.rados.rados.0.smithi107.stdout:2801: writing smithi10729407-19 from 1851392 to 2236416 tid 3 2020-06-12T02:37:46.423 INFO:tasks.rados.rados.0.smithi107.stdout:2802: delete oid 5 current snap is 0 2020-06-12T02:37:46.424 INFO:tasks.rados.rados.0.smithi107.stdout:2798: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:46.424 INFO:tasks.rados.rados.0.smithi107.stdout:2798: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:46.424 INFO:tasks.rados.rados.0.smithi107.stdout:2798: finishing write tid 5 to smithi10729407-1 2020-06-12T02:37:46.424 INFO:tasks.rados.rados.0.smithi107.stdout:2798: finishing write tid 6 to smithi10729407-1 2020-06-12T02:37:46.424 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 571 (ObjNum 2797 snap 0 seq_num 2797) dirty exists 2020-06-12T02:37:46.425 INFO:tasks.rados.rados.0.smithi107.stdout:2798: left oid 1 (ObjNum 2797 snap 0 seq_num 2797) 2020-06-12T02:37:46.427 INFO:tasks.rados.rados.0.smithi107.stdout:2798: done (2 left) 2020-06-12T02:37:46.427 INFO:tasks.rados.rados.0.smithi107.stdout:2802: done (1 left) 2020-06-12T02:37:46.427 INFO:tasks.rados.rados.0.smithi107.stdout:2803: delete oid 18 current snap is 0 2020-06-12T02:37:46.428 INFO:tasks.rados.rados.0.smithi107.stdout:2801: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:46.428 INFO:tasks.rados.rados.0.smithi107.stdout:2801: finishing write tid 2 to smithi10729407-19 2020-06-12T02:37:46.430 INFO:tasks.rados.rados.0.smithi107.stdout:2803: done (1 left) 2020-06-12T02:37:46.430 INFO:tasks.rados.rados.0.smithi107.stdout:2804: delete oid 2 current snap is 0 2020-06-12T02:37:46.430 INFO:tasks.rados.rados.0.smithi107.stdout:2801: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:46.431 INFO:tasks.rados.rados.0.smithi107.stdout:2804: done (1 left) 2020-06-12T02:37:46.432 INFO:tasks.rados.rados.0.smithi107.stdout:2805: delete oid 20 current snap is 0 2020-06-12T02:37:46.433 INFO:tasks.rados.rados.0.smithi107.stdout:2801: finishing write tid 5 to smithi10729407-19 2020-06-12T02:37:46.434 INFO:tasks.rados.rados.0.smithi107.stdout:2805: done (1 left) 2020-06-12T02:37:46.434 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:37:46.434 INFO:tasks.rados.rados.0.smithi107.stdout:2806: seq_num 2805 ranges {1187840=581632,1769472=638976,2408448=163840} 2020-06-12T02:37:46.443 INFO:tasks.rados.rados.0.smithi107.stdout:2806: writing smithi10729407-9 from 1187840 to 1769472 tid 1 2020-06-12T02:37:46.447 INFO:tasks.rados.rados.0.smithi107.stdout:2806: writing smithi10729407-9 from 1769472 to 2408448 tid 2 2020-06-12T02:37:46.449 INFO:tasks.rados.rados.0.smithi107.stdout:2806: writing smithi10729407-9 from 2408448 to 2572288 tid 3 2020-06-12T02:37:46.449 INFO:tasks.rados.rados.0.smithi107.stdout:2807: delete oid 7 current snap is 0 2020-06-12T02:37:46.449 INFO:tasks.rados.rados.0.smithi107.stdout:2801: finishing write tid 6 to smithi10729407-19 2020-06-12T02:37:46.450 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 459 (ObjNum 2800 snap 0 seq_num 2800) dirty exists 2020-06-12T02:37:46.450 INFO:tasks.rados.rados.0.smithi107.stdout:2801: left oid 19 (ObjNum 2800 snap 0 seq_num 2800) 2020-06-12T02:37:46.451 INFO:tasks.rados.rados.0.smithi107.stdout:2801: done (2 left) 2020-06-12T02:37:46.451 INFO:tasks.rados.rados.0.smithi107.stdout:2807: done (1 left) 2020-06-12T02:37:46.451 INFO:tasks.rados.rados.0.smithi107.stdout:2808: delete oid 19 current snap is 0 2020-06-12T02:37:46.453 INFO:tasks.rados.rados.0.smithi107.stdout:2808: done (1 left) 2020-06-12T02:37:46.453 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:37:46.453 INFO:tasks.rados.rados.0.smithi107.stdout:2809: seq_num 2808 ranges {1253376=630784,1884160=745472,2629632=57344} 2020-06-12T02:37:46.463 INFO:tasks.rados.rados.0.smithi107.stdout:2809: writing smithi10729407-1 from 1253376 to 1884160 tid 1 2020-06-12T02:37:46.468 INFO:tasks.rados.rados.0.smithi107.stdout:2809: writing smithi10729407-1 from 1884160 to 2629632 tid 2 2020-06-12T02:37:46.469 INFO:tasks.rados.rados.0.smithi107.stdout:2809: writing smithi10729407-1 from 2629632 to 2686976 tid 3 2020-06-12T02:37:46.470 INFO:tasks.rados.rados.0.smithi107.stdout:2810: delete oid 16 current snap is 0 2020-06-12T02:37:46.470 INFO:tasks.rados.rados.0.smithi107.stdout:2806: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:46.470 INFO:tasks.rados.rados.0.smithi107.stdout:2806: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:46.470 INFO:tasks.rados.rados.0.smithi107.stdout:2806: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:46.470 INFO:tasks.rados.rados.0.smithi107.stdout:2806: finishing write tid 5 to smithi10729407-9 2020-06-12T02:37:46.471 INFO:tasks.rados.rados.0.smithi107.stdout:2806: finishing write tid 6 to smithi10729407-9 2020-06-12T02:37:46.471 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 480 (ObjNum 2805 snap 0 seq_num 2805) dirty exists 2020-06-12T02:37:46.471 INFO:tasks.rados.rados.0.smithi107.stdout:2806: left oid 9 (ObjNum 2805 snap 0 seq_num 2805) 2020-06-12T02:37:46.471 INFO:tasks.rados.rados.0.smithi107.stdout:2806: done (2 left) 2020-06-12T02:37:46.472 INFO:tasks.rados.rados.0.smithi107.stdout:2810: done (1 left) 2020-06-12T02:37:46.472 INFO:tasks.rados.rados.0.smithi107.stdout:2811: delete oid 11 current snap is 0 2020-06-12T02:37:46.474 INFO:tasks.rados.rados.0.smithi107.stdout:2809: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:46.475 INFO:tasks.rados.rados.0.smithi107.stdout:2809: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:46.479 INFO:tasks.rados.rados.0.smithi107.stdout:2809: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:46.479 INFO:tasks.rados.rados.0.smithi107.stdout:2809: finishing write tid 5 to smithi10729407-1 2020-06-12T02:37:46.480 INFO:tasks.rados.rados.0.smithi107.stdout:2811: done (1 left) 2020-06-12T02:37:46.480 INFO:tasks.rados.rados.0.smithi107.stdout:2812: delete oid 12 current snap is 0 2020-06-12T02:37:46.480 INFO:tasks.rados.rados.0.smithi107.stdout:2809: finishing write tid 6 to smithi10729407-1 2020-06-12T02:37:46.481 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 575 (ObjNum 2808 snap 0 seq_num 2808) dirty exists 2020-06-12T02:37:46.481 INFO:tasks.rados.rados.0.smithi107.stdout:2809: left oid 1 (ObjNum 2808 snap 0 seq_num 2808) 2020-06-12T02:37:46.481 INFO:tasks.rados.rados.0.smithi107.stdout:2809: done (1 left) 2020-06-12T02:37:46.482 INFO:tasks.rados.rados.0.smithi107.stdout:2812: done (0 left) 2020-06-12T02:37:46.482 INFO:tasks.rados.rados.0.smithi107.stdout:2813: delete oid 14 current snap is 0 2020-06-12T02:37:46.483 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:46.484 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:46.485 INFO:tasks.rados.rados.0.smithi107.stdout:2813: done (0 left) 2020-06-12T02:37:46.485 INFO:tasks.rados.rados.0.smithi107.stdout:2814: delete oid 18 current snap is 0 2020-06-12T02:37:46.487 INFO:tasks.rados.rados.0.smithi107.stdout:2814: done (0 left) 2020-06-12T02:37:46.487 INFO:tasks.rados.rados.0.smithi107.stdout:2815: delete oid 10 current snap is 0 2020-06-12T02:37:46.489 INFO:tasks.rados.rados.0.smithi107.stdout:2815: done (0 left) 2020-06-12T02:37:46.489 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:37:46.490 INFO:tasks.rados.rados.0.smithi107.stdout:2816: seq_num 2815 ranges {2572288=753664,3325952=786432,4112384=16384} 2020-06-12T02:37:46.496 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-06-12T02:37:46.507 INFO:tasks.rados.rados.0.smithi107.stdout:2816: writing smithi10729407-9 from 2572288 to 3325952 tid 1 2020-06-12T02:37:46.512 INFO:tasks.rados.rados.0.smithi107.stdout:2816: writing smithi10729407-9 from 3325952 to 4112384 tid 2 2020-06-12T02:37:46.513 INFO:tasks.rados.rados.0.smithi107.stdout:2816: writing smithi10729407-9 from 4112384 to 4128768 tid 3 2020-06-12T02:37:46.513 INFO:tasks.rados.rados.0.smithi107.stdout:2817: delete oid 8 current snap is 0 2020-06-12T02:37:46.515 INFO:tasks.rados.rados.0.smithi107.stdout:2816: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:46.515 INFO:tasks.rados.rados.0.smithi107.stdout:2817: done (1 left) 2020-06-12T02:37:46.515 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:37:46.516 INFO:tasks.rados.rados.0.smithi107.stdout:2818: seq_num 2817 ranges {0=786432,786432=548864,1335296=253952} 2020-06-12T02:37:46.520 INFO:tasks.rados.rados.0.smithi107.stdout:2818: writing smithi10729407-17 from 0 to 786432 tid 1 2020-06-12T02:37:46.522 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:46.519+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:46.523 INFO:tasks.rados.rados.0.smithi107.stdout:2818: writing smithi10729407-17 from 786432 to 1335296 tid 2 2020-06-12T02:37:46.525 INFO:tasks.rados.rados.0.smithi107.stdout:2818: writing smithi10729407-17 from 1335296 to 1589248 tid 3 2020-06-12T02:37:46.526 INFO:tasks.rados.rados.0.smithi107.stdout:2819: delete oid 6 current snap is 0 2020-06-12T02:37:46.526 INFO:tasks.rados.rados.0.smithi107.stdout:2816: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:46.526 INFO:tasks.rados.rados.0.smithi107.stdout:2816: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:46.526 INFO:tasks.rados.rados.0.smithi107.stdout:2816: finishing write tid 5 to smithi10729407-9 2020-06-12T02:37:46.527 INFO:tasks.rados.rados.0.smithi107.stdout:2816: finishing write tid 6 to smithi10729407-9 2020-06-12T02:37:46.527 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 485 (ObjNum 2815 snap 0 seq_num 2815) dirty exists 2020-06-12T02:37:46.527 INFO:tasks.rados.rados.0.smithi107.stdout:2816: left oid 9 (ObjNum 2815 snap 0 seq_num 2815) 2020-06-12T02:37:46.529 INFO:tasks.rados.rados.0.smithi107.stdout:2816: done (2 left) 2020-06-12T02:37:46.529 INFO:tasks.rados.rados.0.smithi107.stdout:2819: done (1 left) 2020-06-12T02:37:46.529 INFO:tasks.rados.rados.0.smithi107.stdout:2820: delete oid 4 current snap is 0 2020-06-12T02:37:46.532 INFO:tasks.rados.rados.0.smithi107.stdout:2818: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:46.532 INFO:tasks.rados.rados.0.smithi107.stdout:2818: finishing write tid 2 to smithi10729407-17 2020-06-12T02:37:46.533 INFO:tasks.rados.rados.0.smithi107.stdout:2818: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:46.533 INFO:tasks.rados.rados.0.smithi107.stdout:2818: finishing write tid 5 to smithi10729407-17 2020-06-12T02:37:46.534 INFO:tasks.rados.rados.0.smithi107.stdout:2820: done (1 left) 2020-06-12T02:37:46.534 INFO:tasks.rados.rados.0.smithi107.stdout:2821: delete oid 15 current snap is 0 2020-06-12T02:37:46.536 INFO:tasks.rados.rados.0.smithi107.stdout:2818: finishing write tid 6 to smithi10729407-17 2020-06-12T02:37:46.536 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 529 (ObjNum 2817 snap 0 seq_num 2817) dirty exists 2020-06-12T02:37:46.536 INFO:tasks.rados.rados.0.smithi107.stdout:2818: left oid 17 (ObjNum 2817 snap 0 seq_num 2817) 2020-06-12T02:37:46.536 INFO:tasks.rados.rados.0.smithi107.stdout:2818: done (1 left) 2020-06-12T02:37:46.537 INFO:tasks.rados.rados.0.smithi107.stdout:2821: done (0 left) 2020-06-12T02:37:46.537 INFO:tasks.rados.rados.0.smithi107.stdout:2822: delete oid 7 current snap is 0 2020-06-12T02:37:46.538 INFO:tasks.rados.rados.0.smithi107.stdout:2822: done (0 left) 2020-06-12T02:37:46.538 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:37:46.538 INFO:tasks.rados.rados.0.smithi107.stdout:2823: seq_num 2822 ranges {0=466944,466944=417792,884736=581632,1466368=204800} 2020-06-12T02:37:46.541 INFO:tasks.rados.rados.0.smithi107.stdout:2823: writing smithi10729407-20 from 0 to 466944 tid 1 2020-06-12T02:37:46.544 INFO:tasks.rados.rados.0.smithi107.stdout:2823: writing smithi10729407-20 from 466944 to 884736 tid 2 2020-06-12T02:37:46.547 INFO:tasks.rados.rados.0.smithi107.stdout:2823: writing smithi10729407-20 from 884736 to 1466368 tid 3 2020-06-12T02:37:46.549 INFO:tasks.rados.rados.0.smithi107.stdout:2823: writing smithi10729407-20 from 1466368 to 1671168 tid 4 2020-06-12T02:37:46.549 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:37:46.549 INFO:tasks.rados.rados.0.smithi107.stdout:2824: seq_num 2823 ranges {0=483328,483328=704512,1187840=499712} 2020-06-12T02:37:46.552 INFO:tasks.rados.rados.0.smithi107.stdout:2824: writing smithi10729407-7 from 0 to 483328 tid 1 2020-06-12T02:37:46.556 INFO:tasks.rados.rados.0.smithi107.stdout:2824: writing smithi10729407-7 from 483328 to 1187840 tid 2 2020-06-12T02:37:46.560 INFO:tasks.rados.rados.0.smithi107.stdout:2824: writing smithi10729407-7 from 1187840 to 1687552 tid 3 2020-06-12T02:37:46.560 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:37:46.560 INFO:tasks.rados.rados.0.smithi107.stdout:2825: seq_num 2824 ranges {0=499712,499712=589824,1089536=548864,1638400=65536} 2020-06-12T02:37:46.563 INFO:tasks.rados.rados.0.smithi107.stdout:2825: writing smithi10729407-16 from 0 to 499712 tid 1 2020-06-12T02:37:46.567 INFO:tasks.rados.rados.0.smithi107.stdout:2825: writing smithi10729407-16 from 499712 to 1089536 tid 2 2020-06-12T02:37:46.570 INFO:tasks.rados.rados.0.smithi107.stdout:2825: writing smithi10729407-16 from 1089536 to 1638400 tid 3 2020-06-12T02:37:46.571 INFO:tasks.rados.rados.0.smithi107.stdout:2825: writing smithi10729407-16 from 1638400 to 1703936 tid 4 2020-06-12T02:37:46.572 INFO:tasks.rados.rados.0.smithi107.stdout:2826: delete oid 15 current snap is 0 2020-06-12T02:37:46.572 INFO:tasks.rados.rados.0.smithi107.stdout:2823: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:46.572 INFO:tasks.rados.rados.0.smithi107.stdout:2824: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:46.572 INFO:tasks.rados.rados.0.smithi107.stdout:2823: finishing write tid 2 to smithi10729407-20 2020-06-12T02:37:46.573 INFO:tasks.rados.rados.0.smithi107.stdout:2823: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:46.573 INFO:tasks.rados.rados.0.smithi107.stdout:2823: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:46.573 INFO:tasks.rados.rados.0.smithi107.stdout:2823: finishing write tid 6 to smithi10729407-20 2020-06-12T02:37:46.573 INFO:tasks.rados.rados.0.smithi107.stdout:2824: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:46.573 INFO:tasks.rados.rados.0.smithi107.stdout:2824: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:46.574 INFO:tasks.rados.rados.0.smithi107.stdout:2824: finishing write tid 5 to smithi10729407-7 2020-06-12T02:37:46.574 INFO:tasks.rados.rados.0.smithi107.stdout:2823: finishing write tid 7 to smithi10729407-20 2020-06-12T02:37:46.575 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 185 (ObjNum 2822 snap 0 seq_num 2822) dirty exists 2020-06-12T02:37:46.575 INFO:tasks.rados.rados.0.smithi107.stdout:2823: left oid 20 (ObjNum 2822 snap 0 seq_num 2822) 2020-06-12T02:37:46.575 INFO:tasks.rados.rados.0.smithi107.stdout:2823: done (3 left) 2020-06-12T02:37:46.575 INFO:tasks.rados.rados.0.smithi107.stdout:2826: done (2 left) 2020-06-12T02:37:46.575 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:37:46.576 INFO:tasks.rados.rados.0.smithi107.stdout:2827: seq_num 2826 ranges {0=532480,532480=761856,1294336=442368} 2020-06-12T02:37:46.577 INFO:tasks.rados.rados.0.smithi107.stdout:2827: writing smithi10729407-14 from 0 to 532480 tid 1 2020-06-12T02:37:46.582 INFO:tasks.rados.rados.0.smithi107.stdout:2827: writing smithi10729407-14 from 532480 to 1294336 tid 2 2020-06-12T02:37:46.585 INFO:tasks.rados.rados.0.smithi107.stdout:2827: writing smithi10729407-14 from 1294336 to 1736704 tid 3 2020-06-12T02:37:46.585 INFO:tasks.rados.rados.0.smithi107.stdout:2828: delete oid 2 current snap is 0 2020-06-12T02:37:46.586 INFO:tasks.rados.rados.0.smithi107.stdout:2824: finishing write tid 6 to smithi10729407-7 2020-06-12T02:37:46.586 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 549 (ObjNum 2823 snap 0 seq_num 2823) dirty exists 2020-06-12T02:37:46.586 INFO:tasks.rados.rados.0.smithi107.stdout:2824: left oid 7 (ObjNum 2823 snap 0 seq_num 2823) 2020-06-12T02:37:46.586 INFO:tasks.rados.rados.0.smithi107.stdout:2825: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:46.591 INFO:tasks.rados.rados.0.smithi107.stdout:2825: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:46.591 INFO:tasks.rados.rados.0.smithi107.stdout:2825: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:46.592 INFO:tasks.rados.rados.0.smithi107.stdout:2825: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:46.592 INFO:tasks.rados.rados.0.smithi107.stdout:2825: finishing write tid 6 to smithi10729407-16 2020-06-12T02:37:46.593 INFO:tasks.rados.rados.0.smithi107.stdout:2827: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:46.594 INFO:tasks.rados.rados.0.smithi107.stdout:2824: done (3 left) 2020-06-12T02:37:46.594 INFO:tasks.rados.rados.0.smithi107.stdout:2828: done (2 left) 2020-06-12T02:37:46.595 INFO:tasks.rados.rados.0.smithi107.stdout:2829: delete oid 13 current snap is 0 2020-06-12T02:37:46.595 INFO:tasks.rados.rados.0.smithi107.stdout:2827: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:46.595 INFO:tasks.rados.rados.0.smithi107.stdout:2825: finishing write tid 7 to smithi10729407-16 2020-06-12T02:37:46.595 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 474 (ObjNum 2824 snap 0 seq_num 2824) dirty exists 2020-06-12T02:37:46.596 INFO:tasks.rados.rados.0.smithi107.stdout:2825: left oid 16 (ObjNum 2824 snap 0 seq_num 2824) 2020-06-12T02:37:46.606 INFO:tasks.rados.rados.0.smithi107.stdout:2827: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:46.606 INFO:tasks.rados.rados.0.smithi107.stdout:2827: finishing write tid 5 to smithi10729407-14 2020-06-12T02:37:46.606 INFO:tasks.rados.rados.0.smithi107.stdout:2825: done (2 left) 2020-06-12T02:37:46.606 INFO:tasks.rados.rados.0.smithi107.stdout:2829: done (1 left) 2020-06-12T02:37:46.607 INFO:tasks.rados.rados.0.smithi107.stdout:2830: delete oid 3 current snap is 0 2020-06-12T02:37:46.607 INFO:tasks.rados.rados.0.smithi107.stdout:2827: finishing write tid 6 to smithi10729407-14 2020-06-12T02:37:46.607 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 478 (ObjNum 2826 snap 0 seq_num 2826) dirty exists 2020-06-12T02:37:46.607 INFO:tasks.rados.rados.0.smithi107.stdout:2827: left oid 14 (ObjNum 2826 snap 0 seq_num 2826) 2020-06-12T02:37:46.607 INFO:tasks.rados.rados.0.smithi107.stdout:2827: done (1 left) 2020-06-12T02:37:46.608 INFO:tasks.rados.rados.0.smithi107.stdout:2830: done (0 left) 2020-06-12T02:37:46.608 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:37:46.608 INFO:tasks.rados.rados.0.smithi107.stdout:2831: seq_num 2830 ranges {0=606208,606208=745472,1351680=458752} 2020-06-12T02:37:46.608 INFO:tasks.rados.rados.0.smithi107.stdout:2831: writing smithi10729407-18 from 0 to 606208 tid 1 2020-06-12T02:37:46.610 INFO:tasks.rados.rados.0.smithi107.stdout:2831: writing smithi10729407-18 from 606208 to 1351680 tid 2 2020-06-12T02:37:46.613 INFO:tasks.rados.rados.0.smithi107.stdout:2831: writing smithi10729407-18 from 1351680 to 1810432 tid 3 2020-06-12T02:37:46.614 INFO:tasks.rados.rados.0.smithi107.stdout:2832: delete oid 11 current snap is 0 2020-06-12T02:37:46.619 INFO:tasks.rados.rados.0.smithi107.stdout:2832: done (1 left) 2020-06-12T02:37:46.619 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:37:46.620 INFO:tasks.rados.rados.0.smithi107.stdout:2833: seq_num 2832 ranges {0=638976,638976=516096,1155072=507904,1662976=180224} 2020-06-12T02:37:46.623 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:37:46.620+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:37:46.623 INFO:tasks.rados.rados.0.smithi107.stdout:2833: writing smithi10729407-4 from 0 to 638976 tid 1 2020-06-12T02:37:46.626 INFO:tasks.rados.rados.0.smithi107.stdout:2833: writing smithi10729407-4 from 638976 to 1155072 tid 2 2020-06-12T02:37:46.630 INFO:tasks.rados.rados.0.smithi107.stdout:2833: writing smithi10729407-4 from 1155072 to 1662976 tid 3 2020-06-12T02:37:46.631 INFO:tasks.rados.rados.0.smithi107.stdout:2833: writing smithi10729407-4 from 1662976 to 1843200 tid 4 2020-06-12T02:37:46.631 INFO:tasks.rados.rados.0.smithi107.stdout:2834: delete oid 17 current snap is 0 2020-06-12T02:37:46.631 INFO:tasks.rados.rados.0.smithi107.stdout:2831: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:46.631 INFO:tasks.rados.rados.0.smithi107.stdout:2831: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:46.632 INFO:tasks.rados.rados.0.smithi107.stdout:2831: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:46.632 INFO:tasks.rados.rados.0.smithi107.stdout:2831: finishing write tid 5 to smithi10729407-18 2020-06-12T02:37:46.632 INFO:tasks.rados.rados.0.smithi107.stdout:2831: finishing write tid 6 to smithi10729407-18 2020-06-12T02:37:46.632 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 489 (ObjNum 2830 snap 0 seq_num 2830) dirty exists 2020-06-12T02:37:46.632 INFO:tasks.rados.rados.0.smithi107.stdout:2831: left oid 18 (ObjNum 2830 snap 0 seq_num 2830) 2020-06-12T02:37:46.633 INFO:tasks.rados.rados.0.smithi107.stdout:2833: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:46.637 INFO:tasks.rados.rados.0.smithi107.stdout:2833: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:46.637 INFO:tasks.rados.rados.0.smithi107.stdout:2833: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:46.637 INFO:tasks.rados.rados.0.smithi107.stdout:2833: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:46.638 INFO:tasks.rados.rados.0.smithi107.stdout:2833: finishing write tid 6 to smithi10729407-4 2020-06-12T02:37:46.644 INFO:tasks.rados.rados.0.smithi107.stdout:2831: done (2 left) 2020-06-12T02:37:46.644 INFO:tasks.rados.rados.0.smithi107.stdout:2834: done (1 left) 2020-06-12T02:37:46.644 INFO:tasks.rados.rados.0.smithi107.stdout:2835: delete oid 6 current snap is 0 2020-06-12T02:37:46.645 INFO:tasks.rados.rados.0.smithi107.stdout:2833: finishing write tid 7 to smithi10729407-4 2020-06-12T02:37:46.645 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 535 (ObjNum 2832 snap 0 seq_num 2832) dirty exists 2020-06-12T02:37:46.645 INFO:tasks.rados.rados.0.smithi107.stdout:2833: left oid 4 (ObjNum 2832 snap 0 seq_num 2832) 2020-06-12T02:37:46.645 INFO:tasks.rados.rados.0.smithi107.stdout:2833: done (1 left) 2020-06-12T02:37:46.646 INFO:tasks.rados.rados.0.smithi107.stdout:2835: done (0 left) 2020-06-12T02:37:46.646 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:37:46.646 INFO:tasks.rados.rados.0.smithi107.stdout:2836: seq_num 2835 ranges {0=688128,688128=565248,1253376=638976} 2020-06-12T02:37:46.649 INFO:tasks.rados.rados.0.smithi107.stdout:2836: writing smithi10729407-19 from 0 to 688128 tid 1 2020-06-12T02:37:46.649 INFO:tasks.rados.rados.0.smithi107.stdout:2836: writing smithi10729407-19 from 688128 to 1253376 tid 2 2020-06-12T02:37:46.653 INFO:tasks.rados.rados.0.smithi107.stdout:2836: writing smithi10729407-19 from 1253376 to 1892352 tid 3 2020-06-12T02:37:46.653 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:37:46.653 INFO:tasks.rados.rados.0.smithi107.stdout:2837: seq_num 2836 ranges {1843200=704512,2547712=499712,3047424=483328,3530752=221184} 2020-06-12T02:37:46.664 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:46.664 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:46.666 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-06-12T02:37:46.667 INFO:tasks.rados.rados.0.smithi107.stdout:2837: writing smithi10729407-4 from 1843200 to 2547712 tid 1 2020-06-12T02:37:46.670 INFO:tasks.rados.rados.0.smithi107.stdout:2837: writing smithi10729407-4 from 2547712 to 3047424 tid 2 2020-06-12T02:37:46.674 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-06-12T02:37:46.675 INFO:tasks.rados.rados.0.smithi107.stdout:2837: writing smithi10729407-4 from 3047424 to 3530752 tid 3 2020-06-12T02:37:46.676 INFO:tasks.rados.rados.0.smithi107.stdout:2837: writing smithi10729407-4 from 3530752 to 3751936 tid 4 2020-06-12T02:37:46.676 INFO:tasks.rados.rados.0.smithi107.stdout:2838: delete oid 7 current snap is 0 2020-06-12T02:37:46.676 INFO:tasks.rados.rados.0.smithi107.stdout:2836: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:46.676 INFO:tasks.rados.rados.0.smithi107.stdout:2836: finishing write tid 2 to smithi10729407-19 2020-06-12T02:37:46.676 INFO:tasks.rados.rados.0.smithi107.stdout:2836: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:46.677 INFO:tasks.rados.rados.0.smithi107.stdout:2836: finishing write tid 5 to smithi10729407-19 2020-06-12T02:37:46.677 INFO:tasks.rados.rados.0.smithi107.stdout:2836: finishing write tid 6 to smithi10729407-19 2020-06-12T02:37:46.677 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 465 (ObjNum 2835 snap 0 seq_num 2835) dirty exists 2020-06-12T02:37:46.677 INFO:tasks.rados.rados.0.smithi107.stdout:2836: left oid 19 (ObjNum 2835 snap 0 seq_num 2835) 2020-06-12T02:37:46.683 INFO:tasks.rados.rados.0.smithi107.stdout:2836: done (2 left) 2020-06-12T02:37:46.684 INFO:tasks.rados.rados.0.smithi107.stdout:2838: done (1 left) 2020-06-12T02:37:46.684 INFO:tasks.rados.rados.0.smithi107.stdout:2839: delete oid 14 current snap is 0 2020-06-12T02:37:46.684 INFO:tasks.rados.rados.0.smithi107.stdout:2837: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:46.685 INFO:tasks.rados.rados.0.smithi107.stdout:2837: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:46.686 INFO:tasks.rados.rados.0.smithi107.stdout:2839: done (1 left) 2020-06-12T02:37:46.687 INFO:tasks.rados.rados.0.smithi107.stdout:2840: delete oid 14 current snap is 0 2020-06-12T02:37:46.687 INFO:tasks.rados.rados.0.smithi107.stdout:2837: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:46.691 INFO:tasks.rados.rados.0.smithi107.stdout:2837: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:46.691 INFO:tasks.rados.rados.0.smithi107.stdout:2837: finishing write tid 6 to smithi10729407-4 2020-06-12T02:37:46.692 INFO:tasks.rados.rados.0.smithi107.stdout:2840: done (1 left) 2020-06-12T02:37:46.692 INFO:tasks.rados.rados.0.smithi107.stdout:2841: delete oid 8 current snap is 0 2020-06-12T02:37:46.693 INFO:tasks.rados.rados.0.smithi107.stdout:2837: finishing write tid 7 to smithi10729407-4 2020-06-12T02:37:46.694 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 541 (ObjNum 2836 snap 0 seq_num 2836) dirty exists 2020-06-12T02:37:46.694 INFO:tasks.rados.rados.0.smithi107.stdout:2837: left oid 4 (ObjNum 2836 snap 0 seq_num 2836) 2020-06-12T02:37:46.696 INFO:tasks.rados.rados.0.smithi107.stdout:2837: done (1 left) 2020-06-12T02:37:46.696 INFO:tasks.rados.rados.0.smithi107.stdout:2841: done (0 left) 2020-06-12T02:37:46.696 INFO:tasks.rados.rados.0.smithi107.stdout:2842: delete oid 2 current snap is 0 2020-06-12T02:37:46.704 INFO:tasks.rados.rados.0.smithi107.stdout:2842: done (0 left) 2020-06-12T02:37:46.704 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:37:46.704 INFO:tasks.rados.rados.0.smithi107.stdout:2843: seq_num 2842 ranges {0=802816,802816=606208,1409024=483328,1892352=114688} 2020-06-12T02:37:46.705 INFO:tasks.rados.rados.0.smithi107.stdout:2843: writing smithi10729407-5 from 0 to 802816 tid 1 2020-06-12T02:37:46.709 INFO:tasks.rados.rados.0.smithi107.stdout:2843: writing smithi10729407-5 from 802816 to 1409024 tid 2 2020-06-12T02:37:46.713 INFO:tasks.rados.rados.0.smithi107.stdout:2843: writing smithi10729407-5 from 1409024 to 1892352 tid 3 2020-06-12T02:37:46.714 INFO:tasks.rados.rados.0.smithi107.stdout:2843: writing smithi10729407-5 from 1892352 to 2007040 tid 4 2020-06-12T02:37:46.714 INFO:tasks.rados.rados.0.smithi107.stdout:2844: delete oid 7 current snap is 0 2020-06-12T02:37:46.714 INFO:tasks.rados.rados.0.smithi107.stdout:2843: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:46.717 INFO:tasks.rados.rados.0.smithi107.stdout:2843: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:46.718 INFO:tasks.rados.rados.0.smithi107.stdout:2844: done (1 left) 2020-06-12T02:37:46.718 INFO:tasks.rados.rados.0.smithi107.stdout:2845: delete oid 17 current snap is 0 2020-06-12T02:37:46.720 INFO:tasks.rados.rados.0.smithi107.stdout:2843: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:46.721 INFO:tasks.rados.rados.0.smithi107.stdout:2845: done (1 left) 2020-06-12T02:37:46.721 INFO:tasks.rados.rados.0.smithi107.stdout:2846: delete oid 8 current snap is 0 2020-06-12T02:37:46.721 INFO:tasks.rados.rados.0.smithi107.stdout:2843: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:46.722 INFO:tasks.rados.rados.0.smithi107.stdout:2843: finishing write tid 6 to smithi10729407-5 2020-06-12T02:37:46.723 INFO:tasks.rados.rados.0.smithi107.stdout:2843: finishing write tid 7 to smithi10729407-5 2020-06-12T02:37:46.724 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 240 (ObjNum 2842 snap 0 seq_num 2842) dirty exists 2020-06-12T02:37:46.724 INFO:tasks.rados.rados.0.smithi107.stdout:2843: left oid 5 (ObjNum 2842 snap 0 seq_num 2842) 2020-06-12T02:37:46.724 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:37:46.720+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:37:46.725 INFO:tasks.rados.rados.0.smithi107.stdout:2843: done (1 left) 2020-06-12T02:37:46.725 INFO:tasks.rados.rados.0.smithi107.stdout:2846: done (0 left) 2020-06-12T02:37:46.725 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:37:46.726 INFO:tasks.rados.rados.0.smithi107.stdout:2847: seq_num 2846 ranges {1703936=466944,2170880=598016,2768896=458752,3227648=548864} 2020-06-12T02:37:46.739 INFO:tasks.rados.rados.0.smithi107.stdout:2847: writing smithi10729407-16 from 1703936 to 2170880 tid 1 2020-06-12T02:37:46.745 INFO:tasks.rados.rados.0.smithi107.stdout:2847: writing smithi10729407-16 from 2170880 to 2768896 tid 2 2020-06-12T02:37:46.746 INFO:tasks.rados.rados.0.smithi107.stdout:2847: writing smithi10729407-16 from 2768896 to 3227648 tid 3 2020-06-12T02:37:46.752 INFO:tasks.rados.rados.0.smithi107.stdout:2847: writing smithi10729407-16 from 3227648 to 3776512 tid 4 2020-06-12T02:37:46.753 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:37:46.753 INFO:tasks.rados.rados.0.smithi107.stdout:2848: seq_num 2847 ranges {4128768=491520,4620288=475136,5095424=671744,5767168=458752} 2020-06-12T02:37:46.763 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:37:46.765 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2020-06-12T02:37:46.778 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":49,"stamp":"2020-06-12T02:37:45.599321+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":16400384,"num_objects":9,"num_object_clones":0,"num_object_copies":25,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1297,"num_read_kb":692,"num_write":2806,"num_write_kb":790602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3735,"ondisk_log_size":3735,"up":62,"acting":62,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8408916,"kb_used_data":19856,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746565804,"statfs":{"total":773094113280,"available":764483383296,"internally_reserved":8589934592,"allocated":20332544,"data_stored":18974041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2637824,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":477,"num_read_kb":255,"num_write":1033,"num_write_kb":285552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001266"},"pg_stats":[{"pgid":"3.f","version":"28'399","reported_seq":"488","reported_epoch":"28","state":"active+undersized+degraded","last_fresh":"2020-06-12T02:37:41.290823+0000","last_change":"2020-06-12T02:37:40.776334+0000","last_active":"2020-06-12T02:37:41.290823+0000","last_peered":"2020-06-12T02:37:41.290823+0000","last_clean":"2020-06-12T02:37:26.883460+0000","last_became_active":"2020-06-12T02:37:27.901005+0000","last_became_peered":"2020-06-12T02:37:27.901005+0000","last_unstale":"2020-06-12T02:37:41.290823+0000","last_undegraded":"2020-06-12T02:37:40.770120+0000","last_fullsized":"2020-06-12T02:37:27.892208+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":399,"ondisk_log_size":399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4579328,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":148,"num_read_kb":76,"num_write":295,"num_write_kb":79520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,4,0],"acting":[2147483647,4,0],"avail_no_missing":["4(1)","0(2)"],"object_location_counts":[{"shards":"0(2),4(1)","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"28'171","reported_seq":"213","reported_epoch":"28","state":"active+undersized","last_fresh":"2020-06-12T02:37:41.331317+0000","last_change":"2020-06-12T02:37:41.138167+0000","last_active":"2020-06-12T02:37:41.331317+0000","last_peered":"2020-06-12T02:37:41.331317+0000","last_clean":"2020-06-12T02:37:26.883460+0000","last_became_active":"2020-06-12T02:37:27.902891+0000","last_became_peered":"2020-06-12T02:37:27.902891+0000","last_unstale":"2020-06-12T02:37:41.331317+0000","last_undegraded":"2020-06-12T02:37:41.331317+0000","last_fullsized":"2020-06-12T02:37:27.892202+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":171,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":59,"num_read_kb":29,"num_write":128,"num_write_kb":36744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,4,1],"acting":[2147483647,4,1],"avail_no_missing":["4(1)","1(2)"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"28'436","reported_seq":"538","reported_epoch":"29","state":"peering","last_fresh":"2020-06-12T02:37:41.367861+0000","last_change":"2020-06-12T02:37:41.368053+0000","last_active":"2020-06-12T02:37:41.367861+0000","last_peered":"2020-06-12T02:37:41.327920+0000","last_clean":"2020-06-12T02:37:26.883460+0000","last_became_active":"2020-06-12T02:37:27.914253+0000","last_became_peered":"2020-06-12T02:37:27.914253+0000","last_unstale":"2020-06-12T02:37:41.367861+0000","last_undegraded":"2020-06-12T02:37:41.367861+0000","last_fullsized":"2020-06-12T02:37:41.367861+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":436,"ondisk_log_size":436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5521408,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":152,"num_read_kb":87,"num_write":354,"num_write_kb":101112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,5,3],"acting":[2147483647,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"3","reported_epoch":"29","state":"peering","last_fresh":"2020-06-12T02:37:41.366929+0000","last_change":"2020-06-12T02:37:41.367099+0000","last_active":"2020-06-12T02:37:26.883460+0000","last_peered":"2020-06-12T02:37:26.883460+0000","last_clean":"2020-06-12T02:37:26.883460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2020-06-12T02:37:41.366929+0000","last_undegraded":"2020-06-12T02:37:41.366929+0000","last_fullsized":"2020-06-12T02:37:41.366929+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"29'369","reported_seq":"451","reported_epoch":"29","state":"active+clean","last_fresh":"2020-06-12T02:37:41.366859+0000","last_change":"2020-06-12T02:37:27.902643+0000","last_active":"2020-06-12T02:37:41.366859+0000","last_peered":"2020-06-12T02:37:41.366859+0000","last_clean":"2020-06-12T02:37:41.366859+0000","last_became_active":"2020-06-12T02:37:27.902204+0000","last_became_peered":"2020-06-12T02:37:27.902204+0000","last_unstale":"2020-06-12T02:37:41.366859+0000","last_undegraded":"2020-06-12T02:37:41.366859+0000","last_fullsized":"2020-06-12T02:37:41.366859+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":369,"ondisk_log_size":369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":122,"num_read_kb":66,"num_write":270,"num_write_kb":75712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"28'356","reported_seq":"443","reported_epoch":"28","state":"active+clean","last_fresh":"2020-06-12T02:37:41.360312+0000","last_change":"2020-06-12T02:37:27.905454+0000","last_active":"2020-06-12T02:37:41.360312+0000","last_peered":"2020-06-12T02:37:41.360312+0000","last_clean":"2020-06-12T02:37:41.360312+0000","last_became_active":"2020-06-12T02:37:27.904857+0000","last_became_peered":"2020-06-12T02:37:27.904857+0000","last_unstale":"2020-06-12T02:37:41.360312+0000","last_undegraded":"2020-06-12T02:37:41.360312+0000","last_fullsized":"2020-06-12T02:37:41.360312+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":356,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1425408,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":137,"num_read_kb":71,"num_write":271,"num_write_kb":74824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"28'316","reported_seq":"384","reported_epoch":"28","state":"active+undersized","last_fresh":"2020-06-12T02:37:41.329640+0000","last_change":"2020-06-12T02:37:41.136182+0000","last_active":"2020-06-12T02:37:41.329640+0000","last_peered":"2020-06-12T02:37:41.329640+0000","last_clean":"2020-06-12T02:37:26.883460+0000","last_became_active":"2020-06-12T02:37:27.900741+0000","last_became_peered":"2020-06-12T02:37:27.900741+0000","last_unstale":"2020-06-12T02:37:41.329640+0000","last_undegraded":"2020-06-12T02:37:41.329640+0000","last_fullsized":"2020-06-12T02:37:27.892202+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":316,"ondisk_log_size":316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":106,"num_read_kb":55,"num_write":219,"num_write_kb":58112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,2147483647],"acting":[4,2,2147483647],"avail_no_missing":["4(0)","2(1)"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"28'211","reported_seq":"264","reported_epoch":"28","state":"active+undersized","last_fresh":"2020-06-12T02:37:41.354874+0000","last_change":"2020-06-12T02:37:41.366643+0000","last_active":"2020-06-12T02:37:41.354874+0000","last_peered":"2020-06-12T02:37:41.354874+0000","last_clean":"2020-06-12T02:37:26.883460+0000","last_became_active":"2020-06-12T02:37:27.906124+0000","last_became_peered":"2020-06-12T02:37:27.906124+0000","last_unstale":"2020-06-12T02:37:41.354874+0000","last_undegraded":"2020-06-12T02:37:41.197672+0000","last_fullsized":"2020-06-12T02:37:27.889445+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":211,"ondisk_log_size":211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":73,"num_read_kb":40,"num_write":163,"num_write_kb":46664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2147483647],"acting":[3,1,2147483647],"avail_no_missing":["3(0)","1(1)"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"20","reported_epoch":"28","state":"active+clean","last_fresh":"2020-06-12T02:37:39.460975+0000","last_change":"2020-06-12T02:37:16.868466+0000","last_active":"2020-06-12T02:37:39.460975+0000","last_peered":"2020-06-12T02:37:39.460975+0000","last_clean":"2020-06-12T02:37:39.460975+0000","last_became_active":"2020-06-12T02:37:16.868098+0000","last_became_peered":"2020-06-12T02:37:16.868098+0000","last_unstale":"2020-06-12T02:37:39.460975+0000","last_undegraded":"2020-06-12T02:37:39.460975+0000","last_fullsized":"2020-06-12T02:37:39.460975+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"28'381","reported_seq":"458","reported_epoch":"29","state":"peering","last_fresh":"2020-06-12T02:37:41.366696+0000","last_change":"2020-06-12T02:37:41.366849+0000","last_active":"2020-06-12T02:37:41.366696+0000","last_peered":"2020-06-12T02:37:40.940521+0000","last_clean":"2020-06-12T02:37:40.940521+0000","last_became_active":"2020-06-12T02:37:27.900459+0000","last_became_peered":"2020-06-12T02:37:27.900459+0000","last_unstale":"2020-06-12T02:37:41.366696+0000","last_undegraded":"2020-06-12T02:37:41.366696+0000","last_fullsized":"2020-06-12T02:37:41.366696+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":381,"ondisk_log_size":381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":126,"num_read_kb":68,"num_write":278,"num_write_kb":79120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,5],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"3","reported_epoch":"29","state":"peering","last_fresh":"2020-06-12T02:37:41.364122+0000","last_change":"2020-06-12T02:37:41.364359+0000","last_active":"2020-06-12T02:37:15.738781+0000","last_peered":"2020-06-12T02:37:15.738781+0000","last_clean":"2020-06-12T02:37:15.738781+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2020-06-12T02:37:41.364122+0000","last_undegraded":"2020-06-12T02:37:41.364122+0000","last_fullsized":"2020-06-12T02:37:41.364122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"28'399","reported_seq":"483","reported_epoch":"28","state":"active+clean","last_fresh":"2020-06-12T02:37:41.335709+0000","last_change":"2020-06-12T02:37:27.903386+0000","last_active":"2020-06-12T02:37:41.335709+0000","last_peered":"2020-06-12T02:37:41.335709+0000","last_clean":"2020-06-12T02:37:41.335709+0000","last_became_active":"2020-06-12T02:37:27.902910+0000","last_became_peered":"2020-06-12T02:37:27.902910+0000","last_unstale":"2020-06-12T02:37:41.335709+0000","last_undegraded":"2020-06-12T02:37:41.335709+0000","last_fullsized":"2020-06-12T02:37:41.335709+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":399,"ondisk_log_size":399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":128,"num_read_kb":68,"num_write":301,"num_write_kb":90672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"20","reported_epoch":"28","state":"active+clean","last_fresh":"2020-06-12T02:37:39.463992+0000","last_change":"2020-06-12T02:37:16.765460+0000","last_active":"2020-06-12T02:37:39.463992+0000","last_peered":"2020-06-12T02:37:39.463992+0000","last_clean":"2020-06-12T02:37:39.463992+0000","last_became_active":"2020-06-12T02:37:16.764958+0000","last_became_peered":"2020-06-12T02:37:16.764958+0000","last_unstale":"2020-06-12T02:37:39.463992+0000","last_undegraded":"2020-06-12T02:37:39.463992+0000","last_fullsized":"2020-06-12T02:37:39.463992+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"28'193","reported_seq":"243","reported_epoch":"28","state":"active+clean","last_fresh":"2020-06-12T02:37:41.347038+0000","last_change":"2020-06-12T02:37:27.905010+0000","last_active":"2020-06-12T02:37:41.347038+0000","last_peered":"2020-06-12T02:37:41.347038+0000","last_clean":"2020-06-12T02:37:41.347038+0000","last_became_active":"2020-06-12T02:37:27.904552+0000","last_became_peered":"2020-06-12T02:37:27.904552+0000","last_unstale":"2020-06-12T02:37:41.347038+0000","last_undegraded":"2020-06-12T02:37:41.347038+0000","last_fullsized":"2020-06-12T02:37:41.347038+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":193,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":65,"num_read_kb":34,"num_write":138,"num_write_kb":39736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"3","reported_epoch":"29","state":"peering","last_fresh":"2020-06-12T02:37:41.368370+0000","last_change":"2020-06-12T02:37:41.368576+0000","last_active":"2020-06-12T02:37:15.738781+0000","last_peered":"2020-06-12T02:37:15.738781+0000","last_clean":"2020-06-12T02:37:15.738781+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2020-06-12T02:37:41.368370+0000","last_undegraded":"2020-06-12T02:37:41.368370+0000","last_fullsized":"2020-06-12T02:37:41.368370+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"15'2","reported_seq":"26","reported_epoch":"28","state":"active+undersized+degraded","last_fresh":"2020-06-12T02:37:39.469342+0000","last_change":"2020-06-12T02:37:25.889337+0000","last_active":"2020-06-12T02:37:39.469342+0000","last_peered":"2020-06-12T02:37:39.469342+0000","last_clean":"2020-06-12T02:37:13.609964+0000","last_became_active":"2020-06-12T02:37:25.889337+0000","last_became_peered":"2020-06-12T02:37:25.889337+0000","last_unstale":"2020-06-12T02:37:39.469342+0000","last_undegraded":"2020-06-12T02:37:25.888627+0000","last_fullsized":"2020-06-12T02:37:25.888542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":3,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:04.419204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:04.419204+0000","last_clean_scrub_stamp":"2020-06-12T02:37:04.419204+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"28'143","reported_seq":"185","reported_epoch":"28","state":"active+clean","last_fresh":"2020-06-12T02:37:41.235675+0000","last_change":"2020-06-12T02:37:27.898985+0000","last_active":"2020-06-12T02:37:41.235675+0000","last_peered":"2020-06-12T02:37:41.235675+0000","last_clean":"2020-06-12T02:37:41.235675+0000","last_became_active":"2020-06-12T02:37:27.898516+0000","last_became_peered":"2020-06-12T02:37:27.898516+0000","last_unstale":"2020-06-12T02:37:41.235675+0000","last_undegraded":"2020-06-12T02:37:41.235675+0000","last_fullsized":"2020-06-12T02:37:41.235675+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":143,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1130496,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":52,"num_read_kb":26,"num_write":105,"num_write_kb":29536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"17","reported_epoch":"28","state":"active+undersized","last_fresh":"2020-06-12T02:37:39.460876+0000","last_change":"2020-06-12T02:37:25.887233+0000","last_active":"2020-06-12T02:37:39.460876+0000","last_peered":"2020-06-12T02:37:39.460876+0000","last_clean":"2020-06-12T02:37:15.738781+0000","last_became_active":"2020-06-12T02:37:25.887233+0000","last_became_peered":"2020-06-12T02:37:25.887233+0000","last_unstale":"2020-06-12T02:37:39.460876+0000","last_undegraded":"2020-06-12T02:37:39.460876+0000","last_fullsized":"2020-06-12T02:37:25.886610+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"16","reported_epoch":"28","state":"active+clean","last_fresh":"2020-06-12T02:37:39.469276+0000","last_change":"2020-06-12T02:37:27.901368+0000","last_active":"2020-06-12T02:37:39.469276+0000","last_peered":"2020-06-12T02:37:39.469276+0000","last_clean":"2020-06-12T02:37:39.469276+0000","last_became_active":"2020-06-12T02:37:27.900888+0000","last_became_peered":"2020-06-12T02:37:27.900888+0000","last_unstale":"2020-06-12T02:37:39.469276+0000","last_undegraded":"2020-06-12T02:37:39.469276+0000","last_fullsized":"2020-06-12T02:37:39.469276+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"17","reported_epoch":"28","state":"active+undersized","last_fresh":"2020-06-12T02:37:39.469504+0000","last_change":"2020-06-12T02:37:25.889086+0000","last_active":"2020-06-12T02:37:39.469504+0000","last_peered":"2020-06-12T02:37:39.469504+0000","last_clean":"2020-06-12T02:37:15.738781+0000","last_became_active":"2020-06-12T02:37:25.889086+0000","last_became_peered":"2020-06-12T02:37:25.889086+0000","last_unstale":"2020-06-12T02:37:39.469504+0000","last_undegraded":"2020-06-12T02:37:39.469504+0000","last_fullsized":"2020-06-12T02:37:25.888533+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"16","reported_epoch":"28","state":"active+clean","last_fresh":"2020-06-12T02:37:39.460912+0000","last_change":"2020-06-12T02:37:27.901486+0000","last_active":"2020-06-12T02:37:39.460912+0000","last_peered":"2020-06-12T02:37:39.460912+0000","last_clean":"2020-06-12T02:37:39.460912+0000","last_became_active":"2020-06-12T02:37:27.901018+0000","last_became_peered":"2020-06-12T02:37:27.901018+0000","last_unstale":"2020-06-12T02:37:39.460912+0000","last_undegraded":"2020-06-12T02:37:39.460912+0000","last_fullsized":"2020-06-12T02:37:39.460912+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"28'359","reported_seq":"444","reported_epoch":"28","state":"active+undersized+degraded","last_fresh":"2020-06-12T02:37:41.281082+0000","last_change":"2020-06-12T02:37:41.078810+0000","last_active":"2020-06-12T02:37:41.281082+0000","last_peered":"2020-06-12T02:37:41.281082+0000","last_clean":"2020-06-12T02:37:26.883460+0000","last_became_active":"2020-06-12T02:37:27.903056+0000","last_became_peered":"2020-06-12T02:37:27.903056+0000","last_unstale":"2020-06-12T02:37:41.281082+0000","last_undegraded":"2020-06-12T02:37:41.078497+0000","last_fullsized":"2020-06-12T02:37:27.889516+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":359,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3743744,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":129,"num_read_kb":72,"num_write":282,"num_write_kb":78840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647],"acting":[3,0,2147483647],"avail_no_missing":["3(0)","0(1)"],"object_location_counts":[{"shards":"0(1),3(0)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"20","reported_epoch":"28","state":"active+clean","last_fresh":"2020-06-12T02:37:39.461115+0000","last_change":"2020-06-12T02:37:16.763049+0000","last_active":"2020-06-12T02:37:39.461115+0000","last_peered":"2020-06-12T02:37:39.461115+0000","last_clean":"2020-06-12T02:37:39.461115+0000","last_became_active":"2020-06-12T02:37:16.762626+0000","last_became_peered":"2020-06-12T02:37:16.762626+0000","last_unstale":"2020-06-12T02:37:39.461115+0000","last_undegraded":"2020-06-12T02:37:39.461115+0000","last_fullsized":"2020-06-12T02:37:39.461115+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"25","reported_epoch":"28","state":"active+undersized","last_fresh":"2020-06-12T02:37:39.466170+0000","last_change":"2020-06-12T02:37:25.886626+0000","last_active":"2020-06-12T02:37:39.466170+0000","last_peered":"2020-06-12T02:37:39.466170+0000","last_clean":"2020-06-12T02:37:17.765801+0000","last_became_active":"2020-06-12T02:37:25.886626+0000","last_became_peered":"2020-06-12T02:37:25.886626+0000","last_unstale":"2020-06-12T02:37:39.466170+0000","last_undegraded":"2020-06-12T02:37:39.466170+0000","last_fullsized":"2020-06-12T02:37:25.885994+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"16","reported_epoch":"28","state":"active+clean","last_fresh":"2020-06-12T02:37:39.469595+0000","last_change":"2020-06-12T02:37:27.906478+0000","last_active":"2020-06-12T02:37:39.469595+0000","last_peered":"2020-06-12T02:37:39.469595+0000","last_clean":"2020-06-12T02:37:39.469595+0000","last_became_active":"2020-06-12T02:37:27.905639+0000","last_became_peered":"2020-06-12T02:37:27.905639+0000","last_unstale":"2020-06-12T02:37:39.469595+0000","last_undegraded":"2020-06-12T02:37:39.469595+0000","last_fullsized":"2020-06-12T02:37:39.469595+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":16400384,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1297,"num_read_kb":692,"num_write":2804,"num_write_kb":790592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17678336,"data_stored":17678336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3733,"ondisk_log_size":3733,"up":48,"acting":48,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048760,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323080,"statfs":{"total":96636764160,"available":95562833920,"internally_reserved":1073741824,"allocated":180224,"data_stored":62762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97599999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6910000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2629999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1359999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5900000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1100000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049060,"kb_used_data":476,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322780,"statfs":{"total":96636764160,"available":95562526720,"internally_reserved":1073741824,"allocated":487424,"data_stored":310999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.655}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0309999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1160000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81899999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90900000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.784}]}]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050956,"kb_used_data":2372,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93320884,"statfs":{"total":96636764160,"available":95560585216,"internally_reserved":1073741824,"allocated":2428928,"data_stored":2252503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.876}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96799999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.03}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95099999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0549999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1052428,"kb_used_data":3456,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93319412,"statfs":{"total":96636764160,"available":95559077888,"internally_reserved":1073741824,"allocated":3538944,"data_stored":3362519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84599999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1850000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.079}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.034}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]}]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1054576,"kb_used_data":5992,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93317264,"statfs":{"total":96636764160,"available":95556878336,"internally_reserved":1073741824,"allocated":6135808,"data_stored":5959383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9259999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.21}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0249999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1419999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1839999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050212,"kb_used_data":1628,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321628,"statfs":{"total":96636764160,"available":95561347072,"internally_reserved":1073741824,"allocated":1667072,"data_stored":1490647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2609999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.1079999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1479999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.0410000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5720000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1051824,"kb_used_data":3240,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93320016,"statfs":{"total":96636764160,"available":95559696384,"internally_reserved":1073741824,"allocated":3317760,"data_stored":3135269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.1480000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.879}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76300000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3759999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1051100,"kb_used_data":2516,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93320740,"statfs":{"total":96636764160,"available":95560437760,"internally_reserved":1073741824,"allocated":2576384,"data_stored":2399959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1220000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91000000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.899}]}]}]}} 2020-06-12T02:37:46.781 INFO:tasks.rados.rados.0.smithi107.stdout:2848: writing smithi10729407-9 from 4128768 to 4620288 tid 1 2020-06-12T02:37:46.784 INFO:tasks.ceph.ceph_manager.ceph:would drop below min_pgs, proj -9, currently 1 2020-06-12T02:37:46.785 INFO:tasks.rados.rados.0.smithi107.stdout:2848: writing smithi10729407-9 from 4620288 to 5095424 tid 2 2020-06-12T02:37:46.786 INFO:tasks.rados.rados.0.smithi107.stdout:2848: writing smithi10729407-9 from 5095424 to 5767168 tid 3 2020-06-12T02:37:46.789 INFO:tasks.rados.rados.0.smithi107.stdout:2848: writing smithi10729407-9 from 5767168 to 6225920 tid 4 2020-06-12T02:37:46.789 INFO:tasks.rados.rados.0.smithi107.stdout:2849: delete oid 18 current snap is 0 2020-06-12T02:37:46.789 INFO:tasks.rados.rados.0.smithi107.stdout:2847: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:46.790 INFO:tasks.rados.rados.0.smithi107.stdout:2847: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:46.790 INFO:tasks.rados.rados.0.smithi107.stdout:2847: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:46.790 INFO:tasks.rados.rados.0.smithi107.stdout:2847: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:46.791 INFO:tasks.rados.rados.0.smithi107.stdout:2847: finishing write tid 6 to smithi10729407-16 2020-06-12T02:37:46.791 INFO:tasks.rados.rados.0.smithi107.stdout:2847: finishing write tid 7 to smithi10729407-16 2020-06-12T02:37:46.791 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 480 (ObjNum 2846 snap 0 seq_num 2846) dirty exists 2020-06-12T02:37:46.791 INFO:tasks.rados.rados.0.smithi107.stdout:2847: left oid 16 (ObjNum 2846 snap 0 seq_num 2846) 2020-06-12T02:37:46.792 INFO:tasks.rados.rados.0.smithi107.stdout:2848: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:46.792 INFO:tasks.rados.rados.0.smithi107.stdout:2848: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:46.797 INFO:tasks.rados.rados.0.smithi107.stdout:2848: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:46.798 INFO:tasks.rados.rados.0.smithi107.stdout:2848: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:46.798 INFO:tasks.rados.rados.0.smithi107.stdout:2848: finishing write tid 6 to smithi10729407-9 2020-06-12T02:37:46.800 INFO:tasks.rados.rados.0.smithi107.stdout:2848: finishing write tid 7 to smithi10729407-9 2020-06-12T02:37:46.800 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 494 (ObjNum 2847 snap 0 seq_num 2847) dirty exists 2020-06-12T02:37:46.800 INFO:tasks.rados.rados.0.smithi107.stdout:2848: left oid 9 (ObjNum 2847 snap 0 seq_num 2847) 2020-06-12T02:37:46.802 INFO:tasks.rados.rados.0.smithi107.stdout:2847: done (2 left) 2020-06-12T02:37:46.803 INFO:tasks.rados.rados.0.smithi107.stdout:2848: done (1 left) 2020-06-12T02:37:46.803 INFO:tasks.rados.rados.0.smithi107.stdout:2849: done (0 left) 2020-06-12T02:37:46.803 INFO:tasks.rados.rados.0.smithi107.stdout:2850: delete oid 10 current snap is 0 2020-06-12T02:37:46.805 INFO:tasks.rados.rados.0.smithi107.stdout:2850: done (0 left) 2020-06-12T02:37:46.805 INFO:tasks.rados.rados.0.smithi107.stdout:2851: delete oid 5 current snap is 0 2020-06-12T02:37:46.807 INFO:tasks.rados.rados.0.smithi107.stdout:2851: done (0 left) 2020-06-12T02:37:46.807 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:37:46.808 INFO:tasks.rados.rados.0.smithi107.stdout:2852: seq_num 2851 ranges {0=557056,557056=466944,1024000=647168,1671168=491520} 2020-06-12T02:37:46.810 INFO:tasks.rados.rados.0.smithi107.stdout:2852: writing smithi10729407-3 from 0 to 557056 tid 1 2020-06-12T02:37:46.814 INFO:tasks.rados.rados.0.smithi107.stdout:2852: writing smithi10729407-3 from 557056 to 1024000 tid 2 2020-06-12T02:37:46.821 INFO:tasks.rados.rados.0.smithi107.stdout:2852: writing smithi10729407-3 from 1024000 to 1671168 tid 3 2020-06-12T02:37:46.822 INFO:tasks.rados.rados.0.smithi107.stdout:2852: writing smithi10729407-3 from 1671168 to 2162688 tid 4 2020-06-12T02:37:46.822 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:37:46.823 INFO:tasks.rados.rados.0.smithi107.stdout:2853: seq_num 2852 ranges {0=573440,573440=753664,1327104=409600,1736704=442368} 2020-06-12T02:37:46.825 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:37:46.822+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:37:46.826 INFO:tasks.rados.rados.0.smithi107.stdout:2853: writing smithi10729407-11 from 0 to 573440 tid 1 2020-06-12T02:37:46.831 INFO:tasks.rados.rados.0.smithi107.stdout:2853: writing smithi10729407-11 from 573440 to 1327104 tid 2 2020-06-12T02:37:46.834 INFO:tasks.rados.rados.0.smithi107.stdout:2853: writing smithi10729407-11 from 1327104 to 1736704 tid 3 2020-06-12T02:37:46.835 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:46.836 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:46.837 INFO:tasks.rados.rados.0.smithi107.stdout:2853: writing smithi10729407-11 from 1736704 to 2179072 tid 4 2020-06-12T02:37:46.837 INFO:tasks.rados.rados.0.smithi107.stdout:2854: delete oid 4 current snap is 0 2020-06-12T02:37:46.838 INFO:tasks.rados.rados.0.smithi107.stdout:2852: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:46.838 INFO:tasks.rados.rados.0.smithi107.stdout:2852: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:46.839 INFO:tasks.rados.rados.0.smithi107.stdout:2852: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:46.839 INFO:tasks.rados.rados.0.smithi107.stdout:2852: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:46.839 INFO:tasks.rados.rados.0.smithi107.stdout:2852: finishing write tid 6 to smithi10729407-3 2020-06-12T02:37:46.840 INFO:tasks.rados.rados.0.smithi107.stdout:2853: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:46.843 INFO:tasks.rados.rados.0.smithi107.stdout:2852: finishing write tid 7 to smithi10729407-3 2020-06-12T02:37:46.843 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 485 (ObjNum 2851 snap 0 seq_num 2851) dirty exists 2020-06-12T02:37:46.843 INFO:tasks.rados.rados.0.smithi107.stdout:2852: left oid 3 (ObjNum 2851 snap 0 seq_num 2851) 2020-06-12T02:37:46.844 INFO:tasks.rados.rados.0.smithi107.stdout:2852: done (2 left) 2020-06-12T02:37:46.844 INFO:tasks.rados.rados.0.smithi107.stdout:2854: done (1 left) 2020-06-12T02:37:46.844 INFO:tasks.rados.rados.0.smithi107.stdout:2855: delete oid 7 current snap is 0 2020-06-12T02:37:46.846 INFO:tasks.rados.rados.0.smithi107.stdout:2855: done (1 left) 2020-06-12T02:37:46.846 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:37:46.847 INFO:tasks.rados.rados.0.smithi107.stdout:2856: seq_num 2855 ranges {0=622592,622592=802816,1425408=630784,2056192=172032} 2020-06-12T02:37:46.848 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-06-12T02:37:46.849 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:37:46.850 INFO:tasks.rados.rados.0.smithi107.stdout:2856: writing smithi10729407-13 from 0 to 622592 tid 1 2020-06-12T02:37:46.857 INFO:tasks.rados.rados.0.smithi107.stdout:2856: writing smithi10729407-13 from 622592 to 1425408 tid 2 2020-06-12T02:37:46.860 INFO:tasks.rados.rados.0.smithi107.stdout:2856: writing smithi10729407-13 from 1425408 to 2056192 tid 3 2020-06-12T02:37:46.861 INFO:tasks.rados.rados.0.smithi107.stdout:2856: writing smithi10729407-13 from 2056192 to 2228224 tid 4 2020-06-12T02:37:46.861 INFO:tasks.rados.rados.0.smithi107.stdout:2857: delete oid 14 current snap is 0 2020-06-12T02:37:46.862 INFO:tasks.rados.rados.0.smithi107.stdout:2853: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:46.862 INFO:tasks.rados.rados.0.smithi107.stdout:2853: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:46.862 INFO:tasks.rados.rados.0.smithi107.stdout:2853: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:46.863 INFO:tasks.rados.rados.0.smithi107.stdout:2853: finishing write tid 6 to smithi10729407-11 2020-06-12T02:37:46.863 INFO:tasks.rados.rados.0.smithi107.stdout:2853: finishing write tid 7 to smithi10729407-11 2020-06-12T02:37:46.864 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 486 (ObjNum 2852 snap 0 seq_num 2852) dirty exists 2020-06-12T02:37:46.864 INFO:tasks.rados.rados.0.smithi107.stdout:2853: left oid 11 (ObjNum 2852 snap 0 seq_num 2852) 2020-06-12T02:37:46.867 INFO:tasks.rados.rados.0.smithi107.stdout:2856: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:46.869 INFO:tasks.rados.rados.0.smithi107.stdout:2853: done (2 left) 2020-06-12T02:37:46.869 INFO:tasks.rados.rados.0.smithi107.stdout:2857: done (1 left) 2020-06-12T02:37:46.870 INFO:tasks.rados.rados.0.smithi107.stdout:2858: delete oid 1 current snap is 0 2020-06-12T02:37:46.871 INFO:tasks.rados.rados.0.smithi107.stdout:2856: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:46.871 INFO:tasks.rados.rados.0.smithi107.stdout:2856: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:46.871 INFO:tasks.rados.rados.0.smithi107.stdout:2856: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:46.872 INFO:tasks.rados.rados.0.smithi107.stdout:2856: finishing write tid 6 to smithi10729407-13 2020-06-12T02:37:46.875 INFO:tasks.rados.rados.0.smithi107.stdout:2858: done (1 left) 2020-06-12T02:37:46.875 INFO:tasks.rados.rados.0.smithi107.stdout:2859: delete oid 11 current snap is 0 2020-06-12T02:37:46.875 INFO:tasks.rados.rados.0.smithi107.stdout:2856: finishing write tid 7 to smithi10729407-13 2020-06-12T02:37:46.875 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 558 (ObjNum 2855 snap 0 seq_num 2855) dirty exists 2020-06-12T02:37:46.876 INFO:tasks.rados.rados.0.smithi107.stdout:2856: left oid 13 (ObjNum 2855 snap 0 seq_num 2855) 2020-06-12T02:37:46.877 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:37:46.878 INFO:tasks.rados.rados.0.smithi107.stdout:2856: done (1 left) 2020-06-12T02:37:46.878 INFO:tasks.rados.rados.0.smithi107.stdout:2859: done (0 left) 2020-06-12T02:37:46.879 INFO:tasks.rados.rados.0.smithi107.stdout:2860: delete oid 16 current snap is 0 2020-06-12T02:37:46.884 INFO:tasks.rados.rados.0.smithi107.stdout:2860: done (0 left) 2020-06-12T02:37:46.885 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:37:46.885 INFO:tasks.rados.rados.0.smithi107.stdout:2861: seq_num 2860 ranges {0=704512,704512=671744,1376256=417792,1794048=516096} 2020-06-12T02:37:46.888 INFO:tasks.rados.rados.0.smithi107.stdout:2861: writing smithi10729407-18 from 0 to 704512 tid 1 2020-06-12T02:37:46.893 INFO:tasks.rados.rados.0.smithi107.stdout:2861: writing smithi10729407-18 from 704512 to 1376256 tid 2 2020-06-12T02:37:46.896 INFO:tasks.rados.rados.0.smithi107.stdout:2861: writing smithi10729407-18 from 1376256 to 1794048 tid 3 2020-06-12T02:37:46.899 INFO:tasks.rados.rados.0.smithi107.stdout:2861: writing smithi10729407-18 from 1794048 to 2310144 tid 4 2020-06-12T02:37:46.899 INFO:tasks.rados.rados.0.smithi107.stdout:2862: delete oid 13 current snap is 0 2020-06-12T02:37:46.900 INFO:tasks.rados.rados.0.smithi107.stdout:2861: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:46.904 INFO:tasks.rados.rados.0.smithi107.stdout:2862: done (1 left) 2020-06-12T02:37:46.904 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:37:46.905 INFO:tasks.rados.rados.0.smithi107.stdout:2863: seq_num 2862 ranges {0=737280,737280=442368,1179648=786432,1966080=376832} 2020-06-12T02:37:46.911 INFO:tasks.rados.rados.0.smithi107.stdout:2863: writing smithi10729407-8 from 0 to 737280 tid 1 2020-06-12T02:37:46.912 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:37:46.914 INFO:tasks.rados.rados.0.smithi107.stdout:2863: writing smithi10729407-8 from 737280 to 1179648 tid 2 2020-06-12T02:37:46.920 INFO:tasks.rados.rados.0.smithi107.stdout:2863: writing smithi10729407-8 from 1179648 to 1966080 tid 3 2020-06-12T02:37:46.922 INFO:tasks.rados.rados.0.smithi107.stdout:2863: writing smithi10729407-8 from 1966080 to 2342912 tid 4 2020-06-12T02:37:46.923 INFO:tasks.rados.rados.0.smithi107.stdout:2864: delete oid 7 current snap is 0 2020-06-12T02:37:46.923 INFO:tasks.rados.rados.0.smithi107.stdout:2861: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:46.923 INFO:tasks.rados.rados.0.smithi107.stdout:2861: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:46.923 INFO:tasks.rados.rados.0.smithi107.stdout:2861: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:46.923 INFO:tasks.rados.rados.0.smithi107.stdout:2861: finishing write tid 6 to smithi10729407-18 2020-06-12T02:37:46.924 INFO:tasks.rados.rados.0.smithi107.stdout:2861: finishing write tid 7 to smithi10729407-18 2020-06-12T02:37:46.924 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 500 (ObjNum 2860 snap 0 seq_num 2860) dirty exists 2020-06-12T02:37:46.924 INFO:tasks.rados.rados.0.smithi107.stdout:2861: left oid 18 (ObjNum 2860 snap 0 seq_num 2860) 2020-06-12T02:37:46.924 INFO:tasks.rados.rados.0.smithi107.stdout:2863: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:46.927 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:46.924+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:46.927 INFO:tasks.rados.rados.0.smithi107.stdout:2861: done (2 left) 2020-06-12T02:37:46.928 INFO:tasks.rados.rados.0.smithi107.stdout:2864: done (1 left) 2020-06-12T02:37:46.928 INFO:tasks.rados.rados.0.smithi107.stdout:2865: delete oid 4 current snap is 0 2020-06-12T02:37:46.928 INFO:tasks.rados.rados.0.smithi107.stdout:2863: finishing write tid 2 to smithi10729407-8 2020-06-12T02:37:46.931 INFO:tasks.rados.rados.0.smithi107.stdout:2865: done (1 left) 2020-06-12T02:37:46.931 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:37:46.931 INFO:tasks.rados.rados.0.smithi107.stdout:2866: seq_num 2865 ranges {1671168=786432,2457600=499712,2957312=557056,3514368=548864} 2020-06-12T02:37:46.941 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:37:46.945 INFO:tasks.rados.rados.0.smithi107.stdout:2866: writing smithi10729407-20 from 1671168 to 2457600 tid 1 2020-06-12T02:37:46.950 INFO:tasks.rados.rados.0.smithi107.stdout:2866: writing smithi10729407-20 from 2457600 to 2957312 tid 2 2020-06-12T02:37:46.955 INFO:tasks.rados.rados.0.smithi107.stdout:2866: writing smithi10729407-20 from 2957312 to 3514368 tid 3 2020-06-12T02:37:46.958 INFO:tasks.rados.rados.0.smithi107.stdout:2866: writing smithi10729407-20 from 3514368 to 4063232 tid 4 2020-06-12T02:37:46.959 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:37:46.959 INFO:tasks.rados.rados.0.smithi107.stdout:2867: seq_num 2866 ranges {2310144=802816,3112960=778240,3891200=761856,4653056=65536} 2020-06-12T02:37:46.976 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:37:46.977 INFO:tasks.rados.rados.0.smithi107.stdout:2867: writing smithi10729407-18 from 2310144 to 3112960 tid 1 2020-06-12T02:37:46.983 INFO:tasks.rados.rados.0.smithi107.stdout:2867: writing smithi10729407-18 from 3112960 to 3891200 tid 2 2020-06-12T02:37:46.992 INFO:tasks.rados.rados.0.smithi107.stdout:2867: writing smithi10729407-18 from 3891200 to 4653056 tid 3 2020-06-12T02:37:46.992 INFO:tasks.rados.rados.0.smithi107.stdout:2867: writing smithi10729407-18 from 4653056 to 4718592 tid 4 2020-06-12T02:37:46.992 INFO:tasks.rados.rados.0.smithi107.stdout:2868: delete oid 14 current snap is 0 2020-06-12T02:37:46.993 INFO:tasks.rados.rados.0.smithi107.stdout:2863: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:46.993 INFO:tasks.rados.rados.0.smithi107.stdout:2863: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:46.993 INFO:tasks.rados.rados.0.smithi107.stdout:2863: finishing write tid 6 to smithi10729407-8 2020-06-12T02:37:46.993 INFO:tasks.rados.rados.0.smithi107.stdout:2863: finishing write tid 7 to smithi10729407-8 2020-06-12T02:37:46.993 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 480 (ObjNum 2862 snap 0 seq_num 2862) dirty exists 2020-06-12T02:37:46.994 INFO:tasks.rados.rados.0.smithi107.stdout:2863: left oid 8 (ObjNum 2862 snap 0 seq_num 2862) 2020-06-12T02:37:46.994 INFO:tasks.rados.rados.0.smithi107.stdout:2866: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:46.994 INFO:tasks.rados.rados.0.smithi107.stdout:2866: finishing write tid 2 to smithi10729407-20 2020-06-12T02:37:46.994 INFO:tasks.rados.rados.0.smithi107.stdout:2866: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:46.995 INFO:tasks.rados.rados.0.smithi107.stdout:2866: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:46.995 INFO:tasks.rados.rados.0.smithi107.stdout:2866: finishing write tid 6 to smithi10729407-20 2020-06-12T02:37:46.995 INFO:tasks.rados.rados.0.smithi107.stdout:2866: finishing write tid 7 to smithi10729407-20 2020-06-12T02:37:46.999 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 190 (ObjNum 2865 snap 0 seq_num 2865) dirty exists 2020-06-12T02:37:46.999 INFO:tasks.rados.rados.0.smithi107.stdout:2866: left oid 20 (ObjNum 2865 snap 0 seq_num 2865) 2020-06-12T02:37:46.999 INFO:tasks.rados.rados.0.smithi107.stdout:2867: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:47.000 INFO:tasks.rados.rados.0.smithi107.stdout:2863: done (3 left) 2020-06-12T02:37:47.000 INFO:tasks.rados.rados.0.smithi107.stdout:2866: done (2 left) 2020-06-12T02:37:47.000 INFO:tasks.rados.rados.0.smithi107.stdout:2868: done (1 left) 2020-06-12T02:37:47.001 INFO:tasks.rados.rados.0.smithi107.stdout:2869: delete oid 13 current snap is 0 2020-06-12T02:37:47.001 INFO:tasks.rados.rados.0.smithi107.stdout:2867: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:47.001 INFO:tasks.rados.rados.0.smithi107.stdout:2867: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:47.002 INFO:tasks.rados.rados.0.smithi107.stdout:2869: done (1 left) 2020-06-12T02:37:47.002 INFO:tasks.rados.rados.0.smithi107.stdout:2870: delete oid 19 current snap is 0 2020-06-12T02:37:47.003 INFO:tasks.rados.rados.0.smithi107.stdout:2867: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:47.003 INFO:tasks.rados.rados.0.smithi107.stdout:2867: finishing write tid 6 to smithi10729407-18 2020-06-12T02:37:47.005 INFO:tasks.rados.rados.0.smithi107.stdout:2867: finishing write tid 7 to smithi10729407-18 2020-06-12T02:37:47.005 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 505 (ObjNum 2866 snap 0 seq_num 2866) dirty exists 2020-06-12T02:37:47.005 INFO:tasks.rados.rados.0.smithi107.stdout:2867: left oid 18 (ObjNum 2866 snap 0 seq_num 2866) 2020-06-12T02:37:47.005 INFO:tasks.rados.rados.0.smithi107.stdout:2867: done (1 left) 2020-06-12T02:37:47.006 INFO:tasks.rados.rados.0.smithi107.stdout:2870: done (0 left) 2020-06-12T02:37:47.006 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:47.006 INFO:tasks.rados.rados.0.smithi107.stdout:2871: seq_num 2870 ranges {0=73728} 2020-06-12T02:37:47.006 INFO:tasks.rados.rados.0.smithi107.stdout:2871: writing smithi10729407-2 from 0 to 73728 tid 1 2020-06-12T02:37:47.007 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:37:47.007 INFO:tasks.rados.rados.0.smithi107.stdout:2872: seq_num 2871 ranges {0=90112} 2020-06-12T02:37:47.007 INFO:tasks.rados.rados.0.smithi107.stdout:2872: writing smithi10729407-15 from 0 to 90112 tid 1 2020-06-12T02:37:47.007 INFO:tasks.rados.rados.0.smithi107.stdout:2873: delete oid 18 current snap is 0 2020-06-12T02:37:47.008 INFO:tasks.rados.rados.0.smithi107.stdout:2871: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:47.008 INFO:tasks.rados.rados.0.smithi107.stdout:2873: done (2 left) 2020-06-12T02:37:47.008 INFO:tasks.rados.rados.0.smithi107.stdout:2874: delete oid 9 current snap is 0 2020-06-12T02:37:47.009 INFO:tasks.rados.rados.0.smithi107.stdout:2872: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:47.009 INFO:tasks.rados.rados.0.smithi107.stdout:2872: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:47.009 INFO:tasks.rados.rados.0.smithi107.stdout:2872: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:47.010 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 482 (ObjNum 2871 snap 0 seq_num 2871) dirty exists 2020-06-12T02:37:47.010 INFO:tasks.rados.rados.0.smithi107.stdout:2872: left oid 15 (ObjNum 2871 snap 0 seq_num 2871) 2020-06-12T02:37:47.011 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:37:47.011 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:37:47.011 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2020-06-12T02:37:47.011 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2020-06-12T02:37:47.012 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:37:47.012 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:37:47.012 INFO:tasks.rados.rados.0.smithi107.stdout:2871: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:47.014 INFO:tasks.rados.rados.0.smithi107.stdout:2871: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:47.014 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 300 (ObjNum 2870 snap 0 seq_num 2870) dirty exists 2020-06-12T02:37:47.014 INFO:tasks.rados.rados.0.smithi107.stdout:2871: left oid 2 (ObjNum 2870 snap 0 seq_num 2870) 2020-06-12T02:37:47.014 INFO:tasks.rados.rados.0.smithi107.stdout:2871: done (2 left) 2020-06-12T02:37:47.014 INFO:tasks.rados.rados.0.smithi107.stdout:2872: done (1 left) 2020-06-12T02:37:47.015 INFO:tasks.rados.rados.0.smithi107.stdout:2874: done (0 left) 2020-06-12T02:37:47.015 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:37:47.015 INFO:tasks.rados.rados.0.smithi107.stdout:2875: seq_num 2874 ranges {4063232=139264} 2020-06-12T02:37:47.022 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-06-12T02:37:47.036 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:37:47.036 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:47.033+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:47.037 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-06-12T02:37:47.037 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-06-12T02:37:47.037 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-06-12T02:37:47.037 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-06-12T02:37:47.038 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-06-12T02:37:47.038 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-06-12T02:37:47.038 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-06-12T02:37:47.038 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.7 2020-06-12T02:37:47.038 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: problem getting command descriptions from osd.7 2020-06-12T02:37:47.041 INFO:tasks.rados.rados.0.smithi107.stdout:2875: writing smithi10729407-20 from 4063232 to 4202496 tid 1 2020-06-12T02:37:47.041 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:37:47.041 INFO:tasks.rados.rados.0.smithi107.stdout:2876: seq_num 2875 ranges {0=155648} 2020-06-12T02:37:47.042 INFO:tasks.rados.rados.0.smithi107.stdout:2876: writing smithi10729407-7 from 0 to 155648 tid 1 2020-06-12T02:37:47.042 INFO:tasks.rados.rados.0.smithi107.stdout:2877: delete oid 1 current snap is 0 2020-06-12T02:37:47.042 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-06-12T02:37:47.042 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-06-12T02:37:47.043 INFO:tasks.rados.rados.0.smithi107.stdout:2875: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:47.044 INFO:tasks.rados.rados.0.smithi107.stdout:2877: done (2 left) 2020-06-12T02:37:47.044 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:37:47.044 INFO:tasks.rados.rados.0.smithi107.stdout:2878: seq_num 2877 ranges {0=188416} 2020-06-12T02:37:47.045 INFO:tasks.rados.rados.0.smithi107.stdout:2878: writing smithi10729407-12 from 0 to 188416 tid 1 2020-06-12T02:37:47.045 INFO:tasks.rados.rados.0.smithi107.stdout:2879: delete oid 8 current snap is 0 2020-06-12T02:37:47.045 INFO:tasks.rados.rados.0.smithi107.stdout:2876: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:47.045 INFO:tasks.rados.rados.0.smithi107.stdout:2875: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:47.046 INFO:tasks.rados.rados.0.smithi107.stdout:2875: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:47.046 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 192 (ObjNum 2874 snap 0 seq_num 2874) dirty exists 2020-06-12T02:37:47.046 INFO:tasks.rados.rados.0.smithi107.stdout:2875: left oid 20 (ObjNum 2874 snap 0 seq_num 2874) 2020-06-12T02:37:47.048 INFO:tasks.rados.rados.0.smithi107.stdout:2876: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:47.052 INFO:tasks.rados.rados.0.smithi107.stdout:2876: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:47.052 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 563 (ObjNum 2875 snap 0 seq_num 2875) dirty exists 2020-06-12T02:37:47.052 INFO:tasks.rados.rados.0.smithi107.stdout:2876: left oid 7 (ObjNum 2875 snap 0 seq_num 2875) 2020-06-12T02:37:47.052 INFO:tasks.rados.rados.0.smithi107.stdout:2875: done (3 left) 2020-06-12T02:37:47.053 INFO:tasks.rados.rados.0.smithi107.stdout:2876: done (2 left) 2020-06-12T02:37:47.053 INFO:tasks.rados.rados.0.smithi107.stdout:2879: done (1 left) 2020-06-12T02:37:47.053 INFO:tasks.rados.rados.0.smithi107.stdout:2880: delete oid 15 current snap is 0 2020-06-12T02:37:47.054 INFO:tasks.rados.rados.0.smithi107.stdout:2878: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:47.055 INFO:tasks.rados.rados.0.smithi107.stdout:2878: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:47.057 INFO:tasks.rados.rados.0.smithi107.stdout:2880: done (1 left) 2020-06-12T02:37:47.058 INFO:tasks.rados.rados.0.smithi107.stdout:2881: delete oid 20 current snap is 0 2020-06-12T02:37:47.059 INFO:tasks.rados.rados.0.smithi107.stdout:2878: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:47.061 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 468 (ObjNum 2877 snap 0 seq_num 2877) dirty exists 2020-06-12T02:37:47.061 INFO:tasks.rados.rados.0.smithi107.stdout:2878: left oid 12 (ObjNum 2877 snap 0 seq_num 2877) 2020-06-12T02:37:47.061 INFO:tasks.rados.rados.0.smithi107.stdout:2878: done (1 left) 2020-06-12T02:37:47.062 INFO:tasks.rados.rados.0.smithi107.stdout:2881: done (0 left) 2020-06-12T02:37:47.062 INFO:tasks.rados.rados.0.smithi107.stdout:2882: delete oid 14 current snap is 0 2020-06-12T02:37:47.062 INFO:tasks.rados.rados.0.smithi107.stdout:2882: done (0 left) 2020-06-12T02:37:47.063 INFO:tasks.rados.rados.0.smithi107.stdout:2883: delete oid 12 current snap is 0 2020-06-12T02:37:47.065 INFO:tasks.rados.rados.0.smithi107.stdout:2883: done (0 left) 2020-06-12T02:37:47.065 INFO:tasks.rados.rados.0.smithi107.stdout:2884: delete oid 3 current snap is 0 2020-06-12T02:37:47.067 INFO:tasks.rados.rados.0.smithi107.stdout:2884: done (0 left) 2020-06-12T02:37:47.067 INFO:tasks.rados.rados.0.smithi107.stdout:2885: delete oid 2 current snap is 0 2020-06-12T02:37:47.069 INFO:tasks.rados.rados.0.smithi107.stdout:2885: done (0 left) 2020-06-12T02:37:47.069 INFO:tasks.rados.rados.0.smithi107.stdout:2886: delete oid 12 current snap is 0 2020-06-12T02:37:47.071 INFO:tasks.rados.rados.0.smithi107.stdout:2886: done (0 left) 2020-06-12T02:37:47.071 INFO:tasks.rados.rados.0.smithi107.stdout:2887: delete oid 6 current snap is 0 2020-06-12T02:37:47.072 INFO:tasks.rados.rados.0.smithi107.stdout:2887: done (0 left) 2020-06-12T02:37:47.073 INFO:tasks.rados.rados.0.smithi107.stdout:2888: delete oid 11 current snap is 0 2020-06-12T02:37:47.075 INFO:tasks.rados.rados.0.smithi107.stdout:2888: done (0 left) 2020-06-12T02:37:47.075 INFO:tasks.rados.rados.0.smithi107.stdout:2889: delete oid 13 current snap is 0 2020-06-12T02:37:47.080 INFO:tasks.rados.rados.0.smithi107.stdout:2889: done (0 left) 2020-06-12T02:37:47.080 INFO:tasks.rados.rados.0.smithi107.stdout:2890: delete oid 13 current snap is 0 2020-06-12T02:37:47.081 INFO:tasks.rados.rados.0.smithi107.stdout:2890: done (0 left) 2020-06-12T02:37:47.082 INFO:tasks.rados.rados.0.smithi107.stdout:2891: delete oid 18 current snap is 0 2020-06-12T02:37:47.084 INFO:tasks.rados.rados.0.smithi107.stdout:2891: done (0 left) 2020-06-12T02:37:47.084 INFO:tasks.rados.rados.0.smithi107.stdout:2892: delete oid 15 current snap is 0 2020-06-12T02:37:47.086 INFO:tasks.rados.rados.0.smithi107.stdout:2892: done (0 left) 2020-06-12T02:37:47.087 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:37:47.087 INFO:tasks.rados.rados.0.smithi107.stdout:2893: seq_num 2892 ranges {0=442368} 2020-06-12T02:37:47.089 INFO:tasks.rados.rados.0.smithi107.stdout:2893: writing smithi10729407-18 from 0 to 442368 tid 1 2020-06-12T02:37:47.089 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:37:47.089 INFO:tasks.rados.rados.0.smithi107.stdout:2894: seq_num 2893 ranges {0=458752} 2020-06-12T02:37:47.092 INFO:tasks.rados.rados.0.smithi107.stdout:2894: writing smithi10729407-20 from 0 to 458752 tid 1 2020-06-12T02:37:47.092 INFO:tasks.rados.rados.0.smithi107.stdout:2895: delete oid 17 current snap is 0 2020-06-12T02:37:47.092 INFO:tasks.rados.rados.0.smithi107.stdout:2893: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:47.095 INFO:tasks.rados.rados.0.smithi107.stdout:2895: done (2 left) 2020-06-12T02:37:47.095 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:37:47.095 INFO:tasks.rados.rados.0.smithi107.stdout:2896: seq_num 2895 ranges {0=491520} 2020-06-12T02:37:47.098 INFO:tasks.rados.rados.0.smithi107.stdout:2896: writing smithi10729407-11 from 0 to 491520 tid 1 2020-06-12T02:37:47.098 INFO:tasks.rados.rados.0.smithi107.stdout:2897: delete oid 15 current snap is 0 2020-06-12T02:37:47.098 INFO:tasks.rados.rados.0.smithi107.stdout:2894: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:47.099 INFO:tasks.rados.rados.0.smithi107.stdout:2894: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:47.099 INFO:tasks.rados.rados.0.smithi107.stdout:2893: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:47.100 INFO:tasks.rados.rados.0.smithi107.stdout:2894: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:47.100 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 195 (ObjNum 2893 snap 0 seq_num 2893) dirty exists 2020-06-12T02:37:47.100 INFO:tasks.rados.rados.0.smithi107.stdout:2894: left oid 20 (ObjNum 2893 snap 0 seq_num 2893) 2020-06-12T02:37:47.102 INFO:tasks.rados.rados.0.smithi107.stdout:2893: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:47.102 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 510 (ObjNum 2892 snap 0 seq_num 2892) dirty exists 2020-06-12T02:37:47.103 INFO:tasks.rados.rados.0.smithi107.stdout:2893: left oid 18 (ObjNum 2892 snap 0 seq_num 2892) 2020-06-12T02:37:47.103 INFO:tasks.rados.rados.0.smithi107.stdout:2893: done (3 left) 2020-06-12T02:37:47.103 INFO:tasks.rados.rados.0.smithi107.stdout:2894: done (2 left) 2020-06-12T02:37:47.104 INFO:tasks.rados.rados.0.smithi107.stdout:2897: done (1 left) 2020-06-12T02:37:47.104 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:37:47.104 INFO:tasks.rados.rados.0.smithi107.stdout:2898: seq_num 2897 ranges {0=524288} 2020-06-12T02:37:47.112 INFO:tasks.rados.rados.0.smithi107.stdout:2898: writing smithi10729407-3 from 0 to 524288 tid 1 2020-06-12T02:37:47.113 INFO:tasks.rados.rados.0.smithi107.stdout:2899: delete oid 15 current snap is 0 2020-06-12T02:37:47.113 INFO:tasks.rados.rados.0.smithi107.stdout:2896: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:47.113 INFO:tasks.rados.rados.0.smithi107.stdout:2896: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:47.113 INFO:tasks.rados.rados.0.smithi107.stdout:2896: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:47.113 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 493 (ObjNum 2895 snap 0 seq_num 2895) dirty exists 2020-06-12T02:37:47.114 INFO:tasks.rados.rados.0.smithi107.stdout:2896: left oid 11 (ObjNum 2895 snap 0 seq_num 2895) 2020-06-12T02:37:47.114 INFO:tasks.rados.rados.0.smithi107.stdout:2896: done (2 left) 2020-06-12T02:37:47.114 INFO:tasks.rados.rados.0.smithi107.stdout:2899: done (1 left) 2020-06-12T02:37:47.114 INFO:tasks.rados.rados.0.smithi107.stdout:2900: delete oid 11 current snap is 0 2020-06-12T02:37:47.115 INFO:tasks.rados.rados.0.smithi107.stdout:2900: done (1 left) 2020-06-12T02:37:47.115 INFO:tasks.rados.rados.0.smithi107.stdout:2901: delete oid 7 current snap is 0 2020-06-12T02:37:47.115 INFO:tasks.rados.rados.0.smithi107.stdout:2898: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:47.116 INFO:tasks.rados.rados.0.smithi107.stdout:2901: done (1 left) 2020-06-12T02:37:47.119 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:37:47.119 INFO:tasks.rados.rados.0.smithi107.stdout:2902: seq_num 2901 ranges {0=589824} 2020-06-12T02:37:47.120 INFO:tasks.rados.rados.0.smithi107.stdout:2902: writing smithi10729407-10 from 0 to 589824 tid 1 2020-06-12T02:37:47.121 INFO:tasks.rados.rados.0.smithi107.stdout:2903: delete oid 17 current snap is 0 2020-06-12T02:37:47.121 INFO:tasks.rados.rados.0.smithi107.stdout:2898: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:47.121 INFO:tasks.rados.rados.0.smithi107.stdout:2898: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:47.121 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 489 (ObjNum 2897 snap 0 seq_num 2897) dirty exists 2020-06-12T02:37:47.121 INFO:tasks.rados.rados.0.smithi107.stdout:2898: left oid 3 (ObjNum 2897 snap 0 seq_num 2897) 2020-06-12T02:37:47.124 INFO:tasks.rados.rados.0.smithi107.stdout:2898: done (2 left) 2020-06-12T02:37:47.124 INFO:tasks.rados.rados.0.smithi107.stdout:2903: done (1 left) 2020-06-12T02:37:47.125 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:37:47.125 INFO:tasks.rados.rados.0.smithi107.stdout:2904: seq_num 2903 ranges {0=622592} 2020-06-12T02:37:47.128 INFO:tasks.rados.rados.0.smithi107.stdout:2904: writing smithi10729407-9 from 0 to 622592 tid 1 2020-06-12T02:37:47.142 INFO:tasks.rados.rados.0.smithi107.stdout:2905: delete oid 12 current snap is 0 2020-06-12T02:37:47.142 INFO:tasks.rados.rados.0.smithi107.stdout:2902: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:47.142 INFO:tasks.rados.rados.0.smithi107.stdout:2902: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:47.142 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:47.125+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:47.144 INFO:tasks.rados.rados.0.smithi107.stdout:2902: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:47.144 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 276 (ObjNum 2901 snap 0 seq_num 2901) dirty exists 2020-06-12T02:37:47.144 INFO:tasks.rados.rados.0.smithi107.stdout:2902: left oid 10 (ObjNum 2901 snap 0 seq_num 2901) 2020-06-12T02:37:47.144 INFO:tasks.rados.rados.0.smithi107.stdout:2902: done (2 left) 2020-06-12T02:37:47.145 INFO:tasks.rados.rados.0.smithi107.stdout:2905: done (1 left) 2020-06-12T02:37:47.145 INFO:tasks.rados.rados.0.smithi107.stdout:2906: delete oid 5 current snap is 0 2020-06-12T02:37:47.145 INFO:tasks.rados.rados.0.smithi107.stdout:2906: done (1 left) 2020-06-12T02:37:47.145 INFO:tasks.rados.rados.0.smithi107.stdout:2907: delete oid 14 current snap is 0 2020-06-12T02:37:47.145 INFO:tasks.rados.rados.0.smithi107.stdout:2904: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:47.146 INFO:tasks.rados.rados.0.smithi107.stdout:2907: done (1 left) 2020-06-12T02:37:47.146 INFO:tasks.rados.rados.0.smithi107.stdout:2908: delete oid 8 current snap is 0 2020-06-12T02:37:47.146 INFO:tasks.rados.rados.0.smithi107.stdout:2908: done (1 left) 2020-06-12T02:37:47.146 INFO:tasks.rados.rados.0.smithi107.stdout:2909: delete oid 17 current snap is 0 2020-06-12T02:37:47.146 INFO:tasks.rados.rados.0.smithi107.stdout:2904: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:47.146 INFO:tasks.rados.rados.0.smithi107.stdout:2904: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:47.147 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 512 (ObjNum 2903 snap 0 seq_num 2903) dirty exists 2020-06-12T02:37:47.147 INFO:tasks.rados.rados.0.smithi107.stdout:2904: left oid 9 (ObjNum 2903 snap 0 seq_num 2903) 2020-06-12T02:37:47.147 INFO:tasks.rados.rados.0.smithi107.stdout:2904: done (1 left) 2020-06-12T02:37:47.147 INFO:tasks.rados.rados.0.smithi107.stdout:2909: done (0 left) 2020-06-12T02:37:47.148 INFO:tasks.rados.rados.0.smithi107.stdout:2910: delete oid 2 current snap is 0 2020-06-12T02:37:47.148 INFO:tasks.rados.rados.0.smithi107.stdout:2910: done (0 left) 2020-06-12T02:37:47.148 INFO:tasks.rados.rados.0.smithi107.stdout:2911: delete oid 5 current snap is 0 2020-06-12T02:37:47.148 INFO:tasks.rados.rados.0.smithi107.stdout:2911: done (0 left) 2020-06-12T02:37:47.148 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:37:47.148 INFO:tasks.rados.rados.0.smithi107.stdout:2912: seq_num 2911 ranges {0=761856} 2020-06-12T02:37:47.150 INFO:tasks.rados.rados.0.smithi107.stdout:2912: writing smithi10729407-13 from 0 to 761856 tid 1 2020-06-12T02:37:47.150 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:37:47.150 INFO:tasks.rados.rados.0.smithi107.stdout:2913: seq_num 2912 ranges {0=778240} 2020-06-12T02:37:47.152 INFO:tasks.rados.rados.0.smithi107.stdout:2913: writing smithi10729407-4 from 0 to 778240 tid 1 2020-06-12T02:37:47.152 INFO:tasks.rados.rados.0.smithi107.stdout:2914: delete oid 8 current snap is 0 2020-06-12T02:37:47.154 INFO:tasks.rados.rados.0.smithi107.stdout:2914: done (2 left) 2020-06-12T02:37:47.154 INFO:tasks.rados.rados.0.smithi107.stdout:2915: delete oid 9 current snap is 0 2020-06-12T02:37:47.156 INFO:tasks.rados.rados.0.smithi107.stdout:2912: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:47.156 INFO:tasks.rados.rados.0.smithi107.stdout:2915: done (2 left) 2020-06-12T02:37:47.157 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:37:47.157 INFO:tasks.rados.rados.0.smithi107.stdout:2916: seq_num 2915 ranges {524288=425984,950272=401408} 2020-06-12T02:37:47.162 INFO:tasks.rados.rados.0.smithi107.stdout:2916: writing smithi10729407-3 from 524288 to 950272 tid 1 2020-06-12T02:37:47.165 INFO:tasks.rados.rados.0.smithi107.stdout:2916: writing smithi10729407-3 from 950272 to 1351680 tid 2 2020-06-12T02:37:47.165 INFO:tasks.rados.rados.0.smithi107.stdout:2917: delete oid 9 current snap is 0 2020-06-12T02:37:47.165 INFO:tasks.rados.rados.0.smithi107.stdout:2913: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:47.165 INFO:tasks.rados.rados.0.smithi107.stdout:2912: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:47.166 INFO:tasks.rados.rados.0.smithi107.stdout:2913: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:47.166 INFO:tasks.rados.rados.0.smithi107.stdout:2912: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:47.166 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 568 (ObjNum 2911 snap 0 seq_num 2911) dirty exists 2020-06-12T02:37:47.166 INFO:tasks.rados.rados.0.smithi107.stdout:2912: left oid 13 (ObjNum 2911 snap 0 seq_num 2911) 2020-06-12T02:37:47.167 INFO:tasks.rados.rados.0.smithi107.stdout:2913: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:47.167 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 549 (ObjNum 2912 snap 0 seq_num 2912) dirty exists 2020-06-12T02:37:47.167 INFO:tasks.rados.rados.0.smithi107.stdout:2913: left oid 4 (ObjNum 2912 snap 0 seq_num 2912) 2020-06-12T02:37:47.170 INFO:tasks.rados.rados.0.smithi107.stdout:2916: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:47.172 INFO:tasks.rados.rados.0.smithi107.stdout:2912: done (3 left) 2020-06-12T02:37:47.172 INFO:tasks.rados.rados.0.smithi107.stdout:2913: done (2 left) 2020-06-12T02:37:47.173 INFO:tasks.rados.rados.0.smithi107.stdout:2917: done (1 left) 2020-06-12T02:37:47.173 INFO:tasks.rados.rados.0.smithi107.stdout:2918: delete oid 11 current snap is 0 2020-06-12T02:37:47.173 INFO:tasks.rados.rados.0.smithi107.stdout:2916: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:47.174 INFO:tasks.rados.rados.0.smithi107.stdout:2916: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:47.174 INFO:tasks.rados.rados.0.smithi107.stdout:2916: finishing write tid 5 to smithi10729407-3 2020-06-12T02:37:47.174 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 492 (ObjNum 2915 snap 0 seq_num 2915) dirty exists 2020-06-12T02:37:47.175 INFO:tasks.rados.rados.0.smithi107.stdout:2916: left oid 3 (ObjNum 2915 snap 0 seq_num 2915) 2020-06-12T02:37:47.177 INFO:tasks.rados.rados.0.smithi107.stdout:2916: done (1 left) 2020-06-12T02:37:47.178 INFO:tasks.rados.rados.0.smithi107.stdout:2918: done (0 left) 2020-06-12T02:37:47.178 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:37:47.178 INFO:tasks.rados.rados.0.smithi107.stdout:2919: seq_num 2918 ranges {1351680=475136,1826816=401408} 2020-06-12T02:37:47.183 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:37:47.183 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2020-06-12T02:37:47.183 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2020-06-12T02:37:47.183 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2020-06-12T02:37:47.184 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.184 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:962 3.7s0 3:ea271007:::smithi10729407-9:head [append 0~598016] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:47.184 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:32.236108+0000", 2020-06-12T02:37:47.184 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 14.943333703, 2020-06-12T02:37:47.184 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018271632999999999, 2020-06-12T02:37:47.184 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:47.185 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:47.185 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:47.185 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:47.185 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:47.185 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 962 2020-06-12T02:37:47.186 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.186 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:47.186 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.186 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:47.186 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.236108+0000", 2020-06-12T02:37:47.187 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:47.187 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.187 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.187 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:47.187 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.236108+0000", 2020-06-12T02:37:47.188 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019838700000000001 2020-06-12T02:37:47.188 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.188 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.188 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:47.188 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.236307+0000", 2020-06-12T02:37:47.188 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4619999999999996e-06 2020-06-12T02:37:47.189 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.189 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.189 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:47.189 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.236315+0000", 2020-06-12T02:37:47.189 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1665999999999999e-05 2020-06-12T02:37:47.190 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.190 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.190 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:47.190 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.236337+0000", 2020-06-12T02:37:47.191 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5630000000000003e-06 2020-06-12T02:37:47.192 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.192 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.192 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:47.192 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.236344+0000", 2020-06-12T02:37:47.192 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023269440000000001 2020-06-12T02:37:47.193 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.193 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.193 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:47.193 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.238671+0000", 2020-06-12T02:37:47.193 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5969000000000003e-05 2020-06-12T02:37:47.194 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.194 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.194 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:47.194 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.238757+0000", 2020-06-12T02:37:47.194 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046101010000000001 2020-06-12T02:37:47.194 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.195 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.195 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:47.195 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.243367+0000", 2020-06-12T02:37:47.195 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7991000000000005e-05 2020-06-12T02:37:47.195 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.196 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.196 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:47.196 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.243455+0000", 2020-06-12T02:37:47.196 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00062561500000000005 2020-06-12T02:37:47.196 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.197 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.197 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:47.197 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.244081+0000", 2020-06-12T02:37:47.197 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010040778 2020-06-12T02:37:47.197 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.197 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.198 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:47.198 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.254122+0000", 2020-06-12T02:37:47.198 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010453100000000001 2020-06-12T02:37:47.198 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.198 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.199 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:47.199 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.254226+0000", 2020-06-12T02:37:47.199 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000153626 2020-06-12T02:37:47.199 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.199 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.200 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:47.200 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.254380+0000", 2020-06-12T02:37:47.200 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018271632999999999 2020-06-12T02:37:47.200 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:47.201 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:47.201 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:47.201 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.201 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.201 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1252 3.8s0 3:14244450:::smithi10729407-2:head [append 0~516096] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:47.201 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:32.999793+0000", 2020-06-12T02:37:47.202 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 14.179649123000001, 2020-06-12T02:37:47.202 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017380335, 2020-06-12T02:37:47.202 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:47.202 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:47.202 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:47.203 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:47.203 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:47.203 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1252 2020-06-12T02:37:47.203 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.203 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:47.203 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.204 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:47.204 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.999793+0000", 2020-06-12T02:37:47.204 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:47.204 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.204 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.205 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:47.205 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.999793+0000", 2020-06-12T02:37:47.205 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000208051 2020-06-12T02:37:47.206 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.206 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.206 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:47.206 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.000001+0000", 2020-06-12T02:37:47.207 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9750000000000002e-06 2020-06-12T02:37:47.207 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.207 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.207 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:47.207 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.000011+0000", 2020-06-12T02:37:47.207 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7733000000000001e-05 2020-06-12T02:37:47.208 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.208 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.208 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:47.208 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.000028+0000", 2020-06-12T02:37:47.208 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7379999999999994e-06 2020-06-12T02:37:47.209 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.209 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.209 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:47.209 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.000036+0000", 2020-06-12T02:37:47.209 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.414e-05 2020-06-12T02:37:47.209 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.210 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.210 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:47.210 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.000090+0000", 2020-06-12T02:37:47.210 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012561699999999999 2020-06-12T02:37:47.210 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.211 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.211 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:47.211 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.000216+0000", 2020-06-12T02:37:47.211 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00079436199999999995 2020-06-12T02:37:47.211 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.211 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.212 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:47.212 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.001010+0000", 2020-06-12T02:37:47.212 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015996506000000001 2020-06-12T02:37:47.212 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.212 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.213 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:47.213 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.017007+0000", 2020-06-12T02:37:47.213 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0599999999999997e-05 2020-06-12T02:37:47.213 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.213 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.213 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:47.214 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.017057+0000", 2020-06-12T02:37:47.214 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011561300000000001 2020-06-12T02:37:47.214 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.214 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.214 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:47.215 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.017173+0000", 2020-06-12T02:37:47.215 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017380335 2020-06-12T02:37:47.215 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:47.215 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:47.216 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:47.216 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.216 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.216 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1299 3.bs0 3:d0ebae7d:::smithi10729407-3:head [append 0~770048] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:47.216 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.162816+0000", 2020-06-12T02:37:47.217 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 14.016625984999999, 2020-06-12T02:37:47.217 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016787209000000001, 2020-06-12T02:37:47.217 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:47.217 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:47.217 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:47.217 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:47.218 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:47.218 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1299 2020-06-12T02:37:47.218 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.218 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:47.218 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.219 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:47.219 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.162816+0000", 2020-06-12T02:37:47.219 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:47.219 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.219 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.219 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:47.220 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.162816+0000", 2020-06-12T02:37:47.220 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044992799999999999 2020-06-12T02:37:47.220 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.220 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.220 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:47.221 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.163266+0000", 2020-06-12T02:37:47.221 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6959999999999999e-06 2020-06-12T02:37:47.221 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.221 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.222 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:47.222 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.163272+0000", 2020-06-12T02:37:47.222 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3028e-05 2020-06-12T02:37:47.222 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.222 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.223 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:47.223 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.163286+0000", 2020-06-12T02:37:47.223 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2829999999999998e-06 2020-06-12T02:37:47.223 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.223 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.223 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:47.224 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.163292+0000", 2020-06-12T02:37:47.224 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1761000000000002e-05 2020-06-12T02:37:47.224 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.224 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.224 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:47.225 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.163334+0000", 2020-06-12T02:37:47.225 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6691000000000001e-05 2020-06-12T02:37:47.225 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.227 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.227 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:47.228 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.163410+0000", 2020-06-12T02:37:47.228 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004070176 2020-06-12T02:37:47.228 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.228 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.229 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:47.229 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.167480+0000", 2020-06-12T02:37:47.230 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7137000000000003e-05 2020-06-12T02:37:47.230 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.230 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.231 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:47.231 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.167568+0000", 2020-06-12T02:37:47.231 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000789648 2020-06-12T02:37:47.232 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.232 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.232 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:47.232 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168357+0000", 2020-06-12T02:37:47.233 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011078226 2020-06-12T02:37:47.233 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.233 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.234 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:47.234 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.179435+0000", 2020-06-12T02:37:47.234 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6649000000000003e-05 2020-06-12T02:37:47.235 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.235 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.235 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:47.236 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.179522+0000", 2020-06-12T02:37:47.236 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0986e-05 2020-06-12T02:37:47.237 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.237 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.237 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:47.238 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.179603+0000", 2020-06-12T02:37:47.238 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016787209000000001 2020-06-12T02:37:47.238 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:47.239 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:47.239 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:47.239 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.240 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.240 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1300 3.bs0 3:d0ebae7d:::smithi10729407-3:head [append 0~557056] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:47.240 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.166253+0000", 2020-06-12T02:37:47.241 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 14.013188477, 2020-06-12T02:37:47.241 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019037724999999998, 2020-06-12T02:37:47.242 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:47.242 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:47.242 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:47.242 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:47.242 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:47.243 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1300 2020-06-12T02:37:47.243 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.243 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:47.243 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.243 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:47.244 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.166253+0000", 2020-06-12T02:37:47.244 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:47.244 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.244 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.244 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:47.244 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.166253+0000", 2020-06-12T02:37:47.245 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034533600000000002 2020-06-12T02:37:47.245 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.245 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.245 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:47.245 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.166599+0000", 2020-06-12T02:37:47.246 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7699999999999996e-06 2020-06-12T02:37:47.246 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.246 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.246 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:47.247 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.166608+0000", 2020-06-12T02:37:47.247 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7314999999999999e-05 2020-06-12T02:37:47.247 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.247 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.247 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:47.248 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.166626+0000", 2020-06-12T02:37:47.248 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9260000000000003e-06 2020-06-12T02:37:47.248 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.248 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.248 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:47.248 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.166633+0000", 2020-06-12T02:37:47.249 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0266000000000002e-05 2020-06-12T02:37:47.249 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.249 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.249 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:47.249 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.166693+0000", 2020-06-12T02:37:47.250 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6692000000000005e-05 2020-06-12T02:37:47.250 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.250 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.250 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:47.250 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.166760+0000", 2020-06-12T02:37:47.251 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0057626860000000004 2020-06-12T02:37:47.251 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.251 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.251 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:47.251 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.172522+0000", 2020-06-12T02:37:47.251 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000130493 2020-06-12T02:37:47.252 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.252 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.252 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:47.252 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.172653+0000", 2020-06-12T02:37:47.252 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00083376299999999995 2020-06-12T02:37:47.253 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.253 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.253 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:47.253 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.173487+0000", 2020-06-12T02:37:47.253 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0061197739999999997 2020-06-12T02:37:47.253 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.254 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.254 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:47.254 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.179606+0000", 2020-06-12T02:37:47.254 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0056160699999999999 2020-06-12T02:37:47.254 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.255 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.255 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:47.255 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.185222+0000", 2020-06-12T02:37:47.255 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8633999999999994e-05 2020-06-12T02:37:47.255 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.255 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.256 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:47.256 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.185291+0000", 2020-06-12T02:37:47.256 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019037724999999998 2020-06-12T02:37:47.256 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:47.256 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:47.257 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:47.257 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.257 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.257 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1301 3.bs0 3:d0ebae7d:::smithi10729407-3:head [append 0~278528] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:47.258 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.168146+0000", 2020-06-12T02:37:47.258 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 14.01129592, 2020-06-12T02:37:47.258 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017691083999999999, 2020-06-12T02:37:47.258 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:47.259 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:47.259 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:47.259 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:47.259 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:47.259 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1301 2020-06-12T02:37:47.260 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.260 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:47.260 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.260 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:47.260 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168146+0000", 2020-06-12T02:37:47.261 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:47.261 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.261 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.261 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:47.261 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168146+0000", 2020-06-12T02:37:47.262 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000213666 2020-06-12T02:37:47.262 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.262 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.262 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:47.262 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168360+0000", 2020-06-12T02:37:47.262 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7009999999999996e-06 2020-06-12T02:37:47.263 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.263 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.263 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:47.263 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168367+0000", 2020-06-12T02:37:47.263 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5096000000000001e-05 2020-06-12T02:37:47.264 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.264 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.264 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:47.264 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168382+0000", 2020-06-12T02:37:47.264 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8739999999999997e-06 2020-06-12T02:37:47.264 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.265 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.265 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:47.265 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168389+0000", 2020-06-12T02:37:47.265 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0076120199999999997 2020-06-12T02:37:47.265 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.266 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.266 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:47.266 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.176001+0000", 2020-06-12T02:37:47.266 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9181000000000001e-05 2020-06-12T02:37:47.267 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.267 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.267 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:47.267 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.176060+0000", 2020-06-12T02:37:47.268 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00039587599999999998 2020-06-12T02:37:47.268 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.268 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.268 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:47.268 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.176456+0000", 2020-06-12T02:37:47.268 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0031522329999999999 2020-06-12T02:37:47.269 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.269 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.269 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:47.269 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.179609+0000", 2020-06-12T02:37:47.269 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0061683670000000001 2020-06-12T02:37:47.270 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.270 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.270 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:47.270 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.185777+0000", 2020-06-12T02:37:47.270 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0069999999999999e-05 2020-06-12T02:37:47.271 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.271 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.271 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:47.271 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.185837+0000", 2020-06-12T02:37:47.271 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017691083999999999 2020-06-12T02:37:47.271 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:47.272 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:47.272 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:47.272 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.272 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.272 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1302 3.bs0 3:d0ebae7d:::smithi10729407-3:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:47.273 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.168445+0000", 2020-06-12T02:37:47.273 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 14.01099664, 2020-06-12T02:37:47.273 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017581657000000001, 2020-06-12T02:37:47.273 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:47.273 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:47.274 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:47.274 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:47.274 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:47.274 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1302 2020-06-12T02:37:47.274 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.275 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:47.275 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.275 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:47.275 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168445+0000", 2020-06-12T02:37:47.275 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:47.275 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.276 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.276 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:47.276 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168445+0000", 2020-06-12T02:37:47.276 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3532e-05 2020-06-12T02:37:47.276 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.277 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.277 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:47.277 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168519+0000", 2020-06-12T02:37:47.277 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2600000000000002e-06 2020-06-12T02:37:47.278 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.278 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.278 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:47.278 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168525+0000", 2020-06-12T02:37:47.278 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4338999999999999e-05 2020-06-12T02:37:47.279 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.279 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.279 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:47.279 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168539+0000", 2020-06-12T02:37:47.279 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7330000000000003e-06 2020-06-12T02:37:47.280 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.280 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.280 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:47.280 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168544+0000", 2020-06-12T02:37:47.280 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0095046090000000007 2020-06-12T02:37:47.280 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.281 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.281 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:47.281 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.178049+0000", 2020-06-12T02:37:47.281 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1524000000000003e-05 2020-06-12T02:37:47.281 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.282 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.282 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:47.282 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.178110+0000", 2020-06-12T02:37:47.282 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030514799999999999 2020-06-12T02:37:47.282 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.283 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.283 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:47.283 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.178415+0000", 2020-06-12T02:37:47.284 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011944480000000001 2020-06-12T02:37:47.284 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.285 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.285 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:47.285 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.179610+0000", 2020-06-12T02:37:47.286 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0063372849999999998 2020-06-12T02:37:47.286 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.286 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.286 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:47.286 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.185947+0000", 2020-06-12T02:37:47.287 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9778999999999996e-05 2020-06-12T02:37:47.287 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.287 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.287 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:47.287 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.186027+0000", 2020-06-12T02:37:47.287 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017581657000000001 2020-06-12T02:37:47.288 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:47.288 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:47.288 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:47.288 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.288 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.289 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1303 3.bs0 3:d0ebae7d:::smithi10729407-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:47.289 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.168587+0000", 2020-06-12T02:37:47.289 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 14.010854721999999, 2020-06-12T02:37:47.289 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019127155, 2020-06-12T02:37:47.289 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:47.290 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:37:47.290 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:47.290 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:47.290 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:47.291 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1303 2020-06-12T02:37:47.291 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.291 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:47.291 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.291 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:47.292 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168587+0000", 2020-06-12T02:37:47.292 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:47.292 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.292 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.292 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:47.292 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168587+0000", 2020-06-12T02:37:47.293 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3099999999999997e-05 2020-06-12T02:37:47.293 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.293 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.294 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:47.294 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168630+0000", 2020-06-12T02:37:47.294 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1600000000000003e-06 2020-06-12T02:37:47.294 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.294 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.294 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:47.295 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168636+0000", 2020-06-12T02:37:47.295 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3439e-05 2020-06-12T02:37:47.295 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.295 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.295 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:47.296 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168650+0000", 2020-06-12T02:37:47.296 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7850000000000002e-06 2020-06-12T02:37:47.296 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.296 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.296 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:47.296 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168654+0000", 2020-06-12T02:37:47.297 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010326317 2020-06-12T02:37:47.297 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.297 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.297 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:47.297 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.178980+0000", 2020-06-12T02:37:47.298 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6593000000000004e-05 2020-06-12T02:37:47.298 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.298 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.298 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:47.298 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.179047+0000", 2020-06-12T02:37:47.298 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0073410990000000002 2020-06-12T02:37:47.299 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.299 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.299 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:47.299 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.186388+0000", 2020-06-12T02:37:47.299 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9405000000000001e-05 2020-06-12T02:37:47.300 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.300 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.300 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:47.300 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.186487+0000", 2020-06-12T02:37:47.300 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001227257 2020-06-12T02:37:47.301 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.301 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.301 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:47.301 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.187714+0000", 2020-06-12T02:37:47.303 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019127155 2020-06-12T02:37:47.303 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:47.303 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:47.303 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:47.303 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.304 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.304 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1527 3.2s0 3:4afffc6f:::smithi10729407-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e26)", 2020-06-12T02:37:47.304 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.750793+0000", 2020-06-12T02:37:47.304 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 13.428648568, 2020-06-12T02:37:47.304 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016832799999999998, 2020-06-12T02:37:47.305 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:47.305 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:37:47.305 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:47.305 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:47.305 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:47.306 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1527 2020-06-12T02:37:47.306 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.306 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:47.306 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.306 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:47.306 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750793+0000", 2020-06-12T02:37:47.307 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:47.307 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.307 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.307 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:47.307 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750793+0000", 2020-06-12T02:37:47.308 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3218e-05 2020-06-12T02:37:47.308 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.308 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.308 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:47.309 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750817+0000", 2020-06-12T02:37:47.309 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0939999999999999e-06 2020-06-12T02:37:47.309 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.309 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.309 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:47.316 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750820+0000", 2020-06-12T02:37:47.317 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4810000000000003e-06 2020-06-12T02:37:47.317 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.317 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.318 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:47.318 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750826+0000", 2020-06-12T02:37:47.318 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.277e-06 2020-06-12T02:37:47.318 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.319 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.319 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:47.319 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750828+0000", 2020-06-12T02:37:47.319 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7738000000000005e-05 2020-06-12T02:37:47.320 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.320 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.320 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:47.320 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750926+0000", 2020-06-12T02:37:47.320 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6310000000000005e-05 2020-06-12T02:37:47.321 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.321 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.321 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:47.321 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.751012+0000", 2020-06-12T02:37:47.321 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010605622 2020-06-12T02:37:47.322 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.322 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.322 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:47.322 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.761618+0000", 2020-06-12T02:37:47.322 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2778000000000002e-05 2020-06-12T02:37:47.322 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.323 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.323 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:47.323 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.761671+0000", 2020-06-12T02:37:47.323 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.005069044 2020-06-12T02:37:47.323 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.324 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.324 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:47.324 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.766740+0000", 2020-06-12T02:37:47.324 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6335999999999997e-05 2020-06-12T02:37:47.325 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.325 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.325 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:47.325 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.766796+0000", 2020-06-12T02:37:47.325 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00082990199999999998 2020-06-12T02:37:47.326 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.326 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.326 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:47.327 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.767626+0000", 2020-06-12T02:37:47.327 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016832799999999998 2020-06-12T02:37:47.328 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:47.328 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:47.328 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:47.329 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.329 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.330 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s1 26/21 ECSubWrite(tid=408, reqid=client.4512.0:1572, at_version=26'99, trim_to=0'0, roll_forward_to=26'97))", 2020-06-12T02:37:47.330 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.887944+0000", 2020-06-12T02:37:47.330 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 13.291498088999999, 2020-06-12T02:37:47.331 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019136641999999999, 2020-06-12T02:37:47.331 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:47.331 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:47.332 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:47.332 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.332 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:47.333 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.887944+0000", 2020-06-12T02:37:47.333 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:47.333 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.334 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.334 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:47.334 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.887944+0000", 2020-06-12T02:37:47.334 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00063708500000000002 2020-06-12T02:37:47.335 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.335 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.335 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:47.336 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888581+0000", 2020-06-12T02:37:47.336 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9644e-05 2020-06-12T02:37:47.336 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.337 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.337 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:47.337 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888600+0000", 2020-06-12T02:37:47.338 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.721e-05 2020-06-12T02:37:47.338 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.338 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.339 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:47.339 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888618+0000", 2020-06-12T02:37:47.339 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7500000000000003e-06 2020-06-12T02:37:47.340 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.340 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.340 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:47.341 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888625+0000", 2020-06-12T02:37:47.341 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2707999999999998e-05 2020-06-12T02:37:47.341 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.341 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.342 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:47.342 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888668+0000", 2020-06-12T02:37:47.342 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0491e-05 2020-06-12T02:37:47.343 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.343 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.343 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:47.344 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888679+0000", 2020-06-12T02:37:47.344 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018351985000000001 2020-06-12T02:37:47.344 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.344 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.345 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:47.345 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.907031+0000", 2020-06-12T02:37:47.345 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9768999999999997e-05 2020-06-12T02:37:47.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.345 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.346 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:47.346 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.907080+0000", 2020-06-12T02:37:47.346 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019136641999999999 2020-06-12T02:37:47.346 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:47.346 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:47.346 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:47.347 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.347 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.347 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s1 26/21 ECSubWrite(tid=409, reqid=client.4512.0:1573, at_version=26'100, trim_to=0'0, roll_forward_to=26'98))", 2020-06-12T02:37:47.347 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.892432+0000", 2020-06-12T02:37:47.347 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 13.287009499, 2020-06-12T02:37:47.348 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01685733, 2020-06-12T02:37:47.348 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:47.348 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:47.348 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:47.348 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.349 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:47.349 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.892432+0000", 2020-06-12T02:37:47.349 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:47.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.349 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.349 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:47.350 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.892432+0000", 2020-06-12T02:37:47.350 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059245900000000004 2020-06-12T02:37:47.350 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.350 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.350 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:47.351 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.893025+0000", 2020-06-12T02:37:47.351 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1591e-05 2020-06-12T02:37:47.351 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.351 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.351 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:47.351 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.893046+0000", 2020-06-12T02:37:47.352 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2429000000000002e-05 2020-06-12T02:37:47.352 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.352 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.352 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:47.352 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.893069+0000", 2020-06-12T02:37:47.353 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0389999999999993e-06 2020-06-12T02:37:47.353 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.353 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.353 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:47.353 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.893078+0000", 2020-06-12T02:37:47.354 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012883099999999999 2020-06-12T02:37:47.354 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.354 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.354 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:47.354 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.893207+0000", 2020-06-12T02:37:47.354 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2407e-05 2020-06-12T02:37:47.355 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.355 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.355 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:47.355 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.893219+0000", 2020-06-12T02:37:47.356 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016029015000000001 2020-06-12T02:37:47.356 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.356 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.356 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:47.356 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.909248+0000", 2020-06-12T02:37:47.357 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1559e-05 2020-06-12T02:37:47.357 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.357 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.357 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:47.357 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.909290+0000", 2020-06-12T02:37:47.357 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01685733 2020-06-12T02:37:47.358 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:47.358 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:47.358 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:47.358 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.359 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.359 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:3329 3.bs0 3:d09543d2:::smithi10729407-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:37:47.359 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:38.600071+0000", 2020-06-12T02:37:47.359 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 8.5793704460000004, 2020-06-12T02:37:47.359 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016512401999999999, 2020-06-12T02:37:47.359 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:47.360 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:37:47.360 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:47.360 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:47.360 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:47.360 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 3329 2020-06-12T02:37:47.361 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.361 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:47.361 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.361 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:47.361 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.600071+0000", 2020-06-12T02:37:47.362 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:47.362 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.362 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.362 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:47.362 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.600071+0000", 2020-06-12T02:37:47.363 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5961999999999998e-05 2020-06-12T02:37:47.363 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.363 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.363 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:47.363 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.600117+0000", 2020-06-12T02:37:47.364 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6890000000000001e-06 2020-06-12T02:37:47.364 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.364 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.364 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:47.364 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.600123+0000", 2020-06-12T02:37:47.364 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2027e-05 2020-06-12T02:37:47.365 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.365 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.365 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:47.365 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.600135+0000", 2020-06-12T02:37:47.365 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.9840000000000003e-06 2020-06-12T02:37:47.366 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.366 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.366 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:47.366 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.600139+0000", 2020-06-12T02:37:47.366 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0073161570000000002 2020-06-12T02:37:47.366 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.367 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.367 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:47.367 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.607455+0000", 2020-06-12T02:37:47.367 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8794000000000004e-05 2020-06-12T02:37:47.367 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.368 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.368 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:47.368 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.607544+0000", 2020-06-12T02:37:47.368 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0068056540000000004 2020-06-12T02:37:47.369 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.369 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.369 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:47.369 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.614350+0000", 2020-06-12T02:37:47.369 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6471999999999994e-05 2020-06-12T02:37:47.369 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.370 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.370 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:47.370 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.614446+0000", 2020-06-12T02:37:47.370 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0021376630000000001 2020-06-12T02:37:47.370 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.371 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.371 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:47.371 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.616584+0000", 2020-06-12T02:37:47.371 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016512401999999999 2020-06-12T02:37:47.371 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:47.371 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:47.372 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:47.372 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.372 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.372 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:3607 3.7s0 3:e653d00f:::smithi10729407-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:37:47.372 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:39.292176+0000", 2020-06-12T02:37:47.373 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 7.8872660090000002, 2020-06-12T02:37:47.373 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016904665999999999, 2020-06-12T02:37:47.373 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:47.373 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:37:47.373 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:47.373 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:47.374 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:47.374 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 3607 2020-06-12T02:37:47.374 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.374 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:47.374 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.375 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:47.375 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.292176+0000", 2020-06-12T02:37:47.375 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:47.375 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.375 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.375 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:47.376 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.292176+0000", 2020-06-12T02:37:47.376 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4786000000000001e-05 2020-06-12T02:37:47.376 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.376 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.376 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:47.377 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.292221+0000", 2020-06-12T02:37:47.377 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0159999999999997e-06 2020-06-12T02:37:47.377 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.377 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.377 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:47.378 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.292227+0000", 2020-06-12T02:37:47.378 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2269999999999999e-05 2020-06-12T02:37:47.378 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.378 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.378 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:47.378 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.292239+0000", 2020-06-12T02:37:47.379 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1849999999999997e-06 2020-06-12T02:37:47.379 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.379 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.379 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:47.379 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.292243+0000", 2020-06-12T02:37:47.379 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0077395820000000001 2020-06-12T02:37:47.380 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.380 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.380 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:47.380 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.299983+0000", 2020-06-12T02:37:47.380 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7809999999999997e-05 2020-06-12T02:37:47.381 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.381 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.381 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:47.381 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.300040+0000", 2020-06-12T02:37:47.381 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0049104680000000003 2020-06-12T02:37:47.382 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.382 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.382 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:47.382 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.304951+0000", 2020-06-12T02:37:47.382 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2177999999999997e-05 2020-06-12T02:37:47.383 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.383 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.383 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:47.383 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.305043+0000", 2020-06-12T02:37:47.383 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0040373709999999997 2020-06-12T02:37:47.383 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.384 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.384 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:47.384 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.309081+0000", 2020-06-12T02:37:47.384 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016904665999999999 2020-06-12T02:37:47.384 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:47.384 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:47.385 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:47.385 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.385 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.385 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:4483 3.7s0 3:e653d00f:::smithi10729407-18:head [append 0~417792] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-06-12T02:37:47.385 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:42.533832+0000", 2020-06-12T02:37:47.386 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 4.645610317, 2020-06-12T02:37:47.386 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018187756999999999, 2020-06-12T02:37:47.386 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:47.386 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:47.386 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:47.387 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:47.387 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:47.387 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 4483 2020-06-12T02:37:47.387 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.387 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:47.387 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.388 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:47.388 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.533832+0000", 2020-06-12T02:37:47.388 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:47.388 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.388 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.388 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:47.389 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.533832+0000", 2020-06-12T02:37:47.389 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016872100000000001 2020-06-12T02:37:47.389 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.389 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.389 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:47.390 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.534000+0000", 2020-06-12T02:37:47.390 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3569999999999998e-06 2020-06-12T02:37:47.390 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.390 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.390 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:47.390 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.534009+0000", 2020-06-12T02:37:47.391 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6013000000000001e-05 2020-06-12T02:37:47.391 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.391 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.391 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:47.391 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.534025+0000", 2020-06-12T02:37:47.392 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.841e-06 2020-06-12T02:37:47.393 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.393 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.393 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:47.393 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.534032+0000", 2020-06-12T02:37:47.393 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014511190000000001 2020-06-12T02:37:47.393 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.394 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.394 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:47.394 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.535484+0000", 2020-06-12T02:37:47.394 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013402299999999999 2020-06-12T02:37:47.394 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.395 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.395 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:47.395 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.535618+0000", 2020-06-12T02:37:47.395 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0026343590000000002 2020-06-12T02:37:47.395 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.395 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.396 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:47.396 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.538252+0000", 2020-06-12T02:37:47.396 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2380000000000007e-05 2020-06-12T02:37:47.396 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.396 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.397 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:47.397 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.538344+0000", 2020-06-12T02:37:47.397 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00068647700000000001 2020-06-12T02:37:47.397 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.397 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.398 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:47.398 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.539031+0000", 2020-06-12T02:37:47.398 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011685509 2020-06-12T02:37:47.398 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.398 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.398 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:47.399 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.550716+0000", 2020-06-12T02:37:47.399 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011673790000000001 2020-06-12T02:37:47.399 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.399 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.399 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:47.400 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.551884+0000", 2020-06-12T02:37:47.400 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013557900000000001 2020-06-12T02:37:47.400 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.400 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.400 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:47.400 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.552019+0000", 2020-06-12T02:37:47.401 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018187756999999999 2020-06-12T02:37:47.401 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:47.401 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:47.401 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:47.401 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.402 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.402 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:4485 3.7s0 3:e653d00f:::smithi10729407-18:head [append 0~180224] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-06-12T02:37:47.402 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:42.538857+0000", 2020-06-12T02:37:47.402 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 4.6405848719999998, 2020-06-12T02:37:47.402 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017291731000000001, 2020-06-12T02:37:47.402 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:47.403 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:47.403 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:47.403 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:47.403 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:47.403 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 4485 2020-06-12T02:37:47.405 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.405 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:47.405 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.405 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:47.405 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.538857+0000", 2020-06-12T02:37:47.406 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:47.406 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.406 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.406 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:47.406 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.538857+0000", 2020-06-12T02:37:47.407 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000170361 2020-06-12T02:37:47.407 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.407 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.407 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:47.407 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.539027+0000", 2020-06-12T02:37:47.407 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1885000000000001e-05 2020-06-12T02:37:47.408 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.408 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.408 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:47.408 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.539039+0000", 2020-06-12T02:37:47.408 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3631e-05 2020-06-12T02:37:47.409 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.409 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.409 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:47.409 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.539053+0000", 2020-06-12T02:37:47.409 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8189999999999997e-06 2020-06-12T02:37:47.410 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.410 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.410 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:47.410 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.539059+0000", 2020-06-12T02:37:47.410 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093035110000000004 2020-06-12T02:37:47.410 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.411 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.411 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:47.411 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.548362+0000", 2020-06-12T02:37:47.411 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114157 2020-06-12T02:37:47.411 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.412 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.412 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:47.412 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.548476+0000", 2020-06-12T02:37:47.412 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056751900000000005 2020-06-12T02:37:47.413 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.413 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.413 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:47.413 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.549044+0000", 2020-06-12T02:37:47.413 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0042106590000000003 2020-06-12T02:37:47.414 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.414 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.414 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:47.414 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.553255+0000", 2020-06-12T02:37:47.414 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027535060000000002 2020-06-12T02:37:47.414 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.415 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.415 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:47.415 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.556008+0000", 2020-06-12T02:37:47.415 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014068300000000001 2020-06-12T02:37:47.415 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.416 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.416 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:47.416 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.556149+0000", 2020-06-12T02:37:47.416 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017291731000000001 2020-06-12T02:37:47.416 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:47.417 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:47.417 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:47.417 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.417 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.417 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:4486 3.7s0 3:e653d00f:::smithi10729407-18:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-06-12T02:37:47.418 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:42.539122+0000", 2020-06-12T02:37:47.418 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 4.6403198889999997, 2020-06-12T02:37:47.418 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017248556000000002, 2020-06-12T02:37:47.418 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:47.418 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:47.419 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:47.419 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:47.419 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:47.419 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 4486 2020-06-12T02:37:47.419 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.419 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:47.420 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.420 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:47.420 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.539122+0000", 2020-06-12T02:37:47.420 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:47.420 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.421 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.421 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:47.421 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.539122+0000", 2020-06-12T02:37:47.421 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7909000000000003e-05 2020-06-12T02:37:47.421 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.421 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.422 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:47.422 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.539190+0000", 2020-06-12T02:37:47.422 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5819999999999998e-06 2020-06-12T02:37:47.422 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.423 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.423 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:47.423 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.539196+0000", 2020-06-12T02:37:47.423 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2004999999999999e-05 2020-06-12T02:37:47.423 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.424 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.424 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:47.424 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.539208+0000", 2020-06-12T02:37:47.424 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8359999999999999e-06 2020-06-12T02:37:47.424 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.425 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.425 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:47.425 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.539213+0000", 2020-06-12T02:37:47.425 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011535996 2020-06-12T02:37:47.425 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.425 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.426 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:47.426 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.550749+0000", 2020-06-12T02:37:47.426 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3753999999999997e-05 2020-06-12T02:37:47.426 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.426 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.427 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:47.428 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.550843+0000", 2020-06-12T02:37:47.428 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00062439500000000003 2020-06-12T02:37:47.428 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.428 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.428 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:47.429 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.551467+0000", 2020-06-12T02:37:47.429 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0020300309999999998 2020-06-12T02:37:47.429 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.429 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.429 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:47.430 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.553497+0000", 2020-06-12T02:37:47.430 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027478680000000001 2020-06-12T02:37:47.430 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.430 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.430 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:47.431 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.556245+0000", 2020-06-12T02:37:47.431 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012517999999999999 2020-06-12T02:37:47.431 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.431 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.431 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:47.432 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.556371+0000", 2020-06-12T02:37:47.432 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017248556000000002 2020-06-12T02:37:47.432 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:47.432 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:47.432 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:47.432 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.433 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.433 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:4487 3.7s0 3:e653d00f:::smithi10729407-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-06-12T02:37:47.433 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:42.539305+0000", 2020-06-12T02:37:47.433 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 4.6401368420000004, 2020-06-12T02:37:47.433 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019352590999999999, 2020-06-12T02:37:47.434 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:47.434 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:37:47.434 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:47.434 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:47.434 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:47.435 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 4487 2020-06-12T02:37:47.435 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.435 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:47.435 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.435 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:47.436 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.539305+0000", 2020-06-12T02:37:47.436 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:47.436 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.436 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.436 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:47.437 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.539305+0000", 2020-06-12T02:37:47.437 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7207999999999999e-05 2020-06-12T02:37:47.437 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.437 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.438 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:47.438 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.539352+0000", 2020-06-12T02:37:47.438 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6440000000000003e-06 2020-06-12T02:37:47.438 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.438 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.439 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:47.439 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.539359+0000", 2020-06-12T02:37:47.439 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0995999999999999e-05 2020-06-12T02:37:47.439 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.439 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.440 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:47.440 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.539370+0000", 2020-06-12T02:37:47.440 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4049999999999999e-06 2020-06-12T02:37:47.440 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.440 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.441 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:47.441 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.539375+0000", 2020-06-12T02:37:47.441 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013353090999999999 2020-06-12T02:37:47.441 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.441 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.442 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:47.442 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.552728+0000", 2020-06-12T02:37:47.442 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000113626 2020-06-12T02:37:47.442 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.442 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.443 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:47.443 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.552842+0000", 2020-06-12T02:37:47.443 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0039644459999999999 2020-06-12T02:37:47.443 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.443 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.443 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:47.444 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.556806+0000", 2020-06-12T02:37:47.444 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5099000000000005e-05 2020-06-12T02:37:47.444 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.444 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.444 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:47.444 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.556902+0000", 2020-06-12T02:37:47.445 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017560760000000001 2020-06-12T02:37:47.445 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.445 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.445 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:47.445 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.558658+0000", 2020-06-12T02:37:47.446 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019352590999999999 2020-06-12T02:37:47.446 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:47.446 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:47.446 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:47.446 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.446 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.447 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:4493 3.5s1 3:a446366d:::smithi10729407-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-06-12T02:37:47.447 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:42.552291+0000", 2020-06-12T02:37:47.447 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 4.6271512870000002, 2020-06-12T02:37:47.447 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016471244999999999, 2020-06-12T02:37:47.447 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:47.448 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:37:47.448 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:47.448 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:47.448 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:47.448 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 4493 2020-06-12T02:37:47.449 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.449 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:47.449 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.449 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:47.449 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.552291+0000", 2020-06-12T02:37:47.449 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:47.450 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.450 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.450 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:47.450 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.552291+0000", 2020-06-12T02:37:47.450 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8744e-05 2020-06-12T02:37:47.451 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.451 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.451 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:47.451 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.552339+0000", 2020-06-12T02:37:47.451 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1639999999999998e-06 2020-06-12T02:37:47.451 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.452 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.452 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:47.452 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.552345+0000", 2020-06-12T02:37:47.452 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2143999999999999e-05 2020-06-12T02:37:47.452 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.452 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.453 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:47.453 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.552358+0000", 2020-06-12T02:37:47.453 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1060000000000002e-06 2020-06-12T02:37:47.453 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.453 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.454 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:47.454 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.552362+0000", 2020-06-12T02:37:47.454 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0099490629999999993 2020-06-12T02:37:47.454 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.454 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.455 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:47.455 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.562311+0000", 2020-06-12T02:37:47.455 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7677999999999998e-05 2020-06-12T02:37:47.455 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.455 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.455 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:47.456 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.562378+0000", 2020-06-12T02:37:47.456 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043169699999999998 2020-06-12T02:37:47.456 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.456 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.457 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:47.457 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.566695+0000", 2020-06-12T02:37:47.457 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3007000000000003e-05 2020-06-12T02:37:47.457 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.457 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.458 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:47.458 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.566748+0000", 2020-06-12T02:37:47.458 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0020133690000000001 2020-06-12T02:37:47.458 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.458 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.459 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:47.459 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.568762+0000", 2020-06-12T02:37:47.459 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016471244999999999 2020-06-12T02:37:47.459 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:47.460 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:47.460 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:47.460 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.460 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.460 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:5072 3.5s1 3:a9510bda:::smithi10729407-14:head [append 0~770048] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-06-12T02:37:47.461 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:44.163594+0000", 2020-06-12T02:37:47.461 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.015847618, 2020-06-12T02:37:47.461 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018215139000000002, 2020-06-12T02:37:47.461 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:47.461 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:47.462 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:47.462 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:47.462 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:47.462 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 5072 2020-06-12T02:37:47.462 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.463 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:47.463 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.463 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:47.463 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.163594+0000", 2020-06-12T02:37:47.463 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:47.463 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.464 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.464 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:47.464 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.163594+0000", 2020-06-12T02:37:47.464 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00046902400000000001 2020-06-12T02:37:47.464 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.465 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.465 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:47.465 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.164063+0000", 2020-06-12T02:37:47.465 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3689999999999996e-06 2020-06-12T02:37:47.465 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.465 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.466 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:47.466 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.164073+0000", 2020-06-12T02:37:47.466 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0157999999999999e-05 2020-06-12T02:37:47.466 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.466 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.467 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:47.467 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.164093+0000", 2020-06-12T02:37:47.467 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6800000000000005e-06 2020-06-12T02:37:47.467 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.467 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.467 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:47.468 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.164102+0000", 2020-06-12T02:37:47.468 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9597999999999993e-05 2020-06-12T02:37:47.468 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.468 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.468 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:47.469 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.164172+0000", 2020-06-12T02:37:47.469 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012974699999999999 2020-06-12T02:37:47.469 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.469 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.469 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:47.469 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.164302+0000", 2020-06-12T02:37:47.470 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00094603199999999997 2020-06-12T02:37:47.470 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.470 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.470 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:47.470 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.165248+0000", 2020-06-12T02:37:47.471 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016405492000000001 2020-06-12T02:37:47.471 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.471 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.471 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:47.471 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.181653+0000", 2020-06-12T02:37:47.472 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8615000000000005e-05 2020-06-12T02:37:47.472 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.472 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.472 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:47.472 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.181742+0000", 2020-06-12T02:37:47.472 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7423999999999997e-05 2020-06-12T02:37:47.473 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.473 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.473 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:47.473 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.181809+0000", 2020-06-12T02:37:47.473 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018215139000000002 2020-06-12T02:37:47.474 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:47.474 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:47.474 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:47.474 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.474 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.474 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:5077 3.5s1 3:a9510bda:::smithi10729407-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-06-12T02:37:47.475 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:44.171464+0000", 2020-06-12T02:37:47.475 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.0079775660000001, 2020-06-12T02:37:47.475 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017114018000000002, 2020-06-12T02:37:47.475 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:47.475 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:37:47.476 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:47.476 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:47.476 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:47.476 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 5077 2020-06-12T02:37:47.476 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.477 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:47.477 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.477 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:47.477 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.171464+0000", 2020-06-12T02:37:47.477 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:47.478 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.479 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.479 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:47.479 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.171464+0000", 2020-06-12T02:37:47.479 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8872000000000002e-05 2020-06-12T02:37:47.479 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.480 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.480 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:47.480 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.171513+0000", 2020-06-12T02:37:47.480 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2840000000000003e-06 2020-06-12T02:37:47.480 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.480 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.481 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:47.481 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.171519+0000", 2020-06-12T02:37:47.481 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3760999999999999e-05 2020-06-12T02:37:47.481 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.481 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.482 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:47.482 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.171533+0000", 2020-06-12T02:37:47.482 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1999999999999996e-06 2020-06-12T02:37:47.482 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.482 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.483 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:47.483 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.171537+0000", 2020-06-12T02:37:47.483 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011095657 2020-06-12T02:37:47.483 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.483 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.483 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:47.484 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.182633+0000", 2020-06-12T02:37:47.484 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3004999999999997e-05 2020-06-12T02:37:47.484 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.484 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.485 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:47.485 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.182696+0000", 2020-06-12T02:37:47.485 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0048429900000000001 2020-06-12T02:37:47.485 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.485 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.486 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:47.486 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.187539+0000", 2020-06-12T02:37:47.486 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7890999999999999e-05 2020-06-12T02:37:47.486 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.486 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.486 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:47.487 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.187597+0000", 2020-06-12T02:37:47.487 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0009813580000000001 2020-06-12T02:37:47.487 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.487 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.487 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:47.488 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.188578+0000", 2020-06-12T02:37:47.488 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017114018000000002 2020-06-12T02:37:47.488 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:47.488 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:47.488 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:47.488 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.489 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.489 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:5093 3.5s1 3:a446366d:::smithi10729407-11:head [append 0~417792] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-06-12T02:37:47.489 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:44.211930+0000", 2020-06-12T02:37:47.489 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 2.967511896, 2020-06-12T02:37:47.489 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017278828999999999, 2020-06-12T02:37:47.490 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:47.490 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:47.490 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:47.490 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:47.490 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:47.490 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 5093 2020-06-12T02:37:47.491 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.491 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:47.491 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.491 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:47.491 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.211930+0000", 2020-06-12T02:37:47.492 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:47.492 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.492 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.492 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:47.492 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.211930+0000", 2020-06-12T02:37:47.492 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000318793 2020-06-12T02:37:47.493 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.493 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.493 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:47.493 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.212249+0000", 2020-06-12T02:37:47.493 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.7708e-05 2020-06-12T02:37:47.494 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.494 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.494 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:47.494 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.212276+0000", 2020-06-12T02:37:47.494 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3105999999999999e-05 2020-06-12T02:37:47.494 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.495 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.495 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:47.495 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.212300+0000", 2020-06-12T02:37:47.495 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2350000000000002e-06 2020-06-12T02:37:47.495 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.496 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.496 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:47.496 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.212307+0000", 2020-06-12T02:37:47.496 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036863500000000002 2020-06-12T02:37:47.496 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.496 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.497 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:47.497 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.212675+0000", 2020-06-12T02:37:47.497 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2799999999999994e-05 2020-06-12T02:37:47.497 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.497 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.498 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:47.498 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.212748+0000", 2020-06-12T02:37:47.498 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027952469999999998 2020-06-12T02:37:47.498 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.498 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.498 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:47.499 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.215543+0000", 2020-06-12T02:37:47.499 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00032576600000000001 2020-06-12T02:37:47.499 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.499 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.499 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:47.500 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.215869+0000", 2020-06-12T02:37:47.500 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00080737599999999995 2020-06-12T02:37:47.500 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.500 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.500 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:47.500 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.216677+0000", 2020-06-12T02:37:47.501 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012350573 2020-06-12T02:37:47.501 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.501 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.501 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:47.501 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.229027+0000", 2020-06-12T02:37:47.502 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4758000000000003e-05 2020-06-12T02:37:47.502 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.503 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.503 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:47.504 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.229122+0000", 2020-06-12T02:37:47.504 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6831999999999998e-05 2020-06-12T02:37:47.504 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:47.504 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:47.504 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:47.505 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.229209+0000", 2020-06-12T02:37:47.505 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017278828999999999 2020-06-12T02:37:47.505 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:47.506 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:47.506 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:47.506 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:47.506 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:47.507 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:37:47.507 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd set noscrub 2020-06-12T02:37:47.533 INFO:tasks.rados.rados.0.smithi107.stdout:2919: writing smithi10729407-3 from 1351680 to 1826816 tid 1 2020-06-12T02:37:47.533 INFO:tasks.rados.rados.0.smithi107.stdout:2919: writing smithi10729407-3 from 1826816 to 2228224 tid 2 2020-06-12T02:37:47.533 INFO:tasks.rados.rados.0.smithi107.stdout:2920: delete oid 8 current snap is 0 2020-06-12T02:37:47.534 INFO:tasks.rados.rados.0.smithi107.stdout:2920: done (1 left) 2020-06-12T02:37:47.534 INFO:tasks.rados.rados.0.smithi107.stdout:2921: delete oid 14 current snap is 0 2020-06-12T02:37:47.534 INFO:tasks.rados.rados.0.smithi107.stdout:2921: done (1 left) 2020-06-12T02:37:47.534 INFO:tasks.rados.rados.0.smithi107.stdout:2922: delete oid 18 current snap is 0 2020-06-12T02:37:47.534 INFO:tasks.rados.rados.0.smithi107.stdout:2919: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:47.535 INFO:tasks.rados.rados.0.smithi107.stdout:2919: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:47.535 INFO:tasks.rados.rados.0.smithi107.stdout:2919: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:47.535 INFO:tasks.rados.rados.0.smithi107.stdout:2922: done (1 left) 2020-06-12T02:37:47.535 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:37:47.535 INFO:tasks.rados.rados.0.smithi107.stdout:2923: seq_num 2922 ranges {761856=540672,1302528=401408} 2020-06-12T02:37:47.536 INFO:tasks.rados.rados.0.smithi107.stdout:2923: writing smithi10729407-13 from 761856 to 1302528 tid 1 2020-06-12T02:37:47.536 INFO:tasks.rados.rados.0.smithi107.stdout:2923: writing smithi10729407-13 from 1302528 to 1703936 tid 2 2020-06-12T02:37:47.536 INFO:tasks.rados.rados.0.smithi107.stdout:2924: delete oid 5 current snap is 0 2020-06-12T02:37:47.536 INFO:tasks.rados.rados.0.smithi107.stdout:2919: finishing write tid 5 to smithi10729407-3 2020-06-12T02:37:47.536 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 495 (ObjNum 2918 snap 0 seq_num 2918) dirty exists 2020-06-12T02:37:47.537 INFO:tasks.rados.rados.0.smithi107.stdout:2919: left oid 3 (ObjNum 2918 snap 0 seq_num 2918) 2020-06-12T02:37:47.537 INFO:tasks.rados.rados.0.smithi107.stdout:2919: done (2 left) 2020-06-12T02:37:47.537 INFO:tasks.rados.rados.0.smithi107.stdout:2924: done (1 left) 2020-06-12T02:37:47.537 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:37:47.538 INFO:tasks.rados.rados.0.smithi107.stdout:2925: seq_num 2924 ranges {0=573440,573440=401408} 2020-06-12T02:37:47.538 INFO:tasks.rados.rados.0.smithi107.stdout:2925: writing smithi10729407-12 from 0 to 573440 tid 1 2020-06-12T02:37:47.538 INFO:tasks.rados.rados.0.smithi107.stdout:2925: writing smithi10729407-12 from 573440 to 974848 tid 2 2020-06-12T02:37:47.538 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:37:47.539 INFO:tasks.rados.rados.0.smithi107.stdout:2926: seq_num 2925 ranges {0=598016,598016=401408} 2020-06-12T02:37:47.539 INFO:tasks.rados.rados.0.smithi107.stdout:2926: writing smithi10729407-17 from 0 to 598016 tid 1 2020-06-12T02:37:47.539 INFO:tasks.rados.rados.0.smithi107.stdout:2926: writing smithi10729407-17 from 598016 to 999424 tid 2 2020-06-12T02:37:47.539 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:37:47.539 INFO:tasks.rados.rados.0.smithi107.stdout:2927: seq_num 2926 ranges {0=614400,614400=401408} 2020-06-12T02:37:47.540 INFO:tasks.rados.rados.0.smithi107.stdout:2927: writing smithi10729407-6 from 0 to 614400 tid 1 2020-06-12T02:37:47.540 INFO:tasks.rados.rados.0.smithi107.stdout:2927: writing smithi10729407-6 from 614400 to 1015808 tid 2 2020-06-12T02:37:47.540 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:37:47.540 INFO:tasks.rados.rados.0.smithi107.stdout:2928: seq_num 2927 ranges {0=630784,630784=401408} 2020-06-12T02:37:47.540 INFO:tasks.rados.rados.0.smithi107.stdout:2928: writing smithi10729407-14 from 0 to 630784 tid 1 2020-06-12T02:37:47.541 INFO:tasks.rados.rados.0.smithi107.stdout:2928: writing smithi10729407-14 from 630784 to 1032192 tid 2 2020-06-12T02:37:47.541 INFO:tasks.rados.rados.0.smithi107.stdout:2929: delete oid 15 current snap is 0 2020-06-12T02:37:47.541 INFO:tasks.rados.rados.0.smithi107.stdout:2923: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:47.541 INFO:tasks.rados.rados.0.smithi107.stdout:2923: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:47.541 INFO:tasks.rados.rados.0.smithi107.stdout:2923: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:47.542 INFO:tasks.rados.rados.0.smithi107.stdout:2923: finishing write tid 5 to smithi10729407-13 2020-06-12T02:37:47.542 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 571 (ObjNum 2922 snap 0 seq_num 2922) dirty exists 2020-06-12T02:37:47.542 INFO:tasks.rados.rados.0.smithi107.stdout:2923: left oid 13 (ObjNum 2922 snap 0 seq_num 2922) 2020-06-12T02:37:47.542 INFO:tasks.rados.rados.0.smithi107.stdout:2925: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:47.542 INFO:tasks.rados.rados.0.smithi107.stdout:2925: finishing write tid 2 to smithi10729407-12 2020-06-12T02:37:47.543 INFO:tasks.rados.rados.0.smithi107.stdout:2925: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:47.543 INFO:tasks.rados.rados.0.smithi107.stdout:2925: finishing write tid 5 to smithi10729407-12 2020-06-12T02:37:47.543 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 474 (ObjNum 2924 snap 0 seq_num 2924) dirty exists 2020-06-12T02:37:47.543 INFO:tasks.rados.rados.0.smithi107.stdout:2925: left oid 12 (ObjNum 2924 snap 0 seq_num 2924) 2020-06-12T02:37:47.543 INFO:tasks.rados.rados.0.smithi107.stdout:2926: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:47.544 INFO:tasks.rados.rados.0.smithi107.stdout:2926: finishing write tid 2 to smithi10729407-17 2020-06-12T02:37:47.544 INFO:tasks.rados.rados.0.smithi107.stdout:2926: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:47.544 INFO:tasks.rados.rados.0.smithi107.stdout:2927: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:47.544 INFO:tasks.rados.rados.0.smithi107.stdout:2926: finishing write tid 5 to smithi10729407-17 2020-06-12T02:37:47.545 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 552 (ObjNum 2925 snap 0 seq_num 2925) dirty exists 2020-06-12T02:37:47.545 INFO:tasks.rados.rados.0.smithi107.stdout:2926: left oid 17 (ObjNum 2925 snap 0 seq_num 2925) 2020-06-12T02:37:47.546 INFO:tasks.rados.rados.0.smithi107.stdout:2923: done (5 left) 2020-06-12T02:37:47.546 INFO:tasks.rados.rados.0.smithi107.stdout:2925: done (4 left) 2020-06-12T02:37:47.546 INFO:tasks.rados.rados.0.smithi107.stdout:2926: done (3 left) 2020-06-12T02:37:47.546 INFO:tasks.rados.rados.0.smithi107.stdout:2929: done (2 left) 2020-06-12T02:37:47.546 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:37:47.547 INFO:tasks.rados.rados.0.smithi107.stdout:2930: seq_num 2929 ranges {0=663552,663552=401408} 2020-06-12T02:37:47.547 INFO:tasks.rados.rados.0.smithi107.stdout:2930: writing smithi10729407-8 from 0 to 663552 tid 1 2020-06-12T02:37:47.547 INFO:tasks.rados.rados.0.smithi107.stdout:2930: writing smithi10729407-8 from 663552 to 1064960 tid 2 2020-06-12T02:37:47.547 INFO:tasks.rados.rados.0.smithi107.stdout:2931: delete oid 16 current snap is 0 2020-06-12T02:37:47.547 INFO:tasks.rados.rados.0.smithi107.stdout:2927: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:47.548 INFO:tasks.rados.rados.0.smithi107.stdout:2927: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:47.548 INFO:tasks.rados.rados.0.smithi107.stdout:2928: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:47.548 INFO:tasks.rados.rados.0.smithi107.stdout:2928: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:47.548 INFO:tasks.rados.rados.0.smithi107.stdout:2927: finishing write tid 5 to smithi10729407-6 2020-06-12T02:37:47.548 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 583 (ObjNum 2926 snap 0 seq_num 2926) dirty exists 2020-06-12T02:37:47.549 INFO:tasks.rados.rados.0.smithi107.stdout:2927: left oid 6 (ObjNum 2926 snap 0 seq_num 2926) 2020-06-12T02:37:47.549 INFO:tasks.rados.rados.0.smithi107.stdout:2928: finishing write tid 4 to smithi10729407-14 2020-06-12T02:37:47.549 INFO:tasks.rados.rados.0.smithi107.stdout:2928: finishing write tid 5 to smithi10729407-14 2020-06-12T02:37:47.549 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 500 (ObjNum 2927 snap 0 seq_num 2927) dirty exists 2020-06-12T02:37:47.549 INFO:tasks.rados.rados.0.smithi107.stdout:2928: left oid 14 (ObjNum 2927 snap 0 seq_num 2927) 2020-06-12T02:37:47.550 INFO:tasks.rados.rados.0.smithi107.stdout:2930: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:47.550 INFO:tasks.rados.rados.0.smithi107.stdout:2927: done (3 left) 2020-06-12T02:37:47.550 INFO:tasks.rados.rados.0.smithi107.stdout:2928: done (2 left) 2020-06-12T02:37:47.550 INFO:tasks.rados.rados.0.smithi107.stdout:2931: done (1 left) 2020-06-12T02:37:47.551 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:37:47.551 INFO:tasks.rados.rados.0.smithi107.stdout:2932: seq_num 2931 ranges {1032192=696320,1728512=401408} 2020-06-12T02:37:47.551 INFO:tasks.rados.rados.0.smithi107.stdout:2932: writing smithi10729407-14 from 1032192 to 1728512 tid 1 2020-06-12T02:37:47.551 INFO:tasks.rados.rados.0.smithi107.stdout:2932: writing smithi10729407-14 from 1728512 to 2129920 tid 2 2020-06-12T02:37:47.551 INFO:tasks.rados.rados.0.smithi107.stdout:2933: delete oid 20 current snap is 0 2020-06-12T02:37:47.552 INFO:tasks.rados.rados.0.smithi107.stdout:2930: finishing write tid 2 to smithi10729407-8 2020-06-12T02:37:47.552 INFO:tasks.rados.rados.0.smithi107.stdout:2930: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:47.552 INFO:tasks.rados.rados.0.smithi107.stdout:2930: finishing write tid 5 to smithi10729407-8 2020-06-12T02:37:47.552 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 494 (ObjNum 2929 snap 0 seq_num 2929) dirty exists 2020-06-12T02:37:47.552 INFO:tasks.rados.rados.0.smithi107.stdout:2930: left oid 8 (ObjNum 2929 snap 0 seq_num 2929) 2020-06-12T02:37:47.553 INFO:tasks.rados.rados.0.smithi107.stdout:2930: done (2 left) 2020-06-12T02:37:47.553 INFO:tasks.rados.rados.0.smithi107.stdout:2933: done (1 left) 2020-06-12T02:37:47.553 INFO:tasks.rados.rados.0.smithi107.stdout:2934: delete oid 5 current snap is 0 2020-06-12T02:37:47.553 INFO:tasks.rados.rados.0.smithi107.stdout:2934: done (1 left) 2020-06-12T02:37:47.553 INFO:tasks.rados.rados.0.smithi107.stdout:2935: delete oid 7 current snap is 0 2020-06-12T02:37:47.554 INFO:tasks.rados.rados.0.smithi107.stdout:2932: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:47.554 INFO:tasks.rados.rados.0.smithi107.stdout:2935: done (1 left) 2020-06-12T02:37:47.554 INFO:tasks.rados.rados.0.smithi107.stdout:2936: delete oid 8 current snap is 0 2020-06-12T02:37:47.554 INFO:tasks.rados.rados.0.smithi107.stdout:2936: done (1 left) 2020-06-12T02:37:47.555 INFO:tasks.rados.rados.0.smithi107.stdout:2937: delete oid 9 current snap is 0 2020-06-12T02:37:47.555 INFO:tasks.rados.rados.0.smithi107.stdout:2932: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:47.555 INFO:tasks.rados.rados.0.smithi107.stdout:2932: finishing write tid 4 to smithi10729407-14 2020-06-12T02:37:47.555 INFO:tasks.rados.rados.0.smithi107.stdout:2932: finishing write tid 5 to smithi10729407-14 2020-06-12T02:37:47.556 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 503 (ObjNum 2931 snap 0 seq_num 2931) dirty exists 2020-06-12T02:37:47.556 INFO:tasks.rados.rados.0.smithi107.stdout:2932: left oid 14 (ObjNum 2931 snap 0 seq_num 2931) 2020-06-12T02:37:47.556 INFO:tasks.rados.rados.0.smithi107.stdout:2932: done (1 left) 2020-06-12T02:37:47.556 INFO:tasks.rados.rados.0.smithi107.stdout:2937: done (0 left) 2020-06-12T02:37:47.556 INFO:tasks.rados.rados.0.smithi107.stdout:2938: delete oid 10 current snap is 0 2020-06-12T02:37:47.557 INFO:tasks.rados.rados.0.smithi107.stdout:2938: done (0 left) 2020-06-12T02:37:47.557 INFO:tasks.rados.rados.0.smithi107.stdout:2939: delete oid 12 current snap is 0 2020-06-12T02:37:47.557 INFO:tasks.rados.rados.0.smithi107.stdout:2939: done (0 left) 2020-06-12T02:37:47.557 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:37:47.557 INFO:tasks.rados.rados.0.smithi107.stdout:2940: seq_num 2939 ranges {0=425984,425984=434176,860160=368640} 2020-06-12T02:37:47.558 INFO:tasks.rados.rados.0.smithi107.stdout:2940: writing smithi10729407-9 from 0 to 425984 tid 1 2020-06-12T02:37:47.558 INFO:tasks.rados.rados.0.smithi107.stdout:2940: writing smithi10729407-9 from 425984 to 860160 tid 2 2020-06-12T02:37:47.558 INFO:tasks.rados.rados.0.smithi107.stdout:2940: writing smithi10729407-9 from 860160 to 1228800 tid 3 2020-06-12T02:37:47.558 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:47.558 INFO:tasks.rados.rados.0.smithi107.stdout:2941: seq_num 2940 ranges {0=442368,442368=720896,1163264=81920} 2020-06-12T02:37:47.559 INFO:tasks.rados.rados.0.smithi107.stdout:2941: writing smithi10729407-2 from 0 to 442368 tid 1 2020-06-12T02:37:47.559 INFO:tasks.rados.rados.0.smithi107.stdout:2941: writing smithi10729407-2 from 442368 to 1163264 tid 2 2020-06-12T02:37:47.559 INFO:tasks.rados.rados.0.smithi107.stdout:2941: writing smithi10729407-2 from 1163264 to 1245184 tid 3 2020-06-12T02:37:47.559 INFO:tasks.rados.rados.0.smithi107.stdout:2942: delete oid 10 current snap is 0 2020-06-12T02:37:47.559 INFO:tasks.rados.rados.0.smithi107.stdout:2940: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:47.560 INFO:tasks.rados.rados.0.smithi107.stdout:2941: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:47.560 INFO:tasks.rados.rados.0.smithi107.stdout:2940: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:47.560 INFO:tasks.rados.rados.0.smithi107.stdout:2940: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:47.560 INFO:tasks.rados.rados.0.smithi107.stdout:2940: finishing write tid 5 to smithi10729407-9 2020-06-12T02:37:47.560 INFO:tasks.rados.rados.0.smithi107.stdout:2942: done (2 left) 2020-06-12T02:37:47.561 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:37:47.561 INFO:tasks.rados.rados.0.smithi107.stdout:2943: seq_num 2942 ranges {0=475136,475136=483328,958464=319488} 2020-06-12T02:37:47.561 INFO:tasks.rados.rados.0.smithi107.stdout:2943: writing smithi10729407-10 from 0 to 475136 tid 1 2020-06-12T02:37:47.561 INFO:tasks.rados.rados.0.smithi107.stdout:2943: writing smithi10729407-10 from 475136 to 958464 tid 2 2020-06-12T02:37:47.561 INFO:tasks.rados.rados.0.smithi107.stdout:2943: writing smithi10729407-10 from 958464 to 1277952 tid 3 2020-06-12T02:37:47.562 INFO:tasks.rados.rados.0.smithi107.stdout:2944: delete oid 17 current snap is 0 2020-06-12T02:37:47.562 INFO:tasks.rados.rados.0.smithi107.stdout:2940: finishing write tid 6 to smithi10729407-9 2020-06-12T02:37:47.565 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 520 (ObjNum 2939 snap 0 seq_num 2939) dirty exists 2020-06-12T02:37:47.565 INFO:tasks.rados.rados.0.smithi107.stdout:2940: left oid 9 (ObjNum 2939 snap 0 seq_num 2939) 2020-06-12T02:37:47.565 INFO:tasks.rados.rados.0.smithi107.stdout:2941: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:47.565 INFO:tasks.rados.rados.0.smithi107.stdout:2941: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:47.566 INFO:tasks.rados.rados.0.smithi107.stdout:2941: finishing write tid 5 to smithi10729407-2 2020-06-12T02:37:47.566 INFO:tasks.rados.rados.0.smithi107.stdout:2941: finishing write tid 6 to smithi10729407-2 2020-06-12T02:37:47.566 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 306 (ObjNum 2940 snap 0 seq_num 2940) dirty exists 2020-06-12T02:37:47.566 INFO:tasks.rados.rados.0.smithi107.stdout:2941: left oid 2 (ObjNum 2940 snap 0 seq_num 2940) 2020-06-12T02:37:47.566 INFO:tasks.rados.rados.0.smithi107.stdout:2943: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:47.566 INFO:tasks.rados.rados.0.smithi107.stdout:2940: done (3 left) 2020-06-12T02:37:47.567 INFO:tasks.rados.rados.0.smithi107.stdout:2941: done (2 left) 2020-06-12T02:37:47.567 INFO:tasks.rados.rados.0.smithi107.stdout:2944: done (1 left) 2020-06-12T02:37:47.567 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:37:47.567 INFO:tasks.rados.rados.0.smithi107.stdout:2945: seq_num 2944 ranges {0=516096,516096=704512,1220608=98304} 2020-06-12T02:37:47.567 INFO:tasks.rados.rados.0.smithi107.stdout:2945: writing smithi10729407-5 from 0 to 516096 tid 1 2020-06-12T02:37:47.568 INFO:tasks.rados.rados.0.smithi107.stdout:2945: writing smithi10729407-5 from 516096 to 1220608 tid 2 2020-06-12T02:37:47.568 INFO:tasks.rados.rados.0.smithi107.stdout:2945: writing smithi10729407-5 from 1220608 to 1318912 tid 3 2020-06-12T02:37:47.568 INFO:tasks.rados.rados.0.smithi107.stdout:2946: delete oid 18 current snap is 0 2020-06-12T02:37:47.568 INFO:tasks.rados.rados.0.smithi107.stdout:2943: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:47.568 INFO:tasks.rados.rados.0.smithi107.stdout:2943: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:47.569 INFO:tasks.rados.rados.0.smithi107.stdout:2943: finishing write tid 5 to smithi10729407-10 2020-06-12T02:37:47.569 INFO:tasks.rados.rados.0.smithi107.stdout:2943: finishing write tid 6 to smithi10729407-10 2020-06-12T02:37:47.569 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 282 (ObjNum 2942 snap 0 seq_num 2942) dirty exists 2020-06-12T02:37:47.569 INFO:tasks.rados.rados.0.smithi107.stdout:2943: left oid 10 (ObjNum 2942 snap 0 seq_num 2942) 2020-06-12T02:37:47.569 INFO:tasks.rados.rados.0.smithi107.stdout:2945: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:47.570 INFO:tasks.rados.rados.0.smithi107.stdout:2943: done (2 left) 2020-06-12T02:37:47.570 INFO:tasks.rados.rados.0.smithi107.stdout:2946: done (1 left) 2020-06-12T02:37:47.570 INFO:tasks.rados.rados.0.smithi107.stdout:2947: delete oid 8 current snap is 0 2020-06-12T02:37:47.570 INFO:tasks.rados.rados.0.smithi107.stdout:2947: done (1 left) 2020-06-12T02:37:47.570 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:37:47.570 INFO:tasks.rados.rados.0.smithi107.stdout:2948: seq_num 2947 ranges {0=565248,565248=761856,1327104=40960} 2020-06-12T02:37:47.571 INFO:tasks.rados.rados.0.smithi107.stdout:2948: writing smithi10729407-16 from 0 to 565248 tid 1 2020-06-12T02:37:47.571 INFO:tasks.rados.rados.0.smithi107.stdout:2948: writing smithi10729407-16 from 565248 to 1327104 tid 2 2020-06-12T02:37:47.571 INFO:tasks.rados.rados.0.smithi107.stdout:2948: writing smithi10729407-16 from 1327104 to 1368064 tid 3 2020-06-12T02:37:47.571 INFO:tasks.rados.rados.0.smithi107.stdout:2949: delete oid 18 current snap is 0 2020-06-12T02:37:47.571 INFO:tasks.rados.rados.0.smithi107.stdout:2945: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:47.572 INFO:tasks.rados.rados.0.smithi107.stdout:2945: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:47.572 INFO:tasks.rados.rados.0.smithi107.stdout:2945: finishing write tid 5 to smithi10729407-5 2020-06-12T02:37:47.572 INFO:tasks.rados.rados.0.smithi107.stdout:2945: finishing write tid 6 to smithi10729407-5 2020-06-12T02:37:47.572 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 249 (ObjNum 2944 snap 0 seq_num 2944) dirty exists 2020-06-12T02:37:47.572 INFO:tasks.rados.rados.0.smithi107.stdout:2945: left oid 5 (ObjNum 2944 snap 0 seq_num 2944) 2020-06-12T02:37:47.573 INFO:tasks.rados.rados.0.smithi107.stdout:2948: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:47.573 INFO:tasks.rados.rados.0.smithi107.stdout:2945: done (2 left) 2020-06-12T02:37:47.573 INFO:tasks.rados.rados.0.smithi107.stdout:2949: done (1 left) 2020-06-12T02:37:47.573 INFO:tasks.rados.rados.0.smithi107.stdout:2950: delete oid 6 current snap is 0 2020-06-12T02:37:47.573 INFO:tasks.rados.rados.0.smithi107.stdout:2950: done (1 left) 2020-06-12T02:37:47.573 INFO:tasks.rados.rados.0.smithi107.stdout:2951: delete oid 15 current snap is 0 2020-06-12T02:37:47.574 INFO:tasks.rados.rados.0.smithi107.stdout:2951: done (1 left) 2020-06-12T02:37:47.574 INFO:tasks.rados.rados.0.smithi107.stdout:2952: delete oid 17 current snap is 0 2020-06-12T02:37:47.574 INFO:tasks.rados.rados.0.smithi107.stdout:2948: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:47.574 INFO:tasks.rados.rados.0.smithi107.stdout:2948: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:47.574 INFO:tasks.rados.rados.0.smithi107.stdout:2948: finishing write tid 5 to smithi10729407-16 2020-06-12T02:37:47.575 INFO:tasks.rados.rados.0.smithi107.stdout:2952: done (1 left) 2020-06-12T02:37:47.575 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:37:47.575 INFO:tasks.rados.rados.0.smithi107.stdout:2953: seq_num 2952 ranges {0=647168,647168=630784,1277952=172032} 2020-06-12T02:37:47.575 INFO:tasks.rados.rados.0.smithi107.stdout:2953: writing smithi10729407-15 from 0 to 647168 tid 1 2020-06-12T02:37:47.575 INFO:tasks.rados.rados.0.smithi107.stdout:2953: writing smithi10729407-15 from 647168 to 1277952 tid 2 2020-06-12T02:37:47.576 INFO:tasks.rados.rados.0.smithi107.stdout:2953: writing smithi10729407-15 from 1277952 to 1449984 tid 3 2020-06-12T02:37:47.576 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:37:47.576 INFO:tasks.rados.rados.0.smithi107.stdout:2954: seq_num 2953 ranges {0=663552,663552=516096,1179648=286720} 2020-06-12T02:37:47.576 INFO:tasks.rados.rados.0.smithi107.stdout:2954: writing smithi10729407-12 from 0 to 663552 tid 1 2020-06-12T02:37:47.576 INFO:tasks.rados.rados.0.smithi107.stdout:2954: writing smithi10729407-12 from 663552 to 1179648 tid 2 2020-06-12T02:37:47.576 INFO:tasks.rados.rados.0.smithi107.stdout:2954: writing smithi10729407-12 from 1179648 to 1466368 tid 3 2020-06-12T02:37:47.577 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:37:47.577 INFO:tasks.rados.rados.0.smithi107.stdout:2955: seq_num 2954 ranges {1318912=679936,1998848=802816} 2020-06-12T02:37:47.577 INFO:tasks.rados.rados.0.smithi107.stdout:2955: writing smithi10729407-5 from 1318912 to 1998848 tid 1 2020-06-12T02:37:47.577 INFO:tasks.rados.rados.0.smithi107.stdout:2955: writing smithi10729407-5 from 1998848 to 2801664 tid 2 2020-06-12T02:37:47.577 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:37:47.578 INFO:tasks.rados.rados.0.smithi107.stdout:2956: seq_num 2955 ranges {0=696320,696320=679936,1376256=122880} 2020-06-12T02:37:47.579 INFO:tasks.rados.rados.0.smithi107.stdout:2956: writing smithi10729407-18 from 0 to 696320 tid 1 2020-06-12T02:37:47.579 INFO:tasks.rados.rados.0.smithi107.stdout:2956: writing smithi10729407-18 from 696320 to 1376256 tid 2 2020-06-12T02:37:47.579 INFO:tasks.rados.rados.0.smithi107.stdout:2956: writing smithi10729407-18 from 1376256 to 1499136 tid 3 2020-06-12T02:37:47.579 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:37:47.580 INFO:tasks.rados.rados.0.smithi107.stdout:2957: seq_num 2956 ranges {778240=712704,1490944=565248,2056192=237568} 2020-06-12T02:37:47.580 INFO:tasks.rados.rados.0.smithi107.stdout:2957: writing smithi10729407-4 from 778240 to 1490944 tid 1 2020-06-12T02:37:47.580 INFO:tasks.rados.rados.0.smithi107.stdout:2957: writing smithi10729407-4 from 1490944 to 2056192 tid 2 2020-06-12T02:37:47.580 INFO:tasks.rados.rados.0.smithi107.stdout:2957: writing smithi10729407-4 from 2056192 to 2293760 tid 3 2020-06-12T02:37:47.580 INFO:tasks.rados.rados.0.smithi107.stdout:2958: delete oid 10 current snap is 0 2020-06-12T02:37:47.581 INFO:tasks.rados.rados.0.smithi107.stdout:2948: finishing write tid 6 to smithi10729407-16 2020-06-12T02:37:47.581 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 500 (ObjNum 2947 snap 0 seq_num 2947) dirty exists 2020-06-12T02:37:47.581 INFO:tasks.rados.rados.0.smithi107.stdout:2948: left oid 16 (ObjNum 2947 snap 0 seq_num 2947) 2020-06-12T02:37:47.581 INFO:tasks.rados.rados.0.smithi107.stdout:2953: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:47.581 INFO:tasks.rados.rados.0.smithi107.stdout:2953: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:47.582 INFO:tasks.rados.rados.0.smithi107.stdout:2953: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:47.582 INFO:tasks.rados.rados.0.smithi107.stdout:2953: finishing write tid 5 to smithi10729407-15 2020-06-12T02:37:47.582 INFO:tasks.rados.rados.0.smithi107.stdout:2953: finishing write tid 6 to smithi10729407-15 2020-06-12T02:37:47.582 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 501 (ObjNum 2952 snap 0 seq_num 2952) dirty exists 2020-06-12T02:37:47.582 INFO:tasks.rados.rados.0.smithi107.stdout:2953: left oid 15 (ObjNum 2952 snap 0 seq_num 2952) 2020-06-12T02:37:47.582 INFO:tasks.rados.rados.0.smithi107.stdout:2954: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:47.583 INFO:tasks.rados.rados.0.smithi107.stdout:2954: finishing write tid 2 to smithi10729407-12 2020-06-12T02:37:47.583 INFO:tasks.rados.rados.0.smithi107.stdout:2954: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:47.583 INFO:tasks.rados.rados.0.smithi107.stdout:2954: finishing write tid 5 to smithi10729407-12 2020-06-12T02:37:47.583 INFO:tasks.rados.rados.0.smithi107.stdout:2954: finishing write tid 6 to smithi10729407-12 2020-06-12T02:37:47.583 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 479 (ObjNum 2953 snap 0 seq_num 2953) dirty exists 2020-06-12T02:37:47.583 INFO:tasks.rados.rados.0.smithi107.stdout:2954: left oid 12 (ObjNum 2953 snap 0 seq_num 2953) 2020-06-12T02:37:47.584 INFO:tasks.rados.rados.0.smithi107.stdout:2955: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:47.584 INFO:tasks.rados.rados.0.smithi107.stdout:2955: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:47.584 INFO:tasks.rados.rados.0.smithi107.stdout:2955: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:47.584 INFO:tasks.rados.rados.0.smithi107.stdout:2955: finishing write tid 5 to smithi10729407-5 2020-06-12T02:37:47.584 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 252 (ObjNum 2954 snap 0 seq_num 2954) dirty exists 2020-06-12T02:37:47.585 INFO:tasks.rados.rados.0.smithi107.stdout:2955: left oid 5 (ObjNum 2954 snap 0 seq_num 2954) 2020-06-12T02:37:47.585 INFO:tasks.rados.rados.0.smithi107.stdout:2956: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:47.585 INFO:tasks.rados.rados.0.smithi107.stdout:2956: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:47.586 INFO:tasks.rados.rados.0.smithi107.stdout:2956: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:47.586 INFO:tasks.rados.rados.0.smithi107.stdout:2956: finishing write tid 5 to smithi10729407-18 2020-06-12T02:37:47.587 INFO:tasks.rados.rados.0.smithi107.stdout:2956: finishing write tid 6 to smithi10729407-18 2020-06-12T02:37:47.587 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 526 (ObjNum 2955 snap 0 seq_num 2955) dirty exists 2020-06-12T02:37:47.587 INFO:tasks.rados.rados.0.smithi107.stdout:2956: left oid 18 (ObjNum 2955 snap 0 seq_num 2955) 2020-06-12T02:37:47.588 INFO:tasks.rados.rados.0.smithi107.stdout:2957: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:47.588 INFO:tasks.rados.rados.0.smithi107.stdout:2948: done (6 left) 2020-06-12T02:37:47.588 INFO:tasks.rados.rados.0.smithi107.stdout:2953: done (5 left) 2020-06-12T02:37:47.588 INFO:tasks.rados.rados.0.smithi107.stdout:2954: done (4 left) 2020-06-12T02:37:47.588 INFO:tasks.rados.rados.0.smithi107.stdout:2955: done (3 left) 2020-06-12T02:37:47.588 INFO:tasks.rados.rados.0.smithi107.stdout:2956: done (2 left) 2020-06-12T02:37:47.589 INFO:tasks.rados.rados.0.smithi107.stdout:2958: done (1 left) 2020-06-12T02:37:47.589 INFO:tasks.rados.rados.0.smithi107.stdout:2959: delete oid 15 current snap is 0 2020-06-12T02:37:47.589 INFO:tasks.rados.rados.0.smithi107.stdout:2959: done (1 left) 2020-06-12T02:37:47.589 INFO:tasks.rados.rados.0.smithi107.stdout:2960: delete oid 15 current snap is 0 2020-06-12T02:37:47.589 INFO:tasks.rados.rados.0.smithi107.stdout:2957: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:47.590 INFO:tasks.rados.rados.0.smithi107.stdout:2960: done (1 left) 2020-06-12T02:37:47.590 INFO:tasks.rados.rados.0.smithi107.stdout:2961: delete oid 7 current snap is 0 2020-06-12T02:37:47.590 INFO:tasks.rados.rados.0.smithi107.stdout:2957: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:47.590 INFO:tasks.rados.rados.0.smithi107.stdout:2957: finishing write tid 5 to smithi10729407-4 2020-06-12T02:37:47.590 INFO:tasks.rados.rados.0.smithi107.stdout:2957: finishing write tid 6 to smithi10729407-4 2020-06-12T02:37:47.591 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 558 (ObjNum 2956 snap 0 seq_num 2956) dirty exists 2020-06-12T02:37:47.591 INFO:tasks.rados.rados.0.smithi107.stdout:2957: left oid 4 (ObjNum 2956 snap 0 seq_num 2956) 2020-06-12T02:37:47.591 INFO:tasks.rados.rados.0.smithi107.stdout:2957: done (1 left) 2020-06-12T02:37:47.591 INFO:tasks.rados.rados.0.smithi107.stdout:2961: done (0 left) 2020-06-12T02:37:47.592 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:37:47.592 INFO:tasks.rados.rados.0.smithi107.stdout:2962: seq_num 2961 ranges {1499136=794624,2293760=434176,2727936=368640} 2020-06-12T02:37:47.592 INFO:tasks.rados.rados.0.smithi107.stdout:2962: writing smithi10729407-18 from 1499136 to 2293760 tid 1 2020-06-12T02:37:47.592 INFO:tasks.rados.rados.0.smithi107.stdout:2962: writing smithi10729407-18 from 2293760 to 2727936 tid 2 2020-06-12T02:37:47.592 INFO:tasks.rados.rados.0.smithi107.stdout:2962: writing smithi10729407-18 from 2727936 to 3096576 tid 3 2020-06-12T02:37:47.593 INFO:tasks.rados.rados.0.smithi107.stdout:2963: delete oid 8 current snap is 0 2020-06-12T02:37:47.593 INFO:tasks.rados.rados.0.smithi107.stdout:2963: done (1 left) 2020-06-12T02:37:47.593 INFO:tasks.rados.rados.0.smithi107.stdout:2964: delete oid 2 current snap is 0 2020-06-12T02:37:47.593 INFO:tasks.rados.rados.0.smithi107.stdout:2964: done (1 left) 2020-06-12T02:37:47.593 INFO:tasks.rados.rados.0.smithi107.stdout:2965: delete oid 8 current snap is 0 2020-06-12T02:37:47.594 INFO:tasks.rados.rados.0.smithi107.stdout:2962: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:47.594 INFO:tasks.rados.rados.0.smithi107.stdout:2962: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:47.594 INFO:tasks.rados.rados.0.smithi107.stdout:2965: done (1 left) 2020-06-12T02:37:47.594 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:37:47.594 INFO:tasks.rados.rados.0.smithi107.stdout:2966: seq_num 2965 ranges {1368064=466944,1835008=425984,2260992=516096,2777088=262144} 2020-06-12T02:37:47.594 INFO:tasks.rados.rados.0.smithi107.stdout:2966: writing smithi10729407-16 from 1368064 to 1835008 tid 1 2020-06-12T02:37:47.595 INFO:tasks.rados.rados.0.smithi107.stdout:2966: writing smithi10729407-16 from 1835008 to 2260992 tid 2 2020-06-12T02:37:47.595 INFO:tasks.rados.rados.0.smithi107.stdout:2966: writing smithi10729407-16 from 2260992 to 2777088 tid 3 2020-06-12T02:37:47.595 INFO:tasks.rados.rados.0.smithi107.stdout:2966: writing smithi10729407-16 from 2777088 to 3039232 tid 4 2020-06-12T02:37:47.595 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:37:47.596 INFO:tasks.rados.rados.0.smithi107.stdout:2967: seq_num 2966 ranges {0=483328,483328=712704,1196032=491520} 2020-06-12T02:37:47.596 INFO:tasks.rados.rados.0.smithi107.stdout:2967: writing smithi10729407-17 from 0 to 483328 tid 1 2020-06-12T02:37:47.596 INFO:tasks.rados.rados.0.smithi107.stdout:2967: writing smithi10729407-17 from 483328 to 1196032 tid 2 2020-06-12T02:37:47.596 INFO:tasks.rados.rados.0.smithi107.stdout:2967: writing smithi10729407-17 from 1196032 to 1687552 tid 3 2020-06-12T02:37:47.597 INFO:tasks.rados.rados.0.smithi107.stdout:2968: delete oid 8 current snap is 0 2020-06-12T02:37:47.597 INFO:tasks.rados.rados.0.smithi107.stdout:2962: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:47.597 INFO:tasks.rados.rados.0.smithi107.stdout:2962: finishing write tid 5 to smithi10729407-18 2020-06-12T02:37:47.597 INFO:tasks.rados.rados.0.smithi107.stdout:2962: finishing write tid 6 to smithi10729407-18 2020-06-12T02:37:47.597 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 530 (ObjNum 2961 snap 0 seq_num 2961) dirty exists 2020-06-12T02:37:47.598 INFO:tasks.rados.rados.0.smithi107.stdout:2962: left oid 18 (ObjNum 2961 snap 0 seq_num 2961) 2020-06-12T02:37:47.599 INFO:tasks.rados.rados.0.smithi107.stdout:2966: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:47.599 INFO:tasks.rados.rados.0.smithi107.stdout:2966: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:47.599 INFO:tasks.rados.rados.0.smithi107.stdout:2966: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:47.600 INFO:tasks.rados.rados.0.smithi107.stdout:2966: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:47.600 INFO:tasks.rados.rados.0.smithi107.stdout:2966: finishing write tid 6 to smithi10729407-16 2020-06-12T02:37:47.600 INFO:tasks.rados.rados.0.smithi107.stdout:2966: finishing write tid 7 to smithi10729407-16 2020-06-12T02:37:47.600 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 505 (ObjNum 2965 snap 0 seq_num 2965) dirty exists 2020-06-12T02:37:47.600 INFO:tasks.rados.rados.0.smithi107.stdout:2966: left oid 16 (ObjNum 2965 snap 0 seq_num 2965) 2020-06-12T02:37:47.601 INFO:tasks.rados.rados.0.smithi107.stdout:2967: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:47.601 INFO:tasks.rados.rados.0.smithi107.stdout:2962: done (3 left) 2020-06-12T02:37:47.601 INFO:tasks.rados.rados.0.smithi107.stdout:2966: done (2 left) 2020-06-12T02:37:47.601 INFO:tasks.rados.rados.0.smithi107.stdout:2968: done (1 left) 2020-06-12T02:37:47.601 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:37:47.601 INFO:tasks.rados.rados.0.smithi107.stdout:2969: seq_num 2968 ranges {2801664=516096,3317760=475136,3792896=688128,4481024=40960} 2020-06-12T02:37:47.602 INFO:tasks.rados.rados.0.smithi107.stdout:2969: writing smithi10729407-5 from 2801664 to 3317760 tid 1 2020-06-12T02:37:47.602 INFO:tasks.rados.rados.0.smithi107.stdout:2969: writing smithi10729407-5 from 3317760 to 3792896 tid 2 2020-06-12T02:37:47.602 INFO:tasks.rados.rados.0.smithi107.stdout:2969: writing smithi10729407-5 from 3792896 to 4481024 tid 3 2020-06-12T02:37:47.602 INFO:tasks.rados.rados.0.smithi107.stdout:2969: writing smithi10729407-5 from 4481024 to 4521984 tid 4 2020-06-12T02:37:47.602 INFO:tasks.rados.rados.0.smithi107.stdout:2970: delete oid 6 current snap is 0 2020-06-12T02:37:47.603 INFO:tasks.rados.rados.0.smithi107.stdout:2967: finishing write tid 2 to smithi10729407-17 2020-06-12T02:37:47.603 INFO:tasks.rados.rados.0.smithi107.stdout:2967: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:47.603 INFO:tasks.rados.rados.0.smithi107.stdout:2967: finishing write tid 5 to smithi10729407-17 2020-06-12T02:37:47.603 INFO:tasks.rados.rados.0.smithi107.stdout:2967: finishing write tid 6 to smithi10729407-17 2020-06-12T02:37:47.603 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 562 (ObjNum 2966 snap 0 seq_num 2966) dirty exists 2020-06-12T02:37:47.603 INFO:tasks.rados.rados.0.smithi107.stdout:2967: left oid 17 (ObjNum 2966 snap 0 seq_num 2966) 2020-06-12T02:37:47.604 INFO:tasks.rados.rados.0.smithi107.stdout:2969: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:47.604 INFO:tasks.rados.rados.0.smithi107.stdout:2967: done (2 left) 2020-06-12T02:37:47.604 INFO:tasks.rados.rados.0.smithi107.stdout:2970: done (1 left) 2020-06-12T02:37:47.604 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:37:47.604 INFO:tasks.rados.rados.0.smithi107.stdout:2971: seq_num 2970 ranges {2293760=548864,2842624=647168,3489792=557056} 2020-06-12T02:37:47.604 INFO:tasks.rados.rados.0.smithi107.stdout:2971: writing smithi10729407-4 from 2293760 to 2842624 tid 1 2020-06-12T02:37:47.605 INFO:tasks.rados.rados.0.smithi107.stdout:2971: writing smithi10729407-4 from 2842624 to 3489792 tid 2 2020-06-12T02:37:47.605 INFO:tasks.rados.rados.0.smithi107.stdout:2971: writing smithi10729407-4 from 3489792 to 4046848 tid 3 2020-06-12T02:37:47.605 INFO:tasks.rados.rados.0.smithi107.stdout:2972: delete oid 19 current snap is 0 2020-06-12T02:37:47.605 INFO:tasks.rados.rados.0.smithi107.stdout:2969: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:47.605 INFO:tasks.rados.rados.0.smithi107.stdout:2969: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:47.606 INFO:tasks.rados.rados.0.smithi107.stdout:2969: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:47.606 INFO:tasks.rados.rados.0.smithi107.stdout:2969: finishing write tid 6 to smithi10729407-5 2020-06-12T02:37:47.606 INFO:tasks.rados.rados.0.smithi107.stdout:2969: finishing write tid 7 to smithi10729407-5 2020-06-12T02:37:47.606 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 257 (ObjNum 2968 snap 0 seq_num 2968) dirty exists 2020-06-12T02:37:47.606 INFO:tasks.rados.rados.0.smithi107.stdout:2969: left oid 5 (ObjNum 2968 snap 0 seq_num 2968) 2020-06-12T02:37:47.606 INFO:tasks.rados.rados.0.smithi107.stdout:2971: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:47.607 INFO:tasks.rados.rados.0.smithi107.stdout:2969: done (2 left) 2020-06-12T02:37:47.607 INFO:tasks.rados.rados.0.smithi107.stdout:2972: done (1 left) 2020-06-12T02:37:47.607 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:37:47.607 INFO:tasks.rados.rados.0.smithi107.stdout:2973: seq_num 2972 ranges {0=581632,581632=417792,999424=671744,1671168=114688} 2020-06-12T02:37:47.607 INFO:tasks.rados.rados.0.smithi107.stdout:2973: writing smithi10729407-20 from 0 to 581632 tid 1 2020-06-12T02:37:47.608 INFO:tasks.rados.rados.0.smithi107.stdout:2973: writing smithi10729407-20 from 581632 to 999424 tid 2 2020-06-12T02:37:47.608 INFO:tasks.rados.rados.0.smithi107.stdout:2973: writing smithi10729407-20 from 999424 to 1671168 tid 3 2020-06-12T02:37:47.608 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-06-12T02:37:47.609 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:37:47.505+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:37:47.616 INFO:tasks.rados.rados.0.smithi107.stdout:2973: writing smithi10729407-20 from 1671168 to 1785856 tid 4 2020-06-12T02:37:47.616 INFO:tasks.rados.rados.0.smithi107.stdout:2974: delete oid 5 current snap is 0 2020-06-12T02:37:47.616 INFO:tasks.rados.rados.0.smithi107.stdout:2971: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:47.617 INFO:tasks.rados.rados.0.smithi107.stdout:2971: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:47.617 INFO:tasks.rados.rados.0.smithi107.stdout:2971: finishing write tid 5 to smithi10729407-4 2020-06-12T02:37:47.617 INFO:tasks.rados.rados.0.smithi107.stdout:2971: finishing write tid 6 to smithi10729407-4 2020-06-12T02:37:47.617 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 566 (ObjNum 2970 snap 0 seq_num 2970) dirty exists 2020-06-12T02:37:47.617 INFO:tasks.rados.rados.0.smithi107.stdout:2971: left oid 4 (ObjNum 2970 snap 0 seq_num 2970) 2020-06-12T02:37:47.618 INFO:tasks.rados.rados.0.smithi107.stdout:2973: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:47.618 INFO:tasks.rados.rados.0.smithi107.stdout:2971: done (2 left) 2020-06-12T02:37:47.618 INFO:tasks.rados.rados.0.smithi107.stdout:2974: done (1 left) 2020-06-12T02:37:47.618 INFO:tasks.rados.rados.0.smithi107.stdout:2975: delete oid 3 current snap is 0 2020-06-12T02:37:47.618 INFO:tasks.rados.rados.0.smithi107.stdout:2973: finishing write tid 2 to smithi10729407-20 2020-06-12T02:37:47.618 INFO:tasks.rados.rados.0.smithi107.stdout:2975: done (1 left) 2020-06-12T02:37:47.619 INFO:tasks.rados.rados.0.smithi107.stdout:2976: delete oid 16 current snap is 0 2020-06-12T02:37:47.619 INFO:tasks.rados.rados.0.smithi107.stdout:2973: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:47.619 INFO:tasks.rados.rados.0.smithi107.stdout:2973: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:47.619 INFO:tasks.rados.rados.0.smithi107.stdout:2973: finishing write tid 6 to smithi10729407-20 2020-06-12T02:37:47.619 INFO:tasks.rados.rados.0.smithi107.stdout:2973: finishing write tid 7 to smithi10729407-20 2020-06-12T02:37:47.620 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 201 (ObjNum 2972 snap 0 seq_num 2972) dirty exists 2020-06-12T02:37:47.621 INFO:tasks.rados.rados.0.smithi107.stdout:2973: left oid 20 (ObjNum 2972 snap 0 seq_num 2972) 2020-06-12T02:37:47.621 INFO:tasks.rados.rados.0.smithi107.stdout:2973: done (1 left) 2020-06-12T02:37:47.621 INFO:tasks.rados.rados.0.smithi107.stdout:2976: done (0 left) 2020-06-12T02:37:47.622 INFO:tasks.rados.rados.0.smithi107.stdout:2977: delete oid 18 current snap is 0 2020-06-12T02:37:47.622 INFO:tasks.rados.rados.0.smithi107.stdout:2977: done (0 left) 2020-06-12T02:37:47.622 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:37:47.622 INFO:tasks.rados.rados.0.smithi107.stdout:2978: seq_num 2977 ranges {0=663552,663552=688128,1351680=458752,1810432=57344} 2020-06-12T02:37:47.622 INFO:tasks.rados.rados.0.smithi107.stdout:2978: writing smithi10729407-3 from 0 to 663552 tid 1 2020-06-12T02:37:47.623 INFO:tasks.rados.rados.0.smithi107.stdout:2978: writing smithi10729407-3 from 663552 to 1351680 tid 2 2020-06-12T02:37:47.623 INFO:tasks.rados.rados.0.smithi107.stdout:2978: writing smithi10729407-3 from 1351680 to 1810432 tid 3 2020-06-12T02:37:47.623 INFO:tasks.rados.rados.0.smithi107.stdout:2978: writing smithi10729407-3 from 1810432 to 1867776 tid 4 2020-06-12T02:37:47.623 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:37:47.623 INFO:tasks.rados.rados.0.smithi107.stdout:2979: seq_num 2978 ranges {0=679936,679936=573440,1253376=630784} 2020-06-12T02:37:47.623 INFO:tasks.rados.rados.0.smithi107.stdout:2979: writing smithi10729407-18 from 0 to 679936 tid 1 2020-06-12T02:37:47.624 INFO:tasks.rados.rados.0.smithi107.stdout:2979: writing smithi10729407-18 from 679936 to 1253376 tid 2 2020-06-12T02:37:47.624 INFO:tasks.rados.rados.0.smithi107.stdout:2979: writing smithi10729407-18 from 1253376 to 1884160 tid 3 2020-06-12T02:37:47.624 INFO:tasks.rados.rados.0.smithi107.stdout:2980: delete oid 12 current snap is 0 2020-06-12T02:37:47.625 INFO:tasks.rados.rados.0.smithi107.stdout:2978: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:47.625 INFO:tasks.rados.rados.0.smithi107.stdout:2978: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:47.625 INFO:tasks.rados.rados.0.smithi107.stdout:2978: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:47.625 INFO:tasks.rados.rados.0.smithi107.stdout:2978: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:47.625 INFO:tasks.rados.rados.0.smithi107.stdout:2978: finishing write tid 6 to smithi10729407-3 2020-06-12T02:37:47.626 INFO:tasks.rados.rados.0.smithi107.stdout:2979: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:47.626 INFO:tasks.rados.rados.0.smithi107.stdout:2978: finishing write tid 7 to smithi10729407-3 2020-06-12T02:37:47.626 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 512 (ObjNum 2977 snap 0 seq_num 2977) dirty exists 2020-06-12T02:37:47.626 INFO:tasks.rados.rados.0.smithi107.stdout:2978: left oid 3 (ObjNum 2977 snap 0 seq_num 2977) 2020-06-12T02:37:47.626 INFO:tasks.rados.rados.0.smithi107.stdout:2978: done (2 left) 2020-06-12T02:37:47.627 INFO:tasks.rados.rados.0.smithi107.stdout:2980: done (1 left) 2020-06-12T02:37:47.627 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:37:47.628 INFO:tasks.rados.rados.0.smithi107.stdout:2981: seq_num 2980 ranges {0=712704,712704=737280,1449984=466944} 2020-06-12T02:37:47.628 INFO:tasks.rados.rados.0.smithi107.stdout:2981: writing smithi10729407-10 from 0 to 712704 tid 1 2020-06-12T02:37:47.628 INFO:tasks.rados.rados.0.smithi107.stdout:2981: writing smithi10729407-10 from 712704 to 1449984 tid 2 2020-06-12T02:37:47.628 INFO:tasks.rados.rados.0.smithi107.stdout:2981: writing smithi10729407-10 from 1449984 to 1916928 tid 3 2020-06-12T02:37:47.628 INFO:tasks.rados.rados.0.smithi107.stdout:2982: delete oid 5 current snap is 0 2020-06-12T02:37:47.629 INFO:tasks.rados.rados.0.smithi107.stdout:2979: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:47.629 INFO:tasks.rados.rados.0.smithi107.stdout:2979: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:47.629 INFO:tasks.rados.rados.0.smithi107.stdout:2979: finishing write tid 5 to smithi10729407-18 2020-06-12T02:37:47.629 INFO:tasks.rados.rados.0.smithi107.stdout:2979: finishing write tid 6 to smithi10729407-18 2020-06-12T02:37:47.629 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 535 (ObjNum 2978 snap 0 seq_num 2978) dirty exists 2020-06-12T02:37:47.630 INFO:tasks.rados.rados.0.smithi107.stdout:2979: left oid 18 (ObjNum 2978 snap 0 seq_num 2978) 2020-06-12T02:37:47.630 INFO:tasks.rados.rados.0.smithi107.stdout:2981: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:47.630 INFO:tasks.rados.rados.0.smithi107.stdout:2979: done (2 left) 2020-06-12T02:37:47.630 INFO:tasks.rados.rados.0.smithi107.stdout:2982: done (1 left) 2020-06-12T02:37:47.630 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:37:47.631 INFO:tasks.rados.rados.0.smithi107.stdout:2983: seq_num 2982 ranges {0=745472,745472=557056,1302528=647168} 2020-06-12T02:37:47.631 INFO:tasks.rados.rados.0.smithi107.stdout:2983: writing smithi10729407-7 from 0 to 745472 tid 1 2020-06-12T02:37:47.631 INFO:tasks.rados.rados.0.smithi107.stdout:2983: writing smithi10729407-7 from 745472 to 1302528 tid 2 2020-06-12T02:37:47.631 INFO:tasks.rados.rados.0.smithi107.stdout:2983: writing smithi10729407-7 from 1302528 to 1949696 tid 3 2020-06-12T02:37:47.631 INFO:tasks.rados.rados.0.smithi107.stdout:2984: delete oid 11 current snap is 0 2020-06-12T02:37:47.632 INFO:tasks.rados.rados.0.smithi107.stdout:2981: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:47.632 INFO:tasks.rados.rados.0.smithi107.stdout:2981: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:47.632 INFO:tasks.rados.rados.0.smithi107.stdout:2981: finishing write tid 5 to smithi10729407-10 2020-06-12T02:37:47.632 INFO:tasks.rados.rados.0.smithi107.stdout:2981: finishing write tid 6 to smithi10729407-10 2020-06-12T02:37:47.632 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 287 (ObjNum 2980 snap 0 seq_num 2980) dirty exists 2020-06-12T02:37:47.632 INFO:tasks.rados.rados.0.smithi107.stdout:2981: left oid 10 (ObjNum 2980 snap 0 seq_num 2980) 2020-06-12T02:37:47.633 INFO:tasks.rados.rados.0.smithi107.stdout:2983: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:47.633 INFO:tasks.rados.rados.0.smithi107.stdout:2981: done (2 left) 2020-06-12T02:37:47.633 INFO:tasks.rados.rados.0.smithi107.stdout:2984: done (1 left) 2020-06-12T02:37:47.633 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:37:47.634 INFO:tasks.rados.rados.0.smithi107.stdout:2985: seq_num 2984 ranges {0=786432,786432=729088,1515520=475136} 2020-06-12T02:37:47.635 INFO:tasks.rados.rados.0.smithi107.stdout:2985: writing smithi10729407-12 from 0 to 786432 tid 1 2020-06-12T02:37:47.636 INFO:tasks.rados.rados.0.smithi107.stdout:2985: writing smithi10729407-12 from 786432 to 1515520 tid 2 2020-06-12T02:37:47.636 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:37:47.608+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:37:47.639 INFO:tasks.rados.rados.0.smithi107.stdout:2985: writing smithi10729407-12 from 1515520 to 1990656 tid 3 2020-06-12T02:37:47.639 INFO:tasks.rados.rados.0.smithi107.stdout:2986: delete oid 4 current snap is 0 2020-06-12T02:37:47.639 INFO:tasks.rados.rados.0.smithi107.stdout:2983: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:47.640 INFO:tasks.rados.rados.0.smithi107.stdout:2983: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:47.640 INFO:tasks.rados.rados.0.smithi107.stdout:2983: finishing write tid 5 to smithi10729407-7 2020-06-12T02:37:47.640 INFO:tasks.rados.rados.0.smithi107.stdout:2983: finishing write tid 6 to smithi10729407-7 2020-06-12T02:37:47.640 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 577 (ObjNum 2982 snap 0 seq_num 2982) dirty exists 2020-06-12T02:37:47.640 INFO:tasks.rados.rados.0.smithi107.stdout:2983: left oid 7 (ObjNum 2982 snap 0 seq_num 2982) 2020-06-12T02:37:47.640 INFO:tasks.rados.rados.0.smithi107.stdout:2985: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:47.641 INFO:tasks.rados.rados.0.smithi107.stdout:2985: finishing write tid 2 to smithi10729407-12 2020-06-12T02:37:47.641 INFO:tasks.rados.rados.0.smithi107.stdout:2983: done (2 left) 2020-06-12T02:37:47.641 INFO:tasks.rados.rados.0.smithi107.stdout:2986: done (1 left) 2020-06-12T02:37:47.641 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:37:47.641 INFO:tasks.rados.rados.0.smithi107.stdout:2987: seq_num 2986 ranges {0=417792,417792=499712,917504=630784,1548288=475136} 2020-06-12T02:37:47.642 INFO:tasks.rados.rados.0.smithi107.stdout:2987: writing smithi10729407-15 from 0 to 417792 tid 1 2020-06-12T02:37:47.642 INFO:tasks.rados.rados.0.smithi107.stdout:2987: writing smithi10729407-15 from 417792 to 917504 tid 2 2020-06-12T02:37:47.642 INFO:tasks.rados.rados.0.smithi107.stdout:2987: writing smithi10729407-15 from 917504 to 1548288 tid 3 2020-06-12T02:37:47.642 INFO:tasks.rados.rados.0.smithi107.stdout:2987: writing smithi10729407-15 from 1548288 to 2023424 tid 4 2020-06-12T02:37:47.642 INFO:tasks.rados.rados.0.smithi107.stdout:2988: delete oid 4 current snap is 0 2020-06-12T02:37:47.642 INFO:tasks.rados.rados.0.smithi107.stdout:2985: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:47.643 INFO:tasks.rados.rados.0.smithi107.stdout:2985: finishing write tid 5 to smithi10729407-12 2020-06-12T02:37:47.643 INFO:tasks.rados.rados.0.smithi107.stdout:2987: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:47.643 INFO:tasks.rados.rados.0.smithi107.stdout:2985: finishing write tid 6 to smithi10729407-12 2020-06-12T02:37:47.643 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 485 (ObjNum 2984 snap 0 seq_num 2984) dirty exists 2020-06-12T02:37:47.643 INFO:tasks.rados.rados.0.smithi107.stdout:2985: left oid 12 (ObjNum 2984 snap 0 seq_num 2984) 2020-06-12T02:37:47.644 INFO:tasks.rados.rados.0.smithi107.stdout:2987: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:47.645 INFO:tasks.rados.rados.0.smithi107.stdout:2985: done (2 left) 2020-06-12T02:37:47.645 INFO:tasks.rados.rados.0.smithi107.stdout:2988: done (1 left) 2020-06-12T02:37:47.645 INFO:tasks.rados.rados.0.smithi107.stdout:2989: delete oid 10 current snap is 0 2020-06-12T02:37:47.645 INFO:tasks.rados.rados.0.smithi107.stdout:2989: done (1 left) 2020-06-12T02:37:47.645 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:37:47.646 INFO:tasks.rados.rados.0.smithi107.stdout:2990: seq_num 2989 ranges {1687552=466944,2154496=598016,2752512=802816,3555328=204800} 2020-06-12T02:37:47.655 INFO:tasks.rados.rados.0.smithi107.stdout:2990: writing smithi10729407-17 from 1687552 to 2154496 tid 1 2020-06-12T02:37:47.659 INFO:tasks.rados.rados.0.smithi107.stdout:2990: writing smithi10729407-17 from 2154496 to 2752512 tid 2 2020-06-12T02:37:47.664 INFO:tasks.rados.rados.0.smithi107.stdout:2990: writing smithi10729407-17 from 2752512 to 3555328 tid 3 2020-06-12T02:37:47.666 INFO:tasks.rados.rados.0.smithi107.stdout:2990: writing smithi10729407-17 from 3555328 to 3760128 tid 4 2020-06-12T02:37:47.666 INFO:tasks.rados.rados.0.smithi107.stdout:2991: delete oid 14 current snap is 0 2020-06-12T02:37:47.666 INFO:tasks.rados.rados.0.smithi107.stdout:2987: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:47.667 INFO:tasks.rados.rados.0.smithi107.stdout:2987: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:47.667 INFO:tasks.rados.rados.0.smithi107.stdout:2987: finishing write tid 6 to smithi10729407-15 2020-06-12T02:37:47.667 INFO:tasks.rados.rados.0.smithi107.stdout:2987: finishing write tid 7 to smithi10729407-15 2020-06-12T02:37:47.667 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 511 (ObjNum 2986 snap 0 seq_num 2986) dirty exists 2020-06-12T02:37:47.667 INFO:tasks.rados.rados.0.smithi107.stdout:2987: left oid 15 (ObjNum 2986 snap 0 seq_num 2986) 2020-06-12T02:37:47.668 INFO:tasks.rados.rados.0.smithi107.stdout:2990: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:47.668 INFO:tasks.rados.rados.0.smithi107.stdout:2990: finishing write tid 2 to smithi10729407-17 2020-06-12T02:37:47.668 INFO:tasks.rados.rados.0.smithi107.stdout:2987: done (2 left) 2020-06-12T02:37:47.669 INFO:tasks.rados.rados.0.smithi107.stdout:2991: done (1 left) 2020-06-12T02:37:47.669 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:37:47.669 INFO:tasks.rados.rados.0.smithi107.stdout:2992: seq_num 2991 ranges {0=499712,499712=770048,1269760=417792,1687552=417792} 2020-06-12T02:37:47.671 INFO:tasks.rados.rados.0.smithi107.stdout:2992: writing smithi10729407-4 from 0 to 499712 tid 1 2020-06-12T02:37:47.676 INFO:tasks.rados.rados.0.smithi107.stdout:2992: writing smithi10729407-4 from 499712 to 1269760 tid 2 2020-06-12T02:37:47.678 INFO:tasks.rados.rados.0.smithi107.stdout:2992: writing smithi10729407-4 from 1269760 to 1687552 tid 3 2020-06-12T02:37:47.682 INFO:tasks.rados.rados.0.smithi107.stdout:2992: writing smithi10729407-4 from 1687552 to 2105344 tid 4 2020-06-12T02:37:47.682 INFO:tasks.rados.rados.0.smithi107.stdout:2993: delete oid 19 current snap is 0 2020-06-12T02:37:47.682 INFO:tasks.rados.rados.0.smithi107.stdout:2990: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:47.682 INFO:tasks.rados.rados.0.smithi107.stdout:2990: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:47.683 INFO:tasks.rados.rados.0.smithi107.stdout:2990: finishing write tid 6 to smithi10729407-17 2020-06-12T02:37:47.683 INFO:tasks.rados.rados.0.smithi107.stdout:2992: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:47.683 INFO:tasks.rados.rados.0.smithi107.stdout:2990: finishing write tid 7 to smithi10729407-17 2020-06-12T02:37:47.683 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 573 (ObjNum 2989 snap 0 seq_num 2989) dirty exists 2020-06-12T02:37:47.683 INFO:tasks.rados.rados.0.smithi107.stdout:2990: left oid 17 (ObjNum 2989 snap 0 seq_num 2989) 2020-06-12T02:37:47.684 INFO:tasks.rados.rados.0.smithi107.stdout:2990: done (2 left) 2020-06-12T02:37:47.684 INFO:tasks.rados.rados.0.smithi107.stdout:2993: done (1 left) 2020-06-12T02:37:47.685 INFO:tasks.rados.rados.0.smithi107.stdout:2994: delete oid 1 current snap is 0 2020-06-12T02:37:47.687 INFO:tasks.rados.rados.0.smithi107.stdout:2992: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:47.687 INFO:tasks.rados.rados.0.smithi107.stdout:2992: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:47.688 INFO:tasks.rados.rados.0.smithi107.stdout:2994: done (1 left) 2020-06-12T02:37:47.688 INFO:tasks.rados.rados.0.smithi107.stdout:2995: delete oid 9 current snap is 0 2020-06-12T02:37:47.691 INFO:tasks.rados.rados.0.smithi107.stdout:2992: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:47.691 INFO:tasks.rados.rados.0.smithi107.stdout:2992: finishing write tid 6 to smithi10729407-4 2020-06-12T02:37:47.692 INFO:tasks.rados.rados.0.smithi107.stdout:2992: finishing write tid 7 to smithi10729407-4 2020-06-12T02:37:47.692 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 578 (ObjNum 2991 snap 0 seq_num 2991) dirty exists 2020-06-12T02:37:47.692 INFO:tasks.rados.rados.0.smithi107.stdout:2992: left oid 4 (ObjNum 2991 snap 0 seq_num 2991) 2020-06-12T02:37:47.693 INFO:tasks.rados.rados.0.smithi107.stdout:2992: done (1 left) 2020-06-12T02:37:47.693 INFO:tasks.rados.rados.0.smithi107.stdout:2995: done (0 left) 2020-06-12T02:37:47.693 INFO:tasks.rados.rados.0.smithi107.stdout:2996: delete oid 4 current snap is 0 2020-06-12T02:37:47.694 INFO:tasks.rados.rados.0.smithi107.stdout:2996: done (0 left) 2020-06-12T02:37:47.694 INFO:tasks.rados.rados.0.smithi107.stdout:2997: delete oid 15 current snap is 0 2020-06-12T02:37:47.698 INFO:tasks.rados.rados.0.smithi107.stdout:2997: done (0 left) 2020-06-12T02:37:47.698 INFO:tasks.rados.rados.0.smithi107.stdout:2998: delete oid 20 current snap is 0 2020-06-12T02:37:47.700 INFO:tasks.rados.rados.0.smithi107.stdout:2998: done (0 left) 2020-06-12T02:37:47.700 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:37:47.700 INFO:tasks.rados.rados.0.smithi107.stdout:2999: seq_num 2998 ranges {1884160=614400,2498560=409600,2908160=565248,3473408=425984,3899392=204800} 2020-06-12T02:37:47.711 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:37:47.709+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:37:47.714 INFO:tasks.rados.rados.0.smithi107.stdout:2999: writing smithi10729407-18 from 1884160 to 2498560 tid 1 2020-06-12T02:37:47.717 INFO:tasks.rados.rados.0.smithi107.stdout:2999: writing smithi10729407-18 from 2498560 to 2908160 tid 2 2020-06-12T02:37:47.726 INFO:tasks.rados.rados.0.smithi107.stdout:2999: writing smithi10729407-18 from 2908160 to 3473408 tid 3 2020-06-12T02:37:47.726 INFO:tasks.rados.rados.0.smithi107.stdout:2999: writing smithi10729407-18 from 3473408 to 3899392 tid 4 2020-06-12T02:37:47.726 INFO:tasks.rados.rados.0.smithi107.stdout:2999: writing smithi10729407-18 from 3899392 to 4104192 tid 5 2020-06-12T02:37:47.727 INFO:tasks.rados.rados.0.smithi107.stdout:3000: delete oid 3 current snap is 0 2020-06-12T02:37:47.727 INFO:tasks.rados.rados.0.smithi107.stdout:2999: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:47.727 INFO:tasks.rados.rados.0.smithi107.stdout:2999: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:47.728 INFO:tasks.rados.rados.0.smithi107.stdout:2999: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:47.728 INFO:tasks.rados.rados.0.smithi107.stdout:3000: done (1 left) 2020-06-12T02:37:47.728 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:37:47.728 INFO:tasks.rados.rados.0.smithi107.stdout:3001: seq_num 3000 ranges {0=647168,647168=573440,1220608=532480,1753088=499712} 2020-06-12T02:37:47.734 INFO:tasks.rados.rados.0.smithi107.stdout:3001: writing smithi10729407-3 from 0 to 647168 tid 1 2020-06-12T02:37:47.735 INFO:tasks.rados.rados.0.smithi107.stdout:3001: writing smithi10729407-3 from 647168 to 1220608 tid 2 2020-06-12T02:37:47.738 INFO:tasks.rados.rados.0.smithi107.stdout:3001: writing smithi10729407-3 from 1220608 to 1753088 tid 3 2020-06-12T02:37:47.742 INFO:tasks.rados.rados.0.smithi107.stdout:3001: writing smithi10729407-3 from 1753088 to 2252800 tid 4 2020-06-12T02:37:47.742 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:37:47.742 INFO:tasks.rados.rados.0.smithi107.stdout:3002: seq_num 3001 ranges {0=663552,663552=458752,1122304=745472,1867776=401408} 2020-06-12T02:37:47.745 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:37:47.746 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:37:47.746 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2020-06-12T02:37:47.746 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:37:47.746 INFO:tasks.rados.rados.0.smithi107.stdout:3002: writing smithi10729407-20 from 0 to 663552 tid 1 2020-06-12T02:37:47.749 INFO:tasks.rados.rados.0.smithi107.stdout:3002: writing smithi10729407-20 from 663552 to 1122304 tid 2 2020-06-12T02:37:47.754 INFO:tasks.rados.rados.0.smithi107.stdout:3002: writing smithi10729407-20 from 1122304 to 1867776 tid 3 2020-06-12T02:37:47.757 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-06-12T02:37:47.758 INFO:tasks.rados.rados.0.smithi107.stdout:3002: writing smithi10729407-20 from 1867776 to 2269184 tid 4 2020-06-12T02:37:47.758 INFO:tasks.rados.rados.0.smithi107.stdout:3003: delete oid 4 current snap is 0 2020-06-12T02:37:47.758 INFO:tasks.rados.rados.0.smithi107.stdout:2999: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:47.759 INFO:tasks.rados.rados.0.smithi107.stdout:2999: finishing write tid 5 to smithi10729407-18 2020-06-12T02:37:47.759 INFO:tasks.rados.rados.0.smithi107.stdout:2999: finishing write tid 7 to smithi10729407-18 2020-06-12T02:37:47.759 INFO:tasks.rados.rados.0.smithi107.stdout:3001: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:47.759 INFO:tasks.rados.rados.0.smithi107.stdout:2999: finishing write tid 8 to smithi10729407-18 2020-06-12T02:37:47.759 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 542 (ObjNum 2998 snap 0 seq_num 2998) dirty exists 2020-06-12T02:37:47.760 INFO:tasks.rados.rados.0.smithi107.stdout:2999: left oid 18 (ObjNum 2998 snap 0 seq_num 2998) 2020-06-12T02:37:47.760 INFO:tasks.rados.rados.0.smithi107.stdout:3001: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:47.760 INFO:tasks.rados.rados.0.smithi107.stdout:3001: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:47.760 INFO:tasks.rados.rados.0.smithi107.stdout:3001: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:47.760 INFO:tasks.rados.rados.0.smithi107.stdout:3001: finishing write tid 6 to smithi10729407-3 2020-06-12T02:37:47.761 INFO:tasks.rados.rados.0.smithi107.stdout:3001: finishing write tid 7 to smithi10729407-3 2020-06-12T02:37:47.761 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 518 (ObjNum 3000 snap 0 seq_num 3000) dirty exists 2020-06-12T02:37:47.761 INFO:tasks.rados.rados.0.smithi107.stdout:3001: left oid 3 (ObjNum 3000 snap 0 seq_num 3000) 2020-06-12T02:37:47.762 INFO:tasks.rados.rados.0.smithi107.stdout:2999: done (3 left) 2020-06-12T02:37:47.762 INFO:tasks.rados.rados.0.smithi107.stdout:3001: done (2 left) 2020-06-12T02:37:47.762 INFO:tasks.rados.rados.0.smithi107.stdout:3003: done (1 left) 2020-06-12T02:37:47.762 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:37:47.763 INFO:tasks.rados.rados.0.smithi107.stdout:3004: seq_num 3003 ranges {4104192=704512,4808704=679936,5488640=753664,6242304=172032} 2020-06-12T02:37:47.786 INFO:tasks.rados.rados.0.smithi107.stdout:3004: writing smithi10729407-18 from 4104192 to 4808704 tid 1 2020-06-12T02:37:47.789 INFO:tasks.rados.rados.0.smithi107.stdout:3004: writing smithi10729407-18 from 4808704 to 5488640 tid 2 2020-06-12T02:37:47.795 INFO:tasks.rados.rados.0.smithi107.stdout:3004: writing smithi10729407-18 from 5488640 to 6242304 tid 3 2020-06-12T02:37:47.796 INFO:tasks.rados.rados.0.smithi107.stdout:3004: writing smithi10729407-18 from 6242304 to 6414336 tid 4 2020-06-12T02:37:47.797 INFO:tasks.rados.rados.0.smithi107.stdout:3005: delete oid 7 current snap is 0 2020-06-12T02:37:47.797 INFO:tasks.rados.rados.0.smithi107.stdout:3002: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:47.797 INFO:tasks.rados.rados.0.smithi107.stdout:3002: finishing write tid 2 to smithi10729407-20 2020-06-12T02:37:47.797 INFO:tasks.rados.rados.0.smithi107.stdout:3002: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:47.798 INFO:tasks.rados.rados.0.smithi107.stdout:3002: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:47.798 INFO:tasks.rados.rados.0.smithi107.stdout:3002: finishing write tid 6 to smithi10729407-20 2020-06-12T02:37:47.798 INFO:tasks.rados.rados.0.smithi107.stdout:3002: finishing write tid 7 to smithi10729407-20 2020-06-12T02:37:47.798 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 207 (ObjNum 3001 snap 0 seq_num 3001) dirty exists 2020-06-12T02:37:47.798 INFO:tasks.rados.rados.0.smithi107.stdout:3002: left oid 20 (ObjNum 3001 snap 0 seq_num 3001) 2020-06-12T02:37:47.799 INFO:tasks.rados.rados.0.smithi107.stdout:3004: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:47.799 INFO:tasks.rados.rados.0.smithi107.stdout:3002: done (2 left) 2020-06-12T02:37:47.799 INFO:tasks.rados.rados.0.smithi107.stdout:3005: done (1 left) 2020-06-12T02:37:47.799 INFO:tasks.rados.rados.0.smithi107.stdout:3006: delete oid 9 current snap is 0 2020-06-12T02:37:47.805 INFO:tasks.rados.rados.0.smithi107.stdout:3004: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:47.805 INFO:tasks.rados.rados.0.smithi107.stdout:3004: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:47.806 INFO:tasks.rados.rados.0.smithi107.stdout:3004: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:47.806 INFO:tasks.rados.rados.0.smithi107.stdout:3004: finishing write tid 6 to smithi10729407-18 2020-06-12T02:37:47.806 INFO:tasks.rados.rados.0.smithi107.stdout:3006: done (1 left) 2020-06-12T02:37:47.807 INFO:tasks.rados.rados.0.smithi107.stdout:3007: delete oid 19 current snap is 0 2020-06-12T02:37:47.807 INFO:tasks.rados.rados.0.smithi107.stdout:3004: finishing write tid 7 to smithi10729407-18 2020-06-12T02:37:47.807 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 547 (ObjNum 3003 snap 0 seq_num 3003) dirty exists 2020-06-12T02:37:47.807 INFO:tasks.rados.rados.0.smithi107.stdout:3004: left oid 18 (ObjNum 3003 snap 0 seq_num 3003) 2020-06-12T02:37:47.808 INFO:tasks.rados.rados.0.smithi107.stdout:3004: done (1 left) 2020-06-12T02:37:47.808 INFO:tasks.rados.rados.0.smithi107.stdout:3007: done (0 left) 2020-06-12T02:37:47.808 INFO:tasks.rados.rados.0.smithi107.stdout:3008: delete oid 13 current snap is 0 2020-06-12T02:37:47.810 INFO:tasks.rados.rados.0.smithi107.stdout:3008: done (0 left) 2020-06-12T02:37:47.810 INFO:tasks.rados.rados.0.smithi107.stdout:3009: delete oid 10 current snap is 0 2020-06-12T02:37:47.812 INFO:tasks.rados.rados.0.smithi107.stdout:3009: done (0 left) 2020-06-12T02:37:47.812 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:37:47.812 INFO:tasks.rados.rados.0.smithi107.stdout:3010: seq_num 3009 ranges {0=802816,802816=786432,1589248=704512,2293760=114688} 2020-06-12T02:37:47.813 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:37:47.810+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:37:47.816 INFO:tasks.rados.rados.0.smithi107.stdout:3010: writing smithi10729407-1 from 0 to 802816 tid 1 2020-06-12T02:37:47.821 INFO:tasks.rados.rados.0.smithi107.stdout:3010: writing smithi10729407-1 from 802816 to 1589248 tid 2 2020-06-12T02:37:47.825 INFO:tasks.rados.rados.0.smithi107.stdout:3010: writing smithi10729407-1 from 1589248 to 2293760 tid 3 2020-06-12T02:37:47.826 INFO:tasks.rados.rados.0.smithi107.stdout:3010: writing smithi10729407-1 from 2293760 to 2408448 tid 4 2020-06-12T02:37:47.826 INFO:tasks.rados.rados.0.smithi107.stdout:3011: delete oid 19 current snap is 0 2020-06-12T02:37:47.827 INFO:tasks.rados.rados.0.smithi107.stdout:3010: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:47.828 INFO:tasks.rados.rados.0.smithi107.stdout:3011: done (1 left) 2020-06-12T02:37:47.828 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:37:47.829 INFO:tasks.rados.rados.0.smithi107.stdout:3012: seq_num 3011 ranges {0=32768} 2020-06-12T02:37:47.829 INFO:tasks.rados.rados.0.smithi107.stdout:3012: writing smithi10729407-15 from 0 to 32768 tid 1 2020-06-12T02:37:47.829 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:37:47.829 INFO:tasks.rados.rados.0.smithi107.stdout:3013: seq_num 3012 ranges {0=49152} 2020-06-12T02:37:47.829 INFO:tasks.rados.rados.0.smithi107.stdout:3013: writing smithi10729407-5 from 0 to 49152 tid 1 2020-06-12T02:37:47.830 INFO:tasks.rados.rados.0.smithi107.stdout:3014: delete oid 9 current snap is 0 2020-06-12T02:37:47.831 INFO:tasks.rados.rados.0.smithi107.stdout:3014: done (3 left) 2020-06-12T02:37:47.831 INFO:tasks.rados.rados.0.smithi107.stdout:3015: delete oid 17 current snap is 0 2020-06-12T02:37:47.832 INFO:tasks.rados.rados.0.smithi107.stdout:3010: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:47.832 INFO:tasks.rados.rados.0.smithi107.stdout:3013: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:47.833 INFO:tasks.rados.rados.0.smithi107.stdout:3010: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:47.833 INFO:tasks.rados.rados.0.smithi107.stdout:3012: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:47.834 INFO:tasks.rados.rados.0.smithi107.stdout:3012: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:47.834 INFO:tasks.rados.rados.0.smithi107.stdout:3010: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:47.834 INFO:tasks.rados.rados.0.smithi107.stdout:3015: done (3 left) 2020-06-12T02:37:47.835 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:37:47.835 INFO:tasks.rados.rados.0.smithi107.stdout:3016: seq_num 3015 ranges {0=98304} 2020-06-12T02:37:47.836 INFO:tasks.rados.rados.0.smithi107.stdout:3016: writing smithi10729407-10 from 0 to 98304 tid 1 2020-06-12T02:37:47.836 INFO:tasks.rados.rados.0.smithi107.stdout:3017: delete oid 17 current snap is 0 2020-06-12T02:37:47.836 INFO:tasks.rados.rados.0.smithi107.stdout:3012: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:47.836 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 514 (ObjNum 3011 snap 0 seq_num 3011) dirty exists 2020-06-12T02:37:47.836 INFO:tasks.rados.rados.0.smithi107.stdout:3012: left oid 15 (ObjNum 3011 snap 0 seq_num 3011) 2020-06-12T02:37:47.837 INFO:tasks.rados.rados.0.smithi107.stdout:3010: finishing write tid 6 to smithi10729407-1 2020-06-12T02:37:47.837 INFO:tasks.rados.rados.0.smithi107.stdout:3013: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:47.837 INFO:tasks.rados.rados.0.smithi107.stdout:3010: finishing write tid 7 to smithi10729407-1 2020-06-12T02:37:47.838 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 591 (ObjNum 3009 snap 0 seq_num 3009) dirty exists 2020-06-12T02:37:47.838 INFO:tasks.rados.rados.0.smithi107.stdout:3010: left oid 1 (ObjNum 3009 snap 0 seq_num 3009) 2020-06-12T02:37:47.839 INFO:tasks.rados.rados.0.smithi107.stdout:3013: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:47.839 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 261 (ObjNum 3012 snap 0 seq_num 3012) dirty exists 2020-06-12T02:37:47.839 INFO:tasks.rados.rados.0.smithi107.stdout:3013: left oid 5 (ObjNum 3012 snap 0 seq_num 3012) 2020-06-12T02:37:47.839 INFO:tasks.rados.rados.0.smithi107.stdout:3010: done (4 left) 2020-06-12T02:37:47.839 INFO:tasks.rados.rados.0.smithi107.stdout:3012: done (3 left) 2020-06-12T02:37:47.840 INFO:tasks.rados.rados.0.smithi107.stdout:3013: done (2 left) 2020-06-12T02:37:47.840 INFO:tasks.rados.rados.0.smithi107.stdout:3017: done (1 left) 2020-06-12T02:37:47.840 INFO:tasks.rados.rados.0.smithi107.stdout:3018: delete oid 15 current snap is 0 2020-06-12T02:37:47.840 INFO:tasks.rados.rados.0.smithi107.stdout:3016: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:47.845 INFO:tasks.rados.rados.0.smithi107.stdout:3018: done (1 left) 2020-06-12T02:37:47.846 INFO:tasks.rados.rados.0.smithi107.stdout:3019: delete oid 3 current snap is 0 2020-06-12T02:37:47.846 INFO:tasks.rados.rados.0.smithi107.stdout:3016: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:47.846 INFO:tasks.rados.rados.0.smithi107.stdout:3016: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:47.847 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 291 (ObjNum 3015 snap 0 seq_num 3015) dirty exists 2020-06-12T02:37:47.847 INFO:tasks.rados.rados.0.smithi107.stdout:3016: left oid 10 (ObjNum 3015 snap 0 seq_num 3015) 2020-06-12T02:37:47.847 INFO:tasks.rados.rados.0.smithi107.stdout:3016: done (1 left) 2020-06-12T02:37:47.847 INFO:tasks.rados.rados.0.smithi107.stdout:3019: done (0 left) 2020-06-12T02:37:47.848 INFO:tasks.rados.rados.0.smithi107.stdout:3020: delete oid 2 current snap is 0 2020-06-12T02:37:47.849 INFO:tasks.rados.rados.0.smithi107.stdout:3020: done (0 left) 2020-06-12T02:37:47.849 INFO:tasks.rados.rados.0.smithi107.stdout:3021: delete oid 19 current snap is 0 2020-06-12T02:37:47.852 INFO:tasks.rados.rados.0.smithi107.stdout:3021: done (0 left) 2020-06-12T02:37:47.852 INFO:tasks.rados.rados.0.smithi107.stdout:3022: delete oid 11 current snap is 0 2020-06-12T02:37:47.854 INFO:tasks.rados.rados.0.smithi107.stdout:3022: done (0 left) 2020-06-12T02:37:47.854 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:37:47.854 INFO:tasks.rados.rados.0.smithi107.stdout:3023: seq_num 3022 ranges {49152=221184} 2020-06-12T02:37:47.855 INFO:tasks.rados.rados.0.smithi107.stdout:3023: writing smithi10729407-5 from 49152 to 270336 tid 1 2020-06-12T02:37:47.855 INFO:tasks.rados.rados.0.smithi107.stdout:3024: delete oid 16 current snap is 0 2020-06-12T02:37:47.857 INFO:tasks.rados.rados.0.smithi107.stdout:3024: done (1 left) 2020-06-12T02:37:47.858 INFO:tasks.rados.rados.0.smithi107.stdout:3025: delete oid 6 current snap is 0 2020-06-12T02:37:47.858 INFO:tasks.rados.rados.0.smithi107.stdout:3023: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:47.860 INFO:tasks.rados.rados.0.smithi107.stdout:3025: done (1 left) 2020-06-12T02:37:47.860 INFO:tasks.rados.rados.0.smithi107.stdout:3026: delete oid 7 current snap is 0 2020-06-12T02:37:47.862 INFO:tasks.rados.rados.0.smithi107.stdout:3023: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:47.862 INFO:tasks.rados.rados.0.smithi107.stdout:3026: done (1 left) 2020-06-12T02:37:47.862 INFO:tasks.rados.rados.0.smithi107.stdout:3027: delete oid 8 current snap is 0 2020-06-12T02:37:47.863 INFO:tasks.rados.rados.0.smithi107.stdout:3023: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:47.863 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 263 (ObjNum 3022 snap 0 seq_num 3022) dirty exists 2020-06-12T02:37:47.864 INFO:tasks.rados.rados.0.smithi107.stdout:3023: left oid 5 (ObjNum 3022 snap 0 seq_num 3022) 2020-06-12T02:37:47.866 INFO:tasks.rados.rados.0.smithi107.stdout:3023: done (1 left) 2020-06-12T02:37:47.866 INFO:tasks.rados.rados.0.smithi107.stdout:3027: done (0 left) 2020-06-12T02:37:47.866 INFO:tasks.rados.rados.0.smithi107.stdout:3028: delete oid 7 current snap is 0 2020-06-12T02:37:47.868 INFO:tasks.rados.rados.0.smithi107.stdout:3028: done (0 left) 2020-06-12T02:37:47.869 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:37:47.869 INFO:tasks.rados.rados.0.smithi107.stdout:3029: seq_num 3028 ranges {0=319488} 2020-06-12T02:37:47.871 INFO:tasks.rados.rados.0.smithi107.stdout:3029: writing smithi10729407-16 from 0 to 319488 tid 1 2020-06-12T02:37:47.871 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:37:47.872 INFO:tasks.rados.rados.0.smithi107.stdout:3030: seq_num 3029 ranges {0=335872} 2020-06-12T02:37:47.873 INFO:tasks.rados.rados.0.smithi107.stdout:3030: writing smithi10729407-6 from 0 to 335872 tid 1 2020-06-12T02:37:47.873 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:37:47.873 INFO:tasks.rados.rados.0.smithi107.stdout:3031: seq_num 3030 ranges {0=352256} 2020-06-12T02:37:47.877 INFO:tasks.rados.rados.0.smithi107.stdout:3031: writing smithi10729407-8 from 0 to 352256 tid 1 2020-06-12T02:37:47.878 INFO:tasks.rados.rados.0.smithi107.stdout:3032: delete oid 17 current snap is 0 2020-06-12T02:37:47.878 INFO:tasks.rados.rados.0.smithi107.stdout:3029: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:47.878 INFO:tasks.rados.rados.0.smithi107.stdout:3030: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:47.880 INFO:tasks.rados.rados.0.smithi107.stdout:3029: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:47.882 INFO:tasks.rados.rados.0.smithi107.stdout:3029: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:47.883 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 522 (ObjNum 3028 snap 0 seq_num 3028) dirty exists 2020-06-12T02:37:47.883 INFO:tasks.rados.rados.0.smithi107.stdout:3029: left oid 16 (ObjNum 3028 snap 0 seq_num 3028) 2020-06-12T02:37:47.883 INFO:tasks.rados.rados.0.smithi107.stdout:3029: done (3 left) 2020-06-12T02:37:47.884 INFO:tasks.rados.rados.0.smithi107.stdout:3032: done (2 left) 2020-06-12T02:37:47.884 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:37:47.884 INFO:tasks.rados.rados.0.smithi107.stdout:3033: seq_num 3032 ranges {0=385024} 2020-06-12T02:37:47.886 INFO:tasks.rados.rados.0.smithi107.stdout:3033: writing smithi10729407-15 from 0 to 385024 tid 1 2020-06-12T02:37:47.886 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:37:47.886 INFO:tasks.rados.rados.0.smithi107.stdout:3034: seq_num 3033 ranges {98304=401408} 2020-06-12T02:37:47.889 INFO:tasks.rados.rados.0.smithi107.stdout:3034: writing smithi10729407-10 from 98304 to 499712 tid 1 2020-06-12T02:37:47.890 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:37:47.890 INFO:tasks.rados.rados.0.smithi107.stdout:3031: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:47.890 INFO:tasks.rados.rados.0.smithi107.stdout:3035: seq_num 3034 ranges {0=417792} 2020-06-12T02:37:47.892 INFO:tasks.rados.rados.0.smithi107.stdout:3035: writing smithi10729407-2 from 0 to 417792 tid 1 2020-06-12T02:37:47.893 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:37:47.893 INFO:tasks.rados.rados.0.smithi107.stdout:3036: seq_num 3035 ranges {2408448=434176} 2020-06-12T02:37:47.896 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:37:47.896 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:37:47.897 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:37:47.897 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:37:47.897 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:37:47.906 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-06-12T02:37:47.910 INFO:tasks.rados.rados.0.smithi107.stdout:3036: writing smithi10729407-1 from 2408448 to 2842624 tid 1 2020-06-12T02:37:47.910 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:37:47.910 INFO:tasks.rados.rados.0.smithi107.stdout:3037: seq_num 3036 ranges {2269184=450560} 2020-06-12T02:37:47.913 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:37:47.910+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:37:47.925 INFO:tasks.rados.rados.0.smithi107.stdout:3037: writing smithi10729407-20 from 2269184 to 2719744 tid 1 2020-06-12T02:37:47.925 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:37:47.926 INFO:tasks.rados.rados.0.smithi107.stdout:3038: seq_num 3037 ranges {0=466944} 2020-06-12T02:37:47.929 INFO:tasks.rados.rados.0.smithi107.stdout:3038: writing smithi10729407-19 from 0 to 466944 tid 1 2020-06-12T02:37:47.929 INFO:tasks.rados.rados.0.smithi107.stdout: waiting on 8 2020-06-12T02:37:47.929 INFO:tasks.rados.rados.0.smithi107.stdout:3031: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:47.929 INFO:tasks.rados.rados.0.smithi107.stdout:3030: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:47.929 INFO:tasks.rados.rados.0.smithi107.stdout:3030: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:47.930 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 594 (ObjNum 3029 snap 0 seq_num 3029) dirty exists 2020-06-12T02:37:47.930 INFO:tasks.rados.rados.0.smithi107.stdout:3030: left oid 6 (ObjNum 3029 snap 0 seq_num 3029) 2020-06-12T02:37:47.930 INFO:tasks.rados.rados.0.smithi107.stdout:3031: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:47.930 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 518 (ObjNum 3030 snap 0 seq_num 3030) dirty exists 2020-06-12T02:37:47.930 INFO:tasks.rados.rados.0.smithi107.stdout:3031: left oid 8 (ObjNum 3030 snap 0 seq_num 3030) 2020-06-12T02:37:47.931 INFO:tasks.rados.rados.0.smithi107.stdout:3033: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:47.931 INFO:tasks.rados.rados.0.smithi107.stdout:3033: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:47.931 INFO:tasks.rados.rados.0.smithi107.stdout:3034: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:47.931 INFO:tasks.rados.rados.0.smithi107.stdout:3033: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:47.931 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 520 (ObjNum 3032 snap 0 seq_num 3032) dirty exists 2020-06-12T02:37:47.932 INFO:tasks.rados.rados.0.smithi107.stdout:3033: left oid 15 (ObjNum 3032 snap 0 seq_num 3032) 2020-06-12T02:37:47.932 INFO:tasks.rados.rados.0.smithi107.stdout:3034: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:47.932 INFO:tasks.rados.rados.0.smithi107.stdout:3030: done (7 left) 2020-06-12T02:37:47.932 INFO:tasks.rados.rados.0.smithi107.stdout:3031: done (6 left) 2020-06-12T02:37:47.932 INFO:tasks.rados.rados.0.smithi107.stdout:3033: done (5 left) 2020-06-12T02:37:47.933 INFO:tasks.rados.rados.0.smithi107.stdout:3039: delete oid 12 current snap is 0 2020-06-12T02:37:47.933 INFO:tasks.rados.rados.0.smithi107.stdout:3034: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:47.933 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 293 (ObjNum 3033 snap 0 seq_num 3033) dirty exists 2020-06-12T02:37:47.933 INFO:tasks.rados.rados.0.smithi107.stdout:3034: left oid 10 (ObjNum 3033 snap 0 seq_num 3033) 2020-06-12T02:37:47.933 INFO:tasks.rados.rados.0.smithi107.stdout:3035: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:47.934 INFO:tasks.rados.rados.0.smithi107.stdout:3035: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:47.934 INFO:tasks.rados.rados.0.smithi107.stdout:3035: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:47.934 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 310 (ObjNum 3034 snap 0 seq_num 3034) dirty exists 2020-06-12T02:37:47.934 INFO:tasks.rados.rados.0.smithi107.stdout:3035: left oid 2 (ObjNum 3034 snap 0 seq_num 3034) 2020-06-12T02:37:47.934 INFO:tasks.rados.rados.0.smithi107.stdout:3036: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:47.935 INFO:tasks.rados.rados.0.smithi107.stdout:3036: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:47.935 INFO:tasks.rados.rados.0.smithi107.stdout:3036: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:47.935 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 596 (ObjNum 3035 snap 0 seq_num 3035) dirty exists 2020-06-12T02:37:47.935 INFO:tasks.rados.rados.0.smithi107.stdout:3036: left oid 1 (ObjNum 3035 snap 0 seq_num 3035) 2020-06-12T02:37:47.936 INFO:tasks.rados.rados.0.smithi107.stdout:3037: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:47.936 INFO:tasks.rados.rados.0.smithi107.stdout:3038: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:47.936 INFO:tasks.rados.rados.0.smithi107.stdout:3038: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:47.937 INFO:tasks.rados.rados.0.smithi107.stdout:3038: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:47.938 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 491 (ObjNum 3037 snap 0 seq_num 3037) dirty exists 2020-06-12T02:37:47.938 INFO:tasks.rados.rados.0.smithi107.stdout:3038: left oid 19 (ObjNum 3037 snap 0 seq_num 3037) 2020-06-12T02:37:47.938 INFO:tasks.rados.rados.0.smithi107.stdout:3034: done (5 left) 2020-06-12T02:37:47.938 INFO:tasks.rados.rados.0.smithi107.stdout:3035: done (4 left) 2020-06-12T02:37:47.938 INFO:tasks.rados.rados.0.smithi107.stdout:3036: done (3 left) 2020-06-12T02:37:47.939 INFO:tasks.rados.rados.0.smithi107.stdout:3038: done (2 left) 2020-06-12T02:37:47.939 INFO:tasks.rados.rados.0.smithi107.stdout:3039: done (1 left) 2020-06-12T02:37:47.939 INFO:tasks.rados.rados.0.smithi107.stdout:3040: delete oid 2 current snap is 0 2020-06-12T02:37:47.939 INFO:tasks.rados.rados.0.smithi107.stdout:3037: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:47.940 INFO:tasks.rados.rados.0.smithi107.stdout:3037: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:47.940 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 209 (ObjNum 3036 snap 0 seq_num 3036) dirty exists 2020-06-12T02:37:47.940 INFO:tasks.rados.rados.0.smithi107.stdout:3037: left oid 20 (ObjNum 3036 snap 0 seq_num 3036) 2020-06-12T02:37:47.940 INFO:tasks.rados.rados.0.smithi107.stdout:3037: done (1 left) 2020-06-12T02:37:47.941 INFO:tasks.rados.rados.0.smithi107.stdout:3040: done (0 left) 2020-06-12T02:37:47.941 INFO:tasks.rados.rados.0.smithi107.stdout:3041: delete oid 4 current snap is 0 2020-06-12T02:37:47.942 INFO:tasks.rados.rados.0.smithi107.stdout:3041: done (0 left) 2020-06-12T02:37:47.943 INFO:tasks.rados.rados.0.smithi107.stdout:3042: delete oid 3 current snap is 0 2020-06-12T02:37:47.945 INFO:tasks.rados.rados.0.smithi107.stdout:3042: done (0 left) 2020-06-12T02:37:47.946 INFO:tasks.rados.rados.0.smithi107.stdout:3043: delete oid 10 current snap is 0 2020-06-12T02:37:47.947 INFO:tasks.rados.rados.0.smithi107.stdout:3043: done (0 left) 2020-06-12T02:37:47.947 INFO:tasks.rados.rados.0.smithi107.stdout:3044: delete oid 12 current snap is 0 2020-06-12T02:37:47.951 INFO:tasks.rados.rados.0.smithi107.stdout:3044: done (0 left) 2020-06-12T02:37:47.951 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:37:47.951 INFO:tasks.rados.rados.0.smithi107.stdout:3045: seq_num 3044 ranges {0=589824} 2020-06-12T02:37:47.954 INFO:tasks.rados.rados.0.smithi107.stdout:3045: writing smithi10729407-10 from 0 to 589824 tid 1 2020-06-12T02:37:47.954 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:37:47.955 INFO:tasks.rados.rados.0.smithi107.stdout:3046: seq_num 3045 ranges {466944=606208} 2020-06-12T02:37:47.961 INFO:tasks.rados.rados.0.smithi107.stdout:3046: writing smithi10729407-19 from 466944 to 1073152 tid 1 2020-06-12T02:37:47.962 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:37:47.962 INFO:tasks.rados.rados.0.smithi107.stdout:3047: seq_num 3046 ranges {0=622592} 2020-06-12T02:37:47.964 INFO:tasks.rados.rados.0.smithi107.stdout:3047: writing smithi10729407-13 from 0 to 622592 tid 1 2020-06-12T02:37:47.965 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:37:47.965 INFO:tasks.rados.rados.0.smithi107.stdout:3048: seq_num 3047 ranges {0=638976} 2020-06-12T02:37:47.969 INFO:tasks.rados.rados.0.smithi107.stdout:3048: writing smithi10729407-11 from 0 to 638976 tid 1 2020-06-12T02:37:47.969 INFO:tasks.rados.rados.0.smithi107.stdout:3049: delete oid 8 current snap is 0 2020-06-12T02:37:47.969 INFO:tasks.rados.rados.0.smithi107.stdout:3045: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:47.969 INFO:tasks.rados.rados.0.smithi107.stdout:3045: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:47.970 INFO:tasks.rados.rados.0.smithi107.stdout:3045: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:47.970 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 296 (ObjNum 3044 snap 0 seq_num 3044) dirty exists 2020-06-12T02:37:47.970 INFO:tasks.rados.rados.0.smithi107.stdout:3045: left oid 10 (ObjNum 3044 snap 0 seq_num 3044) 2020-06-12T02:37:47.970 INFO:tasks.rados.rados.0.smithi107.stdout:3046: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:47.971 INFO:tasks.rados.rados.0.smithi107.stdout:3047: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:47.974 INFO:tasks.rados.rados.0.smithi107.stdout:3045: done (4 left) 2020-06-12T02:37:47.974 INFO:tasks.rados.rados.0.smithi107.stdout:3049: done (3 left) 2020-06-12T02:37:47.974 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:37:47.975 INFO:tasks.rados.rados.0.smithi107.stdout:3050: seq_num 3049 ranges {319488=671744} 2020-06-12T02:37:47.980 INFO:tasks.rados.rados.0.smithi107.stdout:3050: writing smithi10729407-16 from 319488 to 991232 tid 1 2020-06-12T02:37:47.980 INFO:tasks.rados.rados.0.smithi107.stdout:3051: delete oid 17 current snap is 0 2020-06-12T02:37:47.980 INFO:tasks.rados.rados.0.smithi107.stdout:3047: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:47.981 INFO:tasks.rados.rados.0.smithi107.stdout:3048: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:47.981 INFO:tasks.rados.rados.0.smithi107.stdout:3046: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:47.981 INFO:tasks.rados.rados.0.smithi107.stdout:3047: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:47.981 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 583 (ObjNum 3046 snap 0 seq_num 3046) dirty exists 2020-06-12T02:37:47.982 INFO:tasks.rados.rados.0.smithi107.stdout:3047: left oid 13 (ObjNum 3046 snap 0 seq_num 3046) 2020-06-12T02:37:47.982 INFO:tasks.rados.rados.0.smithi107.stdout:3048: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:47.982 INFO:tasks.rados.rados.0.smithi107.stdout:3046: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:47.982 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 495 (ObjNum 3045 snap 0 seq_num 3045) dirty exists 2020-06-12T02:37:47.982 INFO:tasks.rados.rados.0.smithi107.stdout:3046: left oid 19 (ObjNum 3045 snap 0 seq_num 3045) 2020-06-12T02:37:47.982 INFO:tasks.rados.rados.0.smithi107.stdout:3048: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:47.983 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 508 (ObjNum 3047 snap 0 seq_num 3047) dirty exists 2020-06-12T02:37:47.983 INFO:tasks.rados.rados.0.smithi107.stdout:3048: left oid 11 (ObjNum 3047 snap 0 seq_num 3047) 2020-06-12T02:37:47.984 INFO:tasks.rados.rados.0.smithi107.stdout:3046: done (4 left) 2020-06-12T02:37:47.984 INFO:tasks.rados.rados.0.smithi107.stdout:3047: done (3 left) 2020-06-12T02:37:47.984 INFO:tasks.rados.rados.0.smithi107.stdout:3048: done (2 left) 2020-06-12T02:37:47.985 INFO:tasks.rados.rados.0.smithi107.stdout:3051: done (1 left) 2020-06-12T02:37:47.985 INFO:tasks.rados.rados.0.smithi107.stdout:3052: delete oid 17 current snap is 0 2020-06-12T02:37:47.985 INFO:tasks.rados.rados.0.smithi107.stdout:3052: done (1 left) 2020-06-12T02:37:47.985 INFO:tasks.rados.rados.0.smithi107.stdout:3053: delete oid 6 current snap is 0 2020-06-12T02:37:47.987 INFO:tasks.rados.rados.0.smithi107.stdout:3050: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:47.988 INFO:tasks.rados.rados.0.smithi107.stdout:3053: done (1 left) 2020-06-12T02:37:47.988 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:37:47.988 INFO:tasks.rados.rados.0.smithi107.stdout:3054: seq_num 3053 ranges {0=737280} 2020-06-12T02:37:47.991 INFO:tasks.rados.rados.0.smithi107.stdout:3054: writing smithi10729407-17 from 0 to 737280 tid 1 2020-06-12T02:37:47.991 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:37:47.991 INFO:tasks.rados.rados.0.smithi107.stdout:3055: seq_num 3054 ranges {0=753664} 2020-06-12T02:37:47.995 INFO:tasks.rados.rados.0.smithi107.stdout:3055: writing smithi10729407-9 from 0 to 753664 tid 1 2020-06-12T02:37:47.996 INFO:tasks.rados.rados.0.smithi107.stdout:3056: delete oid 5 current snap is 0 2020-06-12T02:37:47.996 INFO:tasks.rados.rados.0.smithi107.stdout:3050: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:47.996 INFO:tasks.rados.rados.0.smithi107.stdout:3050: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:47.996 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 525 (ObjNum 3049 snap 0 seq_num 3049) dirty exists 2020-06-12T02:37:47.997 INFO:tasks.rados.rados.0.smithi107.stdout:3050: left oid 16 (ObjNum 3049 snap 0 seq_num 3049) 2020-06-12T02:37:47.997 INFO:tasks.rados.rados.0.smithi107.stdout:3054: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:47.999 INFO:tasks.rados.rados.0.smithi107.stdout:3054: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:47.999 INFO:tasks.rados.rados.0.smithi107.stdout:3050: done (3 left) 2020-06-12T02:37:48.000 INFO:tasks.rados.rados.0.smithi107.stdout:3056: done (2 left) 2020-06-12T02:37:48.000 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:37:48.000 INFO:tasks.rados.rados.0.smithi107.stdout:3057: seq_num 3056 ranges {589824=786432} 2020-06-12T02:37:48.008 INFO:tasks.rados.rados.0.smithi107.stdout:3057: writing smithi10729407-10 from 589824 to 1376256 tid 1 2020-06-12T02:37:48.008 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:37:48.008 INFO:tasks.rados.rados.0.smithi107.stdout:3054: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:48.008 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 588 (ObjNum 3053 snap 0 seq_num 3053) dirty exists 2020-06-12T02:37:48.008 INFO:tasks.rados.rados.0.smithi107.stdout:3054: left oid 17 (ObjNum 3053 snap 0 seq_num 3053) 2020-06-12T02:37:48.009 INFO:tasks.rados.rados.0.smithi107.stdout:3058: seq_num 3057 ranges {0=802816} 2020-06-12T02:37:48.013 INFO:tasks.rados.rados.0.smithi107.stdout:3058: writing smithi10729407-6 from 0 to 802816 tid 1 2020-06-12T02:37:48.013 INFO:tasks.rados.rados.0.smithi107.stdout:3054: done (3 left) 2020-06-12T02:37:48.013 INFO:tasks.rados.rados.0.smithi107.stdout:3059: delete oid 15 current snap is 0 2020-06-12T02:37:48.013 INFO:tasks.rados.rados.0.smithi107.stdout:3055: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:48.013 INFO:tasks.rados.rados.0.smithi107.stdout:3055: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:48.014 INFO:tasks.rados.rados.0.smithi107.stdout:3055: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:48.014 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 551 (ObjNum 3054 snap 0 seq_num 3054) dirty exists 2020-06-12T02:37:48.014 INFO:tasks.rados.rados.0.smithi107.stdout:3055: left oid 9 (ObjNum 3054 snap 0 seq_num 3054) 2020-06-12T02:37:48.015 INFO:tasks.rados.rados.0.smithi107.stdout:3057: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:48.015 INFO:tasks.rados.rados.0.smithi107.stdout:3055: done (3 left) 2020-06-12T02:37:48.015 INFO:tasks.rados.rados.0.smithi107.stdout:3059: done (2 left) 2020-06-12T02:37:48.016 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:37:48.016 INFO:tasks.rados.rados.0.smithi107.stdout:3060: seq_num 3059 ranges {2719744=434176,3153920=401408} 2020-06-12T02:37:48.016 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:48.012+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:48.032 INFO:tasks.rados.rados.0.smithi107.stdout:3060: writing smithi10729407-20 from 2719744 to 3153920 tid 1 2020-06-12T02:37:48.035 INFO:tasks.rados.rados.0.smithi107.stdout:3060: writing smithi10729407-20 from 3153920 to 3555328 tid 2 2020-06-12T02:37:48.035 INFO:tasks.rados.rados.0.smithi107.stdout:3061: delete oid 13 current snap is 0 2020-06-12T02:37:48.035 INFO:tasks.rados.rados.0.smithi107.stdout:3057: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:48.035 INFO:tasks.rados.rados.0.smithi107.stdout:3057: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:48.035 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 298 (ObjNum 3056 snap 0 seq_num 3056) dirty exists 2020-06-12T02:37:48.036 INFO:tasks.rados.rados.0.smithi107.stdout:3057: left oid 10 (ObjNum 3056 snap 0 seq_num 3056) 2020-06-12T02:37:48.036 INFO:tasks.rados.rados.0.smithi107.stdout:3058: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:48.036 INFO:tasks.rados.rados.0.smithi107.stdout:3058: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:48.036 INFO:tasks.rados.rados.0.smithi107.stdout:3058: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:48.036 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 599 (ObjNum 3057 snap 0 seq_num 3057) dirty exists 2020-06-12T02:37:48.037 INFO:tasks.rados.rados.0.smithi107.stdout:3058: left oid 6 (ObjNum 3057 snap 0 seq_num 3057) 2020-06-12T02:37:48.037 INFO:tasks.rados.rados.0.smithi107.stdout:3057: done (3 left) 2020-06-12T02:37:48.037 INFO:tasks.rados.rados.0.smithi107.stdout:3058: done (2 left) 2020-06-12T02:37:48.038 INFO:tasks.rados.rados.0.smithi107.stdout:3061: done (1 left) 2020-06-12T02:37:48.038 INFO:tasks.rados.rados.0.smithi107.stdout:3062: delete oid 2 current snap is 0 2020-06-12T02:37:48.039 INFO:tasks.rados.rados.0.smithi107.stdout:3062: done (1 left) 2020-06-12T02:37:48.039 INFO:tasks.rados.rados.0.smithi107.stdout:3063: delete oid 17 current snap is 0 2020-06-12T02:37:48.040 INFO:tasks.rados.rados.0.smithi107.stdout:3060: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:48.040 INFO:tasks.rados.rados.0.smithi107.stdout:3060: finishing write tid 2 to smithi10729407-20 2020-06-12T02:37:48.041 INFO:tasks.rados.rados.0.smithi107.stdout:3063: done (1 left) 2020-06-12T02:37:48.042 INFO:tasks.rados.rados.0.smithi107.stdout:3064: delete oid 17 current snap is 0 2020-06-12T02:37:48.042 INFO:tasks.rados.rados.0.smithi107.stdout:3060: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:48.043 INFO:tasks.rados.rados.0.smithi107.stdout:3060: finishing write tid 5 to smithi10729407-20 2020-06-12T02:37:48.044 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 212 (ObjNum 3059 snap 0 seq_num 3059) dirty exists 2020-06-12T02:37:48.044 INFO:tasks.rados.rados.0.smithi107.stdout:3060: left oid 20 (ObjNum 3059 snap 0 seq_num 3059) 2020-06-12T02:37:48.047 INFO:tasks.rados.rados.0.smithi107.stdout:3060: done (1 left) 2020-06-12T02:37:48.047 INFO:tasks.rados.rados.0.smithi107.stdout:3064: done (0 left) 2020-06-12T02:37:48.047 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:37:48.048 INFO:tasks.rados.rados.0.smithi107.stdout:3065: seq_num 3064 ranges {0=524288,524288=401408} 2020-06-12T02:37:48.050 INFO:tasks.rados.rados.0.smithi107.stdout:3065: writing smithi10729407-7 from 0 to 524288 tid 1 2020-06-12T02:37:48.052 INFO:tasks.rados.rados.0.smithi107.stdout:3065: writing smithi10729407-7 from 524288 to 925696 tid 2 2020-06-12T02:37:48.053 INFO:tasks.rados.rados.0.smithi107.stdout:3066: delete oid 4 current snap is 0 2020-06-12T02:37:48.057 INFO:tasks.rados.rados.0.smithi107.stdout:3066: done (1 left) 2020-06-12T02:37:48.058 INFO:tasks.rados.rados.0.smithi107.stdout:3067: delete oid 19 current snap is 0 2020-06-12T02:37:48.058 INFO:tasks.rados.rados.0.smithi107.stdout:3065: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:48.059 INFO:tasks.rados.rados.0.smithi107.stdout:3065: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:48.060 INFO:tasks.rados.rados.0.smithi107.stdout:3067: done (1 left) 2020-06-12T02:37:48.060 INFO:tasks.rados.rados.0.smithi107.stdout:3068: delete oid 15 current snap is 0 2020-06-12T02:37:48.060 INFO:tasks.rados.rados.0.smithi107.stdout:3065: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:48.062 INFO:tasks.rados.rados.0.smithi107.stdout:3065: finishing write tid 5 to smithi10729407-7 2020-06-12T02:37:48.062 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 587 (ObjNum 3064 snap 0 seq_num 3064) dirty exists 2020-06-12T02:37:48.062 INFO:tasks.rados.rados.0.smithi107.stdout:3065: left oid 7 (ObjNum 3064 snap 0 seq_num 3064) 2020-06-12T02:37:48.066 INFO:tasks.rados.rados.0.smithi107.stdout:3065: done (1 left) 2020-06-12T02:37:48.067 INFO:tasks.rados.rados.0.smithi107.stdout:3068: done (0 left) 2020-06-12T02:37:48.067 INFO:tasks.rados.rados.0.smithi107.stdout:3069: delete oid 2 current snap is 0 2020-06-12T02:37:48.068 INFO:tasks.rados.rados.0.smithi107.stdout:3069: done (0 left) 2020-06-12T02:37:48.068 INFO:tasks.rados.rados.0.smithi107.stdout:3070: delete oid 19 current snap is 0 2020-06-12T02:37:48.069 INFO:tasks.rados.rados.0.smithi107.stdout:3070: done (0 left) 2020-06-12T02:37:48.070 INFO:tasks.rados.rados.0.smithi107.stdout:3071: delete oid 2 current snap is 0 2020-06-12T02:37:48.071 INFO:tasks.rados.rados.0.smithi107.stdout:3071: done (0 left) 2020-06-12T02:37:48.071 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:37:48.072 INFO:tasks.rados.rados.0.smithi107.stdout:3072: seq_num 3071 ranges {753664=638976,1392640=401408} 2020-06-12T02:37:48.079 INFO:tasks.rados.rados.0.smithi107.stdout:3072: writing smithi10729407-9 from 753664 to 1392640 tid 1 2020-06-12T02:37:48.082 INFO:tasks.rados.rados.0.smithi107.stdout:3072: writing smithi10729407-9 from 1392640 to 1794048 tid 2 2020-06-12T02:37:48.082 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:37:48.082 INFO:tasks.rados.rados.0.smithi107.stdout:3073: seq_num 3072 ranges {0=655360,655360=401408} 2020-06-12T02:37:48.084 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:37:48.084 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:37:48.085 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:37:48.085 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:37:48.085 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.085 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:858 3.3s0 3:ce45c70a:::smithi10729407-7:head [append 0~565248] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:48.086 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.924643+0000", 2020-06-12T02:37:48.086 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 16.155073445999999, 2020-06-12T02:37:48.086 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015332551999999999, 2020-06-12T02:37:48.086 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:48.086 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:48.086 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:48.087 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:48.087 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:48.087 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 858 2020-06-12T02:37:48.087 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.087 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:48.088 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.088 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:48.088 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.924643+0000", 2020-06-12T02:37:48.088 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:48.089 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.089 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.089 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:48.090 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.924643+0000", 2020-06-12T02:37:48.090 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00067535299999999996 2020-06-12T02:37:48.090 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.090 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.090 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:48.090 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925318+0000", 2020-06-12T02:37:48.091 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9629999999999993e-06 2020-06-12T02:37:48.091 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.091 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.091 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:48.091 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925328+0000", 2020-06-12T02:37:48.092 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7357999999999999e-05 2020-06-12T02:37:48.092 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.092 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.092 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:48.092 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925345+0000", 2020-06-12T02:37:48.092 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7019999999999996e-06 2020-06-12T02:37:48.093 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.093 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.093 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:48.093 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925355+0000", 2020-06-12T02:37:48.094 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6289999999999998e-05 2020-06-12T02:37:48.094 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.094 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.094 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.095 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925411+0000", 2020-06-12T02:37:48.095 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8855000000000004e-05 2020-06-12T02:37:48.095 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.095 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.095 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:48.096 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925480+0000", 2020-06-12T02:37:48.096 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0074521149999999996 2020-06-12T02:37:48.096 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.096 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.096 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.096 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.932932+0000", 2020-06-12T02:37:48.097 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3647999999999995e-05 2020-06-12T02:37:48.097 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.097 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.097 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:48.097 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.933026+0000", 2020-06-12T02:37:48.098 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00078753199999999997 2020-06-12T02:37:48.098 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.098 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.098 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:48.098 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.933813+0000", 2020-06-12T02:37:48.099 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.006013582 2020-06-12T02:37:48.099 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.099 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.100 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:48.100 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.939827+0000", 2020-06-12T02:37:48.100 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0111999999999996e-05 2020-06-12T02:37:48.100 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.100 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.101 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:48.101 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.939897+0000", 2020-06-12T02:37:48.101 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8041999999999996e-05 2020-06-12T02:37:48.101 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.102 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.102 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:48.102 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.939975+0000", 2020-06-12T02:37:48.102 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015332551999999999 2020-06-12T02:37:48.102 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:48.102 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:48.103 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:48.103 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.103 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.103 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:859 3.3s0 3:ce45c70a:::smithi10729407-7:head [append 0~557056] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:48.103 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.927973+0000", 2020-06-12T02:37:48.104 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 16.151743235000001, 2020-06-12T02:37:48.104 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014425955000000001, 2020-06-12T02:37:48.104 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:48.104 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:48.104 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:48.104 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:48.105 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:48.105 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 859 2020-06-12T02:37:48.105 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.105 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:48.105 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.106 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:48.106 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.927973+0000", 2020-06-12T02:37:48.106 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:48.106 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.106 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.106 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:48.107 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.927973+0000", 2020-06-12T02:37:48.107 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00058871900000000003 2020-06-12T02:37:48.107 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.107 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.107 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:48.108 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928562+0000", 2020-06-12T02:37:48.108 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5679999999999999e-06 2020-06-12T02:37:48.108 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.108 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.108 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:48.108 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928567+0000", 2020-06-12T02:37:48.109 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1883e-05 2020-06-12T02:37:48.109 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.110 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.110 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:48.110 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928579+0000", 2020-06-12T02:37:48.110 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.097e-06 2020-06-12T02:37:48.110 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.111 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.111 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:48.111 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928586+0000", 2020-06-12T02:37:48.111 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6851999999999998e-05 2020-06-12T02:37:48.111 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.112 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.112 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.112 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928633+0000", 2020-06-12T02:37:48.112 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6867999999999997e-05 2020-06-12T02:37:48.112 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.112 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.113 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:48.113 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928700+0000", 2020-06-12T02:37:48.113 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0079226580000000008 2020-06-12T02:37:48.113 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.113 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.114 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.114 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.936623+0000", 2020-06-12T02:37:48.115 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000108597 2020-06-12T02:37:48.115 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.115 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.115 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:48.115 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.936731+0000", 2020-06-12T02:37:48.116 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054516799999999995 2020-06-12T02:37:48.116 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.116 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.116 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:48.116 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.937276+0000", 2020-06-12T02:37:48.117 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0043134339999999997 2020-06-12T02:37:48.117 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.117 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.117 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:48.117 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941590+0000", 2020-06-12T02:37:48.118 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00074530099999999997 2020-06-12T02:37:48.118 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.118 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.118 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:48.118 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.942335+0000", 2020-06-12T02:37:48.118 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3810000000000001e-05 2020-06-12T02:37:48.119 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.119 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.119 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:48.119 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.942399+0000", 2020-06-12T02:37:48.119 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014425955000000001 2020-06-12T02:37:48.120 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:48.120 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:48.120 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:48.120 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.120 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.120 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:860 3.3s0 3:ce45c70a:::smithi10729407-7:head [append 0~81920] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:48.121 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.928619+0000", 2020-06-12T02:37:48.121 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 16.15109756, 2020-06-12T02:37:48.121 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017431941999999999, 2020-06-12T02:37:48.121 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:48.121 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:48.122 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:48.122 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:48.122 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:48.122 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 860 2020-06-12T02:37:48.122 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.123 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:48.123 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.123 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:48.123 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928619+0000", 2020-06-12T02:37:48.123 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:48.124 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.124 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.124 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:48.124 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928619+0000", 2020-06-12T02:37:48.124 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000105809 2020-06-12T02:37:48.124 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.125 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.125 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:48.125 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928724+0000", 2020-06-12T02:37:48.125 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2339999999999994e-06 2020-06-12T02:37:48.125 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.126 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.126 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:48.132 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928733+0000", 2020-06-12T02:37:48.132 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0268000000000001e-05 2020-06-12T02:37:48.132 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.133 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.133 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:48.133 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928743+0000", 2020-06-12T02:37:48.133 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5190000000000003e-06 2020-06-12T02:37:48.133 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.134 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.134 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:48.134 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928747+0000", 2020-06-12T02:37:48.134 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0103999999999998e-05 2020-06-12T02:37:48.134 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.134 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.135 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.135 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928777+0000", 2020-06-12T02:37:48.135 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8249000000000002e-05 2020-06-12T02:37:48.135 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.135 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.136 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:48.136 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928866+0000", 2020-06-12T02:37:48.136 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011151629 2020-06-12T02:37:48.136 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.136 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.136 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.137 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.940017+0000", 2020-06-12T02:37:48.137 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8281e-05 2020-06-12T02:37:48.137 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.137 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.137 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:48.138 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.940076+0000", 2020-06-12T02:37:48.138 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000346736 2020-06-12T02:37:48.138 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.138 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.138 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:48.139 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.940422+0000", 2020-06-12T02:37:48.139 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024135049999999998 2020-06-12T02:37:48.139 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.139 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.139 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:48.139 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.942836+0000", 2020-06-12T02:37:48.140 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003170258 2020-06-12T02:37:48.140 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.140 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.140 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:48.140 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946006+0000", 2020-06-12T02:37:48.141 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4350000000000001e-05 2020-06-12T02:37:48.141 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.141 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.141 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:48.141 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946050+0000", 2020-06-12T02:37:48.142 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017431941999999999 2020-06-12T02:37:48.142 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:48.142 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:48.142 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:48.142 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.142 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.143 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:861 3.3s0 3:ce45c70a:::smithi10729407-7:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:48.143 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.928778+0000", 2020-06-12T02:37:48.143 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 16.150937946999999, 2020-06-12T02:37:48.143 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017522738, 2020-06-12T02:37:48.143 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:48.144 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:48.144 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:48.144 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:48.144 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:48.144 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 861 2020-06-12T02:37:48.144 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.145 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:48.145 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.145 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:48.145 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928778+0000", 2020-06-12T02:37:48.145 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:48.146 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.146 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.146 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:48.146 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928778+0000", 2020-06-12T02:37:48.146 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8186000000000002e-05 2020-06-12T02:37:48.146 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.147 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.147 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:48.147 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928826+0000", 2020-06-12T02:37:48.147 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6689999999999997e-06 2020-06-12T02:37:48.147 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.148 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.148 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:48.148 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928831+0000", 2020-06-12T02:37:48.148 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.686e-06 2020-06-12T02:37:48.148 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.148 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.149 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:48.149 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928838+0000", 2020-06-12T02:37:48.149 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7159999999999999e-06 2020-06-12T02:37:48.149 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.149 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.150 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:48.150 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928840+0000", 2020-06-12T02:37:48.150 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4212999999999997e-05 2020-06-12T02:37:48.150 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.150 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.151 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.151 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928895+0000", 2020-06-12T02:37:48.151 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5853999999999994e-05 2020-06-12T02:37:48.151 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.151 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.151 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:48.152 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928990+0000", 2020-06-12T02:37:48.152 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012023982000000001 2020-06-12T02:37:48.152 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.152 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.152 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.153 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941014+0000", 2020-06-12T02:37:48.153 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5104e-05 2020-06-12T02:37:48.153 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.153 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.153 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:48.153 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941070+0000", 2020-06-12T02:37:48.154 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00026444499999999998 2020-06-12T02:37:48.154 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.154 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.154 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:48.154 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941334+0000", 2020-06-12T02:37:48.155 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001704075 2020-06-12T02:37:48.155 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.155 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.155 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:48.155 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.943038+0000", 2020-06-12T02:37:48.155 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031891570000000002 2020-06-12T02:37:48.156 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.156 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.156 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:48.156 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946227+0000", 2020-06-12T02:37:48.156 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3651000000000003e-05 2020-06-12T02:37:48.157 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.157 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.157 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:48.157 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946301+0000", 2020-06-12T02:37:48.157 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017522738 2020-06-12T02:37:48.157 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:48.158 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:48.158 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:48.158 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.158 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.158 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:862 3.3s0 3:ce45c70a:::smithi10729407-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:48.159 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.928876+0000", 2020-06-12T02:37:48.159 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 16.150840451000001, 2020-06-12T02:37:48.159 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019206890000000001, 2020-06-12T02:37:48.159 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:48.159 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:48.159 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:48.160 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:48.160 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:48.160 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 862 2020-06-12T02:37:48.160 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.160 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:48.161 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.161 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:48.161 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928876+0000", 2020-06-12T02:37:48.161 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:48.161 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.161 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.162 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:48.162 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928876+0000", 2020-06-12T02:37:48.162 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9268999999999999e-05 2020-06-12T02:37:48.162 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.162 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.163 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:48.163 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928925+0000", 2020-06-12T02:37:48.163 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8120000000000002e-06 2020-06-12T02:37:48.163 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.163 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.163 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:48.164 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928932+0000", 2020-06-12T02:37:48.164 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3471e-05 2020-06-12T02:37:48.164 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.164 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.164 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:48.164 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928945+0000", 2020-06-12T02:37:48.165 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4279999999999998e-06 2020-06-12T02:37:48.165 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.165 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.165 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:48.165 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928950+0000", 2020-06-12T02:37:48.166 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6358000000000004e-05 2020-06-12T02:37:48.166 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.166 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.166 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.166 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.929016+0000", 2020-06-12T02:37:48.167 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9697999999999996e-05 2020-06-12T02:37:48.167 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.167 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.167 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:48.167 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.929086+0000", 2020-06-12T02:37:48.167 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012409205 2020-06-12T02:37:48.168 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.168 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.168 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.168 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941495+0000", 2020-06-12T02:37:48.168 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9660000000000006e-05 2020-06-12T02:37:48.169 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.169 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.169 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:48.169 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941585+0000", 2020-06-12T02:37:48.169 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050329099999999998 2020-06-12T02:37:48.169 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.170 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.170 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.170 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946617+0000", 2020-06-12T02:37:48.170 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8437999999999996e-05 2020-06-12T02:37:48.170 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.171 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.171 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:48.171 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946706+0000", 2020-06-12T02:37:48.171 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0013766410000000001 2020-06-12T02:37:48.171 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.171 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.172 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:48.172 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.948083+0000", 2020-06-12T02:37:48.172 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019206890000000001 2020-06-12T02:37:48.172 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:48.172 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:48.173 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:48.173 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.173 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.173 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:879 3.fs1 3:fc7e0bf7:::smithi10729407-17:head [append 0~491520] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:48.173 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.000881+0000", 2020-06-12T02:37:48.174 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 16.078835269999999, 2020-06-12T02:37:48.174 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017891845, 2020-06-12T02:37:48.174 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:48.174 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:48.174 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:48.174 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:48.175 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:48.175 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 879 2020-06-12T02:37:48.175 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.175 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:48.175 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.176 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:48.176 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.000881+0000", 2020-06-12T02:37:48.176 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:48.176 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.176 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.176 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:48.177 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.000881+0000", 2020-06-12T02:37:48.177 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00044151599999999998 2020-06-12T02:37:48.177 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.177 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.177 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:48.178 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001322+0000", 2020-06-12T02:37:48.178 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0210000000000001e-06 2020-06-12T02:37:48.178 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.178 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.178 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:48.178 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001326+0000", 2020-06-12T02:37:48.179 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6659999999999997e-06 2020-06-12T02:37:48.179 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.179 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.179 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:48.179 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001335+0000", 2020-06-12T02:37:48.179 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4299999999999999e-06 2020-06-12T02:37:48.180 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.180 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.180 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:48.180 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001339+0000", 2020-06-12T02:37:48.180 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0028741510000000001 2020-06-12T02:37:48.181 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.181 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.181 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.181 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.004214+0000", 2020-06-12T02:37:48.181 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010480999999999999 2020-06-12T02:37:48.182 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.182 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.182 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:48.182 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.004318+0000", 2020-06-12T02:37:48.182 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021112359999999998 2020-06-12T02:37:48.182 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.183 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.183 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.183 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006430+0000", 2020-06-12T02:37:48.183 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2935000000000001e-05 2020-06-12T02:37:48.183 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.184 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.184 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:48.184 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006483+0000", 2020-06-12T02:37:48.184 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00056411099999999995 2020-06-12T02:37:48.184 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.184 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.185 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:48.185 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.007047+0000", 2020-06-12T02:37:48.185 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0076749369999999997 2020-06-12T02:37:48.185 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.185 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.186 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:48.186 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.014722+0000", 2020-06-12T02:37:48.186 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00395693 2020-06-12T02:37:48.186 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.186 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.186 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:48.187 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.018679+0000", 2020-06-12T02:37:48.187 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4102000000000001e-05 2020-06-12T02:37:48.187 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.187 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.187 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:48.188 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.018773+0000", 2020-06-12T02:37:48.188 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017891845 2020-06-12T02:37:48.188 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:48.188 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:48.188 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:48.188 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.189 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.189 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:883 3.fs1 3:fc7e0bf7:::smithi10729407-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:48.189 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.006322+0000", 2020-06-12T02:37:48.189 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 16.073394108999999, 2020-06-12T02:37:48.189 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01444995, 2020-06-12T02:37:48.190 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:48.190 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:48.190 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:48.190 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:48.190 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:48.191 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 883 2020-06-12T02:37:48.191 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.191 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:48.191 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.191 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:48.191 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006322+0000", 2020-06-12T02:37:48.192 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:48.192 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.192 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.192 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:48.192 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006322+0000", 2020-06-12T02:37:48.192 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1342999999999999e-05 2020-06-12T02:37:48.193 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.193 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.193 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:48.193 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006343+0000", 2020-06-12T02:37:48.193 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.021e-06 2020-06-12T02:37:48.194 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.194 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.194 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:48.194 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006346+0000", 2020-06-12T02:37:48.194 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2990000000000002e-06 2020-06-12T02:37:48.194 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.195 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.195 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:48.195 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006353+0000", 2020-06-12T02:37:48.195 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3240000000000001e-06 2020-06-12T02:37:48.195 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.196 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.196 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:48.196 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006355+0000", 2020-06-12T02:37:48.196 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0090315139999999992 2020-06-12T02:37:48.196 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.196 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.197 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.197 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.015386+0000", 2020-06-12T02:37:48.197 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5129000000000003e-05 2020-06-12T02:37:48.197 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.197 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.198 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:48.198 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.015482+0000", 2020-06-12T02:37:48.198 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0043414839999999996 2020-06-12T02:37:48.198 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.198 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.198 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.199 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.019823+0000", 2020-06-12T02:37:48.199 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6991999999999997e-05 2020-06-12T02:37:48.199 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.199 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.199 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:48.200 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.019910+0000", 2020-06-12T02:37:48.200 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00086184399999999996 2020-06-12T02:37:48.200 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.200 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.200 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:48.200 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.020772+0000", 2020-06-12T02:37:48.201 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01444995 2020-06-12T02:37:48.201 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:48.201 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:48.201 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:48.201 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.202 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.202 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:933 3.es1 3:78f01648:::smithi10729407-5:head [append 0~417792] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:48.202 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.156677+0000", 2020-06-12T02:37:48.202 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 15.923039288, 2020-06-12T02:37:48.202 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016833311, 2020-06-12T02:37:48.203 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:48.203 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:48.203 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:48.203 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:48.203 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:48.203 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 933 2020-06-12T02:37:48.204 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.204 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:48.204 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.204 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:48.204 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.156677+0000", 2020-06-12T02:37:48.205 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:48.205 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.205 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.205 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:48.205 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.156677+0000", 2020-06-12T02:37:48.205 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00062571700000000003 2020-06-12T02:37:48.206 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.206 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.206 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:48.206 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157303+0000", 2020-06-12T02:37:48.206 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0740000000000003e-06 2020-06-12T02:37:48.207 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.207 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.207 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:48.207 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157307+0000", 2020-06-12T02:37:48.207 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4419999999999998e-06 2020-06-12T02:37:48.207 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.208 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.208 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:48.208 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157315+0000", 2020-06-12T02:37:48.208 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5300000000000001e-06 2020-06-12T02:37:48.208 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.209 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.209 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:48.209 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157319+0000", 2020-06-12T02:37:48.209 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4995e-05 2020-06-12T02:37:48.209 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.209 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.210 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.210 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157354+0000", 2020-06-12T02:37:48.210 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5883000000000006e-05 2020-06-12T02:37:48.210 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.210 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.211 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:48.211 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157419+0000", 2020-06-12T02:37:48.211 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045316280000000002 2020-06-12T02:37:48.211 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.211 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.211 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.212 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.161951+0000", 2020-06-12T02:37:48.212 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3300999999999996e-05 2020-06-12T02:37:48.212 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.212 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.212 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:48.213 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.162014+0000", 2020-06-12T02:37:48.213 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00039050100000000002 2020-06-12T02:37:48.213 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.213 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.213 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:48.213 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.162405+0000", 2020-06-12T02:37:48.214 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050686250000000002 2020-06-12T02:37:48.214 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.214 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.214 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:48.214 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.167474+0000", 2020-06-12T02:37:48.215 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.005942653 2020-06-12T02:37:48.215 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.215 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.215 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:48.215 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.173416+0000", 2020-06-12T02:37:48.215 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3962000000000006e-05 2020-06-12T02:37:48.216 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.216 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.216 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:48.216 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.173510+0000", 2020-06-12T02:37:48.216 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016833311 2020-06-12T02:37:48.217 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:48.217 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:48.217 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:48.217 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.217 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.217 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:936 3.es1 3:78f01648:::smithi10729407-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:48.218 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.160357+0000", 2020-06-12T02:37:48.218 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 15.919359577, 2020-06-12T02:37:48.218 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015435590000000001, 2020-06-12T02:37:48.218 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:48.218 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:48.219 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:48.219 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:48.219 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:48.219 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 936 2020-06-12T02:37:48.219 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.219 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:48.220 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.220 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:48.220 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160357+0000", 2020-06-12T02:37:48.220 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:48.220 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.221 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.221 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:48.221 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160357+0000", 2020-06-12T02:37:48.221 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4435999999999999e-05 2020-06-12T02:37:48.221 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.221 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.222 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:48.222 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160411+0000", 2020-06-12T02:37:48.222 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2329999999999995e-06 2020-06-12T02:37:48.222 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.222 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.223 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:48.223 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160420+0000", 2020-06-12T02:37:48.223 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7003e-05 2020-06-12T02:37:48.223 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.223 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.223 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:48.224 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160437+0000", 2020-06-12T02:37:48.224 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8370000000000002e-06 2020-06-12T02:37:48.224 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.224 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.224 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:48.225 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160443+0000", 2020-06-12T02:37:48.225 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0070635489999999997 2020-06-12T02:37:48.225 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.225 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.225 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.225 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.167507+0000", 2020-06-12T02:37:48.226 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010589700000000001 2020-06-12T02:37:48.226 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.226 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.226 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:48.226 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.167612+0000", 2020-06-12T02:37:48.227 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0065032299999999996 2020-06-12T02:37:48.227 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.227 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.227 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.227 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.174116+0000", 2020-06-12T02:37:48.227 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3128999999999999e-05 2020-06-12T02:37:48.228 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.228 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.228 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:48.228 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.174169+0000", 2020-06-12T02:37:48.228 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0016232760000000001 2020-06-12T02:37:48.229 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.229 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.229 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:48.229 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.175792+0000", 2020-06-12T02:37:48.229 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015435590000000001 2020-06-12T02:37:48.229 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:48.230 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:48.230 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:48.230 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.230 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.230 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:950 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~778240] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:48.231 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.207643+0000", 2020-06-12T02:37:48.231 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 15.872072994, 2020-06-12T02:37:48.231 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014844661, 2020-06-12T02:37:48.231 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:48.231 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:48.231 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:48.232 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:48.232 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:48.232 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 950 2020-06-12T02:37:48.232 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.232 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:48.233 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.233 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:48.233 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.207643+0000", 2020-06-12T02:37:48.233 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:48.233 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.234 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.234 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:48.234 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.207643+0000", 2020-06-12T02:37:48.234 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00072843099999999998 2020-06-12T02:37:48.234 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.234 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.235 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:48.235 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208372+0000", 2020-06-12T02:37:48.235 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9120000000000001e-06 2020-06-12T02:37:48.235 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.235 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.236 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:48.236 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208378+0000", 2020-06-12T02:37:48.236 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9660000000000008e-06 2020-06-12T02:37:48.236 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.236 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.236 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:48.237 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208388+0000", 2020-06-12T02:37:48.237 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1279999999999999e-06 2020-06-12T02:37:48.237 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.237 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.237 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:48.238 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208394+0000", 2020-06-12T02:37:48.238 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0439999999999998e-05 2020-06-12T02:37:48.238 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.238 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.238 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.238 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208444+0000", 2020-06-12T02:37:48.239 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2611000000000001e-05 2020-06-12T02:37:48.239 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.239 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.239 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:48.239 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208507+0000", 2020-06-12T02:37:48.240 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001228301 2020-06-12T02:37:48.240 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.240 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.240 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.240 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.209735+0000", 2020-06-12T02:37:48.240 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8875000000000001e-05 2020-06-12T02:37:48.241 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.241 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.241 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:48.241 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.209794+0000", 2020-06-12T02:37:48.241 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053218899999999997 2020-06-12T02:37:48.242 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.242 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.242 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:48.242 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.210326+0000", 2020-06-12T02:37:48.242 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0078198439999999994 2020-06-12T02:37:48.243 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.243 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.243 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:48.243 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.218146+0000", 2020-06-12T02:37:48.243 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004300496 2020-06-12T02:37:48.243 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.244 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.244 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:48.244 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.222446+0000", 2020-06-12T02:37:48.244 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1467999999999997e-05 2020-06-12T02:37:48.244 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.244 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.245 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:48.245 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.222488+0000", 2020-06-12T02:37:48.245 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014844661 2020-06-12T02:37:48.245 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:48.245 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:48.246 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:48.246 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.246 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.246 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:954 3.3s0 3:c2ca77da:::smithi10729407-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:48.246 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.212923+0000", 2020-06-12T02:37:48.247 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 15.866793218, 2020-06-12T02:37:48.247 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015411546999999999, 2020-06-12T02:37:48.247 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:48.247 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:48.247 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:48.247 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:48.248 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:48.248 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 954 2020-06-12T02:37:48.248 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.248 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:48.248 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.249 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:48.249 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212923+0000", 2020-06-12T02:37:48.249 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:48.249 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.249 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.249 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:48.250 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212923+0000", 2020-06-12T02:37:48.250 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3598e-05 2020-06-12T02:37:48.250 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.250 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.250 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:48.251 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212946+0000", 2020-06-12T02:37:48.251 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.061e-06 2020-06-12T02:37:48.251 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.251 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.251 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:48.251 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212950+0000", 2020-06-12T02:37:48.252 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8700000000000003e-06 2020-06-12T02:37:48.252 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.252 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.252 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:48.252 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212956+0000", 2020-06-12T02:37:48.252 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0770000000000002e-06 2020-06-12T02:37:48.253 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.253 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.253 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:48.253 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212958+0000", 2020-06-12T02:37:48.253 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051235170000000002 2020-06-12T02:37:48.254 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.254 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.254 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.254 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.218082+0000", 2020-06-12T02:37:48.254 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9246999999999997e-05 2020-06-12T02:37:48.254 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.255 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.255 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:48.255 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.218141+0000", 2020-06-12T02:37:48.255 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0087981910000000003 2020-06-12T02:37:48.255 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.256 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.256 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.256 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.226939+0000", 2020-06-12T02:37:48.256 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4119999999999997e-05 2020-06-12T02:37:48.256 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.256 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.257 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:48.257 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.226994+0000", 2020-06-12T02:37:48.257 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001340866 2020-06-12T02:37:48.257 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.257 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.258 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:48.258 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.228334+0000", 2020-06-12T02:37:48.258 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015411546999999999 2020-06-12T02:37:48.258 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:48.258 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:48.258 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:48.259 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.259 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.259 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1097 3.es1 3:78f01648:::smithi10729407-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:48.259 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.576496+0000", 2020-06-12T02:37:48.259 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 15.503219736, 2020-06-12T02:37:48.260 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014992914, 2020-06-12T02:37:48.260 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:48.260 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:48.260 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:48.260 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:48.260 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:48.261 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1097 2020-06-12T02:37:48.261 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.261 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:48.261 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.261 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:48.262 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576496+0000", 2020-06-12T02:37:48.262 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:48.262 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.262 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.262 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:48.262 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576496+0000", 2020-06-12T02:37:48.263 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3221999999999999e-05 2020-06-12T02:37:48.263 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.263 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.263 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:48.263 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576520+0000", 2020-06-12T02:37:48.264 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.988e-06 2020-06-12T02:37:48.264 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.264 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.264 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:48.264 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576523+0000", 2020-06-12T02:37:48.264 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2940000000000002e-06 2020-06-12T02:37:48.265 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.265 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.265 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:48.265 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576529+0000", 2020-06-12T02:37:48.265 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6060000000000002e-06 2020-06-12T02:37:48.266 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.266 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.266 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:48.266 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576534+0000", 2020-06-12T02:37:48.266 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001254554 2020-06-12T02:37:48.266 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.267 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.267 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.267 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.577789+0000", 2020-06-12T02:37:48.267 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3177999999999998e-05 2020-06-12T02:37:48.267 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.267 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.268 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:48.268 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.577842+0000", 2020-06-12T02:37:48.268 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0063091220000000003 2020-06-12T02:37:48.268 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.268 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.269 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.269 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.584151+0000", 2020-06-12T02:37:48.269 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.783e-05 2020-06-12T02:37:48.269 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.269 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.269 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:48.270 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.584209+0000", 2020-06-12T02:37:48.270 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045863650000000002 2020-06-12T02:37:48.270 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.270 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.270 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.271 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.588796+0000", 2020-06-12T02:37:48.271 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7614000000000006e-05 2020-06-12T02:37:48.271 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.271 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.271 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:48.272 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.588873+0000", 2020-06-12T02:37:48.272 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0026161410000000002 2020-06-12T02:37:48.272 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.272 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.272 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:48.272 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.591489+0000", 2020-06-12T02:37:48.273 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014992914 2020-06-12T02:37:48.273 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:48.273 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:48.273 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:48.273 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.274 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.274 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1207 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~630784] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:48.274 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.859578+0000", 2020-06-12T02:37:48.274 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 15.220138199999999, 2020-06-12T02:37:48.274 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014710433, 2020-06-12T02:37:48.274 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:48.275 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:48.275 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:48.275 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:48.275 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:48.275 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1207 2020-06-12T02:37:48.276 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.276 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:48.276 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.276 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:48.276 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.859578+0000", 2020-06-12T02:37:48.276 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:48.277 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.277 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.277 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:48.277 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.859578+0000", 2020-06-12T02:37:48.277 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000775511 2020-06-12T02:37:48.278 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.278 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.278 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:48.278 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.860353+0000", 2020-06-12T02:37:48.278 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5325999999999999e-05 2020-06-12T02:37:48.278 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.279 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.279 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:48.279 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.860369+0000", 2020-06-12T02:37:48.279 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7430000000000001e-05 2020-06-12T02:37:48.279 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.280 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.280 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:48.280 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.860386+0000", 2020-06-12T02:37:48.280 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.426e-06 2020-06-12T02:37:48.280 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.280 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.281 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:48.281 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.860393+0000", 2020-06-12T02:37:48.281 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3585999999999999e-05 2020-06-12T02:37:48.281 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.281 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.282 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.282 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.860436+0000", 2020-06-12T02:37:48.282 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2559999999999996e-05 2020-06-12T02:37:48.282 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.282 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.283 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:48.283 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.860509+0000", 2020-06-12T02:37:48.283 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00063177700000000003 2020-06-12T02:37:48.283 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.283 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.283 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:48.284 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.861141+0000", 2020-06-12T02:37:48.284 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0026307570000000001 2020-06-12T02:37:48.284 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.284 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.284 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:48.285 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.863771+0000", 2020-06-12T02:37:48.285 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010445545000000001 2020-06-12T02:37:48.285 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.285 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.285 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:48.285 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.874217+0000", 2020-06-12T02:37:48.286 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1514999999999995e-05 2020-06-12T02:37:48.286 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.286 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.286 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:48.286 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.874288+0000", 2020-06-12T02:37:48.287 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014710433 2020-06-12T02:37:48.287 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:48.287 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:48.287 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:48.287 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.287 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.288 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2788 3.fs1 3:fc7e0bf7:::smithi10729407-17:head [append 0~679936] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:48.288 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:36.820585+0000", 2020-06-12T02:37:48.288 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.259131137000001, 2020-06-12T02:37:48.288 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01440234, 2020-06-12T02:37:48.288 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:48.289 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:48.289 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:48.289 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:48.289 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:48.289 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2788 2020-06-12T02:37:48.290 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.290 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:48.290 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.290 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:48.290 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.820585+0000", 2020-06-12T02:37:48.290 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:48.291 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.291 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.291 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:48.291 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.820585+0000", 2020-06-12T02:37:48.291 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001119386 2020-06-12T02:37:48.292 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.292 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.292 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:48.292 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.821704+0000", 2020-06-12T02:37:48.292 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2069000000000001e-05 2020-06-12T02:37:48.292 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.293 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.293 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:48.293 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.821716+0000", 2020-06-12T02:37:48.293 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2909e-05 2020-06-12T02:37:48.293 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.294 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.294 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:48.294 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.821729+0000", 2020-06-12T02:37:48.294 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4559999999999997e-06 2020-06-12T02:37:48.294 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.294 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.295 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:48.295 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.821736+0000", 2020-06-12T02:37:48.295 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6715000000000003e-05 2020-06-12T02:37:48.295 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.295 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.295 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.296 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.821772+0000", 2020-06-12T02:37:48.296 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7078999999999998e-05 2020-06-12T02:37:48.296 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.296 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.296 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:48.297 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.821850+0000", 2020-06-12T02:37:48.297 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00049280600000000004 2020-06-12T02:37:48.297 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.297 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.297 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:48.297 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.822342+0000", 2020-06-12T02:37:48.298 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051665629999999999 2020-06-12T02:37:48.298 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.298 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.298 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:48.298 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.827509+0000", 2020-06-12T02:37:48.299 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0073895280000000002 2020-06-12T02:37:48.299 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.299 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.299 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:48.299 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.834898+0000", 2020-06-12T02:37:48.299 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8828999999999999e-05 2020-06-12T02:37:48.300 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.300 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.300 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:48.300 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.834987+0000", 2020-06-12T02:37:48.300 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01440234 2020-06-12T02:37:48.301 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:48.301 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:48.301 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:48.301 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.301 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.301 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:4813 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~450560] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-06-12T02:37:48.302 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:43.343096+0000", 2020-06-12T02:37:48.302 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.7366203929999999, 2020-06-12T02:37:48.302 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014820242000000001, 2020-06-12T02:37:48.302 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:48.302 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:48.303 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:48.303 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:48.303 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:48.303 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 4813 2020-06-12T02:37:48.303 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.303 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:48.304 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.304 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:48.304 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343096+0000", 2020-06-12T02:37:48.304 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:48.304 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.304 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.305 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:48.305 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343096+0000", 2020-06-12T02:37:48.305 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00068963 2020-06-12T02:37:48.305 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.305 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.306 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:48.306 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343785+0000", 2020-06-12T02:37:48.306 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1479999999999997e-06 2020-06-12T02:37:48.306 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.306 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.307 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:48.307 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343790+0000", 2020-06-12T02:37:48.307 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4060000000000005e-06 2020-06-12T02:37:48.307 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.307 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.307 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:48.308 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343799+0000", 2020-06-12T02:37:48.308 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0970000000000002e-06 2020-06-12T02:37:48.308 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.308 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.308 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:48.309 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343805+0000", 2020-06-12T02:37:48.309 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00023736499999999999 2020-06-12T02:37:48.309 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.309 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.309 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.309 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.344042+0000", 2020-06-12T02:37:48.310 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000113866 2020-06-12T02:37:48.310 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.310 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.310 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:48.310 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.344156+0000", 2020-06-12T02:37:48.311 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036114760000000002 2020-06-12T02:37:48.311 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.311 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.311 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.311 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.347768+0000", 2020-06-12T02:37:48.311 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1842999999999996e-05 2020-06-12T02:37:48.312 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.312 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.312 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:48.312 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.347850+0000", 2020-06-12T02:37:48.312 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00048835999999999997 2020-06-12T02:37:48.313 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.313 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.313 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:48.313 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.348338+0000", 2020-06-12T02:37:48.313 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0049373139999999999 2020-06-12T02:37:48.313 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.314 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.314 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:48.314 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.353275+0000", 2020-06-12T02:37:48.314 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045727119999999996 2020-06-12T02:37:48.314 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.315 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.315 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:48.315 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.357848+0000", 2020-06-12T02:37:48.315 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8025e-05 2020-06-12T02:37:48.315 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.315 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.316 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:48.316 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.357916+0000", 2020-06-12T02:37:48.316 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014820242000000001 2020-06-12T02:37:48.316 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:48.316 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:48.317 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:48.317 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.317 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.317 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:4921 3.9s0 3:93ea0cad:::smithi10729407-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-06-12T02:37:48.317 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:43.595696+0000", 2020-06-12T02:37:48.317 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.4840200269999997, 2020-06-12T02:37:48.318 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016503190000000001, 2020-06-12T02:37:48.318 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:48.318 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:48.318 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:48.318 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:48.319 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:48.319 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 4921 2020-06-12T02:37:48.319 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.319 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:48.319 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.319 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:48.320 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595696+0000", 2020-06-12T02:37:48.320 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:48.320 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.320 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.320 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:48.321 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595696+0000", 2020-06-12T02:37:48.321 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3694999999999999e-05 2020-06-12T02:37:48.321 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.321 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.321 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:48.321 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595720+0000", 2020-06-12T02:37:48.322 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1700000000000001e-06 2020-06-12T02:37:48.322 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.322 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.322 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:48.322 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595723+0000", 2020-06-12T02:37:48.323 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8849999999999999e-06 2020-06-12T02:37:48.323 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.323 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.323 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:48.323 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595731+0000", 2020-06-12T02:37:48.323 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5689999999999999e-06 2020-06-12T02:37:48.324 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.324 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.324 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:48.324 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595733+0000", 2020-06-12T02:37:48.324 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0030116880000000002 2020-06-12T02:37:48.325 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.325 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.325 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.325 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.598745+0000", 2020-06-12T02:37:48.325 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1353999999999996e-05 2020-06-12T02:37:48.326 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.326 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.326 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:48.326 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.598806+0000", 2020-06-12T02:37:48.327 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051955819999999998 2020-06-12T02:37:48.327 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.327 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.327 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.327 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.604002+0000", 2020-06-12T02:37:48.327 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9615000000000001e-05 2020-06-12T02:37:48.328 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.328 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.328 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:48.328 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.604102+0000", 2020-06-12T02:37:48.328 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0052050960000000002 2020-06-12T02:37:48.329 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.329 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.329 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.329 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.609307+0000", 2020-06-12T02:37:48.329 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5025000000000003e-05 2020-06-12T02:37:48.329 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.330 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.330 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:48.330 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.609362+0000", 2020-06-12T02:37:48.330 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002837511 2020-06-12T02:37:48.330 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.331 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.331 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:48.331 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.612199+0000", 2020-06-12T02:37:48.331 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016503190000000001 2020-06-12T02:37:48.331 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:48.331 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:48.332 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:48.332 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.332 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.332 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:5068 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~663552] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e30)", 2020-06-12T02:37:48.332 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:44.157497+0000", 2020-06-12T02:37:48.333 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.922219401, 2020-06-12T02:37:48.333 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015164129, 2020-06-12T02:37:48.333 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:48.333 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:48.333 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:48.334 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:48.334 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:48.334 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 5068 2020-06-12T02:37:48.334 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.339 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:48.340 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.340 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:48.340 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.157497+0000", 2020-06-12T02:37:48.340 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:48.340 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.341 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.341 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:48.341 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.157497+0000", 2020-06-12T02:37:48.341 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00068511099999999997 2020-06-12T02:37:48.341 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.342 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.342 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:48.342 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.158182+0000", 2020-06-12T02:37:48.342 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4630000000000004e-06 2020-06-12T02:37:48.342 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.342 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.343 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:48.343 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.158189+0000", 2020-06-12T02:37:48.343 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4199000000000001e-05 2020-06-12T02:37:48.343 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.343 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.344 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:48.344 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.158203+0000", 2020-06-12T02:37:48.344 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1250000000000004e-06 2020-06-12T02:37:48.344 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.344 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.344 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:48.345 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.158211+0000", 2020-06-12T02:37:48.345 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6754000000000002e-05 2020-06-12T02:37:48.345 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.345 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.345 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.346 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.158237+0000", 2020-06-12T02:37:48.346 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2948999999999996e-05 2020-06-12T02:37:48.346 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.346 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.346 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:48.347 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.158310+0000", 2020-06-12T02:37:48.347 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0032401610000000001 2020-06-12T02:37:48.347 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.347 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.347 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.347 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.161550+0000", 2020-06-12T02:37:48.348 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5550000000000002e-05 2020-06-12T02:37:48.348 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.348 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.348 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:48.348 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.161606+0000", 2020-06-12T02:37:48.348 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00062448199999999999 2020-06-12T02:37:48.349 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.349 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.349 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:48.349 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.162230+0000", 2020-06-12T02:37:48.349 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0073257130000000002 2020-06-12T02:37:48.350 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.350 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.350 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:48.350 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.169556+0000", 2020-06-12T02:37:48.350 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0030226419999999999 2020-06-12T02:37:48.350 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.351 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.351 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:48.351 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.172579+0000", 2020-06-12T02:37:48.351 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1979999999999998e-05 2020-06-12T02:37:48.351 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.352 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.352 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:48.352 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.172661+0000", 2020-06-12T02:37:48.352 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015164129 2020-06-12T02:37:48.352 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:48.352 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:48.353 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:48.353 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.353 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.353 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:5071 3.3s0 3:c2ca77da:::smithi10729407-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-06-12T02:37:48.353 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:44.159788+0000", 2020-06-12T02:37:48.354 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.9199281789999998, 2020-06-12T02:37:48.354 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016812273999999999, 2020-06-12T02:37:48.354 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:48.354 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:48.354 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:48.355 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:48.355 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:48.355 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 5071 2020-06-12T02:37:48.355 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.355 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:48.355 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.356 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:48.356 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159788+0000", 2020-06-12T02:37:48.356 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:48.356 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.356 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.356 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:48.357 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159788+0000", 2020-06-12T02:37:48.357 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1421000000000001e-05 2020-06-12T02:37:48.357 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.357 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.357 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:48.358 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159839+0000", 2020-06-12T02:37:48.358 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9009999999999994e-06 2020-06-12T02:37:48.358 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.358 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.358 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:48.358 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159847+0000", 2020-06-12T02:37:48.359 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3149999999999999e-05 2020-06-12T02:37:48.359 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.359 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.359 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:48.359 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159860+0000", 2020-06-12T02:37:48.360 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1749999999999996e-06 2020-06-12T02:37:48.360 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.360 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.360 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:48.360 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159866+0000", 2020-06-12T02:37:48.360 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6035999999999999e-05 2020-06-12T02:37:48.361 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.361 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.361 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.361 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159892+0000", 2020-06-12T02:37:48.361 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2627999999999998e-05 2020-06-12T02:37:48.362 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.362 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.362 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:48.362 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159944+0000", 2020-06-12T02:37:48.362 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010270133000000001 2020-06-12T02:37:48.362 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.363 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.363 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.363 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.170214+0000", 2020-06-12T02:37:48.363 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5490999999999998e-05 2020-06-12T02:37:48.363 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.364 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.364 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:48.364 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.170310+0000", 2020-06-12T02:37:48.364 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003215828 2020-06-12T02:37:48.364 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.364 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.365 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.365 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.173526+0000", 2020-06-12T02:37:48.365 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0290000000000005e-05 2020-06-12T02:37:48.365 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.365 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.366 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:48.366 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.173616+0000", 2020-06-12T02:37:48.366 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002984221 2020-06-12T02:37:48.366 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.366 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.366 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:48.367 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.176600+0000", 2020-06-12T02:37:48.367 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016812273999999999 2020-06-12T02:37:48.367 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:48.367 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:48.367 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:48.368 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.368 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.368 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:6431 3.9s0 3:93ea0cad:::smithi10729407-19:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-06-12T02:37:48.368 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:47.960027+0000", 2020-06-12T02:37:48.368 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.119689351, 2020-06-12T02:37:48.368 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014549095, 2020-06-12T02:37:48.369 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:48.369 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:48.369 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:48.369 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:48.369 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:48.370 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 6431 2020-06-12T02:37:48.370 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.370 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:48.370 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.370 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:48.370 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960027+0000", 2020-06-12T02:37:48.371 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:48.371 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.371 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.371 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:48.371 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960027+0000", 2020-06-12T02:37:48.372 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9313000000000001e-05 2020-06-12T02:37:48.372 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.372 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.372 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:48.372 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960066+0000", 2020-06-12T02:37:48.372 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0790000000000002e-06 2020-06-12T02:37:48.373 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.373 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.373 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:48.373 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960070+0000", 2020-06-12T02:37:48.373 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4510000000000003e-06 2020-06-12T02:37:48.374 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.374 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.374 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:48.374 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960078+0000", 2020-06-12T02:37:48.374 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4140000000000001e-06 2020-06-12T02:37:48.374 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.375 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.375 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:48.375 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960080+0000", 2020-06-12T02:37:48.375 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004337268 2020-06-12T02:37:48.375 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.375 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.376 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.376 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.964417+0000", 2020-06-12T02:37:48.376 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010161 2020-06-12T02:37:48.376 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.376 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.377 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:48.377 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.964519+0000", 2020-06-12T02:37:48.377 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024899399999999999 2020-06-12T02:37:48.377 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.377 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.377 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.378 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.967009+0000", 2020-06-12T02:37:48.378 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7747000000000001e-05 2020-06-12T02:37:48.378 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.378 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.378 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:48.379 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.967067+0000", 2020-06-12T02:37:48.379 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00025406499999999999 2020-06-12T02:37:48.379 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.379 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.379 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:48.379 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.967321+0000", 2020-06-12T02:37:48.380 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053713999999999997 2020-06-12T02:37:48.380 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.380 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.380 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:48.380 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.967858+0000", 2020-06-12T02:37:48.381 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0065322330000000001 2020-06-12T02:37:48.381 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.381 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.381 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:48.381 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.974390+0000", 2020-06-12T02:37:48.381 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000185835 2020-06-12T02:37:48.382 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.382 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.382 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:48.382 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.974576+0000", 2020-06-12T02:37:48.382 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014549095 2020-06-12T02:37:48.383 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:48.383 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:48.383 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:48.383 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.383 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.384 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:6432 3.9s0 3:93ea0cad:::smithi10729407-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e31)", 2020-06-12T02:37:48.384 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:47.960104+0000", 2020-06-12T02:37:48.384 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.11961208299999999, 2020-06-12T02:37:48.384 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016025815999999998, 2020-06-12T02:37:48.384 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:48.384 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:48.385 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:48.385 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:48.385 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:48.385 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 6432 2020-06-12T02:37:48.385 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.386 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:48.386 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.386 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:48.386 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960104+0000", 2020-06-12T02:37:48.386 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:48.386 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.387 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.387 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:48.387 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960104+0000", 2020-06-12T02:37:48.387 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2351999999999999e-05 2020-06-12T02:37:48.387 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.388 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.388 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:48.388 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960126+0000", 2020-06-12T02:37:48.388 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.061e-06 2020-06-12T02:37:48.388 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.388 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.389 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:48.389 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960129+0000", 2020-06-12T02:37:48.389 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4849999999999997e-06 2020-06-12T02:37:48.389 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.389 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.390 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:48.390 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960136+0000", 2020-06-12T02:37:48.390 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.176e-06 2020-06-12T02:37:48.390 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.390 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.390 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:48.391 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960138+0000", 2020-06-12T02:37:48.391 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0044054139999999999 2020-06-12T02:37:48.391 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.391 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.391 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.392 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.964543+0000", 2020-06-12T02:37:48.392 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8077000000000006e-05 2020-06-12T02:37:48.392 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.392 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.392 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:48.392 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.964612+0000", 2020-06-12T02:37:48.393 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0029921150000000001 2020-06-12T02:37:48.393 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.393 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.393 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.393 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.967604+0000", 2020-06-12T02:37:48.394 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2983000000000001e-05 2020-06-12T02:37:48.394 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.394 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.394 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:48.394 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.967697+0000", 2020-06-12T02:37:48.394 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0072496510000000002 2020-06-12T02:37:48.395 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.395 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.395 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:48.395 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.974946+0000", 2020-06-12T02:37:48.395 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8278e-05 2020-06-12T02:37:48.395 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.396 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.396 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:48.396 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.975005+0000", 2020-06-12T02:37:48.396 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001125224 2020-06-12T02:37:48.396 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:48.397 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:48.397 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:48.397 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.976130+0000", 2020-06-12T02:37:48.397 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016025815999999998 2020-06-12T02:37:48.397 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:48.397 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:48.398 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:48.398 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:48.398 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:48.398 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:37:48.413 INFO:tasks.rados.rados.0.smithi107.stdout:3073: writing smithi10729407-3 from 0 to 655360 tid 1 2020-06-12T02:37:48.414 INFO:tasks.rados.rados.0.smithi107.stdout:3073: writing smithi10729407-3 from 655360 to 1056768 tid 2 2020-06-12T02:37:48.414 INFO:tasks.rados.rados.0.smithi107.stdout:3074: delete oid 13 current snap is 0 2020-06-12T02:37:48.414 INFO:tasks.rados.rados.0.smithi107.stdout:3072: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:48.414 INFO:tasks.rados.rados.0.smithi107.stdout:3072: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:48.414 INFO:tasks.rados.rados.0.smithi107.stdout:3072: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:48.415 INFO:tasks.rados.rados.0.smithi107.stdout:3074: done (2 left) 2020-06-12T02:37:48.415 INFO:tasks.rados.rados.0.smithi107.stdout:3075: delete oid 18 current snap is 0 2020-06-12T02:37:48.415 INFO:tasks.rados.rados.0.smithi107.stdout:3072: finishing write tid 5 to smithi10729407-9 2020-06-12T02:37:48.415 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 554 (ObjNum 3071 snap 0 seq_num 3071) dirty exists 2020-06-12T02:37:48.415 INFO:tasks.rados.rados.0.smithi107.stdout:3072: left oid 9 (ObjNum 3071 snap 0 seq_num 3071) 2020-06-12T02:37:48.415 INFO:tasks.rados.rados.0.smithi107.stdout:3073: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:48.416 INFO:tasks.rados.rados.0.smithi107.stdout:3073: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:48.416 INFO:tasks.rados.rados.0.smithi107.stdout:3073: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:48.416 INFO:tasks.rados.rados.0.smithi107.stdout:3073: finishing write tid 5 to smithi10729407-3 2020-06-12T02:37:48.416 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 528 (ObjNum 3072 snap 0 seq_num 3072) dirty exists 2020-06-12T02:37:48.416 INFO:tasks.rados.rados.0.smithi107.stdout:3073: left oid 3 (ObjNum 3072 snap 0 seq_num 3072) 2020-06-12T02:37:48.417 INFO:tasks.rados.rados.0.smithi107.stdout:3072: done (2 left) 2020-06-12T02:37:48.417 INFO:tasks.rados.rados.0.smithi107.stdout:3073: done (1 left) 2020-06-12T02:37:48.417 INFO:tasks.rados.rados.0.smithi107.stdout:3075: done (0 left) 2020-06-12T02:37:48.417 INFO:tasks.rados.rados.0.smithi107.stdout:3076: delete oid 18 current snap is 0 2020-06-12T02:37:48.417 INFO:tasks.rados.rados.0.smithi107.stdout:3076: done (0 left) 2020-06-12T02:37:48.417 INFO:tasks.rados.rados.0.smithi107.stdout:3077: delete oid 3 current snap is 0 2020-06-12T02:37:48.418 INFO:tasks.rados.rados.0.smithi107.stdout:3077: done (0 left) 2020-06-12T02:37:48.418 INFO:tasks.rados.rados.0.smithi107.stdout:3078: delete oid 15 current snap is 0 2020-06-12T02:37:48.418 INFO:tasks.rados.rados.0.smithi107.stdout:3078: done (0 left) 2020-06-12T02:37:48.418 INFO:tasks.rados.rados.0.smithi107.stdout:3079: delete oid 2 current snap is 0 2020-06-12T02:37:48.418 INFO:tasks.rados.rados.0.smithi107.stdout:3079: done (0 left) 2020-06-12T02:37:48.419 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:37:48.419 INFO:tasks.rados.rados.0.smithi107.stdout:3080: seq_num 3079 ranges {0=770048,770048=401408} 2020-06-12T02:37:48.419 INFO:tasks.rados.rados.0.smithi107.stdout:3080: writing smithi10729407-19 from 0 to 770048 tid 1 2020-06-12T02:37:48.419 INFO:tasks.rados.rados.0.smithi107.stdout:3080: writing smithi10729407-19 from 770048 to 1171456 tid 2 2020-06-12T02:37:48.419 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:37:48.419 INFO:tasks.rados.rados.0.smithi107.stdout:3081: seq_num 3080 ranges {1376256=794624,2170880=401408} 2020-06-12T02:37:48.420 INFO:tasks.rados.rados.0.smithi107.stdout:3081: writing smithi10729407-10 from 1376256 to 2170880 tid 1 2020-06-12T02:37:48.420 INFO:tasks.rados.rados.0.smithi107.stdout:3081: writing smithi10729407-10 from 2170880 to 2572288 tid 2 2020-06-12T02:37:48.420 INFO:tasks.rados.rados.0.smithi107.stdout:3082: delete oid 9 current snap is 0 2020-06-12T02:37:48.420 INFO:tasks.rados.rados.0.smithi107.stdout:3080: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:48.420 INFO:tasks.rados.rados.0.smithi107.stdout:3080: finishing write tid 2 to smithi10729407-19 2020-06-12T02:37:48.421 INFO:tasks.rados.rados.0.smithi107.stdout:3080: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:48.421 INFO:tasks.rados.rados.0.smithi107.stdout:3080: finishing write tid 5 to smithi10729407-19 2020-06-12T02:37:48.421 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 500 (ObjNum 3079 snap 0 seq_num 3079) dirty exists 2020-06-12T02:37:48.421 INFO:tasks.rados.rados.0.smithi107.stdout:3080: left oid 19 (ObjNum 3079 snap 0 seq_num 3079) 2020-06-12T02:37:48.421 INFO:tasks.rados.rados.0.smithi107.stdout:3080: done (2 left) 2020-06-12T02:37:48.421 INFO:tasks.rados.rados.0.smithi107.stdout:3082: done (1 left) 2020-06-12T02:37:48.422 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:37:48.422 INFO:tasks.rados.rados.0.smithi107.stdout:3083: seq_num 3082 ranges {0=425984,425984=434176,860160=368640} 2020-06-12T02:37:48.422 INFO:tasks.rados.rados.0.smithi107.stdout:3083: writing smithi10729407-3 from 0 to 425984 tid 1 2020-06-12T02:37:48.422 INFO:tasks.rados.rados.0.smithi107.stdout:3083: writing smithi10729407-3 from 425984 to 860160 tid 2 2020-06-12T02:37:48.422 INFO:tasks.rados.rados.0.smithi107.stdout:3083: writing smithi10729407-3 from 860160 to 1228800 tid 3 2020-06-12T02:37:48.423 INFO:tasks.rados.rados.0.smithi107.stdout:3084: delete oid 13 current snap is 0 2020-06-12T02:37:48.423 INFO:tasks.rados.rados.0.smithi107.stdout:3081: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:48.423 INFO:tasks.rados.rados.0.smithi107.stdout:3083: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:48.423 INFO:tasks.rados.rados.0.smithi107.stdout:3081: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:48.423 INFO:tasks.rados.rados.0.smithi107.stdout:3081: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:48.423 INFO:tasks.rados.rados.0.smithi107.stdout:3081: finishing write tid 5 to smithi10729407-10 2020-06-12T02:37:48.424 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 301 (ObjNum 3080 snap 0 seq_num 3080) dirty exists 2020-06-12T02:37:48.424 INFO:tasks.rados.rados.0.smithi107.stdout:3081: left oid 10 (ObjNum 3080 snap 0 seq_num 3080) 2020-06-12T02:37:48.424 INFO:tasks.rados.rados.0.smithi107.stdout:3081: done (2 left) 2020-06-12T02:37:48.424 INFO:tasks.rados.rados.0.smithi107.stdout:3084: done (1 left) 2020-06-12T02:37:48.424 INFO:tasks.rados.rados.0.smithi107.stdout:3085: delete oid 11 current snap is 0 2020-06-12T02:37:48.425 INFO:tasks.rados.rados.0.smithi107.stdout:3083: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:48.425 INFO:tasks.rados.rados.0.smithi107.stdout:3085: done (1 left) 2020-06-12T02:37:48.425 INFO:tasks.rados.rados.0.smithi107.stdout:3086: delete oid 6 current snap is 0 2020-06-12T02:37:48.425 INFO:tasks.rados.rados.0.smithi107.stdout:3083: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:48.425 INFO:tasks.rados.rados.0.smithi107.stdout:3083: finishing write tid 5 to smithi10729407-3 2020-06-12T02:37:48.425 INFO:tasks.rados.rados.0.smithi107.stdout:3086: done (1 left) 2020-06-12T02:37:48.426 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:37:48.426 INFO:tasks.rados.rados.0.smithi107.stdout:3087: seq_num 3086 ranges {991232=491520,1482752=778240,2260992=24576} 2020-06-12T02:37:48.426 INFO:tasks.rados.rados.0.smithi107.stdout:3087: writing smithi10729407-16 from 991232 to 1482752 tid 1 2020-06-12T02:37:48.426 INFO:tasks.rados.rados.0.smithi107.stdout:3087: writing smithi10729407-16 from 1482752 to 2260992 tid 2 2020-06-12T02:37:48.426 INFO:tasks.rados.rados.0.smithi107.stdout:3087: writing smithi10729407-16 from 2260992 to 2285568 tid 3 2020-06-12T02:37:48.427 INFO:tasks.rados.rados.0.smithi107.stdout:3088: delete oid 4 current snap is 0 2020-06-12T02:37:48.427 INFO:tasks.rados.rados.0.smithi107.stdout:3083: finishing write tid 6 to smithi10729407-3 2020-06-12T02:37:48.427 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 533 (ObjNum 3082 snap 0 seq_num 3082) dirty exists 2020-06-12T02:37:48.427 INFO:tasks.rados.rados.0.smithi107.stdout:3083: left oid 3 (ObjNum 3082 snap 0 seq_num 3082) 2020-06-12T02:37:48.427 INFO:tasks.rados.rados.0.smithi107.stdout:3087: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:48.427 INFO:tasks.rados.rados.0.smithi107.stdout:3083: done (2 left) 2020-06-12T02:37:48.428 INFO:tasks.rados.rados.0.smithi107.stdout:3088: done (1 left) 2020-06-12T02:37:48.428 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:37:48.428 INFO:tasks.rados.rados.0.smithi107.stdout:3089: seq_num 3088 ranges {0=524288,524288=589824,1114112=212992} 2020-06-12T02:37:48.428 INFO:tasks.rados.rados.0.smithi107.stdout:3089: writing smithi10729407-4 from 0 to 524288 tid 1 2020-06-12T02:37:48.428 INFO:tasks.rados.rados.0.smithi107.stdout:3089: writing smithi10729407-4 from 524288 to 1114112 tid 2 2020-06-12T02:37:48.429 INFO:tasks.rados.rados.0.smithi107.stdout:3089: writing smithi10729407-4 from 1114112 to 1327104 tid 3 2020-06-12T02:37:48.429 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:37:48.429 INFO:tasks.rados.rados.0.smithi107.stdout:3090: seq_num 3089 ranges {0=540672,540672=475136,1015808=327680} 2020-06-12T02:37:48.429 INFO:tasks.rados.rados.0.smithi107.stdout:3090: writing smithi10729407-12 from 0 to 540672 tid 1 2020-06-12T02:37:48.429 INFO:tasks.rados.rados.0.smithi107.stdout:3090: writing smithi10729407-12 from 540672 to 1015808 tid 2 2020-06-12T02:37:48.429 INFO:tasks.rados.rados.0.smithi107.stdout:3090: writing smithi10729407-12 from 1015808 to 1343488 tid 3 2020-06-12T02:37:48.430 INFO:tasks.rados.rados.0.smithi107.stdout:3091: delete oid 18 current snap is 0 2020-06-12T02:37:48.430 INFO:tasks.rados.rados.0.smithi107.stdout:3087: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:48.430 INFO:tasks.rados.rados.0.smithi107.stdout:3087: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:48.430 INFO:tasks.rados.rados.0.smithi107.stdout:3087: finishing write tid 5 to smithi10729407-16 2020-06-12T02:37:48.430 INFO:tasks.rados.rados.0.smithi107.stdout:3087: finishing write tid 6 to smithi10729407-16 2020-06-12T02:37:48.431 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 537 (ObjNum 3086 snap 0 seq_num 3086) dirty exists 2020-06-12T02:37:48.431 INFO:tasks.rados.rados.0.smithi107.stdout:3087: left oid 16 (ObjNum 3086 snap 0 seq_num 3086) 2020-06-12T02:37:48.431 INFO:tasks.rados.rados.0.smithi107.stdout:3089: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:48.431 INFO:tasks.rados.rados.0.smithi107.stdout:3089: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:48.431 INFO:tasks.rados.rados.0.smithi107.stdout:3089: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:48.431 INFO:tasks.rados.rados.0.smithi107.stdout:3089: finishing write tid 5 to smithi10729407-4 2020-06-12T02:37:48.432 INFO:tasks.rados.rados.0.smithi107.stdout:3090: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:48.432 INFO:tasks.rados.rados.0.smithi107.stdout:3087: done (3 left) 2020-06-12T02:37:48.432 INFO:tasks.rados.rados.0.smithi107.stdout:3091: done (2 left) 2020-06-12T02:37:48.432 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:37:48.432 INFO:tasks.rados.rados.0.smithi107.stdout:3092: seq_num 3091 ranges {0=573440,573440=647168,1220608=155648} 2020-06-12T02:37:48.433 INFO:tasks.rados.rados.0.smithi107.stdout:3092: writing smithi10729407-17 from 0 to 573440 tid 1 2020-06-12T02:37:48.433 INFO:tasks.rados.rados.0.smithi107.stdout:3092: writing smithi10729407-17 from 573440 to 1220608 tid 2 2020-06-12T02:37:48.433 INFO:tasks.rados.rados.0.smithi107.stdout:3092: writing smithi10729407-17 from 1220608 to 1376256 tid 3 2020-06-12T02:37:48.433 INFO:tasks.rados.rados.0.smithi107.stdout:3093: delete oid 5 current snap is 0 2020-06-12T02:37:48.433 INFO:tasks.rados.rados.0.smithi107.stdout:3089: finishing write tid 6 to smithi10729407-4 2020-06-12T02:37:48.434 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 596 (ObjNum 3088 snap 0 seq_num 3088) dirty exists 2020-06-12T02:37:48.434 INFO:tasks.rados.rados.0.smithi107.stdout:3089: left oid 4 (ObjNum 3088 snap 0 seq_num 3088) 2020-06-12T02:37:48.434 INFO:tasks.rados.rados.0.smithi107.stdout:3090: finishing write tid 2 to smithi10729407-12 2020-06-12T02:37:48.434 INFO:tasks.rados.rados.0.smithi107.stdout:3090: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:48.434 INFO:tasks.rados.rados.0.smithi107.stdout:3090: finishing write tid 5 to smithi10729407-12 2020-06-12T02:37:48.434 INFO:tasks.rados.rados.0.smithi107.stdout:3090: finishing write tid 6 to smithi10729407-12 2020-06-12T02:37:48.435 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 504 (ObjNum 3089 snap 0 seq_num 3089) dirty exists 2020-06-12T02:37:48.435 INFO:tasks.rados.rados.0.smithi107.stdout:3090: left oid 12 (ObjNum 3089 snap 0 seq_num 3089) 2020-06-12T02:37:48.435 INFO:tasks.rados.rados.0.smithi107.stdout:3089: done (3 left) 2020-06-12T02:37:48.435 INFO:tasks.rados.rados.0.smithi107.stdout:3090: done (2 left) 2020-06-12T02:37:48.435 INFO:tasks.rados.rados.0.smithi107.stdout:3093: done (1 left) 2020-06-12T02:37:48.435 INFO:tasks.rados.rados.0.smithi107.stdout:3094: delete oid 5 current snap is 0 2020-06-12T02:37:48.436 INFO:tasks.rados.rados.0.smithi107.stdout:3094: done (1 left) 2020-06-12T02:37:48.436 INFO:tasks.rados.rados.0.smithi107.stdout:3095: delete oid 15 current snap is 0 2020-06-12T02:37:48.436 INFO:tasks.rados.rados.0.smithi107.stdout:3092: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:48.436 INFO:tasks.rados.rados.0.smithi107.stdout:3092: finishing write tid 2 to smithi10729407-17 2020-06-12T02:37:48.436 INFO:tasks.rados.rados.0.smithi107.stdout:3092: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:48.437 INFO:tasks.rados.rados.0.smithi107.stdout:3095: done (1 left) 2020-06-12T02:37:48.437 INFO:tasks.rados.rados.0.smithi107.stdout:3096: delete oid 13 current snap is 0 2020-06-12T02:37:48.437 INFO:tasks.rados.rados.0.smithi107.stdout:3092: finishing write tid 5 to smithi10729407-17 2020-06-12T02:37:48.437 INFO:tasks.rados.rados.0.smithi107.stdout:3092: finishing write tid 6 to smithi10729407-17 2020-06-12T02:37:48.437 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 600 (ObjNum 3091 snap 0 seq_num 3091) dirty exists 2020-06-12T02:37:48.437 INFO:tasks.rados.rados.0.smithi107.stdout:3092: left oid 17 (ObjNum 3091 snap 0 seq_num 3091) 2020-06-12T02:37:48.438 INFO:tasks.rados.rados.0.smithi107.stdout:3092: done (1 left) 2020-06-12T02:37:48.438 INFO:tasks.rados.rados.0.smithi107.stdout:3096: done (0 left) 2020-06-12T02:37:48.438 INFO:tasks.rados.rados.0.smithi107.stdout:3097: delete oid 14 current snap is 0 2020-06-12T02:37:48.438 INFO:tasks.rados.rados.0.smithi107.stdout:3097: done (0 left) 2020-06-12T02:37:48.438 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:48.439 INFO:tasks.rados.rados.0.smithi107.stdout:3098: seq_num 3097 ranges {0=671744,671744=802816} 2020-06-12T02:37:48.439 INFO:tasks.rados.rados.0.smithi107.stdout:3098: writing smithi10729407-2 from 0 to 671744 tid 1 2020-06-12T02:37:48.439 INFO:tasks.rados.rados.0.smithi107.stdout:3098: writing smithi10729407-2 from 671744 to 1474560 tid 2 2020-06-12T02:37:48.439 INFO:tasks.rados.rados.0.smithi107.stdout:3099: delete oid 6 current snap is 0 2020-06-12T02:37:48.439 INFO:tasks.rados.rados.0.smithi107.stdout:3098: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:48.439 INFO:tasks.rados.rados.0.smithi107.stdout:3099: done (1 left) 2020-06-12T02:37:48.440 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:37:48.440 INFO:tasks.rados.rados.0.smithi107.stdout:3100: seq_num 3099 ranges {2285568=712704,2998272=573440,3571712=229376} 2020-06-12T02:37:48.440 INFO:tasks.rados.rados.0.smithi107.stdout:3100: writing smithi10729407-16 from 2285568 to 2998272 tid 1 2020-06-12T02:37:48.440 INFO:tasks.rados.rados.0.smithi107.stdout:3100: writing smithi10729407-16 from 2998272 to 3571712 tid 2 2020-06-12T02:37:48.440 INFO:tasks.rados.rados.0.smithi107.stdout:3100: writing smithi10729407-16 from 3571712 to 3801088 tid 3 2020-06-12T02:37:48.441 INFO:tasks.rados.rados.0.smithi107.stdout:3101: delete oid 10 current snap is 0 2020-06-12T02:37:48.441 INFO:tasks.rados.rados.0.smithi107.stdout:3098: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:48.441 INFO:tasks.rados.rados.0.smithi107.stdout:3098: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:48.441 INFO:tasks.rados.rados.0.smithi107.stdout:3098: finishing write tid 5 to smithi10729407-2 2020-06-12T02:37:48.441 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 318 (ObjNum 3097 snap 0 seq_num 3097) dirty exists 2020-06-12T02:37:48.441 INFO:tasks.rados.rados.0.smithi107.stdout:3098: left oid 2 (ObjNum 3097 snap 0 seq_num 3097) 2020-06-12T02:37:48.442 INFO:tasks.rados.rados.0.smithi107.stdout:3100: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:48.442 INFO:tasks.rados.rados.0.smithi107.stdout:3098: done (2 left) 2020-06-12T02:37:48.442 INFO:tasks.rados.rados.0.smithi107.stdout:3101: done (1 left) 2020-06-12T02:37:48.442 INFO:tasks.rados.rados.0.smithi107.stdout:3102: delete oid 10 current snap is 0 2020-06-12T02:37:48.442 INFO:tasks.rados.rados.0.smithi107.stdout:3102: done (1 left) 2020-06-12T02:37:48.442 INFO:tasks.rados.rados.0.smithi107.stdout:3103: delete oid 8 current snap is 0 2020-06-12T02:37:48.443 INFO:tasks.rados.rados.0.smithi107.stdout:3103: done (1 left) 2020-06-12T02:37:48.443 INFO:tasks.rados.rados.0.smithi107.stdout:3104: delete oid 7 current snap is 0 2020-06-12T02:37:48.443 INFO:tasks.rados.rados.0.smithi107.stdout:3100: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:48.443 INFO:tasks.rados.rados.0.smithi107.stdout:3104: done (1 left) 2020-06-12T02:37:48.443 INFO:tasks.rados.rados.0.smithi107.stdout:3105: delete oid 4 current snap is 0 2020-06-12T02:37:48.444 INFO:tasks.rados.rados.0.smithi107.stdout:3100: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:48.444 INFO:tasks.rados.rados.0.smithi107.stdout:3100: finishing write tid 5 to smithi10729407-16 2020-06-12T02:37:48.444 INFO:tasks.rados.rados.0.smithi107.stdout:3105: done (1 left) 2020-06-12T02:37:48.444 INFO:tasks.rados.rados.0.smithi107.stdout:3106: delete oid 8 current snap is 0 2020-06-12T02:37:48.444 INFO:tasks.rados.rados.0.smithi107.stdout:3100: finishing write tid 6 to smithi10729407-16 2020-06-12T02:37:48.444 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 541 (ObjNum 3099 snap 0 seq_num 3099) dirty exists 2020-06-12T02:37:48.445 INFO:tasks.rados.rados.0.smithi107.stdout:3100: left oid 16 (ObjNum 3099 snap 0 seq_num 3099) 2020-06-12T02:37:48.445 INFO:tasks.rados.rados.0.smithi107.stdout:3100: done (1 left) 2020-06-12T02:37:48.445 INFO:tasks.rados.rados.0.smithi107.stdout:3106: done (0 left) 2020-06-12T02:37:48.445 INFO:tasks.rados.rados.0.smithi107.stdout:3107: delete oid 11 current snap is 0 2020-06-12T02:37:48.445 INFO:tasks.rados.rados.0.smithi107.stdout:3107: done (0 left) 2020-06-12T02:37:48.445 INFO:tasks.rados.rados.0.smithi107.stdout:3108: delete oid 1 current snap is 0 2020-06-12T02:37:48.446 INFO:tasks.rados.rados.0.smithi107.stdout:3108: done (0 left) 2020-06-12T02:37:48.446 INFO:tasks.rados.rados.0.smithi107.stdout:3109: delete oid 1 current snap is 0 2020-06-12T02:37:48.446 INFO:tasks.rados.rados.0.smithi107.stdout:3109: done (0 left) 2020-06-12T02:37:48.446 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:37:48.446 INFO:tasks.rados.rados.0.smithi107.stdout:3110: seq_num 3109 ranges {0=475136,475136=663552,1138688=540672} 2020-06-12T02:37:48.447 INFO:tasks.rados.rados.0.smithi107.stdout:3110: writing smithi10729407-18 from 0 to 475136 tid 1 2020-06-12T02:37:48.447 INFO:tasks.rados.rados.0.smithi107.stdout:3110: writing smithi10729407-18 from 475136 to 1138688 tid 2 2020-06-12T02:37:48.447 INFO:tasks.rados.rados.0.smithi107.stdout:3110: writing smithi10729407-18 from 1138688 to 1679360 tid 3 2020-06-12T02:37:48.447 INFO:tasks.rados.rados.0.smithi107.stdout:3111: delete oid 19 current snap is 0 2020-06-12T02:37:48.447 INFO:tasks.rados.rados.0.smithi107.stdout:3110: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:48.448 INFO:tasks.rados.rados.0.smithi107.stdout:3111: done (1 left) 2020-06-12T02:37:48.448 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:37:48.448 INFO:tasks.rados.rados.0.smithi107.stdout:3112: seq_num 3111 ranges {0=507904,507904=483328,991232=630784,1622016=90112} 2020-06-12T02:37:48.448 INFO:tasks.rados.rados.0.smithi107.stdout:3112: writing smithi10729407-14 from 0 to 507904 tid 1 2020-06-12T02:37:48.448 INFO:tasks.rados.rados.0.smithi107.stdout:3112: writing smithi10729407-14 from 507904 to 991232 tid 2 2020-06-12T02:37:48.448 INFO:tasks.rados.rados.0.smithi107.stdout:3112: writing smithi10729407-14 from 991232 to 1622016 tid 3 2020-06-12T02:37:48.449 INFO:tasks.rados.rados.0.smithi107.stdout:3112: writing smithi10729407-14 from 1622016 to 1712128 tid 4 2020-06-12T02:37:48.449 INFO:tasks.rados.rados.0.smithi107.stdout:3113: delete oid 16 current snap is 0 2020-06-12T02:37:48.449 INFO:tasks.rados.rados.0.smithi107.stdout:3110: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:48.449 INFO:tasks.rados.rados.0.smithi107.stdout:3110: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:48.449 INFO:tasks.rados.rados.0.smithi107.stdout:3110: finishing write tid 5 to smithi10729407-18 2020-06-12T02:37:48.449 INFO:tasks.rados.rados.0.smithi107.stdout:3110: finishing write tid 6 to smithi10729407-18 2020-06-12T02:37:48.450 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 562 (ObjNum 3109 snap 0 seq_num 3109) dirty exists 2020-06-12T02:37:48.450 INFO:tasks.rados.rados.0.smithi107.stdout:3110: left oid 18 (ObjNum 3109 snap 0 seq_num 3109) 2020-06-12T02:37:48.450 INFO:tasks.rados.rados.0.smithi107.stdout:3112: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:48.450 INFO:tasks.rados.rados.0.smithi107.stdout:3110: done (2 left) 2020-06-12T02:37:48.450 INFO:tasks.rados.rados.0.smithi107.stdout:3113: done (1 left) 2020-06-12T02:37:48.451 INFO:tasks.rados.rados.0.smithi107.stdout:3114: delete oid 4 current snap is 0 2020-06-12T02:37:48.451 INFO:tasks.rados.rados.0.smithi107.stdout:3114: done (1 left) 2020-06-12T02:37:48.451 INFO:tasks.rados.rados.0.smithi107.stdout:3115: delete oid 18 current snap is 0 2020-06-12T02:37:48.451 INFO:tasks.rados.rados.0.smithi107.stdout:3112: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:48.451 INFO:tasks.rados.rados.0.smithi107.stdout:3112: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:48.451 INFO:tasks.rados.rados.0.smithi107.stdout:3115: done (1 left) 2020-06-12T02:37:48.452 INFO:tasks.rados.rados.0.smithi107.stdout:3116: delete oid 8 current snap is 0 2020-06-12T02:37:48.452 INFO:tasks.rados.rados.0.smithi107.stdout:3112: finishing write tid 4 to smithi10729407-14 2020-06-12T02:37:48.452 INFO:tasks.rados.rados.0.smithi107.stdout:3112: finishing write tid 6 to smithi10729407-14 2020-06-12T02:37:48.452 INFO:tasks.rados.rados.0.smithi107.stdout:3116: done (1 left) 2020-06-12T02:37:48.452 INFO:tasks.rados.rados.0.smithi107.stdout:3117: delete oid 11 current snap is 0 2020-06-12T02:37:48.453 INFO:tasks.rados.rados.0.smithi107.stdout:3117: done (1 left) 2020-06-12T02:37:48.453 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:37:48.453 INFO:tasks.rados.rados.0.smithi107.stdout:3118: seq_num 3117 ranges {0=606208,606208=638976,1245184=565248} 2020-06-12T02:37:48.453 INFO:tasks.rados.rados.0.smithi107.stdout:3118: writing smithi10729407-10 from 0 to 606208 tid 1 2020-06-12T02:37:48.453 INFO:tasks.rados.rados.0.smithi107.stdout:3118: writing smithi10729407-10 from 606208 to 1245184 tid 2 2020-06-12T02:37:48.453 INFO:tasks.rados.rados.0.smithi107.stdout:3118: writing smithi10729407-10 from 1245184 to 1810432 tid 3 2020-06-12T02:37:48.454 INFO:tasks.rados.rados.0.smithi107.stdout:3119: delete oid 1 current snap is 0 2020-06-12T02:37:48.454 INFO:tasks.rados.rados.0.smithi107.stdout:3112: finishing write tid 7 to smithi10729407-14 2020-06-12T02:37:48.454 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 516 (ObjNum 3111 snap 0 seq_num 3111) dirty exists 2020-06-12T02:37:48.454 INFO:tasks.rados.rados.0.smithi107.stdout:3112: left oid 14 (ObjNum 3111 snap 0 seq_num 3111) 2020-06-12T02:37:48.454 INFO:tasks.rados.rados.0.smithi107.stdout:3118: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:48.454 INFO:tasks.rados.rados.0.smithi107.stdout:3112: done (2 left) 2020-06-12T02:37:48.455 INFO:tasks.rados.rados.0.smithi107.stdout:3119: done (1 left) 2020-06-12T02:37:48.455 INFO:tasks.rados.rados.0.smithi107.stdout:3120: delete oid 2 current snap is 0 2020-06-12T02:37:48.455 INFO:tasks.rados.rados.0.smithi107.stdout:3118: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:48.455 INFO:tasks.rados.rados.0.smithi107.stdout:3118: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:48.455 INFO:tasks.rados.rados.0.smithi107.stdout:3118: finishing write tid 5 to smithi10729407-10 2020-06-12T02:37:48.456 INFO:tasks.rados.rados.0.smithi107.stdout:3120: done (1 left) 2020-06-12T02:37:48.456 INFO:tasks.rados.rados.0.smithi107.stdout:3121: delete oid 19 current snap is 0 2020-06-12T02:37:48.456 INFO:tasks.rados.rados.0.smithi107.stdout:3118: finishing write tid 6 to smithi10729407-10 2020-06-12T02:37:48.456 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 307 (ObjNum 3117 snap 0 seq_num 3117) dirty exists 2020-06-12T02:37:48.456 INFO:tasks.rados.rados.0.smithi107.stdout:3118: left oid 10 (ObjNum 3117 snap 0 seq_num 3117) 2020-06-12T02:37:48.456 INFO:tasks.rados.rados.0.smithi107.stdout:3118: done (1 left) 2020-06-12T02:37:48.457 INFO:tasks.rados.rados.0.smithi107.stdout:3121: done (0 left) 2020-06-12T02:37:48.457 INFO:tasks.rados.rados.0.smithi107.stdout:3122: delete oid 17 current snap is 0 2020-06-12T02:37:48.457 INFO:tasks.rados.rados.0.smithi107.stdout:3122: done (0 left) 2020-06-12T02:37:48.457 INFO:tasks.rados.rados.0.smithi107.stdout:3123: delete oid 19 current snap is 0 2020-06-12T02:37:48.457 INFO:tasks.rados.rados.0.smithi107.stdout:3123: done (0 left) 2020-06-12T02:37:48.458 INFO:tasks.rados.rados.0.smithi107.stdout:3124: delete oid 19 current snap is 0 2020-06-12T02:37:48.458 INFO:tasks.rados.rados.0.smithi107.stdout:3124: done (0 left) 2020-06-12T02:37:48.458 INFO:tasks.rados.rados.0.smithi107.stdout:3125: delete oid 1 current snap is 0 2020-06-12T02:37:48.458 INFO:tasks.rados.rados.0.smithi107.stdout:3125: done (0 left) 2020-06-12T02:37:48.458 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:37:48.458 INFO:tasks.rados.rados.0.smithi107.stdout:3126: seq_num 3125 ranges {0=745472,745472=565248,1310720=581632,1892352=57344} 2020-06-12T02:37:48.459 INFO:tasks.rados.rados.0.smithi107.stdout:3126: writing smithi10729407-13 from 0 to 745472 tid 1 2020-06-12T02:37:48.459 INFO:tasks.rados.rados.0.smithi107.stdout:3126: writing smithi10729407-13 from 745472 to 1310720 tid 2 2020-06-12T02:37:48.459 INFO:tasks.rados.rados.0.smithi107.stdout:3126: writing smithi10729407-13 from 1310720 to 1892352 tid 3 2020-06-12T02:37:48.459 INFO:tasks.rados.rados.0.smithi107.stdout:3126: writing smithi10729407-13 from 1892352 to 1949696 tid 4 2020-06-12T02:37:48.459 INFO:tasks.rados.rados.0.smithi107.stdout:3127: delete oid 14 current snap is 0 2020-06-12T02:37:48.460 INFO:tasks.rados.rados.0.smithi107.stdout:3126: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:48.460 INFO:tasks.rados.rados.0.smithi107.stdout:3127: done (1 left) 2020-06-12T02:37:48.460 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:37:48.460 INFO:tasks.rados.rados.0.smithi107.stdout:3128: seq_num 3127 ranges {0=778240,778240=729088,1507328=475136} 2020-06-12T02:37:48.460 INFO:tasks.rados.rados.0.smithi107.stdout:3128: writing smithi10729407-2 from 0 to 778240 tid 1 2020-06-12T02:37:48.460 INFO:tasks.rados.rados.0.smithi107.stdout:3128: writing smithi10729407-2 from 778240 to 1507328 tid 2 2020-06-12T02:37:48.461 INFO:tasks.rados.rados.0.smithi107.stdout:3128: writing smithi10729407-2 from 1507328 to 1982464 tid 3 2020-06-12T02:37:48.461 INFO:tasks.rados.rados.0.smithi107.stdout:3129: delete oid 11 current snap is 0 2020-06-12T02:37:48.461 INFO:tasks.rados.rados.0.smithi107.stdout:3126: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:48.461 INFO:tasks.rados.rados.0.smithi107.stdout:3126: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:48.461 INFO:tasks.rados.rados.0.smithi107.stdout:3126: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:48.461 INFO:tasks.rados.rados.0.smithi107.stdout:3126: finishing write tid 6 to smithi10729407-13 2020-06-12T02:37:48.462 INFO:tasks.rados.rados.0.smithi107.stdout:3126: finishing write tid 7 to smithi10729407-13 2020-06-12T02:37:48.462 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 596 (ObjNum 3125 snap 0 seq_num 3125) dirty exists 2020-06-12T02:37:48.462 INFO:tasks.rados.rados.0.smithi107.stdout:3126: left oid 13 (ObjNum 3125 snap 0 seq_num 3125) 2020-06-12T02:37:48.462 INFO:tasks.rados.rados.0.smithi107.stdout:3126: done (2 left) 2020-06-12T02:37:48.462 INFO:tasks.rados.rados.0.smithi107.stdout:3129: done (1 left) 2020-06-12T02:37:48.463 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:37:48.463 INFO:tasks.rados.rados.0.smithi107.stdout:3130: seq_num 3129 ranges {0=409600,409600=499712,909312=565248,1474560=540672} 2020-06-12T02:37:48.463 INFO:tasks.rados.rados.0.smithi107.stdout:3130: writing smithi10729407-18 from 0 to 409600 tid 1 2020-06-12T02:37:48.463 INFO:tasks.rados.rados.0.smithi107.stdout:3130: writing smithi10729407-18 from 409600 to 909312 tid 2 2020-06-12T02:37:48.463 INFO:tasks.rados.rados.0.smithi107.stdout:3130: writing smithi10729407-18 from 909312 to 1474560 tid 3 2020-06-12T02:37:48.463 INFO:tasks.rados.rados.0.smithi107.stdout:3130: writing smithi10729407-18 from 1474560 to 2015232 tid 4 2020-06-12T02:37:48.464 INFO:tasks.rados.rados.0.smithi107.stdout:3131: delete oid 7 current snap is 0 2020-06-12T02:37:48.464 INFO:tasks.rados.rados.0.smithi107.stdout:3128: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:48.464 INFO:tasks.rados.rados.0.smithi107.stdout:3128: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:48.464 INFO:tasks.rados.rados.0.smithi107.stdout:3128: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:48.464 INFO:tasks.rados.rados.0.smithi107.stdout:3128: finishing write tid 5 to smithi10729407-2 2020-06-12T02:37:48.465 INFO:tasks.rados.rados.0.smithi107.stdout:3130: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:48.465 INFO:tasks.rados.rados.0.smithi107.stdout:3128: finishing write tid 6 to smithi10729407-2 2020-06-12T02:37:48.465 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 323 (ObjNum 3127 snap 0 seq_num 3127) dirty exists 2020-06-12T02:37:48.465 INFO:tasks.rados.rados.0.smithi107.stdout:3128: left oid 2 (ObjNum 3127 snap 0 seq_num 3127) 2020-06-12T02:37:48.465 INFO:tasks.rados.rados.0.smithi107.stdout:3128: done (2 left) 2020-06-12T02:37:48.466 INFO:tasks.rados.rados.0.smithi107.stdout:3131: done (1 left) 2020-06-12T02:37:48.466 INFO:tasks.rados.rados.0.smithi107.stdout:3132: delete oid 9 current snap is 0 2020-06-12T02:37:48.466 INFO:tasks.rados.rados.0.smithi107.stdout:3130: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:48.466 INFO:tasks.rados.rados.0.smithi107.stdout:3130: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:48.466 INFO:tasks.rados.rados.0.smithi107.stdout:3130: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:48.467 INFO:tasks.rados.rados.0.smithi107.stdout:3130: finishing write tid 6 to smithi10729407-18 2020-06-12T02:37:48.467 INFO:tasks.rados.rados.0.smithi107.stdout:3132: done (1 left) 2020-06-12T02:37:48.467 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:37:48.467 INFO:tasks.rados.rados.0.smithi107.stdout:3133: seq_num 3132 ranges {1810432=458752,2269184=548864,2818048=737280,3555328=319488} 2020-06-12T02:37:48.467 INFO:tasks.rados.rados.0.smithi107.stdout:3133: writing smithi10729407-10 from 1810432 to 2269184 tid 1 2020-06-12T02:37:48.467 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2020-06-12T02:37:48.468 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:48.396+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:48.468 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-06-12T02:37:48.471 INFO:tasks.rados.rados.0.smithi107.stdout:3133: writing smithi10729407-10 from 2269184 to 2818048 tid 2 2020-06-12T02:37:48.471 INFO:tasks.rados.rados.0.smithi107.stdout:3133: writing smithi10729407-10 from 2818048 to 3555328 tid 3 2020-06-12T02:37:48.472 INFO:tasks.rados.rados.0.smithi107.stdout:3133: writing smithi10729407-10 from 3555328 to 3874816 tid 4 2020-06-12T02:37:48.472 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:37:48.472 INFO:tasks.rados.rados.0.smithi107.stdout:3134: seq_num 3133 ranges {0=475136,475136=483328,958464=548864,1507328=409600,1916928=163840} 2020-06-12T02:37:48.472 INFO:tasks.rados.rados.0.smithi107.stdout:3134: writing smithi10729407-16 from 0 to 475136 tid 1 2020-06-12T02:37:48.472 INFO:tasks.rados.rados.0.smithi107.stdout:3134: writing smithi10729407-16 from 475136 to 958464 tid 2 2020-06-12T02:37:48.473 INFO:tasks.rados.rados.0.smithi107.stdout:3134: writing smithi10729407-16 from 958464 to 1507328 tid 3 2020-06-12T02:37:48.473 INFO:tasks.rados.rados.0.smithi107.stdout:3134: writing smithi10729407-16 from 1507328 to 1916928 tid 4 2020-06-12T02:37:48.473 INFO:tasks.rados.rados.0.smithi107.stdout:3134: writing smithi10729407-16 from 1916928 to 2080768 tid 5 2020-06-12T02:37:48.473 INFO:tasks.rados.rados.0.smithi107.stdout:3135: delete oid 20 current snap is 0 2020-06-12T02:37:48.473 INFO:tasks.rados.rados.0.smithi107.stdout:3130: finishing write tid 7 to smithi10729407-18 2020-06-12T02:37:48.474 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 568 (ObjNum 3129 snap 0 seq_num 3129) dirty exists 2020-06-12T02:37:48.474 INFO:tasks.rados.rados.0.smithi107.stdout:3130: left oid 18 (ObjNum 3129 snap 0 seq_num 3129) 2020-06-12T02:37:48.474 INFO:tasks.rados.rados.0.smithi107.stdout:3133: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:48.474 INFO:tasks.rados.rados.0.smithi107.stdout:3133: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:48.474 INFO:tasks.rados.rados.0.smithi107.stdout:3133: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:48.475 INFO:tasks.rados.rados.0.smithi107.stdout:3133: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:48.475 INFO:tasks.rados.rados.0.smithi107.stdout:3133: finishing write tid 6 to smithi10729407-10 2020-06-12T02:37:48.475 INFO:tasks.rados.rados.0.smithi107.stdout:3134: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:48.475 INFO:tasks.rados.rados.0.smithi107.stdout:3133: finishing write tid 7 to smithi10729407-10 2020-06-12T02:37:48.475 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 312 (ObjNum 3132 snap 0 seq_num 3132) dirty exists 2020-06-12T02:37:48.475 INFO:tasks.rados.rados.0.smithi107.stdout:3133: left oid 10 (ObjNum 3132 snap 0 seq_num 3132) 2020-06-12T02:37:48.476 INFO:tasks.rados.rados.0.smithi107.stdout:3130: done (3 left) 2020-06-12T02:37:48.476 INFO:tasks.rados.rados.0.smithi107.stdout:3133: done (2 left) 2020-06-12T02:37:48.476 INFO:tasks.rados.rados.0.smithi107.stdout:3135: done (1 left) 2020-06-12T02:37:48.476 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:37:48.476 INFO:tasks.rados.rados.0.smithi107.stdout:3136: seq_num 3135 ranges {3874816=507904,4382720=655360,5038080=507904,5545984=442368} 2020-06-12T02:37:48.476 INFO:tasks.rados.rados.0.smithi107.stdout:3136: writing smithi10729407-10 from 3874816 to 4382720 tid 1 2020-06-12T02:37:48.477 INFO:tasks.rados.rados.0.smithi107.stdout:3136: writing smithi10729407-10 from 4382720 to 5038080 tid 2 2020-06-12T02:37:48.477 INFO:tasks.rados.rados.0.smithi107.stdout:3136: writing smithi10729407-10 from 5038080 to 5545984 tid 3 2020-06-12T02:37:48.477 INFO:tasks.rados.rados.0.smithi107.stdout:3136: writing smithi10729407-10 from 5545984 to 5988352 tid 4 2020-06-12T02:37:48.477 INFO:tasks.rados.rados.0.smithi107.stdout:3137: delete oid 6 current snap is 0 2020-06-12T02:37:48.477 INFO:tasks.rados.rados.0.smithi107.stdout:3134: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:48.478 INFO:tasks.rados.rados.0.smithi107.stdout:3134: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:48.478 INFO:tasks.rados.rados.0.smithi107.stdout:3134: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:48.478 INFO:tasks.rados.rados.0.smithi107.stdout:3134: finishing write tid 5 to smithi10729407-16 2020-06-12T02:37:48.478 INFO:tasks.rados.rados.0.smithi107.stdout:3134: finishing write tid 7 to smithi10729407-16 2020-06-12T02:37:48.478 INFO:tasks.rados.rados.0.smithi107.stdout:3134: finishing write tid 8 to smithi10729407-16 2020-06-12T02:37:48.478 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 548 (ObjNum 3133 snap 0 seq_num 3133) dirty exists 2020-06-12T02:37:48.479 INFO:tasks.rados.rados.0.smithi107.stdout:3134: left oid 16 (ObjNum 3133 snap 0 seq_num 3133) 2020-06-12T02:37:48.479 INFO:tasks.rados.rados.0.smithi107.stdout:3136: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:48.480 INFO:tasks.rados.rados.0.smithi107.stdout:3134: done (2 left) 2020-06-12T02:37:48.480 INFO:tasks.rados.rados.0.smithi107.stdout:3137: done (1 left) 2020-06-12T02:37:48.480 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:37:48.481 INFO:tasks.rados.rados.0.smithi107.stdout:3138: seq_num 3137 ranges {0=540672,540672=425984,966656=524288,1490944=458752,1949696=196608} 2020-06-12T02:37:48.481 INFO:tasks.rados.rados.0.smithi107.stdout:3138: writing smithi10729407-5 from 0 to 540672 tid 1 2020-06-12T02:37:48.481 INFO:tasks.rados.rados.0.smithi107.stdout:3138: writing smithi10729407-5 from 540672 to 966656 tid 2 2020-06-12T02:37:48.483 INFO:tasks.rados.rados.0.smithi107.stdout:3138: writing smithi10729407-5 from 966656 to 1490944 tid 3 2020-06-12T02:37:48.485 INFO:tasks.rados.rados.0.smithi107.stdout:3138: writing smithi10729407-5 from 1490944 to 1949696 tid 4 2020-06-12T02:37:48.487 INFO:tasks.rados.rados.0.smithi107.stdout:3138: writing smithi10729407-5 from 1949696 to 2146304 tid 5 2020-06-12T02:37:48.487 INFO:tasks.rados.rados.0.smithi107.stdout:3139: delete oid 18 current snap is 0 2020-06-12T02:37:48.487 INFO:tasks.rados.rados.0.smithi107.stdout:3136: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:48.487 INFO:tasks.rados.rados.0.smithi107.stdout:3136: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:48.487 INFO:tasks.rados.rados.0.smithi107.stdout:3136: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:48.488 INFO:tasks.rados.rados.0.smithi107.stdout:3136: finishing write tid 6 to smithi10729407-10 2020-06-12T02:37:48.488 INFO:tasks.rados.rados.0.smithi107.stdout:3136: finishing write tid 7 to smithi10729407-10 2020-06-12T02:37:48.488 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 317 (ObjNum 3135 snap 0 seq_num 3135) dirty exists 2020-06-12T02:37:48.488 INFO:tasks.rados.rados.0.smithi107.stdout:3136: left oid 10 (ObjNum 3135 snap 0 seq_num 3135) 2020-06-12T02:37:48.488 INFO:tasks.rados.rados.0.smithi107.stdout:3138: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:48.489 INFO:tasks.rados.rados.0.smithi107.stdout:3136: done (2 left) 2020-06-12T02:37:48.489 INFO:tasks.rados.rados.0.smithi107.stdout:3139: done (1 left) 2020-06-12T02:37:48.489 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:37:48.489 INFO:tasks.rados.rados.0.smithi107.stdout:3140: seq_num 3139 ranges {0=581632,581632=589824,1171456=491520,1662976=524288} 2020-06-12T02:37:48.492 INFO:tasks.rados.rados.0.smithi107.stdout:3140: writing smithi10729407-19 from 0 to 581632 tid 1 2020-06-12T02:37:48.495 INFO:tasks.rados.rados.0.smithi107.stdout:3140: writing smithi10729407-19 from 581632 to 1171456 tid 2 2020-06-12T02:37:48.498 INFO:tasks.rados.rados.0.smithi107.stdout:3140: writing smithi10729407-19 from 1171456 to 1662976 tid 3 2020-06-12T02:37:48.502 INFO:tasks.rados.rados.0.smithi107.stdout:3140: writing smithi10729407-19 from 1662976 to 2187264 tid 4 2020-06-12T02:37:48.502 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:37:48.502 INFO:tasks.rados.rados.0.smithi107.stdout:3141: seq_num 3140 ranges {1982464=598016,2580480=524288,3104768=696320,3801088=385024} 2020-06-12T02:37:48.516 INFO:tasks.rados.rados.0.smithi107.stdout:3141: writing smithi10729407-2 from 1982464 to 2580480 tid 1 2020-06-12T02:37:48.519 INFO:tasks.rados.rados.0.smithi107.stdout:3141: writing smithi10729407-2 from 2580480 to 3104768 tid 2 2020-06-12T02:37:48.523 INFO:tasks.rados.rados.0.smithi107.stdout:3141: writing smithi10729407-2 from 3104768 to 3801088 tid 3 2020-06-12T02:37:48.523 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:48.521+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:48.525 INFO:tasks.rados.rados.0.smithi107.stdout:3141: writing smithi10729407-2 from 3801088 to 4186112 tid 4 2020-06-12T02:37:48.525 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:37:48.526 INFO:tasks.rados.rados.0.smithi107.stdout:3142: seq_num 3141 ranges {0=614400,614400=409600,1024000=507904,1531904=516096,2048000=172032} 2020-06-12T02:37:48.529 INFO:tasks.rados.rados.0.smithi107.stdout:3142: writing smithi10729407-11 from 0 to 614400 tid 1 2020-06-12T02:37:48.532 INFO:tasks.rados.rados.0.smithi107.stdout:3142: writing smithi10729407-11 from 614400 to 1024000 tid 2 2020-06-12T02:37:48.535 INFO:tasks.rados.rados.0.smithi107.stdout:3142: writing smithi10729407-11 from 1024000 to 1531904 tid 3 2020-06-12T02:37:48.547 INFO:tasks.rados.rados.0.smithi107.stdout:3142: writing smithi10729407-11 from 1531904 to 2048000 tid 4 2020-06-12T02:37:48.547 INFO:tasks.rados.rados.0.smithi107.stdout:3142: writing smithi10729407-11 from 2048000 to 2220032 tid 5 2020-06-12T02:37:48.547 INFO:tasks.rados.rados.0.smithi107.stdout:3143: delete oid 14 current snap is 0 2020-06-12T02:37:48.547 INFO:tasks.rados.rados.0.smithi107.stdout:3138: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:48.547 INFO:tasks.rados.rados.0.smithi107.stdout:3138: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:48.548 INFO:tasks.rados.rados.0.smithi107.stdout:3138: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:48.548 INFO:tasks.rados.rados.0.smithi107.stdout:3138: finishing write tid 5 to smithi10729407-5 2020-06-12T02:37:48.548 INFO:tasks.rados.rados.0.smithi107.stdout:3138: finishing write tid 7 to smithi10729407-5 2020-06-12T02:37:48.548 INFO:tasks.rados.rados.0.smithi107.stdout:3138: finishing write tid 8 to smithi10729407-5 2020-06-12T02:37:48.548 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 272 (ObjNum 3137 snap 0 seq_num 3137) dirty exists 2020-06-12T02:37:48.549 INFO:tasks.rados.rados.0.smithi107.stdout:3138: left oid 5 (ObjNum 3137 snap 0 seq_num 3137) 2020-06-12T02:37:48.549 INFO:tasks.rados.rados.0.smithi107.stdout:3140: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:48.549 INFO:tasks.rados.rados.0.smithi107.stdout:3140: finishing write tid 2 to smithi10729407-19 2020-06-12T02:37:48.549 INFO:tasks.rados.rados.0.smithi107.stdout:3140: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:48.549 INFO:tasks.rados.rados.0.smithi107.stdout:3140: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:48.551 INFO:tasks.rados.rados.0.smithi107.stdout:3140: finishing write tid 6 to smithi10729407-19 2020-06-12T02:37:48.551 INFO:tasks.rados.rados.0.smithi107.stdout:3140: finishing write tid 7 to smithi10729407-19 2020-06-12T02:37:48.551 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 513 (ObjNum 3139 snap 0 seq_num 3139) dirty exists 2020-06-12T02:37:48.552 INFO:tasks.rados.rados.0.smithi107.stdout:3140: left oid 19 (ObjNum 3139 snap 0 seq_num 3139) 2020-06-12T02:37:48.552 INFO:tasks.rados.rados.0.smithi107.stdout:3141: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:48.552 INFO:tasks.rados.rados.0.smithi107.stdout:3141: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:48.552 INFO:tasks.rados.rados.0.smithi107.stdout:3141: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:48.553 INFO:tasks.rados.rados.0.smithi107.stdout:3141: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:48.553 INFO:tasks.rados.rados.0.smithi107.stdout:3141: finishing write tid 6 to smithi10729407-2 2020-06-12T02:37:48.553 INFO:tasks.rados.rados.0.smithi107.stdout:3141: finishing write tid 7 to smithi10729407-2 2020-06-12T02:37:48.553 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 328 (ObjNum 3140 snap 0 seq_num 3140) dirty exists 2020-06-12T02:37:48.553 INFO:tasks.rados.rados.0.smithi107.stdout:3141: left oid 2 (ObjNum 3140 snap 0 seq_num 3140) 2020-06-12T02:37:48.554 INFO:tasks.rados.rados.0.smithi107.stdout:3142: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:48.554 INFO:tasks.rados.rados.0.smithi107.stdout:3142: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:48.554 INFO:tasks.rados.rados.0.smithi107.stdout:3142: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:48.554 INFO:tasks.rados.rados.0.smithi107.stdout:3142: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:48.555 INFO:tasks.rados.rados.0.smithi107.stdout:3142: finishing write tid 5 to smithi10729407-11 2020-06-12T02:37:48.555 INFO:tasks.rados.rados.0.smithi107.stdout:3142: finishing write tid 7 to smithi10729407-11 2020-06-12T02:37:48.556 INFO:tasks.rados.rados.0.smithi107.stdout:3138: done (4 left) 2020-06-12T02:37:48.556 INFO:tasks.rados.rados.0.smithi107.stdout:3140: done (3 left) 2020-06-12T02:37:48.556 INFO:tasks.rados.rados.0.smithi107.stdout:3141: done (2 left) 2020-06-12T02:37:48.556 INFO:tasks.rados.rados.0.smithi107.stdout:3143: done (1 left) 2020-06-12T02:37:48.556 INFO:tasks.rados.rados.0.smithi107.stdout:3144: delete oid 4 current snap is 0 2020-06-12T02:37:48.556 INFO:tasks.rados.rados.0.smithi107.stdout:3142: finishing write tid 8 to smithi10729407-11 2020-06-12T02:37:48.557 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 525 (ObjNum 3141 snap 0 seq_num 3141) dirty exists 2020-06-12T02:37:48.557 INFO:tasks.rados.rados.0.smithi107.stdout:3142: left oid 11 (ObjNum 3141 snap 0 seq_num 3141) 2020-06-12T02:37:48.557 INFO:tasks.rados.rados.0.smithi107.stdout:3142: done (1 left) 2020-06-12T02:37:48.558 INFO:tasks.rados.rados.0.smithi107.stdout:3144: done (0 left) 2020-06-12T02:37:48.558 INFO:tasks.rados.rados.0.smithi107.stdout:3145: delete oid 1 current snap is 0 2020-06-12T02:37:48.558 INFO:tasks.rados.rados.0.smithi107.stdout:3145: done (0 left) 2020-06-12T02:37:48.559 INFO:tasks.rados.rados.0.smithi107.stdout:3146: delete oid 16 current snap is 0 2020-06-12T02:37:48.562 INFO:tasks.rados.rados.0.smithi107.stdout:3146: done (0 left) 2020-06-12T02:37:48.562 INFO:tasks.rados.rados.0.smithi107.stdout:3147: delete oid 9 current snap is 0 2020-06-12T02:37:48.565 INFO:tasks.rados.rados.0.smithi107.stdout:3147: done (0 left) 2020-06-12T02:37:48.565 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:37:48.566 INFO:tasks.rados.rados.0.smithi107.stdout:3148: seq_num 3147 ranges {0=712704,712704=516096,1228800=450560,1679360=638976} 2020-06-12T02:37:48.569 INFO:tasks.rados.rados.0.smithi107.stdout:3148: writing smithi10729407-20 from 0 to 712704 tid 1 2020-06-12T02:37:48.572 INFO:tasks.rados.rados.0.smithi107.stdout:3148: writing smithi10729407-20 from 712704 to 1228800 tid 2 2020-06-12T02:37:48.575 INFO:tasks.rados.rados.0.smithi107.stdout:3148: writing smithi10729407-20 from 1228800 to 1679360 tid 3 2020-06-12T02:37:48.579 INFO:tasks.rados.rados.0.smithi107.stdout:3148: writing smithi10729407-20 from 1679360 to 2318336 tid 4 2020-06-12T02:37:48.579 INFO:tasks.rados.rados.0.smithi107.stdout:3149: delete oid 16 current snap is 0 2020-06-12T02:37:48.581 INFO:tasks.rados.rados.0.smithi107.stdout:3149: done (1 left) 2020-06-12T02:37:48.581 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:37:48.582 INFO:tasks.rados.rados.0.smithi107.stdout:3150: seq_num 3149 ranges {1228800=745472,1974272=737280,2711552=466944,3178496=401408} 2020-06-12T02:37:48.593 INFO:tasks.rados.rados.0.smithi107.stdout:3150: writing smithi10729407-3 from 1228800 to 1974272 tid 1 2020-06-12T02:37:48.601 INFO:tasks.rados.rados.0.smithi107.stdout:3150: writing smithi10729407-3 from 1974272 to 2711552 tid 2 2020-06-12T02:37:48.601 INFO:tasks.rados.rados.0.smithi107.stdout:3150: writing smithi10729407-3 from 2711552 to 3178496 tid 3 2020-06-12T02:37:48.602 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:37:48.599+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:37:48.603 INFO:tasks.rados.rados.0.smithi107.stdout:3150: writing smithi10729407-3 from 3178496 to 3579904 tid 4 2020-06-12T02:37:48.604 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:37:48.604 INFO:tasks.rados.rados.0.smithi107.stdout:3151: seq_num 3150 ranges {0=761856,761856=622592,1384448=671744,2056192=311296} 2020-06-12T02:37:48.608 INFO:tasks.rados.rados.0.smithi107.stdout:3151: writing smithi10729407-9 from 0 to 761856 tid 1 2020-06-12T02:37:48.611 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:37:48.611 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:37:48.611 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2020-06-12T02:37:48.612 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:37:48.613 INFO:tasks.rados.rados.0.smithi107.stdout:3151: writing smithi10729407-9 from 761856 to 1384448 tid 2 2020-06-12T02:37:48.617 INFO:tasks.rados.rados.0.smithi107.stdout:3151: writing smithi10729407-9 from 1384448 to 2056192 tid 3 2020-06-12T02:37:48.619 INFO:tasks.rados.rados.0.smithi107.stdout:3151: writing smithi10729407-9 from 2056192 to 2367488 tid 4 2020-06-12T02:37:48.619 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:37:48.619 INFO:tasks.rados.rados.0.smithi107.stdout:3152: seq_num 3151 ranges {0=778240,778240=499712,1277952=434176,1712128=442368,2154496=229376} 2020-06-12T02:37:48.621 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-06-12T02:37:48.624 INFO:tasks.rados.rados.0.smithi107.stdout:3152: writing smithi10729407-8 from 0 to 778240 tid 1 2020-06-12T02:37:48.627 INFO:tasks.rados.rados.0.smithi107.stdout:3152: writing smithi10729407-8 from 778240 to 1277952 tid 2 2020-06-12T02:37:48.630 INFO:tasks.rados.rados.0.smithi107.stdout:3152: writing smithi10729407-8 from 1277952 to 1712128 tid 3 2020-06-12T02:37:48.633 INFO:tasks.rados.rados.0.smithi107.stdout:3152: writing smithi10729407-8 from 1712128 to 2154496 tid 4 2020-06-12T02:37:48.634 INFO:tasks.rados.rados.0.smithi107.stdout:3152: writing smithi10729407-8 from 2154496 to 2383872 tid 5 2020-06-12T02:37:48.634 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:37:48.634 INFO:tasks.rados.rados.0.smithi107.stdout:3153: seq_num 3152 ranges {2146304=794624,2940928=786432,3727360=638976,4366336=180224} 2020-06-12T02:37:48.649 INFO:tasks.rados.rados.0.smithi107.stdout:3153: writing smithi10729407-5 from 2146304 to 2940928 tid 1 2020-06-12T02:37:48.655 INFO:tasks.rados.rados.0.smithi107.stdout:3153: writing smithi10729407-5 from 2940928 to 3727360 tid 2 2020-06-12T02:37:48.658 INFO:tasks.rados.rados.0.smithi107.stdout:3153: writing smithi10729407-5 from 3727360 to 4366336 tid 3 2020-06-12T02:37:48.662 INFO:tasks.rados.rados.0.smithi107.stdout:3153: writing smithi10729407-5 from 4366336 to 4546560 tid 4 2020-06-12T02:37:48.662 INFO:tasks.rados.rados.0.smithi107.stdout:3154: delete oid 16 current snap is 0 2020-06-12T02:37:48.662 INFO:tasks.rados.rados.0.smithi107.stdout:3148: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:48.663 INFO:tasks.rados.rados.0.smithi107.stdout:3148: finishing write tid 2 to smithi10729407-20 2020-06-12T02:37:48.663 INFO:tasks.rados.rados.0.smithi107.stdout:3148: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:48.663 INFO:tasks.rados.rados.0.smithi107.stdout:3148: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:48.663 INFO:tasks.rados.rados.0.smithi107.stdout:3148: finishing write tid 6 to smithi10729407-20 2020-06-12T02:37:48.663 INFO:tasks.rados.rados.0.smithi107.stdout:3148: finishing write tid 7 to smithi10729407-20 2020-06-12T02:37:48.663 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 218 (ObjNum 3147 snap 0 seq_num 3147) dirty exists 2020-06-12T02:37:48.665 INFO:tasks.rados.rados.0.smithi107.stdout:3148: left oid 20 (ObjNum 3147 snap 0 seq_num 3147) 2020-06-12T02:37:48.665 INFO:tasks.rados.rados.0.smithi107.stdout:3150: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:48.665 INFO:tasks.rados.rados.0.smithi107.stdout:3150: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:48.665 INFO:tasks.rados.rados.0.smithi107.stdout:3150: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:48.666 INFO:tasks.rados.rados.0.smithi107.stdout:3150: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:48.666 INFO:tasks.rados.rados.0.smithi107.stdout:3150: finishing write tid 6 to smithi10729407-3 2020-06-12T02:37:48.666 INFO:tasks.rados.rados.0.smithi107.stdout:3151: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:48.666 INFO:tasks.rados.rados.0.smithi107.stdout:3150: finishing write tid 7 to smithi10729407-3 2020-06-12T02:37:48.666 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 555 (ObjNum 3149 snap 0 seq_num 3149) dirty exists 2020-06-12T02:37:48.667 INFO:tasks.rados.rados.0.smithi107.stdout:3150: left oid 3 (ObjNum 3149 snap 0 seq_num 3149) 2020-06-12T02:37:48.667 INFO:tasks.rados.rados.0.smithi107.stdout:3151: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:48.667 INFO:tasks.rados.rados.0.smithi107.stdout:3151: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:48.667 INFO:tasks.rados.rados.0.smithi107.stdout:3151: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:48.667 INFO:tasks.rados.rados.0.smithi107.stdout:3151: finishing write tid 6 to smithi10729407-9 2020-06-12T02:37:48.667 INFO:tasks.rados.rados.0.smithi107.stdout:3151: finishing write tid 7 to smithi10729407-9 2020-06-12T02:37:48.668 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 576 (ObjNum 3150 snap 0 seq_num 3150) dirty exists 2020-06-12T02:37:48.668 INFO:tasks.rados.rados.0.smithi107.stdout:3151: left oid 9 (ObjNum 3150 snap 0 seq_num 3150) 2020-06-12T02:37:48.668 INFO:tasks.rados.rados.0.smithi107.stdout:3152: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:48.668 INFO:tasks.rados.rados.0.smithi107.stdout:3152: finishing write tid 2 to smithi10729407-8 2020-06-12T02:37:48.668 INFO:tasks.rados.rados.0.smithi107.stdout:3152: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:48.669 INFO:tasks.rados.rados.0.smithi107.stdout:3152: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:48.669 INFO:tasks.rados.rados.0.smithi107.stdout:3152: finishing write tid 5 to smithi10729407-8 2020-06-12T02:37:48.669 INFO:tasks.rados.rados.0.smithi107.stdout:3152: finishing write tid 7 to smithi10729407-8 2020-06-12T02:37:48.669 INFO:tasks.rados.rados.0.smithi107.stdout:3152: finishing write tid 8 to smithi10729407-8 2020-06-12T02:37:48.669 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 534 (ObjNum 3151 snap 0 seq_num 3151) dirty exists 2020-06-12T02:37:48.669 INFO:tasks.rados.rados.0.smithi107.stdout:3152: left oid 8 (ObjNum 3151 snap 0 seq_num 3151) 2020-06-12T02:37:48.670 INFO:tasks.rados.rados.0.smithi107.stdout:3153: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:48.670 INFO:tasks.rados.rados.0.smithi107.stdout:3148: done (5 left) 2020-06-12T02:37:48.670 INFO:tasks.rados.rados.0.smithi107.stdout:3150: done (4 left) 2020-06-12T02:37:48.670 INFO:tasks.rados.rados.0.smithi107.stdout:3151: done (3 left) 2020-06-12T02:37:48.670 INFO:tasks.rados.rados.0.smithi107.stdout:3152: done (2 left) 2020-06-12T02:37:48.671 INFO:tasks.rados.rados.0.smithi107.stdout:3154: done (1 left) 2020-06-12T02:37:48.671 INFO:tasks.rados.rados.0.smithi107.stdout:3155: delete oid 9 current snap is 0 2020-06-12T02:37:48.671 INFO:tasks.rados.rados.0.smithi107.stdout:3153: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:48.672 INFO:tasks.rados.rados.0.smithi107.stdout:3155: done (1 left) 2020-06-12T02:37:48.673 INFO:tasks.rados.rados.0.smithi107.stdout:3156: delete oid 18 current snap is 0 2020-06-12T02:37:48.673 INFO:tasks.rados.rados.0.smithi107.stdout:3153: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:48.673 INFO:tasks.rados.rados.0.smithi107.stdout:3153: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:48.673 INFO:tasks.rados.rados.0.smithi107.stdout:3153: finishing write tid 6 to smithi10729407-5 2020-06-12T02:37:48.674 INFO:tasks.rados.rados.0.smithi107.stdout:3153: finishing write tid 7 to smithi10729407-5 2020-06-12T02:37:48.674 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 277 (ObjNum 3152 snap 0 seq_num 3152) dirty exists 2020-06-12T02:37:48.674 INFO:tasks.rados.rados.0.smithi107.stdout:3153: left oid 5 (ObjNum 3152 snap 0 seq_num 3152) 2020-06-12T02:37:48.674 INFO:tasks.rados.rados.0.smithi107.stdout:3153: done (1 left) 2020-06-12T02:37:48.675 INFO:tasks.rados.rados.0.smithi107.stdout:3156: done (0 left) 2020-06-12T02:37:48.675 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:37:48.675 INFO:tasks.rados.rados.0.smithi107.stdout:3157: seq_num 3156 ranges {2220032=57344} 2020-06-12T02:37:48.679 INFO:tasks.rados.rados.0.smithi107.stdout:3157: writing smithi10729407-11 from 2220032 to 2277376 tid 1 2020-06-12T02:37:48.679 INFO:tasks.rados.rados.0.smithi107.stdout:3158: delete oid 10 current snap is 0 2020-06-12T02:37:48.681 INFO:tasks.rados.rados.0.smithi107.stdout:3158: done (1 left) 2020-06-12T02:37:48.681 INFO:tasks.rados.rados.0.smithi107.stdout:3159: delete oid 3 current snap is 0 2020-06-12T02:37:48.682 INFO:tasks.rados.rados.0.smithi107.stdout:3157: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:48.683 INFO:tasks.rados.rados.0.smithi107.stdout:3157: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:48.685 INFO:tasks.rados.rados.0.smithi107.stdout:3157: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:48.685 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 528 (ObjNum 3156 snap 0 seq_num 3156) dirty exists 2020-06-12T02:37:48.685 INFO:tasks.rados.rados.0.smithi107.stdout:3157: left oid 11 (ObjNum 3156 snap 0 seq_num 3156) 2020-06-12T02:37:48.686 INFO:tasks.rados.rados.0.smithi107.stdout:3157: done (1 left) 2020-06-12T02:37:48.686 INFO:tasks.rados.rados.0.smithi107.stdout:3159: done (0 left) 2020-06-12T02:37:48.686 INFO:tasks.rados.rados.0.smithi107.stdout:3160: delete oid 18 current snap is 0 2020-06-12T02:37:48.688 INFO:tasks.rados.rados.0.smithi107.stdout:3160: done (0 left) 2020-06-12T02:37:48.688 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:37:48.688 INFO:tasks.rados.rados.0.smithi107.stdout:3161: seq_num 3160 ranges {0=131072} 2020-06-12T02:37:48.689 INFO:tasks.rados.rados.0.smithi107.stdout:3161: writing smithi10729407-7 from 0 to 131072 tid 1 2020-06-12T02:37:48.689 INFO:tasks.rados.rados.0.smithi107.stdout:3162: delete oid 8 current snap is 0 2020-06-12T02:37:48.692 INFO:tasks.rados.rados.0.smithi107.stdout:3162: done (1 left) 2020-06-12T02:37:48.693 INFO:tasks.rados.rados.0.smithi107.stdout:3163: delete oid 5 current snap is 0 2020-06-12T02:37:48.693 INFO:tasks.rados.rados.0.smithi107.stdout:3161: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:48.694 INFO:tasks.rados.rados.0.smithi107.stdout:3163: done (1 left) 2020-06-12T02:37:48.695 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:37:48.695 INFO:tasks.rados.rados.0.smithi107.stdout:3164: seq_num 3163 ranges {0=180224} 2020-06-12T02:37:48.695 INFO:tasks.rados.rados.0.smithi107.stdout:3164: writing smithi10729407-9 from 0 to 180224 tid 1 2020-06-12T02:37:48.696 INFO:tasks.rados.rados.0.smithi107.stdout:3165: delete oid 8 current snap is 0 2020-06-12T02:37:48.697 INFO:tasks.rados.rados.0.smithi107.stdout:3161: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:48.699 INFO:tasks.rados.rados.0.smithi107.stdout:3161: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:48.699 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 599 (ObjNum 3160 snap 0 seq_num 3160) dirty exists 2020-06-12T02:37:48.699 INFO:tasks.rados.rados.0.smithi107.stdout:3161: left oid 7 (ObjNum 3160 snap 0 seq_num 3160) 2020-06-12T02:37:48.700 INFO:tasks.rados.rados.0.smithi107.stdout:3161: done (2 left) 2020-06-12T02:37:48.700 INFO:tasks.rados.rados.0.smithi107.stdout:3165: done (1 left) 2020-06-12T02:37:48.700 INFO:tasks.rados.rados.0.smithi107.stdout:3166: delete oid 6 current snap is 0 2020-06-12T02:37:48.700 INFO:tasks.rados.rados.0.smithi107.stdout:3164: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:48.703 INFO:tasks.rados.rados.0.smithi107.stdout:3166: done (1 left) 2020-06-12T02:37:48.703 INFO:tasks.rados.rados.0.smithi107.stdout:3167: delete oid 12 current snap is 0 2020-06-12T02:37:48.703 INFO:tasks.rados.rados.0.smithi107.stdout:3164: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:48.705 INFO:tasks.rados.rados.0.smithi107.stdout:3164: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:48.706 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 581 (ObjNum 3163 snap 0 seq_num 3163) dirty exists 2020-06-12T02:37:48.706 INFO:tasks.rados.rados.0.smithi107.stdout:3164: left oid 9 (ObjNum 3163 snap 0 seq_num 3163) 2020-06-12T02:37:48.706 INFO:tasks.rados.rados.0.smithi107.stdout:3164: done (1 left) 2020-06-12T02:37:48.706 INFO:tasks.rados.rados.0.smithi107.stdout:3167: done (0 left) 2020-06-12T02:37:48.707 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:37:48.707 INFO:tasks.rados.rados.0.smithi107.stdout:3168: seq_num 3167 ranges {0=245760} 2020-06-12T02:37:48.707 INFO:tasks.rados.rados.0.smithi107.stdout:3168: writing smithi10729407-12 from 0 to 245760 tid 1 2020-06-12T02:37:48.708 INFO:tasks.rados.rados.0.smithi107.stdout:3169: delete oid 13 current snap is 0 2020-06-12T02:37:48.709 INFO:tasks.rados.rados.0.smithi107.stdout:3169: done (1 left) 2020-06-12T02:37:48.709 INFO:tasks.rados.rados.0.smithi107.stdout:3170: delete oid 17 current snap is 0 2020-06-12T02:37:48.710 INFO:tasks.rados.rados.0.smithi107.stdout:3168: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:48.711 INFO:tasks.rados.rados.0.smithi107.stdout:3168: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:48.711 INFO:tasks.rados.rados.0.smithi107.stdout:3170: done (1 left) 2020-06-12T02:37:48.711 INFO:tasks.rados.rados.0.smithi107.stdout:3171: delete oid 18 current snap is 0 2020-06-12T02:37:48.714 INFO:tasks.rados.rados.0.smithi107.stdout:3168: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:48.714 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 516 (ObjNum 3167 snap 0 seq_num 3167) dirty exists 2020-06-12T02:37:48.714 INFO:tasks.rados.rados.0.smithi107.stdout:3168: left oid 12 (ObjNum 3167 snap 0 seq_num 3167) 2020-06-12T02:37:48.716 INFO:tasks.rados.rados.0.smithi107.stdout:3168: done (1 left) 2020-06-12T02:37:48.716 INFO:tasks.rados.rados.0.smithi107.stdout:3171: done (0 left) 2020-06-12T02:37:48.716 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:37:48.717 INFO:tasks.rados.rados.0.smithi107.stdout:3172: seq_num 3171 ranges {0=311296} 2020-06-12T02:37:48.718 INFO:tasks.rados.rados.0.smithi107.stdout:3172: writing smithi10729407-4 from 0 to 311296 tid 1 2020-06-12T02:37:48.718 INFO:tasks.rados.rados.0.smithi107.stdout:3173: delete oid 3 current snap is 0 2020-06-12T02:37:48.719 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:37:48.715+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:37:48.719 INFO:tasks.rados.rados.0.smithi107.stdout:3173: done (1 left) 2020-06-12T02:37:48.720 INFO:tasks.rados.rados.0.smithi107.stdout:3174: delete oid 19 current snap is 0 2020-06-12T02:37:48.722 INFO:tasks.rados.rados.0.smithi107.stdout:3174: done (1 left) 2020-06-12T02:37:48.722 INFO:tasks.rados.rados.0.smithi107.stdout:3175: delete oid 7 current snap is 0 2020-06-12T02:37:48.723 INFO:tasks.rados.rados.0.smithi107.stdout:3172: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:48.723 INFO:tasks.rados.rados.0.smithi107.stdout:3172: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:48.724 INFO:tasks.rados.rados.0.smithi107.stdout:3175: done (1 left) 2020-06-12T02:37:48.724 INFO:tasks.rados.rados.0.smithi107.stdout:3176: delete oid 14 current snap is 0 2020-06-12T02:37:48.725 INFO:tasks.rados.rados.0.smithi107.stdout:3172: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:48.725 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 607 (ObjNum 3171 snap 0 seq_num 3171) dirty exists 2020-06-12T02:37:48.725 INFO:tasks.rados.rados.0.smithi107.stdout:3172: left oid 4 (ObjNum 3171 snap 0 seq_num 3171) 2020-06-12T02:37:48.726 INFO:tasks.rados.rados.0.smithi107.stdout:3172: done (1 left) 2020-06-12T02:37:48.726 INFO:tasks.rados.rados.0.smithi107.stdout:3176: done (0 left) 2020-06-12T02:37:48.726 INFO:tasks.rados.rados.0.smithi107.stdout:3177: delete oid 7 current snap is 0 2020-06-12T02:37:48.728 INFO:tasks.rados.rados.0.smithi107.stdout:3177: done (0 left) 2020-06-12T02:37:48.728 INFO:tasks.rados.rados.0.smithi107.stdout:3178: delete oid 15 current snap is 0 2020-06-12T02:37:48.730 INFO:tasks.rados.rados.0.smithi107.stdout:3178: done (0 left) 2020-06-12T02:37:48.730 INFO:tasks.rados.rados.0.smithi107.stdout:3179: delete oid 11 current snap is 0 2020-06-12T02:37:48.731 INFO:tasks.rados.rados.0.smithi107.stdout:3179: done (0 left) 2020-06-12T02:37:48.732 INFO:tasks.rados.rados.0.smithi107.stdout:3180: delete oid 19 current snap is 0 2020-06-12T02:37:48.733 INFO:tasks.rados.rados.0.smithi107.stdout:3180: done (0 left) 2020-06-12T02:37:48.734 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:37:48.734 INFO:tasks.rados.rados.0.smithi107.stdout:3181: seq_num 3180 ranges {2318336=466944} 2020-06-12T02:37:48.749 INFO:tasks.rados.rados.0.smithi107.stdout:3181: writing smithi10729407-20 from 2318336 to 2785280 tid 1 2020-06-12T02:37:48.749 INFO:tasks.rados.rados.0.smithi107.stdout:3182: delete oid 5 current snap is 0 2020-06-12T02:37:48.751 INFO:tasks.rados.rados.0.smithi107.stdout:3182: done (1 left) 2020-06-12T02:37:48.751 INFO:tasks.rados.rados.0.smithi107.stdout:3183: delete oid 1 current snap is 0 2020-06-12T02:37:48.754 INFO:tasks.rados.rados.0.smithi107.stdout:3183: done (1 left) 2020-06-12T02:37:48.754 INFO:tasks.rados.rados.0.smithi107.stdout:3184: delete oid 3 current snap is 0 2020-06-12T02:37:48.755 INFO:tasks.rados.rados.0.smithi107.stdout:3184: done (1 left) 2020-06-12T02:37:48.755 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:37:48.755 INFO:tasks.rados.rados.0.smithi107.stdout:3185: seq_num 3184 ranges {0=532480} 2020-06-12T02:37:48.758 INFO:tasks.rados.rados.0.smithi107.stdout:3185: writing smithi10729407-19 from 0 to 532480 tid 1 2020-06-12T02:37:48.758 INFO:tasks.rados.rados.0.smithi107.stdout:3186: delete oid 3 current snap is 0 2020-06-12T02:37:48.759 INFO:tasks.rados.rados.0.smithi107.stdout:3181: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:48.759 INFO:tasks.rados.rados.0.smithi107.stdout:3181: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:48.760 INFO:tasks.rados.rados.0.smithi107.stdout:3181: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:48.760 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 220 (ObjNum 3180 snap 0 seq_num 3180) dirty exists 2020-06-12T02:37:48.761 INFO:tasks.rados.rados.0.smithi107.stdout:3181: left oid 20 (ObjNum 3180 snap 0 seq_num 3180) 2020-06-12T02:37:48.763 INFO:tasks.rados.rados.0.smithi107.stdout:3181: done (2 left) 2020-06-12T02:37:48.763 INFO:tasks.rados.rados.0.smithi107.stdout:3186: done (1 left) 2020-06-12T02:37:48.763 INFO:tasks.rados.rados.0.smithi107.stdout:3187: delete oid 20 current snap is 0 2020-06-12T02:37:48.764 INFO:tasks.rados.rados.0.smithi107.stdout:3185: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:48.765 INFO:tasks.rados.rados.0.smithi107.stdout:3185: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:48.765 INFO:tasks.rados.rados.0.smithi107.stdout:3187: done (1 left) 2020-06-12T02:37:48.766 INFO:tasks.rados.rados.0.smithi107.stdout:3188: delete oid 3 current snap is 0 2020-06-12T02:37:48.767 INFO:tasks.rados.rados.0.smithi107.stdout:3188: done (1 left) 2020-06-12T02:37:48.767 INFO:tasks.rados.rados.0.smithi107.stdout:3189: delete oid 4 current snap is 0 2020-06-12T02:37:48.768 INFO:tasks.rados.rados.0.smithi107.stdout:3185: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:48.768 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 520 (ObjNum 3184 snap 0 seq_num 3184) dirty exists 2020-06-12T02:37:48.768 INFO:tasks.rados.rados.0.smithi107.stdout:3185: left oid 19 (ObjNum 3184 snap 0 seq_num 3184) 2020-06-12T02:37:48.769 INFO:tasks.rados.rados.0.smithi107.stdout:3185: done (1 left) 2020-06-12T02:37:48.769 INFO:tasks.rados.rados.0.smithi107.stdout:3189: done (0 left) 2020-06-12T02:37:48.770 INFO:tasks.rados.rados.0.smithi107.stdout:3190: delete oid 18 current snap is 0 2020-06-12T02:37:48.771 INFO:tasks.rados.rados.0.smithi107.stdout:3190: done (0 left) 2020-06-12T02:37:48.771 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:37:48.771 INFO:tasks.rados.rados.0.smithi107.stdout:3191: seq_num 3190 ranges {0=630784} 2020-06-12T02:37:48.775 INFO:tasks.rados.rados.0.smithi107.stdout:3191: writing smithi10729407-8 from 0 to 630784 tid 1 2020-06-12T02:37:48.775 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:37:48.775 INFO:tasks.rados.rados.0.smithi107.stdout:3192: seq_num 3191 ranges {0=647168} 2020-06-12T02:37:48.778 INFO:tasks.rados.rados.0.smithi107.stdout:3192: writing smithi10729407-13 from 0 to 647168 tid 1 2020-06-12T02:37:48.778 INFO:tasks.rados.rados.0.smithi107.stdout:3193: delete oid 12 current snap is 0 2020-06-12T02:37:48.781 INFO:tasks.rados.rados.0.smithi107.stdout:3193: done (2 left) 2020-06-12T02:37:48.782 INFO:tasks.rados.rados.0.smithi107.stdout:3194: delete oid 15 current snap is 0 2020-06-12T02:37:48.782 INFO:tasks.rados.rados.0.smithi107.stdout:3191: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:48.783 INFO:tasks.rados.rados.0.smithi107.stdout:3191: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:48.785 INFO:tasks.rados.rados.0.smithi107.stdout:3191: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:48.785 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 539 (ObjNum 3190 snap 0 seq_num 3190) dirty exists 2020-06-12T02:37:48.785 INFO:tasks.rados.rados.0.smithi107.stdout:3191: left oid 8 (ObjNum 3190 snap 0 seq_num 3190) 2020-06-12T02:37:48.786 INFO:tasks.rados.rados.0.smithi107.stdout:3191: done (2 left) 2020-06-12T02:37:48.786 INFO:tasks.rados.rados.0.smithi107.stdout:3194: done (1 left) 2020-06-12T02:37:48.786 INFO:tasks.rados.rados.0.smithi107.stdout:3195: delete oid 14 current snap is 0 2020-06-12T02:37:48.787 INFO:tasks.rados.rados.0.smithi107.stdout:3195: done (1 left) 2020-06-12T02:37:48.787 INFO:tasks.rados.rados.0.smithi107.stdout:3196: delete oid 5 current snap is 0 2020-06-12T02:37:48.787 INFO:tasks.rados.rados.0.smithi107.stdout:3192: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:48.788 INFO:tasks.rados.rados.0.smithi107.stdout:3196: done (1 left) 2020-06-12T02:37:48.788 INFO:tasks.rados.rados.0.smithi107.stdout:3197: delete oid 14 current snap is 0 2020-06-12T02:37:48.791 INFO:tasks.rados.rados.0.smithi107.stdout:3197: done (1 left) 2020-06-12T02:37:48.791 INFO:tasks.rados.rados.0.smithi107.stdout:3198: delete oid 11 current snap is 0 2020-06-12T02:37:48.792 INFO:tasks.rados.rados.0.smithi107.stdout:3192: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:48.793 INFO:tasks.rados.rados.0.smithi107.stdout:3192: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:48.793 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 604 (ObjNum 3191 snap 0 seq_num 3191) dirty exists 2020-06-12T02:37:48.793 INFO:tasks.rados.rados.0.smithi107.stdout:3192: left oid 13 (ObjNum 3191 snap 0 seq_num 3191) 2020-06-12T02:37:48.794 INFO:tasks.rados.rados.0.smithi107.stdout:3192: done (1 left) 2020-06-12T02:37:48.794 INFO:tasks.rados.rados.0.smithi107.stdout:3198: done (0 left) 2020-06-12T02:37:48.794 INFO:tasks.rados.rados.0.smithi107.stdout:3199: delete oid 12 current snap is 0 2020-06-12T02:37:48.795 INFO:tasks.rados.rados.0.smithi107.stdout:3199: done (0 left) 2020-06-12T02:37:48.796 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:37:48.796 INFO:tasks.rados.rados.0.smithi107.stdout:3200: seq_num 3199 ranges {0=786432} 2020-06-12T02:37:48.800 INFO:tasks.rados.rados.0.smithi107.stdout:3200: writing smithi10729407-12 from 0 to 786432 tid 1 2020-06-12T02:37:48.800 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:37:48.800 INFO:tasks.rados.rados.0.smithi107.stdout:3201: seq_num 3200 ranges {0=802816} 2020-06-12T02:37:48.804 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:37:48.802+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:37:48.805 INFO:tasks.rados.rados.0.smithi107.stdout:3201: writing smithi10729407-3 from 0 to 802816 tid 1 2020-06-12T02:37:48.806 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:37:48.806 INFO:tasks.rados.rados.0.smithi107.stdout:3202: seq_num 3201 ranges {0=417792,417792=401408} 2020-06-12T02:37:48.810 INFO:tasks.rados.rados.0.smithi107.stdout:3202: writing smithi10729407-17 from 0 to 417792 tid 1 2020-06-12T02:37:48.811 INFO:tasks.rados.rados.0.smithi107.stdout:3202: writing smithi10729407-17 from 417792 to 819200 tid 2 2020-06-12T02:37:48.811 INFO:tasks.rados.rados.0.smithi107.stdout:3203: delete oid 8 current snap is 0 2020-06-12T02:37:48.811 INFO:tasks.rados.rados.0.smithi107.stdout:3200: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:48.811 INFO:tasks.rados.rados.0.smithi107.stdout:3200: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:48.814 INFO:tasks.rados.rados.0.smithi107.stdout:3203: done (3 left) 2020-06-12T02:37:48.814 INFO:tasks.rados.rados.0.smithi107.stdout:3204: delete oid 4 current snap is 0 2020-06-12T02:37:48.815 INFO:tasks.rados.rados.0.smithi107.stdout:3200: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:48.815 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 524 (ObjNum 3199 snap 0 seq_num 3199) dirty exists 2020-06-12T02:37:48.815 INFO:tasks.rados.rados.0.smithi107.stdout:3200: left oid 12 (ObjNum 3199 snap 0 seq_num 3199) 2020-06-12T02:37:48.815 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:37:48.816 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:37:48.816 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:37:48.816 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:37:48.816 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:37:48.817 INFO:tasks.rados.rados.0.smithi107.stdout:3201: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:48.817 INFO:tasks.rados.rados.0.smithi107.stdout:3202: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:48.820 INFO:tasks.rados.rados.0.smithi107.stdout:3202: finishing write tid 2 to smithi10729407-17 2020-06-12T02:37:48.821 INFO:tasks.rados.rados.0.smithi107.stdout:3202: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:48.822 INFO:tasks.rados.rados.0.smithi107.stdout:3202: finishing write tid 5 to smithi10729407-17 2020-06-12T02:37:48.822 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 611 (ObjNum 3201 snap 0 seq_num 3201) dirty exists 2020-06-12T02:37:48.822 INFO:tasks.rados.rados.0.smithi107.stdout:3202: left oid 17 (ObjNum 3201 snap 0 seq_num 3201) 2020-06-12T02:37:48.823 INFO:tasks.rados.rados.0.smithi107.stdout:3201: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:48.823 INFO:tasks.rados.rados.0.smithi107.stdout:3200: done (3 left) 2020-06-12T02:37:48.823 INFO:tasks.rados.rados.0.smithi107.stdout:3202: done (2 left) 2020-06-12T02:37:48.823 INFO:tasks.rados.rados.0.smithi107.stdout:3204: done (1 left) 2020-06-12T02:37:48.823 INFO:tasks.rados.rados.0.smithi107.stdout:3205: delete oid 12 current snap is 0 2020-06-12T02:37:48.825 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-06-12T02:37:48.826 INFO:tasks.rados.rados.0.smithi107.stdout:3201: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:48.826 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 563 (ObjNum 3200 snap 0 seq_num 3200) dirty exists 2020-06-12T02:37:48.826 INFO:tasks.rados.rados.0.smithi107.stdout:3201: left oid 3 (ObjNum 3200 snap 0 seq_num 3200) 2020-06-12T02:37:48.826 INFO:tasks.rados.rados.0.smithi107.stdout:3201: done (1 left) 2020-06-12T02:37:48.826 INFO:tasks.rados.rados.0.smithi107.stdout:3205: done (0 left) 2020-06-12T02:37:48.827 INFO:tasks.rados.rados.0.smithi107.stdout:3206: delete oid 14 current snap is 0 2020-06-12T02:37:48.831 INFO:tasks.rados.rados.0.smithi107.stdout:3206: done (0 left) 2020-06-12T02:37:48.831 INFO:tasks.rados.rados.0.smithi107.stdout:3207: delete oid 3 current snap is 0 2020-06-12T02:37:48.833 INFO:tasks.rados.rados.0.smithi107.stdout:3207: done (0 left) 2020-06-12T02:37:48.833 INFO:tasks.rados.rados.0.smithi107.stdout:3208: delete oid 3 current snap is 0 2020-06-12T02:37:48.837 INFO:tasks.rados.rados.0.smithi107.stdout:3208: done (0 left) 2020-06-12T02:37:48.837 INFO:tasks.rados.rados.0.smithi107.stdout:3209: delete oid 4 current snap is 0 2020-06-12T02:37:48.840 INFO:tasks.rados.rados.0.smithi107.stdout:3209: done (0 left) 2020-06-12T02:37:48.840 INFO:tasks.rados.rados.0.smithi107.stdout:3210: delete oid 16 current snap is 0 2020-06-12T02:37:48.842 INFO:tasks.rados.rados.0.smithi107.stdout:3210: done (0 left) 2020-06-12T02:37:48.842 INFO:tasks.rados.rados.0.smithi107.stdout:3211: delete oid 17 current snap is 0 2020-06-12T02:37:48.845 INFO:tasks.rados.rados.0.smithi107.stdout:3211: done (0 left) 2020-06-12T02:37:48.845 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:37:48.845 INFO:tasks.rados.rados.0.smithi107.stdout:3212: seq_num 3211 ranges {0=581632,581632=401408} 2020-06-12T02:37:48.848 INFO:tasks.rados.rados.0.smithi107.stdout:3212: writing smithi10729407-3 from 0 to 581632 tid 1 2020-06-12T02:37:48.850 INFO:tasks.rados.rados.0.smithi107.stdout:3212: writing smithi10729407-3 from 581632 to 983040 tid 2 2020-06-12T02:37:48.851 INFO:tasks.rados.rados.0.smithi107.stdout:3213: delete oid 4 current snap is 0 2020-06-12T02:37:48.852 INFO:tasks.rados.rados.0.smithi107.stdout:3213: done (1 left) 2020-06-12T02:37:48.853 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:37:48.853 INFO:tasks.rados.rados.0.smithi107.stdout:3214: seq_num 3213 ranges {0=614400,614400=401408} 2020-06-12T02:37:48.856 INFO:tasks.rados.rados.0.smithi107.stdout:3214: writing smithi10729407-16 from 0 to 614400 tid 1 2020-06-12T02:37:48.859 INFO:tasks.rados.rados.0.smithi107.stdout:3214: writing smithi10729407-16 from 614400 to 1015808 tid 2 2020-06-12T02:37:48.860 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:37:48.860 INFO:tasks.rados.rados.0.smithi107.stdout:3215: seq_num 3214 ranges {0=630784,630784=401408} 2020-06-12T02:37:48.863 INFO:tasks.rados.rados.0.smithi107.stdout:3215: writing smithi10729407-14 from 0 to 630784 tid 1 2020-06-12T02:37:48.866 INFO:tasks.rados.rados.0.smithi107.stdout:3215: writing smithi10729407-14 from 630784 to 1032192 tid 2 2020-06-12T02:37:48.867 INFO:tasks.rados.rados.0.smithi107.stdout:3216: delete oid 2 current snap is 0 2020-06-12T02:37:48.867 INFO:tasks.rados.rados.0.smithi107.stdout:3212: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:48.867 INFO:tasks.rados.rados.0.smithi107.stdout:3212: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:48.867 INFO:tasks.rados.rados.0.smithi107.stdout:3212: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:48.867 INFO:tasks.rados.rados.0.smithi107.stdout:3212: finishing write tid 5 to smithi10729407-3 2020-06-12T02:37:48.868 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 569 (ObjNum 3211 snap 0 seq_num 3211) dirty exists 2020-06-12T02:37:48.868 INFO:tasks.rados.rados.0.smithi107.stdout:3212: left oid 3 (ObjNum 3211 snap 0 seq_num 3211) 2020-06-12T02:37:48.869 INFO:tasks.rados.rados.0.smithi107.stdout:3214: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:48.869 INFO:tasks.rados.rados.0.smithi107.stdout:3212: done (3 left) 2020-06-12T02:37:48.870 INFO:tasks.rados.rados.0.smithi107.stdout:3216: done (2 left) 2020-06-12T02:37:48.870 INFO:tasks.rados.rados.0.smithi107.stdout:3217: delete oid 18 current snap is 0 2020-06-12T02:37:48.871 INFO:tasks.rados.rados.0.smithi107.stdout:3215: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:48.874 INFO:tasks.rados.rados.0.smithi107.stdout:3217: done (2 left) 2020-06-12T02:37:48.874 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:37:48.874 INFO:tasks.rados.rados.0.smithi107.stdout:3218: seq_num 3217 ranges {983040=688128,1671168=401408} 2020-06-12T02:37:48.883 INFO:tasks.rados.rados.0.smithi107.stdout:3218: writing smithi10729407-3 from 983040 to 1671168 tid 1 2020-06-12T02:37:48.886 INFO:tasks.rados.rados.0.smithi107.stdout:3218: writing smithi10729407-3 from 1671168 to 2072576 tid 2 2020-06-12T02:37:48.886 INFO:tasks.rados.rados.0.smithi107.stdout:3219: delete oid 19 current snap is 0 2020-06-12T02:37:48.886 INFO:tasks.rados.rados.0.smithi107.stdout:3214: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:48.887 INFO:tasks.rados.rados.0.smithi107.stdout:3214: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:48.889 INFO:tasks.rados.rados.0.smithi107.stdout:3214: finishing write tid 5 to smithi10729407-16 2020-06-12T02:37:48.889 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 572 (ObjNum 3213 snap 0 seq_num 3213) dirty exists 2020-06-12T02:37:48.889 INFO:tasks.rados.rados.0.smithi107.stdout:3214: left oid 16 (ObjNum 3213 snap 0 seq_num 3213) 2020-06-12T02:37:48.890 INFO:tasks.rados.rados.0.smithi107.stdout:3215: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:48.890 INFO:tasks.rados.rados.0.smithi107.stdout:3215: finishing write tid 4 to smithi10729407-14 2020-06-12T02:37:48.890 INFO:tasks.rados.rados.0.smithi107.stdout:3215: finishing write tid 5 to smithi10729407-14 2020-06-12T02:37:48.890 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 537 (ObjNum 3214 snap 0 seq_num 3214) dirty exists 2020-06-12T02:37:48.891 INFO:tasks.rados.rados.0.smithi107.stdout:3215: left oid 14 (ObjNum 3214 snap 0 seq_num 3214) 2020-06-12T02:37:48.891 INFO:tasks.rados.rados.0.smithi107.stdout:3214: done (3 left) 2020-06-12T02:37:48.891 INFO:tasks.rados.rados.0.smithi107.stdout:3215: done (2 left) 2020-06-12T02:37:48.892 INFO:tasks.rados.rados.0.smithi107.stdout:3219: done (1 left) 2020-06-12T02:37:48.892 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:37:48.892 INFO:tasks.rados.rados.0.smithi107.stdout:3220: seq_num 3219 ranges {0=720896,720896=401408} 2020-06-12T02:37:48.894 INFO:tasks.rados.rados.0.smithi107.stdout:3220: writing smithi10729407-11 from 0 to 720896 tid 1 2020-06-12T02:37:48.897 INFO:tasks.rados.rados.0.smithi107.stdout:3220: writing smithi10729407-11 from 720896 to 1122304 tid 2 2020-06-12T02:37:48.898 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:37:48.898 INFO:tasks.rados.rados.0.smithi107.stdout:3221: seq_num 3220 ranges {0=737280,737280=401408} 2020-06-12T02:37:48.902 INFO:tasks.rados.rados.0.smithi107.stdout:3221: writing smithi10729407-15 from 0 to 737280 tid 1 2020-06-12T02:37:48.905 INFO:tasks.rados.rados.0.smithi107.stdout:3221: writing smithi10729407-15 from 737280 to 1138688 tid 2 2020-06-12T02:37:48.905 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:37:48.905 INFO:tasks.rados.rados.0.smithi107.stdout:3222: seq_num 3221 ranges {1032192=753664,1785856=401408} 2020-06-12T02:37:48.906 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:48.902+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:48.915 INFO:tasks.rados.rados.0.smithi107.stdout:3222: writing smithi10729407-14 from 1032192 to 1785856 tid 1 2020-06-12T02:37:48.918 INFO:tasks.rados.rados.0.smithi107.stdout:3222: writing smithi10729407-14 from 1785856 to 2187264 tid 2 2020-06-12T02:37:48.919 INFO:tasks.rados.rados.0.smithi107.stdout:3223: delete oid 13 current snap is 0 2020-06-12T02:37:48.919 INFO:tasks.rados.rados.0.smithi107.stdout:3218: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:48.919 INFO:tasks.rados.rados.0.smithi107.stdout:3218: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:48.919 INFO:tasks.rados.rados.0.smithi107.stdout:3218: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:48.920 INFO:tasks.rados.rados.0.smithi107.stdout:3218: finishing write tid 5 to smithi10729407-3 2020-06-12T02:37:48.920 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 575 (ObjNum 3217 snap 0 seq_num 3217) dirty exists 2020-06-12T02:37:48.920 INFO:tasks.rados.rados.0.smithi107.stdout:3218: left oid 3 (ObjNum 3217 snap 0 seq_num 3217) 2020-06-12T02:37:48.920 INFO:tasks.rados.rados.0.smithi107.stdout:3220: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:48.920 INFO:tasks.rados.rados.0.smithi107.stdout:3220: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:48.921 INFO:tasks.rados.rados.0.smithi107.stdout:3221: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:48.921 INFO:tasks.rados.rados.0.smithi107.stdout:3220: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:48.922 INFO:tasks.rados.rados.0.smithi107.stdout:3220: finishing write tid 5 to smithi10729407-11 2020-06-12T02:37:48.922 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 540 (ObjNum 3219 snap 0 seq_num 3219) dirty exists 2020-06-12T02:37:48.924 INFO:tasks.rados.rados.0.smithi107.stdout:3220: left oid 11 (ObjNum 3219 snap 0 seq_num 3219) 2020-06-12T02:37:48.924 INFO:tasks.rados.rados.0.smithi107.stdout:3221: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:48.925 INFO:tasks.rados.rados.0.smithi107.stdout:3221: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:48.925 INFO:tasks.rados.rados.0.smithi107.stdout:3221: finishing write tid 5 to smithi10729407-15 2020-06-12T02:37:48.925 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 544 (ObjNum 3220 snap 0 seq_num 3220) dirty exists 2020-06-12T02:37:48.925 INFO:tasks.rados.rados.0.smithi107.stdout:3221: left oid 15 (ObjNum 3220 snap 0 seq_num 3220) 2020-06-12T02:37:48.926 INFO:tasks.rados.rados.0.smithi107.stdout:3218: done (4 left) 2020-06-12T02:37:48.926 INFO:tasks.rados.rados.0.smithi107.stdout:3220: done (3 left) 2020-06-12T02:37:48.926 INFO:tasks.rados.rados.0.smithi107.stdout:3221: done (2 left) 2020-06-12T02:37:48.926 INFO:tasks.rados.rados.0.smithi107.stdout:3223: done (1 left) 2020-06-12T02:37:48.927 INFO:tasks.rados.rados.0.smithi107.stdout:3224: delete oid 17 current snap is 0 2020-06-12T02:37:48.927 INFO:tasks.rados.rados.0.smithi107.stdout:3222: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:48.927 INFO:tasks.rados.rados.0.smithi107.stdout:3224: done (1 left) 2020-06-12T02:37:48.927 INFO:tasks.rados.rados.0.smithi107.stdout:3225: delete oid 10 current snap is 0 2020-06-12T02:37:48.928 INFO:tasks.rados.rados.0.smithi107.stdout:3225: done (1 left) 2020-06-12T02:37:48.928 INFO:tasks.rados.rados.0.smithi107.stdout:3226: delete oid 3 current snap is 0 2020-06-12T02:37:48.928 INFO:tasks.rados.rados.0.smithi107.stdout:3222: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:48.930 INFO:tasks.rados.rados.0.smithi107.stdout:3222: finishing write tid 4 to smithi10729407-14 2020-06-12T02:37:48.931 INFO:tasks.rados.rados.0.smithi107.stdout:3222: finishing write tid 5 to smithi10729407-14 2020-06-12T02:37:48.931 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 543 (ObjNum 3221 snap 0 seq_num 3221) dirty exists 2020-06-12T02:37:48.931 INFO:tasks.rados.rados.0.smithi107.stdout:3222: left oid 14 (ObjNum 3221 snap 0 seq_num 3221) 2020-06-12T02:37:48.932 INFO:tasks.rados.rados.0.smithi107.stdout:3222: done (1 left) 2020-06-12T02:37:48.932 INFO:tasks.rados.rados.0.smithi107.stdout:3226: done (0 left) 2020-06-12T02:37:48.932 INFO:tasks.rados.rados.0.smithi107.stdout:3227: delete oid 2 current snap is 0 2020-06-12T02:37:48.937 INFO:tasks.rados.rados.0.smithi107.stdout:3227: done (0 left) 2020-06-12T02:37:48.937 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:37:48.938 INFO:tasks.rados.rados.0.smithi107.stdout:3228: seq_num 3227 ranges {2187264=450560,2637824=606208,3244032=196608} 2020-06-12T02:37:48.954 INFO:tasks.rados.rados.0.smithi107.stdout:3228: writing smithi10729407-14 from 2187264 to 2637824 tid 1 2020-06-12T02:37:48.957 INFO:tasks.rados.rados.0.smithi107.stdout:3228: writing smithi10729407-14 from 2637824 to 3244032 tid 2 2020-06-12T02:37:48.958 INFO:tasks.rados.rados.0.smithi107.stdout:3228: writing smithi10729407-14 from 3244032 to 3440640 tid 3 2020-06-12T02:37:48.958 INFO:tasks.rados.rados.0.smithi107.stdout:3229: delete oid 13 current snap is 0 2020-06-12T02:37:48.958 INFO:tasks.rados.rados.0.smithi107.stdout:3229: done (1 left) 2020-06-12T02:37:48.958 INFO:tasks.rados.rados.0.smithi107.stdout:3230: delete oid 2 current snap is 0 2020-06-12T02:37:48.962 INFO:tasks.rados.rados.0.smithi107.stdout:3230: done (1 left) 2020-06-12T02:37:48.963 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:48.963 INFO:tasks.rados.rados.0.smithi107.stdout:3231: seq_num 3230 ranges {0=499712,499712=663552,1163264=139264} 2020-06-12T02:37:48.964 INFO:tasks.rados.rados.0.smithi107.stdout:3231: writing smithi10729407-2 from 0 to 499712 tid 1 2020-06-12T02:37:48.970 INFO:tasks.rados.rados.0.smithi107.stdout:3231: writing smithi10729407-2 from 499712 to 1163264 tid 2 2020-06-12T02:37:48.972 INFO:tasks.rados.rados.0.smithi107.stdout:3231: writing smithi10729407-2 from 1163264 to 1302528 tid 3 2020-06-12T02:37:48.972 INFO:tasks.rados.rados.0.smithi107.stdout:3232: delete oid 1 current snap is 0 2020-06-12T02:37:48.972 INFO:tasks.rados.rados.0.smithi107.stdout:3228: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:48.972 INFO:tasks.rados.rados.0.smithi107.stdout:3228: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:48.973 INFO:tasks.rados.rados.0.smithi107.stdout:3228: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:48.973 INFO:tasks.rados.rados.0.smithi107.stdout:3228: finishing write tid 5 to smithi10729407-14 2020-06-12T02:37:48.973 INFO:tasks.rados.rados.0.smithi107.stdout:3228: finishing write tid 6 to smithi10729407-14 2020-06-12T02:37:48.973 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 547 (ObjNum 3227 snap 0 seq_num 3227) dirty exists 2020-06-12T02:37:48.973 INFO:tasks.rados.rados.0.smithi107.stdout:3228: left oid 14 (ObjNum 3227 snap 0 seq_num 3227) 2020-06-12T02:37:48.974 INFO:tasks.rados.rados.0.smithi107.stdout:3231: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:48.974 INFO:tasks.rados.rados.0.smithi107.stdout:3228: done (2 left) 2020-06-12T02:37:48.974 INFO:tasks.rados.rados.0.smithi107.stdout:3232: done (1 left) 2020-06-12T02:37:48.974 INFO:tasks.rados.rados.0.smithi107.stdout:3233: delete oid 7 current snap is 0 2020-06-12T02:37:48.975 INFO:tasks.rados.rados.0.smithi107.stdout:3233: done (1 left) 2020-06-12T02:37:48.975 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:37:48.975 INFO:tasks.rados.rados.0.smithi107.stdout:3234: seq_num 3233 ranges {0=548864,548864=761856,1310720=40960} 2020-06-12T02:37:48.977 INFO:tasks.rados.rados.0.smithi107.stdout:3234: writing smithi10729407-10 from 0 to 548864 tid 1 2020-06-12T02:37:48.981 INFO:tasks.rados.rados.0.smithi107.stdout:3234: writing smithi10729407-10 from 548864 to 1310720 tid 2 2020-06-12T02:37:48.983 INFO:tasks.rados.rados.0.smithi107.stdout:3234: writing smithi10729407-10 from 1310720 to 1351680 tid 3 2020-06-12T02:37:48.983 INFO:tasks.rados.rados.0.smithi107.stdout:3235: delete oid 9 current snap is 0 2020-06-12T02:37:48.983 INFO:tasks.rados.rados.0.smithi107.stdout:3231: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:48.983 INFO:tasks.rados.rados.0.smithi107.stdout:3231: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:48.984 INFO:tasks.rados.rados.0.smithi107.stdout:3231: finishing write tid 5 to smithi10729407-2 2020-06-12T02:37:48.984 INFO:tasks.rados.rados.0.smithi107.stdout:3231: finishing write tid 6 to smithi10729407-2 2020-06-12T02:37:48.984 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 335 (ObjNum 3230 snap 0 seq_num 3230) dirty exists 2020-06-12T02:37:48.984 INFO:tasks.rados.rados.0.smithi107.stdout:3231: left oid 2 (ObjNum 3230 snap 0 seq_num 3230) 2020-06-12T02:37:48.984 INFO:tasks.rados.rados.0.smithi107.stdout:3234: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:48.985 INFO:tasks.rados.rados.0.smithi107.stdout:3231: done (2 left) 2020-06-12T02:37:48.985 INFO:tasks.rados.rados.0.smithi107.stdout:3235: done (1 left) 2020-06-12T02:37:48.985 INFO:tasks.rados.rados.0.smithi107.stdout:3236: delete oid 15 current snap is 0 2020-06-12T02:37:48.987 INFO:tasks.rados.rados.0.smithi107.stdout:3236: done (1 left) 2020-06-12T02:37:48.987 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:37:48.987 INFO:tasks.rados.rados.0.smithi107.stdout:3237: seq_num 3236 ranges {0=606208,606208=417792,1024000=385024} 2020-06-12T02:37:48.994 INFO:tasks.rados.rados.0.smithi107.stdout:3237: writing smithi10729407-1 from 0 to 606208 tid 1 2020-06-12T02:37:48.995 INFO:tasks.rados.rados.0.smithi107.stdout:3237: writing smithi10729407-1 from 606208 to 1024000 tid 2 2020-06-12T02:37:48.996 INFO:tasks.rados.rados.0.smithi107.stdout:3237: writing smithi10729407-1 from 1024000 to 1409024 tid 3 2020-06-12T02:37:48.996 INFO:tasks.rados.rados.0.smithi107.stdout:3238: delete oid 20 current snap is 0 2020-06-12T02:37:48.996 INFO:tasks.rados.rados.0.smithi107.stdout:3234: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:48.997 INFO:tasks.rados.rados.0.smithi107.stdout:3234: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:48.997 INFO:tasks.rados.rados.0.smithi107.stdout:3234: finishing write tid 5 to smithi10729407-10 2020-06-12T02:37:48.997 INFO:tasks.rados.rados.0.smithi107.stdout:3234: finishing write tid 6 to smithi10729407-10 2020-06-12T02:37:48.997 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 323 (ObjNum 3233 snap 0 seq_num 3233) dirty exists 2020-06-12T02:37:48.997 INFO:tasks.rados.rados.0.smithi107.stdout:3234: left oid 10 (ObjNum 3233 snap 0 seq_num 3233) 2020-06-12T02:37:48.998 INFO:tasks.rados.rados.0.smithi107.stdout:3237: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:49.002 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:37:49.002 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:37:49.002 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:37:49.002 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:37:49.003 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.003 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1206 3.5s0 3:a9510bda:::smithi10729407-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:49.003 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.856688+0000", 2020-06-12T02:37:49.003 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 16.137516512000001, 2020-06-12T02:37:49.004 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019116942000000001, 2020-06-12T02:37:49.004 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:49.006 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:49.006 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:49.007 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:49.007 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:49.007 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1206 2020-06-12T02:37:49.007 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.008 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:49.008 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.008 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:49.009 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856688+0000", 2020-06-12T02:37:49.009 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:49.009 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.009 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.010 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:49.010 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856688+0000", 2020-06-12T02:37:49.010 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4355e-05 2020-06-12T02:37:49.010 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.011 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.011 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:49.011 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856713+0000", 2020-06-12T02:37:49.012 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0850000000000001e-06 2020-06-12T02:37:49.012 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.012 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.012 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:49.013 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856716+0000", 2020-06-12T02:37:49.013 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4509999999999997e-06 2020-06-12T02:37:49.013 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.014 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.014 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:49.014 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856722+0000", 2020-06-12T02:37:49.014 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2879999999999999e-06 2020-06-12T02:37:49.015 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.015 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.015 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:49.016 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856724+0000", 2020-06-12T02:37:49.016 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0057507449999999998 2020-06-12T02:37:49.016 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.017 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.017 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:49.017 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.862475+0000", 2020-06-12T02:37:49.017 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5759000000000003e-05 2020-06-12T02:37:49.018 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.018 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.018 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:49.019 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.862571+0000", 2020-06-12T02:37:49.019 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012199018000000001 2020-06-12T02:37:49.019 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.019 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.020 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:49.020 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.874770+0000", 2020-06-12T02:37:49.020 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7260999999999999e-05 2020-06-12T02:37:49.020 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.021 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.021 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:49.021 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.874827+0000", 2020-06-12T02:37:49.021 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00097798000000000004 2020-06-12T02:37:49.022 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.022 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.022 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:49.023 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.875805+0000", 2020-06-12T02:37:49.024 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019116942000000001 2020-06-12T02:37:49.024 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:49.024 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:49.025 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:49.025 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.025 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.025 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1572 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~745472] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-06-12T02:37:49.025 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.886439+0000", 2020-06-12T02:37:49.026 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 15.107765794000001, 2020-06-12T02:37:49.026 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021442571000000001, 2020-06-12T02:37:49.026 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:49.026 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:49.026 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:49.027 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:49.027 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:49.027 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1572 2020-06-12T02:37:49.027 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.027 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:49.028 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.028 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:49.028 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.886439+0000", 2020-06-12T02:37:49.028 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:49.028 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.029 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.029 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:49.029 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.886439+0000", 2020-06-12T02:37:49.029 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00073027599999999997 2020-06-12T02:37:49.029 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.029 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.030 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:49.030 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887169+0000", 2020-06-12T02:37:49.030 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.49e-06 2020-06-12T02:37:49.030 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.031 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.031 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:49.031 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887175+0000", 2020-06-12T02:37:49.031 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7899999999999994e-06 2020-06-12T02:37:49.031 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.031 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.032 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:49.032 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887185+0000", 2020-06-12T02:37:49.032 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.42e-06 2020-06-12T02:37:49.032 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.032 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.033 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:49.033 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887189+0000", 2020-06-12T02:37:49.033 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1619999999999999e-05 2020-06-12T02:37:49.033 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.033 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.034 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:49.034 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887221+0000", 2020-06-12T02:37:49.034 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9720000000000002e-05 2020-06-12T02:37:49.034 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.034 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.034 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:49.035 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887300+0000", 2020-06-12T02:37:49.035 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047354100000000001 2020-06-12T02:37:49.035 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.035 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.035 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:49.036 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887774+0000", 2020-06-12T02:37:49.036 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0030649750000000002 2020-06-12T02:37:49.036 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.036 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.036 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:49.036 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.890839+0000", 2020-06-12T02:37:49.037 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016995321000000001 2020-06-12T02:37:49.037 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.037 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.037 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:49.037 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.907834+0000", 2020-06-12T02:37:49.038 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7417999999999999e-05 2020-06-12T02:37:49.038 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.038 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.038 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:49.038 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.907882+0000", 2020-06-12T02:37:49.039 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021442571000000001 2020-06-12T02:37:49.039 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:49.039 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:49.039 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:49.039 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.040 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.040 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1573 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~712704] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-06-12T02:37:49.040 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.890888+0000", 2020-06-12T02:37:49.040 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 15.103317059, 2020-06-12T02:37:49.040 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018868001999999998, 2020-06-12T02:37:49.041 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:49.041 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:49.041 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:49.041 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:49.041 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:49.042 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1573 2020-06-12T02:37:49.042 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.042 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:49.042 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.042 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:49.043 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.890888+0000", 2020-06-12T02:37:49.043 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:49.043 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.043 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.043 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:49.043 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.890888+0000", 2020-06-12T02:37:49.044 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000683605 2020-06-12T02:37:49.044 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.044 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.044 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:49.044 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891571+0000", 2020-06-12T02:37:49.045 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4999999999999999e-06 2020-06-12T02:37:49.045 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.045 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.045 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:49.045 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891577+0000", 2020-06-12T02:37:49.046 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.171e-05 2020-06-12T02:37:49.046 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.046 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.046 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:49.046 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891589+0000", 2020-06-12T02:37:49.046 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7650000000000001e-06 2020-06-12T02:37:49.047 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.047 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.047 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:49.047 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891593+0000", 2020-06-12T02:37:49.047 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6689000000000001e-05 2020-06-12T02:37:49.048 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.048 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.048 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:49.048 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891630+0000", 2020-06-12T02:37:49.048 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9877999999999997e-05 2020-06-12T02:37:49.048 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.049 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.049 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:49.049 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891710+0000", 2020-06-12T02:37:49.049 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053367200000000001 2020-06-12T02:37:49.049 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.050 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.050 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:49.050 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.892244+0000", 2020-06-12T02:37:49.050 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004927196 2020-06-12T02:37:49.050 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.051 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.051 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:49.051 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.897171+0000", 2020-06-12T02:37:49.051 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012496419 2020-06-12T02:37:49.051 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.052 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.052 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:49.052 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.909667+0000", 2020-06-12T02:37:49.052 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8567999999999996e-05 2020-06-12T02:37:49.052 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.053 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.053 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:49.053 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.909756+0000", 2020-06-12T02:37:49.053 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018868001999999998 2020-06-12T02:37:49.054 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:49.054 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:49.054 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:49.054 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.055 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.055 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1576 3.5s0 3:a446366d:::smithi10729407-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e26)", 2020-06-12T02:37:49.055 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.891955+0000", 2020-06-12T02:37:49.055 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 15.102249819000001, 2020-06-12T02:37:49.056 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019893246999999999, 2020-06-12T02:37:49.056 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:49.056 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:49.056 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:49.056 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:49.057 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:49.057 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1576 2020-06-12T02:37:49.057 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.057 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:49.057 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.058 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:49.058 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891955+0000", 2020-06-12T02:37:49.058 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:49.058 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.058 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.059 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:49.059 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891955+0000", 2020-06-12T02:37:49.059 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2067999999999999e-05 2020-06-12T02:37:49.059 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.059 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.060 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:49.060 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891977+0000", 2020-06-12T02:37:49.060 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.18e-06 2020-06-12T02:37:49.060 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.060 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.061 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:49.061 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891980+0000", 2020-06-12T02:37:49.061 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3629999999999999e-06 2020-06-12T02:37:49.061 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.061 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.061 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:49.062 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891987+0000", 2020-06-12T02:37:49.062 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.181e-06 2020-06-12T02:37:49.062 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.062 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.063 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:49.063 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891989+0000", 2020-06-12T02:37:49.063 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050959120000000002 2020-06-12T02:37:49.063 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.064 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.064 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:49.064 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.897085+0000", 2020-06-12T02:37:49.065 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1089999999999998e-05 2020-06-12T02:37:49.065 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.065 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.065 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:49.066 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.897166+0000", 2020-06-12T02:37:49.066 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013400235 2020-06-12T02:37:49.066 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.066 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.066 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:49.067 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.910566+0000", 2020-06-12T02:37:49.067 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000104981 2020-06-12T02:37:49.067 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.067 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.067 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:49.068 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.910671+0000", 2020-06-12T02:37:49.068 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001177237 2020-06-12T02:37:49.068 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.068 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.068 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:49.069 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.911848+0000", 2020-06-12T02:37:49.069 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019893246999999999 2020-06-12T02:37:49.069 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:49.069 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:49.069 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:49.069 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.070 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.070 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2913 3.5s0 3:a9510bda:::smithi10729407-14:head [append 0~770048,stat] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:49.070 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.070354+0000", 2020-06-12T02:37:49.070 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.923851018000001, 2020-06-12T02:37:49.070 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60690798599999995, 2020-06-12T02:37:49.071 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:49.071 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:49.071 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:49.071 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:49.071 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:49.072 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2913 2020-06-12T02:37:49.072 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.072 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:49.072 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.072 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:49.073 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.070354+0000", 2020-06-12T02:37:49.073 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:49.073 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.073 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.073 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:49.073 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.070354+0000", 2020-06-12T02:37:49.074 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00098478900000000002 2020-06-12T02:37:49.074 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.074 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.074 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:49.074 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071339+0000", 2020-06-12T02:37:49.075 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8349999999999993e-06 2020-06-12T02:37:49.075 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.075 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.075 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:49.075 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071347+0000", 2020-06-12T02:37:49.075 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0749e-05 2020-06-12T02:37:49.076 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.076 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.076 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:49.076 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071358+0000", 2020-06-12T02:37:49.076 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9369999999999997e-06 2020-06-12T02:37:49.077 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.077 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.077 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:49.077 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071362+0000", 2020-06-12T02:37:49.077 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0426e-05 2020-06-12T02:37:49.078 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.078 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.078 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:49.078 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071423+0000", 2020-06-12T02:37:49.078 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4129999999999998e-05 2020-06-12T02:37:49.079 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.079 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.079 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:49.079 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071487+0000", 2020-06-12T02:37:49.079 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053752599999999998 2020-06-12T02:37:49.080 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.080 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.080 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:49.080 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072024+0000", 2020-06-12T02:37:49.080 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036162579999999998 2020-06-12T02:37:49.081 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.081 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.081 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:49.081 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075640+0000", 2020-06-12T02:37:49.081 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60152201999999999 2020-06-12T02:37:49.082 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.082 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.082 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:49.082 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677162+0000", 2020-06-12T02:37:49.082 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9315999999999996e-05 2020-06-12T02:37:49.082 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.083 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.083 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:49.083 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677262+0000", 2020-06-12T02:37:49.083 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60690798599999995 2020-06-12T02:37:49.083 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:49.084 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:49.084 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:49.084 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.084 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.084 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2914 3.5s0 3:a9510bda:::smithi10729407-14:head [append 0~401408] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:49.084 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.072409+0000", 2020-06-12T02:37:49.085 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.921795574000001, 2020-06-12T02:37:49.085 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.61956801100000003, 2020-06-12T02:37:49.085 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:49.085 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:49.085 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:49.086 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:49.086 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:49.086 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2914 2020-06-12T02:37:49.086 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.086 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:49.087 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.087 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:49.087 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072409+0000", 2020-06-12T02:37:49.087 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:49.087 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.087 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.088 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:49.088 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072409+0000", 2020-06-12T02:37:49.088 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000446317 2020-06-12T02:37:49.088 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.088 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.089 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:49.089 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072856+0000", 2020-06-12T02:37:49.089 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2140000000000004e-06 2020-06-12T02:37:49.089 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.089 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.089 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:49.090 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072861+0000", 2020-06-12T02:37:49.090 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8710000000000003e-06 2020-06-12T02:37:49.090 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.090 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.091 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:49.091 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072870+0000", 2020-06-12T02:37:49.091 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2270000000000004e-06 2020-06-12T02:37:49.091 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.091 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.092 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:49.092 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072874+0000", 2020-06-12T02:37:49.092 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024649960000000001 2020-06-12T02:37:49.092 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.092 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.092 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:49.093 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075339+0000", 2020-06-12T02:37:49.093 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9179999999999996e-05 2020-06-12T02:37:49.093 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.093 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.093 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:49.094 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075438+0000", 2020-06-12T02:37:49.094 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60189281400000005 2020-06-12T02:37:49.094 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.094 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.094 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:49.094 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677331+0000", 2020-06-12T02:37:49.095 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012602800000000001 2020-06-12T02:37:49.095 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.095 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.095 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:49.095 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677457+0000", 2020-06-12T02:37:49.096 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00088167699999999998 2020-06-12T02:37:49.096 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.096 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.096 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:49.096 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.678339+0000", 2020-06-12T02:37:49.096 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004312995 2020-06-12T02:37:49.097 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.097 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.097 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:49.097 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.682652+0000", 2020-06-12T02:37:49.097 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0092869140000000003 2020-06-12T02:37:49.098 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.098 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.098 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:49.098 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691938+0000", 2020-06-12T02:37:49.098 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8778e-05 2020-06-12T02:37:49.098 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.099 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.099 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:49.099 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691977+0000", 2020-06-12T02:37:49.099 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.61956801100000003 2020-06-12T02:37:49.099 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:49.100 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:49.100 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:49.100 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.100 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.100 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2915 3.5s0 3:a9510bda:::smithi10729407-14:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:49.100 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.072912+0000", 2020-06-12T02:37:49.101 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.921292834000001, 2020-06-12T02:37:49.101 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.6192204, 2020-06-12T02:37:49.101 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:49.101 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:49.101 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:49.102 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:49.102 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:49.102 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2915 2020-06-12T02:37:49.102 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.102 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:49.102 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.103 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:49.103 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072912+0000", 2020-06-12T02:37:49.103 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:49.103 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.104 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.104 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:49.104 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072912+0000", 2020-06-12T02:37:49.104 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9201e-05 2020-06-12T02:37:49.104 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.104 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.105 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:49.105 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072951+0000", 2020-06-12T02:37:49.105 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5480000000000002e-06 2020-06-12T02:37:49.105 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.105 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.106 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:49.106 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072955+0000", 2020-06-12T02:37:49.106 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3879999999999997e-06 2020-06-12T02:37:49.106 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.106 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.106 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:49.107 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072961+0000", 2020-06-12T02:37:49.107 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6589999999999999e-06 2020-06-12T02:37:49.107 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.107 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.107 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:49.108 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072964+0000", 2020-06-12T02:37:49.108 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025221039999999998 2020-06-12T02:37:49.108 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.108 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.108 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:49.108 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075486+0000", 2020-06-12T02:37:49.109 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5038999999999998e-05 2020-06-12T02:37:49.109 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.109 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.109 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:49.109 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075561+0000", 2020-06-12T02:37:49.110 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60452397499999999 2020-06-12T02:37:49.110 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.110 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.110 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:49.110 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680085+0000", 2020-06-12T02:37:49.110 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1398999999999999e-05 2020-06-12T02:37:49.111 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.111 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.111 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:49.111 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680156+0000", 2020-06-12T02:37:49.111 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000274465 2020-06-12T02:37:49.111 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.112 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.112 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:49.112 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680431+0000", 2020-06-12T02:37:49.112 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025071999999999998 2020-06-12T02:37:49.112 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.113 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.113 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:49.113 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.682938+0000", 2020-06-12T02:37:49.113 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0091116449999999998 2020-06-12T02:37:49.113 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.114 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.114 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:49.114 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692050+0000", 2020-06-12T02:37:49.114 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2776999999999997e-05 2020-06-12T02:37:49.114 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.114 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.115 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:49.115 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692132+0000", 2020-06-12T02:37:49.115 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.6192204 2020-06-12T02:37:49.115 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:49.116 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:49.116 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:49.116 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.116 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.116 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2916 3.5s0 3:a9510bda:::smithi10729407-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:37:49.116 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.072986+0000", 2020-06-12T02:37:49.117 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.921218594999999, 2020-06-12T02:37:49.117 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.62063941099999997, 2020-06-12T02:37:49.117 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:49.117 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:49.117 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:49.118 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:49.118 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:49.118 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2916 2020-06-12T02:37:49.118 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.118 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:49.119 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.119 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:49.119 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072986+0000", 2020-06-12T02:37:49.119 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:49.119 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.120 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.120 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:49.120 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072986+0000", 2020-06-12T02:37:49.120 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2101000000000001e-05 2020-06-12T02:37:49.120 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.120 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.121 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:49.121 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073008+0000", 2020-06-12T02:37:49.121 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1549999999999999e-06 2020-06-12T02:37:49.121 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.121 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.122 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:49.122 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073011+0000", 2020-06-12T02:37:49.122 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2360000000000001e-06 2020-06-12T02:37:49.122 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.122 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.123 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:49.123 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073018+0000", 2020-06-12T02:37:49.123 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2790000000000001e-06 2020-06-12T02:37:49.123 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.123 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.123 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:49.124 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073020+0000", 2020-06-12T02:37:49.124 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025637670000000002 2020-06-12T02:37:49.124 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.124 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.124 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:49.125 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075584+0000", 2020-06-12T02:37:49.125 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1814999999999998e-05 2020-06-12T02:37:49.125 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.125 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.125 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:49.125 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075636+0000", 2020-06-12T02:37:49.126 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60499943199999995 2020-06-12T02:37:49.126 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.126 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.126 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:49.126 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680635+0000", 2020-06-12T02:37:49.127 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0968000000000002e-05 2020-06-12T02:37:49.127 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.127 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.127 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:49.127 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680706+0000", 2020-06-12T02:37:49.128 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011777101999999999 2020-06-12T02:37:49.128 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.128 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.128 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:49.128 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692483+0000", 2020-06-12T02:37:49.129 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5890999999999995e-05 2020-06-12T02:37:49.129 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.129 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.129 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:49.130 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692569+0000", 2020-06-12T02:37:49.130 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001056665 2020-06-12T02:37:49.131 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.131 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.131 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:49.131 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.693626+0000", 2020-06-12T02:37:49.132 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.62063941099999997 2020-06-12T02:37:49.132 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:49.132 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:49.133 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:49.133 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.133 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.134 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2922 3.ds1 3:b983fd28:::smithi10729407-1:head [append 0~466944] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:49.134 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.097192+0000", 2020-06-12T02:37:49.134 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.897013228, 2020-06-12T02:37:49.135 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58672644699999998, 2020-06-12T02:37:49.135 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:49.135 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:49.135 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:49.136 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:49.136 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:49.136 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2922 2020-06-12T02:37:49.137 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.137 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:49.137 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.138 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:49.138 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097192+0000", 2020-06-12T02:37:49.138 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:49.139 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.139 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.139 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:49.139 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097192+0000", 2020-06-12T02:37:49.140 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047206100000000002 2020-06-12T02:37:49.140 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.140 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.141 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:49.141 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097664+0000", 2020-06-12T02:37:49.141 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8359999999999999e-06 2020-06-12T02:37:49.142 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.142 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.142 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:49.143 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097668+0000", 2020-06-12T02:37:49.143 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0815999999999999e-05 2020-06-12T02:37:49.143 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.144 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.144 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:49.144 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097679+0000", 2020-06-12T02:37:49.145 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.028e-06 2020-06-12T02:37:49.145 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.145 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.146 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:49.146 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097684+0000", 2020-06-12T02:37:49.146 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3932999999999999e-05 2020-06-12T02:37:49.146 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.147 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.147 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:49.147 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097748+0000", 2020-06-12T02:37:49.148 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2214999999999999e-05 2020-06-12T02:37:49.148 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.148 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.149 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:49.149 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097820+0000", 2020-06-12T02:37:49.149 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00041583800000000003 2020-06-12T02:37:49.150 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.150 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.150 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:49.150 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.098236+0000", 2020-06-12T02:37:49.151 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0020339960000000002 2020-06-12T02:37:49.151 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.151 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.152 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:49.152 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.100270+0000", 2020-06-12T02:37:49.152 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58355657299999997 2020-06-12T02:37:49.153 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.153 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.153 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:49.160 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.683827+0000", 2020-06-12T02:37:49.160 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1150999999999995e-05 2020-06-12T02:37:49.160 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.161 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.161 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:49.161 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.683918+0000", 2020-06-12T02:37:49.162 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58672644699999998 2020-06-12T02:37:49.162 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:49.162 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:49.163 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:49.163 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.163 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.163 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2923 3.ds1 3:b983fd28:::smithi10729407-1:head [append 0~614400] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:49.164 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.100504+0000", 2020-06-12T02:37:49.164 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.893700587, 2020-06-12T02:37:49.164 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58853946999999995, 2020-06-12T02:37:49.165 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:49.165 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:49.165 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:49.165 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:49.166 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:49.166 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2923 2020-06-12T02:37:49.166 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.166 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:49.167 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.167 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:49.167 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.100504+0000", 2020-06-12T02:37:49.167 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:49.168 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.168 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.168 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:49.168 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.100504+0000", 2020-06-12T02:37:49.169 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00068395399999999996 2020-06-12T02:37:49.169 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.169 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.169 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:49.170 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101188+0000", 2020-06-12T02:37:49.170 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4050000000000003e-06 2020-06-12T02:37:49.170 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.171 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.171 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:49.172 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101196+0000", 2020-06-12T02:37:49.172 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.328e-05 2020-06-12T02:37:49.172 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.173 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.173 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:49.173 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101209+0000", 2020-06-12T02:37:49.173 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3419999999999996e-06 2020-06-12T02:37:49.174 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.174 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.174 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:49.174 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101215+0000", 2020-06-12T02:37:49.175 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1389000000000005e-05 2020-06-12T02:37:49.177 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.178 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.178 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:49.178 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101277+0000", 2020-06-12T02:37:49.179 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7724000000000004e-05 2020-06-12T02:37:49.179 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.179 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.180 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:49.180 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101344+0000", 2020-06-12T02:37:49.180 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047852499999999999 2020-06-12T02:37:49.181 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.181 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.181 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:49.181 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101823+0000", 2020-06-12T02:37:49.182 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0039255510000000002 2020-06-12T02:37:49.182 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.182 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.183 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:49.183 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105748+0000", 2020-06-12T02:37:49.183 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58321340099999996 2020-06-12T02:37:49.184 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.184 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.185 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:49.185 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.688962+0000", 2020-06-12T02:37:49.185 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1898999999999996e-05 2020-06-12T02:37:49.186 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.186 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.186 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:49.187 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689044+0000", 2020-06-12T02:37:49.187 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58853946999999995 2020-06-12T02:37:49.187 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:49.188 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:49.188 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:49.188 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.188 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.189 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2924 3.ds1 3:b983fd28:::smithi10729407-1:head [append 0~188416] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:49.189 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.101633+0000", 2020-06-12T02:37:49.190 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.892571593, 2020-06-12T02:37:49.190 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58758259199999996, 2020-06-12T02:37:49.190 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:49.190 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:49.191 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:49.191 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:49.191 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:49.192 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2924 2020-06-12T02:37:49.192 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.192 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:49.193 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.193 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:49.193 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101633+0000", 2020-06-12T02:37:49.194 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:49.194 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.194 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.195 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:49.195 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101633+0000", 2020-06-12T02:37:49.195 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00018775399999999999 2020-06-12T02:37:49.195 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.196 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.196 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:49.196 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101821+0000", 2020-06-12T02:37:49.196 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7669999999999998e-06 2020-06-12T02:37:49.196 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.197 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.197 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:49.197 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101828+0000", 2020-06-12T02:37:49.197 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2716e-05 2020-06-12T02:37:49.197 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.198 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.198 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:49.198 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101840+0000", 2020-06-12T02:37:49.198 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.215e-06 2020-06-12T02:37:49.198 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.199 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.199 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:49.199 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101846+0000", 2020-06-12T02:37:49.199 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021214250000000001 2020-06-12T02:37:49.199 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.200 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.200 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:49.200 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.103967+0000", 2020-06-12T02:37:49.200 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1018000000000003e-05 2020-06-12T02:37:49.200 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.201 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.201 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:49.201 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.104038+0000", 2020-06-12T02:37:49.201 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000333364 2020-06-12T02:37:49.201 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.202 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.202 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:49.202 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.104371+0000", 2020-06-12T02:37:49.202 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001378546 2020-06-12T02:37:49.202 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.203 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.203 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:49.203 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105750+0000", 2020-06-12T02:37:49.203 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58338272899999999 2020-06-12T02:37:49.203 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.204 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.204 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:49.204 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689133+0000", 2020-06-12T02:37:49.204 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3058000000000003e-05 2020-06-12T02:37:49.204 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.205 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.205 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:49.205 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689216+0000", 2020-06-12T02:37:49.206 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58758259199999996 2020-06-12T02:37:49.206 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:49.206 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:49.206 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:49.206 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.207 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.207 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2925 3.ds1 3:b983fd28:::smithi10729407-1:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:49.207 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.101899+0000", 2020-06-12T02:37:49.207 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.892305928000001, 2020-06-12T02:37:49.207 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58747810899999997, 2020-06-12T02:37:49.208 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:49.208 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:49.208 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:49.208 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:49.208 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:49.209 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2925 2020-06-12T02:37:49.209 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.209 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:49.209 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.209 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:49.210 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101899+0000", 2020-06-12T02:37:49.210 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:49.210 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.210 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.210 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:49.211 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101899+0000", 2020-06-12T02:37:49.211 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2906999999999998e-05 2020-06-12T02:37:49.211 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.211 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.211 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:49.212 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101972+0000", 2020-06-12T02:37:49.212 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1789999999999996e-06 2020-06-12T02:37:49.212 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.212 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.212 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:49.212 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101978+0000", 2020-06-12T02:37:49.213 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1358e-05 2020-06-12T02:37:49.213 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.213 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.213 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:49.214 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101989+0000", 2020-06-12T02:37:49.214 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.031e-06 2020-06-12T02:37:49.214 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.214 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.214 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:49.215 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101993+0000", 2020-06-12T02:37:49.215 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033218990000000001 2020-06-12T02:37:49.215 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.215 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.216 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:49.216 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105315+0000", 2020-06-12T02:37:49.216 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6324999999999999e-05 2020-06-12T02:37:49.216 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.216 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.217 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:49.217 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105382+0000", 2020-06-12T02:37:49.217 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000216624 2020-06-12T02:37:49.217 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.217 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.218 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:49.218 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105598+0000", 2020-06-12T02:37:49.218 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001236586 2020-06-12T02:37:49.218 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.218 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.219 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:49.219 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.106835+0000", 2020-06-12T02:37:49.219 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58245509200000001 2020-06-12T02:37:49.219 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.219 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.220 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:49.220 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689290+0000", 2020-06-12T02:37:49.220 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7107999999999993e-05 2020-06-12T02:37:49.220 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.220 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.221 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:49.221 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689377+0000", 2020-06-12T02:37:49.221 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58747810899999997 2020-06-12T02:37:49.221 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:49.221 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:49.221 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:49.222 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.222 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.222 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2926 3.ds1 3:b983fd28:::smithi10729407-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:37:49.222 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.102036+0000", 2020-06-12T02:37:49.222 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.892168944, 2020-06-12T02:37:49.223 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58868048699999997, 2020-06-12T02:37:49.223 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:49.223 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:49.223 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:49.223 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:49.224 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:49.224 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2926 2020-06-12T02:37:49.224 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.224 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:49.224 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.225 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:49.225 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102036+0000", 2020-06-12T02:37:49.225 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:49.225 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.226 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.226 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:49.226 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102036+0000", 2020-06-12T02:37:49.226 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3786999999999999e-05 2020-06-12T02:37:49.226 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.227 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.227 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:49.227 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102080+0000", 2020-06-12T02:37:49.227 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7660000000000004e-06 2020-06-12T02:37:49.227 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.228 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.228 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:49.228 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102085+0000", 2020-06-12T02:37:49.228 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3145e-05 2020-06-12T02:37:49.228 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.229 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.229 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:49.229 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102099+0000", 2020-06-12T02:37:49.229 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2440000000000004e-06 2020-06-12T02:37:49.229 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.230 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.230 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:49.230 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102103+0000", 2020-06-12T02:37:49.230 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036826990000000002 2020-06-12T02:37:49.230 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.231 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.231 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:49.231 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105786+0000", 2020-06-12T02:37:49.231 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.6774000000000006e-05 2020-06-12T02:37:49.231 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.232 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.232 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:49.232 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105882+0000", 2020-06-12T02:37:49.232 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58352084900000001 2020-06-12T02:37:49.232 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.233 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.233 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:49.233 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689403+0000", 2020-06-12T02:37:49.233 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3788999999999995e-05 2020-06-12T02:37:49.233 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.234 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.234 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:49.234 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689467+0000", 2020-06-12T02:37:49.234 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001249434 2020-06-12T02:37:49.234 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.235 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.235 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:49.235 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690716+0000", 2020-06-12T02:37:49.235 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58868048699999997 2020-06-12T02:37:49.236 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:49.236 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:49.236 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:49.236 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.236 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.237 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2950 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~598016] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:49.237 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.162562+0000", 2020-06-12T02:37:49.237 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.83164257, 2020-06-12T02:37:49.237 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52780611399999999, 2020-06-12T02:37:49.237 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:49.238 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:49.238 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:49.238 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:49.238 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:49.238 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2950 2020-06-12T02:37:49.239 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.239 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:49.239 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.239 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:49.239 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.162562+0000", 2020-06-12T02:37:49.240 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:49.240 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.240 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.240 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:49.240 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.162562+0000", 2020-06-12T02:37:49.241 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00070718099999999998 2020-06-12T02:37:49.241 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.241 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.241 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:49.241 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163269+0000", 2020-06-12T02:37:49.242 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2029999999999997e-06 2020-06-12T02:37:49.242 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.242 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.242 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:49.242 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163276+0000", 2020-06-12T02:37:49.243 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2734999999999999e-05 2020-06-12T02:37:49.243 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.243 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.243 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:49.243 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163288+0000", 2020-06-12T02:37:49.244 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.404e-06 2020-06-12T02:37:49.244 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.244 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.244 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:49.244 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163295+0000", 2020-06-12T02:37:49.245 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5784000000000006e-05 2020-06-12T02:37:49.245 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.245 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.245 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:49.245 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163361+0000", 2020-06-12T02:37:49.246 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5318e-05 2020-06-12T02:37:49.246 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.246 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.246 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:49.247 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163436+0000", 2020-06-12T02:37:49.247 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054801100000000005 2020-06-12T02:37:49.247 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.247 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.247 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:49.248 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163984+0000", 2020-06-12T02:37:49.248 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.007023077 2020-06-12T02:37:49.248 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.248 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.248 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:49.249 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171007+0000", 2020-06-12T02:37:49.249 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51932051199999996 2020-06-12T02:37:49.249 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.249 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.249 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:49.250 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690327+0000", 2020-06-12T02:37:49.250 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0889000000000001e-05 2020-06-12T02:37:49.250 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.250 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.250 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:49.251 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690368+0000", 2020-06-12T02:37:49.251 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52780611399999999 2020-06-12T02:37:49.251 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:49.251 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:49.251 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:49.251 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.252 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.252 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2951 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~532480] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:49.252 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.165470+0000", 2020-06-12T02:37:49.252 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.828734962, 2020-06-12T02:37:49.252 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52507917900000001, 2020-06-12T02:37:49.253 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:49.253 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:49.253 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:49.253 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:49.253 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:49.254 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2951 2020-06-12T02:37:49.254 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.254 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:49.254 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.254 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:49.255 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.165470+0000", 2020-06-12T02:37:49.255 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:49.255 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.255 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.255 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:49.256 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.165470+0000", 2020-06-12T02:37:49.256 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00062231600000000004 2020-06-12T02:37:49.256 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.256 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.257 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:49.257 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166092+0000", 2020-06-12T02:37:49.257 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1610000000000002e-06 2020-06-12T02:37:49.257 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.258 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.258 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:49.258 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166097+0000", 2020-06-12T02:37:49.258 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7780000000000002e-06 2020-06-12T02:37:49.258 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.258 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.259 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:49.259 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166107+0000", 2020-06-12T02:37:49.259 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7169999999999999e-06 2020-06-12T02:37:49.259 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.259 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.260 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:49.260 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166112+0000", 2020-06-12T02:37:49.260 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4792000000000002e-05 2020-06-12T02:37:49.260 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.260 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.261 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:49.261 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166147+0000", 2020-06-12T02:37:49.261 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9336e-05 2020-06-12T02:37:49.261 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.261 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.262 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:49.262 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166236+0000", 2020-06-12T02:37:49.262 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00067747200000000004 2020-06-12T02:37:49.262 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.262 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.263 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:49.263 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166913+0000", 2020-06-12T02:37:49.263 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0047718669999999999 2020-06-12T02:37:49.263 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.263 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.264 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:49.264 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171685+0000", 2020-06-12T02:37:49.264 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51881990700000002 2020-06-12T02:37:49.264 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.264 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.265 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:49.265 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690505+0000", 2020-06-12T02:37:49.265 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3832999999999999e-05 2020-06-12T02:37:49.265 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.265 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.266 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:49.266 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690549+0000", 2020-06-12T02:37:49.266 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52507917900000001 2020-06-12T02:37:49.266 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:49.267 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:49.267 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:49.267 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.267 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.267 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2952 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~270336] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:49.268 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.166891+0000", 2020-06-12T02:37:49.268 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.827313312999999, 2020-06-12T02:37:49.268 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52426417000000003, 2020-06-12T02:37:49.268 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:49.268 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:49.269 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:49.269 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:49.269 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:49.269 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2952 2020-06-12T02:37:49.269 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.270 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:49.270 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.270 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:49.270 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166891+0000", 2020-06-12T02:37:49.270 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:49.271 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.271 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.271 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:49.271 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166891+0000", 2020-06-12T02:37:49.271 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00031878599999999999 2020-06-12T02:37:49.272 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.272 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.272 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:49.272 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167210+0000", 2020-06-12T02:37:49.272 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2360000000000004e-06 2020-06-12T02:37:49.273 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.273 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.273 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:49.273 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167219+0000", 2020-06-12T02:37:49.273 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3298e-05 2020-06-12T02:37:49.274 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.274 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.274 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:49.274 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167232+0000", 2020-06-12T02:37:49.274 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5849999999999999e-06 2020-06-12T02:37:49.275 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.275 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.275 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:49.275 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167237+0000", 2020-06-12T02:37:49.275 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0018755740000000001 2020-06-12T02:37:49.275 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.276 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.276 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:49.276 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.169113+0000", 2020-06-12T02:37:49.276 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000104768 2020-06-12T02:37:49.277 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.277 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.277 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:49.277 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.169218+0000", 2020-06-12T02:37:49.278 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00057389799999999999 2020-06-12T02:37:49.278 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.278 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.278 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:49.278 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.169792+0000", 2020-06-12T02:37:49.279 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0019333989999999999 2020-06-12T02:37:49.279 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.279 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.279 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:49.279 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171725+0000", 2020-06-12T02:37:49.280 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51938508699999997 2020-06-12T02:37:49.280 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.280 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.280 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:49.280 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691110+0000", 2020-06-12T02:37:49.280 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5538999999999999e-05 2020-06-12T02:37:49.281 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.281 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.281 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:49.281 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691156+0000", 2020-06-12T02:37:49.281 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52426417000000003 2020-06-12T02:37:49.282 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:49.282 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:49.282 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:49.282 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.282 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.283 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2953 3.5s0 3:a446366d:::smithi10729407-11:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:49.283 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.167293+0000", 2020-06-12T02:37:49.283 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.826911866, 2020-06-12T02:37:49.283 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52428350999999995, 2020-06-12T02:37:49.283 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:49.284 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:49.284 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:49.284 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:49.284 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:49.284 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2953 2020-06-12T02:37:49.285 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.285 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:49.285 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.285 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:49.285 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167293+0000", 2020-06-12T02:37:49.286 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:49.286 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.286 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.286 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:49.286 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167293+0000", 2020-06-12T02:37:49.287 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3880000000000004e-05 2020-06-12T02:37:49.287 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.287 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.287 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:49.288 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167367+0000", 2020-06-12T02:37:49.288 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.319e-06 2020-06-12T02:37:49.288 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.288 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.288 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:49.289 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167373+0000", 2020-06-12T02:37:49.289 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1719000000000001e-05 2020-06-12T02:37:49.289 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.289 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.290 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:49.290 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167385+0000", 2020-06-12T02:37:49.290 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4580000000000003e-06 2020-06-12T02:37:49.290 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.291 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.291 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:49.291 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167389+0000", 2020-06-12T02:37:49.291 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036341110000000002 2020-06-12T02:37:49.291 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.292 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.292 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:49.292 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171023+0000", 2020-06-12T02:37:49.292 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9418000000000005e-05 2020-06-12T02:37:49.292 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.293 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.293 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:49.293 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171093+0000", 2020-06-12T02:37:49.293 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00029087900000000002 2020-06-12T02:37:49.293 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.294 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.294 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:49.294 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171384+0000", 2020-06-12T02:37:49.294 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000489347 2020-06-12T02:37:49.294 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.294 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.295 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:49.295 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171873+0000", 2020-06-12T02:37:49.295 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51963691300000003 2020-06-12T02:37:49.295 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.295 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.296 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:49.296 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691510+0000", 2020-06-12T02:37:49.296 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6465999999999997e-05 2020-06-12T02:37:49.296 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.296 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.296 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:49.297 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691576+0000", 2020-06-12T02:37:49.297 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52428350999999995 2020-06-12T02:37:49.297 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:49.298 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:49.298 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:49.298 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.298 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.298 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2954 3.5s0 3:a446366d:::smithi10729407-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:37:49.299 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.167432+0000", 2020-06-12T02:37:49.299 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.826773117, 2020-06-12T02:37:49.299 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52568652699999996, 2020-06-12T02:37:49.299 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:49.299 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:49.299 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:49.300 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:49.300 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:49.300 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2954 2020-06-12T02:37:49.300 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.300 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:49.301 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.301 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:49.301 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167432+0000", 2020-06-12T02:37:49.301 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:49.301 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.302 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.302 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:49.302 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167432+0000", 2020-06-12T02:37:49.302 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3556000000000001e-05 2020-06-12T02:37:49.302 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.302 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.303 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:49.303 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167475+0000", 2020-06-12T02:37:49.303 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7350000000000001e-06 2020-06-12T02:37:49.303 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.303 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.303 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:49.304 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167481+0000", 2020-06-12T02:37:49.304 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3128e-05 2020-06-12T02:37:49.304 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.304 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.304 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:49.305 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167494+0000", 2020-06-12T02:37:49.305 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1579999999999998e-06 2020-06-12T02:37:49.305 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.305 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.305 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:49.306 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167498+0000", 2020-06-12T02:37:49.306 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004082174 2020-06-12T02:37:49.306 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.306 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.306 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:49.306 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171580+0000", 2020-06-12T02:37:49.307 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8954000000000002e-05 2020-06-12T02:37:49.307 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.307 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.307 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:49.308 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171679+0000", 2020-06-12T02:37:49.308 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51993235100000001 2020-06-12T02:37:49.308 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.308 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.308 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:49.309 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691612+0000", 2020-06-12T02:37:49.309 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4329000000000001e-05 2020-06-12T02:37:49.309 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.309 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.309 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:49.310 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691666+0000", 2020-06-12T02:37:49.310 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001452142 2020-06-12T02:37:49.310 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.310 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.310 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:49.310 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.693118+0000", 2020-06-12T02:37:49.311 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52568652699999996 2020-06-12T02:37:49.311 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:49.311 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:49.311 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:49.311 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.312 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.312 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2959 3.ds1 3:b3445631:::smithi10729407-6:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e27)", 2020-06-12T02:37:49.312 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.176248+0000", 2020-06-12T02:37:49.312 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.817956376, 2020-06-12T02:37:49.312 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51549877099999997, 2020-06-12T02:37:49.312 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:49.313 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:49.313 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:49.313 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:49.313 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:49.313 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2959 2020-06-12T02:37:49.314 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.314 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:49.314 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.314 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:49.314 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176248+0000", 2020-06-12T02:37:49.315 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:49.315 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.315 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.315 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:49.315 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176248+0000", 2020-06-12T02:37:49.315 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5120999999999999e-05 2020-06-12T02:37:49.316 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.316 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.316 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:49.316 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176274+0000", 2020-06-12T02:37:49.316 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9189999999999998e-06 2020-06-12T02:37:49.317 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.317 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.317 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:49.317 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176278+0000", 2020-06-12T02:37:49.317 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6339e-05 2020-06-12T02:37:49.317 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.318 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.318 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:49.318 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176305+0000", 2020-06-12T02:37:49.318 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9089999999999999e-06 2020-06-12T02:37:49.319 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.319 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.319 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:49.319 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176310+0000", 2020-06-12T02:37:49.319 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6095999999999999e-05 2020-06-12T02:37:49.320 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.320 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.320 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:49.320 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176376+0000", 2020-06-12T02:37:49.320 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8590999999999994e-05 2020-06-12T02:37:49.320 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.321 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.321 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:49.321 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176444+0000", 2020-06-12T02:37:49.321 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00024346400000000001 2020-06-12T02:37:49.321 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.322 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.322 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:49.322 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176688+0000", 2020-06-12T02:37:49.322 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00044220899999999999 2020-06-12T02:37:49.322 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.323 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.323 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:49.323 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.177130+0000", 2020-06-12T02:37:49.323 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51452984400000001 2020-06-12T02:37:49.323 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.323 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.324 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:49.324 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691660+0000", 2020-06-12T02:37:49.324 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7279000000000002e-05 2020-06-12T02:37:49.324 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.324 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.325 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:49.325 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691747+0000", 2020-06-12T02:37:49.325 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51549877099999997 2020-06-12T02:37:49.325 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:49.325 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:49.325 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:49.326 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.326 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.326 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:3032 3.ds1 3:b983fd28:::smithi10729407-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:37:49.326 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.864289+0000", 2020-06-12T02:37:49.326 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.129915457999999, 2020-06-12T02:37:49.327 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019572216999999999, 2020-06-12T02:37:49.327 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:49.327 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:49.327 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:49.327 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:49.327 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:49.328 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 3032 2020-06-12T02:37:49.328 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.328 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:49.328 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.329 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:49.329 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864289+0000", 2020-06-12T02:37:49.329 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:49.329 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.329 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.330 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:49.330 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864289+0000", 2020-06-12T02:37:49.330 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5701999999999997e-05 2020-06-12T02:37:49.330 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.330 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.330 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:49.331 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864335+0000", 2020-06-12T02:37:49.331 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0910000000000004e-06 2020-06-12T02:37:49.331 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.331 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.331 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:49.332 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864342+0000", 2020-06-12T02:37:49.332 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2301e-05 2020-06-12T02:37:49.332 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.332 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.332 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:49.332 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864354+0000", 2020-06-12T02:37:49.333 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5199999999999997e-06 2020-06-12T02:37:49.333 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.333 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.333 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:49.333 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864360+0000", 2020-06-12T02:37:49.334 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00049180999999999997 2020-06-12T02:37:49.334 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.334 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.334 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:49.334 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864852+0000", 2020-06-12T02:37:49.334 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6725000000000003e-05 2020-06-12T02:37:49.335 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.335 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.335 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:49.335 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864908+0000", 2020-06-12T02:37:49.335 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0089687010000000008 2020-06-12T02:37:49.336 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.336 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.336 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:49.336 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.873877+0000", 2020-06-12T02:37:49.336 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3382999999999999e-05 2020-06-12T02:37:49.336 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.337 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.337 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:49.337 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.873941+0000", 2020-06-12T02:37:49.337 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.006408927 2020-06-12T02:37:49.337 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.338 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.338 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:49.338 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.880350+0000", 2020-06-12T02:37:49.338 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3905e-05 2020-06-12T02:37:49.339 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.339 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.339 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:49.339 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.880403+0000", 2020-06-12T02:37:49.339 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0034581519999999999 2020-06-12T02:37:49.339 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:49.340 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:49.340 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:49.340 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.883862+0000", 2020-06-12T02:37:49.340 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019572216999999999 2020-06-12T02:37:49.340 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:49.341 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:49.341 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:49.341 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:49.341 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:49.341 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:37:49.342 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-06-12T02:37:49.358 INFO:tasks.rados.rados.0.smithi107.stdout:3234: done (2 left) 2020-06-12T02:37:49.358 INFO:tasks.rados.rados.0.smithi107.stdout:3238: done (1 left) 2020-06-12T02:37:49.364 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:37:49.365 INFO:tasks.rados.rados.0.smithi107.stdout:3239: seq_num 3238 ranges {0=638976,638976=589824,1228800=212992} 2020-06-12T02:37:49.365 INFO:tasks.rados.rados.0.smithi107.stdout:3239: writing smithi10729407-13 from 0 to 638976 tid 1 2020-06-12T02:37:49.365 INFO:tasks.rados.rados.0.smithi107.stdout:3239: writing smithi10729407-13 from 638976 to 1228800 tid 2 2020-06-12T02:37:49.365 INFO:tasks.rados.rados.0.smithi107.stdout:3239: writing smithi10729407-13 from 1228800 to 1441792 tid 3 2020-06-12T02:37:49.366 INFO:tasks.rados.rados.0.smithi107.stdout:3240: delete oid 19 current snap is 0 2020-06-12T02:37:49.366 INFO:tasks.rados.rados.0.smithi107.stdout:3237: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:49.366 INFO:tasks.rados.rados.0.smithi107.stdout:3237: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:49.366 INFO:tasks.rados.rados.0.smithi107.stdout:3237: finishing write tid 5 to smithi10729407-1 2020-06-12T02:37:49.366 INFO:tasks.rados.rados.0.smithi107.stdout:3237: finishing write tid 6 to smithi10729407-1 2020-06-12T02:37:49.366 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 614 (ObjNum 3236 snap 0 seq_num 3236) dirty exists 2020-06-12T02:37:49.367 INFO:tasks.rados.rados.0.smithi107.stdout:3237: left oid 1 (ObjNum 3236 snap 0 seq_num 3236) 2020-06-12T02:37:49.367 INFO:tasks.rados.rados.0.smithi107.stdout:3237: done (2 left) 2020-06-12T02:37:49.367 INFO:tasks.rados.rados.0.smithi107.stdout:3240: done (1 left) 2020-06-12T02:37:49.367 INFO:tasks.rados.rados.0.smithi107.stdout:3241: delete oid 14 current snap is 0 2020-06-12T02:37:49.367 INFO:tasks.rados.rados.0.smithi107.stdout:3239: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:49.368 INFO:tasks.rados.rados.0.smithi107.stdout:3239: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:49.368 INFO:tasks.rados.rados.0.smithi107.stdout:3241: done (1 left) 2020-06-12T02:37:49.368 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:37:49.368 INFO:tasks.rados.rados.0.smithi107.stdout:3242: seq_num 3241 ranges {0=688128,688128=688128,1376256=114688} 2020-06-12T02:37:49.368 INFO:tasks.rados.rados.0.smithi107.stdout:3242: writing smithi10729407-15 from 0 to 688128 tid 1 2020-06-12T02:37:49.368 INFO:tasks.rados.rados.0.smithi107.stdout:3242: writing smithi10729407-15 from 688128 to 1376256 tid 2 2020-06-12T02:37:49.369 INFO:tasks.rados.rados.0.smithi107.stdout:3242: writing smithi10729407-15 from 1376256 to 1490944 tid 3 2020-06-12T02:37:49.369 INFO:tasks.rados.rados.0.smithi107.stdout:3243: delete oid 6 current snap is 0 2020-06-12T02:37:49.369 INFO:tasks.rados.rados.0.smithi107.stdout:3239: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:49.369 INFO:tasks.rados.rados.0.smithi107.stdout:3239: finishing write tid 5 to smithi10729407-13 2020-06-12T02:37:49.369 INFO:tasks.rados.rados.0.smithi107.stdout:3239: finishing write tid 6 to smithi10729407-13 2020-06-12T02:37:49.370 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 611 (ObjNum 3238 snap 0 seq_num 3238) dirty exists 2020-06-12T02:37:49.370 INFO:tasks.rados.rados.0.smithi107.stdout:3239: left oid 13 (ObjNum 3238 snap 0 seq_num 3238) 2020-06-12T02:37:49.370 INFO:tasks.rados.rados.0.smithi107.stdout:3239: done (2 left) 2020-06-12T02:37:49.370 INFO:tasks.rados.rados.0.smithi107.stdout:3243: done (1 left) 2020-06-12T02:37:49.370 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:37:49.370 INFO:tasks.rados.rados.0.smithi107.stdout:3244: seq_num 3243 ranges {0=720896,720896=458752,1179648=344064} 2020-06-12T02:37:49.371 INFO:tasks.rados.rados.0.smithi107.stdout:3244: writing smithi10729407-20 from 0 to 720896 tid 1 2020-06-12T02:37:49.371 INFO:tasks.rados.rados.0.smithi107.stdout:3244: writing smithi10729407-20 from 720896 to 1179648 tid 2 2020-06-12T02:37:49.371 INFO:tasks.rados.rados.0.smithi107.stdout:3244: writing smithi10729407-20 from 1179648 to 1523712 tid 3 2020-06-12T02:37:49.371 INFO:tasks.rados.rados.0.smithi107.stdout:3245: delete oid 13 current snap is 0 2020-06-12T02:37:49.371 INFO:tasks.rados.rados.0.smithi107.stdout:3242: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:49.372 INFO:tasks.rados.rados.0.smithi107.stdout:3242: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:49.372 INFO:tasks.rados.rados.0.smithi107.stdout:3242: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:49.372 INFO:tasks.rados.rados.0.smithi107.stdout:3242: finishing write tid 5 to smithi10729407-15 2020-06-12T02:37:49.372 INFO:tasks.rados.rados.0.smithi107.stdout:3242: finishing write tid 6 to smithi10729407-15 2020-06-12T02:37:49.372 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 549 (ObjNum 3241 snap 0 seq_num 3241) dirty exists 2020-06-12T02:37:49.372 INFO:tasks.rados.rados.0.smithi107.stdout:3242: left oid 15 (ObjNum 3241 snap 0 seq_num 3241) 2020-06-12T02:37:49.373 INFO:tasks.rados.rados.0.smithi107.stdout:3242: done (2 left) 2020-06-12T02:37:49.373 INFO:tasks.rados.rados.0.smithi107.stdout:3245: done (1 left) 2020-06-12T02:37:49.373 INFO:tasks.rados.rados.0.smithi107.stdout:3246: delete oid 5 current snap is 0 2020-06-12T02:37:49.373 INFO:tasks.rados.rados.0.smithi107.stdout:3244: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:49.373 INFO:tasks.rados.rados.0.smithi107.stdout:3246: done (1 left) 2020-06-12T02:37:49.374 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:49.374 INFO:tasks.rados.rados.0.smithi107.stdout:3247: seq_num 3246 ranges {1302528=770048,2072576=507904,2580480=294912} 2020-06-12T02:37:49.374 INFO:tasks.rados.rados.0.smithi107.stdout:3247: writing smithi10729407-2 from 1302528 to 2072576 tid 1 2020-06-12T02:37:49.374 INFO:tasks.rados.rados.0.smithi107.stdout:3247: writing smithi10729407-2 from 2072576 to 2580480 tid 2 2020-06-12T02:37:49.375 INFO:tasks.rados.rados.0.smithi107.stdout:3247: writing smithi10729407-2 from 2580480 to 2875392 tid 3 2020-06-12T02:37:49.375 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:37:49.375 INFO:tasks.rados.rados.0.smithi107.stdout:3248: seq_num 3247 ranges {0=786432,786432=442368,1228800=360448} 2020-06-12T02:37:49.375 INFO:tasks.rados.rados.0.smithi107.stdout:3248: writing smithi10729407-4 from 0 to 786432 tid 1 2020-06-12T02:37:49.375 INFO:tasks.rados.rados.0.smithi107.stdout:3248: writing smithi10729407-4 from 786432 to 1228800 tid 2 2020-06-12T02:37:49.376 INFO:tasks.rados.rados.0.smithi107.stdout:3248: writing smithi10729407-4 from 1228800 to 1589248 tid 3 2020-06-12T02:37:49.376 INFO:tasks.rados.rados.0.smithi107.stdout:3249: delete oid 19 current snap is 0 2020-06-12T02:37:49.376 INFO:tasks.rados.rados.0.smithi107.stdout:3244: finishing write tid 2 to smithi10729407-20 2020-06-12T02:37:49.376 INFO:tasks.rados.rados.0.smithi107.stdout:3244: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:49.376 INFO:tasks.rados.rados.0.smithi107.stdout:3244: finishing write tid 5 to smithi10729407-20 2020-06-12T02:37:49.376 INFO:tasks.rados.rados.0.smithi107.stdout:3244: finishing write tid 6 to smithi10729407-20 2020-06-12T02:37:49.377 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 226 (ObjNum 3243 snap 0 seq_num 3243) dirty exists 2020-06-12T02:37:49.377 INFO:tasks.rados.rados.0.smithi107.stdout:3244: left oid 20 (ObjNum 3243 snap 0 seq_num 3243) 2020-06-12T02:37:49.377 INFO:tasks.rados.rados.0.smithi107.stdout:3247: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:49.377 INFO:tasks.rados.rados.0.smithi107.stdout:3247: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:49.377 INFO:tasks.rados.rados.0.smithi107.stdout:3247: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:49.378 INFO:tasks.rados.rados.0.smithi107.stdout:3247: finishing write tid 5 to smithi10729407-2 2020-06-12T02:37:49.378 INFO:tasks.rados.rados.0.smithi107.stdout:3247: finishing write tid 6 to smithi10729407-2 2020-06-12T02:37:49.378 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 339 (ObjNum 3246 snap 0 seq_num 3246) dirty exists 2020-06-12T02:37:49.378 INFO:tasks.rados.rados.0.smithi107.stdout:3247: left oid 2 (ObjNum 3246 snap 0 seq_num 3246) 2020-06-12T02:37:49.378 INFO:tasks.rados.rados.0.smithi107.stdout:3244: done (3 left) 2020-06-12T02:37:49.378 INFO:tasks.rados.rados.0.smithi107.stdout:3247: done (2 left) 2020-06-12T02:37:49.379 INFO:tasks.rados.rados.0.smithi107.stdout:3249: done (1 left) 2020-06-12T02:37:49.379 INFO:tasks.rados.rados.0.smithi107.stdout:3250: delete oid 5 current snap is 0 2020-06-12T02:37:49.379 INFO:tasks.rados.rados.0.smithi107.stdout:3250: done (1 left) 2020-06-12T02:37:49.379 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:37:49.379 INFO:tasks.rados.rados.0.smithi107.stdout:3251: seq_num 3250 ranges {2875392=434176,3309568=499712,3809280=589824,4399104=114688} 2020-06-12T02:37:49.380 INFO:tasks.rados.rados.0.smithi107.stdout:3251: writing smithi10729407-2 from 2875392 to 3309568 tid 1 2020-06-12T02:37:49.380 INFO:tasks.rados.rados.0.smithi107.stdout:3251: writing smithi10729407-2 from 3309568 to 3809280 tid 2 2020-06-12T02:37:49.380 INFO:tasks.rados.rados.0.smithi107.stdout:3251: writing smithi10729407-2 from 3809280 to 4399104 tid 3 2020-06-12T02:37:49.380 INFO:tasks.rados.rados.0.smithi107.stdout:3251: writing smithi10729407-2 from 4399104 to 4513792 tid 4 2020-06-12T02:37:49.380 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:37:49.380 INFO:tasks.rados.rados.0.smithi107.stdout:3252: seq_num 3251 ranges {1351680=450560,1802240=786432,2588672=417792} 2020-06-12T02:37:49.381 INFO:tasks.rados.rados.0.smithi107.stdout:3252: writing smithi10729407-10 from 1351680 to 1802240 tid 1 2020-06-12T02:37:49.381 INFO:tasks.rados.rados.0.smithi107.stdout:3252: writing smithi10729407-10 from 1802240 to 2588672 tid 2 2020-06-12T02:37:49.381 INFO:tasks.rados.rados.0.smithi107.stdout:3252: writing smithi10729407-10 from 2588672 to 3006464 tid 3 2020-06-12T02:37:49.381 INFO:tasks.rados.rados.0.smithi107.stdout:3253: delete oid 20 current snap is 0 2020-06-12T02:37:49.381 INFO:tasks.rados.rados.0.smithi107.stdout:3248: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:49.381 INFO:tasks.rados.rados.0.smithi107.stdout:3248: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:49.382 INFO:tasks.rados.rados.0.smithi107.stdout:3248: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:49.382 INFO:tasks.rados.rados.0.smithi107.stdout:3248: finishing write tid 5 to smithi10729407-4 2020-06-12T02:37:49.382 INFO:tasks.rados.rados.0.smithi107.stdout:3248: finishing write tid 6 to smithi10729407-4 2020-06-12T02:37:49.382 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 620 (ObjNum 3247 snap 0 seq_num 3247) dirty exists 2020-06-12T02:37:49.382 INFO:tasks.rados.rados.0.smithi107.stdout:3248: left oid 4 (ObjNum 3247 snap 0 seq_num 3247) 2020-06-12T02:37:49.383 INFO:tasks.rados.rados.0.smithi107.stdout:3251: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:49.383 INFO:tasks.rados.rados.0.smithi107.stdout:3251: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:49.383 INFO:tasks.rados.rados.0.smithi107.stdout:3251: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:49.383 INFO:tasks.rados.rados.0.smithi107.stdout:3251: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:49.383 INFO:tasks.rados.rados.0.smithi107.stdout:3251: finishing write tid 6 to smithi10729407-2 2020-06-12T02:37:49.383 INFO:tasks.rados.rados.0.smithi107.stdout:3252: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:49.384 INFO:tasks.rados.rados.0.smithi107.stdout:3251: finishing write tid 7 to smithi10729407-2 2020-06-12T02:37:49.384 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 344 (ObjNum 3250 snap 0 seq_num 3250) dirty exists 2020-06-12T02:37:49.384 INFO:tasks.rados.rados.0.smithi107.stdout:3251: left oid 2 (ObjNum 3250 snap 0 seq_num 3250) 2020-06-12T02:37:49.384 INFO:tasks.rados.rados.0.smithi107.stdout:3248: done (3 left) 2020-06-12T02:37:49.385 INFO:tasks.rados.rados.0.smithi107.stdout:3251: done (2 left) 2020-06-12T02:37:49.385 INFO:tasks.rados.rados.0.smithi107.stdout:3253: done (1 left) 2020-06-12T02:37:49.385 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:37:49.385 INFO:tasks.rados.rados.0.smithi107.stdout:3254: seq_num 3253 ranges {0=483328,483328=548864,1032192=655360} 2020-06-12T02:37:49.385 INFO:tasks.rados.rados.0.smithi107.stdout:3254: writing smithi10729407-14 from 0 to 483328 tid 1 2020-06-12T02:37:49.386 INFO:tasks.rados.rados.0.smithi107.stdout:3254: writing smithi10729407-14 from 483328 to 1032192 tid 2 2020-06-12T02:37:49.386 INFO:tasks.rados.rados.0.smithi107.stdout:3254: writing smithi10729407-14 from 1032192 to 1687552 tid 3 2020-06-12T02:37:49.386 INFO:tasks.rados.rados.0.smithi107.stdout:3255: delete oid 6 current snap is 0 2020-06-12T02:37:49.386 INFO:tasks.rados.rados.0.smithi107.stdout:3252: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:49.386 INFO:tasks.rados.rados.0.smithi107.stdout:3252: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:49.386 INFO:tasks.rados.rados.0.smithi107.stdout:3252: finishing write tid 5 to smithi10729407-10 2020-06-12T02:37:49.387 INFO:tasks.rados.rados.0.smithi107.stdout:3252: finishing write tid 6 to smithi10729407-10 2020-06-12T02:37:49.387 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 327 (ObjNum 3251 snap 0 seq_num 3251) dirty exists 2020-06-12T02:37:49.387 INFO:tasks.rados.rados.0.smithi107.stdout:3252: left oid 10 (ObjNum 3251 snap 0 seq_num 3251) 2020-06-12T02:37:49.387 INFO:tasks.rados.rados.0.smithi107.stdout:3254: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:49.387 INFO:tasks.rados.rados.0.smithi107.stdout:3254: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:49.388 INFO:tasks.rados.rados.0.smithi107.stdout:3254: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:49.388 INFO:tasks.rados.rados.0.smithi107.stdout:3252: done (2 left) 2020-06-12T02:37:49.388 INFO:tasks.rados.rados.0.smithi107.stdout:3255: done (1 left) 2020-06-12T02:37:49.388 INFO:tasks.rados.rados.0.smithi107.stdout:3256: delete oid 12 current snap is 0 2020-06-12T02:37:49.388 INFO:tasks.rados.rados.0.smithi107.stdout:3254: finishing write tid 5 to smithi10729407-14 2020-06-12T02:37:49.388 INFO:tasks.rados.rados.0.smithi107.stdout:3254: finishing write tid 6 to smithi10729407-14 2020-06-12T02:37:49.389 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 552 (ObjNum 3253 snap 0 seq_num 3253) dirty exists 2020-06-12T02:37:49.389 INFO:tasks.rados.rados.0.smithi107.stdout:3254: left oid 14 (ObjNum 3253 snap 0 seq_num 3253) 2020-06-12T02:37:49.389 INFO:tasks.rados.rados.0.smithi107.stdout:3254: done (1 left) 2020-06-12T02:37:49.389 INFO:tasks.rados.rados.0.smithi107.stdout:3256: done (0 left) 2020-06-12T02:37:49.389 INFO:tasks.rados.rados.0.smithi107.stdout:3257: delete oid 12 current snap is 0 2020-06-12T02:37:49.389 INFO:tasks.rados.rados.0.smithi107.stdout:3257: done (0 left) 2020-06-12T02:37:49.390 INFO:tasks.rados.rados.0.smithi107.stdout:3258: delete oid 3 current snap is 0 2020-06-12T02:37:49.390 INFO:tasks.rados.rados.0.smithi107.stdout:3258: done (0 left) 2020-06-12T02:37:49.390 INFO:tasks.rados.rados.0.smithi107.stdout:3259: delete oid 11 current snap is 0 2020-06-12T02:37:49.390 INFO:tasks.rados.rados.0.smithi107.stdout:3259: done (0 left) 2020-06-12T02:37:49.390 INFO:tasks.rados.rados.0.smithi107.stdout:3260: delete oid 5 current snap is 0 2020-06-12T02:37:49.391 INFO:tasks.rados.rados.0.smithi107.stdout:3260: done (0 left) 2020-06-12T02:37:49.391 INFO:tasks.rados.rados.0.smithi107.stdout:3261: delete oid 1 current snap is 0 2020-06-12T02:37:49.391 INFO:tasks.rados.rados.0.smithi107.stdout:3261: done (0 left) 2020-06-12T02:37:49.391 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:37:49.391 INFO:tasks.rados.rados.0.smithi107.stdout:3262: seq_num 3261 ranges {0=622592,622592=475136,1097728=729088} 2020-06-12T02:37:49.391 INFO:tasks.rados.rados.0.smithi107.stdout:3262: writing smithi10729407-6 from 0 to 622592 tid 1 2020-06-12T02:37:49.392 INFO:tasks.rados.rados.0.smithi107.stdout:3262: writing smithi10729407-6 from 622592 to 1097728 tid 2 2020-06-12T02:37:49.392 INFO:tasks.rados.rados.0.smithi107.stdout:3262: writing smithi10729407-6 from 1097728 to 1826816 tid 3 2020-06-12T02:37:49.392 INFO:tasks.rados.rados.0.smithi107.stdout:3263: delete oid 17 current snap is 0 2020-06-12T02:37:49.392 INFO:tasks.rados.rados.0.smithi107.stdout:3262: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:49.392 INFO:tasks.rados.rados.0.smithi107.stdout:3263: done (1 left) 2020-06-12T02:37:49.393 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:37:49.393 INFO:tasks.rados.rados.0.smithi107.stdout:3264: seq_num 3263 ranges {1015808=655360,1671168=696320,2367488=507904} 2020-06-12T02:37:49.393 INFO:tasks.rados.rados.0.smithi107.stdout:3264: writing smithi10729407-16 from 1015808 to 1671168 tid 1 2020-06-12T02:37:49.393 INFO:tasks.rados.rados.0.smithi107.stdout:3264: writing smithi10729407-16 from 1671168 to 2367488 tid 2 2020-06-12T02:37:49.393 INFO:tasks.rados.rados.0.smithi107.stdout:3264: writing smithi10729407-16 from 2367488 to 2875392 tid 3 2020-06-12T02:37:49.393 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:37:49.394 INFO:tasks.rados.rados.0.smithi107.stdout:3265: seq_num 3264 ranges {1490944=671744,2162688=581632,2744320=540672,3284992=81920} 2020-06-12T02:37:49.394 INFO:tasks.rados.rados.0.smithi107.stdout:3265: writing smithi10729407-15 from 1490944 to 2162688 tid 1 2020-06-12T02:37:49.394 INFO:tasks.rados.rados.0.smithi107.stdout:3265: writing smithi10729407-15 from 2162688 to 2744320 tid 2 2020-06-12T02:37:49.394 INFO:tasks.rados.rados.0.smithi107.stdout:3265: writing smithi10729407-15 from 2744320 to 3284992 tid 3 2020-06-12T02:37:49.395 INFO:tasks.rados.rados.0.smithi107.stdout:3265: writing smithi10729407-15 from 3284992 to 3366912 tid 4 2020-06-12T02:37:49.395 INFO:tasks.rados.rados.0.smithi107.stdout:3266: delete oid 20 current snap is 0 2020-06-12T02:37:49.395 INFO:tasks.rados.rados.0.smithi107.stdout:3262: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:49.395 INFO:tasks.rados.rados.0.smithi107.stdout:3262: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:49.395 INFO:tasks.rados.rados.0.smithi107.stdout:3262: finishing write tid 5 to smithi10729407-6 2020-06-12T02:37:49.395 INFO:tasks.rados.rados.0.smithi107.stdout:3262: finishing write tid 6 to smithi10729407-6 2020-06-12T02:37:49.396 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 621 (ObjNum 3261 snap 0 seq_num 3261) dirty exists 2020-06-12T02:37:49.396 INFO:tasks.rados.rados.0.smithi107.stdout:3262: left oid 6 (ObjNum 3261 snap 0 seq_num 3261) 2020-06-12T02:37:49.396 INFO:tasks.rados.rados.0.smithi107.stdout:3264: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:49.396 INFO:tasks.rados.rados.0.smithi107.stdout:3264: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:49.396 INFO:tasks.rados.rados.0.smithi107.stdout:3264: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:49.397 INFO:tasks.rados.rados.0.smithi107.stdout:3264: finishing write tid 5 to smithi10729407-16 2020-06-12T02:37:49.397 INFO:tasks.rados.rados.0.smithi107.stdout:3264: finishing write tid 6 to smithi10729407-16 2020-06-12T02:37:49.397 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 581 (ObjNum 3263 snap 0 seq_num 3263) dirty exists 2020-06-12T02:37:49.397 INFO:tasks.rados.rados.0.smithi107.stdout:3264: left oid 16 (ObjNum 3263 snap 0 seq_num 3263) 2020-06-12T02:37:49.397 INFO:tasks.rados.rados.0.smithi107.stdout:3265: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:49.398 INFO:tasks.rados.rados.0.smithi107.stdout:3265: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:49.398 INFO:tasks.rados.rados.0.smithi107.stdout:3265: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:49.398 INFO:tasks.rados.rados.0.smithi107.stdout:3262: done (3 left) 2020-06-12T02:37:49.398 INFO:tasks.rados.rados.0.smithi107.stdout:3264: done (2 left) 2020-06-12T02:37:49.398 INFO:tasks.rados.rados.0.smithi107.stdout:3266: done (1 left) 2020-06-12T02:37:49.398 INFO:tasks.rados.rados.0.smithi107.stdout:3267: delete oid 1 current snap is 0 2020-06-12T02:37:49.399 INFO:tasks.rados.rados.0.smithi107.stdout:3265: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:49.399 INFO:tasks.rados.rados.0.smithi107.stdout:3265: finishing write tid 6 to smithi10729407-15 2020-06-12T02:37:49.399 INFO:tasks.rados.rados.0.smithi107.stdout:3267: done (1 left) 2020-06-12T02:37:49.399 INFO:tasks.rados.rados.0.smithi107.stdout:3268: delete oid 3 current snap is 0 2020-06-12T02:37:49.399 INFO:tasks.rados.rados.0.smithi107.stdout:3265: finishing write tid 7 to smithi10729407-15 2020-06-12T02:37:49.400 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 554 (ObjNum 3264 snap 0 seq_num 3264) dirty exists 2020-06-12T02:37:49.400 INFO:tasks.rados.rados.0.smithi107.stdout:3265: left oid 15 (ObjNum 3264 snap 0 seq_num 3264) 2020-06-12T02:37:49.400 INFO:tasks.rados.rados.0.smithi107.stdout:3265: done (1 left) 2020-06-12T02:37:49.400 INFO:tasks.rados.rados.0.smithi107.stdout:3268: done (0 left) 2020-06-12T02:37:49.400 INFO:tasks.rados.rados.0.smithi107.stdout:3269: delete oid 17 current snap is 0 2020-06-12T02:37:49.400 INFO:tasks.rados.rados.0.smithi107.stdout:3269: done (0 left) 2020-06-12T02:37:49.401 INFO:tasks.rados.rados.0.smithi107.stdout:3270: delete oid 20 current snap is 0 2020-06-12T02:37:49.401 INFO:tasks.rados.rados.0.smithi107.stdout:3270: done (0 left) 2020-06-12T02:37:49.401 INFO:tasks.rados.rados.0.smithi107.stdout:3271: delete oid 10 current snap is 0 2020-06-12T02:37:49.401 INFO:tasks.rados.rados.0.smithi107.stdout:3271: done (0 left) 2020-06-12T02:37:49.401 INFO:tasks.rados.rados.0.smithi107.stdout:3272: delete oid 20 current snap is 0 2020-06-12T02:37:49.402 INFO:tasks.rados.rados.0.smithi107.stdout:3272: done (0 left) 2020-06-12T02:37:49.402 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:37:49.402 INFO:tasks.rados.rados.0.smithi107.stdout:3273: seq_num 3272 ranges {1826816=802816,2629632=499712,3129344=499712,3629056=204800} 2020-06-12T02:37:49.402 INFO:tasks.rados.rados.0.smithi107.stdout:3273: writing smithi10729407-6 from 1826816 to 2629632 tid 1 2020-06-12T02:37:49.402 INFO:tasks.rados.rados.0.smithi107.stdout:3273: writing smithi10729407-6 from 2629632 to 3129344 tid 2 2020-06-12T02:37:49.402 INFO:tasks.rados.rados.0.smithi107.stdout:3273: writing smithi10729407-6 from 3129344 to 3629056 tid 3 2020-06-12T02:37:49.403 INFO:tasks.rados.rados.0.smithi107.stdout:3273: writing smithi10729407-6 from 3629056 to 3833856 tid 4 2020-06-12T02:37:49.403 INFO:tasks.rados.rados.0.smithi107.stdout:3274: delete oid 18 current snap is 0 2020-06-12T02:37:49.403 INFO:tasks.rados.rados.0.smithi107.stdout:3273: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:49.403 INFO:tasks.rados.rados.0.smithi107.stdout:3274: done (1 left) 2020-06-12T02:37:49.403 INFO:tasks.rados.rados.0.smithi107.stdout:3275: delete oid 16 current snap is 0 2020-06-12T02:37:49.403 INFO:tasks.rados.rados.0.smithi107.stdout:3275: done (1 left) 2020-06-12T02:37:49.404 INFO:tasks.rados.rados.0.smithi107.stdout:3276: delete oid 7 current snap is 0 2020-06-12T02:37:49.404 INFO:tasks.rados.rados.0.smithi107.stdout:3273: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:49.404 INFO:tasks.rados.rados.0.smithi107.stdout:3273: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:49.404 INFO:tasks.rados.rados.0.smithi107.stdout:3273: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:49.404 INFO:tasks.rados.rados.0.smithi107.stdout:3273: finishing write tid 6 to smithi10729407-6 2020-06-12T02:37:49.405 INFO:tasks.rados.rados.0.smithi107.stdout:3276: done (1 left) 2020-06-12T02:37:49.405 INFO:tasks.rados.rados.0.smithi107.stdout:3277: delete oid 16 current snap is 0 2020-06-12T02:37:49.405 INFO:tasks.rados.rados.0.smithi107.stdout:3273: finishing write tid 7 to smithi10729407-6 2020-06-12T02:37:49.405 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 627 (ObjNum 3272 snap 0 seq_num 3272) dirty exists 2020-06-12T02:37:49.406 INFO:tasks.rados.rados.0.smithi107.stdout:3273: left oid 6 (ObjNum 3272 snap 0 seq_num 3272) 2020-06-12T02:37:49.406 INFO:tasks.rados.rados.0.smithi107.stdout:3273: done (1 left) 2020-06-12T02:37:49.406 INFO:tasks.rados.rados.0.smithi107.stdout:3277: done (0 left) 2020-06-12T02:37:49.406 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:37:49.406 INFO:tasks.rados.rados.0.smithi107.stdout:3278: seq_num 3277 ranges {0=491520,491520=778240,1269760=688128,1957888=139264} 2020-06-12T02:37:49.407 INFO:tasks.rados.rados.0.smithi107.stdout:3278: writing smithi10729407-13 from 0 to 491520 tid 1 2020-06-12T02:37:49.407 INFO:tasks.rados.rados.0.smithi107.stdout:3278: writing smithi10729407-13 from 491520 to 1269760 tid 2 2020-06-12T02:37:49.407 INFO:tasks.rados.rados.0.smithi107.stdout:3278: writing smithi10729407-13 from 1269760 to 1957888 tid 3 2020-06-12T02:37:49.407 INFO:tasks.rados.rados.0.smithi107.stdout:3278: writing smithi10729407-13 from 1957888 to 2097152 tid 4 2020-06-12T02:37:49.407 INFO:tasks.rados.rados.0.smithi107.stdout:3279: delete oid 6 current snap is 0 2020-06-12T02:37:49.407 INFO:tasks.rados.rados.0.smithi107.stdout:3278: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:49.408 INFO:tasks.rados.rados.0.smithi107.stdout:3279: done (1 left) 2020-06-12T02:37:49.408 INFO:tasks.rados.rados.0.smithi107.stdout:3280: delete oid 4 current snap is 0 2020-06-12T02:37:49.408 INFO:tasks.rados.rados.0.smithi107.stdout:3280: done (1 left) 2020-06-12T02:37:49.408 INFO:tasks.rados.rados.0.smithi107.stdout:3281: delete oid 3 current snap is 0 2020-06-12T02:37:49.408 INFO:tasks.rados.rados.0.smithi107.stdout:3278: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:49.409 INFO:tasks.rados.rados.0.smithi107.stdout:3278: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:49.409 INFO:tasks.rados.rados.0.smithi107.stdout:3278: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:49.409 INFO:tasks.rados.rados.0.smithi107.stdout:3278: finishing write tid 6 to smithi10729407-13 2020-06-12T02:37:49.409 INFO:tasks.rados.rados.0.smithi107.stdout:3278: finishing write tid 7 to smithi10729407-13 2020-06-12T02:37:49.409 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 618 (ObjNum 3277 snap 0 seq_num 3277) dirty exists 2020-06-12T02:37:49.409 INFO:tasks.rados.rados.0.smithi107.stdout:3278: left oid 13 (ObjNum 3277 snap 0 seq_num 3277) 2020-06-12T02:37:49.410 INFO:tasks.rados.rados.0.smithi107.stdout:3278: done (1 left) 2020-06-12T02:37:49.410 INFO:tasks.rados.rados.0.smithi107.stdout:3281: done (0 left) 2020-06-12T02:37:49.410 INFO:tasks.rados.rados.0.smithi107.stdout:3282: delete oid 13 current snap is 0 2020-06-12T02:37:49.410 INFO:tasks.rados.rados.0.smithi107.stdout:3282: done (0 left) 2020-06-12T02:37:49.410 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:37:49.410 INFO:tasks.rados.rados.0.smithi107.stdout:3283: seq_num 3282 ranges {0=573440,573440=598016,1171456=425984,1597440=417792,2015232=163840} 2020-06-12T02:37:49.411 INFO:tasks.rados.rados.0.smithi107.stdout:3283: writing smithi10729407-12 from 0 to 573440 tid 1 2020-06-12T02:37:49.411 INFO:tasks.rados.rados.0.smithi107.stdout:3283: writing smithi10729407-12 from 573440 to 1171456 tid 2 2020-06-12T02:37:49.411 INFO:tasks.rados.rados.0.smithi107.stdout:3283: writing smithi10729407-12 from 1171456 to 1597440 tid 3 2020-06-12T02:37:49.411 INFO:tasks.rados.rados.0.smithi107.stdout:3283: writing smithi10729407-12 from 1597440 to 2015232 tid 4 2020-06-12T02:37:49.411 INFO:tasks.rados.rados.0.smithi107.stdout:3283: writing smithi10729407-12 from 2015232 to 2179072 tid 5 2020-06-12T02:37:49.412 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:37:49.412 INFO:tasks.rados.rados.0.smithi107.stdout:3284: seq_num 3283 ranges {0=589824,589824=483328,1073152=638976,1712128=483328} 2020-06-12T02:37:49.412 INFO:tasks.rados.rados.0.smithi107.stdout:3284: writing smithi10729407-13 from 0 to 589824 tid 1 2020-06-12T02:37:49.412 INFO:tasks.rados.rados.0.smithi107.stdout:3284: writing smithi10729407-13 from 589824 to 1073152 tid 2 2020-06-12T02:37:49.412 INFO:tasks.rados.rados.0.smithi107.stdout:3284: writing smithi10729407-13 from 1073152 to 1712128 tid 3 2020-06-12T02:37:49.412 INFO:tasks.rados.rados.0.smithi107.stdout:3284: writing smithi10729407-13 from 1712128 to 2195456 tid 4 2020-06-12T02:37:49.413 INFO:tasks.rados.rados.0.smithi107.stdout:3285: delete oid 19 current snap is 0 2020-06-12T02:37:49.413 INFO:tasks.rados.rados.0.smithi107.stdout:3283: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:49.413 INFO:tasks.rados.rados.0.smithi107.stdout:3283: finishing write tid 2 to smithi10729407-12 2020-06-12T02:37:49.413 INFO:tasks.rados.rados.0.smithi107.stdout:3283: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:49.413 INFO:tasks.rados.rados.0.smithi107.stdout:3283: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:49.414 INFO:tasks.rados.rados.0.smithi107.stdout:3283: finishing write tid 5 to smithi10729407-12 2020-06-12T02:37:49.414 INFO:tasks.rados.rados.0.smithi107.stdout:3283: finishing write tid 7 to smithi10729407-12 2020-06-12T02:37:49.414 INFO:tasks.rados.rados.0.smithi107.stdout:3283: finishing write tid 8 to smithi10729407-12 2020-06-12T02:37:49.414 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 536 (ObjNum 3282 snap 0 seq_num 3282) dirty exists 2020-06-12T02:37:49.414 INFO:tasks.rados.rados.0.smithi107.stdout:3283: left oid 12 (ObjNum 3282 snap 0 seq_num 3282) 2020-06-12T02:37:49.414 INFO:tasks.rados.rados.0.smithi107.stdout:3284: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:49.415 INFO:tasks.rados.rados.0.smithi107.stdout:3283: done (2 left) 2020-06-12T02:37:49.415 INFO:tasks.rados.rados.0.smithi107.stdout:3285: done (1 left) 2020-06-12T02:37:49.415 INFO:tasks.rados.rados.0.smithi107.stdout:3286: delete oid 6 current snap is 0 2020-06-12T02:37:49.415 INFO:tasks.rados.rados.0.smithi107.stdout:3286: done (1 left) 2020-06-12T02:37:49.416 INFO:tasks.rados.rados.0.smithi107.stdout:3287: delete oid 7 current snap is 0 2020-06-12T02:37:49.416 INFO:tasks.rados.rados.0.smithi107.stdout:3284: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:49.416 INFO:tasks.rados.rados.0.smithi107.stdout:3284: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:49.416 INFO:tasks.rados.rados.0.smithi107.stdout:3284: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:49.416 INFO:tasks.rados.rados.0.smithi107.stdout:3284: finishing write tid 6 to smithi10729407-13 2020-06-12T02:37:49.417 INFO:tasks.rados.rados.0.smithi107.stdout:3287: done (1 left) 2020-06-12T02:37:49.417 INFO:tasks.rados.rados.0.smithi107.stdout:3288: delete oid 14 current snap is 0 2020-06-12T02:37:49.417 INFO:tasks.rados.rados.0.smithi107.stdout:3284: finishing write tid 7 to smithi10729407-13 2020-06-12T02:37:49.417 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 624 (ObjNum 3283 snap 0 seq_num 3283) dirty exists 2020-06-12T02:37:49.417 INFO:tasks.rados.rados.0.smithi107.stdout:3284: left oid 13 (ObjNum 3283 snap 0 seq_num 3283) 2020-06-12T02:37:49.417 INFO:tasks.rados.rados.0.smithi107.stdout:3284: done (1 left) 2020-06-12T02:37:49.418 INFO:tasks.rados.rados.0.smithi107.stdout:3288: done (0 left) 2020-06-12T02:37:49.418 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:37:49.418 INFO:tasks.rados.rados.0.smithi107.stdout:3289: seq_num 3288 ranges {0=671744,671744=753664,1425408=425984,1851392=425984} 2020-06-12T02:37:49.418 INFO:tasks.rados.rados.0.smithi107.stdout:3289: writing smithi10729407-16 from 0 to 671744 tid 1 2020-06-12T02:37:49.418 INFO:tasks.rados.rados.0.smithi107.stdout:3289: writing smithi10729407-16 from 671744 to 1425408 tid 2 2020-06-12T02:37:49.419 INFO:tasks.rados.rados.0.smithi107.stdout:3289: writing smithi10729407-16 from 1425408 to 1851392 tid 3 2020-06-12T02:37:49.419 INFO:tasks.rados.rados.0.smithi107.stdout:3289: writing smithi10729407-16 from 1851392 to 2277376 tid 4 2020-06-12T02:37:49.419 INFO:tasks.rados.rados.0.smithi107.stdout:3289: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:49.419 INFO:tasks.rados.rados.0.smithi107.stdout:3290: delete oid 20 current snap is 0 2020-06-12T02:37:49.419 INFO:tasks.rados.rados.0.smithi107.stdout:3290: done (1 left) 2020-06-12T02:37:49.419 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:37:49.420 INFO:tasks.rados.rados.0.smithi107.stdout:3291: seq_num 3290 ranges {2179072=704512,2883584=516096,3399680=794624,4194304=294912} 2020-06-12T02:37:49.420 INFO:tasks.rados.rados.0.smithi107.stdout:3291: writing smithi10729407-12 from 2179072 to 2883584 tid 1 2020-06-12T02:37:49.420 INFO:tasks.rados.rados.0.smithi107.stdout:3291: writing smithi10729407-12 from 2883584 to 3399680 tid 2 2020-06-12T02:37:49.420 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:37:49.340+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:37:49.421 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-06-12T02:37:49.425 INFO:tasks.rados.rados.0.smithi107.stdout:3291: writing smithi10729407-12 from 3399680 to 4194304 tid 3 2020-06-12T02:37:49.425 INFO:tasks.rados.rados.0.smithi107.stdout:3291: writing smithi10729407-12 from 4194304 to 4489216 tid 4 2020-06-12T02:37:49.426 INFO:tasks.rados.rados.0.smithi107.stdout:3292: delete oid 10 current snap is 0 2020-06-12T02:37:49.426 INFO:tasks.rados.rados.0.smithi107.stdout:3289: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:49.426 INFO:tasks.rados.rados.0.smithi107.stdout:3289: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:49.426 INFO:tasks.rados.rados.0.smithi107.stdout:3289: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:49.426 INFO:tasks.rados.rados.0.smithi107.stdout:3289: finishing write tid 6 to smithi10729407-16 2020-06-12T02:37:49.427 INFO:tasks.rados.rados.0.smithi107.stdout:3289: finishing write tid 7 to smithi10729407-16 2020-06-12T02:37:49.427 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 590 (ObjNum 3288 snap 0 seq_num 3288) dirty exists 2020-06-12T02:37:49.427 INFO:tasks.rados.rados.0.smithi107.stdout:3289: left oid 16 (ObjNum 3288 snap 0 seq_num 3288) 2020-06-12T02:37:49.427 INFO:tasks.rados.rados.0.smithi107.stdout:3291: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:49.427 INFO:tasks.rados.rados.0.smithi107.stdout:3291: finishing write tid 2 to smithi10729407-12 2020-06-12T02:37:49.427 INFO:tasks.rados.rados.0.smithi107.stdout:3289: done (2 left) 2020-06-12T02:37:49.428 INFO:tasks.rados.rados.0.smithi107.stdout:3292: done (1 left) 2020-06-12T02:37:49.428 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:37:49.428 INFO:tasks.rados.rados.0.smithi107.stdout:3293: seq_num 3292 ranges {0=737280,737280=737280,1474560=409600,1884160=458752} 2020-06-12T02:37:49.428 INFO:tasks.rados.rados.0.smithi107.stdout:3293: writing smithi10729407-9 from 0 to 737280 tid 1 2020-06-12T02:37:49.428 INFO:tasks.rados.rados.0.smithi107.stdout:3293: writing smithi10729407-9 from 737280 to 1474560 tid 2 2020-06-12T02:37:49.429 INFO:tasks.rados.rados.0.smithi107.stdout:3293: writing smithi10729407-9 from 1474560 to 1884160 tid 3 2020-06-12T02:37:49.429 INFO:tasks.rados.rados.0.smithi107.stdout:3293: writing smithi10729407-9 from 1884160 to 2342912 tid 4 2020-06-12T02:37:49.429 INFO:tasks.rados.rados.0.smithi107.stdout:3294: delete oid 2 current snap is 0 2020-06-12T02:37:49.429 INFO:tasks.rados.rados.0.smithi107.stdout:3291: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:49.429 INFO:tasks.rados.rados.0.smithi107.stdout:3291: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:49.429 INFO:tasks.rados.rados.0.smithi107.stdout:3291: finishing write tid 6 to smithi10729407-12 2020-06-12T02:37:49.430 INFO:tasks.rados.rados.0.smithi107.stdout:3291: finishing write tid 7 to smithi10729407-12 2020-06-12T02:37:49.430 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 542 (ObjNum 3290 snap 0 seq_num 3290) dirty exists 2020-06-12T02:37:49.430 INFO:tasks.rados.rados.0.smithi107.stdout:3291: left oid 12 (ObjNum 3290 snap 0 seq_num 3290) 2020-06-12T02:37:49.430 INFO:tasks.rados.rados.0.smithi107.stdout:3293: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:49.430 INFO:tasks.rados.rados.0.smithi107.stdout:3291: done (2 left) 2020-06-12T02:37:49.431 INFO:tasks.rados.rados.0.smithi107.stdout:3294: done (1 left) 2020-06-12T02:37:49.431 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:37:49.431 INFO:tasks.rados.rados.0.smithi107.stdout:3295: seq_num 3294 ranges {0=778240,778240=507904,1286144=770048,2056192=327680} 2020-06-12T02:37:49.431 INFO:tasks.rados.rados.0.smithi107.stdout:3295: writing smithi10729407-11 from 0 to 778240 tid 1 2020-06-12T02:37:49.432 INFO:tasks.rados.rados.0.smithi107.stdout:3295: writing smithi10729407-11 from 778240 to 1286144 tid 2 2020-06-12T02:37:49.432 INFO:tasks.rados.rados.0.smithi107.stdout:3295: writing smithi10729407-11 from 1286144 to 2056192 tid 3 2020-06-12T02:37:49.432 INFO:tasks.rados.rados.0.smithi107.stdout:3295: writing smithi10729407-11 from 2056192 to 2383872 tid 4 2020-06-12T02:37:49.432 INFO:tasks.rados.rados.0.smithi107.stdout:3296: delete oid 14 current snap is 0 2020-06-12T02:37:49.432 INFO:tasks.rados.rados.0.smithi107.stdout:3293: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:49.433 INFO:tasks.rados.rados.0.smithi107.stdout:3293: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:49.433 INFO:tasks.rados.rados.0.smithi107.stdout:3293: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:49.433 INFO:tasks.rados.rados.0.smithi107.stdout:3293: finishing write tid 6 to smithi10729407-9 2020-06-12T02:37:49.433 INFO:tasks.rados.rados.0.smithi107.stdout:3293: finishing write tid 7 to smithi10729407-9 2020-06-12T02:37:49.433 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 591 (ObjNum 3292 snap 0 seq_num 3292) dirty exists 2020-06-12T02:37:49.433 INFO:tasks.rados.rados.0.smithi107.stdout:3293: left oid 9 (ObjNum 3292 snap 0 seq_num 3292) 2020-06-12T02:37:49.434 INFO:tasks.rados.rados.0.smithi107.stdout:3295: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:49.434 INFO:tasks.rados.rados.0.smithi107.stdout:3295: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:49.434 INFO:tasks.rados.rados.0.smithi107.stdout:3295: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:49.434 INFO:tasks.rados.rados.0.smithi107.stdout:3295: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:49.434 INFO:tasks.rados.rados.0.smithi107.stdout:3295: finishing write tid 6 to smithi10729407-11 2020-06-12T02:37:49.434 INFO:tasks.rados.rados.0.smithi107.stdout:3293: done (2 left) 2020-06-12T02:37:49.435 INFO:tasks.rados.rados.0.smithi107.stdout:3296: done (1 left) 2020-06-12T02:37:49.435 INFO:tasks.rados.rados.0.smithi107.stdout:3297: delete oid 3 current snap is 0 2020-06-12T02:37:49.435 INFO:tasks.rados.rados.0.smithi107.stdout:3295: finishing write tid 7 to smithi10729407-11 2020-06-12T02:37:49.435 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 559 (ObjNum 3294 snap 0 seq_num 3294) dirty exists 2020-06-12T02:37:49.436 INFO:tasks.rados.rados.0.smithi107.stdout:3295: left oid 11 (ObjNum 3294 snap 0 seq_num 3294) 2020-06-12T02:37:49.436 INFO:tasks.rados.rados.0.smithi107.stdout:3295: done (1 left) 2020-06-12T02:37:49.436 INFO:tasks.rados.rados.0.smithi107.stdout:3297: done (0 left) 2020-06-12T02:37:49.436 INFO:tasks.rados.rados.0.smithi107.stdout:3298: delete oid 12 current snap is 0 2020-06-12T02:37:49.436 INFO:tasks.rados.rados.0.smithi107.stdout:3298: done (0 left) 2020-06-12T02:37:49.437 INFO:tasks.rados.rados.0.smithi107.stdout:3299: delete oid 19 current snap is 0 2020-06-12T02:37:49.437 INFO:tasks.rados.rados.0.smithi107.stdout:3299: done (0 left) 2020-06-12T02:37:49.437 INFO:tasks.rados.rados.0.smithi107.stdout:3300: delete oid 1 current snap is 0 2020-06-12T02:37:49.437 INFO:tasks.rados.rados.0.smithi107.stdout:3300: done (0 left) 2020-06-12T02:37:49.437 INFO:tasks.rados.rados.0.smithi107.stdout:3301: delete oid 14 current snap is 0 2020-06-12T02:37:49.437 INFO:tasks.rados.rados.0.smithi107.stdout:3301: done (0 left) 2020-06-12T02:37:49.438 INFO:tasks.rados.rados.0.smithi107.stdout:3302: delete oid 2 current snap is 0 2020-06-12T02:37:49.440 INFO:tasks.rados.rados.0.smithi107.stdout:3302: done (0 left) 2020-06-12T02:37:49.440 INFO:tasks.rados.rados.0.smithi107.stdout:3303: delete oid 16 current snap is 0 2020-06-12T02:37:49.440 INFO:tasks.rados.rados.0.smithi107.stdout:3303: done (0 left) 2020-06-12T02:37:49.440 INFO:tasks.rados.rados.0.smithi107.stdout:3304: delete oid 12 current snap is 0 2020-06-12T02:37:49.440 INFO:tasks.rados.rados.0.smithi107.stdout:3304: done (0 left) 2020-06-12T02:37:49.441 INFO:tasks.rados.rados.0.smithi107.stdout:3305: delete oid 14 current snap is 0 2020-06-12T02:37:49.441 INFO:tasks.rados.rados.0.smithi107.stdout:3305: done (0 left) 2020-06-12T02:37:49.441 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:37:49.441 INFO:tasks.rados.rados.0.smithi107.stdout:3306: seq_num 3305 ranges {0=155648} 2020-06-12T02:37:49.441 INFO:tasks.rados.rados.0.smithi107.stdout:3306: writing smithi10729407-10 from 0 to 155648 tid 1 2020-06-12T02:37:49.442 INFO:tasks.rados.rados.0.smithi107.stdout:3307: delete oid 7 current snap is 0 2020-06-12T02:37:49.442 INFO:tasks.rados.rados.0.smithi107.stdout:3307: done (1 left) 2020-06-12T02:37:49.442 INFO:tasks.rados.rados.0.smithi107.stdout:3308: delete oid 19 current snap is 0 2020-06-12T02:37:49.442 INFO:tasks.rados.rados.0.smithi107.stdout:3306: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:49.444 INFO:tasks.rados.rados.0.smithi107.stdout:3306: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:49.445 INFO:tasks.rados.rados.0.smithi107.stdout:3308: done (1 left) 2020-06-12T02:37:49.445 INFO:tasks.rados.rados.0.smithi107.stdout:3309: delete oid 14 current snap is 0 2020-06-12T02:37:49.445 INFO:tasks.rados.rados.0.smithi107.stdout:3306: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:49.445 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 331 (ObjNum 3305 snap 0 seq_num 3305) dirty exists 2020-06-12T02:37:49.445 INFO:tasks.rados.rados.0.smithi107.stdout:3306: left oid 10 (ObjNum 3305 snap 0 seq_num 3305) 2020-06-12T02:37:49.446 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:49.440+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:49.447 INFO:tasks.rados.rados.0.smithi107.stdout:3306: done (1 left) 2020-06-12T02:37:49.447 INFO:tasks.rados.rados.0.smithi107.stdout:3309: done (0 left) 2020-06-12T02:37:49.447 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:37:49.447 INFO:tasks.rados.rados.0.smithi107.stdout:3310: seq_num 3309 ranges {0=221184} 2020-06-12T02:37:49.448 INFO:tasks.rados.rados.0.smithi107.stdout:3310: writing smithi10729407-7 from 0 to 221184 tid 1 2020-06-12T02:37:49.448 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:37:49.449 INFO:tasks.rados.rados.0.smithi107.stdout:3311: seq_num 3310 ranges {0=237568} 2020-06-12T02:37:49.449 INFO:tasks.rados.rados.0.smithi107.stdout:3311: writing smithi10729407-18 from 0 to 237568 tid 1 2020-06-12T02:37:49.450 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:37:49.450 INFO:tasks.rados.rados.0.smithi107.stdout:3312: seq_num 3311 ranges {0=253952} 2020-06-12T02:37:49.452 INFO:tasks.rados.rados.0.smithi107.stdout:3312: writing smithi10729407-8 from 0 to 253952 tid 1 2020-06-12T02:37:49.452 INFO:tasks.rados.rados.0.smithi107.stdout:3313: delete oid 12 current snap is 0 2020-06-12T02:37:49.452 INFO:tasks.rados.rados.0.smithi107.stdout:3310: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:49.453 INFO:tasks.rados.rados.0.smithi107.stdout:3310: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:49.455 INFO:tasks.rados.rados.0.smithi107.stdout:3310: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:49.455 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 628 (ObjNum 3309 snap 0 seq_num 3309) dirty exists 2020-06-12T02:37:49.456 INFO:tasks.rados.rados.0.smithi107.stdout:3310: left oid 7 (ObjNum 3309 snap 0 seq_num 3309) 2020-06-12T02:37:49.456 INFO:tasks.rados.rados.0.smithi107.stdout:3310: done (3 left) 2020-06-12T02:37:49.456 INFO:tasks.rados.rados.0.smithi107.stdout:3313: done (2 left) 2020-06-12T02:37:49.456 INFO:tasks.rados.rados.0.smithi107.stdout:3314: delete oid 1 current snap is 0 2020-06-12T02:37:49.456 INFO:tasks.rados.rados.0.smithi107.stdout:3311: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:49.457 INFO:tasks.rados.rados.0.smithi107.stdout:3312: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:49.458 INFO:tasks.rados.rados.0.smithi107.stdout:3311: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:49.458 INFO:tasks.rados.rados.0.smithi107.stdout:3314: done (2 left) 2020-06-12T02:37:49.458 INFO:tasks.rados.rados.0.smithi107.stdout:3315: delete oid 2 current snap is 0 2020-06-12T02:37:49.459 INFO:tasks.rados.rados.0.smithi107.stdout:3311: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:49.459 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 593 (ObjNum 3310 snap 0 seq_num 3310) dirty exists 2020-06-12T02:37:49.460 INFO:tasks.rados.rados.0.smithi107.stdout:3311: left oid 18 (ObjNum 3310 snap 0 seq_num 3310) 2020-06-12T02:37:49.460 INFO:tasks.rados.rados.0.smithi107.stdout:3311: done (2 left) 2020-06-12T02:37:49.460 INFO:tasks.rados.rados.0.smithi107.stdout:3315: done (1 left) 2020-06-12T02:37:49.460 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:37:49.461 INFO:tasks.rados.rados.0.smithi107.stdout:3316: seq_num 3315 ranges {0=327680} 2020-06-12T02:37:49.465 INFO:tasks.rados.rados.0.smithi107.stdout:3316: writing smithi10729407-20 from 0 to 327680 tid 1 2020-06-12T02:37:49.465 INFO:tasks.rados.rados.0.smithi107.stdout:3317: delete oid 6 current snap is 0 2020-06-12T02:37:49.465 INFO:tasks.rados.rados.0.smithi107.stdout:3312: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:49.465 INFO:tasks.rados.rados.0.smithi107.stdout:3312: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:49.466 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 556 (ObjNum 3311 snap 0 seq_num 3311) dirty exists 2020-06-12T02:37:49.466 INFO:tasks.rados.rados.0.smithi107.stdout:3312: left oid 8 (ObjNum 3311 snap 0 seq_num 3311) 2020-06-12T02:37:49.466 INFO:tasks.rados.rados.0.smithi107.stdout:3312: done (2 left) 2020-06-12T02:37:49.467 INFO:tasks.rados.rados.0.smithi107.stdout:3317: done (1 left) 2020-06-12T02:37:49.467 INFO:tasks.rados.rados.0.smithi107.stdout:3318: delete oid 5 current snap is 0 2020-06-12T02:37:49.467 INFO:tasks.rados.rados.0.smithi107.stdout:3316: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:49.468 INFO:tasks.rados.rados.0.smithi107.stdout:3316: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:49.468 INFO:tasks.rados.rados.0.smithi107.stdout:3318: done (1 left) 2020-06-12T02:37:49.468 INFO:tasks.rados.rados.0.smithi107.stdout:3319: delete oid 6 current snap is 0 2020-06-12T02:37:49.469 INFO:tasks.rados.rados.0.smithi107.stdout:3316: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:49.469 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 233 (ObjNum 3315 snap 0 seq_num 3315) dirty exists 2020-06-12T02:37:49.469 INFO:tasks.rados.rados.0.smithi107.stdout:3316: left oid 20 (ObjNum 3315 snap 0 seq_num 3315) 2020-06-12T02:37:49.470 INFO:tasks.rados.rados.0.smithi107.stdout:3316: done (1 left) 2020-06-12T02:37:49.470 INFO:tasks.rados.rados.0.smithi107.stdout:3319: done (0 left) 2020-06-12T02:37:49.470 INFO:tasks.rados.rados.0.smithi107.stdout:3320: delete oid 4 current snap is 0 2020-06-12T02:37:49.472 INFO:tasks.rados.rados.0.smithi107.stdout:3320: done (0 left) 2020-06-12T02:37:49.473 INFO:tasks.rados.rados.0.smithi107.stdout:3321: delete oid 20 current snap is 0 2020-06-12T02:37:49.474 INFO:tasks.rados.rados.0.smithi107.stdout:3321: done (0 left) 2020-06-12T02:37:49.474 INFO:tasks.rados.rados.0.smithi107.stdout:3322: delete oid 19 current snap is 0 2020-06-12T02:37:49.477 INFO:tasks.rados.rados.0.smithi107.stdout:3322: done (0 left) 2020-06-12T02:37:49.477 INFO:tasks.rados.rados.0.smithi107.stdout:3323: delete oid 20 current snap is 0 2020-06-12T02:37:49.478 INFO:tasks.rados.rados.0.smithi107.stdout:3323: done (0 left) 2020-06-12T02:37:49.478 INFO:tasks.rados.rados.0.smithi107.stdout:3324: delete oid 12 current snap is 0 2020-06-12T02:37:49.482 INFO:tasks.rados.rados.0.smithi107.stdout:3324: done (0 left) 2020-06-12T02:37:49.482 INFO:tasks.rados.rados.0.smithi107.stdout:3325: delete oid 16 current snap is 0 2020-06-12T02:37:49.483 INFO:tasks.rados.rados.0.smithi107.stdout:3325: done (0 left) 2020-06-12T02:37:49.483 INFO:tasks.rados.rados.0.smithi107.stdout:3326: delete oid 9 current snap is 0 2020-06-12T02:37:49.485 INFO:tasks.rados.rados.0.smithi107.stdout:3326: done (0 left) 2020-06-12T02:37:49.485 INFO:tasks.rados.rados.0.smithi107.stdout:3327: delete oid 14 current snap is 0 2020-06-12T02:37:49.487 INFO:tasks.rados.rados.0.smithi107.stdout:3327: done (0 left) 2020-06-12T02:37:49.487 INFO:tasks.rados.rados.0.smithi107.stdout:3328: delete oid 13 current snap is 0 2020-06-12T02:37:49.490 INFO:tasks.rados.rados.0.smithi107.stdout:3328: done (0 left) 2020-06-12T02:37:49.490 INFO:tasks.rados.rados.0.smithi107.stdout:3329: delete oid 11 current snap is 0 2020-06-12T02:37:49.492 INFO:tasks.rados.rados.0.smithi107.stdout:3329: done (0 left) 2020-06-12T02:37:49.492 INFO:tasks.rados.rados.0.smithi107.stdout:3330: delete oid 3 current snap is 0 2020-06-12T02:37:49.495 INFO:tasks.rados.rados.0.smithi107.stdout:3330: done (0 left) 2020-06-12T02:37:49.495 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:37:49.496 INFO:tasks.rados.rados.0.smithi107.stdout:3331: seq_num 3330 ranges {0=573440} 2020-06-12T02:37:49.498 INFO:tasks.rados.rados.0.smithi107.stdout:3331: writing smithi10729407-20 from 0 to 573440 tid 1 2020-06-12T02:37:49.499 INFO:tasks.rados.rados.0.smithi107.stdout:3332: delete oid 6 current snap is 0 2020-06-12T02:37:49.501 INFO:tasks.rados.rados.0.smithi107.stdout:3332: done (1 left) 2020-06-12T02:37:49.501 INFO:tasks.rados.rados.0.smithi107.stdout:3333: delete oid 12 current snap is 0 2020-06-12T02:37:49.504 INFO:tasks.rados.rados.0.smithi107.stdout:3331: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:49.505 INFO:tasks.rados.rados.0.smithi107.stdout:3331: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:49.506 INFO:tasks.rados.rados.0.smithi107.stdout:3333: done (1 left) 2020-06-12T02:37:49.506 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:37:49.506 INFO:tasks.rados.rados.0.smithi107.stdout:3334: seq_num 3333 ranges {0=630784} 2020-06-12T02:37:49.509 INFO:tasks.rados.rados.0.smithi107.stdout:3334: writing smithi10729407-3 from 0 to 630784 tid 1 2020-06-12T02:37:49.509 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:37:49.510 INFO:tasks.rados.rados.0.smithi107.stdout:3335: seq_num 3334 ranges {221184=647168} 2020-06-12T02:37:49.516 INFO:tasks.rados.rados.0.smithi107.stdout:3335: writing smithi10729407-7 from 221184 to 868352 tid 1 2020-06-12T02:37:49.517 INFO:tasks.rados.rados.0.smithi107.stdout:3336: delete oid 8 current snap is 0 2020-06-12T02:37:49.518 INFO:tasks.rados.rados.0.smithi107.stdout:3331: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:49.518 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 237 (ObjNum 3330 snap 0 seq_num 3330) dirty exists 2020-06-12T02:37:49.519 INFO:tasks.rados.rados.0.smithi107.stdout:3331: left oid 20 (ObjNum 3330 snap 0 seq_num 3330) 2020-06-12T02:37:49.519 INFO:tasks.rados.rados.0.smithi107.stdout:3334: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:49.519 INFO:tasks.rados.rados.0.smithi107.stdout:3334: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:49.520 INFO:tasks.rados.rados.0.smithi107.stdout:3331: done (3 left) 2020-06-12T02:37:49.521 INFO:tasks.rados.rados.0.smithi107.stdout:3336: done (2 left) 2020-06-12T02:37:49.522 INFO:tasks.rados.rados.0.smithi107.stdout:3337: delete oid 1 current snap is 0 2020-06-12T02:37:49.522 INFO:tasks.rados.rados.0.smithi107.stdout:3334: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:49.522 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 596 (ObjNum 3333 snap 0 seq_num 3333) dirty exists 2020-06-12T02:37:49.522 INFO:tasks.rados.rados.0.smithi107.stdout:3334: left oid 3 (ObjNum 3333 snap 0 seq_num 3333) 2020-06-12T02:37:49.523 INFO:tasks.rados.rados.0.smithi107.stdout:3334: done (2 left) 2020-06-12T02:37:49.523 INFO:tasks.rados.rados.0.smithi107.stdout:3337: done (1 left) 2020-06-12T02:37:49.523 INFO:tasks.rados.rados.0.smithi107.stdout:3338: delete oid 12 current snap is 0 2020-06-12T02:37:49.524 INFO:tasks.rados.rados.0.smithi107.stdout:3335: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:49.525 INFO:tasks.rados.rados.0.smithi107.stdout:3338: done (1 left) 2020-06-12T02:37:49.525 INFO:tasks.rados.rados.0.smithi107.stdout:3339: delete oid 4 current snap is 0 2020-06-12T02:37:49.525 INFO:tasks.rados.rados.0.smithi107.stdout:3335: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:49.526 INFO:tasks.rados.rados.0.smithi107.stdout:3335: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:49.526 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 631 (ObjNum 3334 snap 0 seq_num 3334) dirty exists 2020-06-12T02:37:49.526 INFO:tasks.rados.rados.0.smithi107.stdout:3335: left oid 7 (ObjNum 3334 snap 0 seq_num 3334) 2020-06-12T02:37:49.527 INFO:tasks.rados.rados.0.smithi107.stdout:3335: done (1 left) 2020-06-12T02:37:49.527 INFO:tasks.rados.rados.0.smithi107.stdout:3339: done (0 left) 2020-06-12T02:37:49.527 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:37:49.528 INFO:tasks.rados.rados.0.smithi107.stdout:3340: seq_num 3339 ranges {3366912=729088} 2020-06-12T02:37:49.546 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:37:49.543+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:37:49.549 INFO:tasks.rados.rados.0.smithi107.stdout:3340: writing smithi10729407-15 from 3366912 to 4096000 tid 1 2020-06-12T02:37:49.549 INFO:tasks.rados.rados.0.smithi107.stdout:3341: delete oid 2 current snap is 0 2020-06-12T02:37:49.551 INFO:tasks.rados.rados.0.smithi107.stdout:3341: done (1 left) 2020-06-12T02:37:49.551 INFO:tasks.rados.rados.0.smithi107.stdout:3342: delete oid 17 current snap is 0 2020-06-12T02:37:49.553 INFO:tasks.rados.rados.0.smithi107.stdout:3342: done (1 left) 2020-06-12T02:37:49.553 INFO:tasks.rados.rados.0.smithi107.stdout:3343: delete oid 12 current snap is 0 2020-06-12T02:37:49.555 INFO:tasks.rados.rados.0.smithi107.stdout:3343: done (1 left) 2020-06-12T02:37:49.555 INFO:tasks.rados.rados.0.smithi107.stdout:3344: delete oid 7 current snap is 0 2020-06-12T02:37:49.557 INFO:tasks.rados.rados.0.smithi107.stdout:3340: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:49.558 INFO:tasks.rados.rados.0.smithi107.stdout:3344: done (1 left) 2020-06-12T02:37:49.558 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:37:49.558 INFO:tasks.rados.rados.0.smithi107.stdout:3345: seq_num 3344 ranges {0=409600,409600=401408} 2020-06-12T02:37:49.560 INFO:tasks.rados.rados.0.smithi107.stdout:3345: writing smithi10729407-9 from 0 to 409600 tid 1 2020-06-12T02:37:49.563 INFO:tasks.rados.rados.0.smithi107.stdout:3345: writing smithi10729407-9 from 409600 to 811008 tid 2 2020-06-12T02:37:49.563 INFO:tasks.rados.rados.0.smithi107.stdout:3346: delete oid 5 current snap is 0 2020-06-12T02:37:49.564 INFO:tasks.rados.rados.0.smithi107.stdout:3340: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:49.564 INFO:tasks.rados.rados.0.smithi107.stdout:3340: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:49.565 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 559 (ObjNum 3339 snap 0 seq_num 3339) dirty exists 2020-06-12T02:37:49.565 INFO:tasks.rados.rados.0.smithi107.stdout:3340: left oid 15 (ObjNum 3339 snap 0 seq_num 3339) 2020-06-12T02:37:49.565 INFO:tasks.rados.rados.0.smithi107.stdout:3340: done (2 left) 2020-06-12T02:37:49.566 INFO:tasks.rados.rados.0.smithi107.stdout:3346: done (1 left) 2020-06-12T02:37:49.566 INFO:tasks.rados.rados.0.smithi107.stdout:3347: delete oid 14 current snap is 0 2020-06-12T02:37:49.574 INFO:tasks.rados.rados.0.smithi107.stdout:3345: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:49.575 INFO:tasks.rados.rados.0.smithi107.stdout:3347: done (1 left) 2020-06-12T02:37:49.575 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:37:49.575 INFO:tasks.rados.rados.0.smithi107.stdout:3348: seq_num 3347 ranges {0=458752,458752=401408} 2020-06-12T02:37:49.575 INFO:tasks.rados.rados.0.smithi107.stdout:3348: writing smithi10729407-7 from 0 to 458752 tid 1 2020-06-12T02:37:49.576 INFO:tasks.rados.rados.0.smithi107.stdout:3348: writing smithi10729407-7 from 458752 to 860160 tid 2 2020-06-12T02:37:49.576 INFO:tasks.rados.rados.0.smithi107.stdout:3349: delete oid 6 current snap is 0 2020-06-12T02:37:49.576 INFO:tasks.rados.rados.0.smithi107.stdout:3345: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:49.576 INFO:tasks.rados.rados.0.smithi107.stdout:3345: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:49.576 INFO:tasks.rados.rados.0.smithi107.stdout:3345: finishing write tid 5 to smithi10729407-9 2020-06-12T02:37:49.576 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 597 (ObjNum 3344 snap 0 seq_num 3344) dirty exists 2020-06-12T02:37:49.577 INFO:tasks.rados.rados.0.smithi107.stdout:3345: left oid 9 (ObjNum 3344 snap 0 seq_num 3344) 2020-06-12T02:37:49.577 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:37:49.578 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:37:49.578 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2020-06-12T02:37:49.578 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:37:49.579 INFO:tasks.rados.rados.0.smithi107.stdout:3348: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:49.579 INFO:tasks.rados.rados.0.smithi107.stdout:3345: done (2 left) 2020-06-12T02:37:49.579 INFO:tasks.rados.rados.0.smithi107.stdout:3349: done (1 left) 2020-06-12T02:37:49.579 INFO:tasks.rados.rados.0.smithi107.stdout:3350: delete oid 17 current snap is 0 2020-06-12T02:37:49.580 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-06-12T02:37:49.581 INFO:tasks.rados.rados.0.smithi107.stdout:3350: done (1 left) 2020-06-12T02:37:49.581 INFO:tasks.rados.rados.0.smithi107.stdout:3351: delete oid 6 current snap is 0 2020-06-12T02:37:49.582 INFO:tasks.rados.rados.0.smithi107.stdout:3348: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:49.582 INFO:tasks.rados.rados.0.smithi107.stdout:3348: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:49.583 INFO:tasks.rados.rados.0.smithi107.stdout:3351: done (1 left) 2020-06-12T02:37:49.583 INFO:tasks.rados.rados.0.smithi107.stdout:3352: delete oid 17 current snap is 0 2020-06-12T02:37:49.584 INFO:tasks.rados.rados.0.smithi107.stdout:3348: finishing write tid 5 to smithi10729407-7 2020-06-12T02:37:49.585 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 635 (ObjNum 3347 snap 0 seq_num 3347) dirty exists 2020-06-12T02:37:49.585 INFO:tasks.rados.rados.0.smithi107.stdout:3348: left oid 7 (ObjNum 3347 snap 0 seq_num 3347) 2020-06-12T02:37:49.585 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:37:49.586 INFO:tasks.rados.rados.0.smithi107.stdout:3348: done (1 left) 2020-06-12T02:37:49.587 INFO:tasks.rados.rados.0.smithi107.stdout:3352: done (0 left) 2020-06-12T02:37:49.587 INFO:tasks.rados.rados.0.smithi107.stdout:3353: delete oid 1 current snap is 0 2020-06-12T02:37:49.588 INFO:tasks.rados.rados.0.smithi107.stdout:3353: done (0 left) 2020-06-12T02:37:49.588 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:37:49.588 INFO:tasks.rados.rados.0.smithi107.stdout:3354: seq_num 3353 ranges {0=565248,565248=401408} 2020-06-12T02:37:49.591 INFO:tasks.rados.rados.0.smithi107.stdout:3354: writing smithi10729407-5 from 0 to 565248 tid 1 2020-06-12T02:37:49.593 INFO:tasks.rados.rados.0.smithi107.stdout:3354: writing smithi10729407-5 from 565248 to 966656 tid 2 2020-06-12T02:37:49.593 INFO:tasks.rados.rados.0.smithi107.stdout:3355: delete oid 9 current snap is 0 2020-06-12T02:37:49.595 INFO:tasks.rados.rados.0.smithi107.stdout:3355: done (1 left) 2020-06-12T02:37:49.595 INFO:tasks.rados.rados.0.smithi107.stdout:3356: delete oid 6 current snap is 0 2020-06-12T02:37:49.599 INFO:tasks.rados.rados.0.smithi107.stdout:3354: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:49.599 INFO:tasks.rados.rados.0.smithi107.stdout:3356: done (1 left) 2020-06-12T02:37:49.600 INFO:tasks.rados.rados.0.smithi107.stdout:3357: delete oid 14 current snap is 0 2020-06-12T02:37:49.600 INFO:tasks.rados.rados.0.smithi107.stdout:3354: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:49.600 INFO:tasks.rados.rados.0.smithi107.stdout:3354: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:49.602 INFO:tasks.rados.rados.0.smithi107.stdout:3357: done (1 left) 2020-06-12T02:37:49.602 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:37:49.602 INFO:tasks.rados.rados.0.smithi107.stdout:3358: seq_num 3357 ranges {4096000=630784,4726784=401408} 2020-06-12T02:37:49.608 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:37:49.628 INFO:tasks.rados.rados.0.smithi107.stdout:3358: writing smithi10729407-15 from 4096000 to 4726784 tid 1 2020-06-12T02:37:49.628 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:37:49.630 INFO:tasks.rados.rados.0.smithi107.stdout:3358: writing smithi10729407-15 from 4726784 to 5128192 tid 2 2020-06-12T02:37:49.630 INFO:tasks.rados.rados.0.smithi107.stdout:3359: delete oid 4 current snap is 0 2020-06-12T02:37:49.630 INFO:tasks.rados.rados.0.smithi107.stdout:3354: finishing write tid 5 to smithi10729407-5 2020-06-12T02:37:49.630 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 288 (ObjNum 3353 snap 0 seq_num 3353) dirty exists 2020-06-12T02:37:49.630 INFO:tasks.rados.rados.0.smithi107.stdout:3354: left oid 5 (ObjNum 3353 snap 0 seq_num 3353) 2020-06-12T02:37:49.632 INFO:tasks.rados.rados.0.smithi107.stdout:3354: done (2 left) 2020-06-12T02:37:49.632 INFO:tasks.rados.rados.0.smithi107.stdout:3359: done (1 left) 2020-06-12T02:37:49.632 INFO:tasks.rados.rados.0.smithi107.stdout:3360: delete oid 20 current snap is 0 2020-06-12T02:37:49.636 INFO:tasks.rados.rados.0.smithi107.stdout:3358: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:49.638 INFO:tasks.rados.rados.0.smithi107.stdout:3358: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:49.640 INFO:tasks.rados.rados.0.smithi107.stdout:3358: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:49.640 INFO:tasks.rados.rados.0.smithi107.stdout:3360: done (1 left) 2020-06-12T02:37:49.641 INFO:tasks.rados.rados.0.smithi107.stdout:3361: delete oid 5 current snap is 0 2020-06-12T02:37:49.643 INFO:tasks.rados.rados.0.smithi107.stdout:3358: finishing write tid 5 to smithi10729407-15 2020-06-12T02:37:49.643 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 562 (ObjNum 3357 snap 0 seq_num 3357) dirty exists 2020-06-12T02:37:49.643 INFO:tasks.rados.rados.0.smithi107.stdout:3358: left oid 15 (ObjNum 3357 snap 0 seq_num 3357) 2020-06-12T02:37:49.646 INFO:tasks.rados.rados.0.smithi107.stdout:3358: done (1 left) 2020-06-12T02:37:49.646 INFO:tasks.rados.rados.0.smithi107.stdout:3361: done (0 left) 2020-06-12T02:37:49.646 INFO:tasks.rados.rados.0.smithi107.stdout:3362: delete oid 6 current snap is 0 2020-06-12T02:37:49.647 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:37:49.643+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:37:49.648 INFO:tasks.rados.rados.0.smithi107.stdout:3362: done (0 left) 2020-06-12T02:37:49.648 INFO:tasks.rados.rados.0.smithi107.stdout:3363: delete oid 3 current snap is 0 2020-06-12T02:37:49.650 INFO:tasks.rados.rados.0.smithi107.stdout:3363: done (0 left) 2020-06-12T02:37:49.650 INFO:tasks.rados.rados.0.smithi107.stdout:3364: delete oid 1 current snap is 0 2020-06-12T02:37:49.652 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:37:49.652 INFO:tasks.rados.rados.0.smithi107.stdout:3364: done (0 left) 2020-06-12T02:37:49.652 INFO:tasks.rados.rados.0.smithi107.stdout:3365: delete oid 13 current snap is 0 2020-06-12T02:37:49.654 INFO:tasks.rados.rados.0.smithi107.stdout:3365: done (0 left) 2020-06-12T02:37:49.654 INFO:tasks.rados.rados.0.smithi107.stdout:3366: delete oid 8 current snap is 0 2020-06-12T02:37:49.657 INFO:tasks.rados.rados.0.smithi107.stdout:3366: done (0 left) 2020-06-12T02:37:49.657 INFO:tasks.rados.rados.0.smithi107.stdout:3367: delete oid 19 current snap is 0 2020-06-12T02:37:49.660 INFO:tasks.rados.rados.0.smithi107.stdout:3367: done (0 left) 2020-06-12T02:37:49.660 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:37:49.660 INFO:tasks.rados.rados.0.smithi107.stdout:3368: seq_num 3367 ranges {0=794624,794624=401408} 2020-06-12T02:37:49.664 INFO:tasks.rados.rados.0.smithi107.stdout:3368: writing smithi10729407-8 from 0 to 794624 tid 1 2020-06-12T02:37:49.668 INFO:tasks.rados.rados.0.smithi107.stdout:3368: writing smithi10729407-8 from 794624 to 1196032 tid 2 2020-06-12T02:37:49.668 INFO:tasks.rados.rados.0.smithi107.stdout:3369: delete oid 4 current snap is 0 2020-06-12T02:37:49.670 INFO:tasks.rados.rados.0.smithi107.stdout:3369: done (1 left) 2020-06-12T02:37:49.670 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:37:49.671 INFO:tasks.rados.rados.0.smithi107.stdout:3370: seq_num 3369 ranges {0=425984,425984=729088,1155072=73728} 2020-06-12T02:37:49.672 INFO:tasks.rados.rados.0.smithi107.stdout:3370: writing smithi10729407-2 from 0 to 425984 tid 1 2020-06-12T02:37:49.678 INFO:tasks.rados.rados.0.smithi107.stdout:3370: writing smithi10729407-2 from 425984 to 1155072 tid 2 2020-06-12T02:37:49.679 INFO:tasks.rados.rados.0.smithi107.stdout:3370: writing smithi10729407-2 from 1155072 to 1228800 tid 3 2020-06-12T02:37:49.679 INFO:tasks.rados.rados.0.smithi107.stdout:3371: delete oid 20 current snap is 0 2020-06-12T02:37:49.679 INFO:tasks.rados.rados.0.smithi107.stdout:3368: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:49.679 INFO:tasks.rados.rados.0.smithi107.stdout:3370: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:49.679 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:37:49.681 INFO:tasks.rados.rados.0.smithi107.stdout:3368: finishing write tid 2 to smithi10729407-8 2020-06-12T02:37:49.681 INFO:tasks.rados.rados.0.smithi107.stdout:3368: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:49.682 INFO:tasks.rados.rados.0.smithi107.stdout:3371: done (2 left) 2020-06-12T02:37:49.682 INFO:tasks.rados.rados.0.smithi107.stdout:3372: delete oid 19 current snap is 0 2020-06-12T02:37:49.683 INFO:tasks.rados.rados.0.smithi107.stdout:3368: finishing write tid 5 to smithi10729407-8 2020-06-12T02:37:49.683 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 566 (ObjNum 3367 snap 0 seq_num 3367) dirty exists 2020-06-12T02:37:49.684 INFO:tasks.rados.rados.0.smithi107.stdout:3368: left oid 8 (ObjNum 3367 snap 0 seq_num 3367) 2020-06-12T02:37:49.685 INFO:tasks.rados.rados.0.smithi107.stdout:3370: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:49.686 INFO:tasks.rados.rados.0.smithi107.stdout:3370: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:49.686 INFO:tasks.rados.rados.0.smithi107.stdout:3370: finishing write tid 5 to smithi10729407-2 2020-06-12T02:37:49.687 INFO:tasks.rados.rados.0.smithi107.stdout:3368: done (2 left) 2020-06-12T02:37:49.687 INFO:tasks.rados.rados.0.smithi107.stdout:3372: done (1 left) 2020-06-12T02:37:49.687 INFO:tasks.rados.rados.0.smithi107.stdout:3373: delete oid 5 current snap is 0 2020-06-12T02:37:49.689 INFO:tasks.rados.rados.0.smithi107.stdout:3370: finishing write tid 6 to smithi10729407-2 2020-06-12T02:37:49.689 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 352 (ObjNum 3369 snap 0 seq_num 3369) dirty exists 2020-06-12T02:37:49.690 INFO:tasks.rados.rados.0.smithi107.stdout:3370: left oid 2 (ObjNum 3369 snap 0 seq_num 3369) 2020-06-12T02:37:49.690 INFO:tasks.rados.rados.0.smithi107.stdout:3370: done (1 left) 2020-06-12T02:37:49.690 INFO:tasks.rados.rados.0.smithi107.stdout:3373: done (0 left) 2020-06-12T02:37:49.690 INFO:tasks.rados.rados.0.smithi107.stdout:3374: delete oid 18 current snap is 0 2020-06-12T02:37:49.691 INFO:tasks.rados.rados.0.smithi107.stdout:3374: done (0 left) 2020-06-12T02:37:49.691 INFO:tasks.rados.rados.0.smithi107.stdout:3375: delete oid 16 current snap is 0 2020-06-12T02:37:49.694 INFO:tasks.rados.rados.0.smithi107.stdout:3375: done (0 left) 2020-06-12T02:37:49.695 INFO:tasks.rados.rados.0.smithi107.stdout:3376: delete oid 4 current snap is 0 2020-06-12T02:37:49.696 INFO:tasks.rados.rados.0.smithi107.stdout:3376: done (0 left) 2020-06-12T02:37:49.696 INFO:tasks.rados.rados.0.smithi107.stdout:3377: delete oid 20 current snap is 0 2020-06-12T02:37:49.698 INFO:tasks.rados.rados.0.smithi107.stdout:3377: done (0 left) 2020-06-12T02:37:49.698 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:37:49.699 INFO:tasks.rados.rados.0.smithi107.stdout:3378: seq_num 3377 ranges {860160=565248,1425408=655360,2080768=147456} 2020-06-12T02:37:49.703 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:37:49.706 INFO:tasks.rados.rados.0.smithi107.stdout:3378: writing smithi10729407-7 from 860160 to 1425408 tid 1 2020-06-12T02:37:49.710 INFO:tasks.rados.rados.0.smithi107.stdout:3378: writing smithi10729407-7 from 1425408 to 2080768 tid 2 2020-06-12T02:37:49.711 INFO:tasks.rados.rados.0.smithi107.stdout:3378: writing smithi10729407-7 from 2080768 to 2228224 tid 3 2020-06-12T02:37:49.711 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:37:49.711 INFO:tasks.rados.rados.0.smithi107.stdout:3379: seq_num 3378 ranges {0=581632,581632=540672,1122304=262144} 2020-06-12T02:37:49.714 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:37:49.715 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:37:49.715 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:37:49.715 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:37:49.715 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:37:49.716 INFO:tasks.rados.rados.0.smithi107.stdout:3379: writing smithi10729407-11 from 0 to 581632 tid 1 2020-06-12T02:37:49.718 INFO:tasks.rados.rados.0.smithi107.stdout:3379: writing smithi10729407-11 from 581632 to 1122304 tid 2 2020-06-12T02:37:49.720 INFO:tasks.rados.rados.0.smithi107.stdout:3379: writing smithi10729407-11 from 1122304 to 1384448 tid 3 2020-06-12T02:37:49.721 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:37:49.721 INFO:tasks.rados.rados.0.smithi107.stdout:3380: seq_num 3379 ranges {5128192=598016,5726208=417792,6144000=385024} 2020-06-12T02:37:49.725 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-06-12T02:37:49.731 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:37:49.732 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-06-12T02:37:49.733 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-06-12T02:37:49.733 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-06-12T02:37:49.733 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-06-12T02:37:49.733 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-06-12T02:37:49.734 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-06-12T02:37:49.734 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-06-12T02:37:49.734 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.7 2020-06-12T02:37:49.734 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: problem getting command descriptions from osd.7 2020-06-12T02:37:49.735 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-06-12T02:37:49.735 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-06-12T02:37:49.752 INFO:tasks.rados.rados.0.smithi107.stdout:3380: writing smithi10729407-15 from 5128192 to 5726208 tid 1 2020-06-12T02:37:49.753 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:49.749+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:49.765 INFO:tasks.rados.rados.0.smithi107.stdout:3380: writing smithi10729407-15 from 5726208 to 6144000 tid 2 2020-06-12T02:37:49.765 INFO:tasks.rados.rados.0.smithi107.stdout:3380: writing smithi10729407-15 from 6144000 to 6529024 tid 3 2020-06-12T02:37:49.765 INFO:tasks.rados.rados.0.smithi107.stdout:3381: delete oid 12 current snap is 0 2020-06-12T02:37:49.766 INFO:tasks.rados.rados.0.smithi107.stdout:3378: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:49.766 INFO:tasks.rados.rados.0.smithi107.stdout:3378: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:49.766 INFO:tasks.rados.rados.0.smithi107.stdout:3378: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:49.766 INFO:tasks.rados.rados.0.smithi107.stdout:3378: finishing write tid 5 to smithi10729407-7 2020-06-12T02:37:49.767 INFO:tasks.rados.rados.0.smithi107.stdout:3379: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:49.767 INFO:tasks.rados.rados.0.smithi107.stdout:3378: finishing write tid 6 to smithi10729407-7 2020-06-12T02:37:49.767 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 640 (ObjNum 3377 snap 0 seq_num 3377) dirty exists 2020-06-12T02:37:49.767 INFO:tasks.rados.rados.0.smithi107.stdout:3378: left oid 7 (ObjNum 3377 snap 0 seq_num 3377) 2020-06-12T02:37:49.767 INFO:tasks.rados.rados.0.smithi107.stdout:3379: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:49.768 INFO:tasks.rados.rados.0.smithi107.stdout:3379: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:49.768 INFO:tasks.rados.rados.0.smithi107.stdout:3379: finishing write tid 5 to smithi10729407-11 2020-06-12T02:37:49.768 INFO:tasks.rados.rados.0.smithi107.stdout:3379: finishing write tid 6 to smithi10729407-11 2020-06-12T02:37:49.768 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 571 (ObjNum 3378 snap 0 seq_num 3378) dirty exists 2020-06-12T02:37:49.769 INFO:tasks.rados.rados.0.smithi107.stdout:3379: left oid 11 (ObjNum 3378 snap 0 seq_num 3378) 2020-06-12T02:37:49.769 INFO:tasks.rados.rados.0.smithi107.stdout:3380: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:49.769 INFO:tasks.rados.rados.0.smithi107.stdout:3378: done (3 left) 2020-06-12T02:37:49.769 INFO:tasks.rados.rados.0.smithi107.stdout:3379: done (2 left) 2020-06-12T02:37:49.769 INFO:tasks.rados.rados.0.smithi107.stdout:3381: done (1 left) 2020-06-12T02:37:49.770 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:37:49.770 INFO:tasks.rados.rados.0.smithi107.stdout:3382: seq_num 3381 ranges {0=630784,630784=589824,1220608=212992} 2020-06-12T02:37:49.770 INFO:tasks.rados.rados.0.smithi107.stdout:3382: writing smithi10729407-1 from 0 to 630784 tid 1 2020-06-12T02:37:49.771 INFO:tasks.rados.rados.0.smithi107.stdout:3382: writing smithi10729407-1 from 630784 to 1220608 tid 2 2020-06-12T02:37:49.772 INFO:tasks.rados.rados.0.smithi107.stdout:3382: writing smithi10729407-1 from 1220608 to 1433600 tid 3 2020-06-12T02:37:49.772 INFO:tasks.rados.rados.0.smithi107.stdout:3383: delete oid 12 current snap is 0 2020-06-12T02:37:49.772 INFO:tasks.rados.rados.0.smithi107.stdout:3380: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:49.772 INFO:tasks.rados.rados.0.smithi107.stdout:3380: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:49.772 INFO:tasks.rados.rados.0.smithi107.stdout:3380: finishing write tid 5 to smithi10729407-15 2020-06-12T02:37:49.773 INFO:tasks.rados.rados.0.smithi107.stdout:3380: finishing write tid 6 to smithi10729407-15 2020-06-12T02:37:49.773 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 570 (ObjNum 3379 snap 0 seq_num 3379) dirty exists 2020-06-12T02:37:49.773 INFO:tasks.rados.rados.0.smithi107.stdout:3380: left oid 15 (ObjNum 3379 snap 0 seq_num 3379) 2020-06-12T02:37:49.773 INFO:tasks.rados.rados.0.smithi107.stdout:3380: done (2 left) 2020-06-12T02:37:49.774 INFO:tasks.rados.rados.0.smithi107.stdout:3383: done (1 left) 2020-06-12T02:37:49.774 INFO:tasks.rados.rados.0.smithi107.stdout:3384: delete oid 5 current snap is 0 2020-06-12T02:37:49.775 INFO:tasks.rados.rados.0.smithi107.stdout:3384: done (1 left) 2020-06-12T02:37:49.775 INFO:tasks.rados.rados.0.smithi107.stdout:3385: delete oid 16 current snap is 0 2020-06-12T02:37:49.781 INFO:tasks.rados.rados.0.smithi107.stdout:3382: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:49.782 INFO:tasks.rados.rados.0.smithi107.stdout:3382: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:49.783 INFO:tasks.rados.rados.0.smithi107.stdout:3385: done (1 left) 2020-06-12T02:37:49.783 INFO:tasks.rados.rados.0.smithi107.stdout:3386: delete oid 17 current snap is 0 2020-06-12T02:37:49.783 INFO:tasks.rados.rados.0.smithi107.stdout:3382: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:49.784 INFO:tasks.rados.rados.0.smithi107.stdout:3382: finishing write tid 5 to smithi10729407-1 2020-06-12T02:37:49.784 INFO:tasks.rados.rados.0.smithi107.stdout:3386: done (1 left) 2020-06-12T02:37:49.784 INFO:tasks.rados.rados.0.smithi107.stdout:3387: delete oid 4 current snap is 0 2020-06-12T02:37:49.786 INFO:tasks.rados.rados.0.smithi107.stdout:3387: done (1 left) 2020-06-12T02:37:49.786 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:37:49.786 INFO:tasks.rados.rados.0.smithi107.stdout:3388: seq_num 3387 ranges {2228224=729088,2957312=745472,3702784=57344} 2020-06-12T02:37:49.797 INFO:tasks.rados.rados.0.smithi107.stdout:3388: writing smithi10729407-7 from 2228224 to 2957312 tid 1 2020-06-12T02:37:49.802 INFO:tasks.rados.rados.0.smithi107.stdout:3388: writing smithi10729407-7 from 2957312 to 3702784 tid 2 2020-06-12T02:37:49.802 INFO:tasks.rados.rados.0.smithi107.stdout:3388: writing smithi10729407-7 from 3702784 to 3760128 tid 3 2020-06-12T02:37:49.803 INFO:tasks.rados.rados.0.smithi107.stdout:3389: delete oid 11 current snap is 0 2020-06-12T02:37:49.803 INFO:tasks.rados.rados.0.smithi107.stdout:3382: finishing write tid 6 to smithi10729407-1 2020-06-12T02:37:49.803 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 645 (ObjNum 3381 snap 0 seq_num 3381) dirty exists 2020-06-12T02:37:49.803 INFO:tasks.rados.rados.0.smithi107.stdout:3382: left oid 1 (ObjNum 3381 snap 0 seq_num 3381) 2020-06-12T02:37:49.805 INFO:tasks.rados.rados.0.smithi107.stdout:3382: done (2 left) 2020-06-12T02:37:49.805 INFO:tasks.rados.rados.0.smithi107.stdout:3389: done (1 left) 2020-06-12T02:37:49.805 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:37:49.806 INFO:tasks.rados.rados.0.smithi107.stdout:3390: seq_num 3389 ranges {0=761856,761856=516096,1277952=286720} 2020-06-12T02:37:49.810 INFO:tasks.rados.rados.0.smithi107.stdout:3390: writing smithi10729407-13 from 0 to 761856 tid 1 2020-06-12T02:37:49.815 INFO:tasks.rados.rados.0.smithi107.stdout:3390: writing smithi10729407-13 from 761856 to 1277952 tid 2 2020-06-12T02:37:49.815 INFO:tasks.rados.rados.0.smithi107.stdout:3390: writing smithi10729407-13 from 1277952 to 1564672 tid 3 2020-06-12T02:37:49.816 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:37:49.816 INFO:tasks.rados.rados.0.smithi107.stdout:3391: seq_num 3390 ranges {0=786432,786432=802816} 2020-06-12T02:37:49.820 INFO:tasks.rados.rados.0.smithi107.stdout:3391: writing smithi10729407-12 from 0 to 786432 tid 1 2020-06-12T02:37:49.825 INFO:tasks.rados.rados.0.smithi107.stdout:3391: writing smithi10729407-12 from 786432 to 1589248 tid 2 2020-06-12T02:37:49.825 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:37:49.826 INFO:tasks.rados.rados.0.smithi107.stdout:3392: seq_num 3391 ranges {0=802816,802816=737280,1540096=65536} 2020-06-12T02:37:49.830 INFO:tasks.rados.rados.0.smithi107.stdout:3392: writing smithi10729407-20 from 0 to 802816 tid 1 2020-06-12T02:37:49.834 INFO:tasks.rados.rados.0.smithi107.stdout:3392: writing smithi10729407-20 from 802816 to 1540096 tid 2 2020-06-12T02:37:49.836 INFO:tasks.rados.rados.0.smithi107.stdout:3392: writing smithi10729407-20 from 1540096 to 1605632 tid 3 2020-06-12T02:37:49.836 INFO:tasks.rados.rados.0.smithi107.stdout:3393: delete oid 19 current snap is 0 2020-06-12T02:37:49.836 INFO:tasks.rados.rados.0.smithi107.stdout:3388: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:49.836 INFO:tasks.rados.rados.0.smithi107.stdout:3388: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:49.837 INFO:tasks.rados.rados.0.smithi107.stdout:3388: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:49.837 INFO:tasks.rados.rados.0.smithi107.stdout:3388: finishing write tid 5 to smithi10729407-7 2020-06-12T02:37:49.837 INFO:tasks.rados.rados.0.smithi107.stdout:3388: finishing write tid 6 to smithi10729407-7 2020-06-12T02:37:49.837 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 644 (ObjNum 3387 snap 0 seq_num 3387) dirty exists 2020-06-12T02:37:49.837 INFO:tasks.rados.rados.0.smithi107.stdout:3388: left oid 7 (ObjNum 3387 snap 0 seq_num 3387) 2020-06-12T02:37:49.840 INFO:tasks.rados.rados.0.smithi107.stdout:3390: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:49.840 INFO:tasks.rados.rados.0.smithi107.stdout:3390: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:49.840 INFO:tasks.rados.rados.0.smithi107.stdout:3390: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:49.840 INFO:tasks.rados.rados.0.smithi107.stdout:3390: finishing write tid 5 to smithi10729407-13 2020-06-12T02:37:49.840 INFO:tasks.rados.rados.0.smithi107.stdout:3390: finishing write tid 6 to smithi10729407-13 2020-06-12T02:37:49.841 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 648 (ObjNum 3389 snap 0 seq_num 3389) dirty exists 2020-06-12T02:37:49.841 INFO:tasks.rados.rados.0.smithi107.stdout:3390: left oid 13 (ObjNum 3389 snap 0 seq_num 3389) 2020-06-12T02:37:49.841 INFO:tasks.rados.rados.0.smithi107.stdout:3391: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:49.841 INFO:tasks.rados.rados.0.smithi107.stdout:3391: finishing write tid 2 to smithi10729407-12 2020-06-12T02:37:49.841 INFO:tasks.rados.rados.0.smithi107.stdout:3391: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:49.843 INFO:tasks.rados.rados.0.smithi107.stdout:3391: finishing write tid 5 to smithi10729407-12 2020-06-12T02:37:49.843 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 559 (ObjNum 3390 snap 0 seq_num 3390) dirty exists 2020-06-12T02:37:49.844 INFO:tasks.rados.rados.0.smithi107.stdout:3391: left oid 12 (ObjNum 3390 snap 0 seq_num 3390) 2020-06-12T02:37:49.844 INFO:tasks.rados.rados.0.smithi107.stdout:3388: done (4 left) 2020-06-12T02:37:49.845 INFO:tasks.rados.rados.0.smithi107.stdout:3390: done (3 left) 2020-06-12T02:37:49.846 INFO:tasks.rados.rados.0.smithi107.stdout:3391: done (2 left) 2020-06-12T02:37:49.846 INFO:tasks.rados.rados.0.smithi107.stdout:3393: done (1 left) 2020-06-12T02:37:49.846 INFO:tasks.rados.rados.0.smithi107.stdout:3394: delete oid 4 current snap is 0 2020-06-12T02:37:49.846 INFO:tasks.rados.rados.0.smithi107.stdout:3392: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:49.847 INFO:tasks.rados.rados.0.smithi107.stdout:3394: done (1 left) 2020-06-12T02:37:49.847 INFO:tasks.rados.rados.0.smithi107.stdout:3395: delete oid 17 current snap is 0 2020-06-12T02:37:49.847 INFO:tasks.rados.rados.0.smithi107.stdout:3395: done (1 left) 2020-06-12T02:37:49.847 INFO:tasks.rados.rados.0.smithi107.stdout:3396: delete oid 11 current snap is 0 2020-06-12T02:37:49.848 INFO:tasks.rados.rados.0.smithi107.stdout:3392: finishing write tid 2 to smithi10729407-20 2020-06-12T02:37:49.848 INFO:tasks.rados.rados.0.smithi107.stdout:3396: done (1 left) 2020-06-12T02:37:49.848 INFO:tasks.rados.rados.0.smithi107.stdout:3397: delete oid 18 current snap is 0 2020-06-12T02:37:49.849 INFO:tasks.rados.rados.0.smithi107.stdout:3392: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:49.849 INFO:tasks.rados.rados.0.smithi107.stdout:3392: finishing write tid 5 to smithi10729407-20 2020-06-12T02:37:49.849 INFO:tasks.rados.rados.0.smithi107.stdout:3397: done (1 left) 2020-06-12T02:37:49.850 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:37:49.850 INFO:tasks.rados.rados.0.smithi107.stdout:3398: seq_num 3397 ranges {155648=499712,655360=442368,1097728=507904,1605632=253952} 2020-06-12T02:37:49.851 INFO:tasks.rados.rados.0.smithi107.stdout:3398: writing smithi10729407-10 from 155648 to 655360 tid 1 2020-06-12T02:37:49.853 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:49.849+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:37:49.856 INFO:tasks.rados.rados.0.smithi107.stdout:3398: writing smithi10729407-10 from 655360 to 1097728 tid 2 2020-06-12T02:37:49.857 INFO:tasks.rados.rados.0.smithi107.stdout:3398: writing smithi10729407-10 from 1097728 to 1605632 tid 3 2020-06-12T02:37:49.859 INFO:tasks.rados.rados.0.smithi107.stdout:3398: writing smithi10729407-10 from 1605632 to 1859584 tid 4 2020-06-12T02:37:49.859 INFO:tasks.rados.rados.0.smithi107.stdout:3399: delete oid 17 current snap is 0 2020-06-12T02:37:49.860 INFO:tasks.rados.rados.0.smithi107.stdout:3392: finishing write tid 6 to smithi10729407-20 2020-06-12T02:37:49.860 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 244 (ObjNum 3391 snap 0 seq_num 3391) dirty exists 2020-06-12T02:37:49.860 INFO:tasks.rados.rados.0.smithi107.stdout:3392: left oid 20 (ObjNum 3391 snap 0 seq_num 3391) 2020-06-12T02:37:49.860 INFO:tasks.rados.rados.0.smithi107.stdout:3398: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:49.862 INFO:tasks.rados.rados.0.smithi107.stdout:3392: done (2 left) 2020-06-12T02:37:49.862 INFO:tasks.rados.rados.0.smithi107.stdout:3399: done (1 left) 2020-06-12T02:37:49.862 INFO:tasks.rados.rados.0.smithi107.stdout:3400: delete oid 20 current snap is 0 2020-06-12T02:37:49.863 INFO:tasks.rados.rados.0.smithi107.stdout:3398: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:49.864 INFO:tasks.rados.rados.0.smithi107.stdout:3400: done (1 left) 2020-06-12T02:37:49.864 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:37:49.864 INFO:tasks.rados.rados.0.smithi107.stdout:3401: seq_num 3400 ranges {0=548864,548864=540672,1089536=663552} 2020-06-12T02:37:49.865 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:49.866 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:49.867 INFO:tasks.rados.rados.0.smithi107.stdout:3401: writing smithi10729407-16 from 0 to 548864 tid 1 2020-06-12T02:37:49.870 INFO:tasks.rados.rados.0.smithi107.stdout:3401: writing smithi10729407-16 from 548864 to 1089536 tid 2 2020-06-12T02:37:49.876 INFO:tasks.rados.rados.0.smithi107.stdout:3401: writing smithi10729407-16 from 1089536 to 1753088 tid 3 2020-06-12T02:37:49.877 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:37:49.877 INFO:tasks.rados.rados.0.smithi107.stdout:3402: seq_num 3401 ranges {0=565248,565248=425984,991232=483328,1474560=294912} 2020-06-12T02:37:49.877 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-06-12T02:37:49.878 INFO:tasks.rados.rados.0.smithi107.stdout:3402: writing smithi10729407-11 from 0 to 565248 tid 1 2020-06-12T02:37:49.883 INFO:tasks.rados.rados.0.smithi107.stdout:3402: writing smithi10729407-11 from 565248 to 991232 tid 2 2020-06-12T02:37:49.884 INFO:tasks.rados.rados.0.smithi107.stdout:3402: writing smithi10729407-11 from 991232 to 1474560 tid 3 2020-06-12T02:37:49.886 INFO:tasks.rados.rados.0.smithi107.stdout:3402: writing smithi10729407-11 from 1474560 to 1769472 tid 4 2020-06-12T02:37:49.886 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:37:49.887 INFO:tasks.rados.rados.0.smithi107.stdout:3403: seq_num 3402 ranges {0=581632,581632=712704,1294336=491520} 2020-06-12T02:37:49.890 INFO:tasks.rados.rados.0.smithi107.stdout:3403: writing smithi10729407-19 from 0 to 581632 tid 1 2020-06-12T02:37:49.898 INFO:tasks.rados.rados.0.smithi107.stdout:3403: writing smithi10729407-19 from 581632 to 1294336 tid 2 2020-06-12T02:37:49.898 INFO:tasks.rados.rados.0.smithi107.stdout:3403: writing smithi10729407-19 from 1294336 to 1785856 tid 3 2020-06-12T02:37:49.898 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:37:49.899 INFO:tasks.rados.rados.0.smithi107.stdout:3404: seq_num 3403 ranges {3760128=598016,4358144=598016,4956160=499712,5455872=106496} 2020-06-12T02:37:49.923 INFO:tasks.rados.rados.0.smithi107.stdout:3404: writing smithi10729407-7 from 3760128 to 4358144 tid 1 2020-06-12T02:37:49.926 INFO:tasks.rados.rados.0.smithi107.stdout:3404: writing smithi10729407-7 from 4358144 to 4956160 tid 2 2020-06-12T02:37:49.930 INFO:tasks.rados.rados.0.smithi107.stdout:3404: writing smithi10729407-7 from 4956160 to 5455872 tid 3 2020-06-12T02:37:49.931 INFO:tasks.rados.rados.0.smithi107.stdout:3404: writing smithi10729407-7 from 5455872 to 5562368 tid 4 2020-06-12T02:37:49.931 INFO:tasks.rados.rados.0.smithi107.stdout:3405: delete oid 4 current snap is 0 2020-06-12T02:37:49.931 INFO:tasks.rados.rados.0.smithi107.stdout:3398: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:49.932 INFO:tasks.rados.rados.0.smithi107.stdout:3398: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:49.932 INFO:tasks.rados.rados.0.smithi107.stdout:3398: finishing write tid 6 to smithi10729407-10 2020-06-12T02:37:49.932 INFO:tasks.rados.rados.0.smithi107.stdout:3398: finishing write tid 7 to smithi10729407-10 2020-06-12T02:37:49.932 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 336 (ObjNum 3397 snap 0 seq_num 3397) dirty exists 2020-06-12T02:37:49.932 INFO:tasks.rados.rados.0.smithi107.stdout:3398: left oid 10 (ObjNum 3397 snap 0 seq_num 3397) 2020-06-12T02:37:49.932 INFO:tasks.rados.rados.0.smithi107.stdout:3401: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:49.933 INFO:tasks.rados.rados.0.smithi107.stdout:3402: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:49.933 INFO:tasks.rados.rados.0.smithi107.stdout:3401: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:49.933 INFO:tasks.rados.rados.0.smithi107.stdout:3401: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:49.933 INFO:tasks.rados.rados.0.smithi107.stdout:3401: finishing write tid 5 to smithi10729407-16 2020-06-12T02:37:49.933 INFO:tasks.rados.rados.0.smithi107.stdout:3401: finishing write tid 6 to smithi10729407-16 2020-06-12T02:37:49.934 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 603 (ObjNum 3400 snap 0 seq_num 3400) dirty exists 2020-06-12T02:37:49.934 INFO:tasks.rados.rados.0.smithi107.stdout:3401: left oid 16 (ObjNum 3400 snap 0 seq_num 3400) 2020-06-12T02:37:49.934 INFO:tasks.rados.rados.0.smithi107.stdout:3403: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:49.934 INFO:tasks.rados.rados.0.smithi107.stdout:3402: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:49.934 INFO:tasks.rados.rados.0.smithi107.stdout:3402: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:49.934 INFO:tasks.rados.rados.0.smithi107.stdout:3402: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:49.935 INFO:tasks.rados.rados.0.smithi107.stdout:3403: finishing write tid 2 to smithi10729407-19 2020-06-12T02:37:49.935 INFO:tasks.rados.rados.0.smithi107.stdout:3402: finishing write tid 6 to smithi10729407-11 2020-06-12T02:37:49.935 INFO:tasks.rados.rados.0.smithi107.stdout:3403: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:49.935 INFO:tasks.rados.rados.0.smithi107.stdout:3402: finishing write tid 7 to smithi10729407-11 2020-06-12T02:37:49.935 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 578 (ObjNum 3401 snap 0 seq_num 3401) dirty exists 2020-06-12T02:37:49.935 INFO:tasks.rados.rados.0.smithi107.stdout:3402: left oid 11 (ObjNum 3401 snap 0 seq_num 3401) 2020-06-12T02:37:49.936 INFO:tasks.rados.rados.0.smithi107.stdout:3403: finishing write tid 5 to smithi10729407-19 2020-06-12T02:37:49.936 INFO:tasks.rados.rados.0.smithi107.stdout:3403: finishing write tid 6 to smithi10729407-19 2020-06-12T02:37:49.936 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 564 (ObjNum 3402 snap 0 seq_num 3402) dirty exists 2020-06-12T02:37:49.937 INFO:tasks.rados.rados.0.smithi107.stdout:3403: left oid 19 (ObjNum 3402 snap 0 seq_num 3402) 2020-06-12T02:37:49.937 INFO:tasks.rados.rados.0.smithi107.stdout:3404: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:49.939 INFO:tasks.rados.rados.0.smithi107.stdout:3398: done (5 left) 2020-06-12T02:37:49.939 INFO:tasks.rados.rados.0.smithi107.stdout:3401: done (4 left) 2020-06-12T02:37:49.939 INFO:tasks.rados.rados.0.smithi107.stdout:3402: done (3 left) 2020-06-12T02:37:49.940 INFO:tasks.rados.rados.0.smithi107.stdout:3403: done (2 left) 2020-06-12T02:37:49.940 INFO:tasks.rados.rados.0.smithi107.stdout:3405: done (1 left) 2020-06-12T02:37:49.940 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:37:49.940 INFO:tasks.rados.rados.0.smithi107.stdout:3406: seq_num 3405 ranges {0=630784,630784=761856,1392640=442368} 2020-06-12T02:37:49.940 INFO:tasks.rados.rados.0.smithi107.stdout:3406: writing smithi10729407-6 from 0 to 630784 tid 1 2020-06-12T02:37:49.941 INFO:tasks.rados.rados.0.smithi107.stdout:3406: writing smithi10729407-6 from 630784 to 1392640 tid 2 2020-06-12T02:37:49.945 INFO:tasks.rados.rados.0.smithi107.stdout:3406: writing smithi10729407-6 from 1392640 to 1835008 tid 3 2020-06-12T02:37:49.945 INFO:tasks.rados.rados.0.smithi107.stdout:3407: delete oid 2 current snap is 0 2020-06-12T02:37:49.945 INFO:tasks.rados.rados.0.smithi107.stdout:3404: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:49.945 INFO:tasks.rados.rados.0.smithi107.stdout:3404: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:49.945 INFO:tasks.rados.rados.0.smithi107.stdout:3404: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:49.946 INFO:tasks.rados.rados.0.smithi107.stdout:3404: finishing write tid 6 to smithi10729407-7 2020-06-12T02:37:49.946 INFO:tasks.rados.rados.0.smithi107.stdout:3404: finishing write tid 7 to smithi10729407-7 2020-06-12T02:37:49.946 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 653 (ObjNum 3403 snap 0 seq_num 3403) dirty exists 2020-06-12T02:37:49.946 INFO:tasks.rados.rados.0.smithi107.stdout:3404: left oid 7 (ObjNum 3403 snap 0 seq_num 3403) 2020-06-12T02:37:49.946 INFO:tasks.rados.rados.0.smithi107.stdout:3406: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:49.947 INFO:tasks.rados.rados.0.smithi107.stdout:3404: done (2 left) 2020-06-12T02:37:49.947 INFO:tasks.rados.rados.0.smithi107.stdout:3407: done (1 left) 2020-06-12T02:37:49.947 INFO:tasks.rados.rados.0.smithi107.stdout:3408: delete oid 17 current snap is 0 2020-06-12T02:37:49.954 INFO:tasks.rados.rados.0.smithi107.stdout:3408: done (1 left) 2020-06-12T02:37:49.954 INFO:tasks.rados.rados.0.smithi107.stdout:3409: delete oid 16 current snap is 0 2020-06-12T02:37:49.954 INFO:tasks.rados.rados.0.smithi107.stdout:3406: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:49.955 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:37:49.951+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:37:49.955 INFO:tasks.rados.rados.0.smithi107.stdout:3409: done (1 left) 2020-06-12T02:37:49.957 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:37:49.957 INFO:tasks.rados.rados.0.smithi107.stdout:3410: seq_num 3409 ranges {0=704512,704512=753664,1458176=442368,1900544=8192} 2020-06-12T02:37:49.959 INFO:tasks.rados.rados.0.smithi107.stdout:3410: writing smithi10729407-14 from 0 to 704512 tid 1 2020-06-12T02:37:49.963 INFO:tasks.rados.rados.0.smithi107.stdout:3410: writing smithi10729407-14 from 704512 to 1458176 tid 2 2020-06-12T02:37:49.967 INFO:tasks.rados.rados.0.smithi107.stdout:3410: writing smithi10729407-14 from 1458176 to 1900544 tid 3 2020-06-12T02:37:49.967 INFO:tasks.rados.rados.0.smithi107.stdout:3410: writing smithi10729407-14 from 1900544 to 1908736 tid 4 2020-06-12T02:37:49.967 INFO:tasks.rados.rados.0.smithi107.stdout:3411: delete oid 13 current snap is 0 2020-06-12T02:37:49.967 INFO:tasks.rados.rados.0.smithi107.stdout:3406: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:49.967 INFO:tasks.rados.rados.0.smithi107.stdout:3406: finishing write tid 5 to smithi10729407-6 2020-06-12T02:37:49.968 INFO:tasks.rados.rados.0.smithi107.stdout:3406: finishing write tid 6 to smithi10729407-6 2020-06-12T02:37:49.968 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 649 (ObjNum 3405 snap 0 seq_num 3405) dirty exists 2020-06-12T02:37:49.968 INFO:tasks.rados.rados.0.smithi107.stdout:3406: left oid 6 (ObjNum 3405 snap 0 seq_num 3405) 2020-06-12T02:37:49.968 INFO:tasks.rados.rados.0.smithi107.stdout:3410: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:49.971 INFO:tasks.rados.rados.0.smithi107.stdout:3406: done (2 left) 2020-06-12T02:37:49.972 INFO:tasks.rados.rados.0.smithi107.stdout:3411: done (1 left) 2020-06-12T02:37:49.972 INFO:tasks.rados.rados.0.smithi107.stdout:3412: delete oid 8 current snap is 0 2020-06-12T02:37:49.980 INFO:tasks.rados.rados.0.smithi107.stdout:3410: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:49.982 INFO:tasks.rados.rados.0.smithi107.stdout:3412: done (1 left) 2020-06-12T02:37:49.982 INFO:tasks.rados.rados.0.smithi107.stdout:3413: delete oid 5 current snap is 0 2020-06-12T02:37:49.982 INFO:tasks.rados.rados.0.smithi107.stdout:3410: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:49.982 INFO:tasks.rados.rados.0.smithi107.stdout:3410: finishing write tid 4 to smithi10729407-14 2020-06-12T02:37:49.983 INFO:tasks.rados.rados.0.smithi107.stdout:3413: done (1 left) 2020-06-12T02:37:49.983 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:37:49.983 INFO:tasks.rados.rados.0.smithi107.stdout:3414: seq_num 3413 ranges {1769472=770048,2539520=688128,3227648=425984,3653632=90112} 2020-06-12T02:37:49.993 INFO:tasks.rados.rados.0.smithi107.stdout:3414: writing smithi10729407-11 from 1769472 to 2539520 tid 1 2020-06-12T02:37:49.996 INFO:tasks.rados.rados.0.smithi107.stdout:3414: writing smithi10729407-11 from 2539520 to 3227648 tid 2 2020-06-12T02:37:49.999 INFO:tasks.rados.rados.0.smithi107.stdout:3414: writing smithi10729407-11 from 3227648 to 3653632 tid 3 2020-06-12T02:37:49.999 INFO:tasks.rados.rados.0.smithi107.stdout:3414: writing smithi10729407-11 from 3653632 to 3743744 tid 4 2020-06-12T02:37:50.000 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:37:50.000 INFO:tasks.rados.rados.0.smithi107.stdout:3415: seq_num 3414 ranges {1835008=786432,2621440=622592,3244032=581632} 2020-06-12T02:37:50.014 INFO:tasks.rados.rados.0.smithi107.stdout:3415: writing smithi10729407-6 from 1835008 to 2621440 tid 1 2020-06-12T02:37:50.018 INFO:tasks.rados.rados.0.smithi107.stdout:3415: writing smithi10729407-6 from 2621440 to 3244032 tid 2 2020-06-12T02:37:50.022 INFO:tasks.rados.rados.0.smithi107.stdout:3415: writing smithi10729407-6 from 3244032 to 3825664 tid 3 2020-06-12T02:37:50.022 INFO:tasks.rados.rados.0.smithi107.stdout:3416: delete oid 12 current snap is 0 2020-06-12T02:37:50.022 INFO:tasks.rados.rados.0.smithi107.stdout:3410: finishing write tid 6 to smithi10729407-14 2020-06-12T02:37:50.022 INFO:tasks.rados.rados.0.smithi107.stdout:3410: finishing write tid 7 to smithi10729407-14 2020-06-12T02:37:50.023 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 583 (ObjNum 3409 snap 0 seq_num 3409) dirty exists 2020-06-12T02:37:50.023 INFO:tasks.rados.rados.0.smithi107.stdout:3410: left oid 14 (ObjNum 3409 snap 0 seq_num 3409) 2020-06-12T02:37:50.023 INFO:tasks.rados.rados.0.smithi107.stdout:3414: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:50.023 INFO:tasks.rados.rados.0.smithi107.stdout:3414: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:50.023 INFO:tasks.rados.rados.0.smithi107.stdout:3414: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:50.024 INFO:tasks.rados.rados.0.smithi107.stdout:3414: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:50.024 INFO:tasks.rados.rados.0.smithi107.stdout:3414: finishing write tid 6 to smithi10729407-11 2020-06-12T02:37:50.024 INFO:tasks.rados.rados.0.smithi107.stdout:3414: finishing write tid 7 to smithi10729407-11 2020-06-12T02:37:50.024 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 588 (ObjNum 3413 snap 0 seq_num 3413) dirty exists 2020-06-12T02:37:50.024 INFO:tasks.rados.rados.0.smithi107.stdout:3414: left oid 11 (ObjNum 3413 snap 0 seq_num 3413) 2020-06-12T02:37:50.025 INFO:tasks.rados.rados.0.smithi107.stdout:3415: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:50.025 INFO:tasks.rados.rados.0.smithi107.stdout:3410: done (3 left) 2020-06-12T02:37:50.026 INFO:tasks.rados.rados.0.smithi107.stdout:3414: done (2 left) 2020-06-12T02:37:50.027 INFO:tasks.rados.rados.0.smithi107.stdout:3416: done (1 left) 2020-06-12T02:37:50.027 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:37:50.027 INFO:tasks.rados.rados.0.smithi107.stdout:3417: seq_num 3416 ranges {0=417792,417792=794624,1212416=647168,1859584=163840} 2020-06-12T02:37:50.027 INFO:tasks.rados.rados.0.smithi107.stdout:3417: writing smithi10729407-18 from 0 to 417792 tid 1 2020-06-12T02:37:50.032 INFO:tasks.rados.rados.0.smithi107.stdout:3417: writing smithi10729407-18 from 417792 to 1212416 tid 2 2020-06-12T02:37:50.035 INFO:tasks.rados.rados.0.smithi107.stdout:3417: writing smithi10729407-18 from 1212416 to 1859584 tid 3 2020-06-12T02:37:50.037 INFO:tasks.rados.rados.0.smithi107.stdout:3417: writing smithi10729407-18 from 1859584 to 2023424 tid 4 2020-06-12T02:37:50.037 INFO:tasks.rados.rados.0.smithi107.stdout:3418: delete oid 7 current snap is 0 2020-06-12T02:37:50.037 INFO:tasks.rados.rados.0.smithi107.stdout:3415: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:50.037 INFO:tasks.rados.rados.0.smithi107.stdout:3415: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:50.038 INFO:tasks.rados.rados.0.smithi107.stdout:3415: finishing write tid 5 to smithi10729407-6 2020-06-12T02:37:50.038 INFO:tasks.rados.rados.0.smithi107.stdout:3417: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:50.038 INFO:tasks.rados.rados.0.smithi107.stdout:3415: finishing write tid 6 to smithi10729407-6 2020-06-12T02:37:50.038 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 653 (ObjNum 3414 snap 0 seq_num 3414) dirty exists 2020-06-12T02:37:50.039 INFO:tasks.rados.rados.0.smithi107.stdout:3415: left oid 6 (ObjNum 3414 snap 0 seq_num 3414) 2020-06-12T02:37:50.040 INFO:tasks.rados.rados.0.smithi107.stdout:3415: done (2 left) 2020-06-12T02:37:50.040 INFO:tasks.rados.rados.0.smithi107.stdout:3418: done (1 left) 2020-06-12T02:37:50.040 INFO:tasks.rados.rados.0.smithi107.stdout:3419: delete oid 16 current snap is 0 2020-06-12T02:37:50.044 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:50.044 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:50.047 INFO:tasks.rados.rados.0.smithi107.stdout:3417: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:50.048 INFO:tasks.rados.rados.0.smithi107.stdout:3419: done (1 left) 2020-06-12T02:37:50.048 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:37:50.048 INFO:tasks.rados.rados.0.smithi107.stdout:3420: seq_num 3419 ranges {0=466944,466944=442368,909312=425984,1335296=671744,2007040=65536} 2020-06-12T02:37:50.050 INFO:tasks.rados.rados.0.smithi107.stdout:3420: writing smithi10729407-20 from 0 to 466944 tid 1 2020-06-12T02:37:50.053 INFO:tasks.rados.rados.0.smithi107.stdout:3420: writing smithi10729407-20 from 466944 to 909312 tid 2 2020-06-12T02:37:50.056 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-06-12T02:37:50.057 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:50.052+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:50.057 INFO:tasks.rados.rados.0.smithi107.stdout:3420: writing smithi10729407-20 from 909312 to 1335296 tid 3 2020-06-12T02:37:50.060 INFO:tasks.rados.rados.0.smithi107.stdout:3420: writing smithi10729407-20 from 1335296 to 2007040 tid 4 2020-06-12T02:37:50.061 INFO:tasks.rados.rados.0.smithi107.stdout:3420: writing smithi10729407-20 from 2007040 to 2072576 tid 5 2020-06-12T02:37:50.061 INFO:tasks.rados.rados.0.smithi107.stdout:3421: delete oid 19 current snap is 0 2020-06-12T02:37:50.062 INFO:tasks.rados.rados.0.smithi107.stdout:3417: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:50.062 INFO:tasks.rados.rados.0.smithi107.stdout:3417: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:50.062 INFO:tasks.rados.rados.0.smithi107.stdout:3417: finishing write tid 6 to smithi10729407-18 2020-06-12T02:37:50.063 INFO:tasks.rados.rados.0.smithi107.stdout:3417: finishing write tid 7 to smithi10729407-18 2020-06-12T02:37:50.063 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 605 (ObjNum 3416 snap 0 seq_num 3416) dirty exists 2020-06-12T02:37:50.063 INFO:tasks.rados.rados.0.smithi107.stdout:3417: left oid 18 (ObjNum 3416 snap 0 seq_num 3416) 2020-06-12T02:37:50.063 INFO:tasks.rados.rados.0.smithi107.stdout:3420: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:50.063 INFO:tasks.rados.rados.0.smithi107.stdout:3420: finishing write tid 2 to smithi10729407-20 2020-06-12T02:37:50.064 INFO:tasks.rados.rados.0.smithi107.stdout:3417: done (2 left) 2020-06-12T02:37:50.064 INFO:tasks.rados.rados.0.smithi107.stdout:3421: done (1 left) 2020-06-12T02:37:50.064 INFO:tasks.rados.rados.0.smithi107.stdout:3422: delete oid 12 current snap is 0 2020-06-12T02:37:50.065 INFO:tasks.rados.rados.0.smithi107.stdout:3420: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:50.066 INFO:tasks.rados.rados.0.smithi107.stdout:3422: done (1 left) 2020-06-12T02:37:50.066 INFO:tasks.rados.rados.0.smithi107.stdout:3423: delete oid 14 current snap is 0 2020-06-12T02:37:50.067 INFO:tasks.rados.rados.0.smithi107.stdout:3420: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:50.068 INFO:tasks.rados.rados.0.smithi107.stdout:3420: finishing write tid 5 to smithi10729407-20 2020-06-12T02:37:50.068 INFO:tasks.rados.rados.0.smithi107.stdout:3420: finishing write tid 7 to smithi10729407-20 2020-06-12T02:37:50.069 INFO:tasks.rados.rados.0.smithi107.stdout:3420: finishing write tid 8 to smithi10729407-20 2020-06-12T02:37:50.069 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 251 (ObjNum 3419 snap 0 seq_num 3419) dirty exists 2020-06-12T02:37:50.070 INFO:tasks.rados.rados.0.smithi107.stdout:3420: left oid 20 (ObjNum 3419 snap 0 seq_num 3419) 2020-06-12T02:37:50.070 INFO:tasks.rados.rados.0.smithi107.stdout:3420: done (1 left) 2020-06-12T02:37:50.070 INFO:tasks.rados.rados.0.smithi107.stdout:3423: done (0 left) 2020-06-12T02:37:50.071 INFO:tasks.rados.rados.0.smithi107.stdout:3424: delete oid 2 current snap is 0 2020-06-12T02:37:50.072 INFO:tasks.rados.rados.0.smithi107.stdout:3424: done (0 left) 2020-06-12T02:37:50.073 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:37:50.073 INFO:tasks.rados.rados.0.smithi107.stdout:3425: seq_num 3424 ranges {3743744=548864,4292608=712704,5005312=614400,5619712=278528} 2020-06-12T02:37:50.098 INFO:tasks.rados.rados.0.smithi107.stdout:3425: writing smithi10729407-11 from 3743744 to 4292608 tid 1 2020-06-12T02:37:50.102 INFO:tasks.rados.rados.0.smithi107.stdout:3425: writing smithi10729407-11 from 4292608 to 5005312 tid 2 2020-06-12T02:37:50.107 INFO:tasks.rados.rados.0.smithi107.stdout:3425: writing smithi10729407-11 from 5005312 to 5619712 tid 3 2020-06-12T02:37:50.109 INFO:tasks.rados.rados.0.smithi107.stdout:3425: writing smithi10729407-11 from 5619712 to 5898240 tid 4 2020-06-12T02:37:50.109 INFO:tasks.rados.rados.0.smithi107.stdout:3426: delete oid 9 current snap is 0 2020-06-12T02:37:50.110 INFO:tasks.rados.rados.0.smithi107.stdout:3425: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:50.116 INFO:tasks.rados.rados.0.smithi107.stdout:3426: done (1 left) 2020-06-12T02:37:50.116 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:37:50.116 INFO:tasks.rados.rados.0.smithi107.stdout:3427: seq_num 3426 ranges {0=581632,581632=483328,1064960=573440,1638400=442368,2080768=106496} 2020-06-12T02:37:50.117 INFO:tasks.rados.rados.0.smithi107.stdout:3427: writing smithi10729407-13 from 0 to 581632 tid 1 2020-06-12T02:37:50.125 INFO:tasks.rados.rados.0.smithi107.stdout:3427: writing smithi10729407-13 from 581632 to 1064960 tid 2 2020-06-12T02:37:50.125 INFO:tasks.rados.rados.0.smithi107.stdout:3427: writing smithi10729407-13 from 1064960 to 1638400 tid 3 2020-06-12T02:37:50.127 INFO:tasks.rados.rados.0.smithi107.stdout:3427: writing smithi10729407-13 from 1638400 to 2080768 tid 4 2020-06-12T02:37:50.130 INFO:tasks.rados.rados.0.smithi107.stdout:3427: writing smithi10729407-13 from 2080768 to 2187264 tid 5 2020-06-12T02:37:50.131 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:37:50.131 INFO:tasks.rados.rados.0.smithi107.stdout:3428: seq_num 3427 ranges {0=598016,598016=770048,1368064=786432,2154496=49152} 2020-06-12T02:37:50.131 INFO:tasks.rados.rados.0.smithi107.stdout:3428: writing smithi10729407-4 from 0 to 598016 tid 1 2020-06-12T02:37:50.137 INFO:tasks.rados.rados.0.smithi107.stdout:3428: writing smithi10729407-4 from 598016 to 1368064 tid 2 2020-06-12T02:37:50.141 INFO:tasks.rados.rados.0.smithi107.stdout:3428: writing smithi10729407-4 from 1368064 to 2154496 tid 3 2020-06-12T02:37:50.142 INFO:tasks.rados.rados.0.smithi107.stdout:3428: writing smithi10729407-4 from 2154496 to 2203648 tid 4 2020-06-12T02:37:50.146 INFO:tasks.rados.rados.0.smithi107.stdout:3429: delete oid 20 current snap is 0 2020-06-12T02:37:50.146 INFO:tasks.rados.rados.0.smithi107.stdout:3425: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:50.147 INFO:tasks.rados.rados.0.smithi107.stdout:3425: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:50.147 INFO:tasks.rados.rados.0.smithi107.stdout:3425: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:50.147 INFO:tasks.rados.rados.0.smithi107.stdout:3425: finishing write tid 6 to smithi10729407-11 2020-06-12T02:37:50.147 INFO:tasks.rados.rados.0.smithi107.stdout:3425: finishing write tid 7 to smithi10729407-11 2020-06-12T02:37:50.147 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 594 (ObjNum 3424 snap 0 seq_num 3424) dirty exists 2020-06-12T02:37:50.148 INFO:tasks.rados.rados.0.smithi107.stdout:3425: left oid 11 (ObjNum 3424 snap 0 seq_num 3424) 2020-06-12T02:37:50.148 INFO:tasks.rados.rados.0.smithi107.stdout:3427: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:50.148 INFO:tasks.rados.rados.0.smithi107.stdout:3427: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:50.148 INFO:tasks.rados.rados.0.smithi107.stdout:3427: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:50.148 INFO:tasks.rados.rados.0.smithi107.stdout:3427: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:50.149 INFO:tasks.rados.rados.0.smithi107.stdout:3427: finishing write tid 5 to smithi10729407-13 2020-06-12T02:37:50.149 INFO:tasks.rados.rados.0.smithi107.stdout:3427: finishing write tid 7 to smithi10729407-13 2020-06-12T02:37:50.149 INFO:tasks.rados.rados.0.smithi107.stdout:3428: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:50.149 INFO:tasks.rados.rados.0.smithi107.stdout:3427: finishing write tid 8 to smithi10729407-13 2020-06-12T02:37:50.149 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 661 (ObjNum 3426 snap 0 seq_num 3426) dirty exists 2020-06-12T02:37:50.149 INFO:tasks.rados.rados.0.smithi107.stdout:3427: left oid 13 (ObjNum 3426 snap 0 seq_num 3426) 2020-06-12T02:37:50.150 INFO:tasks.rados.rados.0.smithi107.stdout:3425: done (3 left) 2020-06-12T02:37:50.151 INFO:tasks.rados.rados.0.smithi107.stdout:3427: done (2 left) 2020-06-12T02:37:50.151 INFO:tasks.rados.rados.0.smithi107.stdout:3429: done (1 left) 2020-06-12T02:37:50.151 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:37:50.151 INFO:tasks.rados.rados.0.smithi107.stdout:3430: seq_num 3429 ranges {0=638976,638976=581632,1220608=753664,1974272=270336} 2020-06-12T02:37:50.151 INFO:tasks.rados.rados.0.smithi107.stdout:3430: writing smithi10729407-20 from 0 to 638976 tid 1 2020-06-12T02:37:50.152 INFO:tasks.rados.rados.0.smithi107.stdout:3430: writing smithi10729407-20 from 638976 to 1220608 tid 2 2020-06-12T02:37:50.157 INFO:tasks.rados.rados.0.smithi107.stdout:3430: writing smithi10729407-20 from 1220608 to 1974272 tid 3 2020-06-12T02:37:50.157 INFO:tasks.rados.rados.0.smithi107.stdout:3430: writing smithi10729407-20 from 1974272 to 2244608 tid 4 2020-06-12T02:37:50.157 INFO:tasks.rados.rados.0.smithi107.stdout:3431: delete oid 3 current snap is 0 2020-06-12T02:37:50.157 INFO:tasks.rados.rados.0.smithi107.stdout:3428: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:50.158 INFO:tasks.rados.rados.0.smithi107.stdout:3428: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:50.158 INFO:tasks.rados.rados.0.smithi107.stdout:3428: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:50.158 INFO:tasks.rados.rados.0.smithi107.stdout:3428: finishing write tid 6 to smithi10729407-4 2020-06-12T02:37:50.158 INFO:tasks.rados.rados.0.smithi107.stdout:3430: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:50.158 INFO:tasks.rados.rados.0.smithi107.stdout:3428: finishing write tid 7 to smithi10729407-4 2020-06-12T02:37:50.159 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 643 (ObjNum 3427 snap 0 seq_num 3427) dirty exists 2020-06-12T02:37:50.159 INFO:tasks.rados.rados.0.smithi107.stdout:3428: left oid 4 (ObjNum 3427 snap 0 seq_num 3427) 2020-06-12T02:37:50.159 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:50.154+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:37:50.160 INFO:tasks.rados.rados.0.smithi107.stdout:3428: done (2 left) 2020-06-12T02:37:50.160 INFO:tasks.rados.rados.0.smithi107.stdout:3431: done (1 left) 2020-06-12T02:37:50.160 INFO:tasks.rados.rados.0.smithi107.stdout:3432: delete oid 8 current snap is 0 2020-06-12T02:37:50.165 INFO:tasks.rados.rados.0.smithi107.stdout:3432: done (1 left) 2020-06-12T02:37:50.165 INFO:tasks.rados.rados.0.smithi107.stdout:3433: delete oid 17 current snap is 0 2020-06-12T02:37:50.167 INFO:tasks.rados.rados.0.smithi107.stdout:3430: finishing write tid 2 to smithi10729407-20 2020-06-12T02:37:50.167 INFO:tasks.rados.rados.0.smithi107.stdout:3433: done (1 left) 2020-06-12T02:37:50.168 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:37:50.168 INFO:tasks.rados.rados.0.smithi107.stdout:3434: seq_num 3433 ranges {0=704512,704512=524288,1228800=729088,1957888=352256} 2020-06-12T02:37:50.171 INFO:tasks.rados.rados.0.smithi107.stdout:3434: writing smithi10729407-3 from 0 to 704512 tid 1 2020-06-12T02:37:50.176 INFO:tasks.rados.rados.0.smithi107.stdout:3434: writing smithi10729407-3 from 704512 to 1228800 tid 2 2020-06-12T02:37:50.185 INFO:tasks.rados.rados.0.smithi107.stdout:3434: writing smithi10729407-3 from 1228800 to 1957888 tid 3 2020-06-12T02:37:50.185 INFO:tasks.rados.rados.0.smithi107.stdout:3434: writing smithi10729407-3 from 1957888 to 2310144 tid 4 2020-06-12T02:37:50.185 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:37:50.185 INFO:tasks.rados.rados.0.smithi107.stdout:3435: seq_num 3434 ranges {6529024=720896,7249920=409600,7659520=540672,8200192=548864,8749056=106496} 2020-06-12T02:37:50.209 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:50.210 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:50.218 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-06-12T02:37:50.221 INFO:tasks.rados.rados.0.smithi107.stdout:3435: writing smithi10729407-15 from 6529024 to 7249920 tid 1 2020-06-12T02:37:50.223 INFO:tasks.rados.rados.0.smithi107.stdout:3435: writing smithi10729407-15 from 7249920 to 7659520 tid 2 2020-06-12T02:37:50.227 INFO:tasks.rados.rados.0.smithi107.stdout:3435: writing smithi10729407-15 from 7659520 to 8200192 tid 3 2020-06-12T02:37:50.230 INFO:tasks.rados.rados.0.smithi107.stdout:3435: writing smithi10729407-15 from 8200192 to 8749056 tid 4 2020-06-12T02:37:50.231 INFO:tasks.rados.rados.0.smithi107.stdout:3435: writing smithi10729407-15 from 8749056 to 8855552 tid 5 2020-06-12T02:37:50.231 INFO:tasks.rados.rados.0.smithi107.stdout:3436: delete oid 8 current snap is 0 2020-06-12T02:37:50.231 INFO:tasks.rados.rados.0.smithi107.stdout:3430: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:50.232 INFO:tasks.rados.rados.0.smithi107.stdout:3430: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:50.232 INFO:tasks.rados.rados.0.smithi107.stdout:3430: finishing write tid 6 to smithi10729407-20 2020-06-12T02:37:50.232 INFO:tasks.rados.rados.0.smithi107.stdout:3430: finishing write tid 7 to smithi10729407-20 2020-06-12T02:37:50.232 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 257 (ObjNum 3429 snap 0 seq_num 3429) dirty exists 2020-06-12T02:37:50.232 INFO:tasks.rados.rados.0.smithi107.stdout:3430: left oid 20 (ObjNum 3429 snap 0 seq_num 3429) 2020-06-12T02:37:50.233 INFO:tasks.rados.rados.0.smithi107.stdout:3434: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:50.233 INFO:tasks.rados.rados.0.smithi107.stdout:3434: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:50.233 INFO:tasks.rados.rados.0.smithi107.stdout:3434: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:50.233 INFO:tasks.rados.rados.0.smithi107.stdout:3434: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:50.233 INFO:tasks.rados.rados.0.smithi107.stdout:3434: finishing write tid 6 to smithi10729407-3 2020-06-12T02:37:50.234 INFO:tasks.rados.rados.0.smithi107.stdout:3434: finishing write tid 7 to smithi10729407-3 2020-06-12T02:37:50.234 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 611 (ObjNum 3433 snap 0 seq_num 3433) dirty exists 2020-06-12T02:37:50.234 INFO:tasks.rados.rados.0.smithi107.stdout:3434: left oid 3 (ObjNum 3433 snap 0 seq_num 3433) 2020-06-12T02:37:50.234 INFO:tasks.rados.rados.0.smithi107.stdout:3435: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:50.235 INFO:tasks.rados.rados.0.smithi107.stdout:3435: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:50.236 INFO:tasks.rados.rados.0.smithi107.stdout:3435: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:50.237 INFO:tasks.rados.rados.0.smithi107.stdout:3435: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:50.239 INFO:tasks.rados.rados.0.smithi107.stdout:3435: finishing write tid 5 to smithi10729407-15 2020-06-12T02:37:50.240 INFO:tasks.rados.rados.0.smithi107.stdout:3435: finishing write tid 7 to smithi10729407-15 2020-06-12T02:37:50.241 INFO:tasks.rados.rados.0.smithi107.stdout:3430: done (3 left) 2020-06-12T02:37:50.241 INFO:tasks.rados.rados.0.smithi107.stdout:3434: done (2 left) 2020-06-12T02:37:50.241 INFO:tasks.rados.rados.0.smithi107.stdout:3436: done (1 left) 2020-06-12T02:37:50.241 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:37:50.241 INFO:tasks.rados.rados.0.smithi107.stdout:3437: seq_num 3436 ranges {0=753664,753664=573440,1327104=548864,1875968=483328} 2020-06-12T02:37:50.245 INFO:tasks.rados.rados.0.smithi107.stdout:3437: writing smithi10729407-5 from 0 to 753664 tid 1 2020-06-12T02:37:50.248 INFO:tasks.rados.rados.0.smithi107.stdout:3437: writing smithi10729407-5 from 753664 to 1327104 tid 2 2020-06-12T02:37:50.252 INFO:tasks.rados.rados.0.smithi107.stdout:3437: writing smithi10729407-5 from 1327104 to 1875968 tid 3 2020-06-12T02:37:50.255 INFO:tasks.rados.rados.0.smithi107.stdout:3437: writing smithi10729407-5 from 1875968 to 2359296 tid 4 2020-06-12T02:37:50.255 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:37:50.255 INFO:tasks.rados.rados.0.smithi107.stdout:3438: seq_num 3437 ranges {0=770048,770048=507904,1277952=761856,2039808=335872} 2020-06-12T02:37:50.257 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:50.256+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:50.259 INFO:tasks.rados.rados.0.smithi107.stdout:3438: writing smithi10729407-16 from 0 to 770048 tid 1 2020-06-12T02:37:50.263 INFO:tasks.rados.rados.0.smithi107.stdout:3438: writing smithi10729407-16 from 770048 to 1277952 tid 2 2020-06-12T02:37:50.267 INFO:tasks.rados.rados.0.smithi107.stdout:3438: writing smithi10729407-16 from 1277952 to 2039808 tid 3 2020-06-12T02:37:50.269 INFO:tasks.rados.rados.0.smithi107.stdout:3438: writing smithi10729407-16 from 2039808 to 2375680 tid 4 2020-06-12T02:37:50.270 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:37:50.270 INFO:tasks.rados.rados.0.smithi107.stdout:3439: seq_num 3438 ranges {0=786432,786432=794624,1581056=516096,2097152=294912} 2020-06-12T02:37:50.275 INFO:tasks.rados.rados.0.smithi107.stdout:3439: writing smithi10729407-9 from 0 to 786432 tid 1 2020-06-12T02:37:50.280 INFO:tasks.rados.rados.0.smithi107.stdout:3439: writing smithi10729407-9 from 786432 to 1581056 tid 2 2020-06-12T02:37:50.283 INFO:tasks.rados.rados.0.smithi107.stdout:3439: writing smithi10729407-9 from 1581056 to 2097152 tid 3 2020-06-12T02:37:50.286 INFO:tasks.rados.rados.0.smithi107.stdout:3439: writing smithi10729407-9 from 2097152 to 2392064 tid 4 2020-06-12T02:37:50.286 INFO:tasks.rados.rados.0.smithi107.stdout:3440: delete oid 19 current snap is 0 2020-06-12T02:37:50.286 INFO:tasks.rados.rados.0.smithi107.stdout:3435: finishing write tid 8 to smithi10729407-15 2020-06-12T02:37:50.287 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 578 (ObjNum 3434 snap 0 seq_num 3434) dirty exists 2020-06-12T02:37:50.287 INFO:tasks.rados.rados.0.smithi107.stdout:3435: left oid 15 (ObjNum 3434 snap 0 seq_num 3434) 2020-06-12T02:37:50.287 INFO:tasks.rados.rados.0.smithi107.stdout:3437: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:50.287 INFO:tasks.rados.rados.0.smithi107.stdout:3437: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:50.287 INFO:tasks.rados.rados.0.smithi107.stdout:3437: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:50.288 INFO:tasks.rados.rados.0.smithi107.stdout:3437: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:50.288 INFO:tasks.rados.rados.0.smithi107.stdout:3437: finishing write tid 6 to smithi10729407-5 2020-06-12T02:37:50.288 INFO:tasks.rados.rados.0.smithi107.stdout:3437: finishing write tid 7 to smithi10729407-5 2020-06-12T02:37:50.288 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 297 (ObjNum 3436 snap 0 seq_num 3436) dirty exists 2020-06-12T02:37:50.288 INFO:tasks.rados.rados.0.smithi107.stdout:3437: left oid 5 (ObjNum 3436 snap 0 seq_num 3436) 2020-06-12T02:37:50.288 INFO:tasks.rados.rados.0.smithi107.stdout:3438: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:50.289 INFO:tasks.rados.rados.0.smithi107.stdout:3438: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:50.289 INFO:tasks.rados.rados.0.smithi107.stdout:3438: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:50.289 INFO:tasks.rados.rados.0.smithi107.stdout:3438: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:50.290 INFO:tasks.rados.rados.0.smithi107.stdout:3438: finishing write tid 6 to smithi10729407-16 2020-06-12T02:37:50.290 INFO:tasks.rados.rados.0.smithi107.stdout:3435: done (4 left) 2020-06-12T02:37:50.290 INFO:tasks.rados.rados.0.smithi107.stdout:3437: done (3 left) 2020-06-12T02:37:50.291 INFO:tasks.rados.rados.0.smithi107.stdout:3440: done (2 left) 2020-06-12T02:37:50.291 INFO:tasks.rados.rados.0.smithi107.stdout:3441: delete oid 8 current snap is 0 2020-06-12T02:37:50.292 INFO:tasks.rados.rados.0.smithi107.stdout:3441: done (2 left) 2020-06-12T02:37:50.292 INFO:tasks.rados.rados.0.smithi107.stdout:3442: delete oid 6 current snap is 0 2020-06-12T02:37:50.294 INFO:tasks.rados.rados.0.smithi107.stdout:3438: finishing write tid 7 to smithi10729407-16 2020-06-12T02:37:50.294 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 616 (ObjNum 3437 snap 0 seq_num 3437) dirty exists 2020-06-12T02:37:50.294 INFO:tasks.rados.rados.0.smithi107.stdout:3438: left oid 16 (ObjNum 3437 snap 0 seq_num 3437) 2020-06-12T02:37:50.296 INFO:tasks.rados.rados.0.smithi107.stdout:3439: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:50.296 INFO:tasks.rados.rados.0.smithi107.stdout:3438: done (2 left) 2020-06-12T02:37:50.297 INFO:tasks.rados.rados.0.smithi107.stdout:3442: done (1 left) 2020-06-12T02:37:50.297 INFO:tasks.rados.rados.0.smithi107.stdout:3443: delete oid 15 current snap is 0 2020-06-12T02:37:50.297 INFO:tasks.rados.rados.0.smithi107.stdout:3439: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:50.297 INFO:tasks.rados.rados.0.smithi107.stdout:3439: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:50.297 INFO:tasks.rados.rados.0.smithi107.stdout:3439: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:50.298 INFO:tasks.rados.rados.0.smithi107.stdout:3443: done (1 left) 2020-06-12T02:37:50.298 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:37:50.299 INFO:tasks.rados.rados.0.smithi107.stdout:3444: seq_num 3443 ranges {0=65536} 2020-06-12T02:37:50.299 INFO:tasks.rados.rados.0.smithi107.stdout:3444: writing smithi10729407-15 from 0 to 65536 tid 1 2020-06-12T02:37:50.299 INFO:tasks.rados.rados.0.smithi107.stdout:3445: delete oid 12 current snap is 0 2020-06-12T02:37:50.300 INFO:tasks.rados.rados.0.smithi107.stdout:3439: finishing write tid 6 to smithi10729407-9 2020-06-12T02:37:50.301 INFO:tasks.rados.rados.0.smithi107.stdout:3439: finishing write tid 7 to smithi10729407-9 2020-06-12T02:37:50.301 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 611 (ObjNum 3438 snap 0 seq_num 3438) dirty exists 2020-06-12T02:37:50.301 INFO:tasks.rados.rados.0.smithi107.stdout:3439: left oid 9 (ObjNum 3438 snap 0 seq_num 3438) 2020-06-12T02:37:50.302 INFO:tasks.rados.rados.0.smithi107.stdout:3439: done (2 left) 2020-06-12T02:37:50.302 INFO:tasks.rados.rados.0.smithi107.stdout:3445: done (1 left) 2020-06-12T02:37:50.302 INFO:tasks.rados.rados.0.smithi107.stdout:3446: delete oid 7 current snap is 0 2020-06-12T02:37:50.304 INFO:tasks.rados.rados.0.smithi107.stdout:3446: done (1 left) 2020-06-12T02:37:50.304 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:37:50.304 INFO:tasks.rados.rados.0.smithi107.stdout:3447: seq_num 3446 ranges {2359296=114688} 2020-06-12T02:37:50.317 INFO:tasks.rados.rados.0.smithi107.stdout:3447: writing smithi10729407-5 from 2359296 to 2473984 tid 1 2020-06-12T02:37:50.318 INFO:tasks.rados.rados.0.smithi107.stdout:3448: delete oid 10 current snap is 0 2020-06-12T02:37:50.318 INFO:tasks.rados.rados.0.smithi107.stdout:3444: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:50.318 INFO:tasks.rados.rados.0.smithi107.stdout:3444: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:50.318 INFO:tasks.rados.rados.0.smithi107.stdout:3444: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:50.318 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 583 (ObjNum 3443 snap 0 seq_num 3443) dirty exists 2020-06-12T02:37:50.319 INFO:tasks.rados.rados.0.smithi107.stdout:3444: left oid 15 (ObjNum 3443 snap 0 seq_num 3443) 2020-06-12T02:37:50.319 INFO:tasks.rados.rados.0.smithi107.stdout:3444: done (2 left) 2020-06-12T02:37:50.319 INFO:tasks.rados.rados.0.smithi107.stdout:3448: done (1 left) 2020-06-12T02:37:50.320 INFO:tasks.rados.rados.0.smithi107.stdout:3449: delete oid 13 current snap is 0 2020-06-12T02:37:50.321 INFO:tasks.rados.rados.0.smithi107.stdout:3447: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:50.322 INFO:tasks.rados.rados.0.smithi107.stdout:3449: done (1 left) 2020-06-12T02:37:50.322 INFO:tasks.rados.rados.0.smithi107.stdout:3450: delete oid 10 current snap is 0 2020-06-12T02:37:50.324 INFO:tasks.rados.rados.0.smithi107.stdout:3447: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:50.325 INFO:tasks.rados.rados.0.smithi107.stdout:3450: done (1 left) 2020-06-12T02:37:50.325 INFO:tasks.rados.rados.0.smithi107.stdout:3451: delete oid 7 current snap is 0 2020-06-12T02:37:50.326 INFO:tasks.rados.rados.0.smithi107.stdout:3447: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:50.326 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 299 (ObjNum 3446 snap 0 seq_num 3446) dirty exists 2020-06-12T02:37:50.326 INFO:tasks.rados.rados.0.smithi107.stdout:3447: left oid 5 (ObjNum 3446 snap 0 seq_num 3446) 2020-06-12T02:37:50.332 INFO:tasks.rados.rados.0.smithi107.stdout:3447: done (1 left) 2020-06-12T02:37:50.333 INFO:tasks.rados.rados.0.smithi107.stdout:3451: done (0 left) 2020-06-12T02:37:50.333 INFO:tasks.rados.rados.0.smithi107.stdout:3452: delete oid 1 current snap is 0 2020-06-12T02:37:50.334 INFO:tasks.rados.rados.0.smithi107.stdout:3452: done (0 left) 2020-06-12T02:37:50.334 INFO:tasks.rados.rados.0.smithi107.stdout:3453: delete oid 19 current snap is 0 2020-06-12T02:37:50.336 INFO:tasks.rados.rados.0.smithi107.stdout:3453: done (0 left) 2020-06-12T02:37:50.336 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:37:50.336 INFO:tasks.rados.rados.0.smithi107.stdout:3454: seq_num 3453 ranges {0=237568} 2020-06-12T02:37:50.337 INFO:tasks.rados.rados.0.smithi107.stdout:3454: writing smithi10729407-1 from 0 to 237568 tid 1 2020-06-12T02:37:50.338 INFO:tasks.rados.rados.0.smithi107.stdout:3455: delete oid 19 current snap is 0 2020-06-12T02:37:50.340 INFO:tasks.rados.rados.0.smithi107.stdout:3455: done (1 left) 2020-06-12T02:37:50.340 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:37:50.340 INFO:tasks.rados.rados.0.smithi107.stdout:3456: seq_num 3455 ranges {0=270336} 2020-06-12T02:37:50.341 INFO:tasks.rados.rados.0.smithi107.stdout:3456: writing smithi10729407-14 from 0 to 270336 tid 1 2020-06-12T02:37:50.341 INFO:tasks.rados.rados.0.smithi107.stdout:3457: delete oid 7 current snap is 0 2020-06-12T02:37:50.341 INFO:tasks.rados.rados.0.smithi107.stdout:3454: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:50.343 INFO:tasks.rados.rados.0.smithi107.stdout:3457: done (2 left) 2020-06-12T02:37:50.343 INFO:tasks.rados.rados.0.smithi107.stdout:3458: delete oid 3 current snap is 0 2020-06-12T02:37:50.344 INFO:tasks.rados.rados.0.smithi107.stdout:3454: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:50.347 INFO:tasks.rados.rados.0.smithi107.stdout:3458: done (2 left) 2020-06-12T02:37:50.348 INFO:tasks.rados.rados.0.smithi107.stdout:3459: delete oid 19 current snap is 0 2020-06-12T02:37:50.348 INFO:tasks.rados.rados.0.smithi107.stdout:3454: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:50.348 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 657 (ObjNum 3453 snap 0 seq_num 3453) dirty exists 2020-06-12T02:37:50.348 INFO:tasks.rados.rados.0.smithi107.stdout:3454: left oid 1 (ObjNum 3453 snap 0 seq_num 3453) 2020-06-12T02:37:50.348 INFO:tasks.rados.rados.0.smithi107.stdout:3456: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:50.348 INFO:tasks.rados.rados.0.smithi107.stdout:3456: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:50.349 INFO:tasks.rados.rados.0.smithi107.stdout:3456: finishing write tid 4 to smithi10729407-14 2020-06-12T02:37:50.350 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 596 (ObjNum 3455 snap 0 seq_num 3455) dirty exists 2020-06-12T02:37:50.350 INFO:tasks.rados.rados.0.smithi107.stdout:3456: left oid 14 (ObjNum 3455 snap 0 seq_num 3455) 2020-06-12T02:37:50.353 INFO:tasks.rados.rados.0.smithi107.stdout:3454: done (2 left) 2020-06-12T02:37:50.353 INFO:tasks.rados.rados.0.smithi107.stdout:3456: done (1 left) 2020-06-12T02:37:50.354 INFO:tasks.rados.rados.0.smithi107.stdout:3459: done (0 left) 2020-06-12T02:37:50.354 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:37:50.354 INFO:tasks.rados.rados.0.smithi107.stdout:3460: seq_num 3459 ranges {2244608=335872} 2020-06-12T02:37:50.358 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:50.356+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:50.367 INFO:tasks.rados.rados.0.smithi107.stdout:3460: writing smithi10729407-20 from 2244608 to 2580480 tid 1 2020-06-12T02:37:50.367 INFO:tasks.rados.rados.0.smithi107.stdout:3461: delete oid 13 current snap is 0 2020-06-12T02:37:50.370 INFO:tasks.rados.rados.0.smithi107.stdout:3461: done (1 left) 2020-06-12T02:37:50.371 INFO:tasks.rados.rados.0.smithi107.stdout:3462: delete oid 8 current snap is 0 2020-06-12T02:37:50.371 INFO:tasks.rados.rados.0.smithi107.stdout:3460: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:50.372 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:50.372 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:50.373 INFO:tasks.rados.rados.0.smithi107.stdout:3460: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:50.375 INFO:tasks.rados.rados.0.smithi107.stdout:3460: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:50.375 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 259 (ObjNum 3459 snap 0 seq_num 3459) dirty exists 2020-06-12T02:37:50.375 INFO:tasks.rados.rados.0.smithi107.stdout:3460: left oid 20 (ObjNum 3459 snap 0 seq_num 3459) 2020-06-12T02:37:50.375 INFO:tasks.rados.rados.0.smithi107.stdout:3460: done (1 left) 2020-06-12T02:37:50.376 INFO:tasks.rados.rados.0.smithi107.stdout:3462: done (0 left) 2020-06-12T02:37:50.376 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:37:50.376 INFO:tasks.rados.rados.0.smithi107.stdout:3463: seq_num 3462 ranges {0=385024} 2020-06-12T02:37:50.377 INFO:tasks.rados.rados.0.smithi107.stdout:3463: writing smithi10729407-7 from 0 to 385024 tid 1 2020-06-12T02:37:50.377 INFO:tasks.rados.rados.0.smithi107.stdout:3464: delete oid 6 current snap is 0 2020-06-12T02:37:50.379 INFO:tasks.rados.rados.0.smithi107.stdout:3464: done (1 left) 2020-06-12T02:37:50.379 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:37:50.379 INFO:tasks.rados.rados.0.smithi107.stdout:3465: seq_num 3464 ranges {2473984=417792} 2020-06-12T02:37:50.390 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-06-12T02:37:50.395 INFO:tasks.rados.rados.0.smithi107.stdout:3465: writing smithi10729407-5 from 2473984 to 2891776 tid 1 2020-06-12T02:37:50.396 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:37:50.396 INFO:tasks.rados.rados.0.smithi107.stdout:3466: seq_num 3465 ranges {0=434176} 2020-06-12T02:37:50.398 INFO:tasks.rados.rados.0.smithi107.stdout:3466: writing smithi10729407-13 from 0 to 434176 tid 1 2020-06-12T02:37:50.398 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:37:50.398 INFO:tasks.rados.rados.0.smithi107.stdout:3467: seq_num 3466 ranges {5898240=450560} 2020-06-12T02:37:50.434 INFO:tasks.rados.rados.0.smithi107.stdout:3467: writing smithi10729407-11 from 5898240 to 6348800 tid 1 2020-06-12T02:37:50.434 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:37:50.435 INFO:tasks.rados.rados.0.smithi107.stdout:3468: seq_num 3467 ranges {2580480=475136} 2020-06-12T02:37:50.450 INFO:tasks.rados.rados.0.smithi107.stdout:3468: writing smithi10729407-20 from 2580480 to 3055616 tid 1 2020-06-12T02:37:50.451 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:37:50.451 INFO:tasks.rados.rados.0.smithi107.stdout:3463: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:50.451 INFO:tasks.rados.rados.0.smithi107.stdout:3469: seq_num 3468 ranges {0=491520} 2020-06-12T02:37:50.453 INFO:tasks.rados.rados.0.smithi107.stdout:3469: writing smithi10729407-17 from 0 to 491520 tid 1 2020-06-12T02:37:50.453 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:37:50.454 INFO:tasks.rados.rados.0.smithi107.stdout:3470: seq_num 3469 ranges {2203648=507904} 2020-06-12T02:37:50.459 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:37:50.457+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:37:50.468 INFO:tasks.rados.rados.0.smithi107.stdout:3470: writing smithi10729407-4 from 2203648 to 2711552 tid 1 2020-06-12T02:37:50.468 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:37:50.469 INFO:tasks.rados.rados.0.smithi107.stdout:3471: seq_num 3470 ranges {270336=524288} 2020-06-12T02:37:50.470 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd set nodeep-scrub 2020-06-12T02:37:50.476 INFO:tasks.rados.rados.0.smithi107.stdout:3471: writing smithi10729407-14 from 270336 to 794624 tid 1 2020-06-12T02:37:50.476 INFO:tasks.rados.rados.0.smithi107.stdout: waiting on 8 2020-06-12T02:37:50.476 INFO:tasks.rados.rados.0.smithi107.stdout:3463: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:50.477 INFO:tasks.rados.rados.0.smithi107.stdout:3463: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:50.477 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 668 (ObjNum 3462 snap 0 seq_num 3462) dirty exists 2020-06-12T02:37:50.477 INFO:tasks.rados.rados.0.smithi107.stdout:3463: left oid 7 (ObjNum 3462 snap 0 seq_num 3462) 2020-06-12T02:37:50.477 INFO:tasks.rados.rados.0.smithi107.stdout:3465: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:50.477 INFO:tasks.rados.rados.0.smithi107.stdout:3465: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:50.478 INFO:tasks.rados.rados.0.smithi107.stdout:3465: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:50.478 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 301 (ObjNum 3464 snap 0 seq_num 3464) dirty exists 2020-06-12T02:37:50.478 INFO:tasks.rados.rados.0.smithi107.stdout:3465: left oid 5 (ObjNum 3464 snap 0 seq_num 3464) 2020-06-12T02:37:50.478 INFO:tasks.rados.rados.0.smithi107.stdout:3466: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:50.478 INFO:tasks.rados.rados.0.smithi107.stdout:3466: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:50.479 INFO:tasks.rados.rados.0.smithi107.stdout:3466: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:50.480 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 670 (ObjNum 3465 snap 0 seq_num 3465) dirty exists 2020-06-12T02:37:50.480 INFO:tasks.rados.rados.0.smithi107.stdout:3466: left oid 13 (ObjNum 3465 snap 0 seq_num 3465) 2020-06-12T02:37:50.480 INFO:tasks.rados.rados.0.smithi107.stdout:3467: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:50.481 INFO:tasks.rados.rados.0.smithi107.stdout:3463: done (7 left) 2020-06-12T02:37:50.481 INFO:tasks.rados.rados.0.smithi107.stdout:3465: done (6 left) 2020-06-12T02:37:50.481 INFO:tasks.rados.rados.0.smithi107.stdout:3466: done (5 left) 2020-06-12T02:37:50.481 INFO:tasks.rados.rados.0.smithi107.stdout:3472: delete oid 7 current snap is 0 2020-06-12T02:37:50.481 INFO:tasks.rados.rados.0.smithi107.stdout:3467: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:50.482 INFO:tasks.rados.rados.0.smithi107.stdout:3467: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:50.482 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 598 (ObjNum 3466 snap 0 seq_num 3466) dirty exists 2020-06-12T02:37:50.482 INFO:tasks.rados.rados.0.smithi107.stdout:3467: left oid 11 (ObjNum 3466 snap 0 seq_num 3466) 2020-06-12T02:37:50.482 INFO:tasks.rados.rados.0.smithi107.stdout:3468: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:50.482 INFO:tasks.rados.rados.0.smithi107.stdout:3469: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:50.483 INFO:tasks.rados.rados.0.smithi107.stdout:3469: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:50.483 INFO:tasks.rados.rados.0.smithi107.stdout:3468: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:50.483 INFO:tasks.rados.rados.0.smithi107.stdout:3469: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:50.483 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 646 (ObjNum 3468 snap 0 seq_num 3468) dirty exists 2020-06-12T02:37:50.483 INFO:tasks.rados.rados.0.smithi107.stdout:3469: left oid 17 (ObjNum 3468 snap 0 seq_num 3468) 2020-06-12T02:37:50.483 INFO:tasks.rados.rados.0.smithi107.stdout:3468: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:50.484 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 261 (ObjNum 3467 snap 0 seq_num 3467) dirty exists 2020-06-12T02:37:50.484 INFO:tasks.rados.rados.0.smithi107.stdout:3468: left oid 20 (ObjNum 3467 snap 0 seq_num 3467) 2020-06-12T02:37:50.484 INFO:tasks.rados.rados.0.smithi107.stdout:3470: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:50.485 INFO:tasks.rados.rados.0.smithi107.stdout:3467: done (5 left) 2020-06-12T02:37:50.485 INFO:tasks.rados.rados.0.smithi107.stdout:3468: done (4 left) 2020-06-12T02:37:50.486 INFO:tasks.rados.rados.0.smithi107.stdout:3469: done (3 left) 2020-06-12T02:37:50.486 INFO:tasks.rados.rados.0.smithi107.stdout:3472: done (2 left) 2020-06-12T02:37:50.486 INFO:tasks.rados.rados.0.smithi107.stdout:3473: delete oid 19 current snap is 0 2020-06-12T02:37:50.486 INFO:tasks.rados.rados.0.smithi107.stdout:3470: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:50.486 INFO:tasks.rados.rados.0.smithi107.stdout:3471: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:50.487 INFO:tasks.rados.rados.0.smithi107.stdout:3473: done (2 left) 2020-06-12T02:37:50.487 INFO:tasks.rados.rados.0.smithi107.stdout:3474: delete oid 1 current snap is 0 2020-06-12T02:37:50.487 INFO:tasks.rados.rados.0.smithi107.stdout:3470: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:50.487 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 648 (ObjNum 3469 snap 0 seq_num 3469) dirty exists 2020-06-12T02:37:50.487 INFO:tasks.rados.rados.0.smithi107.stdout:3470: left oid 4 (ObjNum 3469 snap 0 seq_num 3469) 2020-06-12T02:37:50.487 INFO:tasks.rados.rados.0.smithi107.stdout:3471: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:50.488 INFO:tasks.rados.rados.0.smithi107.stdout:3471: finishing write tid 4 to smithi10729407-14 2020-06-12T02:37:50.488 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 600 (ObjNum 3470 snap 0 seq_num 3470) dirty exists 2020-06-12T02:37:50.488 INFO:tasks.rados.rados.0.smithi107.stdout:3471: left oid 14 (ObjNum 3470 snap 0 seq_num 3470) 2020-06-12T02:37:50.488 INFO:tasks.rados.rados.0.smithi107.stdout:3470: done (2 left) 2020-06-12T02:37:50.488 INFO:tasks.rados.rados.0.smithi107.stdout:3471: done (1 left) 2020-06-12T02:37:50.489 INFO:tasks.rados.rados.0.smithi107.stdout:3474: done (0 left) 2020-06-12T02:37:50.489 INFO:tasks.rados.rados.0.smithi107.stdout:3475: delete oid 15 current snap is 0 2020-06-12T02:37:50.489 INFO:tasks.rados.rados.0.smithi107.stdout:3475: done (0 left) 2020-06-12T02:37:50.490 INFO:tasks.rados.rados.0.smithi107.stdout:3476: delete oid 5 current snap is 0 2020-06-12T02:37:50.490 INFO:tasks.rados.rados.0.smithi107.stdout:3476: done (0 left) 2020-06-12T02:37:50.490 INFO:tasks.rados.rados.0.smithi107.stdout:3477: delete oid 15 current snap is 0 2020-06-12T02:37:50.490 INFO:tasks.rados.rados.0.smithi107.stdout:3477: done (0 left) 2020-06-12T02:37:50.491 INFO:tasks.rados.rados.0.smithi107.stdout:3478: delete oid 13 current snap is 0 2020-06-12T02:37:50.493 INFO:tasks.rados.rados.0.smithi107.stdout:3478: done (0 left) 2020-06-12T02:37:50.493 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:37:50.494 INFO:tasks.rados.rados.0.smithi107.stdout:3479: seq_num 3478 ranges {0=655360} 2020-06-12T02:37:50.497 INFO:tasks.rados.rados.0.smithi107.stdout:3479: writing smithi10729407-19 from 0 to 655360 tid 1 2020-06-12T02:37:50.497 INFO:tasks.rados.rados.0.smithi107.stdout:3480: delete oid 5 current snap is 0 2020-06-12T02:37:50.499 INFO:tasks.rados.rados.0.smithi107.stdout:3480: done (1 left) 2020-06-12T02:37:50.499 INFO:tasks.rados.rados.0.smithi107.stdout:3481: delete oid 7 current snap is 0 2020-06-12T02:37:50.501 INFO:tasks.rados.rados.0.smithi107.stdout:3481: done (1 left) 2020-06-12T02:37:50.501 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:37:50.501 INFO:tasks.rados.rados.0.smithi107.stdout:3482: seq_num 3481 ranges {3055616=704512} 2020-06-12T02:37:50.523 INFO:tasks.rados.rados.0.smithi107.stdout:3482: writing smithi10729407-20 from 3055616 to 3760128 tid 1 2020-06-12T02:37:50.523 INFO:tasks.rados.rados.0.smithi107.stdout:3483: delete oid 4 current snap is 0 2020-06-12T02:37:50.523 INFO:tasks.rados.rados.0.smithi107.stdout:3479: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:50.523 INFO:tasks.rados.rados.0.smithi107.stdout:3479: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:50.524 INFO:tasks.rados.rados.0.smithi107.stdout:3479: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:50.524 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 575 (ObjNum 3478 snap 0 seq_num 3478) dirty exists 2020-06-12T02:37:50.524 INFO:tasks.rados.rados.0.smithi107.stdout:3479: left oid 19 (ObjNum 3478 snap 0 seq_num 3478) 2020-06-12T02:37:50.524 INFO:tasks.rados.rados.0.smithi107.stdout:3479: done (2 left) 2020-06-12T02:37:50.525 INFO:tasks.rados.rados.0.smithi107.stdout:3483: done (1 left) 2020-06-12T02:37:50.525 INFO:tasks.rados.rados.0.smithi107.stdout:3484: delete oid 14 current snap is 0 2020-06-12T02:37:50.530 INFO:tasks.rados.rados.0.smithi107.stdout:3484: done (1 left) 2020-06-12T02:37:50.530 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:37:50.531 INFO:tasks.rados.rados.0.smithi107.stdout:3485: seq_num 3484 ranges {6348800=753664} 2020-06-12T02:37:50.534 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:50.534 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:50.543 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-06-12T02:37:50.567 INFO:tasks.rados.rados.0.smithi107.stdout:3485: writing smithi10729407-11 from 6348800 to 7102464 tid 1 2020-06-12T02:37:50.568 INFO:tasks.rados.rados.0.smithi107.stdout:3486: delete oid 19 current snap is 0 2020-06-12T02:37:50.568 INFO:tasks.rados.rados.0.smithi107.stdout:3482: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:50.568 INFO:tasks.rados.rados.0.smithi107.stdout:3482: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:50.568 INFO:tasks.rados.rados.0.smithi107.stdout:3482: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:50.568 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 263 (ObjNum 3481 snap 0 seq_num 3481) dirty exists 2020-06-12T02:37:50.569 INFO:tasks.rados.rados.0.smithi107.stdout:3482: left oid 20 (ObjNum 3481 snap 0 seq_num 3481) 2020-06-12T02:37:50.569 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:37:50.565+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:37:50.570 INFO:tasks.rados.rados.0.smithi107.stdout:3482: done (2 left) 2020-06-12T02:37:50.570 INFO:tasks.rados.rados.0.smithi107.stdout:3486: done (1 left) 2020-06-12T02:37:50.570 INFO:tasks.rados.rados.0.smithi107.stdout:3487: delete oid 18 current snap is 0 2020-06-12T02:37:50.574 INFO:tasks.rados.rados.0.smithi107.stdout:3485: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:50.577 INFO:tasks.rados.rados.0.smithi107.stdout:3487: done (1 left) 2020-06-12T02:37:50.578 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:37:50.578 INFO:tasks.rados.rados.0.smithi107.stdout:3488: seq_num 3487 ranges {0=409600,409600=401408} 2020-06-12T02:37:50.580 INFO:tasks.rados.rados.0.smithi107.stdout:3488: writing smithi10729407-18 from 0 to 409600 tid 1 2020-06-12T02:37:50.583 INFO:tasks.rados.rados.0.smithi107.stdout:3488: writing smithi10729407-18 from 409600 to 811008 tid 2 2020-06-12T02:37:50.583 INFO:tasks.rados.rados.0.smithi107.stdout:3489: delete oid 13 current snap is 0 2020-06-12T02:37:50.584 INFO:tasks.rados.rados.0.smithi107.stdout:3485: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:50.584 INFO:tasks.rados.rados.0.smithi107.stdout:3485: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:50.584 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 603 (ObjNum 3484 snap 0 seq_num 3484) dirty exists 2020-06-12T02:37:50.584 INFO:tasks.rados.rados.0.smithi107.stdout:3485: left oid 11 (ObjNum 3484 snap 0 seq_num 3484) 2020-06-12T02:37:50.592 INFO:tasks.rados.rados.0.smithi107.stdout:3488: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:50.593 INFO:tasks.rados.rados.0.smithi107.stdout:3485: done (2 left) 2020-06-12T02:37:50.593 INFO:tasks.rados.rados.0.smithi107.stdout:3489: done (1 left) 2020-06-12T02:37:50.593 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:37:50.593 INFO:tasks.rados.rados.0.smithi107.stdout:3490: seq_num 3489 ranges {0=442368,442368=401408} 2020-06-12T02:37:50.593 INFO:tasks.rados.rados.0.smithi107.stdout:3490: writing smithi10729407-10 from 0 to 442368 tid 1 2020-06-12T02:37:50.594 INFO:tasks.rados.rados.0.smithi107.stdout:3490: writing smithi10729407-10 from 442368 to 843776 tid 2 2020-06-12T02:37:50.594 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:37:50.595 INFO:tasks.rados.rados.0.smithi107.stdout:3491: seq_num 3490 ranges {0=458752,458752=401408} 2020-06-12T02:37:50.596 INFO:tasks.rados.rados.0.smithi107.stdout:3491: writing smithi10729407-5 from 0 to 458752 tid 1 2020-06-12T02:37:50.600 INFO:tasks.rados.rados.0.smithi107.stdout:3491: writing smithi10729407-5 from 458752 to 860160 tid 2 2020-06-12T02:37:50.600 INFO:tasks.rados.rados.0.smithi107.stdout:3492: delete oid 12 current snap is 0 2020-06-12T02:37:50.600 INFO:tasks.rados.rados.0.smithi107.stdout:3488: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:50.600 INFO:tasks.rados.rados.0.smithi107.stdout:3490: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:50.602 INFO:tasks.rados.rados.0.smithi107.stdout:3488: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:50.602 INFO:tasks.rados.rados.0.smithi107.stdout:3490: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:50.602 INFO:tasks.rados.rados.0.smithi107.stdout:3491: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:50.603 INFO:tasks.rados.rados.0.smithi107.stdout:3490: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:50.603 INFO:tasks.rados.rados.0.smithi107.stdout:3492: done (3 left) 2020-06-12T02:37:50.604 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:37:50.604 INFO:tasks.rados.rados.0.smithi107.stdout:3493: seq_num 3492 ranges {0=491520,491520=401408} 2020-06-12T02:37:50.606 INFO:tasks.rados.rados.0.smithi107.stdout:3493: writing smithi10729407-14 from 0 to 491520 tid 1 2020-06-12T02:37:50.609 INFO:tasks.rados.rados.0.smithi107.stdout:3493: writing smithi10729407-14 from 491520 to 892928 tid 2 2020-06-12T02:37:50.609 INFO:tasks.rados.rados.0.smithi107.stdout:3494: delete oid 2 current snap is 0 2020-06-12T02:37:50.610 INFO:tasks.rados.rados.0.smithi107.stdout:3490: finishing write tid 5 to smithi10729407-10 2020-06-12T02:37:50.610 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 341 (ObjNum 3489 snap 0 seq_num 3489) dirty exists 2020-06-12T02:37:50.610 INFO:tasks.rados.rados.0.smithi107.stdout:3490: left oid 10 (ObjNum 3489 snap 0 seq_num 3489) 2020-06-12T02:37:50.610 INFO:tasks.rados.rados.0.smithi107.stdout:3488: finishing write tid 5 to smithi10729407-18 2020-06-12T02:37:50.611 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 615 (ObjNum 3487 snap 0 seq_num 3487) dirty exists 2020-06-12T02:37:50.611 INFO:tasks.rados.rados.0.smithi107.stdout:3488: left oid 18 (ObjNum 3487 snap 0 seq_num 3487) 2020-06-12T02:37:50.611 INFO:tasks.rados.rados.0.smithi107.stdout:3491: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:50.611 INFO:tasks.rados.rados.0.smithi107.stdout:3491: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:50.611 INFO:tasks.rados.rados.0.smithi107.stdout:3491: finishing write tid 5 to smithi10729407-5 2020-06-12T02:37:50.612 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 306 (ObjNum 3490 snap 0 seq_num 3490) dirty exists 2020-06-12T02:37:50.612 INFO:tasks.rados.rados.0.smithi107.stdout:3491: left oid 5 (ObjNum 3490 snap 0 seq_num 3490) 2020-06-12T02:37:50.612 INFO:tasks.rados.rados.0.smithi107.stdout:3488: done (4 left) 2020-06-12T02:37:50.613 INFO:tasks.rados.rados.0.smithi107.stdout:3490: done (3 left) 2020-06-12T02:37:50.613 INFO:tasks.rados.rados.0.smithi107.stdout:3491: done (2 left) 2020-06-12T02:37:50.613 INFO:tasks.rados.rados.0.smithi107.stdout:3494: done (1 left) 2020-06-12T02:37:50.613 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:37:50.614 INFO:tasks.rados.rados.0.smithi107.stdout:3495: seq_num 3494 ranges {0=524288,524288=401408} 2020-06-12T02:37:50.615 INFO:tasks.rados.rados.0.smithi107.stdout:3495: writing smithi10729407-19 from 0 to 524288 tid 1 2020-06-12T02:37:50.618 INFO:tasks.rados.rados.0.smithi107.stdout:3495: writing smithi10729407-19 from 524288 to 925696 tid 2 2020-06-12T02:37:50.618 INFO:tasks.rados.rados.0.smithi107.stdout:3496: delete oid 8 current snap is 0 2020-06-12T02:37:50.618 INFO:tasks.rados.rados.0.smithi107.stdout:3493: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:50.621 INFO:tasks.rados.rados.0.smithi107.stdout:3496: done (2 left) 2020-06-12T02:37:50.621 INFO:tasks.rados.rados.0.smithi107.stdout:3497: delete oid 11 current snap is 0 2020-06-12T02:37:50.623 INFO:tasks.rados.rados.0.smithi107.stdout:3493: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:50.623 INFO:tasks.rados.rados.0.smithi107.stdout:3495: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:50.623 INFO:tasks.rados.rados.0.smithi107.stdout:3493: finishing write tid 4 to smithi10729407-14 2020-06-12T02:37:50.624 INFO:tasks.rados.rados.0.smithi107.stdout:3493: finishing write tid 5 to smithi10729407-14 2020-06-12T02:37:50.624 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 606 (ObjNum 3492 snap 0 seq_num 3492) dirty exists 2020-06-12T02:37:50.625 INFO:tasks.rados.rados.0.smithi107.stdout:3493: left oid 14 (ObjNum 3492 snap 0 seq_num 3492) 2020-06-12T02:37:50.627 INFO:tasks.rados.rados.0.smithi107.stdout:3495: finishing write tid 2 to smithi10729407-19 2020-06-12T02:37:50.628 INFO:tasks.rados.rados.0.smithi107.stdout:3495: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:50.629 INFO:tasks.rados.rados.0.smithi107.stdout:3495: finishing write tid 5 to smithi10729407-19 2020-06-12T02:37:50.630 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 580 (ObjNum 3494 snap 0 seq_num 3494) dirty exists 2020-06-12T02:37:50.630 INFO:tasks.rados.rados.0.smithi107.stdout:3495: left oid 19 (ObjNum 3494 snap 0 seq_num 3494) 2020-06-12T02:37:50.630 INFO:tasks.rados.rados.0.smithi107.stdout:3493: done (2 left) 2020-06-12T02:37:50.631 INFO:tasks.rados.rados.0.smithi107.stdout:3495: done (1 left) 2020-06-12T02:37:50.631 INFO:tasks.rados.rados.0.smithi107.stdout:3497: done (0 left) 2020-06-12T02:37:50.631 INFO:tasks.rados.rados.0.smithi107.stdout:3498: delete oid 18 current snap is 0 2020-06-12T02:37:50.632 INFO:tasks.rados.rados.0.smithi107.stdout:3498: done (0 left) 2020-06-12T02:37:50.632 INFO:tasks.rados.rados.0.smithi107.stdout:3499: delete oid 7 current snap is 0 2020-06-12T02:37:50.635 INFO:tasks.rados.rados.0.smithi107.stdout:3499: done (0 left) 2020-06-12T02:37:50.635 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:37:50.635 INFO:tasks.rados.rados.0.smithi107.stdout:3500: seq_num 3499 ranges {0=606208,606208=401408} 2020-06-12T02:37:50.639 INFO:tasks.rados.rados.0.smithi107.stdout:3500: writing smithi10729407-18 from 0 to 606208 tid 1 2020-06-12T02:37:50.642 INFO:tasks.rados.rados.0.smithi107.stdout:3500: writing smithi10729407-18 from 606208 to 1007616 tid 2 2020-06-12T02:37:50.643 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:37:50.643 INFO:tasks.rados.rados.0.smithi107.stdout:3501: seq_num 3500 ranges {2392064=622592,3014656=401408} 2020-06-12T02:37:50.661 INFO:tasks.rados.rados.0.smithi107.stdout:3501: writing smithi10729407-9 from 2392064 to 3014656 tid 1 2020-06-12T02:37:50.661 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:50.659+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:50.664 INFO:tasks.rados.rados.0.smithi107.stdout:3501: writing smithi10729407-9 from 3014656 to 3416064 tid 2 2020-06-12T02:37:50.664 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:50.664 INFO:tasks.rados.rados.0.smithi107.stdout:3502: seq_num 3501 ranges {0=638976,638976=401408} 2020-06-12T02:37:50.674 INFO:tasks.rados.rados.0.smithi107.stdout:3502: writing smithi10729407-2 from 0 to 638976 tid 1 2020-06-12T02:37:50.675 INFO:tasks.rados.rados.0.smithi107.stdout:3502: writing smithi10729407-2 from 638976 to 1040384 tid 2 2020-06-12T02:37:50.675 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:37:50.675 INFO:tasks.rados.rados.0.smithi107.stdout:3503: seq_num 3502 ranges {3760128=655360,4415488=401408} 2020-06-12T02:37:50.697 INFO:tasks.rados.rados.0.smithi107.stdout:3503: writing smithi10729407-20 from 3760128 to 4415488 tid 1 2020-06-12T02:37:50.701 INFO:tasks.rados.rados.0.smithi107.stdout:3503: writing smithi10729407-20 from 4415488 to 4816896 tid 2 2020-06-12T02:37:50.701 INFO:tasks.rados.rados.0.smithi107.stdout:3504: delete oid 11 current snap is 0 2020-06-12T02:37:50.701 INFO:tasks.rados.rados.0.smithi107.stdout:3500: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:50.701 INFO:tasks.rados.rados.0.smithi107.stdout:3500: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:50.701 INFO:tasks.rados.rados.0.smithi107.stdout:3500: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:50.702 INFO:tasks.rados.rados.0.smithi107.stdout:3500: finishing write tid 5 to smithi10729407-18 2020-06-12T02:37:50.702 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 619 (ObjNum 3499 snap 0 seq_num 3499) dirty exists 2020-06-12T02:37:50.702 INFO:tasks.rados.rados.0.smithi107.stdout:3500: left oid 18 (ObjNum 3499 snap 0 seq_num 3499) 2020-06-12T02:37:50.702 INFO:tasks.rados.rados.0.smithi107.stdout:3501: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:50.702 INFO:tasks.rados.rados.0.smithi107.stdout:3501: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:50.703 INFO:tasks.rados.rados.0.smithi107.stdout:3501: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:50.703 INFO:tasks.rados.rados.0.smithi107.stdout:3501: finishing write tid 5 to smithi10729407-9 2020-06-12T02:37:50.703 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 622 (ObjNum 3500 snap 0 seq_num 3500) dirty exists 2020-06-12T02:37:50.703 INFO:tasks.rados.rados.0.smithi107.stdout:3501: left oid 9 (ObjNum 3500 snap 0 seq_num 3500) 2020-06-12T02:37:50.703 INFO:tasks.rados.rados.0.smithi107.stdout:3502: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:50.704 INFO:tasks.rados.rados.0.smithi107.stdout:3502: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:50.704 INFO:tasks.rados.rados.0.smithi107.stdout:3502: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:50.704 INFO:tasks.rados.rados.0.smithi107.stdout:3502: finishing write tid 5 to smithi10729407-2 2020-06-12T02:37:50.704 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 358 (ObjNum 3501 snap 0 seq_num 3501) dirty exists 2020-06-12T02:37:50.704 INFO:tasks.rados.rados.0.smithi107.stdout:3502: left oid 2 (ObjNum 3501 snap 0 seq_num 3501) 2020-06-12T02:37:50.705 INFO:tasks.rados.rados.0.smithi107.stdout:3503: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:50.706 INFO:tasks.rados.rados.0.smithi107.stdout:3500: done (4 left) 2020-06-12T02:37:50.706 INFO:tasks.rados.rados.0.smithi107.stdout:3501: done (3 left) 2020-06-12T02:37:50.706 INFO:tasks.rados.rados.0.smithi107.stdout:3502: done (2 left) 2020-06-12T02:37:50.706 INFO:tasks.rados.rados.0.smithi107.stdout:3504: done (1 left) 2020-06-12T02:37:50.706 INFO:tasks.rados.rados.0.smithi107.stdout:3505: delete oid 6 current snap is 0 2020-06-12T02:37:50.711 INFO:tasks.rados.rados.0.smithi107.stdout:3505: done (1 left) 2020-06-12T02:37:50.712 INFO:tasks.rados.rados.0.smithi107.stdout:3506: delete oid 7 current snap is 0 2020-06-12T02:37:50.712 INFO:tasks.rados.rados.0.smithi107.stdout:3503: finishing write tid 2 to smithi10729407-20 2020-06-12T02:37:50.712 INFO:tasks.rados.rados.0.smithi107.stdout:3506: done (1 left) 2020-06-12T02:37:50.712 INFO:tasks.rados.rados.0.smithi107.stdout:3507: delete oid 8 current snap is 0 2020-06-12T02:37:50.712 INFO:tasks.rados.rados.0.smithi107.stdout:3503: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:50.713 INFO:tasks.rados.rados.0.smithi107.stdout:3503: finishing write tid 5 to smithi10729407-20 2020-06-12T02:37:50.713 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 266 (ObjNum 3502 snap 0 seq_num 3502) dirty exists 2020-06-12T02:37:50.713 INFO:tasks.rados.rados.0.smithi107.stdout:3503: left oid 20 (ObjNum 3502 snap 0 seq_num 3502) 2020-06-12T02:37:50.714 INFO:tasks.rados.rados.0.smithi107.stdout:3503: done (1 left) 2020-06-12T02:37:50.714 INFO:tasks.rados.rados.0.smithi107.stdout:3507: done (0 left) 2020-06-12T02:37:50.714 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:37:50.715 INFO:tasks.rados.rados.0.smithi107.stdout:3508: seq_num 3507 ranges {0=745472,745472=401408} 2020-06-12T02:37:50.719 INFO:tasks.rados.rados.0.smithi107.stdout:3508: writing smithi10729407-11 from 0 to 745472 tid 1 2020-06-12T02:37:50.722 INFO:tasks.rados.rados.0.smithi107.stdout:3508: writing smithi10729407-11 from 745472 to 1146880 tid 2 2020-06-12T02:37:50.722 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:37:50.722 INFO:tasks.rados.rados.0.smithi107.stdout:3509: seq_num 3508 ranges {2375680=761856,3137536=401408} 2020-06-12T02:37:50.740 INFO:tasks.rados.rados.0.smithi107.stdout:3509: writing smithi10729407-16 from 2375680 to 3137536 tid 1 2020-06-12T02:37:50.743 INFO:tasks.rados.rados.0.smithi107.stdout:3509: writing smithi10729407-16 from 3137536 to 3538944 tid 2 2020-06-12T02:37:50.743 INFO:tasks.rados.rados.0.smithi107.stdout:3510: delete oid 4 current snap is 0 2020-06-12T02:37:50.743 INFO:tasks.rados.rados.0.smithi107.stdout:3508: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:50.743 INFO:tasks.rados.rados.0.smithi107.stdout:3508: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:50.743 INFO:tasks.rados.rados.0.smithi107.stdout:3508: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:50.744 INFO:tasks.rados.rados.0.smithi107.stdout:3508: finishing write tid 5 to smithi10729407-11 2020-06-12T02:37:50.744 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 611 (ObjNum 3507 snap 0 seq_num 3507) dirty exists 2020-06-12T02:37:50.744 INFO:tasks.rados.rados.0.smithi107.stdout:3508: left oid 11 (ObjNum 3507 snap 0 seq_num 3507) 2020-06-12T02:37:50.745 INFO:tasks.rados.rados.0.smithi107.stdout:3508: done (2 left) 2020-06-12T02:37:50.745 INFO:tasks.rados.rados.0.smithi107.stdout:3510: done (1 left) 2020-06-12T02:37:50.745 INFO:tasks.rados.rados.0.smithi107.stdout:3511: delete oid 19 current snap is 0 2020-06-12T02:37:50.748 INFO:tasks.rados.rados.0.smithi107.stdout:3511: done (1 left) 2020-06-12T02:37:50.748 INFO:tasks.rados.rados.0.smithi107.stdout:3512: delete oid 5 current snap is 0 2020-06-12T02:37:50.750 INFO:tasks.rados.rados.0.smithi107.stdout:3509: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:50.752 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:50.752 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:50.754 INFO:tasks.rados.rados.0.smithi107.stdout:3509: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:50.754 INFO:tasks.rados.rados.0.smithi107.stdout:3512: done (1 left) 2020-06-12T02:37:50.754 INFO:tasks.rados.rados.0.smithi107.stdout:3513: delete oid 19 current snap is 0 2020-06-12T02:37:50.755 INFO:tasks.rados.rados.0.smithi107.stdout:3509: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:50.757 INFO:tasks.rados.rados.0.smithi107.stdout:3509: finishing write tid 5 to smithi10729407-16 2020-06-12T02:37:50.757 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 620 (ObjNum 3508 snap 0 seq_num 3508) dirty exists 2020-06-12T02:37:50.757 INFO:tasks.rados.rados.0.smithi107.stdout:3509: left oid 16 (ObjNum 3508 snap 0 seq_num 3508) 2020-06-12T02:37:50.758 INFO:tasks.rados.rados.0.smithi107.stdout:3509: done (1 left) 2020-06-12T02:37:50.758 INFO:tasks.rados.rados.0.smithi107.stdout:3513: done (0 left) 2020-06-12T02:37:50.758 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:37:50.758 INFO:tasks.rados.rados.0.smithi107.stdout:3514: seq_num 3513 ranges {1040384=442368,1482752=614400,2097152=188416} 2020-06-12T02:37:50.762 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-06-12T02:37:50.763 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:37:50.759+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:37:50.766 INFO:tasks.rados.rados.0.smithi107.stdout:3514: writing smithi10729407-2 from 1040384 to 1482752 tid 1 2020-06-12T02:37:50.769 INFO:tasks.rados.rados.0.smithi107.stdout:3514: writing smithi10729407-2 from 1482752 to 2097152 tid 2 2020-06-12T02:37:50.771 INFO:tasks.rados.rados.0.smithi107.stdout:3514: writing smithi10729407-2 from 2097152 to 2285568 tid 3 2020-06-12T02:37:50.771 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:37:50.771 INFO:tasks.rados.rados.0.smithi107.stdout:3515: seq_num 3514 ranges {1146880=458752,1605632=499712,2105344=303104} 2020-06-12T02:37:50.780 INFO:tasks.rados.rados.0.smithi107.stdout:3515: writing smithi10729407-11 from 1146880 to 1605632 tid 1 2020-06-12T02:37:50.784 INFO:tasks.rados.rados.0.smithi107.stdout:3515: writing smithi10729407-11 from 1605632 to 2105344 tid 2 2020-06-12T02:37:50.786 INFO:tasks.rados.rados.0.smithi107.stdout:3515: writing smithi10729407-11 from 2105344 to 2408448 tid 3 2020-06-12T02:37:50.786 INFO:tasks.rados.rados.0.smithi107.stdout:3516: delete oid 18 current snap is 0 2020-06-12T02:37:50.786 INFO:tasks.rados.rados.0.smithi107.stdout:3514: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:50.786 INFO:tasks.rados.rados.0.smithi107.stdout:3514: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:50.787 INFO:tasks.rados.rados.0.smithi107.stdout:3514: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:50.787 INFO:tasks.rados.rados.0.smithi107.stdout:3514: finishing write tid 5 to smithi10729407-2 2020-06-12T02:37:50.787 INFO:tasks.rados.rados.0.smithi107.stdout:3514: finishing write tid 6 to smithi10729407-2 2020-06-12T02:37:50.787 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 362 (ObjNum 3513 snap 0 seq_num 3513) dirty exists 2020-06-12T02:37:50.787 INFO:tasks.rados.rados.0.smithi107.stdout:3514: left oid 2 (ObjNum 3513 snap 0 seq_num 3513) 2020-06-12T02:37:50.788 INFO:tasks.rados.rados.0.smithi107.stdout:3515: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:50.788 INFO:tasks.rados.rados.0.smithi107.stdout:3514: done (2 left) 2020-06-12T02:37:50.789 INFO:tasks.rados.rados.0.smithi107.stdout:3516: done (1 left) 2020-06-12T02:37:50.789 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:37:50.789 INFO:tasks.rados.rados.0.smithi107.stdout:3517: seq_num 3516 ranges {843776=491520,1335296=671744,2007040=131072} 2020-06-12T02:37:50.799 INFO:tasks.rados.rados.0.smithi107.stdout:3517: writing smithi10729407-10 from 843776 to 1335296 tid 1 2020-06-12T02:37:50.801 INFO:tasks.rados.rados.0.smithi107.stdout:3517: writing smithi10729407-10 from 1335296 to 2007040 tid 2 2020-06-12T02:37:50.802 INFO:tasks.rados.rados.0.smithi107.stdout:3517: writing smithi10729407-10 from 2007040 to 2138112 tid 3 2020-06-12T02:37:50.802 INFO:tasks.rados.rados.0.smithi107.stdout:3518: delete oid 9 current snap is 0 2020-06-12T02:37:50.803 INFO:tasks.rados.rados.0.smithi107.stdout:3515: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:50.803 INFO:tasks.rados.rados.0.smithi107.stdout:3515: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:50.803 INFO:tasks.rados.rados.0.smithi107.stdout:3515: finishing write tid 5 to smithi10729407-11 2020-06-12T02:37:50.803 INFO:tasks.rados.rados.0.smithi107.stdout:3515: finishing write tid 6 to smithi10729407-11 2020-06-12T02:37:50.803 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 615 (ObjNum 3514 snap 0 seq_num 3514) dirty exists 2020-06-12T02:37:50.804 INFO:tasks.rados.rados.0.smithi107.stdout:3515: left oid 11 (ObjNum 3514 snap 0 seq_num 3514) 2020-06-12T02:37:50.804 INFO:tasks.rados.rados.0.smithi107.stdout:3515: done (2 left) 2020-06-12T02:37:50.805 INFO:tasks.rados.rados.0.smithi107.stdout:3518: done (1 left) 2020-06-12T02:37:50.805 INFO:tasks.rados.rados.0.smithi107.stdout:3519: delete oid 11 current snap is 0 2020-06-12T02:37:50.807 INFO:tasks.rados.rados.0.smithi107.stdout:3517: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:50.814 INFO:tasks.rados.rados.0.smithi107.stdout:3519: done (1 left) 2020-06-12T02:37:50.814 INFO:tasks.rados.rados.0.smithi107.stdout:3520: delete oid 13 current snap is 0 2020-06-12T02:37:50.815 INFO:tasks.rados.rados.0.smithi107.stdout:3520: done (1 left) 2020-06-12T02:37:50.815 INFO:tasks.rados.rados.0.smithi107.stdout:3521: delete oid 15 current snap is 0 2020-06-12T02:37:50.816 INFO:tasks.rados.rados.0.smithi107.stdout:3517: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:50.816 INFO:tasks.rados.rados.0.smithi107.stdout:3517: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:50.817 INFO:tasks.rados.rados.0.smithi107.stdout:3517: finishing write tid 5 to smithi10729407-10 2020-06-12T02:37:50.819 INFO:tasks.rados.rados.0.smithi107.stdout:3517: finishing write tid 6 to smithi10729407-10 2020-06-12T02:37:50.819 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 345 (ObjNum 3516 snap 0 seq_num 3516) dirty exists 2020-06-12T02:37:50.819 INFO:tasks.rados.rados.0.smithi107.stdout:3517: left oid 10 (ObjNum 3516 snap 0 seq_num 3516) 2020-06-12T02:37:50.819 INFO:tasks.rados.rados.0.smithi107.stdout:3517: done (1 left) 2020-06-12T02:37:50.820 INFO:tasks.rados.rados.0.smithi107.stdout:3521: done (0 left) 2020-06-12T02:37:50.820 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:37:50.820 INFO:tasks.rados.rados.0.smithi107.stdout:3522: seq_num 3521 ranges {0=573440,573440=540672,1114112=262144} 2020-06-12T02:37:50.822 INFO:tasks.rados.rados.0.smithi107.stdout:3522: writing smithi10729407-15 from 0 to 573440 tid 1 2020-06-12T02:37:50.825 INFO:tasks.rados.rados.0.smithi107.stdout:3522: writing smithi10729407-15 from 573440 to 1114112 tid 2 2020-06-12T02:37:50.828 INFO:tasks.rados.rados.0.smithi107.stdout:3522: writing smithi10729407-15 from 1114112 to 1376256 tid 3 2020-06-12T02:37:50.828 INFO:tasks.rados.rados.0.smithi107.stdout:3523: delete oid 7 current snap is 0 2020-06-12T02:37:50.828 INFO:tasks.rados.rados.0.smithi107.stdout:3522: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:50.830 INFO:tasks.rados.rados.0.smithi107.stdout:3523: done (1 left) 2020-06-12T02:37:50.830 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:37:50.830 INFO:tasks.rados.rados.0.smithi107.stdout:3524: seq_num 3523 ranges {0=606208,606208=712704,1318912=90112} 2020-06-12T02:37:50.833 INFO:tasks.rados.rados.0.smithi107.stdout:3524: writing smithi10729407-13 from 0 to 606208 tid 1 2020-06-12T02:37:50.838 INFO:tasks.rados.rados.0.smithi107.stdout:3524: writing smithi10729407-13 from 606208 to 1318912 tid 2 2020-06-12T02:37:50.844 INFO:tasks.rados.rados.0.smithi107.stdout:3524: writing smithi10729407-13 from 1318912 to 1409024 tid 3 2020-06-12T02:37:50.844 INFO:tasks.rados.rados.0.smithi107.stdout:3525: delete oid 12 current snap is 0 2020-06-12T02:37:50.844 INFO:tasks.rados.rados.0.smithi107.stdout:3522: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:50.845 INFO:tasks.rados.rados.0.smithi107.stdout:3522: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:50.845 INFO:tasks.rados.rados.0.smithi107.stdout:3522: finishing write tid 5 to smithi10729407-15 2020-06-12T02:37:50.845 INFO:tasks.rados.rados.0.smithi107.stdout:3522: finishing write tid 6 to smithi10729407-15 2020-06-12T02:37:50.845 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 593 (ObjNum 3521 snap 0 seq_num 3521) dirty exists 2020-06-12T02:37:50.845 INFO:tasks.rados.rados.0.smithi107.stdout:3522: left oid 15 (ObjNum 3521 snap 0 seq_num 3521) 2020-06-12T02:37:50.846 INFO:tasks.rados.rados.0.smithi107.stdout:3522: done (2 left) 2020-06-12T02:37:50.846 INFO:tasks.rados.rados.0.smithi107.stdout:3525: done (1 left) 2020-06-12T02:37:50.846 INFO:tasks.rados.rados.0.smithi107.stdout:3526: delete oid 10 current snap is 0 2020-06-12T02:37:50.853 INFO:tasks.rados.rados.0.smithi107.stdout:3526: done (1 left) 2020-06-12T02:37:50.853 INFO:tasks.rados.rados.0.smithi107.stdout:3527: delete oid 4 current snap is 0 2020-06-12T02:37:50.853 INFO:tasks.rados.rados.0.smithi107.stdout:3524: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:50.853 INFO:tasks.rados.rados.0.smithi107.stdout:3527: done (1 left) 2020-06-12T02:37:50.854 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:37:50.854 INFO:tasks.rados.rados.0.smithi107.stdout:3528: seq_num 3527 ranges {3538944=679936,4218880=647168,4866048=155648} 2020-06-12T02:37:50.863 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:50.861+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:50.872 INFO:tasks.rados.rados.0.smithi107.stdout:3528: writing smithi10729407-16 from 3538944 to 4218880 tid 1 2020-06-12T02:37:50.876 INFO:tasks.rados.rados.0.smithi107.stdout:3528: writing smithi10729407-16 from 4218880 to 4866048 tid 2 2020-06-12T02:37:50.878 INFO:tasks.rados.rados.0.smithi107.stdout:3528: writing smithi10729407-16 from 4866048 to 5021696 tid 3 2020-06-12T02:37:50.879 INFO:tasks.rados.rados.0.smithi107.stdout:3529: delete oid 8 current snap is 0 2020-06-12T02:37:50.879 INFO:tasks.rados.rados.0.smithi107.stdout:3524: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:50.880 INFO:tasks.rados.rados.0.smithi107.stdout:3524: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:50.880 INFO:tasks.rados.rados.0.smithi107.stdout:3524: finishing write tid 5 to smithi10729407-13 2020-06-12T02:37:50.880 INFO:tasks.rados.rados.0.smithi107.stdout:3524: finishing write tid 6 to smithi10729407-13 2020-06-12T02:37:50.881 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 682 (ObjNum 3523 snap 0 seq_num 3523) dirty exists 2020-06-12T02:37:50.881 INFO:tasks.rados.rados.0.smithi107.stdout:3524: left oid 13 (ObjNum 3523 snap 0 seq_num 3523) 2020-06-12T02:37:50.887 INFO:tasks.rados.rados.0.smithi107.stdout:3528: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:50.888 INFO:tasks.rados.rados.0.smithi107.stdout:3524: done (2 left) 2020-06-12T02:37:50.888 INFO:tasks.rados.rados.0.smithi107.stdout:3529: done (1 left) 2020-06-12T02:37:50.888 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:37:50.889 INFO:tasks.rados.rados.0.smithi107.stdout:3530: seq_num 3529 ranges {491520=712704,1204224=466944,1671168=335872} 2020-06-12T02:37:50.889 INFO:tasks.rados.rados.0.smithi107.stdout:3530: writing smithi10729407-17 from 491520 to 1204224 tid 1 2020-06-12T02:37:50.890 INFO:tasks.rados.rados.0.smithi107.stdout:3530: writing smithi10729407-17 from 1204224 to 1671168 tid 2 2020-06-12T02:37:50.892 INFO:tasks.rados.rados.0.smithi107.stdout:3530: writing smithi10729407-17 from 1671168 to 2007040 tid 3 2020-06-12T02:37:50.892 INFO:tasks.rados.rados.0.smithi107.stdout:3531: delete oid 14 current snap is 0 2020-06-12T02:37:50.893 INFO:tasks.rados.rados.0.smithi107.stdout:3528: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:50.893 INFO:tasks.rados.rados.0.smithi107.stdout:3528: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:50.893 INFO:tasks.rados.rados.0.smithi107.stdout:3528: finishing write tid 5 to smithi10729407-16 2020-06-12T02:37:50.894 INFO:tasks.rados.rados.0.smithi107.stdout:3528: finishing write tid 6 to smithi10729407-16 2020-06-12T02:37:50.894 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 624 (ObjNum 3527 snap 0 seq_num 3527) dirty exists 2020-06-12T02:37:50.894 INFO:tasks.rados.rados.0.smithi107.stdout:3528: left oid 16 (ObjNum 3527 snap 0 seq_num 3527) 2020-06-12T02:37:50.895 INFO:tasks.rados.rados.0.smithi107.stdout:3530: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:50.896 INFO:tasks.rados.rados.0.smithi107.stdout:3528: done (2 left) 2020-06-12T02:37:50.896 INFO:tasks.rados.rados.0.smithi107.stdout:3531: done (1 left) 2020-06-12T02:37:50.897 INFO:tasks.rados.rados.0.smithi107.stdout:3532: delete oid 5 current snap is 0 2020-06-12T02:37:50.899 INFO:tasks.rados.rados.0.smithi107.stdout:3532: done (1 left) 2020-06-12T02:37:50.900 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:37:50.900 INFO:tasks.rados.rados.0.smithi107.stdout:3533: seq_num 3532 ranges {0=761856,761856=516096,1277952=286720} 2020-06-12T02:37:50.902 INFO:tasks.rados.rados.0.smithi107.stdout:3533: writing smithi10729407-19 from 0 to 761856 tid 1 2020-06-12T02:37:50.905 INFO:tasks.rados.rados.0.smithi107.stdout:3533: writing smithi10729407-19 from 761856 to 1277952 tid 2 2020-06-12T02:37:50.907 INFO:tasks.rados.rados.0.smithi107.stdout:3533: writing smithi10729407-19 from 1277952 to 1564672 tid 3 2020-06-12T02:37:50.907 INFO:tasks.rados.rados.0.smithi107.stdout:3534: delete oid 6 current snap is 0 2020-06-12T02:37:50.908 INFO:tasks.rados.rados.0.smithi107.stdout:3530: finishing write tid 2 to smithi10729407-17 2020-06-12T02:37:50.908 INFO:tasks.rados.rados.0.smithi107.stdout:3530: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:50.908 INFO:tasks.rados.rados.0.smithi107.stdout:3530: finishing write tid 5 to smithi10729407-17 2020-06-12T02:37:50.909 INFO:tasks.rados.rados.0.smithi107.stdout:3530: finishing write tid 6 to smithi10729407-17 2020-06-12T02:37:50.909 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 655 (ObjNum 3529 snap 0 seq_num 3529) dirty exists 2020-06-12T02:37:50.910 INFO:tasks.rados.rados.0.smithi107.stdout:3530: left oid 17 (ObjNum 3529 snap 0 seq_num 3529) 2020-06-12T02:37:50.910 INFO:tasks.rados.rados.0.smithi107.stdout:3530: done (2 left) 2020-06-12T02:37:50.911 INFO:tasks.rados.rados.0.smithi107.stdout:3534: done (1 left) 2020-06-12T02:37:50.911 INFO:tasks.rados.rados.0.smithi107.stdout:3535: delete oid 10 current snap is 0 2020-06-12T02:37:50.912 INFO:tasks.rados.rados.0.smithi107.stdout:3535: done (1 left) 2020-06-12T02:37:50.912 INFO:tasks.rados.rados.0.smithi107.stdout:3536: delete oid 4 current snap is 0 2020-06-12T02:37:50.913 INFO:tasks.rados.rados.0.smithi107.stdout:3533: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:50.913 INFO:tasks.rados.rados.0.smithi107.stdout:3533: finishing write tid 2 to smithi10729407-19 2020-06-12T02:37:50.915 INFO:tasks.rados.rados.0.smithi107.stdout:3536: done (1 left) 2020-06-12T02:37:50.915 INFO:tasks.rados.rados.0.smithi107.stdout:3537: delete oid 1 current snap is 0 2020-06-12T02:37:50.915 INFO:tasks.rados.rados.0.smithi107.stdout:3533: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:50.917 INFO:tasks.rados.rados.0.smithi107.stdout:3537: done (1 left) 2020-06-12T02:37:50.917 INFO:tasks.rados.rados.0.smithi107.stdout:3538: delete oid 14 current snap is 0 2020-06-12T02:37:50.918 INFO:tasks.rados.rados.0.smithi107.stdout:3533: finishing write tid 5 to smithi10729407-19 2020-06-12T02:37:50.919 INFO:tasks.rados.rados.0.smithi107.stdout:3533: finishing write tid 6 to smithi10729407-19 2020-06-12T02:37:50.920 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 587 (ObjNum 3532 snap 0 seq_num 3532) dirty exists 2020-06-12T02:37:50.920 INFO:tasks.rados.rados.0.smithi107.stdout:3533: left oid 19 (ObjNum 3532 snap 0 seq_num 3532) 2020-06-12T02:37:50.920 INFO:tasks.rados.rados.0.smithi107.stdout:3533: done (1 left) 2020-06-12T02:37:50.921 INFO:tasks.rados.rados.0.smithi107.stdout:3538: done (0 left) 2020-06-12T02:37:50.921 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:37:50.921 INFO:tasks.rados.rados.0.smithi107.stdout:3539: seq_num 3538 ranges {0=458752,458752=671744,1130496=475136,1605632=57344} 2020-06-12T02:37:50.923 INFO:tasks.rados.rados.0.smithi107.stdout:3539: writing smithi10729407-10 from 0 to 458752 tid 1 2020-06-12T02:37:50.927 INFO:tasks.rados.rados.0.smithi107.stdout:3539: writing smithi10729407-10 from 458752 to 1130496 tid 2 2020-06-12T02:37:50.930 INFO:tasks.rados.rados.0.smithi107.stdout:3539: writing smithi10729407-10 from 1130496 to 1605632 tid 3 2020-06-12T02:37:50.931 INFO:tasks.rados.rados.0.smithi107.stdout:3539: writing smithi10729407-10 from 1605632 to 1662976 tid 4 2020-06-12T02:37:50.931 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:37:50.931 INFO:tasks.rados.rados.0.smithi107.stdout:3540: seq_num 3539 ranges {0=475136,475136=557056,1032192=647168} 2020-06-12T02:37:50.933 INFO:tasks.rados.rados.0.smithi107.stdout:3540: writing smithi10729407-6 from 0 to 475136 tid 1 2020-06-12T02:37:50.937 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:50.938 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:50.938 INFO:tasks.rados.rados.0.smithi107.stdout:3540: writing smithi10729407-6 from 475136 to 1032192 tid 2 2020-06-12T02:37:50.943 INFO:tasks.rados.rados.0.smithi107.stdout:3540: writing smithi10729407-6 from 1032192 to 1679360 tid 3 2020-06-12T02:37:50.943 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:37:50.943 INFO:tasks.rados.rados.0.smithi107.stdout:3541: seq_num 3540 ranges {5021696=491520,5513216=442368,5955584=442368,6397952=319488} 2020-06-12T02:37:50.948 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-06-12T02:37:50.972 INFO:tasks.rados.rados.0.smithi107.stdout:3541: writing smithi10729407-16 from 5021696 to 5513216 tid 1 2020-06-12T02:37:50.973 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:50.970+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:50.975 INFO:tasks.rados.rados.0.smithi107.stdout:3541: writing smithi10729407-16 from 5513216 to 5955584 tid 2 2020-06-12T02:37:50.978 INFO:tasks.rados.rados.0.smithi107.stdout:3541: writing smithi10729407-16 from 5955584 to 6397952 tid 3 2020-06-12T02:37:50.980 INFO:tasks.rados.rados.0.smithi107.stdout:3541: writing smithi10729407-16 from 6397952 to 6717440 tid 4 2020-06-12T02:37:50.981 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:37:50.981 INFO:tasks.rados.rados.0.smithi107.stdout:3542: seq_num 3541 ranges {0=507904,507904=729088,1236992=475136} 2020-06-12T02:37:50.984 INFO:tasks.rados.rados.0.smithi107.stdout:3542: writing smithi10729407-5 from 0 to 507904 tid 1 2020-06-12T02:37:50.994 INFO:tasks.rados.rados.0.smithi107.stdout:3542: writing smithi10729407-5 from 507904 to 1236992 tid 2 2020-06-12T02:37:51.000 INFO:tasks.rados.rados.0.smithi107.stdout:3542: writing smithi10729407-5 from 1236992 to 1712128 tid 3 2020-06-12T02:37:51.001 INFO:tasks.rados.rados.0.smithi107.stdout:3543: delete oid 18 current snap is 0 2020-06-12T02:37:51.001 INFO:tasks.rados.rados.0.smithi107.stdout:3539: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:51.001 INFO:tasks.rados.rados.0.smithi107.stdout:3539: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:51.001 INFO:tasks.rados.rados.0.smithi107.stdout:3539: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:51.001 INFO:tasks.rados.rados.0.smithi107.stdout:3539: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:51.001 INFO:tasks.rados.rados.0.smithi107.stdout:3539: finishing write tid 6 to smithi10729407-10 2020-06-12T02:37:51.003 INFO:tasks.rados.rados.0.smithi107.stdout:3539: finishing write tid 7 to smithi10729407-10 2020-06-12T02:37:51.003 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 352 (ObjNum 3538 snap 0 seq_num 3538) dirty exists 2020-06-12T02:37:51.003 INFO:tasks.rados.rados.0.smithi107.stdout:3539: left oid 10 (ObjNum 3538 snap 0 seq_num 3538) 2020-06-12T02:37:51.004 INFO:tasks.rados.rados.0.smithi107.stdout:3540: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:51.004 INFO:tasks.rados.rados.0.smithi107.stdout:3540: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:51.004 INFO:tasks.rados.rados.0.smithi107.stdout:3540: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:51.004 INFO:tasks.rados.rados.0.smithi107.stdout:3540: finishing write tid 5 to smithi10729407-6 2020-06-12T02:37:51.004 INFO:tasks.rados.rados.0.smithi107.stdout:3540: finishing write tid 6 to smithi10729407-6 2020-06-12T02:37:51.004 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 666 (ObjNum 3539 snap 0 seq_num 3539) dirty exists 2020-06-12T02:37:51.005 INFO:tasks.rados.rados.0.smithi107.stdout:3540: left oid 6 (ObjNum 3539 snap 0 seq_num 3539) 2020-06-12T02:37:51.005 INFO:tasks.rados.rados.0.smithi107.stdout:3541: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:51.005 INFO:tasks.rados.rados.0.smithi107.stdout:3541: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:51.005 INFO:tasks.rados.rados.0.smithi107.stdout:3542: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:51.005 INFO:tasks.rados.rados.0.smithi107.stdout:3541: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:51.006 INFO:tasks.rados.rados.0.smithi107.stdout:3541: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:51.006 INFO:tasks.rados.rados.0.smithi107.stdout:3541: finishing write tid 6 to smithi10729407-16 2020-06-12T02:37:51.007 INFO:tasks.rados.rados.0.smithi107.stdout:3539: done (4 left) 2020-06-12T02:37:51.007 INFO:tasks.rados.rados.0.smithi107.stdout:3540: done (3 left) 2020-06-12T02:37:51.008 INFO:tasks.rados.rados.0.smithi107.stdout:3543: done (2 left) 2020-06-12T02:37:51.008 INFO:tasks.rados.rados.0.smithi107.stdout:3544: delete oid 17 current snap is 0 2020-06-12T02:37:51.008 INFO:tasks.rados.rados.0.smithi107.stdout:3541: finishing write tid 7 to smithi10729407-16 2020-06-12T02:37:51.008 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 629 (ObjNum 3540 snap 0 seq_num 3540) dirty exists 2020-06-12T02:37:51.008 INFO:tasks.rados.rados.0.smithi107.stdout:3541: left oid 16 (ObjNum 3540 snap 0 seq_num 3540) 2020-06-12T02:37:51.009 INFO:tasks.rados.rados.0.smithi107.stdout:3542: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:51.009 INFO:tasks.rados.rados.0.smithi107.stdout:3542: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:51.009 INFO:tasks.rados.rados.0.smithi107.stdout:3541: done (2 left) 2020-06-12T02:37:51.009 INFO:tasks.rados.rados.0.smithi107.stdout:3544: done (1 left) 2020-06-12T02:37:51.009 INFO:tasks.rados.rados.0.smithi107.stdout:3545: delete oid 1 current snap is 0 2020-06-12T02:37:51.009 INFO:tasks.rados.rados.0.smithi107.stdout:3542: finishing write tid 5 to smithi10729407-5 2020-06-12T02:37:51.010 INFO:tasks.rados.rados.0.smithi107.stdout:3542: finishing write tid 6 to smithi10729407-5 2020-06-12T02:37:51.010 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 312 (ObjNum 3541 snap 0 seq_num 3541) dirty exists 2020-06-12T02:37:51.010 INFO:tasks.rados.rados.0.smithi107.stdout:3542: left oid 5 (ObjNum 3541 snap 0 seq_num 3541) 2020-06-12T02:37:51.010 INFO:tasks.rados.rados.0.smithi107.stdout:3542: done (1 left) 2020-06-12T02:37:51.010 INFO:tasks.rados.rados.0.smithi107.stdout:3545: done (0 left) 2020-06-12T02:37:51.011 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:37:51.011 INFO:tasks.rados.rados.0.smithi107.stdout:3546: seq_num 3545 ranges {0=581632,581632=712704,1294336=491520} 2020-06-12T02:37:51.011 INFO:tasks.rados.rados.0.smithi107.stdout:3546: writing smithi10729407-11 from 0 to 581632 tid 1 2020-06-12T02:37:51.012 INFO:tasks.rados.rados.0.smithi107.stdout:3546: writing smithi10729407-11 from 581632 to 1294336 tid 2 2020-06-12T02:37:51.016 INFO:tasks.rados.rados.0.smithi107.stdout:3546: writing smithi10729407-11 from 1294336 to 1785856 tid 3 2020-06-12T02:37:51.016 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:37:51.016 INFO:tasks.rados.rados.0.smithi107.stdout:3547: seq_num 3546 ranges {1376256=598016,1974272=598016,2572288=442368,3014656=163840} 2020-06-12T02:37:51.029 INFO:tasks.rados.rados.0.smithi107.stdout:3547: writing smithi10729407-15 from 1376256 to 1974272 tid 1 2020-06-12T02:37:51.036 INFO:tasks.rados.rados.0.smithi107.stdout:3547: writing smithi10729407-15 from 1974272 to 2572288 tid 2 2020-06-12T02:37:51.037 INFO:tasks.rados.rados.0.smithi107.stdout:3547: writing smithi10729407-15 from 2572288 to 3014656 tid 3 2020-06-12T02:37:51.037 INFO:tasks.rados.rados.0.smithi107.stdout:3547: writing smithi10729407-15 from 3014656 to 3178496 tid 4 2020-06-12T02:37:51.037 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:37:51.037 INFO:tasks.rados.rados.0.smithi107.stdout:3548: seq_num 3547 ranges {1564672=614400,2179072=483328,2662400=647168,3309568=73728} 2020-06-12T02:37:51.047 INFO:tasks.rados.rados.0.smithi107.stdout:3548: writing smithi10729407-19 from 1564672 to 2179072 tid 1 2020-06-12T02:37:51.050 INFO:tasks.rados.rados.0.smithi107.stdout:3548: writing smithi10729407-19 from 2179072 to 2662400 tid 2 2020-06-12T02:37:51.054 INFO:tasks.rados.rados.0.smithi107.stdout:3548: writing smithi10729407-19 from 2662400 to 3309568 tid 3 2020-06-12T02:37:51.055 INFO:tasks.rados.rados.0.smithi107.stdout:3548: writing smithi10729407-19 from 3309568 to 3383296 tid 4 2020-06-12T02:37:51.055 INFO:tasks.rados.rados.0.smithi107.stdout:3549: delete oid 6 current snap is 0 2020-06-12T02:37:51.055 INFO:tasks.rados.rados.0.smithi107.stdout:3546: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:51.055 INFO:tasks.rados.rados.0.smithi107.stdout:3546: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:51.055 INFO:tasks.rados.rados.0.smithi107.stdout:3546: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:51.056 INFO:tasks.rados.rados.0.smithi107.stdout:3546: finishing write tid 5 to smithi10729407-11 2020-06-12T02:37:51.056 INFO:tasks.rados.rados.0.smithi107.stdout:3546: finishing write tid 6 to smithi10729407-11 2020-06-12T02:37:51.056 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 622 (ObjNum 3545 snap 0 seq_num 3545) dirty exists 2020-06-12T02:37:51.056 INFO:tasks.rados.rados.0.smithi107.stdout:3546: left oid 11 (ObjNum 3545 snap 0 seq_num 3545) 2020-06-12T02:37:51.057 INFO:tasks.rados.rados.0.smithi107.stdout:3547: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:51.057 INFO:tasks.rados.rados.0.smithi107.stdout:3547: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:51.057 INFO:tasks.rados.rados.0.smithi107.stdout:3547: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:51.057 INFO:tasks.rados.rados.0.smithi107.stdout:3547: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:51.057 INFO:tasks.rados.rados.0.smithi107.stdout:3547: finishing write tid 6 to smithi10729407-15 2020-06-12T02:37:51.058 INFO:tasks.rados.rados.0.smithi107.stdout:3547: finishing write tid 7 to smithi10729407-15 2020-06-12T02:37:51.058 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 599 (ObjNum 3546 snap 0 seq_num 3546) dirty exists 2020-06-12T02:37:51.058 INFO:tasks.rados.rados.0.smithi107.stdout:3547: left oid 15 (ObjNum 3546 snap 0 seq_num 3546) 2020-06-12T02:37:51.058 INFO:tasks.rados.rados.0.smithi107.stdout:3548: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:51.059 INFO:tasks.rados.rados.0.smithi107.stdout:3546: done (3 left) 2020-06-12T02:37:51.059 INFO:tasks.rados.rados.0.smithi107.stdout:3547: done (2 left) 2020-06-12T02:37:51.059 INFO:tasks.rados.rados.0.smithi107.stdout:3549: done (1 left) 2020-06-12T02:37:51.060 INFO:tasks.rados.rados.0.smithi107.stdout:3550: delete oid 11 current snap is 0 2020-06-12T02:37:51.060 INFO:tasks.rados.rados.0.smithi107.stdout:3550: done (1 left) 2020-06-12T02:37:51.060 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:37:51.060 INFO:tasks.rados.rados.0.smithi107.stdout:3551: seq_num 3550 ranges {0=663552,663552=532480,1196032=417792,1613824=253952} 2020-06-12T02:37:51.062 INFO:tasks.rados.rados.0.smithi107.stdout:3551: writing smithi10729407-4 from 0 to 663552 tid 1 2020-06-12T02:37:51.064 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:37:51.063+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:37:51.066 INFO:tasks.rados.rados.0.smithi107.stdout:3551: writing smithi10729407-4 from 663552 to 1196032 tid 2 2020-06-12T02:37:51.068 INFO:tasks.rados.rados.0.smithi107.stdout:3551: writing smithi10729407-4 from 1196032 to 1613824 tid 3 2020-06-12T02:37:51.070 INFO:tasks.rados.rados.0.smithi107.stdout:3551: writing smithi10729407-4 from 1613824 to 1867776 tid 4 2020-06-12T02:37:51.070 INFO:tasks.rados.rados.0.smithi107.stdout:3552: delete oid 13 current snap is 0 2020-06-12T02:37:51.070 INFO:tasks.rados.rados.0.smithi107.stdout:3548: finishing write tid 2 to smithi10729407-19 2020-06-12T02:37:51.070 INFO:tasks.rados.rados.0.smithi107.stdout:3548: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:51.071 INFO:tasks.rados.rados.0.smithi107.stdout:3548: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:51.071 INFO:tasks.rados.rados.0.smithi107.stdout:3548: finishing write tid 6 to smithi10729407-19 2020-06-12T02:37:51.071 INFO:tasks.rados.rados.0.smithi107.stdout:3548: finishing write tid 7 to smithi10729407-19 2020-06-12T02:37:51.071 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 592 (ObjNum 3547 snap 0 seq_num 3547) dirty exists 2020-06-12T02:37:51.071 INFO:tasks.rados.rados.0.smithi107.stdout:3548: left oid 19 (ObjNum 3547 snap 0 seq_num 3547) 2020-06-12T02:37:51.073 INFO:tasks.rados.rados.0.smithi107.stdout:3551: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:51.075 INFO:tasks.rados.rados.0.smithi107.stdout:3548: done (2 left) 2020-06-12T02:37:51.076 INFO:tasks.rados.rados.0.smithi107.stdout:3552: done (1 left) 2020-06-12T02:37:51.076 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:37:51.076 INFO:tasks.rados.rados.0.smithi107.stdout:3553: seq_num 3552 ranges {3178496=696320,3874816=753664,4628480=450560} 2020-06-12T02:37:51.097 INFO:tasks.rados.rados.0.smithi107.stdout:3553: writing smithi10729407-15 from 3178496 to 3874816 tid 1 2020-06-12T02:37:51.101 INFO:tasks.rados.rados.0.smithi107.stdout:3553: writing smithi10729407-15 from 3874816 to 4628480 tid 2 2020-06-12T02:37:51.104 INFO:tasks.rados.rados.0.smithi107.stdout:3553: writing smithi10729407-15 from 4628480 to 5079040 tid 3 2020-06-12T02:37:51.104 INFO:tasks.rados.rados.0.smithi107.stdout:3554: delete oid 13 current snap is 0 2020-06-12T02:37:51.105 INFO:tasks.rados.rados.0.smithi107.stdout:3551: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:51.105 INFO:tasks.rados.rados.0.smithi107.stdout:3551: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:51.105 INFO:tasks.rados.rados.0.smithi107.stdout:3551: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:51.105 INFO:tasks.rados.rados.0.smithi107.stdout:3551: finishing write tid 6 to smithi10729407-4 2020-06-12T02:37:51.105 INFO:tasks.rados.rados.0.smithi107.stdout:3551: finishing write tid 7 to smithi10729407-4 2020-06-12T02:37:51.106 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 662 (ObjNum 3550 snap 0 seq_num 3550) dirty exists 2020-06-12T02:37:51.106 INFO:tasks.rados.rados.0.smithi107.stdout:3551: left oid 4 (ObjNum 3550 snap 0 seq_num 3550) 2020-06-12T02:37:51.106 INFO:tasks.rados.rados.0.smithi107.stdout:3551: done (2 left) 2020-06-12T02:37:51.106 INFO:tasks.rados.rados.0.smithi107.stdout:3554: done (1 left) 2020-06-12T02:37:51.107 INFO:tasks.rados.rados.0.smithi107.stdout:3555: delete oid 2 current snap is 0 2020-06-12T02:37:51.109 INFO:tasks.rados.rados.0.smithi107.stdout:3555: done (1 left) 2020-06-12T02:37:51.109 INFO:tasks.rados.rados.0.smithi107.stdout:3556: delete oid 17 current snap is 0 2020-06-12T02:37:51.111 INFO:tasks.rados.rados.0.smithi107.stdout:3553: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:51.111 INFO:tasks.rados.rados.0.smithi107.stdout:3556: done (1 left) 2020-06-12T02:37:51.112 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:37:51.112 INFO:tasks.rados.rados.0.smithi107.stdout:3557: seq_num 3556 ranges {0=761856,761856=688128,1449984=516096} 2020-06-12T02:37:51.116 INFO:tasks.rados.rados.0.smithi107.stdout:3557: writing smithi10729407-9 from 0 to 761856 tid 1 2020-06-12T02:37:51.122 INFO:tasks.rados.rados.0.smithi107.stdout:3557: writing smithi10729407-9 from 761856 to 1449984 tid 2 2020-06-12T02:37:51.125 INFO:tasks.rados.rados.0.smithi107.stdout:3557: writing smithi10729407-9 from 1449984 to 1966080 tid 3 2020-06-12T02:37:51.126 INFO:tasks.rados.rados.0.smithi107.stdout:3558: delete oid 6 current snap is 0 2020-06-12T02:37:51.126 INFO:tasks.rados.rados.0.smithi107.stdout:3553: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:51.127 INFO:tasks.rados.rados.0.smithi107.stdout:3553: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:51.127 INFO:tasks.rados.rados.0.smithi107.stdout:3553: finishing write tid 5 to smithi10729407-15 2020-06-12T02:37:51.127 INFO:tasks.rados.rados.0.smithi107.stdout:3553: finishing write tid 6 to smithi10729407-15 2020-06-12T02:37:51.127 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 603 (ObjNum 3552 snap 0 seq_num 3552) dirty exists 2020-06-12T02:37:51.127 INFO:tasks.rados.rados.0.smithi107.stdout:3553: left oid 15 (ObjNum 3552 snap 0 seq_num 3552) 2020-06-12T02:37:51.128 INFO:tasks.rados.rados.0.smithi107.stdout:3557: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:51.128 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:51.128 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:51.129 INFO:tasks.rados.rados.0.smithi107.stdout:3553: done (2 left) 2020-06-12T02:37:51.129 INFO:tasks.rados.rados.0.smithi107.stdout:3558: done (1 left) 2020-06-12T02:37:51.129 INFO:tasks.rados.rados.0.smithi107.stdout:3559: delete oid 6 current snap is 0 2020-06-12T02:37:51.130 INFO:tasks.rados.rados.0.smithi107.stdout:3559: done (1 left) 2020-06-12T02:37:51.130 INFO:tasks.rados.rados.0.smithi107.stdout:3560: delete oid 5 current snap is 0 2020-06-12T02:37:51.131 INFO:tasks.rados.rados.0.smithi107.stdout:3557: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:51.132 INFO:tasks.rados.rados.0.smithi107.stdout:3557: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:51.132 INFO:tasks.rados.rados.0.smithi107.stdout:3557: finishing write tid 5 to smithi10729407-9 2020-06-12T02:37:51.132 INFO:tasks.rados.rados.0.smithi107.stdout:3560: done (1 left) 2020-06-12T02:37:51.133 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:37:51.133 INFO:tasks.rados.rados.0.smithi107.stdout:3561: seq_num 3560 ranges {5079040=425984,5505024=679936,6184960=745472,6930432=180224} 2020-06-12T02:37:51.137 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-06-12T02:37:51.162 INFO:tasks.rados.rados.0.smithi107.stdout:3561: writing smithi10729407-15 from 5079040 to 5505024 tid 1 2020-06-12T02:37:51.166 INFO:tasks.rados.rados.0.smithi107.stdout:3561: writing smithi10729407-15 from 5505024 to 6184960 tid 2 2020-06-12T02:37:51.167 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:37:51.165+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:37:51.171 INFO:tasks.rados.rados.0.smithi107.stdout:3561: writing smithi10729407-15 from 6184960 to 6930432 tid 3 2020-06-12T02:37:51.172 INFO:tasks.rados.rados.0.smithi107.stdout:3561: writing smithi10729407-15 from 6930432 to 7110656 tid 4 2020-06-12T02:37:51.172 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:51.173 INFO:tasks.rados.rados.0.smithi107.stdout:3562: seq_num 3561 ranges {0=442368,442368=565248,1007616=557056,1564672=466944,2031616=16384} 2020-06-12T02:37:51.175 INFO:tasks.rados.rados.0.smithi107.stdout:3562: writing smithi10729407-2 from 0 to 442368 tid 1 2020-06-12T02:37:51.179 INFO:tasks.rados.rados.0.smithi107.stdout:3562: writing smithi10729407-2 from 442368 to 1007616 tid 2 2020-06-12T02:37:51.182 INFO:tasks.rados.rados.0.smithi107.stdout:3562: writing smithi10729407-2 from 1007616 to 1564672 tid 3 2020-06-12T02:37:51.185 INFO:tasks.rados.rados.0.smithi107.stdout:3562: writing smithi10729407-2 from 1564672 to 2031616 tid 4 2020-06-12T02:37:51.186 INFO:tasks.rados.rados.0.smithi107.stdout:3562: writing smithi10729407-2 from 2031616 to 2048000 tid 5 2020-06-12T02:37:51.186 INFO:tasks.rados.rados.0.smithi107.stdout:3563: delete oid 17 current snap is 0 2020-06-12T02:37:51.186 INFO:tasks.rados.rados.0.smithi107.stdout:3557: finishing write tid 6 to smithi10729407-9 2020-06-12T02:37:51.186 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 629 (ObjNum 3556 snap 0 seq_num 3556) dirty exists 2020-06-12T02:37:51.187 INFO:tasks.rados.rados.0.smithi107.stdout:3557: left oid 9 (ObjNum 3556 snap 0 seq_num 3556) 2020-06-12T02:37:51.187 INFO:tasks.rados.rados.0.smithi107.stdout:3561: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:51.188 INFO:tasks.rados.rados.0.smithi107.stdout:3562: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:51.188 INFO:tasks.rados.rados.0.smithi107.stdout:3561: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:51.189 INFO:tasks.rados.rados.0.smithi107.stdout:3561: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:51.189 INFO:tasks.rados.rados.0.smithi107.stdout:3561: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:51.190 INFO:tasks.rados.rados.0.smithi107.stdout:3557: done (3 left) 2020-06-12T02:37:51.190 INFO:tasks.rados.rados.0.smithi107.stdout:3563: done (2 left) 2020-06-12T02:37:51.190 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:37:51.190 INFO:tasks.rados.rados.0.smithi107.stdout:3564: seq_num 3563 ranges {3383296=475136,3858432=729088,4587520=565248,5152768=311296} 2020-06-12T02:37:51.210 INFO:tasks.rados.rados.0.smithi107.stdout:3564: writing smithi10729407-19 from 3383296 to 3858432 tid 1 2020-06-12T02:37:51.215 INFO:tasks.rados.rados.0.smithi107.stdout:3564: writing smithi10729407-19 from 3858432 to 4587520 tid 2 2020-06-12T02:37:51.219 INFO:tasks.rados.rados.0.smithi107.stdout:3564: writing smithi10729407-19 from 4587520 to 5152768 tid 3 2020-06-12T02:37:51.222 INFO:tasks.rados.rados.0.smithi107.stdout:3564: writing smithi10729407-19 from 5152768 to 5464064 tid 4 2020-06-12T02:37:51.222 INFO:tasks.rados.rados.0.smithi107.stdout:3565: delete oid 4 current snap is 0 2020-06-12T02:37:51.222 INFO:tasks.rados.rados.0.smithi107.stdout:3561: finishing write tid 6 to smithi10729407-15 2020-06-12T02:37:51.223 INFO:tasks.rados.rados.0.smithi107.stdout:3561: finishing write tid 7 to smithi10729407-15 2020-06-12T02:37:51.223 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 608 (ObjNum 3560 snap 0 seq_num 3560) dirty exists 2020-06-12T02:37:51.223 INFO:tasks.rados.rados.0.smithi107.stdout:3561: left oid 15 (ObjNum 3560 snap 0 seq_num 3560) 2020-06-12T02:37:51.223 INFO:tasks.rados.rados.0.smithi107.stdout:3562: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:51.224 INFO:tasks.rados.rados.0.smithi107.stdout:3562: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:51.224 INFO:tasks.rados.rados.0.smithi107.stdout:3562: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:51.224 INFO:tasks.rados.rados.0.smithi107.stdout:3562: finishing write tid 5 to smithi10729407-2 2020-06-12T02:37:51.224 INFO:tasks.rados.rados.0.smithi107.stdout:3562: finishing write tid 7 to smithi10729407-2 2020-06-12T02:37:51.224 INFO:tasks.rados.rados.0.smithi107.stdout:3562: finishing write tid 8 to smithi10729407-2 2020-06-12T02:37:51.225 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 369 (ObjNum 3561 snap 0 seq_num 3561) dirty exists 2020-06-12T02:37:51.225 INFO:tasks.rados.rados.0.smithi107.stdout:3562: left oid 2 (ObjNum 3561 snap 0 seq_num 3561) 2020-06-12T02:37:51.225 INFO:tasks.rados.rados.0.smithi107.stdout:3564: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:51.226 INFO:tasks.rados.rados.0.smithi107.stdout:3564: finishing write tid 2 to smithi10729407-19 2020-06-12T02:37:51.226 INFO:tasks.rados.rados.0.smithi107.stdout:3561: done (3 left) 2020-06-12T02:37:51.226 INFO:tasks.rados.rados.0.smithi107.stdout:3562: done (2 left) 2020-06-12T02:37:51.227 INFO:tasks.rados.rados.0.smithi107.stdout:3565: done (1 left) 2020-06-12T02:37:51.227 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:51.228 INFO:tasks.rados.rados.0.smithi107.stdout:3566: seq_num 3565 ranges {2048000=516096,2564096=499712,3063808=532480,3596288=524288,4120576=49152} 2020-06-12T02:37:51.238 INFO:tasks.rados.rados.0.smithi107.stdout:3566: writing smithi10729407-2 from 2048000 to 2564096 tid 1 2020-06-12T02:37:51.242 INFO:tasks.rados.rados.0.smithi107.stdout:3566: writing smithi10729407-2 from 2564096 to 3063808 tid 2 2020-06-12T02:37:51.245 INFO:tasks.rados.rados.0.smithi107.stdout:3566: writing smithi10729407-2 from 3063808 to 3596288 tid 3 2020-06-12T02:37:51.248 INFO:tasks.rados.rados.0.smithi107.stdout:3566: writing smithi10729407-2 from 3596288 to 4120576 tid 4 2020-06-12T02:37:51.249 INFO:tasks.rados.rados.0.smithi107.stdout:3566: writing smithi10729407-2 from 4120576 to 4169728 tid 5 2020-06-12T02:37:51.249 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:37:51.250 INFO:tasks.rados.rados.0.smithi107.stdout:3567: seq_num 3566 ranges {0=532480,532480=434176,966656=745472,1712128=409600,2121728=16384} 2020-06-12T02:37:51.252 INFO:tasks.rados.rados.0.smithi107.stdout:3567: writing smithi10729407-14 from 0 to 532480 tid 1 2020-06-12T02:37:51.255 INFO:tasks.rados.rados.0.smithi107.stdout:3567: writing smithi10729407-14 from 532480 to 966656 tid 2 2020-06-12T02:37:51.260 INFO:tasks.rados.rados.0.smithi107.stdout:3567: writing smithi10729407-14 from 966656 to 1712128 tid 3 2020-06-12T02:37:51.263 INFO:tasks.rados.rados.0.smithi107.stdout:3567: writing smithi10729407-14 from 1712128 to 2121728 tid 4 2020-06-12T02:37:51.263 INFO:tasks.rados.rados.0.smithi107.stdout:3567: writing smithi10729407-14 from 2121728 to 2138112 tid 5 2020-06-12T02:37:51.263 INFO:tasks.rados.rados.0.smithi107.stdout:3568: delete oid 12 current snap is 0 2020-06-12T02:37:51.264 INFO:tasks.rados.rados.0.smithi107.stdout:3564: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:51.264 INFO:tasks.rados.rados.0.smithi107.stdout:3564: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:51.264 INFO:tasks.rados.rados.0.smithi107.stdout:3564: finishing write tid 6 to smithi10729407-19 2020-06-12T02:37:51.265 INFO:tasks.rados.rados.0.smithi107.stdout:3564: finishing write tid 7 to smithi10729407-19 2020-06-12T02:37:51.265 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 597 (ObjNum 3563 snap 0 seq_num 3563) dirty exists 2020-06-12T02:37:51.265 INFO:tasks.rados.rados.0.smithi107.stdout:3564: left oid 19 (ObjNum 3563 snap 0 seq_num 3563) 2020-06-12T02:37:51.265 INFO:tasks.rados.rados.0.smithi107.stdout:3566: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:51.265 INFO:tasks.rados.rados.0.smithi107.stdout:3566: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:51.266 INFO:tasks.rados.rados.0.smithi107.stdout:3566: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:51.266 INFO:tasks.rados.rados.0.smithi107.stdout:3566: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:51.266 INFO:tasks.rados.rados.0.smithi107.stdout:3566: finishing write tid 5 to smithi10729407-2 2020-06-12T02:37:51.266 INFO:tasks.rados.rados.0.smithi107.stdout:3566: finishing write tid 7 to smithi10729407-2 2020-06-12T02:37:51.266 INFO:tasks.rados.rados.0.smithi107.stdout:3567: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:51.266 INFO:tasks.rados.rados.0.smithi107.stdout:3566: finishing write tid 8 to smithi10729407-2 2020-06-12T02:37:51.267 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 375 (ObjNum 3565 snap 0 seq_num 3565) dirty exists 2020-06-12T02:37:51.267 INFO:tasks.rados.rados.0.smithi107.stdout:3566: left oid 2 (ObjNum 3565 snap 0 seq_num 3565) 2020-06-12T02:37:51.268 INFO:tasks.rados.rados.0.smithi107.stdout:3564: done (3 left) 2020-06-12T02:37:51.268 INFO:tasks.rados.rados.0.smithi107.stdout:3566: done (2 left) 2020-06-12T02:37:51.269 INFO:tasks.rados.rados.0.smithi107.stdout:3568: done (1 left) 2020-06-12T02:37:51.269 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:37:51.269 INFO:tasks.rados.rados.0.smithi107.stdout:3569: seq_num 3568 ranges {0=565248,565248=606208,1171456=704512,1875968=294912} 2020-06-12T02:37:51.269 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:51.266+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:37:51.270 INFO:tasks.rados.rados.0.smithi107.stdout:3569: writing smithi10729407-3 from 0 to 565248 tid 1 2020-06-12T02:37:51.276 INFO:tasks.rados.rados.0.smithi107.stdout:3569: writing smithi10729407-3 from 565248 to 1171456 tid 2 2020-06-12T02:37:51.277 INFO:tasks.rados.rados.0.smithi107.stdout:3569: writing smithi10729407-3 from 1171456 to 1875968 tid 3 2020-06-12T02:37:51.279 INFO:tasks.rados.rados.0.smithi107.stdout:3569: writing smithi10729407-3 from 1875968 to 2170880 tid 4 2020-06-12T02:37:51.279 INFO:tasks.rados.rados.0.smithi107.stdout:3570: delete oid 20 current snap is 0 2020-06-12T02:37:51.280 INFO:tasks.rados.rados.0.smithi107.stdout:3567: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:51.280 INFO:tasks.rados.rados.0.smithi107.stdout:3567: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:51.280 INFO:tasks.rados.rados.0.smithi107.stdout:3567: finishing write tid 4 to smithi10729407-14 2020-06-12T02:37:51.280 INFO:tasks.rados.rados.0.smithi107.stdout:3569: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:51.280 INFO:tasks.rados.rados.0.smithi107.stdout:3567: finishing write tid 5 to smithi10729407-14 2020-06-12T02:37:51.282 INFO:tasks.rados.rados.0.smithi107.stdout:3567: finishing write tid 7 to smithi10729407-14 2020-06-12T02:37:51.282 INFO:tasks.rados.rados.0.smithi107.stdout:3567: finishing write tid 8 to smithi10729407-14 2020-06-12T02:37:51.282 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 629 (ObjNum 3566 snap 0 seq_num 3566) dirty exists 2020-06-12T02:37:51.282 INFO:tasks.rados.rados.0.smithi107.stdout:3567: left oid 14 (ObjNum 3566 snap 0 seq_num 3566) 2020-06-12T02:37:51.286 INFO:tasks.rados.rados.0.smithi107.stdout:3569: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:51.287 INFO:tasks.rados.rados.0.smithi107.stdout:3567: done (2 left) 2020-06-12T02:37:51.287 INFO:tasks.rados.rados.0.smithi107.stdout:3570: done (1 left) 2020-06-12T02:37:51.287 INFO:tasks.rados.rados.0.smithi107.stdout:3571: delete oid 9 current snap is 0 2020-06-12T02:37:51.290 INFO:tasks.rados.rados.0.smithi107.stdout:3569: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:51.292 INFO:tasks.rados.rados.0.smithi107.stdout:3569: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:51.292 INFO:tasks.rados.rados.0.smithi107.stdout:3569: finishing write tid 6 to smithi10729407-3 2020-06-12T02:37:51.294 INFO:tasks.rados.rados.0.smithi107.stdout:3569: finishing write tid 7 to smithi10729407-3 2020-06-12T02:37:51.294 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 634 (ObjNum 3568 snap 0 seq_num 3568) dirty exists 2020-06-12T02:37:51.294 INFO:tasks.rados.rados.0.smithi107.stdout:3569: left oid 3 (ObjNum 3568 snap 0 seq_num 3568) 2020-06-12T02:37:51.296 INFO:tasks.rados.rados.0.smithi107.stdout:3569: done (1 left) 2020-06-12T02:37:51.297 INFO:tasks.rados.rados.0.smithi107.stdout:3571: done (0 left) 2020-06-12T02:37:51.297 INFO:tasks.rados.rados.0.smithi107.stdout:3572: delete oid 18 current snap is 0 2020-06-12T02:37:51.300 INFO:tasks.rados.rados.0.smithi107.stdout:3572: done (0 left) 2020-06-12T02:37:51.300 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:37:51.300 INFO:tasks.rados.rados.0.smithi107.stdout:3573: seq_num 3572 ranges {6717440=630784,7348224=540672,7888896=737280,8626176=327680} 2020-06-12T02:37:51.316 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:51.316 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:51.325 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-06-12T02:37:51.339 INFO:tasks.rados.rados.0.smithi107.stdout:3573: writing smithi10729407-16 from 6717440 to 7348224 tid 1 2020-06-12T02:37:51.342 INFO:tasks.rados.rados.0.smithi107.stdout:3573: writing smithi10729407-16 from 7348224 to 7888896 tid 2 2020-06-12T02:37:51.348 INFO:tasks.rados.rados.0.smithi107.stdout:3573: writing smithi10729407-16 from 7888896 to 8626176 tid 3 2020-06-12T02:37:51.350 INFO:tasks.rados.rados.0.smithi107.stdout:3573: writing smithi10729407-16 from 8626176 to 8953856 tid 4 2020-06-12T02:37:51.350 INFO:tasks.rados.rados.0.smithi107.stdout:3574: delete oid 6 current snap is 0 2020-06-12T02:37:51.350 INFO:tasks.rados.rados.0.smithi107.stdout:3573: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:51.352 INFO:tasks.rados.rados.0.smithi107.stdout:3574: done (1 left) 2020-06-12T02:37:51.352 INFO:tasks.rados.rados.0.smithi107.stdout:3575: delete oid 12 current snap is 0 2020-06-12T02:37:51.354 INFO:tasks.rados.rados.0.smithi107.stdout:3575: done (1 left) 2020-06-12T02:37:51.355 INFO:tasks.rados.rados.0.smithi107.stdout:3576: delete oid 14 current snap is 0 2020-06-12T02:37:51.357 INFO:tasks.rados.rados.0.smithi107.stdout:3576: done (1 left) 2020-06-12T02:37:51.357 INFO:tasks.rados.rados.0.smithi107.stdout:3577: delete oid 18 current snap is 0 2020-06-12T02:37:51.360 INFO:tasks.rados.rados.0.smithi107.stdout:3573: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:51.360 INFO:tasks.rados.rados.0.smithi107.stdout:3573: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:51.362 INFO:tasks.rados.rados.0.smithi107.stdout:3573: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:51.363 INFO:tasks.rados.rados.0.smithi107.stdout:3573: finishing write tid 6 to smithi10729407-16 2020-06-12T02:37:51.363 INFO:tasks.rados.rados.0.smithi107.stdout:3577: done (1 left) 2020-06-12T02:37:51.363 INFO:tasks.rados.rados.0.smithi107.stdout:3578: delete oid 15 current snap is 0 2020-06-12T02:37:51.364 INFO:tasks.rados.rados.0.smithi107.stdout:3573: finishing write tid 7 to smithi10729407-16 2020-06-12T02:37:51.365 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 639 (ObjNum 3572 snap 0 seq_num 3572) dirty exists 2020-06-12T02:37:51.365 INFO:tasks.rados.rados.0.smithi107.stdout:3573: left oid 16 (ObjNum 3572 snap 0 seq_num 3572) 2020-06-12T02:37:51.365 INFO:tasks.rados.rados.0.smithi107.stdout:3573: done (1 left) 2020-06-12T02:37:51.366 INFO:tasks.rados.rados.0.smithi107.stdout:3578: done (0 left) 2020-06-12T02:37:51.366 INFO:tasks.rados.rados.0.smithi107.stdout:3579: delete oid 18 current snap is 0 2020-06-12T02:37:51.367 INFO:tasks.rados.rados.0.smithi107.stdout:3579: done (0 left) 2020-06-12T02:37:51.367 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:37:51.368 INFO:tasks.rados.rados.0.smithi107.stdout:3580: seq_num 3579 ranges {0=745472,745472=581632,1327104=491520,1818624=458752,2277376=73728} 2020-06-12T02:37:51.370 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:37:51.368+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:37:51.371 INFO:tasks.rados.rados.0.smithi107.stdout:3580: writing smithi10729407-17 from 0 to 745472 tid 1 2020-06-12T02:37:51.375 INFO:tasks.rados.rados.0.smithi107.stdout:3580: writing smithi10729407-17 from 745472 to 1327104 tid 2 2020-06-12T02:37:51.379 INFO:tasks.rados.rados.0.smithi107.stdout:3580: writing smithi10729407-17 from 1327104 to 1818624 tid 3 2020-06-12T02:37:51.382 INFO:tasks.rados.rados.0.smithi107.stdout:3580: writing smithi10729407-17 from 1818624 to 2277376 tid 4 2020-06-12T02:37:51.382 INFO:tasks.rados.rados.0.smithi107.stdout:3580: writing smithi10729407-17 from 2277376 to 2351104 tid 5 2020-06-12T02:37:51.383 INFO:tasks.rados.rados.0.smithi107.stdout:3581: delete oid 14 current snap is 0 2020-06-12T02:37:51.383 INFO:tasks.rados.rados.0.smithi107.stdout:3580: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:51.385 INFO:tasks.rados.rados.0.smithi107.stdout:3581: done (1 left) 2020-06-12T02:37:51.385 INFO:tasks.rados.rados.0.smithi107.stdout:3582: delete oid 1 current snap is 0 2020-06-12T02:37:51.385 INFO:tasks.rados.rados.0.smithi107.stdout:3580: finishing write tid 2 to smithi10729407-17 2020-06-12T02:37:51.387 INFO:tasks.rados.rados.0.smithi107.stdout:3580: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:51.387 INFO:tasks.rados.rados.0.smithi107.stdout:3582: done (1 left) 2020-06-12T02:37:51.387 INFO:tasks.rados.rados.0.smithi107.stdout:3583: delete oid 6 current snap is 0 2020-06-12T02:37:51.388 INFO:tasks.rados.rados.0.smithi107.stdout:3580: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:51.388 INFO:tasks.rados.rados.0.smithi107.stdout:3580: finishing write tid 5 to smithi10729407-17 2020-06-12T02:37:51.388 INFO:tasks.rados.rados.0.smithi107.stdout:3580: finishing write tid 7 to smithi10729407-17 2020-06-12T02:37:51.390 INFO:tasks.rados.rados.0.smithi107.stdout:3583: done (1 left) 2020-06-12T02:37:51.390 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:37:51.390 INFO:tasks.rados.rados.0.smithi107.stdout:3584: seq_num 3583 ranges {0=16384} 2020-06-12T02:37:51.390 INFO:tasks.rados.rados.0.smithi107.stdout:3584: writing smithi10729407-13 from 0 to 16384 tid 1 2020-06-12T02:37:51.391 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:37:51.391 INFO:tasks.rados.rados.0.smithi107.stdout:3585: seq_num 3584 ranges {0=32768} 2020-06-12T02:37:51.391 INFO:tasks.rados.rados.0.smithi107.stdout:3585: writing smithi10729407-12 from 0 to 32768 tid 1 2020-06-12T02:37:51.391 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:37:51.391 INFO:tasks.rados.rados.0.smithi107.stdout:3586: seq_num 3585 ranges {0=49152} 2020-06-12T02:37:51.392 INFO:tasks.rados.rados.0.smithi107.stdout:3586: writing smithi10729407-7 from 0 to 49152 tid 1 2020-06-12T02:37:51.392 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:37:51.392 INFO:tasks.rados.rados.0.smithi107.stdout:3587: seq_num 3586 ranges {5464064=65536} 2020-06-12T02:37:51.409 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2020-06-12T02:37:51.421 INFO:tasks.rados.rados.0.smithi107.stdout:3587: writing smithi10729407-19 from 5464064 to 5529600 tid 1 2020-06-12T02:37:51.421 INFO:tasks.rados.rados.0.smithi107.stdout:3588: delete oid 10 current snap is 0 2020-06-12T02:37:51.421 INFO:tasks.rados.rados.0.smithi107.stdout:3580: finishing write tid 8 to smithi10729407-17 2020-06-12T02:37:51.422 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 671 (ObjNum 3579 snap 0 seq_num 3579) dirty exists 2020-06-12T02:37:51.422 INFO:tasks.rados.rados.0.smithi107.stdout:3580: left oid 17 (ObjNum 3579 snap 0 seq_num 3579) 2020-06-12T02:37:51.422 INFO:tasks.rados.rados.0.smithi107.stdout:3585: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:51.422 INFO:tasks.rados.rados.0.smithi107.stdout:3585: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:51.422 INFO:tasks.rados.rados.0.smithi107.stdout:3584: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:51.423 INFO:tasks.rados.rados.0.smithi107.stdout:3586: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:51.423 INFO:tasks.rados.rados.0.smithi107.stdout:3586: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:51.423 INFO:tasks.rados.rados.0.smithi107.stdout:3585: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:51.423 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 601 (ObjNum 3584 snap 0 seq_num 3584) dirty exists 2020-06-12T02:37:51.423 INFO:tasks.rados.rados.0.smithi107.stdout:3585: left oid 12 (ObjNum 3584 snap 0 seq_num 3584) 2020-06-12T02:37:51.423 INFO:tasks.rados.rados.0.smithi107.stdout:3586: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:51.424 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 687 (ObjNum 3585 snap 0 seq_num 3585) dirty exists 2020-06-12T02:37:51.424 INFO:tasks.rados.rados.0.smithi107.stdout:3586: left oid 7 (ObjNum 3585 snap 0 seq_num 3585) 2020-06-12T02:37:51.424 INFO:tasks.rados.rados.0.smithi107.stdout:3584: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:51.424 INFO:tasks.rados.rados.0.smithi107.stdout:3584: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:51.424 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 688 (ObjNum 3583 snap 0 seq_num 3583) dirty exists 2020-06-12T02:37:51.425 INFO:tasks.rados.rados.0.smithi107.stdout:3584: left oid 13 (ObjNum 3583 snap 0 seq_num 3583) 2020-06-12T02:37:51.425 INFO:tasks.rados.rados.0.smithi107.stdout:3580: done (5 left) 2020-06-12T02:37:51.425 INFO:tasks.rados.rados.0.smithi107.stdout:3584: done (4 left) 2020-06-12T02:37:51.426 INFO:tasks.rados.rados.0.smithi107.stdout:3585: done (3 left) 2020-06-12T02:37:51.426 INFO:tasks.rados.rados.0.smithi107.stdout:3586: done (2 left) 2020-06-12T02:37:51.426 INFO:tasks.rados.rados.0.smithi107.stdout:3588: done (1 left) 2020-06-12T02:37:51.426 INFO:tasks.rados.rados.0.smithi107.stdout:3589: delete oid 7 current snap is 0 2020-06-12T02:37:51.427 INFO:tasks.rados.rados.0.smithi107.stdout:3587: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:51.427 INFO:tasks.rados.rados.0.smithi107.stdout:3587: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:51.427 INFO:tasks.rados.rados.0.smithi107.stdout:3589: done (1 left) 2020-06-12T02:37:51.427 INFO:tasks.rados.rados.0.smithi107.stdout:3590: delete oid 10 current snap is 0 2020-06-12T02:37:51.428 INFO:tasks.rados.rados.0.smithi107.stdout:3587: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:51.428 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 603 (ObjNum 3586 snap 0 seq_num 3586) dirty exists 2020-06-12T02:37:51.428 INFO:tasks.rados.rados.0.smithi107.stdout:3587: left oid 19 (ObjNum 3586 snap 0 seq_num 3586) 2020-06-12T02:37:51.429 INFO:tasks.rados.rados.0.smithi107.stdout:3587: done (1 left) 2020-06-12T02:37:51.429 INFO:tasks.rados.rados.0.smithi107.stdout:3590: done (0 left) 2020-06-12T02:37:51.429 INFO:tasks.rados.rados.0.smithi107.stdout:3591: delete oid 6 current snap is 0 2020-06-12T02:37:51.431 INFO:tasks.rados.rados.0.smithi107.stdout:3591: done (0 left) 2020-06-12T02:37:51.431 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:37:51.431 INFO:tasks.rados.rados.0.smithi107.stdout:3592: seq_num 3591 ranges {0=147456} 2020-06-12T02:37:51.432 INFO:tasks.rados.rados.0.smithi107.stdout:3592: writing smithi10729407-10 from 0 to 147456 tid 1 2020-06-12T02:37:51.432 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:37:51.432 INFO:tasks.rados.rados.0.smithi107.stdout:3593: seq_num 3592 ranges {0=163840} 2020-06-12T02:37:51.433 INFO:tasks.rados.rados.0.smithi107.stdout:3593: writing smithi10729407-15 from 0 to 163840 tid 1 2020-06-12T02:37:51.433 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:37:51.433 INFO:tasks.rados.rados.0.smithi107.stdout:3594: seq_num 3593 ranges {4169728=180224} 2020-06-12T02:37:51.457 INFO:tasks.rados.rados.0.smithi107.stdout:3594: writing smithi10729407-2 from 4169728 to 4349952 tid 1 2020-06-12T02:37:51.457 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:37:51.458 INFO:tasks.rados.rados.0.smithi107.stdout:3595: seq_num 3594 ranges {2170880=196608} 2020-06-12T02:37:51.471 INFO:tasks.rados.rados.0.smithi107.stdout:3595: writing smithi10729407-3 from 2170880 to 2367488 tid 1 2020-06-12T02:37:51.471 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:37:51.472 INFO:tasks.rados.rados.0.smithi107.stdout:3596: seq_num 3595 ranges {0=212992} 2020-06-12T02:37:51.472 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:51.468+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:37:51.473 INFO:tasks.rados.rados.0.smithi107.stdout:3596: writing smithi10729407-20 from 0 to 212992 tid 1 2020-06-12T02:37:51.474 INFO:tasks.rados.rados.0.smithi107.stdout:3597: delete oid 6 current snap is 0 2020-06-12T02:37:51.474 INFO:tasks.rados.rados.0.smithi107.stdout:3592: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:51.474 INFO:tasks.rados.rados.0.smithi107.stdout:3593: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:51.475 INFO:tasks.rados.rados.0.smithi107.stdout:3592: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:51.483 INFO:tasks.rados.rados.0.smithi107.stdout:3592: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:51.483 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 356 (ObjNum 3591 snap 0 seq_num 3591) dirty exists 2020-06-12T02:37:51.484 INFO:tasks.rados.rados.0.smithi107.stdout:3592: left oid 10 (ObjNum 3591 snap 0 seq_num 3591) 2020-06-12T02:37:51.484 INFO:tasks.rados.rados.0.smithi107.stdout:3593: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:51.484 INFO:tasks.rados.rados.0.smithi107.stdout:3593: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:51.484 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 611 (ObjNum 3592 snap 0 seq_num 3592) dirty exists 2020-06-12T02:37:51.485 INFO:tasks.rados.rados.0.smithi107.stdout:3593: left oid 15 (ObjNum 3592 snap 0 seq_num 3592) 2020-06-12T02:37:51.485 INFO:tasks.rados.rados.0.smithi107.stdout:3594: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:51.485 INFO:tasks.rados.rados.0.smithi107.stdout:3594: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:51.485 INFO:tasks.rados.rados.0.smithi107.stdout:3594: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:51.486 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 377 (ObjNum 3593 snap 0 seq_num 3593) dirty exists 2020-06-12T02:37:51.486 INFO:tasks.rados.rados.0.smithi107.stdout:3594: left oid 2 (ObjNum 3593 snap 0 seq_num 3593) 2020-06-12T02:37:51.487 INFO:tasks.rados.rados.0.smithi107.stdout:3595: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:51.487 INFO:tasks.rados.rados.0.smithi107.stdout:3592: done (5 left) 2020-06-12T02:37:51.487 INFO:tasks.rados.rados.0.smithi107.stdout:3593: done (4 left) 2020-06-12T02:37:51.487 INFO:tasks.rados.rados.0.smithi107.stdout:3594: done (3 left) 2020-06-12T02:37:51.487 INFO:tasks.rados.rados.0.smithi107.stdout:3597: done (2 left) 2020-06-12T02:37:51.488 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:37:51.488 INFO:tasks.rados.rados.0.smithi107.stdout:3598: seq_num 3597 ranges {0=245760} 2020-06-12T02:37:51.488 INFO:tasks.rados.rados.0.smithi107.stdout:3598: writing smithi10729407-4 from 0 to 245760 tid 1 2020-06-12T02:37:51.488 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:37:51.488 INFO:tasks.rados.rados.0.smithi107.stdout:3599: seq_num 3598 ranges {5529600=262144} 2020-06-12T02:37:51.489 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:51.489 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:51.498 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-06-12T02:37:51.510 INFO:tasks.rados.rados.0.smithi107.stdout:3599: writing smithi10729407-19 from 5529600 to 5791744 tid 1 2020-06-12T02:37:51.510 INFO:tasks.rados.rados.0.smithi107.stdout:3600: delete oid 10 current snap is 0 2020-06-12T02:37:51.510 INFO:tasks.rados.rados.0.smithi107.stdout:3596: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:51.511 INFO:tasks.rados.rados.0.smithi107.stdout:3595: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:51.511 INFO:tasks.rados.rados.0.smithi107.stdout:3596: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:51.511 INFO:tasks.rados.rados.0.smithi107.stdout:3595: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:51.511 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 641 (ObjNum 3594 snap 0 seq_num 3594) dirty exists 2020-06-12T02:37:51.511 INFO:tasks.rados.rados.0.smithi107.stdout:3595: left oid 3 (ObjNum 3594 snap 0 seq_num 3594) 2020-06-12T02:37:51.512 INFO:tasks.rados.rados.0.smithi107.stdout:3596: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:51.512 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 269 (ObjNum 3595 snap 0 seq_num 3595) dirty exists 2020-06-12T02:37:51.512 INFO:tasks.rados.rados.0.smithi107.stdout:3596: left oid 20 (ObjNum 3595 snap 0 seq_num 3595) 2020-06-12T02:37:51.512 INFO:tasks.rados.rados.0.smithi107.stdout:3598: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:51.512 INFO:tasks.rados.rados.0.smithi107.stdout:3598: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:51.513 INFO:tasks.rados.rados.0.smithi107.stdout:3598: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:51.513 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 673 (ObjNum 3597 snap 0 seq_num 3597) dirty exists 2020-06-12T02:37:51.513 INFO:tasks.rados.rados.0.smithi107.stdout:3598: left oid 4 (ObjNum 3597 snap 0 seq_num 3597) 2020-06-12T02:37:51.514 INFO:tasks.rados.rados.0.smithi107.stdout:3595: done (4 left) 2020-06-12T02:37:51.514 INFO:tasks.rados.rados.0.smithi107.stdout:3596: done (3 left) 2020-06-12T02:37:51.514 INFO:tasks.rados.rados.0.smithi107.stdout:3598: done (2 left) 2020-06-12T02:37:51.514 INFO:tasks.rados.rados.0.smithi107.stdout:3600: done (1 left) 2020-06-12T02:37:51.515 INFO:tasks.rados.rados.0.smithi107.stdout:3601: delete oid 12 current snap is 0 2020-06-12T02:37:51.515 INFO:tasks.rados.rados.0.smithi107.stdout:3599: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:51.515 INFO:tasks.rados.rados.0.smithi107.stdout:3599: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:51.516 INFO:tasks.rados.rados.0.smithi107.stdout:3601: done (1 left) 2020-06-12T02:37:51.516 INFO:tasks.rados.rados.0.smithi107.stdout:3602: delete oid 12 current snap is 0 2020-06-12T02:37:51.516 INFO:tasks.rados.rados.0.smithi107.stdout:3599: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:51.517 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 605 (ObjNum 3598 snap 0 seq_num 3598) dirty exists 2020-06-12T02:37:51.517 INFO:tasks.rados.rados.0.smithi107.stdout:3599: left oid 19 (ObjNum 3598 snap 0 seq_num 3598) 2020-06-12T02:37:51.517 INFO:tasks.rados.rados.0.smithi107.stdout:3599: done (1 left) 2020-06-12T02:37:51.517 INFO:tasks.rados.rados.0.smithi107.stdout:3602: done (0 left) 2020-06-12T02:37:51.517 INFO:tasks.rados.rados.0.smithi107.stdout:3603: delete oid 2 current snap is 0 2020-06-12T02:37:51.518 INFO:tasks.rados.rados.0.smithi107.stdout:3603: done (0 left) 2020-06-12T02:37:51.518 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:37:51.519 INFO:tasks.rados.rados.0.smithi107.stdout:3604: seq_num 3603 ranges {0=352256} 2020-06-12T02:37:51.520 INFO:tasks.rados.rados.0.smithi107.stdout:3604: writing smithi10729407-12 from 0 to 352256 tid 1 2020-06-12T02:37:51.521 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:37:51.521 INFO:tasks.rados.rados.0.smithi107.stdout:3605: seq_num 3604 ranges {212992=368640} 2020-06-12T02:37:51.524 INFO:tasks.rados.rados.0.smithi107.stdout:3605: writing smithi10729407-20 from 212992 to 581632 tid 1 2020-06-12T02:37:51.525 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:37:51.525 INFO:tasks.rados.rados.0.smithi107.stdout:3606: seq_num 3605 ranges {0=385024} 2020-06-12T02:37:51.527 INFO:tasks.rados.rados.0.smithi107.stdout:3606: writing smithi10729407-6 from 0 to 385024 tid 1 2020-06-12T02:37:51.527 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:37:51.527 INFO:tasks.rados.rados.0.smithi107.stdout:3607: seq_num 3606 ranges {0=401408} 2020-06-12T02:37:51.530 INFO:tasks.rados.rados.0.smithi107.stdout:3607: writing smithi10729407-5 from 0 to 401408 tid 1 2020-06-12T02:37:51.530 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:37:51.530 INFO:tasks.rados.rados.0.smithi107.stdout:3608: seq_num 3607 ranges {2367488=417792} 2020-06-12T02:37:51.546 INFO:tasks.rados.rados.0.smithi107.stdout:3608: writing smithi10729407-3 from 2367488 to 2785280 tid 1 2020-06-12T02:37:51.546 INFO:tasks.rados.rados.0.smithi107.stdout:3609: delete oid 16 current snap is 0 2020-06-12T02:37:51.546 INFO:tasks.rados.rados.0.smithi107.stdout:3604: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:51.546 INFO:tasks.rados.rados.0.smithi107.stdout:3604: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:51.547 INFO:tasks.rados.rados.0.smithi107.stdout:3604: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:51.547 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 609 (ObjNum 3603 snap 0 seq_num 3603) dirty exists 2020-06-12T02:37:51.547 INFO:tasks.rados.rados.0.smithi107.stdout:3604: left oid 12 (ObjNum 3603 snap 0 seq_num 3603) 2020-06-12T02:37:51.547 INFO:tasks.rados.rados.0.smithi107.stdout:3605: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:51.547 INFO:tasks.rados.rados.0.smithi107.stdout:3605: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:51.547 INFO:tasks.rados.rados.0.smithi107.stdout:3605: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:51.548 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 271 (ObjNum 3604 snap 0 seq_num 3604) dirty exists 2020-06-12T02:37:51.548 INFO:tasks.rados.rados.0.smithi107.stdout:3605: left oid 20 (ObjNum 3604 snap 0 seq_num 3604) 2020-06-12T02:37:51.548 INFO:tasks.rados.rados.0.smithi107.stdout:3606: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:51.548 INFO:tasks.rados.rados.0.smithi107.stdout:3606: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:51.548 INFO:tasks.rados.rados.0.smithi107.stdout:3607: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:51.549 INFO:tasks.rados.rados.0.smithi107.stdout:3607: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:51.549 INFO:tasks.rados.rados.0.smithi107.stdout:3607: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:51.549 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 315 (ObjNum 3606 snap 0 seq_num 3606) dirty exists 2020-06-12T02:37:51.549 INFO:tasks.rados.rados.0.smithi107.stdout:3607: left oid 5 (ObjNum 3606 snap 0 seq_num 3606) 2020-06-12T02:37:51.549 INFO:tasks.rados.rados.0.smithi107.stdout:3606: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:51.550 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 677 (ObjNum 3605 snap 0 seq_num 3605) dirty exists 2020-06-12T02:37:51.550 INFO:tasks.rados.rados.0.smithi107.stdout:3606: left oid 6 (ObjNum 3605 snap 0 seq_num 3605) 2020-06-12T02:37:51.551 INFO:tasks.rados.rados.0.smithi107.stdout:3608: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:51.552 INFO:tasks.rados.rados.0.smithi107.stdout:3608: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:51.552 INFO:tasks.rados.rados.0.smithi107.stdout:3604: done (5 left) 2020-06-12T02:37:51.553 INFO:tasks.rados.rados.0.smithi107.stdout:3605: done (4 left) 2020-06-12T02:37:51.553 INFO:tasks.rados.rados.0.smithi107.stdout:3606: done (3 left) 2020-06-12T02:37:51.553 INFO:tasks.rados.rados.0.smithi107.stdout:3607: done (2 left) 2020-06-12T02:37:51.560 INFO:tasks.rados.rados.0.smithi107.stdout:3609: done (1 left) 2020-06-12T02:37:51.567 INFO:tasks.rados.rados.0.smithi107.stdout:3610: delete oid 11 current snap is 0 2020-06-12T02:37:51.569 INFO:tasks.rados.rados.0.smithi107.stdout:3610: done (1 left) 2020-06-12T02:37:51.569 INFO:tasks.rados.rados.0.smithi107.stdout:3611: delete oid 6 current snap is 0 2020-06-12T02:37:51.569 INFO:tasks.rados.rados.0.smithi107.stdout:3608: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:51.570 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 643 (ObjNum 3607 snap 0 seq_num 3607) dirty exists 2020-06-12T02:37:51.570 INFO:tasks.rados.rados.0.smithi107.stdout:3608: left oid 3 (ObjNum 3607 snap 0 seq_num 3607) 2020-06-12T02:37:51.570 INFO:tasks.rados.rados.0.smithi107.stdout:3608: done (1 left) 2020-06-12T02:37:51.571 INFO:tasks.rados.rados.0.smithi107.stdout:3611: done (0 left) 2020-06-12T02:37:51.571 INFO:tasks.rados.rados.0.smithi107.stdout:3612: delete oid 11 current snap is 0 2020-06-12T02:37:51.571 INFO:tasks.rados.rados.0.smithi107.stdout:3612: done (0 left) 2020-06-12T02:37:51.572 INFO:tasks.rados.rados.0.smithi107.stdout:3613: delete oid 5 current snap is 0 2020-06-12T02:37:51.572 INFO:tasks.rados.rados.0.smithi107.stdout:3613: done (0 left) 2020-06-12T02:37:51.572 INFO:tasks.rados.rados.0.smithi107.stdout:3614: delete oid 18 current snap is 0 2020-06-12T02:37:51.573 INFO:tasks.rados.rados.0.smithi107.stdout:3614: done (0 left) 2020-06-12T02:37:51.573 INFO:tasks.rados.rados.0.smithi107.stdout:3615: delete oid 6 current snap is 0 2020-06-12T02:37:51.573 INFO:tasks.rados.rados.0.smithi107.stdout:3615: done (0 left) 2020-06-12T02:37:51.573 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:37:51.574 INFO:tasks.rados.rados.0.smithi107.stdout:3616: seq_num 3615 ranges {0=548864} 2020-06-12T02:37:51.577 INFO:tasks.rados.rados.0.smithi107.stdout:3616: writing smithi10729407-10 from 0 to 548864 tid 1 2020-06-12T02:37:51.577 INFO:tasks.rados.rados.0.smithi107.stdout:3617: delete oid 19 current snap is 0 2020-06-12T02:37:51.577 INFO:tasks.rados.rados.0.smithi107.stdout:3617: done (1 left) 2020-06-12T02:37:51.578 INFO:tasks.rados.rados.0.smithi107.stdout:3618: delete oid 2 current snap is 0 2020-06-12T02:37:51.578 INFO:tasks.rados.rados.0.smithi107.stdout:3616: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:51.578 INFO:tasks.rados.rados.0.smithi107.stdout:3618: done (1 left) 2020-06-12T02:37:51.578 INFO:tasks.rados.rados.0.smithi107.stdout:3619: delete oid 4 current snap is 0 2020-06-12T02:37:51.578 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:51.573+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:51.579 INFO:tasks.rados.rados.0.smithi107.stdout:3616: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:51.580 INFO:tasks.rados.rados.0.smithi107.stdout:3616: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:51.580 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 359 (ObjNum 3615 snap 0 seq_num 3615) dirty exists 2020-06-12T02:37:51.580 INFO:tasks.rados.rados.0.smithi107.stdout:3616: left oid 10 (ObjNum 3615 snap 0 seq_num 3615) 2020-06-12T02:37:51.580 INFO:tasks.rados.rados.0.smithi107.stdout:3616: done (1 left) 2020-06-12T02:37:51.580 INFO:tasks.rados.rados.0.smithi107.stdout:3619: done (0 left) 2020-06-12T02:37:51.581 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:37:51.581 INFO:tasks.rados.rados.0.smithi107.stdout:3620: seq_num 3619 ranges {0=614400} 2020-06-12T02:37:51.583 INFO:tasks.rados.rados.0.smithi107.stdout:3620: writing smithi10729407-5 from 0 to 614400 tid 1 2020-06-12T02:37:51.583 INFO:tasks.rados.rados.0.smithi107.stdout:3621: delete oid 4 current snap is 0 2020-06-12T02:37:51.585 INFO:tasks.rados.rados.0.smithi107.stdout:3621: done (1 left) 2020-06-12T02:37:51.585 INFO:tasks.rados.rados.0.smithi107.stdout:3622: delete oid 18 current snap is 0 2020-06-12T02:37:51.587 INFO:tasks.rados.rados.0.smithi107.stdout:3622: done (1 left) 2020-06-12T02:37:51.588 INFO:tasks.rados.rados.0.smithi107.stdout:3623: delete oid 1 current snap is 0 2020-06-12T02:37:51.588 INFO:tasks.rados.rados.0.smithi107.stdout:3620: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:51.589 INFO:tasks.rados.rados.0.smithi107.stdout:3620: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:51.590 INFO:tasks.rados.rados.0.smithi107.stdout:3620: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:51.590 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 318 (ObjNum 3619 snap 0 seq_num 3619) dirty exists 2020-06-12T02:37:51.590 INFO:tasks.rados.rados.0.smithi107.stdout:3620: left oid 5 (ObjNum 3619 snap 0 seq_num 3619) 2020-06-12T02:37:51.590 INFO:tasks.rados.rados.0.smithi107.stdout:3620: done (1 left) 2020-06-12T02:37:51.590 INFO:tasks.rados.rados.0.smithi107.stdout:3623: done (0 left) 2020-06-12T02:37:51.591 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:37:51.591 INFO:tasks.rados.rados.0.smithi107.stdout:3624: seq_num 3623 ranges {0=688128} 2020-06-12T02:37:51.594 INFO:tasks.rados.rados.0.smithi107.stdout:3624: writing smithi10729407-19 from 0 to 688128 tid 1 2020-06-12T02:37:51.594 INFO:tasks.rados.rados.0.smithi107.stdout:3625: delete oid 15 current snap is 0 2020-06-12T02:37:51.596 INFO:tasks.rados.rados.0.smithi107.stdout:3625: done (1 left) 2020-06-12T02:37:51.596 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:37:51.596 INFO:tasks.rados.rados.0.smithi107.stdout:3626: seq_num 3625 ranges {0=720896} 2020-06-12T02:37:51.601 INFO:tasks.rados.rados.0.smithi107.stdout:3626: writing smithi10729407-11 from 0 to 720896 tid 1 2020-06-12T02:37:51.602 INFO:tasks.rados.rados.0.smithi107.stdout:3627: delete oid 17 current snap is 0 2020-06-12T02:37:51.602 INFO:tasks.rados.rados.0.smithi107.stdout:3624: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:51.602 INFO:tasks.rados.rados.0.smithi107.stdout:3624: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:51.603 INFO:tasks.rados.rados.0.smithi107.stdout:3627: done (2 left) 2020-06-12T02:37:51.603 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:37:51.604 INFO:tasks.rados.rados.0.smithi107.stdout:3628: seq_num 3627 ranges {0=753664} 2020-06-12T02:37:51.608 INFO:tasks.rados.rados.0.smithi107.stdout:3628: writing smithi10729407-9 from 0 to 753664 tid 1 2020-06-12T02:37:51.609 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:37:51.609 INFO:tasks.rados.rados.0.smithi107.stdout:3629: seq_num 3628 ranges {352256=770048} 2020-06-12T02:37:51.618 INFO:tasks.rados.rados.0.smithi107.stdout:3629: writing smithi10729407-12 from 352256 to 1122304 tid 1 2020-06-12T02:37:51.618 INFO:tasks.rados.rados.0.smithi107.stdout:3630: delete oid 15 current snap is 0 2020-06-12T02:37:51.618 INFO:tasks.rados.rados.0.smithi107.stdout:3624: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:51.618 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 612 (ObjNum 3623 snap 0 seq_num 3623) dirty exists 2020-06-12T02:37:51.619 INFO:tasks.rados.rados.0.smithi107.stdout:3624: left oid 19 (ObjNum 3623 snap 0 seq_num 3623) 2020-06-12T02:37:51.619 INFO:tasks.rados.rados.0.smithi107.stdout:3626: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:51.619 INFO:tasks.rados.rados.0.smithi107.stdout:3626: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:51.619 INFO:tasks.rados.rados.0.smithi107.stdout:3628: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:51.619 INFO:tasks.rados.rados.0.smithi107.stdout:3626: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:51.620 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 635 (ObjNum 3625 snap 0 seq_num 3625) dirty exists 2020-06-12T02:37:51.620 INFO:tasks.rados.rados.0.smithi107.stdout:3626: left oid 11 (ObjNum 3625 snap 0 seq_num 3625) 2020-06-12T02:37:51.620 INFO:tasks.rados.rados.0.smithi107.stdout:3624: done (4 left) 2020-06-12T02:37:51.621 INFO:tasks.rados.rados.0.smithi107.stdout:3626: done (3 left) 2020-06-12T02:37:51.621 INFO:tasks.rados.rados.0.smithi107.stdout:3630: done (2 left) 2020-06-12T02:37:51.621 INFO:tasks.rados.rados.0.smithi107.stdout:3631: delete oid 5 current snap is 0 2020-06-12T02:37:51.621 INFO:tasks.rados.rados.0.smithi107.stdout:3628: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:51.622 INFO:tasks.rados.rados.0.smithi107.stdout:3628: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:51.622 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 637 (ObjNum 3627 snap 0 seq_num 3627) dirty exists 2020-06-12T02:37:51.622 INFO:tasks.rados.rados.0.smithi107.stdout:3628: left oid 9 (ObjNum 3627 snap 0 seq_num 3627) 2020-06-12T02:37:51.623 INFO:tasks.rados.rados.0.smithi107.stdout:3628: done (2 left) 2020-06-12T02:37:51.623 INFO:tasks.rados.rados.0.smithi107.stdout:3631: done (1 left) 2020-06-12T02:37:51.623 INFO:tasks.rados.rados.0.smithi107.stdout:3632: delete oid 5 current snap is 0 2020-06-12T02:37:51.628 INFO:tasks.rados.rados.0.smithi107.stdout:3629: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:51.628 INFO:tasks.rados.rados.0.smithi107.stdout:3629: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:51.629 INFO:tasks.rados.rados.0.smithi107.stdout:3632: done (1 left) 2020-06-12T02:37:51.629 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:37:51.629 INFO:tasks.rados.rados.0.smithi107.stdout:3633: seq_num 3632 ranges {0=434176,434176=401408} 2020-06-12T02:37:51.632 INFO:tasks.rados.rados.0.smithi107.stdout:3633: writing smithi10729407-16 from 0 to 434176 tid 1 2020-06-12T02:37:51.634 INFO:tasks.rados.rados.0.smithi107.stdout:3633: writing smithi10729407-16 from 434176 to 835584 tid 2 2020-06-12T02:37:51.634 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:37:51.634 INFO:tasks.rados.rados.0.smithi107.stdout:3634: seq_num 3633 ranges {0=450560,450560=401408} 2020-06-12T02:37:51.637 INFO:tasks.rados.rados.0.smithi107.stdout:3634: writing smithi10729407-5 from 0 to 450560 tid 1 2020-06-12T02:37:51.640 INFO:tasks.rados.rados.0.smithi107.stdout:3634: writing smithi10729407-5 from 450560 to 851968 tid 2 2020-06-12T02:37:51.640 INFO:tasks.rados.rados.0.smithi107.stdout:3629: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:51.640 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 614 (ObjNum 3628 snap 0 seq_num 3628) dirty exists 2020-06-12T02:37:51.640 INFO:tasks.rados.rados.0.smithi107.stdout:3629: left oid 12 (ObjNum 3628 snap 0 seq_num 3628) 2020-06-12T02:37:51.641 INFO:tasks.rados.rados.0.smithi107.stdout:3629: done (2 left) 2020-06-12T02:37:51.641 INFO:tasks.rados.rados.0.smithi107.stdout:3635: delete oid 2 current snap is 0 2020-06-12T02:37:51.642 INFO:tasks.rados.rados.0.smithi107.stdout:3633: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:51.642 INFO:tasks.rados.rados.0.smithi107.stdout:3633: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:51.642 INFO:tasks.rados.rados.0.smithi107.stdout:3633: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:51.642 INFO:tasks.rados.rados.0.smithi107.stdout:3634: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:51.651 INFO:tasks.rados.rados.0.smithi107.stdout:3633: finishing write tid 5 to smithi10729407-16 2020-06-12T02:37:51.652 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 647 (ObjNum 3632 snap 0 seq_num 3632) dirty exists 2020-06-12T02:37:51.652 INFO:tasks.rados.rados.0.smithi107.stdout:3633: left oid 16 (ObjNum 3632 snap 0 seq_num 3632) 2020-06-12T02:37:51.652 INFO:tasks.rados.rados.0.smithi107.stdout:3633: done (2 left) 2020-06-12T02:37:51.652 INFO:tasks.rados.rados.0.smithi107.stdout:3635: done (1 left) 2020-06-12T02:37:51.653 INFO:tasks.rados.rados.0.smithi107.stdout:3636: delete oid 15 current snap is 0 2020-06-12T02:37:51.653 INFO:tasks.rados.rados.0.smithi107.stdout:3636: done (1 left) 2020-06-12T02:37:51.653 INFO:tasks.rados.rados.0.smithi107.stdout:3637: delete oid 11 current snap is 0 2020-06-12T02:37:51.653 INFO:tasks.rados.rados.0.smithi107.stdout:3634: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:51.653 INFO:tasks.rados.rados.0.smithi107.stdout:3634: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:51.654 INFO:tasks.rados.rados.0.smithi107.stdout:3637: done (1 left) 2020-06-12T02:37:51.654 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:37:51.654 INFO:tasks.rados.rados.0.smithi107.stdout:3638: seq_num 3637 ranges {548864=516096,1064960=401408} 2020-06-12T02:37:51.654 INFO:tasks.rados.rados.0.smithi107.stdout:3638: writing smithi10729407-10 from 548864 to 1064960 tid 1 2020-06-12T02:37:51.656 INFO:tasks.rados.rados.0.smithi107.stdout:3638: writing smithi10729407-10 from 1064960 to 1466368 tid 2 2020-06-12T02:37:51.657 INFO:tasks.rados.rados.0.smithi107.stdout:3639: delete oid 1 current snap is 0 2020-06-12T02:37:51.657 INFO:tasks.rados.rados.0.smithi107.stdout:3634: finishing write tid 5 to smithi10729407-5 2020-06-12T02:37:51.657 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 323 (ObjNum 3633 snap 0 seq_num 3633) dirty exists 2020-06-12T02:37:51.657 INFO:tasks.rados.rados.0.smithi107.stdout:3634: left oid 5 (ObjNum 3633 snap 0 seq_num 3633) 2020-06-12T02:37:51.658 INFO:tasks.rados.rados.0.smithi107.stdout:3634: done (2 left) 2020-06-12T02:37:51.659 INFO:tasks.rados.rados.0.smithi107.stdout:3639: done (1 left) 2020-06-12T02:37:51.659 INFO:tasks.rados.rados.0.smithi107.stdout:3640: delete oid 6 current snap is 0 2020-06-12T02:37:51.661 INFO:tasks.rados.rados.0.smithi107.stdout:3638: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:51.661 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:51.662 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:51.662 INFO:tasks.rados.rados.0.smithi107.stdout:3638: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:51.663 INFO:tasks.rados.rados.0.smithi107.stdout:3638: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:51.663 INFO:tasks.rados.rados.0.smithi107.stdout:3640: done (1 left) 2020-06-12T02:37:51.664 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:37:51.664 INFO:tasks.rados.rados.0.smithi107.stdout:3641: seq_num 3640 ranges {851968=565248,1417216=401408} 2020-06-12T02:37:51.671 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-06-12T02:37:51.672 INFO:tasks.rados.rados.0.smithi107.stdout:3641: writing smithi10729407-5 from 851968 to 1417216 tid 1 2020-06-12T02:37:51.675 INFO:tasks.rados.rados.0.smithi107.stdout:3641: writing smithi10729407-5 from 1417216 to 1818624 tid 2 2020-06-12T02:37:51.675 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:37:51.675 INFO:tasks.rados.rados.0.smithi107.stdout:3642: seq_num 3641 ranges {2785280=589824,3375104=401408} 2020-06-12T02:37:51.677 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:51.675+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:51.695 INFO:tasks.rados.rados.0.smithi107.stdout:3642: writing smithi10729407-3 from 2785280 to 3375104 tid 1 2020-06-12T02:37:51.696 INFO:tasks.rados.rados.0.smithi107.stdout:3642: writing smithi10729407-3 from 3375104 to 3776512 tid 2 2020-06-12T02:37:51.696 INFO:tasks.rados.rados.0.smithi107.stdout:3643: delete oid 2 current snap is 0 2020-06-12T02:37:51.696 INFO:tasks.rados.rados.0.smithi107.stdout:3638: finishing write tid 5 to smithi10729407-10 2020-06-12T02:37:51.697 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 362 (ObjNum 3637 snap 0 seq_num 3637) dirty exists 2020-06-12T02:37:51.697 INFO:tasks.rados.rados.0.smithi107.stdout:3638: left oid 10 (ObjNum 3637 snap 0 seq_num 3637) 2020-06-12T02:37:51.697 INFO:tasks.rados.rados.0.smithi107.stdout:3641: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:51.697 INFO:tasks.rados.rados.0.smithi107.stdout:3641: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:51.697 INFO:tasks.rados.rados.0.smithi107.stdout:3641: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:51.698 INFO:tasks.rados.rados.0.smithi107.stdout:3641: finishing write tid 5 to smithi10729407-5 2020-06-12T02:37:51.698 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 326 (ObjNum 3640 snap 0 seq_num 3640) dirty exists 2020-06-12T02:37:51.698 INFO:tasks.rados.rados.0.smithi107.stdout:3641: left oid 5 (ObjNum 3640 snap 0 seq_num 3640) 2020-06-12T02:37:51.699 INFO:tasks.rados.rados.0.smithi107.stdout:3638: done (3 left) 2020-06-12T02:37:51.699 INFO:tasks.rados.rados.0.smithi107.stdout:3641: done (2 left) 2020-06-12T02:37:51.699 INFO:tasks.rados.rados.0.smithi107.stdout:3643: done (1 left) 2020-06-12T02:37:51.699 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:37:51.700 INFO:tasks.rados.rados.0.smithi107.stdout:3644: seq_num 3643 ranges {1818624=622592,2441216=401408} 2020-06-12T02:37:51.711 INFO:tasks.rados.rados.0.smithi107.stdout:3644: writing smithi10729407-5 from 1818624 to 2441216 tid 1 2020-06-12T02:37:51.713 INFO:tasks.rados.rados.0.smithi107.stdout:3644: writing smithi10729407-5 from 2441216 to 2842624 tid 2 2020-06-12T02:37:51.713 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:37:51.714 INFO:tasks.rados.rados.0.smithi107.stdout:3645: seq_num 3644 ranges {0=638976,638976=401408} 2020-06-12T02:37:51.717 INFO:tasks.rados.rados.0.smithi107.stdout:3645: writing smithi10729407-11 from 0 to 638976 tid 1 2020-06-12T02:37:51.720 INFO:tasks.rados.rados.0.smithi107.stdout:3645: writing smithi10729407-11 from 638976 to 1040384 tid 2 2020-06-12T02:37:51.720 INFO:tasks.rados.rados.0.smithi107.stdout:3646: delete oid 12 current snap is 0 2020-06-12T02:37:51.721 INFO:tasks.rados.rados.0.smithi107.stdout:3642: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:51.721 INFO:tasks.rados.rados.0.smithi107.stdout:3642: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:51.721 INFO:tasks.rados.rados.0.smithi107.stdout:3642: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:51.721 INFO:tasks.rados.rados.0.smithi107.stdout:3642: finishing write tid 5 to smithi10729407-3 2020-06-12T02:37:51.721 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 650 (ObjNum 3641 snap 0 seq_num 3641) dirty exists 2020-06-12T02:37:51.722 INFO:tasks.rados.rados.0.smithi107.stdout:3642: left oid 3 (ObjNum 3641 snap 0 seq_num 3641) 2020-06-12T02:37:51.722 INFO:tasks.rados.rados.0.smithi107.stdout:3644: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:51.722 INFO:tasks.rados.rados.0.smithi107.stdout:3644: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:51.723 INFO:tasks.rados.rados.0.smithi107.stdout:3644: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:51.723 INFO:tasks.rados.rados.0.smithi107.stdout:3642: done (3 left) 2020-06-12T02:37:51.723 INFO:tasks.rados.rados.0.smithi107.stdout:3646: done (2 left) 2020-06-12T02:37:51.723 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:37:51.724 INFO:tasks.rados.rados.0.smithi107.stdout:3647: seq_num 3646 ranges {0=671744,671744=401408} 2020-06-12T02:37:51.726 INFO:tasks.rados.rados.0.smithi107.stdout:3647: writing smithi10729407-8 from 0 to 671744 tid 1 2020-06-12T02:37:51.728 INFO:tasks.rados.rados.0.smithi107.stdout:3647: writing smithi10729407-8 from 671744 to 1073152 tid 2 2020-06-12T02:37:51.729 INFO:tasks.rados.rados.0.smithi107.stdout:3648: delete oid 18 current snap is 0 2020-06-12T02:37:51.732 INFO:tasks.rados.rados.0.smithi107.stdout:3644: finishing write tid 5 to smithi10729407-5 2020-06-12T02:37:51.732 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 329 (ObjNum 3643 snap 0 seq_num 3643) dirty exists 2020-06-12T02:37:51.732 INFO:tasks.rados.rados.0.smithi107.stdout:3644: left oid 5 (ObjNum 3643 snap 0 seq_num 3643) 2020-06-12T02:37:51.732 INFO:tasks.rados.rados.0.smithi107.stdout:3645: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:51.733 INFO:tasks.rados.rados.0.smithi107.stdout:3645: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:51.733 INFO:tasks.rados.rados.0.smithi107.stdout:3645: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:51.733 INFO:tasks.rados.rados.0.smithi107.stdout:3645: finishing write tid 5 to smithi10729407-11 2020-06-12T02:37:51.733 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 639 (ObjNum 3644 snap 0 seq_num 3644) dirty exists 2020-06-12T02:37:51.733 INFO:tasks.rados.rados.0.smithi107.stdout:3645: left oid 11 (ObjNum 3644 snap 0 seq_num 3644) 2020-06-12T02:37:51.734 INFO:tasks.rados.rados.0.smithi107.stdout:3644: done (3 left) 2020-06-12T02:37:51.734 INFO:tasks.rados.rados.0.smithi107.stdout:3645: done (2 left) 2020-06-12T02:37:51.734 INFO:tasks.rados.rados.0.smithi107.stdout:3648: done (1 left) 2020-06-12T02:37:51.734 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:51.735 INFO:tasks.rados.rados.0.smithi107.stdout:3649: seq_num 3648 ranges {0=704512,704512=401408} 2020-06-12T02:37:51.736 INFO:tasks.rados.rados.0.smithi107.stdout:3649: writing smithi10729407-2 from 0 to 704512 tid 1 2020-06-12T02:37:51.738 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-06-12T02:37:51.739 INFO:tasks.rados.rados.0.smithi107.stdout:3649: writing smithi10729407-2 from 704512 to 1105920 tid 2 2020-06-12T02:37:51.739 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:37:51.739 INFO:tasks.rados.rados.0.smithi107.stdout:3650: seq_num 3649 ranges {3776512=720896,4497408=401408} 2020-06-12T02:37:51.772 INFO:tasks.rados.rados.0.smithi107.stdout:3650: writing smithi10729407-3 from 3776512 to 4497408 tid 1 2020-06-12T02:37:51.772 INFO:tasks.rados.rados.0.smithi107.stdout:3650: writing smithi10729407-3 from 4497408 to 4898816 tid 2 2020-06-12T02:37:51.772 INFO:tasks.rados.rados.0.smithi107.stdout:3651: delete oid 19 current snap is 0 2020-06-12T02:37:51.773 INFO:tasks.rados.rados.0.smithi107.stdout:3647: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:51.773 INFO:tasks.rados.rados.0.smithi107.stdout:3647: finishing write tid 2 to smithi10729407-8 2020-06-12T02:37:51.773 INFO:tasks.rados.rados.0.smithi107.stdout:3647: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:51.773 INFO:tasks.rados.rados.0.smithi107.stdout:3647: finishing write tid 5 to smithi10729407-8 2020-06-12T02:37:51.773 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 617 (ObjNum 3646 snap 0 seq_num 3646) dirty exists 2020-06-12T02:37:51.774 INFO:tasks.rados.rados.0.smithi107.stdout:3647: left oid 8 (ObjNum 3646 snap 0 seq_num 3646) 2020-06-12T02:37:51.774 INFO:tasks.rados.rados.0.smithi107.stdout:3649: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:51.774 INFO:tasks.rados.rados.0.smithi107.stdout:3649: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:51.774 INFO:tasks.rados.rados.0.smithi107.stdout:3649: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:51.774 INFO:tasks.rados.rados.0.smithi107.stdout:3649: finishing write tid 5 to smithi10729407-2 2020-06-12T02:37:51.774 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 384 (ObjNum 3648 snap 0 seq_num 3648) dirty exists 2020-06-12T02:37:51.775 INFO:tasks.rados.rados.0.smithi107.stdout:3649: left oid 2 (ObjNum 3648 snap 0 seq_num 3648) 2020-06-12T02:37:51.775 INFO:tasks.rados.rados.0.smithi107.stdout:3647: done (3 left) 2020-06-12T02:37:51.775 INFO:tasks.rados.rados.0.smithi107.stdout:3649: done (2 left) 2020-06-12T02:37:51.775 INFO:tasks.rados.rados.0.smithi107.stdout:3651: done (1 left) 2020-06-12T02:37:51.775 INFO:tasks.rados.rados.0.smithi107.stdout:3652: delete oid 13 current snap is 0 2020-06-12T02:37:51.776 INFO:tasks.rados.rados.0.smithi107.stdout:3650: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:51.776 INFO:tasks.rados.rados.0.smithi107.stdout:3652: done (1 left) 2020-06-12T02:37:51.777 INFO:tasks.rados.rados.0.smithi107.stdout:3653: delete oid 5 current snap is 0 2020-06-12T02:37:51.777 INFO:tasks.rados.rados.0.smithi107.stdout:3650: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:51.777 INFO:tasks.rados.rados.0.smithi107.stdout:3650: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:51.780 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:37:51.776+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:37:51.780 INFO:tasks.rados.rados.0.smithi107.stdout:3650: finishing write tid 5 to smithi10729407-3 2020-06-12T02:37:51.780 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 653 (ObjNum 3649 snap 0 seq_num 3649) dirty exists 2020-06-12T02:37:51.781 INFO:tasks.rados.rados.0.smithi107.stdout:3650: left oid 3 (ObjNum 3649 snap 0 seq_num 3649) 2020-06-12T02:37:51.781 INFO:tasks.rados.rados.0.smithi107.stdout:3650: done (1 left) 2020-06-12T02:37:51.781 INFO:tasks.rados.rados.0.smithi107.stdout:3653: done (0 left) 2020-06-12T02:37:51.782 INFO:tasks.rados.rados.0.smithi107.stdout:3654: delete oid 20 current snap is 0 2020-06-12T02:37:51.782 INFO:tasks.rados.rados.0.smithi107.stdout:3654: done (0 left) 2020-06-12T02:37:51.782 INFO:tasks.rados.rados.0.smithi107.stdout:3655: delete oid 16 current snap is 0 2020-06-12T02:37:51.784 INFO:tasks.rados.rados.0.smithi107.stdout:3655: done (0 left) 2020-06-12T02:37:51.784 INFO:tasks.rados.rados.0.smithi107.stdout:3656: delete oid 2 current snap is 0 2020-06-12T02:37:51.786 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [7] live_osds: [0, 1, 2, 3, 4, 5, 6] 2020-06-12T02:37:51.786 INFO:tasks.thrashosds.thrasher:Reviving osd 7 2020-06-12T02:37:51.786 INFO:tasks.ceph.osd.7:Restarting daemon 2020-06-12T02:37:51.786 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2020-06-12T02:37:51.788 INFO:tasks.rados.rados.0.smithi107.stdout:3656: done (0 left) 2020-06-12T02:37:51.788 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:37:51.789 INFO:tasks.rados.rados.0.smithi107.stdout:3657: seq_num 3656 ranges {4898816=434176,5332992=573440,5906432=229376} 2020-06-12T02:37:51.791 INFO:tasks.ceph.osd.7:Started 2020-06-12T02:37:51.791 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-06-12T02:37:51.824 INFO:tasks.rados.rados.0.smithi107.stdout:3657: writing smithi10729407-3 from 4898816 to 5332992 tid 1 2020-06-12T02:37:51.824 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:51.824 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:51.826 INFO:tasks.rados.rados.0.smithi107.stdout:3657: writing smithi10729407-3 from 5332992 to 5906432 tid 2 2020-06-12T02:37:51.826 INFO:tasks.rados.rados.0.smithi107.stdout:3657: writing smithi10729407-3 from 5906432 to 6135808 tid 3 2020-06-12T02:37:51.827 INFO:tasks.rados.rados.0.smithi107.stdout:3658: delete oid 17 current snap is 0 2020-06-12T02:37:51.827 INFO:tasks.rados.rados.0.smithi107.stdout:3657: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:51.827 INFO:tasks.rados.rados.0.smithi107.stdout:3658: done (1 left) 2020-06-12T02:37:51.827 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:37:51.827 INFO:tasks.rados.rados.0.smithi107.stdout:3659: seq_num 3658 ranges {0=466944,466944=786432,1253376=16384} 2020-06-12T02:37:51.828 INFO:tasks.rados.rados.0.smithi107.stdout:3659: writing smithi10729407-17 from 0 to 466944 tid 1 2020-06-12T02:37:51.829 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-06-12T02:37:51.831 INFO:tasks.rados.rados.0.smithi107.stdout:3659: writing smithi10729407-17 from 466944 to 1253376 tid 2 2020-06-12T02:37:51.832 INFO:tasks.rados.rados.0.smithi107.stdout:3659: writing smithi10729407-17 from 1253376 to 1269760 tid 3 2020-06-12T02:37:51.832 INFO:tasks.rados.rados.0.smithi107.stdout:3660: delete oid 9 current snap is 0 2020-06-12T02:37:51.833 INFO:tasks.rados.rados.0.smithi107.stdout:3657: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:51.833 INFO:tasks.rados.rados.0.smithi107.stdout:3657: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:51.834 INFO:tasks.rados.rados.0.smithi107.stdout:3659: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:51.835 INFO:tasks.rados.rados.0.smithi107.stdout:3660: done (2 left) 2020-06-12T02:37:51.836 INFO:tasks.rados.rados.0.smithi107.stdout:3661: delete oid 5 current snap is 0 2020-06-12T02:37:51.836 INFO:tasks.rados.rados.0.smithi107.stdout:3657: finishing write tid 5 to smithi10729407-3 2020-06-12T02:37:51.838 INFO:tasks.rados.rados.0.smithi107.stdout:3657: finishing write tid 6 to smithi10729407-3 2020-06-12T02:37:51.838 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 658 (ObjNum 3656 snap 0 seq_num 3656) dirty exists 2020-06-12T02:37:51.839 INFO:tasks.rados.rados.0.smithi107.stdout:3657: left oid 3 (ObjNum 3656 snap 0 seq_num 3656) 2020-06-12T02:37:51.839 INFO:tasks.rados.rados.0.smithi107.stdout:3657: done (2 left) 2020-06-12T02:37:51.839 INFO:tasks.rados.rados.0.smithi107.stdout:3661: done (1 left) 2020-06-12T02:37:51.839 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:37:51.839 INFO:tasks.rados.rados.0.smithi107.stdout:3662: seq_num 3661 ranges {0=524288,524288=442368,966656=360448} 2020-06-12T02:37:51.841 INFO:tasks.rados.rados.0.smithi107.stdout:3662: writing smithi10729407-20 from 0 to 524288 tid 1 2020-06-12T02:37:51.844 INFO:tasks.rados.rados.0.smithi107.stdout:3662: writing smithi10729407-20 from 524288 to 966656 tid 2 2020-06-12T02:37:51.849 INFO:tasks.rados.rados.0.smithi107.stdout:3662: writing smithi10729407-20 from 966656 to 1327104 tid 3 2020-06-12T02:37:51.849 INFO:tasks.rados.rados.0.smithi107.stdout:3663: delete oid 14 current snap is 0 2020-06-12T02:37:51.849 INFO:tasks.rados.rados.0.smithi107.stdout:3659: finishing write tid 2 to smithi10729407-17 2020-06-12T02:37:51.849 INFO:tasks.rados.rados.0.smithi107.stdout:3659: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:51.850 INFO:tasks.rados.rados.0.smithi107.stdout:3659: finishing write tid 5 to smithi10729407-17 2020-06-12T02:37:51.850 INFO:tasks.rados.rados.0.smithi107.stdout:3659: finishing write tid 6 to smithi10729407-17 2020-06-12T02:37:51.850 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 681 (ObjNum 3658 snap 0 seq_num 3658) dirty exists 2020-06-12T02:37:51.850 INFO:tasks.rados.rados.0.smithi107.stdout:3659: left oid 17 (ObjNum 3658 snap 0 seq_num 3658) 2020-06-12T02:37:51.851 INFO:tasks.rados.rados.0.smithi107.stdout:3659: done (2 left) 2020-06-12T02:37:51.851 INFO:tasks.rados.rados.0.smithi107.stdout:3663: done (1 left) 2020-06-12T02:37:51.851 INFO:tasks.rados.rados.0.smithi107.stdout:3664: delete oid 3 current snap is 0 2020-06-12T02:37:51.853 INFO:tasks.rados.rados.0.smithi107.stdout:3664: done (1 left) 2020-06-12T02:37:51.853 INFO:tasks.rados.rados.0.smithi107.stdout:3665: delete oid 3 current snap is 0 2020-06-12T02:37:51.854 INFO:tasks.rados.rados.0.smithi107.stdout:3662: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:51.854 INFO:tasks.rados.rados.0.smithi107.stdout:3662: finishing write tid 2 to smithi10729407-20 2020-06-12T02:37:51.855 INFO:tasks.rados.rados.0.smithi107.stdout:3662: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:51.858 INFO:tasks.rados.rados.0.smithi107.stdout:3662: finishing write tid 5 to smithi10729407-20 2020-06-12T02:37:51.860 INFO:tasks.rados.rados.0.smithi107.stdout:3662: finishing write tid 6 to smithi10729407-20 2020-06-12T02:37:51.860 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 276 (ObjNum 3661 snap 0 seq_num 3661) dirty exists 2020-06-12T02:37:51.860 INFO:tasks.rados.rados.0.smithi107.stdout:3662: left oid 20 (ObjNum 3661 snap 0 seq_num 3661) 2020-06-12T02:37:51.861 INFO:tasks.rados.rados.0.smithi107.stdout:3662: done (1 left) 2020-06-12T02:37:51.861 INFO:tasks.rados.rados.0.smithi107.stdout:3665: done (0 left) 2020-06-12T02:37:51.861 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:37:51.862 INFO:tasks.rados.rados.0.smithi107.stdout:3666: seq_num 3665 ranges {0=589824,589824=425984,1015808=376832} 2020-06-12T02:37:51.864 INFO:tasks.rados.rados.0.smithi107.stdout:3666: writing smithi10729407-16 from 0 to 589824 tid 1 2020-06-12T02:37:51.866 INFO:tasks.rados.rados.0.smithi107.stdout:3666: writing smithi10729407-16 from 589824 to 1015808 tid 2 2020-06-12T02:37:51.869 INFO:tasks.rados.rados.0.smithi107.stdout:3666: writing smithi10729407-16 from 1015808 to 1392640 tid 3 2020-06-12T02:37:51.869 INFO:tasks.rados.rados.0.smithi107.stdout:3667: delete oid 14 current snap is 0 2020-06-12T02:37:51.872 INFO:tasks.rados.rados.0.smithi107.stdout:3667: done (1 left) 2020-06-12T02:37:51.872 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:37:51.872 INFO:tasks.rados.rados.0.smithi107.stdout:3668: seq_num 3667 ranges {0=622592,622592=598016,1220608=204800} 2020-06-12T02:37:51.876 INFO:tasks.rados.rados.0.smithi107.stdout:3668: writing smithi10729407-15 from 0 to 622592 tid 1 2020-06-12T02:37:51.878 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:37:51.876+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:37:51.881 INFO:tasks.rados.rados.0.smithi107.stdout:3668: writing smithi10729407-15 from 622592 to 1220608 tid 2 2020-06-12T02:37:51.883 INFO:tasks.rados.rados.0.smithi107.stdout:3668: writing smithi10729407-15 from 1220608 to 1425408 tid 3 2020-06-12T02:37:51.883 INFO:tasks.rados.rados.0.smithi107.stdout:3669: delete oid 13 current snap is 0 2020-06-12T02:37:51.883 INFO:tasks.rados.rados.0.smithi107.stdout:3666: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:51.884 INFO:tasks.rados.rados.0.smithi107.stdout:3666: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:51.884 INFO:tasks.rados.rados.0.smithi107.stdout:3666: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:51.884 INFO:tasks.rados.rados.0.smithi107.stdout:3666: finishing write tid 5 to smithi10729407-16 2020-06-12T02:37:51.885 INFO:tasks.rados.rados.0.smithi107.stdout:3666: finishing write tid 6 to smithi10729407-16 2020-06-12T02:37:51.885 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 664 (ObjNum 3665 snap 0 seq_num 3665) dirty exists 2020-06-12T02:37:51.885 INFO:tasks.rados.rados.0.smithi107.stdout:3666: left oid 16 (ObjNum 3665 snap 0 seq_num 3665) 2020-06-12T02:37:51.888 INFO:tasks.rados.rados.0.smithi107.stdout:3666: done (2 left) 2020-06-12T02:37:51.888 INFO:tasks.rados.rados.0.smithi107.stdout:3669: done (1 left) 2020-06-12T02:37:51.889 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:37:51.889 INFO:tasks.rados.rados.0.smithi107.stdout:3670: seq_num 3669 ranges {0=655360,655360=770048,1425408=32768} 2020-06-12T02:37:51.898 INFO:tasks.rados.rados.0.smithi107.stdout:3670: writing smithi10729407-18 from 0 to 655360 tid 1 2020-06-12T02:37:51.898 INFO:tasks.rados.rados.0.smithi107.stdout:3670: writing smithi10729407-18 from 655360 to 1425408 tid 2 2020-06-12T02:37:51.899 INFO:tasks.rados.rados.0.smithi107.stdout:3670: writing smithi10729407-18 from 1425408 to 1458176 tid 3 2020-06-12T02:37:51.899 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:37:51.899 INFO:tasks.rados.rados.0.smithi107.stdout:3671: seq_num 3670 ranges {0=671744,671744=647168,1318912=155648} 2020-06-12T02:37:51.903 INFO:tasks.rados.rados.0.smithi107.stdout:3671: writing smithi10729407-5 from 0 to 671744 tid 1 2020-06-12T02:37:51.908 INFO:tasks.rados.rados.0.smithi107.stdout:3671: writing smithi10729407-5 from 671744 to 1318912 tid 2 2020-06-12T02:37:51.909 INFO:tasks.rados.rados.0.smithi107.stdout:3671: writing smithi10729407-5 from 1318912 to 1474560 tid 3 2020-06-12T02:37:51.909 INFO:tasks.rados.rados.0.smithi107.stdout:3672: delete oid 17 current snap is 0 2020-06-12T02:37:51.910 INFO:tasks.rados.rados.0.smithi107.stdout:3668: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:51.910 INFO:tasks.rados.rados.0.smithi107.stdout:3668: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:51.910 INFO:tasks.rados.rados.0.smithi107.stdout:3668: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:51.910 INFO:tasks.rados.rados.0.smithi107.stdout:3668: finishing write tid 5 to smithi10729407-15 2020-06-12T02:37:51.910 INFO:tasks.rados.rados.0.smithi107.stdout:3668: finishing write tid 6 to smithi10729407-15 2020-06-12T02:37:51.911 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 621 (ObjNum 3667 snap 0 seq_num 3667) dirty exists 2020-06-12T02:37:51.911 INFO:tasks.rados.rados.0.smithi107.stdout:3668: left oid 15 (ObjNum 3667 snap 0 seq_num 3667) 2020-06-12T02:37:51.911 INFO:tasks.rados.rados.0.smithi107.stdout:3670: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:51.911 INFO:tasks.rados.rados.0.smithi107.stdout:3670: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:51.911 INFO:tasks.rados.rados.0.smithi107.stdout:3670: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:51.912 INFO:tasks.rados.rados.0.smithi107.stdout:3670: finishing write tid 5 to smithi10729407-18 2020-06-12T02:37:51.913 INFO:tasks.rados.rados.0.smithi107.stdout:3668: done (3 left) 2020-06-12T02:37:51.913 INFO:tasks.rados.rados.0.smithi107.stdout:3672: done (2 left) 2020-06-12T02:37:51.914 INFO:tasks.rados.rados.0.smithi107.stdout:3673: delete oid 11 current snap is 0 2020-06-12T02:37:51.914 INFO:tasks.rados.rados.0.smithi107.stdout:3670: finishing write tid 6 to smithi10729407-18 2020-06-12T02:37:51.915 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 643 (ObjNum 3669 snap 0 seq_num 3669) dirty exists 2020-06-12T02:37:51.915 INFO:tasks.rados.rados.0.smithi107.stdout:3670: left oid 18 (ObjNum 3669 snap 0 seq_num 3669) 2020-06-12T02:37:51.916 INFO:tasks.rados.rados.0.smithi107.stdout:3671: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:51.916 INFO:tasks.rados.rados.0.smithi107.stdout:3671: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:51.916 INFO:tasks.rados.rados.0.smithi107.stdout:3671: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:51.920 INFO:tasks.rados.rados.0.smithi107.stdout:3670: done (2 left) 2020-06-12T02:37:51.920 INFO:tasks.rados.rados.0.smithi107.stdout:3673: done (1 left) 2020-06-12T02:37:51.920 INFO:tasks.rados.rados.0.smithi107.stdout:3674: delete oid 18 current snap is 0 2020-06-12T02:37:51.920 INFO:tasks.rados.rados.0.smithi107.stdout:3671: finishing write tid 5 to smithi10729407-5 2020-06-12T02:37:51.921 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:37:51.921 INFO:tasks.rados.rados.0.smithi107.stdout:3671: finishing write tid 6 to smithi10729407-5 2020-06-12T02:37:51.922 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 335 (ObjNum 3670 snap 0 seq_num 3670) dirty exists 2020-06-12T02:37:51.922 INFO:tasks.rados.rados.0.smithi107.stdout:3671: left oid 5 (ObjNum 3670 snap 0 seq_num 3670) 2020-06-12T02:37:51.922 INFO:tasks.rados.rados.0.smithi107.stdout:3671: done (1 left) 2020-06-12T02:37:51.922 INFO:tasks.rados.rados.0.smithi107.stdout:3674: done (0 left) 2020-06-12T02:37:51.923 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:37:51.923 INFO:tasks.rados.rados.0.smithi107.stdout:3675: seq_num 3674 ranges {0=737280,737280=638976,1376256=163840} 2020-06-12T02:37:51.924 INFO:teuthology.orchestra.run.smithi188.stderr:no valid command found; 10 closest matches: 2020-06-12T02:37:51.924 INFO:teuthology.orchestra.run.smithi188.stderr:0 2020-06-12T02:37:51.925 INFO:teuthology.orchestra.run.smithi188.stderr:1 2020-06-12T02:37:51.925 INFO:teuthology.orchestra.run.smithi188.stderr:2 2020-06-12T02:37:51.925 INFO:teuthology.orchestra.run.smithi188.stderr:abort 2020-06-12T02:37:51.925 INFO:teuthology.orchestra.run.smithi188.stderr:assert 2020-06-12T02:37:51.926 INFO:teuthology.orchestra.run.smithi188.stderr:config diff 2020-06-12T02:37:51.926 INFO:teuthology.orchestra.run.smithi188.stderr:config diff get 2020-06-12T02:37:51.926 INFO:teuthology.orchestra.run.smithi188.stderr:config get 2020-06-12T02:37:51.926 INFO:teuthology.orchestra.run.smithi188.stderr:config help [] 2020-06-12T02:37:51.926 INFO:teuthology.orchestra.run.smithi188.stderr:config set ... 2020-06-12T02:37:51.926 INFO:teuthology.orchestra.run.smithi188.stderr:admin_socket: invalid command 2020-06-12T02:37:51.927 INFO:tasks.rados.rados.0.smithi107.stdout:3675: writing smithi10729407-18 from 0 to 737280 tid 1 2020-06-12T02:37:51.928 DEBUG:teuthology.orchestra.run:got remote process result: 22 2020-06-12T02:37:51.928 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2020-06-12T02:37:51.931 INFO:tasks.rados.rados.0.smithi107.stdout:3675: writing smithi10729407-18 from 737280 to 1376256 tid 2 2020-06-12T02:37:51.931 INFO:tasks.rados.rados.0.smithi107.stdout:3675: writing smithi10729407-18 from 1376256 to 1540096 tid 3 2020-06-12T02:37:51.932 INFO:tasks.rados.rados.0.smithi107.stdout:3676: delete oid 16 current snap is 0 2020-06-12T02:37:51.935 INFO:tasks.rados.rados.0.smithi107.stdout:3676: done (1 left) 2020-06-12T02:37:51.935 INFO:tasks.rados.rados.0.smithi107.stdout:3677: delete oid 20 current snap is 0 2020-06-12T02:37:51.938 INFO:tasks.rados.rados.0.smithi107.stdout:3677: done (1 left) 2020-06-12T02:37:51.939 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:37:51.939 INFO:tasks.rados.rados.0.smithi107.stdout:3678: seq_num 3677 ranges {1474560=786432,2260992=688128,2949120=114688} 2020-06-12T02:37:51.948 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:37:51.952 INFO:tasks.rados.rados.0.smithi107.stdout:3678: writing smithi10729407-5 from 1474560 to 2260992 tid 1 2020-06-12T02:37:51.957 INFO:tasks.rados.rados.0.smithi107.stdout:3678: writing smithi10729407-5 from 2260992 to 2949120 tid 2 2020-06-12T02:37:51.957 INFO:tasks.rados.rados.0.smithi107.stdout:3678: writing smithi10729407-5 from 2949120 to 3063808 tid 3 2020-06-12T02:37:51.957 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:37:51.958 INFO:tasks.rados.rados.0.smithi107.stdout:3679: seq_num 3678 ranges {0=802816,802816=573440,1376256=229376} 2020-06-12T02:37:51.962 INFO:tasks.rados.rados.0.smithi107.stdout:3679: writing smithi10729407-13 from 0 to 802816 tid 1 2020-06-12T02:37:51.965 INFO:tasks.rados.rados.0.smithi107.stdout:3679: writing smithi10729407-13 from 802816 to 1376256 tid 2 2020-06-12T02:37:51.967 INFO:tasks.rados.rados.0.smithi107.stdout:3679: writing smithi10729407-13 from 1376256 to 1605632 tid 3 2020-06-12T02:37:51.967 INFO:tasks.rados.rados.0.smithi107.stdout:3680: delete oid 11 current snap is 0 2020-06-12T02:37:51.967 INFO:tasks.rados.rados.0.smithi107.stdout:3675: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:51.967 INFO:tasks.rados.rados.0.smithi107.stdout:3675: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:51.968 INFO:tasks.rados.rados.0.smithi107.stdout:3675: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:51.968 INFO:tasks.rados.rados.0.smithi107.stdout:3675: finishing write tid 5 to smithi10729407-18 2020-06-12T02:37:51.968 INFO:tasks.rados.rados.0.smithi107.stdout:3675: finishing write tid 6 to smithi10729407-18 2020-06-12T02:37:51.968 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 648 (ObjNum 3674 snap 0 seq_num 3674) dirty exists 2020-06-12T02:37:51.968 INFO:tasks.rados.rados.0.smithi107.stdout:3675: left oid 18 (ObjNum 3674 snap 0 seq_num 3674) 2020-06-12T02:37:51.969 INFO:tasks.rados.rados.0.smithi107.stdout:3678: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:51.969 INFO:tasks.rados.rados.0.smithi107.stdout:3678: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:51.969 INFO:tasks.rados.rados.0.smithi107.stdout:3678: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:51.969 INFO:tasks.rados.rados.0.smithi107.stdout:3678: finishing write tid 5 to smithi10729407-5 2020-06-12T02:37:51.969 INFO:tasks.rados.rados.0.smithi107.stdout:3678: finishing write tid 6 to smithi10729407-5 2020-06-12T02:37:51.970 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 339 (ObjNum 3677 snap 0 seq_num 3677) dirty exists 2020-06-12T02:37:51.970 INFO:tasks.rados.rados.0.smithi107.stdout:3678: left oid 5 (ObjNum 3677 snap 0 seq_num 3677) 2020-06-12T02:37:51.971 INFO:tasks.rados.rados.0.smithi107.stdout:3675: done (3 left) 2020-06-12T02:37:51.972 INFO:tasks.rados.rados.0.smithi107.stdout:3678: done (2 left) 2020-06-12T02:37:51.972 INFO:tasks.rados.rados.0.smithi107.stdout:3680: done (1 left) 2020-06-12T02:37:51.972 INFO:tasks.rados.rados.0.smithi107.stdout:3681: delete oid 9 current snap is 0 2020-06-12T02:37:51.973 INFO:tasks.rados.rados.0.smithi107.stdout:3681: done (1 left) 2020-06-12T02:37:51.973 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:37:51.973 INFO:tasks.rados.rados.0.smithi107.stdout:3682: seq_num 3681 ranges {0=458752,458752=679936,1138688=417792,1556480=106496} 2020-06-12T02:37:51.974 INFO:tasks.rados.rados.0.smithi107.stdout:3682: writing smithi10729407-6 from 0 to 458752 tid 1 2020-06-12T02:37:51.975 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:37:51.979 INFO:tasks.rados.rados.0.smithi107.stdout:3682: writing smithi10729407-6 from 458752 to 1138688 tid 2 2020-06-12T02:37:51.980 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:51.978+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:51.982 INFO:tasks.rados.rados.0.smithi107.stdout:3682: writing smithi10729407-6 from 1138688 to 1556480 tid 3 2020-06-12T02:37:51.983 INFO:tasks.rados.rados.0.smithi107.stdout:3682: writing smithi10729407-6 from 1556480 to 1662976 tid 4 2020-06-12T02:37:51.983 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:37:51.983 INFO:tasks.rados.rados.0.smithi107.stdout:3683: seq_num 3682 ranges {1466368=475136,1941504=565248,2506752=622592,3129344=16384} 2020-06-12T02:37:51.993 INFO:tasks.rados.rados.0.smithi107.stdout:3683: writing smithi10729407-10 from 1466368 to 1941504 tid 1 2020-06-12T02:37:51.998 INFO:tasks.rados.rados.0.smithi107.stdout:3683: writing smithi10729407-10 from 1941504 to 2506752 tid 2 2020-06-12T02:37:52.002 INFO:tasks.rados.rados.0.smithi107.stdout:3683: writing smithi10729407-10 from 2506752 to 3129344 tid 3 2020-06-12T02:37:52.002 INFO:tasks.rados.rados.0.smithi107.stdout:3683: writing smithi10729407-10 from 3129344 to 3145728 tid 4 2020-06-12T02:37:52.003 INFO:tasks.rados.rados.0.smithi107.stdout:3684: delete oid 18 current snap is 0 2020-06-12T02:37:52.005 INFO:tasks.rados.rados.0.smithi107.stdout:3679: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:52.005 INFO:tasks.rados.rados.0.smithi107.stdout:3679: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:52.006 INFO:tasks.rados.rados.0.smithi107.stdout:3679: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:52.006 INFO:tasks.rados.rados.0.smithi107.stdout:3679: finishing write tid 5 to smithi10729407-13 2020-06-12T02:37:52.006 INFO:tasks.rados.rados.0.smithi107.stdout:3682: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:52.006 INFO:tasks.rados.rados.0.smithi107.stdout:3679: finishing write tid 6 to smithi10729407-13 2020-06-12T02:37:52.006 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 695 (ObjNum 3678 snap 0 seq_num 3678) dirty exists 2020-06-12T02:37:52.007 INFO:tasks.rados.rados.0.smithi107.stdout:3679: left oid 13 (ObjNum 3678 snap 0 seq_num 3678) 2020-06-12T02:37:52.007 INFO:tasks.rados.rados.0.smithi107.stdout:3682: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:52.007 INFO:tasks.rados.rados.0.smithi107.stdout:3682: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:52.007 INFO:tasks.rados.rados.0.smithi107.stdout:3682: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:52.007 INFO:tasks.rados.rados.0.smithi107.stdout:3682: finishing write tid 6 to smithi10729407-6 2020-06-12T02:37:52.007 INFO:tasks.rados.rados.0.smithi107.stdout:3682: finishing write tid 7 to smithi10729407-6 2020-06-12T02:37:52.008 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 687 (ObjNum 3681 snap 0 seq_num 3681) dirty exists 2020-06-12T02:37:52.008 INFO:tasks.rados.rados.0.smithi107.stdout:3682: left oid 6 (ObjNum 3681 snap 0 seq_num 3681) 2020-06-12T02:37:52.008 INFO:tasks.rados.rados.0.smithi107.stdout:3683: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:52.008 INFO:tasks.rados.rados.0.smithi107.stdout:3679: done (3 left) 2020-06-12T02:37:52.008 INFO:tasks.rados.rados.0.smithi107.stdout:3682: done (2 left) 2020-06-12T02:37:52.009 INFO:tasks.rados.rados.0.smithi107.stdout:3684: done (1 left) 2020-06-12T02:37:52.009 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:37:52.009 INFO:tasks.rados.rados.0.smithi107.stdout:3685: seq_num 3684 ranges {1425408=507904,1933312=729088,2662400=475136} 2020-06-12T02:37:52.014 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:37:52.014 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:37:52.014 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2020-06-12T02:37:52.015 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2020-06-12T02:37:52.015 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2020-06-12T02:37:52.015 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.015 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:962 3.7s0 3:ea271007:::smithi10729407-9:head [append 0~598016] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:52.015 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:32.236108+0000", 2020-06-12T02:37:52.016 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 19.773866467000001, 2020-06-12T02:37:52.016 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018271632999999999, 2020-06-12T02:37:52.016 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:52.016 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:52.016 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:52.016 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:52.017 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:52.017 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 962 2020-06-12T02:37:52.017 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.017 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:52.017 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.018 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:52.024 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.236108+0000", 2020-06-12T02:37:52.024 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:52.024 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.024 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.025 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:52.025 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.236108+0000", 2020-06-12T02:37:52.025 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019838700000000001 2020-06-12T02:37:52.025 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.025 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.026 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:52.026 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.236307+0000", 2020-06-12T02:37:52.026 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4619999999999996e-06 2020-06-12T02:37:52.026 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.026 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.026 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:52.027 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.236315+0000", 2020-06-12T02:37:52.027 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1665999999999999e-05 2020-06-12T02:37:52.027 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.027 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.027 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:52.028 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.236337+0000", 2020-06-12T02:37:52.028 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5630000000000003e-06 2020-06-12T02:37:52.028 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.028 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.028 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:52.028 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.236344+0000", 2020-06-12T02:37:52.029 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023269440000000001 2020-06-12T02:37:52.029 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.029 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.029 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:52.029 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.238671+0000", 2020-06-12T02:37:52.030 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5969000000000003e-05 2020-06-12T02:37:52.030 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.030 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.030 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:52.030 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.238757+0000", 2020-06-12T02:37:52.031 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046101010000000001 2020-06-12T02:37:52.031 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.031 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.031 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:52.031 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.243367+0000", 2020-06-12T02:37:52.031 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7991000000000005e-05 2020-06-12T02:37:52.032 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.032 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.032 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:52.032 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.243455+0000", 2020-06-12T02:37:52.032 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00062561500000000005 2020-06-12T02:37:52.033 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.033 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.033 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:52.033 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.244081+0000", 2020-06-12T02:37:52.033 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010040778 2020-06-12T02:37:52.033 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.034 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.034 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:52.034 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.254122+0000", 2020-06-12T02:37:52.034 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010453100000000001 2020-06-12T02:37:52.034 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.035 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.035 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:52.035 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.254226+0000", 2020-06-12T02:37:52.035 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000153626 2020-06-12T02:37:52.035 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.035 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.036 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:52.036 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.254380+0000", 2020-06-12T02:37:52.036 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018271632999999999 2020-06-12T02:37:52.036 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:52.036 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:52.037 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:52.037 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.037 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.037 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1252 3.8s0 3:14244450:::smithi10729407-2:head [append 0~516096] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:52.037 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:32.999793+0000", 2020-06-12T02:37:52.037 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 19.010181887000002, 2020-06-12T02:37:52.038 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017380335, 2020-06-12T02:37:52.038 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:52.038 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:52.038 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:52.038 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:52.039 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:52.039 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1252 2020-06-12T02:37:52.039 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.039 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:52.039 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.039 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:52.040 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.999793+0000", 2020-06-12T02:37:52.040 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:52.040 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.040 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.040 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:52.041 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.999793+0000", 2020-06-12T02:37:52.041 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000208051 2020-06-12T02:37:52.041 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.041 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.041 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:52.042 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.000001+0000", 2020-06-12T02:37:52.042 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9750000000000002e-06 2020-06-12T02:37:52.042 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.042 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.042 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:52.042 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.000011+0000", 2020-06-12T02:37:52.043 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7733000000000001e-05 2020-06-12T02:37:52.043 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.043 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.043 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:52.043 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.000028+0000", 2020-06-12T02:37:52.044 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7379999999999994e-06 2020-06-12T02:37:52.044 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.044 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.044 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:52.044 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.000036+0000", 2020-06-12T02:37:52.044 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.414e-05 2020-06-12T02:37:52.045 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.045 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.045 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:52.045 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.000090+0000", 2020-06-12T02:37:52.045 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012561699999999999 2020-06-12T02:37:52.045 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.046 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.046 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:52.046 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.000216+0000", 2020-06-12T02:37:52.046 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00079436199999999995 2020-06-12T02:37:52.046 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.047 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.047 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:52.047 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.001010+0000", 2020-06-12T02:37:52.047 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015996506000000001 2020-06-12T02:37:52.047 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.047 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.048 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:52.048 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.017007+0000", 2020-06-12T02:37:52.048 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0599999999999997e-05 2020-06-12T02:37:52.048 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.048 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.049 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:52.049 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.017057+0000", 2020-06-12T02:37:52.049 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011561300000000001 2020-06-12T02:37:52.049 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.049 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.049 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:52.050 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.017173+0000", 2020-06-12T02:37:52.050 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017380335 2020-06-12T02:37:52.050 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:52.050 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:52.050 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:52.051 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.051 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.051 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1300 3.bs0 3:d0ebae7d:::smithi10729407-3:head [append 0~557056] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:52.051 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.166253+0000", 2020-06-12T02:37:52.051 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 18.843721241000001, 2020-06-12T02:37:52.051 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019037724999999998, 2020-06-12T02:37:52.052 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:52.052 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:52.052 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:52.052 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:52.052 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:52.053 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1300 2020-06-12T02:37:52.053 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.053 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:52.053 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.053 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:52.053 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.166253+0000", 2020-06-12T02:37:52.054 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:52.054 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.054 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.054 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:52.054 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.166253+0000", 2020-06-12T02:37:52.054 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034533600000000002 2020-06-12T02:37:52.055 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.055 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.055 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:52.055 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.166599+0000", 2020-06-12T02:37:52.055 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7699999999999996e-06 2020-06-12T02:37:52.056 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.056 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.056 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:52.056 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.166608+0000", 2020-06-12T02:37:52.056 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7314999999999999e-05 2020-06-12T02:37:52.057 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.057 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.057 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:52.057 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.166626+0000", 2020-06-12T02:37:52.057 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9260000000000003e-06 2020-06-12T02:37:52.057 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.058 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.058 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:52.058 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.166633+0000", 2020-06-12T02:37:52.058 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0266000000000002e-05 2020-06-12T02:37:52.058 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.059 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.059 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:52.059 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.166693+0000", 2020-06-12T02:37:52.059 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6692000000000005e-05 2020-06-12T02:37:52.059 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.059 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.060 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:52.060 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.166760+0000", 2020-06-12T02:37:52.060 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0057626860000000004 2020-06-12T02:37:52.060 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.060 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.061 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:52.061 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.172522+0000", 2020-06-12T02:37:52.061 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000130493 2020-06-12T02:37:52.061 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.061 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.061 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:52.062 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.172653+0000", 2020-06-12T02:37:52.062 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00083376299999999995 2020-06-12T02:37:52.062 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.062 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.062 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:52.063 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.173487+0000", 2020-06-12T02:37:52.063 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0061197739999999997 2020-06-12T02:37:52.063 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.063 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.063 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:52.063 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.179606+0000", 2020-06-12T02:37:52.064 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0056160699999999999 2020-06-12T02:37:52.064 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.064 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.064 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:52.064 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.185222+0000", 2020-06-12T02:37:52.065 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8633999999999994e-05 2020-06-12T02:37:52.065 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.065 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.065 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:52.065 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.185291+0000", 2020-06-12T02:37:52.065 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019037724999999998 2020-06-12T02:37:52.066 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:52.066 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:52.066 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:52.066 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.066 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.067 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1301 3.bs0 3:d0ebae7d:::smithi10729407-3:head [append 0~278528] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:52.067 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.168146+0000", 2020-06-12T02:37:52.067 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 18.841828683999999, 2020-06-12T02:37:52.067 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017691083999999999, 2020-06-12T02:37:52.067 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:52.067 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:52.068 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:52.068 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:52.068 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:52.068 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1301 2020-06-12T02:37:52.068 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.069 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:52.069 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.069 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:52.069 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168146+0000", 2020-06-12T02:37:52.069 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:52.070 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.070 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.070 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:52.070 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168146+0000", 2020-06-12T02:37:52.070 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000213666 2020-06-12T02:37:52.070 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.071 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.071 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:52.071 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168360+0000", 2020-06-12T02:37:52.071 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7009999999999996e-06 2020-06-12T02:37:52.071 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.071 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.072 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:52.072 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168367+0000", 2020-06-12T02:37:52.072 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5096000000000001e-05 2020-06-12T02:37:52.072 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.072 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.073 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:52.073 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168382+0000", 2020-06-12T02:37:52.073 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8739999999999997e-06 2020-06-12T02:37:52.073 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.073 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.074 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:52.074 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168389+0000", 2020-06-12T02:37:52.074 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0076120199999999997 2020-06-12T02:37:52.074 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.074 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.074 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:52.075 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.176001+0000", 2020-06-12T02:37:52.075 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9181000000000001e-05 2020-06-12T02:37:52.075 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.075 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.075 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:52.076 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.176060+0000", 2020-06-12T02:37:52.076 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00039587599999999998 2020-06-12T02:37:52.076 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.076 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.076 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:52.076 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.176456+0000", 2020-06-12T02:37:52.077 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0031522329999999999 2020-06-12T02:37:52.077 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.077 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.077 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:52.077 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.179609+0000", 2020-06-12T02:37:52.078 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0061683670000000001 2020-06-12T02:37:52.078 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.078 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.078 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:52.078 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.185777+0000", 2020-06-12T02:37:52.078 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0069999999999999e-05 2020-06-12T02:37:52.079 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.079 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.079 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:52.079 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.185837+0000", 2020-06-12T02:37:52.079 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017691083999999999 2020-06-12T02:37:52.079 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:52.080 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:52.080 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:52.080 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.080 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.080 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1302 3.bs0 3:d0ebae7d:::smithi10729407-3:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:52.081 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.168445+0000", 2020-06-12T02:37:52.081 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 18.841529403999999, 2020-06-12T02:37:52.081 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017581657000000001, 2020-06-12T02:37:52.081 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:52.081 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:52.082 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:52.082 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:52.082 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:52.082 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1302 2020-06-12T02:37:52.082 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.082 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:52.083 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.083 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:52.083 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168445+0000", 2020-06-12T02:37:52.083 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:52.083 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.083 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.084 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:52.084 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168445+0000", 2020-06-12T02:37:52.084 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3532e-05 2020-06-12T02:37:52.084 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.084 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.085 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:52.085 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168519+0000", 2020-06-12T02:37:52.085 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2600000000000002e-06 2020-06-12T02:37:52.085 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.085 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.085 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:52.086 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168525+0000", 2020-06-12T02:37:52.086 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4338999999999999e-05 2020-06-12T02:37:52.086 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.086 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.086 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:52.087 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168539+0000", 2020-06-12T02:37:52.087 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7330000000000003e-06 2020-06-12T02:37:52.087 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.087 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.087 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:52.087 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168544+0000", 2020-06-12T02:37:52.088 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0095046090000000007 2020-06-12T02:37:52.088 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.088 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.088 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:52.088 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.178049+0000", 2020-06-12T02:37:52.089 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1524000000000003e-05 2020-06-12T02:37:52.089 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.089 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.089 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:52.089 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.178110+0000", 2020-06-12T02:37:52.089 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030514799999999999 2020-06-12T02:37:52.090 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.090 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.090 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:52.090 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.178415+0000", 2020-06-12T02:37:52.090 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011944480000000001 2020-06-12T02:37:52.091 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.091 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.091 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:52.091 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.179610+0000", 2020-06-12T02:37:52.091 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0063372849999999998 2020-06-12T02:37:52.091 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.092 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.092 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:52.092 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.185947+0000", 2020-06-12T02:37:52.092 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9778999999999996e-05 2020-06-12T02:37:52.092 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.093 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.093 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:52.093 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.186027+0000", 2020-06-12T02:37:52.093 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017581657000000001 2020-06-12T02:37:52.093 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:52.093 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:52.094 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:52.094 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.094 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.094 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1303 3.bs0 3:d0ebae7d:::smithi10729407-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:52.094 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.168587+0000", 2020-06-12T02:37:52.095 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 18.841387485999999, 2020-06-12T02:37:52.095 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019127155, 2020-06-12T02:37:52.095 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:52.095 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:37:52.095 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:52.095 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:52.096 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:52.096 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1303 2020-06-12T02:37:52.096 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.096 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:52.096 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.097 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:52.097 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168587+0000", 2020-06-12T02:37:52.097 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:52.097 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.097 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.097 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:52.098 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168587+0000", 2020-06-12T02:37:52.098 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3099999999999997e-05 2020-06-12T02:37:52.098 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.098 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.098 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:52.099 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168630+0000", 2020-06-12T02:37:52.099 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1600000000000003e-06 2020-06-12T02:37:52.099 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.099 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.100 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:52.100 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168636+0000", 2020-06-12T02:37:52.100 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3439e-05 2020-06-12T02:37:52.100 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.100 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.101 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:52.101 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168650+0000", 2020-06-12T02:37:52.101 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7850000000000002e-06 2020-06-12T02:37:52.101 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.101 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.102 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:52.102 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.168654+0000", 2020-06-12T02:37:52.102 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010326317 2020-06-12T02:37:52.102 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.102 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.102 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:52.103 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.178980+0000", 2020-06-12T02:37:52.103 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6593000000000004e-05 2020-06-12T02:37:52.103 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.103 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.103 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:52.103 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.179047+0000", 2020-06-12T02:37:52.104 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0073410990000000002 2020-06-12T02:37:52.104 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.104 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.104 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:52.104 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.186388+0000", 2020-06-12T02:37:52.105 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9405000000000001e-05 2020-06-12T02:37:52.105 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.105 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.105 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:52.105 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.186487+0000", 2020-06-12T02:37:52.106 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001227257 2020-06-12T02:37:52.106 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.106 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.106 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:52.106 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.187714+0000", 2020-06-12T02:37:52.106 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019127155 2020-06-12T02:37:52.107 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:52.107 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:52.107 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:52.107 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.107 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.107 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:1527 3.2s0 3:4afffc6f:::smithi10729407-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e26)", 2020-06-12T02:37:52.108 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.750793+0000", 2020-06-12T02:37:52.108 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 18.259181332000001, 2020-06-12T02:37:52.108 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016832799999999998, 2020-06-12T02:37:52.108 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:52.108 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:37:52.109 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:52.109 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:52.109 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:52.109 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 1527 2020-06-12T02:37:52.109 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.109 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:52.110 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.110 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:52.110 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750793+0000", 2020-06-12T02:37:52.110 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:52.110 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.111 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.111 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:52.111 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750793+0000", 2020-06-12T02:37:52.111 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3218e-05 2020-06-12T02:37:52.111 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.111 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.112 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:52.112 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750817+0000", 2020-06-12T02:37:52.112 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0939999999999999e-06 2020-06-12T02:37:52.112 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.112 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.113 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:52.113 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750820+0000", 2020-06-12T02:37:52.113 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4810000000000003e-06 2020-06-12T02:37:52.113 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.113 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.113 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:52.114 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750826+0000", 2020-06-12T02:37:52.114 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.277e-06 2020-06-12T02:37:52.114 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.114 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.114 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:52.115 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750828+0000", 2020-06-12T02:37:52.115 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7738000000000005e-05 2020-06-12T02:37:52.115 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.115 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.115 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:52.115 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.750926+0000", 2020-06-12T02:37:52.116 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6310000000000005e-05 2020-06-12T02:37:52.116 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.116 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.116 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:52.117 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.751012+0000", 2020-06-12T02:37:52.117 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010605622 2020-06-12T02:37:52.117 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.117 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.117 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:52.117 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.761618+0000", 2020-06-12T02:37:52.118 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2778000000000002e-05 2020-06-12T02:37:52.118 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.118 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.118 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:52.119 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.761671+0000", 2020-06-12T02:37:52.119 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.005069044 2020-06-12T02:37:52.119 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.119 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.119 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:52.120 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.766740+0000", 2020-06-12T02:37:52.120 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6335999999999997e-05 2020-06-12T02:37:52.120 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.120 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.120 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:52.120 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.766796+0000", 2020-06-12T02:37:52.121 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00082990199999999998 2020-06-12T02:37:52.121 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.121 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.121 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:52.121 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.767626+0000", 2020-06-12T02:37:52.122 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016832799999999998 2020-06-12T02:37:52.122 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:52.122 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:52.122 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:52.122 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.122 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.123 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s1 26/21 ECSubWrite(tid=408, reqid=client.4512.0:1572, at_version=26'99, trim_to=0'0, roll_forward_to=26'97))", 2020-06-12T02:37:52.123 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.887944+0000", 2020-06-12T02:37:52.123 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 18.122030852999998, 2020-06-12T02:37:52.123 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019136641999999999, 2020-06-12T02:37:52.123 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:52.124 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:52.124 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:52.124 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.124 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:52.124 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.887944+0000", 2020-06-12T02:37:52.124 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:52.125 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.125 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.125 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:52.125 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.887944+0000", 2020-06-12T02:37:52.125 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00063708500000000002 2020-06-12T02:37:52.126 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.126 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.126 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:52.126 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888581+0000", 2020-06-12T02:37:52.126 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9644e-05 2020-06-12T02:37:52.126 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.127 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.127 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:52.127 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888600+0000", 2020-06-12T02:37:52.127 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.721e-05 2020-06-12T02:37:52.127 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.128 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.128 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:52.128 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888618+0000", 2020-06-12T02:37:52.128 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7500000000000003e-06 2020-06-12T02:37:52.128 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.128 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.130 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:52.130 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888625+0000", 2020-06-12T02:37:52.130 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2707999999999998e-05 2020-06-12T02:37:52.130 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.131 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.131 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:52.131 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888668+0000", 2020-06-12T02:37:52.131 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0491e-05 2020-06-12T02:37:52.131 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.132 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.132 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:52.132 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888679+0000", 2020-06-12T02:37:52.132 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018351985000000001 2020-06-12T02:37:52.132 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.132 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.133 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:52.133 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.907031+0000", 2020-06-12T02:37:52.133 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9768999999999997e-05 2020-06-12T02:37:52.133 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.133 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.134 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:52.134 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.907080+0000", 2020-06-12T02:37:52.134 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019136641999999999 2020-06-12T02:37:52.134 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:52.134 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:52.135 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:52.135 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.135 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.135 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s1 26/21 ECSubWrite(tid=409, reqid=client.4512.0:1573, at_version=26'100, trim_to=0'0, roll_forward_to=26'98))", 2020-06-12T02:37:52.135 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.892432+0000", 2020-06-12T02:37:52.135 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 18.117542263000001, 2020-06-12T02:37:52.136 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01685733, 2020-06-12T02:37:52.136 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:52.136 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:37:52.136 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:52.136 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.137 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:52.137 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.892432+0000", 2020-06-12T02:37:52.137 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:52.137 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.137 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.137 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:52.138 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.892432+0000", 2020-06-12T02:37:52.138 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059245900000000004 2020-06-12T02:37:52.138 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.138 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.138 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:52.139 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.893025+0000", 2020-06-12T02:37:52.139 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1591e-05 2020-06-12T02:37:52.139 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.139 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.139 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:52.140 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.893046+0000", 2020-06-12T02:37:52.140 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2429000000000002e-05 2020-06-12T02:37:52.140 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.140 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.140 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:52.140 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.893069+0000", 2020-06-12T02:37:52.141 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0389999999999993e-06 2020-06-12T02:37:52.141 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.141 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.141 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:52.141 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.893078+0000", 2020-06-12T02:37:52.142 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012883099999999999 2020-06-12T02:37:52.142 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.142 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.142 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:52.142 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.893207+0000", 2020-06-12T02:37:52.142 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2407e-05 2020-06-12T02:37:52.143 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.143 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.143 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:52.143 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.893219+0000", 2020-06-12T02:37:52.143 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016029015000000001 2020-06-12T02:37:52.144 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.144 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.144 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:52.144 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.909248+0000", 2020-06-12T02:37:52.144 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1559e-05 2020-06-12T02:37:52.144 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.145 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.145 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:52.145 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.909290+0000", 2020-06-12T02:37:52.145 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01685733 2020-06-12T02:37:52.145 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:52.146 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:52.146 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:52.146 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.146 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.146 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:3607 3.7s0 3:e653d00f:::smithi10729407-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:37:52.147 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:39.292176+0000", 2020-06-12T02:37:52.147 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 12.717798773, 2020-06-12T02:37:52.147 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016904665999999999, 2020-06-12T02:37:52.147 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:52.147 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:37:52.147 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:52.148 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:52.148 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:52.148 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 3607 2020-06-12T02:37:52.148 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.148 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:52.148 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.149 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:52.149 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.292176+0000", 2020-06-12T02:37:52.149 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:52.150 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.150 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.150 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:52.150 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.292176+0000", 2020-06-12T02:37:52.150 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4786000000000001e-05 2020-06-12T02:37:52.151 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.151 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.151 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:52.151 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.292221+0000", 2020-06-12T02:37:52.151 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0159999999999997e-06 2020-06-12T02:37:52.151 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.152 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.152 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:52.152 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.292227+0000", 2020-06-12T02:37:52.152 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2269999999999999e-05 2020-06-12T02:37:52.152 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.153 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.153 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:52.153 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.292239+0000", 2020-06-12T02:37:52.153 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1849999999999997e-06 2020-06-12T02:37:52.153 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.153 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.154 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:52.154 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.292243+0000", 2020-06-12T02:37:52.154 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0077395820000000001 2020-06-12T02:37:52.154 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.154 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.155 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:52.155 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.299983+0000", 2020-06-12T02:37:52.155 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7809999999999997e-05 2020-06-12T02:37:52.155 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.155 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.155 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:52.156 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.300040+0000", 2020-06-12T02:37:52.156 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0049104680000000003 2020-06-12T02:37:52.156 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.156 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.156 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:52.157 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.304951+0000", 2020-06-12T02:37:52.157 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2177999999999997e-05 2020-06-12T02:37:52.157 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.157 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.157 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:52.157 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.305043+0000", 2020-06-12T02:37:52.158 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0040373709999999997 2020-06-12T02:37:52.158 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.158 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.158 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:52.158 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:39.309081+0000", 2020-06-12T02:37:52.159 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016904665999999999 2020-06-12T02:37:52.159 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:52.159 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:52.159 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:52.159 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.159 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.160 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:4483 3.7s0 3:e653d00f:::smithi10729407-18:head [append 0~417792] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-06-12T02:37:52.160 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:42.533832+0000", 2020-06-12T02:37:52.160 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 9.476143081, 2020-06-12T02:37:52.160 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018187756999999999, 2020-06-12T02:37:52.160 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:52.160 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:52.161 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:52.161 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:52.161 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:52.161 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 4483 2020-06-12T02:37:52.161 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.162 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:52.162 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.162 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:52.162 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.533832+0000", 2020-06-12T02:37:52.162 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:52.162 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.163 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.163 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:52.163 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.533832+0000", 2020-06-12T02:37:52.163 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016872100000000001 2020-06-12T02:37:52.163 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.164 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.164 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:52.164 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.534000+0000", 2020-06-12T02:37:52.164 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3569999999999998e-06 2020-06-12T02:37:52.164 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.165 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.165 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:52.165 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.534009+0000", 2020-06-12T02:37:52.165 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6013000000000001e-05 2020-06-12T02:37:52.165 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.165 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.166 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:52.166 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.534025+0000", 2020-06-12T02:37:52.166 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.841e-06 2020-06-12T02:37:52.166 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.166 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.167 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:52.167 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.534032+0000", 2020-06-12T02:37:52.167 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014511190000000001 2020-06-12T02:37:52.167 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.167 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.167 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:52.168 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.535484+0000", 2020-06-12T02:37:52.168 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013402299999999999 2020-06-12T02:37:52.168 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.168 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.168 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:52.169 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.535618+0000", 2020-06-12T02:37:52.169 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0026343590000000002 2020-06-12T02:37:52.169 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.169 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.169 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:52.169 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.538252+0000", 2020-06-12T02:37:52.170 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2380000000000007e-05 2020-06-12T02:37:52.170 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.170 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.170 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:52.170 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.538344+0000", 2020-06-12T02:37:52.171 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00068647700000000001 2020-06-12T02:37:52.171 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.171 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.171 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:52.171 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.539031+0000", 2020-06-12T02:37:52.171 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011685509 2020-06-12T02:37:52.172 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.172 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.172 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:52.172 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.550716+0000", 2020-06-12T02:37:52.172 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011673790000000001 2020-06-12T02:37:52.173 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.173 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.173 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:52.173 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.551884+0000", 2020-06-12T02:37:52.173 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013557900000000001 2020-06-12T02:37:52.173 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.174 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.174 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:52.174 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.552019+0000", 2020-06-12T02:37:52.174 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018187756999999999 2020-06-12T02:37:52.174 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:52.175 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:52.175 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:52.175 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.175 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.175 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:4485 3.7s0 3:e653d00f:::smithi10729407-18:head [append 0~180224] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-06-12T02:37:52.175 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:42.538857+0000", 2020-06-12T02:37:52.176 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 9.4711176360000007, 2020-06-12T02:37:52.176 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017291731000000001, 2020-06-12T02:37:52.176 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:52.176 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:52.176 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:52.177 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:52.177 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:52.177 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 4485 2020-06-12T02:37:52.177 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.177 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:52.178 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.178 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:52.178 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.538857+0000", 2020-06-12T02:37:52.178 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:52.178 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.178 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.179 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:52.179 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.538857+0000", 2020-06-12T02:37:52.179 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000170361 2020-06-12T02:37:52.179 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.179 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.179 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:52.180 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.539027+0000", 2020-06-12T02:37:52.180 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1885000000000001e-05 2020-06-12T02:37:52.180 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.180 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.180 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:52.181 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.539039+0000", 2020-06-12T02:37:52.181 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3631e-05 2020-06-12T02:37:52.181 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.181 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.181 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:52.181 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.539053+0000", 2020-06-12T02:37:52.182 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8189999999999997e-06 2020-06-12T02:37:52.182 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.182 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.182 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:52.182 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.539059+0000", 2020-06-12T02:37:52.183 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093035110000000004 2020-06-12T02:37:52.183 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.183 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.183 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:52.183 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.548362+0000", 2020-06-12T02:37:52.183 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114157 2020-06-12T02:37:52.184 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.184 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.184 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:52.184 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.548476+0000", 2020-06-12T02:37:52.184 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056751900000000005 2020-06-12T02:37:52.185 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.185 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.185 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:52.185 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.549044+0000", 2020-06-12T02:37:52.185 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0042106590000000003 2020-06-12T02:37:52.185 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.186 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.186 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:52.186 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.553255+0000", 2020-06-12T02:37:52.186 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027535060000000002 2020-06-12T02:37:52.186 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.187 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.187 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:52.187 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.556008+0000", 2020-06-12T02:37:52.187 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014068300000000001 2020-06-12T02:37:52.187 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.187 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.188 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:52.188 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.556149+0000", 2020-06-12T02:37:52.188 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017291731000000001 2020-06-12T02:37:52.188 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:52.188 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:52.189 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:52.189 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.189 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.189 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:4486 3.7s0 3:e653d00f:::smithi10729407-18:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-06-12T02:37:52.189 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:42.539122+0000", 2020-06-12T02:37:52.189 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 9.4708526529999997, 2020-06-12T02:37:52.190 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017248556000000002, 2020-06-12T02:37:52.190 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:52.190 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:52.190 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:52.190 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:52.191 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:52.191 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 4486 2020-06-12T02:37:52.191 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.191 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:52.191 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.191 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:52.192 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.539122+0000", 2020-06-12T02:37:52.192 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:52.192 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.192 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.192 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:52.192 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.539122+0000", 2020-06-12T02:37:52.193 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7909000000000003e-05 2020-06-12T02:37:52.193 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.193 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.193 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:52.193 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.539190+0000", 2020-06-12T02:37:52.194 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5819999999999998e-06 2020-06-12T02:37:52.194 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.194 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.194 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:52.194 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.539196+0000", 2020-06-12T02:37:52.194 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2004999999999999e-05 2020-06-12T02:37:52.195 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.195 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.195 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:52.195 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.539208+0000", 2020-06-12T02:37:52.195 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8359999999999999e-06 2020-06-12T02:37:52.196 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.196 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.196 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:52.196 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.539213+0000", 2020-06-12T02:37:52.196 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011535996 2020-06-12T02:37:52.196 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.197 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.197 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:52.197 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.550749+0000", 2020-06-12T02:37:52.197 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3753999999999997e-05 2020-06-12T02:37:52.197 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.198 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.198 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:52.198 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.550843+0000", 2020-06-12T02:37:52.198 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00062439500000000003 2020-06-12T02:37:52.198 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.198 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.199 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:52.199 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.551467+0000", 2020-06-12T02:37:52.199 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0020300309999999998 2020-06-12T02:37:52.199 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.199 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.200 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:52.200 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.553497+0000", 2020-06-12T02:37:52.200 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027478680000000001 2020-06-12T02:37:52.200 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.200 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.200 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:52.201 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.556245+0000", 2020-06-12T02:37:52.201 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012517999999999999 2020-06-12T02:37:52.201 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.201 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.201 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:52.202 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.556371+0000", 2020-06-12T02:37:52.202 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017248556000000002 2020-06-12T02:37:52.202 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:52.202 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:52.202 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:52.202 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.203 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.203 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:4487 3.7s0 3:e653d00f:::smithi10729407-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-06-12T02:37:52.203 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:42.539305+0000", 2020-06-12T02:37:52.203 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 9.4706696059999995, 2020-06-12T02:37:52.203 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019352590999999999, 2020-06-12T02:37:52.204 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:52.204 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:37:52.204 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:52.204 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:52.204 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:52.204 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 4487 2020-06-12T02:37:52.205 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.205 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:52.205 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.205 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:52.205 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.539305+0000", 2020-06-12T02:37:52.206 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:52.206 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.206 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.206 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:52.206 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.539305+0000", 2020-06-12T02:37:52.206 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7207999999999999e-05 2020-06-12T02:37:52.207 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.207 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.207 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:52.207 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.539352+0000", 2020-06-12T02:37:52.207 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6440000000000003e-06 2020-06-12T02:37:52.208 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.208 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.208 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:52.208 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.539359+0000", 2020-06-12T02:37:52.208 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0995999999999999e-05 2020-06-12T02:37:52.208 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.209 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.209 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:52.209 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.539370+0000", 2020-06-12T02:37:52.209 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4049999999999999e-06 2020-06-12T02:37:52.209 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.210 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.210 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:52.210 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.539375+0000", 2020-06-12T02:37:52.210 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013353090999999999 2020-06-12T02:37:52.210 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.210 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.211 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:52.211 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.552728+0000", 2020-06-12T02:37:52.211 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000113626 2020-06-12T02:37:52.211 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.211 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.212 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:52.212 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.552842+0000", 2020-06-12T02:37:52.212 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0039644459999999999 2020-06-12T02:37:52.212 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.212 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.212 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:52.213 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.556806+0000", 2020-06-12T02:37:52.213 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5099000000000005e-05 2020-06-12T02:37:52.213 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.213 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.213 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:52.214 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.556902+0000", 2020-06-12T02:37:52.214 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017560760000000001 2020-06-12T02:37:52.214 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.214 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.214 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:52.214 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.558658+0000", 2020-06-12T02:37:52.215 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019352590999999999 2020-06-12T02:37:52.215 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:52.215 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:52.215 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:52.215 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.216 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.216 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:5072 3.5s1 3:a9510bda:::smithi10729407-14:head [append 0~770048] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-06-12T02:37:52.216 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:44.163594+0000", 2020-06-12T02:37:52.216 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 7.8463803820000004, 2020-06-12T02:37:52.216 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018215139000000002, 2020-06-12T02:37:52.216 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:52.217 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:52.217 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:52.217 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:52.217 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:52.217 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 5072 2020-06-12T02:37:52.218 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.218 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:52.218 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.218 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:52.218 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.163594+0000", 2020-06-12T02:37:52.218 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:52.219 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.219 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.219 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:52.219 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.163594+0000", 2020-06-12T02:37:52.219 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00046902400000000001 2020-06-12T02:37:52.220 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.220 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.220 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:52.220 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.164063+0000", 2020-06-12T02:37:52.220 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3689999999999996e-06 2020-06-12T02:37:52.220 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.221 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.221 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:52.221 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.164073+0000", 2020-06-12T02:37:52.221 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0157999999999999e-05 2020-06-12T02:37:52.221 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.222 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.222 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:52.222 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.164093+0000", 2020-06-12T02:37:52.222 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6800000000000005e-06 2020-06-12T02:37:52.222 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.222 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.223 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:52.223 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.164102+0000", 2020-06-12T02:37:52.223 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9597999999999993e-05 2020-06-12T02:37:52.223 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.223 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.223 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:52.224 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.164172+0000", 2020-06-12T02:37:52.224 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012974699999999999 2020-06-12T02:37:52.224 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.224 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.224 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:52.225 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.164302+0000", 2020-06-12T02:37:52.225 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00094603199999999997 2020-06-12T02:37:52.225 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.225 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.225 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:52.232 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.165248+0000", 2020-06-12T02:37:52.232 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016405492000000001 2020-06-12T02:37:52.233 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.233 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.233 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:52.233 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.181653+0000", 2020-06-12T02:37:52.234 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8615000000000005e-05 2020-06-12T02:37:52.234 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.235 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.235 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:52.235 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.181742+0000", 2020-06-12T02:37:52.236 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7423999999999997e-05 2020-06-12T02:37:52.236 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.236 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.236 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:52.236 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.181809+0000", 2020-06-12T02:37:52.237 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018215139000000002 2020-06-12T02:37:52.237 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:52.237 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:52.237 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:52.237 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.238 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.238 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:5077 3.5s1 3:a9510bda:::smithi10729407-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-06-12T02:37:52.238 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:44.171464+0000", 2020-06-12T02:37:52.238 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 7.8385103300000001, 2020-06-12T02:37:52.239 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017114018000000002, 2020-06-12T02:37:52.239 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:52.239 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:37:52.240 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:52.240 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:52.240 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:52.240 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 5077 2020-06-12T02:37:52.240 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.241 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:52.241 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.241 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:52.241 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.171464+0000", 2020-06-12T02:37:52.242 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:52.242 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.242 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.242 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:52.243 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.171464+0000", 2020-06-12T02:37:52.243 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8872000000000002e-05 2020-06-12T02:37:52.244 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.244 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.244 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:52.244 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.171513+0000", 2020-06-12T02:37:52.244 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2840000000000003e-06 2020-06-12T02:37:52.245 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.245 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.245 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:52.245 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.171519+0000", 2020-06-12T02:37:52.246 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3760999999999999e-05 2020-06-12T02:37:52.246 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.246 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.246 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:52.246 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.171533+0000", 2020-06-12T02:37:52.247 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1999999999999996e-06 2020-06-12T02:37:52.247 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.247 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.247 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:52.247 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.171537+0000", 2020-06-12T02:37:52.247 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011095657 2020-06-12T02:37:52.248 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.248 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.248 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:52.248 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.182633+0000", 2020-06-12T02:37:52.248 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3004999999999997e-05 2020-06-12T02:37:52.249 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.249 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.249 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:52.249 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.182696+0000", 2020-06-12T02:37:52.249 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0048429900000000001 2020-06-12T02:37:52.250 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.250 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.250 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:52.250 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.187539+0000", 2020-06-12T02:37:52.250 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7890999999999999e-05 2020-06-12T02:37:52.250 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.251 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.251 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:52.251 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.187597+0000", 2020-06-12T02:37:52.251 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0009813580000000001 2020-06-12T02:37:52.251 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.252 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.252 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:52.252 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.188578+0000", 2020-06-12T02:37:52.252 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017114018000000002 2020-06-12T02:37:52.252 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:52.252 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:52.253 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:52.253 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.253 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.253 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:5093 3.5s1 3:a446366d:::smithi10729407-11:head [append 0~417792] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-06-12T02:37:52.253 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:44.211930+0000", 2020-06-12T02:37:52.254 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 7.7980446600000004, 2020-06-12T02:37:52.254 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017278828999999999, 2020-06-12T02:37:52.254 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:52.254 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:52.254 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:52.255 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:52.255 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:52.255 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 5093 2020-06-12T02:37:52.255 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.255 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:52.255 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.256 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:52.256 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.211930+0000", 2020-06-12T02:37:52.256 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:52.256 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.256 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.257 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:52.257 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.211930+0000", 2020-06-12T02:37:52.257 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000318793 2020-06-12T02:37:52.257 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.257 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.257 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:52.258 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.212249+0000", 2020-06-12T02:37:52.258 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.7708e-05 2020-06-12T02:37:52.258 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.258 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.258 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:52.259 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.212276+0000", 2020-06-12T02:37:52.259 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3105999999999999e-05 2020-06-12T02:37:52.259 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.259 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.259 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:52.259 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.212300+0000", 2020-06-12T02:37:52.260 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2350000000000002e-06 2020-06-12T02:37:52.260 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.260 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.260 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:52.260 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.212307+0000", 2020-06-12T02:37:52.261 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036863500000000002 2020-06-12T02:37:52.261 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.261 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.261 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:52.261 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.212675+0000", 2020-06-12T02:37:52.261 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2799999999999994e-05 2020-06-12T02:37:52.262 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.262 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.262 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:52.262 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.212748+0000", 2020-06-12T02:37:52.262 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027952469999999998 2020-06-12T02:37:52.263 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.263 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.263 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:52.263 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.215543+0000", 2020-06-12T02:37:52.263 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00032576600000000001 2020-06-12T02:37:52.263 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.264 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.264 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:52.264 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.215869+0000", 2020-06-12T02:37:52.264 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00080737599999999995 2020-06-12T02:37:52.264 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.265 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.265 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:37:52.265 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.216677+0000", 2020-06-12T02:37:52.265 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012350573 2020-06-12T02:37:52.265 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.265 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.266 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:37:52.266 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.229027+0000", 2020-06-12T02:37:52.266 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4758000000000003e-05 2020-06-12T02:37:52.266 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.266 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.267 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:37:52.267 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.229122+0000", 2020-06-12T02:37:52.267 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6831999999999998e-05 2020-06-12T02:37:52.267 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.267 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.267 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:52.268 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:44.229209+0000", 2020-06-12T02:37:52.268 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017278828999999999 2020-06-12T02:37:52.268 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:52.268 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:52.268 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:52.269 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.269 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.269 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:6566 3.5s1 3:a9510bda:::smithi10729407-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e31)", 2020-06-12T02:37:52.269 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:48.290232+0000", 2020-06-12T02:37:52.269 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.7197430219999998, 2020-06-12T02:37:52.269 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018673937000000002, 2020-06-12T02:37:52.270 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:52.270 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:37:52.270 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:52.270 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:52.270 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:52.271 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 6566 2020-06-12T02:37:52.271 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.271 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:52.271 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.271 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:52.271 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:48.290232+0000", 2020-06-12T02:37:52.272 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:52.272 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.272 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.272 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:52.272 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:48.290232+0000", 2020-06-12T02:37:52.273 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.26e-05 2020-06-12T02:37:52.273 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.273 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.273 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:52.273 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:48.290254+0000", 2020-06-12T02:37:52.273 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0570000000000001e-06 2020-06-12T02:37:52.274 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.274 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.274 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:52.274 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:48.290257+0000", 2020-06-12T02:37:52.274 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1720000000000004e-06 2020-06-12T02:37:52.275 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.275 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.275 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:52.275 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:48.290263+0000", 2020-06-12T02:37:52.275 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3609999999999999e-06 2020-06-12T02:37:52.275 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.276 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.276 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:52.276 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:48.290266+0000", 2020-06-12T02:37:52.276 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0088469959999999993 2020-06-12T02:37:52.276 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.276 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.277 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:52.277 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:48.299113+0000", 2020-06-12T02:37:52.277 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5782000000000002e-05 2020-06-12T02:37:52.277 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.277 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.278 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:52.278 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:48.299169+0000", 2020-06-12T02:37:52.278 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0029321310000000001 2020-06-12T02:37:52.278 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.278 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.279 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:52.279 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:48.302101+0000", 2020-06-12T02:37:52.279 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2227999999999999e-05 2020-06-12T02:37:52.279 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.279 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.279 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:52.280 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:48.302193+0000", 2020-06-12T02:37:52.280 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0067126099999999999 2020-06-12T02:37:52.280 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.280 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.280 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:52.281 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:48.308906+0000", 2020-06-12T02:37:52.281 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018673937000000002 2020-06-12T02:37:52.281 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:52.281 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:52.281 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:52.281 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.282 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.282 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:6759 3.bs0 3:d0ebae7d:::smithi10729407-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e32)", 2020-06-12T02:37:52.282 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:48.804647+0000", 2020-06-12T02:37:52.282 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.205327692, 2020-06-12T02:37:52.282 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019135273000000001, 2020-06-12T02:37:52.283 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:52.283 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:37:52.283 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:52.283 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:52.283 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:52.283 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 6759 2020-06-12T02:37:52.284 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.284 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:52.284 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.284 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:52.284 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:48.804647+0000", 2020-06-12T02:37:52.285 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:52.285 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.285 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.285 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:52.285 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:48.804647+0000", 2020-06-12T02:37:52.285 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4632999999999998e-05 2020-06-12T02:37:52.286 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.286 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.286 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:52.286 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:48.804692+0000", 2020-06-12T02:37:52.286 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0098999999999999e-05 2020-06-12T02:37:52.287 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.287 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.287 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:52.287 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:48.804702+0000", 2020-06-12T02:37:52.287 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5077e-05 2020-06-12T02:37:52.287 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.288 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.288 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:52.288 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:48.804717+0000", 2020-06-12T02:37:52.288 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0459999999999996e-06 2020-06-12T02:37:52.288 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.289 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.289 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:52.289 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:48.804722+0000", 2020-06-12T02:37:52.289 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041246670000000003 2020-06-12T02:37:52.289 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.289 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.290 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:52.290 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:48.808846+0000", 2020-06-12T02:37:52.290 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4011999999999994e-05 2020-06-12T02:37:52.290 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.290 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.291 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:52.291 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:48.808940+0000", 2020-06-12T02:37:52.291 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071690419999999996 2020-06-12T02:37:52.291 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.291 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.291 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:52.292 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:48.816110+0000", 2020-06-12T02:37:52.292 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2596999999999995e-05 2020-06-12T02:37:52.292 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.292 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.292 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:52.293 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:48.816202+0000", 2020-06-12T02:37:52.293 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0055060669999999999 2020-06-12T02:37:52.293 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.293 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.293 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:52.293 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:48.821708+0000", 2020-06-12T02:37:52.294 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7193000000000004e-05 2020-06-12T02:37:52.294 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.294 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.294 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:52.294 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:48.821795+0000", 2020-06-12T02:37:52.295 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0019868400000000001 2020-06-12T02:37:52.295 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.295 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.295 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:52.295 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:48.823782+0000", 2020-06-12T02:37:52.295 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019135273000000001 2020-06-12T02:37:52.296 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:52.296 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:52.296 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:52.296 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.296 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.297 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:6781 3.bs0 3:d09543d2:::smithi10729407-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e32)", 2020-06-12T02:37:52.297 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:48.858413+0000", 2020-06-12T02:37:52.297 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.1515613660000001, 2020-06-12T02:37:52.297 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017431960999999999, 2020-06-12T02:37:52.297 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:37:52.297 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:37:52.298 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:37:52.298 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:37:52.298 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:52.298 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 6781 2020-06-12T02:37:52.298 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.299 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:37:52.299 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.299 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:37:52.299 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:48.858413+0000", 2020-06-12T02:37:52.299 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:37:52.299 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.300 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.300 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:37:52.300 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:48.858413+0000", 2020-06-12T02:37:52.300 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1511000000000002e-05 2020-06-12T02:37:52.300 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.300 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.301 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:37:52.301 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:48.858465+0000", 2020-06-12T02:37:52.301 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2589999999999997e-06 2020-06-12T02:37:52.301 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.301 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.302 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:37:52.302 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:48.858471+0000", 2020-06-12T02:37:52.302 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4929000000000001e-05 2020-06-12T02:37:52.302 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.302 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.302 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:37:52.303 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:48.858486+0000", 2020-06-12T02:37:52.303 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.95e-06 2020-06-12T02:37:52.303 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.303 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.303 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:37:52.304 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:48.858491+0000", 2020-06-12T02:37:52.304 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0050484429999999997 2020-06-12T02:37:52.304 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.304 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.304 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:52.304 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:48.863539+0000", 2020-06-12T02:37:52.305 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2377e-05 2020-06-12T02:37:52.305 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.305 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.305 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:52.305 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:48.863632+0000", 2020-06-12T02:37:52.306 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010296015 2020-06-12T02:37:52.306 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.306 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.306 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:52.306 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:48.873928+0000", 2020-06-12T02:37:52.306 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8671999999999996e-05 2020-06-12T02:37:52.307 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.307 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.307 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:52.307 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:48.873996+0000", 2020-06-12T02:37:52.307 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00069740000000000004 2020-06-12T02:37:52.308 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.308 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.308 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:37:52.308 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:48.874694+0000", 2020-06-12T02:37:52.308 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3091999999999997e-05 2020-06-12T02:37:52.308 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.309 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.309 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:37:52.309 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:48.874767+0000", 2020-06-12T02:37:52.309 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001078313 2020-06-12T02:37:52.309 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:37:52.310 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:37:52.310 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:37:52.310 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:48.875845+0000", 2020-06-12T02:37:52.310 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017431960999999999 2020-06-12T02:37:52.310 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:52.310 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:52.311 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:52.311 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:37:52.311 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:37:52.311 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:37:52.326 INFO:tasks.rados.rados.0.smithi107.stdout:3685: writing smithi10729407-15 from 1425408 to 1933312 tid 1 2020-06-12T02:37:52.326 INFO:tasks.rados.rados.0.smithi107.stdout:3685: writing smithi10729407-15 from 1933312 to 2662400 tid 2 2020-06-12T02:37:52.326 INFO:tasks.rados.rados.0.smithi107.stdout:3685: writing smithi10729407-15 from 2662400 to 3137536 tid 3 2020-06-12T02:37:52.327 INFO:tasks.rados.rados.0.smithi107.stdout:3686: delete oid 12 current snap is 0 2020-06-12T02:37:52.327 INFO:tasks.rados.rados.0.smithi107.stdout:3683: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:52.327 INFO:tasks.rados.rados.0.smithi107.stdout:3683: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:52.327 INFO:tasks.rados.rados.0.smithi107.stdout:3683: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:52.327 INFO:tasks.rados.rados.0.smithi107.stdout:3683: finishing write tid 6 to smithi10729407-10 2020-06-12T02:37:52.328 INFO:tasks.rados.rados.0.smithi107.stdout:3683: finishing write tid 7 to smithi10729407-10 2020-06-12T02:37:52.328 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 367 (ObjNum 3682 snap 0 seq_num 3682) dirty exists 2020-06-12T02:37:52.328 INFO:tasks.rados.rados.0.smithi107.stdout:3683: left oid 10 (ObjNum 3682 snap 0 seq_num 3682) 2020-06-12T02:37:52.328 INFO:tasks.rados.rados.0.smithi107.stdout:3685: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:52.328 INFO:tasks.rados.rados.0.smithi107.stdout:3683: done (2 left) 2020-06-12T02:37:52.328 INFO:tasks.rados.rados.0.smithi107.stdout:3686: done (1 left) 2020-06-12T02:37:52.329 INFO:tasks.rados.rados.0.smithi107.stdout:3687: delete oid 2 current snap is 0 2020-06-12T02:37:52.329 INFO:tasks.rados.rados.0.smithi107.stdout:3687: done (1 left) 2020-06-12T02:37:52.329 INFO:tasks.rados.rados.0.smithi107.stdout:3688: delete oid 7 current snap is 0 2020-06-12T02:37:52.329 INFO:tasks.rados.rados.0.smithi107.stdout:3685: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:52.329 INFO:tasks.rados.rados.0.smithi107.stdout:3688: done (1 left) 2020-06-12T02:37:52.330 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:37:52.330 INFO:tasks.rados.rados.0.smithi107.stdout:3689: seq_num 3688 ranges {0=573440,573440=720896,1294336=483328} 2020-06-12T02:37:52.330 INFO:tasks.rados.rados.0.smithi107.stdout:3689: writing smithi10729407-11 from 0 to 573440 tid 1 2020-06-12T02:37:52.330 INFO:tasks.rados.rados.0.smithi107.stdout:3689: writing smithi10729407-11 from 573440 to 1294336 tid 2 2020-06-12T02:37:52.330 INFO:tasks.rados.rados.0.smithi107.stdout:3689: writing smithi10729407-11 from 1294336 to 1777664 tid 3 2020-06-12T02:37:52.331 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:37:52.331 INFO:tasks.rados.rados.0.smithi107.stdout:3690: seq_num 3689 ranges {0=589824,589824=598016,1187840=606208} 2020-06-12T02:37:52.331 INFO:tasks.rados.rados.0.smithi107.stdout:3690: writing smithi10729407-1 from 0 to 589824 tid 1 2020-06-12T02:37:52.331 INFO:tasks.rados.rados.0.smithi107.stdout:3690: writing smithi10729407-1 from 589824 to 1187840 tid 2 2020-06-12T02:37:52.331 INFO:tasks.rados.rados.0.smithi107.stdout:3690: writing smithi10729407-1 from 1187840 to 1794048 tid 3 2020-06-12T02:37:52.331 INFO:tasks.rados.rados.0.smithi107.stdout:3691: delete oid 18 current snap is 0 2020-06-12T02:37:52.332 INFO:tasks.rados.rados.0.smithi107.stdout:3685: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:52.332 INFO:tasks.rados.rados.0.smithi107.stdout:3685: finishing write tid 5 to smithi10729407-15 2020-06-12T02:37:52.332 INFO:tasks.rados.rados.0.smithi107.stdout:3685: finishing write tid 6 to smithi10729407-15 2020-06-12T02:37:52.332 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 625 (ObjNum 3684 snap 0 seq_num 3684) dirty exists 2020-06-12T02:37:52.332 INFO:tasks.rados.rados.0.smithi107.stdout:3685: left oid 15 (ObjNum 3684 snap 0 seq_num 3684) 2020-06-12T02:37:52.332 INFO:tasks.rados.rados.0.smithi107.stdout:3689: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:52.333 INFO:tasks.rados.rados.0.smithi107.stdout:3689: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:52.333 INFO:tasks.rados.rados.0.smithi107.stdout:3689: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:52.333 INFO:tasks.rados.rados.0.smithi107.stdout:3689: finishing write tid 5 to smithi10729407-11 2020-06-12T02:37:52.333 INFO:tasks.rados.rados.0.smithi107.stdout:3690: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:52.333 INFO:tasks.rados.rados.0.smithi107.stdout:3689: finishing write tid 6 to smithi10729407-11 2020-06-12T02:37:52.334 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 647 (ObjNum 3688 snap 0 seq_num 3688) dirty exists 2020-06-12T02:37:52.334 INFO:tasks.rados.rados.0.smithi107.stdout:3689: left oid 11 (ObjNum 3688 snap 0 seq_num 3688) 2020-06-12T02:37:52.334 INFO:tasks.rados.rados.0.smithi107.stdout:3685: done (3 left) 2020-06-12T02:37:52.334 INFO:tasks.rados.rados.0.smithi107.stdout:3689: done (2 left) 2020-06-12T02:37:52.334 INFO:tasks.rados.rados.0.smithi107.stdout:3691: done (1 left) 2020-06-12T02:37:52.335 INFO:tasks.rados.rados.0.smithi107.stdout:3692: delete oid 11 current snap is 0 2020-06-12T02:37:52.335 INFO:tasks.rados.rados.0.smithi107.stdout:3690: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:52.335 INFO:tasks.rados.rados.0.smithi107.stdout:3690: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:52.335 INFO:tasks.rados.rados.0.smithi107.stdout:3690: finishing write tid 5 to smithi10729407-1 2020-06-12T02:37:52.335 INFO:tasks.rados.rados.0.smithi107.stdout:3692: done (1 left) 2020-06-12T02:37:52.335 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:37:52.336 INFO:tasks.rados.rados.0.smithi107.stdout:3693: seq_num 3692 ranges {0=638976,638976=655360,1294336=548864} 2020-06-12T02:37:52.336 INFO:tasks.rados.rados.0.smithi107.stdout:3693: writing smithi10729407-11 from 0 to 638976 tid 1 2020-06-12T02:37:52.336 INFO:tasks.rados.rados.0.smithi107.stdout:3693: writing smithi10729407-11 from 638976 to 1294336 tid 2 2020-06-12T02:37:52.336 INFO:tasks.rados.rados.0.smithi107.stdout:3693: writing smithi10729407-11 from 1294336 to 1843200 tid 3 2020-06-12T02:37:52.336 INFO:tasks.rados.rados.0.smithi107.stdout:3694: delete oid 9 current snap is 0 2020-06-12T02:37:52.336 INFO:tasks.rados.rados.0.smithi107.stdout:3690: finishing write tid 6 to smithi10729407-1 2020-06-12T02:37:52.337 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 691 (ObjNum 3689 snap 0 seq_num 3689) dirty exists 2020-06-12T02:37:52.337 INFO:tasks.rados.rados.0.smithi107.stdout:3690: left oid 1 (ObjNum 3689 snap 0 seq_num 3689) 2020-06-12T02:37:52.337 INFO:tasks.rados.rados.0.smithi107.stdout:3693: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:52.337 INFO:tasks.rados.rados.0.smithi107.stdout:3690: done (2 left) 2020-06-12T02:37:52.337 INFO:tasks.rados.rados.0.smithi107.stdout:3694: done (1 left) 2020-06-12T02:37:52.338 INFO:tasks.rados.rados.0.smithi107.stdout:3695: delete oid 7 current snap is 0 2020-06-12T02:37:52.338 INFO:tasks.rados.rados.0.smithi107.stdout:3695: done (1 left) 2020-06-12T02:37:52.338 INFO:tasks.rados.rados.0.smithi107.stdout:3696: delete oid 17 current snap is 0 2020-06-12T02:37:52.338 INFO:tasks.rados.rados.0.smithi107.stdout:3693: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:52.338 INFO:tasks.rados.rados.0.smithi107.stdout:3693: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:52.338 INFO:tasks.rados.rados.0.smithi107.stdout:3693: finishing write tid 5 to smithi10729407-11 2020-06-12T02:37:52.339 INFO:tasks.rados.rados.0.smithi107.stdout:3696: done (1 left) 2020-06-12T02:37:52.339 INFO:tasks.rados.rados.0.smithi107.stdout:3697: delete oid 16 current snap is 0 2020-06-12T02:37:52.339 INFO:tasks.rados.rados.0.smithi107.stdout:3697: done (1 left) 2020-06-12T02:37:52.339 INFO:tasks.rados.rados.0.smithi107.stdout:3698: delete oid 7 current snap is 0 2020-06-12T02:37:52.339 INFO:tasks.rados.rados.0.smithi107.stdout:3693: finishing write tid 6 to smithi10729407-11 2020-06-12T02:37:52.340 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 652 (ObjNum 3692 snap 0 seq_num 3692) dirty exists 2020-06-12T02:37:52.340 INFO:tasks.rados.rados.0.smithi107.stdout:3693: left oid 11 (ObjNum 3692 snap 0 seq_num 3692) 2020-06-12T02:37:52.340 INFO:tasks.rados.rados.0.smithi107.stdout:3693: done (1 left) 2020-06-12T02:37:52.340 INFO:tasks.rados.rados.0.smithi107.stdout:3698: done (0 left) 2020-06-12T02:37:52.340 INFO:tasks.rados.rados.0.smithi107.stdout:3699: delete oid 12 current snap is 0 2020-06-12T02:37:52.340 INFO:tasks.rados.rados.0.smithi107.stdout:3699: done (0 left) 2020-06-12T02:37:52.341 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:37:52.341 INFO:tasks.rados.rados.0.smithi107.stdout:3700: seq_num 3699 ranges {0=761856,761856=696320,1458176=507904} 2020-06-12T02:37:52.341 INFO:tasks.rados.rados.0.smithi107.stdout:3700: writing smithi10729407-20 from 0 to 761856 tid 1 2020-06-12T02:37:52.341 INFO:tasks.rados.rados.0.smithi107.stdout:3700: writing smithi10729407-20 from 761856 to 1458176 tid 2 2020-06-12T02:37:52.341 INFO:tasks.rados.rados.0.smithi107.stdout:3700: writing smithi10729407-20 from 1458176 to 1966080 tid 3 2020-06-12T02:37:52.342 INFO:tasks.rados.rados.0.smithi107.stdout:3701: delete oid 2 current snap is 0 2020-06-12T02:37:52.342 INFO:tasks.rados.rados.0.smithi107.stdout:3700: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:52.342 INFO:tasks.rados.rados.0.smithi107.stdout:3701: done (1 left) 2020-06-12T02:37:52.342 INFO:tasks.rados.rados.0.smithi107.stdout:3702: delete oid 10 current snap is 0 2020-06-12T02:37:52.342 INFO:tasks.rados.rados.0.smithi107.stdout:3702: done (1 left) 2020-06-12T02:37:52.342 INFO:tasks.rados.rados.0.smithi107.stdout:3703: delete oid 14 current snap is 0 2020-06-12T02:37:52.343 INFO:tasks.rados.rados.0.smithi107.stdout:3700: finishing write tid 2 to smithi10729407-20 2020-06-12T02:37:52.343 INFO:tasks.rados.rados.0.smithi107.stdout:3700: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:52.343 INFO:tasks.rados.rados.0.smithi107.stdout:3703: done (1 left) 2020-06-12T02:37:52.343 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:37:52.343 INFO:tasks.rados.rados.0.smithi107.stdout:3704: seq_num 3703 ranges {0=425984,425984=679936,1105920=737280,1843200=188416} 2020-06-12T02:37:52.343 INFO:tasks.rados.rados.0.smithi107.stdout:3704: writing smithi10729407-17 from 0 to 425984 tid 1 2020-06-12T02:37:52.344 INFO:tasks.rados.rados.0.smithi107.stdout:3704: writing smithi10729407-17 from 425984 to 1105920 tid 2 2020-06-12T02:37:52.344 INFO:tasks.rados.rados.0.smithi107.stdout:3704: writing smithi10729407-17 from 1105920 to 1843200 tid 3 2020-06-12T02:37:52.344 INFO:tasks.rados.rados.0.smithi107.stdout:3704: writing smithi10729407-17 from 1843200 to 2031616 tid 4 2020-06-12T02:37:52.344 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:37:52.344 INFO:tasks.rados.rados.0.smithi107.stdout:3705: seq_num 3704 ranges {0=442368,442368=565248,1007616=491520,1499136=548864} 2020-06-12T02:37:52.345 INFO:tasks.rados.rados.0.smithi107.stdout:3705: writing smithi10729407-9 from 0 to 442368 tid 1 2020-06-12T02:37:52.345 INFO:tasks.rados.rados.0.smithi107.stdout:3705: writing smithi10729407-9 from 442368 to 1007616 tid 2 2020-06-12T02:37:52.345 INFO:tasks.rados.rados.0.smithi107.stdout:3705: writing smithi10729407-9 from 1007616 to 1499136 tid 3 2020-06-12T02:37:52.345 INFO:tasks.rados.rados.0.smithi107.stdout:3705: writing smithi10729407-9 from 1499136 to 2048000 tid 4 2020-06-12T02:37:52.345 INFO:tasks.rados.rados.0.smithi107.stdout:3706: delete oid 1 current snap is 0 2020-06-12T02:37:52.345 INFO:tasks.rados.rados.0.smithi107.stdout:3700: finishing write tid 5 to smithi10729407-20 2020-06-12T02:37:52.346 INFO:tasks.rados.rados.0.smithi107.stdout:3700: finishing write tid 6 to smithi10729407-20 2020-06-12T02:37:52.346 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 281 (ObjNum 3699 snap 0 seq_num 3699) dirty exists 2020-06-12T02:37:52.346 INFO:tasks.rados.rados.0.smithi107.stdout:3700: left oid 20 (ObjNum 3699 snap 0 seq_num 3699) 2020-06-12T02:37:52.346 INFO:tasks.rados.rados.0.smithi107.stdout:3704: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:52.346 INFO:tasks.rados.rados.0.smithi107.stdout:3704: finishing write tid 2 to smithi10729407-17 2020-06-12T02:37:52.347 INFO:tasks.rados.rados.0.smithi107.stdout:3704: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:52.347 INFO:tasks.rados.rados.0.smithi107.stdout:3704: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:52.347 INFO:tasks.rados.rados.0.smithi107.stdout:3705: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:52.347 INFO:tasks.rados.rados.0.smithi107.stdout:3704: finishing write tid 6 to smithi10729407-17 2020-06-12T02:37:52.347 INFO:tasks.rados.rados.0.smithi107.stdout:3704: finishing write tid 7 to smithi10729407-17 2020-06-12T02:37:52.347 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 688 (ObjNum 3703 snap 0 seq_num 3703) dirty exists 2020-06-12T02:37:52.348 INFO:tasks.rados.rados.0.smithi107.stdout:3704: left oid 17 (ObjNum 3703 snap 0 seq_num 3703) 2020-06-12T02:37:52.348 INFO:tasks.rados.rados.0.smithi107.stdout:3700: done (3 left) 2020-06-12T02:37:52.348 INFO:tasks.rados.rados.0.smithi107.stdout:3704: done (2 left) 2020-06-12T02:37:52.348 INFO:tasks.rados.rados.0.smithi107.stdout:3706: done (1 left) 2020-06-12T02:37:52.348 INFO:tasks.rados.rados.0.smithi107.stdout:3707: delete oid 20 current snap is 0 2020-06-12T02:37:52.349 INFO:tasks.rados.rados.0.smithi107.stdout:3705: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:52.349 INFO:tasks.rados.rados.0.smithi107.stdout:3705: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:52.349 INFO:tasks.rados.rados.0.smithi107.stdout:3705: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:52.349 INFO:tasks.rados.rados.0.smithi107.stdout:3705: finishing write tid 6 to smithi10729407-9 2020-06-12T02:37:52.349 INFO:tasks.rados.rados.0.smithi107.stdout:3707: done (1 left) 2020-06-12T02:37:52.349 INFO:tasks.rados.rados.0.smithi107.stdout:3708: delete oid 11 current snap is 0 2020-06-12T02:37:52.350 INFO:tasks.rados.rados.0.smithi107.stdout:3705: finishing write tid 7 to smithi10729407-9 2020-06-12T02:37:52.350 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 657 (ObjNum 3704 snap 0 seq_num 3704) dirty exists 2020-06-12T02:37:52.350 INFO:tasks.rados.rados.0.smithi107.stdout:3705: left oid 9 (ObjNum 3704 snap 0 seq_num 3704) 2020-06-12T02:37:52.350 INFO:tasks.rados.rados.0.smithi107.stdout:3705: done (1 left) 2020-06-12T02:37:52.350 INFO:tasks.rados.rados.0.smithi107.stdout:3708: done (0 left) 2020-06-12T02:37:52.350 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:37:52.351 INFO:tasks.rados.rados.0.smithi107.stdout:3709: seq_num 3708 ranges {1073152=507904,1581056=499712,2080768=475136,2555904=606208,3162112=24576} 2020-06-12T02:37:52.351 INFO:tasks.rados.rados.0.smithi107.stdout:3709: writing smithi10729407-8 from 1073152 to 1581056 tid 1 2020-06-12T02:37:52.351 INFO:tasks.rados.rados.0.smithi107.stdout:3709: writing smithi10729407-8 from 1581056 to 2080768 tid 2 2020-06-12T02:37:52.351 INFO:tasks.rados.rados.0.smithi107.stdout:3709: writing smithi10729407-8 from 2080768 to 2555904 tid 3 2020-06-12T02:37:52.351 INFO:tasks.rados.rados.0.smithi107.stdout:3709: writing smithi10729407-8 from 2555904 to 3162112 tid 4 2020-06-12T02:37:52.352 INFO:tasks.rados.rados.0.smithi107.stdout:3709: writing smithi10729407-8 from 3162112 to 3186688 tid 5 2020-06-12T02:37:52.352 INFO:tasks.rados.rados.0.smithi107.stdout:3710: delete oid 1 current snap is 0 2020-06-12T02:37:52.352 INFO:tasks.rados.rados.0.smithi107.stdout:3709: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:52.352 INFO:tasks.rados.rados.0.smithi107.stdout:3710: done (1 left) 2020-06-12T02:37:52.352 INFO:tasks.rados.rados.0.smithi107.stdout:3711: delete oid 13 current snap is 0 2020-06-12T02:37:52.352 INFO:tasks.rados.rados.0.smithi107.stdout:3711: done (1 left) 2020-06-12T02:37:52.353 INFO:tasks.rados.rados.0.smithi107.stdout:3712: delete oid 9 current snap is 0 2020-06-12T02:37:52.353 INFO:tasks.rados.rados.0.smithi107.stdout:3709: finishing write tid 2 to smithi10729407-8 2020-06-12T02:37:52.353 INFO:tasks.rados.rados.0.smithi107.stdout:3709: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:52.353 INFO:tasks.rados.rados.0.smithi107.stdout:3709: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:52.353 INFO:tasks.rados.rados.0.smithi107.stdout:3712: done (1 left) 2020-06-12T02:37:52.354 INFO:tasks.rados.rados.0.smithi107.stdout:3713: delete oid 17 current snap is 0 2020-06-12T02:37:52.354 INFO:tasks.rados.rados.0.smithi107.stdout:3713: done (1 left) 2020-06-12T02:37:52.354 INFO:tasks.rados.rados.0.smithi107.stdout:3714: delete oid 9 current snap is 0 2020-06-12T02:37:52.354 INFO:tasks.rados.rados.0.smithi107.stdout:3709: finishing write tid 5 to smithi10729407-8 2020-06-12T02:37:52.354 INFO:tasks.rados.rados.0.smithi107.stdout:3709: finishing write tid 7 to smithi10729407-8 2020-06-12T02:37:52.354 INFO:tasks.rados.rados.0.smithi107.stdout:3709: finishing write tid 8 to smithi10729407-8 2020-06-12T02:37:52.355 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 631 (ObjNum 3708 snap 0 seq_num 3708) dirty exists 2020-06-12T02:37:52.355 INFO:tasks.rados.rados.0.smithi107.stdout:3709: left oid 8 (ObjNum 3708 snap 0 seq_num 3708) 2020-06-12T02:37:52.355 INFO:tasks.rados.rados.0.smithi107.stdout:3709: done (1 left) 2020-06-12T02:37:52.355 INFO:tasks.rados.rados.0.smithi107.stdout:3714: done (0 left) 2020-06-12T02:37:52.355 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:37:52.355 INFO:tasks.rados.rados.0.smithi107.stdout:3715: seq_num 3714 ranges {0=606208,606208=655360,1261568=466944,1728512=483328} 2020-06-12T02:37:52.356 INFO:tasks.rados.rados.0.smithi107.stdout:3715: writing smithi10729407-13 from 0 to 606208 tid 1 2020-06-12T02:37:52.356 INFO:tasks.rados.rados.0.smithi107.stdout:3715: writing smithi10729407-13 from 606208 to 1261568 tid 2 2020-06-12T02:37:52.356 INFO:tasks.rados.rados.0.smithi107.stdout:3715: writing smithi10729407-13 from 1261568 to 1728512 tid 3 2020-06-12T02:37:52.356 INFO:tasks.rados.rados.0.smithi107.stdout:3715: writing smithi10729407-13 from 1728512 to 2211840 tid 4 2020-06-12T02:37:52.356 INFO:tasks.rados.rados.0.smithi107.stdout:3716: delete oid 11 current snap is 0 2020-06-12T02:37:52.357 INFO:tasks.rados.rados.0.smithi107.stdout:3715: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:52.357 INFO:tasks.rados.rados.0.smithi107.stdout:3716: done (1 left) 2020-06-12T02:37:52.357 INFO:tasks.rados.rados.0.smithi107.stdout:3717: delete oid 20 current snap is 0 2020-06-12T02:37:52.357 INFO:tasks.rados.rados.0.smithi107.stdout:3717: done (1 left) 2020-06-12T02:37:52.357 INFO:tasks.rados.rados.0.smithi107.stdout:3718: delete oid 12 current snap is 0 2020-06-12T02:37:52.357 INFO:tasks.rados.rados.0.smithi107.stdout:3715: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:52.358 INFO:tasks.rados.rados.0.smithi107.stdout:3715: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:52.358 INFO:tasks.rados.rados.0.smithi107.stdout:3715: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:52.358 INFO:tasks.rados.rados.0.smithi107.stdout:3718: done (1 left) 2020-06-12T02:37:52.358 INFO:tasks.rados.rados.0.smithi107.stdout:3719: delete oid 11 current snap is 0 2020-06-12T02:37:52.359 INFO:tasks.rados.rados.0.smithi107.stdout:3715: finishing write tid 6 to smithi10729407-13 2020-06-12T02:37:52.359 INFO:tasks.rados.rados.0.smithi107.stdout:3715: finishing write tid 7 to smithi10729407-13 2020-06-12T02:37:52.359 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 704 (ObjNum 3714 snap 0 seq_num 3714) dirty exists 2020-06-12T02:37:52.359 INFO:tasks.rados.rados.0.smithi107.stdout:3715: left oid 13 (ObjNum 3714 snap 0 seq_num 3714) 2020-06-12T02:37:52.359 INFO:tasks.rados.rados.0.smithi107.stdout:3715: done (1 left) 2020-06-12T02:37:52.360 INFO:tasks.rados.rados.0.smithi107.stdout:3719: done (0 left) 2020-06-12T02:37:52.360 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:37:52.360 INFO:tasks.rados.rados.0.smithi107.stdout:3720: seq_num 3719 ranges {0=696320,696320=532480,1228800=655360,1884160=417792} 2020-06-12T02:37:52.360 INFO:tasks.rados.rados.0.smithi107.stdout:3720: writing smithi10729407-10 from 0 to 696320 tid 1 2020-06-12T02:37:52.360 INFO:tasks.rados.rados.0.smithi107.stdout:3720: writing smithi10729407-10 from 696320 to 1228800 tid 2 2020-06-12T02:37:52.360 INFO:tasks.rados.rados.0.smithi107.stdout:3720: writing smithi10729407-10 from 1228800 to 1884160 tid 3 2020-06-12T02:37:52.361 INFO:tasks.rados.rados.0.smithi107.stdout:3720: writing smithi10729407-10 from 1884160 to 2301952 tid 4 2020-06-12T02:37:52.361 INFO:tasks.rados.rados.0.smithi107.stdout:3721: delete oid 12 current snap is 0 2020-06-12T02:37:52.361 INFO:tasks.rados.rados.0.smithi107.stdout:3720: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:52.361 INFO:tasks.rados.rados.0.smithi107.stdout:3721: done (1 left) 2020-06-12T02:37:52.361 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:37:52.361 INFO:tasks.rados.rados.0.smithi107.stdout:3722: seq_num 3721 ranges {3186688=729088,3915776=696320,4612096=622592,5234688=286720} 2020-06-12T02:37:52.362 INFO:tasks.rados.rados.0.smithi107.stdout:3722: writing smithi10729407-8 from 3186688 to 3915776 tid 1 2020-06-12T02:37:52.362 INFO:tasks.rados.rados.0.smithi107.stdout:3722: writing smithi10729407-8 from 3915776 to 4612096 tid 2 2020-06-12T02:37:52.362 INFO:tasks.rados.rados.0.smithi107.stdout:3722: writing smithi10729407-8 from 4612096 to 5234688 tid 3 2020-06-12T02:37:52.362 INFO:tasks.rados.rados.0.smithi107.stdout:3722: writing smithi10729407-8 from 5234688 to 5521408 tid 4 2020-06-12T02:37:52.362 INFO:tasks.rados.rados.0.smithi107.stdout:3723: delete oid 11 current snap is 0 2020-06-12T02:37:52.363 INFO:tasks.rados.rados.0.smithi107.stdout:3720: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:52.364 INFO:tasks.rados.rados.0.smithi107.stdout:3720: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:52.365 INFO:tasks.rados.rados.0.smithi107.stdout:3720: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:52.365 INFO:tasks.rados.rados.0.smithi107.stdout:3720: finishing write tid 6 to smithi10729407-10 2020-06-12T02:37:52.365 INFO:tasks.rados.rados.0.smithi107.stdout:3720: finishing write tid 7 to smithi10729407-10 2020-06-12T02:37:52.365 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 373 (ObjNum 3719 snap 0 seq_num 3719) dirty exists 2020-06-12T02:37:52.365 INFO:tasks.rados.rados.0.smithi107.stdout:3720: left oid 10 (ObjNum 3719 snap 0 seq_num 3719) 2020-06-12T02:37:52.366 INFO:tasks.rados.rados.0.smithi107.stdout:3722: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:52.366 INFO:tasks.rados.rados.0.smithi107.stdout:3720: done (2 left) 2020-06-12T02:37:52.366 INFO:tasks.rados.rados.0.smithi107.stdout:3723: done (1 left) 2020-06-12T02:37:52.366 INFO:tasks.rados.rados.0.smithi107.stdout:3724: delete oid 7 current snap is 0 2020-06-12T02:37:52.366 INFO:tasks.rados.rados.0.smithi107.stdout:3724: done (1 left) 2020-06-12T02:37:52.366 INFO:tasks.rados.rados.0.smithi107.stdout:3725: delete oid 13 current snap is 0 2020-06-12T02:37:52.367 INFO:tasks.rados.rados.0.smithi107.stdout:3722: finishing write tid 2 to smithi10729407-8 2020-06-12T02:37:52.367 INFO:tasks.rados.rados.0.smithi107.stdout:3722: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:52.367 INFO:tasks.rados.rados.0.smithi107.stdout:3725: done (1 left) 2020-06-12T02:37:52.367 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:37:52.367 INFO:tasks.rados.rados.0.smithi107.stdout:3726: seq_num 3725 ranges {0=794624,794624=638976,1433600=598016,2031616=368640} 2020-06-12T02:37:52.368 INFO:tasks.rados.rados.0.smithi107.stdout:3726: writing smithi10729407-20 from 0 to 794624 tid 1 2020-06-12T02:37:52.368 INFO:tasks.rados.rados.0.smithi107.stdout:3726: writing smithi10729407-20 from 794624 to 1433600 tid 2 2020-06-12T02:37:52.368 INFO:tasks.rados.rados.0.smithi107.stdout:3726: writing smithi10729407-20 from 1433600 to 2031616 tid 3 2020-06-12T02:37:52.368 INFO:tasks.rados.rados.0.smithi107.stdout:3726: writing smithi10729407-20 from 2031616 to 2400256 tid 4 2020-06-12T02:37:52.368 INFO:tasks.rados.rados.0.smithi107.stdout:3727: delete oid 6 current snap is 0 2020-06-12T02:37:52.368 INFO:tasks.rados.rados.0.smithi107.stdout:3722: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:52.369 INFO:tasks.rados.rados.0.smithi107.stdout:3722: finishing write tid 6 to smithi10729407-8 2020-06-12T02:37:52.369 INFO:tasks.rados.rados.0.smithi107.stdout:3722: finishing write tid 7 to smithi10729407-8 2020-06-12T02:37:52.369 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 636 (ObjNum 3721 snap 0 seq_num 3721) dirty exists 2020-06-12T02:37:52.369 INFO:tasks.rados.rados.0.smithi107.stdout:3722: left oid 8 (ObjNum 3721 snap 0 seq_num 3721) 2020-06-12T02:37:52.369 INFO:tasks.rados.rados.0.smithi107.stdout:3726: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:52.370 INFO:tasks.rados.rados.0.smithi107.stdout:3722: done (2 left) 2020-06-12T02:37:52.370 INFO:tasks.rados.rados.0.smithi107.stdout:3727: done (1 left) 2020-06-12T02:37:52.370 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:37:52.370 INFO:tasks.rados.rados.0.smithi107.stdout:3728: seq_num 3727 ranges {0=24576} 2020-06-12T02:37:52.370 INFO:tasks.rados.rados.0.smithi107.stdout:3728: writing smithi10729407-17 from 0 to 24576 tid 1 2020-06-12T02:37:52.371 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:37:52.371 INFO:tasks.rados.rados.0.smithi107.stdout:3729: seq_num 3728 ranges {5521408=40960} 2020-06-12T02:37:52.371 INFO:tasks.rados.rados.0.smithi107.stdout:3729: writing smithi10729407-8 from 5521408 to 5562368 tid 1 2020-06-12T02:37:52.371 INFO:tasks.rados.rados.0.smithi107.stdout:3730: delete oid 12 current snap is 0 2020-06-12T02:37:52.371 INFO:tasks.rados.rados.0.smithi107.stdout:3726: finishing write tid 2 to smithi10729407-20 2020-06-12T02:37:52.371 INFO:tasks.rados.rados.0.smithi107.stdout:3726: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:52.372 INFO:tasks.rados.rados.0.smithi107.stdout:3726: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:52.372 INFO:tasks.rados.rados.0.smithi107.stdout:3728: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:52.372 INFO:tasks.rados.rados.0.smithi107.stdout:3726: finishing write tid 6 to smithi10729407-20 2020-06-12T02:37:52.372 INFO:tasks.rados.rados.0.smithi107.stdout:3728: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:52.372 INFO:tasks.rados.rados.0.smithi107.stdout:3726: finishing write tid 7 to smithi10729407-20 2020-06-12T02:37:52.373 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 288 (ObjNum 3725 snap 0 seq_num 3725) dirty exists 2020-06-12T02:37:52.373 INFO:tasks.rados.rados.0.smithi107.stdout:3726: left oid 20 (ObjNum 3725 snap 0 seq_num 3725) 2020-06-12T02:37:52.373 INFO:tasks.rados.rados.0.smithi107.stdout:3728: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:52.373 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 691 (ObjNum 3727 snap 0 seq_num 3727) dirty exists 2020-06-12T02:37:52.374 INFO:tasks.rados.rados.0.smithi107.stdout:3728: left oid 17 (ObjNum 3727 snap 0 seq_num 3727) 2020-06-12T02:37:52.374 INFO:tasks.rados.rados.0.smithi107.stdout:3726: done (3 left) 2020-06-12T02:37:52.374 INFO:tasks.rados.rados.0.smithi107.stdout:3728: done (2 left) 2020-06-12T02:37:52.374 INFO:tasks.rados.rados.0.smithi107.stdout:3730: done (1 left) 2020-06-12T02:37:52.374 INFO:tasks.rados.rados.0.smithi107.stdout:3731: delete oid 15 current snap is 0 2020-06-12T02:37:52.374 INFO:tasks.rados.rados.0.smithi107.stdout:3729: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:52.375 INFO:tasks.rados.rados.0.smithi107.stdout:3729: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:52.375 INFO:tasks.rados.rados.0.smithi107.stdout:3731: done (1 left) 2020-06-12T02:37:52.375 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:37:52.375 INFO:tasks.rados.rados.0.smithi107.stdout:3732: seq_num 3731 ranges {0=90112} 2020-06-12T02:37:52.375 INFO:tasks.rados.rados.0.smithi107.stdout:3732: writing smithi10729407-12 from 0 to 90112 tid 1 2020-06-12T02:37:52.376 INFO:tasks.rados.rados.0.smithi107.stdout:3733: delete oid 15 current snap is 0 2020-06-12T02:37:52.376 INFO:tasks.rados.rados.0.smithi107.stdout:3729: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:52.376 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 638 (ObjNum 3728 snap 0 seq_num 3728) dirty exists 2020-06-12T02:37:52.376 INFO:tasks.rados.rados.0.smithi107.stdout:3729: left oid 8 (ObjNum 3728 snap 0 seq_num 3728) 2020-06-12T02:37:52.376 INFO:tasks.rados.rados.0.smithi107.stdout:3732: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:52.376 INFO:tasks.rados.rados.0.smithi107.stdout:3729: done (2 left) 2020-06-12T02:37:52.377 INFO:tasks.rados.rados.0.smithi107.stdout:3733: done (1 left) 2020-06-12T02:37:52.377 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:37:52.377 INFO:tasks.rados.rados.0.smithi107.stdout:3734: seq_num 3733 ranges {0=122880} 2020-06-12T02:37:52.377 INFO:tasks.rados.rados.0.smithi107.stdout:3734: writing smithi10729407-18 from 0 to 122880 tid 1 2020-06-12T02:37:52.377 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:37:52.378 INFO:tasks.rados.rados.0.smithi107.stdout:3735: seq_num 3734 ranges {0=139264} 2020-06-12T02:37:52.378 INFO:tasks.rados.rados.0.smithi107.stdout:3735: writing smithi10729407-15 from 0 to 139264 tid 1 2020-06-12T02:37:52.378 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:37:52.378 INFO:tasks.rados.rados.0.smithi107.stdout:3736: seq_num 3735 ranges {3063808=155648} 2020-06-12T02:37:52.379 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-06-12T02:37:52.379 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:37:52.380 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:37:52.380 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:37:52.380 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-06-12T02:37:52.380 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-06-12T02:37:52.381 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-06-12T02:37:52.381 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-06-12T02:37:52.381 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-06-12T02:37:52.381 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-06-12T02:37:52.381 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-06-12T02:37:52.382 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.7 2020-06-12T02:37:52.382 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: problem getting command descriptions from osd.7 2020-06-12T02:37:52.382 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-06-12T02:37:52.382 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-06-12T02:37:52.383 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:52.310+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:52.387 INFO:tasks.rados.rados.0.smithi107.stdout:3736: writing smithi10729407-5 from 3063808 to 3219456 tid 1 2020-06-12T02:37:52.387 INFO:tasks.rados.rados.0.smithi107.stdout:3737: delete oid 3 current snap is 0 2020-06-12T02:37:52.387 INFO:tasks.rados.rados.0.smithi107.stdout:3732: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:52.387 INFO:tasks.rados.rados.0.smithi107.stdout:3734: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:52.387 INFO:tasks.rados.rados.0.smithi107.stdout:3735: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:52.388 INFO:tasks.rados.rados.0.smithi107.stdout:3732: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:52.388 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 623 (ObjNum 3731 snap 0 seq_num 3731) dirty exists 2020-06-12T02:37:52.388 INFO:tasks.rados.rados.0.smithi107.stdout:3732: left oid 12 (ObjNum 3731 snap 0 seq_num 3731) 2020-06-12T02:37:52.388 INFO:tasks.rados.rados.0.smithi107.stdout:3734: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:52.388 INFO:tasks.rados.rados.0.smithi107.stdout:3735: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:52.389 INFO:tasks.rados.rados.0.smithi107.stdout:3734: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:52.389 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 661 (ObjNum 3733 snap 0 seq_num 3733) dirty exists 2020-06-12T02:37:52.389 INFO:tasks.rados.rados.0.smithi107.stdout:3734: left oid 18 (ObjNum 3733 snap 0 seq_num 3733) 2020-06-12T02:37:52.390 INFO:tasks.rados.rados.0.smithi107.stdout:3735: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:52.390 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 642 (ObjNum 3734 snap 0 seq_num 3734) dirty exists 2020-06-12T02:37:52.390 INFO:tasks.rados.rados.0.smithi107.stdout:3735: left oid 15 (ObjNum 3734 snap 0 seq_num 3734) 2020-06-12T02:37:52.390 INFO:tasks.rados.rados.0.smithi107.stdout:3732: done (4 left) 2020-06-12T02:37:52.390 INFO:tasks.rados.rados.0.smithi107.stdout:3734: done (3 left) 2020-06-12T02:37:52.391 INFO:tasks.rados.rados.0.smithi107.stdout:3735: done (2 left) 2020-06-12T02:37:52.391 INFO:tasks.rados.rados.0.smithi107.stdout:3737: done (1 left) 2020-06-12T02:37:52.391 INFO:tasks.rados.rados.0.smithi107.stdout:3738: delete oid 18 current snap is 0 2020-06-12T02:37:52.391 INFO:tasks.rados.rados.0.smithi107.stdout:3736: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:52.391 INFO:tasks.rados.rados.0.smithi107.stdout:3738: done (1 left) 2020-06-12T02:37:52.391 INFO:tasks.rados.rados.0.smithi107.stdout:3739: delete oid 20 current snap is 0 2020-06-12T02:37:52.392 INFO:tasks.rados.rados.0.smithi107.stdout:3736: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:52.392 INFO:tasks.rados.rados.0.smithi107.stdout:3736: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:52.392 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 341 (ObjNum 3735 snap 0 seq_num 3735) dirty exists 2020-06-12T02:37:52.392 INFO:tasks.rados.rados.0.smithi107.stdout:3736: left oid 5 (ObjNum 3735 snap 0 seq_num 3735) 2020-06-12T02:37:52.392 INFO:tasks.rados.rados.0.smithi107.stdout:3736: done (1 left) 2020-06-12T02:37:52.393 INFO:tasks.rados.rados.0.smithi107.stdout:3739: done (0 left) 2020-06-12T02:37:52.393 INFO:tasks.rados.rados.0.smithi107.stdout:3740: delete oid 4 current snap is 0 2020-06-12T02:37:52.393 INFO:tasks.rados.rados.0.smithi107.stdout:3740: done (0 left) 2020-06-12T02:37:52.393 INFO:tasks.rados.rados.0.smithi107.stdout:3741: delete oid 16 current snap is 0 2020-06-12T02:37:52.393 INFO:tasks.rados.rados.0.smithi107.stdout:3741: done (0 left) 2020-06-12T02:37:52.393 INFO:tasks.rados.rados.0.smithi107.stdout:3742: delete oid 14 current snap is 0 2020-06-12T02:37:52.394 INFO:tasks.rados.rados.0.smithi107.stdout:3742: done (0 left) 2020-06-12T02:37:52.394 INFO:tasks.rados.rados.0.smithi107.stdout:3743: delete oid 9 current snap is 0 2020-06-12T02:37:52.394 INFO:tasks.rados.rados.0.smithi107.stdout:3743: done (0 left) 2020-06-12T02:37:52.394 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:37:52.394 INFO:tasks.rados.rados.0.smithi107.stdout:3744: seq_num 3743 ranges {90112=294912} 2020-06-12T02:37:52.395 INFO:tasks.rados.rados.0.smithi107.stdout:3744: writing smithi10729407-12 from 90112 to 385024 tid 1 2020-06-12T02:37:52.395 INFO:tasks.rados.rados.0.smithi107.stdout:3745: delete oid 13 current snap is 0 2020-06-12T02:37:52.395 INFO:tasks.rados.rados.0.smithi107.stdout:3745: done (1 left) 2020-06-12T02:37:52.395 INFO:tasks.rados.rados.0.smithi107.stdout:3746: delete oid 1 current snap is 0 2020-06-12T02:37:52.395 INFO:tasks.rados.rados.0.smithi107.stdout:3744: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:52.395 INFO:tasks.rados.rados.0.smithi107.stdout:3746: done (1 left) 2020-06-12T02:37:52.396 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:37:52.396 INFO:tasks.rados.rados.0.smithi107.stdout:3747: seq_num 3746 ranges {139264=344064} 2020-06-12T02:37:52.396 INFO:tasks.rados.rados.0.smithi107.stdout:3747: writing smithi10729407-15 from 139264 to 483328 tid 1 2020-06-12T02:37:52.396 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:37:52.396 INFO:tasks.rados.rados.0.smithi107.stdout:3748: seq_num 3747 ranges {0=360448} 2020-06-12T02:37:52.397 INFO:tasks.rados.rados.0.smithi107.stdout:3748: writing smithi10729407-3 from 0 to 360448 tid 1 2020-06-12T02:37:52.397 INFO:tasks.rados.rados.0.smithi107.stdout:3749: delete oid 1 current snap is 0 2020-06-12T02:37:52.397 INFO:tasks.rados.rados.0.smithi107.stdout:3744: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:52.398 INFO:tasks.rados.rados.0.smithi107.stdout:3744: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:52.398 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 625 (ObjNum 3743 snap 0 seq_num 3743) dirty exists 2020-06-12T02:37:52.398 INFO:tasks.rados.rados.0.smithi107.stdout:3744: left oid 12 (ObjNum 3743 snap 0 seq_num 3743) 2020-06-12T02:37:52.398 INFO:tasks.rados.rados.0.smithi107.stdout:3747: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:52.398 INFO:tasks.rados.rados.0.smithi107.stdout:3744: done (3 left) 2020-06-12T02:37:52.399 INFO:tasks.rados.rados.0.smithi107.stdout:3749: done (2 left) 2020-06-12T02:37:52.399 INFO:tasks.rados.rados.0.smithi107.stdout:3750: delete oid 9 current snap is 0 2020-06-12T02:37:52.399 INFO:tasks.rados.rados.0.smithi107.stdout:3747: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:52.399 INFO:tasks.rados.rados.0.smithi107.stdout:3748: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:52.399 INFO:tasks.rados.rados.0.smithi107.stdout:3747: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:52.399 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 644 (ObjNum 3746 snap 0 seq_num 3746) dirty exists 2020-06-12T02:37:52.400 INFO:tasks.rados.rados.0.smithi107.stdout:3747: left oid 15 (ObjNum 3746 snap 0 seq_num 3746) 2020-06-12T02:37:52.400 INFO:tasks.rados.rados.0.smithi107.stdout:3748: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:52.400 INFO:tasks.rados.rados.0.smithi107.stdout:3748: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:52.400 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 670 (ObjNum 3747 snap 0 seq_num 3747) dirty exists 2020-06-12T02:37:52.400 INFO:tasks.rados.rados.0.smithi107.stdout:3748: left oid 3 (ObjNum 3747 snap 0 seq_num 3747) 2020-06-12T02:37:52.401 INFO:tasks.rados.rados.0.smithi107.stdout:3747: done (2 left) 2020-06-12T02:37:52.401 INFO:tasks.rados.rados.0.smithi107.stdout:3748: done (1 left) 2020-06-12T02:37:52.401 INFO:tasks.rados.rados.0.smithi107.stdout:3750: done (0 left) 2020-06-12T02:37:52.401 INFO:tasks.rados.rados.0.smithi107.stdout:3751: delete oid 2 current snap is 0 2020-06-12T02:37:52.401 INFO:tasks.rados.rados.0.smithi107.stdout:3751: done (0 left) 2020-06-12T02:37:52.402 INFO:tasks.rados.rados.0.smithi107.stdout:3752: delete oid 5 current snap is 0 2020-06-12T02:37:52.402 INFO:tasks.rados.rados.0.smithi107.stdout:3752: done (0 left) 2020-06-12T02:37:52.402 INFO:tasks.rados.rados.0.smithi107.stdout:3753: delete oid 16 current snap is 0 2020-06-12T02:37:52.402 INFO:tasks.rados.rados.0.smithi107.stdout:3753: done (0 left) 2020-06-12T02:37:52.403 INFO:tasks.rados.rados.0.smithi107.stdout:3754: delete oid 1 current snap is 0 2020-06-12T02:37:52.403 INFO:tasks.rados.rados.0.smithi107.stdout:3754: done (0 left) 2020-06-12T02:37:52.403 INFO:tasks.rados.rados.0.smithi107.stdout:3755: delete oid 17 current snap is 0 2020-06-12T02:37:52.405 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:37:52.388+0000 7f235bf1fec0 -1 Falling back to public interface 2020-06-12T02:37:52.406 INFO:tasks.rados.rados.0.smithi107.stdout:3755: done (0 left) 2020-06-12T02:37:52.406 INFO:tasks.rados.rados.0.smithi107.stdout:3756: delete oid 2 current snap is 0 2020-06-12T02:37:52.406 INFO:tasks.rados.rados.0.smithi107.stdout:3756: done (0 left) 2020-06-12T02:37:52.406 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:37:52.407 INFO:tasks.rados.rados.0.smithi107.stdout:3757: seq_num 3756 ranges {0=507904} 2020-06-12T02:37:52.407 INFO:tasks.rados.rados.0.smithi107.stdout:3757: writing smithi10729407-6 from 0 to 507904 tid 1 2020-06-12T02:37:52.407 INFO:tasks.rados.rados.0.smithi107.stdout:3758: delete oid 8 current snap is 0 2020-06-12T02:37:52.407 INFO:tasks.rados.rados.0.smithi107.stdout:3758: done (1 left) 2020-06-12T02:37:52.407 INFO:tasks.rados.rados.0.smithi107.stdout:3759: delete oid 5 current snap is 0 2020-06-12T02:37:52.408 INFO:tasks.rados.rados.0.smithi107.stdout:3759: done (1 left) 2020-06-12T02:37:52.408 INFO:tasks.rados.rados.0.smithi107.stdout:3760: delete oid 7 current snap is 0 2020-06-12T02:37:52.408 INFO:tasks.rados.rados.0.smithi107.stdout:3757: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:52.408 INFO:tasks.rados.rados.0.smithi107.stdout:3757: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:52.408 INFO:tasks.rados.rados.0.smithi107.stdout:3757: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:52.408 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 699 (ObjNum 3756 snap 0 seq_num 3756) dirty exists 2020-06-12T02:37:52.409 INFO:tasks.rados.rados.0.smithi107.stdout:3757: left oid 6 (ObjNum 3756 snap 0 seq_num 3756) 2020-06-12T02:37:52.409 INFO:tasks.rados.rados.0.smithi107.stdout:3757: done (1 left) 2020-06-12T02:37:52.409 INFO:tasks.rados.rados.0.smithi107.stdout:3760: done (0 left) 2020-06-12T02:37:52.409 INFO:tasks.rados.rados.0.smithi107.stdout:3761: delete oid 8 current snap is 0 2020-06-12T02:37:52.409 INFO:tasks.rados.rados.0.smithi107.stdout:3761: done (0 left) 2020-06-12T02:37:52.410 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:37:52.410 INFO:tasks.rados.rados.0.smithi107.stdout:3762: seq_num 3761 ranges {0=598016} 2020-06-12T02:37:52.411 INFO:tasks.rados.rados.0.smithi107.stdout:3762: writing smithi10729407-7 from 0 to 598016 tid 1 2020-06-12T02:37:52.411 INFO:tasks.rados.rados.0.smithi107.stdout:3763: delete oid 5 current snap is 0 2020-06-12T02:37:52.411 INFO:tasks.rados.rados.0.smithi107.stdout:3763: done (1 left) 2020-06-12T02:37:52.411 INFO:tasks.rados.rados.0.smithi107.stdout:3764: delete oid 13 current snap is 0 2020-06-12T02:37:52.413 INFO:tasks.rados.rados.0.smithi107.stdout:3762: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:52.414 INFO:tasks.rados.rados.0.smithi107.stdout:3762: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:52.415 INFO:tasks.rados.rados.0.smithi107.stdout:3762: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:52.415 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 710 (ObjNum 3761 snap 0 seq_num 3761) dirty exists 2020-06-12T02:37:52.415 INFO:tasks.rados.rados.0.smithi107.stdout:3762: left oid 7 (ObjNum 3761 snap 0 seq_num 3761) 2020-06-12T02:37:52.415 INFO:tasks.rados.rados.0.smithi107.stdout:3762: done (1 left) 2020-06-12T02:37:52.415 INFO:tasks.rados.rados.0.smithi107.stdout:3764: done (0 left) 2020-06-12T02:37:52.416 INFO:tasks.rados.rados.0.smithi107.stdout:3765: delete oid 19 current snap is 0 2020-06-12T02:37:52.417 INFO:tasks.rados.rados.0.smithi107.stdout:3765: done (0 left) 2020-06-12T02:37:52.418 INFO:tasks.rados.rados.0.smithi107.stdout:3766: delete oid 3 current snap is 0 2020-06-12T02:37:52.421 INFO:tasks.rados.rados.0.smithi107.stdout:3766: done (0 left) 2020-06-12T02:37:52.421 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:37:52.421 INFO:tasks.rados.rados.0.smithi107.stdout:3767: seq_num 3766 ranges {507904=679936} 2020-06-12T02:37:52.427 INFO:tasks.rados.rados.0.smithi107.stdout:3767: writing smithi10729407-6 from 507904 to 1187840 tid 1 2020-06-12T02:37:52.428 INFO:tasks.rados.rados.0.smithi107.stdout:3768: delete oid 18 current snap is 0 2020-06-12T02:37:52.429 INFO:tasks.rados.rados.0.smithi107.stdout:3768: done (1 left) 2020-06-12T02:37:52.429 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:37:52.436 INFO:tasks.rados.rados.0.smithi107.stdout:3769: seq_num 3768 ranges {0=712704} 2020-06-12T02:37:52.437 INFO:tasks.rados.rados.0.smithi107.stdout:3769: writing smithi10729407-16 from 0 to 712704 tid 1 2020-06-12T02:37:52.438 INFO:tasks.rados.rados.0.smithi107.stdout:3770: delete oid 15 current snap is 0 2020-06-12T02:37:52.438 INFO:tasks.rados.rados.0.smithi107.stdout:3767: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:52.438 INFO:tasks.rados.rados.0.smithi107.stdout:3767: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:52.438 INFO:tasks.rados.rados.0.smithi107.stdout:3770: done (2 left) 2020-06-12T02:37:52.438 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:37:52.439 INFO:tasks.rados.rados.0.smithi107.stdout:3771: seq_num 3770 ranges {0=745472} 2020-06-12T02:37:52.441 INFO:tasks.rados.rados.0.smithi107.stdout:3771: writing smithi10729407-5 from 0 to 745472 tid 1 2020-06-12T02:37:52.441 INFO:tasks.rados.rados.0.smithi107.stdout:3772: delete oid 9 current snap is 0 2020-06-12T02:37:52.442 INFO:tasks.rados.rados.0.smithi107.stdout:3767: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:52.442 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 701 (ObjNum 3766 snap 0 seq_num 3766) dirty exists 2020-06-12T02:37:52.442 INFO:tasks.rados.rados.0.smithi107.stdout:3767: left oid 6 (ObjNum 3766 snap 0 seq_num 3766) 2020-06-12T02:37:52.442 INFO:tasks.rados.rados.0.smithi107.stdout:3769: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:52.443 INFO:tasks.rados.rados.0.smithi107.stdout:3769: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:52.444 INFO:tasks.rados.rados.0.smithi107.stdout:3769: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:52.445 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 674 (ObjNum 3768 snap 0 seq_num 3768) dirty exists 2020-06-12T02:37:52.445 INFO:tasks.rados.rados.0.smithi107.stdout:3769: left oid 16 (ObjNum 3768 snap 0 seq_num 3768) 2020-06-12T02:37:52.447 INFO:tasks.rados.rados.0.smithi107.stdout:3767: done (3 left) 2020-06-12T02:37:52.447 INFO:tasks.rados.rados.0.smithi107.stdout:3769: done (2 left) 2020-06-12T02:37:52.448 INFO:tasks.rados.rados.0.smithi107.stdout:3772: done (1 left) 2020-06-12T02:37:52.448 INFO:tasks.rados.rados.0.smithi107.stdout:3773: delete oid 1 current snap is 0 2020-06-12T02:37:52.451 INFO:tasks.rados.rados.0.smithi107.stdout:3771: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:52.451 INFO:tasks.rados.rados.0.smithi107.stdout:3773: done (1 left) 2020-06-12T02:37:52.451 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:52.452 INFO:tasks.rados.rados.0.smithi107.stdout:3774: seq_num 3773 ranges {0=794624} 2020-06-12T02:37:52.452 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:52.449+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:37:52.456 INFO:tasks.rados.rados.0.smithi107.stdout:3774: writing smithi10729407-2 from 0 to 794624 tid 1 2020-06-12T02:37:52.456 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:37:52.456 INFO:tasks.rados.rados.0.smithi107.stdout:3775: seq_num 3774 ranges {0=409600,409600=401408} 2020-06-12T02:37:52.458 INFO:tasks.rados.rados.0.smithi107.stdout:3775: writing smithi10729407-9 from 0 to 409600 tid 1 2020-06-12T02:37:52.461 INFO:tasks.rados.rados.0.smithi107.stdout:3775: writing smithi10729407-9 from 409600 to 811008 tid 2 2020-06-12T02:37:52.462 INFO:tasks.rados.rados.0.smithi107.stdout:3776: delete oid 11 current snap is 0 2020-06-12T02:37:52.462 INFO:tasks.rados.rados.0.smithi107.stdout:3771: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:52.462 INFO:tasks.rados.rados.0.smithi107.stdout:3771: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:52.462 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 346 (ObjNum 3770 snap 0 seq_num 3770) dirty exists 2020-06-12T02:37:52.462 INFO:tasks.rados.rados.0.smithi107.stdout:3771: left oid 5 (ObjNum 3770 snap 0 seq_num 3770) 2020-06-12T02:37:52.463 INFO:tasks.rados.rados.0.smithi107.stdout:3774: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:52.463 INFO:tasks.rados.rados.0.smithi107.stdout:3775: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:52.464 INFO:tasks.rados.rados.0.smithi107.stdout:3771: done (3 left) 2020-06-12T02:37:52.464 INFO:tasks.rados.rados.0.smithi107.stdout:3776: done (2 left) 2020-06-12T02:37:52.464 INFO:tasks.rados.rados.0.smithi107.stdout:3777: delete oid 3 current snap is 0 2020-06-12T02:37:52.468 INFO:tasks.rados.rados.0.smithi107.stdout:3774: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:52.468 INFO:tasks.rados.rados.0.smithi107.stdout:3777: done (2 left) 2020-06-12T02:37:52.469 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:37:52.469 INFO:tasks.rados.rados.0.smithi107.stdout:3778: seq_num 3777 ranges {0=466944,466944=401408} 2020-06-12T02:37:52.471 INFO:tasks.rados.rados.0.smithi107.stdout:3778: writing smithi10729407-4 from 0 to 466944 tid 1 2020-06-12T02:37:52.474 INFO:tasks.rados.rados.0.smithi107.stdout:3778: writing smithi10729407-4 from 466944 to 868352 tid 2 2020-06-12T02:37:52.474 INFO:tasks.rados.rados.0.smithi107.stdout:3779: delete oid 11 current snap is 0 2020-06-12T02:37:52.474 INFO:tasks.rados.rados.0.smithi107.stdout:3775: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:52.475 INFO:tasks.rados.rados.0.smithi107.stdout:3775: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:52.475 INFO:tasks.rados.rados.0.smithi107.stdout:3774: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:52.475 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 391 (ObjNum 3773 snap 0 seq_num 3773) dirty exists 2020-06-12T02:37:52.475 INFO:tasks.rados.rados.0.smithi107.stdout:3774: left oid 2 (ObjNum 3773 snap 0 seq_num 3773) 2020-06-12T02:37:52.475 INFO:tasks.rados.rados.0.smithi107.stdout:3775: finishing write tid 5 to smithi10729407-9 2020-06-12T02:37:52.476 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 669 (ObjNum 3774 snap 0 seq_num 3774) dirty exists 2020-06-12T02:37:52.476 INFO:tasks.rados.rados.0.smithi107.stdout:3775: left oid 9 (ObjNum 3774 snap 0 seq_num 3774) 2020-06-12T02:37:52.476 INFO:tasks.rados.rados.0.smithi107.stdout:3774: done (3 left) 2020-06-12T02:37:52.476 INFO:tasks.rados.rados.0.smithi107.stdout:3775: done (2 left) 2020-06-12T02:37:52.477 INFO:tasks.rados.rados.0.smithi107.stdout:3779: done (1 left) 2020-06-12T02:37:52.477 INFO:tasks.rados.rados.0.smithi107.stdout:3780: delete oid 5 current snap is 0 2020-06-12T02:37:52.478 INFO:tasks.rados.rados.0.smithi107.stdout:3780: done (1 left) 2020-06-12T02:37:52.478 INFO:tasks.rados.rados.0.smithi107.stdout:3781: delete oid 5 current snap is 0 2020-06-12T02:37:52.478 INFO:tasks.rados.rados.0.smithi107.stdout:3778: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:52.479 INFO:tasks.rados.rados.0.smithi107.stdout:3778: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:52.479 INFO:tasks.rados.rados.0.smithi107.stdout:3778: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:52.480 INFO:tasks.rados.rados.0.smithi107.stdout:3778: finishing write tid 5 to smithi10729407-4 2020-06-12T02:37:52.480 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 696 (ObjNum 3777 snap 0 seq_num 3777) dirty exists 2020-06-12T02:37:52.480 INFO:tasks.rados.rados.0.smithi107.stdout:3778: left oid 4 (ObjNum 3777 snap 0 seq_num 3777) 2020-06-12T02:37:52.481 INFO:tasks.rados.rados.0.smithi107.stdout:3778: done (1 left) 2020-06-12T02:37:52.482 INFO:tasks.rados.rados.0.smithi107.stdout:3781: done (0 left) 2020-06-12T02:37:52.482 INFO:tasks.rados.rados.0.smithi107.stdout:3782: delete oid 20 current snap is 0 2020-06-12T02:37:52.483 INFO:tasks.rados.rados.0.smithi107.stdout:3782: done (0 left) 2020-06-12T02:37:52.483 INFO:tasks.rados.rados.0.smithi107.stdout:3783: delete oid 14 current snap is 0 2020-06-12T02:37:52.485 INFO:tasks.rados.rados.0.smithi107.stdout:3783: done (0 left) 2020-06-12T02:37:52.485 INFO:tasks.rados.rados.0.smithi107.stdout:3784: delete oid 2 current snap is 0 2020-06-12T02:37:52.486 INFO:tasks.rados.rados.0.smithi107.stdout:3784: done (0 left) 2020-06-12T02:37:52.487 INFO:tasks.rados.rados.0.smithi107.stdout:3785: delete oid 7 current snap is 0 2020-06-12T02:37:52.492 INFO:tasks.rados.rados.0.smithi107.stdout:3785: done (0 left) 2020-06-12T02:37:52.492 INFO:tasks.rados.rados.0.smithi107.stdout:3786: delete oid 14 current snap is 0 2020-06-12T02:37:52.495 INFO:tasks.rados.rados.0.smithi107.stdout:3786: done (0 left) 2020-06-12T02:37:52.495 INFO:tasks.rados.rados.0.smithi107.stdout:3787: delete oid 12 current snap is 0 2020-06-12T02:37:52.496 INFO:tasks.rados.rados.0.smithi107.stdout:3787: done (0 left) 2020-06-12T02:37:52.497 INFO:tasks.rados.rados.0.smithi107.stdout:3788: delete oid 6 current snap is 0 2020-06-12T02:37:52.498 INFO:tasks.rados.rados.0.smithi107.stdout:3788: done (0 left) 2020-06-12T02:37:52.499 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:37:52.499 INFO:tasks.rados.rados.0.smithi107.stdout:3789: seq_num 3788 ranges {0=647168,647168=401408} 2020-06-12T02:37:52.502 INFO:tasks.rados.rados.0.smithi107.stdout:3789: writing smithi10729407-12 from 0 to 647168 tid 1 2020-06-12T02:37:52.505 INFO:tasks.rados.rados.0.smithi107.stdout:3789: writing smithi10729407-12 from 647168 to 1048576 tid 2 2020-06-12T02:37:52.505 INFO:tasks.rados.rados.0.smithi107.stdout:3790: delete oid 7 current snap is 0 2020-06-12T02:37:52.508 INFO:tasks.rados.rados.0.smithi107.stdout:3790: done (1 left) 2020-06-12T02:37:52.508 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:37:52.508 INFO:tasks.rados.rados.0.smithi107.stdout:3791: seq_num 3790 ranges {0=679936,679936=401408} 2020-06-12T02:37:52.512 INFO:tasks.rados.rados.0.smithi107.stdout:3791: writing smithi10729407-15 from 0 to 679936 tid 1 2020-06-12T02:37:52.514 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:52.511+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:52.514 INFO:tasks.rados.rados.0.smithi107.stdout:3791: writing smithi10729407-15 from 679936 to 1081344 tid 2 2020-06-12T02:37:52.515 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:37:52.515 INFO:tasks.rados.rados.0.smithi107.stdout:3792: seq_num 3791 ranges {0=696320,696320=401408} 2020-06-12T02:37:52.518 INFO:tasks.rados.rados.0.smithi107.stdout:3792: writing smithi10729407-19 from 0 to 696320 tid 1 2020-06-12T02:37:52.521 INFO:tasks.rados.rados.0.smithi107.stdout:3792: writing smithi10729407-19 from 696320 to 1097728 tid 2 2020-06-12T02:37:52.521 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:37:52.521 INFO:tasks.rados.rados.0.smithi107.stdout:3793: seq_num 3792 ranges {0=712704,712704=401408} 2020-06-12T02:37:52.526 INFO:tasks.rados.rados.0.smithi107.stdout:3793: writing smithi10729407-18 from 0 to 712704 tid 1 2020-06-12T02:37:52.528 INFO:tasks.rados.rados.0.smithi107.stdout:3793: writing smithi10729407-18 from 712704 to 1114112 tid 2 2020-06-12T02:37:52.529 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:52.529 INFO:tasks.rados.rados.0.smithi107.stdout:3794: seq_num 3793 ranges {0=729088,729088=401408} 2020-06-12T02:37:52.530 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:37:52.530 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:37:52.531 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.531 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8055 3.9s0 3:93ea0cad:::smithi10729407-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e33)", 2020-06-12T02:37:52.531 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.520386+0000", 2020-06-12T02:37:52.531 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.0077088599999999997, 2020-06-12T02:37:52.531 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0077298599999999999, 2020-06-12T02:37:52.532 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:52.532 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "delayed", 2020-06-12T02:37:52.532 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:52.532 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:52.532 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:52.533 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8055 2020-06-12T02:37:52.533 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.533 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:52.533 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.533 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:52.534 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.520386+0000", 2020-06-12T02:37:52.534 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:52.534 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.535 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.535 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:52.535 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.520386+0000", 2020-06-12T02:37:52.535 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2271000000000002e-05 2020-06-12T02:37:52.535 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.536 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.536 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:52.536 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.520428+0000", 2020-06-12T02:37:52.536 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0299999999999999e-06 2020-06-12T02:37:52.536 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.537 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.538 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:52.540 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.520434+0000", 2020-06-12T02:37:52.540 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6714999999999998e-05 2020-06-12T02:37:52.540 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.540 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.541 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:52.541 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.520451+0000", 2020-06-12T02:37:52.541 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3919999999999996e-06 2020-06-12T02:37:52.541 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.542 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.542 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:52.542 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.520455+0000", 2020-06-12T02:37:52.542 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.006513558 2020-06-12T02:37:52.542 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.543 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.543 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:52.543 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.526968+0000", 2020-06-12T02:37:52.544 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000113571 2020-06-12T02:37:52.544 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.544 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.544 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:52.545 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.527082+0000", 2020-06-12T02:37:52.545 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0066965369999999998 2020-06-12T02:37:52.545 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:52.545 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:52.545 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:52.546 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.546 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.546 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 33/21 ECSubWriteReply(tid=3765, last_complete=33'633, committed=1, applied=1))", 2020-06-12T02:37:52.546 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.527717+0000", 2020-06-12T02:37:52.546 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.00037689499999999998, 2020-06-12T02:37:52.547 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00044297599999999998, 2020-06-12T02:37:52.547 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:52.547 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:37:52.547 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:52.547 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.548 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:52.548 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.527717+0000", 2020-06-12T02:37:52.548 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:52.548 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.548 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.549 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:52.549 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.527717+0000", 2020-06-12T02:37:52.549 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6141999999999999e-05 2020-06-12T02:37:52.549 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.550 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.550 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:52.550 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.527774+0000", 2020-06-12T02:37:52.550 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1799e-05 2020-06-12T02:37:52.551 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.552 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.552 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:52.552 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.527785+0000", 2020-06-12T02:37:52.552 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4827999999999999e-05 2020-06-12T02:37:52.553 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.553 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.553 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:52.553 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.527800+0000", 2020-06-12T02:37:52.553 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8110000000000002e-06 2020-06-12T02:37:52.554 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.554 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.554 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:52.554 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.527805+0000", 2020-06-12T02:37:52.554 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4790999999999999e-05 2020-06-12T02:37:52.554 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.555 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.555 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:52.555 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.527840+0000", 2020-06-12T02:37:52.555 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000122371 2020-06-12T02:37:52.555 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:52.556 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:52.556 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:52.556 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:52.556 INFO:teuthology.orchestra.run.smithi188.stdout: ], 2020-06-12T02:37:52.556 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 2 2020-06-12T02:37:52.556 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:37:52.559 INFO:tasks.rados.rados.0.smithi107.stdout:3794: writing smithi10729407-2 from 0 to 729088 tid 1 2020-06-12T02:37:52.560 INFO:tasks.rados.rados.0.smithi107.stdout:3794: writing smithi10729407-2 from 729088 to 1130496 tid 2 2020-06-12T02:37:52.560 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:37:52.560 INFO:tasks.rados.rados.0.smithi107.stdout:3795: seq_num 3794 ranges {868352=745472,1613824=401408} 2020-06-12T02:37:52.560 INFO:tasks.rados.rados.0.smithi107.stdout:3795: writing smithi10729407-4 from 868352 to 1613824 tid 1 2020-06-12T02:37:52.561 INFO:tasks.rados.rados.0.smithi107.stdout:3795: writing smithi10729407-4 from 1613824 to 2015232 tid 2 2020-06-12T02:37:52.561 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:37:52.561 INFO:tasks.rados.rados.0.smithi107.stdout:3796: seq_num 3795 ranges {0=761856,761856=401408} 2020-06-12T02:37:52.561 INFO:tasks.rados.rados.0.smithi107.stdout:3796: writing smithi10729407-5 from 0 to 761856 tid 1 2020-06-12T02:37:52.561 INFO:tasks.rados.rados.0.smithi107.stdout:3796: writing smithi10729407-5 from 761856 to 1163264 tid 2 2020-06-12T02:37:52.561 INFO:tasks.rados.rados.0.smithi107.stdout:3797: delete oid 3 current snap is 0 2020-06-12T02:37:52.562 INFO:tasks.rados.rados.0.smithi107.stdout:3789: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:52.562 INFO:tasks.rados.rados.0.smithi107.stdout:3789: finishing write tid 2 to smithi10729407-12 2020-06-12T02:37:52.562 INFO:tasks.rados.rados.0.smithi107.stdout:3789: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:52.562 INFO:tasks.rados.rados.0.smithi107.stdout:3789: finishing write tid 5 to smithi10729407-12 2020-06-12T02:37:52.562 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 630 (ObjNum 3788 snap 0 seq_num 3788) dirty exists 2020-06-12T02:37:52.563 INFO:tasks.rados.rados.0.smithi107.stdout:3789: left oid 12 (ObjNum 3788 snap 0 seq_num 3788) 2020-06-12T02:37:52.563 INFO:tasks.rados.rados.0.smithi107.stdout:3791: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:52.563 INFO:tasks.rados.rados.0.smithi107.stdout:3791: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:52.563 INFO:tasks.rados.rados.0.smithi107.stdout:3791: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:52.563 INFO:tasks.rados.rados.0.smithi107.stdout:3791: finishing write tid 5 to smithi10729407-15 2020-06-12T02:37:52.563 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 650 (ObjNum 3790 snap 0 seq_num 3790) dirty exists 2020-06-12T02:37:52.564 INFO:tasks.rados.rados.0.smithi107.stdout:3791: left oid 15 (ObjNum 3790 snap 0 seq_num 3790) 2020-06-12T02:37:52.564 INFO:tasks.rados.rados.0.smithi107.stdout:3792: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:52.564 INFO:tasks.rados.rados.0.smithi107.stdout:3792: finishing write tid 2 to smithi10729407-19 2020-06-12T02:37:52.564 INFO:tasks.rados.rados.0.smithi107.stdout:3792: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:52.564 INFO:tasks.rados.rados.0.smithi107.stdout:3792: finishing write tid 5 to smithi10729407-19 2020-06-12T02:37:52.565 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 633 (ObjNum 3791 snap 0 seq_num 3791) dirty exists 2020-06-12T02:37:52.565 INFO:tasks.rados.rados.0.smithi107.stdout:3792: left oid 19 (ObjNum 3791 snap 0 seq_num 3791) 2020-06-12T02:37:52.565 INFO:tasks.rados.rados.0.smithi107.stdout:3793: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:52.565 INFO:tasks.rados.rados.0.smithi107.stdout:3793: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:52.565 INFO:tasks.rados.rados.0.smithi107.stdout:3793: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:52.565 INFO:tasks.rados.rados.0.smithi107.stdout:3794: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:52.566 INFO:tasks.rados.rados.0.smithi107.stdout:3793: finishing write tid 5 to smithi10729407-18 2020-06-12T02:37:52.566 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 672 (ObjNum 3792 snap 0 seq_num 3792) dirty exists 2020-06-12T02:37:52.566 INFO:tasks.rados.rados.0.smithi107.stdout:3793: left oid 18 (ObjNum 3792 snap 0 seq_num 3792) 2020-06-12T02:37:52.566 INFO:tasks.rados.rados.0.smithi107.stdout:3794: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:52.566 INFO:tasks.rados.rados.0.smithi107.stdout:3794: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:52.567 INFO:tasks.rados.rados.0.smithi107.stdout:3794: finishing write tid 5 to smithi10729407-2 2020-06-12T02:37:52.567 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 395 (ObjNum 3793 snap 0 seq_num 3793) dirty exists 2020-06-12T02:37:52.567 INFO:tasks.rados.rados.0.smithi107.stdout:3794: left oid 2 (ObjNum 3793 snap 0 seq_num 3793) 2020-06-12T02:37:52.567 INFO:tasks.rados.rados.0.smithi107.stdout:3795: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:52.567 INFO:tasks.rados.rados.0.smithi107.stdout:3795: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:52.567 INFO:tasks.rados.rados.0.smithi107.stdout:3795: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:52.568 INFO:tasks.rados.rados.0.smithi107.stdout:3795: finishing write tid 5 to smithi10729407-4 2020-06-12T02:37:52.568 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 699 (ObjNum 3794 snap 0 seq_num 3794) dirty exists 2020-06-12T02:37:52.568 INFO:tasks.rados.rados.0.smithi107.stdout:3795: left oid 4 (ObjNum 3794 snap 0 seq_num 3794) 2020-06-12T02:37:52.568 INFO:tasks.rados.rados.0.smithi107.stdout:3789: done (7 left) 2020-06-12T02:37:52.569 INFO:tasks.rados.rados.0.smithi107.stdout:3791: done (6 left) 2020-06-12T02:37:52.569 INFO:tasks.rados.rados.0.smithi107.stdout:3792: done (5 left) 2020-06-12T02:37:52.569 INFO:tasks.rados.rados.0.smithi107.stdout:3793: done (4 left) 2020-06-12T02:37:52.569 INFO:tasks.rados.rados.0.smithi107.stdout:3794: done (3 left) 2020-06-12T02:37:52.569 INFO:tasks.rados.rados.0.smithi107.stdout:3795: done (2 left) 2020-06-12T02:37:52.570 INFO:tasks.rados.rados.0.smithi107.stdout:3797: done (1 left) 2020-06-12T02:37:52.570 INFO:tasks.rados.rados.0.smithi107.stdout:3798: delete oid 6 current snap is 0 2020-06-12T02:37:52.570 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-06-12T02:37:52.572 INFO:tasks.rados.rados.0.smithi107.stdout:3798: done (1 left) 2020-06-12T02:37:52.572 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:37:52.572 INFO:tasks.rados.rados.0.smithi107.stdout:3799: seq_num 3798 ranges {0=417792,417792=688128,1105920=114688} 2020-06-12T02:37:52.573 INFO:tasks.rados.rados.0.smithi107.stdout:3799: writing smithi10729407-1 from 0 to 417792 tid 1 2020-06-12T02:37:52.573 INFO:tasks.rados.rados.0.smithi107.stdout:3799: writing smithi10729407-1 from 417792 to 1105920 tid 2 2020-06-12T02:37:52.573 INFO:tasks.rados.rados.0.smithi107.stdout:3799: writing smithi10729407-1 from 1105920 to 1220608 tid 3 2020-06-12T02:37:52.573 INFO:tasks.rados.rados.0.smithi107.stdout:3800: delete oid 7 current snap is 0 2020-06-12T02:37:52.573 INFO:tasks.rados.rados.0.smithi107.stdout:3796: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:52.574 INFO:tasks.rados.rados.0.smithi107.stdout:3796: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:52.574 INFO:tasks.rados.rados.0.smithi107.stdout:3796: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:52.574 INFO:tasks.rados.rados.0.smithi107.stdout:3796: finishing write tid 5 to smithi10729407-5 2020-06-12T02:37:52.574 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 351 (ObjNum 3795 snap 0 seq_num 3795) dirty exists 2020-06-12T02:37:52.574 INFO:tasks.rados.rados.0.smithi107.stdout:3796: left oid 5 (ObjNum 3795 snap 0 seq_num 3795) 2020-06-12T02:37:52.574 INFO:tasks.rados.rados.0.smithi107.stdout:3799: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:52.575 INFO:tasks.rados.rados.0.smithi107.stdout:3796: done (2 left) 2020-06-12T02:37:52.575 INFO:tasks.rados.rados.0.smithi107.stdout:3800: done (1 left) 2020-06-12T02:37:52.575 INFO:tasks.rados.rados.0.smithi107.stdout:3801: delete oid 12 current snap is 0 2020-06-12T02:37:52.575 INFO:tasks.rados.rados.0.smithi107.stdout:3801: done (1 left) 2020-06-12T02:37:52.575 INFO:tasks.rados.rados.0.smithi107.stdout:3802: delete oid 9 current snap is 0 2020-06-12T02:37:52.576 INFO:tasks.rados.rados.0.smithi107.stdout:3799: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:52.576 INFO:tasks.rados.rados.0.smithi107.stdout:3799: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:52.576 INFO:tasks.rados.rados.0.smithi107.stdout:3802: done (1 left) 2020-06-12T02:37:52.577 INFO:tasks.rados.rados.0.smithi107.stdout:3803: delete oid 14 current snap is 0 2020-06-12T02:37:52.577 INFO:tasks.rados.rados.0.smithi107.stdout:3799: finishing write tid 5 to smithi10729407-1 2020-06-12T02:37:52.577 INFO:tasks.rados.rados.0.smithi107.stdout:3803: done (1 left) 2020-06-12T02:37:52.579 INFO:tasks.rados.rados.0.smithi107.stdout:3804: delete oid 20 current snap is 0 2020-06-12T02:37:52.579 INFO:tasks.rados.rados.0.smithi107.stdout:3799: finishing write tid 6 to smithi10729407-1 2020-06-12T02:37:52.579 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 708 (ObjNum 3798 snap 0 seq_num 3798) dirty exists 2020-06-12T02:37:52.580 INFO:tasks.rados.rados.0.smithi107.stdout:3799: left oid 1 (ObjNum 3798 snap 0 seq_num 3798) 2020-06-12T02:37:52.580 INFO:tasks.rados.rados.0.smithi107.stdout:3799: done (1 left) 2020-06-12T02:37:52.580 INFO:tasks.rados.rados.0.smithi107.stdout:3804: done (0 left) 2020-06-12T02:37:52.581 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:37:52.581 INFO:tasks.rados.rados.0.smithi107.stdout:3805: seq_num 3804 ranges {1163264=516096,1679360=442368,2121728=360448} 2020-06-12T02:37:52.587 INFO:tasks.rados.rados.0.smithi107.stdout:3805: writing smithi10729407-5 from 1163264 to 1679360 tid 1 2020-06-12T02:37:52.590 INFO:tasks.rados.rados.0.smithi107.stdout:3805: writing smithi10729407-5 from 1679360 to 2121728 tid 2 2020-06-12T02:37:52.592 INFO:tasks.rados.rados.0.smithi107.stdout:3805: writing smithi10729407-5 from 2121728 to 2482176 tid 3 2020-06-12T02:37:52.592 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:37:52.592 INFO:tasks.rados.rados.0.smithi107.stdout:3806: seq_num 3805 ranges {712704=532480,1245184=729088,1974272=73728} 2020-06-12T02:37:52.599 INFO:tasks.rados.rados.0.smithi107.stdout:3806: writing smithi10729407-16 from 712704 to 1245184 tid 1 2020-06-12T02:37:52.603 INFO:tasks.rados.rados.0.smithi107.stdout:3806: writing smithi10729407-16 from 1245184 to 1974272 tid 2 2020-06-12T02:37:52.604 INFO:tasks.rados.rados.0.smithi107.stdout:3806: writing smithi10729407-16 from 1974272 to 2048000 tid 3 2020-06-12T02:37:52.604 INFO:tasks.rados.rados.0.smithi107.stdout:3807: delete oid 13 current snap is 0 2020-06-12T02:37:52.604 INFO:tasks.rados.rados.0.smithi107.stdout:3805: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:52.604 INFO:tasks.rados.rados.0.smithi107.stdout:3805: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:52.605 INFO:tasks.rados.rados.0.smithi107.stdout:3805: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:52.605 INFO:tasks.rados.rados.0.smithi107.stdout:3805: finishing write tid 5 to smithi10729407-5 2020-06-12T02:37:52.605 INFO:tasks.rados.rados.0.smithi107.stdout:3805: finishing write tid 6 to smithi10729407-5 2020-06-12T02:37:52.605 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 355 (ObjNum 3804 snap 0 seq_num 3804) dirty exists 2020-06-12T02:37:52.605 INFO:tasks.rados.rados.0.smithi107.stdout:3805: left oid 5 (ObjNum 3804 snap 0 seq_num 3804) 2020-06-12T02:37:52.606 INFO:tasks.rados.rados.0.smithi107.stdout:3806: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:52.606 INFO:tasks.rados.rados.0.smithi107.stdout:3805: done (2 left) 2020-06-12T02:37:52.606 INFO:tasks.rados.rados.0.smithi107.stdout:3807: done (1 left) 2020-06-12T02:37:52.607 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:37:52.607 INFO:tasks.rados.rados.0.smithi107.stdout:3808: seq_num 3807 ranges {1220608=565248,1785856=499712,2285568=303104} 2020-06-12T02:37:52.614 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:37:52.612+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:37:52.616 INFO:tasks.rados.rados.0.smithi107.stdout:3808: writing smithi10729407-1 from 1220608 to 1785856 tid 1 2020-06-12T02:37:52.618 INFO:tasks.rados.rados.0.smithi107.stdout:3808: writing smithi10729407-1 from 1785856 to 2285568 tid 2 2020-06-12T02:37:52.620 INFO:tasks.rados.rados.0.smithi107.stdout:3808: writing smithi10729407-1 from 2285568 to 2588672 tid 3 2020-06-12T02:37:52.620 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:37:52.621 INFO:tasks.rados.rados.0.smithi107.stdout:3809: seq_num 3808 ranges {0=581632,581632=786432,1368064=16384} 2020-06-12T02:37:52.624 INFO:tasks.rados.rados.0.smithi107.stdout:3809: writing smithi10729407-7 from 0 to 581632 tid 1 2020-06-12T02:37:52.629 INFO:tasks.rados.rados.0.smithi107.stdout:3809: writing smithi10729407-7 from 581632 to 1368064 tid 2 2020-06-12T02:37:52.629 INFO:tasks.rados.rados.0.smithi107.stdout:3809: writing smithi10729407-7 from 1368064 to 1384448 tid 3 2020-06-12T02:37:52.630 INFO:tasks.rados.rados.0.smithi107.stdout:3810: delete oid 4 current snap is 0 2020-06-12T02:37:52.631 INFO:tasks.rados.rados.0.smithi107.stdout:3806: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:52.631 INFO:tasks.rados.rados.0.smithi107.stdout:3806: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:52.631 INFO:tasks.rados.rados.0.smithi107.stdout:3806: finishing write tid 5 to smithi10729407-16 2020-06-12T02:37:52.631 INFO:tasks.rados.rados.0.smithi107.stdout:3806: finishing write tid 6 to smithi10729407-16 2020-06-12T02:37:52.632 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 680 (ObjNum 3805 snap 0 seq_num 3805) dirty exists 2020-06-12T02:37:52.632 INFO:tasks.rados.rados.0.smithi107.stdout:3806: left oid 16 (ObjNum 3805 snap 0 seq_num 3805) 2020-06-12T02:37:52.632 INFO:tasks.rados.rados.0.smithi107.stdout:3808: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:52.632 INFO:tasks.rados.rados.0.smithi107.stdout:3808: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:52.632 INFO:tasks.rados.rados.0.smithi107.stdout:3808: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:52.632 INFO:tasks.rados.rados.0.smithi107.stdout:3808: finishing write tid 5 to smithi10729407-1 2020-06-12T02:37:52.633 INFO:tasks.rados.rados.0.smithi107.stdout:3808: finishing write tid 6 to smithi10729407-1 2020-06-12T02:37:52.633 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 712 (ObjNum 3807 snap 0 seq_num 3807) dirty exists 2020-06-12T02:37:52.633 INFO:tasks.rados.rados.0.smithi107.stdout:3808: left oid 1 (ObjNum 3807 snap 0 seq_num 3807) 2020-06-12T02:37:52.634 INFO:tasks.rados.rados.0.smithi107.stdout:3809: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:52.634 INFO:tasks.rados.rados.0.smithi107.stdout:3806: done (3 left) 2020-06-12T02:37:52.640 INFO:tasks.rados.rados.0.smithi107.stdout:3808: done (2 left) 2020-06-12T02:37:52.640 INFO:tasks.rados.rados.0.smithi107.stdout:3810: done (1 left) 2020-06-12T02:37:52.640 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:37:52.640 INFO:tasks.rados.rados.0.smithi107.stdout:3811: seq_num 3810 ranges {1097728=614400,1712128=598016,2310144=204800} 2020-06-12T02:37:52.643 INFO:tasks.rados.rados.0.smithi107.stdout:3811: writing smithi10729407-19 from 1097728 to 1712128 tid 1 2020-06-12T02:37:52.647 INFO:tasks.rados.rados.0.smithi107.stdout:3811: writing smithi10729407-19 from 1712128 to 2310144 tid 2 2020-06-12T02:37:52.649 INFO:tasks.rados.rados.0.smithi107.stdout:3811: writing smithi10729407-19 from 2310144 to 2514944 tid 3 2020-06-12T02:37:52.649 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:37:52.649 INFO:tasks.rados.rados.0.smithi107.stdout:3812: seq_num 3811 ranges {0=630784,630784=483328,1114112=319488} 2020-06-12T02:37:52.652 INFO:tasks.rados.rados.0.smithi107.stdout:3812: writing smithi10729407-6 from 0 to 630784 tid 1 2020-06-12T02:37:52.655 INFO:tasks.rados.rados.0.smithi107.stdout:3812: writing smithi10729407-6 from 630784 to 1114112 tid 2 2020-06-12T02:37:52.657 INFO:tasks.rados.rados.0.smithi107.stdout:3812: writing smithi10729407-6 from 1114112 to 1433600 tid 3 2020-06-12T02:37:52.658 INFO:tasks.rados.rados.0.smithi107.stdout:3813: delete oid 3 current snap is 0 2020-06-12T02:37:52.658 INFO:tasks.rados.rados.0.smithi107.stdout:3809: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:52.658 INFO:tasks.rados.rados.0.smithi107.stdout:3809: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:52.658 INFO:tasks.rados.rados.0.smithi107.stdout:3809: finishing write tid 5 to smithi10729407-7 2020-06-12T02:37:52.658 INFO:tasks.rados.rados.0.smithi107.stdout:3809: finishing write tid 6 to smithi10729407-7 2020-06-12T02:37:52.659 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 719 (ObjNum 3808 snap 0 seq_num 3808) dirty exists 2020-06-12T02:37:52.659 INFO:tasks.rados.rados.0.smithi107.stdout:3809: left oid 7 (ObjNum 3808 snap 0 seq_num 3808) 2020-06-12T02:37:52.659 INFO:tasks.rados.rados.0.smithi107.stdout:3811: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:52.659 INFO:tasks.rados.rados.0.smithi107.stdout:3811: finishing write tid 2 to smithi10729407-19 2020-06-12T02:37:52.659 INFO:tasks.rados.rados.0.smithi107.stdout:3811: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:52.660 INFO:tasks.rados.rados.0.smithi107.stdout:3811: finishing write tid 5 to smithi10729407-19 2020-06-12T02:37:52.660 INFO:tasks.rados.rados.0.smithi107.stdout:3812: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:52.660 INFO:tasks.rados.rados.0.smithi107.stdout:3811: finishing write tid 6 to smithi10729407-19 2020-06-12T02:37:52.661 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 638 (ObjNum 3810 snap 0 seq_num 3810) dirty exists 2020-06-12T02:37:52.661 INFO:tasks.rados.rados.0.smithi107.stdout:3811: left oid 19 (ObjNum 3810 snap 0 seq_num 3810) 2020-06-12T02:37:52.662 INFO:tasks.rados.rados.0.smithi107.stdout:3809: done (3 left) 2020-06-12T02:37:52.663 INFO:tasks.rados.rados.0.smithi107.stdout:3811: done (2 left) 2020-06-12T02:37:52.663 INFO:tasks.rados.rados.0.smithi107.stdout:3813: done (1 left) 2020-06-12T02:37:52.663 INFO:tasks.rados.rados.0.smithi107.stdout:3814: delete oid 18 current snap is 0 2020-06-12T02:37:52.663 INFO:tasks.rados.rados.0.smithi107.stdout:3812: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:52.665 INFO:tasks.rados.rados.0.smithi107.stdout:3812: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:52.666 INFO:tasks.rados.rados.0.smithi107.stdout:3812: finishing write tid 5 to smithi10729407-6 2020-06-12T02:37:52.666 INFO:tasks.rados.rados.0.smithi107.stdout:3814: done (1 left) 2020-06-12T02:37:52.666 INFO:tasks.rados.rados.0.smithi107.stdout:3815: delete oid 2 current snap is 0 2020-06-12T02:37:52.668 INFO:tasks.rados.rados.0.smithi107.stdout:3815: done (1 left) 2020-06-12T02:37:52.668 INFO:tasks.rados.rados.0.smithi107.stdout:3816: delete oid 20 current snap is 0 2020-06-12T02:37:52.668 INFO:tasks.rados.rados.0.smithi107.stdout:3812: finishing write tid 6 to smithi10729407-6 2020-06-12T02:37:52.669 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 716 (ObjNum 3811 snap 0 seq_num 3811) dirty exists 2020-06-12T02:37:52.669 INFO:tasks.rados.rados.0.smithi107.stdout:3812: left oid 6 (ObjNum 3811 snap 0 seq_num 3811) 2020-06-12T02:37:52.670 INFO:tasks.rados.rados.0.smithi107.stdout:3812: done (1 left) 2020-06-12T02:37:52.670 INFO:tasks.rados.rados.0.smithi107.stdout:3816: done (0 left) 2020-06-12T02:37:52.670 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:37:52.670 INFO:tasks.rados.rados.0.smithi107.stdout:3817: seq_num 3816 ranges {2482176=720896,3203072=704512,3907584=98304} 2020-06-12T02:37:52.687 INFO:tasks.rados.rados.0.smithi107.stdout:3817: writing smithi10729407-5 from 2482176 to 3203072 tid 1 2020-06-12T02:37:52.691 INFO:tasks.rados.rados.0.smithi107.stdout:3817: writing smithi10729407-5 from 3203072 to 3907584 tid 2 2020-06-12T02:37:52.691 INFO:tasks.rados.rados.0.smithi107.stdout:3817: writing smithi10729407-5 from 3907584 to 4005888 tid 3 2020-06-12T02:37:52.692 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:37:52.692 INFO:tasks.rados.rados.0.smithi107.stdout:3818: seq_num 3817 ranges {0=737280,737280=638976,1376256=163840} 2020-06-12T02:37:52.695 INFO:tasks.rados.rados.0.smithi107.stdout:3818: writing smithi10729407-8 from 0 to 737280 tid 1 2020-06-12T02:37:52.699 INFO:tasks.rados.rados.0.smithi107.stdout:3818: writing smithi10729407-8 from 737280 to 1376256 tid 2 2020-06-12T02:37:52.701 INFO:tasks.rados.rados.0.smithi107.stdout:3818: writing smithi10729407-8 from 1376256 to 1540096 tid 3 2020-06-12T02:37:52.701 INFO:tasks.rados.rados.0.smithi107.stdout:3819: delete oid 13 current snap is 0 2020-06-12T02:37:52.701 INFO:tasks.rados.rados.0.smithi107.stdout:3817: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:52.702 INFO:tasks.rados.rados.0.smithi107.stdout:3817: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:52.703 INFO:tasks.rados.rados.0.smithi107.stdout:3818: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:52.704 INFO:tasks.rados.rados.0.smithi107.stdout:3817: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:52.708 INFO:tasks.rados.rados.0.smithi107.stdout:3819: done (2 left) 2020-06-12T02:37:52.708 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:37:52.708 INFO:tasks.rados.rados.0.smithi107.stdout:3820: seq_num 3819 ranges {0=770048,770048=409600,1179648=393216} 2020-06-12T02:37:52.712 INFO:tasks.rados.rados.0.smithi107.stdout:3820: writing smithi10729407-11 from 0 to 770048 tid 1 2020-06-12T02:37:52.715 INFO:tasks.rados.rados.0.smithi107.stdout:3820: writing smithi10729407-11 from 770048 to 1179648 tid 2 2020-06-12T02:37:52.715 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:52.713+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:52.718 INFO:tasks.rados.rados.0.smithi107.stdout:3820: writing smithi10729407-11 from 1179648 to 1572864 tid 3 2020-06-12T02:37:52.718 INFO:tasks.rados.rados.0.smithi107.stdout:3821: delete oid 12 current snap is 0 2020-06-12T02:37:52.718 INFO:tasks.rados.rados.0.smithi107.stdout:3817: finishing write tid 5 to smithi10729407-5 2020-06-12T02:37:52.719 INFO:tasks.rados.rados.0.smithi107.stdout:3818: finishing write tid 2 to smithi10729407-8 2020-06-12T02:37:52.719 INFO:tasks.rados.rados.0.smithi107.stdout:3817: finishing write tid 6 to smithi10729407-5 2020-06-12T02:37:52.719 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 359 (ObjNum 3816 snap 0 seq_num 3816) dirty exists 2020-06-12T02:37:52.719 INFO:tasks.rados.rados.0.smithi107.stdout:3817: left oid 5 (ObjNum 3816 snap 0 seq_num 3816) 2020-06-12T02:37:52.719 INFO:tasks.rados.rados.0.smithi107.stdout:3818: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:52.720 INFO:tasks.rados.rados.0.smithi107.stdout:3818: finishing write tid 5 to smithi10729407-8 2020-06-12T02:37:52.720 INFO:tasks.rados.rados.0.smithi107.stdout:3818: finishing write tid 6 to smithi10729407-8 2020-06-12T02:37:52.720 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 654 (ObjNum 3817 snap 0 seq_num 3817) dirty exists 2020-06-12T02:37:52.720 INFO:tasks.rados.rados.0.smithi107.stdout:3818: left oid 8 (ObjNum 3817 snap 0 seq_num 3817) 2020-06-12T02:37:52.721 INFO:tasks.rados.rados.0.smithi107.stdout:3817: done (3 left) 2020-06-12T02:37:52.721 INFO:tasks.rados.rados.0.smithi107.stdout:3818: done (2 left) 2020-06-12T02:37:52.721 INFO:tasks.rados.rados.0.smithi107.stdout:3821: done (1 left) 2020-06-12T02:37:52.721 INFO:tasks.rados.rados.0.smithi107.stdout:3822: delete oid 15 current snap is 0 2020-06-12T02:37:52.721 INFO:tasks.rados.rados.0.smithi107.stdout:3820: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:52.722 INFO:tasks.rados.rados.0.smithi107.stdout:3822: done (1 left) 2020-06-12T02:37:52.723 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:37:52.723 INFO:tasks.rados.rados.0.smithi107.stdout:3823: seq_num 3822 ranges {0=417792,417792=458752,876544=737280,1613824=8192} 2020-06-12T02:37:52.724 INFO:tasks.rados.rados.0.smithi107.stdout:3823: writing smithi10729407-13 from 0 to 417792 tid 1 2020-06-12T02:37:52.728 INFO:tasks.rados.rados.0.smithi107.stdout:3823: writing smithi10729407-13 from 417792 to 876544 tid 2 2020-06-12T02:37:52.733 INFO:tasks.rados.rados.0.smithi107.stdout:3823: writing smithi10729407-13 from 876544 to 1613824 tid 3 2020-06-12T02:37:52.733 INFO:tasks.rados.rados.0.smithi107.stdout:3823: writing smithi10729407-13 from 1613824 to 1622016 tid 4 2020-06-12T02:37:52.733 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:37:52.733 INFO:tasks.rados.rados.0.smithi107.stdout:3824: seq_num 3823 ranges {0=434176,434176=745472,1179648=458752} 2020-06-12T02:37:52.735 INFO:tasks.rados.rados.0.smithi107.stdout:3824: writing smithi10729407-3 from 0 to 434176 tid 1 2020-06-12T02:37:52.740 INFO:tasks.rados.rados.0.smithi107.stdout:3824: writing smithi10729407-3 from 434176 to 1179648 tid 2 2020-06-12T02:37:52.743 INFO:tasks.rados.rados.0.smithi107.stdout:3824: writing smithi10729407-3 from 1179648 to 1638400 tid 3 2020-06-12T02:37:52.743 INFO:tasks.rados.rados.0.smithi107.stdout:3825: delete oid 10 current snap is 0 2020-06-12T02:37:52.743 INFO:tasks.rados.rados.0.smithi107.stdout:3820: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:52.744 INFO:tasks.rados.rados.0.smithi107.stdout:3820: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:52.744 INFO:tasks.rados.rados.0.smithi107.stdout:3820: finishing write tid 5 to smithi10729407-11 2020-06-12T02:37:52.744 INFO:tasks.rados.rados.0.smithi107.stdout:3823: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:52.744 INFO:tasks.rados.rados.0.smithi107.stdout:3820: finishing write tid 6 to smithi10729407-11 2020-06-12T02:37:52.744 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 667 (ObjNum 3819 snap 0 seq_num 3819) dirty exists 2020-06-12T02:37:52.745 INFO:tasks.rados.rados.0.smithi107.stdout:3820: left oid 11 (ObjNum 3819 snap 0 seq_num 3819) 2020-06-12T02:37:52.745 INFO:tasks.rados.rados.0.smithi107.stdout:3823: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:52.745 INFO:tasks.rados.rados.0.smithi107.stdout:3823: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:52.745 INFO:tasks.rados.rados.0.smithi107.stdout:3823: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:52.746 INFO:tasks.rados.rados.0.smithi107.stdout:3823: finishing write tid 6 to smithi10729407-13 2020-06-12T02:37:52.746 INFO:tasks.rados.rados.0.smithi107.stdout:3824: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:52.746 INFO:tasks.rados.rados.0.smithi107.stdout:3823: finishing write tid 7 to smithi10729407-13 2020-06-12T02:37:52.747 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 725 (ObjNum 3822 snap 0 seq_num 3822) dirty exists 2020-06-12T02:37:52.747 INFO:tasks.rados.rados.0.smithi107.stdout:3823: left oid 13 (ObjNum 3822 snap 0 seq_num 3822) 2020-06-12T02:37:52.748 INFO:tasks.rados.rados.0.smithi107.stdout:3820: done (3 left) 2020-06-12T02:37:52.748 INFO:tasks.rados.rados.0.smithi107.stdout:3823: done (2 left) 2020-06-12T02:37:52.748 INFO:tasks.rados.rados.0.smithi107.stdout:3825: done (1 left) 2020-06-12T02:37:52.749 INFO:tasks.rados.rados.0.smithi107.stdout:3826: delete oid 19 current snap is 0 2020-06-12T02:37:52.749 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:37:52.750 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:37:52.750 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:37:52.750 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:37:52.750 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:37:52.751 INFO:tasks.rados.rados.0.smithi107.stdout:3824: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:52.751 INFO:tasks.rados.rados.0.smithi107.stdout:3824: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:52.753 INFO:tasks.rados.rados.0.smithi107.stdout:3826: done (1 left) 2020-06-12T02:37:52.753 INFO:tasks.rados.rados.0.smithi107.stdout:3827: delete oid 9 current snap is 0 2020-06-12T02:37:52.753 INFO:tasks.rados.rados.0.smithi107.stdout:3824: finishing write tid 5 to smithi10729407-3 2020-06-12T02:37:52.755 INFO:tasks.rados.rados.0.smithi107.stdout:3824: finishing write tid 6 to smithi10729407-3 2020-06-12T02:37:52.755 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 685 (ObjNum 3823 snap 0 seq_num 3823) dirty exists 2020-06-12T02:37:52.756 INFO:tasks.rados.rados.0.smithi107.stdout:3824: left oid 3 (ObjNum 3823 snap 0 seq_num 3823) 2020-06-12T02:37:52.757 INFO:tasks.rados.rados.0.smithi107.stdout:3824: done (1 left) 2020-06-12T02:37:52.757 INFO:tasks.rados.rados.0.smithi107.stdout:3827: done (0 left) 2020-06-12T02:37:52.757 INFO:tasks.rados.rados.0.smithi107.stdout:3828: delete oid 19 current snap is 0 2020-06-12T02:37:52.759 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-06-12T02:37:52.760 INFO:tasks.rados.rados.0.smithi107.stdout:3828: done (0 left) 2020-06-12T02:37:52.760 INFO:tasks.rados.rados.0.smithi107.stdout:3829: delete oid 5 current snap is 0 2020-06-12T02:37:52.761 INFO:tasks.rados.rados.0.smithi107.stdout:3829: done (0 left) 2020-06-12T02:37:52.761 INFO:tasks.rados.rados.0.smithi107.stdout:3830: delete oid 19 current snap is 0 2020-06-12T02:37:52.763 INFO:tasks.rados.rados.0.smithi107.stdout:3830: done (0 left) 2020-06-12T02:37:52.765 INFO:tasks.rados.rados.0.smithi107.stdout:3831: delete oid 17 current snap is 0 2020-06-12T02:37:52.765 INFO:tasks.rados.rados.0.smithi107.stdout:3831: done (0 left) 2020-06-12T02:37:52.766 INFO:tasks.rados.rados.0.smithi107.stdout:3832: delete oid 1 current snap is 0 2020-06-12T02:37:52.767 INFO:tasks.rados.rados.0.smithi107.stdout:3832: done (0 left) 2020-06-12T02:37:52.767 INFO:tasks.rados.rados.0.smithi107.stdout:3833: delete oid 10 current snap is 0 2020-06-12T02:37:52.769 INFO:tasks.rados.rados.0.smithi107.stdout:3833: done (0 left) 2020-06-12T02:37:52.770 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:37:52.770 INFO:tasks.rados.rados.0.smithi107.stdout:3834: seq_num 3833 ranges {1638400=598016,2236416=491520,2727936=524288,3252224=188416} 2020-06-12T02:37:52.786 INFO:tasks.rados.rados.0.smithi107.stdout:3834: writing smithi10729407-3 from 1638400 to 2236416 tid 1 2020-06-12T02:37:52.786 INFO:tasks.rados.rados.0.smithi107.stdout:3834: writing smithi10729407-3 from 2236416 to 2727936 tid 2 2020-06-12T02:37:52.788 INFO:tasks.rados.rados.0.smithi107.stdout:3834: writing smithi10729407-3 from 2727936 to 3252224 tid 3 2020-06-12T02:37:52.789 INFO:tasks.rados.rados.0.smithi107.stdout:3834: writing smithi10729407-3 from 3252224 to 3440640 tid 4 2020-06-12T02:37:52.790 INFO:tasks.rados.rados.0.smithi107.stdout:3835: delete oid 8 current snap is 0 2020-06-12T02:37:52.792 INFO:tasks.rados.rados.0.smithi107.stdout:3835: done (1 left) 2020-06-12T02:37:52.792 INFO:tasks.rados.rados.0.smithi107.stdout:3836: delete oid 9 current snap is 0 2020-06-12T02:37:52.793 INFO:tasks.rados.rados.0.smithi107.stdout:3834: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:52.794 INFO:tasks.rados.rados.0.smithi107.stdout:3836: done (1 left) 2020-06-12T02:37:52.795 INFO:tasks.rados.rados.0.smithi107.stdout:3837: delete oid 2 current snap is 0 2020-06-12T02:37:52.796 INFO:tasks.rados.rados.0.smithi107.stdout:3834: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:52.796 INFO:tasks.rados.rados.0.smithi107.stdout:3834: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:52.797 INFO:tasks.rados.rados.0.smithi107.stdout:3834: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:52.797 INFO:tasks.rados.rados.0.smithi107.stdout:3837: done (1 left) 2020-06-12T02:37:52.798 INFO:tasks.rados.rados.0.smithi107.stdout:3838: delete oid 11 current snap is 0 2020-06-12T02:37:52.800 INFO:tasks.rados.rados.0.smithi107.stdout:3834: finishing write tid 6 to smithi10729407-3 2020-06-12T02:37:52.801 INFO:tasks.rados.rados.0.smithi107.stdout:3834: finishing write tid 7 to smithi10729407-3 2020-06-12T02:37:52.801 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 690 (ObjNum 3833 snap 0 seq_num 3833) dirty exists 2020-06-12T02:37:52.801 INFO:tasks.rados.rados.0.smithi107.stdout:3834: left oid 3 (ObjNum 3833 snap 0 seq_num 3833) 2020-06-12T02:37:52.802 INFO:tasks.rados.rados.0.smithi107.stdout:3834: done (1 left) 2020-06-12T02:37:52.802 INFO:tasks.rados.rados.0.smithi107.stdout:3838: done (0 left) 2020-06-12T02:37:52.803 INFO:tasks.rados.rados.0.smithi107.stdout:3839: delete oid 9 current snap is 0 2020-06-12T02:37:52.807 INFO:tasks.rados.rados.0.smithi107.stdout:3839: done (0 left) 2020-06-12T02:37:52.808 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:37:52.808 INFO:tasks.rados.rados.0.smithi107.stdout:3840: seq_num 3839 ranges {0=704512,704512=598016,1302528=466944,1769472=139264} 2020-06-12T02:37:52.811 INFO:tasks.rados.rados.0.smithi107.stdout:3840: writing smithi10729407-19 from 0 to 704512 tid 1 2020-06-12T02:37:52.816 INFO:tasks.rados.rados.0.smithi107.stdout:3840: writing smithi10729407-19 from 704512 to 1302528 tid 2 2020-06-12T02:37:52.817 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:52.814+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:37:52.818 INFO:tasks.rados.rados.0.smithi107.stdout:3840: writing smithi10729407-19 from 1302528 to 1769472 tid 3 2020-06-12T02:37:52.819 INFO:tasks.rados.rados.0.smithi107.stdout:3840: writing smithi10729407-19 from 1769472 to 1908736 tid 4 2020-06-12T02:37:52.819 INFO:tasks.rados.rados.0.smithi107.stdout:3841: delete oid 6 current snap is 0 2020-06-12T02:37:52.820 INFO:tasks.rados.rados.0.smithi107.stdout:3840: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:52.821 INFO:tasks.rados.rados.0.smithi107.stdout:3841: done (1 left) 2020-06-12T02:37:52.821 INFO:tasks.rados.rados.0.smithi107.stdout:3842: delete oid 11 current snap is 0 2020-06-12T02:37:52.824 INFO:tasks.rados.rados.0.smithi107.stdout:3842: done (1 left) 2020-06-12T02:37:52.824 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:37:52.824 INFO:tasks.rados.rados.0.smithi107.stdout:3843: seq_num 3842 ranges {0=753664,753664=696320,1449984=507904} 2020-06-12T02:37:52.828 INFO:tasks.rados.rados.0.smithi107.stdout:3843: writing smithi10729407-12 from 0 to 753664 tid 1 2020-06-12T02:37:52.832 INFO:tasks.rados.rados.0.smithi107.stdout:3843: writing smithi10729407-12 from 753664 to 1449984 tid 2 2020-06-12T02:37:52.835 INFO:tasks.rados.rados.0.smithi107.stdout:3843: writing smithi10729407-12 from 1449984 to 1957888 tid 3 2020-06-12T02:37:52.835 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:37:52.835 INFO:tasks.rados.rados.0.smithi107.stdout:3844: seq_num 3843 ranges {3440640=770048,4210688=581632,4792320=450560,5242880=172032} 2020-06-12T02:37:52.857 INFO:tasks.rados.rados.0.smithi107.stdout:3844: writing smithi10729407-3 from 3440640 to 4210688 tid 1 2020-06-12T02:37:52.860 INFO:tasks.rados.rados.0.smithi107.stdout:3844: writing smithi10729407-3 from 4210688 to 4792320 tid 2 2020-06-12T02:37:52.864 INFO:tasks.rados.rados.0.smithi107.stdout:3844: writing smithi10729407-3 from 4792320 to 5242880 tid 3 2020-06-12T02:37:52.866 INFO:tasks.rados.rados.0.smithi107.stdout:3844: writing smithi10729407-3 from 5242880 to 5414912 tid 4 2020-06-12T02:37:52.866 INFO:tasks.rados.rados.0.smithi107.stdout:3845: delete oid 16 current snap is 0 2020-06-12T02:37:52.866 INFO:tasks.rados.rados.0.smithi107.stdout:3840: finishing write tid 2 to smithi10729407-19 2020-06-12T02:37:52.866 INFO:tasks.rados.rados.0.smithi107.stdout:3840: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:52.867 INFO:tasks.rados.rados.0.smithi107.stdout:3840: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:52.867 INFO:tasks.rados.rados.0.smithi107.stdout:3840: finishing write tid 6 to smithi10729407-19 2020-06-12T02:37:52.867 INFO:tasks.rados.rados.0.smithi107.stdout:3840: finishing write tid 7 to smithi10729407-19 2020-06-12T02:37:52.867 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 647 (ObjNum 3839 snap 0 seq_num 3839) dirty exists 2020-06-12T02:37:52.868 INFO:tasks.rados.rados.0.smithi107.stdout:3840: left oid 19 (ObjNum 3839 snap 0 seq_num 3839) 2020-06-12T02:37:52.868 INFO:tasks.rados.rados.0.smithi107.stdout:3843: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:52.868 INFO:tasks.rados.rados.0.smithi107.stdout:3843: finishing write tid 2 to smithi10729407-12 2020-06-12T02:37:52.868 INFO:tasks.rados.rados.0.smithi107.stdout:3843: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:52.868 INFO:tasks.rados.rados.0.smithi107.stdout:3843: finishing write tid 5 to smithi10729407-12 2020-06-12T02:37:52.869 INFO:tasks.rados.rados.0.smithi107.stdout:3843: finishing write tid 6 to smithi10729407-12 2020-06-12T02:37:52.869 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 651 (ObjNum 3842 snap 0 seq_num 3842) dirty exists 2020-06-12T02:37:52.869 INFO:tasks.rados.rados.0.smithi107.stdout:3843: left oid 12 (ObjNum 3842 snap 0 seq_num 3842) 2020-06-12T02:37:52.879 INFO:tasks.rados.rados.0.smithi107.stdout:3844: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:52.879 INFO:tasks.rados.rados.0.smithi107.stdout:3844: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:52.879 INFO:tasks.rados.rados.0.smithi107.stdout:3844: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:52.879 INFO:tasks.rados.rados.0.smithi107.stdout:3844: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:52.880 INFO:tasks.rados.rados.0.smithi107.stdout:3844: finishing write tid 6 to smithi10729407-3 2020-06-12T02:37:52.880 INFO:tasks.rados.rados.0.smithi107.stdout:3844: finishing write tid 7 to smithi10729407-3 2020-06-12T02:37:52.880 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 695 (ObjNum 3843 snap 0 seq_num 3843) dirty exists 2020-06-12T02:37:52.880 INFO:tasks.rados.rados.0.smithi107.stdout:3844: left oid 3 (ObjNum 3843 snap 0 seq_num 3843) 2020-06-12T02:37:52.880 INFO:tasks.rados.rados.0.smithi107.stdout:3840: done (3 left) 2020-06-12T02:37:52.880 INFO:tasks.rados.rados.0.smithi107.stdout:3843: done (2 left) 2020-06-12T02:37:52.881 INFO:tasks.rados.rados.0.smithi107.stdout:3844: done (1 left) 2020-06-12T02:37:52.881 INFO:tasks.rados.rados.0.smithi107.stdout:3845: done (0 left) 2020-06-12T02:37:52.881 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:37:52.882 INFO:tasks.rados.rados.0.smithi107.stdout:3846: seq_num 3845 ranges {1622016=802816,2424832=753664,3178496=450560} 2020-06-12T02:37:52.891 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:37:52.891 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:37:52.892 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:37:52.892 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:37:52.892 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.892 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:858 3.3s0 3:ce45c70a:::smithi10729407-7:head [append 0~565248] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:52.892 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.924643+0000", 2020-06-12T02:37:52.893 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.960113951, 2020-06-12T02:37:52.893 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015332551999999999, 2020-06-12T02:37:52.893 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:52.893 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:52.893 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:52.894 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:52.894 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:52.894 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 858 2020-06-12T02:37:52.894 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.895 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:52.895 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.895 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:52.895 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.924643+0000", 2020-06-12T02:37:52.895 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:52.896 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.896 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.896 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:52.896 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.924643+0000", 2020-06-12T02:37:52.897 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00067535299999999996 2020-06-12T02:37:52.897 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.897 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.900 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:52.900 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925318+0000", 2020-06-12T02:37:52.900 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9629999999999993e-06 2020-06-12T02:37:52.900 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.900 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.901 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:52.901 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925328+0000", 2020-06-12T02:37:52.901 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7357999999999999e-05 2020-06-12T02:37:52.901 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.901 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.902 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:52.902 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925345+0000", 2020-06-12T02:37:52.902 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7019999999999996e-06 2020-06-12T02:37:52.902 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.902 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.903 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:52.903 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925355+0000", 2020-06-12T02:37:52.903 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6289999999999998e-05 2020-06-12T02:37:52.903 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.903 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.904 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:52.904 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925411+0000", 2020-06-12T02:37:52.904 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8855000000000004e-05 2020-06-12T02:37:52.904 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.904 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.905 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:52.905 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925480+0000", 2020-06-12T02:37:52.905 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0074521149999999996 2020-06-12T02:37:52.905 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.905 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.905 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:52.906 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.932932+0000", 2020-06-12T02:37:52.906 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3647999999999995e-05 2020-06-12T02:37:52.906 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.906 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.906 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:52.907 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.933026+0000", 2020-06-12T02:37:52.907 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00078753199999999997 2020-06-12T02:37:52.907 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.907 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.907 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:52.908 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.933813+0000", 2020-06-12T02:37:52.908 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.006013582 2020-06-12T02:37:52.908 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.908 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.908 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:52.908 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.939827+0000", 2020-06-12T02:37:52.909 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0111999999999996e-05 2020-06-12T02:37:52.909 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.909 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.909 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:52.909 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.939897+0000", 2020-06-12T02:37:52.910 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8041999999999996e-05 2020-06-12T02:37:52.910 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.910 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.910 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:52.910 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.939975+0000", 2020-06-12T02:37:52.911 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015332551999999999 2020-06-12T02:37:52.911 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:52.911 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:52.911 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:52.911 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.911 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.912 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:860 3.3s0 3:ce45c70a:::smithi10729407-7:head [append 0~81920] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:52.912 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.928619+0000", 2020-06-12T02:37:52.912 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.956138065000001, 2020-06-12T02:37:52.912 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017431941999999999, 2020-06-12T02:37:52.912 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:52.913 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:52.913 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:52.913 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:52.913 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:52.913 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 860 2020-06-12T02:37:52.914 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.914 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:52.914 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.914 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:52.916 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928619+0000", 2020-06-12T02:37:52.916 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:52.916 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.916 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.916 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:52.917 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928619+0000", 2020-06-12T02:37:52.917 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000105809 2020-06-12T02:37:52.917 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.917 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.917 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:52.918 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928724+0000", 2020-06-12T02:37:52.918 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2339999999999994e-06 2020-06-12T02:37:52.918 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.918 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.918 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:52.919 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928733+0000", 2020-06-12T02:37:52.919 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0268000000000001e-05 2020-06-12T02:37:52.919 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.919 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.920 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:52.920 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928743+0000", 2020-06-12T02:37:52.920 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5190000000000003e-06 2020-06-12T02:37:52.920 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.920 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.921 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:52.921 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928747+0000", 2020-06-12T02:37:52.921 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0103999999999998e-05 2020-06-12T02:37:52.921 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.921 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.921 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:52.922 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928777+0000", 2020-06-12T02:37:52.922 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8249000000000002e-05 2020-06-12T02:37:52.922 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.922 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.922 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:52.923 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928866+0000", 2020-06-12T02:37:52.923 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011151629 2020-06-12T02:37:52.923 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.923 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.923 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:52.923 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.940017+0000", 2020-06-12T02:37:52.924 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8281e-05 2020-06-12T02:37:52.924 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.924 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.924 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:52.924 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.940076+0000", 2020-06-12T02:37:52.925 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000346736 2020-06-12T02:37:52.925 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.925 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.925 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:52.925 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.940422+0000", 2020-06-12T02:37:52.926 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024135049999999998 2020-06-12T02:37:52.926 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.926 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.926 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:52.927 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.942836+0000", 2020-06-12T02:37:52.927 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003170258 2020-06-12T02:37:52.927 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.927 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.927 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:52.927 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946006+0000", 2020-06-12T02:37:52.928 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4350000000000001e-05 2020-06-12T02:37:52.928 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.928 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.928 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:52.928 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946050+0000", 2020-06-12T02:37:52.929 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017431941999999999 2020-06-12T02:37:52.929 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:52.929 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:52.929 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:52.929 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.929 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.930 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:861 3.3s0 3:ce45c70a:::smithi10729407-7:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:52.930 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.928778+0000", 2020-06-12T02:37:52.930 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.955978452, 2020-06-12T02:37:52.930 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017522738, 2020-06-12T02:37:52.930 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:52.931 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:52.931 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:52.931 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:52.931 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:52.931 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 861 2020-06-12T02:37:52.932 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.932 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:52.932 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.932 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:52.932 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928778+0000", 2020-06-12T02:37:52.933 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:52.933 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.933 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.933 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:52.933 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928778+0000", 2020-06-12T02:37:52.933 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8186000000000002e-05 2020-06-12T02:37:52.934 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.934 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.934 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:52.934 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928826+0000", 2020-06-12T02:37:52.934 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6689999999999997e-06 2020-06-12T02:37:52.935 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.935 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.935 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:52.935 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928831+0000", 2020-06-12T02:37:52.935 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.686e-06 2020-06-12T02:37:52.936 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.936 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.936 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:52.937 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928838+0000", 2020-06-12T02:37:52.937 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7159999999999999e-06 2020-06-12T02:37:52.937 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.938 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.938 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:52.938 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928840+0000", 2020-06-12T02:37:52.938 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4212999999999997e-05 2020-06-12T02:37:52.939 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.940 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.940 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:52.940 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928895+0000", 2020-06-12T02:37:52.940 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5853999999999994e-05 2020-06-12T02:37:52.940 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.941 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.941 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:52.941 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928990+0000", 2020-06-12T02:37:52.942 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012023982000000001 2020-06-12T02:37:52.943 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.943 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.944 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:52.944 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941014+0000", 2020-06-12T02:37:52.944 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5104e-05 2020-06-12T02:37:52.944 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.945 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.945 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:52.945 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941070+0000", 2020-06-12T02:37:52.945 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00026444499999999998 2020-06-12T02:37:52.946 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.946 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.946 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:52.947 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941334+0000", 2020-06-12T02:37:52.947 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001704075 2020-06-12T02:37:52.947 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.947 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.948 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:52.948 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.943038+0000", 2020-06-12T02:37:52.948 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031891570000000002 2020-06-12T02:37:52.948 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.948 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.949 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:52.949 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946227+0000", 2020-06-12T02:37:52.949 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3651000000000003e-05 2020-06-12T02:37:52.949 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.949 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.949 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:52.950 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946301+0000", 2020-06-12T02:37:52.950 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017522738 2020-06-12T02:37:52.950 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:52.950 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:52.950 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:52.951 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.951 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.951 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:862 3.3s0 3:ce45c70a:::smithi10729407-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:52.951 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.928876+0000", 2020-06-12T02:37:52.951 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.955880956000001, 2020-06-12T02:37:52.952 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019206890000000001, 2020-06-12T02:37:52.952 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:52.952 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:52.952 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:52.953 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:52.953 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:52.953 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 862 2020-06-12T02:37:52.953 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.953 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:52.954 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.954 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:52.954 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928876+0000", 2020-06-12T02:37:52.954 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:52.955 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.955 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.955 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:52.955 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928876+0000", 2020-06-12T02:37:52.955 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9268999999999999e-05 2020-06-12T02:37:52.955 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.956 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.956 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:52.956 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928925+0000", 2020-06-12T02:37:52.956 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8120000000000002e-06 2020-06-12T02:37:52.957 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.957 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.957 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:52.957 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928932+0000", 2020-06-12T02:37:52.958 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3471e-05 2020-06-12T02:37:52.958 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.958 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.958 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:52.958 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928945+0000", 2020-06-12T02:37:52.958 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4279999999999998e-06 2020-06-12T02:37:52.959 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.959 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.959 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:52.959 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928950+0000", 2020-06-12T02:37:52.959 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6358000000000004e-05 2020-06-12T02:37:52.960 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.960 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.960 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:52.960 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.929016+0000", 2020-06-12T02:37:52.960 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9697999999999996e-05 2020-06-12T02:37:52.960 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.961 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.961 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:52.961 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.929086+0000", 2020-06-12T02:37:52.961 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012409205 2020-06-12T02:37:52.961 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.962 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.963 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:52.963 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941495+0000", 2020-06-12T02:37:52.963 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9660000000000006e-05 2020-06-12T02:37:52.963 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.964 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.964 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:52.964 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941585+0000", 2020-06-12T02:37:52.964 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050329099999999998 2020-06-12T02:37:52.964 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.964 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.965 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:52.965 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946617+0000", 2020-06-12T02:37:52.965 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8437999999999996e-05 2020-06-12T02:37:52.965 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.965 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.966 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:52.966 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946706+0000", 2020-06-12T02:37:52.966 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0013766410000000001 2020-06-12T02:37:52.966 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.966 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.967 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:52.967 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.948083+0000", 2020-06-12T02:37:52.967 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019206890000000001 2020-06-12T02:37:52.967 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:52.967 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:52.967 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:52.968 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.968 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.968 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:879 3.fs1 3:fc7e0bf7:::smithi10729407-17:head [append 0~491520] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:52.968 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.000881+0000", 2020-06-12T02:37:52.968 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.883875775, 2020-06-12T02:37:52.969 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017891845, 2020-06-12T02:37:52.969 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:52.969 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:52.969 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:52.969 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:52.970 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:52.970 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 879 2020-06-12T02:37:52.970 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.970 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:52.970 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.970 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:52.971 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.000881+0000", 2020-06-12T02:37:52.971 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:52.972 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.972 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.972 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:52.972 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.000881+0000", 2020-06-12T02:37:52.972 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00044151599999999998 2020-06-12T02:37:52.973 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.973 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.973 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:52.973 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001322+0000", 2020-06-12T02:37:52.973 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0210000000000001e-06 2020-06-12T02:37:52.973 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.974 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.974 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:52.974 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001326+0000", 2020-06-12T02:37:52.974 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6659999999999997e-06 2020-06-12T02:37:52.974 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.975 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.975 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:52.975 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001335+0000", 2020-06-12T02:37:52.975 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4299999999999999e-06 2020-06-12T02:37:52.975 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.975 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.976 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:52.976 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001339+0000", 2020-06-12T02:37:52.976 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0028741510000000001 2020-06-12T02:37:52.976 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.976 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.977 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:52.977 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.004214+0000", 2020-06-12T02:37:52.977 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010480999999999999 2020-06-12T02:37:52.977 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.977 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.978 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:52.978 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.004318+0000", 2020-06-12T02:37:52.978 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021112359999999998 2020-06-12T02:37:52.978 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.978 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.979 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:52.979 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006430+0000", 2020-06-12T02:37:52.979 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2935000000000001e-05 2020-06-12T02:37:52.979 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.979 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.979 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:52.980 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006483+0000", 2020-06-12T02:37:52.980 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00056411099999999995 2020-06-12T02:37:52.980 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.980 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.980 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:52.980 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.007047+0000", 2020-06-12T02:37:52.981 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0076749369999999997 2020-06-12T02:37:52.982 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.982 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.982 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:52.982 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.014722+0000", 2020-06-12T02:37:52.982 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00395693 2020-06-12T02:37:52.983 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.983 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.983 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:52.983 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.018679+0000", 2020-06-12T02:37:52.983 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4102000000000001e-05 2020-06-12T02:37:52.984 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.984 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.984 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:52.984 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.018773+0000", 2020-06-12T02:37:52.984 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017891845 2020-06-12T02:37:52.985 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:52.985 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:52.985 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:52.985 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.985 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.985 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:883 3.fs1 3:fc7e0bf7:::smithi10729407-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:52.986 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.006322+0000", 2020-06-12T02:37:52.986 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.878434614, 2020-06-12T02:37:52.986 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01444995, 2020-06-12T02:37:52.986 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:52.986 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:52.987 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:52.987 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:52.987 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:52.987 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 883 2020-06-12T02:37:52.987 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.988 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:52.988 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.989 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:52.990 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006322+0000", 2020-06-12T02:37:52.990 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:52.990 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.991 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.991 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:52.991 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006322+0000", 2020-06-12T02:37:52.992 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1342999999999999e-05 2020-06-12T02:37:52.992 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.992 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.993 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:52.993 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006343+0000", 2020-06-12T02:37:52.993 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.021e-06 2020-06-12T02:37:52.994 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.994 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.994 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:52.995 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006346+0000", 2020-06-12T02:37:52.995 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2990000000000002e-06 2020-06-12T02:37:52.995 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.996 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.996 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:52.996 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006353+0000", 2020-06-12T02:37:52.996 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3240000000000001e-06 2020-06-12T02:37:52.997 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.997 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:52.998 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:52.998 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006355+0000", 2020-06-12T02:37:52.998 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0090315139999999992 2020-06-12T02:37:52.998 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:52.999 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.000 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.000 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.015386+0000", 2020-06-12T02:37:53.000 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5129000000000003e-05 2020-06-12T02:37:53.000 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.001 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.001 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:53.001 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.015482+0000", 2020-06-12T02:37:53.001 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0043414839999999996 2020-06-12T02:37:53.001 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.002 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.002 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.002 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.019823+0000", 2020-06-12T02:37:53.002 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6991999999999997e-05 2020-06-12T02:37:53.002 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.003 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.003 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:53.003 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.019910+0000", 2020-06-12T02:37:53.003 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00086184399999999996 2020-06-12T02:37:53.003 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.003 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.004 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:53.004 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.020772+0000", 2020-06-12T02:37:53.004 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01444995 2020-06-12T02:37:53.004 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.004 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:53.005 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.005 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.005 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.005 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:933 3.es1 3:78f01648:::smithi10729407-5:head [append 0~417792] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:53.005 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.156677+0000", 2020-06-12T02:37:53.006 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.728079792999999, 2020-06-12T02:37:53.006 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016833311, 2020-06-12T02:37:53.006 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:53.006 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:53.006 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:53.006 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:53.007 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:53.007 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 933 2020-06-12T02:37:53.007 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.007 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:53.007 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.008 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:53.008 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.156677+0000", 2020-06-12T02:37:53.008 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:53.008 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.008 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.008 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:53.009 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.156677+0000", 2020-06-12T02:37:53.009 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00062571700000000003 2020-06-12T02:37:53.009 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.009 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.009 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:53.011 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157303+0000", 2020-06-12T02:37:53.011 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0740000000000003e-06 2020-06-12T02:37:53.011 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.011 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.011 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:53.012 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157307+0000", 2020-06-12T02:37:53.012 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4419999999999998e-06 2020-06-12T02:37:53.012 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.012 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.012 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:53.013 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157315+0000", 2020-06-12T02:37:53.013 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5300000000000001e-06 2020-06-12T02:37:53.013 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.013 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.014 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:53.014 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157319+0000", 2020-06-12T02:37:53.014 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4995e-05 2020-06-12T02:37:53.014 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.014 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.014 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.015 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157354+0000", 2020-06-12T02:37:53.015 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5883000000000006e-05 2020-06-12T02:37:53.015 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.015 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.015 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:53.016 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157419+0000", 2020-06-12T02:37:53.016 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045316280000000002 2020-06-12T02:37:53.016 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.016 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.016 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.017 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.161951+0000", 2020-06-12T02:37:53.017 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3300999999999996e-05 2020-06-12T02:37:53.017 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.017 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.017 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:53.018 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.162014+0000", 2020-06-12T02:37:53.018 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00039050100000000002 2020-06-12T02:37:53.018 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.018 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.018 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:53.018 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.162405+0000", 2020-06-12T02:37:53.019 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050686250000000002 2020-06-12T02:37:53.019 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.019 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.019 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:53.019 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.167474+0000", 2020-06-12T02:37:53.020 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.005942653 2020-06-12T02:37:53.020 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.020 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.020 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:53.020 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.173416+0000", 2020-06-12T02:37:53.020 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3962000000000006e-05 2020-06-12T02:37:53.021 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.021 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.021 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:53.021 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.173510+0000", 2020-06-12T02:37:53.021 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016833311 2020-06-12T02:37:53.022 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.022 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:53.022 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.022 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.022 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.023 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:936 3.es1 3:78f01648:::smithi10729407-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:53.023 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.160357+0000", 2020-06-12T02:37:53.023 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.724400081999999, 2020-06-12T02:37:53.023 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015435590000000001, 2020-06-12T02:37:53.023 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:53.023 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:53.024 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:53.024 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:53.024 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:53.024 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 936 2020-06-12T02:37:53.024 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.025 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:53.025 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.025 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:53.025 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160357+0000", 2020-06-12T02:37:53.025 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:53.025 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.026 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.026 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:53.026 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160357+0000", 2020-06-12T02:37:53.026 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4435999999999999e-05 2020-06-12T02:37:53.026 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.027 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.027 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:53.027 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160411+0000", 2020-06-12T02:37:53.027 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2329999999999995e-06 2020-06-12T02:37:53.027 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.028 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.028 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:53.028 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160420+0000", 2020-06-12T02:37:53.028 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7003e-05 2020-06-12T02:37:53.028 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.028 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.029 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:53.029 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160437+0000", 2020-06-12T02:37:53.029 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8370000000000002e-06 2020-06-12T02:37:53.029 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.029 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.030 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:53.030 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160443+0000", 2020-06-12T02:37:53.030 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0070635489999999997 2020-06-12T02:37:53.030 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.030 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.031 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.031 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.167507+0000", 2020-06-12T02:37:53.031 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010589700000000001 2020-06-12T02:37:53.031 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.031 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.031 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:53.032 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.167612+0000", 2020-06-12T02:37:53.032 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0065032299999999996 2020-06-12T02:37:53.032 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.032 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.032 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.033 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.174116+0000", 2020-06-12T02:37:53.033 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3128999999999999e-05 2020-06-12T02:37:53.033 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.033 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.033 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:53.033 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.174169+0000", 2020-06-12T02:37:53.034 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0016232760000000001 2020-06-12T02:37:53.034 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.034 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.034 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:53.034 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.175792+0000", 2020-06-12T02:37:53.035 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015435590000000001 2020-06-12T02:37:53.035 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.035 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:53.035 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.035 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.035 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.036 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:950 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~778240] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:53.036 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.207643+0000", 2020-06-12T02:37:53.036 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.677113499000001, 2020-06-12T02:37:53.036 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014844661, 2020-06-12T02:37:53.036 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:53.037 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:53.037 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:53.037 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:53.037 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:53.037 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 950 2020-06-12T02:37:53.038 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.038 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:53.038 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.038 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:53.038 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.207643+0000", 2020-06-12T02:37:53.038 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:53.039 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.039 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.039 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:53.039 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.207643+0000", 2020-06-12T02:37:53.039 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00072843099999999998 2020-06-12T02:37:53.040 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.040 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.040 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:53.040 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208372+0000", 2020-06-12T02:37:53.040 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9120000000000001e-06 2020-06-12T02:37:53.041 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.041 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.041 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:53.041 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208378+0000", 2020-06-12T02:37:53.041 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9660000000000008e-06 2020-06-12T02:37:53.041 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.042 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.042 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:53.042 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208388+0000", 2020-06-12T02:37:53.042 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1279999999999999e-06 2020-06-12T02:37:53.042 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.043 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.043 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:53.043 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208394+0000", 2020-06-12T02:37:53.043 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0439999999999998e-05 2020-06-12T02:37:53.043 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.043 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.044 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.044 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208444+0000", 2020-06-12T02:37:53.044 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2611000000000001e-05 2020-06-12T02:37:53.044 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.044 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.045 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:53.045 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208507+0000", 2020-06-12T02:37:53.045 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001228301 2020-06-12T02:37:53.045 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.045 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.053 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.053 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.209735+0000", 2020-06-12T02:37:53.054 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8875000000000001e-05 2020-06-12T02:37:53.054 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.054 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.054 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:53.055 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.209794+0000", 2020-06-12T02:37:53.055 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053218899999999997 2020-06-12T02:37:53.055 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.056 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.056 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:53.056 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.210326+0000", 2020-06-12T02:37:53.057 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0078198439999999994 2020-06-12T02:37:53.057 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.057 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.058 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:53.058 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.218146+0000", 2020-06-12T02:37:53.059 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004300496 2020-06-12T02:37:53.059 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.059 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.060 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:53.060 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.222446+0000", 2020-06-12T02:37:53.060 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1467999999999997e-05 2020-06-12T02:37:53.061 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.061 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.062 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:53.062 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.222488+0000", 2020-06-12T02:37:53.062 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014844661 2020-06-12T02:37:53.063 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.063 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:53.063 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.064 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.064 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.064 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:954 3.3s0 3:c2ca77da:::smithi10729407-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:53.065 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.212923+0000", 2020-06-12T02:37:53.065 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.671833722999999, 2020-06-12T02:37:53.065 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015411546999999999, 2020-06-12T02:37:53.066 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:53.066 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:53.066 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:53.067 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:53.067 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:53.068 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 954 2020-06-12T02:37:53.068 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.068 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:53.068 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.069 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:53.069 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212923+0000", 2020-06-12T02:37:53.069 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:53.069 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.069 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.070 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:53.070 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212923+0000", 2020-06-12T02:37:53.070 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3598e-05 2020-06-12T02:37:53.070 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.070 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.071 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:53.071 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212946+0000", 2020-06-12T02:37:53.071 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.061e-06 2020-06-12T02:37:53.071 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.071 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.072 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:53.072 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212950+0000", 2020-06-12T02:37:53.072 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8700000000000003e-06 2020-06-12T02:37:53.075 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.075 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.076 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:53.076 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212956+0000", 2020-06-12T02:37:53.076 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0770000000000002e-06 2020-06-12T02:37:53.076 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.077 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.077 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:53.077 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212958+0000", 2020-06-12T02:37:53.077 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051235170000000002 2020-06-12T02:37:53.077 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.078 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.078 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.078 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.218082+0000", 2020-06-12T02:37:53.078 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9246999999999997e-05 2020-06-12T02:37:53.078 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.079 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.079 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:53.079 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.218141+0000", 2020-06-12T02:37:53.079 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0087981910000000003 2020-06-12T02:37:53.079 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.080 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.080 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.080 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.226939+0000", 2020-06-12T02:37:53.080 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4119999999999997e-05 2020-06-12T02:37:53.080 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.081 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.081 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:53.081 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.226994+0000", 2020-06-12T02:37:53.081 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001340866 2020-06-12T02:37:53.081 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.082 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.082 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:53.082 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.228334+0000", 2020-06-12T02:37:53.082 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015411546999999999 2020-06-12T02:37:53.082 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.083 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:53.083 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.083 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.083 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.083 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1097 3.es1 3:78f01648:::smithi10729407-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:53.084 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.576496+0000", 2020-06-12T02:37:53.084 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.308260240999999, 2020-06-12T02:37:53.084 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014992914, 2020-06-12T02:37:53.084 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:53.085 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:53.085 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:53.085 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:53.086 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:53.086 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1097 2020-06-12T02:37:53.087 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.087 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:53.087 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.087 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:53.087 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576496+0000", 2020-06-12T02:37:53.088 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:53.088 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.088 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.088 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:53.088 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576496+0000", 2020-06-12T02:37:53.089 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3221999999999999e-05 2020-06-12T02:37:53.089 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.089 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.089 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:53.089 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576520+0000", 2020-06-12T02:37:53.090 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.988e-06 2020-06-12T02:37:53.090 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.090 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.090 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:53.090 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576523+0000", 2020-06-12T02:37:53.091 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2940000000000002e-06 2020-06-12T02:37:53.091 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.091 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.091 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:53.091 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576529+0000", 2020-06-12T02:37:53.092 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6060000000000002e-06 2020-06-12T02:37:53.092 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.092 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.092 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:53.092 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576534+0000", 2020-06-12T02:37:53.092 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001254554 2020-06-12T02:37:53.093 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.093 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.093 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.093 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.577789+0000", 2020-06-12T02:37:53.093 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3177999999999998e-05 2020-06-12T02:37:53.094 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.094 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.094 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:53.094 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.577842+0000", 2020-06-12T02:37:53.094 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0063091220000000003 2020-06-12T02:37:53.095 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.095 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.095 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.096 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.584151+0000", 2020-06-12T02:37:53.096 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.783e-05 2020-06-12T02:37:53.096 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.097 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.097 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:53.097 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.584209+0000", 2020-06-12T02:37:53.097 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045863650000000002 2020-06-12T02:37:53.098 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.098 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.098 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.098 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.588796+0000", 2020-06-12T02:37:53.099 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7614000000000006e-05 2020-06-12T02:37:53.099 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.099 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.099 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:53.099 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.588873+0000", 2020-06-12T02:37:53.100 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0026161410000000002 2020-06-12T02:37:53.100 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.100 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.100 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:53.100 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.591489+0000", 2020-06-12T02:37:53.101 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014992914 2020-06-12T02:37:53.101 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.101 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:53.101 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.101 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.102 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.102 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1207 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~630784] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:53.102 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.859578+0000", 2020-06-12T02:37:53.102 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.025178704999998, 2020-06-12T02:37:53.102 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014710433, 2020-06-12T02:37:53.103 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:53.103 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:53.103 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:53.103 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:53.103 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:53.103 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1207 2020-06-12T02:37:53.104 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.104 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:53.104 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.104 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:53.104 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.859578+0000", 2020-06-12T02:37:53.105 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:53.105 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.105 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.106 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:53.106 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.859578+0000", 2020-06-12T02:37:53.106 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000775511 2020-06-12T02:37:53.107 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.107 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.107 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:53.107 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.860353+0000", 2020-06-12T02:37:53.107 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5325999999999999e-05 2020-06-12T02:37:53.108 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.108 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.108 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:53.108 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.860369+0000", 2020-06-12T02:37:53.108 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7430000000000001e-05 2020-06-12T02:37:53.109 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.109 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.109 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:53.109 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.860386+0000", 2020-06-12T02:37:53.109 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.426e-06 2020-06-12T02:37:53.110 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.110 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.110 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:53.110 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.860393+0000", 2020-06-12T02:37:53.110 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3585999999999999e-05 2020-06-12T02:37:53.111 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.111 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.111 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.111 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.860436+0000", 2020-06-12T02:37:53.111 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2559999999999996e-05 2020-06-12T02:37:53.111 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.112 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.112 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:53.112 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.860509+0000", 2020-06-12T02:37:53.112 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00063177700000000003 2020-06-12T02:37:53.112 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.113 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.113 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:53.113 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.861141+0000", 2020-06-12T02:37:53.113 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0026307570000000001 2020-06-12T02:37:53.113 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.114 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.114 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:53.114 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.863771+0000", 2020-06-12T02:37:53.114 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010445545000000001 2020-06-12T02:37:53.114 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.115 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.115 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:53.115 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.874217+0000", 2020-06-12T02:37:53.115 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1514999999999995e-05 2020-06-12T02:37:53.116 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.116 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.116 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:53.117 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.874288+0000", 2020-06-12T02:37:53.117 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014710433 2020-06-12T02:37:53.117 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.117 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:53.118 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.118 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.118 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.118 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:4813 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~450560] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-06-12T02:37:53.119 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:43.343096+0000", 2020-06-12T02:37:53.119 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.5416608979999999, 2020-06-12T02:37:53.119 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014820242000000001, 2020-06-12T02:37:53.120 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:53.120 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:53.120 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:53.120 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:53.121 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:53.121 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 4813 2020-06-12T02:37:53.121 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.121 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:53.121 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.122 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:53.122 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343096+0000", 2020-06-12T02:37:53.122 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:53.122 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.122 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.123 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:53.123 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343096+0000", 2020-06-12T02:37:53.123 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00068963 2020-06-12T02:37:53.123 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.124 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.124 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:53.124 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343785+0000", 2020-06-12T02:37:53.124 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1479999999999997e-06 2020-06-12T02:37:53.125 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.125 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.125 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:53.125 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343790+0000", 2020-06-12T02:37:53.126 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4060000000000005e-06 2020-06-12T02:37:53.126 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.126 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.127 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:53.127 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343799+0000", 2020-06-12T02:37:53.128 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0970000000000002e-06 2020-06-12T02:37:53.128 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.128 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.128 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:53.128 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343805+0000", 2020-06-12T02:37:53.129 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00023736499999999999 2020-06-12T02:37:53.129 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.129 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.129 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.129 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.344042+0000", 2020-06-12T02:37:53.130 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000113866 2020-06-12T02:37:53.130 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.130 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.130 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:53.130 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.344156+0000", 2020-06-12T02:37:53.131 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036114760000000002 2020-06-12T02:37:53.131 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.131 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.131 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.131 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.347768+0000", 2020-06-12T02:37:53.131 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1842999999999996e-05 2020-06-12T02:37:53.132 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.132 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.132 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:53.132 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.347850+0000", 2020-06-12T02:37:53.132 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00048835999999999997 2020-06-12T02:37:53.133 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.133 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.133 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:53.133 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.348338+0000", 2020-06-12T02:37:53.133 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0049373139999999999 2020-06-12T02:37:53.134 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.134 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.134 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:53.134 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.353275+0000", 2020-06-12T02:37:53.134 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045727119999999996 2020-06-12T02:37:53.134 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.135 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.135 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:53.135 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.357848+0000", 2020-06-12T02:37:53.135 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8025e-05 2020-06-12T02:37:53.135 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.136 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.136 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:53.136 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.357916+0000", 2020-06-12T02:37:53.136 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014820242000000001 2020-06-12T02:37:53.137 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.137 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:53.137 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.138 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.138 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.138 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:4921 3.9s0 3:93ea0cad:::smithi10729407-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-06-12T02:37:53.138 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:43.595696+0000", 2020-06-12T02:37:53.138 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.2890605320000006, 2020-06-12T02:37:53.138 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016503190000000001, 2020-06-12T02:37:53.139 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:53.139 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:53.139 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:53.139 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:53.139 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:53.140 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 4921 2020-06-12T02:37:53.140 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.140 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:53.140 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.140 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:53.141 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595696+0000", 2020-06-12T02:37:53.141 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:53.141 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.141 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.141 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:53.141 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595696+0000", 2020-06-12T02:37:53.142 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3694999999999999e-05 2020-06-12T02:37:53.142 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.142 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.142 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:53.142 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595720+0000", 2020-06-12T02:37:53.143 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1700000000000001e-06 2020-06-12T02:37:53.143 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.143 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.143 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:53.143 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595723+0000", 2020-06-12T02:37:53.143 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8849999999999999e-06 2020-06-12T02:37:53.144 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.144 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.144 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:53.144 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595731+0000", 2020-06-12T02:37:53.144 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5689999999999999e-06 2020-06-12T02:37:53.145 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.145 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.145 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:53.146 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595733+0000", 2020-06-12T02:37:53.146 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0030116880000000002 2020-06-12T02:37:53.146 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.146 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.146 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.147 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.598745+0000", 2020-06-12T02:37:53.147 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1353999999999996e-05 2020-06-12T02:37:53.147 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.148 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.148 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:53.148 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.598806+0000", 2020-06-12T02:37:53.148 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051955819999999998 2020-06-12T02:37:53.149 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.149 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.149 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.149 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.604002+0000", 2020-06-12T02:37:53.149 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9615000000000001e-05 2020-06-12T02:37:53.149 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.150 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.150 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:53.150 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.604102+0000", 2020-06-12T02:37:53.150 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0052050960000000002 2020-06-12T02:37:53.150 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.151 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.151 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.151 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.609307+0000", 2020-06-12T02:37:53.151 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5025000000000003e-05 2020-06-12T02:37:53.151 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.152 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.152 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:53.152 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.609362+0000", 2020-06-12T02:37:53.152 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002837511 2020-06-12T02:37:53.153 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.153 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.153 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:53.153 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.612199+0000", 2020-06-12T02:37:53.154 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016503190000000001 2020-06-12T02:37:53.154 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.154 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:53.154 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.155 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.155 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.155 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:5068 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~663552] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e30)", 2020-06-12T02:37:53.155 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:44.157497+0000", 2020-06-12T02:37:53.155 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 8.7272599060000005, 2020-06-12T02:37:53.156 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015164129, 2020-06-12T02:37:53.156 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:53.156 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:53.156 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:53.156 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:53.157 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:53.157 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 5068 2020-06-12T02:37:53.157 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.157 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:53.158 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.158 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:53.158 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.157497+0000", 2020-06-12T02:37:53.159 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:53.159 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.159 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.159 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:53.159 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.157497+0000", 2020-06-12T02:37:53.160 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00068511099999999997 2020-06-12T02:37:53.160 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.160 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.160 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:53.160 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.158182+0000", 2020-06-12T02:37:53.160 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4630000000000004e-06 2020-06-12T02:37:53.161 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.161 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.161 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:53.161 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.158189+0000", 2020-06-12T02:37:53.161 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4199000000000001e-05 2020-06-12T02:37:53.162 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.162 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.162 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:53.162 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.158203+0000", 2020-06-12T02:37:53.162 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1250000000000004e-06 2020-06-12T02:37:53.163 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.163 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.163 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:53.163 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.158211+0000", 2020-06-12T02:37:53.163 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6754000000000002e-05 2020-06-12T02:37:53.163 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.164 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.164 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.164 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.158237+0000", 2020-06-12T02:37:53.164 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2948999999999996e-05 2020-06-12T02:37:53.164 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.165 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.165 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:53.165 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.158310+0000", 2020-06-12T02:37:53.165 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0032401610000000001 2020-06-12T02:37:53.165 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.165 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.166 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.166 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.161550+0000", 2020-06-12T02:37:53.166 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5550000000000002e-05 2020-06-12T02:37:53.166 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.166 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.167 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:53.167 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.161606+0000", 2020-06-12T02:37:53.167 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00062448199999999999 2020-06-12T02:37:53.167 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.168 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.168 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:53.168 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.162230+0000", 2020-06-12T02:37:53.169 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0073257130000000002 2020-06-12T02:37:53.169 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.169 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.169 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:53.169 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.169556+0000", 2020-06-12T02:37:53.169 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0030226419999999999 2020-06-12T02:37:53.170 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.170 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.170 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:53.170 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.172579+0000", 2020-06-12T02:37:53.170 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1979999999999998e-05 2020-06-12T02:37:53.171 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.171 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.171 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:53.171 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.172661+0000", 2020-06-12T02:37:53.171 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015164129 2020-06-12T02:37:53.172 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.172 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:53.172 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.172 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.172 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.172 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:5071 3.3s0 3:c2ca77da:::smithi10729407-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-06-12T02:37:53.173 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:44.159788+0000", 2020-06-12T02:37:53.173 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 8.7249686840000003, 2020-06-12T02:37:53.173 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016812273999999999, 2020-06-12T02:37:53.173 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:53.173 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:53.174 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:53.174 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:53.174 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:53.174 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 5071 2020-06-12T02:37:53.174 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.174 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:53.175 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.175 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:53.175 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159788+0000", 2020-06-12T02:37:53.175 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:53.175 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.176 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.176 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:53.176 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159788+0000", 2020-06-12T02:37:53.176 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1421000000000001e-05 2020-06-12T02:37:53.176 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.176 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.177 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:53.177 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159839+0000", 2020-06-12T02:37:53.177 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9009999999999994e-06 2020-06-12T02:37:53.177 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.178 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.178 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:53.178 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159847+0000", 2020-06-12T02:37:53.178 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3149999999999999e-05 2020-06-12T02:37:53.179 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.179 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.179 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:53.179 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159860+0000", 2020-06-12T02:37:53.179 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1749999999999996e-06 2020-06-12T02:37:53.180 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.180 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.180 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:53.180 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159866+0000", 2020-06-12T02:37:53.180 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6035999999999999e-05 2020-06-12T02:37:53.181 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.181 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.181 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.181 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159892+0000", 2020-06-12T02:37:53.181 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2627999999999998e-05 2020-06-12T02:37:53.182 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.182 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.182 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:53.182 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159944+0000", 2020-06-12T02:37:53.182 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010270133000000001 2020-06-12T02:37:53.182 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.183 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.183 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.183 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.170214+0000", 2020-06-12T02:37:53.183 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5490999999999998e-05 2020-06-12T02:37:53.183 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.184 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.184 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:53.184 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.170310+0000", 2020-06-12T02:37:53.184 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003215828 2020-06-12T02:37:53.184 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.184 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.185 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.185 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.173526+0000", 2020-06-12T02:37:53.185 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0290000000000005e-05 2020-06-12T02:37:53.185 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.185 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.186 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:53.186 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.173616+0000", 2020-06-12T02:37:53.186 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002984221 2020-06-12T02:37:53.186 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.186 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.187 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:53.187 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.176600+0000", 2020-06-12T02:37:53.187 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016812273999999999 2020-06-12T02:37:53.187 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.188 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:53.188 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.188 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.188 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.189 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:6431 3.9s0 3:93ea0cad:::smithi10729407-19:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-06-12T02:37:53.189 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:47.960027+0000", 2020-06-12T02:37:53.189 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.9247298559999999, 2020-06-12T02:37:53.189 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014549095, 2020-06-12T02:37:53.190 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:53.190 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:53.190 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:53.190 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:53.190 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:53.191 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 6431 2020-06-12T02:37:53.191 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.191 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:53.191 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.191 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:53.191 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960027+0000", 2020-06-12T02:37:53.192 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:53.192 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.192 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.192 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:53.192 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960027+0000", 2020-06-12T02:37:53.193 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9313000000000001e-05 2020-06-12T02:37:53.193 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.193 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.193 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:53.193 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960066+0000", 2020-06-12T02:37:53.194 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0790000000000002e-06 2020-06-12T02:37:53.194 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.194 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.195 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:53.196 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960070+0000", 2020-06-12T02:37:53.196 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4510000000000003e-06 2020-06-12T02:37:53.197 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.197 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.197 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:53.198 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960078+0000", 2020-06-12T02:37:53.198 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4140000000000001e-06 2020-06-12T02:37:53.198 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.199 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.199 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:53.199 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960080+0000", 2020-06-12T02:37:53.200 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004337268 2020-06-12T02:37:53.200 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.200 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.200 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.201 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.964417+0000", 2020-06-12T02:37:53.201 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010161 2020-06-12T02:37:53.201 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.202 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.202 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:53.203 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.964519+0000", 2020-06-12T02:37:53.203 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024899399999999999 2020-06-12T02:37:53.203 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.204 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.204 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.204 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.967009+0000", 2020-06-12T02:37:53.205 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7747000000000001e-05 2020-06-12T02:37:53.205 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.205 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.206 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:53.206 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.967067+0000", 2020-06-12T02:37:53.206 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00025406499999999999 2020-06-12T02:37:53.207 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.207 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.207 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:53.208 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.967321+0000", 2020-06-12T02:37:53.208 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053713999999999997 2020-06-12T02:37:53.208 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.209 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.210 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:53.210 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.967858+0000", 2020-06-12T02:37:53.210 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0065322330000000001 2020-06-12T02:37:53.211 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.211 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.211 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:53.212 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.974390+0000", 2020-06-12T02:37:53.212 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000185835 2020-06-12T02:37:53.212 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.213 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.213 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:53.214 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.974576+0000", 2020-06-12T02:37:53.214 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014549095 2020-06-12T02:37:53.214 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.215 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:53.215 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.215 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.215 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.216 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:6432 3.9s0 3:93ea0cad:::smithi10729407-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e31)", 2020-06-12T02:37:53.216 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:47.960104+0000", 2020-06-12T02:37:53.217 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.9246525879999998, 2020-06-12T02:37:53.217 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016025815999999998, 2020-06-12T02:37:53.217 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:53.218 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:53.218 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:53.218 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:53.219 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:53.219 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 6432 2020-06-12T02:37:53.219 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.220 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:53.220 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.220 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:53.221 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960104+0000", 2020-06-12T02:37:53.221 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:53.221 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.222 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.222 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:53.222 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960104+0000", 2020-06-12T02:37:53.223 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2351999999999999e-05 2020-06-12T02:37:53.224 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.224 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.224 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:53.225 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960126+0000", 2020-06-12T02:37:53.225 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.061e-06 2020-06-12T02:37:53.225 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.225 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.226 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:53.226 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960129+0000", 2020-06-12T02:37:53.226 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4849999999999997e-06 2020-06-12T02:37:53.226 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.226 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.227 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:53.227 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960136+0000", 2020-06-12T02:37:53.227 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.176e-06 2020-06-12T02:37:53.227 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.227 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.227 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:53.228 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960138+0000", 2020-06-12T02:37:53.228 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0044054139999999999 2020-06-12T02:37:53.228 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.228 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.228 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.229 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.964543+0000", 2020-06-12T02:37:53.229 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8077000000000006e-05 2020-06-12T02:37:53.229 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.229 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.229 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:53.230 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.964612+0000", 2020-06-12T02:37:53.230 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0029921150000000001 2020-06-12T02:37:53.230 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.230 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.230 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.231 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.967604+0000", 2020-06-12T02:37:53.231 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2983000000000001e-05 2020-06-12T02:37:53.231 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.231 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.231 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:53.231 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.967697+0000", 2020-06-12T02:37:53.232 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0072496510000000002 2020-06-12T02:37:53.232 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.232 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.233 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.233 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.974946+0000", 2020-06-12T02:37:53.233 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8278e-05 2020-06-12T02:37:53.233 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.234 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.234 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:53.234 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.975005+0000", 2020-06-12T02:37:53.234 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001125224 2020-06-12T02:37:53.234 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.235 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.235 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:53.235 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.976130+0000", 2020-06-12T02:37:53.235 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016025815999999998 2020-06-12T02:37:53.235 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.235 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:53.236 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.236 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.238 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.238 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8123 3.es1 3:78f01648:::smithi10729407-5:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-06-12T02:37:53.238 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.690721+0000", 2020-06-12T02:37:53.238 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.19403548400000001, 2020-06-12T02:37:53.238 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016503126999999999, 2020-06-12T02:37:53.239 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:53.239 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:53.239 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:53.239 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:53.239 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:53.240 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8123 2020-06-12T02:37:53.240 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.240 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:53.240 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.240 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:53.241 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690721+0000", 2020-06-12T02:37:53.241 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:53.241 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.241 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.241 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:53.242 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690721+0000", 2020-06-12T02:37:53.242 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7526e-05 2020-06-12T02:37:53.242 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.242 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.242 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:53.243 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690759+0000", 2020-06-12T02:37:53.243 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1999999999999999e-06 2020-06-12T02:37:53.243 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.243 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.243 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:53.243 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690762+0000", 2020-06-12T02:37:53.244 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9229999999999999e-06 2020-06-12T02:37:53.244 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.244 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.244 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:53.244 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690768+0000", 2020-06-12T02:37:53.245 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4480000000000001e-06 2020-06-12T02:37:53.245 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.245 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.245 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:53.245 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690770+0000", 2020-06-12T02:37:53.246 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00079317799999999996 2020-06-12T02:37:53.246 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.246 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.246 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.246 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.691563+0000", 2020-06-12T02:37:53.246 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5299e-05 2020-06-12T02:37:53.247 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.247 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.247 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:53.247 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.691619+0000", 2020-06-12T02:37:53.247 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010070076000000001 2020-06-12T02:37:53.248 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.248 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.248 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.248 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.701689+0000", 2020-06-12T02:37:53.248 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000110234 2020-06-12T02:37:53.248 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.249 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.249 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:53.249 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.701799+0000", 2020-06-12T02:37:53.249 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00037311100000000003 2020-06-12T02:37:53.249 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.250 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.250 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:53.256 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.702172+0000", 2020-06-12T02:37:53.257 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0044460050000000003 2020-06-12T02:37:53.257 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.257 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.257 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:53.258 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.706618+0000", 2020-06-12T02:37:53.258 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00043550199999999999 2020-06-12T02:37:53.258 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.258 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.258 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:53.258 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.707054+0000", 2020-06-12T02:37:53.259 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017062499999999999 2020-06-12T02:37:53.259 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.259 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.259 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:53.259 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.707224+0000", 2020-06-12T02:37:53.260 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016503126999999999 2020-06-12T02:37:53.260 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.260 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:53.260 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.260 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.261 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.261 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8124 3.es1 3:78f01648:::smithi10729407-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e33)", 2020-06-12T02:37:53.261 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.690794+0000", 2020-06-12T02:37:53.261 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.19396229800000001, 2020-06-12T02:37:53.261 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019298497000000001, 2020-06-12T02:37:53.262 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:53.262 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:53.262 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:53.262 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:53.262 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:53.263 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8124 2020-06-12T02:37:53.263 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.263 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:53.263 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.263 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:53.264 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690794+0000", 2020-06-12T02:37:53.264 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:53.264 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.264 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.264 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:53.265 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690794+0000", 2020-06-12T02:37:53.266 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1869999999999999e-05 2020-06-12T02:37:53.266 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.266 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.266 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:53.267 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690816+0000", 2020-06-12T02:37:53.267 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.162e-06 2020-06-12T02:37:53.267 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.267 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.267 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:53.268 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690819+0000", 2020-06-12T02:37:53.268 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2280000000000003e-06 2020-06-12T02:37:53.268 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.268 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.268 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:53.269 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690826+0000", 2020-06-12T02:37:53.269 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.452e-06 2020-06-12T02:37:53.269 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.269 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.269 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:53.270 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690828+0000", 2020-06-12T02:37:53.270 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00081402600000000003 2020-06-12T02:37:53.270 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.270 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.270 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.270 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.691642+0000", 2020-06-12T02:37:53.271 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00024263 2020-06-12T02:37:53.271 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.271 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.271 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:53.271 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.691885+0000", 2020-06-12T02:37:53.272 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010968739 2020-06-12T02:37:53.272 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.272 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.272 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.272 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.702853+0000", 2020-06-12T02:37:53.272 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8985000000000002e-05 2020-06-12T02:37:53.273 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.273 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.273 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:53.273 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.702952+0000", 2020-06-12T02:37:53.273 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0046973329999999997 2020-06-12T02:37:53.274 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.274 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.274 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.274 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.707650+0000", 2020-06-12T02:37:53.274 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3773e-05 2020-06-12T02:37:53.274 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.275 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.275 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:53.275 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.707723+0000", 2020-06-12T02:37:53.275 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023692990000000001 2020-06-12T02:37:53.275 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.276 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.276 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:53.276 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.710093+0000", 2020-06-12T02:37:53.276 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019298497000000001 2020-06-12T02:37:53.276 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.276 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:53.277 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.277 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.277 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:53.277 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:37:53.298 INFO:tasks.rados.rados.0.smithi107.stdout:3846: writing smithi10729407-13 from 1622016 to 2424832 tid 1 2020-06-12T02:37:53.298 INFO:tasks.rados.rados.0.smithi107.stdout:3846: writing smithi10729407-13 from 2424832 to 3178496 tid 2 2020-06-12T02:37:53.298 INFO:tasks.rados.rados.0.smithi107.stdout:3846: writing smithi10729407-13 from 3178496 to 3629056 tid 3 2020-06-12T02:37:53.298 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:37:53.299 INFO:tasks.rados.rados.0.smithi107.stdout:3847: seq_num 3846 ranges {0=417792,417792=638976,1056768=671744,1728512=294912} 2020-06-12T02:37:53.299 INFO:tasks.rados.rados.0.smithi107.stdout:3847: writing smithi10729407-5 from 0 to 417792 tid 1 2020-06-12T02:37:53.299 INFO:tasks.rados.rados.0.smithi107.stdout:3847: writing smithi10729407-5 from 417792 to 1056768 tid 2 2020-06-12T02:37:53.299 INFO:tasks.rados.rados.0.smithi107.stdout:3847: writing smithi10729407-5 from 1056768 to 1728512 tid 3 2020-06-12T02:37:53.299 INFO:tasks.rados.rados.0.smithi107.stdout:3847: writing smithi10729407-5 from 1728512 to 2023424 tid 4 2020-06-12T02:37:53.300 INFO:tasks.rados.rados.0.smithi107.stdout:3848: delete oid 9 current snap is 0 2020-06-12T02:37:53.300 INFO:tasks.rados.rados.0.smithi107.stdout:3846: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:53.300 INFO:tasks.rados.rados.0.smithi107.stdout:3847: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:53.300 INFO:tasks.rados.rados.0.smithi107.stdout:3846: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:53.300 INFO:tasks.rados.rados.0.smithi107.stdout:3848: done (2 left) 2020-06-12T02:37:53.301 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:37:53.301 INFO:tasks.rados.rados.0.smithi107.stdout:3849: seq_num 3848 ranges {0=450560,450560=450560,901120=638976,1540096=516096} 2020-06-12T02:37:53.301 INFO:tasks.rados.rados.0.smithi107.stdout:3849: writing smithi10729407-6 from 0 to 450560 tid 1 2020-06-12T02:37:53.301 INFO:tasks.rados.rados.0.smithi107.stdout:3849: writing smithi10729407-6 from 450560 to 901120 tid 2 2020-06-12T02:37:53.302 INFO:tasks.rados.rados.0.smithi107.stdout:3849: writing smithi10729407-6 from 901120 to 1540096 tid 3 2020-06-12T02:37:53.302 INFO:tasks.rados.rados.0.smithi107.stdout:3849: writing smithi10729407-6 from 1540096 to 2056192 tid 4 2020-06-12T02:37:53.302 INFO:tasks.rados.rados.0.smithi107.stdout:3850: delete oid 4 current snap is 0 2020-06-12T02:37:53.302 INFO:tasks.rados.rados.0.smithi107.stdout:3846: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:53.302 INFO:tasks.rados.rados.0.smithi107.stdout:3846: finishing write tid 5 to smithi10729407-13 2020-06-12T02:37:53.302 INFO:tasks.rados.rados.0.smithi107.stdout:3846: finishing write tid 6 to smithi10729407-13 2020-06-12T02:37:53.303 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 729 (ObjNum 3845 snap 0 seq_num 3845) dirty exists 2020-06-12T02:37:53.303 INFO:tasks.rados.rados.0.smithi107.stdout:3846: left oid 13 (ObjNum 3845 snap 0 seq_num 3845) 2020-06-12T02:37:53.303 INFO:tasks.rados.rados.0.smithi107.stdout:3847: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:53.303 INFO:tasks.rados.rados.0.smithi107.stdout:3847: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:53.303 INFO:tasks.rados.rados.0.smithi107.stdout:3847: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:53.304 INFO:tasks.rados.rados.0.smithi107.stdout:3847: finishing write tid 6 to smithi10729407-5 2020-06-12T02:37:53.304 INFO:tasks.rados.rados.0.smithi107.stdout:3847: finishing write tid 7 to smithi10729407-5 2020-06-12T02:37:53.304 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 365 (ObjNum 3846 snap 0 seq_num 3846) dirty exists 2020-06-12T02:37:53.304 INFO:tasks.rados.rados.0.smithi107.stdout:3847: left oid 5 (ObjNum 3846 snap 0 seq_num 3846) 2020-06-12T02:37:53.304 INFO:tasks.rados.rados.0.smithi107.stdout:3849: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:53.304 INFO:tasks.rados.rados.0.smithi107.stdout:3846: done (3 left) 2020-06-12T02:37:53.305 INFO:tasks.rados.rados.0.smithi107.stdout:3847: done (2 left) 2020-06-12T02:37:53.305 INFO:tasks.rados.rados.0.smithi107.stdout:3850: done (1 left) 2020-06-12T02:37:53.305 INFO:tasks.rados.rados.0.smithi107.stdout:3851: delete oid 13 current snap is 0 2020-06-12T02:37:53.305 INFO:tasks.rados.rados.0.smithi107.stdout:3849: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:53.305 INFO:tasks.rados.rados.0.smithi107.stdout:3851: done (1 left) 2020-06-12T02:37:53.306 INFO:tasks.rados.rados.0.smithi107.stdout:3852: delete oid 5 current snap is 0 2020-06-12T02:37:53.306 INFO:tasks.rados.rados.0.smithi107.stdout:3849: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:53.306 INFO:tasks.rados.rados.0.smithi107.stdout:3849: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:53.306 INFO:tasks.rados.rados.0.smithi107.stdout:3849: finishing write tid 6 to smithi10729407-6 2020-06-12T02:37:53.306 INFO:tasks.rados.rados.0.smithi107.stdout:3852: done (1 left) 2020-06-12T02:37:53.306 INFO:tasks.rados.rados.0.smithi107.stdout:3853: delete oid 17 current snap is 0 2020-06-12T02:37:53.307 INFO:tasks.rados.rados.0.smithi107.stdout:3849: finishing write tid 7 to smithi10729407-6 2020-06-12T02:37:53.307 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 723 (ObjNum 3848 snap 0 seq_num 3848) dirty exists 2020-06-12T02:37:53.307 INFO:tasks.rados.rados.0.smithi107.stdout:3849: left oid 6 (ObjNum 3848 snap 0 seq_num 3848) 2020-06-12T02:37:53.307 INFO:tasks.rados.rados.0.smithi107.stdout:3849: done (1 left) 2020-06-12T02:37:53.307 INFO:tasks.rados.rados.0.smithi107.stdout:3853: done (0 left) 2020-06-12T02:37:53.308 INFO:tasks.rados.rados.0.smithi107.stdout:3854: delete oid 1 current snap is 0 2020-06-12T02:37:53.308 INFO:tasks.rados.rados.0.smithi107.stdout:3854: done (0 left) 2020-06-12T02:37:53.308 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:37:53.308 INFO:tasks.rados.rados.0.smithi107.stdout:3855: seq_num 3854 ranges {0=557056,557056=557056,1114112=630784,1744896=417792} 2020-06-12T02:37:53.308 INFO:tasks.rados.rados.0.smithi107.stdout:3855: writing smithi10729407-5 from 0 to 557056 tid 1 2020-06-12T02:37:53.308 INFO:tasks.rados.rados.0.smithi107.stdout:3855: writing smithi10729407-5 from 557056 to 1114112 tid 2 2020-06-12T02:37:53.309 INFO:tasks.rados.rados.0.smithi107.stdout:3855: writing smithi10729407-5 from 1114112 to 1744896 tid 3 2020-06-12T02:37:53.309 INFO:tasks.rados.rados.0.smithi107.stdout:3855: writing smithi10729407-5 from 1744896 to 2162688 tid 4 2020-06-12T02:37:53.309 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:37:53.309 INFO:tasks.rados.rados.0.smithi107.stdout:3856: seq_num 3855 ranges {0=573440,573440=491520,1064960=794624,1859584=319488} 2020-06-12T02:37:53.309 INFO:tasks.rados.rados.0.smithi107.stdout:3856: writing smithi10729407-4 from 0 to 573440 tid 1 2020-06-12T02:37:53.310 INFO:tasks.rados.rados.0.smithi107.stdout:3856: writing smithi10729407-4 from 573440 to 1064960 tid 2 2020-06-12T02:37:53.310 INFO:tasks.rados.rados.0.smithi107.stdout:3856: writing smithi10729407-4 from 1064960 to 1859584 tid 3 2020-06-12T02:37:53.310 INFO:tasks.rados.rados.0.smithi107.stdout:3856: writing smithi10729407-4 from 1859584 to 2179072 tid 4 2020-06-12T02:37:53.310 INFO:tasks.rados.rados.0.smithi107.stdout:3857: delete oid 18 current snap is 0 2020-06-12T02:37:53.310 INFO:tasks.rados.rados.0.smithi107.stdout:3855: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:53.310 INFO:tasks.rados.rados.0.smithi107.stdout:3855: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:53.311 INFO:tasks.rados.rados.0.smithi107.stdout:3855: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:53.311 INFO:tasks.rados.rados.0.smithi107.stdout:3855: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:53.311 INFO:tasks.rados.rados.0.smithi107.stdout:3855: finishing write tid 6 to smithi10729407-5 2020-06-12T02:37:53.311 INFO:tasks.rados.rados.0.smithi107.stdout:3855: finishing write tid 7 to smithi10729407-5 2020-06-12T02:37:53.311 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 371 (ObjNum 3854 snap 0 seq_num 3854) dirty exists 2020-06-12T02:37:53.312 INFO:tasks.rados.rados.0.smithi107.stdout:3855: left oid 5 (ObjNum 3854 snap 0 seq_num 3854) 2020-06-12T02:37:53.312 INFO:tasks.rados.rados.0.smithi107.stdout:3856: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:53.312 INFO:tasks.rados.rados.0.smithi107.stdout:3856: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:53.312 INFO:tasks.rados.rados.0.smithi107.stdout:3855: done (2 left) 2020-06-12T02:37:53.312 INFO:tasks.rados.rados.0.smithi107.stdout:3857: done (1 left) 2020-06-12T02:37:53.312 INFO:tasks.rados.rados.0.smithi107.stdout:3858: delete oid 5 current snap is 0 2020-06-12T02:37:53.313 INFO:tasks.rados.rados.0.smithi107.stdout:3858: done (1 left) 2020-06-12T02:37:53.313 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:37:53.313 INFO:tasks.rados.rados.0.smithi107.stdout:3859: seq_num 3858 ranges {0=622592,622592=548864,1171456=614400,1785856=442368} 2020-06-12T02:37:53.313 INFO:tasks.rados.rados.0.smithi107.stdout:3859: writing smithi10729407-8 from 0 to 622592 tid 1 2020-06-12T02:37:53.313 INFO:tasks.rados.rados.0.smithi107.stdout:3859: writing smithi10729407-8 from 622592 to 1171456 tid 2 2020-06-12T02:37:53.314 INFO:tasks.rados.rados.0.smithi107.stdout:3859: writing smithi10729407-8 from 1171456 to 1785856 tid 3 2020-06-12T02:37:53.314 INFO:tasks.rados.rados.0.smithi107.stdout:3859: writing smithi10729407-8 from 1785856 to 2228224 tid 4 2020-06-12T02:37:53.314 INFO:tasks.rados.rados.0.smithi107.stdout:3860: delete oid 19 current snap is 0 2020-06-12T02:37:53.314 INFO:tasks.rados.rados.0.smithi107.stdout:3856: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:53.314 INFO:tasks.rados.rados.0.smithi107.stdout:3856: finishing write tid 4 to smithi10729407-4 2020-06-12T02:37:53.315 INFO:tasks.rados.rados.0.smithi107.stdout:3856: finishing write tid 6 to smithi10729407-4 2020-06-12T02:37:53.315 INFO:tasks.rados.rados.0.smithi107.stdout:3856: finishing write tid 7 to smithi10729407-4 2020-06-12T02:37:53.315 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 708 (ObjNum 3855 snap 0 seq_num 3855) dirty exists 2020-06-12T02:37:53.315 INFO:tasks.rados.rados.0.smithi107.stdout:3856: left oid 4 (ObjNum 3855 snap 0 seq_num 3855) 2020-06-12T02:37:53.315 INFO:tasks.rados.rados.0.smithi107.stdout:3859: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:53.315 INFO:tasks.rados.rados.0.smithi107.stdout:3856: done (2 left) 2020-06-12T02:37:53.316 INFO:tasks.rados.rados.0.smithi107.stdout:3860: done (1 left) 2020-06-12T02:37:53.316 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:37:53.316 INFO:tasks.rados.rados.0.smithi107.stdout:3861: seq_num 3860 ranges {0=655360,655360=712704,1368064=581632,1949696=311296} 2020-06-12T02:37:53.316 INFO:tasks.rados.rados.0.smithi107.stdout:3861: writing smithi10729407-5 from 0 to 655360 tid 1 2020-06-12T02:37:53.316 INFO:tasks.rados.rados.0.smithi107.stdout:3861: writing smithi10729407-5 from 655360 to 1368064 tid 2 2020-06-12T02:37:53.317 INFO:tasks.rados.rados.0.smithi107.stdout:3861: writing smithi10729407-5 from 1368064 to 1949696 tid 3 2020-06-12T02:37:53.317 INFO:tasks.rados.rados.0.smithi107.stdout:3861: writing smithi10729407-5 from 1949696 to 2260992 tid 4 2020-06-12T02:37:53.317 INFO:tasks.rados.rados.0.smithi107.stdout:3862: delete oid 14 current snap is 0 2020-06-12T02:37:53.317 INFO:tasks.rados.rados.0.smithi107.stdout:3859: finishing write tid 2 to smithi10729407-8 2020-06-12T02:37:53.317 INFO:tasks.rados.rados.0.smithi107.stdout:3859: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:53.317 INFO:tasks.rados.rados.0.smithi107.stdout:3859: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:53.318 INFO:tasks.rados.rados.0.smithi107.stdout:3859: finishing write tid 6 to smithi10729407-8 2020-06-12T02:37:53.318 INFO:tasks.rados.rados.0.smithi107.stdout:3861: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:53.318 INFO:tasks.rados.rados.0.smithi107.stdout:3859: finishing write tid 7 to smithi10729407-8 2020-06-12T02:37:53.318 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 661 (ObjNum 3858 snap 0 seq_num 3858) dirty exists 2020-06-12T02:37:53.318 INFO:tasks.rados.rados.0.smithi107.stdout:3859: left oid 8 (ObjNum 3858 snap 0 seq_num 3858) 2020-06-12T02:37:53.318 INFO:tasks.rados.rados.0.smithi107.stdout:3859: done (2 left) 2020-06-12T02:37:53.319 INFO:tasks.rados.rados.0.smithi107.stdout:3862: done (1 left) 2020-06-12T02:37:53.319 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:37:53.319 INFO:tasks.rados.rados.0.smithi107.stdout:3863: seq_num 3862 ranges {1957888=688128,2646016=483328,3129344=589824,3719168=532480} 2020-06-12T02:37:53.319 INFO:tasks.rados.rados.0.smithi107.stdout:3863: writing smithi10729407-12 from 1957888 to 2646016 tid 1 2020-06-12T02:37:53.319 INFO:tasks.rados.rados.0.smithi107.stdout:3863: writing smithi10729407-12 from 2646016 to 3129344 tid 2 2020-06-12T02:37:53.320 INFO:tasks.rados.rados.0.smithi107.stdout:3863: writing smithi10729407-12 from 3129344 to 3719168 tid 3 2020-06-12T02:37:53.320 INFO:tasks.rados.rados.0.smithi107.stdout:3863: writing smithi10729407-12 from 3719168 to 4251648 tid 4 2020-06-12T02:37:53.320 INFO:tasks.rados.rados.0.smithi107.stdout:3864: delete oid 10 current snap is 0 2020-06-12T02:37:53.320 INFO:tasks.rados.rados.0.smithi107.stdout:3861: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:53.320 INFO:tasks.rados.rados.0.smithi107.stdout:3861: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:53.320 INFO:tasks.rados.rados.0.smithi107.stdout:3861: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:53.321 INFO:tasks.rados.rados.0.smithi107.stdout:3861: finishing write tid 6 to smithi10729407-5 2020-06-12T02:37:53.321 INFO:tasks.rados.rados.0.smithi107.stdout:3861: finishing write tid 7 to smithi10729407-5 2020-06-12T02:37:53.321 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 377 (ObjNum 3860 snap 0 seq_num 3860) dirty exists 2020-06-12T02:37:53.321 INFO:tasks.rados.rados.0.smithi107.stdout:3861: left oid 5 (ObjNum 3860 snap 0 seq_num 3860) 2020-06-12T02:37:53.321 INFO:tasks.rados.rados.0.smithi107.stdout:3863: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:53.322 INFO:tasks.rados.rados.0.smithi107.stdout:3863: finishing write tid 2 to smithi10729407-12 2020-06-12T02:37:53.322 INFO:tasks.rados.rados.0.smithi107.stdout:3861: done (2 left) 2020-06-12T02:37:53.322 INFO:tasks.rados.rados.0.smithi107.stdout:3864: done (1 left) 2020-06-12T02:37:53.322 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:37:53.322 INFO:tasks.rados.rados.0.smithi107.stdout:3865: seq_num 3864 ranges {0=720896,720896=704512,1425408=557056,1982464=344064} 2020-06-12T02:37:53.322 INFO:tasks.rados.rados.0.smithi107.stdout:3865: writing smithi10729407-20 from 0 to 720896 tid 1 2020-06-12T02:37:53.323 INFO:tasks.rados.rados.0.smithi107.stdout:3865: writing smithi10729407-20 from 720896 to 1425408 tid 2 2020-06-12T02:37:53.323 INFO:tasks.rados.rados.0.smithi107.stdout:3865: writing smithi10729407-20 from 1425408 to 1982464 tid 3 2020-06-12T02:37:53.323 INFO:tasks.rados.rados.0.smithi107.stdout:3865: writing smithi10729407-20 from 1982464 to 2326528 tid 4 2020-06-12T02:37:53.323 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:37:53.323 INFO:tasks.rados.rados.0.smithi107.stdout:3866: seq_num 3865 ranges {0=737280,737280=589824,1327104=770048,2097152=245760} 2020-06-12T02:37:53.323 INFO:tasks.rados.rados.0.smithi107.stdout:3866: writing smithi10729407-11 from 0 to 737280 tid 1 2020-06-12T02:37:53.324 INFO:tasks.rados.rados.0.smithi107.stdout:3866: writing smithi10729407-11 from 737280 to 1327104 tid 2 2020-06-12T02:37:53.324 INFO:tasks.rados.rados.0.smithi107.stdout:3866: writing smithi10729407-11 from 1327104 to 2097152 tid 3 2020-06-12T02:37:53.324 INFO:tasks.rados.rados.0.smithi107.stdout:3866: writing smithi10729407-11 from 2097152 to 2342912 tid 4 2020-06-12T02:37:53.324 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:37:53.324 INFO:tasks.rados.rados.0.smithi107.stdout:3867: seq_num 3866 ranges {0=753664,753664=466944,1220608=573440,1794048=565248} 2020-06-12T02:37:53.325 INFO:tasks.rados.rados.0.smithi107.stdout:3867: writing smithi10729407-13 from 0 to 753664 tid 1 2020-06-12T02:37:53.325 INFO:tasks.rados.rados.0.smithi107.stdout:3867: writing smithi10729407-13 from 753664 to 1220608 tid 2 2020-06-12T02:37:53.325 INFO:tasks.rados.rados.0.smithi107.stdout:3867: writing smithi10729407-13 from 1220608 to 1794048 tid 3 2020-06-12T02:37:53.325 INFO:tasks.rados.rados.0.smithi107.stdout:3867: writing smithi10729407-13 from 1794048 to 2359296 tid 4 2020-06-12T02:37:53.325 INFO:tasks.rados.rados.0.smithi107.stdout:3868: delete oid 14 current snap is 0 2020-06-12T02:37:53.325 INFO:tasks.rados.rados.0.smithi107.stdout:3863: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:53.326 INFO:tasks.rados.rados.0.smithi107.stdout:3863: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:53.326 INFO:tasks.rados.rados.0.smithi107.stdout:3863: finishing write tid 6 to smithi10729407-12 2020-06-12T02:37:53.326 INFO:tasks.rados.rados.0.smithi107.stdout:3863: finishing write tid 7 to smithi10729407-12 2020-06-12T02:37:53.326 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 657 (ObjNum 3862 snap 0 seq_num 3862) dirty exists 2020-06-12T02:37:53.326 INFO:tasks.rados.rados.0.smithi107.stdout:3863: left oid 12 (ObjNum 3862 snap 0 seq_num 3862) 2020-06-12T02:37:53.326 INFO:tasks.rados.rados.0.smithi107.stdout:3865: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:53.327 INFO:tasks.rados.rados.0.smithi107.stdout:3865: finishing write tid 2 to smithi10729407-20 2020-06-12T02:37:53.327 INFO:tasks.rados.rados.0.smithi107.stdout:3865: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:53.327 INFO:tasks.rados.rados.0.smithi107.stdout:3865: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:53.327 INFO:tasks.rados.rados.0.smithi107.stdout:3865: finishing write tid 6 to smithi10729407-20 2020-06-12T02:37:53.327 INFO:tasks.rados.rados.0.smithi107.stdout:3865: finishing write tid 7 to smithi10729407-20 2020-06-12T02:37:53.328 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 297 (ObjNum 3864 snap 0 seq_num 3864) dirty exists 2020-06-12T02:37:53.328 INFO:tasks.rados.rados.0.smithi107.stdout:3865: left oid 20 (ObjNum 3864 snap 0 seq_num 3864) 2020-06-12T02:37:53.328 INFO:tasks.rados.rados.0.smithi107.stdout:3866: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:53.328 INFO:tasks.rados.rados.0.smithi107.stdout:3866: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:53.328 INFO:tasks.rados.rados.0.smithi107.stdout:3866: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:53.328 INFO:tasks.rados.rados.0.smithi107.stdout:3866: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:53.329 INFO:tasks.rados.rados.0.smithi107.stdout:3866: finishing write tid 6 to smithi10729407-11 2020-06-12T02:37:53.329 INFO:tasks.rados.rados.0.smithi107.stdout:3866: finishing write tid 7 to smithi10729407-11 2020-06-12T02:37:53.329 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 675 (ObjNum 3865 snap 0 seq_num 3865) dirty exists 2020-06-12T02:37:53.329 INFO:tasks.rados.rados.0.smithi107.stdout:3866: left oid 11 (ObjNum 3865 snap 0 seq_num 3865) 2020-06-12T02:37:53.329 INFO:tasks.rados.rados.0.smithi107.stdout:3867: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:53.330 INFO:tasks.rados.rados.0.smithi107.stdout:3863: done (4 left) 2020-06-12T02:37:53.330 INFO:tasks.rados.rados.0.smithi107.stdout:3865: done (3 left) 2020-06-12T02:37:53.330 INFO:tasks.rados.rados.0.smithi107.stdout:3866: done (2 left) 2020-06-12T02:37:53.330 INFO:tasks.rados.rados.0.smithi107.stdout:3868: done (1 left) 2020-06-12T02:37:53.330 INFO:tasks.rados.rados.0.smithi107.stdout:3869: delete oid 1 current snap is 0 2020-06-12T02:37:53.331 INFO:tasks.rados.rados.0.smithi107.stdout:3869: done (1 left) 2020-06-12T02:37:53.331 INFO:tasks.rados.rados.0.smithi107.stdout:3870: delete oid 8 current snap is 0 2020-06-12T02:37:53.331 INFO:tasks.rados.rados.0.smithi107.stdout:3867: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:53.331 INFO:tasks.rados.rados.0.smithi107.stdout:3867: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:53.332 INFO:tasks.rados.rados.0.smithi107.stdout:3867: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:53.332 INFO:tasks.rados.rados.0.smithi107.stdout:3867: finishing write tid 6 to smithi10729407-13 2020-06-12T02:37:53.332 INFO:tasks.rados.rados.0.smithi107.stdout:3870: done (1 left) 2020-06-12T02:37:53.332 INFO:tasks.rados.rados.0.smithi107.stdout:3871: delete oid 18 current snap is 0 2020-06-12T02:37:53.332 INFO:tasks.rados.rados.0.smithi107.stdout:3867: finishing write tid 7 to smithi10729407-13 2020-06-12T02:37:53.332 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 735 (ObjNum 3866 snap 0 seq_num 3866) dirty exists 2020-06-12T02:37:53.333 INFO:tasks.rados.rados.0.smithi107.stdout:3867: left oid 13 (ObjNum 3866 snap 0 seq_num 3866) 2020-06-12T02:37:53.333 INFO:tasks.rados.rados.0.smithi107.stdout:3867: done (1 left) 2020-06-12T02:37:53.333 INFO:tasks.rados.rados.0.smithi107.stdout:3871: done (0 left) 2020-06-12T02:37:53.333 INFO:tasks.rados.rados.0.smithi107.stdout:3872: delete oid 5 current snap is 0 2020-06-12T02:37:53.333 INFO:tasks.rados.rados.0.smithi107.stdout:3872: done (0 left) 2020-06-12T02:37:53.334 INFO:tasks.rados.rados.0.smithi107.stdout:3873: delete oid 14 current snap is 0 2020-06-12T02:37:53.334 INFO:tasks.rados.rados.0.smithi107.stdout:3873: done (0 left) 2020-06-12T02:37:53.334 INFO:tasks.rados.rados.0.smithi107.stdout:3874: delete oid 8 current snap is 0 2020-06-12T02:37:53.334 INFO:tasks.rados.rados.0.smithi107.stdout:3874: done (0 left) 2020-06-12T02:37:53.334 INFO:tasks.rados.rados.0.smithi107.stdout:3875: delete oid 6 current snap is 0 2020-06-12T02:37:53.335 INFO:tasks.rados.rados.0.smithi107.stdout:3875: done (0 left) 2020-06-12T02:37:53.335 INFO:tasks.rados.rados.0.smithi107.stdout:3876: delete oid 20 current snap is 0 2020-06-12T02:37:53.335 INFO:tasks.rados.rados.0.smithi107.stdout:3876: done (0 left) 2020-06-12T02:37:53.335 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:37:53.335 INFO:tasks.rados.rados.0.smithi107.stdout:3877: seq_num 3876 ranges {0=122880} 2020-06-12T02:37:53.335 INFO:tasks.rados.rados.0.smithi107.stdout:3877: writing smithi10729407-6 from 0 to 122880 tid 1 2020-06-12T02:37:53.336 INFO:tasks.rados.rados.0.smithi107.stdout:3878: delete oid 3 current snap is 0 2020-06-12T02:37:53.336 INFO:tasks.rados.rados.0.smithi107.stdout:3878: done (1 left) 2020-06-12T02:37:53.336 INFO:tasks.rados.rados.0.smithi107.stdout:3879: delete oid 1 current snap is 0 2020-06-12T02:37:53.336 INFO:tasks.rados.rados.0.smithi107.stdout:3877: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:53.336 INFO:tasks.rados.rados.0.smithi107.stdout:3879: done (1 left) 2020-06-12T02:37:53.336 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:37:53.337 INFO:tasks.rados.rados.0.smithi107.stdout:3880: seq_num 3879 ranges {0=172032} 2020-06-12T02:37:53.337 INFO:tasks.rados.rados.0.smithi107.stdout:3880: writing smithi10729407-1 from 0 to 172032 tid 1 2020-06-12T02:37:53.337 INFO:tasks.rados.rados.0.smithi107.stdout:3881: delete oid 18 current snap is 0 2020-06-12T02:37:53.337 INFO:tasks.rados.rados.0.smithi107.stdout:3877: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:53.337 INFO:tasks.rados.rados.0.smithi107.stdout:3877: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:53.338 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 729 (ObjNum 3876 snap 0 seq_num 3876) dirty exists 2020-06-12T02:37:53.338 INFO:tasks.rados.rados.0.smithi107.stdout:3877: left oid 6 (ObjNum 3876 snap 0 seq_num 3876) 2020-06-12T02:37:53.338 INFO:tasks.rados.rados.0.smithi107.stdout:3877: done (2 left) 2020-06-12T02:37:53.338 INFO:tasks.rados.rados.0.smithi107.stdout:3881: done (1 left) 2020-06-12T02:37:53.338 INFO:tasks.rados.rados.0.smithi107.stdout:3882: delete oid 6 current snap is 0 2020-06-12T02:37:53.338 INFO:tasks.rados.rados.0.smithi107.stdout:3880: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:53.339 INFO:tasks.rados.rados.0.smithi107.stdout:3880: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:53.339 INFO:tasks.rados.rados.0.smithi107.stdout:3882: done (1 left) 2020-06-12T02:37:53.339 INFO:tasks.rados.rados.0.smithi107.stdout:3883: delete oid 10 current snap is 0 2020-06-12T02:37:53.339 INFO:tasks.rados.rados.0.smithi107.stdout:3880: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:53.339 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 731 (ObjNum 3879 snap 0 seq_num 3879) dirty exists 2020-06-12T02:37:53.340 INFO:tasks.rados.rados.0.smithi107.stdout:3880: left oid 1 (ObjNum 3879 snap 0 seq_num 3879) 2020-06-12T02:37:53.340 INFO:tasks.rados.rados.0.smithi107.stdout:3880: done (1 left) 2020-06-12T02:37:53.340 INFO:tasks.rados.rados.0.smithi107.stdout:3883: done (0 left) 2020-06-12T02:37:53.340 INFO:tasks.rados.rados.0.smithi107.stdout:3884: delete oid 10 current snap is 0 2020-06-12T02:37:53.340 INFO:tasks.rados.rados.0.smithi107.stdout:3884: done (0 left) 2020-06-12T02:37:53.340 INFO:tasks.rados.rados.0.smithi107.stdout:3885: delete oid 13 current snap is 0 2020-06-12T02:37:53.341 INFO:tasks.rados.rados.0.smithi107.stdout:3885: done (0 left) 2020-06-12T02:37:53.341 INFO:tasks.rados.rados.0.smithi107.stdout:3886: delete oid 4 current snap is 0 2020-06-12T02:37:53.341 INFO:tasks.rados.rados.0.smithi107.stdout:3886: done (0 left) 2020-06-12T02:37:53.341 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:37:53.341 INFO:tasks.rados.rados.0.smithi107.stdout:3887: seq_num 3886 ranges {0=286720} 2020-06-12T02:37:53.342 INFO:tasks.rados.rados.0.smithi107.stdout:3887: writing smithi10729407-16 from 0 to 286720 tid 1 2020-06-12T02:37:53.342 INFO:tasks.rados.rados.0.smithi107.stdout:3888: delete oid 5 current snap is 0 2020-06-12T02:37:53.342 INFO:tasks.rados.rados.0.smithi107.stdout:3888: done (1 left) 2020-06-12T02:37:53.342 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:37:53.342 INFO:tasks.rados.rados.0.smithi107.stdout:3889: seq_num 3888 ranges {0=319488} 2020-06-12T02:37:53.342 INFO:tasks.rados.rados.0.smithi107.stdout:3889: writing smithi10729407-9 from 0 to 319488 tid 1 2020-06-12T02:37:53.343 INFO:tasks.rados.rados.0.smithi107.stdout:3890: delete oid 4 current snap is 0 2020-06-12T02:37:53.343 INFO:tasks.rados.rados.0.smithi107.stdout:3887: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:53.343 INFO:tasks.rados.rados.0.smithi107.stdout:3887: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:53.343 INFO:tasks.rados.rados.0.smithi107.stdout:3887: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:53.343 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 699 (ObjNum 3886 snap 0 seq_num 3886) dirty exists 2020-06-12T02:37:53.343 INFO:tasks.rados.rados.0.smithi107.stdout:3887: left oid 16 (ObjNum 3886 snap 0 seq_num 3886) 2020-06-12T02:37:53.344 INFO:tasks.rados.rados.0.smithi107.stdout:3887: done (2 left) 2020-06-12T02:37:53.344 INFO:tasks.rados.rados.0.smithi107.stdout:3890: done (1 left) 2020-06-12T02:37:53.344 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:37:53.344 INFO:tasks.rados.rados.0.smithi107.stdout:3891: seq_num 3890 ranges {0=352256} 2020-06-12T02:37:53.344 INFO:tasks.rados.rados.0.smithi107.stdout:3891: writing smithi10729407-8 from 0 to 352256 tid 1 2020-06-12T02:37:53.345 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:37:53.345 INFO:tasks.rados.rados.0.smithi107.stdout:3892: seq_num 3891 ranges {0=368640} 2020-06-12T02:37:53.345 INFO:tasks.rados.rados.0.smithi107.stdout:3892: writing smithi10729407-10 from 0 to 368640 tid 1 2020-06-12T02:37:53.345 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:37:53.345 INFO:tasks.rados.rados.0.smithi107.stdout:3893: seq_num 3892 ranges {2342912=385024} 2020-06-12T02:37:53.345 INFO:tasks.rados.rados.0.smithi107.stdout:3893: writing smithi10729407-11 from 2342912 to 2727936 tid 1 2020-06-12T02:37:53.346 INFO:tasks.rados.rados.0.smithi107.stdout:3894: delete oid 4 current snap is 0 2020-06-12T02:37:53.346 INFO:tasks.rados.rados.0.smithi107.stdout:3889: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:53.346 INFO:tasks.rados.rados.0.smithi107.stdout:3889: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:53.346 INFO:tasks.rados.rados.0.smithi107.stdout:3889: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:53.346 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 683 (ObjNum 3888 snap 0 seq_num 3888) dirty exists 2020-06-12T02:37:53.347 INFO:tasks.rados.rados.0.smithi107.stdout:3889: left oid 9 (ObjNum 3888 snap 0 seq_num 3888) 2020-06-12T02:37:53.347 INFO:tasks.rados.rados.0.smithi107.stdout:3891: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:53.347 INFO:tasks.rados.rados.0.smithi107.stdout:3892: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:53.347 INFO:tasks.rados.rados.0.smithi107.stdout:3891: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:53.347 INFO:tasks.rados.rados.0.smithi107.stdout:3892: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:53.347 INFO:tasks.rados.rados.0.smithi107.stdout:3891: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:53.348 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 665 (ObjNum 3890 snap 0 seq_num 3890) dirty exists 2020-06-12T02:37:53.348 INFO:tasks.rados.rados.0.smithi107.stdout:3891: left oid 8 (ObjNum 3890 snap 0 seq_num 3890) 2020-06-12T02:37:53.348 INFO:tasks.rados.rados.0.smithi107.stdout:3892: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:53.348 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 380 (ObjNum 3891 snap 0 seq_num 3891) dirty exists 2020-06-12T02:37:53.348 INFO:tasks.rados.rados.0.smithi107.stdout:3892: left oid 10 (ObjNum 3891 snap 0 seq_num 3891) 2020-06-12T02:37:53.348 INFO:tasks.rados.rados.0.smithi107.stdout:3889: done (4 left) 2020-06-12T02:37:53.349 INFO:tasks.rados.rados.0.smithi107.stdout:3891: done (3 left) 2020-06-12T02:37:53.349 INFO:tasks.rados.rados.0.smithi107.stdout:3892: done (2 left) 2020-06-12T02:37:53.349 INFO:tasks.rados.rados.0.smithi107.stdout:3894: done (1 left) 2020-06-12T02:37:53.349 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:37:53.349 INFO:tasks.rados.rados.0.smithi107.stdout:3895: seq_num 3894 ranges {172032=425984} 2020-06-12T02:37:53.350 INFO:tasks.rados.rados.0.smithi107.stdout:3895: writing smithi10729407-1 from 172032 to 598016 tid 1 2020-06-12T02:37:53.350 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:37:53.350 INFO:tasks.rados.rados.0.smithi107.stdout:3896: seq_num 3895 ranges {319488=442368} 2020-06-12T02:37:53.350 INFO:tasks.rados.rados.0.smithi107.stdout:3896: writing smithi10729407-9 from 319488 to 761856 tid 1 2020-06-12T02:37:53.350 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:37:53.350 INFO:tasks.rados.rados.0.smithi107.stdout:3897: seq_num 3896 ranges {0=458752} 2020-06-12T02:37:53.351 INFO:tasks.rados.rados.0.smithi107.stdout:3897: writing smithi10729407-20 from 0 to 458752 tid 1 2020-06-12T02:37:53.351 INFO:tasks.rados.rados.0.smithi107.stdout:3898: delete oid 13 current snap is 0 2020-06-12T02:37:53.351 INFO:tasks.rados.rados.0.smithi107.stdout:3893: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:53.351 INFO:tasks.rados.rados.0.smithi107.stdout:3893: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:53.351 INFO:tasks.rados.rados.0.smithi107.stdout:3893: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:53.352 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 679 (ObjNum 3892 snap 0 seq_num 3892) dirty exists 2020-06-12T02:37:53.352 INFO:tasks.rados.rados.0.smithi107.stdout:3893: left oid 11 (ObjNum 3892 snap 0 seq_num 3892) 2020-06-12T02:37:53.352 INFO:tasks.rados.rados.0.smithi107.stdout:3895: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:53.352 INFO:tasks.rados.rados.0.smithi107.stdout:3895: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:53.352 INFO:tasks.rados.rados.0.smithi107.stdout:3896: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:53.352 INFO:tasks.rados.rados.0.smithi107.stdout:3895: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:53.353 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 734 (ObjNum 3894 snap 0 seq_num 3894) dirty exists 2020-06-12T02:37:53.353 INFO:tasks.rados.rados.0.smithi107.stdout:3895: left oid 1 (ObjNum 3894 snap 0 seq_num 3894) 2020-06-12T02:37:53.353 INFO:tasks.rados.rados.0.smithi107.stdout:3896: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:53.353 INFO:tasks.rados.rados.0.smithi107.stdout:3893: done (4 left) 2020-06-12T02:37:53.353 INFO:tasks.rados.rados.0.smithi107.stdout:3895: done (3 left) 2020-06-12T02:37:53.354 INFO:tasks.rados.rados.0.smithi107.stdout:3898: done (2 left) 2020-06-12T02:37:53.354 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:37:53.354 INFO:tasks.rados.rados.0.smithi107.stdout:3899: seq_num 3898 ranges {0=491520} 2020-06-12T02:37:53.354 INFO:tasks.rados.rados.0.smithi107.stdout:3899: writing smithi10729407-14 from 0 to 491520 tid 1 2020-06-12T02:37:53.354 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:37:53.354 INFO:tasks.rados.rados.0.smithi107.stdout:3900: seq_num 3899 ranges {2727936=507904} 2020-06-12T02:37:53.355 INFO:tasks.rados.rados.0.smithi107.stdout:3900: writing smithi10729407-11 from 2727936 to 3235840 tid 1 2020-06-12T02:37:53.355 INFO:tasks.rados.rados.0.smithi107.stdout:3901: delete oid 13 current snap is 0 2020-06-12T02:37:53.355 INFO:tasks.rados.rados.0.smithi107.stdout:3896: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:53.355 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 685 (ObjNum 3895 snap 0 seq_num 3895) dirty exists 2020-06-12T02:37:53.355 INFO:tasks.rados.rados.0.smithi107.stdout:3896: left oid 9 (ObjNum 3895 snap 0 seq_num 3895) 2020-06-12T02:37:53.356 INFO:tasks.rados.rados.0.smithi107.stdout:3897: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:53.356 INFO:tasks.rados.rados.0.smithi107.stdout:3897: finishing write tid 3 to smithi10729407-20 2020-06-12T02:37:53.356 INFO:tasks.rados.rados.0.smithi107.stdout:3897: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:53.356 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 300 (ObjNum 3896 snap 0 seq_num 3896) dirty exists 2020-06-12T02:37:53.356 INFO:tasks.rados.rados.0.smithi107.stdout:3897: left oid 20 (ObjNum 3896 snap 0 seq_num 3896) 2020-06-12T02:37:53.356 INFO:tasks.rados.rados.0.smithi107.stdout:3899: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:53.357 INFO:tasks.rados.rados.0.smithi107.stdout:3899: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:53.357 INFO:tasks.rados.rados.0.smithi107.stdout:3899: finishing write tid 4 to smithi10729407-14 2020-06-12T02:37:53.357 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 681 (ObjNum 3898 snap 0 seq_num 3898) dirty exists 2020-06-12T02:37:53.357 INFO:tasks.rados.rados.0.smithi107.stdout:3899: left oid 14 (ObjNum 3898 snap 0 seq_num 3898) 2020-06-12T02:37:53.357 INFO:tasks.rados.rados.0.smithi107.stdout:3896: done (4 left) 2020-06-12T02:37:53.358 INFO:tasks.rados.rados.0.smithi107.stdout:3897: done (3 left) 2020-06-12T02:37:53.358 INFO:tasks.rados.rados.0.smithi107.stdout:3899: done (2 left) 2020-06-12T02:37:53.358 INFO:tasks.rados.rados.0.smithi107.stdout:3901: done (1 left) 2020-06-12T02:37:53.358 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:37:53.358 INFO:tasks.rados.rados.0.smithi107.stdout:3902: seq_num 3901 ranges {491520=540672} 2020-06-12T02:37:53.358 INFO:tasks.rados.rados.0.smithi107.stdout:3902: writing smithi10729407-14 from 491520 to 1032192 tid 1 2020-06-12T02:37:53.359 INFO:tasks.rados.rados.0.smithi107.stdout:3903: delete oid 15 current snap is 0 2020-06-12T02:37:53.359 INFO:tasks.rados.rados.0.smithi107.stdout:3900: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:53.359 INFO:tasks.rados.rados.0.smithi107.stdout:3900: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:53.359 INFO:tasks.rados.rados.0.smithi107.stdout:3900: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:53.359 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 683 (ObjNum 3899 snap 0 seq_num 3899) dirty exists 2020-06-12T02:37:53.359 INFO:tasks.rados.rados.0.smithi107.stdout:3900: left oid 11 (ObjNum 3899 snap 0 seq_num 3899) 2020-06-12T02:37:53.360 INFO:tasks.rados.rados.0.smithi107.stdout:3900: done (2 left) 2020-06-12T02:37:53.360 INFO:tasks.rados.rados.0.smithi107.stdout:3903: done (1 left) 2020-06-12T02:37:53.360 INFO:tasks.rados.rados.0.smithi107.stdout:3904: delete oid 2 current snap is 0 2020-06-12T02:37:53.360 INFO:tasks.rados.rados.0.smithi107.stdout:3904: done (1 left) 2020-06-12T02:37:53.360 INFO:tasks.rados.rados.0.smithi107.stdout:3905: delete oid 13 current snap is 0 2020-06-12T02:37:53.361 INFO:tasks.rados.rados.0.smithi107.stdout:3902: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:53.361 INFO:tasks.rados.rados.0.smithi107.stdout:3902: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:53.361 INFO:tasks.rados.rados.0.smithi107.stdout:3905: done (1 left) 2020-06-12T02:37:53.361 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:37:53.361 INFO:tasks.rados.rados.0.smithi107.stdout:3906: seq_num 3905 ranges {0=606208} 2020-06-12T02:37:53.361 INFO:tasks.rados.rados.0.smithi107.stdout:3906: writing smithi10729407-15 from 0 to 606208 tid 1 2020-06-12T02:37:53.362 INFO:tasks.rados.rados.0.smithi107.stdout:3907: delete oid 3 current snap is 0 2020-06-12T02:37:53.362 INFO:tasks.rados.rados.0.smithi107.stdout:3902: finishing write tid 4 to smithi10729407-14 2020-06-12T02:37:53.362 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 685 (ObjNum 3901 snap 0 seq_num 3901) dirty exists 2020-06-12T02:37:53.362 INFO:tasks.rados.rados.0.smithi107.stdout:3902: left oid 14 (ObjNum 3901 snap 0 seq_num 3901) 2020-06-12T02:37:53.362 INFO:tasks.rados.rados.0.smithi107.stdout:3902: done (2 left) 2020-06-12T02:37:53.363 INFO:tasks.rados.rados.0.smithi107.stdout:3907: done (1 left) 2020-06-12T02:37:53.363 INFO:tasks.rados.rados.0.smithi107.stdout:3908: delete oid 14 current snap is 0 2020-06-12T02:37:53.363 INFO:tasks.rados.rados.0.smithi107.stdout:3908: done (1 left) 2020-06-12T02:37:53.363 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:37:53.363 INFO:tasks.rados.rados.0.smithi107.stdout:3909: seq_num 3908 ranges {0=655360} 2020-06-12T02:37:53.363 INFO:tasks.rados.rados.0.smithi107.stdout:3909: writing smithi10729407-6 from 0 to 655360 tid 1 2020-06-12T02:37:53.364 INFO:tasks.rados.rados.0.smithi107.stdout:3910: delete oid 3 current snap is 0 2020-06-12T02:37:53.364 INFO:tasks.rados.rados.0.smithi107.stdout:3906: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:53.364 INFO:tasks.rados.rados.0.smithi107.stdout:3906: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:53.364 INFO:tasks.rados.rados.0.smithi107.stdout:3906: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:53.364 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 668 (ObjNum 3905 snap 0 seq_num 3905) dirty exists 2020-06-12T02:37:53.365 INFO:tasks.rados.rados.0.smithi107.stdout:3906: left oid 15 (ObjNum 3905 snap 0 seq_num 3905) 2020-06-12T02:37:53.365 INFO:tasks.rados.rados.0.smithi107.stdout:3906: done (2 left) 2020-06-12T02:37:53.365 INFO:tasks.rados.rados.0.smithi107.stdout:3910: done (1 left) 2020-06-12T02:37:53.365 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:37:53.365 INFO:tasks.rados.rados.0.smithi107.stdout:3911: seq_num 3910 ranges {0=688128} 2020-06-12T02:37:53.365 INFO:tasks.rados.rados.0.smithi107.stdout:3911: writing smithi10729407-17 from 0 to 688128 tid 1 2020-06-12T02:37:53.366 INFO:tasks.rados.rados.0.smithi107.stdout:3912: delete oid 15 current snap is 0 2020-06-12T02:37:53.366 INFO:tasks.rados.rados.0.smithi107.stdout:3909: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:53.366 INFO:tasks.rados.rados.0.smithi107.stdout:3909: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:53.366 INFO:tasks.rados.rados.0.smithi107.stdout:3909: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:53.366 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 736 (ObjNum 3908 snap 0 seq_num 3908) dirty exists 2020-06-12T02:37:53.367 INFO:tasks.rados.rados.0.smithi107.stdout:3909: left oid 6 (ObjNum 3908 snap 0 seq_num 3908) 2020-06-12T02:37:53.367 INFO:tasks.rados.rados.0.smithi107.stdout:3909: done (2 left) 2020-06-12T02:37:53.367 INFO:tasks.rados.rados.0.smithi107.stdout:3912: done (1 left) 2020-06-12T02:37:53.367 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:37:53.367 INFO:tasks.rados.rados.0.smithi107.stdout:3913: seq_num 3912 ranges {3235840=720896} 2020-06-12T02:37:53.367 INFO:tasks.rados.rados.0.smithi107.stdout:3913: writing smithi10729407-11 from 3235840 to 3956736 tid 1 2020-06-12T02:37:53.368 INFO:tasks.rados.rados.0.smithi107.stdout:3914: delete oid 10 current snap is 0 2020-06-12T02:37:53.368 INFO:tasks.rados.rados.0.smithi107.stdout:3911: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:53.368 INFO:tasks.rados.rados.0.smithi107.stdout:3911: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:53.368 INFO:tasks.rados.rados.0.smithi107.stdout:3911: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:53.368 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 713 (ObjNum 3910 snap 0 seq_num 3910) dirty exists 2020-06-12T02:37:53.369 INFO:tasks.rados.rados.0.smithi107.stdout:3911: left oid 17 (ObjNum 3910 snap 0 seq_num 3910) 2020-06-12T02:37:53.369 INFO:tasks.rados.rados.0.smithi107.stdout:3911: done (2 left) 2020-06-12T02:37:53.369 INFO:tasks.rados.rados.0.smithi107.stdout:3914: done (1 left) 2020-06-12T02:37:53.369 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:53.369 INFO:tasks.rados.rados.0.smithi107.stdout:3915: seq_num 3914 ranges {0=761856} 2020-06-12T02:37:53.369 INFO:tasks.rados.rados.0.smithi107.stdout:3915: writing smithi10729407-2 from 0 to 761856 tid 1 2020-06-12T02:37:53.370 INFO:tasks.rados.rados.0.smithi107.stdout:3916: delete oid 3 current snap is 0 2020-06-12T02:37:53.370 INFO:tasks.rados.rados.0.smithi107.stdout:3913: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:53.370 INFO:tasks.rados.rados.0.smithi107.stdout:3913: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:53.370 INFO:tasks.rados.rados.0.smithi107.stdout:3913: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:53.371 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 688 (ObjNum 3912 snap 0 seq_num 3912) dirty exists 2020-06-12T02:37:53.371 INFO:tasks.rados.rados.0.smithi107.stdout:3913: left oid 11 (ObjNum 3912 snap 0 seq_num 3912) 2020-06-12T02:37:53.371 INFO:tasks.rados.rados.0.smithi107.stdout:3913: done (2 left) 2020-06-12T02:37:53.371 INFO:tasks.rados.rados.0.smithi107.stdout:3916: done (1 left) 2020-06-12T02:37:53.371 INFO:tasks.rados.rados.0.smithi107.stdout:3917: delete oid 15 current snap is 0 2020-06-12T02:37:53.372 INFO:tasks.rados.rados.0.smithi107.stdout:3917: done (1 left) 2020-06-12T02:37:53.372 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:37:53.372 INFO:tasks.rados.rados.0.smithi107.stdout:3918: seq_num 3917 ranges {0=409600,409600=401408} 2020-06-12T02:37:53.372 INFO:tasks.rados.rados.0.smithi107.stdout:3918: writing smithi10729407-15 from 0 to 409600 tid 1 2020-06-12T02:37:53.372 INFO:tasks.rados.rados.0.smithi107.stdout:3918: writing smithi10729407-15 from 409600 to 811008 tid 2 2020-06-12T02:37:53.373 INFO:tasks.rados.rados.0.smithi107.stdout:3919: delete oid 4 current snap is 0 2020-06-12T02:37:53.373 INFO:tasks.rados.rados.0.smithi107.stdout:3915: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:53.373 INFO:tasks.rados.rados.0.smithi107.stdout:3915: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:53.373 INFO:tasks.rados.rados.0.smithi107.stdout:3915: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:53.373 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 400 (ObjNum 3914 snap 0 seq_num 3914) dirty exists 2020-06-12T02:37:53.373 INFO:tasks.rados.rados.0.smithi107.stdout:3915: left oid 2 (ObjNum 3914 snap 0 seq_num 3914) 2020-06-12T02:37:53.374 INFO:tasks.rados.rados.0.smithi107.stdout:3915: done (2 left) 2020-06-12T02:37:53.374 INFO:tasks.rados.rados.0.smithi107.stdout:3919: done (1 left) 2020-06-12T02:37:53.374 INFO:tasks.rados.rados.0.smithi107.stdout:3920: delete oid 2 current snap is 0 2020-06-12T02:37:53.374 INFO:tasks.rados.rados.0.smithi107.stdout:3920: done (1 left) 2020-06-12T02:37:53.374 INFO:tasks.rados.rados.0.smithi107.stdout:3921: delete oid 12 current snap is 0 2020-06-12T02:37:53.375 INFO:tasks.rados.rados.0.smithi107.stdout:3918: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:53.375 INFO:tasks.rados.rados.0.smithi107.stdout:3918: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:53.375 INFO:tasks.rados.rados.0.smithi107.stdout:3921: done (1 left) 2020-06-12T02:37:53.375 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:37:53.375 INFO:tasks.rados.rados.0.smithi107.stdout:3922: seq_num 3921 ranges {0=475136,475136=401408} 2020-06-12T02:37:53.375 INFO:tasks.rados.rados.0.smithi107.stdout:3922: writing smithi10729407-18 from 0 to 475136 tid 1 2020-06-12T02:37:53.376 INFO:tasks.rados.rados.0.smithi107.stdout:3922: writing smithi10729407-18 from 475136 to 876544 tid 2 2020-06-12T02:37:53.376 INFO:tasks.rados.rados.0.smithi107.stdout:3923: delete oid 9 current snap is 0 2020-06-12T02:37:53.376 INFO:tasks.rados.rados.0.smithi107.stdout:3918: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:53.376 INFO:tasks.rados.rados.0.smithi107.stdout:3918: finishing write tid 5 to smithi10729407-15 2020-06-12T02:37:53.376 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 673 (ObjNum 3917 snap 0 seq_num 3917) dirty exists 2020-06-12T02:37:53.376 INFO:tasks.rados.rados.0.smithi107.stdout:3918: left oid 15 (ObjNum 3917 snap 0 seq_num 3917) 2020-06-12T02:37:53.377 INFO:tasks.rados.rados.0.smithi107.stdout:3922: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:53.377 INFO:tasks.rados.rados.0.smithi107.stdout:3918: done (2 left) 2020-06-12T02:37:53.377 INFO:tasks.rados.rados.0.smithi107.stdout:3923: done (1 left) 2020-06-12T02:37:53.377 INFO:tasks.rados.rados.0.smithi107.stdout:3924: delete oid 10 current snap is 0 2020-06-12T02:37:53.377 INFO:tasks.rados.rados.0.smithi107.stdout:3924: done (1 left) 2020-06-12T02:37:53.378 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:37:53.378 INFO:tasks.rados.rados.0.smithi107.stdout:3925: seq_num 3924 ranges {0=524288,524288=401408} 2020-06-12T02:37:53.378 INFO:tasks.rados.rados.0.smithi107.stdout:3925: writing smithi10729407-5 from 0 to 524288 tid 1 2020-06-12T02:37:53.378 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:53.277+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:53.379 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-06-12T02:37:53.384 INFO:tasks.rados.rados.0.smithi107.stdout:3925: writing smithi10729407-5 from 524288 to 925696 tid 2 2020-06-12T02:37:53.385 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:37:53.385 INFO:tasks.rados.rados.0.smithi107.stdout:3926: seq_num 3925 ranges {458752=540672,999424=401408} 2020-06-12T02:37:53.385 INFO:tasks.rados.rados.0.smithi107.stdout:3926: writing smithi10729407-20 from 458752 to 999424 tid 1 2020-06-12T02:37:53.385 INFO:tasks.rados.rados.0.smithi107.stdout:3926: writing smithi10729407-20 from 999424 to 1400832 tid 2 2020-06-12T02:37:53.385 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:37:53.386 INFO:tasks.rados.rados.0.smithi107.stdout:3922: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:53.386 INFO:tasks.rados.rados.0.smithi107.stdout:3927: seq_num 3926 ranges {688128=557056,1245184=401408} 2020-06-12T02:37:53.386 INFO:tasks.rados.rados.0.smithi107.stdout:3927: writing smithi10729407-17 from 688128 to 1245184 tid 1 2020-06-12T02:37:53.386 INFO:tasks.rados.rados.0.smithi107.stdout:3927: writing smithi10729407-17 from 1245184 to 1646592 tid 2 2020-06-12T02:37:53.386 INFO:tasks.rados.rados.0.smithi107.stdout:3928: delete oid 12 current snap is 0 2020-06-12T02:37:53.387 INFO:tasks.rados.rados.0.smithi107.stdout:3922: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:53.387 INFO:tasks.rados.rados.0.smithi107.stdout:3922: finishing write tid 5 to smithi10729407-18 2020-06-12T02:37:53.387 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 689 (ObjNum 3921 snap 0 seq_num 3921) dirty exists 2020-06-12T02:37:53.387 INFO:tasks.rados.rados.0.smithi107.stdout:3922: left oid 18 (ObjNum 3921 snap 0 seq_num 3921) 2020-06-12T02:37:53.388 INFO:tasks.rados.rados.0.smithi107.stdout:3925: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:53.388 INFO:tasks.rados.rados.0.smithi107.stdout:3925: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:53.388 INFO:tasks.rados.rados.0.smithi107.stdout:3925: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:53.388 INFO:tasks.rados.rados.0.smithi107.stdout:3925: finishing write tid 5 to smithi10729407-5 2020-06-12T02:37:53.388 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 382 (ObjNum 3924 snap 0 seq_num 3924) dirty exists 2020-06-12T02:37:53.389 INFO:tasks.rados.rados.0.smithi107.stdout:3925: left oid 5 (ObjNum 3924 snap 0 seq_num 3924) 2020-06-12T02:37:53.389 INFO:tasks.rados.rados.0.smithi107.stdout:3926: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:53.389 INFO:tasks.rados.rados.0.smithi107.stdout:3926: finishing write tid 2 to smithi10729407-20 2020-06-12T02:37:53.389 INFO:tasks.rados.rados.0.smithi107.stdout:3926: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:53.389 INFO:tasks.rados.rados.0.smithi107.stdout:3922: done (4 left) 2020-06-12T02:37:53.389 INFO:tasks.rados.rados.0.smithi107.stdout:3925: done (3 left) 2020-06-12T02:37:53.390 INFO:tasks.rados.rados.0.smithi107.stdout:3928: done (2 left) 2020-06-12T02:37:53.390 INFO:tasks.rados.rados.0.smithi107.stdout:3929: delete oid 12 current snap is 0 2020-06-12T02:37:53.390 INFO:tasks.rados.rados.0.smithi107.stdout:3926: finishing write tid 5 to smithi10729407-20 2020-06-12T02:37:53.390 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 303 (ObjNum 3925 snap 0 seq_num 3925) dirty exists 2020-06-12T02:37:53.390 INFO:tasks.rados.rados.0.smithi107.stdout:3926: left oid 20 (ObjNum 3925 snap 0 seq_num 3925) 2020-06-12T02:37:53.391 INFO:tasks.rados.rados.0.smithi107.stdout:3926: done (2 left) 2020-06-12T02:37:53.391 INFO:tasks.rados.rados.0.smithi107.stdout:3929: done (1 left) 2020-06-12T02:37:53.391 INFO:tasks.rados.rados.0.smithi107.stdout:3930: delete oid 11 current snap is 0 2020-06-12T02:37:53.391 INFO:tasks.rados.rados.0.smithi107.stdout:3927: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:53.392 INFO:tasks.rados.rados.0.smithi107.stdout:3927: finishing write tid 2 to smithi10729407-17 2020-06-12T02:37:53.392 INFO:tasks.rados.rados.0.smithi107.stdout:3927: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:53.392 INFO:tasks.rados.rados.0.smithi107.stdout:3927: finishing write tid 5 to smithi10729407-17 2020-06-12T02:37:53.392 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 717 (ObjNum 3926 snap 0 seq_num 3926) dirty exists 2020-06-12T02:37:53.392 INFO:tasks.rados.rados.0.smithi107.stdout:3927: left oid 17 (ObjNum 3926 snap 0 seq_num 3926) 2020-06-12T02:37:53.392 INFO:tasks.rados.rados.0.smithi107.stdout:3927: done (1 left) 2020-06-12T02:37:53.393 INFO:tasks.rados.rados.0.smithi107.stdout:3930: done (0 left) 2020-06-12T02:37:53.393 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:37:53.393 INFO:tasks.rados.rados.0.smithi107.stdout:3931: seq_num 3930 ranges {876544=622592,1499136=401408} 2020-06-12T02:37:53.393 INFO:tasks.rados.rados.0.smithi107.stdout:3931: writing smithi10729407-18 from 876544 to 1499136 tid 1 2020-06-12T02:37:53.393 INFO:tasks.rados.rados.0.smithi107.stdout:3931: writing smithi10729407-18 from 1499136 to 1900544 tid 2 2020-06-12T02:37:53.394 INFO:tasks.rados.rados.0.smithi107.stdout:3932: delete oid 9 current snap is 0 2020-06-12T02:37:53.394 INFO:tasks.rados.rados.0.smithi107.stdout:3931: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:53.394 INFO:tasks.rados.rados.0.smithi107.stdout:3931: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:53.394 INFO:tasks.rados.rados.0.smithi107.stdout:3931: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:53.394 INFO:tasks.rados.rados.0.smithi107.stdout:3931: finishing write tid 5 to smithi10729407-18 2020-06-12T02:37:53.395 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 692 (ObjNum 3930 snap 0 seq_num 3930) dirty exists 2020-06-12T02:37:53.395 INFO:tasks.rados.rados.0.smithi107.stdout:3931: left oid 18 (ObjNum 3930 snap 0 seq_num 3930) 2020-06-12T02:37:53.395 INFO:tasks.rados.rados.0.smithi107.stdout:3931: done (1 left) 2020-06-12T02:37:53.395 INFO:tasks.rados.rados.0.smithi107.stdout:3932: done (0 left) 2020-06-12T02:37:53.396 INFO:tasks.rados.rados.0.smithi107.stdout:3933: delete oid 13 current snap is 0 2020-06-12T02:37:53.396 INFO:tasks.rados.rados.0.smithi107.stdout:3933: done (0 left) 2020-06-12T02:37:53.396 INFO:tasks.rados.rados.0.smithi107.stdout:3934: delete oid 5 current snap is 0 2020-06-12T02:37:53.396 INFO:tasks.rados.rados.0.smithi107.stdout:3934: done (0 left) 2020-06-12T02:37:53.396 INFO:tasks.rados.rados.0.smithi107.stdout:3935: delete oid 15 current snap is 0 2020-06-12T02:37:53.397 INFO:tasks.rados.rados.0.smithi107.stdout:3935: done (0 left) 2020-06-12T02:37:53.397 INFO:tasks.rados.rados.0.smithi107.stdout:3936: delete oid 16 current snap is 0 2020-06-12T02:37:53.397 INFO:tasks.rados.rados.0.smithi107.stdout:3936: done (0 left) 2020-06-12T02:37:53.397 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:37:53.397 INFO:tasks.rados.rados.0.smithi107.stdout:3937: seq_num 3936 ranges {1384448=729088,2113536=401408} 2020-06-12T02:37:53.397 INFO:tasks.rados.rados.0.smithi107.stdout:3937: writing smithi10729407-7 from 1384448 to 2113536 tid 1 2020-06-12T02:37:53.398 INFO:tasks.rados.rados.0.smithi107.stdout:3937: writing smithi10729407-7 from 2113536 to 2514944 tid 2 2020-06-12T02:37:53.398 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:37:53.398 INFO:tasks.rados.rados.0.smithi107.stdout:3938: seq_num 3937 ranges {0=745472,745472=401408} 2020-06-12T02:37:53.398 INFO:tasks.rados.rados.0.smithi107.stdout:3938: writing smithi10729407-9 from 0 to 745472 tid 1 2020-06-12T02:37:53.398 INFO:tasks.rados.rados.0.smithi107.stdout:3938: writing smithi10729407-9 from 745472 to 1146880 tid 2 2020-06-12T02:37:53.399 INFO:tasks.rados.rados.0.smithi107.stdout:3939: delete oid 13 current snap is 0 2020-06-12T02:37:53.399 INFO:tasks.rados.rados.0.smithi107.stdout:3937: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:53.399 INFO:tasks.rados.rados.0.smithi107.stdout:3937: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:53.399 INFO:tasks.rados.rados.0.smithi107.stdout:3937: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:53.399 INFO:tasks.rados.rados.0.smithi107.stdout:3937: finishing write tid 5 to smithi10729407-7 2020-06-12T02:37:53.399 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 743 (ObjNum 3936 snap 0 seq_num 3936) dirty exists 2020-06-12T02:37:53.400 INFO:tasks.rados.rados.0.smithi107.stdout:3937: left oid 7 (ObjNum 3936 snap 0 seq_num 3936) 2020-06-12T02:37:53.400 INFO:tasks.rados.rados.0.smithi107.stdout:3937: done (2 left) 2020-06-12T02:37:53.400 INFO:tasks.rados.rados.0.smithi107.stdout:3939: done (1 left) 2020-06-12T02:37:53.400 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:37:53.400 INFO:tasks.rados.rados.0.smithi107.stdout:3940: seq_num 3939 ranges {0=778240,778240=401408} 2020-06-12T02:37:53.400 INFO:tasks.rados.rados.0.smithi107.stdout:3940: writing smithi10729407-13 from 0 to 778240 tid 1 2020-06-12T02:37:53.401 INFO:tasks.rados.rados.0.smithi107.stdout:3940: writing smithi10729407-13 from 778240 to 1179648 tid 2 2020-06-12T02:37:53.401 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:37:53.401 INFO:tasks.rados.rados.0.smithi107.stdout:3941: seq_num 3940 ranges {655360=794624,1449984=401408} 2020-06-12T02:37:53.403 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:53.378+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:53.404 INFO:tasks.rados.rados.0.smithi107.stdout:3941: writing smithi10729407-6 from 655360 to 1449984 tid 1 2020-06-12T02:37:53.404 INFO:tasks.rados.rados.0.smithi107.stdout:3941: writing smithi10729407-6 from 1449984 to 1851392 tid 2 2020-06-12T02:37:53.405 INFO:tasks.rados.rados.0.smithi107.stdout:3942: delete oid 5 current snap is 0 2020-06-12T02:37:53.405 INFO:tasks.rados.rados.0.smithi107.stdout:3938: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:53.405 INFO:tasks.rados.rados.0.smithi107.stdout:3938: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:53.405 INFO:tasks.rados.rados.0.smithi107.stdout:3938: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:53.405 INFO:tasks.rados.rados.0.smithi107.stdout:3938: finishing write tid 5 to smithi10729407-9 2020-06-12T02:37:53.405 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 696 (ObjNum 3937 snap 0 seq_num 3937) dirty exists 2020-06-12T02:37:53.406 INFO:tasks.rados.rados.0.smithi107.stdout:3938: left oid 9 (ObjNum 3937 snap 0 seq_num 3937) 2020-06-12T02:37:53.406 INFO:tasks.rados.rados.0.smithi107.stdout:3940: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:53.406 INFO:tasks.rados.rados.0.smithi107.stdout:3940: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:53.406 INFO:tasks.rados.rados.0.smithi107.stdout:3940: finishing write tid 4 to smithi10729407-13 2020-06-12T02:37:53.406 INFO:tasks.rados.rados.0.smithi107.stdout:3940: finishing write tid 5 to smithi10729407-13 2020-06-12T02:37:53.407 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 747 (ObjNum 3939 snap 0 seq_num 3939) dirty exists 2020-06-12T02:37:53.407 INFO:tasks.rados.rados.0.smithi107.stdout:3940: left oid 13 (ObjNum 3939 snap 0 seq_num 3939) 2020-06-12T02:37:53.407 INFO:tasks.rados.rados.0.smithi107.stdout:3938: done (3 left) 2020-06-12T02:37:53.407 INFO:tasks.rados.rados.0.smithi107.stdout:3940: done (2 left) 2020-06-12T02:37:53.407 INFO:tasks.rados.rados.0.smithi107.stdout:3942: done (1 left) 2020-06-12T02:37:53.407 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:37:53.408 INFO:tasks.rados.rados.0.smithi107.stdout:3943: seq_num 3942 ranges {352256=425984,778240=573440,1351680=229376} 2020-06-12T02:37:53.408 INFO:tasks.rados.rados.0.smithi107.stdout:3943: writing smithi10729407-8 from 352256 to 778240 tid 1 2020-06-12T02:37:53.409 INFO:tasks.rados.rados.0.smithi107.stdout:3943: writing smithi10729407-8 from 778240 to 1351680 tid 2 2020-06-12T02:37:53.409 INFO:tasks.rados.rados.0.smithi107.stdout:3943: writing smithi10729407-8 from 1351680 to 1581056 tid 3 2020-06-12T02:37:53.410 INFO:tasks.rados.rados.0.smithi107.stdout:3944: delete oid 17 current snap is 0 2020-06-12T02:37:53.410 INFO:tasks.rados.rados.0.smithi107.stdout:3941: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:53.410 INFO:tasks.rados.rados.0.smithi107.stdout:3941: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:53.410 INFO:tasks.rados.rados.0.smithi107.stdout:3941: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:53.410 INFO:tasks.rados.rados.0.smithi107.stdout:3941: finishing write tid 5 to smithi10729407-6 2020-06-12T02:37:53.411 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 739 (ObjNum 3940 snap 0 seq_num 3940) dirty exists 2020-06-12T02:37:53.411 INFO:tasks.rados.rados.0.smithi107.stdout:3941: left oid 6 (ObjNum 3940 snap 0 seq_num 3940) 2020-06-12T02:37:53.411 INFO:tasks.rados.rados.0.smithi107.stdout:3941: done (2 left) 2020-06-12T02:37:53.411 INFO:tasks.rados.rados.0.smithi107.stdout:3944: done (1 left) 2020-06-12T02:37:53.412 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:37:53.412 INFO:tasks.rados.rados.0.smithi107.stdout:3945: seq_num 3944 ranges {1851392=458752,2310144=745472,3055616=57344} 2020-06-12T02:37:53.418 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd unset noscrub 2020-06-12T02:37:53.422 INFO:tasks.rados.rados.0.smithi107.stdout:3945: writing smithi10729407-6 from 1851392 to 2310144 tid 1 2020-06-12T02:37:53.426 INFO:tasks.rados.rados.0.smithi107.stdout:3945: writing smithi10729407-6 from 2310144 to 3055616 tid 2 2020-06-12T02:37:53.427 INFO:tasks.rados.rados.0.smithi107.stdout:3945: writing smithi10729407-6 from 3055616 to 3112960 tid 3 2020-06-12T02:37:53.428 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:37:53.428 INFO:tasks.rados.rados.0.smithi107.stdout:3946: seq_num 3945 ranges {0=475136,475136=630784,1105920=172032} 2020-06-12T02:37:53.430 INFO:tasks.rados.rados.0.smithi107.stdout:3946: writing smithi10729407-11 from 0 to 475136 tid 1 2020-06-12T02:37:53.434 INFO:tasks.rados.rados.0.smithi107.stdout:3946: writing smithi10729407-11 from 475136 to 1105920 tid 2 2020-06-12T02:37:53.435 INFO:tasks.rados.rados.0.smithi107.stdout:3946: writing smithi10729407-11 from 1105920 to 1277952 tid 3 2020-06-12T02:37:53.435 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:37:53.436 INFO:tasks.rados.rados.0.smithi107.stdout:3947: seq_num 3946 ranges {0=491520,491520=565248,1056768=237568} 2020-06-12T02:37:53.438 INFO:tasks.rados.rados.0.smithi107.stdout:3947: writing smithi10729407-15 from 0 to 491520 tid 1 2020-06-12T02:37:53.443 INFO:tasks.rados.rados.0.smithi107.stdout:3947: writing smithi10729407-15 from 491520 to 1056768 tid 2 2020-06-12T02:37:53.444 INFO:tasks.rados.rados.0.smithi107.stdout:3947: writing smithi10729407-15 from 1056768 to 1294336 tid 3 2020-06-12T02:37:53.444 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:53.445 INFO:tasks.rados.rados.0.smithi107.stdout:3948: seq_num 3947 ranges {0=507904,507904=450560,958464=352256} 2020-06-12T02:37:53.447 INFO:tasks.rados.rados.0.smithi107.stdout:3948: writing smithi10729407-2 from 0 to 507904 tid 1 2020-06-12T02:37:53.450 INFO:tasks.rados.rados.0.smithi107.stdout:3948: writing smithi10729407-2 from 507904 to 958464 tid 2 2020-06-12T02:37:53.453 INFO:tasks.rados.rados.0.smithi107.stdout:3948: writing smithi10729407-2 from 958464 to 1310720 tid 3 2020-06-12T02:37:53.453 INFO:tasks.rados.rados.0.smithi107.stdout:3949: delete oid 5 current snap is 0 2020-06-12T02:37:53.454 INFO:tasks.rados.rados.0.smithi107.stdout:3943: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:53.454 INFO:tasks.rados.rados.0.smithi107.stdout:3943: finishing write tid 2 to smithi10729407-8 2020-06-12T02:37:53.461 INFO:tasks.rados.rados.0.smithi107.stdout:3943: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:53.461 INFO:tasks.rados.rados.0.smithi107.stdout:3943: finishing write tid 5 to smithi10729407-8 2020-06-12T02:37:53.461 INFO:tasks.rados.rados.0.smithi107.stdout:3943: finishing write tid 6 to smithi10729407-8 2020-06-12T02:37:53.462 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 678 (ObjNum 3942 snap 0 seq_num 3942) dirty exists 2020-06-12T02:37:53.462 INFO:tasks.rados.rados.0.smithi107.stdout:3943: left oid 8 (ObjNum 3942 snap 0 seq_num 3942) 2020-06-12T02:37:53.462 INFO:tasks.rados.rados.0.smithi107.stdout:3945: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:53.462 INFO:tasks.rados.rados.0.smithi107.stdout:3945: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:53.462 INFO:tasks.rados.rados.0.smithi107.stdout:3945: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:53.462 INFO:tasks.rados.rados.0.smithi107.stdout:3945: finishing write tid 5 to smithi10729407-6 2020-06-12T02:37:53.463 INFO:tasks.rados.rados.0.smithi107.stdout:3946: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:53.463 INFO:tasks.rados.rados.0.smithi107.stdout:3945: finishing write tid 6 to smithi10729407-6 2020-06-12T02:37:53.463 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 743 (ObjNum 3944 snap 0 seq_num 3944) dirty exists 2020-06-12T02:37:53.463 INFO:tasks.rados.rados.0.smithi107.stdout:3945: left oid 6 (ObjNum 3944 snap 0 seq_num 3944) 2020-06-12T02:37:53.463 INFO:tasks.rados.rados.0.smithi107.stdout:3946: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:53.464 INFO:tasks.rados.rados.0.smithi107.stdout:3946: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:53.464 INFO:tasks.rados.rados.0.smithi107.stdout:3946: finishing write tid 5 to smithi10729407-11 2020-06-12T02:37:53.464 INFO:tasks.rados.rados.0.smithi107.stdout:3947: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:53.464 INFO:tasks.rados.rados.0.smithi107.stdout:3947: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:53.464 INFO:tasks.rados.rados.0.smithi107.stdout:3947: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:53.464 INFO:tasks.rados.rados.0.smithi107.stdout:3947: finishing write tid 5 to smithi10729407-15 2020-06-12T02:37:53.466 INFO:tasks.rados.rados.0.smithi107.stdout:3948: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:53.466 INFO:tasks.rados.rados.0.smithi107.stdout:3946: finishing write tid 6 to smithi10729407-11 2020-06-12T02:37:53.466 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 693 (ObjNum 3945 snap 0 seq_num 3945) dirty exists 2020-06-12T02:37:53.466 INFO:tasks.rados.rados.0.smithi107.stdout:3946: left oid 11 (ObjNum 3945 snap 0 seq_num 3945) 2020-06-12T02:37:53.467 INFO:tasks.rados.rados.0.smithi107.stdout:3947: finishing write tid 6 to smithi10729407-15 2020-06-12T02:37:53.467 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 682 (ObjNum 3946 snap 0 seq_num 3946) dirty exists 2020-06-12T02:37:53.467 INFO:tasks.rados.rados.0.smithi107.stdout:3947: left oid 15 (ObjNum 3946 snap 0 seq_num 3946) 2020-06-12T02:37:53.467 INFO:tasks.rados.rados.0.smithi107.stdout:3943: done (5 left) 2020-06-12T02:37:53.467 INFO:tasks.rados.rados.0.smithi107.stdout:3945: done (4 left) 2020-06-12T02:37:53.467 INFO:tasks.rados.rados.0.smithi107.stdout:3946: done (3 left) 2020-06-12T02:37:53.468 INFO:tasks.rados.rados.0.smithi107.stdout:3947: done (2 left) 2020-06-12T02:37:53.468 INFO:tasks.rados.rados.0.smithi107.stdout:3949: done (1 left) 2020-06-12T02:37:53.468 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:37:53.468 INFO:tasks.rados.rados.0.smithi107.stdout:3950: seq_num 3949 ranges {0=540672,540672=614400,1155072=188416} 2020-06-12T02:37:53.468 INFO:tasks.rados.rados.0.smithi107.stdout:3950: writing smithi10729407-14 from 0 to 540672 tid 1 2020-06-12T02:37:53.469 INFO:tasks.rados.rados.0.smithi107.stdout:3950: writing smithi10729407-14 from 540672 to 1155072 tid 2 2020-06-12T02:37:53.469 INFO:tasks.rados.rados.0.smithi107.stdout:3950: writing smithi10729407-14 from 1155072 to 1343488 tid 3 2020-06-12T02:37:53.469 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:37:53.469 INFO:tasks.rados.rados.0.smithi107.stdout:3951: seq_num 3950 ranges {1179648=557056,1736704=499712,2236416=303104} 2020-06-12T02:37:53.470 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:37:53.463+0000 7f235bf1fec0 -1 osd.7 18 log_to_monitors {default=true} 2020-06-12T02:37:53.475 INFO:tasks.rados.rados.0.smithi107.stdout:3951: writing smithi10729407-13 from 1179648 to 1736704 tid 1 2020-06-12T02:37:53.478 INFO:tasks.rados.rados.0.smithi107.stdout:3951: writing smithi10729407-13 from 1736704 to 2236416 tid 2 2020-06-12T02:37:53.480 INFO:tasks.rados.rados.0.smithi107.stdout:3951: writing smithi10729407-13 from 2236416 to 2539520 tid 3 2020-06-12T02:37:53.480 INFO:tasks.rados.rados.0.smithi107.stdout:3952: delete oid 7 current snap is 0 2020-06-12T02:37:53.480 INFO:tasks.rados.rados.0.smithi107.stdout:3948: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:53.481 INFO:tasks.rados.rados.0.smithi107.stdout:3948: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:53.481 INFO:tasks.rados.rados.0.smithi107.stdout:3948: finishing write tid 5 to smithi10729407-2 2020-06-12T02:37:53.481 INFO:tasks.rados.rados.0.smithi107.stdout:3948: finishing write tid 6 to smithi10729407-2 2020-06-12T02:37:53.481 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 405 (ObjNum 3947 snap 0 seq_num 3947) dirty exists 2020-06-12T02:37:53.481 INFO:tasks.rados.rados.0.smithi107.stdout:3948: left oid 2 (ObjNum 3947 snap 0 seq_num 3947) 2020-06-12T02:37:53.482 INFO:tasks.rados.rados.0.smithi107.stdout:3950: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:53.482 INFO:tasks.rados.rados.0.smithi107.stdout:3950: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:53.482 INFO:tasks.rados.rados.0.smithi107.stdout:3950: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:53.482 INFO:tasks.rados.rados.0.smithi107.stdout:3950: finishing write tid 5 to smithi10729407-14 2020-06-12T02:37:53.482 INFO:tasks.rados.rados.0.smithi107.stdout:3950: finishing write tid 6 to smithi10729407-14 2020-06-12T02:37:53.483 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 697 (ObjNum 3949 snap 0 seq_num 3949) dirty exists 2020-06-12T02:37:53.483 INFO:tasks.rados.rados.0.smithi107.stdout:3950: left oid 14 (ObjNum 3949 snap 0 seq_num 3949) 2020-06-12T02:37:53.485 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:37:53.483+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:37:53.487 INFO:tasks.rados.rados.0.smithi107.stdout:3951: finishing write tid 1 to smithi10729407-13 2020-06-12T02:37:53.488 INFO:tasks.rados.rados.0.smithi107.stdout:3951: finishing write tid 2 to smithi10729407-13 2020-06-12T02:37:53.489 INFO:tasks.rados.rados.0.smithi107.stdout:3951: finishing write tid 3 to smithi10729407-13 2020-06-12T02:37:53.489 INFO:tasks.rados.rados.0.smithi107.stdout:3951: finishing write tid 5 to smithi10729407-13 2020-06-12T02:37:53.490 INFO:tasks.rados.rados.0.smithi107.stdout:3948: done (3 left) 2020-06-12T02:37:53.491 INFO:tasks.rados.rados.0.smithi107.stdout:3950: done (2 left) 2020-06-12T02:37:53.491 INFO:tasks.rados.rados.0.smithi107.stdout:3952: done (1 left) 2020-06-12T02:37:53.491 INFO:tasks.rados.rados.0.smithi107.stdout:3953: delete oid 15 current snap is 0 2020-06-12T02:37:53.492 INFO:tasks.rados.rados.0.smithi107.stdout:3951: finishing write tid 6 to smithi10729407-13 2020-06-12T02:37:53.493 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 751 (ObjNum 3950 snap 0 seq_num 3950) dirty exists 2020-06-12T02:37:53.493 INFO:tasks.rados.rados.0.smithi107.stdout:3951: left oid 13 (ObjNum 3950 snap 0 seq_num 3950) 2020-06-12T02:37:53.496 INFO:tasks.rados.rados.0.smithi107.stdout:3951: done (1 left) 2020-06-12T02:37:53.496 INFO:tasks.rados.rados.0.smithi107.stdout:3953: done (0 left) 2020-06-12T02:37:53.496 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:37:53.496 INFO:tasks.rados.rados.0.smithi107.stdout:3954: seq_num 3953 ranges {0=614400,614400=557056,1171456=245760} 2020-06-12T02:37:53.499 INFO:tasks.rados.rados.0.smithi107.stdout:3954: writing smithi10729407-19 from 0 to 614400 tid 1 2020-06-12T02:37:53.503 INFO:tasks.rados.rados.0.smithi107.stdout:3954: writing smithi10729407-19 from 614400 to 1171456 tid 2 2020-06-12T02:37:53.504 INFO:tasks.rados.rados.0.smithi107.stdout:3954: writing smithi10729407-19 from 1171456 to 1417216 tid 3 2020-06-12T02:37:53.505 INFO:tasks.rados.rados.0.smithi107.stdout:3955: delete oid 3 current snap is 0 2020-06-12T02:37:53.506 INFO:tasks.rados.rados.0.smithi107.stdout:3954: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:53.506 INFO:tasks.rados.rados.0.smithi107.stdout:3955: done (1 left) 2020-06-12T02:37:53.507 INFO:tasks.rados.rados.0.smithi107.stdout:3956: delete oid 16 current snap is 0 2020-06-12T02:37:53.508 INFO:tasks.rados.rados.0.smithi107.stdout:3956: done (1 left) 2020-06-12T02:37:53.508 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:37:53.509 INFO:tasks.rados.rados.0.smithi107.stdout:3957: seq_num 3956 ranges {0=663552,663552=655360,1318912=147456} 2020-06-12T02:37:53.513 INFO:tasks.rados.rados.0.smithi107.stdout:3957: writing smithi10729407-5 from 0 to 663552 tid 1 2020-06-12T02:37:53.517 INFO:tasks.rados.rados.0.smithi107.stdout:3957: writing smithi10729407-5 from 663552 to 1318912 tid 2 2020-06-12T02:37:53.518 INFO:tasks.rados.rados.0.smithi107.stdout:3957: writing smithi10729407-5 from 1318912 to 1466368 tid 3 2020-06-12T02:37:53.519 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:53.519 INFO:tasks.rados.rados.0.smithi107.stdout:3958: seq_num 3957 ranges {1310720=679936,1990656=540672,2531328=262144} 2020-06-12T02:37:53.530 INFO:tasks.rados.rados.0.smithi107.stdout:3958: writing smithi10729407-2 from 1310720 to 1990656 tid 1 2020-06-12T02:37:53.533 INFO:tasks.rados.rados.0.smithi107.stdout:3958: writing smithi10729407-2 from 1990656 to 2531328 tid 2 2020-06-12T02:37:53.535 INFO:tasks.rados.rados.0.smithi107.stdout:3958: writing smithi10729407-2 from 2531328 to 2793472 tid 3 2020-06-12T02:37:53.535 INFO:tasks.rados.rados.0.smithi107.stdout:3959: delete oid 17 current snap is 0 2020-06-12T02:37:53.535 INFO:tasks.rados.rados.0.smithi107.stdout:3954: finishing write tid 2 to smithi10729407-19 2020-06-12T02:37:53.536 INFO:tasks.rados.rados.0.smithi107.stdout:3954: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:53.536 INFO:tasks.rados.rados.0.smithi107.stdout:3954: finishing write tid 5 to smithi10729407-19 2020-06-12T02:37:53.536 INFO:tasks.rados.rados.0.smithi107.stdout:3954: finishing write tid 6 to smithi10729407-19 2020-06-12T02:37:53.536 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 664 (ObjNum 3953 snap 0 seq_num 3953) dirty exists 2020-06-12T02:37:53.536 INFO:tasks.rados.rados.0.smithi107.stdout:3954: left oid 19 (ObjNum 3953 snap 0 seq_num 3953) 2020-06-12T02:37:53.537 INFO:tasks.rados.rados.0.smithi107.stdout:3957: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:53.537 INFO:tasks.rados.rados.0.smithi107.stdout:3957: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:53.537 INFO:tasks.rados.rados.0.smithi107.stdout:3957: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:53.537 INFO:tasks.rados.rados.0.smithi107.stdout:3957: finishing write tid 5 to smithi10729407-5 2020-06-12T02:37:53.537 INFO:tasks.rados.rados.0.smithi107.stdout:3957: finishing write tid 6 to smithi10729407-5 2020-06-12T02:37:53.538 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 389 (ObjNum 3956 snap 0 seq_num 3956) dirty exists 2020-06-12T02:37:53.538 INFO:tasks.rados.rados.0.smithi107.stdout:3957: left oid 5 (ObjNum 3956 snap 0 seq_num 3956) 2020-06-12T02:37:53.538 INFO:tasks.rados.rados.0.smithi107.stdout:3958: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:53.539 INFO:tasks.rados.rados.0.smithi107.stdout:3954: done (3 left) 2020-06-12T02:37:53.539 INFO:tasks.rados.rados.0.smithi107.stdout:3957: done (2 left) 2020-06-12T02:37:53.539 INFO:tasks.rados.rados.0.smithi107.stdout:3959: done (1 left) 2020-06-12T02:37:53.539 INFO:tasks.rados.rados.0.smithi107.stdout:3960: delete oid 17 current snap is 0 2020-06-12T02:37:53.540 INFO:tasks.rados.rados.0.smithi107.stdout:3960: done (1 left) 2020-06-12T02:37:53.540 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:37:53.541 INFO:tasks.rados.rados.0.smithi107.stdout:3961: seq_num 3960 ranges {0=729088,729088=598016,1327104=204800} 2020-06-12T02:37:53.545 INFO:tasks.rados.rados.0.smithi107.stdout:3961: writing smithi10729407-4 from 0 to 729088 tid 1 2020-06-12T02:37:53.549 INFO:tasks.rados.rados.0.smithi107.stdout:3961: writing smithi10729407-4 from 729088 to 1327104 tid 2 2020-06-12T02:37:53.550 INFO:tasks.rados.rados.0.smithi107.stdout:3961: writing smithi10729407-4 from 1327104 to 1531904 tid 3 2020-06-12T02:37:53.551 INFO:tasks.rados.rados.0.smithi107.stdout:3962: delete oid 20 current snap is 0 2020-06-12T02:37:53.551 INFO:tasks.rados.rados.0.smithi107.stdout:3958: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:53.551 INFO:tasks.rados.rados.0.smithi107.stdout:3958: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:53.551 INFO:tasks.rados.rados.0.smithi107.stdout:3958: finishing write tid 5 to smithi10729407-2 2020-06-12T02:37:53.551 INFO:tasks.rados.rados.0.smithi107.stdout:3958: finishing write tid 6 to smithi10729407-2 2020-06-12T02:37:53.551 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 409 (ObjNum 3957 snap 0 seq_num 3957) dirty exists 2020-06-12T02:37:53.552 INFO:tasks.rados.rados.0.smithi107.stdout:3958: left oid 2 (ObjNum 3957 snap 0 seq_num 3957) 2020-06-12T02:37:53.557 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:37:53.557 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:37:53.557 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2020-06-12T02:37:53.558 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:37:53.560 INFO:tasks.rados.rados.0.smithi107.stdout:3958: done (2 left) 2020-06-12T02:37:53.560 INFO:tasks.rados.rados.0.smithi107.stdout:3962: done (1 left) 2020-06-12T02:37:53.561 INFO:tasks.rados.rados.0.smithi107.stdout:3963: delete oid 11 current snap is 0 2020-06-12T02:37:53.575 INFO:tasks.rados.rados.0.smithi107.stdout:3963: done (1 left) 2020-06-12T02:37:53.576 INFO:tasks.rados.rados.0.smithi107.stdout:3964: delete oid 8 current snap is 0 2020-06-12T02:37:53.576 INFO:tasks.rados.rados.0.smithi107.stdout:3964: done (1 left) 2020-06-12T02:37:53.577 INFO:tasks.rados.rados.0.smithi107.stdout:3965: delete oid 18 current snap is 0 2020-06-12T02:37:53.577 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-06-12T02:37:53.622 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:37:53.620+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:37:53.687 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:53.684+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:37:53.694 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:37:53.695 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:37:53.695 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:37:53.695 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:37:53.695 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:37:53.704 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-06-12T02:37:53.787 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:53.785+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:37:53.862 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:37:53.862 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:37:53.862 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:37:53.863 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:37:53.863 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.869 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1206 3.5s0 3:a9510bda:::smithi10729407-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:53.869 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.856688+0000", 2020-06-12T02:37:53.869 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 21.001410363000002, 2020-06-12T02:37:53.870 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019116942000000001, 2020-06-12T02:37:53.870 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:53.870 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:53.870 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:53.870 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:53.871 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:53.871 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1206 2020-06-12T02:37:53.871 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.871 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:53.871 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.872 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:53.872 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856688+0000", 2020-06-12T02:37:53.872 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:53.872 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.872 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.872 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:53.873 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856688+0000", 2020-06-12T02:37:53.873 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4355e-05 2020-06-12T02:37:53.873 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.873 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.873 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:53.874 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856713+0000", 2020-06-12T02:37:53.874 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0850000000000001e-06 2020-06-12T02:37:53.874 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.874 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.874 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:53.875 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856716+0000", 2020-06-12T02:37:53.875 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4509999999999997e-06 2020-06-12T02:37:53.875 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.875 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.875 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:53.876 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856722+0000", 2020-06-12T02:37:53.876 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2879999999999999e-06 2020-06-12T02:37:53.876 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.876 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.876 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:53.877 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856724+0000", 2020-06-12T02:37:53.877 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0057507449999999998 2020-06-12T02:37:53.877 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.877 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.877 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.878 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.862475+0000", 2020-06-12T02:37:53.878 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5759000000000003e-05 2020-06-12T02:37:53.878 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.878 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.878 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:53.879 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.862571+0000", 2020-06-12T02:37:53.879 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012199018000000001 2020-06-12T02:37:53.879 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.879 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.879 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.880 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.874770+0000", 2020-06-12T02:37:53.880 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7260999999999999e-05 2020-06-12T02:37:53.880 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.880 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.880 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:53.880 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.874827+0000", 2020-06-12T02:37:53.881 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00097798000000000004 2020-06-12T02:37:53.881 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.881 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.881 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:53.881 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.875805+0000", 2020-06-12T02:37:53.882 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019116942000000001 2020-06-12T02:37:53.882 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.882 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:53.882 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.882 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.882 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.883 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1572 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~745472] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-06-12T02:37:53.883 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.886439+0000", 2020-06-12T02:37:53.883 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 19.971659644999999, 2020-06-12T02:37:53.883 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021442571000000001, 2020-06-12T02:37:53.883 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:53.884 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:53.884 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:53.884 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:53.884 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:53.884 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1572 2020-06-12T02:37:53.884 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.885 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:53.885 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.885 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:53.885 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.886439+0000", 2020-06-12T02:37:53.885 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:53.886 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.886 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.886 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:53.886 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.886439+0000", 2020-06-12T02:37:53.887 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00073027599999999997 2020-06-12T02:37:53.887 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.887 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.887 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:53.887 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887169+0000", 2020-06-12T02:37:53.888 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.49e-06 2020-06-12T02:37:53.888 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.888 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.888 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:53.888 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887175+0000", 2020-06-12T02:37:53.889 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7899999999999994e-06 2020-06-12T02:37:53.889 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.889 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.889 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:53.889 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887185+0000", 2020-06-12T02:37:53.889 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.42e-06 2020-06-12T02:37:53.890 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.890 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.890 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:53.890 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887189+0000", 2020-06-12T02:37:53.890 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1619999999999999e-05 2020-06-12T02:37:53.891 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.891 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.891 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.891 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887221+0000", 2020-06-12T02:37:53.891 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9720000000000002e-05 2020-06-12T02:37:53.891 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.892 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.892 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:53.892 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887300+0000", 2020-06-12T02:37:53.892 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047354100000000001 2020-06-12T02:37:53.892 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.892 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.893 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:53.893 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887774+0000", 2020-06-12T02:37:53.893 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0030649750000000002 2020-06-12T02:37:53.893 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.893 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.894 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:53.894 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.890839+0000", 2020-06-12T02:37:53.894 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016995321000000001 2020-06-12T02:37:53.894 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.894 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.895 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:53.895 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.907834+0000", 2020-06-12T02:37:53.895 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7417999999999999e-05 2020-06-12T02:37:53.895 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.895 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.895 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:53.896 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.907882+0000", 2020-06-12T02:37:53.896 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021442571000000001 2020-06-12T02:37:53.896 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.897 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:53.897 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.897 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.897 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.897 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1573 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~712704] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-06-12T02:37:53.897 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.890888+0000", 2020-06-12T02:37:53.898 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 19.967210909999999, 2020-06-12T02:37:53.898 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018868001999999998, 2020-06-12T02:37:53.898 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:53.898 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:53.898 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:53.899 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:53.899 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:53.899 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1573 2020-06-12T02:37:53.899 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.899 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:53.899 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.900 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:53.900 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.890888+0000", 2020-06-12T02:37:53.900 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:53.900 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.900 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.901 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:53.901 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.890888+0000", 2020-06-12T02:37:53.901 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000683605 2020-06-12T02:37:53.901 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.901 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.901 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:53.902 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891571+0000", 2020-06-12T02:37:53.902 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4999999999999999e-06 2020-06-12T02:37:53.902 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.902 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.902 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:53.903 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891577+0000", 2020-06-12T02:37:53.903 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.171e-05 2020-06-12T02:37:53.903 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.903 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.903 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:53.903 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891589+0000", 2020-06-12T02:37:53.904 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7650000000000001e-06 2020-06-12T02:37:53.904 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.904 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.904 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:53.904 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891593+0000", 2020-06-12T02:37:53.905 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6689000000000001e-05 2020-06-12T02:37:53.905 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.905 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.905 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.905 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891630+0000", 2020-06-12T02:37:53.906 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9877999999999997e-05 2020-06-12T02:37:53.906 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.906 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.906 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:53.907 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891710+0000", 2020-06-12T02:37:53.908 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053367200000000001 2020-06-12T02:37:53.908 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.908 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.909 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:53.909 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.892244+0000", 2020-06-12T02:37:53.909 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004927196 2020-06-12T02:37:53.909 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.909 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.910 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:53.910 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.897171+0000", 2020-06-12T02:37:53.910 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012496419 2020-06-12T02:37:53.910 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.910 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.911 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:53.911 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.909667+0000", 2020-06-12T02:37:53.911 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8567999999999996e-05 2020-06-12T02:37:53.911 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.911 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.911 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:53.912 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.909756+0000", 2020-06-12T02:37:53.912 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018868001999999998 2020-06-12T02:37:53.912 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.912 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:53.912 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.913 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.913 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.913 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1576 3.5s0 3:a446366d:::smithi10729407-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e26)", 2020-06-12T02:37:53.913 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.891955+0000", 2020-06-12T02:37:53.913 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 19.966143670000001, 2020-06-12T02:37:53.914 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019893246999999999, 2020-06-12T02:37:53.914 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:53.914 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:53.914 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:53.914 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:53.915 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:53.915 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1576 2020-06-12T02:37:53.915 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.915 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:53.915 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.915 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:53.916 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891955+0000", 2020-06-12T02:37:53.916 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:53.916 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.916 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.916 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:53.917 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891955+0000", 2020-06-12T02:37:53.917 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2067999999999999e-05 2020-06-12T02:37:53.917 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.917 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.917 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:53.918 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891977+0000", 2020-06-12T02:37:53.918 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.18e-06 2020-06-12T02:37:53.918 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.918 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.918 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:53.918 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891980+0000", 2020-06-12T02:37:53.919 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3629999999999999e-06 2020-06-12T02:37:53.919 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.919 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.919 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:53.919 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891987+0000", 2020-06-12T02:37:53.920 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.181e-06 2020-06-12T02:37:53.920 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.920 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.920 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:53.920 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891989+0000", 2020-06-12T02:37:53.920 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050959120000000002 2020-06-12T02:37:53.921 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.921 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.921 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.921 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.897085+0000", 2020-06-12T02:37:53.921 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1089999999999998e-05 2020-06-12T02:37:53.922 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.922 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.922 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:53.922 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.897166+0000", 2020-06-12T02:37:53.922 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013400235 2020-06-12T02:37:53.922 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.923 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.923 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.923 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.910566+0000", 2020-06-12T02:37:53.923 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000104981 2020-06-12T02:37:53.923 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.924 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.924 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:53.924 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.910671+0000", 2020-06-12T02:37:53.924 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001177237 2020-06-12T02:37:53.924 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.924 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.925 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:53.925 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.911848+0000", 2020-06-12T02:37:53.925 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019893246999999999 2020-06-12T02:37:53.925 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.925 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:53.926 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.926 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.926 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.926 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2913 3.5s0 3:a9510bda:::smithi10729407-14:head [append 0~770048,stat] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:53.926 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.070354+0000", 2020-06-12T02:37:53.927 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 16.787744869000001, 2020-06-12T02:37:53.927 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60690798599999995, 2020-06-12T02:37:53.928 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:53.928 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:53.928 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:53.928 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:53.928 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:53.929 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2913 2020-06-12T02:37:53.929 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.929 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:53.929 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.929 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:53.930 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.070354+0000", 2020-06-12T02:37:53.930 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:53.930 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.930 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.931 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:53.931 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.070354+0000", 2020-06-12T02:37:53.931 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00098478900000000002 2020-06-12T02:37:53.931 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.931 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.932 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:53.932 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071339+0000", 2020-06-12T02:37:53.932 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8349999999999993e-06 2020-06-12T02:37:53.932 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.932 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.932 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:53.933 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071347+0000", 2020-06-12T02:37:53.933 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0749e-05 2020-06-12T02:37:53.933 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.933 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.933 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:53.934 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071358+0000", 2020-06-12T02:37:53.934 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9369999999999997e-06 2020-06-12T02:37:53.934 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.934 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.934 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:53.934 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071362+0000", 2020-06-12T02:37:53.935 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0426e-05 2020-06-12T02:37:53.935 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.935 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.935 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.936 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071423+0000", 2020-06-12T02:37:53.936 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4129999999999998e-05 2020-06-12T02:37:53.936 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.936 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.936 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:53.937 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071487+0000", 2020-06-12T02:37:53.937 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053752599999999998 2020-06-12T02:37:53.937 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.937 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.937 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:53.937 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072024+0000", 2020-06-12T02:37:53.938 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036162579999999998 2020-06-12T02:37:53.938 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.938 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.938 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:53.938 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075640+0000", 2020-06-12T02:37:53.939 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60152201999999999 2020-06-12T02:37:53.939 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.939 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.939 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:53.939 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677162+0000", 2020-06-12T02:37:53.939 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9315999999999996e-05 2020-06-12T02:37:53.940 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.940 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.940 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:53.940 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677262+0000", 2020-06-12T02:37:53.940 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60690798599999995 2020-06-12T02:37:53.941 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.941 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:53.941 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.942 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.942 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.942 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2914 3.5s0 3:a9510bda:::smithi10729407-14:head [append 0~401408] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:53.942 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.072409+0000", 2020-06-12T02:37:53.942 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 16.785689425000001, 2020-06-12T02:37:53.943 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.61956801100000003, 2020-06-12T02:37:53.943 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:53.943 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:53.943 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:53.943 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:53.943 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:53.944 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2914 2020-06-12T02:37:53.944 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.944 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:53.944 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.944 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:53.945 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072409+0000", 2020-06-12T02:37:53.946 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:53.946 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.946 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.946 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:53.946 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072409+0000", 2020-06-12T02:37:53.947 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000446317 2020-06-12T02:37:53.947 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.947 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.947 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:53.947 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072856+0000", 2020-06-12T02:37:53.948 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2140000000000004e-06 2020-06-12T02:37:53.948 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.948 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.948 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:53.948 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072861+0000", 2020-06-12T02:37:53.948 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8710000000000003e-06 2020-06-12T02:37:53.949 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.949 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.949 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:53.949 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072870+0000", 2020-06-12T02:37:53.949 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2270000000000004e-06 2020-06-12T02:37:53.950 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.950 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.950 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:53.950 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072874+0000", 2020-06-12T02:37:53.950 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024649960000000001 2020-06-12T02:37:53.950 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.951 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.951 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.951 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075339+0000", 2020-06-12T02:37:53.951 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9179999999999996e-05 2020-06-12T02:37:53.951 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.952 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.952 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:53.952 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075438+0000", 2020-06-12T02:37:53.952 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60189281400000005 2020-06-12T02:37:53.952 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.952 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.953 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.953 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677331+0000", 2020-06-12T02:37:53.953 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012602800000000001 2020-06-12T02:37:53.953 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.953 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.954 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:53.954 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677457+0000", 2020-06-12T02:37:53.954 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00088167699999999998 2020-06-12T02:37:53.954 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.954 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.954 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:53.955 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.678339+0000", 2020-06-12T02:37:53.955 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004312995 2020-06-12T02:37:53.955 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.955 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.955 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:53.956 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.682652+0000", 2020-06-12T02:37:53.956 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0092869140000000003 2020-06-12T02:37:53.956 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.956 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.956 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:53.956 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691938+0000", 2020-06-12T02:37:53.957 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8778e-05 2020-06-12T02:37:53.957 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.957 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.957 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:53.957 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691977+0000", 2020-06-12T02:37:53.958 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.61956801100000003 2020-06-12T02:37:53.958 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.958 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:53.958 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.958 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.958 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.959 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2915 3.5s0 3:a9510bda:::smithi10729407-14:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:53.959 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.072912+0000", 2020-06-12T02:37:53.959 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 16.785186684999999, 2020-06-12T02:37:53.959 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.6192204, 2020-06-12T02:37:53.959 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:53.959 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:53.960 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:53.960 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:53.960 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:53.960 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2915 2020-06-12T02:37:53.960 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.961 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:53.961 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.961 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:53.961 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072912+0000", 2020-06-12T02:37:53.961 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:53.961 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.962 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.962 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:53.962 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072912+0000", 2020-06-12T02:37:53.962 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9201e-05 2020-06-12T02:37:53.962 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.963 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.963 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:53.963 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072951+0000", 2020-06-12T02:37:53.963 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5480000000000002e-06 2020-06-12T02:37:53.963 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.963 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.964 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:53.964 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072955+0000", 2020-06-12T02:37:53.964 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3879999999999997e-06 2020-06-12T02:37:53.964 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.964 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.965 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:53.965 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072961+0000", 2020-06-12T02:37:53.965 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6589999999999999e-06 2020-06-12T02:37:53.965 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.965 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.965 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:53.966 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072964+0000", 2020-06-12T02:37:53.966 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025221039999999998 2020-06-12T02:37:53.966 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.966 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.966 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.967 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075486+0000", 2020-06-12T02:37:53.967 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5038999999999998e-05 2020-06-12T02:37:53.967 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.967 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.967 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:53.967 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075561+0000", 2020-06-12T02:37:53.968 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60452397499999999 2020-06-12T02:37:53.968 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.968 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.968 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.968 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680085+0000", 2020-06-12T02:37:53.969 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1398999999999999e-05 2020-06-12T02:37:53.969 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.969 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.969 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:53.969 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680156+0000", 2020-06-12T02:37:53.969 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000274465 2020-06-12T02:37:53.970 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.970 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.970 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:53.970 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680431+0000", 2020-06-12T02:37:53.970 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025071999999999998 2020-06-12T02:37:53.970 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.971 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.971 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:53.971 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.682938+0000", 2020-06-12T02:37:53.971 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0091116449999999998 2020-06-12T02:37:53.971 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.972 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.972 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:53.972 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692050+0000", 2020-06-12T02:37:53.972 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2776999999999997e-05 2020-06-12T02:37:53.972 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.972 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.973 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:53.973 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692132+0000", 2020-06-12T02:37:53.973 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.6192204 2020-06-12T02:37:53.973 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.973 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:53.974 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.974 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.974 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.974 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2916 3.5s0 3:a9510bda:::smithi10729407-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:37:53.974 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.072986+0000", 2020-06-12T02:37:53.974 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 16.785112445999999, 2020-06-12T02:37:53.975 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.62063941099999997, 2020-06-12T02:37:53.975 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:53.975 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:53.975 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:53.975 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:53.976 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:53.976 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2916 2020-06-12T02:37:53.976 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.976 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:53.976 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.976 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:53.977 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072986+0000", 2020-06-12T02:37:53.977 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:53.977 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.977 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.977 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:53.977 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072986+0000", 2020-06-12T02:37:53.978 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2101000000000001e-05 2020-06-12T02:37:53.978 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.978 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.978 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:53.978 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073008+0000", 2020-06-12T02:37:53.979 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1549999999999999e-06 2020-06-12T02:37:53.979 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.979 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.979 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:53.979 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073011+0000", 2020-06-12T02:37:53.980 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2360000000000001e-06 2020-06-12T02:37:53.980 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.980 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.980 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:53.981 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073018+0000", 2020-06-12T02:37:53.981 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2790000000000001e-06 2020-06-12T02:37:53.981 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.981 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.981 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:53.981 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073020+0000", 2020-06-12T02:37:53.982 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025637670000000002 2020-06-12T02:37:53.982 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.982 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.982 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.982 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075584+0000", 2020-06-12T02:37:53.983 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1814999999999998e-05 2020-06-12T02:37:53.983 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.983 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.983 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:53.983 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075636+0000", 2020-06-12T02:37:53.983 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60499943199999995 2020-06-12T02:37:53.984 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.984 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.984 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.984 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680635+0000", 2020-06-12T02:37:53.984 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0968000000000002e-05 2020-06-12T02:37:53.985 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.985 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.985 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:53.985 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680706+0000", 2020-06-12T02:37:53.985 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011777101999999999 2020-06-12T02:37:53.985 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.986 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.986 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.986 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692483+0000", 2020-06-12T02:37:53.986 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5890999999999995e-05 2020-06-12T02:37:53.986 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.987 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.987 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:53.987 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692569+0000", 2020-06-12T02:37:53.987 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001056665 2020-06-12T02:37:53.987 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.987 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.988 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:53.988 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.693626+0000", 2020-06-12T02:37:53.988 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.62063941099999997 2020-06-12T02:37:53.988 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.988 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:53.989 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:53.989 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.989 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.989 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2922 3.ds1 3:b983fd28:::smithi10729407-1:head [append 0~466944] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:53.989 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.097192+0000", 2020-06-12T02:37:53.989 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 16.760907078999999, 2020-06-12T02:37:53.990 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58672644699999998, 2020-06-12T02:37:53.990 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:53.990 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:53.990 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:53.991 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:53.991 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:53.991 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2922 2020-06-12T02:37:53.991 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.992 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:53.992 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.992 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:53.992 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097192+0000", 2020-06-12T02:37:53.992 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:53.992 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.993 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.993 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:53.993 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097192+0000", 2020-06-12T02:37:53.993 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047206100000000002 2020-06-12T02:37:53.993 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.994 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.994 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:53.994 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097664+0000", 2020-06-12T02:37:53.994 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8359999999999999e-06 2020-06-12T02:37:53.994 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.995 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.995 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:53.995 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097668+0000", 2020-06-12T02:37:53.995 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0815999999999999e-05 2020-06-12T02:37:53.995 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.995 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.996 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:53.996 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097679+0000", 2020-06-12T02:37:53.996 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.028e-06 2020-06-12T02:37:53.996 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.996 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.996 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:53.997 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097684+0000", 2020-06-12T02:37:53.997 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3932999999999999e-05 2020-06-12T02:37:53.997 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.997 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.997 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:53.998 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097748+0000", 2020-06-12T02:37:53.998 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2214999999999999e-05 2020-06-12T02:37:53.998 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.998 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.998 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:53.998 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097820+0000", 2020-06-12T02:37:53.999 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00041583800000000003 2020-06-12T02:37:53.999 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:53.999 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:53.999 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:53.999 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.098236+0000", 2020-06-12T02:37:54.000 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0020339960000000002 2020-06-12T02:37:54.000 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.000 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.000 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:54.000 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.100270+0000", 2020-06-12T02:37:54.000 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58355657299999997 2020-06-12T02:37:54.001 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.001 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.001 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:54.002 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.683827+0000", 2020-06-12T02:37:54.002 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1150999999999995e-05 2020-06-12T02:37:54.002 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.002 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.002 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:54.003 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.683918+0000", 2020-06-12T02:37:54.003 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58672644699999998 2020-06-12T02:37:54.003 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:54.003 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:54.003 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:54.004 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.004 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.004 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2923 3.ds1 3:b983fd28:::smithi10729407-1:head [append 0~614400] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:54.004 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.100504+0000", 2020-06-12T02:37:54.004 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 16.757594438000002, 2020-06-12T02:37:54.004 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58853946999999995, 2020-06-12T02:37:54.005 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:54.005 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:54.005 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:54.005 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:54.005 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:54.006 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2923 2020-06-12T02:37:54.006 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.006 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:54.006 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.006 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:54.006 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.100504+0000", 2020-06-12T02:37:54.007 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:54.007 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.007 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.007 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:54.007 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.100504+0000", 2020-06-12T02:37:54.007 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00068395399999999996 2020-06-12T02:37:54.008 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.008 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.008 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:54.008 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101188+0000", 2020-06-12T02:37:54.008 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4050000000000003e-06 2020-06-12T02:37:54.009 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.009 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.009 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:54.009 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101196+0000", 2020-06-12T02:37:54.009 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.328e-05 2020-06-12T02:37:54.009 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.010 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.010 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:54.010 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101209+0000", 2020-06-12T02:37:54.010 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3419999999999996e-06 2020-06-12T02:37:54.010 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.011 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.011 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:54.011 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101215+0000", 2020-06-12T02:37:54.011 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1389000000000005e-05 2020-06-12T02:37:54.011 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.011 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.012 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:54.012 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101277+0000", 2020-06-12T02:37:54.012 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7724000000000004e-05 2020-06-12T02:37:54.012 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.012 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.012 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:54.013 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101344+0000", 2020-06-12T02:37:54.013 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047852499999999999 2020-06-12T02:37:54.013 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.013 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.013 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:54.014 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101823+0000", 2020-06-12T02:37:54.014 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0039255510000000002 2020-06-12T02:37:54.014 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.014 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.014 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:54.014 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105748+0000", 2020-06-12T02:37:54.015 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58321340099999996 2020-06-12T02:37:54.015 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.015 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.015 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:54.015 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.688962+0000", 2020-06-12T02:37:54.016 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1898999999999996e-05 2020-06-12T02:37:54.016 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.016 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.016 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:54.016 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689044+0000", 2020-06-12T02:37:54.016 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58853946999999995 2020-06-12T02:37:54.017 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:54.017 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:54.017 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:54.017 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.017 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.018 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2924 3.ds1 3:b983fd28:::smithi10729407-1:head [append 0~188416] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:54.018 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.101633+0000", 2020-06-12T02:37:54.018 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 16.756465444, 2020-06-12T02:37:54.018 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58758259199999996, 2020-06-12T02:37:54.018 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:54.018 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:54.019 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:54.019 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:54.019 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:54.019 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2924 2020-06-12T02:37:54.019 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.020 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:54.020 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.020 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:54.020 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101633+0000", 2020-06-12T02:37:54.020 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:54.020 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.021 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.021 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:54.021 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101633+0000", 2020-06-12T02:37:54.021 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00018775399999999999 2020-06-12T02:37:54.021 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.022 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.022 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:54.022 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101821+0000", 2020-06-12T02:37:54.022 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7669999999999998e-06 2020-06-12T02:37:54.022 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.022 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.023 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:54.023 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101828+0000", 2020-06-12T02:37:54.023 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2716e-05 2020-06-12T02:37:54.023 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.023 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.024 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:54.024 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101840+0000", 2020-06-12T02:37:54.024 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.215e-06 2020-06-12T02:37:54.024 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.024 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.024 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:54.025 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101846+0000", 2020-06-12T02:37:54.025 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021214250000000001 2020-06-12T02:37:54.025 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.025 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.025 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:54.025 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.103967+0000", 2020-06-12T02:37:54.026 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1018000000000003e-05 2020-06-12T02:37:54.026 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.026 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.026 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:54.026 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.104038+0000", 2020-06-12T02:37:54.027 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000333364 2020-06-12T02:37:54.027 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.027 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.027 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:54.027 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.104371+0000", 2020-06-12T02:37:54.027 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001378546 2020-06-12T02:37:54.028 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.028 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.028 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:54.028 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105750+0000", 2020-06-12T02:37:54.028 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58338272899999999 2020-06-12T02:37:54.029 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.029 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.029 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:54.029 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689133+0000", 2020-06-12T02:37:54.029 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3058000000000003e-05 2020-06-12T02:37:54.029 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.030 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.030 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:54.030 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689216+0000", 2020-06-12T02:37:54.030 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58758259199999996 2020-06-12T02:37:54.030 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:54.030 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:54.031 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:54.031 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.031 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.031 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2925 3.ds1 3:b983fd28:::smithi10729407-1:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:54.031 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.101899+0000", 2020-06-12T02:37:54.032 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 16.756199778999999, 2020-06-12T02:37:54.032 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58747810899999997, 2020-06-12T02:37:54.032 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:54.032 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:54.032 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:54.032 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:54.033 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:54.033 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2925 2020-06-12T02:37:54.033 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.033 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:54.033 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.034 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:54.034 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101899+0000", 2020-06-12T02:37:54.034 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:54.034 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.034 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.034 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:54.035 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101899+0000", 2020-06-12T02:37:54.035 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2906999999999998e-05 2020-06-12T02:37:54.035 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.035 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.036 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:54.036 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101972+0000", 2020-06-12T02:37:54.036 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1789999999999996e-06 2020-06-12T02:37:54.036 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.036 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.036 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:54.037 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101978+0000", 2020-06-12T02:37:54.037 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1358e-05 2020-06-12T02:37:54.037 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.037 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.037 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:54.038 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101989+0000", 2020-06-12T02:37:54.038 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.031e-06 2020-06-12T02:37:54.038 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.038 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.038 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:54.038 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101993+0000", 2020-06-12T02:37:54.039 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033218990000000001 2020-06-12T02:37:54.039 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.040 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.040 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:54.040 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105315+0000", 2020-06-12T02:37:54.040 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6324999999999999e-05 2020-06-12T02:37:54.041 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.041 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.041 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:54.041 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105382+0000", 2020-06-12T02:37:54.041 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000216624 2020-06-12T02:37:54.042 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.042 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.042 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:54.042 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105598+0000", 2020-06-12T02:37:54.042 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001236586 2020-06-12T02:37:54.043 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.043 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.043 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:54.043 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.106835+0000", 2020-06-12T02:37:54.043 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58245509200000001 2020-06-12T02:37:54.044 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.044 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.044 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:54.044 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689290+0000", 2020-06-12T02:37:54.044 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7107999999999993e-05 2020-06-12T02:37:54.044 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.045 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.045 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:54.045 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689377+0000", 2020-06-12T02:37:54.045 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58747810899999997 2020-06-12T02:37:54.045 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:54.046 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:54.046 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:54.046 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.046 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.046 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2926 3.ds1 3:b983fd28:::smithi10729407-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:37:54.046 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.102036+0000", 2020-06-12T02:37:54.047 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 16.756062794999998, 2020-06-12T02:37:54.047 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58868048699999997, 2020-06-12T02:37:54.047 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:54.047 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:54.047 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:54.048 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:54.048 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:54.048 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2926 2020-06-12T02:37:54.048 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.048 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:54.048 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.049 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:54.049 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102036+0000", 2020-06-12T02:37:54.049 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:54.049 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.049 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.050 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:54.050 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102036+0000", 2020-06-12T02:37:54.050 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3786999999999999e-05 2020-06-12T02:37:54.050 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.050 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.050 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:54.051 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102080+0000", 2020-06-12T02:37:54.051 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7660000000000004e-06 2020-06-12T02:37:54.051 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.051 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.051 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:54.052 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102085+0000", 2020-06-12T02:37:54.052 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3145e-05 2020-06-12T02:37:54.052 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.052 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.052 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:54.053 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102099+0000", 2020-06-12T02:37:54.053 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2440000000000004e-06 2020-06-12T02:37:54.053 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.053 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.053 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:54.053 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102103+0000", 2020-06-12T02:37:54.054 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036826990000000002 2020-06-12T02:37:54.054 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.054 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.054 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:54.054 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105786+0000", 2020-06-12T02:37:54.055 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.6774000000000006e-05 2020-06-12T02:37:54.055 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.055 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.055 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:54.055 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105882+0000", 2020-06-12T02:37:54.055 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58352084900000001 2020-06-12T02:37:54.056 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.056 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.056 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:54.056 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689403+0000", 2020-06-12T02:37:54.056 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3788999999999995e-05 2020-06-12T02:37:54.057 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.057 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.057 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:54.057 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689467+0000", 2020-06-12T02:37:54.057 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001249434 2020-06-12T02:37:54.057 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.058 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.058 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:54.058 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690716+0000", 2020-06-12T02:37:54.058 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58868048699999997 2020-06-12T02:37:54.058 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:54.059 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:54.059 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:54.059 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.059 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.059 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2950 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~598016] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:54.059 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.162562+0000", 2020-06-12T02:37:54.060 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 16.695536421, 2020-06-12T02:37:54.060 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52780611399999999, 2020-06-12T02:37:54.060 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:54.060 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:54.060 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:54.061 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:54.061 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:54.061 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2950 2020-06-12T02:37:54.061 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.061 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:54.061 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.062 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:54.062 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.162562+0000", 2020-06-12T02:37:54.062 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:54.062 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.062 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.063 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:54.063 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.162562+0000", 2020-06-12T02:37:54.064 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00070718099999999998 2020-06-12T02:37:54.064 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.064 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.064 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:54.064 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163269+0000", 2020-06-12T02:37:54.065 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2029999999999997e-06 2020-06-12T02:37:54.065 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.065 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.065 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:54.065 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163276+0000", 2020-06-12T02:37:54.065 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2734999999999999e-05 2020-06-12T02:37:54.066 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.066 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.066 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:54.066 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163288+0000", 2020-06-12T02:37:54.066 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.404e-06 2020-06-12T02:37:54.067 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.067 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.067 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:54.067 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163295+0000", 2020-06-12T02:37:54.067 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5784000000000006e-05 2020-06-12T02:37:54.067 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.068 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.068 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:54.068 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163361+0000", 2020-06-12T02:37:54.068 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5318e-05 2020-06-12T02:37:54.068 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.068 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.069 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:54.069 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163436+0000", 2020-06-12T02:37:54.069 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054801100000000005 2020-06-12T02:37:54.069 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.069 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.070 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:54.070 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163984+0000", 2020-06-12T02:37:54.070 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.007023077 2020-06-12T02:37:54.070 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.070 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.070 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:54.071 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171007+0000", 2020-06-12T02:37:54.071 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51932051199999996 2020-06-12T02:37:54.071 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.071 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.071 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:54.072 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690327+0000", 2020-06-12T02:37:54.072 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0889000000000001e-05 2020-06-12T02:37:54.072 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.072 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.072 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:54.072 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690368+0000", 2020-06-12T02:37:54.073 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52780611399999999 2020-06-12T02:37:54.073 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:54.073 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:54.073 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:54.073 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.074 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.074 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2951 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~532480] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:54.075 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.165470+0000", 2020-06-12T02:37:54.075 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 16.692628812999999, 2020-06-12T02:37:54.075 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52507917900000001, 2020-06-12T02:37:54.075 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:54.075 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:54.076 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:54.076 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:54.076 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:54.076 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2951 2020-06-12T02:37:54.076 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.077 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:54.077 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.077 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:54.077 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.165470+0000", 2020-06-12T02:37:54.077 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:54.077 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.078 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.078 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:54.078 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.165470+0000", 2020-06-12T02:37:54.078 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00062231600000000004 2020-06-12T02:37:54.078 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.078 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.079 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:54.079 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166092+0000", 2020-06-12T02:37:54.079 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1610000000000002e-06 2020-06-12T02:37:54.079 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.079 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.080 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:54.080 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166097+0000", 2020-06-12T02:37:54.080 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7780000000000002e-06 2020-06-12T02:37:54.080 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.080 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.080 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:54.081 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166107+0000", 2020-06-12T02:37:54.081 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7169999999999999e-06 2020-06-12T02:37:54.081 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.081 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.081 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:54.082 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166112+0000", 2020-06-12T02:37:54.082 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4792000000000002e-05 2020-06-12T02:37:54.082 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.082 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.082 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:54.082 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166147+0000", 2020-06-12T02:37:54.083 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9336e-05 2020-06-12T02:37:54.083 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.083 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.083 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:54.083 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166236+0000", 2020-06-12T02:37:54.084 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00067747200000000004 2020-06-12T02:37:54.084 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.084 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.084 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:54.084 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166913+0000", 2020-06-12T02:37:54.085 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0047718669999999999 2020-06-12T02:37:54.085 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.085 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.085 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:54.085 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171685+0000", 2020-06-12T02:37:54.086 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51881990700000002 2020-06-12T02:37:54.086 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.086 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.086 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:54.086 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690505+0000", 2020-06-12T02:37:54.086 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3832999999999999e-05 2020-06-12T02:37:54.087 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.087 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.087 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:54.087 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690549+0000", 2020-06-12T02:37:54.087 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52507917900000001 2020-06-12T02:37:54.087 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:54.088 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:54.088 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:54.088 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.088 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.088 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2952 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~270336] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:54.089 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.166891+0000", 2020-06-12T02:37:54.089 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 16.691207164000001, 2020-06-12T02:37:54.089 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52426417000000003, 2020-06-12T02:37:54.089 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:54.089 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:54.090 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:54.090 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:54.090 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:54.090 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2952 2020-06-12T02:37:54.090 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.090 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:54.091 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.091 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:54.091 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166891+0000", 2020-06-12T02:37:54.091 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:54.091 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.092 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.092 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:54.092 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166891+0000", 2020-06-12T02:37:54.093 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00031878599999999999 2020-06-12T02:37:54.093 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.093 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.093 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:54.094 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167210+0000", 2020-06-12T02:37:54.094 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2360000000000004e-06 2020-06-12T02:37:54.094 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.094 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.094 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:54.095 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167219+0000", 2020-06-12T02:37:54.095 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3298e-05 2020-06-12T02:37:54.095 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.095 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.095 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:54.096 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167232+0000", 2020-06-12T02:37:54.096 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5849999999999999e-06 2020-06-12T02:37:54.096 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.096 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.096 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:54.096 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167237+0000", 2020-06-12T02:37:54.097 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0018755740000000001 2020-06-12T02:37:54.097 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.097 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.097 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:54.097 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.169113+0000", 2020-06-12T02:37:54.098 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000104768 2020-06-12T02:37:54.098 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.098 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.098 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:54.098 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.169218+0000", 2020-06-12T02:37:54.099 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00057389799999999999 2020-06-12T02:37:54.099 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.099 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.099 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:54.099 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.169792+0000", 2020-06-12T02:37:54.100 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0019333989999999999 2020-06-12T02:37:54.100 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.101 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.101 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:54.101 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171725+0000", 2020-06-12T02:37:54.101 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51938508699999997 2020-06-12T02:37:54.101 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.102 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.102 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:54.102 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691110+0000", 2020-06-12T02:37:54.102 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5538999999999999e-05 2020-06-12T02:37:54.102 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.103 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.103 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:54.103 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691156+0000", 2020-06-12T02:37:54.103 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52426417000000003 2020-06-12T02:37:54.103 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:54.103 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:54.104 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:54.104 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.104 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.104 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2953 3.5s0 3:a446366d:::smithi10729407-11:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:54.104 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.167293+0000", 2020-06-12T02:37:54.105 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 16.690805717, 2020-06-12T02:37:54.105 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52428350999999995, 2020-06-12T02:37:54.105 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:54.105 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:54.105 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:54.106 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:54.106 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:54.106 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2953 2020-06-12T02:37:54.106 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.106 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:54.106 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.107 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:54.107 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167293+0000", 2020-06-12T02:37:54.107 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:54.107 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.108 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.108 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:54.108 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167293+0000", 2020-06-12T02:37:54.108 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3880000000000004e-05 2020-06-12T02:37:54.108 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.109 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.109 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:54.109 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167367+0000", 2020-06-12T02:37:54.109 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.319e-06 2020-06-12T02:37:54.109 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.109 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.110 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:54.110 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167373+0000", 2020-06-12T02:37:54.110 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1719000000000001e-05 2020-06-12T02:37:54.110 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.110 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.111 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:54.111 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167385+0000", 2020-06-12T02:37:54.111 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4580000000000003e-06 2020-06-12T02:37:54.111 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.111 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.111 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:54.112 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167389+0000", 2020-06-12T02:37:54.112 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036341110000000002 2020-06-12T02:37:54.112 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.112 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.112 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:54.113 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171023+0000", 2020-06-12T02:37:54.113 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9418000000000005e-05 2020-06-12T02:37:54.113 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.113 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.113 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:54.113 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171093+0000", 2020-06-12T02:37:54.114 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00029087900000000002 2020-06-12T02:37:54.114 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.114 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.114 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:54.114 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171384+0000", 2020-06-12T02:37:54.115 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000489347 2020-06-12T02:37:54.115 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.115 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.115 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:54.115 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171873+0000", 2020-06-12T02:37:54.115 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51963691300000003 2020-06-12T02:37:54.116 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.116 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.116 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:54.116 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691510+0000", 2020-06-12T02:37:54.116 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6465999999999997e-05 2020-06-12T02:37:54.117 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.117 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.117 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:54.117 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691576+0000", 2020-06-12T02:37:54.117 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52428350999999995 2020-06-12T02:37:54.117 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:54.118 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:54.118 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:54.118 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.118 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.118 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2954 3.5s0 3:a446366d:::smithi10729407-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:37:54.119 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.167432+0000", 2020-06-12T02:37:54.119 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 16.690666967999999, 2020-06-12T02:37:54.119 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52568652699999996, 2020-06-12T02:37:54.119 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:54.119 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:54.119 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:54.120 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:54.120 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:54.120 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2954 2020-06-12T02:37:54.120 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.120 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:54.121 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.121 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:54.121 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167432+0000", 2020-06-12T02:37:54.121 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:54.121 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.121 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.122 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:54.122 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167432+0000", 2020-06-12T02:37:54.122 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3556000000000001e-05 2020-06-12T02:37:54.122 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.122 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.123 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:54.123 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167475+0000", 2020-06-12T02:37:54.123 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7350000000000001e-06 2020-06-12T02:37:54.123 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.123 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.124 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:54.124 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167481+0000", 2020-06-12T02:37:54.124 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3128e-05 2020-06-12T02:37:54.124 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.124 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.125 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:54.125 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167494+0000", 2020-06-12T02:37:54.125 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1579999999999998e-06 2020-06-12T02:37:54.125 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.125 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.126 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:54.126 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167498+0000", 2020-06-12T02:37:54.126 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004082174 2020-06-12T02:37:54.126 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.126 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.126 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:54.127 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171580+0000", 2020-06-12T02:37:54.127 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8954000000000002e-05 2020-06-12T02:37:54.127 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.127 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.127 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:54.128 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171679+0000", 2020-06-12T02:37:54.128 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51993235100000001 2020-06-12T02:37:54.128 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.128 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.128 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:54.128 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691612+0000", 2020-06-12T02:37:54.129 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4329000000000001e-05 2020-06-12T02:37:54.129 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.129 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.129 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:54.129 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691666+0000", 2020-06-12T02:37:54.130 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001452142 2020-06-12T02:37:54.130 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.130 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.130 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:54.130 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.693118+0000", 2020-06-12T02:37:54.130 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52568652699999996 2020-06-12T02:37:54.131 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:54.131 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:54.131 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:54.131 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.131 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.132 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2959 3.ds1 3:b3445631:::smithi10729407-6:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e27)", 2020-06-12T02:37:54.132 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.176248+0000", 2020-06-12T02:37:54.132 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 16.681850227000002, 2020-06-12T02:37:54.132 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51549877099999997, 2020-06-12T02:37:54.132 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:54.132 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:54.133 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:54.133 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:54.133 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:54.133 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2959 2020-06-12T02:37:54.134 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.134 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:54.134 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.134 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:54.134 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176248+0000", 2020-06-12T02:37:54.135 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:54.135 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.135 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.135 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:54.135 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176248+0000", 2020-06-12T02:37:54.135 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5120999999999999e-05 2020-06-12T02:37:54.136 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.136 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.136 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:54.136 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176274+0000", 2020-06-12T02:37:54.136 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9189999999999998e-06 2020-06-12T02:37:54.137 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.137 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.137 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:54.137 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176278+0000", 2020-06-12T02:37:54.137 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6339e-05 2020-06-12T02:37:54.137 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.138 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.138 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:54.138 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176305+0000", 2020-06-12T02:37:54.138 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9089999999999999e-06 2020-06-12T02:37:54.138 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.139 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.139 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:54.139 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176310+0000", 2020-06-12T02:37:54.139 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6095999999999999e-05 2020-06-12T02:37:54.139 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.139 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.140 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:54.140 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176376+0000", 2020-06-12T02:37:54.140 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8590999999999994e-05 2020-06-12T02:37:54.140 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.140 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.141 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:54.141 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176444+0000", 2020-06-12T02:37:54.141 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00024346400000000001 2020-06-12T02:37:54.141 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.141 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.141 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:54.142 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176688+0000", 2020-06-12T02:37:54.142 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00044220899999999999 2020-06-12T02:37:54.142 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.142 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.142 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:54.142 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.177130+0000", 2020-06-12T02:37:54.143 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51452984400000001 2020-06-12T02:37:54.143 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.143 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.143 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:54.143 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691660+0000", 2020-06-12T02:37:54.144 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7279000000000002e-05 2020-06-12T02:37:54.144 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.144 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.144 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:54.144 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691747+0000", 2020-06-12T02:37:54.144 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51549877099999997 2020-06-12T02:37:54.145 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:54.145 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:54.145 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:54.145 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.145 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.146 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:3032 3.ds1 3:b983fd28:::smithi10729407-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:37:54.146 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.864289+0000", 2020-06-12T02:37:54.146 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 15.993809309, 2020-06-12T02:37:54.146 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019572216999999999, 2020-06-12T02:37:54.146 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:54.146 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:54.147 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:54.147 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:54.147 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:54.147 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 3032 2020-06-12T02:37:54.147 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.148 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:54.148 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.148 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:54.148 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864289+0000", 2020-06-12T02:37:54.148 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:54.148 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.149 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.149 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:54.149 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864289+0000", 2020-06-12T02:37:54.149 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5701999999999997e-05 2020-06-12T02:37:54.149 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.150 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.150 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:54.150 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864335+0000", 2020-06-12T02:37:54.150 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0910000000000004e-06 2020-06-12T02:37:54.150 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.150 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.151 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:54.151 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864342+0000", 2020-06-12T02:37:54.151 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2301e-05 2020-06-12T02:37:54.151 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.151 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.151 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:54.152 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864354+0000", 2020-06-12T02:37:54.152 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5199999999999997e-06 2020-06-12T02:37:54.152 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.152 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.153 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:54.153 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864360+0000", 2020-06-12T02:37:54.153 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00049180999999999997 2020-06-12T02:37:54.153 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.153 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.153 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:54.154 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864852+0000", 2020-06-12T02:37:54.154 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6725000000000003e-05 2020-06-12T02:37:54.154 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.154 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.154 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:54.155 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864908+0000", 2020-06-12T02:37:54.155 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0089687010000000008 2020-06-12T02:37:54.155 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.155 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.155 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:54.155 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.873877+0000", 2020-06-12T02:37:54.156 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3382999999999999e-05 2020-06-12T02:37:54.156 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.156 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.156 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:54.157 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.873941+0000", 2020-06-12T02:37:54.157 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.006408927 2020-06-12T02:37:54.157 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.157 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.157 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:54.158 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.880350+0000", 2020-06-12T02:37:54.158 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3905e-05 2020-06-12T02:37:54.158 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.158 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.158 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:54.158 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.880403+0000", 2020-06-12T02:37:54.159 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0034581519999999999 2020-06-12T02:37:54.159 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:54.159 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:54.159 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:54.159 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.883862+0000", 2020-06-12T02:37:54.160 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019572216999999999 2020-06-12T02:37:54.160 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:54.160 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:54.160 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:54.160 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:54.160 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:54.161 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:37:54.162 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-06-12T02:37:54.163 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:54.159+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:37:54.263 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:54.261+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:54.320 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:37:54.321 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:37:54.321 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2020-06-12T02:37:54.321 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:37:54.329 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-06-12T02:37:54.364 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:54.362+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:54.383 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-06-12T02:37:54.467 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:54.464+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:37:54.489 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:37:54.490 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:37:54.490 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:37:54.490 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:37:54.491 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:37:54.494 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-06-12T02:37:54.527 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:37:54.547 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:37:54.558 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:54.554+0000 7f4626716700 -1 osd.5 35 build_incremental_map_msg missing incremental map 35 2020-06-12T02:37:54.558 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:54.554+0000 7f4626716700 -1 osd.5 35 build_incremental_map_msg missing incremental map 35 2020-06-12T02:37:54.560 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2020-06-12T02:37:54.568 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:54.566+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:54.572 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:37:54.592 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:37:54.613 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:37:54.633 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:37:54.653 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:37:54.659 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:54.659 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:54.668 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-06-12T02:37:54.673 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2020-06-12T02:37:54.675 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-06-12T02:37:54.675 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-06-12T02:37:54.675 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-06-12T02:37:54.675 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-06-12T02:37:54.676 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-06-12T02:37:54.676 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-06-12T02:37:54.676 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-06-12T02:37:54.676 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-06-12T02:37:54.708 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:54.706+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:37:54.770 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:54.768+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:54.785 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:54.785 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:54.802 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-06-12T02:37:54.872 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:37:54.870+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:37:54.921 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:54.921 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:54.935 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-06-12T02:37:54.973 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:37:54.971+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:37:55.074 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:37:55.072+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:37:55.091 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:55.091 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:55.104 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-06-12T02:37:55.175 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:55.173+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:55.261 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:55.262 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:55.270 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-06-12T02:37:55.314 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:55.312+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:55.378 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:55.376+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:55.387 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:55.387 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:55.396 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-06-12T02:37:55.479 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:55.476+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:37:55.556 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:55.557 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:55.567 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-06-12T02:37:55.579 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:55.577+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:55.589 INFO:tasks.rados.rados.0.smithi107.stdout:3961: finishing write tid 1 to smithi10729407-4 2020-06-12T02:37:55.595 INFO:tasks.rados.rados.0.smithi107.stdout:3965: done (1 left) 2020-06-12T02:37:55.595 INFO:tasks.rados.rados.0.smithi107.stdout:3966: delete oid 17 current snap is 0 2020-06-12T02:37:55.598 INFO:tasks.rados.rados.0.smithi107.stdout:3961: finishing write tid 2 to smithi10729407-4 2020-06-12T02:37:55.598 INFO:tasks.rados.rados.0.smithi107.stdout:3961: finishing write tid 3 to smithi10729407-4 2020-06-12T02:37:55.598 INFO:tasks.rados.rados.0.smithi107.stdout:3961: finishing write tid 5 to smithi10729407-4 2020-06-12T02:37:55.600 INFO:tasks.rados.rados.0.smithi107.stdout:3961: finishing write tid 6 to smithi10729407-4 2020-06-12T02:37:55.601 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 724 (ObjNum 3960 snap 0 seq_num 3960) dirty exists 2020-06-12T02:37:55.601 INFO:tasks.rados.rados.0.smithi107.stdout:3961: left oid 4 (ObjNum 3960 snap 0 seq_num 3960) 2020-06-12T02:37:55.605 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:55.603+0000 7f4626716700 -1 osd.5 36 build_incremental_map_msg missing incremental map 35 2020-06-12T02:37:55.606 INFO:tasks.rados.rados.0.smithi107.stdout:3961: done (1 left) 2020-06-12T02:37:55.606 INFO:tasks.rados.rados.0.smithi107.stdout:3966: done (0 left) 2020-06-12T02:37:55.607 INFO:tasks.rados.rados.0.smithi107.stdout:3967: delete oid 10 current snap is 0 2020-06-12T02:37:55.609 INFO:tasks.rados.rados.0.smithi107.stdout:3967: done (0 left) 2020-06-12T02:37:55.610 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:37:55.610 INFO:tasks.rados.rados.0.smithi107.stdout:3968: seq_num 3967 ranges {1146880=442368,1589248=630784,2220032=573440} 2020-06-12T02:37:55.618 INFO:tasks.rados.rados.0.smithi107.stdout:3968: writing smithi10729407-9 from 1146880 to 1589248 tid 1 2020-06-12T02:37:55.622 INFO:tasks.rados.rados.0.smithi107.stdout:3968: writing smithi10729407-9 from 1589248 to 2220032 tid 2 2020-06-12T02:37:55.627 INFO:tasks.rados.rados.0.smithi107.stdout:3968: writing smithi10729407-9 from 2220032 to 2793472 tid 3 2020-06-12T02:37:55.627 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:37:55.627 INFO:tasks.rados.rados.0.smithi107.stdout:3969: seq_num 3968 ranges {598016=458752,1056768=516096,1572864=499712,2072576=188416} 2020-06-12T02:37:55.633 INFO:tasks.rados.rados.0.smithi107.stdout:3969: writing smithi10729407-1 from 598016 to 1056768 tid 1 2020-06-12T02:37:55.637 INFO:tasks.rados.rados.0.smithi107.stdout:3969: writing smithi10729407-1 from 1056768 to 1572864 tid 2 2020-06-12T02:37:55.640 INFO:tasks.rados.rados.0.smithi107.stdout:3969: writing smithi10729407-1 from 1572864 to 2072576 tid 3 2020-06-12T02:37:55.641 INFO:tasks.rados.rados.0.smithi107.stdout:3969: writing smithi10729407-1 from 2072576 to 2260992 tid 4 2020-06-12T02:37:55.641 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:37:55.642 INFO:tasks.rados.rados.0.smithi107.stdout:3970: seq_num 3969 ranges {1466368=475136,1941504=450560,2392064=655360,3047424=98304} 2020-06-12T02:37:55.652 INFO:tasks.rados.rados.0.smithi107.stdout:3970: writing smithi10729407-5 from 1466368 to 1941504 tid 1 2020-06-12T02:37:55.654 INFO:tasks.rados.rados.0.smithi107.stdout:3970: writing smithi10729407-5 from 1941504 to 2392064 tid 2 2020-06-12T02:37:55.659 INFO:tasks.rados.rados.0.smithi107.stdout:3970: writing smithi10729407-5 from 2392064 to 3047424 tid 3 2020-06-12T02:37:55.660 INFO:tasks.rados.rados.0.smithi107.stdout:3970: writing smithi10729407-5 from 3047424 to 3145728 tid 4 2020-06-12T02:37:55.660 INFO:tasks.rados.rados.0.smithi107.stdout:3971: delete oid 12 current snap is 0 2020-06-12T02:37:55.660 INFO:tasks.rados.rados.0.smithi107.stdout:3968: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:55.661 INFO:tasks.rados.rados.0.smithi107.stdout:3968: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:55.661 INFO:tasks.rados.rados.0.smithi107.stdout:3968: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:55.661 INFO:tasks.rados.rados.0.smithi107.stdout:3968: finishing write tid 5 to smithi10729407-9 2020-06-12T02:37:55.661 INFO:tasks.rados.rados.0.smithi107.stdout:3968: finishing write tid 6 to smithi10729407-9 2020-06-12T02:37:55.661 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 701 (ObjNum 3967 snap 0 seq_num 3967) dirty exists 2020-06-12T02:37:55.661 INFO:tasks.rados.rados.0.smithi107.stdout:3968: left oid 9 (ObjNum 3967 snap 0 seq_num 3967) 2020-06-12T02:37:55.662 INFO:tasks.rados.rados.0.smithi107.stdout:3969: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:55.662 INFO:tasks.rados.rados.0.smithi107.stdout:3969: finishing write tid 2 to smithi10729407-1 2020-06-12T02:37:55.662 INFO:tasks.rados.rados.0.smithi107.stdout:3969: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:55.664 INFO:tasks.rados.rados.0.smithi107.stdout:3969: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:55.665 INFO:tasks.rados.rados.0.smithi107.stdout:3969: finishing write tid 6 to smithi10729407-1 2020-06-12T02:37:55.667 INFO:tasks.rados.rados.0.smithi107.stdout:3969: finishing write tid 7 to smithi10729407-1 2020-06-12T02:37:55.667 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 748 (ObjNum 3968 snap 0 seq_num 3968) dirty exists 2020-06-12T02:37:55.667 INFO:tasks.rados.rados.0.smithi107.stdout:3969: left oid 1 (ObjNum 3968 snap 0 seq_num 3968) 2020-06-12T02:37:55.670 INFO:tasks.rados.rados.0.smithi107.stdout:3970: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:55.671 INFO:tasks.rados.rados.0.smithi107.stdout:3970: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:55.671 INFO:tasks.rados.rados.0.smithi107.stdout:3968: done (3 left) 2020-06-12T02:37:55.672 INFO:tasks.rados.rados.0.smithi107.stdout:3969: done (2 left) 2020-06-12T02:37:55.672 INFO:tasks.rados.rados.0.smithi107.stdout:3971: done (1 left) 2020-06-12T02:37:55.672 INFO:tasks.rados.rados.0.smithi107.stdout:3972: delete oid 19 current snap is 0 2020-06-12T02:37:55.672 INFO:tasks.rados.rados.0.smithi107.stdout:3970: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:55.672 INFO:tasks.rados.rados.0.smithi107.stdout:3970: finishing write tid 4 to smithi10729407-5 2020-06-12T02:37:55.673 INFO:tasks.rados.rados.0.smithi107.stdout:3970: finishing write tid 6 to smithi10729407-5 2020-06-12T02:37:55.674 INFO:tasks.rados.rados.0.smithi107.stdout:3970: finishing write tid 7 to smithi10729407-5 2020-06-12T02:37:55.674 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 394 (ObjNum 3969 snap 0 seq_num 3969) dirty exists 2020-06-12T02:37:55.674 INFO:tasks.rados.rados.0.smithi107.stdout:3970: left oid 5 (ObjNum 3969 snap 0 seq_num 3969) 2020-06-12T02:37:55.676 INFO:tasks.rados.rados.0.smithi107.stdout:3970: done (1 left) 2020-06-12T02:37:55.676 INFO:tasks.rados.rados.0.smithi107.stdout:3972: done (0 left) 2020-06-12T02:37:55.676 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:37:55.677 INFO:tasks.rados.rados.0.smithi107.stdout:3973: seq_num 3972 ranges {0=532480,532480=507904,1040384=483328,1523712=212992} 2020-06-12T02:37:55.679 INFO:tasks.rados.rados.0.smithi107.stdout:3973: writing smithi10729407-11 from 0 to 532480 tid 1 2020-06-12T02:37:55.680 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:55.677+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:37:55.683 INFO:tasks.rados.rados.0.smithi107.stdout:3973: writing smithi10729407-11 from 532480 to 1040384 tid 2 2020-06-12T02:37:55.686 INFO:tasks.rados.rados.0.smithi107.stdout:3973: writing smithi10729407-11 from 1040384 to 1523712 tid 3 2020-06-12T02:37:55.688 INFO:tasks.rados.rados.0.smithi107.stdout:3973: writing smithi10729407-11 from 1523712 to 1736704 tid 4 2020-06-12T02:37:55.689 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:37:55.689 INFO:tasks.rados.rados.0.smithi107.stdout:3974: seq_num 3973 ranges {3145728=548864,3694592=794624,4489216=409600} 2020-06-12T02:37:55.715 INFO:tasks.rados.rados.0.smithi107.stdout:3974: writing smithi10729407-5 from 3145728 to 3694592 tid 1 2020-06-12T02:37:55.715 INFO:tasks.rados.rados.0.smithi107.stdout:3974: writing smithi10729407-5 from 3694592 to 4489216 tid 2 2020-06-12T02:37:55.717 INFO:tasks.rados.rados.0.smithi107.stdout:3974: writing smithi10729407-5 from 4489216 to 4898816 tid 3 2020-06-12T02:37:55.717 INFO:tasks.rados.rados.0.smithi107.stdout:3975: delete oid 17 current snap is 0 2020-06-12T02:37:55.718 INFO:tasks.rados.rados.0.smithi107.stdout:3973: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:55.718 INFO:tasks.rados.rados.0.smithi107.stdout:3973: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:55.718 INFO:tasks.rados.rados.0.smithi107.stdout:3973: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:55.718 INFO:tasks.rados.rados.0.smithi107.stdout:3973: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:55.718 INFO:tasks.rados.rados.0.smithi107.stdout:3973: finishing write tid 6 to smithi10729407-11 2020-06-12T02:37:55.719 INFO:tasks.rados.rados.0.smithi107.stdout:3973: finishing write tid 7 to smithi10729407-11 2020-06-12T02:37:55.719 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 703 (ObjNum 3972 snap 0 seq_num 3972) dirty exists 2020-06-12T02:37:55.719 INFO:tasks.rados.rados.0.smithi107.stdout:3973: left oid 11 (ObjNum 3972 snap 0 seq_num 3972) 2020-06-12T02:37:55.719 INFO:tasks.rados.rados.0.smithi107.stdout:3974: finishing write tid 1 to smithi10729407-5 2020-06-12T02:37:55.720 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:55.720 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:55.720 INFO:tasks.rados.rados.0.smithi107.stdout:3973: done (2 left) 2020-06-12T02:37:55.721 INFO:tasks.rados.rados.0.smithi107.stdout:3975: done (1 left) 2020-06-12T02:37:55.721 INFO:tasks.rados.rados.0.smithi107.stdout:3976: delete oid 9 current snap is 0 2020-06-12T02:37:55.724 INFO:tasks.rados.rados.0.smithi107.stdout:3976: done (1 left) 2020-06-12T02:37:55.724 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:37:55.724 INFO:tasks.rados.rados.0.smithi107.stdout:3977: seq_num 3976 ranges {2793472=598016,3391488=442368,3833856=458752,4292608=303104} 2020-06-12T02:37:55.730 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-06-12T02:37:55.742 INFO:tasks.rados.rados.0.smithi107.stdout:3977: writing smithi10729407-2 from 2793472 to 3391488 tid 1 2020-06-12T02:37:55.751 INFO:tasks.rados.rados.0.smithi107.stdout:3977: writing smithi10729407-2 from 3391488 to 3833856 tid 2 2020-06-12T02:37:55.752 INFO:tasks.rados.rados.0.smithi107.stdout:3977: writing smithi10729407-2 from 3833856 to 4292608 tid 3 2020-06-12T02:37:55.752 INFO:tasks.rados.rados.0.smithi107.stdout:3977: writing smithi10729407-2 from 4292608 to 4595712 tid 4 2020-06-12T02:37:55.752 INFO:tasks.rados.rados.0.smithi107.stdout:3978: delete oid 20 current snap is 0 2020-06-12T02:37:55.752 INFO:tasks.rados.rados.0.smithi107.stdout:3974: finishing write tid 2 to smithi10729407-5 2020-06-12T02:37:55.752 INFO:tasks.rados.rados.0.smithi107.stdout:3974: finishing write tid 3 to smithi10729407-5 2020-06-12T02:37:55.753 INFO:tasks.rados.rados.0.smithi107.stdout:3974: finishing write tid 5 to smithi10729407-5 2020-06-12T02:37:55.753 INFO:tasks.rados.rados.0.smithi107.stdout:3974: finishing write tid 6 to smithi10729407-5 2020-06-12T02:37:55.753 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 398 (ObjNum 3973 snap 0 seq_num 3973) dirty exists 2020-06-12T02:37:55.753 INFO:tasks.rados.rados.0.smithi107.stdout:3974: left oid 5 (ObjNum 3973 snap 0 seq_num 3973) 2020-06-12T02:37:55.753 INFO:tasks.rados.rados.0.smithi107.stdout:3977: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:55.754 INFO:tasks.rados.rados.0.smithi107.stdout:3974: done (2 left) 2020-06-12T02:37:55.754 INFO:tasks.rados.rados.0.smithi107.stdout:3978: done (1 left) 2020-06-12T02:37:55.754 INFO:tasks.rados.rados.0.smithi107.stdout:3979: delete oid 16 current snap is 0 2020-06-12T02:37:55.756 INFO:tasks.rados.rados.0.smithi107.stdout:3977: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:55.756 INFO:tasks.rados.rados.0.smithi107.stdout:3979: done (1 left) 2020-06-12T02:37:55.756 INFO:tasks.rados.rados.0.smithi107.stdout:3980: delete oid 11 current snap is 0 2020-06-12T02:37:55.759 INFO:tasks.rados.rados.0.smithi107.stdout:3980: done (1 left) 2020-06-12T02:37:55.759 INFO:tasks.rados.rados.0.smithi107.stdout:3981: delete oid 9 current snap is 0 2020-06-12T02:37:55.760 INFO:tasks.rados.rados.0.smithi107.stdout:3977: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:55.761 INFO:tasks.rados.rados.0.smithi107.stdout:3977: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:55.762 INFO:tasks.rados.rados.0.smithi107.stdout:3981: done (1 left) 2020-06-12T02:37:55.762 INFO:tasks.rados.rados.0.smithi107.stdout:3982: delete oid 16 current snap is 0 2020-06-12T02:37:55.764 INFO:tasks.rados.rados.0.smithi107.stdout:3977: finishing write tid 6 to smithi10729407-2 2020-06-12T02:37:55.764 INFO:tasks.rados.rados.0.smithi107.stdout:3982: done (1 left) 2020-06-12T02:37:55.765 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:37:55.765 INFO:tasks.rados.rados.0.smithi107.stdout:3983: seq_num 3982 ranges {3112960=696320,3809280=598016,4407296=409600,4816896=196608} 2020-06-12T02:37:55.780 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:55.778+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:55.785 INFO:tasks.rados.rados.0.smithi107.stdout:3983: writing smithi10729407-6 from 3112960 to 3809280 tid 1 2020-06-12T02:37:55.789 INFO:tasks.rados.rados.0.smithi107.stdout:3983: writing smithi10729407-6 from 3809280 to 4407296 tid 2 2020-06-12T02:37:55.791 INFO:tasks.rados.rados.0.smithi107.stdout:3983: writing smithi10729407-6 from 4407296 to 4816896 tid 3 2020-06-12T02:37:55.793 INFO:tasks.rados.rados.0.smithi107.stdout:3983: writing smithi10729407-6 from 4816896 to 5013504 tid 4 2020-06-12T02:37:55.793 INFO:tasks.rados.rados.0.smithi107.stdout:3984: delete oid 15 current snap is 0 2020-06-12T02:37:55.793 INFO:tasks.rados.rados.0.smithi107.stdout:3977: finishing write tid 7 to smithi10729407-2 2020-06-12T02:37:55.793 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 414 (ObjNum 3976 snap 0 seq_num 3976) dirty exists 2020-06-12T02:37:55.793 INFO:tasks.rados.rados.0.smithi107.stdout:3977: left oid 2 (ObjNum 3976 snap 0 seq_num 3976) 2020-06-12T02:37:55.795 INFO:tasks.rados.rados.0.smithi107.stdout:3977: done (2 left) 2020-06-12T02:37:55.795 INFO:tasks.rados.rados.0.smithi107.stdout:3984: done (1 left) 2020-06-12T02:37:55.796 INFO:tasks.rados.rados.0.smithi107.stdout:3985: delete oid 16 current snap is 0 2020-06-12T02:37:55.798 INFO:tasks.rados.rados.0.smithi107.stdout:3985: done (1 left) 2020-06-12T02:37:55.798 INFO:tasks.rados.rados.0.smithi107.stdout:3986: delete oid 11 current snap is 0 2020-06-12T02:37:55.799 INFO:tasks.rados.rados.0.smithi107.stdout:3983: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:55.799 INFO:tasks.rados.rados.0.smithi107.stdout:3983: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:55.799 INFO:tasks.rados.rados.0.smithi107.stdout:3983: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:55.801 INFO:tasks.rados.rados.0.smithi107.stdout:3983: finishing write tid 4 to smithi10729407-6 2020-06-12T02:37:55.802 INFO:tasks.rados.rados.0.smithi107.stdout:3986: done (1 left) 2020-06-12T02:37:55.802 INFO:tasks.rados.rados.0.smithi107.stdout:3987: delete oid 1 current snap is 0 2020-06-12T02:37:55.802 INFO:tasks.rados.rados.0.smithi107.stdout:3983: finishing write tid 6 to smithi10729407-6 2020-06-12T02:37:55.804 INFO:tasks.rados.rados.0.smithi107.stdout:3983: finishing write tid 7 to smithi10729407-6 2020-06-12T02:37:55.804 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 753 (ObjNum 3982 snap 0 seq_num 3982) dirty exists 2020-06-12T02:37:55.804 INFO:tasks.rados.rados.0.smithi107.stdout:3983: left oid 6 (ObjNum 3982 snap 0 seq_num 3982) 2020-06-12T02:37:55.807 INFO:tasks.rados.rados.0.smithi107.stdout:3983: done (1 left) 2020-06-12T02:37:55.807 INFO:tasks.rados.rados.0.smithi107.stdout:3987: done (0 left) 2020-06-12T02:37:55.807 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:37:55.808 INFO:tasks.rados.rados.0.smithi107.stdout:3988: seq_num 3987 ranges {4595712=778240,5373952=466944,5840896=598016,6438912=139264} 2020-06-12T02:37:55.836 INFO:tasks.rados.rados.0.smithi107.stdout:3988: writing smithi10729407-2 from 4595712 to 5373952 tid 1 2020-06-12T02:37:55.840 INFO:tasks.rados.rados.0.smithi107.stdout:3988: writing smithi10729407-2 from 5373952 to 5840896 tid 2 2020-06-12T02:37:55.842 INFO:tasks.rados.rados.0.smithi107.stdout:3988: writing smithi10729407-2 from 5840896 to 6438912 tid 3 2020-06-12T02:37:55.844 INFO:tasks.rados.rados.0.smithi107.stdout:3988: writing smithi10729407-2 from 6438912 to 6578176 tid 4 2020-06-12T02:37:55.844 INFO:tasks.rados.rados.0.smithi107.stdout:3989: delete oid 13 current snap is 0 2020-06-12T02:37:55.846 INFO:tasks.rados.rados.0.smithi107.stdout:3989: done (1 left) 2020-06-12T02:37:55.847 INFO:tasks.rados.rados.0.smithi107.stdout:3990: delete oid 20 current snap is 0 2020-06-12T02:37:55.849 INFO:tasks.rados.rados.0.smithi107.stdout:3990: done (1 left) 2020-06-12T02:37:55.849 INFO:tasks.rados.rados.0.smithi107.stdout:3991: delete oid 8 current snap is 0 2020-06-12T02:37:55.871 INFO:tasks.rados.rados.0.smithi107.stdout:3988: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:55.877 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:55.877 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:55.881 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:55.880+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:55.886 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-06-12T02:37:55.915 INFO:tasks.rados.rados.0.smithi107.stdout:3991: done (1 left) 2020-06-12T02:37:55.916 INFO:tasks.rados.rados.0.smithi107.stdout:3992: delete oid 16 current snap is 0 2020-06-12T02:37:55.916 INFO:tasks.rados.rados.0.smithi107.stdout:3988: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:55.916 INFO:tasks.rados.rados.0.smithi107.stdout:3988: finishing write tid 3 to smithi10729407-2 2020-06-12T02:37:55.917 INFO:tasks.rados.rados.0.smithi107.stdout:3988: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:55.917 INFO:tasks.rados.rados.0.smithi107.stdout:3988: finishing write tid 6 to smithi10729407-2 2020-06-12T02:37:55.917 INFO:tasks.rados.rados.0.smithi107.stdout:3992: done (1 left) 2020-06-12T02:37:55.918 INFO:tasks.rados.rados.0.smithi107.stdout:3993: delete oid 9 current snap is 0 2020-06-12T02:37:55.918 INFO:tasks.rados.rados.0.smithi107.stdout:3988: finishing write tid 7 to smithi10729407-2 2020-06-12T02:37:55.918 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 419 (ObjNum 3987 snap 0 seq_num 3987) dirty exists 2020-06-12T02:37:55.919 INFO:tasks.rados.rados.0.smithi107.stdout:3988: left oid 2 (ObjNum 3987 snap 0 seq_num 3987) 2020-06-12T02:37:55.920 INFO:tasks.rados.rados.0.smithi107.stdout:3988: done (1 left) 2020-06-12T02:37:55.920 INFO:tasks.rados.rados.0.smithi107.stdout:3993: done (0 left) 2020-06-12T02:37:55.921 INFO:tasks.rados.rados.0.smithi107.stdout:3994: delete oid 1 current snap is 0 2020-06-12T02:37:55.921 INFO:tasks.rados.rados.0.smithi107.stdout:3994: done (0 left) 2020-06-12T02:37:55.922 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:37:55.923 INFO:tasks.rados.rados.0.smithi107.stdout:3995: seq_num 3994 ranges {0=499712,499712=507904,1007616=745472,1753088=352256} 2020-06-12T02:37:55.924 INFO:tasks.rados.rados.0.smithi107.stdout:3995: writing smithi10729407-10 from 0 to 499712 tid 1 2020-06-12T02:37:55.926 INFO:tasks.rados.rados.0.smithi107.stdout:3995: writing smithi10729407-10 from 499712 to 1007616 tid 2 2020-06-12T02:37:55.931 INFO:tasks.rados.rados.0.smithi107.stdout:3995: writing smithi10729407-10 from 1007616 to 1753088 tid 3 2020-06-12T02:37:55.933 INFO:tasks.rados.rados.0.smithi107.stdout:3995: writing smithi10729407-10 from 1753088 to 2105344 tid 4 2020-06-12T02:37:55.934 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:37:55.934 INFO:tasks.rados.rados.0.smithi107.stdout:3996: seq_num 3995 ranges {0=516096,516096=794624,1310720=557056,1867776=253952} 2020-06-12T02:37:55.936 INFO:tasks.rados.rados.0.smithi107.stdout:3996: writing smithi10729407-7 from 0 to 516096 tid 1 2020-06-12T02:37:55.942 INFO:tasks.rados.rados.0.smithi107.stdout:3996: writing smithi10729407-7 from 516096 to 1310720 tid 2 2020-06-12T02:37:55.945 INFO:tasks.rados.rados.0.smithi107.stdout:3996: writing smithi10729407-7 from 1310720 to 1867776 tid 3 2020-06-12T02:37:55.947 INFO:tasks.rados.rados.0.smithi107.stdout:3996: writing smithi10729407-7 from 1867776 to 2121728 tid 4 2020-06-12T02:37:55.948 INFO:tasks.rados.rados.0.smithi107.stdout:3997: delete oid 8 current snap is 0 2020-06-12T02:37:55.948 INFO:tasks.rados.rados.0.smithi107.stdout:3995: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:55.948 INFO:tasks.rados.rados.0.smithi107.stdout:3995: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:55.948 INFO:tasks.rados.rados.0.smithi107.stdout:3995: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:55.948 INFO:tasks.rados.rados.0.smithi107.stdout:3995: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:55.949 INFO:tasks.rados.rados.0.smithi107.stdout:3996: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:55.949 INFO:tasks.rados.rados.0.smithi107.stdout:3995: finishing write tid 6 to smithi10729407-10 2020-06-12T02:37:55.949 INFO:tasks.rados.rados.0.smithi107.stdout:3995: finishing write tid 7 to smithi10729407-10 2020-06-12T02:37:55.949 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 388 (ObjNum 3994 snap 0 seq_num 3994) dirty exists 2020-06-12T02:37:55.949 INFO:tasks.rados.rados.0.smithi107.stdout:3995: left oid 10 (ObjNum 3994 snap 0 seq_num 3994) 2020-06-12T02:37:55.950 INFO:tasks.rados.rados.0.smithi107.stdout:3995: done (2 left) 2020-06-12T02:37:55.950 INFO:tasks.rados.rados.0.smithi107.stdout:3997: done (1 left) 2020-06-12T02:37:55.950 INFO:tasks.rados.rados.0.smithi107.stdout:3998: delete oid 19 current snap is 0 2020-06-12T02:37:55.954 INFO:tasks.rados.rados.0.smithi107.stdout:3998: done (1 left) 2020-06-12T02:37:55.954 INFO:tasks.rados.rados.0.smithi107.stdout:3999: delete oid 18 current snap is 0 2020-06-12T02:37:55.954 INFO:tasks.rados.rados.0.smithi107.stdout:3996: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:55.956 INFO:tasks.rados.rados.0.smithi107.stdout:3999: done (1 left) 2020-06-12T02:37:55.956 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:37:55.957 INFO:tasks.rados.rados.0.smithi107.stdout:4000: seq_num 3999 ranges {0=581632,581632=778240,1359872=532480,1892352=294912} 2020-06-12T02:37:55.960 INFO:tasks.rados.rados.0.smithi107.stdout:4000: writing smithi10729407-19 from 0 to 581632 tid 1 2020-06-12T02:37:55.964 INFO:tasks.rados.rados.0.smithi107.stdout:4000: writing smithi10729407-19 from 581632 to 1359872 tid 2 2020-06-12T02:37:55.967 INFO:tasks.rados.rados.0.smithi107.stdout:4000: writing smithi10729407-19 from 1359872 to 1892352 tid 3 2020-06-12T02:37:55.969 INFO:tasks.rados.rados.0.smithi107.stdout:4000: writing smithi10729407-19 from 1892352 to 2187264 tid 4 2020-06-12T02:37:55.969 INFO:tasks.rados.rados.0.smithi107.stdout:4001: delete oid 1 current snap is 0 2020-06-12T02:37:55.970 INFO:tasks.rados.rados.0.smithi107.stdout:3996: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:55.970 INFO:tasks.rados.rados.0.smithi107.stdout:3996: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:55.970 INFO:tasks.rados.rados.0.smithi107.stdout:3996: finishing write tid 6 to smithi10729407-7 2020-06-12T02:37:55.970 INFO:tasks.rados.rados.0.smithi107.stdout:3996: finishing write tid 7 to smithi10729407-7 2020-06-12T02:37:55.970 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 758 (ObjNum 3995 snap 0 seq_num 3995) dirty exists 2020-06-12T02:37:55.971 INFO:tasks.rados.rados.0.smithi107.stdout:3996: left oid 7 (ObjNum 3995 snap 0 seq_num 3995) 2020-06-12T02:37:55.971 INFO:tasks.rados.rados.0.smithi107.stdout:4000: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:55.972 INFO:tasks.rados.rados.0.smithi107.stdout:3996: done (2 left) 2020-06-12T02:37:55.972 INFO:tasks.rados.rados.0.smithi107.stdout:4001: done (1 left) 2020-06-12T02:37:55.973 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:37:55.975 INFO:tasks.rados.rados.0.smithi107.stdout:4002: seq_num 4001 ranges {0=614400,614400=548864,1163264=548864,1712128=507904} 2020-06-12T02:37:55.975 INFO:tasks.rados.rados.0.smithi107.stdout:4002: writing smithi10729407-3 from 0 to 614400 tid 1 2020-06-12T02:37:55.980 INFO:tasks.rados.rados.0.smithi107.stdout:4002: writing smithi10729407-3 from 614400 to 1163264 tid 2 2020-06-12T02:37:55.987 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:55.981+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:55.987 INFO:tasks.rados.rados.0.smithi107.stdout:4002: writing smithi10729407-3 from 1163264 to 1712128 tid 3 2020-06-12T02:37:55.988 INFO:tasks.rados.rados.0.smithi107.stdout:4002: writing smithi10729407-3 from 1712128 to 2220032 tid 4 2020-06-12T02:37:55.988 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:37:55.988 INFO:tasks.rados.rados.0.smithi107.stdout:4003: seq_num 4002 ranges {2121728=630784,2752512=434176,3186688=712704,3899392=458752} 2020-06-12T02:37:56.002 INFO:tasks.rados.rados.0.smithi107.stdout:4003: writing smithi10729407-7 from 2121728 to 2752512 tid 1 2020-06-12T02:37:56.004 INFO:tasks.rados.rados.0.smithi107.stdout:4003: writing smithi10729407-7 from 2752512 to 3186688 tid 2 2020-06-12T02:37:56.009 INFO:tasks.rados.rados.0.smithi107.stdout:4003: writing smithi10729407-7 from 3186688 to 3899392 tid 3 2020-06-12T02:37:56.012 INFO:tasks.rados.rados.0.smithi107.stdout:4003: writing smithi10729407-7 from 3899392 to 4358144 tid 4 2020-06-12T02:37:56.012 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:37:56.012 INFO:tasks.rados.rados.0.smithi107.stdout:4004: seq_num 4003 ranges {0=647168,647168=720896,1368064=516096,1884160=368640} 2020-06-12T02:37:56.016 INFO:tasks.rados.rados.0.smithi107.stdout:4004: writing smithi10729407-15 from 0 to 647168 tid 1 2020-06-12T02:37:56.021 INFO:tasks.rados.rados.0.smithi107.stdout:4004: writing smithi10729407-15 from 647168 to 1368064 tid 2 2020-06-12T02:37:56.024 INFO:tasks.rados.rados.0.smithi107.stdout:4004: writing smithi10729407-15 from 1368064 to 1884160 tid 3 2020-06-12T02:37:56.026 INFO:tasks.rados.rados.0.smithi107.stdout:4004: writing smithi10729407-15 from 1884160 to 2252800 tid 4 2020-06-12T02:37:56.026 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:37:56.027 INFO:tasks.rados.rados.0.smithi107.stdout:4005: seq_num 4004 ranges {0=663552,663552=606208,1269760=720896,1990656=278528} 2020-06-12T02:37:56.030 INFO:tasks.rados.rados.0.smithi107.stdout:4005: writing smithi10729407-8 from 0 to 663552 tid 1 2020-06-12T02:37:56.033 INFO:tasks.rados.rados.0.smithi107.stdout:4005: writing smithi10729407-8 from 663552 to 1269760 tid 2 2020-06-12T02:37:56.038 INFO:tasks.rados.rados.0.smithi107.stdout:4005: writing smithi10729407-8 from 1269760 to 1990656 tid 3 2020-06-12T02:37:56.040 INFO:tasks.rados.rados.0.smithi107.stdout:4005: writing smithi10729407-8 from 1990656 to 2269184 tid 4 2020-06-12T02:37:56.040 INFO:tasks.rados.rados.0.smithi107.stdout:4006: delete oid 11 current snap is 0 2020-06-12T02:37:56.040 INFO:tasks.rados.rados.0.smithi107.stdout:4000: finishing write tid 2 to smithi10729407-19 2020-06-12T02:37:56.041 INFO:tasks.rados.rados.0.smithi107.stdout:4000: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:56.041 INFO:tasks.rados.rados.0.smithi107.stdout:4000: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:56.041 INFO:tasks.rados.rados.0.smithi107.stdout:4000: finishing write tid 6 to smithi10729407-19 2020-06-12T02:37:56.041 INFO:tasks.rados.rados.0.smithi107.stdout:4000: finishing write tid 7 to smithi10729407-19 2020-06-12T02:37:56.041 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 672 (ObjNum 3999 snap 0 seq_num 3999) dirty exists 2020-06-12T02:37:56.042 INFO:tasks.rados.rados.0.smithi107.stdout:4000: left oid 19 (ObjNum 3999 snap 0 seq_num 3999) 2020-06-12T02:37:56.042 INFO:tasks.rados.rados.0.smithi107.stdout:4002: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:56.042 INFO:tasks.rados.rados.0.smithi107.stdout:4002: finishing write tid 2 to smithi10729407-3 2020-06-12T02:37:56.042 INFO:tasks.rados.rados.0.smithi107.stdout:4002: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:56.042 INFO:tasks.rados.rados.0.smithi107.stdout:4002: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:56.043 INFO:tasks.rados.rados.0.smithi107.stdout:4002: finishing write tid 6 to smithi10729407-3 2020-06-12T02:37:56.043 INFO:tasks.rados.rados.0.smithi107.stdout:4002: finishing write tid 7 to smithi10729407-3 2020-06-12T02:37:56.043 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 714 (ObjNum 4001 snap 0 seq_num 4001) dirty exists 2020-06-12T02:37:56.043 INFO:tasks.rados.rados.0.smithi107.stdout:4002: left oid 3 (ObjNum 4001 snap 0 seq_num 4001) 2020-06-12T02:37:56.043 INFO:tasks.rados.rados.0.smithi107.stdout:4003: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:56.044 INFO:tasks.rados.rados.0.smithi107.stdout:4003: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:56.044 INFO:tasks.rados.rados.0.smithi107.stdout:4003: finishing write tid 3 to smithi10729407-7 2020-06-12T02:37:56.044 INFO:tasks.rados.rados.0.smithi107.stdout:4003: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:56.044 INFO:tasks.rados.rados.0.smithi107.stdout:4003: finishing write tid 6 to smithi10729407-7 2020-06-12T02:37:56.044 INFO:tasks.rados.rados.0.smithi107.stdout:4003: finishing write tid 7 to smithi10729407-7 2020-06-12T02:37:56.044 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 763 (ObjNum 4002 snap 0 seq_num 4002) dirty exists 2020-06-12T02:37:56.045 INFO:tasks.rados.rados.0.smithi107.stdout:4003: left oid 7 (ObjNum 4002 snap 0 seq_num 4002) 2020-06-12T02:37:56.045 INFO:tasks.rados.rados.0.smithi107.stdout:4004: finishing write tid 1 to smithi10729407-15 2020-06-12T02:37:56.045 INFO:tasks.rados.rados.0.smithi107.stdout:4004: finishing write tid 2 to smithi10729407-15 2020-06-12T02:37:56.045 INFO:tasks.rados.rados.0.smithi107.stdout:4004: finishing write tid 3 to smithi10729407-15 2020-06-12T02:37:56.045 INFO:tasks.rados.rados.0.smithi107.stdout:4004: finishing write tid 4 to smithi10729407-15 2020-06-12T02:37:56.045 INFO:tasks.rados.rados.0.smithi107.stdout:4004: finishing write tid 6 to smithi10729407-15 2020-06-12T02:37:56.046 INFO:tasks.rados.rados.0.smithi107.stdout:4004: finishing write tid 7 to smithi10729407-15 2020-06-12T02:37:56.046 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 692 (ObjNum 4003 snap 0 seq_num 4003) dirty exists 2020-06-12T02:37:56.046 INFO:tasks.rados.rados.0.smithi107.stdout:4004: left oid 15 (ObjNum 4003 snap 0 seq_num 4003) 2020-06-12T02:37:56.046 INFO:tasks.rados.rados.0.smithi107.stdout:4005: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:56.047 INFO:tasks.rados.rados.0.smithi107.stdout:4000: done (5 left) 2020-06-12T02:37:56.047 INFO:tasks.rados.rados.0.smithi107.stdout:4002: done (4 left) 2020-06-12T02:37:56.047 INFO:tasks.rados.rados.0.smithi107.stdout:4003: done (3 left) 2020-06-12T02:37:56.048 INFO:tasks.rados.rados.0.smithi107.stdout:4004: done (2 left) 2020-06-12T02:37:56.048 INFO:tasks.rados.rados.0.smithi107.stdout:4006: done (1 left) 2020-06-12T02:37:56.048 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:37:56.048 INFO:tasks.rados.rados.0.smithi107.stdout:4007: seq_num 4006 ranges {0=696320,696320=770048,1466368=688128,2154496=147456} 2020-06-12T02:37:56.049 INFO:tasks.rados.rados.0.smithi107.stdout:4007: writing smithi10729407-11 from 0 to 696320 tid 1 2020-06-12T02:37:56.050 INFO:tasks.rados.rados.0.smithi107.stdout:4007: writing smithi10729407-11 from 696320 to 1466368 tid 2 2020-06-12T02:37:56.054 INFO:tasks.rados.rados.0.smithi107.stdout:4007: writing smithi10729407-11 from 1466368 to 2154496 tid 3 2020-06-12T02:37:56.056 INFO:tasks.rados.rados.0.smithi107.stdout:4007: writing smithi10729407-11 from 2154496 to 2301952 tid 4 2020-06-12T02:37:56.057 INFO:tasks.rados.rados.0.smithi107.stdout:4008: delete oid 4 current snap is 0 2020-06-12T02:37:56.057 INFO:tasks.rados.rados.0.smithi107.stdout:4005: finishing write tid 2 to smithi10729407-8 2020-06-12T02:37:56.058 INFO:tasks.rados.rados.0.smithi107.stdout:4005: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:56.058 INFO:tasks.rados.rados.0.smithi107.stdout:4005: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:56.058 INFO:tasks.rados.rados.0.smithi107.stdout:4005: finishing write tid 6 to smithi10729407-8 2020-06-12T02:37:56.058 INFO:tasks.rados.rados.0.smithi107.stdout:4005: finishing write tid 7 to smithi10729407-8 2020-06-12T02:37:56.059 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 697 (ObjNum 4004 snap 0 seq_num 4004) dirty exists 2020-06-12T02:37:56.059 INFO:tasks.rados.rados.0.smithi107.stdout:4005: left oid 8 (ObjNum 4004 snap 0 seq_num 4004) 2020-06-12T02:37:56.059 INFO:tasks.rados.rados.0.smithi107.stdout:4005: done (2 left) 2020-06-12T02:37:56.060 INFO:tasks.rados.rados.0.smithi107.stdout:4008: done (1 left) 2020-06-12T02:37:56.060 INFO:tasks.rados.rados.0.smithi107.stdout:4009: delete oid 17 current snap is 0 2020-06-12T02:37:56.062 INFO:tasks.rados.rados.0.smithi107.stdout:4009: done (1 left) 2020-06-12T02:37:56.063 INFO:tasks.rados.rados.0.smithi107.stdout:4010: delete oid 17 current snap is 0 2020-06-12T02:37:56.064 INFO:tasks.rados.rados.0.smithi107.stdout:4007: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:56.065 INFO:tasks.rados.rados.0.smithi107.stdout:4010: done (1 left) 2020-06-12T02:37:56.065 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:37:56.066 INFO:tasks.rados.rados.0.smithi107.stdout:4011: seq_num 4010 ranges {0=770048,770048=761856,1531904=671744,2203648=172032} 2020-06-12T02:37:56.070 INFO:tasks.rados.rados.0.smithi107.stdout:4011: writing smithi10729407-9 from 0 to 770048 tid 1 2020-06-12T02:37:56.074 INFO:tasks.rados.rados.0.smithi107.stdout:4011: writing smithi10729407-9 from 770048 to 1531904 tid 2 2020-06-12T02:37:56.075 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:56.075 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:56.078 INFO:tasks.rados.rados.0.smithi107.stdout:4011: writing smithi10729407-9 from 1531904 to 2203648 tid 3 2020-06-12T02:37:56.080 INFO:tasks.rados.rados.0.smithi107.stdout:4011: writing smithi10729407-9 from 2203648 to 2375680 tid 4 2020-06-12T02:37:56.080 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:37:56.080 INFO:tasks.rados.rados.0.smithi107.stdout:4012: seq_num 4011 ranges {0=786432,786432=647168,1433600=475136,1908736=483328} 2020-06-12T02:37:56.084 INFO:tasks.rados.rados.0.smithi107.stdout:4012: writing smithi10729407-18 from 0 to 786432 tid 1 2020-06-12T02:37:56.087 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-06-12T02:37:56.088 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:37:56.085+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:37:56.089 INFO:tasks.rados.rados.0.smithi107.stdout:4012: writing smithi10729407-18 from 786432 to 1433600 tid 2 2020-06-12T02:37:56.091 INFO:tasks.rados.rados.0.smithi107.stdout:4012: writing smithi10729407-18 from 1433600 to 1908736 tid 3 2020-06-12T02:37:56.095 INFO:tasks.rados.rados.0.smithi107.stdout:4012: writing smithi10729407-18 from 1908736 to 2392064 tid 4 2020-06-12T02:37:56.095 INFO:tasks.rados.rados.0.smithi107.stdout:4013: delete oid 16 current snap is 0 2020-06-12T02:37:56.095 INFO:tasks.rados.rados.0.smithi107.stdout:4007: finishing write tid 2 to smithi10729407-11 2020-06-12T02:37:56.095 INFO:tasks.rados.rados.0.smithi107.stdout:4007: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:56.096 INFO:tasks.rados.rados.0.smithi107.stdout:4007: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:56.096 INFO:tasks.rados.rados.0.smithi107.stdout:4007: finishing write tid 6 to smithi10729407-11 2020-06-12T02:37:56.096 INFO:tasks.rados.rados.0.smithi107.stdout:4007: finishing write tid 7 to smithi10729407-11 2020-06-12T02:37:56.096 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 711 (ObjNum 4006 snap 0 seq_num 4006) dirty exists 2020-06-12T02:37:56.097 INFO:tasks.rados.rados.0.smithi107.stdout:4007: left oid 11 (ObjNum 4006 snap 0 seq_num 4006) 2020-06-12T02:37:56.097 INFO:tasks.rados.rados.0.smithi107.stdout:4011: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:56.098 INFO:tasks.rados.rados.0.smithi107.stdout:4011: finishing write tid 2 to smithi10729407-9 2020-06-12T02:37:56.098 INFO:tasks.rados.rados.0.smithi107.stdout:4011: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:56.098 INFO:tasks.rados.rados.0.smithi107.stdout:4011: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:56.099 INFO:tasks.rados.rados.0.smithi107.stdout:4011: finishing write tid 6 to smithi10729407-9 2020-06-12T02:37:56.099 INFO:tasks.rados.rados.0.smithi107.stdout:4011: finishing write tid 7 to smithi10729407-9 2020-06-12T02:37:56.099 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 710 (ObjNum 4010 snap 0 seq_num 4010) dirty exists 2020-06-12T02:37:56.099 INFO:tasks.rados.rados.0.smithi107.stdout:4011: left oid 9 (ObjNum 4010 snap 0 seq_num 4010) 2020-06-12T02:37:56.100 INFO:tasks.rados.rados.0.smithi107.stdout:4012: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:56.100 INFO:tasks.rados.rados.0.smithi107.stdout:4007: done (3 left) 2020-06-12T02:37:56.101 INFO:tasks.rados.rados.0.smithi107.stdout:4011: done (2 left) 2020-06-12T02:37:56.101 INFO:tasks.rados.rados.0.smithi107.stdout:4013: done (1 left) 2020-06-12T02:37:56.101 INFO:tasks.rados.rados.0.smithi107.stdout:4014: delete oid 16 current snap is 0 2020-06-12T02:37:56.102 INFO:tasks.rados.rados.0.smithi107.stdout:4014: done (1 left) 2020-06-12T02:37:56.102 INFO:tasks.rados.rados.0.smithi107.stdout:4015: delete oid 12 current snap is 0 2020-06-12T02:37:56.105 INFO:tasks.rados.rados.0.smithi107.stdout:4015: done (1 left) 2020-06-12T02:37:56.105 INFO:tasks.rados.rados.0.smithi107.stdout:4016: delete oid 7 current snap is 0 2020-06-12T02:37:56.107 INFO:tasks.rados.rados.0.smithi107.stdout:4016: done (1 left) 2020-06-12T02:37:56.107 INFO:tasks.rados.rados.0.smithi107.stdout:4017: delete oid 17 current snap is 0 2020-06-12T02:37:56.110 INFO:tasks.rados.rados.0.smithi107.stdout:4012: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:56.111 INFO:tasks.rados.rados.0.smithi107.stdout:4012: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:56.112 INFO:tasks.rados.rados.0.smithi107.stdout:4012: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:56.115 INFO:tasks.rados.rados.0.smithi107.stdout:4012: finishing write tid 6 to smithi10729407-18 2020-06-12T02:37:56.115 INFO:tasks.rados.rados.0.smithi107.stdout:4017: done (1 left) 2020-06-12T02:37:56.116 INFO:tasks.rados.rados.0.smithi107.stdout:4018: delete oid 8 current snap is 0 2020-06-12T02:37:56.117 INFO:tasks.rados.rados.0.smithi107.stdout:4012: finishing write tid 7 to smithi10729407-18 2020-06-12T02:37:56.117 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 715 (ObjNum 4011 snap 0 seq_num 4011) dirty exists 2020-06-12T02:37:56.117 INFO:tasks.rados.rados.0.smithi107.stdout:4012: left oid 18 (ObjNum 4011 snap 0 seq_num 4011) 2020-06-12T02:37:56.118 INFO:tasks.rados.rados.0.smithi107.stdout:4012: done (1 left) 2020-06-12T02:37:56.118 INFO:tasks.rados.rados.0.smithi107.stdout:4018: done (0 left) 2020-06-12T02:37:56.118 INFO:tasks.rados.rados.0.smithi107.stdout:4019: delete oid 6 current snap is 0 2020-06-12T02:37:56.120 INFO:tasks.rados.rados.0.smithi107.stdout:4019: done (0 left) 2020-06-12T02:37:56.121 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:37:56.121 INFO:tasks.rados.rados.0.smithi107.stdout:4020: seq_num 4019 ranges {0=114688} 2020-06-12T02:37:56.122 INFO:tasks.rados.rados.0.smithi107.stdout:4020: writing smithi10729407-12 from 0 to 114688 tid 1 2020-06-12T02:37:56.122 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:37:56.122 INFO:tasks.rados.rados.0.smithi107.stdout:4021: seq_num 4020 ranges {2375680=131072} 2020-06-12T02:37:56.135 INFO:tasks.rados.rados.0.smithi107.stdout:4021: writing smithi10729407-9 from 2375680 to 2506752 tid 1 2020-06-12T02:37:56.136 INFO:tasks.rados.rados.0.smithi107.stdout:4022: delete oid 1 current snap is 0 2020-06-12T02:37:56.136 INFO:tasks.rados.rados.0.smithi107.stdout:4020: finishing write tid 1 to smithi10729407-12 2020-06-12T02:37:56.137 INFO:tasks.rados.rados.0.smithi107.stdout:4020: finishing write tid 3 to smithi10729407-12 2020-06-12T02:37:56.137 INFO:tasks.rados.rados.0.smithi107.stdout:4020: finishing write tid 4 to smithi10729407-12 2020-06-12T02:37:56.137 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 675 (ObjNum 4019 snap 0 seq_num 4019) dirty exists 2020-06-12T02:37:56.138 INFO:tasks.rados.rados.0.smithi107.stdout:4020: left oid 12 (ObjNum 4019 snap 0 seq_num 4019) 2020-06-12T02:37:56.138 INFO:tasks.rados.rados.0.smithi107.stdout:4020: done (2 left) 2020-06-12T02:37:56.139 INFO:tasks.rados.rados.0.smithi107.stdout:4022: done (1 left) 2020-06-12T02:37:56.140 INFO:tasks.rados.rados.0.smithi107.stdout:4023: delete oid 3 current snap is 0 2020-06-12T02:37:56.142 INFO:tasks.rados.rados.0.smithi107.stdout:4023: done (1 left) 2020-06-12T02:37:56.143 INFO:tasks.rados.rados.0.smithi107.stdout:4024: delete oid 6 current snap is 0 2020-06-12T02:37:56.144 INFO:tasks.rados.rados.0.smithi107.stdout:4021: finishing write tid 1 to smithi10729407-9 2020-06-12T02:37:56.148 INFO:tasks.rados.rados.0.smithi107.stdout:4024: done (1 left) 2020-06-12T02:37:56.148 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:37:56.149 INFO:tasks.rados.rados.0.smithi107.stdout:4025: seq_num 4024 ranges {0=196608} 2020-06-12T02:37:56.150 INFO:tasks.rados.rados.0.smithi107.stdout:4025: writing smithi10729407-8 from 0 to 196608 tid 1 2020-06-12T02:37:56.150 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:37:56.150 INFO:tasks.rados.rados.0.smithi107.stdout:4026: seq_num 4025 ranges {2301952=212992} 2020-06-12T02:37:56.163 INFO:tasks.rados.rados.0.smithi107.stdout:4026: writing smithi10729407-11 from 2301952 to 2514944 tid 1 2020-06-12T02:37:56.164 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:37:56.164 INFO:tasks.rados.rados.0.smithi107.stdout:4027: seq_num 4026 ranges {2392064=229376} 2020-06-12T02:37:56.178 INFO:tasks.rados.rados.0.smithi107.stdout:4027: writing smithi10729407-18 from 2392064 to 2621440 tid 1 2020-06-12T02:37:56.179 INFO:tasks.rados.rados.0.smithi107.stdout:4028: delete oid 4 current snap is 0 2020-06-12T02:37:56.179 INFO:tasks.rados.rados.0.smithi107.stdout:4021: finishing write tid 3 to smithi10729407-9 2020-06-12T02:37:56.179 INFO:tasks.rados.rados.0.smithi107.stdout:4021: finishing write tid 4 to smithi10729407-9 2020-06-12T02:37:56.179 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 717 (ObjNum 4020 snap 0 seq_num 4020) dirty exists 2020-06-12T02:37:56.179 INFO:tasks.rados.rados.0.smithi107.stdout:4021: left oid 9 (ObjNum 4020 snap 0 seq_num 4020) 2020-06-12T02:37:56.180 INFO:tasks.rados.rados.0.smithi107.stdout:4025: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:56.180 INFO:tasks.rados.rados.0.smithi107.stdout:4025: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:56.180 INFO:tasks.rados.rados.0.smithi107.stdout:4025: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:56.180 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 700 (ObjNum 4024 snap 0 seq_num 4024) dirty exists 2020-06-12T02:37:56.181 INFO:tasks.rados.rados.0.smithi107.stdout:4025: left oid 8 (ObjNum 4024 snap 0 seq_num 4024) 2020-06-12T02:37:56.181 INFO:tasks.rados.rados.0.smithi107.stdout:4026: finishing write tid 1 to smithi10729407-11 2020-06-12T02:37:56.181 INFO:tasks.rados.rados.0.smithi107.stdout:4026: finishing write tid 3 to smithi10729407-11 2020-06-12T02:37:56.181 INFO:tasks.rados.rados.0.smithi107.stdout:4026: finishing write tid 4 to smithi10729407-11 2020-06-12T02:37:56.182 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 713 (ObjNum 4025 snap 0 seq_num 4025) dirty exists 2020-06-12T02:37:56.182 INFO:tasks.rados.rados.0.smithi107.stdout:4026: left oid 11 (ObjNum 4025 snap 0 seq_num 4025) 2020-06-12T02:37:56.183 INFO:tasks.rados.rados.0.smithi107.stdout:4021: done (4 left) 2020-06-12T02:37:56.183 INFO:tasks.rados.rados.0.smithi107.stdout:4025: done (3 left) 2020-06-12T02:37:56.184 INFO:tasks.rados.rados.0.smithi107.stdout:4026: done (2 left) 2020-06-12T02:37:56.184 INFO:tasks.rados.rados.0.smithi107.stdout:4028: done (1 left) 2020-06-12T02:37:56.184 INFO:tasks.rados.rados.0.smithi107.stdout:4029: delete oid 9 current snap is 0 2020-06-12T02:37:56.184 INFO:tasks.rados.rados.0.smithi107.stdout:4027: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:56.185 INFO:tasks.rados.rados.0.smithi107.stdout:4027: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:56.185 INFO:tasks.rados.rados.0.smithi107.stdout:4029: done (1 left) 2020-06-12T02:37:56.186 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:37:56.186 INFO:tasks.rados.rados.0.smithi107.stdout:4030: seq_num 4029 ranges {0=286720} 2020-06-12T02:37:56.187 INFO:tasks.rados.rados.0.smithi107.stdout:4030: writing smithi10729407-17 from 0 to 286720 tid 1 2020-06-12T02:37:56.188 INFO:tasks.rados.rados.0.smithi107.stdout:4031: delete oid 10 current snap is 0 2020-06-12T02:37:56.189 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:56.186+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:56.189 INFO:tasks.rados.rados.0.smithi107.stdout:4027: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:56.190 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 719 (ObjNum 4026 snap 0 seq_num 4026) dirty exists 2020-06-12T02:37:56.190 INFO:tasks.rados.rados.0.smithi107.stdout:4027: left oid 18 (ObjNum 4026 snap 0 seq_num 4026) 2020-06-12T02:37:56.191 INFO:tasks.rados.rados.0.smithi107.stdout:4027: done (2 left) 2020-06-12T02:37:56.191 INFO:tasks.rados.rados.0.smithi107.stdout:4031: done (1 left) 2020-06-12T02:37:56.191 INFO:tasks.rados.rados.0.smithi107.stdout:4032: delete oid 20 current snap is 0 2020-06-12T02:37:56.192 INFO:tasks.rados.rados.0.smithi107.stdout:4030: finishing write tid 1 to smithi10729407-17 2020-06-12T02:37:56.194 INFO:tasks.rados.rados.0.smithi107.stdout:4032: done (1 left) 2020-06-12T02:37:56.195 INFO:tasks.rados.rados.0.smithi107.stdout:4033: delete oid 12 current snap is 0 2020-06-12T02:37:56.195 INFO:tasks.rados.rados.0.smithi107.stdout:4030: finishing write tid 3 to smithi10729407-17 2020-06-12T02:37:56.196 INFO:tasks.rados.rados.0.smithi107.stdout:4033: done (1 left) 2020-06-12T02:37:56.196 INFO:tasks.rados.rados.0.smithi107.stdout:4034: delete oid 15 current snap is 0 2020-06-12T02:37:56.196 INFO:tasks.rados.rados.0.smithi107.stdout:4030: finishing write tid 4 to smithi10729407-17 2020-06-12T02:37:56.197 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 733 (ObjNum 4029 snap 0 seq_num 4029) dirty exists 2020-06-12T02:37:56.197 INFO:tasks.rados.rados.0.smithi107.stdout:4030: left oid 17 (ObjNum 4029 snap 0 seq_num 4029) 2020-06-12T02:37:56.198 INFO:tasks.rados.rados.0.smithi107.stdout:4030: done (1 left) 2020-06-12T02:37:56.198 INFO:tasks.rados.rados.0.smithi107.stdout:4034: done (0 left) 2020-06-12T02:37:56.199 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:37:56.199 INFO:tasks.rados.rados.0.smithi107.stdout:4035: seq_num 4034 ranges {0=368640} 2020-06-12T02:37:56.200 INFO:tasks.rados.rados.0.smithi107.stdout:4035: writing smithi10729407-1 from 0 to 368640 tid 1 2020-06-12T02:37:56.201 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:37:56.201 INFO:tasks.rados.rados.0.smithi107.stdout:4036: seq_num 4035 ranges {0=385024} 2020-06-12T02:37:56.202 INFO:tasks.rados.rados.0.smithi107.stdout:4036: writing smithi10729407-10 from 0 to 385024 tid 1 2020-06-12T02:37:56.203 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:37:56.203 INFO:tasks.rados.rados.0.smithi107.stdout:4037: seq_num 4036 ranges {0=401408} 2020-06-12T02:37:56.205 INFO:tasks.rados.rados.0.smithi107.stdout:4037: writing smithi10729407-3 from 0 to 401408 tid 1 2020-06-12T02:37:56.205 INFO:tasks.rados.rados.0.smithi107.stdout:4038: delete oid 5 current snap is 0 2020-06-12T02:37:56.205 INFO:tasks.rados.rados.0.smithi107.stdout:4035: finishing write tid 1 to smithi10729407-1 2020-06-12T02:37:56.210 INFO:tasks.rados.rados.0.smithi107.stdout:4036: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:56.210 INFO:tasks.rados.rados.0.smithi107.stdout:4036: finishing write tid 3 to smithi10729407-10 2020-06-12T02:37:56.210 INFO:tasks.rados.rados.0.smithi107.stdout:4038: done (3 left) 2020-06-12T02:37:56.210 INFO:tasks.rados.rados.0.smithi107.stdout:4039: delete oid 14 current snap is 0 2020-06-12T02:37:56.211 INFO:tasks.rados.rados.0.smithi107.stdout:4036: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:56.211 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 391 (ObjNum 4035 snap 0 seq_num 4035) dirty exists 2020-06-12T02:37:56.211 INFO:tasks.rados.rados.0.smithi107.stdout:4036: left oid 10 (ObjNum 4035 snap 0 seq_num 4035) 2020-06-12T02:37:56.212 INFO:tasks.rados.rados.0.smithi107.stdout:4035: finishing write tid 3 to smithi10729407-1 2020-06-12T02:37:56.213 INFO:tasks.rados.rados.0.smithi107.stdout:4036: done (3 left) 2020-06-12T02:37:56.213 INFO:tasks.rados.rados.0.smithi107.stdout:4039: done (2 left) 2020-06-12T02:37:56.213 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:37:56.214 INFO:tasks.rados.rados.0.smithi107.stdout:4040: seq_num 4039 ranges {196608=450560} 2020-06-12T02:37:56.217 INFO:tasks.rados.rados.0.smithi107.stdout:4040: writing smithi10729407-8 from 196608 to 647168 tid 1 2020-06-12T02:37:56.217 INFO:tasks.rados.rados.0.smithi107.stdout:4041: delete oid 10 current snap is 0 2020-06-12T02:37:56.218 INFO:tasks.rados.rados.0.smithi107.stdout:4037: finishing write tid 1 to smithi10729407-3 2020-06-12T02:37:56.218 INFO:tasks.rados.rados.0.smithi107.stdout:4035: finishing write tid 4 to smithi10729407-1 2020-06-12T02:37:56.218 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 761 (ObjNum 4034 snap 0 seq_num 4034) dirty exists 2020-06-12T02:37:56.218 INFO:tasks.rados.rados.0.smithi107.stdout:4035: left oid 1 (ObjNum 4034 snap 0 seq_num 4034) 2020-06-12T02:37:56.219 INFO:tasks.rados.rados.0.smithi107.stdout:4037: finishing write tid 3 to smithi10729407-3 2020-06-12T02:37:56.219 INFO:tasks.rados.rados.0.smithi107.stdout:4035: done (3 left) 2020-06-12T02:37:56.219 INFO:tasks.rados.rados.0.smithi107.stdout:4041: done (2 left) 2020-06-12T02:37:56.220 INFO:tasks.rados.rados.0.smithi107.stdout:4042: delete oid 19 current snap is 0 2020-06-12T02:37:56.221 INFO:tasks.rados.rados.0.smithi107.stdout:4037: finishing write tid 4 to smithi10729407-3 2020-06-12T02:37:56.221 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 719 (ObjNum 4036 snap 0 seq_num 4036) dirty exists 2020-06-12T02:37:56.221 INFO:tasks.rados.rados.0.smithi107.stdout:4037: left oid 3 (ObjNum 4036 snap 0 seq_num 4036) 2020-06-12T02:37:56.222 INFO:tasks.rados.rados.0.smithi107.stdout:4037: done (2 left) 2020-06-12T02:37:56.223 INFO:tasks.rados.rados.0.smithi107.stdout:4042: done (1 left) 2020-06-12T02:37:56.223 INFO:tasks.rados.rados.0.smithi107.stdout:4043: delete oid 10 current snap is 0 2020-06-12T02:37:56.223 INFO:tasks.rados.rados.0.smithi107.stdout:4040: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:56.224 INFO:tasks.rados.rados.0.smithi107.stdout:4040: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:56.225 INFO:tasks.rados.rados.0.smithi107.stdout:4040: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:56.225 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 703 (ObjNum 4039 snap 0 seq_num 4039) dirty exists 2020-06-12T02:37:56.225 INFO:tasks.rados.rados.0.smithi107.stdout:4040: left oid 8 (ObjNum 4039 snap 0 seq_num 4039) 2020-06-12T02:37:56.226 INFO:tasks.rados.rados.0.smithi107.stdout:4040: done (1 left) 2020-06-12T02:37:56.226 INFO:tasks.rados.rados.0.smithi107.stdout:4043: done (0 left) 2020-06-12T02:37:56.227 INFO:tasks.rados.rados.0.smithi107.stdout:4044: delete oid 8 current snap is 0 2020-06-12T02:37:56.231 INFO:tasks.rados.rados.0.smithi107.stdout:4044: done (0 left) 2020-06-12T02:37:56.232 INFO:tasks.rados.rados.0.smithi107.stdout:4045: delete oid 1 current snap is 0 2020-06-12T02:37:56.234 INFO:tasks.rados.rados.0.smithi107.stdout:4045: done (0 left) 2020-06-12T02:37:56.234 INFO:tasks.rados.rados.0.smithi107.stdout:4046: delete oid 3 current snap is 0 2020-06-12T02:37:56.236 INFO:tasks.rados.rados.0.smithi107.stdout:4046: done (0 left) 2020-06-12T02:37:56.236 INFO:tasks.rados.rados.0.smithi107.stdout:4047: delete oid 20 current snap is 0 2020-06-12T02:37:56.238 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:56.238 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:56.239 INFO:tasks.rados.rados.0.smithi107.stdout:4047: done (0 left) 2020-06-12T02:37:56.239 INFO:tasks.rados.rados.0.smithi107.stdout:4048: delete oid 9 current snap is 0 2020-06-12T02:37:56.241 INFO:tasks.rados.rados.0.smithi107.stdout:4048: done (0 left) 2020-06-12T02:37:56.241 INFO:tasks.rados.rados.0.smithi107.stdout:4049: delete oid 12 current snap is 0 2020-06-12T02:37:56.243 INFO:tasks.rados.rados.0.smithi107.stdout:4049: done (0 left) 2020-06-12T02:37:56.243 INFO:tasks.rados.rados.0.smithi107.stdout:4050: delete oid 4 current snap is 0 2020-06-12T02:37:56.245 INFO:tasks.rados.rados.0.smithi107.stdout:4050: done (0 left) 2020-06-12T02:37:56.245 INFO:tasks.rados.rados.0.smithi107.stdout:4051: delete oid 17 current snap is 0 2020-06-12T02:37:56.247 INFO:tasks.rados.rados.0.smithi107.stdout:4051: done (0 left) 2020-06-12T02:37:56.247 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:37:56.247 INFO:tasks.rados.rados.0.smithi107.stdout:4052: seq_num 4051 ranges {0=655360} 2020-06-12T02:37:56.249 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-06-12T02:37:56.251 INFO:tasks.rados.rados.0.smithi107.stdout:4052: writing smithi10729407-8 from 0 to 655360 tid 1 2020-06-12T02:37:56.251 INFO:tasks.rados.rados.0.smithi107.stdout:4053: delete oid 19 current snap is 0 2020-06-12T02:37:56.253 INFO:tasks.rados.rados.0.smithi107.stdout:4053: done (1 left) 2020-06-12T02:37:56.253 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:37:56.253 INFO:tasks.rados.rados.0.smithi107.stdout:4054: seq_num 4053 ranges {0=688128} 2020-06-12T02:37:56.257 INFO:tasks.rados.rados.0.smithi107.stdout:4054: writing smithi10729407-16 from 0 to 688128 tid 1 2020-06-12T02:37:56.258 INFO:tasks.rados.rados.0.smithi107.stdout:4055: delete oid 19 current snap is 0 2020-06-12T02:37:56.259 INFO:tasks.rados.rados.0.smithi107.stdout:4052: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:56.259 INFO:tasks.rados.rados.0.smithi107.stdout:4052: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:56.260 INFO:tasks.rados.rados.0.smithi107.stdout:4055: done (2 left) 2020-06-12T02:37:56.260 INFO:tasks.rados.rados.0.smithi107.stdout:4056: delete oid 5 current snap is 0 2020-06-12T02:37:56.261 INFO:tasks.rados.rados.0.smithi107.stdout:4052: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:56.262 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 706 (ObjNum 4051 snap 0 seq_num 4051) dirty exists 2020-06-12T02:37:56.262 INFO:tasks.rados.rados.0.smithi107.stdout:4052: left oid 8 (ObjNum 4051 snap 0 seq_num 4051) 2020-06-12T02:37:56.263 INFO:tasks.rados.rados.0.smithi107.stdout:4052: done (2 left) 2020-06-12T02:37:56.263 INFO:tasks.rados.rados.0.smithi107.stdout:4056: done (1 left) 2020-06-12T02:37:56.264 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:37:56.264 INFO:tasks.rados.rados.0.smithi107.stdout:4057: seq_num 4056 ranges {0=737280} 2020-06-12T02:37:56.266 INFO:tasks.rados.rados.0.smithi107.stdout:4057: writing smithi10729407-19 from 0 to 737280 tid 1 2020-06-12T02:37:56.267 INFO:tasks.rados.rados.0.smithi107.stdout:4058: delete oid 12 current snap is 0 2020-06-12T02:37:56.267 INFO:tasks.rados.rados.0.smithi107.stdout:4054: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:56.268 INFO:tasks.rados.rados.0.smithi107.stdout:4054: finishing write tid 3 to smithi10729407-16 2020-06-12T02:37:56.268 INFO:tasks.rados.rados.0.smithi107.stdout:4054: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:56.268 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 722 (ObjNum 4053 snap 0 seq_num 4053) dirty exists 2020-06-12T02:37:56.268 INFO:tasks.rados.rados.0.smithi107.stdout:4054: left oid 16 (ObjNum 4053 snap 0 seq_num 4053) 2020-06-12T02:37:56.273 INFO:tasks.rados.rados.0.smithi107.stdout:4057: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:56.274 INFO:tasks.rados.rados.0.smithi107.stdout:4054: done (2 left) 2020-06-12T02:37:56.275 INFO:tasks.rados.rados.0.smithi107.stdout:4058: done (1 left) 2020-06-12T02:37:56.275 INFO:tasks.rados.rados.0.smithi107.stdout:4059: delete oid 2 current snap is 0 2020-06-12T02:37:56.277 INFO:tasks.rados.rados.0.smithi107.stdout:4057: finishing write tid 3 to smithi10729407-19 2020-06-12T02:37:56.278 INFO:tasks.rados.rados.0.smithi107.stdout:4059: done (1 left) 2020-06-12T02:37:56.278 INFO:tasks.rados.rados.0.smithi107.stdout:4060: delete oid 2 current snap is 0 2020-06-12T02:37:56.279 INFO:tasks.rados.rados.0.smithi107.stdout:4057: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:56.279 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 683 (ObjNum 4056 snap 0 seq_num 4056) dirty exists 2020-06-12T02:37:56.280 INFO:tasks.rados.rados.0.smithi107.stdout:4057: left oid 19 (ObjNum 4056 snap 0 seq_num 4056) 2020-06-12T02:37:56.284 INFO:tasks.rados.rados.0.smithi107.stdout:4057: done (1 left) 2020-06-12T02:37:56.284 INFO:tasks.rados.rados.0.smithi107.stdout:4060: done (0 left) 2020-06-12T02:37:56.284 INFO:tasks.rados.rados.0.smithi107.stdout:4061: delete oid 3 current snap is 0 2020-06-12T02:37:56.286 INFO:tasks.rados.rados.0.smithi107.stdout:4061: done (0 left) 2020-06-12T02:37:56.286 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:37:56.286 INFO:tasks.rados.rados.0.smithi107.stdout:4062: seq_num 4061 ranges {737280=417792,1155072=401408} 2020-06-12T02:37:56.289 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:56.286+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:56.292 INFO:tasks.rados.rados.0.smithi107.stdout:4062: writing smithi10729407-19 from 737280 to 1155072 tid 1 2020-06-12T02:37:56.295 INFO:tasks.rados.rados.0.smithi107.stdout:4062: writing smithi10729407-19 from 1155072 to 1556480 tid 2 2020-06-12T02:37:56.295 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:37:56.295 INFO:tasks.rados.rados.0.smithi107.stdout:4063: seq_num 4062 ranges {655360=434176,1089536=401408} 2020-06-12T02:37:56.301 INFO:tasks.rados.rados.0.smithi107.stdout:4063: writing smithi10729407-8 from 655360 to 1089536 tid 1 2020-06-12T02:37:56.303 INFO:tasks.rados.rados.0.smithi107.stdout:4063: writing smithi10729407-8 from 1089536 to 1490944 tid 2 2020-06-12T02:37:56.304 INFO:tasks.rados.rados.0.smithi107.stdout:4064: delete oid 15 current snap is 0 2020-06-12T02:37:56.304 INFO:tasks.rados.rados.0.smithi107.stdout:4062: finishing write tid 1 to smithi10729407-19 2020-06-12T02:37:56.304 INFO:tasks.rados.rados.0.smithi107.stdout:4062: finishing write tid 2 to smithi10729407-19 2020-06-12T02:37:56.304 INFO:tasks.rados.rados.0.smithi107.stdout:4062: finishing write tid 4 to smithi10729407-19 2020-06-12T02:37:56.305 INFO:tasks.rados.rados.0.smithi107.stdout:4062: finishing write tid 5 to smithi10729407-19 2020-06-12T02:37:56.305 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 686 (ObjNum 4061 snap 0 seq_num 4061) dirty exists 2020-06-12T02:37:56.305 INFO:tasks.rados.rados.0.smithi107.stdout:4062: left oid 19 (ObjNum 4061 snap 0 seq_num 4061) 2020-06-12T02:37:56.310 INFO:tasks.rados.rados.0.smithi107.stdout:4063: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:56.310 INFO:tasks.rados.rados.0.smithi107.stdout:4063: finishing write tid 2 to smithi10729407-8 2020-06-12T02:37:56.311 INFO:tasks.rados.rados.0.smithi107.stdout:4063: finishing write tid 4 to smithi10729407-8 2020-06-12T02:37:56.312 INFO:tasks.rados.rados.0.smithi107.stdout:4062: done (2 left) 2020-06-12T02:37:56.313 INFO:tasks.rados.rados.0.smithi107.stdout:4064: done (1 left) 2020-06-12T02:37:56.313 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:37:56.313 INFO:tasks.rados.rados.0.smithi107.stdout:4065: seq_num 4064 ranges {0=466944,466944=401408} 2020-06-12T02:37:56.314 INFO:tasks.rados.rados.0.smithi107.stdout:4065: writing smithi10729407-10 from 0 to 466944 tid 1 2020-06-12T02:37:56.316 INFO:tasks.rados.rados.0.smithi107.stdout:4065: writing smithi10729407-10 from 466944 to 868352 tid 2 2020-06-12T02:37:56.317 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:37:56.317 INFO:tasks.rados.rados.0.smithi107.stdout:4066: seq_num 4065 ranges {2621440=483328,3104768=401408} 2020-06-12T02:37:56.333 INFO:tasks.rados.rados.0.smithi107.stdout:4066: writing smithi10729407-18 from 2621440 to 3104768 tid 1 2020-06-12T02:37:56.336 INFO:tasks.rados.rados.0.smithi107.stdout:4066: writing smithi10729407-18 from 3104768 to 3506176 tid 2 2020-06-12T02:37:56.336 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:37:56.337 INFO:tasks.rados.rados.0.smithi107.stdout:4067: seq_num 4066 ranges {0=499712,499712=401408} 2020-06-12T02:37:56.339 INFO:tasks.rados.rados.0.smithi107.stdout:4067: writing smithi10729407-20 from 0 to 499712 tid 1 2020-06-12T02:37:56.342 INFO:tasks.rados.rados.0.smithi107.stdout:4067: writing smithi10729407-20 from 499712 to 901120 tid 2 2020-06-12T02:37:56.342 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:37:56.343 INFO:tasks.rados.rados.0.smithi107.stdout:4068: seq_num 4067 ranges {0=524288,524288=401408} 2020-06-12T02:37:56.345 INFO:tasks.rados.rados.0.smithi107.stdout:4068: writing smithi10729407-7 from 0 to 524288 tid 1 2020-06-12T02:37:56.348 INFO:tasks.rados.rados.0.smithi107.stdout:4068: writing smithi10729407-7 from 524288 to 925696 tid 2 2020-06-12T02:37:56.349 INFO:tasks.rados.rados.0.smithi107.stdout:4069: delete oid 9 current snap is 0 2020-06-12T02:37:56.351 INFO:tasks.rados.rados.0.smithi107.stdout:4063: finishing write tid 5 to smithi10729407-8 2020-06-12T02:37:56.351 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 709 (ObjNum 4062 snap 0 seq_num 4062) dirty exists 2020-06-12T02:37:56.352 INFO:tasks.rados.rados.0.smithi107.stdout:4063: left oid 8 (ObjNum 4062 snap 0 seq_num 4062) 2020-06-12T02:37:56.352 INFO:tasks.rados.rados.0.smithi107.stdout:4065: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:56.352 INFO:tasks.rados.rados.0.smithi107.stdout:4065: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:56.352 INFO:tasks.rados.rados.0.smithi107.stdout:4065: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:56.352 INFO:tasks.rados.rados.0.smithi107.stdout:4065: finishing write tid 5 to smithi10729407-10 2020-06-12T02:37:56.353 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 396 (ObjNum 4064 snap 0 seq_num 4064) dirty exists 2020-06-12T02:37:56.354 INFO:tasks.rados.rados.0.smithi107.stdout:4065: left oid 10 (ObjNum 4064 snap 0 seq_num 4064) 2020-06-12T02:37:56.354 INFO:tasks.rados.rados.0.smithi107.stdout:4066: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:56.354 INFO:tasks.rados.rados.0.smithi107.stdout:4066: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:56.354 INFO:tasks.rados.rados.0.smithi107.stdout:4066: finishing write tid 4 to smithi10729407-18 2020-06-12T02:37:56.355 INFO:tasks.rados.rados.0.smithi107.stdout:4067: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:56.355 INFO:tasks.rados.rados.0.smithi107.stdout:4066: finishing write tid 5 to smithi10729407-18 2020-06-12T02:37:56.355 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 724 (ObjNum 4065 snap 0 seq_num 4065) dirty exists 2020-06-12T02:37:56.355 INFO:tasks.rados.rados.0.smithi107.stdout:4066: left oid 18 (ObjNum 4065 snap 0 seq_num 4065) 2020-06-12T02:37:56.356 INFO:tasks.rados.rados.0.smithi107.stdout:4067: finishing write tid 2 to smithi10729407-20 2020-06-12T02:37:56.356 INFO:tasks.rados.rados.0.smithi107.stdout:4063: done (5 left) 2020-06-12T02:37:56.357 INFO:tasks.rados.rados.0.smithi107.stdout:4065: done (4 left) 2020-06-12T02:37:56.357 INFO:tasks.rados.rados.0.smithi107.stdout:4066: done (3 left) 2020-06-12T02:37:56.357 INFO:tasks.rados.rados.0.smithi107.stdout:4069: done (2 left) 2020-06-12T02:37:56.357 INFO:tasks.rados.rados.0.smithi107.stdout:4070: delete oid 12 current snap is 0 2020-06-12T02:37:56.357 INFO:tasks.rados.rados.0.smithi107.stdout:4067: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:56.358 INFO:tasks.rados.rados.0.smithi107.stdout:4067: finishing write tid 5 to smithi10729407-20 2020-06-12T02:37:56.358 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 311 (ObjNum 4066 snap 0 seq_num 4066) dirty exists 2020-06-12T02:37:56.358 INFO:tasks.rados.rados.0.smithi107.stdout:4067: left oid 20 (ObjNum 4066 snap 0 seq_num 4066) 2020-06-12T02:37:56.358 INFO:tasks.rados.rados.0.smithi107.stdout:4067: done (2 left) 2020-06-12T02:37:56.358 INFO:tasks.rados.rados.0.smithi107.stdout:4070: done (1 left) 2020-06-12T02:37:56.359 INFO:tasks.rados.rados.0.smithi107.stdout:4071: delete oid 3 current snap is 0 2020-06-12T02:37:56.359 INFO:tasks.rados.rados.0.smithi107.stdout:4068: finishing write tid 1 to smithi10729407-7 2020-06-12T02:37:56.361 INFO:tasks.rados.rados.0.smithi107.stdout:4071: done (1 left) 2020-06-12T02:37:56.361 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:37:56.361 INFO:tasks.rados.rados.0.smithi107.stdout:4072: seq_num 4071 ranges {688128=589824,1277952=401408} 2020-06-12T02:37:56.368 INFO:tasks.rados.rados.0.smithi107.stdout:4072: writing smithi10729407-16 from 688128 to 1277952 tid 1 2020-06-12T02:37:56.371 INFO:tasks.rados.rados.0.smithi107.stdout:4072: writing smithi10729407-16 from 1277952 to 1679360 tid 2 2020-06-12T02:37:56.371 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:37:56.371 INFO:tasks.rados.rados.0.smithi107.stdout:4073: seq_num 4072 ranges {901120=606208,1507328=401408} 2020-06-12T02:37:56.379 INFO:tasks.rados.rados.0.smithi107.stdout:4073: writing smithi10729407-20 from 901120 to 1507328 tid 1 2020-06-12T02:37:56.382 INFO:tasks.rados.rados.0.smithi107.stdout:4073: writing smithi10729407-20 from 1507328 to 1908736 tid 2 2020-06-12T02:37:56.382 INFO:tasks.rados.rados.0.smithi107.stdout:4074: delete oid 13 current snap is 0 2020-06-12T02:37:56.383 INFO:tasks.rados.rados.0.smithi107.stdout:4068: finishing write tid 2 to smithi10729407-7 2020-06-12T02:37:56.383 INFO:tasks.rados.rados.0.smithi107.stdout:4068: finishing write tid 4 to smithi10729407-7 2020-06-12T02:37:56.383 INFO:tasks.rados.rados.0.smithi107.stdout:4068: finishing write tid 5 to smithi10729407-7 2020-06-12T02:37:56.383 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 767 (ObjNum 4067 snap 0 seq_num 4067) dirty exists 2020-06-12T02:37:56.383 INFO:tasks.rados.rados.0.smithi107.stdout:4068: left oid 7 (ObjNum 4067 snap 0 seq_num 4067) 2020-06-12T02:37:56.384 INFO:tasks.rados.rados.0.smithi107.stdout:4072: finishing write tid 1 to smithi10729407-16 2020-06-12T02:37:56.384 INFO:tasks.rados.rados.0.smithi107.stdout:4072: finishing write tid 2 to smithi10729407-16 2020-06-12T02:37:56.384 INFO:tasks.rados.rados.0.smithi107.stdout:4072: finishing write tid 4 to smithi10729407-16 2020-06-12T02:37:56.384 INFO:tasks.rados.rados.0.smithi107.stdout:4072: finishing write tid 5 to smithi10729407-16 2020-06-12T02:37:56.384 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 727 (ObjNum 4071 snap 0 seq_num 4071) dirty exists 2020-06-12T02:37:56.385 INFO:tasks.rados.rados.0.smithi107.stdout:4072: left oid 16 (ObjNum 4071 snap 0 seq_num 4071) 2020-06-12T02:37:56.385 INFO:tasks.rados.rados.0.smithi107.stdout:4068: done (3 left) 2020-06-12T02:37:56.386 INFO:tasks.rados.rados.0.smithi107.stdout:4072: done (2 left) 2020-06-12T02:37:56.386 INFO:tasks.rados.rados.0.smithi107.stdout:4074: done (1 left) 2020-06-12T02:37:56.386 INFO:tasks.rados.rados.0.smithi107.stdout:4075: delete oid 6 current snap is 0 2020-06-12T02:37:56.387 INFO:tasks.rados.rados.0.smithi107.stdout:4075: done (1 left) 2020-06-12T02:37:56.388 INFO:tasks.rados.rados.0.smithi107.stdout:4076: delete oid 14 current snap is 0 2020-06-12T02:37:56.388 INFO:tasks.rados.rados.0.smithi107.stdout:4073: finishing write tid 1 to smithi10729407-20 2020-06-12T02:37:56.390 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:56.388+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:56.391 INFO:tasks.rados.rados.0.smithi107.stdout:4073: finishing write tid 2 to smithi10729407-20 2020-06-12T02:37:56.392 INFO:tasks.rados.rados.0.smithi107.stdout:4076: done (1 left) 2020-06-12T02:37:56.392 INFO:tasks.rados.rados.0.smithi107.stdout:4077: delete oid 2 current snap is 0 2020-06-12T02:37:56.392 INFO:tasks.rados.rados.0.smithi107.stdout:4073: finishing write tid 4 to smithi10729407-20 2020-06-12T02:37:56.393 INFO:tasks.rados.rados.0.smithi107.stdout:4073: finishing write tid 5 to smithi10729407-20 2020-06-12T02:37:56.393 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 314 (ObjNum 4072 snap 0 seq_num 4072) dirty exists 2020-06-12T02:37:56.393 INFO:tasks.rados.rados.0.smithi107.stdout:4073: left oid 20 (ObjNum 4072 snap 0 seq_num 4072) 2020-06-12T02:37:56.394 INFO:tasks.rados.rados.0.smithi107.stdout:4073: done (1 left) 2020-06-12T02:37:56.394 INFO:tasks.rados.rados.0.smithi107.stdout:4077: done (0 left) 2020-06-12T02:37:56.395 INFO:tasks.rados.rados.0.smithi107.stdout:4078: delete oid 17 current snap is 0 2020-06-12T02:37:56.396 INFO:tasks.rados.rados.0.smithi107.stdout:4078: done (0 left) 2020-06-12T02:37:56.396 INFO:tasks.rados.rados.0.smithi107.stdout:4079: delete oid 10 current snap is 0 2020-06-12T02:37:56.398 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:56.398 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:56.398 INFO:tasks.rados.rados.0.smithi107.stdout:4079: done (0 left) 2020-06-12T02:37:56.399 INFO:tasks.rados.rados.0.smithi107.stdout:4080: delete oid 5 current snap is 0 2020-06-12T02:37:56.402 INFO:tasks.rados.rados.0.smithi107.stdout:4080: done (0 left) 2020-06-12T02:37:56.403 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:37:56.403 INFO:tasks.rados.rados.0.smithi107.stdout:4081: seq_num 4080 ranges {0=737280,737280=401408} 2020-06-12T02:37:56.407 INFO:tasks.rados.rados.0.smithi107.stdout:4081: writing smithi10729407-2 from 0 to 737280 tid 1 2020-06-12T02:37:56.407 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-06-12T02:37:56.409 INFO:tasks.rados.rados.0.smithi107.stdout:4081: writing smithi10729407-2 from 737280 to 1138688 tid 2 2020-06-12T02:37:56.409 INFO:tasks.rados.rados.0.smithi107.stdout:4082: delete oid 4 current snap is 0 2020-06-12T02:37:56.412 INFO:tasks.rados.rados.0.smithi107.stdout:4082: done (1 left) 2020-06-12T02:37:56.412 INFO:tasks.rados.rados.0.smithi107.stdout:4083: delete oid 9 current snap is 0 2020-06-12T02:37:56.416 INFO:tasks.rados.rados.0.smithi107.stdout:4081: finishing write tid 1 to smithi10729407-2 2020-06-12T02:37:56.416 INFO:tasks.rados.rados.0.smithi107.stdout:4083: done (1 left) 2020-06-12T02:37:56.417 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:37:56.417 INFO:tasks.rados.rados.0.smithi107.stdout:4084: seq_num 4083 ranges {0=786432,786432=401408} 2020-06-12T02:37:56.421 INFO:tasks.rados.rados.0.smithi107.stdout:4084: writing smithi10729407-10 from 0 to 786432 tid 1 2020-06-12T02:37:56.424 INFO:tasks.rados.rados.0.smithi107.stdout:4084: writing smithi10729407-10 from 786432 to 1187840 tid 2 2020-06-12T02:37:56.424 INFO:tasks.rados.rados.0.smithi107.stdout:4085: delete oid 19 current snap is 0 2020-06-12T02:37:56.425 INFO:tasks.rados.rados.0.smithi107.stdout:4081: finishing write tid 2 to smithi10729407-2 2020-06-12T02:37:56.425 INFO:tasks.rados.rados.0.smithi107.stdout:4081: finishing write tid 4 to smithi10729407-2 2020-06-12T02:37:56.425 INFO:tasks.rados.rados.0.smithi107.stdout:4081: finishing write tid 5 to smithi10729407-2 2020-06-12T02:37:56.425 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 425 (ObjNum 4080 snap 0 seq_num 4080) dirty exists 2020-06-12T02:37:56.426 INFO:tasks.rados.rados.0.smithi107.stdout:4081: left oid 2 (ObjNum 4080 snap 0 seq_num 4080) 2020-06-12T02:37:56.426 INFO:tasks.rados.rados.0.smithi107.stdout:4081: done (2 left) 2020-06-12T02:37:56.427 INFO:tasks.rados.rados.0.smithi107.stdout:4085: done (1 left) 2020-06-12T02:37:56.427 INFO:tasks.rados.rados.0.smithi107.stdout:4086: delete oid 19 current snap is 0 2020-06-12T02:37:56.428 INFO:tasks.rados.rados.0.smithi107.stdout:4084: finishing write tid 1 to smithi10729407-10 2020-06-12T02:37:56.430 INFO:tasks.rados.rados.0.smithi107.stdout:4086: done (1 left) 2020-06-12T02:37:56.430 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:37:56.430 INFO:tasks.rados.rados.0.smithi107.stdout:4087: seq_num 4086 ranges {0=434176,434176=466944,901120=335872} 2020-06-12T02:37:56.432 INFO:tasks.rados.rados.0.smithi107.stdout:4087: writing smithi10729407-14 from 0 to 434176 tid 1 2020-06-12T02:37:56.436 INFO:tasks.rados.rados.0.smithi107.stdout:4087: writing smithi10729407-14 from 434176 to 901120 tid 2 2020-06-12T02:37:56.438 INFO:tasks.rados.rados.0.smithi107.stdout:4087: writing smithi10729407-14 from 901120 to 1236992 tid 3 2020-06-12T02:37:56.438 INFO:tasks.rados.rados.0.smithi107.stdout:4088: delete oid 8 current snap is 0 2020-06-12T02:37:56.439 INFO:tasks.rados.rados.0.smithi107.stdout:4084: finishing write tid 2 to smithi10729407-10 2020-06-12T02:37:56.439 INFO:tasks.rados.rados.0.smithi107.stdout:4084: finishing write tid 4 to smithi10729407-10 2020-06-12T02:37:56.441 INFO:tasks.rados.rados.0.smithi107.stdout:4084: finishing write tid 5 to smithi10729407-10 2020-06-12T02:37:56.442 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 400 (ObjNum 4083 snap 0 seq_num 4083) dirty exists 2020-06-12T02:37:56.442 INFO:tasks.rados.rados.0.smithi107.stdout:4084: left oid 10 (ObjNum 4083 snap 0 seq_num 4083) 2020-06-12T02:37:56.442 INFO:tasks.rados.rados.0.smithi107.stdout:4084: done (2 left) 2020-06-12T02:37:56.442 INFO:tasks.rados.rados.0.smithi107.stdout:4088: done (1 left) 2020-06-12T02:37:56.443 INFO:tasks.rados.rados.0.smithi107.stdout:4089: delete oid 16 current snap is 0 2020-06-12T02:37:56.443 INFO:tasks.rados.rados.0.smithi107.stdout:4089: done (1 left) 2020-06-12T02:37:56.444 INFO:tasks.rados.rados.0.smithi107.stdout:4090: delete oid 16 current snap is 0 2020-06-12T02:37:56.444 INFO:tasks.rados.rados.0.smithi107.stdout:4087: finishing write tid 1 to smithi10729407-14 2020-06-12T02:37:56.445 INFO:tasks.rados.rados.0.smithi107.stdout:4087: finishing write tid 2 to smithi10729407-14 2020-06-12T02:37:56.447 INFO:tasks.rados.rados.0.smithi107.stdout:4087: finishing write tid 3 to smithi10729407-14 2020-06-12T02:37:56.448 INFO:tasks.rados.rados.0.smithi107.stdout:4087: finishing write tid 5 to smithi10729407-14 2020-06-12T02:37:56.448 INFO:tasks.rados.rados.0.smithi107.stdout:4090: done (1 left) 2020-06-12T02:37:56.448 INFO:tasks.rados.rados.0.smithi107.stdout:4091: delete oid 9 current snap is 0 2020-06-12T02:37:56.451 INFO:tasks.rados.rados.0.smithi107.stdout:4087: finishing write tid 6 to smithi10729407-14 2020-06-12T02:37:56.451 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 719 (ObjNum 4086 snap 0 seq_num 4086) dirty exists 2020-06-12T02:37:56.451 INFO:tasks.rados.rados.0.smithi107.stdout:4087: left oid 14 (ObjNum 4086 snap 0 seq_num 4086) 2020-06-12T02:37:56.453 INFO:tasks.rados.rados.0.smithi107.stdout:4087: done (1 left) 2020-06-12T02:37:56.453 INFO:tasks.rados.rados.0.smithi107.stdout:4091: done (0 left) 2020-06-12T02:37:56.453 INFO:tasks.rados.rados.0.smithi107.stdout:4092: delete oid 5 current snap is 0 2020-06-12T02:37:56.456 INFO:tasks.rados.rados.0.smithi107.stdout:4092: done (0 left) 2020-06-12T02:37:56.456 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:37:56.456 INFO:tasks.rados.rados.0.smithi107.stdout:4093: seq_num 4092 ranges {0=540672,540672=622592,1163264=180224} 2020-06-12T02:37:56.458 INFO:tasks.rados.rados.0.smithi107.stdout:4093: writing smithi10729407-8 from 0 to 540672 tid 1 2020-06-12T02:37:56.462 INFO:tasks.rados.rados.0.smithi107.stdout:4093: writing smithi10729407-8 from 540672 to 1163264 tid 2 2020-06-12T02:37:56.463 INFO:tasks.rados.rados.0.smithi107.stdout:4093: writing smithi10729407-8 from 1163264 to 1343488 tid 3 2020-06-12T02:37:56.463 INFO:tasks.rados.rados.0.smithi107.stdout:4094: delete oid 9 current snap is 0 2020-06-12T02:37:56.464 INFO:tasks.rados.rados.0.smithi107.stdout:4093: finishing write tid 1 to smithi10729407-8 2020-06-12T02:37:56.466 INFO:tasks.rados.rados.0.smithi107.stdout:4094: done (1 left) 2020-06-12T02:37:56.466 INFO:tasks.rados.rados.0.smithi107.stdout:4095: delete oid 20 current snap is 0 2020-06-12T02:37:56.471 INFO:tasks.rados.rados.0.smithi107.stdout:4093: finishing write tid 2 to smithi10729407-8 2020-06-12T02:37:56.471 INFO:tasks.rados.rados.0.smithi107.stdout:4093: finishing write tid 3 to smithi10729407-8 2020-06-12T02:37:56.472 INFO:tasks.rados.rados.0.smithi107.stdout:4095: done (1 left) 2020-06-12T02:37:56.472 INFO:tasks.rados.rados.0.smithi107.stdout:4096: delete oid 14 current snap is 0 2020-06-12T02:37:56.473 INFO:tasks.rados.rados.0.smithi107.stdout:4093: finishing write tid 5 to smithi10729407-8 2020-06-12T02:37:56.474 INFO:tasks.rados.rados.0.smithi107.stdout:4093: finishing write tid 6 to smithi10729407-8 2020-06-12T02:37:56.474 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 715 (ObjNum 4092 snap 0 seq_num 4092) dirty exists 2020-06-12T02:37:56.474 INFO:tasks.rados.rados.0.smithi107.stdout:4093: left oid 8 (ObjNum 4092 snap 0 seq_num 4092) 2020-06-12T02:37:56.475 INFO:tasks.rados.rados.0.smithi107.stdout:4093: done (1 left) 2020-06-12T02:37:56.475 INFO:tasks.rados.rados.0.smithi107.stdout:4096: done (0 left) 2020-06-12T02:37:56.475 INFO:tasks.rados.rados.0.smithi107.stdout:4097: delete oid 10 current snap is 0 2020-06-12T02:37:56.477 INFO:tasks.rados.rados.0.smithi107.stdout:4097: done (0 left) 2020-06-12T02:37:56.478 INFO:tasks.rados.rados.0.smithi107.stdout:4098: delete oid 3 current snap is 0 2020-06-12T02:37:56.480 INFO:tasks.rados.rados.0.smithi107.stdout:4098: done (0 left) 2020-06-12T02:37:56.480 INFO:tasks.rados.rados.0.smithi107.stdout:4099: delete oid 6 current snap is 0 2020-06-12T02:37:56.482 INFO:tasks.rados.rados.0.smithi107.stdout:4099: done (0 left) 2020-06-12T02:37:56.482 INFO:tasks.rados.rados.0.smithi107.stdout:4100: delete oid 13 current snap is 0 2020-06-12T02:37:56.484 INFO:tasks.rados.rados.0.smithi107.stdout:4100: done (0 left) 2020-06-12T02:37:56.484 INFO:tasks.rados.rados.0.smithi107.stdout:4101: delete oid 6 current snap is 0 2020-06-12T02:37:56.486 INFO:tasks.rados.rados.0.smithi107.stdout:4101: done (0 left) 2020-06-12T02:37:56.486 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:37:56.487 INFO:tasks.rados.rados.0.smithi107.stdout:4102: seq_num 4101 ranges {0=688128,688128=425984,1114112=376832} 2020-06-12T02:37:56.490 INFO:tasks.rados.rados.0.smithi107.stdout:4102: writing smithi10729407-6 from 0 to 688128 tid 1 2020-06-12T02:37:56.491 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:56.489+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:56.492 INFO:tasks.rados.rados.0.smithi107.stdout:4102: writing smithi10729407-6 from 688128 to 1114112 tid 2 2020-06-12T02:37:56.495 INFO:tasks.rados.rados.0.smithi107.stdout:4102: writing smithi10729407-6 from 1114112 to 1490944 tid 3 2020-06-12T02:37:56.495 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:37:56.495 INFO:tasks.rados.rados.0.smithi107.stdout:4103: seq_num 4102 ranges {3506176=704512,4210688=712704,4923392=90112} 2020-06-12T02:37:56.518 INFO:tasks.rados.rados.0.smithi107.stdout:4103: writing smithi10729407-18 from 3506176 to 4210688 tid 1 2020-06-12T02:37:56.523 INFO:tasks.rados.rados.0.smithi107.stdout:4103: writing smithi10729407-18 from 4210688 to 4923392 tid 2 2020-06-12T02:37:56.524 INFO:tasks.rados.rados.0.smithi107.stdout:4103: writing smithi10729407-18 from 4923392 to 5013504 tid 3 2020-06-12T02:37:56.525 INFO:tasks.rados.rados.0.smithi107.stdout:4104: delete oid 1 current snap is 0 2020-06-12T02:37:56.525 INFO:tasks.rados.rados.0.smithi107.stdout:4102: finishing write tid 1 to smithi10729407-6 2020-06-12T02:37:56.525 INFO:tasks.rados.rados.0.smithi107.stdout:4102: finishing write tid 2 to smithi10729407-6 2020-06-12T02:37:56.525 INFO:tasks.rados.rados.0.smithi107.stdout:4102: finishing write tid 3 to smithi10729407-6 2020-06-12T02:37:56.526 INFO:tasks.rados.rados.0.smithi107.stdout:4102: finishing write tid 5 to smithi10729407-6 2020-06-12T02:37:56.526 INFO:tasks.rados.rados.0.smithi107.stdout:4102: finishing write tid 6 to smithi10729407-6 2020-06-12T02:37:56.527 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 769 (ObjNum 4101 snap 0 seq_num 4101) dirty exists 2020-06-12T02:37:56.527 INFO:tasks.rados.rados.0.smithi107.stdout:4102: left oid 6 (ObjNum 4101 snap 0 seq_num 4101) 2020-06-12T02:37:56.528 INFO:tasks.rados.rados.0.smithi107.stdout:4103: finishing write tid 1 to smithi10729407-18 2020-06-12T02:37:56.528 INFO:tasks.rados.rados.0.smithi107.stdout:4102: done (2 left) 2020-06-12T02:37:56.529 INFO:tasks.rados.rados.0.smithi107.stdout:4104: done (1 left) 2020-06-12T02:37:56.529 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:37:56.529 INFO:tasks.rados.rados.0.smithi107.stdout:4105: seq_num 4104 ranges {0=737280,737280=483328,1220608=319488} 2020-06-12T02:37:56.535 INFO:tasks.rados.rados.0.smithi107.stdout:4105: writing smithi10729407-17 from 0 to 737280 tid 1 2020-06-12T02:37:56.535 INFO:tasks.rados.rados.0.smithi107.stdout:4105: writing smithi10729407-17 from 737280 to 1220608 tid 2 2020-06-12T02:37:56.536 INFO:tasks.rados.rados.0.smithi107.stdout:4105: writing smithi10729407-17 from 1220608 to 1540096 tid 3 2020-06-12T02:37:56.536 INFO:tasks.rados.rados.0.smithi107.stdout:4106: delete oid 4 current snap is 0 2020-06-12T02:37:56.552 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:56.552 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:56.567 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-06-12T02:37:56.569 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd unset nodeep-scrub 2020-06-12T02:37:56.611 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:56.609+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:56.683 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:56.683 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:56.691 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-06-12T02:37:56.694 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-06-12T02:37:56.695 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:37:56.692+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:37:56.795 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:37:56.793+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:37:56.831 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:37:56.848 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:56.848 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:56.852 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:37:56.856 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-06-12T02:37:56.883 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:37:56.895 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:37:56.898 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:37:56.895+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:37:56.914 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:37:56.930 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-06-12T02:37:56.935 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:37:56.955 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:37:56.976 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2020-06-12T02:37:56.977 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-06-12T02:37:56.978 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-06-12T02:37:56.978 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-06-12T02:37:56.978 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-06-12T02:37:56.978 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-06-12T02:37:56.978 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-06-12T02:37:56.979 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-06-12T02:37:56.979 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-06-12T02:37:56.998 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:56.996+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:57.020 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:37:57.021 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:37:57.021 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:37:57.021 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:37:57.022 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.022 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:858 3.3s0 3:ce45c70a:::smithi10729407-7:head [append 0~565248] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:57.022 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.924643+0000", 2020-06-12T02:37:57.022 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 25.091846734000001, 2020-06-12T02:37:57.022 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015332551999999999, 2020-06-12T02:37:57.023 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:57.023 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:57.023 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:57.024 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:57.025 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:57.026 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 858 2020-06-12T02:37:57.026 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.026 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:57.026 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.027 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:57.027 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.924643+0000", 2020-06-12T02:37:57.027 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:57.027 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.027 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.028 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:57.040 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.924643+0000", 2020-06-12T02:37:57.041 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00067535299999999996 2020-06-12T02:37:57.041 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.041 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.041 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:57.042 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925318+0000", 2020-06-12T02:37:57.047 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9629999999999993e-06 2020-06-12T02:37:57.047 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.048 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.048 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:57.048 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925328+0000", 2020-06-12T02:37:57.049 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7357999999999999e-05 2020-06-12T02:37:57.049 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.049 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.049 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:57.049 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925345+0000", 2020-06-12T02:37:57.051 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7019999999999996e-06 2020-06-12T02:37:57.051 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.051 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.051 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:57.051 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925355+0000", 2020-06-12T02:37:57.052 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6289999999999998e-05 2020-06-12T02:37:57.052 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.052 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.052 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.052 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925411+0000", 2020-06-12T02:37:57.053 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8855000000000004e-05 2020-06-12T02:37:57.053 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.053 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.053 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:57.053 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925480+0000", 2020-06-12T02:37:57.054 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0074521149999999996 2020-06-12T02:37:57.054 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.054 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.055 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.055 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.932932+0000", 2020-06-12T02:37:57.056 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3647999999999995e-05 2020-06-12T02:37:57.056 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.056 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.056 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:57.056 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.933026+0000", 2020-06-12T02:37:57.057 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00078753199999999997 2020-06-12T02:37:57.057 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.057 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.057 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:57.057 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.933813+0000", 2020-06-12T02:37:57.058 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.006013582 2020-06-12T02:37:57.058 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.058 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.058 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:57.059 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.939827+0000", 2020-06-12T02:37:57.059 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0111999999999996e-05 2020-06-12T02:37:57.059 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.059 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.059 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:57.060 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.939897+0000", 2020-06-12T02:37:57.060 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8041999999999996e-05 2020-06-12T02:37:57.060 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.061 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.061 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:57.061 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.939975+0000", 2020-06-12T02:37:57.061 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015332551999999999 2020-06-12T02:37:57.062 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.062 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:57.062 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.062 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.062 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.062 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:860 3.3s0 3:ce45c70a:::smithi10729407-7:head [append 0~81920] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:57.063 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.928619+0000", 2020-06-12T02:37:57.063 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 25.087870848000001, 2020-06-12T02:37:57.063 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017431941999999999, 2020-06-12T02:37:57.063 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:57.064 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:57.064 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:57.064 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:57.064 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:57.064 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 860 2020-06-12T02:37:57.065 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.065 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:57.065 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.065 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:57.065 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928619+0000", 2020-06-12T02:37:57.066 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:57.066 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.066 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.066 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:57.066 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928619+0000", 2020-06-12T02:37:57.067 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000105809 2020-06-12T02:37:57.067 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.067 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.067 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:57.067 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928724+0000", 2020-06-12T02:37:57.068 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2339999999999994e-06 2020-06-12T02:37:57.068 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.068 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.068 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:57.068 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928733+0000", 2020-06-12T02:37:57.069 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0268000000000001e-05 2020-06-12T02:37:57.069 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.069 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.069 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:57.069 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928743+0000", 2020-06-12T02:37:57.070 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5190000000000003e-06 2020-06-12T02:37:57.070 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.070 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.070 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:57.070 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928747+0000", 2020-06-12T02:37:57.071 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0103999999999998e-05 2020-06-12T02:37:57.071 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.072 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.072 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.072 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928777+0000", 2020-06-12T02:37:57.072 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8249000000000002e-05 2020-06-12T02:37:57.072 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.073 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.073 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:57.073 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928866+0000", 2020-06-12T02:37:57.073 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011151629 2020-06-12T02:37:57.073 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.074 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.074 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.074 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.940017+0000", 2020-06-12T02:37:57.074 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8281e-05 2020-06-12T02:37:57.074 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.075 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.075 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:57.075 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.940076+0000", 2020-06-12T02:37:57.075 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000346736 2020-06-12T02:37:57.075 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.076 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.076 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:57.076 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.940422+0000", 2020-06-12T02:37:57.076 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024135049999999998 2020-06-12T02:37:57.077 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.077 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.077 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:57.077 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.942836+0000", 2020-06-12T02:37:57.077 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003170258 2020-06-12T02:37:57.078 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.078 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.078 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:57.078 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946006+0000", 2020-06-12T02:37:57.079 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4350000000000001e-05 2020-06-12T02:37:57.079 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.079 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.079 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:57.079 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946050+0000", 2020-06-12T02:37:57.080 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017431941999999999 2020-06-12T02:37:57.080 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.080 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:57.081 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.081 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.081 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.081 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:861 3.3s0 3:ce45c70a:::smithi10729407-7:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:57.082 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.928778+0000", 2020-06-12T02:37:57.082 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 25.087711235, 2020-06-12T02:37:57.082 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017522738, 2020-06-12T02:37:57.082 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:57.082 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:57.083 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:57.083 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:57.083 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:57.083 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 861 2020-06-12T02:37:57.083 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.084 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:57.084 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.084 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:57.084 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928778+0000", 2020-06-12T02:37:57.085 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:57.085 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.085 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.085 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:57.086 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928778+0000", 2020-06-12T02:37:57.086 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8186000000000002e-05 2020-06-12T02:37:57.086 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.086 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.086 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:57.086 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928826+0000", 2020-06-12T02:37:57.087 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6689999999999997e-06 2020-06-12T02:37:57.087 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.087 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.087 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:57.087 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928831+0000", 2020-06-12T02:37:57.088 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.686e-06 2020-06-12T02:37:57.088 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.088 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.088 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:57.089 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928838+0000", 2020-06-12T02:37:57.089 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7159999999999999e-06 2020-06-12T02:37:57.089 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.089 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.089 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:57.090 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928840+0000", 2020-06-12T02:37:57.090 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4212999999999997e-05 2020-06-12T02:37:57.090 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.090 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.090 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.091 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928895+0000", 2020-06-12T02:37:57.091 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5853999999999994e-05 2020-06-12T02:37:57.091 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.092 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.092 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:57.092 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928990+0000", 2020-06-12T02:37:57.092 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012023982000000001 2020-06-12T02:37:57.092 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.093 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.093 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.093 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941014+0000", 2020-06-12T02:37:57.093 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5104e-05 2020-06-12T02:37:57.094 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.094 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.094 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:57.094 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941070+0000", 2020-06-12T02:37:57.094 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00026444499999999998 2020-06-12T02:37:57.095 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.095 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.095 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:57.095 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941334+0000", 2020-06-12T02:37:57.095 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001704075 2020-06-12T02:37:57.096 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.096 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.096 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:57.096 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.943038+0000", 2020-06-12T02:37:57.096 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031891570000000002 2020-06-12T02:37:57.097 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.097 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.097 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:57.097 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946227+0000", 2020-06-12T02:37:57.097 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3651000000000003e-05 2020-06-12T02:37:57.098 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.098 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.098 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:57.098 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946301+0000", 2020-06-12T02:37:57.098 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017522738 2020-06-12T02:37:57.099 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.099 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:57.099 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.099 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.099 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.099 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:862 3.3s0 3:ce45c70a:::smithi10729407-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:57.100 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.928876+0000", 2020-06-12T02:37:57.100 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 25.087613738999998, 2020-06-12T02:37:57.100 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019206890000000001, 2020-06-12T02:37:57.100 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:57.100 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:57.101 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:57.101 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:57.101 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:57.101 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 862 2020-06-12T02:37:57.102 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.102 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:57.102 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.102 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:57.103 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928876+0000", 2020-06-12T02:37:57.103 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:57.103 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.103 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.103 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:57.104 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928876+0000", 2020-06-12T02:37:57.104 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9268999999999999e-05 2020-06-12T02:37:57.104 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.104 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.104 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:57.105 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928925+0000", 2020-06-12T02:37:57.105 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8120000000000002e-06 2020-06-12T02:37:57.105 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.105 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.106 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:57.106 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928932+0000", 2020-06-12T02:37:57.106 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3471e-05 2020-06-12T02:37:57.106 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.106 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.106 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:57.107 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928945+0000", 2020-06-12T02:37:57.107 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4279999999999998e-06 2020-06-12T02:37:57.107 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.107 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.107 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:57.108 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928950+0000", 2020-06-12T02:37:57.108 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6358000000000004e-05 2020-06-12T02:37:57.108 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.108 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.108 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.109 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.929016+0000", 2020-06-12T02:37:57.109 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9697999999999996e-05 2020-06-12T02:37:57.109 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.109 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.109 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:57.110 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.929086+0000", 2020-06-12T02:37:57.110 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012409205 2020-06-12T02:37:57.110 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.110 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.111 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.111 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941495+0000", 2020-06-12T02:37:57.111 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9660000000000006e-05 2020-06-12T02:37:57.111 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.111 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.112 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:57.112 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941585+0000", 2020-06-12T02:37:57.112 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050329099999999998 2020-06-12T02:37:57.112 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.113 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.113 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.113 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946617+0000", 2020-06-12T02:37:57.113 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8437999999999996e-05 2020-06-12T02:37:57.114 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.114 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.114 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:57.114 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946706+0000", 2020-06-12T02:37:57.114 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0013766410000000001 2020-06-12T02:37:57.115 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.115 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.115 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:57.115 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.948083+0000", 2020-06-12T02:37:57.115 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019206890000000001 2020-06-12T02:37:57.115 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.116 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:57.116 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.116 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.116 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.116 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:879 3.fs1 3:fc7e0bf7:::smithi10729407-17:head [append 0~491520] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:57.117 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.000881+0000", 2020-06-12T02:37:57.117 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 25.015608558, 2020-06-12T02:37:57.117 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017891845, 2020-06-12T02:37:57.117 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:57.118 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:57.118 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:57.118 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:57.118 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:57.118 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 879 2020-06-12T02:37:57.119 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.119 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:57.119 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.119 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:57.119 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.000881+0000", 2020-06-12T02:37:57.120 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:57.120 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.120 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.120 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:57.120 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.000881+0000", 2020-06-12T02:37:57.121 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00044151599999999998 2020-06-12T02:37:57.121 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.121 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.121 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:57.122 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001322+0000", 2020-06-12T02:37:57.122 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0210000000000001e-06 2020-06-12T02:37:57.122 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.122 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.123 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:57.123 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001326+0000", 2020-06-12T02:37:57.123 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6659999999999997e-06 2020-06-12T02:37:57.123 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.123 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.124 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:57.124 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001335+0000", 2020-06-12T02:37:57.124 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4299999999999999e-06 2020-06-12T02:37:57.124 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.124 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.125 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:57.125 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001339+0000", 2020-06-12T02:37:57.125 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0028741510000000001 2020-06-12T02:37:57.125 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.125 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.126 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.126 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.004214+0000", 2020-06-12T02:37:57.126 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010480999999999999 2020-06-12T02:37:57.126 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.126 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.127 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:57.127 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.004318+0000", 2020-06-12T02:37:57.127 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021112359999999998 2020-06-12T02:37:57.127 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.127 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.128 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.128 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006430+0000", 2020-06-12T02:37:57.128 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2935000000000001e-05 2020-06-12T02:37:57.128 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.129 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.129 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:57.129 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006483+0000", 2020-06-12T02:37:57.129 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00056411099999999995 2020-06-12T02:37:57.130 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.130 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.130 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:57.130 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.007047+0000", 2020-06-12T02:37:57.130 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0076749369999999997 2020-06-12T02:37:57.131 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.131 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.131 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:57.132 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.014722+0000", 2020-06-12T02:37:57.132 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00395693 2020-06-12T02:37:57.132 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.133 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.134 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:57.134 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.018679+0000", 2020-06-12T02:37:57.134 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4102000000000001e-05 2020-06-12T02:37:57.134 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.134 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.135 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:57.135 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.018773+0000", 2020-06-12T02:37:57.135 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017891845 2020-06-12T02:37:57.135 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.135 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:57.135 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.136 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.136 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.136 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:933 3.es1 3:78f01648:::smithi10729407-5:head [append 0~417792] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:57.136 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.156677+0000", 2020-06-12T02:37:57.136 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 24.859812575999999, 2020-06-12T02:37:57.137 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016833311, 2020-06-12T02:37:57.137 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:57.137 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:57.137 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:57.137 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:57.138 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:57.138 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 933 2020-06-12T02:37:57.138 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.138 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:57.139 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.139 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:57.139 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.156677+0000", 2020-06-12T02:37:57.139 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:57.139 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.140 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.140 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:57.140 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.156677+0000", 2020-06-12T02:37:57.140 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00062571700000000003 2020-06-12T02:37:57.140 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.141 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.141 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:57.141 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157303+0000", 2020-06-12T02:37:57.141 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0740000000000003e-06 2020-06-12T02:37:57.141 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.141 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.142 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:57.142 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157307+0000", 2020-06-12T02:37:57.142 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4419999999999998e-06 2020-06-12T02:37:57.142 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.142 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.143 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:57.143 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157315+0000", 2020-06-12T02:37:57.143 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5300000000000001e-06 2020-06-12T02:37:57.143 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.143 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.143 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:57.146 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157319+0000", 2020-06-12T02:37:57.146 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4995e-05 2020-06-12T02:37:57.146 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.146 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.147 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.147 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157354+0000", 2020-06-12T02:37:57.147 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5883000000000006e-05 2020-06-12T02:37:57.147 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.147 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.148 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:57.148 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157419+0000", 2020-06-12T02:37:57.148 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045316280000000002 2020-06-12T02:37:57.148 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.148 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.148 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.149 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.161951+0000", 2020-06-12T02:37:57.149 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3300999999999996e-05 2020-06-12T02:37:57.149 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.150 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.150 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:57.150 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.162014+0000", 2020-06-12T02:37:57.150 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00039050100000000002 2020-06-12T02:37:57.150 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.151 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.151 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:57.151 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.162405+0000", 2020-06-12T02:37:57.151 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050686250000000002 2020-06-12T02:37:57.151 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.152 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.152 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:57.152 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.167474+0000", 2020-06-12T02:37:57.152 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.005942653 2020-06-12T02:37:57.152 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.152 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.153 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:57.153 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.173416+0000", 2020-06-12T02:37:57.153 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3962000000000006e-05 2020-06-12T02:37:57.153 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.153 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.154 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:57.154 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.173510+0000", 2020-06-12T02:37:57.154 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016833311 2020-06-12T02:37:57.154 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.154 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:57.154 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.155 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.155 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.155 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:936 3.es1 3:78f01648:::smithi10729407-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:57.155 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.160357+0000", 2020-06-12T02:37:57.155 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 24.856132864999999, 2020-06-12T02:37:57.156 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015435590000000001, 2020-06-12T02:37:57.156 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:57.156 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:57.156 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:57.156 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:57.156 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:57.157 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 936 2020-06-12T02:37:57.157 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.157 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:57.157 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.157 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:57.158 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160357+0000", 2020-06-12T02:37:57.158 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:57.158 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.158 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.158 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:57.158 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160357+0000", 2020-06-12T02:37:57.159 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4435999999999999e-05 2020-06-12T02:37:57.159 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.159 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.160 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:57.160 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160411+0000", 2020-06-12T02:37:57.160 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2329999999999995e-06 2020-06-12T02:37:57.160 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.160 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.160 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:57.161 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160420+0000", 2020-06-12T02:37:57.161 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7003e-05 2020-06-12T02:37:57.161 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.161 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.161 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:57.162 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160437+0000", 2020-06-12T02:37:57.162 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8370000000000002e-06 2020-06-12T02:37:57.162 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.162 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.162 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:57.163 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160443+0000", 2020-06-12T02:37:57.163 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0070635489999999997 2020-06-12T02:37:57.163 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.163 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.163 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.163 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.167507+0000", 2020-06-12T02:37:57.164 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010589700000000001 2020-06-12T02:37:57.164 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.164 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.164 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:57.164 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.167612+0000", 2020-06-12T02:37:57.164 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0065032299999999996 2020-06-12T02:37:57.165 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.165 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.165 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.165 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.174116+0000", 2020-06-12T02:37:57.165 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3128999999999999e-05 2020-06-12T02:37:57.166 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.166 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.166 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:57.166 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.174169+0000", 2020-06-12T02:37:57.166 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0016232760000000001 2020-06-12T02:37:57.167 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.167 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.167 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:57.167 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.175792+0000", 2020-06-12T02:37:57.167 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015435590000000001 2020-06-12T02:37:57.168 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.168 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:57.168 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.168 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.168 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.168 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:950 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~778240] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:37:57.169 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.207643+0000", 2020-06-12T02:37:57.169 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 24.808846282000001, 2020-06-12T02:37:57.169 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014844661, 2020-06-12T02:37:57.169 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:57.170 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:57.170 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:57.170 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:57.170 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:57.171 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 950 2020-06-12T02:37:57.171 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.171 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:57.171 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.171 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:57.171 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.207643+0000", 2020-06-12T02:37:57.172 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:57.172 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.172 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.172 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:57.172 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.207643+0000", 2020-06-12T02:37:57.173 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00072843099999999998 2020-06-12T02:37:57.173 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.173 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.173 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:57.173 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208372+0000", 2020-06-12T02:37:57.173 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9120000000000001e-06 2020-06-12T02:37:57.174 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.174 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.174 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:57.174 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208378+0000", 2020-06-12T02:37:57.174 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9660000000000008e-06 2020-06-12T02:37:57.175 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.175 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.175 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:57.175 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208388+0000", 2020-06-12T02:37:57.175 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1279999999999999e-06 2020-06-12T02:37:57.175 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.176 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.176 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:57.176 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208394+0000", 2020-06-12T02:37:57.176 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0439999999999998e-05 2020-06-12T02:37:57.176 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.177 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.177 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.177 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208444+0000", 2020-06-12T02:37:57.177 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2611000000000001e-05 2020-06-12T02:37:57.177 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.177 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.178 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:57.178 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208507+0000", 2020-06-12T02:37:57.178 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001228301 2020-06-12T02:37:57.178 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.178 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.179 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.179 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.209735+0000", 2020-06-12T02:37:57.179 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8875000000000001e-05 2020-06-12T02:37:57.179 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.179 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.179 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:57.180 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.209794+0000", 2020-06-12T02:37:57.180 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053218899999999997 2020-06-12T02:37:57.180 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.181 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.181 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:57.181 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.210326+0000", 2020-06-12T02:37:57.181 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0078198439999999994 2020-06-12T02:37:57.181 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.182 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.182 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:57.182 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.218146+0000", 2020-06-12T02:37:57.182 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004300496 2020-06-12T02:37:57.182 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.182 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.183 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:57.183 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.222446+0000", 2020-06-12T02:37:57.183 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1467999999999997e-05 2020-06-12T02:37:57.183 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.183 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.184 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:57.184 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.222488+0000", 2020-06-12T02:37:57.184 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014844661 2020-06-12T02:37:57.184 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.184 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:57.184 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.185 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.185 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.185 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:954 3.3s0 3:c2ca77da:::smithi10729407-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:57.185 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.212923+0000", 2020-06-12T02:37:57.185 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 24.803566505999999, 2020-06-12T02:37:57.186 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015411546999999999, 2020-06-12T02:37:57.186 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:57.186 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:57.186 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:57.186 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:57.187 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:57.187 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 954 2020-06-12T02:37:57.187 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.187 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:57.187 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.187 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:57.188 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212923+0000", 2020-06-12T02:37:57.188 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:57.188 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.188 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.188 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:57.189 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212923+0000", 2020-06-12T02:37:57.189 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3598e-05 2020-06-12T02:37:57.189 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.189 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.189 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:57.189 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212946+0000", 2020-06-12T02:37:57.190 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.061e-06 2020-06-12T02:37:57.190 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.190 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.190 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:57.191 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212950+0000", 2020-06-12T02:37:57.191 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8700000000000003e-06 2020-06-12T02:37:57.191 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.191 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.191 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:57.192 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212956+0000", 2020-06-12T02:37:57.192 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0770000000000002e-06 2020-06-12T02:37:57.192 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.192 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.192 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:57.193 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212958+0000", 2020-06-12T02:37:57.193 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051235170000000002 2020-06-12T02:37:57.193 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.193 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.193 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.193 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.218082+0000", 2020-06-12T02:37:57.194 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9246999999999997e-05 2020-06-12T02:37:57.194 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.194 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.194 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:57.194 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.218141+0000", 2020-06-12T02:37:57.195 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0087981910000000003 2020-06-12T02:37:57.195 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.195 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.195 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.195 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.226939+0000", 2020-06-12T02:37:57.195 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4119999999999997e-05 2020-06-12T02:37:57.196 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.196 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.196 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:57.196 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.226994+0000", 2020-06-12T02:37:57.196 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001340866 2020-06-12T02:37:57.197 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.197 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.197 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:57.197 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.228334+0000", 2020-06-12T02:37:57.197 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015411546999999999 2020-06-12T02:37:57.197 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.198 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:57.198 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.198 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.198 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.198 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1097 3.es1 3:78f01648:::smithi10729407-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:57.199 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.576496+0000", 2020-06-12T02:37:57.199 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 24.439993024, 2020-06-12T02:37:57.199 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014992914, 2020-06-12T02:37:57.199 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:57.199 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:57.199 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:57.200 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:57.200 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:57.200 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1097 2020-06-12T02:37:57.200 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.201 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:57.201 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.201 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:57.201 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576496+0000", 2020-06-12T02:37:57.201 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:57.202 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.202 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.202 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:57.202 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576496+0000", 2020-06-12T02:37:57.202 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3221999999999999e-05 2020-06-12T02:37:57.203 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.203 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.203 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:57.203 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576520+0000", 2020-06-12T02:37:57.203 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.988e-06 2020-06-12T02:37:57.203 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.204 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.204 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:57.204 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576523+0000", 2020-06-12T02:37:57.204 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2940000000000002e-06 2020-06-12T02:37:57.204 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.205 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.205 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:57.205 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576529+0000", 2020-06-12T02:37:57.205 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6060000000000002e-06 2020-06-12T02:37:57.205 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.206 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.206 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:57.206 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576534+0000", 2020-06-12T02:37:57.206 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001254554 2020-06-12T02:37:57.206 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.206 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.207 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.207 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.577789+0000", 2020-06-12T02:37:57.207 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3177999999999998e-05 2020-06-12T02:37:57.207 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.207 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.208 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:57.208 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.577842+0000", 2020-06-12T02:37:57.208 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0063091220000000003 2020-06-12T02:37:57.208 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.208 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.208 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.209 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.584151+0000", 2020-06-12T02:37:57.209 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.783e-05 2020-06-12T02:37:57.209 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.209 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.209 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:57.210 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.584209+0000", 2020-06-12T02:37:57.210 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045863650000000002 2020-06-12T02:37:57.210 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.210 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.210 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.210 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.588796+0000", 2020-06-12T02:37:57.211 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7614000000000006e-05 2020-06-12T02:37:57.211 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.211 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.212 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:57.212 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.588873+0000", 2020-06-12T02:37:57.212 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0026161410000000002 2020-06-12T02:37:57.212 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.212 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.212 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:57.213 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.591489+0000", 2020-06-12T02:37:57.213 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014992914 2020-06-12T02:37:57.213 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.213 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:57.213 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.214 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.214 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.214 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:4813 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~450560] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-06-12T02:37:57.214 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:43.343096+0000", 2020-06-12T02:37:57.214 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 13.673393681, 2020-06-12T02:37:57.214 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014820242000000001, 2020-06-12T02:37:57.215 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:57.215 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:57.215 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:57.215 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:57.215 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:57.216 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 4813 2020-06-12T02:37:57.216 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.216 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:57.216 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.216 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:57.216 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343096+0000", 2020-06-12T02:37:57.217 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:57.217 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.217 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.217 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:57.217 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343096+0000", 2020-06-12T02:37:57.218 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00068963 2020-06-12T02:37:57.218 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.218 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.218 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:57.218 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343785+0000", 2020-06-12T02:37:57.218 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1479999999999997e-06 2020-06-12T02:37:57.219 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.219 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.219 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:57.219 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343790+0000", 2020-06-12T02:37:57.219 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4060000000000005e-06 2020-06-12T02:37:57.220 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.220 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.220 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:57.220 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343799+0000", 2020-06-12T02:37:57.220 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0970000000000002e-06 2020-06-12T02:37:57.220 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.221 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.221 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:57.221 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343805+0000", 2020-06-12T02:37:57.222 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00023736499999999999 2020-06-12T02:37:57.222 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.222 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.222 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.222 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.344042+0000", 2020-06-12T02:37:57.223 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000113866 2020-06-12T02:37:57.223 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.223 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.223 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:57.223 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.344156+0000", 2020-06-12T02:37:57.223 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036114760000000002 2020-06-12T02:37:57.224 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.224 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.224 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.224 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.347768+0000", 2020-06-12T02:37:57.224 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1842999999999996e-05 2020-06-12T02:37:57.225 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.225 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.225 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:57.225 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.347850+0000", 2020-06-12T02:37:57.225 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00048835999999999997 2020-06-12T02:37:57.225 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.226 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.226 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:57.226 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.348338+0000", 2020-06-12T02:37:57.226 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0049373139999999999 2020-06-12T02:37:57.226 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.227 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.227 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:57.227 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.353275+0000", 2020-06-12T02:37:57.227 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045727119999999996 2020-06-12T02:37:57.227 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.227 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.228 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:57.228 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.357848+0000", 2020-06-12T02:37:57.228 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8025e-05 2020-06-12T02:37:57.228 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.228 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.229 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:57.229 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.357916+0000", 2020-06-12T02:37:57.229 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014820242000000001 2020-06-12T02:37:57.229 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.229 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:57.229 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.230 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.230 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.230 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:4921 3.9s0 3:93ea0cad:::smithi10729407-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-06-12T02:37:57.230 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:43.595696+0000", 2020-06-12T02:37:57.230 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 13.420793314999999, 2020-06-12T02:37:57.231 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016503190000000001, 2020-06-12T02:37:57.231 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:57.231 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:57.231 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:57.232 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:57.232 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:57.232 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 4921 2020-06-12T02:37:57.232 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.232 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:57.233 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.233 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:57.233 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595696+0000", 2020-06-12T02:37:57.233 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:57.233 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.234 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.234 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:57.234 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595696+0000", 2020-06-12T02:37:57.234 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3694999999999999e-05 2020-06-12T02:37:57.234 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.235 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.235 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:57.235 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595720+0000", 2020-06-12T02:37:57.235 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1700000000000001e-06 2020-06-12T02:37:57.235 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.235 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.236 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:57.236 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595723+0000", 2020-06-12T02:37:57.236 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8849999999999999e-06 2020-06-12T02:37:57.236 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.236 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.237 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:57.237 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595731+0000", 2020-06-12T02:37:57.237 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5689999999999999e-06 2020-06-12T02:37:57.237 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.237 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.237 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:57.238 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595733+0000", 2020-06-12T02:37:57.238 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0030116880000000002 2020-06-12T02:37:57.238 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.238 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.238 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.239 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.598745+0000", 2020-06-12T02:37:57.239 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1353999999999996e-05 2020-06-12T02:37:57.239 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.239 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.239 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:57.239 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.598806+0000", 2020-06-12T02:37:57.240 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051955819999999998 2020-06-12T02:37:57.240 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.240 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.240 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.240 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.604002+0000", 2020-06-12T02:37:57.241 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9615000000000001e-05 2020-06-12T02:37:57.241 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.241 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.241 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:57.241 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.604102+0000", 2020-06-12T02:37:57.242 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0052050960000000002 2020-06-12T02:37:57.242 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.242 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.242 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.243 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.609307+0000", 2020-06-12T02:37:57.243 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5025000000000003e-05 2020-06-12T02:37:57.243 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.243 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.243 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:57.243 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.609362+0000", 2020-06-12T02:37:57.244 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002837511 2020-06-12T02:37:57.244 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.244 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.244 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:57.244 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.612199+0000", 2020-06-12T02:37:57.245 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016503190000000001 2020-06-12T02:37:57.245 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.245 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:57.245 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.245 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.246 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.246 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:5068 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~663552] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e30)", 2020-06-12T02:37:57.246 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:44.157497+0000", 2020-06-12T02:37:57.246 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 12.858992689000001, 2020-06-12T02:37:57.246 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015164129, 2020-06-12T02:37:57.246 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:57.247 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:57.247 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:57.247 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:57.247 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:57.247 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 5068 2020-06-12T02:37:57.248 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.248 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:57.248 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.248 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:57.248 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.157497+0000", 2020-06-12T02:37:57.248 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:57.249 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.249 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.249 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:57.249 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.157497+0000", 2020-06-12T02:37:57.249 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00068511099999999997 2020-06-12T02:37:57.250 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.250 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.250 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:57.250 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.158182+0000", 2020-06-12T02:37:57.250 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4630000000000004e-06 2020-06-12T02:37:57.250 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.251 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.251 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:57.251 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.158189+0000", 2020-06-12T02:37:57.251 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4199000000000001e-05 2020-06-12T02:37:57.251 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.252 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.252 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:57.253 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.158203+0000", 2020-06-12T02:37:57.253 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1250000000000004e-06 2020-06-12T02:37:57.253 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.254 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.254 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:57.254 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.158211+0000", 2020-06-12T02:37:57.254 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6754000000000002e-05 2020-06-12T02:37:57.255 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.255 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.255 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.255 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.158237+0000", 2020-06-12T02:37:57.255 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2948999999999996e-05 2020-06-12T02:37:57.256 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.256 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.256 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:57.256 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.158310+0000", 2020-06-12T02:37:57.256 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0032401610000000001 2020-06-12T02:37:57.257 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.257 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.257 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.257 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.161550+0000", 2020-06-12T02:37:57.257 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5550000000000002e-05 2020-06-12T02:37:57.258 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.258 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.258 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:57.258 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.161606+0000", 2020-06-12T02:37:57.258 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00062448199999999999 2020-06-12T02:37:57.258 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.259 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.259 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:57.259 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.162230+0000", 2020-06-12T02:37:57.259 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0073257130000000002 2020-06-12T02:37:57.259 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.260 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.260 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:57.260 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.169556+0000", 2020-06-12T02:37:57.260 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0030226419999999999 2020-06-12T02:37:57.261 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.261 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.261 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:57.261 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.172579+0000", 2020-06-12T02:37:57.261 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1979999999999998e-05 2020-06-12T02:37:57.261 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.262 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.262 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:57.262 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.172661+0000", 2020-06-12T02:37:57.262 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015164129 2020-06-12T02:37:57.263 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.263 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:57.263 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.263 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.263 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.264 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:5071 3.3s0 3:c2ca77da:::smithi10729407-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-06-12T02:37:57.264 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:44.159788+0000", 2020-06-12T02:37:57.264 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 12.856701467000001, 2020-06-12T02:37:57.264 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016812273999999999, 2020-06-12T02:37:57.264 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:57.265 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:57.265 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:57.265 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:57.265 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:57.265 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 5071 2020-06-12T02:37:57.265 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.266 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:57.266 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.266 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:57.266 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159788+0000", 2020-06-12T02:37:57.266 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:57.267 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.267 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.267 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:57.267 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159788+0000", 2020-06-12T02:37:57.267 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1421000000000001e-05 2020-06-12T02:37:57.267 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.268 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.268 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:57.268 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159839+0000", 2020-06-12T02:37:57.268 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9009999999999994e-06 2020-06-12T02:37:57.269 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.269 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.269 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:57.269 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159847+0000", 2020-06-12T02:37:57.269 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3149999999999999e-05 2020-06-12T02:37:57.270 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.270 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.270 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:57.270 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159860+0000", 2020-06-12T02:37:57.271 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1749999999999996e-06 2020-06-12T02:37:57.271 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.271 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.271 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:57.271 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159866+0000", 2020-06-12T02:37:57.272 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6035999999999999e-05 2020-06-12T02:37:57.272 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.272 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.272 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.273 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159892+0000", 2020-06-12T02:37:57.273 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2627999999999998e-05 2020-06-12T02:37:57.273 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.273 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.273 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:57.274 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159944+0000", 2020-06-12T02:37:57.274 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010270133000000001 2020-06-12T02:37:57.274 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.274 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.274 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.275 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.170214+0000", 2020-06-12T02:37:57.275 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5490999999999998e-05 2020-06-12T02:37:57.275 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.275 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.275 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:57.275 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.170310+0000", 2020-06-12T02:37:57.276 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003215828 2020-06-12T02:37:57.276 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.276 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.276 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.277 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.173526+0000", 2020-06-12T02:37:57.277 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0290000000000005e-05 2020-06-12T02:37:57.277 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.277 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.278 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:57.278 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.173616+0000", 2020-06-12T02:37:57.278 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002984221 2020-06-12T02:37:57.278 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.279 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.279 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:57.279 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.176600+0000", 2020-06-12T02:37:57.279 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016812273999999999 2020-06-12T02:37:57.280 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.280 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:57.280 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.281 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.281 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.281 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:6432 3.9s0 3:93ea0cad:::smithi10729407-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e31)", 2020-06-12T02:37:57.282 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:47.960104+0000", 2020-06-12T02:37:57.282 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.0563853709999993, 2020-06-12T02:37:57.282 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016025815999999998, 2020-06-12T02:37:57.282 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:57.282 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:57.283 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:57.283 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:57.283 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:57.283 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 6432 2020-06-12T02:37:57.283 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.284 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:57.284 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.284 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:57.284 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960104+0000", 2020-06-12T02:37:57.284 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:57.284 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.285 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.285 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:57.285 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960104+0000", 2020-06-12T02:37:57.285 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2351999999999999e-05 2020-06-12T02:37:57.285 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.286 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.286 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:57.286 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960126+0000", 2020-06-12T02:37:57.287 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.061e-06 2020-06-12T02:37:57.287 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.287 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.287 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:57.288 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960129+0000", 2020-06-12T02:37:57.288 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4849999999999997e-06 2020-06-12T02:37:57.288 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.289 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.289 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:57.289 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960136+0000", 2020-06-12T02:37:57.290 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.176e-06 2020-06-12T02:37:57.290 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.290 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.291 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:57.291 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960138+0000", 2020-06-12T02:37:57.291 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0044054139999999999 2020-06-12T02:37:57.292 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.292 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.292 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.293 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.964543+0000", 2020-06-12T02:37:57.293 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8077000000000006e-05 2020-06-12T02:37:57.294 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.294 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.294 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:57.295 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.964612+0000", 2020-06-12T02:37:57.295 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0029921150000000001 2020-06-12T02:37:57.296 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.296 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.296 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.296 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.967604+0000", 2020-06-12T02:37:57.297 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2983000000000001e-05 2020-06-12T02:37:57.297 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.298 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.298 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:57.299 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.967697+0000", 2020-06-12T02:37:57.299 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0072496510000000002 2020-06-12T02:37:57.299 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.300 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.300 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.300 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.974946+0000", 2020-06-12T02:37:57.301 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8278e-05 2020-06-12T02:37:57.301 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.302 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.302 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:57.302 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.975005+0000", 2020-06-12T02:37:57.303 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001125224 2020-06-12T02:37:57.303 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.304 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.304 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:57.304 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.976130+0000", 2020-06-12T02:37:57.305 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016025815999999998 2020-06-12T02:37:57.305 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.306 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:57.306 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.306 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.307 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.307 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8123 3.es1 3:78f01648:::smithi10729407-5:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-06-12T02:37:57.308 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.690721+0000", 2020-06-12T02:37:57.308 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.3257682669999999, 2020-06-12T02:37:57.308 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016503126999999999, 2020-06-12T02:37:57.308 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:57.309 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:57.309 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:57.309 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:57.310 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:57.310 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8123 2020-06-12T02:37:57.311 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.311 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:57.311 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.312 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:57.312 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690721+0000", 2020-06-12T02:37:57.312 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:57.313 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.313 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.313 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:57.314 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690721+0000", 2020-06-12T02:37:57.314 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7526e-05 2020-06-12T02:37:57.314 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.314 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.315 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:57.315 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690759+0000", 2020-06-12T02:37:57.315 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1999999999999999e-06 2020-06-12T02:37:57.315 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.315 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.316 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:57.316 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690762+0000", 2020-06-12T02:37:57.316 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9229999999999999e-06 2020-06-12T02:37:57.316 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.317 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.317 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:57.317 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690768+0000", 2020-06-12T02:37:57.317 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4480000000000001e-06 2020-06-12T02:37:57.317 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.318 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.318 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:57.318 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690770+0000", 2020-06-12T02:37:57.318 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00079317799999999996 2020-06-12T02:37:57.319 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.319 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.319 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.319 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.691563+0000", 2020-06-12T02:37:57.320 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5299e-05 2020-06-12T02:37:57.320 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.320 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.320 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:57.321 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.691619+0000", 2020-06-12T02:37:57.321 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010070076000000001 2020-06-12T02:37:57.321 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.321 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.322 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.322 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.701689+0000", 2020-06-12T02:37:57.322 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000110234 2020-06-12T02:37:57.323 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.323 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.323 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:57.323 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.701799+0000", 2020-06-12T02:37:57.324 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00037311100000000003 2020-06-12T02:37:57.324 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.324 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.324 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:57.326 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.702172+0000", 2020-06-12T02:37:57.327 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0044460050000000003 2020-06-12T02:37:57.327 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.327 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.327 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:57.327 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.706618+0000", 2020-06-12T02:37:57.328 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00043550199999999999 2020-06-12T02:37:57.328 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.328 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.328 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:57.328 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.707054+0000", 2020-06-12T02:37:57.329 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017062499999999999 2020-06-12T02:37:57.329 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.329 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.329 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:57.329 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.707224+0000", 2020-06-12T02:37:57.330 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016503126999999999 2020-06-12T02:37:57.330 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.330 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:57.330 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.330 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.330 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.331 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8124 3.es1 3:78f01648:::smithi10729407-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e33)", 2020-06-12T02:37:57.331 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.690794+0000", 2020-06-12T02:37:57.331 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.3256950810000001, 2020-06-12T02:37:57.331 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019298497000000001, 2020-06-12T02:37:57.331 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:57.332 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:57.332 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:57.332 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:57.332 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:57.332 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8124 2020-06-12T02:37:57.332 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.333 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:57.333 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.333 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:57.333 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690794+0000", 2020-06-12T02:37:57.333 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:57.334 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.334 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.334 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:57.334 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690794+0000", 2020-06-12T02:37:57.334 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1869999999999999e-05 2020-06-12T02:37:57.335 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.335 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.335 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:57.335 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690816+0000", 2020-06-12T02:37:57.335 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.162e-06 2020-06-12T02:37:57.336 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.336 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.336 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:57.336 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690819+0000", 2020-06-12T02:37:57.336 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2280000000000003e-06 2020-06-12T02:37:57.336 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.337 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.337 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:57.337 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690826+0000", 2020-06-12T02:37:57.337 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.452e-06 2020-06-12T02:37:57.337 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.338 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.338 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:57.338 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690828+0000", 2020-06-12T02:37:57.338 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00081402600000000003 2020-06-12T02:37:57.338 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.338 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.339 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.339 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.691642+0000", 2020-06-12T02:37:57.339 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00024263 2020-06-12T02:37:57.339 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.339 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.340 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:57.340 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.691885+0000", 2020-06-12T02:37:57.340 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010968739 2020-06-12T02:37:57.340 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.340 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.341 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.341 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.702853+0000", 2020-06-12T02:37:57.341 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8985000000000002e-05 2020-06-12T02:37:57.341 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.341 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.341 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:57.342 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.702952+0000", 2020-06-12T02:37:57.342 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0046973329999999997 2020-06-12T02:37:57.342 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.342 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.342 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.343 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.707650+0000", 2020-06-12T02:37:57.343 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3773e-05 2020-06-12T02:37:57.343 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.343 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.343 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:57.343 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.707723+0000", 2020-06-12T02:37:57.344 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023692990000000001 2020-06-12T02:37:57.344 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.344 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.344 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:57.344 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.710093+0000", 2020-06-12T02:37:57.345 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019298497000000001 2020-06-12T02:37:57.345 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.345 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:57.345 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.345 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.346 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.346 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8190 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~753664] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-06-12T02:37:57.346 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.894592+0000", 2020-06-12T02:37:57.346 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.1218974230000001, 2020-06-12T02:37:57.346 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016165763999999999, 2020-06-12T02:37:57.346 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:57.347 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:57.347 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:57.347 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:57.347 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:57.347 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8190 2020-06-12T02:37:57.348 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.350 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:57.350 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.350 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:57.351 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.894592+0000", 2020-06-12T02:37:57.351 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:57.351 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.351 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.351 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:57.352 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.894592+0000", 2020-06-12T02:37:57.352 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00074881500000000001 2020-06-12T02:37:57.352 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.352 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.352 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:57.353 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.895341+0000", 2020-06-12T02:37:57.353 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1919999999999999e-06 2020-06-12T02:37:57.353 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.354 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.354 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:57.354 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.895348+0000", 2020-06-12T02:37:57.354 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.059e-05 2020-06-12T02:37:57.354 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.355 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.355 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:57.355 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.895359+0000", 2020-06-12T02:37:57.355 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2310000000000001e-06 2020-06-12T02:37:57.355 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.355 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.356 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:57.356 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.895365+0000", 2020-06-12T02:37:57.356 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00049215800000000003 2020-06-12T02:37:57.356 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.356 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.357 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.357 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.895857+0000", 2020-06-12T02:37:57.357 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1001999999999995e-05 2020-06-12T02:37:57.357 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.357 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.358 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:57.358 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.895938+0000", 2020-06-12T02:37:57.358 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0059219040000000004 2020-06-12T02:37:57.358 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.358 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.359 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.359 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.901860+0000", 2020-06-12T02:37:57.359 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010156499999999999 2020-06-12T02:37:57.359 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.360 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.360 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:57.360 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.901961+0000", 2020-06-12T02:37:57.360 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00085242800000000002 2020-06-12T02:37:57.360 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.360 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.361 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:57.361 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.902814+0000", 2020-06-12T02:37:57.361 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0057806180000000004 2020-06-12T02:37:57.361 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.361 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.362 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:57.362 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.908594+0000", 2020-06-12T02:37:57.362 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021229600000000001 2020-06-12T02:37:57.362 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.362 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.363 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:57.363 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.910717+0000", 2020-06-12T02:37:57.363 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0300999999999999e-05 2020-06-12T02:37:57.363 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.364 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.364 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:57.364 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.910758+0000", 2020-06-12T02:37:57.364 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016165763999999999 2020-06-12T02:37:57.364 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.365 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:57.365 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.365 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.365 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.365 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8191 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~450560] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-06-12T02:37:57.365 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.897242+0000", 2020-06-12T02:37:57.366 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.1192469699999998, 2020-06-12T02:37:57.366 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014857576000000001, 2020-06-12T02:37:57.366 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:57.366 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:57.366 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:57.367 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:57.367 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:57.367 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8191 2020-06-12T02:37:57.367 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.367 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:57.367 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.368 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:57.368 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897242+0000", 2020-06-12T02:37:57.368 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:57.368 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.368 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.369 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:57.369 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897242+0000", 2020-06-12T02:37:57.369 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000531592 2020-06-12T02:37:57.369 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.369 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.370 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:57.370 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897774+0000", 2020-06-12T02:37:57.370 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0288999999999999e-05 2020-06-12T02:37:57.370 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.370 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.371 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:57.371 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897784+0000", 2020-06-12T02:37:57.371 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7149999999999993e-06 2020-06-12T02:37:57.371 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.371 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.371 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:57.372 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897794+0000", 2020-06-12T02:37:57.372 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8960000000000002e-06 2020-06-12T02:37:57.372 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.372 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.372 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:57.373 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897799+0000", 2020-06-12T02:37:57.373 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6251999999999998e-05 2020-06-12T02:37:57.373 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.373 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.373 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.374 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897835+0000", 2020-06-12T02:37:57.374 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.768e-05 2020-06-12T02:37:57.374 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.374 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.374 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:57.374 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897933+0000", 2020-06-12T02:37:57.375 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0082988100000000002 2020-06-12T02:37:57.375 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.375 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.375 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.375 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.906232+0000", 2020-06-12T02:37:57.376 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7865999999999998e-05 2020-06-12T02:37:57.376 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.376 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.376 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:57.376 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.906290+0000", 2020-06-12T02:37:57.377 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047895700000000002 2020-06-12T02:37:57.377 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.377 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.377 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:57.377 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.906768+0000", 2020-06-12T02:37:57.377 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002658442 2020-06-12T02:37:57.378 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.378 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.378 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:57.378 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.909427+0000", 2020-06-12T02:37:57.378 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0026066230000000002 2020-06-12T02:37:57.379 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.379 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.379 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:57.379 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.912034+0000", 2020-06-12T02:37:57.379 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6453999999999998e-05 2020-06-12T02:37:57.380 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.380 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.380 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:57.380 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.912100+0000", 2020-06-12T02:37:57.380 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014857576000000001 2020-06-12T02:37:57.380 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.381 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:57.381 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.381 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.381 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.381 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8193 3.3s0 3:c2ca77da:::smithi10729407-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e33)", 2020-06-12T02:37:57.382 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.897900+0000", 2020-06-12T02:37:57.382 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.1185890379999996, 2020-06-12T02:37:57.382 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015951213999999998, 2020-06-12T02:37:57.382 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:57.382 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:57.383 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:57.383 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:57.383 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:57.383 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8193 2020-06-12T02:37:57.383 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.383 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:57.384 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.384 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:57.384 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897900+0000", 2020-06-12T02:37:57.384 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:57.384 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.385 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.385 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:57.385 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897900+0000", 2020-06-12T02:37:57.385 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2331e-05 2020-06-12T02:37:57.385 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.386 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.386 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:57.386 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897923+0000", 2020-06-12T02:37:57.386 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.021e-06 2020-06-12T02:37:57.386 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.386 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.387 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:57.387 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897926+0000", 2020-06-12T02:37:57.387 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4150000000000002e-06 2020-06-12T02:37:57.387 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.387 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.388 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:57.388 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897933+0000", 2020-06-12T02:37:57.388 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2359999999999999e-06 2020-06-12T02:37:57.388 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.388 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.389 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:57.389 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897935+0000", 2020-06-12T02:37:57.389 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000118301 2020-06-12T02:37:57.389 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.389 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.390 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.390 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.898054+0000", 2020-06-12T02:37:57.390 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8914999999999998e-05 2020-06-12T02:37:57.390 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.390 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.390 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:57.391 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.898143+0000", 2020-06-12T02:37:57.391 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011203013 2020-06-12T02:37:57.391 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.391 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.391 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.392 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.909346+0000", 2020-06-12T02:37:57.392 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7075000000000003e-05 2020-06-12T02:37:57.392 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.392 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.392 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:57.392 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.909423+0000", 2020-06-12T02:37:57.393 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033600969999999998 2020-06-12T02:37:57.393 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.393 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.393 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.393 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.912783+0000", 2020-06-12T02:37:57.394 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6118000000000005e-05 2020-06-12T02:37:57.394 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.394 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.394 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:57.394 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.912869+0000", 2020-06-12T02:37:57.394 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00098269200000000007 2020-06-12T02:37:57.395 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.395 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.395 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:57.395 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.913852+0000", 2020-06-12T02:37:57.395 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015951213999999998 2020-06-12T02:37:57.396 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.396 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:57.396 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.396 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.396 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:57.396 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:37:57.398 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-06-12T02:37:57.399 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:37:57.399 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:37:57.399 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2020-06-12T02:37:57.400 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:37:57.400 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd reweight 0 1 2020-06-12T02:37:57.402 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:57.399+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:37:57.500 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:37:57.498+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:37:57.535 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:37:57.536 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:37:57.536 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2020-06-12T02:37:57.536 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:37:57.544 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-06-12T02:37:57.569 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2020-06-12T02:37:57.569 INFO:tasks.rados.rados.0.smithi107.stdout:4103: finishing write tid 2 to smithi10729407-18 2020-06-12T02:37:57.570 INFO:tasks.rados.rados.0.smithi107.stdout:4103: finishing write tid 3 to smithi10729407-18 2020-06-12T02:37:57.570 INFO:tasks.rados.rados.0.smithi107.stdout:4103: finishing write tid 5 to smithi10729407-18 2020-06-12T02:37:57.571 INFO:tasks.rados.rados.0.smithi107.stdout:4103: finishing write tid 6 to smithi10729407-18 2020-06-12T02:37:57.571 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 732 (ObjNum 4102 snap 0 seq_num 4102) dirty exists 2020-06-12T02:37:57.571 INFO:tasks.rados.rados.0.smithi107.stdout:4103: left oid 18 (ObjNum 4102 snap 0 seq_num 4102) 2020-06-12T02:37:57.599 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:37:57.597+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:37:57.667 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:37:57.667 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:37:57.667 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:37:57.667 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:37:57.668 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:37:57.675 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-06-12T02:37:57.717 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:57.714+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:37:57.802 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:57.799+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:37:57.834 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:37:57.834 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:37:57.834 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:37:57.835 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:37:57.835 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.835 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1206 3.5s0 3:a9510bda:::smithi10729407-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:37:57.835 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.856688+0000", 2020-06-12T02:37:57.835 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 24.973620931999999, 2020-06-12T02:37:57.836 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019116942000000001, 2020-06-12T02:37:57.836 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:57.836 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:57.836 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:57.836 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:57.837 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:57.837 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1206 2020-06-12T02:37:57.837 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.837 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:57.837 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.838 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:57.838 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856688+0000", 2020-06-12T02:37:57.838 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:57.839 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.839 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.839 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:57.839 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856688+0000", 2020-06-12T02:37:57.839 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4355e-05 2020-06-12T02:37:57.840 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.840 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.840 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:57.840 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856713+0000", 2020-06-12T02:37:57.840 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0850000000000001e-06 2020-06-12T02:37:57.841 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.841 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.841 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:57.841 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856716+0000", 2020-06-12T02:37:57.841 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4509999999999997e-06 2020-06-12T02:37:57.842 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.842 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.842 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:57.842 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856722+0000", 2020-06-12T02:37:57.842 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2879999999999999e-06 2020-06-12T02:37:57.843 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.843 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.844 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:57.844 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856724+0000", 2020-06-12T02:37:57.844 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0057507449999999998 2020-06-12T02:37:57.844 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.845 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.845 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.845 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.862475+0000", 2020-06-12T02:37:57.845 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5759000000000003e-05 2020-06-12T02:37:57.845 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.846 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.846 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:57.846 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.862571+0000", 2020-06-12T02:37:57.846 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012199018000000001 2020-06-12T02:37:57.846 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.847 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.847 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.847 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.874770+0000", 2020-06-12T02:37:57.847 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7260999999999999e-05 2020-06-12T02:37:57.847 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.848 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.849 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:57.849 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.874827+0000", 2020-06-12T02:37:57.850 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00097798000000000004 2020-06-12T02:37:57.850 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.850 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.850 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:57.850 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.875805+0000", 2020-06-12T02:37:57.851 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019116942000000001 2020-06-12T02:37:57.851 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.851 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:57.851 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.851 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.852 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.852 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1572 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~745472] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-06-12T02:37:57.852 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.886439+0000", 2020-06-12T02:37:57.852 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 23.943870214, 2020-06-12T02:37:57.852 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021442571000000001, 2020-06-12T02:37:57.853 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:57.853 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:57.853 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:57.853 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:57.853 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:57.854 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1572 2020-06-12T02:37:57.854 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.854 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:57.854 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.854 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:57.855 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.886439+0000", 2020-06-12T02:37:57.855 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:57.855 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.855 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.855 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:57.855 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.886439+0000", 2020-06-12T02:37:57.856 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00073027599999999997 2020-06-12T02:37:57.856 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.856 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.856 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:57.856 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887169+0000", 2020-06-12T02:37:57.857 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.49e-06 2020-06-12T02:37:57.857 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.857 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.857 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:57.857 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887175+0000", 2020-06-12T02:37:57.858 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7899999999999994e-06 2020-06-12T02:37:57.858 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.858 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.859 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:57.859 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887185+0000", 2020-06-12T02:37:57.859 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.42e-06 2020-06-12T02:37:57.859 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.860 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.860 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:57.860 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887189+0000", 2020-06-12T02:37:57.860 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1619999999999999e-05 2020-06-12T02:37:57.860 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.861 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.861 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.861 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887221+0000", 2020-06-12T02:37:57.861 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9720000000000002e-05 2020-06-12T02:37:57.861 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.862 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.862 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:57.862 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887300+0000", 2020-06-12T02:37:57.862 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047354100000000001 2020-06-12T02:37:57.862 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.862 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.863 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:57.863 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887774+0000", 2020-06-12T02:37:57.863 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0030649750000000002 2020-06-12T02:37:57.863 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.863 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.864 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:57.864 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.890839+0000", 2020-06-12T02:37:57.864 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016995321000000001 2020-06-12T02:37:57.864 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.865 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.865 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:57.865 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.907834+0000", 2020-06-12T02:37:57.865 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7417999999999999e-05 2020-06-12T02:37:57.865 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.866 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.866 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:57.866 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.907882+0000", 2020-06-12T02:37:57.866 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021442571000000001 2020-06-12T02:37:57.866 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.867 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:57.867 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.867 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.867 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.867 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1573 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~712704] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-06-12T02:37:57.867 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.890888+0000", 2020-06-12T02:37:57.868 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 23.939421479, 2020-06-12T02:37:57.868 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018868001999999998, 2020-06-12T02:37:57.868 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:57.868 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:57.871 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:57.872 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:57.873 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:57.873 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1573 2020-06-12T02:37:57.873 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.873 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:57.874 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.874 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:57.874 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.890888+0000", 2020-06-12T02:37:57.874 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:57.874 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.875 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.875 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:57.875 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.890888+0000", 2020-06-12T02:37:57.875 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000683605 2020-06-12T02:37:57.875 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.876 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.876 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:57.876 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891571+0000", 2020-06-12T02:37:57.876 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4999999999999999e-06 2020-06-12T02:37:57.876 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.876 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.877 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:57.877 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891577+0000", 2020-06-12T02:37:57.877 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.171e-05 2020-06-12T02:37:57.877 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.877 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.878 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:57.878 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891589+0000", 2020-06-12T02:37:57.878 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7650000000000001e-06 2020-06-12T02:37:57.878 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.878 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.878 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:57.879 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891593+0000", 2020-06-12T02:37:57.879 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6689000000000001e-05 2020-06-12T02:37:57.879 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.880 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.880 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.880 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891630+0000", 2020-06-12T02:37:57.880 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9877999999999997e-05 2020-06-12T02:37:57.880 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.881 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.881 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:57.887 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891710+0000", 2020-06-12T02:37:57.888 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053367200000000001 2020-06-12T02:37:57.888 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.888 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.888 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:57.888 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.892244+0000", 2020-06-12T02:37:57.889 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004927196 2020-06-12T02:37:57.889 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.889 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.889 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:57.890 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.897171+0000", 2020-06-12T02:37:57.890 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012496419 2020-06-12T02:37:57.890 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.890 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.890 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:57.891 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.909667+0000", 2020-06-12T02:37:57.891 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8567999999999996e-05 2020-06-12T02:37:57.891 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.891 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.891 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:57.891 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.909756+0000", 2020-06-12T02:37:57.892 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018868001999999998 2020-06-12T02:37:57.892 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.892 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:57.892 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.892 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.893 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.893 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1576 3.5s0 3:a446366d:::smithi10729407-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e26)", 2020-06-12T02:37:57.893 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.891955+0000", 2020-06-12T02:37:57.893 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 23.938354238999999, 2020-06-12T02:37:57.893 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019893246999999999, 2020-06-12T02:37:57.894 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:57.894 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:57.894 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:57.894 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:57.894 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:57.894 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1576 2020-06-12T02:37:57.895 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.895 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:57.895 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.895 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:57.895 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891955+0000", 2020-06-12T02:37:57.896 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:57.896 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.896 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.896 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:57.896 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891955+0000", 2020-06-12T02:37:57.896 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2067999999999999e-05 2020-06-12T02:37:57.897 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.897 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.897 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:57.897 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891977+0000", 2020-06-12T02:37:57.897 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.18e-06 2020-06-12T02:37:57.898 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.898 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.898 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:57.898 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891980+0000", 2020-06-12T02:37:57.898 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3629999999999999e-06 2020-06-12T02:37:57.899 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.899 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.899 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:57.899 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891987+0000", 2020-06-12T02:37:57.900 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.181e-06 2020-06-12T02:37:57.900 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.900 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.900 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:57.900 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891989+0000", 2020-06-12T02:37:57.900 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050959120000000002 2020-06-12T02:37:57.901 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.901 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.901 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.901 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.897085+0000", 2020-06-12T02:37:57.901 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1089999999999998e-05 2020-06-12T02:37:57.902 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.902 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.902 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:57.902 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.897166+0000", 2020-06-12T02:37:57.902 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013400235 2020-06-12T02:37:57.903 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.903 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.903 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.903 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.910566+0000", 2020-06-12T02:37:57.903 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000104981 2020-06-12T02:37:57.903 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.904 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.904 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:57.904 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.910671+0000", 2020-06-12T02:37:57.904 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001177237 2020-06-12T02:37:57.904 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.905 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.905 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:57.909 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.911848+0000", 2020-06-12T02:37:57.909 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019893246999999999 2020-06-12T02:37:57.909 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.909 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:57.910 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.910 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.910 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.910 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2913 3.5s0 3:a9510bda:::smithi10729407-14:head [append 0~770048,stat] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:57.910 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.070354+0000", 2020-06-12T02:37:57.911 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.759955437999999, 2020-06-12T02:37:57.911 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60690798599999995, 2020-06-12T02:37:57.911 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:57.911 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:57.911 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:57.911 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:57.912 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:57.912 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2913 2020-06-12T02:37:57.912 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.912 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:57.912 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.913 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:57.913 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.070354+0000", 2020-06-12T02:37:57.913 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:57.913 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.913 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.914 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:57.914 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.070354+0000", 2020-06-12T02:37:57.914 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00098478900000000002 2020-06-12T02:37:57.914 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.914 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.914 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:57.915 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071339+0000", 2020-06-12T02:37:57.915 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8349999999999993e-06 2020-06-12T02:37:57.915 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.915 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.915 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:57.916 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071347+0000", 2020-06-12T02:37:57.916 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0749e-05 2020-06-12T02:37:57.916 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.916 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.916 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:57.917 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071358+0000", 2020-06-12T02:37:57.917 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9369999999999997e-06 2020-06-12T02:37:57.918 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.918 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.918 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:57.918 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071362+0000", 2020-06-12T02:37:57.918 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0426e-05 2020-06-12T02:37:57.919 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.919 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.919 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.919 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071423+0000", 2020-06-12T02:37:57.919 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4129999999999998e-05 2020-06-12T02:37:57.920 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.920 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.920 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:57.920 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071487+0000", 2020-06-12T02:37:57.920 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053752599999999998 2020-06-12T02:37:57.920 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.921 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.921 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:57.921 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072024+0000", 2020-06-12T02:37:57.921 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036162579999999998 2020-06-12T02:37:57.921 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.922 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.922 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:57.922 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075640+0000", 2020-06-12T02:37:57.922 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60152201999999999 2020-06-12T02:37:57.922 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.923 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.923 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:57.923 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677162+0000", 2020-06-12T02:37:57.923 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9315999999999996e-05 2020-06-12T02:37:57.923 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.924 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.924 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:57.924 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677262+0000", 2020-06-12T02:37:57.924 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60690798599999995 2020-06-12T02:37:57.925 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.925 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:57.925 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.925 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.925 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.926 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2914 3.5s0 3:a9510bda:::smithi10729407-14:head [append 0~401408] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:57.926 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.072409+0000", 2020-06-12T02:37:57.926 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.757899993999999, 2020-06-12T02:37:57.926 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.61956801100000003, 2020-06-12T02:37:57.926 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:57.926 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:57.927 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:57.927 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:57.927 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:57.927 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2914 2020-06-12T02:37:57.927 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.928 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:57.928 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.928 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:57.928 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072409+0000", 2020-06-12T02:37:57.928 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:57.928 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.929 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.929 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:57.929 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072409+0000", 2020-06-12T02:37:57.929 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000446317 2020-06-12T02:37:57.929 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.930 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.930 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:57.930 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072856+0000", 2020-06-12T02:37:57.930 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2140000000000004e-06 2020-06-12T02:37:57.930 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.931 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.931 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:57.931 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072861+0000", 2020-06-12T02:37:57.931 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8710000000000003e-06 2020-06-12T02:37:57.932 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.932 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.932 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:57.932 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072870+0000", 2020-06-12T02:37:57.932 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2270000000000004e-06 2020-06-12T02:37:57.933 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.933 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.933 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:57.933 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072874+0000", 2020-06-12T02:37:57.933 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024649960000000001 2020-06-12T02:37:57.934 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.934 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.934 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.934 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075339+0000", 2020-06-12T02:37:57.934 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9179999999999996e-05 2020-06-12T02:37:57.935 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.935 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.935 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:57.935 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075438+0000", 2020-06-12T02:37:57.935 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60189281400000005 2020-06-12T02:37:57.936 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.936 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.936 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.936 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677331+0000", 2020-06-12T02:37:57.936 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012602800000000001 2020-06-12T02:37:57.936 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.937 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.937 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:57.937 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677457+0000", 2020-06-12T02:37:57.937 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00088167699999999998 2020-06-12T02:37:57.937 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.938 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.938 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:57.938 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.678339+0000", 2020-06-12T02:37:57.938 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004312995 2020-06-12T02:37:57.938 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.939 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.939 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:57.939 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.682652+0000", 2020-06-12T02:37:57.939 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0092869140000000003 2020-06-12T02:37:57.939 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.939 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.940 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:57.940 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691938+0000", 2020-06-12T02:37:57.940 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8778e-05 2020-06-12T02:37:57.940 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.940 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.941 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:57.941 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691977+0000", 2020-06-12T02:37:57.941 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.61956801100000003 2020-06-12T02:37:57.941 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.941 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:57.942 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.942 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.942 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.942 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2915 3.5s0 3:a9510bda:::smithi10729407-14:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:57.942 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.072912+0000", 2020-06-12T02:37:57.943 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.757397254000001, 2020-06-12T02:37:57.943 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.6192204, 2020-06-12T02:37:57.943 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:57.943 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:57.944 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:57.944 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:57.944 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:57.944 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2915 2020-06-12T02:37:57.944 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.944 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:57.945 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.945 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:57.945 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072912+0000", 2020-06-12T02:37:57.945 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:57.945 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.946 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.946 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:57.946 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072912+0000", 2020-06-12T02:37:57.946 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9201e-05 2020-06-12T02:37:57.946 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.947 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.947 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:57.947 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072951+0000", 2020-06-12T02:37:57.947 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5480000000000002e-06 2020-06-12T02:37:57.947 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.948 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.948 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:57.948 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072955+0000", 2020-06-12T02:37:57.948 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3879999999999997e-06 2020-06-12T02:37:57.948 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.949 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.949 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:57.949 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072961+0000", 2020-06-12T02:37:57.949 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6589999999999999e-06 2020-06-12T02:37:57.949 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.950 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.950 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:57.950 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072964+0000", 2020-06-12T02:37:57.950 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025221039999999998 2020-06-12T02:37:57.950 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.951 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.951 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.951 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075486+0000", 2020-06-12T02:37:57.951 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5038999999999998e-05 2020-06-12T02:37:57.951 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.951 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.952 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:57.952 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075561+0000", 2020-06-12T02:37:57.952 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60452397499999999 2020-06-12T02:37:57.952 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.952 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.953 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.953 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680085+0000", 2020-06-12T02:37:57.953 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1398999999999999e-05 2020-06-12T02:37:57.953 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.953 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.953 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:57.954 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680156+0000", 2020-06-12T02:37:57.954 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000274465 2020-06-12T02:37:57.954 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.954 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.954 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:57.955 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680431+0000", 2020-06-12T02:37:57.955 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025071999999999998 2020-06-12T02:37:57.955 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.955 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.955 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:57.955 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.682938+0000", 2020-06-12T02:37:57.956 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0091116449999999998 2020-06-12T02:37:57.956 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.956 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.956 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:57.956 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692050+0000", 2020-06-12T02:37:57.957 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2776999999999997e-05 2020-06-12T02:37:57.957 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.957 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.957 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:57.957 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692132+0000", 2020-06-12T02:37:57.958 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.6192204 2020-06-12T02:37:57.958 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.958 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:57.958 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.958 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.958 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.959 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2916 3.5s0 3:a9510bda:::smithi10729407-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:37:57.959 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.072986+0000", 2020-06-12T02:37:57.959 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.757323015000001, 2020-06-12T02:37:57.959 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.62063941099999997, 2020-06-12T02:37:57.959 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:57.960 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:57.960 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:57.960 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:57.960 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:57.960 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2916 2020-06-12T02:37:57.960 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.961 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:57.961 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.961 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:57.961 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072986+0000", 2020-06-12T02:37:57.961 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:57.962 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.962 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.962 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:57.962 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072986+0000", 2020-06-12T02:37:57.962 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2101000000000001e-05 2020-06-12T02:37:57.962 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.963 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.963 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:57.963 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073008+0000", 2020-06-12T02:37:57.963 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1549999999999999e-06 2020-06-12T02:37:57.966 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.966 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.967 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:57.968 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073011+0000", 2020-06-12T02:37:57.968 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2360000000000001e-06 2020-06-12T02:37:57.968 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.968 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.969 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:57.969 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073018+0000", 2020-06-12T02:37:57.969 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2790000000000001e-06 2020-06-12T02:37:57.969 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.969 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.970 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:57.970 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073020+0000", 2020-06-12T02:37:57.970 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025637670000000002 2020-06-12T02:37:57.970 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.970 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.971 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.971 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075584+0000", 2020-06-12T02:37:57.971 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1814999999999998e-05 2020-06-12T02:37:57.971 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.971 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.972 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:57.972 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075636+0000", 2020-06-12T02:37:57.973 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60499943199999995 2020-06-12T02:37:57.973 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.974 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.974 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.974 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680635+0000", 2020-06-12T02:37:57.975 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0968000000000002e-05 2020-06-12T02:37:57.975 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.975 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.975 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:57.975 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680706+0000", 2020-06-12T02:37:57.976 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011777101999999999 2020-06-12T02:37:57.976 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.976 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.976 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.976 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692483+0000", 2020-06-12T02:37:57.977 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5890999999999995e-05 2020-06-12T02:37:57.977 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.977 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.977 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:57.977 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692569+0000", 2020-06-12T02:37:57.978 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001056665 2020-06-12T02:37:57.978 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.978 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.978 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:57.978 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.693626+0000", 2020-06-12T02:37:57.978 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.62063941099999997 2020-06-12T02:37:57.979 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.979 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:57.979 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.979 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.979 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.980 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2922 3.ds1 3:b983fd28:::smithi10729407-1:head [append 0~466944] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:57.980 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.097192+0000", 2020-06-12T02:37:57.980 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.733117648, 2020-06-12T02:37:57.980 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58672644699999998, 2020-06-12T02:37:57.980 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:57.981 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:57.981 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:57.981 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:57.981 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:57.981 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2922 2020-06-12T02:37:57.982 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.982 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:57.982 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.983 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:57.983 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097192+0000", 2020-06-12T02:37:57.983 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:57.983 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.983 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.983 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:57.984 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097192+0000", 2020-06-12T02:37:57.984 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047206100000000002 2020-06-12T02:37:57.984 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.984 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.984 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:57.985 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097664+0000", 2020-06-12T02:37:57.985 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8359999999999999e-06 2020-06-12T02:37:57.985 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.985 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.985 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:57.986 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097668+0000", 2020-06-12T02:37:57.986 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0815999999999999e-05 2020-06-12T02:37:57.986 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.986 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.986 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:57.987 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097679+0000", 2020-06-12T02:37:57.987 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.028e-06 2020-06-12T02:37:57.987 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.987 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.988 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:57.988 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097684+0000", 2020-06-12T02:37:57.988 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3932999999999999e-05 2020-06-12T02:37:57.988 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.988 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.988 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:57.989 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097748+0000", 2020-06-12T02:37:57.989 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2214999999999999e-05 2020-06-12T02:37:57.989 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.989 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.989 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:57.990 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097820+0000", 2020-06-12T02:37:57.990 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00041583800000000003 2020-06-12T02:37:57.990 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.990 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.990 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:57.991 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.098236+0000", 2020-06-12T02:37:57.991 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0020339960000000002 2020-06-12T02:37:57.991 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.993 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.993 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:57.993 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.100270+0000", 2020-06-12T02:37:57.994 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58355657299999997 2020-06-12T02:37:57.994 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.994 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.994 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:57.994 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.683827+0000", 2020-06-12T02:37:57.995 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1150999999999995e-05 2020-06-12T02:37:57.995 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.995 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.995 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:57.995 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.683918+0000", 2020-06-12T02:37:57.995 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58672644699999998 2020-06-12T02:37:57.996 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.996 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:57.996 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:57.996 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.996 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.997 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2923 3.ds1 3:b983fd28:::smithi10729407-1:head [append 0~614400] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:57.997 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.100504+0000", 2020-06-12T02:37:57.997 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.729805007, 2020-06-12T02:37:57.997 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58853946999999995, 2020-06-12T02:37:57.997 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:57.998 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:57.998 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:57.998 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:57.998 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:57.998 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2923 2020-06-12T02:37:57.998 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:57.999 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:57.999 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:57.999 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:57.999 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.100504+0000", 2020-06-12T02:37:57.999 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:58.000 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.000 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.000 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:58.000 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.100504+0000", 2020-06-12T02:37:58.000 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00068395399999999996 2020-06-12T02:37:58.000 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.001 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.001 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:58.001 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101188+0000", 2020-06-12T02:37:58.001 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4050000000000003e-06 2020-06-12T02:37:58.001 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.002 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.002 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:58.002 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101196+0000", 2020-06-12T02:37:58.002 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.328e-05 2020-06-12T02:37:58.002 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.002 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.003 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:58.004 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101209+0000", 2020-06-12T02:37:58.004 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3419999999999996e-06 2020-06-12T02:37:58.004 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.005 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.005 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:58.005 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101215+0000", 2020-06-12T02:37:58.005 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1389000000000005e-05 2020-06-12T02:37:58.005 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.006 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.006 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.006 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101277+0000", 2020-06-12T02:37:58.007 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7724000000000004e-05 2020-06-12T02:37:58.007 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.007 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.007 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:58.007 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101344+0000", 2020-06-12T02:37:58.007 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047852499999999999 2020-06-12T02:37:58.008 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.008 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.008 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:58.008 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101823+0000", 2020-06-12T02:37:58.008 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0039255510000000002 2020-06-12T02:37:58.009 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.009 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.009 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:58.009 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105748+0000", 2020-06-12T02:37:58.009 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58321340099999996 2020-06-12T02:37:58.010 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.010 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.010 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:58.010 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.688962+0000", 2020-06-12T02:37:58.010 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1898999999999996e-05 2020-06-12T02:37:58.011 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.011 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.011 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:58.011 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689044+0000", 2020-06-12T02:37:58.012 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58853946999999995 2020-06-12T02:37:58.012 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.012 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:58.012 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.012 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.013 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.013 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2924 3.ds1 3:b983fd28:::smithi10729407-1:head [append 0~188416] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:58.013 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.101633+0000", 2020-06-12T02:37:58.013 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.728676013000001, 2020-06-12T02:37:58.013 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58758259199999996, 2020-06-12T02:37:58.013 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:58.014 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:58.014 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:58.014 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:58.015 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:58.015 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2924 2020-06-12T02:37:58.015 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.015 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:58.015 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.015 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:58.016 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101633+0000", 2020-06-12T02:37:58.016 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:58.016 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.016 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.017 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:58.017 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101633+0000", 2020-06-12T02:37:58.017 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00018775399999999999 2020-06-12T02:37:58.017 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.018 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.018 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:58.018 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101821+0000", 2020-06-12T02:37:58.018 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7669999999999998e-06 2020-06-12T02:37:58.018 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.019 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.019 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:58.019 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101828+0000", 2020-06-12T02:37:58.019 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2716e-05 2020-06-12T02:37:58.019 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.019 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.020 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:58.020 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101840+0000", 2020-06-12T02:37:58.020 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.215e-06 2020-06-12T02:37:58.020 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.020 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.021 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:58.021 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101846+0000", 2020-06-12T02:37:58.021 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021214250000000001 2020-06-12T02:37:58.021 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.021 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.021 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.022 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.103967+0000", 2020-06-12T02:37:58.022 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1018000000000003e-05 2020-06-12T02:37:58.022 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.022 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.022 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:58.023 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.104038+0000", 2020-06-12T02:37:58.023 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000333364 2020-06-12T02:37:58.023 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.023 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.024 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:58.024 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.104371+0000", 2020-06-12T02:37:58.024 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001378546 2020-06-12T02:37:58.024 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.025 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.025 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:58.025 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105750+0000", 2020-06-12T02:37:58.025 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58338272899999999 2020-06-12T02:37:58.025 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.026 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.026 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:58.026 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689133+0000", 2020-06-12T02:37:58.026 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3058000000000003e-05 2020-06-12T02:37:58.026 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.027 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.027 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:58.027 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689216+0000", 2020-06-12T02:37:58.027 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58758259199999996 2020-06-12T02:37:58.028 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.028 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:58.028 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.028 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.028 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.028 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2925 3.ds1 3:b983fd28:::smithi10729407-1:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:58.029 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.101899+0000", 2020-06-12T02:37:58.029 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.728410348000001, 2020-06-12T02:37:58.029 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58747810899999997, 2020-06-12T02:37:58.029 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:58.029 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:58.030 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:58.030 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:58.030 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:58.030 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2925 2020-06-12T02:37:58.030 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.031 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:58.031 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.031 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:58.031 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101899+0000", 2020-06-12T02:37:58.031 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:58.031 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.032 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.032 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:58.032 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101899+0000", 2020-06-12T02:37:58.032 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2906999999999998e-05 2020-06-12T02:37:58.032 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.033 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.033 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:58.033 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101972+0000", 2020-06-12T02:37:58.033 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1789999999999996e-06 2020-06-12T02:37:58.033 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.033 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.034 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:58.034 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101978+0000", 2020-06-12T02:37:58.034 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1358e-05 2020-06-12T02:37:58.035 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.035 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.035 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:58.035 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101989+0000", 2020-06-12T02:37:58.035 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.031e-06 2020-06-12T02:37:58.036 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.036 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.036 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:58.036 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101993+0000", 2020-06-12T02:37:58.036 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033218990000000001 2020-06-12T02:37:58.037 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.037 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.037 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.037 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105315+0000", 2020-06-12T02:37:58.038 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6324999999999999e-05 2020-06-12T02:37:58.038 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.038 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.038 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:58.038 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105382+0000", 2020-06-12T02:37:58.039 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000216624 2020-06-12T02:37:58.039 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.039 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.039 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:58.039 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105598+0000", 2020-06-12T02:37:58.039 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001236586 2020-06-12T02:37:58.040 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.040 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.040 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:58.040 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.106835+0000", 2020-06-12T02:37:58.040 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58245509200000001 2020-06-12T02:37:58.041 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.041 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.041 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:58.041 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689290+0000", 2020-06-12T02:37:58.041 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7107999999999993e-05 2020-06-12T02:37:58.041 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.042 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.042 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:58.042 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689377+0000", 2020-06-12T02:37:58.042 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58747810899999997 2020-06-12T02:37:58.042 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.043 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:58.043 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.043 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.043 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.043 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2926 3.ds1 3:b983fd28:::smithi10729407-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:37:58.043 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.102036+0000", 2020-06-12T02:37:58.044 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.728273364, 2020-06-12T02:37:58.044 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58868048699999997, 2020-06-12T02:37:58.044 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:58.044 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:58.044 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:58.045 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:58.045 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:58.045 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2926 2020-06-12T02:37:58.045 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.046 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:58.046 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.046 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:58.046 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102036+0000", 2020-06-12T02:37:58.046 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:58.047 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.047 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.047 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:58.047 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102036+0000", 2020-06-12T02:37:58.047 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3786999999999999e-05 2020-06-12T02:37:58.048 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.048 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.048 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:58.048 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102080+0000", 2020-06-12T02:37:58.048 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7660000000000004e-06 2020-06-12T02:37:58.049 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.049 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.049 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:58.049 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102085+0000", 2020-06-12T02:37:58.049 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3145e-05 2020-06-12T02:37:58.050 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.050 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.050 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:58.050 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102099+0000", 2020-06-12T02:37:58.050 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2440000000000004e-06 2020-06-12T02:37:58.051 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.051 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.051 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:58.051 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102103+0000", 2020-06-12T02:37:58.051 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036826990000000002 2020-06-12T02:37:58.051 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.052 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.052 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.052 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105786+0000", 2020-06-12T02:37:58.052 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.6774000000000006e-05 2020-06-12T02:37:58.052 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.053 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.053 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:58.053 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105882+0000", 2020-06-12T02:37:58.053 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58352084900000001 2020-06-12T02:37:58.053 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.053 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.054 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.054 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689403+0000", 2020-06-12T02:37:58.054 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3788999999999995e-05 2020-06-12T02:37:58.054 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.054 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.055 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:58.055 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689467+0000", 2020-06-12T02:37:58.055 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001249434 2020-06-12T02:37:58.055 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.056 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.056 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:58.056 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690716+0000", 2020-06-12T02:37:58.056 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58868048699999997 2020-06-12T02:37:58.056 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.057 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:58.057 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.057 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.057 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.057 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2950 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~598016] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:58.057 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.162562+0000", 2020-06-12T02:37:58.058 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.667746990000001, 2020-06-12T02:37:58.058 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52780611399999999, 2020-06-12T02:37:58.058 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:58.058 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:58.058 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:58.059 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:58.059 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:58.059 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2950 2020-06-12T02:37:58.059 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.059 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:58.059 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.060 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:58.060 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.162562+0000", 2020-06-12T02:37:58.060 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:58.060 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.060 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.061 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:58.061 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.162562+0000", 2020-06-12T02:37:58.061 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00070718099999999998 2020-06-12T02:37:58.061 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.061 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.062 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:58.062 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163269+0000", 2020-06-12T02:37:58.062 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2029999999999997e-06 2020-06-12T02:37:58.062 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.062 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.062 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:58.063 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163276+0000", 2020-06-12T02:37:58.063 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2734999999999999e-05 2020-06-12T02:37:58.063 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.063 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.063 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:58.064 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163288+0000", 2020-06-12T02:37:58.064 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.404e-06 2020-06-12T02:37:58.064 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.064 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.064 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:58.064 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163295+0000", 2020-06-12T02:37:58.065 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5784000000000006e-05 2020-06-12T02:37:58.065 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.066 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.066 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.066 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163361+0000", 2020-06-12T02:37:58.066 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5318e-05 2020-06-12T02:37:58.067 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.067 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.067 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:58.067 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163436+0000", 2020-06-12T02:37:58.067 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054801100000000005 2020-06-12T02:37:58.068 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.068 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.068 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:58.068 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163984+0000", 2020-06-12T02:37:58.068 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.007023077 2020-06-12T02:37:58.069 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.069 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.069 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:58.069 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171007+0000", 2020-06-12T02:37:58.069 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51932051199999996 2020-06-12T02:37:58.070 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.070 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.070 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:58.070 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690327+0000", 2020-06-12T02:37:58.070 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0889000000000001e-05 2020-06-12T02:37:58.070 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.071 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.071 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:58.071 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690368+0000", 2020-06-12T02:37:58.071 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52780611399999999 2020-06-12T02:37:58.071 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.072 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:58.072 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.072 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.072 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.073 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2951 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~532480] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:58.073 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.165470+0000", 2020-06-12T02:37:58.073 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.664839382, 2020-06-12T02:37:58.073 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52507917900000001, 2020-06-12T02:37:58.073 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:58.074 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:58.074 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:58.074 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:58.074 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:58.074 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2951 2020-06-12T02:37:58.075 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.075 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:58.075 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.075 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:58.075 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.165470+0000", 2020-06-12T02:37:58.075 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:58.076 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.076 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.076 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:58.077 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.165470+0000", 2020-06-12T02:37:58.077 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00062231600000000004 2020-06-12T02:37:58.077 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.077 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.078 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:58.078 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166092+0000", 2020-06-12T02:37:58.078 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1610000000000002e-06 2020-06-12T02:37:58.078 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.079 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.079 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:58.079 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166097+0000", 2020-06-12T02:37:58.079 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7780000000000002e-06 2020-06-12T02:37:58.079 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.080 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.080 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:58.080 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166107+0000", 2020-06-12T02:37:58.080 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7169999999999999e-06 2020-06-12T02:37:58.080 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.080 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.081 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:58.081 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166112+0000", 2020-06-12T02:37:58.081 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4792000000000002e-05 2020-06-12T02:37:58.081 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.081 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.082 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.082 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166147+0000", 2020-06-12T02:37:58.082 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9336e-05 2020-06-12T02:37:58.083 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.083 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.083 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:58.083 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166236+0000", 2020-06-12T02:37:58.083 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00067747200000000004 2020-06-12T02:37:58.084 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.085 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.085 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:58.085 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166913+0000", 2020-06-12T02:37:58.085 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0047718669999999999 2020-06-12T02:37:58.086 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.086 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.086 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:58.086 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171685+0000", 2020-06-12T02:37:58.087 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51881990700000002 2020-06-12T02:37:58.087 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.087 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.087 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:58.087 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690505+0000", 2020-06-12T02:37:58.087 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3832999999999999e-05 2020-06-12T02:37:58.088 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.088 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.088 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:58.088 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690549+0000", 2020-06-12T02:37:58.088 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52507917900000001 2020-06-12T02:37:58.089 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.089 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:58.089 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.089 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.089 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.090 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2952 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~270336] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:58.090 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.166891+0000", 2020-06-12T02:37:58.090 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.663417732999999, 2020-06-12T02:37:58.090 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52426417000000003, 2020-06-12T02:37:58.090 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:58.091 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:58.091 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:58.091 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:58.092 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:58.092 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2952 2020-06-12T02:37:58.092 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.092 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:58.093 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.093 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:58.093 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166891+0000", 2020-06-12T02:37:58.093 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:58.094 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.094 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.094 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:58.094 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166891+0000", 2020-06-12T02:37:58.094 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00031878599999999999 2020-06-12T02:37:58.094 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.095 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.095 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:58.095 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167210+0000", 2020-06-12T02:37:58.095 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2360000000000004e-06 2020-06-12T02:37:58.095 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.096 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.096 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:58.096 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167219+0000", 2020-06-12T02:37:58.096 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3298e-05 2020-06-12T02:37:58.097 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.097 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.097 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:58.098 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167232+0000", 2020-06-12T02:37:58.098 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5849999999999999e-06 2020-06-12T02:37:58.098 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.098 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.098 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:58.099 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167237+0000", 2020-06-12T02:37:58.099 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0018755740000000001 2020-06-12T02:37:58.099 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.099 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.100 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.100 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.169113+0000", 2020-06-12T02:37:58.100 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000104768 2020-06-12T02:37:58.100 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.101 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.101 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:58.101 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.169218+0000", 2020-06-12T02:37:58.101 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00057389799999999999 2020-06-12T02:37:58.101 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.102 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.102 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:58.102 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.169792+0000", 2020-06-12T02:37:58.102 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0019333989999999999 2020-06-12T02:37:58.102 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.103 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.103 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:58.103 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171725+0000", 2020-06-12T02:37:58.103 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51938508699999997 2020-06-12T02:37:58.104 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.104 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.104 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:58.104 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691110+0000", 2020-06-12T02:37:58.104 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5538999999999999e-05 2020-06-12T02:37:58.105 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.105 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.105 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:58.105 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691156+0000", 2020-06-12T02:37:58.105 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52426417000000003 2020-06-12T02:37:58.106 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.106 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:58.106 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.106 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.106 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.106 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2953 3.5s0 3:a446366d:::smithi10729407-11:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:58.107 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.167293+0000", 2020-06-12T02:37:58.107 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.663016286000001, 2020-06-12T02:37:58.107 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52428350999999995, 2020-06-12T02:37:58.108 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:58.108 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:58.108 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:58.108 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:58.108 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:58.108 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2953 2020-06-12T02:37:58.109 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.109 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:58.109 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.109 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:58.109 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167293+0000", 2020-06-12T02:37:58.110 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:58.110 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.110 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.110 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:58.110 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167293+0000", 2020-06-12T02:37:58.111 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3880000000000004e-05 2020-06-12T02:37:58.111 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.111 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.111 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:58.111 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167367+0000", 2020-06-12T02:37:58.111 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.319e-06 2020-06-12T02:37:58.112 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.112 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.112 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:58.112 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167373+0000", 2020-06-12T02:37:58.113 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1719000000000001e-05 2020-06-12T02:37:58.113 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.114 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.114 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:58.114 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167385+0000", 2020-06-12T02:37:58.114 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4580000000000003e-06 2020-06-12T02:37:58.114 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.115 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.115 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:58.115 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167389+0000", 2020-06-12T02:37:58.115 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036341110000000002 2020-06-12T02:37:58.115 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.115 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.116 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.116 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171023+0000", 2020-06-12T02:37:58.116 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9418000000000005e-05 2020-06-12T02:37:58.116 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.117 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.117 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:58.117 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171093+0000", 2020-06-12T02:37:58.117 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00029087900000000002 2020-06-12T02:37:58.118 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.118 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.118 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:58.118 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171384+0000", 2020-06-12T02:37:58.118 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000489347 2020-06-12T02:37:58.119 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.119 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.119 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:58.119 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171873+0000", 2020-06-12T02:37:58.119 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51963691300000003 2020-06-12T02:37:58.119 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.120 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.120 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:58.120 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691510+0000", 2020-06-12T02:37:58.120 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6465999999999997e-05 2020-06-12T02:37:58.120 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.121 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.121 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:58.121 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691576+0000", 2020-06-12T02:37:58.121 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52428350999999995 2020-06-12T02:37:58.121 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.121 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:58.122 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.122 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.122 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.122 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2954 3.5s0 3:a446366d:::smithi10729407-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:37:58.122 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.167432+0000", 2020-06-12T02:37:58.123 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.662877537, 2020-06-12T02:37:58.123 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52568652699999996, 2020-06-12T02:37:58.123 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:58.123 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:58.123 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:58.124 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:58.124 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:58.124 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2954 2020-06-12T02:37:58.124 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.125 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:58.125 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.125 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:58.125 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167432+0000", 2020-06-12T02:37:58.125 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:58.125 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.126 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.126 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:58.127 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167432+0000", 2020-06-12T02:37:58.127 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3556000000000001e-05 2020-06-12T02:37:58.128 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.128 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.128 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:58.128 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167475+0000", 2020-06-12T02:37:58.128 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7350000000000001e-06 2020-06-12T02:37:58.129 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.129 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.129 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:58.129 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167481+0000", 2020-06-12T02:37:58.129 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3128e-05 2020-06-12T02:37:58.130 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.130 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.130 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:58.130 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167494+0000", 2020-06-12T02:37:58.130 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1579999999999998e-06 2020-06-12T02:37:58.130 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.131 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.131 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:58.131 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167498+0000", 2020-06-12T02:37:58.131 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004082174 2020-06-12T02:37:58.131 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.132 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.132 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.132 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171580+0000", 2020-06-12T02:37:58.132 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8954000000000002e-05 2020-06-12T02:37:58.132 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.132 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.133 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:58.133 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171679+0000", 2020-06-12T02:37:58.133 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51993235100000001 2020-06-12T02:37:58.133 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.133 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.134 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.134 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691612+0000", 2020-06-12T02:37:58.134 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4329000000000001e-05 2020-06-12T02:37:58.134 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.135 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.135 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:58.135 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691666+0000", 2020-06-12T02:37:58.135 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001452142 2020-06-12T02:37:58.135 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.136 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.136 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:58.136 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.693118+0000", 2020-06-12T02:37:58.136 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52568652699999996 2020-06-12T02:37:58.136 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.137 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:58.137 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.137 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.137 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.137 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2959 3.ds1 3:b3445631:::smithi10729407-6:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e27)", 2020-06-12T02:37:58.138 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.176248+0000", 2020-06-12T02:37:58.138 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.654060796, 2020-06-12T02:37:58.138 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51549877099999997, 2020-06-12T02:37:58.138 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:58.138 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:58.139 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:58.139 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:58.139 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:58.139 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2959 2020-06-12T02:37:58.139 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.140 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:58.140 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.140 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:58.140 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176248+0000", 2020-06-12T02:37:58.140 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:58.140 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.141 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.141 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:58.141 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176248+0000", 2020-06-12T02:37:58.141 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5120999999999999e-05 2020-06-12T02:37:58.141 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.142 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.142 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:58.142 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176274+0000", 2020-06-12T02:37:58.142 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9189999999999998e-06 2020-06-12T02:37:58.142 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.143 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.143 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:58.143 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176278+0000", 2020-06-12T02:37:58.143 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6339e-05 2020-06-12T02:37:58.143 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.143 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.144 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:58.144 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176305+0000", 2020-06-12T02:37:58.144 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9089999999999999e-06 2020-06-12T02:37:58.144 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.145 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.145 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:58.145 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176310+0000", 2020-06-12T02:37:58.145 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6095999999999999e-05 2020-06-12T02:37:58.145 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.146 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.146 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.146 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176376+0000", 2020-06-12T02:37:58.146 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8590999999999994e-05 2020-06-12T02:37:58.146 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.147 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.147 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:58.147 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176444+0000", 2020-06-12T02:37:58.147 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00024346400000000001 2020-06-12T02:37:58.147 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.147 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.148 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:58.148 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176688+0000", 2020-06-12T02:37:58.148 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00044220899999999999 2020-06-12T02:37:58.148 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.149 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.149 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:58.149 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.177130+0000", 2020-06-12T02:37:58.149 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51452984400000001 2020-06-12T02:37:58.149 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.150 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.150 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:58.150 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691660+0000", 2020-06-12T02:37:58.150 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7279000000000002e-05 2020-06-12T02:37:58.150 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.151 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.151 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:58.151 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691747+0000", 2020-06-12T02:37:58.151 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51549877099999997 2020-06-12T02:37:58.151 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.151 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:58.152 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.152 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.152 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.152 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:3032 3.ds1 3:b983fd28:::smithi10729407-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:37:58.152 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.864289+0000", 2020-06-12T02:37:58.153 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 19.966019878000001, 2020-06-12T02:37:58.153 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019572216999999999, 2020-06-12T02:37:58.153 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:58.153 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:58.153 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:58.153 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:58.154 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:58.154 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 3032 2020-06-12T02:37:58.154 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.154 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:58.155 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.156 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:58.156 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864289+0000", 2020-06-12T02:37:58.156 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:58.156 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.156 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.157 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:58.157 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864289+0000", 2020-06-12T02:37:58.157 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5701999999999997e-05 2020-06-12T02:37:58.157 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.157 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.158 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:58.158 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864335+0000", 2020-06-12T02:37:58.158 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0910000000000004e-06 2020-06-12T02:37:58.158 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.158 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.159 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:58.159 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864342+0000", 2020-06-12T02:37:58.159 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2301e-05 2020-06-12T02:37:58.159 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.159 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.160 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:58.160 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864354+0000", 2020-06-12T02:37:58.160 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5199999999999997e-06 2020-06-12T02:37:58.160 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.160 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.160 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:58.161 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864360+0000", 2020-06-12T02:37:58.161 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00049180999999999997 2020-06-12T02:37:58.161 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.161 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.161 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.162 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864852+0000", 2020-06-12T02:37:58.162 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6725000000000003e-05 2020-06-12T02:37:58.162 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.162 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.162 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:58.163 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864908+0000", 2020-06-12T02:37:58.163 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0089687010000000008 2020-06-12T02:37:58.163 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.163 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.164 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.164 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.873877+0000", 2020-06-12T02:37:58.164 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3382999999999999e-05 2020-06-12T02:37:58.164 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.164 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.164 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:58.165 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.873941+0000", 2020-06-12T02:37:58.165 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.006408927 2020-06-12T02:37:58.165 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.165 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.165 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.166 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.880350+0000", 2020-06-12T02:37:58.166 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3905e-05 2020-06-12T02:37:58.166 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.166 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.166 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:58.167 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.880403+0000", 2020-06-12T02:37:58.167 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0034581519999999999 2020-06-12T02:37:58.167 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.167 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.167 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:58.168 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.883862+0000", 2020-06-12T02:37:58.168 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019572216999999999 2020-06-12T02:37:58.171 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.171 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:58.171 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.171 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.171 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:58.172 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:37:58.173 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:58.171+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:37:58.174 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-06-12T02:37:58.274 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:37:58.272+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:37:58.332 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:37:58.333 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:37:58.333 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2020-06-12T02:37:58.333 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:37:58.341 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-06-12T02:37:58.381 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:58.379+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:58.477 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:58.475+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:58.497 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:37:58.497 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:37:58.497 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:37:58.497 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:37:58.498 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:37:58.505 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-06-12T02:37:58.573 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.0 to 1 (10000) 2020-06-12T02:37:58.581 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:58.579+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:58.585 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd reweight 1 1 2020-06-12T02:37:58.626 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:37:58.627 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:37:58.627 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:37:58.627 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:37:58.627 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.627 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1558 3.as0 3:5cb723d2:::smithi10729407-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e26)", 2020-06-12T02:37:58.628 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.839131+0000", 2020-06-12T02:37:58.628 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 24.783445223000001, 2020-06-12T02:37:58.628 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013057691, 2020-06-12T02:37:58.628 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:58.628 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:58.629 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:58.629 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:58.629 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:58.629 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1558 2020-06-12T02:37:58.629 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.630 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:58.630 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.630 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:58.630 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.839131+0000", 2020-06-12T02:37:58.630 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:58.631 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.631 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.631 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:58.631 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.839131+0000", 2020-06-12T02:37:58.631 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2228999999999998e-05 2020-06-12T02:37:58.632 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.632 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.632 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:58.632 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.839173+0000", 2020-06-12T02:37:58.632 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9789999999999998e-06 2020-06-12T02:37:58.632 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.633 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.633 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:58.633 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.839179+0000", 2020-06-12T02:37:58.633 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3011e-05 2020-06-12T02:37:58.633 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.634 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.634 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:58.634 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.839192+0000", 2020-06-12T02:37:58.634 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7929999999999999e-06 2020-06-12T02:37:58.635 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.635 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.635 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:58.635 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.839196+0000", 2020-06-12T02:37:58.635 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0083729379999999999 2020-06-12T02:37:58.636 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.636 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.636 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.636 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.847569+0000", 2020-06-12T02:37:58.636 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5887999999999998e-05 2020-06-12T02:37:58.636 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.637 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.637 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:58.637 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.847625+0000", 2020-06-12T02:37:58.637 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0027582330000000001 2020-06-12T02:37:58.637 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.638 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.638 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.638 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.850383+0000", 2020-06-12T02:37:58.638 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00037230500000000002 2020-06-12T02:37:58.638 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.638 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.639 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:58.639 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.850755+0000", 2020-06-12T02:37:58.639 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0014333149999999999 2020-06-12T02:37:58.639 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.639 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.640 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:58.640 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.852189+0000", 2020-06-12T02:37:58.640 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013057691 2020-06-12T02:37:58.640 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.640 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:58.640 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.641 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.641 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.641 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2746 3.as0 3:5cb723d2:::smithi10729407-15:head [append 0~524288] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:58.641 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:36.723084+0000", 2020-06-12T02:37:58.641 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 21.899492426999998, 2020-06-12T02:37:58.642 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013015831, 2020-06-12T02:37:58.642 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:58.642 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:58.642 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:58.642 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:58.642 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:58.643 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2746 2020-06-12T02:37:58.643 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.643 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:58.643 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.643 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:58.644 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.723084+0000", 2020-06-12T02:37:58.644 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:58.644 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.644 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.644 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:58.644 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.723084+0000", 2020-06-12T02:37:58.645 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0012755659999999999 2020-06-12T02:37:58.645 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.645 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.645 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:58.645 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724359+0000", 2020-06-12T02:37:58.646 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2599999999999996e-06 2020-06-12T02:37:58.646 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.646 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.646 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:58.647 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724365+0000", 2020-06-12T02:37:58.647 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5499999999999995e-06 2020-06-12T02:37:58.647 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.647 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.647 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:58.647 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724373+0000", 2020-06-12T02:37:58.648 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3590000000000001e-06 2020-06-12T02:37:58.648 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.648 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.648 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:58.648 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724378+0000", 2020-06-12T02:37:58.649 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0034266629999999999 2020-06-12T02:37:58.649 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.649 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.649 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.649 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.727804+0000", 2020-06-12T02:37:58.649 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6347000000000005e-05 2020-06-12T02:37:58.650 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.650 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.650 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:58.650 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.727891+0000", 2020-06-12T02:37:58.650 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054319199999999996 2020-06-12T02:37:58.651 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.651 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.651 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:58.651 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.728434+0000", 2020-06-12T02:37:58.651 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0029938999999999999 2020-06-12T02:37:58.652 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.652 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.652 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:58.652 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.731428+0000", 2020-06-12T02:37:58.652 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0046291250000000004 2020-06-12T02:37:58.652 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.653 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.653 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:58.653 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.736057+0000", 2020-06-12T02:37:58.653 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2868999999999999e-05 2020-06-12T02:37:58.653 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.654 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.654 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:58.654 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.736100+0000", 2020-06-12T02:37:58.654 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013015831 2020-06-12T02:37:58.654 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.655 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:58.655 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.655 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.655 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.655 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2747 3.as0 3:5cb723d2:::smithi10729407-15:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:58.655 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:36.724407+0000", 2020-06-12T02:37:58.656 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 21.89816926, 2020-06-12T02:37:58.656 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014982544, 2020-06-12T02:37:58.656 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:58.656 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:58.656 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:58.657 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:58.657 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:58.657 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2747 2020-06-12T02:37:58.657 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.657 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:58.657 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.658 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:58.658 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724407+0000", 2020-06-12T02:37:58.658 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:58.658 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.658 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.659 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:58.659 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724407+0000", 2020-06-12T02:37:58.659 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6766e-05 2020-06-12T02:37:58.659 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.659 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.659 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:58.660 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724444+0000", 2020-06-12T02:37:58.660 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.315e-06 2020-06-12T02:37:58.660 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.660 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.660 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:58.660 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724447+0000", 2020-06-12T02:37:58.661 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9100000000000002e-06 2020-06-12T02:37:58.661 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.661 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.661 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:58.661 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724453+0000", 2020-06-12T02:37:58.662 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.43e-06 2020-06-12T02:37:58.662 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.662 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.662 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:58.662 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724455+0000", 2020-06-12T02:37:58.663 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0061404010000000002 2020-06-12T02:37:58.663 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.663 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.663 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.663 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.730596+0000", 2020-06-12T02:37:58.664 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8402e-05 2020-06-12T02:37:58.664 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.664 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.664 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:58.664 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.730664+0000", 2020-06-12T02:37:58.665 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00031661500000000002 2020-06-12T02:37:58.665 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.665 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.665 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:58.666 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.730981+0000", 2020-06-12T02:37:58.666 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0049794890000000001 2020-06-12T02:37:58.666 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.666 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.666 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:58.667 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.735960+0000", 2020-06-12T02:37:58.667 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033245739999999998 2020-06-12T02:37:58.667 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.667 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.667 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:58.668 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.739285+0000", 2020-06-12T02:37:58.668 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010464200000000001 2020-06-12T02:37:58.668 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.668 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.668 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:58.668 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.739390+0000", 2020-06-12T02:37:58.669 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014982544 2020-06-12T02:37:58.669 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.669 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:58.669 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.669 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.670 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.670 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2748 3.as0 3:5cb723d2:::smithi10729407-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:37:58.670 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:36.724477+0000", 2020-06-12T02:37:58.670 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 21.898099121000001, 2020-06-12T02:37:58.670 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020852102000000001, 2020-06-12T02:37:58.670 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:58.671 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:58.671 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:58.671 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:58.671 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:58.671 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2748 2020-06-12T02:37:58.672 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.672 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:58.672 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.672 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:58.672 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724477+0000", 2020-06-12T02:37:58.672 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:58.673 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.673 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.673 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:58.673 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724477+0000", 2020-06-12T02:37:58.673 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2065e-05 2020-06-12T02:37:58.674 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.674 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.674 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:58.674 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724499+0000", 2020-06-12T02:37:58.675 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9340000000000002e-06 2020-06-12T02:37:58.675 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.675 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.675 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:58.675 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724502+0000", 2020-06-12T02:37:58.675 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.246e-06 2020-06-12T02:37:58.676 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.676 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.676 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:58.676 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724508+0000", 2020-06-12T02:37:58.676 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2900000000000001e-06 2020-06-12T02:37:58.677 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.677 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.677 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:58.677 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724511+0000", 2020-06-12T02:37:58.677 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0068556700000000003 2020-06-12T02:37:58.677 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.678 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.678 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.678 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.731366+0000", 2020-06-12T02:37:58.678 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4738999999999999e-05 2020-06-12T02:37:58.678 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.679 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.679 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:58.679 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.731421+0000", 2020-06-12T02:37:58.679 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0080153150000000003 2020-06-12T02:37:58.679 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.680 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.680 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.680 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.739436+0000", 2020-06-12T02:37:58.680 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010418 2020-06-12T02:37:58.680 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.681 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.681 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:58.681 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.739541+0000", 2020-06-12T02:37:58.681 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0057886630000000003 2020-06-12T02:37:58.682 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.682 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.682 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:58.682 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.745329+0000", 2020-06-12T02:37:58.683 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020852102000000001 2020-06-12T02:37:58.683 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.683 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:58.683 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.683 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.684 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.684 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:3055 3.as0 3:5cb723d2:::smithi10729407-15:head [append 0~614400] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:58.684 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.918209+0000", 2020-06-12T02:37:58.684 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.704367405999999, 2020-06-12T02:37:58.684 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014165394, 2020-06-12T02:37:58.685 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:58.685 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:58.685 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:58.685 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:58.685 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:58.685 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 3055 2020-06-12T02:37:58.686 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.686 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:58.686 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.686 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:58.686 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.918209+0000", 2020-06-12T02:37:58.687 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:58.687 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.687 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.687 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:58.687 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.918209+0000", 2020-06-12T02:37:58.687 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00081606199999999999 2020-06-12T02:37:58.688 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.688 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.688 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:58.688 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.919025+0000", 2020-06-12T02:37:58.688 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.251e-06 2020-06-12T02:37:58.689 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.689 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.689 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:58.689 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.919031+0000", 2020-06-12T02:37:58.689 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1460999999999999e-05 2020-06-12T02:37:58.689 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.690 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.690 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:58.690 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.919043+0000", 2020-06-12T02:37:58.690 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.592e-06 2020-06-12T02:37:58.690 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.691 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.691 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:58.691 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.919048+0000", 2020-06-12T02:37:58.691 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0020427959999999999 2020-06-12T02:37:58.691 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.691 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.692 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.692 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.921091+0000", 2020-06-12T02:37:58.692 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012382800000000001 2020-06-12T02:37:58.692 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.692 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.693 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:58.693 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.921215+0000", 2020-06-12T02:37:58.693 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053208300000000004 2020-06-12T02:37:58.693 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.693 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.694 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:58.694 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.921747+0000", 2020-06-12T02:37:58.694 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0075648599999999996 2020-06-12T02:37:58.694 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.694 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.694 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:58.695 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.929312+0000", 2020-06-12T02:37:58.695 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0029879630000000002 2020-06-12T02:37:58.695 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.695 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.695 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:58.696 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.932300+0000", 2020-06-12T02:37:58.696 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4498000000000004e-05 2020-06-12T02:37:58.696 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.696 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.696 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:58.696 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.932374+0000", 2020-06-12T02:37:58.697 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014165394 2020-06-12T02:37:58.697 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.697 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:58.697 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.697 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.698 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.698 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:3217 3.as0 3:5732055a:::smithi10729407-8:head [append 0~147456] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:58.698 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:38.300193+0000", 2020-06-12T02:37:58.698 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.322383296999998, 2020-06-12T02:37:58.698 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012845314, 2020-06-12T02:37:58.699 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:58.699 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:58.699 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:58.699 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:58.699 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:58.699 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 3217 2020-06-12T02:37:58.700 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.700 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:58.700 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.700 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:58.700 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300193+0000", 2020-06-12T02:37:58.701 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:58.701 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.701 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.701 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:58.701 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300193+0000", 2020-06-12T02:37:58.702 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017722699999999999 2020-06-12T02:37:58.702 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.702 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.702 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:58.702 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300370+0000", 2020-06-12T02:37:58.702 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4009999999999996e-06 2020-06-12T02:37:58.703 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.703 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.703 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:58.703 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300376+0000", 2020-06-12T02:37:58.703 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9109999999999993e-06 2020-06-12T02:37:58.703 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.704 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.704 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:58.704 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300384+0000", 2020-06-12T02:37:58.704 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2740000000000001e-06 2020-06-12T02:37:58.704 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.705 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.705 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:58.705 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300388+0000", 2020-06-12T02:37:58.705 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3261999999999999e-05 2020-06-12T02:37:58.705 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.705 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.706 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.706 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300411+0000", 2020-06-12T02:37:58.706 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9906e-05 2020-06-12T02:37:58.706 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.706 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.707 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:58.707 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300461+0000", 2020-06-12T02:37:58.707 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0059439150000000001 2020-06-12T02:37:58.707 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.707 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.707 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.708 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.306405+0000", 2020-06-12T02:37:58.708 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7945000000000002e-05 2020-06-12T02:37:58.708 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.708 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.708 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:58.709 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.306463+0000", 2020-06-12T02:37:58.709 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00042195500000000001 2020-06-12T02:37:58.709 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.709 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.709 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:58.709 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.306885+0000", 2020-06-12T02:37:58.710 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0032273800000000002 2020-06-12T02:37:58.710 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.710 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.710 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:58.710 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.310112+0000", 2020-06-12T02:37:58.711 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0028430420000000001 2020-06-12T02:37:58.711 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.711 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.711 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:58.711 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.312955+0000", 2020-06-12T02:37:58.711 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3096000000000005e-05 2020-06-12T02:37:58.712 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.712 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.712 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:58.712 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.313038+0000", 2020-06-12T02:37:58.712 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012845314 2020-06-12T02:37:58.713 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.713 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:58.713 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.713 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.713 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.713 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:3218 3.as0 3:5732055a:::smithi10729407-8:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:37:58.714 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:38.300417+0000", 2020-06-12T02:37:58.714 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.322158779999999, 2020-06-12T02:37:58.714 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012861745000000001, 2020-06-12T02:37:58.714 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:58.714 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:58.715 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:58.715 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:58.715 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:58.715 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 3218 2020-06-12T02:37:58.715 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.716 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:58.716 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.716 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:58.716 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300417+0000", 2020-06-12T02:37:58.716 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:58.716 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.717 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.717 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:58.717 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300417+0000", 2020-06-12T02:37:58.717 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9110999999999999e-05 2020-06-12T02:37:58.717 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.718 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.718 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:58.719 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300457+0000", 2020-06-12T02:37:58.719 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3289999999999998e-06 2020-06-12T02:37:58.719 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.719 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.719 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:58.719 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300460+0000", 2020-06-12T02:37:58.720 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8569999999999997e-06 2020-06-12T02:37:58.720 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.720 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.720 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:58.720 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300467+0000", 2020-06-12T02:37:58.721 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7080000000000002e-06 2020-06-12T02:37:58.721 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.721 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.721 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:58.721 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300469+0000", 2020-06-12T02:37:58.721 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0064999999999998e-05 2020-06-12T02:37:58.722 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.722 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.722 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.722 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300490+0000", 2020-06-12T02:37:58.722 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0943999999999999e-05 2020-06-12T02:37:58.723 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.723 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.723 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:58.723 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300540+0000", 2020-06-12T02:37:58.723 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0073891770000000002 2020-06-12T02:37:58.723 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.724 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.724 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.724 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.307930+0000", 2020-06-12T02:37:58.724 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010105199999999999 2020-06-12T02:37:58.724 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.725 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.725 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:58.725 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.308031+0000", 2020-06-12T02:37:58.725 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00043489400000000001 2020-06-12T02:37:58.725 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.725 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.726 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:58.726 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.308466+0000", 2020-06-12T02:37:58.726 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0018799070000000001 2020-06-12T02:37:58.726 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.726 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.727 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:58.727 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.310346+0000", 2020-06-12T02:37:58.727 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0028380839999999998 2020-06-12T02:37:58.727 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.727 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.727 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:58.728 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.313184+0000", 2020-06-12T02:37:58.728 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5617000000000003e-05 2020-06-12T02:37:58.728 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.728 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.728 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:58.728 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.313279+0000", 2020-06-12T02:37:58.729 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012861745000000001 2020-06-12T02:37:58.729 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.729 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:58.729 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.729 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.730 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.730 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:3219 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:37:58.730 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:38.300493+0000", 2020-06-12T02:37:58.730 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.322083414000002, 2020-06-12T02:37:58.730 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018386370999999999, 2020-06-12T02:37:58.730 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:58.731 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:58.731 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:58.731 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:58.731 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:58.731 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 3219 2020-06-12T02:37:58.732 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.732 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:58.732 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.732 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:58.732 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300493+0000", 2020-06-12T02:37:58.733 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:58.733 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.733 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.733 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:58.733 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300493+0000", 2020-06-12T02:37:58.733 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3008000000000001e-05 2020-06-12T02:37:58.734 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.734 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.734 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:58.734 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300516+0000", 2020-06-12T02:37:58.735 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1329999999999999e-06 2020-06-12T02:37:58.735 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.735 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.735 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:58.735 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300519+0000", 2020-06-12T02:37:58.736 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8260000000000004e-06 2020-06-12T02:37:58.736 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.736 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.736 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:58.736 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300526+0000", 2020-06-12T02:37:58.737 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.525e-06 2020-06-12T02:37:58.737 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.737 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.737 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:58.737 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300528+0000", 2020-06-12T02:37:58.737 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9839e-05 2020-06-12T02:37:58.738 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.738 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.738 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.738 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300558+0000", 2020-06-12T02:37:58.738 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3198000000000001e-05 2020-06-12T02:37:58.739 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.739 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.739 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:58.739 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300611+0000", 2020-06-12T02:37:58.739 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0082281509999999995 2020-06-12T02:37:58.739 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.740 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.740 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.740 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.308840+0000", 2020-06-12T02:37:58.740 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5921000000000005e-05 2020-06-12T02:37:58.741 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.741 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.741 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:58.741 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.308935+0000", 2020-06-12T02:37:58.741 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050007560000000003 2020-06-12T02:37:58.742 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.742 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.742 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.742 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.313936+0000", 2020-06-12T02:37:58.742 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1600000000000007e-05 2020-06-12T02:37:58.742 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.743 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.743 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:58.743 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.313998+0000", 2020-06-12T02:37:58.743 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0048814139999999997 2020-06-12T02:37:58.743 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.744 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.744 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:58.744 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.318879+0000", 2020-06-12T02:37:58.744 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018386370999999999 2020-06-12T02:37:58.744 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.745 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:58.745 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.745 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.745 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.745 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:4478 3.as0 3:5cb723d2:::smithi10729407-15:head [append 0~483328] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-06-12T02:37:58.745 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:42.522285+0000", 2020-06-12T02:37:58.746 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 16.100291451, 2020-06-12T02:37:58.746 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013258748000000001, 2020-06-12T02:37:58.746 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:58.746 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:58.746 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:58.747 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:58.747 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:58.747 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 4478 2020-06-12T02:37:58.747 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.747 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:58.748 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.748 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:58.748 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.522285+0000", 2020-06-12T02:37:58.748 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:58.749 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.749 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.749 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:58.749 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.522285+0000", 2020-06-12T02:37:58.749 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00055732500000000001 2020-06-12T02:37:58.750 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.750 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.750 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:58.750 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.522842+0000", 2020-06-12T02:37:58.750 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1270000000000002e-06 2020-06-12T02:37:58.750 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.751 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.751 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:58.751 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.522847+0000", 2020-06-12T02:37:58.751 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1314e-05 2020-06-12T02:37:58.751 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.752 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.752 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:58.752 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.522859+0000", 2020-06-12T02:37:58.752 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4339999999999997e-06 2020-06-12T02:37:58.752 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.752 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.753 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:58.753 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.522865+0000", 2020-06-12T02:37:58.753 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033831099999999999 2020-06-12T02:37:58.753 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.753 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.754 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.754 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.526248+0000", 2020-06-12T02:37:58.754 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010359199999999999 2020-06-12T02:37:58.754 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.754 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.754 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:58.755 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.526352+0000", 2020-06-12T02:37:58.755 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031925009999999999 2020-06-12T02:37:58.755 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.755 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.755 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.755 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.529544+0000", 2020-06-12T02:37:58.756 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6626000000000003e-05 2020-06-12T02:37:58.756 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.756 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.756 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:58.756 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.529601+0000", 2020-06-12T02:37:58.757 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047735099999999999 2020-06-12T02:37:58.757 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.757 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.757 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:58.757 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.530078+0000", 2020-06-12T02:37:58.757 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0053046090000000001 2020-06-12T02:37:58.758 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.758 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.758 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:58.758 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.535383+0000", 2020-06-12T02:37:58.758 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011821599999999999 2020-06-12T02:37:58.759 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.759 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.759 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:58.759 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.535501+0000", 2020-06-12T02:37:58.759 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2543000000000002e-05 2020-06-12T02:37:58.759 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.760 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.760 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:58.760 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.535544+0000", 2020-06-12T02:37:58.760 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013258748000000001 2020-06-12T02:37:58.760 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.761 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:58.761 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.761 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.761 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.761 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:4535 3.as0 3:5732055a:::smithi10729407-8:head [append 0~712704] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-06-12T02:37:58.761 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:42.670365+0000", 2020-06-12T02:37:58.762 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 15.952211119999999, 2020-06-12T02:37:58.762 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013998371000000001, 2020-06-12T02:37:58.762 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:58.762 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:58.762 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:58.763 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:58.763 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:58.763 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 4535 2020-06-12T02:37:58.763 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.763 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:58.764 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.764 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:58.764 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.670365+0000", 2020-06-12T02:37:58.764 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:58.764 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.765 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.765 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:58.765 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.670365+0000", 2020-06-12T02:37:58.765 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00059204600000000004 2020-06-12T02:37:58.765 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.766 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.766 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:58.766 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.670957+0000", 2020-06-12T02:37:58.766 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.122e-06 2020-06-12T02:37:58.766 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.767 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.767 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:58.767 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.670963+0000", 2020-06-12T02:37:58.767 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1518999999999999e-05 2020-06-12T02:37:58.767 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.768 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.768 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:58.768 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.670975+0000", 2020-06-12T02:37:58.768 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9290000000000001e-06 2020-06-12T02:37:58.768 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.768 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.769 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:58.769 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.670982+0000", 2020-06-12T02:37:58.769 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4758000000000001e-05 2020-06-12T02:37:58.769 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.769 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.770 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.770 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.671027+0000", 2020-06-12T02:37:58.770 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9733999999999993e-05 2020-06-12T02:37:58.770 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.770 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.770 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:58.771 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.671096+0000", 2020-06-12T02:37:58.771 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000544448 2020-06-12T02:37:58.771 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.771 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.771 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:58.772 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.671641+0000", 2020-06-12T02:37:58.772 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012592534000000001 2020-06-12T02:37:58.772 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.772 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.772 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:58.772 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.684233+0000", 2020-06-12T02:37:58.773 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9785999999999998e-05 2020-06-12T02:37:58.773 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.773 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.773 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:58.773 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.684313+0000", 2020-06-12T02:37:58.774 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0494999999999997e-05 2020-06-12T02:37:58.774 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.774 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.774 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:58.775 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.684364+0000", 2020-06-12T02:37:58.775 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013998371000000001 2020-06-12T02:37:58.775 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.775 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:58.775 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.776 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.776 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.776 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:4975 3.as0 3:5732055a:::smithi10729407-8:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e30)", 2020-06-12T02:37:58.776 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:43.721794+0000", 2020-06-12T02:37:58.776 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 14.900782627, 2020-06-12T02:37:58.776 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.1825135, 2020-06-12T02:37:58.777 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:58.777 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:58.777 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:58.777 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:58.777 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:58.778 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 4975 2020-06-12T02:37:58.778 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.778 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:58.778 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.778 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:58.778 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721794+0000", 2020-06-12T02:37:58.779 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:58.779 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.779 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.779 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:58.779 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721794+0000", 2020-06-12T02:37:58.780 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4542e-05 2020-06-12T02:37:58.780 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.780 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.780 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:58.780 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721818+0000", 2020-06-12T02:37:58.781 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9169999999999997e-06 2020-06-12T02:37:58.781 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.781 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.781 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:58.782 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721823+0000", 2020-06-12T02:37:58.782 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1184e-05 2020-06-12T02:37:58.782 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.782 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.782 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:58.783 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721834+0000", 2020-06-12T02:37:58.783 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.254e-06 2020-06-12T02:37:58.783 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.783 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.784 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:58.784 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721840+0000", 2020-06-12T02:37:58.784 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1075999999999998e-05 2020-06-12T02:37:58.784 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.784 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.784 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.785 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721881+0000", 2020-06-12T02:37:58.785 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7416999999999995e-05 2020-06-12T02:37:58.785 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.785 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.785 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:58.786 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721948+0000", 2020-06-12T02:37:58.786 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.18235910999999999 2020-06-12T02:37:58.786 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.786 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.786 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:58.786 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.904307+0000", 2020-06-12T02:37:58.787 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.1825135 2020-06-12T02:37:58.787 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.787 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:58.787 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.787 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.788 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.788 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:6205 3.as0 3:5cb723d2:::smithi10729407-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e31)", 2020-06-12T02:37:58.788 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:47.353583+0000", 2020-06-12T02:37:58.788 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.268993417000001, 2020-06-12T02:37:58.788 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013040737, 2020-06-12T02:37:58.788 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:58.789 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:58.789 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:58.789 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:58.789 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:58.789 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 6205 2020-06-12T02:37:58.790 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.790 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:58.790 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.790 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:58.790 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.353583+0000", 2020-06-12T02:37:58.791 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:58.791 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.791 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.791 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:58.791 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.353583+0000", 2020-06-12T02:37:58.792 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3133000000000002e-05 2020-06-12T02:37:58.792 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.792 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.792 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:58.792 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.353626+0000", 2020-06-12T02:37:58.792 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9570000000000008e-06 2020-06-12T02:37:58.793 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.793 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.793 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:58.793 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.353635+0000", 2020-06-12T02:37:58.793 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1754e-05 2020-06-12T02:37:58.794 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.794 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.794 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:58.794 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.353647+0000", 2020-06-12T02:37:58.794 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0809999999999998e-06 2020-06-12T02:37:58.794 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.795 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.795 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:58.795 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.353650+0000", 2020-06-12T02:37:58.795 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0064807490000000001 2020-06-12T02:37:58.795 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.796 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.796 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.796 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.360131+0000", 2020-06-12T02:37:58.796 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7163000000000001e-05 2020-06-12T02:37:58.796 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.796 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.797 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:58.797 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.360188+0000", 2020-06-12T02:37:58.797 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0053632230000000003 2020-06-12T02:37:58.797 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.797 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.798 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.798 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.365551+0000", 2020-06-12T02:37:58.798 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8078999999999997e-05 2020-06-12T02:37:58.798 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.798 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.798 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:58.799 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.365609+0000", 2020-06-12T02:37:58.799 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001014598 2020-06-12T02:37:58.799 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.799 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.799 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:58.800 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.366624+0000", 2020-06-12T02:37:58.800 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013040737 2020-06-12T02:37:58.800 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.800 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:58.800 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.800 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.801 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.801 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:6678 3.as0 3:5732055a:::smithi10729407-8:head [append 0~499712] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-06-12T02:37:58.801 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:48.626196+0000", 2020-06-12T02:37:58.801 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.9963806440000003, 2020-06-12T02:37:58.801 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014093700000000001, 2020-06-12T02:37:58.802 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:58.802 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:58.802 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:58.802 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:58.802 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:58.802 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 6678 2020-06-12T02:37:58.803 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.803 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:58.803 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.803 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:58.803 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.626196+0000", 2020-06-12T02:37:58.804 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:58.804 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.804 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.804 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:58.804 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.626196+0000", 2020-06-12T02:37:58.804 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00046254199999999998 2020-06-12T02:37:58.805 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.805 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.805 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:58.805 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.626658+0000", 2020-06-12T02:37:58.805 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3400000000000003e-06 2020-06-12T02:37:58.806 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.806 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.806 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:58.806 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.626665+0000", 2020-06-12T02:37:58.806 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0079e-05 2020-06-12T02:37:58.806 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.807 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.807 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:58.807 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.626675+0000", 2020-06-12T02:37:58.807 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0999999999999997e-06 2020-06-12T02:37:58.807 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.807 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.808 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:58.808 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.626679+0000", 2020-06-12T02:37:58.808 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025692100000000002 2020-06-12T02:37:58.808 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.808 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.809 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.809 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.629248+0000", 2020-06-12T02:37:58.809 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000107944 2020-06-12T02:37:58.809 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.809 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.810 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:58.810 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.629356+0000", 2020-06-12T02:37:58.810 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001434999 2020-06-12T02:37:58.810 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.810 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.810 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.811 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.630791+0000", 2020-06-12T02:37:58.811 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0716e-05 2020-06-12T02:37:58.811 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.811 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.811 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:58.811 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.630842+0000", 2020-06-12T02:37:58.812 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00045107600000000002 2020-06-12T02:37:58.812 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.812 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.812 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:58.812 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.631293+0000", 2020-06-12T02:37:58.813 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0048208330000000001 2020-06-12T02:37:58.813 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.813 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.813 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:58.813 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.636113+0000", 2020-06-12T02:37:58.814 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0041129679999999998 2020-06-12T02:37:58.814 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.814 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.814 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:58.814 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.640226+0000", 2020-06-12T02:37:58.814 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2892999999999996e-05 2020-06-12T02:37:58.815 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.815 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.815 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:58.815 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.640289+0000", 2020-06-12T02:37:58.815 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014093700000000001 2020-06-12T02:37:58.816 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.816 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:58.816 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.816 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.816 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.816 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:7084 3.as0 3:5732055a:::smithi10729407-8:head [append 0~401408] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-06-12T02:37:58.817 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:49.666656+0000", 2020-06-12T02:37:58.817 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 8.9559198999999996, 2020-06-12T02:37:58.817 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013133027, 2020-06-12T02:37:58.817 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:58.817 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:58.818 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:58.818 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:58.818 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:58.819 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 7084 2020-06-12T02:37:58.819 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.819 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:58.819 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.819 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:58.819 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.666656+0000", 2020-06-12T02:37:58.820 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:58.820 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.820 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.820 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:58.820 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.666656+0000", 2020-06-12T02:37:58.821 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00040755700000000001 2020-06-12T02:37:58.821 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.821 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.821 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:58.821 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667064+0000", 2020-06-12T02:37:58.822 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9479999999999999e-06 2020-06-12T02:37:58.822 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.822 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.822 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:58.822 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667069+0000", 2020-06-12T02:37:58.822 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.568e-06 2020-06-12T02:37:58.823 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.823 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.823 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:58.823 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667077+0000", 2020-06-12T02:37:58.823 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4170000000000002e-06 2020-06-12T02:37:58.824 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.824 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.824 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:58.824 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667082+0000", 2020-06-12T02:37:58.824 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0034817770000000001 2020-06-12T02:37:58.824 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.825 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.825 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.825 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.670564+0000", 2020-06-12T02:37:58.825 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011071200000000001 2020-06-12T02:37:58.825 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.826 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.826 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:58.826 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.670674+0000", 2020-06-12T02:37:58.826 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0011185780000000001 2020-06-12T02:37:58.826 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.826 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.827 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.827 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.671793+0000", 2020-06-12T02:37:58.827 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2209e-05 2020-06-12T02:37:58.827 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.827 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.828 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:58.828 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.671855+0000", 2020-06-12T02:37:58.828 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054805100000000003 2020-06-12T02:37:58.828 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.828 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.829 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:58.829 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.672403+0000", 2020-06-12T02:37:58.829 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0044889480000000004 2020-06-12T02:37:58.829 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.829 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.829 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:58.830 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.676892+0000", 2020-06-12T02:37:58.830 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0028428730000000001 2020-06-12T02:37:58.830 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.830 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.830 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:58.831 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.679735+0000", 2020-06-12T02:37:58.831 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4388999999999997e-05 2020-06-12T02:37:58.831 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.831 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.831 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:58.832 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.679789+0000", 2020-06-12T02:37:58.832 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013133027 2020-06-12T02:37:58.832 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.832 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:58.832 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.832 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.833 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.833 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:7085 3.as0 3:5732055a:::smithi10729407-8:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-06-12T02:37:58.833 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:49.667120+0000", 2020-06-12T02:37:58.833 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 8.9554566659999999, 2020-06-12T02:37:58.833 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012883739, 2020-06-12T02:37:58.834 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:58.834 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:58.834 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:58.834 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:58.834 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:58.834 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 7085 2020-06-12T02:37:58.835 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.835 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:58.835 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.835 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:58.835 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667120+0000", 2020-06-12T02:37:58.836 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:58.836 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.836 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.836 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:58.836 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667120+0000", 2020-06-12T02:37:58.836 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.773e-05 2020-06-12T02:37:58.837 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.837 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.837 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:58.837 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667157+0000", 2020-06-12T02:37:58.837 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7740000000000002e-06 2020-06-12T02:37:58.837 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.838 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.838 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:58.838 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667161+0000", 2020-06-12T02:37:58.838 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1190000000000002e-06 2020-06-12T02:37:58.838 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.839 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.839 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:58.839 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667167+0000", 2020-06-12T02:37:58.839 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6970000000000002e-06 2020-06-12T02:37:58.839 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.839 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.840 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:58.840 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667170+0000", 2020-06-12T02:37:58.840 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036418029999999999 2020-06-12T02:37:58.840 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.840 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.841 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.841 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.670812+0000", 2020-06-12T02:37:58.841 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000107107 2020-06-12T02:37:58.841 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.841 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.841 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:58.842 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.670919+0000", 2020-06-12T02:37:58.842 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0034625480000000002 2020-06-12T02:37:58.842 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.842 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.842 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.843 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.674381+0000", 2020-06-12T02:37:58.843 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010428 2020-06-12T02:37:58.843 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.843 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.843 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:58.843 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.674486+0000", 2020-06-12T02:37:58.844 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000426382 2020-06-12T02:37:58.844 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.844 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.844 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:58.844 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.674912+0000", 2020-06-12T02:37:58.845 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0044864739999999998 2020-06-12T02:37:58.845 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.845 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.845 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:58.845 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.679399+0000", 2020-06-12T02:37:58.845 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00051632499999999999 2020-06-12T02:37:58.846 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.846 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.846 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:58.846 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.679915+0000", 2020-06-12T02:37:58.846 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8499999999999996e-05 2020-06-12T02:37:58.847 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.847 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.847 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:58.847 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.680003+0000", 2020-06-12T02:37:58.847 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012883739 2020-06-12T02:37:58.848 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.848 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:58.848 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.848 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.848 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.848 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:7086 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e32)", 2020-06-12T02:37:58.849 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:49.667192+0000", 2020-06-12T02:37:58.849 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 8.9553843149999999, 2020-06-12T02:37:58.849 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014618023000000001, 2020-06-12T02:37:58.849 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:58.849 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:58.850 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:58.850 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:58.850 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:58.850 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 7086 2020-06-12T02:37:58.850 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.850 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:58.851 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.851 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:58.851 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667192+0000", 2020-06-12T02:37:58.851 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:58.851 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.852 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.852 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:58.852 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667192+0000", 2020-06-12T02:37:58.852 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1912e-05 2020-06-12T02:37:58.852 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.853 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.853 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:58.853 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667214+0000", 2020-06-12T02:37:58.853 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0469999999999998e-06 2020-06-12T02:37:58.853 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.854 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.854 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:58.854 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667217+0000", 2020-06-12T02:37:58.854 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7170000000000002e-06 2020-06-12T02:37:58.855 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.855 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.855 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:58.855 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667224+0000", 2020-06-12T02:37:58.856 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2850000000000001e-06 2020-06-12T02:37:58.856 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.856 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.856 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:58.857 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667226+0000", 2020-06-12T02:37:58.857 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0037197950000000001 2020-06-12T02:37:58.857 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.857 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.857 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.858 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.670946+0000", 2020-06-12T02:37:58.858 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4549999999999994e-05 2020-06-12T02:37:58.858 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.858 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.858 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:58.859 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.671040+0000", 2020-06-12T02:37:58.859 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0043129199999999996 2020-06-12T02:37:58.859 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.859 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.859 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.859 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.675353+0000", 2020-06-12T02:37:58.860 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1982000000000001e-05 2020-06-12T02:37:58.860 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.860 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.860 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:58.860 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.675445+0000", 2020-06-12T02:37:58.861 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.005067401 2020-06-12T02:37:58.861 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.861 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.861 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.861 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.680513+0000", 2020-06-12T02:37:58.862 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000101508 2020-06-12T02:37:58.862 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.862 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.862 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:58.862 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.680614+0000", 2020-06-12T02:37:58.863 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0011959060000000001 2020-06-12T02:37:58.863 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.863 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.863 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:58.863 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.681810+0000", 2020-06-12T02:37:58.864 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014618023000000001 2020-06-12T02:37:58.864 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.864 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:58.864 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.864 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.865 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.865 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:7902 3.as0 3:5732055a:::smithi10729407-8:head [append 0~499712] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-06-12T02:37:58.865 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.159351+0000", 2020-06-12T02:37:58.865 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 6.4632256379999999, 2020-06-12T02:37:58.865 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013425812000000001, 2020-06-12T02:37:58.866 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:58.866 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:58.866 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:58.866 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:58.867 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:58.867 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 7902 2020-06-12T02:37:58.867 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.867 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:58.867 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.867 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:58.868 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.159351+0000", 2020-06-12T02:37:58.868 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:58.868 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.868 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.868 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:58.869 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.159351+0000", 2020-06-12T02:37:58.869 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00048251199999999999 2020-06-12T02:37:58.869 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.869 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.869 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:58.869 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.159833+0000", 2020-06-12T02:37:58.870 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7509999999999997e-06 2020-06-12T02:37:58.870 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.870 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.870 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:58.870 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.159839+0000", 2020-06-12T02:37:58.871 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9840000000000007e-06 2020-06-12T02:37:58.871 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.871 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.871 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:58.871 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.159848+0000", 2020-06-12T02:37:58.871 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4619999999999998e-06 2020-06-12T02:37:58.872 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.872 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.872 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:58.872 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.159852+0000", 2020-06-12T02:37:58.872 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5348000000000001e-05 2020-06-12T02:37:58.873 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.873 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.873 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.873 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.159888+0000", 2020-06-12T02:37:58.873 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9824000000000001e-05 2020-06-12T02:37:58.873 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.874 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.874 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:58.874 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.159958+0000", 2020-06-12T02:37:58.874 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0014605149999999999 2020-06-12T02:37:58.874 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.875 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.875 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.875 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.161418+0000", 2020-06-12T02:37:58.875 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0667999999999997e-05 2020-06-12T02:37:58.875 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.876 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.876 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:58.876 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.161469+0000", 2020-06-12T02:37:58.876 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00058577200000000003 2020-06-12T02:37:58.876 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.876 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.877 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:58.877 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.162054+0000", 2020-06-12T02:37:58.877 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010590761000000001 2020-06-12T02:37:58.877 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.877 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.878 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:58.878 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.172645+0000", 2020-06-12T02:37:58.878 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8467000000000006e-05 2020-06-12T02:37:58.878 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.878 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.879 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:58.879 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.172724+0000", 2020-06-12T02:37:58.879 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2747999999999997e-05 2020-06-12T02:37:58.879 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.879 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.879 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:58.880 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.172776+0000", 2020-06-12T02:37:58.880 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013425812000000001 2020-06-12T02:37:58.880 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.880 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:58.880 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.880 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.881 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.881 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:7907 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e33)", 2020-06-12T02:37:58.881 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.166559+0000", 2020-06-12T02:37:58.881 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 6.4560169370000002, 2020-06-12T02:37:58.881 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013016145999999999, 2020-06-12T02:37:58.882 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:58.882 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:58.882 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:58.883 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:58.883 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:58.883 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 7907 2020-06-12T02:37:58.883 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.884 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:58.884 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.884 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:58.884 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.166559+0000", 2020-06-12T02:37:58.884 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:58.885 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.885 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.885 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:58.885 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.166559+0000", 2020-06-12T02:37:58.885 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1719999999999999e-05 2020-06-12T02:37:58.885 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.886 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.886 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:58.886 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.166581+0000", 2020-06-12T02:37:58.886 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8660000000000002e-06 2020-06-12T02:37:58.886 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.887 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.887 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:58.887 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.166584+0000", 2020-06-12T02:37:58.887 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3439999999999997e-06 2020-06-12T02:37:58.887 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.887 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.888 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:58.888 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.166590+0000", 2020-06-12T02:37:58.888 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0640000000000001e-06 2020-06-12T02:37:58.888 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.888 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.889 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:58.889 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.166592+0000", 2020-06-12T02:37:58.889 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0074059160000000002 2020-06-12T02:37:58.889 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.889 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.889 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.890 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.173998+0000", 2020-06-12T02:37:58.890 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8678999999999998e-05 2020-06-12T02:37:58.890 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.890 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.890 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:58.891 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.174057+0000", 2020-06-12T02:37:58.891 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045994779999999997 2020-06-12T02:37:58.891 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.891 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.891 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.891 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.178656+0000", 2020-06-12T02:37:58.892 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6825999999999999e-05 2020-06-12T02:37:58.892 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.892 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.892 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:58.892 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.178733+0000", 2020-06-12T02:37:58.893 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00084225299999999995 2020-06-12T02:37:58.893 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.893 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.893 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:58.893 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.179575+0000", 2020-06-12T02:37:58.893 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013016145999999999 2020-06-12T02:37:58.894 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.894 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:58.894 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.894 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.894 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.895 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:7931 3.as0 3:5732055a:::smithi10729407-8:head [append 0~696320] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-06-12T02:37:58.895 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.246494+0000", 2020-06-12T02:37:58.895 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 6.3760818209999996, 2020-06-12T02:37:58.895 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014082301, 2020-06-12T02:37:58.895 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:58.895 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:37:58.896 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:58.896 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:58.896 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:58.896 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 7931 2020-06-12T02:37:58.896 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.897 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:58.897 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.897 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:58.897 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.246494+0000", 2020-06-12T02:37:58.897 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:58.897 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.898 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.898 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:58.898 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.246494+0000", 2020-06-12T02:37:58.898 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00077570099999999995 2020-06-12T02:37:58.898 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.899 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.899 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:58.899 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.247270+0000", 2020-06-12T02:37:58.899 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4179999999999997e-06 2020-06-12T02:37:58.899 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.899 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.900 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:58.900 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.247279+0000", 2020-06-12T02:37:58.900 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6297999999999999e-05 2020-06-12T02:37:58.900 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.900 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.901 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:58.901 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.247295+0000", 2020-06-12T02:37:58.901 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3309999999999998e-06 2020-06-12T02:37:58.901 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.901 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.902 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:58.902 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.247302+0000", 2020-06-12T02:37:58.902 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4283999999999999e-05 2020-06-12T02:37:58.903 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.903 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.903 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.903 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.247336+0000", 2020-06-12T02:37:58.903 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8157999999999994e-05 2020-06-12T02:37:58.904 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.904 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.904 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:58.904 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.247405+0000", 2020-06-12T02:37:58.904 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031257059999999998 2020-06-12T02:37:58.904 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.905 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.905 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.905 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.250530+0000", 2020-06-12T02:37:58.905 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7546999999999995e-05 2020-06-12T02:37:58.905 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.906 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.906 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:58.906 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.250608+0000", 2020-06-12T02:37:58.906 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00074126599999999997 2020-06-12T02:37:58.906 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.907 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.907 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:37:58.907 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.251349+0000", 2020-06-12T02:37:58.907 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.008999211 2020-06-12T02:37:58.907 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.907 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.908 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:37:58.908 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.260348+0000", 2020-06-12T02:37:58.908 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00016373200000000001 2020-06-12T02:37:58.908 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.908 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.909 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:37:58.909 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.260512+0000", 2020-06-12T02:37:58.909 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4648999999999997e-05 2020-06-12T02:37:58.909 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.909 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.910 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:58.910 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.260577+0000", 2020-06-12T02:37:58.910 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014082301 2020-06-12T02:37:58.910 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.910 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:58.910 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.911 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.911 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.911 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8129 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e33)", 2020-06-12T02:37:58.911 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.700485+0000", 2020-06-12T02:37:58.911 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 5.9220914520000001, 2020-06-12T02:37:58.912 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01339515, 2020-06-12T02:37:58.912 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:37:58.912 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:37:58.912 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:37:58.915 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:37:58.915 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:37:58.916 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8129 2020-06-12T02:37:58.916 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.916 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:37:58.916 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.916 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:37:58.917 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.700485+0000", 2020-06-12T02:37:58.917 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:37:58.917 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.917 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.917 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:37:58.918 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.700485+0000", 2020-06-12T02:37:58.918 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4323999999999998e-05 2020-06-12T02:37:58.918 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.918 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.918 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:37:58.919 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.700529+0000", 2020-06-12T02:37:58.919 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7100000000000004e-06 2020-06-12T02:37:58.919 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.919 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.920 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:37:58.920 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.700535+0000", 2020-06-12T02:37:58.920 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1664999999999999e-05 2020-06-12T02:37:58.920 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.920 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.921 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:37:58.921 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.700547+0000", 2020-06-12T02:37:58.921 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6150000000000001e-06 2020-06-12T02:37:58.921 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.921 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.922 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:37:58.922 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.700550+0000", 2020-06-12T02:37:58.922 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00049316500000000005 2020-06-12T02:37:58.922 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.922 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.922 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.923 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.701043+0000", 2020-06-12T02:37:58.923 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0063999999999997e-05 2020-06-12T02:37:58.923 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.923 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.923 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:58.924 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.701133+0000", 2020-06-12T02:37:58.924 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0085745109999999999 2020-06-12T02:37:58.924 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.924 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.924 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.924 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.709708+0000", 2020-06-12T02:37:58.925 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8854999999999999e-05 2020-06-12T02:37:58.925 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.925 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.925 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:37:58.925 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.709757+0000", 2020-06-12T02:37:58.926 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002613279 2020-06-12T02:37:58.926 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.926 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.926 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:37:58.926 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.712370+0000", 2020-06-12T02:37:58.927 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4332000000000004e-05 2020-06-12T02:37:58.927 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.927 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.927 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:37:58.927 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.712464+0000", 2020-06-12T02:37:58.927 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00141563 2020-06-12T02:37:58.928 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:37:58.928 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:37:58.928 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:37:58.928 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.713880+0000", 2020-06-12T02:37:58.928 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01339515 2020-06-12T02:37:58.928 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.929 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:58.929 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.929 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:37:58.929 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:37:58.929 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:37:58.931 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-06-12T02:37:58.932 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:37:58.929+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:37:58.988 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-06-12T02:37:59.032 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:37:59.029+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:37:59.050 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:37:59.050 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:37:59.051 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2020-06-12T02:37:59.051 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:37:59.059 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-06-12T02:37:59.129 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:37:59.143 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:59.141+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:59.149 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:37:59.168 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:37:59.188 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:37:59.208 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:37:59.216 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:37:59.216 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:37:59.216 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:37:59.216 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:37:59.217 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:37:59.225 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-06-12T02:37:59.228 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:37:59.232 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:37:59.231+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:37:59.248 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:37:59.269 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2020-06-12T02:37:59.270 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-06-12T02:37:59.270 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-06-12T02:37:59.270 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-06-12T02:37:59.271 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-06-12T02:37:59.271 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-06-12T02:37:59.271 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-06-12T02:37:59.271 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-06-12T02:37:59.271 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-06-12T02:37:59.334 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:37:59.331+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:37:59.382 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:59.382 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:59.391 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-06-12T02:37:59.443 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:37:59.433+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:37:59.507 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:59.507 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:59.515 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-06-12T02:37:59.556 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:37:59.554+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:37:59.578 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.1 to 1 (10000) 2020-06-12T02:37:59.580 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd set noscrub 2020-06-12T02:37:59.588 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd reweight 2 1 2020-06-12T02:37:59.676 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:37:59.674+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:37:59.684 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:59.684 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:59.694 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-06-12T02:37:59.741 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:37:59.738+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:37:59.842 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:37:59.840+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:37:59.858 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:37:59.859 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:37:59.867 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-06-12T02:37:59.943 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:37:59.941+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:00.023 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:00.024 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:00.032 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-06-12T02:38:00.076 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:00.074+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:00.144 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:00.142+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:00.149 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:00.149 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:00.158 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-06-12T02:38:00.246 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:00.244+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:00.314 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:00.314 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:00.322 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-06-12T02:38:00.363 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:00.361+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:00.447 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:00.445+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:00.479 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:00.479 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:00.487 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-06-12T02:38:00.548 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:00.545+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:00.591 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2020-06-12T02:38:00.592 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.2 to 1 (10000) 2020-06-12T02:38:00.607 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd reweight 3 1 2020-06-12T02:38:00.648 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:00.646+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:00.655 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:00.655 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:00.664 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-06-12T02:38:00.749 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:00.747+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:00.825 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:00.826 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:00.834 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-06-12T02:38:00.875 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:00.873+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:00.951 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:00.949+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:00.991 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:00.992 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:01.005 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-06-12T02:38:01.055 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:01.049+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:01.153 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:01.151+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:01.161 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:01.161 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:01.169 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-06-12T02:38:01.253 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:01.252+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:01.281 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-06-12T02:38:01.329 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:01.330 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:01.338 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-06-12T02:38:01.382 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:01.380+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:01.424 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:38:01.435 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:38:01.455 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:38:01.455 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:01.452+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:01.467 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:01.467 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:01.474 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:38:01.477 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-06-12T02:38:01.493 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:38:01.513 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:38:01.533 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:38:01.554 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2020-06-12T02:38:01.555 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-06-12T02:38:01.555 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-06-12T02:38:01.555 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-06-12T02:38:01.556 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-06-12T02:38:01.556 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-06-12T02:38:01.556 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-06-12T02:38:01.556 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-06-12T02:38:01.556 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-06-12T02:38:01.559 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:38:01.557+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:38:01.589 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.3 to 1 (10000) 2020-06-12T02:38:01.600 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd reweight 4 1 2020-06-12T02:38:01.639 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:01.639 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:38:01.639 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:38:01.640 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:38:01.640 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:01.644 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-06-12T02:38:01.687 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:01.686+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:01.757 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:01.755+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:01.803 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:01.804 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:38:01.804 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:38:01.804 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:38:01.804 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.804 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:858 3.3s0 3:ce45c70a:::smithi10729407-7:head [append 0~565248] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:38:01.805 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.924643+0000", 2020-06-12T02:38:01.805 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 29.874967025, 2020-06-12T02:38:01.805 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015332551999999999, 2020-06-12T02:38:01.805 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:01.805 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:01.806 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:01.806 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:01.806 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:01.806 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 858 2020-06-12T02:38:01.806 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.807 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:01.807 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.807 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:01.807 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.924643+0000", 2020-06-12T02:38:01.807 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:01.807 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.808 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.808 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:01.808 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.924643+0000", 2020-06-12T02:38:01.808 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00067535299999999996 2020-06-12T02:38:01.808 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.809 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.809 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:01.809 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925318+0000", 2020-06-12T02:38:01.809 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9629999999999993e-06 2020-06-12T02:38:01.809 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.810 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.810 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:01.810 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925328+0000", 2020-06-12T02:38:01.810 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7357999999999999e-05 2020-06-12T02:38:01.810 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.810 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.811 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:01.811 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925345+0000", 2020-06-12T02:38:01.811 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7019999999999996e-06 2020-06-12T02:38:01.811 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.811 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.812 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:01.812 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925355+0000", 2020-06-12T02:38:01.812 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6289999999999998e-05 2020-06-12T02:38:01.812 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.812 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.812 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:01.813 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925411+0000", 2020-06-12T02:38:01.813 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8855000000000004e-05 2020-06-12T02:38:01.813 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.813 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.813 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:01.814 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925480+0000", 2020-06-12T02:38:01.814 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0074521149999999996 2020-06-12T02:38:01.814 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.814 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.814 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:01.815 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.932932+0000", 2020-06-12T02:38:01.815 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3647999999999995e-05 2020-06-12T02:38:01.815 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.815 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.815 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:01.815 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.933026+0000", 2020-06-12T02:38:01.816 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00078753199999999997 2020-06-12T02:38:01.816 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.816 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.816 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:01.817 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.933813+0000", 2020-06-12T02:38:01.817 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.006013582 2020-06-12T02:38:01.817 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.817 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.817 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:01.818 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.939827+0000", 2020-06-12T02:38:01.818 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0111999999999996e-05 2020-06-12T02:38:01.818 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.818 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.818 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:01.819 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.939897+0000", 2020-06-12T02:38:01.819 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8041999999999996e-05 2020-06-12T02:38:01.819 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.819 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.819 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:01.819 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.939975+0000", 2020-06-12T02:38:01.820 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015332551999999999 2020-06-12T02:38:01.820 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:01.820 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:01.820 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:01.820 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.821 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.821 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:860 3.3s0 3:ce45c70a:::smithi10729407-7:head [append 0~81920] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:38:01.821 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.928619+0000", 2020-06-12T02:38:01.821 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 29.870991139000001, 2020-06-12T02:38:01.821 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017431941999999999, 2020-06-12T02:38:01.821 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:01.822 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:01.822 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:01.822 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:01.822 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:01.822 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 860 2020-06-12T02:38:01.823 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.823 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:01.823 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.823 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:01.823 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928619+0000", 2020-06-12T02:38:01.823 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:01.824 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.824 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.824 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:01.824 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928619+0000", 2020-06-12T02:38:01.824 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000105809 2020-06-12T02:38:01.825 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.825 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.825 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:01.825 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928724+0000", 2020-06-12T02:38:01.826 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2339999999999994e-06 2020-06-12T02:38:01.826 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.826 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.826 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:01.826 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928733+0000", 2020-06-12T02:38:01.827 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0268000000000001e-05 2020-06-12T02:38:01.827 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.827 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.827 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:01.827 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928743+0000", 2020-06-12T02:38:01.827 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5190000000000003e-06 2020-06-12T02:38:01.828 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.828 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.828 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:01.828 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928747+0000", 2020-06-12T02:38:01.828 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0103999999999998e-05 2020-06-12T02:38:01.829 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.829 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.829 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:01.829 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928777+0000", 2020-06-12T02:38:01.829 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8249000000000002e-05 2020-06-12T02:38:01.829 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.830 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.830 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:01.830 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928866+0000", 2020-06-12T02:38:01.830 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011151629 2020-06-12T02:38:01.830 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.831 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.831 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:01.831 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.940017+0000", 2020-06-12T02:38:01.831 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8281e-05 2020-06-12T02:38:01.831 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.831 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.832 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:01.832 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.940076+0000", 2020-06-12T02:38:01.832 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000346736 2020-06-12T02:38:01.832 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.832 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.833 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:01.833 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.940422+0000", 2020-06-12T02:38:01.833 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024135049999999998 2020-06-12T02:38:01.833 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.833 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.833 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:01.834 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.942836+0000", 2020-06-12T02:38:01.834 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003170258 2020-06-12T02:38:01.834 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.834 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.834 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:01.835 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946006+0000", 2020-06-12T02:38:01.835 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4350000000000001e-05 2020-06-12T02:38:01.835 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.835 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.835 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:01.835 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946050+0000", 2020-06-12T02:38:01.836 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017431941999999999 2020-06-12T02:38:01.836 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:01.837 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:01.838 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:01.838 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.838 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.838 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:861 3.3s0 3:ce45c70a:::smithi10729407-7:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:38:01.838 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.928778+0000", 2020-06-12T02:38:01.838 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 29.870831526, 2020-06-12T02:38:01.839 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017522738, 2020-06-12T02:38:01.839 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:01.839 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:01.839 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:01.839 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:01.840 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:01.840 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 861 2020-06-12T02:38:01.840 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.840 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:01.840 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.840 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:01.841 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928778+0000", 2020-06-12T02:38:01.841 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:01.841 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.841 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.841 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:01.842 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928778+0000", 2020-06-12T02:38:01.842 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8186000000000002e-05 2020-06-12T02:38:01.842 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.842 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.842 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:01.843 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928826+0000", 2020-06-12T02:38:01.843 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6689999999999997e-06 2020-06-12T02:38:01.843 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.843 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.843 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:01.844 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928831+0000", 2020-06-12T02:38:01.844 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.686e-06 2020-06-12T02:38:01.844 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.844 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.844 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:01.844 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928838+0000", 2020-06-12T02:38:01.845 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7159999999999999e-06 2020-06-12T02:38:01.845 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.845 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.845 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:01.845 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928840+0000", 2020-06-12T02:38:01.846 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4212999999999997e-05 2020-06-12T02:38:01.846 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.846 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.846 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:01.846 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928895+0000", 2020-06-12T02:38:01.846 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5853999999999994e-05 2020-06-12T02:38:01.847 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.847 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.847 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:01.847 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928990+0000", 2020-06-12T02:38:01.847 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012023982000000001 2020-06-12T02:38:01.848 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.848 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.848 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:01.850 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941014+0000", 2020-06-12T02:38:01.850 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5104e-05 2020-06-12T02:38:01.850 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.851 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.851 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:01.851 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941070+0000", 2020-06-12T02:38:01.851 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00026444499999999998 2020-06-12T02:38:01.851 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.851 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.852 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:01.852 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941334+0000", 2020-06-12T02:38:01.852 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001704075 2020-06-12T02:38:01.852 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.852 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.853 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:01.853 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.943038+0000", 2020-06-12T02:38:01.853 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031891570000000002 2020-06-12T02:38:01.853 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.853 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.854 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:01.854 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946227+0000", 2020-06-12T02:38:01.855 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3651000000000003e-05 2020-06-12T02:38:01.856 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.856 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.856 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:01.856 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946301+0000", 2020-06-12T02:38:01.856 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017522738 2020-06-12T02:38:01.856 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:01.857 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:01.857 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:01.857 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.857 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.857 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:862 3.3s0 3:ce45c70a:::smithi10729407-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:38:01.858 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.928876+0000", 2020-06-12T02:38:01.858 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 29.870734030000001, 2020-06-12T02:38:01.858 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019206890000000001, 2020-06-12T02:38:01.858 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:01.858 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:01.859 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:01.859 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:01.859 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:01.859 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 862 2020-06-12T02:38:01.859 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.859 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:01.860 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.860 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:01.860 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928876+0000", 2020-06-12T02:38:01.860 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:01.860 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.861 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.861 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:01.861 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928876+0000", 2020-06-12T02:38:01.861 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9268999999999999e-05 2020-06-12T02:38:01.861 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.862 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.862 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:01.862 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928925+0000", 2020-06-12T02:38:01.862 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8120000000000002e-06 2020-06-12T02:38:01.862 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.862 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.863 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:01.863 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928932+0000", 2020-06-12T02:38:01.863 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3471e-05 2020-06-12T02:38:01.863 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.863 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.863 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:01.864 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928945+0000", 2020-06-12T02:38:01.864 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4279999999999998e-06 2020-06-12T02:38:01.864 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.864 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.864 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:01.865 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928950+0000", 2020-06-12T02:38:01.865 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6358000000000004e-05 2020-06-12T02:38:01.865 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.865 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.865 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:01.866 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.929016+0000", 2020-06-12T02:38:01.866 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9697999999999996e-05 2020-06-12T02:38:01.866 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.866 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.866 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:01.866 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.929086+0000", 2020-06-12T02:38:01.867 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012409205 2020-06-12T02:38:01.867 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.867 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.867 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:01.867 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941495+0000", 2020-06-12T02:38:01.867 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9660000000000006e-05 2020-06-12T02:38:01.868 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.868 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.868 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:01.868 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941585+0000", 2020-06-12T02:38:01.868 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050329099999999998 2020-06-12T02:38:01.869 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.869 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.869 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:01.869 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946617+0000", 2020-06-12T02:38:01.869 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8437999999999996e-05 2020-06-12T02:38:01.869 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.870 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.870 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:01.870 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946706+0000", 2020-06-12T02:38:01.870 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0013766410000000001 2020-06-12T02:38:01.870 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.871 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.871 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:01.871 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.948083+0000", 2020-06-12T02:38:01.871 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019206890000000001 2020-06-12T02:38:01.871 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:01.871 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:01.872 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:01.872 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.872 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.872 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:879 3.fs1 3:fc7e0bf7:::smithi10729407-17:head [append 0~491520] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:38:01.872 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.000881+0000", 2020-06-12T02:38:01.873 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 29.798728849, 2020-06-12T02:38:01.873 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017891845, 2020-06-12T02:38:01.873 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:01.873 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:01.873 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:01.873 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:01.874 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:01.874 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 879 2020-06-12T02:38:01.874 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.874 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:01.874 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.875 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:01.875 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.000881+0000", 2020-06-12T02:38:01.875 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:01.875 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.875 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.875 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:01.876 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.000881+0000", 2020-06-12T02:38:01.876 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00044151599999999998 2020-06-12T02:38:01.876 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.876 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.876 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:01.877 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001322+0000", 2020-06-12T02:38:01.877 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0210000000000001e-06 2020-06-12T02:38:01.877 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.877 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.878 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:01.878 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001326+0000", 2020-06-12T02:38:01.878 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6659999999999997e-06 2020-06-12T02:38:01.879 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.879 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.880 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:01.880 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001335+0000", 2020-06-12T02:38:01.880 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4299999999999999e-06 2020-06-12T02:38:01.880 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.880 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.881 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:01.881 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001339+0000", 2020-06-12T02:38:01.881 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0028741510000000001 2020-06-12T02:38:01.881 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.881 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.882 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:01.882 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.004214+0000", 2020-06-12T02:38:01.882 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010480999999999999 2020-06-12T02:38:01.882 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.882 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.882 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:01.883 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.004318+0000", 2020-06-12T02:38:01.883 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021112359999999998 2020-06-12T02:38:01.883 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.883 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.883 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:01.884 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006430+0000", 2020-06-12T02:38:01.884 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2935000000000001e-05 2020-06-12T02:38:01.884 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.884 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.884 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:01.885 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006483+0000", 2020-06-12T02:38:01.885 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00056411099999999995 2020-06-12T02:38:01.885 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.885 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.885 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:01.885 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.007047+0000", 2020-06-12T02:38:01.886 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0076749369999999997 2020-06-12T02:38:01.886 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.886 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.886 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:01.886 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.014722+0000", 2020-06-12T02:38:01.887 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00395693 2020-06-12T02:38:01.887 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.887 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.887 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:01.887 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.018679+0000", 2020-06-12T02:38:01.887 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4102000000000001e-05 2020-06-12T02:38:01.888 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.888 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.888 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:01.888 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.018773+0000", 2020-06-12T02:38:01.888 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017891845 2020-06-12T02:38:01.889 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:01.889 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:01.889 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:01.889 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.889 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.889 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:933 3.es1 3:78f01648:::smithi10729407-5:head [append 0~417792] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:38:01.890 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.156677+0000", 2020-06-12T02:38:01.890 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 29.642932866999999, 2020-06-12T02:38:01.890 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016833311, 2020-06-12T02:38:01.890 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:01.890 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:01.891 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:01.891 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:01.891 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:01.891 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 933 2020-06-12T02:38:01.891 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.891 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:01.892 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.892 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:01.892 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.156677+0000", 2020-06-12T02:38:01.892 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:01.892 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.893 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.893 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:01.893 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.156677+0000", 2020-06-12T02:38:01.893 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00062571700000000003 2020-06-12T02:38:01.893 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.894 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.894 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:01.894 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157303+0000", 2020-06-12T02:38:01.894 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0740000000000003e-06 2020-06-12T02:38:01.894 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.894 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.895 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:01.895 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157307+0000", 2020-06-12T02:38:01.895 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4419999999999998e-06 2020-06-12T02:38:01.895 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.895 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.896 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:01.896 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157315+0000", 2020-06-12T02:38:01.896 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5300000000000001e-06 2020-06-12T02:38:01.896 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.896 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.896 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:01.897 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157319+0000", 2020-06-12T02:38:01.897 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4995e-05 2020-06-12T02:38:01.897 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.897 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.897 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:01.898 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157354+0000", 2020-06-12T02:38:01.898 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5883000000000006e-05 2020-06-12T02:38:01.898 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.898 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.898 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:01.898 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157419+0000", 2020-06-12T02:38:01.899 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045316280000000002 2020-06-12T02:38:01.899 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.899 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.899 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:01.899 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.161951+0000", 2020-06-12T02:38:01.900 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3300999999999996e-05 2020-06-12T02:38:01.900 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.900 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.900 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:01.900 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.162014+0000", 2020-06-12T02:38:01.900 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00039050100000000002 2020-06-12T02:38:01.901 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.901 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.901 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:01.901 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.162405+0000", 2020-06-12T02:38:01.901 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050686250000000002 2020-06-12T02:38:01.902 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.902 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.902 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:01.902 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.167474+0000", 2020-06-12T02:38:01.902 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.005942653 2020-06-12T02:38:01.902 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.903 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.903 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:01.903 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.173416+0000", 2020-06-12T02:38:01.903 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3962000000000006e-05 2020-06-12T02:38:01.903 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.904 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.904 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:01.904 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.173510+0000", 2020-06-12T02:38:01.904 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016833311 2020-06-12T02:38:01.904 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:01.904 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:01.905 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:01.905 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.905 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.905 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:936 3.es1 3:78f01648:::smithi10729407-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:38:01.905 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.160357+0000", 2020-06-12T02:38:01.906 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 29.639253155999999, 2020-06-12T02:38:01.906 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015435590000000001, 2020-06-12T02:38:01.906 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:01.906 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:01.906 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:01.906 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:01.907 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:01.907 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 936 2020-06-12T02:38:01.907 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.907 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:01.907 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.908 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:01.908 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160357+0000", 2020-06-12T02:38:01.908 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:01.908 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.908 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.908 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:01.909 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160357+0000", 2020-06-12T02:38:01.909 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4435999999999999e-05 2020-06-12T02:38:01.909 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.909 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.909 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:01.910 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160411+0000", 2020-06-12T02:38:01.910 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2329999999999995e-06 2020-06-12T02:38:01.910 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.910 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.910 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:01.910 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160420+0000", 2020-06-12T02:38:01.911 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7003e-05 2020-06-12T02:38:01.911 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.911 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.911 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:01.911 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160437+0000", 2020-06-12T02:38:01.912 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8370000000000002e-06 2020-06-12T02:38:01.912 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.912 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.912 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:01.912 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160443+0000", 2020-06-12T02:38:01.912 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0070635489999999997 2020-06-12T02:38:01.913 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.913 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.913 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:01.913 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.167507+0000", 2020-06-12T02:38:01.913 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010589700000000001 2020-06-12T02:38:01.914 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.914 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.914 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:01.914 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.167612+0000", 2020-06-12T02:38:01.914 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0065032299999999996 2020-06-12T02:38:01.914 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.915 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.915 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:01.915 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.174116+0000", 2020-06-12T02:38:01.915 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3128999999999999e-05 2020-06-12T02:38:01.915 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.916 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.916 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:01.916 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.174169+0000", 2020-06-12T02:38:01.916 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0016232760000000001 2020-06-12T02:38:01.916 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.917 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.917 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:01.917 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.175792+0000", 2020-06-12T02:38:01.917 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015435590000000001 2020-06-12T02:38:01.918 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:01.918 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:01.918 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:01.918 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.918 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.918 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:950 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~778240] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:38:01.919 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.207643+0000", 2020-06-12T02:38:01.919 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 29.591966573000001, 2020-06-12T02:38:01.919 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014844661, 2020-06-12T02:38:01.919 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:01.919 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:01.920 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:01.920 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:01.920 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:01.920 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 950 2020-06-12T02:38:01.920 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.920 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:01.921 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.921 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:01.921 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.207643+0000", 2020-06-12T02:38:01.921 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:01.921 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.922 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.922 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:01.922 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.207643+0000", 2020-06-12T02:38:01.922 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00072843099999999998 2020-06-12T02:38:01.922 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.923 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.923 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:01.923 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208372+0000", 2020-06-12T02:38:01.923 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9120000000000001e-06 2020-06-12T02:38:01.924 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.924 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.924 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:01.925 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208378+0000", 2020-06-12T02:38:01.925 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9660000000000008e-06 2020-06-12T02:38:01.925 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.925 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.926 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:01.926 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208388+0000", 2020-06-12T02:38:01.926 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1279999999999999e-06 2020-06-12T02:38:01.926 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.926 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.926 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:01.927 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208394+0000", 2020-06-12T02:38:01.927 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0439999999999998e-05 2020-06-12T02:38:01.927 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.927 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.927 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:01.928 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208444+0000", 2020-06-12T02:38:01.928 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2611000000000001e-05 2020-06-12T02:38:01.928 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.928 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.928 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:01.929 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208507+0000", 2020-06-12T02:38:01.930 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001228301 2020-06-12T02:38:01.930 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.930 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.930 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:01.931 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.209735+0000", 2020-06-12T02:38:01.931 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8875000000000001e-05 2020-06-12T02:38:01.931 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.931 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.931 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:01.932 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.209794+0000", 2020-06-12T02:38:01.932 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053218899999999997 2020-06-12T02:38:01.932 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.932 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.932 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:01.932 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.210326+0000", 2020-06-12T02:38:01.933 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0078198439999999994 2020-06-12T02:38:01.933 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.933 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.933 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:01.933 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.218146+0000", 2020-06-12T02:38:01.934 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004300496 2020-06-12T02:38:01.934 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.934 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.934 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:01.934 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.222446+0000", 2020-06-12T02:38:01.934 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1467999999999997e-05 2020-06-12T02:38:01.935 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.935 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.935 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:01.935 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.222488+0000", 2020-06-12T02:38:01.935 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014844661 2020-06-12T02:38:01.936 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:01.936 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:01.936 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:01.936 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.936 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.936 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:954 3.3s0 3:c2ca77da:::smithi10729407-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:38:01.937 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.212923+0000", 2020-06-12T02:38:01.937 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 29.586686796999999, 2020-06-12T02:38:01.937 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015411546999999999, 2020-06-12T02:38:01.937 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:01.937 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:01.938 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:01.938 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:01.939 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:01.939 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 954 2020-06-12T02:38:01.940 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.940 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:01.940 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.940 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:01.940 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212923+0000", 2020-06-12T02:38:01.941 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:01.941 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.941 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.941 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:01.941 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212923+0000", 2020-06-12T02:38:01.942 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3598e-05 2020-06-12T02:38:01.942 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.942 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.942 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:01.942 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212946+0000", 2020-06-12T02:38:01.942 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.061e-06 2020-06-12T02:38:01.943 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.943 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.943 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:01.943 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212950+0000", 2020-06-12T02:38:01.943 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8700000000000003e-06 2020-06-12T02:38:01.944 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.944 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.944 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:01.944 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212956+0000", 2020-06-12T02:38:01.944 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0770000000000002e-06 2020-06-12T02:38:01.944 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.945 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.945 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:01.945 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212958+0000", 2020-06-12T02:38:01.945 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051235170000000002 2020-06-12T02:38:01.945 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.946 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.946 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:01.946 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.218082+0000", 2020-06-12T02:38:01.946 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9246999999999997e-05 2020-06-12T02:38:01.946 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.947 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.947 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:01.947 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.218141+0000", 2020-06-12T02:38:01.947 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0087981910000000003 2020-06-12T02:38:01.947 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.947 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.948 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:01.948 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.226939+0000", 2020-06-12T02:38:01.948 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4119999999999997e-05 2020-06-12T02:38:01.948 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.948 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.949 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:01.949 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.226994+0000", 2020-06-12T02:38:01.949 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001340866 2020-06-12T02:38:01.949 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.949 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.949 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:01.950 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.228334+0000", 2020-06-12T02:38:01.950 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015411546999999999 2020-06-12T02:38:01.950 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:01.951 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:01.951 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:01.951 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.951 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.951 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1097 3.es1 3:78f01648:::smithi10729407-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:38:01.952 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.576496+0000", 2020-06-12T02:38:01.953 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 29.223113314999999, 2020-06-12T02:38:01.953 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014992914, 2020-06-12T02:38:01.953 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:01.953 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:01.953 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:01.954 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:01.954 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:01.954 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1097 2020-06-12T02:38:01.954 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.954 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:01.955 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.955 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:01.955 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576496+0000", 2020-06-12T02:38:01.955 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:01.955 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.956 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.956 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:01.956 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576496+0000", 2020-06-12T02:38:01.956 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3221999999999999e-05 2020-06-12T02:38:01.956 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.956 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.957 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:01.957 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576520+0000", 2020-06-12T02:38:01.957 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.988e-06 2020-06-12T02:38:01.957 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.957 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.958 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:01.958 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576523+0000", 2020-06-12T02:38:01.958 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2940000000000002e-06 2020-06-12T02:38:01.958 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.958 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.959 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:01.959 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576529+0000", 2020-06-12T02:38:01.959 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6060000000000002e-06 2020-06-12T02:38:01.959 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.959 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.959 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:01.960 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576534+0000", 2020-06-12T02:38:01.960 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001254554 2020-06-12T02:38:01.960 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.960 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.960 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:01.961 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.577789+0000", 2020-06-12T02:38:01.961 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3177999999999998e-05 2020-06-12T02:38:01.961 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.961 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.961 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:01.961 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.577842+0000", 2020-06-12T02:38:01.962 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0063091220000000003 2020-06-12T02:38:01.962 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.962 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.962 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:01.962 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.584151+0000", 2020-06-12T02:38:01.963 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.783e-05 2020-06-12T02:38:01.963 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.963 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.963 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:01.963 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.584209+0000", 2020-06-12T02:38:01.964 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045863650000000002 2020-06-12T02:38:01.964 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.964 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.964 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:01.964 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.588796+0000", 2020-06-12T02:38:01.964 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7614000000000006e-05 2020-06-12T02:38:01.965 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.965 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.965 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:01.965 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.588873+0000", 2020-06-12T02:38:01.965 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0026161410000000002 2020-06-12T02:38:01.966 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.966 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.966 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:01.966 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.591489+0000", 2020-06-12T02:38:01.966 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014992914 2020-06-12T02:38:01.967 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:01.967 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:01.967 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:01.967 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.967 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.968 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:4813 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~450560] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-06-12T02:38:01.968 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:43.343096+0000", 2020-06-12T02:38:01.968 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 18.456513972, 2020-06-12T02:38:01.968 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014820242000000001, 2020-06-12T02:38:01.968 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:01.968 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:01.969 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:01.969 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:01.969 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:01.969 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 4813 2020-06-12T02:38:01.969 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.970 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:01.970 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.970 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:01.970 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343096+0000", 2020-06-12T02:38:01.970 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:01.971 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.971 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.971 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:01.971 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343096+0000", 2020-06-12T02:38:01.971 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00068963 2020-06-12T02:38:01.971 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.972 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.972 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:01.972 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343785+0000", 2020-06-12T02:38:01.972 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1479999999999997e-06 2020-06-12T02:38:01.972 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.973 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.973 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:01.973 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343790+0000", 2020-06-12T02:38:01.973 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4060000000000005e-06 2020-06-12T02:38:01.973 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.973 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.974 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:01.974 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343799+0000", 2020-06-12T02:38:01.974 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0970000000000002e-06 2020-06-12T02:38:01.974 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.974 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.975 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:01.975 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343805+0000", 2020-06-12T02:38:01.975 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00023736499999999999 2020-06-12T02:38:01.975 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.975 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.975 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:01.976 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.344042+0000", 2020-06-12T02:38:01.976 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000113866 2020-06-12T02:38:01.976 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.976 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.976 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:01.977 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.344156+0000", 2020-06-12T02:38:01.977 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036114760000000002 2020-06-12T02:38:01.977 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.977 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.977 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:01.977 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.347768+0000", 2020-06-12T02:38:01.978 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1842999999999996e-05 2020-06-12T02:38:01.978 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.978 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.978 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:01.978 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.347850+0000", 2020-06-12T02:38:01.979 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00048835999999999997 2020-06-12T02:38:01.979 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.979 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.979 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:01.979 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.348338+0000", 2020-06-12T02:38:01.980 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0049373139999999999 2020-06-12T02:38:01.980 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.980 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.980 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:01.980 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.353275+0000", 2020-06-12T02:38:01.980 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045727119999999996 2020-06-12T02:38:01.981 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.981 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.981 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:01.981 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.357848+0000", 2020-06-12T02:38:01.981 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8025e-05 2020-06-12T02:38:01.982 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.982 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.982 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:01.982 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.357916+0000", 2020-06-12T02:38:01.982 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014820242000000001 2020-06-12T02:38:01.982 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:01.983 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:01.983 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:01.983 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.983 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.983 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:4921 3.9s0 3:93ea0cad:::smithi10729407-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-06-12T02:38:01.984 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:43.595696+0000", 2020-06-12T02:38:01.984 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 18.203913606, 2020-06-12T02:38:01.984 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016503190000000001, 2020-06-12T02:38:01.984 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:01.984 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:01.984 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:01.985 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:01.985 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:01.985 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 4921 2020-06-12T02:38:01.985 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.985 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:01.986 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.986 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:01.986 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595696+0000", 2020-06-12T02:38:01.986 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:01.986 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.986 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.987 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:01.987 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595696+0000", 2020-06-12T02:38:01.987 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3694999999999999e-05 2020-06-12T02:38:01.987 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.987 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.988 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:01.988 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595720+0000", 2020-06-12T02:38:01.988 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1700000000000001e-06 2020-06-12T02:38:01.988 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.988 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.988 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:01.989 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595723+0000", 2020-06-12T02:38:01.989 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8849999999999999e-06 2020-06-12T02:38:01.989 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.989 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.989 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:01.990 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595731+0000", 2020-06-12T02:38:01.990 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5689999999999999e-06 2020-06-12T02:38:01.990 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.990 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.990 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:01.990 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595733+0000", 2020-06-12T02:38:01.991 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0030116880000000002 2020-06-12T02:38:01.991 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.991 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.991 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:01.991 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.598745+0000", 2020-06-12T02:38:01.992 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1353999999999996e-05 2020-06-12T02:38:01.992 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.992 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.992 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:01.992 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.598806+0000", 2020-06-12T02:38:01.992 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051955819999999998 2020-06-12T02:38:01.993 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.993 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.993 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:01.993 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.604002+0000", 2020-06-12T02:38:01.993 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9615000000000001e-05 2020-06-12T02:38:01.994 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.994 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.994 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:01.994 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.604102+0000", 2020-06-12T02:38:01.994 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0052050960000000002 2020-06-12T02:38:01.995 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.995 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.995 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:01.995 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.609307+0000", 2020-06-12T02:38:01.995 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5025000000000003e-05 2020-06-12T02:38:01.996 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.996 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.996 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:01.996 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.609362+0000", 2020-06-12T02:38:01.996 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002837511 2020-06-12T02:38:01.996 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.997 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.997 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:01.997 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.612199+0000", 2020-06-12T02:38:01.997 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016503190000000001 2020-06-12T02:38:01.997 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:01.998 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:01.998 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:01.998 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:01.998 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:01.998 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:5068 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~663552] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e30)", 2020-06-12T02:38:01.998 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:44.157497+0000", 2020-06-12T02:38:01.999 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 17.64211298, 2020-06-12T02:38:01.999 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015164129, 2020-06-12T02:38:01.999 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:01.999 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:01.999 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:02.000 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:02.000 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:02.000 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 5068 2020-06-12T02:38:02.000 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.000 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:02.000 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.001 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:02.001 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.157497+0000", 2020-06-12T02:38:02.001 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:02.001 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.001 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.002 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:02.002 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.157497+0000", 2020-06-12T02:38:02.002 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00068511099999999997 2020-06-12T02:38:02.002 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.002 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.002 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:02.003 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.158182+0000", 2020-06-12T02:38:02.003 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4630000000000004e-06 2020-06-12T02:38:02.003 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.003 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.003 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:02.004 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.158189+0000", 2020-06-12T02:38:02.004 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4199000000000001e-05 2020-06-12T02:38:02.004 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.004 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.004 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:02.004 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.158203+0000", 2020-06-12T02:38:02.005 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1250000000000004e-06 2020-06-12T02:38:02.005 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.005 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.005 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:02.005 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.158211+0000", 2020-06-12T02:38:02.006 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6754000000000002e-05 2020-06-12T02:38:02.006 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.006 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.006 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.006 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.158237+0000", 2020-06-12T02:38:02.006 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2948999999999996e-05 2020-06-12T02:38:02.007 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.007 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.007 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:02.007 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.158310+0000", 2020-06-12T02:38:02.007 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0032401610000000001 2020-06-12T02:38:02.008 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.008 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.008 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.008 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.161550+0000", 2020-06-12T02:38:02.008 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5550000000000002e-05 2020-06-12T02:38:02.008 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.009 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.009 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:02.009 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.161606+0000", 2020-06-12T02:38:02.009 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00062448199999999999 2020-06-12T02:38:02.009 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.010 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.010 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:02.010 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.162230+0000", 2020-06-12T02:38:02.010 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0073257130000000002 2020-06-12T02:38:02.010 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.010 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.011 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:02.011 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.169556+0000", 2020-06-12T02:38:02.011 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0030226419999999999 2020-06-12T02:38:02.011 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.011 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.012 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:02.012 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.172579+0000", 2020-06-12T02:38:02.012 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1979999999999998e-05 2020-06-12T02:38:02.012 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.012 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.012 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:02.013 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.172661+0000", 2020-06-12T02:38:02.013 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015164129 2020-06-12T02:38:02.013 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.013 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:02.013 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.014 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.014 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.014 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:5071 3.3s0 3:c2ca77da:::smithi10729407-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-06-12T02:38:02.014 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:44.159788+0000", 2020-06-12T02:38:02.014 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 17.639821758, 2020-06-12T02:38:02.014 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016812273999999999, 2020-06-12T02:38:02.015 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:02.015 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:02.015 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:02.015 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:02.015 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:02.016 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 5071 2020-06-12T02:38:02.016 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.016 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:02.016 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.016 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:02.016 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159788+0000", 2020-06-12T02:38:02.017 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:02.017 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.017 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.017 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:02.017 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159788+0000", 2020-06-12T02:38:02.018 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1421000000000001e-05 2020-06-12T02:38:02.018 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.018 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.018 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:02.018 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159839+0000", 2020-06-12T02:38:02.018 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9009999999999994e-06 2020-06-12T02:38:02.019 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.019 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.019 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:02.019 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159847+0000", 2020-06-12T02:38:02.019 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3149999999999999e-05 2020-06-12T02:38:02.020 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.020 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.020 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:02.020 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159860+0000", 2020-06-12T02:38:02.020 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1749999999999996e-06 2020-06-12T02:38:02.020 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.021 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.021 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:02.021 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159866+0000", 2020-06-12T02:38:02.021 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6035999999999999e-05 2020-06-12T02:38:02.021 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.022 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.022 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.022 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159892+0000", 2020-06-12T02:38:02.022 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2627999999999998e-05 2020-06-12T02:38:02.022 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.022 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.023 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:02.023 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159944+0000", 2020-06-12T02:38:02.023 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010270133000000001 2020-06-12T02:38:02.023 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.023 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.024 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.024 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.170214+0000", 2020-06-12T02:38:02.024 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5490999999999998e-05 2020-06-12T02:38:02.024 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.024 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.024 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:02.025 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.170310+0000", 2020-06-12T02:38:02.025 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003215828 2020-06-12T02:38:02.025 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.025 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.025 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.026 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.173526+0000", 2020-06-12T02:38:02.026 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0290000000000005e-05 2020-06-12T02:38:02.026 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.026 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.026 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:02.026 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.173616+0000", 2020-06-12T02:38:02.027 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002984221 2020-06-12T02:38:02.027 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.027 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.027 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:02.027 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.176600+0000", 2020-06-12T02:38:02.028 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016812273999999999 2020-06-12T02:38:02.028 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.028 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:02.028 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.028 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.028 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.029 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:6432 3.9s0 3:93ea0cad:::smithi10729407-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e31)", 2020-06-12T02:38:02.029 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:47.960104+0000", 2020-06-12T02:38:02.029 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 13.839505662000001, 2020-06-12T02:38:02.029 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016025815999999998, 2020-06-12T02:38:02.029 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:02.030 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:02.030 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:02.030 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:02.030 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:02.030 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 6432 2020-06-12T02:38:02.030 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.031 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:02.031 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.031 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:02.031 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960104+0000", 2020-06-12T02:38:02.031 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:02.032 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.032 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.032 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:02.032 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960104+0000", 2020-06-12T02:38:02.032 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2351999999999999e-05 2020-06-12T02:38:02.032 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.033 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.033 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:02.033 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960126+0000", 2020-06-12T02:38:02.033 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.061e-06 2020-06-12T02:38:02.033 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.034 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.034 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:02.034 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960129+0000", 2020-06-12T02:38:02.034 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4849999999999997e-06 2020-06-12T02:38:02.034 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.034 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.035 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:02.035 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960136+0000", 2020-06-12T02:38:02.035 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.176e-06 2020-06-12T02:38:02.035 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.035 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.036 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:02.036 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960138+0000", 2020-06-12T02:38:02.036 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0044054139999999999 2020-06-12T02:38:02.036 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.036 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.036 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.037 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.964543+0000", 2020-06-12T02:38:02.037 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8077000000000006e-05 2020-06-12T02:38:02.037 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.037 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.037 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:02.038 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.964612+0000", 2020-06-12T02:38:02.038 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0029921150000000001 2020-06-12T02:38:02.038 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.038 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.038 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.038 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.967604+0000", 2020-06-12T02:38:02.039 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2983000000000001e-05 2020-06-12T02:38:02.039 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.039 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.039 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:02.039 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.967697+0000", 2020-06-12T02:38:02.040 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0072496510000000002 2020-06-12T02:38:02.040 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.040 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.040 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.040 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.974946+0000", 2020-06-12T02:38:02.041 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8278e-05 2020-06-12T02:38:02.041 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.041 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.041 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:02.041 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.975005+0000", 2020-06-12T02:38:02.041 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001125224 2020-06-12T02:38:02.042 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.042 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.042 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:02.042 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.976130+0000", 2020-06-12T02:38:02.043 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016025815999999998 2020-06-12T02:38:02.043 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.043 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:02.043 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.043 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.043 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.044 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8123 3.es1 3:78f01648:::smithi10729407-5:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-06-12T02:38:02.044 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.690721+0000", 2020-06-12T02:38:02.044 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.1088885580000003, 2020-06-12T02:38:02.044 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016503126999999999, 2020-06-12T02:38:02.044 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:02.045 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:02.045 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:02.045 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:02.045 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:02.045 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8123 2020-06-12T02:38:02.046 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.046 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:02.046 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.046 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:02.046 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690721+0000", 2020-06-12T02:38:02.046 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:02.047 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.047 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.047 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:02.047 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690721+0000", 2020-06-12T02:38:02.047 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7526e-05 2020-06-12T02:38:02.047 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.048 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.048 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:02.048 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690759+0000", 2020-06-12T02:38:02.048 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1999999999999999e-06 2020-06-12T02:38:02.048 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.049 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.049 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:02.049 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690762+0000", 2020-06-12T02:38:02.049 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9229999999999999e-06 2020-06-12T02:38:02.049 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.049 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.050 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:02.050 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690768+0000", 2020-06-12T02:38:02.050 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4480000000000001e-06 2020-06-12T02:38:02.050 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.050 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.051 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:02.051 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690770+0000", 2020-06-12T02:38:02.051 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00079317799999999996 2020-06-12T02:38:02.051 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.051 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.051 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.052 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.691563+0000", 2020-06-12T02:38:02.052 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5299e-05 2020-06-12T02:38:02.052 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.052 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.052 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:02.053 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.691619+0000", 2020-06-12T02:38:02.053 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010070076000000001 2020-06-12T02:38:02.053 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.053 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.053 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.053 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.701689+0000", 2020-06-12T02:38:02.054 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000110234 2020-06-12T02:38:02.054 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.054 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.058 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:02.058 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.701799+0000", 2020-06-12T02:38:02.058 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00037311100000000003 2020-06-12T02:38:02.059 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.059 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.059 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:02.059 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.702172+0000", 2020-06-12T02:38:02.059 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0044460050000000003 2020-06-12T02:38:02.060 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.060 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.060 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:02.060 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.706618+0000", 2020-06-12T02:38:02.060 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00043550199999999999 2020-06-12T02:38:02.060 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.061 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.061 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:02.061 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.707054+0000", 2020-06-12T02:38:02.061 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017062499999999999 2020-06-12T02:38:02.061 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.062 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.062 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:02.062 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.707224+0000", 2020-06-12T02:38:02.062 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016503126999999999 2020-06-12T02:38:02.062 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.062 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:02.063 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.063 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.063 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.063 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8124 3.es1 3:78f01648:::smithi10729407-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e33)", 2020-06-12T02:38:02.063 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.690794+0000", 2020-06-12T02:38:02.064 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.1088153720000005, 2020-06-12T02:38:02.064 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019298497000000001, 2020-06-12T02:38:02.064 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:02.064 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:02.064 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:02.064 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:02.065 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:02.065 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8124 2020-06-12T02:38:02.065 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.065 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:02.065 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.066 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:02.066 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690794+0000", 2020-06-12T02:38:02.066 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:02.066 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.066 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.066 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:02.067 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690794+0000", 2020-06-12T02:38:02.067 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1869999999999999e-05 2020-06-12T02:38:02.067 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.067 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.067 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:02.068 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690816+0000", 2020-06-12T02:38:02.068 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.162e-06 2020-06-12T02:38:02.068 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.068 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.068 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:02.068 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690819+0000", 2020-06-12T02:38:02.069 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2280000000000003e-06 2020-06-12T02:38:02.069 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.069 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.069 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:02.069 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690826+0000", 2020-06-12T02:38:02.070 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.452e-06 2020-06-12T02:38:02.070 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.070 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.070 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:02.070 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690828+0000", 2020-06-12T02:38:02.071 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00081402600000000003 2020-06-12T02:38:02.071 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.071 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.071 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.071 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.691642+0000", 2020-06-12T02:38:02.071 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00024263 2020-06-12T02:38:02.072 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.072 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.072 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:02.072 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.691885+0000", 2020-06-12T02:38:02.072 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010968739 2020-06-12T02:38:02.073 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.073 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.073 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.073 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.702853+0000", 2020-06-12T02:38:02.073 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8985000000000002e-05 2020-06-12T02:38:02.073 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.074 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.074 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:02.074 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.702952+0000", 2020-06-12T02:38:02.074 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0046973329999999997 2020-06-12T02:38:02.074 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.075 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.075 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.075 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.707650+0000", 2020-06-12T02:38:02.075 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3773e-05 2020-06-12T02:38:02.075 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.075 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.076 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:02.076 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.707723+0000", 2020-06-12T02:38:02.076 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023692990000000001 2020-06-12T02:38:02.076 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.076 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.077 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:02.077 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.710093+0000", 2020-06-12T02:38:02.077 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019298497000000001 2020-06-12T02:38:02.077 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.077 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:02.077 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.078 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.078 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.078 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8190 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~753664] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-06-12T02:38:02.078 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.894592+0000", 2020-06-12T02:38:02.078 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 8.9050177139999995, 2020-06-12T02:38:02.079 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016165763999999999, 2020-06-12T02:38:02.079 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:02.079 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:02.079 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:02.079 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:02.079 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:02.080 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8190 2020-06-12T02:38:02.080 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.080 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:02.080 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.080 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:02.081 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.894592+0000", 2020-06-12T02:38:02.081 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:02.081 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.081 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.081 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:02.081 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.894592+0000", 2020-06-12T02:38:02.082 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00074881500000000001 2020-06-12T02:38:02.082 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.082 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.082 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:02.082 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.895341+0000", 2020-06-12T02:38:02.083 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1919999999999999e-06 2020-06-12T02:38:02.083 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.083 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.083 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:02.083 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.895348+0000", 2020-06-12T02:38:02.083 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.059e-05 2020-06-12T02:38:02.084 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.084 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.084 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:02.084 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.895359+0000", 2020-06-12T02:38:02.084 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2310000000000001e-06 2020-06-12T02:38:02.085 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.085 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.085 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:02.085 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.895365+0000", 2020-06-12T02:38:02.085 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00049215800000000003 2020-06-12T02:38:02.085 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.086 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.086 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.086 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.895857+0000", 2020-06-12T02:38:02.086 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1001999999999995e-05 2020-06-12T02:38:02.086 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.087 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.087 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:02.087 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.895938+0000", 2020-06-12T02:38:02.087 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0059219040000000004 2020-06-12T02:38:02.087 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.087 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.088 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.088 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.901860+0000", 2020-06-12T02:38:02.088 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010156499999999999 2020-06-12T02:38:02.088 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.088 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.089 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:02.089 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.901961+0000", 2020-06-12T02:38:02.089 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00085242800000000002 2020-06-12T02:38:02.089 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.089 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.089 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:02.090 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.902814+0000", 2020-06-12T02:38:02.090 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0057806180000000004 2020-06-12T02:38:02.090 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.090 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.090 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:02.091 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.908594+0000", 2020-06-12T02:38:02.091 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021229600000000001 2020-06-12T02:38:02.091 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.091 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.091 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:02.091 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.910717+0000", 2020-06-12T02:38:02.092 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0300999999999999e-05 2020-06-12T02:38:02.092 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.092 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.092 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:02.092 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.910758+0000", 2020-06-12T02:38:02.092 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016165763999999999 2020-06-12T02:38:02.093 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.093 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:02.093 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.093 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.093 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.094 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8191 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~450560] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-06-12T02:38:02.094 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.897242+0000", 2020-06-12T02:38:02.094 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 8.9023672610000002, 2020-06-12T02:38:02.094 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014857576000000001, 2020-06-12T02:38:02.094 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:02.095 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:02.095 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:02.095 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:02.095 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:02.095 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8191 2020-06-12T02:38:02.095 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.096 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:02.096 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.096 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:02.096 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897242+0000", 2020-06-12T02:38:02.096 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:02.096 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.097 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.097 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:02.097 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897242+0000", 2020-06-12T02:38:02.097 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000531592 2020-06-12T02:38:02.097 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.098 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.098 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:02.098 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897774+0000", 2020-06-12T02:38:02.098 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0288999999999999e-05 2020-06-12T02:38:02.098 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.098 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.099 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:02.099 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897784+0000", 2020-06-12T02:38:02.099 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7149999999999993e-06 2020-06-12T02:38:02.099 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.099 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.100 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:02.100 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897794+0000", 2020-06-12T02:38:02.100 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8960000000000002e-06 2020-06-12T02:38:02.100 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.100 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.100 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:02.101 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897799+0000", 2020-06-12T02:38:02.101 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6251999999999998e-05 2020-06-12T02:38:02.101 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.101 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.101 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.102 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897835+0000", 2020-06-12T02:38:02.102 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.768e-05 2020-06-12T02:38:02.102 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.102 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.102 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:02.103 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897933+0000", 2020-06-12T02:38:02.103 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0082988100000000002 2020-06-12T02:38:02.103 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.103 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.103 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.103 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.906232+0000", 2020-06-12T02:38:02.104 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7865999999999998e-05 2020-06-12T02:38:02.104 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.104 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.104 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:02.104 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.906290+0000", 2020-06-12T02:38:02.104 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047895700000000002 2020-06-12T02:38:02.105 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.105 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.105 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:02.105 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.906768+0000", 2020-06-12T02:38:02.105 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002658442 2020-06-12T02:38:02.106 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.106 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.106 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:02.106 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.909427+0000", 2020-06-12T02:38:02.106 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0026066230000000002 2020-06-12T02:38:02.106 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.107 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.107 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:02.107 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.912034+0000", 2020-06-12T02:38:02.107 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6453999999999998e-05 2020-06-12T02:38:02.107 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.108 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.108 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:02.108 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.912100+0000", 2020-06-12T02:38:02.108 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014857576000000001 2020-06-12T02:38:02.108 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.108 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:02.109 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.109 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.109 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.109 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8193 3.3s0 3:c2ca77da:::smithi10729407-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e33)", 2020-06-12T02:38:02.109 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.897900+0000", 2020-06-12T02:38:02.110 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 8.9017093289999991, 2020-06-12T02:38:02.110 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015951213999999998, 2020-06-12T02:38:02.110 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:02.110 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:02.110 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:02.110 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:02.111 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:02.111 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8193 2020-06-12T02:38:02.111 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.111 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:02.111 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.112 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:02.112 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897900+0000", 2020-06-12T02:38:02.112 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:02.112 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.112 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.112 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:02.113 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897900+0000", 2020-06-12T02:38:02.113 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2331e-05 2020-06-12T02:38:02.113 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.113 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.113 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:02.114 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897923+0000", 2020-06-12T02:38:02.114 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.021e-06 2020-06-12T02:38:02.114 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.114 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.114 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:02.114 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897926+0000", 2020-06-12T02:38:02.115 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4150000000000002e-06 2020-06-12T02:38:02.115 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.115 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.115 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:02.115 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897933+0000", 2020-06-12T02:38:02.116 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2359999999999999e-06 2020-06-12T02:38:02.116 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.116 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.116 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:02.116 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897935+0000", 2020-06-12T02:38:02.116 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000118301 2020-06-12T02:38:02.117 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.117 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.117 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.117 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.898054+0000", 2020-06-12T02:38:02.117 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8914999999999998e-05 2020-06-12T02:38:02.118 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.118 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.118 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:02.118 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.898143+0000", 2020-06-12T02:38:02.118 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011203013 2020-06-12T02:38:02.118 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.119 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.119 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.119 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.909346+0000", 2020-06-12T02:38:02.119 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7075000000000003e-05 2020-06-12T02:38:02.119 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.119 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.120 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:02.120 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.909423+0000", 2020-06-12T02:38:02.120 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033600969999999998 2020-06-12T02:38:02.120 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.120 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.121 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.121 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.912783+0000", 2020-06-12T02:38:02.121 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6118000000000005e-05 2020-06-12T02:38:02.121 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.121 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.121 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:02.122 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.912869+0000", 2020-06-12T02:38:02.122 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00098269200000000007 2020-06-12T02:38:02.122 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.122 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.122 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:02.123 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.913852+0000", 2020-06-12T02:38:02.123 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015951213999999998 2020-06-12T02:38:02.123 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.123 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:02.123 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.123 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.124 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:02.124 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:02.125 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:02.123+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:02.126 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-06-12T02:38:02.226 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:02.223+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:02.283 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:02.283 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:38:02.283 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2020-06-12T02:38:02.284 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:02.291 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-06-12T02:38:02.327 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:02.325+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:02.428 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:02.426+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:02.447 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:02.447 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:38:02.447 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:38:02.448 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:38:02.448 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:02.455 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-06-12T02:38:02.531 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:02.528+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:02.601 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.4 to 1 (10000) 2020-06-12T02:38:02.603 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd set nodeep-scrub 2020-06-12T02:38:02.611 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd reweight 5 1 2020-06-12T02:38:02.614 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:02.615 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:38:02.615 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:38:02.615 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:38:02.615 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.615 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1206 3.5s0 3:a9510bda:::smithi10729407-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:38:02.616 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.856688+0000", 2020-06-12T02:38:02.618 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 29.754272625999999, 2020-06-12T02:38:02.622 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019116942000000001, 2020-06-12T02:38:02.622 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:02.622 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:02.622 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:02.623 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:02.623 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:02.623 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1206 2020-06-12T02:38:02.623 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.623 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:02.624 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.624 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:02.624 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856688+0000", 2020-06-12T02:38:02.624 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:02.624 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.625 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.625 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:02.625 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856688+0000", 2020-06-12T02:38:02.625 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4355e-05 2020-06-12T02:38:02.625 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.626 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.626 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:02.626 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856713+0000", 2020-06-12T02:38:02.626 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0850000000000001e-06 2020-06-12T02:38:02.626 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.626 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.627 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:02.627 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856716+0000", 2020-06-12T02:38:02.627 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4509999999999997e-06 2020-06-12T02:38:02.629 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.629 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.629 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:02.629 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856722+0000", 2020-06-12T02:38:02.629 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2879999999999999e-06 2020-06-12T02:38:02.630 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.630 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.630 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:02.630 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856724+0000", 2020-06-12T02:38:02.630 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0057507449999999998 2020-06-12T02:38:02.631 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.631 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.632 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.633 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.862475+0000", 2020-06-12T02:38:02.633 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5759000000000003e-05 2020-06-12T02:38:02.633 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.633 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.633 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:02.634 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.862571+0000", 2020-06-12T02:38:02.634 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012199018000000001 2020-06-12T02:38:02.634 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.634 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.634 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.635 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.874770+0000", 2020-06-12T02:38:02.635 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7260999999999999e-05 2020-06-12T02:38:02.635 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.635 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.635 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:02.635 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.874827+0000", 2020-06-12T02:38:02.636 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00097798000000000004 2020-06-12T02:38:02.636 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.636 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.636 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:02.636 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.875805+0000", 2020-06-12T02:38:02.637 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019116942000000001 2020-06-12T02:38:02.637 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.637 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:02.637 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.637 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.637 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.638 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1572 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~745472] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-06-12T02:38:02.638 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.886439+0000", 2020-06-12T02:38:02.638 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 28.724521908, 2020-06-12T02:38:02.638 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021442571000000001, 2020-06-12T02:38:02.639 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:02.639 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:02.639 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:02.639 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:02.640 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:02.640 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1572 2020-06-12T02:38:02.640 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.640 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:02.640 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.640 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:02.641 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.886439+0000", 2020-06-12T02:38:02.641 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:02.641 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.643 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.644 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:02.644 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.886439+0000", 2020-06-12T02:38:02.644 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00073027599999999997 2020-06-12T02:38:02.644 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.645 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.645 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:02.645 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887169+0000", 2020-06-12T02:38:02.645 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.49e-06 2020-06-12T02:38:02.645 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.646 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.646 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:02.646 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887175+0000", 2020-06-12T02:38:02.646 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7899999999999994e-06 2020-06-12T02:38:02.646 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.646 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.647 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:02.647 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887185+0000", 2020-06-12T02:38:02.647 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.42e-06 2020-06-12T02:38:02.647 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.647 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.648 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:02.648 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887189+0000", 2020-06-12T02:38:02.648 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1619999999999999e-05 2020-06-12T02:38:02.648 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.648 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.648 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.649 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887221+0000", 2020-06-12T02:38:02.649 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9720000000000002e-05 2020-06-12T02:38:02.649 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.649 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.649 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:02.650 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887300+0000", 2020-06-12T02:38:02.650 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047354100000000001 2020-06-12T02:38:02.650 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.650 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.650 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:02.651 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887774+0000", 2020-06-12T02:38:02.651 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0030649750000000002 2020-06-12T02:38:02.651 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.651 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.651 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:02.652 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.890839+0000", 2020-06-12T02:38:02.652 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016995321000000001 2020-06-12T02:38:02.653 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.653 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.653 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:02.653 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.907834+0000", 2020-06-12T02:38:02.653 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7417999999999999e-05 2020-06-12T02:38:02.653 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.654 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.654 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:02.654 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.907882+0000", 2020-06-12T02:38:02.654 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021442571000000001 2020-06-12T02:38:02.654 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.655 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:02.655 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.655 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.655 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.655 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1573 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~712704] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-06-12T02:38:02.656 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.890888+0000", 2020-06-12T02:38:02.656 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 28.720073172999999, 2020-06-12T02:38:02.656 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018868001999999998, 2020-06-12T02:38:02.656 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:02.656 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:02.656 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:02.657 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:02.657 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:02.657 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1573 2020-06-12T02:38:02.657 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.657 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:02.660 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.660 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:02.660 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.890888+0000", 2020-06-12T02:38:02.660 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:02.660 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.661 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.661 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:02.661 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.890888+0000", 2020-06-12T02:38:02.661 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000683605 2020-06-12T02:38:02.661 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.662 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.662 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:02.662 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891571+0000", 2020-06-12T02:38:02.662 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4999999999999999e-06 2020-06-12T02:38:02.663 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.663 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.663 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:02.663 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891577+0000", 2020-06-12T02:38:02.663 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.171e-05 2020-06-12T02:38:02.664 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.664 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.664 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:02.664 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891589+0000", 2020-06-12T02:38:02.664 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7650000000000001e-06 2020-06-12T02:38:02.664 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.665 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.665 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:02.665 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891593+0000", 2020-06-12T02:38:02.665 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6689000000000001e-05 2020-06-12T02:38:02.665 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.666 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.666 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.666 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891630+0000", 2020-06-12T02:38:02.667 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9877999999999997e-05 2020-06-12T02:38:02.667 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.667 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.667 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:02.667 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891710+0000", 2020-06-12T02:38:02.667 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053367200000000001 2020-06-12T02:38:02.668 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.668 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.668 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:02.670 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.892244+0000", 2020-06-12T02:38:02.670 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004927196 2020-06-12T02:38:02.671 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.671 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.671 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:02.671 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.897171+0000", 2020-06-12T02:38:02.671 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012496419 2020-06-12T02:38:02.671 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.672 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.672 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:02.672 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.909667+0000", 2020-06-12T02:38:02.672 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8567999999999996e-05 2020-06-12T02:38:02.673 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.673 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.674 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:02.674 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.909756+0000", 2020-06-12T02:38:02.674 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018868001999999998 2020-06-12T02:38:02.675 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.675 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:02.675 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.675 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.675 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.676 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1576 3.5s0 3:a446366d:::smithi10729407-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e26)", 2020-06-12T02:38:02.676 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.891955+0000", 2020-06-12T02:38:02.676 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 28.719005932999998, 2020-06-12T02:38:02.677 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019893246999999999, 2020-06-12T02:38:02.677 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:02.677 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:02.678 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:02.678 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:02.678 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:02.679 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1576 2020-06-12T02:38:02.679 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.679 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:02.679 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.679 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:02.680 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891955+0000", 2020-06-12T02:38:02.680 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:02.680 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.680 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.680 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:02.680 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891955+0000", 2020-06-12T02:38:02.681 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2067999999999999e-05 2020-06-12T02:38:02.681 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.681 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.681 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:02.681 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891977+0000", 2020-06-12T02:38:02.682 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.18e-06 2020-06-12T02:38:02.682 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.682 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.682 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:02.684 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891980+0000", 2020-06-12T02:38:02.684 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3629999999999999e-06 2020-06-12T02:38:02.684 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.684 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.685 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:02.685 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891987+0000", 2020-06-12T02:38:02.685 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.181e-06 2020-06-12T02:38:02.685 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.685 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.686 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:02.686 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891989+0000", 2020-06-12T02:38:02.686 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050959120000000002 2020-06-12T02:38:02.686 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.687 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.687 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.687 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.897085+0000", 2020-06-12T02:38:02.687 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1089999999999998e-05 2020-06-12T02:38:02.688 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.688 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.688 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:02.688 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.897166+0000", 2020-06-12T02:38:02.688 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013400235 2020-06-12T02:38:02.689 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.689 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.689 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.689 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.910566+0000", 2020-06-12T02:38:02.689 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000104981 2020-06-12T02:38:02.689 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.690 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.690 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:02.690 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.910671+0000", 2020-06-12T02:38:02.690 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001177237 2020-06-12T02:38:02.690 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.691 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.691 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:02.691 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.911848+0000", 2020-06-12T02:38:02.691 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019893246999999999 2020-06-12T02:38:02.691 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.692 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:02.692 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.692 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.692 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.692 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2913 3.5s0 3:a9510bda:::smithi10729407-14:head [append 0~770048,stat] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:02.692 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.070354+0000", 2020-06-12T02:38:02.693 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 25.540607132000002, 2020-06-12T02:38:02.693 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60690798599999995, 2020-06-12T02:38:02.693 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:02.693 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:02.693 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:02.694 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:02.694 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:02.694 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2913 2020-06-12T02:38:02.694 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.695 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:02.695 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.695 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:02.695 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.070354+0000", 2020-06-12T02:38:02.695 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:02.696 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.696 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.696 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:02.696 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.070354+0000", 2020-06-12T02:38:02.696 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00098478900000000002 2020-06-12T02:38:02.696 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.697 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.697 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:02.697 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071339+0000", 2020-06-12T02:38:02.698 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8349999999999993e-06 2020-06-12T02:38:02.698 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.698 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.698 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:02.698 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071347+0000", 2020-06-12T02:38:02.699 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0749e-05 2020-06-12T02:38:02.699 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.699 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.699 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:02.699 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071358+0000", 2020-06-12T02:38:02.699 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9369999999999997e-06 2020-06-12T02:38:02.700 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.700 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.700 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:02.700 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071362+0000", 2020-06-12T02:38:02.700 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0426e-05 2020-06-12T02:38:02.701 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.701 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.701 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.701 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071423+0000", 2020-06-12T02:38:02.701 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4129999999999998e-05 2020-06-12T02:38:02.701 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.702 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.702 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:02.702 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071487+0000", 2020-06-12T02:38:02.702 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053752599999999998 2020-06-12T02:38:02.702 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.703 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.703 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:02.703 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072024+0000", 2020-06-12T02:38:02.703 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036162579999999998 2020-06-12T02:38:02.703 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.703 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.704 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:02.704 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075640+0000", 2020-06-12T02:38:02.704 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60152201999999999 2020-06-12T02:38:02.704 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.705 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.705 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:02.705 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677162+0000", 2020-06-12T02:38:02.705 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9315999999999996e-05 2020-06-12T02:38:02.705 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.706 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.706 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:02.706 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677262+0000", 2020-06-12T02:38:02.706 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60690798599999995 2020-06-12T02:38:02.706 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.707 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:02.707 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.707 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.708 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.708 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2914 3.5s0 3:a9510bda:::smithi10729407-14:head [append 0~401408] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:02.708 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.072409+0000", 2020-06-12T02:38:02.708 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 25.538551687999998, 2020-06-12T02:38:02.708 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.61956801100000003, 2020-06-12T02:38:02.709 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:02.709 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:02.709 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:02.709 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:02.709 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:02.710 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2914 2020-06-12T02:38:02.710 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.710 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:02.712 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.712 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:02.713 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072409+0000", 2020-06-12T02:38:02.713 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:02.713 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.713 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.713 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:02.714 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072409+0000", 2020-06-12T02:38:02.714 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000446317 2020-06-12T02:38:02.714 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.714 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.714 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:02.715 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072856+0000", 2020-06-12T02:38:02.715 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2140000000000004e-06 2020-06-12T02:38:02.715 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.715 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.715 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:02.716 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072861+0000", 2020-06-12T02:38:02.716 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8710000000000003e-06 2020-06-12T02:38:02.716 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.716 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.716 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:02.716 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072870+0000", 2020-06-12T02:38:02.717 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2270000000000004e-06 2020-06-12T02:38:02.717 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.717 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.717 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:02.717 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072874+0000", 2020-06-12T02:38:02.718 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024649960000000001 2020-06-12T02:38:02.718 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.718 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.718 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.718 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075339+0000", 2020-06-12T02:38:02.718 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9179999999999996e-05 2020-06-12T02:38:02.719 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.719 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.719 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:02.719 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075438+0000", 2020-06-12T02:38:02.719 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60189281400000005 2020-06-12T02:38:02.720 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.720 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.720 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.720 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677331+0000", 2020-06-12T02:38:02.720 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012602800000000001 2020-06-12T02:38:02.721 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.721 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.721 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:02.721 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677457+0000", 2020-06-12T02:38:02.721 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00088167699999999998 2020-06-12T02:38:02.721 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.722 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.722 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:02.722 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.678339+0000", 2020-06-12T02:38:02.722 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004312995 2020-06-12T02:38:02.722 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.723 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.723 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:02.723 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.682652+0000", 2020-06-12T02:38:02.723 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0092869140000000003 2020-06-12T02:38:02.723 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.723 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.724 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:02.724 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691938+0000", 2020-06-12T02:38:02.724 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8778e-05 2020-06-12T02:38:02.724 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.724 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.725 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:02.725 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691977+0000", 2020-06-12T02:38:02.725 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.61956801100000003 2020-06-12T02:38:02.725 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.725 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:02.726 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.726 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.726 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.726 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2915 3.5s0 3:a9510bda:::smithi10729407-14:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:02.726 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.072912+0000", 2020-06-12T02:38:02.726 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 25.538048948, 2020-06-12T02:38:02.727 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.6192204, 2020-06-12T02:38:02.727 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:02.727 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:02.727 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:02.727 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:02.728 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:02.728 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2915 2020-06-12T02:38:02.728 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.728 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:02.728 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.728 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:02.729 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072912+0000", 2020-06-12T02:38:02.729 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:02.729 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.729 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.729 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:02.730 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072912+0000", 2020-06-12T02:38:02.730 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9201e-05 2020-06-12T02:38:02.730 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.730 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.730 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:02.731 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072951+0000", 2020-06-12T02:38:02.731 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5480000000000002e-06 2020-06-12T02:38:02.731 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.732 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.732 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:02.732 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072955+0000", 2020-06-12T02:38:02.732 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3879999999999997e-06 2020-06-12T02:38:02.732 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.733 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.733 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:02.733 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072961+0000", 2020-06-12T02:38:02.733 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6589999999999999e-06 2020-06-12T02:38:02.733 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.734 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.734 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:02.734 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072964+0000", 2020-06-12T02:38:02.734 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025221039999999998 2020-06-12T02:38:02.734 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.734 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.735 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.735 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075486+0000", 2020-06-12T02:38:02.735 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5038999999999998e-05 2020-06-12T02:38:02.735 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.735 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.736 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:02.736 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075561+0000", 2020-06-12T02:38:02.736 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60452397499999999 2020-06-12T02:38:02.736 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.736 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.736 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.737 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680085+0000", 2020-06-12T02:38:02.737 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1398999999999999e-05 2020-06-12T02:38:02.737 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.737 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.737 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:02.738 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680156+0000", 2020-06-12T02:38:02.738 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000274465 2020-06-12T02:38:02.738 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.738 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.738 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:02.739 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680431+0000", 2020-06-12T02:38:02.739 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025071999999999998 2020-06-12T02:38:02.739 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.739 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.739 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:02.739 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.682938+0000", 2020-06-12T02:38:02.740 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0091116449999999998 2020-06-12T02:38:02.740 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.740 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.740 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:02.740 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692050+0000", 2020-06-12T02:38:02.741 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2776999999999997e-05 2020-06-12T02:38:02.741 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.741 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.741 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:02.741 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692132+0000", 2020-06-12T02:38:02.742 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.6192204 2020-06-12T02:38:02.742 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.742 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:02.742 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.742 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.742 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.743 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2916 3.5s0 3:a9510bda:::smithi10729407-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:38:02.743 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.072986+0000", 2020-06-12T02:38:02.743 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 25.537974709, 2020-06-12T02:38:02.743 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.62063941099999997, 2020-06-12T02:38:02.744 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:02.744 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:02.744 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:02.744 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:02.744 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:02.745 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2916 2020-06-12T02:38:02.745 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.745 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:02.745 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.745 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:02.746 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072986+0000", 2020-06-12T02:38:02.746 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:02.746 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.746 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.746 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:02.746 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072986+0000", 2020-06-12T02:38:02.747 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2101000000000001e-05 2020-06-12T02:38:02.747 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.747 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.747 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:02.747 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073008+0000", 2020-06-12T02:38:02.748 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1549999999999999e-06 2020-06-12T02:38:02.748 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.748 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.748 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:02.748 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073011+0000", 2020-06-12T02:38:02.748 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2360000000000001e-06 2020-06-12T02:38:02.749 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.749 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.749 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:02.749 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073018+0000", 2020-06-12T02:38:02.749 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2790000000000001e-06 2020-06-12T02:38:02.750 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.750 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.750 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:02.750 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073020+0000", 2020-06-12T02:38:02.750 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025637670000000002 2020-06-12T02:38:02.751 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.751 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.751 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.751 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075584+0000", 2020-06-12T02:38:02.751 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1814999999999998e-05 2020-06-12T02:38:02.751 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.752 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.752 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:02.752 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075636+0000", 2020-06-12T02:38:02.752 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60499943199999995 2020-06-12T02:38:02.752 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.753 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.753 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.753 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680635+0000", 2020-06-12T02:38:02.753 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0968000000000002e-05 2020-06-12T02:38:02.753 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.754 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.754 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:02.754 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680706+0000", 2020-06-12T02:38:02.754 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011777101999999999 2020-06-12T02:38:02.754 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.754 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.755 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.755 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692483+0000", 2020-06-12T02:38:02.755 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5890999999999995e-05 2020-06-12T02:38:02.755 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.755 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.756 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:02.756 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692569+0000", 2020-06-12T02:38:02.756 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001056665 2020-06-12T02:38:02.756 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.756 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.756 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:02.757 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.693626+0000", 2020-06-12T02:38:02.757 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.62063941099999997 2020-06-12T02:38:02.757 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.757 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:02.758 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.758 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.758 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.758 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2922 3.ds1 3:b983fd28:::smithi10729407-1:head [append 0~466944] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:02.758 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.097192+0000", 2020-06-12T02:38:02.759 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 25.513769342, 2020-06-12T02:38:02.759 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58672644699999998, 2020-06-12T02:38:02.759 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:02.759 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:02.759 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:02.759 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:02.760 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:02.760 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2922 2020-06-12T02:38:02.760 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.760 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:02.760 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.761 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:02.761 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097192+0000", 2020-06-12T02:38:02.761 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:02.761 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.761 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.762 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:02.762 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097192+0000", 2020-06-12T02:38:02.762 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047206100000000002 2020-06-12T02:38:02.762 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.762 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.762 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:02.763 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097664+0000", 2020-06-12T02:38:02.763 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8359999999999999e-06 2020-06-12T02:38:02.763 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.763 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.763 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:02.764 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097668+0000", 2020-06-12T02:38:02.764 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0815999999999999e-05 2020-06-12T02:38:02.764 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.764 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.764 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:02.764 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097679+0000", 2020-06-12T02:38:02.765 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.028e-06 2020-06-12T02:38:02.765 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.765 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.765 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:02.765 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097684+0000", 2020-06-12T02:38:02.766 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3932999999999999e-05 2020-06-12T02:38:02.766 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.766 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.766 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.766 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097748+0000", 2020-06-12T02:38:02.766 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2214999999999999e-05 2020-06-12T02:38:02.767 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.767 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.767 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:02.767 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097820+0000", 2020-06-12T02:38:02.767 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00041583800000000003 2020-06-12T02:38:02.768 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.768 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.768 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:02.768 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.098236+0000", 2020-06-12T02:38:02.769 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0020339960000000002 2020-06-12T02:38:02.769 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.769 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.769 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:02.769 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.100270+0000", 2020-06-12T02:38:02.770 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58355657299999997 2020-06-12T02:38:02.770 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.770 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.770 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:02.771 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.683827+0000", 2020-06-12T02:38:02.772 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1150999999999995e-05 2020-06-12T02:38:02.772 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.772 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.772 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:02.772 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.683918+0000", 2020-06-12T02:38:02.773 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58672644699999998 2020-06-12T02:38:02.773 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.773 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:02.773 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.773 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.774 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.774 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2923 3.ds1 3:b983fd28:::smithi10729407-1:head [append 0~614400] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:02.774 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.100504+0000", 2020-06-12T02:38:02.774 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 25.510456700999999, 2020-06-12T02:38:02.774 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58853946999999995, 2020-06-12T02:38:02.775 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:02.775 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:02.775 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:02.775 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:02.775 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:02.775 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2923 2020-06-12T02:38:02.776 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.776 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:02.776 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.776 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:02.776 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.100504+0000", 2020-06-12T02:38:02.777 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:02.777 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.777 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.777 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:02.777 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.100504+0000", 2020-06-12T02:38:02.778 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00068395399999999996 2020-06-12T02:38:02.778 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.778 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.778 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:02.778 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101188+0000", 2020-06-12T02:38:02.778 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4050000000000003e-06 2020-06-12T02:38:02.779 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.779 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.779 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:02.779 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101196+0000", 2020-06-12T02:38:02.779 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.328e-05 2020-06-12T02:38:02.780 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.780 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.780 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:02.780 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101209+0000", 2020-06-12T02:38:02.781 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3419999999999996e-06 2020-06-12T02:38:02.781 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.781 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.781 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:02.781 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101215+0000", 2020-06-12T02:38:02.782 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1389000000000005e-05 2020-06-12T02:38:02.782 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.782 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.782 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.782 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101277+0000", 2020-06-12T02:38:02.782 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7724000000000004e-05 2020-06-12T02:38:02.783 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.783 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.783 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:02.783 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101344+0000", 2020-06-12T02:38:02.783 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047852499999999999 2020-06-12T02:38:02.784 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.784 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.784 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:02.784 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101823+0000", 2020-06-12T02:38:02.784 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0039255510000000002 2020-06-12T02:38:02.784 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.785 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.785 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:02.785 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105748+0000", 2020-06-12T02:38:02.785 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58321340099999996 2020-06-12T02:38:02.785 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.786 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.786 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:02.786 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.688962+0000", 2020-06-12T02:38:02.786 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1898999999999996e-05 2020-06-12T02:38:02.786 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.787 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.787 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:02.787 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689044+0000", 2020-06-12T02:38:02.787 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58853946999999995 2020-06-12T02:38:02.787 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.787 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:02.788 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.788 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.788 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.788 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2924 3.ds1 3:b983fd28:::smithi10729407-1:head [append 0~188416] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:02.788 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.101633+0000", 2020-06-12T02:38:02.789 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 25.509327707000001, 2020-06-12T02:38:02.789 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58758259199999996, 2020-06-12T02:38:02.789 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:02.789 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:02.789 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:02.790 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:02.790 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:02.790 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2924 2020-06-12T02:38:02.790 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.790 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:02.790 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.791 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:02.791 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101633+0000", 2020-06-12T02:38:02.791 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:02.791 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.791 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.792 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:02.792 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101633+0000", 2020-06-12T02:38:02.792 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00018775399999999999 2020-06-12T02:38:02.792 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.792 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.792 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:02.793 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101821+0000", 2020-06-12T02:38:02.793 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7669999999999998e-06 2020-06-12T02:38:02.793 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.793 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.793 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:02.794 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101828+0000", 2020-06-12T02:38:02.794 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2716e-05 2020-06-12T02:38:02.794 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.794 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.794 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:02.795 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101840+0000", 2020-06-12T02:38:02.795 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.215e-06 2020-06-12T02:38:02.795 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.795 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.795 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:02.795 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101846+0000", 2020-06-12T02:38:02.796 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021214250000000001 2020-06-12T02:38:02.796 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.796 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.796 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.796 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.103967+0000", 2020-06-12T02:38:02.797 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1018000000000003e-05 2020-06-12T02:38:02.797 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.797 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.797 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:02.797 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.104038+0000", 2020-06-12T02:38:02.797 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000333364 2020-06-12T02:38:02.798 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.798 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.798 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:02.798 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.104371+0000", 2020-06-12T02:38:02.798 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001378546 2020-06-12T02:38:02.799 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.799 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.800 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:02.800 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105750+0000", 2020-06-12T02:38:02.800 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58338272899999999 2020-06-12T02:38:02.800 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.800 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.800 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:02.801 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689133+0000", 2020-06-12T02:38:02.801 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3058000000000003e-05 2020-06-12T02:38:02.801 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.801 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.801 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:02.802 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689216+0000", 2020-06-12T02:38:02.802 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58758259199999996 2020-06-12T02:38:02.802 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.802 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:02.802 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.803 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.803 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.803 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2925 3.ds1 3:b983fd28:::smithi10729407-1:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:02.803 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.101899+0000", 2020-06-12T02:38:02.803 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 25.509062042, 2020-06-12T02:38:02.804 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58747810899999997, 2020-06-12T02:38:02.804 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:02.804 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:02.804 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:02.804 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:02.804 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:02.805 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2925 2020-06-12T02:38:02.805 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.805 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:02.806 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.806 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:02.806 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101899+0000", 2020-06-12T02:38:02.806 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:02.806 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.806 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.807 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:02.807 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101899+0000", 2020-06-12T02:38:02.807 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2906999999999998e-05 2020-06-12T02:38:02.807 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.807 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.808 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:02.808 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101972+0000", 2020-06-12T02:38:02.808 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1789999999999996e-06 2020-06-12T02:38:02.808 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.808 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.808 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:02.809 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101978+0000", 2020-06-12T02:38:02.809 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1358e-05 2020-06-12T02:38:02.809 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.809 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.809 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:02.810 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101989+0000", 2020-06-12T02:38:02.810 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.031e-06 2020-06-12T02:38:02.810 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.810 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.810 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:02.811 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101993+0000", 2020-06-12T02:38:02.811 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033218990000000001 2020-06-12T02:38:02.811 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.811 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.811 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.811 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105315+0000", 2020-06-12T02:38:02.812 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6324999999999999e-05 2020-06-12T02:38:02.812 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.812 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.812 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:02.812 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105382+0000", 2020-06-12T02:38:02.813 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000216624 2020-06-12T02:38:02.813 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.813 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.813 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:02.813 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105598+0000", 2020-06-12T02:38:02.814 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001236586 2020-06-12T02:38:02.814 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.814 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.814 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:02.814 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.106835+0000", 2020-06-12T02:38:02.814 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58245509200000001 2020-06-12T02:38:02.815 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.815 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.815 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:02.815 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689290+0000", 2020-06-12T02:38:02.815 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7107999999999993e-05 2020-06-12T02:38:02.816 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.816 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.816 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:02.816 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689377+0000", 2020-06-12T02:38:02.816 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58747810899999997 2020-06-12T02:38:02.816 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.817 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:02.817 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.817 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.817 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.817 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2926 3.ds1 3:b983fd28:::smithi10729407-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:38:02.818 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.102036+0000", 2020-06-12T02:38:02.818 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 25.508925057999999, 2020-06-12T02:38:02.818 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58868048699999997, 2020-06-12T02:38:02.818 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:02.818 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:02.819 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:02.819 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:02.819 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:02.819 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2926 2020-06-12T02:38:02.819 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.820 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:02.820 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.820 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:02.820 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102036+0000", 2020-06-12T02:38:02.820 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:02.820 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.821 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.821 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:02.821 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102036+0000", 2020-06-12T02:38:02.821 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3786999999999999e-05 2020-06-12T02:38:02.821 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.822 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.822 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:02.822 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102080+0000", 2020-06-12T02:38:02.822 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7660000000000004e-06 2020-06-12T02:38:02.822 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.822 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.823 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:02.823 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102085+0000", 2020-06-12T02:38:02.823 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3145e-05 2020-06-12T02:38:02.823 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.823 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.824 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:02.824 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102099+0000", 2020-06-12T02:38:02.824 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2440000000000004e-06 2020-06-12T02:38:02.824 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.824 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.825 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:02.825 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102103+0000", 2020-06-12T02:38:02.825 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036826990000000002 2020-06-12T02:38:02.825 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.825 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.825 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.826 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105786+0000", 2020-06-12T02:38:02.826 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.6774000000000006e-05 2020-06-12T02:38:02.826 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.826 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.826 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:02.827 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105882+0000", 2020-06-12T02:38:02.827 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58352084900000001 2020-06-12T02:38:02.827 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.827 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.827 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.828 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689403+0000", 2020-06-12T02:38:02.828 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3788999999999995e-05 2020-06-12T02:38:02.828 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.828 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.828 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:02.828 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689467+0000", 2020-06-12T02:38:02.829 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001249434 2020-06-12T02:38:02.829 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.829 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.829 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:02.829 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690716+0000", 2020-06-12T02:38:02.830 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58868048699999997 2020-06-12T02:38:02.830 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.830 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:02.830 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.830 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.831 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.831 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2950 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~598016] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:02.831 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.162562+0000", 2020-06-12T02:38:02.831 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 25.448398684000001, 2020-06-12T02:38:02.832 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52780611399999999, 2020-06-12T02:38:02.832 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:02.832 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:02.832 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:02.833 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:02.833 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:02.833 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2950 2020-06-12T02:38:02.833 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.833 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:02.833 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.834 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:02.834 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.162562+0000", 2020-06-12T02:38:02.834 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:02.834 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.834 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.835 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:02.835 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.162562+0000", 2020-06-12T02:38:02.835 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00070718099999999998 2020-06-12T02:38:02.835 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.835 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.836 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:02.836 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163269+0000", 2020-06-12T02:38:02.836 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2029999999999997e-06 2020-06-12T02:38:02.836 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.836 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.836 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:02.837 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163276+0000", 2020-06-12T02:38:02.837 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2734999999999999e-05 2020-06-12T02:38:02.837 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.837 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.837 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:02.838 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163288+0000", 2020-06-12T02:38:02.838 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.404e-06 2020-06-12T02:38:02.838 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.838 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.838 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:02.839 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163295+0000", 2020-06-12T02:38:02.839 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5784000000000006e-05 2020-06-12T02:38:02.839 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.839 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.839 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.839 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163361+0000", 2020-06-12T02:38:02.840 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5318e-05 2020-06-12T02:38:02.840 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.840 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.840 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:02.840 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163436+0000", 2020-06-12T02:38:02.841 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054801100000000005 2020-06-12T02:38:02.841 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.841 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.841 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:02.841 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163984+0000", 2020-06-12T02:38:02.841 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.007023077 2020-06-12T02:38:02.842 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.842 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.842 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:02.842 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171007+0000", 2020-06-12T02:38:02.842 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51932051199999996 2020-06-12T02:38:02.843 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.843 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.843 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:02.843 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690327+0000", 2020-06-12T02:38:02.843 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0889000000000001e-05 2020-06-12T02:38:02.843 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.844 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.844 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:02.844 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690368+0000", 2020-06-12T02:38:02.844 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52780611399999999 2020-06-12T02:38:02.844 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.845 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:02.845 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.845 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.845 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.845 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2951 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~532480] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:02.845 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.165470+0000", 2020-06-12T02:38:02.846 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 25.445491076, 2020-06-12T02:38:02.846 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52507917900000001, 2020-06-12T02:38:02.846 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:02.846 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:02.846 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:02.847 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:02.847 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:02.847 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2951 2020-06-12T02:38:02.847 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.847 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:02.847 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.848 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:02.848 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.165470+0000", 2020-06-12T02:38:02.848 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:02.848 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.848 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.849 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:02.849 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.165470+0000", 2020-06-12T02:38:02.849 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00062231600000000004 2020-06-12T02:38:02.849 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.849 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.849 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:02.850 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166092+0000", 2020-06-12T02:38:02.850 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1610000000000002e-06 2020-06-12T02:38:02.850 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.850 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.850 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:02.851 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166097+0000", 2020-06-12T02:38:02.851 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7780000000000002e-06 2020-06-12T02:38:02.851 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.851 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.851 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:02.851 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166107+0000", 2020-06-12T02:38:02.852 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7169999999999999e-06 2020-06-12T02:38:02.852 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.852 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.852 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:02.852 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166112+0000", 2020-06-12T02:38:02.853 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4792000000000002e-05 2020-06-12T02:38:02.853 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.853 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.853 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.853 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166147+0000", 2020-06-12T02:38:02.853 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9336e-05 2020-06-12T02:38:02.854 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.854 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.854 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:02.854 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166236+0000", 2020-06-12T02:38:02.854 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00067747200000000004 2020-06-12T02:38:02.855 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.855 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.855 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:02.855 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166913+0000", 2020-06-12T02:38:02.855 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0047718669999999999 2020-06-12T02:38:02.855 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.856 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.856 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:02.856 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171685+0000", 2020-06-12T02:38:02.856 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51881990700000002 2020-06-12T02:38:02.856 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.857 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.857 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:02.857 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690505+0000", 2020-06-12T02:38:02.857 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3832999999999999e-05 2020-06-12T02:38:02.857 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.858 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.858 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:02.858 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690549+0000", 2020-06-12T02:38:02.858 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52507917900000001 2020-06-12T02:38:02.858 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.859 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:02.859 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.859 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.859 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.859 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2952 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~270336] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:02.859 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.166891+0000", 2020-06-12T02:38:02.860 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 25.444069426999999, 2020-06-12T02:38:02.860 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52426417000000003, 2020-06-12T02:38:02.860 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:02.860 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:02.860 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:02.861 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:02.861 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:02.861 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2952 2020-06-12T02:38:02.861 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.861 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:02.862 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.862 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:02.862 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166891+0000", 2020-06-12T02:38:02.862 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:02.862 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.862 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.863 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:02.863 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166891+0000", 2020-06-12T02:38:02.863 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00031878599999999999 2020-06-12T02:38:02.863 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.863 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.864 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:02.864 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167210+0000", 2020-06-12T02:38:02.864 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2360000000000004e-06 2020-06-12T02:38:02.864 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.864 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.864 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:02.865 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167219+0000", 2020-06-12T02:38:02.865 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3298e-05 2020-06-12T02:38:02.865 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.865 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.865 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:02.866 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167232+0000", 2020-06-12T02:38:02.866 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5849999999999999e-06 2020-06-12T02:38:02.866 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.866 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.866 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:02.866 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167237+0000", 2020-06-12T02:38:02.867 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0018755740000000001 2020-06-12T02:38:02.867 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.867 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.867 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.867 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.169113+0000", 2020-06-12T02:38:02.868 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000104768 2020-06-12T02:38:02.868 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.868 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.868 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:02.868 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.169218+0000", 2020-06-12T02:38:02.868 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00057389799999999999 2020-06-12T02:38:02.869 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.869 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.869 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:02.869 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.169792+0000", 2020-06-12T02:38:02.869 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0019333989999999999 2020-06-12T02:38:02.870 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.870 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.870 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:02.870 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171725+0000", 2020-06-12T02:38:02.870 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51938508699999997 2020-06-12T02:38:02.870 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.871 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.871 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:02.871 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691110+0000", 2020-06-12T02:38:02.871 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5538999999999999e-05 2020-06-12T02:38:02.871 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.872 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.872 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:02.872 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691156+0000", 2020-06-12T02:38:02.872 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52426417000000003 2020-06-12T02:38:02.872 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.872 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:02.873 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.873 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.873 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.873 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2953 3.5s0 3:a446366d:::smithi10729407-11:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:02.873 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.167293+0000", 2020-06-12T02:38:02.874 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 25.443667980000001, 2020-06-12T02:38:02.874 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52428350999999995, 2020-06-12T02:38:02.878 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:02.878 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:02.879 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:02.879 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:02.879 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:02.879 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2953 2020-06-12T02:38:02.879 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.880 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:02.880 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.880 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:02.880 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167293+0000", 2020-06-12T02:38:02.881 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:02.881 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.881 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.881 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:02.881 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167293+0000", 2020-06-12T02:38:02.881 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3880000000000004e-05 2020-06-12T02:38:02.882 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.882 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.882 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:02.882 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167367+0000", 2020-06-12T02:38:02.882 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.319e-06 2020-06-12T02:38:02.883 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.883 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.883 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:02.883 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167373+0000", 2020-06-12T02:38:02.883 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1719000000000001e-05 2020-06-12T02:38:02.884 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.884 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.884 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:02.884 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167385+0000", 2020-06-12T02:38:02.884 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4580000000000003e-06 2020-06-12T02:38:02.884 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.885 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.885 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:02.885 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167389+0000", 2020-06-12T02:38:02.885 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036341110000000002 2020-06-12T02:38:02.885 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.886 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.886 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.886 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171023+0000", 2020-06-12T02:38:02.886 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9418000000000005e-05 2020-06-12T02:38:02.886 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.887 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.887 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:02.887 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171093+0000", 2020-06-12T02:38:02.887 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00029087900000000002 2020-06-12T02:38:02.887 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.887 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.888 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:02.888 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171384+0000", 2020-06-12T02:38:02.888 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000489347 2020-06-12T02:38:02.888 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.888 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.889 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:02.889 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171873+0000", 2020-06-12T02:38:02.889 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51963691300000003 2020-06-12T02:38:02.889 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.889 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.889 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:02.890 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691510+0000", 2020-06-12T02:38:02.890 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6465999999999997e-05 2020-06-12T02:38:02.890 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.890 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.890 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:02.891 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691576+0000", 2020-06-12T02:38:02.891 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52428350999999995 2020-06-12T02:38:02.891 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.891 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:02.891 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.891 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.892 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.892 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2954 3.5s0 3:a446366d:::smithi10729407-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:38:02.892 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.167432+0000", 2020-06-12T02:38:02.892 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 25.443529230999999, 2020-06-12T02:38:02.892 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52568652699999996, 2020-06-12T02:38:02.893 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:02.893 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:02.893 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:02.893 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:02.893 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:02.893 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2954 2020-06-12T02:38:02.894 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.894 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:02.894 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.895 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:02.895 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167432+0000", 2020-06-12T02:38:02.895 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:02.895 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.895 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.895 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:02.896 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167432+0000", 2020-06-12T02:38:02.896 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3556000000000001e-05 2020-06-12T02:38:02.896 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.896 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.896 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:02.897 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167475+0000", 2020-06-12T02:38:02.897 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7350000000000001e-06 2020-06-12T02:38:02.897 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.897 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.897 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:02.897 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167481+0000", 2020-06-12T02:38:02.898 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3128e-05 2020-06-12T02:38:02.898 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.898 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.898 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:02.898 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167494+0000", 2020-06-12T02:38:02.899 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1579999999999998e-06 2020-06-12T02:38:02.899 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.899 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.899 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:02.899 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167498+0000", 2020-06-12T02:38:02.899 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004082174 2020-06-12T02:38:02.900 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.900 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.900 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.900 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171580+0000", 2020-06-12T02:38:02.900 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8954000000000002e-05 2020-06-12T02:38:02.901 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.901 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.901 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:02.901 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171679+0000", 2020-06-12T02:38:02.901 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51993235100000001 2020-06-12T02:38:02.902 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.902 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.902 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.902 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691612+0000", 2020-06-12T02:38:02.902 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4329000000000001e-05 2020-06-12T02:38:02.902 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.903 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.903 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:02.903 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691666+0000", 2020-06-12T02:38:02.903 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001452142 2020-06-12T02:38:02.903 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.904 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.904 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:02.904 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.693118+0000", 2020-06-12T02:38:02.904 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52568652699999996 2020-06-12T02:38:02.904 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.905 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:02.905 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.905 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.905 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.905 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2959 3.ds1 3:b3445631:::smithi10729407-6:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e27)", 2020-06-12T02:38:02.906 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.176248+0000", 2020-06-12T02:38:02.906 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 25.434712489999999, 2020-06-12T02:38:02.906 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51549877099999997, 2020-06-12T02:38:02.906 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:02.906 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:02.907 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:02.907 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:02.907 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:02.907 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2959 2020-06-12T02:38:02.907 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.908 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:02.908 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.908 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:02.908 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176248+0000", 2020-06-12T02:38:02.908 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:02.908 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.909 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.909 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:02.909 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176248+0000", 2020-06-12T02:38:02.909 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5120999999999999e-05 2020-06-12T02:38:02.909 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.910 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.910 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:02.910 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176274+0000", 2020-06-12T02:38:02.910 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9189999999999998e-06 2020-06-12T02:38:02.910 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.910 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.911 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:02.911 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176278+0000", 2020-06-12T02:38:02.911 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6339e-05 2020-06-12T02:38:02.911 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.911 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.912 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:02.912 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176305+0000", 2020-06-12T02:38:02.912 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9089999999999999e-06 2020-06-12T02:38:02.912 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.912 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.912 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:02.913 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176310+0000", 2020-06-12T02:38:02.913 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6095999999999999e-05 2020-06-12T02:38:02.913 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.913 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.913 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.914 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176376+0000", 2020-06-12T02:38:02.914 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8590999999999994e-05 2020-06-12T02:38:02.914 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.914 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.914 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:02.914 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176444+0000", 2020-06-12T02:38:02.915 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00024346400000000001 2020-06-12T02:38:02.915 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.915 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.915 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:02.915 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176688+0000", 2020-06-12T02:38:02.916 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00044220899999999999 2020-06-12T02:38:02.916 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.916 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.916 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:02.916 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.177130+0000", 2020-06-12T02:38:02.916 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51452984400000001 2020-06-12T02:38:02.917 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.917 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.917 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:02.917 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691660+0000", 2020-06-12T02:38:02.917 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7279000000000002e-05 2020-06-12T02:38:02.918 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.918 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.918 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:02.918 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691747+0000", 2020-06-12T02:38:02.918 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51549877099999997 2020-06-12T02:38:02.918 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.919 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:02.919 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.919 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.919 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.919 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:3032 3.ds1 3:b983fd28:::smithi10729407-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:38:02.920 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.864289+0000", 2020-06-12T02:38:02.920 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 24.746671572, 2020-06-12T02:38:02.920 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019572216999999999, 2020-06-12T02:38:02.920 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:02.920 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:02.921 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:02.921 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:02.921 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:02.921 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 3032 2020-06-12T02:38:02.921 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.921 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:02.922 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.922 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:02.922 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864289+0000", 2020-06-12T02:38:02.922 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:02.922 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.923 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.923 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:02.923 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864289+0000", 2020-06-12T02:38:02.923 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5701999999999997e-05 2020-06-12T02:38:02.923 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.924 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.924 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:02.924 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864335+0000", 2020-06-12T02:38:02.924 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0910000000000004e-06 2020-06-12T02:38:02.924 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.924 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.925 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:02.925 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864342+0000", 2020-06-12T02:38:02.925 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2301e-05 2020-06-12T02:38:02.925 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.925 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.926 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:02.926 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864354+0000", 2020-06-12T02:38:02.926 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5199999999999997e-06 2020-06-12T02:38:02.926 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.926 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.926 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:02.927 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864360+0000", 2020-06-12T02:38:02.927 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00049180999999999997 2020-06-12T02:38:02.927 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.927 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.927 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.928 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864852+0000", 2020-06-12T02:38:02.928 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6725000000000003e-05 2020-06-12T02:38:02.928 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.928 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.928 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:02.928 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864908+0000", 2020-06-12T02:38:02.929 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0089687010000000008 2020-06-12T02:38:02.929 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.929 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.929 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.929 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.873877+0000", 2020-06-12T02:38:02.930 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3382999999999999e-05 2020-06-12T02:38:02.930 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.930 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.930 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:02.930 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.873941+0000", 2020-06-12T02:38:02.930 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.006408927 2020-06-12T02:38:02.931 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.931 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.931 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:02.931 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.880350+0000", 2020-06-12T02:38:02.931 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3905e-05 2020-06-12T02:38:02.932 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.932 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.933 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:02.933 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.880403+0000", 2020-06-12T02:38:02.933 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0034581519999999999 2020-06-12T02:38:02.933 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:02.933 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:02.934 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:02.934 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.883862+0000", 2020-06-12T02:38:02.934 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019572216999999999 2020-06-12T02:38:02.934 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.934 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:02.935 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.935 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:02.935 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:02.935 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:02.936 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-06-12T02:38:02.937 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:02.712+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:02.938 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:02.935+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:03.029 INFO:teuthology.orchestra.run.smithi107.stderr:2020-06-12T02:38:03.027+0000 7fe0a37fe700 0 --2- 172.21.15.107:0/3216520242 >> v2:172.21.15.107:3300/0 conn(0x7fe0a409bde0 0x7fe0a40a1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2020-06-12T02:38:03.038 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:03.035+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:03.053 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:03.054 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:38:03.054 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2020-06-12T02:38:03.054 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:03.062 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-06-12T02:38:03.147 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:03.144+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:03.226 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:03.226 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:38:03.226 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:38:03.227 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:38:03.227 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:03.228 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-06-12T02:38:03.239 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:03.237+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:03.340 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:03.338+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:03.385 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:03.386 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:38:03.386 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:38:03.386 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:38:03.386 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.387 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1558 3.as0 3:5cb723d2:::smithi10729407-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e26)", 2020-06-12T02:38:03.387 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.839131+0000", 2020-06-12T02:38:03.387 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 29.54320117, 2020-06-12T02:38:03.387 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013057691, 2020-06-12T02:38:03.387 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:03.387 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:03.388 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:03.388 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:03.388 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:03.388 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1558 2020-06-12T02:38:03.388 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.389 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:03.389 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.389 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:03.389 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.839131+0000", 2020-06-12T02:38:03.389 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:03.390 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.390 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.390 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:03.390 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.839131+0000", 2020-06-12T02:38:03.390 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2228999999999998e-05 2020-06-12T02:38:03.390 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.391 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.391 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:03.391 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.839173+0000", 2020-06-12T02:38:03.391 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9789999999999998e-06 2020-06-12T02:38:03.392 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.392 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.392 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:03.393 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.839179+0000", 2020-06-12T02:38:03.393 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3011e-05 2020-06-12T02:38:03.393 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.393 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.393 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:03.394 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.839192+0000", 2020-06-12T02:38:03.394 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7929999999999999e-06 2020-06-12T02:38:03.394 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.394 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.394 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:03.395 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.839196+0000", 2020-06-12T02:38:03.395 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0083729379999999999 2020-06-12T02:38:03.395 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.395 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.395 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:03.396 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.847569+0000", 2020-06-12T02:38:03.396 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5887999999999998e-05 2020-06-12T02:38:03.396 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.396 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.396 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:03.397 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.847625+0000", 2020-06-12T02:38:03.397 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0027582330000000001 2020-06-12T02:38:03.397 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.397 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.397 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:03.397 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.850383+0000", 2020-06-12T02:38:03.398 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00037230500000000002 2020-06-12T02:38:03.398 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.398 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.398 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:03.398 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.850755+0000", 2020-06-12T02:38:03.399 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0014333149999999999 2020-06-12T02:38:03.399 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.399 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.399 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:03.399 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.852189+0000", 2020-06-12T02:38:03.399 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013057691 2020-06-12T02:38:03.400 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:03.400 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:03.400 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:03.400 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.400 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.401 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2746 3.as0 3:5cb723d2:::smithi10729407-15:head [append 0~524288] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:03.401 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:36.723084+0000", 2020-06-12T02:38:03.401 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 26.659248374000001, 2020-06-12T02:38:03.401 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013015831, 2020-06-12T02:38:03.401 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:03.402 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:03.402 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:03.402 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:03.402 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:03.402 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2746 2020-06-12T02:38:03.402 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.403 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:03.403 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.403 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:03.403 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.723084+0000", 2020-06-12T02:38:03.403 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:03.404 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.404 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.404 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:03.404 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.723084+0000", 2020-06-12T02:38:03.404 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0012755659999999999 2020-06-12T02:38:03.405 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.405 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.405 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:03.405 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724359+0000", 2020-06-12T02:38:03.405 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2599999999999996e-06 2020-06-12T02:38:03.406 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.406 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.406 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:03.406 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724365+0000", 2020-06-12T02:38:03.406 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5499999999999995e-06 2020-06-12T02:38:03.407 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.407 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.407 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:03.407 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724373+0000", 2020-06-12T02:38:03.408 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3590000000000001e-06 2020-06-12T02:38:03.408 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.408 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.408 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:03.408 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724378+0000", 2020-06-12T02:38:03.408 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0034266629999999999 2020-06-12T02:38:03.409 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.409 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.409 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:03.409 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.727804+0000", 2020-06-12T02:38:03.409 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6347000000000005e-05 2020-06-12T02:38:03.410 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.410 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.410 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:03.410 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.727891+0000", 2020-06-12T02:38:03.410 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054319199999999996 2020-06-12T02:38:03.411 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.411 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.411 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:03.411 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.728434+0000", 2020-06-12T02:38:03.411 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0029938999999999999 2020-06-12T02:38:03.411 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.412 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.412 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:03.412 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.731428+0000", 2020-06-12T02:38:03.412 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0046291250000000004 2020-06-12T02:38:03.412 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.412 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.413 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:03.413 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.736057+0000", 2020-06-12T02:38:03.413 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2868999999999999e-05 2020-06-12T02:38:03.413 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.413 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.414 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:03.414 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.736100+0000", 2020-06-12T02:38:03.414 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013015831 2020-06-12T02:38:03.414 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:03.414 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:03.415 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:03.415 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.415 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.415 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2747 3.as0 3:5cb723d2:::smithi10729407-15:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:03.416 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:36.724407+0000", 2020-06-12T02:38:03.416 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 26.657925207000002, 2020-06-12T02:38:03.417 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014982544, 2020-06-12T02:38:03.417 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:03.417 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:03.417 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:03.417 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:03.418 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:03.418 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2747 2020-06-12T02:38:03.418 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.418 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:03.418 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.418 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:03.419 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724407+0000", 2020-06-12T02:38:03.419 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:03.419 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.419 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.419 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:03.420 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724407+0000", 2020-06-12T02:38:03.420 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6766e-05 2020-06-12T02:38:03.420 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.420 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.420 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:03.420 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724444+0000", 2020-06-12T02:38:03.421 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.315e-06 2020-06-12T02:38:03.421 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.421 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.421 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:03.421 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724447+0000", 2020-06-12T02:38:03.422 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9100000000000002e-06 2020-06-12T02:38:03.422 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.422 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.422 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:03.422 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724453+0000", 2020-06-12T02:38:03.422 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.43e-06 2020-06-12T02:38:03.423 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.423 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.423 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:03.423 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724455+0000", 2020-06-12T02:38:03.423 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0061404010000000002 2020-06-12T02:38:03.424 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.424 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.424 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:03.424 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.730596+0000", 2020-06-12T02:38:03.424 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8402e-05 2020-06-12T02:38:03.425 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.425 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.425 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:03.425 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.730664+0000", 2020-06-12T02:38:03.426 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00031661500000000002 2020-06-12T02:38:03.426 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.426 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.426 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:03.426 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.730981+0000", 2020-06-12T02:38:03.426 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0049794890000000001 2020-06-12T02:38:03.427 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.427 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.427 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:03.427 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.735960+0000", 2020-06-12T02:38:03.427 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033245739999999998 2020-06-12T02:38:03.428 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.428 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.428 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:03.428 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.739285+0000", 2020-06-12T02:38:03.428 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010464200000000001 2020-06-12T02:38:03.428 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.429 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.429 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:03.429 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.739390+0000", 2020-06-12T02:38:03.429 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014982544 2020-06-12T02:38:03.429 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:03.430 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:03.430 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:03.430 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.430 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.430 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2748 3.as0 3:5cb723d2:::smithi10729407-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:38:03.430 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:36.724477+0000", 2020-06-12T02:38:03.431 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 26.657855068, 2020-06-12T02:38:03.431 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020852102000000001, 2020-06-12T02:38:03.431 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:03.431 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:03.432 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:03.432 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:03.432 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:03.432 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2748 2020-06-12T02:38:03.432 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.432 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:03.433 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.433 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:03.433 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724477+0000", 2020-06-12T02:38:03.433 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:03.434 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.434 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.434 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:03.434 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724477+0000", 2020-06-12T02:38:03.434 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2065e-05 2020-06-12T02:38:03.434 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.436 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.437 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:03.437 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724499+0000", 2020-06-12T02:38:03.437 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9340000000000002e-06 2020-06-12T02:38:03.437 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.438 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.438 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:03.438 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724502+0000", 2020-06-12T02:38:03.438 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.246e-06 2020-06-12T02:38:03.439 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.439 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.439 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:03.439 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724508+0000", 2020-06-12T02:38:03.439 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2900000000000001e-06 2020-06-12T02:38:03.439 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.440 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.440 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:03.440 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724511+0000", 2020-06-12T02:38:03.440 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0068556700000000003 2020-06-12T02:38:03.440 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.441 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.441 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:03.441 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.731366+0000", 2020-06-12T02:38:03.441 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4738999999999999e-05 2020-06-12T02:38:03.441 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.442 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.442 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:03.442 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.731421+0000", 2020-06-12T02:38:03.442 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0080153150000000003 2020-06-12T02:38:03.442 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.443 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.443 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:03.443 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.739436+0000", 2020-06-12T02:38:03.443 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010418 2020-06-12T02:38:03.443 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.443 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.444 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:03.444 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.739541+0000", 2020-06-12T02:38:03.444 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0057886630000000003 2020-06-12T02:38:03.444 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.445 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.445 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:03.445 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.745329+0000", 2020-06-12T02:38:03.445 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020852102000000001 2020-06-12T02:38:03.445 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:03.446 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:03.446 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:03.446 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.446 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.446 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:3055 3.as0 3:5cb723d2:::smithi10729407-15:head [append 0~614400] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:03.447 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.918209+0000", 2020-06-12T02:38:03.447 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 25.464123353000002, 2020-06-12T02:38:03.447 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014165394, 2020-06-12T02:38:03.447 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:03.448 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:03.448 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:03.448 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:03.448 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:03.448 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 3055 2020-06-12T02:38:03.448 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.449 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:03.449 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.449 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:03.449 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.918209+0000", 2020-06-12T02:38:03.449 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:03.450 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.450 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.450 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:03.450 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.918209+0000", 2020-06-12T02:38:03.450 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00081606199999999999 2020-06-12T02:38:03.451 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.451 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.451 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:03.451 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.919025+0000", 2020-06-12T02:38:03.451 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.251e-06 2020-06-12T02:38:03.451 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.452 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.452 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:03.452 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.919031+0000", 2020-06-12T02:38:03.452 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1460999999999999e-05 2020-06-12T02:38:03.452 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.453 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.453 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:03.453 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.919043+0000", 2020-06-12T02:38:03.453 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.592e-06 2020-06-12T02:38:03.453 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.453 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.454 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:03.454 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.919048+0000", 2020-06-12T02:38:03.454 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0020427959999999999 2020-06-12T02:38:03.454 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.454 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.455 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:03.455 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.921091+0000", 2020-06-12T02:38:03.455 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012382800000000001 2020-06-12T02:38:03.455 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.455 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.455 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:03.456 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.921215+0000", 2020-06-12T02:38:03.456 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053208300000000004 2020-06-12T02:38:03.456 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.456 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.456 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:03.457 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.921747+0000", 2020-06-12T02:38:03.457 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0075648599999999996 2020-06-12T02:38:03.457 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.457 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.458 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:03.458 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.929312+0000", 2020-06-12T02:38:03.458 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0029879630000000002 2020-06-12T02:38:03.458 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.458 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.459 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:03.459 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.932300+0000", 2020-06-12T02:38:03.459 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4498000000000004e-05 2020-06-12T02:38:03.459 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.459 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.459 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:03.460 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.932374+0000", 2020-06-12T02:38:03.460 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014165394 2020-06-12T02:38:03.460 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:03.460 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:03.460 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:03.461 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.461 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.461 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:3217 3.as0 3:5732055a:::smithi10729407-8:head [append 0~147456] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:03.461 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:38.300193+0000", 2020-06-12T02:38:03.461 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 25.082139244, 2020-06-12T02:38:03.461 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012845314, 2020-06-12T02:38:03.462 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:03.462 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:03.462 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:03.462 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:03.462 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:03.463 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 3217 2020-06-12T02:38:03.463 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.463 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:03.463 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.465 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:03.465 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300193+0000", 2020-06-12T02:38:03.466 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:03.466 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.466 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.466 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:03.466 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300193+0000", 2020-06-12T02:38:03.467 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017722699999999999 2020-06-12T02:38:03.467 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.467 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.467 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:03.468 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300370+0000", 2020-06-12T02:38:03.468 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4009999999999996e-06 2020-06-12T02:38:03.468 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.468 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.468 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:03.468 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300376+0000", 2020-06-12T02:38:03.469 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9109999999999993e-06 2020-06-12T02:38:03.469 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.469 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.469 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:03.469 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300384+0000", 2020-06-12T02:38:03.470 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2740000000000001e-06 2020-06-12T02:38:03.470 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.470 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.470 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:03.470 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300388+0000", 2020-06-12T02:38:03.471 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3261999999999999e-05 2020-06-12T02:38:03.471 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.471 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.471 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:03.472 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300411+0000", 2020-06-12T02:38:03.472 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9906e-05 2020-06-12T02:38:03.472 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.472 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.472 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:03.473 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300461+0000", 2020-06-12T02:38:03.473 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0059439150000000001 2020-06-12T02:38:03.473 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.473 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.473 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:03.473 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.306405+0000", 2020-06-12T02:38:03.474 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7945000000000002e-05 2020-06-12T02:38:03.474 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.474 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.474 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:03.474 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.306463+0000", 2020-06-12T02:38:03.475 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00042195500000000001 2020-06-12T02:38:03.475 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.475 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.475 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:03.475 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.306885+0000", 2020-06-12T02:38:03.475 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0032273800000000002 2020-06-12T02:38:03.476 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.476 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.476 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:03.476 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.310112+0000", 2020-06-12T02:38:03.476 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0028430420000000001 2020-06-12T02:38:03.477 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.477 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.477 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:03.477 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.312955+0000", 2020-06-12T02:38:03.478 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3096000000000005e-05 2020-06-12T02:38:03.478 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.478 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.479 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:03.479 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.313038+0000", 2020-06-12T02:38:03.479 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012845314 2020-06-12T02:38:03.479 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:03.479 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:03.480 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:03.480 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.480 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.480 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:3218 3.as0 3:5732055a:::smithi10729407-8:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:03.480 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:38.300417+0000", 2020-06-12T02:38:03.481 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 25.081914727000001, 2020-06-12T02:38:03.481 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012861745000000001, 2020-06-12T02:38:03.481 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:03.481 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:03.481 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:03.482 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:03.482 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:03.482 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 3218 2020-06-12T02:38:03.482 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.482 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:03.482 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.483 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:03.483 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300417+0000", 2020-06-12T02:38:03.483 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:03.483 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.483 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.484 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:03.484 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300417+0000", 2020-06-12T02:38:03.484 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9110999999999999e-05 2020-06-12T02:38:03.484 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.484 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.485 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:03.485 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300457+0000", 2020-06-12T02:38:03.485 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3289999999999998e-06 2020-06-12T02:38:03.485 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.485 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.485 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:03.487 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300460+0000", 2020-06-12T02:38:03.490 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8569999999999997e-06 2020-06-12T02:38:03.491 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.491 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.491 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:03.491 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300467+0000", 2020-06-12T02:38:03.491 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7080000000000002e-06 2020-06-12T02:38:03.492 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.492 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.492 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:03.492 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300469+0000", 2020-06-12T02:38:03.492 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0064999999999998e-05 2020-06-12T02:38:03.492 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.493 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.493 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:03.493 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300490+0000", 2020-06-12T02:38:03.493 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0943999999999999e-05 2020-06-12T02:38:03.493 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.494 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.494 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:03.494 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300540+0000", 2020-06-12T02:38:03.494 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0073891770000000002 2020-06-12T02:38:03.494 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.495 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.495 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:03.495 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.307930+0000", 2020-06-12T02:38:03.495 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010105199999999999 2020-06-12T02:38:03.495 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.495 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.496 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:03.496 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.308031+0000", 2020-06-12T02:38:03.496 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00043489400000000001 2020-06-12T02:38:03.496 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.496 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.497 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:03.497 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.308466+0000", 2020-06-12T02:38:03.497 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0018799070000000001 2020-06-12T02:38:03.497 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.497 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.498 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:03.498 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.310346+0000", 2020-06-12T02:38:03.498 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0028380839999999998 2020-06-12T02:38:03.498 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.498 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.498 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:03.499 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.313184+0000", 2020-06-12T02:38:03.499 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5617000000000003e-05 2020-06-12T02:38:03.499 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.499 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.499 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:03.500 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.313279+0000", 2020-06-12T02:38:03.501 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012861745000000001 2020-06-12T02:38:03.501 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:03.501 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:03.501 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:03.501 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.502 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.502 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:3219 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:38:03.502 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:38.300493+0000", 2020-06-12T02:38:03.502 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 25.081839361, 2020-06-12T02:38:03.502 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018386370999999999, 2020-06-12T02:38:03.503 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:03.503 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:03.503 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:03.503 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:03.503 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:03.504 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 3219 2020-06-12T02:38:03.504 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.504 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:03.504 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.504 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:03.506 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300493+0000", 2020-06-12T02:38:03.506 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:03.506 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.506 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.507 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:03.507 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300493+0000", 2020-06-12T02:38:03.507 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3008000000000001e-05 2020-06-12T02:38:03.507 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.507 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.508 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:03.508 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300516+0000", 2020-06-12T02:38:03.508 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1329999999999999e-06 2020-06-12T02:38:03.508 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.508 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.509 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:03.509 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300519+0000", 2020-06-12T02:38:03.509 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8260000000000004e-06 2020-06-12T02:38:03.509 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.509 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.510 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:03.510 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300526+0000", 2020-06-12T02:38:03.510 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.525e-06 2020-06-12T02:38:03.510 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.510 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.510 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:03.511 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300528+0000", 2020-06-12T02:38:03.511 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9839e-05 2020-06-12T02:38:03.511 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.511 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.511 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:03.512 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300558+0000", 2020-06-12T02:38:03.512 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3198000000000001e-05 2020-06-12T02:38:03.512 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.512 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.512 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:03.513 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300611+0000", 2020-06-12T02:38:03.513 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0082281509999999995 2020-06-12T02:38:03.513 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.513 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.513 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:03.513 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.308840+0000", 2020-06-12T02:38:03.514 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5921000000000005e-05 2020-06-12T02:38:03.514 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.514 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.514 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:03.514 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.308935+0000", 2020-06-12T02:38:03.515 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050007560000000003 2020-06-12T02:38:03.515 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.515 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.515 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:03.515 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.313936+0000", 2020-06-12T02:38:03.516 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1600000000000007e-05 2020-06-12T02:38:03.516 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.516 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.516 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:03.516 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.313998+0000", 2020-06-12T02:38:03.516 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0048814139999999997 2020-06-12T02:38:03.517 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.517 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.517 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:03.517 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.318879+0000", 2020-06-12T02:38:03.517 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018386370999999999 2020-06-12T02:38:03.518 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:03.518 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:03.518 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:03.518 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.518 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.518 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:4478 3.as0 3:5cb723d2:::smithi10729407-15:head [append 0~483328] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-06-12T02:38:03.519 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:42.522285+0000", 2020-06-12T02:38:03.519 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.860047397999999, 2020-06-12T02:38:03.519 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013258748000000001, 2020-06-12T02:38:03.519 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:03.519 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:03.520 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:03.520 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:03.520 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:03.520 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 4478 2020-06-12T02:38:03.520 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.521 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:03.521 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.521 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:03.521 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.522285+0000", 2020-06-12T02:38:03.521 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:03.521 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.522 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.522 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:03.522 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.522285+0000", 2020-06-12T02:38:03.522 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00055732500000000001 2020-06-12T02:38:03.522 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.523 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.523 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:03.523 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.522842+0000", 2020-06-12T02:38:03.523 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1270000000000002e-06 2020-06-12T02:38:03.523 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.523 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.524 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:03.524 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.522847+0000", 2020-06-12T02:38:03.524 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1314e-05 2020-06-12T02:38:03.524 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.524 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.525 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:03.525 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.522859+0000", 2020-06-12T02:38:03.525 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4339999999999997e-06 2020-06-12T02:38:03.525 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.525 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.525 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:03.526 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.522865+0000", 2020-06-12T02:38:03.526 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033831099999999999 2020-06-12T02:38:03.526 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.526 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.526 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:03.527 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.526248+0000", 2020-06-12T02:38:03.527 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010359199999999999 2020-06-12T02:38:03.527 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.527 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.527 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:03.527 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.526352+0000", 2020-06-12T02:38:03.528 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031925009999999999 2020-06-12T02:38:03.528 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.528 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.529 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:03.529 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.529544+0000", 2020-06-12T02:38:03.529 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6626000000000003e-05 2020-06-12T02:38:03.529 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.529 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.530 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:03.530 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.529601+0000", 2020-06-12T02:38:03.530 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047735099999999999 2020-06-12T02:38:03.530 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.530 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.531 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:03.531 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.530078+0000", 2020-06-12T02:38:03.531 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0053046090000000001 2020-06-12T02:38:03.531 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.531 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.531 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:03.532 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.535383+0000", 2020-06-12T02:38:03.532 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011821599999999999 2020-06-12T02:38:03.532 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.532 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.532 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:03.533 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.535501+0000", 2020-06-12T02:38:03.533 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2543000000000002e-05 2020-06-12T02:38:03.533 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.533 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.533 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:03.533 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.535544+0000", 2020-06-12T02:38:03.534 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013258748000000001 2020-06-12T02:38:03.534 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:03.534 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:03.534 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:03.534 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.535 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.535 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:4535 3.as0 3:5732055a:::smithi10729407-8:head [append 0~712704] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-06-12T02:38:03.535 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:42.670365+0000", 2020-06-12T02:38:03.535 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.711967067, 2020-06-12T02:38:03.535 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013998371000000001, 2020-06-12T02:38:03.536 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:03.536 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:03.536 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:03.536 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:03.536 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:03.536 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 4535 2020-06-12T02:38:03.537 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.537 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:03.537 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.537 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:03.537 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.670365+0000", 2020-06-12T02:38:03.538 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:03.538 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.538 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.538 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:03.538 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.670365+0000", 2020-06-12T02:38:03.538 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00059204600000000004 2020-06-12T02:38:03.539 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.539 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.539 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:03.539 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.670957+0000", 2020-06-12T02:38:03.540 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.122e-06 2020-06-12T02:38:03.540 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.540 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.541 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:03.541 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.670963+0000", 2020-06-12T02:38:03.541 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1518999999999999e-05 2020-06-12T02:38:03.541 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.541 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.542 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:03.542 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.670975+0000", 2020-06-12T02:38:03.542 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9290000000000001e-06 2020-06-12T02:38:03.542 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.542 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.543 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:03.543 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.670982+0000", 2020-06-12T02:38:03.543 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4758000000000001e-05 2020-06-12T02:38:03.543 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.543 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.544 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:03.544 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.671027+0000", 2020-06-12T02:38:03.544 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9733999999999993e-05 2020-06-12T02:38:03.544 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.544 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.545 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:03.545 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.671096+0000", 2020-06-12T02:38:03.545 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000544448 2020-06-12T02:38:03.545 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.546 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.546 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:03.546 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.671641+0000", 2020-06-12T02:38:03.546 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012592534000000001 2020-06-12T02:38:03.546 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.547 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.547 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:03.547 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.684233+0000", 2020-06-12T02:38:03.547 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9785999999999998e-05 2020-06-12T02:38:03.547 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.547 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.548 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:03.548 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.684313+0000", 2020-06-12T02:38:03.548 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0494999999999997e-05 2020-06-12T02:38:03.548 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.548 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.549 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:03.549 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.684364+0000", 2020-06-12T02:38:03.549 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013998371000000001 2020-06-12T02:38:03.549 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:03.549 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:03.550 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:03.550 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.550 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.550 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:4975 3.as0 3:5732055a:::smithi10729407-8:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e30)", 2020-06-12T02:38:03.550 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:43.721794+0000", 2020-06-12T02:38:03.550 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 19.660538574, 2020-06-12T02:38:03.551 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.1825135, 2020-06-12T02:38:03.551 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:03.551 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:03.551 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:03.551 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:03.552 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:03.552 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 4975 2020-06-12T02:38:03.552 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.552 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:03.552 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.552 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:03.553 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721794+0000", 2020-06-12T02:38:03.553 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:03.553 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.553 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.553 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:03.554 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721794+0000", 2020-06-12T02:38:03.554 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4542e-05 2020-06-12T02:38:03.554 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.554 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.554 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:03.555 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721818+0000", 2020-06-12T02:38:03.555 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9169999999999997e-06 2020-06-12T02:38:03.555 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.555 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.556 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:03.556 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721823+0000", 2020-06-12T02:38:03.556 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1184e-05 2020-06-12T02:38:03.556 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.556 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.556 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:03.557 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721834+0000", 2020-06-12T02:38:03.557 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.254e-06 2020-06-12T02:38:03.557 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.557 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.557 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:03.558 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721840+0000", 2020-06-12T02:38:03.558 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1075999999999998e-05 2020-06-12T02:38:03.558 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.558 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.558 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:03.559 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721881+0000", 2020-06-12T02:38:03.559 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7416999999999995e-05 2020-06-12T02:38:03.559 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.559 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.559 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:03.559 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721948+0000", 2020-06-12T02:38:03.560 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.18235910999999999 2020-06-12T02:38:03.560 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.560 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.560 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:03.560 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.904307+0000", 2020-06-12T02:38:03.561 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.1825135 2020-06-12T02:38:03.561 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:03.561 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:03.561 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:03.562 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.562 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.562 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:6205 3.as0 3:5cb723d2:::smithi10729407-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e31)", 2020-06-12T02:38:03.562 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:47.353583+0000", 2020-06-12T02:38:03.563 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 16.028749363999999, 2020-06-12T02:38:03.563 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013040737, 2020-06-12T02:38:03.563 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:03.564 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:03.564 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:03.564 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:03.565 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:03.565 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 6205 2020-06-12T02:38:03.566 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.566 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:03.567 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.567 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:03.567 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.353583+0000", 2020-06-12T02:38:03.567 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:03.567 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.567 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.568 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:03.568 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.353583+0000", 2020-06-12T02:38:03.568 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3133000000000002e-05 2020-06-12T02:38:03.568 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.569 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.569 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:03.569 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.353626+0000", 2020-06-12T02:38:03.569 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9570000000000008e-06 2020-06-12T02:38:03.569 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.570 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.570 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:03.570 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.353635+0000", 2020-06-12T02:38:03.570 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1754e-05 2020-06-12T02:38:03.570 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.571 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.571 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:03.571 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.353647+0000", 2020-06-12T02:38:03.571 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0809999999999998e-06 2020-06-12T02:38:03.571 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.572 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.572 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:03.572 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.353650+0000", 2020-06-12T02:38:03.572 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0064807490000000001 2020-06-12T02:38:03.572 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.572 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.573 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:03.573 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.360131+0000", 2020-06-12T02:38:03.573 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7163000000000001e-05 2020-06-12T02:38:03.573 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.573 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.574 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:03.574 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.360188+0000", 2020-06-12T02:38:03.574 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0053632230000000003 2020-06-12T02:38:03.574 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.574 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.575 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:03.575 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.365551+0000", 2020-06-12T02:38:03.575 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8078999999999997e-05 2020-06-12T02:38:03.575 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.575 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.575 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:03.576 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.365609+0000", 2020-06-12T02:38:03.576 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001014598 2020-06-12T02:38:03.576 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.576 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.577 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:03.578 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.366624+0000", 2020-06-12T02:38:03.578 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013040737 2020-06-12T02:38:03.578 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:03.578 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:03.578 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:03.579 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.579 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.579 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:6678 3.as0 3:5732055a:::smithi10729407-8:head [append 0~499712] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-06-12T02:38:03.579 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:48.626196+0000", 2020-06-12T02:38:03.579 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 14.756136591000001, 2020-06-12T02:38:03.580 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014093700000000001, 2020-06-12T02:38:03.580 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:03.580 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:03.580 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:03.580 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:03.581 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:03.581 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 6678 2020-06-12T02:38:03.581 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.581 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:03.581 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.581 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:03.582 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.626196+0000", 2020-06-12T02:38:03.582 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:03.582 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.582 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.582 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:03.583 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.626196+0000", 2020-06-12T02:38:03.583 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00046254199999999998 2020-06-12T02:38:03.583 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.583 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.583 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:03.583 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.626658+0000", 2020-06-12T02:38:03.584 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3400000000000003e-06 2020-06-12T02:38:03.584 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.584 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.584 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:03.584 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.626665+0000", 2020-06-12T02:38:03.585 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0079e-05 2020-06-12T02:38:03.585 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.585 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.585 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:03.585 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.626675+0000", 2020-06-12T02:38:03.586 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0999999999999997e-06 2020-06-12T02:38:03.586 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.586 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.586 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:03.586 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.626679+0000", 2020-06-12T02:38:03.586 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025692100000000002 2020-06-12T02:38:03.587 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.587 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.587 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:03.587 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.629248+0000", 2020-06-12T02:38:03.587 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000107944 2020-06-12T02:38:03.588 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.588 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.588 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:03.588 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.629356+0000", 2020-06-12T02:38:03.589 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001434999 2020-06-12T02:38:03.589 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.589 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.589 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:03.589 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.630791+0000", 2020-06-12T02:38:03.590 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0716e-05 2020-06-12T02:38:03.590 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.590 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.590 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:03.590 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.630842+0000", 2020-06-12T02:38:03.590 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00045107600000000002 2020-06-12T02:38:03.591 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.591 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.591 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:03.591 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.631293+0000", 2020-06-12T02:38:03.591 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0048208330000000001 2020-06-12T02:38:03.592 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.592 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.592 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:03.592 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.636113+0000", 2020-06-12T02:38:03.592 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0041129679999999998 2020-06-12T02:38:03.593 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.593 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.593 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:03.593 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.640226+0000", 2020-06-12T02:38:03.593 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2892999999999996e-05 2020-06-12T02:38:03.593 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.594 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.594 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:03.594 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.640289+0000", 2020-06-12T02:38:03.594 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014093700000000001 2020-06-12T02:38:03.594 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:03.595 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:03.595 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:03.595 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.595 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.595 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:7084 3.as0 3:5732055a:::smithi10729407-8:head [append 0~401408] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-06-12T02:38:03.596 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:49.666656+0000", 2020-06-12T02:38:03.596 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 13.715675847, 2020-06-12T02:38:03.596 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013133027, 2020-06-12T02:38:03.596 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:03.596 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:03.596 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:03.597 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:03.597 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:03.597 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 7084 2020-06-12T02:38:03.597 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.597 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:03.598 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.598 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:03.598 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.666656+0000", 2020-06-12T02:38:03.598 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:03.598 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.598 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.599 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:03.599 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.666656+0000", 2020-06-12T02:38:03.599 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00040755700000000001 2020-06-12T02:38:03.599 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.599 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.600 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:03.600 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667064+0000", 2020-06-12T02:38:03.600 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9479999999999999e-06 2020-06-12T02:38:03.600 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.601 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.601 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:03.601 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667069+0000", 2020-06-12T02:38:03.601 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.568e-06 2020-06-12T02:38:03.601 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.602 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.602 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:03.602 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667077+0000", 2020-06-12T02:38:03.602 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4170000000000002e-06 2020-06-12T02:38:03.602 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.603 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.603 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:03.603 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667082+0000", 2020-06-12T02:38:03.603 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0034817770000000001 2020-06-12T02:38:03.603 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.604 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.604 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:03.604 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.670564+0000", 2020-06-12T02:38:03.604 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011071200000000001 2020-06-12T02:38:03.604 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.605 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.605 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:03.605 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.670674+0000", 2020-06-12T02:38:03.605 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0011185780000000001 2020-06-12T02:38:03.605 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.606 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.606 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:03.606 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.671793+0000", 2020-06-12T02:38:03.606 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2209e-05 2020-06-12T02:38:03.606 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.607 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.607 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:03.607 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.671855+0000", 2020-06-12T02:38:03.607 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054805100000000003 2020-06-12T02:38:03.607 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.608 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.608 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:03.608 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.672403+0000", 2020-06-12T02:38:03.608 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0044889480000000004 2020-06-12T02:38:03.608 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.608 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.609 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:03.609 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.676892+0000", 2020-06-12T02:38:03.609 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0028428730000000001 2020-06-12T02:38:03.609 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.609 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.610 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:03.610 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.679735+0000", 2020-06-12T02:38:03.610 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4388999999999997e-05 2020-06-12T02:38:03.610 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.610 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.610 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:03.611 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.679789+0000", 2020-06-12T02:38:03.611 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013133027 2020-06-12T02:38:03.611 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:03.611 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:03.611 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:03.612 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.612 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.612 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:7085 3.as0 3:5732055a:::smithi10729407-8:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-06-12T02:38:03.612 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:49.667120+0000", 2020-06-12T02:38:03.613 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 13.715212613, 2020-06-12T02:38:03.613 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012883739, 2020-06-12T02:38:03.613 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:03.613 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:03.613 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:03.614 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:03.614 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:03.614 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 7085 2020-06-12T02:38:03.614 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.615 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:03.615 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.615 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:03.615 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667120+0000", 2020-06-12T02:38:03.615 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:03.616 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.616 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.616 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:03.616 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667120+0000", 2020-06-12T02:38:03.616 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.773e-05 2020-06-12T02:38:03.616 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.617 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.617 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:03.617 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667157+0000", 2020-06-12T02:38:03.617 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7740000000000002e-06 2020-06-12T02:38:03.617 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.618 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.618 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:03.618 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667161+0000", 2020-06-12T02:38:03.618 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1190000000000002e-06 2020-06-12T02:38:03.618 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.618 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.619 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:03.619 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667167+0000", 2020-06-12T02:38:03.619 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6970000000000002e-06 2020-06-12T02:38:03.619 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.619 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.620 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:03.620 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667170+0000", 2020-06-12T02:38:03.620 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036418029999999999 2020-06-12T02:38:03.620 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.620 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.620 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:03.621 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.670812+0000", 2020-06-12T02:38:03.621 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000107107 2020-06-12T02:38:03.621 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.621 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.621 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:03.622 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.670919+0000", 2020-06-12T02:38:03.622 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0034625480000000002 2020-06-12T02:38:03.622 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.622 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.622 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:03.623 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.674381+0000", 2020-06-12T02:38:03.623 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010428 2020-06-12T02:38:03.623 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.623 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.623 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:03.624 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.674486+0000", 2020-06-12T02:38:03.624 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000426382 2020-06-12T02:38:03.624 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.624 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.624 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:03.625 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.674912+0000", 2020-06-12T02:38:03.625 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0044864739999999998 2020-06-12T02:38:03.625 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.625 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.626 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:03.626 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.679399+0000", 2020-06-12T02:38:03.626 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00051632499999999999 2020-06-12T02:38:03.626 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.626 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.626 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:03.627 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.679915+0000", 2020-06-12T02:38:03.627 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8499999999999996e-05 2020-06-12T02:38:03.627 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.627 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.627 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:03.628 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.680003+0000", 2020-06-12T02:38:03.628 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012883739 2020-06-12T02:38:03.628 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:03.628 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:03.628 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:03.628 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.629 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.629 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:7086 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e32)", 2020-06-12T02:38:03.629 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:49.667192+0000", 2020-06-12T02:38:03.629 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 13.715140262, 2020-06-12T02:38:03.629 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014618023000000001, 2020-06-12T02:38:03.630 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:03.630 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:03.630 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:03.630 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:03.630 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:03.631 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 7086 2020-06-12T02:38:03.631 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.631 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:03.631 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.631 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:03.631 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667192+0000", 2020-06-12T02:38:03.632 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:03.632 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.632 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.632 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:03.632 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667192+0000", 2020-06-12T02:38:03.633 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1912e-05 2020-06-12T02:38:03.633 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.633 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.633 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:03.633 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667214+0000", 2020-06-12T02:38:03.634 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0469999999999998e-06 2020-06-12T02:38:03.634 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.634 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.634 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:03.634 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667217+0000", 2020-06-12T02:38:03.635 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7170000000000002e-06 2020-06-12T02:38:03.635 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.635 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.636 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:03.636 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667224+0000", 2020-06-12T02:38:03.636 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2850000000000001e-06 2020-06-12T02:38:03.636 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.636 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.637 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:03.637 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667226+0000", 2020-06-12T02:38:03.637 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0037197950000000001 2020-06-12T02:38:03.637 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.637 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.638 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:03.638 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.670946+0000", 2020-06-12T02:38:03.638 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4549999999999994e-05 2020-06-12T02:38:03.638 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.638 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.639 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:03.639 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.671040+0000", 2020-06-12T02:38:03.639 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0043129199999999996 2020-06-12T02:38:03.639 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.639 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.640 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:03.640 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.675353+0000", 2020-06-12T02:38:03.640 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1982000000000001e-05 2020-06-12T02:38:03.640 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.640 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.641 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:03.641 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.675445+0000", 2020-06-12T02:38:03.641 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.005067401 2020-06-12T02:38:03.641 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.641 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.641 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:03.642 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.680513+0000", 2020-06-12T02:38:03.642 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000101508 2020-06-12T02:38:03.642 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.642 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.642 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:03.643 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.680614+0000", 2020-06-12T02:38:03.643 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0011959060000000001 2020-06-12T02:38:03.643 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.643 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.643 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:03.643 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.681810+0000", 2020-06-12T02:38:03.644 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014618023000000001 2020-06-12T02:38:03.644 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:03.644 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:03.644 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:03.644 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.645 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.645 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:7902 3.as0 3:5732055a:::smithi10729407-8:head [append 0~499712] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-06-12T02:38:03.645 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.159351+0000", 2020-06-12T02:38:03.645 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.222981584999999, 2020-06-12T02:38:03.645 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013425812000000001, 2020-06-12T02:38:03.645 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:03.646 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:03.646 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:03.646 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:03.646 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:03.646 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 7902 2020-06-12T02:38:03.647 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.647 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:03.647 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.647 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:03.647 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.159351+0000", 2020-06-12T02:38:03.647 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:03.648 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.648 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.648 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:03.648 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.159351+0000", 2020-06-12T02:38:03.648 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00048251199999999999 2020-06-12T02:38:03.649 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.649 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.649 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:03.649 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.159833+0000", 2020-06-12T02:38:03.649 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7509999999999997e-06 2020-06-12T02:38:03.650 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.650 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.650 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:03.650 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.159839+0000", 2020-06-12T02:38:03.650 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9840000000000007e-06 2020-06-12T02:38:03.650 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.651 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.651 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:03.651 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.159848+0000", 2020-06-12T02:38:03.651 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4619999999999998e-06 2020-06-12T02:38:03.651 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.651 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.652 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:03.652 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.159852+0000", 2020-06-12T02:38:03.652 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5348000000000001e-05 2020-06-12T02:38:03.652 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.652 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.653 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:03.653 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.159888+0000", 2020-06-12T02:38:03.653 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9824000000000001e-05 2020-06-12T02:38:03.653 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.653 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.653 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:03.654 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.159958+0000", 2020-06-12T02:38:03.654 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0014605149999999999 2020-06-12T02:38:03.654 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.654 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.654 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:03.655 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.161418+0000", 2020-06-12T02:38:03.655 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0667999999999997e-05 2020-06-12T02:38:03.655 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.655 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.655 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:03.656 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.161469+0000", 2020-06-12T02:38:03.656 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00058577200000000003 2020-06-12T02:38:03.656 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.656 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.656 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:03.656 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.162054+0000", 2020-06-12T02:38:03.657 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010590761000000001 2020-06-12T02:38:03.657 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.657 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.658 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:03.658 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.172645+0000", 2020-06-12T02:38:03.658 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8467000000000006e-05 2020-06-12T02:38:03.658 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.658 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.659 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:03.659 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.172724+0000", 2020-06-12T02:38:03.659 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2747999999999997e-05 2020-06-12T02:38:03.659 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.659 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.660 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:03.660 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.172776+0000", 2020-06-12T02:38:03.660 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013425812000000001 2020-06-12T02:38:03.660 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:03.660 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:03.660 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:03.661 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.661 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.661 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:7907 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e33)", 2020-06-12T02:38:03.661 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.166559+0000", 2020-06-12T02:38:03.662 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.215772884, 2020-06-12T02:38:03.662 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013016145999999999, 2020-06-12T02:38:03.662 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:03.663 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:03.663 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:03.663 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:03.663 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:03.664 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 7907 2020-06-12T02:38:03.664 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.664 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:03.664 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.664 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:03.664 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.166559+0000", 2020-06-12T02:38:03.665 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:03.665 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.665 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.665 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:03.665 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.166559+0000", 2020-06-12T02:38:03.666 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1719999999999999e-05 2020-06-12T02:38:03.666 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.666 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.666 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:03.666 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.166581+0000", 2020-06-12T02:38:03.666 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8660000000000002e-06 2020-06-12T02:38:03.667 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.667 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.667 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:03.667 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.166584+0000", 2020-06-12T02:38:03.667 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3439999999999997e-06 2020-06-12T02:38:03.668 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.668 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.668 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:03.668 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.166590+0000", 2020-06-12T02:38:03.668 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0640000000000001e-06 2020-06-12T02:38:03.668 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.669 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.669 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:03.669 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.166592+0000", 2020-06-12T02:38:03.669 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0074059160000000002 2020-06-12T02:38:03.669 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.670 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.670 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:03.670 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.173998+0000", 2020-06-12T02:38:03.670 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8678999999999998e-05 2020-06-12T02:38:03.670 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.671 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.671 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:03.671 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.174057+0000", 2020-06-12T02:38:03.671 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045994779999999997 2020-06-12T02:38:03.671 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.671 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.672 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:03.672 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.178656+0000", 2020-06-12T02:38:03.672 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6825999999999999e-05 2020-06-12T02:38:03.672 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.672 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.673 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:03.673 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.178733+0000", 2020-06-12T02:38:03.673 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00084225299999999995 2020-06-12T02:38:03.673 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.673 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.673 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:03.674 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.179575+0000", 2020-06-12T02:38:03.674 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013016145999999999 2020-06-12T02:38:03.674 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:03.674 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:03.675 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:03.675 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.675 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.675 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:7931 3.as0 3:5732055a:::smithi10729407-8:head [append 0~696320] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-06-12T02:38:03.675 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.246494+0000", 2020-06-12T02:38:03.676 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 11.135837768, 2020-06-12T02:38:03.676 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014082301, 2020-06-12T02:38:03.676 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:03.676 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:03.676 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:03.677 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:03.677 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:03.677 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 7931 2020-06-12T02:38:03.677 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.678 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:03.678 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.678 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:03.678 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.246494+0000", 2020-06-12T02:38:03.679 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:03.679 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.679 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.679 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:03.679 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.246494+0000", 2020-06-12T02:38:03.679 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00077570099999999995 2020-06-12T02:38:03.680 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.680 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.680 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:03.680 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.247270+0000", 2020-06-12T02:38:03.680 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4179999999999997e-06 2020-06-12T02:38:03.681 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.681 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.681 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:03.681 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.247279+0000", 2020-06-12T02:38:03.681 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6297999999999999e-05 2020-06-12T02:38:03.681 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.682 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.682 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:03.682 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.247295+0000", 2020-06-12T02:38:03.682 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3309999999999998e-06 2020-06-12T02:38:03.682 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.683 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.683 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:03.683 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.247302+0000", 2020-06-12T02:38:03.683 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4283999999999999e-05 2020-06-12T02:38:03.683 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.683 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.684 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:03.684 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.247336+0000", 2020-06-12T02:38:03.684 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8157999999999994e-05 2020-06-12T02:38:03.684 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.684 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.685 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:03.685 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.247405+0000", 2020-06-12T02:38:03.685 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031257059999999998 2020-06-12T02:38:03.685 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.685 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.685 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:03.686 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.250530+0000", 2020-06-12T02:38:03.686 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7546999999999995e-05 2020-06-12T02:38:03.686 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.686 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.686 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:03.687 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.250608+0000", 2020-06-12T02:38:03.687 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00074126599999999997 2020-06-12T02:38:03.687 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.687 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.687 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:03.688 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.251349+0000", 2020-06-12T02:38:03.688 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.008999211 2020-06-12T02:38:03.688 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.688 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.688 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:03.689 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.260348+0000", 2020-06-12T02:38:03.689 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00016373200000000001 2020-06-12T02:38:03.689 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.689 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.689 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:03.690 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.260512+0000", 2020-06-12T02:38:03.690 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4648999999999997e-05 2020-06-12T02:38:03.690 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.690 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.690 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:03.690 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.260577+0000", 2020-06-12T02:38:03.691 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014082301 2020-06-12T02:38:03.691 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:03.691 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:03.691 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:03.691 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.692 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.692 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8129 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e33)", 2020-06-12T02:38:03.692 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.700485+0000", 2020-06-12T02:38:03.695 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 10.681847399, 2020-06-12T02:38:03.696 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01339515, 2020-06-12T02:38:03.696 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:03.696 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:03.696 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:03.696 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:03.697 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:03.697 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8129 2020-06-12T02:38:03.697 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.697 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:03.697 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.698 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:03.698 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.700485+0000", 2020-06-12T02:38:03.698 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:03.698 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.698 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.698 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:03.699 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.700485+0000", 2020-06-12T02:38:03.699 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4323999999999998e-05 2020-06-12T02:38:03.699 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.699 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.699 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:03.700 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.700529+0000", 2020-06-12T02:38:03.700 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7100000000000004e-06 2020-06-12T02:38:03.700 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.700 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.700 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:03.701 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.700535+0000", 2020-06-12T02:38:03.701 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1664999999999999e-05 2020-06-12T02:38:03.701 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.701 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.701 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:03.702 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.700547+0000", 2020-06-12T02:38:03.702 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6150000000000001e-06 2020-06-12T02:38:03.702 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.702 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.702 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:03.702 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.700550+0000", 2020-06-12T02:38:03.703 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00049316500000000005 2020-06-12T02:38:03.703 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.703 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.703 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:03.703 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.701043+0000", 2020-06-12T02:38:03.704 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0063999999999997e-05 2020-06-12T02:38:03.704 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.704 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.704 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:03.704 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.701133+0000", 2020-06-12T02:38:03.704 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0085745109999999999 2020-06-12T02:38:03.705 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.705 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.705 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:03.705 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.709708+0000", 2020-06-12T02:38:03.705 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8854999999999999e-05 2020-06-12T02:38:03.706 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.706 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.706 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:03.707 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.709757+0000", 2020-06-12T02:38:03.707 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002613279 2020-06-12T02:38:03.707 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.707 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.707 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:03.707 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.712370+0000", 2020-06-12T02:38:03.708 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4332000000000004e-05 2020-06-12T02:38:03.708 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.708 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.708 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:03.708 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.712464+0000", 2020-06-12T02:38:03.709 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00141563 2020-06-12T02:38:03.709 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:03.709 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:03.709 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:03.709 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.713880+0000", 2020-06-12T02:38:03.709 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01339515 2020-06-12T02:38:03.710 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:03.710 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:03.710 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:03.710 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:03.710 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:03.711 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:03.712 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-06-12T02:38:03.712 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:03.462+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:03.713 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-06-12T02:38:03.715 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2020-06-12T02:38:03.716 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.5 to 1 (10000) 2020-06-12T02:38:03.716 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:03.711+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:03.717 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd reweight 6 1 2020-06-12T02:38:03.815 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:03.812+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:03.835 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:03.835 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:38:03.836 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2020-06-12T02:38:03.836 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:03.843 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-06-12T02:38:03.891 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:38:03.916 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:38:03.929 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:03.926+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:03.930 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:38:03.950 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:38:03.971 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:38:03.991 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:38:04.003 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:04.003 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:38:04.004 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:38:04.004 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:38:04.004 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:04.011 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:38:04.012 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-06-12T02:38:04.031 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2020-06-12T02:38:04.032 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-06-12T02:38:04.032 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-06-12T02:38:04.033 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-06-12T02:38:04.033 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-06-12T02:38:04.033 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-06-12T02:38:04.034 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-06-12T02:38:04.034 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-06-12T02:38:04.034 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-06-12T02:38:04.054 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:04.051+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:04.071 INFO:teuthology.orchestra.run.smithi107.stderr:2020-06-12T02:38:04.069+0000 7feb337fe700 0 --2- 172.21.15.107:0/3240761178 >> v2:172.21.15.107:3300/0 conn(0x7feb3409eb70 0x7feb34131fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2020-06-12T02:38:04.118 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:04.115+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:04.171 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:04.171 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:04.177 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-06-12T02:38:04.222 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:04.220+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:04.294 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:04.295 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:04.304 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-06-12T02:38:04.345 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:04.343+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:04.423 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:04.419+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:04.461 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:04.461 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:04.470 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-06-12T02:38:04.555 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:04.553+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:04.616 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.6 to 1 (10000) 2020-06-12T02:38:04.628 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd reweight 7 1 2020-06-12T02:38:04.629 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:04.623+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:04.631 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:04.632 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:04.640 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-06-12T02:38:04.727 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:04.724+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:04.801 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:04.802 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:04.810 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-06-12T02:38:04.827 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:04.825+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:04.928 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:04.926+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:04.966 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:04.967 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:04.975 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-06-12T02:38:05.029 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:05.027+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:05.136 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:05.138 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:05.138 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:05.134+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:05.140 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-06-12T02:38:05.237 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:05.235+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:05.297 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:05.297 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:05.305 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-06-12T02:38:05.346 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:05.344+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:05.439 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:05.438+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:05.461 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:05.462 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:05.470 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-06-12T02:38:05.554 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:05.552+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:05.624 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.7 to 1 (10000) 2020-06-12T02:38:05.629 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:05.629 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:05.635 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2020-06-12T02:38:05.635 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json 2020-06-12T02:38:05.639 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-06-12T02:38:05.641 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:05.639+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:05.718 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd unset noscrub 2020-06-12T02:38:05.765 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:05.763+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:05.803 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:05.803 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:05.812 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-06-12T02:38:05.853 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:05.851+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:05.947 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:05.945+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:05.950 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:38:05.951 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2020-06-12T02:38:05.961 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2020-06-12T02:38:05.619155+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":13967360,"num_objects":9,"num_object_clones":0,"num_object_copies":25,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2516,"num_read_kb":1358,"num_write":5602,"num_write_kb":1598418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7412,"ondisk_log_size":7412,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":66,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8412720,"kb_used_data":24048,"kb_used_omap":25,"kb_used_meta":8388582,"kb_avail":746562000,"statfs":{"total":773094113280,"available":764479488000,"internally_reserved":8589934592,"allocated":24625152,"data_stored":22609562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26157,"internal_metadata":8589908435},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.016814"},"pg_stats":[{"pgid":"3.f","version":"36'742","reported_seq":"913","reported_epoch":"41","state":"active+clean+wait","last_fresh":"2020-06-12T02:38:00.582815+0000","last_change":"2020-06-12T02:37:57.567984+0000","last_active":"2020-06-12T02:38:00.582815+0000","last_peered":"2020-06-12T02:38:00.582815+0000","last_clean":"2020-06-12T02:38:00.582815+0000","last_became_active":"2020-06-12T02:37:57.566633+0000","last_became_peered":"2020-06-12T02:37:57.566633+0000","last_unstale":"2020-06-12T02:38:00.582815+0000","last_undegraded":"2020-06-12T02:38:00.582815+0000","last_fullsized":"2020-06-12T02:38:00.582815+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":742,"ondisk_log_size":742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1540096,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":251,"num_read_kb":128,"num_write":523,"num_write_kb":142928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1531904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"36'402","reported_seq":"518","reported_epoch":"41","state":"active+clean+wait","last_fresh":"2020-06-12T02:38:00.582735+0000","last_change":"2020-06-12T02:37:57.564534+0000","last_active":"2020-06-12T02:38:00.582735+0000","last_peered":"2020-06-12T02:38:00.582735+0000","last_clean":"2020-06-12T02:38:00.582735+0000","last_became_active":"2020-06-12T02:37:57.563869+0000","last_became_peered":"2020-06-12T02:37:57.563869+0000","last_unstale":"2020-06-12T02:38:00.582735+0000","last_undegraded":"2020-06-12T02:38:00.582735+0000","last_fullsized":"2020-06-12T02:38:00.582735+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":402,"ondisk_log_size":402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":131,"num_read_kb":71,"num_write":306,"num_write_kb":88168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1466368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"36'770","reported_seq":"969","reported_epoch":"41","state":"active+clean+wait","last_fresh":"2020-06-12T02:38:00.582525+0000","last_change":"2020-06-12T02:37:57.565160+0000","last_active":"2020-06-12T02:38:00.582525+0000","last_peered":"2020-06-12T02:38:00.582525+0000","last_clean":"2020-06-12T02:38:00.582525+0000","last_became_active":"2020-06-12T02:37:57.564464+0000","last_became_peered":"2020-06-12T02:37:57.564464+0000","last_unstale":"2020-06-12T02:38:00.582525+0000","last_undegraded":"2020-06-12T02:38:00.582525+0000","last_fullsized":"2020-06-12T02:38:00.582525+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":770,"ondisk_log_size":770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1490944,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":252,"num_read_kb":144,"num_write":593,"num_write_kb":168240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3710976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"24","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:01.362480+0000","last_change":"2020-06-12T02:37:42.384656+0000","last_active":"2020-06-12T02:38:01.362480+0000","last_peered":"2020-06-12T02:38:01.362480+0000","last_clean":"2020-06-12T02:38:01.362480+0000","last_became_active":"2020-06-12T02:37:42.384363+0000","last_became_peered":"2020-06-12T02:37:42.384363+0000","last_unstale":"2020-06-12T02:38:01.362480+0000","last_undegraded":"2020-06-12T02:38:01.362480+0000","last_fullsized":"2020-06-12T02:38:01.362480+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"36'730","reported_seq":"895","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:01.362495+0000","last_change":"2020-06-12T02:37:27.902643+0000","last_active":"2020-06-12T02:38:01.362495+0000","last_peered":"2020-06-12T02:38:01.362495+0000","last_clean":"2020-06-12T02:38:01.362495+0000","last_became_active":"2020-06-12T02:37:27.902204+0000","last_became_peered":"2020-06-12T02:37:27.902204+0000","last_unstale":"2020-06-12T02:38:01.362495+0000","last_undegraded":"2020-06-12T02:38:01.362495+0000","last_fullsized":"2020-06-12T02:38:01.362495+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":730,"ondisk_log_size":730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":242,"num_read_kb":136,"num_write":558,"num_write_kb":161104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"36'715","reported_seq":"882","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:01.362662+0000","last_change":"2020-06-12T02:37:27.905454+0000","last_active":"2020-06-12T02:38:01.362662+0000","last_peered":"2020-06-12T02:38:01.362662+0000","last_clean":"2020-06-12T02:38:01.362662+0000","last_became_active":"2020-06-12T02:37:27.904857+0000","last_became_peered":"2020-06-12T02:37:27.904857+0000","last_unstale":"2020-06-12T02:38:01.362662+0000","last_undegraded":"2020-06-12T02:38:01.362662+0000","last_fullsized":"2020-06-12T02:38:01.362662+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":715,"ondisk_log_size":715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1343488,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":256,"num_read_kb":139,"num_write":553,"num_write_kb":155176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"36'689","reported_seq":"863","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:01.235836+0000","last_change":"2020-06-12T02:37:57.567142+0000","last_active":"2020-06-12T02:38:01.235836+0000","last_peered":"2020-06-12T02:38:01.235836+0000","last_clean":"2020-06-12T02:38:01.235836+0000","last_became_active":"2020-06-12T02:37:57.566462+0000","last_became_peered":"2020-06-12T02:37:57.566462+0000","last_unstale":"2020-06-12T02:38:01.235836+0000","last_undegraded":"2020-06-12T02:38:01.235836+0000","last_fullsized":"2020-06-12T02:38:01.235836+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":689,"ondisk_log_size":689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":226,"num_read_kb":124,"num_write":487,"num_write_kb":134808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"36'425","reported_seq":"552","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:01.362465+0000","last_change":"2020-06-12T02:37:57.567506+0000","last_active":"2020-06-12T02:38:01.362465+0000","last_peered":"2020-06-12T02:38:01.362465+0000","last_clean":"2020-06-12T02:38:01.362465+0000","last_became_active":"2020-06-12T02:37:57.566796+0000","last_became_peered":"2020-06-12T02:37:57.566796+0000","last_unstale":"2020-06-12T02:38:01.362465+0000","last_undegraded":"2020-06-12T02:38:01.362465+0000","last_fullsized":"2020-06-12T02:38:01.362465+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":425,"ondisk_log_size":425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1138688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":147,"num_read_kb":78,"num_write":334,"num_write_kb":97824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2793472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"33","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:00.586306+0000","last_change":"2020-06-12T02:37:16.868466+0000","last_active":"2020-06-12T02:38:00.586306+0000","last_peered":"2020-06-12T02:38:00.586306+0000","last_clean":"2020-06-12T02:38:00.586306+0000","last_became_active":"2020-06-12T02:37:16.868098+0000","last_became_peered":"2020-06-12T02:37:16.868098+0000","last_unstale":"2020-06-12T02:38:00.586306+0000","last_undegraded":"2020-06-12T02:38:00.586306+0000","last_fullsized":"2020-06-12T02:38:00.586306+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"36'720","reported_seq":"877","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:01.362556+0000","last_change":"2020-06-12T02:37:42.379294+0000","last_active":"2020-06-12T02:38:01.362556+0000","last_peered":"2020-06-12T02:38:01.362556+0000","last_clean":"2020-06-12T02:38:01.362556+0000","last_became_active":"2020-06-12T02:37:42.378796+0000","last_became_peered":"2020-06-12T02:37:42.378796+0000","last_unstale":"2020-06-12T02:38:01.362556+0000","last_undegraded":"2020-06-12T02:38:01.362556+0000","last_fullsized":"2020-06-12T02:38:01.362556+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":720,"ondisk_log_size":720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2514944,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":237,"num_read_kb":127,"num_write":533,"num_write_kb":155776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"23","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:00.586062+0000","last_change":"2020-06-12T02:37:42.373838+0000","last_active":"2020-06-12T02:38:00.586062+0000","last_peered":"2020-06-12T02:38:00.586062+0000","last_clean":"2020-06-12T02:38:00.586062+0000","last_became_active":"2020-06-12T02:37:42.373205+0000","last_became_peered":"2020-06-12T02:37:42.373205+0000","last_unstale":"2020-06-12T02:38:00.586062+0000","last_undegraded":"2020-06-12T02:38:00.586062+0000","last_fullsized":"2020-06-12T02:38:00.586062+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"36'769","reported_seq":"933","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:01.235834+0000","last_change":"2020-06-12T02:37:27.903386+0000","last_active":"2020-06-12T02:38:01.235834+0000","last_peered":"2020-06-12T02:38:01.235834+0000","last_clean":"2020-06-12T02:38:01.235834+0000","last_became_active":"2020-06-12T02:37:27.902910+0000","last_became_peered":"2020-06-12T02:37:27.902910+0000","last_unstale":"2020-06-12T02:38:01.235834+0000","last_undegraded":"2020-06-12T02:38:01.235834+0000","last_fullsized":"2020-06-12T02:38:01.235834+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":769,"ondisk_log_size":769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":925696,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":256,"num_read_kb":135,"num_write":582,"num_write_kb":169744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"33","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:00.586974+0000","last_change":"2020-06-12T02:37:16.765460+0000","last_active":"2020-06-12T02:38:00.586974+0000","last_peered":"2020-06-12T02:38:00.586974+0000","last_clean":"2020-06-12T02:38:00.586974+0000","last_became_active":"2020-06-12T02:37:16.764958+0000","last_became_peered":"2020-06-12T02:37:16.764958+0000","last_unstale":"2020-06-12T02:38:00.586974+0000","last_undegraded":"2020-06-12T02:38:00.586974+0000","last_fullsized":"2020-06-12T02:38:00.586974+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"36'401","reported_seq":"505","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:00.586077+0000","last_change":"2020-06-12T02:37:27.905010+0000","last_active":"2020-06-12T02:38:00.586077+0000","last_peered":"2020-06-12T02:38:00.586077+0000","last_clean":"2020-06-12T02:38:00.586077+0000","last_became_active":"2020-06-12T02:37:27.904552+0000","last_became_peered":"2020-06-12T02:37:27.904552+0000","last_unstale":"2020-06-12T02:38:00.586077+0000","last_undegraded":"2020-06-12T02:38:00.586077+0000","last_fullsized":"2020-06-12T02:38:00.586077+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":401,"ondisk_log_size":401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":144,"num_read_kb":75,"num_write":313,"num_write_kb":90136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"23","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:00.586919+0000","last_change":"2020-06-12T02:37:42.376260+0000","last_active":"2020-06-12T02:38:00.586919+0000","last_peered":"2020-06-12T02:38:00.586919+0000","last_clean":"2020-06-12T02:38:00.586919+0000","last_became_active":"2020-06-12T02:37:42.375902+0000","last_became_peered":"2020-06-12T02:37:42.375902+0000","last_unstale":"2020-06-12T02:38:00.586919+0000","last_undegraded":"2020-06-12T02:38:00.586919+0000","last_fullsized":"2020-06-12T02:38:00.586919+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"15'2","reported_seq":"30","reported_epoch":"41","state":"active+clean+wait","last_fresh":"2020-06-12T02:38:00.582577+0000","last_change":"2020-06-12T02:37:54.558799+0000","last_active":"2020-06-12T02:38:00.582577+0000","last_peered":"2020-06-12T02:38:00.582577+0000","last_clean":"2020-06-12T02:38:00.582577+0000","last_became_active":"2020-06-12T02:37:54.558125+0000","last_became_peered":"2020-06-12T02:37:54.558125+0000","last_unstale":"2020-06-12T02:38:00.582577+0000","last_undegraded":"2020-06-12T02:38:00.582577+0000","last_fullsized":"2020-06-12T02:38:00.582577+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":3,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:04.419204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:04.419204+0000","last_clean_scrub_stamp":"2020-06-12T02:37:04.419204+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"36'315","reported_seq":"406","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:01.362493+0000","last_change":"2020-06-12T02:37:27.898985+0000","last_active":"2020-06-12T02:38:01.362493+0000","last_peered":"2020-06-12T02:38:01.362493+0000","last_clean":"2020-06-12T02:38:01.362493+0000","last_became_active":"2020-06-12T02:37:27.898516+0000","last_became_peered":"2020-06-12T02:37:27.898516+0000","last_unstale":"2020-06-12T02:38:01.362493+0000","last_undegraded":"2020-06-12T02:38:01.362493+0000","last_fullsized":"2020-06-12T02:38:01.362493+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":315,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":116,"num_read_kb":62,"num_write":248,"num_write_kb":69496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"27","reported_epoch":"41","state":"active+clean+wait","last_fresh":"2020-06-12T02:38:00.582446+0000","last_change":"2020-06-12T02:37:54.558450+0000","last_active":"2020-06-12T02:38:00.582446+0000","last_peered":"2020-06-12T02:38:00.582446+0000","last_clean":"2020-06-12T02:38:00.582446+0000","last_became_active":"2020-06-12T02:37:54.557850+0000","last_became_peered":"2020-06-12T02:37:54.557850+0000","last_unstale":"2020-06-12T02:38:00.582446+0000","last_undegraded":"2020-06-12T02:38:00.582446+0000","last_fullsized":"2020-06-12T02:38:00.582446+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"28","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:00.981519+0000","last_change":"2020-06-12T02:37:27.901368+0000","last_active":"2020-06-12T02:38:00.981519+0000","last_peered":"2020-06-12T02:38:00.981519+0000","last_clean":"2020-06-12T02:38:00.981519+0000","last_became_active":"2020-06-12T02:37:27.900888+0000","last_became_peered":"2020-06-12T02:37:27.900888+0000","last_unstale":"2020-06-12T02:38:00.981519+0000","last_undegraded":"2020-06-12T02:38:00.981519+0000","last_fullsized":"2020-06-12T02:38:00.981519+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"27","reported_epoch":"41","state":"active+clean+wait","last_fresh":"2020-06-12T02:38:00.582686+0000","last_change":"2020-06-12T02:37:54.557777+0000","last_active":"2020-06-12T02:38:00.582686+0000","last_peered":"2020-06-12T02:38:00.582686+0000","last_clean":"2020-06-12T02:38:00.582686+0000","last_became_active":"2020-06-12T02:37:54.557420+0000","last_became_peered":"2020-06-12T02:37:54.557420+0000","last_unstale":"2020-06-12T02:38:00.582686+0000","last_undegraded":"2020-06-12T02:38:00.582686+0000","last_fullsized":"2020-06-12T02:38:00.582686+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"29","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:00.586191+0000","last_change":"2020-06-12T02:37:27.901486+0000","last_active":"2020-06-12T02:38:00.586191+0000","last_peered":"2020-06-12T02:38:00.586191+0000","last_clean":"2020-06-12T02:38:00.586191+0000","last_became_active":"2020-06-12T02:37:27.901018+0000","last_became_peered":"2020-06-12T02:37:27.901018+0000","last_unstale":"2020-06-12T02:38:00.586191+0000","last_undegraded":"2020-06-12T02:38:00.586191+0000","last_fullsized":"2020-06-12T02:38:00.586191+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"36'732","reported_seq":"918","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:01.362569+0000","last_change":"2020-06-12T02:37:57.567683+0000","last_active":"2020-06-12T02:38:01.362569+0000","last_peered":"2020-06-12T02:38:01.362569+0000","last_clean":"2020-06-12T02:38:01.362569+0000","last_became_active":"2020-06-12T02:37:57.566865+0000","last_became_peered":"2020-06-12T02:37:57.566865+0000","last_unstale":"2020-06-12T02:38:01.362569+0000","last_undegraded":"2020-06-12T02:38:01.362569+0000","last_fullsized":"2020-06-12T02:38:01.362569+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":732,"ondisk_log_size":732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5013504,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":258,"num_read_kb":139,"num_write":570,"num_write_kb":165008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2793472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"33","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:00.586524+0000","last_change":"2020-06-12T02:37:16.763049+0000","last_active":"2020-06-12T02:38:00.586524+0000","last_peered":"2020-06-12T02:38:00.586524+0000","last_clean":"2020-06-12T02:38:00.586524+0000","last_became_active":"2020-06-12T02:37:16.762626+0000","last_became_peered":"2020-06-12T02:37:16.762626+0000","last_unstale":"2020-06-12T02:38:00.586524+0000","last_undegraded":"2020-06-12T02:38:00.586524+0000","last_fullsized":"2020-06-12T02:38:00.586524+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"45","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:01.362703+0000","last_change":"2020-06-12T02:37:54.555377+0000","last_active":"2020-06-12T02:38:01.362703+0000","last_peered":"2020-06-12T02:38:01.362703+0000","last_clean":"2020-06-12T02:38:01.362703+0000","last_became_active":"2020-06-12T02:37:54.554986+0000","last_became_peered":"2020-06-12T02:37:54.554986+0000","last_unstale":"2020-06-12T02:38:01.362703+0000","last_undegraded":"2020-06-12T02:38:01.362703+0000","last_fullsized":"2020-06-12T02:38:01.362703+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"28","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:00.981638+0000","last_change":"2020-06-12T02:37:27.906478+0000","last_active":"2020-06-12T02:38:00.981638+0000","last_peered":"2020-06-12T02:38:00.981638+0000","last_clean":"2020-06-12T02:38:00.981638+0000","last_became_active":"2020-06-12T02:37:27.905639+0000","last_became_peered":"2020-06-12T02:37:27.905639+0000","last_unstale":"2020-06-12T02:38:00.981638+0000","last_undegraded":"2020-06-12T02:38:00.981638+0000","last_fullsized":"2020-06-12T02:38:00.981638+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":13967360,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2516,"num_read_kb":1358,"num_write":5600,"num_write_kb":1598408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20951040,"data_stored":20951040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7410,"ondisk_log_size":7410,"up":48,"acting":48,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26157,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":34,"seq":146028888067,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1053532,"kb_used_data":4948,"kb_used_omap":25,"kb_used_meta":1048550,"kb_avail":93318308,"statfs":{"total":96636764160,"available":95557947392,"internally_reserved":1073741824,"allocated":5066752,"data_stored":4806767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26157,"internal_metadata":1073715667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1459999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.03}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2709999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1659999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3089999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574861,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049680,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322160,"statfs":{"total":96636764160,"available":95561891840,"internally_reserved":1073741824,"allocated":1122304,"data_stored":874451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2230000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.081}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0469999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.764}]}]},{"osd":1,"up_from":13,"seq":55834574862,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050700,"kb_used_data":2116,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321140,"statfs":{"total":96636764160,"available":95560847360,"internally_reserved":1073741824,"allocated":2166784,"data_stored":1912865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0129999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3220000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4570000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574862,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1052688,"kb_used_data":4104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93319152,"statfs":{"total":96636764160,"available":95558811648,"internally_reserved":1073741824,"allocated":4202496,"data_stored":3948577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2609999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3959999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8200000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.226}]}]},{"osd":2,"up_from":13,"seq":55834574861,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050264,"kb_used_data":1680,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321576,"statfs":{"total":96636764160,"available":95561293824,"internally_reserved":1073741824,"allocated":1720320,"data_stored":1466401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.876}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4490000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.387}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.677}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6539999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574861,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1053984,"kb_used_data":5400,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93317856,"statfs":{"total":96636764160,"available":95557484544,"internally_reserved":1073741824,"allocated":5529600,"data_stored":5281747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0169999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98599999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6180000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.103}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.45}]}]},{"osd":4,"up_from":13,"seq":55834574860,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050896,"kb_used_data":2312,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93320944,"statfs":{"total":96636764160,"available":95560646656,"internally_reserved":1073741824,"allocated":2367488,"data_stored":2113569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.712}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0029999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.617}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.141}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4330000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.407}]}]},{"osd":5,"up_from":13,"seq":55834574860,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050976,"kb_used_data":2392,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93320864,"statfs":{"total":96636764160,"available":95560564736,"internally_reserved":1073741824,"allocated":2449408,"data_stored":2205185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0760000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1850000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94699999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97599999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.012}]}]}]}} 2020-06-12T02:38:05.965 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json 2020-06-12T02:38:05.972 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:05.973 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:05.982 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-06-12T02:38:06.043 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-06-12T02:38:06.066 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:06.064+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:06.149 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:06.147+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:06.170 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:06.170 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:06.181 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-06-12T02:38:06.204 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:38:06.224 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:38:06.244 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:38:06.250 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:06.248+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:06.264 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:38:06.264 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:38:06.265 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2020-06-12T02:38:06.274 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2020-06-12T02:38:05.619155+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":13967360,"num_objects":9,"num_object_clones":0,"num_object_copies":25,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2516,"num_read_kb":1358,"num_write":5602,"num_write_kb":1598418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7412,"ondisk_log_size":7412,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":66,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8412720,"kb_used_data":24048,"kb_used_omap":25,"kb_used_meta":8388582,"kb_avail":746562000,"statfs":{"total":773094113280,"available":764479488000,"internally_reserved":8589934592,"allocated":24625152,"data_stored":22609562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26157,"internal_metadata":8589908435},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.016814"},"pg_stats":[{"pgid":"3.f","version":"36'742","reported_seq":"913","reported_epoch":"41","state":"active+clean+wait","last_fresh":"2020-06-12T02:38:00.582815+0000","last_change":"2020-06-12T02:37:57.567984+0000","last_active":"2020-06-12T02:38:00.582815+0000","last_peered":"2020-06-12T02:38:00.582815+0000","last_clean":"2020-06-12T02:38:00.582815+0000","last_became_active":"2020-06-12T02:37:57.566633+0000","last_became_peered":"2020-06-12T02:37:57.566633+0000","last_unstale":"2020-06-12T02:38:00.582815+0000","last_undegraded":"2020-06-12T02:38:00.582815+0000","last_fullsized":"2020-06-12T02:38:00.582815+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":742,"ondisk_log_size":742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1540096,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":251,"num_read_kb":128,"num_write":523,"num_write_kb":142928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1531904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"36'402","reported_seq":"518","reported_epoch":"41","state":"active+clean+wait","last_fresh":"2020-06-12T02:38:00.582735+0000","last_change":"2020-06-12T02:37:57.564534+0000","last_active":"2020-06-12T02:38:00.582735+0000","last_peered":"2020-06-12T02:38:00.582735+0000","last_clean":"2020-06-12T02:38:00.582735+0000","last_became_active":"2020-06-12T02:37:57.563869+0000","last_became_peered":"2020-06-12T02:37:57.563869+0000","last_unstale":"2020-06-12T02:38:00.582735+0000","last_undegraded":"2020-06-12T02:38:00.582735+0000","last_fullsized":"2020-06-12T02:38:00.582735+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":402,"ondisk_log_size":402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":131,"num_read_kb":71,"num_write":306,"num_write_kb":88168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1466368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"36'770","reported_seq":"969","reported_epoch":"41","state":"active+clean+wait","last_fresh":"2020-06-12T02:38:00.582525+0000","last_change":"2020-06-12T02:37:57.565160+0000","last_active":"2020-06-12T02:38:00.582525+0000","last_peered":"2020-06-12T02:38:00.582525+0000","last_clean":"2020-06-12T02:38:00.582525+0000","last_became_active":"2020-06-12T02:37:57.564464+0000","last_became_peered":"2020-06-12T02:37:57.564464+0000","last_unstale":"2020-06-12T02:38:00.582525+0000","last_undegraded":"2020-06-12T02:38:00.582525+0000","last_fullsized":"2020-06-12T02:38:00.582525+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":770,"ondisk_log_size":770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1490944,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":252,"num_read_kb":144,"num_write":593,"num_write_kb":168240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3710976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"24","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:01.362480+0000","last_change":"2020-06-12T02:37:42.384656+0000","last_active":"2020-06-12T02:38:01.362480+0000","last_peered":"2020-06-12T02:38:01.362480+0000","last_clean":"2020-06-12T02:38:01.362480+0000","last_became_active":"2020-06-12T02:37:42.384363+0000","last_became_peered":"2020-06-12T02:37:42.384363+0000","last_unstale":"2020-06-12T02:38:01.362480+0000","last_undegraded":"2020-06-12T02:38:01.362480+0000","last_fullsized":"2020-06-12T02:38:01.362480+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"36'730","reported_seq":"895","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:01.362495+0000","last_change":"2020-06-12T02:37:27.902643+0000","last_active":"2020-06-12T02:38:01.362495+0000","last_peered":"2020-06-12T02:38:01.362495+0000","last_clean":"2020-06-12T02:38:01.362495+0000","last_became_active":"2020-06-12T02:37:27.902204+0000","last_became_peered":"2020-06-12T02:37:27.902204+0000","last_unstale":"2020-06-12T02:38:01.362495+0000","last_undegraded":"2020-06-12T02:38:01.362495+0000","last_fullsized":"2020-06-12T02:38:01.362495+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":730,"ondisk_log_size":730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":242,"num_read_kb":136,"num_write":558,"num_write_kb":161104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"36'715","reported_seq":"882","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:01.362662+0000","last_change":"2020-06-12T02:37:27.905454+0000","last_active":"2020-06-12T02:38:01.362662+0000","last_peered":"2020-06-12T02:38:01.362662+0000","last_clean":"2020-06-12T02:38:01.362662+0000","last_became_active":"2020-06-12T02:37:27.904857+0000","last_became_peered":"2020-06-12T02:37:27.904857+0000","last_unstale":"2020-06-12T02:38:01.362662+0000","last_undegraded":"2020-06-12T02:38:01.362662+0000","last_fullsized":"2020-06-12T02:38:01.362662+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":715,"ondisk_log_size":715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1343488,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":256,"num_read_kb":139,"num_write":553,"num_write_kb":155176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"36'689","reported_seq":"863","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:01.235836+0000","last_change":"2020-06-12T02:37:57.567142+0000","last_active":"2020-06-12T02:38:01.235836+0000","last_peered":"2020-06-12T02:38:01.235836+0000","last_clean":"2020-06-12T02:38:01.235836+0000","last_became_active":"2020-06-12T02:37:57.566462+0000","last_became_peered":"2020-06-12T02:37:57.566462+0000","last_unstale":"2020-06-12T02:38:01.235836+0000","last_undegraded":"2020-06-12T02:38:01.235836+0000","last_fullsized":"2020-06-12T02:38:01.235836+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":689,"ondisk_log_size":689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":226,"num_read_kb":124,"num_write":487,"num_write_kb":134808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"36'425","reported_seq":"552","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:01.362465+0000","last_change":"2020-06-12T02:37:57.567506+0000","last_active":"2020-06-12T02:38:01.362465+0000","last_peered":"2020-06-12T02:38:01.362465+0000","last_clean":"2020-06-12T02:38:01.362465+0000","last_became_active":"2020-06-12T02:37:57.566796+0000","last_became_peered":"2020-06-12T02:37:57.566796+0000","last_unstale":"2020-06-12T02:38:01.362465+0000","last_undegraded":"2020-06-12T02:38:01.362465+0000","last_fullsized":"2020-06-12T02:38:01.362465+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":425,"ondisk_log_size":425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1138688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":147,"num_read_kb":78,"num_write":334,"num_write_kb":97824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2793472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"33","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:00.586306+0000","last_change":"2020-06-12T02:37:16.868466+0000","last_active":"2020-06-12T02:38:00.586306+0000","last_peered":"2020-06-12T02:38:00.586306+0000","last_clean":"2020-06-12T02:38:00.586306+0000","last_became_active":"2020-06-12T02:37:16.868098+0000","last_became_peered":"2020-06-12T02:37:16.868098+0000","last_unstale":"2020-06-12T02:38:00.586306+0000","last_undegraded":"2020-06-12T02:38:00.586306+0000","last_fullsized":"2020-06-12T02:38:00.586306+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"36'720","reported_seq":"877","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:01.362556+0000","last_change":"2020-06-12T02:37:42.379294+0000","last_active":"2020-06-12T02:38:01.362556+0000","last_peered":"2020-06-12T02:38:01.362556+0000","last_clean":"2020-06-12T02:38:01.362556+0000","last_became_active":"2020-06-12T02:37:42.378796+0000","last_became_peered":"2020-06-12T02:37:42.378796+0000","last_unstale":"2020-06-12T02:38:01.362556+0000","last_undegraded":"2020-06-12T02:38:01.362556+0000","last_fullsized":"2020-06-12T02:38:01.362556+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":720,"ondisk_log_size":720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2514944,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":237,"num_read_kb":127,"num_write":533,"num_write_kb":155776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"23","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:00.586062+0000","last_change":"2020-06-12T02:37:42.373838+0000","last_active":"2020-06-12T02:38:00.586062+0000","last_peered":"2020-06-12T02:38:00.586062+0000","last_clean":"2020-06-12T02:38:00.586062+0000","last_became_active":"2020-06-12T02:37:42.373205+0000","last_became_peered":"2020-06-12T02:37:42.373205+0000","last_unstale":"2020-06-12T02:38:00.586062+0000","last_undegraded":"2020-06-12T02:38:00.586062+0000","last_fullsized":"2020-06-12T02:38:00.586062+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"36'769","reported_seq":"933","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:01.235834+0000","last_change":"2020-06-12T02:37:27.903386+0000","last_active":"2020-06-12T02:38:01.235834+0000","last_peered":"2020-06-12T02:38:01.235834+0000","last_clean":"2020-06-12T02:38:01.235834+0000","last_became_active":"2020-06-12T02:37:27.902910+0000","last_became_peered":"2020-06-12T02:37:27.902910+0000","last_unstale":"2020-06-12T02:38:01.235834+0000","last_undegraded":"2020-06-12T02:38:01.235834+0000","last_fullsized":"2020-06-12T02:38:01.235834+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":769,"ondisk_log_size":769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":925696,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":256,"num_read_kb":135,"num_write":582,"num_write_kb":169744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"33","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:00.586974+0000","last_change":"2020-06-12T02:37:16.765460+0000","last_active":"2020-06-12T02:38:00.586974+0000","last_peered":"2020-06-12T02:38:00.586974+0000","last_clean":"2020-06-12T02:38:00.586974+0000","last_became_active":"2020-06-12T02:37:16.764958+0000","last_became_peered":"2020-06-12T02:37:16.764958+0000","last_unstale":"2020-06-12T02:38:00.586974+0000","last_undegraded":"2020-06-12T02:38:00.586974+0000","last_fullsized":"2020-06-12T02:38:00.586974+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"36'401","reported_seq":"505","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:00.586077+0000","last_change":"2020-06-12T02:37:27.905010+0000","last_active":"2020-06-12T02:38:00.586077+0000","last_peered":"2020-06-12T02:38:00.586077+0000","last_clean":"2020-06-12T02:38:00.586077+0000","last_became_active":"2020-06-12T02:37:27.904552+0000","last_became_peered":"2020-06-12T02:37:27.904552+0000","last_unstale":"2020-06-12T02:38:00.586077+0000","last_undegraded":"2020-06-12T02:38:00.586077+0000","last_fullsized":"2020-06-12T02:38:00.586077+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":401,"ondisk_log_size":401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":144,"num_read_kb":75,"num_write":313,"num_write_kb":90136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"23","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:00.586919+0000","last_change":"2020-06-12T02:37:42.376260+0000","last_active":"2020-06-12T02:38:00.586919+0000","last_peered":"2020-06-12T02:38:00.586919+0000","last_clean":"2020-06-12T02:38:00.586919+0000","last_became_active":"2020-06-12T02:37:42.375902+0000","last_became_peered":"2020-06-12T02:37:42.375902+0000","last_unstale":"2020-06-12T02:38:00.586919+0000","last_undegraded":"2020-06-12T02:38:00.586919+0000","last_fullsized":"2020-06-12T02:38:00.586919+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"15'2","reported_seq":"30","reported_epoch":"41","state":"active+clean+wait","last_fresh":"2020-06-12T02:38:00.582577+0000","last_change":"2020-06-12T02:37:54.558799+0000","last_active":"2020-06-12T02:38:00.582577+0000","last_peered":"2020-06-12T02:38:00.582577+0000","last_clean":"2020-06-12T02:38:00.582577+0000","last_became_active":"2020-06-12T02:37:54.558125+0000","last_became_peered":"2020-06-12T02:37:54.558125+0000","last_unstale":"2020-06-12T02:38:00.582577+0000","last_undegraded":"2020-06-12T02:38:00.582577+0000","last_fullsized":"2020-06-12T02:38:00.582577+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":3,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:04.419204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:04.419204+0000","last_clean_scrub_stamp":"2020-06-12T02:37:04.419204+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"36'315","reported_seq":"406","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:01.362493+0000","last_change":"2020-06-12T02:37:27.898985+0000","last_active":"2020-06-12T02:38:01.362493+0000","last_peered":"2020-06-12T02:38:01.362493+0000","last_clean":"2020-06-12T02:38:01.362493+0000","last_became_active":"2020-06-12T02:37:27.898516+0000","last_became_peered":"2020-06-12T02:37:27.898516+0000","last_unstale":"2020-06-12T02:38:01.362493+0000","last_undegraded":"2020-06-12T02:38:01.362493+0000","last_fullsized":"2020-06-12T02:38:01.362493+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":315,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":116,"num_read_kb":62,"num_write":248,"num_write_kb":69496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"27","reported_epoch":"41","state":"active+clean+wait","last_fresh":"2020-06-12T02:38:00.582446+0000","last_change":"2020-06-12T02:37:54.558450+0000","last_active":"2020-06-12T02:38:00.582446+0000","last_peered":"2020-06-12T02:38:00.582446+0000","last_clean":"2020-06-12T02:38:00.582446+0000","last_became_active":"2020-06-12T02:37:54.557850+0000","last_became_peered":"2020-06-12T02:37:54.557850+0000","last_unstale":"2020-06-12T02:38:00.582446+0000","last_undegraded":"2020-06-12T02:38:00.582446+0000","last_fullsized":"2020-06-12T02:38:00.582446+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"28","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:00.981519+0000","last_change":"2020-06-12T02:37:27.901368+0000","last_active":"2020-06-12T02:38:00.981519+0000","last_peered":"2020-06-12T02:38:00.981519+0000","last_clean":"2020-06-12T02:38:00.981519+0000","last_became_active":"2020-06-12T02:37:27.900888+0000","last_became_peered":"2020-06-12T02:37:27.900888+0000","last_unstale":"2020-06-12T02:38:00.981519+0000","last_undegraded":"2020-06-12T02:38:00.981519+0000","last_fullsized":"2020-06-12T02:38:00.981519+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"27","reported_epoch":"41","state":"active+clean+wait","last_fresh":"2020-06-12T02:38:00.582686+0000","last_change":"2020-06-12T02:37:54.557777+0000","last_active":"2020-06-12T02:38:00.582686+0000","last_peered":"2020-06-12T02:38:00.582686+0000","last_clean":"2020-06-12T02:38:00.582686+0000","last_became_active":"2020-06-12T02:37:54.557420+0000","last_became_peered":"2020-06-12T02:37:54.557420+0000","last_unstale":"2020-06-12T02:38:00.582686+0000","last_undegraded":"2020-06-12T02:38:00.582686+0000","last_fullsized":"2020-06-12T02:38:00.582686+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"29","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:00.586191+0000","last_change":"2020-06-12T02:37:27.901486+0000","last_active":"2020-06-12T02:38:00.586191+0000","last_peered":"2020-06-12T02:38:00.586191+0000","last_clean":"2020-06-12T02:38:00.586191+0000","last_became_active":"2020-06-12T02:37:27.901018+0000","last_became_peered":"2020-06-12T02:37:27.901018+0000","last_unstale":"2020-06-12T02:38:00.586191+0000","last_undegraded":"2020-06-12T02:38:00.586191+0000","last_fullsized":"2020-06-12T02:38:00.586191+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"36'732","reported_seq":"918","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:01.362569+0000","last_change":"2020-06-12T02:37:57.567683+0000","last_active":"2020-06-12T02:38:01.362569+0000","last_peered":"2020-06-12T02:38:01.362569+0000","last_clean":"2020-06-12T02:38:01.362569+0000","last_became_active":"2020-06-12T02:37:57.566865+0000","last_became_peered":"2020-06-12T02:37:57.566865+0000","last_unstale":"2020-06-12T02:38:01.362569+0000","last_undegraded":"2020-06-12T02:38:01.362569+0000","last_fullsized":"2020-06-12T02:38:01.362569+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":732,"ondisk_log_size":732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5013504,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":258,"num_read_kb":139,"num_write":570,"num_write_kb":165008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2793472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"33","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:00.586524+0000","last_change":"2020-06-12T02:37:16.763049+0000","last_active":"2020-06-12T02:38:00.586524+0000","last_peered":"2020-06-12T02:38:00.586524+0000","last_clean":"2020-06-12T02:38:00.586524+0000","last_became_active":"2020-06-12T02:37:16.762626+0000","last_became_peered":"2020-06-12T02:37:16.762626+0000","last_unstale":"2020-06-12T02:38:00.586524+0000","last_undegraded":"2020-06-12T02:38:00.586524+0000","last_fullsized":"2020-06-12T02:38:00.586524+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"45","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:01.362703+0000","last_change":"2020-06-12T02:37:54.555377+0000","last_active":"2020-06-12T02:38:01.362703+0000","last_peered":"2020-06-12T02:38:01.362703+0000","last_clean":"2020-06-12T02:38:01.362703+0000","last_became_active":"2020-06-12T02:37:54.554986+0000","last_became_peered":"2020-06-12T02:37:54.554986+0000","last_unstale":"2020-06-12T02:38:01.362703+0000","last_undegraded":"2020-06-12T02:38:01.362703+0000","last_fullsized":"2020-06-12T02:38:01.362703+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"28","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:00.981638+0000","last_change":"2020-06-12T02:37:27.906478+0000","last_active":"2020-06-12T02:38:00.981638+0000","last_peered":"2020-06-12T02:38:00.981638+0000","last_clean":"2020-06-12T02:38:00.981638+0000","last_became_active":"2020-06-12T02:37:27.905639+0000","last_became_peered":"2020-06-12T02:37:27.905639+0000","last_unstale":"2020-06-12T02:38:00.981638+0000","last_undegraded":"2020-06-12T02:38:00.981638+0000","last_fullsized":"2020-06-12T02:38:00.981638+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":13967360,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2516,"num_read_kb":1358,"num_write":5600,"num_write_kb":1598408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20951040,"data_stored":20951040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7410,"ondisk_log_size":7410,"up":48,"acting":48,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26157,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":34,"seq":146028888067,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1053532,"kb_used_data":4948,"kb_used_omap":25,"kb_used_meta":1048550,"kb_avail":93318308,"statfs":{"total":96636764160,"available":95557947392,"internally_reserved":1073741824,"allocated":5066752,"data_stored":4806767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26157,"internal_metadata":1073715667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1459999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.03}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2709999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1659999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3089999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574861,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049680,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322160,"statfs":{"total":96636764160,"available":95561891840,"internally_reserved":1073741824,"allocated":1122304,"data_stored":874451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2230000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.081}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0469999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.764}]}]},{"osd":1,"up_from":13,"seq":55834574862,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050700,"kb_used_data":2116,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321140,"statfs":{"total":96636764160,"available":95560847360,"internally_reserved":1073741824,"allocated":2166784,"data_stored":1912865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0129999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3220000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4570000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574862,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1052688,"kb_used_data":4104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93319152,"statfs":{"total":96636764160,"available":95558811648,"internally_reserved":1073741824,"allocated":4202496,"data_stored":3948577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2609999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3959999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8200000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.226}]}]},{"osd":2,"up_from":13,"seq":55834574861,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050264,"kb_used_data":1680,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321576,"statfs":{"total":96636764160,"available":95561293824,"internally_reserved":1073741824,"allocated":1720320,"data_stored":1466401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.876}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4490000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.387}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.677}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6539999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574861,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1053984,"kb_used_data":5400,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93317856,"statfs":{"total":96636764160,"available":95557484544,"internally_reserved":1073741824,"allocated":5529600,"data_stored":5281747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0169999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98599999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6180000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.103}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.45}]}]},{"osd":4,"up_from":13,"seq":55834574860,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050896,"kb_used_data":2312,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93320944,"statfs":{"total":96636764160,"available":95560646656,"internally_reserved":1073741824,"allocated":2367488,"data_stored":2113569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.712}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0029999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.617}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.141}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4330000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.407}]}]},{"osd":5,"up_from":13,"seq":55834574860,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050976,"kb_used_data":2392,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93320864,"statfs":{"total":96636764160,"available":95560564736,"internally_reserved":1073741824,"allocated":2449408,"data_stored":2205185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0760000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1850000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94699999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97599999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.012}]}]}]}} 2020-06-12T02:38:06.277 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2020-06-12T02:38:06.277 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2020-06-12T02:38:06.278 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_0 2020-06-12T02:38:06.278 INFO:tasks.ceph.ceph_manager.ceph:contract_pool unique_pool_0 by 10 min 8 2020-06-12T02:38:06.278 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json 2020-06-12T02:38:06.285 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:38:06.306 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:38:06.311 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:06.312 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:38:06.312 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2020-06-12T02:38:06.312 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:06.320 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-06-12T02:38:06.330 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:38:06.348 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2020-06-12T02:38:06.349 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-06-12T02:38:06.350 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-06-12T02:38:06.350 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-06-12T02:38:06.351 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-06-12T02:38:06.351 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-06-12T02:38:06.351 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-06-12T02:38:06.351 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-06-12T02:38:06.352 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-06-12T02:38:06.363 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:06.359+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:06.459 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:06.457+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:06.479 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:06.479 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:38:06.480 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:38:06.480 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:38:06.480 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:06.488 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-06-12T02:38:06.544 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:38:06.545 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2020-06-12T02:38:06.558 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2020-06-12T02:38:05.619155+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":13967360,"num_objects":9,"num_object_clones":0,"num_object_copies":25,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2516,"num_read_kb":1358,"num_write":5602,"num_write_kb":1598418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7412,"ondisk_log_size":7412,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":66,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8412720,"kb_used_data":24048,"kb_used_omap":25,"kb_used_meta":8388582,"kb_avail":746562000,"statfs":{"total":773094113280,"available":764479488000,"internally_reserved":8589934592,"allocated":24625152,"data_stored":22609562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26157,"internal_metadata":8589908435},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.016814"},"pg_stats":[{"pgid":"3.f","version":"36'742","reported_seq":"913","reported_epoch":"41","state":"active+clean+wait","last_fresh":"2020-06-12T02:38:00.582815+0000","last_change":"2020-06-12T02:37:57.567984+0000","last_active":"2020-06-12T02:38:00.582815+0000","last_peered":"2020-06-12T02:38:00.582815+0000","last_clean":"2020-06-12T02:38:00.582815+0000","last_became_active":"2020-06-12T02:37:57.566633+0000","last_became_peered":"2020-06-12T02:37:57.566633+0000","last_unstale":"2020-06-12T02:38:00.582815+0000","last_undegraded":"2020-06-12T02:38:00.582815+0000","last_fullsized":"2020-06-12T02:38:00.582815+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":742,"ondisk_log_size":742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1540096,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":251,"num_read_kb":128,"num_write":523,"num_write_kb":142928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1531904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"36'402","reported_seq":"518","reported_epoch":"41","state":"active+clean+wait","last_fresh":"2020-06-12T02:38:00.582735+0000","last_change":"2020-06-12T02:37:57.564534+0000","last_active":"2020-06-12T02:38:00.582735+0000","last_peered":"2020-06-12T02:38:00.582735+0000","last_clean":"2020-06-12T02:38:00.582735+0000","last_became_active":"2020-06-12T02:37:57.563869+0000","last_became_peered":"2020-06-12T02:37:57.563869+0000","last_unstale":"2020-06-12T02:38:00.582735+0000","last_undegraded":"2020-06-12T02:38:00.582735+0000","last_fullsized":"2020-06-12T02:38:00.582735+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":402,"ondisk_log_size":402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":131,"num_read_kb":71,"num_write":306,"num_write_kb":88168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1466368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"36'770","reported_seq":"969","reported_epoch":"41","state":"active+clean+wait","last_fresh":"2020-06-12T02:38:00.582525+0000","last_change":"2020-06-12T02:37:57.565160+0000","last_active":"2020-06-12T02:38:00.582525+0000","last_peered":"2020-06-12T02:38:00.582525+0000","last_clean":"2020-06-12T02:38:00.582525+0000","last_became_active":"2020-06-12T02:37:57.564464+0000","last_became_peered":"2020-06-12T02:37:57.564464+0000","last_unstale":"2020-06-12T02:38:00.582525+0000","last_undegraded":"2020-06-12T02:38:00.582525+0000","last_fullsized":"2020-06-12T02:38:00.582525+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":770,"ondisk_log_size":770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1490944,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":252,"num_read_kb":144,"num_write":593,"num_write_kb":168240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3710976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"24","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:01.362480+0000","last_change":"2020-06-12T02:37:42.384656+0000","last_active":"2020-06-12T02:38:01.362480+0000","last_peered":"2020-06-12T02:38:01.362480+0000","last_clean":"2020-06-12T02:38:01.362480+0000","last_became_active":"2020-06-12T02:37:42.384363+0000","last_became_peered":"2020-06-12T02:37:42.384363+0000","last_unstale":"2020-06-12T02:38:01.362480+0000","last_undegraded":"2020-06-12T02:38:01.362480+0000","last_fullsized":"2020-06-12T02:38:01.362480+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"36'730","reported_seq":"895","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:01.362495+0000","last_change":"2020-06-12T02:37:27.902643+0000","last_active":"2020-06-12T02:38:01.362495+0000","last_peered":"2020-06-12T02:38:01.362495+0000","last_clean":"2020-06-12T02:38:01.362495+0000","last_became_active":"2020-06-12T02:37:27.902204+0000","last_became_peered":"2020-06-12T02:37:27.902204+0000","last_unstale":"2020-06-12T02:38:01.362495+0000","last_undegraded":"2020-06-12T02:38:01.362495+0000","last_fullsized":"2020-06-12T02:38:01.362495+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":730,"ondisk_log_size":730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":242,"num_read_kb":136,"num_write":558,"num_write_kb":161104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"36'715","reported_seq":"882","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:01.362662+0000","last_change":"2020-06-12T02:37:27.905454+0000","last_active":"2020-06-12T02:38:01.362662+0000","last_peered":"2020-06-12T02:38:01.362662+0000","last_clean":"2020-06-12T02:38:01.362662+0000","last_became_active":"2020-06-12T02:37:27.904857+0000","last_became_peered":"2020-06-12T02:37:27.904857+0000","last_unstale":"2020-06-12T02:38:01.362662+0000","last_undegraded":"2020-06-12T02:38:01.362662+0000","last_fullsized":"2020-06-12T02:38:01.362662+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":715,"ondisk_log_size":715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1343488,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":256,"num_read_kb":139,"num_write":553,"num_write_kb":155176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"36'689","reported_seq":"863","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:01.235836+0000","last_change":"2020-06-12T02:37:57.567142+0000","last_active":"2020-06-12T02:38:01.235836+0000","last_peered":"2020-06-12T02:38:01.235836+0000","last_clean":"2020-06-12T02:38:01.235836+0000","last_became_active":"2020-06-12T02:37:57.566462+0000","last_became_peered":"2020-06-12T02:37:57.566462+0000","last_unstale":"2020-06-12T02:38:01.235836+0000","last_undegraded":"2020-06-12T02:38:01.235836+0000","last_fullsized":"2020-06-12T02:38:01.235836+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":689,"ondisk_log_size":689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":226,"num_read_kb":124,"num_write":487,"num_write_kb":134808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"36'425","reported_seq":"552","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:01.362465+0000","last_change":"2020-06-12T02:37:57.567506+0000","last_active":"2020-06-12T02:38:01.362465+0000","last_peered":"2020-06-12T02:38:01.362465+0000","last_clean":"2020-06-12T02:38:01.362465+0000","last_became_active":"2020-06-12T02:37:57.566796+0000","last_became_peered":"2020-06-12T02:37:57.566796+0000","last_unstale":"2020-06-12T02:38:01.362465+0000","last_undegraded":"2020-06-12T02:38:01.362465+0000","last_fullsized":"2020-06-12T02:38:01.362465+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":425,"ondisk_log_size":425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1138688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":147,"num_read_kb":78,"num_write":334,"num_write_kb":97824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2793472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"33","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:00.586306+0000","last_change":"2020-06-12T02:37:16.868466+0000","last_active":"2020-06-12T02:38:00.586306+0000","last_peered":"2020-06-12T02:38:00.586306+0000","last_clean":"2020-06-12T02:38:00.586306+0000","last_became_active":"2020-06-12T02:37:16.868098+0000","last_became_peered":"2020-06-12T02:37:16.868098+0000","last_unstale":"2020-06-12T02:38:00.586306+0000","last_undegraded":"2020-06-12T02:38:00.586306+0000","last_fullsized":"2020-06-12T02:38:00.586306+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"36'720","reported_seq":"877","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:01.362556+0000","last_change":"2020-06-12T02:37:42.379294+0000","last_active":"2020-06-12T02:38:01.362556+0000","last_peered":"2020-06-12T02:38:01.362556+0000","last_clean":"2020-06-12T02:38:01.362556+0000","last_became_active":"2020-06-12T02:37:42.378796+0000","last_became_peered":"2020-06-12T02:37:42.378796+0000","last_unstale":"2020-06-12T02:38:01.362556+0000","last_undegraded":"2020-06-12T02:38:01.362556+0000","last_fullsized":"2020-06-12T02:38:01.362556+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":720,"ondisk_log_size":720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2514944,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":237,"num_read_kb":127,"num_write":533,"num_write_kb":155776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"23","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:00.586062+0000","last_change":"2020-06-12T02:37:42.373838+0000","last_active":"2020-06-12T02:38:00.586062+0000","last_peered":"2020-06-12T02:38:00.586062+0000","last_clean":"2020-06-12T02:38:00.586062+0000","last_became_active":"2020-06-12T02:37:42.373205+0000","last_became_peered":"2020-06-12T02:37:42.373205+0000","last_unstale":"2020-06-12T02:38:00.586062+0000","last_undegraded":"2020-06-12T02:38:00.586062+0000","last_fullsized":"2020-06-12T02:38:00.586062+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"36'769","reported_seq":"933","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:01.235834+0000","last_change":"2020-06-12T02:37:27.903386+0000","last_active":"2020-06-12T02:38:01.235834+0000","last_peered":"2020-06-12T02:38:01.235834+0000","last_clean":"2020-06-12T02:38:01.235834+0000","last_became_active":"2020-06-12T02:37:27.902910+0000","last_became_peered":"2020-06-12T02:37:27.902910+0000","last_unstale":"2020-06-12T02:38:01.235834+0000","last_undegraded":"2020-06-12T02:38:01.235834+0000","last_fullsized":"2020-06-12T02:38:01.235834+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":769,"ondisk_log_size":769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":925696,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":256,"num_read_kb":135,"num_write":582,"num_write_kb":169744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"33","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:00.586974+0000","last_change":"2020-06-12T02:37:16.765460+0000","last_active":"2020-06-12T02:38:00.586974+0000","last_peered":"2020-06-12T02:38:00.586974+0000","last_clean":"2020-06-12T02:38:00.586974+0000","last_became_active":"2020-06-12T02:37:16.764958+0000","last_became_peered":"2020-06-12T02:37:16.764958+0000","last_unstale":"2020-06-12T02:38:00.586974+0000","last_undegraded":"2020-06-12T02:38:00.586974+0000","last_fullsized":"2020-06-12T02:38:00.586974+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"36'401","reported_seq":"505","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:00.586077+0000","last_change":"2020-06-12T02:37:27.905010+0000","last_active":"2020-06-12T02:38:00.586077+0000","last_peered":"2020-06-12T02:38:00.586077+0000","last_clean":"2020-06-12T02:38:00.586077+0000","last_became_active":"2020-06-12T02:37:27.904552+0000","last_became_peered":"2020-06-12T02:37:27.904552+0000","last_unstale":"2020-06-12T02:38:00.586077+0000","last_undegraded":"2020-06-12T02:38:00.586077+0000","last_fullsized":"2020-06-12T02:38:00.586077+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":401,"ondisk_log_size":401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":144,"num_read_kb":75,"num_write":313,"num_write_kb":90136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"23","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:00.586919+0000","last_change":"2020-06-12T02:37:42.376260+0000","last_active":"2020-06-12T02:38:00.586919+0000","last_peered":"2020-06-12T02:38:00.586919+0000","last_clean":"2020-06-12T02:38:00.586919+0000","last_became_active":"2020-06-12T02:37:42.375902+0000","last_became_peered":"2020-06-12T02:37:42.375902+0000","last_unstale":"2020-06-12T02:38:00.586919+0000","last_undegraded":"2020-06-12T02:38:00.586919+0000","last_fullsized":"2020-06-12T02:38:00.586919+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"15'2","reported_seq":"30","reported_epoch":"41","state":"active+clean+wait","last_fresh":"2020-06-12T02:38:00.582577+0000","last_change":"2020-06-12T02:37:54.558799+0000","last_active":"2020-06-12T02:38:00.582577+0000","last_peered":"2020-06-12T02:38:00.582577+0000","last_clean":"2020-06-12T02:38:00.582577+0000","last_became_active":"2020-06-12T02:37:54.558125+0000","last_became_peered":"2020-06-12T02:37:54.558125+0000","last_unstale":"2020-06-12T02:38:00.582577+0000","last_undegraded":"2020-06-12T02:38:00.582577+0000","last_fullsized":"2020-06-12T02:38:00.582577+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":3,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:04.419204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:04.419204+0000","last_clean_scrub_stamp":"2020-06-12T02:37:04.419204+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"36'315","reported_seq":"406","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:01.362493+0000","last_change":"2020-06-12T02:37:27.898985+0000","last_active":"2020-06-12T02:38:01.362493+0000","last_peered":"2020-06-12T02:38:01.362493+0000","last_clean":"2020-06-12T02:38:01.362493+0000","last_became_active":"2020-06-12T02:37:27.898516+0000","last_became_peered":"2020-06-12T02:37:27.898516+0000","last_unstale":"2020-06-12T02:38:01.362493+0000","last_undegraded":"2020-06-12T02:38:01.362493+0000","last_fullsized":"2020-06-12T02:38:01.362493+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":315,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":116,"num_read_kb":62,"num_write":248,"num_write_kb":69496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"27","reported_epoch":"41","state":"active+clean+wait","last_fresh":"2020-06-12T02:38:00.582446+0000","last_change":"2020-06-12T02:37:54.558450+0000","last_active":"2020-06-12T02:38:00.582446+0000","last_peered":"2020-06-12T02:38:00.582446+0000","last_clean":"2020-06-12T02:38:00.582446+0000","last_became_active":"2020-06-12T02:37:54.557850+0000","last_became_peered":"2020-06-12T02:37:54.557850+0000","last_unstale":"2020-06-12T02:38:00.582446+0000","last_undegraded":"2020-06-12T02:38:00.582446+0000","last_fullsized":"2020-06-12T02:38:00.582446+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"28","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:00.981519+0000","last_change":"2020-06-12T02:37:27.901368+0000","last_active":"2020-06-12T02:38:00.981519+0000","last_peered":"2020-06-12T02:38:00.981519+0000","last_clean":"2020-06-12T02:38:00.981519+0000","last_became_active":"2020-06-12T02:37:27.900888+0000","last_became_peered":"2020-06-12T02:37:27.900888+0000","last_unstale":"2020-06-12T02:38:00.981519+0000","last_undegraded":"2020-06-12T02:38:00.981519+0000","last_fullsized":"2020-06-12T02:38:00.981519+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"27","reported_epoch":"41","state":"active+clean+wait","last_fresh":"2020-06-12T02:38:00.582686+0000","last_change":"2020-06-12T02:37:54.557777+0000","last_active":"2020-06-12T02:38:00.582686+0000","last_peered":"2020-06-12T02:38:00.582686+0000","last_clean":"2020-06-12T02:38:00.582686+0000","last_became_active":"2020-06-12T02:37:54.557420+0000","last_became_peered":"2020-06-12T02:37:54.557420+0000","last_unstale":"2020-06-12T02:38:00.582686+0000","last_undegraded":"2020-06-12T02:38:00.582686+0000","last_fullsized":"2020-06-12T02:38:00.582686+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"29","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:00.586191+0000","last_change":"2020-06-12T02:37:27.901486+0000","last_active":"2020-06-12T02:38:00.586191+0000","last_peered":"2020-06-12T02:38:00.586191+0000","last_clean":"2020-06-12T02:38:00.586191+0000","last_became_active":"2020-06-12T02:37:27.901018+0000","last_became_peered":"2020-06-12T02:37:27.901018+0000","last_unstale":"2020-06-12T02:38:00.586191+0000","last_undegraded":"2020-06-12T02:38:00.586191+0000","last_fullsized":"2020-06-12T02:38:00.586191+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"36'732","reported_seq":"918","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:01.362569+0000","last_change":"2020-06-12T02:37:57.567683+0000","last_active":"2020-06-12T02:38:01.362569+0000","last_peered":"2020-06-12T02:38:01.362569+0000","last_clean":"2020-06-12T02:38:01.362569+0000","last_became_active":"2020-06-12T02:37:57.566865+0000","last_became_peered":"2020-06-12T02:37:57.566865+0000","last_unstale":"2020-06-12T02:38:01.362569+0000","last_undegraded":"2020-06-12T02:38:01.362569+0000","last_fullsized":"2020-06-12T02:38:01.362569+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":732,"ondisk_log_size":732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5013504,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":258,"num_read_kb":139,"num_write":570,"num_write_kb":165008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2793472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"33","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:00.586524+0000","last_change":"2020-06-12T02:37:16.763049+0000","last_active":"2020-06-12T02:38:00.586524+0000","last_peered":"2020-06-12T02:38:00.586524+0000","last_clean":"2020-06-12T02:38:00.586524+0000","last_became_active":"2020-06-12T02:37:16.762626+0000","last_became_peered":"2020-06-12T02:37:16.762626+0000","last_unstale":"2020-06-12T02:38:00.586524+0000","last_undegraded":"2020-06-12T02:38:00.586524+0000","last_fullsized":"2020-06-12T02:38:00.586524+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"45","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:01.362703+0000","last_change":"2020-06-12T02:37:54.555377+0000","last_active":"2020-06-12T02:38:01.362703+0000","last_peered":"2020-06-12T02:38:01.362703+0000","last_clean":"2020-06-12T02:38:01.362703+0000","last_became_active":"2020-06-12T02:37:54.554986+0000","last_became_peered":"2020-06-12T02:37:54.554986+0000","last_unstale":"2020-06-12T02:38:01.362703+0000","last_undegraded":"2020-06-12T02:38:01.362703+0000","last_fullsized":"2020-06-12T02:38:01.362703+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"28","reported_epoch":"41","state":"active+clean","last_fresh":"2020-06-12T02:38:00.981638+0000","last_change":"2020-06-12T02:37:27.906478+0000","last_active":"2020-06-12T02:38:00.981638+0000","last_peered":"2020-06-12T02:38:00.981638+0000","last_clean":"2020-06-12T02:38:00.981638+0000","last_became_active":"2020-06-12T02:37:27.905639+0000","last_became_peered":"2020-06-12T02:37:27.905639+0000","last_unstale":"2020-06-12T02:38:00.981638+0000","last_undegraded":"2020-06-12T02:38:00.981638+0000","last_fullsized":"2020-06-12T02:38:00.981638+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":13967360,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2516,"num_read_kb":1358,"num_write":5600,"num_write_kb":1598408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":12296192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20951040,"data_stored":20951040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7410,"ondisk_log_size":7410,"up":48,"acting":48,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26157,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":34,"seq":146028888067,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1053532,"kb_used_data":4948,"kb_used_omap":25,"kb_used_meta":1048550,"kb_avail":93318308,"statfs":{"total":96636764160,"available":95557947392,"internally_reserved":1073741824,"allocated":5066752,"data_stored":4806767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26157,"internal_metadata":1073715667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1459999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.03}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2709999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1659999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3089999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574861,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049680,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322160,"statfs":{"total":96636764160,"available":95561891840,"internally_reserved":1073741824,"allocated":1122304,"data_stored":874451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2230000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.081}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0469999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.764}]}]},{"osd":1,"up_from":13,"seq":55834574862,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050700,"kb_used_data":2116,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321140,"statfs":{"total":96636764160,"available":95560847360,"internally_reserved":1073741824,"allocated":2166784,"data_stored":1912865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0129999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3220000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4570000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574862,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1052688,"kb_used_data":4104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93319152,"statfs":{"total":96636764160,"available":95558811648,"internally_reserved":1073741824,"allocated":4202496,"data_stored":3948577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2609999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3959999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8200000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.226}]}]},{"osd":2,"up_from":13,"seq":55834574861,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050264,"kb_used_data":1680,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321576,"statfs":{"total":96636764160,"available":95561293824,"internally_reserved":1073741824,"allocated":1720320,"data_stored":1466401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.876}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4490000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.387}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.677}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6539999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574861,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1053984,"kb_used_data":5400,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93317856,"statfs":{"total":96636764160,"available":95557484544,"internally_reserved":1073741824,"allocated":5529600,"data_stored":5281747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0169999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98599999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6180000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.103}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.45}]}]},{"osd":4,"up_from":13,"seq":55834574860,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050896,"kb_used_data":2312,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93320944,"statfs":{"total":96636764160,"available":95560646656,"internally_reserved":1073741824,"allocated":2367488,"data_stored":2113569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.712}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0029999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.617}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.141}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4330000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.407}]}]},{"osd":5,"up_from":13,"seq":55834574860,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050976,"kb_used_data":2392,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93320864,"statfs":{"total":96636764160,"available":95560564736,"internally_reserved":1073741824,"allocated":2449408,"data_stored":2205185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0760000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1850000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94699999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97599999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.012}]}]}]}} 2020-06-12T02:38:06.561 INFO:tasks.ceph.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2020-06-12T02:38:06.596 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:06.594+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:06.609 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:06.609 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:38:06.610 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:38:06.610 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:38:06.610 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.611 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:858 3.3s0 3:ce45c70a:::smithi10729407-7:head [append 0~565248] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:38:06.611 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.924643+0000", 2020-06-12T02:38:06.611 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.680758634999997, 2020-06-12T02:38:06.611 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015332551999999999, 2020-06-12T02:38:06.611 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:06.612 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:06.612 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:06.612 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:06.612 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:06.612 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 858 2020-06-12T02:38:06.613 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.613 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:06.613 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.613 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:06.613 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.924643+0000", 2020-06-12T02:38:06.614 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:06.614 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.614 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.614 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:06.615 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.924643+0000", 2020-06-12T02:38:06.615 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00067535299999999996 2020-06-12T02:38:06.615 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.615 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.615 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:06.616 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925318+0000", 2020-06-12T02:38:06.616 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9629999999999993e-06 2020-06-12T02:38:06.616 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.616 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.616 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:06.617 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925328+0000", 2020-06-12T02:38:06.617 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7357999999999999e-05 2020-06-12T02:38:06.617 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.617 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.617 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:06.618 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925345+0000", 2020-06-12T02:38:06.618 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7019999999999996e-06 2020-06-12T02:38:06.618 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.619 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.619 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:06.619 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925355+0000", 2020-06-12T02:38:06.619 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6289999999999998e-05 2020-06-12T02:38:06.620 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.620 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.620 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.621 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925411+0000", 2020-06-12T02:38:06.621 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8855000000000004e-05 2020-06-12T02:38:06.621 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.621 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.622 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:06.622 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925480+0000", 2020-06-12T02:38:06.622 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0074521149999999996 2020-06-12T02:38:06.622 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.623 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.623 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.623 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.932932+0000", 2020-06-12T02:38:06.623 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3647999999999995e-05 2020-06-12T02:38:06.623 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.624 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.624 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:06.624 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.933026+0000", 2020-06-12T02:38:06.625 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00078753199999999997 2020-06-12T02:38:06.625 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.626 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.626 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:06.626 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.933813+0000", 2020-06-12T02:38:06.626 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.006013582 2020-06-12T02:38:06.627 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.627 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.627 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:06.627 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.939827+0000", 2020-06-12T02:38:06.627 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0111999999999996e-05 2020-06-12T02:38:06.628 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.628 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.628 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:06.628 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.939897+0000", 2020-06-12T02:38:06.628 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8041999999999996e-05 2020-06-12T02:38:06.629 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.629 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.629 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:06.629 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.939975+0000", 2020-06-12T02:38:06.629 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015332551999999999 2020-06-12T02:38:06.630 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:06.630 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:06.630 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:06.630 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.630 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.631 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:860 3.3s0 3:ce45c70a:::smithi10729407-7:head [append 0~81920] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:38:06.631 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.928619+0000", 2020-06-12T02:38:06.631 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.676782748999997, 2020-06-12T02:38:06.631 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017431941999999999, 2020-06-12T02:38:06.631 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:06.631 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:06.632 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:06.632 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:06.632 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:06.632 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 860 2020-06-12T02:38:06.632 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.633 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:06.633 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.633 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:06.633 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928619+0000", 2020-06-12T02:38:06.633 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:06.634 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.634 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.634 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:06.634 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928619+0000", 2020-06-12T02:38:06.634 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000105809 2020-06-12T02:38:06.635 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.635 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.635 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:06.635 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928724+0000", 2020-06-12T02:38:06.635 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2339999999999994e-06 2020-06-12T02:38:06.636 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.636 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.636 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:06.636 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928733+0000", 2020-06-12T02:38:06.636 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0268000000000001e-05 2020-06-12T02:38:06.637 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.637 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.637 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:06.637 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928743+0000", 2020-06-12T02:38:06.637 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5190000000000003e-06 2020-06-12T02:38:06.638 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.638 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.638 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:06.638 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928747+0000", 2020-06-12T02:38:06.638 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0103999999999998e-05 2020-06-12T02:38:06.639 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.639 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.639 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.639 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928777+0000", 2020-06-12T02:38:06.639 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8249000000000002e-05 2020-06-12T02:38:06.640 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.640 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.640 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:06.640 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928866+0000", 2020-06-12T02:38:06.640 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011151629 2020-06-12T02:38:06.640 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.641 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.641 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.641 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.940017+0000", 2020-06-12T02:38:06.641 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8281e-05 2020-06-12T02:38:06.641 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.642 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.642 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:06.642 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.940076+0000", 2020-06-12T02:38:06.642 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000346736 2020-06-12T02:38:06.642 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.642 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.643 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:06.643 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.940422+0000", 2020-06-12T02:38:06.643 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024135049999999998 2020-06-12T02:38:06.643 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.643 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.644 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:06.644 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.942836+0000", 2020-06-12T02:38:06.644 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003170258 2020-06-12T02:38:06.644 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.644 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.644 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:06.645 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946006+0000", 2020-06-12T02:38:06.645 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4350000000000001e-05 2020-06-12T02:38:06.645 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.645 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.645 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:06.646 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946050+0000", 2020-06-12T02:38:06.646 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017431941999999999 2020-06-12T02:38:06.646 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:06.646 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:06.646 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:06.647 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.647 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.647 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:861 3.3s0 3:ce45c70a:::smithi10729407-7:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:38:06.647 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.928778+0000", 2020-06-12T02:38:06.647 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.676623136000003, 2020-06-12T02:38:06.647 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017522738, 2020-06-12T02:38:06.648 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:06.648 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:06.648 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:06.648 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:06.648 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:06.649 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 861 2020-06-12T02:38:06.649 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.649 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:06.649 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.649 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:06.649 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928778+0000", 2020-06-12T02:38:06.650 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:06.650 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.650 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.650 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:06.650 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928778+0000", 2020-06-12T02:38:06.651 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8186000000000002e-05 2020-06-12T02:38:06.651 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.651 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.651 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:06.651 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928826+0000", 2020-06-12T02:38:06.652 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6689999999999997e-06 2020-06-12T02:38:06.652 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.652 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.652 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:06.652 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928831+0000", 2020-06-12T02:38:06.652 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.686e-06 2020-06-12T02:38:06.653 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.653 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.653 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:06.653 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928838+0000", 2020-06-12T02:38:06.653 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7159999999999999e-06 2020-06-12T02:38:06.654 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.654 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.654 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:06.654 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928840+0000", 2020-06-12T02:38:06.654 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4212999999999997e-05 2020-06-12T02:38:06.654 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.655 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.655 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.655 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928895+0000", 2020-06-12T02:38:06.655 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5853999999999994e-05 2020-06-12T02:38:06.655 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.656 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.656 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:06.656 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928990+0000", 2020-06-12T02:38:06.656 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012023982000000001 2020-06-12T02:38:06.656 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.656 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.657 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.657 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941014+0000", 2020-06-12T02:38:06.657 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5104e-05 2020-06-12T02:38:06.657 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.657 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.658 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:06.658 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941070+0000", 2020-06-12T02:38:06.658 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00026444499999999998 2020-06-12T02:38:06.658 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.658 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.659 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:06.659 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941334+0000", 2020-06-12T02:38:06.659 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001704075 2020-06-12T02:38:06.659 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.659 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.659 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:06.660 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.943038+0000", 2020-06-12T02:38:06.660 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031891570000000002 2020-06-12T02:38:06.660 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.660 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.660 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:06.661 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946227+0000", 2020-06-12T02:38:06.661 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3651000000000003e-05 2020-06-12T02:38:06.661 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.661 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.661 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:06.661 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946301+0000", 2020-06-12T02:38:06.662 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017522738 2020-06-12T02:38:06.662 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:06.662 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:06.662 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:06.662 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.663 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.663 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:862 3.3s0 3:ce45c70a:::smithi10729407-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:38:06.663 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.928876+0000", 2020-06-12T02:38:06.663 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.676525640000001, 2020-06-12T02:38:06.663 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019206890000000001, 2020-06-12T02:38:06.664 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:06.664 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:06.664 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:06.664 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:06.664 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:06.664 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 862 2020-06-12T02:38:06.665 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.665 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:06.665 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.665 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:06.665 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928876+0000", 2020-06-12T02:38:06.666 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:06.666 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.666 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.666 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:06.666 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928876+0000", 2020-06-12T02:38:06.666 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9268999999999999e-05 2020-06-12T02:38:06.667 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.667 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.667 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:06.667 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928925+0000", 2020-06-12T02:38:06.667 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8120000000000002e-06 2020-06-12T02:38:06.668 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.668 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.668 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:06.668 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928932+0000", 2020-06-12T02:38:06.668 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3471e-05 2020-06-12T02:38:06.668 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.669 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.669 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:06.669 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928945+0000", 2020-06-12T02:38:06.669 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4279999999999998e-06 2020-06-12T02:38:06.669 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.670 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.670 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:06.670 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928950+0000", 2020-06-12T02:38:06.670 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6358000000000004e-05 2020-06-12T02:38:06.670 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.670 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.671 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.671 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.929016+0000", 2020-06-12T02:38:06.671 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9697999999999996e-05 2020-06-12T02:38:06.671 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.671 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.672 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:06.672 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.929086+0000", 2020-06-12T02:38:06.672 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012409205 2020-06-12T02:38:06.672 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.672 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.672 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.673 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941495+0000", 2020-06-12T02:38:06.673 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9660000000000006e-05 2020-06-12T02:38:06.673 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.673 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.673 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:06.674 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941585+0000", 2020-06-12T02:38:06.674 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050329099999999998 2020-06-12T02:38:06.674 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.674 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.674 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.674 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946617+0000", 2020-06-12T02:38:06.675 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8437999999999996e-05 2020-06-12T02:38:06.675 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.675 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.675 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:06.675 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946706+0000", 2020-06-12T02:38:06.676 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0013766410000000001 2020-06-12T02:38:06.676 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.676 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.676 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:06.676 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.948083+0000", 2020-06-12T02:38:06.676 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019206890000000001 2020-06-12T02:38:06.677 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:06.677 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:06.677 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:06.677 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.677 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.678 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:879 3.fs1 3:fc7e0bf7:::smithi10729407-17:head [append 0~491520] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:38:06.678 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.000881+0000", 2020-06-12T02:38:06.678 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.604520459, 2020-06-12T02:38:06.678 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017891845, 2020-06-12T02:38:06.678 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:06.679 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:06.679 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:06.679 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:06.679 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:06.679 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 879 2020-06-12T02:38:06.679 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.680 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:06.680 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.680 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:06.680 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.000881+0000", 2020-06-12T02:38:06.680 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:06.681 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.681 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.681 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:06.681 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.000881+0000", 2020-06-12T02:38:06.681 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00044151599999999998 2020-06-12T02:38:06.681 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.682 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.682 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:06.682 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001322+0000", 2020-06-12T02:38:06.682 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0210000000000001e-06 2020-06-12T02:38:06.682 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.683 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.683 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:06.683 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001326+0000", 2020-06-12T02:38:06.683 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6659999999999997e-06 2020-06-12T02:38:06.683 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.683 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.684 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:06.684 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001335+0000", 2020-06-12T02:38:06.684 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4299999999999999e-06 2020-06-12T02:38:06.684 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.684 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.685 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:06.685 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001339+0000", 2020-06-12T02:38:06.685 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0028741510000000001 2020-06-12T02:38:06.685 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.685 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.686 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.686 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.004214+0000", 2020-06-12T02:38:06.686 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010480999999999999 2020-06-12T02:38:06.686 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.686 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.686 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:06.687 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.004318+0000", 2020-06-12T02:38:06.687 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021112359999999998 2020-06-12T02:38:06.687 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.687 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.687 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.688 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006430+0000", 2020-06-12T02:38:06.688 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2935000000000001e-05 2020-06-12T02:38:06.688 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.688 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.688 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:06.688 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006483+0000", 2020-06-12T02:38:06.689 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00056411099999999995 2020-06-12T02:38:06.689 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.689 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.689 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:06.689 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.007047+0000", 2020-06-12T02:38:06.690 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0076749369999999997 2020-06-12T02:38:06.690 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.690 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.690 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:06.690 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.014722+0000", 2020-06-12T02:38:06.690 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00395693 2020-06-12T02:38:06.691 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.691 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.691 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:06.691 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.018679+0000", 2020-06-12T02:38:06.691 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4102000000000001e-05 2020-06-12T02:38:06.692 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.692 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.692 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:06.692 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.018773+0000", 2020-06-12T02:38:06.692 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017891845 2020-06-12T02:38:06.693 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:06.693 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:06.693 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:06.694 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.694 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.694 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:933 3.es1 3:78f01648:::smithi10729407-5:head [append 0~417792] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:38:06.694 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.156677+0000", 2020-06-12T02:38:06.694 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.448724476999999, 2020-06-12T02:38:06.695 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016833311, 2020-06-12T02:38:06.695 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:06.695 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:06.695 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:06.695 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:06.695 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:06.696 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 933 2020-06-12T02:38:06.696 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.696 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:06.696 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.696 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:06.697 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.156677+0000", 2020-06-12T02:38:06.697 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:06.697 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.697 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.697 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:06.698 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.156677+0000", 2020-06-12T02:38:06.698 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00062571700000000003 2020-06-12T02:38:06.698 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.698 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.698 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:06.698 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157303+0000", 2020-06-12T02:38:06.699 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0740000000000003e-06 2020-06-12T02:38:06.699 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.699 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.699 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:06.699 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157307+0000", 2020-06-12T02:38:06.700 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4419999999999998e-06 2020-06-12T02:38:06.700 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.700 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.700 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:06.700 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157315+0000", 2020-06-12T02:38:06.700 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5300000000000001e-06 2020-06-12T02:38:06.701 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.701 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.701 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:06.701 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157319+0000", 2020-06-12T02:38:06.701 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4995e-05 2020-06-12T02:38:06.702 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.702 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.702 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.702 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157354+0000", 2020-06-12T02:38:06.702 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5883000000000006e-05 2020-06-12T02:38:06.702 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.703 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.703 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:06.703 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157419+0000", 2020-06-12T02:38:06.703 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045316280000000002 2020-06-12T02:38:06.703 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.704 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.704 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.704 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.161951+0000", 2020-06-12T02:38:06.704 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3300999999999996e-05 2020-06-12T02:38:06.704 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.704 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.705 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:06.705 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.162014+0000", 2020-06-12T02:38:06.705 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00039050100000000002 2020-06-12T02:38:06.705 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.705 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.706 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:06.706 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.162405+0000", 2020-06-12T02:38:06.706 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050686250000000002 2020-06-12T02:38:06.706 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.706 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.706 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:06.707 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.167474+0000", 2020-06-12T02:38:06.707 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.005942653 2020-06-12T02:38:06.707 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.707 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.707 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:06.708 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.173416+0000", 2020-06-12T02:38:06.708 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3962000000000006e-05 2020-06-12T02:38:06.708 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.708 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.708 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:06.708 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.173510+0000", 2020-06-12T02:38:06.709 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016833311 2020-06-12T02:38:06.709 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:06.709 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:06.709 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:06.709 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.710 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.710 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:936 3.es1 3:78f01648:::smithi10729407-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:38:06.710 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.160357+0000", 2020-06-12T02:38:06.710 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.445044766000002, 2020-06-12T02:38:06.710 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015435590000000001, 2020-06-12T02:38:06.710 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:06.711 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:06.711 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:06.711 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:06.711 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:06.711 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 936 2020-06-12T02:38:06.712 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.712 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:06.712 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.712 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:06.712 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160357+0000", 2020-06-12T02:38:06.712 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:06.713 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.713 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.713 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:06.713 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160357+0000", 2020-06-12T02:38:06.713 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4435999999999999e-05 2020-06-12T02:38:06.714 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.714 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.714 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:06.714 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160411+0000", 2020-06-12T02:38:06.714 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2329999999999995e-06 2020-06-12T02:38:06.715 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.715 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.715 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:06.715 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160420+0000", 2020-06-12T02:38:06.715 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7003e-05 2020-06-12T02:38:06.715 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.716 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.716 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:06.716 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160437+0000", 2020-06-12T02:38:06.716 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8370000000000002e-06 2020-06-12T02:38:06.716 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.716 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.717 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:06.717 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160443+0000", 2020-06-12T02:38:06.717 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0070635489999999997 2020-06-12T02:38:06.717 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.717 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.718 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.718 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.167507+0000", 2020-06-12T02:38:06.718 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010589700000000001 2020-06-12T02:38:06.718 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.718 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.718 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:06.719 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.167612+0000", 2020-06-12T02:38:06.719 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0065032299999999996 2020-06-12T02:38:06.719 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.719 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.719 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.720 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.174116+0000", 2020-06-12T02:38:06.720 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3128999999999999e-05 2020-06-12T02:38:06.720 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.720 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.720 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:06.720 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.174169+0000", 2020-06-12T02:38:06.721 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0016232760000000001 2020-06-12T02:38:06.721 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.721 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.721 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:06.721 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.175792+0000", 2020-06-12T02:38:06.722 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015435590000000001 2020-06-12T02:38:06.722 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:06.722 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:06.722 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:06.722 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.723 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.723 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:950 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~778240] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:38:06.723 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.207643+0000", 2020-06-12T02:38:06.723 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.397758183000001, 2020-06-12T02:38:06.723 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014844661, 2020-06-12T02:38:06.723 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:06.724 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:06.724 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:06.724 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:06.726 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:06.726 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 950 2020-06-12T02:38:06.727 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.727 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:06.727 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.727 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:06.728 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.207643+0000", 2020-06-12T02:38:06.728 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:06.728 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.728 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.729 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:06.729 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.207643+0000", 2020-06-12T02:38:06.729 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00072843099999999998 2020-06-12T02:38:06.729 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.730 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.730 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:06.730 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208372+0000", 2020-06-12T02:38:06.730 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9120000000000001e-06 2020-06-12T02:38:06.731 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.731 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.731 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:06.731 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208378+0000", 2020-06-12T02:38:06.732 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9660000000000008e-06 2020-06-12T02:38:06.732 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.732 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.732 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:06.732 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208388+0000", 2020-06-12T02:38:06.733 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1279999999999999e-06 2020-06-12T02:38:06.733 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.733 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.733 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:06.734 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208394+0000", 2020-06-12T02:38:06.735 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0439999999999998e-05 2020-06-12T02:38:06.735 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.736 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.736 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.736 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208444+0000", 2020-06-12T02:38:06.737 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2611000000000001e-05 2020-06-12T02:38:06.737 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.738 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.738 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:06.738 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208507+0000", 2020-06-12T02:38:06.738 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001228301 2020-06-12T02:38:06.739 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.739 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.739 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.739 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.209735+0000", 2020-06-12T02:38:06.740 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8875000000000001e-05 2020-06-12T02:38:06.740 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.740 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.740 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:06.741 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.209794+0000", 2020-06-12T02:38:06.741 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053218899999999997 2020-06-12T02:38:06.741 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.741 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.742 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:06.742 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.210326+0000", 2020-06-12T02:38:06.742 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0078198439999999994 2020-06-12T02:38:06.743 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.743 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.744 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:06.744 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.218146+0000", 2020-06-12T02:38:06.744 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004300496 2020-06-12T02:38:06.745 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.745 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.745 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:06.745 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.222446+0000", 2020-06-12T02:38:06.746 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1467999999999997e-05 2020-06-12T02:38:06.746 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.746 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.746 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:06.746 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.222488+0000", 2020-06-12T02:38:06.747 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014844661 2020-06-12T02:38:06.747 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:06.747 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:06.747 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:06.748 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.748 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.748 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:954 3.3s0 3:c2ca77da:::smithi10729407-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:38:06.748 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.212923+0000", 2020-06-12T02:38:06.748 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.392478406999999, 2020-06-12T02:38:06.749 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015411546999999999, 2020-06-12T02:38:06.749 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:06.749 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:06.749 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:06.750 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:06.750 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:06.750 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 954 2020-06-12T02:38:06.750 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.750 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:06.751 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.751 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:06.751 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212923+0000", 2020-06-12T02:38:06.751 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:06.752 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.752 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.752 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:06.753 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212923+0000", 2020-06-12T02:38:06.753 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3598e-05 2020-06-12T02:38:06.754 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.754 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.754 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:06.754 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212946+0000", 2020-06-12T02:38:06.755 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.061e-06 2020-06-12T02:38:06.755 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.755 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.755 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:06.756 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212950+0000", 2020-06-12T02:38:06.756 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8700000000000003e-06 2020-06-12T02:38:06.757 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.757 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.757 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:06.758 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212956+0000", 2020-06-12T02:38:06.758 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0770000000000002e-06 2020-06-12T02:38:06.758 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.758 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.758 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:06.759 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212958+0000", 2020-06-12T02:38:06.759 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051235170000000002 2020-06-12T02:38:06.759 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.759 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.759 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.760 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.218082+0000", 2020-06-12T02:38:06.760 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9246999999999997e-05 2020-06-12T02:38:06.760 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.762 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.763 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:06.763 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.218141+0000", 2020-06-12T02:38:06.763 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0087981910000000003 2020-06-12T02:38:06.764 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.764 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.764 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.764 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.226939+0000", 2020-06-12T02:38:06.764 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4119999999999997e-05 2020-06-12T02:38:06.765 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.765 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.765 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:06.765 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.226994+0000", 2020-06-12T02:38:06.765 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001340866 2020-06-12T02:38:06.765 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.766 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.766 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:06.766 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.228334+0000", 2020-06-12T02:38:06.766 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015411546999999999 2020-06-12T02:38:06.766 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:06.767 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:06.767 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:06.767 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.767 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.767 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1097 3.es1 3:78f01648:::smithi10729407-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:38:06.767 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.576496+0000", 2020-06-12T02:38:06.768 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.028904924999999, 2020-06-12T02:38:06.768 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014992914, 2020-06-12T02:38:06.768 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:06.768 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:06.768 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:06.769 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:06.769 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:06.769 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1097 2020-06-12T02:38:06.769 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.769 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:06.770 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.770 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:06.770 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576496+0000", 2020-06-12T02:38:06.770 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:06.770 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.771 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.771 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:06.771 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576496+0000", 2020-06-12T02:38:06.771 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3221999999999999e-05 2020-06-12T02:38:06.771 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.771 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.772 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:06.772 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576520+0000", 2020-06-12T02:38:06.772 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.988e-06 2020-06-12T02:38:06.772 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.773 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.773 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:06.773 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576523+0000", 2020-06-12T02:38:06.773 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2940000000000002e-06 2020-06-12T02:38:06.774 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.774 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.774 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:06.774 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576529+0000", 2020-06-12T02:38:06.774 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6060000000000002e-06 2020-06-12T02:38:06.774 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.775 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.775 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:06.775 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576534+0000", 2020-06-12T02:38:06.775 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001254554 2020-06-12T02:38:06.775 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.776 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.776 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.776 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.577789+0000", 2020-06-12T02:38:06.776 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3177999999999998e-05 2020-06-12T02:38:06.776 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.776 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.777 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:06.777 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.577842+0000", 2020-06-12T02:38:06.777 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0063091220000000003 2020-06-12T02:38:06.777 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.777 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.778 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.778 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.584151+0000", 2020-06-12T02:38:06.778 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.783e-05 2020-06-12T02:38:06.778 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.778 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.779 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:06.779 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.584209+0000", 2020-06-12T02:38:06.779 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045863650000000002 2020-06-12T02:38:06.779 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.779 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.779 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.780 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.588796+0000", 2020-06-12T02:38:06.780 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7614000000000006e-05 2020-06-12T02:38:06.780 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.780 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.780 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:06.781 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.588873+0000", 2020-06-12T02:38:06.781 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0026161410000000002 2020-06-12T02:38:06.781 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.781 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.781 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:06.782 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.591489+0000", 2020-06-12T02:38:06.782 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014992914 2020-06-12T02:38:06.782 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:06.782 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:06.782 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:06.783 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.783 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.783 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:4813 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~450560] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-06-12T02:38:06.783 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:43.343096+0000", 2020-06-12T02:38:06.784 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 23.262305582, 2020-06-12T02:38:06.785 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014820242000000001, 2020-06-12T02:38:06.785 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:06.785 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:06.786 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:06.786 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:06.787 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:06.787 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 4813 2020-06-12T02:38:06.787 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.788 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:06.788 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.788 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:06.790 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343096+0000", 2020-06-12T02:38:06.790 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:06.791 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.791 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.791 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:06.792 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343096+0000", 2020-06-12T02:38:06.792 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00068963 2020-06-12T02:38:06.792 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.793 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.793 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:06.793 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343785+0000", 2020-06-12T02:38:06.794 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1479999999999997e-06 2020-06-12T02:38:06.794 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.794 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.795 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:06.795 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343790+0000", 2020-06-12T02:38:06.795 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4060000000000005e-06 2020-06-12T02:38:06.796 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.796 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.796 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:06.797 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343799+0000", 2020-06-12T02:38:06.797 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0970000000000002e-06 2020-06-12T02:38:06.797 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.797 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.798 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:06.798 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343805+0000", 2020-06-12T02:38:06.798 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00023736499999999999 2020-06-12T02:38:06.799 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.799 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.799 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.799 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.344042+0000", 2020-06-12T02:38:06.800 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000113866 2020-06-12T02:38:06.800 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.800 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.800 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:06.801 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.344156+0000", 2020-06-12T02:38:06.801 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036114760000000002 2020-06-12T02:38:06.801 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.801 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.801 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.802 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.347768+0000", 2020-06-12T02:38:06.802 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1842999999999996e-05 2020-06-12T02:38:06.802 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.802 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.803 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:06.803 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.347850+0000", 2020-06-12T02:38:06.803 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00048835999999999997 2020-06-12T02:38:06.803 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.803 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.804 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:06.804 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.348338+0000", 2020-06-12T02:38:06.804 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0049373139999999999 2020-06-12T02:38:06.804 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.804 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.805 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:06.805 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.353275+0000", 2020-06-12T02:38:06.805 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045727119999999996 2020-06-12T02:38:06.805 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.806 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.806 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:06.806 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.357848+0000", 2020-06-12T02:38:06.806 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8025e-05 2020-06-12T02:38:06.807 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.807 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.807 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:06.808 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.357916+0000", 2020-06-12T02:38:06.808 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014820242000000001 2020-06-12T02:38:06.808 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:06.809 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:06.809 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:06.810 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.810 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.810 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:4921 3.9s0 3:93ea0cad:::smithi10729407-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-06-12T02:38:06.810 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:43.595696+0000", 2020-06-12T02:38:06.811 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 23.009705216, 2020-06-12T02:38:06.811 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016503190000000001, 2020-06-12T02:38:06.811 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:06.811 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:06.811 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:06.812 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:06.812 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:06.812 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 4921 2020-06-12T02:38:06.812 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.813 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:06.813 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.813 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:06.813 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595696+0000", 2020-06-12T02:38:06.815 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:06.815 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.815 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.816 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:06.816 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595696+0000", 2020-06-12T02:38:06.816 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3694999999999999e-05 2020-06-12T02:38:06.817 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.817 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.817 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:06.818 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595720+0000", 2020-06-12T02:38:06.818 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1700000000000001e-06 2020-06-12T02:38:06.818 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.818 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.819 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:06.819 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595723+0000", 2020-06-12T02:38:06.819 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8849999999999999e-06 2020-06-12T02:38:06.819 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.820 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.820 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:06.820 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595731+0000", 2020-06-12T02:38:06.821 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5689999999999999e-06 2020-06-12T02:38:06.821 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.821 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.822 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:06.823 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595733+0000", 2020-06-12T02:38:06.823 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0030116880000000002 2020-06-12T02:38:06.823 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.824 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.824 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.824 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.598745+0000", 2020-06-12T02:38:06.824 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1353999999999996e-05 2020-06-12T02:38:06.825 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.825 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.826 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:06.827 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.598806+0000", 2020-06-12T02:38:06.827 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051955819999999998 2020-06-12T02:38:06.827 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.828 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.828 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.828 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.604002+0000", 2020-06-12T02:38:06.828 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9615000000000001e-05 2020-06-12T02:38:06.828 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.829 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.829 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:06.829 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.604102+0000", 2020-06-12T02:38:06.829 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0052050960000000002 2020-06-12T02:38:06.829 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.830 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.830 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.830 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.609307+0000", 2020-06-12T02:38:06.830 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5025000000000003e-05 2020-06-12T02:38:06.830 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.830 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.831 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:06.831 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.609362+0000", 2020-06-12T02:38:06.831 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002837511 2020-06-12T02:38:06.831 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.831 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.832 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:06.832 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.612199+0000", 2020-06-12T02:38:06.832 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016503190000000001 2020-06-12T02:38:06.832 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:06.832 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:06.833 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:06.833 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.833 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.833 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:5068 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~663552] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e30)", 2020-06-12T02:38:06.833 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:44.157497+0000", 2020-06-12T02:38:06.833 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 22.44790459, 2020-06-12T02:38:06.834 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015164129, 2020-06-12T02:38:06.834 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:06.834 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:06.834 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:06.837 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:06.837 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:06.838 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 5068 2020-06-12T02:38:06.838 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.839 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:06.839 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.839 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:06.840 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.157497+0000", 2020-06-12T02:38:06.840 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:06.841 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.841 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.841 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:06.842 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.157497+0000", 2020-06-12T02:38:06.842 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00068511099999999997 2020-06-12T02:38:06.843 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.843 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.844 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:06.844 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.158182+0000", 2020-06-12T02:38:06.844 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4630000000000004e-06 2020-06-12T02:38:06.845 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.845 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.846 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:06.846 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.158189+0000", 2020-06-12T02:38:06.846 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4199000000000001e-05 2020-06-12T02:38:06.847 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.847 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.847 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:06.848 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.158203+0000", 2020-06-12T02:38:06.848 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1250000000000004e-06 2020-06-12T02:38:06.849 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.849 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.849 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:06.850 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.158211+0000", 2020-06-12T02:38:06.850 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6754000000000002e-05 2020-06-12T02:38:06.850 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.850 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.851 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.851 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.158237+0000", 2020-06-12T02:38:06.851 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2948999999999996e-05 2020-06-12T02:38:06.852 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.852 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.852 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:06.852 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.158310+0000", 2020-06-12T02:38:06.853 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0032401610000000001 2020-06-12T02:38:06.854 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.854 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.854 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.855 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.161550+0000", 2020-06-12T02:38:06.855 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5550000000000002e-05 2020-06-12T02:38:06.855 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.855 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.856 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:06.856 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.161606+0000", 2020-06-12T02:38:06.856 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00062448199999999999 2020-06-12T02:38:06.856 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.856 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.857 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:06.857 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.162230+0000", 2020-06-12T02:38:06.857 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0073257130000000002 2020-06-12T02:38:06.857 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.857 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.858 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:06.858 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.169556+0000", 2020-06-12T02:38:06.858 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0030226419999999999 2020-06-12T02:38:06.858 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.858 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.858 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:06.859 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.172579+0000", 2020-06-12T02:38:06.859 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1979999999999998e-05 2020-06-12T02:38:06.859 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.859 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.859 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:06.860 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.172661+0000", 2020-06-12T02:38:06.860 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015164129 2020-06-12T02:38:06.860 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:06.861 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:06.861 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:06.861 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.862 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.862 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:5071 3.3s0 3:c2ca77da:::smithi10729407-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-06-12T02:38:06.862 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:44.159788+0000", 2020-06-12T02:38:06.862 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 22.445613368, 2020-06-12T02:38:06.863 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016812273999999999, 2020-06-12T02:38:06.863 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:06.863 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:06.863 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:06.863 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:06.864 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:06.864 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 5071 2020-06-12T02:38:06.864 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.864 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:06.864 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.865 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:06.865 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159788+0000", 2020-06-12T02:38:06.865 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:06.865 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.866 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.866 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:06.866 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159788+0000", 2020-06-12T02:38:06.866 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1421000000000001e-05 2020-06-12T02:38:06.866 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.867 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.867 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:06.867 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159839+0000", 2020-06-12T02:38:06.867 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9009999999999994e-06 2020-06-12T02:38:06.867 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.868 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.868 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:06.868 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159847+0000", 2020-06-12T02:38:06.868 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3149999999999999e-05 2020-06-12T02:38:06.868 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.869 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.869 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:06.869 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159860+0000", 2020-06-12T02:38:06.869 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1749999999999996e-06 2020-06-12T02:38:06.870 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.870 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.870 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:06.870 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159866+0000", 2020-06-12T02:38:06.870 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6035999999999999e-05 2020-06-12T02:38:06.871 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.871 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.871 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.871 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159892+0000", 2020-06-12T02:38:06.872 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2627999999999998e-05 2020-06-12T02:38:06.872 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.872 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.872 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:06.872 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159944+0000", 2020-06-12T02:38:06.873 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010270133000000001 2020-06-12T02:38:06.873 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.873 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.874 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.874 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.170214+0000", 2020-06-12T02:38:06.874 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5490999999999998e-05 2020-06-12T02:38:06.874 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.874 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.875 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:06.875 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.170310+0000", 2020-06-12T02:38:06.875 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003215828 2020-06-12T02:38:06.876 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.876 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.876 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.876 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.173526+0000", 2020-06-12T02:38:06.877 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0290000000000005e-05 2020-06-12T02:38:06.877 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.877 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.877 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:06.878 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.173616+0000", 2020-06-12T02:38:06.878 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002984221 2020-06-12T02:38:06.878 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.878 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.878 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:06.879 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.176600+0000", 2020-06-12T02:38:06.879 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016812273999999999 2020-06-12T02:38:06.879 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:06.879 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:06.879 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:06.880 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.880 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.880 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:6432 3.9s0 3:93ea0cad:::smithi10729407-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e31)", 2020-06-12T02:38:06.880 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:47.960104+0000", 2020-06-12T02:38:06.880 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 18.645297272000001, 2020-06-12T02:38:06.881 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016025815999999998, 2020-06-12T02:38:06.881 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:06.881 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:06.881 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:06.881 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:06.882 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:06.882 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 6432 2020-06-12T02:38:06.882 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.882 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:06.882 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.883 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:06.883 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960104+0000", 2020-06-12T02:38:06.883 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:06.883 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.883 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.884 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:06.884 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960104+0000", 2020-06-12T02:38:06.884 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2351999999999999e-05 2020-06-12T02:38:06.884 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.884 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.885 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:06.885 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960126+0000", 2020-06-12T02:38:06.885 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.061e-06 2020-06-12T02:38:06.885 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.885 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.886 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:06.886 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960129+0000", 2020-06-12T02:38:06.886 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4849999999999997e-06 2020-06-12T02:38:06.886 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.887 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.887 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:06.887 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960136+0000", 2020-06-12T02:38:06.887 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.176e-06 2020-06-12T02:38:06.887 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.888 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.888 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:06.888 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960138+0000", 2020-06-12T02:38:06.888 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0044054139999999999 2020-06-12T02:38:06.888 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.889 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.889 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.889 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.964543+0000", 2020-06-12T02:38:06.889 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8077000000000006e-05 2020-06-12T02:38:06.890 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.890 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.890 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:06.890 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.964612+0000", 2020-06-12T02:38:06.891 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0029921150000000001 2020-06-12T02:38:06.891 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.891 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.891 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.891 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.967604+0000", 2020-06-12T02:38:06.892 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2983000000000001e-05 2020-06-12T02:38:06.892 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.892 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.892 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:06.892 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.967697+0000", 2020-06-12T02:38:06.893 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0072496510000000002 2020-06-12T02:38:06.893 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.893 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.893 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.893 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.974946+0000", 2020-06-12T02:38:06.894 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8278e-05 2020-06-12T02:38:06.894 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.894 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.895 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:06.895 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.975005+0000", 2020-06-12T02:38:06.895 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001125224 2020-06-12T02:38:06.895 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.895 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.896 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:06.896 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.976130+0000", 2020-06-12T02:38:06.896 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016025815999999998 2020-06-12T02:38:06.896 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:06.896 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:06.897 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:06.897 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.897 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.897 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8123 3.es1 3:78f01648:::smithi10729407-5:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-06-12T02:38:06.898 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.690721+0000", 2020-06-12T02:38:06.898 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 13.914680168, 2020-06-12T02:38:06.898 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016503126999999999, 2020-06-12T02:38:06.898 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:06.898 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:06.899 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:06.899 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:06.899 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:06.899 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8123 2020-06-12T02:38:06.899 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.900 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:06.900 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.900 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:06.900 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690721+0000", 2020-06-12T02:38:06.900 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:06.901 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.901 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.901 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:06.901 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690721+0000", 2020-06-12T02:38:06.901 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7526e-05 2020-06-12T02:38:06.902 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.902 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.902 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:06.902 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690759+0000", 2020-06-12T02:38:06.902 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1999999999999999e-06 2020-06-12T02:38:06.903 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.903 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.903 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:06.903 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690762+0000", 2020-06-12T02:38:06.903 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9229999999999999e-06 2020-06-12T02:38:06.904 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.904 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.904 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:06.904 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690768+0000", 2020-06-12T02:38:06.904 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4480000000000001e-06 2020-06-12T02:38:06.905 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.905 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.905 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:06.905 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690770+0000", 2020-06-12T02:38:06.905 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00079317799999999996 2020-06-12T02:38:06.906 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.906 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.906 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.907 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.691563+0000", 2020-06-12T02:38:06.907 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5299e-05 2020-06-12T02:38:06.908 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.908 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.908 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:06.908 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.691619+0000", 2020-06-12T02:38:06.908 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010070076000000001 2020-06-12T02:38:06.909 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.909 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.909 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.910 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.701689+0000", 2020-06-12T02:38:06.910 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000110234 2020-06-12T02:38:06.910 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.910 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.910 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:06.911 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.701799+0000", 2020-06-12T02:38:06.911 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00037311100000000003 2020-06-12T02:38:06.911 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.911 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.911 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:06.911 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.702172+0000", 2020-06-12T02:38:06.912 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0044460050000000003 2020-06-12T02:38:06.912 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.912 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.912 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:06.912 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.706618+0000", 2020-06-12T02:38:06.913 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00043550199999999999 2020-06-12T02:38:06.913 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.914 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.914 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:06.914 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.707054+0000", 2020-06-12T02:38:06.914 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017062499999999999 2020-06-12T02:38:06.915 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.915 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.915 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:06.915 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.707224+0000", 2020-06-12T02:38:06.915 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016503126999999999 2020-06-12T02:38:06.916 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:06.916 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:06.916 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:06.916 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.916 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.917 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8124 3.es1 3:78f01648:::smithi10729407-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e33)", 2020-06-12T02:38:06.917 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.690794+0000", 2020-06-12T02:38:06.917 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 13.914606982, 2020-06-12T02:38:06.917 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019298497000000001, 2020-06-12T02:38:06.917 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:06.917 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:06.918 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:06.918 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:06.918 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:06.918 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8124 2020-06-12T02:38:06.918 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.919 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:06.919 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.919 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:06.919 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690794+0000", 2020-06-12T02:38:06.919 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:06.920 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.920 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.920 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:06.920 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690794+0000", 2020-06-12T02:38:06.920 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1869999999999999e-05 2020-06-12T02:38:06.920 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.921 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.921 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:06.921 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690816+0000", 2020-06-12T02:38:06.921 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.162e-06 2020-06-12T02:38:06.921 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.922 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.922 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:06.922 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690819+0000", 2020-06-12T02:38:06.922 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2280000000000003e-06 2020-06-12T02:38:06.922 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.923 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.923 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:06.923 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690826+0000", 2020-06-12T02:38:06.923 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.452e-06 2020-06-12T02:38:06.923 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.923 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.924 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:06.924 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690828+0000", 2020-06-12T02:38:06.924 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00081402600000000003 2020-06-12T02:38:06.924 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.924 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.925 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.925 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.691642+0000", 2020-06-12T02:38:06.925 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00024263 2020-06-12T02:38:06.925 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.925 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.926 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:06.926 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.691885+0000", 2020-06-12T02:38:06.926 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010968739 2020-06-12T02:38:06.926 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.926 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.926 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.927 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.702853+0000", 2020-06-12T02:38:06.927 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8985000000000002e-05 2020-06-12T02:38:06.927 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.927 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.927 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:06.928 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.702952+0000", 2020-06-12T02:38:06.928 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0046973329999999997 2020-06-12T02:38:06.928 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.928 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.928 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.929 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.707650+0000", 2020-06-12T02:38:06.929 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3773e-05 2020-06-12T02:38:06.929 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.929 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.930 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:06.930 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.707723+0000", 2020-06-12T02:38:06.930 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023692990000000001 2020-06-12T02:38:06.930 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.930 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.931 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:06.931 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.710093+0000", 2020-06-12T02:38:06.931 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019298497000000001 2020-06-12T02:38:06.931 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:06.931 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:06.931 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:06.932 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.932 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.932 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8190 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~753664] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-06-12T02:38:06.932 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.894592+0000", 2020-06-12T02:38:06.932 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 13.710809324, 2020-06-12T02:38:06.933 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016165763999999999, 2020-06-12T02:38:06.933 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:06.933 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:06.933 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:06.933 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:06.934 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:06.934 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8190 2020-06-12T02:38:06.934 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.934 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:06.934 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.934 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:06.935 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.894592+0000", 2020-06-12T02:38:06.935 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:06.935 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.935 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.935 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:06.935 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.894592+0000", 2020-06-12T02:38:06.936 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00074881500000000001 2020-06-12T02:38:06.936 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.936 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.936 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:06.936 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.895341+0000", 2020-06-12T02:38:06.937 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1919999999999999e-06 2020-06-12T02:38:06.937 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.938 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.938 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:06.938 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.895348+0000", 2020-06-12T02:38:06.938 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.059e-05 2020-06-12T02:38:06.938 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.939 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.939 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:06.939 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.895359+0000", 2020-06-12T02:38:06.939 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2310000000000001e-06 2020-06-12T02:38:06.939 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.940 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.940 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:06.940 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.895365+0000", 2020-06-12T02:38:06.940 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00049215800000000003 2020-06-12T02:38:06.940 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.940 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.941 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.941 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.895857+0000", 2020-06-12T02:38:06.941 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1001999999999995e-05 2020-06-12T02:38:06.941 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.941 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.942 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:06.942 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.895938+0000", 2020-06-12T02:38:06.942 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0059219040000000004 2020-06-12T02:38:06.942 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.942 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.943 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.943 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.901860+0000", 2020-06-12T02:38:06.943 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010156499999999999 2020-06-12T02:38:06.943 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.943 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.943 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:06.944 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.901961+0000", 2020-06-12T02:38:06.944 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00085242800000000002 2020-06-12T02:38:06.944 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.944 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.944 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:06.945 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.902814+0000", 2020-06-12T02:38:06.945 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0057806180000000004 2020-06-12T02:38:06.945 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.945 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.945 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:06.946 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.908594+0000", 2020-06-12T02:38:06.946 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021229600000000001 2020-06-12T02:38:06.946 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.946 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.946 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:06.946 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.910717+0000", 2020-06-12T02:38:06.947 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0300999999999999e-05 2020-06-12T02:38:06.947 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.947 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.947 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:06.947 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.910758+0000", 2020-06-12T02:38:06.948 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016165763999999999 2020-06-12T02:38:06.948 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:06.948 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:06.948 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:06.948 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.949 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.949 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8191 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~450560] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-06-12T02:38:06.949 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.897242+0000", 2020-06-12T02:38:06.949 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 13.708158871, 2020-06-12T02:38:06.950 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014857576000000001, 2020-06-12T02:38:06.951 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:06.951 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:06.951 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:06.951 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:06.952 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:06.952 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8191 2020-06-12T02:38:06.952 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.952 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:06.952 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.953 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:06.953 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897242+0000", 2020-06-12T02:38:06.953 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:06.953 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.953 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.954 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:06.954 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897242+0000", 2020-06-12T02:38:06.954 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000531592 2020-06-12T02:38:06.954 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.954 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.955 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:06.955 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897774+0000", 2020-06-12T02:38:06.955 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0288999999999999e-05 2020-06-12T02:38:06.955 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.955 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.956 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:06.956 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897784+0000", 2020-06-12T02:38:06.956 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7149999999999993e-06 2020-06-12T02:38:06.956 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.956 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.957 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:06.957 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897794+0000", 2020-06-12T02:38:06.957 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8960000000000002e-06 2020-06-12T02:38:06.957 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.958 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.958 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:06.958 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897799+0000", 2020-06-12T02:38:06.958 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6251999999999998e-05 2020-06-12T02:38:06.958 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.958 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.959 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.959 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897835+0000", 2020-06-12T02:38:06.959 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.768e-05 2020-06-12T02:38:06.959 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.959 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.960 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:06.961 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897933+0000", 2020-06-12T02:38:06.961 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0082988100000000002 2020-06-12T02:38:06.962 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.963 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.967 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.968 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.906232+0000", 2020-06-12T02:38:06.968 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7865999999999998e-05 2020-06-12T02:38:06.968 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.969 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.969 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:06.969 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.906290+0000", 2020-06-12T02:38:06.970 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047895700000000002 2020-06-12T02:38:06.970 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.970 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.971 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:06.971 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.906768+0000", 2020-06-12T02:38:06.971 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002658442 2020-06-12T02:38:06.972 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.972 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.972 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:06.973 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.909427+0000", 2020-06-12T02:38:06.973 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0026066230000000002 2020-06-12T02:38:06.973 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.974 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.974 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:06.974 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.912034+0000", 2020-06-12T02:38:06.974 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6453999999999998e-05 2020-06-12T02:38:06.975 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.975 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.975 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:06.975 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.912100+0000", 2020-06-12T02:38:06.975 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014857576000000001 2020-06-12T02:38:06.976 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:06.976 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:06.976 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:06.976 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.976 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.977 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8193 3.3s0 3:c2ca77da:::smithi10729407-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e33)", 2020-06-12T02:38:06.977 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.897900+0000", 2020-06-12T02:38:06.977 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 13.707500939000001, 2020-06-12T02:38:06.977 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015951213999999998, 2020-06-12T02:38:06.977 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:06.978 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:06.978 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:06.978 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:06.978 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:06.978 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8193 2020-06-12T02:38:06.979 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.979 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:06.979 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.979 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:06.979 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897900+0000", 2020-06-12T02:38:06.980 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:06.980 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.980 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.980 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:06.980 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897900+0000", 2020-06-12T02:38:06.980 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2331e-05 2020-06-12T02:38:06.981 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.981 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.981 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:06.981 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897923+0000", 2020-06-12T02:38:06.981 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.021e-06 2020-06-12T02:38:06.982 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.982 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.982 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:06.982 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897926+0000", 2020-06-12T02:38:06.982 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4150000000000002e-06 2020-06-12T02:38:06.983 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.983 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.983 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:06.983 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897933+0000", 2020-06-12T02:38:06.983 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2359999999999999e-06 2020-06-12T02:38:06.983 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.984 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.984 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:06.984 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897935+0000", 2020-06-12T02:38:06.984 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000118301 2020-06-12T02:38:06.984 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.985 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.985 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.985 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.898054+0000", 2020-06-12T02:38:06.985 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8914999999999998e-05 2020-06-12T02:38:06.985 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.985 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.986 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:06.986 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.898143+0000", 2020-06-12T02:38:06.986 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011203013 2020-06-12T02:38:06.986 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.986 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.987 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.987 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.909346+0000", 2020-06-12T02:38:06.987 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7075000000000003e-05 2020-06-12T02:38:06.987 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.987 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.987 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:06.988 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.909423+0000", 2020-06-12T02:38:06.988 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033600969999999998 2020-06-12T02:38:06.988 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.988 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.988 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:06.989 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.912783+0000", 2020-06-12T02:38:06.989 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6118000000000005e-05 2020-06-12T02:38:06.989 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.989 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.989 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:06.989 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.912869+0000", 2020-06-12T02:38:06.990 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00098269200000000007 2020-06-12T02:38:06.990 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:06.990 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:06.991 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:06.991 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.913852+0000", 2020-06-12T02:38:06.991 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015951213999999998 2020-06-12T02:38:06.991 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:06.991 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:06.991 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:06.992 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:06.992 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:06.992 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:06.994 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2020-06-12T02:38:06.994 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:06.991+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:06.995 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-06-12T02:38:07.094 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:07.093+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:07.111 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:07.111 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:38:07.111 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2020-06-12T02:38:07.112 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:07.119 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-06-12T02:38:07.196 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:07.194+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:07.235 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:07.235 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:38:07.235 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:38:07.236 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:38:07.236 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:07.247 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-06-12T02:38:07.296 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:07.294+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:07.380 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:07.380 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:38:07.386 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:38:07.386 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:38:07.386 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.387 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1206 3.5s0 3:a9510bda:::smithi10729407-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:38:07.387 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.856688+0000", 2020-06-12T02:38:07.387 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.504013366000002, 2020-06-12T02:38:07.388 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019116942000000001, 2020-06-12T02:38:07.388 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:07.388 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:07.388 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:07.388 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:07.389 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:07.389 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1206 2020-06-12T02:38:07.389 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.390 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:07.397 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.397 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:07.397 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856688+0000", 2020-06-12T02:38:07.397 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:07.398 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.398 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.398 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:07.399 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856688+0000", 2020-06-12T02:38:07.399 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4355e-05 2020-06-12T02:38:07.399 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.399 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.399 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:07.399 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856713+0000", 2020-06-12T02:38:07.400 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0850000000000001e-06 2020-06-12T02:38:07.400 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.400 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.400 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:07.401 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856716+0000", 2020-06-12T02:38:07.401 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4509999999999997e-06 2020-06-12T02:38:07.401 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.401 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.401 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:07.402 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856722+0000", 2020-06-12T02:38:07.402 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2879999999999999e-06 2020-06-12T02:38:07.402 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.402 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.402 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:07.402 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856724+0000", 2020-06-12T02:38:07.403 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0057507449999999998 2020-06-12T02:38:07.403 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.403 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.403 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:07.403 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.862475+0000", 2020-06-12T02:38:07.404 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5759000000000003e-05 2020-06-12T02:38:07.404 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.404 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.404 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:07.404 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.862571+0000", 2020-06-12T02:38:07.404 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012199018000000001 2020-06-12T02:38:07.405 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.405 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.405 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:07.405 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.874770+0000", 2020-06-12T02:38:07.405 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7260999999999999e-05 2020-06-12T02:38:07.406 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.406 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.406 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:07.406 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.874827+0000", 2020-06-12T02:38:07.407 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00097798000000000004 2020-06-12T02:38:07.407 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.407 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.407 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:07.408 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.875805+0000", 2020-06-12T02:38:07.409 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019116942000000001 2020-06-12T02:38:07.409 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:07.409 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:07.409 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:07.410 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.410 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.410 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1572 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~745472] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-06-12T02:38:07.410 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.886439+0000", 2020-06-12T02:38:07.411 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 33.474262648, 2020-06-12T02:38:07.411 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021442571000000001, 2020-06-12T02:38:07.411 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:07.411 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:07.411 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:07.412 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:07.412 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:07.412 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1572 2020-06-12T02:38:07.412 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.412 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:07.413 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.413 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:07.413 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.886439+0000", 2020-06-12T02:38:07.413 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:07.413 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.413 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.414 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:07.414 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.886439+0000", 2020-06-12T02:38:07.414 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00073027599999999997 2020-06-12T02:38:07.414 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.415 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.415 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:07.415 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887169+0000", 2020-06-12T02:38:07.415 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.49e-06 2020-06-12T02:38:07.416 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.416 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.416 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:07.416 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887175+0000", 2020-06-12T02:38:07.416 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7899999999999994e-06 2020-06-12T02:38:07.417 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.417 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.417 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:07.417 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887185+0000", 2020-06-12T02:38:07.417 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.42e-06 2020-06-12T02:38:07.417 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.418 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.418 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:07.418 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887189+0000", 2020-06-12T02:38:07.418 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1619999999999999e-05 2020-06-12T02:38:07.418 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.419 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.419 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:07.419 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887221+0000", 2020-06-12T02:38:07.419 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9720000000000002e-05 2020-06-12T02:38:07.420 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.420 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.420 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:07.420 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887300+0000", 2020-06-12T02:38:07.420 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047354100000000001 2020-06-12T02:38:07.421 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.421 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.421 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:07.421 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887774+0000", 2020-06-12T02:38:07.421 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0030649750000000002 2020-06-12T02:38:07.422 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.422 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.422 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:07.422 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.890839+0000", 2020-06-12T02:38:07.423 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016995321000000001 2020-06-12T02:38:07.423 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.423 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.424 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:07.424 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.907834+0000", 2020-06-12T02:38:07.424 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7417999999999999e-05 2020-06-12T02:38:07.425 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.425 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.425 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:07.426 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.907882+0000", 2020-06-12T02:38:07.426 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021442571000000001 2020-06-12T02:38:07.426 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:07.427 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:07.427 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:07.427 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.427 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.428 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1573 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~712704] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-06-12T02:38:07.428 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.890888+0000", 2020-06-12T02:38:07.428 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 33.469813913000003, 2020-06-12T02:38:07.429 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018868001999999998, 2020-06-12T02:38:07.429 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:07.429 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:07.430 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:07.430 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:07.430 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:07.431 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1573 2020-06-12T02:38:07.431 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.431 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:07.432 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.432 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:07.432 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.890888+0000", 2020-06-12T02:38:07.432 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:07.432 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.433 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.433 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:07.433 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.890888+0000", 2020-06-12T02:38:07.433 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000683605 2020-06-12T02:38:07.436 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.438 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.439 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:07.439 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891571+0000", 2020-06-12T02:38:07.440 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4999999999999999e-06 2020-06-12T02:38:07.440 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.440 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.440 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:07.440 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891577+0000", 2020-06-12T02:38:07.441 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.171e-05 2020-06-12T02:38:07.441 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.441 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.441 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:07.441 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891589+0000", 2020-06-12T02:38:07.442 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7650000000000001e-06 2020-06-12T02:38:07.442 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.442 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.442 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:07.442 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891593+0000", 2020-06-12T02:38:07.442 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6689000000000001e-05 2020-06-12T02:38:07.443 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.443 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.443 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:07.443 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891630+0000", 2020-06-12T02:38:07.443 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9877999999999997e-05 2020-06-12T02:38:07.444 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.444 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.444 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:07.444 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891710+0000", 2020-06-12T02:38:07.444 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053367200000000001 2020-06-12T02:38:07.444 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.445 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.445 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:07.445 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.892244+0000", 2020-06-12T02:38:07.445 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004927196 2020-06-12T02:38:07.445 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.446 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.446 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:07.446 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.897171+0000", 2020-06-12T02:38:07.446 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012496419 2020-06-12T02:38:07.446 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.447 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.447 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:07.447 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.909667+0000", 2020-06-12T02:38:07.447 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8567999999999996e-05 2020-06-12T02:38:07.447 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.447 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.448 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:07.448 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.909756+0000", 2020-06-12T02:38:07.448 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018868001999999998 2020-06-12T02:38:07.448 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:07.448 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:07.449 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:07.449 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.449 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.449 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1576 3.5s0 3:a446366d:::smithi10729407-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e26)", 2020-06-12T02:38:07.449 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.891955+0000", 2020-06-12T02:38:07.449 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 33.468746672999998, 2020-06-12T02:38:07.450 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019893246999999999, 2020-06-12T02:38:07.450 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:07.450 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:07.450 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:07.450 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:07.451 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:07.451 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1576 2020-06-12T02:38:07.451 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.451 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:07.451 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.451 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:07.452 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891955+0000", 2020-06-12T02:38:07.452 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:07.452 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.452 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.452 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:07.453 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891955+0000", 2020-06-12T02:38:07.453 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2067999999999999e-05 2020-06-12T02:38:07.453 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.453 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.453 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:07.454 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891977+0000", 2020-06-12T02:38:07.454 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.18e-06 2020-06-12T02:38:07.454 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.456 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.456 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:07.457 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891980+0000", 2020-06-12T02:38:07.457 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3629999999999999e-06 2020-06-12T02:38:07.457 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.457 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.457 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:07.458 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891987+0000", 2020-06-12T02:38:07.458 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.181e-06 2020-06-12T02:38:07.458 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.458 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.458 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:07.458 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891989+0000", 2020-06-12T02:38:07.459 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050959120000000002 2020-06-12T02:38:07.459 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.459 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.459 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:07.459 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.897085+0000", 2020-06-12T02:38:07.460 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1089999999999998e-05 2020-06-12T02:38:07.460 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.460 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.460 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:07.460 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.897166+0000", 2020-06-12T02:38:07.460 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013400235 2020-06-12T02:38:07.461 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.461 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.461 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:07.461 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.910566+0000", 2020-06-12T02:38:07.461 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000104981 2020-06-12T02:38:07.462 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.462 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.462 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:07.462 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.910671+0000", 2020-06-12T02:38:07.462 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001177237 2020-06-12T02:38:07.462 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.463 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.463 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:07.463 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.911848+0000", 2020-06-12T02:38:07.463 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019893246999999999 2020-06-12T02:38:07.464 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:07.465 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:07.465 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:07.465 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.466 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.466 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2913 3.5s0 3:a9510bda:::smithi10729407-14:head [append 0~770048,stat] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:07.466 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.070354+0000", 2020-06-12T02:38:07.467 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 30.290347872000002, 2020-06-12T02:38:07.467 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60690798599999995, 2020-06-12T02:38:07.467 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:07.468 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:07.468 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:07.468 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:07.468 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:07.469 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2913 2020-06-12T02:38:07.469 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.469 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:07.469 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.470 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:07.470 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.070354+0000", 2020-06-12T02:38:07.470 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:07.471 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.471 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.471 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:07.471 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.070354+0000", 2020-06-12T02:38:07.472 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00098478900000000002 2020-06-12T02:38:07.472 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.472 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.473 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:07.473 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071339+0000", 2020-06-12T02:38:07.473 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8349999999999993e-06 2020-06-12T02:38:07.473 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.474 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.475 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:07.475 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071347+0000", 2020-06-12T02:38:07.475 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0749e-05 2020-06-12T02:38:07.476 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.476 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.476 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:07.476 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071358+0000", 2020-06-12T02:38:07.477 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9369999999999997e-06 2020-06-12T02:38:07.477 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.477 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.478 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:07.478 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071362+0000", 2020-06-12T02:38:07.478 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0426e-05 2020-06-12T02:38:07.478 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.478 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.479 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:07.479 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071423+0000", 2020-06-12T02:38:07.479 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4129999999999998e-05 2020-06-12T02:38:07.479 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.479 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.480 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:07.480 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071487+0000", 2020-06-12T02:38:07.480 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053752599999999998 2020-06-12T02:38:07.480 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.480 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.481 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:07.481 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072024+0000", 2020-06-12T02:38:07.481 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036162579999999998 2020-06-12T02:38:07.482 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.483 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.483 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:07.483 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075640+0000", 2020-06-12T02:38:07.483 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60152201999999999 2020-06-12T02:38:07.484 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.484 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.484 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:07.484 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677162+0000", 2020-06-12T02:38:07.485 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9315999999999996e-05 2020-06-12T02:38:07.485 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.485 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.485 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:07.486 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677262+0000", 2020-06-12T02:38:07.486 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60690798599999995 2020-06-12T02:38:07.486 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:07.486 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:07.486 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:07.487 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.487 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.487 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2914 3.5s0 3:a9510bda:::smithi10729407-14:head [append 0~401408] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:07.487 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.072409+0000", 2020-06-12T02:38:07.487 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 30.288292427999998, 2020-06-12T02:38:07.488 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.61956801100000003, 2020-06-12T02:38:07.488 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:07.488 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:07.488 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:07.488 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:07.488 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:07.489 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2914 2020-06-12T02:38:07.489 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.489 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:07.489 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.490 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:07.490 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072409+0000", 2020-06-12T02:38:07.490 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:07.490 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.490 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.491 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:07.491 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072409+0000", 2020-06-12T02:38:07.491 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000446317 2020-06-12T02:38:07.491 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.491 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.492 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:07.492 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072856+0000", 2020-06-12T02:38:07.492 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2140000000000004e-06 2020-06-12T02:38:07.493 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.493 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.493 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:07.494 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072861+0000", 2020-06-12T02:38:07.494 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8710000000000003e-06 2020-06-12T02:38:07.494 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.494 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.494 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:07.495 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072870+0000", 2020-06-12T02:38:07.495 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2270000000000004e-06 2020-06-12T02:38:07.495 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.495 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.496 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:07.496 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072874+0000", 2020-06-12T02:38:07.496 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024649960000000001 2020-06-12T02:38:07.496 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.496 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.497 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:07.497 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075339+0000", 2020-06-12T02:38:07.497 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9179999999999996e-05 2020-06-12T02:38:07.497 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.498 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.498 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:07.498 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075438+0000", 2020-06-12T02:38:07.500 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60189281400000005 2020-06-12T02:38:07.500 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.500 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.501 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:07.501 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677331+0000", 2020-06-12T02:38:07.501 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012602800000000001 2020-06-12T02:38:07.501 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.501 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.502 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:07.502 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677457+0000", 2020-06-12T02:38:07.502 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00088167699999999998 2020-06-12T02:38:07.502 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.502 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.503 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:07.503 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.678339+0000", 2020-06-12T02:38:07.503 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004312995 2020-06-12T02:38:07.503 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.503 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.504 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:07.504 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.682652+0000", 2020-06-12T02:38:07.504 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0092869140000000003 2020-06-12T02:38:07.504 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.504 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.505 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:07.505 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691938+0000", 2020-06-12T02:38:07.505 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8778e-05 2020-06-12T02:38:07.506 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.506 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.506 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:07.506 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691977+0000", 2020-06-12T02:38:07.507 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.61956801100000003 2020-06-12T02:38:07.507 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:07.507 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:07.507 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:07.507 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.508 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.508 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2915 3.5s0 3:a9510bda:::smithi10729407-14:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:07.508 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.072912+0000", 2020-06-12T02:38:07.508 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 30.287789688, 2020-06-12T02:38:07.508 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.6192204, 2020-06-12T02:38:07.509 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:07.509 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:07.510 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:07.510 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:07.510 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:07.510 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2915 2020-06-12T02:38:07.510 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.511 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:07.511 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.511 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:07.511 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072912+0000", 2020-06-12T02:38:07.511 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:07.512 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.512 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.512 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:07.512 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072912+0000", 2020-06-12T02:38:07.512 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9201e-05 2020-06-12T02:38:07.513 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.513 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.514 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:07.514 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072951+0000", 2020-06-12T02:38:07.514 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5480000000000002e-06 2020-06-12T02:38:07.515 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.515 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.516 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:07.517 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072955+0000", 2020-06-12T02:38:07.517 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3879999999999997e-06 2020-06-12T02:38:07.517 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.518 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.518 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:07.518 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072961+0000", 2020-06-12T02:38:07.519 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6589999999999999e-06 2020-06-12T02:38:07.519 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.519 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.520 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:07.520 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072964+0000", 2020-06-12T02:38:07.520 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025221039999999998 2020-06-12T02:38:07.521 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.521 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.521 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:07.522 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075486+0000", 2020-06-12T02:38:07.522 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5038999999999998e-05 2020-06-12T02:38:07.522 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.522 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.523 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:07.523 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075561+0000", 2020-06-12T02:38:07.523 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60452397499999999 2020-06-12T02:38:07.523 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.523 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.524 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:07.524 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680085+0000", 2020-06-12T02:38:07.524 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1398999999999999e-05 2020-06-12T02:38:07.524 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.524 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.525 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:07.525 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680156+0000", 2020-06-12T02:38:07.525 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000274465 2020-06-12T02:38:07.525 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.525 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.526 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:07.526 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680431+0000", 2020-06-12T02:38:07.526 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025071999999999998 2020-06-12T02:38:07.526 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.526 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.527 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:07.527 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.682938+0000", 2020-06-12T02:38:07.527 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0091116449999999998 2020-06-12T02:38:07.527 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.527 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.528 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:07.528 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692050+0000", 2020-06-12T02:38:07.528 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2776999999999997e-05 2020-06-12T02:38:07.528 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.528 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.529 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:07.529 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692132+0000", 2020-06-12T02:38:07.529 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.6192204 2020-06-12T02:38:07.529 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:07.529 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:07.530 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:07.530 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.530 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.530 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2916 3.5s0 3:a9510bda:::smithi10729407-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:38:07.530 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.072986+0000", 2020-06-12T02:38:07.531 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 30.287715449, 2020-06-12T02:38:07.531 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.62063941099999997, 2020-06-12T02:38:07.531 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:07.531 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:07.532 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:07.532 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:07.532 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:07.533 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2916 2020-06-12T02:38:07.533 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.533 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:07.534 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.534 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:07.534 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072986+0000", 2020-06-12T02:38:07.534 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:07.534 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.535 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.535 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:07.535 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072986+0000", 2020-06-12T02:38:07.535 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2101000000000001e-05 2020-06-12T02:38:07.536 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.536 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.536 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:07.536 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073008+0000", 2020-06-12T02:38:07.536 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1549999999999999e-06 2020-06-12T02:38:07.537 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.537 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.537 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:07.537 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073011+0000", 2020-06-12T02:38:07.537 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2360000000000001e-06 2020-06-12T02:38:07.538 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.538 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.538 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:07.538 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073018+0000", 2020-06-12T02:38:07.538 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2790000000000001e-06 2020-06-12T02:38:07.539 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.539 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.539 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:07.539 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073020+0000", 2020-06-12T02:38:07.539 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025637670000000002 2020-06-12T02:38:07.540 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.540 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.540 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:07.540 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075584+0000", 2020-06-12T02:38:07.540 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1814999999999998e-05 2020-06-12T02:38:07.541 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.541 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.542 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:07.542 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075636+0000", 2020-06-12T02:38:07.542 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60499943199999995 2020-06-12T02:38:07.543 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.543 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.543 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:07.544 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680635+0000", 2020-06-12T02:38:07.544 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0968000000000002e-05 2020-06-12T02:38:07.544 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.544 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.545 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:07.545 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680706+0000", 2020-06-12T02:38:07.545 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011777101999999999 2020-06-12T02:38:07.546 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.546 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.546 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:07.546 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692483+0000", 2020-06-12T02:38:07.546 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5890999999999995e-05 2020-06-12T02:38:07.547 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.547 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.547 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:07.547 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692569+0000", 2020-06-12T02:38:07.547 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001056665 2020-06-12T02:38:07.548 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.548 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.548 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:07.548 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.693626+0000", 2020-06-12T02:38:07.548 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.62063941099999997 2020-06-12T02:38:07.549 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:07.549 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:07.549 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:07.549 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.549 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.550 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2922 3.ds1 3:b983fd28:::smithi10729407-1:head [append 0~466944] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:07.550 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.097192+0000", 2020-06-12T02:38:07.550 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 30.263510082, 2020-06-12T02:38:07.550 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58672644699999998, 2020-06-12T02:38:07.550 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:07.551 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:07.551 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:07.551 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:07.551 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:07.551 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2922 2020-06-12T02:38:07.552 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.552 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:07.552 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.552 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:07.552 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097192+0000", 2020-06-12T02:38:07.553 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:07.553 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.553 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.553 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:07.553 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097192+0000", 2020-06-12T02:38:07.554 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047206100000000002 2020-06-12T02:38:07.554 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.554 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.554 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:07.554 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097664+0000", 2020-06-12T02:38:07.555 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8359999999999999e-06 2020-06-12T02:38:07.555 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.555 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.555 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:07.555 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097668+0000", 2020-06-12T02:38:07.556 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0815999999999999e-05 2020-06-12T02:38:07.556 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.556 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.556 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:07.556 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097679+0000", 2020-06-12T02:38:07.558 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.028e-06 2020-06-12T02:38:07.558 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.558 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.558 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:07.558 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097684+0000", 2020-06-12T02:38:07.559 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3932999999999999e-05 2020-06-12T02:38:07.559 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.559 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.559 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:07.559 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097748+0000", 2020-06-12T02:38:07.560 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2214999999999999e-05 2020-06-12T02:38:07.560 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.560 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.560 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:07.560 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097820+0000", 2020-06-12T02:38:07.561 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00041583800000000003 2020-06-12T02:38:07.561 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.561 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.561 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:07.561 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.098236+0000", 2020-06-12T02:38:07.562 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0020339960000000002 2020-06-12T02:38:07.562 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.562 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.562 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:07.563 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.100270+0000", 2020-06-12T02:38:07.563 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58355657299999997 2020-06-12T02:38:07.563 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.564 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.564 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:07.564 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.683827+0000", 2020-06-12T02:38:07.565 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1150999999999995e-05 2020-06-12T02:38:07.565 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.565 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.566 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:07.566 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.683918+0000", 2020-06-12T02:38:07.566 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58672644699999998 2020-06-12T02:38:07.567 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:07.567 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:07.567 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:07.567 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.567 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.568 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2923 3.ds1 3:b983fd28:::smithi10729407-1:head [append 0~614400] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:07.568 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.100504+0000", 2020-06-12T02:38:07.568 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 30.260197440999999, 2020-06-12T02:38:07.568 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58853946999999995, 2020-06-12T02:38:07.568 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:07.569 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:07.569 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:07.569 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:07.569 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:07.569 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2923 2020-06-12T02:38:07.570 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.570 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:07.570 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.570 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:07.570 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.100504+0000", 2020-06-12T02:38:07.571 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:07.571 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.571 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.571 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:07.571 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.100504+0000", 2020-06-12T02:38:07.572 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00068395399999999996 2020-06-12T02:38:07.572 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.572 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.572 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:07.572 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101188+0000", 2020-06-12T02:38:07.573 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4050000000000003e-06 2020-06-12T02:38:07.573 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.573 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.573 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:07.573 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101196+0000", 2020-06-12T02:38:07.574 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.328e-05 2020-06-12T02:38:07.574 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.574 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.574 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:07.574 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101209+0000", 2020-06-12T02:38:07.574 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3419999999999996e-06 2020-06-12T02:38:07.575 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.575 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.575 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:07.575 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101215+0000", 2020-06-12T02:38:07.575 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1389000000000005e-05 2020-06-12T02:38:07.576 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.576 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.576 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:07.576 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101277+0000", 2020-06-12T02:38:07.576 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7724000000000004e-05 2020-06-12T02:38:07.577 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.577 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.577 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:07.577 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101344+0000", 2020-06-12T02:38:07.577 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047852499999999999 2020-06-12T02:38:07.578 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.578 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.578 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:07.578 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101823+0000", 2020-06-12T02:38:07.578 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0039255510000000002 2020-06-12T02:38:07.579 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.579 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.579 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:07.579 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105748+0000", 2020-06-12T02:38:07.580 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58321340099999996 2020-06-12T02:38:07.580 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.581 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.581 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:07.581 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.688962+0000", 2020-06-12T02:38:07.581 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1898999999999996e-05 2020-06-12T02:38:07.582 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.582 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.582 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:07.582 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689044+0000", 2020-06-12T02:38:07.582 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58853946999999995 2020-06-12T02:38:07.583 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:07.583 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:07.583 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:07.583 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.583 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.583 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2924 3.ds1 3:b983fd28:::smithi10729407-1:head [append 0~188416] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:07.584 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.101633+0000", 2020-06-12T02:38:07.584 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 30.259068447000001, 2020-06-12T02:38:07.584 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58758259199999996, 2020-06-12T02:38:07.586 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:07.587 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:07.587 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:07.587 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:07.587 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:07.587 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2924 2020-06-12T02:38:07.588 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.588 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:07.588 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.588 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:07.588 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101633+0000", 2020-06-12T02:38:07.588 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:07.589 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.589 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.589 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:07.589 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101633+0000", 2020-06-12T02:38:07.589 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00018775399999999999 2020-06-12T02:38:07.590 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.590 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.590 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:07.590 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101821+0000", 2020-06-12T02:38:07.590 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7669999999999998e-06 2020-06-12T02:38:07.591 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.591 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.591 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:07.591 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101828+0000", 2020-06-12T02:38:07.591 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2716e-05 2020-06-12T02:38:07.592 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.592 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.592 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:07.592 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101840+0000", 2020-06-12T02:38:07.592 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.215e-06 2020-06-12T02:38:07.593 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.593 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.593 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:07.593 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101846+0000", 2020-06-12T02:38:07.593 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021214250000000001 2020-06-12T02:38:07.594 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.594 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.594 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:07.594 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.103967+0000", 2020-06-12T02:38:07.594 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1018000000000003e-05 2020-06-12T02:38:07.595 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.595 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.595 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:07.595 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.104038+0000", 2020-06-12T02:38:07.595 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000333364 2020-06-12T02:38:07.596 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.596 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.596 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:07.596 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.104371+0000", 2020-06-12T02:38:07.596 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001378546 2020-06-12T02:38:07.596 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.597 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.597 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:07.597 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105750+0000", 2020-06-12T02:38:07.597 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58338272899999999 2020-06-12T02:38:07.597 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.598 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.598 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:07.598 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689133+0000", 2020-06-12T02:38:07.598 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3058000000000003e-05 2020-06-12T02:38:07.598 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.599 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.599 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:07.599 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689216+0000", 2020-06-12T02:38:07.599 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58758259199999996 2020-06-12T02:38:07.599 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:07.600 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:07.600 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:07.600 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.600 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.600 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2925 3.ds1 3:b983fd28:::smithi10729407-1:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:07.601 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.101899+0000", 2020-06-12T02:38:07.601 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 30.258802782, 2020-06-12T02:38:07.601 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58747810899999997, 2020-06-12T02:38:07.601 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:07.601 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:07.602 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:07.602 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:07.602 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:07.602 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2925 2020-06-12T02:38:07.602 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.603 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:07.603 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.603 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:07.603 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101899+0000", 2020-06-12T02:38:07.603 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:07.604 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.604 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.604 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:07.604 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101899+0000", 2020-06-12T02:38:07.604 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2906999999999998e-05 2020-06-12T02:38:07.604 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.605 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.605 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:07.605 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101972+0000", 2020-06-12T02:38:07.605 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1789999999999996e-06 2020-06-12T02:38:07.605 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.606 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.606 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:07.606 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101978+0000", 2020-06-12T02:38:07.606 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1358e-05 2020-06-12T02:38:07.606 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.607 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.607 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:07.607 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101989+0000", 2020-06-12T02:38:07.607 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.031e-06 2020-06-12T02:38:07.607 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.608 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.608 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:07.608 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101993+0000", 2020-06-12T02:38:07.608 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033218990000000001 2020-06-12T02:38:07.608 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.609 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.609 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:07.609 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105315+0000", 2020-06-12T02:38:07.609 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6324999999999999e-05 2020-06-12T02:38:07.609 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.610 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.610 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:07.610 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105382+0000", 2020-06-12T02:38:07.610 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000216624 2020-06-12T02:38:07.610 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.611 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.611 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:07.611 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105598+0000", 2020-06-12T02:38:07.611 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001236586 2020-06-12T02:38:07.612 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.612 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.612 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:07.612 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.106835+0000", 2020-06-12T02:38:07.612 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58245509200000001 2020-06-12T02:38:07.613 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.613 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.613 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:07.613 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689290+0000", 2020-06-12T02:38:07.613 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7107999999999993e-05 2020-06-12T02:38:07.614 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.614 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.614 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:07.614 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689377+0000", 2020-06-12T02:38:07.614 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58747810899999997 2020-06-12T02:38:07.615 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:07.615 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:07.615 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:07.615 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.615 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.616 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2926 3.ds1 3:b983fd28:::smithi10729407-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:38:07.616 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.102036+0000", 2020-06-12T02:38:07.616 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 30.258665797999999, 2020-06-12T02:38:07.616 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58868048699999997, 2020-06-12T02:38:07.616 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:07.617 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:07.617 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:07.617 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:07.617 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:07.617 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2926 2020-06-12T02:38:07.618 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.618 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:07.618 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.618 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:07.618 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102036+0000", 2020-06-12T02:38:07.618 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:07.619 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.619 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.619 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:07.619 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102036+0000", 2020-06-12T02:38:07.619 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3786999999999999e-05 2020-06-12T02:38:07.620 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.620 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.620 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:07.620 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102080+0000", 2020-06-12T02:38:07.620 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7660000000000004e-06 2020-06-12T02:38:07.621 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.621 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.621 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:07.621 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102085+0000", 2020-06-12T02:38:07.621 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3145e-05 2020-06-12T02:38:07.622 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.622 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.622 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:07.622 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102099+0000", 2020-06-12T02:38:07.622 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2440000000000004e-06 2020-06-12T02:38:07.623 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.623 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.624 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:07.624 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102103+0000", 2020-06-12T02:38:07.624 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036826990000000002 2020-06-12T02:38:07.624 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.625 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.625 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:07.625 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105786+0000", 2020-06-12T02:38:07.625 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.6774000000000006e-05 2020-06-12T02:38:07.625 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.626 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.626 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:07.626 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105882+0000", 2020-06-12T02:38:07.626 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58352084900000001 2020-06-12T02:38:07.626 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.627 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.627 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:07.627 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689403+0000", 2020-06-12T02:38:07.627 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3788999999999995e-05 2020-06-12T02:38:07.627 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.628 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.628 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:07.628 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689467+0000", 2020-06-12T02:38:07.628 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001249434 2020-06-12T02:38:07.628 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.628 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.629 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:07.629 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690716+0000", 2020-06-12T02:38:07.629 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58868048699999997 2020-06-12T02:38:07.629 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:07.629 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:07.630 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:07.631 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.631 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.631 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2950 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~598016] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:07.631 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.162562+0000", 2020-06-12T02:38:07.632 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 30.198139424000001, 2020-06-12T02:38:07.632 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52780611399999999, 2020-06-12T02:38:07.632 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:07.632 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:07.633 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:07.633 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:07.633 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:07.633 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2950 2020-06-12T02:38:07.633 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.634 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:07.634 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.634 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:07.634 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.162562+0000", 2020-06-12T02:38:07.634 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:07.634 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.635 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.635 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:07.635 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.162562+0000", 2020-06-12T02:38:07.635 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00070718099999999998 2020-06-12T02:38:07.635 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.636 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.636 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:07.637 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163269+0000", 2020-06-12T02:38:07.637 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2029999999999997e-06 2020-06-12T02:38:07.637 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.637 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.637 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:07.638 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163276+0000", 2020-06-12T02:38:07.638 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2734999999999999e-05 2020-06-12T02:38:07.638 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.638 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.639 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:07.639 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163288+0000", 2020-06-12T02:38:07.639 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.404e-06 2020-06-12T02:38:07.639 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.639 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.640 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:07.640 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163295+0000", 2020-06-12T02:38:07.640 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5784000000000006e-05 2020-06-12T02:38:07.640 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.640 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.641 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:07.641 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163361+0000", 2020-06-12T02:38:07.641 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5318e-05 2020-06-12T02:38:07.641 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.641 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.642 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:07.642 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163436+0000", 2020-06-12T02:38:07.642 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054801100000000005 2020-06-12T02:38:07.642 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.642 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.642 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:07.643 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163984+0000", 2020-06-12T02:38:07.643 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.007023077 2020-06-12T02:38:07.643 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.643 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.643 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:07.644 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171007+0000", 2020-06-12T02:38:07.644 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51932051199999996 2020-06-12T02:38:07.644 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.644 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.644 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:07.645 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690327+0000", 2020-06-12T02:38:07.645 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0889000000000001e-05 2020-06-12T02:38:07.645 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.645 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.645 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:07.646 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690368+0000", 2020-06-12T02:38:07.646 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52780611399999999 2020-06-12T02:38:07.646 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:07.646 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:07.646 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:07.647 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.647 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.647 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2951 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~532480] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:07.647 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.165470+0000", 2020-06-12T02:38:07.648 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 30.195231816, 2020-06-12T02:38:07.648 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52507917900000001, 2020-06-12T02:38:07.648 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:07.648 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:07.648 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:07.649 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:07.649 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:07.649 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2951 2020-06-12T02:38:07.649 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.649 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:07.650 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.650 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:07.650 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.165470+0000", 2020-06-12T02:38:07.650 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:07.650 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.651 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.651 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:07.651 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.165470+0000", 2020-06-12T02:38:07.651 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00062231600000000004 2020-06-12T02:38:07.651 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.652 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.652 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:07.652 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166092+0000", 2020-06-12T02:38:07.652 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1610000000000002e-06 2020-06-12T02:38:07.652 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.653 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.653 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:07.653 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166097+0000", 2020-06-12T02:38:07.653 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7780000000000002e-06 2020-06-12T02:38:07.653 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.654 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.654 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:07.654 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166107+0000", 2020-06-12T02:38:07.654 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7169999999999999e-06 2020-06-12T02:38:07.654 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.655 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.655 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:07.655 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166112+0000", 2020-06-12T02:38:07.655 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4792000000000002e-05 2020-06-12T02:38:07.655 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.656 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.656 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:07.656 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166147+0000", 2020-06-12T02:38:07.656 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9336e-05 2020-06-12T02:38:07.656 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.656 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.657 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:07.657 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166236+0000", 2020-06-12T02:38:07.657 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00067747200000000004 2020-06-12T02:38:07.657 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.658 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.658 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:07.658 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166913+0000", 2020-06-12T02:38:07.658 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0047718669999999999 2020-06-12T02:38:07.659 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.659 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.659 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:07.659 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171685+0000", 2020-06-12T02:38:07.659 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51881990700000002 2020-06-12T02:38:07.660 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.660 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.660 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:07.660 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690505+0000", 2020-06-12T02:38:07.660 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3832999999999999e-05 2020-06-12T02:38:07.661 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.661 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.661 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:07.661 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690549+0000", 2020-06-12T02:38:07.661 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52507917900000001 2020-06-12T02:38:07.662 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:07.662 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:07.662 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:07.662 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.662 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.663 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2952 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~270336] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:07.663 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.166891+0000", 2020-06-12T02:38:07.663 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 30.193810166999999, 2020-06-12T02:38:07.663 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52426417000000003, 2020-06-12T02:38:07.663 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:07.664 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:07.664 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:07.664 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:07.664 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:07.664 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2952 2020-06-12T02:38:07.665 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.665 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:07.665 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.665 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:07.665 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166891+0000", 2020-06-12T02:38:07.666 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:07.666 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.666 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.666 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:07.666 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166891+0000", 2020-06-12T02:38:07.667 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00031878599999999999 2020-06-12T02:38:07.667 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.667 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.667 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:07.667 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167210+0000", 2020-06-12T02:38:07.668 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2360000000000004e-06 2020-06-12T02:38:07.668 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.668 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.668 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:07.668 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167219+0000", 2020-06-12T02:38:07.669 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3298e-05 2020-06-12T02:38:07.669 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.669 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.669 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:07.669 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167232+0000", 2020-06-12T02:38:07.669 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5849999999999999e-06 2020-06-12T02:38:07.670 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.670 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.670 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:07.670 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167237+0000", 2020-06-12T02:38:07.670 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0018755740000000001 2020-06-12T02:38:07.671 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.671 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.671 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:07.671 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.169113+0000", 2020-06-12T02:38:07.671 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000104768 2020-06-12T02:38:07.672 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.672 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.672 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:07.672 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.169218+0000", 2020-06-12T02:38:07.672 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00057389799999999999 2020-06-12T02:38:07.673 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.673 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.673 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:07.673 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.169792+0000", 2020-06-12T02:38:07.673 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0019333989999999999 2020-06-12T02:38:07.674 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.674 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.674 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:07.674 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171725+0000", 2020-06-12T02:38:07.674 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51938508699999997 2020-06-12T02:38:07.675 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.675 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.675 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:07.675 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691110+0000", 2020-06-12T02:38:07.675 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5538999999999999e-05 2020-06-12T02:38:07.676 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.676 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.676 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:07.676 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691156+0000", 2020-06-12T02:38:07.677 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52426417000000003 2020-06-12T02:38:07.677 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:07.677 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:07.677 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:07.677 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.678 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.678 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2953 3.5s0 3:a446366d:::smithi10729407-11:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:07.678 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.167293+0000", 2020-06-12T02:38:07.678 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 30.193408720000001, 2020-06-12T02:38:07.678 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52428350999999995, 2020-06-12T02:38:07.679 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:07.679 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:07.679 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:07.679 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:07.680 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:07.680 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2953 2020-06-12T02:38:07.680 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.680 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:07.680 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.681 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:07.681 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167293+0000", 2020-06-12T02:38:07.681 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:07.681 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.681 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.682 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:07.682 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167293+0000", 2020-06-12T02:38:07.682 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3880000000000004e-05 2020-06-12T02:38:07.682 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.682 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.683 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:07.683 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167367+0000", 2020-06-12T02:38:07.683 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.319e-06 2020-06-12T02:38:07.683 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.683 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.683 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:07.684 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167373+0000", 2020-06-12T02:38:07.684 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1719000000000001e-05 2020-06-12T02:38:07.684 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.684 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.684 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:07.685 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167385+0000", 2020-06-12T02:38:07.685 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4580000000000003e-06 2020-06-12T02:38:07.685 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.685 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.685 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:07.685 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167389+0000", 2020-06-12T02:38:07.686 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036341110000000002 2020-06-12T02:38:07.686 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.686 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.686 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:07.686 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171023+0000", 2020-06-12T02:38:07.687 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9418000000000005e-05 2020-06-12T02:38:07.687 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.687 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.687 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:07.687 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171093+0000", 2020-06-12T02:38:07.687 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00029087900000000002 2020-06-12T02:38:07.688 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.688 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.688 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:07.688 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171384+0000", 2020-06-12T02:38:07.688 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000489347 2020-06-12T02:38:07.689 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.689 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.689 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:07.689 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171873+0000", 2020-06-12T02:38:07.689 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51963691300000003 2020-06-12T02:38:07.689 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.690 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.690 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:07.690 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691510+0000", 2020-06-12T02:38:07.690 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6465999999999997e-05 2020-06-12T02:38:07.690 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.691 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.691 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:07.691 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691576+0000", 2020-06-12T02:38:07.691 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52428350999999995 2020-06-12T02:38:07.691 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:07.691 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:07.692 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:07.692 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.692 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.692 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2954 3.5s0 3:a446366d:::smithi10729407-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:38:07.692 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.167432+0000", 2020-06-12T02:38:07.693 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 30.193269970999999, 2020-06-12T02:38:07.693 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52568652699999996, 2020-06-12T02:38:07.693 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:07.693 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:07.693 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:07.694 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:07.694 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:07.694 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2954 2020-06-12T02:38:07.694 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.694 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:07.694 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.695 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:07.695 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167432+0000", 2020-06-12T02:38:07.695 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:07.695 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.695 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.695 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:07.696 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167432+0000", 2020-06-12T02:38:07.696 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3556000000000001e-05 2020-06-12T02:38:07.696 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.696 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.696 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:07.697 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167475+0000", 2020-06-12T02:38:07.697 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7350000000000001e-06 2020-06-12T02:38:07.697 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.697 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.697 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:07.697 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167481+0000", 2020-06-12T02:38:07.698 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3128e-05 2020-06-12T02:38:07.698 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.698 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.698 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:07.698 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167494+0000", 2020-06-12T02:38:07.699 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1579999999999998e-06 2020-06-12T02:38:07.699 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.699 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.699 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:07.699 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167498+0000", 2020-06-12T02:38:07.699 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004082174 2020-06-12T02:38:07.700 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.700 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.700 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:07.700 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171580+0000", 2020-06-12T02:38:07.700 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8954000000000002e-05 2020-06-12T02:38:07.701 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.701 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.701 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:07.701 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171679+0000", 2020-06-12T02:38:07.701 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51993235100000001 2020-06-12T02:38:07.702 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.702 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.702 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:07.702 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691612+0000", 2020-06-12T02:38:07.702 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4329000000000001e-05 2020-06-12T02:38:07.702 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.703 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.703 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:07.703 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691666+0000", 2020-06-12T02:38:07.703 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001452142 2020-06-12T02:38:07.703 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.703 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.704 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:07.704 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.693118+0000", 2020-06-12T02:38:07.704 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52568652699999996 2020-06-12T02:38:07.704 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:07.704 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:07.705 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:07.705 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.705 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.705 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2959 3.ds1 3:b3445631:::smithi10729407-6:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e27)", 2020-06-12T02:38:07.705 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.176248+0000", 2020-06-12T02:38:07.706 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 30.184453229999999, 2020-06-12T02:38:07.706 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51549877099999997, 2020-06-12T02:38:07.706 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:07.706 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:07.706 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:07.706 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:07.707 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:07.707 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2959 2020-06-12T02:38:07.707 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.707 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:07.707 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.708 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:07.708 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176248+0000", 2020-06-12T02:38:07.708 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:07.708 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.708 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.708 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:07.709 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176248+0000", 2020-06-12T02:38:07.709 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5120999999999999e-05 2020-06-12T02:38:07.709 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.709 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.709 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:07.710 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176274+0000", 2020-06-12T02:38:07.710 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9189999999999998e-06 2020-06-12T02:38:07.710 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.710 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.710 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:07.710 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176278+0000", 2020-06-12T02:38:07.711 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6339e-05 2020-06-12T02:38:07.711 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.711 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.711 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:07.711 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176305+0000", 2020-06-12T02:38:07.712 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9089999999999999e-06 2020-06-12T02:38:07.712 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.712 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.712 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:07.712 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176310+0000", 2020-06-12T02:38:07.712 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6095999999999999e-05 2020-06-12T02:38:07.713 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.713 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.713 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:07.713 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176376+0000", 2020-06-12T02:38:07.713 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8590999999999994e-05 2020-06-12T02:38:07.714 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.714 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.714 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:07.714 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176444+0000", 2020-06-12T02:38:07.714 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00024346400000000001 2020-06-12T02:38:07.714 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.715 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.715 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:07.715 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176688+0000", 2020-06-12T02:38:07.715 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00044220899999999999 2020-06-12T02:38:07.715 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.715 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.716 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:07.716 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.177130+0000", 2020-06-12T02:38:07.716 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51452984400000001 2020-06-12T02:38:07.716 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.716 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.717 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:07.717 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691660+0000", 2020-06-12T02:38:07.717 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7279000000000002e-05 2020-06-12T02:38:07.717 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.717 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.717 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:07.718 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691747+0000", 2020-06-12T02:38:07.718 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51549877099999997 2020-06-12T02:38:07.718 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:07.718 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:07.718 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:07.719 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.719 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.719 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:3032 3.ds1 3:b983fd28:::smithi10729407-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:38:07.719 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.864289+0000", 2020-06-12T02:38:07.719 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 29.496412312, 2020-06-12T02:38:07.719 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019572216999999999, 2020-06-12T02:38:07.720 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:07.720 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:07.720 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:07.720 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:07.720 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:07.721 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 3032 2020-06-12T02:38:07.721 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.721 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:07.721 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.721 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:07.721 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864289+0000", 2020-06-12T02:38:07.722 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:07.722 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.722 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.722 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:07.722 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864289+0000", 2020-06-12T02:38:07.723 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5701999999999997e-05 2020-06-12T02:38:07.723 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.723 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.723 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:07.723 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864335+0000", 2020-06-12T02:38:07.723 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0910000000000004e-06 2020-06-12T02:38:07.724 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.724 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.724 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:07.724 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864342+0000", 2020-06-12T02:38:07.725 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2301e-05 2020-06-12T02:38:07.725 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.725 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.725 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:07.725 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864354+0000", 2020-06-12T02:38:07.726 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5199999999999997e-06 2020-06-12T02:38:07.726 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.726 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.726 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:07.726 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864360+0000", 2020-06-12T02:38:07.727 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00049180999999999997 2020-06-12T02:38:07.727 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.727 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.727 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:07.727 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864852+0000", 2020-06-12T02:38:07.727 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6725000000000003e-05 2020-06-12T02:38:07.728 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.728 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.728 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:07.728 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864908+0000", 2020-06-12T02:38:07.728 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0089687010000000008 2020-06-12T02:38:07.729 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.729 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.729 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:07.729 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.873877+0000", 2020-06-12T02:38:07.729 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3382999999999999e-05 2020-06-12T02:38:07.729 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.730 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.730 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:07.730 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.873941+0000", 2020-06-12T02:38:07.730 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.006408927 2020-06-12T02:38:07.730 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.731 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.731 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:07.731 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.880350+0000", 2020-06-12T02:38:07.731 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3905e-05 2020-06-12T02:38:07.731 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.731 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.732 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:07.732 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.880403+0000", 2020-06-12T02:38:07.732 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0034581519999999999 2020-06-12T02:38:07.732 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:07.732 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:07.733 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:07.733 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.883862+0000", 2020-06-12T02:38:07.733 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019572216999999999 2020-06-12T02:38:07.733 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:07.733 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:07.733 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:07.734 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:07.734 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:07.734 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:07.735 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-06-12T02:38:07.736 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:07.735+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:07.837 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:07.835+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:07.852 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:07.853 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:38:07.853 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2020-06-12T02:38:07.853 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:07.861 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-06-12T02:38:07.939 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:07.937+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:07.979 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:07.979 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:38:07.979 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:38:07.980 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:38:07.980 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:07.988 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-06-12T02:38:08.040 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:08.038+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:08.107 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:08.108 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:38:08.108 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:38:08.108 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:38:08.109 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.109 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1558 3.as0 3:5cb723d2:::smithi10729407-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e26)", 2020-06-12T02:38:08.109 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.839131+0000", 2020-06-12T02:38:08.109 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.264545192, 2020-06-12T02:38:08.109 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013057691, 2020-06-12T02:38:08.109 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:08.110 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:08.110 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:08.110 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:08.110 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:08.110 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1558 2020-06-12T02:38:08.111 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.111 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:08.111 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.112 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:08.112 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.839131+0000", 2020-06-12T02:38:08.112 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:08.113 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.113 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.114 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:08.114 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.839131+0000", 2020-06-12T02:38:08.114 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2228999999999998e-05 2020-06-12T02:38:08.115 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.115 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.115 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:08.115 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.839173+0000", 2020-06-12T02:38:08.116 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9789999999999998e-06 2020-06-12T02:38:08.116 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.116 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.116 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:08.116 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.839179+0000", 2020-06-12T02:38:08.116 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3011e-05 2020-06-12T02:38:08.117 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.117 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.117 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:08.117 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.839192+0000", 2020-06-12T02:38:08.117 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7929999999999999e-06 2020-06-12T02:38:08.118 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.118 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.118 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:08.118 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.839196+0000", 2020-06-12T02:38:08.118 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0083729379999999999 2020-06-12T02:38:08.119 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.119 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.119 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:08.119 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.847569+0000", 2020-06-12T02:38:08.119 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5887999999999998e-05 2020-06-12T02:38:08.119 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.120 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.120 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:08.120 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.847625+0000", 2020-06-12T02:38:08.120 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0027582330000000001 2020-06-12T02:38:08.120 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.124 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.124 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:08.124 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.850383+0000", 2020-06-12T02:38:08.124 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00037230500000000002 2020-06-12T02:38:08.125 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.125 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.125 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:08.125 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.850755+0000", 2020-06-12T02:38:08.125 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0014333149999999999 2020-06-12T02:38:08.126 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.126 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.126 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:08.126 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.852189+0000", 2020-06-12T02:38:08.126 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013057691 2020-06-12T02:38:08.127 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:08.127 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:08.127 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:08.127 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.127 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.127 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2746 3.as0 3:5cb723d2:::smithi10729407-15:head [append 0~524288] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:08.128 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:36.723084+0000", 2020-06-12T02:38:08.128 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 31.380592396000001, 2020-06-12T02:38:08.128 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013015831, 2020-06-12T02:38:08.128 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:08.128 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:08.129 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:08.129 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:08.129 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:08.129 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2746 2020-06-12T02:38:08.130 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.130 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:08.130 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.130 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:08.130 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.723084+0000", 2020-06-12T02:38:08.131 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:08.131 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.131 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.131 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:08.132 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.723084+0000", 2020-06-12T02:38:08.132 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0012755659999999999 2020-06-12T02:38:08.132 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.132 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.132 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:08.133 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724359+0000", 2020-06-12T02:38:08.133 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2599999999999996e-06 2020-06-12T02:38:08.133 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.133 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.134 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:08.134 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724365+0000", 2020-06-12T02:38:08.134 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5499999999999995e-06 2020-06-12T02:38:08.135 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.135 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.135 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:08.135 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724373+0000", 2020-06-12T02:38:08.136 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3590000000000001e-06 2020-06-12T02:38:08.136 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.136 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.136 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:08.136 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724378+0000", 2020-06-12T02:38:08.136 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0034266629999999999 2020-06-12T02:38:08.137 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.137 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.137 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:08.137 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.727804+0000", 2020-06-12T02:38:08.137 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6347000000000005e-05 2020-06-12T02:38:08.138 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.138 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.138 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:08.138 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.727891+0000", 2020-06-12T02:38:08.138 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054319199999999996 2020-06-12T02:38:08.139 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.139 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.139 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:08.139 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.728434+0000", 2020-06-12T02:38:08.139 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0029938999999999999 2020-06-12T02:38:08.139 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.140 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.140 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:08.140 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.731428+0000", 2020-06-12T02:38:08.140 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0046291250000000004 2020-06-12T02:38:08.140 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.141 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.141 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:08.141 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.736057+0000", 2020-06-12T02:38:08.141 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2868999999999999e-05 2020-06-12T02:38:08.141 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.141 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.142 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:08.142 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.736100+0000", 2020-06-12T02:38:08.142 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013015831 2020-06-12T02:38:08.142 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:08.143 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:08.143 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:08.143 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.143 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.143 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2747 3.as0 3:5cb723d2:::smithi10729407-15:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:08.144 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:36.724407+0000", 2020-06-12T02:38:08.144 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 31.379269228999998, 2020-06-12T02:38:08.144 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014982544, 2020-06-12T02:38:08.144 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:08.145 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:08.145 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:08.145 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:08.145 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:08.146 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2747 2020-06-12T02:38:08.146 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.146 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:08.146 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.146 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:08.147 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724407+0000", 2020-06-12T02:38:08.147 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:08.147 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.147 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.147 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:08.148 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724407+0000", 2020-06-12T02:38:08.148 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6766e-05 2020-06-12T02:38:08.148 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.148 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.148 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:08.148 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724444+0000", 2020-06-12T02:38:08.149 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.315e-06 2020-06-12T02:38:08.149 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.149 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.149 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:08.149 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724447+0000", 2020-06-12T02:38:08.150 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9100000000000002e-06 2020-06-12T02:38:08.150 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.151 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.151 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:08.151 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724453+0000", 2020-06-12T02:38:08.151 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.43e-06 2020-06-12T02:38:08.152 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.152 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.152 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:08.152 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724455+0000", 2020-06-12T02:38:08.152 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0061404010000000002 2020-06-12T02:38:08.153 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.153 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.153 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:08.153 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.730596+0000", 2020-06-12T02:38:08.153 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8402e-05 2020-06-12T02:38:08.154 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.154 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.154 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:08.154 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.730664+0000", 2020-06-12T02:38:08.154 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00031661500000000002 2020-06-12T02:38:08.155 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.155 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.155 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:08.155 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.730981+0000", 2020-06-12T02:38:08.156 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0049794890000000001 2020-06-12T02:38:08.156 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.156 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.156 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:08.156 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.735960+0000", 2020-06-12T02:38:08.157 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033245739999999998 2020-06-12T02:38:08.157 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.157 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.157 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:08.157 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.739285+0000", 2020-06-12T02:38:08.158 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010464200000000001 2020-06-12T02:38:08.158 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.158 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.158 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:08.158 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.739390+0000", 2020-06-12T02:38:08.158 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014982544 2020-06-12T02:38:08.159 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:08.159 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:08.159 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:08.160 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.160 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.161 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2748 3.as0 3:5cb723d2:::smithi10729407-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:38:08.161 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:36.724477+0000", 2020-06-12T02:38:08.162 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 31.37919909, 2020-06-12T02:38:08.162 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020852102000000001, 2020-06-12T02:38:08.162 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:08.162 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:08.162 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:08.163 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:08.163 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:08.163 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2748 2020-06-12T02:38:08.163 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.164 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:08.164 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.164 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:08.164 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724477+0000", 2020-06-12T02:38:08.164 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:08.164 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.165 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.165 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:08.165 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724477+0000", 2020-06-12T02:38:08.165 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2065e-05 2020-06-12T02:38:08.166 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.166 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.166 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:08.166 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724499+0000", 2020-06-12T02:38:08.167 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9340000000000002e-06 2020-06-12T02:38:08.167 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.167 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.167 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:08.167 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724502+0000", 2020-06-12T02:38:08.167 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.246e-06 2020-06-12T02:38:08.168 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.168 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.168 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:08.168 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724508+0000", 2020-06-12T02:38:08.168 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2900000000000001e-06 2020-06-12T02:38:08.169 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.169 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.169 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:08.169 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724511+0000", 2020-06-12T02:38:08.169 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0068556700000000003 2020-06-12T02:38:08.170 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.170 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.170 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:08.170 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.731366+0000", 2020-06-12T02:38:08.170 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4738999999999999e-05 2020-06-12T02:38:08.170 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.171 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.171 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:08.171 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.731421+0000", 2020-06-12T02:38:08.171 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0080153150000000003 2020-06-12T02:38:08.171 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.172 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.172 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:08.172 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.739436+0000", 2020-06-12T02:38:08.172 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010418 2020-06-12T02:38:08.172 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.172 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.173 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:08.173 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.739541+0000", 2020-06-12T02:38:08.173 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0057886630000000003 2020-06-12T02:38:08.173 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.173 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.174 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:08.174 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.745329+0000", 2020-06-12T02:38:08.174 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020852102000000001 2020-06-12T02:38:08.174 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:08.174 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:08.175 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:08.175 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.175 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.175 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:3055 3.as0 3:5cb723d2:::smithi10729407-15:head [append 0~614400] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:08.175 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.918209+0000", 2020-06-12T02:38:08.175 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 30.185467375000002, 2020-06-12T02:38:08.176 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014165394, 2020-06-12T02:38:08.176 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:08.176 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:08.177 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:08.177 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:08.177 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:08.177 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 3055 2020-06-12T02:38:08.177 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.177 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:08.178 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.178 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:08.178 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.918209+0000", 2020-06-12T02:38:08.178 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:08.178 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.179 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.179 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:08.179 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.918209+0000", 2020-06-12T02:38:08.179 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00081606199999999999 2020-06-12T02:38:08.179 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.179 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.180 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:08.180 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.919025+0000", 2020-06-12T02:38:08.180 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.251e-06 2020-06-12T02:38:08.180 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.180 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.181 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:08.181 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.919031+0000", 2020-06-12T02:38:08.181 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1460999999999999e-05 2020-06-12T02:38:08.181 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.181 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.182 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:08.182 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.919043+0000", 2020-06-12T02:38:08.182 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.592e-06 2020-06-12T02:38:08.182 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.182 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.182 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:08.183 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.919048+0000", 2020-06-12T02:38:08.183 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0020427959999999999 2020-06-12T02:38:08.183 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.183 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.183 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:08.184 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.921091+0000", 2020-06-12T02:38:08.184 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012382800000000001 2020-06-12T02:38:08.184 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.184 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.184 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:08.184 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.921215+0000", 2020-06-12T02:38:08.185 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053208300000000004 2020-06-12T02:38:08.185 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.185 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.185 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:08.185 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.921747+0000", 2020-06-12T02:38:08.186 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0075648599999999996 2020-06-12T02:38:08.186 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.186 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.186 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:08.187 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.929312+0000", 2020-06-12T02:38:08.187 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0029879630000000002 2020-06-12T02:38:08.187 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.187 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.187 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:08.188 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.932300+0000", 2020-06-12T02:38:08.188 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4498000000000004e-05 2020-06-12T02:38:08.188 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.188 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.188 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:08.189 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.932374+0000", 2020-06-12T02:38:08.189 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014165394 2020-06-12T02:38:08.189 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:08.189 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:08.189 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:08.189 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.190 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.190 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:3217 3.as0 3:5732055a:::smithi10729407-8:head [append 0~147456] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:08.190 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:38.300193+0000", 2020-06-12T02:38:08.190 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 29.803483266000001, 2020-06-12T02:38:08.190 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012845314, 2020-06-12T02:38:08.191 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:08.191 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:08.191 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:08.191 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:08.191 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:08.192 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 3217 2020-06-12T02:38:08.192 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.192 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:08.192 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.192 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:08.193 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300193+0000", 2020-06-12T02:38:08.193 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:08.193 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.193 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.193 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:08.194 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300193+0000", 2020-06-12T02:38:08.194 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017722699999999999 2020-06-12T02:38:08.194 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.194 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.194 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:08.195 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300370+0000", 2020-06-12T02:38:08.195 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4009999999999996e-06 2020-06-12T02:38:08.195 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.195 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.199 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:08.199 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300376+0000", 2020-06-12T02:38:08.200 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9109999999999993e-06 2020-06-12T02:38:08.200 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.200 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.200 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:08.200 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300384+0000", 2020-06-12T02:38:08.201 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2740000000000001e-06 2020-06-12T02:38:08.201 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.201 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.201 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:08.201 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300388+0000", 2020-06-12T02:38:08.202 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3261999999999999e-05 2020-06-12T02:38:08.202 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.207 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.208 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:08.208 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300411+0000", 2020-06-12T02:38:08.208 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9906e-05 2020-06-12T02:38:08.208 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.208 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.209 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:08.209 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300461+0000", 2020-06-12T02:38:08.209 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0059439150000000001 2020-06-12T02:38:08.209 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.209 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.210 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:08.210 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.306405+0000", 2020-06-12T02:38:08.210 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7945000000000002e-05 2020-06-12T02:38:08.210 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.210 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.211 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:08.211 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.306463+0000", 2020-06-12T02:38:08.211 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00042195500000000001 2020-06-12T02:38:08.211 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.211 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.212 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:08.212 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.306885+0000", 2020-06-12T02:38:08.212 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0032273800000000002 2020-06-12T02:38:08.212 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.212 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.213 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:08.213 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.310112+0000", 2020-06-12T02:38:08.213 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0028430420000000001 2020-06-12T02:38:08.213 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.214 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.214 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:08.214 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.312955+0000", 2020-06-12T02:38:08.214 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3096000000000005e-05 2020-06-12T02:38:08.214 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.215 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.215 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:08.215 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.313038+0000", 2020-06-12T02:38:08.215 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012845314 2020-06-12T02:38:08.215 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:08.216 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:08.216 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:08.216 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.216 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.216 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:3218 3.as0 3:5732055a:::smithi10729407-8:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:08.216 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:38.300417+0000", 2020-06-12T02:38:08.217 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 29.803258749000001, 2020-06-12T02:38:08.217 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012861745000000001, 2020-06-12T02:38:08.217 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:08.217 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:08.217 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:08.218 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:08.218 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:08.218 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 3218 2020-06-12T02:38:08.218 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.218 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:08.219 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.219 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:08.219 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300417+0000", 2020-06-12T02:38:08.219 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:08.220 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.220 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.220 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:08.220 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300417+0000", 2020-06-12T02:38:08.220 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9110999999999999e-05 2020-06-12T02:38:08.220 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.221 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.221 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:08.221 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300457+0000", 2020-06-12T02:38:08.221 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3289999999999998e-06 2020-06-12T02:38:08.222 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.222 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.222 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:08.222 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300460+0000", 2020-06-12T02:38:08.223 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8569999999999997e-06 2020-06-12T02:38:08.223 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.223 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.223 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:08.223 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300467+0000", 2020-06-12T02:38:08.224 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7080000000000002e-06 2020-06-12T02:38:08.224 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.224 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.224 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:08.224 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300469+0000", 2020-06-12T02:38:08.225 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0064999999999998e-05 2020-06-12T02:38:08.225 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.225 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.225 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:08.225 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300490+0000", 2020-06-12T02:38:08.225 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0943999999999999e-05 2020-06-12T02:38:08.226 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.226 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.226 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:08.226 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300540+0000", 2020-06-12T02:38:08.226 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0073891770000000002 2020-06-12T02:38:08.227 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.227 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.227 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:08.227 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.307930+0000", 2020-06-12T02:38:08.227 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010105199999999999 2020-06-12T02:38:08.228 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.228 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.228 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:08.228 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.308031+0000", 2020-06-12T02:38:08.228 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00043489400000000001 2020-06-12T02:38:08.228 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.229 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.229 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:08.230 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.308466+0000", 2020-06-12T02:38:08.230 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0018799070000000001 2020-06-12T02:38:08.230 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.230 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.231 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:08.231 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.310346+0000", 2020-06-12T02:38:08.231 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0028380839999999998 2020-06-12T02:38:08.231 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.232 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.232 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:08.232 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.313184+0000", 2020-06-12T02:38:08.232 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5617000000000003e-05 2020-06-12T02:38:08.232 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.233 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.233 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:08.233 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.313279+0000", 2020-06-12T02:38:08.233 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012861745000000001 2020-06-12T02:38:08.233 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:08.234 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:08.234 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:08.234 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.234 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.234 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:3219 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:38:08.235 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:38.300493+0000", 2020-06-12T02:38:08.235 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 29.803183383, 2020-06-12T02:38:08.235 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018386370999999999, 2020-06-12T02:38:08.235 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:08.235 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:08.236 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:08.236 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:08.236 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:08.236 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 3219 2020-06-12T02:38:08.236 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.237 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:08.237 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.237 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:08.237 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300493+0000", 2020-06-12T02:38:08.237 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:08.238 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.238 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.238 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:08.238 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300493+0000", 2020-06-12T02:38:08.239 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3008000000000001e-05 2020-06-12T02:38:08.239 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.239 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.239 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:08.239 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300516+0000", 2020-06-12T02:38:08.240 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1329999999999999e-06 2020-06-12T02:38:08.240 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.240 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.240 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:08.240 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300519+0000", 2020-06-12T02:38:08.241 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8260000000000004e-06 2020-06-12T02:38:08.241 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.241 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.241 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:08.242 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300526+0000", 2020-06-12T02:38:08.242 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.525e-06 2020-06-12T02:38:08.242 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.242 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.242 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:08.243 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300528+0000", 2020-06-12T02:38:08.243 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9839e-05 2020-06-12T02:38:08.243 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.243 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.243 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:08.243 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300558+0000", 2020-06-12T02:38:08.244 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3198000000000001e-05 2020-06-12T02:38:08.244 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.244 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.244 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:08.245 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300611+0000", 2020-06-12T02:38:08.245 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0082281509999999995 2020-06-12T02:38:08.245 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.246 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.246 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:08.246 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.308840+0000", 2020-06-12T02:38:08.246 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5921000000000005e-05 2020-06-12T02:38:08.246 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.247 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.247 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:08.247 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.308935+0000", 2020-06-12T02:38:08.247 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050007560000000003 2020-06-12T02:38:08.247 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.247 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.248 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:08.248 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.313936+0000", 2020-06-12T02:38:08.248 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1600000000000007e-05 2020-06-12T02:38:08.248 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.249 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.249 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:08.249 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.313998+0000", 2020-06-12T02:38:08.249 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0048814139999999997 2020-06-12T02:38:08.250 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.250 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.250 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:08.250 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.318879+0000", 2020-06-12T02:38:08.250 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018386370999999999 2020-06-12T02:38:08.251 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:08.251 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:08.251 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:08.252 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.252 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.252 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:4478 3.as0 3:5cb723d2:::smithi10729407-15:head [append 0~483328] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-06-12T02:38:08.252 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:42.522285+0000", 2020-06-12T02:38:08.252 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 25.581391419999999, 2020-06-12T02:38:08.253 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013258748000000001, 2020-06-12T02:38:08.253 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:08.253 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:08.253 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:08.253 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:08.254 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:08.254 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 4478 2020-06-12T02:38:08.254 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.254 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:08.254 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.254 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:08.255 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.522285+0000", 2020-06-12T02:38:08.255 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:08.255 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.255 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.255 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:08.256 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.522285+0000", 2020-06-12T02:38:08.256 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00055732500000000001 2020-06-12T02:38:08.256 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.256 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.256 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:08.257 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.522842+0000", 2020-06-12T02:38:08.257 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1270000000000002e-06 2020-06-12T02:38:08.257 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.257 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.258 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:08.258 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.522847+0000", 2020-06-12T02:38:08.258 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1314e-05 2020-06-12T02:38:08.258 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.258 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.259 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:08.259 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.522859+0000", 2020-06-12T02:38:08.259 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4339999999999997e-06 2020-06-12T02:38:08.259 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.259 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.259 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:08.260 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.522865+0000", 2020-06-12T02:38:08.260 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033831099999999999 2020-06-12T02:38:08.260 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.260 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.261 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:08.261 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.526248+0000", 2020-06-12T02:38:08.261 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010359199999999999 2020-06-12T02:38:08.261 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.262 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.262 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:08.262 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.526352+0000", 2020-06-12T02:38:08.262 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031925009999999999 2020-06-12T02:38:08.262 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.263 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.263 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:08.263 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.529544+0000", 2020-06-12T02:38:08.263 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6626000000000003e-05 2020-06-12T02:38:08.263 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.264 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.264 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:08.264 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.529601+0000", 2020-06-12T02:38:08.264 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047735099999999999 2020-06-12T02:38:08.264 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.265 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.265 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:08.265 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.530078+0000", 2020-06-12T02:38:08.265 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0053046090000000001 2020-06-12T02:38:08.265 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.266 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.267 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:08.267 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.535383+0000", 2020-06-12T02:38:08.267 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011821599999999999 2020-06-12T02:38:08.267 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.268 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.268 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:08.268 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.535501+0000", 2020-06-12T02:38:08.268 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2543000000000002e-05 2020-06-12T02:38:08.269 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.269 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.269 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:08.269 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.535544+0000", 2020-06-12T02:38:08.269 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013258748000000001 2020-06-12T02:38:08.270 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:08.270 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:08.270 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:08.270 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.270 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.270 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:4535 3.as0 3:5732055a:::smithi10729407-8:head [append 0~712704] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-06-12T02:38:08.271 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:42.670365+0000", 2020-06-12T02:38:08.271 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 25.433311089, 2020-06-12T02:38:08.271 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013998371000000001, 2020-06-12T02:38:08.271 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:08.271 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:08.272 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:08.272 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:08.274 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:08.274 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 4535 2020-06-12T02:38:08.274 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.274 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:08.274 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.275 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:08.275 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.670365+0000", 2020-06-12T02:38:08.275 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:08.275 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.275 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.276 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:08.276 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.670365+0000", 2020-06-12T02:38:08.276 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00059204600000000004 2020-06-12T02:38:08.276 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.276 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.276 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:08.277 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.670957+0000", 2020-06-12T02:38:08.277 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.122e-06 2020-06-12T02:38:08.277 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.277 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.277 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:08.278 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.670963+0000", 2020-06-12T02:38:08.278 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1518999999999999e-05 2020-06-12T02:38:08.278 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.278 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.278 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:08.279 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.670975+0000", 2020-06-12T02:38:08.279 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9290000000000001e-06 2020-06-12T02:38:08.279 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.279 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.279 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:08.279 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.670982+0000", 2020-06-12T02:38:08.280 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4758000000000001e-05 2020-06-12T02:38:08.280 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.280 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.280 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:08.280 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.671027+0000", 2020-06-12T02:38:08.281 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9733999999999993e-05 2020-06-12T02:38:08.281 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.281 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.281 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:08.281 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.671096+0000", 2020-06-12T02:38:08.282 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000544448 2020-06-12T02:38:08.282 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.283 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.283 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:08.284 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.671641+0000", 2020-06-12T02:38:08.284 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012592534000000001 2020-06-12T02:38:08.284 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.284 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.284 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:08.285 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.684233+0000", 2020-06-12T02:38:08.285 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9785999999999998e-05 2020-06-12T02:38:08.285 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.285 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.285 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:08.286 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.684313+0000", 2020-06-12T02:38:08.286 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0494999999999997e-05 2020-06-12T02:38:08.286 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.286 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.286 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:08.287 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.684364+0000", 2020-06-12T02:38:08.287 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013998371000000001 2020-06-12T02:38:08.287 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:08.287 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:08.287 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:08.288 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.288 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.288 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:4975 3.as0 3:5732055a:::smithi10729407-8:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e30)", 2020-06-12T02:38:08.288 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:43.721794+0000", 2020-06-12T02:38:08.288 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 24.381882596000001, 2020-06-12T02:38:08.288 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.1825135, 2020-06-12T02:38:08.289 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:08.289 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:08.289 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:08.289 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:08.289 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:08.290 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 4975 2020-06-12T02:38:08.290 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.290 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:08.290 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.290 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:08.290 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721794+0000", 2020-06-12T02:38:08.291 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:08.291 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.291 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.291 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:08.291 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721794+0000", 2020-06-12T02:38:08.292 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4542e-05 2020-06-12T02:38:08.292 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.292 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.292 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:08.292 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721818+0000", 2020-06-12T02:38:08.292 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9169999999999997e-06 2020-06-12T02:38:08.293 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.293 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.294 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:08.294 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721823+0000", 2020-06-12T02:38:08.294 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1184e-05 2020-06-12T02:38:08.294 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.295 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.295 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:08.295 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721834+0000", 2020-06-12T02:38:08.295 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.254e-06 2020-06-12T02:38:08.295 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.296 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.296 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:08.296 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721840+0000", 2020-06-12T02:38:08.296 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1075999999999998e-05 2020-06-12T02:38:08.296 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.297 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.297 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:08.297 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721881+0000", 2020-06-12T02:38:08.297 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7416999999999995e-05 2020-06-12T02:38:08.297 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.297 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.298 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:08.298 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721948+0000", 2020-06-12T02:38:08.298 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.18235910999999999 2020-06-12T02:38:08.298 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.298 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.299 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:08.299 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.904307+0000", 2020-06-12T02:38:08.299 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.1825135 2020-06-12T02:38:08.299 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:08.299 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:08.299 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:08.300 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.300 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.300 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:6205 3.as0 3:5cb723d2:::smithi10729407-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e31)", 2020-06-12T02:38:08.300 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:47.353583+0000", 2020-06-12T02:38:08.300 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.750093386, 2020-06-12T02:38:08.301 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013040737, 2020-06-12T02:38:08.301 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:08.301 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:08.301 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:08.301 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:08.302 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:08.302 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 6205 2020-06-12T02:38:08.302 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.302 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:08.302 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.302 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:08.303 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.353583+0000", 2020-06-12T02:38:08.303 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:08.303 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.303 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.304 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:08.304 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.353583+0000", 2020-06-12T02:38:08.304 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3133000000000002e-05 2020-06-12T02:38:08.304 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.304 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.305 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:08.305 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.353626+0000", 2020-06-12T02:38:08.305 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9570000000000008e-06 2020-06-12T02:38:08.305 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.305 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.305 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:08.306 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.353635+0000", 2020-06-12T02:38:08.306 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1754e-05 2020-06-12T02:38:08.306 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.306 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.306 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:08.307 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.353647+0000", 2020-06-12T02:38:08.307 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0809999999999998e-06 2020-06-12T02:38:08.307 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.307 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.307 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:08.308 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.353650+0000", 2020-06-12T02:38:08.308 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0064807490000000001 2020-06-12T02:38:08.308 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.308 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.308 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:08.308 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.360131+0000", 2020-06-12T02:38:08.309 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7163000000000001e-05 2020-06-12T02:38:08.309 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.309 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.309 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:08.310 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.360188+0000", 2020-06-12T02:38:08.310 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0053632230000000003 2020-06-12T02:38:08.310 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.310 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.310 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:08.311 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.365551+0000", 2020-06-12T02:38:08.311 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8078999999999997e-05 2020-06-12T02:38:08.311 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.311 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.311 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:08.311 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.365609+0000", 2020-06-12T02:38:08.312 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001014598 2020-06-12T02:38:08.312 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.312 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.312 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:08.312 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.366624+0000", 2020-06-12T02:38:08.313 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013040737 2020-06-12T02:38:08.313 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:08.313 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:08.313 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:08.313 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.314 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.314 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:6678 3.as0 3:5732055a:::smithi10729407-8:head [append 0~499712] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-06-12T02:38:08.314 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:48.626196+0000", 2020-06-12T02:38:08.314 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 19.477480613000001, 2020-06-12T02:38:08.315 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014093700000000001, 2020-06-12T02:38:08.315 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:08.315 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:08.315 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:08.315 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:08.315 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:08.316 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 6678 2020-06-12T02:38:08.316 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.316 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:08.316 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.316 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:08.317 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.626196+0000", 2020-06-12T02:38:08.317 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:08.317 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.317 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.317 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:08.317 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.626196+0000", 2020-06-12T02:38:08.318 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00046254199999999998 2020-06-12T02:38:08.318 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.318 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.318 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:08.318 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.626658+0000", 2020-06-12T02:38:08.319 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3400000000000003e-06 2020-06-12T02:38:08.319 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.319 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.319 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:08.319 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.626665+0000", 2020-06-12T02:38:08.319 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0079e-05 2020-06-12T02:38:08.320 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.320 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.320 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:08.320 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.626675+0000", 2020-06-12T02:38:08.320 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0999999999999997e-06 2020-06-12T02:38:08.321 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.321 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.321 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:08.321 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.626679+0000", 2020-06-12T02:38:08.321 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025692100000000002 2020-06-12T02:38:08.321 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.322 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.322 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:08.322 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.629248+0000", 2020-06-12T02:38:08.322 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000107944 2020-06-12T02:38:08.322 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.323 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.323 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:08.323 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.629356+0000", 2020-06-12T02:38:08.323 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001434999 2020-06-12T02:38:08.323 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.324 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.324 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:08.324 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.630791+0000", 2020-06-12T02:38:08.324 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0716e-05 2020-06-12T02:38:08.325 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.325 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.325 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:08.325 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.630842+0000", 2020-06-12T02:38:08.325 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00045107600000000002 2020-06-12T02:38:08.325 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.326 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.326 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:08.326 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.631293+0000", 2020-06-12T02:38:08.326 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0048208330000000001 2020-06-12T02:38:08.326 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.327 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.327 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:08.327 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.636113+0000", 2020-06-12T02:38:08.327 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0041129679999999998 2020-06-12T02:38:08.327 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.328 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.328 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:08.328 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.640226+0000", 2020-06-12T02:38:08.328 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2892999999999996e-05 2020-06-12T02:38:08.328 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.328 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.329 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:08.329 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.640289+0000", 2020-06-12T02:38:08.329 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014093700000000001 2020-06-12T02:38:08.329 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:08.329 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:08.330 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:08.330 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.330 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.330 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:7084 3.as0 3:5732055a:::smithi10729407-8:head [append 0~401408] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-06-12T02:38:08.330 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:49.666656+0000", 2020-06-12T02:38:08.330 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 18.437019869, 2020-06-12T02:38:08.331 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013133027, 2020-06-12T02:38:08.331 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:08.331 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:08.331 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:08.331 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:08.332 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:08.332 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 7084 2020-06-12T02:38:08.332 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.332 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:08.332 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.332 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:08.333 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.666656+0000", 2020-06-12T02:38:08.333 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:08.333 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.333 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.333 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:08.334 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.666656+0000", 2020-06-12T02:38:08.334 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00040755700000000001 2020-06-12T02:38:08.334 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.334 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.335 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:08.335 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667064+0000", 2020-06-12T02:38:08.335 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9479999999999999e-06 2020-06-12T02:38:08.335 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.335 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.336 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:08.336 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667069+0000", 2020-06-12T02:38:08.336 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.568e-06 2020-06-12T02:38:08.336 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.336 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.336 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:08.337 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667077+0000", 2020-06-12T02:38:08.337 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4170000000000002e-06 2020-06-12T02:38:08.337 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.337 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.337 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:08.338 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667082+0000", 2020-06-12T02:38:08.338 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0034817770000000001 2020-06-12T02:38:08.338 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.338 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.338 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:08.338 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.670564+0000", 2020-06-12T02:38:08.339 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011071200000000001 2020-06-12T02:38:08.339 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.339 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.339 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:08.339 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.670674+0000", 2020-06-12T02:38:08.340 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0011185780000000001 2020-06-12T02:38:08.340 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.340 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.340 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:08.340 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.671793+0000", 2020-06-12T02:38:08.340 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2209e-05 2020-06-12T02:38:08.341 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.341 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.341 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:08.341 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.671855+0000", 2020-06-12T02:38:08.341 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054805100000000003 2020-06-12T02:38:08.342 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.342 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.342 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:08.342 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.672403+0000", 2020-06-12T02:38:08.342 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0044889480000000004 2020-06-12T02:38:08.342 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.343 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.343 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:08.343 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.676892+0000", 2020-06-12T02:38:08.343 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0028428730000000001 2020-06-12T02:38:08.343 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.344 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.344 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:08.344 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.679735+0000", 2020-06-12T02:38:08.344 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4388999999999997e-05 2020-06-12T02:38:08.345 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.345 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.345 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:08.345 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.679789+0000", 2020-06-12T02:38:08.345 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013133027 2020-06-12T02:38:08.346 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:08.346 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:08.346 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:08.347 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.347 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.347 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:7085 3.as0 3:5732055a:::smithi10729407-8:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-06-12T02:38:08.347 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:49.667120+0000", 2020-06-12T02:38:08.347 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 18.436556634999999, 2020-06-12T02:38:08.347 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012883739, 2020-06-12T02:38:08.348 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:08.348 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:08.348 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:08.348 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:08.348 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:08.349 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 7085 2020-06-12T02:38:08.349 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.349 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:08.349 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.349 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:08.350 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667120+0000", 2020-06-12T02:38:08.350 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:08.350 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.350 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.350 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:08.351 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667120+0000", 2020-06-12T02:38:08.351 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.773e-05 2020-06-12T02:38:08.351 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.351 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.351 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:08.352 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667157+0000", 2020-06-12T02:38:08.352 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7740000000000002e-06 2020-06-12T02:38:08.352 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.352 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.352 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:08.352 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667161+0000", 2020-06-12T02:38:08.353 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1190000000000002e-06 2020-06-12T02:38:08.353 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.353 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.353 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:08.353 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667167+0000", 2020-06-12T02:38:08.354 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6970000000000002e-06 2020-06-12T02:38:08.354 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.354 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.354 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:08.354 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667170+0000", 2020-06-12T02:38:08.354 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036418029999999999 2020-06-12T02:38:08.355 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.355 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.355 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:08.356 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.670812+0000", 2020-06-12T02:38:08.356 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000107107 2020-06-12T02:38:08.356 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.356 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.356 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:08.356 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.670919+0000", 2020-06-12T02:38:08.357 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0034625480000000002 2020-06-12T02:38:08.357 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.357 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.357 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:08.357 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.674381+0000", 2020-06-12T02:38:08.358 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010428 2020-06-12T02:38:08.358 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.358 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.358 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:08.358 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.674486+0000", 2020-06-12T02:38:08.358 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000426382 2020-06-12T02:38:08.359 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.359 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.359 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:08.359 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.674912+0000", 2020-06-12T02:38:08.359 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0044864739999999998 2020-06-12T02:38:08.360 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.360 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.360 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:08.360 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.679399+0000", 2020-06-12T02:38:08.360 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00051632499999999999 2020-06-12T02:38:08.360 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.361 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.361 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:08.361 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.679915+0000", 2020-06-12T02:38:08.361 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8499999999999996e-05 2020-06-12T02:38:08.361 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.362 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.362 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:08.362 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.680003+0000", 2020-06-12T02:38:08.362 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012883739 2020-06-12T02:38:08.362 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:08.363 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:08.363 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:08.363 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.363 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.363 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:7086 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e32)", 2020-06-12T02:38:08.363 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:49.667192+0000", 2020-06-12T02:38:08.364 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 18.436484283999999, 2020-06-12T02:38:08.364 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014618023000000001, 2020-06-12T02:38:08.364 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:08.364 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:08.364 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:08.365 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:08.365 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:08.365 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 7086 2020-06-12T02:38:08.365 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.366 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:08.366 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.366 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:08.366 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667192+0000", 2020-06-12T02:38:08.366 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:08.367 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.367 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.367 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:08.367 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667192+0000", 2020-06-12T02:38:08.367 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1912e-05 2020-06-12T02:38:08.367 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.368 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.368 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:08.368 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667214+0000", 2020-06-12T02:38:08.368 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0469999999999998e-06 2020-06-12T02:38:08.368 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.369 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.369 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:08.369 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667217+0000", 2020-06-12T02:38:08.369 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7170000000000002e-06 2020-06-12T02:38:08.369 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.370 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.370 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:08.370 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667224+0000", 2020-06-12T02:38:08.370 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2850000000000001e-06 2020-06-12T02:38:08.370 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.370 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.371 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:08.371 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667226+0000", 2020-06-12T02:38:08.371 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0037197950000000001 2020-06-12T02:38:08.371 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.371 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.372 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:08.372 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.670946+0000", 2020-06-12T02:38:08.372 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4549999999999994e-05 2020-06-12T02:38:08.372 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.372 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.372 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:08.373 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.671040+0000", 2020-06-12T02:38:08.373 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0043129199999999996 2020-06-12T02:38:08.373 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.373 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.373 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:08.374 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.675353+0000", 2020-06-12T02:38:08.374 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1982000000000001e-05 2020-06-12T02:38:08.374 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.374 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.374 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:08.374 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.675445+0000", 2020-06-12T02:38:08.375 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.005067401 2020-06-12T02:38:08.375 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.375 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.375 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:08.376 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.680513+0000", 2020-06-12T02:38:08.376 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000101508 2020-06-12T02:38:08.376 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.376 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.376 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:08.377 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.680614+0000", 2020-06-12T02:38:08.377 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0011959060000000001 2020-06-12T02:38:08.377 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.377 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.377 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:08.378 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.681810+0000", 2020-06-12T02:38:08.378 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014618023000000001 2020-06-12T02:38:08.378 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:08.378 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:08.379 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:08.379 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.379 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.379 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:7902 3.as0 3:5732055a:::smithi10729407-8:head [append 0~499712] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-06-12T02:38:08.380 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.159351+0000", 2020-06-12T02:38:08.380 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 15.944325607, 2020-06-12T02:38:08.380 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013425812000000001, 2020-06-12T02:38:08.380 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:08.380 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:08.381 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:08.381 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:08.381 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:08.381 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 7902 2020-06-12T02:38:08.381 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.381 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:08.382 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.382 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:08.382 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.159351+0000", 2020-06-12T02:38:08.382 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:08.382 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.383 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.383 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:08.383 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.159351+0000", 2020-06-12T02:38:08.383 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00048251199999999999 2020-06-12T02:38:08.383 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.384 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.384 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:08.384 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.159833+0000", 2020-06-12T02:38:08.384 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7509999999999997e-06 2020-06-12T02:38:08.384 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.384 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.385 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:08.385 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.159839+0000", 2020-06-12T02:38:08.385 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9840000000000007e-06 2020-06-12T02:38:08.385 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.386 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.386 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:08.386 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.159848+0000", 2020-06-12T02:38:08.386 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4619999999999998e-06 2020-06-12T02:38:08.387 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.387 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.387 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:08.387 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.159852+0000", 2020-06-12T02:38:08.387 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5348000000000001e-05 2020-06-12T02:38:08.387 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.388 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.388 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:08.388 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.159888+0000", 2020-06-12T02:38:08.388 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9824000000000001e-05 2020-06-12T02:38:08.388 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.389 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.389 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:08.389 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.159958+0000", 2020-06-12T02:38:08.389 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0014605149999999999 2020-06-12T02:38:08.389 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.389 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.390 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:08.390 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.161418+0000", 2020-06-12T02:38:08.390 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0667999999999997e-05 2020-06-12T02:38:08.390 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.390 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.391 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:08.391 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.161469+0000", 2020-06-12T02:38:08.391 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00058577200000000003 2020-06-12T02:38:08.391 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.391 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.391 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:08.392 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.162054+0000", 2020-06-12T02:38:08.392 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010590761000000001 2020-06-12T02:38:08.392 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.392 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.392 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:08.393 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.172645+0000", 2020-06-12T02:38:08.393 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8467000000000006e-05 2020-06-12T02:38:08.393 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.393 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.393 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:08.394 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.172724+0000", 2020-06-12T02:38:08.394 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2747999999999997e-05 2020-06-12T02:38:08.394 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.394 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.394 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:08.394 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.172776+0000", 2020-06-12T02:38:08.395 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013425812000000001 2020-06-12T02:38:08.395 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:08.395 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:08.395 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:08.396 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.396 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.396 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:7907 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e33)", 2020-06-12T02:38:08.396 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.166559+0000", 2020-06-12T02:38:08.397 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 15.937116906, 2020-06-12T02:38:08.397 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013016145999999999, 2020-06-12T02:38:08.397 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:08.397 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:08.397 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:08.398 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:08.398 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:08.398 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 7907 2020-06-12T02:38:08.398 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.398 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:08.399 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.399 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:08.399 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.166559+0000", 2020-06-12T02:38:08.399 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:08.399 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.399 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.400 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:08.400 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.166559+0000", 2020-06-12T02:38:08.400 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1719999999999999e-05 2020-06-12T02:38:08.400 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.400 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.401 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:08.401 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.166581+0000", 2020-06-12T02:38:08.401 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8660000000000002e-06 2020-06-12T02:38:08.401 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.401 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.402 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:08.402 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.166584+0000", 2020-06-12T02:38:08.402 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3439999999999997e-06 2020-06-12T02:38:08.402 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.402 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.402 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:08.403 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.166590+0000", 2020-06-12T02:38:08.403 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0640000000000001e-06 2020-06-12T02:38:08.403 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.403 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.403 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:08.404 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.166592+0000", 2020-06-12T02:38:08.404 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0074059160000000002 2020-06-12T02:38:08.404 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.404 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.407 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:08.407 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.173998+0000", 2020-06-12T02:38:08.407 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8678999999999998e-05 2020-06-12T02:38:08.407 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.407 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.408 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:08.408 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.174057+0000", 2020-06-12T02:38:08.408 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045994779999999997 2020-06-12T02:38:08.408 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.408 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.408 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:08.409 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.178656+0000", 2020-06-12T02:38:08.409 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6825999999999999e-05 2020-06-12T02:38:08.409 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.409 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.409 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:08.410 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.178733+0000", 2020-06-12T02:38:08.410 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00084225299999999995 2020-06-12T02:38:08.410 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.410 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.410 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:08.411 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.179575+0000", 2020-06-12T02:38:08.411 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013016145999999999 2020-06-12T02:38:08.411 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:08.411 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:08.411 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:08.411 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.412 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.412 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:7931 3.as0 3:5732055a:::smithi10729407-8:head [append 0~696320] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-06-12T02:38:08.412 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.246494+0000", 2020-06-12T02:38:08.412 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 15.85718179, 2020-06-12T02:38:08.412 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014082301, 2020-06-12T02:38:08.413 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:08.413 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:08.413 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:08.413 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:08.413 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:08.413 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 7931 2020-06-12T02:38:08.414 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.414 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:08.414 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.414 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:08.414 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.246494+0000", 2020-06-12T02:38:08.415 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:08.415 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.415 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.415 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:08.415 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.246494+0000", 2020-06-12T02:38:08.416 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00077570099999999995 2020-06-12T02:38:08.417 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.417 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.417 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:08.417 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.247270+0000", 2020-06-12T02:38:08.417 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4179999999999997e-06 2020-06-12T02:38:08.418 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.418 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.418 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:08.418 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.247279+0000", 2020-06-12T02:38:08.418 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6297999999999999e-05 2020-06-12T02:38:08.419 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.419 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.419 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:08.419 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.247295+0000", 2020-06-12T02:38:08.419 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3309999999999998e-06 2020-06-12T02:38:08.420 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.420 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.420 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:08.420 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.247302+0000", 2020-06-12T02:38:08.420 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4283999999999999e-05 2020-06-12T02:38:08.420 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.421 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.421 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:08.421 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.247336+0000", 2020-06-12T02:38:08.421 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8157999999999994e-05 2020-06-12T02:38:08.421 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.422 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.422 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:08.422 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.247405+0000", 2020-06-12T02:38:08.422 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031257059999999998 2020-06-12T02:38:08.422 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.422 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.423 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:08.423 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.250530+0000", 2020-06-12T02:38:08.423 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7546999999999995e-05 2020-06-12T02:38:08.423 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.423 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.424 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:08.424 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.250608+0000", 2020-06-12T02:38:08.424 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00074126599999999997 2020-06-12T02:38:08.424 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.424 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.425 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:08.425 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.251349+0000", 2020-06-12T02:38:08.425 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.008999211 2020-06-12T02:38:08.425 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.425 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.425 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:08.426 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.260348+0000", 2020-06-12T02:38:08.426 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00016373200000000001 2020-06-12T02:38:08.426 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.426 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.426 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:08.427 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.260512+0000", 2020-06-12T02:38:08.427 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4648999999999997e-05 2020-06-12T02:38:08.427 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.427 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.427 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:08.427 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.260577+0000", 2020-06-12T02:38:08.428 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014082301 2020-06-12T02:38:08.428 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:08.428 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:08.428 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:08.428 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.429 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.429 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8129 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e33)", 2020-06-12T02:38:08.429 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.700485+0000", 2020-06-12T02:38:08.429 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 15.403191421000001, 2020-06-12T02:38:08.429 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01339515, 2020-06-12T02:38:08.430 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:08.430 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:08.430 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:08.430 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:08.430 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:08.431 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8129 2020-06-12T02:38:08.431 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.431 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:08.431 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.431 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:08.431 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.700485+0000", 2020-06-12T02:38:08.432 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:08.432 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.432 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.432 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:08.432 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.700485+0000", 2020-06-12T02:38:08.433 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4323999999999998e-05 2020-06-12T02:38:08.433 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.433 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.433 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:08.433 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.700529+0000", 2020-06-12T02:38:08.433 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7100000000000004e-06 2020-06-12T02:38:08.434 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.434 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.434 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:08.434 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.700535+0000", 2020-06-12T02:38:08.434 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1664999999999999e-05 2020-06-12T02:38:08.435 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.435 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.435 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:08.435 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.700547+0000", 2020-06-12T02:38:08.435 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6150000000000001e-06 2020-06-12T02:38:08.435 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.436 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.436 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:08.436 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.700550+0000", 2020-06-12T02:38:08.436 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00049316500000000005 2020-06-12T02:38:08.436 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.437 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.437 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:08.437 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.701043+0000", 2020-06-12T02:38:08.437 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0063999999999997e-05 2020-06-12T02:38:08.437 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.438 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.438 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:08.438 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.701133+0000", 2020-06-12T02:38:08.438 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0085745109999999999 2020-06-12T02:38:08.438 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.438 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.439 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:08.439 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.709708+0000", 2020-06-12T02:38:08.439 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8854999999999999e-05 2020-06-12T02:38:08.439 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.439 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.440 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:08.440 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.709757+0000", 2020-06-12T02:38:08.440 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002613279 2020-06-12T02:38:08.440 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.440 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.440 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:08.441 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.712370+0000", 2020-06-12T02:38:08.441 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4332000000000004e-05 2020-06-12T02:38:08.442 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.442 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.442 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:08.442 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.712464+0000", 2020-06-12T02:38:08.442 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00141563 2020-06-12T02:38:08.443 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:08.443 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:08.443 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:08.443 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.713880+0000", 2020-06-12T02:38:08.443 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01339515 2020-06-12T02:38:08.444 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:08.444 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:08.444 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:08.444 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:08.444 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:08.444 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:08.445 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-06-12T02:38:08.447 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:08.444+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:08.447 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-06-12T02:38:08.547 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:08.545+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:08.588 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:38:08.611 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:08.611 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:38:08.611 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2020-06-12T02:38:08.612 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:08.612 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:38:08.623 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-06-12T02:38:08.627 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:38:08.648 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:38:08.648 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:08.646+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:08.668 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:38:08.688 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:38:08.707 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:38:08.728 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2020-06-12T02:38:08.729 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-06-12T02:38:08.729 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-06-12T02:38:08.729 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-06-12T02:38:08.729 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-06-12T02:38:08.730 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-06-12T02:38:08.730 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-06-12T02:38:08.730 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-06-12T02:38:08.731 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-06-12T02:38:08.743 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:08.744 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:08.752 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-06-12T02:38:08.781 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:08.777+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:08.850 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:08.848+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:08.867 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:08.868 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:08.876 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-06-12T02:38:08.951 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:38:08.949+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:38:08.995 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:08.995 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:08.997 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd unset nodeep-scrub 2020-06-12T02:38:09.010 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-06-12T02:38:09.053 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:09.050+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:09.153 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:09.151+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:09.170 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:09.171 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:09.179 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-06-12T02:38:09.255 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:09.253+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:09.298 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:09.298 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:09.306 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-06-12T02:38:09.358 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:09.356+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:09.459 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:38:09.457+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:38:09.464 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:09.464 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:09.473 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-06-12T02:38:09.560 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:09.558+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:09.590 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:09.590 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:09.598 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-06-12T02:38:09.644 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2020-06-12T02:38:09.660 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:09.658+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:09.756 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:09.756 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:09.761 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:09.758+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:09.764 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-06-12T02:38:09.862 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:09.860+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:09.880 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:09.880 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:09.895 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-06-12T02:38:09.978 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:09.976+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:10.051 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:10.051 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:10.060 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-06-12T02:38:10.063 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:10.061+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:10.164 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:10.162+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:10.218 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:10.219 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:10.227 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-06-12T02:38:10.268 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:10.266+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:10.365 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:10.364+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:10.383 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:10.384 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:10.392 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-06-12T02:38:10.466 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:10.464+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:10.548 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:10.548 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:10.556 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-06-12T02:38:10.570 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:10.568+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:10.672 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:10.670+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:10.675 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:10.675 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:10.684 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-06-12T02:38:10.745 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-06-12T02:38:10.790 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:10.787+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:10.813 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:10.813 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:10.821 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-06-12T02:38:10.874 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:10.871+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:10.876 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:38:10.898 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:38:10.917 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:38:10.936 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:38:10.939 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:10.939 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:10.947 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-06-12T02:38:10.956 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:38:10.975 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:10.972+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:10.976 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:38:10.996 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:38:11.016 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2020-06-12T02:38:11.017 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-06-12T02:38:11.018 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-06-12T02:38:11.018 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-06-12T02:38:11.018 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-06-12T02:38:11.019 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-06-12T02:38:11.019 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-06-12T02:38:11.019 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-06-12T02:38:11.019 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-06-12T02:38:11.082 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:11.079+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:11.105 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:11.106 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:38:11.106 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:38:11.106 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:38:11.106 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.106 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:858 3.3s0 3:ce45c70a:::smithi10729407-7:head [append 0~565248] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:38:11.107 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.924643+0000", 2020-06-12T02:38:11.107 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 39.176925367999999, 2020-06-12T02:38:11.107 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015332551999999999, 2020-06-12T02:38:11.107 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:11.107 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:11.108 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:11.108 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:11.108 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:11.108 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 858 2020-06-12T02:38:11.108 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.109 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:11.109 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.109 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:11.109 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.924643+0000", 2020-06-12T02:38:11.109 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:11.109 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.110 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.110 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:11.110 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.924643+0000", 2020-06-12T02:38:11.110 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00067535299999999996 2020-06-12T02:38:11.110 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.111 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.112 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:11.113 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925318+0000", 2020-06-12T02:38:11.113 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9629999999999993e-06 2020-06-12T02:38:11.113 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.114 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.114 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:11.114 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925328+0000", 2020-06-12T02:38:11.114 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7357999999999999e-05 2020-06-12T02:38:11.114 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.115 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.115 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:11.115 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925345+0000", 2020-06-12T02:38:11.115 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7019999999999996e-06 2020-06-12T02:38:11.115 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.116 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.116 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:11.116 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925355+0000", 2020-06-12T02:38:11.116 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6289999999999998e-05 2020-06-12T02:38:11.116 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.117 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.117 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.117 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925411+0000", 2020-06-12T02:38:11.117 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8855000000000004e-05 2020-06-12T02:38:11.117 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.117 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.118 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:11.118 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.925480+0000", 2020-06-12T02:38:11.118 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0074521149999999996 2020-06-12T02:38:11.118 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.118 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.119 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.119 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.932932+0000", 2020-06-12T02:38:11.119 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3647999999999995e-05 2020-06-12T02:38:11.119 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.119 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.120 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:11.120 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.933026+0000", 2020-06-12T02:38:11.120 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00078753199999999997 2020-06-12T02:38:11.120 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.120 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.121 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:11.121 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.933813+0000", 2020-06-12T02:38:11.121 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.006013582 2020-06-12T02:38:11.122 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.122 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.122 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:11.122 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.939827+0000", 2020-06-12T02:38:11.123 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0111999999999996e-05 2020-06-12T02:38:11.123 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.123 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.123 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:11.123 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.939897+0000", 2020-06-12T02:38:11.124 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8041999999999996e-05 2020-06-12T02:38:11.124 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.124 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.124 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:11.124 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.939975+0000", 2020-06-12T02:38:11.124 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015332551999999999 2020-06-12T02:38:11.125 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.125 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:11.125 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.125 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.125 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.126 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:860 3.3s0 3:ce45c70a:::smithi10729407-7:head [append 0~81920] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:38:11.126 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.928619+0000", 2020-06-12T02:38:11.126 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 39.172949482, 2020-06-12T02:38:11.126 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017431941999999999, 2020-06-12T02:38:11.126 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:11.127 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:11.127 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:11.127 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:11.127 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:11.127 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 860 2020-06-12T02:38:11.127 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.128 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:11.128 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.128 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:11.128 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928619+0000", 2020-06-12T02:38:11.128 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:11.129 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.129 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.129 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:11.129 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928619+0000", 2020-06-12T02:38:11.129 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000105809 2020-06-12T02:38:11.129 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.130 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.130 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:11.130 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928724+0000", 2020-06-12T02:38:11.130 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2339999999999994e-06 2020-06-12T02:38:11.130 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.131 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.131 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:11.131 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928733+0000", 2020-06-12T02:38:11.131 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0268000000000001e-05 2020-06-12T02:38:11.132 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.132 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.132 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:11.132 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928743+0000", 2020-06-12T02:38:11.133 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5190000000000003e-06 2020-06-12T02:38:11.133 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.133 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.133 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:11.134 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928747+0000", 2020-06-12T02:38:11.134 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0103999999999998e-05 2020-06-12T02:38:11.134 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.134 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.134 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.135 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928777+0000", 2020-06-12T02:38:11.135 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8249000000000002e-05 2020-06-12T02:38:11.135 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.135 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.135 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:11.136 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928866+0000", 2020-06-12T02:38:11.136 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011151629 2020-06-12T02:38:11.136 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.138 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.138 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.138 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.940017+0000", 2020-06-12T02:38:11.139 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8281e-05 2020-06-12T02:38:11.139 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.139 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.139 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:11.139 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.940076+0000", 2020-06-12T02:38:11.140 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000346736 2020-06-12T02:38:11.140 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.140 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.140 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:11.140 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.940422+0000", 2020-06-12T02:38:11.141 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024135049999999998 2020-06-12T02:38:11.141 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.141 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.141 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:11.141 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.942836+0000", 2020-06-12T02:38:11.141 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003170258 2020-06-12T02:38:11.142 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.142 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.142 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:11.143 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946006+0000", 2020-06-12T02:38:11.143 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4350000000000001e-05 2020-06-12T02:38:11.143 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.144 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.144 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:11.144 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946050+0000", 2020-06-12T02:38:11.144 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017431941999999999 2020-06-12T02:38:11.144 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.145 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:11.146 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.146 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.146 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.147 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:861 3.3s0 3:ce45c70a:::smithi10729407-7:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:38:11.147 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.928778+0000", 2020-06-12T02:38:11.148 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 39.172789868999999, 2020-06-12T02:38:11.148 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017522738, 2020-06-12T02:38:11.148 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:11.151 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:11.152 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:11.153 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:11.153 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:11.153 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 861 2020-06-12T02:38:11.154 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.154 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:11.154 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.154 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:11.154 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928778+0000", 2020-06-12T02:38:11.154 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:11.155 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.155 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.155 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:11.155 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928778+0000", 2020-06-12T02:38:11.155 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8186000000000002e-05 2020-06-12T02:38:11.156 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.156 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.156 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:11.156 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928826+0000", 2020-06-12T02:38:11.156 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6689999999999997e-06 2020-06-12T02:38:11.157 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.157 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.157 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:11.157 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928831+0000", 2020-06-12T02:38:11.157 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.686e-06 2020-06-12T02:38:11.157 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.158 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.158 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:11.158 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928838+0000", 2020-06-12T02:38:11.158 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7159999999999999e-06 2020-06-12T02:38:11.158 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.159 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.159 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:11.159 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928840+0000", 2020-06-12T02:38:11.159 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4212999999999997e-05 2020-06-12T02:38:11.160 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.160 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.160 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.160 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928895+0000", 2020-06-12T02:38:11.160 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5853999999999994e-05 2020-06-12T02:38:11.161 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.161 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.161 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:11.161 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928990+0000", 2020-06-12T02:38:11.161 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012023982000000001 2020-06-12T02:38:11.162 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.162 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.162 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.162 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941014+0000", 2020-06-12T02:38:11.163 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5104e-05 2020-06-12T02:38:11.163 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.164 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.164 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:11.164 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941070+0000", 2020-06-12T02:38:11.164 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00026444499999999998 2020-06-12T02:38:11.164 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.164 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.165 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:11.165 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941334+0000", 2020-06-12T02:38:11.165 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001704075 2020-06-12T02:38:11.165 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.165 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.166 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:11.166 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.943038+0000", 2020-06-12T02:38:11.166 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031891570000000002 2020-06-12T02:38:11.166 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.166 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.167 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:11.167 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946227+0000", 2020-06-12T02:38:11.167 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3651000000000003e-05 2020-06-12T02:38:11.167 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.167 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.167 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:11.168 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946301+0000", 2020-06-12T02:38:11.168 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017522738 2020-06-12T02:38:11.168 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.168 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:11.168 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.169 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.169 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.169 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:862 3.3s0 3:ce45c70a:::smithi10729407-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:38:11.169 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.928876+0000", 2020-06-12T02:38:11.169 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 39.172692372999997, 2020-06-12T02:38:11.169 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019206890000000001, 2020-06-12T02:38:11.170 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:11.170 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:11.170 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:11.170 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:11.170 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:11.171 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 862 2020-06-12T02:38:11.171 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.171 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:11.171 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.171 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:11.171 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928876+0000", 2020-06-12T02:38:11.172 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:11.172 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.172 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.172 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:11.172 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928876+0000", 2020-06-12T02:38:11.173 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9268999999999999e-05 2020-06-12T02:38:11.173 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.174 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.174 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:11.174 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928925+0000", 2020-06-12T02:38:11.174 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8120000000000002e-06 2020-06-12T02:38:11.174 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.175 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.175 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:11.175 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928932+0000", 2020-06-12T02:38:11.175 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3471e-05 2020-06-12T02:38:11.176 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.176 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.176 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:11.176 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928945+0000", 2020-06-12T02:38:11.176 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4279999999999998e-06 2020-06-12T02:38:11.176 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.177 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.177 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:11.177 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928950+0000", 2020-06-12T02:38:11.177 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6358000000000004e-05 2020-06-12T02:38:11.177 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.178 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.178 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.178 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.929016+0000", 2020-06-12T02:38:11.178 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9697999999999996e-05 2020-06-12T02:38:11.178 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.178 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.179 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:11.179 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.929086+0000", 2020-06-12T02:38:11.180 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012409205 2020-06-12T02:38:11.180 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.180 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.180 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.181 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941495+0000", 2020-06-12T02:38:11.181 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9660000000000006e-05 2020-06-12T02:38:11.181 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.181 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.182 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:11.182 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941585+0000", 2020-06-12T02:38:11.182 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050329099999999998 2020-06-12T02:38:11.182 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.183 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.183 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.184 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946617+0000", 2020-06-12T02:38:11.184 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8437999999999996e-05 2020-06-12T02:38:11.184 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.184 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.184 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:11.185 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946706+0000", 2020-06-12T02:38:11.185 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0013766410000000001 2020-06-12T02:38:11.185 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.185 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.186 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:11.186 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.948083+0000", 2020-06-12T02:38:11.186 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019206890000000001 2020-06-12T02:38:11.186 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.186 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:11.186 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.187 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.187 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.187 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:879 3.fs1 3:fc7e0bf7:::smithi10729407-17:head [append 0~491520] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:38:11.187 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.000881+0000", 2020-06-12T02:38:11.187 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 39.100687192000002, 2020-06-12T02:38:11.188 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017891845, 2020-06-12T02:38:11.188 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:11.188 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:11.188 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:11.188 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:11.188 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:11.189 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 879 2020-06-12T02:38:11.189 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.189 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:11.189 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.189 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:11.190 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.000881+0000", 2020-06-12T02:38:11.190 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:11.190 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.190 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.190 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:11.191 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.000881+0000", 2020-06-12T02:38:11.191 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00044151599999999998 2020-06-12T02:38:11.191 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.191 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.191 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:11.191 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001322+0000", 2020-06-12T02:38:11.192 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0210000000000001e-06 2020-06-12T02:38:11.192 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.192 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.192 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:11.192 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001326+0000", 2020-06-12T02:38:11.192 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6659999999999997e-06 2020-06-12T02:38:11.193 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.193 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.193 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:11.194 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001335+0000", 2020-06-12T02:38:11.194 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4299999999999999e-06 2020-06-12T02:38:11.194 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.194 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.194 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:11.195 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001339+0000", 2020-06-12T02:38:11.195 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0028741510000000001 2020-06-12T02:38:11.195 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.195 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.195 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.195 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.004214+0000", 2020-06-12T02:38:11.196 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010480999999999999 2020-06-12T02:38:11.196 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.196 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.196 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:11.196 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.004318+0000", 2020-06-12T02:38:11.197 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021112359999999998 2020-06-12T02:38:11.197 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.197 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.197 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.197 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006430+0000", 2020-06-12T02:38:11.198 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2935000000000001e-05 2020-06-12T02:38:11.198 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.198 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.198 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:11.198 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006483+0000", 2020-06-12T02:38:11.199 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00056411099999999995 2020-06-12T02:38:11.199 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.199 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.199 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:11.199 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.007047+0000", 2020-06-12T02:38:11.199 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0076749369999999997 2020-06-12T02:38:11.200 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.200 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.200 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:11.200 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.014722+0000", 2020-06-12T02:38:11.201 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00395693 2020-06-12T02:38:11.201 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.201 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.201 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:11.201 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.018679+0000", 2020-06-12T02:38:11.202 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4102000000000001e-05 2020-06-12T02:38:11.202 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.202 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.202 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:11.202 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.018773+0000", 2020-06-12T02:38:11.202 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017891845 2020-06-12T02:38:11.203 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.203 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:11.203 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.203 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.203 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.204 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:933 3.es1 3:78f01648:::smithi10729407-5:head [append 0~417792] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:38:11.204 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.156677+0000", 2020-06-12T02:38:11.204 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 38.944891210000002, 2020-06-12T02:38:11.204 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016833311, 2020-06-12T02:38:11.205 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:11.205 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:11.205 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:11.205 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:11.205 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:11.206 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 933 2020-06-12T02:38:11.206 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.206 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:11.206 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.206 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:11.206 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.156677+0000", 2020-06-12T02:38:11.207 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:11.207 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.207 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.207 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:11.207 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.156677+0000", 2020-06-12T02:38:11.208 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00062571700000000003 2020-06-12T02:38:11.208 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.208 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.208 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:11.208 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157303+0000", 2020-06-12T02:38:11.209 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0740000000000003e-06 2020-06-12T02:38:11.209 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.209 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.209 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:11.209 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157307+0000", 2020-06-12T02:38:11.210 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4419999999999998e-06 2020-06-12T02:38:11.210 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.210 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.210 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:11.210 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157315+0000", 2020-06-12T02:38:11.210 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5300000000000001e-06 2020-06-12T02:38:11.211 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.211 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.211 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:11.211 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157319+0000", 2020-06-12T02:38:11.211 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4995e-05 2020-06-12T02:38:11.212 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.212 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.212 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.212 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157354+0000", 2020-06-12T02:38:11.212 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5883000000000006e-05 2020-06-12T02:38:11.212 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.213 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.213 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:11.213 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157419+0000", 2020-06-12T02:38:11.214 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045316280000000002 2020-06-12T02:38:11.214 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.214 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.215 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.215 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.161951+0000", 2020-06-12T02:38:11.215 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3300999999999996e-05 2020-06-12T02:38:11.215 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.215 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.216 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:11.216 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.162014+0000", 2020-06-12T02:38:11.216 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00039050100000000002 2020-06-12T02:38:11.216 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.216 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.217 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:11.217 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.162405+0000", 2020-06-12T02:38:11.217 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050686250000000002 2020-06-12T02:38:11.217 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.217 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.218 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:11.218 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.167474+0000", 2020-06-12T02:38:11.218 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.005942653 2020-06-12T02:38:11.218 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.218 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.218 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:11.219 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.173416+0000", 2020-06-12T02:38:11.219 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3962000000000006e-05 2020-06-12T02:38:11.219 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.219 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.219 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:11.220 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.173510+0000", 2020-06-12T02:38:11.220 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016833311 2020-06-12T02:38:11.220 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.220 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:11.220 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.220 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.221 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.221 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:936 3.es1 3:78f01648:::smithi10729407-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:38:11.221 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.160357+0000", 2020-06-12T02:38:11.221 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 38.941211498999998, 2020-06-12T02:38:11.221 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015435590000000001, 2020-06-12T02:38:11.222 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:11.222 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:11.222 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:11.222 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:11.222 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:11.223 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 936 2020-06-12T02:38:11.223 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.223 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:11.223 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.223 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:11.223 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160357+0000", 2020-06-12T02:38:11.224 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:11.224 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.224 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.224 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:11.225 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160357+0000", 2020-06-12T02:38:11.225 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4435999999999999e-05 2020-06-12T02:38:11.225 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.225 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.225 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:11.226 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160411+0000", 2020-06-12T02:38:11.226 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2329999999999995e-06 2020-06-12T02:38:11.226 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.226 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.226 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:11.227 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160420+0000", 2020-06-12T02:38:11.227 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7003e-05 2020-06-12T02:38:11.227 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.227 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.227 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:11.227 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160437+0000", 2020-06-12T02:38:11.228 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8370000000000002e-06 2020-06-12T02:38:11.228 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.228 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.228 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:11.228 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.160443+0000", 2020-06-12T02:38:11.229 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0070635489999999997 2020-06-12T02:38:11.229 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.229 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.229 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.229 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.167507+0000", 2020-06-12T02:38:11.229 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010589700000000001 2020-06-12T02:38:11.230 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.230 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.230 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:11.230 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.167612+0000", 2020-06-12T02:38:11.231 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0065032299999999996 2020-06-12T02:38:11.231 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.232 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.232 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.232 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.174116+0000", 2020-06-12T02:38:11.232 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3128999999999999e-05 2020-06-12T02:38:11.233 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.233 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.233 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:11.233 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.174169+0000", 2020-06-12T02:38:11.233 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0016232760000000001 2020-06-12T02:38:11.234 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.234 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.234 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:11.234 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.175792+0000", 2020-06-12T02:38:11.234 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015435590000000001 2020-06-12T02:38:11.235 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.235 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:11.235 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.235 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.236 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.236 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:950 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~778240] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:38:11.236 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.207643+0000", 2020-06-12T02:38:11.236 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 38.893924916000003, 2020-06-12T02:38:11.237 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014844661, 2020-06-12T02:38:11.237 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:11.237 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:11.237 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:11.237 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:11.238 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:11.240 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 950 2020-06-12T02:38:11.240 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.240 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:11.240 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.240 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:11.241 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.207643+0000", 2020-06-12T02:38:11.241 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:11.241 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.242 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.242 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:11.242 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.207643+0000", 2020-06-12T02:38:11.242 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00072843099999999998 2020-06-12T02:38:11.242 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.243 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.243 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:11.243 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208372+0000", 2020-06-12T02:38:11.243 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9120000000000001e-06 2020-06-12T02:38:11.243 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.243 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.244 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:11.244 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208378+0000", 2020-06-12T02:38:11.244 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9660000000000008e-06 2020-06-12T02:38:11.244 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.245 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.245 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:11.245 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208388+0000", 2020-06-12T02:38:11.245 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1279999999999999e-06 2020-06-12T02:38:11.245 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.246 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.246 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:11.246 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208394+0000", 2020-06-12T02:38:11.247 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0439999999999998e-05 2020-06-12T02:38:11.247 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.247 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.247 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.247 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208444+0000", 2020-06-12T02:38:11.247 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2611000000000001e-05 2020-06-12T02:38:11.248 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.248 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.248 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:11.248 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.208507+0000", 2020-06-12T02:38:11.248 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001228301 2020-06-12T02:38:11.249 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.249 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.249 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.249 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.209735+0000", 2020-06-12T02:38:11.249 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8875000000000001e-05 2020-06-12T02:38:11.250 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.250 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.250 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:11.250 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.209794+0000", 2020-06-12T02:38:11.251 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053218899999999997 2020-06-12T02:38:11.251 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.251 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.251 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:11.251 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.210326+0000", 2020-06-12T02:38:11.252 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0078198439999999994 2020-06-12T02:38:11.252 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.252 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.252 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:11.252 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.218146+0000", 2020-06-12T02:38:11.252 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004300496 2020-06-12T02:38:11.253 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.253 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.253 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:11.253 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.222446+0000", 2020-06-12T02:38:11.253 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1467999999999997e-05 2020-06-12T02:38:11.254 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.254 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.254 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:11.254 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.222488+0000", 2020-06-12T02:38:11.254 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014844661 2020-06-12T02:38:11.255 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.255 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:11.255 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.255 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.255 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.256 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:954 3.3s0 3:c2ca77da:::smithi10729407-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:38:11.256 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.212923+0000", 2020-06-12T02:38:11.256 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 38.888645140000001, 2020-06-12T02:38:11.256 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015411546999999999, 2020-06-12T02:38:11.256 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:11.257 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:11.257 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:11.257 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:11.257 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:11.257 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 954 2020-06-12T02:38:11.257 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.258 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:11.258 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.258 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:11.258 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212923+0000", 2020-06-12T02:38:11.258 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:11.259 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.259 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.259 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:11.259 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212923+0000", 2020-06-12T02:38:11.260 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3598e-05 2020-06-12T02:38:11.260 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.260 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.260 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:11.260 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212946+0000", 2020-06-12T02:38:11.261 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.061e-06 2020-06-12T02:38:11.261 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.261 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.261 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:11.261 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212950+0000", 2020-06-12T02:38:11.262 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8700000000000003e-06 2020-06-12T02:38:11.262 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.262 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.262 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:11.262 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212956+0000", 2020-06-12T02:38:11.264 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0770000000000002e-06 2020-06-12T02:38:11.264 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.264 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.264 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:11.265 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.212958+0000", 2020-06-12T02:38:11.265 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051235170000000002 2020-06-12T02:38:11.265 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.265 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.265 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.266 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.218082+0000", 2020-06-12T02:38:11.266 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9246999999999997e-05 2020-06-12T02:38:11.266 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.266 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.267 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:11.267 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.218141+0000", 2020-06-12T02:38:11.267 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0087981910000000003 2020-06-12T02:38:11.268 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.268 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.268 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.268 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.226939+0000", 2020-06-12T02:38:11.269 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4119999999999997e-05 2020-06-12T02:38:11.269 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.269 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.269 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:11.269 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.226994+0000", 2020-06-12T02:38:11.269 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001340866 2020-06-12T02:38:11.270 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.270 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.270 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:11.270 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.228334+0000", 2020-06-12T02:38:11.270 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015411546999999999 2020-06-12T02:38:11.271 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.271 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:11.271 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.271 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.275 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.275 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1097 3.es1 3:78f01648:::smithi10729407-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:38:11.275 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.576496+0000", 2020-06-12T02:38:11.275 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 38.525071658000002, 2020-06-12T02:38:11.276 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014992914, 2020-06-12T02:38:11.276 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:11.276 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:11.276 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:11.277 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:11.277 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:11.277 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1097 2020-06-12T02:38:11.277 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.278 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:11.278 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.278 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:11.278 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576496+0000", 2020-06-12T02:38:11.278 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:11.279 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.279 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.279 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:11.279 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576496+0000", 2020-06-12T02:38:11.279 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3221999999999999e-05 2020-06-12T02:38:11.282 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.282 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.283 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:11.283 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576520+0000", 2020-06-12T02:38:11.283 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.988e-06 2020-06-12T02:38:11.283 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.283 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.284 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:11.284 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576523+0000", 2020-06-12T02:38:11.284 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2940000000000002e-06 2020-06-12T02:38:11.284 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.284 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.285 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:11.285 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576529+0000", 2020-06-12T02:38:11.285 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6060000000000002e-06 2020-06-12T02:38:11.285 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.285 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.285 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:11.286 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.576534+0000", 2020-06-12T02:38:11.286 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001254554 2020-06-12T02:38:11.286 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.286 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.286 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.287 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.577789+0000", 2020-06-12T02:38:11.287 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3177999999999998e-05 2020-06-12T02:38:11.288 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.288 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.288 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:11.288 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.577842+0000", 2020-06-12T02:38:11.288 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0063091220000000003 2020-06-12T02:38:11.289 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.289 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.289 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.289 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.584151+0000", 2020-06-12T02:38:11.289 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.783e-05 2020-06-12T02:38:11.290 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.290 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.290 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:11.290 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.584209+0000", 2020-06-12T02:38:11.290 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045863650000000002 2020-06-12T02:38:11.291 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.291 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.291 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.291 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.588796+0000", 2020-06-12T02:38:11.291 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7614000000000006e-05 2020-06-12T02:38:11.291 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.292 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.292 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:11.292 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.588873+0000", 2020-06-12T02:38:11.293 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0026161410000000002 2020-06-12T02:38:11.293 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.293 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.293 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:11.293 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.591489+0000", 2020-06-12T02:38:11.294 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014992914 2020-06-12T02:38:11.294 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.294 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:11.294 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.294 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.295 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.295 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:4813 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~450560] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-06-12T02:38:11.295 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:43.343096+0000", 2020-06-12T02:38:11.295 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 27.758472314999999, 2020-06-12T02:38:11.295 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014820242000000001, 2020-06-12T02:38:11.295 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:11.296 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:11.296 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:11.296 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:11.296 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:11.296 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 4813 2020-06-12T02:38:11.297 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.297 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:11.298 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.298 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:11.298 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343096+0000", 2020-06-12T02:38:11.298 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:11.298 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.299 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.299 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:11.299 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343096+0000", 2020-06-12T02:38:11.299 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00068963 2020-06-12T02:38:11.299 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.300 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.300 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:11.300 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343785+0000", 2020-06-12T02:38:11.300 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1479999999999997e-06 2020-06-12T02:38:11.300 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.301 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.301 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:11.301 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343790+0000", 2020-06-12T02:38:11.301 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4060000000000005e-06 2020-06-12T02:38:11.302 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.302 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.302 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:11.302 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343799+0000", 2020-06-12T02:38:11.302 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0970000000000002e-06 2020-06-12T02:38:11.303 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.304 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.304 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:11.304 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.343805+0000", 2020-06-12T02:38:11.304 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00023736499999999999 2020-06-12T02:38:11.304 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.305 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.305 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.305 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.344042+0000", 2020-06-12T02:38:11.305 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000113866 2020-06-12T02:38:11.306 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.306 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.306 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:11.306 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.344156+0000", 2020-06-12T02:38:11.306 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036114760000000002 2020-06-12T02:38:11.307 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.307 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.307 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.307 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.347768+0000", 2020-06-12T02:38:11.308 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1842999999999996e-05 2020-06-12T02:38:11.308 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.308 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.308 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:11.308 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.347850+0000", 2020-06-12T02:38:11.309 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00048835999999999997 2020-06-12T02:38:11.309 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.309 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.309 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:11.309 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.348338+0000", 2020-06-12T02:38:11.310 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0049373139999999999 2020-06-12T02:38:11.310 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.310 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.310 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:11.310 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.353275+0000", 2020-06-12T02:38:11.311 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045727119999999996 2020-06-12T02:38:11.311 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.311 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.311 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:11.311 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.357848+0000", 2020-06-12T02:38:11.311 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8025e-05 2020-06-12T02:38:11.312 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.312 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.312 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:11.312 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.357916+0000", 2020-06-12T02:38:11.312 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014820242000000001 2020-06-12T02:38:11.313 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.313 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:11.313 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.313 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.313 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.313 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:4921 3.9s0 3:93ea0cad:::smithi10729407-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-06-12T02:38:11.314 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:43.595696+0000", 2020-06-12T02:38:11.314 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 27.505871948999999, 2020-06-12T02:38:11.314 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016503190000000001, 2020-06-12T02:38:11.314 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:11.314 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:11.315 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:11.315 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:11.315 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:11.315 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 4921 2020-06-12T02:38:11.315 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.316 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:11.316 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.316 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:11.316 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595696+0000", 2020-06-12T02:38:11.317 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:11.317 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.317 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.317 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:11.317 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595696+0000", 2020-06-12T02:38:11.318 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3694999999999999e-05 2020-06-12T02:38:11.318 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.318 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.318 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:11.319 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595720+0000", 2020-06-12T02:38:11.319 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1700000000000001e-06 2020-06-12T02:38:11.319 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.319 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.319 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:11.320 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595723+0000", 2020-06-12T02:38:11.320 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8849999999999999e-06 2020-06-12T02:38:11.320 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.320 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.320 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:11.320 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595731+0000", 2020-06-12T02:38:11.321 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5689999999999999e-06 2020-06-12T02:38:11.321 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.321 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.321 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:11.321 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595733+0000", 2020-06-12T02:38:11.322 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0030116880000000002 2020-06-12T02:38:11.322 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.322 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.322 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.322 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.598745+0000", 2020-06-12T02:38:11.322 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1353999999999996e-05 2020-06-12T02:38:11.323 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.323 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.323 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:11.323 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.598806+0000", 2020-06-12T02:38:11.323 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051955819999999998 2020-06-12T02:38:11.324 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.324 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.324 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.324 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.604002+0000", 2020-06-12T02:38:11.324 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9615000000000001e-05 2020-06-12T02:38:11.325 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.325 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.325 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:11.325 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.604102+0000", 2020-06-12T02:38:11.325 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0052050960000000002 2020-06-12T02:38:11.326 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.326 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.327 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.327 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.609307+0000", 2020-06-12T02:38:11.327 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5025000000000003e-05 2020-06-12T02:38:11.327 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.328 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.328 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:11.328 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.609362+0000", 2020-06-12T02:38:11.328 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002837511 2020-06-12T02:38:11.328 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.329 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.329 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:11.330 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.612199+0000", 2020-06-12T02:38:11.331 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016503190000000001 2020-06-12T02:38:11.331 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.331 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:11.331 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.332 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.332 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.332 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:5068 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~663552] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e30)", 2020-06-12T02:38:11.332 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:44.157497+0000", 2020-06-12T02:38:11.332 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 26.944071322999999, 2020-06-12T02:38:11.333 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015164129, 2020-06-12T02:38:11.333 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:11.333 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:11.333 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:11.333 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:11.334 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:11.334 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 5068 2020-06-12T02:38:11.334 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.334 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:11.334 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.334 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:11.335 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.157497+0000", 2020-06-12T02:38:11.335 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:11.335 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.335 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.335 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:11.336 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.157497+0000", 2020-06-12T02:38:11.336 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00068511099999999997 2020-06-12T02:38:11.336 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.336 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.336 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:11.337 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.158182+0000", 2020-06-12T02:38:11.337 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4630000000000004e-06 2020-06-12T02:38:11.337 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.337 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.337 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:11.337 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.158189+0000", 2020-06-12T02:38:11.338 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4199000000000001e-05 2020-06-12T02:38:11.338 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.338 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.338 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:11.339 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.158203+0000", 2020-06-12T02:38:11.339 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1250000000000004e-06 2020-06-12T02:38:11.339 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.339 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.339 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:11.339 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.158211+0000", 2020-06-12T02:38:11.340 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6754000000000002e-05 2020-06-12T02:38:11.341 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.341 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.341 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.341 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.158237+0000", 2020-06-12T02:38:11.342 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2948999999999996e-05 2020-06-12T02:38:11.342 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.342 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.343 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:11.343 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.158310+0000", 2020-06-12T02:38:11.343 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0032401610000000001 2020-06-12T02:38:11.344 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.344 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.344 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.345 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.161550+0000", 2020-06-12T02:38:11.345 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5550000000000002e-05 2020-06-12T02:38:11.345 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.345 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.346 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:11.346 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.161606+0000", 2020-06-12T02:38:11.346 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00062448199999999999 2020-06-12T02:38:11.347 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.347 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.347 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:11.347 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.162230+0000", 2020-06-12T02:38:11.348 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0073257130000000002 2020-06-12T02:38:11.348 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.348 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.349 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:11.349 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.169556+0000", 2020-06-12T02:38:11.349 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0030226419999999999 2020-06-12T02:38:11.349 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.350 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.351 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:11.351 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.172579+0000", 2020-06-12T02:38:11.352 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1979999999999998e-05 2020-06-12T02:38:11.352 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.352 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.353 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:11.354 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.172661+0000", 2020-06-12T02:38:11.354 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015164129 2020-06-12T02:38:11.354 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.354 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:11.355 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.355 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.355 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.355 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:5071 3.3s0 3:c2ca77da:::smithi10729407-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-06-12T02:38:11.355 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:44.159788+0000", 2020-06-12T02:38:11.356 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 26.941780100999999, 2020-06-12T02:38:11.356 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016812273999999999, 2020-06-12T02:38:11.356 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:11.356 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:11.356 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:11.356 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:11.357 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:11.357 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 5071 2020-06-12T02:38:11.357 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.357 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:11.357 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.358 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:11.358 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159788+0000", 2020-06-12T02:38:11.358 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:11.358 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.358 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.358 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:11.359 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159788+0000", 2020-06-12T02:38:11.359 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1421000000000001e-05 2020-06-12T02:38:11.359 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.360 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.360 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:11.360 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159839+0000", 2020-06-12T02:38:11.360 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9009999999999994e-06 2020-06-12T02:38:11.360 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.361 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.362 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:11.362 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159847+0000", 2020-06-12T02:38:11.362 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3149999999999999e-05 2020-06-12T02:38:11.362 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.363 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.363 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:11.363 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159860+0000", 2020-06-12T02:38:11.364 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1749999999999996e-06 2020-06-12T02:38:11.364 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.364 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.365 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:11.365 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159866+0000", 2020-06-12T02:38:11.365 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6035999999999999e-05 2020-06-12T02:38:11.366 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.366 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.366 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.367 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159892+0000", 2020-06-12T02:38:11.367 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2627999999999998e-05 2020-06-12T02:38:11.367 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.368 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.368 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:11.369 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159944+0000", 2020-06-12T02:38:11.369 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010270133000000001 2020-06-12T02:38:11.370 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.370 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.370 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.371 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.170214+0000", 2020-06-12T02:38:11.372 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5490999999999998e-05 2020-06-12T02:38:11.372 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.372 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.372 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:11.373 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.170310+0000", 2020-06-12T02:38:11.373 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003215828 2020-06-12T02:38:11.373 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.373 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.373 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.374 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.173526+0000", 2020-06-12T02:38:11.374 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0290000000000005e-05 2020-06-12T02:38:11.374 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.374 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.374 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:11.375 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.173616+0000", 2020-06-12T02:38:11.375 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002984221 2020-06-12T02:38:11.375 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.375 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.375 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:11.376 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.176600+0000", 2020-06-12T02:38:11.376 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016812273999999999 2020-06-12T02:38:11.376 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.376 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:11.376 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.376 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.377 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.377 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:6432 3.9s0 3:93ea0cad:::smithi10729407-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e31)", 2020-06-12T02:38:11.377 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:47.960104+0000", 2020-06-12T02:38:11.377 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 23.141464005, 2020-06-12T02:38:11.377 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016025815999999998, 2020-06-12T02:38:11.378 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:11.378 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:11.378 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:11.378 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:11.378 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:11.379 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 6432 2020-06-12T02:38:11.379 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.379 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:11.379 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.379 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:11.379 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960104+0000", 2020-06-12T02:38:11.380 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:11.380 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.380 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.380 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:11.380 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960104+0000", 2020-06-12T02:38:11.381 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2351999999999999e-05 2020-06-12T02:38:11.381 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.381 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.381 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:11.381 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960126+0000", 2020-06-12T02:38:11.382 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.061e-06 2020-06-12T02:38:11.382 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.382 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.382 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:11.382 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960129+0000", 2020-06-12T02:38:11.382 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4849999999999997e-06 2020-06-12T02:38:11.383 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.383 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.383 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:11.383 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960136+0000", 2020-06-12T02:38:11.383 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.176e-06 2020-06-12T02:38:11.384 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.384 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.384 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:11.384 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960138+0000", 2020-06-12T02:38:11.384 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0044054139999999999 2020-06-12T02:38:11.384 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.385 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.385 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.385 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.964543+0000", 2020-06-12T02:38:11.385 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8077000000000006e-05 2020-06-12T02:38:11.385 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.386 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.386 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:11.386 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.964612+0000", 2020-06-12T02:38:11.386 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0029921150000000001 2020-06-12T02:38:11.386 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.386 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.387 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.387 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.967604+0000", 2020-06-12T02:38:11.387 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2983000000000001e-05 2020-06-12T02:38:11.387 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.387 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.388 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:11.388 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.967697+0000", 2020-06-12T02:38:11.388 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0072496510000000002 2020-06-12T02:38:11.388 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.388 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.388 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.389 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.974946+0000", 2020-06-12T02:38:11.389 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8278e-05 2020-06-12T02:38:11.389 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.389 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.389 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:11.390 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.975005+0000", 2020-06-12T02:38:11.390 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001125224 2020-06-12T02:38:11.390 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.390 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.390 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:11.390 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.976130+0000", 2020-06-12T02:38:11.391 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016025815999999998 2020-06-12T02:38:11.391 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.391 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:11.391 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.391 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.392 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.392 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8123 3.es1 3:78f01648:::smithi10729407-5:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-06-12T02:38:11.392 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.690721+0000", 2020-06-12T02:38:11.392 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 18.410846900999999, 2020-06-12T02:38:11.392 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016503126999999999, 2020-06-12T02:38:11.392 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:11.393 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:11.393 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:11.393 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:11.393 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:11.393 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8123 2020-06-12T02:38:11.394 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.394 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:11.394 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.394 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:11.394 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690721+0000", 2020-06-12T02:38:11.394 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:11.395 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.395 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.395 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:11.395 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690721+0000", 2020-06-12T02:38:11.395 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7526e-05 2020-06-12T02:38:11.396 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.396 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.396 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:11.396 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690759+0000", 2020-06-12T02:38:11.396 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1999999999999999e-06 2020-06-12T02:38:11.396 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.397 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.397 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:11.397 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690762+0000", 2020-06-12T02:38:11.397 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9229999999999999e-06 2020-06-12T02:38:11.397 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.398 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.398 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:11.398 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690768+0000", 2020-06-12T02:38:11.398 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4480000000000001e-06 2020-06-12T02:38:11.398 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.398 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.399 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:11.399 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690770+0000", 2020-06-12T02:38:11.399 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00079317799999999996 2020-06-12T02:38:11.399 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.399 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.400 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.400 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.691563+0000", 2020-06-12T02:38:11.400 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5299e-05 2020-06-12T02:38:11.400 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.400 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.400 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:11.401 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.691619+0000", 2020-06-12T02:38:11.401 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010070076000000001 2020-06-12T02:38:11.401 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.401 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.401 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.402 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.701689+0000", 2020-06-12T02:38:11.402 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000110234 2020-06-12T02:38:11.402 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.402 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.402 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:11.402 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.701799+0000", 2020-06-12T02:38:11.403 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00037311100000000003 2020-06-12T02:38:11.403 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.403 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.403 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:11.403 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.702172+0000", 2020-06-12T02:38:11.404 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0044460050000000003 2020-06-12T02:38:11.404 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.405 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.405 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:11.405 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.706618+0000", 2020-06-12T02:38:11.405 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00043550199999999999 2020-06-12T02:38:11.405 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.405 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.406 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:11.406 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.707054+0000", 2020-06-12T02:38:11.406 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017062499999999999 2020-06-12T02:38:11.406 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.406 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.407 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:11.407 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.707224+0000", 2020-06-12T02:38:11.407 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016503126999999999 2020-06-12T02:38:11.407 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.407 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:11.408 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.408 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.408 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.408 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8124 3.es1 3:78f01648:::smithi10729407-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e33)", 2020-06-12T02:38:11.408 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.690794+0000", 2020-06-12T02:38:11.408 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 18.410773715000001, 2020-06-12T02:38:11.409 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019298497000000001, 2020-06-12T02:38:11.409 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:11.409 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:11.409 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:11.409 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:11.410 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:11.410 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8124 2020-06-12T02:38:11.410 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.410 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:11.410 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.410 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:11.411 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690794+0000", 2020-06-12T02:38:11.411 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:11.411 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.411 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.411 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:11.412 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690794+0000", 2020-06-12T02:38:11.412 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1869999999999999e-05 2020-06-12T02:38:11.412 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.412 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.412 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:11.413 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690816+0000", 2020-06-12T02:38:11.413 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.162e-06 2020-06-12T02:38:11.413 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.413 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.413 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:11.413 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690819+0000", 2020-06-12T02:38:11.414 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2280000000000003e-06 2020-06-12T02:38:11.414 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.414 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.414 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:11.414 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690826+0000", 2020-06-12T02:38:11.415 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.452e-06 2020-06-12T02:38:11.415 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.415 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.415 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:11.415 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690828+0000", 2020-06-12T02:38:11.416 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00081402600000000003 2020-06-12T02:38:11.416 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.416 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.416 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.417 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.691642+0000", 2020-06-12T02:38:11.417 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00024263 2020-06-12T02:38:11.417 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.417 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.417 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:11.418 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.691885+0000", 2020-06-12T02:38:11.418 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010968739 2020-06-12T02:38:11.418 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.418 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.418 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.419 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.702853+0000", 2020-06-12T02:38:11.419 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8985000000000002e-05 2020-06-12T02:38:11.419 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.419 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.419 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:11.419 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.702952+0000", 2020-06-12T02:38:11.420 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0046973329999999997 2020-06-12T02:38:11.420 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.420 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.420 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.420 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.707650+0000", 2020-06-12T02:38:11.421 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3773e-05 2020-06-12T02:38:11.421 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.421 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.421 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:11.421 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.707723+0000", 2020-06-12T02:38:11.421 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023692990000000001 2020-06-12T02:38:11.422 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.422 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.422 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:11.422 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.710093+0000", 2020-06-12T02:38:11.422 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019298497000000001 2020-06-12T02:38:11.423 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.423 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:11.423 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.423 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.423 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.423 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8190 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~753664] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-06-12T02:38:11.424 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.894592+0000", 2020-06-12T02:38:11.424 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 18.206976056999999, 2020-06-12T02:38:11.424 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016165763999999999, 2020-06-12T02:38:11.424 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:11.424 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:11.425 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:11.425 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:11.425 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:11.425 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8190 2020-06-12T02:38:11.425 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.425 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:11.426 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.426 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:11.426 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.894592+0000", 2020-06-12T02:38:11.426 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:11.426 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.427 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.427 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:11.427 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.894592+0000", 2020-06-12T02:38:11.427 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00074881500000000001 2020-06-12T02:38:11.427 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.427 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.428 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:11.428 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.895341+0000", 2020-06-12T02:38:11.428 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1919999999999999e-06 2020-06-12T02:38:11.428 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.428 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.429 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:11.429 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.895348+0000", 2020-06-12T02:38:11.429 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.059e-05 2020-06-12T02:38:11.429 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.429 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.429 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:11.430 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.895359+0000", 2020-06-12T02:38:11.430 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2310000000000001e-06 2020-06-12T02:38:11.430 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.430 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.430 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:11.431 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.895365+0000", 2020-06-12T02:38:11.431 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00049215800000000003 2020-06-12T02:38:11.431 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.431 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.431 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.431 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.895857+0000", 2020-06-12T02:38:11.432 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1001999999999995e-05 2020-06-12T02:38:11.432 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.432 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.432 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:11.432 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.895938+0000", 2020-06-12T02:38:11.433 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0059219040000000004 2020-06-12T02:38:11.433 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.433 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.433 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.433 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.901860+0000", 2020-06-12T02:38:11.434 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010156499999999999 2020-06-12T02:38:11.434 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.434 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.434 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:11.434 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.901961+0000", 2020-06-12T02:38:11.435 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00085242800000000002 2020-06-12T02:38:11.435 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.435 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.435 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:11.435 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.902814+0000", 2020-06-12T02:38:11.435 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0057806180000000004 2020-06-12T02:38:11.436 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.436 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.436 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:11.436 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.908594+0000", 2020-06-12T02:38:11.436 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021229600000000001 2020-06-12T02:38:11.436 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.437 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.437 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:11.437 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.910717+0000", 2020-06-12T02:38:11.437 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0300999999999999e-05 2020-06-12T02:38:11.437 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.438 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.438 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:11.438 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.910758+0000", 2020-06-12T02:38:11.438 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016165763999999999 2020-06-12T02:38:11.438 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.438 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:11.439 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.439 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.439 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.439 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8191 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~450560] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-06-12T02:38:11.439 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.897242+0000", 2020-06-12T02:38:11.440 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 18.204325604000001, 2020-06-12T02:38:11.440 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014857576000000001, 2020-06-12T02:38:11.440 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:11.440 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:11.440 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:11.440 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:11.441 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:11.441 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8191 2020-06-12T02:38:11.441 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.441 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:11.441 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.442 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:11.442 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897242+0000", 2020-06-12T02:38:11.442 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:11.442 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.442 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.442 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:11.443 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897242+0000", 2020-06-12T02:38:11.443 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000531592 2020-06-12T02:38:11.443 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.443 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.443 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:11.444 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897774+0000", 2020-06-12T02:38:11.444 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0288999999999999e-05 2020-06-12T02:38:11.444 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.444 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.444 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:11.444 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897784+0000", 2020-06-12T02:38:11.445 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7149999999999993e-06 2020-06-12T02:38:11.445 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.445 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.445 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:11.445 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897794+0000", 2020-06-12T02:38:11.446 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8960000000000002e-06 2020-06-12T02:38:11.446 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.446 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.446 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:11.446 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897799+0000", 2020-06-12T02:38:11.447 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6251999999999998e-05 2020-06-12T02:38:11.447 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.447 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.447 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.447 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897835+0000", 2020-06-12T02:38:11.448 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.768e-05 2020-06-12T02:38:11.448 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.448 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.448 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:11.448 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897933+0000", 2020-06-12T02:38:11.449 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0082988100000000002 2020-06-12T02:38:11.449 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.449 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.449 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.449 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.906232+0000", 2020-06-12T02:38:11.449 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7865999999999998e-05 2020-06-12T02:38:11.450 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.450 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.450 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:11.450 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.906290+0000", 2020-06-12T02:38:11.450 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047895700000000002 2020-06-12T02:38:11.451 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.451 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.451 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:11.451 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.906768+0000", 2020-06-12T02:38:11.451 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002658442 2020-06-12T02:38:11.451 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.452 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.452 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:11.452 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.909427+0000", 2020-06-12T02:38:11.452 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0026066230000000002 2020-06-12T02:38:11.452 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.453 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.453 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:11.453 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.912034+0000", 2020-06-12T02:38:11.453 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6453999999999998e-05 2020-06-12T02:38:11.453 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.453 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.454 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:11.454 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.912100+0000", 2020-06-12T02:38:11.454 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014857576000000001 2020-06-12T02:38:11.454 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.454 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:11.455 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.455 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.455 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.455 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8193 3.3s0 3:c2ca77da:::smithi10729407-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e33)", 2020-06-12T02:38:11.455 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.897900+0000", 2020-06-12T02:38:11.455 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 18.203667672000002, 2020-06-12T02:38:11.456 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015951213999999998, 2020-06-12T02:38:11.456 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:11.456 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:11.456 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:11.456 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:11.457 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:11.457 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8193 2020-06-12T02:38:11.457 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.457 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:11.457 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.458 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:11.458 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897900+0000", 2020-06-12T02:38:11.458 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:11.458 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.458 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.458 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:11.459 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897900+0000", 2020-06-12T02:38:11.459 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2331e-05 2020-06-12T02:38:11.459 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.459 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.459 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:11.460 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897923+0000", 2020-06-12T02:38:11.460 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.021e-06 2020-06-12T02:38:11.460 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.460 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.460 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:11.461 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897926+0000", 2020-06-12T02:38:11.461 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4150000000000002e-06 2020-06-12T02:38:11.461 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.461 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.461 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:11.462 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897933+0000", 2020-06-12T02:38:11.462 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2359999999999999e-06 2020-06-12T02:38:11.462 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.462 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.462 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:11.462 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897935+0000", 2020-06-12T02:38:11.463 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000118301 2020-06-12T02:38:11.463 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.463 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.463 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.463 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.898054+0000", 2020-06-12T02:38:11.463 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8914999999999998e-05 2020-06-12T02:38:11.464 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.464 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.464 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:11.464 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.898143+0000", 2020-06-12T02:38:11.464 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011203013 2020-06-12T02:38:11.465 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.465 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.465 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.465 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.909346+0000", 2020-06-12T02:38:11.465 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7075000000000003e-05 2020-06-12T02:38:11.465 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.466 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.466 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:11.466 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.909423+0000", 2020-06-12T02:38:11.466 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033600969999999998 2020-06-12T02:38:11.466 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.467 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.467 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.467 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.912783+0000", 2020-06-12T02:38:11.467 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6118000000000005e-05 2020-06-12T02:38:11.467 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.467 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.468 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:11.468 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.912869+0000", 2020-06-12T02:38:11.468 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00098269200000000007 2020-06-12T02:38:11.468 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.468 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.469 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:11.469 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.913852+0000", 2020-06-12T02:38:11.469 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015951213999999998 2020-06-12T02:38:11.469 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.469 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:11.469 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.470 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.470 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:11.470 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:11.471 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:11.469+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:11.472 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-06-12T02:38:11.562 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2020-06-12T02:38:11.563 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2020-06-12T02:38:11.563 INFO:tasks.thrashosds.thrasher:Reweighting osd 1 to 0.18982494206385925 2020-06-12T02:38:11.563 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd reweight 1 0.18982494206385925 2020-06-12T02:38:11.590 INFO:tasks.rados.rados.0.smithi107.stdout:4105: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:11.591 INFO:tasks.rados.rados.0.smithi107.stdout:4105: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:11.591 INFO:tasks.rados.rados.0.smithi107.stdout:4105: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:11.591 INFO:tasks.rados.rados.0.smithi107.stdout:4105: finishing write tid 5 to smithi10729407-17 2020-06-12T02:38:11.591 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:11.587+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:11.592 INFO:tasks.rados.rados.0.smithi107.stdout:4103: done (2 left) 2020-06-12T02:38:11.592 INFO:tasks.rados.rados.0.smithi107.stdout:4106: done (1 left) 2020-06-12T02:38:11.592 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:11.593 INFO:tasks.rados.rados.0.smithi107.stdout:4107: seq_num 4106 ranges {1343488=778240,2121728=696320,2818048=106496} 2020-06-12T02:38:11.611 INFO:tasks.rados.rados.0.smithi107.stdout:4107: writing smithi10729407-8 from 1343488 to 2121728 tid 1 2020-06-12T02:38:11.611 INFO:tasks.rados.rados.0.smithi107.stdout:4107: writing smithi10729407-8 from 2121728 to 2818048 tid 2 2020-06-12T02:38:11.611 INFO:tasks.rados.rados.0.smithi107.stdout:4107: writing smithi10729407-8 from 2818048 to 2924544 tid 3 2020-06-12T02:38:11.611 INFO:tasks.rados.rados.0.smithi107.stdout:4108: delete oid 15 current snap is 0 2020-06-12T02:38:11.611 INFO:tasks.rados.rados.0.smithi107.stdout:4105: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:11.612 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 741 (ObjNum 4104 snap 0 seq_num 4104) dirty exists 2020-06-12T02:38:11.612 INFO:tasks.rados.rados.0.smithi107.stdout:4105: left oid 17 (ObjNum 4104 snap 0 seq_num 4104) 2020-06-12T02:38:11.612 INFO:tasks.rados.rados.0.smithi107.stdout:4107: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:11.618 INFO:tasks.rados.rados.0.smithi107.stdout:4105: done (2 left) 2020-06-12T02:38:11.619 INFO:tasks.rados.rados.0.smithi107.stdout:4108: done (1 left) 2020-06-12T02:38:11.619 INFO:tasks.rados.rados.0.smithi107.stdout:4109: delete oid 2 current snap is 0 2020-06-12T02:38:11.619 INFO:tasks.rados.rados.0.smithi107.stdout:4107: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:11.622 INFO:tasks.rados.rados.0.smithi107.stdout:4107: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:11.622 INFO:tasks.rados.rados.0.smithi107.stdout:4109: done (1 left) 2020-06-12T02:38:11.622 INFO:tasks.rados.rados.0.smithi107.stdout:4110: delete oid 3 current snap is 0 2020-06-12T02:38:11.624 INFO:tasks.rados.rados.0.smithi107.stdout:4107: finishing write tid 5 to smithi10729407-8 2020-06-12T02:38:11.627 INFO:tasks.rados.rados.0.smithi107.stdout:4110: done (1 left) 2020-06-12T02:38:11.627 INFO:tasks.rados.rados.0.smithi107.stdout:4111: delete oid 4 current snap is 0 2020-06-12T02:38:11.628 INFO:tasks.rados.rados.0.smithi107.stdout:4107: finishing write tid 6 to smithi10729407-8 2020-06-12T02:38:11.628 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 720 (ObjNum 4106 snap 0 seq_num 4106) dirty exists 2020-06-12T02:38:11.628 INFO:tasks.rados.rados.0.smithi107.stdout:4107: left oid 8 (ObjNum 4106 snap 0 seq_num 4106) 2020-06-12T02:38:11.630 INFO:tasks.rados.rados.0.smithi107.stdout:4107: done (1 left) 2020-06-12T02:38:11.630 INFO:tasks.rados.rados.0.smithi107.stdout:4111: done (0 left) 2020-06-12T02:38:11.630 INFO:tasks.rados.rados.0.smithi107.stdout:4112: delete oid 1 current snap is 0 2020-06-12T02:38:11.632 INFO:tasks.rados.rados.0.smithi107.stdout:4112: done (0 left) 2020-06-12T02:38:11.632 INFO:tasks.rados.rados.0.smithi107.stdout:4113: delete oid 14 current snap is 0 2020-06-12T02:38:11.635 INFO:tasks.rados.rados.0.smithi107.stdout:4113: done (0 left) 2020-06-12T02:38:11.635 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:11.635 INFO:tasks.rados.rados.0.smithi107.stdout:4114: seq_num 4113 ranges {2924544=491520,3416064=737280,4153344=466944} 2020-06-12T02:38:11.636 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:11.636 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:38:11.637 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2020-06-12T02:38:11.637 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:11.645 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-06-12T02:38:11.654 INFO:tasks.rados.rados.0.smithi107.stdout:4114: writing smithi10729407-8 from 2924544 to 3416064 tid 1 2020-06-12T02:38:11.656 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd set noscrub 2020-06-12T02:38:11.658 INFO:tasks.rados.rados.0.smithi107.stdout:4114: writing smithi10729407-8 from 3416064 to 4153344 tid 2 2020-06-12T02:38:11.660 INFO:tasks.rados.rados.0.smithi107.stdout:4114: writing smithi10729407-8 from 4153344 to 4620288 tid 3 2020-06-12T02:38:11.660 INFO:tasks.rados.rados.0.smithi107.stdout:4115: delete oid 1 current snap is 0 2020-06-12T02:38:11.663 INFO:tasks.rados.rados.0.smithi107.stdout:4115: done (1 left) 2020-06-12T02:38:11.664 INFO:tasks.rados.rados.0.smithi107.stdout:4116: delete oid 16 current snap is 0 2020-06-12T02:38:11.664 INFO:tasks.rados.rados.0.smithi107.stdout:4114: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:11.668 INFO:tasks.rados.rados.0.smithi107.stdout:4114: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:11.669 INFO:tasks.rados.rados.0.smithi107.stdout:4116: done (1 left) 2020-06-12T02:38:11.669 INFO:tasks.rados.rados.0.smithi107.stdout:4117: delete oid 19 current snap is 0 2020-06-12T02:38:11.669 INFO:tasks.rados.rados.0.smithi107.stdout:4114: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:11.670 INFO:tasks.rados.rados.0.smithi107.stdout:4114: finishing write tid 5 to smithi10729407-8 2020-06-12T02:38:11.671 INFO:tasks.rados.rados.0.smithi107.stdout:4117: done (1 left) 2020-06-12T02:38:11.671 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:11.671 INFO:tasks.rados.rados.0.smithi107.stdout:4118: seq_num 4117 ranges {1490944=557056,2048000=679936,2727936=524288} 2020-06-12T02:38:11.674 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:11.671+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:11.683 INFO:tasks.rados.rados.0.smithi107.stdout:4118: writing smithi10729407-6 from 1490944 to 2048000 tid 1 2020-06-12T02:38:11.686 INFO:tasks.rados.rados.0.smithi107.stdout:4118: writing smithi10729407-6 from 2048000 to 2727936 tid 2 2020-06-12T02:38:11.690 INFO:tasks.rados.rados.0.smithi107.stdout:4118: writing smithi10729407-6 from 2727936 to 3252224 tid 3 2020-06-12T02:38:11.690 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:38:11.691 INFO:tasks.rados.rados.0.smithi107.stdout:4119: seq_num 4118 ranges {0=573440,573440=565248,1138688=589824,1728512=49152} 2020-06-12T02:38:11.693 INFO:tasks.rados.rados.0.smithi107.stdout:4119: writing smithi10729407-3 from 0 to 573440 tid 1 2020-06-12T02:38:11.697 INFO:tasks.rados.rados.0.smithi107.stdout:4119: writing smithi10729407-3 from 573440 to 1138688 tid 2 2020-06-12T02:38:11.701 INFO:tasks.rados.rados.0.smithi107.stdout:4119: writing smithi10729407-3 from 1138688 to 1728512 tid 3 2020-06-12T02:38:11.701 INFO:tasks.rados.rados.0.smithi107.stdout:4119: writing smithi10729407-3 from 1728512 to 1777664 tid 4 2020-06-12T02:38:11.702 INFO:tasks.rados.rados.0.smithi107.stdout:4120: delete oid 7 current snap is 0 2020-06-12T02:38:11.703 INFO:tasks.rados.rados.0.smithi107.stdout:4114: finishing write tid 6 to smithi10729407-8 2020-06-12T02:38:11.704 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 724 (ObjNum 4113 snap 0 seq_num 4113) dirty exists 2020-06-12T02:38:11.704 INFO:tasks.rados.rados.0.smithi107.stdout:4114: left oid 8 (ObjNum 4113 snap 0 seq_num 4113) 2020-06-12T02:38:11.704 INFO:tasks.rados.rados.0.smithi107.stdout:4118: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:11.704 INFO:tasks.rados.rados.0.smithi107.stdout:4118: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:11.705 INFO:tasks.rados.rados.0.smithi107.stdout:4114: done (3 left) 2020-06-12T02:38:11.705 INFO:tasks.rados.rados.0.smithi107.stdout:4120: done (2 left) 2020-06-12T02:38:11.705 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:38:11.705 INFO:tasks.rados.rados.0.smithi107.stdout:4121: seq_num 4120 ranges {0=606208,606208=729088,1335296=475136} 2020-06-12T02:38:11.708 INFO:tasks.rados.rados.0.smithi107.stdout:4121: writing smithi10729407-10 from 0 to 606208 tid 1 2020-06-12T02:38:11.713 INFO:tasks.rados.rados.0.smithi107.stdout:4121: writing smithi10729407-10 from 606208 to 1335296 tid 2 2020-06-12T02:38:11.717 INFO:tasks.rados.rados.0.smithi107.stdout:4121: writing smithi10729407-10 from 1335296 to 1810432 tid 3 2020-06-12T02:38:11.718 INFO:tasks.rados.rados.0.smithi107.stdout:4122: delete oid 7 current snap is 0 2020-06-12T02:38:11.718 INFO:tasks.rados.rados.0.smithi107.stdout:4118: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:11.718 INFO:tasks.rados.rados.0.smithi107.stdout:4118: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:11.719 INFO:tasks.rados.rados.0.smithi107.stdout:4118: finishing write tid 6 to smithi10729407-6 2020-06-12T02:38:11.719 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 776 (ObjNum 4117 snap 0 seq_num 4117) dirty exists 2020-06-12T02:38:11.719 INFO:tasks.rados.rados.0.smithi107.stdout:4118: left oid 6 (ObjNum 4117 snap 0 seq_num 4117) 2020-06-12T02:38:11.720 INFO:tasks.rados.rados.0.smithi107.stdout:4119: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:11.720 INFO:tasks.rados.rados.0.smithi107.stdout:4119: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:11.721 INFO:tasks.rados.rados.0.smithi107.stdout:4119: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:11.722 INFO:tasks.rados.rados.0.smithi107.stdout:4118: done (3 left) 2020-06-12T02:38:11.722 INFO:tasks.rados.rados.0.smithi107.stdout:4122: done (2 left) 2020-06-12T02:38:11.722 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:11.723 INFO:tasks.rados.rados.0.smithi107.stdout:4123: seq_num 4122 ranges {0=638976,638976=548864,1187840=573440,1761280=81920} 2020-06-12T02:38:11.726 INFO:tasks.rados.rados.0.smithi107.stdout:4123: writing smithi10729407-19 from 0 to 638976 tid 1 2020-06-12T02:38:11.730 INFO:tasks.rados.rados.0.smithi107.stdout:4123: writing smithi10729407-19 from 638976 to 1187840 tid 2 2020-06-12T02:38:11.734 INFO:tasks.rados.rados.0.smithi107.stdout:4123: writing smithi10729407-19 from 1187840 to 1761280 tid 3 2020-06-12T02:38:11.735 INFO:tasks.rados.rados.0.smithi107.stdout:4123: writing smithi10729407-19 from 1761280 to 1843200 tid 4 2020-06-12T02:38:11.735 INFO:tasks.rados.rados.0.smithi107.stdout:4124: delete oid 12 current snap is 0 2020-06-12T02:38:11.735 INFO:tasks.rados.rados.0.smithi107.stdout:4121: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:11.736 INFO:tasks.rados.rados.0.smithi107.stdout:4121: finishing write tid 2 to smithi10729407-10 2020-06-12T02:38:11.736 INFO:tasks.rados.rados.0.smithi107.stdout:4119: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:11.737 INFO:tasks.rados.rados.0.smithi107.stdout:4121: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:11.737 INFO:tasks.rados.rados.0.smithi107.stdout:4121: finishing write tid 5 to smithi10729407-10 2020-06-12T02:38:11.737 INFO:tasks.rados.rados.0.smithi107.stdout:4119: finishing write tid 6 to smithi10729407-3 2020-06-12T02:38:11.737 INFO:tasks.rados.rados.0.smithi107.stdout:4121: finishing write tid 6 to smithi10729407-10 2020-06-12T02:38:11.737 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 405 (ObjNum 4120 snap 0 seq_num 4120) dirty exists 2020-06-12T02:38:11.738 INFO:tasks.rados.rados.0.smithi107.stdout:4121: left oid 10 (ObjNum 4120 snap 0 seq_num 4120) 2020-06-12T02:38:11.738 INFO:tasks.rados.rados.0.smithi107.stdout:4119: finishing write tid 7 to smithi10729407-3 2020-06-12T02:38:11.738 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 737 (ObjNum 4118 snap 0 seq_num 4118) dirty exists 2020-06-12T02:38:11.738 INFO:tasks.rados.rados.0.smithi107.stdout:4119: left oid 3 (ObjNum 4118 snap 0 seq_num 4118) 2020-06-12T02:38:11.740 INFO:tasks.rados.rados.0.smithi107.stdout:4123: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:11.741 INFO:tasks.rados.rados.0.smithi107.stdout:4123: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:11.741 INFO:tasks.rados.rados.0.smithi107.stdout:4123: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:11.742 INFO:tasks.rados.rados.0.smithi107.stdout:4123: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:11.745 INFO:tasks.rados.rados.0.smithi107.stdout:4123: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:11.749 INFO:tasks.rados.rados.0.smithi107.stdout:4119: done (3 left) 2020-06-12T02:38:11.749 INFO:tasks.rados.rados.0.smithi107.stdout:4121: done (2 left) 2020-06-12T02:38:11.749 INFO:tasks.rados.rados.0.smithi107.stdout:4124: done (1 left) 2020-06-12T02:38:11.750 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:11.750 INFO:tasks.rados.rados.0.smithi107.stdout:4125: seq_num 4124 ranges {1777664=671744,2449408=720896,3170304=483328} 2020-06-12T02:38:11.769 INFO:tasks.rados.rados.0.smithi107.stdout:4125: writing smithi10729407-3 from 1777664 to 2449408 tid 1 2020-06-12T02:38:11.770 INFO:tasks.rados.rados.0.smithi107.stdout:4125: writing smithi10729407-3 from 2449408 to 3170304 tid 2 2020-06-12T02:38:11.773 INFO:tasks.rados.rados.0.smithi107.stdout:4125: writing smithi10729407-3 from 3170304 to 3653632 tid 3 2020-06-12T02:38:11.773 INFO:tasks.rados.rados.0.smithi107.stdout:4126: delete oid 10 current snap is 0 2020-06-12T02:38:11.773 INFO:tasks.rados.rados.0.smithi107.stdout:4123: finishing write tid 7 to smithi10729407-19 2020-06-12T02:38:11.773 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 695 (ObjNum 4122 snap 0 seq_num 4122) dirty exists 2020-06-12T02:38:11.774 INFO:tasks.rados.rados.0.smithi107.stdout:4123: left oid 19 (ObjNum 4122 snap 0 seq_num 4122) 2020-06-12T02:38:11.774 INFO:tasks.rados.rados.0.smithi107.stdout:4125: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:11.776 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:11.773+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:11.777 INFO:tasks.rados.rados.0.smithi107.stdout:4123: done (2 left) 2020-06-12T02:38:11.777 INFO:tasks.rados.rados.0.smithi107.stdout:4126: done (1 left) 2020-06-12T02:38:11.777 INFO:tasks.rados.rados.0.smithi107.stdout:4127: delete oid 13 current snap is 0 2020-06-12T02:38:11.782 INFO:tasks.rados.rados.0.smithi107.stdout:4125: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:11.782 INFO:tasks.rados.rados.0.smithi107.stdout:4127: done (1 left) 2020-06-12T02:38:11.783 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:38:11.783 INFO:tasks.rados.rados.0.smithi107.stdout:4128: seq_num 4127 ranges {0=729088,729088=770048,1499136=434176} 2020-06-12T02:38:11.788 INFO:tasks.rados.rados.0.smithi107.stdout:4128: writing smithi10729407-5 from 0 to 729088 tid 1 2020-06-12T02:38:11.788 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:11.789 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:38:11.789 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:38:11.789 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:38:11.789 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:11.793 INFO:tasks.rados.rados.0.smithi107.stdout:4128: writing smithi10729407-5 from 729088 to 1499136 tid 2 2020-06-12T02:38:11.796 INFO:tasks.rados.rados.0.smithi107.stdout:4128: writing smithi10729407-5 from 1499136 to 1933312 tid 3 2020-06-12T02:38:11.796 INFO:tasks.rados.rados.0.smithi107.stdout:4129: delete oid 11 current snap is 0 2020-06-12T02:38:11.796 INFO:tasks.rados.rados.0.smithi107.stdout:4125: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:11.796 INFO:tasks.rados.rados.0.smithi107.stdout:4125: finishing write tid 5 to smithi10729407-3 2020-06-12T02:38:11.797 INFO:tasks.rados.rados.0.smithi107.stdout:4125: finishing write tid 6 to smithi10729407-3 2020-06-12T02:38:11.797 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 741 (ObjNum 4124 snap 0 seq_num 4124) dirty exists 2020-06-12T02:38:11.797 INFO:tasks.rados.rados.0.smithi107.stdout:4125: left oid 3 (ObjNum 4124 snap 0 seq_num 4124) 2020-06-12T02:38:11.797 INFO:tasks.rados.rados.0.smithi107.stdout:4128: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:11.798 INFO:tasks.rados.rados.0.smithi107.stdout:4125: done (2 left) 2020-06-12T02:38:11.799 INFO:tasks.rados.rados.0.smithi107.stdout:4129: done (1 left) 2020-06-12T02:38:11.799 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:38:11.799 INFO:tasks.rados.rados.0.smithi107.stdout:4130: seq_num 4129 ranges {0=761856,761856=589824,1351680=614400} 2020-06-12T02:38:11.801 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-06-12T02:38:11.803 INFO:tasks.rados.rados.0.smithi107.stdout:4130: writing smithi10729407-1 from 0 to 761856 tid 1 2020-06-12T02:38:11.807 INFO:tasks.rados.rados.0.smithi107.stdout:4130: writing smithi10729407-1 from 761856 to 1351680 tid 2 2020-06-12T02:38:11.813 INFO:tasks.rados.rados.0.smithi107.stdout:4130: writing smithi10729407-1 from 1351680 to 1966080 tid 3 2020-06-12T02:38:11.813 INFO:tasks.rados.rados.0.smithi107.stdout:4131: delete oid 3 current snap is 0 2020-06-12T02:38:11.813 INFO:tasks.rados.rados.0.smithi107.stdout:4128: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:11.814 INFO:tasks.rados.rados.0.smithi107.stdout:4128: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:11.814 INFO:tasks.rados.rados.0.smithi107.stdout:4128: finishing write tid 5 to smithi10729407-5 2020-06-12T02:38:11.814 INFO:tasks.rados.rados.0.smithi107.stdout:4128: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:11.814 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 406 (ObjNum 4127 snap 0 seq_num 4127) dirty exists 2020-06-12T02:38:11.814 INFO:tasks.rados.rados.0.smithi107.stdout:4128: left oid 5 (ObjNum 4127 snap 0 seq_num 4127) 2020-06-12T02:38:11.815 INFO:tasks.rados.rados.0.smithi107.stdout:4130: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:11.815 INFO:tasks.rados.rados.0.smithi107.stdout:4128: done (2 left) 2020-06-12T02:38:11.816 INFO:tasks.rados.rados.0.smithi107.stdout:4131: done (1 left) 2020-06-12T02:38:11.816 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:11.816 INFO:tasks.rados.rados.0.smithi107.stdout:4132: seq_num 4131 ranges {0=794624,794624=761856,1556480=442368} 2020-06-12T02:38:11.822 INFO:tasks.rados.rados.0.smithi107.stdout:4132: writing smithi10729407-7 from 0 to 794624 tid 1 2020-06-12T02:38:11.827 INFO:tasks.rados.rados.0.smithi107.stdout:4132: writing smithi10729407-7 from 794624 to 1556480 tid 2 2020-06-12T02:38:11.830 INFO:tasks.rados.rados.0.smithi107.stdout:4132: writing smithi10729407-7 from 1556480 to 1998848 tid 3 2020-06-12T02:38:11.830 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:38:11.831 INFO:tasks.rados.rados.0.smithi107.stdout:4133: seq_num 4132 ranges {0=409600,409600=638976,1048576=548864,1597440=417792} 2020-06-12T02:38:11.833 INFO:tasks.rados.rados.0.smithi107.stdout:4133: writing smithi10729407-2 from 0 to 409600 tid 1 2020-06-12T02:38:11.838 INFO:tasks.rados.rados.0.smithi107.stdout:4133: writing smithi10729407-2 from 409600 to 1048576 tid 2 2020-06-12T02:38:11.842 INFO:tasks.rados.rados.0.smithi107.stdout:4133: writing smithi10729407-2 from 1048576 to 1597440 tid 3 2020-06-12T02:38:11.844 INFO:tasks.rados.rados.0.smithi107.stdout:4133: writing smithi10729407-2 from 1597440 to 2015232 tid 4 2020-06-12T02:38:11.845 INFO:tasks.rados.rados.0.smithi107.stdout:4134: delete oid 5 current snap is 0 2020-06-12T02:38:11.845 INFO:tasks.rados.rados.0.smithi107.stdout:4130: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:11.845 INFO:tasks.rados.rados.0.smithi107.stdout:4130: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:11.845 INFO:tasks.rados.rados.0.smithi107.stdout:4130: finishing write tid 5 to smithi10729407-1 2020-06-12T02:38:11.846 INFO:tasks.rados.rados.0.smithi107.stdout:4130: finishing write tid 6 to smithi10729407-1 2020-06-12T02:38:11.846 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 780 (ObjNum 4129 snap 0 seq_num 4129) dirty exists 2020-06-12T02:38:11.846 INFO:tasks.rados.rados.0.smithi107.stdout:4130: left oid 1 (ObjNum 4129 snap 0 seq_num 4129) 2020-06-12T02:38:11.846 INFO:tasks.rados.rados.0.smithi107.stdout:4132: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:11.847 INFO:tasks.rados.rados.0.smithi107.stdout:4132: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:11.847 INFO:tasks.rados.rados.0.smithi107.stdout:4132: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:11.847 INFO:tasks.rados.rados.0.smithi107.stdout:4132: finishing write tid 5 to smithi10729407-7 2020-06-12T02:38:11.847 INFO:tasks.rados.rados.0.smithi107.stdout:4132: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:11.848 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 776 (ObjNum 4131 snap 0 seq_num 4131) dirty exists 2020-06-12T02:38:11.848 INFO:tasks.rados.rados.0.smithi107.stdout:4132: left oid 7 (ObjNum 4131 snap 0 seq_num 4131) 2020-06-12T02:38:11.848 INFO:tasks.rados.rados.0.smithi107.stdout:4133: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:11.849 INFO:tasks.rados.rados.0.smithi107.stdout:4130: done (3 left) 2020-06-12T02:38:11.850 INFO:tasks.rados.rados.0.smithi107.stdout:4132: done (2 left) 2020-06-12T02:38:11.850 INFO:tasks.rados.rados.0.smithi107.stdout:4134: done (1 left) 2020-06-12T02:38:11.850 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:38:11.850 INFO:tasks.rados.rados.0.smithi107.stdout:4135: seq_num 4134 ranges {0=442368,442368=409600,851968=516096,1368064=442368,1810432=237568} 2020-06-12T02:38:11.852 INFO:tasks.rados.rados.0.smithi107.stdout:4135: writing smithi10729407-10 from 0 to 442368 tid 1 2020-06-12T02:38:11.855 INFO:tasks.rados.rados.0.smithi107.stdout:4135: writing smithi10729407-10 from 442368 to 851968 tid 2 2020-06-12T02:38:11.860 INFO:tasks.rados.rados.0.smithi107.stdout:4135: writing smithi10729407-10 from 851968 to 1368064 tid 3 2020-06-12T02:38:11.863 INFO:tasks.rados.rados.0.smithi107.stdout:4135: writing smithi10729407-10 from 1368064 to 1810432 tid 4 2020-06-12T02:38:11.865 INFO:tasks.rados.rados.0.smithi107.stdout:4135: writing smithi10729407-10 from 1810432 to 2048000 tid 5 2020-06-12T02:38:11.865 INFO:tasks.rados.rados.0.smithi107.stdout:4136: delete oid 19 current snap is 0 2020-06-12T02:38:11.867 INFO:tasks.rados.rados.0.smithi107.stdout:4133: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:11.867 INFO:tasks.rados.rados.0.smithi107.stdout:4133: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:11.867 INFO:tasks.rados.rados.0.smithi107.stdout:4135: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:11.868 INFO:tasks.rados.rados.0.smithi107.stdout:4133: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:11.868 INFO:tasks.rados.rados.0.smithi107.stdout:4133: finishing write tid 6 to smithi10729407-2 2020-06-12T02:38:11.868 INFO:tasks.rados.rados.0.smithi107.stdout:4133: finishing write tid 7 to smithi10729407-2 2020-06-12T02:38:11.868 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 431 (ObjNum 4132 snap 0 seq_num 4132) dirty exists 2020-06-12T02:38:11.868 INFO:tasks.rados.rados.0.smithi107.stdout:4133: left oid 2 (ObjNum 4132 snap 0 seq_num 4132) 2020-06-12T02:38:11.870 INFO:tasks.rados.rados.0.smithi107.stdout:4133: done (2 left) 2020-06-12T02:38:11.870 INFO:tasks.rados.rados.0.smithi107.stdout:4136: done (1 left) 2020-06-12T02:38:11.870 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:38:11.870 INFO:tasks.rados.rados.0.smithi107.stdout:4137: seq_num 4136 ranges {0=475136,475136=630784,1105920=524288,1630208=450560} 2020-06-12T02:38:11.872 INFO:tasks.rados.rados.0.smithi107.stdout:4137: writing smithi10729407-13 from 0 to 475136 tid 1 2020-06-12T02:38:11.877 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:11.874+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:11.877 INFO:tasks.rados.rados.0.smithi107.stdout:4137: writing smithi10729407-13 from 475136 to 1105920 tid 2 2020-06-12T02:38:11.881 INFO:tasks.rados.rados.0.smithi107.stdout:4137: writing smithi10729407-13 from 1105920 to 1630208 tid 3 2020-06-12T02:38:11.887 INFO:tasks.rados.rados.0.smithi107.stdout:4137: writing smithi10729407-13 from 1630208 to 2080768 tid 4 2020-06-12T02:38:11.887 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:38:11.888 INFO:tasks.rados.rados.0.smithi107.stdout:4138: seq_num 4137 ranges {0=491520,491520=516096,1007616=688128,1695744=401408} 2020-06-12T02:38:11.888 INFO:tasks.rados.rados.0.smithi107.stdout:4138: writing smithi10729407-11 from 0 to 491520 tid 1 2020-06-12T02:38:11.892 INFO:tasks.rados.rados.0.smithi107.stdout:4138: writing smithi10729407-11 from 491520 to 1007616 tid 2 2020-06-12T02:38:11.895 INFO:tasks.rados.rados.0.smithi107.stdout:4138: writing smithi10729407-11 from 1007616 to 1695744 tid 3 2020-06-12T02:38:11.899 INFO:tasks.rados.rados.0.smithi107.stdout:4138: writing smithi10729407-11 from 1695744 to 2097152 tid 4 2020-06-12T02:38:11.899 INFO:tasks.rados.rados.0.smithi107.stdout:4139: delete oid 12 current snap is 0 2020-06-12T02:38:11.899 INFO:tasks.rados.rados.0.smithi107.stdout:4135: finishing write tid 2 to smithi10729407-10 2020-06-12T02:38:11.899 INFO:tasks.rados.rados.0.smithi107.stdout:4135: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:11.900 INFO:tasks.rados.rados.0.smithi107.stdout:4137: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:11.900 INFO:tasks.rados.rados.0.smithi107.stdout:4135: finishing write tid 4 to smithi10729407-10 2020-06-12T02:38:11.900 INFO:tasks.rados.rados.0.smithi107.stdout:4135: finishing write tid 5 to smithi10729407-10 2020-06-12T02:38:11.900 INFO:tasks.rados.rados.0.smithi107.stdout:4135: finishing write tid 7 to smithi10729407-10 2020-06-12T02:38:11.900 INFO:tasks.rados.rados.0.smithi107.stdout:4137: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:11.901 INFO:tasks.rados.rados.0.smithi107.stdout:4137: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:11.901 INFO:tasks.rados.rados.0.smithi107.stdout:4138: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:11.901 INFO:tasks.rados.rados.0.smithi107.stdout:4135: finishing write tid 8 to smithi10729407-10 2020-06-12T02:38:11.901 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 412 (ObjNum 4134 snap 0 seq_num 4134) dirty exists 2020-06-12T02:38:11.901 INFO:tasks.rados.rados.0.smithi107.stdout:4135: left oid 10 (ObjNum 4134 snap 0 seq_num 4134) 2020-06-12T02:38:11.902 INFO:tasks.rados.rados.0.smithi107.stdout:4137: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:11.902 INFO:tasks.rados.rados.0.smithi107.stdout:4137: finishing write tid 6 to smithi10729407-13 2020-06-12T02:38:11.903 INFO:tasks.rados.rados.0.smithi107.stdout:4135: done (3 left) 2020-06-12T02:38:11.904 INFO:tasks.rados.rados.0.smithi107.stdout:4139: done (2 left) 2020-06-12T02:38:11.904 INFO:tasks.rados.rados.0.smithi107.stdout:4140: delete oid 1 current snap is 0 2020-06-12T02:38:11.905 INFO:tasks.rados.rados.0.smithi107.stdout:4137: finishing write tid 7 to smithi10729407-13 2020-06-12T02:38:11.905 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 781 (ObjNum 4136 snap 0 seq_num 4136) dirty exists 2020-06-12T02:38:11.905 INFO:tasks.rados.rados.0.smithi107.stdout:4137: left oid 13 (ObjNum 4136 snap 0 seq_num 4136) 2020-06-12T02:38:11.909 INFO:tasks.rados.rados.0.smithi107.stdout:4138: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:11.909 INFO:tasks.rados.rados.0.smithi107.stdout:4138: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:11.912 INFO:tasks.rados.rados.0.smithi107.stdout:4137: done (2 left) 2020-06-12T02:38:11.912 INFO:tasks.rados.rados.0.smithi107.stdout:4140: done (1 left) 2020-06-12T02:38:11.912 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:11.912 INFO:tasks.rados.rados.0.smithi107.stdout:4141: seq_num 4140 ranges {0=540672,540672=565248,1105920=507904,1613824=532480} 2020-06-12T02:38:11.914 INFO:tasks.rados.rados.0.smithi107.stdout:4141: writing smithi10729407-12 from 0 to 540672 tid 1 2020-06-12T02:38:11.918 INFO:tasks.rados.rados.0.smithi107.stdout:4141: writing smithi10729407-12 from 540672 to 1105920 tid 2 2020-06-12T02:38:11.921 INFO:tasks.rados.rados.0.smithi107.stdout:4141: writing smithi10729407-12 from 1105920 to 1613824 tid 3 2020-06-12T02:38:11.925 INFO:tasks.rados.rados.0.smithi107.stdout:4141: writing smithi10729407-12 from 1613824 to 2146304 tid 4 2020-06-12T02:38:11.925 INFO:tasks.rados.rados.0.smithi107.stdout:4142: delete oid 9 current snap is 0 2020-06-12T02:38:11.926 INFO:tasks.rados.rados.0.smithi107.stdout:4138: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:11.926 INFO:tasks.rados.rados.0.smithi107.stdout:4138: finishing write tid 6 to smithi10729407-11 2020-06-12T02:38:11.926 INFO:tasks.rados.rados.0.smithi107.stdout:4138: finishing write tid 7 to smithi10729407-11 2020-06-12T02:38:11.926 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 727 (ObjNum 4137 snap 0 seq_num 4137) dirty exists 2020-06-12T02:38:11.926 INFO:tasks.rados.rados.0.smithi107.stdout:4138: left oid 11 (ObjNum 4137 snap 0 seq_num 4137) 2020-06-12T02:38:11.927 INFO:tasks.rados.rados.0.smithi107.stdout:4141: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:11.927 INFO:tasks.rados.rados.0.smithi107.stdout:4138: done (2 left) 2020-06-12T02:38:11.927 INFO:tasks.rados.rados.0.smithi107.stdout:4142: done (1 left) 2020-06-12T02:38:11.928 INFO:tasks.rados.rados.0.smithi107.stdout:4143: delete oid 3 current snap is 0 2020-06-12T02:38:11.929 INFO:tasks.rados.rados.0.smithi107.stdout:4143: done (1 left) 2020-06-12T02:38:11.929 INFO:tasks.rados.rados.0.smithi107.stdout:4144: delete oid 11 current snap is 0 2020-06-12T02:38:11.931 INFO:tasks.rados.rados.0.smithi107.stdout:4141: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:11.931 INFO:tasks.rados.rados.0.smithi107.stdout:4141: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:11.933 INFO:tasks.rados.rados.0.smithi107.stdout:4144: done (1 left) 2020-06-12T02:38:11.934 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:38:11.934 INFO:tasks.rados.rados.0.smithi107.stdout:4145: seq_num 4144 ranges {2080768=606208,2686976=548864,3235840=491520,3727360=565248} 2020-06-12T02:38:11.948 INFO:tasks.rados.rados.0.smithi107.stdout:4145: writing smithi10729407-13 from 2080768 to 2686976 tid 1 2020-06-12T02:38:11.950 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:11.950 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:38:11.950 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:38:11.950 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:38:11.951 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.951 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1206 3.5s0 3:a9510bda:::smithi10729407-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:38:11.951 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.856688+0000", 2020-06-12T02:38:11.951 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 39.089099425999997, 2020-06-12T02:38:11.951 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019116942000000001, 2020-06-12T02:38:11.952 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:11.952 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:11.952 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:11.952 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:11.952 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:11.952 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1206 2020-06-12T02:38:11.953 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.953 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:11.953 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.953 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:11.953 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856688+0000", 2020-06-12T02:38:11.954 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:11.954 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.954 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.954 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:11.954 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856688+0000", 2020-06-12T02:38:11.955 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4355e-05 2020-06-12T02:38:11.955 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.955 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.955 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:11.955 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856713+0000", 2020-06-12T02:38:11.956 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0850000000000001e-06 2020-06-12T02:38:11.956 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.956 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.956 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:11.956 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856716+0000", 2020-06-12T02:38:11.956 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4509999999999997e-06 2020-06-12T02:38:11.957 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.957 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.957 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:11.957 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856722+0000", 2020-06-12T02:38:11.957 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2879999999999999e-06 2020-06-12T02:38:11.958 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.958 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.958 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:11.958 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856724+0000", 2020-06-12T02:38:11.958 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0057507449999999998 2020-06-12T02:38:11.959 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.959 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.959 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.959 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.862475+0000", 2020-06-12T02:38:11.959 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5759000000000003e-05 2020-06-12T02:38:11.960 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.960 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.960 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:11.960 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.862571+0000", 2020-06-12T02:38:11.961 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012199018000000001 2020-06-12T02:38:11.961 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.961 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.961 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.961 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.874770+0000", 2020-06-12T02:38:11.962 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7260999999999999e-05 2020-06-12T02:38:11.962 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.962 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.962 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:11.963 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.874827+0000", 2020-06-12T02:38:11.963 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00097798000000000004 2020-06-12T02:38:11.963 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.963 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.963 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:11.963 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.875805+0000", 2020-06-12T02:38:11.964 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019116942000000001 2020-06-12T02:38:11.964 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.964 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:11.964 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.964 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.965 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.965 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1572 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~745472] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-06-12T02:38:11.965 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.886439+0000", 2020-06-12T02:38:11.965 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 38.059348708000002, 2020-06-12T02:38:11.965 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021442571000000001, 2020-06-12T02:38:11.965 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:11.966 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:11.966 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:11.967 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:11.967 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:11.967 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1572 2020-06-12T02:38:11.967 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.967 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:11.967 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.968 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:11.968 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.886439+0000", 2020-06-12T02:38:11.968 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:11.968 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.968 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.969 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:11.969 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.886439+0000", 2020-06-12T02:38:11.969 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00073027599999999997 2020-06-12T02:38:11.969 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.970 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.970 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:11.970 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887169+0000", 2020-06-12T02:38:11.970 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.49e-06 2020-06-12T02:38:11.970 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.970 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.971 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:11.971 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887175+0000", 2020-06-12T02:38:11.971 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7899999999999994e-06 2020-06-12T02:38:11.971 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.971 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.972 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:11.972 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887185+0000", 2020-06-12T02:38:11.972 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.42e-06 2020-06-12T02:38:11.972 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.972 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.972 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:11.973 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887189+0000", 2020-06-12T02:38:11.973 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1619999999999999e-05 2020-06-12T02:38:11.973 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.973 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.973 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:11.974 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887221+0000", 2020-06-12T02:38:11.974 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9720000000000002e-05 2020-06-12T02:38:11.974 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.974 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.974 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:11.975 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887300+0000", 2020-06-12T02:38:11.975 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047354100000000001 2020-06-12T02:38:11.975 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.975 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.975 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:11.975 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887774+0000", 2020-06-12T02:38:11.976 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0030649750000000002 2020-06-12T02:38:11.976 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.976 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.976 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:11.976 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.890839+0000", 2020-06-12T02:38:11.977 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016995321000000001 2020-06-12T02:38:11.977 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.977 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.977 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:11.977 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.907834+0000", 2020-06-12T02:38:11.977 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7417999999999999e-05 2020-06-12T02:38:11.978 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.978 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.978 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:11.978 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.907882+0000", 2020-06-12T02:38:11.978 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021442571000000001 2020-06-12T02:38:11.979 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.979 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:11.979 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:11.979 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.979 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.980 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1573 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~712704] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-06-12T02:38:11.980 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.890888+0000", 2020-06-12T02:38:11.980 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 38.054899972999998, 2020-06-12T02:38:11.980 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018868001999999998, 2020-06-12T02:38:11.980 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:11.981 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:11.981 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:11.981 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:11.981 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:11.981 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1573 2020-06-12T02:38:11.982 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.982 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:11.982 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.982 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:11.982 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.890888+0000", 2020-06-12T02:38:11.982 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:11.983 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.983 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.983 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:11.983 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.890888+0000", 2020-06-12T02:38:11.983 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000683605 2020-06-12T02:38:11.984 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.984 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.984 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:11.984 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891571+0000", 2020-06-12T02:38:11.984 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4999999999999999e-06 2020-06-12T02:38:11.984 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:11.985 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:11.985 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:11.985 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891577+0000", 2020-06-12T02:38:11.985 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.171e-05 2020-06-12T02:38:11.985 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.002 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.002 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:12.002 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891589+0000", 2020-06-12T02:38:12.002 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7650000000000001e-06 2020-06-12T02:38:12.003 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.003 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.003 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:12.003 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891593+0000", 2020-06-12T02:38:12.003 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6689000000000001e-05 2020-06-12T02:38:12.004 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.004 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.004 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:12.004 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891630+0000", 2020-06-12T02:38:12.004 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9877999999999997e-05 2020-06-12T02:38:12.005 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.005 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.005 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:12.005 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891710+0000", 2020-06-12T02:38:12.006 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053367200000000001 2020-06-12T02:38:12.006 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.006 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.006 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:12.007 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.892244+0000", 2020-06-12T02:38:12.007 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004927196 2020-06-12T02:38:12.007 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.007 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.007 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:12.008 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.897171+0000", 2020-06-12T02:38:12.008 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012496419 2020-06-12T02:38:12.008 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.008 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.008 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:12.009 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.909667+0000", 2020-06-12T02:38:12.009 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8567999999999996e-05 2020-06-12T02:38:12.009 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.009 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.010 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:12.010 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.909756+0000", 2020-06-12T02:38:12.010 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018868001999999998 2020-06-12T02:38:12.010 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.011 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:12.011 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.011 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.011 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.011 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1576 3.5s0 3:a446366d:::smithi10729407-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e26)", 2020-06-12T02:38:12.012 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.891955+0000", 2020-06-12T02:38:12.012 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 38.053832733, 2020-06-12T02:38:12.012 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019893246999999999, 2020-06-12T02:38:12.012 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:12.012 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:12.013 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:12.013 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:12.013 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:12.013 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1576 2020-06-12T02:38:12.013 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.014 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:12.014 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.014 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:12.014 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891955+0000", 2020-06-12T02:38:12.015 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:12.015 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.015 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.015 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:12.016 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891955+0000", 2020-06-12T02:38:12.016 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2067999999999999e-05 2020-06-12T02:38:12.016 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.016 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.016 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:12.017 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891977+0000", 2020-06-12T02:38:12.017 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.18e-06 2020-06-12T02:38:12.017 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.017 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.017 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:12.017 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891980+0000", 2020-06-12T02:38:12.018 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3629999999999999e-06 2020-06-12T02:38:12.018 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.018 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.018 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:12.018 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891987+0000", 2020-06-12T02:38:12.019 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.181e-06 2020-06-12T02:38:12.019 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.019 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.019 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:12.019 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891989+0000", 2020-06-12T02:38:12.020 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050959120000000002 2020-06-12T02:38:12.020 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.020 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.020 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:12.020 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.897085+0000", 2020-06-12T02:38:12.020 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1089999999999998e-05 2020-06-12T02:38:12.021 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.021 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.021 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:12.021 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.897166+0000", 2020-06-12T02:38:12.021 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013400235 2020-06-12T02:38:12.022 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.022 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.022 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:12.023 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.910566+0000", 2020-06-12T02:38:12.023 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000104981 2020-06-12T02:38:12.023 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.023 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.023 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:12.024 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.910671+0000", 2020-06-12T02:38:12.024 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001177237 2020-06-12T02:38:12.024 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.024 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.024 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:12.025 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.911848+0000", 2020-06-12T02:38:12.025 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019893246999999999 2020-06-12T02:38:12.025 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.025 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:12.025 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.026 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.026 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.026 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2913 3.5s0 3:a9510bda:::smithi10729407-14:head [append 0~770048,stat] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:12.026 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.070354+0000", 2020-06-12T02:38:12.026 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.875433932, 2020-06-12T02:38:12.027 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60690798599999995, 2020-06-12T02:38:12.027 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:12.027 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:12.027 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:12.027 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:12.027 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:12.028 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2913 2020-06-12T02:38:12.028 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.028 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:12.028 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.028 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:12.029 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.070354+0000", 2020-06-12T02:38:12.029 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:12.029 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.029 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.029 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:12.030 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.070354+0000", 2020-06-12T02:38:12.030 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00098478900000000002 2020-06-12T02:38:12.030 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.030 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.030 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:12.031 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071339+0000", 2020-06-12T02:38:12.031 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8349999999999993e-06 2020-06-12T02:38:12.031 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.031 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.032 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:12.032 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071347+0000", 2020-06-12T02:38:12.032 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0749e-05 2020-06-12T02:38:12.032 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.032 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.033 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:12.033 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071358+0000", 2020-06-12T02:38:12.033 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9369999999999997e-06 2020-06-12T02:38:12.033 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.033 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.034 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:12.034 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071362+0000", 2020-06-12T02:38:12.034 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0426e-05 2020-06-12T02:38:12.034 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.034 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.034 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:12.035 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071423+0000", 2020-06-12T02:38:12.035 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4129999999999998e-05 2020-06-12T02:38:12.035 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.035 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.035 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:12.036 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071487+0000", 2020-06-12T02:38:12.036 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053752599999999998 2020-06-12T02:38:12.036 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.036 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.036 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:12.037 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072024+0000", 2020-06-12T02:38:12.037 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036162579999999998 2020-06-12T02:38:12.037 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.037 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.037 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:12.038 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075640+0000", 2020-06-12T02:38:12.038 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60152201999999999 2020-06-12T02:38:12.038 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.038 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.038 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:12.039 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677162+0000", 2020-06-12T02:38:12.039 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9315999999999996e-05 2020-06-12T02:38:12.039 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.039 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.039 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:12.040 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677262+0000", 2020-06-12T02:38:12.040 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60690798599999995 2020-06-12T02:38:12.040 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.040 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:12.041 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.041 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.041 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.041 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2914 3.5s0 3:a9510bda:::smithi10729407-14:head [append 0~401408] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:12.041 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.072409+0000", 2020-06-12T02:38:12.042 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.873378488, 2020-06-12T02:38:12.042 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.61956801100000003, 2020-06-12T02:38:12.042 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:12.042 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:12.042 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:12.043 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:12.043 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:12.043 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2914 2020-06-12T02:38:12.043 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.043 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:12.044 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.044 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:12.044 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072409+0000", 2020-06-12T02:38:12.044 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:12.045 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.045 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.045 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:12.045 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072409+0000", 2020-06-12T02:38:12.045 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000446317 2020-06-12T02:38:12.045 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.046 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.046 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:12.046 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072856+0000", 2020-06-12T02:38:12.046 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2140000000000004e-06 2020-06-12T02:38:12.047 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.047 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.047 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:12.047 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072861+0000", 2020-06-12T02:38:12.047 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8710000000000003e-06 2020-06-12T02:38:12.047 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.048 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.048 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:12.048 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072870+0000", 2020-06-12T02:38:12.048 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2270000000000004e-06 2020-06-12T02:38:12.048 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.049 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.049 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:12.049 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072874+0000", 2020-06-12T02:38:12.049 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024649960000000001 2020-06-12T02:38:12.050 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.050 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.050 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:12.050 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075339+0000", 2020-06-12T02:38:12.051 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9179999999999996e-05 2020-06-12T02:38:12.051 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.051 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.051 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:12.051 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075438+0000", 2020-06-12T02:38:12.052 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60189281400000005 2020-06-12T02:38:12.052 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.052 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.052 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:12.052 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677331+0000", 2020-06-12T02:38:12.053 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012602800000000001 2020-06-12T02:38:12.053 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.053 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.053 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:12.053 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677457+0000", 2020-06-12T02:38:12.054 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00088167699999999998 2020-06-12T02:38:12.054 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.054 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.054 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:12.054 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.678339+0000", 2020-06-12T02:38:12.054 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004312995 2020-06-12T02:38:12.055 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.055 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.055 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:12.055 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.682652+0000", 2020-06-12T02:38:12.056 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0092869140000000003 2020-06-12T02:38:12.056 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.056 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.056 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:12.056 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691938+0000", 2020-06-12T02:38:12.056 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8778e-05 2020-06-12T02:38:12.057 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.057 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.057 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:12.057 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691977+0000", 2020-06-12T02:38:12.057 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.61956801100000003 2020-06-12T02:38:12.058 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.058 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:12.058 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.058 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.059 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.059 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2915 3.5s0 3:a9510bda:::smithi10729407-14:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:12.059 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.072912+0000", 2020-06-12T02:38:12.059 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.872875747999998, 2020-06-12T02:38:12.060 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.6192204, 2020-06-12T02:38:12.060 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:12.060 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:12.060 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:12.060 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:12.060 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:12.061 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2915 2020-06-12T02:38:12.061 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.061 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:12.061 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.061 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:12.062 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072912+0000", 2020-06-12T02:38:12.062 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:12.062 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.062 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.062 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:12.063 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072912+0000", 2020-06-12T02:38:12.063 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9201e-05 2020-06-12T02:38:12.063 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.063 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.063 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:12.063 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072951+0000", 2020-06-12T02:38:12.064 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5480000000000002e-06 2020-06-12T02:38:12.064 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.064 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.064 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:12.064 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072955+0000", 2020-06-12T02:38:12.065 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3879999999999997e-06 2020-06-12T02:38:12.065 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.065 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.065 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:12.065 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072961+0000", 2020-06-12T02:38:12.065 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6589999999999999e-06 2020-06-12T02:38:12.066 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.066 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.066 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:12.066 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072964+0000", 2020-06-12T02:38:12.066 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025221039999999998 2020-06-12T02:38:12.067 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.067 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.067 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:12.067 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075486+0000", 2020-06-12T02:38:12.068 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5038999999999998e-05 2020-06-12T02:38:12.068 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.068 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.068 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:12.068 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075561+0000", 2020-06-12T02:38:12.069 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60452397499999999 2020-06-12T02:38:12.069 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.069 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.069 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:12.069 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680085+0000", 2020-06-12T02:38:12.070 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1398999999999999e-05 2020-06-12T02:38:12.070 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.070 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.070 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:12.070 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680156+0000", 2020-06-12T02:38:12.070 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000274465 2020-06-12T02:38:12.071 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.071 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.071 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:12.071 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680431+0000", 2020-06-12T02:38:12.071 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025071999999999998 2020-06-12T02:38:12.072 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.072 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.072 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:12.072 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.682938+0000", 2020-06-12T02:38:12.072 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0091116449999999998 2020-06-12T02:38:12.072 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.073 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.073 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:12.073 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692050+0000", 2020-06-12T02:38:12.073 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2776999999999997e-05 2020-06-12T02:38:12.073 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.074 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.074 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:12.074 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692132+0000", 2020-06-12T02:38:12.074 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.6192204 2020-06-12T02:38:12.074 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.075 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:12.075 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.075 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.075 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.075 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2916 3.5s0 3:a9510bda:::smithi10729407-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:38:12.075 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.072986+0000", 2020-06-12T02:38:12.076 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.872801508999999, 2020-06-12T02:38:12.076 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.62063941099999997, 2020-06-12T02:38:12.076 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:12.076 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:12.076 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:12.077 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:12.077 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:12.077 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2916 2020-06-12T02:38:12.077 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.078 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:12.078 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.078 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:12.078 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072986+0000", 2020-06-12T02:38:12.078 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:12.079 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.079 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.079 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:12.079 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072986+0000", 2020-06-12T02:38:12.079 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2101000000000001e-05 2020-06-12T02:38:12.079 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.080 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.080 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:12.080 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073008+0000", 2020-06-12T02:38:12.080 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1549999999999999e-06 2020-06-12T02:38:12.080 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.081 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.081 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:12.081 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073011+0000", 2020-06-12T02:38:12.081 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2360000000000001e-06 2020-06-12T02:38:12.081 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.082 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.082 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:12.082 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073018+0000", 2020-06-12T02:38:12.082 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2790000000000001e-06 2020-06-12T02:38:12.082 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.082 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.083 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:12.083 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073020+0000", 2020-06-12T02:38:12.083 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025637670000000002 2020-06-12T02:38:12.083 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.083 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.084 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:12.084 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075584+0000", 2020-06-12T02:38:12.084 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1814999999999998e-05 2020-06-12T02:38:12.084 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.084 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.085 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:12.085 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075636+0000", 2020-06-12T02:38:12.085 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60499943199999995 2020-06-12T02:38:12.085 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.086 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.086 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:12.086 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680635+0000", 2020-06-12T02:38:12.086 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0968000000000002e-05 2020-06-12T02:38:12.087 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.087 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.087 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:12.087 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680706+0000", 2020-06-12T02:38:12.087 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011777101999999999 2020-06-12T02:38:12.088 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.088 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.088 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:12.088 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692483+0000", 2020-06-12T02:38:12.088 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5890999999999995e-05 2020-06-12T02:38:12.089 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.089 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.089 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:12.089 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692569+0000", 2020-06-12T02:38:12.089 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001056665 2020-06-12T02:38:12.089 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.090 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.090 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:12.090 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.693626+0000", 2020-06-12T02:38:12.096 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.62063941099999997 2020-06-12T02:38:12.096 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.096 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:12.097 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.097 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.097 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.097 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2922 3.ds1 3:b983fd28:::smithi10729407-1:head [append 0~466944] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:12.097 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.097192+0000", 2020-06-12T02:38:12.098 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.848596141999998, 2020-06-12T02:38:12.098 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58672644699999998, 2020-06-12T02:38:12.098 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:12.098 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:12.098 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:12.099 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:12.099 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:12.099 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2922 2020-06-12T02:38:12.099 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.099 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:12.099 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.100 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:12.100 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097192+0000", 2020-06-12T02:38:12.100 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:12.100 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.100 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.101 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:12.101 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097192+0000", 2020-06-12T02:38:12.101 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047206100000000002 2020-06-12T02:38:12.101 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.101 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.102 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:12.102 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097664+0000", 2020-06-12T02:38:12.102 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8359999999999999e-06 2020-06-12T02:38:12.102 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.102 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.102 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:12.103 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097668+0000", 2020-06-12T02:38:12.103 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0815999999999999e-05 2020-06-12T02:38:12.103 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.103 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.104 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:12.104 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097679+0000", 2020-06-12T02:38:12.104 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.028e-06 2020-06-12T02:38:12.104 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.104 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.105 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:12.105 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097684+0000", 2020-06-12T02:38:12.105 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3932999999999999e-05 2020-06-12T02:38:12.105 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.106 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.106 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:12.106 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097748+0000", 2020-06-12T02:38:12.106 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2214999999999999e-05 2020-06-12T02:38:12.106 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.106 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.107 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:12.107 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097820+0000", 2020-06-12T02:38:12.107 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00041583800000000003 2020-06-12T02:38:12.107 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.107 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.108 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:12.108 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.098236+0000", 2020-06-12T02:38:12.108 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0020339960000000002 2020-06-12T02:38:12.108 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.108 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.109 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:12.109 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.100270+0000", 2020-06-12T02:38:12.109 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58355657299999997 2020-06-12T02:38:12.109 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.110 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.110 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:12.110 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.683827+0000", 2020-06-12T02:38:12.110 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1150999999999995e-05 2020-06-12T02:38:12.110 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.111 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.111 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:12.111 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.683918+0000", 2020-06-12T02:38:12.111 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58672644699999998 2020-06-12T02:38:12.111 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.111 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:12.112 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.112 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.112 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.112 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2923 3.ds1 3:b983fd28:::smithi10729407-1:head [append 0~614400] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:12.113 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.100504+0000", 2020-06-12T02:38:12.113 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.845283500999997, 2020-06-12T02:38:12.113 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58853946999999995, 2020-06-12T02:38:12.113 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:12.114 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:12.114 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:12.114 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:12.114 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:12.114 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2923 2020-06-12T02:38:12.115 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.115 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:12.115 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.115 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:12.115 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.100504+0000", 2020-06-12T02:38:12.115 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:12.116 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.116 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.116 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:12.116 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.100504+0000", 2020-06-12T02:38:12.116 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00068395399999999996 2020-06-12T02:38:12.117 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.117 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.117 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:12.117 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101188+0000", 2020-06-12T02:38:12.117 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4050000000000003e-06 2020-06-12T02:38:12.118 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.118 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.118 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:12.118 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101196+0000", 2020-06-12T02:38:12.118 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.328e-05 2020-06-12T02:38:12.119 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.119 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.119 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:12.119 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101209+0000", 2020-06-12T02:38:12.119 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3419999999999996e-06 2020-06-12T02:38:12.120 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.120 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.120 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:12.120 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101215+0000", 2020-06-12T02:38:12.120 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1389000000000005e-05 2020-06-12T02:38:12.121 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.121 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.121 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:12.121 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101277+0000", 2020-06-12T02:38:12.121 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7724000000000004e-05 2020-06-12T02:38:12.122 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.122 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.122 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:12.122 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101344+0000", 2020-06-12T02:38:12.123 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047852499999999999 2020-06-12T02:38:12.123 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.123 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.123 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:12.123 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101823+0000", 2020-06-12T02:38:12.124 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0039255510000000002 2020-06-12T02:38:12.124 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.124 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.124 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:12.124 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105748+0000", 2020-06-12T02:38:12.124 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58321340099999996 2020-06-12T02:38:12.125 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.125 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.125 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:12.125 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.688962+0000", 2020-06-12T02:38:12.125 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1898999999999996e-05 2020-06-12T02:38:12.126 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.126 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.126 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:12.126 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689044+0000", 2020-06-12T02:38:12.126 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58853946999999995 2020-06-12T02:38:12.127 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.127 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:12.127 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.127 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.127 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.128 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2924 3.ds1 3:b983fd28:::smithi10729407-1:head [append 0~188416] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:12.128 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.101633+0000", 2020-06-12T02:38:12.128 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.844154506999999, 2020-06-12T02:38:12.128 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58758259199999996, 2020-06-12T02:38:12.128 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:12.128 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:12.129 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:12.129 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:12.129 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:12.129 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2924 2020-06-12T02:38:12.129 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.130 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:12.130 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.130 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:12.130 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101633+0000", 2020-06-12T02:38:12.131 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:12.131 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.131 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.131 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:12.132 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101633+0000", 2020-06-12T02:38:12.132 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00018775399999999999 2020-06-12T02:38:12.132 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.132 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.132 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:12.132 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101821+0000", 2020-06-12T02:38:12.133 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7669999999999998e-06 2020-06-12T02:38:12.133 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.133 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.133 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:12.133 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101828+0000", 2020-06-12T02:38:12.134 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2716e-05 2020-06-12T02:38:12.134 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.134 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.134 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:12.134 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101840+0000", 2020-06-12T02:38:12.134 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.215e-06 2020-06-12T02:38:12.135 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.135 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.135 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:12.135 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101846+0000", 2020-06-12T02:38:12.135 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021214250000000001 2020-06-12T02:38:12.136 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.136 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.136 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:12.136 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.103967+0000", 2020-06-12T02:38:12.136 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1018000000000003e-05 2020-06-12T02:38:12.136 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.137 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.137 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:12.137 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.104038+0000", 2020-06-12T02:38:12.137 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000333364 2020-06-12T02:38:12.137 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.138 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.138 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:12.138 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.104371+0000", 2020-06-12T02:38:12.138 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001378546 2020-06-12T02:38:12.138 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.138 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.139 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:12.139 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105750+0000", 2020-06-12T02:38:12.139 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58338272899999999 2020-06-12T02:38:12.140 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.140 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.140 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:12.140 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689133+0000", 2020-06-12T02:38:12.140 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3058000000000003e-05 2020-06-12T02:38:12.141 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.141 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.141 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:12.141 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689216+0000", 2020-06-12T02:38:12.141 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58758259199999996 2020-06-12T02:38:12.141 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.142 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:12.142 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.142 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.142 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.142 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2925 3.ds1 3:b983fd28:::smithi10729407-1:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:12.143 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.101899+0000", 2020-06-12T02:38:12.143 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.843888841999998, 2020-06-12T02:38:12.143 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58747810899999997, 2020-06-12T02:38:12.143 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:12.143 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:12.144 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:12.144 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:12.144 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:12.144 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2925 2020-06-12T02:38:12.144 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.144 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:12.145 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.145 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:12.145 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101899+0000", 2020-06-12T02:38:12.145 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:12.145 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.146 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.146 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:12.146 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101899+0000", 2020-06-12T02:38:12.146 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2906999999999998e-05 2020-06-12T02:38:12.146 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.147 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.147 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:12.147 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101972+0000", 2020-06-12T02:38:12.147 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1789999999999996e-06 2020-06-12T02:38:12.147 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.147 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.148 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:12.148 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101978+0000", 2020-06-12T02:38:12.148 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1358e-05 2020-06-12T02:38:12.148 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.148 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.148 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:12.149 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101989+0000", 2020-06-12T02:38:12.149 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.031e-06 2020-06-12T02:38:12.149 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.149 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.149 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:12.150 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101993+0000", 2020-06-12T02:38:12.150 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033218990000000001 2020-06-12T02:38:12.150 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.150 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.150 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:12.151 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105315+0000", 2020-06-12T02:38:12.151 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6324999999999999e-05 2020-06-12T02:38:12.151 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.151 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.151 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:12.152 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105382+0000", 2020-06-12T02:38:12.152 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000216624 2020-06-12T02:38:12.152 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.152 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.152 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:12.153 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105598+0000", 2020-06-12T02:38:12.153 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001236586 2020-06-12T02:38:12.153 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.153 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.153 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:12.154 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.106835+0000", 2020-06-12T02:38:12.154 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58245509200000001 2020-06-12T02:38:12.154 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.155 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.155 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:12.155 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689290+0000", 2020-06-12T02:38:12.156 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7107999999999993e-05 2020-06-12T02:38:12.156 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.156 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.156 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:12.156 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689377+0000", 2020-06-12T02:38:12.157 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58747810899999997 2020-06-12T02:38:12.157 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.157 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:12.157 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.157 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.158 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.158 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2926 3.ds1 3:b983fd28:::smithi10729407-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:38:12.158 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.102036+0000", 2020-06-12T02:38:12.158 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.843751857999997, 2020-06-12T02:38:12.158 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58868048699999997, 2020-06-12T02:38:12.158 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:12.159 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:12.159 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:12.159 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:12.159 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:12.159 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2926 2020-06-12T02:38:12.160 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.160 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:12.160 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.160 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:12.160 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102036+0000", 2020-06-12T02:38:12.160 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:12.161 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.161 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.161 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:12.162 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102036+0000", 2020-06-12T02:38:12.162 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3786999999999999e-05 2020-06-12T02:38:12.162 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.162 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.162 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:12.163 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102080+0000", 2020-06-12T02:38:12.163 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7660000000000004e-06 2020-06-12T02:38:12.163 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.163 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.163 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:12.163 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102085+0000", 2020-06-12T02:38:12.164 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3145e-05 2020-06-12T02:38:12.164 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.164 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.164 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:12.164 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102099+0000", 2020-06-12T02:38:12.165 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2440000000000004e-06 2020-06-12T02:38:12.165 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.165 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.165 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:12.165 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102103+0000", 2020-06-12T02:38:12.166 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036826990000000002 2020-06-12T02:38:12.166 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.166 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.166 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:12.166 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105786+0000", 2020-06-12T02:38:12.167 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.6774000000000006e-05 2020-06-12T02:38:12.167 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.167 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.167 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:12.167 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105882+0000", 2020-06-12T02:38:12.168 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58352084900000001 2020-06-12T02:38:12.168 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.168 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.168 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:12.168 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689403+0000", 2020-06-12T02:38:12.169 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3788999999999995e-05 2020-06-12T02:38:12.169 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.169 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.169 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:12.169 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689467+0000", 2020-06-12T02:38:12.170 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001249434 2020-06-12T02:38:12.170 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.170 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.170 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:12.170 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690716+0000", 2020-06-12T02:38:12.171 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58868048699999997 2020-06-12T02:38:12.171 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.171 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:12.171 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.171 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.171 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.172 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2950 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~598016] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:12.172 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.162562+0000", 2020-06-12T02:38:12.172 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.783225483999999, 2020-06-12T02:38:12.173 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52780611399999999, 2020-06-12T02:38:12.173 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:12.173 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:12.173 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:12.173 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:12.174 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:12.174 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2950 2020-06-12T02:38:12.174 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.174 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:12.174 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.175 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:12.175 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.162562+0000", 2020-06-12T02:38:12.175 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:12.175 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.175 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.176 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:12.176 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.162562+0000", 2020-06-12T02:38:12.176 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00070718099999999998 2020-06-12T02:38:12.176 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.176 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.177 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:12.177 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163269+0000", 2020-06-12T02:38:12.177 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2029999999999997e-06 2020-06-12T02:38:12.177 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.177 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.178 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:12.178 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163276+0000", 2020-06-12T02:38:12.178 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2734999999999999e-05 2020-06-12T02:38:12.178 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.178 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.178 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:12.179 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163288+0000", 2020-06-12T02:38:12.179 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.404e-06 2020-06-12T02:38:12.179 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.179 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.179 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:12.180 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163295+0000", 2020-06-12T02:38:12.180 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5784000000000006e-05 2020-06-12T02:38:12.180 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.180 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.180 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:12.180 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163361+0000", 2020-06-12T02:38:12.181 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5318e-05 2020-06-12T02:38:12.181 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.181 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.181 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:12.181 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163436+0000", 2020-06-12T02:38:12.182 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054801100000000005 2020-06-12T02:38:12.182 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.182 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.183 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:12.183 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163984+0000", 2020-06-12T02:38:12.184 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.007023077 2020-06-12T02:38:12.184 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.184 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.184 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:12.184 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171007+0000", 2020-06-12T02:38:12.185 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51932051199999996 2020-06-12T02:38:12.185 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.185 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.185 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:12.185 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690327+0000", 2020-06-12T02:38:12.186 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0889000000000001e-05 2020-06-12T02:38:12.186 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.186 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.186 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:12.186 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690368+0000", 2020-06-12T02:38:12.187 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52780611399999999 2020-06-12T02:38:12.187 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.187 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:12.187 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.187 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.187 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.188 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2951 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~532480] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:12.188 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.165470+0000", 2020-06-12T02:38:12.188 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.780317875999998, 2020-06-12T02:38:12.188 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52507917900000001, 2020-06-12T02:38:12.188 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:12.189 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:12.189 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:12.189 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:12.189 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:12.189 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2951 2020-06-12T02:38:12.190 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.190 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:12.190 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.190 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:12.190 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.165470+0000", 2020-06-12T02:38:12.191 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:12.191 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.191 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.191 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:12.191 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.165470+0000", 2020-06-12T02:38:12.191 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00062231600000000004 2020-06-12T02:38:12.192 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.192 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.192 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:12.192 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166092+0000", 2020-06-12T02:38:12.192 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1610000000000002e-06 2020-06-12T02:38:12.192 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.193 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.193 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:12.193 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166097+0000", 2020-06-12T02:38:12.194 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7780000000000002e-06 2020-06-12T02:38:12.194 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.194 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.194 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:12.194 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166107+0000", 2020-06-12T02:38:12.195 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7169999999999999e-06 2020-06-12T02:38:12.195 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.195 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.195 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:12.195 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166112+0000", 2020-06-12T02:38:12.196 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4792000000000002e-05 2020-06-12T02:38:12.196 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.196 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.196 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:12.196 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166147+0000", 2020-06-12T02:38:12.197 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9336e-05 2020-06-12T02:38:12.197 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.197 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.197 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:12.197 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166236+0000", 2020-06-12T02:38:12.198 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00067747200000000004 2020-06-12T02:38:12.198 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.198 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.199 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:12.200 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166913+0000", 2020-06-12T02:38:12.200 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0047718669999999999 2020-06-12T02:38:12.201 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.201 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.201 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:12.201 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171685+0000", 2020-06-12T02:38:12.201 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51881990700000002 2020-06-12T02:38:12.202 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.202 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.202 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:12.202 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690505+0000", 2020-06-12T02:38:12.202 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3832999999999999e-05 2020-06-12T02:38:12.203 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.203 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.203 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:12.203 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690549+0000", 2020-06-12T02:38:12.204 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52507917900000001 2020-06-12T02:38:12.204 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.204 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:12.204 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.205 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.205 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.205 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2952 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~270336] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:12.205 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.166891+0000", 2020-06-12T02:38:12.205 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.778896226999997, 2020-06-12T02:38:12.206 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52426417000000003, 2020-06-12T02:38:12.206 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:12.206 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:12.206 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:12.206 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:12.207 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:12.207 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2952 2020-06-12T02:38:12.207 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.207 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:12.207 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.207 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:12.208 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166891+0000", 2020-06-12T02:38:12.208 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:12.208 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.208 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.208 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:12.209 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166891+0000", 2020-06-12T02:38:12.209 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00031878599999999999 2020-06-12T02:38:12.209 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.209 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.209 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:12.209 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167210+0000", 2020-06-12T02:38:12.210 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2360000000000004e-06 2020-06-12T02:38:12.210 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.210 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.210 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:12.210 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167219+0000", 2020-06-12T02:38:12.211 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3298e-05 2020-06-12T02:38:12.211 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.211 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.211 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:12.211 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167232+0000", 2020-06-12T02:38:12.212 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5849999999999999e-06 2020-06-12T02:38:12.212 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.214 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.214 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:12.215 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167237+0000", 2020-06-12T02:38:12.215 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0018755740000000001 2020-06-12T02:38:12.215 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.216 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.216 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:12.216 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.169113+0000", 2020-06-12T02:38:12.216 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000104768 2020-06-12T02:38:12.216 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.217 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.217 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:12.217 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.169218+0000", 2020-06-12T02:38:12.217 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00057389799999999999 2020-06-12T02:38:12.217 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.218 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.218 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:12.218 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.169792+0000", 2020-06-12T02:38:12.219 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0019333989999999999 2020-06-12T02:38:12.219 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.219 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.220 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:12.220 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171725+0000", 2020-06-12T02:38:12.220 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51938508699999997 2020-06-12T02:38:12.221 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.221 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.221 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:12.221 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691110+0000", 2020-06-12T02:38:12.222 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5538999999999999e-05 2020-06-12T02:38:12.222 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.222 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.222 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:12.223 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691156+0000", 2020-06-12T02:38:12.223 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52426417000000003 2020-06-12T02:38:12.223 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.224 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:12.224 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.224 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.224 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.225 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2953 3.5s0 3:a446366d:::smithi10729407-11:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:12.225 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.167293+0000", 2020-06-12T02:38:12.225 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.778494780000003, 2020-06-12T02:38:12.226 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52428350999999995, 2020-06-12T02:38:12.226 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:12.226 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:12.227 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:12.227 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:12.227 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:12.228 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2953 2020-06-12T02:38:12.228 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.228 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:12.229 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.229 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:12.229 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167293+0000", 2020-06-12T02:38:12.230 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:12.230 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.230 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.231 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:12.231 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167293+0000", 2020-06-12T02:38:12.231 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3880000000000004e-05 2020-06-12T02:38:12.232 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.232 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.232 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:12.232 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167367+0000", 2020-06-12T02:38:12.233 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.319e-06 2020-06-12T02:38:12.233 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.233 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.234 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:12.234 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167373+0000", 2020-06-12T02:38:12.234 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1719000000000001e-05 2020-06-12T02:38:12.235 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.235 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.235 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:12.236 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167385+0000", 2020-06-12T02:38:12.236 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4580000000000003e-06 2020-06-12T02:38:12.236 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.237 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.237 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:12.237 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167389+0000", 2020-06-12T02:38:12.238 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036341110000000002 2020-06-12T02:38:12.238 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.238 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.238 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:12.239 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171023+0000", 2020-06-12T02:38:12.239 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9418000000000005e-05 2020-06-12T02:38:12.239 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.240 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.240 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:12.240 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171093+0000", 2020-06-12T02:38:12.241 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00029087900000000002 2020-06-12T02:38:12.241 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.241 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.242 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:12.242 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171384+0000", 2020-06-12T02:38:12.242 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000489347 2020-06-12T02:38:12.243 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.243 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.243 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:12.243 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171873+0000", 2020-06-12T02:38:12.244 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51963691300000003 2020-06-12T02:38:12.244 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.244 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.245 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:12.245 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691510+0000", 2020-06-12T02:38:12.245 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6465999999999997e-05 2020-06-12T02:38:12.246 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.246 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.246 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:12.247 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691576+0000", 2020-06-12T02:38:12.247 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52428350999999995 2020-06-12T02:38:12.247 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.247 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:12.247 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.248 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.248 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.248 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2954 3.5s0 3:a446366d:::smithi10729407-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:38:12.248 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.167432+0000", 2020-06-12T02:38:12.248 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.778356031000001, 2020-06-12T02:38:12.249 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52568652699999996, 2020-06-12T02:38:12.249 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:12.249 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:12.249 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:12.249 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:12.250 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:12.250 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2954 2020-06-12T02:38:12.250 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.250 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:12.250 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.251 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:12.251 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167432+0000", 2020-06-12T02:38:12.251 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:12.251 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.251 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.251 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:12.252 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167432+0000", 2020-06-12T02:38:12.253 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3556000000000001e-05 2020-06-12T02:38:12.253 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.253 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.253 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:12.253 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167475+0000", 2020-06-12T02:38:12.254 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7350000000000001e-06 2020-06-12T02:38:12.254 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.254 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.254 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:12.254 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167481+0000", 2020-06-12T02:38:12.255 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3128e-05 2020-06-12T02:38:12.255 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.255 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.256 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:12.256 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167494+0000", 2020-06-12T02:38:12.256 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1579999999999998e-06 2020-06-12T02:38:12.256 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.256 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.257 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:12.257 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167498+0000", 2020-06-12T02:38:12.257 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004082174 2020-06-12T02:38:12.257 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.257 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.258 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:12.258 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171580+0000", 2020-06-12T02:38:12.258 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8954000000000002e-05 2020-06-12T02:38:12.258 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.258 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.259 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:12.259 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171679+0000", 2020-06-12T02:38:12.259 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51993235100000001 2020-06-12T02:38:12.259 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.259 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.259 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:12.260 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691612+0000", 2020-06-12T02:38:12.260 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4329000000000001e-05 2020-06-12T02:38:12.260 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.260 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.260 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:12.261 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691666+0000", 2020-06-12T02:38:12.261 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001452142 2020-06-12T02:38:12.261 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.261 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.261 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:12.262 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.693118+0000", 2020-06-12T02:38:12.262 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52568652699999996 2020-06-12T02:38:12.262 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.263 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:12.263 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.264 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.264 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.264 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2959 3.ds1 3:b3445631:::smithi10729407-6:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e27)", 2020-06-12T02:38:12.264 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.176248+0000", 2020-06-12T02:38:12.265 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.769539289999997, 2020-06-12T02:38:12.265 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51549877099999997, 2020-06-12T02:38:12.265 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:12.266 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:12.266 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:12.266 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:12.266 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:12.267 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2959 2020-06-12T02:38:12.267 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.267 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:12.267 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.267 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:12.268 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176248+0000", 2020-06-12T02:38:12.268 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:12.268 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.268 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.268 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:12.268 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176248+0000", 2020-06-12T02:38:12.269 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5120999999999999e-05 2020-06-12T02:38:12.269 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.269 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.269 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:12.269 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176274+0000", 2020-06-12T02:38:12.270 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9189999999999998e-06 2020-06-12T02:38:12.270 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.270 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.270 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:12.270 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176278+0000", 2020-06-12T02:38:12.270 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6339e-05 2020-06-12T02:38:12.271 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.271 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.271 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:12.271 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176305+0000", 2020-06-12T02:38:12.271 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9089999999999999e-06 2020-06-12T02:38:12.272 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.272 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.272 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:12.272 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176310+0000", 2020-06-12T02:38:12.273 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6095999999999999e-05 2020-06-12T02:38:12.273 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.273 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.273 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:12.273 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176376+0000", 2020-06-12T02:38:12.273 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8590999999999994e-05 2020-06-12T02:38:12.274 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.274 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.274 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:12.274 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176444+0000", 2020-06-12T02:38:12.274 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00024346400000000001 2020-06-12T02:38:12.275 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.275 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.275 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:12.275 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176688+0000", 2020-06-12T02:38:12.275 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00044220899999999999 2020-06-12T02:38:12.276 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.276 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.277 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:12.277 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.177130+0000", 2020-06-12T02:38:12.277 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51452984400000001 2020-06-12T02:38:12.277 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.277 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.278 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:12.278 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691660+0000", 2020-06-12T02:38:12.278 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7279000000000002e-05 2020-06-12T02:38:12.278 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.278 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.278 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:12.279 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691747+0000", 2020-06-12T02:38:12.279 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51549877099999997 2020-06-12T02:38:12.279 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.279 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:12.279 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.280 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.280 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.280 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:3032 3.ds1 3:b983fd28:::smithi10729407-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:38:12.280 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.864289+0000", 2020-06-12T02:38:12.280 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.081498371999999, 2020-06-12T02:38:12.280 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019572216999999999, 2020-06-12T02:38:12.281 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:12.281 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:12.281 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:12.281 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:12.281 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:12.282 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 3032 2020-06-12T02:38:12.282 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.282 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:12.282 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.282 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:12.283 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864289+0000", 2020-06-12T02:38:12.283 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:12.283 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.283 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.283 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:12.284 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864289+0000", 2020-06-12T02:38:12.284 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5701999999999997e-05 2020-06-12T02:38:12.284 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.285 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.285 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:12.285 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864335+0000", 2020-06-12T02:38:12.285 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0910000000000004e-06 2020-06-12T02:38:12.285 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.286 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.286 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:12.286 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864342+0000", 2020-06-12T02:38:12.287 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2301e-05 2020-06-12T02:38:12.287 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.287 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.287 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:12.287 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864354+0000", 2020-06-12T02:38:12.287 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5199999999999997e-06 2020-06-12T02:38:12.288 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.288 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.288 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:12.288 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864360+0000", 2020-06-12T02:38:12.288 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00049180999999999997 2020-06-12T02:38:12.289 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.289 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.289 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:12.289 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864852+0000", 2020-06-12T02:38:12.289 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6725000000000003e-05 2020-06-12T02:38:12.290 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.290 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.290 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:12.290 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864908+0000", 2020-06-12T02:38:12.290 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0089687010000000008 2020-06-12T02:38:12.291 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.291 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.291 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:12.291 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.873877+0000", 2020-06-12T02:38:12.292 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3382999999999999e-05 2020-06-12T02:38:12.292 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.292 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.292 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:12.292 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.873941+0000", 2020-06-12T02:38:12.293 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.006408927 2020-06-12T02:38:12.293 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.293 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.293 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:12.293 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.880350+0000", 2020-06-12T02:38:12.294 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3905e-05 2020-06-12T02:38:12.299 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.299 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.300 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:12.300 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.880403+0000", 2020-06-12T02:38:12.300 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0034581519999999999 2020-06-12T02:38:12.300 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.300 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.301 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:12.301 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.883862+0000", 2020-06-12T02:38:12.301 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019572216999999999 2020-06-12T02:38:12.302 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.302 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:12.302 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.302 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.303 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:12.303 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:12.318 INFO:tasks.rados.rados.0.smithi107.stdout:4145: writing smithi10729407-13 from 2686976 to 3235840 tid 2 2020-06-12T02:38:12.319 INFO:tasks.rados.rados.0.smithi107.stdout:4145: writing smithi10729407-13 from 3235840 to 3727360 tid 3 2020-06-12T02:38:12.319 INFO:tasks.rados.rados.0.smithi107.stdout:4145: writing smithi10729407-13 from 3727360 to 4292608 tid 4 2020-06-12T02:38:12.319 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:38:12.319 INFO:tasks.rados.rados.0.smithi107.stdout:4146: seq_num 4145 ranges {0=630784,630784=434176,1064960=647168,1712128=524288} 2020-06-12T02:38:12.320 INFO:tasks.rados.rados.0.smithi107.stdout:4146: writing smithi10729407-11 from 0 to 630784 tid 1 2020-06-12T02:38:12.320 INFO:tasks.rados.rados.0.smithi107.stdout:4146: writing smithi10729407-11 from 630784 to 1064960 tid 2 2020-06-12T02:38:12.320 INFO:tasks.rados.rados.0.smithi107.stdout:4146: writing smithi10729407-11 from 1064960 to 1712128 tid 3 2020-06-12T02:38:12.320 INFO:tasks.rados.rados.0.smithi107.stdout:4146: writing smithi10729407-11 from 1712128 to 2236416 tid 4 2020-06-12T02:38:12.320 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:38:12.321 INFO:tasks.rados.rados.0.smithi107.stdout:4147: seq_num 4146 ranges {0=647168,647168=720896,1368064=450560,1818624=434176} 2020-06-12T02:38:12.321 INFO:tasks.rados.rados.0.smithi107.stdout:4147: writing smithi10729407-16 from 0 to 647168 tid 1 2020-06-12T02:38:12.321 INFO:tasks.rados.rados.0.smithi107.stdout:4147: writing smithi10729407-16 from 647168 to 1368064 tid 2 2020-06-12T02:38:12.321 INFO:tasks.rados.rados.0.smithi107.stdout:4147: writing smithi10729407-16 from 1368064 to 1818624 tid 3 2020-06-12T02:38:12.321 INFO:tasks.rados.rados.0.smithi107.stdout:4147: writing smithi10729407-16 from 1818624 to 2252800 tid 4 2020-06-12T02:38:12.322 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:12.322 INFO:tasks.rados.rados.0.smithi107.stdout:4148: seq_num 4147 ranges {0=663552,663552=606208,1269760=663552,1933312=335872} 2020-06-12T02:38:12.322 INFO:tasks.rados.rados.0.smithi107.stdout:4148: writing smithi10729407-5 from 0 to 663552 tid 1 2020-06-12T02:38:12.322 INFO:tasks.rados.rados.0.smithi107.stdout:4148: writing smithi10729407-5 from 663552 to 1269760 tid 2 2020-06-12T02:38:12.322 INFO:tasks.rados.rados.0.smithi107.stdout:4148: writing smithi10729407-5 from 1269760 to 1933312 tid 3 2020-06-12T02:38:12.322 INFO:tasks.rados.rados.0.smithi107.stdout:4148: writing smithi10729407-5 from 1933312 to 2269184 tid 4 2020-06-12T02:38:12.323 INFO:tasks.rados.rados.0.smithi107.stdout:4149: delete oid 2 current snap is 0 2020-06-12T02:38:12.323 INFO:tasks.rados.rados.0.smithi107.stdout:4141: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:12.323 INFO:tasks.rados.rados.0.smithi107.stdout:4141: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:12.323 INFO:tasks.rados.rados.0.smithi107.stdout:4141: finishing write tid 7 to smithi10729407-12 2020-06-12T02:38:12.323 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 703 (ObjNum 4140 snap 0 seq_num 4140) dirty exists 2020-06-12T02:38:12.324 INFO:tasks.rados.rados.0.smithi107.stdout:4141: left oid 12 (ObjNum 4140 snap 0 seq_num 4140) 2020-06-12T02:38:12.324 INFO:tasks.rados.rados.0.smithi107.stdout:4145: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:12.324 INFO:tasks.rados.rados.0.smithi107.stdout:4145: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:12.324 INFO:tasks.rados.rados.0.smithi107.stdout:4145: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:12.324 INFO:tasks.rados.rados.0.smithi107.stdout:4145: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:12.325 INFO:tasks.rados.rados.0.smithi107.stdout:4145: finishing write tid 6 to smithi10729407-13 2020-06-12T02:38:12.325 INFO:tasks.rados.rados.0.smithi107.stdout:4145: finishing write tid 7 to smithi10729407-13 2020-06-12T02:38:12.325 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 786 (ObjNum 4144 snap 0 seq_num 4144) dirty exists 2020-06-12T02:38:12.325 INFO:tasks.rados.rados.0.smithi107.stdout:4145: left oid 13 (ObjNum 4144 snap 0 seq_num 4144) 2020-06-12T02:38:12.325 INFO:tasks.rados.rados.0.smithi107.stdout:4146: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:12.326 INFO:tasks.rados.rados.0.smithi107.stdout:4146: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:12.326 INFO:tasks.rados.rados.0.smithi107.stdout:4146: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:12.326 INFO:tasks.rados.rados.0.smithi107.stdout:4146: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:12.326 INFO:tasks.rados.rados.0.smithi107.stdout:4146: finishing write tid 6 to smithi10729407-11 2020-06-12T02:38:12.326 INFO:tasks.rados.rados.0.smithi107.stdout:4147: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:12.327 INFO:tasks.rados.rados.0.smithi107.stdout:4146: finishing write tid 7 to smithi10729407-11 2020-06-12T02:38:12.327 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 733 (ObjNum 4145 snap 0 seq_num 4145) dirty exists 2020-06-12T02:38:12.327 INFO:tasks.rados.rados.0.smithi107.stdout:4146: left oid 11 (ObjNum 4145 snap 0 seq_num 4145) 2020-06-12T02:38:12.327 INFO:tasks.rados.rados.0.smithi107.stdout:4147: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:12.327 INFO:tasks.rados.rados.0.smithi107.stdout:4147: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:12.328 INFO:tasks.rados.rados.0.smithi107.stdout:4147: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:12.328 INFO:tasks.rados.rados.0.smithi107.stdout:4147: finishing write tid 6 to smithi10729407-16 2020-06-12T02:38:12.328 INFO:tasks.rados.rados.0.smithi107.stdout:4147: finishing write tid 7 to smithi10729407-16 2020-06-12T02:38:12.328 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 748 (ObjNum 4146 snap 0 seq_num 4146) dirty exists 2020-06-12T02:38:12.328 INFO:tasks.rados.rados.0.smithi107.stdout:4147: left oid 16 (ObjNum 4146 snap 0 seq_num 4146) 2020-06-12T02:38:12.329 INFO:tasks.rados.rados.0.smithi107.stdout:4141: done (5 left) 2020-06-12T02:38:12.329 INFO:tasks.rados.rados.0.smithi107.stdout:4145: done (4 left) 2020-06-12T02:38:12.329 INFO:tasks.rados.rados.0.smithi107.stdout:4146: done (3 left) 2020-06-12T02:38:12.329 INFO:tasks.rados.rados.0.smithi107.stdout:4147: done (2 left) 2020-06-12T02:38:12.329 INFO:tasks.rados.rados.0.smithi107.stdout:4149: done (1 left) 2020-06-12T02:38:12.329 INFO:tasks.rados.rados.0.smithi107.stdout:4150: delete oid 2 current snap is 0 2020-06-12T02:38:12.330 INFO:tasks.rados.rados.0.smithi107.stdout:4148: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:12.330 INFO:tasks.rados.rados.0.smithi107.stdout:4148: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:12.330 INFO:tasks.rados.rados.0.smithi107.stdout:4148: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:12.330 INFO:tasks.rados.rados.0.smithi107.stdout:4150: done (1 left) 2020-06-12T02:38:12.330 INFO:tasks.rados.rados.0.smithi107.stdout:4151: delete oid 10 current snap is 0 2020-06-12T02:38:12.331 INFO:tasks.rados.rados.0.smithi107.stdout:4148: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:12.331 INFO:tasks.rados.rados.0.smithi107.stdout:4148: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:12.331 INFO:tasks.rados.rados.0.smithi107.stdout:4148: finishing write tid 7 to smithi10729407-5 2020-06-12T02:38:12.331 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 412 (ObjNum 4147 snap 0 seq_num 4147) dirty exists 2020-06-12T02:38:12.331 INFO:tasks.rados.rados.0.smithi107.stdout:4148: left oid 5 (ObjNum 4147 snap 0 seq_num 4147) 2020-06-12T02:38:12.332 INFO:tasks.rados.rados.0.smithi107.stdout:4148: done (1 left) 2020-06-12T02:38:12.332 INFO:tasks.rados.rados.0.smithi107.stdout:4151: done (0 left) 2020-06-12T02:38:12.332 INFO:tasks.rados.rados.0.smithi107.stdout:4152: delete oid 11 current snap is 0 2020-06-12T02:38:12.332 INFO:tasks.rados.rados.0.smithi107.stdout:4152: done (0 left) 2020-06-12T02:38:12.333 INFO:tasks.rados.rados.0.smithi107.stdout:4153: delete oid 9 current snap is 0 2020-06-12T02:38:12.333 INFO:tasks.rados.rados.0.smithi107.stdout:4153: done (0 left) 2020-06-12T02:38:12.333 INFO:tasks.rados.rados.0.smithi107.stdout:4154: delete oid 1 current snap is 0 2020-06-12T02:38:12.333 INFO:tasks.rados.rados.0.smithi107.stdout:4154: done (0 left) 2020-06-12T02:38:12.333 INFO:tasks.rados.rados.0.smithi107.stdout:4155: delete oid 9 current snap is 0 2020-06-12T02:38:12.334 INFO:tasks.rados.rados.0.smithi107.stdout:4155: done (0 left) 2020-06-12T02:38:12.334 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:38:12.334 INFO:tasks.rados.rados.0.smithi107.stdout:4156: seq_num 4155 ranges {1998848=794624,2793472=532480,3325952=622592,3948544=450560} 2020-06-12T02:38:12.334 INFO:tasks.rados.rados.0.smithi107.stdout:4156: writing smithi10729407-7 from 1998848 to 2793472 tid 1 2020-06-12T02:38:12.334 INFO:tasks.rados.rados.0.smithi107.stdout:4156: writing smithi10729407-7 from 2793472 to 3325952 tid 2 2020-06-12T02:38:12.335 INFO:tasks.rados.rados.0.smithi107.stdout:4156: writing smithi10729407-7 from 3325952 to 3948544 tid 3 2020-06-12T02:38:12.335 INFO:tasks.rados.rados.0.smithi107.stdout:4156: writing smithi10729407-7 from 3948544 to 4399104 tid 4 2020-06-12T02:38:12.335 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:38:12.335 INFO:tasks.rados.rados.0.smithi107.stdout:4157: seq_num 4156 ranges {0=8192} 2020-06-12T02:38:12.335 INFO:tasks.rados.rados.0.smithi107.stdout:4157: writing smithi10729407-20 from 0 to 8192 tid 1 2020-06-12T02:38:12.335 INFO:tasks.rados.rados.0.smithi107.stdout:4158: delete oid 5 current snap is 0 2020-06-12T02:38:12.336 INFO:tasks.rados.rados.0.smithi107.stdout:4156: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:12.336 INFO:tasks.rados.rados.0.smithi107.stdout:4157: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:12.336 INFO:tasks.rados.rados.0.smithi107.stdout:4158: done (2 left) 2020-06-12T02:38:12.336 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:12.336 INFO:tasks.rados.rados.0.smithi107.stdout:4159: seq_num 4158 ranges {4620288=40960} 2020-06-12T02:38:12.337 INFO:tasks.rados.rados.0.smithi107.stdout:4159: writing smithi10729407-8 from 4620288 to 4661248 tid 1 2020-06-12T02:38:12.337 INFO:tasks.rados.rados.0.smithi107.stdout:4160: delete oid 16 current snap is 0 2020-06-12T02:38:12.337 INFO:tasks.rados.rados.0.smithi107.stdout:4157: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:12.338 INFO:tasks.rados.rados.0.smithi107.stdout:4157: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:12.338 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 317 (ObjNum 4156 snap 0 seq_num 4156) dirty exists 2020-06-12T02:38:12.338 INFO:tasks.rados.rados.0.smithi107.stdout:4157: left oid 20 (ObjNum 4156 snap 0 seq_num 4156) 2020-06-12T02:38:12.339 INFO:tasks.rados.rados.0.smithi107.stdout:4156: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:12.339 INFO:tasks.rados.rados.0.smithi107.stdout:4156: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:12.339 INFO:tasks.rados.rados.0.smithi107.stdout:4156: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:12.339 INFO:tasks.rados.rados.0.smithi107.stdout:4156: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:12.339 INFO:tasks.rados.rados.0.smithi107.stdout:4156: finishing write tid 7 to smithi10729407-7 2020-06-12T02:38:12.340 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 791 (ObjNum 4155 snap 0 seq_num 4155) dirty exists 2020-06-12T02:38:12.340 INFO:tasks.rados.rados.0.smithi107.stdout:4156: left oid 7 (ObjNum 4155 snap 0 seq_num 4155) 2020-06-12T02:38:12.340 INFO:tasks.rados.rados.0.smithi107.stdout:4156: done (3 left) 2020-06-12T02:38:12.340 INFO:tasks.rados.rados.0.smithi107.stdout:4157: done (2 left) 2020-06-12T02:38:12.340 INFO:tasks.rados.rados.0.smithi107.stdout:4160: done (1 left) 2020-06-12T02:38:12.341 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:38:12.341 INFO:tasks.rados.rados.0.smithi107.stdout:4161: seq_num 4160 ranges {0=73728} 2020-06-12T02:38:12.341 INFO:tasks.rados.rados.0.smithi107.stdout:4161: writing smithi10729407-10 from 0 to 73728 tid 1 2020-06-12T02:38:12.341 INFO:tasks.rados.rados.0.smithi107.stdout:4162: delete oid 1 current snap is 0 2020-06-12T02:38:12.341 INFO:tasks.rados.rados.0.smithi107.stdout:4159: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:12.342 INFO:tasks.rados.rados.0.smithi107.stdout:4159: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:12.342 INFO:tasks.rados.rados.0.smithi107.stdout:4159: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:12.342 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 726 (ObjNum 4158 snap 0 seq_num 4158) dirty exists 2020-06-12T02:38:12.342 INFO:tasks.rados.rados.0.smithi107.stdout:4159: left oid 8 (ObjNum 4158 snap 0 seq_num 4158) 2020-06-12T02:38:12.342 INFO:tasks.rados.rados.0.smithi107.stdout:4159: done (2 left) 2020-06-12T02:38:12.342 INFO:tasks.rados.rados.0.smithi107.stdout:4162: done (1 left) 2020-06-12T02:38:12.343 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:12.343 INFO:tasks.rados.rados.0.smithi107.stdout:4163: seq_num 4162 ranges {4292608=106496} 2020-06-12T02:38:12.343 INFO:tasks.rados.rados.0.smithi107.stdout:4163: writing smithi10729407-13 from 4292608 to 4399104 tid 1 2020-06-12T02:38:12.343 INFO:tasks.rados.rados.0.smithi107.stdout:4164: delete oid 17 current snap is 0 2020-06-12T02:38:12.343 INFO:tasks.rados.rados.0.smithi107.stdout:4161: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:12.344 INFO:tasks.rados.rados.0.smithi107.stdout:4161: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:12.344 INFO:tasks.rados.rados.0.smithi107.stdout:4161: finishing write tid 4 to smithi10729407-10 2020-06-12T02:38:12.344 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 415 (ObjNum 4160 snap 0 seq_num 4160) dirty exists 2020-06-12T02:38:12.344 INFO:tasks.rados.rados.0.smithi107.stdout:4161: left oid 10 (ObjNum 4160 snap 0 seq_num 4160) 2020-06-12T02:38:12.344 INFO:tasks.rados.rados.0.smithi107.stdout:4161: done (2 left) 2020-06-12T02:38:12.344 INFO:tasks.rados.rados.0.smithi107.stdout:4164: done (1 left) 2020-06-12T02:38:12.345 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:12.345 INFO:tasks.rados.rados.0.smithi107.stdout:4165: seq_num 4164 ranges {0=147456} 2020-06-12T02:38:12.345 INFO:tasks.rados.rados.0.smithi107.stdout:4165: writing smithi10729407-5 from 0 to 147456 tid 1 2020-06-12T02:38:12.345 INFO:tasks.rados.rados.0.smithi107.stdout:4166: delete oid 1 current snap is 0 2020-06-12T02:38:12.345 INFO:tasks.rados.rados.0.smithi107.stdout:4163: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:12.346 INFO:tasks.rados.rados.0.smithi107.stdout:4166: done (2 left) 2020-06-12T02:38:12.346 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:12.346 INFO:tasks.rados.rados.0.smithi107.stdout:4167: seq_num 4166 ranges {0=180224} 2020-06-12T02:38:12.346 INFO:tasks.rados.rados.0.smithi107.stdout:4167: writing smithi10729407-14 from 0 to 180224 tid 1 2020-06-12T02:38:12.346 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:12.347 INFO:tasks.rados.rados.0.smithi107.stdout:4168: seq_num 4167 ranges {0=196608} 2020-06-12T02:38:12.347 INFO:tasks.rados.rados.0.smithi107.stdout:4168: writing smithi10729407-19 from 0 to 196608 tid 1 2020-06-12T02:38:12.347 INFO:tasks.rados.rados.0.smithi107.stdout:4169: delete oid 1 current snap is 0 2020-06-12T02:38:12.347 INFO:tasks.rados.rados.0.smithi107.stdout:4163: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:12.347 INFO:tasks.rados.rados.0.smithi107.stdout:4163: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:12.347 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 793 (ObjNum 4162 snap 0 seq_num 4162) dirty exists 2020-06-12T02:38:12.348 INFO:tasks.rados.rados.0.smithi107.stdout:4163: left oid 13 (ObjNum 4162 snap 0 seq_num 4162) 2020-06-12T02:38:12.348 INFO:tasks.rados.rados.0.smithi107.stdout:4165: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:12.348 INFO:tasks.rados.rados.0.smithi107.stdout:4165: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:12.348 INFO:tasks.rados.rados.0.smithi107.stdout:4165: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:12.348 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 415 (ObjNum 4164 snap 0 seq_num 4164) dirty exists 2020-06-12T02:38:12.349 INFO:tasks.rados.rados.0.smithi107.stdout:4165: left oid 5 (ObjNum 4164 snap 0 seq_num 4164) 2020-06-12T02:38:12.349 INFO:tasks.rados.rados.0.smithi107.stdout:4163: done (4 left) 2020-06-12T02:38:12.349 INFO:tasks.rados.rados.0.smithi107.stdout:4165: done (3 left) 2020-06-12T02:38:12.349 INFO:tasks.rados.rados.0.smithi107.stdout:4169: done (2 left) 2020-06-12T02:38:12.349 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:12.350 INFO:tasks.rados.rados.0.smithi107.stdout:4170: seq_num 4169 ranges {4661248=229376} 2020-06-12T02:38:12.350 INFO:tasks.rados.rados.0.smithi107.stdout:4170: writing smithi10729407-8 from 4661248 to 4890624 tid 1 2020-06-12T02:38:12.350 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:12.350 INFO:tasks.rados.rados.0.smithi107.stdout:4171: seq_num 4170 ranges {4399104=245760} 2020-06-12T02:38:12.350 INFO:tasks.rados.rados.0.smithi107.stdout:4171: writing smithi10729407-7 from 4399104 to 4644864 tid 1 2020-06-12T02:38:12.351 INFO:tasks.rados.rados.0.smithi107.stdout:4172: delete oid 5 current snap is 0 2020-06-12T02:38:12.351 INFO:tasks.rados.rados.0.smithi107.stdout:4167: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:12.351 INFO:tasks.rados.rados.0.smithi107.stdout:4168: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:12.351 INFO:tasks.rados.rados.0.smithi107.stdout:4167: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:12.351 INFO:tasks.rados.rados.0.smithi107.stdout:4168: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:12.352 INFO:tasks.rados.rados.0.smithi107.stdout:4167: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:12.352 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 736 (ObjNum 4166 snap 0 seq_num 4166) dirty exists 2020-06-12T02:38:12.352 INFO:tasks.rados.rados.0.smithi107.stdout:4167: left oid 14 (ObjNum 4166 snap 0 seq_num 4166) 2020-06-12T02:38:12.352 INFO:tasks.rados.rados.0.smithi107.stdout:4168: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:12.352 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 705 (ObjNum 4167 snap 0 seq_num 4167) dirty exists 2020-06-12T02:38:12.352 INFO:tasks.rados.rados.0.smithi107.stdout:4168: left oid 19 (ObjNum 4167 snap 0 seq_num 4167) 2020-06-12T02:38:12.353 INFO:tasks.rados.rados.0.smithi107.stdout:4170: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:12.353 INFO:tasks.rados.rados.0.smithi107.stdout:4170: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:12.353 INFO:tasks.rados.rados.0.smithi107.stdout:4170: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:12.353 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 728 (ObjNum 4169 snap 0 seq_num 4169) dirty exists 2020-06-12T02:38:12.353 INFO:tasks.rados.rados.0.smithi107.stdout:4170: left oid 8 (ObjNum 4169 snap 0 seq_num 4169) 2020-06-12T02:38:12.354 INFO:tasks.rados.rados.0.smithi107.stdout:4167: done (4 left) 2020-06-12T02:38:12.354 INFO:tasks.rados.rados.0.smithi107.stdout:4168: done (3 left) 2020-06-12T02:38:12.354 INFO:tasks.rados.rados.0.smithi107.stdout:4170: done (2 left) 2020-06-12T02:38:12.354 INFO:tasks.rados.rados.0.smithi107.stdout:4172: done (1 left) 2020-06-12T02:38:12.354 INFO:tasks.rados.rados.0.smithi107.stdout:4173: delete oid 2 current snap is 0 2020-06-12T02:38:12.355 INFO:tasks.rados.rados.0.smithi107.stdout:4171: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:12.355 INFO:tasks.rados.rados.0.smithi107.stdout:4171: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:12.355 INFO:tasks.rados.rados.0.smithi107.stdout:4173: done (1 left) 2020-06-12T02:38:12.355 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:12.355 INFO:tasks.rados.rados.0.smithi107.stdout:4174: seq_num 4173 ranges {0=294912} 2020-06-12T02:38:12.355 INFO:tasks.rados.rados.0.smithi107.stdout:4174: writing smithi10729407-4 from 0 to 294912 tid 1 2020-06-12T02:38:12.356 INFO:tasks.rados.rados.0.smithi107.stdout:4175: delete oid 14 current snap is 0 2020-06-12T02:38:12.356 INFO:tasks.rados.rados.0.smithi107.stdout:4171: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:12.356 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 795 (ObjNum 4170 snap 0 seq_num 4170) dirty exists 2020-06-12T02:38:12.356 INFO:tasks.rados.rados.0.smithi107.stdout:4171: left oid 7 (ObjNum 4170 snap 0 seq_num 4170) 2020-06-12T02:38:12.357 INFO:tasks.rados.rados.0.smithi107.stdout:4171: done (2 left) 2020-06-12T02:38:12.357 INFO:tasks.rados.rados.0.smithi107.stdout:4175: done (1 left) 2020-06-12T02:38:12.357 INFO:tasks.rados.rados.0.smithi107.stdout:4176: delete oid 3 current snap is 0 2020-06-12T02:38:12.357 INFO:tasks.rados.rados.0.smithi107.stdout:4176: done (1 left) 2020-06-12T02:38:12.357 INFO:tasks.rados.rados.0.smithi107.stdout:4177: delete oid 2 current snap is 0 2020-06-12T02:38:12.357 INFO:tasks.rados.rados.0.smithi107.stdout:4177: done (1 left) 2020-06-12T02:38:12.358 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:12.358 INFO:tasks.rados.rados.0.smithi107.stdout:4178: seq_num 4177 ranges {0=360448} 2020-06-12T02:38:12.358 INFO:tasks.rados.rados.0.smithi107.stdout:4178: writing smithi10729407-2 from 0 to 360448 tid 1 2020-06-12T02:38:12.358 INFO:tasks.rados.rados.0.smithi107.stdout:4179: delete oid 5 current snap is 0 2020-06-12T02:38:12.358 INFO:tasks.rados.rados.0.smithi107.stdout:4174: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:12.359 INFO:tasks.rados.rados.0.smithi107.stdout:4174: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:12.359 INFO:tasks.rados.rados.0.smithi107.stdout:4174: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:12.359 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 746 (ObjNum 4173 snap 0 seq_num 4173) dirty exists 2020-06-12T02:38:12.359 INFO:tasks.rados.rados.0.smithi107.stdout:4174: left oid 4 (ObjNum 4173 snap 0 seq_num 4173) 2020-06-12T02:38:12.359 INFO:tasks.rados.rados.0.smithi107.stdout:4174: done (2 left) 2020-06-12T02:38:12.360 INFO:tasks.rados.rados.0.smithi107.stdout:4179: done (1 left) 2020-06-12T02:38:12.360 INFO:tasks.rados.rados.0.smithi107.stdout:4180: delete oid 19 current snap is 0 2020-06-12T02:38:12.360 INFO:tasks.rados.rados.0.smithi107.stdout:4178: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:12.360 INFO:tasks.rados.rados.0.smithi107.stdout:4180: done (1 left) 2020-06-12T02:38:12.360 INFO:tasks.rados.rados.0.smithi107.stdout:4181: delete oid 6 current snap is 0 2020-06-12T02:38:12.360 INFO:tasks.rados.rados.0.smithi107.stdout:4178: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:12.361 INFO:tasks.rados.rados.0.smithi107.stdout:4178: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:12.361 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 437 (ObjNum 4177 snap 0 seq_num 4177) dirty exists 2020-06-12T02:38:12.361 INFO:tasks.rados.rados.0.smithi107.stdout:4178: left oid 2 (ObjNum 4177 snap 0 seq_num 4177) 2020-06-12T02:38:12.361 INFO:tasks.rados.rados.0.smithi107.stdout:4178: done (1 left) 2020-06-12T02:38:12.361 INFO:tasks.rados.rados.0.smithi107.stdout:4181: done (0 left) 2020-06-12T02:38:12.362 INFO:tasks.rados.rados.0.smithi107.stdout:4182: delete oid 8 current snap is 0 2020-06-12T02:38:12.362 INFO:tasks.rados.rados.0.smithi107.stdout:4182: done (0 left) 2020-06-12T02:38:12.362 INFO:tasks.rados.rados.0.smithi107.stdout:4183: delete oid 9 current snap is 0 2020-06-12T02:38:12.362 INFO:tasks.rados.rados.0.smithi107.stdout:4183: done (0 left) 2020-06-12T02:38:12.362 INFO:tasks.rados.rados.0.smithi107.stdout:4184: delete oid 7 current snap is 0 2020-06-12T02:38:12.363 INFO:tasks.rados.rados.0.smithi107.stdout:4184: done (0 left) 2020-06-12T02:38:12.363 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:12.363 INFO:tasks.rados.rados.0.smithi107.stdout:4185: seq_num 4184 ranges {2146304=483328} 2020-06-12T02:38:12.363 INFO:tasks.rados.rados.0.smithi107.stdout:4185: writing smithi10729407-12 from 2146304 to 2629632 tid 1 2020-06-12T02:38:12.364 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:12.364 INFO:tasks.rados.rados.0.smithi107.stdout:4186: seq_num 4185 ranges {0=499712} 2020-06-12T02:38:12.364 INFO:tasks.rados.rados.0.smithi107.stdout:4186: writing smithi10729407-9 from 0 to 499712 tid 1 2020-06-12T02:38:12.364 INFO:tasks.rados.rados.0.smithi107.stdout:4187: delete oid 6 current snap is 0 2020-06-12T02:38:12.365 INFO:tasks.rados.rados.0.smithi107.stdout:4185: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:12.365 INFO:tasks.rados.rados.0.smithi107.stdout:4185: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:12.365 INFO:tasks.rados.rados.0.smithi107.stdout:4185: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:12.365 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 708 (ObjNum 4184 snap 0 seq_num 4184) dirty exists 2020-06-12T02:38:12.365 INFO:tasks.rados.rados.0.smithi107.stdout:4185: left oid 12 (ObjNum 4184 snap 0 seq_num 4184) 2020-06-12T02:38:12.365 INFO:tasks.rados.rados.0.smithi107.stdout:4185: done (2 left) 2020-06-12T02:38:12.366 INFO:tasks.rados.rados.0.smithi107.stdout:4187: done (1 left) 2020-06-12T02:38:12.366 INFO:tasks.rados.rados.0.smithi107.stdout:4188: delete oid 19 current snap is 0 2020-06-12T02:38:12.366 INFO:tasks.rados.rados.0.smithi107.stdout:4186: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:12.366 INFO:tasks.rados.rados.0.smithi107.stdout:4186: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:12.366 INFO:tasks.rados.rados.0.smithi107.stdout:4188: done (1 left) 2020-06-12T02:38:12.367 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:12.367 INFO:tasks.rados.rados.0.smithi107.stdout:4189: seq_num 4188 ranges {2629632=548864} 2020-06-12T02:38:12.367 INFO:tasks.rados.rados.0.smithi107.stdout:4189: writing smithi10729407-12 from 2629632 to 3178496 tid 1 2020-06-12T02:38:12.367 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:38:12.367 INFO:tasks.rados.rados.0.smithi107.stdout:4190: seq_num 4189 ranges {0=565248} 2020-06-12T02:38:12.368 INFO:tasks.rados.rados.0.smithi107.stdout:4190: writing smithi10729407-5 from 0 to 565248 tid 1 2020-06-12T02:38:12.368 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:38:12.368 INFO:tasks.rados.rados.0.smithi107.stdout:4191: seq_num 4190 ranges {0=581632} 2020-06-12T02:38:12.368 INFO:tasks.rados.rados.0.smithi107.stdout:4191: writing smithi10729407-16 from 0 to 581632 tid 1 2020-06-12T02:38:12.368 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:12.369 INFO:tasks.rados.rados.0.smithi107.stdout:4192: seq_num 4191 ranges {0=598016} 2020-06-12T02:38:12.369 INFO:tasks.rados.rados.0.smithi107.stdout:4192: writing smithi10729407-8 from 0 to 598016 tid 1 2020-06-12T02:38:12.369 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:38:12.369 INFO:tasks.rados.rados.0.smithi107.stdout:4193: seq_num 4192 ranges {73728=614400} 2020-06-12T02:38:12.370 INFO:tasks.rados.rados.0.smithi107.stdout:4193: writing smithi10729407-10 from 73728 to 688128 tid 1 2020-06-12T02:38:12.370 INFO:tasks.rados.rados.0.smithi107.stdout:4194: delete oid 11 current snap is 0 2020-06-12T02:38:12.370 INFO:tasks.rados.rados.0.smithi107.stdout:4186: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:12.370 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 738 (ObjNum 4185 snap 0 seq_num 4185) dirty exists 2020-06-12T02:38:12.370 INFO:tasks.rados.rados.0.smithi107.stdout:4186: left oid 9 (ObjNum 4185 snap 0 seq_num 4185) 2020-06-12T02:38:12.371 INFO:tasks.rados.rados.0.smithi107.stdout:4189: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:12.371 INFO:tasks.rados.rados.0.smithi107.stdout:4189: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:12.371 INFO:tasks.rados.rados.0.smithi107.stdout:4189: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:12.371 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 711 (ObjNum 4188 snap 0 seq_num 4188) dirty exists 2020-06-12T02:38:12.371 INFO:tasks.rados.rados.0.smithi107.stdout:4189: left oid 12 (ObjNum 4188 snap 0 seq_num 4188) 2020-06-12T02:38:12.371 INFO:tasks.rados.rados.0.smithi107.stdout:4190: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:12.372 INFO:tasks.rados.rados.0.smithi107.stdout:4190: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:12.372 INFO:tasks.rados.rados.0.smithi107.stdout:4191: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:12.372 INFO:tasks.rados.rados.0.smithi107.stdout:4190: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:12.372 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 419 (ObjNum 4189 snap 0 seq_num 4189) dirty exists 2020-06-12T02:38:12.373 INFO:tasks.rados.rados.0.smithi107.stdout:4190: left oid 5 (ObjNum 4189 snap 0 seq_num 4189) 2020-06-12T02:38:12.373 INFO:tasks.rados.rados.0.smithi107.stdout:4191: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:12.373 INFO:tasks.rados.rados.0.smithi107.stdout:4191: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:12.373 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 752 (ObjNum 4190 snap 0 seq_num 4190) dirty exists 2020-06-12T02:38:12.374 INFO:tasks.rados.rados.0.smithi107.stdout:4191: left oid 16 (ObjNum 4190 snap 0 seq_num 4190) 2020-06-12T02:38:12.374 INFO:tasks.rados.rados.0.smithi107.stdout:4186: done (6 left) 2020-06-12T02:38:12.374 INFO:tasks.rados.rados.0.smithi107.stdout:4189: done (5 left) 2020-06-12T02:38:12.374 INFO:tasks.rados.rados.0.smithi107.stdout:4190: done (4 left) 2020-06-12T02:38:12.374 INFO:tasks.rados.rados.0.smithi107.stdout:4191: done (3 left) 2020-06-12T02:38:12.374 INFO:tasks.rados.rados.0.smithi107.stdout:4194: done (2 left) 2020-06-12T02:38:12.375 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:12.375 INFO:tasks.rados.rados.0.smithi107.stdout:4195: seq_num 4194 ranges {0=647168} 2020-06-12T02:38:12.375 INFO:tasks.rados.rados.0.smithi107.stdout:4195: writing smithi10729407-14 from 0 to 647168 tid 1 2020-06-12T02:38:12.375 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:12.375 INFO:tasks.rados.rados.0.smithi107.stdout:4196: seq_num 4195 ranges {4399104=663552} 2020-06-12T02:38:12.376 INFO:tasks.rados.rados.0.smithi107.stdout:4196: writing smithi10729407-13 from 4399104 to 5062656 tid 1 2020-06-12T02:38:12.376 INFO:tasks.rados.rados.0.smithi107.stdout:4197: delete oid 17 current snap is 0 2020-06-12T02:38:12.376 INFO:tasks.rados.rados.0.smithi107.stdout:4192: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:12.376 INFO:tasks.rados.rados.0.smithi107.stdout:4192: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:12.376 INFO:tasks.rados.rados.0.smithi107.stdout:4193: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:12.377 INFO:tasks.rados.rados.0.smithi107.stdout:4193: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:12.377 INFO:tasks.rados.rados.0.smithi107.stdout:4192: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:12.377 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 731 (ObjNum 4191 snap 0 seq_num 4191) dirty exists 2020-06-12T02:38:12.377 INFO:tasks.rados.rados.0.smithi107.stdout:4192: left oid 8 (ObjNum 4191 snap 0 seq_num 4191) 2020-06-12T02:38:12.377 INFO:tasks.rados.rados.0.smithi107.stdout:4193: finishing write tid 4 to smithi10729407-10 2020-06-12T02:38:12.378 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 417 (ObjNum 4192 snap 0 seq_num 4192) dirty exists 2020-06-12T02:38:12.378 INFO:tasks.rados.rados.0.smithi107.stdout:4193: left oid 10 (ObjNum 4192 snap 0 seq_num 4192) 2020-06-12T02:38:12.378 INFO:tasks.rados.rados.0.smithi107.stdout:4195: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:12.378 INFO:tasks.rados.rados.0.smithi107.stdout:4195: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:12.378 INFO:tasks.rados.rados.0.smithi107.stdout:4195: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:12.379 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 740 (ObjNum 4194 snap 0 seq_num 4194) dirty exists 2020-06-12T02:38:12.379 INFO:tasks.rados.rados.0.smithi107.stdout:4195: left oid 14 (ObjNum 4194 snap 0 seq_num 4194) 2020-06-12T02:38:12.379 INFO:tasks.rados.rados.0.smithi107.stdout:4192: done (4 left) 2020-06-12T02:38:12.379 INFO:tasks.rados.rados.0.smithi107.stdout:4193: done (3 left) 2020-06-12T02:38:12.379 INFO:tasks.rados.rados.0.smithi107.stdout:4195: done (2 left) 2020-06-12T02:38:12.379 INFO:tasks.rados.rados.0.smithi107.stdout:4197: done (1 left) 2020-06-12T02:38:12.380 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:12.380 INFO:tasks.rados.rados.0.smithi107.stdout:4198: seq_num 4197 ranges {499712=696320} 2020-06-12T02:38:12.380 INFO:tasks.rados.rados.0.smithi107.stdout:4198: writing smithi10729407-9 from 499712 to 1196032 tid 1 2020-06-12T02:38:12.380 INFO:tasks.rados.rados.0.smithi107.stdout:4199: delete oid 18 current snap is 0 2020-06-12T02:38:12.380 INFO:tasks.rados.rados.0.smithi107.stdout:4196: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:12.381 INFO:tasks.rados.rados.0.smithi107.stdout:4196: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:12.381 INFO:tasks.rados.rados.0.smithi107.stdout:4196: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:12.381 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 798 (ObjNum 4195 snap 0 seq_num 4195) dirty exists 2020-06-12T02:38:12.381 INFO:tasks.rados.rados.0.smithi107.stdout:4196: left oid 13 (ObjNum 4195 snap 0 seq_num 4195) 2020-06-12T02:38:12.382 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-06-12T02:38:12.383 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:12.302+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:12.387 INFO:tasks.rados.rados.0.smithi107.stdout:4198: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:12.387 INFO:tasks.rados.rados.0.smithi107.stdout:4198: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:12.388 INFO:tasks.rados.rados.0.smithi107.stdout:4196: done (2 left) 2020-06-12T02:38:12.388 INFO:tasks.rados.rados.0.smithi107.stdout:4199: done (1 left) 2020-06-12T02:38:12.388 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:12.388 INFO:tasks.rados.rados.0.smithi107.stdout:4200: seq_num 4199 ranges {0=729088} 2020-06-12T02:38:12.388 INFO:tasks.rados.rados.0.smithi107.stdout:4200: writing smithi10729407-17 from 0 to 729088 tid 1 2020-06-12T02:38:12.388 INFO:tasks.rados.rados.0.smithi107.stdout:4201: delete oid 3 current snap is 0 2020-06-12T02:38:12.389 INFO:tasks.rados.rados.0.smithi107.stdout:4198: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:12.389 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 740 (ObjNum 4197 snap 0 seq_num 4197) dirty exists 2020-06-12T02:38:12.389 INFO:tasks.rados.rados.0.smithi107.stdout:4198: left oid 9 (ObjNum 4197 snap 0 seq_num 4197) 2020-06-12T02:38:12.389 INFO:tasks.rados.rados.0.smithi107.stdout:4198: done (2 left) 2020-06-12T02:38:12.389 INFO:tasks.rados.rados.0.smithi107.stdout:4201: done (1 left) 2020-06-12T02:38:12.390 INFO:tasks.rados.rados.0.smithi107.stdout:4202: delete oid 16 current snap is 0 2020-06-12T02:38:12.390 INFO:tasks.rados.rados.0.smithi107.stdout:4202: done (1 left) 2020-06-12T02:38:12.390 INFO:tasks.rados.rados.0.smithi107.stdout:4203: delete oid 18 current snap is 0 2020-06-12T02:38:12.391 INFO:tasks.rados.rados.0.smithi107.stdout:4200: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:12.391 INFO:tasks.rados.rados.0.smithi107.stdout:4200: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:12.391 INFO:tasks.rados.rados.0.smithi107.stdout:4200: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:12.391 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 749 (ObjNum 4199 snap 0 seq_num 4199) dirty exists 2020-06-12T02:38:12.392 INFO:tasks.rados.rados.0.smithi107.stdout:4200: left oid 17 (ObjNum 4199 snap 0 seq_num 4199) 2020-06-12T02:38:12.392 INFO:tasks.rados.rados.0.smithi107.stdout:4200: done (1 left) 2020-06-12T02:38:12.392 INFO:tasks.rados.rados.0.smithi107.stdout:4203: done (0 left) 2020-06-12T02:38:12.392 INFO:tasks.rados.rados.0.smithi107.stdout:4204: delete oid 6 current snap is 0 2020-06-12T02:38:12.393 INFO:tasks.rados.rados.0.smithi107.stdout:4204: done (0 left) 2020-06-12T02:38:12.393 INFO:tasks.rados.rados.0.smithi107.stdout:4205: delete oid 5 current snap is 0 2020-06-12T02:38:12.393 INFO:tasks.rados.rados.0.smithi107.stdout:4205: done (0 left) 2020-06-12T02:38:12.393 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:38:12.393 INFO:tasks.rados.rados.0.smithi107.stdout:4206: seq_num 4205 ranges {688128=434176,1122304=401408} 2020-06-12T02:38:12.394 INFO:tasks.rados.rados.0.smithi107.stdout:4206: writing smithi10729407-10 from 688128 to 1122304 tid 1 2020-06-12T02:38:12.394 INFO:tasks.rados.rados.0.smithi107.stdout:4206: writing smithi10729407-10 from 1122304 to 1523712 tid 2 2020-06-12T02:38:12.394 INFO:tasks.rados.rados.0.smithi107.stdout:4207: delete oid 13 current snap is 0 2020-06-12T02:38:12.394 INFO:tasks.rados.rados.0.smithi107.stdout:4207: done (1 left) 2020-06-12T02:38:12.394 INFO:tasks.rados.rados.0.smithi107.stdout:4208: delete oid 12 current snap is 0 2020-06-12T02:38:12.395 INFO:tasks.rados.rados.0.smithi107.stdout:4206: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:12.395 INFO:tasks.rados.rados.0.smithi107.stdout:4206: finishing write tid 2 to smithi10729407-10 2020-06-12T02:38:12.395 INFO:tasks.rados.rados.0.smithi107.stdout:4208: done (1 left) 2020-06-12T02:38:12.395 INFO:tasks.rados.rados.0.smithi107.stdout:4209: delete oid 8 current snap is 0 2020-06-12T02:38:12.395 INFO:tasks.rados.rados.0.smithi107.stdout:4206: finishing write tid 4 to smithi10729407-10 2020-06-12T02:38:12.396 INFO:tasks.rados.rados.0.smithi107.stdout:4206: finishing write tid 5 to smithi10729407-10 2020-06-12T02:38:12.396 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 420 (ObjNum 4205 snap 0 seq_num 4205) dirty exists 2020-06-12T02:38:12.396 INFO:tasks.rados.rados.0.smithi107.stdout:4206: left oid 10 (ObjNum 4205 snap 0 seq_num 4205) 2020-06-12T02:38:12.396 INFO:tasks.rados.rados.0.smithi107.stdout:4206: done (1 left) 2020-06-12T02:38:12.396 INFO:tasks.rados.rados.0.smithi107.stdout:4209: done (0 left) 2020-06-12T02:38:12.397 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:38:12.397 INFO:tasks.rados.rados.0.smithi107.stdout:4210: seq_num 4209 ranges {0=499712,499712=401408} 2020-06-12T02:38:12.397 INFO:tasks.rados.rados.0.smithi107.stdout:4210: writing smithi10729407-16 from 0 to 499712 tid 1 2020-06-12T02:38:12.397 INFO:tasks.rados.rados.0.smithi107.stdout:4210: writing smithi10729407-16 from 499712 to 901120 tid 2 2020-06-12T02:38:12.397 INFO:tasks.rados.rados.0.smithi107.stdout:4211: delete oid 18 current snap is 0 2020-06-12T02:38:12.397 INFO:tasks.rados.rados.0.smithi107.stdout:4211: done (1 left) 2020-06-12T02:38:12.398 INFO:tasks.rados.rados.0.smithi107.stdout:4212: delete oid 6 current snap is 0 2020-06-12T02:38:12.398 INFO:tasks.rados.rados.0.smithi107.stdout:4210: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:12.398 INFO:tasks.rados.rados.0.smithi107.stdout:4210: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:12.398 INFO:tasks.rados.rados.0.smithi107.stdout:4212: done (1 left) 2020-06-12T02:38:12.398 INFO:tasks.rados.rados.0.smithi107.stdout:4213: delete oid 13 current snap is 0 2020-06-12T02:38:12.399 INFO:tasks.rados.rados.0.smithi107.stdout:4210: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:12.399 INFO:tasks.rados.rados.0.smithi107.stdout:4210: finishing write tid 5 to smithi10729407-16 2020-06-12T02:38:12.399 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 757 (ObjNum 4209 snap 0 seq_num 4209) dirty exists 2020-06-12T02:38:12.399 INFO:tasks.rados.rados.0.smithi107.stdout:4210: left oid 16 (ObjNum 4209 snap 0 seq_num 4209) 2020-06-12T02:38:12.400 INFO:tasks.rados.rados.0.smithi107.stdout:4210: done (1 left) 2020-06-12T02:38:12.400 INFO:tasks.rados.rados.0.smithi107.stdout:4213: done (0 left) 2020-06-12T02:38:12.400 INFO:tasks.rados.rados.0.smithi107.stdout:4214: delete oid 4 current snap is 0 2020-06-12T02:38:12.400 INFO:tasks.rados.rados.0.smithi107.stdout:4214: done (0 left) 2020-06-12T02:38:12.401 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:38:12.401 INFO:tasks.rados.rados.0.smithi107.stdout:4215: seq_num 4214 ranges {0=581632,581632=401408} 2020-06-12T02:38:12.401 INFO:tasks.rados.rados.0.smithi107.stdout:4215: writing smithi10729407-5 from 0 to 581632 tid 1 2020-06-12T02:38:12.401 INFO:tasks.rados.rados.0.smithi107.stdout:4215: writing smithi10729407-5 from 581632 to 983040 tid 2 2020-06-12T02:38:12.401 INFO:tasks.rados.rados.0.smithi107.stdout:4216: delete oid 1 current snap is 0 2020-06-12T02:38:12.402 INFO:tasks.rados.rados.0.smithi107.stdout:4216: done (1 left) 2020-06-12T02:38:12.402 INFO:tasks.rados.rados.0.smithi107.stdout:4217: delete oid 8 current snap is 0 2020-06-12T02:38:12.402 INFO:tasks.rados.rados.0.smithi107.stdout:4215: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:12.402 INFO:tasks.rados.rados.0.smithi107.stdout:4217: done (1 left) 2020-06-12T02:38:12.402 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:12.403 INFO:tasks.rados.rados.0.smithi107.stdout:4218: seq_num 4217 ranges {729088=630784,1359872=401408} 2020-06-12T02:38:12.404 INFO:tasks.rados.rados.0.smithi107.stdout:4218: writing smithi10729407-17 from 729088 to 1359872 tid 1 2020-06-12T02:38:12.404 INFO:tasks.rados.rados.0.smithi107.stdout:4218: writing smithi10729407-17 from 1359872 to 1761280 tid 2 2020-06-12T02:38:12.405 INFO:tasks.rados.rados.0.smithi107.stdout:4219: delete oid 4 current snap is 0 2020-06-12T02:38:12.405 INFO:tasks.rados.rados.0.smithi107.stdout:4215: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:12.405 INFO:tasks.rados.rados.0.smithi107.stdout:4215: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:12.405 INFO:tasks.rados.rados.0.smithi107.stdout:4215: finishing write tid 5 to smithi10729407-5 2020-06-12T02:38:12.405 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 423 (ObjNum 4214 snap 0 seq_num 4214) dirty exists 2020-06-12T02:38:12.406 INFO:tasks.rados.rados.0.smithi107.stdout:4215: left oid 5 (ObjNum 4214 snap 0 seq_num 4214) 2020-06-12T02:38:12.406 INFO:tasks.rados.rados.0.smithi107.stdout:4218: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:12.406 INFO:tasks.rados.rados.0.smithi107.stdout:4215: done (2 left) 2020-06-12T02:38:12.406 INFO:tasks.rados.rados.0.smithi107.stdout:4219: done (1 left) 2020-06-12T02:38:12.406 INFO:tasks.rados.rados.0.smithi107.stdout:4220: delete oid 16 current snap is 0 2020-06-12T02:38:12.407 INFO:tasks.rados.rados.0.smithi107.stdout:4218: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:12.407 INFO:tasks.rados.rados.0.smithi107.stdout:4220: done (1 left) 2020-06-12T02:38:12.407 INFO:tasks.rados.rados.0.smithi107.stdout:4221: delete oid 6 current snap is 0 2020-06-12T02:38:12.407 INFO:tasks.rados.rados.0.smithi107.stdout:4218: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:12.408 INFO:tasks.rados.rados.0.smithi107.stdout:4218: finishing write tid 5 to smithi10729407-17 2020-06-12T02:38:12.409 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 754 (ObjNum 4217 snap 0 seq_num 4217) dirty exists 2020-06-12T02:38:12.409 INFO:tasks.rados.rados.0.smithi107.stdout:4218: left oid 17 (ObjNum 4217 snap 0 seq_num 4217) 2020-06-12T02:38:12.409 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:12.405+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:12.410 INFO:tasks.rados.rados.0.smithi107.stdout:4218: done (1 left) 2020-06-12T02:38:12.411 INFO:tasks.rados.rados.0.smithi107.stdout:4221: done (0 left) 2020-06-12T02:38:12.411 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:12.411 INFO:tasks.rados.rados.0.smithi107.stdout:4222: seq_num 4221 ranges {647168=696320,1343488=401408} 2020-06-12T02:38:12.416 INFO:tasks.rados.rados.0.smithi107.stdout:4222: writing smithi10729407-14 from 647168 to 1343488 tid 1 2020-06-12T02:38:12.419 INFO:tasks.rados.rados.0.smithi107.stdout:4222: writing smithi10729407-14 from 1343488 to 1744896 tid 2 2020-06-12T02:38:12.419 INFO:tasks.rados.rados.0.smithi107.stdout:4223: delete oid 18 current snap is 0 2020-06-12T02:38:12.423 INFO:tasks.rados.rados.0.smithi107.stdout:4223: done (1 left) 2020-06-12T02:38:12.423 INFO:tasks.rados.rados.0.smithi107.stdout:4224: delete oid 13 current snap is 0 2020-06-12T02:38:12.426 INFO:tasks.rados.rados.0.smithi107.stdout:4222: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:12.426 INFO:tasks.rados.rados.0.smithi107.stdout:4224: done (1 left) 2020-06-12T02:38:12.427 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:12.427 INFO:tasks.rados.rados.0.smithi107.stdout:4225: seq_num 4224 ranges {0=753664,753664=401408} 2020-06-12T02:38:12.431 INFO:tasks.rados.rados.0.smithi107.stdout:4225: writing smithi10729407-4 from 0 to 753664 tid 1 2020-06-12T02:38:12.434 INFO:tasks.rados.rados.0.smithi107.stdout:4225: writing smithi10729407-4 from 753664 to 1155072 tid 2 2020-06-12T02:38:12.434 INFO:tasks.rados.rados.0.smithi107.stdout:4226: delete oid 17 current snap is 0 2020-06-12T02:38:12.435 INFO:tasks.rados.rados.0.smithi107.stdout:4222: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:12.436 INFO:tasks.rados.rados.0.smithi107.stdout:4222: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:12.436 INFO:tasks.rados.rados.0.smithi107.stdout:4222: finishing write tid 5 to smithi10729407-14 2020-06-12T02:38:12.436 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 743 (ObjNum 4221 snap 0 seq_num 4221) dirty exists 2020-06-12T02:38:12.436 INFO:tasks.rados.rados.0.smithi107.stdout:4222: left oid 14 (ObjNum 4221 snap 0 seq_num 4221) 2020-06-12T02:38:12.441 INFO:tasks.rados.rados.0.smithi107.stdout:4225: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:12.444 INFO:tasks.rados.rados.0.smithi107.stdout:4225: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:12.447 INFO:tasks.rados.rados.0.smithi107.stdout:4225: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:12.447 INFO:tasks.rados.rados.0.smithi107.stdout:4222: done (2 left) 2020-06-12T02:38:12.448 INFO:tasks.rados.rados.0.smithi107.stdout:4226: done (1 left) 2020-06-12T02:38:12.448 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:12.448 INFO:tasks.rados.rados.0.smithi107.stdout:4227: seq_num 4226 ranges {0=786432,786432=401408} 2020-06-12T02:38:12.451 INFO:tasks.rados.rados.0.smithi107.stdout:4227: writing smithi10729407-19 from 0 to 786432 tid 1 2020-06-12T02:38:12.454 INFO:tasks.rados.rados.0.smithi107.stdout:4227: writing smithi10729407-19 from 786432 to 1187840 tid 2 2020-06-12T02:38:12.454 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:12.455 INFO:tasks.rados.rados.0.smithi107.stdout:4225: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:12.455 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 757 (ObjNum 4224 snap 0 seq_num 4224) dirty exists 2020-06-12T02:38:12.455 INFO:tasks.rados.rados.0.smithi107.stdout:4225: left oid 4 (ObjNum 4224 snap 0 seq_num 4224) 2020-06-12T02:38:12.455 INFO:tasks.rados.rados.0.smithi107.stdout:4228: seq_num 4227 ranges {0=802816,802816=401408} 2020-06-12T02:38:12.459 INFO:tasks.rados.rados.0.smithi107.stdout:4228: writing smithi10729407-17 from 0 to 802816 tid 1 2020-06-12T02:38:12.461 INFO:tasks.rados.rados.0.smithi107.stdout:4228: writing smithi10729407-17 from 802816 to 1204224 tid 2 2020-06-12T02:38:12.461 INFO:tasks.rados.rados.0.smithi107.stdout:4225: done (2 left) 2020-06-12T02:38:12.462 INFO:tasks.rados.rados.0.smithi107.stdout:4229: delete oid 6 current snap is 0 2020-06-12T02:38:12.462 INFO:tasks.rados.rados.0.smithi107.stdout:4227: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:12.466 INFO:tasks.rados.rados.0.smithi107.stdout:4227: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:12.467 INFO:tasks.rados.rados.0.smithi107.stdout:4228: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:12.467 INFO:tasks.rados.rados.0.smithi107.stdout:4227: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:12.468 INFO:tasks.rados.rados.0.smithi107.stdout:4229: done (2 left) 2020-06-12T02:38:12.468 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:38:12.469 INFO:tasks.rados.rados.0.smithi107.stdout:4230: seq_num 4229 ranges {1523712=434176,1957888=466944,2424832=335872} 2020-06-12T02:38:12.478 INFO:tasks.rados.rados.0.smithi107.stdout:4230: writing smithi10729407-10 from 1523712 to 1957888 tid 1 2020-06-12T02:38:12.481 INFO:tasks.rados.rados.0.smithi107.stdout:4230: writing smithi10729407-10 from 1957888 to 2424832 tid 2 2020-06-12T02:38:12.483 INFO:tasks.rados.rados.0.smithi107.stdout:4230: writing smithi10729407-10 from 2424832 to 2760704 tid 3 2020-06-12T02:38:12.484 INFO:tasks.rados.rados.0.smithi107.stdout:4231: delete oid 1 current snap is 0 2020-06-12T02:38:12.484 INFO:tasks.rados.rados.0.smithi107.stdout:4227: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:12.484 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 715 (ObjNum 4226 snap 0 seq_num 4226) dirty exists 2020-06-12T02:38:12.484 INFO:tasks.rados.rados.0.smithi107.stdout:4227: left oid 19 (ObjNum 4226 snap 0 seq_num 4226) 2020-06-12T02:38:12.484 INFO:tasks.rados.rados.0.smithi107.stdout:4228: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:12.485 INFO:tasks.rados.rados.0.smithi107.stdout:4228: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:12.485 INFO:tasks.rados.rados.0.smithi107.stdout:4228: finishing write tid 5 to smithi10729407-17 2020-06-12T02:38:12.485 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 761 (ObjNum 4227 snap 0 seq_num 4227) dirty exists 2020-06-12T02:38:12.485 INFO:tasks.rados.rados.0.smithi107.stdout:4228: left oid 17 (ObjNum 4227 snap 0 seq_num 4227) 2020-06-12T02:38:12.485 INFO:tasks.rados.rados.0.smithi107.stdout:4230: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:12.486 INFO:tasks.rados.rados.0.smithi107.stdout:4227: done (3 left) 2020-06-12T02:38:12.486 INFO:tasks.rados.rados.0.smithi107.stdout:4228: done (2 left) 2020-06-12T02:38:12.486 INFO:tasks.rados.rados.0.smithi107.stdout:4231: done (1 left) 2020-06-12T02:38:12.487 INFO:tasks.rados.rados.0.smithi107.stdout:4232: delete oid 5 current snap is 0 2020-06-12T02:38:12.489 INFO:tasks.rados.rados.0.smithi107.stdout:4230: finishing write tid 2 to smithi10729407-10 2020-06-12T02:38:12.489 INFO:tasks.rados.rados.0.smithi107.stdout:4232: done (1 left) 2020-06-12T02:38:12.490 INFO:tasks.rados.rados.0.smithi107.stdout:4233: delete oid 4 current snap is 0 2020-06-12T02:38:12.490 INFO:tasks.rados.rados.0.smithi107.stdout:4230: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:12.491 INFO:tasks.rados.rados.0.smithi107.stdout:4230: finishing write tid 5 to smithi10729407-10 2020-06-12T02:38:12.492 INFO:tasks.rados.rados.0.smithi107.stdout:4230: finishing write tid 6 to smithi10729407-10 2020-06-12T02:38:12.492 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 424 (ObjNum 4229 snap 0 seq_num 4229) dirty exists 2020-06-12T02:38:12.492 INFO:tasks.rados.rados.0.smithi107.stdout:4230: left oid 10 (ObjNum 4229 snap 0 seq_num 4229) 2020-06-12T02:38:12.492 INFO:tasks.rados.rados.0.smithi107.stdout:4230: done (1 left) 2020-06-12T02:38:12.493 INFO:tasks.rados.rados.0.smithi107.stdout:4233: done (0 left) 2020-06-12T02:38:12.493 INFO:tasks.rados.rados.0.smithi107.stdout:4234: delete oid 4 current snap is 0 2020-06-12T02:38:12.494 INFO:tasks.rados.rados.0.smithi107.stdout:4234: done (0 left) 2020-06-12T02:38:12.495 INFO:tasks.rados.rados.0.smithi107.stdout:4235: delete oid 10 current snap is 0 2020-06-12T02:38:12.505 INFO:tasks.rados.rados.0.smithi107.stdout:4235: done (0 left) 2020-06-12T02:38:12.506 INFO:tasks.rados.rados.0.smithi107.stdout:4236: delete oid 11 current snap is 0 2020-06-12T02:38:12.506 INFO:tasks.rados.rados.0.smithi107.stdout:4236: done (0 left) 2020-06-12T02:38:12.506 INFO:tasks.rados.rados.0.smithi107.stdout:4237: delete oid 14 current snap is 0 2020-06-12T02:38:12.506 INFO:tasks.rados.rados.0.smithi107.stdout:4237: done (0 left) 2020-06-12T02:38:12.506 INFO:tasks.rados.rados.0.smithi107.stdout:4238: delete oid 3 current snap is 0 2020-06-12T02:38:12.507 INFO:tasks.rados.rados.0.smithi107.stdout:4238: done (0 left) 2020-06-12T02:38:12.507 INFO:tasks.rados.rados.0.smithi107.stdout:4239: delete oid 15 current snap is 0 2020-06-12T02:38:12.507 INFO:tasks.rados.rados.0.smithi107.stdout:4239: done (0 left) 2020-06-12T02:38:12.507 INFO:tasks.rados.rados.0.smithi107.stdout:4240: delete oid 7 current snap is 0 2020-06-12T02:38:12.508 INFO:tasks.rados.rados.0.smithi107.stdout:4240: done (0 left) 2020-06-12T02:38:12.508 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:12.508 INFO:tasks.rados.rados.0.smithi107.stdout:4241: seq_num 4240 ranges {1196032=614400,1810432=442368,2252800=360448} 2020-06-12T02:38:12.511 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:12.508+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:12.517 INFO:tasks.rados.rados.0.smithi107.stdout:4241: writing smithi10729407-9 from 1196032 to 1810432 tid 1 2020-06-12T02:38:12.520 INFO:tasks.rados.rados.0.smithi107.stdout:4241: writing smithi10729407-9 from 1810432 to 2252800 tid 2 2020-06-12T02:38:12.522 INFO:tasks.rados.rados.0.smithi107.stdout:4241: writing smithi10729407-9 from 2252800 to 2613248 tid 3 2020-06-12T02:38:12.523 INFO:tasks.rados.rados.0.smithi107.stdout:4242: delete oid 13 current snap is 0 2020-06-12T02:38:12.527 INFO:tasks.rados.rados.0.smithi107.stdout:4242: done (1 left) 2020-06-12T02:38:12.528 INFO:tasks.rados.rados.0.smithi107.stdout:4243: delete oid 19 current snap is 0 2020-06-12T02:38:12.528 INFO:tasks.rados.rados.0.smithi107.stdout:4241: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:12.528 INFO:tasks.rados.rados.0.smithi107.stdout:4243: done (1 left) 2020-06-12T02:38:12.529 INFO:tasks.rados.rados.0.smithi107.stdout:4244: delete oid 16 current snap is 0 2020-06-12T02:38:12.529 INFO:tasks.rados.rados.0.smithi107.stdout:4241: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:12.531 INFO:tasks.rados.rados.0.smithi107.stdout:4241: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:12.534 INFO:tasks.rados.rados.0.smithi107.stdout:4241: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:12.534 INFO:tasks.rados.rados.0.smithi107.stdout:4244: done (1 left) 2020-06-12T02:38:12.534 INFO:tasks.rados.rados.0.smithi107.stdout:4245: delete oid 18 current snap is 0 2020-06-12T02:38:12.535 INFO:tasks.rados.rados.0.smithi107.stdout:4241: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:12.536 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 748 (ObjNum 4240 snap 0 seq_num 4240) dirty exists 2020-06-12T02:38:12.536 INFO:tasks.rados.rados.0.smithi107.stdout:4241: left oid 9 (ObjNum 4240 snap 0 seq_num 4240) 2020-06-12T02:38:12.536 INFO:tasks.rados.rados.0.smithi107.stdout:4241: done (1 left) 2020-06-12T02:38:12.537 INFO:tasks.rados.rados.0.smithi107.stdout:4245: done (0 left) 2020-06-12T02:38:12.537 INFO:tasks.rados.rados.0.smithi107.stdout:4246: delete oid 18 current snap is 0 2020-06-12T02:38:12.538 INFO:tasks.rados.rados.0.smithi107.stdout:4246: done (0 left) 2020-06-12T02:38:12.538 INFO:tasks.rados.rados.0.smithi107.stdout:4247: delete oid 12 current snap is 0 2020-06-12T02:38:12.540 INFO:tasks.rados.rados.0.smithi107.stdout:4247: done (0 left) 2020-06-12T02:38:12.540 INFO:tasks.rados.rados.0.smithi107.stdout:4248: delete oid 8 current snap is 0 2020-06-12T02:38:12.542 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:12.542 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:38:12.542 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.543 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:9088 3.as0 3:5732055a:::smithi10729407-8:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e49)", 2020-06-12T02:38:12.543 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:12.539099+0000", 2020-06-12T02:38:12.543 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.0015363180000000001, 2020-06-12T02:38:12.543 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0015575949999999999, 2020-06-12T02:38:12.543 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:12.544 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:12.544 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:12.544 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:12.544 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:12.544 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9088 2020-06-12T02:38:12.545 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.545 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:12.545 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.545 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:12.546 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.539099+0000", 2020-06-12T02:38:12.546 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:12.547 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.547 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.547 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:12.547 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.539099+0000", 2020-06-12T02:38:12.547 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0000999999999998e-05 2020-06-12T02:38:12.548 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.548 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.548 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:12.548 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.539149+0000", 2020-06-12T02:38:12.548 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9729999999999992e-06 2020-06-12T02:38:12.548 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.549 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.549 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:12.549 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.539159+0000", 2020-06-12T02:38:12.549 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7765999999999999e-05 2020-06-12T02:38:12.549 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.550 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.550 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:12.550 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.539177+0000", 2020-06-12T02:38:12.551 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9230000000000008e-06 2020-06-12T02:38:12.551 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.551 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.552 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:12.552 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.539186+0000", 2020-06-12T02:38:12.552 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3742000000000002e-05 2020-06-12T02:38:12.552 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.552 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.553 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:12.553 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.539230+0000", 2020-06-12T02:38:12.553 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8753999999999999e-05 2020-06-12T02:38:12.553 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.553 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.553 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:12.554 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.539299+0000", 2020-06-12T02:38:12.554 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000199159 2020-06-12T02:38:12.554 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.554 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:12.554 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.555 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.555 INFO:teuthology.orchestra.run.smithi188.stdout: ], 2020-06-12T02:38:12.555 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 1 2020-06-12T02:38:12.555 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:12.559 INFO:tasks.rados.rados.0.smithi107.stdout:4248: done (0 left) 2020-06-12T02:38:12.559 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:12.560 INFO:tasks.rados.rados.0.smithi107.stdout:4249: seq_num 4248 ranges {0=753664,753664=770048,1523712=32768} 2020-06-12T02:38:12.560 INFO:tasks.rados.rados.0.smithi107.stdout:4249: writing smithi10729407-18 from 0 to 753664 tid 1 2020-06-12T02:38:12.560 INFO:tasks.rados.rados.0.smithi107.stdout:4249: writing smithi10729407-18 from 753664 to 1523712 tid 2 2020-06-12T02:38:12.560 INFO:tasks.rados.rados.0.smithi107.stdout:4249: writing smithi10729407-18 from 1523712 to 1556480 tid 3 2020-06-12T02:38:12.560 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:12.561 INFO:tasks.rados.rados.0.smithi107.stdout:4250: seq_num 4249 ranges {0=770048,770048=655360,1425408=147456} 2020-06-12T02:38:12.561 INFO:tasks.rados.rados.0.smithi107.stdout:4250: writing smithi10729407-7 from 0 to 770048 tid 1 2020-06-12T02:38:12.561 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-06-12T02:38:12.562 INFO:tasks.rados.rados.0.smithi107.stdout:4250: writing smithi10729407-7 from 770048 to 1425408 tid 2 2020-06-12T02:38:12.563 INFO:tasks.rados.rados.0.smithi107.stdout:4250: writing smithi10729407-7 from 1425408 to 1572864 tid 3 2020-06-12T02:38:12.563 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:12.563 INFO:tasks.rados.rados.0.smithi107.stdout:4251: seq_num 4250 ranges {0=786432,786432=589824,1376256=212992} 2020-06-12T02:38:12.567 INFO:tasks.rados.rados.0.smithi107.stdout:4251: writing smithi10729407-1 from 0 to 786432 tid 1 2020-06-12T02:38:12.571 INFO:tasks.rados.rados.0.smithi107.stdout:4251: writing smithi10729407-1 from 786432 to 1376256 tid 2 2020-06-12T02:38:12.572 INFO:tasks.rados.rados.0.smithi107.stdout:4251: writing smithi10729407-1 from 1376256 to 1589248 tid 3 2020-06-12T02:38:12.573 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:12.573 INFO:tasks.rados.rados.0.smithi107.stdout:4252: seq_num 4251 ranges {0=802816,802816=475136,1277952=327680} 2020-06-12T02:38:12.577 INFO:tasks.rados.rados.0.smithi107.stdout:4252: writing smithi10729407-11 from 0 to 802816 tid 1 2020-06-12T02:38:12.580 INFO:tasks.rados.rados.0.smithi107.stdout:4252: writing smithi10729407-11 from 802816 to 1277952 tid 2 2020-06-12T02:38:12.582 INFO:tasks.rados.rados.0.smithi107.stdout:4252: writing smithi10729407-11 from 1277952 to 1605632 tid 3 2020-06-12T02:38:12.583 INFO:tasks.rados.rados.0.smithi107.stdout:4253: delete oid 14 current snap is 0 2020-06-12T02:38:12.583 INFO:tasks.rados.rados.0.smithi107.stdout:4249: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:12.583 INFO:tasks.rados.rados.0.smithi107.stdout:4249: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:12.583 INFO:tasks.rados.rados.0.smithi107.stdout:4249: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:12.583 INFO:tasks.rados.rados.0.smithi107.stdout:4249: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:12.584 INFO:tasks.rados.rados.0.smithi107.stdout:4249: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:12.584 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 754 (ObjNum 4248 snap 0 seq_num 4248) dirty exists 2020-06-12T02:38:12.584 INFO:tasks.rados.rados.0.smithi107.stdout:4249: left oid 18 (ObjNum 4248 snap 0 seq_num 4248) 2020-06-12T02:38:12.584 INFO:tasks.rados.rados.0.smithi107.stdout:4250: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:12.584 INFO:tasks.rados.rados.0.smithi107.stdout:4250: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:12.585 INFO:tasks.rados.rados.0.smithi107.stdout:4250: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:12.585 INFO:tasks.rados.rados.0.smithi107.stdout:4250: finishing write tid 5 to smithi10729407-7 2020-06-12T02:38:12.585 INFO:tasks.rados.rados.0.smithi107.stdout:4250: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:12.585 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 807 (ObjNum 4249 snap 0 seq_num 4249) dirty exists 2020-06-12T02:38:12.585 INFO:tasks.rados.rados.0.smithi107.stdout:4250: left oid 7 (ObjNum 4249 snap 0 seq_num 4249) 2020-06-12T02:38:12.585 INFO:tasks.rados.rados.0.smithi107.stdout:4251: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:12.586 INFO:tasks.rados.rados.0.smithi107.stdout:4251: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:12.586 INFO:tasks.rados.rados.0.smithi107.stdout:4251: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:12.587 INFO:tasks.rados.rados.0.smithi107.stdout:4252: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:12.589 INFO:tasks.rados.rados.0.smithi107.stdout:4251: finishing write tid 5 to smithi10729407-1 2020-06-12T02:38:12.591 INFO:tasks.rados.rados.0.smithi107.stdout:4252: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:12.591 INFO:tasks.rados.rados.0.smithi107.stdout:4251: finishing write tid 6 to smithi10729407-1 2020-06-12T02:38:12.592 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 797 (ObjNum 4250 snap 0 seq_num 4250) dirty exists 2020-06-12T02:38:12.592 INFO:tasks.rados.rados.0.smithi107.stdout:4251: left oid 1 (ObjNum 4250 snap 0 seq_num 4250) 2020-06-12T02:38:12.592 INFO:tasks.rados.rados.0.smithi107.stdout:4252: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:12.592 INFO:tasks.rados.rados.0.smithi107.stdout:4252: finishing write tid 5 to smithi10729407-11 2020-06-12T02:38:12.592 INFO:tasks.rados.rados.0.smithi107.stdout:4249: done (4 left) 2020-06-12T02:38:12.593 INFO:tasks.rados.rados.0.smithi107.stdout:4250: done (3 left) 2020-06-12T02:38:12.593 INFO:tasks.rados.rados.0.smithi107.stdout:4251: done (2 left) 2020-06-12T02:38:12.593 INFO:tasks.rados.rados.0.smithi107.stdout:4253: done (1 left) 2020-06-12T02:38:12.593 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:38:12.593 INFO:tasks.rados.rados.0.smithi107.stdout:4254: seq_num 4253 ranges {0=434176,434176=638976,1073152=565248} 2020-06-12T02:38:12.594 INFO:tasks.rados.rados.0.smithi107.stdout:4254: writing smithi10729407-13 from 0 to 434176 tid 1 2020-06-12T02:38:12.598 INFO:tasks.rados.rados.0.smithi107.stdout:4254: writing smithi10729407-13 from 434176 to 1073152 tid 2 2020-06-12T02:38:12.601 INFO:tasks.rados.rados.0.smithi107.stdout:4254: writing smithi10729407-13 from 1073152 to 1638400 tid 3 2020-06-12T02:38:12.601 INFO:tasks.rados.rados.0.smithi107.stdout:4255: delete oid 1 current snap is 0 2020-06-12T02:38:12.601 INFO:tasks.rados.rados.0.smithi107.stdout:4252: finishing write tid 6 to smithi10729407-11 2020-06-12T02:38:12.602 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 749 (ObjNum 4251 snap 0 seq_num 4251) dirty exists 2020-06-12T02:38:12.602 INFO:tasks.rados.rados.0.smithi107.stdout:4252: left oid 11 (ObjNum 4251 snap 0 seq_num 4251) 2020-06-12T02:38:12.602 INFO:tasks.rados.rados.0.smithi107.stdout:4254: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:12.606 INFO:tasks.rados.rados.0.smithi107.stdout:4252: done (2 left) 2020-06-12T02:38:12.607 INFO:tasks.rados.rados.0.smithi107.stdout:4255: done (1 left) 2020-06-12T02:38:12.607 INFO:tasks.rados.rados.0.smithi107.stdout:4256: delete oid 10 current snap is 0 2020-06-12T02:38:12.607 INFO:tasks.rados.rados.0.smithi107.stdout:4254: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:12.609 INFO:tasks.rados.rados.0.smithi107.stdout:4254: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:12.612 INFO:tasks.rados.rados.0.smithi107.stdout:4256: done (1 left) 2020-06-12T02:38:12.612 INFO:tasks.rados.rados.0.smithi107.stdout:4257: delete oid 1 current snap is 0 2020-06-12T02:38:12.612 INFO:tasks.rados.rados.0.smithi107.stdout:4254: finishing write tid 5 to smithi10729407-13 2020-06-12T02:38:12.613 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:38:12.609+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:38:12.615 INFO:tasks.rados.rados.0.smithi107.stdout:4254: finishing write tid 6 to smithi10729407-13 2020-06-12T02:38:12.615 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 811 (ObjNum 4253 snap 0 seq_num 4253) dirty exists 2020-06-12T02:38:12.615 INFO:tasks.rados.rados.0.smithi107.stdout:4254: left oid 13 (ObjNum 4253 snap 0 seq_num 4253) 2020-06-12T02:38:12.615 INFO:tasks.rados.rados.0.smithi107.stdout:4254: done (1 left) 2020-06-12T02:38:12.615 INFO:tasks.rados.rados.0.smithi107.stdout:4257: done (0 left) 2020-06-12T02:38:12.616 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:12.616 INFO:tasks.rados.rados.0.smithi107.stdout:4258: seq_num 4257 ranges {1204224=499712,1703936=581632,2285568=548864,2834432=73728} 2020-06-12T02:38:12.624 INFO:tasks.rados.rados.0.smithi107.stdout:4258: writing smithi10729407-17 from 1204224 to 1703936 tid 1 2020-06-12T02:38:12.627 INFO:tasks.rados.rados.0.smithi107.stdout:4258: writing smithi10729407-17 from 1703936 to 2285568 tid 2 2020-06-12T02:38:12.630 INFO:tasks.rados.rados.0.smithi107.stdout:4258: writing smithi10729407-17 from 2285568 to 2834432 tid 3 2020-06-12T02:38:12.631 INFO:tasks.rados.rados.0.smithi107.stdout:4258: writing smithi10729407-17 from 2834432 to 2908160 tid 4 2020-06-12T02:38:12.631 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:38:12.634 INFO:tasks.rados.rados.0.smithi107.stdout:4259: seq_num 4258 ranges {360448=516096,876544=507904,1384448=696320} 2020-06-12T02:38:12.636 INFO:tasks.rados.rados.0.smithi107.stdout:4259: writing smithi10729407-2 from 360448 to 876544 tid 1 2020-06-12T02:38:12.639 INFO:tasks.rados.rados.0.smithi107.stdout:4259: writing smithi10729407-2 from 876544 to 1384448 tid 2 2020-06-12T02:38:12.643 INFO:tasks.rados.rados.0.smithi107.stdout:4259: writing smithi10729407-2 from 1384448 to 2080768 tid 3 2020-06-12T02:38:12.643 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:12.644 INFO:tasks.rados.rados.0.smithi107.stdout:4260: seq_num 4259 ranges {0=532480,532480=794624,1327104=409600} 2020-06-12T02:38:12.646 INFO:tasks.rados.rados.0.smithi107.stdout:4260: writing smithi10729407-12 from 0 to 532480 tid 1 2020-06-12T02:38:12.658 INFO:tasks.rados.rados.0.smithi107.stdout:4260: writing smithi10729407-12 from 532480 to 1327104 tid 2 2020-06-12T02:38:12.658 INFO:tasks.rados.rados.0.smithi107.stdout:4260: writing smithi10729407-12 from 1327104 to 1736704 tid 3 2020-06-12T02:38:12.658 INFO:tasks.rados.rados.0.smithi107.stdout:4261: delete oid 4 current snap is 0 2020-06-12T02:38:12.659 INFO:tasks.rados.rados.0.smithi107.stdout:4258: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:12.659 INFO:tasks.rados.rados.0.smithi107.stdout:4258: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:12.659 INFO:tasks.rados.rados.0.smithi107.stdout:4258: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:12.659 INFO:tasks.rados.rados.0.smithi107.stdout:4258: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:12.659 INFO:tasks.rados.rados.0.smithi107.stdout:4258: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:12.660 INFO:tasks.rados.rados.0.smithi107.stdout:4259: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:12.660 INFO:tasks.rados.rados.0.smithi107.stdout:4258: finishing write tid 7 to smithi10729407-17 2020-06-12T02:38:12.660 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 768 (ObjNum 4257 snap 0 seq_num 4257) dirty exists 2020-06-12T02:38:12.660 INFO:tasks.rados.rados.0.smithi107.stdout:4258: left oid 17 (ObjNum 4257 snap 0 seq_num 4257) 2020-06-12T02:38:12.660 INFO:tasks.rados.rados.0.smithi107.stdout:4259: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:12.660 INFO:tasks.rados.rados.0.smithi107.stdout:4259: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:12.661 INFO:tasks.rados.rados.0.smithi107.stdout:4259: finishing write tid 5 to smithi10729407-2 2020-06-12T02:38:12.661 INFO:tasks.rados.rados.0.smithi107.stdout:4259: finishing write tid 6 to smithi10729407-2 2020-06-12T02:38:12.661 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 441 (ObjNum 4258 snap 0 seq_num 4258) dirty exists 2020-06-12T02:38:12.661 INFO:tasks.rados.rados.0.smithi107.stdout:4259: left oid 2 (ObjNum 4258 snap 0 seq_num 4258) 2020-06-12T02:38:12.664 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2020-06-12T02:38:12.667 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.1 to 0.189825 (3098) 2020-06-12T02:38:12.667 INFO:tasks.rados.rados.0.smithi107.stdout:4260: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:12.668 INFO:tasks.rados.rados.0.smithi107.stdout:4258: done (3 left) 2020-06-12T02:38:12.669 INFO:tasks.rados.rados.0.smithi107.stdout:4259: done (2 left) 2020-06-12T02:38:12.669 INFO:tasks.rados.rados.0.smithi107.stdout:4261: done (1 left) 2020-06-12T02:38:12.669 INFO:tasks.rados.rados.0.smithi107.stdout:4262: delete oid 6 current snap is 0 2020-06-12T02:38:12.670 INFO:tasks.rados.rados.0.smithi107.stdout:4260: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:12.671 INFO:tasks.rados.rados.0.smithi107.stdout:4260: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:12.671 INFO:tasks.rados.rados.0.smithi107.stdout:4260: finishing write tid 5 to smithi10729407-12 2020-06-12T02:38:12.672 INFO:tasks.rados.rados.0.smithi107.stdout:4262: done (1 left) 2020-06-12T02:38:12.672 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:12.672 INFO:tasks.rados.rados.0.smithi107.stdout:4263: seq_num 4262 ranges {0=589824,589824=450560,1040384=737280,1777664=16384} 2020-06-12T02:38:12.675 INFO:tasks.rados.rados.0.smithi107.stdout:4263: writing smithi10729407-4 from 0 to 589824 tid 1 2020-06-12T02:38:12.679 INFO:tasks.rados.rados.0.smithi107.stdout:4263: writing smithi10729407-4 from 589824 to 1040384 tid 2 2020-06-12T02:38:12.684 INFO:tasks.rados.rados.0.smithi107.stdout:4263: writing smithi10729407-4 from 1040384 to 1777664 tid 3 2020-06-12T02:38:12.684 INFO:tasks.rados.rados.0.smithi107.stdout:4263: writing smithi10729407-4 from 1777664 to 1794048 tid 4 2020-06-12T02:38:12.685 INFO:tasks.rados.rados.0.smithi107.stdout:4264: delete oid 18 current snap is 0 2020-06-12T02:38:12.685 INFO:tasks.rados.rados.0.smithi107.stdout:4260: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:12.686 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 721 (ObjNum 4259 snap 0 seq_num 4259) dirty exists 2020-06-12T02:38:12.686 INFO:tasks.rados.rados.0.smithi107.stdout:4260: left oid 12 (ObjNum 4259 snap 0 seq_num 4259) 2020-06-12T02:38:12.686 INFO:tasks.rados.rados.0.smithi107.stdout:4263: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:12.688 INFO:tasks.rados.rados.0.smithi107.stdout:4263: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:12.688 INFO:tasks.rados.rados.0.smithi107.stdout:4260: done (2 left) 2020-06-12T02:38:12.688 INFO:tasks.rados.rados.0.smithi107.stdout:4264: done (1 left) 2020-06-12T02:38:12.688 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:12.688 INFO:tasks.rados.rados.0.smithi107.stdout:4265: seq_num 4264 ranges {0=622592,622592=622592,1245184=581632} 2020-06-12T02:38:12.691 INFO:tasks.rados.rados.0.smithi107.stdout:4265: writing smithi10729407-18 from 0 to 622592 tid 1 2020-06-12T02:38:12.694 INFO:tasks.rados.rados.0.smithi107.stdout:4265: writing smithi10729407-18 from 622592 to 1245184 tid 2 2020-06-12T02:38:12.698 INFO:tasks.rados.rados.0.smithi107.stdout:4265: writing smithi10729407-18 from 1245184 to 1826816 tid 3 2020-06-12T02:38:12.698 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:12.698 INFO:tasks.rados.rados.0.smithi107.stdout:4266: seq_num 4265 ranges {2613248=638976,3252224=548864,3801088=507904,4308992=147456} 2020-06-12T02:38:12.713 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:12.719 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:38:12.719 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:38:12.719 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:38:12.719 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:12.720 INFO:tasks.rados.rados.0.smithi107.stdout:4266: writing smithi10729407-9 from 2613248 to 3252224 tid 1 2020-06-12T02:38:12.730 INFO:tasks.rados.rados.0.smithi107.stdout:4266: writing smithi10729407-9 from 3252224 to 3801088 tid 2 2020-06-12T02:38:12.730 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:12.711+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:12.732 INFO:tasks.rados.rados.0.smithi107.stdout:4266: writing smithi10729407-9 from 3801088 to 4308992 tid 3 2020-06-12T02:38:12.732 INFO:tasks.rados.rados.0.smithi107.stdout:4266: writing smithi10729407-9 from 4308992 to 4456448 tid 4 2020-06-12T02:38:12.732 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:12.732 INFO:tasks.rados.rados.0.smithi107.stdout:4267: seq_num 4266 ranges {0=655360,655360=434176,1089536=720896,1810432=49152} 2020-06-12T02:38:12.733 INFO:tasks.rados.rados.0.smithi107.stdout:4267: writing smithi10729407-3 from 0 to 655360 tid 1 2020-06-12T02:38:12.733 INFO:tasks.rados.rados.0.smithi107.stdout:4267: writing smithi10729407-3 from 655360 to 1089536 tid 2 2020-06-12T02:38:12.734 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-06-12T02:38:12.735 INFO:tasks.rados.rados.0.smithi107.stdout:4267: writing smithi10729407-3 from 1089536 to 1810432 tid 3 2020-06-12T02:38:12.736 INFO:tasks.rados.rados.0.smithi107.stdout:4267: writing smithi10729407-3 from 1810432 to 1859584 tid 4 2020-06-12T02:38:12.736 INFO:tasks.rados.rados.0.smithi107.stdout:4268: delete oid 2 current snap is 0 2020-06-12T02:38:12.736 INFO:tasks.rados.rados.0.smithi107.stdout:4263: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:12.737 INFO:tasks.rados.rados.0.smithi107.stdout:4263: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:12.737 INFO:tasks.rados.rados.0.smithi107.stdout:4263: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:12.737 INFO:tasks.rados.rados.0.smithi107.stdout:4263: finishing write tid 7 to smithi10729407-4 2020-06-12T02:38:12.738 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 774 (ObjNum 4262 snap 0 seq_num 4262) dirty exists 2020-06-12T02:38:12.738 INFO:tasks.rados.rados.0.smithi107.stdout:4263: left oid 4 (ObjNum 4262 snap 0 seq_num 4262) 2020-06-12T02:38:12.738 INFO:tasks.rados.rados.0.smithi107.stdout:4265: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:12.739 INFO:tasks.rados.rados.0.smithi107.stdout:4265: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:12.739 INFO:tasks.rados.rados.0.smithi107.stdout:4265: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:12.739 INFO:tasks.rados.rados.0.smithi107.stdout:4265: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:12.739 INFO:tasks.rados.rados.0.smithi107.stdout:4265: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:12.739 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 759 (ObjNum 4264 snap 0 seq_num 4264) dirty exists 2020-06-12T02:38:12.739 INFO:tasks.rados.rados.0.smithi107.stdout:4265: left oid 18 (ObjNum 4264 snap 0 seq_num 4264) 2020-06-12T02:38:12.740 INFO:tasks.rados.rados.0.smithi107.stdout:4266: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:12.740 INFO:tasks.rados.rados.0.smithi107.stdout:4266: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:12.740 INFO:tasks.rados.rados.0.smithi107.stdout:4266: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:12.740 INFO:tasks.rados.rados.0.smithi107.stdout:4266: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:12.740 INFO:tasks.rados.rados.0.smithi107.stdout:4266: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:12.741 INFO:tasks.rados.rados.0.smithi107.stdout:4267: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:12.741 INFO:tasks.rados.rados.0.smithi107.stdout:4266: finishing write tid 7 to smithi10729407-9 2020-06-12T02:38:12.741 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 764 (ObjNum 4265 snap 0 seq_num 4265) dirty exists 2020-06-12T02:38:12.741 INFO:tasks.rados.rados.0.smithi107.stdout:4266: left oid 9 (ObjNum 4265 snap 0 seq_num 4265) 2020-06-12T02:38:12.742 INFO:tasks.rados.rados.0.smithi107.stdout:4267: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:12.746 INFO:tasks.rados.rados.0.smithi107.stdout:4267: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:12.746 INFO:tasks.rados.rados.0.smithi107.stdout:4267: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:12.746 INFO:tasks.rados.rados.0.smithi107.stdout:4267: finishing write tid 6 to smithi10729407-3 2020-06-12T02:38:12.749 INFO:tasks.rados.rados.0.smithi107.stdout:4267: finishing write tid 7 to smithi10729407-3 2020-06-12T02:38:12.750 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 765 (ObjNum 4266 snap 0 seq_num 4266) dirty exists 2020-06-12T02:38:12.750 INFO:tasks.rados.rados.0.smithi107.stdout:4267: left oid 3 (ObjNum 4266 snap 0 seq_num 4266) 2020-06-12T02:38:12.817 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:38:12.814+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:38:12.892 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:12.892 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:38:12.893 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:38:12.893 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:38:12.893 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.894 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1558 3.as0 3:5cb723d2:::smithi10729407-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e26)", 2020-06-12T02:38:12.894 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.839131+0000", 2020-06-12T02:38:12.894 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 39.048300785999999, 2020-06-12T02:38:12.895 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013057691, 2020-06-12T02:38:12.895 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:12.895 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:12.896 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:12.896 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:12.896 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:12.896 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1558 2020-06-12T02:38:12.896 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.897 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:12.897 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.897 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:12.897 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.839131+0000", 2020-06-12T02:38:12.897 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:12.898 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.898 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.898 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:12.898 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.839131+0000", 2020-06-12T02:38:12.898 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2228999999999998e-05 2020-06-12T02:38:12.899 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.899 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.899 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:12.899 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.839173+0000", 2020-06-12T02:38:12.899 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9789999999999998e-06 2020-06-12T02:38:12.899 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.900 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.900 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:12.900 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.839179+0000", 2020-06-12T02:38:12.900 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3011e-05 2020-06-12T02:38:12.900 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.901 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.901 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:12.901 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.839192+0000", 2020-06-12T02:38:12.901 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7929999999999999e-06 2020-06-12T02:38:12.901 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.902 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.902 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:12.902 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.839196+0000", 2020-06-12T02:38:12.902 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0083729379999999999 2020-06-12T02:38:12.903 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.903 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.903 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:12.903 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.847569+0000", 2020-06-12T02:38:12.903 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5887999999999998e-05 2020-06-12T02:38:12.903 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.904 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.904 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:12.904 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.847625+0000", 2020-06-12T02:38:12.904 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0027582330000000001 2020-06-12T02:38:12.904 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.905 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.905 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:12.905 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.850383+0000", 2020-06-12T02:38:12.905 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00037230500000000002 2020-06-12T02:38:12.905 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.906 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.911 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:12.911 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.850755+0000", 2020-06-12T02:38:12.912 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0014333149999999999 2020-06-12T02:38:12.912 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.912 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.913 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:12.913 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.852189+0000", 2020-06-12T02:38:12.914 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013057691 2020-06-12T02:38:12.916 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.916 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:12.917 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.917 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.917 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.918 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2746 3.as0 3:5cb723d2:::smithi10729407-15:head [append 0~524288] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:12.918 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:36.723084+0000", 2020-06-12T02:38:12.918 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 36.164347990000003, 2020-06-12T02:38:12.920 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013015831, 2020-06-12T02:38:12.920 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:12.920 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:12.920 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:12.921 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:12.921 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:12.921 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2746 2020-06-12T02:38:12.921 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.921 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:12.922 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.922 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:12.922 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.723084+0000", 2020-06-12T02:38:12.923 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:12.923 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.924 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.925 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:12.925 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.723084+0000", 2020-06-12T02:38:12.925 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0012755659999999999 2020-06-12T02:38:12.925 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.926 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.926 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:12.926 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724359+0000", 2020-06-12T02:38:12.927 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2599999999999996e-06 2020-06-12T02:38:12.927 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.927 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.927 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:12.928 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724365+0000", 2020-06-12T02:38:12.928 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5499999999999995e-06 2020-06-12T02:38:12.928 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.929 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.929 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:12.929 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724373+0000", 2020-06-12T02:38:12.930 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3590000000000001e-06 2020-06-12T02:38:12.930 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.930 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.931 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:12.931 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724378+0000", 2020-06-12T02:38:12.931 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0034266629999999999 2020-06-12T02:38:12.932 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.932 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.932 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:12.933 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.727804+0000", 2020-06-12T02:38:12.933 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6347000000000005e-05 2020-06-12T02:38:12.933 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.933 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.933 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:12.934 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.727891+0000", 2020-06-12T02:38:12.934 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054319199999999996 2020-06-12T02:38:12.934 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.934 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.935 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:12.935 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.728434+0000", 2020-06-12T02:38:12.935 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0029938999999999999 2020-06-12T02:38:12.935 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.935 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.936 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:12.936 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.731428+0000", 2020-06-12T02:38:12.936 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0046291250000000004 2020-06-12T02:38:12.936 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.936 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.937 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:12.937 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.736057+0000", 2020-06-12T02:38:12.937 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2868999999999999e-05 2020-06-12T02:38:12.937 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.937 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.938 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:12.938 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.736100+0000", 2020-06-12T02:38:12.938 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013015831 2020-06-12T02:38:12.938 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.939 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:12.939 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.939 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.940 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.940 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2747 3.as0 3:5cb723d2:::smithi10729407-15:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:12.940 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:36.724407+0000", 2020-06-12T02:38:12.941 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 36.163024823000001, 2020-06-12T02:38:12.941 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014982544, 2020-06-12T02:38:12.941 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:12.941 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:12.942 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:12.942 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:12.942 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:12.942 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2747 2020-06-12T02:38:12.943 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.943 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:12.943 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.944 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:12.944 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724407+0000", 2020-06-12T02:38:12.944 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:12.944 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.944 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.945 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:12.945 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724407+0000", 2020-06-12T02:38:12.946 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6766e-05 2020-06-12T02:38:12.946 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.946 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.947 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:12.947 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724444+0000", 2020-06-12T02:38:12.948 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.315e-06 2020-06-12T02:38:12.948 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.948 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.948 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:12.948 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724447+0000", 2020-06-12T02:38:12.949 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9100000000000002e-06 2020-06-12T02:38:12.949 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.949 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.949 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:12.949 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724453+0000", 2020-06-12T02:38:12.950 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.43e-06 2020-06-12T02:38:12.950 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.950 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.950 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:12.950 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724455+0000", 2020-06-12T02:38:12.951 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0061404010000000002 2020-06-12T02:38:12.951 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.951 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.951 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:12.952 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.730596+0000", 2020-06-12T02:38:12.952 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8402e-05 2020-06-12T02:38:12.952 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.952 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.953 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:12.953 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.730664+0000", 2020-06-12T02:38:12.953 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00031661500000000002 2020-06-12T02:38:12.953 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.953 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.954 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:12.954 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.730981+0000", 2020-06-12T02:38:12.954 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0049794890000000001 2020-06-12T02:38:12.954 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.955 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.955 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:12.955 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.735960+0000", 2020-06-12T02:38:12.955 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033245739999999998 2020-06-12T02:38:12.955 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.956 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.956 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:12.956 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.739285+0000", 2020-06-12T02:38:12.957 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010464200000000001 2020-06-12T02:38:12.957 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.957 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.957 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:12.958 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.739390+0000", 2020-06-12T02:38:12.958 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014982544 2020-06-12T02:38:12.958 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.958 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:12.958 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.959 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.959 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.959 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2748 3.as0 3:5cb723d2:::smithi10729407-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:38:12.959 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:36.724477+0000", 2020-06-12T02:38:12.959 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 36.162954683999999, 2020-06-12T02:38:12.960 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020852102000000001, 2020-06-12T02:38:12.960 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:12.960 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:12.960 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:12.960 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:12.960 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:12.961 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2748 2020-06-12T02:38:12.961 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.961 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:12.961 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.961 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:12.962 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724477+0000", 2020-06-12T02:38:12.962 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:12.962 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.962 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.962 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:12.963 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724477+0000", 2020-06-12T02:38:12.963 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2065e-05 2020-06-12T02:38:12.963 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.963 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.963 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:12.964 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724499+0000", 2020-06-12T02:38:12.964 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9340000000000002e-06 2020-06-12T02:38:12.964 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.964 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.964 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:12.964 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724502+0000", 2020-06-12T02:38:12.965 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.246e-06 2020-06-12T02:38:12.966 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.966 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.966 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:12.967 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724508+0000", 2020-06-12T02:38:12.967 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2900000000000001e-06 2020-06-12T02:38:12.967 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.967 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.967 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:12.968 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724511+0000", 2020-06-12T02:38:12.968 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0068556700000000003 2020-06-12T02:38:12.968 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.968 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.968 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:12.969 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.731366+0000", 2020-06-12T02:38:12.969 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4738999999999999e-05 2020-06-12T02:38:12.969 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.969 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.969 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:12.969 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.731421+0000", 2020-06-12T02:38:12.970 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0080153150000000003 2020-06-12T02:38:12.970 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.970 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.970 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:12.970 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.739436+0000", 2020-06-12T02:38:12.971 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010418 2020-06-12T02:38:12.971 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.971 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.971 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:12.972 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.739541+0000", 2020-06-12T02:38:12.972 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0057886630000000003 2020-06-12T02:38:12.972 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.972 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.972 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:12.973 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.745329+0000", 2020-06-12T02:38:12.973 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020852102000000001 2020-06-12T02:38:12.973 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.973 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:12.973 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.974 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.974 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.974 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:3055 3.as0 3:5cb723d2:::smithi10729407-15:head [append 0~614400] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:12.974 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.918209+0000", 2020-06-12T02:38:12.974 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.969222969, 2020-06-12T02:38:12.975 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014165394, 2020-06-12T02:38:12.975 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:12.975 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:12.975 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:12.975 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:12.975 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:12.976 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 3055 2020-06-12T02:38:12.976 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.976 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:12.976 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.976 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:12.977 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.918209+0000", 2020-06-12T02:38:12.977 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:12.977 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.977 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.977 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:12.977 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.918209+0000", 2020-06-12T02:38:12.978 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00081606199999999999 2020-06-12T02:38:12.978 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.978 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.978 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:12.978 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.919025+0000", 2020-06-12T02:38:12.979 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.251e-06 2020-06-12T02:38:12.979 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.979 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.979 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:12.979 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.919031+0000", 2020-06-12T02:38:12.980 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1460999999999999e-05 2020-06-12T02:38:12.980 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.980 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.980 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:12.981 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.919043+0000", 2020-06-12T02:38:12.981 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.592e-06 2020-06-12T02:38:12.981 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.981 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.981 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:12.982 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.919048+0000", 2020-06-12T02:38:12.982 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0020427959999999999 2020-06-12T02:38:12.982 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.982 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.982 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:12.982 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.921091+0000", 2020-06-12T02:38:12.983 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012382800000000001 2020-06-12T02:38:12.983 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.983 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.983 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:12.983 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.921215+0000", 2020-06-12T02:38:12.984 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053208300000000004 2020-06-12T02:38:12.984 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.984 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.984 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:12.984 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.921747+0000", 2020-06-12T02:38:12.984 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0075648599999999996 2020-06-12T02:38:12.985 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.985 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.985 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:12.985 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.929312+0000", 2020-06-12T02:38:12.986 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0029879630000000002 2020-06-12T02:38:12.986 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.986 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.986 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:12.986 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.932300+0000", 2020-06-12T02:38:12.987 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4498000000000004e-05 2020-06-12T02:38:12.987 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.987 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.987 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:12.987 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.932374+0000", 2020-06-12T02:38:12.988 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014165394 2020-06-12T02:38:12.988 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.988 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:12.988 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:12.988 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.988 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.989 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:3219 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:38:12.989 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:38.300493+0000", 2020-06-12T02:38:12.989 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.586938977000003, 2020-06-12T02:38:12.989 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018386370999999999, 2020-06-12T02:38:12.989 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:12.990 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:12.990 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:12.990 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:12.990 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:12.990 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 3219 2020-06-12T02:38:12.991 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.991 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:12.991 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.991 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:12.992 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300493+0000", 2020-06-12T02:38:12.992 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:12.992 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.992 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.992 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:12.992 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300493+0000", 2020-06-12T02:38:12.993 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3008000000000001e-05 2020-06-12T02:38:12.993 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.993 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.993 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:12.993 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300516+0000", 2020-06-12T02:38:12.994 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1329999999999999e-06 2020-06-12T02:38:12.994 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.994 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.994 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:12.994 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300519+0000", 2020-06-12T02:38:12.995 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8260000000000004e-06 2020-06-12T02:38:12.995 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.995 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.995 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:12.995 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300526+0000", 2020-06-12T02:38:12.995 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.525e-06 2020-06-12T02:38:12.996 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.996 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.996 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:12.996 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300528+0000", 2020-06-12T02:38:12.996 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9839e-05 2020-06-12T02:38:12.997 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.997 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.997 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:12.997 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300558+0000", 2020-06-12T02:38:12.997 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3198000000000001e-05 2020-06-12T02:38:12.997 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.998 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.998 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:12.998 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300611+0000", 2020-06-12T02:38:12.998 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0082281509999999995 2020-06-12T02:38:12.998 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:12.999 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:12.999 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:12.999 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.308840+0000", 2020-06-12T02:38:12.999 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5921000000000005e-05 2020-06-12T02:38:12.999 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.000 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.000 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:13.000 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.308935+0000", 2020-06-12T02:38:13.000 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050007560000000003 2020-06-12T02:38:13.000 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.000 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.001 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:13.001 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.313936+0000", 2020-06-12T02:38:13.001 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1600000000000007e-05 2020-06-12T02:38:13.001 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.001 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.002 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:13.002 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.313998+0000", 2020-06-12T02:38:13.002 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0048814139999999997 2020-06-12T02:38:13.002 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.002 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.002 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:13.003 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.318879+0000", 2020-06-12T02:38:13.003 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018386370999999999 2020-06-12T02:38:13.003 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:13.003 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:13.003 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:13.004 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.004 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.004 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:4478 3.as0 3:5cb723d2:::smithi10729407-15:head [append 0~483328] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-06-12T02:38:13.004 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:42.522285+0000", 2020-06-12T02:38:13.004 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 30.365147014000001, 2020-06-12T02:38:13.004 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013258748000000001, 2020-06-12T02:38:13.005 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:13.005 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:13.005 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:13.005 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:13.005 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:13.006 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 4478 2020-06-12T02:38:13.006 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.006 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:13.006 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.007 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:13.007 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.522285+0000", 2020-06-12T02:38:13.007 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:13.007 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.007 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.008 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:13.008 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.522285+0000", 2020-06-12T02:38:13.008 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00055732500000000001 2020-06-12T02:38:13.008 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.008 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.008 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:13.009 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.522842+0000", 2020-06-12T02:38:13.009 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1270000000000002e-06 2020-06-12T02:38:13.009 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.009 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.009 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:13.010 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.522847+0000", 2020-06-12T02:38:13.010 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1314e-05 2020-06-12T02:38:13.010 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.010 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.010 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:13.010 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.522859+0000", 2020-06-12T02:38:13.011 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4339999999999997e-06 2020-06-12T02:38:13.011 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.011 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.011 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:13.011 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.522865+0000", 2020-06-12T02:38:13.012 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033831099999999999 2020-06-12T02:38:13.012 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.012 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.012 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:13.013 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.526248+0000", 2020-06-12T02:38:13.013 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010359199999999999 2020-06-12T02:38:13.013 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.013 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.013 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:13.013 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.526352+0000", 2020-06-12T02:38:13.014 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031925009999999999 2020-06-12T02:38:13.014 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.014 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.014 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:13.014 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.529544+0000", 2020-06-12T02:38:13.015 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6626000000000003e-05 2020-06-12T02:38:13.015 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.015 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.015 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:13.015 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.529601+0000", 2020-06-12T02:38:13.015 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047735099999999999 2020-06-12T02:38:13.016 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.016 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.016 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:13.016 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.530078+0000", 2020-06-12T02:38:13.016 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0053046090000000001 2020-06-12T02:38:13.017 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.017 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.017 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:13.017 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.535383+0000", 2020-06-12T02:38:13.018 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011821599999999999 2020-06-12T02:38:13.018 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.018 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.018 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:13.018 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.535501+0000", 2020-06-12T02:38:13.019 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2543000000000002e-05 2020-06-12T02:38:13.019 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.019 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.019 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:13.019 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.535544+0000", 2020-06-12T02:38:13.019 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013258748000000001 2020-06-12T02:38:13.020 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:13.020 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:13.020 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:13.020 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.020 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.021 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:4535 3.as0 3:5732055a:::smithi10729407-8:head [append 0~712704] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-06-12T02:38:13.021 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:42.670365+0000", 2020-06-12T02:38:13.021 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 30.217066682999999, 2020-06-12T02:38:13.021 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013998371000000001, 2020-06-12T02:38:13.021 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:13.021 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:13.022 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:13.022 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:13.022 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:13.022 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 4535 2020-06-12T02:38:13.022 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.023 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:13.023 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.023 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:13.023 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.670365+0000", 2020-06-12T02:38:13.023 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:13.023 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.024 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.024 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:13.024 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.670365+0000", 2020-06-12T02:38:13.024 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00059204600000000004 2020-06-12T02:38:13.024 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.025 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.025 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:13.025 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.670957+0000", 2020-06-12T02:38:13.025 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.122e-06 2020-06-12T02:38:13.025 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.025 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.026 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:13.026 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.670963+0000", 2020-06-12T02:38:13.026 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1518999999999999e-05 2020-06-12T02:38:13.026 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.026 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.027 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:13.027 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.670975+0000", 2020-06-12T02:38:13.027 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9290000000000001e-06 2020-06-12T02:38:13.027 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.027 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.028 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:13.028 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.670982+0000", 2020-06-12T02:38:13.028 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4758000000000001e-05 2020-06-12T02:38:13.029 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.029 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.029 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:13.029 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.671027+0000", 2020-06-12T02:38:13.029 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9733999999999993e-05 2020-06-12T02:38:13.030 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.030 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.030 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:13.036 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.671096+0000", 2020-06-12T02:38:13.036 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000544448 2020-06-12T02:38:13.036 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.037 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.037 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:13.037 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.671641+0000", 2020-06-12T02:38:13.037 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012592534000000001 2020-06-12T02:38:13.037 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.038 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.038 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:13.038 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.684233+0000", 2020-06-12T02:38:13.038 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9785999999999998e-05 2020-06-12T02:38:13.038 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.038 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.039 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:13.039 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.684313+0000", 2020-06-12T02:38:13.039 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0494999999999997e-05 2020-06-12T02:38:13.039 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.039 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.040 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:13.040 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.684364+0000", 2020-06-12T02:38:13.040 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013998371000000001 2020-06-12T02:38:13.040 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:13.040 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:13.041 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:13.041 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.041 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.041 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:4975 3.as0 3:5732055a:::smithi10729407-8:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e30)", 2020-06-12T02:38:13.041 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:43.721794+0000", 2020-06-12T02:38:13.041 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 29.165638189999999, 2020-06-12T02:38:13.042 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.1825135, 2020-06-12T02:38:13.042 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:13.042 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:13.042 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:13.042 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:13.043 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:13.043 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 4975 2020-06-12T02:38:13.043 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.043 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:13.043 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.044 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:13.044 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721794+0000", 2020-06-12T02:38:13.044 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:13.044 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.044 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.044 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:13.045 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721794+0000", 2020-06-12T02:38:13.045 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4542e-05 2020-06-12T02:38:13.045 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.045 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.045 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:13.046 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721818+0000", 2020-06-12T02:38:13.046 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9169999999999997e-06 2020-06-12T02:38:13.046 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.046 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.046 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:13.046 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721823+0000", 2020-06-12T02:38:13.047 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1184e-05 2020-06-12T02:38:13.047 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.047 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.047 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:13.047 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721834+0000", 2020-06-12T02:38:13.048 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.254e-06 2020-06-12T02:38:13.048 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.048 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.048 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:13.048 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721840+0000", 2020-06-12T02:38:13.048 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1075999999999998e-05 2020-06-12T02:38:13.049 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.049 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.049 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:13.049 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721881+0000", 2020-06-12T02:38:13.049 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7416999999999995e-05 2020-06-12T02:38:13.050 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.050 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.050 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:13.050 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721948+0000", 2020-06-12T02:38:13.050 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.18235910999999999 2020-06-12T02:38:13.050 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.051 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.051 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:13.051 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.904307+0000", 2020-06-12T02:38:13.051 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.1825135 2020-06-12T02:38:13.051 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:13.052 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:13.052 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:13.052 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.052 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.052 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:6205 3.as0 3:5cb723d2:::smithi10729407-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e31)", 2020-06-12T02:38:13.053 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:47.353583+0000", 2020-06-12T02:38:13.053 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 25.533848979999998, 2020-06-12T02:38:13.053 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013040737, 2020-06-12T02:38:13.053 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:13.054 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:13.054 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:13.054 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:13.054 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:13.054 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 6205 2020-06-12T02:38:13.055 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.055 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:13.055 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.055 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:13.055 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.353583+0000", 2020-06-12T02:38:13.055 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:13.056 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.056 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.056 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:13.056 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.353583+0000", 2020-06-12T02:38:13.056 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3133000000000002e-05 2020-06-12T02:38:13.057 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.057 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.057 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:13.057 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.353626+0000", 2020-06-12T02:38:13.057 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9570000000000008e-06 2020-06-12T02:38:13.057 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.058 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.058 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:13.058 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.353635+0000", 2020-06-12T02:38:13.058 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1754e-05 2020-06-12T02:38:13.058 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.059 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.059 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:13.059 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.353647+0000", 2020-06-12T02:38:13.059 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0809999999999998e-06 2020-06-12T02:38:13.059 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.060 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.060 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:13.060 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.353650+0000", 2020-06-12T02:38:13.060 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0064807490000000001 2020-06-12T02:38:13.060 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.060 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.061 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:13.061 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.360131+0000", 2020-06-12T02:38:13.061 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7163000000000001e-05 2020-06-12T02:38:13.061 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.061 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.062 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:13.062 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.360188+0000", 2020-06-12T02:38:13.062 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0053632230000000003 2020-06-12T02:38:13.062 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.062 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.062 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:13.063 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.365551+0000", 2020-06-12T02:38:13.063 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8078999999999997e-05 2020-06-12T02:38:13.063 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.063 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.063 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:13.064 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.365609+0000", 2020-06-12T02:38:13.064 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001014598 2020-06-12T02:38:13.064 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.064 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.064 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:13.064 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.366624+0000", 2020-06-12T02:38:13.065 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013040737 2020-06-12T02:38:13.065 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:13.065 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:13.065 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:13.065 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.066 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.066 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:6678 3.as0 3:5732055a:::smithi10729407-8:head [append 0~499712] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-06-12T02:38:13.066 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:48.626196+0000", 2020-06-12T02:38:13.066 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 24.261236207, 2020-06-12T02:38:13.066 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014093700000000001, 2020-06-12T02:38:13.066 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:13.067 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:13.067 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:13.067 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:13.067 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:13.068 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 6678 2020-06-12T02:38:13.068 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.068 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:13.068 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.068 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:13.069 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.626196+0000", 2020-06-12T02:38:13.069 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:13.069 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.069 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.069 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:13.069 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.626196+0000", 2020-06-12T02:38:13.070 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00046254199999999998 2020-06-12T02:38:13.070 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.070 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.070 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:13.070 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.626658+0000", 2020-06-12T02:38:13.071 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3400000000000003e-06 2020-06-12T02:38:13.071 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.071 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.071 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:13.071 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.626665+0000", 2020-06-12T02:38:13.072 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0079e-05 2020-06-12T02:38:13.072 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.072 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.072 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:13.072 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.626675+0000", 2020-06-12T02:38:13.072 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0999999999999997e-06 2020-06-12T02:38:13.073 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.073 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.073 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:13.073 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.626679+0000", 2020-06-12T02:38:13.073 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025692100000000002 2020-06-12T02:38:13.074 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.074 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.074 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:13.074 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.629248+0000", 2020-06-12T02:38:13.074 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000107944 2020-06-12T02:38:13.075 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.075 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.075 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:13.075 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.629356+0000", 2020-06-12T02:38:13.075 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001434999 2020-06-12T02:38:13.075 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.076 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.076 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:13.076 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.630791+0000", 2020-06-12T02:38:13.076 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0716e-05 2020-06-12T02:38:13.076 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.077 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.077 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:13.077 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.630842+0000", 2020-06-12T02:38:13.077 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00045107600000000002 2020-06-12T02:38:13.077 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.078 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.078 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:13.078 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.631293+0000", 2020-06-12T02:38:13.078 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0048208330000000001 2020-06-12T02:38:13.078 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.078 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.079 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:13.079 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.636113+0000", 2020-06-12T02:38:13.079 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0041129679999999998 2020-06-12T02:38:13.079 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.079 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.080 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:13.080 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.640226+0000", 2020-06-12T02:38:13.080 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2892999999999996e-05 2020-06-12T02:38:13.080 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.080 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.080 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:13.081 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.640289+0000", 2020-06-12T02:38:13.081 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014093700000000001 2020-06-12T02:38:13.081 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:13.081 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:13.081 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:13.082 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.082 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.082 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:7084 3.as0 3:5732055a:::smithi10729407-8:head [append 0~401408] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-06-12T02:38:13.082 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:49.666656+0000", 2020-06-12T02:38:13.082 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 23.220775462999999, 2020-06-12T02:38:13.083 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013133027, 2020-06-12T02:38:13.083 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:13.083 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:13.083 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:13.083 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:13.083 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:13.084 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 7084 2020-06-12T02:38:13.084 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.084 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:13.084 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.084 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:13.085 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.666656+0000", 2020-06-12T02:38:13.085 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:13.085 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.085 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.085 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:13.086 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.666656+0000", 2020-06-12T02:38:13.086 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00040755700000000001 2020-06-12T02:38:13.086 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.086 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.086 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:13.087 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667064+0000", 2020-06-12T02:38:13.087 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9479999999999999e-06 2020-06-12T02:38:13.087 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.087 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.087 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:13.088 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667069+0000", 2020-06-12T02:38:13.088 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.568e-06 2020-06-12T02:38:13.088 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.088 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.088 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:13.088 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667077+0000", 2020-06-12T02:38:13.089 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4170000000000002e-06 2020-06-12T02:38:13.089 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.089 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.089 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:13.089 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667082+0000", 2020-06-12T02:38:13.090 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0034817770000000001 2020-06-12T02:38:13.090 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.090 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.090 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:13.090 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.670564+0000", 2020-06-12T02:38:13.091 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011071200000000001 2020-06-12T02:38:13.091 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.091 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.091 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:13.091 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.670674+0000", 2020-06-12T02:38:13.091 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0011185780000000001 2020-06-12T02:38:13.092 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.092 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.092 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:13.092 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.671793+0000", 2020-06-12T02:38:13.092 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2209e-05 2020-06-12T02:38:13.093 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.093 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.093 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:13.093 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.671855+0000", 2020-06-12T02:38:13.093 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054805100000000003 2020-06-12T02:38:13.093 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.094 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.094 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:13.094 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.672403+0000", 2020-06-12T02:38:13.094 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0044889480000000004 2020-06-12T02:38:13.094 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.095 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.095 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:13.095 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.676892+0000", 2020-06-12T02:38:13.095 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0028428730000000001 2020-06-12T02:38:13.095 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.095 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.096 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:13.096 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.679735+0000", 2020-06-12T02:38:13.096 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4388999999999997e-05 2020-06-12T02:38:13.096 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.096 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.097 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:13.097 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.679789+0000", 2020-06-12T02:38:13.097 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013133027 2020-06-12T02:38:13.097 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:13.097 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:13.097 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:13.098 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.098 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.098 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:7086 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e32)", 2020-06-12T02:38:13.098 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:49.667192+0000", 2020-06-12T02:38:13.098 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 23.220239878000001, 2020-06-12T02:38:13.099 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014618023000000001, 2020-06-12T02:38:13.099 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:13.099 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:13.099 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:13.099 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:13.099 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:13.100 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 7086 2020-06-12T02:38:13.100 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.100 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:13.100 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.100 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:13.101 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667192+0000", 2020-06-12T02:38:13.101 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:13.101 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.101 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.101 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:13.102 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667192+0000", 2020-06-12T02:38:13.102 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1912e-05 2020-06-12T02:38:13.102 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.102 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.102 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:13.103 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667214+0000", 2020-06-12T02:38:13.103 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0469999999999998e-06 2020-06-12T02:38:13.103 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.103 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.103 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:13.104 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667217+0000", 2020-06-12T02:38:13.104 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7170000000000002e-06 2020-06-12T02:38:13.104 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.104 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.104 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:13.105 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667224+0000", 2020-06-12T02:38:13.105 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2850000000000001e-06 2020-06-12T02:38:13.105 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.105 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.105 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:13.106 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667226+0000", 2020-06-12T02:38:13.106 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0037197950000000001 2020-06-12T02:38:13.106 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.106 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.106 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:13.107 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.670946+0000", 2020-06-12T02:38:13.107 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4549999999999994e-05 2020-06-12T02:38:13.107 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.107 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.107 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:13.108 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.671040+0000", 2020-06-12T02:38:13.108 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0043129199999999996 2020-06-12T02:38:13.108 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.108 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.108 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:13.109 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.675353+0000", 2020-06-12T02:38:13.109 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1982000000000001e-05 2020-06-12T02:38:13.109 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.109 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.109 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:13.110 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.675445+0000", 2020-06-12T02:38:13.110 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.005067401 2020-06-12T02:38:13.110 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.110 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.110 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:13.110 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.680513+0000", 2020-06-12T02:38:13.111 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000101508 2020-06-12T02:38:13.111 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.112 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.112 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:13.112 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.680614+0000", 2020-06-12T02:38:13.112 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0011959060000000001 2020-06-12T02:38:13.112 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.113 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.113 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:13.113 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.681810+0000", 2020-06-12T02:38:13.113 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014618023000000001 2020-06-12T02:38:13.113 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:13.114 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:13.114 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:13.114 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.114 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.114 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:7902 3.as0 3:5732055a:::smithi10729407-8:head [append 0~499712] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-06-12T02:38:13.114 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.159351+0000", 2020-06-12T02:38:13.115 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.728081200999998, 2020-06-12T02:38:13.115 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013425812000000001, 2020-06-12T02:38:13.115 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:13.115 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:13.115 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:13.116 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:13.116 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:13.116 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 7902 2020-06-12T02:38:13.116 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.116 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:13.117 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.117 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:13.117 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.159351+0000", 2020-06-12T02:38:13.117 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:13.117 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.118 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.118 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:13.118 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.159351+0000", 2020-06-12T02:38:13.118 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00048251199999999999 2020-06-12T02:38:13.119 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.119 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.119 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:13.120 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.159833+0000", 2020-06-12T02:38:13.120 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7509999999999997e-06 2020-06-12T02:38:13.120 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.120 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.121 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:13.121 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.159839+0000", 2020-06-12T02:38:13.121 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9840000000000007e-06 2020-06-12T02:38:13.121 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.121 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.122 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:13.122 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.159848+0000", 2020-06-12T02:38:13.122 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4619999999999998e-06 2020-06-12T02:38:13.122 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.122 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.123 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:13.123 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.159852+0000", 2020-06-12T02:38:13.123 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5348000000000001e-05 2020-06-12T02:38:13.123 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.123 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.124 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:13.124 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.159888+0000", 2020-06-12T02:38:13.124 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9824000000000001e-05 2020-06-12T02:38:13.124 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.124 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.125 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:13.125 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.159958+0000", 2020-06-12T02:38:13.125 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0014605149999999999 2020-06-12T02:38:13.125 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.125 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.125 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:13.126 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.161418+0000", 2020-06-12T02:38:13.126 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0667999999999997e-05 2020-06-12T02:38:13.126 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.126 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.126 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:13.127 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.161469+0000", 2020-06-12T02:38:13.127 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00058577200000000003 2020-06-12T02:38:13.127 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.127 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.127 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:13.127 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.162054+0000", 2020-06-12T02:38:13.128 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010590761000000001 2020-06-12T02:38:13.128 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.128 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.128 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:13.128 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.172645+0000", 2020-06-12T02:38:13.129 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8467000000000006e-05 2020-06-12T02:38:13.129 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.129 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.129 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:13.129 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.172724+0000", 2020-06-12T02:38:13.130 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2747999999999997e-05 2020-06-12T02:38:13.130 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.130 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.130 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:13.130 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.172776+0000", 2020-06-12T02:38:13.130 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013425812000000001 2020-06-12T02:38:13.131 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:13.131 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:13.131 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:13.131 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.131 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.132 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:7907 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e33)", 2020-06-12T02:38:13.132 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.166559+0000", 2020-06-12T02:38:13.132 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.720872499999999, 2020-06-12T02:38:13.132 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013016145999999999, 2020-06-12T02:38:13.132 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:13.132 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:13.133 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:13.133 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:13.133 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:13.133 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 7907 2020-06-12T02:38:13.133 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.134 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:13.134 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.134 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:13.134 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.166559+0000", 2020-06-12T02:38:13.134 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:13.134 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.135 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.135 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:13.135 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.166559+0000", 2020-06-12T02:38:13.135 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1719999999999999e-05 2020-06-12T02:38:13.135 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.136 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.136 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:13.136 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.166581+0000", 2020-06-12T02:38:13.136 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8660000000000002e-06 2020-06-12T02:38:13.136 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.136 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.137 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:13.137 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.166584+0000", 2020-06-12T02:38:13.137 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3439999999999997e-06 2020-06-12T02:38:13.137 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.137 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.138 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:13.138 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.166590+0000", 2020-06-12T02:38:13.138 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0640000000000001e-06 2020-06-12T02:38:13.138 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.138 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.139 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:13.139 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.166592+0000", 2020-06-12T02:38:13.139 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0074059160000000002 2020-06-12T02:38:13.139 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.139 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.140 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:13.140 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.173998+0000", 2020-06-12T02:38:13.140 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8678999999999998e-05 2020-06-12T02:38:13.140 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.140 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.140 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:13.141 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.174057+0000", 2020-06-12T02:38:13.141 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045994779999999997 2020-06-12T02:38:13.141 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.141 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.141 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:13.142 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.178656+0000", 2020-06-12T02:38:13.142 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6825999999999999e-05 2020-06-12T02:38:13.142 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.142 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.142 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:13.142 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.178733+0000", 2020-06-12T02:38:13.143 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00084225299999999995 2020-06-12T02:38:13.143 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.143 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.143 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:13.143 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.179575+0000", 2020-06-12T02:38:13.144 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013016145999999999 2020-06-12T02:38:13.144 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:13.144 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:13.144 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:13.144 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.145 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.145 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:7931 3.as0 3:5732055a:::smithi10729407-8:head [append 0~696320] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-06-12T02:38:13.145 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.246494+0000", 2020-06-12T02:38:13.145 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.640937384000001, 2020-06-12T02:38:13.145 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014082301, 2020-06-12T02:38:13.146 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:13.146 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:13.146 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:13.146 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:13.146 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:13.146 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 7931 2020-06-12T02:38:13.147 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.147 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:13.147 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.147 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:13.147 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.246494+0000", 2020-06-12T02:38:13.148 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:13.148 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.148 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.148 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:13.148 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.246494+0000", 2020-06-12T02:38:13.148 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00077570099999999995 2020-06-12T02:38:13.149 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.149 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.149 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:13.149 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.247270+0000", 2020-06-12T02:38:13.149 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4179999999999997e-06 2020-06-12T02:38:13.150 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.150 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.150 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:13.150 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.247279+0000", 2020-06-12T02:38:13.150 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6297999999999999e-05 2020-06-12T02:38:13.151 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.151 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.151 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:13.151 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.247295+0000", 2020-06-12T02:38:13.151 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3309999999999998e-06 2020-06-12T02:38:13.151 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.152 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.152 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:13.152 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.247302+0000", 2020-06-12T02:38:13.152 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4283999999999999e-05 2020-06-12T02:38:13.152 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.153 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.153 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:13.153 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.247336+0000", 2020-06-12T02:38:13.153 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8157999999999994e-05 2020-06-12T02:38:13.153 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.153 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.154 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:13.154 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.247405+0000", 2020-06-12T02:38:13.154 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031257059999999998 2020-06-12T02:38:13.154 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.154 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.155 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:13.155 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.250530+0000", 2020-06-12T02:38:13.155 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7546999999999995e-05 2020-06-12T02:38:13.155 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.155 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.155 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:13.156 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.250608+0000", 2020-06-12T02:38:13.156 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00074126599999999997 2020-06-12T02:38:13.156 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.156 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.156 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:13.157 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.251349+0000", 2020-06-12T02:38:13.157 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.008999211 2020-06-12T02:38:13.157 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.157 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.157 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:13.157 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.260348+0000", 2020-06-12T02:38:13.158 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00016373200000000001 2020-06-12T02:38:13.158 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.158 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.158 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:13.158 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.260512+0000", 2020-06-12T02:38:13.159 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4648999999999997e-05 2020-06-12T02:38:13.159 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.159 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.159 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:13.159 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.260577+0000", 2020-06-12T02:38:13.159 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014082301 2020-06-12T02:38:13.160 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:13.160 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:13.160 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:13.160 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.160 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.161 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8129 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e33)", 2020-06-12T02:38:13.161 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.700485+0000", 2020-06-12T02:38:13.161 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 20.186947015000001, 2020-06-12T02:38:13.161 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01339515, 2020-06-12T02:38:13.161 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:13.162 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:13.162 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:13.162 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:13.162 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:13.162 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8129 2020-06-12T02:38:13.162 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.163 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:13.163 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.163 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:13.163 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.700485+0000", 2020-06-12T02:38:13.163 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:13.164 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.164 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.164 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:13.164 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.700485+0000", 2020-06-12T02:38:13.164 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4323999999999998e-05 2020-06-12T02:38:13.164 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.165 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.165 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:13.165 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.700529+0000", 2020-06-12T02:38:13.165 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7100000000000004e-06 2020-06-12T02:38:13.165 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.166 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.166 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:13.166 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.700535+0000", 2020-06-12T02:38:13.166 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1664999999999999e-05 2020-06-12T02:38:13.166 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.166 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.167 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:13.167 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.700547+0000", 2020-06-12T02:38:13.167 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6150000000000001e-06 2020-06-12T02:38:13.167 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.167 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.168 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:13.168 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.700550+0000", 2020-06-12T02:38:13.168 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00049316500000000005 2020-06-12T02:38:13.168 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.169 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.169 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:13.169 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.701043+0000", 2020-06-12T02:38:13.169 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0063999999999997e-05 2020-06-12T02:38:13.169 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.170 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.170 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:13.170 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.701133+0000", 2020-06-12T02:38:13.170 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0085745109999999999 2020-06-12T02:38:13.170 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.170 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.171 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:13.171 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.709708+0000", 2020-06-12T02:38:13.171 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8854999999999999e-05 2020-06-12T02:38:13.171 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.171 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.172 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:13.172 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.709757+0000", 2020-06-12T02:38:13.172 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002613279 2020-06-12T02:38:13.172 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.172 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.172 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:13.173 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.712370+0000", 2020-06-12T02:38:13.173 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4332000000000004e-05 2020-06-12T02:38:13.173 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.173 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.173 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:13.174 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.712464+0000", 2020-06-12T02:38:13.174 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00141563 2020-06-12T02:38:13.174 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.174 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.174 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:13.175 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.713880+0000", 2020-06-12T02:38:13.175 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01339515 2020-06-12T02:38:13.175 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:13.175 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:13.176 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:13.176 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.176 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.176 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8780 3.as0 3:5732055a:::smithi10729407-8:head [append 0~106496] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:13.176 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:11.606644+0000", 2020-06-12T02:38:13.177 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 1.280787557, 2020-06-12T02:38:13.177 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013846951999999999, 2020-06-12T02:38:13.177 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:13.177 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:13.177 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:13.178 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:13.178 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:13.178 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8780 2020-06-12T02:38:13.178 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.178 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:13.178 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.179 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:13.179 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.606644+0000", 2020-06-12T02:38:13.179 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:13.179 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.179 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.180 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:13.180 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.606644+0000", 2020-06-12T02:38:13.180 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000139424 2020-06-12T02:38:13.180 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.180 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.181 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:13.181 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.606784+0000", 2020-06-12T02:38:13.181 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3067999999999999e-05 2020-06-12T02:38:13.181 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.181 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.181 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:13.182 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.606797+0000", 2020-06-12T02:38:13.182 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1382999999999999e-05 2020-06-12T02:38:13.182 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.182 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.183 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:13.184 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.606808+0000", 2020-06-12T02:38:13.184 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2689999999999996e-06 2020-06-12T02:38:13.184 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.184 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.184 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:13.185 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.606814+0000", 2020-06-12T02:38:13.185 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0016745130000000001 2020-06-12T02:38:13.185 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.185 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.185 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:13.186 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.608489+0000", 2020-06-12T02:38:13.186 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000113917 2020-06-12T02:38:13.186 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.186 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.186 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:13.187 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.608603+0000", 2020-06-12T02:38:13.187 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0072825420000000004 2020-06-12T02:38:13.188 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.188 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.188 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:13.188 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.615885+0000", 2020-06-12T02:38:13.188 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0661999999999995e-05 2020-06-12T02:38:13.188 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.189 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.189 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:13.189 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.615956+0000", 2020-06-12T02:38:13.189 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000395478 2020-06-12T02:38:13.189 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.190 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.190 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:13.190 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.616352+0000", 2020-06-12T02:38:13.190 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00217542 2020-06-12T02:38:13.190 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.191 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.191 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:13.191 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.618527+0000", 2020-06-12T02:38:13.191 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001923157 2020-06-12T02:38:13.192 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.192 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.192 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:13.192 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.620450+0000", 2020-06-12T02:38:13.192 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1118999999999997e-05 2020-06-12T02:38:13.193 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.193 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.193 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:13.193 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.620491+0000", 2020-06-12T02:38:13.193 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013846951999999999 2020-06-12T02:38:13.193 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:13.194 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:13.194 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:13.194 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.194 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.194 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8781 3.as0 3:5732055a:::smithi10729407-8:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:13.195 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:11.606865+0000", 2020-06-12T02:38:13.195 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 1.28056638, 2020-06-12T02:38:13.195 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016125024000000002, 2020-06-12T02:38:13.195 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:13.195 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:13.196 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:13.196 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:13.196 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:13.196 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8781 2020-06-12T02:38:13.197 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.197 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:13.197 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.197 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:13.197 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.606865+0000", 2020-06-12T02:38:13.198 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:13.198 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.198 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.198 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:13.198 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.606865+0000", 2020-06-12T02:38:13.199 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4474000000000006e-05 2020-06-12T02:38:13.199 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.199 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.199 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:13.199 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.606940+0000", 2020-06-12T02:38:13.200 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.579e-06 2020-06-12T02:38:13.200 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.200 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.200 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:13.200 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.606946+0000", 2020-06-12T02:38:13.200 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2563e-05 2020-06-12T02:38:13.201 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.201 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.201 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:13.201 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.606959+0000", 2020-06-12T02:38:13.202 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.792e-06 2020-06-12T02:38:13.202 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.202 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.202 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:13.202 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.606964+0000", 2020-06-12T02:38:13.203 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0016792689999999999 2020-06-12T02:38:13.203 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.203 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.203 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:13.204 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.608643+0000", 2020-06-12T02:38:13.204 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9752000000000001e-05 2020-06-12T02:38:13.204 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.204 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.204 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:13.205 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.608693+0000", 2020-06-12T02:38:13.205 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0087402529999999999 2020-06-12T02:38:13.205 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.205 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.205 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:13.206 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.617433+0000", 2020-06-12T02:38:13.206 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8889999999999999e-05 2020-06-12T02:38:13.206 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.206 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.206 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:13.206 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.617502+0000", 2020-06-12T02:38:13.207 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00029944299999999999 2020-06-12T02:38:13.207 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.207 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.207 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:13.207 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.617801+0000", 2020-06-12T02:38:13.208 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024307959999999998 2020-06-12T02:38:13.208 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.208 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.208 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:13.208 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.620232+0000", 2020-06-12T02:38:13.208 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002703567 2020-06-12T02:38:13.209 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.209 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.209 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:13.209 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.622936+0000", 2020-06-12T02:38:13.209 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4645999999999998e-05 2020-06-12T02:38:13.210 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.210 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.210 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:13.210 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.622990+0000", 2020-06-12T02:38:13.210 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016125024000000002 2020-06-12T02:38:13.211 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:13.211 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:13.211 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:13.211 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.212 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.212 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8782 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e49)", 2020-06-12T02:38:13.212 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:11.607007+0000", 2020-06-12T02:38:13.212 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 1.280424644, 2020-06-12T02:38:13.212 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019139309, 2020-06-12T02:38:13.213 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:13.213 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:13.213 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:13.213 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:13.214 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:13.214 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8782 2020-06-12T02:38:13.214 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.214 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:13.215 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.215 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:13.215 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.607007+0000", 2020-06-12T02:38:13.215 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:13.215 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.216 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.216 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:13.216 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.607007+0000", 2020-06-12T02:38:13.216 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4070000000000003e-05 2020-06-12T02:38:13.216 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.217 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.217 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:13.217 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.607051+0000", 2020-06-12T02:38:13.217 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8749999999999997e-06 2020-06-12T02:38:13.217 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.217 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.218 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:13.218 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.607057+0000", 2020-06-12T02:38:13.218 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3715e-05 2020-06-12T02:38:13.218 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.218 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.219 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:13.219 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.607071+0000", 2020-06-12T02:38:13.219 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3939999999999998e-06 2020-06-12T02:38:13.219 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.219 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.219 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:13.220 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.607075+0000", 2020-06-12T02:38:13.220 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001658678 2020-06-12T02:38:13.220 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.220 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.220 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:13.221 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.608734+0000", 2020-06-12T02:38:13.221 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4022000000000002e-05 2020-06-12T02:38:13.221 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.222 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.222 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:13.222 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.608828+0000", 2020-06-12T02:38:13.222 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0093690579999999996 2020-06-12T02:38:13.223 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.223 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.223 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:13.223 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.618197+0000", 2020-06-12T02:38:13.223 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5519999999999994e-05 2020-06-12T02:38:13.223 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.224 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.224 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:13.224 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.618283+0000", 2020-06-12T02:38:13.224 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.005054205 2020-06-12T02:38:13.224 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.225 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.225 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:13.225 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.623337+0000", 2020-06-12T02:38:13.225 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8203999999999998e-05 2020-06-12T02:38:13.225 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.225 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.226 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:13.226 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.623425+0000", 2020-06-12T02:38:13.226 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0027215680000000002 2020-06-12T02:38:13.226 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:13.226 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:13.227 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:13.227 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.626146+0000", 2020-06-12T02:38:13.227 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019139309 2020-06-12T02:38:13.227 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:13.227 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:13.227 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:13.228 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:13.228 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:13.228 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:13.229 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-06-12T02:38:13.230 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-06-12T02:38:13.231 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:13.227+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:13.330 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:13.328+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:13.367 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:38:13.386 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:38:13.389 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:13.390 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:38:13.390 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2020-06-12T02:38:13.390 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:13.398 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-06-12T02:38:13.412 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:38:13.424 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:38:13.439 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:13.437+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:13.444 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:38:13.464 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:38:13.484 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:38:13.503 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2020-06-12T02:38:13.510 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-06-12T02:38:13.510 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-06-12T02:38:13.511 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-06-12T02:38:13.511 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-06-12T02:38:13.511 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-06-12T02:38:13.511 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-06-12T02:38:13.511 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-06-12T02:38:13.511 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-06-12T02:38:13.532 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:13.529+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:13.555 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:13.556 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:13.565 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-06-12T02:38:13.633 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:13.630+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:13.727 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:13.727 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:13.741 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:13.731+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:13.743 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-06-12T02:38:13.834 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:13.832+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:13.859 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:13.860 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:13.870 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-06-12T02:38:13.935 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:13.932+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:13.986 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:13.987 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:13.995 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-06-12T02:38:14.036 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:14.033+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:14.137 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:14.135+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:14.159 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:14.159 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:14.160 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-06-12T02:38:14.238 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:14.235+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:14.282 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:14.282 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:14.291 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-06-12T02:38:14.375 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:14.372+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:14.439 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:38:14.436+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:38:14.447 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:14.448 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:14.456 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-06-12T02:38:14.540 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:14.537+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:14.611 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:14.612 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:14.620 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-06-12T02:38:14.641 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:38:14.639+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:38:14.676 INFO:tasks.rados.rados.0.smithi107.stdout:4263: done (4 left) 2020-06-12T02:38:14.676 INFO:tasks.rados.rados.0.smithi107.stdout:4265: done (3 left) 2020-06-12T02:38:14.676 INFO:tasks.rados.rados.0.smithi107.stdout:4266: done (2 left) 2020-06-12T02:38:14.676 INFO:tasks.rados.rados.0.smithi107.stdout:4267: done (1 left) 2020-06-12T02:38:14.677 INFO:tasks.rados.rados.0.smithi107.stdout:4268: done (0 left) 2020-06-12T02:38:14.677 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:14.677 INFO:tasks.rados.rados.0.smithi107.stdout:4269: seq_num 4268 ranges {1605632=688128,2293760=606208,2899968=598016} 2020-06-12T02:38:14.678 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd set nodeep-scrub 2020-06-12T02:38:14.693 INFO:tasks.rados.rados.0.smithi107.stdout:4269: writing smithi10729407-11 from 1605632 to 2293760 tid 1 2020-06-12T02:38:14.693 INFO:tasks.rados.rados.0.smithi107.stdout:4269: writing smithi10729407-11 from 2293760 to 2899968 tid 2 2020-06-12T02:38:14.697 INFO:tasks.rados.rados.0.smithi107.stdout:4269: writing smithi10729407-11 from 2899968 to 3497984 tid 3 2020-06-12T02:38:14.697 INFO:tasks.rados.rados.0.smithi107.stdout:4270: delete oid 14 current snap is 0 2020-06-12T02:38:14.703 INFO:tasks.rados.rados.0.smithi107.stdout:4269: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:14.706 INFO:tasks.rados.rados.0.smithi107.stdout:4269: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:14.706 INFO:tasks.rados.rados.0.smithi107.stdout:4269: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:14.706 INFO:tasks.rados.rados.0.smithi107.stdout:4269: finishing write tid 5 to smithi10729407-11 2020-06-12T02:38:14.707 INFO:tasks.rados.rados.0.smithi107.stdout:4269: finishing write tid 6 to smithi10729407-11 2020-06-12T02:38:14.707 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 754 (ObjNum 4268 snap 0 seq_num 4268) dirty exists 2020-06-12T02:38:14.708 INFO:tasks.rados.rados.0.smithi107.stdout:4269: left oid 11 (ObjNum 4268 snap 0 seq_num 4268) 2020-06-12T02:38:14.708 INFO:tasks.rados.rados.0.smithi107.stdout:4269: done (1 left) 2020-06-12T02:38:14.708 INFO:tasks.rados.rados.0.smithi107.stdout:4270: done (0 left) 2020-06-12T02:38:14.709 INFO:tasks.rados.rados.0.smithi107.stdout:4271: delete oid 13 current snap is 0 2020-06-12T02:38:14.716 INFO:tasks.rados.rados.0.smithi107.stdout:4271: done (0 left) 2020-06-12T02:38:14.717 INFO:tasks.rados.rados.0.smithi107.stdout:4272: delete oid 15 current snap is 0 2020-06-12T02:38:14.717 INFO:tasks.rados.rados.0.smithi107.stdout:4272: done (0 left) 2020-06-12T02:38:14.717 INFO:tasks.rados.rados.0.smithi107.stdout:4273: delete oid 14 current snap is 0 2020-06-12T02:38:14.718 INFO:tasks.rados.rados.0.smithi107.stdout:4273: done (0 left) 2020-06-12T02:38:14.718 INFO:tasks.rados.rados.0.smithi107.stdout:4274: delete oid 12 current snap is 0 2020-06-12T02:38:14.721 INFO:tasks.rados.rados.0.smithi107.stdout:4274: done (0 left) 2020-06-12T02:38:14.721 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:14.721 INFO:tasks.rados.rados.0.smithi107.stdout:4275: seq_num 4274 ranges {0=786432,786432=761856,1548288=442368} 2020-06-12T02:38:14.726 INFO:tasks.rados.rados.0.smithi107.stdout:4275: writing smithi10729407-14 from 0 to 786432 tid 1 2020-06-12T02:38:14.731 INFO:tasks.rados.rados.0.smithi107.stdout:4275: writing smithi10729407-14 from 786432 to 1548288 tid 2 2020-06-12T02:38:14.734 INFO:tasks.rados.rados.0.smithi107.stdout:4275: writing smithi10729407-14 from 1548288 to 1990656 tid 3 2020-06-12T02:38:14.734 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:14.734 INFO:tasks.rados.rados.0.smithi107.stdout:4276: seq_num 4275 ranges {1794048=802816,2596864=647168,3244032=483328,3727360=73728} 2020-06-12T02:38:14.742 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:14.739+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:14.750 INFO:tasks.rados.rados.0.smithi107.stdout:4276: writing smithi10729407-4 from 1794048 to 2596864 tid 1 2020-06-12T02:38:14.755 INFO:tasks.rados.rados.0.smithi107.stdout:4276: writing smithi10729407-4 from 2596864 to 3244032 tid 2 2020-06-12T02:38:14.758 INFO:tasks.rados.rados.0.smithi107.stdout:4276: writing smithi10729407-4 from 3244032 to 3727360 tid 3 2020-06-12T02:38:14.759 INFO:tasks.rados.rados.0.smithi107.stdout:4276: writing smithi10729407-4 from 3727360 to 3801088 tid 4 2020-06-12T02:38:14.759 INFO:tasks.rados.rados.0.smithi107.stdout:4277: delete oid 7 current snap is 0 2020-06-12T02:38:14.760 INFO:tasks.rados.rados.0.smithi107.stdout:4275: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:14.760 INFO:tasks.rados.rados.0.smithi107.stdout:4275: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:14.760 INFO:tasks.rados.rados.0.smithi107.stdout:4275: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:14.760 INFO:tasks.rados.rados.0.smithi107.stdout:4275: finishing write tid 5 to smithi10729407-14 2020-06-12T02:38:14.760 INFO:tasks.rados.rados.0.smithi107.stdout:4275: finishing write tid 6 to smithi10729407-14 2020-06-12T02:38:14.761 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 760 (ObjNum 4274 snap 0 seq_num 4274) dirty exists 2020-06-12T02:38:14.761 INFO:tasks.rados.rados.0.smithi107.stdout:4275: left oid 14 (ObjNum 4274 snap 0 seq_num 4274) 2020-06-12T02:38:14.761 INFO:tasks.rados.rados.0.smithi107.stdout:4276: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:14.763 INFO:tasks.rados.rados.0.smithi107.stdout:4275: done (2 left) 2020-06-12T02:38:14.763 INFO:tasks.rados.rados.0.smithi107.stdout:4277: done (1 left) 2020-06-12T02:38:14.763 INFO:tasks.rados.rados.0.smithi107.stdout:4278: delete oid 3 current snap is 0 2020-06-12T02:38:14.766 INFO:tasks.rados.rados.0.smithi107.stdout:4278: done (1 left) 2020-06-12T02:38:14.771 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:38:14.771 INFO:tasks.rados.rados.0.smithi107.stdout:4279: seq_num 4278 ranges {0=450560,450560=696320,1146880=655360,1802240=253952} 2020-06-12T02:38:14.772 INFO:tasks.rados.rados.0.smithi107.stdout:4279: writing smithi10729407-13 from 0 to 450560 tid 1 2020-06-12T02:38:14.774 INFO:tasks.rados.rados.0.smithi107.stdout:4279: writing smithi10729407-13 from 450560 to 1146880 tid 2 2020-06-12T02:38:14.778 INFO:tasks.rados.rados.0.smithi107.stdout:4279: writing smithi10729407-13 from 1146880 to 1802240 tid 3 2020-06-12T02:38:14.781 INFO:tasks.rados.rados.0.smithi107.stdout:4279: writing smithi10729407-13 from 1802240 to 2056192 tid 4 2020-06-12T02:38:14.781 INFO:tasks.rados.rados.0.smithi107.stdout:4280: delete oid 11 current snap is 0 2020-06-12T02:38:14.781 INFO:tasks.rados.rados.0.smithi107.stdout:4276: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:14.781 INFO:tasks.rados.rados.0.smithi107.stdout:4276: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:14.781 INFO:tasks.rados.rados.0.smithi107.stdout:4276: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:14.782 INFO:tasks.rados.rados.0.smithi107.stdout:4279: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:14.782 INFO:tasks.rados.rados.0.smithi107.stdout:4276: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:14.782 INFO:tasks.rados.rados.0.smithi107.stdout:4276: finishing write tid 7 to smithi10729407-4 2020-06-12T02:38:14.782 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 779 (ObjNum 4275 snap 0 seq_num 4275) dirty exists 2020-06-12T02:38:14.782 INFO:tasks.rados.rados.0.smithi107.stdout:4276: left oid 4 (ObjNum 4275 snap 0 seq_num 4275) 2020-06-12T02:38:14.786 INFO:tasks.rados.rados.0.smithi107.stdout:4279: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:14.787 INFO:tasks.rados.rados.0.smithi107.stdout:4279: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:14.787 INFO:tasks.rados.rados.0.smithi107.stdout:4276: done (2 left) 2020-06-12T02:38:14.787 INFO:tasks.rados.rados.0.smithi107.stdout:4280: done (1 left) 2020-06-12T02:38:14.788 INFO:tasks.rados.rados.0.smithi107.stdout:4281: delete oid 4 current snap is 0 2020-06-12T02:38:14.789 INFO:tasks.rados.rados.0.smithi107.stdout:4279: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:14.790 INFO:tasks.rados.rados.0.smithi107.stdout:4281: done (1 left) 2020-06-12T02:38:14.790 INFO:tasks.rados.rados.0.smithi107.stdout:4282: delete oid 5 current snap is 0 2020-06-12T02:38:14.790 INFO:tasks.rados.rados.0.smithi107.stdout:4279: finishing write tid 6 to smithi10729407-13 2020-06-12T02:38:14.793 INFO:tasks.rados.rados.0.smithi107.stdout:4282: done (1 left) 2020-06-12T02:38:14.795 INFO:tasks.rados.rados.0.smithi107.stdout:4283: delete oid 1 current snap is 0 2020-06-12T02:38:14.795 INFO:tasks.rados.rados.0.smithi107.stdout:4279: finishing write tid 7 to smithi10729407-13 2020-06-12T02:38:14.795 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 818 (ObjNum 4278 snap 0 seq_num 4278) dirty exists 2020-06-12T02:38:14.796 INFO:tasks.rados.rados.0.smithi107.stdout:4279: left oid 13 (ObjNum 4278 snap 0 seq_num 4278) 2020-06-12T02:38:14.796 INFO:tasks.rados.rados.0.smithi107.stdout:4279: done (1 left) 2020-06-12T02:38:14.796 INFO:tasks.rados.rados.0.smithi107.stdout:4283: done (0 left) 2020-06-12T02:38:14.797 INFO:tasks.rados.rados.0.smithi107.stdout:4284: delete oid 4 current snap is 0 2020-06-12T02:38:14.800 INFO:tasks.rados.rados.0.smithi107.stdout:4284: done (0 left) 2020-06-12T02:38:14.800 INFO:tasks.rados.rados.0.smithi107.stdout:4285: delete oid 10 current snap is 0 2020-06-12T02:38:14.803 INFO:tasks.rados.rados.0.smithi107.stdout:4285: done (0 left) 2020-06-12T02:38:14.803 INFO:tasks.rados.rados.0.smithi107.stdout:4286: delete oid 5 current snap is 0 2020-06-12T02:38:14.804 INFO:tasks.rados.rados.0.smithi107.stdout:4286: done (0 left) 2020-06-12T02:38:14.804 INFO:tasks.rados.rados.0.smithi107.stdout:4287: delete oid 18 current snap is 0 2020-06-12T02:38:14.807 INFO:tasks.rados.rados.0.smithi107.stdout:4287: done (0 left) 2020-06-12T02:38:14.807 INFO:tasks.rados.rados.0.smithi107.stdout:4288: delete oid 11 current snap is 0 2020-06-12T02:38:14.812 INFO:tasks.rados.rados.0.smithi107.stdout:4288: done (0 left) 2020-06-12T02:38:14.812 INFO:tasks.rados.rados.0.smithi107.stdout:4289: delete oid 16 current snap is 0 2020-06-12T02:38:14.813 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:14.813 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:14.816 INFO:tasks.rados.rados.0.smithi107.stdout:4289: done (0 left) 2020-06-12T02:38:14.816 INFO:tasks.rados.rados.0.smithi107.stdout:4290: delete oid 2 current snap is 0 2020-06-12T02:38:14.817 INFO:tasks.rados.rados.0.smithi107.stdout:4290: done (0 left) 2020-06-12T02:38:14.817 INFO:tasks.rados.rados.0.smithi107.stdout:4291: delete oid 4 current snap is 0 2020-06-12T02:38:14.821 INFO:tasks.rados.rados.0.smithi107.stdout:4291: done (0 left) 2020-06-12T02:38:14.821 INFO:tasks.rados.rados.0.smithi107.stdout:4292: delete oid 4 current snap is 0 2020-06-12T02:38:14.822 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-06-12T02:38:14.823 INFO:tasks.rados.rados.0.smithi107.stdout:4292: done (0 left) 2020-06-12T02:38:14.824 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:38:14.824 INFO:tasks.rados.rados.0.smithi107.stdout:4293: seq_num 4292 ranges {0=688128,688128=778240,1466368=614400,2080768=212992} 2020-06-12T02:38:14.829 INFO:tasks.rados.rados.0.smithi107.stdout:4293: writing smithi10729407-1 from 0 to 688128 tid 1 2020-06-12T02:38:14.831 INFO:tasks.rados.rados.0.smithi107.stdout:4293: writing smithi10729407-1 from 688128 to 1466368 tid 2 2020-06-12T02:38:14.836 INFO:tasks.rados.rados.0.smithi107.stdout:4293: writing smithi10729407-1 from 1466368 to 2080768 tid 3 2020-06-12T02:38:14.837 INFO:tasks.rados.rados.0.smithi107.stdout:4293: writing smithi10729407-1 from 2080768 to 2293760 tid 4 2020-06-12T02:38:14.837 INFO:tasks.rados.rados.0.smithi107.stdout:4294: delete oid 17 current snap is 0 2020-06-12T02:38:14.840 INFO:tasks.rados.rados.0.smithi107.stdout:4294: done (1 left) 2020-06-12T02:38:14.841 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:14.841 INFO:tasks.rados.rados.0.smithi107.stdout:4295: seq_num 4294 ranges {0=720896,720896=548864,1269760=581632,1851392=475136} 2020-06-12T02:38:14.855 INFO:tasks.rados.rados.0.smithi107.stdout:4295: writing smithi10729407-5 from 0 to 720896 tid 1 2020-06-12T02:38:14.855 INFO:tasks.rados.rados.0.smithi107.stdout:4295: writing smithi10729407-5 from 720896 to 1269760 tid 2 2020-06-12T02:38:14.855 INFO:tasks.rados.rados.0.smithi107.stdout:4295: writing smithi10729407-5 from 1269760 to 1851392 tid 3 2020-06-12T02:38:14.855 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:14.852+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:14.856 INFO:tasks.rados.rados.0.smithi107.stdout:4295: writing smithi10729407-5 from 1851392 to 2326528 tid 4 2020-06-12T02:38:14.856 INFO:tasks.rados.rados.0.smithi107.stdout:4296: delete oid 20 current snap is 0 2020-06-12T02:38:14.857 INFO:tasks.rados.rados.0.smithi107.stdout:4293: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:14.857 INFO:tasks.rados.rados.0.smithi107.stdout:4293: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:14.857 INFO:tasks.rados.rados.0.smithi107.stdout:4293: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:14.857 INFO:tasks.rados.rados.0.smithi107.stdout:4293: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:14.857 INFO:tasks.rados.rados.0.smithi107.stdout:4293: finishing write tid 6 to smithi10729407-1 2020-06-12T02:38:14.858 INFO:tasks.rados.rados.0.smithi107.stdout:4293: finishing write tid 7 to smithi10729407-1 2020-06-12T02:38:14.858 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 806 (ObjNum 4292 snap 0 seq_num 4292) dirty exists 2020-06-12T02:38:14.858 INFO:tasks.rados.rados.0.smithi107.stdout:4293: left oid 1 (ObjNum 4292 snap 0 seq_num 4292) 2020-06-12T02:38:14.858 INFO:tasks.rados.rados.0.smithi107.stdout:4293: done (2 left) 2020-06-12T02:38:14.859 INFO:tasks.rados.rados.0.smithi107.stdout:4296: done (1 left) 2020-06-12T02:38:14.859 INFO:tasks.rados.rados.0.smithi107.stdout:4297: delete oid 3 current snap is 0 2020-06-12T02:38:14.859 INFO:tasks.rados.rados.0.smithi107.stdout:4295: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:14.860 INFO:tasks.rados.rados.0.smithi107.stdout:4297: done (1 left) 2020-06-12T02:38:14.860 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:38:14.860 INFO:tasks.rados.rados.0.smithi107.stdout:4298: seq_num 4297 ranges {0=770048,770048=647168,1417216=753664,2170880=204800} 2020-06-12T02:38:14.865 INFO:tasks.rados.rados.0.smithi107.stdout:4298: writing smithi10729407-7 from 0 to 770048 tid 1 2020-06-12T02:38:14.868 INFO:tasks.rados.rados.0.smithi107.stdout:4298: writing smithi10729407-7 from 770048 to 1417216 tid 2 2020-06-12T02:38:14.873 INFO:tasks.rados.rados.0.smithi107.stdout:4298: writing smithi10729407-7 from 1417216 to 2170880 tid 3 2020-06-12T02:38:14.874 INFO:tasks.rados.rados.0.smithi107.stdout:4298: writing smithi10729407-7 from 2170880 to 2375680 tid 4 2020-06-12T02:38:14.875 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:14.875 INFO:tasks.rados.rados.0.smithi107.stdout:4299: seq_num 4298 ranges {0=786432,786432=532480,1318912=557056,1875968=516096} 2020-06-12T02:38:14.879 INFO:tasks.rados.rados.0.smithi107.stdout:4299: writing smithi10729407-15 from 0 to 786432 tid 1 2020-06-12T02:38:14.882 INFO:tasks.rados.rados.0.smithi107.stdout:4299: writing smithi10729407-15 from 786432 to 1318912 tid 2 2020-06-12T02:38:14.886 INFO:tasks.rados.rados.0.smithi107.stdout:4299: writing smithi10729407-15 from 1318912 to 1875968 tid 3 2020-06-12T02:38:14.890 INFO:tasks.rados.rados.0.smithi107.stdout:4299: writing smithi10729407-15 from 1875968 to 2392064 tid 4 2020-06-12T02:38:14.890 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:14.890 INFO:tasks.rados.rados.0.smithi107.stdout:4300: seq_num 4299 ranges {0=802816,802816=417792,1220608=770048,1990656=417792} 2020-06-12T02:38:14.894 INFO:tasks.rados.rados.0.smithi107.stdout:4300: writing smithi10729407-11 from 0 to 802816 tid 1 2020-06-12T02:38:14.897 INFO:tasks.rados.rados.0.smithi107.stdout:4300: writing smithi10729407-11 from 802816 to 1220608 tid 2 2020-06-12T02:38:14.902 INFO:tasks.rados.rados.0.smithi107.stdout:4300: writing smithi10729407-11 from 1220608 to 1990656 tid 3 2020-06-12T02:38:14.905 INFO:tasks.rados.rados.0.smithi107.stdout:4300: writing smithi10729407-11 from 1990656 to 2408448 tid 4 2020-06-12T02:38:14.905 INFO:tasks.rados.rados.0.smithi107.stdout:4301: delete oid 12 current snap is 0 2020-06-12T02:38:14.905 INFO:tasks.rados.rados.0.smithi107.stdout:4295: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:14.905 INFO:tasks.rados.rados.0.smithi107.stdout:4295: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:14.906 INFO:tasks.rados.rados.0.smithi107.stdout:4295: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:14.906 INFO:tasks.rados.rados.0.smithi107.stdout:4295: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:14.906 INFO:tasks.rados.rados.0.smithi107.stdout:4295: finishing write tid 7 to smithi10729407-5 2020-06-12T02:38:14.906 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 431 (ObjNum 4294 snap 0 seq_num 4294) dirty exists 2020-06-12T02:38:14.906 INFO:tasks.rados.rados.0.smithi107.stdout:4295: left oid 5 (ObjNum 4294 snap 0 seq_num 4294) 2020-06-12T02:38:14.906 INFO:tasks.rados.rados.0.smithi107.stdout:4298: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:14.907 INFO:tasks.rados.rados.0.smithi107.stdout:4298: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:14.907 INFO:tasks.rados.rados.0.smithi107.stdout:4298: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:14.907 INFO:tasks.rados.rados.0.smithi107.stdout:4298: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:14.907 INFO:tasks.rados.rados.0.smithi107.stdout:4298: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:14.907 INFO:tasks.rados.rados.0.smithi107.stdout:4298: finishing write tid 7 to smithi10729407-7 2020-06-12T02:38:14.908 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 823 (ObjNum 4297 snap 0 seq_num 4297) dirty exists 2020-06-12T02:38:14.908 INFO:tasks.rados.rados.0.smithi107.stdout:4298: left oid 7 (ObjNum 4297 snap 0 seq_num 4297) 2020-06-12T02:38:14.908 INFO:tasks.rados.rados.0.smithi107.stdout:4299: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:14.908 INFO:tasks.rados.rados.0.smithi107.stdout:4299: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:14.908 INFO:tasks.rados.rados.0.smithi107.stdout:4299: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:14.908 INFO:tasks.rados.rados.0.smithi107.stdout:4299: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:14.909 INFO:tasks.rados.rados.0.smithi107.stdout:4299: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:14.909 INFO:tasks.rados.rados.0.smithi107.stdout:4299: finishing write tid 7 to smithi10729407-15 2020-06-12T02:38:14.910 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 741 (ObjNum 4298 snap 0 seq_num 4298) dirty exists 2020-06-12T02:38:14.910 INFO:tasks.rados.rados.0.smithi107.stdout:4299: left oid 15 (ObjNum 4298 snap 0 seq_num 4298) 2020-06-12T02:38:14.910 INFO:tasks.rados.rados.0.smithi107.stdout:4295: done (4 left) 2020-06-12T02:38:14.910 INFO:tasks.rados.rados.0.smithi107.stdout:4298: done (3 left) 2020-06-12T02:38:14.910 INFO:tasks.rados.rados.0.smithi107.stdout:4299: done (2 left) 2020-06-12T02:38:14.911 INFO:tasks.rados.rados.0.smithi107.stdout:4301: done (1 left) 2020-06-12T02:38:14.911 INFO:tasks.rados.rados.0.smithi107.stdout:4302: delete oid 16 current snap is 0 2020-06-12T02:38:14.911 INFO:tasks.rados.rados.0.smithi107.stdout:4300: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:14.911 INFO:tasks.rados.rados.0.smithi107.stdout:4300: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:14.912 INFO:tasks.rados.rados.0.smithi107.stdout:4302: done (1 left) 2020-06-12T02:38:14.912 INFO:tasks.rados.rados.0.smithi107.stdout:4303: delete oid 1 current snap is 0 2020-06-12T02:38:14.915 INFO:tasks.rados.rados.0.smithi107.stdout:4300: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:14.915 INFO:tasks.rados.rados.0.smithi107.stdout:4300: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:14.916 INFO:tasks.rados.rados.0.smithi107.stdout:4300: finishing write tid 6 to smithi10729407-11 2020-06-12T02:38:14.917 INFO:tasks.rados.rados.0.smithi107.stdout:4303: done (1 left) 2020-06-12T02:38:14.917 INFO:tasks.rados.rados.0.smithi107.stdout:4304: delete oid 10 current snap is 0 2020-06-12T02:38:14.919 INFO:tasks.rados.rados.0.smithi107.stdout:4304: done (1 left) 2020-06-12T02:38:14.919 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:14.920 INFO:tasks.rados.rados.0.smithi107.stdout:4305: seq_num 4304 ranges {0=90112} 2020-06-12T02:38:14.920 INFO:tasks.rados.rados.0.smithi107.stdout:4305: writing smithi10729407-8 from 0 to 90112 tid 1 2020-06-12T02:38:14.921 INFO:tasks.rados.rados.0.smithi107.stdout:4306: delete oid 9 current snap is 0 2020-06-12T02:38:14.921 INFO:tasks.rados.rados.0.smithi107.stdout:4300: finishing write tid 7 to smithi10729407-11 2020-06-12T02:38:14.921 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 767 (ObjNum 4299 snap 0 seq_num 4299) dirty exists 2020-06-12T02:38:14.921 INFO:tasks.rados.rados.0.smithi107.stdout:4300: left oid 11 (ObjNum 4299 snap 0 seq_num 4299) 2020-06-12T02:38:14.922 INFO:tasks.rados.rados.0.smithi107.stdout:4300: done (2 left) 2020-06-12T02:38:14.922 INFO:tasks.rados.rados.0.smithi107.stdout:4306: done (1 left) 2020-06-12T02:38:14.922 INFO:tasks.rados.rados.0.smithi107.stdout:4307: delete oid 17 current snap is 0 2020-06-12T02:38:14.923 INFO:tasks.rados.rados.0.smithi107.stdout:4305: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:14.926 INFO:tasks.rados.rados.0.smithi107.stdout:4307: done (1 left) 2020-06-12T02:38:14.926 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:14.926 INFO:tasks.rados.rados.0.smithi107.stdout:4308: seq_num 4307 ranges {0=139264} 2020-06-12T02:38:14.927 INFO:tasks.rados.rados.0.smithi107.stdout:4308: writing smithi10729407-9 from 0 to 139264 tid 1 2020-06-12T02:38:14.927 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:14.927 INFO:tasks.rados.rados.0.smithi107.stdout:4309: seq_num 4308 ranges {0=155648} 2020-06-12T02:38:14.928 INFO:tasks.rados.rados.0.smithi107.stdout:4309: writing smithi10729407-19 from 0 to 155648 tid 1 2020-06-12T02:38:14.928 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:14.928 INFO:tasks.rados.rados.0.smithi107.stdout:4310: seq_num 4309 ranges {0=172032} 2020-06-12T02:38:14.929 INFO:tasks.rados.rados.0.smithi107.stdout:4310: writing smithi10729407-12 from 0 to 172032 tid 1 2020-06-12T02:38:14.929 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:38:14.929 INFO:tasks.rados.rados.0.smithi107.stdout:4311: seq_num 4310 ranges {0=188416} 2020-06-12T02:38:14.930 INFO:tasks.rados.rados.0.smithi107.stdout:4311: writing smithi10729407-10 from 0 to 188416 tid 1 2020-06-12T02:38:14.931 INFO:tasks.rados.rados.0.smithi107.stdout:4312: delete oid 13 current snap is 0 2020-06-12T02:38:14.931 INFO:tasks.rados.rados.0.smithi107.stdout:4305: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:14.931 INFO:tasks.rados.rados.0.smithi107.stdout:4305: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:14.931 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 743 (ObjNum 4304 snap 0 seq_num 4304) dirty exists 2020-06-12T02:38:14.931 INFO:tasks.rados.rados.0.smithi107.stdout:4305: left oid 8 (ObjNum 4304 snap 0 seq_num 4304) 2020-06-12T02:38:14.932 INFO:tasks.rados.rados.0.smithi107.stdout:4308: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:14.934 INFO:tasks.rados.rados.0.smithi107.stdout:4309: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:14.935 INFO:tasks.rados.rados.0.smithi107.stdout:4311: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:14.937 INFO:tasks.rados.rados.0.smithi107.stdout:4305: done (5 left) 2020-06-12T02:38:14.937 INFO:tasks.rados.rados.0.smithi107.stdout:4312: done (4 left) 2020-06-12T02:38:14.938 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:38:14.938 INFO:tasks.rados.rados.0.smithi107.stdout:4313: seq_num 4312 ranges {2408448=221184} 2020-06-12T02:38:14.952 INFO:tasks.rados.rados.0.smithi107.stdout:4313: writing smithi10729407-11 from 2408448 to 2629632 tid 1 2020-06-12T02:38:14.953 INFO:tasks.rados.rados.0.smithi107.stdout:4314: delete oid 1 current snap is 0 2020-06-12T02:38:14.953 INFO:tasks.rados.rados.0.smithi107.stdout:4310: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:14.953 INFO:tasks.rados.rados.0.smithi107.stdout:4308: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:14.953 INFO:tasks.rados.rados.0.smithi107.stdout:4310: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:14.953 INFO:tasks.rados.rados.0.smithi107.stdout:4311: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:14.954 INFO:tasks.rados.rados.0.smithi107.stdout:4309: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:14.954 INFO:tasks.rados.rados.0.smithi107.stdout:4310: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:14.954 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 726 (ObjNum 4309 snap 0 seq_num 4309) dirty exists 2020-06-12T02:38:14.954 INFO:tasks.rados.rados.0.smithi107.stdout:4310: left oid 12 (ObjNum 4309 snap 0 seq_num 4309) 2020-06-12T02:38:14.954 INFO:tasks.rados.rados.0.smithi107.stdout:4309: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:14.955 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 727 (ObjNum 4308 snap 0 seq_num 4308) dirty exists 2020-06-12T02:38:14.955 INFO:tasks.rados.rados.0.smithi107.stdout:4309: left oid 19 (ObjNum 4308 snap 0 seq_num 4308) 2020-06-12T02:38:14.955 INFO:tasks.rados.rados.0.smithi107.stdout:4308: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:14.955 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 768 (ObjNum 4307 snap 0 seq_num 4307) dirty exists 2020-06-12T02:38:14.955 INFO:tasks.rados.rados.0.smithi107.stdout:4308: left oid 9 (ObjNum 4307 snap 0 seq_num 4307) 2020-06-12T02:38:14.955 INFO:tasks.rados.rados.0.smithi107.stdout:4311: finishing write tid 4 to smithi10729407-10 2020-06-12T02:38:14.956 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 430 (ObjNum 4310 snap 0 seq_num 4310) dirty exists 2020-06-12T02:38:14.956 INFO:tasks.rados.rados.0.smithi107.stdout:4311: left oid 10 (ObjNum 4310 snap 0 seq_num 4310) 2020-06-12T02:38:14.957 INFO:tasks.rados.rados.0.smithi107.stdout:4308: done (5 left) 2020-06-12T02:38:14.957 INFO:tasks.rados.rados.0.smithi107.stdout:4309: done (4 left) 2020-06-12T02:38:14.957 INFO:tasks.rados.rados.0.smithi107.stdout:4310: done (3 left) 2020-06-12T02:38:14.958 INFO:tasks.rados.rados.0.smithi107.stdout:4311: done (2 left) 2020-06-12T02:38:14.958 INFO:tasks.rados.rados.0.smithi107.stdout:4314: done (1 left) 2020-06-12T02:38:14.958 INFO:tasks.rados.rados.0.smithi107.stdout:4315: delete oid 4 current snap is 0 2020-06-12T02:38:14.958 INFO:tasks.rados.rados.0.smithi107.stdout:4313: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:14.958 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:38:14.955+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:38:14.964 INFO:tasks.rados.rados.0.smithi107.stdout:4315: done (1 left) 2020-06-12T02:38:14.965 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:14.976 INFO:tasks.rados.rados.0.smithi107.stdout:4316: seq_num 4315 ranges {0=270336} 2020-06-12T02:38:14.976 INFO:tasks.rados.rados.0.smithi107.stdout:4316: writing smithi10729407-3 from 0 to 270336 tid 1 2020-06-12T02:38:14.977 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:14.977 INFO:tasks.rados.rados.0.smithi107.stdout:4313: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:14.977 INFO:tasks.rados.rados.0.smithi107.stdout:4317: seq_num 4316 ranges {0=286720} 2020-06-12T02:38:14.977 INFO:tasks.rados.rados.0.smithi107.stdout:4317: writing smithi10729407-18 from 0 to 286720 tid 1 2020-06-12T02:38:14.977 INFO:tasks.rados.rados.0.smithi107.stdout:4318: delete oid 2 current snap is 0 2020-06-12T02:38:14.978 INFO:tasks.rados.rados.0.smithi107.stdout:4313: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:14.978 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 769 (ObjNum 4312 snap 0 seq_num 4312) dirty exists 2020-06-12T02:38:14.978 INFO:tasks.rados.rados.0.smithi107.stdout:4313: left oid 11 (ObjNum 4312 snap 0 seq_num 4312) 2020-06-12T02:38:14.980 INFO:tasks.rados.rados.0.smithi107.stdout:4316: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:14.980 INFO:tasks.rados.rados.0.smithi107.stdout:4313: done (3 left) 2020-06-12T02:38:14.980 INFO:tasks.rados.rados.0.smithi107.stdout:4318: done (2 left) 2020-06-12T02:38:14.981 INFO:tasks.rados.rados.0.smithi107.stdout:4319: delete oid 13 current snap is 0 2020-06-12T02:38:14.981 INFO:tasks.rados.rados.0.smithi107.stdout:4317: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:14.981 INFO:tasks.rados.rados.0.smithi107.stdout:4317: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:14.981 INFO:tasks.rados.rados.0.smithi107.stdout:4319: done (2 left) 2020-06-12T02:38:14.981 INFO:tasks.rados.rados.0.smithi107.stdout:4320: delete oid 14 current snap is 0 2020-06-12T02:38:14.982 INFO:tasks.rados.rados.0.smithi107.stdout:4316: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:14.983 INFO:tasks.rados.rados.0.smithi107.stdout:4317: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:14.983 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 770 (ObjNum 4316 snap 0 seq_num 4316) dirty exists 2020-06-12T02:38:14.983 INFO:tasks.rados.rados.0.smithi107.stdout:4317: left oid 18 (ObjNum 4316 snap 0 seq_num 4316) 2020-06-12T02:38:14.983 INFO:tasks.rados.rados.0.smithi107.stdout:4316: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:14.983 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 771 (ObjNum 4315 snap 0 seq_num 4315) dirty exists 2020-06-12T02:38:14.983 INFO:tasks.rados.rados.0.smithi107.stdout:4316: left oid 3 (ObjNum 4315 snap 0 seq_num 4315) 2020-06-12T02:38:14.984 INFO:tasks.rados.rados.0.smithi107.stdout:4316: done (2 left) 2020-06-12T02:38:14.984 INFO:tasks.rados.rados.0.smithi107.stdout:4317: done (1 left) 2020-06-12T02:38:14.984 INFO:tasks.rados.rados.0.smithi107.stdout:4320: done (0 left) 2020-06-12T02:38:14.984 INFO:tasks.rados.rados.0.smithi107.stdout:4321: delete oid 6 current snap is 0 2020-06-12T02:38:14.984 INFO:tasks.rados.rados.0.smithi107.stdout:4321: done (0 left) 2020-06-12T02:38:14.985 INFO:tasks.rados.rados.0.smithi107.stdout:4322: delete oid 8 current snap is 0 2020-06-12T02:38:14.985 INFO:tasks.rados.rados.0.smithi107.stdout:4322: done (0 left) 2020-06-12T02:38:14.985 INFO:tasks.rados.rados.0.smithi107.stdout:4323: delete oid 12 current snap is 0 2020-06-12T02:38:14.986 INFO:tasks.rados.rados.0.smithi107.stdout:4323: done (0 left) 2020-06-12T02:38:14.986 INFO:tasks.rados.rados.0.smithi107.stdout:4324: delete oid 5 current snap is 0 2020-06-12T02:38:14.986 INFO:tasks.rados.rados.0.smithi107.stdout:4324: done (0 left) 2020-06-12T02:38:14.986 INFO:tasks.rados.rados.0.smithi107.stdout:4325: delete oid 19 current snap is 0 2020-06-12T02:38:14.987 INFO:tasks.rados.rados.0.smithi107.stdout:4325: done (0 left) 2020-06-12T02:38:14.987 INFO:tasks.rados.rados.0.smithi107.stdout:4326: delete oid 6 current snap is 0 2020-06-12T02:38:14.987 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:14.987 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:14.988 INFO:tasks.rados.rados.0.smithi107.stdout:4326: done (0 left) 2020-06-12T02:38:14.988 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:38:14.989 INFO:tasks.rados.rados.0.smithi107.stdout:4327: seq_num 4326 ranges {2629632=458752} 2020-06-12T02:38:14.995 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-06-12T02:38:15.005 INFO:tasks.rados.rados.0.smithi107.stdout:4327: writing smithi10729407-11 from 2629632 to 3088384 tid 1 2020-06-12T02:38:15.006 INFO:tasks.rados.rados.0.smithi107.stdout:4328: delete oid 14 current snap is 0 2020-06-12T02:38:15.012 INFO:tasks.rados.rados.0.smithi107.stdout:4327: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:15.012 INFO:tasks.rados.rados.0.smithi107.stdout:4328: done (1 left) 2020-06-12T02:38:15.012 INFO:tasks.rados.rados.0.smithi107.stdout:4329: delete oid 20 current snap is 0 2020-06-12T02:38:15.017 INFO:tasks.rados.rados.0.smithi107.stdout:4329: done (1 left) 2020-06-12T02:38:15.017 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:15.017 INFO:tasks.rados.rados.0.smithi107.stdout:4330: seq_num 4329 ranges {0=507904} 2020-06-12T02:38:15.020 INFO:tasks.rados.rados.0.smithi107.stdout:4330: writing smithi10729407-1 from 0 to 507904 tid 1 2020-06-12T02:38:15.020 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:15.020 INFO:tasks.rados.rados.0.smithi107.stdout:4331: seq_num 4330 ranges {0=524288} 2020-06-12T02:38:15.024 INFO:tasks.rados.rados.0.smithi107.stdout:4331: writing smithi10729407-12 from 0 to 524288 tid 1 2020-06-12T02:38:15.024 INFO:tasks.rados.rados.0.smithi107.stdout:4332: delete oid 15 current snap is 0 2020-06-12T02:38:15.024 INFO:tasks.rados.rados.0.smithi107.stdout:4327: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:15.025 INFO:tasks.rados.rados.0.smithi107.stdout:4327: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:15.025 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 773 (ObjNum 4326 snap 0 seq_num 4326) dirty exists 2020-06-12T02:38:15.025 INFO:tasks.rados.rados.0.smithi107.stdout:4327: left oid 11 (ObjNum 4326 snap 0 seq_num 4326) 2020-06-12T02:38:15.026 INFO:tasks.rados.rados.0.smithi107.stdout:4327: done (3 left) 2020-06-12T02:38:15.027 INFO:tasks.rados.rados.0.smithi107.stdout:4332: done (2 left) 2020-06-12T02:38:15.027 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:38:15.027 INFO:tasks.rados.rados.0.smithi107.stdout:4333: seq_num 4332 ranges {188416=557056} 2020-06-12T02:38:15.031 INFO:tasks.rados.rados.0.smithi107.stdout:4333: writing smithi10729407-10 from 188416 to 745472 tid 1 2020-06-12T02:38:15.031 INFO:tasks.rados.rados.0.smithi107.stdout:4334: delete oid 14 current snap is 0 2020-06-12T02:38:15.031 INFO:tasks.rados.rados.0.smithi107.stdout:4330: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:15.032 INFO:tasks.rados.rados.0.smithi107.stdout:4330: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:15.034 INFO:tasks.rados.rados.0.smithi107.stdout:4331: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:15.034 INFO:tasks.rados.rados.0.smithi107.stdout:4330: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:15.034 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 812 (ObjNum 4329 snap 0 seq_num 4329) dirty exists 2020-06-12T02:38:15.035 INFO:tasks.rados.rados.0.smithi107.stdout:4330: left oid 1 (ObjNum 4329 snap 0 seq_num 4329) 2020-06-12T02:38:15.040 INFO:tasks.rados.rados.0.smithi107.stdout:4330: done (3 left) 2020-06-12T02:38:15.040 INFO:tasks.rados.rados.0.smithi107.stdout:4334: done (2 left) 2020-06-12T02:38:15.040 INFO:tasks.rados.rados.0.smithi107.stdout:4335: delete oid 16 current snap is 0 2020-06-12T02:38:15.041 INFO:tasks.rados.rados.0.smithi107.stdout:4331: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:15.042 INFO:tasks.rados.rados.0.smithi107.stdout:4335: done (2 left) 2020-06-12T02:38:15.042 INFO:tasks.rados.rados.0.smithi107.stdout:4336: delete oid 2 current snap is 0 2020-06-12T02:38:15.042 INFO:tasks.rados.rados.0.smithi107.stdout:4333: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:15.042 INFO:tasks.rados.rados.0.smithi107.stdout:4331: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:15.043 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 731 (ObjNum 4330 snap 0 seq_num 4330) dirty exists 2020-06-12T02:38:15.043 INFO:tasks.rados.rados.0.smithi107.stdout:4331: left oid 12 (ObjNum 4330 snap 0 seq_num 4330) 2020-06-12T02:38:15.044 INFO:tasks.rados.rados.0.smithi107.stdout:4333: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:15.045 INFO:tasks.rados.rados.0.smithi107.stdout:4331: done (2 left) 2020-06-12T02:38:15.045 INFO:tasks.rados.rados.0.smithi107.stdout:4336: done (1 left) 2020-06-12T02:38:15.045 INFO:tasks.rados.rados.0.smithi107.stdout:4337: delete oid 5 current snap is 0 2020-06-12T02:38:15.048 INFO:tasks.rados.rados.0.smithi107.stdout:4333: finishing write tid 4 to smithi10729407-10 2020-06-12T02:38:15.048 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 432 (ObjNum 4332 snap 0 seq_num 4332) dirty exists 2020-06-12T02:38:15.048 INFO:tasks.rados.rados.0.smithi107.stdout:4333: left oid 10 (ObjNum 4332 snap 0 seq_num 4332) 2020-06-12T02:38:15.048 INFO:tasks.rados.rados.0.smithi107.stdout:4333: done (1 left) 2020-06-12T02:38:15.049 INFO:tasks.rados.rados.0.smithi107.stdout:4337: done (0 left) 2020-06-12T02:38:15.049 INFO:tasks.rados.rados.0.smithi107.stdout:4338: delete oid 17 current snap is 0 2020-06-12T02:38:15.050 INFO:tasks.rados.rados.0.smithi107.stdout:4338: done (0 left) 2020-06-12T02:38:15.050 INFO:tasks.rados.rados.0.smithi107.stdout:4339: delete oid 2 current snap is 0 2020-06-12T02:38:15.058 INFO:tasks.rados.rados.0.smithi107.stdout:4339: done (0 left) 2020-06-12T02:38:15.058 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:15.058 INFO:tasks.rados.rados.0.smithi107.stdout:4340: seq_num 4339 ranges {0=679936} 2020-06-12T02:38:15.058 INFO:tasks.rados.rados.0.smithi107.stdout:4340: writing smithi10729407-14 from 0 to 679936 tid 1 2020-06-12T02:38:15.059 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:15.059 INFO:tasks.rados.rados.0.smithi107.stdout:4341: seq_num 4340 ranges {0=696320} 2020-06-12T02:38:15.059 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:15.055+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:15.061 INFO:tasks.rados.rados.0.smithi107.stdout:4341: writing smithi10729407-16 from 0 to 696320 tid 1 2020-06-12T02:38:15.061 INFO:tasks.rados.rados.0.smithi107.stdout:4342: delete oid 20 current snap is 0 2020-06-12T02:38:15.063 INFO:tasks.rados.rados.0.smithi107.stdout:4342: done (2 left) 2020-06-12T02:38:15.063 INFO:tasks.rados.rados.0.smithi107.stdout:4343: delete oid 4 current snap is 0 2020-06-12T02:38:15.066 INFO:tasks.rados.rados.0.smithi107.stdout:4343: done (2 left) 2020-06-12T02:38:15.066 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:38:15.066 INFO:tasks.rados.rados.0.smithi107.stdout:4344: seq_num 4343 ranges {3088384=745472} 2020-06-12T02:38:15.089 INFO:tasks.rados.rados.0.smithi107.stdout:4344: writing smithi10729407-11 from 3088384 to 3833856 tid 1 2020-06-12T02:38:15.089 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:15.090 INFO:tasks.rados.rados.0.smithi107.stdout:4345: seq_num 4344 ranges {524288=761856} 2020-06-12T02:38:15.097 INFO:tasks.rados.rados.0.smithi107.stdout:4345: writing smithi10729407-12 from 524288 to 1286144 tid 1 2020-06-12T02:38:15.098 INFO:tasks.rados.rados.0.smithi107.stdout:4346: delete oid 20 current snap is 0 2020-06-12T02:38:15.098 INFO:tasks.rados.rados.0.smithi107.stdout:4340: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:15.098 INFO:tasks.rados.rados.0.smithi107.stdout:4340: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:15.098 INFO:tasks.rados.rados.0.smithi107.stdout:4341: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:15.098 INFO:tasks.rados.rados.0.smithi107.stdout:4340: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:15.099 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 776 (ObjNum 4339 snap 0 seq_num 4339) dirty exists 2020-06-12T02:38:15.099 INFO:tasks.rados.rados.0.smithi107.stdout:4340: left oid 14 (ObjNum 4339 snap 0 seq_num 4339) 2020-06-12T02:38:15.099 INFO:tasks.rados.rados.0.smithi107.stdout:4341: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:15.099 INFO:tasks.rados.rados.0.smithi107.stdout:4341: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:15.099 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 774 (ObjNum 4340 snap 0 seq_num 4340) dirty exists 2020-06-12T02:38:15.100 INFO:tasks.rados.rados.0.smithi107.stdout:4341: left oid 16 (ObjNum 4340 snap 0 seq_num 4340) 2020-06-12T02:38:15.100 INFO:tasks.rados.rados.0.smithi107.stdout:4344: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:15.100 INFO:tasks.rados.rados.0.smithi107.stdout:4344: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:15.101 INFO:tasks.rados.rados.0.smithi107.stdout:4344: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:15.101 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 778 (ObjNum 4343 snap 0 seq_num 4343) dirty exists 2020-06-12T02:38:15.101 INFO:tasks.rados.rados.0.smithi107.stdout:4344: left oid 11 (ObjNum 4343 snap 0 seq_num 4343) 2020-06-12T02:38:15.102 INFO:tasks.rados.rados.0.smithi107.stdout:4340: done (4 left) 2020-06-12T02:38:15.102 INFO:tasks.rados.rados.0.smithi107.stdout:4341: done (3 left) 2020-06-12T02:38:15.103 INFO:tasks.rados.rados.0.smithi107.stdout:4344: done (2 left) 2020-06-12T02:38:15.103 INFO:tasks.rados.rados.0.smithi107.stdout:4346: done (1 left) 2020-06-12T02:38:15.103 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:15.103 INFO:tasks.rados.rados.0.smithi107.stdout:4347: seq_num 4346 ranges {286720=794624} 2020-06-12T02:38:15.108 INFO:tasks.rados.rados.0.smithi107.stdout:4347: writing smithi10729407-18 from 286720 to 1081344 tid 1 2020-06-12T02:38:15.108 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:15.109 INFO:tasks.rados.rados.0.smithi107.stdout:4348: seq_num 4347 ranges {0=409600,409600=401408} 2020-06-12T02:38:15.111 INFO:tasks.rados.rados.0.smithi107.stdout:4348: writing smithi10729407-15 from 0 to 409600 tid 1 2020-06-12T02:38:15.114 INFO:tasks.rados.rados.0.smithi107.stdout:4348: writing smithi10729407-15 from 409600 to 811008 tid 2 2020-06-12T02:38:15.114 INFO:tasks.rados.rados.0.smithi107.stdout:4349: delete oid 7 current snap is 0 2020-06-12T02:38:15.114 INFO:tasks.rados.rados.0.smithi107.stdout:4345: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:15.114 INFO:tasks.rados.rados.0.smithi107.stdout:4345: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:15.114 INFO:tasks.rados.rados.0.smithi107.stdout:4345: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:15.115 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 733 (ObjNum 4344 snap 0 seq_num 4344) dirty exists 2020-06-12T02:38:15.115 INFO:tasks.rados.rados.0.smithi107.stdout:4345: left oid 12 (ObjNum 4344 snap 0 seq_num 4344) 2020-06-12T02:38:15.132 INFO:tasks.rados.rados.0.smithi107.stdout:4347: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:15.132 INFO:tasks.rados.rados.0.smithi107.stdout:4347: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:15.132 INFO:tasks.rados.rados.0.smithi107.stdout:4348: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:15.139 INFO:tasks.rados.rados.0.smithi107.stdout:4345: done (3 left) 2020-06-12T02:38:15.139 INFO:tasks.rados.rados.0.smithi107.stdout:4349: done (2 left) 2020-06-12T02:38:15.139 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:15.140 INFO:tasks.rados.rados.0.smithi107.stdout:4350: seq_num 4349 ranges {139264=442368,581632=401408} 2020-06-12T02:38:15.140 INFO:tasks.rados.rados.0.smithi107.stdout:4350: writing smithi10729407-9 from 139264 to 581632 tid 1 2020-06-12T02:38:15.140 INFO:tasks.rados.rados.0.smithi107.stdout:4350: writing smithi10729407-9 from 581632 to 983040 tid 2 2020-06-12T02:38:15.140 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:15.140 INFO:tasks.rados.rados.0.smithi107.stdout:4351: seq_num 4350 ranges {0=458752,458752=401408} 2020-06-12T02:38:15.141 INFO:tasks.rados.rados.0.smithi107.stdout:4351: writing smithi10729407-4 from 0 to 458752 tid 1 2020-06-12T02:38:15.141 INFO:tasks.rados.rados.0.smithi107.stdout:4351: writing smithi10729407-4 from 458752 to 860160 tid 2 2020-06-12T02:38:15.141 INFO:tasks.rados.rados.0.smithi107.stdout:4352: delete oid 8 current snap is 0 2020-06-12T02:38:15.141 INFO:tasks.rados.rados.0.smithi107.stdout:4347: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:15.142 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 772 (ObjNum 4346 snap 0 seq_num 4346) dirty exists 2020-06-12T02:38:15.142 INFO:tasks.rados.rados.0.smithi107.stdout:4347: left oid 18 (ObjNum 4346 snap 0 seq_num 4346) 2020-06-12T02:38:15.142 INFO:tasks.rados.rados.0.smithi107.stdout:4348: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:15.143 INFO:tasks.rados.rados.0.smithi107.stdout:4348: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:15.143 INFO:tasks.rados.rados.0.smithi107.stdout:4348: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:15.143 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 748 (ObjNum 4347 snap 0 seq_num 4347) dirty exists 2020-06-12T02:38:15.143 INFO:tasks.rados.rados.0.smithi107.stdout:4348: left oid 15 (ObjNum 4347 snap 0 seq_num 4347) 2020-06-12T02:38:15.143 INFO:tasks.rados.rados.0.smithi107.stdout:4350: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:15.145 INFO:tasks.rados.rados.0.smithi107.stdout:4347: done (4 left) 2020-06-12T02:38:15.145 INFO:tasks.rados.rados.0.smithi107.stdout:4348: done (3 left) 2020-06-12T02:38:15.145 INFO:tasks.rados.rados.0.smithi107.stdout:4352: done (2 left) 2020-06-12T02:38:15.145 INFO:tasks.rados.rados.0.smithi107.stdout:4353: delete oid 19 current snap is 0 2020-06-12T02:38:15.146 INFO:tasks.rados.rados.0.smithi107.stdout:4350: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:15.146 INFO:tasks.rados.rados.0.smithi107.stdout:4350: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:15.146 INFO:tasks.rados.rados.0.smithi107.stdout:4353: done (2 left) 2020-06-12T02:38:15.146 INFO:tasks.rados.rados.0.smithi107.stdout:4354: delete oid 16 current snap is 0 2020-06-12T02:38:15.146 INFO:tasks.rados.rados.0.smithi107.stdout:4351: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:15.146 INFO:tasks.rados.rados.0.smithi107.stdout:4350: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:15.147 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 775 (ObjNum 4349 snap 0 seq_num 4349) dirty exists 2020-06-12T02:38:15.147 INFO:tasks.rados.rados.0.smithi107.stdout:4350: left oid 9 (ObjNum 4349 snap 0 seq_num 4349) 2020-06-12T02:38:15.147 INFO:tasks.rados.rados.0.smithi107.stdout:4350: done (2 left) 2020-06-12T02:38:15.147 INFO:tasks.rados.rados.0.smithi107.stdout:4354: done (1 left) 2020-06-12T02:38:15.147 INFO:tasks.rados.rados.0.smithi107.stdout:4355: delete oid 16 current snap is 0 2020-06-12T02:38:15.148 INFO:tasks.rados.rados.0.smithi107.stdout:4351: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:15.148 INFO:tasks.rados.rados.0.smithi107.stdout:4351: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:15.148 INFO:tasks.rados.rados.0.smithi107.stdout:4351: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:15.149 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 791 (ObjNum 4350 snap 0 seq_num 4350) dirty exists 2020-06-12T02:38:15.149 INFO:tasks.rados.rados.0.smithi107.stdout:4351: left oid 4 (ObjNum 4350 snap 0 seq_num 4350) 2020-06-12T02:38:15.149 INFO:tasks.rados.rados.0.smithi107.stdout:4351: done (1 left) 2020-06-12T02:38:15.149 INFO:tasks.rados.rados.0.smithi107.stdout:4355: done (0 left) 2020-06-12T02:38:15.149 INFO:tasks.rados.rados.0.smithi107.stdout:4356: delete oid 20 current snap is 0 2020-06-12T02:38:15.150 INFO:tasks.rados.rados.0.smithi107.stdout:4356: done (0 left) 2020-06-12T02:38:15.150 INFO:tasks.rados.rados.0.smithi107.stdout:4357: delete oid 12 current snap is 0 2020-06-12T02:38:15.152 INFO:tasks.rados.rados.0.smithi107.stdout:4357: done (0 left) 2020-06-12T02:38:15.152 INFO:tasks.rados.rados.0.smithi107.stdout:4358: delete oid 16 current snap is 0 2020-06-12T02:38:15.158 INFO:tasks.rados.rados.0.smithi107.stdout:4358: done (0 left) 2020-06-12T02:38:15.158 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:38:15.158 INFO:tasks.rados.rados.0.smithi107.stdout:4359: seq_num 4358 ranges {0=598016,598016=401408} 2020-06-12T02:38:15.159 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:15.155+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:15.159 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:15.160 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:15.171 INFO:tasks.rados.rados.0.smithi107.stdout:4359: writing smithi10729407-16 from 0 to 598016 tid 1 2020-06-12T02:38:15.172 INFO:tasks.rados.rados.0.smithi107.stdout:4359: writing smithi10729407-16 from 598016 to 999424 tid 2 2020-06-12T02:38:15.172 INFO:tasks.rados.rados.0.smithi107.stdout:4360: delete oid 18 current snap is 0 2020-06-12T02:38:15.172 INFO:tasks.rados.rados.0.smithi107.stdout:4360: done (1 left) 2020-06-12T02:38:15.172 INFO:tasks.rados.rados.0.smithi107.stdout:4361: delete oid 6 current snap is 0 2020-06-12T02:38:15.172 INFO:tasks.rados.rados.0.smithi107.stdout:4359: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:15.173 INFO:tasks.rados.rados.0.smithi107.stdout:4359: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:15.173 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-06-12T02:38:15.174 INFO:tasks.rados.rados.0.smithi107.stdout:4361: done (1 left) 2020-06-12T02:38:15.174 INFO:tasks.rados.rados.0.smithi107.stdout:4362: delete oid 19 current snap is 0 2020-06-12T02:38:15.181 INFO:tasks.rados.rados.0.smithi107.stdout:4362: done (1 left) 2020-06-12T02:38:15.181 INFO:tasks.rados.rados.0.smithi107.stdout:4363: delete oid 9 current snap is 0 2020-06-12T02:38:15.181 INFO:tasks.rados.rados.0.smithi107.stdout:4359: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:15.181 INFO:tasks.rados.rados.0.smithi107.stdout:4359: finishing write tid 5 to smithi10729407-16 2020-06-12T02:38:15.182 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 780 (ObjNum 4358 snap 0 seq_num 4358) dirty exists 2020-06-12T02:38:15.182 INFO:tasks.rados.rados.0.smithi107.stdout:4359: left oid 16 (ObjNum 4358 snap 0 seq_num 4358) 2020-06-12T02:38:15.182 INFO:tasks.rados.rados.0.smithi107.stdout:4359: done (1 left) 2020-06-12T02:38:15.182 INFO:tasks.rados.rados.0.smithi107.stdout:4363: done (0 left) 2020-06-12T02:38:15.183 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:15.183 INFO:tasks.rados.rados.0.smithi107.stdout:4364: seq_num 4363 ranges {0=679936,679936=401408} 2020-06-12T02:38:15.185 INFO:tasks.rados.rados.0.smithi107.stdout:4364: writing smithi10729407-18 from 0 to 679936 tid 1 2020-06-12T02:38:15.187 INFO:tasks.rados.rados.0.smithi107.stdout:4364: writing smithi10729407-18 from 679936 to 1081344 tid 2 2020-06-12T02:38:15.188 INFO:tasks.rados.rados.0.smithi107.stdout:4365: delete oid 13 current snap is 0 2020-06-12T02:38:15.192 INFO:tasks.rados.rados.0.smithi107.stdout:4365: done (1 left) 2020-06-12T02:38:15.192 INFO:tasks.rados.rados.0.smithi107.stdout:4366: delete oid 10 current snap is 0 2020-06-12T02:38:15.194 INFO:tasks.rados.rados.0.smithi107.stdout:4364: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:15.195 INFO:tasks.rados.rados.0.smithi107.stdout:4364: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:15.196 INFO:tasks.rados.rados.0.smithi107.stdout:4366: done (1 left) 2020-06-12T02:38:15.196 INFO:tasks.rados.rados.0.smithi107.stdout:4367: delete oid 14 current snap is 0 2020-06-12T02:38:15.196 INFO:tasks.rados.rados.0.smithi107.stdout:4364: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:15.199 INFO:tasks.rados.rados.0.smithi107.stdout:4364: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:15.199 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 780 (ObjNum 4363 snap 0 seq_num 4363) dirty exists 2020-06-12T02:38:15.199 INFO:tasks.rados.rados.0.smithi107.stdout:4364: left oid 18 (ObjNum 4363 snap 0 seq_num 4363) 2020-06-12T02:38:15.199 INFO:tasks.rados.rados.0.smithi107.stdout:4364: done (1 left) 2020-06-12T02:38:15.200 INFO:tasks.rados.rados.0.smithi107.stdout:4367: done (0 left) 2020-06-12T02:38:15.200 INFO:tasks.rados.rados.0.smithi107.stdout:4368: delete oid 4 current snap is 0 2020-06-12T02:38:15.201 INFO:tasks.rados.rados.0.smithi107.stdout:4368: done (0 left) 2020-06-12T02:38:15.201 INFO:tasks.rados.rados.0.smithi107.stdout:4369: delete oid 12 current snap is 0 2020-06-12T02:38:15.203 INFO:tasks.rados.rados.0.smithi107.stdout:4369: done (0 left) 2020-06-12T02:38:15.203 INFO:tasks.rados.rados.0.smithi107.stdout:4370: delete oid 7 current snap is 0 2020-06-12T02:38:15.206 INFO:tasks.rados.rados.0.smithi107.stdout:4370: done (0 left) 2020-06-12T02:38:15.206 INFO:tasks.rados.rados.0.smithi107.stdout:4371: delete oid 6 current snap is 0 2020-06-12T02:38:15.208 INFO:tasks.rados.rados.0.smithi107.stdout:4371: done (0 left) 2020-06-12T02:38:15.208 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:38:15.208 INFO:tasks.rados.rados.0.smithi107.stdout:4372: seq_num 4371 ranges {0=409600,409600=540672,950272=262144} 2020-06-12T02:38:15.211 INFO:tasks.rados.rados.0.smithi107.stdout:4372: writing smithi10729407-5 from 0 to 409600 tid 1 2020-06-12T02:38:15.215 INFO:tasks.rados.rados.0.smithi107.stdout:4372: writing smithi10729407-5 from 409600 to 950272 tid 2 2020-06-12T02:38:15.217 INFO:tasks.rados.rados.0.smithi107.stdout:4372: writing smithi10729407-5 from 950272 to 1212416 tid 3 2020-06-12T02:38:15.217 INFO:tasks.rados.rados.0.smithi107.stdout:4373: delete oid 10 current snap is 0 2020-06-12T02:38:15.217 INFO:tasks.rados.rados.0.smithi107.stdout:4372: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:15.219 INFO:tasks.rados.rados.0.smithi107.stdout:4373: done (1 left) 2020-06-12T02:38:15.219 INFO:tasks.rados.rados.0.smithi107.stdout:4374: delete oid 14 current snap is 0 2020-06-12T02:38:15.222 INFO:tasks.rados.rados.0.smithi107.stdout:4372: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:15.222 INFO:tasks.rados.rados.0.smithi107.stdout:4372: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:15.222 INFO:tasks.rados.rados.0.smithi107.stdout:4374: done (1 left) 2020-06-12T02:38:15.222 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:15.223 INFO:tasks.rados.rados.0.smithi107.stdout:4375: seq_num 4374 ranges {0=458752,458752=638976,1097728=163840} 2020-06-12T02:38:15.224 INFO:tasks.rados.rados.0.smithi107.stdout:4375: writing smithi10729407-4 from 0 to 458752 tid 1 2020-06-12T02:38:15.228 INFO:tasks.rados.rados.0.smithi107.stdout:4375: writing smithi10729407-4 from 458752 to 1097728 tid 2 2020-06-12T02:38:15.229 INFO:tasks.rados.rados.0.smithi107.stdout:4375: writing smithi10729407-4 from 1097728 to 1261568 tid 3 2020-06-12T02:38:15.230 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:15.230 INFO:tasks.rados.rados.0.smithi107.stdout:4376: seq_num 4375 ranges {0=475136,475136=524288,999424=278528} 2020-06-12T02:38:15.232 INFO:tasks.rados.rados.0.smithi107.stdout:4376: writing smithi10729407-9 from 0 to 475136 tid 1 2020-06-12T02:38:15.236 INFO:tasks.rados.rados.0.smithi107.stdout:4376: writing smithi10729407-9 from 475136 to 999424 tid 2 2020-06-12T02:38:15.238 INFO:tasks.rados.rados.0.smithi107.stdout:4376: writing smithi10729407-9 from 999424 to 1277952 tid 3 2020-06-12T02:38:15.238 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:15.238 INFO:tasks.rados.rados.0.smithi107.stdout:4377: seq_num 4376 ranges {0=491520,491520=409600,901120=393216} 2020-06-12T02:38:15.240 INFO:tasks.rados.rados.0.smithi107.stdout:4377: writing smithi10729407-6 from 0 to 491520 tid 1 2020-06-12T02:38:15.244 INFO:tasks.rados.rados.0.smithi107.stdout:4377: writing smithi10729407-6 from 491520 to 901120 tid 2 2020-06-12T02:38:15.246 INFO:tasks.rados.rados.0.smithi107.stdout:4377: writing smithi10729407-6 from 901120 to 1294336 tid 3 2020-06-12T02:38:15.246 INFO:tasks.rados.rados.0.smithi107.stdout:4378: delete oid 18 current snap is 0 2020-06-12T02:38:15.247 INFO:tasks.rados.rados.0.smithi107.stdout:4372: finishing write tid 5 to smithi10729407-5 2020-06-12T02:38:15.247 INFO:tasks.rados.rados.0.smithi107.stdout:4372: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:15.247 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 437 (ObjNum 4371 snap 0 seq_num 4371) dirty exists 2020-06-12T02:38:15.247 INFO:tasks.rados.rados.0.smithi107.stdout:4372: left oid 5 (ObjNum 4371 snap 0 seq_num 4371) 2020-06-12T02:38:15.247 INFO:tasks.rados.rados.0.smithi107.stdout:4375: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:15.248 INFO:tasks.rados.rados.0.smithi107.stdout:4375: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:15.248 INFO:tasks.rados.rados.0.smithi107.stdout:4375: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:15.248 INFO:tasks.rados.rados.0.smithi107.stdout:4376: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:15.248 INFO:tasks.rados.rados.0.smithi107.stdout:4375: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:15.249 INFO:tasks.rados.rados.0.smithi107.stdout:4375: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:15.249 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 796 (ObjNum 4374 snap 0 seq_num 4374) dirty exists 2020-06-12T02:38:15.249 INFO:tasks.rados.rados.0.smithi107.stdout:4375: left oid 4 (ObjNum 4374 snap 0 seq_num 4374) 2020-06-12T02:38:15.249 INFO:tasks.rados.rados.0.smithi107.stdout:4377: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:15.252 INFO:tasks.rados.rados.0.smithi107.stdout:4376: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:15.252 INFO:tasks.rados.rados.0.smithi107.stdout:4376: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:15.254 INFO:tasks.rados.rados.0.smithi107.stdout:4376: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:15.255 INFO:tasks.rados.rados.0.smithi107.stdout:4377: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:15.255 INFO:tasks.rados.rados.0.smithi107.stdout:4372: done (4 left) 2020-06-12T02:38:15.255 INFO:tasks.rados.rados.0.smithi107.stdout:4375: done (3 left) 2020-06-12T02:38:15.256 INFO:tasks.rados.rados.0.smithi107.stdout:4378: done (2 left) 2020-06-12T02:38:15.256 INFO:tasks.rados.rados.0.smithi107.stdout:4379: delete oid 3 current snap is 0 2020-06-12T02:38:15.257 INFO:tasks.rados.rados.0.smithi107.stdout:4376: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:15.258 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 784 (ObjNum 4375 snap 0 seq_num 4375) dirty exists 2020-06-12T02:38:15.258 INFO:tasks.rados.rados.0.smithi107.stdout:4376: left oid 9 (ObjNum 4375 snap 0 seq_num 4375) 2020-06-12T02:38:15.260 INFO:tasks.rados.rados.0.smithi107.stdout:4376: done (2 left) 2020-06-12T02:38:15.260 INFO:tasks.rados.rados.0.smithi107.stdout:4379: done (1 left) 2020-06-12T02:38:15.260 INFO:tasks.rados.rados.0.smithi107.stdout:4380: delete oid 9 current snap is 0 2020-06-12T02:38:15.261 INFO:tasks.rados.rados.0.smithi107.stdout:4377: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:15.261 INFO:tasks.rados.rados.0.smithi107.stdout:4377: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:15.261 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:15.257+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:15.262 INFO:tasks.rados.rados.0.smithi107.stdout:4377: finishing write tid 6 to smithi10729407-6 2020-06-12T02:38:15.262 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 818 (ObjNum 4376 snap 0 seq_num 4376) dirty exists 2020-06-12T02:38:15.262 INFO:tasks.rados.rados.0.smithi107.stdout:4377: left oid 6 (ObjNum 4376 snap 0 seq_num 4376) 2020-06-12T02:38:15.271 INFO:tasks.rados.rados.0.smithi107.stdout:4377: done (1 left) 2020-06-12T02:38:15.271 INFO:tasks.rados.rados.0.smithi107.stdout:4380: done (0 left) 2020-06-12T02:38:15.271 INFO:tasks.rados.rados.0.smithi107.stdout:4381: delete oid 6 current snap is 0 2020-06-12T02:38:15.272 INFO:tasks.rados.rados.0.smithi107.stdout:4381: done (0 left) 2020-06-12T02:38:15.272 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:15.273 INFO:tasks.rados.rados.0.smithi107.stdout:4382: seq_num 4381 ranges {811008=581632,1392640=679936,2072576=122880} 2020-06-12T02:38:15.280 INFO:tasks.rados.rados.0.smithi107.stdout:4382: writing smithi10729407-15 from 811008 to 1392640 tid 1 2020-06-12T02:38:15.283 INFO:tasks.rados.rados.0.smithi107.stdout:4382: writing smithi10729407-15 from 1392640 to 2072576 tid 2 2020-06-12T02:38:15.286 INFO:tasks.rados.rados.0.smithi107.stdout:4382: writing smithi10729407-15 from 2072576 to 2195456 tid 3 2020-06-12T02:38:15.286 INFO:tasks.rados.rados.0.smithi107.stdout:4383: delete oid 19 current snap is 0 2020-06-12T02:38:15.289 INFO:tasks.rados.rados.0.smithi107.stdout:4383: done (1 left) 2020-06-12T02:38:15.290 INFO:tasks.rados.rados.0.smithi107.stdout:4384: delete oid 17 current snap is 0 2020-06-12T02:38:15.291 INFO:tasks.rados.rados.0.smithi107.stdout:4382: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:15.294 INFO:tasks.rados.rados.0.smithi107.stdout:4382: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:15.296 INFO:tasks.rados.rados.0.smithi107.stdout:4382: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:15.297 INFO:tasks.rados.rados.0.smithi107.stdout:4384: done (1 left) 2020-06-12T02:38:15.297 INFO:tasks.rados.rados.0.smithi107.stdout:4385: delete oid 14 current snap is 0 2020-06-12T02:38:15.297 INFO:tasks.rados.rados.0.smithi107.stdout:4382: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:15.298 INFO:tasks.rados.rados.0.smithi107.stdout:4382: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:15.299 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 753 (ObjNum 4381 snap 0 seq_num 4381) dirty exists 2020-06-12T02:38:15.299 INFO:tasks.rados.rados.0.smithi107.stdout:4382: left oid 15 (ObjNum 4381 snap 0 seq_num 4381) 2020-06-12T02:38:15.300 INFO:tasks.rados.rados.0.smithi107.stdout:4382: done (1 left) 2020-06-12T02:38:15.300 INFO:tasks.rados.rados.0.smithi107.stdout:4385: done (0 left) 2020-06-12T02:38:15.301 INFO:tasks.rados.rados.0.smithi107.stdout:4386: delete oid 14 current snap is 0 2020-06-12T02:38:15.302 INFO:tasks.rados.rados.0.smithi107.stdout:4386: done (0 left) 2020-06-12T02:38:15.303 INFO:tasks.rados.rados.0.smithi107.stdout:4387: delete oid 8 current snap is 0 2020-06-12T02:38:15.304 INFO:tasks.rados.rados.0.smithi107.stdout:4387: done (0 left) 2020-06-12T02:38:15.305 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:38:15.305 INFO:tasks.rados.rados.0.smithi107.stdout:4388: seq_num 4387 ranges {0=679936,679936=434176,1114112=368640} 2020-06-12T02:38:15.308 INFO:tasks.rados.rados.0.smithi107.stdout:4388: writing smithi10729407-6 from 0 to 679936 tid 1 2020-06-12T02:38:15.310 INFO:tasks.rados.rados.0.smithi107.stdout:4388: writing smithi10729407-6 from 679936 to 1114112 tid 2 2020-06-12T02:38:15.313 INFO:tasks.rados.rados.0.smithi107.stdout:4388: writing smithi10729407-6 from 1114112 to 1482752 tid 3 2020-06-12T02:38:15.313 INFO:tasks.rados.rados.0.smithi107.stdout:4389: delete oid 13 current snap is 0 2020-06-12T02:38:15.316 INFO:tasks.rados.rados.0.smithi107.stdout:4389: done (1 left) 2020-06-12T02:38:15.317 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:15.317 INFO:tasks.rados.rados.0.smithi107.stdout:4390: seq_num 4389 ranges {1212416=712704,1925120=606208,2531328=196608} 2020-06-12T02:38:15.317 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:15.318 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:15.328 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-06-12T02:38:15.329 INFO:tasks.rados.rados.0.smithi107.stdout:4390: writing smithi10729407-5 from 1212416 to 1925120 tid 1 2020-06-12T02:38:15.331 INFO:tasks.rados.rados.0.smithi107.stdout:4390: writing smithi10729407-5 from 1925120 to 2531328 tid 2 2020-06-12T02:38:15.333 INFO:tasks.rados.rados.0.smithi107.stdout:4390: writing smithi10729407-5 from 2531328 to 2727936 tid 3 2020-06-12T02:38:15.333 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:15.333 INFO:tasks.rados.rados.0.smithi107.stdout:4391: seq_num 4390 ranges {0=729088,729088=491520,1220608=311296} 2020-06-12T02:38:15.337 INFO:tasks.rados.rados.0.smithi107.stdout:4391: writing smithi10729407-9 from 0 to 729088 tid 1 2020-06-12T02:38:15.340 INFO:tasks.rados.rados.0.smithi107.stdout:4391: writing smithi10729407-9 from 729088 to 1220608 tid 2 2020-06-12T02:38:15.343 INFO:tasks.rados.rados.0.smithi107.stdout:4391: writing smithi10729407-9 from 1220608 to 1531904 tid 3 2020-06-12T02:38:15.343 INFO:tasks.rados.rados.0.smithi107.stdout:4392: delete oid 13 current snap is 0 2020-06-12T02:38:15.343 INFO:tasks.rados.rados.0.smithi107.stdout:4388: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:15.343 INFO:tasks.rados.rados.0.smithi107.stdout:4388: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:15.344 INFO:tasks.rados.rados.0.smithi107.stdout:4388: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:15.344 INFO:tasks.rados.rados.0.smithi107.stdout:4388: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:15.344 INFO:tasks.rados.rados.0.smithi107.stdout:4388: finishing write tid 6 to smithi10729407-6 2020-06-12T02:38:15.344 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 823 (ObjNum 4387 snap 0 seq_num 4387) dirty exists 2020-06-12T02:38:15.344 INFO:tasks.rados.rados.0.smithi107.stdout:4388: left oid 6 (ObjNum 4387 snap 0 seq_num 4387) 2020-06-12T02:38:15.345 INFO:tasks.rados.rados.0.smithi107.stdout:4390: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:15.345 INFO:tasks.rados.rados.0.smithi107.stdout:4390: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:15.345 INFO:tasks.rados.rados.0.smithi107.stdout:4390: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:15.345 INFO:tasks.rados.rados.0.smithi107.stdout:4390: finishing write tid 5 to smithi10729407-5 2020-06-12T02:38:15.345 INFO:tasks.rados.rados.0.smithi107.stdout:4390: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:15.346 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 441 (ObjNum 4389 snap 0 seq_num 4389) dirty exists 2020-06-12T02:38:15.346 INFO:tasks.rados.rados.0.smithi107.stdout:4390: left oid 5 (ObjNum 4389 snap 0 seq_num 4389) 2020-06-12T02:38:15.346 INFO:tasks.rados.rados.0.smithi107.stdout:4391: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:15.346 INFO:tasks.rados.rados.0.smithi107.stdout:4388: done (3 left) 2020-06-12T02:38:15.347 INFO:tasks.rados.rados.0.smithi107.stdout:4390: done (2 left) 2020-06-12T02:38:15.347 INFO:tasks.rados.rados.0.smithi107.stdout:4392: done (1 left) 2020-06-12T02:38:15.347 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:15.347 INFO:tasks.rados.rados.0.smithi107.stdout:4393: seq_num 4392 ranges {0=761856,761856=655360,1417216=147456} 2020-06-12T02:38:15.349 INFO:tasks.rados.rados.0.smithi107.stdout:4393: writing smithi10729407-19 from 0 to 761856 tid 1 2020-06-12T02:38:15.353 INFO:tasks.rados.rados.0.smithi107.stdout:4393: writing smithi10729407-19 from 761856 to 1417216 tid 2 2020-06-12T02:38:15.354 INFO:tasks.rados.rados.0.smithi107.stdout:4393: writing smithi10729407-19 from 1417216 to 1564672 tid 3 2020-06-12T02:38:15.355 INFO:tasks.rados.rados.0.smithi107.stdout:4394: delete oid 20 current snap is 0 2020-06-12T02:38:15.355 INFO:tasks.rados.rados.0.smithi107.stdout:4391: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:15.355 INFO:tasks.rados.rados.0.smithi107.stdout:4391: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:15.355 INFO:tasks.rados.rados.0.smithi107.stdout:4391: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:15.355 INFO:tasks.rados.rados.0.smithi107.stdout:4391: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:15.356 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 790 (ObjNum 4390 snap 0 seq_num 4390) dirty exists 2020-06-12T02:38:15.356 INFO:tasks.rados.rados.0.smithi107.stdout:4391: left oid 9 (ObjNum 4390 snap 0 seq_num 4390) 2020-06-12T02:38:15.358 INFO:tasks.rados.rados.0.smithi107.stdout:4391: done (2 left) 2020-06-12T02:38:15.358 INFO:tasks.rados.rados.0.smithi107.stdout:4394: done (1 left) 2020-06-12T02:38:15.359 INFO:tasks.rados.rados.0.smithi107.stdout:4395: delete oid 15 current snap is 0 2020-06-12T02:38:15.359 INFO:tasks.rados.rados.0.smithi107.stdout:4393: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:15.361 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:15.358+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:15.362 INFO:tasks.rados.rados.0.smithi107.stdout:4393: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:15.362 INFO:tasks.rados.rados.0.smithi107.stdout:4393: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:15.364 INFO:tasks.rados.rados.0.smithi107.stdout:4395: done (1 left) 2020-06-12T02:38:15.365 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:15.365 INFO:tasks.rados.rados.0.smithi107.stdout:4396: seq_num 4395 ranges {3833856=409600,4243456=761856,5005312=442368} 2020-06-12T02:38:15.373 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2020-06-12T02:38:15.388 INFO:tasks.rados.rados.0.smithi107.stdout:4396: writing smithi10729407-11 from 3833856 to 4243456 tid 1 2020-06-12T02:38:15.395 INFO:tasks.rados.rados.0.smithi107.stdout:4396: writing smithi10729407-11 from 4243456 to 5005312 tid 2 2020-06-12T02:38:15.396 INFO:tasks.rados.rados.0.smithi107.stdout:4396: writing smithi10729407-11 from 5005312 to 5447680 tid 3 2020-06-12T02:38:15.397 INFO:tasks.rados.rados.0.smithi107.stdout:4397: delete oid 17 current snap is 0 2020-06-12T02:38:15.397 INFO:tasks.rados.rados.0.smithi107.stdout:4393: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:15.397 INFO:tasks.rados.rados.0.smithi107.stdout:4393: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:15.397 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 742 (ObjNum 4392 snap 0 seq_num 4392) dirty exists 2020-06-12T02:38:15.397 INFO:tasks.rados.rados.0.smithi107.stdout:4393: left oid 19 (ObjNum 4392 snap 0 seq_num 4392) 2020-06-12T02:38:15.398 INFO:tasks.rados.rados.0.smithi107.stdout:4396: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:15.401 INFO:tasks.rados.rados.0.smithi107.stdout:4393: done (2 left) 2020-06-12T02:38:15.402 INFO:tasks.rados.rados.0.smithi107.stdout:4397: done (1 left) 2020-06-12T02:38:15.402 INFO:tasks.rados.rados.0.smithi107.stdout:4398: delete oid 15 current snap is 0 2020-06-12T02:38:15.404 INFO:tasks.rados.rados.0.smithi107.stdout:4396: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:15.405 INFO:tasks.rados.rados.0.smithi107.stdout:4396: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:15.406 INFO:tasks.rados.rados.0.smithi107.stdout:4396: finishing write tid 5 to smithi10729407-11 2020-06-12T02:38:15.407 INFO:tasks.rados.rados.0.smithi107.stdout:4396: finishing write tid 6 to smithi10729407-11 2020-06-12T02:38:15.407 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 786 (ObjNum 4395 snap 0 seq_num 4395) dirty exists 2020-06-12T02:38:15.407 INFO:tasks.rados.rados.0.smithi107.stdout:4396: left oid 11 (ObjNum 4395 snap 0 seq_num 4395) 2020-06-12T02:38:15.462 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:15.460+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:15.470 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:15.470 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:15.479 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-06-12T02:38:15.515 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-06-12T02:38:15.563 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:15.561+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:15.639 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:15.640 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:15.649 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-06-12T02:38:15.653 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:38:15.674 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:38:15.693 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:38:15.694 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:15.691+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:15.712 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:38:15.732 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:38:15.752 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:38:15.766 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:15.763+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:15.767 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:15.768 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:38:15.768 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2020-06-12T02:38:15.768 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:15.771 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:38:15.776 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-06-12T02:38:15.791 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2020-06-12T02:38:15.793 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-06-12T02:38:15.793 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-06-12T02:38:15.793 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-06-12T02:38:15.793 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-06-12T02:38:15.793 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-06-12T02:38:15.794 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-06-12T02:38:15.794 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-06-12T02:38:15.794 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-06-12T02:38:15.867 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:15.865+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:15.933 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:15.933 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:38:15.933 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:38:15.934 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:38:15.934 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:15.941 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-06-12T02:38:15.985 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:15.983+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:16.060 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:16.061 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:38:16.061 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:38:16.062 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:38:16.062 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.062 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:860 3.3s0 3:ce45c70a:::smithi10729407-7:head [append 0~81920] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:38:16.063 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.928619+0000", 2020-06-12T02:38:16.063 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 44.127200061000003, 2020-06-12T02:38:16.063 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017431941999999999, 2020-06-12T02:38:16.064 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:16.064 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:16.064 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:16.065 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:16.065 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:16.065 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 860 2020-06-12T02:38:16.066 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.067 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:16.067 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.067 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:16.068 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928619+0000", 2020-06-12T02:38:16.068 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:16.068 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.069 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.069 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:16.069 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928619+0000", 2020-06-12T02:38:16.070 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000105809 2020-06-12T02:38:16.070 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.070 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.071 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:16.071 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928724+0000", 2020-06-12T02:38:16.071 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2339999999999994e-06 2020-06-12T02:38:16.071 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.072 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.072 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:16.072 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928733+0000", 2020-06-12T02:38:16.073 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0268000000000001e-05 2020-06-12T02:38:16.073 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.073 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.074 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:16.074 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928743+0000", 2020-06-12T02:38:16.074 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5190000000000003e-06 2020-06-12T02:38:16.075 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.075 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.075 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:16.076 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928747+0000", 2020-06-12T02:38:16.077 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0103999999999998e-05 2020-06-12T02:38:16.077 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.078 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.078 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:16.078 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928777+0000", 2020-06-12T02:38:16.079 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8249000000000002e-05 2020-06-12T02:38:16.079 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.079 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.079 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:16.080 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928866+0000", 2020-06-12T02:38:16.080 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011151629 2020-06-12T02:38:16.080 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.081 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.081 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:16.081 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.940017+0000", 2020-06-12T02:38:16.082 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8281e-05 2020-06-12T02:38:16.082 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.082 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.083 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:16.083 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.940076+0000", 2020-06-12T02:38:16.083 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000346736 2020-06-12T02:38:16.084 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.084 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.084 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:16.085 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.940422+0000", 2020-06-12T02:38:16.085 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024135049999999998 2020-06-12T02:38:16.085 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.086 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.086 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:16.087 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.942836+0000", 2020-06-12T02:38:16.087 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003170258 2020-06-12T02:38:16.088 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.088 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.088 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:16.088 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946006+0000", 2020-06-12T02:38:16.089 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4350000000000001e-05 2020-06-12T02:38:16.089 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.089 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.090 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:16.090 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946050+0000", 2020-06-12T02:38:16.090 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017431941999999999 2020-06-12T02:38:16.091 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:16.091 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:16.091 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:16.092 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.092 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.092 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:861 3.3s0 3:ce45c70a:::smithi10729407-7:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:38:16.093 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.928778+0000", 2020-06-12T02:38:16.093 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 44.127040448000002, 2020-06-12T02:38:16.093 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017522738, 2020-06-12T02:38:16.094 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:16.094 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:16.094 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:16.095 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:16.095 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:16.095 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 861 2020-06-12T02:38:16.096 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.096 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:16.097 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.097 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:16.098 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928778+0000", 2020-06-12T02:38:16.098 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:16.098 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.098 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.099 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:16.099 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928778+0000", 2020-06-12T02:38:16.099 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8186000000000002e-05 2020-06-12T02:38:16.099 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.099 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.100 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:16.100 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928826+0000", 2020-06-12T02:38:16.100 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6689999999999997e-06 2020-06-12T02:38:16.100 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.100 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.100 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:16.101 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928831+0000", 2020-06-12T02:38:16.101 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.686e-06 2020-06-12T02:38:16.101 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.101 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.101 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:16.102 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928838+0000", 2020-06-12T02:38:16.102 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7159999999999999e-06 2020-06-12T02:38:16.102 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.102 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.102 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:16.103 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928840+0000", 2020-06-12T02:38:16.103 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4212999999999997e-05 2020-06-12T02:38:16.103 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.103 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.103 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:16.103 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928895+0000", 2020-06-12T02:38:16.104 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5853999999999994e-05 2020-06-12T02:38:16.104 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.104 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.104 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:16.104 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928990+0000", 2020-06-12T02:38:16.105 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012023982000000001 2020-06-12T02:38:16.105 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.105 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.105 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:16.105 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941014+0000", 2020-06-12T02:38:16.106 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5104e-05 2020-06-12T02:38:16.106 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.106 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.106 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:16.107 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941070+0000", 2020-06-12T02:38:16.107 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00026444499999999998 2020-06-12T02:38:16.107 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.108 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.108 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:16.108 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941334+0000", 2020-06-12T02:38:16.108 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001704075 2020-06-12T02:38:16.108 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.109 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.109 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:16.109 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.943038+0000", 2020-06-12T02:38:16.109 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031891570000000002 2020-06-12T02:38:16.109 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.110 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.110 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:16.110 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946227+0000", 2020-06-12T02:38:16.110 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3651000000000003e-05 2020-06-12T02:38:16.110 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.111 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.111 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:16.111 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946301+0000", 2020-06-12T02:38:16.111 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017522738 2020-06-12T02:38:16.111 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:16.112 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:16.112 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:16.113 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.113 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.114 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:862 3.3s0 3:ce45c70a:::smithi10729407-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:38:16.114 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.928876+0000", 2020-06-12T02:38:16.114 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 44.126942952, 2020-06-12T02:38:16.114 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019206890000000001, 2020-06-12T02:38:16.114 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:16.115 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:16.115 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:16.115 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:16.115 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:16.115 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 862 2020-06-12T02:38:16.115 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.116 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:16.116 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.116 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:16.116 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928876+0000", 2020-06-12T02:38:16.116 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:16.117 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.117 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.117 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:16.117 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928876+0000", 2020-06-12T02:38:16.117 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9268999999999999e-05 2020-06-12T02:38:16.118 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.118 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.118 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:16.118 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928925+0000", 2020-06-12T02:38:16.118 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8120000000000002e-06 2020-06-12T02:38:16.119 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.119 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.119 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:16.119 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928932+0000", 2020-06-12T02:38:16.119 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3471e-05 2020-06-12T02:38:16.119 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.120 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.120 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:16.120 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928945+0000", 2020-06-12T02:38:16.120 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4279999999999998e-06 2020-06-12T02:38:16.120 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.121 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.121 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:16.121 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928950+0000", 2020-06-12T02:38:16.121 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6358000000000004e-05 2020-06-12T02:38:16.121 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.122 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.122 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:16.122 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.929016+0000", 2020-06-12T02:38:16.122 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9697999999999996e-05 2020-06-12T02:38:16.123 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.123 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.123 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:16.123 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.929086+0000", 2020-06-12T02:38:16.123 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012409205 2020-06-12T02:38:16.124 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.124 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.124 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:16.124 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941495+0000", 2020-06-12T02:38:16.124 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9660000000000006e-05 2020-06-12T02:38:16.124 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.125 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.125 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:16.125 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941585+0000", 2020-06-12T02:38:16.125 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050329099999999998 2020-06-12T02:38:16.125 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.126 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.126 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:16.126 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946617+0000", 2020-06-12T02:38:16.126 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8437999999999996e-05 2020-06-12T02:38:16.126 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.126 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.127 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:16.127 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946706+0000", 2020-06-12T02:38:16.127 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0013766410000000001 2020-06-12T02:38:16.127 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.127 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.128 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:16.128 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.948083+0000", 2020-06-12T02:38:16.128 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019206890000000001 2020-06-12T02:38:16.128 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:16.128 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:16.128 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:16.129 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.129 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.129 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:879 3.fs1 3:fc7e0bf7:::smithi10729407-17:head [append 0~491520] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:38:16.129 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.000881+0000", 2020-06-12T02:38:16.129 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 44.054937770999999, 2020-06-12T02:38:16.130 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017891845, 2020-06-12T02:38:16.130 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:16.130 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:16.130 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:16.130 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:16.131 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:16.131 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 879 2020-06-12T02:38:16.131 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.131 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:16.132 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.132 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:16.132 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.000881+0000", 2020-06-12T02:38:16.132 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:16.133 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.133 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.133 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:16.133 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.000881+0000", 2020-06-12T02:38:16.134 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00044151599999999998 2020-06-12T02:38:16.134 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.134 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.134 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:16.134 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001322+0000", 2020-06-12T02:38:16.135 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0210000000000001e-06 2020-06-12T02:38:16.135 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.135 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.135 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:16.135 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001326+0000", 2020-06-12T02:38:16.136 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6659999999999997e-06 2020-06-12T02:38:16.136 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.136 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.136 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:16.136 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001335+0000", 2020-06-12T02:38:16.136 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4299999999999999e-06 2020-06-12T02:38:16.137 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.137 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.137 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:16.137 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001339+0000", 2020-06-12T02:38:16.137 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0028741510000000001 2020-06-12T02:38:16.138 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.138 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.138 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:16.138 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.004214+0000", 2020-06-12T02:38:16.138 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010480999999999999 2020-06-12T02:38:16.138 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.139 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.139 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:16.139 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.004318+0000", 2020-06-12T02:38:16.139 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021112359999999998 2020-06-12T02:38:16.139 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.140 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.140 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:16.140 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006430+0000", 2020-06-12T02:38:16.142 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2935000000000001e-05 2020-06-12T02:38:16.142 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.142 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.142 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:16.143 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006483+0000", 2020-06-12T02:38:16.144 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00056411099999999995 2020-06-12T02:38:16.144 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.144 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.144 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:16.144 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.007047+0000", 2020-06-12T02:38:16.144 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0076749369999999997 2020-06-12T02:38:16.145 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.145 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.145 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:16.145 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.014722+0000", 2020-06-12T02:38:16.146 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00395693 2020-06-12T02:38:16.146 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.146 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.146 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:16.146 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.018679+0000", 2020-06-12T02:38:16.146 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4102000000000001e-05 2020-06-12T02:38:16.147 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.147 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.147 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:16.147 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.018773+0000", 2020-06-12T02:38:16.147 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017891845 2020-06-12T02:38:16.148 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:16.148 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:16.148 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:16.148 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.148 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.149 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:933 3.es1 3:78f01648:::smithi10729407-5:head [append 0~417792] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:38:16.149 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.156677+0000", 2020-06-12T02:38:16.149 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 43.899141788999998, 2020-06-12T02:38:16.149 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016833311, 2020-06-12T02:38:16.149 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:16.149 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:16.150 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:16.150 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:16.150 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:16.150 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 933 2020-06-12T02:38:16.150 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.151 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:16.151 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.151 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:16.151 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.156677+0000", 2020-06-12T02:38:16.151 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:16.151 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.152 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.152 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:16.152 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.156677+0000", 2020-06-12T02:38:16.152 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00062571700000000003 2020-06-12T02:38:16.152 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.153 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.153 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:16.153 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157303+0000", 2020-06-12T02:38:16.153 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0740000000000003e-06 2020-06-12T02:38:16.154 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.154 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.154 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:16.154 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157307+0000", 2020-06-12T02:38:16.154 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4419999999999998e-06 2020-06-12T02:38:16.155 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.155 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.155 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:16.155 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157315+0000", 2020-06-12T02:38:16.155 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5300000000000001e-06 2020-06-12T02:38:16.156 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.156 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.156 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:16.156 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157319+0000", 2020-06-12T02:38:16.156 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4995e-05 2020-06-12T02:38:16.156 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.157 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.157 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:16.157 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157354+0000", 2020-06-12T02:38:16.157 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5883000000000006e-05 2020-06-12T02:38:16.157 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.158 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.158 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:16.158 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157419+0000", 2020-06-12T02:38:16.158 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045316280000000002 2020-06-12T02:38:16.158 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.159 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.159 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:16.159 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.161951+0000", 2020-06-12T02:38:16.159 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3300999999999996e-05 2020-06-12T02:38:16.159 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.159 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.160 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:16.160 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.162014+0000", 2020-06-12T02:38:16.160 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00039050100000000002 2020-06-12T02:38:16.160 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.160 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.161 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:16.161 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.162405+0000", 2020-06-12T02:38:16.161 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050686250000000002 2020-06-12T02:38:16.161 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.161 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.161 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:16.162 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.167474+0000", 2020-06-12T02:38:16.162 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.005942653 2020-06-12T02:38:16.162 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.162 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.162 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:16.163 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.173416+0000", 2020-06-12T02:38:16.163 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3962000000000006e-05 2020-06-12T02:38:16.163 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.163 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.164 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:16.164 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.173510+0000", 2020-06-12T02:38:16.164 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016833311 2020-06-12T02:38:16.164 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:16.164 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:16.165 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:16.165 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.165 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.165 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:4921 3.9s0 3:93ea0cad:::smithi10729407-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-06-12T02:38:16.165 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:43.595696+0000", 2020-06-12T02:38:16.165 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 32.460122527999999, 2020-06-12T02:38:16.166 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016503190000000001, 2020-06-12T02:38:16.166 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:16.166 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:16.166 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:16.166 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:16.167 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:16.167 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 4921 2020-06-12T02:38:16.167 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.167 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:16.168 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.168 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:16.168 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595696+0000", 2020-06-12T02:38:16.168 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:16.168 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.169 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.169 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:16.169 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595696+0000", 2020-06-12T02:38:16.169 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3694999999999999e-05 2020-06-12T02:38:16.169 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.170 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.170 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:16.170 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595720+0000", 2020-06-12T02:38:16.170 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1700000000000001e-06 2020-06-12T02:38:16.170 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.170 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.171 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:16.171 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595723+0000", 2020-06-12T02:38:16.171 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8849999999999999e-06 2020-06-12T02:38:16.171 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.172 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.172 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:16.172 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595731+0000", 2020-06-12T02:38:16.172 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5689999999999999e-06 2020-06-12T02:38:16.172 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.173 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.173 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:16.173 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595733+0000", 2020-06-12T02:38:16.175 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0030116880000000002 2020-06-12T02:38:16.175 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.175 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.176 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:16.177 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.598745+0000", 2020-06-12T02:38:16.177 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1353999999999996e-05 2020-06-12T02:38:16.177 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.177 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.177 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:16.178 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.598806+0000", 2020-06-12T02:38:16.178 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051955819999999998 2020-06-12T02:38:16.178 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.178 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.178 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:16.178 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.604002+0000", 2020-06-12T02:38:16.179 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9615000000000001e-05 2020-06-12T02:38:16.179 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.179 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.179 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:16.179 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.604102+0000", 2020-06-12T02:38:16.180 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0052050960000000002 2020-06-12T02:38:16.180 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.180 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.180 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:16.180 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.609307+0000", 2020-06-12T02:38:16.180 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5025000000000003e-05 2020-06-12T02:38:16.181 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.181 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.181 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:16.181 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.609362+0000", 2020-06-12T02:38:16.181 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002837511 2020-06-12T02:38:16.182 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.182 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.182 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:16.182 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.612199+0000", 2020-06-12T02:38:16.182 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016503190000000001 2020-06-12T02:38:16.183 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:16.183 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:16.183 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:16.183 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.183 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.183 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:5071 3.3s0 3:c2ca77da:::smithi10729407-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-06-12T02:38:16.184 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:44.159788+0000", 2020-06-12T02:38:16.184 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 31.896030679999999, 2020-06-12T02:38:16.184 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016812273999999999, 2020-06-12T02:38:16.184 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:16.184 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:16.185 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:16.185 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:16.185 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:16.185 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 5071 2020-06-12T02:38:16.186 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.186 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:16.186 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.186 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:16.186 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159788+0000", 2020-06-12T02:38:16.187 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:16.187 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.187 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.187 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:16.187 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159788+0000", 2020-06-12T02:38:16.188 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1421000000000001e-05 2020-06-12T02:38:16.188 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.188 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.188 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:16.188 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159839+0000", 2020-06-12T02:38:16.189 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9009999999999994e-06 2020-06-12T02:38:16.189 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.189 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.189 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:16.189 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159847+0000", 2020-06-12T02:38:16.190 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3149999999999999e-05 2020-06-12T02:38:16.190 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.190 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.190 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:16.190 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159860+0000", 2020-06-12T02:38:16.190 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1749999999999996e-06 2020-06-12T02:38:16.191 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.191 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.191 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:16.191 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159866+0000", 2020-06-12T02:38:16.191 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6035999999999999e-05 2020-06-12T02:38:16.192 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.192 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.192 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:16.192 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159892+0000", 2020-06-12T02:38:16.192 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2627999999999998e-05 2020-06-12T02:38:16.193 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.193 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.193 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:16.193 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159944+0000", 2020-06-12T02:38:16.194 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010270133000000001 2020-06-12T02:38:16.194 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.194 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.194 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:16.194 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.170214+0000", 2020-06-12T02:38:16.195 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5490999999999998e-05 2020-06-12T02:38:16.195 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.195 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.195 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:16.195 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.170310+0000", 2020-06-12T02:38:16.195 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003215828 2020-06-12T02:38:16.196 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.196 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.196 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:16.196 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.173526+0000", 2020-06-12T02:38:16.196 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0290000000000005e-05 2020-06-12T02:38:16.197 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.197 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.197 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:16.197 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.173616+0000", 2020-06-12T02:38:16.198 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002984221 2020-06-12T02:38:16.198 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.198 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.198 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:16.198 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.176600+0000", 2020-06-12T02:38:16.199 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016812273999999999 2020-06-12T02:38:16.203 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:16.204 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:16.204 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:16.204 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.204 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.204 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:6432 3.9s0 3:93ea0cad:::smithi10729407-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e31)", 2020-06-12T02:38:16.205 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:47.960104+0000", 2020-06-12T02:38:16.205 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 28.095714584, 2020-06-12T02:38:16.205 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016025815999999998, 2020-06-12T02:38:16.205 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:16.205 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:16.206 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:16.206 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:16.206 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:16.206 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 6432 2020-06-12T02:38:16.206 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.207 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:16.207 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.207 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:16.207 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960104+0000", 2020-06-12T02:38:16.207 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:16.207 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.208 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.208 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:16.208 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960104+0000", 2020-06-12T02:38:16.208 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2351999999999999e-05 2020-06-12T02:38:16.208 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.209 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.209 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:16.209 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960126+0000", 2020-06-12T02:38:16.209 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.061e-06 2020-06-12T02:38:16.209 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.209 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.210 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:16.210 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960129+0000", 2020-06-12T02:38:16.210 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4849999999999997e-06 2020-06-12T02:38:16.210 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.210 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.211 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:16.211 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960136+0000", 2020-06-12T02:38:16.211 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.176e-06 2020-06-12T02:38:16.211 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.211 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.211 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:16.212 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.960138+0000", 2020-06-12T02:38:16.212 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0044054139999999999 2020-06-12T02:38:16.212 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.212 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.212 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:16.213 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.964543+0000", 2020-06-12T02:38:16.213 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8077000000000006e-05 2020-06-12T02:38:16.213 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.213 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.213 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:16.213 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.964612+0000", 2020-06-12T02:38:16.214 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0029921150000000001 2020-06-12T02:38:16.214 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.214 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.214 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:16.214 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.967604+0000", 2020-06-12T02:38:16.215 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2983000000000001e-05 2020-06-12T02:38:16.215 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.215 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.215 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:16.215 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.967697+0000", 2020-06-12T02:38:16.215 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0072496510000000002 2020-06-12T02:38:16.216 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.216 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.216 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:16.216 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.974946+0000", 2020-06-12T02:38:16.216 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8278e-05 2020-06-12T02:38:16.217 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.217 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.217 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:16.217 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.975005+0000", 2020-06-12T02:38:16.217 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001125224 2020-06-12T02:38:16.218 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.218 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.218 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:16.218 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:47.976130+0000", 2020-06-12T02:38:16.218 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016025815999999998 2020-06-12T02:38:16.218 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:16.219 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:16.219 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:16.219 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.219 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.219 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8123 3.es1 3:78f01648:::smithi10729407-5:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-06-12T02:38:16.220 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.690721+0000", 2020-06-12T02:38:16.220 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 23.365097479999999, 2020-06-12T02:38:16.220 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016503126999999999, 2020-06-12T02:38:16.220 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:16.220 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:16.220 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:16.221 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:16.221 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:16.221 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8123 2020-06-12T02:38:16.221 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.221 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:16.222 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.222 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:16.223 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690721+0000", 2020-06-12T02:38:16.223 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:16.223 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.223 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.224 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:16.224 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690721+0000", 2020-06-12T02:38:16.224 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7526e-05 2020-06-12T02:38:16.224 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.224 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.224 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:16.225 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690759+0000", 2020-06-12T02:38:16.225 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1999999999999999e-06 2020-06-12T02:38:16.225 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.225 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.225 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:16.226 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690762+0000", 2020-06-12T02:38:16.226 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9229999999999999e-06 2020-06-12T02:38:16.226 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.226 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.226 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:16.227 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690768+0000", 2020-06-12T02:38:16.227 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4480000000000001e-06 2020-06-12T02:38:16.227 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.227 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.227 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:16.227 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690770+0000", 2020-06-12T02:38:16.228 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00079317799999999996 2020-06-12T02:38:16.228 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.228 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.228 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:16.228 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.691563+0000", 2020-06-12T02:38:16.229 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5299e-05 2020-06-12T02:38:16.229 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.229 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.229 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:16.229 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.691619+0000", 2020-06-12T02:38:16.229 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010070076000000001 2020-06-12T02:38:16.230 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.230 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.230 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:16.230 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.701689+0000", 2020-06-12T02:38:16.230 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000110234 2020-06-12T02:38:16.231 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.231 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.231 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:16.231 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.701799+0000", 2020-06-12T02:38:16.231 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00037311100000000003 2020-06-12T02:38:16.231 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.232 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.232 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:16.232 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.702172+0000", 2020-06-12T02:38:16.232 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0044460050000000003 2020-06-12T02:38:16.232 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.233 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.233 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:16.233 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.706618+0000", 2020-06-12T02:38:16.233 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00043550199999999999 2020-06-12T02:38:16.233 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.233 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.234 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:16.234 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.707054+0000", 2020-06-12T02:38:16.234 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017062499999999999 2020-06-12T02:38:16.234 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.234 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.234 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:16.235 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.707224+0000", 2020-06-12T02:38:16.235 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016503126999999999 2020-06-12T02:38:16.235 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:16.235 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:16.235 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:16.236 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.236 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.236 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8124 3.es1 3:78f01648:::smithi10729407-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e33)", 2020-06-12T02:38:16.236 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.690794+0000", 2020-06-12T02:38:16.236 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 23.365024294000001, 2020-06-12T02:38:16.236 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019298497000000001, 2020-06-12T02:38:16.237 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:16.237 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:16.237 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:16.237 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:16.237 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:16.238 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8124 2020-06-12T02:38:16.238 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.238 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:16.238 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.238 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:16.239 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690794+0000", 2020-06-12T02:38:16.239 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:16.239 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.239 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.239 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:16.240 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690794+0000", 2020-06-12T02:38:16.240 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1869999999999999e-05 2020-06-12T02:38:16.240 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.240 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.240 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:16.240 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690816+0000", 2020-06-12T02:38:16.241 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.162e-06 2020-06-12T02:38:16.241 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.241 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.241 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:16.241 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690819+0000", 2020-06-12T02:38:16.242 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2280000000000003e-06 2020-06-12T02:38:16.242 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.242 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.242 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:16.242 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690826+0000", 2020-06-12T02:38:16.242 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.452e-06 2020-06-12T02:38:16.243 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.243 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.243 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:16.243 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690828+0000", 2020-06-12T02:38:16.243 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00081402600000000003 2020-06-12T02:38:16.244 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.244 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.244 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:16.244 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.691642+0000", 2020-06-12T02:38:16.244 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00024263 2020-06-12T02:38:16.244 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.245 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.245 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:16.245 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.691885+0000", 2020-06-12T02:38:16.245 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010968739 2020-06-12T02:38:16.245 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.246 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.246 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:16.246 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.702853+0000", 2020-06-12T02:38:16.246 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8985000000000002e-05 2020-06-12T02:38:16.246 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.246 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.247 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:16.247 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.702952+0000", 2020-06-12T02:38:16.247 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0046973329999999997 2020-06-12T02:38:16.247 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.247 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.248 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:16.248 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.707650+0000", 2020-06-12T02:38:16.248 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3773e-05 2020-06-12T02:38:16.248 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.248 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.248 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:16.249 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.707723+0000", 2020-06-12T02:38:16.249 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023692990000000001 2020-06-12T02:38:16.249 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.249 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.249 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:16.250 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.710093+0000", 2020-06-12T02:38:16.250 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019298497000000001 2020-06-12T02:38:16.250 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:16.251 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:16.251 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:16.251 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.251 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.252 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8190 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~753664] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-06-12T02:38:16.252 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.894592+0000", 2020-06-12T02:38:16.252 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 23.161226635999999, 2020-06-12T02:38:16.252 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016165763999999999, 2020-06-12T02:38:16.254 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:16.254 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:16.254 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:16.254 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:16.255 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:16.255 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8190 2020-06-12T02:38:16.255 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.255 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:16.255 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.256 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:16.256 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.894592+0000", 2020-06-12T02:38:16.256 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:16.256 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.256 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.257 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:16.257 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.894592+0000", 2020-06-12T02:38:16.257 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00074881500000000001 2020-06-12T02:38:16.257 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.257 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.258 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:16.258 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.895341+0000", 2020-06-12T02:38:16.258 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1919999999999999e-06 2020-06-12T02:38:16.258 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.258 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.259 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:16.259 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.895348+0000", 2020-06-12T02:38:16.259 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.059e-05 2020-06-12T02:38:16.259 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.259 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.259 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:16.260 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.895359+0000", 2020-06-12T02:38:16.260 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2310000000000001e-06 2020-06-12T02:38:16.260 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.260 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.260 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:16.261 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.895365+0000", 2020-06-12T02:38:16.261 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00049215800000000003 2020-06-12T02:38:16.261 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.261 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.262 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:16.262 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.895857+0000", 2020-06-12T02:38:16.262 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1001999999999995e-05 2020-06-12T02:38:16.263 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.263 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.263 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:16.263 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.895938+0000", 2020-06-12T02:38:16.263 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0059219040000000004 2020-06-12T02:38:16.264 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.264 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.264 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:16.264 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.901860+0000", 2020-06-12T02:38:16.264 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010156499999999999 2020-06-12T02:38:16.265 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.265 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.265 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:16.265 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.901961+0000", 2020-06-12T02:38:16.265 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00085242800000000002 2020-06-12T02:38:16.266 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.266 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.266 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:16.266 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.902814+0000", 2020-06-12T02:38:16.267 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0057806180000000004 2020-06-12T02:38:16.267 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.267 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.267 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:16.267 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.908594+0000", 2020-06-12T02:38:16.267 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021229600000000001 2020-06-12T02:38:16.268 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.268 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.268 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:16.268 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.910717+0000", 2020-06-12T02:38:16.268 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0300999999999999e-05 2020-06-12T02:38:16.269 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.269 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.269 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:16.269 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.910758+0000", 2020-06-12T02:38:16.269 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016165763999999999 2020-06-12T02:38:16.270 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:16.270 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:16.270 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:16.270 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.270 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.270 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8193 3.3s0 3:c2ca77da:::smithi10729407-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e33)", 2020-06-12T02:38:16.271 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.897900+0000", 2020-06-12T02:38:16.271 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 23.157918251000002, 2020-06-12T02:38:16.271 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015951213999999998, 2020-06-12T02:38:16.271 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:16.272 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:16.272 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:16.272 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:16.272 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:16.272 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8193 2020-06-12T02:38:16.273 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.273 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:16.273 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.273 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:16.273 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897900+0000", 2020-06-12T02:38:16.274 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:16.274 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.274 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.274 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:16.274 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897900+0000", 2020-06-12T02:38:16.274 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2331e-05 2020-06-12T02:38:16.275 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.275 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.275 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:16.275 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897923+0000", 2020-06-12T02:38:16.275 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.021e-06 2020-06-12T02:38:16.276 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.276 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.276 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:16.276 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897926+0000", 2020-06-12T02:38:16.276 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4150000000000002e-06 2020-06-12T02:38:16.277 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.277 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.277 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:16.277 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897933+0000", 2020-06-12T02:38:16.278 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2359999999999999e-06 2020-06-12T02:38:16.278 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.279 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.279 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:16.279 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.897935+0000", 2020-06-12T02:38:16.279 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000118301 2020-06-12T02:38:16.279 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.280 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.280 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:16.280 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.898054+0000", 2020-06-12T02:38:16.280 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8914999999999998e-05 2020-06-12T02:38:16.280 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.281 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.281 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:16.281 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.898143+0000", 2020-06-12T02:38:16.281 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011203013 2020-06-12T02:38:16.281 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.281 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.282 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:16.282 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.909346+0000", 2020-06-12T02:38:16.282 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7075000000000003e-05 2020-06-12T02:38:16.282 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.282 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.283 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:16.283 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.909423+0000", 2020-06-12T02:38:16.283 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033600969999999998 2020-06-12T02:38:16.283 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.283 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.284 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:16.284 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.912783+0000", 2020-06-12T02:38:16.284 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6118000000000005e-05 2020-06-12T02:38:16.284 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.284 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.284 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:16.285 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.912869+0000", 2020-06-12T02:38:16.285 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00098269200000000007 2020-06-12T02:38:16.285 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.285 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.285 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:16.286 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.913852+0000", 2020-06-12T02:38:16.286 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015951213999999998 2020-06-12T02:38:16.286 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:16.286 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:16.286 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:16.287 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.287 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.287 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8841 3.3s0 3:ce45c70a:::smithi10729407-7:head [append 0~794624] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:16.287 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:11.820531+0000", 2020-06-12T02:38:16.287 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.2352873219999996, 2020-06-12T02:38:16.288 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019581604999999998, 2020-06-12T02:38:16.288 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:16.288 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:16.288 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:16.288 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:16.289 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:16.289 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8841 2020-06-12T02:38:16.289 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.289 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:16.289 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.290 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:16.290 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.820531+0000", 2020-06-12T02:38:16.290 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:16.291 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.291 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.291 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:16.291 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.820531+0000", 2020-06-12T02:38:16.292 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00070341999999999996 2020-06-12T02:38:16.292 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.293 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.293 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:16.293 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.821235+0000", 2020-06-12T02:38:16.293 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9590000000000001e-06 2020-06-12T02:38:16.293 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.294 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.294 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:16.294 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.821244+0000", 2020-06-12T02:38:16.294 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6135000000000001e-05 2020-06-12T02:38:16.294 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.295 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.295 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:16.295 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.821260+0000", 2020-06-12T02:38:16.295 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5159999999999997e-06 2020-06-12T02:38:16.295 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.295 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.296 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:16.296 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.821267+0000", 2020-06-12T02:38:16.296 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3677999999999998e-05 2020-06-12T02:38:16.296 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.296 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.297 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:16.297 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.821311+0000", 2020-06-12T02:38:16.297 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2262000000000006e-05 2020-06-12T02:38:16.297 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.297 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.297 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:16.298 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.821393+0000", 2020-06-12T02:38:16.298 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00061887800000000001 2020-06-12T02:38:16.298 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.298 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.298 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:16.299 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.822012+0000", 2020-06-12T02:38:16.299 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0043135860000000003 2020-06-12T02:38:16.299 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.299 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.299 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:16.299 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.826326+0000", 2020-06-12T02:38:16.300 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013738409 2020-06-12T02:38:16.300 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.300 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.300 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:16.300 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.840064+0000", 2020-06-12T02:38:16.301 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8761999999999998e-05 2020-06-12T02:38:16.301 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.301 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.301 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:16.301 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.840113+0000", 2020-06-12T02:38:16.302 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019581604999999998 2020-06-12T02:38:16.302 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:16.302 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:16.302 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:16.302 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.302 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.303 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8866 3.3s0 3:c2ca77da:::smithi10729407-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e49)", 2020-06-12T02:38:16.303 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:11.883598+0000", 2020-06-12T02:38:16.303 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.1722210219999996, 2020-06-12T02:38:16.303 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019905586999999999, 2020-06-12T02:38:16.303 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:16.304 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:16.304 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:16.304 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:16.304 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:16.304 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8866 2020-06-12T02:38:16.304 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.305 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:16.305 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.305 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:16.305 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.883598+0000", 2020-06-12T02:38:16.305 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:16.306 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.306 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.307 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:16.307 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.883598+0000", 2020-06-12T02:38:16.314 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2928000000000001e-05 2020-06-12T02:38:16.314 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.314 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.314 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:16.315 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.883621+0000", 2020-06-12T02:38:16.315 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2760000000000001e-06 2020-06-12T02:38:16.315 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.315 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.315 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:16.316 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.883624+0000", 2020-06-12T02:38:16.316 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2239999999999995e-06 2020-06-12T02:38:16.316 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.316 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.316 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:16.317 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.883632+0000", 2020-06-12T02:38:16.317 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2460000000000002e-06 2020-06-12T02:38:16.317 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.317 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.317 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:16.318 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.883634+0000", 2020-06-12T02:38:16.318 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0055082630000000002 2020-06-12T02:38:16.318 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.318 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.318 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:16.319 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.889143+0000", 2020-06-12T02:38:16.319 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9482999999999998e-05 2020-06-12T02:38:16.319 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.319 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.320 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:16.320 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.889222+0000", 2020-06-12T02:38:16.320 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0099390269999999996 2020-06-12T02:38:16.320 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.320 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.321 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:16.321 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.899161+0000", 2020-06-12T02:38:16.321 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000115758 2020-06-12T02:38:16.321 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.321 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.322 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:16.322 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.899277+0000", 2020-06-12T02:38:16.322 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0042263819999999999 2020-06-12T02:38:16.322 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.322 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.323 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:16.323 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.903503+0000", 2020-06-12T02:38:16.323 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019905586999999999 2020-06-12T02:38:16.323 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:16.324 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:16.324 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:16.324 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.324 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.325 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8916 3.3s0 3:ce45c70a:::smithi10729407-7:head [append 0~532480] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:16.325 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:12.042607+0000", 2020-06-12T02:38:16.325 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.0132120660000004, 2020-06-12T02:38:16.325 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016706051, 2020-06-12T02:38:16.325 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:16.326 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:16.326 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:16.326 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:16.326 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:16.327 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8916 2020-06-12T02:38:16.327 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.327 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:16.327 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.327 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:16.328 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.042607+0000", 2020-06-12T02:38:16.328 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:16.328 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.328 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.328 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:16.329 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.042607+0000", 2020-06-12T02:38:16.329 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00066049299999999995 2020-06-12T02:38:16.329 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.329 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.330 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:16.330 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.043267+0000", 2020-06-12T02:38:16.330 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5260000000000004e-06 2020-06-12T02:38:16.330 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.330 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.331 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:16.331 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.043272+0000", 2020-06-12T02:38:16.331 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0111e-05 2020-06-12T02:38:16.331 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.331 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.332 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:16.332 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.043282+0000", 2020-06-12T02:38:16.332 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9699999999999998e-06 2020-06-12T02:38:16.332 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.332 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.333 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:16.333 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.043287+0000", 2020-06-12T02:38:16.333 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00079113700000000003 2020-06-12T02:38:16.333 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.333 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.334 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:16.334 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.044078+0000", 2020-06-12T02:38:16.334 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4621000000000003e-05 2020-06-12T02:38:16.335 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.335 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.335 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:16.335 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.044152+0000", 2020-06-12T02:38:16.335 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0043291140000000002 2020-06-12T02:38:16.336 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.336 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.336 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:16.336 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.048482+0000", 2020-06-12T02:38:16.337 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1329999999999997e-05 2020-06-12T02:38:16.337 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.337 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.337 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:16.337 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.048543+0000", 2020-06-12T02:38:16.338 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000466186 2020-06-12T02:38:16.338 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.338 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.338 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:16.338 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.049009+0000", 2020-06-12T02:38:16.339 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010209073000000001 2020-06-12T02:38:16.339 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.339 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.340 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:16.340 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.059218+0000", 2020-06-12T02:38:16.341 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.041e-05 2020-06-12T02:38:16.341 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.341 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.342 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:16.342 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.059269+0000", 2020-06-12T02:38:16.342 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4079999999999998e-05 2020-06-12T02:38:16.342 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.342 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.343 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:16.343 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.059313+0000", 2020-06-12T02:38:16.343 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016706051 2020-06-12T02:38:16.343 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:16.343 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:16.343 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:16.344 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.344 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.344 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:9129 3.9s0 3:977e4b9d:::smithi10729407-12:head [append 0~532480] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:16.345 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:12.645731+0000", 2020-06-12T02:38:16.345 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.4100876360000001, 2020-06-12T02:38:16.345 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017582474000000001, 2020-06-12T02:38:16.345 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:16.346 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:16.346 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:16.346 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:16.346 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:16.346 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9129 2020-06-12T02:38:16.347 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.347 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:16.347 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.347 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:16.347 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.645731+0000", 2020-06-12T02:38:16.348 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:16.348 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.348 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.348 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:16.348 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.645731+0000", 2020-06-12T02:38:16.349 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00050370799999999996 2020-06-12T02:38:16.349 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.349 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.349 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:16.349 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.646235+0000", 2020-06-12T02:38:16.350 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0410000000000003e-06 2020-06-12T02:38:16.350 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.350 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.350 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:16.350 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.646244+0000", 2020-06-12T02:38:16.351 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6384000000000001e-05 2020-06-12T02:38:16.351 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.351 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.351 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:16.351 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.646260+0000", 2020-06-12T02:38:16.351 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3039999999999999e-06 2020-06-12T02:38:16.352 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.352 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.352 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:16.352 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.646267+0000", 2020-06-12T02:38:16.352 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0970999999999997e-05 2020-06-12T02:38:16.353 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.353 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.353 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:16.353 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.646318+0000", 2020-06-12T02:38:16.353 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000117429 2020-06-12T02:38:16.354 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.354 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.354 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:16.355 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.646436+0000", 2020-06-12T02:38:16.355 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00084515600000000001 2020-06-12T02:38:16.355 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.355 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.355 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:16.356 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.647281+0000", 2020-06-12T02:38:16.356 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003637389 2020-06-12T02:38:16.356 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.356 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.356 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:16.357 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.650918+0000", 2020-06-12T02:38:16.357 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012318766 2020-06-12T02:38:16.357 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.357 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.357 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:16.358 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.663237+0000", 2020-06-12T02:38:16.358 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6326000000000001e-05 2020-06-12T02:38:16.358 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.358 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.358 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:16.358 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.663313+0000", 2020-06-12T02:38:16.359 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017582474000000001 2020-06-12T02:38:16.359 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:16.359 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:16.359 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:16.359 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.360 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.360 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:9130 3.9s0 3:977e4b9d:::smithi10729407-12:head [append 0~794624] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:16.360 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:12.651124+0000", 2020-06-12T02:38:16.360 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.4046949500000001, 2020-06-12T02:38:16.360 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016509482999999998, 2020-06-12T02:38:16.361 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:16.361 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:16.361 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:16.361 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:16.361 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:16.362 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9130 2020-06-12T02:38:16.362 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.362 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:16.362 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.363 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:16.363 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.651124+0000", 2020-06-12T02:38:16.363 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:16.363 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.363 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.364 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:16.364 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.651124+0000", 2020-06-12T02:38:16.364 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001444587 2020-06-12T02:38:16.364 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.364 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.365 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:16.365 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.652568+0000", 2020-06-12T02:38:16.365 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2469999999999997e-06 2020-06-12T02:38:16.365 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.366 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.366 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:16.366 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.652577+0000", 2020-06-12T02:38:16.366 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7112000000000002e-05 2020-06-12T02:38:16.367 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.367 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.367 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:16.367 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.652595+0000", 2020-06-12T02:38:16.367 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9610000000000005e-06 2020-06-12T02:38:16.368 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.368 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.368 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:16.368 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.652603+0000", 2020-06-12T02:38:16.368 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6863000000000002e-05 2020-06-12T02:38:16.368 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.369 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.369 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:16.369 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.652649+0000", 2020-06-12T02:38:16.369 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000111016 2020-06-12T02:38:16.369 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.370 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.370 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:16.370 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.652760+0000", 2020-06-12T02:38:16.370 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00065590500000000003 2020-06-12T02:38:16.370 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.371 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.371 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:16.371 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.653416+0000", 2020-06-12T02:38:16.371 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0099127380000000008 2020-06-12T02:38:16.372 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.372 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.372 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:16.372 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.663329+0000", 2020-06-12T02:38:16.372 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0042080609999999999 2020-06-12T02:38:16.373 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.373 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.373 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:16.373 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.667537+0000", 2020-06-12T02:38:16.374 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5993000000000001e-05 2020-06-12T02:38:16.374 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.374 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.374 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:16.375 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.667633+0000", 2020-06-12T02:38:16.376 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016509482999999998 2020-06-12T02:38:16.376 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:16.376 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:16.376 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:16.376 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.377 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.377 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:9131 3.9s0 3:977e4b9d:::smithi10729407-12:head [append 0~409600] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:16.377 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:12.653674+0000", 2020-06-12T02:38:16.377 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.4021441299999999, 2020-06-12T02:38:16.377 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016092208, 2020-06-12T02:38:16.378 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:16.378 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:16.378 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:16.378 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:16.378 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:16.379 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9131 2020-06-12T02:38:16.379 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.379 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:16.379 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.380 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:16.380 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.653674+0000", 2020-06-12T02:38:16.380 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:16.380 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.380 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.381 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:16.381 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.653674+0000", 2020-06-12T02:38:16.381 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000788918 2020-06-12T02:38:16.381 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.381 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.382 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:16.382 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.654463+0000", 2020-06-12T02:38:16.382 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8680000000000001e-06 2020-06-12T02:38:16.382 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.382 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.382 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:16.383 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.654470+0000", 2020-06-12T02:38:16.383 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3288999999999999e-05 2020-06-12T02:38:16.383 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.383 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.383 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:16.384 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.654484+0000", 2020-06-12T02:38:16.384 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8220000000000003e-06 2020-06-12T02:38:16.384 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.384 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.384 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:16.385 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.654489+0000", 2020-06-12T02:38:16.385 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0041449900000000003 2020-06-12T02:38:16.385 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.386 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.386 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:16.386 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.658634+0000", 2020-06-12T02:38:16.386 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000123695 2020-06-12T02:38:16.386 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.387 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.387 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:16.387 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.658758+0000", 2020-06-12T02:38:16.387 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00069055199999999998 2020-06-12T02:38:16.387 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.388 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.388 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:16.388 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.659449+0000", 2020-06-12T02:38:16.388 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0047984869999999997 2020-06-12T02:38:16.388 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.389 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.389 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:16.389 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.664247+0000", 2020-06-12T02:38:16.389 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0054567399999999999 2020-06-12T02:38:16.390 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.390 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.390 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:16.390 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.669704+0000", 2020-06-12T02:38:16.390 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2847000000000003e-05 2020-06-12T02:38:16.391 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.391 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.391 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:16.391 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.669767+0000", 2020-06-12T02:38:16.391 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016092208 2020-06-12T02:38:16.392 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:16.392 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:16.392 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:16.392 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.392 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.393 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:9132 3.9s0 3:977e4b9d:::smithi10729407-12:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:16.393 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:12.654544+0000", 2020-06-12T02:38:16.393 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.4012748180000001, 2020-06-12T02:38:16.393 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015543141, 2020-06-12T02:38:16.394 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:16.394 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:16.394 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:16.394 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:16.394 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:16.395 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9132 2020-06-12T02:38:16.395 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.395 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:16.395 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.396 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:16.396 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.654544+0000", 2020-06-12T02:38:16.396 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:16.397 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.397 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.397 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:16.397 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.654544+0000", 2020-06-12T02:38:16.397 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4109999999999993e-05 2020-06-12T02:38:16.398 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.398 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.399 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:16.399 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.654618+0000", 2020-06-12T02:38:16.399 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1079999999999998e-06 2020-06-12T02:38:16.400 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.400 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.400 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:16.400 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.654624+0000", 2020-06-12T02:38:16.400 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1759999999999999e-05 2020-06-12T02:38:16.401 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.401 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.401 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:16.401 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.654636+0000", 2020-06-12T02:38:16.401 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.092e-06 2020-06-12T02:38:16.402 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.407 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.407 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:16.407 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.654640+0000", 2020-06-12T02:38:16.407 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0073042920000000004 2020-06-12T02:38:16.408 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.408 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.408 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:16.408 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.661944+0000", 2020-06-12T02:38:16.409 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000115294 2020-06-12T02:38:16.409 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.409 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.409 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:16.409 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.662059+0000", 2020-06-12T02:38:16.410 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000451216 2020-06-12T02:38:16.410 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.410 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.410 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:16.410 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.662511+0000", 2020-06-12T02:38:16.410 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0017387959999999999 2020-06-12T02:38:16.411 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.411 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.411 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:16.411 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.664249+0000", 2020-06-12T02:38:16.411 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0057476649999999999 2020-06-12T02:38:16.412 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.412 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.412 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:16.412 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.669997+0000", 2020-06-12T02:38:16.412 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9808000000000005e-05 2020-06-12T02:38:16.412 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.413 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.413 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:16.413 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.670087+0000", 2020-06-12T02:38:16.413 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015543141 2020-06-12T02:38:16.413 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:16.414 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:16.414 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:16.414 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.414 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.414 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:9133 3.9s0 3:977e4b9d:::smithi10729407-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e49)", 2020-06-12T02:38:16.415 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:12.654680+0000", 2020-06-12T02:38:16.415 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.4011382459999999, 2020-06-12T02:38:16.415 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017220268, 2020-06-12T02:38:16.415 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:16.415 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:16.416 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:16.417 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:16.417 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:16.417 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9133 2020-06-12T02:38:16.417 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.417 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:16.418 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.418 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:16.418 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.654680+0000", 2020-06-12T02:38:16.418 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:16.418 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.419 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.419 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:16.419 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.654680+0000", 2020-06-12T02:38:16.419 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9465999999999998e-05 2020-06-12T02:38:16.419 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.419 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.420 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:16.420 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.654720+0000", 2020-06-12T02:38:16.420 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7320000000000003e-06 2020-06-12T02:38:16.420 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.420 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.421 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:16.421 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.654726+0000", 2020-06-12T02:38:16.421 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3538e-05 2020-06-12T02:38:16.421 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.421 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.421 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:16.422 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.654739+0000", 2020-06-12T02:38:16.422 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5689999999999998e-06 2020-06-12T02:38:16.422 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.422 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.422 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:16.423 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.654744+0000", 2020-06-12T02:38:16.423 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0086310060000000001 2020-06-12T02:38:16.423 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.423 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.423 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:16.424 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.663375+0000", 2020-06-12T02:38:16.424 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00026180499999999999 2020-06-12T02:38:16.424 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.424 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.424 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:16.424 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.663636+0000", 2020-06-12T02:38:16.425 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0071786749999999998 2020-06-12T02:38:16.425 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.425 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.425 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:16.425 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.670815+0000", 2020-06-12T02:38:16.426 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1920000000000002e-05 2020-06-12T02:38:16.426 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.426 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.426 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:16.426 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.670897+0000", 2020-06-12T02:38:16.427 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001003557 2020-06-12T02:38:16.427 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:16.427 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:16.427 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:16.427 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.671901+0000", 2020-06-12T02:38:16.427 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017220268 2020-06-12T02:38:16.428 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:16.428 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:16.428 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:16.428 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:16.428 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:16.428 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:16.432 INFO:tasks.rados.rados.0.smithi107.stdout:4396: done (1 left) 2020-06-12T02:38:16.432 INFO:tasks.rados.rados.0.smithi107.stdout:4398: done (0 left) 2020-06-12T02:38:16.433 INFO:tasks.rados.rados.0.smithi107.stdout:4399: delete oid 12 current snap is 0 2020-06-12T02:38:16.433 INFO:tasks.rados.rados.0.smithi107.stdout:4399: done (0 left) 2020-06-12T02:38:16.433 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:16.433 INFO:tasks.rados.rados.0.smithi107.stdout:4400: seq_num 4399 ranges {1261568=483328,1744896=696320,2441216=507904} 2020-06-12T02:38:16.433 INFO:tasks.rados.rados.0.smithi107.stdout:4400: writing smithi10729407-4 from 1261568 to 1744896 tid 1 2020-06-12T02:38:16.434 INFO:tasks.rados.rados.0.smithi107.stdout:4400: writing smithi10729407-4 from 1744896 to 2441216 tid 2 2020-06-12T02:38:16.434 INFO:tasks.rados.rados.0.smithi107.stdout:4400: writing smithi10729407-4 from 2441216 to 2949120 tid 3 2020-06-12T02:38:16.434 INFO:tasks.rados.rados.0.smithi107.stdout:4401: delete oid 8 current snap is 0 2020-06-12T02:38:16.434 INFO:tasks.rados.rados.0.smithi107.stdout:4400: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:16.434 INFO:tasks.rados.rados.0.smithi107.stdout:4401: done (1 left) 2020-06-12T02:38:16.435 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:16.435 INFO:tasks.rados.rados.0.smithi107.stdout:4402: seq_num 4401 ranges {0=516096,516096=466944,983040=696320,1679360=40960} 2020-06-12T02:38:16.435 INFO:tasks.rados.rados.0.smithi107.stdout:4402: writing smithi10729407-17 from 0 to 516096 tid 1 2020-06-12T02:38:16.435 INFO:tasks.rados.rados.0.smithi107.stdout:4402: writing smithi10729407-17 from 516096 to 983040 tid 2 2020-06-12T02:38:16.435 INFO:tasks.rados.rados.0.smithi107.stdout:4402: writing smithi10729407-17 from 983040 to 1679360 tid 3 2020-06-12T02:38:16.435 INFO:tasks.rados.rados.0.smithi107.stdout:4402: writing smithi10729407-17 from 1679360 to 1720320 tid 4 2020-06-12T02:38:16.436 INFO:tasks.rados.rados.0.smithi107.stdout:4403: delete oid 12 current snap is 0 2020-06-12T02:38:16.436 INFO:tasks.rados.rados.0.smithi107.stdout:4400: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:16.436 INFO:tasks.rados.rados.0.smithi107.stdout:4400: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:16.436 INFO:tasks.rados.rados.0.smithi107.stdout:4400: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:16.436 INFO:tasks.rados.rados.0.smithi107.stdout:4402: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:16.437 INFO:tasks.rados.rados.0.smithi107.stdout:4400: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:16.437 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 802 (ObjNum 4399 snap 0 seq_num 4399) dirty exists 2020-06-12T02:38:16.437 INFO:tasks.rados.rados.0.smithi107.stdout:4400: left oid 4 (ObjNum 4399 snap 0 seq_num 4399) 2020-06-12T02:38:16.437 INFO:tasks.rados.rados.0.smithi107.stdout:4402: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:16.437 INFO:tasks.rados.rados.0.smithi107.stdout:4400: done (2 left) 2020-06-12T02:38:16.437 INFO:tasks.rados.rados.0.smithi107.stdout:4403: done (1 left) 2020-06-12T02:38:16.438 INFO:tasks.rados.rados.0.smithi107.stdout:4404: delete oid 14 current snap is 0 2020-06-12T02:38:16.438 INFO:tasks.rados.rados.0.smithi107.stdout:4404: done (1 left) 2020-06-12T02:38:16.438 INFO:tasks.rados.rados.0.smithi107.stdout:4405: delete oid 1 current snap is 0 2020-06-12T02:38:16.438 INFO:tasks.rados.rados.0.smithi107.stdout:4405: done (1 left) 2020-06-12T02:38:16.438 INFO:tasks.rados.rados.0.smithi107.stdout:4406: delete oid 3 current snap is 0 2020-06-12T02:38:16.438 INFO:tasks.rados.rados.0.smithi107.stdout:4402: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:16.439 INFO:tasks.rados.rados.0.smithi107.stdout:4402: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:16.439 INFO:tasks.rados.rados.0.smithi107.stdout:4402: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:16.439 INFO:tasks.rados.rados.0.smithi107.stdout:4402: finishing write tid 7 to smithi10729407-17 2020-06-12T02:38:16.439 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 807 (ObjNum 4401 snap 0 seq_num 4401) dirty exists 2020-06-12T02:38:16.439 INFO:tasks.rados.rados.0.smithi107.stdout:4402: left oid 17 (ObjNum 4401 snap 0 seq_num 4401) 2020-06-12T02:38:16.440 INFO:tasks.rados.rados.0.smithi107.stdout:4402: done (1 left) 2020-06-12T02:38:16.440 INFO:tasks.rados.rados.0.smithi107.stdout:4406: done (0 left) 2020-06-12T02:38:16.440 INFO:tasks.rados.rados.0.smithi107.stdout:4407: delete oid 11 current snap is 0 2020-06-12T02:38:16.440 INFO:tasks.rados.rados.0.smithi107.stdout:4407: done (0 left) 2020-06-12T02:38:16.440 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:16.440 INFO:tasks.rados.rados.0.smithi107.stdout:4408: seq_num 4407 ranges {0=614400,614400=622592,1236992=581632} 2020-06-12T02:38:16.441 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:16.427+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:16.441 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-06-12T02:38:16.442 INFO:tasks.rados.rados.0.smithi107.stdout:4408: writing smithi10729407-3 from 0 to 614400 tid 1 2020-06-12T02:38:16.443 INFO:tasks.rados.rados.0.smithi107.stdout:4408: writing smithi10729407-3 from 614400 to 1236992 tid 2 2020-06-12T02:38:16.443 INFO:tasks.rados.rados.0.smithi107.stdout:4408: writing smithi10729407-3 from 1236992 to 1818624 tid 3 2020-06-12T02:38:16.443 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:38:16.443 INFO:tasks.rados.rados.0.smithi107.stdout:4409: seq_num 4408 ranges {0=630784,630784=507904,1138688=450560,1589248=245760} 2020-06-12T02:38:16.446 INFO:tasks.rados.rados.0.smithi107.stdout:4409: writing smithi10729407-20 from 0 to 630784 tid 1 2020-06-12T02:38:16.450 INFO:tasks.rados.rados.0.smithi107.stdout:4409: writing smithi10729407-20 from 630784 to 1138688 tid 2 2020-06-12T02:38:16.453 INFO:tasks.rados.rados.0.smithi107.stdout:4409: writing smithi10729407-20 from 1138688 to 1589248 tid 3 2020-06-12T02:38:16.455 INFO:tasks.rados.rados.0.smithi107.stdout:4409: writing smithi10729407-20 from 1589248 to 1835008 tid 4 2020-06-12T02:38:16.455 INFO:tasks.rados.rados.0.smithi107.stdout:4410: delete oid 4 current snap is 0 2020-06-12T02:38:16.455 INFO:tasks.rados.rados.0.smithi107.stdout:4408: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:16.455 INFO:tasks.rados.rados.0.smithi107.stdout:4408: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:16.456 INFO:tasks.rados.rados.0.smithi107.stdout:4408: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:16.456 INFO:tasks.rados.rados.0.smithi107.stdout:4408: finishing write tid 5 to smithi10729407-3 2020-06-12T02:38:16.456 INFO:tasks.rados.rados.0.smithi107.stdout:4408: finishing write tid 6 to smithi10729407-3 2020-06-12T02:38:16.456 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 786 (ObjNum 4407 snap 0 seq_num 4407) dirty exists 2020-06-12T02:38:16.456 INFO:tasks.rados.rados.0.smithi107.stdout:4408: left oid 3 (ObjNum 4407 snap 0 seq_num 4407) 2020-06-12T02:38:16.457 INFO:tasks.rados.rados.0.smithi107.stdout:4409: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:16.457 INFO:tasks.rados.rados.0.smithi107.stdout:4409: finishing write tid 2 to smithi10729407-20 2020-06-12T02:38:16.458 INFO:tasks.rados.rados.0.smithi107.stdout:4408: done (2 left) 2020-06-12T02:38:16.458 INFO:tasks.rados.rados.0.smithi107.stdout:4410: done (1 left) 2020-06-12T02:38:16.458 INFO:tasks.rados.rados.0.smithi107.stdout:4411: delete oid 18 current snap is 0 2020-06-12T02:38:16.460 INFO:tasks.rados.rados.0.smithi107.stdout:4411: done (1 left) 2020-06-12T02:38:16.461 INFO:tasks.rados.rados.0.smithi107.stdout:4412: delete oid 12 current snap is 0 2020-06-12T02:38:16.461 INFO:tasks.rados.rados.0.smithi107.stdout:4409: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:16.463 INFO:tasks.rados.rados.0.smithi107.stdout:4409: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:16.463 INFO:tasks.rados.rados.0.smithi107.stdout:4409: finishing write tid 6 to smithi10729407-20 2020-06-12T02:38:16.466 INFO:tasks.rados.rados.0.smithi107.stdout:4412: done (1 left) 2020-06-12T02:38:16.466 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:38:16.466 INFO:tasks.rados.rados.0.smithi107.stdout:4413: seq_num 4412 ranges {0=696320,696320=491520,1187840=425984,1613824=286720} 2020-06-12T02:38:16.470 INFO:tasks.rados.rados.0.smithi107.stdout:4413: writing smithi10729407-10 from 0 to 696320 tid 1 2020-06-12T02:38:16.472 INFO:tasks.rados.rados.0.smithi107.stdout:4413: writing smithi10729407-10 from 696320 to 1187840 tid 2 2020-06-12T02:38:16.475 INFO:tasks.rados.rados.0.smithi107.stdout:4413: writing smithi10729407-10 from 1187840 to 1613824 tid 3 2020-06-12T02:38:16.478 INFO:tasks.rados.rados.0.smithi107.stdout:4413: writing smithi10729407-10 from 1613824 to 1900544 tid 4 2020-06-12T02:38:16.478 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:38:16.479 INFO:tasks.rados.rados.0.smithi107.stdout:4414: seq_num 4413 ranges {0=712704,712704=778240,1490944=425984} 2020-06-12T02:38:16.481 INFO:tasks.rados.rados.0.smithi107.stdout:4414: writing smithi10729407-11 from 0 to 712704 tid 1 2020-06-12T02:38:16.485 INFO:tasks.rados.rados.0.smithi107.stdout:4414: writing smithi10729407-11 from 712704 to 1490944 tid 2 2020-06-12T02:38:16.488 INFO:tasks.rados.rados.0.smithi107.stdout:4414: writing smithi10729407-11 from 1490944 to 1916928 tid 3 2020-06-12T02:38:16.488 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:16.489 INFO:tasks.rados.rados.0.smithi107.stdout:4415: seq_num 4414 ranges {0=729088,729088=663552,1392640=442368,1835008=98304} 2020-06-12T02:38:16.493 INFO:tasks.rados.rados.0.smithi107.stdout:4415: writing smithi10729407-12 from 0 to 729088 tid 1 2020-06-12T02:38:16.497 INFO:tasks.rados.rados.0.smithi107.stdout:4415: writing smithi10729407-12 from 729088 to 1392640 tid 2 2020-06-12T02:38:16.500 INFO:tasks.rados.rados.0.smithi107.stdout:4415: writing smithi10729407-12 from 1392640 to 1835008 tid 3 2020-06-12T02:38:16.501 INFO:tasks.rados.rados.0.smithi107.stdout:4415: writing smithi10729407-12 from 1835008 to 1933312 tid 4 2020-06-12T02:38:16.501 INFO:tasks.rados.rados.0.smithi107.stdout:4416: delete oid 9 current snap is 0 2020-06-12T02:38:16.501 INFO:tasks.rados.rados.0.smithi107.stdout:4409: finishing write tid 7 to smithi10729407-20 2020-06-12T02:38:16.502 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 328 (ObjNum 4408 snap 0 seq_num 4408) dirty exists 2020-06-12T02:38:16.502 INFO:tasks.rados.rados.0.smithi107.stdout:4409: left oid 20 (ObjNum 4408 snap 0 seq_num 4408) 2020-06-12T02:38:16.502 INFO:tasks.rados.rados.0.smithi107.stdout:4414: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:16.502 INFO:tasks.rados.rados.0.smithi107.stdout:4414: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:16.502 INFO:tasks.rados.rados.0.smithi107.stdout:4414: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:16.503 INFO:tasks.rados.rados.0.smithi107.stdout:4414: finishing write tid 5 to smithi10729407-11 2020-06-12T02:38:16.503 INFO:tasks.rados.rados.0.smithi107.stdout:4414: finishing write tid 6 to smithi10729407-11 2020-06-12T02:38:16.504 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 792 (ObjNum 4413 snap 0 seq_num 4413) dirty exists 2020-06-12T02:38:16.504 INFO:tasks.rados.rados.0.smithi107.stdout:4414: left oid 11 (ObjNum 4413 snap 0 seq_num 4413) 2020-06-12T02:38:16.504 INFO:tasks.rados.rados.0.smithi107.stdout:4409: done (4 left) 2020-06-12T02:38:16.504 INFO:tasks.rados.rados.0.smithi107.stdout:4414: done (3 left) 2020-06-12T02:38:16.504 INFO:tasks.rados.rados.0.smithi107.stdout:4416: done (2 left) 2020-06-12T02:38:16.504 INFO:tasks.rados.rados.0.smithi107.stdout:4417: delete oid 5 current snap is 0 2020-06-12T02:38:16.505 INFO:tasks.rados.rados.0.smithi107.stdout:4415: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:16.506 INFO:tasks.rados.rados.0.smithi107.stdout:4417: done (2 left) 2020-06-12T02:38:16.506 INFO:tasks.rados.rados.0.smithi107.stdout:4418: delete oid 1 current snap is 0 2020-06-12T02:38:16.510 INFO:tasks.rados.rados.0.smithi107.stdout:4415: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:16.511 INFO:tasks.rados.rados.0.smithi107.stdout:4415: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:16.512 INFO:tasks.rados.rados.0.smithi107.stdout:4418: done (2 left) 2020-06-12T02:38:16.512 INFO:tasks.rados.rados.0.smithi107.stdout:4419: delete oid 4 current snap is 0 2020-06-12T02:38:16.514 INFO:tasks.rados.rados.0.smithi107.stdout:4415: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:16.515 INFO:tasks.rados.rados.0.smithi107.stdout:4415: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:16.515 INFO:tasks.rados.rados.0.smithi107.stdout:4419: done (2 left) 2020-06-12T02:38:16.515 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:38:16.515 INFO:tasks.rados.rados.0.smithi107.stdout:4420: seq_num 4419 ranges {1835008=417792,2252800=532480,2785280=630784,3416064=442368} 2020-06-12T02:38:16.526 INFO:tasks.rados.rados.0.smithi107.stdout:4420: writing smithi10729407-20 from 1835008 to 2252800 tid 1 2020-06-12T02:38:16.530 INFO:tasks.rados.rados.0.smithi107.stdout:4420: writing smithi10729407-20 from 2252800 to 2785280 tid 2 2020-06-12T02:38:16.530 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:16.528+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:16.534 INFO:tasks.rados.rados.0.smithi107.stdout:4420: writing smithi10729407-20 from 2785280 to 3416064 tid 3 2020-06-12T02:38:16.538 INFO:tasks.rados.rados.0.smithi107.stdout:4420: writing smithi10729407-20 from 3416064 to 3858432 tid 4 2020-06-12T02:38:16.539 INFO:tasks.rados.rados.0.smithi107.stdout:4421: delete oid 16 current snap is 0 2020-06-12T02:38:16.539 INFO:tasks.rados.rados.0.smithi107.stdout:4415: finishing write tid 7 to smithi10729407-12 2020-06-12T02:38:16.539 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 750 (ObjNum 4414 snap 0 seq_num 4414) dirty exists 2020-06-12T02:38:16.539 INFO:tasks.rados.rados.0.smithi107.stdout:4415: left oid 12 (ObjNum 4414 snap 0 seq_num 4414) 2020-06-12T02:38:16.540 INFO:tasks.rados.rados.0.smithi107.stdout:4420: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:16.540 INFO:tasks.rados.rados.0.smithi107.stdout:4415: done (3 left) 2020-06-12T02:38:16.540 INFO:tasks.rados.rados.0.smithi107.stdout:4421: done (2 left) 2020-06-12T02:38:16.541 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:16.541 INFO:tasks.rados.rados.0.smithi107.stdout:4422: seq_num 4421 ranges {1564672=450560,2015232=704512,2719744=598016,3317760=303104} 2020-06-12T02:38:16.551 INFO:tasks.rados.rados.0.smithi107.stdout:4422: writing smithi10729407-19 from 1564672 to 2015232 tid 1 2020-06-12T02:38:16.555 INFO:tasks.rados.rados.0.smithi107.stdout:4422: writing smithi10729407-19 from 2015232 to 2719744 tid 2 2020-06-12T02:38:16.559 INFO:tasks.rados.rados.0.smithi107.stdout:4422: writing smithi10729407-19 from 2719744 to 3317760 tid 3 2020-06-12T02:38:16.562 INFO:tasks.rados.rados.0.smithi107.stdout:4422: writing smithi10729407-19 from 3317760 to 3620864 tid 4 2020-06-12T02:38:16.562 INFO:tasks.rados.rados.0.smithi107.stdout:4423: delete oid 18 current snap is 0 2020-06-12T02:38:16.562 INFO:tasks.rados.rados.0.smithi107.stdout:4420: finishing write tid 2 to smithi10729407-20 2020-06-12T02:38:16.562 INFO:tasks.rados.rados.0.smithi107.stdout:4420: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:16.562 INFO:tasks.rados.rados.0.smithi107.stdout:4420: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:16.563 INFO:tasks.rados.rados.0.smithi107.stdout:4420: finishing write tid 6 to smithi10729407-20 2020-06-12T02:38:16.563 INFO:tasks.rados.rados.0.smithi107.stdout:4420: finishing write tid 7 to smithi10729407-20 2020-06-12T02:38:16.563 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 333 (ObjNum 4419 snap 0 seq_num 4419) dirty exists 2020-06-12T02:38:16.563 INFO:tasks.rados.rados.0.smithi107.stdout:4420: left oid 20 (ObjNum 4419 snap 0 seq_num 4419) 2020-06-12T02:38:16.563 INFO:tasks.rados.rados.0.smithi107.stdout:4422: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:16.564 INFO:tasks.rados.rados.0.smithi107.stdout:4420: done (3 left) 2020-06-12T02:38:16.564 INFO:tasks.rados.rados.0.smithi107.stdout:4423: done (2 left) 2020-06-12T02:38:16.564 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:38:16.565 INFO:tasks.rados.rados.0.smithi107.stdout:4424: seq_num 4423 ranges {3858432=483328,4341760=466944,4808704=614400,5423104=524288} 2020-06-12T02:38:16.587 INFO:tasks.rados.rados.0.smithi107.stdout:4424: writing smithi10729407-20 from 3858432 to 4341760 tid 1 2020-06-12T02:38:16.590 INFO:tasks.rados.rados.0.smithi107.stdout:4424: writing smithi10729407-20 from 4341760 to 4808704 tid 2 2020-06-12T02:38:16.594 INFO:tasks.rados.rados.0.smithi107.stdout:4424: writing smithi10729407-20 from 4808704 to 5423104 tid 3 2020-06-12T02:38:16.598 INFO:tasks.rados.rados.0.smithi107.stdout:4424: writing smithi10729407-20 from 5423104 to 5947392 tid 4 2020-06-12T02:38:16.598 INFO:tasks.rados.rados.0.smithi107.stdout:4425: delete oid 17 current snap is 0 2020-06-12T02:38:16.598 INFO:tasks.rados.rados.0.smithi107.stdout:4422: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:16.599 INFO:tasks.rados.rados.0.smithi107.stdout:4422: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:16.599 INFO:tasks.rados.rados.0.smithi107.stdout:4422: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:16.599 INFO:tasks.rados.rados.0.smithi107.stdout:4422: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:16.599 INFO:tasks.rados.rados.0.smithi107.stdout:4422: finishing write tid 7 to smithi10729407-19 2020-06-12T02:38:16.599 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 755 (ObjNum 4421 snap 0 seq_num 4421) dirty exists 2020-06-12T02:38:16.600 INFO:tasks.rados.rados.0.smithi107.stdout:4422: left oid 19 (ObjNum 4421 snap 0 seq_num 4421) 2020-06-12T02:38:16.600 INFO:tasks.rados.rados.0.smithi107.stdout:4424: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:16.600 INFO:tasks.rados.rados.0.smithi107.stdout:4422: done (3 left) 2020-06-12T02:38:16.600 INFO:tasks.rados.rados.0.smithi107.stdout:4425: done (2 left) 2020-06-12T02:38:16.601 INFO:tasks.rados.rados.0.smithi107.stdout:4426: delete oid 3 current snap is 0 2020-06-12T02:38:16.604 INFO:tasks.rados.rados.0.smithi107.stdout:4426: done (2 left) 2020-06-12T02:38:16.605 INFO:tasks.rados.rados.0.smithi107.stdout:4427: delete oid 11 current snap is 0 2020-06-12T02:38:16.605 INFO:tasks.rados.rados.0.smithi107.stdout:4424: finishing write tid 2 to smithi10729407-20 2020-06-12T02:38:16.606 INFO:tasks.rados.rados.0.smithi107.stdout:4424: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:16.606 INFO:tasks.rados.rados.0.smithi107.stdout:4424: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:16.611 INFO:tasks.rados.rados.0.smithi107.stdout:4424: finishing write tid 6 to smithi10729407-20 2020-06-12T02:38:16.611 INFO:tasks.rados.rados.0.smithi107.stdout:4424: finishing write tid 7 to smithi10729407-20 2020-06-12T02:38:16.612 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 338 (ObjNum 4423 snap 0 seq_num 4423) dirty exists 2020-06-12T02:38:16.612 INFO:tasks.rados.rados.0.smithi107.stdout:4424: left oid 20 (ObjNum 4423 snap 0 seq_num 4423) 2020-06-12T02:38:16.612 INFO:tasks.rados.rados.0.smithi107.stdout:4424: done (2 left) 2020-06-12T02:38:16.612 INFO:tasks.rados.rados.0.smithi107.stdout:4427: done (1 left) 2020-06-12T02:38:16.612 INFO:tasks.rados.rados.0.smithi107.stdout:4428: delete oid 1 current snap is 0 2020-06-12T02:38:16.622 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:16.622 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:38:16.630 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2020-06-12T02:38:16.630 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:16.632 INFO:tasks.rados.rados.0.smithi107.stdout:4428: done (1 left) 2020-06-12T02:38:16.632 INFO:tasks.rados.rados.0.smithi107.stdout:4429: delete oid 4 current snap is 0 2020-06-12T02:38:16.632 INFO:tasks.rados.rados.0.smithi107.stdout:4429: done (1 left) 2020-06-12T02:38:16.633 INFO:tasks.rados.rados.0.smithi107.stdout:4430: delete oid 9 current snap is 0 2020-06-12T02:38:16.633 INFO:tasks.rados.rados.0.smithi107.stdout:4430: done (1 left) 2020-06-12T02:38:16.633 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:38:16.633 INFO:tasks.rados.rados.0.smithi107.stdout:4431: seq_num 4430 ranges {0=598016,598016=507904,1105920=770048,1875968=327680} 2020-06-12T02:38:16.633 INFO:tasks.rados.rados.0.smithi107.stdout:4431: writing smithi10729407-5 from 0 to 598016 tid 1 2020-06-12T02:38:16.634 INFO:tasks.rados.rados.0.smithi107.stdout:4431: writing smithi10729407-5 from 598016 to 1105920 tid 2 2020-06-12T02:38:16.634 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:16.628+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:16.635 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-06-12T02:38:16.636 INFO:tasks.rados.rados.0.smithi107.stdout:4431: writing smithi10729407-5 from 1105920 to 1875968 tid 3 2020-06-12T02:38:16.636 INFO:tasks.rados.rados.0.smithi107.stdout:4431: writing smithi10729407-5 from 1875968 to 2203648 tid 4 2020-06-12T02:38:16.636 INFO:tasks.rados.rados.0.smithi107.stdout:4432: delete oid 18 current snap is 0 2020-06-12T02:38:16.636 INFO:tasks.rados.rados.0.smithi107.stdout:4431: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:16.637 INFO:tasks.rados.rados.0.smithi107.stdout:4432: done (2 left) 2020-06-12T02:38:16.637 INFO:tasks.rados.rados.0.smithi107.stdout:4433: delete oid 17 current snap is 0 2020-06-12T02:38:16.638 INFO:tasks.rados.rados.0.smithi107.stdout:4433: done (2 left) 2020-06-12T02:38:16.638 INFO:tasks.rados.rados.0.smithi107.stdout:4434: delete oid 4 current snap is 0 2020-06-12T02:38:16.640 INFO:tasks.rados.rados.0.smithi107.stdout:4434: done (2 left) 2020-06-12T02:38:16.640 INFO:tasks.rados.rados.0.smithi107.stdout:4435: delete oid 11 current snap is 0 2020-06-12T02:38:16.641 INFO:tasks.rados.rados.0.smithi107.stdout:4431: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:16.641 INFO:tasks.rados.rados.0.smithi107.stdout:4431: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:16.642 INFO:tasks.rados.rados.0.smithi107.stdout:4435: done (2 left) 2020-06-12T02:38:16.643 INFO:tasks.rados.rados.0.smithi107.stdout:4436: delete oid 18 current snap is 0 2020-06-12T02:38:16.643 INFO:tasks.rados.rados.0.smithi107.stdout:4431: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:16.644 INFO:tasks.rados.rados.0.smithi107.stdout:4431: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:16.645 INFO:tasks.rados.rados.0.smithi107.stdout:4436: done (2 left) 2020-06-12T02:38:16.645 INFO:tasks.rados.rados.0.smithi107.stdout:4437: delete oid 17 current snap is 0 2020-06-12T02:38:16.646 INFO:tasks.rados.rados.0.smithi107.stdout:4437: done (2 left) 2020-06-12T02:38:16.646 INFO:tasks.rados.rados.0.smithi107.stdout:4438: delete oid 4 current snap is 0 2020-06-12T02:38:16.647 INFO:tasks.rados.rados.0.smithi107.stdout:4431: finishing write tid 7 to smithi10729407-5 2020-06-12T02:38:16.647 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 447 (ObjNum 4430 snap 0 seq_num 4430) dirty exists 2020-06-12T02:38:16.648 INFO:tasks.rados.rados.0.smithi107.stdout:4431: left oid 5 (ObjNum 4430 snap 0 seq_num 4430) 2020-06-12T02:38:16.648 INFO:tasks.rados.rados.0.smithi107.stdout:4431: done (2 left) 2020-06-12T02:38:16.648 INFO:tasks.rados.rados.0.smithi107.stdout:4438: done (1 left) 2020-06-12T02:38:16.649 INFO:tasks.rados.rados.0.smithi107.stdout:4439: delete oid 19 current snap is 0 2020-06-12T02:38:16.650 INFO:tasks.rados.rados.0.smithi107.stdout:4439: done (1 left) 2020-06-12T02:38:16.650 INFO:tasks.rados.rados.0.smithi107.stdout:4440: delete oid 13 current snap is 0 2020-06-12T02:38:16.652 INFO:tasks.rados.rados.0.smithi107.stdout:4440: done (1 left) 2020-06-12T02:38:16.652 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:16.653 INFO:tasks.rados.rados.0.smithi107.stdout:4441: seq_num 4440 ranges {0=770048,770048=655360,1425408=696320,2121728=253952} 2020-06-12T02:38:16.656 INFO:tasks.rados.rados.0.smithi107.stdout:4441: writing smithi10729407-4 from 0 to 770048 tid 1 2020-06-12T02:38:16.683 INFO:tasks.rados.rados.0.smithi107.stdout:4441: writing smithi10729407-4 from 770048 to 1425408 tid 2 2020-06-12T02:38:16.683 INFO:tasks.rados.rados.0.smithi107.stdout:4441: writing smithi10729407-4 from 1425408 to 2121728 tid 3 2020-06-12T02:38:16.684 INFO:tasks.rados.rados.0.smithi107.stdout:4441: writing smithi10729407-4 from 2121728 to 2375680 tid 4 2020-06-12T02:38:16.684 INFO:tasks.rados.rados.0.smithi107.stdout:4442: delete oid 12 current snap is 0 2020-06-12T02:38:16.684 INFO:tasks.rados.rados.0.smithi107.stdout:4441: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:16.684 INFO:tasks.rados.rados.0.smithi107.stdout:4442: done (2 left) 2020-06-12T02:38:16.684 INFO:tasks.rados.rados.0.smithi107.stdout:4443: delete oid 11 current snap is 0 2020-06-12T02:38:16.685 INFO:tasks.rados.rados.0.smithi107.stdout:4441: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:16.685 INFO:tasks.rados.rados.0.smithi107.stdout:4441: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:16.685 INFO:tasks.rados.rados.0.smithi107.stdout:4443: done (2 left) 2020-06-12T02:38:16.685 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:38:16.685 INFO:tasks.rados.rados.0.smithi107.stdout:4444: seq_num 4443 ranges {5947392=16384} 2020-06-12T02:38:16.705 INFO:tasks.rados.rados.0.smithi107.stdout:4444: writing smithi10729407-20 from 5947392 to 5963776 tid 1 2020-06-12T02:38:16.705 INFO:tasks.rados.rados.0.smithi107.stdout:4445: delete oid 2 current snap is 0 2020-06-12T02:38:16.705 INFO:tasks.rados.rados.0.smithi107.stdout:4441: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:16.705 INFO:tasks.rados.rados.0.smithi107.stdout:4441: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:16.706 INFO:tasks.rados.rados.0.smithi107.stdout:4441: finishing write tid 7 to smithi10729407-4 2020-06-12T02:38:16.706 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 820 (ObjNum 4440 snap 0 seq_num 4440) dirty exists 2020-06-12T02:38:16.706 INFO:tasks.rados.rados.0.smithi107.stdout:4441: left oid 4 (ObjNum 4440 snap 0 seq_num 4440) 2020-06-12T02:38:16.707 INFO:tasks.rados.rados.0.smithi107.stdout:4444: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:16.707 INFO:tasks.rados.rados.0.smithi107.stdout:4441: done (3 left) 2020-06-12T02:38:16.707 INFO:tasks.rados.rados.0.smithi107.stdout:4445: done (2 left) 2020-06-12T02:38:16.708 INFO:tasks.rados.rados.0.smithi107.stdout:4446: delete oid 2 current snap is 0 2020-06-12T02:38:16.709 INFO:tasks.rados.rados.0.smithi107.stdout:4446: done (2 left) 2020-06-12T02:38:16.710 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:38:16.710 INFO:tasks.rados.rados.0.smithi107.stdout:4447: seq_num 4446 ranges {0=65536} 2020-06-12T02:38:16.711 INFO:tasks.rados.rados.0.smithi107.stdout:4447: writing smithi10729407-7 from 0 to 65536 tid 1 2020-06-12T02:38:16.711 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:16.711 INFO:tasks.rados.rados.0.smithi107.stdout:4448: seq_num 4447 ranges {0=81920} 2020-06-12T02:38:16.711 INFO:tasks.rados.rados.0.smithi107.stdout:4448: writing smithi10729407-13 from 0 to 81920 tid 1 2020-06-12T02:38:16.711 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:38:16.712 INFO:tasks.rados.rados.0.smithi107.stdout:4449: seq_num 4448 ranges {2203648=98304} 2020-06-12T02:38:16.724 INFO:tasks.rados.rados.0.smithi107.stdout:4449: writing smithi10729407-5 from 2203648 to 2301952 tid 1 2020-06-12T02:38:16.724 INFO:tasks.rados.rados.0.smithi107.stdout:4450: delete oid 4 current snap is 0 2020-06-12T02:38:16.725 INFO:tasks.rados.rados.0.smithi107.stdout:4444: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:16.725 INFO:tasks.rados.rados.0.smithi107.stdout:4444: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:16.725 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 340 (ObjNum 4443 snap 0 seq_num 4443) dirty exists 2020-06-12T02:38:16.726 INFO:tasks.rados.rados.0.smithi107.stdout:4444: left oid 20 (ObjNum 4443 snap 0 seq_num 4443) 2020-06-12T02:38:16.726 INFO:tasks.rados.rados.0.smithi107.stdout:4447: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:16.726 INFO:tasks.rados.rados.0.smithi107.stdout:4448: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:16.726 INFO:tasks.rados.rados.0.smithi107.stdout:4447: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:16.727 INFO:tasks.rados.rados.0.smithi107.stdout:4448: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:16.727 INFO:tasks.rados.rados.0.smithi107.stdout:4447: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:16.727 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 834 (ObjNum 4446 snap 0 seq_num 4446) dirty exists 2020-06-12T02:38:16.727 INFO:tasks.rados.rados.0.smithi107.stdout:4447: left oid 7 (ObjNum 4446 snap 0 seq_num 4446) 2020-06-12T02:38:16.728 INFO:tasks.rados.rados.0.smithi107.stdout:4448: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:16.728 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 835 (ObjNum 4447 snap 0 seq_num 4447) dirty exists 2020-06-12T02:38:16.728 INFO:tasks.rados.rados.0.smithi107.stdout:4448: left oid 13 (ObjNum 4447 snap 0 seq_num 4447) 2020-06-12T02:38:16.729 INFO:tasks.rados.rados.0.smithi107.stdout:4444: done (5 left) 2020-06-12T02:38:16.730 INFO:tasks.rados.rados.0.smithi107.stdout:4447: done (4 left) 2020-06-12T02:38:16.730 INFO:tasks.rados.rados.0.smithi107.stdout:4448: done (3 left) 2020-06-12T02:38:16.730 INFO:tasks.rados.rados.0.smithi107.stdout:4450: done (2 left) 2020-06-12T02:38:16.731 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:16.731 INFO:tasks.rados.rados.0.smithi107.stdout:4451: seq_num 4450 ranges {0=131072} 2020-06-12T02:38:16.731 INFO:tasks.rados.rados.0.smithi107.stdout:4451: writing smithi10729407-11 from 0 to 131072 tid 1 2020-06-12T02:38:16.731 INFO:tasks.rados.rados.0.smithi107.stdout:4452: delete oid 9 current snap is 0 2020-06-12T02:38:16.731 INFO:tasks.rados.rados.0.smithi107.stdout:4449: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:16.732 INFO:tasks.rados.rados.0.smithi107.stdout:4451: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:16.732 INFO:tasks.rados.rados.0.smithi107.stdout:4452: done (3 left) 2020-06-12T02:38:16.733 INFO:tasks.rados.rados.0.smithi107.stdout:4453: delete oid 13 current snap is 0 2020-06-12T02:38:16.733 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:16.730+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:16.734 INFO:tasks.rados.rados.0.smithi107.stdout:4451: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:16.734 INFO:tasks.rados.rados.0.smithi107.stdout:4453: done (3 left) 2020-06-12T02:38:16.734 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:16.734 INFO:tasks.rados.rados.0.smithi107.stdout:4454: seq_num 4453 ranges {0=180224} 2020-06-12T02:38:16.735 INFO:tasks.rados.rados.0.smithi107.stdout:4454: writing smithi10729407-8 from 0 to 180224 tid 1 2020-06-12T02:38:16.735 INFO:tasks.rados.rados.0.smithi107.stdout:4455: delete oid 18 current snap is 0 2020-06-12T02:38:16.735 INFO:tasks.rados.rados.0.smithi107.stdout:4449: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:16.736 INFO:tasks.rados.rados.0.smithi107.stdout:4449: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:16.736 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 449 (ObjNum 4448 snap 0 seq_num 4448) dirty exists 2020-06-12T02:38:16.736 INFO:tasks.rados.rados.0.smithi107.stdout:4449: left oid 5 (ObjNum 4448 snap 0 seq_num 4448) 2020-06-12T02:38:16.736 INFO:tasks.rados.rados.0.smithi107.stdout:4451: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:16.736 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 797 (ObjNum 4450 snap 0 seq_num 4450) dirty exists 2020-06-12T02:38:16.737 INFO:tasks.rados.rados.0.smithi107.stdout:4451: left oid 11 (ObjNum 4450 snap 0 seq_num 4450) 2020-06-12T02:38:16.737 INFO:tasks.rados.rados.0.smithi107.stdout:4449: done (4 left) 2020-06-12T02:38:16.737 INFO:tasks.rados.rados.0.smithi107.stdout:4451: done (3 left) 2020-06-12T02:38:16.738 INFO:tasks.rados.rados.0.smithi107.stdout:4455: done (2 left) 2020-06-12T02:38:16.738 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:38:16.738 INFO:tasks.rados.rados.0.smithi107.stdout:4456: seq_num 4455 ranges {0=221184} 2020-06-12T02:38:16.739 INFO:tasks.rados.rados.0.smithi107.stdout:4456: writing smithi10729407-16 from 0 to 221184 tid 1 2020-06-12T02:38:16.739 INFO:tasks.rados.rados.0.smithi107.stdout:4457: delete oid 6 current snap is 0 2020-06-12T02:38:16.739 INFO:tasks.rados.rados.0.smithi107.stdout:4454: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:16.739 INFO:tasks.rados.rados.0.smithi107.stdout:4454: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:16.742 INFO:tasks.rados.rados.0.smithi107.stdout:4457: done (3 left) 2020-06-12T02:38:16.742 INFO:tasks.rados.rados.0.smithi107.stdout:4458: delete oid 19 current snap is 0 2020-06-12T02:38:16.742 INFO:tasks.rados.rados.0.smithi107.stdout:4454: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:16.743 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 759 (ObjNum 4453 snap 0 seq_num 4453) dirty exists 2020-06-12T02:38:16.743 INFO:tasks.rados.rados.0.smithi107.stdout:4454: left oid 8 (ObjNum 4453 snap 0 seq_num 4453) 2020-06-12T02:38:16.744 INFO:tasks.rados.rados.0.smithi107.stdout:4456: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:16.744 INFO:tasks.rados.rados.0.smithi107.stdout:4456: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:16.745 INFO:tasks.rados.rados.0.smithi107.stdout:4454: done (3 left) 2020-06-12T02:38:16.745 INFO:tasks.rados.rados.0.smithi107.stdout:4458: done (2 left) 2020-06-12T02:38:16.745 INFO:tasks.rados.rados.0.smithi107.stdout:4459: delete oid 12 current snap is 0 2020-06-12T02:38:16.746 INFO:tasks.rados.rados.0.smithi107.stdout:4456: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:16.746 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 790 (ObjNum 4455 snap 0 seq_num 4455) dirty exists 2020-06-12T02:38:16.746 INFO:tasks.rados.rados.0.smithi107.stdout:4456: left oid 16 (ObjNum 4455 snap 0 seq_num 4455) 2020-06-12T02:38:16.747 INFO:tasks.rados.rados.0.smithi107.stdout:4456: done (2 left) 2020-06-12T02:38:16.747 INFO:tasks.rados.rados.0.smithi107.stdout:4459: done (1 left) 2020-06-12T02:38:16.747 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:16.747 INFO:tasks.rados.rados.0.smithi107.stdout:4460: seq_num 4459 ranges {0=286720} 2020-06-12T02:38:16.748 INFO:tasks.rados.rados.0.smithi107.stdout:4460: writing smithi10729407-15 from 0 to 286720 tid 1 2020-06-12T02:38:16.748 INFO:tasks.rados.rados.0.smithi107.stdout:4461: delete oid 16 current snap is 0 2020-06-12T02:38:16.750 INFO:tasks.rados.rados.0.smithi107.stdout:4461: done (2 left) 2020-06-12T02:38:16.750 INFO:tasks.rados.rados.0.smithi107.stdout:4462: delete oid 5 current snap is 0 2020-06-12T02:38:16.753 INFO:tasks.rados.rados.0.smithi107.stdout:4460: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:16.755 INFO:tasks.rados.rados.0.smithi107.stdout:4462: done (2 left) 2020-06-12T02:38:16.755 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:16.755 INFO:tasks.rados.rados.0.smithi107.stdout:4463: seq_num 4462 ranges {0=335872} 2020-06-12T02:38:16.756 INFO:tasks.rados.rados.0.smithi107.stdout:4463: writing smithi10729407-13 from 0 to 335872 tid 1 2020-06-12T02:38:16.757 INFO:tasks.rados.rados.0.smithi107.stdout:4464: delete oid 20 current snap is 0 2020-06-12T02:38:16.757 INFO:tasks.rados.rados.0.smithi107.stdout:4460: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:16.760 INFO:tasks.rados.rados.0.smithi107.stdout:4460: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:16.760 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 761 (ObjNum 4459 snap 0 seq_num 4459) dirty exists 2020-06-12T02:38:16.760 INFO:tasks.rados.rados.0.smithi107.stdout:4460: left oid 15 (ObjNum 4459 snap 0 seq_num 4459) 2020-06-12T02:38:16.764 INFO:tasks.rados.rados.0.smithi107.stdout:4463: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:16.765 INFO:tasks.rados.rados.0.smithi107.stdout:4460: done (3 left) 2020-06-12T02:38:16.765 INFO:tasks.rados.rados.0.smithi107.stdout:4464: done (2 left) 2020-06-12T02:38:16.765 INFO:tasks.rados.rados.0.smithi107.stdout:4465: delete oid 6 current snap is 0 2020-06-12T02:38:16.768 INFO:tasks.rados.rados.0.smithi107.stdout:4465: done (2 left) 2020-06-12T02:38:16.769 INFO:tasks.rados.rados.0.smithi107.stdout:4466: delete oid 3 current snap is 0 2020-06-12T02:38:16.771 INFO:tasks.rados.rados.0.smithi107.stdout:4463: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:16.771 INFO:tasks.rados.rados.0.smithi107.stdout:4463: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:16.771 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 838 (ObjNum 4462 snap 0 seq_num 4462) dirty exists 2020-06-12T02:38:16.771 INFO:tasks.rados.rados.0.smithi107.stdout:4463: left oid 13 (ObjNum 4462 snap 0 seq_num 4462) 2020-06-12T02:38:16.772 INFO:tasks.rados.rados.0.smithi107.stdout:4463: done (2 left) 2020-06-12T02:38:16.772 INFO:tasks.rados.rados.0.smithi107.stdout:4466: done (1 left) 2020-06-12T02:38:16.772 INFO:tasks.rados.rados.0.smithi107.stdout:4467: delete oid 4 current snap is 0 2020-06-12T02:38:16.774 INFO:tasks.rados.rados.0.smithi107.stdout:4467: done (1 left) 2020-06-12T02:38:16.774 INFO:tasks.rados.rados.0.smithi107.stdout:4468: delete oid 3 current snap is 0 2020-06-12T02:38:16.777 INFO:tasks.rados.rados.0.smithi107.stdout:4468: done (1 left) 2020-06-12T02:38:16.777 INFO:tasks.rados.rados.0.smithi107.stdout:4469: delete oid 7 current snap is 0 2020-06-12T02:38:16.778 INFO:tasks.rados.rados.0.smithi107.stdout:4469: done (1 left) 2020-06-12T02:38:16.778 INFO:tasks.rados.rados.0.smithi107.stdout:4470: delete oid 7 current snap is 0 2020-06-12T02:38:16.779 INFO:tasks.rados.rados.0.smithi107.stdout:4470: done (1 left) 2020-06-12T02:38:16.780 INFO:tasks.rados.rados.0.smithi107.stdout:4471: delete oid 5 current snap is 0 2020-06-12T02:38:16.781 INFO:tasks.rados.rados.0.smithi107.stdout:4471: done (1 left) 2020-06-12T02:38:16.782 INFO:tasks.rados.rados.0.smithi107.stdout:4472: delete oid 8 current snap is 0 2020-06-12T02:38:16.784 INFO:tasks.rados.rados.0.smithi107.stdout:4472: done (1 left) 2020-06-12T02:38:16.784 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:16.784 INFO:tasks.rados.rados.0.smithi107.stdout:4473: seq_num 4472 ranges {0=499712} 2020-06-12T02:38:16.787 INFO:tasks.rados.rados.0.smithi107.stdout:4473: writing smithi10729407-12 from 0 to 499712 tid 1 2020-06-12T02:38:16.787 INFO:tasks.rados.rados.0.smithi107.stdout:4474: delete oid 18 current snap is 0 2020-06-12T02:38:16.789 INFO:tasks.rados.rados.0.smithi107.stdout:4474: done (2 left) 2020-06-12T02:38:16.789 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:16.789 INFO:tasks.rados.rados.0.smithi107.stdout:4475: seq_num 4474 ranges {0=540672} 2020-06-12T02:38:16.792 INFO:tasks.rados.rados.0.smithi107.stdout:4475: writing smithi10729407-17 from 0 to 540672 tid 1 2020-06-12T02:38:16.792 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:16.792 INFO:tasks.rados.rados.0.smithi107.stdout:4476: seq_num 4475 ranges {0=557056} 2020-06-12T02:38:16.795 INFO:tasks.rados.rados.0.smithi107.stdout:4476: writing smithi10729407-4 from 0 to 557056 tid 1 2020-06-12T02:38:16.796 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:16.796 INFO:tasks.rados.rados.0.smithi107.stdout:4473: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:16.796 INFO:tasks.rados.rados.0.smithi107.stdout:4477: seq_num 4476 ranges {0=573440} 2020-06-12T02:38:16.799 INFO:tasks.rados.rados.0.smithi107.stdout:4477: writing smithi10729407-19 from 0 to 573440 tid 1 2020-06-12T02:38:16.800 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:16.800 INFO:tasks.rados.rados.0.smithi107.stdout:4478: seq_num 4477 ranges {131072=589824} 2020-06-12T02:38:16.804 INFO:tasks.rados.rados.0.smithi107.stdout:4478: writing smithi10729407-11 from 131072 to 720896 tid 1 2020-06-12T02:38:16.804 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:38:16.804 INFO:tasks.rados.rados.0.smithi107.stdout:4473: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:16.805 INFO:tasks.rados.rados.0.smithi107.stdout:4473: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:16.805 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 761 (ObjNum 4472 snap 0 seq_num 4472) dirty exists 2020-06-12T02:38:16.805 INFO:tasks.rados.rados.0.smithi107.stdout:4473: left oid 12 (ObjNum 4472 snap 0 seq_num 4472) 2020-06-12T02:38:16.805 INFO:tasks.rados.rados.0.smithi107.stdout:4475: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:16.805 INFO:tasks.rados.rados.0.smithi107.stdout:4479: seq_num 4478 ranges {0=606208} 2020-06-12T02:38:16.808 INFO:tasks.rados.rados.0.smithi107.stdout:4479: writing smithi10729407-1 from 0 to 606208 tid 1 2020-06-12T02:38:16.808 INFO:tasks.rados.rados.0.smithi107.stdout:4473: done (6 left) 2020-06-12T02:38:16.809 INFO:tasks.rados.rados.0.smithi107.stdout:4480: delete oid 14 current snap is 0 2020-06-12T02:38:16.809 INFO:tasks.rados.rados.0.smithi107.stdout:4476: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:16.809 INFO:tasks.rados.rados.0.smithi107.stdout:4476: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:16.809 INFO:tasks.rados.rados.0.smithi107.stdout:4475: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:16.809 INFO:tasks.rados.rados.0.smithi107.stdout:4476: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:16.810 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 825 (ObjNum 4475 snap 0 seq_num 4475) dirty exists 2020-06-12T02:38:16.810 INFO:tasks.rados.rados.0.smithi107.stdout:4476: left oid 4 (ObjNum 4475 snap 0 seq_num 4475) 2020-06-12T02:38:16.810 INFO:tasks.rados.rados.0.smithi107.stdout:4477: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:16.811 INFO:tasks.rados.rados.0.smithi107.stdout:4478: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:16.811 INFO:tasks.rados.rados.0.smithi107.stdout:4477: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:16.813 INFO:tasks.rados.rados.0.smithi107.stdout:4475: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:16.813 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 826 (ObjNum 4474 snap 0 seq_num 4474) dirty exists 2020-06-12T02:38:16.813 INFO:tasks.rados.rados.0.smithi107.stdout:4475: left oid 17 (ObjNum 4474 snap 0 seq_num 4474) 2020-06-12T02:38:16.814 INFO:tasks.rados.rados.0.smithi107.stdout:4477: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:16.820 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 763 (ObjNum 4476 snap 0 seq_num 4476) dirty exists 2020-06-12T02:38:16.820 INFO:tasks.rados.rados.0.smithi107.stdout:4477: left oid 19 (ObjNum 4476 snap 0 seq_num 4476) 2020-06-12T02:38:16.821 INFO:tasks.rados.rados.0.smithi107.stdout:4478: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:16.821 INFO:tasks.rados.rados.0.smithi107.stdout:4475: done (6 left) 2020-06-12T02:38:16.822 INFO:tasks.rados.rados.0.smithi107.stdout:4476: done (5 left) 2020-06-12T02:38:16.822 INFO:tasks.rados.rados.0.smithi107.stdout:4477: done (4 left) 2020-06-12T02:38:16.822 INFO:tasks.rados.rados.0.smithi107.stdout:4480: done (3 left) 2020-06-12T02:38:16.822 INFO:tasks.rados.rados.0.smithi107.stdout:4481: delete oid 13 current snap is 0 2020-06-12T02:38:16.823 INFO:tasks.rados.rados.0.smithi107.stdout:4478: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:16.823 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 799 (ObjNum 4477 snap 0 seq_num 4477) dirty exists 2020-06-12T02:38:16.823 INFO:tasks.rados.rados.0.smithi107.stdout:4478: left oid 11 (ObjNum 4477 snap 0 seq_num 4477) 2020-06-12T02:38:16.823 INFO:tasks.rados.rados.0.smithi107.stdout:4479: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:16.823 INFO:tasks.rados.rados.0.smithi107.stdout:4479: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:16.824 INFO:tasks.rados.rados.0.smithi107.stdout:4478: done (3 left) 2020-06-12T02:38:16.824 INFO:tasks.rados.rados.0.smithi107.stdout:4481: done (2 left) 2020-06-12T02:38:16.824 INFO:tasks.rados.rados.0.smithi107.stdout:4482: delete oid 2 current snap is 0 2020-06-12T02:38:16.824 INFO:tasks.rados.rados.0.smithi107.stdout:4479: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:16.824 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 830 (ObjNum 4478 snap 0 seq_num 4478) dirty exists 2020-06-12T02:38:16.825 INFO:tasks.rados.rados.0.smithi107.stdout:4479: left oid 1 (ObjNum 4478 snap 0 seq_num 4478) 2020-06-12T02:38:16.825 INFO:tasks.rados.rados.0.smithi107.stdout:4479: done (2 left) 2020-06-12T02:38:16.825 INFO:tasks.rados.rados.0.smithi107.stdout:4482: done (1 left) 2020-06-12T02:38:16.825 INFO:tasks.rados.rados.0.smithi107.stdout:4483: delete oid 17 current snap is 0 2020-06-12T02:38:16.826 INFO:tasks.rados.rados.0.smithi107.stdout:4483: done (1 left) 2020-06-12T02:38:16.826 INFO:tasks.rados.rados.0.smithi107.stdout:4484: delete oid 7 current snap is 0 2020-06-12T02:38:16.826 INFO:tasks.rados.rados.0.smithi107.stdout:4484: done (1 left) 2020-06-12T02:38:16.826 INFO:tasks.rados.rados.0.smithi107.stdout:4485: delete oid 2 current snap is 0 2020-06-12T02:38:16.827 INFO:tasks.rados.rados.0.smithi107.stdout:4485: done (1 left) 2020-06-12T02:38:16.827 INFO:tasks.rados.rados.0.smithi107.stdout:4486: delete oid 9 current snap is 0 2020-06-12T02:38:16.828 INFO:tasks.rados.rados.0.smithi107.stdout:4486: done (1 left) 2020-06-12T02:38:16.829 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:38:16.829 INFO:tasks.rados.rados.0.smithi107.stdout:4487: seq_num 4486 ranges {0=737280} 2020-06-12T02:38:16.833 INFO:tasks.rados.rados.0.smithi107.stdout:4487: writing smithi10729407-20 from 0 to 737280 tid 1 2020-06-12T02:38:16.833 INFO:tasks.rados.rados.0.smithi107.stdout:4488: delete oid 12 current snap is 0 2020-06-12T02:38:16.833 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:16.830+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:16.834 INFO:tasks.rados.rados.0.smithi107.stdout:4488: done (2 left) 2020-06-12T02:38:16.835 INFO:tasks.rados.rados.0.smithi107.stdout:4489: delete oid 6 current snap is 0 2020-06-12T02:38:16.841 INFO:tasks.rados.rados.0.smithi107.stdout:4489: done (2 left) 2020-06-12T02:38:16.841 INFO:tasks.rados.rados.0.smithi107.stdout:4490: delete oid 8 current snap is 0 2020-06-12T02:38:16.842 INFO:tasks.rados.rados.0.smithi107.stdout:4487: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:16.842 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:16.843 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:38:16.843 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:38:16.843 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:38:16.843 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:16.843 INFO:tasks.rados.rados.0.smithi107.stdout:4490: done (2 left) 2020-06-12T02:38:16.844 INFO:tasks.rados.rados.0.smithi107.stdout:4491: delete oid 15 current snap is 0 2020-06-12T02:38:16.844 INFO:tasks.rados.rados.0.smithi107.stdout:4487: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:16.844 INFO:tasks.rados.rados.0.smithi107.stdout:4487: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:16.845 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 343 (ObjNum 4486 snap 0 seq_num 4486) dirty exists 2020-06-12T02:38:16.845 INFO:tasks.rados.rados.0.smithi107.stdout:4487: left oid 20 (ObjNum 4486 snap 0 seq_num 4486) 2020-06-12T02:38:16.845 INFO:tasks.rados.rados.0.smithi107.stdout:4487: done (2 left) 2020-06-12T02:38:16.845 INFO:tasks.rados.rados.0.smithi107.stdout:4491: done (1 left) 2020-06-12T02:38:16.846 INFO:tasks.rados.rados.0.smithi107.stdout:4492: delete oid 3 current snap is 0 2020-06-12T02:38:16.847 INFO:tasks.rados.rados.0.smithi107.stdout:4492: done (1 left) 2020-06-12T02:38:16.848 INFO:tasks.rados.rados.0.smithi107.stdout:4493: delete oid 15 current snap is 0 2020-06-12T02:38:16.849 INFO:tasks.rados.rados.0.smithi107.stdout:4493: done (1 left) 2020-06-12T02:38:16.849 INFO:tasks.rados.rados.0.smithi107.stdout:4494: delete oid 2 current snap is 0 2020-06-12T02:38:16.851 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-06-12T02:38:16.852 INFO:tasks.rados.rados.0.smithi107.stdout:4494: done (1 left) 2020-06-12T02:38:16.853 INFO:tasks.rados.rados.0.smithi107.stdout:4495: delete oid 19 current snap is 0 2020-06-12T02:38:16.854 INFO:tasks.rados.rados.0.smithi107.stdout:4495: done (1 left) 2020-06-12T02:38:16.854 INFO:tasks.rados.rados.0.smithi107.stdout:4496: delete oid 15 current snap is 0 2020-06-12T02:38:16.855 INFO:tasks.rados.rados.0.smithi107.stdout:4496: done (1 left) 2020-06-12T02:38:16.856 INFO:tasks.rados.rados.0.smithi107.stdout:4497: delete oid 19 current snap is 0 2020-06-12T02:38:16.857 INFO:tasks.rados.rados.0.smithi107.stdout:4497: done (1 left) 2020-06-12T02:38:16.857 INFO:tasks.rados.rados.0.smithi107.stdout:4498: delete oid 16 current snap is 0 2020-06-12T02:38:16.859 INFO:tasks.rados.rados.0.smithi107.stdout:4498: done (1 left) 2020-06-12T02:38:16.859 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:16.859 INFO:tasks.rados.rados.0.smithi107.stdout:4499: seq_num 4498 ranges {557056=540672,1097728=401408} 2020-06-12T02:38:16.865 INFO:tasks.rados.rados.0.smithi107.stdout:4499: writing smithi10729407-4 from 557056 to 1097728 tid 1 2020-06-12T02:38:16.867 INFO:tasks.rados.rados.0.smithi107.stdout:4499: writing smithi10729407-4 from 1097728 to 1499136 tid 2 2020-06-12T02:38:16.868 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:16.868 INFO:tasks.rados.rados.0.smithi107.stdout:4500: seq_num 4499 ranges {0=557056,557056=401408} 2020-06-12T02:38:16.871 INFO:tasks.rados.rados.0.smithi107.stdout:4500: writing smithi10729407-17 from 0 to 557056 tid 1 2020-06-12T02:38:16.873 INFO:tasks.rados.rados.0.smithi107.stdout:4500: writing smithi10729407-17 from 557056 to 958464 tid 2 2020-06-12T02:38:16.873 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:16.873 INFO:tasks.rados.rados.0.smithi107.stdout:4501: seq_num 4500 ranges {0=573440,573440=401408} 2020-06-12T02:38:16.878 INFO:tasks.rados.rados.0.smithi107.stdout:4501: writing smithi10729407-19 from 0 to 573440 tid 1 2020-06-12T02:38:16.879 INFO:tasks.rados.rados.0.smithi107.stdout:4501: writing smithi10729407-19 from 573440 to 974848 tid 2 2020-06-12T02:38:16.879 INFO:tasks.rados.rados.0.smithi107.stdout:4502: delete oid 13 current snap is 0 2020-06-12T02:38:16.879 INFO:tasks.rados.rados.0.smithi107.stdout:4499: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:16.880 INFO:tasks.rados.rados.0.smithi107.stdout:4499: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:16.880 INFO:tasks.rados.rados.0.smithi107.stdout:4499: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:16.884 INFO:tasks.rados.rados.0.smithi107.stdout:4500: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:16.884 INFO:tasks.rados.rados.0.smithi107.stdout:4502: done (4 left) 2020-06-12T02:38:16.884 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:16.884 INFO:tasks.rados.rados.0.smithi107.stdout:4503: seq_num 4502 ranges {0=606208,606208=401408} 2020-06-12T02:38:16.886 INFO:tasks.rados.rados.0.smithi107.stdout:4503: writing smithi10729407-16 from 0 to 606208 tid 1 2020-06-12T02:38:16.888 INFO:tasks.rados.rados.0.smithi107.stdout:4503: writing smithi10729407-16 from 606208 to 1007616 tid 2 2020-06-12T02:38:16.889 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:16.889 INFO:tasks.rados.rados.0.smithi107.stdout:4504: seq_num 4503 ranges {720896=622592,1343488=401408} 2020-06-12T02:38:16.896 INFO:tasks.rados.rados.0.smithi107.stdout:4504: writing smithi10729407-11 from 720896 to 1343488 tid 1 2020-06-12T02:38:16.899 INFO:tasks.rados.rados.0.smithi107.stdout:4504: writing smithi10729407-11 from 1343488 to 1744896 tid 2 2020-06-12T02:38:16.899 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:38:16.899 INFO:tasks.rados.rados.0.smithi107.stdout:4505: seq_num 4504 ranges {0=638976,638976=401408} 2020-06-12T02:38:16.904 INFO:tasks.rados.rados.0.smithi107.stdout:4505: writing smithi10729407-3 from 0 to 638976 tid 1 2020-06-12T02:38:16.906 INFO:tasks.rados.rados.0.smithi107.stdout:4505: writing smithi10729407-3 from 638976 to 1040384 tid 2 2020-06-12T02:38:16.907 INFO:tasks.rados.rados.0.smithi107.stdout:4506: delete oid 6 current snap is 0 2020-06-12T02:38:16.907 INFO:tasks.rados.rados.0.smithi107.stdout:4501: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:16.907 INFO:tasks.rados.rados.0.smithi107.stdout:4499: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:16.907 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 830 (ObjNum 4498 snap 0 seq_num 4498) dirty exists 2020-06-12T02:38:16.908 INFO:tasks.rados.rados.0.smithi107.stdout:4499: left oid 4 (ObjNum 4498 snap 0 seq_num 4498) 2020-06-12T02:38:16.908 INFO:tasks.rados.rados.0.smithi107.stdout:4500: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:16.908 INFO:tasks.rados.rados.0.smithi107.stdout:4500: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:16.908 INFO:tasks.rados.rados.0.smithi107.stdout:4501: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:16.908 INFO:tasks.rados.rados.0.smithi107.stdout:4501: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:16.909 INFO:tasks.rados.rados.0.smithi107.stdout:4500: finishing write tid 5 to smithi10729407-17 2020-06-12T02:38:16.909 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 833 (ObjNum 4499 snap 0 seq_num 4499) dirty exists 2020-06-12T02:38:16.909 INFO:tasks.rados.rados.0.smithi107.stdout:4500: left oid 17 (ObjNum 4499 snap 0 seq_num 4499) 2020-06-12T02:38:16.909 INFO:tasks.rados.rados.0.smithi107.stdout:4503: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:16.909 INFO:tasks.rados.rados.0.smithi107.stdout:4501: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:16.910 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 769 (ObjNum 4500 snap 0 seq_num 4500) dirty exists 2020-06-12T02:38:16.910 INFO:tasks.rados.rados.0.smithi107.stdout:4501: left oid 19 (ObjNum 4500 snap 0 seq_num 4500) 2020-06-12T02:38:16.910 INFO:tasks.rados.rados.0.smithi107.stdout:4503: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:16.911 INFO:tasks.rados.rados.0.smithi107.stdout:4503: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:16.911 INFO:tasks.rados.rados.0.smithi107.stdout:4504: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:16.911 INFO:tasks.rados.rados.0.smithi107.stdout:4504: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:16.912 INFO:tasks.rados.rados.0.smithi107.stdout:4504: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:16.912 INFO:tasks.rados.rados.0.smithi107.stdout:4499: done (7 left) 2020-06-12T02:38:16.912 INFO:tasks.rados.rados.0.smithi107.stdout:4500: done (6 left) 2020-06-12T02:38:16.913 INFO:tasks.rados.rados.0.smithi107.stdout:4501: done (5 left) 2020-06-12T02:38:16.913 INFO:tasks.rados.rados.0.smithi107.stdout:4506: done (4 left) 2020-06-12T02:38:16.913 INFO:tasks.rados.rados.0.smithi107.stdout:4507: delete oid 15 current snap is 0 2020-06-12T02:38:16.913 INFO:tasks.rados.rados.0.smithi107.stdout:4504: finishing write tid 5 to smithi10729407-11 2020-06-12T02:38:16.913 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 803 (ObjNum 4503 snap 0 seq_num 4503) dirty exists 2020-06-12T02:38:16.913 INFO:tasks.rados.rados.0.smithi107.stdout:4504: left oid 11 (ObjNum 4503 snap 0 seq_num 4503) 2020-06-12T02:38:16.914 INFO:tasks.rados.rados.0.smithi107.stdout:4503: finishing write tid 5 to smithi10729407-16 2020-06-12T02:38:16.914 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 798 (ObjNum 4502 snap 0 seq_num 4502) dirty exists 2020-06-12T02:38:16.915 INFO:tasks.rados.rados.0.smithi107.stdout:4503: left oid 16 (ObjNum 4502 snap 0 seq_num 4502) 2020-06-12T02:38:16.915 INFO:tasks.rados.rados.0.smithi107.stdout:4505: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:16.915 INFO:tasks.rados.rados.0.smithi107.stdout:4503: done (4 left) 2020-06-12T02:38:16.915 INFO:tasks.rados.rados.0.smithi107.stdout:4504: done (3 left) 2020-06-12T02:38:16.915 INFO:tasks.rados.rados.0.smithi107.stdout:4507: done (2 left) 2020-06-12T02:38:16.916 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:16.916 INFO:tasks.rados.rados.0.smithi107.stdout:4508: seq_num 4507 ranges {0=688128,688128=401408} 2020-06-12T02:38:16.917 INFO:tasks.rados.rados.0.smithi107.stdout:4508: writing smithi10729407-14 from 0 to 688128 tid 1 2020-06-12T02:38:16.920 INFO:tasks.rados.rados.0.smithi107.stdout:4508: writing smithi10729407-14 from 688128 to 1089536 tid 2 2020-06-12T02:38:16.920 INFO:tasks.rados.rados.0.smithi107.stdout:4509: delete oid 2 current snap is 0 2020-06-12T02:38:16.920 INFO:tasks.rados.rados.0.smithi107.stdout:4505: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:16.920 INFO:tasks.rados.rados.0.smithi107.stdout:4505: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:16.921 INFO:tasks.rados.rados.0.smithi107.stdout:4505: finishing write tid 5 to smithi10729407-3 2020-06-12T02:38:16.921 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 801 (ObjNum 4504 snap 0 seq_num 4504) dirty exists 2020-06-12T02:38:16.921 INFO:tasks.rados.rados.0.smithi107.stdout:4505: left oid 3 (ObjNum 4504 snap 0 seq_num 4504) 2020-06-12T02:38:16.922 INFO:tasks.rados.rados.0.smithi107.stdout:4505: done (3 left) 2020-06-12T02:38:16.923 INFO:tasks.rados.rados.0.smithi107.stdout:4509: done (2 left) 2020-06-12T02:38:16.923 INFO:tasks.rados.rados.0.smithi107.stdout:4510: delete oid 4 current snap is 0 2020-06-12T02:38:16.925 INFO:tasks.rados.rados.0.smithi107.stdout:4510: done (2 left) 2020-06-12T02:38:16.925 INFO:tasks.rados.rados.0.smithi107.stdout:4511: delete oid 7 current snap is 0 2020-06-12T02:38:16.925 INFO:tasks.rados.rados.0.smithi107.stdout:4508: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:16.927 INFO:tasks.rados.rados.0.smithi107.stdout:4508: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:16.928 INFO:tasks.rados.rados.0.smithi107.stdout:4508: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:16.928 INFO:tasks.rados.rados.0.smithi107.stdout:4511: done (2 left) 2020-06-12T02:38:16.928 INFO:tasks.rados.rados.0.smithi107.stdout:4512: delete oid 17 current snap is 0 2020-06-12T02:38:16.929 INFO:tasks.rados.rados.0.smithi107.stdout:4508: finishing write tid 5 to smithi10729407-14 2020-06-12T02:38:16.929 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 806 (ObjNum 4507 snap 0 seq_num 4507) dirty exists 2020-06-12T02:38:16.929 INFO:tasks.rados.rados.0.smithi107.stdout:4508: left oid 14 (ObjNum 4507 snap 0 seq_num 4507) 2020-06-12T02:38:16.931 INFO:tasks.rados.rados.0.smithi107.stdout:4508: done (2 left) 2020-06-12T02:38:16.931 INFO:tasks.rados.rados.0.smithi107.stdout:4512: done (1 left) 2020-06-12T02:38:16.931 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:16.931 INFO:tasks.rados.rados.0.smithi107.stdout:4513: seq_num 4512 ranges {0=770048,770048=401408} 2020-06-12T02:38:16.934 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:16.931+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:16.935 INFO:tasks.rados.rados.0.smithi107.stdout:4513: writing smithi10729407-4 from 0 to 770048 tid 1 2020-06-12T02:38:16.937 INFO:tasks.rados.rados.0.smithi107.stdout:4513: writing smithi10729407-4 from 770048 to 1171456 tid 2 2020-06-12T02:38:16.938 INFO:tasks.rados.rados.0.smithi107.stdout:4514: delete oid 3 current snap is 0 2020-06-12T02:38:16.939 INFO:tasks.rados.rados.0.smithi107.stdout:4514: done (2 left) 2020-06-12T02:38:16.939 INFO:tasks.rados.rados.0.smithi107.stdout:4515: delete oid 6 current snap is 0 2020-06-12T02:38:16.942 INFO:tasks.rados.rados.0.smithi107.stdout:4515: done (2 left) 2020-06-12T02:38:16.942 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:38:16.943 INFO:tasks.rados.rados.0.smithi107.stdout:4516: seq_num 4515 ranges {0=425984,425984=425984,851968=376832} 2020-06-12T02:38:16.945 INFO:tasks.rados.rados.0.smithi107.stdout:4516: writing smithi10729407-13 from 0 to 425984 tid 1 2020-06-12T02:38:16.947 INFO:tasks.rados.rados.0.smithi107.stdout:4516: writing smithi10729407-13 from 425984 to 851968 tid 2 2020-06-12T02:38:16.950 INFO:tasks.rados.rados.0.smithi107.stdout:4516: writing smithi10729407-13 from 851968 to 1228800 tid 3 2020-06-12T02:38:16.950 INFO:tasks.rados.rados.0.smithi107.stdout:4517: delete oid 9 current snap is 0 2020-06-12T02:38:16.951 INFO:tasks.rados.rados.0.smithi107.stdout:4513: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:16.951 INFO:tasks.rados.rados.0.smithi107.stdout:4513: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:16.951 INFO:tasks.rados.rados.0.smithi107.stdout:4513: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:16.951 INFO:tasks.rados.rados.0.smithi107.stdout:4513: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:16.951 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 838 (ObjNum 4512 snap 0 seq_num 4512) dirty exists 2020-06-12T02:38:16.952 INFO:tasks.rados.rados.0.smithi107.stdout:4513: left oid 4 (ObjNum 4512 snap 0 seq_num 4512) 2020-06-12T02:38:16.952 INFO:tasks.rados.rados.0.smithi107.stdout:4513: done (3 left) 2020-06-12T02:38:16.952 INFO:tasks.rados.rados.0.smithi107.stdout:4517: done (2 left) 2020-06-12T02:38:16.953 INFO:tasks.rados.rados.0.smithi107.stdout:4518: delete oid 1 current snap is 0 2020-06-12T02:38:16.953 INFO:tasks.rados.rados.0.smithi107.stdout:4516: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:16.955 INFO:tasks.rados.rados.0.smithi107.stdout:4518: done (2 left) 2020-06-12T02:38:16.955 INFO:tasks.rados.rados.0.smithi107.stdout:4519: delete oid 12 current snap is 0 2020-06-12T02:38:16.957 INFO:tasks.rados.rados.0.smithi107.stdout:4516: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:16.957 INFO:tasks.rados.rados.0.smithi107.stdout:4516: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:16.958 INFO:tasks.rados.rados.0.smithi107.stdout:4519: done (2 left) 2020-06-12T02:38:16.959 INFO:tasks.rados.rados.0.smithi107.stdout:4520: delete oid 3 current snap is 0 2020-06-12T02:38:16.959 INFO:tasks.rados.rados.0.smithi107.stdout:4516: finishing write tid 5 to smithi10729407-13 2020-06-12T02:38:16.961 INFO:tasks.rados.rados.0.smithi107.stdout:4516: finishing write tid 6 to smithi10729407-13 2020-06-12T02:38:16.961 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 848 (ObjNum 4515 snap 0 seq_num 4515) dirty exists 2020-06-12T02:38:16.961 INFO:tasks.rados.rados.0.smithi107.stdout:4516: left oid 13 (ObjNum 4515 snap 0 seq_num 4515) 2020-06-12T02:38:16.962 INFO:tasks.rados.rados.0.smithi107.stdout:4516: done (2 left) 2020-06-12T02:38:16.962 INFO:tasks.rados.rados.0.smithi107.stdout:4520: done (1 left) 2020-06-12T02:38:16.962 INFO:tasks.rados.rados.0.smithi107.stdout:4521: delete oid 9 current snap is 0 2020-06-12T02:38:16.963 INFO:tasks.rados.rados.0.smithi107.stdout:4521: done (1 left) 2020-06-12T02:38:16.963 INFO:tasks.rados.rados.0.smithi107.stdout:4522: delete oid 19 current snap is 0 2020-06-12T02:38:16.965 INFO:tasks.rados.rados.0.smithi107.stdout:4522: done (1 left) 2020-06-12T02:38:16.965 INFO:tasks.rados.rados.0.smithi107.stdout:4523: delete oid 4 current snap is 0 2020-06-12T02:38:16.969 INFO:tasks.rados.rados.0.smithi107.stdout:4523: done (1 left) 2020-06-12T02:38:16.969 INFO:tasks.rados.rados.0.smithi107.stdout:4524: delete oid 9 current snap is 0 2020-06-12T02:38:16.971 INFO:tasks.rados.rados.0.smithi107.stdout:4524: done (1 left) 2020-06-12T02:38:16.971 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:16.971 INFO:tasks.rados.rados.0.smithi107.stdout:4525: seq_num 4524 ranges {0=573440,573440=679936,1253376=122880} 2020-06-12T02:38:16.974 INFO:tasks.rados.rados.0.smithi107.stdout:4525: writing smithi10729407-15 from 0 to 573440 tid 1 2020-06-12T02:38:16.978 INFO:tasks.rados.rados.0.smithi107.stdout:4525: writing smithi10729407-15 from 573440 to 1253376 tid 2 2020-06-12T02:38:16.979 INFO:tasks.rados.rados.0.smithi107.stdout:4525: writing smithi10729407-15 from 1253376 to 1376256 tid 3 2020-06-12T02:38:16.979 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:16.979 INFO:tasks.rados.rados.0.smithi107.stdout:4526: seq_num 4525 ranges {0=589824,589824=565248,1155072=237568} 2020-06-12T02:38:16.982 INFO:tasks.rados.rados.0.smithi107.stdout:4526: writing smithi10729407-3 from 0 to 589824 tid 1 2020-06-12T02:38:16.985 INFO:tasks.rados.rados.0.smithi107.stdout:4526: writing smithi10729407-3 from 589824 to 1155072 tid 2 2020-06-12T02:38:16.987 INFO:tasks.rados.rados.0.smithi107.stdout:4526: writing smithi10729407-3 from 1155072 to 1392640 tid 3 2020-06-12T02:38:16.987 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:16.987 INFO:tasks.rados.rados.0.smithi107.stdout:4527: seq_num 4526 ranges {1089536=606208,1695744=450560,2146304=352256} 2020-06-12T02:38:16.997 INFO:tasks.rados.rados.0.smithi107.stdout:4527: writing smithi10729407-14 from 1089536 to 1695744 tid 1 2020-06-12T02:38:16.999 INFO:tasks.rados.rados.0.smithi107.stdout:4527: writing smithi10729407-14 from 1695744 to 2146304 tid 2 2020-06-12T02:38:17.003 INFO:tasks.rados.rados.0.smithi107.stdout:4527: writing smithi10729407-14 from 2146304 to 2498560 tid 3 2020-06-12T02:38:17.004 INFO:tasks.rados.rados.0.smithi107.stdout:4528: delete oid 17 current snap is 0 2020-06-12T02:38:17.004 INFO:tasks.rados.rados.0.smithi107.stdout:4525: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:17.004 INFO:tasks.rados.rados.0.smithi107.stdout:4526: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:17.004 INFO:tasks.rados.rados.0.smithi107.stdout:4525: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:17.005 INFO:tasks.rados.rados.0.smithi107.stdout:4525: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:17.005 INFO:tasks.rados.rados.0.smithi107.stdout:4525: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:17.005 INFO:tasks.rados.rados.0.smithi107.stdout:4525: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:17.005 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 771 (ObjNum 4524 snap 0 seq_num 4524) dirty exists 2020-06-12T02:38:17.005 INFO:tasks.rados.rados.0.smithi107.stdout:4525: left oid 15 (ObjNum 4524 snap 0 seq_num 4524) 2020-06-12T02:38:17.006 INFO:tasks.rados.rados.0.smithi107.stdout:4526: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:17.007 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:17.008 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:38:17.008 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:38:17.008 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:38:17.008 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.008 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1206 3.5s0 3:a9510bda:::smithi10729407-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:38:17.009 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.856688+0000", 2020-06-12T02:38:17.009 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 44.145373147000001, 2020-06-12T02:38:17.009 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019116942000000001, 2020-06-12T02:38:17.010 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:17.010 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:17.010 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:17.010 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:17.010 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:17.011 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1206 2020-06-12T02:38:17.011 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.011 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:17.011 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.011 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:17.012 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856688+0000", 2020-06-12T02:38:17.012 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:17.012 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.012 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.012 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:17.012 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856688+0000", 2020-06-12T02:38:17.013 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4355e-05 2020-06-12T02:38:17.013 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.013 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.013 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:17.013 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856713+0000", 2020-06-12T02:38:17.014 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0850000000000001e-06 2020-06-12T02:38:17.014 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.014 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.014 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:17.014 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856716+0000", 2020-06-12T02:38:17.015 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4509999999999997e-06 2020-06-12T02:38:17.015 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.015 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.015 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:17.015 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856722+0000", 2020-06-12T02:38:17.016 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2879999999999999e-06 2020-06-12T02:38:17.016 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.016 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.016 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:17.016 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856724+0000", 2020-06-12T02:38:17.017 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0057507449999999998 2020-06-12T02:38:17.017 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.017 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.017 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:17.017 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.862475+0000", 2020-06-12T02:38:17.018 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5759000000000003e-05 2020-06-12T02:38:17.018 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.023 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.023 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:17.024 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.862571+0000", 2020-06-12T02:38:17.024 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012199018000000001 2020-06-12T02:38:17.024 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.025 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.025 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:17.025 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.874770+0000", 2020-06-12T02:38:17.025 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7260999999999999e-05 2020-06-12T02:38:17.025 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.026 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.026 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:17.026 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.874827+0000", 2020-06-12T02:38:17.026 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00097798000000000004 2020-06-12T02:38:17.026 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.027 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.027 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:17.027 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.875805+0000", 2020-06-12T02:38:17.027 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019116942000000001 2020-06-12T02:38:17.027 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.027 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:17.028 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.028 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.028 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.028 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1572 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~745472] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-06-12T02:38:17.028 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.886439+0000", 2020-06-12T02:38:17.029 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 43.115622428999998, 2020-06-12T02:38:17.029 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021442571000000001, 2020-06-12T02:38:17.029 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:17.029 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:17.029 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:17.030 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:17.030 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:17.030 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1572 2020-06-12T02:38:17.030 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.030 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:17.031 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.031 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:17.031 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.886439+0000", 2020-06-12T02:38:17.031 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:17.031 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.031 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.032 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:17.032 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.886439+0000", 2020-06-12T02:38:17.032 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00073027599999999997 2020-06-12T02:38:17.032 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.033 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.033 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:17.033 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887169+0000", 2020-06-12T02:38:17.033 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.49e-06 2020-06-12T02:38:17.033 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.034 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.034 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:17.034 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887175+0000", 2020-06-12T02:38:17.034 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7899999999999994e-06 2020-06-12T02:38:17.034 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.035 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.035 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:17.035 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887185+0000", 2020-06-12T02:38:17.035 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.42e-06 2020-06-12T02:38:17.035 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.036 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.036 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:17.036 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887189+0000", 2020-06-12T02:38:17.036 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1619999999999999e-05 2020-06-12T02:38:17.036 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.037 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.037 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:17.037 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887221+0000", 2020-06-12T02:38:17.037 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9720000000000002e-05 2020-06-12T02:38:17.037 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.037 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.038 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:17.038 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887300+0000", 2020-06-12T02:38:17.038 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047354100000000001 2020-06-12T02:38:17.038 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.038 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.039 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:17.039 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887774+0000", 2020-06-12T02:38:17.039 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0030649750000000002 2020-06-12T02:38:17.039 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.039 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.040 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:17.040 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.890839+0000", 2020-06-12T02:38:17.040 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016995321000000001 2020-06-12T02:38:17.040 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.040 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.041 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:17.041 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.907834+0000", 2020-06-12T02:38:17.041 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7417999999999999e-05 2020-06-12T02:38:17.041 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.041 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.041 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:17.042 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.907882+0000", 2020-06-12T02:38:17.042 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021442571000000001 2020-06-12T02:38:17.042 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.042 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:17.042 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.043 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.043 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.043 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1573 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~712704] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-06-12T02:38:17.043 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.890888+0000", 2020-06-12T02:38:17.043 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 43.111173694000001, 2020-06-12T02:38:17.043 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018868001999999998, 2020-06-12T02:38:17.044 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:17.044 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:17.044 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:17.044 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:17.044 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:17.045 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1573 2020-06-12T02:38:17.045 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.045 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:17.045 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.045 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:17.046 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.890888+0000", 2020-06-12T02:38:17.046 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:17.046 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.046 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.046 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:17.046 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.890888+0000", 2020-06-12T02:38:17.047 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000683605 2020-06-12T02:38:17.047 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.047 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.047 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:17.047 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891571+0000", 2020-06-12T02:38:17.047 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4999999999999999e-06 2020-06-12T02:38:17.048 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.048 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.048 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:17.048 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891577+0000", 2020-06-12T02:38:17.048 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.171e-05 2020-06-12T02:38:17.049 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.049 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.049 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:17.049 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891589+0000", 2020-06-12T02:38:17.049 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7650000000000001e-06 2020-06-12T02:38:17.050 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.050 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.050 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:17.050 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891593+0000", 2020-06-12T02:38:17.050 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6689000000000001e-05 2020-06-12T02:38:17.051 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.051 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.051 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:17.051 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891630+0000", 2020-06-12T02:38:17.051 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9877999999999997e-05 2020-06-12T02:38:17.052 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.052 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.052 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:17.052 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891710+0000", 2020-06-12T02:38:17.052 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053367200000000001 2020-06-12T02:38:17.052 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.053 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.053 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:17.053 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.892244+0000", 2020-06-12T02:38:17.053 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004927196 2020-06-12T02:38:17.053 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.054 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.054 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:17.054 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.897171+0000", 2020-06-12T02:38:17.055 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012496419 2020-06-12T02:38:17.055 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.055 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.055 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:17.055 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.909667+0000", 2020-06-12T02:38:17.055 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8567999999999996e-05 2020-06-12T02:38:17.056 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.056 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.056 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:17.056 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.909756+0000", 2020-06-12T02:38:17.056 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018868001999999998 2020-06-12T02:38:17.057 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.057 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:17.057 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.057 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.057 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.058 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1576 3.5s0 3:a446366d:::smithi10729407-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e26)", 2020-06-12T02:38:17.058 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.891955+0000", 2020-06-12T02:38:17.058 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 43.110106453999997, 2020-06-12T02:38:17.058 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019893246999999999, 2020-06-12T02:38:17.058 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:17.059 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:17.059 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:17.059 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:17.059 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:17.059 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1576 2020-06-12T02:38:17.060 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.060 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:17.060 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.060 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:17.060 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891955+0000", 2020-06-12T02:38:17.060 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:17.061 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.061 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.061 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:17.061 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891955+0000", 2020-06-12T02:38:17.061 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2067999999999999e-05 2020-06-12T02:38:17.062 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.062 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.062 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:17.062 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891977+0000", 2020-06-12T02:38:17.062 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.18e-06 2020-06-12T02:38:17.063 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.063 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.063 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:17.063 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891980+0000", 2020-06-12T02:38:17.063 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3629999999999999e-06 2020-06-12T02:38:17.063 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.064 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.064 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:17.064 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891987+0000", 2020-06-12T02:38:17.064 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.181e-06 2020-06-12T02:38:17.064 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.065 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.065 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:17.065 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891989+0000", 2020-06-12T02:38:17.065 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050959120000000002 2020-06-12T02:38:17.065 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.066 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.066 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:17.066 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.897085+0000", 2020-06-12T02:38:17.066 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1089999999999998e-05 2020-06-12T02:38:17.067 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.067 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.067 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:17.067 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.897166+0000", 2020-06-12T02:38:17.067 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013400235 2020-06-12T02:38:17.068 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.068 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.068 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:17.068 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.910566+0000", 2020-06-12T02:38:17.068 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000104981 2020-06-12T02:38:17.069 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.069 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.069 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:17.069 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.910671+0000", 2020-06-12T02:38:17.069 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001177237 2020-06-12T02:38:17.070 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.070 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.070 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:17.070 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.911848+0000", 2020-06-12T02:38:17.070 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019893246999999999 2020-06-12T02:38:17.071 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.071 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:17.071 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.071 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.071 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.072 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2913 3.5s0 3:a9510bda:::smithi10729407-14:head [append 0~770048,stat] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:17.072 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.070354+0000", 2020-06-12T02:38:17.072 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 39.931707652999997, 2020-06-12T02:38:17.072 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60690798599999995, 2020-06-12T02:38:17.072 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:17.073 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:17.073 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:17.073 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:17.073 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:17.073 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2913 2020-06-12T02:38:17.074 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.074 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:17.074 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.074 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:17.074 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.070354+0000", 2020-06-12T02:38:17.075 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:17.075 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.075 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.075 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:17.076 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.070354+0000", 2020-06-12T02:38:17.076 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00098478900000000002 2020-06-12T02:38:17.076 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.076 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.076 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:17.077 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071339+0000", 2020-06-12T02:38:17.077 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8349999999999993e-06 2020-06-12T02:38:17.077 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.077 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.077 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:17.078 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071347+0000", 2020-06-12T02:38:17.078 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0749e-05 2020-06-12T02:38:17.078 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.078 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.079 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:17.079 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071358+0000", 2020-06-12T02:38:17.079 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9369999999999997e-06 2020-06-12T02:38:17.080 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.080 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.080 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:17.080 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071362+0000", 2020-06-12T02:38:17.080 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0426e-05 2020-06-12T02:38:17.080 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.081 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.081 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:17.081 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071423+0000", 2020-06-12T02:38:17.081 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4129999999999998e-05 2020-06-12T02:38:17.081 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.082 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.082 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:17.082 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071487+0000", 2020-06-12T02:38:17.082 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053752599999999998 2020-06-12T02:38:17.082 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.083 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.083 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:17.083 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072024+0000", 2020-06-12T02:38:17.083 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036162579999999998 2020-06-12T02:38:17.083 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.084 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.084 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:17.084 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075640+0000", 2020-06-12T02:38:17.084 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60152201999999999 2020-06-12T02:38:17.084 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.085 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.085 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:17.085 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677162+0000", 2020-06-12T02:38:17.085 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9315999999999996e-05 2020-06-12T02:38:17.086 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.086 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.086 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:17.086 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677262+0000", 2020-06-12T02:38:17.086 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60690798599999995 2020-06-12T02:38:17.087 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.087 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:17.087 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.087 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.087 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.087 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2914 3.5s0 3:a9510bda:::smithi10729407-14:head [append 0~401408] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:17.088 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.072409+0000", 2020-06-12T02:38:17.088 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 39.929652208999997, 2020-06-12T02:38:17.088 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.61956801100000003, 2020-06-12T02:38:17.088 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:17.088 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:17.089 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:17.089 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:17.089 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:17.089 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2914 2020-06-12T02:38:17.089 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.089 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:17.090 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.090 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:17.090 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072409+0000", 2020-06-12T02:38:17.090 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:17.090 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.091 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.091 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:17.091 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072409+0000", 2020-06-12T02:38:17.091 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000446317 2020-06-12T02:38:17.091 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.091 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.092 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:17.092 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072856+0000", 2020-06-12T02:38:17.092 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2140000000000004e-06 2020-06-12T02:38:17.092 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.092 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.093 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:17.093 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072861+0000", 2020-06-12T02:38:17.093 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8710000000000003e-06 2020-06-12T02:38:17.093 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.093 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.094 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:17.094 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072870+0000", 2020-06-12T02:38:17.094 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2270000000000004e-06 2020-06-12T02:38:17.094 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.094 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.094 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:17.095 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072874+0000", 2020-06-12T02:38:17.095 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024649960000000001 2020-06-12T02:38:17.095 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.095 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.095 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:17.096 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075339+0000", 2020-06-12T02:38:17.096 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9179999999999996e-05 2020-06-12T02:38:17.096 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.096 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.096 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:17.096 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075438+0000", 2020-06-12T02:38:17.097 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60189281400000005 2020-06-12T02:38:17.097 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.097 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.097 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:17.097 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677331+0000", 2020-06-12T02:38:17.098 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012602800000000001 2020-06-12T02:38:17.098 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.098 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.098 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:17.098 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677457+0000", 2020-06-12T02:38:17.099 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00088167699999999998 2020-06-12T02:38:17.099 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.099 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.099 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:17.099 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.678339+0000", 2020-06-12T02:38:17.099 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004312995 2020-06-12T02:38:17.100 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.100 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.100 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:17.100 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.682652+0000", 2020-06-12T02:38:17.100 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0092869140000000003 2020-06-12T02:38:17.101 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.101 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.101 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:17.101 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691938+0000", 2020-06-12T02:38:17.101 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8778e-05 2020-06-12T02:38:17.101 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.102 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.102 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:17.102 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691977+0000", 2020-06-12T02:38:17.102 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.61956801100000003 2020-06-12T02:38:17.103 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.103 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:17.103 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.104 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.104 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.104 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2915 3.5s0 3:a9510bda:::smithi10729407-14:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:17.104 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.072912+0000", 2020-06-12T02:38:17.104 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 39.929149469000002, 2020-06-12T02:38:17.104 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.6192204, 2020-06-12T02:38:17.105 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:17.105 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:17.105 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:17.105 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:17.105 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:17.106 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2915 2020-06-12T02:38:17.106 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.106 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:17.106 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.106 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:17.107 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072912+0000", 2020-06-12T02:38:17.107 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:17.107 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.107 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.107 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:17.107 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072912+0000", 2020-06-12T02:38:17.108 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9201e-05 2020-06-12T02:38:17.108 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.108 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.108 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:17.108 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072951+0000", 2020-06-12T02:38:17.109 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5480000000000002e-06 2020-06-12T02:38:17.109 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.109 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.109 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:17.109 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072955+0000", 2020-06-12T02:38:17.109 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3879999999999997e-06 2020-06-12T02:38:17.110 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.110 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.110 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:17.110 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072961+0000", 2020-06-12T02:38:17.111 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6589999999999999e-06 2020-06-12T02:38:17.111 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.111 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.111 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:17.111 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072964+0000", 2020-06-12T02:38:17.111 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025221039999999998 2020-06-12T02:38:17.112 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.112 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.112 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:17.112 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075486+0000", 2020-06-12T02:38:17.112 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5038999999999998e-05 2020-06-12T02:38:17.113 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.113 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.113 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:17.113 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075561+0000", 2020-06-12T02:38:17.113 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60452397499999999 2020-06-12T02:38:17.114 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.114 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.114 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:17.114 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680085+0000", 2020-06-12T02:38:17.114 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1398999999999999e-05 2020-06-12T02:38:17.114 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.115 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.115 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:17.115 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680156+0000", 2020-06-12T02:38:17.115 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000274465 2020-06-12T02:38:17.115 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.115 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.116 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:17.116 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680431+0000", 2020-06-12T02:38:17.116 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025071999999999998 2020-06-12T02:38:17.116 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.116 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.117 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:17.117 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.682938+0000", 2020-06-12T02:38:17.117 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0091116449999999998 2020-06-12T02:38:17.117 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.117 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.117 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:17.118 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692050+0000", 2020-06-12T02:38:17.118 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2776999999999997e-05 2020-06-12T02:38:17.118 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.118 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.118 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:17.119 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692132+0000", 2020-06-12T02:38:17.119 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.6192204 2020-06-12T02:38:17.119 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.119 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:17.119 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.120 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.120 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.120 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2916 3.5s0 3:a9510bda:::smithi10729407-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:38:17.120 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.072986+0000", 2020-06-12T02:38:17.120 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 39.929075230000002, 2020-06-12T02:38:17.122 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.62063941099999997, 2020-06-12T02:38:17.122 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:17.122 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:17.122 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:17.123 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:17.123 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:17.123 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2916 2020-06-12T02:38:17.123 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.123 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:17.124 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.124 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:17.124 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072986+0000", 2020-06-12T02:38:17.124 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:17.124 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.124 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.125 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:17.125 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072986+0000", 2020-06-12T02:38:17.125 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2101000000000001e-05 2020-06-12T02:38:17.125 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.125 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.126 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:17.126 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073008+0000", 2020-06-12T02:38:17.126 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1549999999999999e-06 2020-06-12T02:38:17.126 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.127 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.127 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:17.127 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073011+0000", 2020-06-12T02:38:17.127 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2360000000000001e-06 2020-06-12T02:38:17.127 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.127 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.128 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:17.128 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073018+0000", 2020-06-12T02:38:17.128 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2790000000000001e-06 2020-06-12T02:38:17.128 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.128 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.129 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:17.129 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073020+0000", 2020-06-12T02:38:17.129 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025637670000000002 2020-06-12T02:38:17.129 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.129 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.130 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:17.130 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075584+0000", 2020-06-12T02:38:17.130 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1814999999999998e-05 2020-06-12T02:38:17.130 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.131 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.131 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:17.131 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075636+0000", 2020-06-12T02:38:17.131 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60499943199999995 2020-06-12T02:38:17.131 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.132 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.132 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:17.132 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680635+0000", 2020-06-12T02:38:17.132 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0968000000000002e-05 2020-06-12T02:38:17.132 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.132 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.133 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:17.133 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680706+0000", 2020-06-12T02:38:17.133 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011777101999999999 2020-06-12T02:38:17.133 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.133 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.134 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:17.134 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692483+0000", 2020-06-12T02:38:17.134 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5890999999999995e-05 2020-06-12T02:38:17.134 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.134 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.134 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:17.135 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692569+0000", 2020-06-12T02:38:17.135 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001056665 2020-06-12T02:38:17.135 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.135 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.135 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:17.136 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.693626+0000", 2020-06-12T02:38:17.136 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.62063941099999997 2020-06-12T02:38:17.136 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.136 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:17.136 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.136 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.137 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.137 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2922 3.ds1 3:b983fd28:::smithi10729407-1:head [append 0~466944] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:17.137 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.097192+0000", 2020-06-12T02:38:17.138 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 39.904869863000002, 2020-06-12T02:38:17.138 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58672644699999998, 2020-06-12T02:38:17.138 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:17.138 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:17.138 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:17.139 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:17.139 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:17.139 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2922 2020-06-12T02:38:17.139 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.139 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:17.140 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.140 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:17.140 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097192+0000", 2020-06-12T02:38:17.140 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:17.141 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.141 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.141 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:17.141 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097192+0000", 2020-06-12T02:38:17.141 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047206100000000002 2020-06-12T02:38:17.142 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.142 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.142 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:17.142 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097664+0000", 2020-06-12T02:38:17.142 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8359999999999999e-06 2020-06-12T02:38:17.143 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.143 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.143 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:17.143 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097668+0000", 2020-06-12T02:38:17.143 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0815999999999999e-05 2020-06-12T02:38:17.143 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.144 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.144 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:17.144 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097679+0000", 2020-06-12T02:38:17.144 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.028e-06 2020-06-12T02:38:17.144 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.145 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.145 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:17.145 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097684+0000", 2020-06-12T02:38:17.145 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3932999999999999e-05 2020-06-12T02:38:17.145 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.146 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.146 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:17.146 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097748+0000", 2020-06-12T02:38:17.146 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2214999999999999e-05 2020-06-12T02:38:17.147 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.147 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.147 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:17.147 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097820+0000", 2020-06-12T02:38:17.147 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00041583800000000003 2020-06-12T02:38:17.148 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.148 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.148 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:17.148 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.098236+0000", 2020-06-12T02:38:17.149 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0020339960000000002 2020-06-12T02:38:17.149 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.149 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.149 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:17.150 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.100270+0000", 2020-06-12T02:38:17.150 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58355657299999997 2020-06-12T02:38:17.150 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.150 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.151 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:17.151 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.683827+0000", 2020-06-12T02:38:17.151 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1150999999999995e-05 2020-06-12T02:38:17.151 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.152 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.152 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:17.152 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.683918+0000", 2020-06-12T02:38:17.152 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58672644699999998 2020-06-12T02:38:17.152 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.152 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:17.153 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.153 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.153 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.153 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2923 3.ds1 3:b983fd28:::smithi10729407-1:head [append 0~614400] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:17.153 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.100504+0000", 2020-06-12T02:38:17.154 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 39.901557222000001, 2020-06-12T02:38:17.154 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58853946999999995, 2020-06-12T02:38:17.154 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:17.154 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:17.154 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:17.155 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:17.155 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:17.155 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2923 2020-06-12T02:38:17.155 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.155 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:17.155 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.156 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:17.156 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.100504+0000", 2020-06-12T02:38:17.156 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:17.156 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.156 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.157 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:17.157 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.100504+0000", 2020-06-12T02:38:17.157 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00068395399999999996 2020-06-12T02:38:17.157 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.157 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.157 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:17.158 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101188+0000", 2020-06-12T02:38:17.158 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4050000000000003e-06 2020-06-12T02:38:17.158 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.158 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.158 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:17.159 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101196+0000", 2020-06-12T02:38:17.159 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.328e-05 2020-06-12T02:38:17.159 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.159 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.159 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:17.159 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101209+0000", 2020-06-12T02:38:17.160 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3419999999999996e-06 2020-06-12T02:38:17.160 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.160 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.160 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:17.160 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101215+0000", 2020-06-12T02:38:17.161 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1389000000000005e-05 2020-06-12T02:38:17.161 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.161 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.161 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:17.162 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101277+0000", 2020-06-12T02:38:17.162 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7724000000000004e-05 2020-06-12T02:38:17.162 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.162 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.162 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:17.163 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101344+0000", 2020-06-12T02:38:17.163 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047852499999999999 2020-06-12T02:38:17.163 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.163 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.163 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:17.163 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101823+0000", 2020-06-12T02:38:17.164 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0039255510000000002 2020-06-12T02:38:17.164 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.164 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.164 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:17.165 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105748+0000", 2020-06-12T02:38:17.165 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58321340099999996 2020-06-12T02:38:17.165 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.166 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.166 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:17.166 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.688962+0000", 2020-06-12T02:38:17.166 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1898999999999996e-05 2020-06-12T02:38:17.167 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.167 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.167 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:17.167 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689044+0000", 2020-06-12T02:38:17.168 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58853946999999995 2020-06-12T02:38:17.168 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.168 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:17.168 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.168 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.169 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.169 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2924 3.ds1 3:b983fd28:::smithi10729407-1:head [append 0~188416] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:17.169 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.101633+0000", 2020-06-12T02:38:17.169 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 39.900428228000003, 2020-06-12T02:38:17.169 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58758259199999996, 2020-06-12T02:38:17.169 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:17.170 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:17.170 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:17.170 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:17.170 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:17.170 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2924 2020-06-12T02:38:17.171 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.171 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:17.171 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.171 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:17.172 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101633+0000", 2020-06-12T02:38:17.172 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:17.172 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.172 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.172 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:17.173 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101633+0000", 2020-06-12T02:38:17.173 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00018775399999999999 2020-06-12T02:38:17.173 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.173 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.173 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:17.174 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101821+0000", 2020-06-12T02:38:17.174 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7669999999999998e-06 2020-06-12T02:38:17.174 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.174 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.174 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:17.174 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101828+0000", 2020-06-12T02:38:17.175 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2716e-05 2020-06-12T02:38:17.175 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.175 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.175 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:17.176 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101840+0000", 2020-06-12T02:38:17.176 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.215e-06 2020-06-12T02:38:17.176 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.177 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.177 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:17.178 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101846+0000", 2020-06-12T02:38:17.178 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021214250000000001 2020-06-12T02:38:17.178 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.178 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.178 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:17.179 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.103967+0000", 2020-06-12T02:38:17.179 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1018000000000003e-05 2020-06-12T02:38:17.179 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.179 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.179 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:17.180 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.104038+0000", 2020-06-12T02:38:17.180 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000333364 2020-06-12T02:38:17.180 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.180 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.180 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:17.181 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.104371+0000", 2020-06-12T02:38:17.181 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001378546 2020-06-12T02:38:17.181 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.181 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.181 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:17.181 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105750+0000", 2020-06-12T02:38:17.182 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58338272899999999 2020-06-12T02:38:17.182 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.182 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.182 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:17.182 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689133+0000", 2020-06-12T02:38:17.183 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3058000000000003e-05 2020-06-12T02:38:17.183 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.183 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.183 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:17.183 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689216+0000", 2020-06-12T02:38:17.183 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58758259199999996 2020-06-12T02:38:17.184 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.184 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:17.184 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.184 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.184 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.185 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2925 3.ds1 3:b983fd28:::smithi10729407-1:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:17.185 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.101899+0000", 2020-06-12T02:38:17.185 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 39.900162563000002, 2020-06-12T02:38:17.185 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58747810899999997, 2020-06-12T02:38:17.185 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:17.186 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:17.186 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:17.186 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:17.186 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:17.186 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2925 2020-06-12T02:38:17.187 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.187 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:17.187 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.187 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:17.187 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101899+0000", 2020-06-12T02:38:17.187 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:17.188 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.188 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.188 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:17.188 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101899+0000", 2020-06-12T02:38:17.189 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2906999999999998e-05 2020-06-12T02:38:17.189 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.189 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.189 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:17.189 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101972+0000", 2020-06-12T02:38:17.189 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1789999999999996e-06 2020-06-12T02:38:17.190 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.190 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.190 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:17.190 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101978+0000", 2020-06-12T02:38:17.190 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1358e-05 2020-06-12T02:38:17.191 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.191 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.191 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:17.191 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101989+0000", 2020-06-12T02:38:17.191 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.031e-06 2020-06-12T02:38:17.192 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.192 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.192 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:17.192 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101993+0000", 2020-06-12T02:38:17.192 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033218990000000001 2020-06-12T02:38:17.192 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.193 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.193 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:17.193 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105315+0000", 2020-06-12T02:38:17.193 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6324999999999999e-05 2020-06-12T02:38:17.193 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.194 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.194 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:17.194 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105382+0000", 2020-06-12T02:38:17.194 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000216624 2020-06-12T02:38:17.194 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.194 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.195 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:17.195 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105598+0000", 2020-06-12T02:38:17.195 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001236586 2020-06-12T02:38:17.195 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.195 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.196 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:17.196 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.106835+0000", 2020-06-12T02:38:17.196 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58245509200000001 2020-06-12T02:38:17.197 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.197 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.197 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:17.198 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689290+0000", 2020-06-12T02:38:17.198 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7107999999999993e-05 2020-06-12T02:38:17.200 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.200 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.200 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:17.200 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689377+0000", 2020-06-12T02:38:17.200 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58747810899999997 2020-06-12T02:38:17.201 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.201 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:17.201 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.201 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.201 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.202 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2926 3.ds1 3:b983fd28:::smithi10729407-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:38:17.202 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.102036+0000", 2020-06-12T02:38:17.202 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 39.900025579000001, 2020-06-12T02:38:17.202 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58868048699999997, 2020-06-12T02:38:17.202 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:17.203 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:17.203 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:17.203 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:17.203 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:17.203 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2926 2020-06-12T02:38:17.204 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.204 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:17.204 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.204 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:17.204 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102036+0000", 2020-06-12T02:38:17.204 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:17.205 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.205 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.205 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:17.205 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102036+0000", 2020-06-12T02:38:17.205 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3786999999999999e-05 2020-06-12T02:38:17.206 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.206 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.206 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:17.206 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102080+0000", 2020-06-12T02:38:17.206 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7660000000000004e-06 2020-06-12T02:38:17.207 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.207 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.207 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:17.207 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102085+0000", 2020-06-12T02:38:17.207 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3145e-05 2020-06-12T02:38:17.208 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.208 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.208 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:17.208 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102099+0000", 2020-06-12T02:38:17.208 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2440000000000004e-06 2020-06-12T02:38:17.209 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.209 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.209 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:17.209 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102103+0000", 2020-06-12T02:38:17.209 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036826990000000002 2020-06-12T02:38:17.209 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.210 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.210 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:17.210 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105786+0000", 2020-06-12T02:38:17.210 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.6774000000000006e-05 2020-06-12T02:38:17.210 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.211 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.211 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:17.211 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105882+0000", 2020-06-12T02:38:17.211 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58352084900000001 2020-06-12T02:38:17.211 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.212 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.212 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:17.212 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689403+0000", 2020-06-12T02:38:17.212 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3788999999999995e-05 2020-06-12T02:38:17.212 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.212 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.213 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:17.213 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689467+0000", 2020-06-12T02:38:17.213 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001249434 2020-06-12T02:38:17.213 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.213 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.214 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:17.214 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690716+0000", 2020-06-12T02:38:17.214 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58868048699999997 2020-06-12T02:38:17.214 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.214 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:17.214 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.215 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.215 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.215 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2950 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~598016] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:17.215 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.162562+0000", 2020-06-12T02:38:17.215 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 39.839499205000003, 2020-06-12T02:38:17.216 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52780611399999999, 2020-06-12T02:38:17.216 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:17.216 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:17.216 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:17.216 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:17.217 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:17.217 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2950 2020-06-12T02:38:17.217 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.217 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:17.217 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.217 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:17.218 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.162562+0000", 2020-06-12T02:38:17.218 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:17.218 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.218 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.218 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:17.219 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.162562+0000", 2020-06-12T02:38:17.219 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00070718099999999998 2020-06-12T02:38:17.219 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.219 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.219 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:17.219 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163269+0000", 2020-06-12T02:38:17.220 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2029999999999997e-06 2020-06-12T02:38:17.220 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.220 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.220 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:17.220 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163276+0000", 2020-06-12T02:38:17.221 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2734999999999999e-05 2020-06-12T02:38:17.221 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.221 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.221 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:17.221 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163288+0000", 2020-06-12T02:38:17.221 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.404e-06 2020-06-12T02:38:17.222 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.222 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.222 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:17.222 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163295+0000", 2020-06-12T02:38:17.222 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5784000000000006e-05 2020-06-12T02:38:17.223 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.223 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.223 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:17.223 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163361+0000", 2020-06-12T02:38:17.223 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5318e-05 2020-06-12T02:38:17.224 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.224 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.224 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:17.224 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163436+0000", 2020-06-12T02:38:17.225 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054801100000000005 2020-06-12T02:38:17.225 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.225 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.226 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:17.226 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163984+0000", 2020-06-12T02:38:17.226 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.007023077 2020-06-12T02:38:17.226 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.227 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.227 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:17.227 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171007+0000", 2020-06-12T02:38:17.227 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51932051199999996 2020-06-12T02:38:17.227 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.227 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.228 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:17.228 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690327+0000", 2020-06-12T02:38:17.228 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0889000000000001e-05 2020-06-12T02:38:17.228 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.228 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.229 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:17.229 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690368+0000", 2020-06-12T02:38:17.229 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52780611399999999 2020-06-12T02:38:17.229 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.229 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:17.229 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.230 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.230 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.230 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2951 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~532480] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:17.230 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.165470+0000", 2020-06-12T02:38:17.230 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 39.836591597000002, 2020-06-12T02:38:17.231 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52507917900000001, 2020-06-12T02:38:17.231 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:17.231 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:17.231 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:17.231 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:17.231 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:17.232 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2951 2020-06-12T02:38:17.232 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.232 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:17.232 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.232 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:17.233 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.165470+0000", 2020-06-12T02:38:17.233 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:17.233 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.233 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.233 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:17.233 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.165470+0000", 2020-06-12T02:38:17.234 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00062231600000000004 2020-06-12T02:38:17.234 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.234 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.234 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:17.234 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166092+0000", 2020-06-12T02:38:17.235 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1610000000000002e-06 2020-06-12T02:38:17.235 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.235 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.235 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:17.236 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166097+0000", 2020-06-12T02:38:17.236 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7780000000000002e-06 2020-06-12T02:38:17.236 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.236 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.236 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:17.237 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166107+0000", 2020-06-12T02:38:17.237 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7169999999999999e-06 2020-06-12T02:38:17.237 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.237 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.238 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:17.238 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166112+0000", 2020-06-12T02:38:17.238 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4792000000000002e-05 2020-06-12T02:38:17.238 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.238 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.238 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:17.239 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166147+0000", 2020-06-12T02:38:17.239 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9336e-05 2020-06-12T02:38:17.239 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.239 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.239 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:17.240 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166236+0000", 2020-06-12T02:38:17.240 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00067747200000000004 2020-06-12T02:38:17.240 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.240 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.240 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:17.240 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166913+0000", 2020-06-12T02:38:17.241 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0047718669999999999 2020-06-12T02:38:17.241 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.241 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.241 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:17.241 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171685+0000", 2020-06-12T02:38:17.241 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51881990700000002 2020-06-12T02:38:17.242 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.242 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.242 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:17.242 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690505+0000", 2020-06-12T02:38:17.242 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3832999999999999e-05 2020-06-12T02:38:17.243 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.243 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.243 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:17.243 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690549+0000", 2020-06-12T02:38:17.243 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52507917900000001 2020-06-12T02:38:17.244 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.244 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:17.244 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.244 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.244 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.245 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2952 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~270336] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:17.245 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.166891+0000", 2020-06-12T02:38:17.245 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 39.835169948000001, 2020-06-12T02:38:17.245 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52426417000000003, 2020-06-12T02:38:17.245 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:17.246 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:17.246 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:17.246 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:17.246 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:17.246 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2952 2020-06-12T02:38:17.247 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.247 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:17.247 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.247 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:17.247 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166891+0000", 2020-06-12T02:38:17.247 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:17.248 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.248 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.248 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:17.248 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166891+0000", 2020-06-12T02:38:17.248 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00031878599999999999 2020-06-12T02:38:17.248 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.249 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.249 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:17.249 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167210+0000", 2020-06-12T02:38:17.249 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2360000000000004e-06 2020-06-12T02:38:17.249 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.250 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.250 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:17.250 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167219+0000", 2020-06-12T02:38:17.250 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3298e-05 2020-06-12T02:38:17.250 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.250 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.251 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:17.251 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167232+0000", 2020-06-12T02:38:17.251 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5849999999999999e-06 2020-06-12T02:38:17.251 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.251 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.252 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:17.252 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167237+0000", 2020-06-12T02:38:17.252 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0018755740000000001 2020-06-12T02:38:17.252 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.252 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.252 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:17.253 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.169113+0000", 2020-06-12T02:38:17.253 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000104768 2020-06-12T02:38:17.253 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.253 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.253 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:17.254 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.169218+0000", 2020-06-12T02:38:17.254 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00057389799999999999 2020-06-12T02:38:17.254 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.254 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.254 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:17.254 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.169792+0000", 2020-06-12T02:38:17.255 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0019333989999999999 2020-06-12T02:38:17.255 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.255 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.255 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:17.255 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171725+0000", 2020-06-12T02:38:17.256 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51938508699999997 2020-06-12T02:38:17.256 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.256 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.256 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:17.256 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691110+0000", 2020-06-12T02:38:17.256 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5538999999999999e-05 2020-06-12T02:38:17.257 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.257 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.257 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:17.257 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691156+0000", 2020-06-12T02:38:17.257 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52426417000000003 2020-06-12T02:38:17.258 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.258 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:17.258 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.258 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.258 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.258 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2953 3.5s0 3:a446366d:::smithi10729407-11:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:17.259 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.167293+0000", 2020-06-12T02:38:17.259 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 39.834768500999999, 2020-06-12T02:38:17.259 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52428350999999995, 2020-06-12T02:38:17.259 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:17.259 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:17.260 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:17.260 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:17.260 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:17.260 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2953 2020-06-12T02:38:17.260 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.260 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:17.261 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.261 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:17.261 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167293+0000", 2020-06-12T02:38:17.261 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:17.261 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.262 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.262 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:17.262 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167293+0000", 2020-06-12T02:38:17.262 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3880000000000004e-05 2020-06-12T02:38:17.262 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.262 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.263 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:17.263 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167367+0000", 2020-06-12T02:38:17.263 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.319e-06 2020-06-12T02:38:17.263 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.263 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.264 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:17.264 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167373+0000", 2020-06-12T02:38:17.264 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1719000000000001e-05 2020-06-12T02:38:17.264 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.264 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.264 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:17.265 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167385+0000", 2020-06-12T02:38:17.265 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4580000000000003e-06 2020-06-12T02:38:17.265 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.265 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.265 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:17.266 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167389+0000", 2020-06-12T02:38:17.266 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036341110000000002 2020-06-12T02:38:17.266 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.266 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.266 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:17.266 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171023+0000", 2020-06-12T02:38:17.267 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9418000000000005e-05 2020-06-12T02:38:17.267 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.267 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.267 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:17.267 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171093+0000", 2020-06-12T02:38:17.268 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00029087900000000002 2020-06-12T02:38:17.268 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.268 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.268 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:17.268 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171384+0000", 2020-06-12T02:38:17.268 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000489347 2020-06-12T02:38:17.269 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.269 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.269 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:17.269 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171873+0000", 2020-06-12T02:38:17.269 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51963691300000003 2020-06-12T02:38:17.270 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.270 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.270 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:17.270 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691510+0000", 2020-06-12T02:38:17.270 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6465999999999997e-05 2020-06-12T02:38:17.270 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.271 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.271 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:17.271 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691576+0000", 2020-06-12T02:38:17.271 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52428350999999995 2020-06-12T02:38:17.271 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.271 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:17.272 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.272 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.272 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.272 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2954 3.5s0 3:a446366d:::smithi10729407-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:38:17.272 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.167432+0000", 2020-06-12T02:38:17.273 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 39.834629751999998, 2020-06-12T02:38:17.273 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52568652699999996, 2020-06-12T02:38:17.273 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:17.273 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:17.273 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:17.274 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:17.274 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:17.274 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2954 2020-06-12T02:38:17.274 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.274 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:17.274 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.275 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:17.275 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167432+0000", 2020-06-12T02:38:17.275 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:17.275 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.275 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.276 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:17.276 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167432+0000", 2020-06-12T02:38:17.276 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3556000000000001e-05 2020-06-12T02:38:17.276 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.276 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.276 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:17.277 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167475+0000", 2020-06-12T02:38:17.277 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7350000000000001e-06 2020-06-12T02:38:17.277 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.277 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.277 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:17.278 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167481+0000", 2020-06-12T02:38:17.278 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3128e-05 2020-06-12T02:38:17.278 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.278 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.278 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:17.278 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167494+0000", 2020-06-12T02:38:17.279 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1579999999999998e-06 2020-06-12T02:38:17.279 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.279 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.279 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:17.279 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167498+0000", 2020-06-12T02:38:17.280 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004082174 2020-06-12T02:38:17.280 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.280 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.280 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:17.280 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171580+0000", 2020-06-12T02:38:17.280 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8954000000000002e-05 2020-06-12T02:38:17.281 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.281 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.281 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:17.281 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171679+0000", 2020-06-12T02:38:17.281 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51993235100000001 2020-06-12T02:38:17.282 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.282 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.282 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:17.282 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691612+0000", 2020-06-12T02:38:17.282 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4329000000000001e-05 2020-06-12T02:38:17.282 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.283 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.283 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:17.283 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691666+0000", 2020-06-12T02:38:17.283 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001452142 2020-06-12T02:38:17.283 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.284 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.284 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:17.284 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.693118+0000", 2020-06-12T02:38:17.284 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52568652699999996 2020-06-12T02:38:17.284 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.284 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:17.285 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.285 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.285 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.285 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2959 3.ds1 3:b3445631:::smithi10729407-6:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e27)", 2020-06-12T02:38:17.285 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.176248+0000", 2020-06-12T02:38:17.286 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 39.825813011000001, 2020-06-12T02:38:17.287 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51549877099999997, 2020-06-12T02:38:17.287 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:17.288 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:17.288 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:17.288 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:17.288 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:17.288 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2959 2020-06-12T02:38:17.289 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.289 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:17.289 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.289 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:17.289 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176248+0000", 2020-06-12T02:38:17.290 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:17.290 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.290 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.290 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:17.290 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176248+0000", 2020-06-12T02:38:17.291 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5120999999999999e-05 2020-06-12T02:38:17.291 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.291 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.291 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:17.291 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176274+0000", 2020-06-12T02:38:17.292 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9189999999999998e-06 2020-06-12T02:38:17.292 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.292 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.292 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:17.292 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176278+0000", 2020-06-12T02:38:17.292 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6339e-05 2020-06-12T02:38:17.293 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.293 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.293 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:17.293 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176305+0000", 2020-06-12T02:38:17.293 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9089999999999999e-06 2020-06-12T02:38:17.294 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.294 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.294 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:17.294 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176310+0000", 2020-06-12T02:38:17.294 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6095999999999999e-05 2020-06-12T02:38:17.295 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.295 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.295 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:17.296 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176376+0000", 2020-06-12T02:38:17.296 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8590999999999994e-05 2020-06-12T02:38:17.296 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.296 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.296 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:17.296 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176444+0000", 2020-06-12T02:38:17.297 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00024346400000000001 2020-06-12T02:38:17.297 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.297 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.297 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:17.297 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176688+0000", 2020-06-12T02:38:17.298 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00044220899999999999 2020-06-12T02:38:17.298 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.298 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.298 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:17.298 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.177130+0000", 2020-06-12T02:38:17.299 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51452984400000001 2020-06-12T02:38:17.299 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.299 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.299 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:17.299 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691660+0000", 2020-06-12T02:38:17.300 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7279000000000002e-05 2020-06-12T02:38:17.300 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.300 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.300 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:17.300 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691747+0000", 2020-06-12T02:38:17.301 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51549877099999997 2020-06-12T02:38:17.301 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.301 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:17.301 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.301 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.302 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.302 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:3032 3.ds1 3:b983fd28:::smithi10729407-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:38:17.302 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.864289+0000", 2020-06-12T02:38:17.302 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 39.137772093000002, 2020-06-12T02:38:17.303 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019572216999999999, 2020-06-12T02:38:17.303 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:17.303 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:17.304 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:17.304 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:17.304 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:17.305 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 3032 2020-06-12T02:38:17.305 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.305 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:17.305 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.306 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:17.306 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864289+0000", 2020-06-12T02:38:17.306 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:17.306 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.306 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.307 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:17.307 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864289+0000", 2020-06-12T02:38:17.307 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5701999999999997e-05 2020-06-12T02:38:17.307 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.307 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.308 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:17.308 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864335+0000", 2020-06-12T02:38:17.308 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0910000000000004e-06 2020-06-12T02:38:17.308 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.308 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.308 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:17.309 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864342+0000", 2020-06-12T02:38:17.309 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2301e-05 2020-06-12T02:38:17.309 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.309 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.310 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:17.310 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864354+0000", 2020-06-12T02:38:17.310 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5199999999999997e-06 2020-06-12T02:38:17.310 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.310 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.310 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:17.311 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864360+0000", 2020-06-12T02:38:17.311 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00049180999999999997 2020-06-12T02:38:17.311 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.311 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.311 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:17.312 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864852+0000", 2020-06-12T02:38:17.312 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6725000000000003e-05 2020-06-12T02:38:17.312 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.312 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.312 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:17.312 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864908+0000", 2020-06-12T02:38:17.313 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0089687010000000008 2020-06-12T02:38:17.313 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.313 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.313 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:17.313 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.873877+0000", 2020-06-12T02:38:17.314 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3382999999999999e-05 2020-06-12T02:38:17.314 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.314 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.314 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:17.314 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.873941+0000", 2020-06-12T02:38:17.315 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.006408927 2020-06-12T02:38:17.315 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.315 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.315 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:17.316 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.880350+0000", 2020-06-12T02:38:17.316 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3905e-05 2020-06-12T02:38:17.316 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.316 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.316 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:17.317 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.880403+0000", 2020-06-12T02:38:17.317 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0034581519999999999 2020-06-12T02:38:17.317 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.317 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.317 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:17.318 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.883862+0000", 2020-06-12T02:38:17.318 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019572216999999999 2020-06-12T02:38:17.318 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.318 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:17.319 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.319 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.319 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:17.319 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:17.319 INFO:tasks.rados.rados.0.smithi107.stdout:4527: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:17.320 INFO:tasks.rados.rados.0.smithi107.stdout:4526: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:17.320 INFO:tasks.rados.rados.0.smithi107.stdout:4526: finishing write tid 5 to smithi10729407-3 2020-06-12T02:38:17.320 INFO:tasks.rados.rados.0.smithi107.stdout:4526: finishing write tid 6 to smithi10729407-3 2020-06-12T02:38:17.320 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 807 (ObjNum 4525 snap 0 seq_num 4525) dirty exists 2020-06-12T02:38:17.320 INFO:tasks.rados.rados.0.smithi107.stdout:4526: left oid 3 (ObjNum 4525 snap 0 seq_num 4525) 2020-06-12T02:38:17.338 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-06-12T02:38:17.339 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:17.319+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:17.339 INFO:tasks.rados.rados.0.smithi107.stdout:4525: done (4 left) 2020-06-12T02:38:17.339 INFO:tasks.rados.rados.0.smithi107.stdout:4526: done (3 left) 2020-06-12T02:38:17.340 INFO:tasks.rados.rados.0.smithi107.stdout:4528: done (2 left) 2020-06-12T02:38:17.340 INFO:tasks.rados.rados.0.smithi107.stdout:4529: delete oid 20 current snap is 0 2020-06-12T02:38:17.340 INFO:tasks.rados.rados.0.smithi107.stdout:4527: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:17.340 INFO:tasks.rados.rados.0.smithi107.stdout:4529: done (2 left) 2020-06-12T02:38:17.340 INFO:tasks.rados.rados.0.smithi107.stdout:4530: delete oid 2 current snap is 0 2020-06-12T02:38:17.340 INFO:tasks.rados.rados.0.smithi107.stdout:4527: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:17.341 INFO:tasks.rados.rados.0.smithi107.stdout:4527: finishing write tid 5 to smithi10729407-14 2020-06-12T02:38:17.341 INFO:tasks.rados.rados.0.smithi107.stdout:4527: finishing write tid 6 to smithi10729407-14 2020-06-12T02:38:17.341 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 810 (ObjNum 4526 snap 0 seq_num 4526) dirty exists 2020-06-12T02:38:17.341 INFO:tasks.rados.rados.0.smithi107.stdout:4527: left oid 14 (ObjNum 4526 snap 0 seq_num 4526) 2020-06-12T02:38:17.341 INFO:tasks.rados.rados.0.smithi107.stdout:4527: done (2 left) 2020-06-12T02:38:17.342 INFO:tasks.rados.rados.0.smithi107.stdout:4530: done (1 left) 2020-06-12T02:38:17.342 INFO:tasks.rados.rados.0.smithi107.stdout:4531: delete oid 12 current snap is 0 2020-06-12T02:38:17.342 INFO:tasks.rados.rados.0.smithi107.stdout:4531: done (1 left) 2020-06-12T02:38:17.342 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:17.342 INFO:tasks.rados.rados.0.smithi107.stdout:4532: seq_num 4531 ranges {0=688128,688128=671744,1359872=131072} 2020-06-12T02:38:17.343 INFO:tasks.rados.rados.0.smithi107.stdout:4532: writing smithi10729407-5 from 0 to 688128 tid 1 2020-06-12T02:38:17.343 INFO:tasks.rados.rados.0.smithi107.stdout:4532: writing smithi10729407-5 from 688128 to 1359872 tid 2 2020-06-12T02:38:17.343 INFO:tasks.rados.rados.0.smithi107.stdout:4532: writing smithi10729407-5 from 1359872 to 1490944 tid 3 2020-06-12T02:38:17.343 INFO:tasks.rados.rados.0.smithi107.stdout:4533: delete oid 20 current snap is 0 2020-06-12T02:38:17.343 INFO:tasks.rados.rados.0.smithi107.stdout:4533: done (2 left) 2020-06-12T02:38:17.344 INFO:tasks.rados.rados.0.smithi107.stdout:4534: delete oid 4 current snap is 0 2020-06-12T02:38:17.344 INFO:tasks.rados.rados.0.smithi107.stdout:4532: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:17.344 INFO:tasks.rados.rados.0.smithi107.stdout:4534: done (2 left) 2020-06-12T02:38:17.344 INFO:tasks.rados.rados.0.smithi107.stdout:4535: delete oid 16 current snap is 0 2020-06-12T02:38:17.344 INFO:tasks.rados.rados.0.smithi107.stdout:4532: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:17.345 INFO:tasks.rados.rados.0.smithi107.stdout:4532: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:17.345 INFO:tasks.rados.rados.0.smithi107.stdout:4532: finishing write tid 5 to smithi10729407-5 2020-06-12T02:38:17.345 INFO:tasks.rados.rados.0.smithi107.stdout:4535: done (2 left) 2020-06-12T02:38:17.345 INFO:tasks.rados.rados.0.smithi107.stdout:4536: delete oid 17 current snap is 0 2020-06-12T02:38:17.345 INFO:tasks.rados.rados.0.smithi107.stdout:4532: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:17.346 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 455 (ObjNum 4531 snap 0 seq_num 4531) dirty exists 2020-06-12T02:38:17.346 INFO:tasks.rados.rados.0.smithi107.stdout:4532: left oid 5 (ObjNum 4531 snap 0 seq_num 4531) 2020-06-12T02:38:17.346 INFO:tasks.rados.rados.0.smithi107.stdout:4532: done (2 left) 2020-06-12T02:38:17.346 INFO:tasks.rados.rados.0.smithi107.stdout:4536: done (1 left) 2020-06-12T02:38:17.346 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:38:17.347 INFO:tasks.rados.rados.0.smithi107.stdout:4537: seq_num 4536 ranges {0=778240,778240=548864,1327104=253952} 2020-06-12T02:38:17.347 INFO:tasks.rados.rados.0.smithi107.stdout:4537: writing smithi10729407-6 from 0 to 778240 tid 1 2020-06-12T02:38:17.347 INFO:tasks.rados.rados.0.smithi107.stdout:4537: writing smithi10729407-6 from 778240 to 1327104 tid 2 2020-06-12T02:38:17.347 INFO:tasks.rados.rados.0.smithi107.stdout:4537: writing smithi10729407-6 from 1327104 to 1581056 tid 3 2020-06-12T02:38:17.347 INFO:tasks.rados.rados.0.smithi107.stdout:4538: delete oid 16 current snap is 0 2020-06-12T02:38:17.347 INFO:tasks.rados.rados.0.smithi107.stdout:4538: done (2 left) 2020-06-12T02:38:17.348 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:38:17.348 INFO:tasks.rados.rados.0.smithi107.stdout:4539: seq_num 4538 ranges {0=409600,409600=712704,1122304=491520} 2020-06-12T02:38:17.348 INFO:tasks.rados.rados.0.smithi107.stdout:4539: writing smithi10729407-1 from 0 to 409600 tid 1 2020-06-12T02:38:17.348 INFO:tasks.rados.rados.0.smithi107.stdout:4539: writing smithi10729407-1 from 409600 to 1122304 tid 2 2020-06-12T02:38:17.348 INFO:tasks.rados.rados.0.smithi107.stdout:4539: writing smithi10729407-1 from 1122304 to 1613824 tid 3 2020-06-12T02:38:17.349 INFO:tasks.rados.rados.0.smithi107.stdout:4540: delete oid 11 current snap is 0 2020-06-12T02:38:17.349 INFO:tasks.rados.rados.0.smithi107.stdout:4537: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:17.349 INFO:tasks.rados.rados.0.smithi107.stdout:4537: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:17.349 INFO:tasks.rados.rados.0.smithi107.stdout:4537: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:17.349 INFO:tasks.rados.rados.0.smithi107.stdout:4537: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:17.349 INFO:tasks.rados.rados.0.smithi107.stdout:4537: finishing write tid 6 to smithi10729407-6 2020-06-12T02:38:17.350 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 838 (ObjNum 4536 snap 0 seq_num 4536) dirty exists 2020-06-12T02:38:17.350 INFO:tasks.rados.rados.0.smithi107.stdout:4537: left oid 6 (ObjNum 4536 snap 0 seq_num 4536) 2020-06-12T02:38:17.350 INFO:tasks.rados.rados.0.smithi107.stdout:4539: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:17.350 INFO:tasks.rados.rados.0.smithi107.stdout:4537: done (3 left) 2020-06-12T02:38:17.350 INFO:tasks.rados.rados.0.smithi107.stdout:4540: done (2 left) 2020-06-12T02:38:17.351 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:38:17.351 INFO:tasks.rados.rados.0.smithi107.stdout:4541: seq_num 4540 ranges {0=442368,442368=532480,974848=655360,1630208=16384} 2020-06-12T02:38:17.351 INFO:tasks.rados.rados.0.smithi107.stdout:4541: writing smithi10729407-11 from 0 to 442368 tid 1 2020-06-12T02:38:17.351 INFO:tasks.rados.rados.0.smithi107.stdout:4541: writing smithi10729407-11 from 442368 to 974848 tid 2 2020-06-12T02:38:17.351 INFO:tasks.rados.rados.0.smithi107.stdout:4541: writing smithi10729407-11 from 974848 to 1630208 tid 3 2020-06-12T02:38:17.351 INFO:tasks.rados.rados.0.smithi107.stdout:4541: writing smithi10729407-11 from 1630208 to 1646592 tid 4 2020-06-12T02:38:17.352 INFO:tasks.rados.rados.0.smithi107.stdout:4542: delete oid 18 current snap is 0 2020-06-12T02:38:17.352 INFO:tasks.rados.rados.0.smithi107.stdout:4539: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:17.353 INFO:tasks.rados.rados.0.smithi107.stdout:4539: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:17.353 INFO:tasks.rados.rados.0.smithi107.stdout:4539: finishing write tid 5 to smithi10729407-1 2020-06-12T02:38:17.353 INFO:tasks.rados.rados.0.smithi107.stdout:4541: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:17.354 INFO:tasks.rados.rados.0.smithi107.stdout:4539: finishing write tid 6 to smithi10729407-1 2020-06-12T02:38:17.354 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 842 (ObjNum 4538 snap 0 seq_num 4538) dirty exists 2020-06-12T02:38:17.354 INFO:tasks.rados.rados.0.smithi107.stdout:4539: left oid 1 (ObjNum 4538 snap 0 seq_num 4538) 2020-06-12T02:38:17.354 INFO:tasks.rados.rados.0.smithi107.stdout:4539: done (3 left) 2020-06-12T02:38:17.355 INFO:tasks.rados.rados.0.smithi107.stdout:4542: done (2 left) 2020-06-12T02:38:17.355 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:17.355 INFO:tasks.rados.rados.0.smithi107.stdout:4543: seq_num 4542 ranges {0=475136,475136=704512,1179648=499712} 2020-06-12T02:38:17.356 INFO:tasks.rados.rados.0.smithi107.stdout:4543: writing smithi10729407-19 from 0 to 475136 tid 1 2020-06-12T02:38:17.356 INFO:tasks.rados.rados.0.smithi107.stdout:4543: writing smithi10729407-19 from 475136 to 1179648 tid 2 2020-06-12T02:38:17.356 INFO:tasks.rados.rados.0.smithi107.stdout:4543: writing smithi10729407-19 from 1179648 to 1679360 tid 3 2020-06-12T02:38:17.357 INFO:tasks.rados.rados.0.smithi107.stdout:4544: delete oid 7 current snap is 0 2020-06-12T02:38:17.357 INFO:tasks.rados.rados.0.smithi107.stdout:4541: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:17.357 INFO:tasks.rados.rados.0.smithi107.stdout:4541: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:17.357 INFO:tasks.rados.rados.0.smithi107.stdout:4541: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:17.357 INFO:tasks.rados.rados.0.smithi107.stdout:4541: finishing write tid 6 to smithi10729407-11 2020-06-12T02:38:17.358 INFO:tasks.rados.rados.0.smithi107.stdout:4541: finishing write tid 7 to smithi10729407-11 2020-06-12T02:38:17.358 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 816 (ObjNum 4540 snap 0 seq_num 4540) dirty exists 2020-06-12T02:38:17.358 INFO:tasks.rados.rados.0.smithi107.stdout:4541: left oid 11 (ObjNum 4540 snap 0 seq_num 4540) 2020-06-12T02:38:17.358 INFO:tasks.rados.rados.0.smithi107.stdout:4541: done (3 left) 2020-06-12T02:38:17.358 INFO:tasks.rados.rados.0.smithi107.stdout:4544: done (2 left) 2020-06-12T02:38:17.359 INFO:tasks.rados.rados.0.smithi107.stdout:4545: delete oid 7 current snap is 0 2020-06-12T02:38:17.359 INFO:tasks.rados.rados.0.smithi107.stdout:4543: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:17.359 INFO:tasks.rados.rados.0.smithi107.stdout:4543: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:17.359 INFO:tasks.rados.rados.0.smithi107.stdout:4543: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:17.359 INFO:tasks.rados.rados.0.smithi107.stdout:4543: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:17.359 INFO:tasks.rados.rados.0.smithi107.stdout:4545: done (2 left) 2020-06-12T02:38:17.360 INFO:tasks.rados.rados.0.smithi107.stdout:4546: delete oid 14 current snap is 0 2020-06-12T02:38:17.360 INFO:tasks.rados.rados.0.smithi107.stdout:4543: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:17.362 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 776 (ObjNum 4542 snap 0 seq_num 4542) dirty exists 2020-06-12T02:38:17.362 INFO:tasks.rados.rados.0.smithi107.stdout:4543: left oid 19 (ObjNum 4542 snap 0 seq_num 4542) 2020-06-12T02:38:17.362 INFO:tasks.rados.rados.0.smithi107.stdout:4543: done (2 left) 2020-06-12T02:38:17.362 INFO:tasks.rados.rados.0.smithi107.stdout:4546: done (1 left) 2020-06-12T02:38:17.363 INFO:tasks.rados.rados.0.smithi107.stdout:4547: delete oid 18 current snap is 0 2020-06-12T02:38:17.363 INFO:tasks.rados.rados.0.smithi107.stdout:4547: done (1 left) 2020-06-12T02:38:17.363 INFO:tasks.rados.rados.0.smithi107.stdout:4548: delete oid 14 current snap is 0 2020-06-12T02:38:17.363 INFO:tasks.rados.rados.0.smithi107.stdout:4548: done (1 left) 2020-06-12T02:38:17.363 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:17.364 INFO:tasks.rados.rados.0.smithi107.stdout:4549: seq_num 4548 ranges {0=573440,573440=458752,1032192=614400,1646592=131072} 2020-06-12T02:38:17.364 INFO:tasks.rados.rados.0.smithi107.stdout:4549: writing smithi10729407-7 from 0 to 573440 tid 1 2020-06-12T02:38:17.364 INFO:tasks.rados.rados.0.smithi107.stdout:4549: writing smithi10729407-7 from 573440 to 1032192 tid 2 2020-06-12T02:38:17.364 INFO:tasks.rados.rados.0.smithi107.stdout:4549: writing smithi10729407-7 from 1032192 to 1646592 tid 3 2020-06-12T02:38:17.365 INFO:tasks.rados.rados.0.smithi107.stdout:4549: writing smithi10729407-7 from 1646592 to 1777664 tid 4 2020-06-12T02:38:17.365 INFO:tasks.rados.rados.0.smithi107.stdout:4550: delete oid 17 current snap is 0 2020-06-12T02:38:17.365 INFO:tasks.rados.rados.0.smithi107.stdout:4550: done (2 left) 2020-06-12T02:38:17.365 INFO:tasks.rados.rados.0.smithi107.stdout:4551: delete oid 2 current snap is 0 2020-06-12T02:38:17.365 INFO:tasks.rados.rados.0.smithi107.stdout:4549: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:17.366 INFO:tasks.rados.rados.0.smithi107.stdout:4549: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:17.366 INFO:tasks.rados.rados.0.smithi107.stdout:4551: done (2 left) 2020-06-12T02:38:17.366 INFO:tasks.rados.rados.0.smithi107.stdout:4552: delete oid 18 current snap is 0 2020-06-12T02:38:17.366 INFO:tasks.rados.rados.0.smithi107.stdout:4549: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:17.367 INFO:tasks.rados.rados.0.smithi107.stdout:4549: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:17.367 INFO:tasks.rados.rados.0.smithi107.stdout:4549: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:17.367 INFO:tasks.rados.rados.0.smithi107.stdout:4552: done (2 left) 2020-06-12T02:38:17.367 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:17.367 INFO:tasks.rados.rados.0.smithi107.stdout:4553: seq_num 4552 ranges {0=647168,647168=794624,1441792=409600} 2020-06-12T02:38:17.367 INFO:tasks.rados.rados.0.smithi107.stdout:4553: writing smithi10729407-16 from 0 to 647168 tid 1 2020-06-12T02:38:17.368 INFO:tasks.rados.rados.0.smithi107.stdout:4553: writing smithi10729407-16 from 647168 to 1441792 tid 2 2020-06-12T02:38:17.368 INFO:tasks.rados.rados.0.smithi107.stdout:4553: writing smithi10729407-16 from 1441792 to 1851392 tid 3 2020-06-12T02:38:17.368 INFO:tasks.rados.rados.0.smithi107.stdout:4554: delete oid 1 current snap is 0 2020-06-12T02:38:17.368 INFO:tasks.rados.rados.0.smithi107.stdout:4549: finishing write tid 7 to smithi10729407-7 2020-06-12T02:38:17.368 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 855 (ObjNum 4548 snap 0 seq_num 4548) dirty exists 2020-06-12T02:38:17.369 INFO:tasks.rados.rados.0.smithi107.stdout:4549: left oid 7 (ObjNum 4548 snap 0 seq_num 4548) 2020-06-12T02:38:17.369 INFO:tasks.rados.rados.0.smithi107.stdout:4553: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:17.369 INFO:tasks.rados.rados.0.smithi107.stdout:4549: done (3 left) 2020-06-12T02:38:17.369 INFO:tasks.rados.rados.0.smithi107.stdout:4554: done (2 left) 2020-06-12T02:38:17.369 INFO:tasks.rados.rados.0.smithi107.stdout:4555: delete oid 9 current snap is 0 2020-06-12T02:38:17.369 INFO:tasks.rados.rados.0.smithi107.stdout:4555: done (2 left) 2020-06-12T02:38:17.370 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:38:17.370 INFO:tasks.rados.rados.0.smithi107.stdout:4556: seq_num 4555 ranges {0=696320,696320=499712,1196032=704512} 2020-06-12T02:38:17.370 INFO:tasks.rados.rados.0.smithi107.stdout:4556: writing smithi10729407-20 from 0 to 696320 tid 1 2020-06-12T02:38:17.370 INFO:tasks.rados.rados.0.smithi107.stdout:4556: writing smithi10729407-20 from 696320 to 1196032 tid 2 2020-06-12T02:38:17.370 INFO:tasks.rados.rados.0.smithi107.stdout:4556: writing smithi10729407-20 from 1196032 to 1900544 tid 3 2020-06-12T02:38:17.371 INFO:tasks.rados.rados.0.smithi107.stdout:4557: delete oid 3 current snap is 0 2020-06-12T02:38:17.371 INFO:tasks.rados.rados.0.smithi107.stdout:4553: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:17.371 INFO:tasks.rados.rados.0.smithi107.stdout:4553: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:17.371 INFO:tasks.rados.rados.0.smithi107.stdout:4553: finishing write tid 5 to smithi10729407-16 2020-06-12T02:38:17.371 INFO:tasks.rados.rados.0.smithi107.stdout:4553: finishing write tid 6 to smithi10729407-16 2020-06-12T02:38:17.372 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 813 (ObjNum 4552 snap 0 seq_num 4552) dirty exists 2020-06-12T02:38:17.372 INFO:tasks.rados.rados.0.smithi107.stdout:4553: left oid 16 (ObjNum 4552 snap 0 seq_num 4552) 2020-06-12T02:38:17.372 INFO:tasks.rados.rados.0.smithi107.stdout:4556: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:17.372 INFO:tasks.rados.rados.0.smithi107.stdout:4553: done (3 left) 2020-06-12T02:38:17.372 INFO:tasks.rados.rados.0.smithi107.stdout:4557: done (2 left) 2020-06-12T02:38:17.373 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:38:17.373 INFO:tasks.rados.rados.0.smithi107.stdout:4558: seq_num 4557 ranges {0=729088,729088=663552,1392640=540672} 2020-06-12T02:38:17.373 INFO:tasks.rados.rados.0.smithi107.stdout:4558: writing smithi10729407-2 from 0 to 729088 tid 1 2020-06-12T02:38:17.373 INFO:tasks.rados.rados.0.smithi107.stdout:4558: writing smithi10729407-2 from 729088 to 1392640 tid 2 2020-06-12T02:38:17.373 INFO:tasks.rados.rados.0.smithi107.stdout:4558: writing smithi10729407-2 from 1392640 to 1933312 tid 3 2020-06-12T02:38:17.374 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:17.374 INFO:tasks.rados.rados.0.smithi107.stdout:4559: seq_num 4558 ranges {1679360=745472,2424832=548864,2973696=589824,3563520=65536} 2020-06-12T02:38:17.374 INFO:tasks.rados.rados.0.smithi107.stdout:4559: writing smithi10729407-19 from 1679360 to 2424832 tid 1 2020-06-12T02:38:17.374 INFO:tasks.rados.rados.0.smithi107.stdout:4559: writing smithi10729407-19 from 2424832 to 2973696 tid 2 2020-06-12T02:38:17.375 INFO:tasks.rados.rados.0.smithi107.stdout:4559: writing smithi10729407-19 from 2973696 to 3563520 tid 3 2020-06-12T02:38:17.375 INFO:tasks.rados.rados.0.smithi107.stdout:4559: writing smithi10729407-19 from 3563520 to 3629056 tid 4 2020-06-12T02:38:17.375 INFO:tasks.rados.rados.0.smithi107.stdout:4560: delete oid 9 current snap is 0 2020-06-12T02:38:17.375 INFO:tasks.rados.rados.0.smithi107.stdout:4556: finishing write tid 2 to smithi10729407-20 2020-06-12T02:38:17.375 INFO:tasks.rados.rados.0.smithi107.stdout:4556: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:17.376 INFO:tasks.rados.rados.0.smithi107.stdout:4556: finishing write tid 5 to smithi10729407-20 2020-06-12T02:38:17.376 INFO:tasks.rados.rados.0.smithi107.stdout:4556: finishing write tid 6 to smithi10729407-20 2020-06-12T02:38:17.376 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 349 (ObjNum 4555 snap 0 seq_num 4555) dirty exists 2020-06-12T02:38:17.376 INFO:tasks.rados.rados.0.smithi107.stdout:4556: left oid 20 (ObjNum 4555 snap 0 seq_num 4555) 2020-06-12T02:38:17.376 INFO:tasks.rados.rados.0.smithi107.stdout:4558: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:17.378 INFO:tasks.rados.rados.0.smithi107.stdout:4558: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:17.378 INFO:tasks.rados.rados.0.smithi107.stdout:4558: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:17.378 INFO:tasks.rados.rados.0.smithi107.stdout:4558: finishing write tid 5 to smithi10729407-2 2020-06-12T02:38:17.379 INFO:tasks.rados.rados.0.smithi107.stdout:4558: finishing write tid 6 to smithi10729407-2 2020-06-12T02:38:17.379 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 458 (ObjNum 4557 snap 0 seq_num 4557) dirty exists 2020-06-12T02:38:17.379 INFO:tasks.rados.rados.0.smithi107.stdout:4558: left oid 2 (ObjNum 4557 snap 0 seq_num 4557) 2020-06-12T02:38:17.379 INFO:tasks.rados.rados.0.smithi107.stdout:4556: done (4 left) 2020-06-12T02:38:17.379 INFO:tasks.rados.rados.0.smithi107.stdout:4558: done (3 left) 2020-06-12T02:38:17.380 INFO:tasks.rados.rados.0.smithi107.stdout:4560: done (2 left) 2020-06-12T02:38:17.380 INFO:tasks.rados.rados.0.smithi107.stdout:4561: delete oid 4 current snap is 0 2020-06-12T02:38:17.380 INFO:tasks.rados.rados.0.smithi107.stdout:4561: done (2 left) 2020-06-12T02:38:17.380 INFO:tasks.rados.rados.0.smithi107.stdout:4562: delete oid 2 current snap is 0 2020-06-12T02:38:17.380 INFO:tasks.rados.rados.0.smithi107.stdout:4559: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:17.380 INFO:tasks.rados.rados.0.smithi107.stdout:4559: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:17.381 INFO:tasks.rados.rados.0.smithi107.stdout:4559: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:17.381 INFO:tasks.rados.rados.0.smithi107.stdout:4562: done (2 left) 2020-06-12T02:38:17.381 INFO:tasks.rados.rados.0.smithi107.stdout:4563: delete oid 18 current snap is 0 2020-06-12T02:38:17.381 INFO:tasks.rados.rados.0.smithi107.stdout:4559: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:17.381 INFO:tasks.rados.rados.0.smithi107.stdout:4559: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:17.382 INFO:tasks.rados.rados.0.smithi107.stdout:4559: finishing write tid 7 to smithi10729407-19 2020-06-12T02:38:17.382 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 781 (ObjNum 4558 snap 0 seq_num 4558) dirty exists 2020-06-12T02:38:17.382 INFO:tasks.rados.rados.0.smithi107.stdout:4559: left oid 19 (ObjNum 4558 snap 0 seq_num 4558) 2020-06-12T02:38:17.382 INFO:tasks.rados.rados.0.smithi107.stdout:4559: done (2 left) 2020-06-12T02:38:17.382 INFO:tasks.rados.rados.0.smithi107.stdout:4563: done (1 left) 2020-06-12T02:38:17.383 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:17.383 INFO:tasks.rados.rados.0.smithi107.stdout:4564: seq_num 4563 ranges {0=425984,425984=417792,843776=729088,1572864=458752} 2020-06-12T02:38:17.383 INFO:tasks.rados.rados.0.smithi107.stdout:4564: writing smithi10729407-3 from 0 to 425984 tid 1 2020-06-12T02:38:17.383 INFO:tasks.rados.rados.0.smithi107.stdout:4564: writing smithi10729407-3 from 425984 to 843776 tid 2 2020-06-12T02:38:17.383 INFO:tasks.rados.rados.0.smithi107.stdout:4564: writing smithi10729407-3 from 843776 to 1572864 tid 3 2020-06-12T02:38:17.384 INFO:tasks.rados.rados.0.smithi107.stdout:4564: writing smithi10729407-3 from 1572864 to 2031616 tid 4 2020-06-12T02:38:17.384 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:17.384 INFO:tasks.rados.rados.0.smithi107.stdout:4565: seq_num 4564 ranges {0=442368,442368=704512,1146880=532480,1679360=368640} 2020-06-12T02:38:17.384 INFO:tasks.rados.rados.0.smithi107.stdout:4565: writing smithi10729407-9 from 0 to 442368 tid 1 2020-06-12T02:38:17.385 INFO:tasks.rados.rados.0.smithi107.stdout:4565: writing smithi10729407-9 from 442368 to 1146880 tid 2 2020-06-12T02:38:17.385 INFO:tasks.rados.rados.0.smithi107.stdout:4565: writing smithi10729407-9 from 1146880 to 1679360 tid 3 2020-06-12T02:38:17.385 INFO:tasks.rados.rados.0.smithi107.stdout:4565: writing smithi10729407-9 from 1679360 to 2048000 tid 4 2020-06-12T02:38:17.385 INFO:tasks.rados.rados.0.smithi107.stdout:4566: delete oid 20 current snap is 0 2020-06-12T02:38:17.385 INFO:tasks.rados.rados.0.smithi107.stdout:4564: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:17.386 INFO:tasks.rados.rados.0.smithi107.stdout:4564: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:17.386 INFO:tasks.rados.rados.0.smithi107.stdout:4564: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:17.386 INFO:tasks.rados.rados.0.smithi107.stdout:4564: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:17.386 INFO:tasks.rados.rados.0.smithi107.stdout:4564: finishing write tid 6 to smithi10729407-3 2020-06-12T02:38:17.386 INFO:tasks.rados.rados.0.smithi107.stdout:4565: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:17.387 INFO:tasks.rados.rados.0.smithi107.stdout:4564: finishing write tid 7 to smithi10729407-3 2020-06-12T02:38:17.387 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 819 (ObjNum 4563 snap 0 seq_num 4563) dirty exists 2020-06-12T02:38:17.387 INFO:tasks.rados.rados.0.smithi107.stdout:4564: left oid 3 (ObjNum 4563 snap 0 seq_num 4563) 2020-06-12T02:38:17.387 INFO:tasks.rados.rados.0.smithi107.stdout:4564: done (3 left) 2020-06-12T02:38:17.387 INFO:tasks.rados.rados.0.smithi107.stdout:4566: done (2 left) 2020-06-12T02:38:17.387 INFO:tasks.rados.rados.0.smithi107.stdout:4567: delete oid 15 current snap is 0 2020-06-12T02:38:17.388 INFO:tasks.rados.rados.0.smithi107.stdout:4565: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:17.388 INFO:tasks.rados.rados.0.smithi107.stdout:4567: done (2 left) 2020-06-12T02:38:17.388 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:17.388 INFO:tasks.rados.rados.0.smithi107.stdout:4568: seq_num 4567 ranges {0=491520,491520=761856,1253376=712704,1966080=131072} 2020-06-12T02:38:17.388 INFO:tasks.rados.rados.0.smithi107.stdout:4568: writing smithi10729407-18 from 0 to 491520 tid 1 2020-06-12T02:38:17.389 INFO:tasks.rados.rados.0.smithi107.stdout:4568: writing smithi10729407-18 from 491520 to 1253376 tid 2 2020-06-12T02:38:17.389 INFO:tasks.rados.rados.0.smithi107.stdout:4568: writing smithi10729407-18 from 1253376 to 1966080 tid 3 2020-06-12T02:38:17.389 INFO:tasks.rados.rados.0.smithi107.stdout:4568: writing smithi10729407-18 from 1966080 to 2097152 tid 4 2020-06-12T02:38:17.389 INFO:tasks.rados.rados.0.smithi107.stdout:4569: delete oid 20 current snap is 0 2020-06-12T02:38:17.390 INFO:tasks.rados.rados.0.smithi107.stdout:4565: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:17.390 INFO:tasks.rados.rados.0.smithi107.stdout:4565: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:17.390 INFO:tasks.rados.rados.0.smithi107.stdout:4565: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:17.390 INFO:tasks.rados.rados.0.smithi107.stdout:4565: finishing write tid 7 to smithi10729407-9 2020-06-12T02:38:17.390 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 814 (ObjNum 4564 snap 0 seq_num 4564) dirty exists 2020-06-12T02:38:17.391 INFO:tasks.rados.rados.0.smithi107.stdout:4565: left oid 9 (ObjNum 4564 snap 0 seq_num 4564) 2020-06-12T02:38:17.391 INFO:tasks.rados.rados.0.smithi107.stdout:4568: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:17.391 INFO:tasks.rados.rados.0.smithi107.stdout:4565: done (3 left) 2020-06-12T02:38:17.391 INFO:tasks.rados.rados.0.smithi107.stdout:4569: done (2 left) 2020-06-12T02:38:17.391 INFO:tasks.rados.rados.0.smithi107.stdout:4570: delete oid 15 current snap is 0 2020-06-12T02:38:17.392 INFO:tasks.rados.rados.0.smithi107.stdout:4570: done (2 left) 2020-06-12T02:38:17.392 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:17.392 INFO:tasks.rados.rados.0.smithi107.stdout:4571: seq_num 4570 ranges {3629056=540672,4169728=458752,4628480=532480,5160960=565248,5726208=49152} 2020-06-12T02:38:17.392 INFO:tasks.rados.rados.0.smithi107.stdout:4571: writing smithi10729407-19 from 3629056 to 4169728 tid 1 2020-06-12T02:38:17.392 INFO:tasks.rados.rados.0.smithi107.stdout:4571: writing smithi10729407-19 from 4169728 to 4628480 tid 2 2020-06-12T02:38:17.392 INFO:tasks.rados.rados.0.smithi107.stdout:4571: writing smithi10729407-19 from 4628480 to 5160960 tid 3 2020-06-12T02:38:17.393 INFO:tasks.rados.rados.0.smithi107.stdout:4571: writing smithi10729407-19 from 5160960 to 5726208 tid 4 2020-06-12T02:38:17.393 INFO:tasks.rados.rados.0.smithi107.stdout:4571: writing smithi10729407-19 from 5726208 to 5775360 tid 5 2020-06-12T02:38:17.393 INFO:tasks.rados.rados.0.smithi107.stdout:4572: delete oid 5 current snap is 0 2020-06-12T02:38:17.393 INFO:tasks.rados.rados.0.smithi107.stdout:4568: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:17.393 INFO:tasks.rados.rados.0.smithi107.stdout:4568: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:17.394 INFO:tasks.rados.rados.0.smithi107.stdout:4568: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:17.394 INFO:tasks.rados.rados.0.smithi107.stdout:4568: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:17.394 INFO:tasks.rados.rados.0.smithi107.stdout:4568: finishing write tid 7 to smithi10729407-18 2020-06-12T02:38:17.394 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 819 (ObjNum 4567 snap 0 seq_num 4567) dirty exists 2020-06-12T02:38:17.394 INFO:tasks.rados.rados.0.smithi107.stdout:4568: left oid 18 (ObjNum 4567 snap 0 seq_num 4567) 2020-06-12T02:38:17.395 INFO:tasks.rados.rados.0.smithi107.stdout:4571: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:17.395 INFO:tasks.rados.rados.0.smithi107.stdout:4568: done (3 left) 2020-06-12T02:38:17.395 INFO:tasks.rados.rados.0.smithi107.stdout:4572: done (2 left) 2020-06-12T02:38:17.395 INFO:tasks.rados.rados.0.smithi107.stdout:4573: delete oid 1 current snap is 0 2020-06-12T02:38:17.395 INFO:tasks.rados.rados.0.smithi107.stdout:4573: done (2 left) 2020-06-12T02:38:17.396 INFO:tasks.rados.rados.0.smithi107.stdout:4574: delete oid 18 current snap is 0 2020-06-12T02:38:17.396 INFO:tasks.rados.rados.0.smithi107.stdout:4574: done (2 left) 2020-06-12T02:38:17.396 INFO:tasks.rados.rados.0.smithi107.stdout:4575: delete oid 17 current snap is 0 2020-06-12T02:38:17.396 INFO:tasks.rados.rados.0.smithi107.stdout:4571: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:17.396 INFO:tasks.rados.rados.0.smithi107.stdout:4575: done (2 left) 2020-06-12T02:38:17.397 INFO:tasks.rados.rados.0.smithi107.stdout:4576: delete oid 13 current snap is 0 2020-06-12T02:38:17.397 INFO:tasks.rados.rados.0.smithi107.stdout:4571: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:17.397 INFO:tasks.rados.rados.0.smithi107.stdout:4571: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:17.397 INFO:tasks.rados.rados.0.smithi107.stdout:4571: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:17.397 INFO:tasks.rados.rados.0.smithi107.stdout:4571: finishing write tid 7 to smithi10729407-19 2020-06-12T02:38:17.397 INFO:tasks.rados.rados.0.smithi107.stdout:4576: done (2 left) 2020-06-12T02:38:17.398 INFO:tasks.rados.rados.0.smithi107.stdout:4577: delete oid 7 current snap is 0 2020-06-12T02:38:17.398 INFO:tasks.rados.rados.0.smithi107.stdout:4571: finishing write tid 8 to smithi10729407-19 2020-06-12T02:38:17.398 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 787 (ObjNum 4570 snap 0 seq_num 4570) dirty exists 2020-06-12T02:38:17.398 INFO:tasks.rados.rados.0.smithi107.stdout:4571: left oid 19 (ObjNum 4570 snap 0 seq_num 4570) 2020-06-12T02:38:17.398 INFO:tasks.rados.rados.0.smithi107.stdout:4571: done (2 left) 2020-06-12T02:38:17.399 INFO:tasks.rados.rados.0.smithi107.stdout:4577: done (1 left) 2020-06-12T02:38:17.399 INFO:tasks.rados.rados.0.smithi107.stdout:4578: delete oid 4 current snap is 0 2020-06-12T02:38:17.399 INFO:tasks.rados.rados.0.smithi107.stdout:4578: done (1 left) 2020-06-12T02:38:17.399 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:38:17.399 INFO:tasks.rados.rados.0.smithi107.stdout:4579: seq_num 4578 ranges {1646592=679936,2326528=786432,3112960=491520,3604480=327680} 2020-06-12T02:38:17.400 INFO:tasks.rados.rados.0.smithi107.stdout:4579: writing smithi10729407-11 from 1646592 to 2326528 tid 1 2020-06-12T02:38:17.401 INFO:tasks.rados.rados.0.smithi107.stdout:4579: writing smithi10729407-11 from 2326528 to 3112960 tid 2 2020-06-12T02:38:17.401 INFO:tasks.rados.rados.0.smithi107.stdout:4579: writing smithi10729407-11 from 3112960 to 3604480 tid 3 2020-06-12T02:38:17.401 INFO:tasks.rados.rados.0.smithi107.stdout:4579: writing smithi10729407-11 from 3604480 to 3932160 tid 4 2020-06-12T02:38:17.401 INFO:tasks.rados.rados.0.smithi107.stdout:4580: delete oid 16 current snap is 0 2020-06-12T02:38:17.402 INFO:tasks.rados.rados.0.smithi107.stdout:4579: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:17.402 INFO:tasks.rados.rados.0.smithi107.stdout:4580: done (2 left) 2020-06-12T02:38:17.402 INFO:tasks.rados.rados.0.smithi107.stdout:4581: delete oid 12 current snap is 0 2020-06-12T02:38:17.402 INFO:tasks.rados.rados.0.smithi107.stdout:4581: done (2 left) 2020-06-12T02:38:17.402 INFO:tasks.rados.rados.0.smithi107.stdout:4582: delete oid 20 current snap is 0 2020-06-12T02:38:17.403 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd unset noscrub 2020-06-12T02:38:17.407 INFO:tasks.rados.rados.0.smithi107.stdout:4579: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:17.407 INFO:tasks.rados.rados.0.smithi107.stdout:4579: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:17.407 INFO:tasks.rados.rados.0.smithi107.stdout:4582: done (2 left) 2020-06-12T02:38:17.408 INFO:tasks.rados.rados.0.smithi107.stdout:4583: delete oid 7 current snap is 0 2020-06-12T02:38:17.408 INFO:tasks.rados.rados.0.smithi107.stdout:4579: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:17.408 INFO:tasks.rados.rados.0.smithi107.stdout:4579: finishing write tid 6 to smithi10729407-11 2020-06-12T02:38:17.408 INFO:tasks.rados.rados.0.smithi107.stdout:4579: finishing write tid 7 to smithi10729407-11 2020-06-12T02:38:17.408 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 823 (ObjNum 4578 snap 0 seq_num 4578) dirty exists 2020-06-12T02:38:17.408 INFO:tasks.rados.rados.0.smithi107.stdout:4579: left oid 11 (ObjNum 4578 snap 0 seq_num 4578) 2020-06-12T02:38:17.409 INFO:tasks.rados.rados.0.smithi107.stdout:4579: done (2 left) 2020-06-12T02:38:17.409 INFO:tasks.rados.rados.0.smithi107.stdout:4583: done (1 left) 2020-06-12T02:38:17.409 INFO:tasks.rados.rados.0.smithi107.stdout:4584: delete oid 9 current snap is 0 2020-06-12T02:38:17.409 INFO:tasks.rados.rados.0.smithi107.stdout:4584: done (1 left) 2020-06-12T02:38:17.409 INFO:tasks.rados.rados.0.smithi107.stdout:4585: delete oid 9 current snap is 0 2020-06-12T02:38:17.410 INFO:tasks.rados.rados.0.smithi107.stdout:4585: done (1 left) 2020-06-12T02:38:17.410 INFO:tasks.rados.rados.0.smithi107.stdout:4586: delete oid 3 current snap is 0 2020-06-12T02:38:17.410 INFO:tasks.rados.rados.0.smithi107.stdout:4586: done (1 left) 2020-06-12T02:38:17.410 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:17.410 INFO:tasks.rados.rados.0.smithi107.stdout:4587: seq_num 4586 ranges {0=8192} 2020-06-12T02:38:17.411 INFO:tasks.rados.rados.0.smithi107.stdout:4587: writing smithi10729407-4 from 0 to 8192 tid 1 2020-06-12T02:38:17.411 INFO:tasks.rados.rados.0.smithi107.stdout:4588: delete oid 15 current snap is 0 2020-06-12T02:38:17.411 INFO:tasks.rados.rados.0.smithi107.stdout:4588: done (2 left) 2020-06-12T02:38:17.411 INFO:tasks.rados.rados.0.smithi107.stdout:4589: delete oid 13 current snap is 0 2020-06-12T02:38:17.411 INFO:tasks.rados.rados.0.smithi107.stdout:4587: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:17.411 INFO:tasks.rados.rados.0.smithi107.stdout:4589: done (2 left) 2020-06-12T02:38:17.412 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:17.412 INFO:tasks.rados.rados.0.smithi107.stdout:4590: seq_num 4589 ranges {5775360=57344} 2020-06-12T02:38:17.412 INFO:tasks.rados.rados.0.smithi107.stdout:4590: writing smithi10729407-19 from 5775360 to 5832704 tid 1 2020-06-12T02:38:17.412 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:17.412 INFO:tasks.rados.rados.0.smithi107.stdout:4591: seq_num 4590 ranges {0=81920} 2020-06-12T02:38:17.413 INFO:tasks.rados.rados.0.smithi107.stdout:4591: writing smithi10729407-14 from 0 to 81920 tid 1 2020-06-12T02:38:17.413 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:17.413 INFO:tasks.rados.rados.0.smithi107.stdout:4592: seq_num 4591 ranges {0=98304} 2020-06-12T02:38:17.413 INFO:tasks.rados.rados.0.smithi107.stdout:4592: writing smithi10729407-17 from 0 to 98304 tid 1 2020-06-12T02:38:17.413 INFO:tasks.rados.rados.0.smithi107.stdout:4593: delete oid 9 current snap is 0 2020-06-12T02:38:17.413 INFO:tasks.rados.rados.0.smithi107.stdout:4587: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:17.414 INFO:tasks.rados.rados.0.smithi107.stdout:4587: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:17.414 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 848 (ObjNum 4586 snap 0 seq_num 4586) dirty exists 2020-06-12T02:38:17.414 INFO:tasks.rados.rados.0.smithi107.stdout:4587: left oid 4 (ObjNum 4586 snap 0 seq_num 4586) 2020-06-12T02:38:17.414 INFO:tasks.rados.rados.0.smithi107.stdout:4590: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:17.414 INFO:tasks.rados.rados.0.smithi107.stdout:4591: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:17.415 INFO:tasks.rados.rados.0.smithi107.stdout:4587: done (5 left) 2020-06-12T02:38:17.415 INFO:tasks.rados.rados.0.smithi107.stdout:4593: done (4 left) 2020-06-12T02:38:17.415 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:17.415 INFO:tasks.rados.rados.0.smithi107.stdout:4594: seq_num 4593 ranges {3932160=131072} 2020-06-12T02:38:17.426 INFO:tasks.rados.rados.0.smithi107.stdout:4594: writing smithi10729407-11 from 3932160 to 4063232 tid 1 2020-06-12T02:38:17.427 INFO:tasks.rados.rados.0.smithi107.stdout:4595: delete oid 3 current snap is 0 2020-06-12T02:38:17.427 INFO:tasks.rados.rados.0.smithi107.stdout:4592: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:17.427 INFO:tasks.rados.rados.0.smithi107.stdout:4591: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:17.427 INFO:tasks.rados.rados.0.smithi107.stdout:4590: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:17.427 INFO:tasks.rados.rados.0.smithi107.stdout:4591: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:17.428 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 825 (ObjNum 4590 snap 0 seq_num 4590) dirty exists 2020-06-12T02:38:17.428 INFO:tasks.rados.rados.0.smithi107.stdout:4591: left oid 14 (ObjNum 4590 snap 0 seq_num 4590) 2020-06-12T02:38:17.428 INFO:tasks.rados.rados.0.smithi107.stdout:4592: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:17.428 INFO:tasks.rados.rados.0.smithi107.stdout:4590: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:17.428 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 790 (ObjNum 4589 snap 0 seq_num 4589) dirty exists 2020-06-12T02:38:17.429 INFO:tasks.rados.rados.0.smithi107.stdout:4590: left oid 19 (ObjNum 4589 snap 0 seq_num 4589) 2020-06-12T02:38:17.429 INFO:tasks.rados.rados.0.smithi107.stdout:4592: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:17.429 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 850 (ObjNum 4591 snap 0 seq_num 4591) dirty exists 2020-06-12T02:38:17.429 INFO:tasks.rados.rados.0.smithi107.stdout:4592: left oid 17 (ObjNum 4591 snap 0 seq_num 4591) 2020-06-12T02:38:17.429 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:17.422+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:17.435 INFO:tasks.rados.rados.0.smithi107.stdout:4590: done (5 left) 2020-06-12T02:38:17.436 INFO:tasks.rados.rados.0.smithi107.stdout:4591: done (4 left) 2020-06-12T02:38:17.436 INFO:tasks.rados.rados.0.smithi107.stdout:4592: done (3 left) 2020-06-12T02:38:17.436 INFO:tasks.rados.rados.0.smithi107.stdout:4595: done (2 left) 2020-06-12T02:38:17.437 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:17.437 INFO:tasks.rados.rados.0.smithi107.stdout:4596: seq_num 4595 ranges {5832704=163840} 2020-06-12T02:38:17.461 INFO:tasks.rados.rados.0.smithi107.stdout:4596: writing smithi10729407-19 from 5832704 to 5996544 tid 1 2020-06-12T02:38:17.462 INFO:tasks.rados.rados.0.smithi107.stdout:4597: delete oid 9 current snap is 0 2020-06-12T02:38:17.462 INFO:tasks.rados.rados.0.smithi107.stdout:4594: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:17.462 INFO:tasks.rados.rados.0.smithi107.stdout:4594: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:17.462 INFO:tasks.rados.rados.0.smithi107.stdout:4594: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:17.462 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 827 (ObjNum 4593 snap 0 seq_num 4593) dirty exists 2020-06-12T02:38:17.463 INFO:tasks.rados.rados.0.smithi107.stdout:4594: left oid 11 (ObjNum 4593 snap 0 seq_num 4593) 2020-06-12T02:38:17.463 INFO:tasks.rados.rados.0.smithi107.stdout:4594: done (3 left) 2020-06-12T02:38:17.464 INFO:tasks.rados.rados.0.smithi107.stdout:4597: done (2 left) 2020-06-12T02:38:17.464 INFO:tasks.rados.rados.0.smithi107.stdout:4598: delete oid 15 current snap is 0 2020-06-12T02:38:17.464 INFO:tasks.rados.rados.0.smithi107.stdout:4596: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:17.466 INFO:tasks.rados.rados.0.smithi107.stdout:4598: done (2 left) 2020-06-12T02:38:17.466 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:17.467 INFO:tasks.rados.rados.0.smithi107.stdout:4599: seq_num 4598 ranges {0=212992} 2020-06-12T02:38:17.467 INFO:tasks.rados.rados.0.smithi107.stdout:4599: writing smithi10729407-1 from 0 to 212992 tid 1 2020-06-12T02:38:17.468 INFO:tasks.rados.rados.0.smithi107.stdout:4600: delete oid 7 current snap is 0 2020-06-12T02:38:17.468 INFO:tasks.rados.rados.0.smithi107.stdout:4596: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:17.469 INFO:tasks.rados.rados.0.smithi107.stdout:4596: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:17.469 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 792 (ObjNum 4595 snap 0 seq_num 4595) dirty exists 2020-06-12T02:38:17.469 INFO:tasks.rados.rados.0.smithi107.stdout:4596: left oid 19 (ObjNum 4595 snap 0 seq_num 4595) 2020-06-12T02:38:17.470 INFO:tasks.rados.rados.0.smithi107.stdout:4596: done (3 left) 2020-06-12T02:38:17.470 INFO:tasks.rados.rados.0.smithi107.stdout:4600: done (2 left) 2020-06-12T02:38:17.471 INFO:tasks.rados.rados.0.smithi107.stdout:4601: delete oid 13 current snap is 0 2020-06-12T02:38:17.473 INFO:tasks.rados.rados.0.smithi107.stdout:4599: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:17.478 INFO:tasks.rados.rados.0.smithi107.stdout:4601: done (2 left) 2020-06-12T02:38:17.479 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:17.479 INFO:tasks.rados.rados.0.smithi107.stdout:4602: seq_num 4601 ranges {0=262144} 2020-06-12T02:38:17.480 INFO:tasks.rados.rados.0.smithi107.stdout:4602: writing smithi10729407-7 from 0 to 262144 tid 1 2020-06-12T02:38:17.480 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:17.480 INFO:tasks.rados.rados.0.smithi107.stdout:4603: seq_num 4602 ranges {0=278528} 2020-06-12T02:38:17.482 INFO:tasks.rados.rados.0.smithi107.stdout:4603: writing smithi10729407-15 from 0 to 278528 tid 1 2020-06-12T02:38:17.482 INFO:tasks.rados.rados.0.smithi107.stdout:4604: delete oid 4 current snap is 0 2020-06-12T02:38:17.482 INFO:tasks.rados.rados.0.smithi107.stdout:4599: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:17.482 INFO:tasks.rados.rados.0.smithi107.stdout:4599: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:17.482 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 846 (ObjNum 4598 snap 0 seq_num 4598) dirty exists 2020-06-12T02:38:17.483 INFO:tasks.rados.rados.0.smithi107.stdout:4599: left oid 1 (ObjNum 4598 snap 0 seq_num 4598) 2020-06-12T02:38:17.487 INFO:tasks.rados.rados.0.smithi107.stdout:4599: done (4 left) 2020-06-12T02:38:17.488 INFO:tasks.rados.rados.0.smithi107.stdout:4604: done (3 left) 2020-06-12T02:38:17.488 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:17.488 INFO:tasks.rados.rados.0.smithi107.stdout:4605: seq_num 4604 ranges {0=311296} 2020-06-12T02:38:17.489 INFO:tasks.rados.rados.0.smithi107.stdout:4605: writing smithi10729407-9 from 0 to 311296 tid 1 2020-06-12T02:38:17.489 INFO:tasks.rados.rados.0.smithi107.stdout:4606: delete oid 13 current snap is 0 2020-06-12T02:38:17.490 INFO:tasks.rados.rados.0.smithi107.stdout:4602: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:17.490 INFO:tasks.rados.rados.0.smithi107.stdout:4603: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:17.490 INFO:tasks.rados.rados.0.smithi107.stdout:4603: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:17.490 INFO:tasks.rados.rados.0.smithi107.stdout:4602: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:17.491 INFO:tasks.rados.rados.0.smithi107.stdout:4603: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:17.491 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 777 (ObjNum 4602 snap 0 seq_num 4602) dirty exists 2020-06-12T02:38:17.491 INFO:tasks.rados.rados.0.smithi107.stdout:4603: left oid 15 (ObjNum 4602 snap 0 seq_num 4602) 2020-06-12T02:38:17.492 INFO:tasks.rados.rados.0.smithi107.stdout:4602: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:17.492 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 863 (ObjNum 4601 snap 0 seq_num 4601) dirty exists 2020-06-12T02:38:17.492 INFO:tasks.rados.rados.0.smithi107.stdout:4602: left oid 7 (ObjNum 4601 snap 0 seq_num 4601) 2020-06-12T02:38:17.496 INFO:tasks.rados.rados.0.smithi107.stdout:4605: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:17.496 INFO:tasks.rados.rados.0.smithi107.stdout:4602: done (4 left) 2020-06-12T02:38:17.496 INFO:tasks.rados.rados.0.smithi107.stdout:4603: done (3 left) 2020-06-12T02:38:17.497 INFO:tasks.rados.rados.0.smithi107.stdout:4606: done (2 left) 2020-06-12T02:38:17.497 INFO:tasks.rados.rados.0.smithi107.stdout:4607: delete oid 5 current snap is 0 2020-06-12T02:38:17.498 INFO:tasks.rados.rados.0.smithi107.stdout:4607: done (2 left) 2020-06-12T02:38:17.499 INFO:tasks.rados.rados.0.smithi107.stdout:4608: delete oid 16 current snap is 0 2020-06-12T02:38:17.501 INFO:tasks.rados.rados.0.smithi107.stdout:4608: done (2 left) 2020-06-12T02:38:17.501 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:17.502 INFO:tasks.rados.rados.0.smithi107.stdout:4609: seq_num 4608 ranges {0=376832} 2020-06-12T02:38:17.504 INFO:tasks.rados.rados.0.smithi107.stdout:4609: writing smithi10729407-4 from 0 to 376832 tid 1 2020-06-12T02:38:17.504 INFO:tasks.rados.rados.0.smithi107.stdout:4610: delete oid 15 current snap is 0 2020-06-12T02:38:17.504 INFO:tasks.rados.rados.0.smithi107.stdout:4605: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:17.505 INFO:tasks.rados.rados.0.smithi107.stdout:4605: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:17.505 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 826 (ObjNum 4604 snap 0 seq_num 4604) dirty exists 2020-06-12T02:38:17.505 INFO:tasks.rados.rados.0.smithi107.stdout:4605: left oid 9 (ObjNum 4604 snap 0 seq_num 4604) 2020-06-12T02:38:17.506 INFO:tasks.rados.rados.0.smithi107.stdout:4605: done (3 left) 2020-06-12T02:38:17.506 INFO:tasks.rados.rados.0.smithi107.stdout:4610: done (2 left) 2020-06-12T02:38:17.507 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:17.507 INFO:tasks.rados.rados.0.smithi107.stdout:4611: seq_num 4610 ranges {5996544=417792} 2020-06-12T02:38:17.523 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:17.520+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:17.538 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:17.538 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:38:17.540 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2020-06-12T02:38:17.540 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:17.542 INFO:tasks.rados.rados.0.smithi107.stdout:4611: writing smithi10729407-19 from 5996544 to 6414336 tid 1 2020-06-12T02:38:17.542 INFO:tasks.rados.rados.0.smithi107.stdout:4612: delete oid 2 current snap is 0 2020-06-12T02:38:17.542 INFO:tasks.rados.rados.0.smithi107.stdout:4609: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:17.542 INFO:tasks.rados.rados.0.smithi107.stdout:4609: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:17.542 INFO:tasks.rados.rados.0.smithi107.stdout:4609: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:17.543 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 853 (ObjNum 4608 snap 0 seq_num 4608) dirty exists 2020-06-12T02:38:17.543 INFO:tasks.rados.rados.0.smithi107.stdout:4609: left oid 4 (ObjNum 4608 snap 0 seq_num 4608) 2020-06-12T02:38:17.546 INFO:tasks.rados.rados.0.smithi107.stdout:4609: done (3 left) 2020-06-12T02:38:17.546 INFO:tasks.rados.rados.0.smithi107.stdout:4612: done (2 left) 2020-06-12T02:38:17.547 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:17.547 INFO:tasks.rados.rados.0.smithi107.stdout:4613: seq_num 4612 ranges {0=450560} 2020-06-12T02:38:17.547 INFO:tasks.rados.rados.0.smithi107.stdout:4613: writing smithi10729407-12 from 0 to 450560 tid 1 2020-06-12T02:38:17.547 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:17.547 INFO:tasks.rados.rados.0.smithi107.stdout:4614: seq_num 4613 ranges {98304=466944} 2020-06-12T02:38:17.548 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-06-12T02:38:17.549 INFO:tasks.rados.rados.0.smithi107.stdout:4614: writing smithi10729407-17 from 98304 to 565248 tid 1 2020-06-12T02:38:17.550 INFO:tasks.rados.rados.0.smithi107.stdout:4615: delete oid 2 current snap is 0 2020-06-12T02:38:17.550 INFO:tasks.rados.rados.0.smithi107.stdout:4611: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:17.551 INFO:tasks.rados.rados.0.smithi107.stdout:4611: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:17.551 INFO:tasks.rados.rados.0.smithi107.stdout:4613: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:17.552 INFO:tasks.rados.rados.0.smithi107.stdout:4611: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:17.552 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 794 (ObjNum 4610 snap 0 seq_num 4610) dirty exists 2020-06-12T02:38:17.553 INFO:tasks.rados.rados.0.smithi107.stdout:4611: left oid 19 (ObjNum 4610 snap 0 seq_num 4610) 2020-06-12T02:38:17.554 INFO:tasks.rados.rados.0.smithi107.stdout:4611: done (4 left) 2020-06-12T02:38:17.554 INFO:tasks.rados.rados.0.smithi107.stdout:4615: done (3 left) 2020-06-12T02:38:17.554 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:17.554 INFO:tasks.rados.rados.0.smithi107.stdout:4616: seq_num 4615 ranges {376832=499712} 2020-06-12T02:38:17.560 INFO:tasks.rados.rados.0.smithi107.stdout:4616: writing smithi10729407-4 from 376832 to 876544 tid 1 2020-06-12T02:38:17.560 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:17.560 INFO:tasks.rados.rados.0.smithi107.stdout:4617: seq_num 4616 ranges {0=516096} 2020-06-12T02:38:17.562 INFO:tasks.rados.rados.0.smithi107.stdout:4617: writing smithi10729407-5 from 0 to 516096 tid 1 2020-06-12T02:38:17.562 INFO:tasks.rados.rados.0.smithi107.stdout:4618: delete oid 1 current snap is 0 2020-06-12T02:38:17.563 INFO:tasks.rados.rados.0.smithi107.stdout:4613: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:17.563 INFO:tasks.rados.rados.0.smithi107.stdout:4614: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:17.563 INFO:tasks.rados.rados.0.smithi107.stdout:4613: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:17.563 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 796 (ObjNum 4612 snap 0 seq_num 4612) dirty exists 2020-06-12T02:38:17.563 INFO:tasks.rados.rados.0.smithi107.stdout:4613: left oid 12 (ObjNum 4612 snap 0 seq_num 4612) 2020-06-12T02:38:17.565 INFO:tasks.rados.rados.0.smithi107.stdout:4614: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:17.567 INFO:tasks.rados.rados.0.smithi107.stdout:4614: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:17.567 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 855 (ObjNum 4613 snap 0 seq_num 4613) dirty exists 2020-06-12T02:38:17.567 INFO:tasks.rados.rados.0.smithi107.stdout:4614: left oid 17 (ObjNum 4613 snap 0 seq_num 4613) 2020-06-12T02:38:17.568 INFO:tasks.rados.rados.0.smithi107.stdout:4616: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:17.568 INFO:tasks.rados.rados.0.smithi107.stdout:4616: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:17.570 INFO:tasks.rados.rados.0.smithi107.stdout:4617: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:17.570 INFO:tasks.rados.rados.0.smithi107.stdout:4613: done (5 left) 2020-06-12T02:38:17.570 INFO:tasks.rados.rados.0.smithi107.stdout:4614: done (4 left) 2020-06-12T02:38:17.571 INFO:tasks.rados.rados.0.smithi107.stdout:4618: done (3 left) 2020-06-12T02:38:17.571 INFO:tasks.rados.rados.0.smithi107.stdout:4619: delete oid 7 current snap is 0 2020-06-12T02:38:17.571 INFO:tasks.rados.rados.0.smithi107.stdout:4616: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:17.571 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 857 (ObjNum 4615 snap 0 seq_num 4615) dirty exists 2020-06-12T02:38:17.571 INFO:tasks.rados.rados.0.smithi107.stdout:4616: left oid 4 (ObjNum 4615 snap 0 seq_num 4615) 2020-06-12T02:38:17.572 INFO:tasks.rados.rados.0.smithi107.stdout:4617: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:17.572 INFO:tasks.rados.rados.0.smithi107.stdout:4617: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:17.573 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 459 (ObjNum 4616 snap 0 seq_num 4616) dirty exists 2020-06-12T02:38:17.573 INFO:tasks.rados.rados.0.smithi107.stdout:4617: left oid 5 (ObjNum 4616 snap 0 seq_num 4616) 2020-06-12T02:38:17.573 INFO:tasks.rados.rados.0.smithi107.stdout:4616: done (3 left) 2020-06-12T02:38:17.573 INFO:tasks.rados.rados.0.smithi107.stdout:4617: done (2 left) 2020-06-12T02:38:17.574 INFO:tasks.rados.rados.0.smithi107.stdout:4619: done (1 left) 2020-06-12T02:38:17.574 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:17.574 INFO:tasks.rados.rados.0.smithi107.stdout:4620: seq_num 4619 ranges {311296=565248} 2020-06-12T02:38:17.578 INFO:tasks.rados.rados.0.smithi107.stdout:4620: writing smithi10729407-9 from 311296 to 876544 tid 1 2020-06-12T02:38:17.578 INFO:tasks.rados.rados.0.smithi107.stdout:4621: delete oid 3 current snap is 0 2020-06-12T02:38:17.583 INFO:tasks.rados.rados.0.smithi107.stdout:4621: done (2 left) 2020-06-12T02:38:17.584 INFO:tasks.rados.rados.0.smithi107.stdout:4622: delete oid 2 current snap is 0 2020-06-12T02:38:17.584 INFO:tasks.rados.rados.0.smithi107.stdout:4620: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:17.584 INFO:tasks.rados.rados.0.smithi107.stdout:4620: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:17.585 INFO:tasks.rados.rados.0.smithi107.stdout:4620: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:17.585 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 828 (ObjNum 4619 snap 0 seq_num 4619) dirty exists 2020-06-12T02:38:17.585 INFO:tasks.rados.rados.0.smithi107.stdout:4620: left oid 9 (ObjNum 4619 snap 0 seq_num 4619) 2020-06-12T02:38:17.585 INFO:tasks.rados.rados.0.smithi107.stdout:4620: done (2 left) 2020-06-12T02:38:17.586 INFO:tasks.rados.rados.0.smithi107.stdout:4622: done (1 left) 2020-06-12T02:38:17.586 INFO:tasks.rados.rados.0.smithi107.stdout:4623: delete oid 6 current snap is 0 2020-06-12T02:38:17.587 INFO:tasks.rados.rados.0.smithi107.stdout:4623: done (1 left) 2020-06-12T02:38:17.587 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:38:17.587 INFO:tasks.rados.rados.0.smithi107.stdout:4624: seq_num 4623 ranges {0=630784} 2020-06-12T02:38:17.590 INFO:tasks.rados.rados.0.smithi107.stdout:4624: writing smithi10729407-13 from 0 to 630784 tid 1 2020-06-12T02:38:17.591 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:17.591 INFO:tasks.rados.rados.0.smithi107.stdout:4625: seq_num 4624 ranges {0=647168} 2020-06-12T02:38:17.594 INFO:tasks.rados.rados.0.smithi107.stdout:4625: writing smithi10729407-8 from 0 to 647168 tid 1 2020-06-12T02:38:17.594 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:17.594 INFO:tasks.rados.rados.0.smithi107.stdout:4626: seq_num 4625 ranges {565248=663552} 2020-06-12T02:38:17.601 INFO:tasks.rados.rados.0.smithi107.stdout:4626: writing smithi10729407-17 from 565248 to 1228800 tid 1 2020-06-12T02:38:17.601 INFO:tasks.rados.rados.0.smithi107.stdout:4627: delete oid 1 current snap is 0 2020-06-12T02:38:17.602 INFO:tasks.rados.rados.0.smithi107.stdout:4624: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:17.602 INFO:tasks.rados.rados.0.smithi107.stdout:4624: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:17.602 INFO:tasks.rados.rados.0.smithi107.stdout:4624: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:17.603 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 867 (ObjNum 4623 snap 0 seq_num 4623) dirty exists 2020-06-12T02:38:17.603 INFO:tasks.rados.rados.0.smithi107.stdout:4624: left oid 13 (ObjNum 4623 snap 0 seq_num 4623) 2020-06-12T02:38:17.605 INFO:tasks.rados.rados.0.smithi107.stdout:4625: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:17.609 INFO:tasks.rados.rados.0.smithi107.stdout:4625: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:17.610 INFO:tasks.rados.rados.0.smithi107.stdout:4624: done (4 left) 2020-06-12T02:38:17.610 INFO:tasks.rados.rados.0.smithi107.stdout:4627: done (3 left) 2020-06-12T02:38:17.611 INFO:tasks.rados.rados.0.smithi107.stdout:4628: delete oid 2 current snap is 0 2020-06-12T02:38:17.611 INFO:tasks.rados.rados.0.smithi107.stdout:4626: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:17.612 INFO:tasks.rados.rados.0.smithi107.stdout:4626: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:17.612 INFO:tasks.rados.rados.0.smithi107.stdout:4625: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:17.613 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 780 (ObjNum 4624 snap 0 seq_num 4624) dirty exists 2020-06-12T02:38:17.613 INFO:tasks.rados.rados.0.smithi107.stdout:4625: left oid 8 (ObjNum 4624 snap 0 seq_num 4624) 2020-06-12T02:38:17.614 INFO:tasks.rados.rados.0.smithi107.stdout:4626: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:17.614 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 859 (ObjNum 4625 snap 0 seq_num 4625) dirty exists 2020-06-12T02:38:17.614 INFO:tasks.rados.rados.0.smithi107.stdout:4626: left oid 17 (ObjNum 4625 snap 0 seq_num 4625) 2020-06-12T02:38:17.616 INFO:tasks.rados.rados.0.smithi107.stdout:4625: done (3 left) 2020-06-12T02:38:17.616 INFO:tasks.rados.rados.0.smithi107.stdout:4626: done (2 left) 2020-06-12T02:38:17.616 INFO:tasks.rados.rados.0.smithi107.stdout:4628: done (1 left) 2020-06-12T02:38:17.616 INFO:tasks.rados.rados.0.smithi107.stdout:4629: delete oid 7 current snap is 0 2020-06-12T02:38:17.617 INFO:tasks.rados.rados.0.smithi107.stdout:4629: done (1 left) 2020-06-12T02:38:17.617 INFO:tasks.rados.rados.0.smithi107.stdout:4630: delete oid 15 current snap is 0 2020-06-12T02:38:17.619 INFO:tasks.rados.rados.0.smithi107.stdout:4630: done (1 left) 2020-06-12T02:38:17.620 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:17.620 INFO:tasks.rados.rados.0.smithi107.stdout:4631: seq_num 4630 ranges {0=753664} 2020-06-12T02:38:17.624 INFO:tasks.rados.rados.0.smithi107.stdout:4631: writing smithi10729407-7 from 0 to 753664 tid 1 2020-06-12T02:38:17.625 INFO:tasks.rados.rados.0.smithi107.stdout:4632: delete oid 4 current snap is 0 2020-06-12T02:38:17.626 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:17.622+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:17.627 INFO:tasks.rados.rados.0.smithi107.stdout:4632: done (2 left) 2020-06-12T02:38:17.627 INFO:tasks.rados.rados.0.smithi107.stdout:4633: delete oid 16 current snap is 0 2020-06-12T02:38:17.631 INFO:tasks.rados.rados.0.smithi107.stdout:4633: done (2 left) 2020-06-12T02:38:17.631 INFO:tasks.rados.rados.0.smithi107.stdout:4634: delete oid 15 current snap is 0 2020-06-12T02:38:17.632 INFO:tasks.rados.rados.0.smithi107.stdout:4631: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:17.633 INFO:tasks.rados.rados.0.smithi107.stdout:4631: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:17.634 INFO:tasks.rados.rados.0.smithi107.stdout:4634: done (2 left) 2020-06-12T02:38:17.635 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:38:17.635 INFO:tasks.rados.rados.0.smithi107.stdout:4635: seq_num 4634 ranges {0=417792,417792=401408} 2020-06-12T02:38:17.636 INFO:tasks.rados.rados.0.smithi107.stdout:4635: writing smithi10729407-20 from 0 to 417792 tid 1 2020-06-12T02:38:17.643 INFO:tasks.rados.rados.0.smithi107.stdout:4635: writing smithi10729407-20 from 417792 to 819200 tid 2 2020-06-12T02:38:17.643 INFO:tasks.rados.rados.0.smithi107.stdout:4636: delete oid 2 current snap is 0 2020-06-12T02:38:17.644 INFO:tasks.rados.rados.0.smithi107.stdout:4631: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:17.644 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 870 (ObjNum 4630 snap 0 seq_num 4630) dirty exists 2020-06-12T02:38:17.644 INFO:tasks.rados.rados.0.smithi107.stdout:4631: left oid 7 (ObjNum 4630 snap 0 seq_num 4630) 2020-06-12T02:38:17.645 INFO:tasks.rados.rados.0.smithi107.stdout:4631: done (3 left) 2020-06-12T02:38:17.646 INFO:tasks.rados.rados.0.smithi107.stdout:4636: done (2 left) 2020-06-12T02:38:17.646 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:17.646 INFO:tasks.rados.rados.0.smithi107.stdout:4637: seq_num 4636 ranges {0=450560,450560=401408} 2020-06-12T02:38:17.647 INFO:tasks.rados.rados.0.smithi107.stdout:4637: writing smithi10729407-6 from 0 to 450560 tid 1 2020-06-12T02:38:17.648 INFO:tasks.rados.rados.0.smithi107.stdout:4637: writing smithi10729407-6 from 450560 to 851968 tid 2 2020-06-12T02:38:17.648 INFO:tasks.rados.rados.0.smithi107.stdout:4638: delete oid 17 current snap is 0 2020-06-12T02:38:17.648 INFO:tasks.rados.rados.0.smithi107.stdout:4635: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:17.649 INFO:tasks.rados.rados.0.smithi107.stdout:4635: finishing write tid 2 to smithi10729407-20 2020-06-12T02:38:17.650 INFO:tasks.rados.rados.0.smithi107.stdout:4635: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:17.651 INFO:tasks.rados.rados.0.smithi107.stdout:4637: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:17.651 INFO:tasks.rados.rados.0.smithi107.stdout:4638: done (3 left) 2020-06-12T02:38:17.652 INFO:tasks.rados.rados.0.smithi107.stdout:4639: delete oid 1 current snap is 0 2020-06-12T02:38:17.652 INFO:tasks.rados.rados.0.smithi107.stdout:4635: finishing write tid 5 to smithi10729407-20 2020-06-12T02:38:17.652 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 355 (ObjNum 4634 snap 0 seq_num 4634) dirty exists 2020-06-12T02:38:17.653 INFO:tasks.rados.rados.0.smithi107.stdout:4635: left oid 20 (ObjNum 4634 snap 0 seq_num 4634) 2020-06-12T02:38:17.657 INFO:tasks.rados.rados.0.smithi107.stdout:4637: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:17.657 INFO:tasks.rados.rados.0.smithi107.stdout:4637: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:17.659 INFO:tasks.rados.rados.0.smithi107.stdout:4635: done (3 left) 2020-06-12T02:38:17.659 INFO:tasks.rados.rados.0.smithi107.stdout:4639: done (2 left) 2020-06-12T02:38:17.659 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:38:17.659 INFO:tasks.rados.rados.0.smithi107.stdout:4640: seq_num 4639 ranges {0=499712,499712=401408} 2020-06-12T02:38:17.661 INFO:tasks.rados.rados.0.smithi107.stdout:4640: writing smithi10729407-3 from 0 to 499712 tid 1 2020-06-12T02:38:17.664 INFO:tasks.rados.rados.0.smithi107.stdout:4640: writing smithi10729407-3 from 499712 to 901120 tid 2 2020-06-12T02:38:17.664 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:17.664 INFO:tasks.rados.rados.0.smithi107.stdout:4641: seq_num 4640 ranges {0=516096,516096=401408} 2020-06-12T02:38:17.668 INFO:tasks.rados.rados.0.smithi107.stdout:4641: writing smithi10729407-16 from 0 to 516096 tid 1 2020-06-12T02:38:17.670 INFO:tasks.rados.rados.0.smithi107.stdout:4641: writing smithi10729407-16 from 516096 to 917504 tid 2 2020-06-12T02:38:17.670 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:17.671 INFO:tasks.rados.rados.0.smithi107.stdout:4642: seq_num 4641 ranges {0=532480,532480=401408} 2020-06-12T02:38:17.674 INFO:tasks.rados.rados.0.smithi107.stdout:4642: writing smithi10729407-1 from 0 to 532480 tid 1 2020-06-12T02:38:17.678 INFO:tasks.rados.rados.0.smithi107.stdout:4642: writing smithi10729407-1 from 532480 to 933888 tid 2 2020-06-12T02:38:17.678 INFO:tasks.rados.rados.0.smithi107.stdout:4643: delete oid 14 current snap is 0 2020-06-12T02:38:17.678 INFO:tasks.rados.rados.0.smithi107.stdout:4637: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:17.678 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 852 (ObjNum 4636 snap 0 seq_num 4636) dirty exists 2020-06-12T02:38:17.679 INFO:tasks.rados.rados.0.smithi107.stdout:4637: left oid 6 (ObjNum 4636 snap 0 seq_num 4636) 2020-06-12T02:38:17.679 INFO:tasks.rados.rados.0.smithi107.stdout:4640: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:17.679 INFO:tasks.rados.rados.0.smithi107.stdout:4640: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:17.679 INFO:tasks.rados.rados.0.smithi107.stdout:4640: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:17.679 INFO:tasks.rados.rados.0.smithi107.stdout:4640: finishing write tid 5 to smithi10729407-3 2020-06-12T02:38:17.680 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 828 (ObjNum 4639 snap 0 seq_num 4639) dirty exists 2020-06-12T02:38:17.680 INFO:tasks.rados.rados.0.smithi107.stdout:4640: left oid 3 (ObjNum 4639 snap 0 seq_num 4639) 2020-06-12T02:38:17.680 INFO:tasks.rados.rados.0.smithi107.stdout:4641: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:17.680 INFO:tasks.rados.rados.0.smithi107.stdout:4637: done (5 left) 2020-06-12T02:38:17.681 INFO:tasks.rados.rados.0.smithi107.stdout:4640: done (4 left) 2020-06-12T02:38:17.681 INFO:tasks.rados.rados.0.smithi107.stdout:4643: done (3 left) 2020-06-12T02:38:17.681 INFO:tasks.rados.rados.0.smithi107.stdout:4644: delete oid 11 current snap is 0 2020-06-12T02:38:17.682 INFO:tasks.rados.rados.0.smithi107.stdout:4642: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:17.684 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2020-06-12T02:38:17.684 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2020-06-12T02:38:17.684 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump -f json-pretty 2020-06-12T02:38:17.685 INFO:tasks.rados.rados.0.smithi107.stdout:4644: done (3 left) 2020-06-12T02:38:17.686 INFO:tasks.rados.rados.0.smithi107.stdout:4645: delete oid 7 current snap is 0 2020-06-12T02:38:17.690 INFO:tasks.rados.rados.0.smithi107.stdout:4641: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:17.690 INFO:tasks.rados.rados.0.smithi107.stdout:4641: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:17.690 INFO:tasks.rados.rados.0.smithi107.stdout:4642: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:17.690 INFO:tasks.rados.rados.0.smithi107.stdout:4645: done (3 left) 2020-06-12T02:38:17.691 INFO:tasks.rados.rados.0.smithi107.stdout:4646: delete oid 4 current snap is 0 2020-06-12T02:38:17.693 INFO:tasks.rados.rados.0.smithi107.stdout:4641: finishing write tid 5 to smithi10729407-16 2020-06-12T02:38:17.693 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 831 (ObjNum 4640 snap 0 seq_num 4640) dirty exists 2020-06-12T02:38:17.693 INFO:tasks.rados.rados.0.smithi107.stdout:4641: left oid 16 (ObjNum 4640 snap 0 seq_num 4640) 2020-06-12T02:38:17.695 INFO:tasks.rados.rados.0.smithi107.stdout:4642: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:17.697 INFO:tasks.rados.rados.0.smithi107.stdout:4642: finishing write tid 5 to smithi10729407-1 2020-06-12T02:38:17.697 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 856 (ObjNum 4641 snap 0 seq_num 4641) dirty exists 2020-06-12T02:38:17.697 INFO:tasks.rados.rados.0.smithi107.stdout:4642: left oid 1 (ObjNum 4641 snap 0 seq_num 4641) 2020-06-12T02:38:17.698 INFO:tasks.rados.rados.0.smithi107.stdout:4641: done (3 left) 2020-06-12T02:38:17.698 INFO:tasks.rados.rados.0.smithi107.stdout:4642: done (2 left) 2020-06-12T02:38:17.698 INFO:tasks.rados.rados.0.smithi107.stdout:4646: done (1 left) 2020-06-12T02:38:17.698 INFO:tasks.rados.rados.0.smithi107.stdout:4647: delete oid 19 current snap is 0 2020-06-12T02:38:17.700 INFO:tasks.rados.rados.0.smithi107.stdout:4647: done (1 left) 2020-06-12T02:38:17.700 INFO:tasks.rados.rados.0.smithi107.stdout:4648: delete oid 1 current snap is 0 2020-06-12T02:38:17.704 INFO:tasks.rados.rados.0.smithi107.stdout:4648: done (1 left) 2020-06-12T02:38:17.704 INFO:tasks.rados.rados.0.smithi107.stdout:4649: delete oid 9 current snap is 0 2020-06-12T02:38:17.707 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:17.707 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:38:17.708 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:38:17.708 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:38:17.708 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:17.709 INFO:tasks.rados.rados.0.smithi107.stdout:4649: done (1 left) 2020-06-12T02:38:17.709 INFO:tasks.rados.rados.0.smithi107.stdout:4650: delete oid 20 current snap is 0 2020-06-12T02:38:17.714 INFO:tasks.rados.rados.0.smithi107.stdout:4650: done (1 left) 2020-06-12T02:38:17.714 INFO:tasks.rados.rados.0.smithi107.stdout:4651: delete oid 16 current snap is 0 2020-06-12T02:38:17.717 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-06-12T02:38:17.718 INFO:tasks.rados.rados.0.smithi107.stdout:4651: done (1 left) 2020-06-12T02:38:17.719 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:17.719 INFO:tasks.rados.rados.0.smithi107.stdout:4652: seq_num 4651 ranges {0=704512,704512=401408} 2020-06-12T02:38:17.721 INFO:tasks.rados.rados.0.smithi107.stdout:4652: writing smithi10729407-14 from 0 to 704512 tid 1 2020-06-12T02:38:17.724 INFO:tasks.rados.rados.0.smithi107.stdout:4652: writing smithi10729407-14 from 704512 to 1105920 tid 2 2020-06-12T02:38:17.724 INFO:tasks.rados.rados.0.smithi107.stdout:4653: delete oid 3 current snap is 0 2020-06-12T02:38:17.730 INFO:tasks.rados.rados.0.smithi107.stdout:4653: done (2 left) 2020-06-12T02:38:17.730 INFO:tasks.rados.rados.0.smithi107.stdout:4654: delete oid 15 current snap is 0 2020-06-12T02:38:17.731 INFO:tasks.rados.rados.0.smithi107.stdout:4652: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:17.734 INFO:tasks.rados.rados.0.smithi107.stdout:4654: done (2 left) 2020-06-12T02:38:17.734 INFO:tasks.rados.rados.0.smithi107.stdout:4655: delete oid 13 current snap is 0 2020-06-12T02:38:17.736 INFO:tasks.rados.rados.0.smithi107.stdout:4652: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:17.739 INFO:tasks.rados.rados.0.smithi107.stdout:4655: done (2 left) 2020-06-12T02:38:17.740 INFO:tasks.rados.rados.0.smithi107.stdout:4656: delete oid 1 current snap is 0 2020-06-12T02:38:17.740 INFO:tasks.rados.rados.0.smithi107.stdout:4652: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:17.742 INFO:tasks.rados.rados.0.smithi107.stdout:4652: finishing write tid 5 to smithi10729407-14 2020-06-12T02:38:17.742 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 832 (ObjNum 4651 snap 0 seq_num 4651) dirty exists 2020-06-12T02:38:17.742 INFO:tasks.rados.rados.0.smithi107.stdout:4652: left oid 14 (ObjNum 4651 snap 0 seq_num 4651) 2020-06-12T02:38:17.745 INFO:tasks.rados.rados.0.smithi107.stdout:4652: done (2 left) 2020-06-12T02:38:17.746 INFO:tasks.rados.rados.0.smithi107.stdout:4656: done (1 left) 2020-06-12T02:38:17.746 INFO:tasks.rados.rados.0.smithi107.stdout:4657: delete oid 16 current snap is 0 2020-06-12T02:38:17.747 INFO:tasks.rados.rados.0.smithi107.stdout:4657: done (1 left) 2020-06-12T02:38:17.748 INFO:tasks.rados.rados.0.smithi107.stdout:4658: delete oid 3 current snap is 0 2020-06-12T02:38:17.749 INFO:tasks.rados.rados.0.smithi107.stdout:4658: done (1 left) 2020-06-12T02:38:17.750 INFO:tasks.rados.rados.0.smithi107.stdout:4659: delete oid 20 current snap is 0 2020-06-12T02:38:17.752 INFO:tasks.rados.rados.0.smithi107.stdout:4659: done (1 left) 2020-06-12T02:38:17.752 INFO:tasks.rados.rados.0.smithi107.stdout:4660: delete oid 9 current snap is 0 2020-06-12T02:38:17.753 INFO:tasks.rados.rados.0.smithi107.stdout:4660: done (1 left) 2020-06-12T02:38:17.753 INFO:tasks.rados.rados.0.smithi107.stdout:4661: delete oid 11 current snap is 0 2020-06-12T02:38:17.758 INFO:tasks.rados.rados.0.smithi107.stdout:4661: done (1 left) 2020-06-12T02:38:17.758 INFO:tasks.rados.rados.0.smithi107.stdout:4662: delete oid 9 current snap is 0 2020-06-12T02:38:17.759 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:38:17.756+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:38:17.760 INFO:tasks.rados.rados.0.smithi107.stdout:4662: done (1 left) 2020-06-12T02:38:17.760 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:17.760 INFO:tasks.rados.rados.0.smithi107.stdout:4663: seq_num 4662 ranges {1105920=483328,1589248=417792,2007040=385024} 2020-06-12T02:38:17.768 INFO:tasks.rados.rados.0.smithi107.stdout:4663: writing smithi10729407-14 from 1105920 to 1589248 tid 1 2020-06-12T02:38:17.771 INFO:tasks.rados.rados.0.smithi107.stdout:4663: writing smithi10729407-14 from 1589248 to 2007040 tid 2 2020-06-12T02:38:17.774 INFO:tasks.rados.rados.0.smithi107.stdout:4663: writing smithi10729407-14 from 2007040 to 2392064 tid 3 2020-06-12T02:38:17.774 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:17.774 INFO:tasks.rados.rados.0.smithi107.stdout:4664: seq_num 4663 ranges {0=499712,499712=696320,1196032=106496} 2020-06-12T02:38:17.777 INFO:tasks.rados.rados.0.smithi107.stdout:4664: writing smithi10729407-4 from 0 to 499712 tid 1 2020-06-12T02:38:17.782 INFO:tasks.rados.rados.0.smithi107.stdout:4664: writing smithi10729407-4 from 499712 to 1196032 tid 2 2020-06-12T02:38:17.783 INFO:tasks.rados.rados.0.smithi107.stdout:4664: writing smithi10729407-4 from 1196032 to 1302528 tid 3 2020-06-12T02:38:17.783 INFO:tasks.rados.rados.0.smithi107.stdout:4665: delete oid 1 current snap is 0 2020-06-12T02:38:17.784 INFO:tasks.rados.rados.0.smithi107.stdout:4663: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:17.784 INFO:tasks.rados.rados.0.smithi107.stdout:4663: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:17.784 INFO:tasks.rados.rados.0.smithi107.stdout:4663: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:17.784 INFO:tasks.rados.rados.0.smithi107.stdout:4663: finishing write tid 5 to smithi10729407-14 2020-06-12T02:38:17.785 INFO:tasks.rados.rados.0.smithi107.stdout:4664: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:17.785 INFO:tasks.rados.rados.0.smithi107.stdout:4663: finishing write tid 6 to smithi10729407-14 2020-06-12T02:38:17.785 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 837 (ObjNum 4662 snap 0 seq_num 4662) dirty exists 2020-06-12T02:38:17.785 INFO:tasks.rados.rados.0.smithi107.stdout:4663: left oid 14 (ObjNum 4662 snap 0 seq_num 4662) 2020-06-12T02:38:17.788 INFO:tasks.rados.rados.0.smithi107.stdout:4663: done (3 left) 2020-06-12T02:38:17.789 INFO:tasks.rados.rados.0.smithi107.stdout:4665: done (2 left) 2020-06-12T02:38:17.789 INFO:tasks.rados.rados.0.smithi107.stdout:4666: delete oid 12 current snap is 0 2020-06-12T02:38:17.794 INFO:tasks.rados.rados.0.smithi107.stdout:4664: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:17.794 INFO:tasks.rados.rados.0.smithi107.stdout:4664: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:17.796 INFO:tasks.rados.rados.0.smithi107.stdout:4664: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:17.797 INFO:tasks.rados.rados.0.smithi107.stdout:4666: done (2 left) 2020-06-12T02:38:17.798 INFO:tasks.rados.rados.0.smithi107.stdout:4667: delete oid 1 current snap is 0 2020-06-12T02:38:17.798 INFO:tasks.rados.rados.0.smithi107.stdout:4664: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:17.798 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 866 (ObjNum 4663 snap 0 seq_num 4663) dirty exists 2020-06-12T02:38:17.798 INFO:tasks.rados.rados.0.smithi107.stdout:4664: left oid 4 (ObjNum 4663 snap 0 seq_num 4663) 2020-06-12T02:38:17.798 INFO:tasks.rados.rados.0.smithi107.stdout:4664: done (2 left) 2020-06-12T02:38:17.799 INFO:tasks.rados.rados.0.smithi107.stdout:4667: done (1 left) 2020-06-12T02:38:17.799 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:17.799 INFO:tasks.rados.rados.0.smithi107.stdout:4668: seq_num 4667 ranges {0=565248,565248=638976,1204224=163840} 2020-06-12T02:38:17.800 INFO:tasks.rados.rados.0.smithi107.stdout:4668: writing smithi10729407-19 from 0 to 565248 tid 1 2020-06-12T02:38:17.806 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-06-12T02:38:17.807 INFO:tasks.rados.rados.0.smithi107.stdout:4668: writing smithi10729407-19 from 565248 to 1204224 tid 2 2020-06-12T02:38:17.807 INFO:tasks.rados.rados.0.smithi107.stdout:4668: writing smithi10729407-19 from 1204224 to 1368064 tid 3 2020-06-12T02:38:17.808 INFO:tasks.rados.rados.0.smithi107.stdout:4669: delete oid 4 current snap is 0 2020-06-12T02:38:17.811 INFO:tasks.rados.rados.0.smithi107.stdout:4668: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:17.812 INFO:tasks.rados.rados.0.smithi107.stdout:4669: done (2 left) 2020-06-12T02:38:17.812 INFO:tasks.rados.rados.0.smithi107.stdout:4670: delete oid 7 current snap is 0 2020-06-12T02:38:17.815 INFO:tasks.rados.rados.0.smithi107.stdout:4670: done (2 left) 2020-06-12T02:38:17.815 INFO:tasks.rados.rados.0.smithi107.stdout:4671: delete oid 2 current snap is 0 2020-06-12T02:38:17.816 INFO:tasks.rados.rados.0.smithi107.stdout:4671: done (2 left) 2020-06-12T02:38:17.816 INFO:tasks.rados.rados.0.smithi107.stdout:4672: delete oid 16 current snap is 0 2020-06-12T02:38:17.817 INFO:tasks.rados.rados.0.smithi107.stdout:4668: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:17.817 INFO:tasks.rados.rados.0.smithi107.stdout:4668: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:17.817 INFO:tasks.rados.rados.0.smithi107.stdout:4668: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:17.818 INFO:tasks.rados.rados.0.smithi107.stdout:4668: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:17.819 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 802 (ObjNum 4667 snap 0 seq_num 4667) dirty exists 2020-06-12T02:38:17.819 INFO:tasks.rados.rados.0.smithi107.stdout:4668: left oid 19 (ObjNum 4667 snap 0 seq_num 4667) 2020-06-12T02:38:17.819 INFO:tasks.rados.rados.0.smithi107.stdout:4668: done (2 left) 2020-06-12T02:38:17.819 INFO:tasks.rados.rados.0.smithi107.stdout:4672: done (1 left) 2020-06-12T02:38:17.819 INFO:tasks.rados.rados.0.smithi107.stdout:4673: delete oid 6 current snap is 0 2020-06-12T02:38:17.821 INFO:tasks.rados.rados.0.smithi107.stdout:4673: done (1 left) 2020-06-12T02:38:17.821 INFO:tasks.rados.rados.0.smithi107.stdout:4674: delete oid 13 current snap is 0 2020-06-12T02:38:17.823 INFO:tasks.rados.rados.0.smithi107.stdout:4674: done (1 left) 2020-06-12T02:38:17.824 INFO:tasks.rados.rados.0.smithi107.stdout:4675: delete oid 13 current snap is 0 2020-06-12T02:38:17.826 INFO:tasks.rados.rados.0.smithi107.stdout:4675: done (1 left) 2020-06-12T02:38:17.826 INFO:tasks.rados.rados.0.smithi107.stdout:4676: delete oid 20 current snap is 0 2020-06-12T02:38:17.827 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:17.824+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:17.828 INFO:tasks.rados.rados.0.smithi107.stdout:4676: done (1 left) 2020-06-12T02:38:17.828 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:17.828 INFO:tasks.rados.rados.0.smithi107.stdout:4677: seq_num 4676 ranges {1368064=720896,2088960=491520,2580480=311296} 2020-06-12T02:38:17.839 INFO:tasks.rados.rados.0.smithi107.stdout:4677: writing smithi10729407-19 from 1368064 to 2088960 tid 1 2020-06-12T02:38:17.842 INFO:tasks.rados.rados.0.smithi107.stdout:4677: writing smithi10729407-19 from 2088960 to 2580480 tid 2 2020-06-12T02:38:17.847 INFO:tasks.rados.rados.0.smithi107.stdout:4677: writing smithi10729407-19 from 2580480 to 2891776 tid 3 2020-06-12T02:38:17.847 INFO:tasks.rados.rados.0.smithi107.stdout:4678: delete oid 12 current snap is 0 2020-06-12T02:38:17.847 INFO:tasks.rados.rados.0.smithi107.stdout:4677: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:17.854 INFO:tasks.rados.rados.0.smithi107.stdout:4677: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:17.856 INFO:tasks.rados.rados.0.smithi107.stdout:4677: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:17.857 INFO:tasks.rados.rados.0.smithi107.stdout:4677: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:17.858 INFO:tasks.rados.rados.0.smithi107.stdout:4677: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:17.859 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 806 (ObjNum 4676 snap 0 seq_num 4676) dirty exists 2020-06-12T02:38:17.859 INFO:tasks.rados.rados.0.smithi107.stdout:4677: left oid 19 (ObjNum 4676 snap 0 seq_num 4676) 2020-06-12T02:38:17.860 INFO:tasks.rados.rados.0.smithi107.stdout:4677: done (2 left) 2020-06-12T02:38:17.860 INFO:tasks.rados.rados.0.smithi107.stdout:4678: done (1 left) 2020-06-12T02:38:17.860 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:17.860 INFO:tasks.rados.rados.0.smithi107.stdout:4679: seq_num 4678 ranges {647168=753664,1400832=663552,2064384=139264} 2020-06-12T02:38:17.867 INFO:tasks.rados.rados.0.smithi107.stdout:4679: writing smithi10729407-8 from 647168 to 1400832 tid 1 2020-06-12T02:38:17.872 INFO:tasks.rados.rados.0.smithi107.stdout:4679: writing smithi10729407-8 from 1400832 to 2064384 tid 2 2020-06-12T02:38:17.874 INFO:tasks.rados.rados.0.smithi107.stdout:4679: writing smithi10729407-8 from 2064384 to 2203648 tid 3 2020-06-12T02:38:17.874 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:17.875 INFO:tasks.rados.rados.0.smithi107.stdout:4680: seq_num 4679 ranges {0=770048,770048=548864,1318912=253952} 2020-06-12T02:38:17.881 INFO:tasks.rados.rados.0.smithi107.stdout:4680: writing smithi10729407-4 from 0 to 770048 tid 1 2020-06-12T02:38:17.883 INFO:tasks.rados.rados.0.smithi107.stdout:4680: writing smithi10729407-4 from 770048 to 1318912 tid 2 2020-06-12T02:38:17.885 INFO:tasks.rados.rados.0.smithi107.stdout:4680: writing smithi10729407-4 from 1318912 to 1572864 tid 3 2020-06-12T02:38:17.886 INFO:tasks.rados.rados.0.smithi107.stdout:4681: delete oid 7 current snap is 0 2020-06-12T02:38:17.886 INFO:tasks.rados.rados.0.smithi107.stdout:4679: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:17.889 INFO:tasks.rados.rados.0.smithi107.stdout:4679: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:17.890 INFO:tasks.rados.rados.0.smithi107.stdout:4679: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:17.890 INFO:tasks.rados.rados.0.smithi107.stdout:4680: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:17.890 INFO:tasks.rados.rados.0.smithi107.stdout:4681: done (3 left) 2020-06-12T02:38:17.890 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:17.891 INFO:tasks.rados.rados.0.smithi107.stdout:4682: seq_num 4681 ranges {0=802816,802816=720896,1523712=81920} 2020-06-12T02:38:17.896 INFO:tasks.rados.rados.0.smithi107.stdout:4682: writing smithi10729407-9 from 0 to 802816 tid 1 2020-06-12T02:38:17.905 INFO:tasks.rados.rados.0.smithi107.stdout:4682: writing smithi10729407-9 from 802816 to 1523712 tid 2 2020-06-12T02:38:17.906 INFO:tasks.rados.rados.0.smithi107.stdout:4682: writing smithi10729407-9 from 1523712 to 1605632 tid 3 2020-06-12T02:38:17.906 INFO:tasks.rados.rados.0.smithi107.stdout:4679: finishing write tid 5 to smithi10729407-8 2020-06-12T02:38:17.906 INFO:tasks.rados.rados.0.smithi107.stdout:4679: finishing write tid 6 to smithi10729407-8 2020-06-12T02:38:17.906 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 787 (ObjNum 4678 snap 0 seq_num 4678) dirty exists 2020-06-12T02:38:17.906 INFO:tasks.rados.rados.0.smithi107.stdout:4679: left oid 8 (ObjNum 4678 snap 0 seq_num 4678) 2020-06-12T02:38:17.907 INFO:tasks.rados.rados.0.smithi107.stdout:4680: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:17.907 INFO:tasks.rados.rados.0.smithi107.stdout:4680: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:17.907 INFO:tasks.rados.rados.0.smithi107.stdout:4679: done (3 left) 2020-06-12T02:38:17.907 INFO:tasks.rados.rados.0.smithi107.stdout:4683: delete oid 3 current snap is 0 2020-06-12T02:38:17.907 INFO:tasks.rados.rados.0.smithi107.stdout:4680: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:17.907 INFO:tasks.rados.rados.0.smithi107.stdout:4680: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:17.908 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 871 (ObjNum 4679 snap 0 seq_num 4679) dirty exists 2020-06-12T02:38:17.908 INFO:tasks.rados.rados.0.smithi107.stdout:4680: left oid 4 (ObjNum 4679 snap 0 seq_num 4679) 2020-06-12T02:38:17.908 INFO:tasks.rados.rados.0.smithi107.stdout:4682: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:17.908 INFO:tasks.rados.rados.0.smithi107.stdout:4680: done (3 left) 2020-06-12T02:38:17.908 INFO:tasks.rados.rados.0.smithi107.stdout:4683: done (2 left) 2020-06-12T02:38:17.909 INFO:tasks.rados.rados.0.smithi107.stdout:4684: delete oid 3 current snap is 0 2020-06-12T02:38:17.910 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:17.910 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:38:17.911 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:38:17.911 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:38:17.911 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.911 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2747 3.as0 3:5cb723d2:::smithi10729407-15:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:17.911 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:36.724407+0000", 2020-06-12T02:38:17.912 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 41.181199513999999, 2020-06-12T02:38:17.912 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014982544, 2020-06-12T02:38:17.912 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:17.912 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:17.912 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:17.912 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:17.913 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:17.913 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2747 2020-06-12T02:38:17.913 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.913 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:17.913 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.914 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:17.914 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724407+0000", 2020-06-12T02:38:17.914 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:17.915 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.915 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.915 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:17.915 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724407+0000", 2020-06-12T02:38:17.915 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6766e-05 2020-06-12T02:38:17.916 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.916 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.916 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:17.916 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724444+0000", 2020-06-12T02:38:17.916 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.315e-06 2020-06-12T02:38:17.916 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.917 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.917 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:17.917 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724447+0000", 2020-06-12T02:38:17.917 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9100000000000002e-06 2020-06-12T02:38:17.917 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.918 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.918 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:17.918 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724453+0000", 2020-06-12T02:38:17.918 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.43e-06 2020-06-12T02:38:17.918 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.919 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.919 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:17.919 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724455+0000", 2020-06-12T02:38:17.919 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0061404010000000002 2020-06-12T02:38:17.919 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.919 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.920 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:17.920 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.730596+0000", 2020-06-12T02:38:17.920 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8402e-05 2020-06-12T02:38:17.920 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.920 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.921 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:17.921 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.730664+0000", 2020-06-12T02:38:17.922 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00031661500000000002 2020-06-12T02:38:17.923 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.923 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.924 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:17.925 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.730981+0000", 2020-06-12T02:38:17.925 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0049794890000000001 2020-06-12T02:38:17.925 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.925 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.926 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:17.926 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.735960+0000", 2020-06-12T02:38:17.926 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033245739999999998 2020-06-12T02:38:17.926 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.926 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.927 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:17.927 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.739285+0000", 2020-06-12T02:38:17.927 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010464200000000001 2020-06-12T02:38:17.927 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.927 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.927 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:17.928 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.739390+0000", 2020-06-12T02:38:17.928 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014982544 2020-06-12T02:38:17.928 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.928 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:17.928 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.929 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.929 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.929 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2748 3.as0 3:5cb723d2:::smithi10729407-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:38:17.929 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:36.724477+0000", 2020-06-12T02:38:17.930 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 41.181129374999998, 2020-06-12T02:38:17.930 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020852102000000001, 2020-06-12T02:38:17.930 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:17.930 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:17.930 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:17.931 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:17.931 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:17.931 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2748 2020-06-12T02:38:17.931 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.931 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:17.932 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.932 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:17.932 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724477+0000", 2020-06-12T02:38:17.932 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:17.932 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.933 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.933 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:17.933 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724477+0000", 2020-06-12T02:38:17.933 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2065e-05 2020-06-12T02:38:17.934 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.934 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.934 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:17.934 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724499+0000", 2020-06-12T02:38:17.934 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9340000000000002e-06 2020-06-12T02:38:17.935 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.935 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.935 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:17.935 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724502+0000", 2020-06-12T02:38:17.936 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.246e-06 2020-06-12T02:38:17.936 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.936 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.936 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:17.936 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724508+0000", 2020-06-12T02:38:17.937 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2900000000000001e-06 2020-06-12T02:38:17.937 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.937 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.937 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:17.937 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.724511+0000", 2020-06-12T02:38:17.937 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0068556700000000003 2020-06-12T02:38:17.938 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.938 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.938 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:17.938 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.731366+0000", 2020-06-12T02:38:17.938 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4738999999999999e-05 2020-06-12T02:38:17.939 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.939 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.939 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:17.939 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.731421+0000", 2020-06-12T02:38:17.939 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0080153150000000003 2020-06-12T02:38:17.940 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.940 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.940 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:17.940 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.739436+0000", 2020-06-12T02:38:17.940 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010418 2020-06-12T02:38:17.940 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.941 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.941 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:17.941 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.739541+0000", 2020-06-12T02:38:17.941 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0057886630000000003 2020-06-12T02:38:17.941 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.942 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.942 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:17.942 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:36.745329+0000", 2020-06-12T02:38:17.942 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020852102000000001 2020-06-12T02:38:17.942 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.943 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:17.943 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.943 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.943 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.943 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:3055 3.as0 3:5cb723d2:::smithi10729407-15:head [append 0~614400] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:17.943 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.918209+0000", 2020-06-12T02:38:17.944 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 39.987397659999999, 2020-06-12T02:38:17.944 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014165394, 2020-06-12T02:38:17.944 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:17.944 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:17.944 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:17.945 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:17.945 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:17.945 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 3055 2020-06-12T02:38:17.945 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.946 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:17.946 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.946 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:17.946 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.918209+0000", 2020-06-12T02:38:17.946 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:17.947 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.947 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.947 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:17.947 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.918209+0000", 2020-06-12T02:38:17.948 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00081606199999999999 2020-06-12T02:38:17.948 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.948 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.948 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:17.949 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.919025+0000", 2020-06-12T02:38:17.949 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.251e-06 2020-06-12T02:38:17.949 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.949 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.949 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:17.950 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.919031+0000", 2020-06-12T02:38:17.950 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1460999999999999e-05 2020-06-12T02:38:17.950 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.950 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.950 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:17.951 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.919043+0000", 2020-06-12T02:38:17.951 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.592e-06 2020-06-12T02:38:17.951 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.951 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.951 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:17.951 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.919048+0000", 2020-06-12T02:38:17.952 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0020427959999999999 2020-06-12T02:38:17.952 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.952 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.952 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:17.952 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.921091+0000", 2020-06-12T02:38:17.953 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012382800000000001 2020-06-12T02:38:17.953 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.953 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.953 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:17.953 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.921215+0000", 2020-06-12T02:38:17.954 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053208300000000004 2020-06-12T02:38:17.954 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.954 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.954 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:17.955 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.921747+0000", 2020-06-12T02:38:17.955 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0075648599999999996 2020-06-12T02:38:17.955 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.955 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.956 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:17.956 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.929312+0000", 2020-06-12T02:38:17.956 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0029879630000000002 2020-06-12T02:38:17.956 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.956 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.957 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:17.957 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.932300+0000", 2020-06-12T02:38:17.957 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4498000000000004e-05 2020-06-12T02:38:17.957 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.957 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.958 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:17.958 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.932374+0000", 2020-06-12T02:38:17.958 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014165394 2020-06-12T02:38:17.958 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.958 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:17.959 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.959 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.959 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.959 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:3219 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:38:17.959 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:38.300493+0000", 2020-06-12T02:38:17.959 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 39.605113668000001, 2020-06-12T02:38:17.960 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018386370999999999, 2020-06-12T02:38:17.960 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:17.960 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:17.960 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:17.960 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:17.961 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:17.961 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 3219 2020-06-12T02:38:17.961 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.961 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:17.961 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.962 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:17.962 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300493+0000", 2020-06-12T02:38:17.963 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:17.963 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.963 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.963 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:17.963 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300493+0000", 2020-06-12T02:38:17.963 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3008000000000001e-05 2020-06-12T02:38:17.964 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.964 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.964 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:17.964 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300516+0000", 2020-06-12T02:38:17.964 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1329999999999999e-06 2020-06-12T02:38:17.965 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.965 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.965 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:17.965 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300519+0000", 2020-06-12T02:38:17.965 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8260000000000004e-06 2020-06-12T02:38:17.966 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.966 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.966 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:17.966 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300526+0000", 2020-06-12T02:38:17.967 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.525e-06 2020-06-12T02:38:17.967 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.967 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.967 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:17.967 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300528+0000", 2020-06-12T02:38:17.968 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9839e-05 2020-06-12T02:38:17.968 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.968 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.968 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:17.969 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300558+0000", 2020-06-12T02:38:17.969 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3198000000000001e-05 2020-06-12T02:38:17.969 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.969 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.969 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:17.969 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.300611+0000", 2020-06-12T02:38:17.970 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0082281509999999995 2020-06-12T02:38:17.970 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.970 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.970 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:17.970 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.308840+0000", 2020-06-12T02:38:17.971 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5921000000000005e-05 2020-06-12T02:38:17.971 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.971 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.971 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:17.971 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.308935+0000", 2020-06-12T02:38:17.972 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050007560000000003 2020-06-12T02:38:17.972 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.972 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.972 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:17.972 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.313936+0000", 2020-06-12T02:38:17.973 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1600000000000007e-05 2020-06-12T02:38:17.973 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.973 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.973 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:17.974 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.313998+0000", 2020-06-12T02:38:17.974 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0048814139999999997 2020-06-12T02:38:17.974 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.974 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.974 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:17.974 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:38.318879+0000", 2020-06-12T02:38:17.975 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018386370999999999 2020-06-12T02:38:17.975 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.975 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:17.975 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.975 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.976 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.976 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:4478 3.as0 3:5cb723d2:::smithi10729407-15:head [append 0~483328] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-06-12T02:38:17.976 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:42.522285+0000", 2020-06-12T02:38:17.976 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 35.383321705, 2020-06-12T02:38:17.977 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013258748000000001, 2020-06-12T02:38:17.977 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:17.977 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:17.977 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:17.977 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:17.977 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:17.978 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 4478 2020-06-12T02:38:17.978 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.978 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:17.978 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.978 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:17.979 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.522285+0000", 2020-06-12T02:38:17.979 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:17.979 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.981 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.981 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:17.981 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.522285+0000", 2020-06-12T02:38:17.981 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00055732500000000001 2020-06-12T02:38:17.982 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.982 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.982 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:17.982 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.522842+0000", 2020-06-12T02:38:17.982 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1270000000000002e-06 2020-06-12T02:38:17.983 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.983 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.983 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:17.983 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.522847+0000", 2020-06-12T02:38:17.983 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1314e-05 2020-06-12T02:38:17.984 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.984 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.984 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:17.984 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.522859+0000", 2020-06-12T02:38:17.984 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4339999999999997e-06 2020-06-12T02:38:17.984 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.985 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.985 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:17.985 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.522865+0000", 2020-06-12T02:38:17.985 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033831099999999999 2020-06-12T02:38:17.985 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.986 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.986 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:17.986 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.526248+0000", 2020-06-12T02:38:17.986 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010359199999999999 2020-06-12T02:38:17.986 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.987 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.987 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:17.987 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.526352+0000", 2020-06-12T02:38:17.987 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031925009999999999 2020-06-12T02:38:17.987 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.987 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.988 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:17.988 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.529544+0000", 2020-06-12T02:38:17.988 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6626000000000003e-05 2020-06-12T02:38:17.988 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.988 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.989 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:17.989 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.529601+0000", 2020-06-12T02:38:17.989 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047735099999999999 2020-06-12T02:38:17.989 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.989 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.989 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:17.990 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.530078+0000", 2020-06-12T02:38:17.990 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0053046090000000001 2020-06-12T02:38:17.990 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.990 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.990 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:17.991 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.535383+0000", 2020-06-12T02:38:17.991 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011821599999999999 2020-06-12T02:38:17.991 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.991 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.991 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:17.991 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.535501+0000", 2020-06-12T02:38:17.992 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2543000000000002e-05 2020-06-12T02:38:17.992 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.992 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.992 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:17.992 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.535544+0000", 2020-06-12T02:38:17.993 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013258748000000001 2020-06-12T02:38:17.993 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.993 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:17.993 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:17.993 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.993 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.994 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:4535 3.as0 3:5732055a:::smithi10729407-8:head [append 0~712704] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-06-12T02:38:17.994 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:42.670365+0000", 2020-06-12T02:38:17.994 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 35.235241373999997, 2020-06-12T02:38:17.994 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013998371000000001, 2020-06-12T02:38:17.994 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:17.995 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:17.995 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:17.995 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:17.995 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:17.995 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 4535 2020-06-12T02:38:17.996 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.996 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:17.996 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.996 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:17.996 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.670365+0000", 2020-06-12T02:38:17.996 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:17.997 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.997 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.997 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:17.997 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.670365+0000", 2020-06-12T02:38:17.997 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00059204600000000004 2020-06-12T02:38:17.998 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.998 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.998 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:17.998 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.670957+0000", 2020-06-12T02:38:17.998 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.122e-06 2020-06-12T02:38:17.998 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:17.999 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:17.999 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:17.999 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.670963+0000", 2020-06-12T02:38:17.999 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1518999999999999e-05 2020-06-12T02:38:17.999 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.000 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.000 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:18.000 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.670975+0000", 2020-06-12T02:38:18.000 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9290000000000001e-06 2020-06-12T02:38:18.000 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.000 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.001 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:18.001 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.670982+0000", 2020-06-12T02:38:18.001 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4758000000000001e-05 2020-06-12T02:38:18.001 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.001 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.002 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:18.002 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.671027+0000", 2020-06-12T02:38:18.002 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9733999999999993e-05 2020-06-12T02:38:18.002 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.002 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.002 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:18.003 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.671096+0000", 2020-06-12T02:38:18.003 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000544448 2020-06-12T02:38:18.003 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.003 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.003 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:18.004 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.671641+0000", 2020-06-12T02:38:18.004 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012592534000000001 2020-06-12T02:38:18.004 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.004 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.004 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:18.004 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.684233+0000", 2020-06-12T02:38:18.005 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9785999999999998e-05 2020-06-12T02:38:18.005 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.005 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.005 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:18.005 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.684313+0000", 2020-06-12T02:38:18.006 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0494999999999997e-05 2020-06-12T02:38:18.006 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.006 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.006 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:18.006 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:42.684364+0000", 2020-06-12T02:38:18.006 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013998371000000001 2020-06-12T02:38:18.007 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:18.007 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:18.007 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:18.007 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.007 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.008 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:4975 3.as0 3:5732055a:::smithi10729407-8:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e30)", 2020-06-12T02:38:18.008 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:43.721794+0000", 2020-06-12T02:38:18.008 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.183812881000001, 2020-06-12T02:38:18.008 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.1825135, 2020-06-12T02:38:18.008 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:18.008 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:18.009 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:18.009 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:18.009 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:18.009 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 4975 2020-06-12T02:38:18.009 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.010 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:18.010 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.010 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:18.010 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721794+0000", 2020-06-12T02:38:18.010 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:18.010 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.011 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.011 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:18.011 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721794+0000", 2020-06-12T02:38:18.011 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4542e-05 2020-06-12T02:38:18.011 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.012 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.012 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:18.012 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721818+0000", 2020-06-12T02:38:18.012 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9169999999999997e-06 2020-06-12T02:38:18.012 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.012 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.013 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:18.013 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721823+0000", 2020-06-12T02:38:18.013 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1184e-05 2020-06-12T02:38:18.013 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.013 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.014 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:18.014 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721834+0000", 2020-06-12T02:38:18.014 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.254e-06 2020-06-12T02:38:18.014 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.014 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.014 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:18.015 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721840+0000", 2020-06-12T02:38:18.015 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1075999999999998e-05 2020-06-12T02:38:18.015 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.015 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.015 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:18.016 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721881+0000", 2020-06-12T02:38:18.016 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7416999999999995e-05 2020-06-12T02:38:18.016 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.016 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.016 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:18.016 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721948+0000", 2020-06-12T02:38:18.017 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.18235910999999999 2020-06-12T02:38:18.017 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.017 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.017 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:18.017 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.904307+0000", 2020-06-12T02:38:18.018 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.1825135 2020-06-12T02:38:18.018 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:18.018 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:18.018 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:18.018 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.018 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.019 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:6678 3.as0 3:5732055a:::smithi10729407-8:head [append 0~499712] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-06-12T02:38:18.019 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:48.626196+0000", 2020-06-12T02:38:18.019 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 29.279410897999998, 2020-06-12T02:38:18.019 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014093700000000001, 2020-06-12T02:38:18.019 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:18.020 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:18.020 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:18.020 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:18.020 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:18.020 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 6678 2020-06-12T02:38:18.020 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.021 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:18.021 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.021 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:18.021 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.626196+0000", 2020-06-12T02:38:18.021 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:18.022 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.022 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.022 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:18.022 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.626196+0000", 2020-06-12T02:38:18.022 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00046254199999999998 2020-06-12T02:38:18.022 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.023 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.023 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:18.023 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.626658+0000", 2020-06-12T02:38:18.023 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3400000000000003e-06 2020-06-12T02:38:18.023 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.024 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.024 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:18.024 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.626665+0000", 2020-06-12T02:38:18.024 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0079e-05 2020-06-12T02:38:18.024 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.024 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.025 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:18.025 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.626675+0000", 2020-06-12T02:38:18.025 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0999999999999997e-06 2020-06-12T02:38:18.025 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.025 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.026 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:18.026 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.626679+0000", 2020-06-12T02:38:18.026 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025692100000000002 2020-06-12T02:38:18.026 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.026 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.026 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:18.027 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.629248+0000", 2020-06-12T02:38:18.027 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000107944 2020-06-12T02:38:18.027 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.027 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.027 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:18.028 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.629356+0000", 2020-06-12T02:38:18.028 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001434999 2020-06-12T02:38:18.028 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.028 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.028 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:18.028 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.630791+0000", 2020-06-12T02:38:18.029 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0716e-05 2020-06-12T02:38:18.029 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.029 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.029 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:18.029 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.630842+0000", 2020-06-12T02:38:18.030 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00045107600000000002 2020-06-12T02:38:18.030 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.030 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.030 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:18.030 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.631293+0000", 2020-06-12T02:38:18.030 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0048208330000000001 2020-06-12T02:38:18.031 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.031 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.031 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:18.031 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.636113+0000", 2020-06-12T02:38:18.031 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0041129679999999998 2020-06-12T02:38:18.032 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.032 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.032 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:18.032 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.640226+0000", 2020-06-12T02:38:18.032 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2892999999999996e-05 2020-06-12T02:38:18.032 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.033 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.033 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:18.033 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:48.640289+0000", 2020-06-12T02:38:18.033 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014093700000000001 2020-06-12T02:38:18.033 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:18.034 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:18.034 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:18.034 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.034 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.034 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:7086 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e32)", 2020-06-12T02:38:18.034 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:49.667192+0000", 2020-06-12T02:38:18.035 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 28.238414569, 2020-06-12T02:38:18.035 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014618023000000001, 2020-06-12T02:38:18.035 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:18.035 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:18.035 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:18.036 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:18.036 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:18.036 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 7086 2020-06-12T02:38:18.036 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.036 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:18.036 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.037 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:18.037 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667192+0000", 2020-06-12T02:38:18.037 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:18.037 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.037 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.038 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:18.038 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667192+0000", 2020-06-12T02:38:18.038 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1912e-05 2020-06-12T02:38:18.038 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.038 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.038 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:18.039 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667214+0000", 2020-06-12T02:38:18.039 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0469999999999998e-06 2020-06-12T02:38:18.039 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.039 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.039 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:18.040 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667217+0000", 2020-06-12T02:38:18.040 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7170000000000002e-06 2020-06-12T02:38:18.040 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.040 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.040 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:18.040 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667224+0000", 2020-06-12T02:38:18.041 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2850000000000001e-06 2020-06-12T02:38:18.041 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.041 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.041 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:18.041 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.667226+0000", 2020-06-12T02:38:18.042 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0037197950000000001 2020-06-12T02:38:18.042 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.042 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.042 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:18.042 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.670946+0000", 2020-06-12T02:38:18.043 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4549999999999994e-05 2020-06-12T02:38:18.043 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.043 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.043 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:18.043 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.671040+0000", 2020-06-12T02:38:18.043 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0043129199999999996 2020-06-12T02:38:18.044 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.044 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.044 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:18.044 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.675353+0000", 2020-06-12T02:38:18.044 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1982000000000001e-05 2020-06-12T02:38:18.045 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.045 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.045 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:18.045 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.675445+0000", 2020-06-12T02:38:18.045 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.005067401 2020-06-12T02:38:18.046 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.046 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.046 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:18.046 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.680513+0000", 2020-06-12T02:38:18.046 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000101508 2020-06-12T02:38:18.046 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.047 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.047 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:18.047 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.680614+0000", 2020-06-12T02:38:18.047 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0011959060000000001 2020-06-12T02:38:18.047 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.047 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.048 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:18.048 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:49.681810+0000", 2020-06-12T02:38:18.048 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014618023000000001 2020-06-12T02:38:18.051 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:18.051 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:18.051 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:18.051 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.051 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.052 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:7902 3.as0 3:5732055a:::smithi10729407-8:head [append 0~499712] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-06-12T02:38:18.052 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.159351+0000", 2020-06-12T02:38:18.052 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 25.746255892000001, 2020-06-12T02:38:18.052 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013425812000000001, 2020-06-12T02:38:18.052 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:18.053 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:18.053 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:18.053 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:18.053 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:18.054 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 7902 2020-06-12T02:38:18.054 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.054 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:18.054 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.054 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:18.055 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.159351+0000", 2020-06-12T02:38:18.055 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:18.055 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.055 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.055 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:18.056 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.159351+0000", 2020-06-12T02:38:18.056 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00048251199999999999 2020-06-12T02:38:18.056 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.056 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.056 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:18.057 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.159833+0000", 2020-06-12T02:38:18.057 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7509999999999997e-06 2020-06-12T02:38:18.057 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.057 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.057 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:18.057 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.159839+0000", 2020-06-12T02:38:18.058 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9840000000000007e-06 2020-06-12T02:38:18.058 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.058 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.058 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:18.058 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.159848+0000", 2020-06-12T02:38:18.059 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4619999999999998e-06 2020-06-12T02:38:18.059 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.059 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.059 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:18.060 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.159852+0000", 2020-06-12T02:38:18.060 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5348000000000001e-05 2020-06-12T02:38:18.060 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.060 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.060 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:18.060 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.159888+0000", 2020-06-12T02:38:18.061 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9824000000000001e-05 2020-06-12T02:38:18.061 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.061 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.061 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:18.061 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.159958+0000", 2020-06-12T02:38:18.062 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0014605149999999999 2020-06-12T02:38:18.062 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.062 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.062 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:18.062 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.161418+0000", 2020-06-12T02:38:18.062 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0667999999999997e-05 2020-06-12T02:38:18.063 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.063 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.063 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:18.063 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.161469+0000", 2020-06-12T02:38:18.063 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00058577200000000003 2020-06-12T02:38:18.064 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.064 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.064 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:18.064 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.162054+0000", 2020-06-12T02:38:18.064 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010590761000000001 2020-06-12T02:38:18.064 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.065 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.065 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:18.065 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.172645+0000", 2020-06-12T02:38:18.065 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8467000000000006e-05 2020-06-12T02:38:18.065 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.066 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.066 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:18.066 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.172724+0000", 2020-06-12T02:38:18.066 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2747999999999997e-05 2020-06-12T02:38:18.066 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.066 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.067 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:18.067 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.172776+0000", 2020-06-12T02:38:18.067 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013425812000000001 2020-06-12T02:38:18.067 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:18.067 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:18.068 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:18.068 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.068 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.068 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:7931 3.as0 3:5732055a:::smithi10729407-8:head [append 0~696320] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-06-12T02:38:18.068 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.246494+0000", 2020-06-12T02:38:18.068 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 25.659112074999999, 2020-06-12T02:38:18.069 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014082301, 2020-06-12T02:38:18.069 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:18.069 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:18.069 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:18.069 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:18.070 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:18.070 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 7931 2020-06-12T02:38:18.070 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.070 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:18.070 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.071 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:18.071 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.246494+0000", 2020-06-12T02:38:18.071 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:18.071 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.071 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.071 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:18.072 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.246494+0000", 2020-06-12T02:38:18.072 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00077570099999999995 2020-06-12T02:38:18.072 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.072 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.072 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:18.072 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.247270+0000", 2020-06-12T02:38:18.073 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4179999999999997e-06 2020-06-12T02:38:18.073 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.073 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.073 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:18.073 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.247279+0000", 2020-06-12T02:38:18.074 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6297999999999999e-05 2020-06-12T02:38:18.074 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.074 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.074 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:18.074 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.247295+0000", 2020-06-12T02:38:18.075 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3309999999999998e-06 2020-06-12T02:38:18.075 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.075 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.075 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:18.075 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.247302+0000", 2020-06-12T02:38:18.075 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4283999999999999e-05 2020-06-12T02:38:18.076 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.076 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.076 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:18.076 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.247336+0000", 2020-06-12T02:38:18.076 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8157999999999994e-05 2020-06-12T02:38:18.077 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.077 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.077 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:18.077 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.247405+0000", 2020-06-12T02:38:18.077 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031257059999999998 2020-06-12T02:38:18.077 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.078 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.078 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:18.078 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.250530+0000", 2020-06-12T02:38:18.078 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7546999999999995e-05 2020-06-12T02:38:18.078 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.079 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.079 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:18.079 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.250608+0000", 2020-06-12T02:38:18.079 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00074126599999999997 2020-06-12T02:38:18.079 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.079 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.080 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:18.080 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.251349+0000", 2020-06-12T02:38:18.080 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.008999211 2020-06-12T02:38:18.080 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.080 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.080 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:18.081 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.260348+0000", 2020-06-12T02:38:18.081 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00016373200000000001 2020-06-12T02:38:18.081 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.081 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.081 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:18.082 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.260512+0000", 2020-06-12T02:38:18.082 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4648999999999997e-05 2020-06-12T02:38:18.082 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.082 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.082 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:18.082 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.260577+0000", 2020-06-12T02:38:18.083 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014082301 2020-06-12T02:38:18.083 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:18.083 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:18.083 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:18.083 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.084 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.084 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8129 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e33)", 2020-06-12T02:38:18.084 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.700485+0000", 2020-06-12T02:38:18.084 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 25.205121706, 2020-06-12T02:38:18.084 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01339515, 2020-06-12T02:38:18.084 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:18.085 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:18.085 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:18.085 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:18.085 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:18.085 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8129 2020-06-12T02:38:18.086 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.086 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:18.086 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.086 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:18.086 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.700485+0000", 2020-06-12T02:38:18.086 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:18.087 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.087 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.087 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:18.087 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.700485+0000", 2020-06-12T02:38:18.087 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4323999999999998e-05 2020-06-12T02:38:18.088 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.088 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.088 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:18.088 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.700529+0000", 2020-06-12T02:38:18.088 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7100000000000004e-06 2020-06-12T02:38:18.088 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.089 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.089 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:18.089 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.700535+0000", 2020-06-12T02:38:18.089 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1664999999999999e-05 2020-06-12T02:38:18.089 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.090 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.090 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:18.090 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.700547+0000", 2020-06-12T02:38:18.090 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6150000000000001e-06 2020-06-12T02:38:18.090 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.090 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.091 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:18.091 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.700550+0000", 2020-06-12T02:38:18.091 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00049316500000000005 2020-06-12T02:38:18.091 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.091 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.092 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:18.092 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.701043+0000", 2020-06-12T02:38:18.092 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0063999999999997e-05 2020-06-12T02:38:18.092 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.092 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.092 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:18.093 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.701133+0000", 2020-06-12T02:38:18.093 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0085745109999999999 2020-06-12T02:38:18.093 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.093 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.093 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:18.094 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.709708+0000", 2020-06-12T02:38:18.094 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8854999999999999e-05 2020-06-12T02:38:18.094 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.094 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.094 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:18.094 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.709757+0000", 2020-06-12T02:38:18.095 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002613279 2020-06-12T02:38:18.095 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.095 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.095 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:18.095 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.712370+0000", 2020-06-12T02:38:18.096 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4332000000000004e-05 2020-06-12T02:38:18.096 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.096 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.096 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:18.096 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.712464+0000", 2020-06-12T02:38:18.096 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00141563 2020-06-12T02:38:18.097 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.097 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.097 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:18.097 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.713880+0000", 2020-06-12T02:38:18.097 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01339515 2020-06-12T02:38:18.098 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:18.098 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:18.098 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:18.098 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.098 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.098 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8780 3.as0 3:5732055a:::smithi10729407-8:head [append 0~106496] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:18.099 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:11.606644+0000", 2020-06-12T02:38:18.099 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 6.2989622479999996, 2020-06-12T02:38:18.099 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013846951999999999, 2020-06-12T02:38:18.099 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:18.099 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:18.100 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:18.100 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:18.100 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:18.100 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8780 2020-06-12T02:38:18.100 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.100 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:18.101 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.101 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:18.101 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.606644+0000", 2020-06-12T02:38:18.101 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:18.101 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.102 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.102 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:18.102 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.606644+0000", 2020-06-12T02:38:18.102 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000139424 2020-06-12T02:38:18.102 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.103 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.103 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:18.103 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.606784+0000", 2020-06-12T02:38:18.103 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3067999999999999e-05 2020-06-12T02:38:18.103 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.103 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.104 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:18.104 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.606797+0000", 2020-06-12T02:38:18.104 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1382999999999999e-05 2020-06-12T02:38:18.104 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.104 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.104 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:18.105 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.606808+0000", 2020-06-12T02:38:18.105 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2689999999999996e-06 2020-06-12T02:38:18.105 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.105 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.105 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:18.106 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.606814+0000", 2020-06-12T02:38:18.106 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0016745130000000001 2020-06-12T02:38:18.106 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.106 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.106 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:18.107 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.608489+0000", 2020-06-12T02:38:18.107 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000113917 2020-06-12T02:38:18.107 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.107 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.107 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:18.107 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.608603+0000", 2020-06-12T02:38:18.108 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0072825420000000004 2020-06-12T02:38:18.108 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.108 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.108 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:18.108 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.615885+0000", 2020-06-12T02:38:18.109 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0661999999999995e-05 2020-06-12T02:38:18.109 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.109 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.109 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:18.109 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.615956+0000", 2020-06-12T02:38:18.109 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000395478 2020-06-12T02:38:18.110 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.110 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.110 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:18.110 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.616352+0000", 2020-06-12T02:38:18.110 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00217542 2020-06-12T02:38:18.111 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.111 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.111 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:18.111 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.618527+0000", 2020-06-12T02:38:18.111 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001923157 2020-06-12T02:38:18.111 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.112 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.112 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:18.112 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.620450+0000", 2020-06-12T02:38:18.112 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1118999999999997e-05 2020-06-12T02:38:18.112 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.113 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.113 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:18.113 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.620491+0000", 2020-06-12T02:38:18.113 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013846951999999999 2020-06-12T02:38:18.113 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:18.113 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:18.114 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:18.114 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.114 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.114 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8781 3.as0 3:5732055a:::smithi10729407-8:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:18.114 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:11.606865+0000", 2020-06-12T02:38:18.115 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 6.2987410710000002, 2020-06-12T02:38:18.115 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016125024000000002, 2020-06-12T02:38:18.115 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:18.115 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:18.115 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:18.115 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:18.116 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:18.116 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8781 2020-06-12T02:38:18.116 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.116 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:18.116 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.117 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:18.117 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.606865+0000", 2020-06-12T02:38:18.117 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:18.117 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.117 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.117 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:18.118 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.606865+0000", 2020-06-12T02:38:18.118 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4474000000000006e-05 2020-06-12T02:38:18.118 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.118 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.118 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:18.119 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.606940+0000", 2020-06-12T02:38:18.119 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.579e-06 2020-06-12T02:38:18.119 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.119 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.119 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:18.119 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.606946+0000", 2020-06-12T02:38:18.120 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2563e-05 2020-06-12T02:38:18.120 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.120 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.120 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:18.120 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.606959+0000", 2020-06-12T02:38:18.121 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.792e-06 2020-06-12T02:38:18.121 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.121 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.121 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:18.121 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.606964+0000", 2020-06-12T02:38:18.121 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0016792689999999999 2020-06-12T02:38:18.122 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.122 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.122 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:18.122 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.608643+0000", 2020-06-12T02:38:18.122 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9752000000000001e-05 2020-06-12T02:38:18.123 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.123 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.123 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:18.123 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.608693+0000", 2020-06-12T02:38:18.123 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0087402529999999999 2020-06-12T02:38:18.123 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.124 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.124 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:18.124 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.617433+0000", 2020-06-12T02:38:18.124 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8889999999999999e-05 2020-06-12T02:38:18.124 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.124 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.125 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:18.125 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.617502+0000", 2020-06-12T02:38:18.125 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00029944299999999999 2020-06-12T02:38:18.125 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.125 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.126 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:18.126 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.617801+0000", 2020-06-12T02:38:18.126 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024307959999999998 2020-06-12T02:38:18.126 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.126 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.127 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:18.127 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.620232+0000", 2020-06-12T02:38:18.127 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002703567 2020-06-12T02:38:18.127 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.127 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.127 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:18.128 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.622936+0000", 2020-06-12T02:38:18.128 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4645999999999998e-05 2020-06-12T02:38:18.128 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.128 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.128 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:18.128 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.622990+0000", 2020-06-12T02:38:18.129 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016125024000000002 2020-06-12T02:38:18.129 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:18.129 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:18.129 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:18.129 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.130 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.130 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8782 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e49)", 2020-06-12T02:38:18.130 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:11.607007+0000", 2020-06-12T02:38:18.130 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 6.2985993349999996, 2020-06-12T02:38:18.130 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019139309, 2020-06-12T02:38:18.130 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:18.131 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:18.131 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:18.131 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:18.131 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:18.131 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8782 2020-06-12T02:38:18.132 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.132 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:18.132 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.132 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:18.132 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.607007+0000", 2020-06-12T02:38:18.132 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:18.133 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.133 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.133 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:18.133 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.607007+0000", 2020-06-12T02:38:18.133 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4070000000000003e-05 2020-06-12T02:38:18.134 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.134 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.134 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:18.134 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.607051+0000", 2020-06-12T02:38:18.134 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8749999999999997e-06 2020-06-12T02:38:18.134 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.135 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.135 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:18.135 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.607057+0000", 2020-06-12T02:38:18.135 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3715e-05 2020-06-12T02:38:18.135 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.136 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.136 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:18.136 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.607071+0000", 2020-06-12T02:38:18.136 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3939999999999998e-06 2020-06-12T02:38:18.136 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.137 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.137 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:18.137 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.607075+0000", 2020-06-12T02:38:18.137 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001658678 2020-06-12T02:38:18.137 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.137 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.138 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:18.138 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.608734+0000", 2020-06-12T02:38:18.138 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4022000000000002e-05 2020-06-12T02:38:18.138 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.138 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.139 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:18.139 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.608828+0000", 2020-06-12T02:38:18.139 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0093690579999999996 2020-06-12T02:38:18.139 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.139 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.140 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:18.140 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.618197+0000", 2020-06-12T02:38:18.140 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5519999999999994e-05 2020-06-12T02:38:18.140 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.140 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.140 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:18.141 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.618283+0000", 2020-06-12T02:38:18.141 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.005054205 2020-06-12T02:38:18.141 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.141 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.141 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:18.142 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.623337+0000", 2020-06-12T02:38:18.142 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8203999999999998e-05 2020-06-12T02:38:18.142 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.142 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.142 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:18.142 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.623425+0000", 2020-06-12T02:38:18.143 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0027215680000000002 2020-06-12T02:38:18.143 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.143 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.143 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:18.143 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.626146+0000", 2020-06-12T02:38:18.144 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019139309 2020-06-12T02:38:18.144 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:18.144 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:18.144 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:18.144 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.144 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.145 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:9668 3.as0 3:5cb723d2:::smithi10729407-15:head [append 0~122880] snapc 0=[] ondisk+write+known_if_redirected e54)", 2020-06-12T02:38:18.145 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:16.977947+0000", 2020-06-12T02:38:18.145 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.92765927299999995, 2020-06-12T02:38:18.145 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014098585, 2020-06-12T02:38:18.145 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:18.146 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:18.146 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:18.146 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:18.146 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:18.146 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9668 2020-06-12T02:38:18.147 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.147 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:18.147 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.147 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:18.147 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.977947+0000", 2020-06-12T02:38:18.147 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:18.148 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.148 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.148 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:18.148 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.977947+0000", 2020-06-12T02:38:18.148 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011696699999999999 2020-06-12T02:38:18.148 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.149 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.149 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:18.149 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.978064+0000", 2020-06-12T02:38:18.149 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5039999999999996e-06 2020-06-12T02:38:18.149 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.150 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.150 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:18.150 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.978072+0000", 2020-06-12T02:38:18.150 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1867e-05 2020-06-12T02:38:18.150 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.151 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.151 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:18.151 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.978084+0000", 2020-06-12T02:38:18.151 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6469999999999998e-06 2020-06-12T02:38:18.151 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.152 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.152 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:18.153 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.978091+0000", 2020-06-12T02:38:18.153 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000440016 2020-06-12T02:38:18.153 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.153 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.153 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:18.154 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.978531+0000", 2020-06-12T02:38:18.154 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4606999999999997e-05 2020-06-12T02:38:18.154 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.154 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.154 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:18.155 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.978626+0000", 2020-06-12T02:38:18.155 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0053153219999999999 2020-06-12T02:38:18.155 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.155 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.155 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:18.155 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.983941+0000", 2020-06-12T02:38:18.156 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000107105 2020-06-12T02:38:18.156 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.156 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.156 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:18.156 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.984048+0000", 2020-06-12T02:38:18.157 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00061992099999999995 2020-06-12T02:38:18.157 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.157 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.157 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:18.157 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.984668+0000", 2020-06-12T02:38:18.157 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002674297 2020-06-12T02:38:18.158 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.158 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.158 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:18.158 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.987342+0000", 2020-06-12T02:38:18.158 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00460642 2020-06-12T02:38:18.159 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.159 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.159 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:18.159 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.991949+0000", 2020-06-12T02:38:18.159 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.6911999999999996e-05 2020-06-12T02:38:18.159 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.160 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.160 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:18.160 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.992046+0000", 2020-06-12T02:38:18.160 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014098585 2020-06-12T02:38:18.160 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:18.161 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:18.161 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:18.161 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.161 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.161 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:9669 3.as0 3:5cb723d2:::smithi10729407-15:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e54)", 2020-06-12T02:38:18.161 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:16.978138+0000", 2020-06-12T02:38:18.162 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.92746837299999996, 2020-06-12T02:38:18.162 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014120600000000001, 2020-06-12T02:38:18.162 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:18.162 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:18.162 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:18.163 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:18.163 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:18.163 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9669 2020-06-12T02:38:18.163 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.163 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:18.163 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.164 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:18.164 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.978138+0000", 2020-06-12T02:38:18.164 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:18.164 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.164 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.165 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:18.165 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.978138+0000", 2020-06-12T02:38:18.165 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6625000000000006e-05 2020-06-12T02:38:18.165 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.165 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.165 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:18.166 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.978215+0000", 2020-06-12T02:38:18.166 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1489999999999999e-06 2020-06-12T02:38:18.166 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.166 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.166 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:18.167 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.978221+0000", 2020-06-12T02:38:18.167 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0340000000000002e-06 2020-06-12T02:38:18.167 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.167 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.167 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:18.167 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.978230+0000", 2020-06-12T02:38:18.168 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.349e-06 2020-06-12T02:38:18.168 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.168 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.168 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:18.168 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.978233+0000", 2020-06-12T02:38:18.169 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00042449000000000002 2020-06-12T02:38:18.169 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.169 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.169 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:18.169 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.978658+0000", 2020-06-12T02:38:18.169 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6187000000000007e-05 2020-06-12T02:38:18.170 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.170 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.170 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:18.170 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.978744+0000", 2020-06-12T02:38:18.170 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.007427955 2020-06-12T02:38:18.171 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.171 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.171 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:18.171 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.986172+0000", 2020-06-12T02:38:18.171 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9823999999999998e-05 2020-06-12T02:38:18.171 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.172 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.172 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:18.172 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.986272+0000", 2020-06-12T02:38:18.172 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00044918900000000001 2020-06-12T02:38:18.172 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.173 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.173 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:18.173 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.986721+0000", 2020-06-12T02:38:18.173 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004752554 2020-06-12T02:38:18.173 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.174 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.174 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:18.174 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.991473+0000", 2020-06-12T02:38:18.174 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00070009399999999996 2020-06-12T02:38:18.174 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.174 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.175 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:18.175 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.992174+0000", 2020-06-12T02:38:18.175 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5149999999999996e-05 2020-06-12T02:38:18.175 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.175 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.176 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:18.176 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.992259+0000", 2020-06-12T02:38:18.176 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014120600000000001 2020-06-12T02:38:18.176 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:18.176 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:18.176 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:18.177 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.177 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.177 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:9670 3.as0 3:5cb723d2:::smithi10729407-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e54)", 2020-06-12T02:38:18.177 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:16.978257+0000", 2020-06-12T02:38:18.177 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.92734972100000002, 2020-06-12T02:38:18.178 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017543729000000001, 2020-06-12T02:38:18.178 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:18.178 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:18.178 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:18.178 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:18.178 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:18.179 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9670 2020-06-12T02:38:18.179 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.179 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:18.179 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.179 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:18.180 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.978257+0000", 2020-06-12T02:38:18.180 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:18.180 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.180 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.180 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:18.181 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.978257+0000", 2020-06-12T02:38:18.181 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2262000000000001e-05 2020-06-12T02:38:18.181 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.181 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.181 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:18.181 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.978279+0000", 2020-06-12T02:38:18.182 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1190000000000001e-06 2020-06-12T02:38:18.182 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.182 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.182 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:18.182 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.978282+0000", 2020-06-12T02:38:18.183 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2090000000000005e-06 2020-06-12T02:38:18.183 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.183 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.183 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:18.183 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.978290+0000", 2020-06-12T02:38:18.183 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3360000000000002e-06 2020-06-12T02:38:18.184 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.184 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.184 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:18.184 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.978293+0000", 2020-06-12T02:38:18.184 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00046790999999999998 2020-06-12T02:38:18.185 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.185 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.185 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:18.185 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.978761+0000", 2020-06-12T02:38:18.185 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2028999999999999e-05 2020-06-12T02:38:18.185 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.186 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.186 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:18.186 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.978813+0000", 2020-06-12T02:38:18.186 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0083788709999999995 2020-06-12T02:38:18.186 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.187 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.187 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:18.187 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.987192+0000", 2020-06-12T02:38:18.187 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8042999999999996e-05 2020-06-12T02:38:18.187 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.187 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.188 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:18.188 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.987290+0000", 2020-06-12T02:38:18.188 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0057722880000000004 2020-06-12T02:38:18.188 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.188 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.189 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:18.189 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.993062+0000", 2020-06-12T02:38:18.189 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000116085 2020-06-12T02:38:18.189 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.189 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.189 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:18.190 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.993178+0000", 2020-06-12T02:38:18.190 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0026225770000000001 2020-06-12T02:38:18.190 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.190 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.190 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:18.191 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.995801+0000", 2020-06-12T02:38:18.191 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017543729000000001 2020-06-12T02:38:18.191 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:18.191 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:18.191 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:18.191 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.192 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.192 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:9884 3.as0 3:5732055a:::smithi10729407-8:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e54)", 2020-06-12T02:38:18.192 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:17.594551+0000", 2020-06-12T02:38:18.192 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.31105519399999998, 2020-06-12T02:38:18.192 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013549509, 2020-06-12T02:38:18.193 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:18.193 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:18.193 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:18.193 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:18.193 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:18.193 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9884 2020-06-12T02:38:18.194 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.194 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:18.194 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.194 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:18.194 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.594551+0000", 2020-06-12T02:38:18.195 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:18.195 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.195 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.195 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:18.195 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.594551+0000", 2020-06-12T02:38:18.195 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3703999999999998e-05 2020-06-12T02:38:18.196 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.196 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.196 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:18.196 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.594625+0000", 2020-06-12T02:38:18.196 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3749999999999997e-06 2020-06-12T02:38:18.197 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.197 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.197 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:18.197 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.594632+0000", 2020-06-12T02:38:18.197 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2921e-05 2020-06-12T02:38:18.197 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.198 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.198 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:18.198 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.594645+0000", 2020-06-12T02:38:18.198 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4709999999999998e-06 2020-06-12T02:38:18.198 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.199 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.199 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:18.199 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.594651+0000", 2020-06-12T02:38:18.199 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0053362790000000002 2020-06-12T02:38:18.199 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.199 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.200 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:18.200 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.599987+0000", 2020-06-12T02:38:18.200 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000140731 2020-06-12T02:38:18.200 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.200 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.201 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:18.201 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.600128+0000", 2020-06-12T02:38:18.201 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0032092879999999998 2020-06-12T02:38:18.201 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.201 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.201 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:18.202 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.603337+0000", 2020-06-12T02:38:18.202 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000104684 2020-06-12T02:38:18.202 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.202 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.202 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:18.203 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.603442+0000", 2020-06-12T02:38:18.203 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047894300000000001 2020-06-12T02:38:18.203 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.203 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.203 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:18.203 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.603921+0000", 2020-06-12T02:38:18.204 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001319172 2020-06-12T02:38:18.204 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.204 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.204 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:18.204 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.605240+0000", 2020-06-12T02:38:18.205 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0027798660000000002 2020-06-12T02:38:18.205 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.205 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.205 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:18.205 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.608020+0000", 2020-06-12T02:38:18.205 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1075000000000006e-05 2020-06-12T02:38:18.206 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.206 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.206 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:18.206 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.608101+0000", 2020-06-12T02:38:18.206 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013549509 2020-06-12T02:38:18.207 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:18.207 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:18.207 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:18.207 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.207 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.207 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:9885 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e54)", 2020-06-12T02:38:18.208 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:17.594695+0000", 2020-06-12T02:38:18.208 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.31091113999999997, 2020-06-12T02:38:18.208 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015441514999999999, 2020-06-12T02:38:18.208 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:18.208 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:18.209 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:18.209 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:18.209 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:18.209 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9885 2020-06-12T02:38:18.209 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.209 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:18.210 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.210 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:18.210 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.594695+0000", 2020-06-12T02:38:18.210 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:18.210 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.211 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.211 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:18.211 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.594695+0000", 2020-06-12T02:38:18.211 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4107000000000003e-05 2020-06-12T02:38:18.211 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.211 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.212 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:18.212 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.594739+0000", 2020-06-12T02:38:18.212 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4849999999999997e-06 2020-06-12T02:38:18.212 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.212 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.213 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:18.213 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.594746+0000", 2020-06-12T02:38:18.213 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2763e-05 2020-06-12T02:38:18.213 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.213 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.213 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:18.214 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.594759+0000", 2020-06-12T02:38:18.214 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0930000000000002e-06 2020-06-12T02:38:18.214 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.214 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.214 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:18.215 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.594764+0000", 2020-06-12T02:38:18.215 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0053941049999999997 2020-06-12T02:38:18.215 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.215 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.215 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:18.216 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.600158+0000", 2020-06-12T02:38:18.216 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010055700000000001 2020-06-12T02:38:18.216 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.217 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.217 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:18.217 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.600258+0000", 2020-06-12T02:38:18.217 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004881898 2020-06-12T02:38:18.217 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.218 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.218 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:18.218 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.605140+0000", 2020-06-12T02:38:18.218 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4969000000000005e-05 2020-06-12T02:38:18.218 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.218 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.219 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:18.219 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.605235+0000", 2020-06-12T02:38:18.219 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003487474 2020-06-12T02:38:18.219 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.219 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.220 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:18.220 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.608723+0000", 2020-06-12T02:38:18.220 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000100076 2020-06-12T02:38:18.220 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.220 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.220 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:18.221 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.608823+0000", 2020-06-12T02:38:18.221 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0013139880000000001 2020-06-12T02:38:18.221 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:18.222 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:18.222 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:18.222 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.610137+0000", 2020-06-12T02:38:18.222 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015441514999999999 2020-06-12T02:38:18.222 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:18.222 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:18.223 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:18.223 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:18.223 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:18.223 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:18.241 INFO:tasks.rados.rados.0.smithi107.stdout:4684: done (2 left) 2020-06-12T02:38:18.242 INFO:tasks.rados.rados.0.smithi107.stdout:4685: delete oid 17 current snap is 0 2020-06-12T02:38:18.242 INFO:tasks.rados.rados.0.smithi107.stdout:4682: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:18.242 INFO:tasks.rados.rados.0.smithi107.stdout:4682: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:18.243 INFO:tasks.rados.rados.0.smithi107.stdout:4682: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:18.243 INFO:tasks.rados.rados.0.smithi107.stdout:4685: done (2 left) 2020-06-12T02:38:18.243 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:18.243 INFO:tasks.rados.rados.0.smithi107.stdout:4686: seq_num 4685 ranges {0=466944,466944=704512,1171456=499712} 2020-06-12T02:38:18.243 INFO:tasks.rados.rados.0.smithi107.stdout:4686: writing smithi10729407-7 from 0 to 466944 tid 1 2020-06-12T02:38:18.243 INFO:tasks.rados.rados.0.smithi107.stdout:4686: writing smithi10729407-7 from 466944 to 1171456 tid 2 2020-06-12T02:38:18.244 INFO:tasks.rados.rados.0.smithi107.stdout:4686: writing smithi10729407-7 from 1171456 to 1671168 tid 3 2020-06-12T02:38:18.244 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:18.244 INFO:tasks.rados.rados.0.smithi107.stdout:4687: seq_num 4686 ranges {2891776=483328,3375104=589824,3964928=614400} 2020-06-12T02:38:18.244 INFO:tasks.rados.rados.0.smithi107.stdout:4687: writing smithi10729407-19 from 2891776 to 3375104 tid 1 2020-06-12T02:38:18.244 INFO:tasks.rados.rados.0.smithi107.stdout:4687: writing smithi10729407-19 from 3375104 to 3964928 tid 2 2020-06-12T02:38:18.245 INFO:tasks.rados.rados.0.smithi107.stdout:4687: writing smithi10729407-19 from 3964928 to 4579328 tid 3 2020-06-12T02:38:18.245 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:18.245 INFO:tasks.rados.rados.0.smithi107.stdout:4688: seq_num 4687 ranges {0=507904,507904=475136,983040=573440,1556480=155648} 2020-06-12T02:38:18.245 INFO:tasks.rados.rados.0.smithi107.stdout:4688: writing smithi10729407-2 from 0 to 507904 tid 1 2020-06-12T02:38:18.245 INFO:tasks.rados.rados.0.smithi107.stdout:4688: writing smithi10729407-2 from 507904 to 983040 tid 2 2020-06-12T02:38:18.246 INFO:tasks.rados.rados.0.smithi107.stdout:4688: writing smithi10729407-2 from 983040 to 1556480 tid 3 2020-06-12T02:38:18.246 INFO:tasks.rados.rados.0.smithi107.stdout:4688: writing smithi10729407-2 from 1556480 to 1712128 tid 4 2020-06-12T02:38:18.246 INFO:tasks.rados.rados.0.smithi107.stdout:4689: delete oid 3 current snap is 0 2020-06-12T02:38:18.246 INFO:tasks.rados.rados.0.smithi107.stdout:4682: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:18.246 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 835 (ObjNum 4681 snap 0 seq_num 4681) dirty exists 2020-06-12T02:38:18.246 INFO:tasks.rados.rados.0.smithi107.stdout:4682: left oid 9 (ObjNum 4681 snap 0 seq_num 4681) 2020-06-12T02:38:18.247 INFO:tasks.rados.rados.0.smithi107.stdout:4686: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:18.247 INFO:tasks.rados.rados.0.smithi107.stdout:4686: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:18.247 INFO:tasks.rados.rados.0.smithi107.stdout:4686: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:18.247 INFO:tasks.rados.rados.0.smithi107.stdout:4686: finishing write tid 5 to smithi10729407-7 2020-06-12T02:38:18.247 INFO:tasks.rados.rados.0.smithi107.stdout:4686: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:18.247 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 880 (ObjNum 4685 snap 0 seq_num 4685) dirty exists 2020-06-12T02:38:18.248 INFO:tasks.rados.rados.0.smithi107.stdout:4686: left oid 7 (ObjNum 4685 snap 0 seq_num 4685) 2020-06-12T02:38:18.248 INFO:tasks.rados.rados.0.smithi107.stdout:4687: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:18.248 INFO:tasks.rados.rados.0.smithi107.stdout:4687: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:18.248 INFO:tasks.rados.rados.0.smithi107.stdout:4687: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:18.248 INFO:tasks.rados.rados.0.smithi107.stdout:4687: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:18.249 INFO:tasks.rados.rados.0.smithi107.stdout:4682: done (5 left) 2020-06-12T02:38:18.249 INFO:tasks.rados.rados.0.smithi107.stdout:4686: done (4 left) 2020-06-12T02:38:18.249 INFO:tasks.rados.rados.0.smithi107.stdout:4689: done (3 left) 2020-06-12T02:38:18.249 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:18.249 INFO:tasks.rados.rados.0.smithi107.stdout:4690: seq_num 4689 ranges {0=540672,540672=638976,1179648=565248} 2020-06-12T02:38:18.249 INFO:tasks.rados.rados.0.smithi107.stdout:4690: writing smithi10729407-12 from 0 to 540672 tid 1 2020-06-12T02:38:18.255 INFO:tasks.rados.rados.0.smithi107.stdout:4690: writing smithi10729407-12 from 540672 to 1179648 tid 2 2020-06-12T02:38:18.255 INFO:tasks.rados.rados.0.smithi107.stdout:4690: writing smithi10729407-12 from 1179648 to 1744896 tid 3 2020-06-12T02:38:18.255 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:18.255 INFO:tasks.rados.rados.0.smithi107.stdout:4691: seq_num 4690 ranges {0=557056,557056=524288,1081344=679936} 2020-06-12T02:38:18.256 INFO:tasks.rados.rados.0.smithi107.stdout:4691: writing smithi10729407-16 from 0 to 557056 tid 1 2020-06-12T02:38:18.256 INFO:tasks.rados.rados.0.smithi107.stdout:4691: writing smithi10729407-16 from 557056 to 1081344 tid 2 2020-06-12T02:38:18.256 INFO:tasks.rados.rados.0.smithi107.stdout:4691: writing smithi10729407-16 from 1081344 to 1761280 tid 3 2020-06-12T02:38:18.256 INFO:tasks.rados.rados.0.smithi107.stdout:4692: delete oid 11 current snap is 0 2020-06-12T02:38:18.256 INFO:tasks.rados.rados.0.smithi107.stdout:4687: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:18.257 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 811 (ObjNum 4686 snap 0 seq_num 4686) dirty exists 2020-06-12T02:38:18.257 INFO:tasks.rados.rados.0.smithi107.stdout:4687: left oid 19 (ObjNum 4686 snap 0 seq_num 4686) 2020-06-12T02:38:18.257 INFO:tasks.rados.rados.0.smithi107.stdout:4688: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:18.257 INFO:tasks.rados.rados.0.smithi107.stdout:4688: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:18.257 INFO:tasks.rados.rados.0.smithi107.stdout:4688: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:18.257 INFO:tasks.rados.rados.0.smithi107.stdout:4688: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:18.258 INFO:tasks.rados.rados.0.smithi107.stdout:4688: finishing write tid 6 to smithi10729407-2 2020-06-12T02:38:18.258 INFO:tasks.rados.rados.0.smithi107.stdout:4688: finishing write tid 7 to smithi10729407-2 2020-06-12T02:38:18.258 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 470 (ObjNum 4687 snap 0 seq_num 4687) dirty exists 2020-06-12T02:38:18.258 INFO:tasks.rados.rados.0.smithi107.stdout:4688: left oid 2 (ObjNum 4687 snap 0 seq_num 4687) 2020-06-12T02:38:18.258 INFO:tasks.rados.rados.0.smithi107.stdout:4690: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:18.259 INFO:tasks.rados.rados.0.smithi107.stdout:4690: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:18.259 INFO:tasks.rados.rados.0.smithi107.stdout:4690: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:18.259 INFO:tasks.rados.rados.0.smithi107.stdout:4690: finishing write tid 5 to smithi10729407-12 2020-06-12T02:38:18.259 INFO:tasks.rados.rados.0.smithi107.stdout:4690: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:18.259 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 815 (ObjNum 4689 snap 0 seq_num 4689) dirty exists 2020-06-12T02:38:18.259 INFO:tasks.rados.rados.0.smithi107.stdout:4690: left oid 12 (ObjNum 4689 snap 0 seq_num 4689) 2020-06-12T02:38:18.260 INFO:tasks.rados.rados.0.smithi107.stdout:4687: done (5 left) 2020-06-12T02:38:18.260 INFO:tasks.rados.rados.0.smithi107.stdout:4688: done (4 left) 2020-06-12T02:38:18.260 INFO:tasks.rados.rados.0.smithi107.stdout:4690: done (3 left) 2020-06-12T02:38:18.261 INFO:tasks.rados.rados.0.smithi107.stdout:4692: done (2 left) 2020-06-12T02:38:18.261 INFO:tasks.rados.rados.0.smithi107.stdout:4693: delete oid 12 current snap is 0 2020-06-12T02:38:18.261 INFO:tasks.rados.rados.0.smithi107.stdout:4691: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:18.261 INFO:tasks.rados.rados.0.smithi107.stdout:4693: done (2 left) 2020-06-12T02:38:18.262 INFO:tasks.rados.rados.0.smithi107.stdout:4694: delete oid 6 current snap is 0 2020-06-12T02:38:18.262 INFO:tasks.rados.rados.0.smithi107.stdout:4694: done (2 left) 2020-06-12T02:38:18.262 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:38:18.262 INFO:tasks.rados.rados.0.smithi107.stdout:4695: seq_num 4694 ranges {0=622592,622592=516096,1138688=688128} 2020-06-12T02:38:18.262 INFO:tasks.rados.rados.0.smithi107.stdout:4695: writing smithi10729407-6 from 0 to 622592 tid 1 2020-06-12T02:38:18.262 INFO:tasks.rados.rados.0.smithi107.stdout:4695: writing smithi10729407-6 from 622592 to 1138688 tid 2 2020-06-12T02:38:18.263 INFO:tasks.rados.rados.0.smithi107.stdout:4695: writing smithi10729407-6 from 1138688 to 1826816 tid 3 2020-06-12T02:38:18.263 INFO:tasks.rados.rados.0.smithi107.stdout:4696: delete oid 8 current snap is 0 2020-06-12T02:38:18.263 INFO:tasks.rados.rados.0.smithi107.stdout:4691: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:18.263 INFO:tasks.rados.rados.0.smithi107.stdout:4691: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:18.263 INFO:tasks.rados.rados.0.smithi107.stdout:4691: finishing write tid 5 to smithi10729407-16 2020-06-12T02:38:18.264 INFO:tasks.rados.rados.0.smithi107.stdout:4691: finishing write tid 6 to smithi10729407-16 2020-06-12T02:38:18.264 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 843 (ObjNum 4690 snap 0 seq_num 4690) dirty exists 2020-06-12T02:38:18.264 INFO:tasks.rados.rados.0.smithi107.stdout:4691: left oid 16 (ObjNum 4690 snap 0 seq_num 4690) 2020-06-12T02:38:18.264 INFO:tasks.rados.rados.0.smithi107.stdout:4695: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:18.264 INFO:tasks.rados.rados.0.smithi107.stdout:4695: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:18.265 INFO:tasks.rados.rados.0.smithi107.stdout:4695: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:18.265 INFO:tasks.rados.rados.0.smithi107.stdout:4695: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:18.265 INFO:tasks.rados.rados.0.smithi107.stdout:4691: done (3 left) 2020-06-12T02:38:18.265 INFO:tasks.rados.rados.0.smithi107.stdout:4696: done (2 left) 2020-06-12T02:38:18.265 INFO:tasks.rados.rados.0.smithi107.stdout:4697: delete oid 11 current snap is 0 2020-06-12T02:38:18.266 INFO:tasks.rados.rados.0.smithi107.stdout:4695: finishing write tid 6 to smithi10729407-6 2020-06-12T02:38:18.266 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 866 (ObjNum 4694 snap 0 seq_num 4694) dirty exists 2020-06-12T02:38:18.266 INFO:tasks.rados.rados.0.smithi107.stdout:4695: left oid 6 (ObjNum 4694 snap 0 seq_num 4694) 2020-06-12T02:38:18.266 INFO:tasks.rados.rados.0.smithi107.stdout:4695: done (2 left) 2020-06-12T02:38:18.266 INFO:tasks.rados.rados.0.smithi107.stdout:4697: done (1 left) 2020-06-12T02:38:18.266 INFO:tasks.rados.rados.0.smithi107.stdout:4698: delete oid 4 current snap is 0 2020-06-12T02:38:18.267 INFO:tasks.rados.rados.0.smithi107.stdout:4698: done (1 left) 2020-06-12T02:38:18.267 INFO:tasks.rados.rados.0.smithi107.stdout:4699: delete oid 3 current snap is 0 2020-06-12T02:38:18.267 INFO:tasks.rados.rados.0.smithi107.stdout:4699: done (1 left) 2020-06-12T02:38:18.267 INFO:tasks.rados.rados.0.smithi107.stdout:4700: delete oid 6 current snap is 0 2020-06-12T02:38:18.267 INFO:tasks.rados.rados.0.smithi107.stdout:4700: done (1 left) 2020-06-12T02:38:18.268 INFO:tasks.rados.rados.0.smithi107.stdout:4701: delete oid 20 current snap is 0 2020-06-12T02:38:18.268 INFO:tasks.rados.rados.0.smithi107.stdout:4701: done (1 left) 2020-06-12T02:38:18.269 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:38:18.269 INFO:tasks.rados.rados.0.smithi107.stdout:4702: seq_num 4701 ranges {0=737280,737280=548864,1286144=524288,1810432=131072} 2020-06-12T02:38:18.269 INFO:tasks.rados.rados.0.smithi107.stdout:4702: writing smithi10729407-11 from 0 to 737280 tid 1 2020-06-12T02:38:18.269 INFO:tasks.rados.rados.0.smithi107.stdout:4702: writing smithi10729407-11 from 737280 to 1286144 tid 2 2020-06-12T02:38:18.270 INFO:tasks.rados.rados.0.smithi107.stdout:4702: writing smithi10729407-11 from 1286144 to 1810432 tid 3 2020-06-12T02:38:18.270 INFO:tasks.rados.rados.0.smithi107.stdout:4702: writing smithi10729407-11 from 1810432 to 1941504 tid 4 2020-06-12T02:38:18.270 INFO:tasks.rados.rados.0.smithi107.stdout:4703: delete oid 3 current snap is 0 2020-06-12T02:38:18.270 INFO:tasks.rados.rados.0.smithi107.stdout:4702: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:18.270 INFO:tasks.rados.rados.0.smithi107.stdout:4703: done (2 left) 2020-06-12T02:38:18.271 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:18.271 INFO:tasks.rados.rados.0.smithi107.stdout:4704: seq_num 4703 ranges {0=770048,770048=720896,1490944=483328} 2020-06-12T02:38:18.271 INFO:tasks.rados.rados.0.smithi107.stdout:4704: writing smithi10729407-15 from 0 to 770048 tid 1 2020-06-12T02:38:18.271 INFO:tasks.rados.rados.0.smithi107.stdout:4704: writing smithi10729407-15 from 770048 to 1490944 tid 2 2020-06-12T02:38:18.271 INFO:tasks.rados.rados.0.smithi107.stdout:4704: writing smithi10729407-15 from 1490944 to 1974272 tid 3 2020-06-12T02:38:18.272 INFO:tasks.rados.rados.0.smithi107.stdout:4702: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:18.272 INFO:tasks.rados.rados.0.smithi107.stdout:4705: delete oid 20 current snap is 0 2020-06-12T02:38:18.272 INFO:tasks.rados.rados.0.smithi107.stdout:4702: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:18.272 INFO:tasks.rados.rados.0.smithi107.stdout:4702: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:18.272 INFO:tasks.rados.rados.0.smithi107.stdout:4702: finishing write tid 6 to smithi10729407-11 2020-06-12T02:38:18.273 INFO:tasks.rados.rados.0.smithi107.stdout:4702: finishing write tid 7 to smithi10729407-11 2020-06-12T02:38:18.273 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 844 (ObjNum 4701 snap 0 seq_num 4701) dirty exists 2020-06-12T02:38:18.273 INFO:tasks.rados.rados.0.smithi107.stdout:4702: left oid 11 (ObjNum 4701 snap 0 seq_num 4701) 2020-06-12T02:38:18.273 INFO:tasks.rados.rados.0.smithi107.stdout:4704: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:18.273 INFO:tasks.rados.rados.0.smithi107.stdout:4702: done (3 left) 2020-06-12T02:38:18.273 INFO:tasks.rados.rados.0.smithi107.stdout:4705: done (2 left) 2020-06-12T02:38:18.274 INFO:tasks.rados.rados.0.smithi107.stdout:4706: delete oid 13 current snap is 0 2020-06-12T02:38:18.274 INFO:tasks.rados.rados.0.smithi107.stdout:4704: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:18.274 INFO:tasks.rados.rados.0.smithi107.stdout:4704: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:18.274 INFO:tasks.rados.rados.0.smithi107.stdout:4704: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:18.275 INFO:tasks.rados.rados.0.smithi107.stdout:4706: done (2 left) 2020-06-12T02:38:18.275 INFO:tasks.rados.rados.0.smithi107.stdout:4707: delete oid 9 current snap is 0 2020-06-12T02:38:18.275 INFO:tasks.rados.rados.0.smithi107.stdout:4704: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:18.275 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 792 (ObjNum 4703 snap 0 seq_num 4703) dirty exists 2020-06-12T02:38:18.276 INFO:tasks.rados.rados.0.smithi107.stdout:4704: left oid 15 (ObjNum 4703 snap 0 seq_num 4703) 2020-06-12T02:38:18.276 INFO:tasks.rados.rados.0.smithi107.stdout:4704: done (2 left) 2020-06-12T02:38:18.276 INFO:tasks.rados.rados.0.smithi107.stdout:4707: done (1 left) 2020-06-12T02:38:18.276 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:18.276 INFO:tasks.rados.rados.0.smithi107.stdout:4708: seq_num 4707 ranges {0=442368,442368=704512,1146880=475136,1622016=425984} 2020-06-12T02:38:18.277 INFO:tasks.rados.rados.0.smithi107.stdout:4708: writing smithi10729407-4 from 0 to 442368 tid 1 2020-06-12T02:38:18.277 INFO:tasks.rados.rados.0.smithi107.stdout:4708: writing smithi10729407-4 from 442368 to 1146880 tid 2 2020-06-12T02:38:18.277 INFO:tasks.rados.rados.0.smithi107.stdout:4708: writing smithi10729407-4 from 1146880 to 1622016 tid 3 2020-06-12T02:38:18.277 INFO:tasks.rados.rados.0.smithi107.stdout:4708: writing smithi10729407-4 from 1622016 to 2048000 tid 4 2020-06-12T02:38:18.277 INFO:tasks.rados.rados.0.smithi107.stdout:4709: delete oid 13 current snap is 0 2020-06-12T02:38:18.278 INFO:tasks.rados.rados.0.smithi107.stdout:4708: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:18.278 INFO:tasks.rados.rados.0.smithi107.stdout:4709: done (2 left) 2020-06-12T02:38:18.278 INFO:tasks.rados.rados.0.smithi107.stdout:4710: delete oid 11 current snap is 0 2020-06-12T02:38:18.278 INFO:tasks.rados.rados.0.smithi107.stdout:4710: done (2 left) 2020-06-12T02:38:18.278 INFO:tasks.rados.rados.0.smithi107.stdout:4711: delete oid 12 current snap is 0 2020-06-12T02:38:18.278 INFO:tasks.rados.rados.0.smithi107.stdout:4708: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:18.279 INFO:tasks.rados.rados.0.smithi107.stdout:4708: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:18.279 INFO:tasks.rados.rados.0.smithi107.stdout:4708: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:18.279 INFO:tasks.rados.rados.0.smithi107.stdout:4708: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:18.280 INFO:tasks.rados.rados.0.smithi107.stdout:4708: finishing write tid 7 to smithi10729407-4 2020-06-12T02:38:18.280 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 878 (ObjNum 4707 snap 0 seq_num 4707) dirty exists 2020-06-12T02:38:18.280 INFO:tasks.rados.rados.0.smithi107.stdout:4708: left oid 4 (ObjNum 4707 snap 0 seq_num 4707) 2020-06-12T02:38:18.280 INFO:tasks.rados.rados.0.smithi107.stdout:4708: done (2 left) 2020-06-12T02:38:18.280 INFO:tasks.rados.rados.0.smithi107.stdout:4711: done (1 left) 2020-06-12T02:38:18.280 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:18.281 INFO:tasks.rados.rados.0.smithi107.stdout:4712: seq_num 4711 ranges {0=507904,507904=647168,1155072=450560,1605632=425984,2031616=81920} 2020-06-12T02:38:18.281 INFO:tasks.rados.rados.0.smithi107.stdout:4712: writing smithi10729407-9 from 0 to 507904 tid 1 2020-06-12T02:38:18.281 INFO:tasks.rados.rados.0.smithi107.stdout:4712: writing smithi10729407-9 from 507904 to 1155072 tid 2 2020-06-12T02:38:18.281 INFO:tasks.rados.rados.0.smithi107.stdout:4712: writing smithi10729407-9 from 1155072 to 1605632 tid 3 2020-06-12T02:38:18.282 INFO:tasks.rados.rados.0.smithi107.stdout:4712: writing smithi10729407-9 from 1605632 to 2031616 tid 4 2020-06-12T02:38:18.283 INFO:tasks.rados.rados.0.smithi107.stdout:4712: writing smithi10729407-9 from 2031616 to 2113536 tid 5 2020-06-12T02:38:18.283 INFO:tasks.rados.rados.0.smithi107.stdout:4713: delete oid 5 current snap is 0 2020-06-12T02:38:18.283 INFO:tasks.rados.rados.0.smithi107.stdout:4713: done (2 left) 2020-06-12T02:38:18.283 INFO:tasks.rados.rados.0.smithi107.stdout:4714: delete oid 7 current snap is 0 2020-06-12T02:38:18.283 INFO:tasks.rados.rados.0.smithi107.stdout:4712: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:18.284 INFO:tasks.rados.rados.0.smithi107.stdout:4712: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:18.284 INFO:tasks.rados.rados.0.smithi107.stdout:4712: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:18.284 INFO:tasks.rados.rados.0.smithi107.stdout:4712: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:18.284 INFO:tasks.rados.rados.0.smithi107.stdout:4712: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:18.284 INFO:tasks.rados.rados.0.smithi107.stdout:4712: finishing write tid 7 to smithi10729407-9 2020-06-12T02:38:18.285 INFO:tasks.rados.rados.0.smithi107.stdout:4714: done (2 left) 2020-06-12T02:38:18.285 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:18.285 INFO:tasks.rados.rados.0.smithi107.stdout:4715: seq_num 4714 ranges {0=557056,557056=745472,1302528=630784,1933312=229376} 2020-06-12T02:38:18.285 INFO:tasks.rados.rados.0.smithi107.stdout:4715: writing smithi10729407-6 from 0 to 557056 tid 1 2020-06-12T02:38:18.285 INFO:tasks.rados.rados.0.smithi107.stdout:4715: writing smithi10729407-6 from 557056 to 1302528 tid 2 2020-06-12T02:38:18.286 INFO:tasks.rados.rados.0.smithi107.stdout:4715: writing smithi10729407-6 from 1302528 to 1933312 tid 3 2020-06-12T02:38:18.286 INFO:tasks.rados.rados.0.smithi107.stdout:4715: writing smithi10729407-6 from 1933312 to 2162688 tid 4 2020-06-12T02:38:18.286 INFO:tasks.rados.rados.0.smithi107.stdout:4716: delete oid 11 current snap is 0 2020-06-12T02:38:18.286 INFO:tasks.rados.rados.0.smithi107.stdout:4712: finishing write tid 8 to smithi10729407-9 2020-06-12T02:38:18.287 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 842 (ObjNum 4711 snap 0 seq_num 4711) dirty exists 2020-06-12T02:38:18.287 INFO:tasks.rados.rados.0.smithi107.stdout:4712: left oid 9 (ObjNum 4711 snap 0 seq_num 4711) 2020-06-12T02:38:18.287 INFO:tasks.rados.rados.0.smithi107.stdout:4715: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:18.287 INFO:tasks.rados.rados.0.smithi107.stdout:4712: done (3 left) 2020-06-12T02:38:18.287 INFO:tasks.rados.rados.0.smithi107.stdout:4716: done (2 left) 2020-06-12T02:38:18.287 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:18.288 INFO:tasks.rados.rados.0.smithi107.stdout:4717: seq_num 4716 ranges {0=589824,589824=516096,1105920=638976,1744896=450560} 2020-06-12T02:38:18.288 INFO:tasks.rados.rados.0.smithi107.stdout:4717: writing smithi10729407-12 from 0 to 589824 tid 1 2020-06-12T02:38:18.288 INFO:tasks.rados.rados.0.smithi107.stdout:4717: writing smithi10729407-12 from 589824 to 1105920 tid 2 2020-06-12T02:38:18.288 INFO:tasks.rados.rados.0.smithi107.stdout:4717: writing smithi10729407-12 from 1105920 to 1744896 tid 3 2020-06-12T02:38:18.288 INFO:tasks.rados.rados.0.smithi107.stdout:4717: writing smithi10729407-12 from 1744896 to 2195456 tid 4 2020-06-12T02:38:18.289 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:18.289 INFO:tasks.rados.rados.0.smithi107.stdout:4718: seq_num 4717 ranges {1974272=606208,2580480=802816,3383296=450560,3833856=352256} 2020-06-12T02:38:18.289 INFO:tasks.rados.rados.0.smithi107.stdout:4718: writing smithi10729407-15 from 1974272 to 2580480 tid 1 2020-06-12T02:38:18.289 INFO:tasks.rados.rados.0.smithi107.stdout:4718: writing smithi10729407-15 from 2580480 to 3383296 tid 2 2020-06-12T02:38:18.290 INFO:tasks.rados.rados.0.smithi107.stdout:4718: writing smithi10729407-15 from 3383296 to 3833856 tid 3 2020-06-12T02:38:18.290 INFO:tasks.rados.rados.0.smithi107.stdout:4718: writing smithi10729407-15 from 3833856 to 4186112 tid 4 2020-06-12T02:38:18.290 INFO:tasks.rados.rados.0.smithi107.stdout:4719: delete oid 11 current snap is 0 2020-06-12T02:38:18.290 INFO:tasks.rados.rados.0.smithi107.stdout:4715: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:18.290 INFO:tasks.rados.rados.0.smithi107.stdout:4715: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:18.291 INFO:tasks.rados.rados.0.smithi107.stdout:4715: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:18.291 INFO:tasks.rados.rados.0.smithi107.stdout:4715: finishing write tid 6 to smithi10729407-6 2020-06-12T02:38:18.291 INFO:tasks.rados.rados.0.smithi107.stdout:4715: finishing write tid 7 to smithi10729407-6 2020-06-12T02:38:18.291 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 872 (ObjNum 4714 snap 0 seq_num 4714) dirty exists 2020-06-12T02:38:18.291 INFO:tasks.rados.rados.0.smithi107.stdout:4715: left oid 6 (ObjNum 4714 snap 0 seq_num 4714) 2020-06-12T02:38:18.291 INFO:tasks.rados.rados.0.smithi107.stdout:4717: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:18.292 INFO:tasks.rados.rados.0.smithi107.stdout:4717: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:18.292 INFO:tasks.rados.rados.0.smithi107.stdout:4717: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:18.292 INFO:tasks.rados.rados.0.smithi107.stdout:4717: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:18.292 INFO:tasks.rados.rados.0.smithi107.stdout:4717: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:18.292 INFO:tasks.rados.rados.0.smithi107.stdout:4717: finishing write tid 7 to smithi10729407-12 2020-06-12T02:38:18.293 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 822 (ObjNum 4716 snap 0 seq_num 4716) dirty exists 2020-06-12T02:38:18.293 INFO:tasks.rados.rados.0.smithi107.stdout:4717: left oid 12 (ObjNum 4716 snap 0 seq_num 4716) 2020-06-12T02:38:18.293 INFO:tasks.rados.rados.0.smithi107.stdout:4718: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:18.293 INFO:tasks.rados.rados.0.smithi107.stdout:4715: done (4 left) 2020-06-12T02:38:18.293 INFO:tasks.rados.rados.0.smithi107.stdout:4717: done (3 left) 2020-06-12T02:38:18.293 INFO:tasks.rados.rados.0.smithi107.stdout:4719: done (2 left) 2020-06-12T02:38:18.294 INFO:tasks.rados.rados.0.smithi107.stdout:4720: delete oid 9 current snap is 0 2020-06-12T02:38:18.294 INFO:tasks.rados.rados.0.smithi107.stdout:4720: done (2 left) 2020-06-12T02:38:18.294 INFO:tasks.rados.rados.0.smithi107.stdout:4721: delete oid 8 current snap is 0 2020-06-12T02:38:18.294 INFO:tasks.rados.rados.0.smithi107.stdout:4718: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:18.294 INFO:tasks.rados.rados.0.smithi107.stdout:4718: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:18.295 INFO:tasks.rados.rados.0.smithi107.stdout:4718: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:18.295 INFO:tasks.rados.rados.0.smithi107.stdout:4718: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:18.295 INFO:tasks.rados.rados.0.smithi107.stdout:4721: done (2 left) 2020-06-12T02:38:18.295 INFO:tasks.rados.rados.0.smithi107.stdout:4722: delete oid 4 current snap is 0 2020-06-12T02:38:18.295 INFO:tasks.rados.rados.0.smithi107.stdout:4718: finishing write tid 7 to smithi10729407-15 2020-06-12T02:38:18.295 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 797 (ObjNum 4717 snap 0 seq_num 4717) dirty exists 2020-06-12T02:38:18.296 INFO:tasks.rados.rados.0.smithi107.stdout:4718: left oid 15 (ObjNum 4717 snap 0 seq_num 4717) 2020-06-12T02:38:18.296 INFO:tasks.rados.rados.0.smithi107.stdout:4718: done (2 left) 2020-06-12T02:38:18.296 INFO:tasks.rados.rados.0.smithi107.stdout:4722: done (1 left) 2020-06-12T02:38:18.296 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:18.297 INFO:tasks.rados.rados.0.smithi107.stdout:4723: seq_num 4722 ranges {0=688128,688128=671744,1359872=589824,1949696=344064} 2020-06-12T02:38:18.297 INFO:tasks.rados.rados.0.smithi107.stdout:4723: writing smithi10729407-1 from 0 to 688128 tid 1 2020-06-12T02:38:18.297 INFO:tasks.rados.rados.0.smithi107.stdout:4723: writing smithi10729407-1 from 688128 to 1359872 tid 2 2020-06-12T02:38:18.297 INFO:tasks.rados.rados.0.smithi107.stdout:4723: writing smithi10729407-1 from 1359872 to 1949696 tid 3 2020-06-12T02:38:18.297 INFO:tasks.rados.rados.0.smithi107.stdout:4723: writing smithi10729407-1 from 1949696 to 2293760 tid 4 2020-06-12T02:38:18.298 INFO:tasks.rados.rados.0.smithi107.stdout:4724: delete oid 8 current snap is 0 2020-06-12T02:38:18.298 INFO:tasks.rados.rados.0.smithi107.stdout:4723: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:18.298 INFO:tasks.rados.rados.0.smithi107.stdout:4724: done (2 left) 2020-06-12T02:38:18.299 INFO:tasks.rados.rados.0.smithi107.stdout:4725: delete oid 11 current snap is 0 2020-06-12T02:38:18.299 INFO:tasks.rados.rados.0.smithi107.stdout:4725: done (2 left) 2020-06-12T02:38:18.299 INFO:tasks.rados.rados.0.smithi107.stdout:4726: delete oid 11 current snap is 0 2020-06-12T02:38:18.300 INFO:tasks.rados.rados.0.smithi107.stdout:4723: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:18.300 INFO:tasks.rados.rados.0.smithi107.stdout:4726: done (2 left) 2020-06-12T02:38:18.300 INFO:tasks.rados.rados.0.smithi107.stdout:4727: delete oid 4 current snap is 0 2020-06-12T02:38:18.300 INFO:tasks.rados.rados.0.smithi107.stdout:4723: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:18.300 INFO:tasks.rados.rados.0.smithi107.stdout:4727: done (2 left) 2020-06-12T02:38:18.301 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:18.301 INFO:tasks.rados.rados.0.smithi107.stdout:4728: seq_num 4727 ranges {0=778240,778240=491520,1269760=778240,2048000=335872} 2020-06-12T02:38:18.301 INFO:tasks.rados.rados.0.smithi107.stdout:4728: writing smithi10729407-8 from 0 to 778240 tid 1 2020-06-12T02:38:18.301 INFO:tasks.rados.rados.0.smithi107.stdout:4728: writing smithi10729407-8 from 778240 to 1269760 tid 2 2020-06-12T02:38:18.301 INFO:tasks.rados.rados.0.smithi107.stdout:4728: writing smithi10729407-8 from 1269760 to 2048000 tid 3 2020-06-12T02:38:18.301 INFO:tasks.rados.rados.0.smithi107.stdout:4728: writing smithi10729407-8 from 2048000 to 2383872 tid 4 2020-06-12T02:38:18.302 INFO:tasks.rados.rados.0.smithi107.stdout:4729: delete oid 16 current snap is 0 2020-06-12T02:38:18.302 INFO:tasks.rados.rados.0.smithi107.stdout:4723: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:18.302 INFO:tasks.rados.rados.0.smithi107.stdout:4723: finishing write tid 6 to smithi10729407-1 2020-06-12T02:38:18.303 INFO:tasks.rados.rados.0.smithi107.stdout:4723: finishing write tid 7 to smithi10729407-1 2020-06-12T02:38:18.303 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 877 (ObjNum 4722 snap 0 seq_num 4722) dirty exists 2020-06-12T02:38:18.303 INFO:tasks.rados.rados.0.smithi107.stdout:4723: left oid 1 (ObjNum 4722 snap 0 seq_num 4722) 2020-06-12T02:38:18.304 INFO:tasks.rados.rados.0.smithi107.stdout:4723: done (3 left) 2020-06-12T02:38:18.304 INFO:tasks.rados.rados.0.smithi107.stdout:4729: done (2 left) 2020-06-12T02:38:18.304 INFO:tasks.rados.rados.0.smithi107.stdout:4730: delete oid 15 current snap is 0 2020-06-12T02:38:18.304 INFO:tasks.rados.rados.0.smithi107.stdout:4728: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:18.304 INFO:tasks.rados.rados.0.smithi107.stdout:4728: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:18.305 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:38:18.305 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:38:18.305 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:38:18.305 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:38:18.306 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:38:18.306 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:38:18.306 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:38:18.306 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2020-06-12T02:38:18.307 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:38:18.307 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:18.307 INFO:teuthology.orchestra.run.smithi107.stdout: "epoch": 54, 2020-06-12T02:38:18.307 INFO:teuthology.orchestra.run.smithi107.stdout: "fsid": "2f5030a1-3206-43c9-8471-d10429688d36", 2020-06-12T02:38:18.307 INFO:teuthology.orchestra.run.smithi107.stdout: "created": "2020-06-12T02:37:00.578493+0000", 2020-06-12T02:38:18.308 INFO:teuthology.orchestra.run.smithi107.stdout: "modified": "2020-06-12T02:38:16.359885+0000", 2020-06-12T02:38:18.308 INFO:teuthology.orchestra.run.smithi107.stdout: "last_up_change": "2020-06-12T02:37:53.542920+0000", 2020-06-12T02:38:18.308 INFO:teuthology.orchestra.run.smithi107.stdout: "last_in_change": "2020-06-12T02:37:10.705891+0000", 2020-06-12T02:38:18.308 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-06-12T02:38:18.308 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_num": 5806080, 2020-06-12T02:38:18.309 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_set": [ 2020-06-12T02:38:18.309 INFO:teuthology.orchestra.run.smithi107.stdout: "nodeep-scrub", 2020-06-12T02:38:18.309 INFO:teuthology.orchestra.run.smithi107.stdout: "noscrub", 2020-06-12T02:38:18.309 INFO:teuthology.orchestra.run.smithi107.stdout: "pglog_hardlimit", 2020-06-12T02:38:18.309 INFO:teuthology.orchestra.run.smithi107.stdout: "purged_snapdirs", 2020-06-12T02:38:18.310 INFO:teuthology.orchestra.run.smithi107.stdout: "recovery_deletes", 2020-06-12T02:38:18.310 INFO:teuthology.orchestra.run.smithi107.stdout: "sortbitwise" 2020-06-12T02:38:18.310 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2020-06-12T02:38:18.311 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_version": 5, 2020-06-12T02:38:18.311 INFO:teuthology.orchestra.run.smithi107.stdout: "full_ratio": 0.94999998807907104, 2020-06-12T02:38:18.311 INFO:teuthology.orchestra.run.smithi107.stdout: "backfillfull_ratio": 0.89999997615814209, 2020-06-12T02:38:18.311 INFO:teuthology.orchestra.run.smithi107.stdout: "nearfull_ratio": 0.85000002384185791, 2020-06-12T02:38:18.311 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_snapshot": "", 2020-06-12T02:38:18.312 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_max": 3, 2020-06-12T02:38:18.312 INFO:teuthology.orchestra.run.smithi107.stdout: "max_osd": 8, 2020-06-12T02:38:18.312 INFO:teuthology.orchestra.run.smithi107.stdout: "require_min_compat_client": "jewel", 2020-06-12T02:38:18.312 INFO:teuthology.orchestra.run.smithi107.stdout: "min_compat_client": "jewel", 2020-06-12T02:38:18.312 INFO:teuthology.orchestra.run.smithi107.stdout: "require_osd_release": "octopus", 2020-06-12T02:38:18.313 INFO:teuthology.orchestra.run.smithi107.stdout: "pools": [ 2020-06-12T02:38:18.313 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.313 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 1, 2020-06-12T02:38:18.313 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "device_health_metrics", 2020-06-12T02:38:18.313 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2020-06-12T02:37:03.610773+0000", 2020-06-12T02:38:18.314 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2020-06-12T02:38:18.314 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2020-06-12T02:38:18.314 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2020-06-12T02:38:18.314 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 2, 2020-06-12T02:38:18.314 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 1, 2020-06-12T02:38:18.315 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2020-06-12T02:38:18.315 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2020-06-12T02:38:18.315 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2020-06-12T02:38:18.315 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 1, 2020-06-12T02:38:18.316 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 1, 2020-06-12T02:38:18.316 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 1, 2020-06-12T02:38:18.316 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 1, 2020-06-12T02:38:18.317 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 1, 2020-06-12T02:38:18.317 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2020-06-12T02:38:18.317 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2020-06-12T02:38:18.317 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2020-06-12T02:38:18.317 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2020-06-12T02:38:18.318 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2020-06-12T02:38:18.318 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2020-06-12T02:38:18.318 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2020-06-12T02:38:18.318 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.318 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "15", 2020-06-12T02:38:18.319 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2020-06-12T02:38:18.319 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2020-06-12T02:38:18.319 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2020-06-12T02:38:18.319 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2020-06-12T02:38:18.319 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2020-06-12T02:38:18.320 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2020-06-12T02:38:18.320 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2020-06-12T02:38:18.320 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2020-06-12T02:38:18.320 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2020-06-12T02:38:18.321 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2020-06-12T02:38:18.321 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2020-06-12T02:38:18.321 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2020-06-12T02:38:18.321 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2020-06-12T02:38:18.322 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2020-06-12T02:38:18.322 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2020-06-12T02:38:18.322 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2020-06-12T02:38:18.322 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2020-06-12T02:38:18.322 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2020-06-12T02:38:18.323 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-06-12T02:38:18.323 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-06-12T02:38:18.323 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2020-06-12T02:38:18.323 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2020-06-12T02:38:18.323 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2020-06-12T02:38:18.324 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2020-06-12T02:38:18.324 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2020-06-12T02:38:18.324 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2020-06-12T02:38:18.324 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.324 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2020-06-12T02:38:18.325 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2020-06-12T02:38:18.325 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2020-06-12T02:38:18.325 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2020-06-12T02:38:18.325 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2020-06-12T02:38:18.325 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2020-06-12T02:38:18.326 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2020-06-12T02:38:18.326 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2020-06-12T02:38:18.326 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2020-06-12T02:38:18.326 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2020-06-12T02:38:18.326 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2020-06-12T02:38:18.327 INFO:teuthology.orchestra.run.smithi107.stdout: "options": { 2020-06-12T02:38:18.327 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_min": 1 2020-06-12T02:38:18.327 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.327 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2020-06-12T02:38:18.327 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_devicehealth": {} 2020-06-12T02:38:18.328 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:18.328 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.328 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.328 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 2, 2020-06-12T02:38:18.328 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "rbd", 2020-06-12T02:38:18.329 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2020-06-12T02:37:14.831188+0000", 2020-06-12T02:38:18.329 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2020-06-12T02:38:18.329 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2020-06-12T02:38:18.329 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2020-06-12T02:38:18.329 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 2, 2020-06-12T02:38:18.330 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 1, 2020-06-12T02:38:18.330 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2020-06-12T02:38:18.330 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2020-06-12T02:38:18.330 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2020-06-12T02:38:18.331 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 8, 2020-06-12T02:38:18.331 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 8, 2020-06-12T02:38:18.332 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 8, 2020-06-12T02:38:18.332 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 8, 2020-06-12T02:38:18.332 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 8, 2020-06-12T02:38:18.332 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2020-06-12T02:38:18.332 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2020-06-12T02:38:18.333 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2020-06-12T02:38:18.333 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2020-06-12T02:38:18.333 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2020-06-12T02:38:18.333 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2020-06-12T02:38:18.333 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2020-06-12T02:38:18.334 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.334 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "18", 2020-06-12T02:38:18.334 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2020-06-12T02:38:18.334 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2020-06-12T02:38:18.334 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2020-06-12T02:38:18.335 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2020-06-12T02:38:18.335 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2020-06-12T02:38:18.335 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2020-06-12T02:38:18.335 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2020-06-12T02:38:18.335 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2020-06-12T02:38:18.336 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2020-06-12T02:38:18.336 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2020-06-12T02:38:18.336 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2020-06-12T02:38:18.336 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2020-06-12T02:38:18.336 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2020-06-12T02:38:18.337 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2020-06-12T02:38:18.337 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2020-06-12T02:38:18.337 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2020-06-12T02:38:18.337 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2020-06-12T02:38:18.337 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2020-06-12T02:38:18.338 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-06-12T02:38:18.338 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-06-12T02:38:18.338 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2020-06-12T02:38:18.338 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2020-06-12T02:38:18.338 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2020-06-12T02:38:18.339 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2020-06-12T02:38:18.339 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2020-06-12T02:38:18.339 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2020-06-12T02:38:18.339 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.339 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2020-06-12T02:38:18.340 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2020-06-12T02:38:18.340 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2020-06-12T02:38:18.340 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2020-06-12T02:38:18.340 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2020-06-12T02:38:18.340 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2020-06-12T02:38:18.340 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2020-06-12T02:38:18.341 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2020-06-12T02:38:18.341 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2020-06-12T02:38:18.341 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2020-06-12T02:38:18.342 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2020-06-12T02:38:18.342 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2020-06-12T02:38:18.342 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2020-06-12T02:38:18.342 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd": {} 2020-06-12T02:38:18.342 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:18.343 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.343 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.343 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 3, 2020-06-12T02:38:18.344 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "unique_pool_0", 2020-06-12T02:38:18.344 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2020-06-12T02:37:25.893619+0000", 2020-06-12T02:38:18.344 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2020-06-12T02:38:18.344 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2020-06-12T02:38:18.344 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 3, 2020-06-12T02:38:18.345 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2020-06-12T02:38:18.345 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2020-06-12T02:38:18.345 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 1, 2020-06-12T02:38:18.345 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2020-06-12T02:38:18.345 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2020-06-12T02:38:18.346 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 16, 2020-06-12T02:38:18.346 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 16, 2020-06-12T02:38:18.346 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 16, 2020-06-12T02:38:18.346 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 16, 2020-06-12T02:38:18.346 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 16, 2020-06-12T02:38:18.347 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2020-06-12T02:38:18.347 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2020-06-12T02:38:18.347 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2020-06-12T02:38:18.347 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2020-06-12T02:38:18.347 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2020-06-12T02:38:18.348 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2020-06-12T02:38:18.348 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2020-06-12T02:38:18.348 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.348 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "24", 2020-06-12T02:38:18.348 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2020-06-12T02:38:18.349 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2020-06-12T02:38:18.349 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2020-06-12T02:38:18.349 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2020-06-12T02:38:18.349 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2020-06-12T02:38:18.349 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2020-06-12T02:38:18.350 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2020-06-12T02:38:18.350 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2020-06-12T02:38:18.350 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2020-06-12T02:38:18.350 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2020-06-12T02:38:18.350 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2020-06-12T02:38:18.350 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2020-06-12T02:38:18.351 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2020-06-12T02:38:18.351 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2020-06-12T02:38:18.351 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2020-06-12T02:38:18.351 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2020-06-12T02:38:18.352 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2020-06-12T02:38:18.352 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2020-06-12T02:38:18.352 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-06-12T02:38:18.352 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-06-12T02:38:18.353 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2020-06-12T02:38:18.353 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2020-06-12T02:38:18.353 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2020-06-12T02:38:18.353 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "teuthologyprofile", 2020-06-12T02:38:18.353 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2020-06-12T02:38:18.354 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2020-06-12T02:38:18.354 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.354 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2020-06-12T02:38:18.354 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2020-06-12T02:38:18.354 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2020-06-12T02:38:18.355 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2020-06-12T02:38:18.355 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2020-06-12T02:38:18.355 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2020-06-12T02:38:18.355 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2020-06-12T02:38:18.355 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2020-06-12T02:38:18.356 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 8192, 2020-06-12T02:38:18.356 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2020-06-12T02:38:18.356 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": true, 2020-06-12T02:38:18.356 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2020-06-12T02:38:18.356 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2020-06-12T02:38:18.357 INFO:teuthology.orchestra.run.smithi107.stdout: "rados": {} 2020-06-12T02:38:18.357 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:18.357 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:18.357 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2020-06-12T02:38:18.357 INFO:teuthology.orchestra.run.smithi107.stdout: "osds": [ 2020-06-12T02:38:18.358 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.358 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 0, 2020-06-12T02:38:18.358 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "ad4b4814-83e7-4a2e-9b1f-d10406174f7e", 2020-06-12T02:38:18.358 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2020-06-12T02:38:18.358 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2020-06-12T02:38:18.359 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2020-06-12T02:38:18.359 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2020-06-12T02:38:18.359 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2020-06-12T02:38:18.359 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2020-06-12T02:38:18.359 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 13, 2020-06-12T02:38:18.360 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 53, 2020-06-12T02:38:18.360 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2020-06-12T02:38:18.360 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2020-06-12T02:38:18.360 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2020-06-12T02:38:18.360 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:18.361 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.361 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:18.361 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6816", 2020-06-12T02:38:18.361 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26614 2020-06-12T02:38:18.361 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.362 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.362 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:18.362 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6817", 2020-06-12T02:38:18.363 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26614 2020-06-12T02:38:18.363 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:18.363 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:18.363 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.363 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2020-06-12T02:38:18.364 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:18.364 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.364 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:18.364 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6819", 2020-06-12T02:38:18.365 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26614 2020-06-12T02:38:18.365 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.365 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.365 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:18.365 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6821", 2020-06-12T02:38:18.366 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26614 2020-06-12T02:38:18.366 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:18.366 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:18.366 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.366 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2020-06-12T02:38:18.367 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:18.367 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.367 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:18.367 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6827", 2020-06-12T02:38:18.367 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26614 2020-06-12T02:38:18.368 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.368 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.368 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:18.368 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6829", 2020-06-12T02:38:18.368 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26614 2020-06-12T02:38:18.369 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:18.369 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:18.369 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.369 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2020-06-12T02:38:18.369 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:18.370 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.370 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:18.370 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6823", 2020-06-12T02:38:18.370 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26614 2020-06-12T02:38:18.370 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.371 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.371 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:18.371 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6825", 2020-06-12T02:38:18.371 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26614 2020-06-12T02:38:18.371 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:18.372 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:18.372 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.373 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.107:6817/26614", 2020-06-12T02:38:18.373 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.107:6821/26614", 2020-06-12T02:38:18.373 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.107:6829/26614", 2020-06-12T02:38:18.373 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.107:6825/26614", 2020-06-12T02:38:18.374 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2020-06-12T02:38:18.374 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2020-06-12T02:38:18.374 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2020-06-12T02:38:18.374 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:18.374 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.375 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.375 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 1, 2020-06-12T02:38:18.375 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "046adf0d-b9e9-41ad-a7c9-3841f8b8f790", 2020-06-12T02:38:18.375 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2020-06-12T02:38:18.375 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2020-06-12T02:38:18.376 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 0.1898193359375, 2020-06-12T02:38:18.376 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2020-06-12T02:38:18.376 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2020-06-12T02:38:18.376 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2020-06-12T02:38:18.376 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 13, 2020-06-12T02:38:18.377 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 29, 2020-06-12T02:38:18.377 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2020-06-12T02:38:18.377 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2020-06-12T02:38:18.377 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2020-06-12T02:38:18.377 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:18.378 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.378 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:18.378 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6800", 2020-06-12T02:38:18.378 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26611 2020-06-12T02:38:18.378 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.379 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.379 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:18.379 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6801", 2020-06-12T02:38:18.379 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26611 2020-06-12T02:38:18.380 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:18.380 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:18.380 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.380 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2020-06-12T02:38:18.380 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:18.381 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.381 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:18.381 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6802", 2020-06-12T02:38:18.381 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26611 2020-06-12T02:38:18.381 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.382 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.382 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:18.382 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6803", 2020-06-12T02:38:18.382 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26611 2020-06-12T02:38:18.382 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:18.383 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:18.383 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.383 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2020-06-12T02:38:18.383 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:18.384 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.384 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:18.384 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6806", 2020-06-12T02:38:18.384 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26611 2020-06-12T02:38:18.384 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.385 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.385 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:18.385 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6807", 2020-06-12T02:38:18.385 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26611 2020-06-12T02:38:18.385 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:18.386 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:18.386 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.386 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2020-06-12T02:38:18.386 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:18.386 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.387 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:18.387 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6804", 2020-06-12T02:38:18.387 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26611 2020-06-12T02:38:18.387 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.387 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.388 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:18.388 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6805", 2020-06-12T02:38:18.388 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26611 2020-06-12T02:38:18.388 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:18.388 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:18.389 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.389 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.107:6801/26611", 2020-06-12T02:38:18.389 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.107:6803/26611", 2020-06-12T02:38:18.389 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.107:6807/26611", 2020-06-12T02:38:18.389 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.107:6805/26611", 2020-06-12T02:38:18.390 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2020-06-12T02:38:18.390 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2020-06-12T02:38:18.390 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2020-06-12T02:38:18.390 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:18.390 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.391 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.391 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 2, 2020-06-12T02:38:18.391 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "e15da6b7-dc79-4185-a5f9-0f443d8b52b6", 2020-06-12T02:38:18.391 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2020-06-12T02:38:18.391 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2020-06-12T02:38:18.392 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2020-06-12T02:38:18.392 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2020-06-12T02:38:18.392 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2020-06-12T02:38:18.392 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2020-06-12T02:38:18.392 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 13, 2020-06-12T02:38:18.393 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 52, 2020-06-12T02:38:18.393 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2020-06-12T02:38:18.393 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2020-06-12T02:38:18.394 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2020-06-12T02:38:18.394 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:18.394 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.394 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:18.394 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6818", 2020-06-12T02:38:18.395 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26613 2020-06-12T02:38:18.395 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.395 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.395 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:18.395 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6820", 2020-06-12T02:38:18.395 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26613 2020-06-12T02:38:18.396 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:18.396 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:18.396 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.396 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2020-06-12T02:38:18.396 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:18.397 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.397 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:18.397 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6822", 2020-06-12T02:38:18.397 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26613 2020-06-12T02:38:18.397 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.398 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.398 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:18.398 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6824", 2020-06-12T02:38:18.398 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26613 2020-06-12T02:38:18.398 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:18.399 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:18.399 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.399 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2020-06-12T02:38:18.399 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:18.399 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.400 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:18.400 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6830", 2020-06-12T02:38:18.400 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26613 2020-06-12T02:38:18.401 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.401 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.402 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:18.402 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6831", 2020-06-12T02:38:18.402 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26613 2020-06-12T02:38:18.403 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:18.403 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:18.403 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.403 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2020-06-12T02:38:18.404 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:18.404 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.404 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:18.404 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6826", 2020-06-12T02:38:18.404 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26613 2020-06-12T02:38:18.405 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.405 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.405 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:18.405 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6828", 2020-06-12T02:38:18.405 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26613 2020-06-12T02:38:18.406 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:18.406 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:18.406 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.406 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.107:6820/26613", 2020-06-12T02:38:18.406 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.107:6824/26613", 2020-06-12T02:38:18.407 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.107:6831/26613", 2020-06-12T02:38:18.407 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.107:6828/26613", 2020-06-12T02:38:18.407 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2020-06-12T02:38:18.407 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2020-06-12T02:38:18.407 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2020-06-12T02:38:18.408 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:18.408 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.408 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.408 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 3, 2020-06-12T02:38:18.408 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "843e0dd3-5ce4-46b9-9004-69849048d2bd", 2020-06-12T02:38:18.409 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2020-06-12T02:38:18.409 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2020-06-12T02:38:18.409 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2020-06-12T02:38:18.409 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2020-06-12T02:38:18.409 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2020-06-12T02:38:18.410 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2020-06-12T02:38:18.410 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 13, 2020-06-12T02:38:18.410 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 53, 2020-06-12T02:38:18.410 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2020-06-12T02:38:18.410 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2020-06-12T02:38:18.411 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2020-06-12T02:38:18.411 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:18.411 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.411 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:18.411 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6808", 2020-06-12T02:38:18.411 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26612 2020-06-12T02:38:18.412 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.412 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.412 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:18.412 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6809", 2020-06-12T02:38:18.412 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26612 2020-06-12T02:38:18.413 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:18.413 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:18.413 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.413 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2020-06-12T02:38:18.414 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:18.414 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.414 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:18.414 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6810", 2020-06-12T02:38:18.415 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26612 2020-06-12T02:38:18.415 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.415 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.415 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:18.415 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6811", 2020-06-12T02:38:18.416 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26612 2020-06-12T02:38:18.416 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:18.417 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:18.417 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.417 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2020-06-12T02:38:18.417 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:18.417 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.418 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:18.418 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6814", 2020-06-12T02:38:18.418 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26612 2020-06-12T02:38:18.418 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.418 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.419 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:18.419 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6815", 2020-06-12T02:38:18.419 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26612 2020-06-12T02:38:18.419 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:18.419 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:18.420 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.420 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2020-06-12T02:38:18.420 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:18.420 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.420 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:18.421 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6812", 2020-06-12T02:38:18.421 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26612 2020-06-12T02:38:18.421 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.421 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.421 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:18.422 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6813", 2020-06-12T02:38:18.422 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26612 2020-06-12T02:38:18.422 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:18.422 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:18.422 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.423 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.107:6809/26612", 2020-06-12T02:38:18.423 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.107:6811/26612", 2020-06-12T02:38:18.423 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.107:6815/26612", 2020-06-12T02:38:18.423 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.107:6813/26612", 2020-06-12T02:38:18.423 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2020-06-12T02:38:18.424 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2020-06-12T02:38:18.424 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2020-06-12T02:38:18.424 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:18.424 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.425 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.425 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 4, 2020-06-12T02:38:18.425 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "c066ab09-e031-4b49-9df3-d54cd9f9b580", 2020-06-12T02:38:18.425 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2020-06-12T02:38:18.425 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2020-06-12T02:38:18.426 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2020-06-12T02:38:18.426 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2020-06-12T02:38:18.426 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2020-06-12T02:38:18.426 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2020-06-12T02:38:18.426 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 13, 2020-06-12T02:38:18.427 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 53, 2020-06-12T02:38:18.427 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2020-06-12T02:38:18.427 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2020-06-12T02:38:18.427 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2020-06-12T02:38:18.427 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:18.428 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.428 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:18.428 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6818", 2020-06-12T02:38:18.428 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26685 2020-06-12T02:38:18.428 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.429 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.429 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:18.429 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6820", 2020-06-12T02:38:18.429 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26685 2020-06-12T02:38:18.429 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:18.430 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:18.430 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.430 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2020-06-12T02:38:18.430 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:18.430 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.431 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:18.431 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6822", 2020-06-12T02:38:18.431 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26685 2020-06-12T02:38:18.431 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.431 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.432 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:18.432 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6823", 2020-06-12T02:38:18.432 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26685 2020-06-12T02:38:18.433 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:18.433 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:18.433 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.433 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2020-06-12T02:38:18.433 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:18.434 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.434 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:18.434 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6830", 2020-06-12T02:38:18.435 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26685 2020-06-12T02:38:18.435 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.435 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.435 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:18.435 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6832", 2020-06-12T02:38:18.436 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26685 2020-06-12T02:38:18.436 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:18.436 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:18.436 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.436 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2020-06-12T02:38:18.437 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:18.437 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.437 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:18.437 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6825", 2020-06-12T02:38:18.437 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26685 2020-06-12T02:38:18.438 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.438 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.438 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:18.438 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6828", 2020-06-12T02:38:18.438 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26685 2020-06-12T02:38:18.438 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:18.439 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:18.439 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.439 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.188:6820/26685", 2020-06-12T02:38:18.439 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.188:6823/26685", 2020-06-12T02:38:18.439 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.188:6832/26685", 2020-06-12T02:38:18.440 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.188:6828/26685", 2020-06-12T02:38:18.440 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2020-06-12T02:38:18.440 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2020-06-12T02:38:18.440 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2020-06-12T02:38:18.440 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:18.441 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.441 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.441 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 5, 2020-06-12T02:38:18.441 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "56a24e45-1b1b-4a65-b4b4-57cfab296f65", 2020-06-12T02:38:18.441 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2020-06-12T02:38:18.442 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2020-06-12T02:38:18.442 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2020-06-12T02:38:18.442 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 0, 2020-06-12T02:38:18.442 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2020-06-12T02:38:18.442 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2020-06-12T02:38:18.443 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 13, 2020-06-12T02:38:18.443 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 36, 2020-06-12T02:38:18.443 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2020-06-12T02:38:18.443 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2020-06-12T02:38:18.443 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2020-06-12T02:38:18.444 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:18.444 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.445 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:18.445 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6819", 2020-06-12T02:38:18.445 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26684 2020-06-12T02:38:18.445 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.445 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.446 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:18.446 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6821", 2020-06-12T02:38:18.446 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26684 2020-06-12T02:38:18.446 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:18.446 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:18.447 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.447 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2020-06-12T02:38:18.447 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:18.447 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.447 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:18.448 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6824", 2020-06-12T02:38:18.448 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26684 2020-06-12T02:38:18.448 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.448 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.448 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:18.449 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6826", 2020-06-12T02:38:18.449 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26684 2020-06-12T02:38:18.449 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:18.449 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:18.449 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.450 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2020-06-12T02:38:18.450 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:18.450 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.450 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:18.450 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6831", 2020-06-12T02:38:18.451 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26684 2020-06-12T02:38:18.451 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.451 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.451 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:18.451 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6833", 2020-06-12T02:38:18.452 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26684 2020-06-12T02:38:18.452 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:18.452 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:18.452 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.452 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2020-06-12T02:38:18.453 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:18.453 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.453 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:18.453 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6827", 2020-06-12T02:38:18.453 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26684 2020-06-12T02:38:18.454 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.459 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.460 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:18.460 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6829", 2020-06-12T02:38:18.460 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26684 2020-06-12T02:38:18.460 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:18.460 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:18.461 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.461 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.188:6821/26684", 2020-06-12T02:38:18.461 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.188:6826/26684", 2020-06-12T02:38:18.461 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.188:6833/26684", 2020-06-12T02:38:18.461 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.188:6829/26684", 2020-06-12T02:38:18.462 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2020-06-12T02:38:18.462 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2020-06-12T02:38:18.462 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2020-06-12T02:38:18.462 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:18.463 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.463 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.463 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 6, 2020-06-12T02:38:18.463 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "f207a2a8-b3c7-4fd8-ae90-4c2d7a5c1c38", 2020-06-12T02:38:18.463 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2020-06-12T02:38:18.464 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2020-06-12T02:38:18.464 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2020-06-12T02:38:18.464 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2020-06-12T02:38:18.464 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2020-06-12T02:38:18.465 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2020-06-12T02:38:18.465 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 13, 2020-06-12T02:38:18.465 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 53, 2020-06-12T02:38:18.465 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2020-06-12T02:38:18.466 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2020-06-12T02:38:18.466 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2020-06-12T02:38:18.466 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:18.466 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.466 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:18.467 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6810", 2020-06-12T02:38:18.467 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26682 2020-06-12T02:38:18.467 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.467 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.467 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:18.468 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6811", 2020-06-12T02:38:18.468 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26682 2020-06-12T02:38:18.468 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:18.468 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:18.468 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.468 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2020-06-12T02:38:18.469 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:18.469 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.469 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:18.469 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6812", 2020-06-12T02:38:18.470 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26682 2020-06-12T02:38:18.470 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.470 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.470 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:18.471 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6813", 2020-06-12T02:38:18.471 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26682 2020-06-12T02:38:18.471 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:18.471 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:18.471 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.472 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2020-06-12T02:38:18.472 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:18.472 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.472 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:18.472 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6816", 2020-06-12T02:38:18.473 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26682 2020-06-12T02:38:18.473 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.473 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.473 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:18.474 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6817", 2020-06-12T02:38:18.474 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26682 2020-06-12T02:38:18.474 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:18.474 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:18.474 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.475 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2020-06-12T02:38:18.475 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:18.475 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.475 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:18.475 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6814", 2020-06-12T02:38:18.476 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26682 2020-06-12T02:38:18.476 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.476 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.476 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:18.476 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6815", 2020-06-12T02:38:18.477 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26682 2020-06-12T02:38:18.477 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:18.477 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:18.477 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.477 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.188:6811/26682", 2020-06-12T02:38:18.478 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.188:6813/26682", 2020-06-12T02:38:18.478 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.188:6817/26682", 2020-06-12T02:38:18.478 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.188:6815/26682", 2020-06-12T02:38:18.478 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2020-06-12T02:38:18.478 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2020-06-12T02:38:18.479 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2020-06-12T02:38:18.479 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:18.479 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.479 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.479 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 7, 2020-06-12T02:38:18.480 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "4e5cc44f-af17-47c9-973f-8e87d1bd98db", 2020-06-12T02:38:18.480 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2020-06-12T02:38:18.480 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2020-06-12T02:38:18.480 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2020-06-12T02:38:18.481 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2020-06-12T02:38:18.481 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 13, 2020-06-12T02:38:18.481 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 18, 2020-06-12T02:38:18.481 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 34, 2020-06-12T02:38:18.482 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 50, 2020-06-12T02:38:18.482 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 19, 2020-06-12T02:38:18.482 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2020-06-12T02:38:18.482 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2020-06-12T02:38:18.482 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:18.482 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.483 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:18.483 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6802", 2020-06-12T02:38:18.483 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 29046 2020-06-12T02:38:18.483 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.483 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.484 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:18.484 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6803", 2020-06-12T02:38:18.484 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 29046 2020-06-12T02:38:18.484 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:18.484 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:18.485 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.485 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2020-06-12T02:38:18.485 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:18.485 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.485 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:18.486 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6804", 2020-06-12T02:38:18.486 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 29046 2020-06-12T02:38:18.486 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.486 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.486 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:18.487 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6805", 2020-06-12T02:38:18.487 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 29046 2020-06-12T02:38:18.487 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:18.487 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:18.487 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.488 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2020-06-12T02:38:18.488 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:18.488 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.488 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:18.488 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6808", 2020-06-12T02:38:18.489 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 29046 2020-06-12T02:38:18.489 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.489 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.489 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:18.489 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6809", 2020-06-12T02:38:18.489 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 29046 2020-06-12T02:38:18.490 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:18.490 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:18.490 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.491 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2020-06-12T02:38:18.491 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:18.491 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.491 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:18.491 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6806", 2020-06-12T02:38:18.492 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 29046 2020-06-12T02:38:18.492 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.492 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.492 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:18.492 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6807", 2020-06-12T02:38:18.493 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 29046 2020-06-12T02:38:18.493 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:18.493 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:18.493 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.493 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.188:6803/29046", 2020-06-12T02:38:18.494 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.188:6805/29046", 2020-06-12T02:38:18.494 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.188:6809/29046", 2020-06-12T02:38:18.494 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.188:6807/29046", 2020-06-12T02:38:18.494 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2020-06-12T02:38:18.494 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2020-06-12T02:38:18.495 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2020-06-12T02:38:18.495 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:18.495 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:18.495 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2020-06-12T02:38:18.495 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_xinfo": [ 2020-06-12T02:38:18.496 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.496 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 0, 2020-06-12T02:38:18.496 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2020-06-12T02:38:18.496 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2020-06-12T02:38:18.496 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2020-06-12T02:38:18.496 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138292836696063, 2020-06-12T02:38:18.497 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2020-06-12T02:38:18.497 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2020-06-12T02:38:09.980294+0000", 2020-06-12T02:38:18.497 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2020-06-12T02:38:18.497 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.498 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.498 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 1, 2020-06-12T02:38:18.498 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2020-06-12T02:38:18.498 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2020-06-12T02:38:18.498 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2020-06-12T02:38:18.499 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138292836696063, 2020-06-12T02:38:18.499 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2020-06-12T02:38:18.499 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2020-06-12T02:38:09.373534+0000", 2020-06-12T02:38:18.499 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2020-06-12T02:38:18.499 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.499 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.500 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 2, 2020-06-12T02:38:18.500 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2020-06-12T02:38:18.500 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2020-06-12T02:38:18.501 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2020-06-12T02:38:18.501 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138292836696063, 2020-06-12T02:38:18.501 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2020-06-12T02:38:18.501 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2020-06-12T02:38:09.542448+0000", 2020-06-12T02:38:18.501 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2020-06-12T02:38:18.502 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.502 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.502 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 3, 2020-06-12T02:38:18.502 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2020-06-12T02:38:18.502 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2020-06-12T02:38:18.503 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2020-06-12T02:38:18.503 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138292836696063, 2020-06-12T02:38:18.503 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2020-06-12T02:38:18.503 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2020-06-12T02:38:09.528988+0000", 2020-06-12T02:38:18.503 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2020-06-12T02:38:18.504 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.504 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.504 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 4, 2020-06-12T02:38:18.504 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2020-06-12T02:38:18.504 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2020-06-12T02:38:18.505 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2020-06-12T02:38:18.505 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138292836696063, 2020-06-12T02:38:18.505 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2020-06-12T02:38:18.505 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2020-06-12T02:38:08.934675+0000", 2020-06-12T02:38:18.505 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2020-06-12T02:38:18.506 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.506 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.506 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 5, 2020-06-12T02:38:18.506 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2020-06-12T02:38:18.506 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2020-06-12T02:38:18.507 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2020-06-12T02:38:18.507 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138292836696063, 2020-06-12T02:38:18.507 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2020-06-12T02:38:18.507 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2020-06-12T02:38:09.183394+0000", 2020-06-12T02:38:18.507 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2020-06-12T02:38:18.508 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.508 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.508 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 6, 2020-06-12T02:38:18.508 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2020-06-12T02:38:18.508 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2020-06-12T02:38:18.509 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2020-06-12T02:38:18.509 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138292836696063, 2020-06-12T02:38:18.509 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2020-06-12T02:38:18.509 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2020-06-12T02:38:09.953928+0000", 2020-06-12T02:38:18.509 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2020-06-12T02:38:18.510 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.510 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:18.510 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 7, 2020-06-12T02:38:18.510 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2020-06-12T02:38:18.510 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2020-06-12T02:38:18.510 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2020-06-12T02:38:18.511 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138292836696063, 2020-06-12T02:38:18.511 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2020-06-12T02:38:18.511 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2020-06-12T02:38:09.294465+0000", 2020-06-12T02:38:18.512 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 19 2020-06-12T02:38:18.512 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:18.512 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2020-06-12T02:38:18.512 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_upmap": [], 2020-06-12T02:38:18.512 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_upmap_items": [], 2020-06-12T02:38:18.513 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_temp": [], 2020-06-12T02:38:18.513 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_temp": [], 2020-06-12T02:38:18.513 INFO:teuthology.orchestra.run.smithi107.stdout: "blacklist": {}, 2020-06-12T02:38:18.513 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profiles": { 2020-06-12T02:38:18.513 INFO:teuthology.orchestra.run.smithi107.stdout: "default": { 2020-06-12T02:38:18.514 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-failure-domain": "osd", 2020-06-12T02:38:18.514 INFO:teuthology.orchestra.run.smithi107.stdout: "k": "2", 2020-06-12T02:38:18.514 INFO:teuthology.orchestra.run.smithi107.stdout: "m": "1", 2020-06-12T02:38:18.514 INFO:teuthology.orchestra.run.smithi107.stdout: "plugin": "jerasure", 2020-06-12T02:38:18.514 INFO:teuthology.orchestra.run.smithi107.stdout: "ruleset-failure-domain": "osd", 2020-06-12T02:38:18.515 INFO:teuthology.orchestra.run.smithi107.stdout: "technique": "reed_sol_van" 2020-06-12T02:38:18.515 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.515 INFO:teuthology.orchestra.run.smithi107.stdout: "teuthologyprofile": { 2020-06-12T02:38:18.515 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-device-class": "", 2020-06-12T02:38:18.515 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-failure-domain": "osd", 2020-06-12T02:38:18.516 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-root": "default", 2020-06-12T02:38:18.516 INFO:teuthology.orchestra.run.smithi107.stdout: "jerasure-per-chunk-alignment": "false", 2020-06-12T02:38:18.516 INFO:teuthology.orchestra.run.smithi107.stdout: "k": "2", 2020-06-12T02:38:18.516 INFO:teuthology.orchestra.run.smithi107.stdout: "m": "1", 2020-06-12T02:38:18.516 INFO:teuthology.orchestra.run.smithi107.stdout: "plugin": "jerasure", 2020-06-12T02:38:18.517 INFO:teuthology.orchestra.run.smithi107.stdout: "ruleset-failure-domain": "osd", 2020-06-12T02:38:18.517 INFO:teuthology.orchestra.run.smithi107.stdout: "technique": "reed_sol_van", 2020-06-12T02:38:18.517 INFO:teuthology.orchestra.run.smithi107.stdout: "w": "8" 2020-06-12T02:38:18.518 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:18.518 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:18.519 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps_queue": [], 2020-06-12T02:38:18.519 INFO:teuthology.orchestra.run.smithi107.stdout: "new_removed_snaps": [], 2020-06-12T02:38:18.519 INFO:teuthology.orchestra.run.smithi107.stdout: "new_purged_snaps": [], 2020-06-12T02:38:18.520 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_node_flags": {}, 2020-06-12T02:38:18.520 INFO:teuthology.orchestra.run.smithi107.stdout: "device_class_flags": {} 2020-06-12T02:38:18.520 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:18.521 INFO:tasks.thrashosds.thrasher:j is {'epoch': 54, 'fsid': '2f5030a1-3206-43c9-8471-d10429688d36', 'created': '2020-06-12T02:37:00.578493+0000', 'modified': '2020-06-12T02:38:16.359885+0000', 'last_up_change': '2020-06-12T02:37:53.542920+0000', 'last_in_change': '2020-06-12T02:37:10.705891+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'jewel', 'min_compat_client': 'jewel', 'require_osd_release': 'octopus', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-06-12T02:37:03.610773+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-06-12T02:37:14.831188+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'unique_pool_0', 'create_time': '2020-06-12T02:37:25.893619+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 3, 'size': 3, 'min_size': 2, 'crush_rule': 1, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'teuthologyprofile', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': True, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'ad4b4814-83e7-4a2e-9b1f-d10406174f7e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 53, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6816', 'nonce': 26614}, {'type': 'v1', 'addr': '172.21.15.107:6817', 'nonce': 26614}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6819', 'nonce': 26614}, {'type': 'v1', 'addr': '172.21.15.107:6821', 'nonce': 26614}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6827', 'nonce': 26614}, {'type': 'v1', 'addr': '172.21.15.107:6829', 'nonce': 26614}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6823', 'nonce': 26614}, {'type': 'v1', 'addr': '172.21.15.107:6825', 'nonce': 26614}]}, 'public_addr': '172.21.15.107:6817/26614', 'cluster_addr': '172.21.15.107:6821/26614', 'heartbeat_back_addr': '172.21.15.107:6829/26614', 'heartbeat_front_addr': '172.21.15.107:6825/26614', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '046adf0d-b9e9-41ad-a7c9-3841f8b8f790', 'up': 1, 'in': 1, 'weight': 0.1898193359375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 29, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6800', 'nonce': 26611}, {'type': 'v1', 'addr': '172.21.15.107:6801', 'nonce': 26611}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6802', 'nonce': 26611}, {'type': 'v1', 'addr': '172.21.15.107:6803', 'nonce': 26611}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6806', 'nonce': 26611}, {'type': 'v1', 'addr': '172.21.15.107:6807', 'nonce': 26611}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6804', 'nonce': 26611}, {'type': 'v1', 'addr': '172.21.15.107:6805', 'nonce': 26611}]}, 'public_addr': '172.21.15.107:6801/26611', 'cluster_addr': '172.21.15.107:6803/26611', 'heartbeat_back_addr': '172.21.15.107:6807/26611', 'heartbeat_front_addr': '172.21.15.107:6805/26611', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'e15da6b7-dc79-4185-a5f9-0f443d8b52b6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 52, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6818', 'nonce': 26613}, {'type': 'v1', 'addr': '172.21.15.107:6820', 'nonce': 26613}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6822', 'nonce': 26613}, {'type': 'v1', 'addr': '172.21.15.107:6824', 'nonce': 26613}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6830', 'nonce': 26613}, {'type': 'v1', 'addr': '172.21.15.107:6831', 'nonce': 26613}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6826', 'nonce': 26613}, {'type': 'v1', 'addr': '172.21.15.107:6828', 'nonce': 26613}]}, 'public_addr': '172.21.15.107:6820/26613', 'cluster_addr': '172.21.15.107:6824/26613', 'heartbeat_back_addr': '172.21.15.107:6831/26613', 'heartbeat_front_addr': '172.21.15.107:6828/26613', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '843e0dd3-5ce4-46b9-9004-69849048d2bd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 53, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6808', 'nonce': 26612}, {'type': 'v1', 'addr': '172.21.15.107:6809', 'nonce': 26612}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6810', 'nonce': 26612}, {'type': 'v1', 'addr': '172.21.15.107:6811', 'nonce': 26612}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6814', 'nonce': 26612}, {'type': 'v1', 'addr': '172.21.15.107:6815', 'nonce': 26612}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6812', 'nonce': 26612}, {'type': 'v1', 'addr': '172.21.15.107:6813', 'nonce': 26612}]}, 'public_addr': '172.21.15.107:6809/26612', 'cluster_addr': '172.21.15.107:6811/26612', 'heartbeat_back_addr': '172.21.15.107:6815/26612', 'heartbeat_front_addr': '172.21.15.107:6813/26612', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'c066ab09-e031-4b49-9df3-d54cd9f9b580', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 53, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6818', 'nonce': 26685}, {'type': 'v1', 'addr': '172.21.15.188:6820', 'nonce': 26685}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6822', 'nonce': 26685}, {'type': 'v1', 'addr': '172.21.15.188:6823', 'nonce': 26685}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6830', 'nonce': 26685}, {'type': 'v1', 'addr': '172.21.15.188:6832', 'nonce': 26685}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6825', 'nonce': 26685}, {'type': 'v1', 'addr': '172.21.15.188:6828', 'nonce': 26685}]}, 'public_addr': '172.21.15.188:6820/26685', 'cluster_addr': '172.21.15.188:6823/26685', 'heartbeat_back_addr': '172.21.15.188:6832/26685', 'heartbeat_front_addr': '172.21.15.188:6828/26685', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '56a24e45-1b1b-4a65-b4b4-57cfab296f65', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 36, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6819', 'nonce': 26684}, {'type': 'v1', 'addr': '172.21.15.188:6821', 'nonce': 26684}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6824', 'nonce': 26684}, {'type': 'v1', 'addr': '172.21.15.188:6826', 'nonce': 26684}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6831', 'nonce': 26684}, {'type': 'v1', 'addr': '172.21.15.188:6833', 'nonce': 26684}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6827', 'nonce': 26684}, {'type': 'v1', 'addr': '172.21.15.188:6829', 'nonce': 26684}]}, 'public_addr': '172.21.15.188:6821/26684', 'cluster_addr': '172.21.15.188:6826/26684', 'heartbeat_back_addr': '172.21.15.188:6833/26684', 'heartbeat_front_addr': '172.21.15.188:6829/26684', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'f207a2a8-b3c7-4fd8-ae90-4c2d7a5c1c38', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 53, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6810', 'nonce': 26682}, {'type': 'v1', 'addr': '172.21.15.188:6811', 'nonce': 26682}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6812', 'nonce': 26682}, {'type': 'v1', 'addr': '172.21.15.188:6813', 'nonce': 26682}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6816', 'nonce': 26682}, {'type': 'v1', 'addr': '172.21.15.188:6817', 'nonce': 26682}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6814', 'nonce': 26682}, {'type': 'v1', 'addr': '172.21.15.188:6815', 'nonce': 26682}]}, 'public_addr': '172.21.15.188:6811/26682', 'cluster_addr': '172.21.15.188:6813/26682', 'heartbeat_back_addr': '172.21.15.188:6817/26682', 'heartbeat_front_addr': '172.21.15.188:6815/26682', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '4e5cc44f-af17-47c9-973f-8e87d1bd98db', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 18, 'up_from': 34, 'up_thru': 50, 'down_at': 19, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6802', 'nonce': 29046}, {'type': 'v1', 'addr': '172.21.15.188:6803', 'nonce': 29046}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6804', 'nonce': 29046}, {'type': 'v1', 'addr': '172.21.15.188:6805', 'nonce': 29046}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6808', 'nonce': 29046}, {'type': 'v1', 'addr': '172.21.15.188:6809', 'nonce': 29046}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6806', 'nonce': 29046}, {'type': 'v1', 'addr': '172.21.15.188:6807', 'nonce': 29046}]}, 'public_addr': '172.21.15.188:6803/29046', 'cluster_addr': '172.21.15.188:6805/29046', 'heartbeat_back_addr': '172.21.15.188:6809/29046', 'heartbeat_front_addr': '172.21.15.188:6807/29046', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138292836696063, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-06-12T02:38:09.980294+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138292836696063, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-06-12T02:38:09.373534+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138292836696063, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-06-12T02:38:09.542448+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138292836696063, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-06-12T02:38:09.528988+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138292836696063, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-06-12T02:38:08.934675+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138292836696063, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-06-12T02:38:09.183394+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138292836696063, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-06-12T02:38:09.953928+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138292836696063, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-06-12T02:38:09.294465+0000', 'dead_epoch': 19}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_temp': [], 'primary_temp': [], 'blacklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}, 'teuthologyprofile': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}} 2020-06-12T02:38:18.521 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2020-06-12T02:38:18.522 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-06-12T02:38:18.522 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-06-12T02:38:18.522 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-06-12T02:38:18.522 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-06-12T02:38:18.523 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-06-12T02:38:18.523 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-06-12T02:38:18.523 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-06-12T02:38:18.523 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-06-12T02:38:18.524 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-06-12T02:38:18.525 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:18.222+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:18.539 INFO:tasks.rados.rados.0.smithi107.stdout:4728: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:18.539 INFO:tasks.rados.rados.0.smithi107.stdout:4728: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:18.540 INFO:tasks.rados.rados.0.smithi107.stdout:4728: finishing write tid 6 to smithi10729407-8 2020-06-12T02:38:18.540 INFO:tasks.rados.rados.0.smithi107.stdout:4728: finishing write tid 7 to smithi10729407-8 2020-06-12T02:38:18.540 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 804 (ObjNum 4727 snap 0 seq_num 4727) dirty exists 2020-06-12T02:38:18.540 INFO:tasks.rados.rados.0.smithi107.stdout:4728: left oid 8 (ObjNum 4727 snap 0 seq_num 4727) 2020-06-12T02:38:18.540 INFO:tasks.rados.rados.0.smithi107.stdout:4728: done (2 left) 2020-06-12T02:38:18.540 INFO:tasks.rados.rados.0.smithi107.stdout:4730: done (1 left) 2020-06-12T02:38:18.541 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:38:18.541 INFO:tasks.rados.rados.0.smithi107.stdout:4731: seq_num 4730 ranges {2293760=24576} 2020-06-12T02:38:18.541 INFO:tasks.rados.rados.0.smithi107.stdout:4731: writing smithi10729407-1 from 2293760 to 2318336 tid 1 2020-06-12T02:38:18.541 INFO:tasks.rados.rados.0.smithi107.stdout:4732: delete oid 12 current snap is 0 2020-06-12T02:38:18.541 INFO:tasks.rados.rados.0.smithi107.stdout:4732: done (2 left) 2020-06-12T02:38:18.542 INFO:tasks.rados.rados.0.smithi107.stdout:4733: delete oid 3 current snap is 0 2020-06-12T02:38:18.542 INFO:tasks.rados.rados.0.smithi107.stdout:4731: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:18.542 INFO:tasks.rados.rados.0.smithi107.stdout:4731: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:18.542 INFO:tasks.rados.rados.0.smithi107.stdout:4733: done (2 left) 2020-06-12T02:38:18.542 INFO:tasks.rados.rados.0.smithi107.stdout:4734: delete oid 9 current snap is 0 2020-06-12T02:38:18.543 INFO:tasks.rados.rados.0.smithi107.stdout:4731: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:18.543 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 879 (ObjNum 4730 snap 0 seq_num 4730) dirty exists 2020-06-12T02:38:18.543 INFO:tasks.rados.rados.0.smithi107.stdout:4731: left oid 1 (ObjNum 4730 snap 0 seq_num 4730) 2020-06-12T02:38:18.543 INFO:tasks.rados.rados.0.smithi107.stdout:4731: done (2 left) 2020-06-12T02:38:18.543 INFO:tasks.rados.rados.0.smithi107.stdout:4734: done (1 left) 2020-06-12T02:38:18.543 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:38:18.544 INFO:tasks.rados.rados.0.smithi107.stdout:4735: seq_num 4734 ranges {2162688=90112} 2020-06-12T02:38:18.544 INFO:tasks.rados.rados.0.smithi107.stdout:4735: writing smithi10729407-6 from 2162688 to 2252800 tid 1 2020-06-12T02:38:18.544 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:38:18.544 INFO:tasks.rados.rados.0.smithi107.stdout:4736: seq_num 4735 ranges {2318336=106496} 2020-06-12T02:38:18.544 INFO:tasks.rados.rados.0.smithi107.stdout:4736: writing smithi10729407-1 from 2318336 to 2424832 tid 1 2020-06-12T02:38:18.545 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:18.545 INFO:tasks.rados.rados.0.smithi107.stdout:4737: seq_num 4736 ranges {4579328=122880} 2020-06-12T02:38:18.545 INFO:tasks.rados.rados.0.smithi107.stdout:4737: writing smithi10729407-19 from 4579328 to 4702208 tid 1 2020-06-12T02:38:18.545 INFO:tasks.rados.rados.0.smithi107.stdout:4738: delete oid 15 current snap is 0 2020-06-12T02:38:18.545 INFO:tasks.rados.rados.0.smithi107.stdout:4735: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:18.545 INFO:tasks.rados.rados.0.smithi107.stdout:4735: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:18.546 INFO:tasks.rados.rados.0.smithi107.stdout:4735: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:18.546 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 881 (ObjNum 4734 snap 0 seq_num 4734) dirty exists 2020-06-12T02:38:18.546 INFO:tasks.rados.rados.0.smithi107.stdout:4735: left oid 6 (ObjNum 4734 snap 0 seq_num 4734) 2020-06-12T02:38:18.546 INFO:tasks.rados.rados.0.smithi107.stdout:4736: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:18.546 INFO:tasks.rados.rados.0.smithi107.stdout:4736: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:18.547 INFO:tasks.rados.rados.0.smithi107.stdout:4736: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:18.547 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 883 (ObjNum 4735 snap 0 seq_num 4735) dirty exists 2020-06-12T02:38:18.547 INFO:tasks.rados.rados.0.smithi107.stdout:4736: left oid 1 (ObjNum 4735 snap 0 seq_num 4735) 2020-06-12T02:38:18.547 INFO:tasks.rados.rados.0.smithi107.stdout:4735: done (4 left) 2020-06-12T02:38:18.547 INFO:tasks.rados.rados.0.smithi107.stdout:4736: done (3 left) 2020-06-12T02:38:18.547 INFO:tasks.rados.rados.0.smithi107.stdout:4738: done (2 left) 2020-06-12T02:38:18.548 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:18.548 INFO:tasks.rados.rados.0.smithi107.stdout:4739: seq_num 4738 ranges {0=155648} 2020-06-12T02:38:18.548 INFO:tasks.rados.rados.0.smithi107.stdout:4739: writing smithi10729407-13 from 0 to 155648 tid 1 2020-06-12T02:38:18.548 INFO:tasks.rados.rados.0.smithi107.stdout:4740: delete oid 3 current snap is 0 2020-06-12T02:38:18.549 INFO:tasks.rados.rados.0.smithi107.stdout:4737: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:18.549 INFO:tasks.rados.rados.0.smithi107.stdout:4737: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:18.549 INFO:tasks.rados.rados.0.smithi107.stdout:4739: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:18.549 INFO:tasks.rados.rados.0.smithi107.stdout:4740: done (3 left) 2020-06-12T02:38:18.549 INFO:tasks.rados.rados.0.smithi107.stdout:4741: delete oid 4 current snap is 0 2020-06-12T02:38:18.550 INFO:tasks.rados.rados.0.smithi107.stdout:4737: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:18.550 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 825 (ObjNum 4736 snap 0 seq_num 4736) dirty exists 2020-06-12T02:38:18.550 INFO:tasks.rados.rados.0.smithi107.stdout:4737: left oid 19 (ObjNum 4736 snap 0 seq_num 4736) 2020-06-12T02:38:18.550 INFO:tasks.rados.rados.0.smithi107.stdout:4739: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:18.550 INFO:tasks.rados.rados.0.smithi107.stdout:4737: done (3 left) 2020-06-12T02:38:18.551 INFO:tasks.rados.rados.0.smithi107.stdout:4741: done (2 left) 2020-06-12T02:38:18.551 INFO:tasks.rados.rados.0.smithi107.stdout:4742: delete oid 14 current snap is 0 2020-06-12T02:38:18.551 INFO:tasks.rados.rados.0.smithi107.stdout:4739: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:18.551 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 885 (ObjNum 4738 snap 0 seq_num 4738) dirty exists 2020-06-12T02:38:18.551 INFO:tasks.rados.rados.0.smithi107.stdout:4739: left oid 13 (ObjNum 4738 snap 0 seq_num 4738) 2020-06-12T02:38:18.551 INFO:tasks.rados.rados.0.smithi107.stdout:4739: done (2 left) 2020-06-12T02:38:18.552 INFO:tasks.rados.rados.0.smithi107.stdout:4742: done (1 left) 2020-06-12T02:38:18.552 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:18.552 INFO:tasks.rados.rados.0.smithi107.stdout:4743: seq_num 4742 ranges {0=221184} 2020-06-12T02:38:18.552 INFO:tasks.rados.rados.0.smithi107.stdout:4743: writing smithi10729407-14 from 0 to 221184 tid 1 2020-06-12T02:38:18.552 INFO:tasks.rados.rados.0.smithi107.stdout:4744: delete oid 5 current snap is 0 2020-06-12T02:38:18.552 INFO:tasks.rados.rados.0.smithi107.stdout:4744: done (2 left) 2020-06-12T02:38:18.553 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:38:18.553 INFO:tasks.rados.rados.0.smithi107.stdout:4745: seq_num 4744 ranges {0=253952} 2020-06-12T02:38:18.553 INFO:tasks.rados.rados.0.smithi107.stdout:4745: writing smithi10729407-7 from 0 to 253952 tid 1 2020-06-12T02:38:18.553 INFO:tasks.rados.rados.0.smithi107.stdout:4746: delete oid 17 current snap is 0 2020-06-12T02:38:18.553 INFO:tasks.rados.rados.0.smithi107.stdout:4743: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:18.554 INFO:tasks.rados.rados.0.smithi107.stdout:4746: done (3 left) 2020-06-12T02:38:18.554 INFO:tasks.rados.rados.0.smithi107.stdout:4747: delete oid 8 current snap is 0 2020-06-12T02:38:18.554 INFO:tasks.rados.rados.0.smithi107.stdout:4745: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:18.554 INFO:tasks.rados.rados.0.smithi107.stdout:4747: done (3 left) 2020-06-12T02:38:18.554 INFO:tasks.rados.rados.0.smithi107.stdout:4748: delete oid 11 current snap is 0 2020-06-12T02:38:18.555 INFO:tasks.rados.rados.0.smithi107.stdout:4743: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:18.555 INFO:tasks.rados.rados.0.smithi107.stdout:4743: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:18.555 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 852 (ObjNum 4742 snap 0 seq_num 4742) dirty exists 2020-06-12T02:38:18.555 INFO:tasks.rados.rados.0.smithi107.stdout:4743: left oid 14 (ObjNum 4742 snap 0 seq_num 4742) 2020-06-12T02:38:18.555 INFO:tasks.rados.rados.0.smithi107.stdout:4745: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:18.555 INFO:tasks.rados.rados.0.smithi107.stdout:4745: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:18.556 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 887 (ObjNum 4744 snap 0 seq_num 4744) dirty exists 2020-06-12T02:38:18.556 INFO:tasks.rados.rados.0.smithi107.stdout:4745: left oid 7 (ObjNum 4744 snap 0 seq_num 4744) 2020-06-12T02:38:18.556 INFO:tasks.rados.rados.0.smithi107.stdout:4743: done (3 left) 2020-06-12T02:38:18.556 INFO:tasks.rados.rados.0.smithi107.stdout:4745: done (2 left) 2020-06-12T02:38:18.556 INFO:tasks.rados.rados.0.smithi107.stdout:4748: done (1 left) 2020-06-12T02:38:18.557 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:18.557 INFO:tasks.rados.rados.0.smithi107.stdout:4749: seq_num 4748 ranges {0=327680} 2020-06-12T02:38:18.557 INFO:tasks.rados.rados.0.smithi107.stdout:4749: writing smithi10729407-18 from 0 to 327680 tid 1 2020-06-12T02:38:18.557 INFO:tasks.rados.rados.0.smithi107.stdout:4750: delete oid 11 current snap is 0 2020-06-12T02:38:18.557 INFO:tasks.rados.rados.0.smithi107.stdout:4750: done (2 left) 2020-06-12T02:38:18.557 INFO:tasks.rados.rados.0.smithi107.stdout:4751: delete oid 4 current snap is 0 2020-06-12T02:38:18.558 INFO:tasks.rados.rados.0.smithi107.stdout:4749: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:18.558 INFO:tasks.rados.rados.0.smithi107.stdout:4749: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:18.558 INFO:tasks.rados.rados.0.smithi107.stdout:4751: done (2 left) 2020-06-12T02:38:18.558 INFO:tasks.rados.rados.0.smithi107.stdout:4752: delete oid 5 current snap is 0 2020-06-12T02:38:18.558 INFO:tasks.rados.rados.0.smithi107.stdout:4749: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:18.559 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 846 (ObjNum 4748 snap 0 seq_num 4748) dirty exists 2020-06-12T02:38:18.559 INFO:tasks.rados.rados.0.smithi107.stdout:4749: left oid 18 (ObjNum 4748 snap 0 seq_num 4748) 2020-06-12T02:38:18.559 INFO:tasks.rados.rados.0.smithi107.stdout:4749: done (2 left) 2020-06-12T02:38:18.559 INFO:tasks.rados.rados.0.smithi107.stdout:4752: done (1 left) 2020-06-12T02:38:18.559 INFO:tasks.rados.rados.0.smithi107.stdout:4753: delete oid 16 current snap is 0 2020-06-12T02:38:18.559 INFO:tasks.rados.rados.0.smithi107.stdout:4753: done (1 left) 2020-06-12T02:38:18.560 INFO:tasks.rados.rados.0.smithi107.stdout:4754: delete oid 8 current snap is 0 2020-06-12T02:38:18.560 INFO:tasks.rados.rados.0.smithi107.stdout:4754: done (1 left) 2020-06-12T02:38:18.560 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:18.560 INFO:tasks.rados.rados.0.smithi107.stdout:4755: seq_num 4754 ranges {0=425984} 2020-06-12T02:38:18.560 INFO:tasks.rados.rados.0.smithi107.stdout:4755: writing smithi10729407-5 from 0 to 425984 tid 1 2020-06-12T02:38:18.561 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:18.561 INFO:tasks.rados.rados.0.smithi107.stdout:4756: seq_num 4755 ranges {0=442368} 2020-06-12T02:38:18.561 INFO:tasks.rados.rados.0.smithi107.stdout:4756: writing smithi10729407-3 from 0 to 442368 tid 1 2020-06-12T02:38:18.561 INFO:tasks.rados.rados.0.smithi107.stdout:4757: delete oid 9 current snap is 0 2020-06-12T02:38:18.561 INFO:tasks.rados.rados.0.smithi107.stdout:4755: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:18.561 INFO:tasks.rados.rados.0.smithi107.stdout:4757: done (3 left) 2020-06-12T02:38:18.562 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:18.562 INFO:tasks.rados.rados.0.smithi107.stdout:4758: seq_num 4757 ranges {221184=475136} 2020-06-12T02:38:18.562 INFO:tasks.rados.rados.0.smithi107.stdout:4758: writing smithi10729407-14 from 221184 to 696320 tid 1 2020-06-12T02:38:18.562 INFO:tasks.rados.rados.0.smithi107.stdout:4759: delete oid 9 current snap is 0 2020-06-12T02:38:18.562 INFO:tasks.rados.rados.0.smithi107.stdout:4755: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:18.562 INFO:tasks.rados.rados.0.smithi107.stdout:4756: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:18.563 INFO:tasks.rados.rados.0.smithi107.stdout:4755: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:18.563 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 464 (ObjNum 4754 snap 0 seq_num 4754) dirty exists 2020-06-12T02:38:18.563 INFO:tasks.rados.rados.0.smithi107.stdout:4755: left oid 5 (ObjNum 4754 snap 0 seq_num 4754) 2020-06-12T02:38:18.563 INFO:tasks.rados.rados.0.smithi107.stdout:4756: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:18.563 INFO:tasks.rados.rados.0.smithi107.stdout:4756: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:18.564 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 851 (ObjNum 4755 snap 0 seq_num 4755) dirty exists 2020-06-12T02:38:18.564 INFO:tasks.rados.rados.0.smithi107.stdout:4756: left oid 3 (ObjNum 4755 snap 0 seq_num 4755) 2020-06-12T02:38:18.564 INFO:tasks.rados.rados.0.smithi107.stdout:4755: done (4 left) 2020-06-12T02:38:18.564 INFO:tasks.rados.rados.0.smithi107.stdout:4756: done (3 left) 2020-06-12T02:38:18.564 INFO:tasks.rados.rados.0.smithi107.stdout:4759: done (2 left) 2020-06-12T02:38:18.564 INFO:tasks.rados.rados.0.smithi107.stdout:4760: delete oid 4 current snap is 0 2020-06-12T02:38:18.565 INFO:tasks.rados.rados.0.smithi107.stdout:4760: done (2 left) 2020-06-12T02:38:18.565 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:18.565 INFO:tasks.rados.rados.0.smithi107.stdout:4761: seq_num 4760 ranges {253952=524288} 2020-06-12T02:38:18.565 INFO:tasks.rados.rados.0.smithi107.stdout:4761: writing smithi10729407-7 from 253952 to 778240 tid 1 2020-06-12T02:38:18.565 INFO:tasks.rados.rados.0.smithi107.stdout:4762: delete oid 9 current snap is 0 2020-06-12T02:38:18.566 INFO:tasks.rados.rados.0.smithi107.stdout:4758: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:18.566 INFO:tasks.rados.rados.0.smithi107.stdout:4758: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:18.566 INFO:tasks.rados.rados.0.smithi107.stdout:4762: done (3 left) 2020-06-12T02:38:18.566 INFO:tasks.rados.rados.0.smithi107.stdout:4763: delete oid 19 current snap is 0 2020-06-12T02:38:18.566 INFO:tasks.rados.rados.0.smithi107.stdout:4758: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:18.566 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 856 (ObjNum 4757 snap 0 seq_num 4757) dirty exists 2020-06-12T02:38:18.567 INFO:tasks.rados.rados.0.smithi107.stdout:4758: left oid 14 (ObjNum 4757 snap 0 seq_num 4757) 2020-06-12T02:38:18.567 INFO:tasks.rados.rados.0.smithi107.stdout:4758: done (3 left) 2020-06-12T02:38:18.567 INFO:tasks.rados.rados.0.smithi107.stdout:4763: done (2 left) 2020-06-12T02:38:18.567 INFO:tasks.rados.rados.0.smithi107.stdout:4764: delete oid 18 current snap is 0 2020-06-12T02:38:18.567 INFO:tasks.rados.rados.0.smithi107.stdout:4761: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:18.568 INFO:tasks.rados.rados.0.smithi107.stdout:4761: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:18.568 INFO:tasks.rados.rados.0.smithi107.stdout:4761: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:18.568 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 889 (ObjNum 4760 snap 0 seq_num 4760) dirty exists 2020-06-12T02:38:18.568 INFO:tasks.rados.rados.0.smithi107.stdout:4761: left oid 7 (ObjNum 4760 snap 0 seq_num 4760) 2020-06-12T02:38:18.568 INFO:tasks.rados.rados.0.smithi107.stdout:4761: done (2 left) 2020-06-12T02:38:18.569 INFO:tasks.rados.rados.0.smithi107.stdout:4764: done (1 left) 2020-06-12T02:38:18.569 INFO:tasks.rados.rados.0.smithi107.stdout:4765: delete oid 16 current snap is 0 2020-06-12T02:38:18.569 INFO:tasks.rados.rados.0.smithi107.stdout:4765: done (1 left) 2020-06-12T02:38:18.569 INFO:tasks.rados.rados.0.smithi107.stdout:4766: delete oid 2 current snap is 0 2020-06-12T02:38:18.569 INFO:tasks.rados.rados.0.smithi107.stdout:4766: done (1 left) 2020-06-12T02:38:18.570 INFO:tasks.rados.rados.0.smithi107.stdout:4767: delete oid 1 current snap is 0 2020-06-12T02:38:18.570 INFO:tasks.rados.rados.0.smithi107.stdout:4767: done (1 left) 2020-06-12T02:38:18.570 INFO:tasks.rados.rados.0.smithi107.stdout:4768: delete oid 20 current snap is 0 2020-06-12T02:38:18.570 INFO:tasks.rados.rados.0.smithi107.stdout:4768: done (1 left) 2020-06-12T02:38:18.570 INFO:tasks.rados.rados.0.smithi107.stdout:4769: delete oid 20 current snap is 0 2020-06-12T02:38:18.570 INFO:tasks.rados.rados.0.smithi107.stdout:4769: done (1 left) 2020-06-12T02:38:18.571 INFO:tasks.rados.rados.0.smithi107.stdout:4770: delete oid 18 current snap is 0 2020-06-12T02:38:18.571 INFO:tasks.rados.rados.0.smithi107.stdout:4770: done (1 left) 2020-06-12T02:38:18.572 INFO:tasks.rados.rados.0.smithi107.stdout:4771: delete oid 8 current snap is 0 2020-06-12T02:38:18.572 INFO:tasks.rados.rados.0.smithi107.stdout:4771: done (1 left) 2020-06-12T02:38:18.572 INFO:tasks.rados.rados.0.smithi107.stdout:4772: delete oid 14 current snap is 0 2020-06-12T02:38:18.573 INFO:tasks.rados.rados.0.smithi107.stdout:4772: done (1 left) 2020-06-12T02:38:18.573 INFO:tasks.rados.rados.0.smithi107.stdout:4773: delete oid 19 current snap is 0 2020-06-12T02:38:18.573 INFO:tasks.rados.rados.0.smithi107.stdout:4773: done (1 left) 2020-06-12T02:38:18.573 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:18.573 INFO:tasks.rados.rados.0.smithi107.stdout:4774: seq_num 4773 ranges {0=745472} 2020-06-12T02:38:18.574 INFO:tasks.rados.rados.0.smithi107.stdout:4774: writing smithi10729407-19 from 0 to 745472 tid 1 2020-06-12T02:38:18.574 INFO:tasks.rados.rados.0.smithi107.stdout:4775: delete oid 6 current snap is 0 2020-06-12T02:38:18.574 INFO:tasks.rados.rados.0.smithi107.stdout:4775: done (2 left) 2020-06-12T02:38:18.574 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:18.574 INFO:tasks.rados.rados.0.smithi107.stdout:4776: seq_num 4775 ranges {0=778240} 2020-06-12T02:38:18.575 INFO:tasks.rados.rados.0.smithi107.stdout:4776: writing smithi10729407-12 from 0 to 778240 tid 1 2020-06-12T02:38:18.575 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:18.575 INFO:tasks.rados.rados.0.smithi107.stdout:4777: seq_num 4776 ranges {0=794624} 2020-06-12T02:38:18.575 INFO:tasks.rados.rados.0.smithi107.stdout:4777: writing smithi10729407-14 from 0 to 794624 tid 1 2020-06-12T02:38:18.575 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:18.575 INFO:tasks.rados.rados.0.smithi107.stdout:4778: seq_num 4777 ranges {0=409600,409600=401408} 2020-06-12T02:38:18.576 INFO:tasks.rados.rados.0.smithi107.stdout:4778: writing smithi10729407-15 from 0 to 409600 tid 1 2020-06-12T02:38:18.576 INFO:tasks.rados.rados.0.smithi107.stdout:4778: writing smithi10729407-15 from 409600 to 811008 tid 2 2020-06-12T02:38:18.576 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:18.576 INFO:tasks.rados.rados.0.smithi107.stdout:4779: seq_num 4778 ranges {0=425984,425984=401408} 2020-06-12T02:38:18.576 INFO:tasks.rados.rados.0.smithi107.stdout:4779: writing smithi10729407-1 from 0 to 425984 tid 1 2020-06-12T02:38:18.577 INFO:tasks.rados.rados.0.smithi107.stdout:4779: writing smithi10729407-1 from 425984 to 827392 tid 2 2020-06-12T02:38:18.577 INFO:tasks.rados.rados.0.smithi107.stdout:4780: delete oid 4 current snap is 0 2020-06-12T02:38:18.577 INFO:tasks.rados.rados.0.smithi107.stdout:4774: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:18.577 INFO:tasks.rados.rados.0.smithi107.stdout:4774: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:18.577 INFO:tasks.rados.rados.0.smithi107.stdout:4774: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:18.578 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 829 (ObjNum 4773 snap 0 seq_num 4773) dirty exists 2020-06-12T02:38:18.578 INFO:tasks.rados.rados.0.smithi107.stdout:4774: left oid 19 (ObjNum 4773 snap 0 seq_num 4773) 2020-06-12T02:38:18.578 INFO:tasks.rados.rados.0.smithi107.stdout:4776: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:18.578 INFO:tasks.rados.rados.0.smithi107.stdout:4776: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:18.578 INFO:tasks.rados.rados.0.smithi107.stdout:4776: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:18.578 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 831 (ObjNum 4775 snap 0 seq_num 4775) dirty exists 2020-06-12T02:38:18.579 INFO:tasks.rados.rados.0.smithi107.stdout:4776: left oid 12 (ObjNum 4775 snap 0 seq_num 4775) 2020-06-12T02:38:18.579 INFO:tasks.rados.rados.0.smithi107.stdout:4777: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:18.579 INFO:tasks.rados.rados.0.smithi107.stdout:4778: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:18.579 INFO:tasks.rados.rados.0.smithi107.stdout:4779: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:18.579 INFO:tasks.rados.rados.0.smithi107.stdout:4777: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:18.580 INFO:tasks.rados.rados.0.smithi107.stdout:4774: done (6 left) 2020-06-12T02:38:18.580 INFO:tasks.rados.rados.0.smithi107.stdout:4776: done (5 left) 2020-06-12T02:38:18.580 INFO:tasks.rados.rados.0.smithi107.stdout:4780: done (4 left) 2020-06-12T02:38:18.580 INFO:tasks.rados.rados.0.smithi107.stdout:4781: delete oid 17 current snap is 0 2020-06-12T02:38:18.580 INFO:tasks.rados.rados.0.smithi107.stdout:4777: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:18.580 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 859 (ObjNum 4776 snap 0 seq_num 4776) dirty exists 2020-06-12T02:38:18.581 INFO:tasks.rados.rados.0.smithi107.stdout:4777: left oid 14 (ObjNum 4776 snap 0 seq_num 4776) 2020-06-12T02:38:18.581 INFO:tasks.rados.rados.0.smithi107.stdout:4778: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:18.581 INFO:tasks.rados.rados.0.smithi107.stdout:4778: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:18.581 INFO:tasks.rados.rados.0.smithi107.stdout:4777: done (4 left) 2020-06-12T02:38:18.581 INFO:tasks.rados.rados.0.smithi107.stdout:4781: done (3 left) 2020-06-12T02:38:18.582 INFO:tasks.rados.rados.0.smithi107.stdout:4782: delete oid 2 current snap is 0 2020-06-12T02:38:18.582 INFO:tasks.rados.rados.0.smithi107.stdout:4779: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:18.582 INFO:tasks.rados.rados.0.smithi107.stdout:4778: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:18.582 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 812 (ObjNum 4777 snap 0 seq_num 4777) dirty exists 2020-06-12T02:38:18.582 INFO:tasks.rados.rados.0.smithi107.stdout:4778: left oid 15 (ObjNum 4777 snap 0 seq_num 4777) 2020-06-12T02:38:18.583 INFO:tasks.rados.rados.0.smithi107.stdout:4779: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:18.583 INFO:tasks.rados.rados.0.smithi107.stdout:4778: done (3 left) 2020-06-12T02:38:18.583 INFO:tasks.rados.rados.0.smithi107.stdout:4782: done (2 left) 2020-06-12T02:38:18.583 INFO:tasks.rados.rados.0.smithi107.stdout:4783: delete oid 11 current snap is 0 2020-06-12T02:38:18.583 INFO:tasks.rados.rados.0.smithi107.stdout:4779: finishing write tid 5 to smithi10729407-1 2020-06-12T02:38:18.583 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 888 (ObjNum 4778 snap 0 seq_num 4778) dirty exists 2020-06-12T02:38:18.584 INFO:tasks.rados.rados.0.smithi107.stdout:4779: left oid 1 (ObjNum 4778 snap 0 seq_num 4778) 2020-06-12T02:38:18.584 INFO:tasks.rados.rados.0.smithi107.stdout:4779: done (2 left) 2020-06-12T02:38:18.584 INFO:tasks.rados.rados.0.smithi107.stdout:4783: done (1 left) 2020-06-12T02:38:18.584 INFO:tasks.rados.rados.0.smithi107.stdout:4784: delete oid 13 current snap is 0 2020-06-12T02:38:18.584 INFO:tasks.rados.rados.0.smithi107.stdout:4784: done (1 left) 2020-06-12T02:38:18.585 INFO:tasks.rados.rados.0.smithi107.stdout:4785: delete oid 20 current snap is 0 2020-06-12T02:38:18.585 INFO:tasks.rados.rados.0.smithi107.stdout:4785: done (1 left) 2020-06-12T02:38:18.585 INFO:tasks.rados.rados.0.smithi107.stdout:4786: delete oid 14 current snap is 0 2020-06-12T02:38:18.585 INFO:tasks.rados.rados.0.smithi107.stdout:4786: done (1 left) 2020-06-12T02:38:18.585 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:18.585 INFO:tasks.rados.rados.0.smithi107.stdout:4787: seq_num 4786 ranges {778240=565248,1343488=401408} 2020-06-12T02:38:18.586 INFO:tasks.rados.rados.0.smithi107.stdout:4787: writing smithi10729407-7 from 778240 to 1343488 tid 1 2020-06-12T02:38:18.586 INFO:tasks.rados.rados.0.smithi107.stdout:4787: writing smithi10729407-7 from 1343488 to 1744896 tid 2 2020-06-12T02:38:18.586 INFO:tasks.rados.rados.0.smithi107.stdout:4788: delete oid 9 current snap is 0 2020-06-12T02:38:18.586 INFO:tasks.rados.rados.0.smithi107.stdout:4788: done (2 left) 2020-06-12T02:38:18.587 INFO:tasks.rados.rados.0.smithi107.stdout:4789: delete oid 14 current snap is 0 2020-06-12T02:38:18.587 INFO:tasks.rados.rados.0.smithi107.stdout:4789: done (2 left) 2020-06-12T02:38:18.587 INFO:tasks.rados.rados.0.smithi107.stdout:4790: delete oid 11 current snap is 0 2020-06-12T02:38:18.587 INFO:tasks.rados.rados.0.smithi107.stdout:4787: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:18.587 INFO:tasks.rados.rados.0.smithi107.stdout:4787: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:18.587 INFO:tasks.rados.rados.0.smithi107.stdout:4790: done (2 left) 2020-06-12T02:38:18.588 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:18.588 INFO:tasks.rados.rados.0.smithi107.stdout:4791: seq_num 4790 ranges {0=630784,630784=401408} 2020-06-12T02:38:18.588 INFO:tasks.rados.rados.0.smithi107.stdout:4791: writing smithi10729407-6 from 0 to 630784 tid 1 2020-06-12T02:38:18.588 INFO:tasks.rados.rados.0.smithi107.stdout:4791: writing smithi10729407-6 from 630784 to 1032192 tid 2 2020-06-12T02:38:18.588 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:18.588 INFO:tasks.rados.rados.0.smithi107.stdout:4792: seq_num 4791 ranges {778240=647168,1425408=401408} 2020-06-12T02:38:18.589 INFO:tasks.rados.rados.0.smithi107.stdout:4792: writing smithi10729407-12 from 778240 to 1425408 tid 1 2020-06-12T02:38:18.589 INFO:tasks.rados.rados.0.smithi107.stdout:4792: writing smithi10729407-12 from 1425408 to 1826816 tid 2 2020-06-12T02:38:18.589 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:38:18.589 INFO:tasks.rados.rados.0.smithi107.stdout:4793: seq_num 4792 ranges {0=663552,663552=401408} 2020-06-12T02:38:18.589 INFO:tasks.rados.rados.0.smithi107.stdout:4793: writing smithi10729407-2 from 0 to 663552 tid 1 2020-06-12T02:38:18.590 INFO:tasks.rados.rados.0.smithi107.stdout:4793: writing smithi10729407-2 from 663552 to 1064960 tid 2 2020-06-12T02:38:18.590 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:18.590 INFO:tasks.rados.rados.0.smithi107.stdout:4794: seq_num 4793 ranges {0=679936,679936=401408} 2020-06-12T02:38:18.590 INFO:tasks.rados.rados.0.smithi107.stdout:4794: writing smithi10729407-4 from 0 to 679936 tid 1 2020-06-12T02:38:18.590 INFO:tasks.rados.rados.0.smithi107.stdout:4794: writing smithi10729407-4 from 679936 to 1081344 tid 2 2020-06-12T02:38:18.591 INFO:tasks.rados.rados.0.smithi107.stdout:4795: delete oid 3 current snap is 0 2020-06-12T02:38:18.591 INFO:tasks.rados.rados.0.smithi107.stdout:4787: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:18.591 INFO:tasks.rados.rados.0.smithi107.stdout:4787: finishing write tid 5 to smithi10729407-7 2020-06-12T02:38:18.591 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 893 (ObjNum 4786 snap 0 seq_num 4786) dirty exists 2020-06-12T02:38:18.591 INFO:tasks.rados.rados.0.smithi107.stdout:4787: left oid 7 (ObjNum 4786 snap 0 seq_num 4786) 2020-06-12T02:38:18.591 INFO:tasks.rados.rados.0.smithi107.stdout:4791: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:18.592 INFO:tasks.rados.rados.0.smithi107.stdout:4791: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:18.592 INFO:tasks.rados.rados.0.smithi107.stdout:4791: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:18.592 INFO:tasks.rados.rados.0.smithi107.stdout:4791: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:18.592 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 891 (ObjNum 4790 snap 0 seq_num 4790) dirty exists 2020-06-12T02:38:18.592 INFO:tasks.rados.rados.0.smithi107.stdout:4791: left oid 6 (ObjNum 4790 snap 0 seq_num 4790) 2020-06-12T02:38:18.593 INFO:tasks.rados.rados.0.smithi107.stdout:4792: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:18.593 INFO:tasks.rados.rados.0.smithi107.stdout:4793: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:18.593 INFO:tasks.rados.rados.0.smithi107.stdout:4792: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:18.593 INFO:tasks.rados.rados.0.smithi107.stdout:4792: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:18.593 INFO:tasks.rados.rados.0.smithi107.stdout:4792: finishing write tid 5 to smithi10729407-12 2020-06-12T02:38:18.593 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 834 (ObjNum 4791 snap 0 seq_num 4791) dirty exists 2020-06-12T02:38:18.594 INFO:tasks.rados.rados.0.smithi107.stdout:4792: left oid 12 (ObjNum 4791 snap 0 seq_num 4791) 2020-06-12T02:38:18.594 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2020-06-12T02:38:18.595 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:18.523+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:18.597 INFO:tasks.rados.rados.0.smithi107.stdout:4787: done (6 left) 2020-06-12T02:38:18.598 INFO:tasks.rados.rados.0.smithi107.stdout:4791: done (5 left) 2020-06-12T02:38:18.598 INFO:tasks.rados.rados.0.smithi107.stdout:4792: done (4 left) 2020-06-12T02:38:18.598 INFO:tasks.rados.rados.0.smithi107.stdout:4795: done (3 left) 2020-06-12T02:38:18.598 INFO:tasks.rados.rados.0.smithi107.stdout:4796: delete oid 11 current snap is 0 2020-06-12T02:38:18.598 INFO:tasks.rados.rados.0.smithi107.stdout:4796: done (3 left) 2020-06-12T02:38:18.599 INFO:tasks.rados.rados.0.smithi107.stdout:4797: delete oid 20 current snap is 0 2020-06-12T02:38:18.599 INFO:tasks.rados.rados.0.smithi107.stdout:4797: done (3 left) 2020-06-12T02:38:18.599 INFO:tasks.rados.rados.0.smithi107.stdout:4798: delete oid 13 current snap is 0 2020-06-12T02:38:18.599 INFO:tasks.rados.rados.0.smithi107.stdout:4794: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:18.600 INFO:tasks.rados.rados.0.smithi107.stdout:4794: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:18.600 INFO:tasks.rados.rados.0.smithi107.stdout:4793: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:18.600 INFO:tasks.rados.rados.0.smithi107.stdout:4793: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:18.600 INFO:tasks.rados.rados.0.smithi107.stdout:4794: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:18.600 INFO:tasks.rados.rados.0.smithi107.stdout:4793: finishing write tid 5 to smithi10729407-2 2020-06-12T02:38:18.600 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 475 (ObjNum 4792 snap 0 seq_num 4792) dirty exists 2020-06-12T02:38:18.601 INFO:tasks.rados.rados.0.smithi107.stdout:4793: left oid 2 (ObjNum 4792 snap 0 seq_num 4792) 2020-06-12T02:38:18.601 INFO:tasks.rados.rados.0.smithi107.stdout:4793: done (3 left) 2020-06-12T02:38:18.601 INFO:tasks.rados.rados.0.smithi107.stdout:4798: done (2 left) 2020-06-12T02:38:18.601 INFO:tasks.rados.rados.0.smithi107.stdout:4799: delete oid 5 current snap is 0 2020-06-12T02:38:18.601 INFO:tasks.rados.rados.0.smithi107.stdout:4794: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:18.602 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 889 (ObjNum 4793 snap 0 seq_num 4793) dirty exists 2020-06-12T02:38:18.602 INFO:tasks.rados.rados.0.smithi107.stdout:4794: left oid 4 (ObjNum 4793 snap 0 seq_num 4793) 2020-06-12T02:38:18.602 INFO:tasks.rados.rados.0.smithi107.stdout:4794: done (2 left) 2020-06-12T02:38:18.602 INFO:tasks.rados.rados.0.smithi107.stdout:4799: done (1 left) 2020-06-12T02:38:18.602 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:18.602 INFO:tasks.rados.rados.0.smithi107.stdout:4800: seq_num 4799 ranges {0=778240,778240=401408} 2020-06-12T02:38:18.603 INFO:tasks.rados.rados.0.smithi107.stdout:4800: writing smithi10729407-9 from 0 to 778240 tid 1 2020-06-12T02:38:18.603 INFO:tasks.rados.rados.0.smithi107.stdout:4800: writing smithi10729407-9 from 778240 to 1179648 tid 2 2020-06-12T02:38:18.603 INFO:tasks.rados.rados.0.smithi107.stdout:4801: delete oid 17 current snap is 0 2020-06-12T02:38:18.603 INFO:tasks.rados.rados.0.smithi107.stdout:4801: done (2 left) 2020-06-12T02:38:18.603 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:18.604 INFO:tasks.rados.rados.0.smithi107.stdout:4802: seq_num 4801 ranges {1826816=409600,2236416=434176,2670592=368640} 2020-06-12T02:38:18.604 INFO:tasks.rados.rados.0.smithi107.stdout:4802: writing smithi10729407-12 from 1826816 to 2236416 tid 1 2020-06-12T02:38:18.604 INFO:tasks.rados.rados.0.smithi107.stdout:4802: writing smithi10729407-12 from 2236416 to 2670592 tid 2 2020-06-12T02:38:18.604 INFO:tasks.rados.rados.0.smithi107.stdout:4802: writing smithi10729407-12 from 2670592 to 3039232 tid 3 2020-06-12T02:38:18.604 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:18.605 INFO:tasks.rados.rados.0.smithi107.stdout:4803: seq_num 4802 ranges {1081344=425984,1507328=712704,2220032=90112} 2020-06-12T02:38:18.605 INFO:tasks.rados.rados.0.smithi107.stdout:4803: writing smithi10729407-4 from 1081344 to 1507328 tid 1 2020-06-12T02:38:18.605 INFO:tasks.rados.rados.0.smithi107.stdout:4803: writing smithi10729407-4 from 1507328 to 2220032 tid 2 2020-06-12T02:38:18.605 INFO:tasks.rados.rados.0.smithi107.stdout:4803: writing smithi10729407-4 from 2220032 to 2310144 tid 3 2020-06-12T02:38:18.605 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:18.605 INFO:tasks.rados.rados.0.smithi107.stdout:4804: seq_num 4803 ranges {0=450560,450560=598016,1048576=204800} 2020-06-12T02:38:18.606 INFO:tasks.rados.rados.0.smithi107.stdout:4804: writing smithi10729407-18 from 0 to 450560 tid 1 2020-06-12T02:38:18.608 INFO:tasks.rados.rados.0.smithi107.stdout:4804: writing smithi10729407-18 from 450560 to 1048576 tid 2 2020-06-12T02:38:18.608 INFO:tasks.rados.rados.0.smithi107.stdout:4804: writing smithi10729407-18 from 1048576 to 1253376 tid 3 2020-06-12T02:38:18.608 INFO:tasks.rados.rados.0.smithi107.stdout:4805: delete oid 6 current snap is 0 2020-06-12T02:38:18.608 INFO:tasks.rados.rados.0.smithi107.stdout:4800: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:18.609 INFO:tasks.rados.rados.0.smithi107.stdout:4800: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:18.609 INFO:tasks.rados.rados.0.smithi107.stdout:4800: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:18.609 INFO:tasks.rados.rados.0.smithi107.stdout:4800: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:18.609 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 855 (ObjNum 4799 snap 0 seq_num 4799) dirty exists 2020-06-12T02:38:18.609 INFO:tasks.rados.rados.0.smithi107.stdout:4800: left oid 9 (ObjNum 4799 snap 0 seq_num 4799) 2020-06-12T02:38:18.610 INFO:tasks.rados.rados.0.smithi107.stdout:4802: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:18.610 INFO:tasks.rados.rados.0.smithi107.stdout:4802: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:18.610 INFO:tasks.rados.rados.0.smithi107.stdout:4802: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:18.610 INFO:tasks.rados.rados.0.smithi107.stdout:4802: finishing write tid 5 to smithi10729407-12 2020-06-12T02:38:18.610 INFO:tasks.rados.rados.0.smithi107.stdout:4802: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:18.610 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 838 (ObjNum 4801 snap 0 seq_num 4801) dirty exists 2020-06-12T02:38:18.611 INFO:tasks.rados.rados.0.smithi107.stdout:4802: left oid 12 (ObjNum 4801 snap 0 seq_num 4801) 2020-06-12T02:38:18.611 INFO:tasks.rados.rados.0.smithi107.stdout:4803: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:18.611 INFO:tasks.rados.rados.0.smithi107.stdout:4803: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:18.611 INFO:tasks.rados.rados.0.smithi107.stdout:4803: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:18.611 INFO:tasks.rados.rados.0.smithi107.stdout:4803: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:18.612 INFO:tasks.rados.rados.0.smithi107.stdout:4800: done (5 left) 2020-06-12T02:38:18.612 INFO:tasks.rados.rados.0.smithi107.stdout:4802: done (4 left) 2020-06-12T02:38:18.612 INFO:tasks.rados.rados.0.smithi107.stdout:4805: done (3 left) 2020-06-12T02:38:18.612 INFO:tasks.rados.rados.0.smithi107.stdout:4806: delete oid 6 current snap is 0 2020-06-12T02:38:18.612 INFO:tasks.rados.rados.0.smithi107.stdout:4803: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:18.612 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 894 (ObjNum 4802 snap 0 seq_num 4802) dirty exists 2020-06-12T02:38:18.613 INFO:tasks.rados.rados.0.smithi107.stdout:4803: left oid 4 (ObjNum 4802 snap 0 seq_num 4802) 2020-06-12T02:38:18.613 INFO:tasks.rados.rados.0.smithi107.stdout:4804: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:18.613 INFO:tasks.rados.rados.0.smithi107.stdout:4804: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:18.613 INFO:tasks.rados.rados.0.smithi107.stdout:4803: done (3 left) 2020-06-12T02:38:18.613 INFO:tasks.rados.rados.0.smithi107.stdout:4806: done (2 left) 2020-06-12T02:38:18.614 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:18.614 INFO:tasks.rados.rados.0.smithi107.stdout:4807: seq_num 4806 ranges {0=499712,499712=704512,1204224=98304} 2020-06-12T02:38:18.614 INFO:tasks.rados.rados.0.smithi107.stdout:4807: writing smithi10729407-3 from 0 to 499712 tid 1 2020-06-12T02:38:18.615 INFO:tasks.rados.rados.0.smithi107.stdout:4807: writing smithi10729407-3 from 499712 to 1204224 tid 2 2020-06-12T02:38:18.616 INFO:tasks.rados.rados.0.smithi107.stdout:4807: writing smithi10729407-3 from 1204224 to 1302528 tid 3 2020-06-12T02:38:18.616 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:18.616 INFO:tasks.rados.rados.0.smithi107.stdout:4808: seq_num 4807 ranges {811008=516096,1327104=589824,1916928=212992} 2020-06-12T02:38:18.623 INFO:tasks.rados.rados.0.smithi107.stdout:4808: writing smithi10729407-15 from 811008 to 1327104 tid 1 2020-06-12T02:38:18.627 INFO:tasks.rados.rados.0.smithi107.stdout:4808: writing smithi10729407-15 from 1327104 to 1916928 tid 2 2020-06-12T02:38:18.628 INFO:tasks.rados.rados.0.smithi107.stdout:4808: writing smithi10729407-15 from 1916928 to 2129920 tid 3 2020-06-12T02:38:18.629 INFO:tasks.rados.rados.0.smithi107.stdout:4809: delete oid 5 current snap is 0 2020-06-12T02:38:18.629 INFO:tasks.rados.rados.0.smithi107.stdout:4804: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:18.629 INFO:tasks.rados.rados.0.smithi107.stdout:4804: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:18.629 INFO:tasks.rados.rados.0.smithi107.stdout:4804: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:18.629 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 859 (ObjNum 4803 snap 0 seq_num 4803) dirty exists 2020-06-12T02:38:18.630 INFO:tasks.rados.rados.0.smithi107.stdout:4804: left oid 18 (ObjNum 4803 snap 0 seq_num 4803) 2020-06-12T02:38:18.630 INFO:tasks.rados.rados.0.smithi107.stdout:4807: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:18.630 INFO:tasks.rados.rados.0.smithi107.stdout:4807: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:18.630 INFO:tasks.rados.rados.0.smithi107.stdout:4807: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:18.630 INFO:tasks.rados.rados.0.smithi107.stdout:4807: finishing write tid 5 to smithi10729407-3 2020-06-12T02:38:18.631 INFO:tasks.rados.rados.0.smithi107.stdout:4807: finishing write tid 6 to smithi10729407-3 2020-06-12T02:38:18.631 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 857 (ObjNum 4806 snap 0 seq_num 4806) dirty exists 2020-06-12T02:38:18.631 INFO:tasks.rados.rados.0.smithi107.stdout:4807: left oid 3 (ObjNum 4806 snap 0 seq_num 4806) 2020-06-12T02:38:18.632 INFO:tasks.rados.rados.0.smithi107.stdout:4808: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:18.632 INFO:tasks.rados.rados.0.smithi107.stdout:4804: done (4 left) 2020-06-12T02:38:18.632 INFO:tasks.rados.rados.0.smithi107.stdout:4807: done (3 left) 2020-06-12T02:38:18.632 INFO:tasks.rados.rados.0.smithi107.stdout:4809: done (2 left) 2020-06-12T02:38:18.632 INFO:tasks.rados.rados.0.smithi107.stdout:4810: delete oid 4 current snap is 0 2020-06-12T02:38:18.635 INFO:tasks.rados.rados.0.smithi107.stdout:4810: done (2 left) 2020-06-12T02:38:18.635 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:18.636 INFO:tasks.rados.rados.0.smithi107.stdout:4811: seq_num 4810 ranges {0=565248,565248=638976,1204224=163840} 2020-06-12T02:38:18.638 INFO:tasks.rados.rados.0.smithi107.stdout:4811: writing smithi10729407-16 from 0 to 565248 tid 1 2020-06-12T02:38:18.642 INFO:tasks.rados.rados.0.smithi107.stdout:4811: writing smithi10729407-16 from 565248 to 1204224 tid 2 2020-06-12T02:38:18.643 INFO:tasks.rados.rados.0.smithi107.stdout:4811: writing smithi10729407-16 from 1204224 to 1368064 tid 3 2020-06-12T02:38:18.643 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:18.643 INFO:tasks.rados.rados.0.smithi107.stdout:4812: seq_num 4811 ranges {0=581632,581632=524288,1105920=278528} 2020-06-12T02:38:18.647 INFO:tasks.rados.rados.0.smithi107.stdout:4812: writing smithi10729407-8 from 0 to 581632 tid 1 2020-06-12T02:38:18.649 INFO:tasks.rados.rados.0.smithi107.stdout:4812: writing smithi10729407-8 from 581632 to 1105920 tid 2 2020-06-12T02:38:18.651 INFO:tasks.rados.rados.0.smithi107.stdout:4812: writing smithi10729407-8 from 1105920 to 1384448 tid 3 2020-06-12T02:38:18.652 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:18.652 INFO:tasks.rados.rados.0.smithi107.stdout:4813: seq_num 4812 ranges {0=598016,598016=409600,1007616=393216} 2020-06-12T02:38:18.655 INFO:tasks.rados.rados.0.smithi107.stdout:4813: writing smithi10729407-4 from 0 to 598016 tid 1 2020-06-12T02:38:18.655 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:18.653+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:18.658 INFO:tasks.rados.rados.0.smithi107.stdout:4813: writing smithi10729407-4 from 598016 to 1007616 tid 2 2020-06-12T02:38:18.664 INFO:tasks.rados.rados.0.smithi107.stdout:4813: writing smithi10729407-4 from 1007616 to 1400832 tid 3 2020-06-12T02:38:18.664 INFO:tasks.rados.rados.0.smithi107.stdout:4814: delete oid 13 current snap is 0 2020-06-12T02:38:18.665 INFO:tasks.rados.rados.0.smithi107.stdout:4808: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:18.665 INFO:tasks.rados.rados.0.smithi107.stdout:4808: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:18.665 INFO:tasks.rados.rados.0.smithi107.stdout:4808: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:18.665 INFO:tasks.rados.rados.0.smithi107.stdout:4808: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:18.665 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 816 (ObjNum 4807 snap 0 seq_num 4807) dirty exists 2020-06-12T02:38:18.665 INFO:tasks.rados.rados.0.smithi107.stdout:4808: left oid 15 (ObjNum 4807 snap 0 seq_num 4807) 2020-06-12T02:38:18.666 INFO:tasks.rados.rados.0.smithi107.stdout:4811: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:18.666 INFO:tasks.rados.rados.0.smithi107.stdout:4811: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:18.666 INFO:tasks.rados.rados.0.smithi107.stdout:4811: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:18.666 INFO:tasks.rados.rados.0.smithi107.stdout:4812: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:18.666 INFO:tasks.rados.rados.0.smithi107.stdout:4811: finishing write tid 5 to smithi10729407-16 2020-06-12T02:38:18.667 INFO:tasks.rados.rados.0.smithi107.stdout:4811: finishing write tid 6 to smithi10729407-16 2020-06-12T02:38:18.667 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 861 (ObjNum 4810 snap 0 seq_num 4810) dirty exists 2020-06-12T02:38:18.667 INFO:tasks.rados.rados.0.smithi107.stdout:4811: left oid 16 (ObjNum 4810 snap 0 seq_num 4810) 2020-06-12T02:38:18.668 INFO:tasks.rados.rados.0.smithi107.stdout:4808: done (5 left) 2020-06-12T02:38:18.668 INFO:tasks.rados.rados.0.smithi107.stdout:4811: done (4 left) 2020-06-12T02:38:18.668 INFO:tasks.rados.rados.0.smithi107.stdout:4814: done (3 left) 2020-06-12T02:38:18.668 INFO:tasks.rados.rados.0.smithi107.stdout:4815: delete oid 19 current snap is 0 2020-06-12T02:38:18.668 INFO:tasks.rados.rados.0.smithi107.stdout:4812: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:18.669 INFO:tasks.rados.rados.0.smithi107.stdout:4815: done (3 left) 2020-06-12T02:38:18.669 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:38:18.670 INFO:tasks.rados.rados.0.smithi107.stdout:4816: seq_num 4815 ranges {0=647168,647168=507904,1155072=294912} 2020-06-12T02:38:18.674 INFO:tasks.rados.rados.0.smithi107.stdout:4816: writing smithi10729407-20 from 0 to 647168 tid 1 2020-06-12T02:38:18.675 INFO:tasks.rados.rados.0.smithi107.stdout:4816: writing smithi10729407-20 from 647168 to 1155072 tid 2 2020-06-12T02:38:18.677 INFO:tasks.rados.rados.0.smithi107.stdout:4816: writing smithi10729407-20 from 1155072 to 1449984 tid 3 2020-06-12T02:38:18.678 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:38:18.678 INFO:tasks.rados.rados.0.smithi107.stdout:4817: seq_num 4816 ranges {0=663552,663552=794624,1458176=8192} 2020-06-12T02:38:18.681 INFO:tasks.rados.rados.0.smithi107.stdout:4817: writing smithi10729407-5 from 0 to 663552 tid 1 2020-06-12T02:38:18.686 INFO:tasks.rados.rados.0.smithi107.stdout:4817: writing smithi10729407-5 from 663552 to 1458176 tid 2 2020-06-12T02:38:18.687 INFO:tasks.rados.rados.0.smithi107.stdout:4817: writing smithi10729407-5 from 1458176 to 1466368 tid 3 2020-06-12T02:38:18.687 INFO:tasks.rados.rados.0.smithi107.stdout:4818: delete oid 19 current snap is 0 2020-06-12T02:38:18.687 INFO:tasks.rados.rados.0.smithi107.stdout:4812: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:18.687 INFO:tasks.rados.rados.0.smithi107.stdout:4812: finishing write tid 5 to smithi10729407-8 2020-06-12T02:38:18.687 INFO:tasks.rados.rados.0.smithi107.stdout:4813: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:18.688 INFO:tasks.rados.rados.0.smithi107.stdout:4812: finishing write tid 6 to smithi10729407-8 2020-06-12T02:38:18.688 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 820 (ObjNum 4811 snap 0 seq_num 4811) dirty exists 2020-06-12T02:38:18.688 INFO:tasks.rados.rados.0.smithi107.stdout:4812: left oid 8 (ObjNum 4811 snap 0 seq_num 4811) 2020-06-12T02:38:18.688 INFO:tasks.rados.rados.0.smithi107.stdout:4813: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:18.688 INFO:tasks.rados.rados.0.smithi107.stdout:4813: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:18.689 INFO:tasks.rados.rados.0.smithi107.stdout:4813: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:18.689 INFO:tasks.rados.rados.0.smithi107.stdout:4813: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:18.689 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 899 (ObjNum 4812 snap 0 seq_num 4812) dirty exists 2020-06-12T02:38:18.689 INFO:tasks.rados.rados.0.smithi107.stdout:4813: left oid 4 (ObjNum 4812 snap 0 seq_num 4812) 2020-06-12T02:38:18.689 INFO:tasks.rados.rados.0.smithi107.stdout:4816: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:18.690 INFO:tasks.rados.rados.0.smithi107.stdout:4816: finishing write tid 2 to smithi10729407-20 2020-06-12T02:38:18.690 INFO:tasks.rados.rados.0.smithi107.stdout:4816: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:18.690 INFO:tasks.rados.rados.0.smithi107.stdout:4816: finishing write tid 5 to smithi10729407-20 2020-06-12T02:38:18.691 INFO:tasks.rados.rados.0.smithi107.stdout:4816: finishing write tid 6 to smithi10729407-20 2020-06-12T02:38:18.691 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 368 (ObjNum 4815 snap 0 seq_num 4815) dirty exists 2020-06-12T02:38:18.691 INFO:tasks.rados.rados.0.smithi107.stdout:4816: left oid 20 (ObjNum 4815 snap 0 seq_num 4815) 2020-06-12T02:38:18.692 INFO:tasks.rados.rados.0.smithi107.stdout:4817: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:18.693 INFO:tasks.rados.rados.0.smithi107.stdout:4812: done (5 left) 2020-06-12T02:38:18.693 INFO:tasks.rados.rados.0.smithi107.stdout:4813: done (4 left) 2020-06-12T02:38:18.693 INFO:tasks.rados.rados.0.smithi107.stdout:4816: done (3 left) 2020-06-12T02:38:18.693 INFO:tasks.rados.rados.0.smithi107.stdout:4818: done (2 left) 2020-06-12T02:38:18.694 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:18.694 INFO:tasks.rados.rados.0.smithi107.stdout:4819: seq_num 4818 ranges {1384448=696320,2080768=565248,2646016=237568} 2020-06-12T02:38:18.705 INFO:tasks.rados.rados.0.smithi107.stdout:4819: writing smithi10729407-8 from 1384448 to 2080768 tid 1 2020-06-12T02:38:18.707 INFO:tasks.rados.rados.0.smithi107.stdout:4819: writing smithi10729407-8 from 2080768 to 2646016 tid 2 2020-06-12T02:38:18.709 INFO:tasks.rados.rados.0.smithi107.stdout:4819: writing smithi10729407-8 from 2646016 to 2883584 tid 3 2020-06-12T02:38:18.710 INFO:tasks.rados.rados.0.smithi107.stdout:4820: delete oid 17 current snap is 0 2020-06-12T02:38:18.710 INFO:tasks.rados.rados.0.smithi107.stdout:4817: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:18.710 INFO:tasks.rados.rados.0.smithi107.stdout:4817: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:18.710 INFO:tasks.rados.rados.0.smithi107.stdout:4817: finishing write tid 5 to smithi10729407-5 2020-06-12T02:38:18.710 INFO:tasks.rados.rados.0.smithi107.stdout:4817: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:18.711 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 470 (ObjNum 4816 snap 0 seq_num 4816) dirty exists 2020-06-12T02:38:18.711 INFO:tasks.rados.rados.0.smithi107.stdout:4817: left oid 5 (ObjNum 4816 snap 0 seq_num 4816) 2020-06-12T02:38:18.713 INFO:tasks.rados.rados.0.smithi107.stdout:4817: done (3 left) 2020-06-12T02:38:18.713 INFO:tasks.rados.rados.0.smithi107.stdout:4820: done (2 left) 2020-06-12T02:38:18.714 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:18.714 INFO:tasks.rados.rados.0.smithi107.stdout:4821: seq_num 4820 ranges {1400832=729088,2129920=786432,2916352=16384} 2020-06-12T02:38:18.725 INFO:tasks.rados.rados.0.smithi107.stdout:4821: writing smithi10729407-4 from 1400832 to 2129920 tid 1 2020-06-12T02:38:18.727 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:18.725+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:18.729 INFO:tasks.rados.rados.0.smithi107.stdout:4821: writing smithi10729407-4 from 2129920 to 2916352 tid 2 2020-06-12T02:38:18.729 INFO:tasks.rados.rados.0.smithi107.stdout:4821: writing smithi10729407-4 from 2916352 to 2932736 tid 3 2020-06-12T02:38:18.730 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:38:18.730 INFO:tasks.rados.rados.0.smithi107.stdout:4819: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:18.730 INFO:tasks.rados.rados.0.smithi107.stdout:4822: seq_num 4821 ranges {827392=745472,1572864=663552,2236416=139264} 2020-06-12T02:38:18.738 INFO:tasks.rados.rados.0.smithi107.stdout:4822: writing smithi10729407-1 from 827392 to 1572864 tid 1 2020-06-12T02:38:18.742 INFO:tasks.rados.rados.0.smithi107.stdout:4822: writing smithi10729407-1 from 1572864 to 2236416 tid 2 2020-06-12T02:38:18.743 INFO:tasks.rados.rados.0.smithi107.stdout:4822: writing smithi10729407-1 from 2236416 to 2375680 tid 3 2020-06-12T02:38:18.743 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:38:18.743 INFO:tasks.rados.rados.0.smithi107.stdout:4823: seq_num 4822 ranges {0=761856,761856=548864,1310720=253952} 2020-06-12T02:38:18.747 INFO:tasks.rados.rados.0.smithi107.stdout:4823: writing smithi10729407-11 from 0 to 761856 tid 1 2020-06-12T02:38:18.750 INFO:tasks.rados.rados.0.smithi107.stdout:4823: writing smithi10729407-11 from 761856 to 1310720 tid 2 2020-06-12T02:38:18.752 INFO:tasks.rados.rados.0.smithi107.stdout:4823: writing smithi10729407-11 from 1310720 to 1564672 tid 3 2020-06-12T02:38:18.753 INFO:tasks.rados.rados.0.smithi107.stdout:4824: delete oid 6 current snap is 0 2020-06-12T02:38:18.754 INFO:tasks.rados.rados.0.smithi107.stdout:4819: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:18.754 INFO:tasks.rados.rados.0.smithi107.stdout:4819: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:18.754 INFO:tasks.rados.rados.0.smithi107.stdout:4819: finishing write tid 5 to smithi10729407-8 2020-06-12T02:38:18.754 INFO:tasks.rados.rados.0.smithi107.stdout:4819: finishing write tid 6 to smithi10729407-8 2020-06-12T02:38:18.755 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 824 (ObjNum 4818 snap 0 seq_num 4818) dirty exists 2020-06-12T02:38:18.755 INFO:tasks.rados.rados.0.smithi107.stdout:4819: left oid 8 (ObjNum 4818 snap 0 seq_num 4818) 2020-06-12T02:38:18.755 INFO:tasks.rados.rados.0.smithi107.stdout:4821: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:18.755 INFO:tasks.rados.rados.0.smithi107.stdout:4821: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:18.755 INFO:tasks.rados.rados.0.smithi107.stdout:4821: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:18.756 INFO:tasks.rados.rados.0.smithi107.stdout:4821: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:18.756 INFO:tasks.rados.rados.0.smithi107.stdout:4821: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:18.756 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 904 (ObjNum 4820 snap 0 seq_num 4820) dirty exists 2020-06-12T02:38:18.757 INFO:tasks.rados.rados.0.smithi107.stdout:4821: left oid 4 (ObjNum 4820 snap 0 seq_num 4820) 2020-06-12T02:38:18.757 INFO:tasks.rados.rados.0.smithi107.stdout:4822: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:18.757 INFO:tasks.rados.rados.0.smithi107.stdout:4822: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:18.757 INFO:tasks.rados.rados.0.smithi107.stdout:4822: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:18.758 INFO:tasks.rados.rados.0.smithi107.stdout:4822: finishing write tid 5 to smithi10729407-1 2020-06-12T02:38:18.758 INFO:tasks.rados.rados.0.smithi107.stdout:4822: finishing write tid 6 to smithi10729407-1 2020-06-12T02:38:18.758 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 897 (ObjNum 4821 snap 0 seq_num 4821) dirty exists 2020-06-12T02:38:18.759 INFO:tasks.rados.rados.0.smithi107.stdout:4822: left oid 1 (ObjNum 4821 snap 0 seq_num 4821) 2020-06-12T02:38:18.759 INFO:tasks.rados.rados.0.smithi107.stdout:4819: done (5 left) 2020-06-12T02:38:18.759 INFO:tasks.rados.rados.0.smithi107.stdout:4821: done (4 left) 2020-06-12T02:38:18.759 INFO:tasks.rados.rados.0.smithi107.stdout:4822: done (3 left) 2020-06-12T02:38:18.759 INFO:tasks.rados.rados.0.smithi107.stdout:4824: done (2 left) 2020-06-12T02:38:18.760 INFO:tasks.rados.rados.0.smithi107.stdout:4825: delete oid 15 current snap is 0 2020-06-12T02:38:18.760 INFO:tasks.rados.rados.0.smithi107.stdout:4823: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:18.760 INFO:tasks.rados.rados.0.smithi107.stdout:4825: done (2 left) 2020-06-12T02:38:18.761 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:18.761 INFO:tasks.rados.rados.0.smithi107.stdout:4826: seq_num 4825 ranges {2883584=417792,3301376=606208,3907584=598016} 2020-06-12T02:38:18.762 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:18.762 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:18.771 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-06-12T02:38:18.778 INFO:tasks.rados.rados.0.smithi107.stdout:4826: writing smithi10729407-8 from 2883584 to 3301376 tid 1 2020-06-12T02:38:18.781 INFO:tasks.rados.rados.0.smithi107.stdout:4826: writing smithi10729407-8 from 3301376 to 3907584 tid 2 2020-06-12T02:38:18.785 INFO:tasks.rados.rados.0.smithi107.stdout:4826: writing smithi10729407-8 from 3907584 to 4505600 tid 3 2020-06-12T02:38:18.785 INFO:tasks.rados.rados.0.smithi107.stdout:4827: delete oid 3 current snap is 0 2020-06-12T02:38:18.785 INFO:tasks.rados.rados.0.smithi107.stdout:4823: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:18.785 INFO:tasks.rados.rados.0.smithi107.stdout:4823: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:18.785 INFO:tasks.rados.rados.0.smithi107.stdout:4823: finishing write tid 5 to smithi10729407-11 2020-06-12T02:38:18.786 INFO:tasks.rados.rados.0.smithi107.stdout:4823: finishing write tid 6 to smithi10729407-11 2020-06-12T02:38:18.786 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 868 (ObjNum 4822 snap 0 seq_num 4822) dirty exists 2020-06-12T02:38:18.786 INFO:tasks.rados.rados.0.smithi107.stdout:4823: left oid 11 (ObjNum 4822 snap 0 seq_num 4822) 2020-06-12T02:38:18.789 INFO:tasks.rados.rados.0.smithi107.stdout:4823: done (3 left) 2020-06-12T02:38:18.790 INFO:tasks.rados.rados.0.smithi107.stdout:4827: done (2 left) 2020-06-12T02:38:18.790 INFO:tasks.rados.rados.0.smithi107.stdout:4828: delete oid 7 current snap is 0 2020-06-12T02:38:18.791 INFO:tasks.rados.rados.0.smithi107.stdout:4826: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:18.791 INFO:tasks.rados.rados.0.smithi107.stdout:4828: done (2 left) 2020-06-12T02:38:18.792 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:38:18.792 INFO:tasks.rados.rados.0.smithi107.stdout:4829: seq_num 4828 ranges {0=466944,466944=704512,1171456=499712} 2020-06-12T02:38:18.793 INFO:tasks.rados.rados.0.smithi107.stdout:4829: writing smithi10729407-3 from 0 to 466944 tid 1 2020-06-12T02:38:18.797 INFO:tasks.rados.rados.0.smithi107.stdout:4829: writing smithi10729407-3 from 466944 to 1171456 tid 2 2020-06-12T02:38:18.801 INFO:tasks.rados.rados.0.smithi107.stdout:4829: writing smithi10729407-3 from 1171456 to 1671168 tid 3 2020-06-12T02:38:18.801 INFO:tasks.rados.rados.0.smithi107.stdout:4830: delete oid 9 current snap is 0 2020-06-12T02:38:18.801 INFO:tasks.rados.rados.0.smithi107.stdout:4826: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:18.802 INFO:tasks.rados.rados.0.smithi107.stdout:4826: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:18.802 INFO:tasks.rados.rados.0.smithi107.stdout:4826: finishing write tid 5 to smithi10729407-8 2020-06-12T02:38:18.802 INFO:tasks.rados.rados.0.smithi107.stdout:4826: finishing write tid 6 to smithi10729407-8 2020-06-12T02:38:18.802 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 829 (ObjNum 4825 snap 0 seq_num 4825) dirty exists 2020-06-12T02:38:18.802 INFO:tasks.rados.rados.0.smithi107.stdout:4826: left oid 8 (ObjNum 4825 snap 0 seq_num 4825) 2020-06-12T02:38:18.803 INFO:tasks.rados.rados.0.smithi107.stdout:4829: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:18.804 INFO:tasks.rados.rados.0.smithi107.stdout:4826: done (3 left) 2020-06-12T02:38:18.804 INFO:tasks.rados.rados.0.smithi107.stdout:4830: done (2 left) 2020-06-12T02:38:18.804 INFO:tasks.rados.rados.0.smithi107.stdout:4831: delete oid 13 current snap is 0 2020-06-12T02:38:18.807 INFO:tasks.rados.rados.0.smithi107.stdout:4831: done (2 left) 2020-06-12T02:38:18.807 INFO:tasks.rados.rados.0.smithi107.stdout:4832: delete oid 17 current snap is 0 2020-06-12T02:38:18.807 INFO:tasks.rados.rados.0.smithi107.stdout:4829: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:18.808 INFO:tasks.rados.rados.0.smithi107.stdout:4829: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:18.809 INFO:tasks.rados.rados.0.smithi107.stdout:4829: finishing write tid 5 to smithi10729407-3 2020-06-12T02:38:18.810 INFO:tasks.rados.rados.0.smithi107.stdout:4829: finishing write tid 6 to smithi10729407-3 2020-06-12T02:38:18.811 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 866 (ObjNum 4828 snap 0 seq_num 4828) dirty exists 2020-06-12T02:38:18.811 INFO:tasks.rados.rados.0.smithi107.stdout:4829: left oid 3 (ObjNum 4828 snap 0 seq_num 4828) 2020-06-12T02:38:18.812 INFO:tasks.rados.rados.0.smithi107.stdout:4829: done (2 left) 2020-06-12T02:38:18.812 INFO:tasks.rados.rados.0.smithi107.stdout:4832: done (1 left) 2020-06-12T02:38:18.813 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:18.813 INFO:tasks.rados.rados.0.smithi107.stdout:4833: seq_num 4832 ranges {0=532480,532480=647168,1179648=524288,1703936=32768} 2020-06-12T02:38:18.815 INFO:tasks.rados.rados.0.smithi107.stdout:4833: writing smithi10729407-9 from 0 to 532480 tid 1 2020-06-12T02:38:18.819 INFO:tasks.rados.rados.0.smithi107.stdout:4833: writing smithi10729407-9 from 532480 to 1179648 tid 2 2020-06-12T02:38:18.823 INFO:tasks.rados.rados.0.smithi107.stdout:4833: writing smithi10729407-9 from 1179648 to 1703936 tid 3 2020-06-12T02:38:18.823 INFO:tasks.rados.rados.0.smithi107.stdout:4833: writing smithi10729407-9 from 1703936 to 1736704 tid 4 2020-06-12T02:38:18.824 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:38:18.824 INFO:tasks.rados.rados.0.smithi107.stdout:4834: seq_num 4833 ranges {0=548864,548864=532480,1081344=671744} 2020-06-12T02:38:18.826 INFO:tasks.rados.rados.0.smithi107.stdout:4834: writing smithi10729407-6 from 0 to 548864 tid 1 2020-06-12T02:38:18.830 INFO:tasks.rados.rados.0.smithi107.stdout:4834: writing smithi10729407-6 from 548864 to 1081344 tid 2 2020-06-12T02:38:18.835 INFO:tasks.rados.rados.0.smithi107.stdout:4834: writing smithi10729407-6 from 1081344 to 1753088 tid 3 2020-06-12T02:38:18.837 INFO:tasks.rados.rados.0.smithi107.stdout:4835: delete oid 4 current snap is 0 2020-06-12T02:38:18.838 INFO:tasks.rados.rados.0.smithi107.stdout:4833: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:18.838 INFO:tasks.rados.rados.0.smithi107.stdout:4833: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:18.838 INFO:tasks.rados.rados.0.smithi107.stdout:4833: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:18.838 INFO:tasks.rados.rados.0.smithi107.stdout:4833: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:18.839 INFO:tasks.rados.rados.0.smithi107.stdout:4833: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:18.839 INFO:tasks.rados.rados.0.smithi107.stdout:4833: finishing write tid 7 to smithi10729407-9 2020-06-12T02:38:18.839 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 865 (ObjNum 4832 snap 0 seq_num 4832) dirty exists 2020-06-12T02:38:18.839 INFO:tasks.rados.rados.0.smithi107.stdout:4833: left oid 9 (ObjNum 4832 snap 0 seq_num 4832) 2020-06-12T02:38:18.839 INFO:tasks.rados.rados.0.smithi107.stdout:4834: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:18.840 INFO:tasks.rados.rados.0.smithi107.stdout:4833: done (3 left) 2020-06-12T02:38:18.840 INFO:tasks.rados.rados.0.smithi107.stdout:4835: done (2 left) 2020-06-12T02:38:18.840 INFO:tasks.rados.rados.0.smithi107.stdout:4836: delete oid 7 current snap is 0 2020-06-12T02:38:18.841 INFO:tasks.rados.rados.0.smithi107.stdout:4834: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:18.843 INFO:tasks.rados.rados.0.smithi107.stdout:4836: done (2 left) 2020-06-12T02:38:18.844 INFO:tasks.rados.rados.0.smithi107.stdout:4837: delete oid 12 current snap is 0 2020-06-12T02:38:18.845 INFO:tasks.rados.rados.0.smithi107.stdout:4834: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:18.845 INFO:tasks.rados.rados.0.smithi107.stdout:4834: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:18.846 INFO:tasks.rados.rados.0.smithi107.stdout:4834: finishing write tid 6 to smithi10729407-6 2020-06-12T02:38:18.846 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 902 (ObjNum 4833 snap 0 seq_num 4833) dirty exists 2020-06-12T02:38:18.847 INFO:tasks.rados.rados.0.smithi107.stdout:4834: left oid 6 (ObjNum 4833 snap 0 seq_num 4833) 2020-06-12T02:38:18.849 INFO:tasks.rados.rados.0.smithi107.stdout:4834: done (2 left) 2020-06-12T02:38:18.849 INFO:tasks.rados.rados.0.smithi107.stdout:4837: done (1 left) 2020-06-12T02:38:18.849 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:38:18.849 INFO:tasks.rados.rados.0.smithi107.stdout:4838: seq_num 4837 ranges {0=614400,614400=516096,1130496=663552,1794048=24576} 2020-06-12T02:38:18.852 INFO:tasks.rados.rados.0.smithi107.stdout:4838: writing smithi10729407-7 from 0 to 614400 tid 1 2020-06-12T02:38:18.855 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:18.853+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:18.855 INFO:tasks.rados.rados.0.smithi107.stdout:4838: writing smithi10729407-7 from 614400 to 1130496 tid 2 2020-06-12T02:38:18.859 INFO:tasks.rados.rados.0.smithi107.stdout:4838: writing smithi10729407-7 from 1130496 to 1794048 tid 3 2020-06-12T02:38:18.859 INFO:tasks.rados.rados.0.smithi107.stdout:4838: writing smithi10729407-7 from 1794048 to 1818624 tid 4 2020-06-12T02:38:18.860 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:18.860 INFO:tasks.rados.rados.0.smithi107.stdout:4839: seq_num 4838 ranges {0=630784,630784=802816,1433600=401408} 2020-06-12T02:38:18.868 INFO:tasks.rados.rados.0.smithi107.stdout:4839: writing smithi10729407-19 from 0 to 630784 tid 1 2020-06-12T02:38:18.869 INFO:tasks.rados.rados.0.smithi107.stdout:4839: writing smithi10729407-19 from 630784 to 1433600 tid 2 2020-06-12T02:38:18.870 INFO:tasks.rados.rados.0.smithi107.stdout:4839: writing smithi10729407-19 from 1433600 to 1835008 tid 3 2020-06-12T02:38:18.871 INFO:tasks.rados.rados.0.smithi107.stdout:4840: delete oid 1 current snap is 0 2020-06-12T02:38:18.871 INFO:tasks.rados.rados.0.smithi107.stdout:4838: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:18.871 INFO:tasks.rados.rados.0.smithi107.stdout:4838: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:18.871 INFO:tasks.rados.rados.0.smithi107.stdout:4838: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:18.884 INFO:tasks.rados.rados.0.smithi107.stdout:4838: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:18.885 INFO:tasks.rados.rados.0.smithi107.stdout:4838: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:18.885 INFO:tasks.rados.rados.0.smithi107.stdout:4839: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:18.885 INFO:tasks.rados.rados.0.smithi107.stdout:4838: finishing write tid 7 to smithi10729407-7 2020-06-12T02:38:18.885 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 903 (ObjNum 4837 snap 0 seq_num 4837) dirty exists 2020-06-12T02:38:18.886 INFO:tasks.rados.rados.0.smithi107.stdout:4838: left oid 7 (ObjNum 4837 snap 0 seq_num 4837) 2020-06-12T02:38:18.886 INFO:tasks.rados.rados.0.smithi107.stdout:4838: done (3 left) 2020-06-12T02:38:18.886 INFO:tasks.rados.rados.0.smithi107.stdout:4840: done (2 left) 2020-06-12T02:38:18.886 INFO:tasks.rados.rados.0.smithi107.stdout:4841: delete oid 16 current snap is 0 2020-06-12T02:38:18.886 INFO:tasks.rados.rados.0.smithi107.stdout:4839: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:18.887 INFO:tasks.rados.rados.0.smithi107.stdout:4841: done (2 left) 2020-06-12T02:38:18.887 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:38:18.887 INFO:tasks.rados.rados.0.smithi107.stdout:4842: seq_num 4841 ranges {1671168=679936,2351104=450560,2801664=647168,3448832=106496} 2020-06-12T02:38:18.895 INFO:tasks.rados.rados.0.smithi107.stdout:4842: writing smithi10729407-3 from 1671168 to 2351104 tid 1 2020-06-12T02:38:18.897 INFO:tasks.rados.rados.0.smithi107.stdout:4842: writing smithi10729407-3 from 2351104 to 2801664 tid 2 2020-06-12T02:38:18.901 INFO:tasks.rados.rados.0.smithi107.stdout:4842: writing smithi10729407-3 from 2801664 to 3448832 tid 3 2020-06-12T02:38:18.902 INFO:tasks.rados.rados.0.smithi107.stdout:4842: writing smithi10729407-3 from 3448832 to 3555328 tid 4 2020-06-12T02:38:18.902 INFO:tasks.rados.rados.0.smithi107.stdout:4843: delete oid 1 current snap is 0 2020-06-12T02:38:18.902 INFO:tasks.rados.rados.0.smithi107.stdout:4839: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:18.903 INFO:tasks.rados.rados.0.smithi107.stdout:4839: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:18.914 INFO:tasks.rados.rados.0.smithi107.stdout:4839: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:18.914 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 845 (ObjNum 4838 snap 0 seq_num 4838) dirty exists 2020-06-12T02:38:18.914 INFO:tasks.rados.rados.0.smithi107.stdout:4839: left oid 19 (ObjNum 4838 snap 0 seq_num 4838) 2020-06-12T02:38:18.916 INFO:tasks.rados.rados.0.smithi107.stdout:4839: done (3 left) 2020-06-12T02:38:18.916 INFO:tasks.rados.rados.0.smithi107.stdout:4843: done (2 left) 2020-06-12T02:38:18.916 INFO:tasks.rados.rados.0.smithi107.stdout:4844: delete oid 1 current snap is 0 2020-06-12T02:38:18.916 INFO:tasks.rados.rados.0.smithi107.stdout:4842: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:18.917 INFO:tasks.rados.rados.0.smithi107.stdout:4844: done (2 left) 2020-06-12T02:38:18.917 INFO:tasks.rados.rados.0.smithi107.stdout:4845: delete oid 15 current snap is 0 2020-06-12T02:38:18.917 INFO:tasks.rados.rados.0.smithi107.stdout:4842: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:18.917 INFO:tasks.rados.rados.0.smithi107.stdout:4842: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:18.917 INFO:tasks.rados.rados.0.smithi107.stdout:4842: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:18.926 INFO:tasks.rados.rados.0.smithi107.stdout:4845: done (2 left) 2020-06-12T02:38:18.926 INFO:tasks.rados.rados.0.smithi107.stdout:4846: delete oid 13 current snap is 0 2020-06-12T02:38:18.926 INFO:tasks.rados.rados.0.smithi107.stdout:4842: finishing write tid 6 to smithi10729407-3 2020-06-12T02:38:18.926 INFO:tasks.rados.rados.0.smithi107.stdout:4846: done (2 left) 2020-06-12T02:38:18.927 INFO:tasks.rados.rados.0.smithi107.stdout:4847: delete oid 15 current snap is 0 2020-06-12T02:38:18.927 INFO:tasks.rados.rados.0.smithi107.stdout:4842: finishing write tid 7 to smithi10729407-3 2020-06-12T02:38:18.927 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 872 (ObjNum 4841 snap 0 seq_num 4841) dirty exists 2020-06-12T02:38:18.927 INFO:tasks.rados.rados.0.smithi107.stdout:4842: left oid 3 (ObjNum 4841 snap 0 seq_num 4841) 2020-06-12T02:38:18.927 INFO:tasks.rados.rados.0.smithi107.stdout:4842: done (2 left) 2020-06-12T02:38:18.928 INFO:tasks.rados.rados.0.smithi107.stdout:4847: done (1 left) 2020-06-12T02:38:18.928 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:18.928 INFO:tasks.rados.rados.0.smithi107.stdout:4848: seq_num 4847 ranges {0=786432,786432=606208,1392640=598016} 2020-06-12T02:38:18.929 INFO:tasks.rados.rados.0.smithi107.stdout:4848: writing smithi10729407-17 from 0 to 786432 tid 1 2020-06-12T02:38:18.930 INFO:tasks.rados.rados.0.smithi107.stdout:4848: writing smithi10729407-17 from 786432 to 1392640 tid 2 2020-06-12T02:38:18.930 INFO:tasks.rados.rados.0.smithi107.stdout:4848: writing smithi10729407-17 from 1392640 to 1990656 tid 3 2020-06-12T02:38:18.930 INFO:tasks.rados.rados.0.smithi107.stdout:4849: delete oid 5 current snap is 0 2020-06-12T02:38:18.931 INFO:tasks.rados.rados.0.smithi107.stdout:4848: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:18.931 INFO:tasks.rados.rados.0.smithi107.stdout:4849: done (2 left) 2020-06-12T02:38:18.931 INFO:tasks.rados.rados.0.smithi107.stdout:4850: delete oid 16 current snap is 0 2020-06-12T02:38:18.934 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:18.927+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:18.934 INFO:tasks.rados.rados.0.smithi107.stdout:4850: done (2 left) 2020-06-12T02:38:18.935 INFO:tasks.rados.rados.0.smithi107.stdout:4851: delete oid 7 current snap is 0 2020-06-12T02:38:18.935 INFO:tasks.rados.rados.0.smithi107.stdout:4851: done (2 left) 2020-06-12T02:38:18.935 INFO:tasks.rados.rados.0.smithi107.stdout:4852: delete oid 7 current snap is 0 2020-06-12T02:38:18.936 INFO:tasks.rados.rados.0.smithi107.stdout:4848: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:18.936 INFO:tasks.rados.rados.0.smithi107.stdout:4848: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:18.938 INFO:tasks.rados.rados.0.smithi107.stdout:4848: finishing write tid 5 to smithi10729407-17 2020-06-12T02:38:18.938 INFO:tasks.rados.rados.0.smithi107.stdout:4852: done (2 left) 2020-06-12T02:38:18.939 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:18.939 INFO:tasks.rados.rados.0.smithi107.stdout:4853: seq_num 4852 ranges {0=466944,466944=483328,950272=425984,1376256=696320} 2020-06-12T02:38:18.941 INFO:tasks.rados.rados.0.smithi107.stdout:4853: writing smithi10729407-1 from 0 to 466944 tid 1 2020-06-12T02:38:18.944 INFO:tasks.rados.rados.0.smithi107.stdout:4853: writing smithi10729407-1 from 466944 to 950272 tid 2 2020-06-12T02:38:18.946 INFO:tasks.rados.rados.0.smithi107.stdout:4853: writing smithi10729407-1 from 950272 to 1376256 tid 3 2020-06-12T02:38:18.951 INFO:tasks.rados.rados.0.smithi107.stdout:4853: writing smithi10729407-1 from 1376256 to 2072576 tid 4 2020-06-12T02:38:18.951 INFO:tasks.rados.rados.0.smithi107.stdout:4854: delete oid 7 current snap is 0 2020-06-12T02:38:18.952 INFO:tasks.rados.rados.0.smithi107.stdout:4848: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:18.952 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 910 (ObjNum 4847 snap 0 seq_num 4847) dirty exists 2020-06-12T02:38:18.952 INFO:tasks.rados.rados.0.smithi107.stdout:4848: left oid 17 (ObjNum 4847 snap 0 seq_num 4847) 2020-06-12T02:38:18.952 INFO:tasks.rados.rados.0.smithi107.stdout:4853: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:18.954 INFO:tasks.rados.rados.0.smithi107.stdout:4853: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:18.955 INFO:tasks.rados.rados.0.smithi107.stdout:4848: done (3 left) 2020-06-12T02:38:18.955 INFO:tasks.rados.rados.0.smithi107.stdout:4854: done (2 left) 2020-06-12T02:38:18.955 INFO:tasks.rados.rados.0.smithi107.stdout:4855: delete oid 15 current snap is 0 2020-06-12T02:38:18.961 INFO:tasks.rados.rados.0.smithi107.stdout:4855: done (2 left) 2020-06-12T02:38:18.961 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:18.961 INFO:tasks.rados.rados.0.smithi107.stdout:4856: seq_num 4855 ranges {0=516096,516096=532480,1048576=598016,1646592=442368,2088960=32768} 2020-06-12T02:38:18.965 INFO:tasks.rados.rados.0.smithi107.stdout:4856: writing smithi10729407-13 from 0 to 516096 tid 1 2020-06-12T02:38:18.966 INFO:tasks.rados.rados.0.smithi107.stdout:4856: writing smithi10729407-13 from 516096 to 1048576 tid 2 2020-06-12T02:38:18.967 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:18.967 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:18.969 INFO:tasks.rados.rados.0.smithi107.stdout:4856: writing smithi10729407-13 from 1048576 to 1646592 tid 3 2020-06-12T02:38:18.972 INFO:tasks.rados.rados.0.smithi107.stdout:4856: writing smithi10729407-13 from 1646592 to 2088960 tid 4 2020-06-12T02:38:18.973 INFO:tasks.rados.rados.0.smithi107.stdout:4856: writing smithi10729407-13 from 2088960 to 2121728 tid 5 2020-06-12T02:38:18.973 INFO:tasks.rados.rados.0.smithi107.stdout:4857: delete oid 11 current snap is 0 2020-06-12T02:38:18.973 INFO:tasks.rados.rados.0.smithi107.stdout:4853: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:18.973 INFO:tasks.rados.rados.0.smithi107.stdout:4853: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:18.974 INFO:tasks.rados.rados.0.smithi107.stdout:4853: finishing write tid 6 to smithi10729407-1 2020-06-12T02:38:18.974 INFO:tasks.rados.rados.0.smithi107.stdout:4856: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:18.974 INFO:tasks.rados.rados.0.smithi107.stdout:4853: finishing write tid 7 to smithi10729407-1 2020-06-12T02:38:18.974 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 910 (ObjNum 4852 snap 0 seq_num 4852) dirty exists 2020-06-12T02:38:18.974 INFO:tasks.rados.rados.0.smithi107.stdout:4853: left oid 1 (ObjNum 4852 snap 0 seq_num 4852) 2020-06-12T02:38:18.977 INFO:tasks.rados.rados.0.smithi107.stdout:4853: done (3 left) 2020-06-12T02:38:18.977 INFO:tasks.rados.rados.0.smithi107.stdout:4857: done (2 left) 2020-06-12T02:38:18.977 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:18.977 INFO:tasks.rados.rados.0.smithi107.stdout:4858: seq_num 4857 ranges {0=548864,548864=704512,1253376=565248,1818624=335872} 2020-06-12T02:38:18.978 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-06-12T02:38:18.980 INFO:tasks.rados.rados.0.smithi107.stdout:4858: writing smithi10729407-4 from 0 to 548864 tid 1 2020-06-12T02:38:18.984 INFO:tasks.rados.rados.0.smithi107.stdout:4858: writing smithi10729407-4 from 548864 to 1253376 tid 2 2020-06-12T02:38:18.993 INFO:tasks.rados.rados.0.smithi107.stdout:4858: writing smithi10729407-4 from 1253376 to 1818624 tid 3 2020-06-12T02:38:18.993 INFO:tasks.rados.rados.0.smithi107.stdout:4858: writing smithi10729407-4 from 1818624 to 2154496 tid 4 2020-06-12T02:38:18.995 INFO:tasks.rados.rados.0.smithi107.stdout:4859: delete oid 5 current snap is 0 2020-06-12T02:38:18.995 INFO:tasks.rados.rados.0.smithi107.stdout:4856: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:18.995 INFO:tasks.rados.rados.0.smithi107.stdout:4856: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:18.995 INFO:tasks.rados.rados.0.smithi107.stdout:4856: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:18.996 INFO:tasks.rados.rados.0.smithi107.stdout:4856: finishing write tid 5 to smithi10729407-13 2020-06-12T02:38:18.996 INFO:tasks.rados.rados.0.smithi107.stdout:4856: finishing write tid 7 to smithi10729407-13 2020-06-12T02:38:18.996 INFO:tasks.rados.rados.0.smithi107.stdout:4856: finishing write tid 8 to smithi10729407-13 2020-06-12T02:38:18.996 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 913 (ObjNum 4855 snap 0 seq_num 4855) dirty exists 2020-06-12T02:38:18.996 INFO:tasks.rados.rados.0.smithi107.stdout:4856: left oid 13 (ObjNum 4855 snap 0 seq_num 4855) 2020-06-12T02:38:18.997 INFO:tasks.rados.rados.0.smithi107.stdout:4856: done (3 left) 2020-06-12T02:38:18.997 INFO:tasks.rados.rados.0.smithi107.stdout:4859: done (2 left) 2020-06-12T02:38:18.997 INFO:tasks.rados.rados.0.smithi107.stdout:4860: delete oid 13 current snap is 0 2020-06-12T02:38:18.998 INFO:tasks.rados.rados.0.smithi107.stdout:4860: done (2 left) 2020-06-12T02:38:18.998 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:38:18.998 INFO:tasks.rados.rados.0.smithi107.stdout:4861: seq_num 4860 ranges {0=598016,598016=802816,1400832=786432,2187264=16384} 2020-06-12T02:38:18.999 INFO:tasks.rados.rados.0.smithi107.stdout:4861: writing smithi10729407-11 from 0 to 598016 tid 1 2020-06-12T02:38:19.007 INFO:tasks.rados.rados.0.smithi107.stdout:4861: writing smithi10729407-11 from 598016 to 1400832 tid 2 2020-06-12T02:38:19.009 INFO:tasks.rados.rados.0.smithi107.stdout:4861: writing smithi10729407-11 from 1400832 to 2187264 tid 3 2020-06-12T02:38:19.009 INFO:tasks.rados.rados.0.smithi107.stdout:4861: writing smithi10729407-11 from 2187264 to 2203648 tid 4 2020-06-12T02:38:19.009 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:38:19.009 INFO:tasks.rados.rados.0.smithi107.stdout:4862: seq_num 4861 ranges {1449984=622592,2072576=688128,2760704=548864,3309568=368640} 2020-06-12T02:38:19.021 INFO:tasks.rados.rados.0.smithi107.stdout:4862: writing smithi10729407-20 from 1449984 to 2072576 tid 1 2020-06-12T02:38:19.025 INFO:tasks.rados.rados.0.smithi107.stdout:4862: writing smithi10729407-20 from 2072576 to 2760704 tid 2 2020-06-12T02:38:19.030 INFO:tasks.rados.rados.0.smithi107.stdout:4862: writing smithi10729407-20 from 2760704 to 3309568 tid 3 2020-06-12T02:38:19.030 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:19.027+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:19.032 INFO:tasks.rados.rados.0.smithi107.stdout:4862: writing smithi10729407-20 from 3309568 to 3678208 tid 4 2020-06-12T02:38:19.032 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:19.033 INFO:tasks.rados.rados.0.smithi107.stdout:4863: seq_num 4862 ranges {0=638976,638976=573440,1212416=753664,1966080=278528} 2020-06-12T02:38:19.036 INFO:tasks.rados.rados.0.smithi107.stdout:4863: writing smithi10729407-14 from 0 to 638976 tid 1 2020-06-12T02:38:19.040 INFO:tasks.rados.rados.0.smithi107.stdout:4863: writing smithi10729407-14 from 638976 to 1212416 tid 2 2020-06-12T02:38:19.045 INFO:tasks.rados.rados.0.smithi107.stdout:4863: writing smithi10729407-14 from 1212416 to 1966080 tid 3 2020-06-12T02:38:19.047 INFO:tasks.rados.rados.0.smithi107.stdout:4863: writing smithi10729407-14 from 1966080 to 2244608 tid 4 2020-06-12T02:38:19.047 INFO:tasks.rados.rados.0.smithi107.stdout:4864: delete oid 9 current snap is 0 2020-06-12T02:38:19.047 INFO:tasks.rados.rados.0.smithi107.stdout:4858: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:19.048 INFO:tasks.rados.rados.0.smithi107.stdout:4858: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:19.048 INFO:tasks.rados.rados.0.smithi107.stdout:4858: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:19.048 INFO:tasks.rados.rados.0.smithi107.stdout:4858: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:19.048 INFO:tasks.rados.rados.0.smithi107.stdout:4858: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:19.048 INFO:tasks.rados.rados.0.smithi107.stdout:4858: finishing write tid 7 to smithi10729407-4 2020-06-12T02:38:19.049 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 915 (ObjNum 4857 snap 0 seq_num 4857) dirty exists 2020-06-12T02:38:19.049 INFO:tasks.rados.rados.0.smithi107.stdout:4858: left oid 4 (ObjNum 4857 snap 0 seq_num 4857) 2020-06-12T02:38:19.049 INFO:tasks.rados.rados.0.smithi107.stdout:4861: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:19.049 INFO:tasks.rados.rados.0.smithi107.stdout:4861: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:19.049 INFO:tasks.rados.rados.0.smithi107.stdout:4861: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:19.050 INFO:tasks.rados.rados.0.smithi107.stdout:4861: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:19.050 INFO:tasks.rados.rados.0.smithi107.stdout:4861: finishing write tid 6 to smithi10729407-11 2020-06-12T02:38:19.051 INFO:tasks.rados.rados.0.smithi107.stdout:4861: finishing write tid 7 to smithi10729407-11 2020-06-12T02:38:19.051 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 874 (ObjNum 4860 snap 0 seq_num 4860) dirty exists 2020-06-12T02:38:19.051 INFO:tasks.rados.rados.0.smithi107.stdout:4861: left oid 11 (ObjNum 4860 snap 0 seq_num 4860) 2020-06-12T02:38:19.051 INFO:tasks.rados.rados.0.smithi107.stdout:4862: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:19.051 INFO:tasks.rados.rados.0.smithi107.stdout:4862: finishing write tid 2 to smithi10729407-20 2020-06-12T02:38:19.052 INFO:tasks.rados.rados.0.smithi107.stdout:4862: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:19.052 INFO:tasks.rados.rados.0.smithi107.stdout:4862: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:19.052 INFO:tasks.rados.rados.0.smithi107.stdout:4862: finishing write tid 6 to smithi10729407-20 2020-06-12T02:38:19.052 INFO:tasks.rados.rados.0.smithi107.stdout:4862: finishing write tid 7 to smithi10729407-20 2020-06-12T02:38:19.052 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 373 (ObjNum 4861 snap 0 seq_num 4861) dirty exists 2020-06-12T02:38:19.052 INFO:tasks.rados.rados.0.smithi107.stdout:4862: left oid 20 (ObjNum 4861 snap 0 seq_num 4861) 2020-06-12T02:38:19.053 INFO:tasks.rados.rados.0.smithi107.stdout:4863: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:19.053 INFO:tasks.rados.rados.0.smithi107.stdout:4863: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:19.054 INFO:tasks.rados.rados.0.smithi107.stdout:4858: done (5 left) 2020-06-12T02:38:19.054 INFO:tasks.rados.rados.0.smithi107.stdout:4861: done (4 left) 2020-06-12T02:38:19.054 INFO:tasks.rados.rados.0.smithi107.stdout:4862: done (3 left) 2020-06-12T02:38:19.055 INFO:tasks.rados.rados.0.smithi107.stdout:4864: done (2 left) 2020-06-12T02:38:19.055 INFO:tasks.rados.rados.0.smithi107.stdout:4865: delete oid 3 current snap is 0 2020-06-12T02:38:19.055 INFO:tasks.rados.rados.0.smithi107.stdout:4863: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:19.055 INFO:tasks.rados.rados.0.smithi107.stdout:4863: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:19.056 INFO:tasks.rados.rados.0.smithi107.stdout:4863: finishing write tid 6 to smithi10729407-14 2020-06-12T02:38:19.056 INFO:tasks.rados.rados.0.smithi107.stdout:4865: done (2 left) 2020-06-12T02:38:19.057 INFO:tasks.rados.rados.0.smithi107.stdout:4866: delete oid 12 current snap is 0 2020-06-12T02:38:19.058 INFO:tasks.rados.rados.0.smithi107.stdout:4863: finishing write tid 7 to smithi10729407-14 2020-06-12T02:38:19.058 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 879 (ObjNum 4862 snap 0 seq_num 4862) dirty exists 2020-06-12T02:38:19.059 INFO:tasks.rados.rados.0.smithi107.stdout:4863: left oid 14 (ObjNum 4862 snap 0 seq_num 4862) 2020-06-12T02:38:19.060 INFO:tasks.rados.rados.0.smithi107.stdout:4863: done (2 left) 2020-06-12T02:38:19.060 INFO:tasks.rados.rados.0.smithi107.stdout:4866: done (1 left) 2020-06-12T02:38:19.061 INFO:tasks.rados.rados.0.smithi107.stdout:4867: delete oid 7 current snap is 0 2020-06-12T02:38:19.063 INFO:tasks.rados.rados.0.smithi107.stdout:4867: done (1 left) 2020-06-12T02:38:19.063 INFO:tasks.rados.rados.0.smithi107.stdout:4868: delete oid 16 current snap is 0 2020-06-12T02:38:19.065 INFO:tasks.rados.rados.0.smithi107.stdout:4868: done (1 left) 2020-06-12T02:38:19.065 INFO:tasks.rados.rados.0.smithi107.stdout:4869: delete oid 12 current snap is 0 2020-06-12T02:38:19.072 INFO:tasks.rados.rados.0.smithi107.stdout:4869: done (1 left) 2020-06-12T02:38:19.073 INFO:tasks.rados.rados.0.smithi107.stdout:4870: delete oid 6 current snap is 0 2020-06-12T02:38:19.073 INFO:tasks.rados.rados.0.smithi107.stdout:4870: done (1 left) 2020-06-12T02:38:19.073 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:19.073 INFO:tasks.rados.rados.0.smithi107.stdout:4871: seq_num 4870 ranges {0=770048,770048=499712,1269760=712704,1982464=393216} 2020-06-12T02:38:19.074 INFO:tasks.rados.rados.0.smithi107.stdout:4871: writing smithi10729407-16 from 0 to 770048 tid 1 2020-06-12T02:38:19.077 INFO:tasks.rados.rados.0.smithi107.stdout:4871: writing smithi10729407-16 from 770048 to 1269760 tid 2 2020-06-12T02:38:19.082 INFO:tasks.rados.rados.0.smithi107.stdout:4871: writing smithi10729407-16 from 1269760 to 1982464 tid 3 2020-06-12T02:38:19.084 INFO:tasks.rados.rados.0.smithi107.stdout:4871: writing smithi10729407-16 from 1982464 to 2375680 tid 4 2020-06-12T02:38:19.085 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:38:19.085 INFO:tasks.rados.rados.0.smithi107.stdout:4872: seq_num 4871 ranges {0=786432,786432=786432,1572864=524288,2097152=294912} 2020-06-12T02:38:19.089 INFO:tasks.rados.rados.0.smithi107.stdout:4872: writing smithi10729407-13 from 0 to 786432 tid 1 2020-06-12T02:38:19.095 INFO:tasks.rados.rados.0.smithi107.stdout:4872: writing smithi10729407-13 from 786432 to 1572864 tid 2 2020-06-12T02:38:19.098 INFO:tasks.rados.rados.0.smithi107.stdout:4872: writing smithi10729407-13 from 1572864 to 2097152 tid 3 2020-06-12T02:38:19.100 INFO:tasks.rados.rados.0.smithi107.stdout:4872: writing smithi10729407-13 from 2097152 to 2392064 tid 4 2020-06-12T02:38:19.101 INFO:tasks.rados.rados.0.smithi107.stdout:4873: delete oid 1 current snap is 0 2020-06-12T02:38:19.101 INFO:tasks.rados.rados.0.smithi107.stdout:4871: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:19.101 INFO:tasks.rados.rados.0.smithi107.stdout:4871: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:19.101 INFO:tasks.rados.rados.0.smithi107.stdout:4871: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:19.101 INFO:tasks.rados.rados.0.smithi107.stdout:4871: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:19.102 INFO:tasks.rados.rados.0.smithi107.stdout:4871: finishing write tid 6 to smithi10729407-16 2020-06-12T02:38:19.102 INFO:tasks.rados.rados.0.smithi107.stdout:4871: finishing write tid 7 to smithi10729407-16 2020-06-12T02:38:19.102 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 880 (ObjNum 4870 snap 0 seq_num 4870) dirty exists 2020-06-12T02:38:19.102 INFO:tasks.rados.rados.0.smithi107.stdout:4871: left oid 16 (ObjNum 4870 snap 0 seq_num 4870) 2020-06-12T02:38:19.102 INFO:tasks.rados.rados.0.smithi107.stdout:4872: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:19.104 INFO:tasks.rados.rados.0.smithi107.stdout:4871: done (3 left) 2020-06-12T02:38:19.104 INFO:tasks.rados.rados.0.smithi107.stdout:4873: done (2 left) 2020-06-12T02:38:19.104 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:19.105 INFO:tasks.rados.rados.0.smithi107.stdout:4874: seq_num 4873 ranges {4505600=16384} 2020-06-12T02:38:19.128 INFO:tasks.rados.rados.0.smithi107.stdout:4874: writing smithi10729407-8 from 4505600 to 4521984 tid 1 2020-06-12T02:38:19.129 INFO:tasks.rados.rados.0.smithi107.stdout:4875: delete oid 14 current snap is 0 2020-06-12T02:38:19.129 INFO:tasks.rados.rados.0.smithi107.stdout:4872: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:19.129 INFO:tasks.rados.rados.0.smithi107.stdout:4872: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:19.129 INFO:tasks.rados.rados.0.smithi107.stdout:4872: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:19.130 INFO:tasks.rados.rados.0.smithi107.stdout:4872: finishing write tid 6 to smithi10729407-13 2020-06-12T02:38:19.130 INFO:tasks.rados.rados.0.smithi107.stdout:4872: finishing write tid 7 to smithi10729407-13 2020-06-12T02:38:19.130 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 920 (ObjNum 4871 snap 0 seq_num 4871) dirty exists 2020-06-12T02:38:19.130 INFO:tasks.rados.rados.0.smithi107.stdout:4872: left oid 13 (ObjNum 4871 snap 0 seq_num 4871) 2020-06-12T02:38:19.131 INFO:tasks.rados.rados.0.smithi107.stdout:4872: done (3 left) 2020-06-12T02:38:19.132 INFO:tasks.rados.rados.0.smithi107.stdout:4875: done (2 left) 2020-06-12T02:38:19.132 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:19.132 INFO:tasks.rados.rados.0.smithi107.stdout:4876: seq_num 4875 ranges {0=49152} 2020-06-12T02:38:19.132 INFO:tasks.rados.rados.0.smithi107.stdout:4876: writing smithi10729407-12 from 0 to 49152 tid 1 2020-06-12T02:38:19.132 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:38:19.133 INFO:tasks.rados.rados.0.smithi107.stdout:4877: seq_num 4876 ranges {2375680=65536} 2020-06-12T02:38:19.133 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:19.128+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:19.146 INFO:tasks.rados.rados.0.smithi107.stdout:4877: writing smithi10729407-16 from 2375680 to 2441216 tid 1 2020-06-12T02:38:19.147 INFO:tasks.rados.rados.0.smithi107.stdout:4878: delete oid 13 current snap is 0 2020-06-12T02:38:19.147 INFO:tasks.rados.rados.0.smithi107.stdout:4874: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:19.147 INFO:tasks.rados.rados.0.smithi107.stdout:4874: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:19.147 INFO:tasks.rados.rados.0.smithi107.stdout:4876: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:19.147 INFO:tasks.rados.rados.0.smithi107.stdout:4876: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:19.148 INFO:tasks.rados.rados.0.smithi107.stdout:4874: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:19.148 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 834 (ObjNum 4873 snap 0 seq_num 4873) dirty exists 2020-06-12T02:38:19.148 INFO:tasks.rados.rados.0.smithi107.stdout:4874: left oid 8 (ObjNum 4873 snap 0 seq_num 4873) 2020-06-12T02:38:19.148 INFO:tasks.rados.rados.0.smithi107.stdout:4876: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:19.148 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 849 (ObjNum 4875 snap 0 seq_num 4875) dirty exists 2020-06-12T02:38:19.148 INFO:tasks.rados.rados.0.smithi107.stdout:4876: left oid 12 (ObjNum 4875 snap 0 seq_num 4875) 2020-06-12T02:38:19.149 INFO:tasks.rados.rados.0.smithi107.stdout:4874: done (4 left) 2020-06-12T02:38:19.149 INFO:tasks.rados.rados.0.smithi107.stdout:4876: done (3 left) 2020-06-12T02:38:19.150 INFO:tasks.rados.rados.0.smithi107.stdout:4878: done (2 left) 2020-06-12T02:38:19.150 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:38:19.150 INFO:tasks.rados.rados.0.smithi107.stdout:4879: seq_num 4878 ranges {2203648=98304} 2020-06-12T02:38:19.160 INFO:tasks.rados.rados.0.smithi107.stdout:4879: writing smithi10729407-11 from 2203648 to 2301952 tid 1 2020-06-12T02:38:19.160 INFO:tasks.rados.rados.0.smithi107.stdout:4880: delete oid 18 current snap is 0 2020-06-12T02:38:19.160 INFO:tasks.rados.rados.0.smithi107.stdout:4877: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:19.160 INFO:tasks.rados.rados.0.smithi107.stdout:4877: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:19.161 INFO:tasks.rados.rados.0.smithi107.stdout:4877: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:19.161 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 882 (ObjNum 4876 snap 0 seq_num 4876) dirty exists 2020-06-12T02:38:19.161 INFO:tasks.rados.rados.0.smithi107.stdout:4877: left oid 16 (ObjNum 4876 snap 0 seq_num 4876) 2020-06-12T02:38:19.162 INFO:tasks.rados.rados.0.smithi107.stdout:4877: done (3 left) 2020-06-12T02:38:19.162 INFO:tasks.rados.rados.0.smithi107.stdout:4880: done (2 left) 2020-06-12T02:38:19.162 INFO:tasks.rados.rados.0.smithi107.stdout:4881: delete oid 3 current snap is 0 2020-06-12T02:38:19.163 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:19.163 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:19.164 INFO:tasks.rados.rados.0.smithi107.stdout:4879: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:19.166 INFO:tasks.rados.rados.0.smithi107.stdout:4881: done (2 left) 2020-06-12T02:38:19.166 INFO:tasks.rados.rados.0.smithi107.stdout:4882: delete oid 14 current snap is 0 2020-06-12T02:38:19.167 INFO:tasks.rados.rados.0.smithi107.stdout:4879: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:19.167 INFO:tasks.rados.rados.0.smithi107.stdout:4882: done (2 left) 2020-06-12T02:38:19.168 INFO:tasks.rados.rados.0.smithi107.stdout:4883: delete oid 15 current snap is 0 2020-06-12T02:38:19.168 INFO:tasks.rados.rados.0.smithi107.stdout:4879: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:19.168 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 882 (ObjNum 4878 snap 0 seq_num 4878) dirty exists 2020-06-12T02:38:19.168 INFO:tasks.rados.rados.0.smithi107.stdout:4879: left oid 11 (ObjNum 4878 snap 0 seq_num 4878) 2020-06-12T02:38:19.169 INFO:tasks.rados.rados.0.smithi107.stdout:4879: done (2 left) 2020-06-12T02:38:19.169 INFO:tasks.rados.rados.0.smithi107.stdout:4883: done (1 left) 2020-06-12T02:38:19.169 INFO:tasks.rados.rados.0.smithi107.stdout:4884: delete oid 14 current snap is 0 2020-06-12T02:38:19.171 INFO:tasks.rados.rados.0.smithi107.stdout:4884: done (1 left) 2020-06-12T02:38:19.172 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:19.172 INFO:tasks.rados.rados.0.smithi107.stdout:4885: seq_num 4884 ranges {0=204800} 2020-06-12T02:38:19.173 INFO:tasks.rados.rados.0.smithi107.stdout:4885: writing smithi10729407-7 from 0 to 204800 tid 1 2020-06-12T02:38:19.173 INFO:tasks.rados.rados.0.smithi107.stdout:4886: delete oid 17 current snap is 0 2020-06-12T02:38:19.174 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-06-12T02:38:19.175 INFO:tasks.rados.rados.0.smithi107.stdout:4886: done (2 left) 2020-06-12T02:38:19.176 INFO:tasks.rados.rados.0.smithi107.stdout:4887: delete oid 12 current snap is 0 2020-06-12T02:38:19.178 INFO:tasks.rados.rados.0.smithi107.stdout:4885: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:19.179 INFO:tasks.rados.rados.0.smithi107.stdout:4885: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:19.179 INFO:tasks.rados.rados.0.smithi107.stdout:4887: done (2 left) 2020-06-12T02:38:19.180 INFO:tasks.rados.rados.0.smithi107.stdout:4888: delete oid 15 current snap is 0 2020-06-12T02:38:19.180 INFO:tasks.rados.rados.0.smithi107.stdout:4885: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:19.180 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 923 (ObjNum 4884 snap 0 seq_num 4884) dirty exists 2020-06-12T02:38:19.181 INFO:tasks.rados.rados.0.smithi107.stdout:4885: left oid 7 (ObjNum 4884 snap 0 seq_num 4884) 2020-06-12T02:38:19.181 INFO:tasks.rados.rados.0.smithi107.stdout:4885: done (2 left) 2020-06-12T02:38:19.182 INFO:tasks.rados.rados.0.smithi107.stdout:4888: done (1 left) 2020-06-12T02:38:19.182 INFO:tasks.rados.rados.0.smithi107.stdout:4889: delete oid 6 current snap is 0 2020-06-12T02:38:19.184 INFO:tasks.rados.rados.0.smithi107.stdout:4889: done (1 left) 2020-06-12T02:38:19.184 INFO:tasks.rados.rados.0.smithi107.stdout:4890: delete oid 1 current snap is 0 2020-06-12T02:38:19.185 INFO:tasks.rados.rados.0.smithi107.stdout:4890: done (1 left) 2020-06-12T02:38:19.185 INFO:tasks.rados.rados.0.smithi107.stdout:4891: delete oid 8 current snap is 0 2020-06-12T02:38:19.187 INFO:tasks.rados.rados.0.smithi107.stdout:4891: done (1 left) 2020-06-12T02:38:19.187 INFO:tasks.rados.rados.0.smithi107.stdout:4892: delete oid 11 current snap is 0 2020-06-12T02:38:19.192 INFO:tasks.rados.rados.0.smithi107.stdout:4892: done (1 left) 2020-06-12T02:38:19.192 INFO:tasks.rados.rados.0.smithi107.stdout:4893: delete oid 15 current snap is 0 2020-06-12T02:38:19.195 INFO:tasks.rados.rados.0.smithi107.stdout:4893: done (1 left) 2020-06-12T02:38:19.195 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:19.195 INFO:tasks.rados.rados.0.smithi107.stdout:4894: seq_num 4893 ranges {0=352256} 2020-06-12T02:38:19.198 INFO:tasks.rados.rados.0.smithi107.stdout:4894: writing smithi10729407-9 from 0 to 352256 tid 1 2020-06-12T02:38:19.198 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:19.198 INFO:tasks.rados.rados.0.smithi107.stdout:4895: seq_num 4894 ranges {0=368640} 2020-06-12T02:38:19.199 INFO:tasks.rados.rados.0.smithi107.stdout:4895: writing smithi10729407-1 from 0 to 368640 tid 1 2020-06-12T02:38:19.199 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:19.200 INFO:tasks.rados.rados.0.smithi107.stdout:4896: seq_num 4895 ranges {0=385024} 2020-06-12T02:38:19.202 INFO:tasks.rados.rados.0.smithi107.stdout:4896: writing smithi10729407-15 from 0 to 385024 tid 1 2020-06-12T02:38:19.202 INFO:tasks.rados.rados.0.smithi107.stdout:4897: delete oid 14 current snap is 0 2020-06-12T02:38:19.203 INFO:tasks.rados.rados.0.smithi107.stdout:4894: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:19.206 INFO:tasks.rados.rados.0.smithi107.stdout:4894: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:19.206 INFO:tasks.rados.rados.0.smithi107.stdout:4897: done (4 left) 2020-06-12T02:38:19.207 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:19.207 INFO:tasks.rados.rados.0.smithi107.stdout:4898: seq_num 4897 ranges {0=417792} 2020-06-12T02:38:19.209 INFO:tasks.rados.rados.0.smithi107.stdout:4898: writing smithi10729407-14 from 0 to 417792 tid 1 2020-06-12T02:38:19.209 INFO:tasks.rados.rados.0.smithi107.stdout:4899: delete oid 2 current snap is 0 2020-06-12T02:38:19.210 INFO:tasks.rados.rados.0.smithi107.stdout:4896: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:19.210 INFO:tasks.rados.rados.0.smithi107.stdout:4895: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:19.210 INFO:tasks.rados.rados.0.smithi107.stdout:4894: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:19.210 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 869 (ObjNum 4893 snap 0 seq_num 4893) dirty exists 2020-06-12T02:38:19.210 INFO:tasks.rados.rados.0.smithi107.stdout:4894: left oid 9 (ObjNum 4893 snap 0 seq_num 4893) 2020-06-12T02:38:19.211 INFO:tasks.rados.rados.0.smithi107.stdout:4896: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:19.211 INFO:tasks.rados.rados.0.smithi107.stdout:4896: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:19.212 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 840 (ObjNum 4895 snap 0 seq_num 4895) dirty exists 2020-06-12T02:38:19.212 INFO:tasks.rados.rados.0.smithi107.stdout:4896: left oid 15 (ObjNum 4895 snap 0 seq_num 4895) 2020-06-12T02:38:19.213 INFO:tasks.rados.rados.0.smithi107.stdout:4895: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:19.214 INFO:tasks.rados.rados.0.smithi107.stdout:4894: done (5 left) 2020-06-12T02:38:19.215 INFO:tasks.rados.rados.0.smithi107.stdout:4896: done (4 left) 2020-06-12T02:38:19.215 INFO:tasks.rados.rados.0.smithi107.stdout:4899: done (3 left) 2020-06-12T02:38:19.215 INFO:tasks.rados.rados.0.smithi107.stdout:4900: delete oid 12 current snap is 0 2020-06-12T02:38:19.216 INFO:tasks.rados.rados.0.smithi107.stdout:4895: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:19.216 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 916 (ObjNum 4894 snap 0 seq_num 4894) dirty exists 2020-06-12T02:38:19.216 INFO:tasks.rados.rados.0.smithi107.stdout:4895: left oid 1 (ObjNum 4894 snap 0 seq_num 4894) 2020-06-12T02:38:19.217 INFO:tasks.rados.rados.0.smithi107.stdout:4898: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:19.218 INFO:tasks.rados.rados.0.smithi107.stdout:4895: done (3 left) 2020-06-12T02:38:19.218 INFO:tasks.rados.rados.0.smithi107.stdout:4900: done (2 left) 2020-06-12T02:38:19.218 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:38:19.218 INFO:tasks.rados.rados.0.smithi107.stdout:4901: seq_num 4900 ranges {0=475136} 2020-06-12T02:38:19.220 INFO:tasks.rados.rados.0.smithi107.stdout:4901: writing smithi10729407-6 from 0 to 475136 tid 1 2020-06-12T02:38:19.221 INFO:tasks.rados.rados.0.smithi107.stdout:4902: delete oid 11 current snap is 0 2020-06-12T02:38:19.221 INFO:tasks.rados.rados.0.smithi107.stdout:4898: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:19.221 INFO:tasks.rados.rados.0.smithi107.stdout:4898: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:19.221 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 888 (ObjNum 4897 snap 0 seq_num 4897) dirty exists 2020-06-12T02:38:19.221 INFO:tasks.rados.rados.0.smithi107.stdout:4898: left oid 14 (ObjNum 4897 snap 0 seq_num 4897) 2020-06-12T02:38:19.225 INFO:tasks.rados.rados.0.smithi107.stdout:4898: done (3 left) 2020-06-12T02:38:19.226 INFO:tasks.rados.rados.0.smithi107.stdout:4902: done (2 left) 2020-06-12T02:38:19.226 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:38:19.226 INFO:tasks.rados.rados.0.smithi107.stdout:4903: seq_num 4902 ranges {0=507904} 2020-06-12T02:38:19.229 INFO:tasks.rados.rados.0.smithi107.stdout:4903: writing smithi10729407-3 from 0 to 507904 tid 1 2020-06-12T02:38:19.229 INFO:tasks.rados.rados.0.smithi107.stdout:4904: delete oid 16 current snap is 0 2020-06-12T02:38:19.229 INFO:tasks.rados.rados.0.smithi107.stdout:4901: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:19.230 INFO:tasks.rados.rados.0.smithi107.stdout:4901: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:19.230 INFO:tasks.rados.rados.0.smithi107.stdout:4901: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:19.230 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 918 (ObjNum 4900 snap 0 seq_num 4900) dirty exists 2020-06-12T02:38:19.230 INFO:tasks.rados.rados.0.smithi107.stdout:4901: left oid 6 (ObjNum 4900 snap 0 seq_num 4900) 2020-06-12T02:38:19.232 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:19.230+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:19.234 INFO:tasks.rados.rados.0.smithi107.stdout:4903: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:19.236 INFO:tasks.rados.rados.0.smithi107.stdout:4903: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:19.238 INFO:tasks.rados.rados.0.smithi107.stdout:4901: done (3 left) 2020-06-12T02:38:19.239 INFO:tasks.rados.rados.0.smithi107.stdout:4904: done (2 left) 2020-06-12T02:38:19.239 INFO:tasks.rados.rados.0.smithi107.stdout:4905: delete oid 14 current snap is 0 2020-06-12T02:38:19.240 INFO:tasks.rados.rados.0.smithi107.stdout:4903: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:19.240 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 885 (ObjNum 4902 snap 0 seq_num 4902) dirty exists 2020-06-12T02:38:19.240 INFO:tasks.rados.rados.0.smithi107.stdout:4903: left oid 3 (ObjNum 4902 snap 0 seq_num 4902) 2020-06-12T02:38:19.240 INFO:tasks.rados.rados.0.smithi107.stdout:4903: done (2 left) 2020-06-12T02:38:19.241 INFO:tasks.rados.rados.0.smithi107.stdout:4905: done (1 left) 2020-06-12T02:38:19.241 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:19.241 INFO:tasks.rados.rados.0.smithi107.stdout:4906: seq_num 4905 ranges {385024=557056} 2020-06-12T02:38:19.245 INFO:tasks.rados.rados.0.smithi107.stdout:4906: writing smithi10729407-15 from 385024 to 942080 tid 1 2020-06-12T02:38:19.245 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:19.246 INFO:tasks.rados.rados.0.smithi107.stdout:4907: seq_num 4906 ranges {0=573440} 2020-06-12T02:38:19.248 INFO:tasks.rados.rados.0.smithi107.stdout:4907: writing smithi10729407-13 from 0 to 573440 tid 1 2020-06-12T02:38:19.249 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:19.249 INFO:tasks.rados.rados.0.smithi107.stdout:4908: seq_num 4907 ranges {507904=589824} 2020-06-12T02:38:19.255 INFO:tasks.rados.rados.0.smithi107.stdout:4908: writing smithi10729407-3 from 507904 to 1097728 tid 1 2020-06-12T02:38:19.255 INFO:tasks.rados.rados.0.smithi107.stdout:4909: delete oid 12 current snap is 0 2020-06-12T02:38:19.255 INFO:tasks.rados.rados.0.smithi107.stdout:4906: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:19.255 INFO:tasks.rados.rados.0.smithi107.stdout:4906: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:19.256 INFO:tasks.rados.rados.0.smithi107.stdout:4906: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:19.256 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 842 (ObjNum 4905 snap 0 seq_num 4905) dirty exists 2020-06-12T02:38:19.256 INFO:tasks.rados.rados.0.smithi107.stdout:4906: left oid 15 (ObjNum 4905 snap 0 seq_num 4905) 2020-06-12T02:38:19.256 INFO:tasks.rados.rados.0.smithi107.stdout:4907: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:19.257 INFO:tasks.rados.rados.0.smithi107.stdout:4906: done (4 left) 2020-06-12T02:38:19.257 INFO:tasks.rados.rados.0.smithi107.stdout:4909: done (3 left) 2020-06-12T02:38:19.257 INFO:tasks.rados.rados.0.smithi107.stdout:4910: delete oid 8 current snap is 0 2020-06-12T02:38:19.259 INFO:tasks.rados.rados.0.smithi107.stdout:4910: done (3 left) 2020-06-12T02:38:19.259 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:38:19.260 INFO:tasks.rados.rados.0.smithi107.stdout:4911: seq_num 4910 ranges {0=638976} 2020-06-12T02:38:19.263 INFO:tasks.rados.rados.0.smithi107.stdout:4911: writing smithi10729407-5 from 0 to 638976 tid 1 2020-06-12T02:38:19.264 INFO:tasks.rados.rados.0.smithi107.stdout:4912: delete oid 6 current snap is 0 2020-06-12T02:38:19.264 INFO:tasks.rados.rados.0.smithi107.stdout:4907: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:19.264 INFO:tasks.rados.rados.0.smithi107.stdout:4908: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:19.264 INFO:tasks.rados.rados.0.smithi107.stdout:4907: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:19.264 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 925 (ObjNum 4906 snap 0 seq_num 4906) dirty exists 2020-06-12T02:38:19.265 INFO:tasks.rados.rados.0.smithi107.stdout:4907: left oid 13 (ObjNum 4906 snap 0 seq_num 4906) 2020-06-12T02:38:19.265 INFO:tasks.rados.rados.0.smithi107.stdout:4908: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:19.267 INFO:tasks.rados.rados.0.smithi107.stdout:4908: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:19.267 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 888 (ObjNum 4907 snap 0 seq_num 4907) dirty exists 2020-06-12T02:38:19.267 INFO:tasks.rados.rados.0.smithi107.stdout:4908: left oid 3 (ObjNum 4907 snap 0 seq_num 4907) 2020-06-12T02:38:19.267 INFO:tasks.rados.rados.0.smithi107.stdout:4907: done (4 left) 2020-06-12T02:38:19.268 INFO:tasks.rados.rados.0.smithi107.stdout:4908: done (3 left) 2020-06-12T02:38:19.268 INFO:tasks.rados.rados.0.smithi107.stdout:4912: done (2 left) 2020-06-12T02:38:19.268 INFO:tasks.rados.rados.0.smithi107.stdout:4913: delete oid 3 current snap is 0 2020-06-12T02:38:19.276 INFO:tasks.rados.rados.0.smithi107.stdout:4913: done (2 left) 2020-06-12T02:38:19.276 INFO:tasks.rados.rados.0.smithi107.stdout:4914: delete oid 12 current snap is 0 2020-06-12T02:38:19.276 INFO:tasks.rados.rados.0.smithi107.stdout:4911: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:19.277 INFO:tasks.rados.rados.0.smithi107.stdout:4914: done (2 left) 2020-06-12T02:38:19.277 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:19.277 INFO:tasks.rados.rados.0.smithi107.stdout:4915: seq_num 4914 ranges {573440=704512} 2020-06-12T02:38:19.282 INFO:tasks.rados.rados.0.smithi107.stdout:4915: writing smithi10729407-13 from 573440 to 1277952 tid 1 2020-06-12T02:38:19.282 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:38:19.283 INFO:tasks.rados.rados.0.smithi107.stdout:4916: seq_num 4915 ranges {3678208=720896} 2020-06-12T02:38:19.304 INFO:tasks.rados.rados.0.smithi107.stdout:4916: writing smithi10729407-20 from 3678208 to 4399104 tid 1 2020-06-12T02:38:19.304 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:38:19.305 INFO:tasks.rados.rados.0.smithi107.stdout:4917: seq_num 4916 ranges {0=737280} 2020-06-12T02:38:19.309 INFO:tasks.rados.rados.0.smithi107.stdout:4917: writing smithi10729407-16 from 0 to 737280 tid 1 2020-06-12T02:38:19.309 INFO:tasks.rados.rados.0.smithi107.stdout:4918: delete oid 4 current snap is 0 2020-06-12T02:38:19.309 INFO:tasks.rados.rados.0.smithi107.stdout:4911: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:19.310 INFO:tasks.rados.rados.0.smithi107.stdout:4911: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:19.310 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 474 (ObjNum 4910 snap 0 seq_num 4910) dirty exists 2020-06-12T02:38:19.310 INFO:tasks.rados.rados.0.smithi107.stdout:4911: left oid 5 (ObjNum 4910 snap 0 seq_num 4910) 2020-06-12T02:38:19.310 INFO:tasks.rados.rados.0.smithi107.stdout:4915: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:19.310 INFO:tasks.rados.rados.0.smithi107.stdout:4915: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:19.310 INFO:tasks.rados.rados.0.smithi107.stdout:4915: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:19.311 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 927 (ObjNum 4914 snap 0 seq_num 4914) dirty exists 2020-06-12T02:38:19.311 INFO:tasks.rados.rados.0.smithi107.stdout:4915: left oid 13 (ObjNum 4914 snap 0 seq_num 4914) 2020-06-12T02:38:19.312 INFO:tasks.rados.rados.0.smithi107.stdout:4911: done (5 left) 2020-06-12T02:38:19.312 INFO:tasks.rados.rados.0.smithi107.stdout:4915: done (4 left) 2020-06-12T02:38:19.312 INFO:tasks.rados.rados.0.smithi107.stdout:4918: done (3 left) 2020-06-12T02:38:19.313 INFO:tasks.rados.rados.0.smithi107.stdout:4919: delete oid 1 current snap is 0 2020-06-12T02:38:19.315 INFO:tasks.rados.rados.0.smithi107.stdout:4916: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:19.315 INFO:tasks.rados.rados.0.smithi107.stdout:4919: done (3 left) 2020-06-12T02:38:19.315 INFO:tasks.rados.rados.0.smithi107.stdout:4920: delete oid 14 current snap is 0 2020-06-12T02:38:19.318 INFO:tasks.rados.rados.0.smithi107.stdout:4917: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:19.318 INFO:tasks.rados.rados.0.smithi107.stdout:4917: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:19.319 INFO:tasks.rados.rados.0.smithi107.stdout:4917: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:19.320 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 891 (ObjNum 4916 snap 0 seq_num 4916) dirty exists 2020-06-12T02:38:19.320 INFO:tasks.rados.rados.0.smithi107.stdout:4917: left oid 16 (ObjNum 4916 snap 0 seq_num 4916) 2020-06-12T02:38:19.320 INFO:tasks.rados.rados.0.smithi107.stdout:4917: done (3 left) 2020-06-12T02:38:19.321 INFO:tasks.rados.rados.0.smithi107.stdout:4920: done (2 left) 2020-06-12T02:38:19.321 INFO:tasks.rados.rados.0.smithi107.stdout:4921: delete oid 9 current snap is 0 2020-06-12T02:38:19.321 INFO:tasks.rados.rados.0.smithi107.stdout:4916: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:19.322 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:19.322 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:19.322 INFO:tasks.rados.rados.0.smithi107.stdout:4916: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:19.323 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 375 (ObjNum 4915 snap 0 seq_num 4915) dirty exists 2020-06-12T02:38:19.323 INFO:tasks.rados.rados.0.smithi107.stdout:4916: left oid 20 (ObjNum 4915 snap 0 seq_num 4915) 2020-06-12T02:38:19.324 INFO:tasks.rados.rados.0.smithi107.stdout:4916: done (2 left) 2020-06-12T02:38:19.324 INFO:tasks.rados.rados.0.smithi107.stdout:4921: done (1 left) 2020-06-12T02:38:19.324 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:19.324 INFO:tasks.rados.rados.0.smithi107.stdout:4922: seq_num 4921 ranges {0=425984,425984=401408} 2020-06-12T02:38:19.325 INFO:tasks.rados.rados.0.smithi107.stdout:4922: writing smithi10729407-8 from 0 to 425984 tid 1 2020-06-12T02:38:19.328 INFO:tasks.rados.rados.0.smithi107.stdout:4922: writing smithi10729407-8 from 425984 to 827392 tid 2 2020-06-12T02:38:19.328 INFO:tasks.rados.rados.0.smithi107.stdout:4923: delete oid 20 current snap is 0 2020-06-12T02:38:19.331 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-06-12T02:38:19.335 INFO:tasks.rados.rados.0.smithi107.stdout:4922: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:19.335 INFO:tasks.rados.rados.0.smithi107.stdout:4922: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:19.335 INFO:tasks.rados.rados.0.smithi107.stdout:4922: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:19.336 INFO:tasks.rados.rados.0.smithi107.stdout:4923: done (2 left) 2020-06-12T02:38:19.336 INFO:tasks.rados.rados.0.smithi107.stdout:4924: delete oid 16 current snap is 0 2020-06-12T02:38:19.336 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:19.332+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:19.337 INFO:tasks.rados.rados.0.smithi107.stdout:4922: finishing write tid 5 to smithi10729407-8 2020-06-12T02:38:19.338 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 846 (ObjNum 4921 snap 0 seq_num 4921) dirty exists 2020-06-12T02:38:19.339 INFO:tasks.rados.rados.0.smithi107.stdout:4922: left oid 8 (ObjNum 4921 snap 0 seq_num 4921) 2020-06-12T02:38:19.340 INFO:tasks.rados.rados.0.smithi107.stdout:4922: done (2 left) 2020-06-12T02:38:19.340 INFO:tasks.rados.rados.0.smithi107.stdout:4924: done (1 left) 2020-06-12T02:38:19.340 INFO:tasks.rados.rados.0.smithi107.stdout:4925: delete oid 8 current snap is 0 2020-06-12T02:38:19.341 INFO:tasks.rados.rados.0.smithi107.stdout:4925: done (1 left) 2020-06-12T02:38:19.341 INFO:tasks.rados.rados.0.smithi107.stdout:4926: delete oid 15 current snap is 0 2020-06-12T02:38:19.345 INFO:tasks.rados.rados.0.smithi107.stdout:4926: done (1 left) 2020-06-12T02:38:19.346 INFO:tasks.rados.rados.0.smithi107.stdout:4927: delete oid 18 current snap is 0 2020-06-12T02:38:19.347 INFO:tasks.rados.rados.0.smithi107.stdout:4927: done (1 left) 2020-06-12T02:38:19.348 INFO:tasks.rados.rados.0.smithi107.stdout:4928: delete oid 19 current snap is 0 2020-06-12T02:38:19.349 INFO:tasks.rados.rados.0.smithi107.stdout:4928: done (1 left) 2020-06-12T02:38:19.350 INFO:tasks.rados.rados.0.smithi107.stdout:4929: delete oid 19 current snap is 0 2020-06-12T02:38:19.353 INFO:tasks.rados.rados.0.smithi107.stdout:4929: done (1 left) 2020-06-12T02:38:19.353 INFO:tasks.rados.rados.0.smithi107.stdout:4930: delete oid 12 current snap is 0 2020-06-12T02:38:19.354 INFO:tasks.rados.rados.0.smithi107.stdout:4930: done (1 left) 2020-06-12T02:38:19.354 INFO:tasks.rados.rados.0.smithi107.stdout:4931: delete oid 12 current snap is 0 2020-06-12T02:38:19.355 INFO:tasks.rados.rados.0.smithi107.stdout:4931: done (1 left) 2020-06-12T02:38:19.355 INFO:tasks.rados.rados.0.smithi107.stdout:4932: delete oid 17 current snap is 0 2020-06-12T02:38:19.357 INFO:tasks.rados.rados.0.smithi107.stdout:4932: done (1 left) 2020-06-12T02:38:19.357 INFO:tasks.rados.rados.0.smithi107.stdout:4933: delete oid 20 current snap is 0 2020-06-12T02:38:19.359 INFO:tasks.rados.rados.0.smithi107.stdout:4933: done (1 left) 2020-06-12T02:38:19.359 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:38:19.359 INFO:tasks.rados.rados.0.smithi107.stdout:4934: seq_num 4933 ranges {204800=622592,827392=401408} 2020-06-12T02:38:19.363 INFO:tasks.rados.rados.0.smithi107.stdout:4934: writing smithi10729407-7 from 204800 to 827392 tid 1 2020-06-12T02:38:19.366 INFO:tasks.rados.rados.0.smithi107.stdout:4934: writing smithi10729407-7 from 827392 to 1228800 tid 2 2020-06-12T02:38:19.366 INFO:tasks.rados.rados.0.smithi107.stdout:4935: delete oid 14 current snap is 0 2020-06-12T02:38:19.369 INFO:tasks.rados.rados.0.smithi107.stdout:4935: done (2 left) 2020-06-12T02:38:19.369 INFO:tasks.rados.rados.0.smithi107.stdout:4936: delete oid 6 current snap is 0 2020-06-12T02:38:19.372 INFO:tasks.rados.rados.0.smithi107.stdout:4936: done (2 left) 2020-06-12T02:38:19.373 INFO:tasks.rados.rados.0.smithi107.stdout:4937: delete oid 12 current snap is 0 2020-06-12T02:38:19.373 INFO:tasks.rados.rados.0.smithi107.stdout:4934: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:19.374 INFO:tasks.rados.rados.0.smithi107.stdout:4937: done (2 left) 2020-06-12T02:38:19.374 INFO:tasks.rados.rados.0.smithi107.stdout:4938: delete oid 18 current snap is 0 2020-06-12T02:38:19.377 INFO:tasks.rados.rados.0.smithi107.stdout:4938: done (2 left) 2020-06-12T02:38:19.377 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:19.377 INFO:tasks.rados.rados.0.smithi107.stdout:4939: seq_num 4938 ranges {0=704512,704512=401408} 2020-06-12T02:38:19.381 INFO:tasks.rados.rados.0.smithi107.stdout:4939: writing smithi10729407-14 from 0 to 704512 tid 1 2020-06-12T02:38:19.386 INFO:tasks.rados.rados.0.smithi107.stdout:4939: writing smithi10729407-14 from 704512 to 1105920 tid 2 2020-06-12T02:38:19.387 INFO:tasks.rados.rados.0.smithi107.stdout:4940: delete oid 16 current snap is 0 2020-06-12T02:38:19.387 INFO:tasks.rados.rados.0.smithi107.stdout:4934: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:19.387 INFO:tasks.rados.rados.0.smithi107.stdout:4934: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:19.387 INFO:tasks.rados.rados.0.smithi107.stdout:4934: finishing write tid 5 to smithi10729407-7 2020-06-12T02:38:19.387 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 930 (ObjNum 4933 snap 0 seq_num 4933) dirty exists 2020-06-12T02:38:19.388 INFO:tasks.rados.rados.0.smithi107.stdout:4934: left oid 7 (ObjNum 4933 snap 0 seq_num 4933) 2020-06-12T02:38:19.388 INFO:tasks.rados.rados.0.smithi107.stdout:4934: done (3 left) 2020-06-12T02:38:19.388 INFO:tasks.rados.rados.0.smithi107.stdout:4940: done (2 left) 2020-06-12T02:38:19.389 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:19.389 INFO:tasks.rados.rados.0.smithi107.stdout:4941: seq_num 4940 ranges {1277952=745472,2023424=401408} 2020-06-12T02:38:19.399 INFO:tasks.rados.rados.0.smithi107.stdout:4941: writing smithi10729407-13 from 1277952 to 2023424 tid 1 2020-06-12T02:38:19.402 INFO:tasks.rados.rados.0.smithi107.stdout:4941: writing smithi10729407-13 from 2023424 to 2424832 tid 2 2020-06-12T02:38:19.402 INFO:tasks.rados.rados.0.smithi107.stdout:4942: delete oid 15 current snap is 0 2020-06-12T02:38:19.402 INFO:tasks.rados.rados.0.smithi107.stdout:4939: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:19.403 INFO:tasks.rados.rados.0.smithi107.stdout:4939: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:19.403 INFO:tasks.rados.rados.0.smithi107.stdout:4939: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:19.403 INFO:tasks.rados.rados.0.smithi107.stdout:4939: finishing write tid 5 to smithi10729407-14 2020-06-12T02:38:19.403 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 895 (ObjNum 4938 snap 0 seq_num 4938) dirty exists 2020-06-12T02:38:19.403 INFO:tasks.rados.rados.0.smithi107.stdout:4939: left oid 14 (ObjNum 4938 snap 0 seq_num 4938) 2020-06-12T02:38:19.404 INFO:tasks.rados.rados.0.smithi107.stdout:4939: done (3 left) 2020-06-12T02:38:19.405 INFO:tasks.rados.rados.0.smithi107.stdout:4942: done (2 left) 2020-06-12T02:38:19.405 INFO:tasks.rados.rados.0.smithi107.stdout:4943: delete oid 7 current snap is 0 2020-06-12T02:38:19.406 INFO:tasks.rados.rados.0.smithi107.stdout:4941: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:19.410 INFO:tasks.rados.rados.0.smithi107.stdout:4943: done (2 left) 2020-06-12T02:38:19.410 INFO:tasks.rados.rados.0.smithi107.stdout:4944: delete oid 7 current snap is 0 2020-06-12T02:38:19.411 INFO:tasks.rados.rados.0.smithi107.stdout:4941: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:19.411 INFO:tasks.rados.rados.0.smithi107.stdout:4941: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:19.413 INFO:tasks.rados.rados.0.smithi107.stdout:4944: done (2 left) 2020-06-12T02:38:19.413 INFO:tasks.rados.rados.0.smithi107.stdout:4945: delete oid 9 current snap is 0 2020-06-12T02:38:19.414 INFO:tasks.rados.rados.0.smithi107.stdout:4941: finishing write tid 5 to smithi10729407-13 2020-06-12T02:38:19.414 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 934 (ObjNum 4940 snap 0 seq_num 4940) dirty exists 2020-06-12T02:38:19.414 INFO:tasks.rados.rados.0.smithi107.stdout:4941: left oid 13 (ObjNum 4940 snap 0 seq_num 4940) 2020-06-12T02:38:19.415 INFO:tasks.rados.rados.0.smithi107.stdout:4941: done (2 left) 2020-06-12T02:38:19.415 INFO:tasks.rados.rados.0.smithi107.stdout:4945: done (1 left) 2020-06-12T02:38:19.415 INFO:tasks.rados.rados.0.smithi107.stdout:4946: delete oid 7 current snap is 0 2020-06-12T02:38:19.417 INFO:tasks.rados.rados.0.smithi107.stdout:4946: done (1 left) 2020-06-12T02:38:19.417 INFO:tasks.rados.rados.0.smithi107.stdout:4947: delete oid 3 current snap is 0 2020-06-12T02:38:19.419 INFO:tasks.rados.rados.0.smithi107.stdout:4947: done (1 left) 2020-06-12T02:38:19.419 INFO:tasks.rados.rados.0.smithi107.stdout:4948: delete oid 11 current snap is 0 2020-06-12T02:38:19.421 INFO:tasks.rados.rados.0.smithi107.stdout:4948: done (1 left) 2020-06-12T02:38:19.421 INFO:tasks.rados.rados.0.smithi107.stdout:4949: delete oid 5 current snap is 0 2020-06-12T02:38:19.423 INFO:tasks.rados.rados.0.smithi107.stdout:4949: done (1 left) 2020-06-12T02:38:19.423 INFO:tasks.rados.rados.0.smithi107.stdout:4950: delete oid 19 current snap is 0 2020-06-12T02:38:19.427 INFO:tasks.rados.rados.0.smithi107.stdout:4950: done (1 left) 2020-06-12T02:38:19.427 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:19.427 INFO:tasks.rados.rados.0.smithi107.stdout:4951: seq_num 4950 ranges {0=507904,507904=589824,1097728=212992} 2020-06-12T02:38:19.429 INFO:tasks.rados.rados.0.smithi107.stdout:4951: writing smithi10729407-18 from 0 to 507904 tid 1 2020-06-12T02:38:19.433 INFO:tasks.rados.rados.0.smithi107.stdout:4951: writing smithi10729407-18 from 507904 to 1097728 tid 2 2020-06-12T02:38:19.434 INFO:tasks.rados.rados.0.smithi107.stdout:4951: writing smithi10729407-18 from 1097728 to 1310720 tid 3 2020-06-12T02:38:19.434 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:19.435 INFO:tasks.rados.rados.0.smithi107.stdout:4952: seq_num 4951 ranges {0=524288,524288=475136,999424=327680} 2020-06-12T02:38:19.437 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:19.434+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:19.438 INFO:tasks.rados.rados.0.smithi107.stdout:4952: writing smithi10729407-19 from 0 to 524288 tid 1 2020-06-12T02:38:19.441 INFO:tasks.rados.rados.0.smithi107.stdout:4952: writing smithi10729407-19 from 524288 to 999424 tid 2 2020-06-12T02:38:19.444 INFO:tasks.rados.rados.0.smithi107.stdout:4952: writing smithi10729407-19 from 999424 to 1327104 tid 3 2020-06-12T02:38:19.444 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:38:19.444 INFO:tasks.rados.rados.0.smithi107.stdout:4951: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:19.444 INFO:tasks.rados.rados.0.smithi107.stdout:4951: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:19.444 INFO:tasks.rados.rados.0.smithi107.stdout:4953: seq_num 4952 ranges {0=540672,540672=761856,1302528=40960} 2020-06-12T02:38:19.447 INFO:tasks.rados.rados.0.smithi107.stdout:4953: writing smithi10729407-7 from 0 to 540672 tid 1 2020-06-12T02:38:19.451 INFO:tasks.rados.rados.0.smithi107.stdout:4953: writing smithi10729407-7 from 540672 to 1302528 tid 2 2020-06-12T02:38:19.452 INFO:tasks.rados.rados.0.smithi107.stdout:4953: writing smithi10729407-7 from 1302528 to 1343488 tid 3 2020-06-12T02:38:19.452 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:19.452 INFO:tasks.rados.rados.0.smithi107.stdout:4954: seq_num 4953 ranges {0=557056,557056=647168,1204224=155648} 2020-06-12T02:38:19.455 INFO:tasks.rados.rados.0.smithi107.stdout:4954: writing smithi10729407-1 from 0 to 557056 tid 1 2020-06-12T02:38:19.459 INFO:tasks.rados.rados.0.smithi107.stdout:4954: writing smithi10729407-1 from 557056 to 1204224 tid 2 2020-06-12T02:38:19.460 INFO:tasks.rados.rados.0.smithi107.stdout:4954: writing smithi10729407-1 from 1204224 to 1359872 tid 3 2020-06-12T02:38:19.460 INFO:tasks.rados.rados.0.smithi107.stdout:4955: delete oid 3 current snap is 0 2020-06-12T02:38:19.461 INFO:tasks.rados.rados.0.smithi107.stdout:4951: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:19.461 INFO:tasks.rados.rados.0.smithi107.stdout:4951: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:19.461 INFO:tasks.rados.rados.0.smithi107.stdout:4951: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:19.461 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 877 (ObjNum 4950 snap 0 seq_num 4950) dirty exists 2020-06-12T02:38:19.462 INFO:tasks.rados.rados.0.smithi107.stdout:4951: left oid 18 (ObjNum 4950 snap 0 seq_num 4950) 2020-06-12T02:38:19.462 INFO:tasks.rados.rados.0.smithi107.stdout:4952: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:19.462 INFO:tasks.rados.rados.0.smithi107.stdout:4953: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:19.462 INFO:tasks.rados.rados.0.smithi107.stdout:4952: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:19.462 INFO:tasks.rados.rados.0.smithi107.stdout:4952: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:19.463 INFO:tasks.rados.rados.0.smithi107.stdout:4952: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:19.464 INFO:tasks.rados.rados.0.smithi107.stdout:4952: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:19.464 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 863 (ObjNum 4951 snap 0 seq_num 4951) dirty exists 2020-06-12T02:38:19.464 INFO:tasks.rados.rados.0.smithi107.stdout:4952: left oid 19 (ObjNum 4951 snap 0 seq_num 4951) 2020-06-12T02:38:19.465 INFO:tasks.rados.rados.0.smithi107.stdout:4954: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:19.466 INFO:tasks.rados.rados.0.smithi107.stdout:4953: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:19.467 INFO:tasks.rados.rados.0.smithi107.stdout:4951: done (5 left) 2020-06-12T02:38:19.467 INFO:tasks.rados.rados.0.smithi107.stdout:4952: done (4 left) 2020-06-12T02:38:19.467 INFO:tasks.rados.rados.0.smithi107.stdout:4955: done (3 left) 2020-06-12T02:38:19.468 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:19.468 INFO:tasks.rados.rados.0.smithi107.stdout:4956: seq_num 4955 ranges {1105920=589824,1695744=409600,2105344=393216} 2020-06-12T02:38:19.476 INFO:tasks.rados.rados.0.smithi107.stdout:4956: writing smithi10729407-14 from 1105920 to 1695744 tid 1 2020-06-12T02:38:19.477 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:19.477 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:19.485 INFO:tasks.rados.rados.0.smithi107.stdout:4956: writing smithi10729407-14 from 1695744 to 2105344 tid 2 2020-06-12T02:38:19.485 INFO:tasks.rados.rados.0.smithi107.stdout:4956: writing smithi10729407-14 from 2105344 to 2498560 tid 3 2020-06-12T02:38:19.485 INFO:tasks.rados.rados.0.smithi107.stdout:4957: delete oid 5 current snap is 0 2020-06-12T02:38:19.486 INFO:tasks.rados.rados.0.smithi107.stdout:4953: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:19.486 INFO:tasks.rados.rados.0.smithi107.stdout:4953: finishing write tid 5 to smithi10729407-7 2020-06-12T02:38:19.488 INFO:tasks.rados.rados.0.smithi107.stdout:4953: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:19.488 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 940 (ObjNum 4952 snap 0 seq_num 4952) dirty exists 2020-06-12T02:38:19.488 INFO:tasks.rados.rados.0.smithi107.stdout:4953: left oid 7 (ObjNum 4952 snap 0 seq_num 4952) 2020-06-12T02:38:19.488 INFO:tasks.rados.rados.0.smithi107.stdout:4954: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:19.489 INFO:tasks.rados.rados.0.smithi107.stdout:4954: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:19.489 INFO:tasks.rados.rados.0.smithi107.stdout:4954: finishing write tid 5 to smithi10729407-1 2020-06-12T02:38:19.489 INFO:tasks.rados.rados.0.smithi107.stdout:4954: finishing write tid 6 to smithi10729407-1 2020-06-12T02:38:19.489 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 925 (ObjNum 4953 snap 0 seq_num 4953) dirty exists 2020-06-12T02:38:19.489 INFO:tasks.rados.rados.0.smithi107.stdout:4954: left oid 1 (ObjNum 4953 snap 0 seq_num 4953) 2020-06-12T02:38:19.490 INFO:tasks.rados.rados.0.smithi107.stdout:4953: done (4 left) 2020-06-12T02:38:19.490 INFO:tasks.rados.rados.0.smithi107.stdout:4954: done (3 left) 2020-06-12T02:38:19.491 INFO:tasks.rados.rados.0.smithi107.stdout:4957: done (2 left) 2020-06-12T02:38:19.491 INFO:tasks.rados.rados.0.smithi107.stdout:4958: delete oid 2 current snap is 0 2020-06-12T02:38:19.492 INFO:tasks.rados.rados.0.smithi107.stdout:4956: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:19.493 INFO:tasks.rados.rados.0.smithi107.stdout:4958: done (2 left) 2020-06-12T02:38:19.493 INFO:tasks.rados.rados.0.smithi107.stdout:4959: delete oid 9 current snap is 0 2020-06-12T02:38:19.493 INFO:tasks.rados.rados.0.smithi107.stdout:4956: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:19.493 INFO:tasks.rados.rados.0.smithi107.stdout:4956: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:19.494 INFO:tasks.rados.rados.0.smithi107.stdout:4959: done (2 left) 2020-06-12T02:38:19.494 INFO:tasks.rados.rados.0.smithi107.stdout:4960: delete oid 20 current snap is 0 2020-06-12T02:38:19.494 INFO:tasks.rados.rados.0.smithi107.stdout:4956: finishing write tid 5 to smithi10729407-14 2020-06-12T02:38:19.495 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-06-12T02:38:19.496 INFO:tasks.rados.rados.0.smithi107.stdout:4956: finishing write tid 6 to smithi10729407-14 2020-06-12T02:38:19.496 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 900 (ObjNum 4955 snap 0 seq_num 4955) dirty exists 2020-06-12T02:38:19.497 INFO:tasks.rados.rados.0.smithi107.stdout:4956: left oid 14 (ObjNum 4955 snap 0 seq_num 4955) 2020-06-12T02:38:19.497 INFO:tasks.rados.rados.0.smithi107.stdout:4956: done (2 left) 2020-06-12T02:38:19.497 INFO:tasks.rados.rados.0.smithi107.stdout:4960: done (1 left) 2020-06-12T02:38:19.497 INFO:tasks.rados.rados.0.smithi107.stdout:4961: delete oid 16 current snap is 0 2020-06-12T02:38:19.497 INFO:tasks.rados.rados.0.smithi107.stdout:4961: done (1 left) 2020-06-12T02:38:19.498 INFO:tasks.rados.rados.0.smithi107.stdout:4962: delete oid 7 current snap is 0 2020-06-12T02:38:19.498 INFO:tasks.rados.rados.0.smithi107.stdout:4962: done (1 left) 2020-06-12T02:38:19.498 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:38:19.499 INFO:tasks.rados.rados.0.smithi107.stdout:4963: seq_num 4962 ranges {0=712704,712704=450560,1163264=352256} 2020-06-12T02:38:19.502 INFO:tasks.rados.rados.0.smithi107.stdout:4963: writing smithi10729407-7 from 0 to 712704 tid 1 2020-06-12T02:38:19.505 INFO:tasks.rados.rados.0.smithi107.stdout:4963: writing smithi10729407-7 from 712704 to 1163264 tid 2 2020-06-12T02:38:19.507 INFO:tasks.rados.rados.0.smithi107.stdout:4963: writing smithi10729407-7 from 1163264 to 1515520 tid 3 2020-06-12T02:38:19.507 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:19.508 INFO:tasks.rados.rados.0.smithi107.stdout:4964: seq_num 4963 ranges {0=729088,729088=737280,1466368=65536} 2020-06-12T02:38:19.512 INFO:tasks.rados.rados.0.smithi107.stdout:4964: writing smithi10729407-8 from 0 to 729088 tid 1 2020-06-12T02:38:19.516 INFO:tasks.rados.rados.0.smithi107.stdout:4964: writing smithi10729407-8 from 729088 to 1466368 tid 2 2020-06-12T02:38:19.517 INFO:tasks.rados.rados.0.smithi107.stdout:4964: writing smithi10729407-8 from 1466368 to 1531904 tid 3 2020-06-12T02:38:19.517 INFO:tasks.rados.rados.0.smithi107.stdout:4965: delete oid 4 current snap is 0 2020-06-12T02:38:19.518 INFO:tasks.rados.rados.0.smithi107.stdout:4963: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:19.518 INFO:tasks.rados.rados.0.smithi107.stdout:4963: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:19.518 INFO:tasks.rados.rados.0.smithi107.stdout:4963: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:19.518 INFO:tasks.rados.rados.0.smithi107.stdout:4963: finishing write tid 5 to smithi10729407-7 2020-06-12T02:38:19.522 INFO:tasks.rados.rados.0.smithi107.stdout:4963: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:19.522 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 945 (ObjNum 4962 snap 0 seq_num 4962) dirty exists 2020-06-12T02:38:19.522 INFO:tasks.rados.rados.0.smithi107.stdout:4963: left oid 7 (ObjNum 4962 snap 0 seq_num 4962) 2020-06-12T02:38:19.523 INFO:tasks.rados.rados.0.smithi107.stdout:4964: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:19.528 INFO:tasks.rados.rados.0.smithi107.stdout:4963: done (3 left) 2020-06-12T02:38:19.528 INFO:tasks.rados.rados.0.smithi107.stdout:4965: done (2 left) 2020-06-12T02:38:19.529 INFO:tasks.rados.rados.0.smithi107.stdout:4966: delete oid 19 current snap is 0 2020-06-12T02:38:19.534 INFO:tasks.rados.rados.0.smithi107.stdout:4964: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:19.535 INFO:tasks.rados.rados.0.smithi107.stdout:4966: done (2 left) 2020-06-12T02:38:19.536 INFO:tasks.rados.rados.0.smithi107.stdout:4967: delete oid 11 current snap is 0 2020-06-12T02:38:19.537 INFO:tasks.rados.rados.0.smithi107.stdout:4964: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:19.538 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:19.534+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:19.538 INFO:tasks.rados.rados.0.smithi107.stdout:4964: finishing write tid 5 to smithi10729407-8 2020-06-12T02:38:19.539 INFO:tasks.rados.rados.0.smithi107.stdout:4967: done (2 left) 2020-06-12T02:38:19.539 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:19.539 INFO:tasks.rados.rados.0.smithi107.stdout:4968: seq_num 4967 ranges {0=794624,794624=720896,1515520=81920} 2020-06-12T02:38:19.543 INFO:tasks.rados.rados.0.smithi107.stdout:4968: writing smithi10729407-5 from 0 to 794624 tid 1 2020-06-12T02:38:19.548 INFO:tasks.rados.rados.0.smithi107.stdout:4968: writing smithi10729407-5 from 794624 to 1515520 tid 2 2020-06-12T02:38:19.549 INFO:tasks.rados.rados.0.smithi107.stdout:4968: writing smithi10729407-5 from 1515520 to 1597440 tid 3 2020-06-12T02:38:19.549 INFO:tasks.rados.rados.0.smithi107.stdout:4969: delete oid 7 current snap is 0 2020-06-12T02:38:19.549 INFO:tasks.rados.rados.0.smithi107.stdout:4964: finishing write tid 6 to smithi10729407-8 2020-06-12T02:38:19.549 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 853 (ObjNum 4963 snap 0 seq_num 4963) dirty exists 2020-06-12T02:38:19.549 INFO:tasks.rados.rados.0.smithi107.stdout:4964: left oid 8 (ObjNum 4963 snap 0 seq_num 4963) 2020-06-12T02:38:19.550 INFO:tasks.rados.rados.0.smithi107.stdout:4968: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:19.551 INFO:tasks.rados.rados.0.smithi107.stdout:4964: done (3 left) 2020-06-12T02:38:19.551 INFO:tasks.rados.rados.0.smithi107.stdout:4969: done (2 left) 2020-06-12T02:38:19.551 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:19.552 INFO:tasks.rados.rados.0.smithi107.stdout:4970: seq_num 4969 ranges {0=425984,425984=491520,917504=466944,1384448=245760} 2020-06-12T02:38:19.553 INFO:tasks.rados.rados.0.smithi107.stdout:4970: writing smithi10729407-17 from 0 to 425984 tid 1 2020-06-12T02:38:19.556 INFO:tasks.rados.rados.0.smithi107.stdout:4970: writing smithi10729407-17 from 425984 to 917504 tid 2 2020-06-12T02:38:19.559 INFO:tasks.rados.rados.0.smithi107.stdout:4970: writing smithi10729407-17 from 917504 to 1384448 tid 3 2020-06-12T02:38:19.561 INFO:tasks.rados.rados.0.smithi107.stdout:4970: writing smithi10729407-17 from 1384448 to 1630208 tid 4 2020-06-12T02:38:19.561 INFO:tasks.rados.rados.0.smithi107.stdout:4971: delete oid 16 current snap is 0 2020-06-12T02:38:19.561 INFO:tasks.rados.rados.0.smithi107.stdout:4968: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:19.561 INFO:tasks.rados.rados.0.smithi107.stdout:4968: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:19.561 INFO:tasks.rados.rados.0.smithi107.stdout:4968: finishing write tid 5 to smithi10729407-5 2020-06-12T02:38:19.562 INFO:tasks.rados.rados.0.smithi107.stdout:4970: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:19.562 INFO:tasks.rados.rados.0.smithi107.stdout:4968: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:19.562 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 480 (ObjNum 4967 snap 0 seq_num 4967) dirty exists 2020-06-12T02:38:19.562 INFO:tasks.rados.rados.0.smithi107.stdout:4968: left oid 5 (ObjNum 4967 snap 0 seq_num 4967) 2020-06-12T02:38:19.563 INFO:tasks.rados.rados.0.smithi107.stdout:4968: done (3 left) 2020-06-12T02:38:19.563 INFO:tasks.rados.rados.0.smithi107.stdout:4971: done (2 left) 2020-06-12T02:38:19.563 INFO:tasks.rados.rados.0.smithi107.stdout:4972: delete oid 1 current snap is 0 2020-06-12T02:38:19.567 INFO:tasks.rados.rados.0.smithi107.stdout:4972: done (2 left) 2020-06-12T02:38:19.567 INFO:tasks.rados.rados.0.smithi107.stdout:4973: delete oid 8 current snap is 0 2020-06-12T02:38:19.574 INFO:tasks.rados.rados.0.smithi107.stdout:4973: done (2 left) 2020-06-12T02:38:19.575 INFO:tasks.rados.rados.0.smithi107.stdout:4974: delete oid 9 current snap is 0 2020-06-12T02:38:19.575 INFO:tasks.rados.rados.0.smithi107.stdout:4970: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:19.575 INFO:tasks.rados.rados.0.smithi107.stdout:4974: done (2 left) 2020-06-12T02:38:19.575 INFO:tasks.rados.rados.0.smithi107.stdout:4975: delete oid 12 current snap is 0 2020-06-12T02:38:19.576 INFO:tasks.rados.rados.0.smithi107.stdout:4970: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:19.576 INFO:tasks.rados.rados.0.smithi107.stdout:4970: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:19.576 INFO:tasks.rados.rados.0.smithi107.stdout:4970: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:19.577 INFO:tasks.rados.rados.0.smithi107.stdout:4975: done (2 left) 2020-06-12T02:38:19.577 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:19.577 INFO:tasks.rados.rados.0.smithi107.stdout:4976: seq_num 4975 ranges {0=524288,524288=647168,1171456=458752,1630208=98304} 2020-06-12T02:38:19.578 INFO:tasks.rados.rados.0.smithi107.stdout:4976: writing smithi10729407-9 from 0 to 524288 tid 1 2020-06-12T02:38:19.582 INFO:tasks.rados.rados.0.smithi107.stdout:4976: writing smithi10729407-9 from 524288 to 1171456 tid 2 2020-06-12T02:38:19.585 INFO:tasks.rados.rados.0.smithi107.stdout:4976: writing smithi10729407-9 from 1171456 to 1630208 tid 3 2020-06-12T02:38:19.589 INFO:tasks.rados.rados.0.smithi107.stdout:4976: writing smithi10729407-9 from 1630208 to 1728512 tid 4 2020-06-12T02:38:19.590 INFO:tasks.rados.rados.0.smithi107.stdout:4977: delete oid 18 current snap is 0 2020-06-12T02:38:19.590 INFO:tasks.rados.rados.0.smithi107.stdout:4970: finishing write tid 7 to smithi10729407-17 2020-06-12T02:38:19.590 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 924 (ObjNum 4969 snap 0 seq_num 4969) dirty exists 2020-06-12T02:38:19.590 INFO:tasks.rados.rados.0.smithi107.stdout:4970: left oid 17 (ObjNum 4969 snap 0 seq_num 4969) 2020-06-12T02:38:19.590 INFO:tasks.rados.rados.0.smithi107.stdout:4976: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:19.592 INFO:tasks.rados.rados.0.smithi107.stdout:4976: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:19.593 INFO:tasks.rados.rados.0.smithi107.stdout:4976: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:19.593 INFO:tasks.rados.rados.0.smithi107.stdout:4976: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:19.596 INFO:tasks.rados.rados.0.smithi107.stdout:4976: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:19.596 INFO:tasks.rados.rados.0.smithi107.stdout:4970: done (3 left) 2020-06-12T02:38:19.597 INFO:tasks.rados.rados.0.smithi107.stdout:4977: done (2 left) 2020-06-12T02:38:19.597 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:19.597 INFO:tasks.rados.rados.0.smithi107.stdout:4978: seq_num 4977 ranges {0=557056,557056=417792,974848=425984,1400832=360448} 2020-06-12T02:38:19.601 INFO:tasks.rados.rados.0.smithi107.stdout:4978: writing smithi10729407-15 from 0 to 557056 tid 1 2020-06-12T02:38:19.602 INFO:tasks.rados.rados.0.smithi107.stdout:4978: writing smithi10729407-15 from 557056 to 974848 tid 2 2020-06-12T02:38:19.605 INFO:tasks.rados.rados.0.smithi107.stdout:4978: writing smithi10729407-15 from 974848 to 1400832 tid 3 2020-06-12T02:38:19.607 INFO:tasks.rados.rados.0.smithi107.stdout:4978: writing smithi10729407-15 from 1400832 to 1761280 tid 4 2020-06-12T02:38:19.607 INFO:tasks.rados.rados.0.smithi107.stdout:4979: delete oid 8 current snap is 0 2020-06-12T02:38:19.608 INFO:tasks.rados.rados.0.smithi107.stdout:4976: finishing write tid 7 to smithi10729407-9 2020-06-12T02:38:19.608 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 884 (ObjNum 4975 snap 0 seq_num 4975) dirty exists 2020-06-12T02:38:19.608 INFO:tasks.rados.rados.0.smithi107.stdout:4976: left oid 9 (ObjNum 4975 snap 0 seq_num 4975) 2020-06-12T02:38:19.613 INFO:tasks.rados.rados.0.smithi107.stdout:4978: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:19.614 INFO:tasks.rados.rados.0.smithi107.stdout:4978: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:19.614 INFO:tasks.rados.rados.0.smithi107.stdout:4978: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:19.619 INFO:tasks.rados.rados.0.smithi107.stdout:4978: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:19.620 INFO:tasks.rados.rados.0.smithi107.stdout:4978: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:19.621 INFO:tasks.rados.rados.0.smithi107.stdout:4978: finishing write tid 7 to smithi10729407-15 2020-06-12T02:38:19.622 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 859 (ObjNum 4977 snap 0 seq_num 4977) dirty exists 2020-06-12T02:38:19.622 INFO:tasks.rados.rados.0.smithi107.stdout:4978: left oid 15 (ObjNum 4977 snap 0 seq_num 4977) 2020-06-12T02:38:19.622 INFO:tasks.rados.rados.0.smithi107.stdout:4976: done (3 left) 2020-06-12T02:38:19.622 INFO:tasks.rados.rados.0.smithi107.stdout:4978: done (2 left) 2020-06-12T02:38:19.622 INFO:tasks.rados.rados.0.smithi107.stdout:4979: done (1 left) 2020-06-12T02:38:19.623 INFO:tasks.rados.rados.0.smithi107.stdout:4980: delete oid 11 current snap is 0 2020-06-12T02:38:19.625 INFO:tasks.rados.rados.0.smithi107.stdout:4980: done (1 left) 2020-06-12T02:38:19.625 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:38:19.625 INFO:tasks.rados.rados.0.smithi107.stdout:4981: seq_num 4980 ranges {0=614400,614400=516096,1130496=598016,1728512=90112} 2020-06-12T02:38:19.627 INFO:tasks.rados.rados.0.smithi107.stdout:4981: writing smithi10729407-1 from 0 to 614400 tid 1 2020-06-12T02:38:19.631 INFO:tasks.rados.rados.0.smithi107.stdout:4981: writing smithi10729407-1 from 614400 to 1130496 tid 2 2020-06-12T02:38:19.632 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:19.633 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:19.635 INFO:tasks.rados.rados.0.smithi107.stdout:4981: writing smithi10729407-1 from 1130496 to 1728512 tid 3 2020-06-12T02:38:19.635 INFO:tasks.rados.rados.0.smithi107.stdout:4981: writing smithi10729407-1 from 1728512 to 1818624 tid 4 2020-06-12T02:38:19.636 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:19.636 INFO:tasks.rados.rados.0.smithi107.stdout:4982: seq_num 4981 ranges {0=630784,630784=802816,1433600=401408} 2020-06-12T02:38:19.639 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:19.635+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:19.640 INFO:tasks.rados.rados.0.smithi107.stdout:4982: writing smithi10729407-2 from 0 to 630784 tid 1 2020-06-12T02:38:19.643 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-06-12T02:38:19.644 INFO:tasks.rados.rados.0.smithi107.stdout:4982: writing smithi10729407-2 from 630784 to 1433600 tid 2 2020-06-12T02:38:19.648 INFO:tasks.rados.rados.0.smithi107.stdout:4982: writing smithi10729407-2 from 1433600 to 1835008 tid 3 2020-06-12T02:38:19.648 INFO:tasks.rados.rados.0.smithi107.stdout:4983: delete oid 3 current snap is 0 2020-06-12T02:38:19.648 INFO:tasks.rados.rados.0.smithi107.stdout:4981: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:19.648 INFO:tasks.rados.rados.0.smithi107.stdout:4981: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:19.649 INFO:tasks.rados.rados.0.smithi107.stdout:4981: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:19.649 INFO:tasks.rados.rados.0.smithi107.stdout:4981: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:19.649 INFO:tasks.rados.rados.0.smithi107.stdout:4981: finishing write tid 6 to smithi10729407-1 2020-06-12T02:38:19.649 INFO:tasks.rados.rados.0.smithi107.stdout:4981: finishing write tid 7 to smithi10729407-1 2020-06-12T02:38:19.649 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 931 (ObjNum 4980 snap 0 seq_num 4980) dirty exists 2020-06-12T02:38:19.650 INFO:tasks.rados.rados.0.smithi107.stdout:4981: left oid 1 (ObjNum 4980 snap 0 seq_num 4980) 2020-06-12T02:38:19.650 INFO:tasks.rados.rados.0.smithi107.stdout:4981: done (3 left) 2020-06-12T02:38:19.650 INFO:tasks.rados.rados.0.smithi107.stdout:4983: done (2 left) 2020-06-12T02:38:19.651 INFO:tasks.rados.rados.0.smithi107.stdout:4984: delete oid 5 current snap is 0 2020-06-12T02:38:19.651 INFO:tasks.rados.rados.0.smithi107.stdout:4982: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:19.652 INFO:tasks.rados.rados.0.smithi107.stdout:4984: done (2 left) 2020-06-12T02:38:19.652 INFO:tasks.rados.rados.0.smithi107.stdout:4985: delete oid 4 current snap is 0 2020-06-12T02:38:19.655 INFO:tasks.rados.rados.0.smithi107.stdout:4982: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:19.656 INFO:tasks.rados.rados.0.smithi107.stdout:4982: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:19.656 INFO:tasks.rados.rados.0.smithi107.stdout:4982: finishing write tid 5 to smithi10729407-2 2020-06-12T02:38:19.656 INFO:tasks.rados.rados.0.smithi107.stdout:4985: done (2 left) 2020-06-12T02:38:19.656 INFO:tasks.rados.rados.0.smithi107.stdout:4986: delete oid 17 current snap is 0 2020-06-12T02:38:19.657 INFO:tasks.rados.rados.0.smithi107.stdout:4982: finishing write tid 6 to smithi10729407-2 2020-06-12T02:38:19.657 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 481 (ObjNum 4981 snap 0 seq_num 4981) dirty exists 2020-06-12T02:38:19.657 INFO:tasks.rados.rados.0.smithi107.stdout:4982: left oid 2 (ObjNum 4981 snap 0 seq_num 4981) 2020-06-12T02:38:19.659 INFO:tasks.rados.rados.0.smithi107.stdout:4982: done (2 left) 2020-06-12T02:38:19.659 INFO:tasks.rados.rados.0.smithi107.stdout:4986: done (1 left) 2020-06-12T02:38:19.659 INFO:tasks.rados.rados.0.smithi107.stdout:4987: delete oid 5 current snap is 0 2020-06-12T02:38:19.661 INFO:tasks.rados.rados.0.smithi107.stdout:4987: done (1 left) 2020-06-12T02:38:19.661 INFO:tasks.rados.rados.0.smithi107.stdout:4988: delete oid 8 current snap is 0 2020-06-12T02:38:19.668 INFO:tasks.rados.rados.0.smithi107.stdout:4988: done (1 left) 2020-06-12T02:38:19.668 INFO:tasks.rados.rados.0.smithi107.stdout:4989: delete oid 2 current snap is 0 2020-06-12T02:38:19.668 INFO:tasks.rados.rados.0.smithi107.stdout:4989: done (1 left) 2020-06-12T02:38:19.668 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:38:19.668 INFO:tasks.rados.rados.0.smithi107.stdout:4990: seq_num 4989 ranges {0=761856,761856=729088,1490944=475136} 2020-06-12T02:38:19.670 INFO:tasks.rados.rados.0.smithi107.stdout:4990: writing smithi10729407-7 from 0 to 761856 tid 1 2020-06-12T02:38:19.675 INFO:tasks.rados.rados.0.smithi107.stdout:4990: writing smithi10729407-7 from 761856 to 1490944 tid 2 2020-06-12T02:38:19.678 INFO:tasks.rados.rados.0.smithi107.stdout:4990: writing smithi10729407-7 from 1490944 to 1966080 tid 3 2020-06-12T02:38:19.679 INFO:tasks.rados.rados.0.smithi107.stdout:4991: delete oid 4 current snap is 0 2020-06-12T02:38:19.679 INFO:tasks.rados.rados.0.smithi107.stdout:4990: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:19.689 INFO:tasks.rados.rados.0.smithi107.stdout:4991: done (2 left) 2020-06-12T02:38:19.689 INFO:tasks.rados.rados.0.smithi107.stdout:4992: delete oid 12 current snap is 0 2020-06-12T02:38:19.689 INFO:tasks.rados.rados.0.smithi107.stdout:4992: done (2 left) 2020-06-12T02:38:19.690 INFO:tasks.rados.rados.0.smithi107.stdout:4993: delete oid 11 current snap is 0 2020-06-12T02:38:19.690 INFO:tasks.rados.rados.0.smithi107.stdout:4993: done (2 left) 2020-06-12T02:38:19.690 INFO:tasks.rados.rados.0.smithi107.stdout:4994: delete oid 11 current snap is 0 2020-06-12T02:38:19.690 INFO:tasks.rados.rados.0.smithi107.stdout:4990: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:19.690 INFO:tasks.rados.rados.0.smithi107.stdout:4990: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:19.691 INFO:tasks.rados.rados.0.smithi107.stdout:4990: finishing write tid 5 to smithi10729407-7 2020-06-12T02:38:19.691 INFO:tasks.rados.rados.0.smithi107.stdout:4990: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:19.691 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 950 (ObjNum 4989 snap 0 seq_num 4989) dirty exists 2020-06-12T02:38:19.692 INFO:tasks.rados.rados.0.smithi107.stdout:4990: left oid 7 (ObjNum 4989 snap 0 seq_num 4989) 2020-06-12T02:38:19.692 INFO:tasks.rados.rados.0.smithi107.stdout:4990: done (2 left) 2020-06-12T02:38:19.692 INFO:tasks.rados.rados.0.smithi107.stdout:4994: done (1 left) 2020-06-12T02:38:19.692 INFO:tasks.rados.rados.0.smithi107.stdout:4995: delete oid 2 current snap is 0 2020-06-12T02:38:19.694 INFO:tasks.rados.rados.0.smithi107.stdout:4995: done (1 left) 2020-06-12T02:38:19.695 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:19.695 INFO:tasks.rados.rados.0.smithi107.stdout:4996: seq_num 4995 ranges {2498560=458752,2957312=483328,3440640=761856,4202496=360448} 2020-06-12T02:38:19.710 INFO:tasks.rados.rados.0.smithi107.stdout:4996: writing smithi10729407-14 from 2498560 to 2957312 tid 1 2020-06-12T02:38:19.713 INFO:tasks.rados.rados.0.smithi107.stdout:4996: writing smithi10729407-14 from 2957312 to 3440640 tid 2 2020-06-12T02:38:19.719 INFO:tasks.rados.rados.0.smithi107.stdout:4996: writing smithi10729407-14 from 3440640 to 4202496 tid 3 2020-06-12T02:38:19.721 INFO:tasks.rados.rados.0.smithi107.stdout:4996: writing smithi10729407-14 from 4202496 to 4562944 tid 4 2020-06-12T02:38:19.721 INFO:tasks.rados.rados.0.smithi107.stdout:4997: delete oid 12 current snap is 0 2020-06-12T02:38:19.721 INFO:tasks.rados.rados.0.smithi107.stdout:4996: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:19.724 INFO:tasks.rados.rados.0.smithi107.stdout:4997: done (2 left) 2020-06-12T02:38:19.724 INFO:tasks.rados.rados.0.smithi107.stdout:4998: delete oid 12 current snap is 0 2020-06-12T02:38:19.728 INFO:tasks.rados.rados.0.smithi107.stdout:4996: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:19.729 INFO:tasks.rados.rados.0.smithi107.stdout:4996: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:19.729 INFO:tasks.rados.rados.0.smithi107.stdout:4998: done (2 left) 2020-06-12T02:38:19.729 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:19.730 INFO:tasks.rados.rados.0.smithi107.stdout:4999: seq_num 4998 ranges {0=516096,516096=532480,1048576=540672,1589248=532480} 2020-06-12T02:38:19.731 INFO:tasks.rados.rados.0.smithi107.stdout:4999: writing smithi10729407-8 from 0 to 516096 tid 1 2020-06-12T02:38:19.734 INFO:tasks.rados.rados.0.smithi107.stdout:4999: writing smithi10729407-8 from 516096 to 1048576 tid 2 2020-06-12T02:38:19.738 INFO:tasks.rados.rados.0.smithi107.stdout:4999: writing smithi10729407-8 from 1048576 to 1589248 tid 3 2020-06-12T02:38:19.740 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:19.737+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:19.741 INFO:tasks.rados.rados.0.smithi107.stdout:4999: writing smithi10729407-8 from 1589248 to 2121728 tid 4 2020-06-12T02:38:19.741 INFO:tasks.rados.rados.0.smithi107.stdout:5000: delete oid 15 current snap is 0 2020-06-12T02:38:19.741 INFO:tasks.rados.rados.0.smithi107.stdout:4996: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:19.742 INFO:tasks.rados.rados.0.smithi107.stdout:4996: finishing write tid 6 to smithi10729407-14 2020-06-12T02:38:19.742 INFO:tasks.rados.rados.0.smithi107.stdout:4996: finishing write tid 7 to smithi10729407-14 2020-06-12T02:38:19.742 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 909 (ObjNum 4995 snap 0 seq_num 4995) dirty exists 2020-06-12T02:38:19.742 INFO:tasks.rados.rados.0.smithi107.stdout:4996: left oid 14 (ObjNum 4995 snap 0 seq_num 4995) 2020-06-12T02:38:19.742 INFO:tasks.rados.rados.0.smithi107.stdout:4999: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:19.752 INFO:tasks.rados.rados.0.smithi107.stdout:4999: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:19.752 INFO:tasks.rados.rados.0.smithi107.stdout:4999: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:19.753 INFO:tasks.rados.rados.0.smithi107.stdout:4999: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:19.753 INFO:tasks.rados.rados.0.smithi107.stdout:4999: finishing write tid 6 to smithi10729407-8 2020-06-12T02:38:19.753 INFO:tasks.rados.rados.0.smithi107.stdout:4996: done (3 left) 2020-06-12T02:38:19.754 INFO:tasks.rados.rados.0.smithi107.stdout:5000: done (2 left) 2020-06-12T02:38:19.754 INFO:tasks.rados.rados.0.smithi107.stdout:5001: delete oid 3 current snap is 0 2020-06-12T02:38:19.755 INFO:tasks.rados.rados.0.smithi107.stdout:4999: finishing write tid 7 to smithi10729407-8 2020-06-12T02:38:19.755 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 866 (ObjNum 4998 snap 0 seq_num 4998) dirty exists 2020-06-12T02:38:19.755 INFO:tasks.rados.rados.0.smithi107.stdout:4999: left oid 8 (ObjNum 4998 snap 0 seq_num 4998) 2020-06-12T02:38:19.756 INFO:tasks.rados.rados.0.smithi107.stdout:4999: done (2 left) 2020-06-12T02:38:19.756 INFO:tasks.rados.rados.0.smithi107.stdout:5001: done (1 left) 2020-06-12T02:38:19.756 INFO:tasks.rados.rados.0.smithi107.stdout:5002: delete oid 4 current snap is 0 2020-06-12T02:38:19.758 INFO:tasks.rados.rados.0.smithi107.stdout:5002: done (1 left) 2020-06-12T02:38:19.758 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:19.759 INFO:tasks.rados.rados.0.smithi107.stdout:5003: seq_num 5002 ranges {1966080=581632,2547712=475136,3022848=516096,3538944=614400} 2020-06-12T02:38:19.773 INFO:tasks.rados.rados.0.smithi107.stdout:5003: writing smithi10729407-7 from 1966080 to 2547712 tid 1 2020-06-12T02:38:19.775 INFO:tasks.rados.rados.0.smithi107.stdout:5003: writing smithi10729407-7 from 2547712 to 3022848 tid 2 2020-06-12T02:38:19.778 INFO:tasks.rados.rados.0.smithi107.stdout:5003: writing smithi10729407-7 from 3022848 to 3538944 tid 3 2020-06-12T02:38:19.780 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:19.780 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:19.781 INFO:tasks.rados.rados.0.smithi107.stdout:5003: writing smithi10729407-7 from 3538944 to 4153344 tid 4 2020-06-12T02:38:19.782 INFO:tasks.rados.rados.0.smithi107.stdout:5004: delete oid 17 current snap is 0 2020-06-12T02:38:19.782 INFO:tasks.rados.rados.0.smithi107.stdout:5003: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:19.784 INFO:tasks.rados.rados.0.smithi107.stdout:5004: done (2 left) 2020-06-12T02:38:19.784 INFO:tasks.rados.rados.0.smithi107.stdout:5005: delete oid 11 current snap is 0 2020-06-12T02:38:19.786 INFO:tasks.rados.rados.0.smithi107.stdout:5005: done (2 left) 2020-06-12T02:38:19.787 INFO:tasks.rados.rados.0.smithi107.stdout:5006: delete oid 8 current snap is 0 2020-06-12T02:38:19.789 INFO:tasks.rados.rados.0.smithi107.stdout:5003: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:19.789 INFO:tasks.rados.rados.0.smithi107.stdout:5003: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:19.791 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-06-12T02:38:19.792 INFO:tasks.rados.rados.0.smithi107.stdout:5006: done (2 left) 2020-06-12T02:38:19.792 INFO:tasks.rados.rados.0.smithi107.stdout:5007: delete oid 18 current snap is 0 2020-06-12T02:38:19.793 INFO:tasks.rados.rados.0.smithi107.stdout:5003: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:19.793 INFO:tasks.rados.rados.0.smithi107.stdout:5003: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:19.794 INFO:tasks.rados.rados.0.smithi107.stdout:5007: done (2 left) 2020-06-12T02:38:19.794 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:19.795 INFO:tasks.rados.rados.0.smithi107.stdout:5008: seq_num 5007 ranges {0=663552,663552=745472,1409024=704512,2113536=155648} 2020-06-12T02:38:19.798 INFO:tasks.rados.rados.0.smithi107.stdout:5008: writing smithi10729407-15 from 0 to 663552 tid 1 2020-06-12T02:38:19.802 INFO:tasks.rados.rados.0.smithi107.stdout:5008: writing smithi10729407-15 from 663552 to 1409024 tid 2 2020-06-12T02:38:19.806 INFO:tasks.rados.rados.0.smithi107.stdout:5008: writing smithi10729407-15 from 1409024 to 2113536 tid 3 2020-06-12T02:38:19.807 INFO:tasks.rados.rados.0.smithi107.stdout:5008: writing smithi10729407-15 from 2113536 to 2269184 tid 4 2020-06-12T02:38:19.807 INFO:tasks.rados.rados.0.smithi107.stdout:5009: delete oid 6 current snap is 0 2020-06-12T02:38:19.808 INFO:tasks.rados.rados.0.smithi107.stdout:5003: finishing write tid 7 to smithi10729407-7 2020-06-12T02:38:19.808 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 955 (ObjNum 5002 snap 0 seq_num 5002) dirty exists 2020-06-12T02:38:19.808 INFO:tasks.rados.rados.0.smithi107.stdout:5003: left oid 7 (ObjNum 5002 snap 0 seq_num 5002) 2020-06-12T02:38:19.808 INFO:tasks.rados.rados.0.smithi107.stdout:5008: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:19.811 INFO:tasks.rados.rados.0.smithi107.stdout:5003: done (3 left) 2020-06-12T02:38:19.812 INFO:tasks.rados.rados.0.smithi107.stdout:5009: done (2 left) 2020-06-12T02:38:19.812 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:38:19.812 INFO:tasks.rados.rados.0.smithi107.stdout:5010: seq_num 5009 ranges {0=696320,696320=516096,1212416=671744,1884160=417792} 2020-06-12T02:38:19.815 INFO:tasks.rados.rados.0.smithi107.stdout:5010: writing smithi10729407-5 from 0 to 696320 tid 1 2020-06-12T02:38:19.818 INFO:tasks.rados.rados.0.smithi107.stdout:5010: writing smithi10729407-5 from 696320 to 1212416 tid 2 2020-06-12T02:38:19.823 INFO:tasks.rados.rados.0.smithi107.stdout:5010: writing smithi10729407-5 from 1212416 to 1884160 tid 3 2020-06-12T02:38:19.825 INFO:tasks.rados.rados.0.smithi107.stdout:5010: writing smithi10729407-5 from 1884160 to 2301952 tid 4 2020-06-12T02:38:19.826 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:19.826 INFO:tasks.rados.rados.0.smithi107.stdout:5011: seq_num 5010 ranges {1728512=712704,2441216=450560,2891776=475136,3366912=679936} 2020-06-12T02:38:19.839 INFO:tasks.rados.rados.0.smithi107.stdout:5011: writing smithi10729407-9 from 1728512 to 2441216 tid 1 2020-06-12T02:38:19.842 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:19.839+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:19.842 INFO:tasks.rados.rados.0.smithi107.stdout:5011: writing smithi10729407-9 from 2441216 to 2891776 tid 2 2020-06-12T02:38:19.845 INFO:tasks.rados.rados.0.smithi107.stdout:5011: writing smithi10729407-9 from 2891776 to 3366912 tid 3 2020-06-12T02:38:19.849 INFO:tasks.rados.rados.0.smithi107.stdout:5011: writing smithi10729407-9 from 3366912 to 4046848 tid 4 2020-06-12T02:38:19.850 INFO:tasks.rados.rados.0.smithi107.stdout:5012: delete oid 16 current snap is 0 2020-06-12T02:38:19.850 INFO:tasks.rados.rados.0.smithi107.stdout:5008: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:19.850 INFO:tasks.rados.rados.0.smithi107.stdout:5008: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:19.850 INFO:tasks.rados.rados.0.smithi107.stdout:5008: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:19.850 INFO:tasks.rados.rados.0.smithi107.stdout:5008: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:19.851 INFO:tasks.rados.rados.0.smithi107.stdout:5008: finishing write tid 7 to smithi10729407-15 2020-06-12T02:38:19.851 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 873 (ObjNum 5007 snap 0 seq_num 5007) dirty exists 2020-06-12T02:38:19.851 INFO:tasks.rados.rados.0.smithi107.stdout:5008: left oid 15 (ObjNum 5007 snap 0 seq_num 5007) 2020-06-12T02:38:19.851 INFO:tasks.rados.rados.0.smithi107.stdout:5010: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:19.851 INFO:tasks.rados.rados.0.smithi107.stdout:5010: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:19.852 INFO:tasks.rados.rados.0.smithi107.stdout:5010: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:19.852 INFO:tasks.rados.rados.0.smithi107.stdout:5010: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:19.852 INFO:tasks.rados.rados.0.smithi107.stdout:5010: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:19.852 INFO:tasks.rados.rados.0.smithi107.stdout:5010: finishing write tid 7 to smithi10729407-5 2020-06-12T02:38:19.852 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 487 (ObjNum 5009 snap 0 seq_num 5009) dirty exists 2020-06-12T02:38:19.853 INFO:tasks.rados.rados.0.smithi107.stdout:5010: left oid 5 (ObjNum 5009 snap 0 seq_num 5009) 2020-06-12T02:38:19.853 INFO:tasks.rados.rados.0.smithi107.stdout:5011: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:19.853 INFO:tasks.rados.rados.0.smithi107.stdout:5008: done (4 left) 2020-06-12T02:38:19.854 INFO:tasks.rados.rados.0.smithi107.stdout:5010: done (3 left) 2020-06-12T02:38:19.854 INFO:tasks.rados.rados.0.smithi107.stdout:5012: done (2 left) 2020-06-12T02:38:19.854 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:19.854 INFO:tasks.rados.rados.0.smithi107.stdout:5013: seq_num 5012 ranges {0=745472,745472=614400,1359872=491520,1851392=499712} 2020-06-12T02:38:19.856 INFO:tasks.rados.rados.0.smithi107.stdout:5013: writing smithi10729407-19 from 0 to 745472 tid 1 2020-06-12T02:38:19.860 INFO:tasks.rados.rados.0.smithi107.stdout:5013: writing smithi10729407-19 from 745472 to 1359872 tid 2 2020-06-12T02:38:19.864 INFO:tasks.rados.rados.0.smithi107.stdout:5013: writing smithi10729407-19 from 1359872 to 1851392 tid 3 2020-06-12T02:38:19.866 INFO:tasks.rados.rados.0.smithi107.stdout:5013: writing smithi10729407-19 from 1851392 to 2351104 tid 4 2020-06-12T02:38:19.867 INFO:tasks.rados.rados.0.smithi107.stdout:5014: delete oid 20 current snap is 0 2020-06-12T02:38:19.867 INFO:tasks.rados.rados.0.smithi107.stdout:5011: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:19.867 INFO:tasks.rados.rados.0.smithi107.stdout:5011: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:19.868 INFO:tasks.rados.rados.0.smithi107.stdout:5011: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:19.869 INFO:tasks.rados.rados.0.smithi107.stdout:5011: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:19.869 INFO:tasks.rados.rados.0.smithi107.stdout:5011: finishing write tid 7 to smithi10729407-9 2020-06-12T02:38:19.869 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 891 (ObjNum 5010 snap 0 seq_num 5010) dirty exists 2020-06-12T02:38:19.869 INFO:tasks.rados.rados.0.smithi107.stdout:5011: left oid 9 (ObjNum 5010 snap 0 seq_num 5010) 2020-06-12T02:38:19.870 INFO:tasks.rados.rados.0.smithi107.stdout:5011: done (3 left) 2020-06-12T02:38:19.870 INFO:tasks.rados.rados.0.smithi107.stdout:5014: done (2 left) 2020-06-12T02:38:19.871 INFO:tasks.rados.rados.0.smithi107.stdout:5015: delete oid 1 current snap is 0 2020-06-12T02:38:19.871 INFO:tasks.rados.rados.0.smithi107.stdout:5013: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:19.871 INFO:tasks.rados.rados.0.smithi107.stdout:5013: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:19.872 INFO:tasks.rados.rados.0.smithi107.stdout:5015: done (2 left) 2020-06-12T02:38:19.872 INFO:tasks.rados.rados.0.smithi107.stdout:5016: delete oid 9 current snap is 0 2020-06-12T02:38:19.875 INFO:tasks.rados.rados.0.smithi107.stdout:5016: done (2 left) 2020-06-12T02:38:19.875 INFO:tasks.rados.rados.0.smithi107.stdout:5017: delete oid 4 current snap is 0 2020-06-12T02:38:19.876 INFO:tasks.rados.rados.0.smithi107.stdout:5013: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:19.879 INFO:tasks.rados.rados.0.smithi107.stdout:5013: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:19.879 INFO:tasks.rados.rados.0.smithi107.stdout:5013: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:19.879 INFO:tasks.rados.rados.0.smithi107.stdout:5017: done (2 left) 2020-06-12T02:38:19.879 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:19.880 INFO:tasks.rados.rados.0.smithi107.stdout:5018: seq_num 5017 ranges {0=32768} 2020-06-12T02:38:19.880 INFO:tasks.rados.rados.0.smithi107.stdout:5018: writing smithi10729407-12 from 0 to 32768 tid 1 2020-06-12T02:38:19.880 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:19.880 INFO:tasks.rados.rados.0.smithi107.stdout:5019: seq_num 5018 ranges {0=49152} 2020-06-12T02:38:19.880 INFO:tasks.rados.rados.0.smithi107.stdout:5019: writing smithi10729407-18 from 0 to 49152 tid 1 2020-06-12T02:38:19.881 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:19.881 INFO:tasks.rados.rados.0.smithi107.stdout:5020: seq_num 5019 ranges {4562944=65536} 2020-06-12T02:38:19.905 INFO:tasks.rados.rados.0.smithi107.stdout:5020: writing smithi10729407-14 from 4562944 to 4628480 tid 1 2020-06-12T02:38:19.905 INFO:tasks.rados.rados.0.smithi107.stdout:5021: delete oid 2 current snap is 0 2020-06-12T02:38:19.905 INFO:tasks.rados.rados.0.smithi107.stdout:5013: finishing write tid 7 to smithi10729407-19 2020-06-12T02:38:19.905 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 873 (ObjNum 5012 snap 0 seq_num 5012) dirty exists 2020-06-12T02:38:19.906 INFO:tasks.rados.rados.0.smithi107.stdout:5013: left oid 19 (ObjNum 5012 snap 0 seq_num 5012) 2020-06-12T02:38:19.906 INFO:tasks.rados.rados.0.smithi107.stdout:5019: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:19.906 INFO:tasks.rados.rados.0.smithi107.stdout:5019: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:19.906 INFO:tasks.rados.rados.0.smithi107.stdout:5018: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:19.906 INFO:tasks.rados.rados.0.smithi107.stdout:5019: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:19.907 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 894 (ObjNum 5018 snap 0 seq_num 5018) dirty exists 2020-06-12T02:38:19.907 INFO:tasks.rados.rados.0.smithi107.stdout:5019: left oid 18 (ObjNum 5018 snap 0 seq_num 5018) 2020-06-12T02:38:19.907 INFO:tasks.rados.rados.0.smithi107.stdout:5018: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:19.907 INFO:tasks.rados.rados.0.smithi107.stdout:5018: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:19.907 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 875 (ObjNum 5017 snap 0 seq_num 5017) dirty exists 2020-06-12T02:38:19.908 INFO:tasks.rados.rados.0.smithi107.stdout:5018: left oid 12 (ObjNum 5017 snap 0 seq_num 5017) 2020-06-12T02:38:19.908 INFO:tasks.rados.rados.0.smithi107.stdout:5013: done (5 left) 2020-06-12T02:38:19.908 INFO:tasks.rados.rados.0.smithi107.stdout:5018: done (4 left) 2020-06-12T02:38:19.909 INFO:tasks.rados.rados.0.smithi107.stdout:5019: done (3 left) 2020-06-12T02:38:19.909 INFO:tasks.rados.rados.0.smithi107.stdout:5021: done (2 left) 2020-06-12T02:38:19.909 INFO:tasks.rados.rados.0.smithi107.stdout:5022: delete oid 9 current snap is 0 2020-06-12T02:38:19.909 INFO:tasks.rados.rados.0.smithi107.stdout:5020: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:19.911 INFO:tasks.rados.rados.0.smithi107.stdout:5022: done (2 left) 2020-06-12T02:38:19.911 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:19.911 INFO:tasks.rados.rados.0.smithi107.stdout:5023: seq_num 5022 ranges {2424832=114688} 2020-06-12T02:38:19.925 INFO:tasks.rados.rados.0.smithi107.stdout:5023: writing smithi10729407-13 from 2424832 to 2539520 tid 1 2020-06-12T02:38:19.926 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:19.926 INFO:tasks.rados.rados.0.smithi107.stdout:5024: seq_num 5023 ranges {0=131072} 2020-06-12T02:38:19.927 INFO:tasks.rados.rados.0.smithi107.stdout:5024: writing smithi10729407-4 from 0 to 131072 tid 1 2020-06-12T02:38:19.927 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:19.928 INFO:tasks.rados.rados.0.smithi107.stdout:5020: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:19.928 INFO:tasks.rados.rados.0.smithi107.stdout:5025: seq_num 5024 ranges {0=147456} 2020-06-12T02:38:19.928 INFO:tasks.rados.rados.0.smithi107.stdout:5025: writing smithi10729407-17 from 0 to 147456 tid 1 2020-06-12T02:38:19.929 INFO:tasks.rados.rados.0.smithi107.stdout:5026: delete oid 6 current snap is 0 2020-06-12T02:38:19.929 INFO:tasks.rados.rados.0.smithi107.stdout:5020: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:19.929 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 912 (ObjNum 5019 snap 0 seq_num 5019) dirty exists 2020-06-12T02:38:19.929 INFO:tasks.rados.rados.0.smithi107.stdout:5020: left oid 14 (ObjNum 5019 snap 0 seq_num 5019) 2020-06-12T02:38:19.930 INFO:tasks.rados.rados.0.smithi107.stdout:5023: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:19.931 INFO:tasks.rados.rados.0.smithi107.stdout:5020: done (5 left) 2020-06-12T02:38:19.931 INFO:tasks.rados.rados.0.smithi107.stdout:5026: done (4 left) 2020-06-12T02:38:19.931 INFO:tasks.rados.rados.0.smithi107.stdout:5027: delete oid 5 current snap is 0 2020-06-12T02:38:19.933 INFO:tasks.rados.rados.0.smithi107.stdout:5023: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:19.934 INFO:tasks.rados.rados.0.smithi107.stdout:5027: done (4 left) 2020-06-12T02:38:19.934 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:19.934 INFO:tasks.rados.rados.0.smithi107.stdout:5028: seq_num 5027 ranges {4628480=196608} 2020-06-12T02:38:19.934 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:19.935 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:19.942 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:19.940+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:19.943 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-06-12T02:38:19.960 INFO:tasks.rados.rados.0.smithi107.stdout:5028: writing smithi10729407-14 from 4628480 to 4825088 tid 1 2020-06-12T02:38:19.960 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:19.960 INFO:tasks.rados.rados.0.smithi107.stdout:5029: seq_num 5028 ranges {0=212992} 2020-06-12T02:38:19.961 INFO:tasks.rados.rados.0.smithi107.stdout:5029: writing smithi10729407-11 from 0 to 212992 tid 1 2020-06-12T02:38:19.961 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:19.962 INFO:tasks.rados.rados.0.smithi107.stdout:5030: seq_num 5029 ranges {4153344=229376} 2020-06-12T02:38:19.986 INFO:tasks.rados.rados.0.smithi107.stdout:5030: writing smithi10729407-7 from 4153344 to 4382720 tid 1 2020-06-12T02:38:19.987 INFO:tasks.rados.rados.0.smithi107.stdout:5031: delete oid 19 current snap is 0 2020-06-12T02:38:19.987 INFO:tasks.rados.rados.0.smithi107.stdout:5024: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:19.987 INFO:tasks.rados.rados.0.smithi107.stdout:5023: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:19.987 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 957 (ObjNum 5022 snap 0 seq_num 5022) dirty exists 2020-06-12T02:38:19.988 INFO:tasks.rados.rados.0.smithi107.stdout:5023: left oid 13 (ObjNum 5022 snap 0 seq_num 5022) 2020-06-12T02:38:19.988 INFO:tasks.rados.rados.0.smithi107.stdout:5025: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:19.988 INFO:tasks.rados.rados.0.smithi107.stdout:5025: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:19.988 INFO:tasks.rados.rados.0.smithi107.stdout:5024: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:19.988 INFO:tasks.rados.rados.0.smithi107.stdout:5025: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:19.989 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 933 (ObjNum 5024 snap 0 seq_num 5024) dirty exists 2020-06-12T02:38:19.989 INFO:tasks.rados.rados.0.smithi107.stdout:5025: left oid 17 (ObjNum 5024 snap 0 seq_num 5024) 2020-06-12T02:38:19.989 INFO:tasks.rados.rados.0.smithi107.stdout:5024: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:19.989 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 934 (ObjNum 5023 snap 0 seq_num 5023) dirty exists 2020-06-12T02:38:19.989 INFO:tasks.rados.rados.0.smithi107.stdout:5024: left oid 4 (ObjNum 5023 snap 0 seq_num 5023) 2020-06-12T02:38:19.990 INFO:tasks.rados.rados.0.smithi107.stdout:5028: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:19.990 INFO:tasks.rados.rados.0.smithi107.stdout:5029: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:19.990 INFO:tasks.rados.rados.0.smithi107.stdout:5029: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:19.990 INFO:tasks.rados.rados.0.smithi107.stdout:5028: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:19.990 INFO:tasks.rados.rados.0.smithi107.stdout:5029: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:19.991 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 915 (ObjNum 5028 snap 0 seq_num 5028) dirty exists 2020-06-12T02:38:19.991 INFO:tasks.rados.rados.0.smithi107.stdout:5029: left oid 11 (ObjNum 5028 snap 0 seq_num 5028) 2020-06-12T02:38:19.991 INFO:tasks.rados.rados.0.smithi107.stdout:5028: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:19.991 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 916 (ObjNum 5027 snap 0 seq_num 5027) dirty exists 2020-06-12T02:38:19.991 INFO:tasks.rados.rados.0.smithi107.stdout:5028: left oid 14 (ObjNum 5027 snap 0 seq_num 5027) 2020-06-12T02:38:19.993 INFO:tasks.rados.rados.0.smithi107.stdout:5023: done (7 left) 2020-06-12T02:38:19.993 INFO:tasks.rados.rados.0.smithi107.stdout:5024: done (6 left) 2020-06-12T02:38:19.993 INFO:tasks.rados.rados.0.smithi107.stdout:5025: done (5 left) 2020-06-12T02:38:19.993 INFO:tasks.rados.rados.0.smithi107.stdout:5028: done (4 left) 2020-06-12T02:38:19.994 INFO:tasks.rados.rados.0.smithi107.stdout:5029: done (3 left) 2020-06-12T02:38:19.994 INFO:tasks.rados.rados.0.smithi107.stdout:5031: done (2 left) 2020-06-12T02:38:19.995 INFO:tasks.rados.rados.0.smithi107.stdout:5032: delete oid 20 current snap is 0 2020-06-12T02:38:19.995 INFO:tasks.rados.rados.0.smithi107.stdout:5030: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:19.995 INFO:tasks.rados.rados.0.smithi107.stdout:5030: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:19.995 INFO:tasks.rados.rados.0.smithi107.stdout:5032: done (2 left) 2020-06-12T02:38:19.995 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:19.996 INFO:tasks.rados.rados.0.smithi107.stdout:5033: seq_num 5032 ranges {0=278528} 2020-06-12T02:38:19.996 INFO:tasks.rados.rados.0.smithi107.stdout:5033: writing smithi10729407-5 from 0 to 278528 tid 1 2020-06-12T02:38:19.996 INFO:tasks.rados.rados.0.smithi107.stdout:5034: delete oid 1 current snap is 0 2020-06-12T02:38:19.996 INFO:tasks.rados.rados.0.smithi107.stdout:5030: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:19.996 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 959 (ObjNum 5029 snap 0 seq_num 5029) dirty exists 2020-06-12T02:38:19.996 INFO:tasks.rados.rados.0.smithi107.stdout:5030: left oid 7 (ObjNum 5029 snap 0 seq_num 5029) 2020-06-12T02:38:19.997 INFO:tasks.rados.rados.0.smithi107.stdout:5030: done (3 left) 2020-06-12T02:38:19.997 INFO:tasks.rados.rados.0.smithi107.stdout:5034: done (2 left) 2020-06-12T02:38:19.998 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:19.998 INFO:tasks.rados.rados.0.smithi107.stdout:5035: seq_num 5034 ranges {32768=311296} 2020-06-12T02:38:19.999 INFO:tasks.rados.rados.0.smithi107.stdout:5035: writing smithi10729407-12 from 32768 to 344064 tid 1 2020-06-12T02:38:19.999 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:19.999 INFO:tasks.rados.rados.0.smithi107.stdout:5036: seq_num 5035 ranges {0=327680} 2020-06-12T02:38:20.000 INFO:tasks.rados.rados.0.smithi107.stdout:5036: writing smithi10729407-9 from 0 to 327680 tid 1 2020-06-12T02:38:20.000 INFO:tasks.rados.rados.0.smithi107.stdout:5037: delete oid 8 current snap is 0 2020-06-12T02:38:20.001 INFO:tasks.rados.rados.0.smithi107.stdout:5033: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:20.001 INFO:tasks.rados.rados.0.smithi107.stdout:5033: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:20.001 INFO:tasks.rados.rados.0.smithi107.stdout:5033: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:20.001 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 490 (ObjNum 5032 snap 0 seq_num 5032) dirty exists 2020-06-12T02:38:20.002 INFO:tasks.rados.rados.0.smithi107.stdout:5033: left oid 5 (ObjNum 5032 snap 0 seq_num 5032) 2020-06-12T02:38:20.003 INFO:tasks.rados.rados.0.smithi107.stdout:5033: done (4 left) 2020-06-12T02:38:20.003 INFO:tasks.rados.rados.0.smithi107.stdout:5037: done (3 left) 2020-06-12T02:38:20.003 INFO:tasks.rados.rados.0.smithi107.stdout:5038: delete oid 3 current snap is 0 2020-06-12T02:38:20.004 INFO:tasks.rados.rados.0.smithi107.stdout:5035: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:20.005 INFO:tasks.rados.rados.0.smithi107.stdout:5035: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:20.006 INFO:tasks.rados.rados.0.smithi107.stdout:5036: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:20.007 INFO:tasks.rados.rados.0.smithi107.stdout:5035: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:20.007 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 878 (ObjNum 5034 snap 0 seq_num 5034) dirty exists 2020-06-12T02:38:20.007 INFO:tasks.rados.rados.0.smithi107.stdout:5035: left oid 12 (ObjNum 5034 snap 0 seq_num 5034) 2020-06-12T02:38:20.007 INFO:tasks.rados.rados.0.smithi107.stdout:5035: done (3 left) 2020-06-12T02:38:20.007 INFO:tasks.rados.rados.0.smithi107.stdout:5038: done (2 left) 2020-06-12T02:38:20.008 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:38:20.008 INFO:tasks.rados.rados.0.smithi107.stdout:5039: seq_num 5038 ranges {0=385024} 2020-06-12T02:38:20.009 INFO:tasks.rados.rados.0.smithi107.stdout:5039: writing smithi10729407-3 from 0 to 385024 tid 1 2020-06-12T02:38:20.009 INFO:tasks.rados.rados.0.smithi107.stdout:5040: delete oid 18 current snap is 0 2020-06-12T02:38:20.009 INFO:tasks.rados.rados.0.smithi107.stdout:5036: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:20.011 INFO:tasks.rados.rados.0.smithi107.stdout:5036: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:20.011 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 897 (ObjNum 5035 snap 0 seq_num 5035) dirty exists 2020-06-12T02:38:20.012 INFO:tasks.rados.rados.0.smithi107.stdout:5036: left oid 9 (ObjNum 5035 snap 0 seq_num 5035) 2020-06-12T02:38:20.014 INFO:tasks.rados.rados.0.smithi107.stdout:5039: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:20.014 INFO:tasks.rados.rados.0.smithi107.stdout:5036: done (3 left) 2020-06-12T02:38:20.015 INFO:tasks.rados.rados.0.smithi107.stdout:5040: done (2 left) 2020-06-12T02:38:20.015 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:20.015 INFO:tasks.rados.rados.0.smithi107.stdout:5041: seq_num 5040 ranges {344064=417792} 2020-06-12T02:38:20.019 INFO:tasks.rados.rados.0.smithi107.stdout:5041: writing smithi10729407-12 from 344064 to 761856 tid 1 2020-06-12T02:38:20.019 INFO:tasks.rados.rados.0.smithi107.stdout:5042: delete oid 6 current snap is 0 2020-06-12T02:38:20.019 INFO:tasks.rados.rados.0.smithi107.stdout:5039: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:20.019 INFO:tasks.rados.rados.0.smithi107.stdout:5039: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:20.019 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 903 (ObjNum 5038 snap 0 seq_num 5038) dirty exists 2020-06-12T02:38:20.020 INFO:tasks.rados.rados.0.smithi107.stdout:5039: left oid 3 (ObjNum 5038 snap 0 seq_num 5038) 2020-06-12T02:38:20.022 INFO:tasks.rados.rados.0.smithi107.stdout:5039: done (3 left) 2020-06-12T02:38:20.022 INFO:tasks.rados.rados.0.smithi107.stdout:5042: done (2 left) 2020-06-12T02:38:20.023 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:20.023 INFO:tasks.rados.rados.0.smithi107.stdout:5043: seq_num 5042 ranges {212992=450560} 2020-06-12T02:38:20.026 INFO:tasks.rados.rados.0.smithi107.stdout:5043: writing smithi10729407-11 from 212992 to 663552 tid 1 2020-06-12T02:38:20.026 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:38:20.027 INFO:tasks.rados.rados.0.smithi107.stdout:5044: seq_num 5043 ranges {4382720=466944} 2020-06-12T02:38:20.043 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:20.041+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:20.052 INFO:tasks.rados.rados.0.smithi107.stdout:5044: writing smithi10729407-7 from 4382720 to 4849664 tid 1 2020-06-12T02:38:20.053 INFO:tasks.rados.rados.0.smithi107.stdout:5045: delete oid 13 current snap is 0 2020-06-12T02:38:20.053 INFO:tasks.rados.rados.0.smithi107.stdout:5041: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:20.053 INFO:tasks.rados.rados.0.smithi107.stdout:5041: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:20.053 INFO:tasks.rados.rados.0.smithi107.stdout:5041: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:20.053 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 880 (ObjNum 5040 snap 0 seq_num 5040) dirty exists 2020-06-12T02:38:20.054 INFO:tasks.rados.rados.0.smithi107.stdout:5041: left oid 12 (ObjNum 5040 snap 0 seq_num 5040) 2020-06-12T02:38:20.054 INFO:tasks.rados.rados.0.smithi107.stdout:5043: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:20.054 INFO:tasks.rados.rados.0.smithi107.stdout:5043: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:20.054 INFO:tasks.rados.rados.0.smithi107.stdout:5043: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:20.054 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 918 (ObjNum 5042 snap 0 seq_num 5042) dirty exists 2020-06-12T02:38:20.055 INFO:tasks.rados.rados.0.smithi107.stdout:5043: left oid 11 (ObjNum 5042 snap 0 seq_num 5042) 2020-06-12T02:38:20.057 INFO:tasks.rados.rados.0.smithi107.stdout:5044: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:20.058 INFO:tasks.rados.rados.0.smithi107.stdout:5041: done (4 left) 2020-06-12T02:38:20.059 INFO:tasks.rados.rados.0.smithi107.stdout:5043: done (3 left) 2020-06-12T02:38:20.059 INFO:tasks.rados.rados.0.smithi107.stdout:5045: done (2 left) 2020-06-12T02:38:20.059 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:20.059 INFO:tasks.rados.rados.0.smithi107.stdout:5046: seq_num 5045 ranges {327680=499712} 2020-06-12T02:38:20.063 INFO:tasks.rados.rados.0.smithi107.stdout:5046: writing smithi10729407-9 from 327680 to 827392 tid 1 2020-06-12T02:38:20.063 INFO:tasks.rados.rados.0.smithi107.stdout:5047: delete oid 16 current snap is 0 2020-06-12T02:38:20.063 INFO:tasks.rados.rados.0.smithi107.stdout:5044: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:20.064 INFO:tasks.rados.rados.0.smithi107.stdout:5044: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:20.064 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 962 (ObjNum 5043 snap 0 seq_num 5043) dirty exists 2020-06-12T02:38:20.064 INFO:tasks.rados.rados.0.smithi107.stdout:5044: left oid 7 (ObjNum 5043 snap 0 seq_num 5043) 2020-06-12T02:38:20.068 INFO:tasks.rados.rados.0.smithi107.stdout:5046: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:20.069 INFO:tasks.rados.rados.0.smithi107.stdout:5046: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:20.069 INFO:tasks.rados.rados.0.smithi107.stdout:5044: done (3 left) 2020-06-12T02:38:20.069 INFO:tasks.rados.rados.0.smithi107.stdout:5047: done (2 left) 2020-06-12T02:38:20.069 INFO:tasks.rados.rados.0.smithi107.stdout:5048: delete oid 8 current snap is 0 2020-06-12T02:38:20.070 INFO:tasks.rados.rados.0.smithi107.stdout:5046: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:20.070 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 900 (ObjNum 5045 snap 0 seq_num 5045) dirty exists 2020-06-12T02:38:20.070 INFO:tasks.rados.rados.0.smithi107.stdout:5046: left oid 9 (ObjNum 5045 snap 0 seq_num 5045) 2020-06-12T02:38:20.071 INFO:tasks.rados.rados.0.smithi107.stdout:5046: done (2 left) 2020-06-12T02:38:20.071 INFO:tasks.rados.rados.0.smithi107.stdout:5048: done (1 left) 2020-06-12T02:38:20.071 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:20.071 INFO:tasks.rados.rados.0.smithi107.stdout:5049: seq_num 5048 ranges {385024=548864} 2020-06-12T02:38:20.076 INFO:tasks.rados.rados.0.smithi107.stdout:5049: writing smithi10729407-3 from 385024 to 933888 tid 1 2020-06-12T02:38:20.076 INFO:tasks.rados.rados.0.smithi107.stdout:5050: delete oid 14 current snap is 0 2020-06-12T02:38:20.080 INFO:tasks.rados.rados.0.smithi107.stdout:5050: done (2 left) 2020-06-12T02:38:20.080 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:20.081 INFO:tasks.rados.rados.0.smithi107.stdout:5051: seq_num 5050 ranges {147456=581632} 2020-06-12T02:38:20.085 INFO:tasks.rados.rados.0.smithi107.stdout:5051: writing smithi10729407-17 from 147456 to 729088 tid 1 2020-06-12T02:38:20.085 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:20.085 INFO:tasks.rados.rados.0.smithi107.stdout:5052: seq_num 5051 ranges {0=598016} 2020-06-12T02:38:20.089 INFO:tasks.rados.rados.0.smithi107.stdout:5052: writing smithi10729407-8 from 0 to 598016 tid 1 2020-06-12T02:38:20.089 INFO:tasks.rados.rados.0.smithi107.stdout:5053: delete oid 12 current snap is 0 2020-06-12T02:38:20.089 INFO:tasks.rados.rados.0.smithi107.stdout:5049: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:20.090 INFO:tasks.rados.rados.0.smithi107.stdout:5049: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:20.090 INFO:tasks.rados.rados.0.smithi107.stdout:5049: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:20.090 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 906 (ObjNum 5048 snap 0 seq_num 5048) dirty exists 2020-06-12T02:38:20.090 INFO:tasks.rados.rados.0.smithi107.stdout:5049: left oid 3 (ObjNum 5048 snap 0 seq_num 5048) 2020-06-12T02:38:20.096 INFO:tasks.rados.rados.0.smithi107.stdout:5049: done (4 left) 2020-06-12T02:38:20.096 INFO:tasks.rados.rados.0.smithi107.stdout:5053: done (3 left) 2020-06-12T02:38:20.096 INFO:tasks.rados.rados.0.smithi107.stdout:5054: delete oid 18 current snap is 0 2020-06-12T02:38:20.097 INFO:tasks.rados.rados.0.smithi107.stdout:5054: done (3 left) 2020-06-12T02:38:20.097 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:20.097 INFO:tasks.rados.rados.0.smithi107.stdout:5055: seq_num 5054 ranges {0=647168} 2020-06-12T02:38:20.098 INFO:tasks.rados.rados.0.smithi107.stdout:5055: writing smithi10729407-6 from 0 to 647168 tid 1 2020-06-12T02:38:20.098 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:20.098 INFO:tasks.rados.rados.0.smithi107.stdout:5056: seq_num 5055 ranges {827392=671744} 2020-06-12T02:38:20.104 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:20.104 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:20.107 INFO:tasks.rados.rados.0.smithi107.stdout:5056: writing smithi10729407-9 from 827392 to 1499136 tid 1 2020-06-12T02:38:20.107 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:20.107 INFO:tasks.rados.rados.0.smithi107.stdout:5051: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:20.107 INFO:tasks.rados.rados.0.smithi107.stdout:5057: seq_num 5056 ranges {2269184=688128} 2020-06-12T02:38:20.114 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-06-12T02:38:20.123 INFO:tasks.rados.rados.0.smithi107.stdout:5057: writing smithi10729407-15 from 2269184 to 2957312 tid 1 2020-06-12T02:38:20.124 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:20.124 INFO:tasks.rados.rados.0.smithi107.stdout:5058: seq_num 5057 ranges {131072=704512} 2020-06-12T02:38:20.129 INFO:tasks.rados.rados.0.smithi107.stdout:5058: writing smithi10729407-4 from 131072 to 835584 tid 1 2020-06-12T02:38:20.129 INFO:tasks.rados.rados.0.smithi107.stdout:5059: delete oid 16 current snap is 0 2020-06-12T02:38:20.129 INFO:tasks.rados.rados.0.smithi107.stdout:5052: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:20.129 INFO:tasks.rados.rados.0.smithi107.stdout:5051: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:20.130 INFO:tasks.rados.rados.0.smithi107.stdout:5051: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:20.130 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 936 (ObjNum 5050 snap 0 seq_num 5050) dirty exists 2020-06-12T02:38:20.130 INFO:tasks.rados.rados.0.smithi107.stdout:5051: left oid 17 (ObjNum 5050 snap 0 seq_num 5050) 2020-06-12T02:38:20.130 INFO:tasks.rados.rados.0.smithi107.stdout:5052: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:20.130 INFO:tasks.rados.rados.0.smithi107.stdout:5055: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:20.131 INFO:tasks.rados.rados.0.smithi107.stdout:5052: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:20.131 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 877 (ObjNum 5051 snap 0 seq_num 5051) dirty exists 2020-06-12T02:38:20.131 INFO:tasks.rados.rados.0.smithi107.stdout:5052: left oid 8 (ObjNum 5051 snap 0 seq_num 5051) 2020-06-12T02:38:20.131 INFO:tasks.rados.rados.0.smithi107.stdout:5055: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:20.131 INFO:tasks.rados.rados.0.smithi107.stdout:5055: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:20.132 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 938 (ObjNum 5054 snap 0 seq_num 5054) dirty exists 2020-06-12T02:38:20.132 INFO:tasks.rados.rados.0.smithi107.stdout:5055: left oid 6 (ObjNum 5054 snap 0 seq_num 5054) 2020-06-12T02:38:20.132 INFO:tasks.rados.rados.0.smithi107.stdout:5056: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:20.132 INFO:tasks.rados.rados.0.smithi107.stdout:5056: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:20.132 INFO:tasks.rados.rados.0.smithi107.stdout:5056: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:20.132 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 903 (ObjNum 5055 snap 0 seq_num 5055) dirty exists 2020-06-12T02:38:20.133 INFO:tasks.rados.rados.0.smithi107.stdout:5056: left oid 9 (ObjNum 5055 snap 0 seq_num 5055) 2020-06-12T02:38:20.133 INFO:tasks.rados.rados.0.smithi107.stdout:5051: done (7 left) 2020-06-12T02:38:20.134 INFO:tasks.rados.rados.0.smithi107.stdout:5052: done (6 left) 2020-06-12T02:38:20.134 INFO:tasks.rados.rados.0.smithi107.stdout:5055: done (5 left) 2020-06-12T02:38:20.134 INFO:tasks.rados.rados.0.smithi107.stdout:5056: done (4 left) 2020-06-12T02:38:20.134 INFO:tasks.rados.rados.0.smithi107.stdout:5059: done (3 left) 2020-06-12T02:38:20.134 INFO:tasks.rados.rados.0.smithi107.stdout:5060: delete oid 5 current snap is 0 2020-06-12T02:38:20.134 INFO:tasks.rados.rados.0.smithi107.stdout:5057: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:20.135 INFO:tasks.rados.rados.0.smithi107.stdout:5057: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:20.135 INFO:tasks.rados.rados.0.smithi107.stdout:5060: done (3 left) 2020-06-12T02:38:20.135 INFO:tasks.rados.rados.0.smithi107.stdout:5061: delete oid 17 current snap is 0 2020-06-12T02:38:20.136 INFO:tasks.rados.rados.0.smithi107.stdout:5057: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:20.136 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 879 (ObjNum 5056 snap 0 seq_num 5056) dirty exists 2020-06-12T02:38:20.136 INFO:tasks.rados.rados.0.smithi107.stdout:5057: left oid 15 (ObjNum 5056 snap 0 seq_num 5056) 2020-06-12T02:38:20.136 INFO:tasks.rados.rados.0.smithi107.stdout:5058: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:20.137 INFO:tasks.rados.rados.0.smithi107.stdout:5057: done (3 left) 2020-06-12T02:38:20.138 INFO:tasks.rados.rados.0.smithi107.stdout:5061: done (2 left) 2020-06-12T02:38:20.138 INFO:tasks.rados.rados.0.smithi107.stdout:5062: delete oid 15 current snap is 0 2020-06-12T02:38:20.139 INFO:tasks.rados.rados.0.smithi107.stdout:5058: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:20.141 INFO:tasks.rados.rados.0.smithi107.stdout:5058: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:20.141 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 939 (ObjNum 5057 snap 0 seq_num 5057) dirty exists 2020-06-12T02:38:20.141 INFO:tasks.rados.rados.0.smithi107.stdout:5058: left oid 4 (ObjNum 5057 snap 0 seq_num 5057) 2020-06-12T02:38:20.142 INFO:tasks.rados.rados.0.smithi107.stdout:5058: done (2 left) 2020-06-12T02:38:20.142 INFO:tasks.rados.rados.0.smithi107.stdout:5062: done (1 left) 2020-06-12T02:38:20.143 INFO:tasks.rados.rados.0.smithi107.stdout:5063: delete oid 3 current snap is 0 2020-06-12T02:38:20.144 INFO:tasks.rados.rados.0.smithi107.stdout:5063: done (1 left) 2020-06-12T02:38:20.144 INFO:tasks.rados.rados.0.smithi107.stdout:5064: delete oid 5 current snap is 0 2020-06-12T02:38:20.146 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:20.143+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:20.147 INFO:tasks.rados.rados.0.smithi107.stdout:5064: done (1 left) 2020-06-12T02:38:20.148 INFO:tasks.rados.rados.0.smithi107.stdout:5065: delete oid 5 current snap is 0 2020-06-12T02:38:20.150 INFO:tasks.rados.rados.0.smithi107.stdout:5065: done (1 left) 2020-06-12T02:38:20.150 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:20.151 INFO:tasks.rados.rados.0.smithi107.stdout:5066: seq_num 5065 ranges {0=434176,434176=401408} 2020-06-12T02:38:20.153 INFO:tasks.rados.rados.0.smithi107.stdout:5066: writing smithi10729407-18 from 0 to 434176 tid 1 2020-06-12T02:38:20.155 INFO:tasks.rados.rados.0.smithi107.stdout:5066: writing smithi10729407-18 from 434176 to 835584 tid 2 2020-06-12T02:38:20.155 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:20.155 INFO:tasks.rados.rados.0.smithi107.stdout:5067: seq_num 5066 ranges {598016=450560,1048576=401408} 2020-06-12T02:38:20.161 INFO:tasks.rados.rados.0.smithi107.stdout:5067: writing smithi10729407-8 from 598016 to 1048576 tid 1 2020-06-12T02:38:20.164 INFO:tasks.rados.rados.0.smithi107.stdout:5067: writing smithi10729407-8 from 1048576 to 1449984 tid 2 2020-06-12T02:38:20.164 INFO:tasks.rados.rados.0.smithi107.stdout:5068: delete oid 11 current snap is 0 2020-06-12T02:38:20.164 INFO:tasks.rados.rados.0.smithi107.stdout:5066: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:20.165 INFO:tasks.rados.rados.0.smithi107.stdout:5066: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:20.165 INFO:tasks.rados.rados.0.smithi107.stdout:5066: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:20.165 INFO:tasks.rados.rados.0.smithi107.stdout:5066: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:20.166 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 906 (ObjNum 5065 snap 0 seq_num 5065) dirty exists 2020-06-12T02:38:20.166 INFO:tasks.rados.rados.0.smithi107.stdout:5066: left oid 18 (ObjNum 5065 snap 0 seq_num 5065) 2020-06-12T02:38:20.166 INFO:tasks.rados.rados.0.smithi107.stdout:5066: done (3 left) 2020-06-12T02:38:20.167 INFO:tasks.rados.rados.0.smithi107.stdout:5068: done (2 left) 2020-06-12T02:38:20.167 INFO:tasks.rados.rados.0.smithi107.stdout:5069: delete oid 9 current snap is 0 2020-06-12T02:38:20.170 INFO:tasks.rados.rados.0.smithi107.stdout:5069: done (2 left) 2020-06-12T02:38:20.170 INFO:tasks.rados.rados.0.smithi107.stdout:5070: delete oid 19 current snap is 0 2020-06-12T02:38:20.170 INFO:tasks.rados.rados.0.smithi107.stdout:5067: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:20.172 INFO:tasks.rados.rados.0.smithi107.stdout:5067: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:20.175 INFO:tasks.rados.rados.0.smithi107.stdout:5070: done (2 left) 2020-06-12T02:38:20.175 INFO:tasks.rados.rados.0.smithi107.stdout:5071: delete oid 12 current snap is 0 2020-06-12T02:38:20.175 INFO:tasks.rados.rados.0.smithi107.stdout:5067: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:20.177 INFO:tasks.rados.rados.0.smithi107.stdout:5067: finishing write tid 5 to smithi10729407-8 2020-06-12T02:38:20.177 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 883 (ObjNum 5066 snap 0 seq_num 5066) dirty exists 2020-06-12T02:38:20.177 INFO:tasks.rados.rados.0.smithi107.stdout:5067: left oid 8 (ObjNum 5066 snap 0 seq_num 5066) 2020-06-12T02:38:20.179 INFO:tasks.rados.rados.0.smithi107.stdout:5067: done (2 left) 2020-06-12T02:38:20.179 INFO:tasks.rados.rados.0.smithi107.stdout:5071: done (1 left) 2020-06-12T02:38:20.179 INFO:tasks.rados.rados.0.smithi107.stdout:5072: delete oid 13 current snap is 0 2020-06-12T02:38:20.182 INFO:tasks.rados.rados.0.smithi107.stdout:5072: done (1 left) 2020-06-12T02:38:20.182 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:20.182 INFO:tasks.rados.rados.0.smithi107.stdout:5073: seq_num 5072 ranges {0=548864,548864=401408} 2020-06-12T02:38:20.184 INFO:tasks.rados.rados.0.smithi107.stdout:5073: writing smithi10729407-13 from 0 to 548864 tid 1 2020-06-12T02:38:20.187 INFO:tasks.rados.rados.0.smithi107.stdout:5073: writing smithi10729407-13 from 548864 to 950272 tid 2 2020-06-12T02:38:20.187 INFO:tasks.rados.rados.0.smithi107.stdout:5074: delete oid 11 current snap is 0 2020-06-12T02:38:20.189 INFO:tasks.rados.rados.0.smithi107.stdout:5074: done (2 left) 2020-06-12T02:38:20.190 INFO:tasks.rados.rados.0.smithi107.stdout:5075: delete oid 6 current snap is 0 2020-06-12T02:38:20.192 INFO:tasks.rados.rados.0.smithi107.stdout:5073: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:20.192 INFO:tasks.rados.rados.0.smithi107.stdout:5075: done (2 left) 2020-06-12T02:38:20.193 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:20.193 INFO:tasks.rados.rados.0.smithi107.stdout:5076: seq_num 5075 ranges {0=606208,606208=401408} 2020-06-12T02:38:20.196 INFO:tasks.rados.rados.0.smithi107.stdout:5076: writing smithi10729407-9 from 0 to 606208 tid 1 2020-06-12T02:38:20.198 INFO:tasks.rados.rados.0.smithi107.stdout:5076: writing smithi10729407-9 from 606208 to 1007616 tid 2 2020-06-12T02:38:20.198 INFO:tasks.rados.rados.0.smithi107.stdout:5077: delete oid 4 current snap is 0 2020-06-12T02:38:20.199 INFO:tasks.rados.rados.0.smithi107.stdout:5073: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:20.202 INFO:tasks.rados.rados.0.smithi107.stdout:5073: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:20.203 INFO:tasks.rados.rados.0.smithi107.stdout:5077: done (3 left) 2020-06-12T02:38:20.203 INFO:tasks.rados.rados.0.smithi107.stdout:5078: delete oid 4 current snap is 0 2020-06-12T02:38:20.204 INFO:tasks.rados.rados.0.smithi107.stdout:5073: finishing write tid 5 to smithi10729407-13 2020-06-12T02:38:20.204 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 966 (ObjNum 5072 snap 0 seq_num 5072) dirty exists 2020-06-12T02:38:20.204 INFO:tasks.rados.rados.0.smithi107.stdout:5073: left oid 13 (ObjNum 5072 snap 0 seq_num 5072) 2020-06-12T02:38:20.206 INFO:tasks.rados.rados.0.smithi107.stdout:5076: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:20.207 INFO:tasks.rados.rados.0.smithi107.stdout:5073: done (3 left) 2020-06-12T02:38:20.207 INFO:tasks.rados.rados.0.smithi107.stdout:5078: done (2 left) 2020-06-12T02:38:20.207 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:20.207 INFO:tasks.rados.rados.0.smithi107.stdout:5079: seq_num 5078 ranges {0=655360,655360=401408} 2020-06-12T02:38:20.211 INFO:tasks.rados.rados.0.smithi107.stdout:5079: writing smithi10729407-19 from 0 to 655360 tid 1 2020-06-12T02:38:20.213 INFO:tasks.rados.rados.0.smithi107.stdout:5079: writing smithi10729407-19 from 655360 to 1056768 tid 2 2020-06-12T02:38:20.213 INFO:tasks.rados.rados.0.smithi107.stdout:5080: delete oid 2 current snap is 0 2020-06-12T02:38:20.214 INFO:tasks.rados.rados.0.smithi107.stdout:5076: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:20.214 INFO:tasks.rados.rados.0.smithi107.stdout:5076: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:20.214 INFO:tasks.rados.rados.0.smithi107.stdout:5076: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:20.214 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 910 (ObjNum 5075 snap 0 seq_num 5075) dirty exists 2020-06-12T02:38:20.214 INFO:tasks.rados.rados.0.smithi107.stdout:5076: left oid 9 (ObjNum 5075 snap 0 seq_num 5075) 2020-06-12T02:38:20.215 INFO:tasks.rados.rados.0.smithi107.stdout:5076: done (3 left) 2020-06-12T02:38:20.215 INFO:tasks.rados.rados.0.smithi107.stdout:5080: done (2 left) 2020-06-12T02:38:20.215 INFO:tasks.rados.rados.0.smithi107.stdout:5081: delete oid 20 current snap is 0 2020-06-12T02:38:20.216 INFO:tasks.rados.rados.0.smithi107.stdout:5081: done (2 left) 2020-06-12T02:38:20.217 INFO:tasks.rados.rados.0.smithi107.stdout:5082: delete oid 3 current snap is 0 2020-06-12T02:38:20.217 INFO:tasks.rados.rados.0.smithi107.stdout:5079: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:20.219 INFO:tasks.rados.rados.0.smithi107.stdout:5082: done (2 left) 2020-06-12T02:38:20.219 INFO:tasks.rados.rados.0.smithi107.stdout:5083: delete oid 2 current snap is 0 2020-06-12T02:38:20.222 INFO:tasks.rados.rados.0.smithi107.stdout:5079: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:20.222 INFO:tasks.rados.rados.0.smithi107.stdout:5083: done (2 left) 2020-06-12T02:38:20.222 INFO:tasks.rados.rados.0.smithi107.stdout:5084: delete oid 7 current snap is 0 2020-06-12T02:38:20.223 INFO:tasks.rados.rados.0.smithi107.stdout:5079: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:20.224 INFO:tasks.rados.rados.0.smithi107.stdout:5084: done (2 left) 2020-06-12T02:38:20.225 INFO:tasks.rados.rados.0.smithi107.stdout:5085: delete oid 2 current snap is 0 2020-06-12T02:38:20.225 INFO:tasks.rados.rados.0.smithi107.stdout:5079: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:20.225 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 886 (ObjNum 5078 snap 0 seq_num 5078) dirty exists 2020-06-12T02:38:20.225 INFO:tasks.rados.rados.0.smithi107.stdout:5079: left oid 19 (ObjNum 5078 snap 0 seq_num 5078) 2020-06-12T02:38:20.227 INFO:tasks.rados.rados.0.smithi107.stdout:5079: done (2 left) 2020-06-12T02:38:20.227 INFO:tasks.rados.rados.0.smithi107.stdout:5085: done (1 left) 2020-06-12T02:38:20.227 INFO:tasks.rados.rados.0.smithi107.stdout:5086: delete oid 3 current snap is 0 2020-06-12T02:38:20.229 INFO:tasks.rados.rados.0.smithi107.stdout:5086: done (1 left) 2020-06-12T02:38:20.230 INFO:tasks.rados.rados.0.smithi107.stdout:5087: delete oid 17 current snap is 0 2020-06-12T02:38:20.232 INFO:tasks.rados.rados.0.smithi107.stdout:5087: done (1 left) 2020-06-12T02:38:20.232 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:20.232 INFO:tasks.rados.rados.0.smithi107.stdout:5088: seq_num 5087 ranges {0=802816,802816=401408} 2020-06-12T02:38:20.236 INFO:tasks.rados.rados.0.smithi107.stdout:5088: writing smithi10729407-7 from 0 to 802816 tid 1 2020-06-12T02:38:20.239 INFO:tasks.rados.rados.0.smithi107.stdout:5088: writing smithi10729407-7 from 802816 to 1204224 tid 2 2020-06-12T02:38:20.239 INFO:tasks.rados.rados.0.smithi107.stdout:5089: delete oid 2 current snap is 0 2020-06-12T02:38:20.241 INFO:tasks.rados.rados.0.smithi107.stdout:5089: done (2 left) 2020-06-12T02:38:20.241 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:20.241 INFO:tasks.rados.rados.0.smithi107.stdout:5090: seq_num 5089 ranges {0=434176,434176=606208,1040384=196608} 2020-06-12T02:38:20.243 INFO:tasks.rados.rados.0.smithi107.stdout:5090: writing smithi10729407-17 from 0 to 434176 tid 1 2020-06-12T02:38:20.246 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:20.243+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:20.247 INFO:tasks.rados.rados.0.smithi107.stdout:5090: writing smithi10729407-17 from 434176 to 1040384 tid 2 2020-06-12T02:38:20.249 INFO:tasks.rados.rados.0.smithi107.stdout:5090: writing smithi10729407-17 from 1040384 to 1236992 tid 3 2020-06-12T02:38:20.249 INFO:tasks.rados.rados.0.smithi107.stdout:5091: delete oid 16 current snap is 0 2020-06-12T02:38:20.249 INFO:tasks.rados.rados.0.smithi107.stdout:5088: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:20.250 INFO:tasks.rados.rados.0.smithi107.stdout:5088: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:20.250 INFO:tasks.rados.rados.0.smithi107.stdout:5088: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:20.250 INFO:tasks.rados.rados.0.smithi107.stdout:5090: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:20.251 INFO:tasks.rados.rados.0.smithi107.stdout:5088: finishing write tid 5 to smithi10729407-7 2020-06-12T02:38:20.251 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 970 (ObjNum 5087 snap 0 seq_num 5087) dirty exists 2020-06-12T02:38:20.251 INFO:tasks.rados.rados.0.smithi107.stdout:5088: left oid 7 (ObjNum 5087 snap 0 seq_num 5087) 2020-06-12T02:38:20.256 INFO:tasks.rados.rados.0.smithi107.stdout:5088: done (3 left) 2020-06-12T02:38:20.257 INFO:tasks.rados.rados.0.smithi107.stdout:5091: done (2 left) 2020-06-12T02:38:20.257 INFO:tasks.rados.rados.0.smithi107.stdout:5092: delete oid 14 current snap is 0 2020-06-12T02:38:20.258 INFO:tasks.rados.rados.0.smithi107.stdout:5092: done (2 left) 2020-06-12T02:38:20.258 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:20.259 INFO:tasks.rados.rados.0.smithi107.stdout:5093: seq_num 5092 ranges {1056768=483328,1540096=663552,2203648=139264} 2020-06-12T02:38:20.264 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:20.264 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:20.266 INFO:tasks.rados.rados.0.smithi107.stdout:5093: writing smithi10729407-19 from 1056768 to 1540096 tid 1 2020-06-12T02:38:20.270 INFO:tasks.rados.rados.0.smithi107.stdout:5093: writing smithi10729407-19 from 1540096 to 2203648 tid 2 2020-06-12T02:38:20.271 INFO:tasks.rados.rados.0.smithi107.stdout:5093: writing smithi10729407-19 from 2203648 to 2342912 tid 3 2020-06-12T02:38:20.271 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:20.271 INFO:tasks.rados.rados.0.smithi107.stdout:5094: seq_num 5093 ranges {0=499712,499712=540672,1040384=262144} 2020-06-12T02:38:20.273 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-06-12T02:38:20.274 INFO:tasks.rados.rados.0.smithi107.stdout:5094: writing smithi10729407-6 from 0 to 499712 tid 1 2020-06-12T02:38:20.278 INFO:tasks.rados.rados.0.smithi107.stdout:5094: writing smithi10729407-6 from 499712 to 1040384 tid 2 2020-06-12T02:38:20.279 INFO:tasks.rados.rados.0.smithi107.stdout:5094: writing smithi10729407-6 from 1040384 to 1302528 tid 3 2020-06-12T02:38:20.280 INFO:tasks.rados.rados.0.smithi107.stdout:5095: delete oid 15 current snap is 0 2020-06-12T02:38:20.280 INFO:tasks.rados.rados.0.smithi107.stdout:5090: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:20.280 INFO:tasks.rados.rados.0.smithi107.stdout:5090: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:20.280 INFO:tasks.rados.rados.0.smithi107.stdout:5090: finishing write tid 5 to smithi10729407-17 2020-06-12T02:38:20.280 INFO:tasks.rados.rados.0.smithi107.stdout:5090: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:20.281 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 946 (ObjNum 5089 snap 0 seq_num 5089) dirty exists 2020-06-12T02:38:20.281 INFO:tasks.rados.rados.0.smithi107.stdout:5090: left oid 17 (ObjNum 5089 snap 0 seq_num 5089) 2020-06-12T02:38:20.281 INFO:tasks.rados.rados.0.smithi107.stdout:5093: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:20.281 INFO:tasks.rados.rados.0.smithi107.stdout:5093: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:20.282 INFO:tasks.rados.rados.0.smithi107.stdout:5094: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:20.282 INFO:tasks.rados.rados.0.smithi107.stdout:5093: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:20.282 INFO:tasks.rados.rados.0.smithi107.stdout:5093: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:20.284 INFO:tasks.rados.rados.0.smithi107.stdout:5090: done (4 left) 2020-06-12T02:38:20.284 INFO:tasks.rados.rados.0.smithi107.stdout:5095: done (3 left) 2020-06-12T02:38:20.284 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:20.284 INFO:tasks.rados.rados.0.smithi107.stdout:5096: seq_num 5095 ranges {0=540672,540672=761856,1302528=40960} 2020-06-12T02:38:20.287 INFO:tasks.rados.rados.0.smithi107.stdout:5096: writing smithi10729407-14 from 0 to 540672 tid 1 2020-06-12T02:38:20.292 INFO:tasks.rados.rados.0.smithi107.stdout:5096: writing smithi10729407-14 from 540672 to 1302528 tid 2 2020-06-12T02:38:20.293 INFO:tasks.rados.rados.0.smithi107.stdout:5096: writing smithi10729407-14 from 1302528 to 1343488 tid 3 2020-06-12T02:38:20.293 INFO:tasks.rados.rados.0.smithi107.stdout:5097: delete oid 3 current snap is 0 2020-06-12T02:38:20.293 INFO:tasks.rados.rados.0.smithi107.stdout:5093: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:20.293 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 890 (ObjNum 5092 snap 0 seq_num 5092) dirty exists 2020-06-12T02:38:20.293 INFO:tasks.rados.rados.0.smithi107.stdout:5093: left oid 19 (ObjNum 5092 snap 0 seq_num 5092) 2020-06-12T02:38:20.294 INFO:tasks.rados.rados.0.smithi107.stdout:5094: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:20.299 INFO:tasks.rados.rados.0.smithi107.stdout:5094: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:20.299 INFO:tasks.rados.rados.0.smithi107.stdout:5094: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:20.300 INFO:tasks.rados.rados.0.smithi107.stdout:5094: finishing write tid 6 to smithi10729407-6 2020-06-12T02:38:20.300 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 943 (ObjNum 5093 snap 0 seq_num 5093) dirty exists 2020-06-12T02:38:20.300 INFO:tasks.rados.rados.0.smithi107.stdout:5094: left oid 6 (ObjNum 5093 snap 0 seq_num 5093) 2020-06-12T02:38:20.301 INFO:tasks.rados.rados.0.smithi107.stdout:5093: done (4 left) 2020-06-12T02:38:20.301 INFO:tasks.rados.rados.0.smithi107.stdout:5094: done (3 left) 2020-06-12T02:38:20.301 INFO:tasks.rados.rados.0.smithi107.stdout:5097: done (2 left) 2020-06-12T02:38:20.301 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:38:20.301 INFO:tasks.rados.rados.0.smithi107.stdout:5098: seq_num 5097 ranges {1204224=573440,1777664=532480,2310144=270336} 2020-06-12T02:38:20.308 INFO:tasks.rados.rados.0.smithi107.stdout:5098: writing smithi10729407-7 from 1204224 to 1777664 tid 1 2020-06-12T02:38:20.311 INFO:tasks.rados.rados.0.smithi107.stdout:5098: writing smithi10729407-7 from 1777664 to 2310144 tid 2 2020-06-12T02:38:20.313 INFO:tasks.rados.rados.0.smithi107.stdout:5098: writing smithi10729407-7 from 2310144 to 2580480 tid 3 2020-06-12T02:38:20.313 INFO:tasks.rados.rados.0.smithi107.stdout:5099: delete oid 15 current snap is 0 2020-06-12T02:38:20.313 INFO:tasks.rados.rados.0.smithi107.stdout:5096: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:20.314 INFO:tasks.rados.rados.0.smithi107.stdout:5096: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:20.314 INFO:tasks.rados.rados.0.smithi107.stdout:5096: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:20.314 INFO:tasks.rados.rados.0.smithi107.stdout:5096: finishing write tid 5 to smithi10729407-14 2020-06-12T02:38:20.314 INFO:tasks.rados.rados.0.smithi107.stdout:5096: finishing write tid 6 to smithi10729407-14 2020-06-12T02:38:20.314 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 926 (ObjNum 5095 snap 0 seq_num 5095) dirty exists 2020-06-12T02:38:20.315 INFO:tasks.rados.rados.0.smithi107.stdout:5096: left oid 14 (ObjNum 5095 snap 0 seq_num 5095) 2020-06-12T02:38:20.315 INFO:tasks.rados.rados.0.smithi107.stdout:5098: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:20.315 INFO:tasks.rados.rados.0.smithi107.stdout:5096: done (3 left) 2020-06-12T02:38:20.316 INFO:tasks.rados.rados.0.smithi107.stdout:5099: done (2 left) 2020-06-12T02:38:20.316 INFO:tasks.rados.rados.0.smithi107.stdout:5100: delete oid 6 current snap is 0 2020-06-12T02:38:20.320 INFO:tasks.rados.rados.0.smithi107.stdout:5100: done (2 left) 2020-06-12T02:38:20.320 INFO:tasks.rados.rados.0.smithi107.stdout:5101: delete oid 4 current snap is 0 2020-06-12T02:38:20.321 INFO:tasks.rados.rados.0.smithi107.stdout:5098: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:20.323 INFO:tasks.rados.rados.0.smithi107.stdout:5098: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:20.325 INFO:tasks.rados.rados.0.smithi107.stdout:5101: done (2 left) 2020-06-12T02:38:20.326 INFO:tasks.rados.rados.0.smithi107.stdout:5102: delete oid 8 current snap is 0 2020-06-12T02:38:20.326 INFO:tasks.rados.rados.0.smithi107.stdout:5098: finishing write tid 5 to smithi10729407-7 2020-06-12T02:38:20.326 INFO:tasks.rados.rados.0.smithi107.stdout:5098: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:20.326 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 974 (ObjNum 5097 snap 0 seq_num 5097) dirty exists 2020-06-12T02:38:20.327 INFO:tasks.rados.rados.0.smithi107.stdout:5098: left oid 7 (ObjNum 5097 snap 0 seq_num 5097) 2020-06-12T02:38:20.330 INFO:tasks.rados.rados.0.smithi107.stdout:5098: done (2 left) 2020-06-12T02:38:20.330 INFO:tasks.rados.rados.0.smithi107.stdout:5102: done (1 left) 2020-06-12T02:38:20.330 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:20.330 INFO:tasks.rados.rados.0.smithi107.stdout:5103: seq_num 5102 ranges {1236992=655360,1892352=802816} 2020-06-12T02:38:20.340 INFO:tasks.rados.rados.0.smithi107.stdout:5103: writing smithi10729407-17 from 1236992 to 1892352 tid 1 2020-06-12T02:38:20.345 INFO:tasks.rados.rados.0.smithi107.stdout:5103: writing smithi10729407-17 from 1892352 to 2695168 tid 2 2020-06-12T02:38:20.345 INFO:tasks.rados.rados.0.smithi107.stdout:5104: delete oid 14 current snap is 0 2020-06-12T02:38:20.347 INFO:tasks.rados.rados.0.smithi107.stdout:5104: done (2 left) 2020-06-12T02:38:20.347 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:20.349 INFO:tasks.rados.rados.0.smithi107.stdout:5105: seq_num 5104 ranges {950272=688128,1638400=573440,2211840=229376} 2020-06-12T02:38:20.350 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:20.344+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:20.360 INFO:tasks.rados.rados.0.smithi107.stdout:5105: writing smithi10729407-13 from 950272 to 1638400 tid 1 2020-06-12T02:38:20.362 INFO:tasks.rados.rados.0.smithi107.stdout:5105: writing smithi10729407-13 from 1638400 to 2211840 tid 2 2020-06-12T02:38:20.362 INFO:tasks.rados.rados.0.smithi107.stdout:5105: writing smithi10729407-13 from 2211840 to 2441216 tid 3 2020-06-12T02:38:20.363 INFO:tasks.rados.rados.0.smithi107.stdout:5106: delete oid 3 current snap is 0 2020-06-12T02:38:20.363 INFO:tasks.rados.rados.0.smithi107.stdout:5103: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:20.363 INFO:tasks.rados.rados.0.smithi107.stdout:5103: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:20.363 INFO:tasks.rados.rados.0.smithi107.stdout:5103: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:20.363 INFO:tasks.rados.rados.0.smithi107.stdout:5103: finishing write tid 5 to smithi10729407-17 2020-06-12T02:38:20.364 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 950 (ObjNum 5102 snap 0 seq_num 5102) dirty exists 2020-06-12T02:38:20.364 INFO:tasks.rados.rados.0.smithi107.stdout:5103: left oid 17 (ObjNum 5102 snap 0 seq_num 5102) 2020-06-12T02:38:20.364 INFO:tasks.rados.rados.0.smithi107.stdout:5103: done (3 left) 2020-06-12T02:38:20.365 INFO:tasks.rados.rados.0.smithi107.stdout:5106: done (2 left) 2020-06-12T02:38:20.365 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:20.365 INFO:tasks.rados.rados.0.smithi107.stdout:5107: seq_num 5106 ranges {2342912=720896,3063808=737280,3801088=65536} 2020-06-12T02:38:20.385 INFO:tasks.rados.rados.0.smithi107.stdout:5107: writing smithi10729407-19 from 2342912 to 3063808 tid 1 2020-06-12T02:38:20.386 INFO:tasks.rados.rados.0.smithi107.stdout:5107: writing smithi10729407-19 from 3063808 to 3801088 tid 2 2020-06-12T02:38:20.387 INFO:tasks.rados.rados.0.smithi107.stdout:5107: writing smithi10729407-19 from 3801088 to 3866624 tid 3 2020-06-12T02:38:20.387 INFO:tasks.rados.rados.0.smithi107.stdout:5108: delete oid 11 current snap is 0 2020-06-12T02:38:20.387 INFO:tasks.rados.rados.0.smithi107.stdout:5105: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:20.387 INFO:tasks.rados.rados.0.smithi107.stdout:5105: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:20.387 INFO:tasks.rados.rados.0.smithi107.stdout:5105: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:20.388 INFO:tasks.rados.rados.0.smithi107.stdout:5105: finishing write tid 5 to smithi10729407-13 2020-06-12T02:38:20.388 INFO:tasks.rados.rados.0.smithi107.stdout:5105: finishing write tid 6 to smithi10729407-13 2020-06-12T02:38:20.388 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 978 (ObjNum 5104 snap 0 seq_num 5104) dirty exists 2020-06-12T02:38:20.388 INFO:tasks.rados.rados.0.smithi107.stdout:5105: left oid 13 (ObjNum 5104 snap 0 seq_num 5104) 2020-06-12T02:38:20.390 INFO:tasks.rados.rados.0.smithi107.stdout:5105: done (3 left) 2020-06-12T02:38:20.390 INFO:tasks.rados.rados.0.smithi107.stdout:5108: done (2 left) 2020-06-12T02:38:20.390 INFO:tasks.rados.rados.0.smithi107.stdout:5109: delete oid 4 current snap is 0 2020-06-12T02:38:20.392 INFO:tasks.rados.rados.0.smithi107.stdout:5109: done (2 left) 2020-06-12T02:38:20.392 INFO:tasks.rados.rados.0.smithi107.stdout:5110: delete oid 18 current snap is 0 2020-06-12T02:38:20.394 INFO:tasks.rados.rados.0.smithi107.stdout:5107: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:20.394 INFO:tasks.rados.rados.0.smithi107.stdout:5107: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:20.395 INFO:tasks.rados.rados.0.smithi107.stdout:5110: done (2 left) 2020-06-12T02:38:20.405 INFO:tasks.rados.rados.0.smithi107.stdout:5111: delete oid 20 current snap is 0 2020-06-12T02:38:20.408 INFO:tasks.rados.rados.0.smithi107.stdout:5111: done (2 left) 2020-06-12T02:38:20.408 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:20.409 INFO:tasks.rados.rados.0.smithi107.stdout:5112: seq_num 5111 ranges {0=802816,802816=614400,1417216=188416} 2020-06-12T02:38:20.409 INFO:tasks.rados.rados.0.smithi107.stdout:5112: writing smithi10729407-18 from 0 to 802816 tid 1 2020-06-12T02:38:20.409 INFO:tasks.rados.rados.0.smithi107.stdout:5112: writing smithi10729407-18 from 802816 to 1417216 tid 2 2020-06-12T02:38:20.410 INFO:tasks.rados.rados.0.smithi107.stdout:5112: writing smithi10729407-18 from 1417216 to 1605632 tid 3 2020-06-12T02:38:20.410 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:20.410 INFO:tasks.rados.rados.0.smithi107.stdout:5113: seq_num 5112 ranges {0=417792,417792=491520,909312=712704} 2020-06-12T02:38:20.410 INFO:tasks.rados.rados.0.smithi107.stdout:5113: writing smithi10729407-14 from 0 to 417792 tid 1 2020-06-12T02:38:20.413 INFO:tasks.rados.rados.0.smithi107.stdout:5113: writing smithi10729407-14 from 417792 to 909312 tid 2 2020-06-12T02:38:20.418 INFO:tasks.rados.rados.0.smithi107.stdout:5113: writing smithi10729407-14 from 909312 to 1622016 tid 3 2020-06-12T02:38:20.418 INFO:tasks.rados.rados.0.smithi107.stdout:5114: delete oid 9 current snap is 0 2020-06-12T02:38:20.418 INFO:tasks.rados.rados.0.smithi107.stdout:5107: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:20.419 INFO:tasks.rados.rados.0.smithi107.stdout:5107: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:20.419 INFO:tasks.rados.rados.0.smithi107.stdout:5107: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:20.419 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 894 (ObjNum 5106 snap 0 seq_num 5106) dirty exists 2020-06-12T02:38:20.419 INFO:tasks.rados.rados.0.smithi107.stdout:5107: left oid 19 (ObjNum 5106 snap 0 seq_num 5106) 2020-06-12T02:38:20.419 INFO:tasks.rados.rados.0.smithi107.stdout:5112: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:20.420 INFO:tasks.rados.rados.0.smithi107.stdout:5112: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:20.422 INFO:tasks.rados.rados.0.smithi107.stdout:5113: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:20.427 INFO:tasks.rados.rados.0.smithi107.stdout:5113: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:20.427 INFO:tasks.rados.rados.0.smithi107.stdout:5112: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:20.427 INFO:tasks.rados.rados.0.smithi107.stdout:5112: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:20.427 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:20.428 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:20.428 INFO:tasks.rados.rados.0.smithi107.stdout:5113: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:20.429 INFO:tasks.rados.rados.0.smithi107.stdout:5113: finishing write tid 5 to smithi10729407-14 2020-06-12T02:38:20.429 INFO:tasks.rados.rados.0.smithi107.stdout:5107: done (4 left) 2020-06-12T02:38:20.429 INFO:tasks.rados.rados.0.smithi107.stdout:5114: done (3 left) 2020-06-12T02:38:20.429 INFO:tasks.rados.rados.0.smithi107.stdout:5115: delete oid 15 current snap is 0 2020-06-12T02:38:20.430 INFO:tasks.rados.rados.0.smithi107.stdout:5112: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:20.430 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 915 (ObjNum 5111 snap 0 seq_num 5111) dirty exists 2020-06-12T02:38:20.430 INFO:tasks.rados.rados.0.smithi107.stdout:5112: left oid 18 (ObjNum 5111 snap 0 seq_num 5111) 2020-06-12T02:38:20.431 INFO:tasks.rados.rados.0.smithi107.stdout:5113: finishing write tid 6 to smithi10729407-14 2020-06-12T02:38:20.431 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 932 (ObjNum 5112 snap 0 seq_num 5112) dirty exists 2020-06-12T02:38:20.431 INFO:tasks.rados.rados.0.smithi107.stdout:5113: left oid 14 (ObjNum 5112 snap 0 seq_num 5112) 2020-06-12T02:38:20.431 INFO:tasks.rados.rados.0.smithi107.stdout:5112: done (3 left) 2020-06-12T02:38:20.432 INFO:tasks.rados.rados.0.smithi107.stdout:5113: done (2 left) 2020-06-12T02:38:20.432 INFO:tasks.rados.rados.0.smithi107.stdout:5115: done (1 left) 2020-06-12T02:38:20.432 INFO:tasks.rados.rados.0.smithi107.stdout:5116: delete oid 20 current snap is 0 2020-06-12T02:38:20.433 INFO:tasks.rados.rados.0.smithi107.stdout:5116: done (1 left) 2020-06-12T02:38:20.433 INFO:tasks.rados.rados.0.smithi107.stdout:5117: delete oid 12 current snap is 0 2020-06-12T02:38:20.435 INFO:tasks.rados.rados.0.smithi107.stdout:5117: done (1 left) 2020-06-12T02:38:20.435 INFO:tasks.rados.rados.0.smithi107.stdout:5118: delete oid 20 current snap is 0 2020-06-12T02:38:20.437 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-06-12T02:38:20.438 INFO:tasks.rados.rados.0.smithi107.stdout:5118: done (1 left) 2020-06-12T02:38:20.438 INFO:tasks.rados.rados.0.smithi107.stdout:5119: delete oid 2 current snap is 0 2020-06-12T02:38:20.439 INFO:tasks.rados.rados.0.smithi107.stdout:5119: done (1 left) 2020-06-12T02:38:20.439 INFO:tasks.rados.rados.0.smithi107.stdout:5120: delete oid 8 current snap is 0 2020-06-12T02:38:20.440 INFO:tasks.rados.rados.0.smithi107.stdout:5120: done (1 left) 2020-06-12T02:38:20.441 INFO:tasks.rados.rados.0.smithi107.stdout:5121: delete oid 9 current snap is 0 2020-06-12T02:38:20.443 INFO:tasks.rados.rados.0.smithi107.stdout:5121: done (1 left) 2020-06-12T02:38:20.443 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:20.443 INFO:tasks.rados.rados.0.smithi107.stdout:5122: seq_num 5121 ranges {0=573440,573440=704512,1277952=499712} 2020-06-12T02:38:20.446 INFO:tasks.rados.rados.0.smithi107.stdout:5122: writing smithi10729407-15 from 0 to 573440 tid 1 2020-06-12T02:38:20.448 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:20.445+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:20.450 INFO:tasks.rados.rados.0.smithi107.stdout:5122: writing smithi10729407-15 from 573440 to 1277952 tid 2 2020-06-12T02:38:20.454 INFO:tasks.rados.rados.0.smithi107.stdout:5122: writing smithi10729407-15 from 1277952 to 1777664 tid 3 2020-06-12T02:38:20.454 INFO:tasks.rados.rados.0.smithi107.stdout:5123: delete oid 13 current snap is 0 2020-06-12T02:38:20.454 INFO:tasks.rados.rados.0.smithi107.stdout:5122: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:20.462 INFO:tasks.rados.rados.0.smithi107.stdout:5123: done (2 left) 2020-06-12T02:38:20.462 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:38:20.462 INFO:tasks.rados.rados.0.smithi107.stdout:5124: seq_num 5123 ranges {0=606208,606208=475136,1081344=540672,1622016=188416} 2020-06-12T02:38:20.466 INFO:tasks.rados.rados.0.smithi107.stdout:5124: writing smithi10729407-3 from 0 to 606208 tid 1 2020-06-12T02:38:20.468 INFO:tasks.rados.rados.0.smithi107.stdout:5124: writing smithi10729407-3 from 606208 to 1081344 tid 2 2020-06-12T02:38:20.472 INFO:tasks.rados.rados.0.smithi107.stdout:5124: writing smithi10729407-3 from 1081344 to 1622016 tid 3 2020-06-12T02:38:20.473 INFO:tasks.rados.rados.0.smithi107.stdout:5124: writing smithi10729407-3 from 1622016 to 1810432 tid 4 2020-06-12T02:38:20.474 INFO:tasks.rados.rados.0.smithi107.stdout:5125: delete oid 20 current snap is 0 2020-06-12T02:38:20.474 INFO:tasks.rados.rados.0.smithi107.stdout:5122: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:20.474 INFO:tasks.rados.rados.0.smithi107.stdout:5122: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:20.474 INFO:tasks.rados.rados.0.smithi107.stdout:5122: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:20.474 INFO:tasks.rados.rados.0.smithi107.stdout:5122: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:20.475 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 892 (ObjNum 5121 snap 0 seq_num 5121) dirty exists 2020-06-12T02:38:20.475 INFO:tasks.rados.rados.0.smithi107.stdout:5122: left oid 15 (ObjNum 5121 snap 0 seq_num 5121) 2020-06-12T02:38:20.475 INFO:tasks.rados.rados.0.smithi107.stdout:5124: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:20.476 INFO:tasks.rados.rados.0.smithi107.stdout:5122: done (3 left) 2020-06-12T02:38:20.476 INFO:tasks.rados.rados.0.smithi107.stdout:5125: done (2 left) 2020-06-12T02:38:20.476 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:20.476 INFO:tasks.rados.rados.0.smithi107.stdout:5126: seq_num 5125 ranges {1777664=638976,2416640=688128,3104768=516096} 2020-06-12T02:38:20.489 INFO:tasks.rados.rados.0.smithi107.stdout:5126: writing smithi10729407-15 from 1777664 to 2416640 tid 1 2020-06-12T02:38:20.493 INFO:tasks.rados.rados.0.smithi107.stdout:5126: writing smithi10729407-15 from 2416640 to 3104768 tid 2 2020-06-12T02:38:20.496 INFO:tasks.rados.rados.0.smithi107.stdout:5126: writing smithi10729407-15 from 3104768 to 3620864 tid 3 2020-06-12T02:38:20.497 INFO:tasks.rados.rados.0.smithi107.stdout:5127: delete oid 20 current snap is 0 2020-06-12T02:38:20.497 INFO:tasks.rados.rados.0.smithi107.stdout:5124: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:20.497 INFO:tasks.rados.rados.0.smithi107.stdout:5124: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:20.497 INFO:tasks.rados.rados.0.smithi107.stdout:5124: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:20.497 INFO:tasks.rados.rados.0.smithi107.stdout:5124: finishing write tid 6 to smithi10729407-3 2020-06-12T02:38:20.498 INFO:tasks.rados.rados.0.smithi107.stdout:5124: finishing write tid 7 to smithi10729407-3 2020-06-12T02:38:20.498 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 918 (ObjNum 5123 snap 0 seq_num 5123) dirty exists 2020-06-12T02:38:20.498 INFO:tasks.rados.rados.0.smithi107.stdout:5124: left oid 3 (ObjNum 5123 snap 0 seq_num 5123) 2020-06-12T02:38:20.505 INFO:tasks.rados.rados.0.smithi107.stdout:5124: done (3 left) 2020-06-12T02:38:20.505 INFO:tasks.rados.rados.0.smithi107.stdout:5127: done (2 left) 2020-06-12T02:38:20.506 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:20.506 INFO:tasks.rados.rados.0.smithi107.stdout:5128: seq_num 5127 ranges {0=671744,671744=458752,1130496=516096,1646592=229376} 2020-06-12T02:38:20.508 INFO:tasks.rados.rados.0.smithi107.stdout:5128: writing smithi10729407-12 from 0 to 671744 tid 1 2020-06-12T02:38:20.511 INFO:tasks.rados.rados.0.smithi107.stdout:5128: writing smithi10729407-12 from 671744 to 1130496 tid 2 2020-06-12T02:38:20.514 INFO:tasks.rados.rados.0.smithi107.stdout:5128: writing smithi10729407-12 from 1130496 to 1646592 tid 3 2020-06-12T02:38:20.515 INFO:tasks.rados.rados.0.smithi107.stdout:5128: writing smithi10729407-12 from 1646592 to 1875968 tid 4 2020-06-12T02:38:20.516 INFO:tasks.rados.rados.0.smithi107.stdout:5129: delete oid 9 current snap is 0 2020-06-12T02:38:20.516 INFO:tasks.rados.rados.0.smithi107.stdout:5126: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:20.516 INFO:tasks.rados.rados.0.smithi107.stdout:5126: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:20.516 INFO:tasks.rados.rados.0.smithi107.stdout:5126: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:20.516 INFO:tasks.rados.rados.0.smithi107.stdout:5126: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:20.517 INFO:tasks.rados.rados.0.smithi107.stdout:5126: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:20.517 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 896 (ObjNum 5125 snap 0 seq_num 5125) dirty exists 2020-06-12T02:38:20.517 INFO:tasks.rados.rados.0.smithi107.stdout:5126: left oid 15 (ObjNum 5125 snap 0 seq_num 5125) 2020-06-12T02:38:20.517 INFO:tasks.rados.rados.0.smithi107.stdout:5128: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:20.518 INFO:tasks.rados.rados.0.smithi107.stdout:5126: done (3 left) 2020-06-12T02:38:20.518 INFO:tasks.rados.rados.0.smithi107.stdout:5129: done (2 left) 2020-06-12T02:38:20.518 INFO:tasks.rados.rados.0.smithi107.stdout:5130: delete oid 15 current snap is 0 2020-06-12T02:38:20.526 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-06-12T02:38:20.527 INFO:tasks.rados.rados.0.smithi107.stdout:5128: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:20.527 INFO:tasks.rados.rados.0.smithi107.stdout:5130: done (2 left) 2020-06-12T02:38:20.527 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:20.528 INFO:tasks.rados.rados.0.smithi107.stdout:5131: seq_num 5130 ranges {3866624=720896,4587520=516096,5103616=688128} 2020-06-12T02:38:20.547 INFO:tasks.rados.rados.0.smithi107.stdout:5131: writing smithi10729407-19 from 3866624 to 4587520 tid 1 2020-06-12T02:38:20.550 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:20.547+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:20.551 INFO:tasks.rados.rados.0.smithi107.stdout:5131: writing smithi10729407-19 from 4587520 to 5103616 tid 2 2020-06-12T02:38:20.555 INFO:tasks.rados.rados.0.smithi107.stdout:5131: writing smithi10729407-19 from 5103616 to 5791744 tid 3 2020-06-12T02:38:20.555 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:20.555 INFO:tasks.rados.rados.0.smithi107.stdout:5132: seq_num 5131 ranges {1810432=737280,2547712=794624,3342336=409600} 2020-06-12T02:38:20.569 INFO:tasks.rados.rados.0.smithi107.stdout:5132: writing smithi10729407-3 from 1810432 to 2547712 tid 1 2020-06-12T02:38:20.574 INFO:tasks.rados.rados.0.smithi107.stdout:5132: writing smithi10729407-3 from 2547712 to 3342336 tid 2 2020-06-12T02:38:20.577 INFO:tasks.rados.rados.0.smithi107.stdout:5132: writing smithi10729407-3 from 3342336 to 3751936 tid 3 2020-06-12T02:38:20.578 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:20.578 INFO:tasks.rados.rados.0.smithi107.stdout:5133: seq_num 5132 ranges {0=753664,753664=729088,1482752=475136} 2020-06-12T02:38:20.578 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:20.578 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:20.582 INFO:tasks.rados.rados.0.smithi107.stdout:5133: writing smithi10729407-5 from 0 to 753664 tid 1 2020-06-12T02:38:20.587 INFO:tasks.rados.rados.0.smithi107.stdout:5133: writing smithi10729407-5 from 753664 to 1482752 tid 2 2020-06-12T02:38:20.590 INFO:tasks.rados.rados.0.smithi107.stdout:5133: writing smithi10729407-5 from 1482752 to 1957888 tid 3 2020-06-12T02:38:20.591 INFO:tasks.rados.rados.0.smithi107.stdout:5134: delete oid 14 current snap is 0 2020-06-12T02:38:20.591 INFO:tasks.rados.rados.0.smithi107.stdout:5128: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:20.591 INFO:tasks.rados.rados.0.smithi107.stdout:5128: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:20.591 INFO:tasks.rados.rados.0.smithi107.stdout:5128: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:20.591 INFO:tasks.rados.rados.0.smithi107.stdout:5128: finishing write tid 7 to smithi10729407-12 2020-06-12T02:38:20.592 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 900 (ObjNum 5127 snap 0 seq_num 5127) dirty exists 2020-06-12T02:38:20.592 INFO:tasks.rados.rados.0.smithi107.stdout:5128: left oid 12 (ObjNum 5127 snap 0 seq_num 5127) 2020-06-12T02:38:20.592 INFO:tasks.rados.rados.0.smithi107.stdout:5131: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:20.592 INFO:tasks.rados.rados.0.smithi107.stdout:5131: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:20.592 INFO:tasks.rados.rados.0.smithi107.stdout:5131: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:20.595 INFO:tasks.rados.rados.0.smithi107.stdout:5131: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:20.595 INFO:tasks.rados.rados.0.smithi107.stdout:5131: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:20.595 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 904 (ObjNum 5130 snap 0 seq_num 5130) dirty exists 2020-06-12T02:38:20.596 INFO:tasks.rados.rados.0.smithi107.stdout:5131: left oid 19 (ObjNum 5130 snap 0 seq_num 5130) 2020-06-12T02:38:20.596 INFO:tasks.rados.rados.0.smithi107.stdout:5132: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:20.596 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd unset nodeep-scrub 2020-06-12T02:38:20.598 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-06-12T02:38:20.599 INFO:tasks.rados.rados.0.smithi107.stdout:5132: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:20.599 INFO:tasks.rados.rados.0.smithi107.stdout:5132: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:20.599 INFO:tasks.rados.rados.0.smithi107.stdout:5132: finishing write tid 5 to smithi10729407-3 2020-06-12T02:38:20.599 INFO:tasks.rados.rados.0.smithi107.stdout:5132: finishing write tid 6 to smithi10729407-3 2020-06-12T02:38:20.599 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 922 (ObjNum 5131 snap 0 seq_num 5131) dirty exists 2020-06-12T02:38:20.600 INFO:tasks.rados.rados.0.smithi107.stdout:5132: left oid 3 (ObjNum 5131 snap 0 seq_num 5131) 2020-06-12T02:38:20.600 INFO:tasks.rados.rados.0.smithi107.stdout:5133: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:20.600 INFO:tasks.rados.rados.0.smithi107.stdout:5128: done (5 left) 2020-06-12T02:38:20.600 INFO:tasks.rados.rados.0.smithi107.stdout:5131: done (4 left) 2020-06-12T02:38:20.600 INFO:tasks.rados.rados.0.smithi107.stdout:5132: done (3 left) 2020-06-12T02:38:20.601 INFO:tasks.rados.rados.0.smithi107.stdout:5134: done (2 left) 2020-06-12T02:38:20.601 INFO:tasks.rados.rados.0.smithi107.stdout:5135: delete oid 12 current snap is 0 2020-06-12T02:38:20.602 INFO:tasks.rados.rados.0.smithi107.stdout:5133: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:20.602 INFO:tasks.rados.rados.0.smithi107.stdout:5135: done (2 left) 2020-06-12T02:38:20.602 INFO:tasks.rados.rados.0.smithi107.stdout:5136: delete oid 16 current snap is 0 2020-06-12T02:38:20.603 INFO:tasks.rados.rados.0.smithi107.stdout:5133: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:20.603 INFO:tasks.rados.rados.0.smithi107.stdout:5133: finishing write tid 5 to smithi10729407-5 2020-06-12T02:38:20.603 INFO:tasks.rados.rados.0.smithi107.stdout:5136: done (2 left) 2020-06-12T02:38:20.603 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:20.603 INFO:tasks.rados.rados.0.smithi107.stdout:5137: seq_num 5136 ranges {0=425984,425984=671744,1097728=688128,1785856=245760} 2020-06-12T02:38:20.606 INFO:tasks.rados.rados.0.smithi107.stdout:5137: writing smithi10729407-9 from 0 to 425984 tid 1 2020-06-12T02:38:20.608 INFO:tasks.rados.rados.0.smithi107.stdout:5137: writing smithi10729407-9 from 425984 to 1097728 tid 2 2020-06-12T02:38:20.613 INFO:tasks.rados.rados.0.smithi107.stdout:5137: writing smithi10729407-9 from 1097728 to 1785856 tid 3 2020-06-12T02:38:20.615 INFO:tasks.rados.rados.0.smithi107.stdout:5137: writing smithi10729407-9 from 1785856 to 2031616 tid 4 2020-06-12T02:38:20.615 INFO:tasks.rados.rados.0.smithi107.stdout:5138: delete oid 2 current snap is 0 2020-06-12T02:38:20.616 INFO:tasks.rados.rados.0.smithi107.stdout:5133: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:20.616 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 497 (ObjNum 5132 snap 0 seq_num 5132) dirty exists 2020-06-12T02:38:20.616 INFO:tasks.rados.rados.0.smithi107.stdout:5133: left oid 5 (ObjNum 5132 snap 0 seq_num 5132) 2020-06-12T02:38:20.616 INFO:tasks.rados.rados.0.smithi107.stdout:5137: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:20.618 INFO:tasks.rados.rados.0.smithi107.stdout:5133: done (3 left) 2020-06-12T02:38:20.618 INFO:tasks.rados.rados.0.smithi107.stdout:5138: done (2 left) 2020-06-12T02:38:20.618 INFO:tasks.rados.rados.0.smithi107.stdout:5139: delete oid 5 current snap is 0 2020-06-12T02:38:20.619 INFO:tasks.rados.rados.0.smithi107.stdout:5137: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:20.621 INFO:tasks.rados.rados.0.smithi107.stdout:5139: done (2 left) 2020-06-12T02:38:20.621 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:20.621 INFO:tasks.rados.rados.0.smithi107.stdout:5140: seq_num 5139 ranges {0=475136,475136=720896,1196032=507904,1703936=376832} 2020-06-12T02:38:20.625 INFO:tasks.rados.rados.0.smithi107.stdout:5140: writing smithi10729407-5 from 0 to 475136 tid 1 2020-06-12T02:38:20.630 INFO:tasks.rados.rados.0.smithi107.stdout:5140: writing smithi10729407-5 from 475136 to 1196032 tid 2 2020-06-12T02:38:20.633 INFO:tasks.rados.rados.0.smithi107.stdout:5140: writing smithi10729407-5 from 1196032 to 1703936 tid 3 2020-06-12T02:38:20.636 INFO:tasks.rados.rados.0.smithi107.stdout:5140: writing smithi10729407-5 from 1703936 to 2080768 tid 4 2020-06-12T02:38:20.636 INFO:tasks.rados.rados.0.smithi107.stdout:5141: delete oid 4 current snap is 0 2020-06-12T02:38:20.636 INFO:tasks.rados.rados.0.smithi107.stdout:5137: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:20.637 INFO:tasks.rados.rados.0.smithi107.stdout:5137: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:20.637 INFO:tasks.rados.rados.0.smithi107.stdout:5137: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:20.637 INFO:tasks.rados.rados.0.smithi107.stdout:5137: finishing write tid 7 to smithi10729407-9 2020-06-12T02:38:20.637 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 923 (ObjNum 5136 snap 0 seq_num 5136) dirty exists 2020-06-12T02:38:20.637 INFO:tasks.rados.rados.0.smithi107.stdout:5137: left oid 9 (ObjNum 5136 snap 0 seq_num 5136) 2020-06-12T02:38:20.638 INFO:tasks.rados.rados.0.smithi107.stdout:5140: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:20.641 INFO:tasks.rados.rados.0.smithi107.stdout:5140: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:20.641 INFO:tasks.rados.rados.0.smithi107.stdout:5137: done (3 left) 2020-06-12T02:38:20.642 INFO:tasks.rados.rados.0.smithi107.stdout:5141: done (2 left) 2020-06-12T02:38:20.642 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:20.642 INFO:tasks.rados.rados.0.smithi107.stdout:5142: seq_num 5141 ranges {2031616=507904,2539520=540672,3080192=475136,3555328=589824} 2020-06-12T02:38:20.651 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:20.648+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:20.655 INFO:tasks.rados.rados.0.smithi107.stdout:5142: writing smithi10729407-9 from 2031616 to 2539520 tid 1 2020-06-12T02:38:20.658 INFO:tasks.rados.rados.0.smithi107.stdout:5142: writing smithi10729407-9 from 2539520 to 3080192 tid 2 2020-06-12T02:38:20.662 INFO:tasks.rados.rados.0.smithi107.stdout:5142: writing smithi10729407-9 from 3080192 to 3555328 tid 3 2020-06-12T02:38:20.666 INFO:tasks.rados.rados.0.smithi107.stdout:5142: writing smithi10729407-9 from 3555328 to 4145152 tid 4 2020-06-12T02:38:20.666 INFO:tasks.rados.rados.0.smithi107.stdout:5140: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:20.666 INFO:tasks.rados.rados.0.smithi107.stdout:5140: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:20.666 INFO:tasks.rados.rados.0.smithi107.stdout:5140: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:20.666 INFO:tasks.rados.rados.0.smithi107.stdout:5140: finishing write tid 7 to smithi10729407-5 2020-06-12T02:38:20.667 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 503 (ObjNum 5139 snap 0 seq_num 5139) dirty exists 2020-06-12T02:38:20.667 INFO:tasks.rados.rados.0.smithi107.stdout:5140: left oid 5 (ObjNum 5139 snap 0 seq_num 5139) 2020-06-12T02:38:20.667 INFO:tasks.rados.rados.0.smithi107.stdout:5140: done (2 left) 2020-06-12T02:38:20.667 INFO:tasks.rados.rados.0.smithi107.stdout:5143: delete oid 4 current snap is 0 2020-06-12T02:38:20.675 INFO:tasks.rados.rados.0.smithi107.stdout:5142: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:20.675 INFO:tasks.rados.rados.0.smithi107.stdout:5143: done (2 left) 2020-06-12T02:38:20.676 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:20.676 INFO:tasks.rados.rados.0.smithi107.stdout:5144: seq_num 5143 ranges {2080768=540672,2621440=712704,3334144=491520,3825664=401408} 2020-06-12T02:38:20.690 INFO:tasks.rados.rados.0.smithi107.stdout:5144: writing smithi10729407-5 from 2080768 to 2621440 tid 1 2020-06-12T02:38:20.697 INFO:tasks.rados.rados.0.smithi107.stdout:5144: writing smithi10729407-5 from 2621440 to 3334144 tid 2 2020-06-12T02:38:20.700 INFO:tasks.rados.rados.0.smithi107.stdout:5144: writing smithi10729407-5 from 3334144 to 3825664 tid 3 2020-06-12T02:38:20.713 INFO:tasks.rados.rados.0.smithi107.stdout:5144: writing smithi10729407-5 from 3825664 to 4227072 tid 4 2020-06-12T02:38:20.714 INFO:tasks.rados.rados.0.smithi107.stdout:5142: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:20.714 INFO:tasks.rados.rados.0.smithi107.stdout:5142: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:20.714 INFO:tasks.rados.rados.0.smithi107.stdout:5142: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:20.714 INFO:tasks.rados.rados.0.smithi107.stdout:5142: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:20.715 INFO:tasks.rados.rados.0.smithi107.stdout:5142: finishing write tid 7 to smithi10729407-9 2020-06-12T02:38:20.715 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 928 (ObjNum 5141 snap 0 seq_num 5141) dirty exists 2020-06-12T02:38:20.715 INFO:tasks.rados.rados.0.smithi107.stdout:5142: left oid 9 (ObjNum 5141 snap 0 seq_num 5141) 2020-06-12T02:38:20.715 INFO:tasks.rados.rados.0.smithi107.stdout:5144: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:20.715 INFO:tasks.rados.rados.0.smithi107.stdout:5142: done (2 left) 2020-06-12T02:38:20.715 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:20.716 INFO:tasks.rados.rados.0.smithi107.stdout:5145: seq_num 5144 ranges {2695168=557056,3252224=589824,3842048=647168,4489216=368640} 2020-06-12T02:38:20.722 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:38:20.722 INFO:tasks.rados.rados.0.smithi107.stdout:5145: writing smithi10729407-17 from 2695168 to 3252224 tid 1 2020-06-12T02:38:20.727 INFO:tasks.rados.rados.0.smithi107.stdout:5145: writing smithi10729407-17 from 3252224 to 3842048 tid 2 2020-06-12T02:38:20.731 INFO:tasks.rados.rados.0.smithi107.stdout:5145: writing smithi10729407-17 from 3842048 to 4489216 tid 3 2020-06-12T02:38:20.734 INFO:tasks.rados.rados.0.smithi107.stdout:5145: writing smithi10729407-17 from 4489216 to 4857856 tid 4 2020-06-12T02:38:20.734 INFO:tasks.rados.rados.0.smithi107.stdout:5144: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:20.734 INFO:tasks.rados.rados.0.smithi107.stdout:5144: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:20.735 INFO:tasks.rados.rados.0.smithi107.stdout:5144: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:20.735 INFO:tasks.rados.rados.0.smithi107.stdout:5144: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:20.735 INFO:tasks.rados.rados.0.smithi107.stdout:5144: finishing write tid 7 to smithi10729407-5 2020-06-12T02:38:20.735 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 508 (ObjNum 5143 snap 0 seq_num 5143) dirty exists 2020-06-12T02:38:20.735 INFO:tasks.rados.rados.0.smithi107.stdout:5144: left oid 5 (ObjNum 5143 snap 0 seq_num 5143) 2020-06-12T02:38:20.736 INFO:tasks.rados.rados.0.smithi107.stdout:5145: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:20.736 INFO:tasks.rados.rados.0.smithi107.stdout:5144: done (2 left) 2020-06-12T02:38:20.736 INFO:tasks.rados.rados.0.smithi107.stdout:5146: delete oid 12 current snap is 0 2020-06-12T02:38:20.736 INFO:tasks.rados.rados.0.smithi107.stdout:5146: done (2 left) 2020-06-12T02:38:20.737 INFO:tasks.rados.rados.0.smithi107.stdout:5147: delete oid 11 current snap is 0 2020-06-12T02:38:20.739 INFO:tasks.rados.rados.0.smithi107.stdout:5147: done (2 left) 2020-06-12T02:38:20.739 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:38:20.739 INFO:tasks.rados.rados.0.smithi107.stdout:5148: seq_num 5147 ranges {2580480=606208,3186688=696320,3883008=466944,4349952=442368} 2020-06-12T02:38:20.752 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:20.750+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:20.753 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:38:20.755 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:20.755 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:20.757 INFO:tasks.rados.rados.0.smithi107.stdout:5148: writing smithi10729407-7 from 2580480 to 3186688 tid 1 2020-06-12T02:38:20.761 INFO:tasks.rados.rados.0.smithi107.stdout:5148: writing smithi10729407-7 from 3186688 to 3883008 tid 2 2020-06-12T02:38:20.764 INFO:tasks.rados.rados.0.smithi107.stdout:5148: writing smithi10729407-7 from 3883008 to 4349952 tid 3 2020-06-12T02:38:20.766 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-06-12T02:38:20.767 INFO:tasks.rados.rados.0.smithi107.stdout:5148: writing smithi10729407-7 from 4349952 to 4792320 tid 4 2020-06-12T02:38:20.767 INFO:tasks.rados.rados.0.smithi107.stdout:5149: delete oid 18 current snap is 0 2020-06-12T02:38:20.767 INFO:tasks.rados.rados.0.smithi107.stdout:5145: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:20.768 INFO:tasks.rados.rados.0.smithi107.stdout:5145: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:20.768 INFO:tasks.rados.rados.0.smithi107.stdout:5145: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:20.768 INFO:tasks.rados.rados.0.smithi107.stdout:5145: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:20.768 INFO:tasks.rados.rados.0.smithi107.stdout:5145: finishing write tid 7 to smithi10729407-17 2020-06-12T02:38:20.768 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 958 (ObjNum 5144 snap 0 seq_num 5144) dirty exists 2020-06-12T02:38:20.769 INFO:tasks.rados.rados.0.smithi107.stdout:5145: left oid 17 (ObjNum 5144 snap 0 seq_num 5144) 2020-06-12T02:38:20.769 INFO:tasks.rados.rados.0.smithi107.stdout:5148: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:20.770 INFO:tasks.rados.rados.0.smithi107.stdout:5145: done (3 left) 2020-06-12T02:38:20.771 INFO:tasks.rados.rados.0.smithi107.stdout:5149: done (2 left) 2020-06-12T02:38:20.771 INFO:tasks.rados.rados.0.smithi107.stdout:5150: delete oid 18 current snap is 0 2020-06-12T02:38:20.774 INFO:tasks.rados.rados.0.smithi107.stdout:5150: done (2 left) 2020-06-12T02:38:20.774 INFO:tasks.rados.rados.0.smithi107.stdout:5151: delete oid 20 current snap is 0 2020-06-12T02:38:20.775 INFO:tasks.rados.rados.0.smithi107.stdout:5148: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:20.776 INFO:tasks.rados.rados.0.smithi107.stdout:5151: done (2 left) 2020-06-12T02:38:20.776 INFO:tasks.rados.rados.0.smithi107.stdout:5152: delete oid 15 current snap is 0 2020-06-12T02:38:20.777 INFO:tasks.rados.rados.0.smithi107.stdout:5148: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:20.778 INFO:tasks.rados.rados.0.smithi107.stdout:5148: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:20.779 INFO:tasks.rados.rados.0.smithi107.stdout:5148: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:20.779 INFO:tasks.rados.rados.0.smithi107.stdout:5152: done (2 left) 2020-06-12T02:38:20.779 INFO:tasks.rados.rados.0.smithi107.stdout:5153: delete oid 11 current snap is 0 2020-06-12T02:38:20.780 INFO:tasks.rados.rados.0.smithi107.stdout:5148: finishing write tid 7 to smithi10729407-7 2020-06-12T02:38:20.780 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 984 (ObjNum 5147 snap 0 seq_num 5147) dirty exists 2020-06-12T02:38:20.781 INFO:tasks.rados.rados.0.smithi107.stdout:5148: left oid 7 (ObjNum 5147 snap 0 seq_num 5147) 2020-06-12T02:38:20.781 INFO:tasks.rados.rados.0.smithi107.stdout:5148: done (2 left) 2020-06-12T02:38:20.782 INFO:tasks.rados.rados.0.smithi107.stdout:5153: done (1 left) 2020-06-12T02:38:20.782 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:20.782 INFO:tasks.rados.rados.0.smithi107.stdout:5154: seq_num 5153 ranges {0=712704,712704=802816,1515520=417792,1933312=385024} 2020-06-12T02:38:20.786 INFO:tasks.rados.rados.0.smithi107.stdout:5154: writing smithi10729407-8 from 0 to 712704 tid 1 2020-06-12T02:38:20.790 INFO:tasks.rados.rados.0.smithi107.stdout:5154: writing smithi10729407-8 from 712704 to 1515520 tid 2 2020-06-12T02:38:20.793 INFO:tasks.rados.rados.0.smithi107.stdout:5154: writing smithi10729407-8 from 1515520 to 1933312 tid 3 2020-06-12T02:38:20.795 INFO:tasks.rados.rados.0.smithi107.stdout:5154: writing smithi10729407-8 from 1933312 to 2318336 tid 4 2020-06-12T02:38:20.796 INFO:tasks.rados.rados.0.smithi107.stdout:5155: delete oid 6 current snap is 0 2020-06-12T02:38:20.797 INFO:tasks.rados.rados.0.smithi107.stdout:5154: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:20.802 INFO:tasks.rados.rados.0.smithi107.stdout:5155: done (2 left) 2020-06-12T02:38:20.803 INFO:tasks.rados.rados.0.smithi107.stdout:5156: delete oid 6 current snap is 0 2020-06-12T02:38:20.807 INFO:tasks.rados.rados.0.smithi107.stdout:5154: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:20.807 INFO:tasks.rados.rados.0.smithi107.stdout:5154: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:20.808 INFO:tasks.rados.rados.0.smithi107.stdout:5156: done (2 left) 2020-06-12T02:38:20.809 INFO:tasks.rados.rados.0.smithi107.stdout:5157: delete oid 1 current snap is 0 2020-06-12T02:38:20.809 INFO:tasks.rados.rados.0.smithi107.stdout:5154: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:20.810 INFO:tasks.rados.rados.0.smithi107.stdout:5154: finishing write tid 6 to smithi10729407-8 2020-06-12T02:38:20.811 INFO:tasks.rados.rados.0.smithi107.stdout:5154: finishing write tid 7 to smithi10729407-8 2020-06-12T02:38:20.811 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 903 (ObjNum 5153 snap 0 seq_num 5153) dirty exists 2020-06-12T02:38:20.811 INFO:tasks.rados.rados.0.smithi107.stdout:5154: left oid 8 (ObjNum 5153 snap 0 seq_num 5153) 2020-06-12T02:38:20.814 INFO:tasks.rados.rados.0.smithi107.stdout:5154: done (2 left) 2020-06-12T02:38:20.814 INFO:tasks.rados.rados.0.smithi107.stdout:5157: done (1 left) 2020-06-12T02:38:20.814 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:20.814 INFO:tasks.rados.rados.0.smithi107.stdout:5158: seq_num 5157 ranges {4857856=778240,5636096=786432,6422528=794624,7217152=24576} 2020-06-12T02:38:20.846 INFO:tasks.rados.rados.0.smithi107.stdout:5158: writing smithi10729407-17 from 4857856 to 5636096 tid 1 2020-06-12T02:38:20.851 INFO:tasks.rados.rados.0.smithi107.stdout:5158: writing smithi10729407-17 from 5636096 to 6422528 tid 2 2020-06-12T02:38:20.854 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:20.852+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:20.857 INFO:tasks.rados.rados.0.smithi107.stdout:5158: writing smithi10729407-17 from 6422528 to 7217152 tid 3 2020-06-12T02:38:20.857 INFO:tasks.rados.rados.0.smithi107.stdout:5158: writing smithi10729407-17 from 7217152 to 7241728 tid 4 2020-06-12T02:38:20.857 INFO:tasks.rados.rados.0.smithi107.stdout:5159: delete oid 18 current snap is 0 2020-06-12T02:38:20.857 INFO:tasks.rados.rados.0.smithi107.stdout:5158: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:20.866 INFO:tasks.rados.rados.0.smithi107.stdout:5159: done (2 left) 2020-06-12T02:38:20.867 INFO:tasks.rados.rados.0.smithi107.stdout:5160: delete oid 7 current snap is 0 2020-06-12T02:38:20.868 INFO:tasks.rados.rados.0.smithi107.stdout:5158: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:20.870 INFO:tasks.rados.rados.0.smithi107.stdout:5160: done (2 left) 2020-06-12T02:38:20.870 INFO:tasks.rados.rados.0.smithi107.stdout:5161: delete oid 15 current snap is 0 2020-06-12T02:38:20.870 INFO:tasks.rados.rados.0.smithi107.stdout:5158: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:20.872 INFO:tasks.rados.rados.0.smithi107.stdout:5158: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:20.873 INFO:tasks.rados.rados.0.smithi107.stdout:5158: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:20.874 INFO:tasks.rados.rados.0.smithi107.stdout:5158: finishing write tid 7 to smithi10729407-17 2020-06-12T02:38:20.875 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 963 (ObjNum 5157 snap 0 seq_num 5157) dirty exists 2020-06-12T02:38:20.875 INFO:tasks.rados.rados.0.smithi107.stdout:5158: left oid 17 (ObjNum 5157 snap 0 seq_num 5157) 2020-06-12T02:38:20.875 INFO:tasks.rados.rados.0.smithi107.stdout:5158: done (2 left) 2020-06-12T02:38:20.875 INFO:tasks.rados.rados.0.smithi107.stdout:5161: done (1 left) 2020-06-12T02:38:20.876 INFO:tasks.rados.rados.0.smithi107.stdout:5162: delete oid 1 current snap is 0 2020-06-12T02:38:20.878 INFO:tasks.rados.rados.0.smithi107.stdout:5162: done (1 left) 2020-06-12T02:38:20.878 INFO:tasks.rados.rados.0.smithi107.stdout:5163: delete oid 20 current snap is 0 2020-06-12T02:38:20.879 INFO:tasks.rados.rados.0.smithi107.stdout:5163: done (1 left) 2020-06-12T02:38:20.879 INFO:tasks.rados.rados.0.smithi107.stdout:5164: delete oid 8 current snap is 0 2020-06-12T02:38:20.882 INFO:tasks.rados.rados.0.smithi107.stdout:5164: done (1 left) 2020-06-12T02:38:20.882 INFO:tasks.rados.rados.0.smithi107.stdout:5165: delete oid 5 current snap is 0 2020-06-12T02:38:20.887 INFO:tasks.rados.rados.0.smithi107.stdout:5165: done (1 left) 2020-06-12T02:38:20.887 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:20.887 INFO:tasks.rados.rados.0.smithi107.stdout:5166: seq_num 5165 ranges {0=106496} 2020-06-12T02:38:20.888 INFO:tasks.rados.rados.0.smithi107.stdout:5166: writing smithi10729407-12 from 0 to 106496 tid 1 2020-06-12T02:38:20.888 INFO:tasks.rados.rados.0.smithi107.stdout:5167: delete oid 2 current snap is 0 2020-06-12T02:38:20.891 INFO:tasks.rados.rados.0.smithi107.stdout:5167: done (2 left) 2020-06-12T02:38:20.891 INFO:tasks.rados.rados.0.smithi107.stdout:5168: delete oid 20 current snap is 0 2020-06-12T02:38:20.894 INFO:tasks.rados.rados.0.smithi107.stdout:5166: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:20.894 INFO:tasks.rados.rados.0.smithi107.stdout:5166: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:20.894 INFO:tasks.rados.rados.0.smithi107.stdout:5168: done (2 left) 2020-06-12T02:38:20.895 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:20.895 INFO:tasks.rados.rados.0.smithi107.stdout:5169: seq_num 5168 ranges {0=155648} 2020-06-12T02:38:20.897 INFO:tasks.rados.rados.0.smithi107.stdout:5169: writing smithi10729407-18 from 0 to 155648 tid 1 2020-06-12T02:38:20.897 INFO:tasks.rados.rados.0.smithi107.stdout:5170: delete oid 3 current snap is 0 2020-06-12T02:38:20.897 INFO:tasks.rados.rados.0.smithi107.stdout:5166: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:20.901 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 908 (ObjNum 5165 snap 0 seq_num 5165) dirty exists 2020-06-12T02:38:20.901 INFO:tasks.rados.rados.0.smithi107.stdout:5166: left oid 12 (ObjNum 5165 snap 0 seq_num 5165) 2020-06-12T02:38:20.902 INFO:tasks.rados.rados.0.smithi107.stdout:5166: done (3 left) 2020-06-12T02:38:20.902 INFO:tasks.rados.rados.0.smithi107.stdout:5170: done (2 left) 2020-06-12T02:38:20.902 INFO:tasks.rados.rados.0.smithi107.stdout:5171: delete oid 5 current snap is 0 2020-06-12T02:38:20.902 INFO:tasks.rados.rados.0.smithi107.stdout:5171: done (2 left) 2020-06-12T02:38:20.903 INFO:tasks.rados.rados.0.smithi107.stdout:5172: delete oid 5 current snap is 0 2020-06-12T02:38:20.903 INFO:tasks.rados.rados.0.smithi107.stdout:5169: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:20.903 INFO:tasks.rados.rados.0.smithi107.stdout:5172: done (2 left) 2020-06-12T02:38:20.904 INFO:tasks.rados.rados.0.smithi107.stdout:5173: delete oid 2 current snap is 0 2020-06-12T02:38:20.904 INFO:tasks.rados.rados.0.smithi107.stdout:5169: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:20.905 INFO:tasks.rados.rados.0.smithi107.stdout:5173: done (2 left) 2020-06-12T02:38:20.905 INFO:tasks.rados.rados.0.smithi107.stdout:5174: delete oid 5 current snap is 0 2020-06-12T02:38:20.906 INFO:tasks.rados.rados.0.smithi107.stdout:5169: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:20.912 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 933 (ObjNum 5168 snap 0 seq_num 5168) dirty exists 2020-06-12T02:38:20.912 INFO:tasks.rados.rados.0.smithi107.stdout:5169: left oid 18 (ObjNum 5168 snap 0 seq_num 5168) 2020-06-12T02:38:20.913 INFO:tasks.rados.rados.0.smithi107.stdout:5169: done (2 left) 2020-06-12T02:38:20.913 INFO:tasks.rados.rados.0.smithi107.stdout:5174: done (1 left) 2020-06-12T02:38:20.914 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:20.914 INFO:tasks.rados.rados.0.smithi107.stdout:5175: seq_num 5174 ranges {106496=262144} 2020-06-12T02:38:20.914 INFO:tasks.rados.rados.0.smithi107.stdout:5175: writing smithi10729407-12 from 106496 to 368640 tid 1 2020-06-12T02:38:20.914 INFO:tasks.rados.rados.0.smithi107.stdout:5176: delete oid 5 current snap is 0 2020-06-12T02:38:20.914 INFO:tasks.rados.rados.0.smithi107.stdout:5176: done (2 left) 2020-06-12T02:38:20.915 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:20.915 INFO:tasks.rados.rados.0.smithi107.stdout:5177: seq_num 5176 ranges {0=294912} 2020-06-12T02:38:20.915 INFO:tasks.rados.rados.0.smithi107.stdout:5177: writing smithi10729407-16 from 0 to 294912 tid 1 2020-06-12T02:38:20.915 INFO:tasks.rados.rados.0.smithi107.stdout:5178: delete oid 4 current snap is 0 2020-06-12T02:38:20.916 INFO:tasks.rados.rados.0.smithi107.stdout:5175: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:20.916 INFO:tasks.rados.rados.0.smithi107.stdout:5175: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:20.917 INFO:tasks.rados.rados.0.smithi107.stdout:5175: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:20.918 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 910 (ObjNum 5174 snap 0 seq_num 5174) dirty exists 2020-06-12T02:38:20.918 INFO:tasks.rados.rados.0.smithi107.stdout:5175: left oid 12 (ObjNum 5174 snap 0 seq_num 5174) 2020-06-12T02:38:20.918 INFO:tasks.rados.rados.0.smithi107.stdout:5175: done (3 left) 2020-06-12T02:38:20.918 INFO:tasks.rados.rados.0.smithi107.stdout:5178: done (2 left) 2020-06-12T02:38:20.920 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:38:20.920 INFO:tasks.rados.rados.0.smithi107.stdout:5179: seq_num 5178 ranges {0=327680} 2020-06-12T02:38:20.920 INFO:tasks.rados.rados.0.smithi107.stdout:5179: writing smithi10729407-13 from 0 to 327680 tid 1 2020-06-12T02:38:20.920 INFO:tasks.rados.rados.0.smithi107.stdout:5180: delete oid 4 current snap is 0 2020-06-12T02:38:20.920 INFO:tasks.rados.rados.0.smithi107.stdout:5177: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:20.922 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:20.922 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:20.923 INFO:tasks.rados.rados.0.smithi107.stdout:5179: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:20.924 INFO:tasks.rados.rados.0.smithi107.stdout:5177: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:20.924 INFO:tasks.rados.rados.0.smithi107.stdout:5179: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:20.925 INFO:tasks.rados.rados.0.smithi107.stdout:5180: done (3 left) 2020-06-12T02:38:20.925 INFO:tasks.rados.rados.0.smithi107.stdout:5181: delete oid 6 current snap is 0 2020-06-12T02:38:20.926 INFO:tasks.rados.rados.0.smithi107.stdout:5179: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:20.927 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 987 (ObjNum 5178 snap 0 seq_num 5178) dirty exists 2020-06-12T02:38:20.927 INFO:tasks.rados.rados.0.smithi107.stdout:5179: left oid 13 (ObjNum 5178 snap 0 seq_num 5178) 2020-06-12T02:38:20.927 INFO:tasks.rados.rados.0.smithi107.stdout:5177: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:20.928 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 926 (ObjNum 5176 snap 0 seq_num 5176) dirty exists 2020-06-12T02:38:20.928 INFO:tasks.rados.rados.0.smithi107.stdout:5177: left oid 16 (ObjNum 5176 snap 0 seq_num 5176) 2020-06-12T02:38:20.930 INFO:tasks.rados.rados.0.smithi107.stdout:5177: done (3 left) 2020-06-12T02:38:20.930 INFO:tasks.rados.rados.0.smithi107.stdout:5179: done (2 left) 2020-06-12T02:38:20.930 INFO:tasks.rados.rados.0.smithi107.stdout:5181: done (1 left) 2020-06-12T02:38:20.931 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:20.931 INFO:tasks.rados.rados.0.smithi107.stdout:5182: seq_num 5181 ranges {0=376832} 2020-06-12T02:38:20.932 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-06-12T02:38:20.933 INFO:tasks.rados.rados.0.smithi107.stdout:5182: writing smithi10729407-15 from 0 to 376832 tid 1 2020-06-12T02:38:20.933 INFO:tasks.rados.rados.0.smithi107.stdout:5183: delete oid 14 current snap is 0 2020-06-12T02:38:20.935 INFO:tasks.rados.rados.0.smithi107.stdout:5183: done (2 left) 2020-06-12T02:38:20.935 INFO:tasks.rados.rados.0.smithi107.stdout:5184: delete oid 16 current snap is 0 2020-06-12T02:38:20.949 INFO:tasks.rados.rados.0.smithi107.stdout:5184: done (2 left) 2020-06-12T02:38:20.949 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:38:20.949 INFO:tasks.rados.rados.0.smithi107.stdout:5185: seq_num 5184 ranges {0=425984} 2020-06-12T02:38:20.950 INFO:tasks.rados.rados.0.smithi107.stdout:5185: writing smithi10729407-20 from 0 to 425984 tid 1 2020-06-12T02:38:20.950 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:20.950 INFO:tasks.rados.rados.0.smithi107.stdout:5182: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:20.950 INFO:tasks.rados.rados.0.smithi107.stdout:5186: seq_num 5185 ranges {155648=442368} 2020-06-12T02:38:20.951 INFO:tasks.rados.rados.0.smithi107.stdout:5186: writing smithi10729407-18 from 155648 to 598016 tid 1 2020-06-12T02:38:20.951 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:20.951 INFO:tasks.rados.rados.0.smithi107.stdout:5187: seq_num 5186 ranges {0=458752} 2020-06-12T02:38:20.951 INFO:tasks.rados.rados.0.smithi107.stdout:5187: writing smithi10729407-14 from 0 to 458752 tid 1 2020-06-12T02:38:20.951 INFO:tasks.rados.rados.0.smithi107.stdout:5188: delete oid 2 current snap is 0 2020-06-12T02:38:20.951 INFO:tasks.rados.rados.0.smithi107.stdout:5182: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:20.952 INFO:tasks.rados.rados.0.smithi107.stdout:5182: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:20.952 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 907 (ObjNum 5181 snap 0 seq_num 5181) dirty exists 2020-06-12T02:38:20.952 INFO:tasks.rados.rados.0.smithi107.stdout:5182: left oid 15 (ObjNum 5181 snap 0 seq_num 5181) 2020-06-12T02:38:20.952 INFO:tasks.rados.rados.0.smithi107.stdout:5185: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:20.954 INFO:tasks.rados.rados.0.smithi107.stdout:5186: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:20.954 INFO:tasks.rados.rados.0.smithi107.stdout:5182: done (5 left) 2020-06-12T02:38:20.954 INFO:tasks.rados.rados.0.smithi107.stdout:5188: done (4 left) 2020-06-12T02:38:20.954 INFO:tasks.rados.rados.0.smithi107.stdout:5189: delete oid 2 current snap is 0 2020-06-12T02:38:20.955 INFO:tasks.rados.rados.0.smithi107.stdout:5185: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:20.956 INFO:tasks.rados.rados.0.smithi107.stdout:5186: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:20.957 INFO:tasks.rados.rados.0.smithi107.stdout:5187: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:20.958 INFO:tasks.rados.rados.0.smithi107.stdout:5187: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:20.958 INFO:tasks.rados.rados.0.smithi107.stdout:5185: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:20.959 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 391 (ObjNum 5184 snap 0 seq_num 5184) dirty exists 2020-06-12T02:38:20.960 INFO:tasks.rados.rados.0.smithi107.stdout:5185: left oid 20 (ObjNum 5184 snap 0 seq_num 5184) 2020-06-12T02:38:20.960 INFO:tasks.rados.rados.0.smithi107.stdout:5186: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:20.960 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 935 (ObjNum 5185 snap 0 seq_num 5185) dirty exists 2020-06-12T02:38:20.960 INFO:tasks.rados.rados.0.smithi107.stdout:5186: left oid 18 (ObjNum 5185 snap 0 seq_num 5185) 2020-06-12T02:38:20.961 INFO:tasks.rados.rados.0.smithi107.stdout:5187: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:20.961 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 938 (ObjNum 5186 snap 0 seq_num 5186) dirty exists 2020-06-12T02:38:20.961 INFO:tasks.rados.rados.0.smithi107.stdout:5187: left oid 14 (ObjNum 5186 snap 0 seq_num 5186) 2020-06-12T02:38:20.962 INFO:tasks.rados.rados.0.smithi107.stdout:5185: done (4 left) 2020-06-12T02:38:20.962 INFO:tasks.rados.rados.0.smithi107.stdout:5186: done (3 left) 2020-06-12T02:38:20.962 INFO:tasks.rados.rados.0.smithi107.stdout:5187: done (2 left) 2020-06-12T02:38:20.963 INFO:tasks.rados.rados.0.smithi107.stdout:5189: done (1 left) 2020-06-12T02:38:20.963 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:20.963 INFO:tasks.rados.rados.0.smithi107.stdout:5190: seq_num 5189 ranges {0=507904} 2020-06-12T02:38:20.964 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:20.954+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:20.965 INFO:tasks.rados.rados.0.smithi107.stdout:5190: writing smithi10729407-2 from 0 to 507904 tid 1 2020-06-12T02:38:20.966 INFO:tasks.rados.rados.0.smithi107.stdout:5191: delete oid 3 current snap is 0 2020-06-12T02:38:20.967 INFO:tasks.rados.rados.0.smithi107.stdout:5191: done (2 left) 2020-06-12T02:38:20.967 INFO:tasks.rados.rados.0.smithi107.stdout:5192: delete oid 12 current snap is 0 2020-06-12T02:38:20.967 INFO:tasks.rados.rados.0.smithi107.stdout:5190: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:20.968 INFO:tasks.rados.rados.0.smithi107.stdout:5192: done (2 left) 2020-06-12T02:38:20.968 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:20.968 INFO:tasks.rados.rados.0.smithi107.stdout:5193: seq_num 5192 ranges {0=565248} 2020-06-12T02:38:20.969 INFO:tasks.rados.rados.0.smithi107.stdout:5193: writing smithi10729407-12 from 0 to 565248 tid 1 2020-06-12T02:38:20.970 INFO:tasks.rados.rados.0.smithi107.stdout:5194: delete oid 16 current snap is 0 2020-06-12T02:38:20.970 INFO:tasks.rados.rados.0.smithi107.stdout:5190: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:20.970 INFO:tasks.rados.rados.0.smithi107.stdout:5190: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:20.971 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 496 (ObjNum 5189 snap 0 seq_num 5189) dirty exists 2020-06-12T02:38:20.971 INFO:tasks.rados.rados.0.smithi107.stdout:5190: left oid 2 (ObjNum 5189 snap 0 seq_num 5189) 2020-06-12T02:38:20.974 INFO:tasks.rados.rados.0.smithi107.stdout:5190: done (3 left) 2020-06-12T02:38:20.974 INFO:tasks.rados.rados.0.smithi107.stdout:5194: done (2 left) 2020-06-12T02:38:20.974 INFO:tasks.rados.rados.0.smithi107.stdout:5195: delete oid 2 current snap is 0 2020-06-12T02:38:20.975 INFO:tasks.rados.rados.0.smithi107.stdout:5193: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:20.976 INFO:tasks.rados.rados.0.smithi107.stdout:5195: done (2 left) 2020-06-12T02:38:20.976 INFO:tasks.rados.rados.0.smithi107.stdout:5196: delete oid 11 current snap is 0 2020-06-12T02:38:20.976 INFO:tasks.rados.rados.0.smithi107.stdout:5193: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:20.977 INFO:tasks.rados.rados.0.smithi107.stdout:5193: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:20.977 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 913 (ObjNum 5192 snap 0 seq_num 5192) dirty exists 2020-06-12T02:38:20.978 INFO:tasks.rados.rados.0.smithi107.stdout:5193: left oid 12 (ObjNum 5192 snap 0 seq_num 5192) 2020-06-12T02:38:20.978 INFO:tasks.rados.rados.0.smithi107.stdout:5193: done (2 left) 2020-06-12T02:38:20.978 INFO:tasks.rados.rados.0.smithi107.stdout:5196: done (1 left) 2020-06-12T02:38:20.979 INFO:tasks.rados.rados.0.smithi107.stdout:5197: delete oid 17 current snap is 0 2020-06-12T02:38:20.980 INFO:tasks.rados.rados.0.smithi107.stdout:5197: done (1 left) 2020-06-12T02:38:20.981 INFO:tasks.rados.rados.0.smithi107.stdout:5198: delete oid 9 current snap is 0 2020-06-12T02:38:20.982 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:38:20.986 INFO:tasks.rados.rados.0.smithi107.stdout:5198: done (1 left) 2020-06-12T02:38:20.986 INFO:tasks.rados.rados.0.smithi107.stdout:5199: delete oid 8 current snap is 0 2020-06-12T02:38:20.988 INFO:tasks.rados.rados.0.smithi107.stdout:5199: done (1 left) 2020-06-12T02:38:20.989 INFO:tasks.rados.rados.0.smithi107.stdout:5200: delete oid 3 current snap is 0 2020-06-12T02:38:20.992 INFO:tasks.rados.rados.0.smithi107.stdout:5200: done (1 left) 2020-06-12T02:38:20.992 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:38:20.993 INFO:tasks.rados.rados.0.smithi107.stdout:5201: seq_num 5200 ranges {0=696320} 2020-06-12T02:38:20.996 INFO:tasks.rados.rados.0.smithi107.stdout:5201: writing smithi10729407-16 from 0 to 696320 tid 1 2020-06-12T02:38:20.996 INFO:tasks.rados.rados.0.smithi107.stdout:5202: delete oid 14 current snap is 0 2020-06-12T02:38:20.999 INFO:tasks.rados.rados.0.smithi107.stdout:5202: done (2 left) 2020-06-12T02:38:20.999 INFO:tasks.rados.rados.0.smithi107.stdout:5203: delete oid 14 current snap is 0 2020-06-12T02:38:21.002 INFO:tasks.rados.rados.0.smithi107.stdout:5203: done (2 left) 2020-06-12T02:38:21.003 INFO:tasks.rados.rados.0.smithi107.stdout:5204: delete oid 4 current snap is 0 2020-06-12T02:38:21.003 INFO:tasks.rados.rados.0.smithi107.stdout:5201: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:21.004 INFO:tasks.rados.rados.0.smithi107.stdout:5201: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:21.007 INFO:tasks.rados.rados.0.smithi107.stdout:5204: done (2 left) 2020-06-12T02:38:21.007 INFO:tasks.rados.rados.0.smithi107.stdout:5205: delete oid 14 current snap is 0 2020-06-12T02:38:21.007 INFO:tasks.rados.rados.0.smithi107.stdout:5201: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:21.007 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 932 (ObjNum 5200 snap 0 seq_num 5200) dirty exists 2020-06-12T02:38:21.007 INFO:tasks.rados.rados.0.smithi107.stdout:5201: left oid 16 (ObjNum 5200 snap 0 seq_num 5200) 2020-06-12T02:38:21.009 INFO:tasks.rados.rados.0.smithi107.stdout:5201: done (2 left) 2020-06-12T02:38:21.009 INFO:tasks.rados.rados.0.smithi107.stdout:5205: done (1 left) 2020-06-12T02:38:21.010 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:38:21.010 INFO:tasks.rados.rados.0.smithi107.stdout:5206: seq_num 5205 ranges {425984=778240} 2020-06-12T02:38:21.014 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:38:21.016 INFO:tasks.rados.rados.0.smithi107.stdout:5206: writing smithi10729407-20 from 425984 to 1204224 tid 1 2020-06-12T02:38:21.016 INFO:tasks.rados.rados.0.smithi107.stdout:5207: delete oid 5 current snap is 0 2020-06-12T02:38:21.018 INFO:tasks.rados.rados.0.smithi107.stdout:5207: done (2 left) 2020-06-12T02:38:21.019 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:21.019 INFO:tasks.rados.rados.0.smithi107.stdout:5208: seq_num 5207 ranges {376832=409600,786432=401408} 2020-06-12T02:38:21.023 INFO:tasks.rados.rados.0.smithi107.stdout:5208: writing smithi10729407-15 from 376832 to 786432 tid 1 2020-06-12T02:38:21.025 INFO:tasks.rados.rados.0.smithi107.stdout:5208: writing smithi10729407-15 from 786432 to 1187840 tid 2 2020-06-12T02:38:21.025 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:21.026 INFO:tasks.rados.rados.0.smithi107.stdout:5209: seq_num 5208 ranges {598016=425984,1024000=401408} 2020-06-12T02:38:21.031 INFO:tasks.rados.rados.0.smithi107.stdout:5209: writing smithi10729407-18 from 598016 to 1024000 tid 1 2020-06-12T02:38:21.033 INFO:tasks.rados.rados.0.smithi107.stdout:5209: writing smithi10729407-18 from 1024000 to 1425408 tid 2 2020-06-12T02:38:21.034 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:38:21.034 INFO:tasks.rados.rados.0.smithi107.stdout:5210: seq_num 5209 ranges {0=442368,442368=401408} 2020-06-12T02:38:21.035 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:38:21.037 INFO:tasks.rados.rados.0.smithi107.stdout:5210: writing smithi10729407-11 from 0 to 442368 tid 1 2020-06-12T02:38:21.039 INFO:tasks.rados.rados.0.smithi107.stdout:5210: writing smithi10729407-11 from 442368 to 843776 tid 2 2020-06-12T02:38:21.039 INFO:tasks.rados.rados.0.smithi107.stdout:5211: delete oid 12 current snap is 0 2020-06-12T02:38:21.040 INFO:tasks.rados.rados.0.smithi107.stdout:5206: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:21.040 INFO:tasks.rados.rados.0.smithi107.stdout:5208: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:21.040 INFO:tasks.rados.rados.0.smithi107.stdout:5206: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:21.040 INFO:tasks.rados.rados.0.smithi107.stdout:5208: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:21.041 INFO:tasks.rados.rados.0.smithi107.stdout:5206: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:21.041 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 393 (ObjNum 5205 snap 0 seq_num 5205) dirty exists 2020-06-12T02:38:21.041 INFO:tasks.rados.rados.0.smithi107.stdout:5206: left oid 20 (ObjNum 5205 snap 0 seq_num 5205) 2020-06-12T02:38:21.041 INFO:tasks.rados.rados.0.smithi107.stdout:5208: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:21.041 INFO:tasks.rados.rados.0.smithi107.stdout:5208: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:21.042 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 911 (ObjNum 5207 snap 0 seq_num 5207) dirty exists 2020-06-12T02:38:21.042 INFO:tasks.rados.rados.0.smithi107.stdout:5208: left oid 15 (ObjNum 5207 snap 0 seq_num 5207) 2020-06-12T02:38:21.045 INFO:tasks.rados.rados.0.smithi107.stdout:5209: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:21.046 INFO:tasks.rados.rados.0.smithi107.stdout:5206: done (5 left) 2020-06-12T02:38:21.046 INFO:tasks.rados.rados.0.smithi107.stdout:5208: done (4 left) 2020-06-12T02:38:21.046 INFO:tasks.rados.rados.0.smithi107.stdout:5211: done (3 left) 2020-06-12T02:38:21.047 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:21.047 INFO:tasks.rados.rados.0.smithi107.stdout:5212: seq_num 5211 ranges {1187840=483328,1671168=401408} 2020-06-12T02:38:21.054 INFO:tasks.rados.rados.0.smithi107.stdout:5212: writing smithi10729407-15 from 1187840 to 1671168 tid 1 2020-06-12T02:38:21.059 INFO:tasks.rados.rados.0.smithi107.stdout:5212: writing smithi10729407-15 from 1671168 to 2072576 tid 2 2020-06-12T02:38:21.059 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:38:21.059 INFO:tasks.rados.rados.0.smithi107.stdout:5213: seq_num 5212 ranges {0=499712,499712=401408} 2020-06-12T02:38:21.059 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:21.056+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:21.060 INFO:tasks.rados.rados.0.smithi107.stdout:5213: writing smithi10729407-2 from 0 to 499712 tid 1 2020-06-12T02:38:21.063 INFO:tasks.rados.rados.0.smithi107.stdout:5213: writing smithi10729407-2 from 499712 to 901120 tid 2 2020-06-12T02:38:21.064 INFO:tasks.rados.rados.0.smithi107.stdout:5214: delete oid 4 current snap is 0 2020-06-12T02:38:21.064 INFO:tasks.rados.rados.0.smithi107.stdout:5210: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:21.064 INFO:tasks.rados.rados.0.smithi107.stdout:5209: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:21.064 INFO:tasks.rados.rados.0.smithi107.stdout:5209: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:21.065 INFO:tasks.rados.rados.0.smithi107.stdout:5209: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:21.065 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 939 (ObjNum 5208 snap 0 seq_num 5208) dirty exists 2020-06-12T02:38:21.065 INFO:tasks.rados.rados.0.smithi107.stdout:5209: left oid 18 (ObjNum 5208 snap 0 seq_num 5208) 2020-06-12T02:38:21.065 INFO:tasks.rados.rados.0.smithi107.stdout:5210: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:21.065 INFO:tasks.rados.rados.0.smithi107.stdout:5210: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:21.066 INFO:tasks.rados.rados.0.smithi107.stdout:5210: finishing write tid 5 to smithi10729407-11 2020-06-12T02:38:21.066 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 945 (ObjNum 5209 snap 0 seq_num 5209) dirty exists 2020-06-12T02:38:21.066 INFO:tasks.rados.rados.0.smithi107.stdout:5210: left oid 11 (ObjNum 5209 snap 0 seq_num 5209) 2020-06-12T02:38:21.066 INFO:tasks.rados.rados.0.smithi107.stdout:5212: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:21.067 INFO:tasks.rados.rados.0.smithi107.stdout:5212: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:21.067 INFO:tasks.rados.rados.0.smithi107.stdout:5212: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:21.068 INFO:tasks.rados.rados.0.smithi107.stdout:5209: done (5 left) 2020-06-12T02:38:21.068 INFO:tasks.rados.rados.0.smithi107.stdout:5210: done (4 left) 2020-06-12T02:38:21.068 INFO:tasks.rados.rados.0.smithi107.stdout:5214: done (3 left) 2020-06-12T02:38:21.068 INFO:tasks.rados.rados.0.smithi107.stdout:5215: delete oid 11 current snap is 0 2020-06-12T02:38:21.068 INFO:tasks.rados.rados.0.smithi107.stdout:5213: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:21.069 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:38:21.069 INFO:tasks.rados.rados.0.smithi107.stdout:5212: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:21.069 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 914 (ObjNum 5211 snap 0 seq_num 5211) dirty exists 2020-06-12T02:38:21.070 INFO:tasks.rados.rados.0.smithi107.stdout:5212: left oid 15 (ObjNum 5211 snap 0 seq_num 5211) 2020-06-12T02:38:21.070 INFO:tasks.rados.rados.0.smithi107.stdout:5212: done (3 left) 2020-06-12T02:38:21.070 INFO:tasks.rados.rados.0.smithi107.stdout:5215: done (2 left) 2020-06-12T02:38:21.070 INFO:tasks.rados.rados.0.smithi107.stdout:5216: delete oid 6 current snap is 0 2020-06-12T02:38:21.072 INFO:tasks.rados.rados.0.smithi107.stdout:5213: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:21.073 INFO:tasks.rados.rados.0.smithi107.stdout:5216: done (2 left) 2020-06-12T02:38:21.074 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:21.074 INFO:tasks.rados.rados.0.smithi107.stdout:5217: seq_num 5216 ranges {327680=565248,892928=401408} 2020-06-12T02:38:21.079 INFO:tasks.rados.rados.0.smithi107.stdout:5217: writing smithi10729407-13 from 327680 to 892928 tid 1 2020-06-12T02:38:21.083 INFO:tasks.rados.rados.0.smithi107.stdout:5217: writing smithi10729407-13 from 892928 to 1294336 tid 2 2020-06-12T02:38:21.083 INFO:tasks.rados.rados.0.smithi107.stdout:5218: delete oid 3 current snap is 0 2020-06-12T02:38:21.083 INFO:tasks.rados.rados.0.smithi107.stdout:5213: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:21.083 INFO:tasks.rados.rados.0.smithi107.stdout:5213: finishing write tid 5 to smithi10729407-2 2020-06-12T02:38:21.083 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 500 (ObjNum 5212 snap 0 seq_num 5212) dirty exists 2020-06-12T02:38:21.084 INFO:tasks.rados.rados.0.smithi107.stdout:5213: left oid 2 (ObjNum 5212 snap 0 seq_num 5212) 2020-06-12T02:38:21.085 INFO:tasks.rados.rados.0.smithi107.stdout:5213: done (3 left) 2020-06-12T02:38:21.086 INFO:tasks.rados.rados.0.smithi107.stdout:5218: done (2 left) 2020-06-12T02:38:21.086 INFO:tasks.rados.rados.0.smithi107.stdout:5219: delete oid 1 current snap is 0 2020-06-12T02:38:21.091 INFO:tasks.rados.rados.0.smithi107.stdout:5217: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:21.093 INFO:tasks.rados.rados.0.smithi107.stdout:5219: done (2 left) 2020-06-12T02:38:21.093 INFO:tasks.rados.rados.0.smithi107.stdout:5220: delete oid 16 current snap is 0 2020-06-12T02:38:21.094 INFO:tasks.rados.rados.0.smithi107.stdout:5220: done (2 left) 2020-06-12T02:38:21.094 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:21.095 INFO:tasks.rados.rados.0.smithi107.stdout:5221: seq_num 5220 ranges {1425408=630784,2056192=401408} 2020-06-12T02:38:21.095 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:38:21.104 INFO:tasks.rados.rados.0.smithi107.stdout:5221: writing smithi10729407-18 from 1425408 to 2056192 tid 1 2020-06-12T02:38:21.107 INFO:tasks.rados.rados.0.smithi107.stdout:5221: writing smithi10729407-18 from 2056192 to 2457600 tid 2 2020-06-12T02:38:21.108 INFO:tasks.rados.rados.0.smithi107.stdout:5222: delete oid 1 current snap is 0 2020-06-12T02:38:21.108 INFO:tasks.rados.rados.0.smithi107.stdout:5217: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:21.108 INFO:tasks.rados.rados.0.smithi107.stdout:5217: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:21.108 INFO:tasks.rados.rados.0.smithi107.stdout:5217: finishing write tid 5 to smithi10729407-13 2020-06-12T02:38:21.109 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 990 (ObjNum 5216 snap 0 seq_num 5216) dirty exists 2020-06-12T02:38:21.109 INFO:tasks.rados.rados.0.smithi107.stdout:5217: left oid 13 (ObjNum 5216 snap 0 seq_num 5216) 2020-06-12T02:38:21.109 INFO:tasks.rados.rados.0.smithi107.stdout:5217: done (3 left) 2020-06-12T02:38:21.110 INFO:tasks.rados.rados.0.smithi107.stdout:5222: done (2 left) 2020-06-12T02:38:21.110 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:21.110 INFO:tasks.rados.rados.0.smithi107.stdout:5223: seq_num 5222 ranges {0=663552,663552=401408} 2020-06-12T02:38:21.113 INFO:tasks.rados.rados.0.smithi107.stdout:5223: writing smithi10729407-12 from 0 to 663552 tid 1 2020-06-12T02:38:21.116 INFO:tasks.rados.rados.0.smithi107.stdout:5223: writing smithi10729407-12 from 663552 to 1064960 tid 2 2020-06-12T02:38:21.116 INFO:tasks.rados.rados.0.smithi107.stdout:5224: delete oid 5 current snap is 0 2020-06-12T02:38:21.116 INFO:tasks.rados.rados.0.smithi107.stdout:5221: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:21.118 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2020-06-12T02:38:21.119 INFO:tasks.rados.rados.0.smithi107.stdout:5224: done (3 left) 2020-06-12T02:38:21.119 INFO:tasks.rados.rados.0.smithi107.stdout:5225: delete oid 6 current snap is 0 2020-06-12T02:38:21.120 INFO:tasks.rados.rados.0.smithi107.stdout:5221: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:21.120 INFO:tasks.rados.rados.0.smithi107.stdout:5221: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:21.121 INFO:tasks.rados.rados.0.smithi107.stdout:5221: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:21.121 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 942 (ObjNum 5220 snap 0 seq_num 5220) dirty exists 2020-06-12T02:38:21.121 INFO:tasks.rados.rados.0.smithi107.stdout:5221: left oid 18 (ObjNum 5220 snap 0 seq_num 5220) 2020-06-12T02:38:21.122 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-06-12T02:38:21.123 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-06-12T02:38:21.123 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-06-12T02:38:21.123 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-06-12T02:38:21.125 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-06-12T02:38:21.125 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-06-12T02:38:21.125 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-06-12T02:38:21.125 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-06-12T02:38:21.126 INFO:tasks.rados.rados.0.smithi107.stdout:5223: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:21.126 INFO:tasks.rados.rados.0.smithi107.stdout:5221: done (3 left) 2020-06-12T02:38:21.126 INFO:tasks.rados.rados.0.smithi107.stdout:5225: done (2 left) 2020-06-12T02:38:21.127 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:21.127 INFO:tasks.rados.rados.0.smithi107.stdout:5226: seq_num 5225 ranges {0=712704,712704=401408} 2020-06-12T02:38:21.130 INFO:tasks.rados.rados.0.smithi107.stdout:5226: writing smithi10729407-9 from 0 to 712704 tid 1 2020-06-12T02:38:21.130 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:21.130 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2020-06-12T02:38:21.131 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2020-06-12T02:38:21.131 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2020-06-12T02:38:21.131 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.131 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s1 26/21 ECSubWrite(tid=408, reqid=client.4512.0:1572, at_version=26'99, trim_to=0'0, roll_forward_to=26'97))", 2020-06-12T02:38:21.131 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.887944+0000", 2020-06-12T02:38:21.132 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 47.237268585000002, 2020-06-12T02:38:21.132 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019136641999999999, 2020-06-12T02:38:21.132 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:21.132 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:21.132 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:21.133 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.133 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:21.133 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.887944+0000", 2020-06-12T02:38:21.133 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:21.133 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.134 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.134 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:21.134 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.887944+0000", 2020-06-12T02:38:21.134 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00063708500000000002 2020-06-12T02:38:21.134 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.134 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.135 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:21.135 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888581+0000", 2020-06-12T02:38:21.135 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9644e-05 2020-06-12T02:38:21.135 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.135 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.136 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:21.136 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888600+0000", 2020-06-12T02:38:21.136 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.721e-05 2020-06-12T02:38:21.136 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.136 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.137 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:21.137 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888618+0000", 2020-06-12T02:38:21.137 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7500000000000003e-06 2020-06-12T02:38:21.137 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.137 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.138 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:21.138 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888625+0000", 2020-06-12T02:38:21.138 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2707999999999998e-05 2020-06-12T02:38:21.138 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.139 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.139 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:21.139 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888668+0000", 2020-06-12T02:38:21.139 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0491e-05 2020-06-12T02:38:21.139 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.140 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.140 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:21.140 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888679+0000", 2020-06-12T02:38:21.140 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018351985000000001 2020-06-12T02:38:21.140 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.140 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.141 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:21.141 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.907031+0000", 2020-06-12T02:38:21.141 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9768999999999997e-05 2020-06-12T02:38:21.141 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.141 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.142 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:21.142 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.907080+0000", 2020-06-12T02:38:21.142 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019136641999999999 2020-06-12T02:38:21.142 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:21.142 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:21.143 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:21.143 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.143 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.143 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:4487 3.7s0 3:e653d00f:::smithi10729407-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-06-12T02:38:21.143 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:42.539305+0000", 2020-06-12T02:38:21.143 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 38.585907337999998, 2020-06-12T02:38:21.144 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019352590999999999, 2020-06-12T02:38:21.144 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:21.144 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:21.144 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:21.144 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:21.145 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:21.145 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 4487 2020-06-12T02:38:21.145 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.145 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:21.145 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.146 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:21.146 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.539305+0000", 2020-06-12T02:38:21.146 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:21.147 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.147 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.147 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:21.147 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.539305+0000", 2020-06-12T02:38:21.147 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7207999999999999e-05 2020-06-12T02:38:21.148 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.148 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.149 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:21.150 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.539352+0000", 2020-06-12T02:38:21.150 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6440000000000003e-06 2020-06-12T02:38:21.150 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.150 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.151 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:21.151 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.539359+0000", 2020-06-12T02:38:21.151 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0995999999999999e-05 2020-06-12T02:38:21.151 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.152 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.152 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:21.153 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.539370+0000", 2020-06-12T02:38:21.153 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4049999999999999e-06 2020-06-12T02:38:21.154 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.154 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.154 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:21.154 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.539375+0000", 2020-06-12T02:38:21.155 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013353090999999999 2020-06-12T02:38:21.155 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.156 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.156 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:21.157 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.552728+0000", 2020-06-12T02:38:21.157 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000113626 2020-06-12T02:38:21.157 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.158 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.158 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:21.158 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.552842+0000", 2020-06-12T02:38:21.158 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0039644459999999999 2020-06-12T02:38:21.158 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.159 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.159 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:21.159 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.556806+0000", 2020-06-12T02:38:21.159 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5099000000000005e-05 2020-06-12T02:38:21.159 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.159 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.160 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:21.160 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.556902+0000", 2020-06-12T02:38:21.160 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017560760000000001 2020-06-12T02:38:21.160 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.160 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.161 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:21.161 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.558658+0000", 2020-06-12T02:38:21.161 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019352590999999999 2020-06-12T02:38:21.161 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:21.161 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:21.162 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:21.162 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.162 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.162 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8416 3.ds2 3:b3445631:::smithi10729407-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e33)", 2020-06-12T02:38:21.162 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:53.425646+0000", 2020-06-12T02:38:21.162 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 27.699566318999999, 2020-06-12T02:38:21.163 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019204078999999999, 2020-06-12T02:38:21.163 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:21.163 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:21.163 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:21.163 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:21.164 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:21.164 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8416 2020-06-12T02:38:21.164 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.164 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:21.165 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.165 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:21.165 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:53.425646+0000", 2020-06-12T02:38:21.165 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:21.166 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.166 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.166 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:21.166 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:53.425646+0000", 2020-06-12T02:38:21.167 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2311e-05 2020-06-12T02:38:21.171 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.171 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.171 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:21.171 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:53.425668+0000", 2020-06-12T02:38:21.172 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0699999999999998e-06 2020-06-12T02:38:21.172 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.172 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.172 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:21.172 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:53.425671+0000", 2020-06-12T02:38:21.173 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8499999999999996e-06 2020-06-12T02:38:21.173 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.173 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.173 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:21.174 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:53.425678+0000", 2020-06-12T02:38:21.174 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2800000000000002e-06 2020-06-12T02:38:21.174 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.175 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.175 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:21.175 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:53.425681+0000", 2020-06-12T02:38:21.175 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0055371559999999997 2020-06-12T02:38:21.175 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.175 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.176 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:21.176 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:53.431218+0000", 2020-06-12T02:38:21.176 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9020000000000001e-05 2020-06-12T02:38:21.176 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.176 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.177 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:21.177 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:53.431277+0000", 2020-06-12T02:38:21.177 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0049408350000000002 2020-06-12T02:38:21.177 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.177 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.178 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:21.178 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:53.436218+0000", 2020-06-12T02:38:21.178 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1631000000000004e-05 2020-06-12T02:38:21.178 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.178 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.179 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:21.179 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:53.436309+0000", 2020-06-12T02:38:21.179 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085409260000000008 2020-06-12T02:38:21.179 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.179 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.179 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:21.180 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:53.444850+0000", 2020-06-12T02:38:21.180 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019204078999999999 2020-06-12T02:38:21.180 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:21.180 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:21.180 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:21.182 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.182 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.182 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8802 3.bs0 3:d0ebae7d:::smithi10729407-3:head [append 0~573440] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:21.183 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.691981+0000", 2020-06-12T02:38:21.183 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 9.4332315290000004, 2020-06-12T02:38:21.183 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.024741973, 2020-06-12T02:38:21.183 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:21.183 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:21.184 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:21.184 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:21.184 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:21.184 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8802 2020-06-12T02:38:21.184 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.185 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:21.185 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.185 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:21.185 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.691981+0000", 2020-06-12T02:38:21.185 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:21.186 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.186 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.186 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:21.186 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.691981+0000", 2020-06-12T02:38:21.186 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000369312 2020-06-12T02:38:21.186 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.187 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.187 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:21.187 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.692350+0000", 2020-06-12T02:38:21.187 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9639999999999998e-06 2020-06-12T02:38:21.188 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.188 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.188 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:21.188 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.692360+0000", 2020-06-12T02:38:21.188 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9335999999999999e-05 2020-06-12T02:38:21.189 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.189 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.189 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:21.189 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.692379+0000", 2020-06-12T02:38:21.189 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8229999999999995e-06 2020-06-12T02:38:21.189 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.190 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.190 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:21.190 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.692387+0000", 2020-06-12T02:38:21.190 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0760999999999997e-05 2020-06-12T02:38:21.191 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.191 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.191 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:21.191 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.692438+0000", 2020-06-12T02:38:21.192 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000113064 2020-06-12T02:38:21.192 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.192 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.192 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:21.192 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.692551+0000", 2020-06-12T02:38:21.193 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00080948800000000003 2020-06-12T02:38:21.193 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.193 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.193 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:21.193 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.693361+0000", 2020-06-12T02:38:21.194 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.023159432000000001 2020-06-12T02:38:21.194 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.194 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.194 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:21.194 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.716520+0000", 2020-06-12T02:38:21.195 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000115021 2020-06-12T02:38:21.195 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.195 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.195 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:21.195 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.716635+0000", 2020-06-12T02:38:21.196 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7771999999999999e-05 2020-06-12T02:38:21.196 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.196 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.196 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:21.196 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.716723+0000", 2020-06-12T02:38:21.197 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.024741973 2020-06-12T02:38:21.197 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:21.197 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:21.197 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:21.197 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.198 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.198 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8803 3.bs0 3:d0ebae7d:::smithi10729407-3:head [append 0~565248] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:21.198 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.695906+0000", 2020-06-12T02:38:21.198 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 9.429306338, 2020-06-12T02:38:21.199 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020991011, 2020-06-12T02:38:21.199 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:21.199 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:21.199 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:21.199 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:21.200 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:21.200 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8803 2020-06-12T02:38:21.200 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.200 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:21.200 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.201 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:21.201 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.695906+0000", 2020-06-12T02:38:21.201 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:21.201 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.201 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.202 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:21.202 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.695906+0000", 2020-06-12T02:38:21.202 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036262499999999999 2020-06-12T02:38:21.202 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.202 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.203 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:21.203 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.696269+0000", 2020-06-12T02:38:21.203 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0131e-05 2020-06-12T02:38:21.203 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.203 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.204 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:21.204 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.696279+0000", 2020-06-12T02:38:21.204 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6390000000000001e-05 2020-06-12T02:38:21.204 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.204 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.204 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:21.205 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.696295+0000", 2020-06-12T02:38:21.205 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0600000000000002e-06 2020-06-12T02:38:21.205 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.205 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.206 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:21.206 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.696302+0000", 2020-06-12T02:38:21.206 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0016389709999999999 2020-06-12T02:38:21.206 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.206 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.207 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:21.207 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.697941+0000", 2020-06-12T02:38:21.207 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011928400000000001 2020-06-12T02:38:21.207 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.207 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.208 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:21.208 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.698060+0000", 2020-06-12T02:38:21.208 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00061229800000000005 2020-06-12T02:38:21.208 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.209 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.209 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:21.209 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.698673+0000", 2020-06-12T02:38:21.209 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018056592 2020-06-12T02:38:21.210 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.210 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.210 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:21.210 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.716729+0000", 2020-06-12T02:38:21.210 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6626999999999996e-05 2020-06-12T02:38:21.211 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.211 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.211 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:21.211 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.716816+0000", 2020-06-12T02:38:21.211 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1032999999999995e-05 2020-06-12T02:38:21.212 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.212 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.212 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:21.212 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.716897+0000", 2020-06-12T02:38:21.212 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020991011 2020-06-12T02:38:21.213 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:21.213 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:21.213 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:21.213 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.214 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.214 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8804 3.bs0 3:d0ebae7d:::smithi10729407-3:head [append 0~589824] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:21.214 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.700177+0000", 2020-06-12T02:38:21.214 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 9.4250352419999999, 2020-06-12T02:38:21.214 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020057443000000001, 2020-06-12T02:38:21.215 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:21.215 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:21.215 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:21.215 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:21.215 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:21.216 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8804 2020-06-12T02:38:21.216 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.216 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:21.216 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.217 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:21.217 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700177+0000", 2020-06-12T02:38:21.217 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:21.217 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.217 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.218 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:21.218 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700177+0000", 2020-06-12T02:38:21.218 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038999500000000002 2020-06-12T02:38:21.218 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.218 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.219 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:21.219 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700567+0000", 2020-06-12T02:38:21.219 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.064e-05 2020-06-12T02:38:21.219 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.219 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.220 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:21.220 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700578+0000", 2020-06-12T02:38:21.220 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8008000000000001e-05 2020-06-12T02:38:21.220 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.220 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.221 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:21.221 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700596+0000", 2020-06-12T02:38:21.221 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7950000000000008e-06 2020-06-12T02:38:21.221 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.222 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.222 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:21.222 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700604+0000", 2020-06-12T02:38:21.222 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0086454970000000003 2020-06-12T02:38:21.223 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.223 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.223 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:21.223 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.709249+0000", 2020-06-12T02:38:21.223 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014904799999999999 2020-06-12T02:38:21.223 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.224 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.224 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:21.224 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.709398+0000", 2020-06-12T02:38:21.224 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00087258400000000003 2020-06-12T02:38:21.224 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.225 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.225 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:21.225 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.710271+0000", 2020-06-12T02:38:21.225 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085096960000000006 2020-06-12T02:38:21.226 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.226 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.226 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:21.226 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.718780+0000", 2020-06-12T02:38:21.226 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013730579999999999 2020-06-12T02:38:21.227 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.227 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.227 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:21.227 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.720153+0000", 2020-06-12T02:38:21.227 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1122e-05 2020-06-12T02:38:21.228 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.228 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.228 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:21.228 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.720235+0000", 2020-06-12T02:38:21.228 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020057443000000001 2020-06-12T02:38:21.229 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:21.229 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:21.229 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:21.229 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.230 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.230 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8805 3.bs0 3:d0ebae7d:::smithi10729407-3:head [append 0~49152] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:21.230 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.700675+0000", 2020-06-12T02:38:21.230 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 9.4245376699999994, 2020-06-12T02:38:21.230 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.023534237999999999, 2020-06-12T02:38:21.231 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:21.231 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:21.231 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:21.231 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:21.231 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:21.232 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8805 2020-06-12T02:38:21.232 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.232 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:21.232 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.232 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:21.232 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700675+0000", 2020-06-12T02:38:21.233 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:21.233 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.233 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.233 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:21.233 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700675+0000", 2020-06-12T02:38:21.234 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119883 2020-06-12T02:38:21.234 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.234 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.235 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:21.235 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700795+0000", 2020-06-12T02:38:21.235 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1179999999999992e-06 2020-06-12T02:38:21.235 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.236 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.236 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:21.236 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700803+0000", 2020-06-12T02:38:21.237 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3040999999999999e-05 2020-06-12T02:38:21.237 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.237 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.237 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:21.237 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700816+0000", 2020-06-12T02:38:21.238 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9499999999999998e-06 2020-06-12T02:38:21.238 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.238 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.238 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:21.238 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700822+0000", 2020-06-12T02:38:21.239 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014038444000000001 2020-06-12T02:38:21.239 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.239 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.239 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:21.239 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.714860+0000", 2020-06-12T02:38:21.240 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000113682 2020-06-12T02:38:21.240 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.240 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.240 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:21.240 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.714974+0000", 2020-06-12T02:38:21.241 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00058496799999999995 2020-06-12T02:38:21.241 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.241 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.241 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:21.242 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.715559+0000", 2020-06-12T02:38:21.242 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0033177940000000002 2020-06-12T02:38:21.242 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.242 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.242 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:21.243 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.718877+0000", 2020-06-12T02:38:21.243 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.005256625 2020-06-12T02:38:21.243 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.243 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.244 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:21.244 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.724133+0000", 2020-06-12T02:38:21.244 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5733000000000001e-05 2020-06-12T02:38:21.244 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.244 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.244 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:21.245 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.724209+0000", 2020-06-12T02:38:21.245 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.023534237999999999 2020-06-12T02:38:21.245 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:21.245 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:21.245 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:21.246 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.246 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.246 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8806 3.bs0 3:d0ebae7d:::smithi10729407-3:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:21.246 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.700882+0000", 2020-06-12T02:38:21.246 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 9.4243308040000002, 2020-06-12T02:38:21.247 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025273714999999999, 2020-06-12T02:38:21.247 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:21.247 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:21.247 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:21.247 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:21.248 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:21.248 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8806 2020-06-12T02:38:21.248 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.248 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:21.248 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.249 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:21.249 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700882+0000", 2020-06-12T02:38:21.249 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:21.249 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.249 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.250 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:21.250 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700882+0000", 2020-06-12T02:38:21.250 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7329000000000004e-05 2020-06-12T02:38:21.250 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.250 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.251 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:21.251 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700969+0000", 2020-06-12T02:38:21.251 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0620000000000001e-06 2020-06-12T02:38:21.251 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.252 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.252 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:21.252 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700977+0000", 2020-06-12T02:38:21.252 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5291e-05 2020-06-12T02:38:21.252 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.253 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.253 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:21.253 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700992+0000", 2020-06-12T02:38:21.253 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.297e-06 2020-06-12T02:38:21.253 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.254 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.254 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:21.254 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700999+0000", 2020-06-12T02:38:21.254 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015970872000000001 2020-06-12T02:38:21.254 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.255 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.255 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:21.255 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.716969+0000", 2020-06-12T02:38:21.255 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105802 2020-06-12T02:38:21.255 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.256 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.256 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:21.256 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.717075+0000", 2020-06-12T02:38:21.256 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00042746400000000001 2020-06-12T02:38:21.256 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.256 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.257 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:21.257 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.717503+0000", 2020-06-12T02:38:21.257 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013889869999999999 2020-06-12T02:38:21.257 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.257 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.258 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:21.258 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.718892+0000", 2020-06-12T02:38:21.258 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071672740000000004 2020-06-12T02:38:21.258 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.258 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.259 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:21.259 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.726059+0000", 2020-06-12T02:38:21.259 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6336999999999996e-05 2020-06-12T02:38:21.259 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.260 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.260 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:21.260 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.726155+0000", 2020-06-12T02:38:21.260 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025273714999999999 2020-06-12T02:38:21.261 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:21.261 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:21.261 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:21.261 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.261 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.262 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8807 3.bs0 3:d0ebae7d:::smithi10729407-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e49)", 2020-06-12T02:38:21.262 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.701041+0000", 2020-06-12T02:38:21.262 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 9.4241712149999994, 2020-06-12T02:38:21.262 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030589063, 2020-06-12T02:38:21.262 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:21.263 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:21.263 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:21.263 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:21.263 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:21.263 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8807 2020-06-12T02:38:21.264 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.264 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:21.264 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.264 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:21.264 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.701041+0000", 2020-06-12T02:38:21.264 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:21.265 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.265 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.265 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:21.265 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.701041+0000", 2020-06-12T02:38:21.265 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6292999999999999e-05 2020-06-12T02:38:21.266 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.266 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.266 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:21.266 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.701087+0000", 2020-06-12T02:38:21.266 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9840000000000004e-06 2020-06-12T02:38:21.267 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.267 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.267 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:21.267 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.701094+0000", 2020-06-12T02:38:21.267 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3713000000000001e-05 2020-06-12T02:38:21.268 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.268 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.268 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:21.268 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.701108+0000", 2020-06-12T02:38:21.268 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6200000000000004e-06 2020-06-12T02:38:21.269 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.269 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.269 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:21.269 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.701114+0000", 2020-06-12T02:38:21.270 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016879646000000002 2020-06-12T02:38:21.270 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.270 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.270 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:21.271 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.717993+0000", 2020-06-12T02:38:21.271 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4378999999999999e-05 2020-06-12T02:38:21.271 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.271 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.271 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:21.272 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.718088+0000", 2020-06-12T02:38:21.272 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0092861850000000006 2020-06-12T02:38:21.272 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.272 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.272 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:21.273 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.727374+0000", 2020-06-12T02:38:21.273 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1155000000000004e-05 2020-06-12T02:38:21.273 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.273 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.273 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:21.273 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.727465+0000", 2020-06-12T02:38:21.274 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004165088 2020-06-12T02:38:21.274 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.274 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.274 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:21.274 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.731630+0000", 2020-06-12T02:38:21.275 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030589063 2020-06-12T02:38:21.275 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:21.275 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:21.275 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:21.276 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.276 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.276 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8847 3.8s0 3:14244450:::smithi10729407-2:head [append 0~638976] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:21.277 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.836821+0000", 2020-06-12T02:38:21.277 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 9.2883911490000006, 2020-06-12T02:38:21.277 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020322752999999999, 2020-06-12T02:38:21.277 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:21.277 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:21.278 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:21.278 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:21.278 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:21.278 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8847 2020-06-12T02:38:21.279 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.279 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:21.279 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.279 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:21.279 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.836821+0000", 2020-06-12T02:38:21.280 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:21.280 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.280 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.280 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:21.281 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.836821+0000", 2020-06-12T02:38:21.281 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00021910399999999999 2020-06-12T02:38:21.281 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.281 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.281 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:21.282 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.837040+0000", 2020-06-12T02:38:21.282 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5099999999999998e-06 2020-06-12T02:38:21.282 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.282 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.282 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:21.283 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.837049+0000", 2020-06-12T02:38:21.283 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4453e-05 2020-06-12T02:38:21.283 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.283 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.283 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:21.283 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.837063+0000", 2020-06-12T02:38:21.284 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.251e-06 2020-06-12T02:38:21.284 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.284 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.284 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:21.284 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.837070+0000", 2020-06-12T02:38:21.285 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0026855310000000001 2020-06-12T02:38:21.285 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.285 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.285 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:21.285 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.839755+0000", 2020-06-12T02:38:21.286 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9023000000000006e-05 2020-06-12T02:38:21.286 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.287 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.287 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:21.287 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.839834+0000", 2020-06-12T02:38:21.288 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0059031170000000003 2020-06-12T02:38:21.288 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.288 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.288 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:21.289 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.845737+0000", 2020-06-12T02:38:21.289 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4455000000000003e-05 2020-06-12T02:38:21.289 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.289 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.291 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:21.291 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.845832+0000", 2020-06-12T02:38:21.291 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00074392800000000004 2020-06-12T02:38:21.291 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.292 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.292 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:21.292 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.846576+0000", 2020-06-12T02:38:21.292 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0070446479999999997 2020-06-12T02:38:21.292 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.293 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.293 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:21.293 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.853620+0000", 2020-06-12T02:38:21.293 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.003447126 2020-06-12T02:38:21.293 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.293 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.294 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:21.294 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.857067+0000", 2020-06-12T02:38:21.294 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6606999999999993e-05 2020-06-12T02:38:21.294 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.295 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.295 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:21.295 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.857144+0000", 2020-06-12T02:38:21.295 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020322752999999999 2020-06-12T02:38:21.295 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:21.296 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:21.296 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:21.296 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.296 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.296 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8849 3.8s0 3:14244450:::smithi10729407-2:head [append 0~417792] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:21.296 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.843163+0000", 2020-06-12T02:38:21.297 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 9.2820498520000001, 2020-06-12T02:38:21.297 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019870644999999999, 2020-06-12T02:38:21.297 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:21.297 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:21.297 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:21.298 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:21.298 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:21.298 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8849 2020-06-12T02:38:21.298 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.298 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:21.298 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.299 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:21.299 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.843163+0000", 2020-06-12T02:38:21.299 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:21.299 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.299 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.300 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:21.300 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.843163+0000", 2020-06-12T02:38:21.300 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000185687 2020-06-12T02:38:21.300 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.300 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.301 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:21.301 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.843348+0000", 2020-06-12T02:38:21.301 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2609999999999999e-06 2020-06-12T02:38:21.301 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.301 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.302 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:21.302 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.843354+0000", 2020-06-12T02:38:21.302 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1477999999999999e-05 2020-06-12T02:38:21.302 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.302 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.303 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:21.303 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.843366+0000", 2020-06-12T02:38:21.303 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2469999999999999e-06 2020-06-12T02:38:21.303 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.303 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.303 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:21.304 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.843371+0000", 2020-06-12T02:38:21.304 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7528999999999999e-05 2020-06-12T02:38:21.304 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.304 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.304 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:21.305 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.843409+0000", 2020-06-12T02:38:21.305 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3548999999999995e-05 2020-06-12T02:38:21.305 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.305 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.305 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:21.306 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.843502+0000", 2020-06-12T02:38:21.306 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010203337999999999 2020-06-12T02:38:21.306 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.306 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.306 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:21.306 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.853706+0000", 2020-06-12T02:38:21.307 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010009399999999999 2020-06-12T02:38:21.307 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.307 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.307 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:21.307 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.853806+0000", 2020-06-12T02:38:21.308 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000696544 2020-06-12T02:38:21.308 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.308 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.308 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:21.308 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.854502+0000", 2020-06-12T02:38:21.308 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0065843669999999998 2020-06-12T02:38:21.309 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.309 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.309 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:21.309 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.861087+0000", 2020-06-12T02:38:21.309 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0018563799999999999 2020-06-12T02:38:21.311 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.312 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.312 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:21.312 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.862943+0000", 2020-06-12T02:38:21.312 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0171000000000001e-05 2020-06-12T02:38:21.313 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.313 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.313 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:21.313 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.863033+0000", 2020-06-12T02:38:21.313 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019870644999999999 2020-06-12T02:38:21.314 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:21.314 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:21.314 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:21.314 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.314 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.315 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8850 3.8s0 3:14244450:::smithi10729407-2:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:21.315 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.843425+0000", 2020-06-12T02:38:21.315 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 9.2817868600000004, 2020-06-12T02:38:21.316 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019923956999999999, 2020-06-12T02:38:21.316 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:21.316 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:21.317 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:21.317 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:21.317 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:21.317 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8850 2020-06-12T02:38:21.317 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.318 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:21.318 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.318 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:21.325 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.843425+0000", 2020-06-12T02:38:21.325 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:21.325 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.326 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.326 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:21.326 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.843425+0000", 2020-06-12T02:38:21.326 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8193000000000002e-05 2020-06-12T02:38:21.326 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.326 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.327 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:21.327 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.843504+0000", 2020-06-12T02:38:21.327 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3110000000000002e-06 2020-06-12T02:38:21.327 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.327 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.328 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:21.328 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.843510+0000", 2020-06-12T02:38:21.328 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8329999999999994e-06 2020-06-12T02:38:21.328 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.328 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.329 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:21.329 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.843518+0000", 2020-06-12T02:38:21.329 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.1860000000000001e-06 2020-06-12T02:38:21.329 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.329 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.329 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:21.330 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.843521+0000", 2020-06-12T02:38:21.330 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8204999999999998e-05 2020-06-12T02:38:21.330 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.330 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.330 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:21.331 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.843549+0000", 2020-06-12T02:38:21.331 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4450000000000006e-05 2020-06-12T02:38:21.331 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.331 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.332 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:21.332 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.843634+0000", 2020-06-12T02:38:21.332 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013555185000000001 2020-06-12T02:38:21.332 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.332 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.333 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:21.333 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.857189+0000", 2020-06-12T02:38:21.333 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7164999999999996e-05 2020-06-12T02:38:21.333 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.333 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.334 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:21.334 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.857286+0000", 2020-06-12T02:38:21.335 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000406809 2020-06-12T02:38:21.335 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.335 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.335 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:21.335 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.857693+0000", 2020-06-12T02:38:21.336 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0033959120000000001 2020-06-12T02:38:21.336 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.336 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.336 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:21.336 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.861089+0000", 2020-06-12T02:38:21.336 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0021525369999999999 2020-06-12T02:38:21.337 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.337 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.337 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:21.337 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.863241+0000", 2020-06-12T02:38:21.337 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000108171 2020-06-12T02:38:21.338 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.338 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.338 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:21.338 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.863349+0000", 2020-06-12T02:38:21.338 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019923956999999999 2020-06-12T02:38:21.338 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:21.339 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:21.339 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:21.339 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.339 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.339 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8851 3.8s0 3:14244450:::smithi10729407-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e49)", 2020-06-12T02:38:21.340 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.843565+0000", 2020-06-12T02:38:21.340 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 9.2816477769999999, 2020-06-12T02:38:21.340 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.02215278, 2020-06-12T02:38:21.340 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:21.340 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:21.340 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:21.341 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:21.341 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:21.341 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8851 2020-06-12T02:38:21.341 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.341 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:21.342 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.342 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:21.342 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.843565+0000", 2020-06-12T02:38:21.342 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:21.342 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.342 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.343 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:21.343 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.843565+0000", 2020-06-12T02:38:21.343 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4527999999999998e-05 2020-06-12T02:38:21.343 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.343 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.344 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:21.344 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.843609+0000", 2020-06-12T02:38:21.345 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0909999999999998e-06 2020-06-12T02:38:21.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.345 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.345 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:21.345 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.843615+0000", 2020-06-12T02:38:21.346 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2254e-05 2020-06-12T02:38:21.346 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.346 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.346 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:21.346 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.843627+0000", 2020-06-12T02:38:21.346 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5539999999999997e-06 2020-06-12T02:38:21.347 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.347 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.347 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:21.347 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.843633+0000", 2020-06-12T02:38:21.347 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2058999999999999e-05 2020-06-12T02:38:21.348 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.348 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.348 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:21.348 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.843665+0000", 2020-06-12T02:38:21.349 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9106e-05 2020-06-12T02:38:21.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.350 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.350 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:21.350 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.843714+0000", 2020-06-12T02:38:21.350 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014386637000000001 2020-06-12T02:38:21.350 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.351 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.351 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:21.351 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.858101+0000", 2020-06-12T02:38:21.351 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000102223 2020-06-12T02:38:21.351 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.352 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.352 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:21.352 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.858203+0000", 2020-06-12T02:38:21.352 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0058912900000000004 2020-06-12T02:38:21.352 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.353 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.353 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:21.353 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864094+0000", 2020-06-12T02:38:21.353 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4738e-05 2020-06-12T02:38:21.353 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.353 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.354 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:21.354 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864189+0000", 2020-06-12T02:38:21.354 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0015283 2020-06-12T02:38:21.354 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.354 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.355 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:21.355 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.865717+0000", 2020-06-12T02:38:21.355 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.02215278 2020-06-12T02:38:21.355 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:21.355 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:21.356 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:21.356 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.357 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.357 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:9628 3.bs0 3:d09543d2:::smithi10729407-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e54)", 2020-06-12T02:38:21.357 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:16.887830+0000", 2020-06-12T02:38:21.357 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 4.2373826430000001, 2020-06-12T02:38:21.357 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022465841, 2020-06-12T02:38:21.358 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:21.358 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:21.358 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:21.358 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:21.358 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:21.359 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 9628 2020-06-12T02:38:21.359 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.359 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:21.359 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.359 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:21.360 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.887830+0000", 2020-06-12T02:38:21.361 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:21.361 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.361 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.362 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:21.362 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.887830+0000", 2020-06-12T02:38:21.362 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6709000000000003e-05 2020-06-12T02:38:21.362 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.362 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.363 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:21.363 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.887876+0000", 2020-06-12T02:38:21.363 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8150000000000002e-06 2020-06-12T02:38:21.363 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.363 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.364 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:21.364 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.887882+0000", 2020-06-12T02:38:21.364 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.359e-05 2020-06-12T02:38:21.364 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.364 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.365 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:21.365 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.887896+0000", 2020-06-12T02:38:21.366 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0029999999999996e-06 2020-06-12T02:38:21.366 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.366 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.366 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:21.367 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.887900+0000", 2020-06-12T02:38:21.367 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012522259999999999 2020-06-12T02:38:21.367 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.367 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.368 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:21.368 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.889152+0000", 2020-06-12T02:38:21.368 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3249000000000002e-05 2020-06-12T02:38:21.368 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.368 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.369 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:21.369 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.889235+0000", 2020-06-12T02:38:21.369 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012485619 2020-06-12T02:38:21.369 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.369 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.370 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:21.370 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.901721+0000", 2020-06-12T02:38:21.370 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2717999999999999e-05 2020-06-12T02:38:21.370 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.370 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.371 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:21.371 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.901774+0000", 2020-06-12T02:38:21.371 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004869305 2020-06-12T02:38:21.371 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.371 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.372 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:21.372 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.906643+0000", 2020-06-12T02:38:21.372 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1521999999999996e-05 2020-06-12T02:38:21.372 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.372 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.372 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:21.373 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.906734+0000", 2020-06-12T02:38:21.373 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0035610849999999999 2020-06-12T02:38:21.373 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.374 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.374 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:21.374 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.910296+0000", 2020-06-12T02:38:21.374 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022465841 2020-06-12T02:38:21.375 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:21.375 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:21.375 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:21.375 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.375 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.376 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:9675 3.bs0 3:d0ebae7d:::smithi10729407-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e54)", 2020-06-12T02:38:21.376 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:16.985984+0000", 2020-06-12T02:38:21.376 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 4.1392284730000002, 2020-06-12T02:38:21.376 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020123386, 2020-06-12T02:38:21.376 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:21.377 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:21.377 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:21.378 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:21.378 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:21.378 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 9675 2020-06-12T02:38:21.378 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.379 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:21.379 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.379 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:21.379 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.985984+0000", 2020-06-12T02:38:21.379 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:21.380 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.380 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.380 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:21.380 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.985984+0000", 2020-06-12T02:38:21.380 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6591999999999999e-05 2020-06-12T02:38:21.381 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.381 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.381 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:21.381 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.986010+0000", 2020-06-12T02:38:21.381 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3119999999999998e-06 2020-06-12T02:38:21.381 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.382 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.382 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:21.382 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.986014+0000", 2020-06-12T02:38:21.382 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9589999999999995e-06 2020-06-12T02:38:21.382 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.383 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.383 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:21.383 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.986022+0000", 2020-06-12T02:38:21.383 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.576e-06 2020-06-12T02:38:21.383 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.383 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.384 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:21.384 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.986024+0000", 2020-06-12T02:38:21.384 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000100881 2020-06-12T02:38:21.384 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.384 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.385 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:21.385 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.986125+0000", 2020-06-12T02:38:21.385 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9279999999999999e-05 2020-06-12T02:38:21.385 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.385 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.386 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:21.386 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.986214+0000", 2020-06-12T02:38:21.386 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0095388030000000002 2020-06-12T02:38:21.386 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.386 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.386 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:21.387 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.995753+0000", 2020-06-12T02:38:21.387 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0044999999999999e-05 2020-06-12T02:38:21.387 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.387 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.387 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:21.388 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.995803+0000", 2020-06-12T02:38:21.388 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0092663399999999996 2020-06-12T02:38:21.388 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.388 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.388 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:21.388 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.005070+0000", 2020-06-12T02:38:21.389 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5955e-05 2020-06-12T02:38:21.389 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.389 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.389 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:21.389 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.005156+0000", 2020-06-12T02:38:21.390 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00095164299999999996 2020-06-12T02:38:21.390 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.390 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.390 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:21.390 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.006107+0000", 2020-06-12T02:38:21.390 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020123386 2020-06-12T02:38:21.391 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:21.391 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:21.391 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:21.391 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.391 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.392 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:9917 3.bs0 3:d09543d2:::smithi10729407-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e54)", 2020-06-12T02:38:21.392 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:17.669696+0000", 2020-06-12T02:38:21.392 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.4555159620000002, 2020-06-12T02:38:21.392 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.021166378, 2020-06-12T02:38:21.392 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:21.393 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:21.393 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:21.393 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:21.393 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:21.394 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 9917 2020-06-12T02:38:21.394 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.394 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:21.394 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.394 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:21.395 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.669696+0000", 2020-06-12T02:38:21.395 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:21.395 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.395 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.395 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:21.395 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.669696+0000", 2020-06-12T02:38:21.396 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2474e-05 2020-06-12T02:38:21.396 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.396 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.396 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:21.396 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.669739+0000", 2020-06-12T02:38:21.397 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5769999999999999e-06 2020-06-12T02:38:21.397 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.397 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.397 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:21.397 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.669745+0000", 2020-06-12T02:38:21.398 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2695e-05 2020-06-12T02:38:21.398 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.398 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.398 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:21.398 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.669758+0000", 2020-06-12T02:38:21.398 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7570000000000004e-06 2020-06-12T02:38:21.399 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.399 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.399 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:21.399 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.669763+0000", 2020-06-12T02:38:21.399 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0049988360000000004 2020-06-12T02:38:21.399 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.400 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.400 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:21.400 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.674762+0000", 2020-06-12T02:38:21.400 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8404000000000002e-05 2020-06-12T02:38:21.400 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.401 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.401 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:21.401 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.674860+0000", 2020-06-12T02:38:21.401 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0056080419999999997 2020-06-12T02:38:21.401 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.401 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.402 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:21.402 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.680468+0000", 2020-06-12T02:38:21.402 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000108749 2020-06-12T02:38:21.402 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.402 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.403 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:21.403 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.680577+0000", 2020-06-12T02:38:21.403 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0074646280000000001 2020-06-12T02:38:21.403 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.403 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.403 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:21.404 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.688042+0000", 2020-06-12T02:38:21.404 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6777e-05 2020-06-12T02:38:21.404 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.404 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.404 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:21.405 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.688118+0000", 2020-06-12T02:38:21.405 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002744439 2020-06-12T02:38:21.405 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.405 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.405 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:21.405 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.690863+0000", 2020-06-12T02:38:21.406 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.021166378 2020-06-12T02:38:21.406 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:21.406 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:21.406 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:21.406 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.407 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.407 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:10022 3.bs0 3:d09543d2:::smithi10729407-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e54)", 2020-06-12T02:38:21.407 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:17.985466+0000", 2020-06-12T02:38:21.407 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.139746175, 2020-06-12T02:38:21.407 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019634347, 2020-06-12T02:38:21.407 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:21.408 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:21.408 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:21.408 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:21.408 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:21.408 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 10022 2020-06-12T02:38:21.409 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.409 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:21.409 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.409 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:21.409 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.985466+0000", 2020-06-12T02:38:21.409 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:21.410 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.410 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.410 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:21.410 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.985466+0000", 2020-06-12T02:38:21.410 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4023999999999997e-05 2020-06-12T02:38:21.411 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.411 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.411 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:21.411 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.985510+0000", 2020-06-12T02:38:21.411 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3829999999999997e-06 2020-06-12T02:38:21.411 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.412 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.412 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:21.412 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.985517+0000", 2020-06-12T02:38:21.412 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1986e-05 2020-06-12T02:38:21.412 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.413 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.413 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:21.413 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.985529+0000", 2020-06-12T02:38:21.413 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1139999999999997e-06 2020-06-12T02:38:21.413 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.413 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.414 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:21.414 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.985534+0000", 2020-06-12T02:38:21.414 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.4019000000000002e-05 2020-06-12T02:38:21.414 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.414 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.415 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:21.415 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.985558+0000", 2020-06-12T02:38:21.415 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010077 2020-06-12T02:38:21.415 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.415 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.415 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:21.416 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.985658+0000", 2020-06-12T02:38:21.416 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014699946 2020-06-12T02:38:21.416 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.416 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.416 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:21.417 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.000358+0000", 2020-06-12T02:38:21.417 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034005400000000001 2020-06-12T02:38:21.417 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.417 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.417 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:21.417 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.000698+0000", 2020-06-12T02:38:21.418 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0031142499999999998 2020-06-12T02:38:21.418 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.418 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.418 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:21.418 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.003813+0000", 2020-06-12T02:38:21.419 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000110443 2020-06-12T02:38:21.419 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.419 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.419 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:21.419 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.003923+0000", 2020-06-12T02:38:21.419 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011773580000000001 2020-06-12T02:38:21.420 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.420 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.420 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:21.420 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.005101+0000", 2020-06-12T02:38:21.420 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019634347 2020-06-12T02:38:21.420 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:21.421 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:21.421 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:21.421 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.421 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.421 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:10040 3.5s1 3:a446366d:::smithi10729407-11:head [append 0~131072] snapc 0=[] ondisk+write+known_if_redirected e54)", 2020-06-12T02:38:21.422 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:18.037838+0000", 2020-06-12T02:38:21.422 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.087374509, 2020-06-12T02:38:21.422 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020329944999999999, 2020-06-12T02:38:21.422 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:21.422 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:21.422 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:21.423 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:21.423 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:21.423 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 10040 2020-06-12T02:38:21.423 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.423 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:21.424 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.424 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:21.424 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.037838+0000", 2020-06-12T02:38:21.424 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:21.424 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.424 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.425 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:21.425 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.037838+0000", 2020-06-12T02:38:21.425 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1982000000000003e-05 2020-06-12T02:38:21.425 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.425 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.426 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:21.426 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.037910+0000", 2020-06-12T02:38:21.426 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3550000000000001e-06 2020-06-12T02:38:21.426 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.426 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.426 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:21.427 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.037916+0000", 2020-06-12T02:38:21.427 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3030000000000001e-05 2020-06-12T02:38:21.427 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.427 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.427 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:21.428 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.037929+0000", 2020-06-12T02:38:21.428 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1019999999999999e-06 2020-06-12T02:38:21.428 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.428 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.428 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:21.428 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.037936+0000", 2020-06-12T02:38:21.429 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3041999999999999e-05 2020-06-12T02:38:21.429 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.429 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.429 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:21.429 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.037979+0000", 2020-06-12T02:38:21.430 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101257 2020-06-12T02:38:21.430 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.430 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.430 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:21.430 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.038081+0000", 2020-06-12T02:38:21.430 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012472613 2020-06-12T02:38:21.431 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.431 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.431 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:21.431 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.050553+0000", 2020-06-12T02:38:21.431 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2694999999999995e-05 2020-06-12T02:38:21.432 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.432 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.432 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:21.432 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.050616+0000", 2020-06-12T02:38:21.432 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036818700000000002 2020-06-12T02:38:21.432 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.433 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.433 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:21.433 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.050984+0000", 2020-06-12T02:38:21.433 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0019458419999999999 2020-06-12T02:38:21.433 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.434 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.434 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:21.434 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.052930+0000", 2020-06-12T02:38:21.434 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0051913590000000004 2020-06-12T02:38:21.434 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.435 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.435 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:21.435 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.058121+0000", 2020-06-12T02:38:21.435 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6480999999999998e-05 2020-06-12T02:38:21.435 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.435 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.436 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:21.436 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.058168+0000", 2020-06-12T02:38:21.436 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020329944999999999 2020-06-12T02:38:21.436 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:21.436 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:21.437 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:21.437 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.437 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.437 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:10041 3.5s1 3:a446366d:::smithi10729407-11:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e54)", 2020-06-12T02:38:21.437 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:18.037986+0000", 2020-06-12T02:38:21.437 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.087226743, 2020-06-12T02:38:21.438 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020334035, 2020-06-12T02:38:21.438 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:21.438 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:21.438 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:21.438 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:21.439 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:21.439 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 10041 2020-06-12T02:38:21.439 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.439 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:21.439 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.440 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:21.440 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.037986+0000", 2020-06-12T02:38:21.440 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:21.440 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.440 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.441 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:21.442 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.037986+0000", 2020-06-12T02:38:21.442 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5748999999999997e-05 2020-06-12T02:38:21.442 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.442 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.443 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:21.443 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.038051+0000", 2020-06-12T02:38:21.443 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1809999999999998e-06 2020-06-12T02:38:21.443 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.443 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.444 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:21.444 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.038058+0000", 2020-06-12T02:38:21.444 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1058e-05 2020-06-12T02:38:21.444 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.444 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.445 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:21.445 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.038069+0000", 2020-06-12T02:38:21.445 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8930000000000004e-06 2020-06-12T02:38:21.445 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.445 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.445 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:21.446 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.038073+0000", 2020-06-12T02:38:21.447 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7157999999999998e-05 2020-06-12T02:38:21.447 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.447 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.447 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:21.447 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.038121+0000", 2020-06-12T02:38:21.448 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6507000000000003e-05 2020-06-12T02:38:21.448 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.448 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.448 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:21.448 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.038217+0000", 2020-06-12T02:38:21.448 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013888977 2020-06-12T02:38:21.449 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.449 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.449 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:21.449 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.052106+0000", 2020-06-12T02:38:21.449 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4517999999999994e-05 2020-06-12T02:38:21.450 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.451 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.451 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:21.451 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.052181+0000", 2020-06-12T02:38:21.451 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040230000000000002 2020-06-12T02:38:21.451 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.452 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.452 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:21.452 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.052583+0000", 2020-06-12T02:38:21.452 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002737279 2020-06-12T02:38:21.452 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.452 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.453 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:21.453 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.055320+0000", 2020-06-12T02:38:21.453 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00294977 2020-06-12T02:38:21.453 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.453 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.454 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:21.454 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.058270+0000", 2020-06-12T02:38:21.454 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9645000000000003e-05 2020-06-12T02:38:21.454 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.454 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.455 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:21.455 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.058320+0000", 2020-06-12T02:38:21.455 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020334035 2020-06-12T02:38:21.455 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:21.455 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:21.456 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:21.456 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.456 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.456 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:10042 3.5s1 3:a446366d:::smithi10729407-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e54)", 2020-06-12T02:38:21.456 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:18.038116+0000", 2020-06-12T02:38:21.457 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.0870967340000002, 2020-06-12T02:38:21.457 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.021560902, 2020-06-12T02:38:21.457 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:21.457 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:21.458 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:21.458 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:21.458 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:21.458 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 10042 2020-06-12T02:38:21.458 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.459 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:21.459 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.459 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:21.459 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.038116+0000", 2020-06-12T02:38:21.459 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:21.459 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.460 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.460 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:21.460 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.038116+0000", 2020-06-12T02:38:21.460 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7672000000000001e-05 2020-06-12T02:38:21.460 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.461 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.461 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:21.461 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.038153+0000", 2020-06-12T02:38:21.461 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4979999999999997e-06 2020-06-12T02:38:21.461 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.461 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.462 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:21.462 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.038159+0000", 2020-06-12T02:38:21.462 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1895999999999999e-05 2020-06-12T02:38:21.462 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.462 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.463 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:21.463 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.038171+0000", 2020-06-12T02:38:21.463 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6460000000000001e-06 2020-06-12T02:38:21.463 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.463 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.463 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:21.464 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.038175+0000", 2020-06-12T02:38:21.464 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4574999999999997e-05 2020-06-12T02:38:21.464 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.464 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.464 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:21.465 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.038250+0000", 2020-06-12T02:38:21.465 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000108447 2020-06-12T02:38:21.465 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.465 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.465 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:21.466 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.038358+0000", 2020-06-12T02:38:21.466 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015001960999999999 2020-06-12T02:38:21.466 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.466 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.466 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:21.466 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.053360+0000", 2020-06-12T02:38:21.467 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7246999999999999e-05 2020-06-12T02:38:21.467 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.467 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.467 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:21.467 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.053458+0000", 2020-06-12T02:38:21.468 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0053373259999999999 2020-06-12T02:38:21.468 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.468 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.468 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:21.469 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.058795+0000", 2020-06-12T02:38:21.469 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4442000000000002e-05 2020-06-12T02:38:21.469 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.469 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.469 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:21.470 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.058879+0000", 2020-06-12T02:38:21.470 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00079719199999999995 2020-06-12T02:38:21.470 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:21.470 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:21.470 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:21.471 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.059677+0000", 2020-06-12T02:38:21.471 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.021560902 2020-06-12T02:38:21.471 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:21.471 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:21.472 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:21.472 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:21.472 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:21.472 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:21.487 INFO:tasks.rados.rados.0.smithi107.stdout:5226: writing smithi10729407-9 from 712704 to 1114112 tid 2 2020-06-12T02:38:21.488 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:38:21.488 INFO:tasks.rados.rados.0.smithi107.stdout:5227: seq_num 5226 ranges {0=729088,729088=401408} 2020-06-12T02:38:21.488 INFO:tasks.rados.rados.0.smithi107.stdout:5227: writing smithi10729407-16 from 0 to 729088 tid 1 2020-06-12T02:38:21.489 INFO:tasks.rados.rados.0.smithi107.stdout:5227: writing smithi10729407-16 from 729088 to 1130496 tid 2 2020-06-12T02:38:21.489 INFO:tasks.rados.rados.0.smithi107.stdout:5228: delete oid 19 current snap is 0 2020-06-12T02:38:21.489 INFO:tasks.rados.rados.0.smithi107.stdout:5223: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:21.489 INFO:tasks.rados.rados.0.smithi107.stdout:5223: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:21.489 INFO:tasks.rados.rados.0.smithi107.stdout:5223: finishing write tid 5 to smithi10729407-12 2020-06-12T02:38:21.490 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 917 (ObjNum 5222 snap 0 seq_num 5222) dirty exists 2020-06-12T02:38:21.490 INFO:tasks.rados.rados.0.smithi107.stdout:5223: left oid 12 (ObjNum 5222 snap 0 seq_num 5222) 2020-06-12T02:38:21.490 INFO:tasks.rados.rados.0.smithi107.stdout:5226: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:21.490 INFO:tasks.rados.rados.0.smithi107.stdout:5223: done (4 left) 2020-06-12T02:38:21.491 INFO:tasks.rados.rados.0.smithi107.stdout:5228: done (3 left) 2020-06-12T02:38:21.491 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:21.491 INFO:tasks.rados.rados.0.smithi107.stdout:5229: seq_num 5228 ranges {0=761856,761856=401408} 2020-06-12T02:38:21.491 INFO:tasks.rados.rados.0.smithi107.stdout:5229: writing smithi10729407-1 from 0 to 761856 tid 1 2020-06-12T02:38:21.491 INFO:tasks.rados.rados.0.smithi107.stdout:5229: writing smithi10729407-1 from 761856 to 1163264 tid 2 2020-06-12T02:38:21.492 INFO:tasks.rados.rados.0.smithi107.stdout:5230: delete oid 17 current snap is 0 2020-06-12T02:38:21.492 INFO:tasks.rados.rados.0.smithi107.stdout:5226: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:21.492 INFO:tasks.rados.rados.0.smithi107.stdout:5226: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:21.492 INFO:tasks.rados.rados.0.smithi107.stdout:5226: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:21.492 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 945 (ObjNum 5225 snap 0 seq_num 5225) dirty exists 2020-06-12T02:38:21.492 INFO:tasks.rados.rados.0.smithi107.stdout:5226: left oid 9 (ObjNum 5225 snap 0 seq_num 5225) 2020-06-12T02:38:21.493 INFO:tasks.rados.rados.0.smithi107.stdout:5227: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:21.493 INFO:tasks.rados.rados.0.smithi107.stdout:5227: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:21.493 INFO:tasks.rados.rados.0.smithi107.stdout:5227: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:21.493 INFO:tasks.rados.rados.0.smithi107.stdout:5227: finishing write tid 5 to smithi10729407-16 2020-06-12T02:38:21.493 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 937 (ObjNum 5226 snap 0 seq_num 5226) dirty exists 2020-06-12T02:38:21.494 INFO:tasks.rados.rados.0.smithi107.stdout:5227: left oid 16 (ObjNum 5226 snap 0 seq_num 5226) 2020-06-12T02:38:21.494 INFO:tasks.rados.rados.0.smithi107.stdout:5226: done (4 left) 2020-06-12T02:38:21.494 INFO:tasks.rados.rados.0.smithi107.stdout:5227: done (3 left) 2020-06-12T02:38:21.494 INFO:tasks.rados.rados.0.smithi107.stdout:5230: done (2 left) 2020-06-12T02:38:21.494 INFO:tasks.rados.rados.0.smithi107.stdout:5231: delete oid 5 current snap is 0 2020-06-12T02:38:21.494 INFO:tasks.rados.rados.0.smithi107.stdout:5229: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:21.495 INFO:tasks.rados.rados.0.smithi107.stdout:5231: done (2 left) 2020-06-12T02:38:21.495 INFO:tasks.rados.rados.0.smithi107.stdout:5232: delete oid 18 current snap is 0 2020-06-12T02:38:21.495 INFO:tasks.rados.rados.0.smithi107.stdout:5229: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:21.495 INFO:tasks.rados.rados.0.smithi107.stdout:5232: done (2 left) 2020-06-12T02:38:21.495 INFO:tasks.rados.rados.0.smithi107.stdout:5233: delete oid 6 current snap is 0 2020-06-12T02:38:21.495 INFO:tasks.rados.rados.0.smithi107.stdout:5229: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:21.496 INFO:tasks.rados.rados.0.smithi107.stdout:5229: finishing write tid 5 to smithi10729407-1 2020-06-12T02:38:21.496 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 956 (ObjNum 5228 snap 0 seq_num 5228) dirty exists 2020-06-12T02:38:21.496 INFO:tasks.rados.rados.0.smithi107.stdout:5229: left oid 1 (ObjNum 5228 snap 0 seq_num 5228) 2020-06-12T02:38:21.496 INFO:tasks.rados.rados.0.smithi107.stdout:5229: done (2 left) 2020-06-12T02:38:21.496 INFO:tasks.rados.rados.0.smithi107.stdout:5233: done (1 left) 2020-06-12T02:38:21.497 INFO:tasks.rados.rados.0.smithi107.stdout:5234: delete oid 15 current snap is 0 2020-06-12T02:38:21.497 INFO:tasks.rados.rados.0.smithi107.stdout:5234: done (1 left) 2020-06-12T02:38:21.497 INFO:tasks.rados.rados.0.smithi107.stdout:5235: delete oid 6 current snap is 0 2020-06-12T02:38:21.497 INFO:tasks.rados.rados.0.smithi107.stdout:5235: done (1 left) 2020-06-12T02:38:21.497 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:38:21.497 INFO:tasks.rados.rados.0.smithi107.stdout:5236: seq_num 5235 ranges {0=483328,483328=663552,1146880=139264} 2020-06-12T02:38:21.498 INFO:tasks.rados.rados.0.smithi107.stdout:5236: writing smithi10729407-6 from 0 to 483328 tid 1 2020-06-12T02:38:21.498 INFO:tasks.rados.rados.0.smithi107.stdout:5236: writing smithi10729407-6 from 483328 to 1146880 tid 2 2020-06-12T02:38:21.498 INFO:tasks.rados.rados.0.smithi107.stdout:5236: writing smithi10729407-6 from 1146880 to 1286144 tid 3 2020-06-12T02:38:21.498 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:21.499 INFO:tasks.rados.rados.0.smithi107.stdout:5237: seq_num 5236 ranges {0=499712,499712=548864,1048576=253952} 2020-06-12T02:38:21.499 INFO:tasks.rados.rados.0.smithi107.stdout:5237: writing smithi10729407-4 from 0 to 499712 tid 1 2020-06-12T02:38:21.499 INFO:tasks.rados.rados.0.smithi107.stdout:5237: writing smithi10729407-4 from 499712 to 1048576 tid 2 2020-06-12T02:38:21.499 INFO:tasks.rados.rados.0.smithi107.stdout:5237: writing smithi10729407-4 from 1048576 to 1302528 tid 3 2020-06-12T02:38:21.499 INFO:tasks.rados.rados.0.smithi107.stdout:5238: delete oid 9 current snap is 0 2020-06-12T02:38:21.500 INFO:tasks.rados.rados.0.smithi107.stdout:5236: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:21.500 INFO:tasks.rados.rados.0.smithi107.stdout:5236: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:21.500 INFO:tasks.rados.rados.0.smithi107.stdout:5236: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:21.500 INFO:tasks.rados.rados.0.smithi107.stdout:5236: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:21.500 INFO:tasks.rados.rados.0.smithi107.stdout:5236: finishing write tid 6 to smithi10729407-6 2020-06-12T02:38:21.500 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 962 (ObjNum 5235 snap 0 seq_num 5235) dirty exists 2020-06-12T02:38:21.501 INFO:tasks.rados.rados.0.smithi107.stdout:5236: left oid 6 (ObjNum 5235 snap 0 seq_num 5235) 2020-06-12T02:38:21.501 INFO:tasks.rados.rados.0.smithi107.stdout:5237: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:21.501 INFO:tasks.rados.rados.0.smithi107.stdout:5237: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:21.501 INFO:tasks.rados.rados.0.smithi107.stdout:5237: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:21.501 INFO:tasks.rados.rados.0.smithi107.stdout:5237: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:21.502 INFO:tasks.rados.rados.0.smithi107.stdout:5236: done (3 left) 2020-06-12T02:38:21.502 INFO:tasks.rados.rados.0.smithi107.stdout:5238: done (2 left) 2020-06-12T02:38:21.502 INFO:tasks.rados.rados.0.smithi107.stdout:5239: delete oid 17 current snap is 0 2020-06-12T02:38:21.502 INFO:tasks.rados.rados.0.smithi107.stdout:5237: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:21.502 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 973 (ObjNum 5236 snap 0 seq_num 5236) dirty exists 2020-06-12T02:38:21.502 INFO:tasks.rados.rados.0.smithi107.stdout:5237: left oid 4 (ObjNum 5236 snap 0 seq_num 5236) 2020-06-12T02:38:21.503 INFO:tasks.rados.rados.0.smithi107.stdout:5237: done (2 left) 2020-06-12T02:38:21.503 INFO:tasks.rados.rados.0.smithi107.stdout:5239: done (1 left) 2020-06-12T02:38:21.503 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:21.503 INFO:tasks.rados.rados.0.smithi107.stdout:5240: seq_num 5239 ranges {0=548864,548864=647168,1196032=155648} 2020-06-12T02:38:21.503 INFO:tasks.rados.rados.0.smithi107.stdout:5240: writing smithi10729407-15 from 0 to 548864 tid 1 2020-06-12T02:38:21.503 INFO:tasks.rados.rados.0.smithi107.stdout:5240: writing smithi10729407-15 from 548864 to 1196032 tid 2 2020-06-12T02:38:21.504 INFO:tasks.rados.rados.0.smithi107.stdout:5240: writing smithi10729407-15 from 1196032 to 1351680 tid 3 2020-06-12T02:38:21.504 INFO:tasks.rados.rados.0.smithi107.stdout:5241: delete oid 17 current snap is 0 2020-06-12T02:38:21.504 INFO:tasks.rados.rados.0.smithi107.stdout:5241: done (2 left) 2020-06-12T02:38:21.504 INFO:tasks.rados.rados.0.smithi107.stdout:5242: delete oid 5 current snap is 0 2020-06-12T02:38:21.504 INFO:tasks.rados.rados.0.smithi107.stdout:5240: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:21.505 INFO:tasks.rados.rados.0.smithi107.stdout:5240: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:21.505 INFO:tasks.rados.rados.0.smithi107.stdout:5242: done (2 left) 2020-06-12T02:38:21.505 INFO:tasks.rados.rados.0.smithi107.stdout:5243: delete oid 11 current snap is 0 2020-06-12T02:38:21.505 INFO:tasks.rados.rados.0.smithi107.stdout:5243: done (2 left) 2020-06-12T02:38:21.505 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:21.505 INFO:tasks.rados.rados.0.smithi107.stdout:5244: seq_num 5243 ranges {1163264=614400,1777664=589824,2367488=212992} 2020-06-12T02:38:21.506 INFO:tasks.rados.rados.0.smithi107.stdout:5244: writing smithi10729407-1 from 1163264 to 1777664 tid 1 2020-06-12T02:38:21.506 INFO:tasks.rados.rados.0.smithi107.stdout:5244: writing smithi10729407-1 from 1777664 to 2367488 tid 2 2020-06-12T02:38:21.506 INFO:tasks.rados.rados.0.smithi107.stdout:5244: writing smithi10729407-1 from 2367488 to 2580480 tid 3 2020-06-12T02:38:21.506 INFO:tasks.rados.rados.0.smithi107.stdout:5245: delete oid 7 current snap is 0 2020-06-12T02:38:21.506 INFO:tasks.rados.rados.0.smithi107.stdout:5240: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:21.507 INFO:tasks.rados.rados.0.smithi107.stdout:5240: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:21.507 INFO:tasks.rados.rados.0.smithi107.stdout:5240: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:21.507 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 919 (ObjNum 5239 snap 0 seq_num 5239) dirty exists 2020-06-12T02:38:21.507 INFO:tasks.rados.rados.0.smithi107.stdout:5240: left oid 15 (ObjNum 5239 snap 0 seq_num 5239) 2020-06-12T02:38:21.507 INFO:tasks.rados.rados.0.smithi107.stdout:5244: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:21.507 INFO:tasks.rados.rados.0.smithi107.stdout:5240: done (3 left) 2020-06-12T02:38:21.508 INFO:tasks.rados.rados.0.smithi107.stdout:5245: done (2 left) 2020-06-12T02:38:21.508 INFO:tasks.rados.rados.0.smithi107.stdout:5246: delete oid 5 current snap is 0 2020-06-12T02:38:21.508 INFO:tasks.rados.rados.0.smithi107.stdout:5246: done (2 left) 2020-06-12T02:38:21.508 INFO:tasks.rados.rados.0.smithi107.stdout:5247: delete oid 16 current snap is 0 2020-06-12T02:38:21.508 INFO:tasks.rados.rados.0.smithi107.stdout:5244: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:21.509 INFO:tasks.rados.rados.0.smithi107.stdout:5247: done (2 left) 2020-06-12T02:38:21.509 INFO:tasks.rados.rados.0.smithi107.stdout:5248: delete oid 14 current snap is 0 2020-06-12T02:38:21.509 INFO:tasks.rados.rados.0.smithi107.stdout:5244: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:21.509 INFO:tasks.rados.rados.0.smithi107.stdout:5244: finishing write tid 5 to smithi10729407-1 2020-06-12T02:38:21.510 INFO:tasks.rados.rados.0.smithi107.stdout:5248: done (2 left) 2020-06-12T02:38:21.510 INFO:tasks.rados.rados.0.smithi107.stdout:5249: delete oid 4 current snap is 0 2020-06-12T02:38:21.510 INFO:tasks.rados.rados.0.smithi107.stdout:5244: finishing write tid 6 to smithi10729407-1 2020-06-12T02:38:21.510 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 966 (ObjNum 5243 snap 0 seq_num 5243) dirty exists 2020-06-12T02:38:21.510 INFO:tasks.rados.rados.0.smithi107.stdout:5244: left oid 1 (ObjNum 5243 snap 0 seq_num 5243) 2020-06-12T02:38:21.510 INFO:tasks.rados.rados.0.smithi107.stdout:5244: done (2 left) 2020-06-12T02:38:21.511 INFO:tasks.rados.rados.0.smithi107.stdout:5249: done (1 left) 2020-06-12T02:38:21.511 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:21.511 INFO:tasks.rados.rados.0.smithi107.stdout:5250: seq_num 5249 ranges {0=720896,720896=745472,1466368=57344} 2020-06-12T02:38:21.511 INFO:tasks.rados.rados.0.smithi107.stdout:5250: writing smithi10729407-17 from 0 to 720896 tid 1 2020-06-12T02:38:21.511 INFO:tasks.rados.rados.0.smithi107.stdout:5250: writing smithi10729407-17 from 720896 to 1466368 tid 2 2020-06-12T02:38:21.511 INFO:tasks.rados.rados.0.smithi107.stdout:5250: writing smithi10729407-17 from 1466368 to 1523712 tid 3 2020-06-12T02:38:21.512 INFO:tasks.rados.rados.0.smithi107.stdout:5251: delete oid 1 current snap is 0 2020-06-12T02:38:21.512 INFO:tasks.rados.rados.0.smithi107.stdout:5250: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:21.512 INFO:tasks.rados.rados.0.smithi107.stdout:5251: done (2 left) 2020-06-12T02:38:21.512 INFO:tasks.rados.rados.0.smithi107.stdout:5252: delete oid 18 current snap is 0 2020-06-12T02:38:21.512 INFO:tasks.rados.rados.0.smithi107.stdout:5250: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:21.513 INFO:tasks.rados.rados.0.smithi107.stdout:5252: done (2 left) 2020-06-12T02:38:21.513 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:38:21.513 INFO:tasks.rados.rados.0.smithi107.stdout:5253: seq_num 5252 ranges {0=770048,770048=794624,1564672=8192} 2020-06-12T02:38:21.513 INFO:tasks.rados.rados.0.smithi107.stdout:5253: writing smithi10729407-7 from 0 to 770048 tid 1 2020-06-12T02:38:21.513 INFO:tasks.rados.rados.0.smithi107.stdout:5253: writing smithi10729407-7 from 770048 to 1564672 tid 2 2020-06-12T02:38:21.513 INFO:tasks.rados.rados.0.smithi107.stdout:5253: writing smithi10729407-7 from 1564672 to 1572864 tid 3 2020-06-12T02:38:21.514 INFO:tasks.rados.rados.0.smithi107.stdout:5254: delete oid 3 current snap is 0 2020-06-12T02:38:21.514 INFO:tasks.rados.rados.0.smithi107.stdout:5250: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:21.514 INFO:tasks.rados.rados.0.smithi107.stdout:5250: finishing write tid 5 to smithi10729407-17 2020-06-12T02:38:21.514 INFO:tasks.rados.rados.0.smithi107.stdout:5250: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:21.514 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 980 (ObjNum 5249 snap 0 seq_num 5249) dirty exists 2020-06-12T02:38:21.515 INFO:tasks.rados.rados.0.smithi107.stdout:5250: left oid 17 (ObjNum 5249 snap 0 seq_num 5249) 2020-06-12T02:38:21.515 INFO:tasks.rados.rados.0.smithi107.stdout:5253: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:21.515 INFO:tasks.rados.rados.0.smithi107.stdout:5250: done (3 left) 2020-06-12T02:38:21.515 INFO:tasks.rados.rados.0.smithi107.stdout:5254: done (2 left) 2020-06-12T02:38:21.515 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:21.515 INFO:tasks.rados.rados.0.smithi107.stdout:5255: seq_num 5254 ranges {1286144=802816,2088960=614400,2703360=188416} 2020-06-12T02:38:21.516 INFO:tasks.rados.rados.0.smithi107.stdout:5255: writing smithi10729407-6 from 1286144 to 2088960 tid 1 2020-06-12T02:38:21.516 INFO:tasks.rados.rados.0.smithi107.stdout:5255: writing smithi10729407-6 from 2088960 to 2703360 tid 2 2020-06-12T02:38:21.516 INFO:tasks.rados.rados.0.smithi107.stdout:5255: writing smithi10729407-6 from 2703360 to 2891776 tid 3 2020-06-12T02:38:21.516 INFO:tasks.rados.rados.0.smithi107.stdout:5256: delete oid 2 current snap is 0 2020-06-12T02:38:21.516 INFO:tasks.rados.rados.0.smithi107.stdout:5253: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:21.517 INFO:tasks.rados.rados.0.smithi107.stdout:5253: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:21.517 INFO:tasks.rados.rados.0.smithi107.stdout:5253: finishing write tid 5 to smithi10729407-7 2020-06-12T02:38:21.517 INFO:tasks.rados.rados.0.smithi107.stdout:5253: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:21.517 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 995 (ObjNum 5252 snap 0 seq_num 5252) dirty exists 2020-06-12T02:38:21.518 INFO:tasks.rados.rados.0.smithi107.stdout:5253: left oid 7 (ObjNum 5252 snap 0 seq_num 5252) 2020-06-12T02:38:21.518 INFO:tasks.rados.rados.0.smithi107.stdout:5253: done (3 left) 2020-06-12T02:38:21.518 INFO:tasks.rados.rados.0.smithi107.stdout:5256: done (2 left) 2020-06-12T02:38:21.518 INFO:tasks.rados.rados.0.smithi107.stdout:5257: delete oid 4 current snap is 0 2020-06-12T02:38:21.518 INFO:tasks.rados.rados.0.smithi107.stdout:5255: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:21.519 INFO:tasks.rados.rados.0.smithi107.stdout:5257: done (2 left) 2020-06-12T02:38:21.519 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:21.519 INFO:tasks.rados.rados.0.smithi107.stdout:5258: seq_num 5257 ranges {1064960=450560,1515520=671744,2187264=532480} 2020-06-12T02:38:21.519 INFO:tasks.rados.rados.0.smithi107.stdout:5258: writing smithi10729407-12 from 1064960 to 1515520 tid 1 2020-06-12T02:38:21.520 INFO:tasks.rados.rados.0.smithi107.stdout:5258: writing smithi10729407-12 from 1515520 to 2187264 tid 2 2020-06-12T02:38:21.520 INFO:tasks.rados.rados.0.smithi107.stdout:5258: writing smithi10729407-12 from 2187264 to 2719744 tid 3 2020-06-12T02:38:21.520 INFO:tasks.rados.rados.0.smithi107.stdout:5259: delete oid 9 current snap is 0 2020-06-12T02:38:21.520 INFO:tasks.rados.rados.0.smithi107.stdout:5255: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:21.520 INFO:tasks.rados.rados.0.smithi107.stdout:5255: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:21.520 INFO:tasks.rados.rados.0.smithi107.stdout:5255: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:21.521 INFO:tasks.rados.rados.0.smithi107.stdout:5255: finishing write tid 6 to smithi10729407-6 2020-06-12T02:38:21.521 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 971 (ObjNum 5254 snap 0 seq_num 5254) dirty exists 2020-06-12T02:38:21.521 INFO:tasks.rados.rados.0.smithi107.stdout:5255: left oid 6 (ObjNum 5254 snap 0 seq_num 5254) 2020-06-12T02:38:21.521 INFO:tasks.rados.rados.0.smithi107.stdout:5258: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:21.521 INFO:tasks.rados.rados.0.smithi107.stdout:5255: done (3 left) 2020-06-12T02:38:21.522 INFO:tasks.rados.rados.0.smithi107.stdout:5259: done (2 left) 2020-06-12T02:38:21.522 INFO:tasks.rados.rados.0.smithi107.stdout:5260: delete oid 3 current snap is 0 2020-06-12T02:38:21.528 INFO:tasks.rados.rados.0.smithi107.stdout:5260: done (2 left) 2020-06-12T02:38:21.528 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:21.529 INFO:tasks.rados.rados.0.smithi107.stdout:5261: seq_num 5260 ranges {1523712=499712,2023424=720896,2744320=483328} 2020-06-12T02:38:21.529 INFO:tasks.rados.rados.0.smithi107.stdout:5261: writing smithi10729407-17 from 1523712 to 2023424 tid 1 2020-06-12T02:38:21.529 INFO:tasks.rados.rados.0.smithi107.stdout:5261: writing smithi10729407-17 from 2023424 to 2744320 tid 2 2020-06-12T02:38:21.530 INFO:tasks.rados.rados.0.smithi107.stdout:5261: writing smithi10729407-17 from 2744320 to 3227648 tid 3 2020-06-12T02:38:21.530 INFO:tasks.rados.rados.0.smithi107.stdout:5262: delete oid 3 current snap is 0 2020-06-12T02:38:21.530 INFO:tasks.rados.rados.0.smithi107.stdout:5258: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:21.530 INFO:tasks.rados.rados.0.smithi107.stdout:5258: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:21.530 INFO:tasks.rados.rados.0.smithi107.stdout:5258: finishing write tid 5 to smithi10729407-12 2020-06-12T02:38:21.530 INFO:tasks.rados.rados.0.smithi107.stdout:5258: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:21.531 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 922 (ObjNum 5257 snap 0 seq_num 5257) dirty exists 2020-06-12T02:38:21.531 INFO:tasks.rados.rados.0.smithi107.stdout:5258: left oid 12 (ObjNum 5257 snap 0 seq_num 5257) 2020-06-12T02:38:21.531 INFO:tasks.rados.rados.0.smithi107.stdout:5258: done (3 left) 2020-06-12T02:38:21.531 INFO:tasks.rados.rados.0.smithi107.stdout:5262: done (2 left) 2020-06-12T02:38:21.531 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:21.532 INFO:tasks.rados.rados.0.smithi107.stdout:5263: seq_num 5262 ranges {0=532480,532480=540672,1073152=499712,1572864=163840} 2020-06-12T02:38:21.532 INFO:tasks.rados.rados.0.smithi107.stdout:5263: writing smithi10729407-5 from 0 to 532480 tid 1 2020-06-12T02:38:21.532 INFO:tasks.rados.rados.0.smithi107.stdout:5263: writing smithi10729407-5 from 532480 to 1073152 tid 2 2020-06-12T02:38:21.532 INFO:tasks.rados.rados.0.smithi107.stdout:5263: writing smithi10729407-5 from 1073152 to 1572864 tid 3 2020-06-12T02:38:21.532 INFO:tasks.rados.rados.0.smithi107.stdout:5263: writing smithi10729407-5 from 1572864 to 1736704 tid 4 2020-06-12T02:38:21.532 INFO:tasks.rados.rados.0.smithi107.stdout:5264: delete oid 19 current snap is 0 2020-06-12T02:38:21.533 INFO:tasks.rados.rados.0.smithi107.stdout:5261: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:21.533 INFO:tasks.rados.rados.0.smithi107.stdout:5261: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:21.533 INFO:tasks.rados.rados.0.smithi107.stdout:5261: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:21.533 INFO:tasks.rados.rados.0.smithi107.stdout:5261: finishing write tid 5 to smithi10729407-17 2020-06-12T02:38:21.533 INFO:tasks.rados.rados.0.smithi107.stdout:5261: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:21.534 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 985 (ObjNum 5260 snap 0 seq_num 5260) dirty exists 2020-06-12T02:38:21.534 INFO:tasks.rados.rados.0.smithi107.stdout:5261: left oid 17 (ObjNum 5260 snap 0 seq_num 5260) 2020-06-12T02:38:21.534 INFO:tasks.rados.rados.0.smithi107.stdout:5263: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:21.534 INFO:tasks.rados.rados.0.smithi107.stdout:5261: done (3 left) 2020-06-12T02:38:21.534 INFO:tasks.rados.rados.0.smithi107.stdout:5264: done (2 left) 2020-06-12T02:38:21.534 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:21.535 INFO:tasks.rados.rados.0.smithi107.stdout:5265: seq_num 5264 ranges {0=565248,565248=704512,1269760=499712} 2020-06-12T02:38:21.535 INFO:tasks.rados.rados.0.smithi107.stdout:5265: writing smithi10729407-11 from 0 to 565248 tid 1 2020-06-12T02:38:21.535 INFO:tasks.rados.rados.0.smithi107.stdout:5265: writing smithi10729407-11 from 565248 to 1269760 tid 2 2020-06-12T02:38:21.535 INFO:tasks.rados.rados.0.smithi107.stdout:5265: writing smithi10729407-11 from 1269760 to 1769472 tid 3 2020-06-12T02:38:21.535 INFO:tasks.rados.rados.0.smithi107.stdout:5266: delete oid 6 current snap is 0 2020-06-12T02:38:21.535 INFO:tasks.rados.rados.0.smithi107.stdout:5263: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:21.536 INFO:tasks.rados.rados.0.smithi107.stdout:5263: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:21.536 INFO:tasks.rados.rados.0.smithi107.stdout:5263: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:21.536 INFO:tasks.rados.rados.0.smithi107.stdout:5263: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:21.536 INFO:tasks.rados.rados.0.smithi107.stdout:5263: finishing write tid 7 to smithi10729407-5 2020-06-12T02:38:21.537 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 523 (ObjNum 5262 snap 0 seq_num 5262) dirty exists 2020-06-12T02:38:21.537 INFO:tasks.rados.rados.0.smithi107.stdout:5263: left oid 5 (ObjNum 5262 snap 0 seq_num 5262) 2020-06-12T02:38:21.537 INFO:tasks.rados.rados.0.smithi107.stdout:5265: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:21.537 INFO:tasks.rados.rados.0.smithi107.stdout:5265: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:21.537 INFO:tasks.rados.rados.0.smithi107.stdout:5263: done (3 left) 2020-06-12T02:38:21.538 INFO:tasks.rados.rados.0.smithi107.stdout:5266: done (2 left) 2020-06-12T02:38:21.538 INFO:tasks.rados.rados.0.smithi107.stdout:5267: delete oid 12 current snap is 0 2020-06-12T02:38:21.538 INFO:tasks.rados.rados.0.smithi107.stdout:5265: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:21.538 INFO:tasks.rados.rados.0.smithi107.stdout:5265: finishing write tid 5 to smithi10729407-11 2020-06-12T02:38:21.538 INFO:tasks.rados.rados.0.smithi107.stdout:5267: done (2 left) 2020-06-12T02:38:21.538 INFO:tasks.rados.rados.0.smithi107.stdout:5268: delete oid 8 current snap is 0 2020-06-12T02:38:21.539 INFO:tasks.rados.rados.0.smithi107.stdout:5265: finishing write tid 6 to smithi10729407-11 2020-06-12T02:38:21.539 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 952 (ObjNum 5264 snap 0 seq_num 5264) dirty exists 2020-06-12T02:38:21.539 INFO:tasks.rados.rados.0.smithi107.stdout:5265: left oid 11 (ObjNum 5264 snap 0 seq_num 5264) 2020-06-12T02:38:21.539 INFO:tasks.rados.rados.0.smithi107.stdout:5265: done (2 left) 2020-06-12T02:38:21.540 INFO:tasks.rados.rados.0.smithi107.stdout:5268: done (1 left) 2020-06-12T02:38:21.540 INFO:tasks.rados.rados.0.smithi107.stdout:5269: delete oid 5 current snap is 0 2020-06-12T02:38:21.540 INFO:tasks.rados.rados.0.smithi107.stdout:5269: done (1 left) 2020-06-12T02:38:21.540 INFO:tasks.rados.rados.0.smithi107.stdout:5270: delete oid 18 current snap is 0 2020-06-12T02:38:21.540 INFO:tasks.rados.rados.0.smithi107.stdout:5270: done (1 left) 2020-06-12T02:38:21.541 INFO:tasks.rados.rados.0.smithi107.stdout:5271: delete oid 15 current snap is 0 2020-06-12T02:38:21.541 INFO:tasks.rados.rados.0.smithi107.stdout:5271: done (1 left) 2020-06-12T02:38:21.541 INFO:tasks.rados.rados.0.smithi107.stdout:5272: delete oid 15 current snap is 0 2020-06-12T02:38:21.541 INFO:tasks.rados.rados.0.smithi107.stdout:5272: done (1 left) 2020-06-12T02:38:21.541 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:21.541 INFO:tasks.rados.rados.0.smithi107.stdout:5273: seq_num 5272 ranges {1294336=704512,1998848=630784,2629632=475136,3104768=98304} 2020-06-12T02:38:21.542 INFO:tasks.rados.rados.0.smithi107.stdout:5273: writing smithi10729407-13 from 1294336 to 1998848 tid 1 2020-06-12T02:38:21.542 INFO:tasks.rados.rados.0.smithi107.stdout:5273: writing smithi10729407-13 from 1998848 to 2629632 tid 2 2020-06-12T02:38:21.542 INFO:tasks.rados.rados.0.smithi107.stdout:5273: writing smithi10729407-13 from 2629632 to 3104768 tid 3 2020-06-12T02:38:21.542 INFO:tasks.rados.rados.0.smithi107.stdout:5273: writing smithi10729407-13 from 3104768 to 3203072 tid 4 2020-06-12T02:38:21.542 INFO:tasks.rados.rados.0.smithi107.stdout:5274: delete oid 20 current snap is 0 2020-06-12T02:38:21.543 INFO:tasks.rados.rados.0.smithi107.stdout:5273: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:21.543 INFO:tasks.rados.rados.0.smithi107.stdout:5274: done (2 left) 2020-06-12T02:38:21.543 INFO:tasks.rados.rados.0.smithi107.stdout:5275: delete oid 5 current snap is 0 2020-06-12T02:38:21.543 INFO:tasks.rados.rados.0.smithi107.stdout:5275: done (2 left) 2020-06-12T02:38:21.543 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:21.543 INFO:tasks.rados.rados.0.smithi107.stdout:5276: seq_num 5275 ranges {0=753664,753664=688128,1441792=516096} 2020-06-12T02:38:21.544 INFO:tasks.rados.rados.0.smithi107.stdout:5276: writing smithi10729407-6 from 0 to 753664 tid 1 2020-06-12T02:38:21.544 INFO:tasks.rados.rados.0.smithi107.stdout:5276: writing smithi10729407-6 from 753664 to 1441792 tid 2 2020-06-12T02:38:21.544 INFO:tasks.rados.rados.0.smithi107.stdout:5276: writing smithi10729407-6 from 1441792 to 1957888 tid 3 2020-06-12T02:38:21.544 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:21.544 INFO:tasks.rados.rados.0.smithi107.stdout:5277: seq_num 5276 ranges {0=770048,770048=573440,1343488=450560,1794048=180224} 2020-06-12T02:38:21.544 INFO:tasks.rados.rados.0.smithi107.stdout:5277: writing smithi10729407-18 from 0 to 770048 tid 1 2020-06-12T02:38:21.545 INFO:tasks.rados.rados.0.smithi107.stdout:5277: writing smithi10729407-18 from 770048 to 1343488 tid 2 2020-06-12T02:38:21.545 INFO:tasks.rados.rados.0.smithi107.stdout:5277: writing smithi10729407-18 from 1343488 to 1794048 tid 3 2020-06-12T02:38:21.545 INFO:tasks.rados.rados.0.smithi107.stdout:5277: writing smithi10729407-18 from 1794048 to 1974272 tid 4 2020-06-12T02:38:21.545 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:21.545 INFO:tasks.rados.rados.0.smithi107.stdout:5278: seq_num 5277 ranges {0=786432,786432=499712,1286144=663552,1949696=40960} 2020-06-12T02:38:21.546 INFO:tasks.rados.rados.0.smithi107.stdout:5278: writing smithi10729407-14 from 0 to 786432 tid 1 2020-06-12T02:38:21.546 INFO:tasks.rados.rados.0.smithi107.stdout:5278: writing smithi10729407-14 from 786432 to 1286144 tid 2 2020-06-12T02:38:21.546 INFO:tasks.rados.rados.0.smithi107.stdout:5278: writing smithi10729407-14 from 1286144 to 1949696 tid 3 2020-06-12T02:38:21.546 INFO:tasks.rados.rados.0.smithi107.stdout:5278: writing smithi10729407-14 from 1949696 to 1990656 tid 4 2020-06-12T02:38:21.546 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:38:21.546 INFO:tasks.rados.rados.0.smithi107.stdout:5279: seq_num 5278 ranges {0=802816,802816=786432,1589248=417792} 2020-06-12T02:38:21.547 INFO:tasks.rados.rados.0.smithi107.stdout:5279: writing smithi10729407-20 from 0 to 802816 tid 1 2020-06-12T02:38:21.547 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-06-12T02:38:21.548 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2020-06-12T02:38:21.548 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:38:21.471+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:38:21.556 INFO:tasks.rados.rados.0.smithi107.stdout:5279: writing smithi10729407-20 from 802816 to 1589248 tid 2 2020-06-12T02:38:21.556 INFO:tasks.rados.rados.0.smithi107.stdout:5279: writing smithi10729407-20 from 1589248 to 2007040 tid 3 2020-06-12T02:38:21.556 INFO:tasks.rados.rados.0.smithi107.stdout:5280: delete oid 8 current snap is 0 2020-06-12T02:38:21.556 INFO:tasks.rados.rados.0.smithi107.stdout:5273: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:21.557 INFO:tasks.rados.rados.0.smithi107.stdout:5273: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:21.557 INFO:tasks.rados.rados.0.smithi107.stdout:5273: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:21.557 INFO:tasks.rados.rados.0.smithi107.stdout:5273: finishing write tid 6 to smithi10729407-13 2020-06-12T02:38:21.557 INFO:tasks.rados.rados.0.smithi107.stdout:5273: finishing write tid 7 to smithi10729407-13 2020-06-12T02:38:21.557 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1000 (ObjNum 5272 snap 0 seq_num 5272) dirty exists 2020-06-12T02:38:21.558 INFO:tasks.rados.rados.0.smithi107.stdout:5273: left oid 13 (ObjNum 5272 snap 0 seq_num 5272) 2020-06-12T02:38:21.558 INFO:tasks.rados.rados.0.smithi107.stdout:5276: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:21.558 INFO:tasks.rados.rados.0.smithi107.stdout:5276: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:21.558 INFO:tasks.rados.rados.0.smithi107.stdout:5276: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:21.558 INFO:tasks.rados.rados.0.smithi107.stdout:5276: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:21.559 INFO:tasks.rados.rados.0.smithi107.stdout:5277: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:21.559 INFO:tasks.rados.rados.0.smithi107.stdout:5276: finishing write tid 6 to smithi10729407-6 2020-06-12T02:38:21.559 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 976 (ObjNum 5275 snap 0 seq_num 5275) dirty exists 2020-06-12T02:38:21.559 INFO:tasks.rados.rados.0.smithi107.stdout:5276: left oid 6 (ObjNum 5275 snap 0 seq_num 5275) 2020-06-12T02:38:21.559 INFO:tasks.rados.rados.0.smithi107.stdout:5277: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:21.559 INFO:tasks.rados.rados.0.smithi107.stdout:5277: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:21.560 INFO:tasks.rados.rados.0.smithi107.stdout:5277: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:21.560 INFO:tasks.rados.rados.0.smithi107.stdout:5277: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:21.560 INFO:tasks.rados.rados.0.smithi107.stdout:5277: finishing write tid 7 to smithi10729407-18 2020-06-12T02:38:21.560 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 955 (ObjNum 5276 snap 0 seq_num 5276) dirty exists 2020-06-12T02:38:21.560 INFO:tasks.rados.rados.0.smithi107.stdout:5277: left oid 18 (ObjNum 5276 snap 0 seq_num 5276) 2020-06-12T02:38:21.561 INFO:tasks.rados.rados.0.smithi107.stdout:5278: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:21.561 INFO:tasks.rados.rados.0.smithi107.stdout:5278: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:21.561 INFO:tasks.rados.rados.0.smithi107.stdout:5278: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:21.561 INFO:tasks.rados.rados.0.smithi107.stdout:5278: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:21.561 INFO:tasks.rados.rados.0.smithi107.stdout:5278: finishing write tid 6 to smithi10729407-14 2020-06-12T02:38:21.562 INFO:tasks.rados.rados.0.smithi107.stdout:5273: done (6 left) 2020-06-12T02:38:21.562 INFO:tasks.rados.rados.0.smithi107.stdout:5276: done (5 left) 2020-06-12T02:38:21.562 INFO:tasks.rados.rados.0.smithi107.stdout:5277: done (4 left) 2020-06-12T02:38:21.562 INFO:tasks.rados.rados.0.smithi107.stdout:5280: done (3 left) 2020-06-12T02:38:21.562 INFO:tasks.rados.rados.0.smithi107.stdout:5281: delete oid 6 current snap is 0 2020-06-12T02:38:21.562 INFO:tasks.rados.rados.0.smithi107.stdout:5279: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:21.563 INFO:tasks.rados.rados.0.smithi107.stdout:5278: finishing write tid 7 to smithi10729407-14 2020-06-12T02:38:21.563 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 957 (ObjNum 5277 snap 0 seq_num 5277) dirty exists 2020-06-12T02:38:21.563 INFO:tasks.rados.rados.0.smithi107.stdout:5278: left oid 14 (ObjNum 5277 snap 0 seq_num 5277) 2020-06-12T02:38:21.563 INFO:tasks.rados.rados.0.smithi107.stdout:5279: finishing write tid 2 to smithi10729407-20 2020-06-12T02:38:21.563 INFO:tasks.rados.rados.0.smithi107.stdout:5278: done (3 left) 2020-06-12T02:38:21.564 INFO:tasks.rados.rados.0.smithi107.stdout:5281: done (2 left) 2020-06-12T02:38:21.564 INFO:tasks.rados.rados.0.smithi107.stdout:5282: delete oid 2 current snap is 0 2020-06-12T02:38:21.564 INFO:tasks.rados.rados.0.smithi107.stdout:5279: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:21.564 INFO:tasks.rados.rados.0.smithi107.stdout:5279: finishing write tid 5 to smithi10729407-20 2020-06-12T02:38:21.564 INFO:tasks.rados.rados.0.smithi107.stdout:5279: finishing write tid 6 to smithi10729407-20 2020-06-12T02:38:21.565 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 398 (ObjNum 5278 snap 0 seq_num 5278) dirty exists 2020-06-12T02:38:21.565 INFO:tasks.rados.rados.0.smithi107.stdout:5279: left oid 20 (ObjNum 5278 snap 0 seq_num 5278) 2020-06-12T02:38:21.565 INFO:tasks.rados.rados.0.smithi107.stdout:5279: done (2 left) 2020-06-12T02:38:21.565 INFO:tasks.rados.rados.0.smithi107.stdout:5282: done (1 left) 2020-06-12T02:38:21.565 INFO:tasks.rados.rados.0.smithi107.stdout:5283: delete oid 8 current snap is 0 2020-06-12T02:38:21.566 INFO:tasks.rados.rados.0.smithi107.stdout:5283: done (1 left) 2020-06-12T02:38:21.566 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:38:21.566 INFO:tasks.rados.rados.0.smithi107.stdout:5284: seq_num 5283 ranges {3203072=483328,3686400=606208,4292608=606208,4898816=393216} 2020-06-12T02:38:21.566 INFO:tasks.rados.rados.0.smithi107.stdout:5284: writing smithi10729407-13 from 3203072 to 3686400 tid 1 2020-06-12T02:38:21.566 INFO:tasks.rados.rados.0.smithi107.stdout:5284: writing smithi10729407-13 from 3686400 to 4292608 tid 2 2020-06-12T02:38:21.567 INFO:tasks.rados.rados.0.smithi107.stdout:5284: writing smithi10729407-13 from 4292608 to 4898816 tid 3 2020-06-12T02:38:21.567 INFO:tasks.rados.rados.0.smithi107.stdout:5284: writing smithi10729407-13 from 4898816 to 5292032 tid 4 2020-06-12T02:38:21.567 INFO:tasks.rados.rados.0.smithi107.stdout:5285: delete oid 19 current snap is 0 2020-06-12T02:38:21.567 INFO:tasks.rados.rados.0.smithi107.stdout:5285: done (2 left) 2020-06-12T02:38:21.567 INFO:tasks.rados.rados.0.smithi107.stdout:5286: delete oid 19 current snap is 0 2020-06-12T02:38:21.568 INFO:tasks.rados.rados.0.smithi107.stdout:5284: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:21.568 INFO:tasks.rados.rados.0.smithi107.stdout:5284: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:21.568 INFO:tasks.rados.rados.0.smithi107.stdout:5286: done (2 left) 2020-06-12T02:38:21.568 INFO:tasks.rados.rados.0.smithi107.stdout:5287: delete oid 8 current snap is 0 2020-06-12T02:38:21.568 INFO:tasks.rados.rados.0.smithi107.stdout:5284: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:21.569 INFO:tasks.rados.rados.0.smithi107.stdout:5287: done (2 left) 2020-06-12T02:38:21.569 INFO:tasks.rados.rados.0.smithi107.stdout:5288: delete oid 8 current snap is 0 2020-06-12T02:38:21.569 INFO:tasks.rados.rados.0.smithi107.stdout:5284: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:21.569 INFO:tasks.rados.rados.0.smithi107.stdout:5284: finishing write tid 6 to smithi10729407-13 2020-06-12T02:38:21.569 INFO:tasks.rados.rados.0.smithi107.stdout:5284: finishing write tid 7 to smithi10729407-13 2020-06-12T02:38:21.570 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1005 (ObjNum 5283 snap 0 seq_num 5283) dirty exists 2020-06-12T02:38:21.570 INFO:tasks.rados.rados.0.smithi107.stdout:5284: left oid 13 (ObjNum 5283 snap 0 seq_num 5283) 2020-06-12T02:38:21.570 INFO:tasks.rados.rados.0.smithi107.stdout:5284: done (2 left) 2020-06-12T02:38:21.570 INFO:tasks.rados.rados.0.smithi107.stdout:5288: done (1 left) 2020-06-12T02:38:21.570 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:21.571 INFO:tasks.rados.rados.0.smithi107.stdout:5289: seq_num 5288 ranges {0=573440,573440=483328,1056768=794624,1851392=327680} 2020-06-12T02:38:21.571 INFO:tasks.rados.rados.0.smithi107.stdout:5289: writing smithi10729407-1 from 0 to 573440 tid 1 2020-06-12T02:38:21.571 INFO:tasks.rados.rados.0.smithi107.stdout:5289: writing smithi10729407-1 from 573440 to 1056768 tid 2 2020-06-12T02:38:21.573 INFO:tasks.rados.rados.0.smithi107.stdout:5289: writing smithi10729407-1 from 1056768 to 1851392 tid 3 2020-06-12T02:38:21.573 INFO:tasks.rados.rados.0.smithi107.stdout:5289: writing smithi10729407-1 from 1851392 to 2179072 tid 4 2020-06-12T02:38:21.573 INFO:tasks.rados.rados.0.smithi107.stdout:5290: delete oid 20 current snap is 0 2020-06-12T02:38:21.573 INFO:tasks.rados.rados.0.smithi107.stdout:5289: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:21.574 INFO:tasks.rados.rados.0.smithi107.stdout:5290: done (2 left) 2020-06-12T02:38:21.574 INFO:tasks.rados.rados.0.smithi107.stdout:5291: delete oid 20 current snap is 0 2020-06-12T02:38:21.574 INFO:tasks.rados.rados.0.smithi107.stdout:5289: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:21.574 INFO:tasks.rados.rados.0.smithi107.stdout:5291: done (2 left) 2020-06-12T02:38:21.574 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:21.574 INFO:tasks.rados.rados.0.smithi107.stdout:5292: seq_num 5291 ranges {0=622592,622592=532480,1155072=565248,1720320=507904} 2020-06-12T02:38:21.576 INFO:tasks.rados.rados.0.smithi107.stdout:5292: writing smithi10729407-4 from 0 to 622592 tid 1 2020-06-12T02:38:21.579 INFO:tasks.rados.rados.0.smithi107.stdout:5292: writing smithi10729407-4 from 622592 to 1155072 tid 2 2020-06-12T02:38:21.583 INFO:tasks.rados.rados.0.smithi107.stdout:5292: writing smithi10729407-4 from 1155072 to 1720320 tid 3 2020-06-12T02:38:21.586 INFO:tasks.rados.rados.0.smithi107.stdout:5292: writing smithi10729407-4 from 1720320 to 2228224 tid 4 2020-06-12T02:38:21.586 INFO:tasks.rados.rados.0.smithi107.stdout:5293: delete oid 9 current snap is 0 2020-06-12T02:38:21.587 INFO:tasks.rados.rados.0.smithi107.stdout:5289: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:21.587 INFO:tasks.rados.rados.0.smithi107.stdout:5289: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:21.587 INFO:tasks.rados.rados.0.smithi107.stdout:5289: finishing write tid 6 to smithi10729407-1 2020-06-12T02:38:21.588 INFO:tasks.rados.rados.0.smithi107.stdout:5289: finishing write tid 7 to smithi10729407-1 2020-06-12T02:38:21.588 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 982 (ObjNum 5288 snap 0 seq_num 5288) dirty exists 2020-06-12T02:38:21.588 INFO:tasks.rados.rados.0.smithi107.stdout:5289: left oid 1 (ObjNum 5288 snap 0 seq_num 5288) 2020-06-12T02:38:21.588 INFO:tasks.rados.rados.0.smithi107.stdout:5292: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:21.594 INFO:tasks.rados.rados.0.smithi107.stdout:5292: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:21.594 INFO:tasks.rados.rados.0.smithi107.stdout:5292: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:21.594 INFO:tasks.rados.rados.0.smithi107.stdout:5289: done (3 left) 2020-06-12T02:38:21.594 INFO:tasks.rados.rados.0.smithi107.stdout:5293: done (2 left) 2020-06-12T02:38:21.595 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:21.595 INFO:tasks.rados.rados.0.smithi107.stdout:5294: seq_num 5293 ranges {1974272=655360,2629632=753664,3383296=581632,3964928=270336} 2020-06-12T02:38:21.607 INFO:tasks.rados.rados.0.smithi107.stdout:5294: writing smithi10729407-18 from 1974272 to 2629632 tid 1 2020-06-12T02:38:21.612 INFO:tasks.rados.rados.0.smithi107.stdout:5294: writing smithi10729407-18 from 2629632 to 3383296 tid 2 2020-06-12T02:38:21.616 INFO:tasks.rados.rados.0.smithi107.stdout:5294: writing smithi10729407-18 from 3383296 to 3964928 tid 3 2020-06-12T02:38:21.617 INFO:tasks.rados.rados.0.smithi107.stdout:5294: writing smithi10729407-18 from 3964928 to 4235264 tid 4 2020-06-12T02:38:21.617 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:21.618 INFO:tasks.rados.rados.0.smithi107.stdout:5295: seq_num 5294 ranges {0=671744,671744=638976,1310720=786432,2097152=180224} 2020-06-12T02:38:21.618 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:38:21.615+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:38:21.622 INFO:tasks.rados.rados.0.smithi107.stdout:5295: writing smithi10729407-8 from 0 to 671744 tid 1 2020-06-12T02:38:21.626 INFO:tasks.rados.rados.0.smithi107.stdout:5295: writing smithi10729407-8 from 671744 to 1310720 tid 2 2020-06-12T02:38:21.631 INFO:tasks.rados.rados.0.smithi107.stdout:5295: writing smithi10729407-8 from 1310720 to 2097152 tid 3 2020-06-12T02:38:21.632 INFO:tasks.rados.rados.0.smithi107.stdout:5295: writing smithi10729407-8 from 2097152 to 2277376 tid 4 2020-06-12T02:38:21.633 INFO:tasks.rados.rados.0.smithi107.stdout:5296: delete oid 14 current snap is 0 2020-06-12T02:38:21.633 INFO:tasks.rados.rados.0.smithi107.stdout:5292: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:21.633 INFO:tasks.rados.rados.0.smithi107.stdout:5292: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:21.633 INFO:tasks.rados.rados.0.smithi107.stdout:5292: finishing write tid 7 to smithi10729407-4 2020-06-12T02:38:21.633 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 990 (ObjNum 5291 snap 0 seq_num 5291) dirty exists 2020-06-12T02:38:21.634 INFO:tasks.rados.rados.0.smithi107.stdout:5292: left oid 4 (ObjNum 5291 snap 0 seq_num 5291) 2020-06-12T02:38:21.634 INFO:tasks.rados.rados.0.smithi107.stdout:5294: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:21.634 INFO:tasks.rados.rados.0.smithi107.stdout:5294: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:21.635 INFO:tasks.rados.rados.0.smithi107.stdout:5295: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:21.635 INFO:tasks.rados.rados.0.smithi107.stdout:5294: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:21.635 INFO:tasks.rados.rados.0.smithi107.stdout:5294: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:21.635 INFO:tasks.rados.rados.0.smithi107.stdout:5294: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:21.636 INFO:tasks.rados.rados.0.smithi107.stdout:5294: finishing write tid 7 to smithi10729407-18 2020-06-12T02:38:21.636 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 961 (ObjNum 5293 snap 0 seq_num 5293) dirty exists 2020-06-12T02:38:21.636 INFO:tasks.rados.rados.0.smithi107.stdout:5294: left oid 18 (ObjNum 5293 snap 0 seq_num 5293) 2020-06-12T02:38:21.637 INFO:tasks.rados.rados.0.smithi107.stdout:5292: done (4 left) 2020-06-12T02:38:21.637 INFO:tasks.rados.rados.0.smithi107.stdout:5294: done (3 left) 2020-06-12T02:38:21.637 INFO:tasks.rados.rados.0.smithi107.stdout:5296: done (2 left) 2020-06-12T02:38:21.637 INFO:tasks.rados.rados.0.smithi107.stdout:5297: delete oid 9 current snap is 0 2020-06-12T02:38:21.639 INFO:tasks.rados.rados.0.smithi107.stdout:5295: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:21.640 INFO:tasks.rados.rados.0.smithi107.stdout:5297: done (2 left) 2020-06-12T02:38:21.640 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:21.640 INFO:tasks.rados.rados.0.smithi107.stdout:5298: seq_num 5297 ranges {0=720896,720896=688128,1409024=565248,1974272=352256} 2020-06-12T02:38:21.644 INFO:tasks.rados.rados.0.smithi107.stdout:5298: writing smithi10729407-15 from 0 to 720896 tid 1 2020-06-12T02:38:21.648 INFO:tasks.rados.rados.0.smithi107.stdout:5298: writing smithi10729407-15 from 720896 to 1409024 tid 2 2020-06-12T02:38:21.651 INFO:tasks.rados.rados.0.smithi107.stdout:5298: writing smithi10729407-15 from 1409024 to 1974272 tid 3 2020-06-12T02:38:21.655 INFO:tasks.rados.rados.0.smithi107.stdout:5298: writing smithi10729407-15 from 1974272 to 2326528 tid 4 2020-06-12T02:38:21.655 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:38:21.655 INFO:tasks.rados.rados.0.smithi107.stdout:5299: seq_num 5298 ranges {0=737280,737280=573440,1310720=770048,2080768=262144} 2020-06-12T02:38:21.657 INFO:tasks.rados.rados.0.smithi107.stdout:5299: writing smithi10729407-20 from 0 to 737280 tid 1 2020-06-12T02:38:21.661 INFO:tasks.rados.rados.0.smithi107.stdout:5299: writing smithi10729407-20 from 737280 to 1310720 tid 2 2020-06-12T02:38:21.665 INFO:tasks.rados.rados.0.smithi107.stdout:5299: writing smithi10729407-20 from 1310720 to 2080768 tid 3 2020-06-12T02:38:21.668 INFO:tasks.rados.rados.0.smithi107.stdout:5299: writing smithi10729407-20 from 2080768 to 2342912 tid 4 2020-06-12T02:38:21.670 INFO:tasks.rados.rados.0.smithi107.stdout:5300: delete oid 1 current snap is 0 2020-06-12T02:38:21.670 INFO:tasks.rados.rados.0.smithi107.stdout:5295: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:21.670 INFO:tasks.rados.rados.0.smithi107.stdout:5295: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:21.670 INFO:tasks.rados.rados.0.smithi107.stdout:5295: finishing write tid 6 to smithi10729407-8 2020-06-12T02:38:21.672 INFO:tasks.rados.rados.0.smithi107.stdout:5295: finishing write tid 7 to smithi10729407-8 2020-06-12T02:38:21.672 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 931 (ObjNum 5294 snap 0 seq_num 5294) dirty exists 2020-06-12T02:38:21.672 INFO:tasks.rados.rados.0.smithi107.stdout:5295: left oid 8 (ObjNum 5294 snap 0 seq_num 5294) 2020-06-12T02:38:21.672 INFO:tasks.rados.rados.0.smithi107.stdout:5298: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:21.673 INFO:tasks.rados.rados.0.smithi107.stdout:5298: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:21.674 INFO:tasks.rados.rados.0.smithi107.stdout:5298: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:21.674 INFO:tasks.rados.rados.0.smithi107.stdout:5299: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:21.674 INFO:tasks.rados.rados.0.smithi107.stdout:5295: done (4 left) 2020-06-12T02:38:21.674 INFO:tasks.rados.rados.0.smithi107.stdout:5300: done (3 left) 2020-06-12T02:38:21.674 INFO:tasks.rados.rados.0.smithi107.stdout:5301: delete oid 8 current snap is 0 2020-06-12T02:38:21.675 INFO:tasks.rados.rados.0.smithi107.stdout:5298: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:21.675 INFO:tasks.rados.rados.0.smithi107.stdout:5298: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:21.677 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:21.675+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:21.681 INFO:tasks.rados.rados.0.smithi107.stdout:5298: finishing write tid 7 to smithi10729407-15 2020-06-12T02:38:21.682 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 936 (ObjNum 5297 snap 0 seq_num 5297) dirty exists 2020-06-12T02:38:21.682 INFO:tasks.rados.rados.0.smithi107.stdout:5298: left oid 15 (ObjNum 5297 snap 0 seq_num 5297) 2020-06-12T02:38:21.685 INFO:tasks.rados.rados.0.smithi107.stdout:5299: finishing write tid 2 to smithi10729407-20 2020-06-12T02:38:21.686 INFO:tasks.rados.rados.0.smithi107.stdout:5298: done (3 left) 2020-06-12T02:38:21.686 INFO:tasks.rados.rados.0.smithi107.stdout:5301: done (2 left) 2020-06-12T02:38:21.686 INFO:tasks.rados.rados.0.smithi107.stdout:5302: delete oid 7 current snap is 0 2020-06-12T02:38:21.686 INFO:tasks.rados.rados.0.smithi107.stdout:5299: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:21.689 INFO:tasks.rados.rados.0.smithi107.stdout:5299: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:21.691 INFO:tasks.rados.rados.0.smithi107.stdout:5299: finishing write tid 6 to smithi10729407-20 2020-06-12T02:38:21.692 INFO:tasks.rados.rados.0.smithi107.stdout:5299: finishing write tid 7 to smithi10729407-20 2020-06-12T02:38:21.692 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 405 (ObjNum 5298 snap 0 seq_num 5298) dirty exists 2020-06-12T02:38:21.693 INFO:tasks.rados.rados.0.smithi107.stdout:5299: left oid 20 (ObjNum 5298 snap 0 seq_num 5298) 2020-06-12T02:38:21.696 INFO:tasks.rados.rados.0.smithi107.stdout:5299: done (2 left) 2020-06-12T02:38:21.696 INFO:tasks.rados.rados.0.smithi107.stdout:5302: done (1 left) 2020-06-12T02:38:21.696 INFO:tasks.rados.rados.0.smithi107.stdout:5303: delete oid 19 current snap is 0 2020-06-12T02:38:21.699 INFO:tasks.rados.rados.0.smithi107.stdout:5303: done (1 left) 2020-06-12T02:38:21.700 INFO:tasks.rados.rados.0.smithi107.stdout:5304: delete oid 14 current snap is 0 2020-06-12T02:38:21.702 INFO:tasks.rados.rados.0.smithi107.stdout:5304: done (1 left) 2020-06-12T02:38:21.702 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:38:21.702 INFO:tasks.rados.rados.0.smithi107.stdout:5305: seq_num 5304 ranges {2342912=32768} 2020-06-12T02:38:21.709 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:21.710 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:38:21.710 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2020-06-12T02:38:21.710 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:21.715 INFO:tasks.rados.rados.0.smithi107.stdout:5305: writing smithi10729407-20 from 2342912 to 2375680 tid 1 2020-06-12T02:38:21.716 INFO:tasks.rados.rados.0.smithi107.stdout:5306: delete oid 15 current snap is 0 2020-06-12T02:38:21.719 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-06-12T02:38:21.739 INFO:tasks.rados.rados.0.smithi107.stdout:5306: done (2 left) 2020-06-12T02:38:21.739 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:38:21.740 INFO:tasks.rados.rados.0.smithi107.stdout:5307: seq_num 5306 ranges {1769472=65536} 2020-06-12T02:38:21.740 INFO:tasks.rados.rados.0.smithi107.stdout:5307: writing smithi10729407-11 from 1769472 to 1835008 tid 1 2020-06-12T02:38:21.740 INFO:tasks.rados.rados.0.smithi107.stdout:5308: delete oid 15 current snap is 0 2020-06-12T02:38:21.740 INFO:tasks.rados.rados.0.smithi107.stdout:5305: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:21.740 INFO:tasks.rados.rados.0.smithi107.stdout:5305: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:21.740 INFO:tasks.rados.rados.0.smithi107.stdout:5305: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:21.741 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 407 (ObjNum 5304 snap 0 seq_num 5304) dirty exists 2020-06-12T02:38:21.741 INFO:tasks.rados.rados.0.smithi107.stdout:5305: left oid 20 (ObjNum 5304 snap 0 seq_num 5304) 2020-06-12T02:38:21.741 INFO:tasks.rados.rados.0.smithi107.stdout:5305: done (3 left) 2020-06-12T02:38:21.741 INFO:tasks.rados.rados.0.smithi107.stdout:5308: done (2 left) 2020-06-12T02:38:21.741 INFO:tasks.rados.rados.0.smithi107.stdout:5309: delete oid 14 current snap is 0 2020-06-12T02:38:21.742 INFO:tasks.rados.rados.0.smithi107.stdout:5307: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:21.742 INFO:tasks.rados.rados.0.smithi107.stdout:5309: done (2 left) 2020-06-12T02:38:21.742 INFO:tasks.rados.rados.0.smithi107.stdout:5310: delete oid 15 current snap is 0 2020-06-12T02:38:21.743 INFO:tasks.rados.rados.0.smithi107.stdout:5310: done (2 left) 2020-06-12T02:38:21.743 INFO:tasks.rados.rados.0.smithi107.stdout:5311: delete oid 1 current snap is 0 2020-06-12T02:38:21.743 INFO:tasks.rados.rados.0.smithi107.stdout:5307: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:21.743 INFO:tasks.rados.rados.0.smithi107.stdout:5307: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:21.744 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 962 (ObjNum 5306 snap 0 seq_num 5306) dirty exists 2020-06-12T02:38:21.744 INFO:tasks.rados.rados.0.smithi107.stdout:5307: left oid 11 (ObjNum 5306 snap 0 seq_num 5306) 2020-06-12T02:38:21.744 INFO:tasks.rados.rados.0.smithi107.stdout:5307: done (2 left) 2020-06-12T02:38:21.744 INFO:tasks.rados.rados.0.smithi107.stdout:5311: done (1 left) 2020-06-12T02:38:21.744 INFO:tasks.rados.rados.0.smithi107.stdout:5312: delete oid 4 current snap is 0 2020-06-12T02:38:21.745 INFO:tasks.rados.rados.0.smithi107.stdout:5312: done (1 left) 2020-06-12T02:38:21.746 INFO:tasks.rados.rados.0.smithi107.stdout:5313: delete oid 13 current snap is 0 2020-06-12T02:38:21.747 INFO:tasks.rados.rados.0.smithi107.stdout:5313: done (1 left) 2020-06-12T02:38:21.748 INFO:tasks.rados.rados.0.smithi107.stdout:5314: delete oid 16 current snap is 0 2020-06-12T02:38:21.758 INFO:tasks.rados.rados.0.smithi107.stdout:5314: done (1 left) 2020-06-12T02:38:21.759 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:38:21.759 INFO:tasks.rados.rados.0.smithi107.stdout:5315: seq_num 5314 ranges {0=204800} 2020-06-12T02:38:21.759 INFO:tasks.rados.rados.0.smithi107.stdout:5315: writing smithi10729407-5 from 0 to 204800 tid 1 2020-06-12T02:38:21.759 INFO:tasks.rados.rados.0.smithi107.stdout:5316: delete oid 7 current snap is 0 2020-06-12T02:38:21.760 INFO:tasks.rados.rados.0.smithi107.stdout:5316: done (2 left) 2020-06-12T02:38:21.760 INFO:tasks.rados.rados.0.smithi107.stdout:5317: delete oid 14 current snap is 0 2020-06-12T02:38:21.760 INFO:tasks.rados.rados.0.smithi107.stdout:5315: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:21.760 INFO:tasks.rados.rados.0.smithi107.stdout:5317: done (2 left) 2020-06-12T02:38:21.760 INFO:tasks.rados.rados.0.smithi107.stdout:5318: delete oid 4 current snap is 0 2020-06-12T02:38:21.761 INFO:tasks.rados.rados.0.smithi107.stdout:5315: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:21.762 INFO:tasks.rados.rados.0.smithi107.stdout:5315: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:21.762 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 527 (ObjNum 5314 snap 0 seq_num 5314) dirty exists 2020-06-12T02:38:21.762 INFO:tasks.rados.rados.0.smithi107.stdout:5315: left oid 5 (ObjNum 5314 snap 0 seq_num 5314) 2020-06-12T02:38:21.762 INFO:tasks.rados.rados.0.smithi107.stdout:5315: done (2 left) 2020-06-12T02:38:21.763 INFO:tasks.rados.rados.0.smithi107.stdout:5318: done (1 left) 2020-06-12T02:38:21.763 INFO:tasks.rados.rados.0.smithi107.stdout:5319: delete oid 5 current snap is 0 2020-06-12T02:38:21.763 INFO:tasks.rados.rados.0.smithi107.stdout:5319: done (1 left) 2020-06-12T02:38:21.763 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:21.763 INFO:tasks.rados.rados.0.smithi107.stdout:5320: seq_num 5319 ranges {0=286720} 2020-06-12T02:38:21.764 INFO:tasks.rados.rados.0.smithi107.stdout:5320: writing smithi10729407-13 from 0 to 286720 tid 1 2020-06-12T02:38:21.765 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:38:21.765 INFO:tasks.rados.rados.0.smithi107.stdout:5321: seq_num 5320 ranges {2375680=303104} 2020-06-12T02:38:21.781 INFO:tasks.rados.rados.0.smithi107.stdout:5321: writing smithi10729407-20 from 2375680 to 2678784 tid 1 2020-06-12T02:38:21.781 INFO:tasks.rados.rados.0.smithi107.stdout:5322: delete oid 6 current snap is 0 2020-06-12T02:38:21.781 INFO:tasks.rados.rados.0.smithi107.stdout:5320: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:21.782 INFO:tasks.rados.rados.0.smithi107.stdout:5320: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:21.782 INFO:tasks.rados.rados.0.smithi107.stdout:5320: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:21.782 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1010 (ObjNum 5319 snap 0 seq_num 5319) dirty exists 2020-06-12T02:38:21.782 INFO:tasks.rados.rados.0.smithi107.stdout:5320: left oid 13 (ObjNum 5319 snap 0 seq_num 5319) 2020-06-12T02:38:21.782 INFO:tasks.rados.rados.0.smithi107.stdout:5320: done (3 left) 2020-06-12T02:38:21.783 INFO:tasks.rados.rados.0.smithi107.stdout:5322: done (2 left) 2020-06-12T02:38:21.783 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:21.783 INFO:tasks.rados.rados.0.smithi107.stdout:5323: seq_num 5322 ranges {0=335872} 2020-06-12T02:38:21.783 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:21.778+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:21.784 INFO:tasks.rados.rados.0.smithi107.stdout:5323: writing smithi10729407-8 from 0 to 335872 tid 1 2020-06-12T02:38:21.784 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:21.785 INFO:tasks.rados.rados.0.smithi107.stdout:5324: seq_num 5323 ranges {0=352256} 2020-06-12T02:38:21.785 INFO:tasks.rados.rados.0.smithi107.stdout:5324: writing smithi10729407-15 from 0 to 352256 tid 1 2020-06-12T02:38:21.785 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:21.785 INFO:tasks.rados.rados.0.smithi107.stdout:5325: seq_num 5324 ranges {0=368640} 2020-06-12T02:38:21.790 INFO:tasks.rados.rados.0.smithi107.stdout:5325: writing smithi10729407-2 from 0 to 368640 tid 1 2020-06-12T02:38:21.790 INFO:tasks.rados.rados.0.smithi107.stdout:5326: delete oid 7 current snap is 0 2020-06-12T02:38:21.791 INFO:tasks.rados.rados.0.smithi107.stdout:5321: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:21.791 INFO:tasks.rados.rados.0.smithi107.stdout:5321: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:21.791 INFO:tasks.rados.rados.0.smithi107.stdout:5321: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:21.791 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 409 (ObjNum 5320 snap 0 seq_num 5320) dirty exists 2020-06-12T02:38:21.791 INFO:tasks.rados.rados.0.smithi107.stdout:5321: left oid 20 (ObjNum 5320 snap 0 seq_num 5320) 2020-06-12T02:38:21.794 INFO:tasks.rados.rados.0.smithi107.stdout:5323: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:21.795 INFO:tasks.rados.rados.0.smithi107.stdout:5325: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:21.795 INFO:tasks.rados.rados.0.smithi107.stdout:5325: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:21.795 INFO:tasks.rados.rados.0.smithi107.stdout:5321: done (5 left) 2020-06-12T02:38:21.795 INFO:tasks.rados.rados.0.smithi107.stdout:5326: done (4 left) 2020-06-12T02:38:21.795 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:21.796 INFO:tasks.rados.rados.0.smithi107.stdout:5327: seq_num 5326 ranges {0=409600} 2020-06-12T02:38:21.796 INFO:tasks.rados.rados.0.smithi107.stdout:5327: writing smithi10729407-7 from 0 to 409600 tid 1 2020-06-12T02:38:21.797 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:21.797 INFO:tasks.rados.rados.0.smithi107.stdout:5328: seq_num 5327 ranges {0=425984} 2020-06-12T02:38:21.799 INFO:tasks.rados.rados.0.smithi107.stdout:5328: writing smithi10729407-16 from 0 to 425984 tid 1 2020-06-12T02:38:21.799 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:21.799 INFO:tasks.rados.rados.0.smithi107.stdout:5329: seq_num 5328 ranges {0=442368} 2020-06-12T02:38:21.802 INFO:tasks.rados.rados.0.smithi107.stdout:5329: writing smithi10729407-3 from 0 to 442368 tid 1 2020-06-12T02:38:21.802 INFO:tasks.rados.rados.0.smithi107.stdout:5330: delete oid 4 current snap is 0 2020-06-12T02:38:21.802 INFO:tasks.rados.rados.0.smithi107.stdout:5325: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:21.802 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 504 (ObjNum 5324 snap 0 seq_num 5324) dirty exists 2020-06-12T02:38:21.803 INFO:tasks.rados.rados.0.smithi107.stdout:5325: left oid 2 (ObjNum 5324 snap 0 seq_num 5324) 2020-06-12T02:38:21.803 INFO:tasks.rados.rados.0.smithi107.stdout:5324: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:21.803 INFO:tasks.rados.rados.0.smithi107.stdout:5323: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:21.805 INFO:tasks.rados.rados.0.smithi107.stdout:5323: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:21.805 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 943 (ObjNum 5322 snap 0 seq_num 5322) dirty exists 2020-06-12T02:38:21.805 INFO:tasks.rados.rados.0.smithi107.stdout:5323: left oid 8 (ObjNum 5322 snap 0 seq_num 5322) 2020-06-12T02:38:21.806 INFO:tasks.rados.rados.0.smithi107.stdout:5324: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:21.806 INFO:tasks.rados.rados.0.smithi107.stdout:5324: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:21.807 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 944 (ObjNum 5323 snap 0 seq_num 5323) dirty exists 2020-06-12T02:38:21.807 INFO:tasks.rados.rados.0.smithi107.stdout:5324: left oid 15 (ObjNum 5323 snap 0 seq_num 5323) 2020-06-12T02:38:21.807 INFO:tasks.rados.rados.0.smithi107.stdout:5328: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:21.807 INFO:tasks.rados.rados.0.smithi107.stdout:5327: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:21.808 INFO:tasks.rados.rados.0.smithi107.stdout:5327: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:21.808 INFO:tasks.rados.rados.0.smithi107.stdout:5323: done (7 left) 2020-06-12T02:38:21.809 INFO:tasks.rados.rados.0.smithi107.stdout:5324: done (6 left) 2020-06-12T02:38:21.809 INFO:tasks.rados.rados.0.smithi107.stdout:5325: done (5 left) 2020-06-12T02:38:21.809 INFO:tasks.rados.rados.0.smithi107.stdout:5330: done (4 left) 2020-06-12T02:38:21.809 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:21.810 INFO:tasks.rados.rados.0.smithi107.stdout:5331: seq_num 5330 ranges {3227648=475136} 2020-06-12T02:38:21.830 INFO:tasks.rados.rados.0.smithi107.stdout:5331: writing smithi10729407-17 from 3227648 to 3702784 tid 1 2020-06-12T02:38:21.830 INFO:tasks.rados.rados.0.smithi107.stdout:5332: delete oid 14 current snap is 0 2020-06-12T02:38:21.830 INFO:tasks.rados.rados.0.smithi107.stdout:5329: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:21.830 INFO:tasks.rados.rados.0.smithi107.stdout:5327: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:21.831 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1013 (ObjNum 5326 snap 0 seq_num 5326) dirty exists 2020-06-12T02:38:21.831 INFO:tasks.rados.rados.0.smithi107.stdout:5327: left oid 7 (ObjNum 5326 snap 0 seq_num 5326) 2020-06-12T02:38:21.831 INFO:tasks.rados.rados.0.smithi107.stdout:5328: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:21.831 INFO:tasks.rados.rados.0.smithi107.stdout:5328: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:21.831 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 945 (ObjNum 5327 snap 0 seq_num 5327) dirty exists 2020-06-12T02:38:21.832 INFO:tasks.rados.rados.0.smithi107.stdout:5328: left oid 16 (ObjNum 5327 snap 0 seq_num 5327) 2020-06-12T02:38:21.832 INFO:tasks.rados.rados.0.smithi107.stdout:5329: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:21.832 INFO:tasks.rados.rados.0.smithi107.stdout:5329: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:21.832 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 946 (ObjNum 5328 snap 0 seq_num 5328) dirty exists 2020-06-12T02:38:21.833 INFO:tasks.rados.rados.0.smithi107.stdout:5329: left oid 3 (ObjNum 5328 snap 0 seq_num 5328) 2020-06-12T02:38:21.833 INFO:tasks.rados.rados.0.smithi107.stdout:5327: done (5 left) 2020-06-12T02:38:21.834 INFO:tasks.rados.rados.0.smithi107.stdout:5328: done (4 left) 2020-06-12T02:38:21.834 INFO:tasks.rados.rados.0.smithi107.stdout:5329: done (3 left) 2020-06-12T02:38:21.834 INFO:tasks.rados.rados.0.smithi107.stdout:5332: done (2 left) 2020-06-12T02:38:21.834 INFO:tasks.rados.rados.0.smithi107.stdout:5333: delete oid 2 current snap is 0 2020-06-12T02:38:21.835 INFO:tasks.rados.rados.0.smithi107.stdout:5331: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:21.835 INFO:tasks.rados.rados.0.smithi107.stdout:5333: done (2 left) 2020-06-12T02:38:21.835 INFO:tasks.rados.rados.0.smithi107.stdout:5334: delete oid 16 current snap is 0 2020-06-12T02:38:21.839 INFO:tasks.rados.rados.0.smithi107.stdout:5331: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:21.839 INFO:tasks.rados.rados.0.smithi107.stdout:5334: done (2 left) 2020-06-12T02:38:21.839 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:21.839 INFO:tasks.rados.rados.0.smithi107.stdout:5335: seq_num 5334 ranges {0=540672} 2020-06-12T02:38:21.842 INFO:tasks.rados.rados.0.smithi107.stdout:5335: writing smithi10729407-19 from 0 to 540672 tid 1 2020-06-12T02:38:21.843 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:21.843 INFO:tasks.rados.rados.0.smithi107.stdout:5336: seq_num 5335 ranges {0=557056} 2020-06-12T02:38:21.846 INFO:tasks.rados.rados.0.smithi107.stdout:5336: writing smithi10729407-16 from 0 to 557056 tid 1 2020-06-12T02:38:21.846 INFO:tasks.rados.rados.0.smithi107.stdout:5337: delete oid 1 current snap is 0 2020-06-12T02:38:21.846 INFO:tasks.rados.rados.0.smithi107.stdout:5331: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:21.847 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 995 (ObjNum 5330 snap 0 seq_num 5330) dirty exists 2020-06-12T02:38:21.847 INFO:tasks.rados.rados.0.smithi107.stdout:5331: left oid 17 (ObjNum 5330 snap 0 seq_num 5330) 2020-06-12T02:38:21.848 INFO:tasks.rados.rados.0.smithi107.stdout:5331: done (4 left) 2020-06-12T02:38:21.848 INFO:tasks.rados.rados.0.smithi107.stdout:5337: done (3 left) 2020-06-12T02:38:21.848 INFO:tasks.rados.rados.0.smithi107.stdout:5338: delete oid 8 current snap is 0 2020-06-12T02:38:21.849 INFO:tasks.rados.rados.0.smithi107.stdout:5335: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:21.849 INFO:tasks.rados.rados.0.smithi107.stdout:5335: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:21.851 INFO:tasks.rados.rados.0.smithi107.stdout:5335: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:21.851 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 929 (ObjNum 5334 snap 0 seq_num 5334) dirty exists 2020-06-12T02:38:21.851 INFO:tasks.rados.rados.0.smithi107.stdout:5335: left oid 19 (ObjNum 5334 snap 0 seq_num 5334) 2020-06-12T02:38:21.853 INFO:tasks.rados.rados.0.smithi107.stdout:5335: done (3 left) 2020-06-12T02:38:21.853 INFO:tasks.rados.rados.0.smithi107.stdout:5338: done (2 left) 2020-06-12T02:38:21.853 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:21.853 INFO:tasks.rados.rados.0.smithi107.stdout:5339: seq_num 5338 ranges {1835008=606208} 2020-06-12T02:38:21.865 INFO:tasks.rados.rados.0.smithi107.stdout:5339: writing smithi10729407-11 from 1835008 to 2441216 tid 1 2020-06-12T02:38:21.865 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:21.865 INFO:tasks.rados.rados.0.smithi107.stdout:5340: seq_num 5339 ranges {0=622592} 2020-06-12T02:38:21.869 INFO:tasks.rados.rados.0.smithi107.stdout:5340: writing smithi10729407-12 from 0 to 622592 tid 1 2020-06-12T02:38:21.869 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:21.869 INFO:tasks.rados.rados.0.smithi107.stdout:5341: seq_num 5340 ranges {0=638976} 2020-06-12T02:38:21.874 INFO:tasks.rados.rados.0.smithi107.stdout:5341: writing smithi10729407-9 from 0 to 638976 tid 1 2020-06-12T02:38:21.875 INFO:tasks.rados.rados.0.smithi107.stdout:5342: delete oid 1 current snap is 0 2020-06-12T02:38:21.875 INFO:tasks.rados.rados.0.smithi107.stdout:5336: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:21.875 INFO:tasks.rados.rados.0.smithi107.stdout:5336: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:21.875 INFO:tasks.rados.rados.0.smithi107.stdout:5336: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:21.876 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 949 (ObjNum 5335 snap 0 seq_num 5335) dirty exists 2020-06-12T02:38:21.876 INFO:tasks.rados.rados.0.smithi107.stdout:5336: left oid 16 (ObjNum 5335 snap 0 seq_num 5335) 2020-06-12T02:38:21.876 INFO:tasks.rados.rados.0.smithi107.stdout:5339: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:21.876 INFO:tasks.rados.rados.0.smithi107.stdout:5339: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:21.876 INFO:tasks.rados.rados.0.smithi107.stdout:5339: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:21.876 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 966 (ObjNum 5338 snap 0 seq_num 5338) dirty exists 2020-06-12T02:38:21.877 INFO:tasks.rados.rados.0.smithi107.stdout:5339: left oid 11 (ObjNum 5338 snap 0 seq_num 5338) 2020-06-12T02:38:21.877 INFO:tasks.rados.rados.0.smithi107.stdout:5336: done (5 left) 2020-06-12T02:38:21.878 INFO:tasks.rados.rados.0.smithi107.stdout:5339: done (4 left) 2020-06-12T02:38:21.878 INFO:tasks.rados.rados.0.smithi107.stdout:5342: done (3 left) 2020-06-12T02:38:21.878 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:21.878 INFO:tasks.rados.rados.0.smithi107.stdout:5343: seq_num 5342 ranges {0=671744} 2020-06-12T02:38:21.880 INFO:tasks.rados.rados.0.smithi107.stdout:5343: writing smithi10729407-2 from 0 to 671744 tid 1 2020-06-12T02:38:21.880 INFO:tasks.rados.rados.0.smithi107.stdout:5344: delete oid 15 current snap is 0 2020-06-12T02:38:21.881 INFO:tasks.rados.rados.0.smithi107.stdout:5340: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:21.887 INFO:tasks.rados.rados.0.smithi107.stdout:5341: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:21.887 INFO:tasks.rados.rados.0.smithi107.stdout:5344: done (4 left) 2020-06-12T02:38:21.887 INFO:tasks.rados.rados.0.smithi107.stdout:5345: delete oid 18 current snap is 0 2020-06-12T02:38:21.888 INFO:tasks.rados.rados.0.smithi107.stdout:5341: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:21.888 INFO:tasks.rados.rados.0.smithi107.stdout:5341: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:21.888 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 964 (ObjNum 5340 snap 0 seq_num 5340) dirty exists 2020-06-12T02:38:21.888 INFO:tasks.rados.rados.0.smithi107.stdout:5341: left oid 9 (ObjNum 5340 snap 0 seq_num 5340) 2020-06-12T02:38:21.888 INFO:tasks.rados.rados.0.smithi107.stdout:5340: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:21.889 INFO:tasks.rados.rados.0.smithi107.stdout:5341: done (4 left) 2020-06-12T02:38:21.889 INFO:tasks.rados.rados.0.smithi107.stdout:5345: done (3 left) 2020-06-12T02:38:21.889 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:21.889 INFO:tasks.rados.rados.0.smithi107.stdout:5346: seq_num 5345 ranges {2441216=720896} 2020-06-12T02:38:21.889 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:21.884+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:21.904 INFO:tasks.rados.rados.0.smithi107.stdout:5346: writing smithi10729407-11 from 2441216 to 3162112 tid 1 2020-06-12T02:38:21.904 INFO:tasks.rados.rados.0.smithi107.stdout:5347: delete oid 18 current snap is 0 2020-06-12T02:38:21.905 INFO:tasks.rados.rados.0.smithi107.stdout:5343: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:21.905 INFO:tasks.rados.rados.0.smithi107.stdout:5343: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:21.905 INFO:tasks.rados.rados.0.smithi107.stdout:5340: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:21.905 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 931 (ObjNum 5339 snap 0 seq_num 5339) dirty exists 2020-06-12T02:38:21.905 INFO:tasks.rados.rados.0.smithi107.stdout:5340: left oid 12 (ObjNum 5339 snap 0 seq_num 5339) 2020-06-12T02:38:21.906 INFO:tasks.rados.rados.0.smithi107.stdout:5343: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:21.906 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 507 (ObjNum 5342 snap 0 seq_num 5342) dirty exists 2020-06-12T02:38:21.906 INFO:tasks.rados.rados.0.smithi107.stdout:5343: left oid 2 (ObjNum 5342 snap 0 seq_num 5342) 2020-06-12T02:38:21.906 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:21.907 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:38:21.907 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:38:21.907 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:38:21.907 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:21.908 INFO:tasks.rados.rados.0.smithi107.stdout:5340: done (4 left) 2020-06-12T02:38:21.908 INFO:tasks.rados.rados.0.smithi107.stdout:5343: done (3 left) 2020-06-12T02:38:21.908 INFO:tasks.rados.rados.0.smithi107.stdout:5347: done (2 left) 2020-06-12T02:38:21.908 INFO:tasks.rados.rados.0.smithi107.stdout:5348: delete oid 7 current snap is 0 2020-06-12T02:38:21.909 INFO:tasks.rados.rados.0.smithi107.stdout:5348: done (2 left) 2020-06-12T02:38:21.910 INFO:tasks.rados.rados.0.smithi107.stdout:5349: delete oid 2 current snap is 0 2020-06-12T02:38:21.910 INFO:tasks.rados.rados.0.smithi107.stdout:5346: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:21.912 INFO:tasks.rados.rados.0.smithi107.stdout:5346: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:21.913 INFO:tasks.rados.rados.0.smithi107.stdout:5346: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:21.913 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 968 (ObjNum 5345 snap 0 seq_num 5345) dirty exists 2020-06-12T02:38:21.913 INFO:tasks.rados.rados.0.smithi107.stdout:5346: left oid 11 (ObjNum 5345 snap 0 seq_num 5345) 2020-06-12T02:38:21.914 INFO:tasks.rados.rados.0.smithi107.stdout:5346: done (2 left) 2020-06-12T02:38:21.914 INFO:tasks.rados.rados.0.smithi107.stdout:5349: done (1 left) 2020-06-12T02:38:21.914 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:21.914 INFO:tasks.rados.rados.0.smithi107.stdout:5350: seq_num 5349 ranges {622592=794624} 2020-06-12T02:38:21.916 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-06-12T02:38:21.921 INFO:tasks.rados.rados.0.smithi107.stdout:5350: writing smithi10729407-12 from 622592 to 1417216 tid 1 2020-06-12T02:38:21.921 INFO:tasks.rados.rados.0.smithi107.stdout:5351: delete oid 20 current snap is 0 2020-06-12T02:38:21.923 INFO:tasks.rados.rados.0.smithi107.stdout:5351: done (2 left) 2020-06-12T02:38:21.924 INFO:tasks.rados.rados.0.smithi107.stdout:5352: delete oid 3 current snap is 0 2020-06-12T02:38:21.925 INFO:tasks.rados.rados.0.smithi107.stdout:5352: done (2 left) 2020-06-12T02:38:21.926 INFO:tasks.rados.rados.0.smithi107.stdout:5353: delete oid 1 current snap is 0 2020-06-12T02:38:21.930 INFO:tasks.rados.rados.0.smithi107.stdout:5350: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:21.935 INFO:tasks.rados.rados.0.smithi107.stdout:5353: done (2 left) 2020-06-12T02:38:21.935 INFO:tasks.rados.rados.0.smithi107.stdout:5354: delete oid 17 current snap is 0 2020-06-12T02:38:21.936 INFO:tasks.rados.rados.0.smithi107.stdout:5350: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:21.937 INFO:tasks.rados.rados.0.smithi107.stdout:5350: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:21.937 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 933 (ObjNum 5349 snap 0 seq_num 5349) dirty exists 2020-06-12T02:38:21.937 INFO:tasks.rados.rados.0.smithi107.stdout:5350: left oid 12 (ObjNum 5349 snap 0 seq_num 5349) 2020-06-12T02:38:21.945 INFO:tasks.rados.rados.0.smithi107.stdout:5350: done (2 left) 2020-06-12T02:38:21.945 INFO:tasks.rados.rados.0.smithi107.stdout:5354: done (1 left) 2020-06-12T02:38:21.946 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:21.946 INFO:tasks.rados.rados.0.smithi107.stdout:5355: seq_num 5354 ranges {0=475136,475136=401408} 2020-06-12T02:38:21.946 INFO:tasks.rados.rados.0.smithi107.stdout:5355: writing smithi10729407-14 from 0 to 475136 tid 1 2020-06-12T02:38:21.947 INFO:tasks.rados.rados.0.smithi107.stdout:5355: writing smithi10729407-14 from 475136 to 876544 tid 2 2020-06-12T02:38:21.947 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:38:21.947 INFO:tasks.rados.rados.0.smithi107.stdout:5356: seq_num 5355 ranges {0=491520,491520=401408} 2020-06-12T02:38:21.948 INFO:tasks.rados.rados.0.smithi107.stdout:5356: writing smithi10729407-2 from 0 to 491520 tid 1 2020-06-12T02:38:21.951 INFO:tasks.rados.rados.0.smithi107.stdout:5356: writing smithi10729407-2 from 491520 to 892928 tid 2 2020-06-12T02:38:21.951 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:21.952 INFO:tasks.rados.rados.0.smithi107.stdout:5357: seq_num 5356 ranges {0=507904,507904=401408} 2020-06-12T02:38:21.954 INFO:tasks.rados.rados.0.smithi107.stdout:5357: writing smithi10729407-3 from 0 to 507904 tid 1 2020-06-12T02:38:21.958 INFO:tasks.rados.rados.0.smithi107.stdout:5357: writing smithi10729407-3 from 507904 to 909312 tid 2 2020-06-12T02:38:21.958 INFO:tasks.rados.rados.0.smithi107.stdout:5358: delete oid 4 current snap is 0 2020-06-12T02:38:21.958 INFO:tasks.rados.rados.0.smithi107.stdout:5355: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:21.959 INFO:tasks.rados.rados.0.smithi107.stdout:5355: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:21.959 INFO:tasks.rados.rados.0.smithi107.stdout:5355: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:21.959 INFO:tasks.rados.rados.0.smithi107.stdout:5356: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:21.959 INFO:tasks.rados.rados.0.smithi107.stdout:5355: finishing write tid 5 to smithi10729407-14 2020-06-12T02:38:21.959 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 971 (ObjNum 5354 snap 0 seq_num 5354) dirty exists 2020-06-12T02:38:21.959 INFO:tasks.rados.rados.0.smithi107.stdout:5355: left oid 14 (ObjNum 5354 snap 0 seq_num 5354) 2020-06-12T02:38:21.960 INFO:tasks.rados.rados.0.smithi107.stdout:5356: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:21.963 INFO:tasks.rados.rados.0.smithi107.stdout:5356: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:21.963 INFO:tasks.rados.rados.0.smithi107.stdout:5357: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:21.964 INFO:tasks.rados.rados.0.smithi107.stdout:5355: done (4 left) 2020-06-12T02:38:21.964 INFO:tasks.rados.rados.0.smithi107.stdout:5358: done (3 left) 2020-06-12T02:38:21.964 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:38:21.964 INFO:tasks.rados.rados.0.smithi107.stdout:5359: seq_num 5358 ranges {286720=540672,827392=401408} 2020-06-12T02:38:21.967 INFO:tasks.rados.rados.0.smithi107.stdout:5359: writing smithi10729407-13 from 286720 to 827392 tid 1 2020-06-12T02:38:21.970 INFO:tasks.rados.rados.0.smithi107.stdout:5359: writing smithi10729407-13 from 827392 to 1228800 tid 2 2020-06-12T02:38:21.970 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:21.970 INFO:tasks.rados.rados.0.smithi107.stdout:5360: seq_num 5359 ranges {638976=557056,1196032=401408} 2020-06-12T02:38:21.976 INFO:tasks.rados.rados.0.smithi107.stdout:5360: writing smithi10729407-9 from 638976 to 1196032 tid 1 2020-06-12T02:38:21.979 INFO:tasks.rados.rados.0.smithi107.stdout:5360: writing smithi10729407-9 from 1196032 to 1597440 tid 2 2020-06-12T02:38:21.980 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:21.980 INFO:tasks.rados.rados.0.smithi107.stdout:5361: seq_num 5360 ranges {0=573440,573440=401408} 2020-06-12T02:38:21.983 INFO:tasks.rados.rados.0.smithi107.stdout:5361: writing smithi10729407-18 from 0 to 573440 tid 1 2020-06-12T02:38:21.986 INFO:tasks.rados.rados.0.smithi107.stdout:5361: writing smithi10729407-18 from 573440 to 974848 tid 2 2020-06-12T02:38:21.986 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:38:21.986 INFO:tasks.rados.rados.0.smithi107.stdout:5362: seq_num 5361 ranges {3162112=589824,3751936=401408} 2020-06-12T02:38:21.988 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:21.985+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:22.006 INFO:tasks.rados.rados.0.smithi107.stdout:5362: writing smithi10729407-11 from 3162112 to 3751936 tid 1 2020-06-12T02:38:22.009 INFO:tasks.rados.rados.0.smithi107.stdout:5362: writing smithi10729407-11 from 3751936 to 4153344 tid 2 2020-06-12T02:38:22.009 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:22.009 INFO:tasks.rados.rados.0.smithi107.stdout:5363: seq_num 5362 ranges {557056=606208,1163264=401408} 2020-06-12T02:38:22.015 INFO:tasks.rados.rados.0.smithi107.stdout:5363: writing smithi10729407-16 from 557056 to 1163264 tid 1 2020-06-12T02:38:22.019 INFO:tasks.rados.rados.0.smithi107.stdout:5363: writing smithi10729407-16 from 1163264 to 1564672 tid 2 2020-06-12T02:38:22.019 INFO:tasks.rados.rados.0.smithi107.stdout: waiting on 8 2020-06-12T02:38:22.019 INFO:tasks.rados.rados.0.smithi107.stdout:5356: finishing write tid 5 to smithi10729407-2 2020-06-12T02:38:22.020 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 511 (ObjNum 5355 snap 0 seq_num 5355) dirty exists 2020-06-12T02:38:22.020 INFO:tasks.rados.rados.0.smithi107.stdout:5356: left oid 2 (ObjNum 5355 snap 0 seq_num 5355) 2020-06-12T02:38:22.020 INFO:tasks.rados.rados.0.smithi107.stdout:5356: done (7 left) 2020-06-12T02:38:22.020 INFO:tasks.rados.rados.0.smithi107.stdout:5364: delete oid 1 current snap is 0 2020-06-12T02:38:22.020 INFO:tasks.rados.rados.0.smithi107.stdout:5357: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:22.021 INFO:tasks.rados.rados.0.smithi107.stdout:5359: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:22.021 INFO:tasks.rados.rados.0.smithi107.stdout:5357: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:22.021 INFO:tasks.rados.rados.0.smithi107.stdout:5359: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:22.021 INFO:tasks.rados.rados.0.smithi107.stdout:5359: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:22.021 INFO:tasks.rados.rados.0.smithi107.stdout:5357: finishing write tid 5 to smithi10729407-3 2020-06-12T02:38:22.022 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 953 (ObjNum 5356 snap 0 seq_num 5356) dirty exists 2020-06-12T02:38:22.022 INFO:tasks.rados.rados.0.smithi107.stdout:5357: left oid 3 (ObjNum 5356 snap 0 seq_num 5356) 2020-06-12T02:38:22.022 INFO:tasks.rados.rados.0.smithi107.stdout:5359: finishing write tid 5 to smithi10729407-13 2020-06-12T02:38:22.022 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1017 (ObjNum 5358 snap 0 seq_num 5358) dirty exists 2020-06-12T02:38:22.022 INFO:tasks.rados.rados.0.smithi107.stdout:5359: left oid 13 (ObjNum 5358 snap 0 seq_num 5358) 2020-06-12T02:38:22.022 INFO:tasks.rados.rados.0.smithi107.stdout:5360: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:22.023 INFO:tasks.rados.rados.0.smithi107.stdout:5360: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:22.023 INFO:tasks.rados.rados.0.smithi107.stdout:5360: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:22.023 INFO:tasks.rados.rados.0.smithi107.stdout:5361: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:22.023 INFO:tasks.rados.rados.0.smithi107.stdout:5360: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:22.023 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 969 (ObjNum 5359 snap 0 seq_num 5359) dirty exists 2020-06-12T02:38:22.023 INFO:tasks.rados.rados.0.smithi107.stdout:5360: left oid 9 (ObjNum 5359 snap 0 seq_num 5359) 2020-06-12T02:38:22.024 INFO:tasks.rados.rados.0.smithi107.stdout:5361: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:22.024 INFO:tasks.rados.rados.0.smithi107.stdout:5361: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:22.024 INFO:tasks.rados.rados.0.smithi107.stdout:5361: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:22.024 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 972 (ObjNum 5360 snap 0 seq_num 5360) dirty exists 2020-06-12T02:38:22.024 INFO:tasks.rados.rados.0.smithi107.stdout:5361: left oid 18 (ObjNum 5360 snap 0 seq_num 5360) 2020-06-12T02:38:22.025 INFO:tasks.rados.rados.0.smithi107.stdout:5362: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:22.025 INFO:tasks.rados.rados.0.smithi107.stdout:5362: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:22.025 INFO:tasks.rados.rados.0.smithi107.stdout:5362: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:22.027 INFO:tasks.rados.rados.0.smithi107.stdout:5362: finishing write tid 5 to smithi10729407-11 2020-06-12T02:38:22.027 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 974 (ObjNum 5361 snap 0 seq_num 5361) dirty exists 2020-06-12T02:38:22.027 INFO:tasks.rados.rados.0.smithi107.stdout:5362: left oid 11 (ObjNum 5361 snap 0 seq_num 5361) 2020-06-12T02:38:22.028 INFO:tasks.rados.rados.0.smithi107.stdout:5357: done (7 left) 2020-06-12T02:38:22.028 INFO:tasks.rados.rados.0.smithi107.stdout:5359: done (6 left) 2020-06-12T02:38:22.028 INFO:tasks.rados.rados.0.smithi107.stdout:5360: done (5 left) 2020-06-12T02:38:22.028 INFO:tasks.rados.rados.0.smithi107.stdout:5361: done (4 left) 2020-06-12T02:38:22.029 INFO:tasks.rados.rados.0.smithi107.stdout:5362: done (3 left) 2020-06-12T02:38:22.029 INFO:tasks.rados.rados.0.smithi107.stdout:5364: done (2 left) 2020-06-12T02:38:22.029 INFO:tasks.rados.rados.0.smithi107.stdout:5365: delete oid 6 current snap is 0 2020-06-12T02:38:22.029 INFO:tasks.rados.rados.0.smithi107.stdout:5363: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:22.029 INFO:tasks.rados.rados.0.smithi107.stdout:5365: done (2 left) 2020-06-12T02:38:22.029 INFO:tasks.rados.rados.0.smithi107.stdout:5366: delete oid 20 current snap is 0 2020-06-12T02:38:22.030 INFO:tasks.rados.rados.0.smithi107.stdout:5363: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:22.031 INFO:tasks.rados.rados.0.smithi107.stdout:5363: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:22.031 INFO:tasks.rados.rados.0.smithi107.stdout:5366: done (2 left) 2020-06-12T02:38:22.031 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:38:22.031 INFO:tasks.rados.rados.0.smithi107.stdout:5367: seq_num 5366 ranges {1228800=679936,1908736=401408} 2020-06-12T02:38:22.038 INFO:tasks.rados.rados.0.smithi107.stdout:5367: writing smithi10729407-13 from 1228800 to 1908736 tid 1 2020-06-12T02:38:22.040 INFO:tasks.rados.rados.0.smithi107.stdout:5367: writing smithi10729407-13 from 1908736 to 2310144 tid 2 2020-06-12T02:38:22.041 INFO:tasks.rados.rados.0.smithi107.stdout:5368: delete oid 2 current snap is 0 2020-06-12T02:38:22.041 INFO:tasks.rados.rados.0.smithi107.stdout:5363: finishing write tid 5 to smithi10729407-16 2020-06-12T02:38:22.041 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 956 (ObjNum 5362 snap 0 seq_num 5362) dirty exists 2020-06-12T02:38:22.041 INFO:tasks.rados.rados.0.smithi107.stdout:5363: left oid 16 (ObjNum 5362 snap 0 seq_num 5362) 2020-06-12T02:38:22.043 INFO:tasks.rados.rados.0.smithi107.stdout:5363: done (3 left) 2020-06-12T02:38:22.043 INFO:tasks.rados.rados.0.smithi107.stdout:5368: done (2 left) 2020-06-12T02:38:22.043 INFO:tasks.rados.rados.0.smithi107.stdout:5369: delete oid 7 current snap is 0 2020-06-12T02:38:22.049 INFO:tasks.rados.rados.0.smithi107.stdout:5367: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:22.051 INFO:tasks.rados.rados.0.smithi107.stdout:5367: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:22.051 INFO:tasks.rados.rados.0.smithi107.stdout:5367: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:22.052 INFO:tasks.rados.rados.0.smithi107.stdout:5367: finishing write tid 5 to smithi10729407-13 2020-06-12T02:38:22.052 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1020 (ObjNum 5366 snap 0 seq_num 5366) dirty exists 2020-06-12T02:38:22.053 INFO:tasks.rados.rados.0.smithi107.stdout:5367: left oid 13 (ObjNum 5366 snap 0 seq_num 5366) 2020-06-12T02:38:22.054 INFO:tasks.rados.rados.0.smithi107.stdout:5367: done (2 left) 2020-06-12T02:38:22.055 INFO:tasks.rados.rados.0.smithi107.stdout:5369: done (1 left) 2020-06-12T02:38:22.055 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:22.056 INFO:tasks.rados.rados.0.smithi107.stdout:5370: seq_num 5369 ranges {876544=729088,1605632=401408} 2020-06-12T02:38:22.062 INFO:tasks.rados.rados.0.smithi107.stdout:5370: writing smithi10729407-14 from 876544 to 1605632 tid 1 2020-06-12T02:38:22.064 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:22.064 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:38:22.064 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:38:22.064 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:38:22.064 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.065 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:860 3.3s0 3:ce45c70a:::smithi10729407-7:head [append 0~81920] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:38:22.065 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.928619+0000", 2020-06-12T02:38:22.065 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 50.130905108999997, 2020-06-12T02:38:22.065 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017431941999999999, 2020-06-12T02:38:22.065 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:22.066 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:22.066 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:22.066 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:22.066 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:22.066 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 860 2020-06-12T02:38:22.067 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.067 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:22.067 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.067 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:22.067 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928619+0000", 2020-06-12T02:38:22.068 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:22.068 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.068 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.068 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:22.068 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928619+0000", 2020-06-12T02:38:22.069 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000105809 2020-06-12T02:38:22.069 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.069 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.069 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:22.069 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928724+0000", 2020-06-12T02:38:22.069 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2339999999999994e-06 2020-06-12T02:38:22.070 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.070 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.070 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:22.070 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928733+0000", 2020-06-12T02:38:22.070 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0268000000000001e-05 2020-06-12T02:38:22.071 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.071 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.071 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:22.071 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928743+0000", 2020-06-12T02:38:22.071 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5190000000000003e-06 2020-06-12T02:38:22.072 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.072 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.072 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:22.072 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928747+0000", 2020-06-12T02:38:22.072 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0103999999999998e-05 2020-06-12T02:38:22.073 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.073 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.073 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.074 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928777+0000", 2020-06-12T02:38:22.074 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8249000000000002e-05 2020-06-12T02:38:22.075 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.075 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.075 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:22.075 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928866+0000", 2020-06-12T02:38:22.075 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011151629 2020-06-12T02:38:22.076 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.076 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.076 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.076 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.940017+0000", 2020-06-12T02:38:22.076 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8281e-05 2020-06-12T02:38:22.077 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.077 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.077 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:22.077 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.940076+0000", 2020-06-12T02:38:22.077 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000346736 2020-06-12T02:38:22.078 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.078 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.078 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:22.078 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.940422+0000", 2020-06-12T02:38:22.078 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024135049999999998 2020-06-12T02:38:22.078 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.079 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.079 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:22.079 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.942836+0000", 2020-06-12T02:38:22.079 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003170258 2020-06-12T02:38:22.079 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.080 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.080 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:22.080 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946006+0000", 2020-06-12T02:38:22.080 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4350000000000001e-05 2020-06-12T02:38:22.080 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.080 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.081 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:22.081 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946050+0000", 2020-06-12T02:38:22.081 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017431941999999999 2020-06-12T02:38:22.081 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.081 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:22.082 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.082 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.082 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.082 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:861 3.3s0 3:ce45c70a:::smithi10729407-7:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:38:22.082 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.928778+0000", 2020-06-12T02:38:22.082 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 50.130745496000003, 2020-06-12T02:38:22.083 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017522738, 2020-06-12T02:38:22.083 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:22.083 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:22.083 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:22.083 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:22.084 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:22.084 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 861 2020-06-12T02:38:22.084 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.084 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:22.084 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.084 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:22.085 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928778+0000", 2020-06-12T02:38:22.085 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:22.085 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.085 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.085 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:22.086 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928778+0000", 2020-06-12T02:38:22.086 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8186000000000002e-05 2020-06-12T02:38:22.086 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.086 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.086 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:22.087 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928826+0000", 2020-06-12T02:38:22.087 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6689999999999997e-06 2020-06-12T02:38:22.087 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.087 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.087 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:22.087 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928831+0000", 2020-06-12T02:38:22.088 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.686e-06 2020-06-12T02:38:22.088 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.088 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.088 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:22.088 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928838+0000", 2020-06-12T02:38:22.089 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7159999999999999e-06 2020-06-12T02:38:22.089 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.089 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.089 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:22.089 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928840+0000", 2020-06-12T02:38:22.089 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4212999999999997e-05 2020-06-12T02:38:22.090 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.090 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.090 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.090 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928895+0000", 2020-06-12T02:38:22.090 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5853999999999994e-05 2020-06-12T02:38:22.091 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.091 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.091 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:22.091 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928990+0000", 2020-06-12T02:38:22.091 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012023982000000001 2020-06-12T02:38:22.091 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.092 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.092 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.092 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941014+0000", 2020-06-12T02:38:22.092 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5104e-05 2020-06-12T02:38:22.092 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.093 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.093 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:22.093 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941070+0000", 2020-06-12T02:38:22.093 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00026444499999999998 2020-06-12T02:38:22.093 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.093 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.094 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:22.094 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941334+0000", 2020-06-12T02:38:22.094 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001704075 2020-06-12T02:38:22.094 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.094 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.095 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:22.095 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.943038+0000", 2020-06-12T02:38:22.095 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031891570000000002 2020-06-12T02:38:22.095 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.095 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.095 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:22.096 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946227+0000", 2020-06-12T02:38:22.096 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3651000000000003e-05 2020-06-12T02:38:22.096 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.096 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.096 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:22.097 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946301+0000", 2020-06-12T02:38:22.097 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017522738 2020-06-12T02:38:22.097 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.097 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:22.097 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.097 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.098 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.098 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:862 3.3s0 3:ce45c70a:::smithi10729407-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:38:22.098 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.928876+0000", 2020-06-12T02:38:22.098 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 50.130648000000001, 2020-06-12T02:38:22.098 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019206890000000001, 2020-06-12T02:38:22.099 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:22.099 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:22.099 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:22.099 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:22.099 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:22.099 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 862 2020-06-12T02:38:22.100 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.100 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:22.100 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.100 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:22.100 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928876+0000", 2020-06-12T02:38:22.101 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:22.101 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.101 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.101 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:22.101 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928876+0000", 2020-06-12T02:38:22.102 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9268999999999999e-05 2020-06-12T02:38:22.102 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.102 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.102 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:22.102 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928925+0000", 2020-06-12T02:38:22.102 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8120000000000002e-06 2020-06-12T02:38:22.103 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.103 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.103 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:22.103 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928932+0000", 2020-06-12T02:38:22.103 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3471e-05 2020-06-12T02:38:22.104 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.104 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.104 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:22.104 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928945+0000", 2020-06-12T02:38:22.104 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4279999999999998e-06 2020-06-12T02:38:22.104 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.105 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.105 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:22.105 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928950+0000", 2020-06-12T02:38:22.105 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6358000000000004e-05 2020-06-12T02:38:22.105 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.106 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.106 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.106 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.929016+0000", 2020-06-12T02:38:22.106 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9697999999999996e-05 2020-06-12T02:38:22.106 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.106 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.107 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:22.107 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.929086+0000", 2020-06-12T02:38:22.107 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012409205 2020-06-12T02:38:22.107 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.107 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.108 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.108 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941495+0000", 2020-06-12T02:38:22.108 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9660000000000006e-05 2020-06-12T02:38:22.108 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.108 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.108 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:22.109 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941585+0000", 2020-06-12T02:38:22.109 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050329099999999998 2020-06-12T02:38:22.109 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.109 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.109 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.110 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946617+0000", 2020-06-12T02:38:22.110 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8437999999999996e-05 2020-06-12T02:38:22.110 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.110 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.110 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:22.110 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946706+0000", 2020-06-12T02:38:22.111 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0013766410000000001 2020-06-12T02:38:22.111 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.111 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.111 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:22.111 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.948083+0000", 2020-06-12T02:38:22.112 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019206890000000001 2020-06-12T02:38:22.112 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.112 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:22.112 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.112 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.112 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.113 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:879 3.fs1 3:fc7e0bf7:::smithi10729407-17:head [append 0~491520] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:38:22.113 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.000881+0000", 2020-06-12T02:38:22.113 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 50.058642818999999, 2020-06-12T02:38:22.113 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017891845, 2020-06-12T02:38:22.113 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:22.114 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:22.114 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:22.114 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:22.114 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:22.114 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 879 2020-06-12T02:38:22.114 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.115 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:22.115 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.115 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:22.115 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.000881+0000", 2020-06-12T02:38:22.115 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:22.116 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.116 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.116 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:22.116 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.000881+0000", 2020-06-12T02:38:22.116 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00044151599999999998 2020-06-12T02:38:22.116 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.117 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.117 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:22.117 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001322+0000", 2020-06-12T02:38:22.117 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0210000000000001e-06 2020-06-12T02:38:22.117 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.118 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.118 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:22.118 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001326+0000", 2020-06-12T02:38:22.118 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6659999999999997e-06 2020-06-12T02:38:22.118 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.118 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.119 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:22.119 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001335+0000", 2020-06-12T02:38:22.119 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4299999999999999e-06 2020-06-12T02:38:22.119 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.119 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.120 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:22.120 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001339+0000", 2020-06-12T02:38:22.120 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0028741510000000001 2020-06-12T02:38:22.120 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.120 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.120 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.121 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.004214+0000", 2020-06-12T02:38:22.121 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010480999999999999 2020-06-12T02:38:22.121 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.121 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.121 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:22.122 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.004318+0000", 2020-06-12T02:38:22.122 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021112359999999998 2020-06-12T02:38:22.122 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.122 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.122 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.123 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006430+0000", 2020-06-12T02:38:22.123 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2935000000000001e-05 2020-06-12T02:38:22.123 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.124 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.124 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:22.124 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006483+0000", 2020-06-12T02:38:22.124 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00056411099999999995 2020-06-12T02:38:22.124 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.124 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.125 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:22.125 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.007047+0000", 2020-06-12T02:38:22.125 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0076749369999999997 2020-06-12T02:38:22.125 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.125 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.126 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:22.126 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.014722+0000", 2020-06-12T02:38:22.126 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00395693 2020-06-12T02:38:22.126 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.126 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.127 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:22.127 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.018679+0000", 2020-06-12T02:38:22.127 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4102000000000001e-05 2020-06-12T02:38:22.127 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.127 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.127 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:22.128 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.018773+0000", 2020-06-12T02:38:22.128 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017891845 2020-06-12T02:38:22.128 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.128 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:22.128 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.129 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.129 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.129 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:933 3.es1 3:78f01648:::smithi10729407-5:head [append 0~417792] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:38:22.129 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.156677+0000", 2020-06-12T02:38:22.129 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 49.902846836999998, 2020-06-12T02:38:22.129 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016833311, 2020-06-12T02:38:22.130 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:22.130 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:22.130 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:22.130 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:22.130 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:22.131 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 933 2020-06-12T02:38:22.131 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.131 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:22.131 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.131 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:22.131 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.156677+0000", 2020-06-12T02:38:22.132 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:22.132 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.132 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.132 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:22.132 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.156677+0000", 2020-06-12T02:38:22.133 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00062571700000000003 2020-06-12T02:38:22.133 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.133 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.133 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:22.133 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157303+0000", 2020-06-12T02:38:22.133 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0740000000000003e-06 2020-06-12T02:38:22.134 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.134 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.134 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:22.134 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157307+0000", 2020-06-12T02:38:22.134 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4419999999999998e-06 2020-06-12T02:38:22.135 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.135 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.135 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:22.135 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157315+0000", 2020-06-12T02:38:22.135 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5300000000000001e-06 2020-06-12T02:38:22.135 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.136 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.136 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:22.136 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157319+0000", 2020-06-12T02:38:22.136 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4995e-05 2020-06-12T02:38:22.136 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.137 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.137 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.137 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157354+0000", 2020-06-12T02:38:22.137 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5883000000000006e-05 2020-06-12T02:38:22.137 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.137 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.138 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:22.138 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.157419+0000", 2020-06-12T02:38:22.138 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045316280000000002 2020-06-12T02:38:22.138 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.138 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.139 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.139 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.161951+0000", 2020-06-12T02:38:22.139 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3300999999999996e-05 2020-06-12T02:38:22.139 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.139 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.139 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:22.140 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.162014+0000", 2020-06-12T02:38:22.140 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00039050100000000002 2020-06-12T02:38:22.140 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.140 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.140 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:22.141 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.162405+0000", 2020-06-12T02:38:22.141 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050686250000000002 2020-06-12T02:38:22.141 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.141 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.141 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:22.141 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.167474+0000", 2020-06-12T02:38:22.147 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.005942653 2020-06-12T02:38:22.147 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.148 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.148 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:22.148 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.173416+0000", 2020-06-12T02:38:22.148 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3962000000000006e-05 2020-06-12T02:38:22.148 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.149 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.149 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:22.149 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.173510+0000", 2020-06-12T02:38:22.149 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016833311 2020-06-12T02:38:22.149 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.149 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:22.150 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.150 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.150 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.150 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:4921 3.9s0 3:93ea0cad:::smithi10729407-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-06-12T02:38:22.150 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:43.595696+0000", 2020-06-12T02:38:22.151 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 38.463827576, 2020-06-12T02:38:22.151 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016503190000000001, 2020-06-12T02:38:22.151 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:22.151 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:22.151 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:22.151 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:22.152 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:22.152 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 4921 2020-06-12T02:38:22.152 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.152 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:22.153 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.153 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:22.153 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595696+0000", 2020-06-12T02:38:22.153 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:22.154 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.154 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.154 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:22.154 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595696+0000", 2020-06-12T02:38:22.154 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3694999999999999e-05 2020-06-12T02:38:22.155 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.155 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.155 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:22.155 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595720+0000", 2020-06-12T02:38:22.155 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1700000000000001e-06 2020-06-12T02:38:22.155 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.156 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.156 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:22.156 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595723+0000", 2020-06-12T02:38:22.156 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8849999999999999e-06 2020-06-12T02:38:22.156 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.157 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.157 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:22.157 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595731+0000", 2020-06-12T02:38:22.157 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5689999999999999e-06 2020-06-12T02:38:22.157 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.157 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.158 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:22.158 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.595733+0000", 2020-06-12T02:38:22.158 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0030116880000000002 2020-06-12T02:38:22.158 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.158 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.159 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.159 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.598745+0000", 2020-06-12T02:38:22.159 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1353999999999996e-05 2020-06-12T02:38:22.159 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.159 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.159 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:22.160 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.598806+0000", 2020-06-12T02:38:22.160 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051955819999999998 2020-06-12T02:38:22.160 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.160 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.160 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.161 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.604002+0000", 2020-06-12T02:38:22.161 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9615000000000001e-05 2020-06-12T02:38:22.161 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.161 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.161 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:22.162 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.604102+0000", 2020-06-12T02:38:22.162 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0052050960000000002 2020-06-12T02:38:22.162 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.162 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.162 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.162 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.609307+0000", 2020-06-12T02:38:22.163 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5025000000000003e-05 2020-06-12T02:38:22.163 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.163 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.163 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:22.163 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.609362+0000", 2020-06-12T02:38:22.164 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002837511 2020-06-12T02:38:22.164 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.164 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.164 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:22.164 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.612199+0000", 2020-06-12T02:38:22.164 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016503190000000001 2020-06-12T02:38:22.165 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.165 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:22.165 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.165 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.166 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.166 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:5071 3.3s0 3:c2ca77da:::smithi10729407-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-06-12T02:38:22.166 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:44.159788+0000", 2020-06-12T02:38:22.166 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 37.899735728000003, 2020-06-12T02:38:22.166 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016812273999999999, 2020-06-12T02:38:22.167 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:22.167 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:22.167 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:22.167 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:22.167 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:22.168 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 5071 2020-06-12T02:38:22.168 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.168 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:22.168 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.168 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:22.168 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159788+0000", 2020-06-12T02:38:22.169 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:22.169 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.169 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.169 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:22.169 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159788+0000", 2020-06-12T02:38:22.170 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1421000000000001e-05 2020-06-12T02:38:22.170 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.170 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.170 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:22.170 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159839+0000", 2020-06-12T02:38:22.171 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9009999999999994e-06 2020-06-12T02:38:22.171 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.171 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.171 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:22.171 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159847+0000", 2020-06-12T02:38:22.171 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3149999999999999e-05 2020-06-12T02:38:22.172 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.172 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.172 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:22.172 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159860+0000", 2020-06-12T02:38:22.172 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1749999999999996e-06 2020-06-12T02:38:22.173 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.173 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.173 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:22.173 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159866+0000", 2020-06-12T02:38:22.173 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6035999999999999e-05 2020-06-12T02:38:22.173 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.174 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.174 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.174 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159892+0000", 2020-06-12T02:38:22.174 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2627999999999998e-05 2020-06-12T02:38:22.174 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.175 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.175 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:22.175 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.159944+0000", 2020-06-12T02:38:22.175 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010270133000000001 2020-06-12T02:38:22.176 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.176 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.176 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.176 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.170214+0000", 2020-06-12T02:38:22.176 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5490999999999998e-05 2020-06-12T02:38:22.177 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.177 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.177 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:22.177 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.170310+0000", 2020-06-12T02:38:22.177 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003215828 2020-06-12T02:38:22.177 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.178 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.178 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.178 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.173526+0000", 2020-06-12T02:38:22.178 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0290000000000005e-05 2020-06-12T02:38:22.178 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.179 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.179 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:22.179 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.173616+0000", 2020-06-12T02:38:22.179 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002984221 2020-06-12T02:38:22.179 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.180 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.180 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:22.180 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:44.176600+0000", 2020-06-12T02:38:22.180 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016812273999999999 2020-06-12T02:38:22.180 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.180 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:22.181 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.181 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.181 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.181 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8123 3.es1 3:78f01648:::smithi10729407-5:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-06-12T02:38:22.181 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.690721+0000", 2020-06-12T02:38:22.182 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 29.368802528, 2020-06-12T02:38:22.182 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016503126999999999, 2020-06-12T02:38:22.182 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:22.182 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:22.182 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:22.182 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:22.183 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:22.183 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8123 2020-06-12T02:38:22.183 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.183 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:22.183 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.184 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:22.184 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690721+0000", 2020-06-12T02:38:22.184 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:22.184 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.184 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.184 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:22.185 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690721+0000", 2020-06-12T02:38:22.185 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7526e-05 2020-06-12T02:38:22.185 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.185 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.185 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:22.186 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690759+0000", 2020-06-12T02:38:22.186 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1999999999999999e-06 2020-06-12T02:38:22.186 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.186 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.186 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:22.187 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690762+0000", 2020-06-12T02:38:22.187 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9229999999999999e-06 2020-06-12T02:38:22.187 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.187 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.187 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:22.187 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690768+0000", 2020-06-12T02:38:22.188 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4480000000000001e-06 2020-06-12T02:38:22.188 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.188 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.188 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:22.188 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690770+0000", 2020-06-12T02:38:22.189 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00079317799999999996 2020-06-12T02:38:22.189 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.189 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.189 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.189 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.691563+0000", 2020-06-12T02:38:22.189 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5299e-05 2020-06-12T02:38:22.190 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.190 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.190 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:22.190 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.691619+0000", 2020-06-12T02:38:22.190 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010070076000000001 2020-06-12T02:38:22.191 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.191 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.191 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.191 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.701689+0000", 2020-06-12T02:38:22.191 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000110234 2020-06-12T02:38:22.191 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.192 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.192 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:22.192 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.701799+0000", 2020-06-12T02:38:22.192 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00037311100000000003 2020-06-12T02:38:22.192 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.193 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.193 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:22.193 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.702172+0000", 2020-06-12T02:38:22.193 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0044460050000000003 2020-06-12T02:38:22.193 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.193 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.194 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:22.194 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.706618+0000", 2020-06-12T02:38:22.194 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00043550199999999999 2020-06-12T02:38:22.194 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.194 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.195 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:22.195 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.707054+0000", 2020-06-12T02:38:22.195 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017062499999999999 2020-06-12T02:38:22.195 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.195 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.195 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:22.196 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.707224+0000", 2020-06-12T02:38:22.196 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016503126999999999 2020-06-12T02:38:22.196 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.196 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:22.196 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.197 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.197 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.197 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8124 3.es1 3:78f01648:::smithi10729407-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e33)", 2020-06-12T02:38:22.197 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.690794+0000", 2020-06-12T02:38:22.197 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 29.368729342000002, 2020-06-12T02:38:22.197 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019298497000000001, 2020-06-12T02:38:22.198 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:22.198 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:22.198 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:22.198 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:22.198 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:22.199 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8124 2020-06-12T02:38:22.199 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.199 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:22.199 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.199 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:22.199 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690794+0000", 2020-06-12T02:38:22.200 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:22.200 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.200 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.200 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:22.200 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690794+0000", 2020-06-12T02:38:22.201 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1869999999999999e-05 2020-06-12T02:38:22.201 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.201 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.201 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:22.201 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690816+0000", 2020-06-12T02:38:22.201 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.162e-06 2020-06-12T02:38:22.202 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.202 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.202 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:22.202 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690819+0000", 2020-06-12T02:38:22.202 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2280000000000003e-06 2020-06-12T02:38:22.203 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.203 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.203 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:22.203 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690826+0000", 2020-06-12T02:38:22.203 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.452e-06 2020-06-12T02:38:22.203 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.204 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.204 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:22.204 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690828+0000", 2020-06-12T02:38:22.204 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00081402600000000003 2020-06-12T02:38:22.204 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.205 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.205 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.205 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.691642+0000", 2020-06-12T02:38:22.205 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00024263 2020-06-12T02:38:22.205 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.205 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.206 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:22.206 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.691885+0000", 2020-06-12T02:38:22.206 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010968739 2020-06-12T02:38:22.207 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.207 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.207 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.207 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.702853+0000", 2020-06-12T02:38:22.208 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8985000000000002e-05 2020-06-12T02:38:22.208 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.208 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.208 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:22.208 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.702952+0000", 2020-06-12T02:38:22.209 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0046973329999999997 2020-06-12T02:38:22.209 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.209 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.209 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.209 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.707650+0000", 2020-06-12T02:38:22.209 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3773e-05 2020-06-12T02:38:22.210 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.210 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.210 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:22.210 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.707723+0000", 2020-06-12T02:38:22.210 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023692990000000001 2020-06-12T02:38:22.211 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.211 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.211 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:22.211 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.710093+0000", 2020-06-12T02:38:22.211 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019298497000000001 2020-06-12T02:38:22.211 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.212 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:22.212 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.212 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.212 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.212 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8841 3.3s0 3:ce45c70a:::smithi10729407-7:head [append 0~794624] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:22.213 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:11.820531+0000", 2020-06-12T02:38:22.213 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 10.23899237, 2020-06-12T02:38:22.213 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019581604999999998, 2020-06-12T02:38:22.213 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:22.213 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:22.213 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:22.214 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:22.214 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:22.214 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8841 2020-06-12T02:38:22.214 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.214 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:22.215 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.215 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:22.215 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.820531+0000", 2020-06-12T02:38:22.215 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:22.215 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.216 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.216 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:22.216 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.820531+0000", 2020-06-12T02:38:22.216 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00070341999999999996 2020-06-12T02:38:22.216 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.216 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.217 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:22.217 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.821235+0000", 2020-06-12T02:38:22.217 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9590000000000001e-06 2020-06-12T02:38:22.217 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.217 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.218 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:22.218 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.821244+0000", 2020-06-12T02:38:22.218 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6135000000000001e-05 2020-06-12T02:38:22.218 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.218 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.218 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:22.219 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.821260+0000", 2020-06-12T02:38:22.219 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5159999999999997e-06 2020-06-12T02:38:22.219 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.219 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.220 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:22.220 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.821267+0000", 2020-06-12T02:38:22.220 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3677999999999998e-05 2020-06-12T02:38:22.220 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.220 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.221 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.221 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.821311+0000", 2020-06-12T02:38:22.221 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2262000000000006e-05 2020-06-12T02:38:22.221 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.221 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.222 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:22.222 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.821393+0000", 2020-06-12T02:38:22.222 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00061887800000000001 2020-06-12T02:38:22.222 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.222 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.223 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:22.223 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.822012+0000", 2020-06-12T02:38:22.223 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0043135860000000003 2020-06-12T02:38:22.223 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.223 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.224 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:22.224 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.826326+0000", 2020-06-12T02:38:22.224 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013738409 2020-06-12T02:38:22.225 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.225 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.225 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:22.225 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.840064+0000", 2020-06-12T02:38:22.225 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8761999999999998e-05 2020-06-12T02:38:22.226 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.226 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.226 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:22.226 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.840113+0000", 2020-06-12T02:38:22.226 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019581604999999998 2020-06-12T02:38:22.226 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.227 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:22.227 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.227 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.227 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.227 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8866 3.3s0 3:c2ca77da:::smithi10729407-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e49)", 2020-06-12T02:38:22.228 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:11.883598+0000", 2020-06-12T02:38:22.228 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 10.175926069999999, 2020-06-12T02:38:22.228 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019905586999999999, 2020-06-12T02:38:22.228 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:22.228 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:22.228 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:22.229 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:22.229 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:22.229 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8866 2020-06-12T02:38:22.229 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.229 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:22.230 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.230 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:22.230 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.883598+0000", 2020-06-12T02:38:22.230 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:22.230 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.230 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.231 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:22.231 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.883598+0000", 2020-06-12T02:38:22.231 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2928000000000001e-05 2020-06-12T02:38:22.231 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.231 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.232 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:22.232 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.883621+0000", 2020-06-12T02:38:22.232 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2760000000000001e-06 2020-06-12T02:38:22.232 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.232 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.232 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:22.233 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.883624+0000", 2020-06-12T02:38:22.233 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2239999999999995e-06 2020-06-12T02:38:22.233 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.233 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.233 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:22.234 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.883632+0000", 2020-06-12T02:38:22.234 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2460000000000002e-06 2020-06-12T02:38:22.234 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.234 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.234 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:22.234 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.883634+0000", 2020-06-12T02:38:22.235 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0055082630000000002 2020-06-12T02:38:22.235 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.235 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.235 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.235 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.889143+0000", 2020-06-12T02:38:22.236 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9482999999999998e-05 2020-06-12T02:38:22.236 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.236 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.236 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:22.236 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.889222+0000", 2020-06-12T02:38:22.236 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0099390269999999996 2020-06-12T02:38:22.237 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.237 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.237 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.237 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.899161+0000", 2020-06-12T02:38:22.237 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000115758 2020-06-12T02:38:22.238 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.238 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.238 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:22.238 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.899277+0000", 2020-06-12T02:38:22.238 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0042263819999999999 2020-06-12T02:38:22.238 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.239 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.239 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:22.239 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.903503+0000", 2020-06-12T02:38:22.239 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019905586999999999 2020-06-12T02:38:22.239 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.240 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:22.240 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.240 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.240 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.240 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8916 3.3s0 3:ce45c70a:::smithi10729407-7:head [append 0~532480] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:22.240 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:12.042607+0000", 2020-06-12T02:38:22.241 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 10.016917114, 2020-06-12T02:38:22.241 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016706051, 2020-06-12T02:38:22.241 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:22.241 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:22.241 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:22.242 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:22.242 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:22.242 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8916 2020-06-12T02:38:22.242 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.242 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:22.243 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.243 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:22.243 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.042607+0000", 2020-06-12T02:38:22.243 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:22.243 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.243 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.244 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:22.244 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.042607+0000", 2020-06-12T02:38:22.244 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00066049299999999995 2020-06-12T02:38:22.244 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.244 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.244 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:22.245 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.043267+0000", 2020-06-12T02:38:22.245 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5260000000000004e-06 2020-06-12T02:38:22.245 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.245 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.245 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:22.246 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.043272+0000", 2020-06-12T02:38:22.246 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0111e-05 2020-06-12T02:38:22.246 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.246 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.246 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:22.247 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.043282+0000", 2020-06-12T02:38:22.247 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9699999999999998e-06 2020-06-12T02:38:22.247 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.247 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.247 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:22.247 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.043287+0000", 2020-06-12T02:38:22.248 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00079113700000000003 2020-06-12T02:38:22.248 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.248 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.248 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.248 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.044078+0000", 2020-06-12T02:38:22.249 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4621000000000003e-05 2020-06-12T02:38:22.249 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.249 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.249 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:22.249 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.044152+0000", 2020-06-12T02:38:22.249 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0043291140000000002 2020-06-12T02:38:22.250 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.250 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.250 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.250 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.048482+0000", 2020-06-12T02:38:22.250 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1329999999999997e-05 2020-06-12T02:38:22.251 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.251 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.251 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:22.251 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.048543+0000", 2020-06-12T02:38:22.251 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000466186 2020-06-12T02:38:22.252 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.252 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.252 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:22.252 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.049009+0000", 2020-06-12T02:38:22.252 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010209073000000001 2020-06-12T02:38:22.252 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.253 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.253 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:22.253 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.059218+0000", 2020-06-12T02:38:22.253 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.041e-05 2020-06-12T02:38:22.253 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.254 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.254 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:22.254 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.059269+0000", 2020-06-12T02:38:22.254 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4079999999999998e-05 2020-06-12T02:38:22.255 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.255 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.255 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:22.255 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.059313+0000", 2020-06-12T02:38:22.255 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016706051 2020-06-12T02:38:22.255 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.256 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:22.256 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.256 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.256 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.256 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:9129 3.9s0 3:977e4b9d:::smithi10729407-12:head [append 0~532480] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:22.257 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:12.645731+0000", 2020-06-12T02:38:22.257 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.4137926840000006, 2020-06-12T02:38:22.257 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017582474000000001, 2020-06-12T02:38:22.257 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:22.257 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:22.258 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:22.258 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:22.258 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:22.258 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9129 2020-06-12T02:38:22.258 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.258 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:22.259 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.259 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:22.259 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.645731+0000", 2020-06-12T02:38:22.259 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:22.259 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.260 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.260 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:22.260 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.645731+0000", 2020-06-12T02:38:22.260 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00050370799999999996 2020-06-12T02:38:22.260 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.260 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.261 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:22.261 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.646235+0000", 2020-06-12T02:38:22.261 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0410000000000003e-06 2020-06-12T02:38:22.261 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.261 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.262 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:22.262 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.646244+0000", 2020-06-12T02:38:22.262 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6384000000000001e-05 2020-06-12T02:38:22.262 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.262 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.262 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:22.263 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.646260+0000", 2020-06-12T02:38:22.263 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3039999999999999e-06 2020-06-12T02:38:22.263 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.263 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.263 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:22.264 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.646267+0000", 2020-06-12T02:38:22.264 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0970999999999997e-05 2020-06-12T02:38:22.264 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.264 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.264 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.264 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.646318+0000", 2020-06-12T02:38:22.265 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000117429 2020-06-12T02:38:22.265 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.265 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.265 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:22.265 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.646436+0000", 2020-06-12T02:38:22.266 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00084515600000000001 2020-06-12T02:38:22.266 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.266 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.266 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:22.266 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.647281+0000", 2020-06-12T02:38:22.266 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003637389 2020-06-12T02:38:22.267 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.267 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.267 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:22.267 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.650918+0000", 2020-06-12T02:38:22.267 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012318766 2020-06-12T02:38:22.268 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.268 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.268 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:22.268 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.663237+0000", 2020-06-12T02:38:22.268 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6326000000000001e-05 2020-06-12T02:38:22.268 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.269 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.269 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:22.269 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.663313+0000", 2020-06-12T02:38:22.269 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017582474000000001 2020-06-12T02:38:22.269 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.270 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:22.270 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.270 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.270 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.270 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:9130 3.9s0 3:977e4b9d:::smithi10729407-12:head [append 0~794624] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:22.270 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:12.651124+0000", 2020-06-12T02:38:22.271 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.4083999980000002, 2020-06-12T02:38:22.271 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016509482999999998, 2020-06-12T02:38:22.271 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:22.271 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:22.271 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:22.272 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:22.272 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:22.272 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9130 2020-06-12T02:38:22.272 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.272 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:22.272 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.273 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:22.273 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.651124+0000", 2020-06-12T02:38:22.273 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:22.274 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.274 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.274 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:22.274 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.651124+0000", 2020-06-12T02:38:22.274 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001444587 2020-06-12T02:38:22.275 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.275 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.275 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:22.275 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.652568+0000", 2020-06-12T02:38:22.275 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2469999999999997e-06 2020-06-12T02:38:22.276 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.276 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.276 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:22.276 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.652577+0000", 2020-06-12T02:38:22.276 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7112000000000002e-05 2020-06-12T02:38:22.277 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.277 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.277 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:22.277 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.652595+0000", 2020-06-12T02:38:22.277 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9610000000000005e-06 2020-06-12T02:38:22.278 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.278 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.278 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:22.278 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.652603+0000", 2020-06-12T02:38:22.278 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6863000000000002e-05 2020-06-12T02:38:22.278 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.279 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.279 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.279 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.652649+0000", 2020-06-12T02:38:22.279 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000111016 2020-06-12T02:38:22.279 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.279 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.280 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:22.280 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.652760+0000", 2020-06-12T02:38:22.280 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00065590500000000003 2020-06-12T02:38:22.280 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.280 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.281 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:22.281 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.653416+0000", 2020-06-12T02:38:22.281 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0099127380000000008 2020-06-12T02:38:22.281 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.281 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.282 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:22.282 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.663329+0000", 2020-06-12T02:38:22.282 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0042080609999999999 2020-06-12T02:38:22.282 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.282 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.282 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:22.283 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.667537+0000", 2020-06-12T02:38:22.283 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5993000000000001e-05 2020-06-12T02:38:22.283 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.283 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.283 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:22.283 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.667633+0000", 2020-06-12T02:38:22.284 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016509482999999998 2020-06-12T02:38:22.284 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.284 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:22.284 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.284 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.285 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.285 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:9133 3.9s0 3:977e4b9d:::smithi10729407-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e49)", 2020-06-12T02:38:22.285 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:12.654680+0000", 2020-06-12T02:38:22.285 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.4048432940000009, 2020-06-12T02:38:22.285 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017220268, 2020-06-12T02:38:22.285 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:22.286 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:22.286 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:22.286 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:22.286 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:22.286 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9133 2020-06-12T02:38:22.287 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.287 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:22.287 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.287 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:22.287 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.654680+0000", 2020-06-12T02:38:22.288 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:22.288 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.288 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.288 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:22.288 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.654680+0000", 2020-06-12T02:38:22.288 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9465999999999998e-05 2020-06-12T02:38:22.289 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.289 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.289 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:22.289 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.654720+0000", 2020-06-12T02:38:22.289 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7320000000000003e-06 2020-06-12T02:38:22.290 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.290 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.290 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:22.290 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.654726+0000", 2020-06-12T02:38:22.290 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3538e-05 2020-06-12T02:38:22.290 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.291 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.291 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:22.291 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.654739+0000", 2020-06-12T02:38:22.291 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5689999999999998e-06 2020-06-12T02:38:22.291 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.291 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.292 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:22.292 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.654744+0000", 2020-06-12T02:38:22.292 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0086310060000000001 2020-06-12T02:38:22.292 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.292 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.293 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.293 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.663375+0000", 2020-06-12T02:38:22.293 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00026180499999999999 2020-06-12T02:38:22.293 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.293 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.294 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:22.294 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.663636+0000", 2020-06-12T02:38:22.294 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0071786749999999998 2020-06-12T02:38:22.294 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.294 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.294 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.295 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.670815+0000", 2020-06-12T02:38:22.295 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1920000000000002e-05 2020-06-12T02:38:22.295 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.295 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.295 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:22.295 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.670897+0000", 2020-06-12T02:38:22.296 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001003557 2020-06-12T02:38:22.296 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.296 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.296 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:22.296 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.671901+0000", 2020-06-12T02:38:22.297 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017220268 2020-06-12T02:38:22.297 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.297 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:22.297 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.297 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.297 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.298 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:9623 3.9s0 3:93ea0cad:::smithi10729407-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e54)", 2020-06-12T02:38:22.298 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:16.878215+0000", 2020-06-12T02:38:22.298 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 5.1813082130000003, 2020-06-12T02:38:22.298 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018281585, 2020-06-12T02:38:22.298 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:22.299 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:22.299 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:22.299 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:22.299 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:22.299 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9623 2020-06-12T02:38:22.299 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.300 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:22.300 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.300 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:22.300 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.878215+0000", 2020-06-12T02:38:22.300 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:22.301 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.301 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.301 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:22.301 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.878215+0000", 2020-06-12T02:38:22.301 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6162000000000001e-05 2020-06-12T02:38:22.301 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.302 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.302 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:22.302 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.878242+0000", 2020-06-12T02:38:22.302 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1810000000000001e-06 2020-06-12T02:38:22.302 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.303 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.303 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:22.303 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.878245+0000", 2020-06-12T02:38:22.303 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5570000000000001e-06 2020-06-12T02:38:22.303 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.303 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.304 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:22.304 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.878251+0000", 2020-06-12T02:38:22.304 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6000000000000001e-06 2020-06-12T02:38:22.304 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.304 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.305 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:22.305 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.878254+0000", 2020-06-12T02:38:22.305 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0026536559999999999 2020-06-12T02:38:22.305 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.305 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.305 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.306 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.880908+0000", 2020-06-12T02:38:22.306 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010170600000000001 2020-06-12T02:38:22.306 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.306 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.306 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:22.307 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.881009+0000", 2020-06-12T02:38:22.307 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0085655460000000003 2020-06-12T02:38:22.307 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.307 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.307 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.307 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.889575+0000", 2020-06-12T02:38:22.308 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3764999999999994e-05 2020-06-12T02:38:22.308 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.308 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.308 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:22.308 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.889669+0000", 2020-06-12T02:38:22.309 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0044753060000000001 2020-06-12T02:38:22.309 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.309 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.309 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.309 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.894144+0000", 2020-06-12T02:38:22.309 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0333999999999994e-05 2020-06-12T02:38:22.310 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.310 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.310 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:22.310 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.894214+0000", 2020-06-12T02:38:22.310 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0022827720000000002 2020-06-12T02:38:22.311 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.311 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.311 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:22.311 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.896497+0000", 2020-06-12T02:38:22.311 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018281585 2020-06-12T02:38:22.311 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.312 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:22.312 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.312 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.312 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.313 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:9783 3.9s0 3:93ea0cad:::smithi10729407-19:head [append 0~458752] snapc 0=[] ondisk+write+known_if_redirected e54)", 2020-06-12T02:38:22.313 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:17.277705+0000", 2020-06-12T02:38:22.313 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.7818184449999999, 2020-06-12T02:38:22.314 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019659603000000001, 2020-06-12T02:38:22.314 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:22.314 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:22.314 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:22.314 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:22.315 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:22.315 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9783 2020-06-12T02:38:22.315 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.315 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:22.315 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.316 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:22.316 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.277705+0000", 2020-06-12T02:38:22.316 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:22.316 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.316 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.317 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:22.317 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.277705+0000", 2020-06-12T02:38:22.317 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047126399999999999 2020-06-12T02:38:22.318 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.318 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.318 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:22.318 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.278176+0000", 2020-06-12T02:38:22.319 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1250000000000001e-06 2020-06-12T02:38:22.319 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.319 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.319 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:22.320 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.278182+0000", 2020-06-12T02:38:22.320 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0852e-05 2020-06-12T02:38:22.320 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.320 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.321 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:22.321 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.278192+0000", 2020-06-12T02:38:22.321 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7230000000000004e-06 2020-06-12T02:38:22.321 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.321 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.322 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:22.322 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.278197+0000", 2020-06-12T02:38:22.322 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001912235 2020-06-12T02:38:22.322 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.322 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.322 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.323 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.280109+0000", 2020-06-12T02:38:22.323 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010991499999999999 2020-06-12T02:38:22.323 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.323 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.323 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:22.324 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.280219+0000", 2020-06-12T02:38:22.324 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0039835069999999998 2020-06-12T02:38:22.324 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.324 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.324 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.325 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.284203+0000", 2020-06-12T02:38:22.325 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4379000000000001e-05 2020-06-12T02:38:22.325 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.325 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.325 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:22.325 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.284277+0000", 2020-06-12T02:38:22.326 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00074622399999999998 2020-06-12T02:38:22.326 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.326 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.326 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:22.327 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.285023+0000", 2020-06-12T02:38:22.327 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0061177180000000003 2020-06-12T02:38:22.328 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.328 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.328 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:22.328 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.291141+0000", 2020-06-12T02:38:22.329 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0061324689999999998 2020-06-12T02:38:22.329 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.329 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.329 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:22.329 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.297274+0000", 2020-06-12T02:38:22.330 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1192000000000004e-05 2020-06-12T02:38:22.330 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.331 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.331 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:22.331 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.297365+0000", 2020-06-12T02:38:22.331 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019659603000000001 2020-06-12T02:38:22.332 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.332 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:22.332 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.332 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.332 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.333 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:9784 3.9s0 3:93ea0cad:::smithi10729407-19:head [append 0~532480] snapc 0=[] ondisk+write+known_if_redirected e54)", 2020-06-12T02:38:22.333 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:17.280918+0000", 2020-06-12T02:38:22.333 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.7786058699999998, 2020-06-12T02:38:22.333 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01666685, 2020-06-12T02:38:22.334 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:22.334 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:22.335 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:22.335 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:22.335 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:22.335 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9784 2020-06-12T02:38:22.336 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.336 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:22.336 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.337 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:22.337 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.280918+0000", 2020-06-12T02:38:22.337 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:22.337 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.337 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.338 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:22.338 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.280918+0000", 2020-06-12T02:38:22.338 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054805299999999995 2020-06-12T02:38:22.338 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.339 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.339 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:22.339 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.281466+0000", 2020-06-12T02:38:22.340 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3109999999999996e-06 2020-06-12T02:38:22.340 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.340 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.340 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:22.341 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.281471+0000", 2020-06-12T02:38:22.341 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1328000000000001e-05 2020-06-12T02:38:22.341 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.341 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.341 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:22.342 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.281482+0000", 2020-06-12T02:38:22.342 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3460000000000004e-06 2020-06-12T02:38:22.342 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.343 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.343 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:22.343 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.281487+0000", 2020-06-12T02:38:22.344 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8671000000000001e-05 2020-06-12T02:38:22.344 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.344 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.344 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.345 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.281545+0000", 2020-06-12T02:38:22.345 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6622e-05 2020-06-12T02:38:22.345 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.345 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.346 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:22.346 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.281612+0000", 2020-06-12T02:38:22.346 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0060043900000000001 2020-06-12T02:38:22.346 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.352 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.352 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.352 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.287616+0000", 2020-06-12T02:38:22.352 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6816000000000004e-05 2020-06-12T02:38:22.353 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.353 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.353 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:22.353 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.287703+0000", 2020-06-12T02:38:22.353 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00072695799999999999 2020-06-12T02:38:22.354 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.354 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.354 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:22.354 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.288430+0000", 2020-06-12T02:38:22.354 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0086734659999999995 2020-06-12T02:38:22.355 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.355 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.355 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:22.355 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.297104+0000", 2020-06-12T02:38:22.355 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00040471499999999999 2020-06-12T02:38:22.356 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.356 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.356 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:22.356 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.297508+0000", 2020-06-12T02:38:22.356 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6174000000000006e-05 2020-06-12T02:38:22.357 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.357 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.357 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:22.357 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.297585+0000", 2020-06-12T02:38:22.357 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01666685 2020-06-12T02:38:22.358 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.358 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:22.358 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.358 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.358 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.359 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11417 3.9s0 3:977e4b9d:::smithi10729407-12:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e56)", 2020-06-12T02:38:22.359 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:21.868452+0000", 2020-06-12T02:38:22.359 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.191071976, 2020-06-12T02:38:22.359 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016949413999999999, 2020-06-12T02:38:22.359 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:22.360 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:22.360 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:22.360 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:22.360 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:22.361 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11417 2020-06-12T02:38:22.361 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.361 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:22.361 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.363 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:22.363 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:21.868452+0000", 2020-06-12T02:38:22.364 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:22.364 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.364 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.364 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:22.364 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:21.868452+0000", 2020-06-12T02:38:22.365 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3787000000000003e-05 2020-06-12T02:38:22.365 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.365 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.365 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:22.365 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:21.868535+0000", 2020-06-12T02:38:22.366 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8190000000000003e-06 2020-06-12T02:38:22.366 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.366 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.366 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:22.366 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:21.868542+0000", 2020-06-12T02:38:22.367 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2604e-05 2020-06-12T02:38:22.367 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.367 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.367 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:22.367 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:21.868555+0000", 2020-06-12T02:38:22.368 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6809999999999997e-06 2020-06-12T02:38:22.368 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.368 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.368 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:22.368 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:21.868560+0000", 2020-06-12T02:38:22.368 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003581894 2020-06-12T02:38:22.369 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.369 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.369 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.369 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:21.872141+0000", 2020-06-12T02:38:22.369 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000113294 2020-06-12T02:38:22.370 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.370 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.370 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:22.370 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:21.872255+0000", 2020-06-12T02:38:22.370 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0032923620000000001 2020-06-12T02:38:22.371 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.371 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.371 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.371 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:21.875547+0000", 2020-06-12T02:38:22.371 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000101587 2020-06-12T02:38:22.372 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.372 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.372 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:22.373 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:21.875649+0000", 2020-06-12T02:38:22.374 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000537851 2020-06-12T02:38:22.374 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.374 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.374 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:22.374 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:21.876187+0000", 2020-06-12T02:38:22.375 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00091652300000000003 2020-06-12T02:38:22.375 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.375 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.375 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:22.375 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:21.877103+0000", 2020-06-12T02:38:22.376 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0081881939999999993 2020-06-12T02:38:22.376 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.376 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.376 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:22.376 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:21.885291+0000", 2020-06-12T02:38:22.376 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000109818 2020-06-12T02:38:22.377 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.377 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.377 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:22.377 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:21.885401+0000", 2020-06-12T02:38:22.377 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016949413999999999 2020-06-12T02:38:22.378 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.378 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:22.378 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.378 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.378 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.379 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11418 3.9s0 3:977e4b9d:::smithi10729407-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e56)", 2020-06-12T02:38:22.379 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:21.868608+0000", 2020-06-12T02:38:22.379 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.19091556600000001, 2020-06-12T02:38:22.379 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019154829000000002, 2020-06-12T02:38:22.379 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:22.380 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:22.381 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:22.381 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:22.381 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:22.381 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11418 2020-06-12T02:38:22.382 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.382 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:22.382 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.382 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:22.383 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:21.868608+0000", 2020-06-12T02:38:22.384 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:22.384 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.384 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.384 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:22.384 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:21.868608+0000", 2020-06-12T02:38:22.385 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6895000000000003e-05 2020-06-12T02:38:22.385 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.385 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.385 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:22.385 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:21.868665+0000", 2020-06-12T02:38:22.386 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4420000000000003e-06 2020-06-12T02:38:22.386 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.386 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.386 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:22.386 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:21.868671+0000", 2020-06-12T02:38:22.387 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4270000000000001e-05 2020-06-12T02:38:22.387 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.387 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.387 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:22.387 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:21.868686+0000", 2020-06-12T02:38:22.387 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2609999999999999e-06 2020-06-12T02:38:22.388 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.388 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.388 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:22.388 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:21.868692+0000", 2020-06-12T02:38:22.388 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036195350000000001 2020-06-12T02:38:22.389 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.389 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.389 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.389 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:21.872311+0000", 2020-06-12T02:38:22.389 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1908000000000002e-05 2020-06-12T02:38:22.390 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.390 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.390 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:22.390 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:21.872403+0000", 2020-06-12T02:38:22.390 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0043083669999999996 2020-06-12T02:38:22.391 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.391 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.391 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.391 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:21.876712+0000", 2020-06-12T02:38:22.391 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5739999999999998e-05 2020-06-12T02:38:22.392 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.392 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.392 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:22.392 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:21.876767+0000", 2020-06-12T02:38:22.392 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0093623630000000003 2020-06-12T02:38:22.393 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.394 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.394 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.394 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:21.886130+0000", 2020-06-12T02:38:22.394 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7858999999999999e-05 2020-06-12T02:38:22.395 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.395 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.395 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:22.395 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:21.886228+0000", 2020-06-12T02:38:22.395 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001535189 2020-06-12T02:38:22.396 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.396 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.396 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:22.396 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:21.887763+0000", 2020-06-12T02:38:22.396 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019154829000000002 2020-06-12T02:38:22.397 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.397 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:22.397 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.397 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.397 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:22.398 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:22.415 INFO:tasks.rados.rados.0.smithi107.stdout:5370: writing smithi10729407-14 from 1605632 to 2007040 tid 2 2020-06-12T02:38:22.415 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:22.415 INFO:tasks.rados.rados.0.smithi107.stdout:5371: seq_num 5370 ranges {4153344=745472,4898816=401408} 2020-06-12T02:38:22.416 INFO:tasks.rados.rados.0.smithi107.stdout:5371: writing smithi10729407-11 from 4153344 to 4898816 tid 1 2020-06-12T02:38:22.416 INFO:tasks.rados.rados.0.smithi107.stdout:5371: writing smithi10729407-11 from 4898816 to 5300224 tid 2 2020-06-12T02:38:22.416 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:38:22.416 INFO:tasks.rados.rados.0.smithi107.stdout:5372: seq_num 5371 ranges {0=761856,761856=401408} 2020-06-12T02:38:22.416 INFO:tasks.rados.rados.0.smithi107.stdout:5372: writing smithi10729407-20 from 0 to 761856 tid 1 2020-06-12T02:38:22.417 INFO:tasks.rados.rados.0.smithi107.stdout:5372: writing smithi10729407-20 from 761856 to 1163264 tid 2 2020-06-12T02:38:22.417 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:22.417 INFO:tasks.rados.rados.0.smithi107.stdout:5373: seq_num 5372 ranges {0=778240,778240=401408} 2020-06-12T02:38:22.417 INFO:tasks.rados.rados.0.smithi107.stdout:5373: writing smithi10729407-15 from 0 to 778240 tid 1 2020-06-12T02:38:22.417 INFO:tasks.rados.rados.0.smithi107.stdout:5373: writing smithi10729407-15 from 778240 to 1179648 tid 2 2020-06-12T02:38:22.417 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:22.418 INFO:tasks.rados.rados.0.smithi107.stdout:5374: seq_num 5373 ranges {0=794624,794624=401408} 2020-06-12T02:38:22.418 INFO:tasks.rados.rados.0.smithi107.stdout:5374: writing smithi10729407-6 from 0 to 794624 tid 1 2020-06-12T02:38:22.418 INFO:tasks.rados.rados.0.smithi107.stdout:5374: writing smithi10729407-6 from 794624 to 1196032 tid 2 2020-06-12T02:38:22.418 INFO:tasks.rados.rados.0.smithi107.stdout:5375: delete oid 3 current snap is 0 2020-06-12T02:38:22.418 INFO:tasks.rados.rados.0.smithi107.stdout:5370: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:22.419 INFO:tasks.rados.rados.0.smithi107.stdout:5370: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:22.419 INFO:tasks.rados.rados.0.smithi107.stdout:5370: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:22.419 INFO:tasks.rados.rados.0.smithi107.stdout:5370: finishing write tid 5 to smithi10729407-14 2020-06-12T02:38:22.420 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 977 (ObjNum 5369 snap 0 seq_num 5369) dirty exists 2020-06-12T02:38:22.420 INFO:tasks.rados.rados.0.smithi107.stdout:5370: left oid 14 (ObjNum 5369 snap 0 seq_num 5369) 2020-06-12T02:38:22.420 INFO:tasks.rados.rados.0.smithi107.stdout:5371: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:22.420 INFO:tasks.rados.rados.0.smithi107.stdout:5371: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:22.421 INFO:tasks.rados.rados.0.smithi107.stdout:5371: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:22.421 INFO:tasks.rados.rados.0.smithi107.stdout:5371: finishing write tid 5 to smithi10729407-11 2020-06-12T02:38:22.421 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 980 (ObjNum 5370 snap 0 seq_num 5370) dirty exists 2020-06-12T02:38:22.421 INFO:tasks.rados.rados.0.smithi107.stdout:5371: left oid 11 (ObjNum 5370 snap 0 seq_num 5370) 2020-06-12T02:38:22.421 INFO:tasks.rados.rados.0.smithi107.stdout:5372: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:22.422 INFO:tasks.rados.rados.0.smithi107.stdout:5370: done (6 left) 2020-06-12T02:38:22.422 INFO:tasks.rados.rados.0.smithi107.stdout:5371: done (5 left) 2020-06-12T02:38:22.422 INFO:tasks.rados.rados.0.smithi107.stdout:5375: done (4 left) 2020-06-12T02:38:22.422 INFO:tasks.rados.rados.0.smithi107.stdout:5376: delete oid 3 current snap is 0 2020-06-12T02:38:22.422 INFO:tasks.rados.rados.0.smithi107.stdout:5372: finishing write tid 2 to smithi10729407-20 2020-06-12T02:38:22.422 INFO:tasks.rados.rados.0.smithi107.stdout:5373: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:22.423 INFO:tasks.rados.rados.0.smithi107.stdout:5372: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:22.423 INFO:tasks.rados.rados.0.smithi107.stdout:5373: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:22.423 INFO:tasks.rados.rados.0.smithi107.stdout:5373: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:22.423 INFO:tasks.rados.rados.0.smithi107.stdout:5372: finishing write tid 5 to smithi10729407-20 2020-06-12T02:38:22.423 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 414 (ObjNum 5371 snap 0 seq_num 5371) dirty exists 2020-06-12T02:38:22.424 INFO:tasks.rados.rados.0.smithi107.stdout:5372: left oid 20 (ObjNum 5371 snap 0 seq_num 5371) 2020-06-12T02:38:22.424 INFO:tasks.rados.rados.0.smithi107.stdout:5373: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:22.424 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 949 (ObjNum 5372 snap 0 seq_num 5372) dirty exists 2020-06-12T02:38:22.425 INFO:tasks.rados.rados.0.smithi107.stdout:5373: left oid 15 (ObjNum 5372 snap 0 seq_num 5372) 2020-06-12T02:38:22.425 INFO:tasks.rados.rados.0.smithi107.stdout:5372: done (4 left) 2020-06-12T02:38:22.425 INFO:tasks.rados.rados.0.smithi107.stdout:5373: done (3 left) 2020-06-12T02:38:22.425 INFO:tasks.rados.rados.0.smithi107.stdout:5376: done (2 left) 2020-06-12T02:38:22.425 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:22.426 INFO:tasks.rados.rados.0.smithi107.stdout:5377: seq_num 5376 ranges {974848=442368,1417216=499712,1916928=303104} 2020-06-12T02:38:22.426 INFO:tasks.rados.rados.0.smithi107.stdout:5377: writing smithi10729407-18 from 974848 to 1417216 tid 1 2020-06-12T02:38:22.426 INFO:tasks.rados.rados.0.smithi107.stdout:5377: writing smithi10729407-18 from 1417216 to 1916928 tid 2 2020-06-12T02:38:22.426 INFO:tasks.rados.rados.0.smithi107.stdout:5377: writing smithi10729407-18 from 1916928 to 2220032 tid 3 2020-06-12T02:38:22.426 INFO:tasks.rados.rados.0.smithi107.stdout:5378: delete oid 12 current snap is 0 2020-06-12T02:38:22.427 INFO:tasks.rados.rados.0.smithi107.stdout:5374: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:22.427 INFO:tasks.rados.rados.0.smithi107.stdout:5374: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:22.427 INFO:tasks.rados.rados.0.smithi107.stdout:5374: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:22.427 INFO:tasks.rados.rados.0.smithi107.stdout:5374: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:22.427 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 993 (ObjNum 5373 snap 0 seq_num 5373) dirty exists 2020-06-12T02:38:22.428 INFO:tasks.rados.rados.0.smithi107.stdout:5374: left oid 6 (ObjNum 5373 snap 0 seq_num 5373) 2020-06-12T02:38:22.428 INFO:tasks.rados.rados.0.smithi107.stdout:5374: done (3 left) 2020-06-12T02:38:22.428 INFO:tasks.rados.rados.0.smithi107.stdout:5378: done (2 left) 2020-06-12T02:38:22.428 INFO:tasks.rados.rados.0.smithi107.stdout:5379: delete oid 9 current snap is 0 2020-06-12T02:38:22.428 INFO:tasks.rados.rados.0.smithi107.stdout:5377: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:22.429 INFO:tasks.rados.rados.0.smithi107.stdout:5377: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:22.429 INFO:tasks.rados.rados.0.smithi107.stdout:5377: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:22.429 INFO:tasks.rados.rados.0.smithi107.stdout:5377: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:22.429 INFO:tasks.rados.rados.0.smithi107.stdout:5379: done (2 left) 2020-06-12T02:38:22.429 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:22.430 INFO:tasks.rados.rados.0.smithi107.stdout:5380: seq_num 5379 ranges {2310144=491520,2801664=548864,3350528=253952} 2020-06-12T02:38:22.430 INFO:tasks.rados.rados.0.smithi107.stdout:5380: writing smithi10729407-13 from 2310144 to 2801664 tid 1 2020-06-12T02:38:22.430 INFO:tasks.rados.rados.0.smithi107.stdout:5380: writing smithi10729407-13 from 2801664 to 3350528 tid 2 2020-06-12T02:38:22.430 INFO:tasks.rados.rados.0.smithi107.stdout:5380: writing smithi10729407-13 from 3350528 to 3604480 tid 3 2020-06-12T02:38:22.431 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:38:22.431 INFO:tasks.rados.rados.0.smithi107.stdout:5381: seq_num 5380 ranges {0=507904,507904=434176,942080=368640} 2020-06-12T02:38:22.431 INFO:tasks.rados.rados.0.smithi107.stdout:5381: writing smithi10729407-2 from 0 to 507904 tid 1 2020-06-12T02:38:22.431 INFO:tasks.rados.rados.0.smithi107.stdout:5381: writing smithi10729407-2 from 507904 to 942080 tid 2 2020-06-12T02:38:22.431 INFO:tasks.rados.rados.0.smithi107.stdout:5381: writing smithi10729407-2 from 942080 to 1310720 tid 3 2020-06-12T02:38:22.431 INFO:tasks.rados.rados.0.smithi107.stdout:5382: delete oid 12 current snap is 0 2020-06-12T02:38:22.432 INFO:tasks.rados.rados.0.smithi107.stdout:5377: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:22.432 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 976 (ObjNum 5376 snap 0 seq_num 5376) dirty exists 2020-06-12T02:38:22.432 INFO:tasks.rados.rados.0.smithi107.stdout:5377: left oid 18 (ObjNum 5376 snap 0 seq_num 5376) 2020-06-12T02:38:22.432 INFO:tasks.rados.rados.0.smithi107.stdout:5380: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:22.432 INFO:tasks.rados.rados.0.smithi107.stdout:5380: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:22.433 INFO:tasks.rados.rados.0.smithi107.stdout:5381: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:22.433 INFO:tasks.rados.rados.0.smithi107.stdout:5380: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:22.433 INFO:tasks.rados.rados.0.smithi107.stdout:5377: done (4 left) 2020-06-12T02:38:22.433 INFO:tasks.rados.rados.0.smithi107.stdout:5382: done (3 left) 2020-06-12T02:38:22.433 INFO:tasks.rados.rados.0.smithi107.stdout:5383: delete oid 5 current snap is 0 2020-06-12T02:38:22.433 INFO:tasks.rados.rados.0.smithi107.stdout:5380: finishing write tid 5 to smithi10729407-13 2020-06-12T02:38:22.434 INFO:tasks.rados.rados.0.smithi107.stdout:5381: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:22.434 INFO:tasks.rados.rados.0.smithi107.stdout:5380: finishing write tid 6 to smithi10729407-13 2020-06-12T02:38:22.434 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1025 (ObjNum 5379 snap 0 seq_num 5379) dirty exists 2020-06-12T02:38:22.435 INFO:tasks.rados.rados.0.smithi107.stdout:5380: left oid 13 (ObjNum 5379 snap 0 seq_num 5379) 2020-06-12T02:38:22.435 INFO:tasks.rados.rados.0.smithi107.stdout:5381: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:22.435 INFO:tasks.rados.rados.0.smithi107.stdout:5380: done (3 left) 2020-06-12T02:38:22.435 INFO:tasks.rados.rados.0.smithi107.stdout:5383: done (2 left) 2020-06-12T02:38:22.435 INFO:tasks.rados.rados.0.smithi107.stdout:5384: delete oid 20 current snap is 0 2020-06-12T02:38:22.436 INFO:tasks.rados.rados.0.smithi107.stdout:5381: finishing write tid 5 to smithi10729407-2 2020-06-12T02:38:22.436 INFO:tasks.rados.rados.0.smithi107.stdout:5381: finishing write tid 6 to smithi10729407-2 2020-06-12T02:38:22.436 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 516 (ObjNum 5380 snap 0 seq_num 5380) dirty exists 2020-06-12T02:38:22.436 INFO:tasks.rados.rados.0.smithi107.stdout:5381: left oid 2 (ObjNum 5380 snap 0 seq_num 5380) 2020-06-12T02:38:22.436 INFO:tasks.rados.rados.0.smithi107.stdout:5381: done (2 left) 2020-06-12T02:38:22.437 INFO:tasks.rados.rados.0.smithi107.stdout:5384: done (1 left) 2020-06-12T02:38:22.437 INFO:tasks.rados.rados.0.smithi107.stdout:5385: delete oid 12 current snap is 0 2020-06-12T02:38:22.437 INFO:tasks.rados.rados.0.smithi107.stdout:5385: done (1 left) 2020-06-12T02:38:22.437 INFO:tasks.rados.rados.0.smithi107.stdout:5386: delete oid 13 current snap is 0 2020-06-12T02:38:22.437 INFO:tasks.rados.rados.0.smithi107.stdout:5386: done (1 left) 2020-06-12T02:38:22.438 INFO:tasks.rados.rados.0.smithi107.stdout:5387: delete oid 8 current snap is 0 2020-06-12T02:38:22.438 INFO:tasks.rados.rados.0.smithi107.stdout:5387: done (1 left) 2020-06-12T02:38:22.438 INFO:tasks.rados.rados.0.smithi107.stdout:5388: delete oid 13 current snap is 0 2020-06-12T02:38:22.438 INFO:tasks.rados.rados.0.smithi107.stdout:5388: done (1 left) 2020-06-12T02:38:22.438 INFO:tasks.rados.rados.0.smithi107.stdout:5389: delete oid 5 current snap is 0 2020-06-12T02:38:22.438 INFO:tasks.rados.rados.0.smithi107.stdout:5389: done (1 left) 2020-06-12T02:38:22.439 INFO:tasks.rados.rados.0.smithi107.stdout:5390: delete oid 18 current snap is 0 2020-06-12T02:38:22.439 INFO:tasks.rados.rados.0.smithi107.stdout:5390: done (1 left) 2020-06-12T02:38:22.439 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:38:22.439 INFO:tasks.rados.rados.0.smithi107.stdout:5391: seq_num 5390 ranges {0=679936,679936=532480,1212416=270336} 2020-06-12T02:38:22.439 INFO:tasks.rados.rados.0.smithi107.stdout:5391: writing smithi10729407-7 from 0 to 679936 tid 1 2020-06-12T02:38:22.440 INFO:tasks.rados.rados.0.smithi107.stdout:5391: writing smithi10729407-7 from 679936 to 1212416 tid 2 2020-06-12T02:38:22.440 INFO:tasks.rados.rados.0.smithi107.stdout:5391: writing smithi10729407-7 from 1212416 to 1482752 tid 3 2020-06-12T02:38:22.440 INFO:tasks.rados.rados.0.smithi107.stdout:5392: delete oid 8 current snap is 0 2020-06-12T02:38:22.440 INFO:tasks.rados.rados.0.smithi107.stdout:5391: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:22.440 INFO:tasks.rados.rados.0.smithi107.stdout:5392: done (2 left) 2020-06-12T02:38:22.440 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:38:22.441 INFO:tasks.rados.rados.0.smithi107.stdout:5393: seq_num 5392 ranges {0=712704,712704=745472,1458176=57344} 2020-06-12T02:38:22.441 INFO:tasks.rados.rados.0.smithi107.stdout:5393: writing smithi10729407-5 from 0 to 712704 tid 1 2020-06-12T02:38:22.441 INFO:tasks.rados.rados.0.smithi107.stdout:5393: writing smithi10729407-5 from 712704 to 1458176 tid 2 2020-06-12T02:38:22.441 INFO:tasks.rados.rados.0.smithi107.stdout:5393: writing smithi10729407-5 from 1458176 to 1515520 tid 3 2020-06-12T02:38:22.441 INFO:tasks.rados.rados.0.smithi107.stdout:5394: delete oid 19 current snap is 0 2020-06-12T02:38:22.442 INFO:tasks.rados.rados.0.smithi107.stdout:5391: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:22.442 INFO:tasks.rados.rados.0.smithi107.stdout:5391: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:22.442 INFO:tasks.rados.rados.0.smithi107.stdout:5391: finishing write tid 5 to smithi10729407-7 2020-06-12T02:38:22.442 INFO:tasks.rados.rados.0.smithi107.stdout:5391: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:22.442 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1031 (ObjNum 5390 snap 0 seq_num 5390) dirty exists 2020-06-12T02:38:22.442 INFO:tasks.rados.rados.0.smithi107.stdout:5391: left oid 7 (ObjNum 5390 snap 0 seq_num 5390) 2020-06-12T02:38:22.443 INFO:tasks.rados.rados.0.smithi107.stdout:5393: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:22.443 INFO:tasks.rados.rados.0.smithi107.stdout:5391: done (3 left) 2020-06-12T02:38:22.443 INFO:tasks.rados.rados.0.smithi107.stdout:5394: done (2 left) 2020-06-12T02:38:22.443 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:22.443 INFO:tasks.rados.rados.0.smithi107.stdout:5395: seq_num 5394 ranges {0=745472,745472=516096,1261568=286720} 2020-06-12T02:38:22.444 INFO:tasks.rados.rados.0.smithi107.stdout:5395: writing smithi10729407-8 from 0 to 745472 tid 1 2020-06-12T02:38:22.444 INFO:tasks.rados.rados.0.smithi107.stdout:5395: writing smithi10729407-8 from 745472 to 1261568 tid 2 2020-06-12T02:38:22.444 INFO:tasks.rados.rados.0.smithi107.stdout:5395: writing smithi10729407-8 from 1261568 to 1548288 tid 3 2020-06-12T02:38:22.444 INFO:tasks.rados.rados.0.smithi107.stdout:5396: delete oid 13 current snap is 0 2020-06-12T02:38:22.444 INFO:tasks.rados.rados.0.smithi107.stdout:5393: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:22.444 INFO:tasks.rados.rados.0.smithi107.stdout:5393: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:22.445 INFO:tasks.rados.rados.0.smithi107.stdout:5393: finishing write tid 5 to smithi10729407-5 2020-06-12T02:38:22.445 INFO:tasks.rados.rados.0.smithi107.stdout:5393: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:22.446 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 534 (ObjNum 5392 snap 0 seq_num 5392) dirty exists 2020-06-12T02:38:22.446 INFO:tasks.rados.rados.0.smithi107.stdout:5393: left oid 5 (ObjNum 5392 snap 0 seq_num 5392) 2020-06-12T02:38:22.446 INFO:tasks.rados.rados.0.smithi107.stdout:5393: done (3 left) 2020-06-12T02:38:22.446 INFO:tasks.rados.rados.0.smithi107.stdout:5396: done (2 left) 2020-06-12T02:38:22.446 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:22.446 INFO:tasks.rados.rados.0.smithi107.stdout:5397: seq_num 5396 ranges {0=778240,778240=688128,1466368=114688} 2020-06-12T02:38:22.447 INFO:tasks.rados.rados.0.smithi107.stdout:5397: writing smithi10729407-3 from 0 to 778240 tid 1 2020-06-12T02:38:22.447 INFO:tasks.rados.rados.0.smithi107.stdout:5397: writing smithi10729407-3 from 778240 to 1466368 tid 2 2020-06-12T02:38:22.447 INFO:tasks.rados.rados.0.smithi107.stdout:5397: writing smithi10729407-3 from 1466368 to 1581056 tid 3 2020-06-12T02:38:22.447 INFO:tasks.rados.rados.0.smithi107.stdout:5398: delete oid 6 current snap is 0 2020-06-12T02:38:22.447 INFO:tasks.rados.rados.0.smithi107.stdout:5395: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:22.448 INFO:tasks.rados.rados.0.smithi107.stdout:5395: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:22.448 INFO:tasks.rados.rados.0.smithi107.stdout:5395: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:22.448 INFO:tasks.rados.rados.0.smithi107.stdout:5395: finishing write tid 5 to smithi10729407-8 2020-06-12T02:38:22.449 INFO:tasks.rados.rados.0.smithi107.stdout:5395: finishing write tid 6 to smithi10729407-8 2020-06-12T02:38:22.449 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 955 (ObjNum 5394 snap 0 seq_num 5394) dirty exists 2020-06-12T02:38:22.449 INFO:tasks.rados.rados.0.smithi107.stdout:5395: left oid 8 (ObjNum 5394 snap 0 seq_num 5394) 2020-06-12T02:38:22.450 INFO:tasks.rados.rados.0.smithi107.stdout:5395: done (3 left) 2020-06-12T02:38:22.450 INFO:tasks.rados.rados.0.smithi107.stdout:5398: done (2 left) 2020-06-12T02:38:22.451 INFO:tasks.rados.rados.0.smithi107.stdout:5399: delete oid 12 current snap is 0 2020-06-12T02:38:22.451 INFO:tasks.rados.rados.0.smithi107.stdout:5397: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:22.451 INFO:tasks.rados.rados.0.smithi107.stdout:5399: done (2 left) 2020-06-12T02:38:22.451 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:22.452 INFO:tasks.rados.rados.0.smithi107.stdout:5400: seq_num 5399 ranges {1179648=425984,1605632=786432,2392064=417792} 2020-06-12T02:38:22.452 INFO:tasks.rados.rados.0.smithi107.stdout:5400: writing smithi10729407-15 from 1179648 to 1605632 tid 1 2020-06-12T02:38:22.452 INFO:tasks.rados.rados.0.smithi107.stdout:5400: writing smithi10729407-15 from 1605632 to 2392064 tid 2 2020-06-12T02:38:22.452 INFO:tasks.rados.rados.0.smithi107.stdout:5400: writing smithi10729407-15 from 2392064 to 2809856 tid 3 2020-06-12T02:38:22.452 INFO:tasks.rados.rados.0.smithi107.stdout:5401: delete oid 17 current snap is 0 2020-06-12T02:38:22.452 INFO:tasks.rados.rados.0.smithi107.stdout:5397: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:22.453 INFO:tasks.rados.rados.0.smithi107.stdout:5397: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:22.453 INFO:tasks.rados.rados.0.smithi107.stdout:5397: finishing write tid 5 to smithi10729407-3 2020-06-12T02:38:22.453 INFO:tasks.rados.rados.0.smithi107.stdout:5397: finishing write tid 6 to smithi10729407-3 2020-06-12T02:38:22.454 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 962 (ObjNum 5396 snap 0 seq_num 5396) dirty exists 2020-06-12T02:38:22.454 INFO:tasks.rados.rados.0.smithi107.stdout:5397: left oid 3 (ObjNum 5396 snap 0 seq_num 5396) 2020-06-12T02:38:22.454 INFO:tasks.rados.rados.0.smithi107.stdout:5400: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:22.454 INFO:tasks.rados.rados.0.smithi107.stdout:5397: done (3 left) 2020-06-12T02:38:22.455 INFO:tasks.rados.rados.0.smithi107.stdout:5401: done (2 left) 2020-06-12T02:38:22.455 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:22.455 INFO:tasks.rados.rados.0.smithi107.stdout:5402: seq_num 5401 ranges {1482752=458752,1941504=557056,2498560=499712,2998272=147456} 2020-06-12T02:38:22.455 INFO:tasks.rados.rados.0.smithi107.stdout:5402: writing smithi10729407-7 from 1482752 to 1941504 tid 1 2020-06-12T02:38:22.455 INFO:tasks.rados.rados.0.smithi107.stdout:5402: writing smithi10729407-7 from 1941504 to 2498560 tid 2 2020-06-12T02:38:22.456 INFO:tasks.rados.rados.0.smithi107.stdout:5402: writing smithi10729407-7 from 2498560 to 2998272 tid 3 2020-06-12T02:38:22.456 INFO:tasks.rados.rados.0.smithi107.stdout:5402: writing smithi10729407-7 from 2998272 to 3145728 tid 4 2020-06-12T02:38:22.456 INFO:tasks.rados.rados.0.smithi107.stdout:5403: delete oid 4 current snap is 0 2020-06-12T02:38:22.456 INFO:tasks.rados.rados.0.smithi107.stdout:5400: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:22.456 INFO:tasks.rados.rados.0.smithi107.stdout:5400: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:22.457 INFO:tasks.rados.rados.0.smithi107.stdout:5400: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:22.457 INFO:tasks.rados.rados.0.smithi107.stdout:5400: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:22.457 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 959 (ObjNum 5399 snap 0 seq_num 5399) dirty exists 2020-06-12T02:38:22.457 INFO:tasks.rados.rados.0.smithi107.stdout:5400: left oid 15 (ObjNum 5399 snap 0 seq_num 5399) 2020-06-12T02:38:22.457 INFO:tasks.rados.rados.0.smithi107.stdout:5402: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:22.458 INFO:tasks.rados.rados.0.smithi107.stdout:5400: done (3 left) 2020-06-12T02:38:22.458 INFO:tasks.rados.rados.0.smithi107.stdout:5403: done (2 left) 2020-06-12T02:38:22.458 INFO:tasks.rados.rados.0.smithi107.stdout:5404: delete oid 14 current snap is 0 2020-06-12T02:38:22.458 INFO:tasks.rados.rados.0.smithi107.stdout:5404: done (2 left) 2020-06-12T02:38:22.458 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:38:22.459 INFO:tasks.rados.rados.0.smithi107.stdout:5405: seq_num 5404 ranges {0=516096,516096=606208,1122304=598016} 2020-06-12T02:38:22.459 INFO:tasks.rados.rados.0.smithi107.stdout:5405: writing smithi10729407-1 from 0 to 516096 tid 1 2020-06-12T02:38:22.459 INFO:tasks.rados.rados.0.smithi107.stdout:5405: writing smithi10729407-1 from 516096 to 1122304 tid 2 2020-06-12T02:38:22.459 INFO:tasks.rados.rados.0.smithi107.stdout:5405: writing smithi10729407-1 from 1122304 to 1720320 tid 3 2020-06-12T02:38:22.459 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:22.459 INFO:tasks.rados.rados.0.smithi107.stdout:5406: seq_num 5405 ranges {0=532480,532480=491520,1024000=483328,1507328=229376} 2020-06-12T02:38:22.460 INFO:tasks.rados.rados.0.smithi107.stdout:5406: writing smithi10729407-19 from 0 to 532480 tid 1 2020-06-12T02:38:22.460 INFO:tasks.rados.rados.0.smithi107.stdout:5406: writing smithi10729407-19 from 532480 to 1024000 tid 2 2020-06-12T02:38:22.460 INFO:tasks.rados.rados.0.smithi107.stdout:5406: writing smithi10729407-19 from 1024000 to 1507328 tid 3 2020-06-12T02:38:22.460 INFO:tasks.rados.rados.0.smithi107.stdout:5406: writing smithi10729407-19 from 1507328 to 1736704 tid 4 2020-06-12T02:38:22.460 INFO:tasks.rados.rados.0.smithi107.stdout:5407: delete oid 8 current snap is 0 2020-06-12T02:38:22.461 INFO:tasks.rados.rados.0.smithi107.stdout:5402: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:22.461 INFO:tasks.rados.rados.0.smithi107.stdout:5402: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:22.461 INFO:tasks.rados.rados.0.smithi107.stdout:5402: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:22.461 INFO:tasks.rados.rados.0.smithi107.stdout:5402: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:22.461 INFO:tasks.rados.rados.0.smithi107.stdout:5402: finishing write tid 7 to smithi10729407-7 2020-06-12T02:38:22.461 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1037 (ObjNum 5401 snap 0 seq_num 5401) dirty exists 2020-06-12T02:38:22.462 INFO:tasks.rados.rados.0.smithi107.stdout:5402: left oid 7 (ObjNum 5401 snap 0 seq_num 5401) 2020-06-12T02:38:22.462 INFO:tasks.rados.rados.0.smithi107.stdout:5405: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:22.462 INFO:tasks.rados.rados.0.smithi107.stdout:5405: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:22.462 INFO:tasks.rados.rados.0.smithi107.stdout:5405: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:22.462 INFO:tasks.rados.rados.0.smithi107.stdout:5405: finishing write tid 5 to smithi10729407-1 2020-06-12T02:38:22.463 INFO:tasks.rados.rados.0.smithi107.stdout:5406: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:22.463 INFO:tasks.rados.rados.0.smithi107.stdout:5405: finishing write tid 6 to smithi10729407-1 2020-06-12T02:38:22.463 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 998 (ObjNum 5404 snap 0 seq_num 5404) dirty exists 2020-06-12T02:38:22.463 INFO:tasks.rados.rados.0.smithi107.stdout:5405: left oid 1 (ObjNum 5404 snap 0 seq_num 5404) 2020-06-12T02:38:22.463 INFO:tasks.rados.rados.0.smithi107.stdout:5402: done (4 left) 2020-06-12T02:38:22.464 INFO:tasks.rados.rados.0.smithi107.stdout:5405: done (3 left) 2020-06-12T02:38:22.464 INFO:tasks.rados.rados.0.smithi107.stdout:5407: done (2 left) 2020-06-12T02:38:22.464 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:22.464 INFO:tasks.rados.rados.0.smithi107.stdout:5408: seq_num 5407 ranges {5300224=565248,5865472=712704,6578176=450560,7028736=40960} 2020-06-12T02:38:22.464 INFO:tasks.rados.rados.0.smithi107.stdout:5408: writing smithi10729407-11 from 5300224 to 5865472 tid 1 2020-06-12T02:38:22.464 INFO:tasks.rados.rados.0.smithi107.stdout:5408: writing smithi10729407-11 from 5865472 to 6578176 tid 2 2020-06-12T02:38:22.465 INFO:tasks.rados.rados.0.smithi107.stdout:5408: writing smithi10729407-11 from 6578176 to 7028736 tid 3 2020-06-12T02:38:22.465 INFO:tasks.rados.rados.0.smithi107.stdout:5408: writing smithi10729407-11 from 7028736 to 7069696 tid 4 2020-06-12T02:38:22.465 INFO:tasks.rados.rados.0.smithi107.stdout:5409: delete oid 9 current snap is 0 2020-06-12T02:38:22.465 INFO:tasks.rados.rados.0.smithi107.stdout:5406: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:22.465 INFO:tasks.rados.rados.0.smithi107.stdout:5406: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:22.466 INFO:tasks.rados.rados.0.smithi107.stdout:5406: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:22.466 INFO:tasks.rados.rados.0.smithi107.stdout:5406: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:22.466 INFO:tasks.rados.rados.0.smithi107.stdout:5406: finishing write tid 7 to smithi10729407-19 2020-06-12T02:38:22.466 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 943 (ObjNum 5405 snap 0 seq_num 5405) dirty exists 2020-06-12T02:38:22.466 INFO:tasks.rados.rados.0.smithi107.stdout:5406: left oid 19 (ObjNum 5405 snap 0 seq_num 5405) 2020-06-12T02:38:22.467 INFO:tasks.rados.rados.0.smithi107.stdout:5406: done (3 left) 2020-06-12T02:38:22.467 INFO:tasks.rados.rados.0.smithi107.stdout:5409: done (2 left) 2020-06-12T02:38:22.467 INFO:tasks.rados.rados.0.smithi107.stdout:5410: delete oid 6 current snap is 0 2020-06-12T02:38:22.467 INFO:tasks.rados.rados.0.smithi107.stdout:5408: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:22.467 INFO:tasks.rados.rados.0.smithi107.stdout:5408: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:22.467 INFO:tasks.rados.rados.0.smithi107.stdout:5408: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:22.468 INFO:tasks.rados.rados.0.smithi107.stdout:5408: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:22.468 INFO:tasks.rados.rados.0.smithi107.stdout:5410: done (2 left) 2020-06-12T02:38:22.468 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:22.468 INFO:tasks.rados.rados.0.smithi107.stdout:5411: seq_num 5410 ranges {1581056=614400,2195456=761856,2957312=442368} 2020-06-12T02:38:22.468 INFO:tasks.rados.rados.0.smithi107.stdout:5411: writing smithi10729407-3 from 1581056 to 2195456 tid 1 2020-06-12T02:38:22.469 INFO:tasks.rados.rados.0.smithi107.stdout:5411: writing smithi10729407-3 from 2195456 to 2957312 tid 2 2020-06-12T02:38:22.469 INFO:tasks.rados.rados.0.smithi107.stdout:5411: writing smithi10729407-3 from 2957312 to 3399680 tid 3 2020-06-12T02:38:22.469 INFO:tasks.rados.rados.0.smithi107.stdout:5412: delete oid 20 current snap is 0 2020-06-12T02:38:22.469 INFO:tasks.rados.rados.0.smithi107.stdout:5408: finishing write tid 6 to smithi10729407-11 2020-06-12T02:38:22.469 INFO:tasks.rados.rados.0.smithi107.stdout:5408: finishing write tid 7 to smithi10729407-11 2020-06-12T02:38:22.470 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 986 (ObjNum 5407 snap 0 seq_num 5407) dirty exists 2020-06-12T02:38:22.470 INFO:tasks.rados.rados.0.smithi107.stdout:5408: left oid 11 (ObjNum 5407 snap 0 seq_num 5407) 2020-06-12T02:38:22.470 INFO:tasks.rados.rados.0.smithi107.stdout:5411: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:22.470 INFO:tasks.rados.rados.0.smithi107.stdout:5408: done (3 left) 2020-06-12T02:38:22.470 INFO:tasks.rados.rados.0.smithi107.stdout:5412: done (2 left) 2020-06-12T02:38:22.470 INFO:tasks.rados.rados.0.smithi107.stdout:5413: delete oid 18 current snap is 0 2020-06-12T02:38:22.471 INFO:tasks.rados.rados.0.smithi107.stdout:5413: done (2 left) 2020-06-12T02:38:22.471 INFO:tasks.rados.rados.0.smithi107.stdout:5414: delete oid 2 current snap is 0 2020-06-12T02:38:22.471 INFO:tasks.rados.rados.0.smithi107.stdout:5414: done (2 left) 2020-06-12T02:38:22.471 INFO:tasks.rados.rados.0.smithi107.stdout:5415: delete oid 14 current snap is 0 2020-06-12T02:38:22.471 INFO:tasks.rados.rados.0.smithi107.stdout:5415: done (2 left) 2020-06-12T02:38:22.472 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:38:22.472 INFO:tasks.rados.rados.0.smithi107.stdout:5416: seq_num 5415 ranges {3145728=696320,3842048=638976,4481024=409600,4890624=155648} 2020-06-12T02:38:22.472 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-06-12T02:38:22.473 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:22.397+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:22.475 INFO:tasks.rados.rados.0.smithi107.stdout:5416: writing smithi10729407-7 from 3145728 to 3842048 tid 1 2020-06-12T02:38:22.476 INFO:tasks.rados.rados.0.smithi107.stdout:5416: writing smithi10729407-7 from 3842048 to 4481024 tid 2 2020-06-12T02:38:22.476 INFO:tasks.rados.rados.0.smithi107.stdout:5416: writing smithi10729407-7 from 4481024 to 4890624 tid 3 2020-06-12T02:38:22.476 INFO:tasks.rados.rados.0.smithi107.stdout:5416: writing smithi10729407-7 from 4890624 to 5046272 tid 4 2020-06-12T02:38:22.476 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:38:22.476 INFO:tasks.rados.rados.0.smithi107.stdout:5417: seq_num 5416 ranges {0=712704,712704=516096,1228800=614400,1843200=73728} 2020-06-12T02:38:22.476 INFO:tasks.rados.rados.0.smithi107.stdout:5417: writing smithi10729407-13 from 0 to 712704 tid 1 2020-06-12T02:38:22.477 INFO:tasks.rados.rados.0.smithi107.stdout:5417: writing smithi10729407-13 from 712704 to 1228800 tid 2 2020-06-12T02:38:22.477 INFO:tasks.rados.rados.0.smithi107.stdout:5417: writing smithi10729407-13 from 1228800 to 1843200 tid 3 2020-06-12T02:38:22.477 INFO:tasks.rados.rados.0.smithi107.stdout:5417: writing smithi10729407-13 from 1843200 to 1916928 tid 4 2020-06-12T02:38:22.477 INFO:tasks.rados.rados.0.smithi107.stdout:5418: delete oid 6 current snap is 0 2020-06-12T02:38:22.477 INFO:tasks.rados.rados.0.smithi107.stdout:5411: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:22.478 INFO:tasks.rados.rados.0.smithi107.stdout:5411: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:22.478 INFO:tasks.rados.rados.0.smithi107.stdout:5411: finishing write tid 5 to smithi10729407-3 2020-06-12T02:38:22.478 INFO:tasks.rados.rados.0.smithi107.stdout:5411: finishing write tid 6 to smithi10729407-3 2020-06-12T02:38:22.478 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 966 (ObjNum 5410 snap 0 seq_num 5410) dirty exists 2020-06-12T02:38:22.478 INFO:tasks.rados.rados.0.smithi107.stdout:5411: left oid 3 (ObjNum 5410 snap 0 seq_num 5410) 2020-06-12T02:38:22.478 INFO:tasks.rados.rados.0.smithi107.stdout:5416: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:22.479 INFO:tasks.rados.rados.0.smithi107.stdout:5416: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:22.479 INFO:tasks.rados.rados.0.smithi107.stdout:5416: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:22.479 INFO:tasks.rados.rados.0.smithi107.stdout:5416: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:22.479 INFO:tasks.rados.rados.0.smithi107.stdout:5411: done (4 left) 2020-06-12T02:38:22.479 INFO:tasks.rados.rados.0.smithi107.stdout:5418: done (3 left) 2020-06-12T02:38:22.480 INFO:tasks.rados.rados.0.smithi107.stdout:5419: delete oid 9 current snap is 0 2020-06-12T02:38:22.480 INFO:tasks.rados.rados.0.smithi107.stdout:5416: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:22.480 INFO:tasks.rados.rados.0.smithi107.stdout:5419: done (3 left) 2020-06-12T02:38:22.480 INFO:tasks.rados.rados.0.smithi107.stdout:5420: delete oid 2 current snap is 0 2020-06-12T02:38:22.480 INFO:tasks.rados.rados.0.smithi107.stdout:5420: done (3 left) 2020-06-12T02:38:22.481 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:22.482 INFO:tasks.rados.rados.0.smithi107.stdout:5421: seq_num 5420 ranges {3399680=778240,4177920=458752,4636672=598016,5234688=147456} 2020-06-12T02:38:22.483 INFO:tasks.rados.rados.0.smithi107.stdout:5421: writing smithi10729407-3 from 3399680 to 4177920 tid 1 2020-06-12T02:38:22.485 INFO:tasks.rados.rados.0.smithi107.stdout:5421: writing smithi10729407-3 from 4177920 to 4636672 tid 2 2020-06-12T02:38:22.489 INFO:tasks.rados.rados.0.smithi107.stdout:5421: writing smithi10729407-3 from 4636672 to 5234688 tid 3 2020-06-12T02:38:22.490 INFO:tasks.rados.rados.0.smithi107.stdout:5421: writing smithi10729407-3 from 5234688 to 5382144 tid 4 2020-06-12T02:38:22.490 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:22.490 INFO:tasks.rados.rados.0.smithi107.stdout:5422: seq_num 5421 ranges {2809856=794624,3604480=794624,4399104=409600} 2020-06-12T02:38:22.500 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:22.497+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:22.510 INFO:tasks.rados.rados.0.smithi107.stdout:5422: writing smithi10729407-15 from 2809856 to 3604480 tid 1 2020-06-12T02:38:22.515 INFO:tasks.rados.rados.0.smithi107.stdout:5422: writing smithi10729407-15 from 3604480 to 4399104 tid 2 2020-06-12T02:38:22.517 INFO:tasks.rados.rados.0.smithi107.stdout:5422: writing smithi10729407-15 from 4399104 to 4808704 tid 3 2020-06-12T02:38:22.518 INFO:tasks.rados.rados.0.smithi107.stdout:5423: delete oid 11 current snap is 0 2020-06-12T02:38:22.518 INFO:tasks.rados.rados.0.smithi107.stdout:5417: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:22.518 INFO:tasks.rados.rados.0.smithi107.stdout:5417: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:22.518 INFO:tasks.rados.rados.0.smithi107.stdout:5416: finishing write tid 7 to smithi10729407-7 2020-06-12T02:38:22.518 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1042 (ObjNum 5415 snap 0 seq_num 5415) dirty exists 2020-06-12T02:38:22.519 INFO:tasks.rados.rados.0.smithi107.stdout:5416: left oid 7 (ObjNum 5415 snap 0 seq_num 5415) 2020-06-12T02:38:22.519 INFO:tasks.rados.rados.0.smithi107.stdout:5417: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:22.519 INFO:tasks.rados.rados.0.smithi107.stdout:5417: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:22.519 INFO:tasks.rados.rados.0.smithi107.stdout:5417: finishing write tid 6 to smithi10729407-13 2020-06-12T02:38:22.519 INFO:tasks.rados.rados.0.smithi107.stdout:5417: finishing write tid 7 to smithi10729407-13 2020-06-12T02:38:22.520 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1047 (ObjNum 5416 snap 0 seq_num 5416) dirty exists 2020-06-12T02:38:22.520 INFO:tasks.rados.rados.0.smithi107.stdout:5417: left oid 13 (ObjNum 5416 snap 0 seq_num 5416) 2020-06-12T02:38:22.520 INFO:tasks.rados.rados.0.smithi107.stdout:5421: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:22.520 INFO:tasks.rados.rados.0.smithi107.stdout:5421: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:22.520 INFO:tasks.rados.rados.0.smithi107.stdout:5421: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:22.521 INFO:tasks.rados.rados.0.smithi107.stdout:5421: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:22.521 INFO:tasks.rados.rados.0.smithi107.stdout:5421: finishing write tid 6 to smithi10729407-3 2020-06-12T02:38:22.521 INFO:tasks.rados.rados.0.smithi107.stdout:5421: finishing write tid 7 to smithi10729407-3 2020-06-12T02:38:22.521 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 971 (ObjNum 5420 snap 0 seq_num 5420) dirty exists 2020-06-12T02:38:22.521 INFO:tasks.rados.rados.0.smithi107.stdout:5421: left oid 3 (ObjNum 5420 snap 0 seq_num 5420) 2020-06-12T02:38:22.522 INFO:tasks.rados.rados.0.smithi107.stdout:5416: done (5 left) 2020-06-12T02:38:22.522 INFO:tasks.rados.rados.0.smithi107.stdout:5417: done (4 left) 2020-06-12T02:38:22.523 INFO:tasks.rados.rados.0.smithi107.stdout:5421: done (3 left) 2020-06-12T02:38:22.523 INFO:tasks.rados.rados.0.smithi107.stdout:5423: done (2 left) 2020-06-12T02:38:22.523 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:38:22.524 INFO:tasks.rados.rados.0.smithi107.stdout:5424: seq_num 5423 ranges {1515520=434176,1949696=557056,2506752=770048,3276800=278528} 2020-06-12T02:38:22.531 INFO:tasks.rados.rados.0.smithi107.stdout:5424: writing smithi10729407-5 from 1515520 to 1949696 tid 1 2020-06-12T02:38:22.535 INFO:tasks.rados.rados.0.smithi107.stdout:5424: writing smithi10729407-5 from 1949696 to 2506752 tid 2 2020-06-12T02:38:22.539 INFO:tasks.rados.rados.0.smithi107.stdout:5424: writing smithi10729407-5 from 2506752 to 3276800 tid 3 2020-06-12T02:38:22.540 INFO:tasks.rados.rados.0.smithi107.stdout:5424: writing smithi10729407-5 from 3276800 to 3555328 tid 4 2020-06-12T02:38:22.541 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:22.541 INFO:tasks.rados.rados.0.smithi107.stdout:5425: seq_num 5424 ranges {0=450560,450560=442368,892928=581632,1474560=581632} 2020-06-12T02:38:22.543 INFO:tasks.rados.rados.0.smithi107.stdout:5425: writing smithi10729407-14 from 0 to 450560 tid 1 2020-06-12T02:38:22.546 INFO:tasks.rados.rados.0.smithi107.stdout:5425: writing smithi10729407-14 from 450560 to 892928 tid 2 2020-06-12T02:38:22.549 INFO:tasks.rados.rados.0.smithi107.stdout:5425: writing smithi10729407-14 from 892928 to 1474560 tid 3 2020-06-12T02:38:22.557 INFO:tasks.rados.rados.0.smithi107.stdout:5425: writing smithi10729407-14 from 1474560 to 2056192 tid 4 2020-06-12T02:38:22.557 INFO:tasks.rados.rados.0.smithi107.stdout:5426: delete oid 12 current snap is 0 2020-06-12T02:38:22.557 INFO:tasks.rados.rados.0.smithi107.stdout:5422: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:22.558 INFO:tasks.rados.rados.0.smithi107.stdout:5422: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:22.558 INFO:tasks.rados.rados.0.smithi107.stdout:5422: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:22.558 INFO:tasks.rados.rados.0.smithi107.stdout:5422: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:22.558 INFO:tasks.rados.rados.0.smithi107.stdout:5422: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:22.578 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 964 (ObjNum 5421 snap 0 seq_num 5421) dirty exists 2020-06-12T02:38:22.578 INFO:tasks.rados.rados.0.smithi107.stdout:5422: left oid 15 (ObjNum 5421 snap 0 seq_num 5421) 2020-06-12T02:38:22.578 INFO:tasks.rados.rados.0.smithi107.stdout:5424: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:22.579 INFO:tasks.rados.rados.0.smithi107.stdout:5424: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:22.579 INFO:tasks.rados.rados.0.smithi107.stdout:5425: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:22.579 INFO:tasks.rados.rados.0.smithi107.stdout:5424: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:22.579 INFO:tasks.rados.rados.0.smithi107.stdout:5424: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:22.579 INFO:tasks.rados.rados.0.smithi107.stdout:5424: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:22.579 INFO:tasks.rados.rados.0.smithi107.stdout:5424: finishing write tid 7 to smithi10729407-5 2020-06-12T02:38:22.580 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 539 (ObjNum 5423 snap 0 seq_num 5423) dirty exists 2020-06-12T02:38:22.580 INFO:tasks.rados.rados.0.smithi107.stdout:5424: left oid 5 (ObjNum 5423 snap 0 seq_num 5423) 2020-06-12T02:38:22.580 INFO:tasks.rados.rados.0.smithi107.stdout:5422: done (4 left) 2020-06-12T02:38:22.580 INFO:tasks.rados.rados.0.smithi107.stdout:5424: done (3 left) 2020-06-12T02:38:22.580 INFO:tasks.rados.rados.0.smithi107.stdout:5426: done (2 left) 2020-06-12T02:38:22.581 INFO:tasks.rados.rados.0.smithi107.stdout:5427: delete oid 18 current snap is 0 2020-06-12T02:38:22.582 INFO:tasks.rados.rados.0.smithi107.stdout:5427: done (2 left) 2020-06-12T02:38:22.582 INFO:tasks.rados.rados.0.smithi107.stdout:5428: delete oid 12 current snap is 0 2020-06-12T02:38:22.583 INFO:tasks.rados.rados.0.smithi107.stdout:5425: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:22.583 INFO:tasks.rados.rados.0.smithi107.stdout:5425: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:22.583 INFO:tasks.rados.rados.0.smithi107.stdout:5425: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:22.583 INFO:tasks.rados.rados.0.smithi107.stdout:5425: finishing write tid 6 to smithi10729407-14 2020-06-12T02:38:22.583 INFO:tasks.rados.rados.0.smithi107.stdout:5428: done (2 left) 2020-06-12T02:38:22.584 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:38:22.584 INFO:tasks.rados.rados.0.smithi107.stdout:5429: seq_num 5428 ranges {0=516096,516096=786432,1302528=557056,1859584=262144} 2020-06-12T02:38:22.584 INFO:tasks.rados.rados.0.smithi107.stdout:5429: writing smithi10729407-20 from 0 to 516096 tid 1 2020-06-12T02:38:22.584 INFO:tasks.rados.rados.0.smithi107.stdout:5429: writing smithi10729407-20 from 516096 to 1302528 tid 2 2020-06-12T02:38:22.584 INFO:tasks.rados.rados.0.smithi107.stdout:5429: writing smithi10729407-20 from 1302528 to 1859584 tid 3 2020-06-12T02:38:22.584 INFO:tasks.rados.rados.0.smithi107.stdout:5429: writing smithi10729407-20 from 1859584 to 2121728 tid 4 2020-06-12T02:38:22.585 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:38:22.585 INFO:tasks.rados.rados.0.smithi107.stdout:5430: seq_num 5429 ranges {1564672=532480,2097152=712704,2809856=770048,3579904=122880} 2020-06-12T02:38:22.590 INFO:tasks.rados.rados.0.smithi107.stdout:5430: writing smithi10729407-16 from 1564672 to 2097152 tid 1 2020-06-12T02:38:22.609 INFO:tasks.rados.rados.0.smithi107.stdout:5430: writing smithi10729407-16 from 2097152 to 2809856 tid 2 2020-06-12T02:38:22.609 INFO:tasks.rados.rados.0.smithi107.stdout:5430: writing smithi10729407-16 from 2809856 to 3579904 tid 3 2020-06-12T02:38:22.609 INFO:tasks.rados.rados.0.smithi107.stdout:5430: writing smithi10729407-16 from 3579904 to 3702784 tid 4 2020-06-12T02:38:22.609 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:22.610 INFO:tasks.rados.rados.0.smithi107.stdout:5431: seq_num 5430 ranges {0=548864,548864=598016,1146880=524288,1671168=483328} 2020-06-12T02:38:22.610 INFO:tasks.rados.rados.0.smithi107.stdout:5431: writing smithi10729407-17 from 0 to 548864 tid 1 2020-06-12T02:38:22.610 INFO:tasks.rados.rados.0.smithi107.stdout:5431: writing smithi10729407-17 from 548864 to 1146880 tid 2 2020-06-12T02:38:22.610 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:22.605+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:22.611 INFO:tasks.rados.rados.0.smithi107.stdout:5431: writing smithi10729407-17 from 1146880 to 1671168 tid 3 2020-06-12T02:38:22.614 INFO:tasks.rados.rados.0.smithi107.stdout:5431: writing smithi10729407-17 from 1671168 to 2154496 tid 4 2020-06-12T02:38:22.615 INFO:tasks.rados.rados.0.smithi107.stdout:5432: delete oid 9 current snap is 0 2020-06-12T02:38:22.615 INFO:tasks.rados.rados.0.smithi107.stdout:5425: finishing write tid 7 to smithi10729407-14 2020-06-12T02:38:22.615 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 993 (ObjNum 5424 snap 0 seq_num 5424) dirty exists 2020-06-12T02:38:22.615 INFO:tasks.rados.rados.0.smithi107.stdout:5425: left oid 14 (ObjNum 5424 snap 0 seq_num 5424) 2020-06-12T02:38:22.616 INFO:tasks.rados.rados.0.smithi107.stdout:5429: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:22.616 INFO:tasks.rados.rados.0.smithi107.stdout:5429: finishing write tid 2 to smithi10729407-20 2020-06-12T02:38:22.616 INFO:tasks.rados.rados.0.smithi107.stdout:5429: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:22.616 INFO:tasks.rados.rados.0.smithi107.stdout:5429: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:22.616 INFO:tasks.rados.rados.0.smithi107.stdout:5429: finishing write tid 6 to smithi10729407-20 2020-06-12T02:38:22.617 INFO:tasks.rados.rados.0.smithi107.stdout:5429: finishing write tid 7 to smithi10729407-20 2020-06-12T02:38:22.617 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 421 (ObjNum 5428 snap 0 seq_num 5428) dirty exists 2020-06-12T02:38:22.617 INFO:tasks.rados.rados.0.smithi107.stdout:5429: left oid 20 (ObjNum 5428 snap 0 seq_num 5428) 2020-06-12T02:38:22.617 INFO:tasks.rados.rados.0.smithi107.stdout:5430: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:22.617 INFO:tasks.rados.rados.0.smithi107.stdout:5430: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:22.617 INFO:tasks.rados.rados.0.smithi107.stdout:5430: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:22.618 INFO:tasks.rados.rados.0.smithi107.stdout:5430: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:22.618 INFO:tasks.rados.rados.0.smithi107.stdout:5430: finishing write tid 6 to smithi10729407-16 2020-06-12T02:38:22.618 INFO:tasks.rados.rados.0.smithi107.stdout:5430: finishing write tid 7 to smithi10729407-16 2020-06-12T02:38:22.618 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 976 (ObjNum 5429 snap 0 seq_num 5429) dirty exists 2020-06-12T02:38:22.618 INFO:tasks.rados.rados.0.smithi107.stdout:5430: left oid 16 (ObjNum 5429 snap 0 seq_num 5429) 2020-06-12T02:38:22.619 INFO:tasks.rados.rados.0.smithi107.stdout:5431: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:22.620 INFO:tasks.rados.rados.0.smithi107.stdout:5431: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:22.621 INFO:tasks.rados.rados.0.smithi107.stdout:5431: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:22.621 INFO:tasks.rados.rados.0.smithi107.stdout:5425: done (5 left) 2020-06-12T02:38:22.621 INFO:tasks.rados.rados.0.smithi107.stdout:5429: done (4 left) 2020-06-12T02:38:22.621 INFO:tasks.rados.rados.0.smithi107.stdout:5430: done (3 left) 2020-06-12T02:38:22.621 INFO:tasks.rados.rados.0.smithi107.stdout:5432: done (2 left) 2020-06-12T02:38:22.622 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:38:22.622 INFO:tasks.rados.rados.0.smithi107.stdout:5433: seq_num 5432 ranges {0=581632,581632=770048,1351680=540672,1892352=294912} 2020-06-12T02:38:22.623 INFO:tasks.rados.rados.0.smithi107.stdout:5433: writing smithi10729407-6 from 0 to 581632 tid 1 2020-06-12T02:38:22.628 INFO:tasks.rados.rados.0.smithi107.stdout:5433: writing smithi10729407-6 from 581632 to 1351680 tid 2 2020-06-12T02:38:22.631 INFO:tasks.rados.rados.0.smithi107.stdout:5433: writing smithi10729407-6 from 1351680 to 1892352 tid 3 2020-06-12T02:38:22.632 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:22.633 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:38:22.633 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.633 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.ds1 56/37 ECSubWrite(tid=994, reqid=client.4512.0:11661, at_version=56'1002, trim_to=0'0, roll_forward_to=56'999))", 2020-06-12T02:38:22.634 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:22.628846+0000", 2020-06-12T02:38:22.634 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.0017664950000000001, 2020-06-12T02:38:22.634 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001793301, 2020-06-12T02:38:22.635 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:22.635 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:38:22.635 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:22.636 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.636 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:22.637 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.628846+0000", 2020-06-12T02:38:22.637 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:22.637 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.638 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.638 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:22.639 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.628846+0000", 2020-06-12T02:38:22.639 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00019534000000000001 2020-06-12T02:38:22.639 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.639 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.642 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:22.642 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.629042+0000", 2020-06-12T02:38:22.643 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2669000000000002e-05 2020-06-12T02:38:22.643 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.643 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.643 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:22.643 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.629074+0000", 2020-06-12T02:38:22.643 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0747999999999999e-05 2020-06-12T02:38:22.644 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.644 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.644 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:22.644 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.629095+0000", 2020-06-12T02:38:22.644 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6510000000000007e-06 2020-06-12T02:38:22.645 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.645 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.645 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:22.645 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.629104+0000", 2020-06-12T02:38:22.645 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1912000000000002e-05 2020-06-12T02:38:22.646 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.646 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.646 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.646 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.629136+0000", 2020-06-12T02:38:22.647 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1232e-05 2020-06-12T02:38:22.647 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.647 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.647 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:22.647 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.629147+0000", 2020-06-12T02:38:22.647 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00030055199999999999 2020-06-12T02:38:22.648 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.648 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:22.648 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.648 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.648 INFO:teuthology.orchestra.run.smithi188.stdout: ], 2020-06-12T02:38:22.649 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 1 2020-06-12T02:38:22.649 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:22.651 INFO:tasks.rados.rados.0.smithi107.stdout:5433: writing smithi10729407-6 from 1892352 to 2187264 tid 4 2020-06-12T02:38:22.651 INFO:tasks.rados.rados.0.smithi107.stdout:5434: delete oid 16 current snap is 0 2020-06-12T02:38:22.651 INFO:tasks.rados.rados.0.smithi107.stdout:5431: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:22.651 INFO:tasks.rados.rados.0.smithi107.stdout:5431: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:22.651 INFO:tasks.rados.rados.0.smithi107.stdout:5431: finishing write tid 7 to smithi10729407-17 2020-06-12T02:38:22.652 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1004 (ObjNum 5430 snap 0 seq_num 5430) dirty exists 2020-06-12T02:38:22.652 INFO:tasks.rados.rados.0.smithi107.stdout:5431: left oid 17 (ObjNum 5430 snap 0 seq_num 5430) 2020-06-12T02:38:22.652 INFO:tasks.rados.rados.0.smithi107.stdout:5433: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:22.652 INFO:tasks.rados.rados.0.smithi107.stdout:5431: done (3 left) 2020-06-12T02:38:22.652 INFO:tasks.rados.rados.0.smithi107.stdout:5434: done (2 left) 2020-06-12T02:38:22.653 INFO:tasks.rados.rados.0.smithi107.stdout:5435: delete oid 11 current snap is 0 2020-06-12T02:38:22.653 INFO:tasks.rados.rados.0.smithi107.stdout:5433: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:22.653 INFO:tasks.rados.rados.0.smithi107.stdout:5435: done (2 left) 2020-06-12T02:38:22.653 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:22.653 INFO:tasks.rados.rados.0.smithi107.stdout:5436: seq_num 5435 ranges {4808704=630784,5439488=417792,5857280=712704,6569984=475136} 2020-06-12T02:38:22.654 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-06-12T02:38:22.672 INFO:tasks.rados.rados.0.smithi107.stdout:5436: writing smithi10729407-15 from 4808704 to 5439488 tid 1 2020-06-12T02:38:22.674 INFO:tasks.rados.rados.0.smithi107.stdout:5436: writing smithi10729407-15 from 5439488 to 5857280 tid 2 2020-06-12T02:38:22.678 INFO:tasks.rados.rados.0.smithi107.stdout:5436: writing smithi10729407-15 from 5857280 to 6569984 tid 3 2020-06-12T02:38:22.680 INFO:tasks.rados.rados.0.smithi107.stdout:5436: writing smithi10729407-15 from 6569984 to 7045120 tid 4 2020-06-12T02:38:22.681 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:22.681 INFO:tasks.rados.rados.0.smithi107.stdout:5437: seq_num 5436 ranges {2154496=647168,2801664=753664,3555328=516096,4071424=335872} 2020-06-12T02:38:22.696 INFO:tasks.rados.rados.0.smithi107.stdout:5437: writing smithi10729407-17 from 2154496 to 2801664 tid 1 2020-06-12T02:38:22.700 INFO:tasks.rados.rados.0.smithi107.stdout:5437: writing smithi10729407-17 from 2801664 to 3555328 tid 2 2020-06-12T02:38:22.703 INFO:tasks.rados.rados.0.smithi107.stdout:5437: writing smithi10729407-17 from 3555328 to 4071424 tid 3 2020-06-12T02:38:22.705 INFO:tasks.rados.rados.0.smithi107.stdout:5437: writing smithi10729407-17 from 4071424 to 4407296 tid 4 2020-06-12T02:38:22.705 INFO:tasks.rados.rados.0.smithi107.stdout:5438: delete oid 2 current snap is 0 2020-06-12T02:38:22.705 INFO:tasks.rados.rados.0.smithi107.stdout:5433: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:22.706 INFO:tasks.rados.rados.0.smithi107.stdout:5433: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:22.706 INFO:tasks.rados.rados.0.smithi107.stdout:5433: finishing write tid 6 to smithi10729407-6 2020-06-12T02:38:22.706 INFO:tasks.rados.rados.0.smithi107.stdout:5433: finishing write tid 7 to smithi10729407-6 2020-06-12T02:38:22.706 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1005 (ObjNum 5432 snap 0 seq_num 5432) dirty exists 2020-06-12T02:38:22.707 INFO:tasks.rados.rados.0.smithi107.stdout:5433: left oid 6 (ObjNum 5432 snap 0 seq_num 5432) 2020-06-12T02:38:22.707 INFO:tasks.rados.rados.0.smithi107.stdout:5436: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:22.707 INFO:tasks.rados.rados.0.smithi107.stdout:5436: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:22.707 INFO:tasks.rados.rados.0.smithi107.stdout:5436: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:22.707 INFO:tasks.rados.rados.0.smithi107.stdout:5436: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:22.707 INFO:tasks.rados.rados.0.smithi107.stdout:5436: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:22.708 INFO:tasks.rados.rados.0.smithi107.stdout:5436: finishing write tid 7 to smithi10729407-15 2020-06-12T02:38:22.708 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 969 (ObjNum 5435 snap 0 seq_num 5435) dirty exists 2020-06-12T02:38:22.708 INFO:tasks.rados.rados.0.smithi107.stdout:5436: left oid 15 (ObjNum 5435 snap 0 seq_num 5435) 2020-06-12T02:38:22.709 INFO:tasks.rados.rados.0.smithi107.stdout:5437: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:22.709 INFO:tasks.rados.rados.0.smithi107.stdout:5433: done (4 left) 2020-06-12T02:38:22.709 INFO:tasks.rados.rados.0.smithi107.stdout:5436: done (3 left) 2020-06-12T02:38:22.710 INFO:tasks.rados.rados.0.smithi107.stdout:5438: done (2 left) 2020-06-12T02:38:22.710 INFO:tasks.rados.rados.0.smithi107.stdout:5439: delete oid 20 current snap is 0 2020-06-12T02:38:22.710 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:22.706+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:22.715 INFO:tasks.rados.rados.0.smithi107.stdout:5439: done (2 left) 2020-06-12T02:38:22.716 INFO:tasks.rados.rados.0.smithi107.stdout:5440: delete oid 7 current snap is 0 2020-06-12T02:38:22.719 INFO:tasks.rados.rados.0.smithi107.stdout:5440: done (2 left) 2020-06-12T02:38:22.719 INFO:tasks.rados.rados.0.smithi107.stdout:5441: delete oid 4 current snap is 0 2020-06-12T02:38:22.721 INFO:tasks.rados.rados.0.smithi107.stdout:5437: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:22.721 INFO:tasks.rados.rados.0.smithi107.stdout:5437: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:22.724 INFO:tasks.rados.rados.0.smithi107.stdout:5437: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:22.725 INFO:tasks.rados.rados.0.smithi107.stdout:5437: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:22.726 INFO:tasks.rados.rados.0.smithi107.stdout:5437: finishing write tid 7 to smithi10729407-17 2020-06-12T02:38:22.726 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1009 (ObjNum 5436 snap 0 seq_num 5436) dirty exists 2020-06-12T02:38:22.727 INFO:tasks.rados.rados.0.smithi107.stdout:5437: left oid 17 (ObjNum 5436 snap 0 seq_num 5436) 2020-06-12T02:38:22.727 INFO:tasks.rados.rados.0.smithi107.stdout:5437: done (2 left) 2020-06-12T02:38:22.727 INFO:tasks.rados.rados.0.smithi107.stdout:5441: done (1 left) 2020-06-12T02:38:22.727 INFO:tasks.rados.rados.0.smithi107.stdout:5442: delete oid 5 current snap is 0 2020-06-12T02:38:22.729 INFO:tasks.rados.rados.0.smithi107.stdout:5442: done (1 left) 2020-06-12T02:38:22.729 INFO:tasks.rados.rados.0.smithi107.stdout:5443: delete oid 6 current snap is 0 2020-06-12T02:38:22.732 INFO:tasks.rados.rados.0.smithi107.stdout:5443: done (1 left) 2020-06-12T02:38:22.732 INFO:tasks.rados.rados.0.smithi107.stdout:5444: delete oid 13 current snap is 0 2020-06-12T02:38:22.738 INFO:tasks.rados.rados.0.smithi107.stdout:5444: done (1 left) 2020-06-12T02:38:22.738 INFO:tasks.rados.rados.0.smithi107.stdout:5445: delete oid 18 current snap is 0 2020-06-12T02:38:22.740 INFO:tasks.rados.rados.0.smithi107.stdout:5445: done (1 left) 2020-06-12T02:38:22.740 INFO:tasks.rados.rados.0.smithi107.stdout:5446: delete oid 12 current snap is 0 2020-06-12T02:38:22.742 INFO:tasks.rados.rados.0.smithi107.stdout:5446: done (1 left) 2020-06-12T02:38:22.742 INFO:tasks.rados.rados.0.smithi107.stdout:5447: delete oid 14 current snap is 0 2020-06-12T02:38:22.747 INFO:tasks.rados.rados.0.smithi107.stdout:5447: done (1 left) 2020-06-12T02:38:22.748 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:22.748 INFO:tasks.rados.rados.0.smithi107.stdout:5448: seq_num 5447 ranges {0=32768} 2020-06-12T02:38:22.748 INFO:tasks.rados.rados.0.smithi107.stdout:5448: writing smithi10729407-5 from 0 to 32768 tid 1 2020-06-12T02:38:22.748 INFO:tasks.rados.rados.0.smithi107.stdout:5449: delete oid 7 current snap is 0 2020-06-12T02:38:22.751 INFO:tasks.rados.rados.0.smithi107.stdout:5449: done (2 left) 2020-06-12T02:38:22.751 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:22.751 INFO:tasks.rados.rados.0.smithi107.stdout:5450: seq_num 5449 ranges {0=65536} 2020-06-12T02:38:22.752 INFO:tasks.rados.rados.0.smithi107.stdout:5450: writing smithi10729407-8 from 0 to 65536 tid 1 2020-06-12T02:38:22.752 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:38:22.752 INFO:tasks.rados.rados.0.smithi107.stdout:5451: seq_num 5450 ranges {0=81920} 2020-06-12T02:38:22.752 INFO:tasks.rados.rados.0.smithi107.stdout:5451: writing smithi10729407-13 from 0 to 81920 tid 1 2020-06-12T02:38:22.753 INFO:tasks.rados.rados.0.smithi107.stdout:5452: delete oid 17 current snap is 0 2020-06-12T02:38:22.753 INFO:tasks.rados.rados.0.smithi107.stdout:5448: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:22.755 INFO:tasks.rados.rados.0.smithi107.stdout:5448: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:22.757 INFO:tasks.rados.rados.0.smithi107.stdout:5450: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:22.758 INFO:tasks.rados.rados.0.smithi107.stdout:5450: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:22.765 INFO:tasks.rados.rados.0.smithi107.stdout:5448: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:22.765 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 542 (ObjNum 5447 snap 0 seq_num 5447) dirty exists 2020-06-12T02:38:22.765 INFO:tasks.rados.rados.0.smithi107.stdout:5448: left oid 5 (ObjNum 5447 snap 0 seq_num 5447) 2020-06-12T02:38:22.765 INFO:tasks.rados.rados.0.smithi107.stdout:5448: done (4 left) 2020-06-12T02:38:22.766 INFO:tasks.rados.rados.0.smithi107.stdout:5452: done (3 left) 2020-06-12T02:38:22.766 INFO:tasks.rados.rados.0.smithi107.stdout:5453: delete oid 6 current snap is 0 2020-06-12T02:38:22.768 INFO:tasks.rados.rados.0.smithi107.stdout:5451: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:22.768 INFO:tasks.rados.rados.0.smithi107.stdout:5451: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:22.768 INFO:tasks.rados.rados.0.smithi107.stdout:5450: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:22.768 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 971 (ObjNum 5449 snap 0 seq_num 5449) dirty exists 2020-06-12T02:38:22.768 INFO:tasks.rados.rados.0.smithi107.stdout:5450: left oid 8 (ObjNum 5449 snap 0 seq_num 5449) 2020-06-12T02:38:22.769 INFO:tasks.rados.rados.0.smithi107.stdout:5450: done (3 left) 2020-06-12T02:38:22.769 INFO:tasks.rados.rados.0.smithi107.stdout:5453: done (2 left) 2020-06-12T02:38:22.769 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:22.769 INFO:tasks.rados.rados.0.smithi107.stdout:5454: seq_num 5453 ranges {0=131072} 2020-06-12T02:38:22.769 INFO:tasks.rados.rados.0.smithi107.stdout:5454: writing smithi10729407-9 from 0 to 131072 tid 1 2020-06-12T02:38:22.770 INFO:tasks.rados.rados.0.smithi107.stdout:5455: delete oid 2 current snap is 0 2020-06-12T02:38:22.770 INFO:tasks.rados.rados.0.smithi107.stdout:5451: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:22.770 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1052 (ObjNum 5450 snap 0 seq_num 5450) dirty exists 2020-06-12T02:38:22.770 INFO:tasks.rados.rados.0.smithi107.stdout:5451: left oid 13 (ObjNum 5450 snap 0 seq_num 5450) 2020-06-12T02:38:22.770 INFO:tasks.rados.rados.0.smithi107.stdout:5451: done (3 left) 2020-06-12T02:38:22.771 INFO:tasks.rados.rados.0.smithi107.stdout:5455: done (2 left) 2020-06-12T02:38:22.771 INFO:tasks.rados.rados.0.smithi107.stdout:5456: delete oid 7 current snap is 0 2020-06-12T02:38:22.771 INFO:tasks.rados.rados.0.smithi107.stdout:5454: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:22.772 INFO:tasks.rados.rados.0.smithi107.stdout:5454: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:22.772 INFO:tasks.rados.rados.0.smithi107.stdout:5456: done (2 left) 2020-06-12T02:38:22.772 INFO:tasks.rados.rados.0.smithi107.stdout:5457: delete oid 5 current snap is 0 2020-06-12T02:38:22.772 INFO:tasks.rados.rados.0.smithi107.stdout:5454: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:22.773 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 986 (ObjNum 5453 snap 0 seq_num 5453) dirty exists 2020-06-12T02:38:22.773 INFO:tasks.rados.rados.0.smithi107.stdout:5454: left oid 9 (ObjNum 5453 snap 0 seq_num 5453) 2020-06-12T02:38:22.773 INFO:tasks.rados.rados.0.smithi107.stdout:5454: done (2 left) 2020-06-12T02:38:22.773 INFO:tasks.rados.rados.0.smithi107.stdout:5457: done (1 left) 2020-06-12T02:38:22.773 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:38:22.774 INFO:tasks.rados.rados.0.smithi107.stdout:5458: seq_num 5457 ranges {0=196608} 2020-06-12T02:38:22.774 INFO:tasks.rados.rados.0.smithi107.stdout:5458: writing smithi10729407-11 from 0 to 196608 tid 1 2020-06-12T02:38:22.774 INFO:tasks.rados.rados.0.smithi107.stdout:5459: delete oid 16 current snap is 0 2020-06-12T02:38:22.777 INFO:tasks.rados.rados.0.smithi107.stdout:5459: done (2 left) 2020-06-12T02:38:22.777 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:38:22.777 INFO:tasks.rados.rados.0.smithi107.stdout:5460: seq_num 5459 ranges {0=229376} 2020-06-12T02:38:22.778 INFO:tasks.rados.rados.0.smithi107.stdout:5460: writing smithi10729407-16 from 0 to 229376 tid 1 2020-06-12T02:38:22.778 INFO:tasks.rados.rados.0.smithi107.stdout:5461: delete oid 18 current snap is 0 2020-06-12T02:38:22.778 INFO:tasks.rados.rados.0.smithi107.stdout:5458: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:22.780 INFO:tasks.rados.rados.0.smithi107.stdout:5458: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:22.782 INFO:tasks.rados.rados.0.smithi107.stdout:5461: done (3 left) 2020-06-12T02:38:22.782 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:22.782 INFO:tasks.rados.rados.0.smithi107.stdout:5462: seq_num 5461 ranges {0=262144} 2020-06-12T02:38:22.784 INFO:tasks.rados.rados.0.smithi107.stdout:5462: writing smithi10729407-4 from 0 to 262144 tid 1 2020-06-12T02:38:22.784 INFO:tasks.rados.rados.0.smithi107.stdout:5463: delete oid 18 current snap is 0 2020-06-12T02:38:22.784 INFO:tasks.rados.rados.0.smithi107.stdout:5458: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:22.785 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 997 (ObjNum 5457 snap 0 seq_num 5457) dirty exists 2020-06-12T02:38:22.785 INFO:tasks.rados.rados.0.smithi107.stdout:5458: left oid 11 (ObjNum 5457 snap 0 seq_num 5457) 2020-06-12T02:38:22.785 INFO:tasks.rados.rados.0.smithi107.stdout:5460: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:22.786 INFO:tasks.rados.rados.0.smithi107.stdout:5460: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:22.786 INFO:tasks.rados.rados.0.smithi107.stdout:5460: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:22.786 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 980 (ObjNum 5459 snap 0 seq_num 5459) dirty exists 2020-06-12T02:38:22.787 INFO:tasks.rados.rados.0.smithi107.stdout:5460: left oid 16 (ObjNum 5459 snap 0 seq_num 5459) 2020-06-12T02:38:22.789 INFO:tasks.rados.rados.0.smithi107.stdout:5458: done (4 left) 2020-06-12T02:38:22.789 INFO:tasks.rados.rados.0.smithi107.stdout:5460: done (3 left) 2020-06-12T02:38:22.789 INFO:tasks.rados.rados.0.smithi107.stdout:5463: done (2 left) 2020-06-12T02:38:22.789 INFO:tasks.rados.rados.0.smithi107.stdout:5464: delete oid 5 current snap is 0 2020-06-12T02:38:22.790 INFO:tasks.rados.rados.0.smithi107.stdout:5462: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:22.791 INFO:tasks.rados.rados.0.smithi107.stdout:5464: done (2 left) 2020-06-12T02:38:22.791 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:22.791 INFO:tasks.rados.rados.0.smithi107.stdout:5465: seq_num 5464 ranges {0=319488} 2020-06-12T02:38:22.792 INFO:tasks.rados.rados.0.smithi107.stdout:5465: writing smithi10729407-12 from 0 to 319488 tid 1 2020-06-12T02:38:22.793 INFO:tasks.rados.rados.0.smithi107.stdout:5466: delete oid 18 current snap is 0 2020-06-12T02:38:22.793 INFO:tasks.rados.rados.0.smithi107.stdout:5462: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:22.796 INFO:tasks.rados.rados.0.smithi107.stdout:5462: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:22.796 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1013 (ObjNum 5461 snap 0 seq_num 5461) dirty exists 2020-06-12T02:38:22.797 INFO:tasks.rados.rados.0.smithi107.stdout:5462: left oid 4 (ObjNum 5461 snap 0 seq_num 5461) 2020-06-12T02:38:22.798 INFO:tasks.rados.rados.0.smithi107.stdout:5462: done (3 left) 2020-06-12T02:38:22.798 INFO:tasks.rados.rados.0.smithi107.stdout:5466: done (2 left) 2020-06-12T02:38:22.798 INFO:tasks.rados.rados.0.smithi107.stdout:5467: delete oid 9 current snap is 0 2020-06-12T02:38:22.799 INFO:tasks.rados.rados.0.smithi107.stdout:5465: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:22.801 INFO:tasks.rados.rados.0.smithi107.stdout:5467: done (2 left) 2020-06-12T02:38:22.801 INFO:tasks.rados.rados.0.smithi107.stdout:5468: delete oid 13 current snap is 0 2020-06-12T02:38:22.802 INFO:tasks.rados.rados.0.smithi107.stdout:5465: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:22.803 INFO:tasks.rados.rados.0.smithi107.stdout:5465: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:22.804 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 948 (ObjNum 5464 snap 0 seq_num 5464) dirty exists 2020-06-12T02:38:22.804 INFO:tasks.rados.rados.0.smithi107.stdout:5465: left oid 12 (ObjNum 5464 snap 0 seq_num 5464) 2020-06-12T02:38:22.804 INFO:tasks.rados.rados.0.smithi107.stdout:5465: done (2 left) 2020-06-12T02:38:22.804 INFO:tasks.rados.rados.0.smithi107.stdout:5468: done (1 left) 2020-06-12T02:38:22.805 INFO:tasks.rados.rados.0.smithi107.stdout:5469: delete oid 16 current snap is 0 2020-06-12T02:38:22.806 INFO:tasks.rados.rados.0.smithi107.stdout:5469: done (1 left) 2020-06-12T02:38:22.806 INFO:tasks.rados.rados.0.smithi107.stdout:5470: delete oid 3 current snap is 0 2020-06-12T02:38:22.808 INFO:tasks.rados.rados.0.smithi107.stdout:5470: done (1 left) 2020-06-12T02:38:22.808 INFO:tasks.rados.rados.0.smithi107.stdout:5471: delete oid 14 current snap is 0 2020-06-12T02:38:22.809 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:22.809 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:38:22.810 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:38:22.810 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:38:22.810 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:22.811 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:22.807+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:22.811 INFO:tasks.rados.rados.0.smithi107.stdout:5471: done (1 left) 2020-06-12T02:38:22.811 INFO:tasks.rados.rados.0.smithi107.stdout:5472: delete oid 19 current snap is 0 2020-06-12T02:38:22.813 INFO:tasks.rados.rados.0.smithi107.stdout:5472: done (1 left) 2020-06-12T02:38:22.813 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:22.813 INFO:tasks.rados.rados.0.smithi107.stdout:5473: seq_num 5472 ranges {0=450560} 2020-06-12T02:38:22.815 INFO:tasks.rados.rados.0.smithi107.stdout:5473: writing smithi10729407-17 from 0 to 450560 tid 1 2020-06-12T02:38:22.816 INFO:tasks.rados.rados.0.smithi107.stdout:5474: delete oid 14 current snap is 0 2020-06-12T02:38:22.819 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-06-12T02:38:22.819 INFO:tasks.rados.rados.0.smithi107.stdout:5474: done (2 left) 2020-06-12T02:38:22.820 INFO:tasks.rados.rados.0.smithi107.stdout:5475: delete oid 3 current snap is 0 2020-06-12T02:38:22.821 INFO:tasks.rados.rados.0.smithi107.stdout:5473: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:22.822 INFO:tasks.rados.rados.0.smithi107.stdout:5475: done (2 left) 2020-06-12T02:38:22.822 INFO:tasks.rados.rados.0.smithi107.stdout:5476: delete oid 16 current snap is 0 2020-06-12T02:38:22.823 INFO:tasks.rados.rados.0.smithi107.stdout:5473: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:22.824 INFO:tasks.rados.rados.0.smithi107.stdout:5476: done (2 left) 2020-06-12T02:38:22.824 INFO:tasks.rados.rados.0.smithi107.stdout:5477: delete oid 8 current snap is 0 2020-06-12T02:38:22.826 INFO:tasks.rados.rados.0.smithi107.stdout:5473: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:22.826 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1015 (ObjNum 5472 snap 0 seq_num 5472) dirty exists 2020-06-12T02:38:22.826 INFO:tasks.rados.rados.0.smithi107.stdout:5473: left oid 17 (ObjNum 5472 snap 0 seq_num 5472) 2020-06-12T02:38:22.828 INFO:tasks.rados.rados.0.smithi107.stdout:5473: done (2 left) 2020-06-12T02:38:22.828 INFO:tasks.rados.rados.0.smithi107.stdout:5477: done (1 left) 2020-06-12T02:38:22.828 INFO:tasks.rados.rados.0.smithi107.stdout:5478: delete oid 5 current snap is 0 2020-06-12T02:38:22.829 INFO:tasks.rados.rados.0.smithi107.stdout:5478: done (1 left) 2020-06-12T02:38:22.829 INFO:tasks.rados.rados.0.smithi107.stdout:5479: delete oid 12 current snap is 0 2020-06-12T02:38:22.831 INFO:tasks.rados.rados.0.smithi107.stdout:5479: done (1 left) 2020-06-12T02:38:22.831 INFO:tasks.rados.rados.0.smithi107.stdout:5480: delete oid 9 current snap is 0 2020-06-12T02:38:22.833 INFO:tasks.rados.rados.0.smithi107.stdout:5480: done (1 left) 2020-06-12T02:38:22.834 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:22.834 INFO:tasks.rados.rados.0.smithi107.stdout:5481: seq_num 5480 ranges {450560=581632} 2020-06-12T02:38:22.839 INFO:tasks.rados.rados.0.smithi107.stdout:5481: writing smithi10729407-17 from 450560 to 1032192 tid 1 2020-06-12T02:38:22.839 INFO:tasks.rados.rados.0.smithi107.stdout:5482: delete oid 5 current snap is 0 2020-06-12T02:38:22.842 INFO:tasks.rados.rados.0.smithi107.stdout:5482: done (2 left) 2020-06-12T02:38:22.842 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:38:22.842 INFO:tasks.rados.rados.0.smithi107.stdout:5483: seq_num 5482 ranges {0=622592} 2020-06-12T02:38:22.846 INFO:tasks.rados.rados.0.smithi107.stdout:5483: writing smithi10729407-3 from 0 to 622592 tid 1 2020-06-12T02:38:22.846 INFO:tasks.rados.rados.0.smithi107.stdout:5484: delete oid 19 current snap is 0 2020-06-12T02:38:22.846 INFO:tasks.rados.rados.0.smithi107.stdout:5481: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:22.846 INFO:tasks.rados.rados.0.smithi107.stdout:5481: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:22.847 INFO:tasks.rados.rados.0.smithi107.stdout:5481: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:22.848 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1017 (ObjNum 5480 snap 0 seq_num 5480) dirty exists 2020-06-12T02:38:22.848 INFO:tasks.rados.rados.0.smithi107.stdout:5481: left oid 17 (ObjNum 5480 snap 0 seq_num 5480) 2020-06-12T02:38:22.848 INFO:tasks.rados.rados.0.smithi107.stdout:5481: done (3 left) 2020-06-12T02:38:22.849 INFO:tasks.rados.rados.0.smithi107.stdout:5484: done (2 left) 2020-06-12T02:38:22.849 INFO:tasks.rados.rados.0.smithi107.stdout:5485: delete oid 20 current snap is 0 2020-06-12T02:38:22.851 INFO:tasks.rados.rados.0.smithi107.stdout:5485: done (2 left) 2020-06-12T02:38:22.851 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:22.852 INFO:tasks.rados.rados.0.smithi107.stdout:5486: seq_num 5485 ranges {7045120=671744} 2020-06-12T02:38:22.892 INFO:tasks.rados.rados.0.smithi107.stdout:5486: writing smithi10729407-15 from 7045120 to 7716864 tid 1 2020-06-12T02:38:22.893 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:22.893 INFO:tasks.rados.rados.0.smithi107.stdout:5487: seq_num 5486 ranges {0=688128} 2020-06-12T02:38:22.896 INFO:tasks.rados.rados.0.smithi107.stdout:5487: writing smithi10729407-5 from 0 to 688128 tid 1 2020-06-12T02:38:22.896 INFO:tasks.rados.rados.0.smithi107.stdout:5488: delete oid 20 current snap is 0 2020-06-12T02:38:22.896 INFO:tasks.rados.rados.0.smithi107.stdout:5483: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:22.897 INFO:tasks.rados.rados.0.smithi107.stdout:5483: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:22.897 INFO:tasks.rados.rados.0.smithi107.stdout:5483: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:22.897 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 986 (ObjNum 5482 snap 0 seq_num 5482) dirty exists 2020-06-12T02:38:22.897 INFO:tasks.rados.rados.0.smithi107.stdout:5483: left oid 3 (ObjNum 5482 snap 0 seq_num 5482) 2020-06-12T02:38:22.901 INFO:tasks.rados.rados.0.smithi107.stdout:5483: done (4 left) 2020-06-12T02:38:22.901 INFO:tasks.rados.rados.0.smithi107.stdout:5488: done (3 left) 2020-06-12T02:38:22.901 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:22.901 INFO:tasks.rados.rados.0.smithi107.stdout:5489: seq_num 5488 ranges {1720320=720896} 2020-06-12T02:38:22.910 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:22.907+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:22.915 INFO:tasks.rados.rados.0.smithi107.stdout:5489: writing smithi10729407-1 from 1720320 to 2441216 tid 1 2020-06-12T02:38:22.915 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:22.915 INFO:tasks.rados.rados.0.smithi107.stdout:5490: seq_num 5489 ranges {196608=737280} 2020-06-12T02:38:22.920 INFO:tasks.rados.rados.0.smithi107.stdout:5490: writing smithi10729407-11 from 196608 to 933888 tid 1 2020-06-12T02:38:22.920 INFO:tasks.rados.rados.0.smithi107.stdout:5491: delete oid 9 current snap is 0 2020-06-12T02:38:22.921 INFO:tasks.rados.rados.0.smithi107.stdout:5486: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:22.921 INFO:tasks.rados.rados.0.smithi107.stdout:5486: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:22.921 INFO:tasks.rados.rados.0.smithi107.stdout:5487: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:22.922 INFO:tasks.rados.rados.0.smithi107.stdout:5486: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:22.922 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 974 (ObjNum 5485 snap 0 seq_num 5485) dirty exists 2020-06-12T02:38:22.922 INFO:tasks.rados.rados.0.smithi107.stdout:5486: left oid 15 (ObjNum 5485 snap 0 seq_num 5485) 2020-06-12T02:38:22.923 INFO:tasks.rados.rados.0.smithi107.stdout:5487: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:22.923 INFO:tasks.rados.rados.0.smithi107.stdout:5487: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:22.923 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 548 (ObjNum 5486 snap 0 seq_num 5486) dirty exists 2020-06-12T02:38:22.924 INFO:tasks.rados.rados.0.smithi107.stdout:5487: left oid 5 (ObjNum 5486 snap 0 seq_num 5486) 2020-06-12T02:38:22.925 INFO:tasks.rados.rados.0.smithi107.stdout:5486: done (5 left) 2020-06-12T02:38:22.927 INFO:tasks.rados.rados.0.smithi107.stdout:5487: done (4 left) 2020-06-12T02:38:22.927 INFO:tasks.rados.rados.0.smithi107.stdout:5491: done (3 left) 2020-06-12T02:38:22.927 INFO:tasks.rados.rados.0.smithi107.stdout:5492: delete oid 7 current snap is 0 2020-06-12T02:38:22.927 INFO:tasks.rados.rados.0.smithi107.stdout:5492: done (3 left) 2020-06-12T02:38:22.927 INFO:tasks.rados.rados.0.smithi107.stdout:5493: delete oid 9 current snap is 0 2020-06-12T02:38:22.928 INFO:tasks.rados.rados.0.smithi107.stdout:5489: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:22.929 INFO:tasks.rados.rados.0.smithi107.stdout:5490: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:22.929 INFO:tasks.rados.rados.0.smithi107.stdout:5493: done (3 left) 2020-06-12T02:38:22.929 INFO:tasks.rados.rados.0.smithi107.stdout:5494: delete oid 16 current snap is 0 2020-06-12T02:38:22.929 INFO:tasks.rados.rados.0.smithi107.stdout:5490: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:22.930 INFO:tasks.rados.rados.0.smithi107.stdout:5490: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:22.930 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1001 (ObjNum 5489 snap 0 seq_num 5489) dirty exists 2020-06-12T02:38:22.930 INFO:tasks.rados.rados.0.smithi107.stdout:5490: left oid 11 (ObjNum 5489 snap 0 seq_num 5489) 2020-06-12T02:38:22.934 INFO:tasks.rados.rados.0.smithi107.stdout:5490: done (3 left) 2020-06-12T02:38:22.934 INFO:tasks.rados.rados.0.smithi107.stdout:5494: done (2 left) 2020-06-12T02:38:22.934 INFO:tasks.rados.rados.0.smithi107.stdout:5495: delete oid 16 current snap is 0 2020-06-12T02:38:22.935 INFO:tasks.rados.rados.0.smithi107.stdout:5489: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:22.935 INFO:tasks.rados.rados.0.smithi107.stdout:5489: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:22.936 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1009 (ObjNum 5488 snap 0 seq_num 5488) dirty exists 2020-06-12T02:38:22.936 INFO:tasks.rados.rados.0.smithi107.stdout:5489: left oid 1 (ObjNum 5488 snap 0 seq_num 5488) 2020-06-12T02:38:22.938 INFO:tasks.rados.rados.0.smithi107.stdout:5489: done (2 left) 2020-06-12T02:38:22.938 INFO:tasks.rados.rados.0.smithi107.stdout:5495: done (1 left) 2020-06-12T02:38:22.938 INFO:tasks.rados.rados.0.smithi107.stdout:5496: delete oid 19 current snap is 0 2020-06-12T02:38:22.939 INFO:tasks.rados.rados.0.smithi107.stdout:5496: done (1 left) 2020-06-12T02:38:22.940 INFO:tasks.rados.rados.0.smithi107.stdout:5497: delete oid 9 current snap is 0 2020-06-12T02:38:22.942 INFO:tasks.rados.rados.0.smithi107.stdout:5497: done (1 left) 2020-06-12T02:38:22.942 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:22.942 INFO:tasks.rados.rados.0.smithi107.stdout:5498: seq_num 5497 ranges {0=466944,466944=401408} 2020-06-12T02:38:22.944 INFO:tasks.rados.rados.0.smithi107.stdout:5498: writing smithi10729407-16 from 0 to 466944 tid 1 2020-06-12T02:38:22.947 INFO:tasks.rados.rados.0.smithi107.stdout:5498: writing smithi10729407-16 from 466944 to 868352 tid 2 2020-06-12T02:38:22.947 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:22.947 INFO:tasks.rados.rados.0.smithi107.stdout:5499: seq_num 5498 ranges {0=483328,483328=401408} 2020-06-12T02:38:22.949 INFO:tasks.rados.rados.0.smithi107.stdout:5499: writing smithi10729407-13 from 0 to 483328 tid 1 2020-06-12T02:38:22.952 INFO:tasks.rados.rados.0.smithi107.stdout:5499: writing smithi10729407-13 from 483328 to 884736 tid 2 2020-06-12T02:38:22.953 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:22.953 INFO:tasks.rados.rados.0.smithi107.stdout:5500: seq_num 5499 ranges {2441216=499712,2940928=401408} 2020-06-12T02:38:22.955 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:22.955 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:38:22.955 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:38:22.955 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:38:22.956 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.956 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1206 3.5s0 3:a9510bda:::smithi10729407-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:38:22.956 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.856688+0000", 2020-06-12T02:38:22.956 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 50.093599431999998, 2020-06-12T02:38:22.956 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019116942000000001, 2020-06-12T02:38:22.957 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:22.957 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:22.957 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:22.957 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:22.957 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:22.957 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1206 2020-06-12T02:38:22.958 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.958 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:22.958 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.958 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:22.958 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856688+0000", 2020-06-12T02:38:22.959 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:22.959 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.959 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.959 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:22.959 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856688+0000", 2020-06-12T02:38:22.960 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4355e-05 2020-06-12T02:38:22.960 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.960 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.960 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:22.960 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856713+0000", 2020-06-12T02:38:22.960 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0850000000000001e-06 2020-06-12T02:38:22.961 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.961 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.961 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:22.961 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856716+0000", 2020-06-12T02:38:22.961 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4509999999999997e-06 2020-06-12T02:38:22.962 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.962 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.962 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:22.962 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856722+0000", 2020-06-12T02:38:22.962 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2879999999999999e-06 2020-06-12T02:38:22.968 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.968 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.968 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:22.968 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856724+0000", 2020-06-12T02:38:22.969 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0057507449999999998 2020-06-12T02:38:22.969 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.969 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.969 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.969 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.862475+0000", 2020-06-12T02:38:22.970 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5759000000000003e-05 2020-06-12T02:38:22.970 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.970 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.970 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:22.970 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.862571+0000", 2020-06-12T02:38:22.971 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012199018000000001 2020-06-12T02:38:22.971 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.971 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.971 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.971 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.874770+0000", 2020-06-12T02:38:22.971 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7260999999999999e-05 2020-06-12T02:38:22.972 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.972 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.972 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:22.972 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.874827+0000", 2020-06-12T02:38:22.972 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00097798000000000004 2020-06-12T02:38:22.973 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.973 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.973 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:22.973 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.875805+0000", 2020-06-12T02:38:22.973 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019116942000000001 2020-06-12T02:38:22.974 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.974 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:22.974 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.974 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.974 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.974 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1572 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~745472] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-06-12T02:38:22.975 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.886439+0000", 2020-06-12T02:38:22.975 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 49.063848714000002, 2020-06-12T02:38:22.975 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021442571000000001, 2020-06-12T02:38:22.975 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:22.975 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:22.976 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:22.976 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:22.976 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:22.976 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1572 2020-06-12T02:38:22.976 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.976 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:22.977 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.977 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:22.977 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.886439+0000", 2020-06-12T02:38:22.977 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:22.977 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.978 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.978 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:22.978 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.886439+0000", 2020-06-12T02:38:22.978 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00073027599999999997 2020-06-12T02:38:22.978 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.979 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.979 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:22.979 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887169+0000", 2020-06-12T02:38:22.979 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.49e-06 2020-06-12T02:38:22.979 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.979 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.980 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:22.980 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887175+0000", 2020-06-12T02:38:22.980 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7899999999999994e-06 2020-06-12T02:38:22.980 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.980 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.981 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:22.981 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887185+0000", 2020-06-12T02:38:22.981 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.42e-06 2020-06-12T02:38:22.981 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.981 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.981 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:22.982 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887189+0000", 2020-06-12T02:38:22.982 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1619999999999999e-05 2020-06-12T02:38:22.982 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.982 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.982 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.983 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887221+0000", 2020-06-12T02:38:22.983 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9720000000000002e-05 2020-06-12T02:38:22.983 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.983 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.983 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:22.983 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887300+0000", 2020-06-12T02:38:22.984 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047354100000000001 2020-06-12T02:38:22.984 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.984 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.984 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:22.984 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887774+0000", 2020-06-12T02:38:22.985 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0030649750000000002 2020-06-12T02:38:22.985 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.986 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.986 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:22.986 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.890839+0000", 2020-06-12T02:38:22.987 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016995321000000001 2020-06-12T02:38:22.987 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.987 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.987 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:22.987 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.907834+0000", 2020-06-12T02:38:22.988 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7417999999999999e-05 2020-06-12T02:38:22.988 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.988 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.988 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:22.988 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.907882+0000", 2020-06-12T02:38:22.988 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021442571000000001 2020-06-12T02:38:22.989 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.989 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:22.989 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:22.989 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.989 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.990 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1573 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~712704] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-06-12T02:38:22.990 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.890888+0000", 2020-06-12T02:38:22.990 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 49.059399978999998, 2020-06-12T02:38:22.990 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018868001999999998, 2020-06-12T02:38:22.990 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:22.991 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:22.991 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:22.991 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:22.991 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:22.992 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1573 2020-06-12T02:38:22.992 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.992 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:22.992 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.992 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:22.992 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.890888+0000", 2020-06-12T02:38:22.993 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:22.993 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.993 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.993 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:22.993 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.890888+0000", 2020-06-12T02:38:22.994 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000683605 2020-06-12T02:38:22.994 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.994 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.994 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:22.994 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891571+0000", 2020-06-12T02:38:22.995 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4999999999999999e-06 2020-06-12T02:38:22.995 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.995 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.995 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:22.995 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891577+0000", 2020-06-12T02:38:22.995 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.171e-05 2020-06-12T02:38:22.996 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.996 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.996 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:22.996 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891589+0000", 2020-06-12T02:38:22.996 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7650000000000001e-06 2020-06-12T02:38:22.996 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.997 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.997 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:22.997 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891593+0000", 2020-06-12T02:38:22.997 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6689000000000001e-05 2020-06-12T02:38:22.997 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.998 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.998 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:22.998 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891630+0000", 2020-06-12T02:38:22.998 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9877999999999997e-05 2020-06-12T02:38:22.998 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.999 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:22.999 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:22.999 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891710+0000", 2020-06-12T02:38:22.999 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053367200000000001 2020-06-12T02:38:22.999 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:22.999 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.000 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:23.000 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.892244+0000", 2020-06-12T02:38:23.000 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004927196 2020-06-12T02:38:23.000 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.000 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.001 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:23.001 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.897171+0000", 2020-06-12T02:38:23.001 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012496419 2020-06-12T02:38:23.001 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.001 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.002 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:23.002 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.909667+0000", 2020-06-12T02:38:23.002 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8567999999999996e-05 2020-06-12T02:38:23.002 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.002 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.003 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:23.003 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.909756+0000", 2020-06-12T02:38:23.003 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018868001999999998 2020-06-12T02:38:23.003 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:23.003 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:23.004 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:23.004 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.004 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.004 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1576 3.5s0 3:a446366d:::smithi10729407-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e26)", 2020-06-12T02:38:23.004 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.891955+0000", 2020-06-12T02:38:23.005 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 49.058332739000001, 2020-06-12T02:38:23.005 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019893246999999999, 2020-06-12T02:38:23.005 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:23.005 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:23.005 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:23.006 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:23.006 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:23.006 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1576 2020-06-12T02:38:23.006 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.006 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:23.006 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.007 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:23.007 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891955+0000", 2020-06-12T02:38:23.007 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:23.007 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.007 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.008 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:23.008 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891955+0000", 2020-06-12T02:38:23.008 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2067999999999999e-05 2020-06-12T02:38:23.008 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.009 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.009 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:23.009 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891977+0000", 2020-06-12T02:38:23.009 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.18e-06 2020-06-12T02:38:23.009 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.010 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.010 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:23.010 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891980+0000", 2020-06-12T02:38:23.010 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3629999999999999e-06 2020-06-12T02:38:23.010 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.011 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.011 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:23.011 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891987+0000", 2020-06-12T02:38:23.011 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.181e-06 2020-06-12T02:38:23.011 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.011 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.012 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:23.012 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891989+0000", 2020-06-12T02:38:23.012 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050959120000000002 2020-06-12T02:38:23.012 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.012 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.013 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:23.014 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.897085+0000", 2020-06-12T02:38:23.014 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1089999999999998e-05 2020-06-12T02:38:23.014 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.014 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.014 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:23.015 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.897166+0000", 2020-06-12T02:38:23.015 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013400235 2020-06-12T02:38:23.015 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.015 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.015 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:23.015 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.910566+0000", 2020-06-12T02:38:23.016 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000104981 2020-06-12T02:38:23.016 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.016 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.016 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:23.016 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.910671+0000", 2020-06-12T02:38:23.017 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001177237 2020-06-12T02:38:23.017 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.017 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.017 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:23.017 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.911848+0000", 2020-06-12T02:38:23.018 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019893246999999999 2020-06-12T02:38:23.018 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:23.018 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:23.018 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:23.018 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.018 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.019 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2913 3.5s0 3:a9510bda:::smithi10729407-14:head [append 0~770048,stat] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:23.019 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.070354+0000", 2020-06-12T02:38:23.019 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 45.879933938000001, 2020-06-12T02:38:23.019 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60690798599999995, 2020-06-12T02:38:23.019 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:23.020 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:23.020 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:23.020 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:23.020 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:23.021 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2913 2020-06-12T02:38:23.021 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.021 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:23.021 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.021 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:23.022 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.070354+0000", 2020-06-12T02:38:23.022 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:23.022 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.022 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.022 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:23.022 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.070354+0000", 2020-06-12T02:38:23.023 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00098478900000000002 2020-06-12T02:38:23.023 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.023 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.023 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:23.023 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071339+0000", 2020-06-12T02:38:23.024 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8349999999999993e-06 2020-06-12T02:38:23.024 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.024 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.024 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:23.024 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071347+0000", 2020-06-12T02:38:23.024 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0749e-05 2020-06-12T02:38:23.025 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.025 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.025 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:23.025 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071358+0000", 2020-06-12T02:38:23.025 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9369999999999997e-06 2020-06-12T02:38:23.026 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.026 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.026 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:23.027 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071362+0000", 2020-06-12T02:38:23.027 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0426e-05 2020-06-12T02:38:23.027 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.028 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.028 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:23.028 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071423+0000", 2020-06-12T02:38:23.028 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4129999999999998e-05 2020-06-12T02:38:23.028 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.029 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.029 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:23.029 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071487+0000", 2020-06-12T02:38:23.029 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053752599999999998 2020-06-12T02:38:23.029 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.029 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.030 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:23.030 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072024+0000", 2020-06-12T02:38:23.030 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036162579999999998 2020-06-12T02:38:23.030 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.030 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.031 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:23.031 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075640+0000", 2020-06-12T02:38:23.031 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60152201999999999 2020-06-12T02:38:23.031 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.031 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.031 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:23.032 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677162+0000", 2020-06-12T02:38:23.032 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9315999999999996e-05 2020-06-12T02:38:23.032 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.032 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.032 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:23.033 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677262+0000", 2020-06-12T02:38:23.033 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60690798599999995 2020-06-12T02:38:23.033 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:23.033 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:23.033 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:23.033 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.034 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.034 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2914 3.5s0 3:a9510bda:::smithi10729407-14:head [append 0~401408] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:23.034 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.072409+0000", 2020-06-12T02:38:23.034 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 45.877878494000001, 2020-06-12T02:38:23.034 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.61956801100000003, 2020-06-12T02:38:23.035 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:23.035 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:23.035 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:23.036 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:23.036 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:23.036 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2914 2020-06-12T02:38:23.036 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.036 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:23.037 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.037 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:23.037 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072409+0000", 2020-06-12T02:38:23.037 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:23.037 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.038 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.038 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:23.038 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072409+0000", 2020-06-12T02:38:23.038 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000446317 2020-06-12T02:38:23.038 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.039 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.039 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:23.039 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072856+0000", 2020-06-12T02:38:23.039 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2140000000000004e-06 2020-06-12T02:38:23.039 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.040 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.040 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:23.040 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072861+0000", 2020-06-12T02:38:23.040 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8710000000000003e-06 2020-06-12T02:38:23.040 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.040 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.041 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:23.041 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072870+0000", 2020-06-12T02:38:23.041 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2270000000000004e-06 2020-06-12T02:38:23.041 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.041 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.042 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:23.042 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072874+0000", 2020-06-12T02:38:23.042 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024649960000000001 2020-06-12T02:38:23.042 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.042 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.043 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:23.043 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075339+0000", 2020-06-12T02:38:23.043 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9179999999999996e-05 2020-06-12T02:38:23.043 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.043 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.043 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:23.044 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075438+0000", 2020-06-12T02:38:23.044 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60189281400000005 2020-06-12T02:38:23.044 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.044 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.044 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:23.045 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677331+0000", 2020-06-12T02:38:23.045 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012602800000000001 2020-06-12T02:38:23.045 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.045 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.045 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:23.045 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677457+0000", 2020-06-12T02:38:23.046 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00088167699999999998 2020-06-12T02:38:23.046 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.046 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.046 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:23.046 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.678339+0000", 2020-06-12T02:38:23.047 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004312995 2020-06-12T02:38:23.047 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.047 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.048 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:23.048 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.682652+0000", 2020-06-12T02:38:23.048 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0092869140000000003 2020-06-12T02:38:23.048 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.049 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.049 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:23.049 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691938+0000", 2020-06-12T02:38:23.049 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8778e-05 2020-06-12T02:38:23.049 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.049 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.050 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:23.050 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691977+0000", 2020-06-12T02:38:23.050 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.61956801100000003 2020-06-12T02:38:23.050 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:23.050 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:23.051 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:23.051 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.051 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.051 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2915 3.5s0 3:a9510bda:::smithi10729407-14:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:23.051 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.072912+0000", 2020-06-12T02:38:23.051 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 45.877375753999999, 2020-06-12T02:38:23.052 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.6192204, 2020-06-12T02:38:23.052 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:23.052 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:23.052 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:23.052 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:23.053 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:23.053 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2915 2020-06-12T02:38:23.053 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.053 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:23.053 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.053 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:23.054 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072912+0000", 2020-06-12T02:38:23.054 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:23.054 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.054 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.054 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:23.055 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072912+0000", 2020-06-12T02:38:23.055 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9201e-05 2020-06-12T02:38:23.055 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.055 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.055 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:23.055 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072951+0000", 2020-06-12T02:38:23.056 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5480000000000002e-06 2020-06-12T02:38:23.056 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.056 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.056 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:23.056 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072955+0000", 2020-06-12T02:38:23.057 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3879999999999997e-06 2020-06-12T02:38:23.057 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.057 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.057 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:23.057 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072961+0000", 2020-06-12T02:38:23.057 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6589999999999999e-06 2020-06-12T02:38:23.058 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.058 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.058 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:23.058 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072964+0000", 2020-06-12T02:38:23.058 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025221039999999998 2020-06-12T02:38:23.059 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.059 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.059 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:23.059 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075486+0000", 2020-06-12T02:38:23.059 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5038999999999998e-05 2020-06-12T02:38:23.059 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.060 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.060 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:23.060 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075561+0000", 2020-06-12T02:38:23.060 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60452397499999999 2020-06-12T02:38:23.060 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.061 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.061 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:23.061 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680085+0000", 2020-06-12T02:38:23.061 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1398999999999999e-05 2020-06-12T02:38:23.061 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.061 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.062 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:23.062 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680156+0000", 2020-06-12T02:38:23.062 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000274465 2020-06-12T02:38:23.062 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.062 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.063 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:23.063 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680431+0000", 2020-06-12T02:38:23.063 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025071999999999998 2020-06-12T02:38:23.063 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.063 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.063 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:23.064 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.682938+0000", 2020-06-12T02:38:23.064 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0091116449999999998 2020-06-12T02:38:23.064 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.064 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.064 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:23.064 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692050+0000", 2020-06-12T02:38:23.065 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2776999999999997e-05 2020-06-12T02:38:23.065 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.065 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.065 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:23.065 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692132+0000", 2020-06-12T02:38:23.066 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.6192204 2020-06-12T02:38:23.067 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:23.067 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:23.067 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:23.067 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.068 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.068 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2916 3.5s0 3:a9510bda:::smithi10729407-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:38:23.068 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.072986+0000", 2020-06-12T02:38:23.068 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 45.877301514999999, 2020-06-12T02:38:23.069 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.62063941099999997, 2020-06-12T02:38:23.069 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:23.069 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:23.069 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:23.070 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:23.070 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:23.070 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2916 2020-06-12T02:38:23.070 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.070 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:23.071 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.071 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:23.071 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072986+0000", 2020-06-12T02:38:23.071 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:23.071 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.071 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.072 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:23.072 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072986+0000", 2020-06-12T02:38:23.072 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2101000000000001e-05 2020-06-12T02:38:23.072 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.072 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.072 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:23.073 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073008+0000", 2020-06-12T02:38:23.073 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1549999999999999e-06 2020-06-12T02:38:23.073 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.073 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.073 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:23.074 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073011+0000", 2020-06-12T02:38:23.074 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2360000000000001e-06 2020-06-12T02:38:23.074 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.074 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.074 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:23.075 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073018+0000", 2020-06-12T02:38:23.075 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2790000000000001e-06 2020-06-12T02:38:23.075 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.075 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.076 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:23.076 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073020+0000", 2020-06-12T02:38:23.076 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025637670000000002 2020-06-12T02:38:23.076 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.076 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.076 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:23.077 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075584+0000", 2020-06-12T02:38:23.077 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1814999999999998e-05 2020-06-12T02:38:23.077 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.077 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.077 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:23.078 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075636+0000", 2020-06-12T02:38:23.078 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60499943199999995 2020-06-12T02:38:23.078 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.078 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.078 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:23.078 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680635+0000", 2020-06-12T02:38:23.079 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0968000000000002e-05 2020-06-12T02:38:23.079 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.079 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.079 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:23.079 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680706+0000", 2020-06-12T02:38:23.080 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011777101999999999 2020-06-12T02:38:23.080 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.080 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.080 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:23.080 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692483+0000", 2020-06-12T02:38:23.081 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5890999999999995e-05 2020-06-12T02:38:23.081 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.081 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.081 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:23.082 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692569+0000", 2020-06-12T02:38:23.082 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001056665 2020-06-12T02:38:23.082 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.082 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.082 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:23.083 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.693626+0000", 2020-06-12T02:38:23.083 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.62063941099999997 2020-06-12T02:38:23.083 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:23.083 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:23.083 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:23.084 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.084 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.084 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2922 3.ds1 3:b983fd28:::smithi10729407-1:head [append 0~466944] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:23.084 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.097192+0000", 2020-06-12T02:38:23.084 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 45.853096147999999, 2020-06-12T02:38:23.085 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58672644699999998, 2020-06-12T02:38:23.085 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:23.085 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:23.085 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:23.085 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:23.085 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:23.086 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2922 2020-06-12T02:38:23.086 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.086 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:23.086 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.086 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:23.087 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097192+0000", 2020-06-12T02:38:23.087 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:23.087 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.087 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.087 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:23.087 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097192+0000", 2020-06-12T02:38:23.088 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047206100000000002 2020-06-12T02:38:23.088 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.088 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.088 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:23.088 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097664+0000", 2020-06-12T02:38:23.089 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8359999999999999e-06 2020-06-12T02:38:23.089 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.089 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.089 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:23.090 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097668+0000", 2020-06-12T02:38:23.090 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0815999999999999e-05 2020-06-12T02:38:23.090 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.090 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.090 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:23.091 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097679+0000", 2020-06-12T02:38:23.091 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.028e-06 2020-06-12T02:38:23.091 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.091 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.091 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:23.092 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097684+0000", 2020-06-12T02:38:23.092 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3932999999999999e-05 2020-06-12T02:38:23.092 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.092 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.092 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:23.093 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097748+0000", 2020-06-12T02:38:23.093 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2214999999999999e-05 2020-06-12T02:38:23.093 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.093 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.093 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:23.094 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097820+0000", 2020-06-12T02:38:23.094 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00041583800000000003 2020-06-12T02:38:23.094 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.094 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.094 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:23.094 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.098236+0000", 2020-06-12T02:38:23.095 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0020339960000000002 2020-06-12T02:38:23.095 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.095 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.095 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:23.095 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.100270+0000", 2020-06-12T02:38:23.096 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58355657299999997 2020-06-12T02:38:23.097 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.097 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.097 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:23.097 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.683827+0000", 2020-06-12T02:38:23.098 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1150999999999995e-05 2020-06-12T02:38:23.098 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.098 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.098 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:23.098 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.683918+0000", 2020-06-12T02:38:23.098 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58672644699999998 2020-06-12T02:38:23.099 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:23.099 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:23.099 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:23.099 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.099 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.100 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2923 3.ds1 3:b983fd28:::smithi10729407-1:head [append 0~614400] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:23.100 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.100504+0000", 2020-06-12T02:38:23.100 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 45.849783506999998, 2020-06-12T02:38:23.100 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58853946999999995, 2020-06-12T02:38:23.100 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:23.101 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:23.101 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:23.101 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:23.101 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:23.101 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2923 2020-06-12T02:38:23.101 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.102 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:23.108 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.108 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:23.109 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.100504+0000", 2020-06-12T02:38:23.109 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:23.109 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.109 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.109 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:23.110 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.100504+0000", 2020-06-12T02:38:23.110 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00068395399999999996 2020-06-12T02:38:23.110 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.110 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.110 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:23.111 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101188+0000", 2020-06-12T02:38:23.111 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4050000000000003e-06 2020-06-12T02:38:23.111 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.111 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.111 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:23.111 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101196+0000", 2020-06-12T02:38:23.112 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.328e-05 2020-06-12T02:38:23.112 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.112 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.112 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:23.112 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101209+0000", 2020-06-12T02:38:23.113 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3419999999999996e-06 2020-06-12T02:38:23.113 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.113 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.113 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:23.113 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101215+0000", 2020-06-12T02:38:23.113 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1389000000000005e-05 2020-06-12T02:38:23.114 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.114 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.114 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:23.114 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101277+0000", 2020-06-12T02:38:23.114 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7724000000000004e-05 2020-06-12T02:38:23.115 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.115 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.115 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:23.115 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101344+0000", 2020-06-12T02:38:23.115 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047852499999999999 2020-06-12T02:38:23.116 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.116 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.117 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:23.117 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101823+0000", 2020-06-12T02:38:23.117 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0039255510000000002 2020-06-12T02:38:23.117 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.117 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.118 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:23.118 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105748+0000", 2020-06-12T02:38:23.118 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58321340099999996 2020-06-12T02:38:23.118 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.119 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.119 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:23.119 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.688962+0000", 2020-06-12T02:38:23.119 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1898999999999996e-05 2020-06-12T02:38:23.119 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.120 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.120 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:23.120 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689044+0000", 2020-06-12T02:38:23.120 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58853946999999995 2020-06-12T02:38:23.120 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:23.121 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:23.121 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:23.121 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.121 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.121 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2924 3.ds1 3:b983fd28:::smithi10729407-1:head [append 0~188416] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:23.121 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.101633+0000", 2020-06-12T02:38:23.122 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 45.848654513, 2020-06-12T02:38:23.122 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58758259199999996, 2020-06-12T02:38:23.122 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:23.122 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:23.122 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:23.123 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:23.123 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:23.124 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2924 2020-06-12T02:38:23.124 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.124 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:23.124 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.124 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:23.125 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101633+0000", 2020-06-12T02:38:23.125 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:23.125 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.125 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.125 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:23.126 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101633+0000", 2020-06-12T02:38:23.126 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00018775399999999999 2020-06-12T02:38:23.126 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.126 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.126 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:23.127 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101821+0000", 2020-06-12T02:38:23.127 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7669999999999998e-06 2020-06-12T02:38:23.127 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.127 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.127 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:23.128 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101828+0000", 2020-06-12T02:38:23.128 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2716e-05 2020-06-12T02:38:23.128 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.128 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.128 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:23.129 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101840+0000", 2020-06-12T02:38:23.129 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.215e-06 2020-06-12T02:38:23.129 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.130 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.130 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:23.130 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101846+0000", 2020-06-12T02:38:23.130 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021214250000000001 2020-06-12T02:38:23.130 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.131 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.131 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:23.131 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.103967+0000", 2020-06-12T02:38:23.131 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1018000000000003e-05 2020-06-12T02:38:23.131 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.131 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.132 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:23.132 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.104038+0000", 2020-06-12T02:38:23.132 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000333364 2020-06-12T02:38:23.132 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.133 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.133 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:23.133 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.104371+0000", 2020-06-12T02:38:23.133 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001378546 2020-06-12T02:38:23.133 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.134 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.134 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:23.134 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105750+0000", 2020-06-12T02:38:23.134 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58338272899999999 2020-06-12T02:38:23.134 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.135 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.135 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:23.135 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689133+0000", 2020-06-12T02:38:23.135 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3058000000000003e-05 2020-06-12T02:38:23.135 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.135 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.137 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:23.137 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689216+0000", 2020-06-12T02:38:23.138 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58758259199999996 2020-06-12T02:38:23.138 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:23.138 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:23.138 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:23.139 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.139 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.139 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2925 3.ds1 3:b983fd28:::smithi10729407-1:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:23.139 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.101899+0000", 2020-06-12T02:38:23.140 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 45.848388847999999, 2020-06-12T02:38:23.140 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58747810899999997, 2020-06-12T02:38:23.140 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:23.140 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:23.141 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:23.141 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:23.141 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:23.141 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2925 2020-06-12T02:38:23.141 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.141 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:23.142 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.142 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:23.142 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101899+0000", 2020-06-12T02:38:23.142 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:23.142 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.143 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.143 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:23.143 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101899+0000", 2020-06-12T02:38:23.143 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2906999999999998e-05 2020-06-12T02:38:23.144 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.144 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.144 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:23.144 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101972+0000", 2020-06-12T02:38:23.144 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1789999999999996e-06 2020-06-12T02:38:23.145 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.145 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.145 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:23.145 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101978+0000", 2020-06-12T02:38:23.145 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1358e-05 2020-06-12T02:38:23.145 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.146 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.147 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:23.147 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101989+0000", 2020-06-12T02:38:23.147 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.031e-06 2020-06-12T02:38:23.147 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.147 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.148 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:23.148 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101993+0000", 2020-06-12T02:38:23.148 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033218990000000001 2020-06-12T02:38:23.148 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.148 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.149 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:23.149 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105315+0000", 2020-06-12T02:38:23.149 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6324999999999999e-05 2020-06-12T02:38:23.149 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.149 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.150 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:23.150 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105382+0000", 2020-06-12T02:38:23.150 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000216624 2020-06-12T02:38:23.150 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.150 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.151 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:23.151 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105598+0000", 2020-06-12T02:38:23.151 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001236586 2020-06-12T02:38:23.151 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.151 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.152 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:23.153 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.106835+0000", 2020-06-12T02:38:23.153 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58245509200000001 2020-06-12T02:38:23.153 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.153 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.153 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:23.154 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689290+0000", 2020-06-12T02:38:23.154 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7107999999999993e-05 2020-06-12T02:38:23.154 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.154 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.154 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:23.155 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689377+0000", 2020-06-12T02:38:23.155 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58747810899999997 2020-06-12T02:38:23.155 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:23.155 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:23.155 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:23.156 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.156 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.156 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2926 3.ds1 3:b983fd28:::smithi10729407-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:38:23.156 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.102036+0000", 2020-06-12T02:38:23.156 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 45.848251863999998, 2020-06-12T02:38:23.157 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58868048699999997, 2020-06-12T02:38:23.157 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:23.157 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:23.157 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:23.157 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:23.158 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:23.158 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2926 2020-06-12T02:38:23.158 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.158 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:23.158 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.159 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:23.159 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102036+0000", 2020-06-12T02:38:23.159 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:23.159 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.159 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.160 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:23.160 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102036+0000", 2020-06-12T02:38:23.160 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3786999999999999e-05 2020-06-12T02:38:23.160 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.161 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.161 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:23.161 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102080+0000", 2020-06-12T02:38:23.162 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7660000000000004e-06 2020-06-12T02:38:23.162 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.162 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.162 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:23.163 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102085+0000", 2020-06-12T02:38:23.163 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3145e-05 2020-06-12T02:38:23.163 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.163 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.163 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:23.164 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102099+0000", 2020-06-12T02:38:23.164 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2440000000000004e-06 2020-06-12T02:38:23.164 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.164 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.164 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:23.165 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102103+0000", 2020-06-12T02:38:23.165 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036826990000000002 2020-06-12T02:38:23.165 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.165 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.166 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:23.166 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105786+0000", 2020-06-12T02:38:23.166 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.6774000000000006e-05 2020-06-12T02:38:23.166 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.166 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.167 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:23.167 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105882+0000", 2020-06-12T02:38:23.167 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58352084900000001 2020-06-12T02:38:23.167 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.167 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.167 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:23.168 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689403+0000", 2020-06-12T02:38:23.168 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3788999999999995e-05 2020-06-12T02:38:23.169 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.169 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.169 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:23.169 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689467+0000", 2020-06-12T02:38:23.170 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001249434 2020-06-12T02:38:23.170 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.170 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.170 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:23.170 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690716+0000", 2020-06-12T02:38:23.171 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58868048699999997 2020-06-12T02:38:23.171 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:23.171 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:23.171 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:23.171 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.172 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.172 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2950 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~598016] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:23.172 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.162562+0000", 2020-06-12T02:38:23.172 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 45.78772549, 2020-06-12T02:38:23.173 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52780611399999999, 2020-06-12T02:38:23.174 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:23.174 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:23.174 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:23.175 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:23.175 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:23.175 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2950 2020-06-12T02:38:23.175 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.176 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:23.176 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.176 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:23.176 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.162562+0000", 2020-06-12T02:38:23.176 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:23.177 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.177 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.177 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:23.177 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.162562+0000", 2020-06-12T02:38:23.177 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00070718099999999998 2020-06-12T02:38:23.178 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.178 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.178 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:23.178 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163269+0000", 2020-06-12T02:38:23.178 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2029999999999997e-06 2020-06-12T02:38:23.179 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.179 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.179 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:23.179 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163276+0000", 2020-06-12T02:38:23.179 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2734999999999999e-05 2020-06-12T02:38:23.179 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.180 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.180 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:23.180 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163288+0000", 2020-06-12T02:38:23.180 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.404e-06 2020-06-12T02:38:23.180 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.181 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.181 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:23.181 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163295+0000", 2020-06-12T02:38:23.182 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5784000000000006e-05 2020-06-12T02:38:23.182 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.182 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.182 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:23.183 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163361+0000", 2020-06-12T02:38:23.183 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5318e-05 2020-06-12T02:38:23.183 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.183 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.184 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:23.184 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163436+0000", 2020-06-12T02:38:23.184 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054801100000000005 2020-06-12T02:38:23.184 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.184 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.184 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:23.185 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163984+0000", 2020-06-12T02:38:23.185 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.007023077 2020-06-12T02:38:23.185 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.185 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.185 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:23.186 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171007+0000", 2020-06-12T02:38:23.186 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51932051199999996 2020-06-12T02:38:23.186 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.186 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.186 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:23.187 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690327+0000", 2020-06-12T02:38:23.187 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0889000000000001e-05 2020-06-12T02:38:23.187 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.187 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.187 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:23.188 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690368+0000", 2020-06-12T02:38:23.188 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52780611399999999 2020-06-12T02:38:23.188 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:23.188 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:23.188 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:23.189 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.189 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.189 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2951 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~532480] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:23.189 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.165470+0000", 2020-06-12T02:38:23.189 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 45.784817881999999, 2020-06-12T02:38:23.190 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52507917900000001, 2020-06-12T02:38:23.190 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:23.190 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:23.190 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:23.190 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:23.191 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:23.191 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2951 2020-06-12T02:38:23.191 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.191 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:23.191 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.192 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:23.192 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.165470+0000", 2020-06-12T02:38:23.192 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:23.192 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.192 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.193 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:23.193 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.165470+0000", 2020-06-12T02:38:23.193 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00062231600000000004 2020-06-12T02:38:23.193 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.194 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.194 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:23.194 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166092+0000", 2020-06-12T02:38:23.194 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1610000000000002e-06 2020-06-12T02:38:23.194 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.195 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.195 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:23.195 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166097+0000", 2020-06-12T02:38:23.195 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7780000000000002e-06 2020-06-12T02:38:23.195 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.196 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.197 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:23.197 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166107+0000", 2020-06-12T02:38:23.197 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7169999999999999e-06 2020-06-12T02:38:23.197 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.198 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.198 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:23.198 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166112+0000", 2020-06-12T02:38:23.198 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4792000000000002e-05 2020-06-12T02:38:23.198 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.199 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.199 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:23.199 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166147+0000", 2020-06-12T02:38:23.199 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9336e-05 2020-06-12T02:38:23.199 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.199 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.200 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:23.200 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166236+0000", 2020-06-12T02:38:23.200 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00067747200000000004 2020-06-12T02:38:23.200 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.200 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.201 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:23.201 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166913+0000", 2020-06-12T02:38:23.201 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0047718669999999999 2020-06-12T02:38:23.201 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.201 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.202 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:23.202 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171685+0000", 2020-06-12T02:38:23.202 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51881990700000002 2020-06-12T02:38:23.202 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.202 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.202 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:23.203 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690505+0000", 2020-06-12T02:38:23.203 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3832999999999999e-05 2020-06-12T02:38:23.203 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.203 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.203 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:23.204 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690549+0000", 2020-06-12T02:38:23.204 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52507917900000001 2020-06-12T02:38:23.204 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:23.204 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:23.204 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:23.204 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.205 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.205 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2952 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~270336] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:23.205 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.166891+0000", 2020-06-12T02:38:23.205 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 45.783396232999998, 2020-06-12T02:38:23.205 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52426417000000003, 2020-06-12T02:38:23.206 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:23.206 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:23.206 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:23.206 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:23.206 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:23.207 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2952 2020-06-12T02:38:23.207 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.207 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:23.207 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.207 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:23.207 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166891+0000", 2020-06-12T02:38:23.208 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:23.208 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.208 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.208 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:23.208 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166891+0000", 2020-06-12T02:38:23.209 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00031878599999999999 2020-06-12T02:38:23.209 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.209 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.209 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:23.209 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167210+0000", 2020-06-12T02:38:23.210 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2360000000000004e-06 2020-06-12T02:38:23.210 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.210 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.211 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:23.211 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167219+0000", 2020-06-12T02:38:23.212 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3298e-05 2020-06-12T02:38:23.212 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.212 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.212 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:23.212 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167232+0000", 2020-06-12T02:38:23.213 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5849999999999999e-06 2020-06-12T02:38:23.213 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.213 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.213 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:23.213 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167237+0000", 2020-06-12T02:38:23.214 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0018755740000000001 2020-06-12T02:38:23.214 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.214 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.214 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:23.214 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.169113+0000", 2020-06-12T02:38:23.215 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000104768 2020-06-12T02:38:23.215 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.215 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.215 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:23.215 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.169218+0000", 2020-06-12T02:38:23.215 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00057389799999999999 2020-06-12T02:38:23.216 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.216 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.216 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:23.216 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.169792+0000", 2020-06-12T02:38:23.217 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0019333989999999999 2020-06-12T02:38:23.217 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.217 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.217 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:23.217 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171725+0000", 2020-06-12T02:38:23.218 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51938508699999997 2020-06-12T02:38:23.218 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.218 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.218 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:23.219 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691110+0000", 2020-06-12T02:38:23.219 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5538999999999999e-05 2020-06-12T02:38:23.219 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.219 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.220 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:23.220 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691156+0000", 2020-06-12T02:38:23.220 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52426417000000003 2020-06-12T02:38:23.220 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:23.220 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:23.220 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:23.221 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.221 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.221 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2953 3.5s0 3:a446366d:::smithi10729407-11:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:23.222 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.167293+0000", 2020-06-12T02:38:23.222 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 45.782994786000003, 2020-06-12T02:38:23.222 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52428350999999995, 2020-06-12T02:38:23.222 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:23.222 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:23.223 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:23.223 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:23.223 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:23.223 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2953 2020-06-12T02:38:23.223 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.223 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:23.224 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.224 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:23.224 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167293+0000", 2020-06-12T02:38:23.224 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:23.224 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.225 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.225 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:23.225 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167293+0000", 2020-06-12T02:38:23.225 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3880000000000004e-05 2020-06-12T02:38:23.225 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.225 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.226 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:23.226 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167367+0000", 2020-06-12T02:38:23.226 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.319e-06 2020-06-12T02:38:23.226 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.227 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.227 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:23.227 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167373+0000", 2020-06-12T02:38:23.227 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1719000000000001e-05 2020-06-12T02:38:23.227 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.228 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.228 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:23.228 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167385+0000", 2020-06-12T02:38:23.228 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4580000000000003e-06 2020-06-12T02:38:23.228 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.229 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.229 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:23.229 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167389+0000", 2020-06-12T02:38:23.229 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036341110000000002 2020-06-12T02:38:23.229 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.229 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.230 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:23.230 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171023+0000", 2020-06-12T02:38:23.230 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9418000000000005e-05 2020-06-12T02:38:23.230 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.230 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.231 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:23.231 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171093+0000", 2020-06-12T02:38:23.231 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00029087900000000002 2020-06-12T02:38:23.231 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.232 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.232 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:23.232 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171384+0000", 2020-06-12T02:38:23.233 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000489347 2020-06-12T02:38:23.233 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.233 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.234 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:23.234 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171873+0000", 2020-06-12T02:38:23.234 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51963691300000003 2020-06-12T02:38:23.234 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.235 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.235 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:23.235 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691510+0000", 2020-06-12T02:38:23.235 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6465999999999997e-05 2020-06-12T02:38:23.235 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.236 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.237 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:23.237 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691576+0000", 2020-06-12T02:38:23.237 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52428350999999995 2020-06-12T02:38:23.238 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:23.238 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:23.238 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:23.238 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.239 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.239 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2954 3.5s0 3:a446366d:::smithi10729407-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:38:23.239 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.167432+0000", 2020-06-12T02:38:23.239 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 45.782856037000002, 2020-06-12T02:38:23.239 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52568652699999996, 2020-06-12T02:38:23.240 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:23.240 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:23.240 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:23.240 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:23.240 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:23.241 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2954 2020-06-12T02:38:23.241 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.242 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:23.242 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.242 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:23.243 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167432+0000", 2020-06-12T02:38:23.243 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:23.243 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.243 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.243 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:23.244 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167432+0000", 2020-06-12T02:38:23.244 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3556000000000001e-05 2020-06-12T02:38:23.244 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.244 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.244 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:23.245 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167475+0000", 2020-06-12T02:38:23.245 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7350000000000001e-06 2020-06-12T02:38:23.245 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.245 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.245 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:23.246 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167481+0000", 2020-06-12T02:38:23.246 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3128e-05 2020-06-12T02:38:23.246 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.246 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.246 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:23.246 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167494+0000", 2020-06-12T02:38:23.247 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1579999999999998e-06 2020-06-12T02:38:23.247 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.247 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.247 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:23.247 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167498+0000", 2020-06-12T02:38:23.248 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004082174 2020-06-12T02:38:23.248 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.248 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.248 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:23.248 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171580+0000", 2020-06-12T02:38:23.249 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8954000000000002e-05 2020-06-12T02:38:23.249 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.249 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.249 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:23.249 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171679+0000", 2020-06-12T02:38:23.249 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51993235100000001 2020-06-12T02:38:23.250 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.250 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.250 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:23.250 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691612+0000", 2020-06-12T02:38:23.251 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4329000000000001e-05 2020-06-12T02:38:23.251 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.251 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.252 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:23.252 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691666+0000", 2020-06-12T02:38:23.252 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001452142 2020-06-12T02:38:23.252 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.252 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.252 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:23.253 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.693118+0000", 2020-06-12T02:38:23.253 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52568652699999996 2020-06-12T02:38:23.253 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:23.253 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:23.253 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:23.254 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.254 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.254 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2959 3.ds1 3:b3445631:::smithi10729407-6:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e27)", 2020-06-12T02:38:23.254 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.176248+0000", 2020-06-12T02:38:23.254 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 45.774039295999998, 2020-06-12T02:38:23.254 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51549877099999997, 2020-06-12T02:38:23.255 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:23.255 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:23.255 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:23.255 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:23.255 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:23.256 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2959 2020-06-12T02:38:23.256 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.256 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:23.256 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.256 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:23.256 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176248+0000", 2020-06-12T02:38:23.257 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:23.257 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.257 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.257 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:23.257 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176248+0000", 2020-06-12T02:38:23.258 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5120999999999999e-05 2020-06-12T02:38:23.258 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.258 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.258 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:23.258 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176274+0000", 2020-06-12T02:38:23.259 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9189999999999998e-06 2020-06-12T02:38:23.259 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.259 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.259 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:23.259 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176278+0000", 2020-06-12T02:38:23.259 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6339e-05 2020-06-12T02:38:23.260 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.260 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.260 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:23.260 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176305+0000", 2020-06-12T02:38:23.260 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9089999999999999e-06 2020-06-12T02:38:23.260 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.261 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.261 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:23.261 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176310+0000", 2020-06-12T02:38:23.262 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6095999999999999e-05 2020-06-12T02:38:23.262 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.262 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.263 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:23.263 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176376+0000", 2020-06-12T02:38:23.263 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8590999999999994e-05 2020-06-12T02:38:23.263 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.263 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.264 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:23.264 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176444+0000", 2020-06-12T02:38:23.265 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00024346400000000001 2020-06-12T02:38:23.265 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.266 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.266 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:23.266 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176688+0000", 2020-06-12T02:38:23.266 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00044220899999999999 2020-06-12T02:38:23.267 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.267 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.268 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:23.268 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.177130+0000", 2020-06-12T02:38:23.268 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51452984400000001 2020-06-12T02:38:23.268 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.268 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.269 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:23.269 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691660+0000", 2020-06-12T02:38:23.269 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7279000000000002e-05 2020-06-12T02:38:23.269 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.269 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.270 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:23.270 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691747+0000", 2020-06-12T02:38:23.270 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51549877099999997 2020-06-12T02:38:23.270 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:23.270 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:23.270 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:23.271 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.271 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.271 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:3032 3.ds1 3:b983fd28:::smithi10729407-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:38:23.271 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.864289+0000", 2020-06-12T02:38:23.271 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 45.085998377999999, 2020-06-12T02:38:23.272 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019572216999999999, 2020-06-12T02:38:23.272 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:23.272 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:23.272 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:23.272 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:23.273 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:23.273 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 3032 2020-06-12T02:38:23.273 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.273 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:23.273 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.273 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:23.274 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864289+0000", 2020-06-12T02:38:23.274 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:23.274 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.274 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.274 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:23.275 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864289+0000", 2020-06-12T02:38:23.275 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5701999999999997e-05 2020-06-12T02:38:23.275 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.275 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.275 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:23.275 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864335+0000", 2020-06-12T02:38:23.276 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0910000000000004e-06 2020-06-12T02:38:23.276 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.276 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.276 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:23.276 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864342+0000", 2020-06-12T02:38:23.277 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2301e-05 2020-06-12T02:38:23.277 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.277 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.277 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:23.277 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864354+0000", 2020-06-12T02:38:23.277 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5199999999999997e-06 2020-06-12T02:38:23.278 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.278 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.278 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:23.278 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864360+0000", 2020-06-12T02:38:23.278 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00049180999999999997 2020-06-12T02:38:23.279 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.279 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.279 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:23.279 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864852+0000", 2020-06-12T02:38:23.279 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6725000000000003e-05 2020-06-12T02:38:23.279 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.280 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.280 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:23.280 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864908+0000", 2020-06-12T02:38:23.280 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0089687010000000008 2020-06-12T02:38:23.280 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.281 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.281 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:23.281 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.873877+0000", 2020-06-12T02:38:23.281 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3382999999999999e-05 2020-06-12T02:38:23.281 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.282 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.282 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:23.282 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.873941+0000", 2020-06-12T02:38:23.282 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.006408927 2020-06-12T02:38:23.282 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.283 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.283 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:23.283 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.880350+0000", 2020-06-12T02:38:23.283 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3905e-05 2020-06-12T02:38:23.283 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.283 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.284 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:23.284 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.880403+0000", 2020-06-12T02:38:23.284 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0034581519999999999 2020-06-12T02:38:23.284 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:23.284 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:23.284 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:23.285 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.883862+0000", 2020-06-12T02:38:23.285 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019572216999999999 2020-06-12T02:38:23.285 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:23.285 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:23.285 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:23.286 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:23.286 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:23.286 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:23.299 INFO:tasks.rados.rados.0.smithi107.stdout:5500: writing smithi10729407-1 from 2441216 to 2940928 tid 1 2020-06-12T02:38:23.299 INFO:tasks.rados.rados.0.smithi107.stdout:5500: writing smithi10729407-1 from 2940928 to 3342336 tid 2 2020-06-12T02:38:23.299 INFO:tasks.rados.rados.0.smithi107.stdout:5501: delete oid 5 current snap is 0 2020-06-12T02:38:23.300 INFO:tasks.rados.rados.0.smithi107.stdout:5498: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:23.300 INFO:tasks.rados.rados.0.smithi107.stdout:5499: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:23.300 INFO:tasks.rados.rados.0.smithi107.stdout:5498: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:23.300 INFO:tasks.rados.rados.0.smithi107.stdout:5498: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:23.300 INFO:tasks.rados.rados.0.smithi107.stdout:5499: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:23.301 INFO:tasks.rados.rados.0.smithi107.stdout:5498: finishing write tid 5 to smithi10729407-16 2020-06-12T02:38:23.301 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 991 (ObjNum 5497 snap 0 seq_num 5497) dirty exists 2020-06-12T02:38:23.301 INFO:tasks.rados.rados.0.smithi107.stdout:5498: left oid 16 (ObjNum 5497 snap 0 seq_num 5497) 2020-06-12T02:38:23.301 INFO:tasks.rados.rados.0.smithi107.stdout:5499: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:23.301 INFO:tasks.rados.rados.0.smithi107.stdout:5499: finishing write tid 5 to smithi10729407-13 2020-06-12T02:38:23.302 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1058 (ObjNum 5498 snap 0 seq_num 5498) dirty exists 2020-06-12T02:38:23.302 INFO:tasks.rados.rados.0.smithi107.stdout:5499: left oid 13 (ObjNum 5498 snap 0 seq_num 5498) 2020-06-12T02:38:23.302 INFO:tasks.rados.rados.0.smithi107.stdout:5498: done (4 left) 2020-06-12T02:38:23.302 INFO:tasks.rados.rados.0.smithi107.stdout:5499: done (3 left) 2020-06-12T02:38:23.302 INFO:tasks.rados.rados.0.smithi107.stdout:5501: done (2 left) 2020-06-12T02:38:23.303 INFO:tasks.rados.rados.0.smithi107.stdout:5502: delete oid 8 current snap is 0 2020-06-12T02:38:23.303 INFO:tasks.rados.rados.0.smithi107.stdout:5500: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:23.303 INFO:tasks.rados.rados.0.smithi107.stdout:5502: done (2 left) 2020-06-12T02:38:23.303 INFO:tasks.rados.rados.0.smithi107.stdout:5503: delete oid 14 current snap is 0 2020-06-12T02:38:23.303 INFO:tasks.rados.rados.0.smithi107.stdout:5503: done (2 left) 2020-06-12T02:38:23.303 INFO:tasks.rados.rados.0.smithi107.stdout:5504: delete oid 8 current snap is 0 2020-06-12T02:38:23.304 INFO:tasks.rados.rados.0.smithi107.stdout:5500: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:23.304 INFO:tasks.rados.rados.0.smithi107.stdout:5500: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:23.304 INFO:tasks.rados.rados.0.smithi107.stdout:5500: finishing write tid 5 to smithi10729407-1 2020-06-12T02:38:23.304 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1012 (ObjNum 5499 snap 0 seq_num 5499) dirty exists 2020-06-12T02:38:23.304 INFO:tasks.rados.rados.0.smithi107.stdout:5500: left oid 1 (ObjNum 5499 snap 0 seq_num 5499) 2020-06-12T02:38:23.304 INFO:tasks.rados.rados.0.smithi107.stdout:5500: done (2 left) 2020-06-12T02:38:23.305 INFO:tasks.rados.rados.0.smithi107.stdout:5504: done (1 left) 2020-06-12T02:38:23.305 INFO:tasks.rados.rados.0.smithi107.stdout:5505: delete oid 8 current snap is 0 2020-06-12T02:38:23.305 INFO:tasks.rados.rados.0.smithi107.stdout:5505: done (1 left) 2020-06-12T02:38:23.305 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:23.305 INFO:tasks.rados.rados.0.smithi107.stdout:5506: seq_num 5505 ranges {0=606208,606208=401408} 2020-06-12T02:38:23.306 INFO:tasks.rados.rados.0.smithi107.stdout:5506: writing smithi10729407-9 from 0 to 606208 tid 1 2020-06-12T02:38:23.306 INFO:tasks.rados.rados.0.smithi107.stdout:5506: writing smithi10729407-9 from 606208 to 1007616 tid 2 2020-06-12T02:38:23.306 INFO:tasks.rados.rados.0.smithi107.stdout:5507: delete oid 3 current snap is 0 2020-06-12T02:38:23.306 INFO:tasks.rados.rados.0.smithi107.stdout:5507: done (2 left) 2020-06-12T02:38:23.306 INFO:tasks.rados.rados.0.smithi107.stdout:5508: delete oid 7 current snap is 0 2020-06-12T02:38:23.307 INFO:tasks.rados.rados.0.smithi107.stdout:5506: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:23.307 INFO:tasks.rados.rados.0.smithi107.stdout:5506: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:23.307 INFO:tasks.rados.rados.0.smithi107.stdout:5506: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:23.307 INFO:tasks.rados.rados.0.smithi107.stdout:5508: done (2 left) 2020-06-12T02:38:23.307 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:23.307 INFO:tasks.rados.rados.0.smithi107.stdout:5509: seq_num 5508 ranges {0=655360,655360=401408} 2020-06-12T02:38:23.308 INFO:tasks.rados.rados.0.smithi107.stdout:5509: writing smithi10729407-19 from 0 to 655360 tid 1 2020-06-12T02:38:23.308 INFO:tasks.rados.rados.0.smithi107.stdout:5509: writing smithi10729407-19 from 655360 to 1056768 tid 2 2020-06-12T02:38:23.308 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:38:23.308 INFO:tasks.rados.rados.0.smithi107.stdout:5510: seq_num 5509 ranges {0=671744,671744=401408} 2020-06-12T02:38:23.308 INFO:tasks.rados.rados.0.smithi107.stdout:5510: writing smithi10729407-3 from 0 to 671744 tid 1 2020-06-12T02:38:23.308 INFO:tasks.rados.rados.0.smithi107.stdout:5510: writing smithi10729407-3 from 671744 to 1073152 tid 2 2020-06-12T02:38:23.309 INFO:tasks.rados.rados.0.smithi107.stdout:5511: delete oid 16 current snap is 0 2020-06-12T02:38:23.309 INFO:tasks.rados.rados.0.smithi107.stdout:5506: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:23.309 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 997 (ObjNum 5505 snap 0 seq_num 5505) dirty exists 2020-06-12T02:38:23.309 INFO:tasks.rados.rados.0.smithi107.stdout:5506: left oid 9 (ObjNum 5505 snap 0 seq_num 5505) 2020-06-12T02:38:23.309 INFO:tasks.rados.rados.0.smithi107.stdout:5509: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:23.310 INFO:tasks.rados.rados.0.smithi107.stdout:5509: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:23.310 INFO:tasks.rados.rados.0.smithi107.stdout:5509: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:23.310 INFO:tasks.rados.rados.0.smithi107.stdout:5509: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:23.310 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 955 (ObjNum 5508 snap 0 seq_num 5508) dirty exists 2020-06-12T02:38:23.310 INFO:tasks.rados.rados.0.smithi107.stdout:5509: left oid 19 (ObjNum 5508 snap 0 seq_num 5508) 2020-06-12T02:38:23.310 INFO:tasks.rados.rados.0.smithi107.stdout:5510: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:23.311 INFO:tasks.rados.rados.0.smithi107.stdout:5510: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:23.311 INFO:tasks.rados.rados.0.smithi107.stdout:5510: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:23.311 INFO:tasks.rados.rados.0.smithi107.stdout:5510: finishing write tid 5 to smithi10729407-3 2020-06-12T02:38:23.311 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 995 (ObjNum 5509 snap 0 seq_num 5509) dirty exists 2020-06-12T02:38:23.311 INFO:tasks.rados.rados.0.smithi107.stdout:5510: left oid 3 (ObjNum 5509 snap 0 seq_num 5509) 2020-06-12T02:38:23.312 INFO:tasks.rados.rados.0.smithi107.stdout:5506: done (4 left) 2020-06-12T02:38:23.312 INFO:tasks.rados.rados.0.smithi107.stdout:5509: done (3 left) 2020-06-12T02:38:23.312 INFO:tasks.rados.rados.0.smithi107.stdout:5510: done (2 left) 2020-06-12T02:38:23.312 INFO:tasks.rados.rados.0.smithi107.stdout:5511: done (1 left) 2020-06-12T02:38:23.312 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:23.312 INFO:tasks.rados.rados.0.smithi107.stdout:5512: seq_num 5511 ranges {0=704512,704512=401408} 2020-06-12T02:38:23.313 INFO:tasks.rados.rados.0.smithi107.stdout:5512: writing smithi10729407-6 from 0 to 704512 tid 1 2020-06-12T02:38:23.313 INFO:tasks.rados.rados.0.smithi107.stdout:5512: writing smithi10729407-6 from 704512 to 1105920 tid 2 2020-06-12T02:38:23.313 INFO:tasks.rados.rados.0.smithi107.stdout:5513: delete oid 1 current snap is 0 2020-06-12T02:38:23.313 INFO:tasks.rados.rados.0.smithi107.stdout:5512: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:23.313 INFO:tasks.rados.rados.0.smithi107.stdout:5513: done (2 left) 2020-06-12T02:38:23.314 INFO:tasks.rados.rados.0.smithi107.stdout:5514: delete oid 9 current snap is 0 2020-06-12T02:38:23.314 INFO:tasks.rados.rados.0.smithi107.stdout:5512: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:23.314 INFO:tasks.rados.rados.0.smithi107.stdout:5512: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:23.314 INFO:tasks.rados.rados.0.smithi107.stdout:5514: done (2 left) 2020-06-12T02:38:23.314 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:23.314 INFO:tasks.rados.rados.0.smithi107.stdout:5515: seq_num 5514 ranges {7716864=753664,8470528=401408} 2020-06-12T02:38:23.315 INFO:tasks.rados.rados.0.smithi107.stdout:5515: writing smithi10729407-15 from 7716864 to 8470528 tid 1 2020-06-12T02:38:23.315 INFO:tasks.rados.rados.0.smithi107.stdout:5515: writing smithi10729407-15 from 8470528 to 8871936 tid 2 2020-06-12T02:38:23.315 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:23.315 INFO:tasks.rados.rados.0.smithi107.stdout:5516: seq_num 5515 ranges {933888=770048,1703936=401408} 2020-06-12T02:38:23.315 INFO:tasks.rados.rados.0.smithi107.stdout:5516: writing smithi10729407-11 from 933888 to 1703936 tid 1 2020-06-12T02:38:23.316 INFO:tasks.rados.rados.0.smithi107.stdout:5516: writing smithi10729407-11 from 1703936 to 2105344 tid 2 2020-06-12T02:38:23.316 INFO:tasks.rados.rados.0.smithi107.stdout:5517: delete oid 19 current snap is 0 2020-06-12T02:38:23.316 INFO:tasks.rados.rados.0.smithi107.stdout:5512: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:23.316 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1016 (ObjNum 5511 snap 0 seq_num 5511) dirty exists 2020-06-12T02:38:23.316 INFO:tasks.rados.rados.0.smithi107.stdout:5512: left oid 6 (ObjNum 5511 snap 0 seq_num 5511) 2020-06-12T02:38:23.316 INFO:tasks.rados.rados.0.smithi107.stdout:5515: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:23.317 INFO:tasks.rados.rados.0.smithi107.stdout:5515: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:23.317 INFO:tasks.rados.rados.0.smithi107.stdout:5515: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:23.317 INFO:tasks.rados.rados.0.smithi107.stdout:5515: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:23.317 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 980 (ObjNum 5514 snap 0 seq_num 5514) dirty exists 2020-06-12T02:38:23.317 INFO:tasks.rados.rados.0.smithi107.stdout:5515: left oid 15 (ObjNum 5514 snap 0 seq_num 5514) 2020-06-12T02:38:23.318 INFO:tasks.rados.rados.0.smithi107.stdout:5512: done (4 left) 2020-06-12T02:38:23.318 INFO:tasks.rados.rados.0.smithi107.stdout:5515: done (3 left) 2020-06-12T02:38:23.318 INFO:tasks.rados.rados.0.smithi107.stdout:5517: done (2 left) 2020-06-12T02:38:23.318 INFO:tasks.rados.rados.0.smithi107.stdout:5518: delete oid 13 current snap is 0 2020-06-12T02:38:23.318 INFO:tasks.rados.rados.0.smithi107.stdout:5516: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:23.318 INFO:tasks.rados.rados.0.smithi107.stdout:5518: done (2 left) 2020-06-12T02:38:23.319 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:23.319 INFO:tasks.rados.rados.0.smithi107.stdout:5519: seq_num 5518 ranges {0=417792,417792=565248,983040=237568} 2020-06-12T02:38:23.319 INFO:tasks.rados.rados.0.smithi107.stdout:5519: writing smithi10729407-9 from 0 to 417792 tid 1 2020-06-12T02:38:23.319 INFO:tasks.rados.rados.0.smithi107.stdout:5519: writing smithi10729407-9 from 417792 to 983040 tid 2 2020-06-12T02:38:23.319 INFO:tasks.rados.rados.0.smithi107.stdout:5519: writing smithi10729407-9 from 983040 to 1220608 tid 3 2020-06-12T02:38:23.319 INFO:tasks.rados.rados.0.smithi107.stdout:5520: delete oid 7 current snap is 0 2020-06-12T02:38:23.320 INFO:tasks.rados.rados.0.smithi107.stdout:5516: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:23.320 INFO:tasks.rados.rados.0.smithi107.stdout:5516: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:23.320 INFO:tasks.rados.rados.0.smithi107.stdout:5516: finishing write tid 5 to smithi10729407-11 2020-06-12T02:38:23.320 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1005 (ObjNum 5515 snap 0 seq_num 5515) dirty exists 2020-06-12T02:38:23.320 INFO:tasks.rados.rados.0.smithi107.stdout:5516: left oid 11 (ObjNum 5515 snap 0 seq_num 5515) 2020-06-12T02:38:23.321 INFO:tasks.rados.rados.0.smithi107.stdout:5519: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:23.321 INFO:tasks.rados.rados.0.smithi107.stdout:5516: done (3 left) 2020-06-12T02:38:23.321 INFO:tasks.rados.rados.0.smithi107.stdout:5520: done (2 left) 2020-06-12T02:38:23.321 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:23.321 INFO:tasks.rados.rados.0.smithi107.stdout:5521: seq_num 5520 ranges {0=458752,458752=786432,1245184=16384} 2020-06-12T02:38:23.321 INFO:tasks.rados.rados.0.smithi107.stdout:5521: writing smithi10729407-16 from 0 to 458752 tid 1 2020-06-12T02:38:23.322 INFO:tasks.rados.rados.0.smithi107.stdout:5521: writing smithi10729407-16 from 458752 to 1245184 tid 2 2020-06-12T02:38:23.322 INFO:tasks.rados.rados.0.smithi107.stdout:5521: writing smithi10729407-16 from 1245184 to 1261568 tid 3 2020-06-12T02:38:23.322 INFO:tasks.rados.rados.0.smithi107.stdout:5522: delete oid 12 current snap is 0 2020-06-12T02:38:23.322 INFO:tasks.rados.rados.0.smithi107.stdout:5519: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:23.322 INFO:tasks.rados.rados.0.smithi107.stdout:5519: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:23.323 INFO:tasks.rados.rados.0.smithi107.stdout:5519: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:23.323 INFO:tasks.rados.rados.0.smithi107.stdout:5519: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:23.323 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1002 (ObjNum 5518 snap 0 seq_num 5518) dirty exists 2020-06-12T02:38:23.323 INFO:tasks.rados.rados.0.smithi107.stdout:5519: left oid 9 (ObjNum 5518 snap 0 seq_num 5518) 2020-06-12T02:38:23.323 INFO:tasks.rados.rados.0.smithi107.stdout:5521: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:23.323 INFO:tasks.rados.rados.0.smithi107.stdout:5519: done (3 left) 2020-06-12T02:38:23.324 INFO:tasks.rados.rados.0.smithi107.stdout:5522: done (2 left) 2020-06-12T02:38:23.324 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:23.324 INFO:tasks.rados.rados.0.smithi107.stdout:5523: seq_num 5522 ranges {0=491520,491520=557056,1048576=245760} 2020-06-12T02:38:23.324 INFO:tasks.rados.rados.0.smithi107.stdout:5523: writing smithi10729407-12 from 0 to 491520 tid 1 2020-06-12T02:38:23.324 INFO:tasks.rados.rados.0.smithi107.stdout:5523: writing smithi10729407-12 from 491520 to 1048576 tid 2 2020-06-12T02:38:23.325 INFO:tasks.rados.rados.0.smithi107.stdout:5523: writing smithi10729407-12 from 1048576 to 1294336 tid 3 2020-06-12T02:38:23.325 INFO:tasks.rados.rados.0.smithi107.stdout:5524: delete oid 8 current snap is 0 2020-06-12T02:38:23.325 INFO:tasks.rados.rados.0.smithi107.stdout:5521: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:23.326 INFO:tasks.rados.rados.0.smithi107.stdout:5521: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:23.326 INFO:tasks.rados.rados.0.smithi107.stdout:5521: finishing write tid 5 to smithi10729407-16 2020-06-12T02:38:23.326 INFO:tasks.rados.rados.0.smithi107.stdout:5524: done (3 left) 2020-06-12T02:38:23.326 INFO:tasks.rados.rados.0.smithi107.stdout:5525: delete oid 9 current snap is 0 2020-06-12T02:38:23.327 INFO:tasks.rados.rados.0.smithi107.stdout:5523: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:23.327 INFO:tasks.rados.rados.0.smithi107.stdout:5521: finishing write tid 6 to smithi10729407-16 2020-06-12T02:38:23.327 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1000 (ObjNum 5520 snap 0 seq_num 5520) dirty exists 2020-06-12T02:38:23.327 INFO:tasks.rados.rados.0.smithi107.stdout:5521: left oid 16 (ObjNum 5520 snap 0 seq_num 5520) 2020-06-12T02:38:23.327 INFO:tasks.rados.rados.0.smithi107.stdout:5523: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:23.328 INFO:tasks.rados.rados.0.smithi107.stdout:5523: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:23.328 INFO:tasks.rados.rados.0.smithi107.stdout:5521: done (3 left) 2020-06-12T02:38:23.328 INFO:tasks.rados.rados.0.smithi107.stdout:5525: done (2 left) 2020-06-12T02:38:23.328 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:23.328 INFO:tasks.rados.rados.0.smithi107.stdout:5526: seq_num 5525 ranges {1032192=540672,1572864=606208,2179072=196608} 2020-06-12T02:38:23.329 INFO:tasks.rados.rados.0.smithi107.stdout:5526: writing smithi10729407-17 from 1032192 to 1572864 tid 1 2020-06-12T02:38:23.329 INFO:tasks.rados.rados.0.smithi107.stdout:5526: writing smithi10729407-17 from 1572864 to 2179072 tid 2 2020-06-12T02:38:23.329 INFO:tasks.rados.rados.0.smithi107.stdout:5526: writing smithi10729407-17 from 2179072 to 2375680 tid 3 2020-06-12T02:38:23.329 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:23.329 INFO:tasks.rados.rados.0.smithi107.stdout:5527: seq_num 5526 ranges {8871936=557056,9428992=491520,9920512=311296} 2020-06-12T02:38:23.329 INFO:tasks.rados.rados.0.smithi107.stdout:5527: writing smithi10729407-15 from 8871936 to 9428992 tid 1 2020-06-12T02:38:23.330 INFO:tasks.rados.rados.0.smithi107.stdout:5527: writing smithi10729407-15 from 9428992 to 9920512 tid 2 2020-06-12T02:38:23.330 INFO:tasks.rados.rados.0.smithi107.stdout:5527: writing smithi10729407-15 from 9920512 to 10231808 tid 3 2020-06-12T02:38:23.330 INFO:tasks.rados.rados.0.smithi107.stdout:5528: delete oid 2 current snap is 0 2020-06-12T02:38:23.330 INFO:tasks.rados.rados.0.smithi107.stdout:5523: finishing write tid 5 to smithi10729407-12 2020-06-12T02:38:23.330 INFO:tasks.rados.rados.0.smithi107.stdout:5523: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:23.331 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 961 (ObjNum 5522 snap 0 seq_num 5522) dirty exists 2020-06-12T02:38:23.331 INFO:tasks.rados.rados.0.smithi107.stdout:5523: left oid 12 (ObjNum 5522 snap 0 seq_num 5522) 2020-06-12T02:38:23.331 INFO:tasks.rados.rados.0.smithi107.stdout:5526: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:23.331 INFO:tasks.rados.rados.0.smithi107.stdout:5526: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:23.331 INFO:tasks.rados.rados.0.smithi107.stdout:5526: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:23.331 INFO:tasks.rados.rados.0.smithi107.stdout:5526: finishing write tid 5 to smithi10729407-17 2020-06-12T02:38:23.332 INFO:tasks.rados.rados.0.smithi107.stdout:5526: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:23.332 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1021 (ObjNum 5525 snap 0 seq_num 5525) dirty exists 2020-06-12T02:38:23.332 INFO:tasks.rados.rados.0.smithi107.stdout:5526: left oid 17 (ObjNum 5525 snap 0 seq_num 5525) 2020-06-12T02:38:23.332 INFO:tasks.rados.rados.0.smithi107.stdout:5523: done (4 left) 2020-06-12T02:38:23.332 INFO:tasks.rados.rados.0.smithi107.stdout:5526: done (3 left) 2020-06-12T02:38:23.333 INFO:tasks.rados.rados.0.smithi107.stdout:5528: done (2 left) 2020-06-12T02:38:23.333 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:23.333 INFO:tasks.rados.rados.0.smithi107.stdout:5529: seq_num 5528 ranges {1294336=589824,1884160=712704,2596864=90112} 2020-06-12T02:38:23.333 INFO:tasks.rados.rados.0.smithi107.stdout:5529: writing smithi10729407-12 from 1294336 to 1884160 tid 1 2020-06-12T02:38:23.333 INFO:tasks.rados.rados.0.smithi107.stdout:5529: writing smithi10729407-12 from 1884160 to 2596864 tid 2 2020-06-12T02:38:23.333 INFO:tasks.rados.rados.0.smithi107.stdout:5529: writing smithi10729407-12 from 2596864 to 2686976 tid 3 2020-06-12T02:38:23.334 INFO:tasks.rados.rados.0.smithi107.stdout:5530: delete oid 17 current snap is 0 2020-06-12T02:38:23.334 INFO:tasks.rados.rados.0.smithi107.stdout:5527: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:23.334 INFO:tasks.rados.rados.0.smithi107.stdout:5527: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:23.334 INFO:tasks.rados.rados.0.smithi107.stdout:5527: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:23.334 INFO:tasks.rados.rados.0.smithi107.stdout:5527: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:23.335 INFO:tasks.rados.rados.0.smithi107.stdout:5527: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:23.335 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 985 (ObjNum 5526 snap 0 seq_num 5526) dirty exists 2020-06-12T02:38:23.335 INFO:tasks.rados.rados.0.smithi107.stdout:5527: left oid 15 (ObjNum 5526 snap 0 seq_num 5526) 2020-06-12T02:38:23.335 INFO:tasks.rados.rados.0.smithi107.stdout:5527: done (3 left) 2020-06-12T02:38:23.335 INFO:tasks.rados.rados.0.smithi107.stdout:5530: done (2 left) 2020-06-12T02:38:23.336 INFO:tasks.rados.rados.0.smithi107.stdout:5531: delete oid 2 current snap is 0 2020-06-12T02:38:23.336 INFO:tasks.rados.rados.0.smithi107.stdout:5529: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:23.336 INFO:tasks.rados.rados.0.smithi107.stdout:5529: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:23.336 INFO:tasks.rados.rados.0.smithi107.stdout:5531: done (2 left) 2020-06-12T02:38:23.336 INFO:tasks.rados.rados.0.smithi107.stdout:5532: delete oid 4 current snap is 0 2020-06-12T02:38:23.336 INFO:tasks.rados.rados.0.smithi107.stdout:5529: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:23.337 INFO:tasks.rados.rados.0.smithi107.stdout:5529: finishing write tid 5 to smithi10729407-12 2020-06-12T02:38:23.337 INFO:tasks.rados.rados.0.smithi107.stdout:5532: done (2 left) 2020-06-12T02:38:23.337 INFO:tasks.rados.rados.0.smithi107.stdout:5533: delete oid 6 current snap is 0 2020-06-12T02:38:23.337 INFO:tasks.rados.rados.0.smithi107.stdout:5529: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:23.338 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 965 (ObjNum 5528 snap 0 seq_num 5528) dirty exists 2020-06-12T02:38:23.338 INFO:tasks.rados.rados.0.smithi107.stdout:5529: left oid 12 (ObjNum 5528 snap 0 seq_num 5528) 2020-06-12T02:38:23.338 INFO:tasks.rados.rados.0.smithi107.stdout:5529: done (2 left) 2020-06-12T02:38:23.338 INFO:tasks.rados.rados.0.smithi107.stdout:5533: done (1 left) 2020-06-12T02:38:23.338 INFO:tasks.rados.rados.0.smithi107.stdout:5534: delete oid 18 current snap is 0 2020-06-12T02:38:23.339 INFO:tasks.rados.rados.0.smithi107.stdout:5534: done (1 left) 2020-06-12T02:38:23.339 INFO:tasks.rados.rados.0.smithi107.stdout:5535: delete oid 1 current snap is 0 2020-06-12T02:38:23.339 INFO:tasks.rados.rados.0.smithi107.stdout:5535: done (1 left) 2020-06-12T02:38:23.339 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:23.339 INFO:tasks.rados.rados.0.smithi107.stdout:5536: seq_num 5535 ranges {0=704512,704512=753664,1458176=49152} 2020-06-12T02:38:23.339 INFO:tasks.rados.rados.0.smithi107.stdout:5536: writing smithi10729407-4 from 0 to 704512 tid 1 2020-06-12T02:38:23.340 INFO:tasks.rados.rados.0.smithi107.stdout:5536: writing smithi10729407-4 from 704512 to 1458176 tid 2 2020-06-12T02:38:23.340 INFO:tasks.rados.rados.0.smithi107.stdout:5536: writing smithi10729407-4 from 1458176 to 1507328 tid 3 2020-06-12T02:38:23.340 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:38:23.340 INFO:tasks.rados.rados.0.smithi107.stdout:5537: seq_num 5536 ranges {0=720896,720896=630784,1351680=172032} 2020-06-12T02:38:23.340 INFO:tasks.rados.rados.0.smithi107.stdout:5537: writing smithi10729407-6 from 0 to 720896 tid 1 2020-06-12T02:38:23.341 INFO:tasks.rados.rados.0.smithi107.stdout:5537: writing smithi10729407-6 from 720896 to 1351680 tid 2 2020-06-12T02:38:23.341 INFO:tasks.rados.rados.0.smithi107.stdout:5537: writing smithi10729407-6 from 1351680 to 1523712 tid 3 2020-06-12T02:38:23.341 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:23.341 INFO:tasks.rados.rados.0.smithi107.stdout:5538: seq_num 5537 ranges {10231808=737280,10969088=516096,11485184=286720} 2020-06-12T02:38:23.342 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-06-12T02:38:23.342 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:23.286+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:23.344 INFO:tasks.rados.rados.0.smithi107.stdout:5538: writing smithi10729407-15 from 10231808 to 10969088 tid 1 2020-06-12T02:38:23.344 INFO:tasks.rados.rados.0.smithi107.stdout:5538: writing smithi10729407-15 from 10969088 to 11485184 tid 2 2020-06-12T02:38:23.344 INFO:tasks.rados.rados.0.smithi107.stdout:5538: writing smithi10729407-15 from 11485184 to 11771904 tid 3 2020-06-12T02:38:23.344 INFO:tasks.rados.rados.0.smithi107.stdout:5539: delete oid 18 current snap is 0 2020-06-12T02:38:23.345 INFO:tasks.rados.rados.0.smithi107.stdout:5536: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:23.345 INFO:tasks.rados.rados.0.smithi107.stdout:5536: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:23.345 INFO:tasks.rados.rados.0.smithi107.stdout:5536: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:23.346 INFO:tasks.rados.rados.0.smithi107.stdout:5536: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:23.346 INFO:tasks.rados.rados.0.smithi107.stdout:5537: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:23.346 INFO:tasks.rados.rados.0.smithi107.stdout:5536: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:23.346 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1027 (ObjNum 5535 snap 0 seq_num 5535) dirty exists 2020-06-12T02:38:23.346 INFO:tasks.rados.rados.0.smithi107.stdout:5536: left oid 4 (ObjNum 5535 snap 0 seq_num 5535) 2020-06-12T02:38:23.346 INFO:tasks.rados.rados.0.smithi107.stdout:5537: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:23.347 INFO:tasks.rados.rados.0.smithi107.stdout:5537: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:23.347 INFO:tasks.rados.rados.0.smithi107.stdout:5537: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:23.347 INFO:tasks.rados.rados.0.smithi107.stdout:5537: finishing write tid 6 to smithi10729407-6 2020-06-12T02:38:23.347 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1022 (ObjNum 5536 snap 0 seq_num 5536) dirty exists 2020-06-12T02:38:23.347 INFO:tasks.rados.rados.0.smithi107.stdout:5537: left oid 6 (ObjNum 5536 snap 0 seq_num 5536) 2020-06-12T02:38:23.348 INFO:tasks.rados.rados.0.smithi107.stdout:5536: done (4 left) 2020-06-12T02:38:23.348 INFO:tasks.rados.rados.0.smithi107.stdout:5537: done (3 left) 2020-06-12T02:38:23.348 INFO:tasks.rados.rados.0.smithi107.stdout:5539: done (2 left) 2020-06-12T02:38:23.348 INFO:tasks.rados.rados.0.smithi107.stdout:5540: delete oid 8 current snap is 0 2020-06-12T02:38:23.348 INFO:tasks.rados.rados.0.smithi107.stdout:5538: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:23.349 INFO:tasks.rados.rados.0.smithi107.stdout:5538: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:23.349 INFO:tasks.rados.rados.0.smithi107.stdout:5538: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:23.350 INFO:tasks.rados.rados.0.smithi107.stdout:5538: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:23.357 INFO:tasks.rados.rados.0.smithi107.stdout:5540: done (2 left) 2020-06-12T02:38:23.358 INFO:tasks.rados.rados.0.smithi107.stdout:5541: delete oid 1 current snap is 0 2020-06-12T02:38:23.358 INFO:tasks.rados.rados.0.smithi107.stdout:5538: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:23.358 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 989 (ObjNum 5537 snap 0 seq_num 5537) dirty exists 2020-06-12T02:38:23.359 INFO:tasks.rados.rados.0.smithi107.stdout:5538: left oid 15 (ObjNum 5537 snap 0 seq_num 5537) 2020-06-12T02:38:23.359 INFO:tasks.rados.rados.0.smithi107.stdout:5538: done (2 left) 2020-06-12T02:38:23.359 INFO:tasks.rados.rados.0.smithi107.stdout:5541: done (1 left) 2020-06-12T02:38:23.359 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:23.360 INFO:tasks.rados.rados.0.smithi107.stdout:5542: seq_num 5541 ranges {11771904=409600,12181504=458752,12640256=622592,13262848=122880} 2020-06-12T02:38:23.403 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:23.399+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:23.416 INFO:tasks.rados.rados.0.smithi107.stdout:5542: writing smithi10729407-15 from 11771904 to 12181504 tid 1 2020-06-12T02:38:23.418 INFO:tasks.rados.rados.0.smithi107.stdout:5542: writing smithi10729407-15 from 12181504 to 12640256 tid 2 2020-06-12T02:38:23.422 INFO:tasks.rados.rados.0.smithi107.stdout:5542: writing smithi10729407-15 from 12640256 to 13262848 tid 3 2020-06-12T02:38:23.423 INFO:tasks.rados.rados.0.smithi107.stdout:5542: writing smithi10729407-15 from 13262848 to 13385728 tid 4 2020-06-12T02:38:23.423 INFO:tasks.rados.rados.0.smithi107.stdout:5543: delete oid 2 current snap is 0 2020-06-12T02:38:23.423 INFO:tasks.rados.rados.0.smithi107.stdout:5542: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:23.426 INFO:tasks.rados.rados.0.smithi107.stdout:5543: done (2 left) 2020-06-12T02:38:23.427 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:23.427 INFO:tasks.rados.rados.0.smithi107.stdout:5544: seq_num 5543 ranges {0=442368,442368=671744,1114112=532480} 2020-06-12T02:38:23.429 INFO:tasks.rados.rados.0.smithi107.stdout:5544: writing smithi10729407-19 from 0 to 442368 tid 1 2020-06-12T02:38:23.433 INFO:tasks.rados.rados.0.smithi107.stdout:5544: writing smithi10729407-19 from 442368 to 1114112 tid 2 2020-06-12T02:38:23.437 INFO:tasks.rados.rados.0.smithi107.stdout:5544: writing smithi10729407-19 from 1114112 to 1646592 tid 3 2020-06-12T02:38:23.437 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:23.437 INFO:tasks.rados.rados.0.smithi107.stdout:5545: seq_num 5544 ranges {0=458752,458752=557056,1015808=442368,1458176=204800} 2020-06-12T02:38:23.438 INFO:tasks.rados.rados.0.smithi107.stdout:5545: writing smithi10729407-14 from 0 to 458752 tid 1 2020-06-12T02:38:23.442 INFO:tasks.rados.rados.0.smithi107.stdout:5545: writing smithi10729407-14 from 458752 to 1015808 tid 2 2020-06-12T02:38:23.445 INFO:tasks.rados.rados.0.smithi107.stdout:5545: writing smithi10729407-14 from 1015808 to 1458176 tid 3 2020-06-12T02:38:23.446 INFO:tasks.rados.rados.0.smithi107.stdout:5545: writing smithi10729407-14 from 1458176 to 1662976 tid 4 2020-06-12T02:38:23.447 INFO:tasks.rados.rados.0.smithi107.stdout:5546: delete oid 2 current snap is 0 2020-06-12T02:38:23.447 INFO:tasks.rados.rados.0.smithi107.stdout:5544: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:23.447 INFO:tasks.rados.rados.0.smithi107.stdout:5542: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:23.447 INFO:tasks.rados.rados.0.smithi107.stdout:5542: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:23.448 INFO:tasks.rados.rados.0.smithi107.stdout:5542: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:23.448 INFO:tasks.rados.rados.0.smithi107.stdout:5542: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:23.449 INFO:tasks.rados.rados.0.smithi107.stdout:5542: finishing write tid 7 to smithi10729407-15 2020-06-12T02:38:23.449 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 995 (ObjNum 5541 snap 0 seq_num 5541) dirty exists 2020-06-12T02:38:23.449 INFO:tasks.rados.rados.0.smithi107.stdout:5542: left oid 15 (ObjNum 5541 snap 0 seq_num 5541) 2020-06-12T02:38:23.450 INFO:tasks.rados.rados.0.smithi107.stdout:5545: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:23.451 INFO:tasks.rados.rados.0.smithi107.stdout:5544: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:23.452 INFO:tasks.rados.rados.0.smithi107.stdout:5544: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:23.452 INFO:tasks.rados.rados.0.smithi107.stdout:5542: done (4 left) 2020-06-12T02:38:23.453 INFO:tasks.rados.rados.0.smithi107.stdout:5546: done (3 left) 2020-06-12T02:38:23.453 INFO:tasks.rados.rados.0.smithi107.stdout:5547: delete oid 15 current snap is 0 2020-06-12T02:38:23.454 INFO:tasks.rados.rados.0.smithi107.stdout:5547: done (3 left) 2020-06-12T02:38:23.454 INFO:tasks.rados.rados.0.smithi107.stdout:5548: delete oid 16 current snap is 0 2020-06-12T02:38:23.454 INFO:tasks.rados.rados.0.smithi107.stdout:5545: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:23.455 INFO:tasks.rados.rados.0.smithi107.stdout:5544: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:23.455 INFO:tasks.rados.rados.0.smithi107.stdout:5545: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:23.456 INFO:tasks.rados.rados.0.smithi107.stdout:5545: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:23.456 INFO:tasks.rados.rados.0.smithi107.stdout:5545: finishing write tid 6 to smithi10729407-14 2020-06-12T02:38:23.458 INFO:tasks.rados.rados.0.smithi107.stdout:5545: finishing write tid 7 to smithi10729407-14 2020-06-12T02:38:23.458 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1010 (ObjNum 5544 snap 0 seq_num 5544) dirty exists 2020-06-12T02:38:23.458 INFO:tasks.rados.rados.0.smithi107.stdout:5545: left oid 14 (ObjNum 5544 snap 0 seq_num 5544) 2020-06-12T02:38:23.459 INFO:tasks.rados.rados.0.smithi107.stdout:5544: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:23.459 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 969 (ObjNum 5543 snap 0 seq_num 5543) dirty exists 2020-06-12T02:38:23.459 INFO:tasks.rados.rados.0.smithi107.stdout:5544: left oid 19 (ObjNum 5543 snap 0 seq_num 5543) 2020-06-12T02:38:23.460 INFO:tasks.rados.rados.0.smithi107.stdout:5544: done (3 left) 2020-06-12T02:38:23.460 INFO:tasks.rados.rados.0.smithi107.stdout:5545: done (2 left) 2020-06-12T02:38:23.460 INFO:tasks.rados.rados.0.smithi107.stdout:5548: done (1 left) 2020-06-12T02:38:23.460 INFO:tasks.rados.rados.0.smithi107.stdout:5549: delete oid 17 current snap is 0 2020-06-12T02:38:23.462 INFO:tasks.rados.rados.0.smithi107.stdout:5549: done (1 left) 2020-06-12T02:38:23.463 INFO:tasks.rados.rados.0.smithi107.stdout:5550: delete oid 1 current snap is 0 2020-06-12T02:38:23.466 INFO:tasks.rados.rados.0.smithi107.stdout:5550: done (1 left) 2020-06-12T02:38:23.466 INFO:tasks.rados.rados.0.smithi107.stdout:5551: delete oid 6 current snap is 0 2020-06-12T02:38:23.469 INFO:tasks.rados.rados.0.smithi107.stdout:5551: done (1 left) 2020-06-12T02:38:23.470 INFO:tasks.rados.rados.0.smithi107.stdout:5552: delete oid 15 current snap is 0 2020-06-12T02:38:23.474 INFO:tasks.rados.rados.0.smithi107.stdout:5552: done (1 left) 2020-06-12T02:38:23.474 INFO:tasks.rados.rados.0.smithi107.stdout:5553: delete oid 1 current snap is 0 2020-06-12T02:38:23.476 INFO:tasks.rados.rados.0.smithi107.stdout:5553: done (1 left) 2020-06-12T02:38:23.476 INFO:tasks.rados.rados.0.smithi107.stdout:5554: delete oid 12 current snap is 0 2020-06-12T02:38:23.478 INFO:tasks.rados.rados.0.smithi107.stdout:5554: done (1 left) 2020-06-12T02:38:23.479 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:23.479 INFO:tasks.rados.rados.0.smithi107.stdout:5555: seq_num 5554 ranges {1646592=622592,2269184=655360,2924544=548864} 2020-06-12T02:38:23.490 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:23.490 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:38:23.491 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2020-06-12T02:38:23.491 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:23.491 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:38:23.488+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:38:23.492 INFO:tasks.rados.rados.0.smithi107.stdout:5555: writing smithi10729407-19 from 1646592 to 2269184 tid 1 2020-06-12T02:38:23.494 INFO:tasks.rados.rados.0.smithi107.stdout:5555: writing smithi10729407-19 from 2269184 to 2924544 tid 2 2020-06-12T02:38:23.498 INFO:tasks.rados.rados.0.smithi107.stdout:5555: writing smithi10729407-19 from 2924544 to 3473408 tid 3 2020-06-12T02:38:23.498 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:23.498 INFO:tasks.rados.rados.0.smithi107.stdout:5556: seq_num 5555 ranges {0=638976,638976=532480,1171456=573440,1744896=98304} 2020-06-12T02:38:23.502 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-06-12T02:38:23.503 INFO:tasks.rados.rados.0.smithi107.stdout:5556: writing smithi10729407-8 from 0 to 638976 tid 1 2020-06-12T02:38:23.505 INFO:tasks.rados.rados.0.smithi107.stdout:5556: writing smithi10729407-8 from 638976 to 1171456 tid 2 2020-06-12T02:38:23.509 INFO:tasks.rados.rados.0.smithi107.stdout:5556: writing smithi10729407-8 from 1171456 to 1744896 tid 3 2020-06-12T02:38:23.510 INFO:tasks.rados.rados.0.smithi107.stdout:5556: writing smithi10729407-8 from 1744896 to 1843200 tid 4 2020-06-12T02:38:23.510 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:23.510 INFO:tasks.rados.rados.0.smithi107.stdout:5557: seq_num 5556 ranges {0=655360,655360=417792,1073152=786432} 2020-06-12T02:38:23.513 INFO:tasks.rados.rados.0.smithi107.stdout:5557: writing smithi10729407-15 from 0 to 655360 tid 1 2020-06-12T02:38:23.515 INFO:tasks.rados.rados.0.smithi107.stdout:5557: writing smithi10729407-15 from 655360 to 1073152 tid 2 2020-06-12T02:38:23.520 INFO:tasks.rados.rados.0.smithi107.stdout:5557: writing smithi10729407-15 from 1073152 to 1859584 tid 3 2020-06-12T02:38:23.520 INFO:tasks.rados.rados.0.smithi107.stdout:5558: delete oid 16 current snap is 0 2020-06-12T02:38:23.521 INFO:tasks.rados.rados.0.smithi107.stdout:5555: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:23.521 INFO:tasks.rados.rados.0.smithi107.stdout:5555: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:23.521 INFO:tasks.rados.rados.0.smithi107.stdout:5555: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:23.522 INFO:tasks.rados.rados.0.smithi107.stdout:5555: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:23.522 INFO:tasks.rados.rados.0.smithi107.stdout:5555: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:23.522 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 974 (ObjNum 5554 snap 0 seq_num 5554) dirty exists 2020-06-12T02:38:23.522 INFO:tasks.rados.rados.0.smithi107.stdout:5555: left oid 19 (ObjNum 5554 snap 0 seq_num 5554) 2020-06-12T02:38:23.522 INFO:tasks.rados.rados.0.smithi107.stdout:5556: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:23.523 INFO:tasks.rados.rados.0.smithi107.stdout:5556: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:23.523 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2020-06-12T02:38:23.523 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2020-06-12T02:38:23.523 INFO:tasks.thrashosds.thrasher:fixing pg num pool device_health_metrics 2020-06-12T02:38:23.524 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json 2020-06-12T02:38:23.525 INFO:tasks.rados.rados.0.smithi107.stdout:5556: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:23.525 INFO:tasks.rados.rados.0.smithi107.stdout:5555: done (4 left) 2020-06-12T02:38:23.525 INFO:tasks.rados.rados.0.smithi107.stdout:5558: done (3 left) 2020-06-12T02:38:23.526 INFO:tasks.rados.rados.0.smithi107.stdout:5559: delete oid 17 current snap is 0 2020-06-12T02:38:23.527 INFO:tasks.rados.rados.0.smithi107.stdout:5559: done (3 left) 2020-06-12T02:38:23.528 INFO:tasks.rados.rados.0.smithi107.stdout:5560: delete oid 11 current snap is 0 2020-06-12T02:38:23.529 INFO:tasks.rados.rados.0.smithi107.stdout:5556: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:23.529 INFO:tasks.rados.rados.0.smithi107.stdout:5556: finishing write tid 6 to smithi10729407-8 2020-06-12T02:38:23.529 INFO:tasks.rados.rados.0.smithi107.stdout:5557: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:23.531 INFO:tasks.rados.rados.0.smithi107.stdout:5560: done (3 left) 2020-06-12T02:38:23.532 INFO:tasks.rados.rados.0.smithi107.stdout:5561: delete oid 9 current snap is 0 2020-06-12T02:38:23.534 INFO:tasks.rados.rados.0.smithi107.stdout:5561: done (3 left) 2020-06-12T02:38:23.534 INFO:tasks.rados.rados.0.smithi107.stdout:5562: delete oid 5 current snap is 0 2020-06-12T02:38:23.536 INFO:tasks.rados.rados.0.smithi107.stdout:5562: done (3 left) 2020-06-12T02:38:23.536 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:23.537 INFO:tasks.rados.rados.0.smithi107.stdout:5563: seq_num 5562 ranges {0=761856,761856=573440,1335296=630784} 2020-06-12T02:38:23.540 INFO:tasks.rados.rados.0.smithi107.stdout:5563: writing smithi10729407-17 from 0 to 761856 tid 1 2020-06-12T02:38:23.544 INFO:tasks.rados.rados.0.smithi107.stdout:5563: writing smithi10729407-17 from 761856 to 1335296 tid 2 2020-06-12T02:38:23.548 INFO:tasks.rados.rados.0.smithi107.stdout:5563: writing smithi10729407-17 from 1335296 to 1966080 tid 3 2020-06-12T02:38:23.548 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:38:23.548 INFO:tasks.rados.rados.0.smithi107.stdout:5564: seq_num 5563 ranges {0=778240,778240=458752,1236992=532480,1769472=212992} 2020-06-12T02:38:23.550 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd set noscrub 2020-06-12T02:38:23.550 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-06-12T02:38:23.554 INFO:tasks.rados.rados.0.smithi107.stdout:5564: writing smithi10729407-20 from 0 to 778240 tid 1 2020-06-12T02:38:23.556 INFO:tasks.rados.rados.0.smithi107.stdout:5564: writing smithi10729407-20 from 778240 to 1236992 tid 2 2020-06-12T02:38:23.560 INFO:tasks.rados.rados.0.smithi107.stdout:5564: writing smithi10729407-20 from 1236992 to 1769472 tid 3 2020-06-12T02:38:23.561 INFO:tasks.rados.rados.0.smithi107.stdout:5564: writing smithi10729407-20 from 1769472 to 1982464 tid 4 2020-06-12T02:38:23.562 INFO:tasks.rados.rados.0.smithi107.stdout:5565: delete oid 2 current snap is 0 2020-06-12T02:38:23.562 INFO:tasks.rados.rados.0.smithi107.stdout:5557: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:23.562 INFO:tasks.rados.rados.0.smithi107.stdout:5556: finishing write tid 7 to smithi10729407-8 2020-06-12T02:38:23.562 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1002 (ObjNum 5555 snap 0 seq_num 5555) dirty exists 2020-06-12T02:38:23.563 INFO:tasks.rados.rados.0.smithi107.stdout:5556: left oid 8 (ObjNum 5555 snap 0 seq_num 5555) 2020-06-12T02:38:23.563 INFO:tasks.rados.rados.0.smithi107.stdout:5557: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:23.563 INFO:tasks.rados.rados.0.smithi107.stdout:5557: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:23.563 INFO:tasks.rados.rados.0.smithi107.stdout:5557: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:23.563 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1006 (ObjNum 5556 snap 0 seq_num 5556) dirty exists 2020-06-12T02:38:23.564 INFO:tasks.rados.rados.0.smithi107.stdout:5557: left oid 15 (ObjNum 5556 snap 0 seq_num 5556) 2020-06-12T02:38:23.564 INFO:tasks.rados.rados.0.smithi107.stdout:5563: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:23.564 INFO:tasks.rados.rados.0.smithi107.stdout:5563: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:23.564 INFO:tasks.rados.rados.0.smithi107.stdout:5563: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:23.564 INFO:tasks.rados.rados.0.smithi107.stdout:5563: finishing write tid 5 to smithi10729407-17 2020-06-12T02:38:23.564 INFO:tasks.rados.rados.0.smithi107.stdout:5563: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:23.565 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1033 (ObjNum 5562 snap 0 seq_num 5562) dirty exists 2020-06-12T02:38:23.565 INFO:tasks.rados.rados.0.smithi107.stdout:5563: left oid 17 (ObjNum 5562 snap 0 seq_num 5562) 2020-06-12T02:38:23.565 INFO:tasks.rados.rados.0.smithi107.stdout:5564: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:23.566 INFO:tasks.rados.rados.0.smithi107.stdout:5556: done (5 left) 2020-06-12T02:38:23.566 INFO:tasks.rados.rados.0.smithi107.stdout:5557: done (4 left) 2020-06-12T02:38:23.566 INFO:tasks.rados.rados.0.smithi107.stdout:5563: done (3 left) 2020-06-12T02:38:23.567 INFO:tasks.rados.rados.0.smithi107.stdout:5565: done (2 left) 2020-06-12T02:38:23.567 INFO:tasks.rados.rados.0.smithi107.stdout:5566: delete oid 13 current snap is 0 2020-06-12T02:38:23.569 INFO:tasks.rados.rados.0.smithi107.stdout:5566: done (2 left) 2020-06-12T02:38:23.569 INFO:tasks.rados.rados.0.smithi107.stdout:5567: delete oid 12 current snap is 0 2020-06-12T02:38:23.571 INFO:tasks.rados.rados.0.smithi107.stdout:5564: finishing write tid 2 to smithi10729407-20 2020-06-12T02:38:23.571 INFO:tasks.rados.rados.0.smithi107.stdout:5564: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:23.572 INFO:tasks.rados.rados.0.smithi107.stdout:5567: done (2 left) 2020-06-12T02:38:23.572 INFO:tasks.rados.rados.0.smithi107.stdout:5568: delete oid 17 current snap is 0 2020-06-12T02:38:23.573 INFO:tasks.rados.rados.0.smithi107.stdout:5564: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:23.574 INFO:tasks.rados.rados.0.smithi107.stdout:5564: finishing write tid 6 to smithi10729407-20 2020-06-12T02:38:23.574 INFO:tasks.rados.rados.0.smithi107.stdout:5568: done (2 left) 2020-06-12T02:38:23.575 INFO:tasks.rados.rados.0.smithi107.stdout:5569: delete oid 13 current snap is 0 2020-06-12T02:38:23.576 INFO:tasks.rados.rados.0.smithi107.stdout:5564: finishing write tid 7 to smithi10729407-20 2020-06-12T02:38:23.576 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 429 (ObjNum 5563 snap 0 seq_num 5563) dirty exists 2020-06-12T02:38:23.576 INFO:tasks.rados.rados.0.smithi107.stdout:5564: left oid 20 (ObjNum 5563 snap 0 seq_num 5563) 2020-06-12T02:38:23.577 INFO:tasks.rados.rados.0.smithi107.stdout:5564: done (2 left) 2020-06-12T02:38:23.578 INFO:tasks.rados.rados.0.smithi107.stdout:5569: done (1 left) 2020-06-12T02:38:23.578 INFO:tasks.rados.rados.0.smithi107.stdout:5570: delete oid 4 current snap is 0 2020-06-12T02:38:23.583 INFO:tasks.rados.rados.0.smithi107.stdout:5570: done (1 left) 2020-06-12T02:38:23.583 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:23.583 INFO:tasks.rados.rados.0.smithi107.stdout:5571: seq_num 5570 ranges {1843200=491520,2334720=499712,2834432=688128,3522560=417792} 2020-06-12T02:38:23.591 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:23.588+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:23.596 INFO:tasks.rados.rados.0.smithi107.stdout:5571: writing smithi10729407-8 from 1843200 to 2334720 tid 1 2020-06-12T02:38:23.599 INFO:tasks.rados.rados.0.smithi107.stdout:5571: writing smithi10729407-8 from 2334720 to 2834432 tid 2 2020-06-12T02:38:23.602 INFO:tasks.rados.rados.0.smithi107.stdout:5571: writing smithi10729407-8 from 2834432 to 3522560 tid 3 2020-06-12T02:38:23.605 INFO:tasks.rados.rados.0.smithi107.stdout:5571: writing smithi10729407-8 from 3522560 to 3940352 tid 4 2020-06-12T02:38:23.605 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:23.605 INFO:tasks.rados.rados.0.smithi107.stdout:5572: seq_num 5571 ranges {3473408=507904,3981312=786432,4767744=499712,5267456=319488} 2020-06-12T02:38:23.627 INFO:tasks.rados.rados.0.smithi107.stdout:5572: writing smithi10729407-19 from 3473408 to 3981312 tid 1 2020-06-12T02:38:23.632 INFO:tasks.rados.rados.0.smithi107.stdout:5572: writing smithi10729407-19 from 3981312 to 4767744 tid 2 2020-06-12T02:38:23.635 INFO:tasks.rados.rados.0.smithi107.stdout:5572: writing smithi10729407-19 from 4767744 to 5267456 tid 3 2020-06-12T02:38:23.637 INFO:tasks.rados.rados.0.smithi107.stdout:5572: writing smithi10729407-19 from 5267456 to 5586944 tid 4 2020-06-12T02:38:23.637 INFO:tasks.rados.rados.0.smithi107.stdout:5573: delete oid 7 current snap is 0 2020-06-12T02:38:23.638 INFO:tasks.rados.rados.0.smithi107.stdout:5571: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:23.638 INFO:tasks.rados.rados.0.smithi107.stdout:5571: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:23.638 INFO:tasks.rados.rados.0.smithi107.stdout:5571: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:23.638 INFO:tasks.rados.rados.0.smithi107.stdout:5571: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:23.638 INFO:tasks.rados.rados.0.smithi107.stdout:5571: finishing write tid 6 to smithi10729407-8 2020-06-12T02:38:23.638 INFO:tasks.rados.rados.0.smithi107.stdout:5571: finishing write tid 7 to smithi10729407-8 2020-06-12T02:38:23.639 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1011 (ObjNum 5570 snap 0 seq_num 5570) dirty exists 2020-06-12T02:38:23.639 INFO:tasks.rados.rados.0.smithi107.stdout:5571: left oid 8 (ObjNum 5570 snap 0 seq_num 5570) 2020-06-12T02:38:23.639 INFO:tasks.rados.rados.0.smithi107.stdout:5572: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:23.640 INFO:tasks.rados.rados.0.smithi107.stdout:5571: done (3 left) 2020-06-12T02:38:23.640 INFO:tasks.rados.rados.0.smithi107.stdout:5573: done (2 left) 2020-06-12T02:38:23.641 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:23.641 INFO:tasks.rados.rados.0.smithi107.stdout:5574: seq_num 5573 ranges {0=540672,540672=606208,1146880=466944,1613824=532480} 2020-06-12T02:38:23.644 INFO:tasks.rados.rados.0.smithi107.stdout:5574: writing smithi10729407-5 from 0 to 540672 tid 1 2020-06-12T02:38:23.649 INFO:tasks.rados.rados.0.smithi107.stdout:5574: writing smithi10729407-5 from 540672 to 1146880 tid 2 2020-06-12T02:38:23.652 INFO:tasks.rados.rados.0.smithi107.stdout:5574: writing smithi10729407-5 from 1146880 to 1613824 tid 3 2020-06-12T02:38:23.656 INFO:tasks.rados.rados.0.smithi107.stdout:5574: writing smithi10729407-5 from 1613824 to 2146304 tid 4 2020-06-12T02:38:23.657 INFO:tasks.rados.rados.0.smithi107.stdout:5572: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:23.657 INFO:tasks.rados.rados.0.smithi107.stdout:5572: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:23.657 INFO:tasks.rados.rados.0.smithi107.stdout:5572: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:23.657 INFO:tasks.rados.rados.0.smithi107.stdout:5574: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:23.658 INFO:tasks.rados.rados.0.smithi107.stdout:5572: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:23.658 INFO:tasks.rados.rados.0.smithi107.stdout:5572: finishing write tid 7 to smithi10729407-19 2020-06-12T02:38:23.658 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 980 (ObjNum 5571 snap 0 seq_num 5571) dirty exists 2020-06-12T02:38:23.658 INFO:tasks.rados.rados.0.smithi107.stdout:5572: left oid 19 (ObjNum 5571 snap 0 seq_num 5571) 2020-06-12T02:38:23.659 INFO:tasks.rados.rados.0.smithi107.stdout:5572: done (2 left) 2020-06-12T02:38:23.659 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:38:23.659 INFO:tasks.rados.rados.0.smithi107.stdout:5575: seq_num 5574 ranges {1982464=557056,2539520=483328,3022848=671744,3694592=450560} 2020-06-12T02:38:23.671 INFO:tasks.rados.rados.0.smithi107.stdout:5575: writing smithi10729407-20 from 1982464 to 2539520 tid 1 2020-06-12T02:38:23.674 INFO:tasks.rados.rados.0.smithi107.stdout:5575: writing smithi10729407-20 from 2539520 to 3022848 tid 2 2020-06-12T02:38:23.679 INFO:tasks.rados.rados.0.smithi107.stdout:5575: writing smithi10729407-20 from 3022848 to 3694592 tid 3 2020-06-12T02:38:23.681 INFO:tasks.rados.rados.0.smithi107.stdout:5575: writing smithi10729407-20 from 3694592 to 4145152 tid 4 2020-06-12T02:38:23.681 INFO:tasks.rados.rados.0.smithi107.stdout:5574: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:23.682 INFO:tasks.rados.rados.0.smithi107.stdout:5574: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:23.682 INFO:tasks.rados.rados.0.smithi107.stdout:5574: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:23.682 INFO:tasks.rados.rados.0.smithi107.stdout:5574: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:23.682 INFO:tasks.rados.rados.0.smithi107.stdout:5574: finishing write tid 7 to smithi10729407-5 2020-06-12T02:38:23.682 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 555 (ObjNum 5573 snap 0 seq_num 5573) dirty exists 2020-06-12T02:38:23.683 INFO:tasks.rados.rados.0.smithi107.stdout:5574: left oid 5 (ObjNum 5573 snap 0 seq_num 5573) 2020-06-12T02:38:23.683 INFO:tasks.rados.rados.0.smithi107.stdout:5575: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:23.683 INFO:tasks.rados.rados.0.smithi107.stdout:5574: done (2 left) 2020-06-12T02:38:23.683 INFO:tasks.rados.rados.0.smithi107.stdout:5576: delete oid 5 current snap is 0 2020-06-12T02:38:23.686 INFO:tasks.rados.rados.0.smithi107.stdout:5576: done (2 left) 2020-06-12T02:38:23.686 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:38:23.686 INFO:tasks.rados.rados.0.smithi107.stdout:5577: seq_num 5576 ranges {0=589824,589824=655360,1245184=688128,1933312=262144} 2020-06-12T02:38:23.690 INFO:tasks.rados.rados.0.smithi107.stdout:5577: writing smithi10729407-2 from 0 to 589824 tid 1 2020-06-12T02:38:23.692 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:23.690+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:23.694 INFO:tasks.rados.rados.0.smithi107.stdout:5577: writing smithi10729407-2 from 589824 to 1245184 tid 2 2020-06-12T02:38:23.698 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:23.698 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:38:23.699 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:38:23.699 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:38:23.699 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:23.699 INFO:tasks.rados.rados.0.smithi107.stdout:5577: writing smithi10729407-2 from 1245184 to 1933312 tid 3 2020-06-12T02:38:23.701 INFO:tasks.rados.rados.0.smithi107.stdout:5577: writing smithi10729407-2 from 1933312 to 2195456 tid 4 2020-06-12T02:38:23.701 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:23.701 INFO:tasks.rados.rados.0.smithi107.stdout:5578: seq_num 5577 ranges {0=606208,606208=540672,1146880=491520,1638400=573440} 2020-06-12T02:38:23.710 INFO:tasks.rados.rados.0.smithi107.stdout:5578: writing smithi10729407-13 from 0 to 606208 tid 1 2020-06-12T02:38:23.712 INFO:tasks.rados.rados.0.smithi107.stdout:5578: writing smithi10729407-13 from 606208 to 1146880 tid 2 2020-06-12T02:38:23.712 INFO:tasks.rados.rados.0.smithi107.stdout:5578: writing smithi10729407-13 from 1146880 to 1638400 tid 3 2020-06-12T02:38:23.713 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-06-12T02:38:23.714 INFO:tasks.rados.rados.0.smithi107.stdout:5578: writing smithi10729407-13 from 1638400 to 2211840 tid 4 2020-06-12T02:38:23.715 INFO:tasks.rados.rados.0.smithi107.stdout:5579: delete oid 17 current snap is 0 2020-06-12T02:38:23.715 INFO:tasks.rados.rados.0.smithi107.stdout:5575: finishing write tid 2 to smithi10729407-20 2020-06-12T02:38:23.715 INFO:tasks.rados.rados.0.smithi107.stdout:5575: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:23.715 INFO:tasks.rados.rados.0.smithi107.stdout:5575: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:23.716 INFO:tasks.rados.rados.0.smithi107.stdout:5575: finishing write tid 6 to smithi10729407-20 2020-06-12T02:38:23.716 INFO:tasks.rados.rados.0.smithi107.stdout:5575: finishing write tid 7 to smithi10729407-20 2020-06-12T02:38:23.716 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 434 (ObjNum 5574 snap 0 seq_num 5574) dirty exists 2020-06-12T02:38:23.716 INFO:tasks.rados.rados.0.smithi107.stdout:5575: left oid 20 (ObjNum 5574 snap 0 seq_num 5574) 2020-06-12T02:38:23.716 INFO:tasks.rados.rados.0.smithi107.stdout:5577: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:23.717 INFO:tasks.rados.rados.0.smithi107.stdout:5577: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:23.717 INFO:tasks.rados.rados.0.smithi107.stdout:5577: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:23.717 INFO:tasks.rados.rados.0.smithi107.stdout:5577: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:23.717 INFO:tasks.rados.rados.0.smithi107.stdout:5577: finishing write tid 6 to smithi10729407-2 2020-06-12T02:38:23.717 INFO:tasks.rados.rados.0.smithi107.stdout:5578: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:23.718 INFO:tasks.rados.rados.0.smithi107.stdout:5577: finishing write tid 7 to smithi10729407-2 2020-06-12T02:38:23.718 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 530 (ObjNum 5576 snap 0 seq_num 5576) dirty exists 2020-06-12T02:38:23.719 INFO:tasks.rados.rados.0.smithi107.stdout:5577: left oid 2 (ObjNum 5576 snap 0 seq_num 5576) 2020-06-12T02:38:23.719 INFO:tasks.rados.rados.0.smithi107.stdout:5575: done (4 left) 2020-06-12T02:38:23.719 INFO:tasks.rados.rados.0.smithi107.stdout:5577: done (3 left) 2020-06-12T02:38:23.719 INFO:tasks.rados.rados.0.smithi107.stdout:5579: done (2 left) 2020-06-12T02:38:23.719 INFO:tasks.rados.rados.0.smithi107.stdout:5580: delete oid 18 current snap is 0 2020-06-12T02:38:23.720 INFO:tasks.rados.rados.0.smithi107.stdout:5580: done (2 left) 2020-06-12T02:38:23.720 INFO:tasks.rados.rados.0.smithi107.stdout:5581: delete oid 20 current snap is 0 2020-06-12T02:38:23.723 INFO:tasks.rados.rados.0.smithi107.stdout:5581: done (2 left) 2020-06-12T02:38:23.723 INFO:tasks.rados.rados.0.smithi107.stdout:5582: delete oid 6 current snap is 0 2020-06-12T02:38:23.727 INFO:tasks.rados.rados.0.smithi107.stdout:5578: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:23.727 INFO:tasks.rados.rados.0.smithi107.stdout:5578: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:23.727 INFO:tasks.rados.rados.0.smithi107.stdout:5578: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:23.728 INFO:tasks.rados.rados.0.smithi107.stdout:5578: finishing write tid 6 to smithi10729407-13 2020-06-12T02:38:23.728 INFO:tasks.rados.rados.0.smithi107.stdout:5582: done (2 left) 2020-06-12T02:38:23.729 INFO:tasks.rados.rados.0.smithi107.stdout:5583: delete oid 7 current snap is 0 2020-06-12T02:38:23.729 INFO:tasks.rados.rados.0.smithi107.stdout:5578: finishing write tid 7 to smithi10729407-13 2020-06-12T02:38:23.730 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1069 (ObjNum 5577 snap 0 seq_num 5577) dirty exists 2020-06-12T02:38:23.730 INFO:tasks.rados.rados.0.smithi107.stdout:5578: left oid 13 (ObjNum 5577 snap 0 seq_num 5577) 2020-06-12T02:38:23.732 INFO:tasks.rados.rados.0.smithi107.stdout:5578: done (2 left) 2020-06-12T02:38:23.732 INFO:tasks.rados.rados.0.smithi107.stdout:5583: done (1 left) 2020-06-12T02:38:23.732 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:23.732 INFO:tasks.rados.rados.0.smithi107.stdout:5584: seq_num 5583 ranges {3940352=712704,4653056=696320,5349376=442368,5791744=466944} 2020-06-12T02:38:23.754 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:38:23.761 INFO:tasks.rados.rados.0.smithi107.stdout:5584: writing smithi10729407-8 from 3940352 to 4653056 tid 1 2020-06-12T02:38:23.767 INFO:tasks.rados.rados.0.smithi107.stdout:5584: writing smithi10729407-8 from 4653056 to 5349376 tid 2 2020-06-12T02:38:23.770 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:38:23.770 INFO:tasks.rados.rados.0.smithi107.stdout:5584: writing smithi10729407-8 from 5349376 to 5791744 tid 3 2020-06-12T02:38:23.774 INFO:tasks.rados.rados.0.smithi107.stdout:5584: writing smithi10729407-8 from 5791744 to 6258688 tid 4 2020-06-12T02:38:23.774 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:23.774 INFO:tasks.rados.rados.0.smithi107.stdout:5585: seq_num 5584 ranges {0=729088,729088=581632,1310720=647168,1957888=376832} 2020-06-12T02:38:23.779 INFO:tasks.rados.rados.0.smithi107.stdout:5585: writing smithi10729407-17 from 0 to 729088 tid 1 2020-06-12T02:38:23.783 INFO:tasks.rados.rados.0.smithi107.stdout:5585: writing smithi10729407-17 from 729088 to 1310720 tid 2 2020-06-12T02:38:23.787 INFO:tasks.rados.rados.0.smithi107.stdout:5585: writing smithi10729407-17 from 1310720 to 1957888 tid 3 2020-06-12T02:38:23.797 INFO:tasks.rados.rados.0.smithi107.stdout:5585: writing smithi10729407-17 from 1957888 to 2334720 tid 4 2020-06-12T02:38:23.797 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:23.797 INFO:tasks.rados.rados.0.smithi107.stdout:5586: seq_num 5585 ranges {0=745472,745472=466944,1212416=450560,1662976=491520,2154496=196608} 2020-06-12T02:38:23.797 INFO:tasks.rados.rados.0.smithi107.stdout:5586: writing smithi10729407-9 from 0 to 745472 tid 1 2020-06-12T02:38:23.798 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:38:23.798 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:23.796+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:23.799 INFO:tasks.rados.rados.0.smithi107.stdout:5586: writing smithi10729407-9 from 745472 to 1212416 tid 2 2020-06-12T02:38:23.802 INFO:tasks.rados.rados.0.smithi107.stdout:5586: writing smithi10729407-9 from 1212416 to 1662976 tid 3 2020-06-12T02:38:23.803 INFO:tasks.rados.rados.0.smithi107.stdout:5586: writing smithi10729407-9 from 1662976 to 2154496 tid 4 2020-06-12T02:38:23.806 INFO:tasks.rados.rados.0.smithi107.stdout:5586: writing smithi10729407-9 from 2154496 to 2351104 tid 5 2020-06-12T02:38:23.806 INFO:tasks.rados.rados.0.smithi107.stdout:5587: delete oid 19 current snap is 0 2020-06-12T02:38:23.806 INFO:tasks.rados.rados.0.smithi107.stdout:5584: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:23.806 INFO:tasks.rados.rados.0.smithi107.stdout:5584: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:23.807 INFO:tasks.rados.rados.0.smithi107.stdout:5584: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:23.807 INFO:tasks.rados.rados.0.smithi107.stdout:5584: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:23.807 INFO:tasks.rados.rados.0.smithi107.stdout:5584: finishing write tid 6 to smithi10729407-8 2020-06-12T02:38:23.807 INFO:tasks.rados.rados.0.smithi107.stdout:5584: finishing write tid 7 to smithi10729407-8 2020-06-12T02:38:23.807 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1016 (ObjNum 5583 snap 0 seq_num 5583) dirty exists 2020-06-12T02:38:23.807 INFO:tasks.rados.rados.0.smithi107.stdout:5584: left oid 8 (ObjNum 5583 snap 0 seq_num 5583) 2020-06-12T02:38:23.808 INFO:tasks.rados.rados.0.smithi107.stdout:5585: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:23.808 INFO:tasks.rados.rados.0.smithi107.stdout:5585: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:23.809 INFO:tasks.rados.rados.0.smithi107.stdout:5585: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:23.809 INFO:tasks.rados.rados.0.smithi107.stdout:5586: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:23.810 INFO:tasks.rados.rados.0.smithi107.stdout:5584: done (4 left) 2020-06-12T02:38:23.810 INFO:tasks.rados.rados.0.smithi107.stdout:5587: done (3 left) 2020-06-12T02:38:23.811 INFO:tasks.rados.rados.0.smithi107.stdout:5588: delete oid 4 current snap is 0 2020-06-12T02:38:23.814 INFO:tasks.rados.rados.0.smithi107.stdout:5585: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:23.823 INFO:tasks.rados.rados.0.smithi107.stdout:5585: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:23.824 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:38:23.825 INFO:tasks.rados.rados.0.smithi107.stdout:5585: finishing write tid 7 to smithi10729407-17 2020-06-12T02:38:23.825 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1041 (ObjNum 5584 snap 0 seq_num 5584) dirty exists 2020-06-12T02:38:23.826 INFO:tasks.rados.rados.0.smithi107.stdout:5585: left oid 17 (ObjNum 5584 snap 0 seq_num 5584) 2020-06-12T02:38:23.826 INFO:tasks.rados.rados.0.smithi107.stdout:5585: done (3 left) 2020-06-12T02:38:23.826 INFO:tasks.rados.rados.0.smithi107.stdout:5588: done (2 left) 2020-06-12T02:38:23.827 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:23.827 INFO:tasks.rados.rados.0.smithi107.stdout:5589: seq_num 5588 ranges {0=794624,794624=565248,1359872=630784,1990656=409600} 2020-06-12T02:38:23.831 INFO:tasks.rados.rados.0.smithi107.stdout:5589: writing smithi10729407-6 from 0 to 794624 tid 1 2020-06-12T02:38:23.834 INFO:tasks.rados.rados.0.smithi107.stdout:5589: writing smithi10729407-6 from 794624 to 1359872 tid 2 2020-06-12T02:38:23.838 INFO:tasks.rados.rados.0.smithi107.stdout:5589: writing smithi10729407-6 from 1359872 to 1990656 tid 3 2020-06-12T02:38:23.840 INFO:tasks.rados.rados.0.smithi107.stdout:5589: writing smithi10729407-6 from 1990656 to 2400256 tid 4 2020-06-12T02:38:23.841 INFO:tasks.rados.rados.0.smithi107.stdout:5590: delete oid 20 current snap is 0 2020-06-12T02:38:23.841 INFO:tasks.rados.rados.0.smithi107.stdout:5586: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:23.841 INFO:tasks.rados.rados.0.smithi107.stdout:5586: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:23.841 INFO:tasks.rados.rados.0.smithi107.stdout:5586: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:23.842 INFO:tasks.rados.rados.0.smithi107.stdout:5586: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:23.842 INFO:tasks.rados.rados.0.smithi107.stdout:5586: finishing write tid 7 to smithi10729407-9 2020-06-12T02:38:23.842 INFO:tasks.rados.rados.0.smithi107.stdout:5586: finishing write tid 8 to smithi10729407-9 2020-06-12T02:38:23.842 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1013 (ObjNum 5585 snap 0 seq_num 5585) dirty exists 2020-06-12T02:38:23.842 INFO:tasks.rados.rados.0.smithi107.stdout:5586: left oid 9 (ObjNum 5585 snap 0 seq_num 5585) 2020-06-12T02:38:23.843 INFO:tasks.rados.rados.0.smithi107.stdout:5589: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:23.843 INFO:tasks.rados.rados.0.smithi107.stdout:5586: done (3 left) 2020-06-12T02:38:23.844 INFO:tasks.rados.rados.0.smithi107.stdout:5590: done (2 left) 2020-06-12T02:38:23.844 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:23.844 INFO:tasks.rados.rados.0.smithi107.stdout:5591: seq_num 5590 ranges {2351104=24576} 2020-06-12T02:38:23.853 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:38:23.857 INFO:tasks.rados.rados.0.smithi107.stdout:5591: writing smithi10729407-9 from 2351104 to 2375680 tid 1 2020-06-12T02:38:23.857 INFO:tasks.rados.rados.0.smithi107.stdout:5592: delete oid 13 current snap is 0 2020-06-12T02:38:23.857 INFO:tasks.rados.rados.0.smithi107.stdout:5589: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:23.857 INFO:tasks.rados.rados.0.smithi107.stdout:5589: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:23.858 INFO:tasks.rados.rados.0.smithi107.stdout:5589: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:23.858 INFO:tasks.rados.rados.0.smithi107.stdout:5589: finishing write tid 6 to smithi10729407-6 2020-06-12T02:38:23.858 INFO:tasks.rados.rados.0.smithi107.stdout:5589: finishing write tid 7 to smithi10729407-6 2020-06-12T02:38:23.858 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1032 (ObjNum 5588 snap 0 seq_num 5588) dirty exists 2020-06-12T02:38:23.858 INFO:tasks.rados.rados.0.smithi107.stdout:5589: left oid 6 (ObjNum 5588 snap 0 seq_num 5588) 2020-06-12T02:38:23.860 INFO:tasks.rados.rados.0.smithi107.stdout:5589: done (3 left) 2020-06-12T02:38:23.860 INFO:tasks.rados.rados.0.smithi107.stdout:5592: done (2 left) 2020-06-12T02:38:23.860 INFO:tasks.rados.rados.0.smithi107.stdout:5593: delete oid 18 current snap is 0 2020-06-12T02:38:23.872 INFO:tasks.rados.rados.0.smithi107.stdout:5591: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:23.872 INFO:tasks.rados.rados.0.smithi107.stdout:5591: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:23.872 INFO:tasks.rados.rados.0.smithi107.stdout:5593: done (2 left) 2020-06-12T02:38:23.872 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:38:23.873 INFO:tasks.rados.rados.0.smithi107.stdout:5594: seq_num 5593 ranges {0=73728} 2020-06-12T02:38:23.873 INFO:tasks.rados.rados.0.smithi107.stdout:5594: writing smithi10729407-20 from 0 to 73728 tid 1 2020-06-12T02:38:23.873 INFO:tasks.rados.rados.0.smithi107.stdout:5595: delete oid 14 current snap is 0 2020-06-12T02:38:23.873 INFO:tasks.rados.rados.0.smithi107.stdout:5591: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:23.873 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1015 (ObjNum 5590 snap 0 seq_num 5590) dirty exists 2020-06-12T02:38:23.874 INFO:tasks.rados.rados.0.smithi107.stdout:5591: left oid 9 (ObjNum 5590 snap 0 seq_num 5590) 2020-06-12T02:38:23.874 INFO:tasks.rados.rados.0.smithi107.stdout:5591: done (3 left) 2020-06-12T02:38:23.874 INFO:tasks.rados.rados.0.smithi107.stdout:5595: done (2 left) 2020-06-12T02:38:23.874 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:23.875 INFO:tasks.rados.rados.0.smithi107.stdout:5596: seq_num 5595 ranges {2375680=106496} 2020-06-12T02:38:23.881 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:38:23.884 INFO:tasks.rados.rados.0.smithi107.stdout:5596: writing smithi10729407-9 from 2375680 to 2482176 tid 1 2020-06-12T02:38:23.885 INFO:tasks.rados.rados.0.smithi107.stdout:5597: delete oid 19 current snap is 0 2020-06-12T02:38:23.885 INFO:tasks.rados.rados.0.smithi107.stdout:5594: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:23.885 INFO:tasks.rados.rados.0.smithi107.stdout:5594: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:23.886 INFO:tasks.rados.rados.0.smithi107.stdout:5594: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:23.886 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 438 (ObjNum 5593 snap 0 seq_num 5593) dirty exists 2020-06-12T02:38:23.886 INFO:tasks.rados.rados.0.smithi107.stdout:5594: left oid 20 (ObjNum 5593 snap 0 seq_num 5593) 2020-06-12T02:38:23.887 INFO:tasks.rados.rados.0.smithi107.stdout:5594: done (3 left) 2020-06-12T02:38:23.887 INFO:tasks.rados.rados.0.smithi107.stdout:5597: done (2 left) 2020-06-12T02:38:23.888 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:23.888 INFO:tasks.rados.rados.0.smithi107.stdout:5598: seq_num 5597 ranges {0=147456} 2020-06-12T02:38:23.889 INFO:tasks.rados.rados.0.smithi107.stdout:5598: writing smithi10729407-19 from 0 to 147456 tid 1 2020-06-12T02:38:23.889 INFO:tasks.rados.rados.0.smithi107.stdout:5599: delete oid 2 current snap is 0 2020-06-12T02:38:23.890 INFO:tasks.rados.rados.0.smithi107.stdout:5596: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:23.891 INFO:tasks.rados.rados.0.smithi107.stdout:5599: done (3 left) 2020-06-12T02:38:23.891 INFO:tasks.rados.rados.0.smithi107.stdout:5600: delete oid 8 current snap is 0 2020-06-12T02:38:23.893 INFO:tasks.rados.rados.0.smithi107.stdout:5596: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:23.895 INFO:tasks.rados.rados.0.smithi107.stdout:5598: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:23.897 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:38:23.898 INFO:tasks.rados.rados.0.smithi107.stdout:5600: done (3 left) 2020-06-12T02:38:23.898 INFO:tasks.rados.rados.0.smithi107.stdout:5601: delete oid 12 current snap is 0 2020-06-12T02:38:23.898 INFO:tasks.rados.rados.0.smithi107.stdout:5598: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:23.898 INFO:tasks.rados.rados.0.smithi107.stdout:5596: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:23.899 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1018 (ObjNum 5595 snap 0 seq_num 5595) dirty exists 2020-06-12T02:38:23.899 INFO:tasks.rados.rados.0.smithi107.stdout:5596: left oid 9 (ObjNum 5595 snap 0 seq_num 5595) 2020-06-12T02:38:23.900 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2020-06-12T02:38:23.901 INFO:tasks.rados.rados.0.smithi107.stdout:5598: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:23.902 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 984 (ObjNum 5597 snap 0 seq_num 5597) dirty exists 2020-06-12T02:38:23.902 INFO:tasks.rados.rados.0.smithi107.stdout:5598: left oid 19 (ObjNum 5597 snap 0 seq_num 5597) 2020-06-12T02:38:23.902 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:38:23.898+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:38:23.903 INFO:tasks.rados.rados.0.smithi107.stdout:5596: done (3 left) 2020-06-12T02:38:23.903 INFO:tasks.rados.rados.0.smithi107.stdout:5598: done (2 left) 2020-06-12T02:38:23.903 INFO:tasks.rados.rados.0.smithi107.stdout:5601: done (1 left) 2020-06-12T02:38:23.904 INFO:tasks.rados.rados.0.smithi107.stdout:5602: delete oid 2 current snap is 0 2020-06-12T02:38:23.905 INFO:tasks.rados.rados.0.smithi107.stdout:5602: done (1 left) 2020-06-12T02:38:23.905 INFO:tasks.rados.rados.0.smithi107.stdout:5603: delete oid 4 current snap is 0 2020-06-12T02:38:23.908 INFO:tasks.rados.rados.0.smithi107.stdout:5603: done (1 left) 2020-06-12T02:38:23.908 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:38:23.908 INFO:tasks.rados.rados.0.smithi107.stdout:5604: seq_num 5603 ranges {1073152=245760} 2020-06-12T02:38:23.909 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:38:23.909 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:23.910 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:23.912 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":93,"stamp":"2020-06-12T02:38:23.606288+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":17473536,"num_objects":8,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":3294,"num_read_kb":1783,"num_write":7323,"num_write_kb":2089658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":13869056,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9678,"ondisk_log_size":9678,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":66,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8414408,"kb_used_data":25736,"kb_used_omap":25,"kb_used_meta":8388582,"kb_avail":746560312,"statfs":{"total":773094113280,"available":764477759488,"internally_reserved":8589934592,"allocated":26353664,"data_stored":23637688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26157,"internal_metadata":8589908435},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,6],"upper_bound":16384},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":622592,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":744,"num_read_kb":407,"num_write":1648,"num_write_kb":469360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1572864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.244596"},"pg_stats":[{"pgid":"3.f","version":"56'1028","reported_seq":"1268","reported_epoch":"56","state":"active+clean","last_fresh":"2020-06-12T02:38:23.275022+0000","last_change":"2020-06-12T02:38:11.587385+0000","last_active":"2020-06-12T02:38:23.275022+0000","last_peered":"2020-06-12T02:38:23.275022+0000","last_clean":"2020-06-12T02:38:23.275022+0000","last_became_active":"2020-06-12T02:37:57.566633+0000","last_became_peered":"2020-06-12T02:37:57.566633+0000","last_unstale":"2020-06-12T02:38:23.275022+0000","last_undegraded":"2020-06-12T02:38:23.275022+0000","last_fullsized":"2020-06-12T02:38:23.275022+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":1028,"ondisk_log_size":1028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1507328,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":355,"num_read_kb":183,"num_write":741,"num_write_kb":206056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1531904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"56'549","reported_seq":"706","reported_epoch":"56","state":"active+clean","last_fresh":"2020-06-12T02:38:22.905384+0000","last_change":"2020-06-12T02:38:11.572385+0000","last_active":"2020-06-12T02:38:22.905384+0000","last_peered":"2020-06-12T02:38:22.905384+0000","last_clean":"2020-06-12T02:38:22.905384+0000","last_became_active":"2020-06-12T02:37:57.563869+0000","last_became_peered":"2020-06-12T02:37:57.563869+0000","last_unstale":"2020-06-12T02:38:22.905384+0000","last_undegraded":"2020-06-12T02:38:22.905384+0000","last_fullsized":"2020-06-12T02:38:22.905384+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":549,"ondisk_log_size":549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":183,"num_read_kb":98,"num_write":424,"num_write_kb":121736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1466368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"56'1026","reported_seq":"1282","reported_epoch":"56","state":"active+clean","last_fresh":"2020-06-12T02:38:23.467337+0000","last_change":"2020-06-12T02:38:11.587290+0000","last_active":"2020-06-12T02:38:23.467337+0000","last_peered":"2020-06-12T02:38:23.467337+0000","last_clean":"2020-06-12T02:38:23.467337+0000","last_became_active":"2020-06-12T02:37:57.564464+0000","last_became_peered":"2020-06-12T02:37:57.564464+0000","last_unstale":"2020-06-12T02:38:23.467337+0000","last_undegraded":"2020-06-12T02:38:23.467337+0000","last_fullsized":"2020-06-12T02:38:23.467337+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":1026,"ondisk_log_size":1026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":336,"num_read_kb":187,"num_write":779,"num_write_kb":222312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3710976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"37","reported_epoch":"55","state":"active+clean","last_fresh":"2020-06-12T02:38:18.377617+0000","last_change":"2020-06-12T02:37:42.384656+0000","last_active":"2020-06-12T02:38:18.377617+0000","last_peered":"2020-06-12T02:38:18.377617+0000","last_clean":"2020-06-12T02:38:18.377617+0000","last_became_active":"2020-06-12T02:37:42.384363+0000","last_became_peered":"2020-06-12T02:37:42.384363+0000","last_unstale":"2020-06-12T02:38:18.377617+0000","last_undegraded":"2020-06-12T02:38:18.377617+0000","last_fullsized":"2020-06-12T02:38:18.377617+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"55'941","reported_seq":"1155","reported_epoch":"55","state":"active+clean","last_fresh":"2020-06-12T02:38:21.350089+0000","last_change":"2020-06-12T02:37:27.902643+0000","last_active":"2020-06-12T02:38:21.350089+0000","last_peered":"2020-06-12T02:38:21.350089+0000","last_clean":"2020-06-12T02:38:21.350089+0000","last_became_active":"2020-06-12T02:37:27.902204+0000","last_became_peered":"2020-06-12T02:37:27.902204+0000","last_unstale":"2020-06-12T02:38:21.350089+0000","last_undegraded":"2020-06-12T02:38:21.350089+0000","last_fullsized":"2020-06-12T02:38:21.350089+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":941,"ondisk_log_size":941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":308,"num_read_kb":172,"num_write":706,"num_write_kb":202032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"56'937","reported_seq":"1183","reported_epoch":"56","state":"active+clean","last_fresh":"2020-06-12T02:38:21.679113+0000","last_change":"2020-06-12T02:38:16.369345+0000","last_active":"2020-06-12T02:38:21.679113+0000","last_peered":"2020-06-12T02:38:21.679113+0000","last_clean":"2020-06-12T02:38:21.679113+0000","last_became_active":"2020-06-12T02:38:16.367448+0000","last_became_peered":"2020-06-12T02:38:16.367448+0000","last_unstale":"2020-06-12T02:38:21.679113+0000","last_undegraded":"2020-06-12T02:38:21.679113+0000","last_fullsized":"2020-06-12T02:38:21.679113+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":937,"ondisk_log_size":937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2326528,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":333,"num_read_kb":182,"num_write":726,"num_write_kb":205888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"56'927","reported_seq":"1165","reported_epoch":"56","state":"active+clean","last_fresh":"2020-06-12T02:38:21.540887+0000","last_change":"2020-06-12T02:37:57.567142+0000","last_active":"2020-06-12T02:38:21.540887+0000","last_peered":"2020-06-12T02:38:21.540887+0000","last_clean":"2020-06-12T02:38:21.540887+0000","last_became_active":"2020-06-12T02:37:57.566462+0000","last_became_peered":"2020-06-12T02:37:57.566462+0000","last_unstale":"2020-06-12T02:38:21.540887+0000","last_undegraded":"2020-06-12T02:38:21.540887+0000","last_fullsized":"2020-06-12T02:38:21.540887+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":927,"ondisk_log_size":927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":307,"num_read_kb":175,"num_write":680,"num_write_kb":187832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"56'502","reported_seq":"672","reported_epoch":"56","state":"active+clean","last_fresh":"2020-06-12T02:38:21.386089+0000","last_change":"2020-06-12T02:38:16.372264+0000","last_active":"2020-06-12T02:38:21.386089+0000","last_peered":"2020-06-12T02:38:21.386089+0000","last_clean":"2020-06-12T02:38:21.386089+0000","last_became_active":"2020-06-12T02:38:16.371502+0000","last_became_peered":"2020-06-12T02:38:16.371502+0000","last_unstale":"2020-06-12T02:38:21.386089+0000","last_undegraded":"2020-06-12T02:38:21.386089+0000","last_fullsized":"2020-06-12T02:38:21.386089+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":502,"ondisk_log_size":502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":161,"num_read_kb":86,"num_write":375,"num_write_kb":109592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2793472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"16","reported_epoch":"55","state":"active+clean","last_fresh":"2020-06-12T02:38:18.379629+0000","last_change":"2020-06-12T02:38:13.661966+0000","last_active":"2020-06-12T02:38:18.379629+0000","last_peered":"2020-06-12T02:38:18.379629+0000","last_clean":"2020-06-12T02:38:18.379629+0000","last_became_active":"2020-06-12T02:38:13.661679+0000","last_became_peered":"2020-06-12T02:38:13.661679+0000","last_unstale":"2020-06-12T02:38:18.379629+0000","last_undegraded":"2020-06-12T02:38:18.379629+0000","last_fullsized":"2020-06-12T02:38:18.379629+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"56'959","reported_seq":"1177","reported_epoch":"56","state":"active+clean","last_fresh":"2020-06-12T02:38:21.634488+0000","last_change":"2020-06-12T02:37:42.379294+0000","last_active":"2020-06-12T02:38:21.634488+0000","last_peered":"2020-06-12T02:38:21.634488+0000","last_clean":"2020-06-12T02:38:21.634488+0000","last_became_active":"2020-06-12T02:37:42.378796+0000","last_became_peered":"2020-06-12T02:37:42.378796+0000","last_unstale":"2020-06-12T02:38:21.634488+0000","last_undegraded":"2020-06-12T02:38:21.634488+0000","last_fullsized":"2020-06-12T02:38:21.634488+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":959,"ondisk_log_size":959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1769472,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":328,"num_read_kb":175,"num_write":718,"num_write_kb":208560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"14","reported_epoch":"55","state":"active+clean","last_fresh":"2020-06-12T02:38:18.421557+0000","last_change":"2020-06-12T02:38:13.664800+0000","last_active":"2020-06-12T02:38:18.421557+0000","last_peered":"2020-06-12T02:38:18.421557+0000","last_clean":"2020-06-12T02:38:18.421557+0000","last_became_active":"2020-06-12T02:38:13.664384+0000","last_became_peered":"2020-06-12T02:38:13.664384+0000","last_unstale":"2020-06-12T02:38:18.421557+0000","last_undegraded":"2020-06-12T02:38:18.421557+0000","last_fullsized":"2020-06-12T02:38:18.421557+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"56'1006","reported_seq":"1255","reported_epoch":"56","state":"active+clean","last_fresh":"2020-06-12T02:38:21.545857+0000","last_change":"2020-06-12T02:38:16.372259+0000","last_active":"2020-06-12T02:38:21.545857+0000","last_peered":"2020-06-12T02:38:21.545857+0000","last_clean":"2020-06-12T02:38:21.545857+0000","last_became_active":"2020-06-12T02:38:16.371577+0000","last_became_peered":"2020-06-12T02:38:16.371577+0000","last_unstale":"2020-06-12T02:38:21.545857+0000","last_undegraded":"2020-06-12T02:38:21.545857+0000","last_fullsized":"2020-06-12T02:38:21.545857+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":1006,"ondisk_log_size":1006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5292032,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":352,"num_read_kb":183,"num_write":777,"num_write_kb":226352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1572864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"49","reported_epoch":"55","state":"active+clean","last_fresh":"2020-06-12T02:38:21.039695+0000","last_change":"2020-06-12T02:38:21.039695+0000","last_active":"2020-06-12T02:38:21.039695+0000","last_peered":"2020-06-12T02:38:21.039695+0000","last_clean":"2020-06-12T02:38:21.039695+0000","last_became_active":"2020-06-12T02:37:16.764958+0000","last_became_peered":"2020-06-12T02:37:16.764958+0000","last_unstale":"2020-06-12T02:38:21.039695+0000","last_undegraded":"2020-06-12T02:38:21.039695+0000","last_fullsized":"2020-06-12T02:38:21.039695+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:38:21.039625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:38:21.039625+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"52'434","reported_seq":"579","reported_epoch":"55","state":"active+clean+wait","last_fresh":"2020-06-12T02:38:18.379589+0000","last_change":"2020-06-12T02:38:16.372481+0000","last_active":"2020-06-12T02:38:18.379589+0000","last_peered":"2020-06-12T02:38:18.379589+0000","last_clean":"2020-06-12T02:38:18.379589+0000","last_became_active":"2020-06-12T02:38:16.371755+0000","last_became_peered":"2020-06-12T02:38:16.371755+0000","last_unstale":"2020-06-12T02:38:18.379589+0000","last_undegraded":"2020-06-12T02:38:18.379589+0000","last_fullsized":"2020-06-12T02:38:18.379589+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":434,"ondisk_log_size":434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":157,"num_read_kb":83,"num_write":342,"num_write_kb":97328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"36","reported_epoch":"55","state":"active+clean","last_fresh":"2020-06-12T02:38:18.372851+0000","last_change":"2020-06-12T02:37:42.376260+0000","last_active":"2020-06-12T02:38:18.372851+0000","last_peered":"2020-06-12T02:38:18.372851+0000","last_clean":"2020-06-12T02:38:18.372851+0000","last_became_active":"2020-06-12T02:37:42.375902+0000","last_became_peered":"2020-06-12T02:37:42.375902+0000","last_unstale":"2020-06-12T02:38:18.372851+0000","last_undegraded":"2020-06-12T02:38:18.372851+0000","last_fullsized":"2020-06-12T02:38:18.372851+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"15'2","reported_seq":"44","reported_epoch":"55","state":"active+clean","last_fresh":"2020-06-12T02:38:18.373846+0000","last_change":"2020-06-12T02:38:05.892468+0000","last_active":"2020-06-12T02:38:18.373846+0000","last_peered":"2020-06-12T02:38:18.373846+0000","last_clean":"2020-06-12T02:38:18.373846+0000","last_became_active":"2020-06-12T02:37:54.558125+0000","last_became_peered":"2020-06-12T02:37:54.558125+0000","last_unstale":"2020-06-12T02:38:18.373846+0000","last_undegraded":"2020-06-12T02:38:18.373846+0000","last_fullsized":"2020-06-12T02:38:18.373846+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":3,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:04.419204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:04.419204+0000","last_clean_scrub_stamp":"2020-06-12T02:37:04.419204+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"56'405","reported_seq":"524","reported_epoch":"56","state":"active+clean","last_fresh":"2020-06-12T02:38:21.689900+0000","last_change":"2020-06-12T02:37:27.898985+0000","last_active":"2020-06-12T02:38:21.689900+0000","last_peered":"2020-06-12T02:38:21.689900+0000","last_clean":"2020-06-12T02:38:21.689900+0000","last_became_active":"2020-06-12T02:37:27.898516+0000","last_became_peered":"2020-06-12T02:37:27.898516+0000","last_unstale":"2020-06-12T02:38:21.689900+0000","last_undegraded":"2020-06-12T02:38:21.689900+0000","last_fullsized":"2020-06-12T02:38:21.689900+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":405,"ondisk_log_size":405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2342912,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":143,"num_read_kb":77,"num_write":308,"num_write_kb":88424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"49","reported_epoch":"55","state":"active+clean","last_fresh":"2020-06-12T02:38:18.373713+0000","last_change":"2020-06-12T02:38:13.661120+0000","last_active":"2020-06-12T02:38:18.373713+0000","last_peered":"2020-06-12T02:38:18.373713+0000","last_clean":"2020-06-12T02:38:18.373713+0000","last_became_active":"2020-06-12T02:38:13.660654+0000","last_became_peered":"2020-06-12T02:38:13.660654+0000","last_unstale":"2020-06-12T02:38:18.373713+0000","last_undegraded":"2020-06-12T02:38:18.373713+0000","last_fullsized":"2020-06-12T02:38:18.373713+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"42","reported_epoch":"55","state":"active+clean","last_fresh":"2020-06-12T02:38:18.379846+0000","last_change":"2020-06-12T02:37:27.901368+0000","last_active":"2020-06-12T02:38:18.379846+0000","last_peered":"2020-06-12T02:38:18.379846+0000","last_clean":"2020-06-12T02:38:18.379846+0000","last_became_active":"2020-06-12T02:37:27.900888+0000","last_became_peered":"2020-06-12T02:37:27.900888+0000","last_unstale":"2020-06-12T02:38:18.379846+0000","last_undegraded":"2020-06-12T02:38:18.379846+0000","last_fullsized":"2020-06-12T02:38:18.379846+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"41","reported_epoch":"55","state":"active+clean","last_fresh":"2020-06-12T02:38:18.373915+0000","last_change":"2020-06-12T02:38:05.892563+0000","last_active":"2020-06-12T02:38:18.373915+0000","last_peered":"2020-06-12T02:38:18.373915+0000","last_clean":"2020-06-12T02:38:18.373915+0000","last_became_active":"2020-06-12T02:37:54.557420+0000","last_became_peered":"2020-06-12T02:37:54.557420+0000","last_unstale":"2020-06-12T02:38:18.373915+0000","last_undegraded":"2020-06-12T02:38:18.373915+0000","last_fullsized":"2020-06-12T02:38:18.373915+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"17","reported_epoch":"55","state":"active+clean","last_fresh":"2020-06-12T02:38:18.374270+0000","last_change":"2020-06-12T02:38:13.666985+0000","last_active":"2020-06-12T02:38:18.374270+0000","last_peered":"2020-06-12T02:38:18.374270+0000","last_clean":"2020-06-12T02:38:18.374270+0000","last_became_active":"2020-06-12T02:38:13.666441+0000","last_became_peered":"2020-06-12T02:38:13.666441+0000","last_unstale":"2020-06-12T02:38:18.374270+0000","last_undegraded":"2020-06-12T02:38:18.374270+0000","last_fullsized":"2020-06-12T02:38:18.374270+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"56'962","reported_seq":"1204","reported_epoch":"56","state":"active+clean","last_fresh":"2020-06-12T02:38:21.634361+0000","last_change":"2020-06-12T02:37:57.567683+0000","last_active":"2020-06-12T02:38:21.634361+0000","last_peered":"2020-06-12T02:38:21.634361+0000","last_clean":"2020-06-12T02:38:21.634361+0000","last_became_active":"2020-06-12T02:37:57.566865+0000","last_became_peered":"2020-06-12T02:37:57.566865+0000","last_unstale":"2020-06-12T02:38:21.634361+0000","last_undegraded":"2020-06-12T02:38:21.634361+0000","last_fullsized":"2020-06-12T02:38:21.634361+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":962,"ondisk_log_size":962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4235264,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":331,"num_read_kb":182,"num_write":745,"num_write_kb":213536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2793472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"16","reported_epoch":"55","state":"active+clean","last_fresh":"2020-06-12T02:38:18.421494+0000","last_change":"2020-06-12T02:38:13.664365+0000","last_active":"2020-06-12T02:38:18.421494+0000","last_peered":"2020-06-12T02:38:18.421494+0000","last_clean":"2020-06-12T02:38:18.421494+0000","last_became_active":"2020-06-12T02:38:13.663999+0000","last_became_peered":"2020-06-12T02:38:13.663999+0000","last_unstale":"2020-06-12T02:38:18.421494+0000","last_undegraded":"2020-06-12T02:38:18.421494+0000","last_fullsized":"2020-06-12T02:38:18.421494+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"58","reported_epoch":"55","state":"active+clean","last_fresh":"2020-06-12T02:38:18.421477+0000","last_change":"2020-06-12T02:37:54.555377+0000","last_active":"2020-06-12T02:38:18.421477+0000","last_peered":"2020-06-12T02:38:18.421477+0000","last_clean":"2020-06-12T02:38:18.421477+0000","last_became_active":"2020-06-12T02:37:54.554986+0000","last_became_peered":"2020-06-12T02:37:54.554986+0000","last_unstale":"2020-06-12T02:38:18.421477+0000","last_undegraded":"2020-06-12T02:38:18.421477+0000","last_fullsized":"2020-06-12T02:38:18.421477+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"51","reported_epoch":"55","state":"active+clean","last_fresh":"2020-06-12T02:38:18.379820+0000","last_change":"2020-06-12T02:38:13.666662+0000","last_active":"2020-06-12T02:38:18.379820+0000","last_peered":"2020-06-12T02:38:18.379820+0000","last_clean":"2020-06-12T02:38:18.379820+0000","last_became_active":"2020-06-12T02:38:13.666144+0000","last_became_peered":"2020-06-12T02:38:13.666144+0000","last_unstale":"2020-06-12T02:38:18.379820+0000","last_undegraded":"2020-06-12T02:38:18.379820+0000","last_fullsized":"2020-06-12T02:38:18.379820+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":17473536,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3294,"num_read_kb":1783,"num_write":7321,"num_write_kb":2089648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":13869056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":30240768,"data_stored":30240768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9676,"ondisk_log_size":9676,"up":48,"acting":48,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26157,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":34,"seq":146028888071,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1052008,"kb_used_data":3424,"kb_used_omap":25,"kb_used_meta":1048550,"kb_avail":93319832,"statfs":{"total":96636764160,"available":95559507968,"internally_reserved":1073741824,"allocated":3506176,"data_stored":3162429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26157,"internal_metadata":1073715667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.806}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.835}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.0099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.5550000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.823}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.9860000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.8969999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574865,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050804,"kb_used_data":2220,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321036,"statfs":{"total":96636764160,"available":95560740864,"internally_reserved":1073741824,"allocated":2273280,"data_stored":1935607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.655,"5min":1.655,"15min":1.655},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.82999999999999996}]},{"osd":1,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.335,"5min":1.335,"15min":1.335},"last":0.621},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.923,"5min":1.923,"15min":1.923},"last":0.94999999999999996}]},{"osd":2,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.01,"15min":1.01},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.6699999999999999,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.653,"5min":1.653,"15min":1.653},"last":0.85699999999999998}]},{"osd":3,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.145,"15min":1.145},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.5649999999999999,"5min":1.5649999999999999,"15min":1.5649999999999999},"last":1.165},{"interface":"front","average":{"1min":1.143,"5min":1.143,"15min":1.143},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.556,"5min":1.556,"15min":1.556},"last":1.1240000000000001}]},{"osd":4,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.246,"5min":1.246,"15min":1.246},"last":0.81000000000000005},{"interface":"front","average":{"1min":1.071,"5min":1.071,"15min":1.071},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.673,"5min":1.673,"15min":1.673},"last":0.97299999999999998}]},{"osd":5,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.8120000000000001,"5min":1.8120000000000001,"15min":1.8120000000000001},"last":1},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.677,"5min":1.677,"15min":1.677},"last":1.034}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2589999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049204,"kb_used_data":620,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322636,"statfs":{"total":96636764160,"available":95562379264,"internally_reserved":1073741824,"allocated":634880,"data_stored":297207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 12 02:38:14 2020","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.548,"5min":1.548,"15min":1.548},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":2.149,"5min":2.149,"15min":2.149},"last":1.216}]},{"osd":2,"last update":"Fri Jun 12 02:38:14 2020","interfaces":[{"interface":"back","average":{"1min":1,"5min":1,"15min":1},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":3.0009999999999999,"5min":3.0009999999999999,"15min":3.0009999999999999},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.014,"5min":1.014,"15min":1.014},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":2.4159999999999999,"5min":2.4159999999999999,"15min":2.4159999999999999},"last":1.079}]},{"osd":3,"last update":"Fri Jun 12 02:38:14 2020","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":2.133,"5min":2.133,"15min":2.133},"last":1.3660000000000001},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":2.4609999999999999,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":0.77800000000000002}]},{"osd":4,"last update":"Fri Jun 12 02:38:14 2020","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":2.1749999999999998,"5min":2.1749999999999998,"15min":2.1749999999999998},"last":1.4259999999999999},{"interface":"front","average":{"1min":1.302,"5min":1.302,"15min":1.302},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":2.2919999999999998,"5min":2.2919999999999998,"15min":2.2919999999999998},"last":1.8149999999999999}]},{"osd":5,"last update":"Fri Jun 12 02:38:14 2020","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.3200000000000001,"15min":1.3200000000000001},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":2.3090000000000002,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":1.044},{"interface":"front","average":{"1min":1.448,"5min":1.448,"15min":1.448},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.835,"5min":2.835,"15min":2.835},"last":0.92300000000000004}]},{"osd":6,"last update":"Fri Jun 12 02:38:14 2020","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.389,"15min":1.389},"min":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"max":{"1min":2.4649999999999999,"5min":2.4649999999999999,"15min":2.4649999999999999},"last":1.1399999999999999},{"interface":"front","average":{"1min":1.6299999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"max":{"1min":3.1589999999999998,"5min":3.1589999999999998,"15min":3.1589999999999998},"last":1.349}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9990000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574866,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1054088,"kb_used_data":5504,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93317752,"statfs":{"total":96636764160,"available":95557378048,"internally_reserved":1073741824,"allocated":5636096,"data_stored":5292349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,6],"upper_bound":16384},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jun 12 02:38:16 2020","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.7789999999999999,"5min":1.7789999999999999,"15min":1.7789999999999999},"last":1.135}]},{"osd":2,"last update":"Fri Jun 12 02:38:16 2020","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":2.141,"5min":2.141,"15min":2.141},"last":1.2130000000000001},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.873,"5min":1.873,"15min":1.873},"last":0.94699999999999995}]},{"osd":3,"last update":"Fri Jun 12 02:38:16 2020","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":2.2320000000000002,"5min":2.2320000000000002,"15min":2.2320000000000002},"last":1.242},{"interface":"front","average":{"1min":1.23,"5min":1.23,"15min":1.23},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.8220000000000001,"5min":1.8220000000000001,"15min":1.8220000000000001},"last":1.3620000000000001}]},{"osd":4,"last update":"Fri Jun 12 02:38:16 2020","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.7709999999999999,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":1.2849999999999999},{"interface":"front","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.554,"5min":1.554,"15min":1.554},"last":1.397}]},{"osd":5,"last update":"Fri Jun 12 02:38:16 2020","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.212,"15min":1.212},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":1.6699999999999999,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":0.96699999999999997},{"interface":"front","average":{"1min":1.502,"5min":1.502,"15min":1.502},"min":{"1min":1.006,"5min":1.006,"15min":1.006},"max":{"1min":2.6240000000000001,"5min":2.6240000000000001,"15min":2.6240000000000001},"last":1.49}]},{"osd":6,"last update":"Fri Jun 12 02:38:16 2020","interfaces":[{"interface":"back","average":{"1min":1.5840000000000001,"5min":1.5840000000000001,"15min":1.5840000000000001},"min":{"1min":1.157,"5min":1.157,"15min":1.157},"max":{"1min":3.2440000000000002,"5min":3.2440000000000002,"15min":3.2440000000000002},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.335,"5min":1.335,"15min":1.335},"min":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"max":{"1min":3.105,"5min":3.105,"15min":3.105},"last":1.427}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7529999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574865,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1052308,"kb_used_data":3724,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93319532,"statfs":{"total":96636764160,"available":95559200768,"internally_reserved":1073741824,"allocated":3813376,"data_stored":3475703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 12 02:38:10 2020","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.871,"15min":0.871},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.9259999999999999,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":3.0049999999999999,"5min":3.0049999999999999,"15min":3.0049999999999999},"last":0.53400000000000003}]},{"osd":1,"last update":"Fri Jun 12 02:38:10 2020","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.046,"15min":1.046},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":2.21,"5min":2.21,"15min":2.21},"last":1.4390000000000001},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":2.5219999999999998,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":1.042}]},{"osd":3,"last update":"Fri Jun 12 02:38:10 2020","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1,"5min":1,"15min":1},"last":1.0580000000000001},{"interface":"front","average":{"1min":1.145,"5min":1.145,"15min":1.145},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.2759999999999998,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":0.93100000000000005}]},{"osd":4,"last update":"Fri Jun 12 02:38:10 2020","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.321,"15min":1.321},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":2.6949999999999998,"5min":2.6949999999999998,"15min":2.6949999999999998},"last":0.71699999999999997},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"min":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"max":{"1min":1.7549999999999999,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":0.82299999999999995}]},{"osd":5,"last update":"Fri Jun 12 02:38:10 2020","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"min":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"max":{"1min":2.1419999999999999,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":0.78100000000000003},{"interface":"front","average":{"1min":1.399,"5min":1.399,"15min":1.399},"min":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"max":{"1min":2.851,"5min":2.851,"15min":2.851},"last":1.23}]},{"osd":6,"last update":"Fri Jun 12 02:38:10 2020","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":1.7110000000000001,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":0.91000000000000003},{"interface":"front","average":{"1min":1.462,"5min":1.462,"15min":1.462},"min":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"max":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":1.0860000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95799999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574865,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1051204,"kb_used_data":2620,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93320636,"statfs":{"total":96636764160,"available":95560331264,"internally_reserved":1073741824,"allocated":2682880,"data_stored":2345207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 12 02:38:12 2020","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":2.0339999999999998,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":1.7989999999999999},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.9219999999999999,"5min":1.9219999999999999,"15min":1.9219999999999999},"last":1.0329999999999999}]},{"osd":1,"last update":"Fri Jun 12 02:38:12 2020","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.9670000000000001,"5min":1.9670000000000001,"15min":1.9670000000000001},"last":1.169},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":2.9969999999999999,"5min":2.9969999999999999,"15min":2.9969999999999999},"last":2.1859999999999999}]},{"osd":2,"last update":"Fri Jun 12 02:38:12 2020","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":4.1079999999999997,"5min":4.1079999999999997,"15min":4.1079999999999997},"last":1.48},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":2.504,"5min":2.504,"15min":2.504},"last":2.6320000000000001}]},{"osd":4,"last update":"Fri Jun 12 02:38:12 2020","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":2.359,"5min":2.359,"15min":2.359},"last":2.5129999999999999},{"interface":"front","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":3.9319999999999999,"5min":3.9319999999999999,"15min":3.9319999999999999},"last":2.2930000000000001}]},{"osd":5,"last update":"Fri Jun 12 02:38:12 2020","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":2.4279999999999999,"5min":2.4279999999999999,"15min":2.4279999999999999},"last":1.7390000000000001},{"interface":"front","average":{"1min":1.452,"5min":1.452,"15min":1.452},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":5.0579999999999998,"5min":5.0579999999999998,"15min":5.0579999999999998},"last":2.351}]},{"osd":6,"last update":"Fri Jun 12 02:38:12 2020","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.29,"15min":1.29},"min":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"max":{"1min":4.0410000000000004,"5min":4.0410000000000004,"15min":4.0410000000000004},"last":2.6880000000000002},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.3620000000000001,"15min":1.3620000000000001},"min":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"max":{"1min":3.0230000000000001,"5min":3.0230000000000001,"15min":3.0230000000000001},"last":2.456}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.028}]}]},{"osd":4,"up_from":13,"seq":55834574864,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1053360,"kb_used_data":4776,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93318480,"statfs":{"total":96636764160,"available":95558123520,"internally_reserved":1073741824,"allocated":4890624,"data_stored":4546877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.712,"5min":1.712,"15min":1.712},"last":0.54200000000000004},{"interface":"front","average":{"1min":1.02,"5min":1.02,"15min":1.02},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":2.855,"5min":2.855,"15min":2.855},"last":0.64900000000000002}]},{"osd":1,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.8240000000000001,"5min":1.8240000000000001,"15min":1.8240000000000001},"last":0.88300000000000001}]},{"osd":2,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":3.1480000000000001,"5min":3.1480000000000001,"15min":3.1480000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"min":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"max":{"1min":3.1259999999999999,"5min":3.1259999999999999,"15min":3.1259999999999999},"last":1.0449999999999999}]},{"osd":3,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.079,"15min":1.079},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":2.411,"5min":2.411,"15min":2.411},"last":1.212},{"interface":"front","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":2.734,"5min":2.734,"15min":2.734},"last":1.25}]},{"osd":5,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.04,"15min":1.04},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.6399999999999999,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.115,"5min":1.115,"15min":1.115},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"last":0.92600000000000005}]},{"osd":6,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":0.998,"15min":0.998},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":1.163},{"interface":"front","average":{"1min":1.341,"5min":1.341,"15min":1.341},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":2.1850000000000001,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":1.2350000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3100000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574864,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1051432,"kb_used_data":2848,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93320408,"statfs":{"total":96636764160,"available":95560097792,"internally_reserved":1073741824,"allocated":2916352,"data_stored":2582309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 12 02:38:13 2020","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.456,"5min":1.456,"15min":1.456},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.597,"5min":1.597,"15min":1.597},"last":0.627}]},{"osd":1,"last update":"Fri Jun 12 02:38:13 2020","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":1.119},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":1.028}]},{"osd":2,"last update":"Fri Jun 12 02:38:13 2020","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":1.1399999999999999},{"interface":"front","average":{"1min":1.012,"5min":1.012,"15min":1.012},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.4259999999999999,"5min":1.4259999999999999,"15min":1.4259999999999999},"last":0.81899999999999995}]},{"osd":3,"last update":"Fri Jun 12 02:38:13 2020","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":2.004,"5min":2.004,"15min":2.004},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.627,"5min":1.627,"15min":1.627},"last":1.379}]},{"osd":4,"last update":"Fri Jun 12 02:38:13 2020","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.997,"15min":0.997},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.796,"5min":1.796,"15min":1.796},"last":1.3200000000000001},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":0.95299999999999996}]},{"osd":6,"last update":"Fri Jun 12 02:38:13 2020","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.827,"5min":1.827,"15min":1.827},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.228,"5min":1.228,"15min":1.228},"min":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"max":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":1.464}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.304}]}]}]}} 2020-06-12T02:38:23.916 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set device_health_metrics pgp_num 1 2020-06-12T02:38:23.918 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-06-12T02:38:23.918 INFO:tasks.rados.rados.0.smithi107.stdout:5604: writing smithi10729407-3 from 1073152 to 1318912 tid 1 2020-06-12T02:38:23.919 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:23.919 INFO:tasks.rados.rados.0.smithi107.stdout:5605: seq_num 5604 ranges {2482176=262144} 2020-06-12T02:38:23.930 INFO:tasks.rados.rados.0.smithi107.stdout:5605: writing smithi10729407-9 from 2482176 to 2744320 tid 1 2020-06-12T02:38:23.930 INFO:tasks.rados.rados.0.smithi107.stdout:5606: delete oid 13 current snap is 0 2020-06-12T02:38:23.931 INFO:tasks.rados.rados.0.smithi107.stdout:5604: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:23.931 INFO:tasks.rados.rados.0.smithi107.stdout:5604: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:23.931 INFO:tasks.rados.rados.0.smithi107.stdout:5604: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:23.931 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1004 (ObjNum 5603 snap 0 seq_num 5603) dirty exists 2020-06-12T02:38:23.931 INFO:tasks.rados.rados.0.smithi107.stdout:5604: left oid 3 (ObjNum 5603 snap 0 seq_num 5603) 2020-06-12T02:38:23.933 INFO:tasks.rados.rados.0.smithi107.stdout:5604: done (3 left) 2020-06-12T02:38:23.933 INFO:tasks.rados.rados.0.smithi107.stdout:5606: done (2 left) 2020-06-12T02:38:23.934 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:23.934 INFO:tasks.rados.rados.0.smithi107.stdout:5607: seq_num 5606 ranges {0=294912} 2020-06-12T02:38:23.935 INFO:tasks.rados.rados.0.smithi107.stdout:5607: writing smithi10729407-12 from 0 to 294912 tid 1 2020-06-12T02:38:23.936 INFO:tasks.rados.rados.0.smithi107.stdout:5608: delete oid 8 current snap is 0 2020-06-12T02:38:23.936 INFO:tasks.rados.rados.0.smithi107.stdout:5605: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:23.936 INFO:tasks.rados.rados.0.smithi107.stdout:5605: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:23.936 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2020-06-12T02:38:23.937 INFO:tasks.rados.rados.0.smithi107.stdout:5605: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:23.937 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1020 (ObjNum 5604 snap 0 seq_num 5604) dirty exists 2020-06-12T02:38:23.937 INFO:tasks.rados.rados.0.smithi107.stdout:5605: left oid 9 (ObjNum 5604 snap 0 seq_num 5604) 2020-06-12T02:38:23.938 INFO:tasks.rados.rados.0.smithi107.stdout:5605: done (3 left) 2020-06-12T02:38:23.938 INFO:tasks.rados.rados.0.smithi107.stdout:5608: done (2 left) 2020-06-12T02:38:23.939 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:23.939 INFO:tasks.rados.rados.0.smithi107.stdout:5609: seq_num 5608 ranges {2744320=327680} 2020-06-12T02:38:23.940 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-06-12T02:38:23.941 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-06-12T02:38:23.941 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-06-12T02:38:23.941 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-06-12T02:38:23.941 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-06-12T02:38:23.942 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-06-12T02:38:23.942 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-06-12T02:38:23.942 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-06-12T02:38:23.956 INFO:tasks.rados.rados.0.smithi107.stdout:5609: writing smithi10729407-9 from 2744320 to 3072000 tid 1 2020-06-12T02:38:23.956 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:23.956 INFO:tasks.rados.rados.0.smithi107.stdout:5610: seq_num 5609 ranges {0=344064} 2020-06-12T02:38:23.958 INFO:tasks.rados.rados.0.smithi107.stdout:5610: writing smithi10729407-14 from 0 to 344064 tid 1 2020-06-12T02:38:23.958 INFO:tasks.rados.rados.0.smithi107.stdout:5611: delete oid 20 current snap is 0 2020-06-12T02:38:23.958 INFO:tasks.rados.rados.0.smithi107.stdout:5607: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:23.958 INFO:tasks.rados.rados.0.smithi107.stdout:5607: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:23.958 INFO:tasks.rados.rados.0.smithi107.stdout:5607: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:23.959 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 987 (ObjNum 5606 snap 0 seq_num 5606) dirty exists 2020-06-12T02:38:23.959 INFO:tasks.rados.rados.0.smithi107.stdout:5607: left oid 12 (ObjNum 5606 snap 0 seq_num 5606) 2020-06-12T02:38:23.961 INFO:tasks.rados.rados.0.smithi107.stdout:5609: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:23.962 INFO:tasks.rados.rados.0.smithi107.stdout:5607: done (4 left) 2020-06-12T02:38:23.962 INFO:tasks.rados.rados.0.smithi107.stdout:5611: done (3 left) 2020-06-12T02:38:23.963 INFO:tasks.rados.rados.0.smithi107.stdout:5612: delete oid 3 current snap is 0 2020-06-12T02:38:23.965 INFO:tasks.rados.rados.0.smithi107.stdout:5610: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:23.965 INFO:tasks.rados.rados.0.smithi107.stdout:5609: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:23.965 INFO:tasks.rados.rados.0.smithi107.stdout:5610: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:23.968 INFO:tasks.rados.rados.0.smithi107.stdout:5612: done (3 left) 2020-06-12T02:38:23.968 INFO:tasks.rados.rados.0.smithi107.stdout:5613: delete oid 1 current snap is 0 2020-06-12T02:38:23.968 INFO:tasks.rados.rados.0.smithi107.stdout:5609: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:23.968 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1022 (ObjNum 5608 snap 0 seq_num 5608) dirty exists 2020-06-12T02:38:23.969 INFO:tasks.rados.rados.0.smithi107.stdout:5609: left oid 9 (ObjNum 5608 snap 0 seq_num 5608) 2020-06-12T02:38:23.969 INFO:tasks.rados.rados.0.smithi107.stdout:5610: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:23.969 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1014 (ObjNum 5609 snap 0 seq_num 5609) dirty exists 2020-06-12T02:38:23.970 INFO:tasks.rados.rados.0.smithi107.stdout:5610: left oid 14 (ObjNum 5609 snap 0 seq_num 5609) 2020-06-12T02:38:23.974 INFO:tasks.rados.rados.0.smithi107.stdout:5609: done (3 left) 2020-06-12T02:38:23.974 INFO:tasks.rados.rados.0.smithi107.stdout:5610: done (2 left) 2020-06-12T02:38:23.974 INFO:tasks.rados.rados.0.smithi107.stdout:5613: done (1 left) 2020-06-12T02:38:23.974 INFO:tasks.rados.rados.0.smithi107.stdout:5614: delete oid 11 current snap is 0 2020-06-12T02:38:23.976 INFO:tasks.rados.rados.0.smithi107.stdout:5614: done (1 left) 2020-06-12T02:38:23.977 INFO:tasks.rados.rados.0.smithi107.stdout:5615: delete oid 1 current snap is 0 2020-06-12T02:38:23.979 INFO:tasks.rados.rados.0.smithi107.stdout:5615: done (1 left) 2020-06-12T02:38:23.979 INFO:tasks.rados.rados.0.smithi107.stdout:5616: delete oid 19 current snap is 0 2020-06-12T02:38:23.981 INFO:tasks.rados.rados.0.smithi107.stdout:5616: done (1 left) 2020-06-12T02:38:23.981 INFO:tasks.rados.rados.0.smithi107.stdout:5617: delete oid 4 current snap is 0 2020-06-12T02:38:23.983 INFO:tasks.rados.rados.0.smithi107.stdout:5617: done (1 left) 2020-06-12T02:38:23.984 INFO:tasks.rados.rados.0.smithi107.stdout:5618: delete oid 19 current snap is 0 2020-06-12T02:38:23.985 INFO:tasks.rados.rados.0.smithi107.stdout:5618: done (1 left) 2020-06-12T02:38:23.985 INFO:tasks.rados.rados.0.smithi107.stdout:5619: delete oid 6 current snap is 0 2020-06-12T02:38:23.987 INFO:tasks.rados.rados.0.smithi107.stdout:5619: done (1 left) 2020-06-12T02:38:23.987 INFO:tasks.rados.rados.0.smithi107.stdout:5620: delete oid 1 current snap is 0 2020-06-12T02:38:23.992 INFO:tasks.rados.rados.0.smithi107.stdout:5620: done (1 left) 2020-06-12T02:38:23.992 INFO:tasks.rados.rados.0.smithi107.stdout:5621: delete oid 17 current snap is 0 2020-06-12T02:38:23.994 INFO:tasks.rados.rados.0.smithi107.stdout:5621: done (1 left) 2020-06-12T02:38:24.000 INFO:tasks.rados.rados.0.smithi107.stdout:5622: delete oid 8 current snap is 0 2020-06-12T02:38:24.001 INFO:tasks.rados.rados.0.smithi107.stdout:5622: done (1 left) 2020-06-12T02:38:24.002 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:24.002 INFO:tasks.rados.rados.0.smithi107.stdout:5623: seq_num 5622 ranges {0=565248} 2020-06-12T02:38:24.002 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:23.999+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:24.003 INFO:tasks.rados.rados.0.smithi107.stdout:5623: writing smithi10729407-19 from 0 to 565248 tid 1 2020-06-12T02:38:24.003 INFO:tasks.rados.rados.0.smithi107.stdout:5624: delete oid 4 current snap is 0 2020-06-12T02:38:24.014 INFO:tasks.rados.rados.0.smithi107.stdout:5624: done (2 left) 2020-06-12T02:38:24.015 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:24.015 INFO:tasks.rados.rados.0.smithi107.stdout:5625: seq_num 5624 ranges {1859584=598016} 2020-06-12T02:38:24.019 INFO:tasks.rados.rados.0.smithi107.stdout:5625: writing smithi10729407-15 from 1859584 to 2457600 tid 1 2020-06-12T02:38:24.020 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:38:24.020 INFO:tasks.rados.rados.0.smithi107.stdout:5626: seq_num 5625 ranges {0=614400} 2020-06-12T02:38:24.023 INFO:tasks.rados.rados.0.smithi107.stdout:5626: writing smithi10729407-20 from 0 to 614400 tid 1 2020-06-12T02:38:24.024 INFO:tasks.rados.rados.0.smithi107.stdout:5627: delete oid 17 current snap is 0 2020-06-12T02:38:24.024 INFO:tasks.rados.rados.0.smithi107.stdout:5623: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:24.024 INFO:tasks.rados.rados.0.smithi107.stdout:5623: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:24.024 INFO:tasks.rados.rados.0.smithi107.stdout:5623: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:24.025 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 991 (ObjNum 5622 snap 0 seq_num 5622) dirty exists 2020-06-12T02:38:24.025 INFO:tasks.rados.rados.0.smithi107.stdout:5623: left oid 19 (ObjNum 5622 snap 0 seq_num 5622) 2020-06-12T02:38:24.028 INFO:tasks.rados.rados.0.smithi107.stdout:5625: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:24.029 INFO:tasks.rados.rados.0.smithi107.stdout:5623: done (4 left) 2020-06-12T02:38:24.029 INFO:tasks.rados.rados.0.smithi107.stdout:5627: done (3 left) 2020-06-12T02:38:24.029 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:24.029 INFO:tasks.rados.rados.0.smithi107.stdout:5628: seq_num 5627 ranges {0=647168} 2020-06-12T02:38:24.032 INFO:tasks.rados.rados.0.smithi107.stdout:5628: writing smithi10729407-17 from 0 to 647168 tid 1 2020-06-12T02:38:24.033 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:38:24.033 INFO:tasks.rados.rados.0.smithi107.stdout:5629: seq_num 5628 ranges {0=663552} 2020-06-12T02:38:24.036 INFO:tasks.rados.rados.0.smithi107.stdout:5629: writing smithi10729407-16 from 0 to 663552 tid 1 2020-06-12T02:38:24.036 INFO:tasks.rados.rados.0.smithi107.stdout:5630: delete oid 1 current snap is 0 2020-06-12T02:38:24.037 INFO:tasks.rados.rados.0.smithi107.stdout:5626: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:24.037 INFO:tasks.rados.rados.0.smithi107.stdout:5625: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:24.042 INFO:tasks.rados.rados.0.smithi107.stdout:5630: done (5 left) 2020-06-12T02:38:24.043 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:24.043 INFO:tasks.rados.rados.0.smithi107.stdout:5631: seq_num 5630 ranges {0=696320} 2020-06-12T02:38:24.047 INFO:tasks.rados.rados.0.smithi107.stdout:5631: writing smithi10729407-8 from 0 to 696320 tid 1 2020-06-12T02:38:24.048 INFO:tasks.rados.rados.0.smithi107.stdout:5632: delete oid 11 current snap is 0 2020-06-12T02:38:24.048 INFO:tasks.rados.rados.0.smithi107.stdout:5625: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:24.048 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1021 (ObjNum 5624 snap 0 seq_num 5624) dirty exists 2020-06-12T02:38:24.049 INFO:tasks.rados.rados.0.smithi107.stdout:5625: left oid 15 (ObjNum 5624 snap 0 seq_num 5624) 2020-06-12T02:38:24.049 INFO:tasks.rados.rados.0.smithi107.stdout:5628: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:24.049 INFO:tasks.rados.rados.0.smithi107.stdout:5626: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:24.050 INFO:tasks.rados.rados.0.smithi107.stdout:5628: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:24.050 INFO:tasks.rados.rados.0.smithi107.stdout:5629: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:24.050 INFO:tasks.rados.rados.0.smithi107.stdout:5626: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:24.051 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 441 (ObjNum 5625 snap 0 seq_num 5625) dirty exists 2020-06-12T02:38:24.051 INFO:tasks.rados.rados.0.smithi107.stdout:5626: left oid 20 (ObjNum 5625 snap 0 seq_num 5625) 2020-06-12T02:38:24.051 INFO:tasks.rados.rados.0.smithi107.stdout:5629: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:24.052 INFO:tasks.rados.rados.0.smithi107.stdout:5628: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:24.052 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1049 (ObjNum 5627 snap 0 seq_num 5627) dirty exists 2020-06-12T02:38:24.052 INFO:tasks.rados.rados.0.smithi107.stdout:5628: left oid 17 (ObjNum 5627 snap 0 seq_num 5627) 2020-06-12T02:38:24.053 INFO:tasks.rados.rados.0.smithi107.stdout:5629: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:24.053 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1007 (ObjNum 5628 snap 0 seq_num 5628) dirty exists 2020-06-12T02:38:24.055 INFO:tasks.rados.rados.0.smithi107.stdout:5629: left oid 16 (ObjNum 5628 snap 0 seq_num 5628) 2020-06-12T02:38:24.057 INFO:tasks.rados.rados.0.smithi107.stdout:5625: done (6 left) 2020-06-12T02:38:24.057 INFO:tasks.rados.rados.0.smithi107.stdout:5626: done (5 left) 2020-06-12T02:38:24.058 INFO:tasks.rados.rados.0.smithi107.stdout:5628: done (4 left) 2020-06-12T02:38:24.058 INFO:tasks.rados.rados.0.smithi107.stdout:5629: done (3 left) 2020-06-12T02:38:24.058 INFO:tasks.rados.rados.0.smithi107.stdout:5632: done (2 left) 2020-06-12T02:38:24.059 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:24.059 INFO:tasks.rados.rados.0.smithi107.stdout:5633: seq_num 5632 ranges {565248=729088} 2020-06-12T02:38:24.059 INFO:tasks.rados.rados.0.smithi107.stdout:5633: writing smithi10729407-19 from 565248 to 1294336 tid 1 2020-06-12T02:38:24.060 INFO:tasks.rados.rados.0.smithi107.stdout:5634: delete oid 15 current snap is 0 2020-06-12T02:38:24.060 INFO:tasks.rados.rados.0.smithi107.stdout:5631: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:24.061 INFO:tasks.rados.rados.0.smithi107.stdout:5631: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:24.062 INFO:tasks.rados.rados.0.smithi107.stdout:5631: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:24.062 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1023 (ObjNum 5630 snap 0 seq_num 5630) dirty exists 2020-06-12T02:38:24.063 INFO:tasks.rados.rados.0.smithi107.stdout:5631: left oid 8 (ObjNum 5630 snap 0 seq_num 5630) 2020-06-12T02:38:24.063 INFO:tasks.rados.rados.0.smithi107.stdout:5631: done (3 left) 2020-06-12T02:38:24.063 INFO:tasks.rados.rados.0.smithi107.stdout:5634: done (2 left) 2020-06-12T02:38:24.064 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:24.064 INFO:tasks.rados.rados.0.smithi107.stdout:5635: seq_num 5634 ranges {696320=761856} 2020-06-12T02:38:24.070 INFO:tasks.rados.rados.0.smithi107.stdout:5635: writing smithi10729407-8 from 696320 to 1458176 tid 1 2020-06-12T02:38:24.070 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:24.070 INFO:tasks.rados.rados.0.smithi107.stdout:5636: seq_num 5635 ranges {294912=778240} 2020-06-12T02:38:24.077 INFO:tasks.rados.rados.0.smithi107.stdout:5636: writing smithi10729407-12 from 294912 to 1073152 tid 1 2020-06-12T02:38:24.077 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:24.077 INFO:tasks.rados.rados.0.smithi107.stdout:5637: seq_num 5636 ranges {0=802816} 2020-06-12T02:38:24.081 INFO:tasks.rados.rados.0.smithi107.stdout:5637: writing smithi10729407-1 from 0 to 802816 tid 1 2020-06-12T02:38:24.081 INFO:tasks.rados.rados.0.smithi107.stdout:5638: delete oid 5 current snap is 0 2020-06-12T02:38:24.082 INFO:tasks.rados.rados.0.smithi107.stdout:5633: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:24.082 INFO:tasks.rados.rados.0.smithi107.stdout:5633: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:24.082 INFO:tasks.rados.rados.0.smithi107.stdout:5633: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:24.083 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 993 (ObjNum 5632 snap 0 seq_num 5632) dirty exists 2020-06-12T02:38:24.083 INFO:tasks.rados.rados.0.smithi107.stdout:5633: left oid 19 (ObjNum 5632 snap 0 seq_num 5632) 2020-06-12T02:38:24.083 INFO:tasks.rados.rados.0.smithi107.stdout:5635: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:24.083 INFO:tasks.rados.rados.0.smithi107.stdout:5635: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:24.084 INFO:tasks.rados.rados.0.smithi107.stdout:5635: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:24.084 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1026 (ObjNum 5634 snap 0 seq_num 5634) dirty exists 2020-06-12T02:38:24.084 INFO:tasks.rados.rados.0.smithi107.stdout:5635: left oid 8 (ObjNum 5634 snap 0 seq_num 5634) 2020-06-12T02:38:24.086 INFO:tasks.rados.rados.0.smithi107.stdout:5636: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:24.087 INFO:tasks.rados.rados.0.smithi107.stdout:5633: done (5 left) 2020-06-12T02:38:24.087 INFO:tasks.rados.rados.0.smithi107.stdout:5635: done (4 left) 2020-06-12T02:38:24.087 INFO:tasks.rados.rados.0.smithi107.stdout:5638: done (3 left) 2020-06-12T02:38:24.087 INFO:tasks.rados.rados.0.smithi107.stdout:5639: delete oid 14 current snap is 0 2020-06-12T02:38:24.089 INFO:tasks.rados.rados.0.smithi107.stdout:5637: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:24.090 INFO:tasks.rados.rados.0.smithi107.stdout:5636: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:24.091 INFO:tasks.rados.rados.0.smithi107.stdout:5639: done (3 left) 2020-06-12T02:38:24.091 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:24.091 INFO:tasks.rados.rados.0.smithi107.stdout:5640: seq_num 5639 ranges {0=450560,450560=401408} 2020-06-12T02:38:24.092 INFO:tasks.rados.rados.0.smithi107.stdout:5640: writing smithi10729407-18 from 0 to 450560 tid 1 2020-06-12T02:38:24.095 INFO:tasks.rados.rados.0.smithi107.stdout:5640: writing smithi10729407-18 from 450560 to 851968 tid 2 2020-06-12T02:38:24.096 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:24.096 INFO:tasks.rados.rados.0.smithi107.stdout:5641: seq_num 5640 ranges {0=466944,466944=401408} 2020-06-12T02:38:24.098 INFO:tasks.rados.rados.0.smithi107.stdout:5641: writing smithi10729407-15 from 0 to 466944 tid 1 2020-06-12T02:38:24.100 INFO:tasks.rados.rados.0.smithi107.stdout:5641: writing smithi10729407-15 from 466944 to 868352 tid 2 2020-06-12T02:38:24.101 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:38:24.101 INFO:tasks.rados.rados.0.smithi107.stdout:5642: seq_num 5641 ranges {0=483328,483328=401408} 2020-06-12T02:38:24.103 INFO:tasks.rados.rados.0.smithi107.stdout:5642: writing smithi10729407-7 from 0 to 483328 tid 1 2020-06-12T02:38:24.104 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:38:24.101+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:38:24.106 INFO:tasks.rados.rados.0.smithi107.stdout:5642: writing smithi10729407-7 from 483328 to 884736 tid 2 2020-06-12T02:38:24.106 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:24.106 INFO:tasks.rados.rados.0.smithi107.stdout:5643: seq_num 5642 ranges {1458176=499712,1957888=401408} 2020-06-12T02:38:24.113 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:24.114 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:24.117 INFO:tasks.rados.rados.0.smithi107.stdout:5643: writing smithi10729407-8 from 1458176 to 1957888 tid 1 2020-06-12T02:38:24.120 INFO:tasks.rados.rados.0.smithi107.stdout:5643: writing smithi10729407-8 from 1957888 to 2359296 tid 2 2020-06-12T02:38:24.120 INFO:tasks.rados.rados.0.smithi107.stdout:5644: delete oid 4 current snap is 0 2020-06-12T02:38:24.120 INFO:tasks.rados.rados.0.smithi107.stdout:5636: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:24.120 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 995 (ObjNum 5635 snap 0 seq_num 5635) dirty exists 2020-06-12T02:38:24.121 INFO:tasks.rados.rados.0.smithi107.stdout:5636: left oid 12 (ObjNum 5635 snap 0 seq_num 5635) 2020-06-12T02:38:24.121 INFO:tasks.rados.rados.0.smithi107.stdout:5637: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:24.121 INFO:tasks.rados.rados.0.smithi107.stdout:5637: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:24.121 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1039 (ObjNum 5636 snap 0 seq_num 5636) dirty exists 2020-06-12T02:38:24.121 INFO:tasks.rados.rados.0.smithi107.stdout:5637: left oid 1 (ObjNum 5636 snap 0 seq_num 5636) 2020-06-12T02:38:24.121 INFO:tasks.rados.rados.0.smithi107.stdout:5640: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:24.122 INFO:tasks.rados.rados.0.smithi107.stdout:5640: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:24.122 INFO:tasks.rados.rados.0.smithi107.stdout:5640: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:24.122 INFO:tasks.rados.rados.0.smithi107.stdout:5640: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:24.122 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1025 (ObjNum 5639 snap 0 seq_num 5639) dirty exists 2020-06-12T02:38:24.122 INFO:tasks.rados.rados.0.smithi107.stdout:5640: left oid 18 (ObjNum 5639 snap 0 seq_num 5639) 2020-06-12T02:38:24.123 INFO:tasks.rados.rados.0.smithi107.stdout:5641: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:24.123 INFO:tasks.rados.rados.0.smithi107.stdout:5641: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:24.123 INFO:tasks.rados.rados.0.smithi107.stdout:5641: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:24.123 INFO:tasks.rados.rados.0.smithi107.stdout:5642: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:24.123 INFO:tasks.rados.rados.0.smithi107.stdout:5641: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:24.123 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1029 (ObjNum 5640 snap 0 seq_num 5640) dirty exists 2020-06-12T02:38:24.124 INFO:tasks.rados.rados.0.smithi107.stdout:5641: left oid 15 (ObjNum 5640 snap 0 seq_num 5640) 2020-06-12T02:38:24.124 INFO:tasks.rados.rados.0.smithi107.stdout:5642: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:24.124 INFO:tasks.rados.rados.0.smithi107.stdout:5642: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:24.125 INFO:tasks.rados.rados.0.smithi107.stdout:5642: finishing write tid 5 to smithi10729407-7 2020-06-12T02:38:24.125 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1075 (ObjNum 5641 snap 0 seq_num 5641) dirty exists 2020-06-12T02:38:24.125 INFO:tasks.rados.rados.0.smithi107.stdout:5642: left oid 7 (ObjNum 5641 snap 0 seq_num 5641) 2020-06-12T02:38:24.126 INFO:tasks.rados.rados.0.smithi107.stdout:5636: done (7 left) 2020-06-12T02:38:24.126 INFO:tasks.rados.rados.0.smithi107.stdout:5637: done (6 left) 2020-06-12T02:38:24.126 INFO:tasks.rados.rados.0.smithi107.stdout:5640: done (5 left) 2020-06-12T02:38:24.126 INFO:tasks.rados.rados.0.smithi107.stdout:5641: done (4 left) 2020-06-12T02:38:24.127 INFO:tasks.rados.rados.0.smithi107.stdout:5642: done (3 left) 2020-06-12T02:38:24.127 INFO:tasks.rados.rados.0.smithi107.stdout:5644: done (2 left) 2020-06-12T02:38:24.127 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:24.127 INFO:tasks.rados.rados.0.smithi107.stdout:5645: seq_num 5644 ranges {0=532480,532480=401408} 2020-06-12T02:38:24.129 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-06-12T02:38:24.129 INFO:tasks.rados.rados.0.smithi107.stdout:5645: writing smithi10729407-3 from 0 to 532480 tid 1 2020-06-12T02:38:24.130 INFO:tasks.rados.rados.0.smithi107.stdout:5645: writing smithi10729407-3 from 532480 to 933888 tid 2 2020-06-12T02:38:24.131 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:38:24.132 INFO:tasks.rados.rados.0.smithi107.stdout:5646: seq_num 5645 ranges {0=548864,548864=401408} 2020-06-12T02:38:24.133 INFO:tasks.rados.rados.0.smithi107.stdout:5646: writing smithi10729407-13 from 0 to 548864 tid 1 2020-06-12T02:38:24.136 INFO:tasks.rados.rados.0.smithi107.stdout:5646: writing smithi10729407-13 from 548864 to 950272 tid 2 2020-06-12T02:38:24.136 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:24.136 INFO:tasks.rados.rados.0.smithi107.stdout:5647: seq_num 5646 ranges {663552=565248,1228800=401408} 2020-06-12T02:38:24.143 INFO:tasks.rados.rados.0.smithi107.stdout:5647: writing smithi10729407-16 from 663552 to 1228800 tid 1 2020-06-12T02:38:24.146 INFO:tasks.rados.rados.0.smithi107.stdout:5647: writing smithi10729407-16 from 1228800 to 1630208 tid 2 2020-06-12T02:38:24.146 INFO:tasks.rados.rados.0.smithi107.stdout:5648: delete oid 20 current snap is 0 2020-06-12T02:38:24.146 INFO:tasks.rados.rados.0.smithi107.stdout:5643: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:24.147 INFO:tasks.rados.rados.0.smithi107.stdout:5643: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:24.147 INFO:tasks.rados.rados.0.smithi107.stdout:5643: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:24.147 INFO:tasks.rados.rados.0.smithi107.stdout:5643: finishing write tid 5 to smithi10729407-8 2020-06-12T02:38:24.147 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1032 (ObjNum 5642 snap 0 seq_num 5642) dirty exists 2020-06-12T02:38:24.147 INFO:tasks.rados.rados.0.smithi107.stdout:5643: left oid 8 (ObjNum 5642 snap 0 seq_num 5642) 2020-06-12T02:38:24.148 INFO:tasks.rados.rados.0.smithi107.stdout:5645: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:24.148 INFO:tasks.rados.rados.0.smithi107.stdout:5646: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:24.148 INFO:tasks.rados.rados.0.smithi107.stdout:5645: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:24.148 INFO:tasks.rados.rados.0.smithi107.stdout:5646: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:24.148 INFO:tasks.rados.rados.0.smithi107.stdout:5645: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:24.149 INFO:tasks.rados.rados.0.smithi107.stdout:5646: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:24.149 INFO:tasks.rados.rados.0.smithi107.stdout:5646: finishing write tid 5 to smithi10729407-13 2020-06-12T02:38:24.149 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1078 (ObjNum 5645 snap 0 seq_num 5645) dirty exists 2020-06-12T02:38:24.149 INFO:tasks.rados.rados.0.smithi107.stdout:5646: left oid 13 (ObjNum 5645 snap 0 seq_num 5645) 2020-06-12T02:38:24.150 INFO:tasks.rados.rados.0.smithi107.stdout:5643: done (5 left) 2020-06-12T02:38:24.150 INFO:tasks.rados.rados.0.smithi107.stdout:5646: done (4 left) 2020-06-12T02:38:24.150 INFO:tasks.rados.rados.0.smithi107.stdout:5648: done (3 left) 2020-06-12T02:38:24.150 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:24.150 INFO:tasks.rados.rados.0.smithi107.stdout:5649: seq_num 5648 ranges {1073152=598016,1671168=401408} 2020-06-12T02:38:24.158 INFO:tasks.rados.rados.0.smithi107.stdout:5649: writing smithi10729407-12 from 1073152 to 1671168 tid 1 2020-06-12T02:38:24.161 INFO:tasks.rados.rados.0.smithi107.stdout:5649: writing smithi10729407-12 from 1671168 to 2072576 tid 2 2020-06-12T02:38:24.162 INFO:tasks.rados.rados.0.smithi107.stdout:5650: delete oid 17 current snap is 0 2020-06-12T02:38:24.162 INFO:tasks.rados.rados.0.smithi107.stdout:5645: finishing write tid 5 to smithi10729407-3 2020-06-12T02:38:24.162 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1010 (ObjNum 5644 snap 0 seq_num 5644) dirty exists 2020-06-12T02:38:24.162 INFO:tasks.rados.rados.0.smithi107.stdout:5645: left oid 3 (ObjNum 5644 snap 0 seq_num 5644) 2020-06-12T02:38:24.162 INFO:tasks.rados.rados.0.smithi107.stdout:5647: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:24.163 INFO:tasks.rados.rados.0.smithi107.stdout:5647: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:24.163 INFO:tasks.rados.rados.0.smithi107.stdout:5647: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:24.163 INFO:tasks.rados.rados.0.smithi107.stdout:5647: finishing write tid 5 to smithi10729407-16 2020-06-12T02:38:24.163 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1013 (ObjNum 5646 snap 0 seq_num 5646) dirty exists 2020-06-12T02:38:24.164 INFO:tasks.rados.rados.0.smithi107.stdout:5647: left oid 16 (ObjNum 5646 snap 0 seq_num 5646) 2020-06-12T02:38:24.164 INFO:tasks.rados.rados.0.smithi107.stdout:5645: done (4 left) 2020-06-12T02:38:24.164 INFO:tasks.rados.rados.0.smithi107.stdout:5647: done (3 left) 2020-06-12T02:38:24.165 INFO:tasks.rados.rados.0.smithi107.stdout:5650: done (2 left) 2020-06-12T02:38:24.165 INFO:tasks.rados.rados.0.smithi107.stdout:5651: delete oid 4 current snap is 0 2020-06-12T02:38:24.165 INFO:tasks.rados.rados.0.smithi107.stdout:5649: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:24.174 INFO:tasks.rados.rados.0.smithi107.stdout:5651: done (2 left) 2020-06-12T02:38:24.174 INFO:tasks.rados.rados.0.smithi107.stdout:5652: delete oid 19 current snap is 0 2020-06-12T02:38:24.174 INFO:tasks.rados.rados.0.smithi107.stdout:5649: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:24.175 INFO:tasks.rados.rados.0.smithi107.stdout:5649: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:24.175 INFO:tasks.rados.rados.0.smithi107.stdout:5649: finishing write tid 5 to smithi10729407-12 2020-06-12T02:38:24.175 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 998 (ObjNum 5648 snap 0 seq_num 5648) dirty exists 2020-06-12T02:38:24.175 INFO:tasks.rados.rados.0.smithi107.stdout:5649: left oid 12 (ObjNum 5648 snap 0 seq_num 5648) 2020-06-12T02:38:24.175 INFO:tasks.rados.rados.0.smithi107.stdout:5649: done (2 left) 2020-06-12T02:38:24.175 INFO:tasks.rados.rados.0.smithi107.stdout:5652: done (1 left) 2020-06-12T02:38:24.176 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:24.176 INFO:tasks.rados.rados.0.smithi107.stdout:5653: seq_num 5652 ranges {0=663552,663552=401408} 2020-06-12T02:38:24.177 INFO:tasks.rados.rados.0.smithi107.stdout:5653: writing smithi10729407-17 from 0 to 663552 tid 1 2020-06-12T02:38:24.179 INFO:tasks.rados.rados.0.smithi107.stdout:5653: writing smithi10729407-17 from 663552 to 1064960 tid 2 2020-06-12T02:38:24.179 INFO:tasks.rados.rados.0.smithi107.stdout:5654: delete oid 13 current snap is 0 2020-06-12T02:38:24.183 INFO:tasks.rados.rados.0.smithi107.stdout:5654: done (2 left) 2020-06-12T02:38:24.183 INFO:tasks.rados.rados.0.smithi107.stdout:5655: delete oid 3 current snap is 0 2020-06-12T02:38:24.187 INFO:tasks.rados.rados.0.smithi107.stdout:5653: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:24.187 INFO:tasks.rados.rados.0.smithi107.stdout:5655: done (2 left) 2020-06-12T02:38:24.188 INFO:tasks.rados.rados.0.smithi107.stdout:5656: delete oid 8 current snap is 0 2020-06-12T02:38:24.189 INFO:tasks.rados.rados.0.smithi107.stdout:5653: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:24.191 INFO:tasks.rados.rados.0.smithi107.stdout:5653: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:24.193 INFO:tasks.rados.rados.0.smithi107.stdout:5656: done (2 left) 2020-06-12T02:38:24.193 INFO:tasks.rados.rados.0.smithi107.stdout:5657: delete oid 13 current snap is 0 2020-06-12T02:38:24.193 INFO:tasks.rados.rados.0.smithi107.stdout:5653: finishing write tid 5 to smithi10729407-17 2020-06-12T02:38:24.194 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1055 (ObjNum 5652 snap 0 seq_num 5652) dirty exists 2020-06-12T02:38:24.194 INFO:tasks.rados.rados.0.smithi107.stdout:5653: left oid 17 (ObjNum 5652 snap 0 seq_num 5652) 2020-06-12T02:38:24.196 INFO:tasks.rados.rados.0.smithi107.stdout:5653: done (2 left) 2020-06-12T02:38:24.196 INFO:tasks.rados.rados.0.smithi107.stdout:5657: done (1 left) 2020-06-12T02:38:24.196 INFO:tasks.rados.rados.0.smithi107.stdout:5658: delete oid 5 current snap is 0 2020-06-12T02:38:24.198 INFO:tasks.rados.rados.0.smithi107.stdout:5658: done (1 left) 2020-06-12T02:38:24.198 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:38:24.198 INFO:tasks.rados.rados.0.smithi107.stdout:5659: seq_num 5658 ranges {1630208=770048,2400256=401408} 2020-06-12T02:38:24.211 INFO:tasks.rados.rados.0.smithi107.stdout:5659: writing smithi10729407-16 from 1630208 to 2400256 tid 1 2020-06-12T02:38:24.213 INFO:tasks.rados.rados.0.smithi107.stdout:5659: writing smithi10729407-16 from 2400256 to 2801664 tid 2 2020-06-12T02:38:24.214 INFO:tasks.rados.rados.0.smithi107.stdout:5660: delete oid 15 current snap is 0 2020-06-12T02:38:24.215 INFO:tasks.rados.rados.0.smithi107.stdout:5660: done (2 left) 2020-06-12T02:38:24.216 INFO:tasks.rados.rados.0.smithi107.stdout:5661: delete oid 5 current snap is 0 2020-06-12T02:38:24.219 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:24.216+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:24.221 INFO:tasks.rados.rados.0.smithi107.stdout:5661: done (2 left) 2020-06-12T02:38:24.221 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:24.221 INFO:tasks.rados.rados.0.smithi107.stdout:5662: seq_num 5661 ranges {0=417792,417792=573440,991232=229376} 2020-06-12T02:38:24.223 INFO:tasks.rados.rados.0.smithi107.stdout:5662: writing smithi10729407-5 from 0 to 417792 tid 1 2020-06-12T02:38:24.229 INFO:tasks.rados.rados.0.smithi107.stdout:5662: writing smithi10729407-5 from 417792 to 991232 tid 2 2020-06-12T02:38:24.229 INFO:tasks.rados.rados.0.smithi107.stdout:5662: writing smithi10729407-5 from 991232 to 1220608 tid 3 2020-06-12T02:38:24.229 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:24.229 INFO:tasks.rados.rados.0.smithi107.stdout:5663: seq_num 5662 ranges {0=434176,434176=458752,892928=344064} 2020-06-12T02:38:24.230 INFO:tasks.rados.rados.0.smithi107.stdout:5663: writing smithi10729407-8 from 0 to 434176 tid 1 2020-06-12T02:38:24.233 INFO:tasks.rados.rados.0.smithi107.stdout:5663: writing smithi10729407-8 from 434176 to 892928 tid 2 2020-06-12T02:38:24.235 INFO:tasks.rados.rados.0.smithi107.stdout:5663: writing smithi10729407-8 from 892928 to 1236992 tid 3 2020-06-12T02:38:24.236 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:24.236 INFO:tasks.rados.rados.0.smithi107.stdout:5664: seq_num 5663 ranges {0=450560,450560=737280,1187840=65536} 2020-06-12T02:38:24.238 INFO:tasks.rados.rados.0.smithi107.stdout:5664: writing smithi10729407-6 from 0 to 450560 tid 1 2020-06-12T02:38:24.242 INFO:tasks.rados.rados.0.smithi107.stdout:5664: writing smithi10729407-6 from 450560 to 1187840 tid 2 2020-06-12T02:38:24.243 INFO:tasks.rados.rados.0.smithi107.stdout:5664: writing smithi10729407-6 from 1187840 to 1253376 tid 3 2020-06-12T02:38:24.243 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:24.243 INFO:tasks.rados.rados.0.smithi107.stdout:5665: seq_num 5664 ranges {3072000=466944,3538944=671744,4210688=131072} 2020-06-12T02:38:24.274 INFO:tasks.rados.rados.0.smithi107.stdout:5665: writing smithi10729407-9 from 3072000 to 3538944 tid 1 2020-06-12T02:38:24.274 INFO:tasks.rados.rados.0.smithi107.stdout:5665: writing smithi10729407-9 from 3538944 to 4210688 tid 2 2020-06-12T02:38:24.274 INFO:tasks.rados.rados.0.smithi107.stdout:5665: writing smithi10729407-9 from 4210688 to 4341760 tid 3 2020-06-12T02:38:24.274 INFO:tasks.rados.rados.0.smithi107.stdout:5666: delete oid 1 current snap is 0 2020-06-12T02:38:24.275 INFO:tasks.rados.rados.0.smithi107.stdout:5659: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:24.275 INFO:tasks.rados.rados.0.smithi107.stdout:5659: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:24.275 INFO:tasks.rados.rados.0.smithi107.stdout:5659: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:24.275 INFO:tasks.rados.rados.0.smithi107.stdout:5659: finishing write tid 5 to smithi10729407-16 2020-06-12T02:38:24.275 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1017 (ObjNum 5658 snap 0 seq_num 5658) dirty exists 2020-06-12T02:38:24.276 INFO:tasks.rados.rados.0.smithi107.stdout:5659: left oid 16 (ObjNum 5658 snap 0 seq_num 5658) 2020-06-12T02:38:24.276 INFO:tasks.rados.rados.0.smithi107.stdout:5662: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:24.276 INFO:tasks.rados.rados.0.smithi107.stdout:5662: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:24.276 INFO:tasks.rados.rados.0.smithi107.stdout:5663: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:24.276 INFO:tasks.rados.rados.0.smithi107.stdout:5662: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:24.276 INFO:tasks.rados.rados.0.smithi107.stdout:5662: finishing write tid 5 to smithi10729407-5 2020-06-12T02:38:24.277 INFO:tasks.rados.rados.0.smithi107.stdout:5662: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:24.277 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 563 (ObjNum 5661 snap 0 seq_num 5661) dirty exists 2020-06-12T02:38:24.277 INFO:tasks.rados.rados.0.smithi107.stdout:5662: left oid 5 (ObjNum 5661 snap 0 seq_num 5661) 2020-06-12T02:38:24.277 INFO:tasks.rados.rados.0.smithi107.stdout:5663: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:24.277 INFO:tasks.rados.rados.0.smithi107.stdout:5664: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:24.278 INFO:tasks.rados.rados.0.smithi107.stdout:5663: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:24.278 INFO:tasks.rados.rados.0.smithi107.stdout:5663: finishing write tid 5 to smithi10729407-8 2020-06-12T02:38:24.278 INFO:tasks.rados.rados.0.smithi107.stdout:5663: finishing write tid 6 to smithi10729407-8 2020-06-12T02:38:24.278 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1038 (ObjNum 5662 snap 0 seq_num 5662) dirty exists 2020-06-12T02:38:24.278 INFO:tasks.rados.rados.0.smithi107.stdout:5663: left oid 8 (ObjNum 5662 snap 0 seq_num 5662) 2020-06-12T02:38:24.278 INFO:tasks.rados.rados.0.smithi107.stdout:5664: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:24.279 INFO:tasks.rados.rados.0.smithi107.stdout:5664: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:24.279 INFO:tasks.rados.rados.0.smithi107.stdout:5664: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:24.279 INFO:tasks.rados.rados.0.smithi107.stdout:5664: finishing write tid 6 to smithi10729407-6 2020-06-12T02:38:24.279 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1043 (ObjNum 5663 snap 0 seq_num 5663) dirty exists 2020-06-12T02:38:24.279 INFO:tasks.rados.rados.0.smithi107.stdout:5664: left oid 6 (ObjNum 5663 snap 0 seq_num 5663) 2020-06-12T02:38:24.280 INFO:tasks.rados.rados.0.smithi107.stdout:5659: done (6 left) 2020-06-12T02:38:24.280 INFO:tasks.rados.rados.0.smithi107.stdout:5662: done (5 left) 2020-06-12T02:38:24.280 INFO:tasks.rados.rados.0.smithi107.stdout:5663: done (4 left) 2020-06-12T02:38:24.280 INFO:tasks.rados.rados.0.smithi107.stdout:5664: done (3 left) 2020-06-12T02:38:24.280 INFO:tasks.rados.rados.0.smithi107.stdout:5666: done (2 left) 2020-06-12T02:38:24.280 INFO:tasks.rados.rados.0.smithi107.stdout:5667: delete oid 15 current snap is 0 2020-06-12T02:38:24.281 INFO:tasks.rados.rados.0.smithi107.stdout:5665: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:24.282 INFO:tasks.rados.rados.0.smithi107.stdout:5665: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:24.282 INFO:tasks.rados.rados.0.smithi107.stdout:5667: done (2 left) 2020-06-12T02:38:24.282 INFO:tasks.rados.rados.0.smithi107.stdout:5668: delete oid 5 current snap is 0 2020-06-12T02:38:24.282 INFO:tasks.rados.rados.0.smithi107.stdout:5665: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:24.283 INFO:tasks.rados.rados.0.smithi107.stdout:5665: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:24.283 INFO:tasks.rados.rados.0.smithi107.stdout:5665: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:24.283 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1029 (ObjNum 5664 snap 0 seq_num 5664) dirty exists 2020-06-12T02:38:24.283 INFO:tasks.rados.rados.0.smithi107.stdout:5665: left oid 9 (ObjNum 5664 snap 0 seq_num 5664) 2020-06-12T02:38:24.283 INFO:tasks.rados.rados.0.smithi107.stdout:5665: done (2 left) 2020-06-12T02:38:24.284 INFO:tasks.rados.rados.0.smithi107.stdout:5668: done (1 left) 2020-06-12T02:38:24.284 INFO:tasks.rados.rados.0.smithi107.stdout:5669: delete oid 7 current snap is 0 2020-06-12T02:38:24.284 INFO:tasks.rados.rados.0.smithi107.stdout:5669: done (1 left) 2020-06-12T02:38:24.284 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:38:24.284 INFO:tasks.rados.rados.0.smithi107.stdout:5670: seq_num 5669 ranges {0=548864,548864=491520,1040384=311296} 2020-06-12T02:38:24.285 INFO:tasks.rados.rados.0.smithi107.stdout:5670: writing smithi10729407-7 from 0 to 548864 tid 1 2020-06-12T02:38:24.285 INFO:tasks.rados.rados.0.smithi107.stdout:5670: writing smithi10729407-7 from 548864 to 1040384 tid 2 2020-06-12T02:38:24.287 INFO:tasks.rados.rados.0.smithi107.stdout:5670: writing smithi10729407-7 from 1040384 to 1351680 tid 3 2020-06-12T02:38:24.287 INFO:tasks.rados.rados.0.smithi107.stdout:5671: delete oid 9 current snap is 0 2020-06-12T02:38:24.288 INFO:tasks.rados.rados.0.smithi107.stdout:5670: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:24.293 INFO:tasks.rados.rados.0.smithi107.stdout:5671: done (2 left) 2020-06-12T02:38:24.293 INFO:tasks.rados.rados.0.smithi107.stdout:5672: delete oid 16 current snap is 0 2020-06-12T02:38:24.295 INFO:tasks.rados.rados.0.smithi107.stdout:5670: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:24.296 INFO:tasks.rados.rados.0.smithi107.stdout:5670: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:24.296 INFO:tasks.rados.rados.0.smithi107.stdout:5672: done (2 left) 2020-06-12T02:38:24.297 INFO:tasks.rados.rados.0.smithi107.stdout:5673: delete oid 12 current snap is 0 2020-06-12T02:38:24.299 INFO:tasks.rados.rados.0.smithi107.stdout:5670: finishing write tid 5 to smithi10729407-7 2020-06-12T02:38:24.300 INFO:tasks.rados.rados.0.smithi107.stdout:5673: done (2 left) 2020-06-12T02:38:24.301 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:24.301 INFO:tasks.rados.rados.0.smithi107.stdout:5674: seq_num 5673 ranges {0=614400,614400=483328,1097728=319488} 2020-06-12T02:38:24.304 INFO:tasks.rados.rados.0.smithi107.stdout:5674: writing smithi10729407-5 from 0 to 614400 tid 1 2020-06-12T02:38:24.307 INFO:tasks.rados.rados.0.smithi107.stdout:5674: writing smithi10729407-5 from 614400 to 1097728 tid 2 2020-06-12T02:38:24.309 INFO:tasks.rados.rados.0.smithi107.stdout:5674: writing smithi10729407-5 from 1097728 to 1417216 tid 3 2020-06-12T02:38:24.309 INFO:tasks.rados.rados.0.smithi107.stdout:5675: delete oid 16 current snap is 0 2020-06-12T02:38:24.309 INFO:tasks.rados.rados.0.smithi107.stdout:5670: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:24.310 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1085 (ObjNum 5669 snap 0 seq_num 5669) dirty exists 2020-06-12T02:38:24.310 INFO:tasks.rados.rados.0.smithi107.stdout:5670: left oid 7 (ObjNum 5669 snap 0 seq_num 5669) 2020-06-12T02:38:24.310 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:24.307+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:24.311 INFO:tasks.rados.rados.0.smithi107.stdout:5670: done (3 left) 2020-06-12T02:38:24.311 INFO:tasks.rados.rados.0.smithi107.stdout:5675: done (2 left) 2020-06-12T02:38:24.311 INFO:tasks.rados.rados.0.smithi107.stdout:5676: delete oid 20 current snap is 0 2020-06-12T02:38:24.313 INFO:tasks.rados.rados.0.smithi107.stdout:5676: done (2 left) 2020-06-12T02:38:24.314 INFO:tasks.rados.rados.0.smithi107.stdout:5677: delete oid 4 current snap is 0 2020-06-12T02:38:24.315 INFO:tasks.rados.rados.0.smithi107.stdout:5674: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:24.315 INFO:tasks.rados.rados.0.smithi107.stdout:5674: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:24.316 INFO:tasks.rados.rados.0.smithi107.stdout:5674: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:24.316 INFO:tasks.rados.rados.0.smithi107.stdout:5674: finishing write tid 5 to smithi10729407-5 2020-06-12T02:38:24.316 INFO:tasks.rados.rados.0.smithi107.stdout:5677: done (2 left) 2020-06-12T02:38:24.317 INFO:tasks.rados.rados.0.smithi107.stdout:5678: delete oid 12 current snap is 0 2020-06-12T02:38:24.318 INFO:tasks.rados.rados.0.smithi107.stdout:5674: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:24.318 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 568 (ObjNum 5673 snap 0 seq_num 5673) dirty exists 2020-06-12T02:38:24.318 INFO:tasks.rados.rados.0.smithi107.stdout:5674: left oid 5 (ObjNum 5673 snap 0 seq_num 5673) 2020-06-12T02:38:24.318 INFO:tasks.rados.rados.0.smithi107.stdout:5674: done (2 left) 2020-06-12T02:38:24.319 INFO:tasks.rados.rados.0.smithi107.stdout:5678: done (1 left) 2020-06-12T02:38:24.319 INFO:tasks.rados.rados.0.smithi107.stdout:5679: delete oid 12 current snap is 0 2020-06-12T02:38:24.320 INFO:tasks.rados.rados.0.smithi107.stdout:5679: done (1 left) 2020-06-12T02:38:24.320 INFO:tasks.rados.rados.0.smithi107.stdout:5680: delete oid 11 current snap is 0 2020-06-12T02:38:24.322 INFO:tasks.rados.rados.0.smithi107.stdout:5680: done (1 left) 2020-06-12T02:38:24.322 INFO:tasks.rados.rados.0.smithi107.stdout:5681: delete oid 13 current snap is 0 2020-06-12T02:38:24.324 INFO:tasks.rados.rados.0.smithi107.stdout:5681: done (1 left) 2020-06-12T02:38:24.325 INFO:tasks.rados.rados.0.smithi107.stdout:5682: delete oid 14 current snap is 0 2020-06-12T02:38:24.326 INFO:tasks.rados.rados.0.smithi107.stdout:5682: done (1 left) 2020-06-12T02:38:24.326 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:24.327 INFO:tasks.rados.rados.0.smithi107.stdout:5683: seq_num 5682 ranges {0=770048,770048=688128,1458176=114688} 2020-06-12T02:38:24.331 INFO:tasks.rados.rados.0.smithi107.stdout:5683: writing smithi10729407-12 from 0 to 770048 tid 1 2020-06-12T02:38:24.332 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:24.333 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:24.335 INFO:tasks.rados.rados.0.smithi107.stdout:5683: writing smithi10729407-12 from 770048 to 1458176 tid 2 2020-06-12T02:38:24.335 INFO:tasks.rados.rados.0.smithi107.stdout:5683: writing smithi10729407-12 from 1458176 to 1572864 tid 3 2020-06-12T02:38:24.336 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:38:24.336 INFO:tasks.rados.rados.0.smithi107.stdout:5684: seq_num 5683 ranges {0=786432,786432=573440,1359872=229376} 2020-06-12T02:38:24.339 INFO:tasks.rados.rados.0.smithi107.stdout:5684: writing smithi10729407-1 from 0 to 786432 tid 1 2020-06-12T02:38:24.342 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-06-12T02:38:24.343 INFO:tasks.rados.rados.0.smithi107.stdout:5684: writing smithi10729407-1 from 786432 to 1359872 tid 2 2020-06-12T02:38:24.344 INFO:tasks.rados.rados.0.smithi107.stdout:5684: writing smithi10729407-1 from 1359872 to 1589248 tid 3 2020-06-12T02:38:24.345 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:24.345 INFO:tasks.rados.rados.0.smithi107.stdout:5685: seq_num 5684 ranges {0=802816,802816=458752,1261568=344064} 2020-06-12T02:38:24.349 INFO:tasks.rados.rados.0.smithi107.stdout:5685: writing smithi10729407-19 from 0 to 802816 tid 1 2020-06-12T02:38:24.352 INFO:tasks.rados.rados.0.smithi107.stdout:5685: writing smithi10729407-19 from 802816 to 1261568 tid 2 2020-06-12T02:38:24.354 INFO:tasks.rados.rados.0.smithi107.stdout:5685: writing smithi10729407-19 from 1261568 to 1605632 tid 3 2020-06-12T02:38:24.355 INFO:tasks.rados.rados.0.smithi107.stdout:5686: delete oid 18 current snap is 0 2020-06-12T02:38:24.355 INFO:tasks.rados.rados.0.smithi107.stdout:5683: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:24.355 INFO:tasks.rados.rados.0.smithi107.stdout:5683: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:24.355 INFO:tasks.rados.rados.0.smithi107.stdout:5683: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:24.356 INFO:tasks.rados.rados.0.smithi107.stdout:5683: finishing write tid 5 to smithi10729407-12 2020-06-12T02:38:24.356 INFO:tasks.rados.rados.0.smithi107.stdout:5683: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:24.356 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1006 (ObjNum 5682 snap 0 seq_num 5682) dirty exists 2020-06-12T02:38:24.356 INFO:tasks.rados.rados.0.smithi107.stdout:5683: left oid 12 (ObjNum 5682 snap 0 seq_num 5682) 2020-06-12T02:38:24.356 INFO:tasks.rados.rados.0.smithi107.stdout:5684: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:24.356 INFO:tasks.rados.rados.0.smithi107.stdout:5684: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:24.357 INFO:tasks.rados.rados.0.smithi107.stdout:5684: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:24.358 INFO:tasks.rados.rados.0.smithi107.stdout:5683: done (4 left) 2020-06-12T02:38:24.358 INFO:tasks.rados.rados.0.smithi107.stdout:5686: done (3 left) 2020-06-12T02:38:24.358 INFO:tasks.rados.rados.0.smithi107.stdout:5687: delete oid 16 current snap is 0 2020-06-12T02:38:24.358 INFO:tasks.rados.rados.0.smithi107.stdout:5684: finishing write tid 5 to smithi10729407-1 2020-06-12T02:38:24.360 INFO:tasks.rados.rados.0.smithi107.stdout:5684: finishing write tid 6 to smithi10729407-1 2020-06-12T02:38:24.360 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1048 (ObjNum 5683 snap 0 seq_num 5683) dirty exists 2020-06-12T02:38:24.360 INFO:tasks.rados.rados.0.smithi107.stdout:5684: left oid 1 (ObjNum 5683 snap 0 seq_num 5683) 2020-06-12T02:38:24.361 INFO:tasks.rados.rados.0.smithi107.stdout:5685: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:24.363 INFO:tasks.rados.rados.0.smithi107.stdout:5684: done (3 left) 2020-06-12T02:38:24.363 INFO:tasks.rados.rados.0.smithi107.stdout:5687: done (2 left) 2020-06-12T02:38:24.364 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:24.364 INFO:tasks.rados.rados.0.smithi107.stdout:5688: seq_num 5687 ranges {0=450560,450560=565248,1015808=638976} 2020-06-12T02:38:24.365 INFO:tasks.rados.rados.0.smithi107.stdout:5688: writing smithi10729407-2 from 0 to 450560 tid 1 2020-06-12T02:38:24.369 INFO:tasks.rados.rados.0.smithi107.stdout:5688: writing smithi10729407-2 from 450560 to 1015808 tid 2 2020-06-12T02:38:24.374 INFO:tasks.rados.rados.0.smithi107.stdout:5688: writing smithi10729407-2 from 1015808 to 1654784 tid 3 2020-06-12T02:38:24.374 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:24.374 INFO:tasks.rados.rados.0.smithi107.stdout:5689: seq_num 5688 ranges {0=466944,466944=442368,909312=540672,1449984=221184} 2020-06-12T02:38:24.376 INFO:tasks.rados.rados.0.smithi107.stdout:5689: writing smithi10729407-18 from 0 to 466944 tid 1 2020-06-12T02:38:24.379 INFO:tasks.rados.rados.0.smithi107.stdout:5689: writing smithi10729407-18 from 466944 to 909312 tid 2 2020-06-12T02:38:24.382 INFO:tasks.rados.rados.0.smithi107.stdout:5689: writing smithi10729407-18 from 909312 to 1449984 tid 3 2020-06-12T02:38:24.384 INFO:tasks.rados.rados.0.smithi107.stdout:5689: writing smithi10729407-18 from 1449984 to 1671168 tid 4 2020-06-12T02:38:24.385 INFO:tasks.rados.rados.0.smithi107.stdout:5690: delete oid 17 current snap is 0 2020-06-12T02:38:24.385 INFO:tasks.rados.rados.0.smithi107.stdout:5685: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:24.385 INFO:tasks.rados.rados.0.smithi107.stdout:5685: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:24.385 INFO:tasks.rados.rados.0.smithi107.stdout:5685: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:24.385 INFO:tasks.rados.rados.0.smithi107.stdout:5685: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:24.386 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1010 (ObjNum 5684 snap 0 seq_num 5684) dirty exists 2020-06-12T02:38:24.386 INFO:tasks.rados.rados.0.smithi107.stdout:5685: left oid 19 (ObjNum 5684 snap 0 seq_num 5684) 2020-06-12T02:38:24.386 INFO:tasks.rados.rados.0.smithi107.stdout:5688: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:24.386 INFO:tasks.rados.rados.0.smithi107.stdout:5688: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:24.386 INFO:tasks.rados.rados.0.smithi107.stdout:5688: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:24.387 INFO:tasks.rados.rados.0.smithi107.stdout:5688: finishing write tid 5 to smithi10729407-2 2020-06-12T02:38:24.387 INFO:tasks.rados.rados.0.smithi107.stdout:5688: finishing write tid 6 to smithi10729407-2 2020-06-12T02:38:24.387 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 536 (ObjNum 5687 snap 0 seq_num 5687) dirty exists 2020-06-12T02:38:24.387 INFO:tasks.rados.rados.0.smithi107.stdout:5688: left oid 2 (ObjNum 5687 snap 0 seq_num 5687) 2020-06-12T02:38:24.388 INFO:tasks.rados.rados.0.smithi107.stdout:5689: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:24.389 INFO:tasks.rados.rados.0.smithi107.stdout:5689: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:24.392 INFO:tasks.rados.rados.0.smithi107.stdout:5689: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:24.392 INFO:tasks.rados.rados.0.smithi107.stdout:5685: done (4 left) 2020-06-12T02:38:24.392 INFO:tasks.rados.rados.0.smithi107.stdout:5688: done (3 left) 2020-06-12T02:38:24.392 INFO:tasks.rados.rados.0.smithi107.stdout:5690: done (2 left) 2020-06-12T02:38:24.392 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:38:24.393 INFO:tasks.rados.rados.0.smithi107.stdout:5691: seq_num 5690 ranges {1351680=499712,1851392=614400,2465792=548864,3014656=40960} 2020-06-12T02:38:24.402 INFO:tasks.rados.rados.0.smithi107.stdout:5691: writing smithi10729407-7 from 1351680 to 1851392 tid 1 2020-06-12T02:38:24.413 INFO:tasks.rados.rados.0.smithi107.stdout:5691: writing smithi10729407-7 from 1851392 to 2465792 tid 2 2020-06-12T02:38:24.414 INFO:tasks.rados.rados.0.smithi107.stdout:5691: writing smithi10729407-7 from 2465792 to 3014656 tid 3 2020-06-12T02:38:24.414 INFO:tasks.rados.rados.0.smithi107.stdout:5691: writing smithi10729407-7 from 3014656 to 3055616 tid 4 2020-06-12T02:38:24.414 INFO:tasks.rados.rados.0.smithi107.stdout:5692: delete oid 1 current snap is 0 2020-06-12T02:38:24.415 INFO:tasks.rados.rados.0.smithi107.stdout:5689: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:24.415 INFO:tasks.rados.rados.0.smithi107.stdout:5689: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:24.415 INFO:tasks.rados.rados.0.smithi107.stdout:5689: finishing write tid 7 to smithi10729407-18 2020-06-12T02:38:24.415 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1036 (ObjNum 5688 snap 0 seq_num 5688) dirty exists 2020-06-12T02:38:24.416 INFO:tasks.rados.rados.0.smithi107.stdout:5689: left oid 18 (ObjNum 5688 snap 0 seq_num 5688) 2020-06-12T02:38:24.416 INFO:tasks.rados.rados.0.smithi107.stdout:5691: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:24.416 INFO:teuthology.orchestra.run.smithi107.stderr:set pool 1 pgp_num to 1 2020-06-12T02:38:24.417 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2020-06-12T02:38:24.417 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:24.410+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:24.418 INFO:tasks.rados.rados.0.smithi107.stdout:5689: done (3 left) 2020-06-12T02:38:24.418 INFO:tasks.rados.rados.0.smithi107.stdout:5692: done (2 left) 2020-06-12T02:38:24.418 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:38:24.418 INFO:tasks.rados.rados.0.smithi107.stdout:5693: seq_num 5692 ranges {0=532480,532480=786432,1318912=417792} 2020-06-12T02:38:24.419 INFO:tasks.rados.rados.0.smithi107.stdout:5693: writing smithi10729407-3 from 0 to 532480 tid 1 2020-06-12T02:38:24.424 INFO:tasks.rados.rados.0.smithi107.stdout:5693: writing smithi10729407-3 from 532480 to 1318912 tid 2 2020-06-12T02:38:24.427 INFO:tasks.rados.rados.0.smithi107.stdout:5693: writing smithi10729407-3 from 1318912 to 1736704 tid 3 2020-06-12T02:38:24.427 INFO:tasks.rados.rados.0.smithi107.stdout:5694: delete oid 11 current snap is 0 2020-06-12T02:38:24.428 INFO:tasks.rados.rados.0.smithi107.stdout:5691: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:24.428 INFO:tasks.rados.rados.0.smithi107.stdout:5691: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:24.428 INFO:tasks.rados.rados.0.smithi107.stdout:5691: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:24.428 INFO:tasks.rados.rados.0.smithi107.stdout:5691: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:24.428 INFO:tasks.rados.rados.0.smithi107.stdout:5691: finishing write tid 7 to smithi10729407-7 2020-06-12T02:38:24.429 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1091 (ObjNum 5690 snap 0 seq_num 5690) dirty exists 2020-06-12T02:38:24.429 INFO:tasks.rados.rados.0.smithi107.stdout:5691: left oid 7 (ObjNum 5690 snap 0 seq_num 5690) 2020-06-12T02:38:24.430 INFO:tasks.rados.rados.0.smithi107.stdout:5693: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:24.430 INFO:tasks.rados.rados.0.smithi107.stdout:5691: done (3 left) 2020-06-12T02:38:24.431 INFO:tasks.rados.rados.0.smithi107.stdout:5694: done (2 left) 2020-06-12T02:38:24.431 INFO:tasks.rados.rados.0.smithi107.stdout:5695: delete oid 13 current snap is 0 2020-06-12T02:38:24.433 INFO:tasks.rados.rados.0.smithi107.stdout:5695: done (2 left) 2020-06-12T02:38:24.433 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:24.433 INFO:tasks.rados.rados.0.smithi107.stdout:5696: seq_num 5695 ranges {1572864=589824,2162688=483328,2646016=720896} 2020-06-12T02:38:24.444 INFO:tasks.rados.rados.0.smithi107.stdout:5696: writing smithi10729407-12 from 1572864 to 2162688 tid 1 2020-06-12T02:38:24.447 INFO:tasks.rados.rados.0.smithi107.stdout:5696: writing smithi10729407-12 from 2162688 to 2646016 tid 2 2020-06-12T02:38:24.452 INFO:tasks.rados.rados.0.smithi107.stdout:5696: writing smithi10729407-12 from 2646016 to 3366912 tid 3 2020-06-12T02:38:24.452 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:24.452 INFO:tasks.rados.rados.0.smithi107.stdout:5697: seq_num 5696 ranges {0=606208,606208=770048,1376256=434176} 2020-06-12T02:38:24.461 INFO:tasks.rados.rados.0.smithi107.stdout:5697: writing smithi10729407-16 from 0 to 606208 tid 1 2020-06-12T02:38:24.462 INFO:tasks.rados.rados.0.smithi107.stdout:5697: writing smithi10729407-16 from 606208 to 1376256 tid 2 2020-06-12T02:38:24.463 INFO:tasks.rados.rados.0.smithi107.stdout:5697: writing smithi10729407-16 from 1376256 to 1810432 tid 3 2020-06-12T02:38:24.463 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:24.463 INFO:tasks.rados.rados.0.smithi107.stdout:5698: seq_num 5697 ranges {1654784=622592,2277376=655360,2932736=548864} 2020-06-12T02:38:24.475 INFO:tasks.rados.rados.0.smithi107.stdout:5698: writing smithi10729407-2 from 1654784 to 2277376 tid 1 2020-06-12T02:38:24.480 INFO:tasks.rados.rados.0.smithi107.stdout:5698: writing smithi10729407-2 from 2277376 to 2932736 tid 2 2020-06-12T02:38:24.484 INFO:tasks.rados.rados.0.smithi107.stdout:5698: writing smithi10729407-2 from 2932736 to 3481600 tid 3 2020-06-12T02:38:24.484 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:24.484 INFO:tasks.rados.rados.0.smithi107.stdout:5699: seq_num 5698 ranges {1417216=638976,2056192=540672,2596864=516096,3112960=147456} 2020-06-12T02:38:24.495 INFO:tasks.rados.rados.0.smithi107.stdout:5699: writing smithi10729407-5 from 1417216 to 2056192 tid 1 2020-06-12T02:38:24.498 INFO:tasks.rados.rados.0.smithi107.stdout:5699: writing smithi10729407-5 from 2056192 to 2596864 tid 2 2020-06-12T02:38:24.502 INFO:tasks.rados.rados.0.smithi107.stdout:5699: writing smithi10729407-5 from 2596864 to 3112960 tid 3 2020-06-12T02:38:24.503 INFO:tasks.rados.rados.0.smithi107.stdout:5699: writing smithi10729407-5 from 3112960 to 3260416 tid 4 2020-06-12T02:38:24.504 INFO:tasks.rados.rados.0.smithi107.stdout:5700: delete oid 17 current snap is 0 2020-06-12T02:38:24.504 INFO:tasks.rados.rados.0.smithi107.stdout:5693: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:24.504 INFO:tasks.rados.rados.0.smithi107.stdout:5693: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:24.504 INFO:tasks.rados.rados.0.smithi107.stdout:5693: finishing write tid 5 to smithi10729407-3 2020-06-12T02:38:24.504 INFO:tasks.rados.rados.0.smithi107.stdout:5693: finishing write tid 6 to smithi10729407-3 2020-06-12T02:38:24.505 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1024 (ObjNum 5692 snap 0 seq_num 5692) dirty exists 2020-06-12T02:38:24.505 INFO:tasks.rados.rados.0.smithi107.stdout:5693: left oid 3 (ObjNum 5692 snap 0 seq_num 5692) 2020-06-12T02:38:24.505 INFO:tasks.rados.rados.0.smithi107.stdout:5696: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:24.505 INFO:tasks.rados.rados.0.smithi107.stdout:5697: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:24.505 INFO:tasks.rados.rados.0.smithi107.stdout:5696: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:24.505 INFO:tasks.rados.rados.0.smithi107.stdout:5696: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:24.506 INFO:tasks.rados.rados.0.smithi107.stdout:5696: finishing write tid 5 to smithi10729407-12 2020-06-12T02:38:24.506 INFO:tasks.rados.rados.0.smithi107.stdout:5696: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:24.506 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1014 (ObjNum 5695 snap 0 seq_num 5695) dirty exists 2020-06-12T02:38:24.506 INFO:tasks.rados.rados.0.smithi107.stdout:5696: left oid 12 (ObjNum 5695 snap 0 seq_num 5695) 2020-06-12T02:38:24.508 INFO:tasks.rados.rados.0.smithi107.stdout:5697: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:24.509 INFO:tasks.rados.rados.0.smithi107.stdout:5697: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:24.509 INFO:tasks.rados.rados.0.smithi107.stdout:5697: finishing write tid 5 to smithi10729407-16 2020-06-12T02:38:24.509 INFO:tasks.rados.rados.0.smithi107.stdout:5697: finishing write tid 6 to smithi10729407-16 2020-06-12T02:38:24.509 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1028 (ObjNum 5696 snap 0 seq_num 5696) dirty exists 2020-06-12T02:38:24.509 INFO:tasks.rados.rados.0.smithi107.stdout:5697: left oid 16 (ObjNum 5696 snap 0 seq_num 5696) 2020-06-12T02:38:24.510 INFO:tasks.rados.rados.0.smithi107.stdout:5698: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:24.510 INFO:tasks.rados.rados.0.smithi107.stdout:5698: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:24.510 INFO:tasks.rados.rados.0.smithi107.stdout:5698: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:24.510 INFO:tasks.rados.rados.0.smithi107.stdout:5698: finishing write tid 5 to smithi10729407-2 2020-06-12T02:38:24.510 INFO:tasks.rados.rados.0.smithi107.stdout:5698: finishing write tid 6 to smithi10729407-2 2020-06-12T02:38:24.511 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 540 (ObjNum 5697 snap 0 seq_num 5697) dirty exists 2020-06-12T02:38:24.511 INFO:tasks.rados.rados.0.smithi107.stdout:5698: left oid 2 (ObjNum 5697 snap 0 seq_num 5697) 2020-06-12T02:38:24.512 INFO:tasks.rados.rados.0.smithi107.stdout:5699: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:24.512 INFO:tasks.rados.rados.0.smithi107.stdout:5693: done (6 left) 2020-06-12T02:38:24.512 INFO:tasks.rados.rados.0.smithi107.stdout:5696: done (5 left) 2020-06-12T02:38:24.512 INFO:tasks.rados.rados.0.smithi107.stdout:5697: done (4 left) 2020-06-12T02:38:24.513 INFO:tasks.rados.rados.0.smithi107.stdout:5698: done (3 left) 2020-06-12T02:38:24.513 INFO:tasks.rados.rados.0.smithi107.stdout:5700: done (2 left) 2020-06-12T02:38:24.513 INFO:tasks.rados.rados.0.smithi107.stdout:5701: delete oid 11 current snap is 0 2020-06-12T02:38:24.513 INFO:tasks.rados.rados.0.smithi107.stdout:5699: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:24.513 INFO:tasks.rados.rados.0.smithi107.stdout:5701: done (2 left) 2020-06-12T02:38:24.514 INFO:tasks.rados.rados.0.smithi107.stdout:5702: delete oid 18 current snap is 0 2020-06-12T02:38:24.514 INFO:tasks.rados.rados.0.smithi107.stdout:5699: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:24.514 INFO:tasks.rados.rados.0.smithi107.stdout:5699: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:24.514 INFO:tasks.rados.rados.0.smithi107.stdout:5699: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:24.515 INFO:tasks.rados.rados.0.smithi107.stdout:5699: finishing write tid 7 to smithi10729407-5 2020-06-12T02:38:24.515 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 573 (ObjNum 5698 snap 0 seq_num 5698) dirty exists 2020-06-12T02:38:24.515 INFO:tasks.rados.rados.0.smithi107.stdout:5699: left oid 5 (ObjNum 5698 snap 0 seq_num 5698) 2020-06-12T02:38:24.517 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:24.513+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:24.517 INFO:tasks.rados.rados.0.smithi107.stdout:5699: done (2 left) 2020-06-12T02:38:24.518 INFO:tasks.rados.rados.0.smithi107.stdout:5702: done (1 left) 2020-06-12T02:38:24.518 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:24.518 INFO:tasks.rados.rados.0.smithi107.stdout:5703: seq_num 5702 ranges {0=704512,704512=524288,1228800=491520,1720320=188416} 2020-06-12T02:38:24.520 INFO:tasks.rados.rados.0.smithi107.stdout:5703: writing smithi10729407-17 from 0 to 704512 tid 1 2020-06-12T02:38:24.523 INFO:tasks.rados.rados.0.smithi107.stdout:5703: writing smithi10729407-17 from 704512 to 1228800 tid 2 2020-06-12T02:38:24.526 INFO:tasks.rados.rados.0.smithi107.stdout:5703: writing smithi10729407-17 from 1228800 to 1720320 tid 3 2020-06-12T02:38:24.528 INFO:tasks.rados.rados.0.smithi107.stdout:5703: writing smithi10729407-17 from 1720320 to 1908736 tid 4 2020-06-12T02:38:24.529 INFO:tasks.rados.rados.0.smithi107.stdout:5704: delete oid 5 current snap is 0 2020-06-12T02:38:24.529 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:24.529 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:24.531 INFO:tasks.rados.rados.0.smithi107.stdout:5704: done (2 left) 2020-06-12T02:38:24.532 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:24.532 INFO:tasks.rados.rados.0.smithi107.stdout:5705: seq_num 5704 ranges {1253376=737280,1990656=696320,2686976=458752,3145728=49152} 2020-06-12T02:38:24.541 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-06-12T02:38:24.542 INFO:tasks.rados.rados.0.smithi107.stdout:5705: writing smithi10729407-6 from 1253376 to 1990656 tid 1 2020-06-12T02:38:24.546 INFO:tasks.rados.rados.0.smithi107.stdout:5705: writing smithi10729407-6 from 1990656 to 2686976 tid 2 2020-06-12T02:38:24.549 INFO:tasks.rados.rados.0.smithi107.stdout:5705: writing smithi10729407-6 from 2686976 to 3145728 tid 3 2020-06-12T02:38:24.550 INFO:tasks.rados.rados.0.smithi107.stdout:5705: writing smithi10729407-6 from 3145728 to 3194880 tid 4 2020-06-12T02:38:24.550 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:24.550 INFO:tasks.rados.rados.0.smithi107.stdout:5706: seq_num 5705 ranges {1236992=753664,1990656=581632,2572288=622592} 2020-06-12T02:38:24.561 INFO:tasks.rados.rados.0.smithi107.stdout:5706: writing smithi10729407-8 from 1236992 to 1990656 tid 1 2020-06-12T02:38:24.564 INFO:tasks.rados.rados.0.smithi107.stdout:5706: writing smithi10729407-8 from 1990656 to 2572288 tid 2 2020-06-12T02:38:24.568 INFO:tasks.rados.rados.0.smithi107.stdout:5706: writing smithi10729407-8 from 2572288 to 3194880 tid 3 2020-06-12T02:38:24.568 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:38:24.569 INFO:tasks.rados.rados.0.smithi107.stdout:5707: seq_num 5706 ranges {3481600=770048,4251648=466944,4718592=475136,5193728=262144} 2020-06-12T02:38:24.592 INFO:tasks.rados.rados.0.smithi107.stdout:5707: writing smithi10729407-2 from 3481600 to 4251648 tid 1 2020-06-12T02:38:24.594 INFO:tasks.rados.rados.0.smithi107.stdout:5707: writing smithi10729407-2 from 4251648 to 4718592 tid 2 2020-06-12T02:38:24.597 INFO:tasks.rados.rados.0.smithi107.stdout:5707: writing smithi10729407-2 from 4718592 to 5193728 tid 3 2020-06-12T02:38:24.601 INFO:tasks.rados.rados.0.smithi107.stdout:5707: writing smithi10729407-2 from 5193728 to 5455872 tid 4 2020-06-12T02:38:24.602 INFO:tasks.rados.rados.0.smithi107.stdout:5708: delete oid 20 current snap is 0 2020-06-12T02:38:24.602 INFO:tasks.rados.rados.0.smithi107.stdout:5703: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:24.602 INFO:tasks.rados.rados.0.smithi107.stdout:5703: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:24.602 INFO:tasks.rados.rados.0.smithi107.stdout:5703: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:24.602 INFO:tasks.rados.rados.0.smithi107.stdout:5703: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:24.603 INFO:tasks.rados.rados.0.smithi107.stdout:5703: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:24.603 INFO:tasks.rados.rados.0.smithi107.stdout:5703: finishing write tid 7 to smithi10729407-17 2020-06-12T02:38:24.603 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1063 (ObjNum 5702 snap 0 seq_num 5702) dirty exists 2020-06-12T02:38:24.603 INFO:tasks.rados.rados.0.smithi107.stdout:5703: left oid 17 (ObjNum 5702 snap 0 seq_num 5702) 2020-06-12T02:38:24.603 INFO:tasks.rados.rados.0.smithi107.stdout:5705: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:24.604 INFO:tasks.rados.rados.0.smithi107.stdout:5705: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:24.604 INFO:tasks.rados.rados.0.smithi107.stdout:5705: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:24.604 INFO:tasks.rados.rados.0.smithi107.stdout:5705: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:24.604 INFO:tasks.rados.rados.0.smithi107.stdout:5705: finishing write tid 6 to smithi10729407-6 2020-06-12T02:38:24.604 INFO:tasks.rados.rados.0.smithi107.stdout:5705: finishing write tid 7 to smithi10729407-6 2020-06-12T02:38:24.604 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1054 (ObjNum 5704 snap 0 seq_num 5704) dirty exists 2020-06-12T02:38:24.605 INFO:tasks.rados.rados.0.smithi107.stdout:5705: left oid 6 (ObjNum 5704 snap 0 seq_num 5704) 2020-06-12T02:38:24.605 INFO:tasks.rados.rados.0.smithi107.stdout:5706: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:24.605 INFO:tasks.rados.rados.0.smithi107.stdout:5706: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:24.605 INFO:tasks.rados.rados.0.smithi107.stdout:5706: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:24.605 INFO:tasks.rados.rados.0.smithi107.stdout:5706: finishing write tid 5 to smithi10729407-8 2020-06-12T02:38:24.606 INFO:tasks.rados.rados.0.smithi107.stdout:5706: finishing write tid 6 to smithi10729407-8 2020-06-12T02:38:24.606 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1043 (ObjNum 5705 snap 0 seq_num 5705) dirty exists 2020-06-12T02:38:24.606 INFO:tasks.rados.rados.0.smithi107.stdout:5706: left oid 8 (ObjNum 5705 snap 0 seq_num 5705) 2020-06-12T02:38:24.606 INFO:tasks.rados.rados.0.smithi107.stdout:5707: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:24.607 INFO:tasks.rados.rados.0.smithi107.stdout:5703: done (5 left) 2020-06-12T02:38:24.607 INFO:tasks.rados.rados.0.smithi107.stdout:5705: done (4 left) 2020-06-12T02:38:24.607 INFO:tasks.rados.rados.0.smithi107.stdout:5706: done (3 left) 2020-06-12T02:38:24.608 INFO:tasks.rados.rados.0.smithi107.stdout:5708: done (2 left) 2020-06-12T02:38:24.608 INFO:tasks.rados.rados.0.smithi107.stdout:5709: delete oid 5 current snap is 0 2020-06-12T02:38:24.608 INFO:tasks.rados.rados.0.smithi107.stdout:5709: done (2 left) 2020-06-12T02:38:24.608 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:24.608 INFO:tasks.rados.rados.0.smithi107.stdout:5710: seq_num 5709 ranges {0=417792,417792=516096,933888=647168,1581056=442368} 2020-06-12T02:38:24.608 INFO:tasks.rados.rados.0.smithi107.stdout:5710: writing smithi10729407-15 from 0 to 417792 tid 1 2020-06-12T02:38:24.612 INFO:tasks.rados.rados.0.smithi107.stdout:5710: writing smithi10729407-15 from 417792 to 933888 tid 2 2020-06-12T02:38:24.613 INFO:tasks.rados.rados.0.smithi107.stdout:5710: writing smithi10729407-15 from 933888 to 1581056 tid 3 2020-06-12T02:38:24.621 INFO:tasks.rados.rados.0.smithi107.stdout:5710: writing smithi10729407-15 from 1581056 to 2023424 tid 4 2020-06-12T02:38:24.621 INFO:tasks.rados.rados.0.smithi107.stdout:5711: delete oid 3 current snap is 0 2020-06-12T02:38:24.621 INFO:tasks.rados.rados.0.smithi107.stdout:5707: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:24.622 INFO:tasks.rados.rados.0.smithi107.stdout:5707: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:24.622 INFO:tasks.rados.rados.0.smithi107.stdout:5707: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:24.622 INFO:tasks.rados.rados.0.smithi107.stdout:5707: finishing write tid 6 to smithi10729407-2 2020-06-12T02:38:24.622 INFO:tasks.rados.rados.0.smithi107.stdout:5707: finishing write tid 7 to smithi10729407-2 2020-06-12T02:38:24.623 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 545 (ObjNum 5706 snap 0 seq_num 5706) dirty exists 2020-06-12T02:38:24.623 INFO:tasks.rados.rados.0.smithi107.stdout:5707: left oid 2 (ObjNum 5706 snap 0 seq_num 5706) 2020-06-12T02:38:24.623 INFO:tasks.rados.rados.0.smithi107.stdout:5710: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:24.623 INFO:tasks.rados.rados.0.smithi107.stdout:5707: done (3 left) 2020-06-12T02:38:24.623 INFO:tasks.rados.rados.0.smithi107.stdout:5711: done (2 left) 2020-06-12T02:38:24.624 INFO:tasks.rados.rados.0.smithi107.stdout:5712: delete oid 5 current snap is 0 2020-06-12T02:38:24.624 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:24.619+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:24.625 INFO:tasks.rados.rados.0.smithi107.stdout:5712: done (2 left) 2020-06-12T02:38:24.625 INFO:tasks.rados.rados.0.smithi107.stdout:5713: delete oid 9 current snap is 0 2020-06-12T02:38:24.625 INFO:tasks.rados.rados.0.smithi107.stdout:5713: done (2 left) 2020-06-12T02:38:24.626 INFO:tasks.rados.rados.0.smithi107.stdout:5714: delete oid 17 current snap is 0 2020-06-12T02:38:24.626 INFO:tasks.rados.rados.0.smithi107.stdout:5710: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:24.626 INFO:tasks.rados.rados.0.smithi107.stdout:5710: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:24.627 INFO:tasks.rados.rados.0.smithi107.stdout:5710: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:24.627 INFO:tasks.rados.rados.0.smithi107.stdout:5710: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:24.628 INFO:tasks.rados.rados.0.smithi107.stdout:5710: finishing write tid 7 to smithi10729407-15 2020-06-12T02:38:24.628 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1048 (ObjNum 5709 snap 0 seq_num 5709) dirty exists 2020-06-12T02:38:24.628 INFO:tasks.rados.rados.0.smithi107.stdout:5710: left oid 15 (ObjNum 5709 snap 0 seq_num 5709) 2020-06-12T02:38:24.629 INFO:tasks.rados.rados.0.smithi107.stdout:5710: done (2 left) 2020-06-12T02:38:24.629 INFO:tasks.rados.rados.0.smithi107.stdout:5714: done (1 left) 2020-06-12T02:38:24.629 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:24.630 INFO:tasks.rados.rados.0.smithi107.stdout:5715: seq_num 5714 ranges {0=507904,507904=786432,1294336=434176,1728512=385024} 2020-06-12T02:38:24.632 INFO:tasks.rados.rados.0.smithi107.stdout:5715: writing smithi10729407-14 from 0 to 507904 tid 1 2020-06-12T02:38:24.638 INFO:tasks.rados.rados.0.smithi107.stdout:5715: writing smithi10729407-14 from 507904 to 1294336 tid 2 2020-06-12T02:38:24.641 INFO:tasks.rados.rados.0.smithi107.stdout:5715: writing smithi10729407-14 from 1294336 to 1728512 tid 3 2020-06-12T02:38:24.644 INFO:tasks.rados.rados.0.smithi107.stdout:5715: writing smithi10729407-14 from 1728512 to 2113536 tid 4 2020-06-12T02:38:24.644 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:24.644 INFO:tasks.rados.rados.0.smithi107.stdout:5716: seq_num 5715 ranges {0=524288,524288=671744,1196032=647168,1843200=286720} 2020-06-12T02:38:24.647 INFO:tasks.rados.rados.0.smithi107.stdout:5716: writing smithi10729407-4 from 0 to 524288 tid 1 2020-06-12T02:38:24.652 INFO:tasks.rados.rados.0.smithi107.stdout:5716: writing smithi10729407-4 from 524288 to 1196032 tid 2 2020-06-12T02:38:24.655 INFO:tasks.rados.rados.0.smithi107.stdout:5716: writing smithi10729407-4 from 1196032 to 1843200 tid 3 2020-06-12T02:38:24.658 INFO:tasks.rados.rados.0.smithi107.stdout:5716: writing smithi10729407-4 from 1843200 to 2129920 tid 4 2020-06-12T02:38:24.658 INFO:tasks.rados.rados.0.smithi107.stdout:5717: delete oid 9 current snap is 0 2020-06-12T02:38:24.658 INFO:tasks.rados.rados.0.smithi107.stdout:5715: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:24.658 INFO:tasks.rados.rados.0.smithi107.stdout:5715: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:24.659 INFO:tasks.rados.rados.0.smithi107.stdout:5715: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:24.659 INFO:tasks.rados.rados.0.smithi107.stdout:5715: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:24.659 INFO:tasks.rados.rados.0.smithi107.stdout:5715: finishing write tid 6 to smithi10729407-14 2020-06-12T02:38:24.659 INFO:tasks.rados.rados.0.smithi107.stdout:5716: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:24.659 INFO:tasks.rados.rados.0.smithi107.stdout:5715: finishing write tid 7 to smithi10729407-14 2020-06-12T02:38:24.660 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1026 (ObjNum 5714 snap 0 seq_num 5714) dirty exists 2020-06-12T02:38:24.660 INFO:tasks.rados.rados.0.smithi107.stdout:5715: left oid 14 (ObjNum 5714 snap 0 seq_num 5714) 2020-06-12T02:38:24.666 INFO:tasks.rados.rados.0.smithi107.stdout:5715: done (3 left) 2020-06-12T02:38:24.666 INFO:tasks.rados.rados.0.smithi107.stdout:5717: done (2 left) 2020-06-12T02:38:24.667 INFO:tasks.rados.rados.0.smithi107.stdout:5718: delete oid 7 current snap is 0 2020-06-12T02:38:24.667 INFO:tasks.rados.rados.0.smithi107.stdout:5716: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:24.668 INFO:tasks.rados.rados.0.smithi107.stdout:5716: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:24.668 INFO:tasks.rados.rados.0.smithi107.stdout:5716: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:24.669 INFO:tasks.rados.rados.0.smithi107.stdout:5718: done (2 left) 2020-06-12T02:38:24.669 INFO:tasks.rados.rados.0.smithi107.stdout:5719: delete oid 5 current snap is 0 2020-06-12T02:38:24.669 INFO:tasks.rados.rados.0.smithi107.stdout:5716: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:24.673 INFO:tasks.rados.rados.0.smithi107.stdout:5716: finishing write tid 7 to smithi10729407-4 2020-06-12T02:38:24.673 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1069 (ObjNum 5715 snap 0 seq_num 5715) dirty exists 2020-06-12T02:38:24.673 INFO:tasks.rados.rados.0.smithi107.stdout:5716: left oid 4 (ObjNum 5715 snap 0 seq_num 5715) 2020-06-12T02:38:24.674 INFO:tasks.rados.rados.0.smithi107.stdout:5716: done (2 left) 2020-06-12T02:38:24.674 INFO:tasks.rados.rados.0.smithi107.stdout:5719: done (1 left) 2020-06-12T02:38:24.675 INFO:tasks.rados.rados.0.smithi107.stdout:5720: delete oid 7 current snap is 0 2020-06-12T02:38:24.676 INFO:tasks.rados.rados.0.smithi107.stdout:5720: done (1 left) 2020-06-12T02:38:24.676 INFO:tasks.rados.rados.0.smithi107.stdout:5721: delete oid 12 current snap is 0 2020-06-12T02:38:24.678 INFO:tasks.rados.rados.0.smithi107.stdout:5721: done (1 left) 2020-06-12T02:38:24.679 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:24.679 INFO:tasks.rados.rados.0.smithi107.stdout:5722: seq_num 5721 ranges {0=622592,622592=425984,1048576=589824,1638400=409600,2048000=180224} 2020-06-12T02:38:24.683 INFO:tasks.rados.rados.0.smithi107.stdout:5722: writing smithi10729407-5 from 0 to 622592 tid 1 2020-06-12T02:38:24.684 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:24.684 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:24.685 INFO:tasks.rados.rados.0.smithi107.stdout:5722: writing smithi10729407-5 from 622592 to 1048576 tid 2 2020-06-12T02:38:24.688 INFO:tasks.rados.rados.0.smithi107.stdout:5722: writing smithi10729407-5 from 1048576 to 1638400 tid 3 2020-06-12T02:38:24.691 INFO:tasks.rados.rados.0.smithi107.stdout:5722: writing smithi10729407-5 from 1638400 to 2048000 tid 4 2020-06-12T02:38:24.692 INFO:tasks.rados.rados.0.smithi107.stdout:5722: writing smithi10729407-5 from 2048000 to 2228224 tid 5 2020-06-12T02:38:24.693 INFO:tasks.rados.rados.0.smithi107.stdout:5723: delete oid 17 current snap is 0 2020-06-12T02:38:24.694 INFO:tasks.rados.rados.0.smithi107.stdout:5722: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:24.695 INFO:tasks.rados.rados.0.smithi107.stdout:5722: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:24.696 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-06-12T02:38:24.697 INFO:tasks.rados.rados.0.smithi107.stdout:5722: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:24.698 INFO:tasks.rados.rados.0.smithi107.stdout:5723: done (2 left) 2020-06-12T02:38:24.698 INFO:tasks.rados.rados.0.smithi107.stdout:5724: delete oid 6 current snap is 0 2020-06-12T02:38:24.701 INFO:tasks.rados.rados.0.smithi107.stdout:5722: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:24.701 INFO:tasks.rados.rados.0.smithi107.stdout:5724: done (2 left) 2020-06-12T02:38:24.701 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:24.701 INFO:tasks.rados.rados.0.smithi107.stdout:5725: seq_num 5724 ranges {0=671744,671744=483328,1155072=409600,1564672=573440,2138112=139264} 2020-06-12T02:38:24.704 INFO:tasks.rados.rados.0.smithi107.stdout:5725: writing smithi10729407-6 from 0 to 671744 tid 1 2020-06-12T02:38:24.707 INFO:tasks.rados.rados.0.smithi107.stdout:5725: writing smithi10729407-6 from 671744 to 1155072 tid 2 2020-06-12T02:38:24.710 INFO:tasks.rados.rados.0.smithi107.stdout:5725: writing smithi10729407-6 from 1155072 to 1564672 tid 3 2020-06-12T02:38:24.713 INFO:tasks.rados.rados.0.smithi107.stdout:5725: writing smithi10729407-6 from 1564672 to 2138112 tid 4 2020-06-12T02:38:24.714 INFO:tasks.rados.rados.0.smithi107.stdout:5725: writing smithi10729407-6 from 2138112 to 2277376 tid 5 2020-06-12T02:38:24.715 INFO:tasks.rados.rados.0.smithi107.stdout:5722: finishing write tid 5 to smithi10729407-5 2020-06-12T02:38:24.715 INFO:tasks.rados.rados.0.smithi107.stdout:5726: delete oid 17 current snap is 0 2020-06-12T02:38:24.715 INFO:tasks.rados.rados.0.smithi107.stdout:5722: finishing write tid 7 to smithi10729407-5 2020-06-12T02:38:24.715 INFO:tasks.rados.rados.0.smithi107.stdout:5722: finishing write tid 8 to smithi10729407-5 2020-06-12T02:38:24.715 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 583 (ObjNum 5721 snap 0 seq_num 5721) dirty exists 2020-06-12T02:38:24.716 INFO:tasks.rados.rados.0.smithi107.stdout:5722: left oid 5 (ObjNum 5721 snap 0 seq_num 5721) 2020-06-12T02:38:24.716 INFO:tasks.rados.rados.0.smithi107.stdout:5725: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:24.719 INFO:tasks.rados.rados.0.smithi107.stdout:5722: done (3 left) 2020-06-12T02:38:24.720 INFO:tasks.rados.rados.0.smithi107.stdout:5726: done (2 left) 2020-06-12T02:38:24.720 INFO:tasks.rados.rados.0.smithi107.stdout:5727: delete oid 8 current snap is 0 2020-06-12T02:38:24.722 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:24.719+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:24.723 INFO:tasks.rados.rados.0.smithi107.stdout:5727: done (2 left) 2020-06-12T02:38:24.724 INFO:tasks.rados.rados.0.smithi107.stdout:5728: delete oid 11 current snap is 0 2020-06-12T02:38:24.728 INFO:tasks.rados.rados.0.smithi107.stdout:5725: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:24.729 INFO:tasks.rados.rados.0.smithi107.stdout:5725: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:24.729 INFO:tasks.rados.rados.0.smithi107.stdout:5725: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:24.729 INFO:tasks.rados.rados.0.smithi107.stdout:5728: done (2 left) 2020-06-12T02:38:24.729 INFO:tasks.rados.rados.0.smithi107.stdout:5729: delete oid 8 current snap is 0 2020-06-12T02:38:24.731 INFO:tasks.rados.rados.0.smithi107.stdout:5725: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:24.731 INFO:tasks.rados.rados.0.smithi107.stdout:5725: finishing write tid 7 to smithi10729407-6 2020-06-12T02:38:24.733 INFO:tasks.rados.rados.0.smithi107.stdout:5725: finishing write tid 8 to smithi10729407-6 2020-06-12T02:38:24.733 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1061 (ObjNum 5724 snap 0 seq_num 5724) dirty exists 2020-06-12T02:38:24.733 INFO:tasks.rados.rados.0.smithi107.stdout:5725: left oid 6 (ObjNum 5724 snap 0 seq_num 5724) 2020-06-12T02:38:24.733 INFO:tasks.rados.rados.0.smithi107.stdout:5725: done (2 left) 2020-06-12T02:38:24.734 INFO:tasks.rados.rados.0.smithi107.stdout:5729: done (1 left) 2020-06-12T02:38:24.734 INFO:tasks.rados.rados.0.smithi107.stdout:5730: delete oid 19 current snap is 0 2020-06-12T02:38:24.735 INFO:tasks.rados.rados.0.smithi107.stdout:5730: done (1 left) 2020-06-12T02:38:24.736 INFO:tasks.rados.rados.0.smithi107.stdout:5731: delete oid 12 current snap is 0 2020-06-12T02:38:24.739 INFO:tasks.rados.rados.0.smithi107.stdout:5731: done (1 left) 2020-06-12T02:38:24.739 INFO:tasks.rados.rados.0.smithi107.stdout:5732: delete oid 7 current snap is 0 2020-06-12T02:38:24.741 INFO:tasks.rados.rados.0.smithi107.stdout:5732: done (1 left) 2020-06-12T02:38:24.742 INFO:tasks.rados.rados.0.smithi107.stdout:5733: delete oid 17 current snap is 0 2020-06-12T02:38:24.743 INFO:tasks.rados.rados.0.smithi107.stdout:5733: done (1 left) 2020-06-12T02:38:24.744 INFO:tasks.rados.rados.0.smithi107.stdout:5734: delete oid 18 current snap is 0 2020-06-12T02:38:24.746 INFO:tasks.rados.rados.0.smithi107.stdout:5734: done (1 left) 2020-06-12T02:38:24.747 INFO:tasks.rados.rados.0.smithi107.stdout:5735: delete oid 1 current snap is 0 2020-06-12T02:38:24.749 INFO:tasks.rados.rados.0.smithi107.stdout:5735: done (1 left) 2020-06-12T02:38:24.749 INFO:tasks.rados.rados.0.smithi107.stdout:5736: delete oid 18 current snap is 0 2020-06-12T02:38:24.750 INFO:tasks.rados.rados.0.smithi107.stdout:5736: done (1 left) 2020-06-12T02:38:24.751 INFO:tasks.rados.rados.0.smithi107.stdout:5737: delete oid 6 current snap is 0 2020-06-12T02:38:24.753 INFO:tasks.rados.rados.0.smithi107.stdout:5737: done (1 left) 2020-06-12T02:38:24.753 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:38:24.754 INFO:tasks.rados.rados.0.smithi107.stdout:5738: seq_num 5737 ranges {0=90112} 2020-06-12T02:38:24.754 INFO:tasks.rados.rados.0.smithi107.stdout:5738: writing smithi10729407-6 from 0 to 90112 tid 1 2020-06-12T02:38:24.755 INFO:tasks.rados.rados.0.smithi107.stdout:5739: delete oid 2 current snap is 0 2020-06-12T02:38:24.756 INFO:tasks.rados.rados.0.smithi107.stdout:5739: done (2 left) 2020-06-12T02:38:24.756 INFO:tasks.rados.rados.0.smithi107.stdout:5740: delete oid 16 current snap is 0 2020-06-12T02:38:24.759 INFO:tasks.rados.rados.0.smithi107.stdout:5740: done (2 left) 2020-06-12T02:38:24.759 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:38:24.759 INFO:tasks.rados.rados.0.smithi107.stdout:5741: seq_num 5740 ranges {2228224=139264} 2020-06-12T02:38:24.772 INFO:tasks.rados.rados.0.smithi107.stdout:5741: writing smithi10729407-5 from 2228224 to 2367488 tid 1 2020-06-12T02:38:24.772 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:24.772 INFO:tasks.rados.rados.0.smithi107.stdout:5742: seq_num 5741 ranges {0=155648} 2020-06-12T02:38:24.773 INFO:tasks.rados.rados.0.smithi107.stdout:5742: writing smithi10729407-3 from 0 to 155648 tid 1 2020-06-12T02:38:24.773 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:24.773 INFO:tasks.rados.rados.0.smithi107.stdout:5743: seq_num 5742 ranges {0=172032} 2020-06-12T02:38:24.774 INFO:tasks.rados.rados.0.smithi107.stdout:5743: writing smithi10729407-19 from 0 to 172032 tid 1 2020-06-12T02:38:24.774 INFO:tasks.rados.rados.0.smithi107.stdout:5744: delete oid 9 current snap is 0 2020-06-12T02:38:24.774 INFO:tasks.rados.rados.0.smithi107.stdout:5738: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:24.775 INFO:tasks.rados.rados.0.smithi107.stdout:5738: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:24.775 INFO:tasks.rados.rados.0.smithi107.stdout:5738: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:24.775 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1065 (ObjNum 5737 snap 0 seq_num 5737) dirty exists 2020-06-12T02:38:24.775 INFO:tasks.rados.rados.0.smithi107.stdout:5738: left oid 6 (ObjNum 5737 snap 0 seq_num 5737) 2020-06-12T02:38:24.776 INFO:tasks.rados.rados.0.smithi107.stdout:5741: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:24.779 INFO:tasks.rados.rados.0.smithi107.stdout:5738: done (5 left) 2020-06-12T02:38:24.779 INFO:tasks.rados.rados.0.smithi107.stdout:5744: done (4 left) 2020-06-12T02:38:24.779 INFO:tasks.rados.rados.0.smithi107.stdout:5745: delete oid 6 current snap is 0 2020-06-12T02:38:24.781 INFO:tasks.rados.rados.0.smithi107.stdout:5742: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:24.782 INFO:tasks.rados.rados.0.smithi107.stdout:5741: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:24.782 INFO:tasks.rados.rados.0.smithi107.stdout:5745: done (4 left) 2020-06-12T02:38:24.782 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:24.783 INFO:tasks.rados.rados.0.smithi107.stdout:5746: seq_num 5745 ranges {0=221184} 2020-06-12T02:38:24.783 INFO:tasks.rados.rados.0.smithi107.stdout:5746: writing smithi10729407-1 from 0 to 221184 tid 1 2020-06-12T02:38:24.784 INFO:tasks.rados.rados.0.smithi107.stdout:5747: delete oid 13 current snap is 0 2020-06-12T02:38:24.784 INFO:tasks.rados.rados.0.smithi107.stdout:5743: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:24.784 INFO:tasks.rados.rados.0.smithi107.stdout:5743: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:24.785 INFO:tasks.rados.rados.0.smithi107.stdout:5743: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:24.785 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1019 (ObjNum 5742 snap 0 seq_num 5742) dirty exists 2020-06-12T02:38:24.785 INFO:tasks.rados.rados.0.smithi107.stdout:5743: left oid 19 (ObjNum 5742 snap 0 seq_num 5742) 2020-06-12T02:38:24.785 INFO:tasks.rados.rados.0.smithi107.stdout:5741: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:24.786 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 585 (ObjNum 5740 snap 0 seq_num 5740) dirty exists 2020-06-12T02:38:24.786 INFO:tasks.rados.rados.0.smithi107.stdout:5741: left oid 5 (ObjNum 5740 snap 0 seq_num 5740) 2020-06-12T02:38:24.789 INFO:tasks.rados.rados.0.smithi107.stdout:5742: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:24.789 INFO:tasks.rados.rados.0.smithi107.stdout:5741: done (5 left) 2020-06-12T02:38:24.789 INFO:tasks.rados.rados.0.smithi107.stdout:5743: done (4 left) 2020-06-12T02:38:24.789 INFO:tasks.rados.rados.0.smithi107.stdout:5747: done (3 left) 2020-06-12T02:38:24.790 INFO:tasks.rados.rados.0.smithi107.stdout:5748: delete oid 14 current snap is 0 2020-06-12T02:38:24.791 INFO:tasks.rados.rados.0.smithi107.stdout:5742: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:24.791 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1032 (ObjNum 5741 snap 0 seq_num 5741) dirty exists 2020-06-12T02:38:24.791 INFO:tasks.rados.rados.0.smithi107.stdout:5742: left oid 3 (ObjNum 5741 snap 0 seq_num 5741) 2020-06-12T02:38:24.791 INFO:tasks.rados.rados.0.smithi107.stdout:5746: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:24.794 INFO:tasks.rados.rados.0.smithi107.stdout:5742: done (3 left) 2020-06-12T02:38:24.794 INFO:tasks.rados.rados.0.smithi107.stdout:5748: done (2 left) 2020-06-12T02:38:24.794 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:24.794 INFO:tasks.rados.rados.0.smithi107.stdout:5749: seq_num 5748 ranges {0=270336} 2020-06-12T02:38:24.796 INFO:tasks.rados.rados.0.smithi107.stdout:5749: writing smithi10729407-18 from 0 to 270336 tid 1 2020-06-12T02:38:24.796 INFO:tasks.rados.rados.0.smithi107.stdout:5750: delete oid 5 current snap is 0 2020-06-12T02:38:24.798 INFO:tasks.rados.rados.0.smithi107.stdout:5750: done (3 left) 2020-06-12T02:38:24.798 INFO:tasks.rados.rados.0.smithi107.stdout:5751: delete oid 16 current snap is 0 2020-06-12T02:38:24.800 INFO:tasks.rados.rados.0.smithi107.stdout:5746: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:24.801 INFO:tasks.rados.rados.0.smithi107.stdout:5746: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:24.802 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1068 (ObjNum 5745 snap 0 seq_num 5745) dirty exists 2020-06-12T02:38:24.802 INFO:tasks.rados.rados.0.smithi107.stdout:5746: left oid 1 (ObjNum 5745 snap 0 seq_num 5745) 2020-06-12T02:38:24.802 INFO:tasks.rados.rados.0.smithi107.stdout:5749: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:24.805 INFO:tasks.rados.rados.0.smithi107.stdout:5749: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:24.805 INFO:tasks.rados.rados.0.smithi107.stdout:5746: done (3 left) 2020-06-12T02:38:24.805 INFO:tasks.rados.rados.0.smithi107.stdout:5751: done (2 left) 2020-06-12T02:38:24.806 INFO:tasks.rados.rados.0.smithi107.stdout:5752: delete oid 5 current snap is 0 2020-06-12T02:38:24.806 INFO:tasks.rados.rados.0.smithi107.stdout:5749: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:24.807 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1044 (ObjNum 5748 snap 0 seq_num 5748) dirty exists 2020-06-12T02:38:24.807 INFO:tasks.rados.rados.0.smithi107.stdout:5749: left oid 18 (ObjNum 5748 snap 0 seq_num 5748) 2020-06-12T02:38:24.807 INFO:tasks.rados.rados.0.smithi107.stdout:5749: done (2 left) 2020-06-12T02:38:24.808 INFO:tasks.rados.rados.0.smithi107.stdout:5752: done (1 left) 2020-06-12T02:38:24.808 INFO:tasks.rados.rados.0.smithi107.stdout:5753: delete oid 13 current snap is 0 2020-06-12T02:38:24.809 INFO:tasks.rados.rados.0.smithi107.stdout:5753: done (1 left) 2020-06-12T02:38:24.810 INFO:tasks.rados.rados.0.smithi107.stdout:5754: delete oid 19 current snap is 0 2020-06-12T02:38:24.811 INFO:tasks.rados.rados.0.smithi107.stdout:5754: done (1 left) 2020-06-12T02:38:24.811 INFO:tasks.rados.rados.0.smithi107.stdout:5755: delete oid 16 current snap is 0 2020-06-12T02:38:24.815 INFO:tasks.rados.rados.0.smithi107.stdout:5755: done (1 left) 2020-06-12T02:38:24.816 INFO:tasks.rados.rados.0.smithi107.stdout:5756: delete oid 7 current snap is 0 2020-06-12T02:38:24.817 INFO:tasks.rados.rados.0.smithi107.stdout:5756: done (1 left) 2020-06-12T02:38:24.818 INFO:tasks.rados.rados.0.smithi107.stdout:5757: delete oid 4 current snap is 0 2020-06-12T02:38:24.827 INFO:tasks.rados.rados.0.smithi107.stdout:5757: done (1 left) 2020-06-12T02:38:24.827 INFO:tasks.rados.rados.0.smithi107.stdout:5758: delete oid 12 current snap is 0 2020-06-12T02:38:24.827 INFO:tasks.rados.rados.0.smithi107.stdout:5758: done (1 left) 2020-06-12T02:38:24.827 INFO:tasks.rados.rados.0.smithi107.stdout:5759: delete oid 14 current snap is 0 2020-06-12T02:38:24.828 INFO:tasks.rados.rados.0.smithi107.stdout:5759: done (1 left) 2020-06-12T02:38:24.828 INFO:tasks.rados.rados.0.smithi107.stdout:5760: delete oid 8 current snap is 0 2020-06-12T02:38:24.828 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:24.823+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:24.829 INFO:tasks.rados.rados.0.smithi107.stdout:5760: done (1 left) 2020-06-12T02:38:24.829 INFO:tasks.rados.rados.0.smithi107.stdout:5761: delete oid 5 current snap is 0 2020-06-12T02:38:24.829 INFO:tasks.rados.rados.0.smithi107.stdout:5761: done (1 left) 2020-06-12T02:38:24.830 INFO:tasks.rados.rados.0.smithi107.stdout:5762: delete oid 19 current snap is 0 2020-06-12T02:38:24.833 INFO:tasks.rados.rados.0.smithi107.stdout:5762: done (1 left) 2020-06-12T02:38:24.834 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:24.834 INFO:tasks.rados.rados.0.smithi107.stdout:5763: seq_num 5762 ranges {0=507904} 2020-06-12T02:38:24.835 INFO:tasks.rados.rados.0.smithi107.stdout:5763: writing smithi10729407-19 from 0 to 507904 tid 1 2020-06-12T02:38:24.835 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:38:24.835 INFO:tasks.rados.rados.0.smithi107.stdout:5764: seq_num 5763 ranges {0=524288} 2020-06-12T02:38:24.836 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:24.836 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:24.838 INFO:tasks.rados.rados.0.smithi107.stdout:5764: writing smithi10729407-5 from 0 to 524288 tid 1 2020-06-12T02:38:24.838 INFO:tasks.rados.rados.0.smithi107.stdout:5765: delete oid 15 current snap is 0 2020-06-12T02:38:24.842 INFO:tasks.rados.rados.0.smithi107.stdout:5763: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:24.844 INFO:tasks.rados.rados.0.smithi107.stdout:5764: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:24.845 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-06-12T02:38:24.846 INFO:tasks.rados.rados.0.smithi107.stdout:5765: done (3 left) 2020-06-12T02:38:24.846 INFO:tasks.rados.rados.0.smithi107.stdout:5766: delete oid 18 current snap is 0 2020-06-12T02:38:24.848 INFO:tasks.rados.rados.0.smithi107.stdout:5763: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:24.850 INFO:tasks.rados.rados.0.smithi107.stdout:5763: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:24.850 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1024 (ObjNum 5762 snap 0 seq_num 5762) dirty exists 2020-06-12T02:38:24.851 INFO:tasks.rados.rados.0.smithi107.stdout:5763: left oid 19 (ObjNum 5762 snap 0 seq_num 5762) 2020-06-12T02:38:24.851 INFO:tasks.rados.rados.0.smithi107.stdout:5763: done (3 left) 2020-06-12T02:38:24.851 INFO:tasks.rados.rados.0.smithi107.stdout:5766: done (2 left) 2020-06-12T02:38:24.851 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:24.851 INFO:tasks.rados.rados.0.smithi107.stdout:5767: seq_num 5766 ranges {0=573440} 2020-06-12T02:38:24.854 INFO:tasks.rados.rados.0.smithi107.stdout:5767: writing smithi10729407-9 from 0 to 573440 tid 1 2020-06-12T02:38:24.854 INFO:tasks.rados.rados.0.smithi107.stdout:5768: delete oid 19 current snap is 0 2020-06-12T02:38:24.854 INFO:tasks.rados.rados.0.smithi107.stdout:5764: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:24.855 INFO:tasks.rados.rados.0.smithi107.stdout:5764: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:24.855 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 590 (ObjNum 5763 snap 0 seq_num 5763) dirty exists 2020-06-12T02:38:24.855 INFO:tasks.rados.rados.0.smithi107.stdout:5764: left oid 5 (ObjNum 5763 snap 0 seq_num 5763) 2020-06-12T02:38:24.858 INFO:tasks.rados.rados.0.smithi107.stdout:5764: done (3 left) 2020-06-12T02:38:24.859 INFO:tasks.rados.rados.0.smithi107.stdout:5768: done (2 left) 2020-06-12T02:38:24.860 INFO:tasks.rados.rados.0.smithi107.stdout:5769: delete oid 11 current snap is 0 2020-06-12T02:38:24.861 INFO:tasks.rados.rados.0.smithi107.stdout:5767: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:24.864 INFO:tasks.rados.rados.0.smithi107.stdout:5769: done (2 left) 2020-06-12T02:38:24.864 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:24.864 INFO:tasks.rados.rados.0.smithi107.stdout:5770: seq_num 5769 ranges {0=622592} 2020-06-12T02:38:24.867 INFO:tasks.rados.rados.0.smithi107.stdout:5770: writing smithi10729407-6 from 0 to 622592 tid 1 2020-06-12T02:38:24.868 INFO:tasks.rados.rados.0.smithi107.stdout:5771: delete oid 1 current snap is 0 2020-06-12T02:38:24.868 INFO:tasks.rados.rados.0.smithi107.stdout:5767: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:24.868 INFO:tasks.rados.rados.0.smithi107.stdout:5767: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:24.869 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1047 (ObjNum 5766 snap 0 seq_num 5766) dirty exists 2020-06-12T02:38:24.869 INFO:tasks.rados.rados.0.smithi107.stdout:5767: left oid 9 (ObjNum 5766 snap 0 seq_num 5766) 2020-06-12T02:38:24.873 INFO:tasks.rados.rados.0.smithi107.stdout:5770: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:24.875 INFO:tasks.rados.rados.0.smithi107.stdout:5767: done (3 left) 2020-06-12T02:38:24.875 INFO:tasks.rados.rados.0.smithi107.stdout:5771: done (2 left) 2020-06-12T02:38:24.875 INFO:tasks.rados.rados.0.smithi107.stdout:5772: delete oid 13 current snap is 0 2020-06-12T02:38:24.877 INFO:tasks.rados.rados.0.smithi107.stdout:5770: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:24.878 INFO:tasks.rados.rados.0.smithi107.stdout:5772: done (2 left) 2020-06-12T02:38:24.878 INFO:tasks.rados.rados.0.smithi107.stdout:5773: delete oid 17 current snap is 0 2020-06-12T02:38:24.886 INFO:tasks.rados.rados.0.smithi107.stdout:5770: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:24.887 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1071 (ObjNum 5769 snap 0 seq_num 5769) dirty exists 2020-06-12T02:38:24.887 INFO:tasks.rados.rados.0.smithi107.stdout:5770: left oid 6 (ObjNum 5769 snap 0 seq_num 5769) 2020-06-12T02:38:24.888 INFO:tasks.rados.rados.0.smithi107.stdout:5770: done (2 left) 2020-06-12T02:38:24.888 INFO:tasks.rados.rados.0.smithi107.stdout:5773: done (1 left) 2020-06-12T02:38:24.888 INFO:tasks.rados.rados.0.smithi107.stdout:5774: delete oid 8 current snap is 0 2020-06-12T02:38:24.888 INFO:tasks.rados.rados.0.smithi107.stdout:5774: done (1 left) 2020-06-12T02:38:24.888 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:24.889 INFO:tasks.rados.rados.0.smithi107.stdout:5775: seq_num 5774 ranges {155648=712704} 2020-06-12T02:38:24.890 INFO:tasks.rados.rados.0.smithi107.stdout:5775: writing smithi10729407-3 from 155648 to 868352 tid 1 2020-06-12T02:38:24.890 INFO:tasks.rados.rados.0.smithi107.stdout:5776: delete oid 16 current snap is 0 2020-06-12T02:38:24.896 INFO:tasks.rados.rados.0.smithi107.stdout:5775: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:24.898 INFO:tasks.rados.rados.0.smithi107.stdout:5776: done (2 left) 2020-06-12T02:38:24.898 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:24.898 INFO:tasks.rados.rados.0.smithi107.stdout:5777: seq_num 5776 ranges {0=745472} 2020-06-12T02:38:24.903 INFO:tasks.rados.rados.0.smithi107.stdout:5777: writing smithi10729407-17 from 0 to 745472 tid 1 2020-06-12T02:38:24.904 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:24.904 INFO:tasks.rados.rados.0.smithi107.stdout:5778: seq_num 5777 ranges {0=761856} 2020-06-12T02:38:24.909 INFO:tasks.rados.rados.0.smithi107.stdout:5778: writing smithi10729407-1 from 0 to 761856 tid 1 2020-06-12T02:38:24.909 INFO:tasks.rados.rados.0.smithi107.stdout:5779: delete oid 4 current snap is 0 2020-06-12T02:38:24.910 INFO:tasks.rados.rados.0.smithi107.stdout:5775: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:24.910 INFO:tasks.rados.rados.0.smithi107.stdout:5775: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:24.910 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1037 (ObjNum 5774 snap 0 seq_num 5774) dirty exists 2020-06-12T02:38:24.911 INFO:tasks.rados.rados.0.smithi107.stdout:5775: left oid 3 (ObjNum 5774 snap 0 seq_num 5774) 2020-06-12T02:38:24.911 INFO:tasks.rados.rados.0.smithi107.stdout:5777: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:24.912 INFO:tasks.rados.rados.0.smithi107.stdout:5777: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:24.912 INFO:tasks.rados.rados.0.smithi107.stdout:5777: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:24.913 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1076 (ObjNum 5776 snap 0 seq_num 5776) dirty exists 2020-06-12T02:38:24.913 INFO:tasks.rados.rados.0.smithi107.stdout:5777: left oid 17 (ObjNum 5776 snap 0 seq_num 5776) 2020-06-12T02:38:24.914 INFO:tasks.rados.rados.0.smithi107.stdout:5775: done (4 left) 2020-06-12T02:38:24.914 INFO:tasks.rados.rados.0.smithi107.stdout:5777: done (3 left) 2020-06-12T02:38:24.914 INFO:tasks.rados.rados.0.smithi107.stdout:5779: done (2 left) 2020-06-12T02:38:24.915 INFO:tasks.rados.rados.0.smithi107.stdout:5780: delete oid 11 current snap is 0 2020-06-12T02:38:24.915 INFO:tasks.rados.rados.0.smithi107.stdout:5780: done (2 left) 2020-06-12T02:38:24.916 INFO:tasks.rados.rados.0.smithi107.stdout:5781: delete oid 3 current snap is 0 2020-06-12T02:38:24.916 INFO:tasks.rados.rados.0.smithi107.stdout:5778: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:24.918 INFO:tasks.rados.rados.0.smithi107.stdout:5781: done (2 left) 2020-06-12T02:38:24.918 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:24.919 INFO:tasks.rados.rados.0.smithi107.stdout:5782: seq_num 5781 ranges {0=425984,425984=401408} 2020-06-12T02:38:24.920 INFO:tasks.rados.rados.0.smithi107.stdout:5782: writing smithi10729407-12 from 0 to 425984 tid 1 2020-06-12T02:38:24.923 INFO:tasks.rados.rados.0.smithi107.stdout:5782: writing smithi10729407-12 from 425984 to 827392 tid 2 2020-06-12T02:38:24.923 INFO:tasks.rados.rados.0.smithi107.stdout:5783: delete oid 18 current snap is 0 2020-06-12T02:38:24.924 INFO:tasks.rados.rados.0.smithi107.stdout:5778: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:24.924 INFO:tasks.rados.rados.0.smithi107.stdout:5778: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:24.924 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1073 (ObjNum 5777 snap 0 seq_num 5777) dirty exists 2020-06-12T02:38:24.925 INFO:tasks.rados.rados.0.smithi107.stdout:5778: left oid 1 (ObjNum 5777 snap 0 seq_num 5777) 2020-06-12T02:38:24.927 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:24.925+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:24.928 INFO:tasks.rados.rados.0.smithi107.stdout:5778: done (3 left) 2020-06-12T02:38:24.928 INFO:tasks.rados.rados.0.smithi107.stdout:5783: done (2 left) 2020-06-12T02:38:24.928 INFO:tasks.rados.rados.0.smithi107.stdout:5784: delete oid 15 current snap is 0 2020-06-12T02:38:24.929 INFO:tasks.rados.rados.0.smithi107.stdout:5782: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:24.931 INFO:tasks.rados.rados.0.smithi107.stdout:5782: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:24.932 INFO:tasks.rados.rados.0.smithi107.stdout:5782: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:24.934 INFO:tasks.rados.rados.0.smithi107.stdout:5782: finishing write tid 5 to smithi10729407-12 2020-06-12T02:38:24.935 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1028 (ObjNum 5781 snap 0 seq_num 5781) dirty exists 2020-06-12T02:38:24.935 INFO:tasks.rados.rados.0.smithi107.stdout:5782: left oid 12 (ObjNum 5781 snap 0 seq_num 5781) 2020-06-12T02:38:24.935 INFO:tasks.rados.rados.0.smithi107.stdout:5782: done (2 left) 2020-06-12T02:38:24.935 INFO:tasks.rados.rados.0.smithi107.stdout:5784: done (1 left) 2020-06-12T02:38:24.936 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:24.936 INFO:tasks.rados.rados.0.smithi107.stdout:5785: seq_num 5784 ranges {0=475136,475136=401408} 2020-06-12T02:38:24.937 INFO:tasks.rados.rados.0.smithi107.stdout:5785: writing smithi10729407-15 from 0 to 475136 tid 1 2020-06-12T02:38:24.939 INFO:tasks.rados.rados.0.smithi107.stdout:5785: writing smithi10729407-15 from 475136 to 876544 tid 2 2020-06-12T02:38:24.939 INFO:tasks.rados.rados.0.smithi107.stdout:5786: delete oid 4 current snap is 0 2020-06-12T02:38:24.944 INFO:tasks.rados.rados.0.smithi107.stdout:5786: done (2 left) 2020-06-12T02:38:24.945 INFO:tasks.rados.rados.0.smithi107.stdout:5787: delete oid 18 current snap is 0 2020-06-12T02:38:24.945 INFO:tasks.rados.rados.0.smithi107.stdout:5785: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:24.946 INFO:tasks.rados.rados.0.smithi107.stdout:5787: done (2 left) 2020-06-12T02:38:24.947 INFO:tasks.rados.rados.0.smithi107.stdout:5788: delete oid 8 current snap is 0 2020-06-12T02:38:24.949 INFO:tasks.rados.rados.0.smithi107.stdout:5785: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:24.950 INFO:tasks.rados.rados.0.smithi107.stdout:5785: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:24.953 INFO:tasks.rados.rados.0.smithi107.stdout:5785: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:24.954 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1057 (ObjNum 5784 snap 0 seq_num 5784) dirty exists 2020-06-12T02:38:24.954 INFO:tasks.rados.rados.0.smithi107.stdout:5785: left oid 15 (ObjNum 5784 snap 0 seq_num 5784) 2020-06-12T02:38:24.954 INFO:tasks.rados.rados.0.smithi107.stdout:5785: done (2 left) 2020-06-12T02:38:24.954 INFO:tasks.rados.rados.0.smithi107.stdout:5788: done (1 left) 2020-06-12T02:38:24.955 INFO:tasks.rados.rados.0.smithi107.stdout:5789: delete oid 15 current snap is 0 2020-06-12T02:38:24.958 INFO:tasks.rados.rados.0.smithi107.stdout:5789: done (1 left) 2020-06-12T02:38:24.958 INFO:tasks.rados.rados.0.smithi107.stdout:5790: delete oid 15 current snap is 0 2020-06-12T02:38:24.963 INFO:tasks.rados.rados.0.smithi107.stdout:5790: done (1 left) 2020-06-12T02:38:24.964 INFO:tasks.rados.rados.0.smithi107.stdout:5791: delete oid 8 current snap is 0 2020-06-12T02:38:24.966 INFO:tasks.rados.rados.0.smithi107.stdout:5791: done (1 left) 2020-06-12T02:38:24.966 INFO:tasks.rados.rados.0.smithi107.stdout:5792: delete oid 5 current snap is 0 2020-06-12T02:38:24.968 INFO:tasks.rados.rados.0.smithi107.stdout:5792: done (1 left) 2020-06-12T02:38:24.969 INFO:tasks.rados.rados.0.smithi107.stdout:5793: delete oid 5 current snap is 0 2020-06-12T02:38:24.974 INFO:tasks.rados.rados.0.smithi107.stdout:5793: done (1 left) 2020-06-12T02:38:24.974 INFO:tasks.rados.rados.0.smithi107.stdout:5794: delete oid 19 current snap is 0 2020-06-12T02:38:24.976 INFO:tasks.rados.rados.0.smithi107.stdout:5794: done (1 left) 2020-06-12T02:38:24.977 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:38:24.977 INFO:tasks.rados.rados.0.smithi107.stdout:5795: seq_num 5794 ranges {0=647168,647168=401408} 2020-06-12T02:38:24.980 INFO:tasks.rados.rados.0.smithi107.stdout:5795: writing smithi10729407-2 from 0 to 647168 tid 1 2020-06-12T02:38:24.982 INFO:tasks.rados.rados.0.smithi107.stdout:5795: writing smithi10729407-2 from 647168 to 1048576 tid 2 2020-06-12T02:38:24.983 INFO:tasks.rados.rados.0.smithi107.stdout:5796: delete oid 4 current snap is 0 2020-06-12T02:38:24.983 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:24.984 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:24.984 INFO:tasks.rados.rados.0.smithi107.stdout:5796: done (2 left) 2020-06-12T02:38:24.985 INFO:tasks.rados.rados.0.smithi107.stdout:5797: delete oid 1 current snap is 0 2020-06-12T02:38:24.987 INFO:tasks.rados.rados.0.smithi107.stdout:5795: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:24.988 INFO:tasks.rados.rados.0.smithi107.stdout:5797: done (2 left) 2020-06-12T02:38:24.988 INFO:tasks.rados.rados.0.smithi107.stdout:5798: delete oid 17 current snap is 0 2020-06-12T02:38:24.992 INFO:tasks.rados.rados.0.smithi107.stdout:5798: done (2 left) 2020-06-12T02:38:24.993 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:24.993 INFO:tasks.rados.rados.0.smithi107.stdout:5799: seq_num 5798 ranges {0=712704,712704=401408} 2020-06-12T02:38:24.995 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-06-12T02:38:24.997 INFO:tasks.rados.rados.0.smithi107.stdout:5799: writing smithi10729407-4 from 0 to 712704 tid 1 2020-06-12T02:38:25.000 INFO:tasks.rados.rados.0.smithi107.stdout:5799: writing smithi10729407-4 from 712704 to 1114112 tid 2 2020-06-12T02:38:25.000 INFO:tasks.rados.rados.0.smithi107.stdout:5800: delete oid 19 current snap is 0 2020-06-12T02:38:25.001 INFO:tasks.rados.rados.0.smithi107.stdout:5795: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:25.001 INFO:tasks.rados.rados.0.smithi107.stdout:5795: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:25.002 INFO:tasks.rados.rados.0.smithi107.stdout:5795: finishing write tid 5 to smithi10729407-2 2020-06-12T02:38:25.002 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 549 (ObjNum 5794 snap 0 seq_num 5794) dirty exists 2020-06-12T02:38:25.002 INFO:tasks.rados.rados.0.smithi107.stdout:5795: left oid 2 (ObjNum 5794 snap 0 seq_num 5794) 2020-06-12T02:38:25.003 INFO:tasks.rados.rados.0.smithi107.stdout:5795: done (3 left) 2020-06-12T02:38:25.003 INFO:tasks.rados.rados.0.smithi107.stdout:5800: done (2 left) 2020-06-12T02:38:25.003 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:38:25.004 INFO:tasks.rados.rados.0.smithi107.stdout:5801: seq_num 5800 ranges {622592=745472,1368064=401408} 2020-06-12T02:38:25.010 INFO:tasks.rados.rados.0.smithi107.stdout:5801: writing smithi10729407-6 from 622592 to 1368064 tid 1 2020-06-12T02:38:25.012 INFO:tasks.rados.rados.0.smithi107.stdout:5801: writing smithi10729407-6 from 1368064 to 1769472 tid 2 2020-06-12T02:38:25.012 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:25.012 INFO:tasks.rados.rados.0.smithi107.stdout:5802: seq_num 5801 ranges {573440=761856,1335296=401408} 2020-06-12T02:38:25.019 INFO:tasks.rados.rados.0.smithi107.stdout:5802: writing smithi10729407-9 from 573440 to 1335296 tid 1 2020-06-12T02:38:25.022 INFO:tasks.rados.rados.0.smithi107.stdout:5802: writing smithi10729407-9 from 1335296 to 1736704 tid 2 2020-06-12T02:38:25.028 INFO:tasks.rados.rados.0.smithi107.stdout:5803: delete oid 19 current snap is 0 2020-06-12T02:38:25.029 INFO:tasks.rados.rados.0.smithi107.stdout:5799: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:25.030 INFO:tasks.rados.rados.0.smithi107.stdout:5799: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:25.030 INFO:tasks.rados.rados.0.smithi107.stdout:5799: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:25.030 INFO:tasks.rados.rados.0.smithi107.stdout:5799: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:25.031 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1083 (ObjNum 5798 snap 0 seq_num 5798) dirty exists 2020-06-12T02:38:25.031 INFO:tasks.rados.rados.0.smithi107.stdout:5799: left oid 4 (ObjNum 5798 snap 0 seq_num 5798) 2020-06-12T02:38:25.031 INFO:tasks.rados.rados.0.smithi107.stdout:5801: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:25.031 INFO:tasks.rados.rados.0.smithi107.stdout:5801: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:25.032 INFO:tasks.rados.rados.0.smithi107.stdout:5801: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:25.032 INFO:tasks.rados.rados.0.smithi107.stdout:5801: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:25.032 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1077 (ObjNum 5800 snap 0 seq_num 5800) dirty exists 2020-06-12T02:38:25.032 INFO:tasks.rados.rados.0.smithi107.stdout:5801: left oid 6 (ObjNum 5800 snap 0 seq_num 5800) 2020-06-12T02:38:25.032 INFO:tasks.rados.rados.0.smithi107.stdout:5799: done (4 left) 2020-06-12T02:38:25.033 INFO:tasks.rados.rados.0.smithi107.stdout:5801: done (3 left) 2020-06-12T02:38:25.033 INFO:tasks.rados.rados.0.smithi107.stdout:5803: done (2 left) 2020-06-12T02:38:25.033 INFO:tasks.rados.rados.0.smithi107.stdout:5804: delete oid 1 current snap is 0 2020-06-12T02:38:25.033 INFO:tasks.rados.rados.0.smithi107.stdout:5804: done (2 left) 2020-06-12T02:38:25.033 INFO:tasks.rados.rados.0.smithi107.stdout:5805: delete oid 17 current snap is 0 2020-06-12T02:38:25.034 INFO:tasks.rados.rados.0.smithi107.stdout:5802: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:25.034 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:25.027+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:25.035 INFO:tasks.rados.rados.0.smithi107.stdout:5802: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:25.035 INFO:tasks.rados.rados.0.smithi107.stdout:5802: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:25.035 INFO:tasks.rados.rados.0.smithi107.stdout:5802: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:25.035 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1052 (ObjNum 5801 snap 0 seq_num 5801) dirty exists 2020-06-12T02:38:25.036 INFO:tasks.rados.rados.0.smithi107.stdout:5802: left oid 9 (ObjNum 5801 snap 0 seq_num 5801) 2020-06-12T02:38:25.036 INFO:tasks.rados.rados.0.smithi107.stdout:5802: done (2 left) 2020-06-12T02:38:25.036 INFO:tasks.rados.rados.0.smithi107.stdout:5805: done (1 left) 2020-06-12T02:38:25.036 INFO:tasks.rados.rados.0.smithi107.stdout:5806: delete oid 4 current snap is 0 2020-06-12T02:38:25.041 INFO:tasks.rados.rados.0.smithi107.stdout:5806: done (1 left) 2020-06-12T02:38:25.041 INFO:tasks.rados.rados.0.smithi107.stdout:5807: delete oid 16 current snap is 0 2020-06-12T02:38:25.045 INFO:tasks.rados.rados.0.smithi107.stdout:5807: done (1 left) 2020-06-12T02:38:25.046 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:25.046 INFO:tasks.rados.rados.0.smithi107.stdout:5808: seq_num 5807 ranges {1736704=458752,2195456=630784,2826240=172032} 2020-06-12T02:38:25.057 INFO:tasks.rados.rados.0.smithi107.stdout:5808: writing smithi10729407-9 from 1736704 to 2195456 tid 1 2020-06-12T02:38:25.061 INFO:tasks.rados.rados.0.smithi107.stdout:5808: writing smithi10729407-9 from 2195456 to 2826240 tid 2 2020-06-12T02:38:25.062 INFO:tasks.rados.rados.0.smithi107.stdout:5808: writing smithi10729407-9 from 2826240 to 2998272 tid 3 2020-06-12T02:38:25.062 INFO:tasks.rados.rados.0.smithi107.stdout:5809: delete oid 16 current snap is 0 2020-06-12T02:38:25.065 INFO:tasks.rados.rados.0.smithi107.stdout:5809: done (2 left) 2020-06-12T02:38:25.065 INFO:tasks.rados.rados.0.smithi107.stdout:5810: delete oid 19 current snap is 0 2020-06-12T02:38:25.066 INFO:tasks.rados.rados.0.smithi107.stdout:5808: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:25.069 INFO:tasks.rados.rados.0.smithi107.stdout:5810: done (2 left) 2020-06-12T02:38:25.069 INFO:tasks.rados.rados.0.smithi107.stdout:5811: delete oid 15 current snap is 0 2020-06-12T02:38:25.069 INFO:tasks.rados.rados.0.smithi107.stdout:5808: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:25.070 INFO:tasks.rados.rados.0.smithi107.stdout:5808: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:25.071 INFO:tasks.rados.rados.0.smithi107.stdout:5811: done (2 left) 2020-06-12T02:38:25.071 INFO:tasks.rados.rados.0.smithi107.stdout:5812: delete oid 18 current snap is 0 2020-06-12T02:38:25.072 INFO:tasks.rados.rados.0.smithi107.stdout:5808: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:25.073 INFO:tasks.rados.rados.0.smithi107.stdout:5808: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:25.073 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1056 (ObjNum 5807 snap 0 seq_num 5807) dirty exists 2020-06-12T02:38:25.074 INFO:tasks.rados.rados.0.smithi107.stdout:5808: left oid 9 (ObjNum 5807 snap 0 seq_num 5807) 2020-06-12T02:38:25.074 INFO:tasks.rados.rados.0.smithi107.stdout:5808: done (2 left) 2020-06-12T02:38:25.074 INFO:tasks.rados.rados.0.smithi107.stdout:5812: done (1 left) 2020-06-12T02:38:25.075 INFO:tasks.rados.rados.0.smithi107.stdout:5813: delete oid 4 current snap is 0 2020-06-12T02:38:25.076 INFO:tasks.rados.rados.0.smithi107.stdout:5813: done (1 left) 2020-06-12T02:38:25.076 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:38:25.076 INFO:tasks.rados.rados.0.smithi107.stdout:5814: seq_num 5813 ranges {0=565248,565248=786432,1351680=16384} 2020-06-12T02:38:25.079 INFO:tasks.rados.rados.0.smithi107.stdout:5814: writing smithi10729407-7 from 0 to 565248 tid 1 2020-06-12T02:38:25.084 INFO:tasks.rados.rados.0.smithi107.stdout:5814: writing smithi10729407-7 from 565248 to 1351680 tid 2 2020-06-12T02:38:25.084 INFO:tasks.rados.rados.0.smithi107.stdout:5814: writing smithi10729407-7 from 1351680 to 1368064 tid 3 2020-06-12T02:38:25.084 INFO:tasks.rados.rados.0.smithi107.stdout:5815: delete oid 2 current snap is 0 2020-06-12T02:38:25.085 INFO:tasks.rados.rados.0.smithi107.stdout:5814: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:25.086 INFO:tasks.rados.rados.0.smithi107.stdout:5815: done (2 left) 2020-06-12T02:38:25.086 INFO:tasks.rados.rados.0.smithi107.stdout:5816: delete oid 11 current snap is 0 2020-06-12T02:38:25.089 INFO:tasks.rados.rados.0.smithi107.stdout:5816: done (2 left) 2020-06-12T02:38:25.089 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:25.089 INFO:tasks.rados.rados.0.smithi107.stdout:5817: seq_num 5816 ranges {1769472=614400,2383872=483328,2867200=319488} 2020-06-12T02:38:25.102 INFO:tasks.rados.rados.0.smithi107.stdout:5817: writing smithi10729407-6 from 1769472 to 2383872 tid 1 2020-06-12T02:38:25.105 INFO:tasks.rados.rados.0.smithi107.stdout:5817: writing smithi10729407-6 from 2383872 to 2867200 tid 2 2020-06-12T02:38:25.108 INFO:tasks.rados.rados.0.smithi107.stdout:5817: writing smithi10729407-6 from 2867200 to 3186688 tid 3 2020-06-12T02:38:25.108 INFO:tasks.rados.rados.0.smithi107.stdout:5818: delete oid 16 current snap is 0 2020-06-12T02:38:25.108 INFO:tasks.rados.rados.0.smithi107.stdout:5814: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:25.108 INFO:tasks.rados.rados.0.smithi107.stdout:5814: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:25.109 INFO:tasks.rados.rados.0.smithi107.stdout:5814: finishing write tid 5 to smithi10729407-7 2020-06-12T02:38:25.109 INFO:tasks.rados.rados.0.smithi107.stdout:5814: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:25.109 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1103 (ObjNum 5813 snap 0 seq_num 5813) dirty exists 2020-06-12T02:38:25.109 INFO:tasks.rados.rados.0.smithi107.stdout:5814: left oid 7 (ObjNum 5813 snap 0 seq_num 5813) 2020-06-12T02:38:25.110 INFO:tasks.rados.rados.0.smithi107.stdout:5817: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:25.111 INFO:tasks.rados.rados.0.smithi107.stdout:5814: done (3 left) 2020-06-12T02:38:25.111 INFO:tasks.rados.rados.0.smithi107.stdout:5818: done (2 left) 2020-06-12T02:38:25.111 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:25.112 INFO:tasks.rados.rados.0.smithi107.stdout:5819: seq_num 5818 ranges {1368064=647168,2015232=655360,2670592=147456} 2020-06-12T02:38:25.120 INFO:tasks.rados.rados.0.smithi107.stdout:5819: writing smithi10729407-7 from 1368064 to 2015232 tid 1 2020-06-12T02:38:25.134 INFO:tasks.rados.rados.0.smithi107.stdout:5819: writing smithi10729407-7 from 2015232 to 2670592 tid 2 2020-06-12T02:38:25.135 INFO:tasks.rados.rados.0.smithi107.stdout:5819: writing smithi10729407-7 from 2670592 to 2818048 tid 3 2020-06-12T02:38:25.135 INFO:tasks.rados.rados.0.smithi107.stdout:5820: delete oid 17 current snap is 0 2020-06-12T02:38:25.136 INFO:tasks.rados.rados.0.smithi107.stdout:5817: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:25.136 INFO:tasks.rados.rados.0.smithi107.stdout:5817: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:25.136 INFO:tasks.rados.rados.0.smithi107.stdout:5817: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:25.136 INFO:tasks.rados.rados.0.smithi107.stdout:5817: finishing write tid 6 to smithi10729407-6 2020-06-12T02:38:25.137 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1082 (ObjNum 5816 snap 0 seq_num 5816) dirty exists 2020-06-12T02:38:25.137 INFO:tasks.rados.rados.0.smithi107.stdout:5817: left oid 6 (ObjNum 5816 snap 0 seq_num 5816) 2020-06-12T02:38:25.137 INFO:tasks.rados.rados.0.smithi107.stdout:5817: done (3 left) 2020-06-12T02:38:25.137 INFO:tasks.rados.rados.0.smithi107.stdout:5820: done (2 left) 2020-06-12T02:38:25.137 INFO:tasks.rados.rados.0.smithi107.stdout:5821: delete oid 1 current snap is 0 2020-06-12T02:38:25.138 INFO:tasks.rados.rados.0.smithi107.stdout:5819: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:25.138 INFO:tasks.rados.rados.0.smithi107.stdout:5821: done (2 left) 2020-06-12T02:38:25.138 INFO:tasks.rados.rados.0.smithi107.stdout:5822: delete oid 13 current snap is 0 2020-06-12T02:38:25.138 INFO:tasks.rados.rados.0.smithi107.stdout:5819: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:25.138 INFO:tasks.rados.rados.0.smithi107.stdout:5819: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:25.139 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:25.139 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:25.140 INFO:tasks.rados.rados.0.smithi107.stdout:5819: finishing write tid 5 to smithi10729407-7 2020-06-12T02:38:25.141 INFO:tasks.rados.rados.0.smithi107.stdout:5822: done (2 left) 2020-06-12T02:38:25.141 INFO:tasks.rados.rados.0.smithi107.stdout:5823: delete oid 11 current snap is 0 2020-06-12T02:38:25.143 INFO:tasks.rados.rados.0.smithi107.stdout:5819: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:25.143 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1107 (ObjNum 5818 snap 0 seq_num 5818) dirty exists 2020-06-12T02:38:25.143 INFO:tasks.rados.rados.0.smithi107.stdout:5819: left oid 7 (ObjNum 5818 snap 0 seq_num 5818) 2020-06-12T02:38:25.143 INFO:tasks.rados.rados.0.smithi107.stdout:5819: done (2 left) 2020-06-12T02:38:25.144 INFO:tasks.rados.rados.0.smithi107.stdout:5823: done (1 left) 2020-06-12T02:38:25.144 INFO:tasks.rados.rados.0.smithi107.stdout:5824: delete oid 13 current snap is 0 2020-06-12T02:38:25.144 INFO:tasks.rados.rados.0.smithi107.stdout:5824: done (1 left) 2020-06-12T02:38:25.144 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:25.144 INFO:tasks.rados.rados.0.smithi107.stdout:5825: seq_num 5824 ranges {2818048=745472,3563520=409600,3973120=393216} 2020-06-12T02:38:25.145 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:25.138+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:25.146 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-06-12T02:38:25.158 INFO:tasks.rados.rados.0.smithi107.stdout:5825: writing smithi10729407-7 from 2818048 to 3563520 tid 1 2020-06-12T02:38:25.160 INFO:tasks.rados.rados.0.smithi107.stdout:5825: writing smithi10729407-7 from 3563520 to 3973120 tid 2 2020-06-12T02:38:25.163 INFO:tasks.rados.rados.0.smithi107.stdout:5825: writing smithi10729407-7 from 3973120 to 4366336 tid 3 2020-06-12T02:38:25.163 INFO:tasks.rados.rados.0.smithi107.stdout:5826: delete oid 16 current snap is 0 2020-06-12T02:38:25.168 INFO:tasks.rados.rados.0.smithi107.stdout:5825: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:25.169 INFO:tasks.rados.rados.0.smithi107.stdout:5825: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:25.171 INFO:tasks.rados.rados.0.smithi107.stdout:5825: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:25.174 INFO:tasks.rados.rados.0.smithi107.stdout:5826: done (2 left) 2020-06-12T02:38:25.174 INFO:tasks.rados.rados.0.smithi107.stdout:5827: delete oid 9 current snap is 0 2020-06-12T02:38:25.174 INFO:tasks.rados.rados.0.smithi107.stdout:5825: finishing write tid 5 to smithi10729407-7 2020-06-12T02:38:25.175 INFO:tasks.rados.rados.0.smithi107.stdout:5825: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:25.175 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1113 (ObjNum 5824 snap 0 seq_num 5824) dirty exists 2020-06-12T02:38:25.176 INFO:tasks.rados.rados.0.smithi107.stdout:5825: left oid 7 (ObjNum 5824 snap 0 seq_num 5824) 2020-06-12T02:38:25.178 INFO:tasks.rados.rados.0.smithi107.stdout:5825: done (2 left) 2020-06-12T02:38:25.179 INFO:tasks.rados.rados.0.smithi107.stdout:5827: done (1 left) 2020-06-12T02:38:25.179 INFO:tasks.rados.rados.0.smithi107.stdout:5828: delete oid 15 current snap is 0 2020-06-12T02:38:25.180 INFO:tasks.rados.rados.0.smithi107.stdout:5828: done (1 left) 2020-06-12T02:38:25.180 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:25.181 INFO:tasks.rados.rados.0.smithi107.stdout:5829: seq_num 5828 ranges {0=409600,409600=745472,1155072=458752} 2020-06-12T02:38:25.181 INFO:tasks.rados.rados.0.smithi107.stdout:5829: writing smithi10729407-11 from 0 to 409600 tid 1 2020-06-12T02:38:25.186 INFO:tasks.rados.rados.0.smithi107.stdout:5829: writing smithi10729407-11 from 409600 to 1155072 tid 2 2020-06-12T02:38:25.189 INFO:tasks.rados.rados.0.smithi107.stdout:5829: writing smithi10729407-11 from 1155072 to 1613824 tid 3 2020-06-12T02:38:25.189 INFO:tasks.rados.rados.0.smithi107.stdout:5830: delete oid 1 current snap is 0 2020-06-12T02:38:25.191 INFO:tasks.rados.rados.0.smithi107.stdout:5829: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:25.210 INFO:tasks.rados.rados.0.smithi107.stdout:5829: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:25.211 INFO:tasks.rados.rados.0.smithi107.stdout:5830: done (2 left) 2020-06-12T02:38:25.211 INFO:tasks.rados.rados.0.smithi107.stdout:5831: delete oid 4 current snap is 0 2020-06-12T02:38:25.211 INFO:tasks.rados.rados.0.smithi107.stdout:5829: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:25.212 INFO:tasks.rados.rados.0.smithi107.stdout:5831: done (2 left) 2020-06-12T02:38:25.212 INFO:tasks.rados.rados.0.smithi107.stdout:5832: delete oid 19 current snap is 0 2020-06-12T02:38:25.212 INFO:tasks.rados.rados.0.smithi107.stdout:5829: finishing write tid 5 to smithi10729407-11 2020-06-12T02:38:25.212 INFO:tasks.rados.rados.0.smithi107.stdout:5829: finishing write tid 6 to smithi10729407-11 2020-06-12T02:38:25.213 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1037 (ObjNum 5828 snap 0 seq_num 5828) dirty exists 2020-06-12T02:38:25.213 INFO:tasks.rados.rados.0.smithi107.stdout:5829: left oid 11 (ObjNum 5828 snap 0 seq_num 5828) 2020-06-12T02:38:25.213 INFO:tasks.rados.rados.0.smithi107.stdout:5829: done (2 left) 2020-06-12T02:38:25.214 INFO:tasks.rados.rados.0.smithi107.stdout:5832: done (1 left) 2020-06-12T02:38:25.214 INFO:tasks.rados.rados.0.smithi107.stdout:5833: delete oid 9 current snap is 0 2020-06-12T02:38:25.214 INFO:tasks.rados.rados.0.smithi107.stdout:5833: done (1 left) 2020-06-12T02:38:25.215 INFO:tasks.rados.rados.0.smithi107.stdout:5834: delete oid 1 current snap is 0 2020-06-12T02:38:25.215 INFO:tasks.rados.rados.0.smithi107.stdout:5834: done (1 left) 2020-06-12T02:38:25.215 INFO:tasks.rados.rados.0.smithi107.stdout:5835: delete oid 12 current snap is 0 2020-06-12T02:38:25.216 INFO:tasks.rados.rados.0.smithi107.stdout:5835: done (1 left) 2020-06-12T02:38:25.216 INFO:tasks.rados.rados.0.smithi107.stdout:5836: delete oid 18 current snap is 0 2020-06-12T02:38:25.217 INFO:tasks.rados.rados.0.smithi107.stdout:5836: done (1 left) 2020-06-12T02:38:25.217 INFO:tasks.rados.rados.0.smithi107.stdout:5837: delete oid 14 current snap is 0 2020-06-12T02:38:25.218 INFO:tasks.rados.rados.0.smithi107.stdout:5837: done (1 left) 2020-06-12T02:38:25.219 INFO:tasks.rados.rados.0.smithi107.stdout:5838: delete oid 1 current snap is 0 2020-06-12T02:38:25.219 INFO:tasks.rados.rados.0.smithi107.stdout:5838: done (1 left) 2020-06-12T02:38:25.220 INFO:tasks.rados.rados.0.smithi107.stdout:5839: delete oid 17 current snap is 0 2020-06-12T02:38:25.221 INFO:tasks.rados.rados.0.smithi107.stdout:5839: done (1 left) 2020-06-12T02:38:25.221 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:25.221 INFO:tasks.rados.rados.0.smithi107.stdout:5840: seq_num 5839 ranges {0=598016,598016=778240,1376256=425984} 2020-06-12T02:38:25.224 INFO:tasks.rados.rados.0.smithi107.stdout:5840: writing smithi10729407-18 from 0 to 598016 tid 1 2020-06-12T02:38:25.229 INFO:tasks.rados.rados.0.smithi107.stdout:5840: writing smithi10729407-18 from 598016 to 1376256 tid 2 2020-06-12T02:38:25.232 INFO:tasks.rados.rados.0.smithi107.stdout:5840: writing smithi10729407-18 from 1376256 to 1802240 tid 3 2020-06-12T02:38:25.232 INFO:tasks.rados.rados.0.smithi107.stdout:5841: delete oid 13 current snap is 0 2020-06-12T02:38:25.232 INFO:tasks.rados.rados.0.smithi107.stdout:5840: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:25.234 INFO:tasks.rados.rados.0.smithi107.stdout:5841: done (2 left) 2020-06-12T02:38:25.234 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:25.235 INFO:tasks.rados.rados.0.smithi107.stdout:5842: seq_num 5841 ranges {0=630784,630784=540672,1171456=450560,1622016=212992} 2020-06-12T02:38:25.240 INFO:tasks.rados.rados.0.smithi107.stdout:5842: writing smithi10729407-1 from 0 to 630784 tid 1 2020-06-12T02:38:25.241 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:25.238+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:25.242 INFO:tasks.rados.rados.0.smithi107.stdout:5842: writing smithi10729407-1 from 630784 to 1171456 tid 2 2020-06-12T02:38:25.245 INFO:tasks.rados.rados.0.smithi107.stdout:5842: writing smithi10729407-1 from 1171456 to 1622016 tid 3 2020-06-12T02:38:25.247 INFO:tasks.rados.rados.0.smithi107.stdout:5842: writing smithi10729407-1 from 1622016 to 1835008 tid 4 2020-06-12T02:38:25.247 INFO:tasks.rados.rados.0.smithi107.stdout:5843: delete oid 15 current snap is 0 2020-06-12T02:38:25.247 INFO:tasks.rados.rados.0.smithi107.stdout:5840: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:25.248 INFO:tasks.rados.rados.0.smithi107.stdout:5840: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:25.248 INFO:tasks.rados.rados.0.smithi107.stdout:5840: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:25.248 INFO:tasks.rados.rados.0.smithi107.stdout:5840: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:25.248 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1064 (ObjNum 5839 snap 0 seq_num 5839) dirty exists 2020-06-12T02:38:25.249 INFO:tasks.rados.rados.0.smithi107.stdout:5840: left oid 18 (ObjNum 5839 snap 0 seq_num 5839) 2020-06-12T02:38:25.249 INFO:tasks.rados.rados.0.smithi107.stdout:5842: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:25.249 INFO:tasks.rados.rados.0.smithi107.stdout:5840: done (3 left) 2020-06-12T02:38:25.250 INFO:tasks.rados.rados.0.smithi107.stdout:5843: done (2 left) 2020-06-12T02:38:25.251 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:38:25.251 INFO:tasks.rados.rados.0.smithi107.stdout:5844: seq_num 5843 ranges {0=663552,663552=712704,1376256=466944,1843200=24576} 2020-06-12T02:38:25.253 INFO:tasks.rados.rados.0.smithi107.stdout:5844: writing smithi10729407-20 from 0 to 663552 tid 1 2020-06-12T02:38:25.258 INFO:tasks.rados.rados.0.smithi107.stdout:5844: writing smithi10729407-20 from 663552 to 1376256 tid 2 2020-06-12T02:38:25.261 INFO:tasks.rados.rados.0.smithi107.stdout:5844: writing smithi10729407-20 from 1376256 to 1843200 tid 3 2020-06-12T02:38:25.261 INFO:tasks.rados.rados.0.smithi107.stdout:5844: writing smithi10729407-20 from 1843200 to 1867776 tid 4 2020-06-12T02:38:25.262 INFO:tasks.rados.rados.0.smithi107.stdout:5845: delete oid 18 current snap is 0 2020-06-12T02:38:25.262 INFO:tasks.rados.rados.0.smithi107.stdout:5842: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:25.262 INFO:tasks.rados.rados.0.smithi107.stdout:5842: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:25.262 INFO:tasks.rados.rados.0.smithi107.stdout:5842: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:25.263 INFO:tasks.rados.rados.0.smithi107.stdout:5842: finishing write tid 6 to smithi10729407-1 2020-06-12T02:38:25.267 INFO:tasks.rados.rados.0.smithi107.stdout:5845: done (3 left) 2020-06-12T02:38:25.267 INFO:tasks.rados.rados.0.smithi107.stdout:5846: delete oid 18 current snap is 0 2020-06-12T02:38:25.267 INFO:tasks.rados.rados.0.smithi107.stdout:5844: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:25.268 INFO:tasks.rados.rados.0.smithi107.stdout:5844: finishing write tid 2 to smithi10729407-20 2020-06-12T02:38:25.269 INFO:tasks.rados.rados.0.smithi107.stdout:5842: finishing write tid 7 to smithi10729407-1 2020-06-12T02:38:25.269 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1091 (ObjNum 5841 snap 0 seq_num 5841) dirty exists 2020-06-12T02:38:25.269 INFO:tasks.rados.rados.0.smithi107.stdout:5842: left oid 1 (ObjNum 5841 snap 0 seq_num 5841) 2020-06-12T02:38:25.270 INFO:tasks.rados.rados.0.smithi107.stdout:5844: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:25.270 INFO:tasks.rados.rados.0.smithi107.stdout:5844: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:25.271 INFO:tasks.rados.rados.0.smithi107.stdout:5842: done (3 left) 2020-06-12T02:38:25.272 INFO:tasks.rados.rados.0.smithi107.stdout:5846: done (2 left) 2020-06-12T02:38:25.272 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:25.272 INFO:tasks.rados.rados.0.smithi107.stdout:5847: seq_num 5846 ranges {4366336=712704,5079040=761856,5840896=442368} 2020-06-12T02:38:25.298 INFO:tasks.rados.rados.0.smithi107.stdout:5847: writing smithi10729407-7 from 4366336 to 5079040 tid 1 2020-06-12T02:38:25.301 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:25.302 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:25.302 INFO:tasks.rados.rados.0.smithi107.stdout:5847: writing smithi10729407-7 from 5079040 to 5840896 tid 2 2020-06-12T02:38:25.305 INFO:tasks.rados.rados.0.smithi107.stdout:5847: writing smithi10729407-7 from 5840896 to 6283264 tid 3 2020-06-12T02:38:25.305 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:25.305 INFO:tasks.rados.rados.0.smithi107.stdout:5848: seq_num 5847 ranges {0=729088,729088=696320,1425408=450560,1875968=57344} 2020-06-12T02:38:25.309 INFO:tasks.rados.rados.0.smithi107.stdout:5848: writing smithi10729407-4 from 0 to 729088 tid 1 2020-06-12T02:38:25.311 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-06-12T02:38:25.313 INFO:tasks.rados.rados.0.smithi107.stdout:5848: writing smithi10729407-4 from 729088 to 1425408 tid 2 2020-06-12T02:38:25.316 INFO:tasks.rados.rados.0.smithi107.stdout:5848: writing smithi10729407-4 from 1425408 to 1875968 tid 3 2020-06-12T02:38:25.316 INFO:tasks.rados.rados.0.smithi107.stdout:5848: writing smithi10729407-4 from 1875968 to 1933312 tid 4 2020-06-12T02:38:25.317 INFO:tasks.rados.rados.0.smithi107.stdout:5849: delete oid 8 current snap is 0 2020-06-12T02:38:25.317 INFO:tasks.rados.rados.0.smithi107.stdout:5844: finishing write tid 6 to smithi10729407-20 2020-06-12T02:38:25.317 INFO:tasks.rados.rados.0.smithi107.stdout:5844: finishing write tid 7 to smithi10729407-20 2020-06-12T02:38:25.318 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 449 (ObjNum 5843 snap 0 seq_num 5843) dirty exists 2020-06-12T02:38:25.318 INFO:tasks.rados.rados.0.smithi107.stdout:5844: left oid 20 (ObjNum 5843 snap 0 seq_num 5843) 2020-06-12T02:38:25.318 INFO:tasks.rados.rados.0.smithi107.stdout:5847: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:25.319 INFO:tasks.rados.rados.0.smithi107.stdout:5847: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:25.319 INFO:tasks.rados.rados.0.smithi107.stdout:5847: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:25.319 INFO:tasks.rados.rados.0.smithi107.stdout:5847: finishing write tid 5 to smithi10729407-7 2020-06-12T02:38:25.323 INFO:tasks.rados.rados.0.smithi107.stdout:5847: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:25.323 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1118 (ObjNum 5846 snap 0 seq_num 5846) dirty exists 2020-06-12T02:38:25.324 INFO:tasks.rados.rados.0.smithi107.stdout:5847: left oid 7 (ObjNum 5846 snap 0 seq_num 5846) 2020-06-12T02:38:25.326 INFO:tasks.rados.rados.0.smithi107.stdout:5844: done (4 left) 2020-06-12T02:38:25.327 INFO:tasks.rados.rados.0.smithi107.stdout:5847: done (3 left) 2020-06-12T02:38:25.327 INFO:tasks.rados.rados.0.smithi107.stdout:5849: done (2 left) 2020-06-12T02:38:25.327 INFO:tasks.rados.rados.0.smithi107.stdout:5850: delete oid 12 current snap is 0 2020-06-12T02:38:25.328 INFO:tasks.rados.rados.0.smithi107.stdout:5848: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:25.328 INFO:tasks.rados.rados.0.smithi107.stdout:5850: done (2 left) 2020-06-12T02:38:25.328 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:25.329 INFO:tasks.rados.rados.0.smithi107.stdout:5851: seq_num 5850 ranges {0=786432,786432=753664,1540096=450560} 2020-06-12T02:38:25.329 INFO:tasks.rados.rados.0.smithi107.stdout:5851: writing smithi10729407-8 from 0 to 786432 tid 1 2020-06-12T02:38:25.330 INFO:tasks.rados.rados.0.smithi107.stdout:5851: writing smithi10729407-8 from 786432 to 1540096 tid 2 2020-06-12T02:38:25.333 INFO:tasks.rados.rados.0.smithi107.stdout:5851: writing smithi10729407-8 from 1540096 to 1990656 tid 3 2020-06-12T02:38:25.334 INFO:tasks.rados.rados.0.smithi107.stdout:5852: delete oid 14 current snap is 0 2020-06-12T02:38:25.335 INFO:tasks.rados.rados.0.smithi107.stdout:5848: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:25.335 INFO:tasks.rados.rados.0.smithi107.stdout:5848: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:25.335 INFO:tasks.rados.rados.0.smithi107.stdout:5848: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:25.336 INFO:tasks.rados.rados.0.smithi107.stdout:5848: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:25.337 INFO:tasks.rados.rados.0.smithi107.stdout:5848: finishing write tid 7 to smithi10729407-4 2020-06-12T02:38:25.337 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1094 (ObjNum 5847 snap 0 seq_num 5847) dirty exists 2020-06-12T02:38:25.338 INFO:tasks.rados.rados.0.smithi107.stdout:5848: left oid 4 (ObjNum 5847 snap 0 seq_num 5847) 2020-06-12T02:38:25.340 INFO:tasks.rados.rados.0.smithi107.stdout:5848: done (3 left) 2020-06-12T02:38:25.340 INFO:tasks.rados.rados.0.smithi107.stdout:5852: done (2 left) 2020-06-12T02:38:25.340 INFO:tasks.rados.rados.0.smithi107.stdout:5853: delete oid 6 current snap is 0 2020-06-12T02:38:25.343 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:25.340+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:25.343 INFO:tasks.rados.rados.0.smithi107.stdout:5851: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:25.344 INFO:tasks.rados.rados.0.smithi107.stdout:5851: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:25.344 INFO:tasks.rados.rados.0.smithi107.stdout:5851: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:25.344 INFO:tasks.rados.rados.0.smithi107.stdout:5853: done (2 left) 2020-06-12T02:38:25.345 INFO:tasks.rados.rados.0.smithi107.stdout:5854: delete oid 19 current snap is 0 2020-06-12T02:38:25.347 INFO:tasks.rados.rados.0.smithi107.stdout:5851: finishing write tid 5 to smithi10729407-8 2020-06-12T02:38:25.349 INFO:tasks.rados.rados.0.smithi107.stdout:5854: done (2 left) 2020-06-12T02:38:25.350 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:25.350 INFO:tasks.rados.rados.0.smithi107.stdout:5855: seq_num 5854 ranges {0=450560,450560=737280,1187840=598016,1785856=270336} 2020-06-12T02:38:25.351 INFO:tasks.rados.rados.0.smithi107.stdout:5855: writing smithi10729407-15 from 0 to 450560 tid 1 2020-06-12T02:38:25.356 INFO:tasks.rados.rados.0.smithi107.stdout:5855: writing smithi10729407-15 from 450560 to 1187840 tid 2 2020-06-12T02:38:25.358 INFO:tasks.rados.rados.0.smithi107.stdout:5855: writing smithi10729407-15 from 1187840 to 1785856 tid 3 2020-06-12T02:38:25.359 INFO:tasks.rados.rados.0.smithi107.stdout:5855: writing smithi10729407-15 from 1785856 to 2056192 tid 4 2020-06-12T02:38:25.359 INFO:tasks.rados.rados.0.smithi107.stdout:5856: delete oid 14 current snap is 0 2020-06-12T02:38:25.360 INFO:tasks.rados.rados.0.smithi107.stdout:5851: finishing write tid 6 to smithi10729407-8 2020-06-12T02:38:25.360 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1069 (ObjNum 5850 snap 0 seq_num 5850) dirty exists 2020-06-12T02:38:25.360 INFO:tasks.rados.rados.0.smithi107.stdout:5851: left oid 8 (ObjNum 5850 snap 0 seq_num 5850) 2020-06-12T02:38:25.360 INFO:tasks.rados.rados.0.smithi107.stdout:5855: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:25.363 INFO:tasks.rados.rados.0.smithi107.stdout:5851: done (3 left) 2020-06-12T02:38:25.363 INFO:tasks.rados.rados.0.smithi107.stdout:5856: done (2 left) 2020-06-12T02:38:25.363 INFO:tasks.rados.rados.0.smithi107.stdout:5857: delete oid 8 current snap is 0 2020-06-12T02:38:25.367 INFO:tasks.rados.rados.0.smithi107.stdout:5855: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:25.367 INFO:tasks.rados.rados.0.smithi107.stdout:5855: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:25.368 INFO:tasks.rados.rados.0.smithi107.stdout:5855: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:25.369 INFO:tasks.rados.rados.0.smithi107.stdout:5855: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:25.372 INFO:tasks.rados.rados.0.smithi107.stdout:5855: finishing write tid 7 to smithi10729407-15 2020-06-12T02:38:25.372 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1074 (ObjNum 5854 snap 0 seq_num 5854) dirty exists 2020-06-12T02:38:25.372 INFO:tasks.rados.rados.0.smithi107.stdout:5855: left oid 15 (ObjNum 5854 snap 0 seq_num 5854) 2020-06-12T02:38:25.373 INFO:tasks.rados.rados.0.smithi107.stdout:5855: done (2 left) 2020-06-12T02:38:25.373 INFO:tasks.rados.rados.0.smithi107.stdout:5857: done (1 left) 2020-06-12T02:38:25.373 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:25.373 INFO:tasks.rados.rados.0.smithi107.stdout:5858: seq_num 5857 ranges {1835008=499712,2334720=794624,3129344=778240,3907584=32768} 2020-06-12T02:38:25.385 INFO:tasks.rados.rados.0.smithi107.stdout:5858: writing smithi10729407-1 from 1835008 to 2334720 tid 1 2020-06-12T02:38:25.390 INFO:tasks.rados.rados.0.smithi107.stdout:5858: writing smithi10729407-1 from 2334720 to 3129344 tid 2 2020-06-12T02:38:25.394 INFO:tasks.rados.rados.0.smithi107.stdout:5858: writing smithi10729407-1 from 3129344 to 3907584 tid 3 2020-06-12T02:38:25.396 INFO:tasks.rados.rados.0.smithi107.stdout:5858: writing smithi10729407-1 from 3907584 to 3940352 tid 4 2020-06-12T02:38:25.397 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:25.398 INFO:tasks.rados.rados.0.smithi107.stdout:5859: seq_num 5858 ranges {0=516096,516096=679936,1196032=581632,1777664=344064} 2020-06-12T02:38:25.398 INFO:tasks.rados.rados.0.smithi107.stdout:5859: writing smithi10729407-14 from 0 to 516096 tid 1 2020-06-12T02:38:25.402 INFO:tasks.rados.rados.0.smithi107.stdout:5859: writing smithi10729407-14 from 516096 to 1196032 tid 2 2020-06-12T02:38:25.406 INFO:tasks.rados.rados.0.smithi107.stdout:5859: writing smithi10729407-14 from 1196032 to 1777664 tid 3 2020-06-12T02:38:25.408 INFO:tasks.rados.rados.0.smithi107.stdout:5859: writing smithi10729407-14 from 1777664 to 2121728 tid 4 2020-06-12T02:38:25.408 INFO:tasks.rados.rados.0.smithi107.stdout:5860: delete oid 15 current snap is 0 2020-06-12T02:38:25.408 INFO:tasks.rados.rados.0.smithi107.stdout:5858: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:25.409 INFO:tasks.rados.rados.0.smithi107.stdout:5858: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:25.409 INFO:tasks.rados.rados.0.smithi107.stdout:5859: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:25.409 INFO:tasks.rados.rados.0.smithi107.stdout:5858: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:25.411 INFO:tasks.rados.rados.0.smithi107.stdout:5860: done (3 left) 2020-06-12T02:38:25.411 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:25.411 INFO:tasks.rados.rados.0.smithi107.stdout:5861: seq_num 5860 ranges {1613824=548864,2162688=442368,2605056=548864,3153920=483328,3637248=131072} 2020-06-12T02:38:25.423 INFO:tasks.rados.rados.0.smithi107.stdout:5861: writing smithi10729407-11 from 1613824 to 2162688 tid 1 2020-06-12T02:38:25.426 INFO:tasks.rados.rados.0.smithi107.stdout:5861: writing smithi10729407-11 from 2162688 to 2605056 tid 2 2020-06-12T02:38:25.437 INFO:tasks.rados.rados.0.smithi107.stdout:5861: writing smithi10729407-11 from 2605056 to 3153920 tid 3 2020-06-12T02:38:25.438 INFO:tasks.rados.rados.0.smithi107.stdout:5861: writing smithi10729407-11 from 3153920 to 3637248 tid 4 2020-06-12T02:38:25.438 INFO:tasks.rados.rados.0.smithi107.stdout:5861: writing smithi10729407-11 from 3637248 to 3768320 tid 5 2020-06-12T02:38:25.438 INFO:tasks.rados.rados.0.smithi107.stdout:5862: delete oid 6 current snap is 0 2020-06-12T02:38:25.439 INFO:tasks.rados.rados.0.smithi107.stdout:5859: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:25.439 INFO:tasks.rados.rados.0.smithi107.stdout:5858: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:25.439 INFO:tasks.rados.rados.0.smithi107.stdout:5858: finishing write tid 6 to smithi10729407-1 2020-06-12T02:38:25.439 INFO:tasks.rados.rados.0.smithi107.stdout:5859: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:25.439 INFO:tasks.rados.rados.0.smithi107.stdout:5858: finishing write tid 7 to smithi10729407-1 2020-06-12T02:38:25.440 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1097 (ObjNum 5857 snap 0 seq_num 5857) dirty exists 2020-06-12T02:38:25.440 INFO:tasks.rados.rados.0.smithi107.stdout:5858: left oid 1 (ObjNum 5857 snap 0 seq_num 5857) 2020-06-12T02:38:25.440 INFO:tasks.rados.rados.0.smithi107.stdout:5859: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:25.440 INFO:tasks.rados.rados.0.smithi107.stdout:5859: finishing write tid 6 to smithi10729407-14 2020-06-12T02:38:25.440 INFO:tasks.rados.rados.0.smithi107.stdout:5859: finishing write tid 7 to smithi10729407-14 2020-06-12T02:38:25.440 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1045 (ObjNum 5858 snap 0 seq_num 5858) dirty exists 2020-06-12T02:38:25.441 INFO:tasks.rados.rados.0.smithi107.stdout:5859: left oid 14 (ObjNum 5858 snap 0 seq_num 5858) 2020-06-12T02:38:25.441 INFO:tasks.rados.rados.0.smithi107.stdout:5861: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:25.441 INFO:tasks.rados.rados.0.smithi107.stdout:5861: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:25.443 INFO:tasks.rados.rados.0.smithi107.stdout:5861: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:25.444 INFO:tasks.rados.rados.0.smithi107.stdout:5861: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:25.444 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:25.440+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:25.444 INFO:tasks.rados.rados.0.smithi107.stdout:5858: done (4 left) 2020-06-12T02:38:25.445 INFO:tasks.rados.rados.0.smithi107.stdout:5859: done (3 left) 2020-06-12T02:38:25.445 INFO:tasks.rados.rados.0.smithi107.stdout:5862: done (2 left) 2020-06-12T02:38:25.445 INFO:tasks.rados.rados.0.smithi107.stdout:5863: delete oid 4 current snap is 0 2020-06-12T02:38:25.446 INFO:tasks.rados.rados.0.smithi107.stdout:5861: finishing write tid 5 to smithi10729407-11 2020-06-12T02:38:25.446 INFO:tasks.rados.rados.0.smithi107.stdout:5861: finishing write tid 7 to smithi10729407-11 2020-06-12T02:38:25.447 INFO:tasks.rados.rados.0.smithi107.stdout:5863: done (2 left) 2020-06-12T02:38:25.447 INFO:tasks.rados.rados.0.smithi107.stdout:5864: delete oid 6 current snap is 0 2020-06-12T02:38:25.447 INFO:tasks.rados.rados.0.smithi107.stdout:5861: finishing write tid 8 to smithi10729407-11 2020-06-12T02:38:25.447 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1051 (ObjNum 5860 snap 0 seq_num 5860) dirty exists 2020-06-12T02:38:25.448 INFO:tasks.rados.rados.0.smithi107.stdout:5861: left oid 11 (ObjNum 5860 snap 0 seq_num 5860) 2020-06-12T02:38:25.451 INFO:tasks.rados.rados.0.smithi107.stdout:5861: done (2 left) 2020-06-12T02:38:25.451 INFO:tasks.rados.rados.0.smithi107.stdout:5864: done (1 left) 2020-06-12T02:38:25.451 INFO:tasks.rados.rados.0.smithi107.stdout:5865: delete oid 2 current snap is 0 2020-06-12T02:38:25.453 INFO:tasks.rados.rados.0.smithi107.stdout:5865: done (1 left) 2020-06-12T02:38:25.453 INFO:tasks.rados.rados.0.smithi107.stdout:5866: delete oid 2 current snap is 0 2020-06-12T02:38:25.455 INFO:tasks.rados.rados.0.smithi107.stdout:5866: done (1 left) 2020-06-12T02:38:25.455 INFO:tasks.rados.rados.0.smithi107.stdout:5867: delete oid 15 current snap is 0 2020-06-12T02:38:25.457 INFO:tasks.rados.rados.0.smithi107.stdout:5867: done (1 left) 2020-06-12T02:38:25.458 INFO:tasks.rados.rados.0.smithi107.stdout:5868: delete oid 2 current snap is 0 2020-06-12T02:38:25.458 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:25.458 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:25.459 INFO:tasks.rados.rados.0.smithi107.stdout:5868: done (1 left) 2020-06-12T02:38:25.459 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:25.459 INFO:tasks.rados.rados.0.smithi107.stdout:5869: seq_num 5868 ranges {0=679936,679936=770048,1449984=507904,1957888=327680} 2020-06-12T02:38:25.463 INFO:tasks.rados.rados.0.smithi107.stdout:5869: writing smithi10729407-12 from 0 to 679936 tid 1 2020-06-12T02:38:25.468 INFO:tasks.rados.rados.0.smithi107.stdout:5869: writing smithi10729407-12 from 679936 to 1449984 tid 2 2020-06-12T02:38:25.468 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-06-12T02:38:25.470 INFO:tasks.rados.rados.0.smithi107.stdout:5869: writing smithi10729407-12 from 1449984 to 1957888 tid 3 2020-06-12T02:38:25.472 INFO:tasks.rados.rados.0.smithi107.stdout:5869: writing smithi10729407-12 from 1957888 to 2285568 tid 4 2020-06-12T02:38:25.473 INFO:tasks.rados.rados.0.smithi107.stdout:5870: delete oid 8 current snap is 0 2020-06-12T02:38:25.473 INFO:tasks.rados.rados.0.smithi107.stdout:5869: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:25.477 INFO:tasks.rados.rados.0.smithi107.stdout:5870: done (2 left) 2020-06-12T02:38:25.478 INFO:tasks.rados.rados.0.smithi107.stdout:5871: delete oid 19 current snap is 0 2020-06-12T02:38:25.480 INFO:tasks.rados.rados.0.smithi107.stdout:5869: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:25.481 INFO:tasks.rados.rados.0.smithi107.stdout:5869: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:25.483 INFO:tasks.rados.rados.0.smithi107.stdout:5869: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:25.486 INFO:tasks.rados.rados.0.smithi107.stdout:5869: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:25.487 INFO:tasks.rados.rados.0.smithi107.stdout:5871: done (2 left) 2020-06-12T02:38:25.488 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:25.488 INFO:tasks.rados.rados.0.smithi107.stdout:5872: seq_num 5871 ranges {0=737280,737280=475136,1212416=729088,1941504=401408} 2020-06-12T02:38:25.491 INFO:tasks.rados.rados.0.smithi107.stdout:5872: writing smithi10729407-9 from 0 to 737280 tid 1 2020-06-12T02:38:25.494 INFO:tasks.rados.rados.0.smithi107.stdout:5872: writing smithi10729407-9 from 737280 to 1212416 tid 2 2020-06-12T02:38:25.513 INFO:tasks.rados.rados.0.smithi107.stdout:5872: writing smithi10729407-9 from 1212416 to 1941504 tid 3 2020-06-12T02:38:25.513 INFO:tasks.rados.rados.0.smithi107.stdout:5872: writing smithi10729407-9 from 1941504 to 2342912 tid 4 2020-06-12T02:38:25.514 INFO:tasks.rados.rados.0.smithi107.stdout:5873: delete oid 1 current snap is 0 2020-06-12T02:38:25.514 INFO:tasks.rados.rados.0.smithi107.stdout:5869: finishing write tid 7 to smithi10729407-12 2020-06-12T02:38:25.514 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1041 (ObjNum 5868 snap 0 seq_num 5868) dirty exists 2020-06-12T02:38:25.514 INFO:tasks.rados.rados.0.smithi107.stdout:5869: left oid 12 (ObjNum 5868 snap 0 seq_num 5868) 2020-06-12T02:38:25.514 INFO:tasks.rados.rados.0.smithi107.stdout:5872: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:25.515 INFO:tasks.rados.rados.0.smithi107.stdout:5869: done (3 left) 2020-06-12T02:38:25.515 INFO:tasks.rados.rados.0.smithi107.stdout:5873: done (2 left) 2020-06-12T02:38:25.515 INFO:tasks.rados.rados.0.smithi107.stdout:5874: delete oid 5 current snap is 0 2020-06-12T02:38:25.515 INFO:tasks.rados.rados.0.smithi107.stdout:5874: done (2 left) 2020-06-12T02:38:25.515 INFO:tasks.rados.rados.0.smithi107.stdout:5875: delete oid 5 current snap is 0 2020-06-12T02:38:25.516 INFO:tasks.rados.rados.0.smithi107.stdout:5875: done (2 left) 2020-06-12T02:38:25.516 INFO:tasks.rados.rados.0.smithi107.stdout:5876: delete oid 17 current snap is 0 2020-06-12T02:38:25.516 INFO:tasks.rados.rados.0.smithi107.stdout:5872: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:25.516 INFO:tasks.rados.rados.0.smithi107.stdout:5872: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:25.516 INFO:tasks.rados.rados.0.smithi107.stdout:5876: done (2 left) 2020-06-12T02:38:25.516 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:38:25.517 INFO:tasks.rados.rados.0.smithi107.stdout:5877: seq_num 5876 ranges {1867776=16384} 2020-06-12T02:38:25.530 INFO:tasks.rados.rados.0.smithi107.stdout:5877: writing smithi10729407-20 from 1867776 to 1884160 tid 1 2020-06-12T02:38:25.530 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:25.531 INFO:tasks.rados.rados.0.smithi107.stdout:5878: seq_num 5877 ranges {0=32768} 2020-06-12T02:38:25.531 INFO:tasks.rados.rados.0.smithi107.stdout:5878: writing smithi10729407-3 from 0 to 32768 tid 1 2020-06-12T02:38:25.531 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:25.531 INFO:tasks.rados.rados.0.smithi107.stdout:5879: seq_num 5878 ranges {0=49152} 2020-06-12T02:38:25.531 INFO:tasks.rados.rados.0.smithi107.stdout:5879: writing smithi10729407-19 from 0 to 49152 tid 1 2020-06-12T02:38:25.532 INFO:tasks.rados.rados.0.smithi107.stdout:5880: delete oid 15 current snap is 0 2020-06-12T02:38:25.532 INFO:tasks.rados.rados.0.smithi107.stdout:5872: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:25.532 INFO:tasks.rados.rados.0.smithi107.stdout:5872: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:25.532 INFO:tasks.rados.rados.0.smithi107.stdout:5872: finishing write tid 7 to smithi10729407-9 2020-06-12T02:38:25.532 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1071 (ObjNum 5871 snap 0 seq_num 5871) dirty exists 2020-06-12T02:38:25.533 INFO:tasks.rados.rados.0.smithi107.stdout:5872: left oid 9 (ObjNum 5871 snap 0 seq_num 5871) 2020-06-12T02:38:25.533 INFO:tasks.rados.rados.0.smithi107.stdout:5877: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:25.533 INFO:tasks.rados.rados.0.smithi107.stdout:5877: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:25.534 INFO:tasks.rados.rados.0.smithi107.stdout:5872: done (5 left) 2020-06-12T02:38:25.534 INFO:tasks.rados.rados.0.smithi107.stdout:5880: done (4 left) 2020-06-12T02:38:25.534 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:38:25.534 INFO:tasks.rados.rados.0.smithi107.stdout:5881: seq_num 5880 ranges {3768320=81920} 2020-06-12T02:38:25.544 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:25.542+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:25.549 INFO:tasks.rados.rados.0.smithi107.stdout:5881: writing smithi10729407-11 from 3768320 to 3850240 tid 1 2020-06-12T02:38:25.550 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:25.550 INFO:tasks.rados.rados.0.smithi107.stdout:5882: seq_num 5881 ranges {0=98304} 2020-06-12T02:38:25.551 INFO:tasks.rados.rados.0.smithi107.stdout:5882: writing smithi10729407-18 from 0 to 98304 tid 1 2020-06-12T02:38:25.551 INFO:tasks.rados.rados.0.smithi107.stdout:5883: delete oid 13 current snap is 0 2020-06-12T02:38:25.551 INFO:tasks.rados.rados.0.smithi107.stdout:5878: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:25.551 INFO:tasks.rados.rados.0.smithi107.stdout:5879: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:25.552 INFO:tasks.rados.rados.0.smithi107.stdout:5879: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:25.552 INFO:tasks.rados.rados.0.smithi107.stdout:5877: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:25.552 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 451 (ObjNum 5876 snap 0 seq_num 5876) dirty exists 2020-06-12T02:38:25.553 INFO:tasks.rados.rados.0.smithi107.stdout:5877: left oid 20 (ObjNum 5876 snap 0 seq_num 5876) 2020-06-12T02:38:25.553 INFO:tasks.rados.rados.0.smithi107.stdout:5879: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:25.553 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1044 (ObjNum 5878 snap 0 seq_num 5878) dirty exists 2020-06-12T02:38:25.553 INFO:tasks.rados.rados.0.smithi107.stdout:5879: left oid 19 (ObjNum 5878 snap 0 seq_num 5878) 2020-06-12T02:38:25.554 INFO:tasks.rados.rados.0.smithi107.stdout:5878: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:25.554 INFO:tasks.rados.rados.0.smithi107.stdout:5878: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:25.554 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1044 (ObjNum 5877 snap 0 seq_num 5877) dirty exists 2020-06-12T02:38:25.554 INFO:tasks.rados.rados.0.smithi107.stdout:5878: left oid 3 (ObjNum 5877 snap 0 seq_num 5877) 2020-06-12T02:38:25.555 INFO:tasks.rados.rados.0.smithi107.stdout:5877: done (6 left) 2020-06-12T02:38:25.555 INFO:tasks.rados.rados.0.smithi107.stdout:5878: done (5 left) 2020-06-12T02:38:25.555 INFO:tasks.rados.rados.0.smithi107.stdout:5879: done (4 left) 2020-06-12T02:38:25.556 INFO:tasks.rados.rados.0.smithi107.stdout:5883: done (3 left) 2020-06-12T02:38:25.556 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:38:25.556 INFO:tasks.rados.rados.0.smithi107.stdout:5884: seq_num 5883 ranges {1884160=131072} 2020-06-12T02:38:25.564 INFO:tasks.rados.rados.0.smithi107.stdout:5884: writing smithi10729407-20 from 1884160 to 2015232 tid 1 2020-06-12T02:38:25.565 INFO:tasks.rados.rados.0.smithi107.stdout:5885: delete oid 8 current snap is 0 2020-06-12T02:38:25.565 INFO:tasks.rados.rados.0.smithi107.stdout:5881: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:25.565 INFO:tasks.rados.rados.0.smithi107.stdout:5882: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:25.566 INFO:tasks.rados.rados.0.smithi107.stdout:5881: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:25.566 INFO:tasks.rados.rados.0.smithi107.stdout:5881: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:25.566 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1053 (ObjNum 5880 snap 0 seq_num 5880) dirty exists 2020-06-12T02:38:25.566 INFO:tasks.rados.rados.0.smithi107.stdout:5881: left oid 11 (ObjNum 5880 snap 0 seq_num 5880) 2020-06-12T02:38:25.566 INFO:tasks.rados.rados.0.smithi107.stdout:5882: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:25.566 INFO:tasks.rados.rados.0.smithi107.stdout:5882: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:25.567 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1073 (ObjNum 5881 snap 0 seq_num 5881) dirty exists 2020-06-12T02:38:25.567 INFO:tasks.rados.rados.0.smithi107.stdout:5882: left oid 18 (ObjNum 5881 snap 0 seq_num 5881) 2020-06-12T02:38:25.567 INFO:tasks.rados.rados.0.smithi107.stdout:5881: done (4 left) 2020-06-12T02:38:25.568 INFO:tasks.rados.rados.0.smithi107.stdout:5882: done (3 left) 2020-06-12T02:38:25.568 INFO:tasks.rados.rados.0.smithi107.stdout:5885: done (2 left) 2020-06-12T02:38:25.568 INFO:tasks.rados.rados.0.smithi107.stdout:5886: delete oid 8 current snap is 0 2020-06-12T02:38:25.568 INFO:tasks.rados.rados.0.smithi107.stdout:5884: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:25.569 INFO:tasks.rados.rados.0.smithi107.stdout:5886: done (2 left) 2020-06-12T02:38:25.569 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:38:25.569 INFO:tasks.rados.rados.0.smithi107.stdout:5887: seq_num 5886 ranges {0=180224} 2020-06-12T02:38:25.570 INFO:tasks.rados.rados.0.smithi107.stdout:5887: writing smithi10729407-13 from 0 to 180224 tid 1 2020-06-12T02:38:25.570 INFO:tasks.rados.rados.0.smithi107.stdout:5888: delete oid 18 current snap is 0 2020-06-12T02:38:25.570 INFO:tasks.rados.rados.0.smithi107.stdout:5884: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:25.571 INFO:tasks.rados.rados.0.smithi107.stdout:5884: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:25.571 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 453 (ObjNum 5883 snap 0 seq_num 5883) dirty exists 2020-06-12T02:38:25.572 INFO:tasks.rados.rados.0.smithi107.stdout:5884: left oid 20 (ObjNum 5883 snap 0 seq_num 5883) 2020-06-12T02:38:25.572 INFO:tasks.rados.rados.0.smithi107.stdout:5884: done (3 left) 2020-06-12T02:38:25.573 INFO:tasks.rados.rados.0.smithi107.stdout:5888: done (2 left) 2020-06-12T02:38:25.573 INFO:tasks.rados.rados.0.smithi107.stdout:5889: delete oid 6 current snap is 0 2020-06-12T02:38:25.574 INFO:tasks.rados.rados.0.smithi107.stdout:5887: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:25.576 INFO:tasks.rados.rados.0.smithi107.stdout:5887: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:25.577 INFO:tasks.rados.rados.0.smithi107.stdout:5889: done (2 left) 2020-06-12T02:38:25.577 INFO:tasks.rados.rados.0.smithi107.stdout:5890: delete oid 2 current snap is 0 2020-06-12T02:38:25.578 INFO:tasks.rados.rados.0.smithi107.stdout:5887: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:25.578 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1121 (ObjNum 5886 snap 0 seq_num 5886) dirty exists 2020-06-12T02:38:25.578 INFO:tasks.rados.rados.0.smithi107.stdout:5887: left oid 13 (ObjNum 5886 snap 0 seq_num 5886) 2020-06-12T02:38:25.579 INFO:tasks.rados.rados.0.smithi107.stdout:5887: done (2 left) 2020-06-12T02:38:25.580 INFO:tasks.rados.rados.0.smithi107.stdout:5890: done (1 left) 2020-06-12T02:38:25.580 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:25.580 INFO:tasks.rados.rados.0.smithi107.stdout:5891: seq_num 5890 ranges {49152=253952} 2020-06-12T02:38:25.581 INFO:tasks.rados.rados.0.smithi107.stdout:5891: writing smithi10729407-19 from 49152 to 303104 tid 1 2020-06-12T02:38:25.581 INFO:tasks.rados.rados.0.smithi107.stdout:5892: delete oid 12 current snap is 0 2020-06-12T02:38:25.585 INFO:tasks.rados.rados.0.smithi107.stdout:5891: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:25.586 INFO:tasks.rados.rados.0.smithi107.stdout:5891: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:25.586 INFO:tasks.rados.rados.0.smithi107.stdout:5891: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:25.587 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1046 (ObjNum 5890 snap 0 seq_num 5890) dirty exists 2020-06-12T02:38:25.587 INFO:tasks.rados.rados.0.smithi107.stdout:5891: left oid 19 (ObjNum 5890 snap 0 seq_num 5890) 2020-06-12T02:38:25.587 INFO:tasks.rados.rados.0.smithi107.stdout:5891: done (2 left) 2020-06-12T02:38:25.588 INFO:tasks.rados.rados.0.smithi107.stdout:5892: done (1 left) 2020-06-12T02:38:25.588 INFO:tasks.rados.rados.0.smithi107.stdout:5893: delete oid 7 current snap is 0 2020-06-12T02:38:25.590 INFO:tasks.rados.rados.0.smithi107.stdout:5893: done (1 left) 2020-06-12T02:38:25.590 INFO:tasks.rados.rados.0.smithi107.stdout:5894: delete oid 5 current snap is 0 2020-06-12T02:38:25.592 INFO:tasks.rados.rados.0.smithi107.stdout:5894: done (1 left) 2020-06-12T02:38:25.592 INFO:tasks.rados.rados.0.smithi107.stdout:5895: delete oid 20 current snap is 0 2020-06-12T02:38:25.594 INFO:tasks.rados.rados.0.smithi107.stdout:5895: done (1 left) 2020-06-12T02:38:25.595 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:38:25.595 INFO:tasks.rados.rados.0.smithi107.stdout:5896: seq_num 5895 ranges {180224=335872} 2020-06-12T02:38:25.597 INFO:tasks.rados.rados.0.smithi107.stdout:5896: writing smithi10729407-13 from 180224 to 516096 tid 1 2020-06-12T02:38:25.597 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:25.598 INFO:tasks.rados.rados.0.smithi107.stdout:5897: seq_num 5896 ranges {0=352256} 2020-06-12T02:38:25.599 INFO:tasks.rados.rados.0.smithi107.stdout:5897: writing smithi10729407-18 from 0 to 352256 tid 1 2020-06-12T02:38:25.599 INFO:tasks.rados.rados.0.smithi107.stdout:5898: delete oid 8 current snap is 0 2020-06-12T02:38:25.602 INFO:tasks.rados.rados.0.smithi107.stdout:5898: done (3 left) 2020-06-12T02:38:25.602 INFO:tasks.rados.rados.0.smithi107.stdout:5899: delete oid 12 current snap is 0 2020-06-12T02:38:25.603 INFO:tasks.rados.rados.0.smithi107.stdout:5896: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:25.604 INFO:tasks.rados.rados.0.smithi107.stdout:5896: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:25.611 INFO:tasks.rados.rados.0.smithi107.stdout:5897: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:25.611 INFO:tasks.rados.rados.0.smithi107.stdout:5896: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:25.611 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1124 (ObjNum 5895 snap 0 seq_num 5895) dirty exists 2020-06-12T02:38:25.611 INFO:tasks.rados.rados.0.smithi107.stdout:5896: left oid 13 (ObjNum 5895 snap 0 seq_num 5895) 2020-06-12T02:38:25.612 INFO:tasks.rados.rados.0.smithi107.stdout:5896: done (3 left) 2020-06-12T02:38:25.612 INFO:tasks.rados.rados.0.smithi107.stdout:5899: done (2 left) 2020-06-12T02:38:25.612 INFO:tasks.rados.rados.0.smithi107.stdout:5900: delete oid 19 current snap is 0 2020-06-12T02:38:25.612 INFO:tasks.rados.rados.0.smithi107.stdout:5900: done (2 left) 2020-06-12T02:38:25.612 INFO:tasks.rados.rados.0.smithi107.stdout:5901: delete oid 8 current snap is 0 2020-06-12T02:38:25.613 INFO:tasks.rados.rados.0.smithi107.stdout:5897: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:25.613 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:25.613 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:25.614 INFO:tasks.rados.rados.0.smithi107.stdout:5897: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:25.614 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1076 (ObjNum 5896 snap 0 seq_num 5896) dirty exists 2020-06-12T02:38:25.614 INFO:tasks.rados.rados.0.smithi107.stdout:5897: left oid 18 (ObjNum 5896 snap 0 seq_num 5896) 2020-06-12T02:38:25.615 INFO:tasks.rados.rados.0.smithi107.stdout:5897: done (2 left) 2020-06-12T02:38:25.616 INFO:tasks.rados.rados.0.smithi107.stdout:5901: done (1 left) 2020-06-12T02:38:25.616 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:25.616 INFO:tasks.rados.rados.0.smithi107.stdout:5902: seq_num 5901 ranges {516096=434176} 2020-06-12T02:38:25.621 INFO:tasks.rados.rados.0.smithi107.stdout:5902: writing smithi10729407-13 from 516096 to 950272 tid 1 2020-06-12T02:38:25.622 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:25.622 INFO:tasks.rados.rados.0.smithi107.stdout:5903: seq_num 5902 ranges {2121728=450560} 2020-06-12T02:38:25.622 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-06-12T02:38:25.641 INFO:tasks.rados.rados.0.smithi107.stdout:5903: writing smithi10729407-14 from 2121728 to 2572288 tid 1 2020-06-12T02:38:25.641 INFO:tasks.rados.rados.0.smithi107.stdout:5904: delete oid 8 current snap is 0 2020-06-12T02:38:25.641 INFO:tasks.rados.rados.0.smithi107.stdout:5902: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:25.642 INFO:tasks.rados.rados.0.smithi107.stdout:5902: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:25.642 INFO:tasks.rados.rados.0.smithi107.stdout:5902: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:25.642 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1126 (ObjNum 5901 snap 0 seq_num 5901) dirty exists 2020-06-12T02:38:25.642 INFO:tasks.rados.rados.0.smithi107.stdout:5902: left oid 13 (ObjNum 5901 snap 0 seq_num 5901) 2020-06-12T02:38:25.642 INFO:tasks.rados.rados.0.smithi107.stdout:5902: done (3 left) 2020-06-12T02:38:25.643 INFO:tasks.rados.rados.0.smithi107.stdout:5904: done (2 left) 2020-06-12T02:38:25.643 INFO:tasks.rados.rados.0.smithi107.stdout:5905: delete oid 9 current snap is 0 2020-06-12T02:38:25.643 INFO:tasks.rados.rados.0.smithi107.stdout:5903: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:25.644 INFO:tasks.rados.rados.0.smithi107.stdout:5905: done (2 left) 2020-06-12T02:38:25.644 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:25.645 INFO:tasks.rados.rados.0.smithi107.stdout:5906: seq_num 5905 ranges {0=499712} 2020-06-12T02:38:25.646 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:25.642+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:25.647 INFO:tasks.rados.rados.0.smithi107.stdout:5906: writing smithi10729407-4 from 0 to 499712 tid 1 2020-06-12T02:38:25.647 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:25.648 INFO:tasks.rados.rados.0.smithi107.stdout:5907: seq_num 5906 ranges {0=516096} 2020-06-12T02:38:25.655 INFO:tasks.rados.rados.0.smithi107.stdout:5907: writing smithi10729407-15 from 0 to 516096 tid 1 2020-06-12T02:38:25.655 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:25.655 INFO:tasks.rados.rados.0.smithi107.stdout:5908: seq_num 5907 ranges {0=540672} 2020-06-12T02:38:25.655 INFO:tasks.rados.rados.0.smithi107.stdout:5908: writing smithi10729407-6 from 0 to 540672 tid 1 2020-06-12T02:38:25.656 INFO:tasks.rados.rados.0.smithi107.stdout:5909: delete oid 8 current snap is 0 2020-06-12T02:38:25.656 INFO:tasks.rados.rados.0.smithi107.stdout:5903: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:25.656 INFO:tasks.rados.rados.0.smithi107.stdout:5903: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:25.656 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1055 (ObjNum 5902 snap 0 seq_num 5902) dirty exists 2020-06-12T02:38:25.656 INFO:tasks.rados.rados.0.smithi107.stdout:5903: left oid 14 (ObjNum 5902 snap 0 seq_num 5902) 2020-06-12T02:38:25.657 INFO:tasks.rados.rados.0.smithi107.stdout:5906: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:25.658 INFO:tasks.rados.rados.0.smithi107.stdout:5907: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:25.658 INFO:tasks.rados.rados.0.smithi107.stdout:5907: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:25.660 INFO:tasks.rados.rados.0.smithi107.stdout:5907: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:25.660 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1086 (ObjNum 5906 snap 0 seq_num 5906) dirty exists 2020-06-12T02:38:25.661 INFO:tasks.rados.rados.0.smithi107.stdout:5907: left oid 15 (ObjNum 5906 snap 0 seq_num 5906) 2020-06-12T02:38:25.662 INFO:tasks.rados.rados.0.smithi107.stdout:5906: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:25.662 INFO:tasks.rados.rados.0.smithi107.stdout:5903: done (5 left) 2020-06-12T02:38:25.663 INFO:tasks.rados.rados.0.smithi107.stdout:5907: done (4 left) 2020-06-12T02:38:25.663 INFO:tasks.rados.rados.0.smithi107.stdout:5909: done (3 left) 2020-06-12T02:38:25.663 INFO:tasks.rados.rados.0.smithi107.stdout:5910: delete oid 18 current snap is 0 2020-06-12T02:38:25.664 INFO:tasks.rados.rados.0.smithi107.stdout:5906: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:25.664 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1098 (ObjNum 5905 snap 0 seq_num 5905) dirty exists 2020-06-12T02:38:25.665 INFO:tasks.rados.rados.0.smithi107.stdout:5906: left oid 4 (ObjNum 5905 snap 0 seq_num 5905) 2020-06-12T02:38:25.665 INFO:tasks.rados.rados.0.smithi107.stdout:5906: done (3 left) 2020-06-12T02:38:25.665 INFO:tasks.rados.rados.0.smithi107.stdout:5910: done (2 left) 2020-06-12T02:38:25.665 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:25.665 INFO:tasks.rados.rados.0.smithi107.stdout:5911: seq_num 5910 ranges {0=589824} 2020-06-12T02:38:25.668 INFO:tasks.rados.rados.0.smithi107.stdout:5911: writing smithi10729407-2 from 0 to 589824 tid 1 2020-06-12T02:38:25.668 INFO:tasks.rados.rados.0.smithi107.stdout:5912: delete oid 13 current snap is 0 2020-06-12T02:38:25.668 INFO:tasks.rados.rados.0.smithi107.stdout:5908: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:25.668 INFO:tasks.rados.rados.0.smithi107.stdout:5908: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:25.670 INFO:tasks.rados.rados.0.smithi107.stdout:5912: done (3 left) 2020-06-12T02:38:25.670 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:25.671 INFO:tasks.rados.rados.0.smithi107.stdout:5913: seq_num 5912 ranges {0=622592} 2020-06-12T02:38:25.674 INFO:tasks.rados.rados.0.smithi107.stdout:5913: writing smithi10729407-8 from 0 to 622592 tid 1 2020-06-12T02:38:25.675 INFO:tasks.rados.rados.0.smithi107.stdout:5914: delete oid 17 current snap is 0 2020-06-12T02:38:25.675 INFO:tasks.rados.rados.0.smithi107.stdout:5908: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:25.675 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1103 (ObjNum 5907 snap 0 seq_num 5907) dirty exists 2020-06-12T02:38:25.675 INFO:tasks.rados.rados.0.smithi107.stdout:5908: left oid 6 (ObjNum 5907 snap 0 seq_num 5907) 2020-06-12T02:38:25.675 INFO:tasks.rados.rados.0.smithi107.stdout:5911: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:25.676 INFO:tasks.rados.rados.0.smithi107.stdout:5911: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:25.676 INFO:tasks.rados.rados.0.smithi107.stdout:5911: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:25.677 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 556 (ObjNum 5910 snap 0 seq_num 5910) dirty exists 2020-06-12T02:38:25.677 INFO:tasks.rados.rados.0.smithi107.stdout:5911: left oid 2 (ObjNum 5910 snap 0 seq_num 5910) 2020-06-12T02:38:25.677 INFO:tasks.rados.rados.0.smithi107.stdout:5908: done (4 left) 2020-06-12T02:38:25.677 INFO:tasks.rados.rados.0.smithi107.stdout:5911: done (3 left) 2020-06-12T02:38:25.678 INFO:tasks.rados.rados.0.smithi107.stdout:5914: done (2 left) 2020-06-12T02:38:25.678 INFO:tasks.rados.rados.0.smithi107.stdout:5915: delete oid 12 current snap is 0 2020-06-12T02:38:25.679 INFO:tasks.rados.rados.0.smithi107.stdout:5915: done (2 left) 2020-06-12T02:38:25.679 INFO:tasks.rados.rados.0.smithi107.stdout:5916: delete oid 9 current snap is 0 2020-06-12T02:38:25.684 INFO:tasks.rados.rados.0.smithi107.stdout:5916: done (2 left) 2020-06-12T02:38:25.685 INFO:tasks.rados.rados.0.smithi107.stdout:5917: delete oid 11 current snap is 0 2020-06-12T02:38:25.685 INFO:tasks.rados.rados.0.smithi107.stdout:5913: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:25.687 INFO:tasks.rados.rados.0.smithi107.stdout:5917: done (2 left) 2020-06-12T02:38:25.687 INFO:tasks.rados.rados.0.smithi107.stdout:5918: delete oid 18 current snap is 0 2020-06-12T02:38:25.688 INFO:tasks.rados.rados.0.smithi107.stdout:5913: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:25.696 INFO:tasks.rados.rados.0.smithi107.stdout:5913: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:25.696 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1089 (ObjNum 5912 snap 0 seq_num 5912) dirty exists 2020-06-12T02:38:25.697 INFO:tasks.rados.rados.0.smithi107.stdout:5913: left oid 8 (ObjNum 5912 snap 0 seq_num 5912) 2020-06-12T02:38:25.697 INFO:tasks.rados.rados.0.smithi107.stdout:5913: done (2 left) 2020-06-12T02:38:25.697 INFO:tasks.rados.rados.0.smithi107.stdout:5918: done (1 left) 2020-06-12T02:38:25.697 INFO:tasks.rados.rados.0.smithi107.stdout:5919: delete oid 12 current snap is 0 2020-06-12T02:38:25.698 INFO:tasks.rados.rados.0.smithi107.stdout:5919: done (1 left) 2020-06-12T02:38:25.698 INFO:tasks.rados.rados.0.smithi107.stdout:5920: delete oid 8 current snap is 0 2020-06-12T02:38:25.698 INFO:tasks.rados.rados.0.smithi107.stdout:5920: done (1 left) 2020-06-12T02:38:25.698 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:25.698 INFO:tasks.rados.rados.0.smithi107.stdout:5921: seq_num 5920 ranges {516096=753664} 2020-06-12T02:38:25.702 INFO:tasks.rados.rados.0.smithi107.stdout:5921: writing smithi10729407-15 from 516096 to 1269760 tid 1 2020-06-12T02:38:25.702 INFO:tasks.rados.rados.0.smithi107.stdout:5922: delete oid 20 current snap is 0 2020-06-12T02:38:25.707 INFO:tasks.rados.rados.0.smithi107.stdout:5922: done (2 left) 2020-06-12T02:38:25.708 INFO:tasks.rados.rados.0.smithi107.stdout:5923: delete oid 3 current snap is 0 2020-06-12T02:38:25.708 INFO:tasks.rados.rados.0.smithi107.stdout:5921: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:25.710 INFO:tasks.rados.rados.0.smithi107.stdout:5923: done (2 left) 2020-06-12T02:38:25.710 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:38:25.710 INFO:tasks.rados.rados.0.smithi107.stdout:5924: seq_num 5923 ranges {0=802816} 2020-06-12T02:38:25.715 INFO:tasks.rados.rados.0.smithi107.stdout:5924: writing smithi10729407-13 from 0 to 802816 tid 1 2020-06-12T02:38:25.715 INFO:tasks.rados.rados.0.smithi107.stdout:5925: delete oid 20 current snap is 0 2020-06-12T02:38:25.715 INFO:tasks.rados.rados.0.smithi107.stdout:5921: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:25.715 INFO:tasks.rados.rados.0.smithi107.stdout:5921: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:25.716 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1092 (ObjNum 5920 snap 0 seq_num 5920) dirty exists 2020-06-12T02:38:25.716 INFO:tasks.rados.rados.0.smithi107.stdout:5921: left oid 15 (ObjNum 5920 snap 0 seq_num 5920) 2020-06-12T02:38:25.717 INFO:tasks.rados.rados.0.smithi107.stdout:5921: done (3 left) 2020-06-12T02:38:25.717 INFO:tasks.rados.rados.0.smithi107.stdout:5925: done (2 left) 2020-06-12T02:38:25.717 INFO:tasks.rados.rados.0.smithi107.stdout:5926: delete oid 5 current snap is 0 2020-06-12T02:38:25.719 INFO:tasks.rados.rados.0.smithi107.stdout:5926: done (2 left) 2020-06-12T02:38:25.719 INFO:tasks.rados.rados.0.smithi107.stdout:5927: delete oid 6 current snap is 0 2020-06-12T02:38:25.721 INFO:tasks.rados.rados.0.smithi107.stdout:5927: done (2 left) 2020-06-12T02:38:25.721 INFO:tasks.rados.rados.0.smithi107.stdout:5928: delete oid 2 current snap is 0 2020-06-12T02:38:25.723 INFO:tasks.rados.rados.0.smithi107.stdout:5924: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:25.724 INFO:tasks.rados.rados.0.smithi107.stdout:5924: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:25.724 INFO:tasks.rados.rados.0.smithi107.stdout:5928: done (2 left) 2020-06-12T02:38:25.724 INFO:tasks.rados.rados.0.smithi107.stdout:5929: delete oid 5 current snap is 0 2020-06-12T02:38:25.726 INFO:tasks.rados.rados.0.smithi107.stdout:5924: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:25.726 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1129 (ObjNum 5923 snap 0 seq_num 5923) dirty exists 2020-06-12T02:38:25.726 INFO:tasks.rados.rados.0.smithi107.stdout:5924: left oid 13 (ObjNum 5923 snap 0 seq_num 5923) 2020-06-12T02:38:25.727 INFO:tasks.rados.rados.0.smithi107.stdout:5924: done (2 left) 2020-06-12T02:38:25.727 INFO:tasks.rados.rados.0.smithi107.stdout:5929: done (1 left) 2020-06-12T02:38:25.727 INFO:tasks.rados.rados.0.smithi107.stdout:5930: delete oid 19 current snap is 0 2020-06-12T02:38:25.728 INFO:tasks.rados.rados.0.smithi107.stdout:5930: done (1 left) 2020-06-12T02:38:25.732 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:25.732 INFO:tasks.rados.rados.0.smithi107.stdout:5931: seq_num 5930 ranges {0=524288,524288=401408} 2020-06-12T02:38:25.735 INFO:tasks.rados.rados.0.smithi107.stdout:5931: writing smithi10729407-2 from 0 to 524288 tid 1 2020-06-12T02:38:25.735 INFO:tasks.rados.rados.0.smithi107.stdout:5931: writing smithi10729407-2 from 524288 to 925696 tid 2 2020-06-12T02:38:25.735 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:25.736 INFO:tasks.rados.rados.0.smithi107.stdout:5932: seq_num 5931 ranges {0=540672,540672=401408} 2020-06-12T02:38:25.737 INFO:tasks.rados.rados.0.smithi107.stdout:5932: writing smithi10729407-6 from 0 to 540672 tid 1 2020-06-12T02:38:25.740 INFO:tasks.rados.rados.0.smithi107.stdout:5932: writing smithi10729407-6 from 540672 to 942080 tid 2 2020-06-12T02:38:25.741 INFO:tasks.rados.rados.0.smithi107.stdout:5933: delete oid 3 current snap is 0 2020-06-12T02:38:25.741 INFO:tasks.rados.rados.0.smithi107.stdout:5931: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:25.741 INFO:tasks.rados.rados.0.smithi107.stdout:5931: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:25.742 INFO:tasks.rados.rados.0.smithi107.stdout:5931: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:25.742 INFO:tasks.rados.rados.0.smithi107.stdout:5931: finishing write tid 5 to smithi10729407-2 2020-06-12T02:38:25.743 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 560 (ObjNum 5930 snap 0 seq_num 5930) dirty exists 2020-06-12T02:38:25.743 INFO:tasks.rados.rados.0.smithi107.stdout:5931: left oid 2 (ObjNum 5930 snap 0 seq_num 5930) 2020-06-12T02:38:25.744 INFO:tasks.rados.rados.0.smithi107.stdout:5931: done (3 left) 2020-06-12T02:38:25.744 INFO:tasks.rados.rados.0.smithi107.stdout:5933: done (2 left) 2020-06-12T02:38:25.745 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:25.745 INFO:tasks.rados.rados.0.smithi107.stdout:5934: seq_num 5933 ranges {0=573440,573440=401408} 2020-06-12T02:38:25.747 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:38:25.745+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:38:25.747 INFO:tasks.rados.rados.0.smithi107.stdout:5934: writing smithi10729407-18 from 0 to 573440 tid 1 2020-06-12T02:38:25.750 INFO:tasks.rados.rados.0.smithi107.stdout:5934: writing smithi10729407-18 from 573440 to 974848 tid 2 2020-06-12T02:38:25.750 INFO:tasks.rados.rados.0.smithi107.stdout:5935: delete oid 14 current snap is 0 2020-06-12T02:38:25.750 INFO:tasks.rados.rados.0.smithi107.stdout:5932: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:25.751 INFO:tasks.rados.rados.0.smithi107.stdout:5932: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:25.751 INFO:tasks.rados.rados.0.smithi107.stdout:5932: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:25.757 INFO:tasks.rados.rados.0.smithi107.stdout:5932: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:25.757 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1107 (ObjNum 5931 snap 0 seq_num 5931) dirty exists 2020-06-12T02:38:25.758 INFO:tasks.rados.rados.0.smithi107.stdout:5932: left oid 6 (ObjNum 5931 snap 0 seq_num 5931) 2020-06-12T02:38:25.758 INFO:tasks.rados.rados.0.smithi107.stdout:5934: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:25.758 INFO:tasks.rados.rados.0.smithi107.stdout:5932: done (3 left) 2020-06-12T02:38:25.758 INFO:tasks.rados.rados.0.smithi107.stdout:5935: done (2 left) 2020-06-12T02:38:25.758 INFO:tasks.rados.rados.0.smithi107.stdout:5936: delete oid 12 current snap is 0 2020-06-12T02:38:25.759 INFO:tasks.rados.rados.0.smithi107.stdout:5934: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:25.759 INFO:tasks.rados.rados.0.smithi107.stdout:5934: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:25.759 INFO:tasks.rados.rados.0.smithi107.stdout:5936: done (2 left) 2020-06-12T02:38:25.760 INFO:tasks.rados.rados.0.smithi107.stdout:5937: delete oid 7 current snap is 0 2020-06-12T02:38:25.760 INFO:tasks.rados.rados.0.smithi107.stdout:5934: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:25.760 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1083 (ObjNum 5933 snap 0 seq_num 5933) dirty exists 2020-06-12T02:38:25.760 INFO:tasks.rados.rados.0.smithi107.stdout:5934: left oid 18 (ObjNum 5933 snap 0 seq_num 5933) 2020-06-12T02:38:25.762 INFO:tasks.rados.rados.0.smithi107.stdout:5934: done (2 left) 2020-06-12T02:38:25.762 INFO:tasks.rados.rados.0.smithi107.stdout:5937: done (1 left) 2020-06-12T02:38:25.762 INFO:tasks.rados.rados.0.smithi107.stdout:5938: delete oid 17 current snap is 0 2020-06-12T02:38:25.764 INFO:tasks.rados.rados.0.smithi107.stdout:5938: done (1 left) 2020-06-12T02:38:25.765 INFO:tasks.rados.rados.0.smithi107.stdout:5939: delete oid 15 current snap is 0 2020-06-12T02:38:25.766 INFO:tasks.rados.rados.0.smithi107.stdout:5939: done (1 left) 2020-06-12T02:38:25.766 INFO:tasks.rados.rados.0.smithi107.stdout:5940: delete oid 11 current snap is 0 2020-06-12T02:38:25.769 INFO:tasks.rados.rados.0.smithi107.stdout:5940: done (1 left) 2020-06-12T02:38:25.769 INFO:tasks.rados.rados.0.smithi107.stdout:5941: delete oid 4 current snap is 0 2020-06-12T02:38:25.772 INFO:tasks.rados.rados.0.smithi107.stdout:5941: done (1 left) 2020-06-12T02:38:25.772 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:25.772 INFO:tasks.rados.rados.0.smithi107.stdout:5942: seq_num 5941 ranges {0=704512,704512=401408} 2020-06-12T02:38:25.776 INFO:tasks.rados.rados.0.smithi107.stdout:5942: writing smithi10729407-1 from 0 to 704512 tid 1 2020-06-12T02:38:25.778 INFO:tasks.rados.rados.0.smithi107.stdout:5942: writing smithi10729407-1 from 704512 to 1105920 tid 2 2020-06-12T02:38:25.779 INFO:tasks.rados.rados.0.smithi107.stdout:5943: delete oid 12 current snap is 0 2020-06-12T02:38:25.781 INFO:tasks.rados.rados.0.smithi107.stdout:5943: done (2 left) 2020-06-12T02:38:25.781 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:25.782 INFO:tasks.rados.rados.0.smithi107.stdout:5944: seq_num 5943 ranges {0=737280,737280=401408} 2020-06-12T02:38:25.783 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:25.783 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:25.786 INFO:tasks.rados.rados.0.smithi107.stdout:5944: writing smithi10729407-15 from 0 to 737280 tid 1 2020-06-12T02:38:25.789 INFO:tasks.rados.rados.0.smithi107.stdout:5944: writing smithi10729407-15 from 737280 to 1138688 tid 2 2020-06-12T02:38:25.789 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:25.789 INFO:tasks.rados.rados.0.smithi107.stdout:5945: seq_num 5944 ranges {0=753664,753664=401408} 2020-06-12T02:38:25.793 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-06-12T02:38:25.794 INFO:tasks.rados.rados.0.smithi107.stdout:5945: writing smithi10729407-11 from 0 to 753664 tid 1 2020-06-12T02:38:25.796 INFO:tasks.rados.rados.0.smithi107.stdout:5945: writing smithi10729407-11 from 753664 to 1155072 tid 2 2020-06-12T02:38:25.796 INFO:tasks.rados.rados.0.smithi107.stdout:5946: delete oid 17 current snap is 0 2020-06-12T02:38:25.796 INFO:tasks.rados.rados.0.smithi107.stdout:5942: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:25.796 INFO:tasks.rados.rados.0.smithi107.stdout:5942: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:25.797 INFO:tasks.rados.rados.0.smithi107.stdout:5942: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:25.797 INFO:tasks.rados.rados.0.smithi107.stdout:5942: finishing write tid 5 to smithi10729407-1 2020-06-12T02:38:25.797 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1110 (ObjNum 5941 snap 0 seq_num 5941) dirty exists 2020-06-12T02:38:25.798 INFO:tasks.rados.rados.0.smithi107.stdout:5942: left oid 1 (ObjNum 5941 snap 0 seq_num 5941) 2020-06-12T02:38:25.798 INFO:tasks.rados.rados.0.smithi107.stdout:5944: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:25.799 INFO:tasks.rados.rados.0.smithi107.stdout:5942: done (4 left) 2020-06-12T02:38:25.799 INFO:tasks.rados.rados.0.smithi107.stdout:5946: done (3 left) 2020-06-12T02:38:25.800 INFO:tasks.rados.rados.0.smithi107.stdout:5947: delete oid 3 current snap is 0 2020-06-12T02:38:25.804 INFO:tasks.rados.rados.0.smithi107.stdout:5944: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:25.804 INFO:tasks.rados.rados.0.smithi107.stdout:5944: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:25.804 INFO:tasks.rados.rados.0.smithi107.stdout:5947: done (3 left) 2020-06-12T02:38:25.804 INFO:tasks.rados.rados.0.smithi107.stdout:5948: delete oid 20 current snap is 0 2020-06-12T02:38:25.804 INFO:tasks.rados.rados.0.smithi107.stdout:5945: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:25.805 INFO:tasks.rados.rados.0.smithi107.stdout:5944: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:25.805 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1096 (ObjNum 5943 snap 0 seq_num 5943) dirty exists 2020-06-12T02:38:25.805 INFO:tasks.rados.rados.0.smithi107.stdout:5944: left oid 15 (ObjNum 5943 snap 0 seq_num 5943) 2020-06-12T02:38:25.805 INFO:tasks.rados.rados.0.smithi107.stdout:5945: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:25.805 INFO:tasks.rados.rados.0.smithi107.stdout:5944: done (3 left) 2020-06-12T02:38:25.806 INFO:tasks.rados.rados.0.smithi107.stdout:5948: done (2 left) 2020-06-12T02:38:25.806 INFO:tasks.rados.rados.0.smithi107.stdout:5949: delete oid 6 current snap is 0 2020-06-12T02:38:25.806 INFO:tasks.rados.rados.0.smithi107.stdout:5945: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:25.807 INFO:tasks.rados.rados.0.smithi107.stdout:5945: finishing write tid 5 to smithi10729407-11 2020-06-12T02:38:25.807 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1061 (ObjNum 5944 snap 0 seq_num 5944) dirty exists 2020-06-12T02:38:25.807 INFO:tasks.rados.rados.0.smithi107.stdout:5945: left oid 11 (ObjNum 5944 snap 0 seq_num 5944) 2020-06-12T02:38:25.807 INFO:tasks.rados.rados.0.smithi107.stdout:5945: done (2 left) 2020-06-12T02:38:25.808 INFO:tasks.rados.rados.0.smithi107.stdout:5949: done (1 left) 2020-06-12T02:38:25.808 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:25.808 INFO:tasks.rados.rados.0.smithi107.stdout:5950: seq_num 5949 ranges {0=442368,442368=745472,1187840=57344} 2020-06-12T02:38:25.808 INFO:tasks.rados.rados.0.smithi107.stdout:5950: writing smithi10729407-14 from 0 to 442368 tid 1 2020-06-12T02:38:25.813 INFO:tasks.rados.rados.0.smithi107.stdout:5950: writing smithi10729407-14 from 442368 to 1187840 tid 2 2020-06-12T02:38:25.814 INFO:tasks.rados.rados.0.smithi107.stdout:5950: writing smithi10729407-14 from 1187840 to 1245184 tid 3 2020-06-12T02:38:25.814 INFO:tasks.rados.rados.0.smithi107.stdout:5951: delete oid 19 current snap is 0 2020-06-12T02:38:25.814 INFO:tasks.rados.rados.0.smithi107.stdout:5950: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:25.818 INFO:tasks.rados.rados.0.smithi107.stdout:5951: done (2 left) 2020-06-12T02:38:25.818 INFO:tasks.rados.rados.0.smithi107.stdout:5952: delete oid 9 current snap is 0 2020-06-12T02:38:25.823 INFO:tasks.rados.rados.0.smithi107.stdout:5950: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:25.823 INFO:tasks.rados.rados.0.smithi107.stdout:5952: done (2 left) 2020-06-12T02:38:25.823 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:38:25.824 INFO:tasks.rados.rados.0.smithi107.stdout:5953: seq_num 5952 ranges {802816=491520,1294336=802816} 2020-06-12T02:38:25.830 INFO:tasks.rados.rados.0.smithi107.stdout:5953: writing smithi10729407-13 from 802816 to 1294336 tid 1 2020-06-12T02:38:25.835 INFO:tasks.rados.rados.0.smithi107.stdout:5953: writing smithi10729407-13 from 1294336 to 2097152 tid 2 2020-06-12T02:38:25.835 INFO:tasks.rados.rados.0.smithi107.stdout:5950: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:25.835 INFO:tasks.rados.rados.0.smithi107.stdout:5950: finishing write tid 5 to smithi10729407-14 2020-06-12T02:38:25.836 INFO:tasks.rados.rados.0.smithi107.stdout:5950: finishing write tid 6 to smithi10729407-14 2020-06-12T02:38:25.836 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1065 (ObjNum 5949 snap 0 seq_num 5949) dirty exists 2020-06-12T02:38:25.836 INFO:tasks.rados.rados.0.smithi107.stdout:5950: left oid 14 (ObjNum 5949 snap 0 seq_num 5949) 2020-06-12T02:38:25.836 INFO:tasks.rados.rados.0.smithi107.stdout:5950: done (2 left) 2020-06-12T02:38:25.836 INFO:tasks.rados.rados.0.smithi107.stdout:5954: delete oid 11 current snap is 0 2020-06-12T02:38:25.837 INFO:tasks.rados.rados.0.smithi107.stdout:5953: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:25.838 INFO:tasks.rados.rados.0.smithi107.stdout:5954: done (2 left) 2020-06-12T02:38:25.838 INFO:tasks.rados.rados.0.smithi107.stdout:5955: delete oid 17 current snap is 0 2020-06-12T02:38:25.840 INFO:tasks.rados.rados.0.smithi107.stdout:5955: done (2 left) 2020-06-12T02:38:25.840 INFO:tasks.rados.rados.0.smithi107.stdout:5956: delete oid 11 current snap is 0 2020-06-12T02:38:25.851 INFO:tasks.rados.rados.0.smithi107.stdout:5956: done (2 left) 2020-06-12T02:38:25.852 INFO:tasks.rados.rados.0.smithi107.stdout:5957: delete oid 19 current snap is 0 2020-06-12T02:38:25.852 INFO:tasks.rados.rados.0.smithi107.stdout:5953: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:25.852 INFO:tasks.rados.rados.0.smithi107.stdout:5953: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:25.852 INFO:tasks.rados.rados.0.smithi107.stdout:5953: finishing write tid 5 to smithi10729407-13 2020-06-12T02:38:25.853 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1133 (ObjNum 5952 snap 0 seq_num 5952) dirty exists 2020-06-12T02:38:25.853 INFO:tasks.rados.rados.0.smithi107.stdout:5953: left oid 13 (ObjNum 5952 snap 0 seq_num 5952) 2020-06-12T02:38:25.853 INFO:tasks.rados.rados.0.smithi107.stdout:5953: done (2 left) 2020-06-12T02:38:25.853 INFO:tasks.rados.rados.0.smithi107.stdout:5957: done (1 left) 2020-06-12T02:38:25.853 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:38:25.854 INFO:tasks.rados.rados.0.smithi107.stdout:5958: seq_num 5957 ranges {0=573440,573440=671744,1245184=131072} 2020-06-12T02:38:25.854 INFO:tasks.rados.rados.0.smithi107.stdout:5958: writing smithi10729407-5 from 0 to 573440 tid 1 2020-06-12T02:38:25.854 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:25.849+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:25.854 INFO:tasks.rados.rados.0.smithi107.stdout:5958: writing smithi10729407-5 from 573440 to 1245184 tid 2 2020-06-12T02:38:25.855 INFO:tasks.rados.rados.0.smithi107.stdout:5958: writing smithi10729407-5 from 1245184 to 1376256 tid 3 2020-06-12T02:38:25.855 INFO:tasks.rados.rados.0.smithi107.stdout:5959: delete oid 7 current snap is 0 2020-06-12T02:38:25.858 INFO:tasks.rados.rados.0.smithi107.stdout:5958: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:25.861 INFO:tasks.rados.rados.0.smithi107.stdout:5959: done (2 left) 2020-06-12T02:38:25.861 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:25.861 INFO:tasks.rados.rados.0.smithi107.stdout:5960: seq_num 5959 ranges {0=606208,606208=442368,1048576=360448} 2020-06-12T02:38:25.864 INFO:tasks.rados.rados.0.smithi107.stdout:5960: writing smithi10729407-19 from 0 to 606208 tid 1 2020-06-12T02:38:25.867 INFO:tasks.rados.rados.0.smithi107.stdout:5960: writing smithi10729407-19 from 606208 to 1048576 tid 2 2020-06-12T02:38:25.869 INFO:tasks.rados.rados.0.smithi107.stdout:5960: writing smithi10729407-19 from 1048576 to 1409024 tid 3 2020-06-12T02:38:25.870 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:25.870 INFO:tasks.rados.rados.0.smithi107.stdout:5958: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:25.871 INFO:tasks.rados.rados.0.smithi107.stdout:5961: seq_num 5960 ranges {1138688=622592,1761280=720896,2482176=81920} 2020-06-12T02:38:25.880 INFO:tasks.rados.rados.0.smithi107.stdout:5961: writing smithi10729407-15 from 1138688 to 1761280 tid 1 2020-06-12T02:38:25.884 INFO:tasks.rados.rados.0.smithi107.stdout:5961: writing smithi10729407-15 from 1761280 to 2482176 tid 2 2020-06-12T02:38:25.885 INFO:tasks.rados.rados.0.smithi107.stdout:5961: writing smithi10729407-15 from 2482176 to 2564096 tid 3 2020-06-12T02:38:25.885 INFO:tasks.rados.rados.0.smithi107.stdout:5962: delete oid 8 current snap is 0 2020-06-12T02:38:25.885 INFO:tasks.rados.rados.0.smithi107.stdout:5958: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:25.885 INFO:tasks.rados.rados.0.smithi107.stdout:5958: finishing write tid 5 to smithi10729407-5 2020-06-12T02:38:25.886 INFO:tasks.rados.rados.0.smithi107.stdout:5958: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:25.886 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 601 (ObjNum 5957 snap 0 seq_num 5957) dirty exists 2020-06-12T02:38:25.886 INFO:tasks.rados.rados.0.smithi107.stdout:5958: left oid 5 (ObjNum 5957 snap 0 seq_num 5957) 2020-06-12T02:38:25.886 INFO:tasks.rados.rados.0.smithi107.stdout:5960: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:25.886 INFO:tasks.rados.rados.0.smithi107.stdout:5960: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:25.887 INFO:tasks.rados.rados.0.smithi107.stdout:5960: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:25.887 INFO:tasks.rados.rados.0.smithi107.stdout:5960: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:25.887 INFO:tasks.rados.rados.0.smithi107.stdout:5960: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:25.887 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1060 (ObjNum 5959 snap 0 seq_num 5959) dirty exists 2020-06-12T02:38:25.888 INFO:tasks.rados.rados.0.smithi107.stdout:5960: left oid 19 (ObjNum 5959 snap 0 seq_num 5959) 2020-06-12T02:38:25.888 INFO:tasks.rados.rados.0.smithi107.stdout:5961: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:25.894 INFO:tasks.rados.rados.0.smithi107.stdout:5958: done (4 left) 2020-06-12T02:38:25.894 INFO:tasks.rados.rados.0.smithi107.stdout:5960: done (3 left) 2020-06-12T02:38:25.895 INFO:tasks.rados.rados.0.smithi107.stdout:5962: done (2 left) 2020-06-12T02:38:25.895 INFO:tasks.rados.rados.0.smithi107.stdout:5963: delete oid 11 current snap is 0 2020-06-12T02:38:25.896 INFO:tasks.rados.rados.0.smithi107.stdout:5961: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:25.896 INFO:tasks.rados.rados.0.smithi107.stdout:5963: done (2 left) 2020-06-12T02:38:25.896 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:25.896 INFO:tasks.rados.rados.0.smithi107.stdout:5964: seq_num 5963 ranges {0=671744,671744=425984,1097728=376832} 2020-06-12T02:38:25.899 INFO:tasks.rados.rados.0.smithi107.stdout:5964: writing smithi10729407-12 from 0 to 671744 tid 1 2020-06-12T02:38:25.902 INFO:tasks.rados.rados.0.smithi107.stdout:5964: writing smithi10729407-12 from 671744 to 1097728 tid 2 2020-06-12T02:38:25.904 INFO:tasks.rados.rados.0.smithi107.stdout:5964: writing smithi10729407-12 from 1097728 to 1474560 tid 3 2020-06-12T02:38:25.904 INFO:tasks.rados.rados.0.smithi107.stdout:5965: delete oid 18 current snap is 0 2020-06-12T02:38:25.905 INFO:tasks.rados.rados.0.smithi107.stdout:5961: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:25.905 INFO:tasks.rados.rados.0.smithi107.stdout:5961: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:25.905 INFO:tasks.rados.rados.0.smithi107.stdout:5961: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:25.905 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1101 (ObjNum 5960 snap 0 seq_num 5960) dirty exists 2020-06-12T02:38:25.905 INFO:tasks.rados.rados.0.smithi107.stdout:5961: left oid 15 (ObjNum 5960 snap 0 seq_num 5960) 2020-06-12T02:38:25.906 INFO:tasks.rados.rados.0.smithi107.stdout:5961: done (3 left) 2020-06-12T02:38:25.907 INFO:tasks.rados.rados.0.smithi107.stdout:5965: done (2 left) 2020-06-12T02:38:25.907 INFO:tasks.rados.rados.0.smithi107.stdout:5966: delete oid 1 current snap is 0 2020-06-12T02:38:25.909 INFO:tasks.rados.rados.0.smithi107.stdout:5966: done (2 left) 2020-06-12T02:38:25.910 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:25.910 INFO:tasks.rados.rados.0.smithi107.stdout:5967: seq_num 5966 ranges {0=729088,729088=475136,1204224=327680} 2020-06-12T02:38:25.914 INFO:tasks.rados.rados.0.smithi107.stdout:5967: writing smithi10729407-4 from 0 to 729088 tid 1 2020-06-12T02:38:25.917 INFO:tasks.rados.rados.0.smithi107.stdout:5967: writing smithi10729407-4 from 729088 to 1204224 tid 2 2020-06-12T02:38:25.919 INFO:tasks.rados.rados.0.smithi107.stdout:5967: writing smithi10729407-4 from 1204224 to 1531904 tid 3 2020-06-12T02:38:25.919 INFO:tasks.rados.rados.0.smithi107.stdout:5968: delete oid 9 current snap is 0 2020-06-12T02:38:25.919 INFO:tasks.rados.rados.0.smithi107.stdout:5964: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:25.920 INFO:tasks.rados.rados.0.smithi107.stdout:5964: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:25.920 INFO:tasks.rados.rados.0.smithi107.stdout:5964: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:25.920 INFO:tasks.rados.rados.0.smithi107.stdout:5964: finishing write tid 5 to smithi10729407-12 2020-06-12T02:38:25.920 INFO:tasks.rados.rados.0.smithi107.stdout:5964: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:25.920 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1064 (ObjNum 5963 snap 0 seq_num 5963) dirty exists 2020-06-12T02:38:25.921 INFO:tasks.rados.rados.0.smithi107.stdout:5964: left oid 12 (ObjNum 5963 snap 0 seq_num 5963) 2020-06-12T02:38:25.926 INFO:tasks.rados.rados.0.smithi107.stdout:5964: done (3 left) 2020-06-12T02:38:25.926 INFO:tasks.rados.rados.0.smithi107.stdout:5968: done (2 left) 2020-06-12T02:38:25.926 INFO:tasks.rados.rados.0.smithi107.stdout:5969: delete oid 7 current snap is 0 2020-06-12T02:38:25.926 INFO:tasks.rados.rados.0.smithi107.stdout:5967: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:25.927 INFO:tasks.rados.rados.0.smithi107.stdout:5967: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:25.928 INFO:tasks.rados.rados.0.smithi107.stdout:5967: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:25.928 INFO:tasks.rados.rados.0.smithi107.stdout:5969: done (2 left) 2020-06-12T02:38:25.929 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:25.929 INFO:tasks.rados.rados.0.smithi107.stdout:5970: seq_num 5969 ranges {0=778240,778240=581632,1359872=221184} 2020-06-12T02:38:25.932 INFO:tasks.rados.rados.0.smithi107.stdout:5970: writing smithi10729407-16 from 0 to 778240 tid 1 2020-06-12T02:38:25.935 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:25.936 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:25.936 INFO:tasks.rados.rados.0.smithi107.stdout:5970: writing smithi10729407-16 from 778240 to 1359872 tid 2 2020-06-12T02:38:25.939 INFO:tasks.rados.rados.0.smithi107.stdout:5970: writing smithi10729407-16 from 1359872 to 1581056 tid 3 2020-06-12T02:38:25.939 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:38:25.939 INFO:tasks.rados.rados.0.smithi107.stdout:5971: seq_num 5970 ranges {2097152=794624,2891776=466944,3358720=335872} 2020-06-12T02:38:25.948 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-06-12T02:38:25.952 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-06-12T02:38:25.953 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:25.950+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:25.955 INFO:tasks.rados.rados.0.smithi107.stdout:5971: writing smithi10729407-13 from 2097152 to 2891776 tid 1 2020-06-12T02:38:25.958 INFO:tasks.rados.rados.0.smithi107.stdout:5971: writing smithi10729407-13 from 2891776 to 3358720 tid 2 2020-06-12T02:38:25.960 INFO:tasks.rados.rados.0.smithi107.stdout:5971: writing smithi10729407-13 from 3358720 to 3694592 tid 3 2020-06-12T02:38:25.961 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:25.961 INFO:tasks.rados.rados.0.smithi107.stdout:5972: seq_num 5971 ranges {0=409600,409600=753664,1163264=450560} 2020-06-12T02:38:25.963 INFO:tasks.rados.rados.0.smithi107.stdout:5972: writing smithi10729407-1 from 0 to 409600 tid 1 2020-06-12T02:38:25.967 INFO:tasks.rados.rados.0.smithi107.stdout:5972: writing smithi10729407-1 from 409600 to 1163264 tid 2 2020-06-12T02:38:25.971 INFO:tasks.rados.rados.0.smithi107.stdout:5972: writing smithi10729407-1 from 1163264 to 1613824 tid 3 2020-06-12T02:38:25.971 INFO:tasks.rados.rados.0.smithi107.stdout:5973: delete oid 5 current snap is 0 2020-06-12T02:38:25.972 INFO:tasks.rados.rados.0.smithi107.stdout:5967: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:25.972 INFO:tasks.rados.rados.0.smithi107.stdout:5967: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:25.972 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1107 (ObjNum 5966 snap 0 seq_num 5966) dirty exists 2020-06-12T02:38:25.972 INFO:tasks.rados.rados.0.smithi107.stdout:5967: left oid 4 (ObjNum 5966 snap 0 seq_num 5966) 2020-06-12T02:38:25.973 INFO:tasks.rados.rados.0.smithi107.stdout:5970: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:25.974 INFO:tasks.rados.rados.0.smithi107.stdout:5970: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:25.974 INFO:tasks.rados.rados.0.smithi107.stdout:5970: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:25.974 INFO:tasks.rados.rados.0.smithi107.stdout:5970: finishing write tid 5 to smithi10729407-16 2020-06-12T02:38:25.974 INFO:tasks.rados.rados.0.smithi107.stdout:5970: finishing write tid 6 to smithi10729407-16 2020-06-12T02:38:25.975 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1051 (ObjNum 5969 snap 0 seq_num 5969) dirty exists 2020-06-12T02:38:25.975 INFO:tasks.rados.rados.0.smithi107.stdout:5970: left oid 16 (ObjNum 5969 snap 0 seq_num 5969) 2020-06-12T02:38:25.975 INFO:tasks.rados.rados.0.smithi107.stdout:5971: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:25.975 INFO:tasks.rados.rados.0.smithi107.stdout:5972: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:25.975 INFO:tasks.rados.rados.0.smithi107.stdout:5971: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:25.976 INFO:tasks.rados.rados.0.smithi107.stdout:5971: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:25.976 INFO:tasks.rados.rados.0.smithi107.stdout:5971: finishing write tid 5 to smithi10729407-13 2020-06-12T02:38:25.977 INFO:tasks.rados.rados.0.smithi107.stdout:5967: done (5 left) 2020-06-12T02:38:25.977 INFO:tasks.rados.rados.0.smithi107.stdout:5970: done (4 left) 2020-06-12T02:38:25.977 INFO:tasks.rados.rados.0.smithi107.stdout:5973: done (3 left) 2020-06-12T02:38:25.977 INFO:tasks.rados.rados.0.smithi107.stdout:5974: delete oid 2 current snap is 0 2020-06-12T02:38:25.978 INFO:tasks.rados.rados.0.smithi107.stdout:5971: finishing write tid 6 to smithi10729407-13 2020-06-12T02:38:25.979 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1139 (ObjNum 5970 snap 0 seq_num 5970) dirty exists 2020-06-12T02:38:25.979 INFO:tasks.rados.rados.0.smithi107.stdout:5971: left oid 13 (ObjNum 5970 snap 0 seq_num 5970) 2020-06-12T02:38:25.979 INFO:tasks.rados.rados.0.smithi107.stdout:5972: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:25.983 INFO:tasks.rados.rados.0.smithi107.stdout:5971: done (3 left) 2020-06-12T02:38:25.983 INFO:tasks.rados.rados.0.smithi107.stdout:5974: done (2 left) 2020-06-12T02:38:25.983 INFO:tasks.rados.rados.0.smithi107.stdout:5975: delete oid 2 current snap is 0 2020-06-12T02:38:25.983 INFO:tasks.rados.rados.0.smithi107.stdout:5972: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:25.984 INFO:tasks.rados.rados.0.smithi107.stdout:5972: finishing write tid 5 to smithi10729407-1 2020-06-12T02:38:25.985 INFO:tasks.rados.rados.0.smithi107.stdout:5972: finishing write tid 6 to smithi10729407-1 2020-06-12T02:38:25.986 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1116 (ObjNum 5971 snap 0 seq_num 5971) dirty exists 2020-06-12T02:38:25.986 INFO:tasks.rados.rados.0.smithi107.stdout:5972: left oid 1 (ObjNum 5971 snap 0 seq_num 5971) 2020-06-12T02:38:25.989 INFO:tasks.rados.rados.0.smithi107.stdout:5972: done (2 left) 2020-06-12T02:38:25.989 INFO:tasks.rados.rados.0.smithi107.stdout:5975: done (1 left) 2020-06-12T02:38:25.989 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:25.990 INFO:tasks.rados.rados.0.smithi107.stdout:5976: seq_num 5975 ranges {1474560=475136,1949696=688128,2637824=516096} 2020-06-12T02:38:25.999 INFO:tasks.rados.rados.0.smithi107.stdout:5976: writing smithi10729407-12 from 1474560 to 1949696 tid 1 2020-06-12T02:38:26.003 INFO:tasks.rados.rados.0.smithi107.stdout:5976: writing smithi10729407-12 from 1949696 to 2637824 tid 2 2020-06-12T02:38:26.006 INFO:tasks.rados.rados.0.smithi107.stdout:5976: writing smithi10729407-12 from 2637824 to 3153920 tid 3 2020-06-12T02:38:26.007 INFO:tasks.rados.rados.0.smithi107.stdout:5977: delete oid 1 current snap is 0 2020-06-12T02:38:26.007 INFO:tasks.rados.rados.0.smithi107.stdout:5976: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:26.009 INFO:tasks.rados.rados.0.smithi107.stdout:5977: done (2 left) 2020-06-12T02:38:26.009 INFO:tasks.rados.rados.0.smithi107.stdout:5978: delete oid 4 current snap is 0 2020-06-12T02:38:26.014 INFO:tasks.rados.rados.0.smithi107.stdout:5978: done (2 left) 2020-06-12T02:38:26.015 INFO:tasks.rados.rados.0.smithi107.stdout:5979: delete oid 5 current snap is 0 2020-06-12T02:38:26.015 INFO:tasks.rados.rados.0.smithi107.stdout:5976: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:26.016 INFO:tasks.rados.rados.0.smithi107.stdout:5976: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:26.017 INFO:tasks.rados.rados.0.smithi107.stdout:5976: finishing write tid 5 to smithi10729407-12 2020-06-12T02:38:26.019 INFO:tasks.rados.rados.0.smithi107.stdout:5976: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:26.019 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1068 (ObjNum 5975 snap 0 seq_num 5975) dirty exists 2020-06-12T02:38:26.019 INFO:tasks.rados.rados.0.smithi107.stdout:5976: left oid 12 (ObjNum 5975 snap 0 seq_num 5975) 2020-06-12T02:38:26.037 INFO:tasks.rados.rados.0.smithi107.stdout:5976: done (2 left) 2020-06-12T02:38:26.038 INFO:tasks.rados.rados.0.smithi107.stdout:5979: done (1 left) 2020-06-12T02:38:26.038 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:26.038 INFO:tasks.rados.rados.0.smithi107.stdout:5980: seq_num 5979 ranges {0=540672,540672=671744,1212416=532480} 2020-06-12T02:38:26.038 INFO:tasks.rados.rados.0.smithi107.stdout:5980: writing smithi10729407-9 from 0 to 540672 tid 1 2020-06-12T02:38:26.038 INFO:tasks.rados.rados.0.smithi107.stdout:5980: writing smithi10729407-9 from 540672 to 1212416 tid 2 2020-06-12T02:38:26.039 INFO:tasks.rados.rados.0.smithi107.stdout:5980: writing smithi10729407-9 from 1212416 to 1744896 tid 3 2020-06-12T02:38:26.039 INFO:tasks.rados.rados.0.smithi107.stdout:5981: delete oid 4 current snap is 0 2020-06-12T02:38:26.039 INFO:tasks.rados.rados.0.smithi107.stdout:5980: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:26.040 INFO:tasks.rados.rados.0.smithi107.stdout:5981: done (2 left) 2020-06-12T02:38:26.040 INFO:tasks.rados.rados.0.smithi107.stdout:5982: delete oid 20 current snap is 0 2020-06-12T02:38:26.040 INFO:tasks.rados.rados.0.smithi107.stdout:5980: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:26.041 INFO:tasks.rados.rados.0.smithi107.stdout:5980: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:26.041 INFO:tasks.rados.rados.0.smithi107.stdout:5980: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:26.041 INFO:tasks.rados.rados.0.smithi107.stdout:5982: done (2 left) 2020-06-12T02:38:26.041 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:26.041 INFO:tasks.rados.rados.0.smithi107.stdout:5983: seq_num 5982 ranges {0=589824,589824=729088,1318912=425984,1744896=49152} 2020-06-12T02:38:26.043 INFO:tasks.rados.rados.0.smithi107.stdout:5983: writing smithi10729407-4 from 0 to 589824 tid 1 2020-06-12T02:38:26.053 INFO:tasks.rados.rados.0.smithi107.stdout:5983: writing smithi10729407-4 from 589824 to 1318912 tid 2 2020-06-12T02:38:26.054 INFO:tasks.rados.rados.0.smithi107.stdout:5983: writing smithi10729407-4 from 1318912 to 1744896 tid 3 2020-06-12T02:38:26.054 INFO:tasks.rados.rados.0.smithi107.stdout:5983: writing smithi10729407-4 from 1744896 to 1794048 tid 4 2020-06-12T02:38:26.054 INFO:tasks.rados.rados.0.smithi107.stdout:5984: delete oid 7 current snap is 0 2020-06-12T02:38:26.054 INFO:tasks.rados.rados.0.smithi107.stdout:5980: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:26.054 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1090 (ObjNum 5979 snap 0 seq_num 5979) dirty exists 2020-06-12T02:38:26.055 INFO:tasks.rados.rados.0.smithi107.stdout:5980: left oid 9 (ObjNum 5979 snap 0 seq_num 5979) 2020-06-12T02:38:26.055 INFO:tasks.rados.rados.0.smithi107.stdout:5983: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:26.055 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:26.051+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:26.056 INFO:tasks.rados.rados.0.smithi107.stdout:5980: done (3 left) 2020-06-12T02:38:26.056 INFO:tasks.rados.rados.0.smithi107.stdout:5984: done (2 left) 2020-06-12T02:38:26.056 INFO:tasks.rados.rados.0.smithi107.stdout:5985: delete oid 5 current snap is 0 2020-06-12T02:38:26.058 INFO:tasks.rados.rados.0.smithi107.stdout:5985: done (2 left) 2020-06-12T02:38:26.059 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:38:26.059 INFO:tasks.rados.rados.0.smithi107.stdout:5986: seq_num 5985 ranges {0=647168,647168=425984,1073152=598016,1671168=180224} 2020-06-12T02:38:26.064 INFO:tasks.rados.rados.0.smithi107.stdout:5986: writing smithi10729407-2 from 0 to 647168 tid 1 2020-06-12T02:38:26.066 INFO:tasks.rados.rados.0.smithi107.stdout:5986: writing smithi10729407-2 from 647168 to 1073152 tid 2 2020-06-12T02:38:26.070 INFO:tasks.rados.rados.0.smithi107.stdout:5986: writing smithi10729407-2 from 1073152 to 1671168 tid 3 2020-06-12T02:38:26.072 INFO:tasks.rados.rados.0.smithi107.stdout:5986: writing smithi10729407-2 from 1671168 to 1851392 tid 4 2020-06-12T02:38:26.072 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:38:26.072 INFO:tasks.rados.rados.0.smithi107.stdout:5987: seq_num 5986 ranges {3694592=663552,4358144=712704,5070848=491520} 2020-06-12T02:38:26.098 INFO:tasks.rados.rados.0.smithi107.stdout:5987: writing smithi10729407-13 from 3694592 to 4358144 tid 1 2020-06-12T02:38:26.106 INFO:tasks.rados.rados.0.smithi107.stdout:5987: writing smithi10729407-13 from 4358144 to 5070848 tid 2 2020-06-12T02:38:26.108 INFO:tasks.rados.rados.0.smithi107.stdout:5987: writing smithi10729407-13 from 5070848 to 5562368 tid 3 2020-06-12T02:38:26.108 INFO:tasks.rados.rados.0.smithi107.stdout:5988: delete oid 19 current snap is 0 2020-06-12T02:38:26.108 INFO:tasks.rados.rados.0.smithi107.stdout:5983: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:26.108 INFO:tasks.rados.rados.0.smithi107.stdout:5983: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:26.108 INFO:tasks.rados.rados.0.smithi107.stdout:5983: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:26.109 INFO:tasks.rados.rados.0.smithi107.stdout:5983: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:26.109 INFO:tasks.rados.rados.0.smithi107.stdout:5986: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:26.109 INFO:tasks.rados.rados.0.smithi107.stdout:5983: finishing write tid 7 to smithi10729407-4 2020-06-12T02:38:26.109 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1114 (ObjNum 5982 snap 0 seq_num 5982) dirty exists 2020-06-12T02:38:26.109 INFO:tasks.rados.rados.0.smithi107.stdout:5983: left oid 4 (ObjNum 5982 snap 0 seq_num 5982) 2020-06-12T02:38:26.110 INFO:tasks.rados.rados.0.smithi107.stdout:5986: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:26.110 INFO:tasks.rados.rados.0.smithi107.stdout:5986: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:26.110 INFO:tasks.rados.rados.0.smithi107.stdout:5986: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:26.110 INFO:tasks.rados.rados.0.smithi107.stdout:5986: finishing write tid 6 to smithi10729407-2 2020-06-12T02:38:26.110 INFO:tasks.rados.rados.0.smithi107.stdout:5986: finishing write tid 7 to smithi10729407-2 2020-06-12T02:38:26.111 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 567 (ObjNum 5985 snap 0 seq_num 5985) dirty exists 2020-06-12T02:38:26.111 INFO:tasks.rados.rados.0.smithi107.stdout:5986: left oid 2 (ObjNum 5985 snap 0 seq_num 5985) 2020-06-12T02:38:26.111 INFO:tasks.rados.rados.0.smithi107.stdout:5983: done (4 left) 2020-06-12T02:38:26.111 INFO:tasks.rados.rados.0.smithi107.stdout:5986: done (3 left) 2020-06-12T02:38:26.112 INFO:tasks.rados.rados.0.smithi107.stdout:5988: done (2 left) 2020-06-12T02:38:26.112 INFO:tasks.rados.rados.0.smithi107.stdout:5989: delete oid 19 current snap is 0 2020-06-12T02:38:26.112 INFO:tasks.rados.rados.0.smithi107.stdout:5987: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:26.113 INFO:tasks.rados.rados.0.smithi107.stdout:5987: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:26.114 INFO:tasks.rados.rados.0.smithi107.stdout:5987: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:26.117 INFO:tasks.rados.rados.0.smithi107.stdout:5987: finishing write tid 5 to smithi10729407-13 2020-06-12T02:38:26.117 INFO:tasks.rados.rados.0.smithi107.stdout:5989: done (2 left) 2020-06-12T02:38:26.117 INFO:tasks.rados.rados.0.smithi107.stdout:5990: delete oid 3 current snap is 0 2020-06-12T02:38:26.118 INFO:tasks.rados.rados.0.smithi107.stdout:5987: finishing write tid 6 to smithi10729407-13 2020-06-12T02:38:26.118 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1144 (ObjNum 5986 snap 0 seq_num 5986) dirty exists 2020-06-12T02:38:26.118 INFO:tasks.rados.rados.0.smithi107.stdout:5987: left oid 13 (ObjNum 5986 snap 0 seq_num 5986) 2020-06-12T02:38:26.119 INFO:tasks.rados.rados.0.smithi107.stdout:5987: done (2 left) 2020-06-12T02:38:26.119 INFO:tasks.rados.rados.0.smithi107.stdout:5990: done (1 left) 2020-06-12T02:38:26.119 INFO:tasks.rados.rados.0.smithi107.stdout:5991: delete oid 13 current snap is 0 2020-06-12T02:38:26.123 INFO:tasks.rados.rados.0.smithi107.stdout:5991: done (1 left) 2020-06-12T02:38:26.123 INFO:tasks.rados.rados.0.smithi107.stdout:5992: delete oid 3 current snap is 0 2020-06-12T02:38:26.127 INFO:tasks.rados.rados.0.smithi107.stdout:5992: done (1 left) 2020-06-12T02:38:26.127 INFO:tasks.rados.rados.0.smithi107.stdout:5993: delete oid 7 current snap is 0 2020-06-12T02:38:26.129 INFO:tasks.rados.rados.0.smithi107.stdout:5993: done (1 left) 2020-06-12T02:38:26.129 INFO:tasks.rados.rados.0.smithi107.stdout:5994: delete oid 18 current snap is 0 2020-06-12T02:38:26.131 INFO:tasks.rados.rados.0.smithi107.stdout:5994: done (1 left) 2020-06-12T02:38:26.132 INFO:tasks.rados.rados.0.smithi107.stdout:5995: delete oid 6 current snap is 0 2020-06-12T02:38:26.133 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:26.133 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:26.134 INFO:tasks.rados.rados.0.smithi107.stdout:5995: done (1 left) 2020-06-12T02:38:26.134 INFO:tasks.rados.rados.0.smithi107.stdout:5996: delete oid 20 current snap is 0 2020-06-12T02:38:26.136 INFO:tasks.rados.rados.0.smithi107.stdout:5996: done (1 left) 2020-06-12T02:38:26.137 INFO:tasks.rados.rados.0.smithi107.stdout:5997: delete oid 4 current snap is 0 2020-06-12T02:38:26.138 INFO:tasks.rados.rados.0.smithi107.stdout:5997: done (1 left) 2020-06-12T02:38:26.138 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:26.139 INFO:tasks.rados.rados.0.smithi107.stdout:5998: seq_num 5997 ranges {2564096=442368,3006464=696320,3702784=540672,4243456=368640} 2020-06-12T02:38:26.139 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:38:26.143 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-06-12T02:38:26.155 INFO:tasks.rados.rados.0.smithi107.stdout:5998: writing smithi10729407-15 from 2564096 to 3006464 tid 1 2020-06-12T02:38:26.156 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:38:26.153+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:38:26.159 INFO:tasks.rados.rados.0.smithi107.stdout:5998: writing smithi10729407-15 from 3006464 to 3702784 tid 2 2020-06-12T02:38:26.161 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:38:26.162 INFO:tasks.rados.rados.0.smithi107.stdout:5998: writing smithi10729407-15 from 3702784 to 4243456 tid 3 2020-06-12T02:38:26.164 INFO:tasks.rados.rados.0.smithi107.stdout:5998: writing smithi10729407-15 from 4243456 to 4612096 tid 4 2020-06-12T02:38:26.165 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:38:26.165 INFO:tasks.rados.rados.0.smithi107.stdout:5999: seq_num 5998 ranges {1851392=458752,2310144=573440,2883584=745472,3629056=286720} 2020-06-12T02:38:26.177 INFO:tasks.rados.rados.0.smithi107.stdout:5999: writing smithi10729407-2 from 1851392 to 2310144 tid 1 2020-06-12T02:38:26.181 INFO:tasks.rados.rados.0.smithi107.stdout:5999: writing smithi10729407-2 from 2310144 to 2883584 tid 2 2020-06-12T02:38:26.183 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:38:26.185 INFO:tasks.rados.rados.0.smithi107.stdout:5999: writing smithi10729407-2 from 2883584 to 3629056 tid 3 2020-06-12T02:38:26.188 INFO:tasks.rados.rados.0.smithi107.stdout:5999: writing smithi10729407-2 from 3629056 to 3915776 tid 4 2020-06-12T02:38:26.188 INFO:tasks.rados.rados.0.smithi107.stdout:6000: delete oid 11 current snap is 0 2020-06-12T02:38:26.189 INFO:tasks.rados.rados.0.smithi107.stdout:5998: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:26.189 INFO:tasks.rados.rados.0.smithi107.stdout:5998: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:26.189 INFO:tasks.rados.rados.0.smithi107.stdout:5998: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:26.189 INFO:tasks.rados.rados.0.smithi107.stdout:5998: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:26.190 INFO:tasks.rados.rados.0.smithi107.stdout:5998: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:26.190 INFO:tasks.rados.rados.0.smithi107.stdout:5998: finishing write tid 7 to smithi10729407-15 2020-06-12T02:38:26.190 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1106 (ObjNum 5997 snap 0 seq_num 5997) dirty exists 2020-06-12T02:38:26.190 INFO:tasks.rados.rados.0.smithi107.stdout:5998: left oid 15 (ObjNum 5997 snap 0 seq_num 5997) 2020-06-12T02:38:26.190 INFO:tasks.rados.rados.0.smithi107.stdout:5999: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:26.200 INFO:tasks.rados.rados.0.smithi107.stdout:5998: done (3 left) 2020-06-12T02:38:26.201 INFO:tasks.rados.rados.0.smithi107.stdout:6000: done (2 left) 2020-06-12T02:38:26.201 INFO:tasks.rados.rados.0.smithi107.stdout:6001: delete oid 1 current snap is 0 2020-06-12T02:38:26.201 INFO:tasks.rados.rados.0.smithi107.stdout:5999: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:26.201 INFO:tasks.rados.rados.0.smithi107.stdout:6001: done (2 left) 2020-06-12T02:38:26.201 INFO:tasks.rados.rados.0.smithi107.stdout:6002: delete oid 13 current snap is 0 2020-06-12T02:38:26.202 INFO:tasks.rados.rados.0.smithi107.stdout:5999: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:26.202 INFO:tasks.rados.rados.0.smithi107.stdout:5999: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:26.202 INFO:tasks.rados.rados.0.smithi107.stdout:6002: done (2 left) 2020-06-12T02:38:26.202 INFO:tasks.rados.rados.0.smithi107.stdout:6003: delete oid 17 current snap is 0 2020-06-12T02:38:26.202 INFO:tasks.rados.rados.0.smithi107.stdout:5999: finishing write tid 6 to smithi10729407-2 2020-06-12T02:38:26.203 INFO:tasks.rados.rados.0.smithi107.stdout:5999: finishing write tid 7 to smithi10729407-2 2020-06-12T02:38:26.203 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 572 (ObjNum 5998 snap 0 seq_num 5998) dirty exists 2020-06-12T02:38:26.203 INFO:tasks.rados.rados.0.smithi107.stdout:5999: left oid 2 (ObjNum 5998 snap 0 seq_num 5998) 2020-06-12T02:38:26.210 INFO:tasks.rados.rados.0.smithi107.stdout:5999: done (2 left) 2020-06-12T02:38:26.210 INFO:tasks.rados.rados.0.smithi107.stdout:6003: done (1 left) 2020-06-12T02:38:26.210 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:26.210 INFO:tasks.rados.rados.0.smithi107.stdout:6004: seq_num 6003 ranges {0=540672,540672=450560,991232=483328,1474560=622592,2097152=49152} 2020-06-12T02:38:26.212 INFO:tasks.rados.rados.0.smithi107.stdout:6004: writing smithi10729407-17 from 0 to 540672 tid 1 2020-06-12T02:38:26.212 INFO:tasks.rados.rados.0.smithi107.stdout:6004: writing smithi10729407-17 from 540672 to 991232 tid 2 2020-06-12T02:38:26.212 INFO:tasks.rados.rados.0.smithi107.stdout:6004: writing smithi10729407-17 from 991232 to 1474560 tid 3 2020-06-12T02:38:26.212 INFO:tasks.rados.rados.0.smithi107.stdout:6004: writing smithi10729407-17 from 1474560 to 2097152 tid 4 2020-06-12T02:38:26.212 INFO:tasks.rados.rados.0.smithi107.stdout:6004: writing smithi10729407-17 from 2097152 to 2146304 tid 5 2020-06-12T02:38:26.213 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:26.213 INFO:tasks.rados.rados.0.smithi107.stdout:6005: seq_num 6004 ranges {0=565248,565248=729088,1294336=696320,1990656=180224} 2020-06-12T02:38:26.213 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:38:26.215 INFO:tasks.rados.rados.0.smithi107.stdout:6005: writing smithi10729407-19 from 0 to 565248 tid 1 2020-06-12T02:38:26.220 INFO:tasks.rados.rados.0.smithi107.stdout:6005: writing smithi10729407-19 from 565248 to 1294336 tid 2 2020-06-12T02:38:26.224 INFO:tasks.rados.rados.0.smithi107.stdout:6005: writing smithi10729407-19 from 1294336 to 1990656 tid 3 2020-06-12T02:38:26.226 INFO:tasks.rados.rados.0.smithi107.stdout:6005: writing smithi10729407-19 from 1990656 to 2170880 tid 4 2020-06-12T02:38:26.226 INFO:tasks.rados.rados.0.smithi107.stdout:6006: delete oid 8 current snap is 0 2020-06-12T02:38:26.227 INFO:tasks.rados.rados.0.smithi107.stdout:6004: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:26.227 INFO:tasks.rados.rados.0.smithi107.stdout:6004: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:26.228 INFO:tasks.rados.rados.0.smithi107.stdout:6004: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:26.228 INFO:tasks.rados.rados.0.smithi107.stdout:6004: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:26.229 INFO:tasks.rados.rados.0.smithi107.stdout:6004: finishing write tid 5 to smithi10729407-17 2020-06-12T02:38:26.229 INFO:tasks.rados.rados.0.smithi107.stdout:6005: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:26.229 INFO:tasks.rados.rados.0.smithi107.stdout:6004: finishing write tid 7 to smithi10729407-17 2020-06-12T02:38:26.230 INFO:tasks.rados.rados.0.smithi107.stdout:6004: finishing write tid 8 to smithi10729407-17 2020-06-12T02:38:26.230 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1122 (ObjNum 6003 snap 0 seq_num 6003) dirty exists 2020-06-12T02:38:26.230 INFO:tasks.rados.rados.0.smithi107.stdout:6004: left oid 17 (ObjNum 6003 snap 0 seq_num 6003) 2020-06-12T02:38:26.233 INFO:tasks.rados.rados.0.smithi107.stdout:6004: done (3 left) 2020-06-12T02:38:26.234 INFO:tasks.rados.rados.0.smithi107.stdout:6006: done (2 left) 2020-06-12T02:38:26.236 INFO:tasks.rados.rados.0.smithi107.stdout:6007: delete oid 15 current snap is 0 2020-06-12T02:38:26.236 INFO:tasks.rados.rados.0.smithi107.stdout:6005: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:26.237 INFO:tasks.rados.rados.0.smithi107.stdout:6005: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:26.237 INFO:tasks.rados.rados.0.smithi107.stdout:6007: done (2 left) 2020-06-12T02:38:26.237 INFO:tasks.rados.rados.0.smithi107.stdout:6008: delete oid 9 current snap is 0 2020-06-12T02:38:26.240 INFO:tasks.rados.rados.0.smithi107.stdout:6005: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:26.241 INFO:tasks.rados.rados.0.smithi107.stdout:6005: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:26.241 INFO:tasks.rados.rados.0.smithi107.stdout:6008: done (2 left) 2020-06-12T02:38:26.241 INFO:tasks.rados.rados.0.smithi107.stdout:6009: delete oid 12 current snap is 0 2020-06-12T02:38:26.244 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:38:26.246 INFO:tasks.rados.rados.0.smithi107.stdout:6005: finishing write tid 7 to smithi10729407-19 2020-06-12T02:38:26.246 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1075 (ObjNum 6004 snap 0 seq_num 6004) dirty exists 2020-06-12T02:38:26.246 INFO:tasks.rados.rados.0.smithi107.stdout:6005: left oid 19 (ObjNum 6004 snap 0 seq_num 6004) 2020-06-12T02:38:26.248 INFO:tasks.rados.rados.0.smithi107.stdout:6005: done (2 left) 2020-06-12T02:38:26.248 INFO:tasks.rados.rados.0.smithi107.stdout:6009: done (1 left) 2020-06-12T02:38:26.248 INFO:tasks.rados.rados.0.smithi107.stdout:6010: delete oid 19 current snap is 0 2020-06-12T02:38:26.259 INFO:tasks.rados.rados.0.smithi107.stdout:6010: done (1 left) 2020-06-12T02:38:26.260 INFO:tasks.rados.rados.0.smithi107.stdout:6011: delete oid 20 current snap is 0 2020-06-12T02:38:26.260 INFO:tasks.rados.rados.0.smithi107.stdout:6011: done (1 left) 2020-06-12T02:38:26.260 INFO:tasks.rados.rados.0.smithi107.stdout:6012: delete oid 14 current snap is 0 2020-06-12T02:38:26.260 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:26.257+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:26.262 INFO:tasks.rados.rados.0.smithi107.stdout:6012: done (1 left) 2020-06-12T02:38:26.262 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:26.262 INFO:tasks.rados.rados.0.smithi107.stdout:6013: seq_num 6012 ranges {0=696320,696320=655360,1351680=655360,2007040=294912} 2020-06-12T02:38:26.266 INFO:tasks.rados.rados.0.smithi107.stdout:6013: writing smithi10729407-19 from 0 to 696320 tid 1 2020-06-12T02:38:26.267 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:38:26.270 INFO:tasks.rados.rados.0.smithi107.stdout:6013: writing smithi10729407-19 from 696320 to 1351680 tid 2 2020-06-12T02:38:26.274 INFO:tasks.rados.rados.0.smithi107.stdout:6013: writing smithi10729407-19 from 1351680 to 2007040 tid 3 2020-06-12T02:38:26.275 INFO:tasks.rados.rados.0.smithi107.stdout:6013: writing smithi10729407-19 from 2007040 to 2301952 tid 4 2020-06-12T02:38:26.276 INFO:tasks.rados.rados.0.smithi107.stdout:6014: delete oid 2 current snap is 0 2020-06-12T02:38:26.276 INFO:tasks.rados.rados.0.smithi107.stdout:6013: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:26.278 INFO:tasks.rados.rados.0.smithi107.stdout:6014: done (2 left) 2020-06-12T02:38:26.278 INFO:tasks.rados.rados.0.smithi107.stdout:6015: delete oid 17 current snap is 0 2020-06-12T02:38:26.281 INFO:tasks.rados.rados.0.smithi107.stdout:6015: done (2 left) 2020-06-12T02:38:26.282 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:26.282 INFO:tasks.rados.rados.0.smithi107.stdout:6016: seq_num 6015 ranges {0=745472,745472=761856,1507328=425984,1933312=417792} 2020-06-12T02:38:26.286 INFO:tasks.rados.rados.0.smithi107.stdout:6016: writing smithi10729407-13 from 0 to 745472 tid 1 2020-06-12T02:38:26.291 INFO:tasks.rados.rados.0.smithi107.stdout:6016: writing smithi10729407-13 from 745472 to 1507328 tid 2 2020-06-12T02:38:26.293 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:38:26.294 INFO:tasks.rados.rados.0.smithi107.stdout:6016: writing smithi10729407-13 from 1507328 to 1933312 tid 3 2020-06-12T02:38:26.296 INFO:tasks.rados.rados.0.smithi107.stdout:6016: writing smithi10729407-13 from 1933312 to 2351104 tid 4 2020-06-12T02:38:26.297 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:26.297 INFO:tasks.rados.rados.0.smithi107.stdout:6017: seq_num 6016 ranges {0=761856,761856=647168,1409024=630784,2039808=327680} 2020-06-12T02:38:26.301 INFO:tasks.rados.rados.0.smithi107.stdout:6017: writing smithi10729407-8 from 0 to 761856 tid 1 2020-06-12T02:38:26.303 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:26.304 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:38:26.304 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2020-06-12T02:38:26.304 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2020-06-12T02:38:26.304 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:26.305 INFO:tasks.rados.rados.0.smithi107.stdout:6017: writing smithi10729407-8 from 761856 to 1409024 tid 2 2020-06-12T02:38:26.308 INFO:tasks.rados.rados.0.smithi107.stdout:6017: writing smithi10729407-8 from 1409024 to 2039808 tid 3 2020-06-12T02:38:26.311 INFO:tasks.rados.rados.0.smithi107.stdout:6017: writing smithi10729407-8 from 2039808 to 2367488 tid 4 2020-06-12T02:38:26.311 INFO:tasks.rados.rados.0.smithi107.stdout:6018: delete oid 7 current snap is 0 2020-06-12T02:38:26.312 INFO:tasks.rados.rados.0.smithi107.stdout:6013: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:26.312 INFO:tasks.rados.rados.0.smithi107.stdout:6013: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:26.312 INFO:tasks.rados.rados.0.smithi107.stdout:6013: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:26.313 INFO:tasks.rados.rados.0.smithi107.stdout:6013: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:26.313 INFO:tasks.rados.rados.0.smithi107.stdout:6013: finishing write tid 7 to smithi10729407-19 2020-06-12T02:38:26.313 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1082 (ObjNum 6012 snap 0 seq_num 6012) dirty exists 2020-06-12T02:38:26.313 INFO:tasks.rados.rados.0.smithi107.stdout:6013: left oid 19 (ObjNum 6012 snap 0 seq_num 6012) 2020-06-12T02:38:26.313 INFO:tasks.rados.rados.0.smithi107.stdout:6016: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:26.314 INFO:tasks.rados.rados.0.smithi107.stdout:6016: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:26.314 INFO:tasks.rados.rados.0.smithi107.stdout:6016: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:26.314 INFO:tasks.rados.rados.0.smithi107.stdout:6016: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:26.314 INFO:tasks.rados.rados.0.smithi107.stdout:6016: finishing write tid 6 to smithi10729407-13 2020-06-12T02:38:26.315 INFO:tasks.rados.rados.0.smithi107.stdout:6016: finishing write tid 7 to smithi10729407-13 2020-06-12T02:38:26.316 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1152 (ObjNum 6015 snap 0 seq_num 6015) dirty exists 2020-06-12T02:38:26.316 INFO:tasks.rados.rados.0.smithi107.stdout:6016: left oid 13 (ObjNum 6015 snap 0 seq_num 6015) 2020-06-12T02:38:26.317 INFO:tasks.rados.rados.0.smithi107.stdout:6017: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:26.317 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-06-12T02:38:26.319 INFO:tasks.rados.rados.0.smithi107.stdout:6013: done (4 left) 2020-06-12T02:38:26.319 INFO:tasks.rados.rados.0.smithi107.stdout:6016: done (3 left) 2020-06-12T02:38:26.319 INFO:tasks.rados.rados.0.smithi107.stdout:6018: done (2 left) 2020-06-12T02:38:26.319 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:38:26.319 INFO:tasks.rados.rados.0.smithi107.stdout:6019: seq_num 6018 ranges {0=794624,794624=409600,1204224=647168,1851392=491520,2342912=57344} 2020-06-12T02:38:26.320 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2020-06-12T02:38:26.320 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-06-12T02:38:26.321 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-06-12T02:38:26.321 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-06-12T02:38:26.321 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-06-12T02:38:26.322 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-06-12T02:38:26.322 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-06-12T02:38:26.322 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-06-12T02:38:26.322 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-06-12T02:38:26.325 INFO:tasks.rados.rados.0.smithi107.stdout:6019: writing smithi10729407-7 from 0 to 794624 tid 1 2020-06-12T02:38:26.325 INFO:tasks.rados.rados.0.smithi107.stdout:6019: writing smithi10729407-7 from 794624 to 1204224 tid 2 2020-06-12T02:38:26.329 INFO:tasks.rados.rados.0.smithi107.stdout:6019: writing smithi10729407-7 from 1204224 to 1851392 tid 3 2020-06-12T02:38:26.337 INFO:tasks.rados.rados.0.smithi107.stdout:6019: writing smithi10729407-7 from 1851392 to 2342912 tid 4 2020-06-12T02:38:26.337 INFO:tasks.rados.rados.0.smithi107.stdout:6019: writing smithi10729407-7 from 2342912 to 2400256 tid 5 2020-06-12T02:38:26.337 INFO:tasks.rados.rados.0.smithi107.stdout:6020: delete oid 19 current snap is 0 2020-06-12T02:38:26.338 INFO:tasks.rados.rados.0.smithi107.stdout:6017: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:26.338 INFO:tasks.rados.rados.0.smithi107.stdout:6017: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:26.338 INFO:tasks.rados.rados.0.smithi107.stdout:6017: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:26.339 INFO:tasks.rados.rados.0.smithi107.stdout:6017: finishing write tid 6 to smithi10729407-8 2020-06-12T02:38:26.339 INFO:tasks.rados.rados.0.smithi107.stdout:6017: finishing write tid 7 to smithi10729407-8 2020-06-12T02:38:26.339 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1113 (ObjNum 6016 snap 0 seq_num 6016) dirty exists 2020-06-12T02:38:26.340 INFO:tasks.rados.rados.0.smithi107.stdout:6017: left oid 8 (ObjNum 6016 snap 0 seq_num 6016) 2020-06-12T02:38:26.340 INFO:tasks.rados.rados.0.smithi107.stdout:6019: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:26.340 INFO:tasks.rados.rados.0.smithi107.stdout:6017: done (3 left) 2020-06-12T02:38:26.340 INFO:tasks.rados.rados.0.smithi107.stdout:6020: done (2 left) 2020-06-12T02:38:26.341 INFO:tasks.rados.rados.0.smithi107.stdout:6021: delete oid 19 current snap is 0 2020-06-12T02:38:26.342 INFO:tasks.rados.rados.0.smithi107.stdout:6021: done (2 left) 2020-06-12T02:38:26.342 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:26.343 INFO:tasks.rados.rados.0.smithi107.stdout:6022: seq_num 6021 ranges {0=40960} 2020-06-12T02:38:26.343 INFO:tasks.rados.rados.0.smithi107.stdout:6022: writing smithi10729407-4 from 0 to 40960 tid 1 2020-06-12T02:38:26.343 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:26.344 INFO:tasks.rados.rados.0.smithi107.stdout:6023: seq_num 6022 ranges {2367488=57344} 2020-06-12T02:38:26.353 INFO:tasks.rados.rados.0.smithi107.stdout:6023: writing smithi10729407-8 from 2367488 to 2424832 tid 1 2020-06-12T02:38:26.353 INFO:tasks.rados.rados.0.smithi107.stdout:6024: delete oid 6 current snap is 0 2020-06-12T02:38:26.354 INFO:tasks.rados.rados.0.smithi107.stdout:6019: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:26.354 INFO:tasks.rados.rados.0.smithi107.stdout:6022: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:26.365 INFO:tasks.rados.rados.0.smithi107.stdout:6022: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:26.365 INFO:tasks.rados.rados.0.smithi107.stdout:6019: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:26.366 INFO:tasks.rados.rados.0.smithi107.stdout:6019: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:26.366 INFO:tasks.rados.rados.0.smithi107.stdout:6022: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:26.366 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1125 (ObjNum 6021 snap 0 seq_num 6021) dirty exists 2020-06-12T02:38:26.366 INFO:tasks.rados.rados.0.smithi107.stdout:6022: left oid 4 (ObjNum 6021 snap 0 seq_num 6021) 2020-06-12T02:38:26.366 INFO:tasks.rados.rados.0.smithi107.stdout:6019: finishing write tid 5 to smithi10729407-7 2020-06-12T02:38:26.367 INFO:tasks.rados.rados.0.smithi107.stdout:6019: finishing write tid 7 to smithi10729407-7 2020-06-12T02:38:26.367 INFO:tasks.rados.rados.0.smithi107.stdout:6019: finishing write tid 8 to smithi10729407-7 2020-06-12T02:38:26.367 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1159 (ObjNum 6018 snap 0 seq_num 6018) dirty exists 2020-06-12T02:38:26.367 INFO:tasks.rados.rados.0.smithi107.stdout:6019: left oid 7 (ObjNum 6018 snap 0 seq_num 6018) 2020-06-12T02:38:26.369 INFO:tasks.rados.rados.0.smithi107.stdout:6019: done (4 left) 2020-06-12T02:38:26.370 INFO:tasks.rados.rados.0.smithi107.stdout:6022: done (3 left) 2020-06-12T02:38:26.370 INFO:tasks.rados.rados.0.smithi107.stdout:6024: done (2 left) 2020-06-12T02:38:26.370 INFO:tasks.rados.rados.0.smithi107.stdout:6025: delete oid 6 current snap is 0 2020-06-12T02:38:26.370 INFO:tasks.rados.rados.0.smithi107.stdout:6023: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:26.371 INFO:tasks.rados.rados.0.smithi107.stdout:6025: done (2 left) 2020-06-12T02:38:26.371 INFO:tasks.rados.rados.0.smithi107.stdout:6026: delete oid 1 current snap is 0 2020-06-12T02:38:26.371 INFO:tasks.rados.rados.0.smithi107.stdout:6023: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:26.371 INFO:tasks.rados.rados.0.smithi107.stdout:6023: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:26.371 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1115 (ObjNum 6022 snap 0 seq_num 6022) dirty exists 2020-06-12T02:38:26.372 INFO:tasks.rados.rados.0.smithi107.stdout:6023: left oid 8 (ObjNum 6022 snap 0 seq_num 6022) 2020-06-12T02:38:26.372 INFO:tasks.rados.rados.0.smithi107.stdout:6023: done (2 left) 2020-06-12T02:38:26.372 INFO:tasks.rados.rados.0.smithi107.stdout:6026: done (1 left) 2020-06-12T02:38:26.372 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:26.372 INFO:tasks.rados.rados.0.smithi107.stdout:6027: seq_num 6026 ranges {40960=131072} 2020-06-12T02:38:26.372 INFO:tasks.rados.rados.0.smithi107.stdout:6027: writing smithi10729407-4 from 40960 to 172032 tid 1 2020-06-12T02:38:26.373 INFO:tasks.rados.rados.0.smithi107.stdout:6028: delete oid 16 current snap is 0 2020-06-12T02:38:26.373 INFO:tasks.rados.rados.0.smithi107.stdout:6028: done (2 left) 2020-06-12T02:38:26.373 INFO:tasks.rados.rados.0.smithi107.stdout:6029: delete oid 17 current snap is 0 2020-06-12T02:38:26.373 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:26.366+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:26.374 INFO:tasks.rados.rados.0.smithi107.stdout:6027: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:26.384 INFO:tasks.rados.rados.0.smithi107.stdout:6029: done (2 left) 2020-06-12T02:38:26.394 INFO:tasks.rados.rados.0.smithi107.stdout:6030: delete oid 18 current snap is 0 2020-06-12T02:38:26.394 INFO:tasks.rados.rados.0.smithi107.stdout:6027: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:26.394 INFO:tasks.rados.rados.0.smithi107.stdout:6027: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:26.395 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1128 (ObjNum 6026 snap 0 seq_num 6026) dirty exists 2020-06-12T02:38:26.395 INFO:tasks.rados.rados.0.smithi107.stdout:6027: left oid 4 (ObjNum 6026 snap 0 seq_num 6026) 2020-06-12T02:38:26.395 INFO:tasks.rados.rados.0.smithi107.stdout:6027: done (2 left) 2020-06-12T02:38:26.395 INFO:tasks.rados.rados.0.smithi107.stdout:6030: done (1 left) 2020-06-12T02:38:26.396 INFO:tasks.rados.rados.0.smithi107.stdout:6031: delete oid 8 current snap is 0 2020-06-12T02:38:26.396 INFO:tasks.rados.rados.0.smithi107.stdout:6031: done (1 left) 2020-06-12T02:38:26.396 INFO:tasks.rados.rados.0.smithi107.stdout:6032: delete oid 8 current snap is 0 2020-06-12T02:38:26.398 INFO:tasks.rados.rados.0.smithi107.stdout:6032: done (1 left) 2020-06-12T02:38:26.398 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:26.398 INFO:tasks.rados.rados.0.smithi107.stdout:6033: seq_num 6032 ranges {0=229376} 2020-06-12T02:38:26.398 INFO:tasks.rados.rados.0.smithi107.stdout:6033: writing smithi10729407-8 from 0 to 229376 tid 1 2020-06-12T02:38:26.398 INFO:tasks.rados.rados.0.smithi107.stdout:6034: delete oid 2 current snap is 0 2020-06-12T02:38:26.399 INFO:tasks.rados.rados.0.smithi107.stdout:6034: done (2 left) 2020-06-12T02:38:26.399 INFO:tasks.rados.rados.0.smithi107.stdout:6035: delete oid 9 current snap is 0 2020-06-12T02:38:26.399 INFO:tasks.rados.rados.0.smithi107.stdout:6033: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:26.399 INFO:tasks.rados.rados.0.smithi107.stdout:6035: done (2 left) 2020-06-12T02:38:26.399 INFO:tasks.rados.rados.0.smithi107.stdout:6036: delete oid 20 current snap is 0 2020-06-12T02:38:26.400 INFO:tasks.rados.rados.0.smithi107.stdout:6033: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:26.405 INFO:tasks.rados.rados.0.smithi107.stdout:6033: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:26.405 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1119 (ObjNum 6032 snap 0 seq_num 6032) dirty exists 2020-06-12T02:38:26.405 INFO:tasks.rados.rados.0.smithi107.stdout:6033: left oid 8 (ObjNum 6032 snap 0 seq_num 6032) 2020-06-12T02:38:26.406 INFO:tasks.rados.rados.0.smithi107.stdout:6033: done (2 left) 2020-06-12T02:38:26.406 INFO:tasks.rados.rados.0.smithi107.stdout:6036: done (1 left) 2020-06-12T02:38:26.406 INFO:tasks.rados.rados.0.smithi107.stdout:6037: delete oid 7 current snap is 0 2020-06-12T02:38:26.410 INFO:tasks.rados.rados.0.smithi107.stdout:6037: done (1 left) 2020-06-12T02:38:26.411 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:38:26.411 INFO:tasks.rados.rados.0.smithi107.stdout:6038: seq_num 6037 ranges {0=311296} 2020-06-12T02:38:26.412 INFO:tasks.rados.rados.0.smithi107.stdout:6038: writing smithi10729407-20 from 0 to 311296 tid 1 2020-06-12T02:38:26.412 INFO:tasks.rados.rados.0.smithi107.stdout:6039: delete oid 1 current snap is 0 2020-06-12T02:38:26.415 INFO:tasks.rados.rados.0.smithi107.stdout:6039: done (2 left) 2020-06-12T02:38:26.416 INFO:tasks.rados.rados.0.smithi107.stdout:6040: delete oid 18 current snap is 0 2020-06-12T02:38:26.416 INFO:tasks.rados.rados.0.smithi107.stdout:6038: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:26.417 INFO:tasks.rados.rados.0.smithi107.stdout:6040: done (2 left) 2020-06-12T02:38:26.417 INFO:tasks.rados.rados.0.smithi107.stdout:6041: delete oid 16 current snap is 0 2020-06-12T02:38:26.419 INFO:tasks.rados.rados.0.smithi107.stdout:6041: done (2 left) 2020-06-12T02:38:26.419 INFO:tasks.rados.rados.0.smithi107.stdout:6042: delete oid 16 current snap is 0 2020-06-12T02:38:26.420 INFO:tasks.rados.rados.0.smithi107.stdout:6038: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:26.421 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd set nodeep-scrub 2020-06-12T02:38:26.422 INFO:tasks.rados.rados.0.smithi107.stdout:6042: done (2 left) 2020-06-12T02:38:26.423 INFO:tasks.rados.rados.0.smithi107.stdout:6043: delete oid 14 current snap is 0 2020-06-12T02:38:26.423 INFO:tasks.rados.rados.0.smithi107.stdout:6038: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:26.423 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 463 (ObjNum 6037 snap 0 seq_num 6037) dirty exists 2020-06-12T02:38:26.423 INFO:tasks.rados.rados.0.smithi107.stdout:6038: left oid 20 (ObjNum 6037 snap 0 seq_num 6037) 2020-06-12T02:38:26.424 INFO:tasks.rados.rados.0.smithi107.stdout:6038: done (2 left) 2020-06-12T02:38:26.424 INFO:tasks.rados.rados.0.smithi107.stdout:6043: done (1 left) 2020-06-12T02:38:26.424 INFO:tasks.rados.rados.0.smithi107.stdout:6044: delete oid 17 current snap is 0 2020-06-12T02:38:26.426 INFO:tasks.rados.rados.0.smithi107.stdout:6044: done (1 left) 2020-06-12T02:38:26.426 INFO:tasks.rados.rados.0.smithi107.stdout:6045: delete oid 11 current snap is 0 2020-06-12T02:38:26.428 INFO:tasks.rados.rados.0.smithi107.stdout:6045: done (1 left) 2020-06-12T02:38:26.428 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:38:26.429 INFO:tasks.rados.rados.0.smithi107.stdout:6046: seq_num 6045 ranges {0=450560} 2020-06-12T02:38:26.431 INFO:tasks.rados.rados.0.smithi107.stdout:6046: writing smithi10729407-1 from 0 to 450560 tid 1 2020-06-12T02:38:26.431 INFO:tasks.rados.rados.0.smithi107.stdout:6047: delete oid 12 current snap is 0 2020-06-12T02:38:26.433 INFO:tasks.rados.rados.0.smithi107.stdout:6047: done (2 left) 2020-06-12T02:38:26.433 INFO:tasks.rados.rados.0.smithi107.stdout:6048: delete oid 15 current snap is 0 2020-06-12T02:38:26.436 INFO:tasks.rados.rados.0.smithi107.stdout:6046: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:26.436 INFO:tasks.rados.rados.0.smithi107.stdout:6048: done (2 left) 2020-06-12T02:38:26.437 INFO:tasks.rados.rados.0.smithi107.stdout:6049: delete oid 4 current snap is 0 2020-06-12T02:38:26.438 INFO:tasks.rados.rados.0.smithi107.stdout:6046: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:26.439 INFO:tasks.rados.rados.0.smithi107.stdout:6049: done (2 left) 2020-06-12T02:38:26.439 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:26.439 INFO:tasks.rados.rados.0.smithi107.stdout:6050: seq_num 6049 ranges {0=516096} 2020-06-12T02:38:26.442 INFO:tasks.rados.rados.0.smithi107.stdout:6050: writing smithi10729407-18 from 0 to 516096 tid 1 2020-06-12T02:38:26.442 INFO:tasks.rados.rados.0.smithi107.stdout:6051: delete oid 5 current snap is 0 2020-06-12T02:38:26.442 INFO:tasks.rados.rados.0.smithi107.stdout:6046: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:26.443 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1125 (ObjNum 6045 snap 0 seq_num 6045) dirty exists 2020-06-12T02:38:26.443 INFO:tasks.rados.rados.0.smithi107.stdout:6046: left oid 1 (ObjNum 6045 snap 0 seq_num 6045) 2020-06-12T02:38:26.444 INFO:tasks.rados.rados.0.smithi107.stdout:6046: done (3 left) 2020-06-12T02:38:26.444 INFO:tasks.rados.rados.0.smithi107.stdout:6051: done (2 left) 2020-06-12T02:38:26.444 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:26.444 INFO:tasks.rados.rados.0.smithi107.stdout:6052: seq_num 6051 ranges {0=548864} 2020-06-12T02:38:26.447 INFO:tasks.rados.rados.0.smithi107.stdout:6052: writing smithi10729407-5 from 0 to 548864 tid 1 2020-06-12T02:38:26.448 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:26.448 INFO:tasks.rados.rados.0.smithi107.stdout:6053: seq_num 6052 ranges {0=565248} 2020-06-12T02:38:26.452 INFO:tasks.rados.rados.0.smithi107.stdout:6053: writing smithi10729407-19 from 0 to 565248 tid 1 2020-06-12T02:38:26.452 INFO:tasks.rados.rados.0.smithi107.stdout:6054: delete oid 11 current snap is 0 2020-06-12T02:38:26.452 INFO:tasks.rados.rados.0.smithi107.stdout:6050: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:26.455 INFO:tasks.rados.rados.0.smithi107.stdout:6050: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:26.455 INFO:tasks.rados.rados.0.smithi107.stdout:6052: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:26.456 INFO:tasks.rados.rados.0.smithi107.stdout:6052: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:26.457 INFO:tasks.rados.rados.0.smithi107.stdout:6054: done (4 left) 2020-06-12T02:38:26.457 INFO:tasks.rados.rados.0.smithi107.stdout:6055: delete oid 8 current snap is 0 2020-06-12T02:38:26.457 INFO:tasks.rados.rados.0.smithi107.stdout:6052: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:26.465 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 607 (ObjNum 6051 snap 0 seq_num 6051) dirty exists 2020-06-12T02:38:26.465 INFO:tasks.rados.rados.0.smithi107.stdout:6052: left oid 5 (ObjNum 6051 snap 0 seq_num 6051) 2020-06-12T02:38:26.465 INFO:tasks.rados.rados.0.smithi107.stdout:6050: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:26.466 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1097 (ObjNum 6049 snap 0 seq_num 6049) dirty exists 2020-06-12T02:38:26.466 INFO:tasks.rados.rados.0.smithi107.stdout:6050: left oid 18 (ObjNum 6049 snap 0 seq_num 6049) 2020-06-12T02:38:26.468 INFO:tasks.rados.rados.0.smithi107.stdout:6053: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:26.468 INFO:tasks.rados.rados.0.smithi107.stdout:6050: done (4 left) 2020-06-12T02:38:26.468 INFO:tasks.rados.rados.0.smithi107.stdout:6052: done (3 left) 2020-06-12T02:38:26.468 INFO:tasks.rados.rados.0.smithi107.stdout:6055: done (2 left) 2020-06-12T02:38:26.469 INFO:tasks.rados.rados.0.smithi107.stdout:6056: delete oid 11 current snap is 0 2020-06-12T02:38:26.469 INFO:tasks.rados.rados.0.smithi107.stdout:6053: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:26.469 INFO:tasks.rados.rados.0.smithi107.stdout:6056: done (2 left) 2020-06-12T02:38:26.469 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:26.469 INFO:tasks.rados.rados.0.smithi107.stdout:6057: seq_num 6056 ranges {516096=630784} 2020-06-12T02:38:26.470 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:26.470 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2020-06-12T02:38:26.470 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2020-06-12T02:38:26.470 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2020-06-12T02:38:26.470 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.471 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8802 3.bs0 3:d0ebae7d:::smithi10729407-3:head [append 0~573440] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:26.471 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.691981+0000", 2020-06-12T02:38:26.471 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 14.767707217, 2020-06-12T02:38:26.471 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.024741973, 2020-06-12T02:38:26.471 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:26.472 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:26.472 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:26.472 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:26.472 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:26.472 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8802 2020-06-12T02:38:26.473 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.473 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:26.473 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.473 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:26.473 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.691981+0000", 2020-06-12T02:38:26.474 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:26.474 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.474 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.474 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:26.474 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.691981+0000", 2020-06-12T02:38:26.474 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000369312 2020-06-12T02:38:26.475 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.475 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.475 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:26.475 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.692350+0000", 2020-06-12T02:38:26.475 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9639999999999998e-06 2020-06-12T02:38:26.476 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.476 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.476 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:26.476 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.692360+0000", 2020-06-12T02:38:26.476 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9335999999999999e-05 2020-06-12T02:38:26.476 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.477 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.477 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:26.477 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.692379+0000", 2020-06-12T02:38:26.477 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8229999999999995e-06 2020-06-12T02:38:26.477 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.478 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.478 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:26.478 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.692387+0000", 2020-06-12T02:38:26.478 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0760999999999997e-05 2020-06-12T02:38:26.478 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.478 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.479 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.479 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.692438+0000", 2020-06-12T02:38:26.479 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000113064 2020-06-12T02:38:26.479 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.479 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.480 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:26.480 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.692551+0000", 2020-06-12T02:38:26.480 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00080948800000000003 2020-06-12T02:38:26.480 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.480 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.481 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:26.481 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.693361+0000", 2020-06-12T02:38:26.481 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.023159432000000001 2020-06-12T02:38:26.481 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.481 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.481 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:26.482 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.716520+0000", 2020-06-12T02:38:26.482 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000115021 2020-06-12T02:38:26.482 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.482 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.482 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:26.483 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.716635+0000", 2020-06-12T02:38:26.483 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7771999999999999e-05 2020-06-12T02:38:26.484 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.484 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.484 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:26.484 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.716723+0000", 2020-06-12T02:38:26.485 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.024741973 2020-06-12T02:38:26.485 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:26.485 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:26.485 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:26.486 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.486 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.486 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8803 3.bs0 3:d0ebae7d:::smithi10729407-3:head [append 0~565248] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:26.486 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.695906+0000", 2020-06-12T02:38:26.486 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 14.763782025999999, 2020-06-12T02:38:26.487 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020991011, 2020-06-12T02:38:26.487 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:26.487 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:26.487 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:26.487 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:26.488 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:26.488 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8803 2020-06-12T02:38:26.489 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.489 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:26.489 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.489 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:26.490 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.695906+0000", 2020-06-12T02:38:26.490 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:26.490 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.490 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.490 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:26.491 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.695906+0000", 2020-06-12T02:38:26.491 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036262499999999999 2020-06-12T02:38:26.491 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.491 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.491 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:26.492 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.696269+0000", 2020-06-12T02:38:26.492 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0131e-05 2020-06-12T02:38:26.492 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.492 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.492 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:26.492 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.696279+0000", 2020-06-12T02:38:26.493 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6390000000000001e-05 2020-06-12T02:38:26.493 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.493 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.493 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:26.493 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.696295+0000", 2020-06-12T02:38:26.494 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0600000000000002e-06 2020-06-12T02:38:26.494 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.494 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.494 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:26.494 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.696302+0000", 2020-06-12T02:38:26.495 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0016389709999999999 2020-06-12T02:38:26.495 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.495 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.495 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.496 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.697941+0000", 2020-06-12T02:38:26.496 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011928400000000001 2020-06-12T02:38:26.496 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.496 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.496 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:26.496 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.698060+0000", 2020-06-12T02:38:26.497 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00061229800000000005 2020-06-12T02:38:26.497 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.497 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.497 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:26.498 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.698673+0000", 2020-06-12T02:38:26.498 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018056592 2020-06-12T02:38:26.498 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.499 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.499 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:26.499 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.716729+0000", 2020-06-12T02:38:26.500 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6626999999999996e-05 2020-06-12T02:38:26.500 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.500 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.500 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:26.500 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.716816+0000", 2020-06-12T02:38:26.501 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1032999999999995e-05 2020-06-12T02:38:26.501 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.501 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.501 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:26.501 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.716897+0000", 2020-06-12T02:38:26.502 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020991011 2020-06-12T02:38:26.502 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:26.502 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:26.502 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:26.502 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.503 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.503 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8804 3.bs0 3:d0ebae7d:::smithi10729407-3:head [append 0~589824] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:26.503 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.700177+0000", 2020-06-12T02:38:26.503 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 14.759510929999999, 2020-06-12T02:38:26.503 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020057443000000001, 2020-06-12T02:38:26.503 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:26.504 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:26.504 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:26.504 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:26.504 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:26.504 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8804 2020-06-12T02:38:26.505 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.505 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:26.506 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.506 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:26.506 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700177+0000", 2020-06-12T02:38:26.506 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:26.507 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.507 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.507 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:26.507 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700177+0000", 2020-06-12T02:38:26.507 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038999500000000002 2020-06-12T02:38:26.508 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.508 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.508 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:26.508 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700567+0000", 2020-06-12T02:38:26.508 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.064e-05 2020-06-12T02:38:26.509 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.509 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.509 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:26.509 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700578+0000", 2020-06-12T02:38:26.509 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8008000000000001e-05 2020-06-12T02:38:26.509 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.510 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.510 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:26.510 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700596+0000", 2020-06-12T02:38:26.510 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7950000000000008e-06 2020-06-12T02:38:26.510 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.511 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.511 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:26.511 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700604+0000", 2020-06-12T02:38:26.511 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0086454970000000003 2020-06-12T02:38:26.511 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.512 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.512 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.512 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.709249+0000", 2020-06-12T02:38:26.512 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014904799999999999 2020-06-12T02:38:26.512 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.513 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.513 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:26.513 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.709398+0000", 2020-06-12T02:38:26.513 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00087258400000000003 2020-06-12T02:38:26.513 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.514 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.514 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:26.514 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.710271+0000", 2020-06-12T02:38:26.514 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085096960000000006 2020-06-12T02:38:26.514 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.514 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.515 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:26.515 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.718780+0000", 2020-06-12T02:38:26.515 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013730579999999999 2020-06-12T02:38:26.515 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.515 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.516 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:26.516 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.720153+0000", 2020-06-12T02:38:26.516 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1122e-05 2020-06-12T02:38:26.516 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.516 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.517 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:26.517 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.720235+0000", 2020-06-12T02:38:26.517 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020057443000000001 2020-06-12T02:38:26.517 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:26.517 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:26.518 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:26.518 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.518 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.518 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8805 3.bs0 3:d0ebae7d:::smithi10729407-3:head [append 0~49152] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:26.518 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.700675+0000", 2020-06-12T02:38:26.519 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 14.759013358000001, 2020-06-12T02:38:26.519 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.023534237999999999, 2020-06-12T02:38:26.519 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:26.519 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:26.519 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:26.519 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:26.520 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:26.520 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8805 2020-06-12T02:38:26.520 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.520 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:26.520 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.521 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:26.521 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700675+0000", 2020-06-12T02:38:26.521 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:26.521 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.521 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.522 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:26.522 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700675+0000", 2020-06-12T02:38:26.522 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119883 2020-06-12T02:38:26.522 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.522 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.522 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:26.523 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700795+0000", 2020-06-12T02:38:26.523 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1179999999999992e-06 2020-06-12T02:38:26.523 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.523 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.523 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:26.524 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700803+0000", 2020-06-12T02:38:26.524 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3040999999999999e-05 2020-06-12T02:38:26.524 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.524 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.524 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:26.525 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700816+0000", 2020-06-12T02:38:26.525 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9499999999999998e-06 2020-06-12T02:38:26.525 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.525 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.525 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:26.526 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700822+0000", 2020-06-12T02:38:26.526 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014038444000000001 2020-06-12T02:38:26.526 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.526 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.526 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.526 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.714860+0000", 2020-06-12T02:38:26.527 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000113682 2020-06-12T02:38:26.527 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.527 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.527 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:26.527 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.714974+0000", 2020-06-12T02:38:26.528 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00058496799999999995 2020-06-12T02:38:26.528 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.528 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.528 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:26.529 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.715559+0000", 2020-06-12T02:38:26.529 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0033177940000000002 2020-06-12T02:38:26.529 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.529 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.529 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:26.529 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.718877+0000", 2020-06-12T02:38:26.530 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.005256625 2020-06-12T02:38:26.530 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.530 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.530 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:26.530 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.724133+0000", 2020-06-12T02:38:26.531 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5733000000000001e-05 2020-06-12T02:38:26.531 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.531 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.531 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:26.531 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.724209+0000", 2020-06-12T02:38:26.531 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.023534237999999999 2020-06-12T02:38:26.532 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:26.532 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:26.532 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:26.532 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.532 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.533 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8806 3.bs0 3:d0ebae7d:::smithi10729407-3:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:26.533 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.700882+0000", 2020-06-12T02:38:26.533 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 14.758806492, 2020-06-12T02:38:26.533 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025273714999999999, 2020-06-12T02:38:26.533 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:26.534 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:26.534 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:26.534 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:26.534 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:26.534 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8806 2020-06-12T02:38:26.535 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.535 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:26.535 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.535 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:26.535 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700882+0000", 2020-06-12T02:38:26.536 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:26.536 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.536 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.536 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:26.536 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700882+0000", 2020-06-12T02:38:26.537 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7329000000000004e-05 2020-06-12T02:38:26.537 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.537 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.537 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:26.537 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700969+0000", 2020-06-12T02:38:26.537 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0620000000000001e-06 2020-06-12T02:38:26.538 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.538 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.538 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:26.538 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700977+0000", 2020-06-12T02:38:26.538 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5291e-05 2020-06-12T02:38:26.539 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.539 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.539 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:26.539 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700992+0000", 2020-06-12T02:38:26.539 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.297e-06 2020-06-12T02:38:26.539 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.540 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.540 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:26.540 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700999+0000", 2020-06-12T02:38:26.540 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015970872000000001 2020-06-12T02:38:26.540 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.541 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.541 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.541 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.716969+0000", 2020-06-12T02:38:26.541 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105802 2020-06-12T02:38:26.541 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.542 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.542 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:26.542 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.717075+0000", 2020-06-12T02:38:26.542 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00042746400000000001 2020-06-12T02:38:26.542 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.542 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.543 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:26.543 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.717503+0000", 2020-06-12T02:38:26.543 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013889869999999999 2020-06-12T02:38:26.543 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.543 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.544 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:26.544 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.718892+0000", 2020-06-12T02:38:26.544 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071672740000000004 2020-06-12T02:38:26.544 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.544 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.545 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:26.545 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.726059+0000", 2020-06-12T02:38:26.545 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6336999999999996e-05 2020-06-12T02:38:26.545 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.546 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.546 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:26.546 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.726155+0000", 2020-06-12T02:38:26.546 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025273714999999999 2020-06-12T02:38:26.546 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:26.547 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:26.547 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:26.547 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.547 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.547 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8807 3.bs0 3:d0ebae7d:::smithi10729407-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e49)", 2020-06-12T02:38:26.547 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.701041+0000", 2020-06-12T02:38:26.548 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 14.758646903000001, 2020-06-12T02:38:26.548 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030589063, 2020-06-12T02:38:26.548 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:26.548 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:26.548 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:26.549 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:26.549 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:26.549 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8807 2020-06-12T02:38:26.549 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.549 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:26.550 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.550 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:26.550 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.701041+0000", 2020-06-12T02:38:26.550 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:26.550 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.550 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.551 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:26.551 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.701041+0000", 2020-06-12T02:38:26.551 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6292999999999999e-05 2020-06-12T02:38:26.551 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.551 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.552 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:26.552 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.701087+0000", 2020-06-12T02:38:26.552 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9840000000000004e-06 2020-06-12T02:38:26.552 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.552 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.553 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:26.553 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.701094+0000", 2020-06-12T02:38:26.553 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3713000000000001e-05 2020-06-12T02:38:26.553 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.554 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.554 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:26.554 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.701108+0000", 2020-06-12T02:38:26.554 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6200000000000004e-06 2020-06-12T02:38:26.554 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.555 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.555 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:26.555 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.701114+0000", 2020-06-12T02:38:26.555 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016879646000000002 2020-06-12T02:38:26.556 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.556 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.556 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.556 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.717993+0000", 2020-06-12T02:38:26.556 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4378999999999999e-05 2020-06-12T02:38:26.557 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.557 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.557 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:26.557 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.718088+0000", 2020-06-12T02:38:26.557 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0092861850000000006 2020-06-12T02:38:26.557 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.558 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.558 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.558 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.727374+0000", 2020-06-12T02:38:26.558 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1155000000000004e-05 2020-06-12T02:38:26.558 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.559 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.559 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:26.559 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.727465+0000", 2020-06-12T02:38:26.559 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004165088 2020-06-12T02:38:26.560 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.560 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.560 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:26.560 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.731630+0000", 2020-06-12T02:38:26.560 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030589063 2020-06-12T02:38:26.561 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:26.561 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:26.561 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:26.561 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.561 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.562 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8847 3.8s0 3:14244450:::smithi10729407-2:head [append 0~638976] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:26.562 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.836821+0000", 2020-06-12T02:38:26.562 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 14.622866837, 2020-06-12T02:38:26.563 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020322752999999999, 2020-06-12T02:38:26.563 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:26.563 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:26.563 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:26.563 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:26.563 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:26.564 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8847 2020-06-12T02:38:26.564 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.564 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:26.566 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.566 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:26.566 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.836821+0000", 2020-06-12T02:38:26.567 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:26.567 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.568 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.568 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:26.568 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.836821+0000", 2020-06-12T02:38:26.568 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00021910399999999999 2020-06-12T02:38:26.569 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.569 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.569 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:26.569 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.837040+0000", 2020-06-12T02:38:26.570 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5099999999999998e-06 2020-06-12T02:38:26.570 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.570 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.571 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:26.571 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.837049+0000", 2020-06-12T02:38:26.571 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4453e-05 2020-06-12T02:38:26.571 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.571 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.572 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:26.572 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.837063+0000", 2020-06-12T02:38:26.572 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.251e-06 2020-06-12T02:38:26.572 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.572 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.573 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:26.573 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.837070+0000", 2020-06-12T02:38:26.573 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0026855310000000001 2020-06-12T02:38:26.573 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.573 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.573 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.574 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.839755+0000", 2020-06-12T02:38:26.574 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9023000000000006e-05 2020-06-12T02:38:26.574 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.574 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.574 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:26.575 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.839834+0000", 2020-06-12T02:38:26.575 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0059031170000000003 2020-06-12T02:38:26.575 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.575 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.575 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.576 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.845737+0000", 2020-06-12T02:38:26.576 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4455000000000003e-05 2020-06-12T02:38:26.576 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.576 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.576 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:26.577 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.845832+0000", 2020-06-12T02:38:26.577 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00074392800000000004 2020-06-12T02:38:26.577 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.577 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.577 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:26.577 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.846576+0000", 2020-06-12T02:38:26.578 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0070446479999999997 2020-06-12T02:38:26.578 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.578 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.578 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:26.579 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.853620+0000", 2020-06-12T02:38:26.579 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.003447126 2020-06-12T02:38:26.579 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.579 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.580 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:26.580 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.857067+0000", 2020-06-12T02:38:26.580 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6606999999999993e-05 2020-06-12T02:38:26.580 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.580 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.586 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:26.586 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.857144+0000", 2020-06-12T02:38:26.586 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020322752999999999 2020-06-12T02:38:26.587 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:26.587 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:26.587 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:26.587 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.587 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.587 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8850 3.8s0 3:14244450:::smithi10729407-2:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:26.588 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.843425+0000", 2020-06-12T02:38:26.588 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 14.616262548, 2020-06-12T02:38:26.588 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019923956999999999, 2020-06-12T02:38:26.588 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:26.588 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:26.589 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:26.589 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:26.589 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:26.589 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8850 2020-06-12T02:38:26.589 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.590 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:26.590 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.590 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:26.590 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.843425+0000", 2020-06-12T02:38:26.590 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:26.590 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.591 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.591 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:26.591 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.843425+0000", 2020-06-12T02:38:26.591 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8193000000000002e-05 2020-06-12T02:38:26.591 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.592 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.592 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:26.592 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.843504+0000", 2020-06-12T02:38:26.592 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3110000000000002e-06 2020-06-12T02:38:26.592 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.592 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.593 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:26.593 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.843510+0000", 2020-06-12T02:38:26.593 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8329999999999994e-06 2020-06-12T02:38:26.593 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.593 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.594 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:26.594 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.843518+0000", 2020-06-12T02:38:26.594 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.1860000000000001e-06 2020-06-12T02:38:26.594 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.594 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.594 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:26.595 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.843521+0000", 2020-06-12T02:38:26.595 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8204999999999998e-05 2020-06-12T02:38:26.595 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.595 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.595 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.596 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.843549+0000", 2020-06-12T02:38:26.596 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4450000000000006e-05 2020-06-12T02:38:26.596 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.596 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.596 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:26.596 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.843634+0000", 2020-06-12T02:38:26.597 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013555185000000001 2020-06-12T02:38:26.597 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.597 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.597 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.597 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.857189+0000", 2020-06-12T02:38:26.598 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7164999999999996e-05 2020-06-12T02:38:26.598 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.598 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.598 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:26.598 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.857286+0000", 2020-06-12T02:38:26.599 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000406809 2020-06-12T02:38:26.599 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.599 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.599 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:26.599 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.857693+0000", 2020-06-12T02:38:26.599 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0033959120000000001 2020-06-12T02:38:26.600 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.600 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.600 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:26.600 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.861089+0000", 2020-06-12T02:38:26.600 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0021525369999999999 2020-06-12T02:38:26.601 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.601 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.601 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:26.601 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.863241+0000", 2020-06-12T02:38:26.601 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000108171 2020-06-12T02:38:26.601 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.602 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.602 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:26.602 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.863349+0000", 2020-06-12T02:38:26.602 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019923956999999999 2020-06-12T02:38:26.602 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:26.603 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:26.603 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:26.603 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.603 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.603 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8851 3.8s0 3:14244450:::smithi10729407-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e49)", 2020-06-12T02:38:26.603 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.843565+0000", 2020-06-12T02:38:26.604 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 14.616123464999999, 2020-06-12T02:38:26.604 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.02215278, 2020-06-12T02:38:26.604 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:26.604 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:26.604 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:26.605 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:26.605 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:26.605 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8851 2020-06-12T02:38:26.605 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.605 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:26.605 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.606 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:26.606 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.843565+0000", 2020-06-12T02:38:26.606 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:26.606 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.606 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.607 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:26.607 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.843565+0000", 2020-06-12T02:38:26.607 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4527999999999998e-05 2020-06-12T02:38:26.607 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.607 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.607 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:26.608 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.843609+0000", 2020-06-12T02:38:26.608 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0909999999999998e-06 2020-06-12T02:38:26.608 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.608 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.608 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:26.609 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.843615+0000", 2020-06-12T02:38:26.609 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2254e-05 2020-06-12T02:38:26.609 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.609 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.609 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:26.609 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.843627+0000", 2020-06-12T02:38:26.610 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5539999999999997e-06 2020-06-12T02:38:26.610 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.610 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.610 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:26.610 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.843633+0000", 2020-06-12T02:38:26.611 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2058999999999999e-05 2020-06-12T02:38:26.611 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.611 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.612 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.612 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.843665+0000", 2020-06-12T02:38:26.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9106e-05 2020-06-12T02:38:26.612 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.613 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:26.613 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.843714+0000", 2020-06-12T02:38:26.613 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014386637000000001 2020-06-12T02:38:26.613 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.613 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.614 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.614 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.858101+0000", 2020-06-12T02:38:26.614 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000102223 2020-06-12T02:38:26.614 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.614 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.614 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:26.615 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.858203+0000", 2020-06-12T02:38:26.615 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0058912900000000004 2020-06-12T02:38:26.615 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.615 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.615 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.616 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864094+0000", 2020-06-12T02:38:26.616 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4738e-05 2020-06-12T02:38:26.616 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.616 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.616 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:26.616 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864189+0000", 2020-06-12T02:38:26.617 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0015283 2020-06-12T02:38:26.617 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.617 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.617 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:26.618 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.865717+0000", 2020-06-12T02:38:26.618 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.02215278 2020-06-12T02:38:26.618 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:26.618 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:26.618 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:26.619 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.619 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.619 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:9628 3.bs0 3:d09543d2:::smithi10729407-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e54)", 2020-06-12T02:38:26.619 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:16.887830+0000", 2020-06-12T02:38:26.619 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 9.5718583309999996, 2020-06-12T02:38:26.620 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022465841, 2020-06-12T02:38:26.620 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:26.620 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:26.620 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:26.620 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:26.620 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:26.621 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 9628 2020-06-12T02:38:26.621 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.621 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:26.621 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.621 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:26.622 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.887830+0000", 2020-06-12T02:38:26.622 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:26.622 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.622 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.622 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:26.622 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.887830+0000", 2020-06-12T02:38:26.623 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6709000000000003e-05 2020-06-12T02:38:26.623 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.623 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.623 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:26.623 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.887876+0000", 2020-06-12T02:38:26.624 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8150000000000002e-06 2020-06-12T02:38:26.624 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.624 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.624 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:26.625 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.887882+0000", 2020-06-12T02:38:26.626 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.359e-05 2020-06-12T02:38:26.626 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.626 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.626 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:26.626 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.887896+0000", 2020-06-12T02:38:26.627 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0029999999999996e-06 2020-06-12T02:38:26.627 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.627 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.627 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:26.627 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.887900+0000", 2020-06-12T02:38:26.628 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012522259999999999 2020-06-12T02:38:26.628 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.628 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.628 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.628 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.889152+0000", 2020-06-12T02:38:26.628 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3249000000000002e-05 2020-06-12T02:38:26.629 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.629 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.629 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:26.629 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.889235+0000", 2020-06-12T02:38:26.629 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012485619 2020-06-12T02:38:26.630 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.630 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.630 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.630 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.901721+0000", 2020-06-12T02:38:26.630 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2717999999999999e-05 2020-06-12T02:38:26.631 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.631 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.631 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:26.632 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.901774+0000", 2020-06-12T02:38:26.633 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004869305 2020-06-12T02:38:26.633 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.633 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.634 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.634 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.906643+0000", 2020-06-12T02:38:26.634 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1521999999999996e-05 2020-06-12T02:38:26.634 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.635 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.635 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:26.636 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.906734+0000", 2020-06-12T02:38:26.636 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0035610849999999999 2020-06-12T02:38:26.636 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.636 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.636 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:26.637 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.910296+0000", 2020-06-12T02:38:26.637 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022465841 2020-06-12T02:38:26.637 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:26.637 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:26.637 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:26.637 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.638 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.638 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:9675 3.bs0 3:d0ebae7d:::smithi10729407-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e54)", 2020-06-12T02:38:26.638 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:16.985984+0000", 2020-06-12T02:38:26.638 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 9.4737041610000006, 2020-06-12T02:38:26.638 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020123386, 2020-06-12T02:38:26.639 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:26.639 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:26.639 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:26.639 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:26.639 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:26.640 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 9675 2020-06-12T02:38:26.640 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.640 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:26.640 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.640 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:26.640 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.985984+0000", 2020-06-12T02:38:26.641 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:26.641 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.641 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.641 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:26.641 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.985984+0000", 2020-06-12T02:38:26.642 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6591999999999999e-05 2020-06-12T02:38:26.642 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.642 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.642 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:26.642 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.986010+0000", 2020-06-12T02:38:26.642 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3119999999999998e-06 2020-06-12T02:38:26.643 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.643 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.643 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:26.643 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.986014+0000", 2020-06-12T02:38:26.643 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9589999999999995e-06 2020-06-12T02:38:26.644 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.644 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.644 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:26.644 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.986022+0000", 2020-06-12T02:38:26.644 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.576e-06 2020-06-12T02:38:26.645 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.645 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.645 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:26.645 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.986024+0000", 2020-06-12T02:38:26.645 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000100881 2020-06-12T02:38:26.645 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.646 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.646 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.646 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.986125+0000", 2020-06-12T02:38:26.646 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9279999999999999e-05 2020-06-12T02:38:26.646 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.647 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.647 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:26.647 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.986214+0000", 2020-06-12T02:38:26.647 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0095388030000000002 2020-06-12T02:38:26.647 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.647 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.648 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.648 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.995753+0000", 2020-06-12T02:38:26.648 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0044999999999999e-05 2020-06-12T02:38:26.648 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.648 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.649 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:26.649 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:16.995803+0000", 2020-06-12T02:38:26.649 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0092663399999999996 2020-06-12T02:38:26.649 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.649 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.650 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.650 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.005070+0000", 2020-06-12T02:38:26.650 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5955e-05 2020-06-12T02:38:26.650 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.650 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.650 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:26.651 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.005156+0000", 2020-06-12T02:38:26.651 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00095164299999999996 2020-06-12T02:38:26.651 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.651 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.651 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:26.651 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.006107+0000", 2020-06-12T02:38:26.652 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020123386 2020-06-12T02:38:26.652 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:26.652 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:26.652 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:26.652 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.653 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.653 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:9917 3.bs0 3:d09543d2:::smithi10729407-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e54)", 2020-06-12T02:38:26.653 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:17.669696+0000", 2020-06-12T02:38:26.653 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 8.7899916499999993, 2020-06-12T02:38:26.653 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.021166378, 2020-06-12T02:38:26.654 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:26.654 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:26.654 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:26.654 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:26.654 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:26.654 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 9917 2020-06-12T02:38:26.655 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.655 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:26.655 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.655 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:26.655 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.669696+0000", 2020-06-12T02:38:26.656 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:26.656 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.656 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.656 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:26.656 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.669696+0000", 2020-06-12T02:38:26.656 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2474e-05 2020-06-12T02:38:26.657 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.657 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.657 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:26.657 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.669739+0000", 2020-06-12T02:38:26.657 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5769999999999999e-06 2020-06-12T02:38:26.658 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.658 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.658 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:26.658 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.669745+0000", 2020-06-12T02:38:26.658 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2695e-05 2020-06-12T02:38:26.658 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.659 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.659 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:26.659 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.669758+0000", 2020-06-12T02:38:26.659 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7570000000000004e-06 2020-06-12T02:38:26.659 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.660 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.660 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:26.660 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.669763+0000", 2020-06-12T02:38:26.660 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0049988360000000004 2020-06-12T02:38:26.660 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.660 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.661 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.661 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.674762+0000", 2020-06-12T02:38:26.661 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8404000000000002e-05 2020-06-12T02:38:26.661 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.661 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.662 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:26.662 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.674860+0000", 2020-06-12T02:38:26.662 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0056080419999999997 2020-06-12T02:38:26.662 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.662 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.668 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.669 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.680468+0000", 2020-06-12T02:38:26.669 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000108749 2020-06-12T02:38:26.669 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.669 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.669 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:26.670 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.680577+0000", 2020-06-12T02:38:26.670 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0074646280000000001 2020-06-12T02:38:26.670 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.670 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.670 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.671 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.688042+0000", 2020-06-12T02:38:26.671 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6777e-05 2020-06-12T02:38:26.671 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.671 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.671 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:26.671 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.688118+0000", 2020-06-12T02:38:26.672 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002744439 2020-06-12T02:38:26.672 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.672 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.673 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:26.673 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.690863+0000", 2020-06-12T02:38:26.673 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.021166378 2020-06-12T02:38:26.673 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:26.673 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:26.674 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:26.674 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.674 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.674 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:10040 3.5s1 3:a446366d:::smithi10729407-11:head [append 0~131072] snapc 0=[] ondisk+write+known_if_redirected e54)", 2020-06-12T02:38:26.674 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:18.037838+0000", 2020-06-12T02:38:26.675 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 8.4218501969999995, 2020-06-12T02:38:26.675 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020329944999999999, 2020-06-12T02:38:26.675 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:26.675 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:26.676 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:26.676 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:26.676 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:26.676 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 10040 2020-06-12T02:38:26.676 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.677 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:26.677 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.677 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:26.677 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.037838+0000", 2020-06-12T02:38:26.677 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:26.678 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.678 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.678 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:26.678 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.037838+0000", 2020-06-12T02:38:26.678 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1982000000000003e-05 2020-06-12T02:38:26.679 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.680 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.680 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:26.680 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.037910+0000", 2020-06-12T02:38:26.680 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3550000000000001e-06 2020-06-12T02:38:26.681 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.681 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.681 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:26.681 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.037916+0000", 2020-06-12T02:38:26.681 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3030000000000001e-05 2020-06-12T02:38:26.682 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.682 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.682 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:26.682 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.037929+0000", 2020-06-12T02:38:26.682 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1019999999999999e-06 2020-06-12T02:38:26.682 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.683 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.683 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:26.683 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.037936+0000", 2020-06-12T02:38:26.683 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3041999999999999e-05 2020-06-12T02:38:26.683 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.684 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.684 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.684 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.037979+0000", 2020-06-12T02:38:26.684 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101257 2020-06-12T02:38:26.684 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.684 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.685 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:26.685 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.038081+0000", 2020-06-12T02:38:26.685 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012472613 2020-06-12T02:38:26.685 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.685 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.686 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.686 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.050553+0000", 2020-06-12T02:38:26.686 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2694999999999995e-05 2020-06-12T02:38:26.686 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.686 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.687 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:26.687 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.050616+0000", 2020-06-12T02:38:26.687 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036818700000000002 2020-06-12T02:38:26.687 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.687 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.687 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:26.688 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.050984+0000", 2020-06-12T02:38:26.688 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0019458419999999999 2020-06-12T02:38:26.688 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.688 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.688 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:26.689 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.052930+0000", 2020-06-12T02:38:26.689 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0051913590000000004 2020-06-12T02:38:26.689 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.689 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.689 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:26.689 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.058121+0000", 2020-06-12T02:38:26.690 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6480999999999998e-05 2020-06-12T02:38:26.690 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.690 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.690 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:26.690 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.058168+0000", 2020-06-12T02:38:26.691 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020329944999999999 2020-06-12T02:38:26.691 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:26.691 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:26.691 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:26.691 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.691 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.692 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:10041 3.5s1 3:a446366d:::smithi10729407-11:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e54)", 2020-06-12T02:38:26.692 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:18.037986+0000", 2020-06-12T02:38:26.692 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 8.4217024309999999, 2020-06-12T02:38:26.692 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020334035, 2020-06-12T02:38:26.692 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:26.693 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:26.693 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:26.693 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:26.693 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:26.693 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 10041 2020-06-12T02:38:26.693 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.694 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:26.694 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.694 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:26.694 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.037986+0000", 2020-06-12T02:38:26.694 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:26.695 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.695 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.695 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:26.695 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.037986+0000", 2020-06-12T02:38:26.695 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5748999999999997e-05 2020-06-12T02:38:26.695 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.696 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.696 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:26.696 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.038051+0000", 2020-06-12T02:38:26.696 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1809999999999998e-06 2020-06-12T02:38:26.696 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.697 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.697 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:26.697 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.038058+0000", 2020-06-12T02:38:26.697 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1058e-05 2020-06-12T02:38:26.697 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.697 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.698 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:26.698 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.038069+0000", 2020-06-12T02:38:26.698 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8930000000000004e-06 2020-06-12T02:38:26.698 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.698 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.699 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:26.699 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.038073+0000", 2020-06-12T02:38:26.699 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7157999999999998e-05 2020-06-12T02:38:26.699 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.699 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.699 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.700 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.038121+0000", 2020-06-12T02:38:26.700 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6507000000000003e-05 2020-06-12T02:38:26.700 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.700 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.700 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:26.701 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.038217+0000", 2020-06-12T02:38:26.701 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013888977 2020-06-12T02:38:26.701 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.701 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.701 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.701 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.052106+0000", 2020-06-12T02:38:26.702 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4517999999999994e-05 2020-06-12T02:38:26.702 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.702 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.702 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:26.702 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.052181+0000", 2020-06-12T02:38:26.703 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040230000000000002 2020-06-12T02:38:26.703 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.703 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.703 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:26.703 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.052583+0000", 2020-06-12T02:38:26.703 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002737279 2020-06-12T02:38:26.704 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.704 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.704 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:26.704 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.055320+0000", 2020-06-12T02:38:26.704 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00294977 2020-06-12T02:38:26.705 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.705 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.705 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:26.705 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.058270+0000", 2020-06-12T02:38:26.705 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9645000000000003e-05 2020-06-12T02:38:26.705 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.706 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.706 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:26.706 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.058320+0000", 2020-06-12T02:38:26.706 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020334035 2020-06-12T02:38:26.706 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:26.707 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:26.707 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:26.707 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.707 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.707 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:10042 3.5s1 3:a446366d:::smithi10729407-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e54)", 2020-06-12T02:38:26.707 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:18.038116+0000", 2020-06-12T02:38:26.708 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 8.4215724220000006, 2020-06-12T02:38:26.708 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.021560902, 2020-06-12T02:38:26.708 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:26.708 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:26.708 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:26.709 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:26.710 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:26.710 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 10042 2020-06-12T02:38:26.710 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.711 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:26.711 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.711 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:26.711 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.038116+0000", 2020-06-12T02:38:26.711 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:26.712 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.712 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.712 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:26.712 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.038116+0000", 2020-06-12T02:38:26.712 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7672000000000001e-05 2020-06-12T02:38:26.712 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.713 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.713 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:26.713 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.038153+0000", 2020-06-12T02:38:26.713 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4979999999999997e-06 2020-06-12T02:38:26.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.714 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:26.714 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.038159+0000", 2020-06-12T02:38:26.714 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1895999999999999e-05 2020-06-12T02:38:26.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.715 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.715 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:26.715 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.038171+0000", 2020-06-12T02:38:26.715 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6460000000000001e-06 2020-06-12T02:38:26.716 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.716 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.716 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:26.716 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.038175+0000", 2020-06-12T02:38:26.716 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4574999999999997e-05 2020-06-12T02:38:26.716 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.717 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.717 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.717 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.038250+0000", 2020-06-12T02:38:26.717 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000108447 2020-06-12T02:38:26.718 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.718 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.718 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:26.719 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.038358+0000", 2020-06-12T02:38:26.719 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015001960999999999 2020-06-12T02:38:26.719 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.719 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.719 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.720 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.053360+0000", 2020-06-12T02:38:26.720 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7246999999999999e-05 2020-06-12T02:38:26.720 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.720 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.721 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:26.722 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.053458+0000", 2020-06-12T02:38:26.723 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0053373259999999999 2020-06-12T02:38:26.723 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.723 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.724 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.724 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.058795+0000", 2020-06-12T02:38:26.724 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4442000000000002e-05 2020-06-12T02:38:26.724 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.724 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.724 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:26.725 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.058879+0000", 2020-06-12T02:38:26.725 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00079719199999999995 2020-06-12T02:38:26.725 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.725 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.725 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:26.726 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:18.059677+0000", 2020-06-12T02:38:26.726 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.021560902 2020-06-12T02:38:26.726 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:26.726 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:26.726 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:26.727 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.727 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.727 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:11341 3.2s0 3:4afffc6f:::smithi10729407-20:head [append 0~573440] snapc 0=[] ondisk+write+known_if_redirected e56)", 2020-06-12T02:38:26.727 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:21.659876+0000", 2020-06-12T02:38:26.727 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 4.7998124710000001, 2020-06-12T02:38:26.728 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.024232958999999998, 2020-06-12T02:38:26.729 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:26.729 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:26.729 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:26.729 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:26.729 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:26.730 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 11341 2020-06-12T02:38:26.730 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.730 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:26.730 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.730 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:26.731 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.659876+0000", 2020-06-12T02:38:26.731 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:26.731 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.731 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.731 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:26.732 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.659876+0000", 2020-06-12T02:38:26.732 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00021518000000000001 2020-06-12T02:38:26.732 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.732 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.732 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:26.732 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.660091+0000", 2020-06-12T02:38:26.733 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7279999999999995e-06 2020-06-12T02:38:26.733 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.733 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.733 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:26.733 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.660098+0000", 2020-06-12T02:38:26.734 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2702e-05 2020-06-12T02:38:26.734 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.734 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.734 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:26.734 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.660111+0000", 2020-06-12T02:38:26.735 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4500000000000003e-06 2020-06-12T02:38:26.735 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.735 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.735 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:26.735 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.660117+0000", 2020-06-12T02:38:26.735 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016948199999999999 2020-06-12T02:38:26.736 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.736 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.736 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.736 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.660286+0000", 2020-06-12T02:38:26.736 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4243000000000001e-05 2020-06-12T02:38:26.737 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.737 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.737 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:26.737 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.660360+0000", 2020-06-12T02:38:26.737 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0095246789999999994 2020-06-12T02:38:26.738 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.738 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.738 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.738 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.669885+0000", 2020-06-12T02:38:26.738 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4608000000000003e-05 2020-06-12T02:38:26.738 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.739 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.739 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:26.739 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.669940+0000", 2020-06-12T02:38:26.739 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050928400000000004 2020-06-12T02:38:26.739 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.740 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.740 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:26.740 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.670449+0000", 2020-06-12T02:38:26.740 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0077268119999999996 2020-06-12T02:38:26.740 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.740 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.741 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:26.741 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.678176+0000", 2020-06-12T02:38:26.741 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0058864490000000002 2020-06-12T02:38:26.741 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.741 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.742 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:26.742 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.684062+0000", 2020-06-12T02:38:26.742 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6341999999999998e-05 2020-06-12T02:38:26.742 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.742 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.743 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:26.743 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.684109+0000", 2020-06-12T02:38:26.743 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.024232958999999998 2020-06-12T02:38:26.743 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:26.743 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:26.743 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:26.744 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.744 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.744 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:11343 3.2s0 3:4afffc6f:::smithi10729407-20:head [append 0~262144] snapc 0=[] ondisk+write+known_if_redirected e56)", 2020-06-12T02:38:26.744 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:21.665815+0000", 2020-06-12T02:38:26.744 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 4.7938726129999996, 2020-06-12T02:38:26.745 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.021621089, 2020-06-12T02:38:26.745 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:26.745 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:26.745 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:26.745 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:26.746 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:26.746 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 11343 2020-06-12T02:38:26.746 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.746 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:26.746 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.746 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:26.747 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.665815+0000", 2020-06-12T02:38:26.747 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:26.747 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.747 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.747 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:26.748 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.665815+0000", 2020-06-12T02:38:26.748 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6261999999999994e-05 2020-06-12T02:38:26.748 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.748 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.748 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:26.748 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.665912+0000", 2020-06-12T02:38:26.749 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1429999999999997e-06 2020-06-12T02:38:26.749 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.749 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.749 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:26.749 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.665917+0000", 2020-06-12T02:38:26.750 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1296e-05 2020-06-12T02:38:26.750 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.750 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.750 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:26.750 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.665928+0000", 2020-06-12T02:38:26.750 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6520000000000003e-06 2020-06-12T02:38:26.751 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.751 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.751 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:26.751 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.665934+0000", 2020-06-12T02:38:26.751 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6580999999999998e-05 2020-06-12T02:38:26.752 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.752 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.752 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.752 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.665960+0000", 2020-06-12T02:38:26.752 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1479e-05 2020-06-12T02:38:26.752 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.753 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.753 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:26.753 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.666012+0000", 2020-06-12T02:38:26.753 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0098223230000000009 2020-06-12T02:38:26.753 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.754 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.754 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.754 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.675834+0000", 2020-06-12T02:38:26.754 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1538000000000006e-05 2020-06-12T02:38:26.754 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.754 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.755 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:26.755 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.675896+0000", 2020-06-12T02:38:26.755 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037683500000000001 2020-06-12T02:38:26.755 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.755 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.756 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:26.756 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.676273+0000", 2020-06-12T02:38:26.756 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0019061219999999999 2020-06-12T02:38:26.756 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.756 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.757 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:26.757 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.678179+0000", 2020-06-12T02:38:26.757 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0092161489999999999 2020-06-12T02:38:26.757 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.757 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.757 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:26.758 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.687395+0000", 2020-06-12T02:38:26.758 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1709000000000003e-05 2020-06-12T02:38:26.758 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.758 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.758 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:26.759 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.687437+0000", 2020-06-12T02:38:26.759 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.021621089 2020-06-12T02:38:26.759 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:26.759 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:26.759 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:26.760 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.760 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.760 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:11344 3.2s0 3:4afffc6f:::smithi10729407-20:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e56)", 2020-06-12T02:38:26.760 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:21.665965+0000", 2020-06-12T02:38:26.760 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 4.7937226239999999, 2020-06-12T02:38:26.760 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.023987715, 2020-06-12T02:38:26.761 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:26.761 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:26.761 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:26.761 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:26.761 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:26.762 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 11344 2020-06-12T02:38:26.762 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.762 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:26.762 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.762 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:26.763 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.665965+0000", 2020-06-12T02:38:26.763 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:26.763 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.763 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.763 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:26.763 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.665965+0000", 2020-06-12T02:38:26.764 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8083000000000001e-05 2020-06-12T02:38:26.764 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.764 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.764 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:26.765 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.666004+0000", 2020-06-12T02:38:26.765 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.366e-06 2020-06-12T02:38:26.765 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.765 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.766 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:26.766 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.666007+0000", 2020-06-12T02:38:26.766 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3110000000000002e-06 2020-06-12T02:38:26.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.766 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.766 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:26.767 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.666013+0000", 2020-06-12T02:38:26.767 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.4540000000000001e-06 2020-06-12T02:38:26.767 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.767 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.767 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:26.768 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.666016+0000", 2020-06-12T02:38:26.768 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7648999999999998e-05 2020-06-12T02:38:26.768 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.768 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.770 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.770 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.666053+0000", 2020-06-12T02:38:26.770 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3727000000000006e-05 2020-06-12T02:38:26.770 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.771 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.771 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:26.771 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.666147+0000", 2020-06-12T02:38:26.771 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011353321 2020-06-12T02:38:26.771 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.772 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.772 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.772 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.677500+0000", 2020-06-12T02:38:26.772 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8938999999999996e-05 2020-06-12T02:38:26.772 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.773 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.773 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:26.773 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.677599+0000", 2020-06-12T02:38:26.773 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030145500000000001 2020-06-12T02:38:26.773 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.774 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.774 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:26.774 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.677901+0000", 2020-06-12T02:38:26.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044933899999999999 2020-06-12T02:38:26.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.776 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:26.776 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.678350+0000", 2020-06-12T02:38:26.776 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011549432 2020-06-12T02:38:26.777 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.777 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.777 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:26.777 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.689899+0000", 2020-06-12T02:38:26.778 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3638999999999999e-05 2020-06-12T02:38:26.778 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.778 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.779 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:26.779 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.689953+0000", 2020-06-12T02:38:26.779 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.023987715 2020-06-12T02:38:26.779 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:26.779 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:26.780 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:26.780 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.780 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.780 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:11345 3.2s0 3:4afffc6f:::smithi10729407-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e56)", 2020-06-12T02:38:26.780 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:21.666049+0000", 2020-06-12T02:38:26.781 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 4.7936393659999998, 2020-06-12T02:38:26.781 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025091826000000001, 2020-06-12T02:38:26.781 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:26.781 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:26.781 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:26.782 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:26.782 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:26.782 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 11345 2020-06-12T02:38:26.783 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.783 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:26.783 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.784 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:26.784 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.666049+0000", 2020-06-12T02:38:26.784 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:26.785 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.785 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.785 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:26.786 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.666049+0000", 2020-06-12T02:38:26.786 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.1736000000000002e-05 2020-06-12T02:38:26.786 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.787 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.787 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:26.787 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.666080+0000", 2020-06-12T02:38:26.788 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8160000000000004e-06 2020-06-12T02:38:26.788 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.788 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.789 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:26.789 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.666084+0000", 2020-06-12T02:38:26.789 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3100000000000003e-06 2020-06-12T02:38:26.790 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.790 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.790 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:26.791 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.666092+0000", 2020-06-12T02:38:26.791 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9490000000000001e-06 2020-06-12T02:38:26.791 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.792 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.792 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:26.792 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.666094+0000", 2020-06-12T02:38:26.793 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4011999999999995e-05 2020-06-12T02:38:26.793 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.793 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.794 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.794 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.666178+0000", 2020-06-12T02:38:26.794 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9424999999999993e-05 2020-06-12T02:38:26.795 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.795 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.795 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:26.796 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.666248+0000", 2020-06-12T02:38:26.796 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011857548000000001 2020-06-12T02:38:26.796 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.797 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.797 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.797 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.678105+0000", 2020-06-12T02:38:26.797 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8922999999999998e-05 2020-06-12T02:38:26.798 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.798 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.799 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:26.799 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.678164+0000", 2020-06-12T02:38:26.799 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012052894 2020-06-12T02:38:26.800 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.800 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.800 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.801 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.690217+0000", 2020-06-12T02:38:26.801 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6490999999999998e-05 2020-06-12T02:38:26.801 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.802 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.802 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:26.802 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.690274+0000", 2020-06-12T02:38:26.803 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000866722 2020-06-12T02:38:26.803 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.803 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.803 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:26.805 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.691141+0000", 2020-06-12T02:38:26.805 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025091826000000001 2020-06-12T02:38:26.806 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:26.806 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:26.807 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:26.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.808 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.808 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:11452 3.bs0 3:d0ebae7d:::smithi10729407-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e56)", 2020-06-12T02:38:26.808 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:21.956864+0000", 2020-06-12T02:38:26.809 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 4.5028240500000001, 2020-06-12T02:38:26.809 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020123195999999999, 2020-06-12T02:38:26.809 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:26.810 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:26.810 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:26.811 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:26.811 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:26.811 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 11452 2020-06-12T02:38:26.814 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.814 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:26.814 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.815 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:26.815 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.956864+0000", 2020-06-12T02:38:26.815 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:26.815 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.815 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.815 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:26.816 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.956864+0000", 2020-06-12T02:38:26.816 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3441000000000001e-05 2020-06-12T02:38:26.816 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.816 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.816 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:26.817 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.956887+0000", 2020-06-12T02:38:26.817 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2590000000000001e-06 2020-06-12T02:38:26.817 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.817 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.817 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:26.818 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.956891+0000", 2020-06-12T02:38:26.818 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1019999999999999e-06 2020-06-12T02:38:26.818 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.818 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.818 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:26.818 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.956898+0000", 2020-06-12T02:38:26.819 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.1329999999999999e-06 2020-06-12T02:38:26.819 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.819 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.819 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:26.820 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.956901+0000", 2020-06-12T02:38:26.820 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00063642400000000002 2020-06-12T02:38:26.820 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.820 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.820 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.821 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.957537+0000", 2020-06-12T02:38:26.821 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0488000000000005e-05 2020-06-12T02:38:26.821 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.821 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.821 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:26.822 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.957628+0000", 2020-06-12T02:38:26.822 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0083444469999999996 2020-06-12T02:38:26.822 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.822 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.823 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.823 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.965972+0000", 2020-06-12T02:38:26.823 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.482e-05 2020-06-12T02:38:26.823 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.823 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.824 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:26.824 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.966027+0000", 2020-06-12T02:38:26.824 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0095845240000000005 2020-06-12T02:38:26.824 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.824 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.824 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:26.825 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.975612+0000", 2020-06-12T02:38:26.825 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1042e-05 2020-06-12T02:38:26.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.825 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.825 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:26.826 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.975703+0000", 2020-06-12T02:38:26.826 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012845160000000001 2020-06-12T02:38:26.826 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:26.826 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:26.826 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:26.827 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.976987+0000", 2020-06-12T02:38:26.827 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020123195999999999 2020-06-12T02:38:26.827 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:26.827 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:26.827 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:26.827 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:26.828 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:26.828 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:26.848 INFO:tasks.rados.rados.0.smithi107.stdout:6057: writing smithi10729407-18 from 516096 to 1146880 tid 1 2020-06-12T02:38:26.849 INFO:tasks.rados.rados.0.smithi107.stdout:6058: delete oid 3 current snap is 0 2020-06-12T02:38:26.849 INFO:tasks.rados.rados.0.smithi107.stdout:6053: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:26.849 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1087 (ObjNum 6052 snap 0 seq_num 6052) dirty exists 2020-06-12T02:38:26.850 INFO:tasks.rados.rados.0.smithi107.stdout:6053: left oid 19 (ObjNum 6052 snap 0 seq_num 6052) 2020-06-12T02:38:26.850 INFO:tasks.rados.rados.0.smithi107.stdout:6053: done (3 left) 2020-06-12T02:38:26.850 INFO:tasks.rados.rados.0.smithi107.stdout:6058: done (2 left) 2020-06-12T02:38:26.850 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:38:26.850 INFO:tasks.rados.rados.0.smithi107.stdout:6059: seq_num 6058 ranges {0=663552} 2020-06-12T02:38:26.851 INFO:tasks.rados.rados.0.smithi107.stdout:6059: writing smithi10729407-16 from 0 to 663552 tid 1 2020-06-12T02:38:26.851 INFO:tasks.rados.rados.0.smithi107.stdout:6060: delete oid 6 current snap is 0 2020-06-12T02:38:26.851 INFO:tasks.rados.rados.0.smithi107.stdout:6057: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:26.851 INFO:tasks.rados.rados.0.smithi107.stdout:6057: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:26.851 INFO:tasks.rados.rados.0.smithi107.stdout:6057: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:26.852 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1099 (ObjNum 6056 snap 0 seq_num 6056) dirty exists 2020-06-12T02:38:26.852 INFO:tasks.rados.rados.0.smithi107.stdout:6057: left oid 18 (ObjNum 6056 snap 0 seq_num 6056) 2020-06-12T02:38:26.852 INFO:tasks.rados.rados.0.smithi107.stdout:6057: done (3 left) 2020-06-12T02:38:26.852 INFO:tasks.rados.rados.0.smithi107.stdout:6060: done (2 left) 2020-06-12T02:38:26.852 INFO:tasks.rados.rados.0.smithi107.stdout:6061: delete oid 14 current snap is 0 2020-06-12T02:38:26.853 INFO:tasks.rados.rados.0.smithi107.stdout:6061: done (2 left) 2020-06-12T02:38:26.853 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:26.853 INFO:tasks.rados.rados.0.smithi107.stdout:6062: seq_num 6061 ranges {0=712704} 2020-06-12T02:38:26.853 INFO:tasks.rados.rados.0.smithi107.stdout:6062: writing smithi10729407-4 from 0 to 712704 tid 1 2020-06-12T02:38:26.853 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:26.854 INFO:tasks.rados.rados.0.smithi107.stdout:6063: seq_num 6062 ranges {450560=737280} 2020-06-12T02:38:26.854 INFO:tasks.rados.rados.0.smithi107.stdout:6063: writing smithi10729407-1 from 450560 to 1187840 tid 1 2020-06-12T02:38:26.854 INFO:tasks.rados.rados.0.smithi107.stdout:6064: delete oid 3 current snap is 0 2020-06-12T02:38:26.854 INFO:tasks.rados.rados.0.smithi107.stdout:6059: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:26.854 INFO:tasks.rados.rados.0.smithi107.stdout:6059: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:26.855 INFO:tasks.rados.rados.0.smithi107.stdout:6059: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:26.855 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1059 (ObjNum 6058 snap 0 seq_num 6058) dirty exists 2020-06-12T02:38:26.855 INFO:tasks.rados.rados.0.smithi107.stdout:6059: left oid 16 (ObjNum 6058 snap 0 seq_num 6058) 2020-06-12T02:38:26.855 INFO:tasks.rados.rados.0.smithi107.stdout:6062: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:26.855 INFO:tasks.rados.rados.0.smithi107.stdout:6059: done (4 left) 2020-06-12T02:38:26.856 INFO:tasks.rados.rados.0.smithi107.stdout:6064: done (3 left) 2020-06-12T02:38:26.856 INFO:tasks.rados.rados.0.smithi107.stdout:6065: delete oid 2 current snap is 0 2020-06-12T02:38:26.856 INFO:tasks.rados.rados.0.smithi107.stdout:6062: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:26.856 INFO:tasks.rados.rados.0.smithi107.stdout:6062: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:26.856 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1132 (ObjNum 6061 snap 0 seq_num 6061) dirty exists 2020-06-12T02:38:26.857 INFO:tasks.rados.rados.0.smithi107.stdout:6062: left oid 4 (ObjNum 6061 snap 0 seq_num 6061) 2020-06-12T02:38:26.857 INFO:tasks.rados.rados.0.smithi107.stdout:6062: done (3 left) 2020-06-12T02:38:26.857 INFO:tasks.rados.rados.0.smithi107.stdout:6065: done (2 left) 2020-06-12T02:38:26.857 INFO:tasks.rados.rados.0.smithi107.stdout:6066: delete oid 17 current snap is 0 2020-06-12T02:38:26.857 INFO:tasks.rados.rados.0.smithi107.stdout:6066: done (2 left) 2020-06-12T02:38:26.858 INFO:tasks.rados.rados.0.smithi107.stdout:6067: delete oid 5 current snap is 0 2020-06-12T02:38:26.858 INFO:tasks.rados.rados.0.smithi107.stdout:6063: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:26.858 INFO:tasks.rados.rados.0.smithi107.stdout:6067: done (2 left) 2020-06-12T02:38:26.858 INFO:tasks.rados.rados.0.smithi107.stdout:6068: delete oid 19 current snap is 0 2020-06-12T02:38:26.858 INFO:tasks.rados.rados.0.smithi107.stdout:6063: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:26.859 INFO:tasks.rados.rados.0.smithi107.stdout:6063: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:26.859 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1128 (ObjNum 6062 snap 0 seq_num 6062) dirty exists 2020-06-12T02:38:26.859 INFO:tasks.rados.rados.0.smithi107.stdout:6063: left oid 1 (ObjNum 6062 snap 0 seq_num 6062) 2020-06-12T02:38:26.859 INFO:tasks.rados.rados.0.smithi107.stdout:6063: done (2 left) 2020-06-12T02:38:26.859 INFO:tasks.rados.rados.0.smithi107.stdout:6068: done (1 left) 2020-06-12T02:38:26.860 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:26.860 INFO:tasks.rados.rados.0.smithi107.stdout:6069: seq_num 6068 ranges {0=434176,434176=401408} 2020-06-12T02:38:26.860 INFO:tasks.rados.rados.0.smithi107.stdout:6069: writing smithi10729407-11 from 0 to 434176 tid 1 2020-06-12T02:38:26.860 INFO:tasks.rados.rados.0.smithi107.stdout:6069: writing smithi10729407-11 from 434176 to 835584 tid 2 2020-06-12T02:38:26.861 INFO:tasks.rados.rados.0.smithi107.stdout:6070: delete oid 5 current snap is 0 2020-06-12T02:38:26.861 INFO:tasks.rados.rados.0.smithi107.stdout:6070: done (2 left) 2020-06-12T02:38:26.861 INFO:tasks.rados.rados.0.smithi107.stdout:6071: delete oid 13 current snap is 0 2020-06-12T02:38:26.861 INFO:tasks.rados.rados.0.smithi107.stdout:6069: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:26.861 INFO:tasks.rados.rados.0.smithi107.stdout:6069: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:26.862 INFO:tasks.rados.rados.0.smithi107.stdout:6071: done (2 left) 2020-06-12T02:38:26.862 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:26.862 INFO:tasks.rados.rados.0.smithi107.stdout:6072: seq_num 6071 ranges {0=483328,483328=401408} 2020-06-12T02:38:26.862 INFO:tasks.rados.rados.0.smithi107.stdout:6072: writing smithi10729407-17 from 0 to 483328 tid 1 2020-06-12T02:38:26.862 INFO:tasks.rados.rados.0.smithi107.stdout:6072: writing smithi10729407-17 from 483328 to 884736 tid 2 2020-06-12T02:38:26.862 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:26.863 INFO:tasks.rados.rados.0.smithi107.stdout:6073: seq_num 6072 ranges {0=499712,499712=401408} 2020-06-12T02:38:26.863 INFO:tasks.rados.rados.0.smithi107.stdout:6073: writing smithi10729407-2 from 0 to 499712 tid 1 2020-06-12T02:38:26.863 INFO:tasks.rados.rados.0.smithi107.stdout:6073: writing smithi10729407-2 from 499712 to 901120 tid 2 2020-06-12T02:38:26.863 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:26.863 INFO:tasks.rados.rados.0.smithi107.stdout:6074: seq_num 6073 ranges {1146880=516096,1662976=401408} 2020-06-12T02:38:26.864 INFO:tasks.rados.rados.0.smithi107.stdout:6074: writing smithi10729407-18 from 1146880 to 1662976 tid 1 2020-06-12T02:38:26.864 INFO:tasks.rados.rados.0.smithi107.stdout:6074: writing smithi10729407-18 from 1662976 to 2064384 tid 2 2020-06-12T02:38:26.864 INFO:tasks.rados.rados.0.smithi107.stdout:6075: delete oid 16 current snap is 0 2020-06-12T02:38:26.864 INFO:tasks.rados.rados.0.smithi107.stdout:6069: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:26.864 INFO:tasks.rados.rados.0.smithi107.stdout:6069: finishing write tid 5 to smithi10729407-11 2020-06-12T02:38:26.865 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1078 (ObjNum 6068 snap 0 seq_num 6068) dirty exists 2020-06-12T02:38:26.865 INFO:tasks.rados.rados.0.smithi107.stdout:6069: left oid 11 (ObjNum 6068 snap 0 seq_num 6068) 2020-06-12T02:38:26.865 INFO:tasks.rados.rados.0.smithi107.stdout:6072: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:26.865 INFO:tasks.rados.rados.0.smithi107.stdout:6072: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:26.865 INFO:tasks.rados.rados.0.smithi107.stdout:6072: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:26.866 INFO:tasks.rados.rados.0.smithi107.stdout:6073: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:26.872 INFO:tasks.rados.rados.0.smithi107.stdout:6072: finishing write tid 5 to smithi10729407-17 2020-06-12T02:38:26.873 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1136 (ObjNum 6071 snap 0 seq_num 6071) dirty exists 2020-06-12T02:38:26.873 INFO:tasks.rados.rados.0.smithi107.stdout:6072: left oid 17 (ObjNum 6071 snap 0 seq_num 6071) 2020-06-12T02:38:26.873 INFO:tasks.rados.rados.0.smithi107.stdout:6073: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:26.874 INFO:tasks.rados.rados.0.smithi107.stdout:6073: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:26.874 INFO:tasks.rados.rados.0.smithi107.stdout:6073: finishing write tid 5 to smithi10729407-2 2020-06-12T02:38:26.875 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 578 (ObjNum 6072 snap 0 seq_num 6072) dirty exists 2020-06-12T02:38:26.875 INFO:tasks.rados.rados.0.smithi107.stdout:6073: left oid 2 (ObjNum 6072 snap 0 seq_num 6072) 2020-06-12T02:38:26.876 INFO:tasks.rados.rados.0.smithi107.stdout:6069: done (5 left) 2020-06-12T02:38:26.876 INFO:tasks.rados.rados.0.smithi107.stdout:6072: done (4 left) 2020-06-12T02:38:26.876 INFO:tasks.rados.rados.0.smithi107.stdout:6073: done (3 left) 2020-06-12T02:38:26.876 INFO:tasks.rados.rados.0.smithi107.stdout:6075: done (2 left) 2020-06-12T02:38:26.877 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:38:26.877 INFO:tasks.rados.rados.0.smithi107.stdout:6076: seq_num 6075 ranges {0=548864,548864=401408} 2020-06-12T02:38:26.877 INFO:tasks.rados.rados.0.smithi107.stdout:6076: writing smithi10729407-3 from 0 to 548864 tid 1 2020-06-12T02:38:26.878 INFO:tasks.rados.rados.0.smithi107.stdout:6076: writing smithi10729407-3 from 548864 to 950272 tid 2 2020-06-12T02:38:26.878 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:26.879 INFO:tasks.rados.rados.0.smithi107.stdout:6077: seq_num 6076 ranges {1187840=565248,1753088=401408} 2020-06-12T02:38:26.879 INFO:tasks.rados.rados.0.smithi107.stdout:6077: writing smithi10729407-1 from 1187840 to 1753088 tid 1 2020-06-12T02:38:26.879 INFO:tasks.rados.rados.0.smithi107.stdout:6077: writing smithi10729407-1 from 1753088 to 2154496 tid 2 2020-06-12T02:38:26.880 INFO:tasks.rados.rados.0.smithi107.stdout:6078: delete oid 19 current snap is 0 2020-06-12T02:38:26.881 INFO:tasks.rados.rados.0.smithi107.stdout:6074: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:26.881 INFO:tasks.rados.rados.0.smithi107.stdout:6074: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:26.881 INFO:tasks.rados.rados.0.smithi107.stdout:6074: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:26.881 INFO:tasks.rados.rados.0.smithi107.stdout:6076: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:26.882 INFO:tasks.rados.rados.0.smithi107.stdout:6076: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:26.882 INFO:tasks.rados.rados.0.smithi107.stdout:6074: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:26.883 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1102 (ObjNum 6073 snap 0 seq_num 6073) dirty exists 2020-06-12T02:38:26.883 INFO:tasks.rados.rados.0.smithi107.stdout:6074: left oid 18 (ObjNum 6073 snap 0 seq_num 6073) 2020-06-12T02:38:26.883 INFO:tasks.rados.rados.0.smithi107.stdout:6076: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:26.884 INFO:tasks.rados.rados.0.smithi107.stdout:6076: finishing write tid 5 to smithi10729407-3 2020-06-12T02:38:26.884 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1064 (ObjNum 6075 snap 0 seq_num 6075) dirty exists 2020-06-12T02:38:26.884 INFO:tasks.rados.rados.0.smithi107.stdout:6076: left oid 3 (ObjNum 6075 snap 0 seq_num 6075) 2020-06-12T02:38:26.884 INFO:tasks.rados.rados.0.smithi107.stdout:6074: done (4 left) 2020-06-12T02:38:26.885 INFO:tasks.rados.rados.0.smithi107.stdout:6076: done (3 left) 2020-06-12T02:38:26.885 INFO:tasks.rados.rados.0.smithi107.stdout:6078: done (2 left) 2020-06-12T02:38:26.885 INFO:tasks.rados.rados.0.smithi107.stdout:6079: delete oid 9 current snap is 0 2020-06-12T02:38:26.886 INFO:tasks.rados.rados.0.smithi107.stdout:6077: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:26.886 INFO:tasks.rados.rados.0.smithi107.stdout:6079: done (2 left) 2020-06-12T02:38:26.886 INFO:tasks.rados.rados.0.smithi107.stdout:6080: delete oid 11 current snap is 0 2020-06-12T02:38:26.886 INFO:tasks.rados.rados.0.smithi107.stdout:6080: done (2 left) 2020-06-12T02:38:26.887 INFO:tasks.rados.rados.0.smithi107.stdout:6081: delete oid 17 current snap is 0 2020-06-12T02:38:26.887 INFO:tasks.rados.rados.0.smithi107.stdout:6077: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:26.887 INFO:tasks.rados.rados.0.smithi107.stdout:6077: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:26.888 INFO:tasks.rados.rados.0.smithi107.stdout:6081: done (2 left) 2020-06-12T02:38:26.888 INFO:tasks.rados.rados.0.smithi107.stdout:6082: delete oid 8 current snap is 0 2020-06-12T02:38:26.888 INFO:tasks.rados.rados.0.smithi107.stdout:6077: finishing write tid 5 to smithi10729407-1 2020-06-12T02:38:26.889 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1131 (ObjNum 6076 snap 0 seq_num 6076) dirty exists 2020-06-12T02:38:26.889 INFO:tasks.rados.rados.0.smithi107.stdout:6077: left oid 1 (ObjNum 6076 snap 0 seq_num 6076) 2020-06-12T02:38:26.889 INFO:tasks.rados.rados.0.smithi107.stdout:6077: done (2 left) 2020-06-12T02:38:26.890 INFO:tasks.rados.rados.0.smithi107.stdout:6082: done (1 left) 2020-06-12T02:38:26.890 INFO:tasks.rados.rados.0.smithi107.stdout:6083: delete oid 4 current snap is 0 2020-06-12T02:38:26.890 INFO:tasks.rados.rados.0.smithi107.stdout:6083: done (1 left) 2020-06-12T02:38:26.891 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:38:26.891 INFO:tasks.rados.rados.0.smithi107.stdout:6084: seq_num 6083 ranges {0=688128,688128=401408} 2020-06-12T02:38:26.891 INFO:tasks.rados.rados.0.smithi107.stdout:6084: writing smithi10729407-16 from 0 to 688128 tid 1 2020-06-12T02:38:26.891 INFO:tasks.rados.rados.0.smithi107.stdout:6084: writing smithi10729407-16 from 688128 to 1089536 tid 2 2020-06-12T02:38:26.891 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:26.892 INFO:tasks.rados.rados.0.smithi107.stdout:6085: seq_num 6084 ranges {0=704512,704512=401408} 2020-06-12T02:38:26.892 INFO:tasks.rados.rados.0.smithi107.stdout:6085: writing smithi10729407-17 from 0 to 704512 tid 1 2020-06-12T02:38:26.892 INFO:tasks.rados.rados.0.smithi107.stdout:6085: writing smithi10729407-17 from 704512 to 1105920 tid 2 2020-06-12T02:38:26.892 INFO:tasks.rados.rados.0.smithi107.stdout:6086: delete oid 9 current snap is 0 2020-06-12T02:38:26.892 INFO:tasks.rados.rados.0.smithi107.stdout:6084: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:26.893 INFO:tasks.rados.rados.0.smithi107.stdout:6084: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:26.893 INFO:tasks.rados.rados.0.smithi107.stdout:6084: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:26.893 INFO:tasks.rados.rados.0.smithi107.stdout:6084: finishing write tid 5 to smithi10729407-16 2020-06-12T02:38:26.893 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1067 (ObjNum 6083 snap 0 seq_num 6083) dirty exists 2020-06-12T02:38:26.894 INFO:tasks.rados.rados.0.smithi107.stdout:6084: left oid 16 (ObjNum 6083 snap 0 seq_num 6083) 2020-06-12T02:38:26.894 INFO:tasks.rados.rados.0.smithi107.stdout:6085: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:26.894 INFO:tasks.rados.rados.0.smithi107.stdout:6084: done (3 left) 2020-06-12T02:38:26.894 INFO:tasks.rados.rados.0.smithi107.stdout:6086: done (2 left) 2020-06-12T02:38:26.894 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:26.895 INFO:tasks.rados.rados.0.smithi107.stdout:6087: seq_num 6086 ranges {0=737280,737280=401408} 2020-06-12T02:38:26.895 INFO:tasks.rados.rados.0.smithi107.stdout:6087: writing smithi10729407-4 from 0 to 737280 tid 1 2020-06-12T02:38:26.895 INFO:tasks.rados.rados.0.smithi107.stdout:6087: writing smithi10729407-4 from 737280 to 1138688 tid 2 2020-06-12T02:38:26.895 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:38:26.895 INFO:tasks.rados.rados.0.smithi107.stdout:6088: seq_num 6087 ranges {0=753664,753664=401408} 2020-06-12T02:38:26.896 INFO:tasks.rados.rados.0.smithi107.stdout:6088: writing smithi10729407-5 from 0 to 753664 tid 1 2020-06-12T02:38:26.896 INFO:tasks.rados.rados.0.smithi107.stdout:6088: writing smithi10729407-5 from 753664 to 1155072 tid 2 2020-06-12T02:38:26.896 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:26.897 INFO:tasks.rados.rados.0.smithi107.stdout:6089: seq_num 6088 ranges {0=770048,770048=401408} 2020-06-12T02:38:26.897 INFO:tasks.rados.rados.0.smithi107.stdout:6089: writing smithi10729407-19 from 0 to 770048 tid 1 2020-06-12T02:38:26.897 INFO:tasks.rados.rados.0.smithi107.stdout:6089: writing smithi10729407-19 from 770048 to 1171456 tid 2 2020-06-12T02:38:26.897 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:26.897 INFO:tasks.rados.rados.0.smithi107.stdout:6090: seq_num 6089 ranges {0=786432,786432=401408} 2020-06-12T02:38:26.898 INFO:tasks.rados.rados.0.smithi107.stdout:6090: writing smithi10729407-8 from 0 to 786432 tid 1 2020-06-12T02:38:26.898 INFO:tasks.rados.rados.0.smithi107.stdout:6090: writing smithi10729407-8 from 786432 to 1187840 tid 2 2020-06-12T02:38:26.898 INFO:tasks.rados.rados.0.smithi107.stdout:6091: delete oid 7 current snap is 0 2020-06-12T02:38:26.898 INFO:tasks.rados.rados.0.smithi107.stdout:6085: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:26.898 INFO:tasks.rados.rados.0.smithi107.stdout:6085: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:26.899 INFO:tasks.rados.rados.0.smithi107.stdout:6085: finishing write tid 5 to smithi10729407-17 2020-06-12T02:38:26.899 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1141 (ObjNum 6084 snap 0 seq_num 6084) dirty exists 2020-06-12T02:38:26.899 INFO:tasks.rados.rados.0.smithi107.stdout:6085: left oid 17 (ObjNum 6084 snap 0 seq_num 6084) 2020-06-12T02:38:26.899 INFO:tasks.rados.rados.0.smithi107.stdout:6087: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:26.899 INFO:tasks.rados.rados.0.smithi107.stdout:6087: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:26.899 INFO:tasks.rados.rados.0.smithi107.stdout:6087: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:26.900 INFO:tasks.rados.rados.0.smithi107.stdout:6088: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:26.900 INFO:tasks.rados.rados.0.smithi107.stdout:6087: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:26.900 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1144 (ObjNum 6086 snap 0 seq_num 6086) dirty exists 2020-06-12T02:38:26.900 INFO:tasks.rados.rados.0.smithi107.stdout:6087: left oid 4 (ObjNum 6086 snap 0 seq_num 6086) 2020-06-12T02:38:26.900 INFO:tasks.rados.rados.0.smithi107.stdout:6088: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:26.901 INFO:tasks.rados.rados.0.smithi107.stdout:6088: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:26.901 INFO:tasks.rados.rados.0.smithi107.stdout:6088: finishing write tid 5 to smithi10729407-5 2020-06-12T02:38:26.901 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 612 (ObjNum 6087 snap 0 seq_num 6087) dirty exists 2020-06-12T02:38:26.901 INFO:tasks.rados.rados.0.smithi107.stdout:6088: left oid 5 (ObjNum 6087 snap 0 seq_num 6087) 2020-06-12T02:38:26.902 INFO:tasks.rados.rados.0.smithi107.stdout:6089: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:26.902 INFO:tasks.rados.rados.0.smithi107.stdout:6085: done (6 left) 2020-06-12T02:38:26.902 INFO:tasks.rados.rados.0.smithi107.stdout:6087: done (5 left) 2020-06-12T02:38:26.902 INFO:tasks.rados.rados.0.smithi107.stdout:6088: done (4 left) 2020-06-12T02:38:26.902 INFO:tasks.rados.rados.0.smithi107.stdout:6091: done (3 left) 2020-06-12T02:38:26.903 INFO:tasks.rados.rados.0.smithi107.stdout:6092: delete oid 13 current snap is 0 2020-06-12T02:38:26.903 INFO:tasks.rados.rados.0.smithi107.stdout:6092: done (3 left) 2020-06-12T02:38:26.903 INFO:tasks.rados.rados.0.smithi107.stdout:6093: delete oid 11 current snap is 0 2020-06-12T02:38:26.903 INFO:tasks.rados.rados.0.smithi107.stdout:6089: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:26.904 INFO:tasks.rados.rados.0.smithi107.stdout:6090: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:26.904 INFO:tasks.rados.rados.0.smithi107.stdout:6090: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:26.904 INFO:tasks.rados.rados.0.smithi107.stdout:6089: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:26.904 INFO:tasks.rados.rados.0.smithi107.stdout:6093: done (3 left) 2020-06-12T02:38:26.904 INFO:tasks.rados.rados.0.smithi107.stdout:6094: delete oid 1 current snap is 0 2020-06-12T02:38:26.905 INFO:tasks.rados.rados.0.smithi107.stdout:6090: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:26.905 INFO:tasks.rados.rados.0.smithi107.stdout:6090: finishing write tid 5 to smithi10729407-8 2020-06-12T02:38:26.905 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1125 (ObjNum 6089 snap 0 seq_num 6089) dirty exists 2020-06-12T02:38:26.905 INFO:tasks.rados.rados.0.smithi107.stdout:6090: left oid 8 (ObjNum 6089 snap 0 seq_num 6089) 2020-06-12T02:38:26.906 INFO:tasks.rados.rados.0.smithi107.stdout:6089: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:26.906 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1092 (ObjNum 6088 snap 0 seq_num 6088) dirty exists 2020-06-12T02:38:26.906 INFO:tasks.rados.rados.0.smithi107.stdout:6089: left oid 19 (ObjNum 6088 snap 0 seq_num 6088) 2020-06-12T02:38:26.906 INFO:tasks.rados.rados.0.smithi107.stdout:6089: done (3 left) 2020-06-12T02:38:26.906 INFO:tasks.rados.rados.0.smithi107.stdout:6090: done (2 left) 2020-06-12T02:38:26.907 INFO:tasks.rados.rados.0.smithi107.stdout:6094: done (1 left) 2020-06-12T02:38:26.907 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:26.907 INFO:tasks.rados.rados.0.smithi107.stdout:6095: seq_num 6094 ranges {1138688=466944,1605632=516096,2121728=286720} 2020-06-12T02:38:26.907 INFO:tasks.rados.rados.0.smithi107.stdout:6095: writing smithi10729407-4 from 1138688 to 1605632 tid 1 2020-06-12T02:38:26.907 INFO:tasks.rados.rados.0.smithi107.stdout:6095: writing smithi10729407-4 from 1605632 to 2121728 tid 2 2020-06-12T02:38:26.907 INFO:tasks.rados.rados.0.smithi107.stdout:6095: writing smithi10729407-4 from 2121728 to 2408448 tid 3 2020-06-12T02:38:26.908 INFO:tasks.rados.rados.0.smithi107.stdout:6096: delete oid 1 current snap is 0 2020-06-12T02:38:26.908 INFO:tasks.rados.rados.0.smithi107.stdout:6095: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:26.908 INFO:tasks.rados.rados.0.smithi107.stdout:6096: done (2 left) 2020-06-12T02:38:26.908 INFO:tasks.rados.rados.0.smithi107.stdout:6097: delete oid 20 current snap is 0 2020-06-12T02:38:26.908 INFO:tasks.rados.rados.0.smithi107.stdout:6097: done (2 left) 2020-06-12T02:38:26.909 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:26.909 INFO:tasks.rados.rados.0.smithi107.stdout:6098: seq_num 6097 ranges {950272=516096,1466368=573440,2039808=229376} 2020-06-12T02:38:26.909 INFO:tasks.rados.rados.0.smithi107.stdout:6098: writing smithi10729407-3 from 950272 to 1466368 tid 1 2020-06-12T02:38:26.909 INFO:tasks.rados.rados.0.smithi107.stdout:6098: writing smithi10729407-3 from 1466368 to 2039808 tid 2 2020-06-12T02:38:26.909 INFO:tasks.rados.rados.0.smithi107.stdout:6098: writing smithi10729407-3 from 2039808 to 2269184 tid 3 2020-06-12T02:38:26.909 INFO:tasks.rados.rados.0.smithi107.stdout:6099: delete oid 15 current snap is 0 2020-06-12T02:38:26.910 INFO:tasks.rados.rados.0.smithi107.stdout:6095: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:26.910 INFO:tasks.rados.rados.0.smithi107.stdout:6095: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:26.910 INFO:tasks.rados.rados.0.smithi107.stdout:6095: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:26.910 INFO:tasks.rados.rados.0.smithi107.stdout:6095: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:26.910 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1148 (ObjNum 6094 snap 0 seq_num 6094) dirty exists 2020-06-12T02:38:26.911 INFO:tasks.rados.rados.0.smithi107.stdout:6095: left oid 4 (ObjNum 6094 snap 0 seq_num 6094) 2020-06-12T02:38:26.911 INFO:tasks.rados.rados.0.smithi107.stdout:6098: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:26.911 INFO:tasks.rados.rados.0.smithi107.stdout:6095: done (3 left) 2020-06-12T02:38:26.911 INFO:tasks.rados.rados.0.smithi107.stdout:6099: done (2 left) 2020-06-12T02:38:26.912 INFO:tasks.rados.rados.0.smithi107.stdout:6100: delete oid 15 current snap is 0 2020-06-12T02:38:26.912 INFO:tasks.rados.rados.0.smithi107.stdout:6100: done (2 left) 2020-06-12T02:38:26.912 INFO:tasks.rados.rados.0.smithi107.stdout:6101: delete oid 5 current snap is 0 2020-06-12T02:38:26.912 INFO:tasks.rados.rados.0.smithi107.stdout:6098: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:26.912 INFO:tasks.rados.rados.0.smithi107.stdout:6098: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:26.912 INFO:tasks.rados.rados.0.smithi107.stdout:6098: finishing write tid 5 to smithi10729407-3 2020-06-12T02:38:26.913 INFO:tasks.rados.rados.0.smithi107.stdout:6098: finishing write tid 6 to smithi10729407-3 2020-06-12T02:38:26.913 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1071 (ObjNum 6097 snap 0 seq_num 6097) dirty exists 2020-06-12T02:38:26.913 INFO:tasks.rados.rados.0.smithi107.stdout:6098: left oid 3 (ObjNum 6097 snap 0 seq_num 6097) 2020-06-12T02:38:26.913 INFO:tasks.rados.rados.0.smithi107.stdout:6098: done (2 left) 2020-06-12T02:38:26.914 INFO:tasks.rados.rados.0.smithi107.stdout:6101: done (1 left) 2020-06-12T02:38:26.914 INFO:tasks.rados.rados.0.smithi107.stdout:6102: delete oid 2 current snap is 0 2020-06-12T02:38:26.914 INFO:tasks.rados.rados.0.smithi107.stdout:6102: done (1 left) 2020-06-12T02:38:26.914 INFO:tasks.rados.rados.0.smithi107.stdout:6103: delete oid 18 current snap is 0 2020-06-12T02:38:26.915 INFO:tasks.rados.rados.0.smithi107.stdout:6103: done (1 left) 2020-06-12T02:38:26.915 INFO:tasks.rados.rados.0.smithi107.stdout:6104: delete oid 16 current snap is 0 2020-06-12T02:38:26.915 INFO:tasks.rados.rados.0.smithi107.stdout:6104: done (1 left) 2020-06-12T02:38:26.915 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:38:26.915 INFO:tasks.rados.rados.0.smithi107.stdout:6105: seq_num 6104 ranges {0=638976,638976=614400,1253376=188416} 2020-06-12T02:38:26.916 INFO:tasks.rados.rados.0.smithi107.stdout:6105: writing smithi10729407-7 from 0 to 638976 tid 1 2020-06-12T02:38:26.916 INFO:tasks.rados.rados.0.smithi107.stdout:6105: writing smithi10729407-7 from 638976 to 1253376 tid 2 2020-06-12T02:38:26.916 INFO:tasks.rados.rados.0.smithi107.stdout:6105: writing smithi10729407-7 from 1253376 to 1441792 tid 3 2020-06-12T02:38:26.916 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:26.917 INFO:tasks.rados.rados.0.smithi107.stdout:6106: seq_num 6105 ranges {1187840=655360,1843200=548864,2392064=253952} 2020-06-12T02:38:26.917 INFO:tasks.rados.rados.0.smithi107.stdout:6106: writing smithi10729407-8 from 1187840 to 1843200 tid 1 2020-06-12T02:38:26.917 INFO:tasks.rados.rados.0.smithi107.stdout:6106: writing smithi10729407-8 from 1843200 to 2392064 tid 2 2020-06-12T02:38:26.917 INFO:tasks.rados.rados.0.smithi107.stdout:6106: writing smithi10729407-8 from 2392064 to 2646016 tid 3 2020-06-12T02:38:26.917 INFO:tasks.rados.rados.0.smithi107.stdout:6107: delete oid 5 current snap is 0 2020-06-12T02:38:26.917 INFO:tasks.rados.rados.0.smithi107.stdout:6105: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:26.918 INFO:tasks.rados.rados.0.smithi107.stdout:6105: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:26.918 INFO:tasks.rados.rados.0.smithi107.stdout:6105: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:26.918 INFO:tasks.rados.rados.0.smithi107.stdout:6105: finishing write tid 5 to smithi10729407-7 2020-06-12T02:38:26.918 INFO:tasks.rados.rados.0.smithi107.stdout:6105: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:26.918 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1167 (ObjNum 6104 snap 0 seq_num 6104) dirty exists 2020-06-12T02:38:26.919 INFO:tasks.rados.rados.0.smithi107.stdout:6105: left oid 7 (ObjNum 6104 snap 0 seq_num 6104) 2020-06-12T02:38:26.919 INFO:tasks.rados.rados.0.smithi107.stdout:6105: done (3 left) 2020-06-12T02:38:26.919 INFO:tasks.rados.rados.0.smithi107.stdout:6107: done (2 left) 2020-06-12T02:38:26.919 INFO:tasks.rados.rados.0.smithi107.stdout:6108: delete oid 20 current snap is 0 2020-06-12T02:38:26.919 INFO:tasks.rados.rados.0.smithi107.stdout:6106: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:26.919 INFO:tasks.rados.rados.0.smithi107.stdout:6106: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:26.920 INFO:tasks.rados.rados.0.smithi107.stdout:6106: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:26.920 INFO:tasks.rados.rados.0.smithi107.stdout:6106: finishing write tid 5 to smithi10729407-8 2020-06-12T02:38:26.920 INFO:tasks.rados.rados.0.smithi107.stdout:6108: done (2 left) 2020-06-12T02:38:26.920 INFO:tasks.rados.rados.0.smithi107.stdout:6109: delete oid 6 current snap is 0 2020-06-12T02:38:26.920 INFO:tasks.rados.rados.0.smithi107.stdout:6106: finishing write tid 6 to smithi10729407-8 2020-06-12T02:38:26.921 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1131 (ObjNum 6105 snap 0 seq_num 6105) dirty exists 2020-06-12T02:38:26.921 INFO:tasks.rados.rados.0.smithi107.stdout:6106: left oid 8 (ObjNum 6105 snap 0 seq_num 6105) 2020-06-12T02:38:26.921 INFO:tasks.rados.rados.0.smithi107.stdout:6106: done (2 left) 2020-06-12T02:38:26.921 INFO:tasks.rados.rados.0.smithi107.stdout:6109: done (1 left) 2020-06-12T02:38:26.921 INFO:tasks.rados.rados.0.smithi107.stdout:6110: delete oid 9 current snap is 0 2020-06-12T02:38:26.922 INFO:tasks.rados.rados.0.smithi107.stdout:6110: done (1 left) 2020-06-12T02:38:26.922 INFO:tasks.rados.rados.0.smithi107.stdout:6111: delete oid 15 current snap is 0 2020-06-12T02:38:26.922 INFO:tasks.rados.rados.0.smithi107.stdout:6111: done (1 left) 2020-06-12T02:38:26.922 INFO:tasks.rados.rados.0.smithi107.stdout:6112: delete oid 7 current snap is 0 2020-06-12T02:38:26.923 INFO:tasks.rados.rados.0.smithi107.stdout:6112: done (1 left) 2020-06-12T02:38:26.923 INFO:tasks.rados.rados.0.smithi107.stdout:6113: delete oid 4 current snap is 0 2020-06-12T02:38:26.923 INFO:tasks.rados.rados.0.smithi107.stdout:6113: done (1 left) 2020-06-12T02:38:26.923 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:38:26.924 INFO:tasks.rados.rados.0.smithi107.stdout:6114: seq_num 6113 ranges {0=786432,786432=466944,1253376=335872} 2020-06-12T02:38:26.924 INFO:tasks.rados.rados.0.smithi107.stdout:6114: writing smithi10729407-6 from 0 to 786432 tid 1 2020-06-12T02:38:26.925 INFO:tasks.rados.rados.0.smithi107.stdout:6114: writing smithi10729407-6 from 786432 to 1253376 tid 2 2020-06-12T02:38:26.925 INFO:tasks.rados.rados.0.smithi107.stdout:6114: writing smithi10729407-6 from 1253376 to 1589248 tid 3 2020-06-12T02:38:26.925 INFO:tasks.rados.rados.0.smithi107.stdout:6115: delete oid 5 current snap is 0 2020-06-12T02:38:26.925 INFO:tasks.rados.rados.0.smithi107.stdout:6115: done (2 left) 2020-06-12T02:38:26.926 INFO:tasks.rados.rados.0.smithi107.stdout:6116: delete oid 14 current snap is 0 2020-06-12T02:38:26.926 INFO:tasks.rados.rados.0.smithi107.stdout:6116: done (2 left) 2020-06-12T02:38:26.926 INFO:tasks.rados.rados.0.smithi107.stdout:6117: delete oid 3 current snap is 0 2020-06-12T02:38:26.927 INFO:tasks.rados.rados.0.smithi107.stdout:6114: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:26.927 INFO:tasks.rados.rados.0.smithi107.stdout:6114: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:26.927 INFO:tasks.rados.rados.0.smithi107.stdout:6117: done (2 left) 2020-06-12T02:38:26.927 INFO:tasks.rados.rados.0.smithi107.stdout:6118: delete oid 2 current snap is 0 2020-06-12T02:38:26.928 INFO:tasks.rados.rados.0.smithi107.stdout:6114: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:26.928 INFO:tasks.rados.rados.0.smithi107.stdout:6114: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:26.928 INFO:tasks.rados.rados.0.smithi107.stdout:6114: finishing write tid 6 to smithi10729407-6 2020-06-12T02:38:26.929 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1138 (ObjNum 6113 snap 0 seq_num 6113) dirty exists 2020-06-12T02:38:26.929 INFO:tasks.rados.rados.0.smithi107.stdout:6114: left oid 6 (ObjNum 6113 snap 0 seq_num 6113) 2020-06-12T02:38:26.929 INFO:tasks.rados.rados.0.smithi107.stdout:6114: done (2 left) 2020-06-12T02:38:26.929 INFO:tasks.rados.rados.0.smithi107.stdout:6118: done (1 left) 2020-06-12T02:38:26.930 INFO:tasks.rados.rados.0.smithi107.stdout:6119: delete oid 13 current snap is 0 2020-06-12T02:38:26.930 INFO:tasks.rados.rados.0.smithi107.stdout:6119: done (1 left) 2020-06-12T02:38:26.930 INFO:tasks.rados.rados.0.smithi107.stdout:6120: delete oid 20 current snap is 0 2020-06-12T02:38:26.931 INFO:tasks.rados.rados.0.smithi107.stdout:6120: done (1 left) 2020-06-12T02:38:26.931 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:26.931 INFO:tasks.rados.rados.0.smithi107.stdout:6121: seq_num 6120 ranges {0=507904,507904=507904,1015808=573440,1589248=122880} 2020-06-12T02:38:26.931 INFO:tasks.rados.rados.0.smithi107.stdout:6121: writing smithi10729407-14 from 0 to 507904 tid 1 2020-06-12T02:38:26.932 INFO:tasks.rados.rados.0.smithi107.stdout:6121: writing smithi10729407-14 from 507904 to 1015808 tid 2 2020-06-12T02:38:26.932 INFO:tasks.rados.rados.0.smithi107.stdout:6121: writing smithi10729407-14 from 1015808 to 1589248 tid 3 2020-06-12T02:38:26.932 INFO:tasks.rados.rados.0.smithi107.stdout:6121: writing smithi10729407-14 from 1589248 to 1712128 tid 4 2020-06-12T02:38:26.933 INFO:tasks.rados.rados.0.smithi107.stdout:6122: delete oid 8 current snap is 0 2020-06-12T02:38:26.933 INFO:tasks.rados.rados.0.smithi107.stdout:6121: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:26.933 INFO:tasks.rados.rados.0.smithi107.stdout:6122: done (2 left) 2020-06-12T02:38:26.933 INFO:tasks.rados.rados.0.smithi107.stdout:6123: delete oid 2 current snap is 0 2020-06-12T02:38:26.934 INFO:tasks.rados.rados.0.smithi107.stdout:6121: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:26.934 INFO:tasks.rados.rados.0.smithi107.stdout:6123: done (2 left) 2020-06-12T02:38:26.934 INFO:tasks.rados.rados.0.smithi107.stdout:6124: delete oid 18 current snap is 0 2020-06-12T02:38:26.935 INFO:tasks.rados.rados.0.smithi107.stdout:6121: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:26.935 INFO:tasks.rados.rados.0.smithi107.stdout:6121: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:26.935 INFO:tasks.rados.rados.0.smithi107.stdout:6121: finishing write tid 6 to smithi10729407-14 2020-06-12T02:38:26.935 INFO:tasks.rados.rados.0.smithi107.stdout:6124: done (2 left) 2020-06-12T02:38:26.935 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:26.935 INFO:tasks.rados.rados.0.smithi107.stdout:6125: seq_num 6124 ranges {0=573440,573440=450560,1024000=557056,1581056=196608} 2020-06-12T02:38:26.936 INFO:tasks.rados.rados.0.smithi107.stdout:6125: writing smithi10729407-4 from 0 to 573440 tid 1 2020-06-12T02:38:26.936 INFO:tasks.rados.rados.0.smithi107.stdout:6125: writing smithi10729407-4 from 573440 to 1024000 tid 2 2020-06-12T02:38:26.936 INFO:tasks.rados.rados.0.smithi107.stdout:6125: writing smithi10729407-4 from 1024000 to 1581056 tid 3 2020-06-12T02:38:26.936 INFO:tasks.rados.rados.0.smithi107.stdout:6125: writing smithi10729407-4 from 1581056 to 1777664 tid 4 2020-06-12T02:38:26.936 INFO:tasks.rados.rados.0.smithi107.stdout:6126: delete oid 18 current snap is 0 2020-06-12T02:38:26.937 INFO:tasks.rados.rados.0.smithi107.stdout:6121: finishing write tid 7 to smithi10729407-14 2020-06-12T02:38:26.937 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1086 (ObjNum 6120 snap 0 seq_num 6120) dirty exists 2020-06-12T02:38:26.937 INFO:tasks.rados.rados.0.smithi107.stdout:6121: left oid 14 (ObjNum 6120 snap 0 seq_num 6120) 2020-06-12T02:38:26.937 INFO:tasks.rados.rados.0.smithi107.stdout:6125: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:26.937 INFO:tasks.rados.rados.0.smithi107.stdout:6125: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:26.938 INFO:tasks.rados.rados.0.smithi107.stdout:6121: done (3 left) 2020-06-12T02:38:26.938 INFO:tasks.rados.rados.0.smithi107.stdout:6126: done (2 left) 2020-06-12T02:38:26.938 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:38:26.938 INFO:tasks.rados.rados.0.smithi107.stdout:6127: seq_num 6126 ranges {0=606208,606208=614400,1220608=524288,1744896=65536} 2020-06-12T02:38:26.938 INFO:tasks.rados.rados.0.smithi107.stdout:6127: writing smithi10729407-2 from 0 to 606208 tid 1 2020-06-12T02:38:26.938 INFO:tasks.rados.rados.0.smithi107.stdout:6127: writing smithi10729407-2 from 606208 to 1220608 tid 2 2020-06-12T02:38:26.939 INFO:tasks.rados.rados.0.smithi107.stdout:6127: writing smithi10729407-2 from 1220608 to 1744896 tid 3 2020-06-12T02:38:26.939 INFO:tasks.rados.rados.0.smithi107.stdout:6127: writing smithi10729407-2 from 1744896 to 1810432 tid 4 2020-06-12T02:38:26.939 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:26.939 INFO:tasks.rados.rados.0.smithi107.stdout:6125: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:26.939 INFO:tasks.rados.rados.0.smithi107.stdout:6128: seq_num 6127 ranges {0=622592,622592=499712,1122304=704512} 2020-06-12T02:38:26.940 INFO:tasks.rados.rados.0.smithi107.stdout:6128: writing smithi10729407-12 from 0 to 622592 tid 1 2020-06-12T02:38:26.940 INFO:tasks.rados.rados.0.smithi107.stdout:6128: writing smithi10729407-12 from 622592 to 1122304 tid 2 2020-06-12T02:38:26.940 INFO:tasks.rados.rados.0.smithi107.stdout:6128: writing smithi10729407-12 from 1122304 to 1826816 tid 3 2020-06-12T02:38:26.940 INFO:tasks.rados.rados.0.smithi107.stdout:6129: delete oid 11 current snap is 0 2020-06-12T02:38:26.940 INFO:tasks.rados.rados.0.smithi107.stdout:6125: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:26.940 INFO:tasks.rados.rados.0.smithi107.stdout:6125: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:26.941 INFO:tasks.rados.rados.0.smithi107.stdout:6125: finishing write tid 7 to smithi10729407-4 2020-06-12T02:38:26.941 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1154 (ObjNum 6124 snap 0 seq_num 6124) dirty exists 2020-06-12T02:38:26.941 INFO:tasks.rados.rados.0.smithi107.stdout:6125: left oid 4 (ObjNum 6124 snap 0 seq_num 6124) 2020-06-12T02:38:26.941 INFO:tasks.rados.rados.0.smithi107.stdout:6127: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:26.941 INFO:tasks.rados.rados.0.smithi107.stdout:6127: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:26.942 INFO:tasks.rados.rados.0.smithi107.stdout:6127: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:26.942 INFO:tasks.rados.rados.0.smithi107.stdout:6128: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:26.942 INFO:tasks.rados.rados.0.smithi107.stdout:6127: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:26.942 INFO:tasks.rados.rados.0.smithi107.stdout:6127: finishing write tid 6 to smithi10729407-2 2020-06-12T02:38:26.942 INFO:tasks.rados.rados.0.smithi107.stdout:6127: finishing write tid 7 to smithi10729407-2 2020-06-12T02:38:26.942 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 586 (ObjNum 6126 snap 0 seq_num 6126) dirty exists 2020-06-12T02:38:26.943 INFO:tasks.rados.rados.0.smithi107.stdout:6127: left oid 2 (ObjNum 6126 snap 0 seq_num 6126) 2020-06-12T02:38:26.943 INFO:tasks.rados.rados.0.smithi107.stdout:6125: done (4 left) 2020-06-12T02:38:26.943 INFO:tasks.rados.rados.0.smithi107.stdout:6127: done (3 left) 2020-06-12T02:38:26.943 INFO:tasks.rados.rados.0.smithi107.stdout:6129: done (2 left) 2020-06-12T02:38:26.943 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:26.943 INFO:tasks.rados.rados.0.smithi107.stdout:6130: seq_num 6129 ranges {0=655360,655360=720896,1376256=483328} 2020-06-12T02:38:26.944 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-06-12T02:38:26.945 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:26.827+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:26.950 INFO:tasks.rados.rados.0.smithi107.stdout:6130: writing smithi10729407-3 from 0 to 655360 tid 1 2020-06-12T02:38:26.950 INFO:tasks.rados.rados.0.smithi107.stdout:6130: writing smithi10729407-3 from 655360 to 1376256 tid 2 2020-06-12T02:38:26.951 INFO:tasks.rados.rados.0.smithi107.stdout:6130: writing smithi10729407-3 from 1376256 to 1859584 tid 3 2020-06-12T02:38:26.951 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:26.951 INFO:tasks.rados.rados.0.smithi107.stdout:6131: seq_num 6130 ranges {1171456=671744,1843200=606208,2449408=499712,2949120=98304} 2020-06-12T02:38:26.951 INFO:tasks.rados.rados.0.smithi107.stdout:6131: writing smithi10729407-19 from 1171456 to 1843200 tid 1 2020-06-12T02:38:26.951 INFO:tasks.rados.rados.0.smithi107.stdout:6131: writing smithi10729407-19 from 1843200 to 2449408 tid 2 2020-06-12T02:38:26.952 INFO:tasks.rados.rados.0.smithi107.stdout:6131: writing smithi10729407-19 from 2449408 to 2949120 tid 3 2020-06-12T02:38:26.952 INFO:tasks.rados.rados.0.smithi107.stdout:6131: writing smithi10729407-19 from 2949120 to 3047424 tid 4 2020-06-12T02:38:26.952 INFO:tasks.rados.rados.0.smithi107.stdout:6132: delete oid 14 current snap is 0 2020-06-12T02:38:26.952 INFO:tasks.rados.rados.0.smithi107.stdout:6128: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:26.952 INFO:tasks.rados.rados.0.smithi107.stdout:6128: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:26.953 INFO:tasks.rados.rados.0.smithi107.stdout:6128: finishing write tid 5 to smithi10729407-12 2020-06-12T02:38:26.953 INFO:tasks.rados.rados.0.smithi107.stdout:6128: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:26.953 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1096 (ObjNum 6127 snap 0 seq_num 6127) dirty exists 2020-06-12T02:38:26.953 INFO:tasks.rados.rados.0.smithi107.stdout:6128: left oid 12 (ObjNum 6127 snap 0 seq_num 6127) 2020-06-12T02:38:26.953 INFO:tasks.rados.rados.0.smithi107.stdout:6130: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:26.953 INFO:tasks.rados.rados.0.smithi107.stdout:6130: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:26.954 INFO:tasks.rados.rados.0.smithi107.stdout:6130: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:26.954 INFO:tasks.rados.rados.0.smithi107.stdout:6130: finishing write tid 5 to smithi10729407-3 2020-06-12T02:38:26.954 INFO:tasks.rados.rados.0.smithi107.stdout:6130: finishing write tid 6 to smithi10729407-3 2020-06-12T02:38:26.954 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1077 (ObjNum 6129 snap 0 seq_num 6129) dirty exists 2020-06-12T02:38:26.954 INFO:tasks.rados.rados.0.smithi107.stdout:6130: left oid 3 (ObjNum 6129 snap 0 seq_num 6129) 2020-06-12T02:38:26.955 INFO:tasks.rados.rados.0.smithi107.stdout:6131: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:26.955 INFO:tasks.rados.rados.0.smithi107.stdout:6128: done (4 left) 2020-06-12T02:38:26.955 INFO:tasks.rados.rados.0.smithi107.stdout:6130: done (3 left) 2020-06-12T02:38:26.955 INFO:tasks.rados.rados.0.smithi107.stdout:6132: done (2 left) 2020-06-12T02:38:26.956 INFO:tasks.rados.rados.0.smithi107.stdout:6133: delete oid 12 current snap is 0 2020-06-12T02:38:26.956 INFO:tasks.rados.rados.0.smithi107.stdout:6131: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:26.956 INFO:tasks.rados.rados.0.smithi107.stdout:6131: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:26.956 INFO:tasks.rados.rados.0.smithi107.stdout:6131: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:26.956 INFO:tasks.rados.rados.0.smithi107.stdout:6131: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:26.957 INFO:tasks.rados.rados.0.smithi107.stdout:6131: finishing write tid 7 to smithi10729407-19 2020-06-12T02:38:26.957 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1101 (ObjNum 6130 snap 0 seq_num 6130) dirty exists 2020-06-12T02:38:26.957 INFO:tasks.rados.rados.0.smithi107.stdout:6131: left oid 19 (ObjNum 6130 snap 0 seq_num 6130) 2020-06-12T02:38:26.957 INFO:tasks.rados.rados.0.smithi107.stdout:6131: done (2 left) 2020-06-12T02:38:26.957 INFO:tasks.rados.rados.0.smithi107.stdout:6133: done (1 left) 2020-06-12T02:38:26.957 INFO:tasks.rados.rados.0.smithi107.stdout:6134: delete oid 8 current snap is 0 2020-06-12T02:38:26.958 INFO:tasks.rados.rados.0.smithi107.stdout:6134: done (1 left) 2020-06-12T02:38:26.958 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:26.958 INFO:tasks.rados.rados.0.smithi107.stdout:6135: seq_num 6134 ranges {0=737280,737280=540672,1277952=483328,1761280=180224} 2020-06-12T02:38:26.958 INFO:tasks.rados.rados.0.smithi107.stdout:6135: writing smithi10729407-14 from 0 to 737280 tid 1 2020-06-12T02:38:26.958 INFO:tasks.rados.rados.0.smithi107.stdout:6135: writing smithi10729407-14 from 737280 to 1277952 tid 2 2020-06-12T02:38:26.959 INFO:tasks.rados.rados.0.smithi107.stdout:6135: writing smithi10729407-14 from 1277952 to 1761280 tid 3 2020-06-12T02:38:26.959 INFO:tasks.rados.rados.0.smithi107.stdout:6135: writing smithi10729407-14 from 1761280 to 1941504 tid 4 2020-06-12T02:38:26.959 INFO:tasks.rados.rados.0.smithi107.stdout:6136: delete oid 9 current snap is 0 2020-06-12T02:38:26.959 INFO:tasks.rados.rados.0.smithi107.stdout:6135: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:26.960 INFO:tasks.rados.rados.0.smithi107.stdout:6136: done (2 left) 2020-06-12T02:38:26.960 INFO:tasks.rados.rados.0.smithi107.stdout:6137: delete oid 16 current snap is 0 2020-06-12T02:38:26.960 INFO:tasks.rados.rados.0.smithi107.stdout:6137: done (2 left) 2020-06-12T02:38:26.960 INFO:tasks.rados.rados.0.smithi107.stdout:6138: delete oid 19 current snap is 0 2020-06-12T02:38:26.960 INFO:tasks.rados.rados.0.smithi107.stdout:6135: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:26.961 INFO:tasks.rados.rados.0.smithi107.stdout:6135: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:26.961 INFO:tasks.rados.rados.0.smithi107.stdout:6138: done (2 left) 2020-06-12T02:38:26.961 INFO:tasks.rados.rados.0.smithi107.stdout:6139: delete oid 5 current snap is 0 2020-06-12T02:38:26.961 INFO:tasks.rados.rados.0.smithi107.stdout:6135: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:26.961 INFO:tasks.rados.rados.0.smithi107.stdout:6135: finishing write tid 6 to smithi10729407-14 2020-06-12T02:38:26.961 INFO:tasks.rados.rados.0.smithi107.stdout:6139: done (2 left) 2020-06-12T02:38:26.962 INFO:tasks.rados.rados.0.smithi107.stdout:6140: delete oid 12 current snap is 0 2020-06-12T02:38:26.962 INFO:tasks.rados.rados.0.smithi107.stdout:6135: finishing write tid 7 to smithi10729407-14 2020-06-12T02:38:26.962 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1093 (ObjNum 6134 snap 0 seq_num 6134) dirty exists 2020-06-12T02:38:26.962 INFO:tasks.rados.rados.0.smithi107.stdout:6135: left oid 14 (ObjNum 6134 snap 0 seq_num 6134) 2020-06-12T02:38:26.962 INFO:tasks.rados.rados.0.smithi107.stdout:6135: done (2 left) 2020-06-12T02:38:26.963 INFO:tasks.rados.rados.0.smithi107.stdout:6140: done (1 left) 2020-06-12T02:38:26.963 INFO:tasks.rados.rados.0.smithi107.stdout:6141: delete oid 18 current snap is 0 2020-06-12T02:38:26.963 INFO:tasks.rados.rados.0.smithi107.stdout:6141: done (1 left) 2020-06-12T02:38:26.963 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:26.963 INFO:tasks.rados.rados.0.smithi107.stdout:6142: seq_num 6141 ranges {0=458752,458752=581632,1040384=688128,1728512=335872} 2020-06-12T02:38:26.963 INFO:tasks.rados.rados.0.smithi107.stdout:6142: writing smithi10729407-13 from 0 to 458752 tid 1 2020-06-12T02:38:26.964 INFO:tasks.rados.rados.0.smithi107.stdout:6142: writing smithi10729407-13 from 458752 to 1040384 tid 2 2020-06-12T02:38:26.964 INFO:tasks.rados.rados.0.smithi107.stdout:6142: writing smithi10729407-13 from 1040384 to 1728512 tid 3 2020-06-12T02:38:26.964 INFO:tasks.rados.rados.0.smithi107.stdout:6142: writing smithi10729407-13 from 1728512 to 2064384 tid 4 2020-06-12T02:38:26.964 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:26.965 INFO:tasks.rados.rados.0.smithi107.stdout:6143: seq_num 6142 ranges {0=475136,475136=466944,942080=442368,1384448=696320} 2020-06-12T02:38:26.965 INFO:tasks.rados.rados.0.smithi107.stdout:6143: writing smithi10729407-12 from 0 to 475136 tid 1 2020-06-12T02:38:26.965 INFO:tasks.rados.rados.0.smithi107.stdout:6143: writing smithi10729407-12 from 475136 to 942080 tid 2 2020-06-12T02:38:26.965 INFO:tasks.rados.rados.0.smithi107.stdout:6143: writing smithi10729407-12 from 942080 to 1384448 tid 3 2020-06-12T02:38:26.966 INFO:tasks.rados.rados.0.smithi107.stdout:6143: writing smithi10729407-12 from 1384448 to 2080768 tid 4 2020-06-12T02:38:26.966 INFO:tasks.rados.rados.0.smithi107.stdout:6144: delete oid 18 current snap is 0 2020-06-12T02:38:26.966 INFO:tasks.rados.rados.0.smithi107.stdout:6142: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:26.966 INFO:tasks.rados.rados.0.smithi107.stdout:6142: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:26.966 INFO:tasks.rados.rados.0.smithi107.stdout:6142: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:26.967 INFO:tasks.rados.rados.0.smithi107.stdout:6142: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:26.967 INFO:tasks.rados.rados.0.smithi107.stdout:6142: finishing write tid 6 to smithi10729407-13 2020-06-12T02:38:26.967 INFO:tasks.rados.rados.0.smithi107.stdout:6143: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:26.967 INFO:tasks.rados.rados.0.smithi107.stdout:6142: finishing write tid 7 to smithi10729407-13 2020-06-12T02:38:26.967 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1174 (ObjNum 6141 snap 0 seq_num 6141) dirty exists 2020-06-12T02:38:26.967 INFO:tasks.rados.rados.0.smithi107.stdout:6142: left oid 13 (ObjNum 6141 snap 0 seq_num 6141) 2020-06-12T02:38:26.968 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:26.943+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:26.969 INFO:tasks.rados.rados.0.smithi107.stdout:6142: done (3 left) 2020-06-12T02:38:26.970 INFO:tasks.rados.rados.0.smithi107.stdout:6144: done (2 left) 2020-06-12T02:38:26.970 INFO:tasks.rados.rados.0.smithi107.stdout:6145: delete oid 13 current snap is 0 2020-06-12T02:38:26.970 INFO:tasks.rados.rados.0.smithi107.stdout:6143: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:26.970 INFO:tasks.rados.rados.0.smithi107.stdout:6143: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:26.970 INFO:tasks.rados.rados.0.smithi107.stdout:6145: done (2 left) 2020-06-12T02:38:26.971 INFO:tasks.rados.rados.0.smithi107.stdout:6146: delete oid 9 current snap is 0 2020-06-12T02:38:26.971 INFO:tasks.rados.rados.0.smithi107.stdout:6146: done (2 left) 2020-06-12T02:38:26.971 INFO:tasks.rados.rados.0.smithi107.stdout:6147: delete oid 4 current snap is 0 2020-06-12T02:38:26.971 INFO:tasks.rados.rados.0.smithi107.stdout:6143: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:26.971 INFO:tasks.rados.rados.0.smithi107.stdout:6143: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:26.972 INFO:tasks.rados.rados.0.smithi107.stdout:6147: done (2 left) 2020-06-12T02:38:26.972 INFO:tasks.rados.rados.0.smithi107.stdout:6148: delete oid 4 current snap is 0 2020-06-12T02:38:26.972 INFO:tasks.rados.rados.0.smithi107.stdout:6143: finishing write tid 7 to smithi10729407-12 2020-06-12T02:38:26.972 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1109 (ObjNum 6142 snap 0 seq_num 6142) dirty exists 2020-06-12T02:38:26.972 INFO:tasks.rados.rados.0.smithi107.stdout:6143: left oid 12 (ObjNum 6142 snap 0 seq_num 6142) 2020-06-12T02:38:26.973 INFO:tasks.rados.rados.0.smithi107.stdout:6143: done (2 left) 2020-06-12T02:38:26.973 INFO:tasks.rados.rados.0.smithi107.stdout:6148: done (1 left) 2020-06-12T02:38:26.973 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:26.974 INFO:tasks.rados.rados.0.smithi107.stdout:6149: seq_num 6148 ranges {0=573440,573440=622592,1196032=434176,1630208=532480,2162688=16384} 2020-06-12T02:38:26.974 INFO:tasks.rados.rados.0.smithi107.stdout:6149: writing smithi10729407-11 from 0 to 573440 tid 1 2020-06-12T02:38:26.977 INFO:tasks.rados.rados.0.smithi107.stdout:6149: writing smithi10729407-11 from 573440 to 1196032 tid 2 2020-06-12T02:38:26.979 INFO:tasks.rados.rados.0.smithi107.stdout:6149: writing smithi10729407-11 from 1196032 to 1630208 tid 3 2020-06-12T02:38:26.983 INFO:tasks.rados.rados.0.smithi107.stdout:6149: writing smithi10729407-11 from 1630208 to 2162688 tid 4 2020-06-12T02:38:26.983 INFO:tasks.rados.rados.0.smithi107.stdout:6149: writing smithi10729407-11 from 2162688 to 2179072 tid 5 2020-06-12T02:38:26.983 INFO:tasks.rados.rados.0.smithi107.stdout:6150: delete oid 8 current snap is 0 2020-06-12T02:38:26.984 INFO:tasks.rados.rados.0.smithi107.stdout:6149: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:26.986 INFO:tasks.rados.rados.0.smithi107.stdout:6150: done (2 left) 2020-06-12T02:38:26.986 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:26.987 INFO:tasks.rados.rados.0.smithi107.stdout:6151: seq_num 6150 ranges {0=606208,606208=786432,1392640=802816,2195456=16384} 2020-06-12T02:38:26.990 INFO:tasks.rados.rados.0.smithi107.stdout:6151: writing smithi10729407-7 from 0 to 606208 tid 1 2020-06-12T02:38:26.994 INFO:tasks.rados.rados.0.smithi107.stdout:6151: writing smithi10729407-7 from 606208 to 1392640 tid 2 2020-06-12T02:38:26.999 INFO:tasks.rados.rados.0.smithi107.stdout:6151: writing smithi10729407-7 from 1392640 to 2195456 tid 3 2020-06-12T02:38:26.999 INFO:tasks.rados.rados.0.smithi107.stdout:6151: writing smithi10729407-7 from 2195456 to 2211840 tid 4 2020-06-12T02:38:27.000 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:27.000 INFO:tasks.rados.rados.0.smithi107.stdout:6152: seq_num 6151 ranges {0=622592,622592=720896,1343488=614400,1957888=270336} 2020-06-12T02:38:27.003 INFO:tasks.rados.rados.0.smithi107.stdout:6152: writing smithi10729407-15 from 0 to 622592 tid 1 2020-06-12T02:38:27.007 INFO:tasks.rados.rados.0.smithi107.stdout:6152: writing smithi10729407-15 from 622592 to 1343488 tid 2 2020-06-12T02:38:27.010 INFO:tasks.rados.rados.0.smithi107.stdout:6152: writing smithi10729407-15 from 1343488 to 1957888 tid 3 2020-06-12T02:38:27.012 INFO:tasks.rados.rados.0.smithi107.stdout:6152: writing smithi10729407-15 from 1957888 to 2228224 tid 4 2020-06-12T02:38:27.012 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:38:27.012 INFO:tasks.rados.rados.0.smithi107.stdout:6153: seq_num 6152 ranges {0=638976,638976=606208,1245184=417792,1662976=581632} 2020-06-12T02:38:27.015 INFO:tasks.rados.rados.0.smithi107.stdout:6153: writing smithi10729407-16 from 0 to 638976 tid 1 2020-06-12T02:38:27.018 INFO:tasks.rados.rados.0.smithi107.stdout:6153: writing smithi10729407-16 from 638976 to 1245184 tid 2 2020-06-12T02:38:27.021 INFO:tasks.rados.rados.0.smithi107.stdout:6153: writing smithi10729407-16 from 1245184 to 1662976 tid 3 2020-06-12T02:38:27.025 INFO:tasks.rados.rados.0.smithi107.stdout:6153: writing smithi10729407-16 from 1662976 to 2244608 tid 4 2020-06-12T02:38:27.026 INFO:tasks.rados.rados.0.smithi107.stdout:6154: delete oid 13 current snap is 0 2020-06-12T02:38:27.026 INFO:tasks.rados.rados.0.smithi107.stdout:6149: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:27.026 INFO:tasks.rados.rados.0.smithi107.stdout:6149: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:27.026 INFO:tasks.rados.rados.0.smithi107.stdout:6149: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:27.026 INFO:tasks.rados.rados.0.smithi107.stdout:6149: finishing write tid 5 to smithi10729407-11 2020-06-12T02:38:27.027 INFO:tasks.rados.rados.0.smithi107.stdout:6149: finishing write tid 7 to smithi10729407-11 2020-06-12T02:38:27.027 INFO:tasks.rados.rados.0.smithi107.stdout:6149: finishing write tid 8 to smithi10729407-11 2020-06-12T02:38:27.027 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1099 (ObjNum 6148 snap 0 seq_num 6148) dirty exists 2020-06-12T02:38:27.028 INFO:tasks.rados.rados.0.smithi107.stdout:6149: left oid 11 (ObjNum 6148 snap 0 seq_num 6148) 2020-06-12T02:38:27.028 INFO:tasks.rados.rados.0.smithi107.stdout:6151: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:27.028 INFO:tasks.rados.rados.0.smithi107.stdout:6151: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:27.028 INFO:tasks.rados.rados.0.smithi107.stdout:6151: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:27.028 INFO:tasks.rados.rados.0.smithi107.stdout:6151: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:27.028 INFO:tasks.rados.rados.0.smithi107.stdout:6151: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:27.029 INFO:tasks.rados.rados.0.smithi107.stdout:6152: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:27.029 INFO:tasks.rados.rados.0.smithi107.stdout:6151: finishing write tid 7 to smithi10729407-7 2020-06-12T02:38:27.029 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1180 (ObjNum 6150 snap 0 seq_num 6150) dirty exists 2020-06-12T02:38:27.029 INFO:tasks.rados.rados.0.smithi107.stdout:6151: left oid 7 (ObjNum 6150 snap 0 seq_num 6150) 2020-06-12T02:38:27.029 INFO:tasks.rados.rados.0.smithi107.stdout:6152: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:27.030 INFO:tasks.rados.rados.0.smithi107.stdout:6153: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:27.030 INFO:tasks.rados.rados.0.smithi107.stdout:6152: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:27.033 INFO:tasks.rados.rados.0.smithi107.stdout:6152: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:27.033 INFO:tasks.rados.rados.0.smithi107.stdout:6152: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:27.035 INFO:tasks.rados.rados.0.smithi107.stdout:6149: done (5 left) 2020-06-12T02:38:27.035 INFO:tasks.rados.rados.0.smithi107.stdout:6151: done (4 left) 2020-06-12T02:38:27.035 INFO:tasks.rados.rados.0.smithi107.stdout:6154: done (3 left) 2020-06-12T02:38:27.036 INFO:tasks.rados.rados.0.smithi107.stdout:6155: delete oid 7 current snap is 0 2020-06-12T02:38:27.036 INFO:tasks.rados.rados.0.smithi107.stdout:6153: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:27.036 INFO:tasks.rados.rados.0.smithi107.stdout:6153: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:27.037 INFO:tasks.rados.rados.0.smithi107.stdout:6152: finishing write tid 7 to smithi10729407-15 2020-06-12T02:38:27.037 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1140 (ObjNum 6151 snap 0 seq_num 6151) dirty exists 2020-06-12T02:38:27.037 INFO:tasks.rados.rados.0.smithi107.stdout:6152: left oid 15 (ObjNum 6151 snap 0 seq_num 6151) 2020-06-12T02:38:27.038 INFO:tasks.rados.rados.0.smithi107.stdout:6153: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:27.038 INFO:tasks.rados.rados.0.smithi107.stdout:6153: finishing write tid 6 to smithi10729407-16 2020-06-12T02:38:27.042 INFO:tasks.rados.rados.0.smithi107.stdout:6152: done (3 left) 2020-06-12T02:38:27.043 INFO:tasks.rados.rados.0.smithi107.stdout:6155: done (2 left) 2020-06-12T02:38:27.043 INFO:tasks.rados.rados.0.smithi107.stdout:6156: delete oid 20 current snap is 0 2020-06-12T02:38:27.043 INFO:tasks.rados.rados.0.smithi107.stdout:6153: finishing write tid 7 to smithi10729407-16 2020-06-12T02:38:27.044 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1083 (ObjNum 6152 snap 0 seq_num 6152) dirty exists 2020-06-12T02:38:27.044 INFO:tasks.rados.rados.0.smithi107.stdout:6153: left oid 16 (ObjNum 6152 snap 0 seq_num 6152) 2020-06-12T02:38:27.044 INFO:tasks.rados.rados.0.smithi107.stdout:6153: done (2 left) 2020-06-12T02:38:27.045 INFO:tasks.rados.rados.0.smithi107.stdout:6156: done (1 left) 2020-06-12T02:38:27.045 INFO:tasks.rados.rados.0.smithi107.stdout:6157: delete oid 16 current snap is 0 2020-06-12T02:38:27.046 INFO:tasks.rados.rados.0.smithi107.stdout:6157: done (1 left) 2020-06-12T02:38:27.046 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:27.046 INFO:tasks.rados.rados.0.smithi107.stdout:6158: seq_num 6157 ranges {0=720896,720896=425984,1146880=606208,1753088=524288,2277376=49152} 2020-06-12T02:38:27.048 INFO:tasks.rados.rados.0.smithi107.stdout:6158: writing smithi10729407-18 from 0 to 720896 tid 1 2020-06-12T02:38:27.049 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:27.046+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:27.051 INFO:tasks.rados.rados.0.smithi107.stdout:6158: writing smithi10729407-18 from 720896 to 1146880 tid 2 2020-06-12T02:38:27.055 INFO:tasks.rados.rados.0.smithi107.stdout:6158: writing smithi10729407-18 from 1146880 to 1753088 tid 3 2020-06-12T02:38:27.058 INFO:tasks.rados.rados.0.smithi107.stdout:6158: writing smithi10729407-18 from 1753088 to 2277376 tid 4 2020-06-12T02:38:27.059 INFO:tasks.rados.rados.0.smithi107.stdout:6158: writing smithi10729407-18 from 2277376 to 2326528 tid 5 2020-06-12T02:38:27.059 INFO:tasks.rados.rados.0.smithi107.stdout:6159: delete oid 9 current snap is 0 2020-06-12T02:38:27.059 INFO:tasks.rados.rados.0.smithi107.stdout:6158: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:27.060 INFO:tasks.rados.rados.0.smithi107.stdout:6158: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:27.066 INFO:tasks.rados.rados.0.smithi107.stdout:6158: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:27.066 INFO:tasks.rados.rados.0.smithi107.stdout:6158: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:27.067 INFO:tasks.rados.rados.0.smithi107.stdout:6158: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:27.068 INFO:tasks.rados.rados.0.smithi107.stdout:6158: finishing write tid 7 to smithi10729407-18 2020-06-12T02:38:27.069 INFO:tasks.rados.rados.0.smithi107.stdout:6159: done (2 left) 2020-06-12T02:38:27.070 INFO:tasks.rados.rados.0.smithi107.stdout:6160: delete oid 15 current snap is 0 2020-06-12T02:38:27.070 INFO:tasks.rados.rados.0.smithi107.stdout:6158: finishing write tid 8 to smithi10729407-18 2020-06-12T02:38:27.070 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1118 (ObjNum 6157 snap 0 seq_num 6157) dirty exists 2020-06-12T02:38:27.071 INFO:tasks.rados.rados.0.smithi107.stdout:6158: left oid 18 (ObjNum 6157 snap 0 seq_num 6157) 2020-06-12T02:38:27.072 INFO:tasks.rados.rados.0.smithi107.stdout:6158: done (2 left) 2020-06-12T02:38:27.072 INFO:tasks.rados.rados.0.smithi107.stdout:6160: done (1 left) 2020-06-12T02:38:27.072 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:27.073 INFO:tasks.rados.rados.0.smithi107.stdout:6161: seq_num 6160 ranges {2080768=778240,2859008=532480,3391488=778240,4169728=294912} 2020-06-12T02:38:27.087 INFO:tasks.rados.rados.0.smithi107.stdout:6161: writing smithi10729407-12 from 2080768 to 2859008 tid 1 2020-06-12T02:38:27.090 INFO:tasks.rados.rados.0.smithi107.stdout:6161: writing smithi10729407-12 from 2859008 to 3391488 tid 2 2020-06-12T02:38:27.095 INFO:tasks.rados.rados.0.smithi107.stdout:6161: writing smithi10729407-12 from 3391488 to 4169728 tid 3 2020-06-12T02:38:27.097 INFO:tasks.rados.rados.0.smithi107.stdout:6161: writing smithi10729407-12 from 4169728 to 4464640 tid 4 2020-06-12T02:38:27.098 INFO:tasks.rados.rados.0.smithi107.stdout:6162: delete oid 18 current snap is 0 2020-06-12T02:38:27.100 INFO:tasks.rados.rados.0.smithi107.stdout:6162: done (2 left) 2020-06-12T02:38:27.100 INFO:tasks.rados.rados.0.smithi107.stdout:6163: delete oid 2 current snap is 0 2020-06-12T02:38:27.102 INFO:tasks.rados.rados.0.smithi107.stdout:6161: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:27.102 INFO:tasks.rados.rados.0.smithi107.stdout:6161: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:27.103 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:27.103 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:38:27.103 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.103 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13205 3.9s0 3:977e4b9d:::smithi10729407-12:head [append 0~778240 in=778240b] snapc 0=[] ondisk+write+known_if_redirected e57)", 2020-06-12T02:38:27.104 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:27.086463+0000", 2020-06-12T02:38:27.104 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.013239762, 2020-06-12T02:38:27.104 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013274136000000001, 2020-06-12T02:38:27.104 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:27.104 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:27.104 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:27.105 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:27.105 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:27.105 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13205 2020-06-12T02:38:27.105 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.105 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:27.106 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.106 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:27.106 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.086463+0000", 2020-06-12T02:38:27.106 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:27.106 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.107 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.107 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:27.107 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.086463+0000", 2020-06-12T02:38:27.107 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00079805999999999996 2020-06-12T02:38:27.107 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.107 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.108 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:27.108 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.087261+0000", 2020-06-12T02:38:27.108 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4359999999999999e-06 2020-06-12T02:38:27.108 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.108 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.109 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:27.109 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.087268+0000", 2020-06-12T02:38:27.109 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2508e-05 2020-06-12T02:38:27.109 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.109 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.110 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:27.110 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.087280+0000", 2020-06-12T02:38:27.110 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4030000000000004e-06 2020-06-12T02:38:27.110 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.110 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.110 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:27.111 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.087286+0000", 2020-06-12T02:38:27.111 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5356000000000002e-05 2020-06-12T02:38:27.111 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.111 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.112 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.112 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.087352+0000", 2020-06-12T02:38:27.112 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4491000000000001e-05 2020-06-12T02:38:27.112 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.112 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.113 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:27.113 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.087436+0000", 2020-06-12T02:38:27.113 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00064190799999999995 2020-06-12T02:38:27.113 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.113 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.114 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:27.114 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.088078+0000", 2020-06-12T02:38:27.114 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.007570678 2020-06-12T02:38:27.114 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.114 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.114 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:27.115 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.095649+0000", 2020-06-12T02:38:27.115 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0091858400000000007 2020-06-12T02:38:27.115 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.115 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:27.115 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.116 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.116 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.116 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13206 3.9s0 3:977e4b9d:::smithi10729407-12:head [append 0~532480 in=532480b] snapc 0=[] ondisk+write+known_if_redirected e57)", 2020-06-12T02:38:27.116 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:27.089456+0000", 2020-06-12T02:38:27.116 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.010247103, 2020-06-12T02:38:27.117 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010392208999999999, 2020-06-12T02:38:27.117 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:27.117 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:27.117 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:27.117 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:27.117 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:27.118 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13206 2020-06-12T02:38:27.118 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.118 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:27.118 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.118 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:27.119 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.089456+0000", 2020-06-12T02:38:27.119 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:27.119 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.119 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.119 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:27.119 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.089456+0000", 2020-06-12T02:38:27.120 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00059486900000000004 2020-06-12T02:38:27.120 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.120 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.120 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:27.120 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.090051+0000", 2020-06-12T02:38:27.121 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2939999999999996e-06 2020-06-12T02:38:27.121 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.121 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.121 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:27.121 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.090056+0000", 2020-06-12T02:38:27.122 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9090000000000006e-06 2020-06-12T02:38:27.122 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.123 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.123 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:27.123 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.090065+0000", 2020-06-12T02:38:27.123 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6600000000000003e-06 2020-06-12T02:38:27.123 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.123 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.124 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:27.124 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.090069+0000", 2020-06-12T02:38:27.124 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0013165270000000001 2020-06-12T02:38:27.124 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.124 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.125 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.125 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.091386+0000", 2020-06-12T02:38:27.125 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1088999999999996e-05 2020-06-12T02:38:27.125 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.125 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.126 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:27.126 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.091467+0000", 2020-06-12T02:38:27.126 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00046413599999999998 2020-06-12T02:38:27.126 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.126 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.126 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:27.127 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.091931+0000", 2020-06-12T02:38:27.127 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002475484 2020-06-12T02:38:27.127 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.127 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:27.127 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.128 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.128 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.128 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13207 3.9s0 3:977e4b9d:::smithi10729407-12:head [append 0~778240 in=778240b] snapc 0=[] ondisk+write+known_if_redirected e57)", 2020-06-12T02:38:27.128 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:27.094678+0000", 2020-06-12T02:38:27.128 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.0050245919999999996, 2020-06-12T02:38:27.128 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0052534269999999997, 2020-06-12T02:38:27.129 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:27.129 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:27.129 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:27.129 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:27.129 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:27.130 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13207 2020-06-12T02:38:27.130 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.130 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:27.130 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.130 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:27.130 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.094678+0000", 2020-06-12T02:38:27.131 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:27.131 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.131 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.131 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:27.131 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.094678+0000", 2020-06-12T02:38:27.132 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00072498099999999998 2020-06-12T02:38:27.132 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.140 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.140 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:27.140 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.095403+0000", 2020-06-12T02:38:27.140 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3790000000000006e-06 2020-06-12T02:38:27.141 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.141 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.141 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:27.141 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.095412+0000", 2020-06-12T02:38:27.141 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7393000000000001e-05 2020-06-12T02:38:27.142 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.142 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.142 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:27.142 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.095429+0000", 2020-06-12T02:38:27.142 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2199999999999998e-06 2020-06-12T02:38:27.143 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.143 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.143 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:27.143 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.095438+0000", 2020-06-12T02:38:27.143 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000321565 2020-06-12T02:38:27.143 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.144 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.144 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.144 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.095760+0000", 2020-06-12T02:38:27.144 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000116196 2020-06-12T02:38:27.144 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.145 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.145 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:27.145 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.095876+0000", 2020-06-12T02:38:27.145 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00077930600000000001 2020-06-12T02:38:27.145 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.145 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.146 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:27.146 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.096655+0000", 2020-06-12T02:38:27.146 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0019770399999999998 2020-06-12T02:38:27.146 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.146 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:27.147 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.147 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.147 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.148 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 57/37 ECSubWriteReply(tid=5439, last_complete=57'1110, committed=1, applied=1))", 2020-06-12T02:38:27.149 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:27.096054+0000", 2020-06-12T02:38:27.149 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.0036489840000000001, 2020-06-12T02:38:27.150 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0039519150000000003, 2020-06-12T02:38:27.150 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:27.150 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "queued for pg", 2020-06-12T02:38:27.150 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:27.150 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.151 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:27.151 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.096054+0000", 2020-06-12T02:38:27.151 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:27.151 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.151 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.151 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:27.152 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.096054+0000", 2020-06-12T02:38:27.152 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5731000000000001e-05 2020-06-12T02:38:27.152 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.152 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.152 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:27.153 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.096090+0000", 2020-06-12T02:38:27.153 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2609999999999996e-06 2020-06-12T02:38:27.153 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.153 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.153 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:27.153 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.096097+0000", 2020-06-12T02:38:27.154 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2031999999999999e-05 2020-06-12T02:38:27.154 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.154 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.154 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:27.154 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.096109+0000", 2020-06-12T02:38:27.155 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.799e-06 2020-06-12T02:38:27.155 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.155 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.155 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:27.155 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.096113+0000", 2020-06-12T02:38:27.156 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8823000000000002e-05 2020-06-12T02:38:27.156 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.156 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:27.156 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.156 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.156 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.157 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 57/37 ECSubWriteReply(tid=5440, last_complete=57'1111, committed=1, applied=1))", 2020-06-12T02:38:27.157 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:27.096662+0000", 2020-06-12T02:38:27.157 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.0030404550000000001, 2020-06-12T02:38:27.157 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033928690000000002, 2020-06-12T02:38:27.157 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:27.158 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "queued for pg", 2020-06-12T02:38:27.158 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:27.158 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.158 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:27.158 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.096662+0000", 2020-06-12T02:38:27.158 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:27.159 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.159 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.159 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:27.160 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.096662+0000", 2020-06-12T02:38:27.160 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1535e-05 2020-06-12T02:38:27.160 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.160 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.160 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:27.161 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.096724+0000", 2020-06-12T02:38:27.161 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1039999999999996e-06 2020-06-12T02:38:27.161 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.161 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.161 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:27.161 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.096733+0000", 2020-06-12T02:38:27.162 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5644000000000001e-05 2020-06-12T02:38:27.162 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.162 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.162 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:27.162 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.096749+0000", 2020-06-12T02:38:27.163 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0310000000000004e-06 2020-06-12T02:38:27.163 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.163 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.163 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:27.163 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.096755+0000", 2020-06-12T02:38:27.163 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2313999999999997e-05 2020-06-12T02:38:27.164 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.164 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:27.164 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.164 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.164 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.165 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13208 3.9s0 3.b9d27ee9 (undecoded) ondisk+write+known_if_redirected e57)", 2020-06-12T02:38:27.165 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:27.096411+0000", 2020-06-12T02:38:27.165 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.0032921360000000002, 2020-06-12T02:38:27.165 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036952589999999998, 2020-06-12T02:38:27.165 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:27.165 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "queued for pg", 2020-06-12T02:38:27.166 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:27.166 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:27.166 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:27.166 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13208 2020-06-12T02:38:27.166 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.167 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:27.167 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.167 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:27.167 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.096411+0000", 2020-06-12T02:38:27.167 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:27.168 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.168 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.168 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:27.168 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.096411+0000", 2020-06-12T02:38:27.168 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00066514999999999999 2020-06-12T02:38:27.168 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.169 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.169 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:27.169 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.097076+0000", 2020-06-12T02:38:27.169 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3159999999999997e-06 2020-06-12T02:38:27.170 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.170 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.170 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:27.170 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.097084+0000", 2020-06-12T02:38:27.170 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6847999999999999e-05 2020-06-12T02:38:27.171 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.171 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.171 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:27.171 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.097101+0000", 2020-06-12T02:38:27.171 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.801e-06 2020-06-12T02:38:27.172 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.172 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.172 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:27.172 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.097107+0000", 2020-06-12T02:38:27.172 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00069611499999999997 2020-06-12T02:38:27.173 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.173 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:27.173 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.173 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.173 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.174 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13209 3.9s0 3.b9d27ee9 (undecoded) ondisk+write+known_if_redirected e57)", 2020-06-12T02:38:27.174 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:27.097167+0000", 2020-06-12T02:38:27.174 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.0025359890000000002, 2020-06-12T02:38:27.174 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0030039979999999999, 2020-06-12T02:38:27.175 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:27.175 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "queued for pg", 2020-06-12T02:38:27.175 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:27.175 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:27.175 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:27.175 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13209 2020-06-12T02:38:27.176 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.176 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:27.176 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.176 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:27.176 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.097167+0000", 2020-06-12T02:38:27.177 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:27.177 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.177 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.177 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:27.177 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.097167+0000", 2020-06-12T02:38:27.177 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6098000000000002e-05 2020-06-12T02:38:27.178 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.178 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.178 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:27.178 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.097243+0000", 2020-06-12T02:38:27.178 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5390000000000002e-06 2020-06-12T02:38:27.179 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.179 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.179 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:27.179 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.097250+0000", 2020-06-12T02:38:27.179 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2687000000000001e-05 2020-06-12T02:38:27.180 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.180 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.180 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:27.180 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.097263+0000", 2020-06-12T02:38:27.181 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8000000000000004e-06 2020-06-12T02:38:27.181 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.181 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.181 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:27.181 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.097269+0000", 2020-06-12T02:38:27.181 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000102124 2020-06-12T02:38:27.182 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.182 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:27.182 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.182 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.182 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.183 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13210 3.9s0 3.b9d27ee9 (undecoded) ondisk+read+rwordered+known_if_redirected e57)", 2020-06-12T02:38:27.183 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:27.097314+0000", 2020-06-12T02:38:27.183 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.002388369, 2020-06-12T02:38:27.183 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0029193790000000002, 2020-06-12T02:38:27.183 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:27.183 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "queued for pg", 2020-06-12T02:38:27.184 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:27.184 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:27.184 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:27.184 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13210 2020-06-12T02:38:27.184 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.185 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:27.185 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.185 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:27.185 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.097314+0000", 2020-06-12T02:38:27.185 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:27.185 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.186 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.186 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:27.186 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.097314+0000", 2020-06-12T02:38:27.186 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6894999999999997e-05 2020-06-12T02:38:27.186 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.187 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.187 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:27.187 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.097361+0000", 2020-06-12T02:38:27.187 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3090000000000001e-06 2020-06-12T02:38:27.188 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.189 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.189 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:27.189 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.097368+0000", 2020-06-12T02:38:27.189 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7268000000000001e-05 2020-06-12T02:38:27.189 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.190 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.190 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:27.190 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.097385+0000", 2020-06-12T02:38:27.190 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.806e-06 2020-06-12T02:38:27.191 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.191 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.191 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:27.191 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.097391+0000", 2020-06-12T02:38:27.191 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6278000000000004e-05 2020-06-12T02:38:27.192 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.192 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:27.192 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.193 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.193 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.193 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.8s1 57/53 ECSubWrite(tid=6803, reqid=client.4512.0:13212, at_version=57'587, trim_to=0'0, roll_forward_to=57'586))", 2020-06-12T02:38:27.193 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:27.099715+0000", 2020-06-12T02:38:27.193 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4294967295.9999881, 2020-06-12T02:38:27.194 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00058157399999999996, 2020-06-12T02:38:27.194 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:27.194 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:38:27.194 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:27.194 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.194 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:27.195 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.099715+0000", 2020-06-12T02:38:27.195 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:27.195 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.195 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.195 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:27.196 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.099715+0000", 2020-06-12T02:38:27.196 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9431999999999999e-05 2020-06-12T02:38:27.196 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.196 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.196 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:27.196 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.099744+0000", 2020-06-12T02:38:27.197 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.279e-05 2020-06-12T02:38:27.197 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.197 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.197 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:27.197 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.099757+0000", 2020-06-12T02:38:27.198 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1099999999999992e-06 2020-06-12T02:38:27.198 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.198 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.198 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:27.198 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.099766+0000", 2020-06-12T02:38:27.198 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1200000000000004e-06 2020-06-12T02:38:27.199 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.199 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.199 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:27.199 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.099770+0000", 2020-06-12T02:38:27.199 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5937000000000003e-05 2020-06-12T02:38:27.200 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.200 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.201 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.202 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.099826+0000", 2020-06-12T02:38:27.202 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2031000000000001e-05 2020-06-12T02:38:27.202 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.202 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.202 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:27.203 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:27.099838+0000", 2020-06-12T02:38:27.203 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012342000000000001 2020-06-12T02:38:27.203 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.203 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:27.203 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.204 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.204 INFO:teuthology.orchestra.run.smithi188.stdout: ], 2020-06-12T02:38:27.204 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 9 2020-06-12T02:38:27.204 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:27.211 INFO:tasks.rados.rados.0.smithi107.stdout:6163: done (2 left) 2020-06-12T02:38:27.212 INFO:tasks.rados.rados.0.smithi107.stdout:6164: delete oid 5 current snap is 0 2020-06-12T02:38:27.212 INFO:tasks.rados.rados.0.smithi107.stdout:6161: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:27.212 INFO:tasks.rados.rados.0.smithi107.stdout:6161: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:27.212 INFO:tasks.rados.rados.0.smithi107.stdout:6164: done (2 left) 2020-06-12T02:38:27.213 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:27.213 INFO:tasks.rados.rados.0.smithi107.stdout:6165: seq_num 6164 ranges {0=40960} 2020-06-12T02:38:27.213 INFO:tasks.rados.rados.0.smithi107.stdout:6165: writing smithi10729407-13 from 0 to 40960 tid 1 2020-06-12T02:38:27.213 INFO:tasks.rados.rados.0.smithi107.stdout:6166: delete oid 4 current snap is 0 2020-06-12T02:38:27.213 INFO:tasks.rados.rados.0.smithi107.stdout:6161: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:27.214 INFO:tasks.rados.rados.0.smithi107.stdout:6161: finishing write tid 7 to smithi10729407-12 2020-06-12T02:38:27.214 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1114 (ObjNum 6160 snap 0 seq_num 6160) dirty exists 2020-06-12T02:38:27.214 INFO:tasks.rados.rados.0.smithi107.stdout:6161: left oid 12 (ObjNum 6160 snap 0 seq_num 6160) 2020-06-12T02:38:27.214 INFO:tasks.rados.rados.0.smithi107.stdout:6161: done (3 left) 2020-06-12T02:38:27.214 INFO:tasks.rados.rados.0.smithi107.stdout:6166: done (2 left) 2020-06-12T02:38:27.215 INFO:tasks.rados.rados.0.smithi107.stdout:6167: delete oid 3 current snap is 0 2020-06-12T02:38:27.215 INFO:tasks.rados.rados.0.smithi107.stdout:6165: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:27.215 INFO:tasks.rados.rados.0.smithi107.stdout:6167: done (2 left) 2020-06-12T02:38:27.215 INFO:tasks.rados.rados.0.smithi107.stdout:6168: delete oid 1 current snap is 0 2020-06-12T02:38:27.215 INFO:tasks.rados.rados.0.smithi107.stdout:6165: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:27.215 INFO:tasks.rados.rados.0.smithi107.stdout:6168: done (2 left) 2020-06-12T02:38:27.216 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:38:27.216 INFO:tasks.rados.rados.0.smithi107.stdout:6169: seq_num 6168 ranges {0=106496} 2020-06-12T02:38:27.216 INFO:tasks.rados.rados.0.smithi107.stdout:6169: writing smithi10729407-20 from 0 to 106496 tid 1 2020-06-12T02:38:27.216 INFO:tasks.rados.rados.0.smithi107.stdout:6170: delete oid 1 current snap is 0 2020-06-12T02:38:27.216 INFO:tasks.rados.rados.0.smithi107.stdout:6165: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:27.217 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1184 (ObjNum 6164 snap 0 seq_num 6164) dirty exists 2020-06-12T02:38:27.217 INFO:tasks.rados.rados.0.smithi107.stdout:6165: left oid 13 (ObjNum 6164 snap 0 seq_num 6164) 2020-06-12T02:38:27.217 INFO:tasks.rados.rados.0.smithi107.stdout:6165: done (3 left) 2020-06-12T02:38:27.217 INFO:tasks.rados.rados.0.smithi107.stdout:6170: done (2 left) 2020-06-12T02:38:27.217 INFO:tasks.rados.rados.0.smithi107.stdout:6171: delete oid 3 current snap is 0 2020-06-12T02:38:27.218 INFO:tasks.rados.rados.0.smithi107.stdout:6169: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:27.218 INFO:tasks.rados.rados.0.smithi107.stdout:6169: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:27.218 INFO:tasks.rados.rados.0.smithi107.stdout:6169: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:27.218 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 469 (ObjNum 6168 snap 0 seq_num 6168) dirty exists 2020-06-12T02:38:27.218 INFO:tasks.rados.rados.0.smithi107.stdout:6169: left oid 20 (ObjNum 6168 snap 0 seq_num 6168) 2020-06-12T02:38:27.218 INFO:tasks.rados.rados.0.smithi107.stdout:6169: done (2 left) 2020-06-12T02:38:27.219 INFO:tasks.rados.rados.0.smithi107.stdout:6171: done (1 left) 2020-06-12T02:38:27.219 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:38:27.219 INFO:tasks.rados.rados.0.smithi107.stdout:6172: seq_num 6171 ranges {0=155648} 2020-06-12T02:38:27.219 INFO:tasks.rados.rados.0.smithi107.stdout:6172: writing smithi10729407-3 from 0 to 155648 tid 1 2020-06-12T02:38:27.219 INFO:tasks.rados.rados.0.smithi107.stdout:6173: delete oid 19 current snap is 0 2020-06-12T02:38:27.220 INFO:tasks.rados.rados.0.smithi107.stdout:6173: done (2 left) 2020-06-12T02:38:27.220 INFO:tasks.rados.rados.0.smithi107.stdout:6174: delete oid 12 current snap is 0 2020-06-12T02:38:27.220 INFO:tasks.rados.rados.0.smithi107.stdout:6172: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:27.220 INFO:tasks.rados.rados.0.smithi107.stdout:6172: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:27.220 INFO:tasks.rados.rados.0.smithi107.stdout:6172: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:27.221 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1088 (ObjNum 6171 snap 0 seq_num 6171) dirty exists 2020-06-12T02:38:27.221 INFO:tasks.rados.rados.0.smithi107.stdout:6172: left oid 3 (ObjNum 6171 snap 0 seq_num 6171) 2020-06-12T02:38:27.221 INFO:tasks.rados.rados.0.smithi107.stdout:6172: done (2 left) 2020-06-12T02:38:27.221 INFO:tasks.rados.rados.0.smithi107.stdout:6174: done (1 left) 2020-06-12T02:38:27.221 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:27.221 INFO:tasks.rados.rados.0.smithi107.stdout:6175: seq_num 6174 ranges {0=204800} 2020-06-12T02:38:27.222 INFO:tasks.rados.rados.0.smithi107.stdout:6175: writing smithi10729407-12 from 0 to 204800 tid 1 2020-06-12T02:38:27.222 INFO:tasks.rados.rados.0.smithi107.stdout:6176: delete oid 1 current snap is 0 2020-06-12T02:38:27.222 INFO:tasks.rados.rados.0.smithi107.stdout:6176: done (2 left) 2020-06-12T02:38:27.222 INFO:tasks.rados.rados.0.smithi107.stdout:6177: delete oid 3 current snap is 0 2020-06-12T02:38:27.223 INFO:tasks.rados.rados.0.smithi107.stdout:6175: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:27.223 INFO:tasks.rados.rados.0.smithi107.stdout:6177: done (2 left) 2020-06-12T02:38:27.223 INFO:tasks.rados.rados.0.smithi107.stdout:6178: delete oid 14 current snap is 0 2020-06-12T02:38:27.223 INFO:tasks.rados.rados.0.smithi107.stdout:6175: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:27.223 INFO:tasks.rados.rados.0.smithi107.stdout:6178: done (2 left) 2020-06-12T02:38:27.226 INFO:tasks.rados.rados.0.smithi107.stdout:6179: delete oid 5 current snap is 0 2020-06-12T02:38:27.226 INFO:tasks.rados.rados.0.smithi107.stdout:6175: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:27.226 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1118 (ObjNum 6174 snap 0 seq_num 6174) dirty exists 2020-06-12T02:38:27.227 INFO:tasks.rados.rados.0.smithi107.stdout:6175: left oid 12 (ObjNum 6174 snap 0 seq_num 6174) 2020-06-12T02:38:27.227 INFO:tasks.rados.rados.0.smithi107.stdout:6175: done (2 left) 2020-06-12T02:38:27.227 INFO:tasks.rados.rados.0.smithi107.stdout:6179: done (1 left) 2020-06-12T02:38:27.227 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:38:27.227 INFO:tasks.rados.rados.0.smithi107.stdout:6180: seq_num 6179 ranges {106496=294912} 2020-06-12T02:38:27.228 INFO:tasks.rados.rados.0.smithi107.stdout:6180: writing smithi10729407-20 from 106496 to 401408 tid 1 2020-06-12T02:38:27.228 INFO:tasks.rados.rados.0.smithi107.stdout:6181: delete oid 9 current snap is 0 2020-06-12T02:38:27.228 INFO:tasks.rados.rados.0.smithi107.stdout:6181: done (2 left) 2020-06-12T02:38:27.228 INFO:tasks.rados.rados.0.smithi107.stdout:6182: delete oid 14 current snap is 0 2020-06-12T02:38:27.228 INFO:tasks.rados.rados.0.smithi107.stdout:6180: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:27.228 INFO:tasks.rados.rados.0.smithi107.stdout:6182: done (2 left) 2020-06-12T02:38:27.229 INFO:tasks.rados.rados.0.smithi107.stdout:6183: delete oid 14 current snap is 0 2020-06-12T02:38:27.229 INFO:tasks.rados.rados.0.smithi107.stdout:6180: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:27.229 INFO:tasks.rados.rados.0.smithi107.stdout:6183: done (2 left) 2020-06-12T02:38:27.229 INFO:tasks.rados.rados.0.smithi107.stdout:6184: delete oid 19 current snap is 0 2020-06-12T02:38:27.229 INFO:tasks.rados.rados.0.smithi107.stdout:6180: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:27.230 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 471 (ObjNum 6179 snap 0 seq_num 6179) dirty exists 2020-06-12T02:38:27.230 INFO:tasks.rados.rados.0.smithi107.stdout:6180: left oid 20 (ObjNum 6179 snap 0 seq_num 6179) 2020-06-12T02:38:27.230 INFO:tasks.rados.rados.0.smithi107.stdout:6180: done (2 left) 2020-06-12T02:38:27.230 INFO:tasks.rados.rados.0.smithi107.stdout:6184: done (1 left) 2020-06-12T02:38:27.230 INFO:tasks.rados.rados.0.smithi107.stdout:6185: delete oid 16 current snap is 0 2020-06-12T02:38:27.231 INFO:tasks.rados.rados.0.smithi107.stdout:6185: done (1 left) 2020-06-12T02:38:27.231 INFO:tasks.rados.rados.0.smithi107.stdout:6186: delete oid 9 current snap is 0 2020-06-12T02:38:27.231 INFO:tasks.rados.rados.0.smithi107.stdout:6186: done (1 left) 2020-06-12T02:38:27.231 INFO:tasks.rados.rados.0.smithi107.stdout:6187: delete oid 11 current snap is 0 2020-06-12T02:38:27.231 INFO:tasks.rados.rados.0.smithi107.stdout:6187: done (1 left) 2020-06-12T02:38:27.231 INFO:tasks.rados.rados.0.smithi107.stdout:6188: delete oid 17 current snap is 0 2020-06-12T02:38:27.232 INFO:tasks.rados.rados.0.smithi107.stdout:6188: done (1 left) 2020-06-12T02:38:27.232 INFO:tasks.rados.rados.0.smithi107.stdout:6189: delete oid 1 current snap is 0 2020-06-12T02:38:27.232 INFO:tasks.rados.rados.0.smithi107.stdout:6189: done (1 left) 2020-06-12T02:38:27.232 INFO:tasks.rados.rados.0.smithi107.stdout:6190: delete oid 2 current snap is 0 2020-06-12T02:38:27.232 INFO:tasks.rados.rados.0.smithi107.stdout:6190: done (1 left) 2020-06-12T02:38:27.233 INFO:tasks.rados.rados.0.smithi107.stdout:6191: delete oid 15 current snap is 0 2020-06-12T02:38:27.233 INFO:tasks.rados.rados.0.smithi107.stdout:6191: done (1 left) 2020-06-12T02:38:27.233 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:27.233 INFO:tasks.rados.rados.0.smithi107.stdout:6192: seq_num 6191 ranges {0=491520} 2020-06-12T02:38:27.234 INFO:tasks.rados.rados.0.smithi107.stdout:6192: writing smithi10729407-19 from 0 to 491520 tid 1 2020-06-12T02:38:27.234 INFO:tasks.rados.rados.0.smithi107.stdout:6193: delete oid 17 current snap is 0 2020-06-12T02:38:27.234 INFO:tasks.rados.rados.0.smithi107.stdout:6193: done (2 left) 2020-06-12T02:38:27.235 INFO:tasks.rados.rados.0.smithi107.stdout:6194: delete oid 3 current snap is 0 2020-06-12T02:38:27.235 INFO:tasks.rados.rados.0.smithi107.stdout:6192: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:27.235 INFO:tasks.rados.rados.0.smithi107.stdout:6192: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:27.235 INFO:tasks.rados.rados.0.smithi107.stdout:6194: done (2 left) 2020-06-12T02:38:27.235 INFO:tasks.rados.rados.0.smithi107.stdout:6195: delete oid 7 current snap is 0 2020-06-12T02:38:27.236 INFO:tasks.rados.rados.0.smithi107.stdout:6192: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:27.236 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1121 (ObjNum 6191 snap 0 seq_num 6191) dirty exists 2020-06-12T02:38:27.236 INFO:tasks.rados.rados.0.smithi107.stdout:6192: left oid 19 (ObjNum 6191 snap 0 seq_num 6191) 2020-06-12T02:38:27.236 INFO:tasks.rados.rados.0.smithi107.stdout:6192: done (2 left) 2020-06-12T02:38:27.237 INFO:tasks.rados.rados.0.smithi107.stdout:6195: done (1 left) 2020-06-12T02:38:27.237 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:38:27.237 INFO:tasks.rados.rados.0.smithi107.stdout:6196: seq_num 6195 ranges {0=557056} 2020-06-12T02:38:27.237 INFO:tasks.rados.rados.0.smithi107.stdout:6196: writing smithi10729407-1 from 0 to 557056 tid 1 2020-06-12T02:38:27.237 INFO:tasks.rados.rados.0.smithi107.stdout:6197: delete oid 3 current snap is 0 2020-06-12T02:38:27.238 INFO:tasks.rados.rados.0.smithi107.stdout:6197: done (2 left) 2020-06-12T02:38:27.238 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:27.238 INFO:tasks.rados.rados.0.smithi107.stdout:6198: seq_num 6197 ranges {0=589824} 2020-06-12T02:38:27.238 INFO:tasks.rados.rados.0.smithi107.stdout:6198: writing smithi10729407-18 from 0 to 589824 tid 1 2020-06-12T02:38:27.239 INFO:tasks.rados.rados.0.smithi107.stdout:6199: delete oid 3 current snap is 0 2020-06-12T02:38:27.239 INFO:tasks.rados.rados.0.smithi107.stdout:6196: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:27.239 INFO:tasks.rados.rados.0.smithi107.stdout:6196: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:27.239 INFO:tasks.rados.rados.0.smithi107.stdout:6196: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:27.239 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1144 (ObjNum 6195 snap 0 seq_num 6195) dirty exists 2020-06-12T02:38:27.240 INFO:tasks.rados.rados.0.smithi107.stdout:6196: left oid 1 (ObjNum 6195 snap 0 seq_num 6195) 2020-06-12T02:38:27.240 INFO:tasks.rados.rados.0.smithi107.stdout:6196: done (3 left) 2020-06-12T02:38:27.240 INFO:tasks.rados.rados.0.smithi107.stdout:6199: done (2 left) 2020-06-12T02:38:27.240 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:27.240 INFO:tasks.rados.rados.0.smithi107.stdout:6200: seq_num 6199 ranges {0=630784} 2020-06-12T02:38:27.242 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-06-12T02:38:27.243 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:38:27.203+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:38:27.245 INFO:tasks.rados.rados.0.smithi107.stdout:6200: writing smithi10729407-3 from 0 to 630784 tid 1 2020-06-12T02:38:27.245 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:38:27.245 INFO:tasks.rados.rados.0.smithi107.stdout:6201: seq_num 6200 ranges {401408=647168} 2020-06-12T02:38:27.245 INFO:tasks.rados.rados.0.smithi107.stdout:6201: writing smithi10729407-20 from 401408 to 1048576 tid 1 2020-06-12T02:38:27.246 INFO:tasks.rados.rados.0.smithi107.stdout:6202: delete oid 6 current snap is 0 2020-06-12T02:38:27.246 INFO:tasks.rados.rados.0.smithi107.stdout:6198: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:27.246 INFO:tasks.rados.rados.0.smithi107.stdout:6198: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:27.246 INFO:tasks.rados.rados.0.smithi107.stdout:6198: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:27.247 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1124 (ObjNum 6197 snap 0 seq_num 6197) dirty exists 2020-06-12T02:38:27.247 INFO:tasks.rados.rados.0.smithi107.stdout:6198: left oid 18 (ObjNum 6197 snap 0 seq_num 6197) 2020-06-12T02:38:27.247 INFO:tasks.rados.rados.0.smithi107.stdout:6200: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:27.247 INFO:tasks.rados.rados.0.smithi107.stdout:6198: done (4 left) 2020-06-12T02:38:27.247 INFO:tasks.rados.rados.0.smithi107.stdout:6202: done (3 left) 2020-06-12T02:38:27.248 INFO:tasks.rados.rados.0.smithi107.stdout:6203: delete oid 12 current snap is 0 2020-06-12T02:38:27.248 INFO:tasks.rados.rados.0.smithi107.stdout:6203: done (3 left) 2020-06-12T02:38:27.248 INFO:tasks.rados.rados.0.smithi107.stdout:6204: delete oid 15 current snap is 0 2020-06-12T02:38:27.248 INFO:tasks.rados.rados.0.smithi107.stdout:6200: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:27.248 INFO:tasks.rados.rados.0.smithi107.stdout:6200: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:27.249 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1095 (ObjNum 6199 snap 0 seq_num 6199) dirty exists 2020-06-12T02:38:27.249 INFO:tasks.rados.rados.0.smithi107.stdout:6200: left oid 3 (ObjNum 6199 snap 0 seq_num 6199) 2020-06-12T02:38:27.249 INFO:tasks.rados.rados.0.smithi107.stdout:6201: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:27.249 INFO:tasks.rados.rados.0.smithi107.stdout:6201: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:27.249 INFO:tasks.rados.rados.0.smithi107.stdout:6200: done (3 left) 2020-06-12T02:38:27.250 INFO:tasks.rados.rados.0.smithi107.stdout:6204: done (2 left) 2020-06-12T02:38:27.250 INFO:tasks.rados.rados.0.smithi107.stdout:6205: delete oid 17 current snap is 0 2020-06-12T02:38:27.250 INFO:tasks.rados.rados.0.smithi107.stdout:6201: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:27.250 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 473 (ObjNum 6200 snap 0 seq_num 6200) dirty exists 2020-06-12T02:38:27.250 INFO:tasks.rados.rados.0.smithi107.stdout:6201: left oid 20 (ObjNum 6200 snap 0 seq_num 6200) 2020-06-12T02:38:27.250 INFO:tasks.rados.rados.0.smithi107.stdout:6201: done (2 left) 2020-06-12T02:38:27.251 INFO:tasks.rados.rados.0.smithi107.stdout:6205: done (1 left) 2020-06-12T02:38:27.251 INFO:tasks.rados.rados.0.smithi107.stdout:6206: delete oid 4 current snap is 0 2020-06-12T02:38:27.251 INFO:tasks.rados.rados.0.smithi107.stdout:6206: done (1 left) 2020-06-12T02:38:27.251 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:38:27.251 INFO:tasks.rados.rados.0.smithi107.stdout:6207: seq_num 6206 ranges {0=745472} 2020-06-12T02:38:27.252 INFO:tasks.rados.rados.0.smithi107.stdout:6207: writing smithi10729407-6 from 0 to 745472 tid 1 2020-06-12T02:38:27.252 INFO:tasks.rados.rados.0.smithi107.stdout:6208: delete oid 17 current snap is 0 2020-06-12T02:38:27.252 INFO:tasks.rados.rados.0.smithi107.stdout:6208: done (2 left) 2020-06-12T02:38:27.252 INFO:tasks.rados.rados.0.smithi107.stdout:6209: delete oid 17 current snap is 0 2020-06-12T02:38:27.252 INFO:tasks.rados.rados.0.smithi107.stdout:6209: done (2 left) 2020-06-12T02:38:27.253 INFO:tasks.rados.rados.0.smithi107.stdout:6210: delete oid 2 current snap is 0 2020-06-12T02:38:27.254 INFO:tasks.rados.rados.0.smithi107.stdout:6207: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:27.255 INFO:tasks.rados.rados.0.smithi107.stdout:6207: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:27.255 INFO:tasks.rados.rados.0.smithi107.stdout:6207: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:27.256 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1147 (ObjNum 6206 snap 0 seq_num 6206) dirty exists 2020-06-12T02:38:27.256 INFO:tasks.rados.rados.0.smithi107.stdout:6207: left oid 6 (ObjNum 6206 snap 0 seq_num 6206) 2020-06-12T02:38:27.256 INFO:tasks.rados.rados.0.smithi107.stdout:6207: done (2 left) 2020-06-12T02:38:27.256 INFO:tasks.rados.rados.0.smithi107.stdout:6210: done (1 left) 2020-06-12T02:38:27.256 INFO:tasks.rados.rados.0.smithi107.stdout:6211: delete oid 20 current snap is 0 2020-06-12T02:38:27.256 INFO:tasks.rados.rados.0.smithi107.stdout:6211: done (1 left) 2020-06-12T02:38:27.257 INFO:tasks.rados.rados.0.smithi107.stdout:6212: delete oid 5 current snap is 0 2020-06-12T02:38:27.257 INFO:tasks.rados.rados.0.smithi107.stdout:6212: done (1 left) 2020-06-12T02:38:27.257 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:27.257 INFO:tasks.rados.rados.0.smithi107.stdout:6213: seq_num 6212 ranges {0=442368,442368=401408} 2020-06-12T02:38:27.257 INFO:tasks.rados.rados.0.smithi107.stdout:6213: writing smithi10729407-16 from 0 to 442368 tid 1 2020-06-12T02:38:27.259 INFO:tasks.rados.rados.0.smithi107.stdout:6213: writing smithi10729407-16 from 442368 to 843776 tid 2 2020-06-12T02:38:27.259 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:38:27.259 INFO:tasks.rados.rados.0.smithi107.stdout:6214: seq_num 6213 ranges {0=458752,458752=401408} 2020-06-12T02:38:27.259 INFO:tasks.rados.rados.0.smithi107.stdout:6214: writing smithi10729407-5 from 0 to 458752 tid 1 2020-06-12T02:38:27.261 INFO:tasks.rados.rados.0.smithi107.stdout:6214: writing smithi10729407-5 from 458752 to 860160 tid 2 2020-06-12T02:38:27.261 INFO:tasks.rados.rados.0.smithi107.stdout:6215: delete oid 6 current snap is 0 2020-06-12T02:38:27.261 INFO:tasks.rados.rados.0.smithi107.stdout:6213: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:27.263 INFO:tasks.rados.rados.0.smithi107.stdout:6213: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:27.264 INFO:tasks.rados.rados.0.smithi107.stdout:6213: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:27.266 INFO:tasks.rados.rados.0.smithi107.stdout:6214: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:27.266 INFO:tasks.rados.rados.0.smithi107.stdout:6215: done (3 left) 2020-06-12T02:38:27.267 INFO:tasks.rados.rados.0.smithi107.stdout:6216: delete oid 7 current snap is 0 2020-06-12T02:38:27.267 INFO:tasks.rados.rados.0.smithi107.stdout:6213: finishing write tid 5 to smithi10729407-16 2020-06-12T02:38:27.267 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1098 (ObjNum 6212 snap 0 seq_num 6212) dirty exists 2020-06-12T02:38:27.267 INFO:tasks.rados.rados.0.smithi107.stdout:6213: left oid 16 (ObjNum 6212 snap 0 seq_num 6212) 2020-06-12T02:38:27.275 INFO:tasks.rados.rados.0.smithi107.stdout:6213: done (3 left) 2020-06-12T02:38:27.275 INFO:tasks.rados.rados.0.smithi107.stdout:6216: done (2 left) 2020-06-12T02:38:27.275 INFO:tasks.rados.rados.0.smithi107.stdout:6217: delete oid 7 current snap is 0 2020-06-12T02:38:27.276 INFO:tasks.rados.rados.0.smithi107.stdout:6214: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:27.276 INFO:tasks.rados.rados.0.smithi107.stdout:6214: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:27.276 INFO:tasks.rados.rados.0.smithi107.stdout:6214: finishing write tid 5 to smithi10729407-5 2020-06-12T02:38:27.277 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 622 (ObjNum 6213 snap 0 seq_num 6213) dirty exists 2020-06-12T02:38:27.277 INFO:tasks.rados.rados.0.smithi107.stdout:6214: left oid 5 (ObjNum 6213 snap 0 seq_num 6213) 2020-06-12T02:38:27.289 INFO:tasks.rados.rados.0.smithi107.stdout:6214: done (2 left) 2020-06-12T02:38:27.289 INFO:tasks.rados.rados.0.smithi107.stdout:6217: done (1 left) 2020-06-12T02:38:27.290 INFO:tasks.rados.rados.0.smithi107.stdout:6218: delete oid 3 current snap is 0 2020-06-12T02:38:27.290 INFO:tasks.rados.rados.0.smithi107.stdout:6218: done (1 left) 2020-06-12T02:38:27.290 INFO:tasks.rados.rados.0.smithi107.stdout:6219: delete oid 14 current snap is 0 2020-06-12T02:38:27.290 INFO:tasks.rados.rados.0.smithi107.stdout:6219: done (1 left) 2020-06-12T02:38:27.290 INFO:tasks.rados.rados.0.smithi107.stdout:6220: delete oid 11 current snap is 0 2020-06-12T02:38:27.290 INFO:tasks.rados.rados.0.smithi107.stdout:6220: done (1 left) 2020-06-12T02:38:27.291 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:27.291 INFO:tasks.rados.rados.0.smithi107.stdout:6221: seq_num 6220 ranges {0=581632,581632=401408} 2020-06-12T02:38:27.292 INFO:tasks.rados.rados.0.smithi107.stdout:6221: writing smithi10729407-2 from 0 to 581632 tid 1 2020-06-12T02:38:27.292 INFO:tasks.rados.rados.0.smithi107.stdout:6221: writing smithi10729407-2 from 581632 to 983040 tid 2 2020-06-12T02:38:27.292 INFO:tasks.rados.rados.0.smithi107.stdout:6222: delete oid 19 current snap is 0 2020-06-12T02:38:27.294 INFO:tasks.rados.rados.0.smithi107.stdout:6222: done (2 left) 2020-06-12T02:38:27.294 INFO:tasks.rados.rados.0.smithi107.stdout:6223: delete oid 6 current snap is 0 2020-06-12T02:38:27.297 INFO:tasks.rados.rados.0.smithi107.stdout:6221: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:27.300 INFO:tasks.rados.rados.0.smithi107.stdout:6221: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:27.300 INFO:tasks.rados.rados.0.smithi107.stdout:6221: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:27.301 INFO:tasks.rados.rados.0.smithi107.stdout:6221: finishing write tid 5 to smithi10729407-2 2020-06-12T02:38:27.301 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 592 (ObjNum 6220 snap 0 seq_num 6220) dirty exists 2020-06-12T02:38:27.302 INFO:tasks.rados.rados.0.smithi107.stdout:6221: left oid 2 (ObjNum 6220 snap 0 seq_num 6220) 2020-06-12T02:38:27.302 INFO:tasks.rados.rados.0.smithi107.stdout:6221: done (2 left) 2020-06-12T02:38:27.302 INFO:tasks.rados.rados.0.smithi107.stdout:6223: done (1 left) 2020-06-12T02:38:27.302 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:27.302 INFO:tasks.rados.rados.0.smithi107.stdout:6224: seq_num 6223 ranges {557056=630784,1187840=401408} 2020-06-12T02:38:27.307 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:27.305+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:27.307 INFO:tasks.rados.rados.0.smithi107.stdout:6224: writing smithi10729407-1 from 557056 to 1187840 tid 1 2020-06-12T02:38:27.309 INFO:tasks.rados.rados.0.smithi107.stdout:6224: writing smithi10729407-1 from 1187840 to 1589248 tid 2 2020-06-12T02:38:27.310 INFO:tasks.rados.rados.0.smithi107.stdout:6225: delete oid 5 current snap is 0 2020-06-12T02:38:27.313 INFO:tasks.rados.rados.0.smithi107.stdout:6225: done (2 left) 2020-06-12T02:38:27.314 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:38:27.314 INFO:tasks.rados.rados.0.smithi107.stdout:6226: seq_num 6225 ranges {0=663552,663552=401408} 2020-06-12T02:38:27.317 INFO:tasks.rados.rados.0.smithi107.stdout:6226: writing smithi10729407-3 from 0 to 663552 tid 1 2020-06-12T02:38:27.320 INFO:tasks.rados.rados.0.smithi107.stdout:6226: writing smithi10729407-3 from 663552 to 1064960 tid 2 2020-06-12T02:38:27.321 INFO:tasks.rados.rados.0.smithi107.stdout:6227: delete oid 20 current snap is 0 2020-06-12T02:38:27.321 INFO:tasks.rados.rados.0.smithi107.stdout:6224: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:27.322 INFO:tasks.rados.rados.0.smithi107.stdout:6224: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:27.323 INFO:tasks.rados.rados.0.smithi107.stdout:6224: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:27.324 INFO:tasks.rados.rados.0.smithi107.stdout:6224: finishing write tid 5 to smithi10729407-1 2020-06-12T02:38:27.325 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1152 (ObjNum 6223 snap 0 seq_num 6223) dirty exists 2020-06-12T02:38:27.325 INFO:tasks.rados.rados.0.smithi107.stdout:6224: left oid 1 (ObjNum 6223 snap 0 seq_num 6223) 2020-06-12T02:38:27.327 INFO:tasks.rados.rados.0.smithi107.stdout:6224: done (3 left) 2020-06-12T02:38:27.327 INFO:tasks.rados.rados.0.smithi107.stdout:6227: done (2 left) 2020-06-12T02:38:27.327 INFO:tasks.rados.rados.0.smithi107.stdout:6228: delete oid 1 current snap is 0 2020-06-12T02:38:27.328 INFO:tasks.rados.rados.0.smithi107.stdout:6226: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:27.329 INFO:tasks.rados.rados.0.smithi107.stdout:6226: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:27.329 INFO:tasks.rados.rados.0.smithi107.stdout:6226: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:27.331 INFO:tasks.rados.rados.0.smithi107.stdout:6226: finishing write tid 5 to smithi10729407-3 2020-06-12T02:38:27.331 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1102 (ObjNum 6225 snap 0 seq_num 6225) dirty exists 2020-06-12T02:38:27.331 INFO:tasks.rados.rados.0.smithi107.stdout:6226: left oid 3 (ObjNum 6225 snap 0 seq_num 6225) 2020-06-12T02:38:27.331 INFO:tasks.rados.rados.0.smithi107.stdout:6226: done (2 left) 2020-06-12T02:38:27.332 INFO:tasks.rados.rados.0.smithi107.stdout:6228: done (1 left) 2020-06-12T02:38:27.332 INFO:tasks.rados.rados.0.smithi107.stdout:6229: delete oid 2 current snap is 0 2020-06-12T02:38:27.334 INFO:tasks.rados.rados.0.smithi107.stdout:6229: done (1 left) 2020-06-12T02:38:27.334 INFO:tasks.rados.rados.0.smithi107.stdout:6230: delete oid 19 current snap is 0 2020-06-12T02:38:27.337 INFO:tasks.rados.rados.0.smithi107.stdout:6230: done (1 left) 2020-06-12T02:38:27.337 INFO:tasks.rados.rados.0.smithi107.stdout:6231: delete oid 15 current snap is 0 2020-06-12T02:38:27.339 INFO:tasks.rados.rados.0.smithi107.stdout:6231: done (1 left) 2020-06-12T02:38:27.339 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:27.340 INFO:tasks.rados.rados.0.smithi107.stdout:6232: seq_num 6231 ranges {0=761856,761856=401408} 2020-06-12T02:38:27.343 INFO:tasks.rados.rados.0.smithi107.stdout:6232: writing smithi10729407-9 from 0 to 761856 tid 1 2020-06-12T02:38:27.346 INFO:tasks.rados.rados.0.smithi107.stdout:6232: writing smithi10729407-9 from 761856 to 1163264 tid 2 2020-06-12T02:38:27.346 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:27.346 INFO:tasks.rados.rados.0.smithi107.stdout:6233: seq_num 6232 ranges {0=778240,778240=401408} 2020-06-12T02:38:27.351 INFO:tasks.rados.rados.0.smithi107.stdout:6233: writing smithi10729407-4 from 0 to 778240 tid 1 2020-06-12T02:38:27.353 INFO:tasks.rados.rados.0.smithi107.stdout:6233: writing smithi10729407-4 from 778240 to 1179648 tid 2 2020-06-12T02:38:27.354 INFO:tasks.rados.rados.0.smithi107.stdout:6234: delete oid 15 current snap is 0 2020-06-12T02:38:27.354 INFO:tasks.rados.rados.0.smithi107.stdout:6232: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:27.357 INFO:tasks.rados.rados.0.smithi107.stdout:6234: done (3 left) 2020-06-12T02:38:27.357 INFO:tasks.rados.rados.0.smithi107.stdout:6235: delete oid 15 current snap is 0 2020-06-12T02:38:27.358 INFO:tasks.rados.rados.0.smithi107.stdout:6232: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:27.359 INFO:tasks.rados.rados.0.smithi107.stdout:6232: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:27.360 INFO:tasks.rados.rados.0.smithi107.stdout:6233: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:27.360 INFO:tasks.rados.rados.0.smithi107.stdout:6232: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:27.361 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1127 (ObjNum 6231 snap 0 seq_num 6231) dirty exists 2020-06-12T02:38:27.361 INFO:tasks.rados.rados.0.smithi107.stdout:6232: left oid 9 (ObjNum 6231 snap 0 seq_num 6231) 2020-06-12T02:38:27.363 INFO:tasks.rados.rados.0.smithi107.stdout:6232: done (3 left) 2020-06-12T02:38:27.363 INFO:tasks.rados.rados.0.smithi107.stdout:6235: done (2 left) 2020-06-12T02:38:27.364 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:27.364 INFO:tasks.rados.rados.0.smithi107.stdout:6236: seq_num 6235 ranges {0=425984,425984=753664,1179648=49152} 2020-06-12T02:38:27.364 INFO:tasks.rados.rados.0.smithi107.stdout:6236: writing smithi10729407-2 from 0 to 425984 tid 1 2020-06-12T02:38:27.367 INFO:tasks.rados.rados.0.smithi107.stdout:6236: writing smithi10729407-2 from 425984 to 1179648 tid 2 2020-06-12T02:38:27.369 INFO:tasks.rados.rados.0.smithi107.stdout:6236: writing smithi10729407-2 from 1179648 to 1228800 tid 3 2020-06-12T02:38:27.369 INFO:tasks.rados.rados.0.smithi107.stdout:6237: delete oid 11 current snap is 0 2020-06-12T02:38:27.369 INFO:tasks.rados.rados.0.smithi107.stdout:6233: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:27.369 INFO:tasks.rados.rados.0.smithi107.stdout:6233: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:27.370 INFO:tasks.rados.rados.0.smithi107.stdout:6236: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:27.370 INFO:tasks.rados.rados.0.smithi107.stdout:6233: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:27.371 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1166 (ObjNum 6232 snap 0 seq_num 6232) dirty exists 2020-06-12T02:38:27.371 INFO:tasks.rados.rados.0.smithi107.stdout:6233: left oid 4 (ObjNum 6232 snap 0 seq_num 6232) 2020-06-12T02:38:27.375 INFO:tasks.rados.rados.0.smithi107.stdout:6233: done (3 left) 2020-06-12T02:38:27.375 INFO:tasks.rados.rados.0.smithi107.stdout:6237: done (2 left) 2020-06-12T02:38:27.376 INFO:tasks.rados.rados.0.smithi107.stdout:6238: delete oid 1 current snap is 0 2020-06-12T02:38:27.376 INFO:tasks.rados.rados.0.smithi107.stdout:6236: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:27.377 INFO:tasks.rados.rados.0.smithi107.stdout:6236: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:27.377 INFO:tasks.rados.rados.0.smithi107.stdout:6236: finishing write tid 5 to smithi10729407-2 2020-06-12T02:38:27.378 INFO:tasks.rados.rados.0.smithi107.stdout:6236: finishing write tid 6 to smithi10729407-2 2020-06-12T02:38:27.378 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 597 (ObjNum 6235 snap 0 seq_num 6235) dirty exists 2020-06-12T02:38:27.379 INFO:tasks.rados.rados.0.smithi107.stdout:6236: left oid 2 (ObjNum 6235 snap 0 seq_num 6235) 2020-06-12T02:38:27.379 INFO:tasks.rados.rados.0.smithi107.stdout:6236: done (2 left) 2020-06-12T02:38:27.380 INFO:tasks.rados.rados.0.smithi107.stdout:6238: done (1 left) 2020-06-12T02:38:27.380 INFO:tasks.rados.rados.0.smithi107.stdout:6239: delete oid 6 current snap is 0 2020-06-12T02:38:27.384 INFO:tasks.rados.rados.0.smithi107.stdout:6239: done (1 left) 2020-06-12T02:38:27.385 INFO:tasks.rados.rados.0.smithi107.stdout:6240: delete oid 6 current snap is 0 2020-06-12T02:38:27.388 INFO:tasks.rados.rados.0.smithi107.stdout:6240: done (1 left) 2020-06-12T02:38:27.388 INFO:tasks.rados.rados.0.smithi107.stdout:6241: delete oid 2 current snap is 0 2020-06-12T02:38:27.391 INFO:tasks.rados.rados.0.smithi107.stdout:6241: done (1 left) 2020-06-12T02:38:27.391 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:38:27.394 INFO:tasks.rados.rados.0.smithi107.stdout:6242: seq_num 6241 ranges {843776=532480,1376256=458752,1835008=344064} 2020-06-12T02:38:27.397 INFO:tasks.rados.rados.0.smithi107.stdout:6242: writing smithi10729407-16 from 843776 to 1376256 tid 1 2020-06-12T02:38:27.403 INFO:tasks.rados.rados.0.smithi107.stdout:6242: writing smithi10729407-16 from 1376256 to 1835008 tid 2 2020-06-12T02:38:27.404 INFO:tasks.rados.rados.0.smithi107.stdout:6242: writing smithi10729407-16 from 1835008 to 2179072 tid 3 2020-06-12T02:38:27.404 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:27.405 INFO:tasks.rados.rados.0.smithi107.stdout:6243: seq_num 6242 ranges {0=548864,548864=794624,1343488=8192} 2020-06-12T02:38:27.408 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:27.409 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:38:27.409 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:38:27.409 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:38:27.410 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:27.414 INFO:tasks.rados.rados.0.smithi107.stdout:6243: writing smithi10729407-5 from 0 to 548864 tid 1 2020-06-12T02:38:27.414 INFO:tasks.rados.rados.0.smithi107.stdout:6243: writing smithi10729407-5 from 548864 to 1343488 tid 2 2020-06-12T02:38:27.414 INFO:tasks.rados.rados.0.smithi107.stdout:6243: writing smithi10729407-5 from 1343488 to 1351680 tid 3 2020-06-12T02:38:27.415 INFO:tasks.rados.rados.0.smithi107.stdout:6244: delete oid 7 current snap is 0 2020-06-12T02:38:27.415 INFO:tasks.rados.rados.0.smithi107.stdout:6242: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:27.417 INFO:tasks.rados.rados.0.smithi107.stdout:6242: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:27.418 INFO:tasks.rados.rados.0.smithi107.stdout:6242: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:27.418 INFO:tasks.rados.rados.0.smithi107.stdout:6242: finishing write tid 5 to smithi10729407-16 2020-06-12T02:38:27.419 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:27.408+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:27.423 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-06-12T02:38:27.424 INFO:tasks.rados.rados.0.smithi107.stdout:6242: finishing write tid 6 to smithi10729407-16 2020-06-12T02:38:27.424 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1106 (ObjNum 6241 snap 0 seq_num 6241) dirty exists 2020-06-12T02:38:27.425 INFO:tasks.rados.rados.0.smithi107.stdout:6242: left oid 16 (ObjNum 6241 snap 0 seq_num 6241) 2020-06-12T02:38:27.425 INFO:tasks.rados.rados.0.smithi107.stdout:6243: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:27.425 INFO:tasks.rados.rados.0.smithi107.stdout:6242: done (3 left) 2020-06-12T02:38:27.425 INFO:tasks.rados.rados.0.smithi107.stdout:6244: done (2 left) 2020-06-12T02:38:27.425 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:38:27.426 INFO:tasks.rados.rados.0.smithi107.stdout:6245: seq_num 6244 ranges {0=581632,581632=565248,1146880=237568} 2020-06-12T02:38:27.426 INFO:tasks.rados.rados.0.smithi107.stdout:6245: writing smithi10729407-2 from 0 to 581632 tid 1 2020-06-12T02:38:27.426 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2020-06-12T02:38:27.427 INFO:tasks.rados.rados.0.smithi107.stdout:6245: writing smithi10729407-2 from 581632 to 1146880 tid 2 2020-06-12T02:38:27.427 INFO:tasks.rados.rados.0.smithi107.stdout:6245: writing smithi10729407-2 from 1146880 to 1384448 tid 3 2020-06-12T02:38:27.427 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:27.428 INFO:tasks.rados.rados.0.smithi107.stdout:6246: seq_num 6245 ranges {0=598016,598016=442368,1040384=360448} 2020-06-12T02:38:27.429 INFO:tasks.rados.rados.0.smithi107.stdout:6246: writing smithi10729407-19 from 0 to 598016 tid 1 2020-06-12T02:38:27.431 INFO:tasks.rados.rados.0.smithi107.stdout:6246: writing smithi10729407-19 from 598016 to 1040384 tid 2 2020-06-12T02:38:27.434 INFO:tasks.rados.rados.0.smithi107.stdout:6246: writing smithi10729407-19 from 1040384 to 1400832 tid 3 2020-06-12T02:38:27.434 INFO:tasks.rados.rados.0.smithi107.stdout:6247: delete oid 20 current snap is 0 2020-06-12T02:38:27.434 INFO:tasks.rados.rados.0.smithi107.stdout:6243: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:27.435 INFO:tasks.rados.rados.0.smithi107.stdout:6243: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:27.435 INFO:tasks.rados.rados.0.smithi107.stdout:6243: finishing write tid 5 to smithi10729407-5 2020-06-12T02:38:27.435 INFO:tasks.rados.rados.0.smithi107.stdout:6243: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:27.435 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 627 (ObjNum 6242 snap 0 seq_num 6242) dirty exists 2020-06-12T02:38:27.435 INFO:tasks.rados.rados.0.smithi107.stdout:6243: left oid 5 (ObjNum 6242 snap 0 seq_num 6242) 2020-06-12T02:38:27.436 INFO:tasks.rados.rados.0.smithi107.stdout:6245: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:27.438 INFO:tasks.rados.rados.0.smithi107.stdout:6246: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:27.438 INFO:tasks.rados.rados.0.smithi107.stdout:6243: done (4 left) 2020-06-12T02:38:27.439 INFO:tasks.rados.rados.0.smithi107.stdout:6247: done (3 left) 2020-06-12T02:38:27.439 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:27.439 INFO:tasks.rados.rados.0.smithi107.stdout:6248: seq_num 6247 ranges {0=630784,630784=614400,1245184=188416} 2020-06-12T02:38:27.442 INFO:tasks.rados.rados.0.smithi107.stdout:6248: writing smithi10729407-12 from 0 to 630784 tid 1 2020-06-12T02:38:27.446 INFO:tasks.rados.rados.0.smithi107.stdout:6248: writing smithi10729407-12 from 630784 to 1245184 tid 2 2020-06-12T02:38:27.447 INFO:tasks.rados.rados.0.smithi107.stdout:6248: writing smithi10729407-12 from 1245184 to 1433600 tid 3 2020-06-12T02:38:27.447 INFO:tasks.rados.rados.0.smithi107.stdout:6249: delete oid 7 current snap is 0 2020-06-12T02:38:27.448 INFO:tasks.rados.rados.0.smithi107.stdout:6245: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:27.448 INFO:tasks.rados.rados.0.smithi107.stdout:6245: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:27.448 INFO:tasks.rados.rados.0.smithi107.stdout:6245: finishing write tid 5 to smithi10729407-2 2020-06-12T02:38:27.448 INFO:tasks.rados.rados.0.smithi107.stdout:6245: finishing write tid 6 to smithi10729407-2 2020-06-12T02:38:27.448 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 602 (ObjNum 6244 snap 0 seq_num 6244) dirty exists 2020-06-12T02:38:27.449 INFO:tasks.rados.rados.0.smithi107.stdout:6245: left oid 2 (ObjNum 6244 snap 0 seq_num 6244) 2020-06-12T02:38:27.449 INFO:tasks.rados.rados.0.smithi107.stdout:6246: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:27.449 INFO:tasks.rados.rados.0.smithi107.stdout:6246: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:27.450 INFO:tasks.rados.rados.0.smithi107.stdout:6246: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:27.450 INFO:tasks.rados.rados.0.smithi107.stdout:6246: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:27.450 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1128 (ObjNum 6245 snap 0 seq_num 6245) dirty exists 2020-06-12T02:38:27.450 INFO:tasks.rados.rados.0.smithi107.stdout:6246: left oid 19 (ObjNum 6245 snap 0 seq_num 6245) 2020-06-12T02:38:27.451 INFO:tasks.rados.rados.0.smithi107.stdout:6245: done (4 left) 2020-06-12T02:38:27.451 INFO:tasks.rados.rados.0.smithi107.stdout:6246: done (3 left) 2020-06-12T02:38:27.451 INFO:tasks.rados.rados.0.smithi107.stdout:6249: done (2 left) 2020-06-12T02:38:27.452 INFO:tasks.rados.rados.0.smithi107.stdout:6250: delete oid 14 current snap is 0 2020-06-12T02:38:27.453 INFO:tasks.rados.rados.0.smithi107.stdout:6248: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:27.454 INFO:tasks.rados.rados.0.smithi107.stdout:6250: done (2 left) 2020-06-12T02:38:27.454 INFO:tasks.rados.rados.0.smithi107.stdout:6251: delete oid 7 current snap is 0 2020-06-12T02:38:27.454 INFO:tasks.rados.rados.0.smithi107.stdout:6248: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:27.456 INFO:tasks.rados.rados.0.smithi107.stdout:6248: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:27.456 INFO:tasks.rados.rados.0.smithi107.stdout:6248: finishing write tid 5 to smithi10729407-12 2020-06-12T02:38:27.456 INFO:tasks.rados.rados.0.smithi107.stdout:6251: done (2 left) 2020-06-12T02:38:27.456 INFO:tasks.rados.rados.0.smithi107.stdout:6252: delete oid 13 current snap is 0 2020-06-12T02:38:27.457 INFO:tasks.rados.rados.0.smithi107.stdout:6248: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:27.458 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1132 (ObjNum 6247 snap 0 seq_num 6247) dirty exists 2020-06-12T02:38:27.458 INFO:tasks.rados.rados.0.smithi107.stdout:6248: left oid 12 (ObjNum 6247 snap 0 seq_num 6247) 2020-06-12T02:38:27.458 INFO:tasks.rados.rados.0.smithi107.stdout:6248: done (2 left) 2020-06-12T02:38:27.459 INFO:tasks.rados.rados.0.smithi107.stdout:6252: done (1 left) 2020-06-12T02:38:27.459 INFO:tasks.rados.rados.0.smithi107.stdout:6253: delete oid 18 current snap is 0 2020-06-12T02:38:27.460 INFO:tasks.rados.rados.0.smithi107.stdout:6253: done (1 left) 2020-06-12T02:38:27.460 INFO:tasks.rados.rados.0.smithi107.stdout:6254: delete oid 18 current snap is 0 2020-06-12T02:38:27.463 INFO:tasks.rados.rados.0.smithi107.stdout:6254: done (1 left) 2020-06-12T02:38:27.463 INFO:tasks.rados.rados.0.smithi107.stdout:6255: delete oid 5 current snap is 0 2020-06-12T02:38:27.465 INFO:tasks.rados.rados.0.smithi107.stdout:6255: done (1 left) 2020-06-12T02:38:27.466 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:38:27.466 INFO:tasks.rados.rados.0.smithi107.stdout:6256: seq_num 6255 ranges {0=761856,761856=540672,1302528=262144} 2020-06-12T02:38:27.469 INFO:tasks.rados.rados.0.smithi107.stdout:6256: writing smithi10729407-20 from 0 to 761856 tid 1 2020-06-12T02:38:27.472 INFO:tasks.rados.rados.0.smithi107.stdout:6256: writing smithi10729407-20 from 761856 to 1302528 tid 2 2020-06-12T02:38:27.474 INFO:tasks.rados.rados.0.smithi107.stdout:6256: writing smithi10729407-20 from 1302528 to 1564672 tid 3 2020-06-12T02:38:27.475 INFO:tasks.rados.rados.0.smithi107.stdout:6257: delete oid 15 current snap is 0 2020-06-12T02:38:27.481 INFO:tasks.rados.rados.0.smithi107.stdout:6257: done (2 left) 2020-06-12T02:38:27.481 INFO:tasks.rados.rados.0.smithi107.stdout:6258: delete oid 9 current snap is 0 2020-06-12T02:38:27.483 INFO:tasks.rados.rados.0.smithi107.stdout:6256: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:27.483 INFO:tasks.rados.rados.0.smithi107.stdout:6256: finishing write tid 2 to smithi10729407-20 2020-06-12T02:38:27.484 INFO:tasks.rados.rados.0.smithi107.stdout:6256: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:27.484 INFO:tasks.rados.rados.0.smithi107.stdout:6256: finishing write tid 5 to smithi10729407-20 2020-06-12T02:38:27.484 INFO:tasks.rados.rados.0.smithi107.stdout:6258: done (2 left) 2020-06-12T02:38:27.485 INFO:tasks.rados.rados.0.smithi107.stdout:6259: delete oid 18 current snap is 0 2020-06-12T02:38:27.485 INFO:tasks.rados.rados.0.smithi107.stdout:6256: finishing write tid 6 to smithi10729407-20 2020-06-12T02:38:27.485 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 480 (ObjNum 6255 snap 0 seq_num 6255) dirty exists 2020-06-12T02:38:27.485 INFO:tasks.rados.rados.0.smithi107.stdout:6256: left oid 20 (ObjNum 6255 snap 0 seq_num 6255) 2020-06-12T02:38:27.495 INFO:tasks.rados.rados.0.smithi107.stdout:6256: done (2 left) 2020-06-12T02:38:27.495 INFO:tasks.rados.rados.0.smithi107.stdout:6259: done (1 left) 2020-06-12T02:38:27.496 INFO:tasks.rados.rados.0.smithi107.stdout:6260: delete oid 11 current snap is 0 2020-06-12T02:38:27.496 INFO:tasks.rados.rados.0.smithi107.stdout:6260: done (1 left) 2020-06-12T02:38:27.496 INFO:tasks.rados.rados.0.smithi107.stdout:6261: delete oid 4 current snap is 0 2020-06-12T02:38:27.496 INFO:tasks.rados.rados.0.smithi107.stdout:6261: done (1 left) 2020-06-12T02:38:27.496 INFO:tasks.rados.rados.0.smithi107.stdout:6262: delete oid 12 current snap is 0 2020-06-12T02:38:27.497 INFO:tasks.rados.rados.0.smithi107.stdout:6262: done (1 left) 2020-06-12T02:38:27.497 INFO:tasks.rados.rados.0.smithi107.stdout:6263: delete oid 1 current snap is 0 2020-06-12T02:38:27.500 INFO:tasks.rados.rados.0.smithi107.stdout:6263: done (1 left) 2020-06-12T02:38:27.501 INFO:tasks.rados.rados.0.smithi107.stdout:6264: delete oid 20 current snap is 0 2020-06-12T02:38:27.503 INFO:tasks.rados.rados.0.smithi107.stdout:6264: done (1 left) 2020-06-12T02:38:27.503 INFO:tasks.rados.rados.0.smithi107.stdout:6265: delete oid 1 current snap is 0 2020-06-12T02:38:27.508 INFO:tasks.rados.rados.0.smithi107.stdout:6265: done (1 left) 2020-06-12T02:38:27.509 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:27.509 INFO:tasks.rados.rados.0.smithi107.stdout:6266: seq_num 6265 ranges {0=532480,532480=679936,1212416=475136,1687552=49152} 2020-06-12T02:38:27.510 INFO:tasks.rados.rados.0.smithi107.stdout:6266: writing smithi10729407-11 from 0 to 532480 tid 1 2020-06-12T02:38:27.512 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:27.510+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:27.514 INFO:tasks.rados.rados.0.smithi107.stdout:6266: writing smithi10729407-11 from 532480 to 1212416 tid 2 2020-06-12T02:38:27.518 INFO:tasks.rados.rados.0.smithi107.stdout:6266: writing smithi10729407-11 from 1212416 to 1687552 tid 3 2020-06-12T02:38:27.518 INFO:tasks.rados.rados.0.smithi107.stdout:6266: writing smithi10729407-11 from 1687552 to 1736704 tid 4 2020-06-12T02:38:27.519 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:27.519 INFO:tasks.rados.rados.0.smithi107.stdout:6267: seq_num 6266 ranges {0=548864,548864=565248,1114112=638976} 2020-06-12T02:38:27.521 INFO:tasks.rados.rados.0.smithi107.stdout:6267: writing smithi10729407-15 from 0 to 548864 tid 1 2020-06-12T02:38:27.524 INFO:tasks.rados.rados.0.smithi107.stdout:6267: writing smithi10729407-15 from 548864 to 1114112 tid 2 2020-06-12T02:38:27.529 INFO:tasks.rados.rados.0.smithi107.stdout:6267: writing smithi10729407-15 from 1114112 to 1753088 tid 3 2020-06-12T02:38:27.529 INFO:tasks.rados.rados.0.smithi107.stdout:6268: delete oid 1 current snap is 0 2020-06-12T02:38:27.530 INFO:tasks.rados.rados.0.smithi107.stdout:6266: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:27.531 INFO:tasks.rados.rados.0.smithi107.stdout:6266: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:27.531 INFO:tasks.rados.rados.0.smithi107.stdout:6266: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:27.531 INFO:tasks.rados.rados.0.smithi107.stdout:6266: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:27.531 INFO:tasks.rados.rados.0.smithi107.stdout:6266: finishing write tid 6 to smithi10729407-11 2020-06-12T02:38:27.532 INFO:tasks.rados.rados.0.smithi107.stdout:6266: finishing write tid 7 to smithi10729407-11 2020-06-12T02:38:27.532 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1113 (ObjNum 6265 snap 0 seq_num 6265) dirty exists 2020-06-12T02:38:27.533 INFO:tasks.rados.rados.0.smithi107.stdout:6266: left oid 11 (ObjNum 6265 snap 0 seq_num 6265) 2020-06-12T02:38:27.534 INFO:tasks.rados.rados.0.smithi107.stdout:6266: done (3 left) 2020-06-12T02:38:27.534 INFO:tasks.rados.rados.0.smithi107.stdout:6268: done (2 left) 2020-06-12T02:38:27.534 INFO:tasks.rados.rados.0.smithi107.stdout:6269: delete oid 8 current snap is 0 2020-06-12T02:38:27.537 INFO:tasks.rados.rados.0.smithi107.stdout:6267: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:27.538 INFO:tasks.rados.rados.0.smithi107.stdout:6267: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:27.538 INFO:tasks.rados.rados.0.smithi107.stdout:6267: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:27.539 INFO:tasks.rados.rados.0.smithi107.stdout:6267: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:27.541 INFO:tasks.rados.rados.0.smithi107.stdout:6269: done (2 left) 2020-06-12T02:38:27.541 INFO:tasks.rados.rados.0.smithi107.stdout:6270: delete oid 11 current snap is 0 2020-06-12T02:38:27.542 INFO:tasks.rados.rados.0.smithi107.stdout:6267: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:27.542 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1151 (ObjNum 6266 snap 0 seq_num 6266) dirty exists 2020-06-12T02:38:27.542 INFO:tasks.rados.rados.0.smithi107.stdout:6267: left oid 15 (ObjNum 6266 snap 0 seq_num 6266) 2020-06-12T02:38:27.543 INFO:tasks.rados.rados.0.smithi107.stdout:6267: done (2 left) 2020-06-12T02:38:27.543 INFO:tasks.rados.rados.0.smithi107.stdout:6270: done (1 left) 2020-06-12T02:38:27.543 INFO:tasks.rados.rados.0.smithi107.stdout:6271: delete oid 16 current snap is 0 2020-06-12T02:38:27.545 INFO:tasks.rados.rados.0.smithi107.stdout:6271: done (1 left) 2020-06-12T02:38:27.546 INFO:tasks.rados.rados.0.smithi107.stdout:6272: delete oid 20 current snap is 0 2020-06-12T02:38:27.550 INFO:tasks.rados.rados.0.smithi107.stdout:6272: done (1 left) 2020-06-12T02:38:27.551 INFO:tasks.rados.rados.0.smithi107.stdout:6273: delete oid 8 current snap is 0 2020-06-12T02:38:27.553 INFO:tasks.rados.rados.0.smithi107.stdout:6273: done (1 left) 2020-06-12T02:38:27.553 INFO:tasks.rados.rados.0.smithi107.stdout:6274: delete oid 13 current snap is 0 2020-06-12T02:38:27.565 INFO:tasks.rados.rados.0.smithi107.stdout:6274: done (1 left) 2020-06-12T02:38:27.566 INFO:tasks.rados.rados.0.smithi107.stdout:6275: delete oid 7 current snap is 0 2020-06-12T02:38:27.566 INFO:tasks.rados.rados.0.smithi107.stdout:6275: done (1 left) 2020-06-12T02:38:27.566 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:27.567 INFO:tasks.rados.rados.0.smithi107.stdout:6276: seq_num 6275 ranges {0=704512,704512=778240,1482752=425984} 2020-06-12T02:38:27.567 INFO:tasks.rados.rados.0.smithi107.stdout:6276: writing smithi10729407-14 from 0 to 704512 tid 1 2020-06-12T02:38:27.567 INFO:tasks.rados.rados.0.smithi107.stdout:6276: writing smithi10729407-14 from 704512 to 1482752 tid 2 2020-06-12T02:38:27.569 INFO:tasks.rados.rados.0.smithi107.stdout:6276: writing smithi10729407-14 from 1482752 to 1908736 tid 3 2020-06-12T02:38:27.570 INFO:tasks.rados.rados.0.smithi107.stdout:6277: delete oid 1 current snap is 0 2020-06-12T02:38:27.571 INFO:tasks.rados.rados.0.smithi107.stdout:6276: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:27.575 INFO:tasks.rados.rados.0.smithi107.stdout:6276: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:27.576 INFO:tasks.rados.rados.0.smithi107.stdout:6277: done (2 left) 2020-06-12T02:38:27.577 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:27.577 INFO:tasks.rados.rados.0.smithi107.stdout:6278: seq_num 6277 ranges {1753088=737280,2490368=540672,3031040=417792,3448832=245760} 2020-06-12T02:38:27.590 INFO:tasks.rados.rados.0.smithi107.stdout:6278: writing smithi10729407-15 from 1753088 to 2490368 tid 1 2020-06-12T02:38:27.592 INFO:tasks.rados.rados.0.smithi107.stdout:6278: writing smithi10729407-15 from 2490368 to 3031040 tid 2 2020-06-12T02:38:27.597 INFO:tasks.rados.rados.0.smithi107.stdout:6278: writing smithi10729407-15 from 3031040 to 3448832 tid 3 2020-06-12T02:38:27.597 INFO:tasks.rados.rados.0.smithi107.stdout:6278: writing smithi10729407-15 from 3448832 to 3694592 tid 4 2020-06-12T02:38:27.598 INFO:tasks.rados.rados.0.smithi107.stdout:6279: delete oid 8 current snap is 0 2020-06-12T02:38:27.598 INFO:tasks.rados.rados.0.smithi107.stdout:6276: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:27.599 INFO:tasks.rados.rados.0.smithi107.stdout:6276: finishing write tid 5 to smithi10729407-14 2020-06-12T02:38:27.599 INFO:tasks.rados.rados.0.smithi107.stdout:6276: finishing write tid 6 to smithi10729407-14 2020-06-12T02:38:27.601 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1118 (ObjNum 6275 snap 0 seq_num 6275) dirty exists 2020-06-12T02:38:27.601 INFO:tasks.rados.rados.0.smithi107.stdout:6276: left oid 14 (ObjNum 6275 snap 0 seq_num 6275) 2020-06-12T02:38:27.602 INFO:tasks.rados.rados.0.smithi107.stdout:6278: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:27.606 INFO:tasks.rados.rados.0.smithi107.stdout:6276: done (3 left) 2020-06-12T02:38:27.606 INFO:tasks.rados.rados.0.smithi107.stdout:6279: done (2 left) 2020-06-12T02:38:27.606 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:27.606 INFO:tasks.rados.rados.0.smithi107.stdout:6280: seq_num 6279 ranges {0=770048,770048=712704,1482752=434176,1916928=57344} 2020-06-12T02:38:27.610 INFO:tasks.rados.rados.0.smithi107.stdout:6280: writing smithi10729407-5 from 0 to 770048 tid 1 2020-06-12T02:38:27.614 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:27.614 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:38:27.614 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:38:27.614 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:38:27.615 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.615 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:860 3.3s0 3:ce45c70a:::smithi10729407-7:head [append 0~81920] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:38:27.615 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.928619+0000", 2020-06-12T02:38:27.615 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 55.680469086999999, 2020-06-12T02:38:27.615 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017431941999999999, 2020-06-12T02:38:27.616 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:27.616 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:27.616 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:27.616 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:27.617 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:27.617 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 860 2020-06-12T02:38:27.617 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.617 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:27.617 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.618 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:27.618 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928619+0000", 2020-06-12T02:38:27.618 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:27.618 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.618 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.619 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:27.619 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928619+0000", 2020-06-12T02:38:27.619 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000105809 2020-06-12T02:38:27.619 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.620 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.620 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:27.620 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928724+0000", 2020-06-12T02:38:27.620 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2339999999999994e-06 2020-06-12T02:38:27.620 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.621 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.621 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:27.621 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928733+0000", 2020-06-12T02:38:27.621 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0268000000000001e-05 2020-06-12T02:38:27.622 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.622 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.622 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:27.623 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928743+0000", 2020-06-12T02:38:27.623 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5190000000000003e-06 2020-06-12T02:38:27.624 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.624 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.624 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:27.624 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928747+0000", 2020-06-12T02:38:27.625 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0103999999999998e-05 2020-06-12T02:38:27.625 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.625 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.625 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.625 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928777+0000", 2020-06-12T02:38:27.626 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8249000000000002e-05 2020-06-12T02:38:27.626 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.626 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.626 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:27.627 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928866+0000", 2020-06-12T02:38:27.627 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011151629 2020-06-12T02:38:27.627 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.628 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.630 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.630 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.940017+0000", 2020-06-12T02:38:27.630 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8281e-05 2020-06-12T02:38:27.630 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.631 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.631 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:27.631 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.940076+0000", 2020-06-12T02:38:27.631 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000346736 2020-06-12T02:38:27.631 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.632 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.632 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:27.632 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.940422+0000", 2020-06-12T02:38:27.632 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024135049999999998 2020-06-12T02:38:27.632 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.633 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.633 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:27.634 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.942836+0000", 2020-06-12T02:38:27.634 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003170258 2020-06-12T02:38:27.634 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.634 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.635 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:27.635 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946006+0000", 2020-06-12T02:38:27.635 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4350000000000001e-05 2020-06-12T02:38:27.635 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.635 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.636 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:27.636 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946050+0000", 2020-06-12T02:38:27.636 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017431941999999999 2020-06-12T02:38:27.636 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.637 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:27.637 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.637 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.637 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.641 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:861 3.3s0 3:ce45c70a:::smithi10729407-7:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:38:27.641 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.928778+0000", 2020-06-12T02:38:27.641 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 55.680309473999998, 2020-06-12T02:38:27.642 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017522738, 2020-06-12T02:38:27.643 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:27.643 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:27.643 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:27.643 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:27.644 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:27.644 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 861 2020-06-12T02:38:27.644 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.645 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:27.645 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.645 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:27.646 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928778+0000", 2020-06-12T02:38:27.646 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:27.646 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.646 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.647 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:27.647 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928778+0000", 2020-06-12T02:38:27.647 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8186000000000002e-05 2020-06-12T02:38:27.647 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.647 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.648 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:27.648 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928826+0000", 2020-06-12T02:38:27.648 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6689999999999997e-06 2020-06-12T02:38:27.648 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.648 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.649 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:27.649 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928831+0000", 2020-06-12T02:38:27.649 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.686e-06 2020-06-12T02:38:27.649 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.649 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.650 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:27.650 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928838+0000", 2020-06-12T02:38:27.650 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7159999999999999e-06 2020-06-12T02:38:27.650 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.650 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.651 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:27.651 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928840+0000", 2020-06-12T02:38:27.651 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4212999999999997e-05 2020-06-12T02:38:27.651 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.651 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.652 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.652 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928895+0000", 2020-06-12T02:38:27.652 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5853999999999994e-05 2020-06-12T02:38:27.652 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.652 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.653 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:27.653 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928990+0000", 2020-06-12T02:38:27.653 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012023982000000001 2020-06-12T02:38:27.653 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.653 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.654 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.654 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941014+0000", 2020-06-12T02:38:27.655 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5104e-05 2020-06-12T02:38:27.655 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.655 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.655 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:27.655 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941070+0000", 2020-06-12T02:38:27.655 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00026444499999999998 2020-06-12T02:38:27.656 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.656 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.656 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:27.656 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941334+0000", 2020-06-12T02:38:27.657 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001704075 2020-06-12T02:38:27.657 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.657 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.657 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:27.658 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.943038+0000", 2020-06-12T02:38:27.659 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031891570000000002 2020-06-12T02:38:27.660 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.660 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.660 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:27.660 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946227+0000", 2020-06-12T02:38:27.661 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3651000000000003e-05 2020-06-12T02:38:27.661 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.661 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.661 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:27.661 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946301+0000", 2020-06-12T02:38:27.662 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017522738 2020-06-12T02:38:27.662 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.662 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:27.662 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.662 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.662 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.663 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:862 3.3s0 3:ce45c70a:::smithi10729407-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:38:27.663 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:31.928876+0000", 2020-06-12T02:38:27.663 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 55.680211978000003, 2020-06-12T02:38:27.663 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019206890000000001, 2020-06-12T02:38:27.663 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:27.664 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:27.664 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:27.664 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:27.664 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:27.664 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 862 2020-06-12T02:38:27.664 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.665 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:27.665 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.665 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:27.665 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928876+0000", 2020-06-12T02:38:27.665 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:27.666 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.666 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.666 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:27.666 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928876+0000", 2020-06-12T02:38:27.666 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9268999999999999e-05 2020-06-12T02:38:27.667 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.667 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.667 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:27.667 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928925+0000", 2020-06-12T02:38:27.668 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8120000000000002e-06 2020-06-12T02:38:27.668 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.668 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.668 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:27.669 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928932+0000", 2020-06-12T02:38:27.669 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3471e-05 2020-06-12T02:38:27.669 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.670 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.670 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:27.670 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928945+0000", 2020-06-12T02:38:27.670 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4279999999999998e-06 2020-06-12T02:38:27.670 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.671 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.671 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:27.671 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.928950+0000", 2020-06-12T02:38:27.671 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6358000000000004e-05 2020-06-12T02:38:27.671 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.671 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.672 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.672 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.929016+0000", 2020-06-12T02:38:27.672 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9697999999999996e-05 2020-06-12T02:38:27.672 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.672 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.673 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:27.673 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.929086+0000", 2020-06-12T02:38:27.673 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012409205 2020-06-12T02:38:27.673 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.673 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.673 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.674 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941495+0000", 2020-06-12T02:38:27.674 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9660000000000006e-05 2020-06-12T02:38:27.674 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.674 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.674 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:27.675 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.941585+0000", 2020-06-12T02:38:27.675 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050329099999999998 2020-06-12T02:38:27.675 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.675 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.675 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.676 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946617+0000", 2020-06-12T02:38:27.676 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8437999999999996e-05 2020-06-12T02:38:27.676 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.676 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.677 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:27.677 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.946706+0000", 2020-06-12T02:38:27.677 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0013766410000000001 2020-06-12T02:38:27.677 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.677 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.677 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:27.678 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:31.948083+0000", 2020-06-12T02:38:27.678 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019206890000000001 2020-06-12T02:38:27.678 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.678 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:27.678 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.679 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.679 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.679 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:879 3.fs1 3:fc7e0bf7:::smithi10729407-17:head [append 0~491520] snapc 0=[] ondisk+write+known_if_redirected e25)", 2020-06-12T02:38:27.679 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.000881+0000", 2020-06-12T02:38:27.680 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 55.608206797000001, 2020-06-12T02:38:27.680 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017891845, 2020-06-12T02:38:27.680 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:27.680 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:27.680 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:27.681 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:27.681 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:27.681 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 879 2020-06-12T02:38:27.681 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.681 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:27.681 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.682 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:27.682 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.000881+0000", 2020-06-12T02:38:27.682 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:27.682 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.682 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.683 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:27.683 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.000881+0000", 2020-06-12T02:38:27.683 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00044151599999999998 2020-06-12T02:38:27.683 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.683 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.683 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:27.684 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001322+0000", 2020-06-12T02:38:27.684 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0210000000000001e-06 2020-06-12T02:38:27.684 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.684 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.684 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:27.685 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001326+0000", 2020-06-12T02:38:27.685 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6659999999999997e-06 2020-06-12T02:38:27.685 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.685 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.685 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:27.685 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001335+0000", 2020-06-12T02:38:27.686 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4299999999999999e-06 2020-06-12T02:38:27.686 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.686 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.686 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:27.686 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.001339+0000", 2020-06-12T02:38:27.687 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0028741510000000001 2020-06-12T02:38:27.687 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.687 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.687 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.687 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.004214+0000", 2020-06-12T02:38:27.687 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010480999999999999 2020-06-12T02:38:27.688 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.688 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.688 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:27.688 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.004318+0000", 2020-06-12T02:38:27.688 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021112359999999998 2020-06-12T02:38:27.689 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.689 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.689 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.689 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006430+0000", 2020-06-12T02:38:27.689 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2935000000000001e-05 2020-06-12T02:38:27.689 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.696 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.697 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:27.697 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.006483+0000", 2020-06-12T02:38:27.697 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00056411099999999995 2020-06-12T02:38:27.697 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.697 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.698 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:27.698 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.007047+0000", 2020-06-12T02:38:27.698 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0076749369999999997 2020-06-12T02:38:27.698 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.699 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.699 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:27.699 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.014722+0000", 2020-06-12T02:38:27.699 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00395693 2020-06-12T02:38:27.699 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.700 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.700 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:27.701 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.018679+0000", 2020-06-12T02:38:27.701 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4102000000000001e-05 2020-06-12T02:38:27.701 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.701 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.702 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:27.702 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.018773+0000", 2020-06-12T02:38:27.702 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017891845 2020-06-12T02:38:27.702 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.702 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:27.703 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.703 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.703 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.703 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8124 3.es1 3:78f01648:::smithi10729407-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e33)", 2020-06-12T02:38:27.703 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:52.690794+0000", 2020-06-12T02:38:27.704 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.918293319999997, 2020-06-12T02:38:27.704 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019298497000000001, 2020-06-12T02:38:27.704 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:27.704 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:27.704 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:27.704 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:27.705 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:27.705 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8124 2020-06-12T02:38:27.705 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.705 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:27.705 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.706 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:27.706 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690794+0000", 2020-06-12T02:38:27.706 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:27.706 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.706 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.707 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:27.707 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690794+0000", 2020-06-12T02:38:27.707 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1869999999999999e-05 2020-06-12T02:38:27.707 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.707 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.707 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:27.708 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690816+0000", 2020-06-12T02:38:27.708 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.162e-06 2020-06-12T02:38:27.708 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.708 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.708 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:27.709 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690819+0000", 2020-06-12T02:38:27.709 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2280000000000003e-06 2020-06-12T02:38:27.709 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.709 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.709 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:27.710 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690826+0000", 2020-06-12T02:38:27.710 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.452e-06 2020-06-12T02:38:27.710 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.710 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.710 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:27.710 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.690828+0000", 2020-06-12T02:38:27.711 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00081402600000000003 2020-06-12T02:38:27.711 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.711 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.711 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.711 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.691642+0000", 2020-06-12T02:38:27.712 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00024263 2020-06-12T02:38:27.712 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.713 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.713 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:27.713 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.691885+0000", 2020-06-12T02:38:27.713 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010968739 2020-06-12T02:38:27.714 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.714 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.714 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.714 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.702853+0000", 2020-06-12T02:38:27.715 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8985000000000002e-05 2020-06-12T02:38:27.715 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.715 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.715 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:27.715 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.702952+0000", 2020-06-12T02:38:27.716 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0046973329999999997 2020-06-12T02:38:27.716 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.716 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.716 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.716 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.707650+0000", 2020-06-12T02:38:27.717 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3773e-05 2020-06-12T02:38:27.717 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.717 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.717 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:27.717 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.707723+0000", 2020-06-12T02:38:27.717 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023692990000000001 2020-06-12T02:38:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:52.710093+0000", 2020-06-12T02:38:27.718 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019298497000000001 2020-06-12T02:38:27.719 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.719 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:27.719 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.719 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.719 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.719 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8841 3.3s0 3:ce45c70a:::smithi10729407-7:head [append 0~794624] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:27.720 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:11.820531+0000", 2020-06-12T02:38:27.720 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 15.788556348, 2020-06-12T02:38:27.720 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019581604999999998, 2020-06-12T02:38:27.720 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:27.720 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:27.721 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:27.721 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:27.721 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:27.721 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8841 2020-06-12T02:38:27.721 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.721 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:27.722 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.722 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:27.723 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.820531+0000", 2020-06-12T02:38:27.724 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:27.724 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.724 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.724 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:27.724 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.820531+0000", 2020-06-12T02:38:27.724 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00070341999999999996 2020-06-12T02:38:27.725 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.725 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.725 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:27.725 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.821235+0000", 2020-06-12T02:38:27.725 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9590000000000001e-06 2020-06-12T02:38:27.726 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.726 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.726 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:27.726 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.821244+0000", 2020-06-12T02:38:27.726 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6135000000000001e-05 2020-06-12T02:38:27.727 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.727 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.727 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:27.727 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.821260+0000", 2020-06-12T02:38:27.727 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5159999999999997e-06 2020-06-12T02:38:27.727 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.728 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.728 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:27.728 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.821267+0000", 2020-06-12T02:38:27.728 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3677999999999998e-05 2020-06-12T02:38:27.728 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.729 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.729 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.729 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.821311+0000", 2020-06-12T02:38:27.729 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2262000000000006e-05 2020-06-12T02:38:27.729 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.729 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.730 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:27.730 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.821393+0000", 2020-06-12T02:38:27.730 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00061887800000000001 2020-06-12T02:38:27.730 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.730 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.731 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:27.731 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.822012+0000", 2020-06-12T02:38:27.731 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0043135860000000003 2020-06-12T02:38:27.731 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.731 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.732 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:27.732 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.826326+0000", 2020-06-12T02:38:27.732 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013738409 2020-06-12T02:38:27.732 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.732 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.732 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:27.733 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.840064+0000", 2020-06-12T02:38:27.733 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8761999999999998e-05 2020-06-12T02:38:27.733 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.733 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.733 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:27.733 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.840113+0000", 2020-06-12T02:38:27.734 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019581604999999998 2020-06-12T02:38:27.734 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.734 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:27.735 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.736 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.736 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.736 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8866 3.3s0 3:c2ca77da:::smithi10729407-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e49)", 2020-06-12T02:38:27.737 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:11.883598+0000", 2020-06-12T02:38:27.737 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 15.725490047999999, 2020-06-12T02:38:27.737 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019905586999999999, 2020-06-12T02:38:27.737 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:27.738 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:27.738 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:27.738 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:27.739 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:27.739 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8866 2020-06-12T02:38:27.739 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.739 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:27.739 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.740 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:27.740 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.883598+0000", 2020-06-12T02:38:27.740 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:27.740 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.740 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.741 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:27.741 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.883598+0000", 2020-06-12T02:38:27.741 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2928000000000001e-05 2020-06-12T02:38:27.741 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.741 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.742 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:27.742 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.883621+0000", 2020-06-12T02:38:27.742 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2760000000000001e-06 2020-06-12T02:38:27.742 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.742 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.742 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:27.743 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.883624+0000", 2020-06-12T02:38:27.743 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2239999999999995e-06 2020-06-12T02:38:27.743 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.743 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.743 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:27.744 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.883632+0000", 2020-06-12T02:38:27.744 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2460000000000002e-06 2020-06-12T02:38:27.744 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.744 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.744 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:27.744 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.883634+0000", 2020-06-12T02:38:27.745 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0055082630000000002 2020-06-12T02:38:27.745 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.745 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.745 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.745 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.889143+0000", 2020-06-12T02:38:27.746 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9482999999999998e-05 2020-06-12T02:38:27.746 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.746 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.746 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:27.746 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.889222+0000", 2020-06-12T02:38:27.746 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0099390269999999996 2020-06-12T02:38:27.747 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.747 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.747 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.747 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.899161+0000", 2020-06-12T02:38:27.748 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000115758 2020-06-12T02:38:27.748 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.748 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.748 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:27.748 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.899277+0000", 2020-06-12T02:38:27.749 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0042263819999999999 2020-06-12T02:38:27.749 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.749 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.749 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:27.749 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:11.903503+0000", 2020-06-12T02:38:27.750 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019905586999999999 2020-06-12T02:38:27.750 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.750 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:27.750 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.750 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.751 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.751 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:9129 3.9s0 3:977e4b9d:::smithi10729407-12:head [append 0~532480] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:27.751 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:12.645731+0000", 2020-06-12T02:38:27.751 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 14.963356662000001, 2020-06-12T02:38:27.751 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017582474000000001, 2020-06-12T02:38:27.752 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:27.752 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:27.752 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:27.752 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:27.752 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:27.752 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9129 2020-06-12T02:38:27.753 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.753 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:27.753 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.753 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:27.753 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.645731+0000", 2020-06-12T02:38:27.754 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:27.754 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.754 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.754 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:27.754 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.645731+0000", 2020-06-12T02:38:27.755 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00050370799999999996 2020-06-12T02:38:27.755 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.755 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.755 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:27.755 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.646235+0000", 2020-06-12T02:38:27.755 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0410000000000003e-06 2020-06-12T02:38:27.756 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.756 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.756 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:27.756 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.646244+0000", 2020-06-12T02:38:27.756 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6384000000000001e-05 2020-06-12T02:38:27.757 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.757 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.757 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:27.757 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.646260+0000", 2020-06-12T02:38:27.757 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3039999999999999e-06 2020-06-12T02:38:27.757 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.758 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.758 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:27.758 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.646267+0000", 2020-06-12T02:38:27.758 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0970999999999997e-05 2020-06-12T02:38:27.759 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.759 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.759 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.759 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.646318+0000", 2020-06-12T02:38:27.759 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000117429 2020-06-12T02:38:27.760 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.760 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.760 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:27.760 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.646436+0000", 2020-06-12T02:38:27.760 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00084515600000000001 2020-06-12T02:38:27.761 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.761 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.761 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:27.761 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.647281+0000", 2020-06-12T02:38:27.761 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003637389 2020-06-12T02:38:27.761 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.762 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.762 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:27.762 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.650918+0000", 2020-06-12T02:38:27.762 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012318766 2020-06-12T02:38:27.762 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.763 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.763 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:27.763 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.663237+0000", 2020-06-12T02:38:27.763 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6326000000000001e-05 2020-06-12T02:38:27.763 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.763 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.764 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:27.764 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:12.663313+0000", 2020-06-12T02:38:27.764 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017582474000000001 2020-06-12T02:38:27.764 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.764 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:27.765 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.765 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.765 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.765 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:9623 3.9s0 3:93ea0cad:::smithi10729407-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e54)", 2020-06-12T02:38:27.765 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:16.878215+0000", 2020-06-12T02:38:27.766 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 10.730872191, 2020-06-12T02:38:27.767 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018281585, 2020-06-12T02:38:27.767 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:27.767 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:27.767 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:27.767 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:27.768 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:27.768 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9623 2020-06-12T02:38:27.768 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.768 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:27.768 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.769 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:27.769 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.878215+0000", 2020-06-12T02:38:27.769 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:27.769 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.769 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.770 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:27.770 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.878215+0000", 2020-06-12T02:38:27.770 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6162000000000001e-05 2020-06-12T02:38:27.770 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.770 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.771 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:27.771 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.878242+0000", 2020-06-12T02:38:27.771 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1810000000000001e-06 2020-06-12T02:38:27.771 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.771 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.771 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:27.772 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.878245+0000", 2020-06-12T02:38:27.772 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5570000000000001e-06 2020-06-12T02:38:27.772 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.772 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.772 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:27.773 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.878251+0000", 2020-06-12T02:38:27.773 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6000000000000001e-06 2020-06-12T02:38:27.773 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.773 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.773 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:27.773 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.878254+0000", 2020-06-12T02:38:27.774 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0026536559999999999 2020-06-12T02:38:27.774 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.774 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.774 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.774 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.880908+0000", 2020-06-12T02:38:27.775 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010170600000000001 2020-06-12T02:38:27.775 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.775 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.775 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:27.775 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.881009+0000", 2020-06-12T02:38:27.775 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0085655460000000003 2020-06-12T02:38:27.776 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.776 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.776 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.776 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.889575+0000", 2020-06-12T02:38:27.776 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3764999999999994e-05 2020-06-12T02:38:27.777 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.777 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.777 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:27.777 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.889669+0000", 2020-06-12T02:38:27.777 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0044753060000000001 2020-06-12T02:38:27.778 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.778 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.778 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.778 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.894144+0000", 2020-06-12T02:38:27.778 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0333999999999994e-05 2020-06-12T02:38:27.779 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.779 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.779 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:27.779 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.894214+0000", 2020-06-12T02:38:27.780 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0022827720000000002 2020-06-12T02:38:27.780 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.780 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.780 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:27.781 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.896497+0000", 2020-06-12T02:38:27.781 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018281585 2020-06-12T02:38:27.781 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.781 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:27.781 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.782 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.782 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.782 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:9783 3.9s0 3:93ea0cad:::smithi10729407-19:head [append 0~458752] snapc 0=[] ondisk+write+known_if_redirected e54)", 2020-06-12T02:38:27.782 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:17.277705+0000", 2020-06-12T02:38:27.782 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 10.331382423000001, 2020-06-12T02:38:27.783 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019659603000000001, 2020-06-12T02:38:27.783 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:27.783 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:27.783 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:27.783 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:27.783 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:27.784 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9783 2020-06-12T02:38:27.784 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.784 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:27.784 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.784 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:27.785 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.277705+0000", 2020-06-12T02:38:27.785 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:27.785 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.785 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.785 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:27.786 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.277705+0000", 2020-06-12T02:38:27.786 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047126399999999999 2020-06-12T02:38:27.786 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.786 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.786 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:27.786 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.278176+0000", 2020-06-12T02:38:27.787 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1250000000000001e-06 2020-06-12T02:38:27.787 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.787 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.787 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:27.787 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.278182+0000", 2020-06-12T02:38:27.788 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0852e-05 2020-06-12T02:38:27.788 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.788 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.788 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:27.788 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.278192+0000", 2020-06-12T02:38:27.788 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7230000000000004e-06 2020-06-12T02:38:27.789 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.789 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.789 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:27.789 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.278197+0000", 2020-06-12T02:38:27.789 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001912235 2020-06-12T02:38:27.790 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.790 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.790 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.790 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.280109+0000", 2020-06-12T02:38:27.790 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010991499999999999 2020-06-12T02:38:27.791 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.791 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.791 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:27.791 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.280219+0000", 2020-06-12T02:38:27.791 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0039835069999999998 2020-06-12T02:38:27.791 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.792 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.792 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.792 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.284203+0000", 2020-06-12T02:38:27.792 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4379000000000001e-05 2020-06-12T02:38:27.792 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.793 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.793 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:27.793 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.284277+0000", 2020-06-12T02:38:27.793 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00074622399999999998 2020-06-12T02:38:27.793 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.793 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.794 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:27.794 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.285023+0000", 2020-06-12T02:38:27.794 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0061177180000000003 2020-06-12T02:38:27.794 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.794 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.795 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:27.795 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.291141+0000", 2020-06-12T02:38:27.795 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0061324689999999998 2020-06-12T02:38:27.795 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.795 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.795 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:27.796 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.297274+0000", 2020-06-12T02:38:27.796 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1192000000000004e-05 2020-06-12T02:38:27.796 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.796 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.796 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:27.797 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:17.297365+0000", 2020-06-12T02:38:27.797 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019659603000000001 2020-06-12T02:38:27.797 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.797 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:27.797 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.797 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.799 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.799 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11418 3.9s0 3:977e4b9d:::smithi10729407-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e56)", 2020-06-12T02:38:27.799 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:21.868608+0000", 2020-06-12T02:38:27.799 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 5.7404795440000003, 2020-06-12T02:38:27.799 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019154829000000002, 2020-06-12T02:38:27.800 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:27.800 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:27.800 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:27.800 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:27.800 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:27.801 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11418 2020-06-12T02:38:27.801 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.801 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:27.801 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.801 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:27.802 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:21.868608+0000", 2020-06-12T02:38:27.802 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:27.802 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.802 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.802 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:27.803 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:21.868608+0000", 2020-06-12T02:38:27.803 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6895000000000003e-05 2020-06-12T02:38:27.803 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.803 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.803 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:27.804 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:21.868665+0000", 2020-06-12T02:38:27.804 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4420000000000003e-06 2020-06-12T02:38:27.804 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.804 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.804 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:27.804 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:21.868671+0000", 2020-06-12T02:38:27.805 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4270000000000001e-05 2020-06-12T02:38:27.805 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.805 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.805 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:27.806 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:21.868686+0000", 2020-06-12T02:38:27.806 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2609999999999999e-06 2020-06-12T02:38:27.807 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.808 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.808 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:27.808 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:21.868692+0000", 2020-06-12T02:38:27.808 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036195350000000001 2020-06-12T02:38:27.808 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.809 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.809 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.809 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:21.872311+0000", 2020-06-12T02:38:27.809 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1908000000000002e-05 2020-06-12T02:38:27.810 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.810 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.810 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:27.810 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:21.872403+0000", 2020-06-12T02:38:27.810 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0043083669999999996 2020-06-12T02:38:27.810 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.811 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.811 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.811 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:21.876712+0000", 2020-06-12T02:38:27.811 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5739999999999998e-05 2020-06-12T02:38:27.811 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.812 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.812 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:27.812 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:21.876767+0000", 2020-06-12T02:38:27.812 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0093623630000000003 2020-06-12T02:38:27.812 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.812 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.813 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.813 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:21.886130+0000", 2020-06-12T02:38:27.813 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7858999999999999e-05 2020-06-12T02:38:27.813 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.813 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.814 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:27.814 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:21.886228+0000", 2020-06-12T02:38:27.814 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001535189 2020-06-12T02:38:27.814 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.814 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.815 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:27.815 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:21.887763+0000", 2020-06-12T02:38:27.815 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019154829000000002 2020-06-12T02:38:27.815 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.815 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:27.816 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.816 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.816 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.816 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11516 3.3s0 3:c2ca77da:::smithi10729407-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e56)", 2020-06-12T02:38:27.816 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:22.161522+0000", 2020-06-12T02:38:27.816 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 5.4475654410000001, 2020-06-12T02:38:27.817 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018818970000000001, 2020-06-12T02:38:27.817 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:27.817 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:27.817 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:27.817 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:27.818 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:27.818 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11516 2020-06-12T02:38:27.818 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.818 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:27.818 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.819 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:27.819 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.161522+0000", 2020-06-12T02:38:27.819 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:27.819 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.819 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.819 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:27.820 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.161522+0000", 2020-06-12T02:38:27.820 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3088999999999999e-05 2020-06-12T02:38:27.820 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.820 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.820 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:27.821 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.161545+0000", 2020-06-12T02:38:27.821 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9570000000000004e-06 2020-06-12T02:38:27.821 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.821 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.821 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:27.821 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.161549+0000", 2020-06-12T02:38:27.822 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6330000000000002e-06 2020-06-12T02:38:27.822 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.822 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.822 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:27.822 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.161558+0000", 2020-06-12T02:38:27.823 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2259999999999999e-06 2020-06-12T02:38:27.823 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.823 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.823 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:27.823 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.161560+0000", 2020-06-12T02:38:27.823 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00065677300000000004 2020-06-12T02:38:27.824 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.824 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.824 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.824 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.162217+0000", 2020-06-12T02:38:27.824 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0732000000000001e-05 2020-06-12T02:38:27.825 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.825 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.825 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:27.825 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.162278+0000", 2020-06-12T02:38:27.825 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0074321960000000003 2020-06-12T02:38:27.825 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.826 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.826 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.826 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.169710+0000", 2020-06-12T02:38:27.826 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000100027 2020-06-12T02:38:27.826 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.827 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.827 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:27.827 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.169810+0000", 2020-06-12T02:38:27.827 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0087077620000000008 2020-06-12T02:38:27.827 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.827 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.828 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.828 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.178518+0000", 2020-06-12T02:38:27.828 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5771000000000002e-05 2020-06-12T02:38:27.828 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.828 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.829 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:27.829 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.178613+0000", 2020-06-12T02:38:27.829 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0017278040000000001 2020-06-12T02:38:27.829 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.829 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.830 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:27.830 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.180341+0000", 2020-06-12T02:38:27.830 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018818970000000001 2020-06-12T02:38:27.830 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.830 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:27.830 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.831 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.831 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.831 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11603 3.3s0 3:ce45c70a:::smithi10729407-7:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e56)", 2020-06-12T02:38:27.831 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:22.436226+0000", 2020-06-12T02:38:27.831 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 5.172861997, 2020-06-12T02:38:27.832 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019471334999999999, 2020-06-12T02:38:27.832 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:27.832 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:27.832 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:27.832 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:27.833 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:27.833 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11603 2020-06-12T02:38:27.833 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.833 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:27.833 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.834 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:27.834 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.436226+0000", 2020-06-12T02:38:27.834 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:27.834 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.834 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.835 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:27.835 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.436226+0000", 2020-06-12T02:38:27.835 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8101e-05 2020-06-12T02:38:27.836 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.836 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.836 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:27.836 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.436264+0000", 2020-06-12T02:38:27.837 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3680000000000002e-06 2020-06-12T02:38:27.837 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.837 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.837 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:27.837 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.436267+0000", 2020-06-12T02:38:27.838 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2990000000000002e-06 2020-06-12T02:38:27.838 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.838 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.838 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:27.838 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.436273+0000", 2020-06-12T02:38:27.839 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5629999999999999e-06 2020-06-12T02:38:27.839 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.839 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.839 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:27.840 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.436276+0000", 2020-06-12T02:38:27.841 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9110000000000001e-05 2020-06-12T02:38:27.841 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.841 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.841 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.842 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.436305+0000", 2020-06-12T02:38:27.842 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6860000000000002e-05 2020-06-12T02:38:27.842 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.842 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.842 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:27.843 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.436352+0000", 2020-06-12T02:38:27.843 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010885654999999999 2020-06-12T02:38:27.843 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.843 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.843 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.843 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.447238+0000", 2020-06-12T02:38:27.844 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.6062000000000003e-05 2020-06-12T02:38:27.844 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.844 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.844 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:27.844 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.447334+0000", 2020-06-12T02:38:27.845 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00042351299999999998 2020-06-12T02:38:27.845 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.845 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.845 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:27.845 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.447757+0000", 2020-06-12T02:38:27.846 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0011521369999999999 2020-06-12T02:38:27.846 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.846 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.847 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:27.847 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.448909+0000", 2020-06-12T02:38:27.848 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0066943519999999998 2020-06-12T02:38:27.848 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.848 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.848 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:27.848 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.455604+0000", 2020-06-12T02:38:27.849 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3314999999999997e-05 2020-06-12T02:38:27.849 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.849 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.849 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:27.849 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.455697+0000", 2020-06-12T02:38:27.850 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019471334999999999 2020-06-12T02:38:27.850 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.850 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:27.850 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.850 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.851 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.851 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11604 3.3s0 3:ce45c70a:::smithi10729407-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e56)", 2020-06-12T02:38:27.851 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:22.436304+0000", 2020-06-12T02:38:27.851 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 5.1727831599999998, 2020-06-12T02:38:27.851 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026739028000000001, 2020-06-12T02:38:27.851 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:27.852 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:27.852 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:27.852 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:27.852 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:27.852 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11604 2020-06-12T02:38:27.853 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.853 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:27.853 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.853 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:27.853 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.436304+0000", 2020-06-12T02:38:27.854 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:27.854 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.854 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.854 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:27.854 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.436304+0000", 2020-06-12T02:38:27.854 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0182e-05 2020-06-12T02:38:27.855 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.856 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.856 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:27.856 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.436335+0000", 2020-06-12T02:38:27.856 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2780000000000002e-06 2020-06-12T02:38:27.857 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.857 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.857 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:27.857 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.436338+0000", 2020-06-12T02:38:27.858 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5050000000000004e-06 2020-06-12T02:38:27.858 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.858 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.859 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:27.859 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.436344+0000", 2020-06-12T02:38:27.859 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6189999999999998e-06 2020-06-12T02:38:27.859 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.859 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.860 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:27.860 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.436347+0000", 2020-06-12T02:38:27.860 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1742000000000002e-05 2020-06-12T02:38:27.860 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.860 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.860 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.861 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.436379+0000", 2020-06-12T02:38:27.861 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5516000000000001e-05 2020-06-12T02:38:27.861 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.861 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.862 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:27.862 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.436444+0000", 2020-06-12T02:38:27.862 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012671909 2020-06-12T02:38:27.862 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.862 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.863 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.863 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.449116+0000", 2020-06-12T02:38:27.863 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5255999999999998e-05 2020-06-12T02:38:27.863 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.863 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.863 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:27.864 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.449201+0000", 2020-06-12T02:38:27.864 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0065621330000000004 2020-06-12T02:38:27.864 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.864 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.864 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.865 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.455764+0000", 2020-06-12T02:38:27.865 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000107519 2020-06-12T02:38:27.865 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.865 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.866 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:27.867 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.455871+0000", 2020-06-12T02:38:27.867 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0071723689999999996 2020-06-12T02:38:27.868 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.868 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.868 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:27.869 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.463043+0000", 2020-06-12T02:38:27.869 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026739028000000001 2020-06-12T02:38:27.869 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.869 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:27.869 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.870 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.870 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.870 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11605 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~712704] snapc 0=[] ondisk+write+known_if_redirected e56)", 2020-06-12T02:38:27.870 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:22.440426+0000", 2020-06-12T02:38:27.871 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 5.1686614769999997, 2020-06-12T02:38:27.871 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022109515, 2020-06-12T02:38:27.871 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:27.871 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:27.871 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:27.871 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:27.872 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:27.872 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11605 2020-06-12T02:38:27.872 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.872 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:27.872 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.874 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:27.874 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.440426+0000", 2020-06-12T02:38:27.874 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:27.874 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.874 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.875 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:27.875 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.440426+0000", 2020-06-12T02:38:27.875 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00086067700000000001 2020-06-12T02:38:27.875 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.875 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.876 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:27.876 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.441287+0000", 2020-06-12T02:38:27.876 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1980000000000007e-06 2020-06-12T02:38:27.876 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.876 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.877 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:27.877 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.441296+0000", 2020-06-12T02:38:27.877 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6762999999999999e-05 2020-06-12T02:38:27.877 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.877 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.878 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:27.878 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.441313+0000", 2020-06-12T02:38:27.878 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2409999999999998e-06 2020-06-12T02:38:27.878 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.878 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.878 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:27.879 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.441320+0000", 2020-06-12T02:38:27.879 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0079076760000000006 2020-06-12T02:38:27.879 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.879 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.879 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.880 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.449228+0000", 2020-06-12T02:38:27.880 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2406999999999998e-05 2020-06-12T02:38:27.880 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.880 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.880 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:27.881 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.449320+0000", 2020-06-12T02:38:27.881 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00067351200000000003 2020-06-12T02:38:27.881 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.881 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.881 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:27.881 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.449994+0000", 2020-06-12T02:38:27.882 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0061543329999999997 2020-06-12T02:38:27.882 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.882 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.882 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:27.882 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.456148+0000", 2020-06-12T02:38:27.883 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0063118419999999998 2020-06-12T02:38:27.883 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.883 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.883 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:27.883 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.462460+0000", 2020-06-12T02:38:27.884 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5865999999999995e-05 2020-06-12T02:38:27.884 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.884 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.884 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:27.884 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.462536+0000", 2020-06-12T02:38:27.884 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022109515 2020-06-12T02:38:27.885 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.885 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:27.885 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.885 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.885 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.886 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11606 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~516096] snapc 0=[] ondisk+write+known_if_redirected e56)", 2020-06-12T02:38:27.886 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:22.444056+0000", 2020-06-12T02:38:27.886 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 5.1650320059999997, 2020-06-12T02:38:27.886 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018730172999999999, 2020-06-12T02:38:27.886 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:27.886 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:27.887 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:27.887 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:27.887 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:27.887 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11606 2020-06-12T02:38:27.887 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.888 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:27.888 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.888 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:27.888 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.444056+0000", 2020-06-12T02:38:27.888 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:27.888 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.889 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.889 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:27.889 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.444056+0000", 2020-06-12T02:38:27.889 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054545899999999998 2020-06-12T02:38:27.889 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.890 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.890 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:27.890 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.444601+0000", 2020-06-12T02:38:27.890 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4110000000000002e-06 2020-06-12T02:38:27.890 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.890 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.891 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:27.891 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.444610+0000", 2020-06-12T02:38:27.891 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5670000000000001e-05 2020-06-12T02:38:27.891 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.891 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.892 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:27.892 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.444626+0000", 2020-06-12T02:38:27.892 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5189999999999998e-06 2020-06-12T02:38:27.892 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.892 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.892 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:27.893 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.444633+0000", 2020-06-12T02:38:27.893 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0081404879999999995 2020-06-12T02:38:27.893 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.893 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.893 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.894 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.452773+0000", 2020-06-12T02:38:27.900 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4142999999999998e-05 2020-06-12T02:38:27.901 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.901 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.901 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:27.901 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.452847+0000", 2020-06-12T02:38:27.902 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00048803099999999999 2020-06-12T02:38:27.902 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.902 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.902 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:27.902 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.453335+0000", 2020-06-12T02:38:27.902 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0029380830000000002 2020-06-12T02:38:27.903 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.903 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.903 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:27.903 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.456273+0000", 2020-06-12T02:38:27.903 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0064353600000000002 2020-06-12T02:38:27.904 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.904 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.904 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:27.904 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.462709+0000", 2020-06-12T02:38:27.904 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7008999999999994e-05 2020-06-12T02:38:27.904 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.905 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.905 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:27.905 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.462786+0000", 2020-06-12T02:38:27.905 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018730172999999999 2020-06-12T02:38:27.905 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.906 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:27.906 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.906 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.906 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.906 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11607 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~614400] snapc 0=[] ondisk+write+known_if_redirected e56)", 2020-06-12T02:38:27.906 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:22.448056+0000", 2020-06-12T02:38:27.907 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 5.1610320730000003, 2020-06-12T02:38:27.907 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017685256999999999, 2020-06-12T02:38:27.907 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:27.907 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:27.907 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:27.908 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:27.908 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:27.908 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11607 2020-06-12T02:38:27.908 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.908 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:27.908 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.909 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:27.909 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.448056+0000", 2020-06-12T02:38:27.909 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:27.909 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.909 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.910 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:27.910 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.448056+0000", 2020-06-12T02:38:27.910 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00059421099999999998 2020-06-12T02:38:27.910 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.910 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.910 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:27.911 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.448650+0000", 2020-06-12T02:38:27.911 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.52e-06 2020-06-12T02:38:27.911 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.911 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.911 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:27.912 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.448657+0000", 2020-06-12T02:38:27.912 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7218e-05 2020-06-12T02:38:27.912 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.912 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.912 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:27.912 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.448674+0000", 2020-06-12T02:38:27.913 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9220000000000002e-06 2020-06-12T02:38:27.913 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.913 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.913 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:27.913 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.448680+0000", 2020-06-12T02:38:27.914 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0076224889999999997 2020-06-12T02:38:27.914 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.914 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.914 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.914 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.456303+0000", 2020-06-12T02:38:27.914 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4634000000000004e-05 2020-06-12T02:38:27.915 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.915 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.915 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:27.915 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.456378+0000", 2020-06-12T02:38:27.915 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00050083799999999998 2020-06-12T02:38:27.916 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.916 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.916 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:27.916 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.456878+0000", 2020-06-12T02:38:27.916 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0056685240000000003 2020-06-12T02:38:27.916 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.917 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.917 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:27.917 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.462547+0000", 2020-06-12T02:38:27.917 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031519260000000002 2020-06-12T02:38:27.917 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.918 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.918 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:27.918 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.465699+0000", 2020-06-12T02:38:27.918 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1974999999999997e-05 2020-06-12T02:38:27.918 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.918 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.919 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:27.919 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.465741+0000", 2020-06-12T02:38:27.919 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017685256999999999 2020-06-12T02:38:27.919 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.919 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:27.920 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.920 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.920 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.920 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11610 3.3s0 3:c2ca77da:::smithi10729407-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e56)", 2020-06-12T02:38:27.920 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:22.449103+0000", 2020-06-12T02:38:27.920 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 5.1599846490000001, 2020-06-12T02:38:27.921 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018211881999999999, 2020-06-12T02:38:27.921 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:27.921 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:27.921 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:27.921 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:27.922 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:27.922 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11610 2020-06-12T02:38:27.922 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.922 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:27.922 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.922 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:27.923 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.449103+0000", 2020-06-12T02:38:27.923 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:27.923 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.923 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.923 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:27.924 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.449103+0000", 2020-06-12T02:38:27.924 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4397000000000002e-05 2020-06-12T02:38:27.924 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.924 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.924 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:27.924 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.449147+0000", 2020-06-12T02:38:27.925 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2489999999999998e-06 2020-06-12T02:38:27.925 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.925 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.925 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:27.925 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.449154+0000", 2020-06-12T02:38:27.926 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1751e-05 2020-06-12T02:38:27.926 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.926 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.927 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:27.927 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.449165+0000", 2020-06-12T02:38:27.927 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6480000000000002e-06 2020-06-12T02:38:27.927 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.927 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.927 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:27.928 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.449170+0000", 2020-06-12T02:38:27.928 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014699393 2020-06-12T02:38:27.928 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.928 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.928 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.929 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.463869+0000", 2020-06-12T02:38:27.929 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4336000000000003e-05 2020-06-12T02:38:27.929 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.929 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.929 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:27.930 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.463924+0000", 2020-06-12T02:38:27.930 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025321150000000001 2020-06-12T02:38:27.930 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.930 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.930 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.930 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.466456+0000", 2020-06-12T02:38:27.931 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9218999999999998e-05 2020-06-12T02:38:27.931 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.931 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.931 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:27.931 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.466505+0000", 2020-06-12T02:38:27.932 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00080977399999999998 2020-06-12T02:38:27.932 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.932 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.932 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:27.932 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.467315+0000", 2020-06-12T02:38:27.932 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018211881999999999 2020-06-12T02:38:27.933 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.933 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:27.933 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.933 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.933 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.934 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11888 3.9s0 3:93ea0cad:::smithi10729407-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e56)", 2020-06-12T02:38:27.934 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:23.435701+0000", 2020-06-12T02:38:27.934 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.173386303, 2020-06-12T02:38:27.934 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020390334, 2020-06-12T02:38:27.934 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:27.934 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:27.935 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:27.935 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:27.935 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:27.935 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11888 2020-06-12T02:38:27.935 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.936 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:27.936 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.936 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:27.936 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.435701+0000", 2020-06-12T02:38:27.936 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:27.936 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.937 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.937 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:27.937 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.435701+0000", 2020-06-12T02:38:27.937 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1142000000000005e-05 2020-06-12T02:38:27.937 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.938 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.938 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:27.938 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.435762+0000", 2020-06-12T02:38:27.938 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1990000000000006e-06 2020-06-12T02:38:27.938 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.939 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.939 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:27.939 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.435771+0000", 2020-06-12T02:38:27.939 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0879000000000001e-05 2020-06-12T02:38:27.939 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.939 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.940 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:27.940 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.435792+0000", 2020-06-12T02:38:27.940 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9399999999999996e-06 2020-06-12T02:38:27.940 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.940 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.940 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:27.941 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.435798+0000", 2020-06-12T02:38:27.941 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010338498 2020-06-12T02:38:27.941 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.941 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.941 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.942 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.446137+0000", 2020-06-12T02:38:27.942 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8717999999999999e-05 2020-06-12T02:38:27.942 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.942 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.942 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:27.942 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.446236+0000", 2020-06-12T02:38:27.943 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0067264899999999999 2020-06-12T02:38:27.943 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.943 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.943 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.943 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.452962+0000", 2020-06-12T02:38:27.944 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000104836 2020-06-12T02:38:27.944 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.944 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.944 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:27.944 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.453067+0000", 2020-06-12T02:38:27.944 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0030246320000000002 2020-06-12T02:38:27.945 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.945 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.945 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:27.945 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.456092+0000", 2020-06-12T02:38:27.945 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020390334 2020-06-12T02:38:27.946 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.946 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:27.946 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.946 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.946 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.946 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:12911 3.3s0 3:ce45c70a:::smithi10729407-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e57)", 2020-06-12T02:38:27.947 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:26.331190+0000", 2020-06-12T02:38:27.947 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 1.2778972120000001, 2020-06-12T02:38:27.947 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019367964000000001, 2020-06-12T02:38:27.947 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:27.947 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:27.948 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:27.948 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:27.948 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:27.948 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 12911 2020-06-12T02:38:27.948 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.948 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:27.949 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.949 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:27.949 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:26.331190+0000", 2020-06-12T02:38:27.949 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:27.949 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.950 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.950 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:27.950 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:26.331190+0000", 2020-06-12T02:38:27.950 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2289999999999998e-05 2020-06-12T02:38:27.950 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.950 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.951 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:27.951 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:26.331213+0000", 2020-06-12T02:38:27.951 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0460000000000001e-06 2020-06-12T02:38:27.951 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.951 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.952 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:27.952 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:26.331216+0000", 2020-06-12T02:38:27.952 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0410000000000003e-06 2020-06-12T02:38:27.952 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.952 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.953 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:27.953 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:26.331222+0000", 2020-06-12T02:38:27.953 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0200000000000001e-06 2020-06-12T02:38:27.953 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.953 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.953 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:27.954 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:26.331224+0000", 2020-06-12T02:38:27.954 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013382335 2020-06-12T02:38:27.954 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.954 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.954 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.955 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:26.344606+0000", 2020-06-12T02:38:27.955 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9628999999999994e-05 2020-06-12T02:38:27.955 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.955 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.955 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:27.955 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:26.344676+0000", 2020-06-12T02:38:27.956 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0037306700000000002 2020-06-12T02:38:27.956 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.956 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.956 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:27.956 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:26.348406+0000", 2020-06-12T02:38:27.956 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4985999999999998e-05 2020-06-12T02:38:27.957 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.957 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.957 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:27.957 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:26.348461+0000", 2020-06-12T02:38:27.957 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002096947 2020-06-12T02:38:27.958 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:27.958 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:27.958 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:27.958 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:26.350558+0000", 2020-06-12T02:38:27.958 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019367964000000001 2020-06-12T02:38:27.958 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.959 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:27.959 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.959 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:27.959 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:27.959 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:27.980 INFO:tasks.rados.rados.0.smithi107.stdout:6280: writing smithi10729407-5 from 770048 to 1482752 tid 2 2020-06-12T02:38:27.980 INFO:tasks.rados.rados.0.smithi107.stdout:6280: writing smithi10729407-5 from 1482752 to 1916928 tid 3 2020-06-12T02:38:27.981 INFO:tasks.rados.rados.0.smithi107.stdout:6280: writing smithi10729407-5 from 1916928 to 1974272 tid 4 2020-06-12T02:38:27.981 INFO:tasks.rados.rados.0.smithi107.stdout:6281: delete oid 9 current snap is 0 2020-06-12T02:38:27.981 INFO:tasks.rados.rados.0.smithi107.stdout:6278: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:27.981 INFO:tasks.rados.rados.0.smithi107.stdout:6278: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:27.981 INFO:tasks.rados.rados.0.smithi107.stdout:6278: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:27.982 INFO:tasks.rados.rados.0.smithi107.stdout:6278: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:27.982 INFO:tasks.rados.rados.0.smithi107.stdout:6281: done (3 left) 2020-06-12T02:38:27.982 INFO:tasks.rados.rados.0.smithi107.stdout:6282: delete oid 19 current snap is 0 2020-06-12T02:38:27.982 INFO:tasks.rados.rados.0.smithi107.stdout:6278: finishing write tid 7 to smithi10729407-15 2020-06-12T02:38:27.982 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1159 (ObjNum 6277 snap 0 seq_num 6277) dirty exists 2020-06-12T02:38:27.982 INFO:tasks.rados.rados.0.smithi107.stdout:6278: left oid 15 (ObjNum 6277 snap 0 seq_num 6277) 2020-06-12T02:38:27.983 INFO:tasks.rados.rados.0.smithi107.stdout:6278: done (3 left) 2020-06-12T02:38:27.983 INFO:tasks.rados.rados.0.smithi107.stdout:6282: done (2 left) 2020-06-12T02:38:27.983 INFO:tasks.rados.rados.0.smithi107.stdout:6283: delete oid 7 current snap is 0 2020-06-12T02:38:27.983 INFO:tasks.rados.rados.0.smithi107.stdout:6283: done (2 left) 2020-06-12T02:38:27.983 INFO:tasks.rados.rados.0.smithi107.stdout:6284: delete oid 2 current snap is 0 2020-06-12T02:38:27.984 INFO:tasks.rados.rados.0.smithi107.stdout:6280: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:27.984 INFO:tasks.rados.rados.0.smithi107.stdout:6280: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:27.984 INFO:tasks.rados.rados.0.smithi107.stdout:6280: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:27.984 INFO:tasks.rados.rados.0.smithi107.stdout:6280: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:27.984 INFO:tasks.rados.rados.0.smithi107.stdout:6280: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:27.984 INFO:tasks.rados.rados.0.smithi107.stdout:6284: done (2 left) 2020-06-12T02:38:27.985 INFO:tasks.rados.rados.0.smithi107.stdout:6285: delete oid 18 current snap is 0 2020-06-12T02:38:27.985 INFO:tasks.rados.rados.0.smithi107.stdout:6280: finishing write tid 7 to smithi10729407-5 2020-06-12T02:38:27.985 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 633 (ObjNum 6279 snap 0 seq_num 6279) dirty exists 2020-06-12T02:38:27.985 INFO:tasks.rados.rados.0.smithi107.stdout:6280: left oid 5 (ObjNum 6279 snap 0 seq_num 6279) 2020-06-12T02:38:27.985 INFO:tasks.rados.rados.0.smithi107.stdout:6280: done (2 left) 2020-06-12T02:38:27.986 INFO:tasks.rados.rados.0.smithi107.stdout:6285: done (1 left) 2020-06-12T02:38:27.986 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:27.986 INFO:tasks.rados.rados.0.smithi107.stdout:6286: seq_num 6285 ranges {0=466944,466944=466944,933888=778240,1712128=360448} 2020-06-12T02:38:27.986 INFO:tasks.rados.rados.0.smithi107.stdout:6286: writing smithi10729407-18 from 0 to 466944 tid 1 2020-06-12T02:38:27.986 INFO:tasks.rados.rados.0.smithi107.stdout:6286: writing smithi10729407-18 from 466944 to 933888 tid 2 2020-06-12T02:38:27.986 INFO:tasks.rados.rados.0.smithi107.stdout:6286: writing smithi10729407-18 from 933888 to 1712128 tid 3 2020-06-12T02:38:27.987 INFO:tasks.rados.rados.0.smithi107.stdout:6286: writing smithi10729407-18 from 1712128 to 2072576 tid 4 2020-06-12T02:38:27.987 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:27.987 INFO:tasks.rados.rados.0.smithi107.stdout:6287: seq_num 6286 ranges {0=483328,483328=753664,1236992=589824,1826816=262144} 2020-06-12T02:38:27.987 INFO:tasks.rados.rados.0.smithi107.stdout:6287: writing smithi10729407-9 from 0 to 483328 tid 1 2020-06-12T02:38:27.987 INFO:tasks.rados.rados.0.smithi107.stdout:6287: writing smithi10729407-9 from 483328 to 1236992 tid 2 2020-06-12T02:38:27.988 INFO:tasks.rados.rados.0.smithi107.stdout:6287: writing smithi10729407-9 from 1236992 to 1826816 tid 3 2020-06-12T02:38:27.988 INFO:tasks.rados.rados.0.smithi107.stdout:6287: writing smithi10729407-9 from 1826816 to 2088960 tid 4 2020-06-12T02:38:27.988 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:27.988 INFO:tasks.rados.rados.0.smithi107.stdout:6288: seq_num 6287 ranges {0=499712,499712=638976,1138688=794624,1933312=172032} 2020-06-12T02:38:27.988 INFO:tasks.rados.rados.0.smithi107.stdout:6288: writing smithi10729407-12 from 0 to 499712 tid 1 2020-06-12T02:38:27.988 INFO:tasks.rados.rados.0.smithi107.stdout:6288: writing smithi10729407-12 from 499712 to 1138688 tid 2 2020-06-12T02:38:27.989 INFO:tasks.rados.rados.0.smithi107.stdout:6288: writing smithi10729407-12 from 1138688 to 1933312 tid 3 2020-06-12T02:38:27.989 INFO:tasks.rados.rados.0.smithi107.stdout:6288: writing smithi10729407-12 from 1933312 to 2105344 tid 4 2020-06-12T02:38:27.989 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:27.989 INFO:tasks.rados.rados.0.smithi107.stdout:6289: seq_num 6288 ranges {0=516096,516096=573440,1089536=606208,1695744=425984} 2020-06-12T02:38:27.989 INFO:tasks.rados.rados.0.smithi107.stdout:6289: writing smithi10729407-17 from 0 to 516096 tid 1 2020-06-12T02:38:27.990 INFO:tasks.rados.rados.0.smithi107.stdout:6289: writing smithi10729407-17 from 516096 to 1089536 tid 2 2020-06-12T02:38:27.990 INFO:tasks.rados.rados.0.smithi107.stdout:6289: writing smithi10729407-17 from 1089536 to 1695744 tid 3 2020-06-12T02:38:27.990 INFO:tasks.rados.rados.0.smithi107.stdout:6289: writing smithi10729407-17 from 1695744 to 2121728 tid 4 2020-06-12T02:38:27.990 INFO:tasks.rados.rados.0.smithi107.stdout:6286: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:27.990 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:27.990 INFO:tasks.rados.rados.0.smithi107.stdout:6290: seq_num 6289 ranges {0=532480,532480=450560,983040=409600,1392640=434176,1826816=311296} 2020-06-12T02:38:27.991 INFO:tasks.rados.rados.0.smithi107.stdout:6290: writing smithi10729407-11 from 0 to 532480 tid 1 2020-06-12T02:38:27.991 INFO:tasks.rados.rados.0.smithi107.stdout:6290: writing smithi10729407-11 from 532480 to 983040 tid 2 2020-06-12T02:38:27.991 INFO:tasks.rados.rados.0.smithi107.stdout:6290: writing smithi10729407-11 from 983040 to 1392640 tid 3 2020-06-12T02:38:27.991 INFO:tasks.rados.rados.0.smithi107.stdout:6290: writing smithi10729407-11 from 1392640 to 1826816 tid 4 2020-06-12T02:38:27.991 INFO:tasks.rados.rados.0.smithi107.stdout:6290: writing smithi10729407-11 from 1826816 to 2138112 tid 5 2020-06-12T02:38:27.992 INFO:tasks.rados.rados.0.smithi107.stdout:6291: delete oid 19 current snap is 0 2020-06-12T02:38:27.992 INFO:tasks.rados.rados.0.smithi107.stdout:6286: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:27.992 INFO:tasks.rados.rados.0.smithi107.stdout:6286: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:27.992 INFO:tasks.rados.rados.0.smithi107.stdout:6286: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:27.992 INFO:tasks.rados.rados.0.smithi107.stdout:6286: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:27.992 INFO:tasks.rados.rados.0.smithi107.stdout:6287: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:27.993 INFO:tasks.rados.rados.0.smithi107.stdout:6286: finishing write tid 7 to smithi10729407-18 2020-06-12T02:38:27.993 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1138 (ObjNum 6285 snap 0 seq_num 6285) dirty exists 2020-06-12T02:38:27.993 INFO:tasks.rados.rados.0.smithi107.stdout:6286: left oid 18 (ObjNum 6285 snap 0 seq_num 6285) 2020-06-12T02:38:27.993 INFO:tasks.rados.rados.0.smithi107.stdout:6287: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:27.993 INFO:tasks.rados.rados.0.smithi107.stdout:6287: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:27.994 INFO:tasks.rados.rados.0.smithi107.stdout:6288: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:27.994 INFO:tasks.rados.rados.0.smithi107.stdout:6287: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:27.994 INFO:tasks.rados.rados.0.smithi107.stdout:6287: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:27.994 INFO:tasks.rados.rados.0.smithi107.stdout:6287: finishing write tid 7 to smithi10729407-9 2020-06-12T02:38:27.994 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1143 (ObjNum 6286 snap 0 seq_num 6286) dirty exists 2020-06-12T02:38:27.994 INFO:tasks.rados.rados.0.smithi107.stdout:6287: left oid 9 (ObjNum 6286 snap 0 seq_num 6286) 2020-06-12T02:38:27.995 INFO:tasks.rados.rados.0.smithi107.stdout:6288: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:27.995 INFO:tasks.rados.rados.0.smithi107.stdout:6288: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:27.995 INFO:tasks.rados.rados.0.smithi107.stdout:6288: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:27.995 INFO:tasks.rados.rados.0.smithi107.stdout:6288: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:27.995 INFO:tasks.rados.rados.0.smithi107.stdout:6288: finishing write tid 7 to smithi10729407-12 2020-06-12T02:38:27.996 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1139 (ObjNum 6287 snap 0 seq_num 6287) dirty exists 2020-06-12T02:38:27.996 INFO:tasks.rados.rados.0.smithi107.stdout:6288: left oid 12 (ObjNum 6287 snap 0 seq_num 6287) 2020-06-12T02:38:27.996 INFO:tasks.rados.rados.0.smithi107.stdout:6289: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:27.996 INFO:tasks.rados.rados.0.smithi107.stdout:6289: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:27.996 INFO:tasks.rados.rados.0.smithi107.stdout:6289: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:27.996 INFO:tasks.rados.rados.0.smithi107.stdout:6289: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:27.997 INFO:tasks.rados.rados.0.smithi107.stdout:6289: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:27.997 INFO:tasks.rados.rados.0.smithi107.stdout:6289: finishing write tid 7 to smithi10729407-17 2020-06-12T02:38:27.997 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1172 (ObjNum 6288 snap 0 seq_num 6288) dirty exists 2020-06-12T02:38:27.997 INFO:tasks.rados.rados.0.smithi107.stdout:6289: left oid 17 (ObjNum 6288 snap 0 seq_num 6288) 2020-06-12T02:38:27.997 INFO:tasks.rados.rados.0.smithi107.stdout:6290: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:27.998 INFO:tasks.rados.rados.0.smithi107.stdout:6290: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:27.998 INFO:tasks.rados.rados.0.smithi107.stdout:6286: done (6 left) 2020-06-12T02:38:27.998 INFO:tasks.rados.rados.0.smithi107.stdout:6287: done (5 left) 2020-06-12T02:38:27.998 INFO:tasks.rados.rados.0.smithi107.stdout:6288: done (4 left) 2020-06-12T02:38:27.998 INFO:tasks.rados.rados.0.smithi107.stdout:6289: done (3 left) 2020-06-12T02:38:27.998 INFO:tasks.rados.rados.0.smithi107.stdout:6291: done (2 left) 2020-06-12T02:38:27.999 INFO:tasks.rados.rados.0.smithi107.stdout:6292: delete oid 20 current snap is 0 2020-06-12T02:38:27.999 INFO:tasks.rados.rados.0.smithi107.stdout:6290: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:27.999 INFO:tasks.rados.rados.0.smithi107.stdout:6290: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:27.999 INFO:tasks.rados.rados.0.smithi107.stdout:6292: done (2 left) 2020-06-12T02:38:27.999 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:28.000 INFO:tasks.rados.rados.0.smithi107.stdout:6293: seq_num 6292 ranges {1064960=581632,1646592=507904,2154496=581632,2736128=516096} 2020-06-12T02:38:28.000 INFO:tasks.rados.rados.0.smithi107.stdout:6293: writing smithi10729407-3 from 1064960 to 1646592 tid 1 2020-06-12T02:38:28.000 INFO:tasks.rados.rados.0.smithi107.stdout:6293: writing smithi10729407-3 from 1646592 to 2154496 tid 2 2020-06-12T02:38:28.000 INFO:tasks.rados.rados.0.smithi107.stdout:6293: writing smithi10729407-3 from 2154496 to 2736128 tid 3 2020-06-12T02:38:28.000 INFO:tasks.rados.rados.0.smithi107.stdout:6293: writing smithi10729407-3 from 2736128 to 3252224 tid 4 2020-06-12T02:38:28.000 INFO:tasks.rados.rados.0.smithi107.stdout:6294: delete oid 8 current snap is 0 2020-06-12T02:38:28.001 INFO:tasks.rados.rados.0.smithi107.stdout:6290: finishing write tid 5 to smithi10729407-11 2020-06-12T02:38:28.001 INFO:tasks.rados.rados.0.smithi107.stdout:6290: finishing write tid 7 to smithi10729407-11 2020-06-12T02:38:28.001 INFO:tasks.rados.rados.0.smithi107.stdout:6290: finishing write tid 8 to smithi10729407-11 2020-06-12T02:38:28.001 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1124 (ObjNum 6289 snap 0 seq_num 6289) dirty exists 2020-06-12T02:38:28.001 INFO:tasks.rados.rados.0.smithi107.stdout:6290: left oid 11 (ObjNum 6289 snap 0 seq_num 6289) 2020-06-12T02:38:28.002 INFO:tasks.rados.rados.0.smithi107.stdout:6293: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:28.002 INFO:tasks.rados.rados.0.smithi107.stdout:6293: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:28.002 INFO:tasks.rados.rados.0.smithi107.stdout:6290: done (3 left) 2020-06-12T02:38:28.002 INFO:tasks.rados.rados.0.smithi107.stdout:6294: done (2 left) 2020-06-12T02:38:28.002 INFO:tasks.rados.rados.0.smithi107.stdout:6295: delete oid 17 current snap is 0 2020-06-12T02:38:28.002 INFO:tasks.rados.rados.0.smithi107.stdout:6295: done (2 left) 2020-06-12T02:38:28.003 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:28.003 INFO:tasks.rados.rados.0.smithi107.stdout:6296: seq_num 6295 ranges {0=638976,638976=606208,1245184=753664,1998848=245760} 2020-06-12T02:38:28.003 INFO:tasks.rados.rados.0.smithi107.stdout:6296: writing smithi10729407-19 from 0 to 638976 tid 1 2020-06-12T02:38:28.003 INFO:tasks.rados.rados.0.smithi107.stdout:6296: writing smithi10729407-19 from 638976 to 1245184 tid 2 2020-06-12T02:38:28.003 INFO:tasks.rados.rados.0.smithi107.stdout:6296: writing smithi10729407-19 from 1245184 to 1998848 tid 3 2020-06-12T02:38:28.003 INFO:tasks.rados.rados.0.smithi107.stdout:6296: writing smithi10729407-19 from 1998848 to 2244608 tid 4 2020-06-12T02:38:28.004 INFO:tasks.rados.rados.0.smithi107.stdout:6297: delete oid 15 current snap is 0 2020-06-12T02:38:28.004 INFO:tasks.rados.rados.0.smithi107.stdout:6293: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:28.004 INFO:tasks.rados.rados.0.smithi107.stdout:6293: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:28.004 INFO:tasks.rados.rados.0.smithi107.stdout:6293: finishing write tid 6 to smithi10729407-3 2020-06-12T02:38:28.004 INFO:tasks.rados.rados.0.smithi107.stdout:6293: finishing write tid 7 to smithi10729407-3 2020-06-12T02:38:28.005 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1112 (ObjNum 6292 snap 0 seq_num 6292) dirty exists 2020-06-12T02:38:28.005 INFO:tasks.rados.rados.0.smithi107.stdout:6293: left oid 3 (ObjNum 6292 snap 0 seq_num 6292) 2020-06-12T02:38:28.005 INFO:tasks.rados.rados.0.smithi107.stdout:6296: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:28.005 INFO:tasks.rados.rados.0.smithi107.stdout:6293: done (3 left) 2020-06-12T02:38:28.005 INFO:tasks.rados.rados.0.smithi107.stdout:6297: done (2 left) 2020-06-12T02:38:28.005 INFO:tasks.rados.rados.0.smithi107.stdout:6298: delete oid 7 current snap is 0 2020-06-12T02:38:28.006 INFO:tasks.rados.rados.0.smithi107.stdout:6296: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:28.006 INFO:tasks.rados.rados.0.smithi107.stdout:6296: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:28.006 INFO:tasks.rados.rados.0.smithi107.stdout:6298: done (2 left) 2020-06-12T02:38:28.006 INFO:tasks.rados.rados.0.smithi107.stdout:6299: delete oid 12 current snap is 0 2020-06-12T02:38:28.006 INFO:tasks.rados.rados.0.smithi107.stdout:6296: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:28.007 INFO:tasks.rados.rados.0.smithi107.stdout:6296: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:28.007 INFO:tasks.rados.rados.0.smithi107.stdout:6299: done (2 left) 2020-06-12T02:38:28.007 INFO:tasks.rados.rados.0.smithi107.stdout:6300: delete oid 13 current snap is 0 2020-06-12T02:38:28.007 INFO:tasks.rados.rados.0.smithi107.stdout:6296: finishing write tid 7 to smithi10729407-19 2020-06-12T02:38:28.007 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1145 (ObjNum 6295 snap 0 seq_num 6295) dirty exists 2020-06-12T02:38:28.007 INFO:tasks.rados.rados.0.smithi107.stdout:6296: left oid 19 (ObjNum 6295 snap 0 seq_num 6295) 2020-06-12T02:38:28.008 INFO:tasks.rados.rados.0.smithi107.stdout:6296: done (2 left) 2020-06-12T02:38:28.008 INFO:tasks.rados.rados.0.smithi107.stdout:6300: done (1 left) 2020-06-12T02:38:28.008 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:28.008 INFO:tasks.rados.rados.0.smithi107.stdout:6301: seq_num 6300 ranges {0=720896,720896=434176,1155072=540672,1695744=630784} 2020-06-12T02:38:28.008 INFO:tasks.rados.rados.0.smithi107.stdout:6301: writing smithi10729407-7 from 0 to 720896 tid 1 2020-06-12T02:38:28.008 INFO:tasks.rados.rados.0.smithi107.stdout:6301: writing smithi10729407-7 from 720896 to 1155072 tid 2 2020-06-12T02:38:28.009 INFO:tasks.rados.rados.0.smithi107.stdout:6301: writing smithi10729407-7 from 1155072 to 1695744 tid 3 2020-06-12T02:38:28.009 INFO:tasks.rados.rados.0.smithi107.stdout:6301: writing smithi10729407-7 from 1695744 to 2326528 tid 4 2020-06-12T02:38:28.009 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:28.009 INFO:tasks.rados.rados.0.smithi107.stdout:6302: seq_num 6301 ranges {2072576=737280,2809856=720896,3530752=753664,4284416=131072} 2020-06-12T02:38:28.009 INFO:tasks.rados.rados.0.smithi107.stdout:6302: writing smithi10729407-18 from 2072576 to 2809856 tid 1 2020-06-12T02:38:28.010 INFO:tasks.rados.rados.0.smithi107.stdout:6302: writing smithi10729407-18 from 2809856 to 3530752 tid 2 2020-06-12T02:38:28.010 INFO:tasks.rados.rados.0.smithi107.stdout:6302: writing smithi10729407-18 from 3530752 to 4284416 tid 3 2020-06-12T02:38:28.010 INFO:tasks.rados.rados.0.smithi107.stdout:6302: writing smithi10729407-18 from 4284416 to 4415488 tid 4 2020-06-12T02:38:28.010 INFO:tasks.rados.rados.0.smithi107.stdout:6303: delete oid 15 current snap is 0 2020-06-12T02:38:28.010 INFO:tasks.rados.rados.0.smithi107.stdout:6301: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:28.011 INFO:tasks.rados.rados.0.smithi107.stdout:6301: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:28.011 INFO:tasks.rados.rados.0.smithi107.stdout:6301: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:28.011 INFO:tasks.rados.rados.0.smithi107.stdout:6301: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:28.011 INFO:tasks.rados.rados.0.smithi107.stdout:6301: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:28.011 INFO:tasks.rados.rados.0.smithi107.stdout:6301: finishing write tid 7 to smithi10729407-7 2020-06-12T02:38:28.011 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1201 (ObjNum 6300 snap 0 seq_num 6300) dirty exists 2020-06-12T02:38:28.012 INFO:tasks.rados.rados.0.smithi107.stdout:6301: left oid 7 (ObjNum 6300 snap 0 seq_num 6300) 2020-06-12T02:38:28.012 INFO:tasks.rados.rados.0.smithi107.stdout:6302: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:28.012 INFO:tasks.rados.rados.0.smithi107.stdout:6301: done (3 left) 2020-06-12T02:38:28.012 INFO:tasks.rados.rados.0.smithi107.stdout:6303: done (2 left) 2020-06-12T02:38:28.012 INFO:tasks.rados.rados.0.smithi107.stdout:6304: delete oid 6 current snap is 0 2020-06-12T02:38:28.013 INFO:tasks.rados.rados.0.smithi107.stdout:6304: done (2 left) 2020-06-12T02:38:28.013 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:28.013 INFO:tasks.rados.rados.0.smithi107.stdout:6305: seq_num 6304 ranges {0=786432,786432=417792,1204224=524288,1728512=663552} 2020-06-12T02:38:28.013 INFO:tasks.rados.rados.0.smithi107.stdout:6305: writing smithi10729407-17 from 0 to 786432 tid 1 2020-06-12T02:38:28.013 INFO:tasks.rados.rados.0.smithi107.stdout:6305: writing smithi10729407-17 from 786432 to 1204224 tid 2 2020-06-12T02:38:28.013 INFO:tasks.rados.rados.0.smithi107.stdout:6305: writing smithi10729407-17 from 1204224 to 1728512 tid 3 2020-06-12T02:38:28.014 INFO:tasks.rados.rados.0.smithi107.stdout:6305: writing smithi10729407-17 from 1728512 to 2392064 tid 4 2020-06-12T02:38:28.014 INFO:tasks.rados.rados.0.smithi107.stdout:6306: delete oid 2 current snap is 0 2020-06-12T02:38:28.014 INFO:tasks.rados.rados.0.smithi107.stdout:6302: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:28.014 INFO:tasks.rados.rados.0.smithi107.stdout:6302: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:28.014 INFO:tasks.rados.rados.0.smithi107.stdout:6302: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:28.015 INFO:tasks.rados.rados.0.smithi107.stdout:6302: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:28.015 INFO:tasks.rados.rados.0.smithi107.stdout:6302: finishing write tid 7 to smithi10729407-18 2020-06-12T02:38:28.015 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1148 (ObjNum 6301 snap 0 seq_num 6301) dirty exists 2020-06-12T02:38:28.015 INFO:tasks.rados.rados.0.smithi107.stdout:6302: left oid 18 (ObjNum 6301 snap 0 seq_num 6301) 2020-06-12T02:38:28.015 INFO:tasks.rados.rados.0.smithi107.stdout:6305: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:28.015 INFO:tasks.rados.rados.0.smithi107.stdout:6302: done (3 left) 2020-06-12T02:38:28.016 INFO:tasks.rados.rados.0.smithi107.stdout:6306: done (2 left) 2020-06-12T02:38:28.016 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:28.016 INFO:tasks.rados.rados.0.smithi107.stdout:6307: seq_num 6306 ranges {2138112=16384} 2020-06-12T02:38:28.016 INFO:tasks.rados.rados.0.smithi107.stdout:6307: writing smithi10729407-11 from 2138112 to 2154496 tid 1 2020-06-12T02:38:28.016 INFO:tasks.rados.rados.0.smithi107.stdout:6308: delete oid 4 current snap is 0 2020-06-12T02:38:28.017 INFO:tasks.rados.rados.0.smithi107.stdout:6305: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:28.017 INFO:tasks.rados.rados.0.smithi107.stdout:6305: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:28.017 INFO:tasks.rados.rados.0.smithi107.stdout:6305: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:28.017 INFO:tasks.rados.rados.0.smithi107.stdout:6305: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:28.017 INFO:tasks.rados.rados.0.smithi107.stdout:6305: finishing write tid 7 to smithi10729407-17 2020-06-12T02:38:28.017 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1178 (ObjNum 6304 snap 0 seq_num 6304) dirty exists 2020-06-12T02:38:28.018 INFO:tasks.rados.rados.0.smithi107.stdout:6305: left oid 17 (ObjNum 6304 snap 0 seq_num 6304) 2020-06-12T02:38:28.018 INFO:tasks.rados.rados.0.smithi107.stdout:6305: done (3 left) 2020-06-12T02:38:28.018 INFO:tasks.rados.rados.0.smithi107.stdout:6308: done (2 left) 2020-06-12T02:38:28.018 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:28.018 INFO:tasks.rados.rados.0.smithi107.stdout:6309: seq_num 6308 ranges {0=49152} 2020-06-12T02:38:28.019 INFO:tasks.rados.rados.0.smithi107.stdout:6309: writing smithi10729407-1 from 0 to 49152 tid 1 2020-06-12T02:38:28.019 INFO:tasks.rados.rados.0.smithi107.stdout:6310: delete oid 5 current snap is 0 2020-06-12T02:38:28.019 INFO:tasks.rados.rados.0.smithi107.stdout:6307: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:28.019 INFO:tasks.rados.rados.0.smithi107.stdout:6307: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:28.019 INFO:tasks.rados.rados.0.smithi107.stdout:6310: done (3 left) 2020-06-12T02:38:28.019 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:28.020 INFO:tasks.rados.rados.0.smithi107.stdout:6311: seq_num 6310 ranges {0=81920} 2020-06-12T02:38:28.020 INFO:tasks.rados.rados.0.smithi107.stdout:6311: writing smithi10729407-8 from 0 to 81920 tid 1 2020-06-12T02:38:28.020 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:28.020 INFO:tasks.rados.rados.0.smithi107.stdout:6312: seq_num 6311 ranges {2088960=98304} 2020-06-12T02:38:28.020 INFO:tasks.rados.rados.0.smithi107.stdout:6312: writing smithi10729407-9 from 2088960 to 2187264 tid 1 2020-06-12T02:38:28.020 INFO:tasks.rados.rados.0.smithi107.stdout:6313: delete oid 17 current snap is 0 2020-06-12T02:38:28.021 INFO:tasks.rados.rados.0.smithi107.stdout:6307: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:28.021 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1126 (ObjNum 6306 snap 0 seq_num 6306) dirty exists 2020-06-12T02:38:28.021 INFO:tasks.rados.rados.0.smithi107.stdout:6307: left oid 11 (ObjNum 6306 snap 0 seq_num 6306) 2020-06-12T02:38:28.021 INFO:tasks.rados.rados.0.smithi107.stdout:6309: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:28.021 INFO:tasks.rados.rados.0.smithi107.stdout:6311: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:28.022 INFO:tasks.rados.rados.0.smithi107.stdout:6311: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:28.022 INFO:tasks.rados.rados.0.smithi107.stdout:6309: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:28.022 INFO:tasks.rados.rados.0.smithi107.stdout:6309: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:28.022 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1163 (ObjNum 6308 snap 0 seq_num 6308) dirty exists 2020-06-12T02:38:28.022 INFO:tasks.rados.rados.0.smithi107.stdout:6309: left oid 1 (ObjNum 6308 snap 0 seq_num 6308) 2020-06-12T02:38:28.022 INFO:tasks.rados.rados.0.smithi107.stdout:6311: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:28.023 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1164 (ObjNum 6310 snap 0 seq_num 6310) dirty exists 2020-06-12T02:38:28.023 INFO:tasks.rados.rados.0.smithi107.stdout:6311: left oid 8 (ObjNum 6310 snap 0 seq_num 6310) 2020-06-12T02:38:28.023 INFO:tasks.rados.rados.0.smithi107.stdout:6307: done (5 left) 2020-06-12T02:38:28.023 INFO:tasks.rados.rados.0.smithi107.stdout:6309: done (4 left) 2020-06-12T02:38:28.023 INFO:tasks.rados.rados.0.smithi107.stdout:6311: done (3 left) 2020-06-12T02:38:28.024 INFO:tasks.rados.rados.0.smithi107.stdout:6313: done (2 left) 2020-06-12T02:38:28.024 INFO:tasks.rados.rados.0.smithi107.stdout:6314: delete oid 8 current snap is 0 2020-06-12T02:38:28.024 INFO:tasks.rados.rados.0.smithi107.stdout:6312: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:28.024 INFO:tasks.rados.rados.0.smithi107.stdout:6312: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:28.024 INFO:tasks.rados.rados.0.smithi107.stdout:6314: done (2 left) 2020-06-12T02:38:28.024 INFO:tasks.rados.rados.0.smithi107.stdout:6315: delete oid 14 current snap is 0 2020-06-12T02:38:28.025 INFO:tasks.rados.rados.0.smithi107.stdout:6312: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:28.025 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1150 (ObjNum 6311 snap 0 seq_num 6311) dirty exists 2020-06-12T02:38:28.025 INFO:tasks.rados.rados.0.smithi107.stdout:6312: left oid 9 (ObjNum 6311 snap 0 seq_num 6311) 2020-06-12T02:38:28.025 INFO:tasks.rados.rados.0.smithi107.stdout:6312: done (2 left) 2020-06-12T02:38:28.025 INFO:tasks.rados.rados.0.smithi107.stdout:6315: done (1 left) 2020-06-12T02:38:28.026 INFO:tasks.rados.rados.0.smithi107.stdout:6316: delete oid 6 current snap is 0 2020-06-12T02:38:28.026 INFO:tasks.rados.rados.0.smithi107.stdout:6316: done (1 left) 2020-06-12T02:38:28.026 INFO:tasks.rados.rados.0.smithi107.stdout:6317: delete oid 12 current snap is 0 2020-06-12T02:38:28.026 INFO:tasks.rados.rados.0.smithi107.stdout:6317: done (1 left) 2020-06-12T02:38:28.026 INFO:tasks.rados.rados.0.smithi107.stdout:6318: delete oid 15 current snap is 0 2020-06-12T02:38:28.027 INFO:tasks.rados.rados.0.smithi107.stdout:6318: done (1 left) 2020-06-12T02:38:28.027 INFO:tasks.rados.rados.0.smithi107.stdout:6319: delete oid 3 current snap is 0 2020-06-12T02:38:28.028 INFO:tasks.rados.rados.0.smithi107.stdout:6319: done (1 left) 2020-06-12T02:38:28.028 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:28.028 INFO:tasks.rados.rados.0.smithi107.stdout:6320: seq_num 6319 ranges {0=237568} 2020-06-12T02:38:28.028 INFO:tasks.rados.rados.0.smithi107.stdout:6320: writing smithi10729407-5 from 0 to 237568 tid 1 2020-06-12T02:38:28.028 INFO:tasks.rados.rados.0.smithi107.stdout:6321: delete oid 12 current snap is 0 2020-06-12T02:38:28.029 INFO:tasks.rados.rados.0.smithi107.stdout:6321: done (2 left) 2020-06-12T02:38:28.029 INFO:tasks.rados.rados.0.smithi107.stdout:6322: delete oid 19 current snap is 0 2020-06-12T02:38:28.029 INFO:tasks.rados.rados.0.smithi107.stdout:6320: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:28.029 INFO:tasks.rados.rados.0.smithi107.stdout:6320: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:28.029 INFO:tasks.rados.rados.0.smithi107.stdout:6322: done (2 left) 2020-06-12T02:38:28.030 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:38:28.030 INFO:tasks.rados.rados.0.smithi107.stdout:6323: seq_num 6322 ranges {0=286720} 2020-06-12T02:38:28.030 INFO:tasks.rados.rados.0.smithi107.stdout:6323: writing smithi10729407-3 from 0 to 286720 tid 1 2020-06-12T02:38:28.030 INFO:tasks.rados.rados.0.smithi107.stdout:6324: delete oid 9 current snap is 0 2020-06-12T02:38:28.030 INFO:tasks.rados.rados.0.smithi107.stdout:6320: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:28.030 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 636 (ObjNum 6319 snap 0 seq_num 6319) dirty exists 2020-06-12T02:38:28.031 INFO:tasks.rados.rados.0.smithi107.stdout:6320: left oid 5 (ObjNum 6319 snap 0 seq_num 6319) 2020-06-12T02:38:28.031 INFO:tasks.rados.rados.0.smithi107.stdout:6320: done (3 left) 2020-06-12T02:38:28.031 INFO:tasks.rados.rados.0.smithi107.stdout:6324: done (2 left) 2020-06-12T02:38:28.031 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:28.031 INFO:tasks.rados.rados.0.smithi107.stdout:6325: seq_num 6324 ranges {0=319488} 2020-06-12T02:38:28.032 INFO:tasks.rados.rados.0.smithi107.stdout:6325: writing smithi10729407-17 from 0 to 319488 tid 1 2020-06-12T02:38:28.032 INFO:tasks.rados.rados.0.smithi107.stdout:6326: delete oid 7 current snap is 0 2020-06-12T02:38:28.032 INFO:tasks.rados.rados.0.smithi107.stdout:6323: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:28.032 INFO:tasks.rados.rados.0.smithi107.stdout:6323: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:28.032 INFO:tasks.rados.rados.0.smithi107.stdout:6323: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:28.032 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1115 (ObjNum 6322 snap 0 seq_num 6322) dirty exists 2020-06-12T02:38:28.033 INFO:tasks.rados.rados.0.smithi107.stdout:6323: left oid 3 (ObjNum 6322 snap 0 seq_num 6322) 2020-06-12T02:38:28.033 INFO:tasks.rados.rados.0.smithi107.stdout:6323: done (3 left) 2020-06-12T02:38:28.033 INFO:tasks.rados.rados.0.smithi107.stdout:6326: done (2 left) 2020-06-12T02:38:28.033 INFO:tasks.rados.rados.0.smithi107.stdout:6327: delete oid 6 current snap is 0 2020-06-12T02:38:28.033 INFO:tasks.rados.rados.0.smithi107.stdout:6325: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:28.033 INFO:tasks.rados.rados.0.smithi107.stdout:6325: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:28.034 INFO:tasks.rados.rados.0.smithi107.stdout:6325: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:28.034 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1182 (ObjNum 6324 snap 0 seq_num 6324) dirty exists 2020-06-12T02:38:28.034 INFO:tasks.rados.rados.0.smithi107.stdout:6325: left oid 17 (ObjNum 6324 snap 0 seq_num 6324) 2020-06-12T02:38:28.034 INFO:tasks.rados.rados.0.smithi107.stdout:6325: done (2 left) 2020-06-12T02:38:28.034 INFO:tasks.rados.rados.0.smithi107.stdout:6327: done (1 left) 2020-06-12T02:38:28.035 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:28.035 INFO:tasks.rados.rados.0.smithi107.stdout:6328: seq_num 6327 ranges {0=368640} 2020-06-12T02:38:28.035 INFO:tasks.rados.rados.0.smithi107.stdout:6328: writing smithi10729407-12 from 0 to 368640 tid 1 2020-06-12T02:38:28.035 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:28.035 INFO:tasks.rados.rados.0.smithi107.stdout:6329: seq_num 6328 ranges {237568=385024} 2020-06-12T02:38:28.035 INFO:tasks.rados.rados.0.smithi107.stdout:6329: writing smithi10729407-5 from 237568 to 622592 tid 1 2020-06-12T02:38:28.036 INFO:tasks.rados.rados.0.smithi107.stdout:6330: delete oid 3 current snap is 0 2020-06-12T02:38:28.036 INFO:tasks.rados.rados.0.smithi107.stdout:6328: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:28.036 INFO:tasks.rados.rados.0.smithi107.stdout:6330: done (3 left) 2020-06-12T02:38:28.036 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:28.036 INFO:tasks.rados.rados.0.smithi107.stdout:6331: seq_num 6330 ranges {0=417792} 2020-06-12T02:38:28.037 INFO:tasks.rados.rados.0.smithi107.stdout:6331: writing smithi10729407-8 from 0 to 417792 tid 1 2020-06-12T02:38:28.037 INFO:tasks.rados.rados.0.smithi107.stdout:6332: delete oid 7 current snap is 0 2020-06-12T02:38:28.037 INFO:tasks.rados.rados.0.smithi107.stdout:6328: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:28.037 INFO:tasks.rados.rados.0.smithi107.stdout:6329: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:28.037 INFO:tasks.rados.rados.0.smithi107.stdout:6328: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:28.037 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1151 (ObjNum 6327 snap 0 seq_num 6327) dirty exists 2020-06-12T02:38:28.038 INFO:tasks.rados.rados.0.smithi107.stdout:6328: left oid 12 (ObjNum 6327 snap 0 seq_num 6327) 2020-06-12T02:38:28.038 INFO:tasks.rados.rados.0.smithi107.stdout:6329: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:28.038 INFO:tasks.rados.rados.0.smithi107.stdout:6328: done (4 left) 2020-06-12T02:38:28.038 INFO:tasks.rados.rados.0.smithi107.stdout:6332: done (3 left) 2020-06-12T02:38:28.038 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:38:28.039 INFO:tasks.rados.rados.0.smithi107.stdout:6333: seq_num 6332 ranges {0=450560} 2020-06-12T02:38:28.039 INFO:tasks.rados.rados.0.smithi107.stdout:6333: writing smithi10729407-3 from 0 to 450560 tid 1 2020-06-12T02:38:28.039 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:28.039 INFO:tasks.rados.rados.0.smithi107.stdout:6334: seq_num 6333 ranges {0=475136} 2020-06-12T02:38:28.039 INFO:tasks.rados.rados.0.smithi107.stdout:6334: writing smithi10729407-13 from 0 to 475136 tid 1 2020-06-12T02:38:28.039 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:28.040 INFO:tasks.rados.rados.0.smithi107.stdout:6335: seq_num 6334 ranges {0=491520} 2020-06-12T02:38:28.040 INFO:tasks.rados.rados.0.smithi107.stdout:6335: writing smithi10729407-9 from 0 to 491520 tid 1 2020-06-12T02:38:28.040 INFO:tasks.rados.rados.0.smithi107.stdout:6336: delete oid 15 current snap is 0 2020-06-12T02:38:28.040 INFO:tasks.rados.rados.0.smithi107.stdout:6329: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:28.040 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 638 (ObjNum 6328 snap 0 seq_num 6328) dirty exists 2020-06-12T02:38:28.040 INFO:tasks.rados.rados.0.smithi107.stdout:6329: left oid 5 (ObjNum 6328 snap 0 seq_num 6328) 2020-06-12T02:38:28.041 INFO:tasks.rados.rados.0.smithi107.stdout:6331: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:28.041 INFO:tasks.rados.rados.0.smithi107.stdout:6331: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:28.041 INFO:tasks.rados.rados.0.smithi107.stdout:6333: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:28.041 INFO:tasks.rados.rados.0.smithi107.stdout:6333: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:28.041 INFO:tasks.rados.rados.0.smithi107.stdout:6331: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:28.042 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1168 (ObjNum 6330 snap 0 seq_num 6330) dirty exists 2020-06-12T02:38:28.042 INFO:tasks.rados.rados.0.smithi107.stdout:6331: left oid 8 (ObjNum 6330 snap 0 seq_num 6330) 2020-06-12T02:38:28.042 INFO:tasks.rados.rados.0.smithi107.stdout:6329: done (6 left) 2020-06-12T02:38:28.042 INFO:tasks.rados.rados.0.smithi107.stdout:6331: done (5 left) 2020-06-12T02:38:28.042 INFO:tasks.rados.rados.0.smithi107.stdout:6336: done (4 left) 2020-06-12T02:38:28.042 INFO:tasks.rados.rados.0.smithi107.stdout:6337: delete oid 4 current snap is 0 2020-06-12T02:38:28.043 INFO:tasks.rados.rados.0.smithi107.stdout:6334: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:28.043 INFO:tasks.rados.rados.0.smithi107.stdout:6335: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:28.043 INFO:tasks.rados.rados.0.smithi107.stdout:6333: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:28.043 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1118 (ObjNum 6332 snap 0 seq_num 6332) dirty exists 2020-06-12T02:38:28.043 INFO:tasks.rados.rados.0.smithi107.stdout:6333: left oid 3 (ObjNum 6332 snap 0 seq_num 6332) 2020-06-12T02:38:28.044 INFO:tasks.rados.rados.0.smithi107.stdout:6333: done (4 left) 2020-06-12T02:38:28.044 INFO:tasks.rados.rados.0.smithi107.stdout:6337: done (3 left) 2020-06-12T02:38:28.044 INFO:tasks.rados.rados.0.smithi107.stdout:6338: delete oid 18 current snap is 0 2020-06-12T02:38:28.044 INFO:tasks.rados.rados.0.smithi107.stdout:6334: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:28.044 INFO:tasks.rados.rados.0.smithi107.stdout:6334: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:28.044 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1205 (ObjNum 6333 snap 0 seq_num 6333) dirty exists 2020-06-12T02:38:28.045 INFO:tasks.rados.rados.0.smithi107.stdout:6334: left oid 13 (ObjNum 6333 snap 0 seq_num 6333) 2020-06-12T02:38:28.045 INFO:tasks.rados.rados.0.smithi107.stdout:6335: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:28.045 INFO:tasks.rados.rados.0.smithi107.stdout:6334: done (3 left) 2020-06-12T02:38:28.045 INFO:tasks.rados.rados.0.smithi107.stdout:6338: done (2 left) 2020-06-12T02:38:28.045 INFO:tasks.rados.rados.0.smithi107.stdout:6339: delete oid 6 current snap is 0 2020-06-12T02:38:28.046 INFO:tasks.rados.rados.0.smithi107.stdout:6339: done (2 left) 2020-06-12T02:38:28.046 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:28.046 INFO:tasks.rados.rados.0.smithi107.stdout:6340: seq_num 6339 ranges {622592=573440} 2020-06-12T02:38:28.046 INFO:tasks.rados.rados.0.smithi107.stdout:6340: writing smithi10729407-5 from 622592 to 1196032 tid 1 2020-06-12T02:38:28.046 INFO:tasks.rados.rados.0.smithi107.stdout:6341: delete oid 18 current snap is 0 2020-06-12T02:38:28.046 INFO:tasks.rados.rados.0.smithi107.stdout:6335: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:28.047 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1153 (ObjNum 6334 snap 0 seq_num 6334) dirty exists 2020-06-12T02:38:28.047 INFO:tasks.rados.rados.0.smithi107.stdout:6335: left oid 9 (ObjNum 6334 snap 0 seq_num 6334) 2020-06-12T02:38:28.047 INFO:tasks.rados.rados.0.smithi107.stdout:6335: done (3 left) 2020-06-12T02:38:28.047 INFO:tasks.rados.rados.0.smithi107.stdout:6341: done (2 left) 2020-06-12T02:38:28.047 INFO:tasks.rados.rados.0.smithi107.stdout:6342: delete oid 3 current snap is 0 2020-06-12T02:38:28.047 INFO:tasks.rados.rados.0.smithi107.stdout:6342: done (2 left) 2020-06-12T02:38:28.048 INFO:tasks.rados.rados.0.smithi107.stdout:6343: delete oid 15 current snap is 0 2020-06-12T02:38:28.048 INFO:tasks.rados.rados.0.smithi107.stdout:6340: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:28.048 INFO:tasks.rados.rados.0.smithi107.stdout:6343: done (2 left) 2020-06-12T02:38:28.048 INFO:tasks.rados.rados.0.smithi107.stdout:6344: delete oid 4 current snap is 0 2020-06-12T02:38:28.048 INFO:tasks.rados.rados.0.smithi107.stdout:6340: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:28.049 INFO:tasks.rados.rados.0.smithi107.stdout:6344: done (2 left) 2020-06-12T02:38:28.049 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:28.049 INFO:tasks.rados.rados.0.smithi107.stdout:6345: seq_num 6344 ranges {491520=655360} 2020-06-12T02:38:28.049 INFO:tasks.rados.rados.0.smithi107.stdout:6345: writing smithi10729407-9 from 491520 to 1146880 tid 1 2020-06-12T02:38:28.049 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:28.049 INFO:tasks.rados.rados.0.smithi107.stdout:6346: seq_num 6345 ranges {0=671744} 2020-06-12T02:38:28.050 INFO:tasks.rados.rados.0.smithi107.stdout:6346: writing smithi10729407-14 from 0 to 671744 tid 1 2020-06-12T02:38:28.050 INFO:tasks.rados.rados.0.smithi107.stdout:6347: delete oid 19 current snap is 0 2020-06-12T02:38:28.050 INFO:tasks.rados.rados.0.smithi107.stdout:6340: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:28.050 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 640 (ObjNum 6339 snap 0 seq_num 6339) dirty exists 2020-06-12T02:38:28.050 INFO:tasks.rados.rados.0.smithi107.stdout:6340: left oid 5 (ObjNum 6339 snap 0 seq_num 6339) 2020-06-12T02:38:28.051 INFO:tasks.rados.rados.0.smithi107.stdout:6345: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:28.051 INFO:tasks.rados.rados.0.smithi107.stdout:6345: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:28.051 INFO:tasks.rados.rados.0.smithi107.stdout:6340: done (4 left) 2020-06-12T02:38:28.051 INFO:tasks.rados.rados.0.smithi107.stdout:6347: done (3 left) 2020-06-12T02:38:28.051 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:38:28.051 INFO:tasks.rados.rados.0.smithi107.stdout:6348: seq_num 6347 ranges {1196032=704512} 2020-06-12T02:38:28.052 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:27.612+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:28.052 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-06-12T02:38:28.053 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:27.958+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:28.057 INFO:tasks.rados.rados.0.smithi107.stdout:6348: writing smithi10729407-5 from 1196032 to 1900544 tid 1 2020-06-12T02:38:28.057 INFO:tasks.rados.rados.0.smithi107.stdout:6349: delete oid 2 current snap is 0 2020-06-12T02:38:28.057 INFO:tasks.rados.rados.0.smithi107.stdout:6345: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:28.057 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1157 (ObjNum 6344 snap 0 seq_num 6344) dirty exists 2020-06-12T02:38:28.058 INFO:tasks.rados.rados.0.smithi107.stdout:6345: left oid 9 (ObjNum 6344 snap 0 seq_num 6344) 2020-06-12T02:38:28.058 INFO:tasks.rados.rados.0.smithi107.stdout:6346: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:28.058 INFO:tasks.rados.rados.0.smithi107.stdout:6346: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:28.058 INFO:tasks.rados.rados.0.smithi107.stdout:6346: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:28.058 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1129 (ObjNum 6345 snap 0 seq_num 6345) dirty exists 2020-06-12T02:38:28.058 INFO:tasks.rados.rados.0.smithi107.stdout:6346: left oid 14 (ObjNum 6345 snap 0 seq_num 6345) 2020-06-12T02:38:28.059 INFO:tasks.rados.rados.0.smithi107.stdout:6345: done (4 left) 2020-06-12T02:38:28.059 INFO:tasks.rados.rados.0.smithi107.stdout:6346: done (3 left) 2020-06-12T02:38:28.059 INFO:tasks.rados.rados.0.smithi107.stdout:6349: done (2 left) 2020-06-12T02:38:28.059 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:28.059 INFO:tasks.rados.rados.0.smithi107.stdout:6350: seq_num 6349 ranges {319488=737280} 2020-06-12T02:38:28.060 INFO:tasks.rados.rados.0.smithi107.stdout:6350: writing smithi10729407-17 from 319488 to 1056768 tid 1 2020-06-12T02:38:28.060 INFO:tasks.rados.rados.0.smithi107.stdout:6351: delete oid 18 current snap is 0 2020-06-12T02:38:28.060 INFO:tasks.rados.rados.0.smithi107.stdout:6348: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:28.060 INFO:tasks.rados.rados.0.smithi107.stdout:6348: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:28.060 INFO:tasks.rados.rados.0.smithi107.stdout:6348: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:28.060 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 642 (ObjNum 6347 snap 0 seq_num 6347) dirty exists 2020-06-12T02:38:28.061 INFO:tasks.rados.rados.0.smithi107.stdout:6348: left oid 5 (ObjNum 6347 snap 0 seq_num 6347) 2020-06-12T02:38:28.061 INFO:tasks.rados.rados.0.smithi107.stdout:6350: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:28.061 INFO:tasks.rados.rados.0.smithi107.stdout:6348: done (3 left) 2020-06-12T02:38:28.061 INFO:tasks.rados.rados.0.smithi107.stdout:6351: done (2 left) 2020-06-12T02:38:28.061 INFO:tasks.rados.rados.0.smithi107.stdout:6352: delete oid 16 current snap is 0 2020-06-12T02:38:28.062 INFO:tasks.rados.rados.0.smithi107.stdout:6350: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:28.062 INFO:tasks.rados.rados.0.smithi107.stdout:6350: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:28.062 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1186 (ObjNum 6349 snap 0 seq_num 6349) dirty exists 2020-06-12T02:38:28.062 INFO:tasks.rados.rados.0.smithi107.stdout:6350: left oid 17 (ObjNum 6349 snap 0 seq_num 6349) 2020-06-12T02:38:28.062 INFO:tasks.rados.rados.0.smithi107.stdout:6350: done (2 left) 2020-06-12T02:38:28.062 INFO:tasks.rados.rados.0.smithi107.stdout:6352: done (1 left) 2020-06-12T02:38:28.063 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:28.063 INFO:tasks.rados.rados.0.smithi107.stdout:6353: seq_num 6352 ranges {1146880=786432} 2020-06-12T02:38:28.063 INFO:tasks.rados.rados.0.smithi107.stdout:6353: writing smithi10729407-9 from 1146880 to 1933312 tid 1 2020-06-12T02:38:28.063 INFO:tasks.rados.rados.0.smithi107.stdout:6354: delete oid 18 current snap is 0 2020-06-12T02:38:28.063 INFO:tasks.rados.rados.0.smithi107.stdout:6353: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:28.064 INFO:tasks.rados.rados.0.smithi107.stdout:6353: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:28.064 INFO:tasks.rados.rados.0.smithi107.stdout:6354: done (2 left) 2020-06-12T02:38:28.064 INFO:tasks.rados.rados.0.smithi107.stdout:6355: delete oid 18 current snap is 0 2020-06-12T02:38:28.064 INFO:tasks.rados.rados.0.smithi107.stdout:6353: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:28.064 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1160 (ObjNum 6352 snap 0 seq_num 6352) dirty exists 2020-06-12T02:38:28.064 INFO:tasks.rados.rados.0.smithi107.stdout:6353: left oid 9 (ObjNum 6352 snap 0 seq_num 6352) 2020-06-12T02:38:28.065 INFO:tasks.rados.rados.0.smithi107.stdout:6353: done (2 left) 2020-06-12T02:38:28.065 INFO:tasks.rados.rados.0.smithi107.stdout:6355: done (1 left) 2020-06-12T02:38:28.065 INFO:tasks.rados.rados.0.smithi107.stdout:6356: delete oid 5 current snap is 0 2020-06-12T02:38:28.065 INFO:tasks.rados.rados.0.smithi107.stdout:6356: done (1 left) 2020-06-12T02:38:28.065 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:28.066 INFO:tasks.rados.rados.0.smithi107.stdout:6357: seq_num 6356 ranges {0=458752,458752=401408} 2020-06-12T02:38:28.066 INFO:tasks.rados.rados.0.smithi107.stdout:6357: writing smithi10729407-7 from 0 to 458752 tid 1 2020-06-12T02:38:28.066 INFO:tasks.rados.rados.0.smithi107.stdout:6357: writing smithi10729407-7 from 458752 to 860160 tid 2 2020-06-12T02:38:28.066 INFO:tasks.rados.rados.0.smithi107.stdout:6358: delete oid 8 current snap is 0 2020-06-12T02:38:28.066 INFO:tasks.rados.rados.0.smithi107.stdout:6358: done (2 left) 2020-06-12T02:38:28.066 INFO:tasks.rados.rados.0.smithi107.stdout:6359: delete oid 2 current snap is 0 2020-06-12T02:38:28.067 INFO:tasks.rados.rados.0.smithi107.stdout:6357: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:28.067 INFO:tasks.rados.rados.0.smithi107.stdout:6357: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:28.067 INFO:tasks.rados.rados.0.smithi107.stdout:6357: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:28.067 INFO:tasks.rados.rados.0.smithi107.stdout:6359: done (2 left) 2020-06-12T02:38:28.067 INFO:tasks.rados.rados.0.smithi107.stdout:6360: delete oid 18 current snap is 0 2020-06-12T02:38:28.068 INFO:tasks.rados.rados.0.smithi107.stdout:6357: finishing write tid 5 to smithi10729407-7 2020-06-12T02:38:28.068 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1208 (ObjNum 6356 snap 0 seq_num 6356) dirty exists 2020-06-12T02:38:28.068 INFO:tasks.rados.rados.0.smithi107.stdout:6357: left oid 7 (ObjNum 6356 snap 0 seq_num 6356) 2020-06-12T02:38:28.068 INFO:tasks.rados.rados.0.smithi107.stdout:6357: done (2 left) 2020-06-12T02:38:28.068 INFO:tasks.rados.rados.0.smithi107.stdout:6360: done (1 left) 2020-06-12T02:38:28.068 INFO:tasks.rados.rados.0.smithi107.stdout:6361: delete oid 18 current snap is 0 2020-06-12T02:38:28.069 INFO:tasks.rados.rados.0.smithi107.stdout:6361: done (1 left) 2020-06-12T02:38:28.069 INFO:tasks.rados.rados.0.smithi107.stdout:6362: delete oid 5 current snap is 0 2020-06-12T02:38:28.069 INFO:tasks.rados.rados.0.smithi107.stdout:6362: done (1 left) 2020-06-12T02:38:28.069 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:28.069 INFO:tasks.rados.rados.0.smithi107.stdout:6363: seq_num 6362 ranges {368640=557056,925696=401408} 2020-06-12T02:38:28.070 INFO:tasks.rados.rados.0.smithi107.stdout:6363: writing smithi10729407-12 from 368640 to 925696 tid 1 2020-06-12T02:38:28.070 INFO:tasks.rados.rados.0.smithi107.stdout:6363: writing smithi10729407-12 from 925696 to 1327104 tid 2 2020-06-12T02:38:28.070 INFO:tasks.rados.rados.0.smithi107.stdout:6364: delete oid 15 current snap is 0 2020-06-12T02:38:28.071 INFO:tasks.rados.rados.0.smithi107.stdout:6364: done (2 left) 2020-06-12T02:38:28.072 INFO:tasks.rados.rados.0.smithi107.stdout:6365: delete oid 8 current snap is 0 2020-06-12T02:38:28.072 INFO:tasks.rados.rados.0.smithi107.stdout:6363: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:28.072 INFO:tasks.rados.rados.0.smithi107.stdout:6363: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:28.072 INFO:tasks.rados.rados.0.smithi107.stdout:6363: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:28.072 INFO:tasks.rados.rados.0.smithi107.stdout:6365: done (2 left) 2020-06-12T02:38:28.073 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:28.073 INFO:tasks.rados.rados.0.smithi107.stdout:6366: seq_num 6365 ranges {1933312=606208,2539520=401408} 2020-06-12T02:38:28.073 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:28.068+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:28.075 INFO:tasks.rados.rados.0.smithi107.stdout:6366: writing smithi10729407-9 from 1933312 to 2539520 tid 1 2020-06-12T02:38:28.078 INFO:tasks.rados.rados.0.smithi107.stdout:6366: writing smithi10729407-9 from 2539520 to 2940928 tid 2 2020-06-12T02:38:28.078 INFO:tasks.rados.rados.0.smithi107.stdout:6367: delete oid 11 current snap is 0 2020-06-12T02:38:28.078 INFO:tasks.rados.rados.0.smithi107.stdout:6363: finishing write tid 5 to smithi10729407-12 2020-06-12T02:38:28.079 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1155 (ObjNum 6362 snap 0 seq_num 6362) dirty exists 2020-06-12T02:38:28.079 INFO:tasks.rados.rados.0.smithi107.stdout:6363: left oid 12 (ObjNum 6362 snap 0 seq_num 6362) 2020-06-12T02:38:28.082 INFO:tasks.rados.rados.0.smithi107.stdout:6363: done (3 left) 2020-06-12T02:38:28.082 INFO:tasks.rados.rados.0.smithi107.stdout:6367: done (2 left) 2020-06-12T02:38:28.082 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:28.083 INFO:tasks.rados.rados.0.smithi107.stdout:6368: seq_num 6367 ranges {0=638976,638976=401408} 2020-06-12T02:38:28.086 INFO:tasks.rados.rados.0.smithi107.stdout:6368: writing smithi10729407-4 from 0 to 638976 tid 1 2020-06-12T02:38:28.089 INFO:tasks.rados.rados.0.smithi107.stdout:6368: writing smithi10729407-4 from 638976 to 1040384 tid 2 2020-06-12T02:38:28.089 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:38:28.089 INFO:tasks.rados.rados.0.smithi107.stdout:6369: seq_num 6368 ranges {0=655360,655360=401408} 2020-06-12T02:38:28.092 INFO:tasks.rados.rados.0.smithi107.stdout:6369: writing smithi10729407-5 from 0 to 655360 tid 1 2020-06-12T02:38:28.095 INFO:tasks.rados.rados.0.smithi107.stdout:6369: writing smithi10729407-5 from 655360 to 1056768 tid 2 2020-06-12T02:38:28.095 INFO:tasks.rados.rados.0.smithi107.stdout:6370: delete oid 2 current snap is 0 2020-06-12T02:38:28.095 INFO:tasks.rados.rados.0.smithi107.stdout:6366: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:28.097 INFO:tasks.rados.rados.0.smithi107.stdout:6366: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:28.097 INFO:tasks.rados.rados.0.smithi107.stdout:6368: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:28.104 INFO:tasks.rados.rados.0.smithi107.stdout:6368: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:28.104 INFO:tasks.rados.rados.0.smithi107.stdout:6366: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:28.105 INFO:tasks.rados.rados.0.smithi107.stdout:6368: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:28.106 INFO:tasks.rados.rados.0.smithi107.stdout:6366: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:28.106 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1167 (ObjNum 6365 snap 0 seq_num 6365) dirty exists 2020-06-12T02:38:28.106 INFO:tasks.rados.rados.0.smithi107.stdout:6366: left oid 9 (ObjNum 6365 snap 0 seq_num 6365) 2020-06-12T02:38:28.106 INFO:tasks.rados.rados.0.smithi107.stdout:6366: done (4 left) 2020-06-12T02:38:28.106 INFO:tasks.rados.rados.0.smithi107.stdout:6370: done (3 left) 2020-06-12T02:38:28.107 INFO:tasks.rados.rados.0.smithi107.stdout:6371: delete oid 16 current snap is 0 2020-06-12T02:38:28.107 INFO:tasks.rados.rados.0.smithi107.stdout:6368: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:28.107 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1189 (ObjNum 6367 snap 0 seq_num 6367) dirty exists 2020-06-12T02:38:28.107 INFO:tasks.rados.rados.0.smithi107.stdout:6368: left oid 4 (ObjNum 6367 snap 0 seq_num 6367) 2020-06-12T02:38:28.107 INFO:tasks.rados.rados.0.smithi107.stdout:6369: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:28.108 INFO:tasks.rados.rados.0.smithi107.stdout:6368: done (3 left) 2020-06-12T02:38:28.108 INFO:tasks.rados.rados.0.smithi107.stdout:6371: done (2 left) 2020-06-12T02:38:28.108 INFO:tasks.rados.rados.0.smithi107.stdout:6372: delete oid 12 current snap is 0 2020-06-12T02:38:28.109 INFO:tasks.rados.rados.0.smithi107.stdout:6369: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:28.109 INFO:tasks.rados.rados.0.smithi107.stdout:6372: done (2 left) 2020-06-12T02:38:28.109 INFO:tasks.rados.rados.0.smithi107.stdout:6373: delete oid 14 current snap is 0 2020-06-12T02:38:28.109 INFO:tasks.rados.rados.0.smithi107.stdout:6369: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:28.111 INFO:tasks.rados.rados.0.smithi107.stdout:6369: finishing write tid 5 to smithi10729407-5 2020-06-12T02:38:28.111 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 647 (ObjNum 6368 snap 0 seq_num 6368) dirty exists 2020-06-12T02:38:28.112 INFO:tasks.rados.rados.0.smithi107.stdout:6369: left oid 5 (ObjNum 6368 snap 0 seq_num 6368) 2020-06-12T02:38:28.114 INFO:tasks.rados.rados.0.smithi107.stdout:6369: done (2 left) 2020-06-12T02:38:28.114 INFO:tasks.rados.rados.0.smithi107.stdout:6373: done (1 left) 2020-06-12T02:38:28.115 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:28.115 INFO:tasks.rados.rados.0.smithi107.stdout:6374: seq_num 6373 ranges {2940928=745472,3686400=401408} 2020-06-12T02:38:28.134 INFO:tasks.rados.rados.0.smithi107.stdout:6374: writing smithi10729407-9 from 2940928 to 3686400 tid 1 2020-06-12T02:38:28.136 INFO:tasks.rados.rados.0.smithi107.stdout:6374: writing smithi10729407-9 from 3686400 to 4087808 tid 2 2020-06-12T02:38:28.137 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:28.137 INFO:tasks.rados.rados.0.smithi107.stdout:6375: seq_num 6374 ranges {1056768=761856,1818624=401408} 2020-06-12T02:38:28.149 INFO:tasks.rados.rados.0.smithi107.stdout:6375: writing smithi10729407-17 from 1056768 to 1818624 tid 1 2020-06-12T02:38:28.150 INFO:tasks.rados.rados.0.smithi107.stdout:6375: writing smithi10729407-17 from 1818624 to 2220032 tid 2 2020-06-12T02:38:28.150 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:38:28.150 INFO:tasks.rados.rados.0.smithi107.stdout:6376: seq_num 6375 ranges {0=778240,778240=401408} 2020-06-12T02:38:28.154 INFO:tasks.rados.rados.0.smithi107.stdout:6376: writing smithi10729407-6 from 0 to 778240 tid 1 2020-06-12T02:38:28.156 INFO:tasks.rados.rados.0.smithi107.stdout:6376: writing smithi10729407-6 from 778240 to 1179648 tid 2 2020-06-12T02:38:28.157 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:28.157 INFO:tasks.rados.rados.0.smithi107.stdout:6377: seq_num 6376 ranges {0=794624,794624=401408} 2020-06-12T02:38:28.175 INFO:tasks.rados.rados.0.smithi107.stdout:6377: writing smithi10729407-8 from 0 to 794624 tid 1 2020-06-12T02:38:28.175 INFO:tasks.rados.rados.0.smithi107.stdout:6377: writing smithi10729407-8 from 794624 to 1196032 tid 2 2020-06-12T02:38:28.176 INFO:tasks.rados.rados.0.smithi107.stdout:6378: delete oid 15 current snap is 0 2020-06-12T02:38:28.176 INFO:tasks.rados.rados.0.smithi107.stdout:6374: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:28.176 INFO:tasks.rados.rados.0.smithi107.stdout:6374: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:28.176 INFO:tasks.rados.rados.0.smithi107.stdout:6374: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:28.176 INFO:tasks.rados.rados.0.smithi107.stdout:6374: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:28.177 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1170 (ObjNum 6373 snap 0 seq_num 6373) dirty exists 2020-06-12T02:38:28.177 INFO:tasks.rados.rados.0.smithi107.stdout:6374: left oid 9 (ObjNum 6373 snap 0 seq_num 6373) 2020-06-12T02:38:28.177 INFO:tasks.rados.rados.0.smithi107.stdout:6375: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:28.177 INFO:tasks.rados.rados.0.smithi107.stdout:6375: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:28.177 INFO:tasks.rados.rados.0.smithi107.stdout:6375: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:28.178 INFO:tasks.rados.rados.0.smithi107.stdout:6376: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:28.178 INFO:tasks.rados.rados.0.smithi107.stdout:6375: finishing write tid 5 to smithi10729407-17 2020-06-12T02:38:28.178 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1192 (ObjNum 6374 snap 0 seq_num 6374) dirty exists 2020-06-12T02:38:28.178 INFO:tasks.rados.rados.0.smithi107.stdout:6375: left oid 17 (ObjNum 6374 snap 0 seq_num 6374) 2020-06-12T02:38:28.178 INFO:tasks.rados.rados.0.smithi107.stdout:6376: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:28.178 INFO:tasks.rados.rados.0.smithi107.stdout:6376: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:28.179 INFO:tasks.rados.rados.0.smithi107.stdout:6376: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:28.179 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1169 (ObjNum 6375 snap 0 seq_num 6375) dirty exists 2020-06-12T02:38:28.179 INFO:tasks.rados.rados.0.smithi107.stdout:6376: left oid 6 (ObjNum 6375 snap 0 seq_num 6375) 2020-06-12T02:38:28.180 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:28.172+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:28.180 INFO:tasks.rados.rados.0.smithi107.stdout:6377: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:28.180 INFO:tasks.rados.rados.0.smithi107.stdout:6377: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:28.181 INFO:tasks.rados.rados.0.smithi107.stdout:6377: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:28.182 INFO:tasks.rados.rados.0.smithi107.stdout:6374: done (5 left) 2020-06-12T02:38:28.182 INFO:tasks.rados.rados.0.smithi107.stdout:6375: done (4 left) 2020-06-12T02:38:28.183 INFO:tasks.rados.rados.0.smithi107.stdout:6376: done (3 left) 2020-06-12T02:38:28.183 INFO:tasks.rados.rados.0.smithi107.stdout:6378: done (2 left) 2020-06-12T02:38:28.183 INFO:tasks.rados.rados.0.smithi107.stdout:6379: delete oid 16 current snap is 0 2020-06-12T02:38:28.183 INFO:tasks.rados.rados.0.smithi107.stdout:6377: finishing write tid 5 to smithi10729407-8 2020-06-12T02:38:28.183 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1176 (ObjNum 6376 snap 0 seq_num 6376) dirty exists 2020-06-12T02:38:28.184 INFO:tasks.rados.rados.0.smithi107.stdout:6377: left oid 8 (ObjNum 6376 snap 0 seq_num 6376) 2020-06-12T02:38:28.184 INFO:tasks.rados.rados.0.smithi107.stdout:6377: done (2 left) 2020-06-12T02:38:28.184 INFO:tasks.rados.rados.0.smithi107.stdout:6379: done (1 left) 2020-06-12T02:38:28.185 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:28.185 INFO:tasks.rados.rados.0.smithi107.stdout:6380: seq_num 6379 ranges {1040384=442368,1482752=638976,2121728=163840} 2020-06-12T02:38:28.191 INFO:tasks.rados.rados.0.smithi107.stdout:6380: writing smithi10729407-4 from 1040384 to 1482752 tid 1 2020-06-12T02:38:28.194 INFO:tasks.rados.rados.0.smithi107.stdout:6380: writing smithi10729407-4 from 1482752 to 2121728 tid 2 2020-06-12T02:38:28.196 INFO:tasks.rados.rados.0.smithi107.stdout:6380: writing smithi10729407-4 from 2121728 to 2285568 tid 3 2020-06-12T02:38:28.196 INFO:tasks.rados.rados.0.smithi107.stdout:6381: delete oid 8 current snap is 0 2020-06-12T02:38:28.197 INFO:tasks.rados.rados.0.smithi107.stdout:6380: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:28.200 INFO:tasks.rados.rados.0.smithi107.stdout:6381: done (2 left) 2020-06-12T02:38:28.200 INFO:tasks.rados.rados.0.smithi107.stdout:6382: delete oid 1 current snap is 0 2020-06-12T02:38:28.212 INFO:tasks.rados.rados.0.smithi107.stdout:6380: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:28.212 INFO:tasks.rados.rados.0.smithi107.stdout:6382: done (2 left) 2020-06-12T02:38:28.212 INFO:tasks.rados.rados.0.smithi107.stdout:6383: delete oid 15 current snap is 0 2020-06-12T02:38:28.212 INFO:tasks.rados.rados.0.smithi107.stdout:6380: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:28.213 INFO:tasks.rados.rados.0.smithi107.stdout:6380: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:28.213 INFO:tasks.rados.rados.0.smithi107.stdout:6380: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:28.213 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1196 (ObjNum 6379 snap 0 seq_num 6379) dirty exists 2020-06-12T02:38:28.213 INFO:tasks.rados.rados.0.smithi107.stdout:6380: left oid 4 (ObjNum 6379 snap 0 seq_num 6379) 2020-06-12T02:38:28.214 INFO:tasks.rados.rados.0.smithi107.stdout:6380: done (2 left) 2020-06-12T02:38:28.214 INFO:tasks.rados.rados.0.smithi107.stdout:6383: done (1 left) 2020-06-12T02:38:28.214 INFO:tasks.rados.rados.0.smithi107.stdout:6384: delete oid 3 current snap is 0 2020-06-12T02:38:28.215 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:28.216 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:38:28.216 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2020-06-12T02:38:28.216 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:28.216 INFO:tasks.rados.rados.0.smithi107.stdout:6384: done (1 left) 2020-06-12T02:38:28.217 INFO:tasks.rados.rados.0.smithi107.stdout:6385: delete oid 15 current snap is 0 2020-06-12T02:38:28.219 INFO:tasks.rados.rados.0.smithi107.stdout:6385: done (1 left) 2020-06-12T02:38:28.219 INFO:tasks.rados.rados.0.smithi107.stdout:6386: delete oid 16 current snap is 0 2020-06-12T02:38:28.221 INFO:tasks.rados.rados.0.smithi107.stdout:6386: done (1 left) 2020-06-12T02:38:28.222 INFO:tasks.rados.rados.0.smithi107.stdout:6387: delete oid 19 current snap is 0 2020-06-12T02:38:28.224 INFO:tasks.rados.rados.0.smithi107.stdout:6387: done (1 left) 2020-06-12T02:38:28.224 INFO:tasks.rados.rados.0.smithi107.stdout:6388: delete oid 6 current snap is 0 2020-06-12T02:38:28.226 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-06-12T02:38:28.226 INFO:tasks.rados.rados.0.smithi107.stdout:6388: done (1 left) 2020-06-12T02:38:28.227 INFO:tasks.rados.rados.0.smithi107.stdout:6389: delete oid 11 current snap is 0 2020-06-12T02:38:28.229 INFO:tasks.rados.rados.0.smithi107.stdout:6389: done (1 left) 2020-06-12T02:38:28.229 INFO:tasks.rados.rados.0.smithi107.stdout:6390: delete oid 2 current snap is 0 2020-06-12T02:38:28.231 INFO:tasks.rados.rados.0.smithi107.stdout:6390: done (1 left) 2020-06-12T02:38:28.231 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:28.231 INFO:tasks.rados.rados.0.smithi107.stdout:6391: seq_num 6390 ranges {2285568=622592,2908160=614400,3522560=188416} 2020-06-12T02:38:28.246 INFO:tasks.rados.rados.0.smithi107.stdout:6391: writing smithi10729407-4 from 2285568 to 2908160 tid 1 2020-06-12T02:38:28.249 INFO:tasks.rados.rados.0.smithi107.stdout:6391: writing smithi10729407-4 from 2908160 to 3522560 tid 2 2020-06-12T02:38:28.250 INFO:tasks.rados.rados.0.smithi107.stdout:6391: writing smithi10729407-4 from 3522560 to 3710976 tid 3 2020-06-12T02:38:28.250 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:28.251 INFO:tasks.rados.rados.0.smithi107.stdout:6392: seq_num 6391 ranges {0=647168,647168=499712,1146880=303104} 2020-06-12T02:38:28.254 INFO:tasks.rados.rados.0.smithi107.stdout:6392: writing smithi10729407-14 from 0 to 647168 tid 1 2020-06-12T02:38:28.258 INFO:tasks.rados.rados.0.smithi107.stdout:6392: writing smithi10729407-14 from 647168 to 1146880 tid 2 2020-06-12T02:38:28.260 INFO:tasks.rados.rados.0.smithi107.stdout:6392: writing smithi10729407-14 from 1146880 to 1449984 tid 3 2020-06-12T02:38:28.260 INFO:tasks.rados.rados.0.smithi107.stdout:6393: delete oid 12 current snap is 0 2020-06-12T02:38:28.260 INFO:tasks.rados.rados.0.smithi107.stdout:6391: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:28.261 INFO:tasks.rados.rados.0.smithi107.stdout:6391: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:28.261 INFO:tasks.rados.rados.0.smithi107.stdout:6392: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:28.262 INFO:tasks.rados.rados.0.smithi107.stdout:6391: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:28.263 INFO:tasks.rados.rados.0.smithi107.stdout:6393: done (3 left) 2020-06-12T02:38:28.263 INFO:tasks.rados.rados.0.smithi107.stdout:6394: delete oid 8 current snap is 0 2020-06-12T02:38:28.263 INFO:tasks.rados.rados.0.smithi107.stdout:6391: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:28.265 INFO:tasks.rados.rados.0.smithi107.stdout:6391: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:28.265 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1200 (ObjNum 6390 snap 0 seq_num 6390) dirty exists 2020-06-12T02:38:28.265 INFO:tasks.rados.rados.0.smithi107.stdout:6391: left oid 4 (ObjNum 6390 snap 0 seq_num 6390) 2020-06-12T02:38:28.266 INFO:tasks.rados.rados.0.smithi107.stdout:6391: done (3 left) 2020-06-12T02:38:28.266 INFO:tasks.rados.rados.0.smithi107.stdout:6394: done (2 left) 2020-06-12T02:38:28.267 INFO:tasks.rados.rados.0.smithi107.stdout:6395: delete oid 2 current snap is 0 2020-06-12T02:38:28.270 INFO:tasks.rados.rados.0.smithi107.stdout:6395: done (2 left) 2020-06-12T02:38:28.270 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:38:28.270 INFO:tasks.rados.rados.0.smithi107.stdout:6396: seq_num 6395 ranges {0=712704,712704=491520,1204224=311296} 2020-06-12T02:38:28.272 INFO:tasks.rados.rados.0.smithi107.stdout:6396: writing smithi10729407-3 from 0 to 712704 tid 1 2020-06-12T02:38:28.275 INFO:tasks.rados.rados.0.smithi107.stdout:6396: writing smithi10729407-3 from 712704 to 1204224 tid 2 2020-06-12T02:38:28.278 INFO:tasks.rados.rados.0.smithi107.stdout:6396: writing smithi10729407-3 from 1204224 to 1515520 tid 3 2020-06-12T02:38:28.278 INFO:tasks.rados.rados.0.smithi107.stdout:6397: delete oid 11 current snap is 0 2020-06-12T02:38:28.278 INFO:tasks.rados.rados.0.smithi107.stdout:6392: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:28.278 INFO:tasks.rados.rados.0.smithi107.stdout:6392: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:28.279 INFO:tasks.rados.rados.0.smithi107.stdout:6392: finishing write tid 5 to smithi10729407-14 2020-06-12T02:38:28.279 INFO:tasks.rados.rados.0.smithi107.stdout:6392: finishing write tid 6 to smithi10729407-14 2020-06-12T02:38:28.279 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1136 (ObjNum 6391 snap 0 seq_num 6391) dirty exists 2020-06-12T02:38:28.279 INFO:tasks.rados.rados.0.smithi107.stdout:6392: left oid 14 (ObjNum 6391 snap 0 seq_num 6391) 2020-06-12T02:38:28.281 INFO:tasks.rados.rados.0.smithi107.stdout:6392: done (3 left) 2020-06-12T02:38:28.281 INFO:tasks.rados.rados.0.smithi107.stdout:6397: done (2 left) 2020-06-12T02:38:28.281 INFO:tasks.rados.rados.0.smithi107.stdout:6398: delete oid 7 current snap is 0 2020-06-12T02:38:28.283 INFO:tasks.rados.rados.0.smithi107.stdout:6396: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:28.284 INFO:tasks.rados.rados.0.smithi107.stdout:6398: done (2 left) 2020-06-12T02:38:28.285 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:28.285 INFO:tasks.rados.rados.0.smithi107.stdout:6399: seq_num 6398 ranges {1449984=761856,2211840=540672,2752512=262144} 2020-06-12T02:38:28.297 INFO:tasks.rados.rados.0.smithi107.stdout:6399: writing smithi10729407-14 from 1449984 to 2211840 tid 1 2020-06-12T02:38:28.300 INFO:tasks.rados.rados.0.smithi107.stdout:6399: writing smithi10729407-14 from 2211840 to 2752512 tid 2 2020-06-12T02:38:28.302 INFO:tasks.rados.rados.0.smithi107.stdout:6399: writing smithi10729407-14 from 2752512 to 3014656 tid 3 2020-06-12T02:38:28.302 INFO:tasks.rados.rados.0.smithi107.stdout:6400: delete oid 1 current snap is 0 2020-06-12T02:38:28.302 INFO:tasks.rados.rados.0.smithi107.stdout:6396: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:28.308 INFO:tasks.rados.rados.0.smithi107.stdout:6396: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:28.308 INFO:tasks.rados.rados.0.smithi107.stdout:6396: finishing write tid 5 to smithi10729407-3 2020-06-12T02:38:28.310 INFO:tasks.rados.rados.0.smithi107.stdout:6396: finishing write tid 6 to smithi10729407-3 2020-06-12T02:38:28.311 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1128 (ObjNum 6395 snap 0 seq_num 6395) dirty exists 2020-06-12T02:38:28.311 INFO:tasks.rados.rados.0.smithi107.stdout:6396: left oid 3 (ObjNum 6395 snap 0 seq_num 6395) 2020-06-12T02:38:28.314 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:28.307+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:28.314 INFO:tasks.rados.rados.0.smithi107.stdout:6399: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:28.314 INFO:tasks.rados.rados.0.smithi107.stdout:6399: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:28.315 INFO:tasks.rados.rados.0.smithi107.stdout:6396: done (3 left) 2020-06-12T02:38:28.315 INFO:tasks.rados.rados.0.smithi107.stdout:6400: done (2 left) 2020-06-12T02:38:28.315 INFO:tasks.rados.rados.0.smithi107.stdout:6401: delete oid 11 current snap is 0 2020-06-12T02:38:28.315 INFO:tasks.rados.rados.0.smithi107.stdout:6399: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:28.315 INFO:tasks.rados.rados.0.smithi107.stdout:6399: finishing write tid 5 to smithi10729407-14 2020-06-12T02:38:28.316 INFO:tasks.rados.rados.0.smithi107.stdout:6401: done (2 left) 2020-06-12T02:38:28.316 INFO:tasks.rados.rados.0.smithi107.stdout:6402: delete oid 3 current snap is 0 2020-06-12T02:38:28.316 INFO:tasks.rados.rados.0.smithi107.stdout:6399: finishing write tid 6 to smithi10729407-14 2020-06-12T02:38:28.317 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1141 (ObjNum 6398 snap 0 seq_num 6398) dirty exists 2020-06-12T02:38:28.317 INFO:tasks.rados.rados.0.smithi107.stdout:6399: left oid 14 (ObjNum 6398 snap 0 seq_num 6398) 2020-06-12T02:38:28.318 INFO:tasks.rados.rados.0.smithi107.stdout:6399: done (2 left) 2020-06-12T02:38:28.318 INFO:tasks.rados.rados.0.smithi107.stdout:6402: done (1 left) 2020-06-12T02:38:28.318 INFO:tasks.rados.rados.0.smithi107.stdout:6403: delete oid 15 current snap is 0 2020-06-12T02:38:28.323 INFO:tasks.rados.rados.0.smithi107.stdout:6403: done (1 left) 2020-06-12T02:38:28.323 INFO:tasks.rados.rados.0.smithi107.stdout:6404: delete oid 14 current snap is 0 2020-06-12T02:38:28.327 INFO:tasks.rados.rados.0.smithi107.stdout:6404: done (1 left) 2020-06-12T02:38:28.327 INFO:tasks.rados.rados.0.smithi107.stdout:6405: delete oid 12 current snap is 0 2020-06-12T02:38:28.330 INFO:tasks.rados.rados.0.smithi107.stdout:6405: done (1 left) 2020-06-12T02:38:28.330 INFO:tasks.rados.rados.0.smithi107.stdout:6406: delete oid 20 current snap is 0 2020-06-12T02:38:28.331 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-06-12T02:38:28.332 INFO:tasks.rados.rados.0.smithi107.stdout:6406: done (1 left) 2020-06-12T02:38:28.332 INFO:tasks.rados.rados.0.smithi107.stdout:6407: delete oid 4 current snap is 0 2020-06-12T02:38:28.334 INFO:tasks.rados.rados.0.smithi107.stdout:6407: done (1 left) 2020-06-12T02:38:28.334 INFO:tasks.rados.rados.0.smithi107.stdout:6408: delete oid 8 current snap is 0 2020-06-12T02:38:28.336 INFO:tasks.rados.rados.0.smithi107.stdout:6408: done (1 left) 2020-06-12T02:38:28.336 INFO:tasks.rados.rados.0.smithi107.stdout:6409: delete oid 4 current snap is 0 2020-06-12T02:38:28.343 INFO:tasks.rados.rados.0.smithi107.stdout:6409: done (1 left) 2020-06-12T02:38:28.343 INFO:tasks.rados.rados.0.smithi107.stdout:6410: delete oid 2 current snap is 0 2020-06-12T02:38:28.344 INFO:tasks.rados.rados.0.smithi107.stdout:6410: done (1 left) 2020-06-12T02:38:28.344 INFO:tasks.rados.rados.0.smithi107.stdout:6411: delete oid 12 current snap is 0 2020-06-12T02:38:28.346 INFO:tasks.rados.rados.0.smithi107.stdout:6411: done (1 left) 2020-06-12T02:38:28.346 INFO:tasks.rados.rados.0.smithi107.stdout:6412: delete oid 17 current snap is 0 2020-06-12T02:38:28.349 INFO:tasks.rados.rados.0.smithi107.stdout:6412: done (1 left) 2020-06-12T02:38:28.351 INFO:tasks.rados.rados.0.smithi107.stdout:6413: delete oid 11 current snap is 0 2020-06-12T02:38:28.352 INFO:tasks.rados.rados.0.smithi107.stdout:6413: done (1 left) 2020-06-12T02:38:28.352 INFO:tasks.rados.rados.0.smithi107.stdout:6414: delete oid 5 current snap is 0 2020-06-12T02:38:28.354 INFO:tasks.rados.rados.0.smithi107.stdout:6414: done (1 left) 2020-06-12T02:38:28.355 INFO:tasks.rados.rados.0.smithi107.stdout:6415: delete oid 2 current snap is 0 2020-06-12T02:38:28.359 INFO:tasks.rados.rados.0.smithi107.stdout:6415: done (1 left) 2020-06-12T02:38:28.359 INFO:tasks.rados.rados.0.smithi107.stdout:6416: delete oid 19 current snap is 0 2020-06-12T02:38:28.361 INFO:tasks.rados.rados.0.smithi107.stdout:6416: done (1 left) 2020-06-12T02:38:28.362 INFO:tasks.rados.rados.0.smithi107.stdout:6417: delete oid 13 current snap is 0 2020-06-12T02:38:28.363 INFO:tasks.rados.rados.0.smithi107.stdout:6417: done (1 left) 2020-06-12T02:38:28.364 INFO:tasks.rados.rados.0.smithi107.stdout:6418: delete oid 4 current snap is 0 2020-06-12T02:38:28.366 INFO:tasks.rados.rados.0.smithi107.stdout:6418: done (1 left) 2020-06-12T02:38:28.366 INFO:tasks.rados.rados.0.smithi107.stdout:6419: delete oid 9 current snap is 0 2020-06-12T02:38:28.368 INFO:tasks.rados.rados.0.smithi107.stdout:6419: done (1 left) 2020-06-12T02:38:28.369 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:28.369 INFO:tasks.rados.rados.0.smithi107.stdout:6420: seq_num 6419 ranges {0=712704,712704=663552,1376256=540672} 2020-06-12T02:38:28.373 INFO:tasks.rados.rados.0.smithi107.stdout:6420: writing smithi10729407-18 from 0 to 712704 tid 1 2020-06-12T02:38:28.376 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:28.374+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:28.378 INFO:tasks.rados.rados.0.smithi107.stdout:6420: writing smithi10729407-18 from 712704 to 1376256 tid 2 2020-06-12T02:38:28.382 INFO:tasks.rados.rados.0.smithi107.stdout:6420: writing smithi10729407-18 from 1376256 to 1916928 tid 3 2020-06-12T02:38:28.382 INFO:tasks.rados.rados.0.smithi107.stdout:6421: delete oid 20 current snap is 0 2020-06-12T02:38:28.387 INFO:tasks.rados.rados.0.smithi107.stdout:6421: done (2 left) 2020-06-12T02:38:28.387 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:28.387 INFO:tasks.rados.rados.0.smithi107.stdout:6422: seq_num 6421 ranges {0=745472,745472=434176,1179648=565248,1744896=204800} 2020-06-12T02:38:28.392 INFO:tasks.rados.rados.0.smithi107.stdout:6422: writing smithi10729407-15 from 0 to 745472 tid 1 2020-06-12T02:38:28.394 INFO:tasks.rados.rados.0.smithi107.stdout:6422: writing smithi10729407-15 from 745472 to 1179648 tid 2 2020-06-12T02:38:28.397 INFO:tasks.rados.rados.0.smithi107.stdout:6422: writing smithi10729407-15 from 1179648 to 1744896 tid 3 2020-06-12T02:38:28.398 INFO:tasks.rados.rados.0.smithi107.stdout:6422: writing smithi10729407-15 from 1744896 to 1949696 tid 4 2020-06-12T02:38:28.399 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:28.399 INFO:tasks.rados.rados.0.smithi107.stdout:6423: seq_num 6422 ranges {0=761856,761856=712704,1474560=491520} 2020-06-12T02:38:28.400 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:28.400 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:38:28.400 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:38:28.401 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:38:28.401 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:28.402 INFO:tasks.rados.rados.0.smithi107.stdout:6423: writing smithi10729407-8 from 0 to 761856 tid 1 2020-06-12T02:38:28.406 INFO:tasks.rados.rados.0.smithi107.stdout:6423: writing smithi10729407-8 from 761856 to 1474560 tid 2 2020-06-12T02:38:28.409 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-06-12T02:38:28.410 INFO:tasks.rados.rados.0.smithi107.stdout:6423: writing smithi10729407-8 from 1474560 to 1966080 tid 3 2020-06-12T02:38:28.410 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:28.411 INFO:tasks.rados.rados.0.smithi107.stdout:6424: seq_num 6423 ranges {0=778240,778240=598016,1376256=581632,1957888=24576} 2020-06-12T02:38:28.414 INFO:tasks.rados.rados.0.smithi107.stdout:6424: writing smithi10729407-16 from 0 to 778240 tid 1 2020-06-12T02:38:28.423 INFO:tasks.rados.rados.0.smithi107.stdout:6424: writing smithi10729407-16 from 778240 to 1376256 tid 2 2020-06-12T02:38:28.426 INFO:tasks.rados.rados.0.smithi107.stdout:6424: writing smithi10729407-16 from 1376256 to 1957888 tid 3 2020-06-12T02:38:28.426 INFO:tasks.rados.rados.0.smithi107.stdout:6424: writing smithi10729407-16 from 1957888 to 1982464 tid 4 2020-06-12T02:38:28.426 INFO:tasks.rados.rados.0.smithi107.stdout:6425: delete oid 20 current snap is 0 2020-06-12T02:38:28.426 INFO:tasks.rados.rados.0.smithi107.stdout:6420: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:28.428 INFO:tasks.rados.rados.0.smithi107.stdout:6420: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:28.429 INFO:tasks.rados.rados.0.smithi107.stdout:6420: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:28.429 INFO:tasks.rados.rados.0.smithi107.stdout:6420: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:28.429 INFO:tasks.rados.rados.0.smithi107.stdout:6420: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:28.430 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1175 (ObjNum 6419 snap 0 seq_num 6419) dirty exists 2020-06-12T02:38:28.430 INFO:tasks.rados.rados.0.smithi107.stdout:6420: left oid 18 (ObjNum 6419 snap 0 seq_num 6419) 2020-06-12T02:38:28.430 INFO:tasks.rados.rados.0.smithi107.stdout:6422: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:28.431 INFO:tasks.rados.rados.0.smithi107.stdout:6424: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:28.433 INFO:tasks.rados.rados.0.smithi107.stdout:6424: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:28.434 INFO:tasks.rados.rados.0.smithi107.stdout:6424: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:28.436 INFO:tasks.rados.rados.0.smithi107.stdout:6424: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:28.436 INFO:tasks.rados.rados.0.smithi107.stdout:6424: finishing write tid 6 to smithi10729407-16 2020-06-12T02:38:28.438 INFO:tasks.rados.rados.0.smithi107.stdout:6424: finishing write tid 7 to smithi10729407-16 2020-06-12T02:38:28.438 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1134 (ObjNum 6423 snap 0 seq_num 6423) dirty exists 2020-06-12T02:38:28.438 INFO:tasks.rados.rados.0.smithi107.stdout:6424: left oid 16 (ObjNum 6423 snap 0 seq_num 6423) 2020-06-12T02:38:28.470 INFO:tasks.rados.rados.0.smithi107.stdout:6420: done (5 left) 2020-06-12T02:38:28.470 INFO:tasks.rados.rados.0.smithi107.stdout:6424: done (4 left) 2020-06-12T02:38:28.471 INFO:tasks.rados.rados.0.smithi107.stdout:6425: done (3 left) 2020-06-12T02:38:28.471 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:28.471 INFO:tasks.rados.rados.0.smithi107.stdout:6426: seq_num 6425 ranges {1916928=409600,2326528=417792,2744320=548864,3293184=638976} 2020-06-12T02:38:28.481 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:38:28.482 INFO:tasks.rados.rados.0.smithi107.stdout:6426: writing smithi10729407-18 from 1916928 to 2326528 tid 1 2020-06-12T02:38:28.485 INFO:tasks.rados.rados.0.smithi107.stdout:6426: writing smithi10729407-18 from 2326528 to 2744320 tid 2 2020-06-12T02:38:28.488 INFO:tasks.rados.rados.0.smithi107.stdout:6426: writing smithi10729407-18 from 2744320 to 3293184 tid 3 2020-06-12T02:38:28.493 INFO:tasks.rados.rados.0.smithi107.stdout:6426: writing smithi10729407-18 from 3293184 to 3932160 tid 4 2020-06-12T02:38:28.493 INFO:tasks.rados.rados.0.smithi107.stdout:6427: delete oid 9 current snap is 0 2020-06-12T02:38:28.493 INFO:tasks.rados.rados.0.smithi107.stdout:6422: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:28.494 INFO:tasks.rados.rados.0.smithi107.stdout:6422: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:28.494 INFO:tasks.rados.rados.0.smithi107.stdout:6422: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:28.494 INFO:tasks.rados.rados.0.smithi107.stdout:6422: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:28.495 INFO:tasks.rados.rados.0.smithi107.stdout:6423: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:28.495 INFO:tasks.rados.rados.0.smithi107.stdout:6422: finishing write tid 7 to smithi10729407-15 2020-06-12T02:38:28.495 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1188 (ObjNum 6421 snap 0 seq_num 6421) dirty exists 2020-06-12T02:38:28.495 INFO:tasks.rados.rados.0.smithi107.stdout:6422: left oid 15 (ObjNum 6421 snap 0 seq_num 6421) 2020-06-12T02:38:28.497 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:38:28.491+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:38:28.497 INFO:tasks.rados.rados.0.smithi107.stdout:6426: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:28.498 INFO:tasks.rados.rados.0.smithi107.stdout:6426: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:28.498 INFO:tasks.rados.rados.0.smithi107.stdout:6423: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:28.498 INFO:tasks.rados.rados.0.smithi107.stdout:6423: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:28.499 INFO:tasks.rados.rados.0.smithi107.stdout:6423: finishing write tid 5 to smithi10729407-8 2020-06-12T02:38:28.499 INFO:tasks.rados.rados.0.smithi107.stdout:6423: finishing write tid 6 to smithi10729407-8 2020-06-12T02:38:28.500 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1192 (ObjNum 6422 snap 0 seq_num 6422) dirty exists 2020-06-12T02:38:28.500 INFO:tasks.rados.rados.0.smithi107.stdout:6423: left oid 8 (ObjNum 6422 snap 0 seq_num 6422) 2020-06-12T02:38:28.501 INFO:tasks.rados.rados.0.smithi107.stdout:6426: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:28.501 INFO:tasks.rados.rados.0.smithi107.stdout:6426: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:28.502 INFO:tasks.rados.rados.0.smithi107.stdout:6426: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:28.503 INFO:tasks.rados.rados.0.smithi107.stdout:6426: finishing write tid 7 to smithi10729407-18 2020-06-12T02:38:28.504 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1180 (ObjNum 6425 snap 0 seq_num 6425) dirty exists 2020-06-12T02:38:28.504 INFO:tasks.rados.rados.0.smithi107.stdout:6426: left oid 18 (ObjNum 6425 snap 0 seq_num 6425) 2020-06-12T02:38:28.504 INFO:tasks.rados.rados.0.smithi107.stdout:6422: done (4 left) 2020-06-12T02:38:28.504 INFO:tasks.rados.rados.0.smithi107.stdout:6423: done (3 left) 2020-06-12T02:38:28.505 INFO:tasks.rados.rados.0.smithi107.stdout:6426: done (2 left) 2020-06-12T02:38:28.505 INFO:tasks.rados.rados.0.smithi107.stdout:6427: done (1 left) 2020-06-12T02:38:28.505 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:38:28.506 INFO:tasks.rados.rados.0.smithi107.stdout:6428: seq_num 6427 ranges {0=442368,442368=589824,1032192=557056,1589248=458752} 2020-06-12T02:38:28.506 INFO:tasks.rados.rados.0.smithi107.stdout:6428: writing smithi10729407-2 from 0 to 442368 tid 1 2020-06-12T02:38:28.508 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:38:28.516 INFO:tasks.rados.rados.0.smithi107.stdout:6428: writing smithi10729407-2 from 442368 to 1032192 tid 2 2020-06-12T02:38:28.522 INFO:tasks.rados.rados.0.smithi107.stdout:6428: writing smithi10729407-2 from 1032192 to 1589248 tid 3 2020-06-12T02:38:28.522 INFO:tasks.rados.rados.0.smithi107.stdout:6428: writing smithi10729407-2 from 1589248 to 2048000 tid 4 2020-06-12T02:38:28.522 INFO:tasks.rados.rados.0.smithi107.stdout:6429: delete oid 14 current snap is 0 2020-06-12T02:38:28.523 INFO:tasks.rados.rados.0.smithi107.stdout:6428: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:28.523 INFO:tasks.rados.rados.0.smithi107.stdout:6429: done (2 left) 2020-06-12T02:38:28.523 INFO:tasks.rados.rados.0.smithi107.stdout:6430: delete oid 14 current snap is 0 2020-06-12T02:38:28.524 INFO:tasks.rados.rados.0.smithi107.stdout:6428: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:28.524 INFO:tasks.rados.rados.0.smithi107.stdout:6428: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:28.524 INFO:tasks.rados.rados.0.smithi107.stdout:6430: done (2 left) 2020-06-12T02:38:28.524 INFO:tasks.rados.rados.0.smithi107.stdout:6431: delete oid 11 current snap is 0 2020-06-12T02:38:28.524 INFO:tasks.rados.rados.0.smithi107.stdout:6428: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:28.525 INFO:tasks.rados.rados.0.smithi107.stdout:6428: finishing write tid 6 to smithi10729407-2 2020-06-12T02:38:28.526 INFO:tasks.rados.rados.0.smithi107.stdout:6431: done (2 left) 2020-06-12T02:38:28.526 INFO:tasks.rados.rados.0.smithi107.stdout:6432: delete oid 9 current snap is 0 2020-06-12T02:38:28.526 INFO:tasks.rados.rados.0.smithi107.stdout:6428: finishing write tid 7 to smithi10729407-2 2020-06-12T02:38:28.527 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 616 (ObjNum 6427 snap 0 seq_num 6427) dirty exists 2020-06-12T02:38:28.527 INFO:tasks.rados.rados.0.smithi107.stdout:6428: left oid 2 (ObjNum 6427 snap 0 seq_num 6427) 2020-06-12T02:38:28.527 INFO:tasks.rados.rados.0.smithi107.stdout:6428: done (2 left) 2020-06-12T02:38:28.527 INFO:tasks.rados.rados.0.smithi107.stdout:6432: done (1 left) 2020-06-12T02:38:28.527 INFO:tasks.rados.rados.0.smithi107.stdout:6433: delete oid 11 current snap is 0 2020-06-12T02:38:28.528 INFO:tasks.rados.rados.0.smithi107.stdout:6433: done (1 left) 2020-06-12T02:38:28.528 INFO:tasks.rados.rados.0.smithi107.stdout:6434: delete oid 1 current snap is 0 2020-06-12T02:38:28.529 INFO:tasks.rados.rados.0.smithi107.stdout:6434: done (1 left) 2020-06-12T02:38:28.529 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:28.529 INFO:tasks.rados.rados.0.smithi107.stdout:6435: seq_num 6434 ranges {0=565248,565248=630784,1196032=712704,1908736=262144} 2020-06-12T02:38:28.532 INFO:tasks.rados.rados.0.smithi107.stdout:6435: writing smithi10729407-5 from 0 to 565248 tid 1 2020-06-12T02:38:28.533 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:38:28.535 INFO:tasks.rados.rados.0.smithi107.stdout:6435: writing smithi10729407-5 from 565248 to 1196032 tid 2 2020-06-12T02:38:28.539 INFO:tasks.rados.rados.0.smithi107.stdout:6435: writing smithi10729407-5 from 1196032 to 1908736 tid 3 2020-06-12T02:38:28.541 INFO:tasks.rados.rados.0.smithi107.stdout:6435: writing smithi10729407-5 from 1908736 to 2170880 tid 4 2020-06-12T02:38:28.542 INFO:tasks.rados.rados.0.smithi107.stdout:6436: delete oid 4 current snap is 0 2020-06-12T02:38:28.542 INFO:tasks.rados.rados.0.smithi107.stdout:6435: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:28.545 INFO:tasks.rados.rados.0.smithi107.stdout:6435: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:28.546 INFO:tasks.rados.rados.0.smithi107.stdout:6436: done (2 left) 2020-06-12T02:38:28.546 INFO:tasks.rados.rados.0.smithi107.stdout:6437: delete oid 19 current snap is 0 2020-06-12T02:38:28.548 INFO:tasks.rados.rados.0.smithi107.stdout:6437: done (2 left) 2020-06-12T02:38:28.548 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:28.549 INFO:tasks.rados.rados.0.smithi107.stdout:6438: seq_num 6437 ranges {0=614400,614400=679936,1294336=483328,1777664=442368} 2020-06-12T02:38:28.551 INFO:tasks.rados.rados.0.smithi107.stdout:6438: writing smithi10729407-17 from 0 to 614400 tid 1 2020-06-12T02:38:28.556 INFO:tasks.rados.rados.0.smithi107.stdout:6438: writing smithi10729407-17 from 614400 to 1294336 tid 2 2020-06-12T02:38:28.557 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:38:28.558 INFO:tasks.rados.rados.0.smithi107.stdout:6438: writing smithi10729407-17 from 1294336 to 1777664 tid 3 2020-06-12T02:38:28.561 INFO:tasks.rados.rados.0.smithi107.stdout:6438: writing smithi10729407-17 from 1777664 to 2220032 tid 4 2020-06-12T02:38:28.561 INFO:tasks.rados.rados.0.smithi107.stdout:6439: delete oid 12 current snap is 0 2020-06-12T02:38:28.562 INFO:tasks.rados.rados.0.smithi107.stdout:6435: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:28.562 INFO:tasks.rados.rados.0.smithi107.stdout:6435: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:28.562 INFO:tasks.rados.rados.0.smithi107.stdout:6435: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:28.562 INFO:tasks.rados.rados.0.smithi107.stdout:6435: finishing write tid 7 to smithi10729407-5 2020-06-12T02:38:28.562 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 653 (ObjNum 6434 snap 0 seq_num 6434) dirty exists 2020-06-12T02:38:28.563 INFO:tasks.rados.rados.0.smithi107.stdout:6435: left oid 5 (ObjNum 6434 snap 0 seq_num 6434) 2020-06-12T02:38:28.563 INFO:tasks.rados.rados.0.smithi107.stdout:6438: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:28.565 INFO:tasks.rados.rados.0.smithi107.stdout:6435: done (3 left) 2020-06-12T02:38:28.565 INFO:tasks.rados.rados.0.smithi107.stdout:6439: done (2 left) 2020-06-12T02:38:28.565 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:28.566 INFO:tasks.rados.rados.0.smithi107.stdout:6440: seq_num 6439 ranges {0=647168,647168=450560,1097728=499712,1597440=655360} 2020-06-12T02:38:28.569 INFO:tasks.rados.rados.0.smithi107.stdout:6440: writing smithi10729407-14 from 0 to 647168 tid 1 2020-06-12T02:38:28.573 INFO:tasks.rados.rados.0.smithi107.stdout:6440: writing smithi10729407-14 from 647168 to 1097728 tid 2 2020-06-12T02:38:28.576 INFO:tasks.rados.rados.0.smithi107.stdout:6440: writing smithi10729407-14 from 1097728 to 1597440 tid 3 2020-06-12T02:38:28.578 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:28.575+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:28.579 INFO:tasks.rados.rados.0.smithi107.stdout:6440: writing smithi10729407-14 from 1597440 to 2252800 tid 4 2020-06-12T02:38:28.579 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:28.580 INFO:tasks.rados.rados.0.smithi107.stdout:6441: seq_num 6440 ranges {1949696=663552,2613248=786432,3399680=712704,4112384=106496} 2020-06-12T02:38:28.587 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:38:28.594 INFO:tasks.rados.rados.0.smithi107.stdout:6441: writing smithi10729407-15 from 1949696 to 2613248 tid 1 2020-06-12T02:38:28.599 INFO:tasks.rados.rados.0.smithi107.stdout:6441: writing smithi10729407-15 from 2613248 to 3399680 tid 2 2020-06-12T02:38:28.603 INFO:tasks.rados.rados.0.smithi107.stdout:6441: writing smithi10729407-15 from 3399680 to 4112384 tid 3 2020-06-12T02:38:28.604 INFO:tasks.rados.rados.0.smithi107.stdout:6441: writing smithi10729407-15 from 4112384 to 4218880 tid 4 2020-06-12T02:38:28.604 INFO:tasks.rados.rados.0.smithi107.stdout:6442: delete oid 4 current snap is 0 2020-06-12T02:38:28.604 INFO:tasks.rados.rados.0.smithi107.stdout:6438: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:28.605 INFO:tasks.rados.rados.0.smithi107.stdout:6438: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:28.605 INFO:tasks.rados.rados.0.smithi107.stdout:6438: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:28.605 INFO:tasks.rados.rados.0.smithi107.stdout:6438: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:28.605 INFO:tasks.rados.rados.0.smithi107.stdout:6440: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:28.605 INFO:tasks.rados.rados.0.smithi107.stdout:6438: finishing write tid 7 to smithi10729407-17 2020-06-12T02:38:28.606 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1210 (ObjNum 6437 snap 0 seq_num 6437) dirty exists 2020-06-12T02:38:28.606 INFO:tasks.rados.rados.0.smithi107.stdout:6438: left oid 17 (ObjNum 6437 snap 0 seq_num 6437) 2020-06-12T02:38:28.606 INFO:tasks.rados.rados.0.smithi107.stdout:6440: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:28.606 INFO:tasks.rados.rados.0.smithi107.stdout:6440: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:28.606 INFO:tasks.rados.rados.0.smithi107.stdout:6440: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:28.606 INFO:tasks.rados.rados.0.smithi107.stdout:6440: finishing write tid 6 to smithi10729407-14 2020-06-12T02:38:28.607 INFO:tasks.rados.rados.0.smithi107.stdout:6440: finishing write tid 7 to smithi10729407-14 2020-06-12T02:38:28.607 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1153 (ObjNum 6439 snap 0 seq_num 6439) dirty exists 2020-06-12T02:38:28.607 INFO:tasks.rados.rados.0.smithi107.stdout:6440: left oid 14 (ObjNum 6439 snap 0 seq_num 6439) 2020-06-12T02:38:28.607 INFO:tasks.rados.rados.0.smithi107.stdout:6441: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:28.608 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:28.608 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:38:28.608 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:38:28.609 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:38:28.609 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.610 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1206 3.5s0 3:a9510bda:::smithi10729407-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:38:28.610 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.856688+0000", 2020-06-12T02:38:28.610 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 55.744779317000003, 2020-06-12T02:38:28.610 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019116942000000001, 2020-06-12T02:38:28.610 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:28.611 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:28.611 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:28.611 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:28.611 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:28.611 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1206 2020-06-12T02:38:28.611 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.612 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:28.612 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.612 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:28.612 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856688+0000", 2020-06-12T02:38:28.612 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:28.613 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.613 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.613 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:28.613 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856688+0000", 2020-06-12T02:38:28.613 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4355e-05 2020-06-12T02:38:28.614 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.614 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.614 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:28.614 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856713+0000", 2020-06-12T02:38:28.614 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0850000000000001e-06 2020-06-12T02:38:28.614 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.615 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.615 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:28.615 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856716+0000", 2020-06-12T02:38:28.615 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4509999999999997e-06 2020-06-12T02:38:28.615 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.616 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.616 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:28.616 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856722+0000", 2020-06-12T02:38:28.616 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2879999999999999e-06 2020-06-12T02:38:28.616 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.616 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.617 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:28.617 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856724+0000", 2020-06-12T02:38:28.617 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0057507449999999998 2020-06-12T02:38:28.617 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.617 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.618 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:28.618 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.862475+0000", 2020-06-12T02:38:28.618 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5759000000000003e-05 2020-06-12T02:38:28.618 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.618 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.619 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:28.620 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.862571+0000", 2020-06-12T02:38:28.620 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012199018000000001 2020-06-12T02:38:28.620 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.620 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.620 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:28.621 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.874770+0000", 2020-06-12T02:38:28.621 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7260999999999999e-05 2020-06-12T02:38:28.621 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.621 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.621 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:28.622 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.874827+0000", 2020-06-12T02:38:28.622 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00097798000000000004 2020-06-12T02:38:28.622 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.622 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.622 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:28.623 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.875805+0000", 2020-06-12T02:38:28.623 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019116942000000001 2020-06-12T02:38:28.623 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:28.623 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:28.623 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:28.623 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.624 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.624 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1572 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~745472] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-06-12T02:38:28.625 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.886439+0000", 2020-06-12T02:38:28.625 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 54.715028599, 2020-06-12T02:38:28.625 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021442571000000001, 2020-06-12T02:38:28.626 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:28.626 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:28.626 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:28.626 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:28.626 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:28.626 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1572 2020-06-12T02:38:28.627 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.627 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:28.627 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.627 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:28.627 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.886439+0000", 2020-06-12T02:38:28.628 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:28.628 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.628 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.628 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:28.628 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.886439+0000", 2020-06-12T02:38:28.629 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00073027599999999997 2020-06-12T02:38:28.629 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.629 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.629 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:28.630 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887169+0000", 2020-06-12T02:38:28.630 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.49e-06 2020-06-12T02:38:28.630 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.631 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.631 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:28.631 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887175+0000", 2020-06-12T02:38:28.631 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7899999999999994e-06 2020-06-12T02:38:28.631 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.631 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.632 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:28.632 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887185+0000", 2020-06-12T02:38:28.632 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.42e-06 2020-06-12T02:38:28.632 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.633 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.634 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:28.634 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887189+0000", 2020-06-12T02:38:28.634 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1619999999999999e-05 2020-06-12T02:38:28.634 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.635 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.635 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:28.635 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887221+0000", 2020-06-12T02:38:28.635 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9720000000000002e-05 2020-06-12T02:38:28.635 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.636 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.636 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:28.636 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887300+0000", 2020-06-12T02:38:28.636 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047354100000000001 2020-06-12T02:38:28.636 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.637 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.637 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:28.637 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887774+0000", 2020-06-12T02:38:28.637 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0030649750000000002 2020-06-12T02:38:28.637 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.637 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.638 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:28.638 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.890839+0000", 2020-06-12T02:38:28.638 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016995321000000001 2020-06-12T02:38:28.638 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.638 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.639 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:28.639 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.907834+0000", 2020-06-12T02:38:28.639 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7417999999999999e-05 2020-06-12T02:38:28.639 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.640 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.640 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:28.640 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.907882+0000", 2020-06-12T02:38:28.641 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021442571000000001 2020-06-12T02:38:28.641 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:28.641 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:28.641 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:28.641 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.641 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.642 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1573 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~712704] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-06-12T02:38:28.642 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.890888+0000", 2020-06-12T02:38:28.642 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 54.710579864000003, 2020-06-12T02:38:28.642 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018868001999999998, 2020-06-12T02:38:28.642 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:28.643 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:28.643 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:28.643 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:28.643 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:28.643 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1573 2020-06-12T02:38:28.644 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.644 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:28.644 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.644 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:28.644 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.890888+0000", 2020-06-12T02:38:28.644 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:28.645 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.645 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.645 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:28.645 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.890888+0000", 2020-06-12T02:38:28.645 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000683605 2020-06-12T02:38:28.646 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.646 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.646 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:28.646 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891571+0000", 2020-06-12T02:38:28.646 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4999999999999999e-06 2020-06-12T02:38:28.646 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.647 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.647 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:28.647 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891577+0000", 2020-06-12T02:38:28.647 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.171e-05 2020-06-12T02:38:28.647 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.647 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.648 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:28.648 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891589+0000", 2020-06-12T02:38:28.648 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7650000000000001e-06 2020-06-12T02:38:28.648 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.648 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.649 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:28.649 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891593+0000", 2020-06-12T02:38:28.649 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6689000000000001e-05 2020-06-12T02:38:28.649 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.649 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.650 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:28.650 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891630+0000", 2020-06-12T02:38:28.650 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9877999999999997e-05 2020-06-12T02:38:28.651 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.651 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.651 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:28.651 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891710+0000", 2020-06-12T02:38:28.652 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053367200000000001 2020-06-12T02:38:28.652 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.652 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.652 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:28.652 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.892244+0000", 2020-06-12T02:38:28.652 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004927196 2020-06-12T02:38:28.653 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.653 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.653 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:28.653 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.897171+0000", 2020-06-12T02:38:28.653 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012496419 2020-06-12T02:38:28.654 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.654 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.654 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:28.654 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.909667+0000", 2020-06-12T02:38:28.654 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8567999999999996e-05 2020-06-12T02:38:28.655 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.655 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.655 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:28.655 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.909756+0000", 2020-06-12T02:38:28.655 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018868001999999998 2020-06-12T02:38:28.655 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:28.656 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:28.656 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:28.656 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.656 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.656 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1576 3.5s0 3:a446366d:::smithi10729407-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e26)", 2020-06-12T02:38:28.657 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.891955+0000", 2020-06-12T02:38:28.657 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 54.709512623999998, 2020-06-12T02:38:28.657 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019893246999999999, 2020-06-12T02:38:28.657 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:28.657 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:28.658 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:28.658 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:28.658 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:28.658 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1576 2020-06-12T02:38:28.658 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.658 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:28.659 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.659 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:28.659 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891955+0000", 2020-06-12T02:38:28.659 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:28.660 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.660 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.660 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:28.660 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891955+0000", 2020-06-12T02:38:28.661 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2067999999999999e-05 2020-06-12T02:38:28.661 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.661 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.662 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:28.662 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891977+0000", 2020-06-12T02:38:28.662 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.18e-06 2020-06-12T02:38:28.662 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.662 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.663 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:28.663 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891980+0000", 2020-06-12T02:38:28.663 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3629999999999999e-06 2020-06-12T02:38:28.663 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.663 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.663 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:28.664 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891987+0000", 2020-06-12T02:38:28.664 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.181e-06 2020-06-12T02:38:28.664 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.664 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.664 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:28.665 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891989+0000", 2020-06-12T02:38:28.665 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050959120000000002 2020-06-12T02:38:28.665 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.665 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.665 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:28.665 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.897085+0000", 2020-06-12T02:38:28.666 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1089999999999998e-05 2020-06-12T02:38:28.666 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.666 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.666 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:28.666 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.897166+0000", 2020-06-12T02:38:28.667 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013400235 2020-06-12T02:38:28.667 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.667 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.667 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:28.667 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.910566+0000", 2020-06-12T02:38:28.667 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000104981 2020-06-12T02:38:28.668 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.668 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.668 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:28.668 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.910671+0000", 2020-06-12T02:38:28.668 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001177237 2020-06-12T02:38:28.669 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.669 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.669 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:28.669 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.911848+0000", 2020-06-12T02:38:28.669 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019893246999999999 2020-06-12T02:38:28.670 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:28.670 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:28.670 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:28.670 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.671 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.671 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2913 3.5s0 3:a9510bda:::smithi10729407-14:head [append 0~770048,stat] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:28.671 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.070354+0000", 2020-06-12T02:38:28.671 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 51.531113822999998, 2020-06-12T02:38:28.672 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60690798599999995, 2020-06-12T02:38:28.672 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:28.672 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:28.672 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:28.672 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:28.673 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:28.673 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2913 2020-06-12T02:38:28.673 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.673 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:28.673 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.674 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:28.674 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.070354+0000", 2020-06-12T02:38:28.674 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:28.674 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.674 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.674 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:28.675 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.070354+0000", 2020-06-12T02:38:28.675 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00098478900000000002 2020-06-12T02:38:28.675 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.675 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.675 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:28.676 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071339+0000", 2020-06-12T02:38:28.676 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8349999999999993e-06 2020-06-12T02:38:28.676 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.676 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.676 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:28.676 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071347+0000", 2020-06-12T02:38:28.677 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0749e-05 2020-06-12T02:38:28.677 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.677 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.677 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:28.677 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071358+0000", 2020-06-12T02:38:28.678 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9369999999999997e-06 2020-06-12T02:38:28.678 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.678 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.678 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:28.678 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071362+0000", 2020-06-12T02:38:28.678 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0426e-05 2020-06-12T02:38:28.679 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.679 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.679 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:28.679 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071423+0000", 2020-06-12T02:38:28.679 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4129999999999998e-05 2020-06-12T02:38:28.680 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.680 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.680 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:28.680 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071487+0000", 2020-06-12T02:38:28.680 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053752599999999998 2020-06-12T02:38:28.680 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.681 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.681 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:28.681 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072024+0000", 2020-06-12T02:38:28.682 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036162579999999998 2020-06-12T02:38:28.682 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.682 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.682 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:28.683 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075640+0000", 2020-06-12T02:38:28.683 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60152201999999999 2020-06-12T02:38:28.683 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.683 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.683 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:28.684 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677162+0000", 2020-06-12T02:38:28.684 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9315999999999996e-05 2020-06-12T02:38:28.684 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.684 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.684 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:28.684 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677262+0000", 2020-06-12T02:38:28.685 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60690798599999995 2020-06-12T02:38:28.685 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:28.685 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:28.685 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:28.685 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.686 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.686 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2914 3.5s0 3:a9510bda:::smithi10729407-14:head [append 0~401408] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:28.686 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.072409+0000", 2020-06-12T02:38:28.686 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 51.529058378999999, 2020-06-12T02:38:28.686 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.61956801100000003, 2020-06-12T02:38:28.686 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:28.687 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:28.687 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:28.687 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:28.687 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:28.687 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2914 2020-06-12T02:38:28.688 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.688 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:28.696 INFO:tasks.rados.rados.0.smithi107.stdout:6438: done (4 left) 2020-06-12T02:38:28.697 INFO:tasks.rados.rados.0.smithi107.stdout:6440: done (3 left) 2020-06-12T02:38:28.697 INFO:tasks.rados.rados.0.smithi107.stdout:6442: done (2 left) 2020-06-12T02:38:28.697 INFO:tasks.rados.rados.0.smithi107.stdout:6443: delete oid 14 current snap is 0 2020-06-12T02:38:28.697 INFO:tasks.rados.rados.0.smithi107.stdout:6443: done (2 left) 2020-06-12T02:38:28.698 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:28.698 INFO:tasks.rados.rados.0.smithi107.stdout:6444: seq_num 6443 ranges {0=712704,712704=434176,1146880=483328,1630208=688128} 2020-06-12T02:38:28.698 INFO:tasks.rados.rados.0.smithi107.stdout:6444: writing smithi10729407-14 from 0 to 712704 tid 1 2020-06-12T02:38:28.698 INFO:tasks.rados.rados.0.smithi107.stdout:6444: writing smithi10729407-14 from 712704 to 1146880 tid 2 2020-06-12T02:38:28.698 INFO:tasks.rados.rados.0.smithi107.stdout:6444: writing smithi10729407-14 from 1146880 to 1630208 tid 3 2020-06-12T02:38:28.699 INFO:tasks.rados.rados.0.smithi107.stdout:6444: writing smithi10729407-14 from 1630208 to 2318336 tid 4 2020-06-12T02:38:28.699 INFO:tasks.rados.rados.0.smithi107.stdout:6445: delete oid 20 current snap is 0 2020-06-12T02:38:28.699 INFO:tasks.rados.rados.0.smithi107.stdout:6441: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:28.699 INFO:tasks.rados.rados.0.smithi107.stdout:6441: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:28.699 INFO:tasks.rados.rados.0.smithi107.stdout:6441: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:28.700 INFO:tasks.rados.rados.0.smithi107.stdout:6441: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:28.700 INFO:tasks.rados.rados.0.smithi107.stdout:6441: finishing write tid 7 to smithi10729407-15 2020-06-12T02:38:28.700 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1197 (ObjNum 6440 snap 0 seq_num 6440) dirty exists 2020-06-12T02:38:28.700 INFO:tasks.rados.rados.0.smithi107.stdout:6441: left oid 15 (ObjNum 6440 snap 0 seq_num 6440) 2020-06-12T02:38:28.700 INFO:tasks.rados.rados.0.smithi107.stdout:6444: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:28.700 INFO:tasks.rados.rados.0.smithi107.stdout:6441: done (3 left) 2020-06-12T02:38:28.701 INFO:tasks.rados.rados.0.smithi107.stdout:6445: done (2 left) 2020-06-12T02:38:28.701 INFO:tasks.rados.rados.0.smithi107.stdout:6446: delete oid 19 current snap is 0 2020-06-12T02:38:28.701 INFO:tasks.rados.rados.0.smithi107.stdout:6446: done (2 left) 2020-06-12T02:38:28.701 INFO:tasks.rados.rados.0.smithi107.stdout:6447: delete oid 2 current snap is 0 2020-06-12T02:38:28.702 INFO:tasks.rados.rados.0.smithi107.stdout:6447: done (2 left) 2020-06-12T02:38:28.702 INFO:tasks.rados.rados.0.smithi107.stdout:6448: delete oid 8 current snap is 0 2020-06-12T02:38:28.702 INFO:tasks.rados.rados.0.smithi107.stdout:6444: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:28.703 INFO:tasks.rados.rados.0.smithi107.stdout:6444: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:28.703 INFO:tasks.rados.rados.0.smithi107.stdout:6444: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:28.703 INFO:tasks.rados.rados.0.smithi107.stdout:6444: finishing write tid 6 to smithi10729407-14 2020-06-12T02:38:28.703 INFO:tasks.rados.rados.0.smithi107.stdout:6448: done (2 left) 2020-06-12T02:38:28.703 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:28.704 INFO:tasks.rados.rados.0.smithi107.stdout:6449: seq_num 6448 ranges {0=794624,794624=704512,1499136=671744,2170880=229376} 2020-06-12T02:38:28.704 INFO:tasks.rados.rados.0.smithi107.stdout:6449: writing smithi10729407-6 from 0 to 794624 tid 1 2020-06-12T02:38:28.704 INFO:tasks.rados.rados.0.smithi107.stdout:6449: writing smithi10729407-6 from 794624 to 1499136 tid 2 2020-06-12T02:38:28.704 INFO:tasks.rados.rados.0.smithi107.stdout:6449: writing smithi10729407-6 from 1499136 to 2170880 tid 3 2020-06-12T02:38:28.704 INFO:tasks.rados.rados.0.smithi107.stdout:6449: writing smithi10729407-6 from 2170880 to 2400256 tid 4 2020-06-12T02:38:28.704 INFO:tasks.rados.rados.0.smithi107.stdout:6450: delete oid 7 current snap is 0 2020-06-12T02:38:28.705 INFO:tasks.rados.rados.0.smithi107.stdout:6444: finishing write tid 7 to smithi10729407-14 2020-06-12T02:38:28.705 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1159 (ObjNum 6443 snap 0 seq_num 6443) dirty exists 2020-06-12T02:38:28.705 INFO:tasks.rados.rados.0.smithi107.stdout:6444: left oid 14 (ObjNum 6443 snap 0 seq_num 6443) 2020-06-12T02:38:28.705 INFO:tasks.rados.rados.0.smithi107.stdout:6449: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:28.705 INFO:tasks.rados.rados.0.smithi107.stdout:6444: done (3 left) 2020-06-12T02:38:28.706 INFO:tasks.rados.rados.0.smithi107.stdout:6450: done (2 left) 2020-06-12T02:38:28.706 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:28.706 INFO:tasks.rados.rados.0.smithi107.stdout:6451: seq_num 6450 ranges {0=32768} 2020-06-12T02:38:28.706 INFO:tasks.rados.rados.0.smithi107.stdout:6451: writing smithi10729407-4 from 0 to 32768 tid 1 2020-06-12T02:38:28.706 INFO:tasks.rados.rados.0.smithi107.stdout:6452: delete oid 7 current snap is 0 2020-06-12T02:38:28.706 INFO:tasks.rados.rados.0.smithi107.stdout:6452: done (3 left) 2020-06-12T02:38:28.707 INFO:tasks.rados.rados.0.smithi107.stdout:6453: delete oid 7 current snap is 0 2020-06-12T02:38:28.707 INFO:tasks.rados.rados.0.smithi107.stdout:6451: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:28.707 INFO:tasks.rados.rados.0.smithi107.stdout:6451: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:28.707 INFO:tasks.rados.rados.0.smithi107.stdout:6449: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:28.707 INFO:tasks.rados.rados.0.smithi107.stdout:6451: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:28.708 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1213 (ObjNum 6450 snap 0 seq_num 6450) dirty exists 2020-06-12T02:38:28.708 INFO:tasks.rados.rados.0.smithi107.stdout:6451: left oid 4 (ObjNum 6450 snap 0 seq_num 6450) 2020-06-12T02:38:28.708 INFO:tasks.rados.rados.0.smithi107.stdout:6451: done (3 left) 2020-06-12T02:38:28.708 INFO:tasks.rados.rados.0.smithi107.stdout:6453: done (2 left) 2020-06-12T02:38:28.708 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:28.708 INFO:tasks.rados.rados.0.smithi107.stdout:6454: seq_num 6453 ranges {0=81920} 2020-06-12T02:38:28.709 INFO:tasks.rados.rados.0.smithi107.stdout:6454: writing smithi10729407-19 from 0 to 81920 tid 1 2020-06-12T02:38:28.709 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:38:28.709 INFO:tasks.rados.rados.0.smithi107.stdout:6455: seq_num 6454 ranges {0=98304} 2020-06-12T02:38:28.709 INFO:tasks.rados.rados.0.smithi107.stdout:6455: writing smithi10729407-1 from 0 to 98304 tid 1 2020-06-12T02:38:28.709 INFO:tasks.rados.rados.0.smithi107.stdout:6456: delete oid 4 current snap is 0 2020-06-12T02:38:28.710 INFO:tasks.rados.rados.0.smithi107.stdout:6449: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:28.710 INFO:tasks.rados.rados.0.smithi107.stdout:6449: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:28.710 INFO:tasks.rados.rados.0.smithi107.stdout:6454: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:28.710 INFO:tasks.rados.rados.0.smithi107.stdout:6456: done (4 left) 2020-06-12T02:38:28.710 INFO:tasks.rados.rados.0.smithi107.stdout:6457: delete oid 18 current snap is 0 2020-06-12T02:38:28.710 INFO:tasks.rados.rados.0.smithi107.stdout:6449: finishing write tid 6 to smithi10729407-6 2020-06-12T02:38:28.711 INFO:tasks.rados.rados.0.smithi107.stdout:6454: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:28.711 INFO:tasks.rados.rados.0.smithi107.stdout:6449: finishing write tid 7 to smithi10729407-6 2020-06-12T02:38:28.711 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1178 (ObjNum 6448 snap 0 seq_num 6448) dirty exists 2020-06-12T02:38:28.711 INFO:tasks.rados.rados.0.smithi107.stdout:6449: left oid 6 (ObjNum 6448 snap 0 seq_num 6448) 2020-06-12T02:38:28.712 INFO:tasks.rados.rados.0.smithi107.stdout:6454: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:28.712 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1166 (ObjNum 6453 snap 0 seq_num 6453) dirty exists 2020-06-12T02:38:28.712 INFO:tasks.rados.rados.0.smithi107.stdout:6454: left oid 19 (ObjNum 6453 snap 0 seq_num 6453) 2020-06-12T02:38:28.712 INFO:tasks.rados.rados.0.smithi107.stdout:6455: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:28.712 INFO:tasks.rados.rados.0.smithi107.stdout:6455: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:28.713 INFO:tasks.rados.rados.0.smithi107.stdout:6449: done (4 left) 2020-06-12T02:38:28.713 INFO:tasks.rados.rados.0.smithi107.stdout:6454: done (3 left) 2020-06-12T02:38:28.713 INFO:tasks.rados.rados.0.smithi107.stdout:6457: done (2 left) 2020-06-12T02:38:28.714 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:28.714 INFO:tasks.rados.rados.0.smithi107.stdout:6458: seq_num 6457 ranges {2318336=147456} 2020-06-12T02:38:28.714 INFO:tasks.rados.rados.0.smithi107.stdout:6458: writing smithi10729407-14 from 2318336 to 2465792 tid 1 2020-06-12T02:38:28.722 INFO:tasks.rados.rados.0.smithi107.stdout:6459: delete oid 8 current snap is 0 2020-06-12T02:38:28.722 INFO:tasks.rados.rados.0.smithi107.stdout:6455: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:28.722 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1180 (ObjNum 6454 snap 0 seq_num 6454) dirty exists 2020-06-12T02:38:28.723 INFO:tasks.rados.rados.0.smithi107.stdout:6455: left oid 1 (ObjNum 6454 snap 0 seq_num 6454) 2020-06-12T02:38:28.723 INFO:tasks.rados.rados.0.smithi107.stdout:6455: done (3 left) 2020-06-12T02:38:28.723 INFO:tasks.rados.rados.0.smithi107.stdout:6459: done (2 left) 2020-06-12T02:38:28.724 INFO:tasks.rados.rados.0.smithi107.stdout:6460: delete oid 15 current snap is 0 2020-06-12T02:38:28.724 INFO:tasks.rados.rados.0.smithi107.stdout:6458: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:28.724 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:38:28.724 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:38:28.724 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2020-06-12T02:38:28.725 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-06-12T02:38:28.725 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-06-12T02:38:28.725 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-06-12T02:38:28.725 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-06-12T02:38:28.726 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-06-12T02:38:28.726 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-06-12T02:38:28.726 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-06-12T02:38:28.727 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-06-12T02:38:28.727 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.727 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:28.727 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072409+0000", 2020-06-12T02:38:28.728 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:28.728 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.728 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.728 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:28.728 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072409+0000", 2020-06-12T02:38:28.728 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000446317 2020-06-12T02:38:28.729 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.729 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.729 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:28.729 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072856+0000", 2020-06-12T02:38:28.729 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2140000000000004e-06 2020-06-12T02:38:28.730 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.730 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.730 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:28.730 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072861+0000", 2020-06-12T02:38:28.730 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8710000000000003e-06 2020-06-12T02:38:28.731 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.731 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.731 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:28.731 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072870+0000", 2020-06-12T02:38:28.731 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2270000000000004e-06 2020-06-12T02:38:28.731 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.732 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.732 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:28.732 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072874+0000", 2020-06-12T02:38:28.732 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024649960000000001 2020-06-12T02:38:28.733 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.733 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.733 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:28.733 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075339+0000", 2020-06-12T02:38:28.734 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9179999999999996e-05 2020-06-12T02:38:28.734 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.734 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.734 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:28.735 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075438+0000", 2020-06-12T02:38:28.735 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60189281400000005 2020-06-12T02:38:28.735 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.735 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.735 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:28.735 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677331+0000", 2020-06-12T02:38:28.736 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012602800000000001 2020-06-12T02:38:28.736 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.737 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.737 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:28.737 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677457+0000", 2020-06-12T02:38:28.737 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00088167699999999998 2020-06-12T02:38:28.738 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.738 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.738 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:28.738 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.678339+0000", 2020-06-12T02:38:28.738 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004312995 2020-06-12T02:38:28.738 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.739 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.739 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:28.739 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.682652+0000", 2020-06-12T02:38:28.739 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0092869140000000003 2020-06-12T02:38:28.739 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.740 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.740 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:28.740 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691938+0000", 2020-06-12T02:38:28.740 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8778e-05 2020-06-12T02:38:28.740 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.740 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.741 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:28.741 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691977+0000", 2020-06-12T02:38:28.741 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.61956801100000003 2020-06-12T02:38:28.741 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:28.741 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:28.742 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:28.742 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.742 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.742 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2915 3.5s0 3:a9510bda:::smithi10729407-14:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:28.742 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.072912+0000", 2020-06-12T02:38:28.743 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 51.528555638999997, 2020-06-12T02:38:28.743 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.6192204, 2020-06-12T02:38:28.743 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:28.743 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:28.743 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:28.744 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:28.744 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:28.744 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2915 2020-06-12T02:38:28.744 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.745 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:28.745 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.745 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:28.745 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072912+0000", 2020-06-12T02:38:28.745 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:28.746 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.746 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.746 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:28.746 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072912+0000", 2020-06-12T02:38:28.746 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9201e-05 2020-06-12T02:38:28.747 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.747 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.747 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:28.747 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072951+0000", 2020-06-12T02:38:28.747 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5480000000000002e-06 2020-06-12T02:38:28.748 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.748 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.748 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:28.748 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072955+0000", 2020-06-12T02:38:28.748 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3879999999999997e-06 2020-06-12T02:38:28.748 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.749 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.749 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:28.749 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072961+0000", 2020-06-12T02:38:28.749 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6589999999999999e-06 2020-06-12T02:38:28.749 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.750 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.750 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:28.750 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072964+0000", 2020-06-12T02:38:28.750 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025221039999999998 2020-06-12T02:38:28.750 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.750 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.751 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:28.751 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075486+0000", 2020-06-12T02:38:28.751 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5038999999999998e-05 2020-06-12T02:38:28.751 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.751 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.752 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:28.752 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075561+0000", 2020-06-12T02:38:28.752 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60452397499999999 2020-06-12T02:38:28.753 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.753 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.754 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:28.754 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680085+0000", 2020-06-12T02:38:28.754 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1398999999999999e-05 2020-06-12T02:38:28.754 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.754 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.755 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:28.755 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680156+0000", 2020-06-12T02:38:28.755 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000274465 2020-06-12T02:38:28.755 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.755 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.756 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:28.756 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680431+0000", 2020-06-12T02:38:28.756 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025071999999999998 2020-06-12T02:38:28.756 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.756 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.756 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:28.757 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.682938+0000", 2020-06-12T02:38:28.757 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0091116449999999998 2020-06-12T02:38:28.757 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.757 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.757 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:28.758 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692050+0000", 2020-06-12T02:38:28.758 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2776999999999997e-05 2020-06-12T02:38:28.758 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.758 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.758 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:28.758 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692132+0000", 2020-06-12T02:38:28.759 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.6192204 2020-06-12T02:38:28.759 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:28.759 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:28.759 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:28.759 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.760 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.760 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2916 3.5s0 3:a9510bda:::smithi10729407-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:38:28.760 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.072986+0000", 2020-06-12T02:38:28.760 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 51.528481399999997, 2020-06-12T02:38:28.760 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.62063941099999997, 2020-06-12T02:38:28.760 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:28.761 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:28.761 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:28.761 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:28.761 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:28.761 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2916 2020-06-12T02:38:28.762 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.762 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:28.762 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.762 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:28.762 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072986+0000", 2020-06-12T02:38:28.763 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:28.763 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.763 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.763 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:28.763 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072986+0000", 2020-06-12T02:38:28.763 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2101000000000001e-05 2020-06-12T02:38:28.764 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.764 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.764 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:28.764 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073008+0000", 2020-06-12T02:38:28.764 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1549999999999999e-06 2020-06-12T02:38:28.765 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.765 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.765 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:28.765 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073011+0000", 2020-06-12T02:38:28.765 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2360000000000001e-06 2020-06-12T02:38:28.765 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.770 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.770 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:28.771 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073018+0000", 2020-06-12T02:38:28.771 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2790000000000001e-06 2020-06-12T02:38:28.771 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.771 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.771 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:28.771 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073020+0000", 2020-06-12T02:38:28.772 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025637670000000002 2020-06-12T02:38:28.772 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.772 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.772 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:28.772 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075584+0000", 2020-06-12T02:38:28.773 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1814999999999998e-05 2020-06-12T02:38:28.773 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.773 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.773 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:28.773 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075636+0000", 2020-06-12T02:38:28.774 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60499943199999995 2020-06-12T02:38:28.774 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.774 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.774 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:28.774 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680635+0000", 2020-06-12T02:38:28.774 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0968000000000002e-05 2020-06-12T02:38:28.775 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.775 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.775 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:28.775 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680706+0000", 2020-06-12T02:38:28.775 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011777101999999999 2020-06-12T02:38:28.776 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.776 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.776 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:28.776 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692483+0000", 2020-06-12T02:38:28.776 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5890999999999995e-05 2020-06-12T02:38:28.777 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.777 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.777 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:28.777 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692569+0000", 2020-06-12T02:38:28.777 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001056665 2020-06-12T02:38:28.777 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.778 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.778 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:28.778 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.693626+0000", 2020-06-12T02:38:28.778 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.62063941099999997 2020-06-12T02:38:28.778 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:28.779 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:28.779 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:28.779 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.779 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.779 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2922 3.ds1 3:b983fd28:::smithi10729407-1:head [append 0~466944] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:28.780 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.097192+0000", 2020-06-12T02:38:28.780 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 51.504276032999996, 2020-06-12T02:38:28.780 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58672644699999998, 2020-06-12T02:38:28.780 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:28.780 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:28.781 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:28.781 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:28.781 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:28.781 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2922 2020-06-12T02:38:28.781 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.781 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:28.782 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.782 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:28.782 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097192+0000", 2020-06-12T02:38:28.782 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:28.782 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.783 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.783 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:28.783 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097192+0000", 2020-06-12T02:38:28.783 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047206100000000002 2020-06-12T02:38:28.783 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.784 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.784 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:28.784 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097664+0000", 2020-06-12T02:38:28.784 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8359999999999999e-06 2020-06-12T02:38:28.784 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.784 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.785 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:28.785 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097668+0000", 2020-06-12T02:38:28.785 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0815999999999999e-05 2020-06-12T02:38:28.785 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.785 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.786 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:28.786 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097679+0000", 2020-06-12T02:38:28.786 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.028e-06 2020-06-12T02:38:28.786 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.786 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.786 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:28.787 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097684+0000", 2020-06-12T02:38:28.787 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3932999999999999e-05 2020-06-12T02:38:28.787 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.787 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.787 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:28.788 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097748+0000", 2020-06-12T02:38:28.788 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2214999999999999e-05 2020-06-12T02:38:28.788 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.788 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.788 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:28.789 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097820+0000", 2020-06-12T02:38:28.789 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00041583800000000003 2020-06-12T02:38:28.789 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.789 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.789 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:28.791 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.098236+0000", 2020-06-12T02:38:28.791 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0020339960000000002 2020-06-12T02:38:28.791 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.791 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.792 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:28.792 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.100270+0000", 2020-06-12T02:38:28.792 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58355657299999997 2020-06-12T02:38:28.792 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.792 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.793 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:28.793 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.683827+0000", 2020-06-12T02:38:28.793 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1150999999999995e-05 2020-06-12T02:38:28.793 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.793 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.793 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:28.794 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.683918+0000", 2020-06-12T02:38:28.794 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58672644699999998 2020-06-12T02:38:28.794 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:28.794 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:28.794 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:28.795 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.795 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.796 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2923 3.ds1 3:b983fd28:::smithi10729407-1:head [append 0~614400] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:28.796 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.100504+0000", 2020-06-12T02:38:28.796 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 51.500963392000003, 2020-06-12T02:38:28.796 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58853946999999995, 2020-06-12T02:38:28.797 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:28.797 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:28.797 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:28.797 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:28.797 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:28.798 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2923 2020-06-12T02:38:28.798 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.798 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:28.798 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.798 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:28.799 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.100504+0000", 2020-06-12T02:38:28.799 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:28.799 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.799 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.799 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:28.800 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.100504+0000", 2020-06-12T02:38:28.800 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00068395399999999996 2020-06-12T02:38:28.800 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.800 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.800 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:28.801 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101188+0000", 2020-06-12T02:38:28.801 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4050000000000003e-06 2020-06-12T02:38:28.801 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.801 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.801 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:28.802 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101196+0000", 2020-06-12T02:38:28.802 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.328e-05 2020-06-12T02:38:28.802 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.803 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.803 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:28.803 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101209+0000", 2020-06-12T02:38:28.803 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3419999999999996e-06 2020-06-12T02:38:28.804 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.804 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.804 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:28.804 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101215+0000", 2020-06-12T02:38:28.804 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1389000000000005e-05 2020-06-12T02:38:28.805 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.805 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.805 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:28.805 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101277+0000", 2020-06-12T02:38:28.805 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7724000000000004e-05 2020-06-12T02:38:28.806 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.806 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.806 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:28.806 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101344+0000", 2020-06-12T02:38:28.807 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047852499999999999 2020-06-12T02:38:28.807 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.807 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.807 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:28.807 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101823+0000", 2020-06-12T02:38:28.808 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0039255510000000002 2020-06-12T02:38:28.808 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.808 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.808 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:28.808 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105748+0000", 2020-06-12T02:38:28.808 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58321340099999996 2020-06-12T02:38:28.809 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.809 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.809 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:28.809 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.688962+0000", 2020-06-12T02:38:28.809 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1898999999999996e-05 2020-06-12T02:38:28.810 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.810 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.810 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:28.810 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689044+0000", 2020-06-12T02:38:28.810 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58853946999999995 2020-06-12T02:38:28.811 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:28.811 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:28.811 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:28.811 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.811 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.812 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2924 3.ds1 3:b983fd28:::smithi10729407-1:head [append 0~188416] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:28.812 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.101633+0000", 2020-06-12T02:38:28.812 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 51.499834397999997, 2020-06-12T02:38:28.812 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58758259199999996, 2020-06-12T02:38:28.813 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:28.813 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:28.813 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:28.813 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:28.813 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:28.814 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2924 2020-06-12T02:38:28.814 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.814 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:28.814 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.814 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:28.815 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101633+0000", 2020-06-12T02:38:28.815 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:28.815 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.815 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.815 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:28.816 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101633+0000", 2020-06-12T02:38:28.816 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00018775399999999999 2020-06-12T02:38:28.816 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.816 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.817 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:28.817 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101821+0000", 2020-06-12T02:38:28.817 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7669999999999998e-06 2020-06-12T02:38:28.817 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.818 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.818 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:28.818 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101828+0000", 2020-06-12T02:38:28.818 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2716e-05 2020-06-12T02:38:28.818 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.818 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.819 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:28.819 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101840+0000", 2020-06-12T02:38:28.819 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.215e-06 2020-06-12T02:38:28.819 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.819 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.820 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:28.820 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101846+0000", 2020-06-12T02:38:28.820 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021214250000000001 2020-06-12T02:38:28.820 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.820 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.820 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:28.821 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.103967+0000", 2020-06-12T02:38:28.821 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1018000000000003e-05 2020-06-12T02:38:28.821 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.821 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.821 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:28.822 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.104038+0000", 2020-06-12T02:38:28.822 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000333364 2020-06-12T02:38:28.822 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.823 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.823 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:28.823 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.104371+0000", 2020-06-12T02:38:28.823 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001378546 2020-06-12T02:38:28.823 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.823 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.824 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:28.824 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105750+0000", 2020-06-12T02:38:28.824 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58338272899999999 2020-06-12T02:38:28.824 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.825 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.825 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:28.825 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689133+0000", 2020-06-12T02:38:28.825 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3058000000000003e-05 2020-06-12T02:38:28.825 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.826 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.826 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:28.826 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689216+0000", 2020-06-12T02:38:28.826 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58758259199999996 2020-06-12T02:38:28.827 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:28.827 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:28.827 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:28.827 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.828 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.828 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2925 3.ds1 3:b983fd28:::smithi10729407-1:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:28.828 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.101899+0000", 2020-06-12T02:38:28.828 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 51.499568732999997, 2020-06-12T02:38:28.828 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58747810899999997, 2020-06-12T02:38:28.829 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:28.829 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:28.829 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:28.829 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:28.829 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:28.830 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2925 2020-06-12T02:38:28.830 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.830 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:28.830 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.830 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:28.830 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101899+0000", 2020-06-12T02:38:28.831 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:28.831 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.831 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.831 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:28.831 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101899+0000", 2020-06-12T02:38:28.832 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2906999999999998e-05 2020-06-12T02:38:28.832 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.832 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.832 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:28.833 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101972+0000", 2020-06-12T02:38:28.833 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1789999999999996e-06 2020-06-12T02:38:28.833 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.833 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.833 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:28.834 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101978+0000", 2020-06-12T02:38:28.834 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1358e-05 2020-06-12T02:38:28.834 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.835 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.835 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:28.835 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101989+0000", 2020-06-12T02:38:28.835 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.031e-06 2020-06-12T02:38:28.835 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.836 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.836 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:28.836 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101993+0000", 2020-06-12T02:38:28.837 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033218990000000001 2020-06-12T02:38:28.837 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.837 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.837 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:28.837 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105315+0000", 2020-06-12T02:38:28.838 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6324999999999999e-05 2020-06-12T02:38:28.838 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.838 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.838 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:28.838 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105382+0000", 2020-06-12T02:38:28.839 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000216624 2020-06-12T02:38:28.839 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.839 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.839 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:28.840 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105598+0000", 2020-06-12T02:38:28.840 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001236586 2020-06-12T02:38:28.840 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.840 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.841 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:28.841 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.106835+0000", 2020-06-12T02:38:28.841 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58245509200000001 2020-06-12T02:38:28.841 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.841 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.842 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:28.842 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689290+0000", 2020-06-12T02:38:28.842 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7107999999999993e-05 2020-06-12T02:38:28.842 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.842 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.843 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:28.843 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689377+0000", 2020-06-12T02:38:28.843 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58747810899999997 2020-06-12T02:38:28.843 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:28.843 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:28.844 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:28.844 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.844 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.844 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2926 3.ds1 3:b983fd28:::smithi10729407-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:38:28.844 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.102036+0000", 2020-06-12T02:38:28.844 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 51.499431749000003, 2020-06-12T02:38:28.845 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58868048699999997, 2020-06-12T02:38:28.845 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:28.845 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:28.846 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:28.846 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:28.846 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:28.847 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2926 2020-06-12T02:38:28.847 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.848 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:28.848 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.848 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:28.849 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102036+0000", 2020-06-12T02:38:28.849 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:28.849 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.850 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.850 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:28.850 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102036+0000", 2020-06-12T02:38:28.851 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3786999999999999e-05 2020-06-12T02:38:28.851 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.851 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.851 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:28.852 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102080+0000", 2020-06-12T02:38:28.852 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7660000000000004e-06 2020-06-12T02:38:28.852 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.853 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.853 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:28.853 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102085+0000", 2020-06-12T02:38:28.854 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3145e-05 2020-06-12T02:38:28.854 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.854 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.855 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:28.855 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102099+0000", 2020-06-12T02:38:28.855 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2440000000000004e-06 2020-06-12T02:38:28.856 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.856 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.856 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:28.857 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102103+0000", 2020-06-12T02:38:28.857 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036826990000000002 2020-06-12T02:38:28.858 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.858 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.858 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:28.859 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105786+0000", 2020-06-12T02:38:28.859 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.6774000000000006e-05 2020-06-12T02:38:28.860 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.860 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.861 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:28.861 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105882+0000", 2020-06-12T02:38:28.861 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58352084900000001 2020-06-12T02:38:28.862 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.862 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.862 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:28.863 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689403+0000", 2020-06-12T02:38:28.863 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3788999999999995e-05 2020-06-12T02:38:28.863 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.863 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.864 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:28.864 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689467+0000", 2020-06-12T02:38:28.864 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001249434 2020-06-12T02:38:28.865 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.865 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.865 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:28.866 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690716+0000", 2020-06-12T02:38:28.866 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58868048699999997 2020-06-12T02:38:28.866 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:28.867 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:28.867 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:28.868 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.868 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.868 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2950 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~598016] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:28.869 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.162562+0000", 2020-06-12T02:38:28.869 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 51.438905374999997, 2020-06-12T02:38:28.869 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52780611399999999, 2020-06-12T02:38:28.870 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:28.870 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:28.870 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:28.871 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:28.871 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:28.871 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2950 2020-06-12T02:38:28.872 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.872 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:28.872 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.872 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:28.873 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.162562+0000", 2020-06-12T02:38:28.873 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:28.873 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.874 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.874 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:28.874 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.162562+0000", 2020-06-12T02:38:28.875 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00070718099999999998 2020-06-12T02:38:28.875 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.875 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.875 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:28.875 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163269+0000", 2020-06-12T02:38:28.876 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2029999999999997e-06 2020-06-12T02:38:28.876 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.876 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.876 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:28.876 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163276+0000", 2020-06-12T02:38:28.876 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2734999999999999e-05 2020-06-12T02:38:28.877 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.877 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.877 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:28.877 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163288+0000", 2020-06-12T02:38:28.877 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.404e-06 2020-06-12T02:38:28.878 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.878 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.878 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:28.879 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163295+0000", 2020-06-12T02:38:28.879 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5784000000000006e-05 2020-06-12T02:38:28.879 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.879 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.879 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:28.880 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163361+0000", 2020-06-12T02:38:28.880 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5318e-05 2020-06-12T02:38:28.880 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.880 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.880 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:28.880 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163436+0000", 2020-06-12T02:38:28.881 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054801100000000005 2020-06-12T02:38:28.881 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.881 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.881 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:28.881 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163984+0000", 2020-06-12T02:38:28.882 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.007023077 2020-06-12T02:38:28.882 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.882 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.882 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:28.882 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171007+0000", 2020-06-12T02:38:28.883 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51932051199999996 2020-06-12T02:38:28.883 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.883 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.883 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:28.883 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690327+0000", 2020-06-12T02:38:28.883 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0889000000000001e-05 2020-06-12T02:38:28.884 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.884 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.884 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:28.884 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690368+0000", 2020-06-12T02:38:28.884 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52780611399999999 2020-06-12T02:38:28.885 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:28.885 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:28.885 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:28.885 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.885 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.885 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2951 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~532480] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:28.886 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.165470+0000", 2020-06-12T02:38:28.886 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 51.435997767000003, 2020-06-12T02:38:28.886 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52507917900000001, 2020-06-12T02:38:28.886 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:28.886 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:28.887 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:28.888 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:28.888 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:28.889 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2951 2020-06-12T02:38:28.889 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.889 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:28.890 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.890 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:28.890 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.165470+0000", 2020-06-12T02:38:28.891 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:28.891 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.891 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.891 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:28.891 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.165470+0000", 2020-06-12T02:38:28.892 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00062231600000000004 2020-06-12T02:38:28.892 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.892 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.892 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:28.892 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166092+0000", 2020-06-12T02:38:28.893 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1610000000000002e-06 2020-06-12T02:38:28.893 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.893 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.893 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:28.893 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166097+0000", 2020-06-12T02:38:28.894 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7780000000000002e-06 2020-06-12T02:38:28.894 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.894 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.894 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:28.894 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166107+0000", 2020-06-12T02:38:28.895 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7169999999999999e-06 2020-06-12T02:38:28.895 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.895 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.895 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:28.895 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166112+0000", 2020-06-12T02:38:28.895 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4792000000000002e-05 2020-06-12T02:38:28.896 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.896 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.896 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:28.898 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166147+0000", 2020-06-12T02:38:28.898 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9336e-05 2020-06-12T02:38:28.898 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.899 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.899 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:28.899 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166236+0000", 2020-06-12T02:38:28.899 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00067747200000000004 2020-06-12T02:38:28.899 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.900 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.900 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:28.900 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166913+0000", 2020-06-12T02:38:28.900 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0047718669999999999 2020-06-12T02:38:28.900 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.901 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.901 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:28.901 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171685+0000", 2020-06-12T02:38:28.901 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51881990700000002 2020-06-12T02:38:28.901 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.901 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.902 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:28.902 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690505+0000", 2020-06-12T02:38:28.902 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3832999999999999e-05 2020-06-12T02:38:28.902 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.902 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.903 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:28.903 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690549+0000", 2020-06-12T02:38:28.903 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52507917900000001 2020-06-12T02:38:28.903 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:28.903 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:28.904 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:28.904 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.904 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.904 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2952 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~270336] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:28.904 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.166891+0000", 2020-06-12T02:38:28.905 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 51.434576118000003, 2020-06-12T02:38:28.905 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52426417000000003, 2020-06-12T02:38:28.905 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:28.905 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:28.905 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:28.905 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:28.906 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:28.906 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2952 2020-06-12T02:38:28.906 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.907 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:28.907 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.907 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:28.907 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166891+0000", 2020-06-12T02:38:28.907 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:28.908 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.908 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.908 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:28.908 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166891+0000", 2020-06-12T02:38:28.908 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00031878599999999999 2020-06-12T02:38:28.909 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.909 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.909 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:28.909 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167210+0000", 2020-06-12T02:38:28.909 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2360000000000004e-06 2020-06-12T02:38:28.910 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.910 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.910 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:28.910 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167219+0000", 2020-06-12T02:38:28.910 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3298e-05 2020-06-12T02:38:28.911 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.911 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.911 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:28.911 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167232+0000", 2020-06-12T02:38:28.911 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5849999999999999e-06 2020-06-12T02:38:28.911 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.912 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.912 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:28.912 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167237+0000", 2020-06-12T02:38:28.912 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0018755740000000001 2020-06-12T02:38:28.912 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.913 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.913 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:28.913 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.169113+0000", 2020-06-12T02:38:28.913 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000104768 2020-06-12T02:38:28.913 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.914 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.914 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:28.914 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.169218+0000", 2020-06-12T02:38:28.914 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00057389799999999999 2020-06-12T02:38:28.914 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.915 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.915 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:28.915 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.169792+0000", 2020-06-12T02:38:28.915 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0019333989999999999 2020-06-12T02:38:28.916 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.916 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.916 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:28.916 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171725+0000", 2020-06-12T02:38:28.916 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51938508699999997 2020-06-12T02:38:28.917 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.917 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.917 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:28.917 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691110+0000", 2020-06-12T02:38:28.917 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5538999999999999e-05 2020-06-12T02:38:28.918 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.918 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.918 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:28.918 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691156+0000", 2020-06-12T02:38:28.918 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52426417000000003 2020-06-12T02:38:28.918 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:28.919 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:28.919 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:28.919 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.919 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.919 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2953 3.5s0 3:a446366d:::smithi10729407-11:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:28.920 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.167293+0000", 2020-06-12T02:38:28.920 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 51.434174671000001, 2020-06-12T02:38:28.920 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52428350999999995, 2020-06-12T02:38:28.920 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:28.920 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:28.921 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:28.921 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:28.921 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:28.921 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2953 2020-06-12T02:38:28.921 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.921 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:28.922 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.922 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:28.931 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167293+0000", 2020-06-12T02:38:28.931 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:28.932 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.932 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.932 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:28.932 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167293+0000", 2020-06-12T02:38:28.932 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3880000000000004e-05 2020-06-12T02:38:28.933 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.933 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.933 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:28.933 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167367+0000", 2020-06-12T02:38:28.933 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.319e-06 2020-06-12T02:38:28.934 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.934 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.934 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:28.934 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167373+0000", 2020-06-12T02:38:28.934 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1719000000000001e-05 2020-06-12T02:38:28.935 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.935 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.935 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:28.935 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167385+0000", 2020-06-12T02:38:28.935 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4580000000000003e-06 2020-06-12T02:38:28.936 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.936 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.936 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:28.936 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167389+0000", 2020-06-12T02:38:28.936 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036341110000000002 2020-06-12T02:38:28.936 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.937 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.937 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:28.937 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171023+0000", 2020-06-12T02:38:28.937 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9418000000000005e-05 2020-06-12T02:38:28.938 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.938 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.938 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:28.939 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171093+0000", 2020-06-12T02:38:28.939 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00029087900000000002 2020-06-12T02:38:28.939 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.939 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.939 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:28.939 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171384+0000", 2020-06-12T02:38:28.940 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000489347 2020-06-12T02:38:28.940 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.940 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.940 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:28.940 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171873+0000", 2020-06-12T02:38:28.941 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51963691300000003 2020-06-12T02:38:28.941 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.941 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.941 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:28.941 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691510+0000", 2020-06-12T02:38:28.942 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6465999999999997e-05 2020-06-12T02:38:28.942 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.942 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.942 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:28.942 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691576+0000", 2020-06-12T02:38:28.942 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52428350999999995 2020-06-12T02:38:28.943 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:28.943 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:28.943 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:28.944 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.944 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.944 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2954 3.5s0 3:a446366d:::smithi10729407-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:38:28.944 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.167432+0000", 2020-06-12T02:38:28.945 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 51.434035922, 2020-06-12T02:38:28.945 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52568652699999996, 2020-06-12T02:38:28.945 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:28.945 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:28.945 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:28.946 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:28.946 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:28.946 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2954 2020-06-12T02:38:28.946 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.946 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:28.946 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.947 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:28.947 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167432+0000", 2020-06-12T02:38:28.947 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:28.947 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.947 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.948 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:28.948 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167432+0000", 2020-06-12T02:38:28.948 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3556000000000001e-05 2020-06-12T02:38:28.948 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.948 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.949 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:28.949 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167475+0000", 2020-06-12T02:38:28.949 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7350000000000001e-06 2020-06-12T02:38:28.949 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.949 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.950 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:28.950 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167481+0000", 2020-06-12T02:38:28.950 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3128e-05 2020-06-12T02:38:28.950 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.950 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.951 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:28.951 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167494+0000", 2020-06-12T02:38:28.951 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1579999999999998e-06 2020-06-12T02:38:28.951 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.952 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.952 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:28.952 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167498+0000", 2020-06-12T02:38:28.952 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004082174 2020-06-12T02:38:28.952 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.953 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.953 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:28.953 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171580+0000", 2020-06-12T02:38:28.953 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8954000000000002e-05 2020-06-12T02:38:28.953 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.954 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.954 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:28.954 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171679+0000", 2020-06-12T02:38:28.954 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51993235100000001 2020-06-12T02:38:28.954 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.955 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.955 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:28.955 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691612+0000", 2020-06-12T02:38:28.955 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4329000000000001e-05 2020-06-12T02:38:28.955 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.955 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.956 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:28.956 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691666+0000", 2020-06-12T02:38:28.956 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001452142 2020-06-12T02:38:28.956 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.956 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.957 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:28.957 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.693118+0000", 2020-06-12T02:38:28.957 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52568652699999996 2020-06-12T02:38:28.957 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:28.957 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:28.958 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:28.958 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.959 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.959 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2959 3.ds1 3:b3445631:::smithi10729407-6:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e27)", 2020-06-12T02:38:28.959 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.176248+0000", 2020-06-12T02:38:28.959 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 51.425219181000003, 2020-06-12T02:38:28.960 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51549877099999997, 2020-06-12T02:38:28.960 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:28.960 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:28.960 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:28.960 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:28.961 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:28.961 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2959 2020-06-12T02:38:28.961 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.961 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:28.961 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.962 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:28.962 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176248+0000", 2020-06-12T02:38:28.962 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:28.962 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.962 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.963 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:28.963 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176248+0000", 2020-06-12T02:38:28.963 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5120999999999999e-05 2020-06-12T02:38:28.963 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.963 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.964 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:28.964 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176274+0000", 2020-06-12T02:38:28.964 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9189999999999998e-06 2020-06-12T02:38:28.964 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.964 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.965 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:28.965 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176278+0000", 2020-06-12T02:38:28.966 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6339e-05 2020-06-12T02:38:28.966 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.966 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.966 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:28.966 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176305+0000", 2020-06-12T02:38:28.967 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9089999999999999e-06 2020-06-12T02:38:28.967 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.967 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.967 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:28.968 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176310+0000", 2020-06-12T02:38:28.968 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6095999999999999e-05 2020-06-12T02:38:28.968 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.968 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.968 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:28.969 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176376+0000", 2020-06-12T02:38:28.969 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8590999999999994e-05 2020-06-12T02:38:28.969 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.969 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.969 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:28.970 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176444+0000", 2020-06-12T02:38:28.970 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00024346400000000001 2020-06-12T02:38:28.970 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.970 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.970 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:28.971 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176688+0000", 2020-06-12T02:38:28.971 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00044220899999999999 2020-06-12T02:38:28.971 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.971 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.971 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:28.972 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.177130+0000", 2020-06-12T02:38:28.972 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51452984400000001 2020-06-12T02:38:28.972 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.972 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.972 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:28.973 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691660+0000", 2020-06-12T02:38:28.973 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7279000000000002e-05 2020-06-12T02:38:28.973 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.973 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.973 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:28.974 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691747+0000", 2020-06-12T02:38:28.974 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51549877099999997 2020-06-12T02:38:28.974 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:28.974 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:28.974 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:28.975 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.975 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.975 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:3032 3.ds1 3:b983fd28:::smithi10729407-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:38:28.975 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.864289+0000", 2020-06-12T02:38:28.975 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 50.737178262999997, 2020-06-12T02:38:28.976 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019572216999999999, 2020-06-12T02:38:28.976 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:28.976 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:28.976 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:28.976 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:28.977 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:28.977 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 3032 2020-06-12T02:38:28.977 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.977 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:28.977 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.978 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:28.978 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864289+0000", 2020-06-12T02:38:28.978 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:28.979 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.979 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.979 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:28.979 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864289+0000", 2020-06-12T02:38:28.979 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5701999999999997e-05 2020-06-12T02:38:28.980 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.980 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.980 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:28.980 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864335+0000", 2020-06-12T02:38:28.980 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0910000000000004e-06 2020-06-12T02:38:28.980 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.981 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.981 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:28.981 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864342+0000", 2020-06-12T02:38:28.981 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2301e-05 2020-06-12T02:38:28.981 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.982 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.982 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:28.982 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864354+0000", 2020-06-12T02:38:28.982 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5199999999999997e-06 2020-06-12T02:38:28.982 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.983 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.983 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:28.983 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864360+0000", 2020-06-12T02:38:28.983 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00049180999999999997 2020-06-12T02:38:28.983 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.984 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.984 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:28.984 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864852+0000", 2020-06-12T02:38:28.984 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6725000000000003e-05 2020-06-12T02:38:28.984 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.985 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.985 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:28.985 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864908+0000", 2020-06-12T02:38:28.985 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0089687010000000008 2020-06-12T02:38:28.985 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.986 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.986 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:28.986 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.873877+0000", 2020-06-12T02:38:28.986 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3382999999999999e-05 2020-06-12T02:38:28.986 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.987 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.987 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:28.987 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.873941+0000", 2020-06-12T02:38:28.988 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.006408927 2020-06-12T02:38:28.988 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.989 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.989 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:28.989 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.880350+0000", 2020-06-12T02:38:28.989 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3905e-05 2020-06-12T02:38:28.989 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.990 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.990 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:28.990 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.880403+0000", 2020-06-12T02:38:28.990 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0034581519999999999 2020-06-12T02:38:28.990 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:28.991 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:28.991 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:28.991 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.883862+0000", 2020-06-12T02:38:28.992 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019572216999999999 2020-06-12T02:38:28.992 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:28.993 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:28.993 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:28.993 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:28.993 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:28.993 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:28.994 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:38:28.687+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:38:28.994 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-06-12T02:38:29.007 INFO:tasks.rados.rados.0.smithi107.stdout:6460: done (2 left) 2020-06-12T02:38:29.008 INFO:tasks.rados.rados.0.smithi107.stdout:6461: delete oid 13 current snap is 0 2020-06-12T02:38:29.008 INFO:tasks.rados.rados.0.smithi107.stdout:6458: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:29.008 INFO:tasks.rados.rados.0.smithi107.stdout:6461: done (2 left) 2020-06-12T02:38:29.008 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:38:29.008 INFO:tasks.rados.rados.0.smithi107.stdout:6462: seq_num 6461 ranges {1982464=212992} 2020-06-12T02:38:29.009 INFO:tasks.rados.rados.0.smithi107.stdout:6462: writing smithi10729407-16 from 1982464 to 2195456 tid 1 2020-06-12T02:38:29.009 INFO:tasks.rados.rados.0.smithi107.stdout:6463: delete oid 18 current snap is 0 2020-06-12T02:38:29.009 INFO:tasks.rados.rados.0.smithi107.stdout:6458: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:29.009 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1161 (ObjNum 6457 snap 0 seq_num 6457) dirty exists 2020-06-12T02:38:29.009 INFO:tasks.rados.rados.0.smithi107.stdout:6458: left oid 14 (ObjNum 6457 snap 0 seq_num 6457) 2020-06-12T02:38:29.010 INFO:tasks.rados.rados.0.smithi107.stdout:6458: done (3 left) 2020-06-12T02:38:29.010 INFO:tasks.rados.rados.0.smithi107.stdout:6463: done (2 left) 2020-06-12T02:38:29.010 INFO:tasks.rados.rados.0.smithi107.stdout:6464: delete oid 7 current snap is 0 2020-06-12T02:38:29.010 INFO:tasks.rados.rados.0.smithi107.stdout:6462: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:29.010 INFO:tasks.rados.rados.0.smithi107.stdout:6462: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:29.010 INFO:tasks.rados.rados.0.smithi107.stdout:6464: done (2 left) 2020-06-12T02:38:29.011 INFO:tasks.rados.rados.0.smithi107.stdout:6465: delete oid 4 current snap is 0 2020-06-12T02:38:29.011 INFO:tasks.rados.rados.0.smithi107.stdout:6462: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:29.011 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1136 (ObjNum 6461 snap 0 seq_num 6461) dirty exists 2020-06-12T02:38:29.011 INFO:tasks.rados.rados.0.smithi107.stdout:6462: left oid 16 (ObjNum 6461 snap 0 seq_num 6461) 2020-06-12T02:38:29.011 INFO:tasks.rados.rados.0.smithi107.stdout:6462: done (2 left) 2020-06-12T02:38:29.012 INFO:tasks.rados.rados.0.smithi107.stdout:6465: done (1 left) 2020-06-12T02:38:29.012 INFO:tasks.rados.rados.0.smithi107.stdout:6466: delete oid 15 current snap is 0 2020-06-12T02:38:29.012 INFO:tasks.rados.rados.0.smithi107.stdout:6466: done (1 left) 2020-06-12T02:38:29.012 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:29.012 INFO:tasks.rados.rados.0.smithi107.stdout:6467: seq_num 6466 ranges {2465792=294912} 2020-06-12T02:38:29.012 INFO:tasks.rados.rados.0.smithi107.stdout:6467: writing smithi10729407-14 from 2465792 to 2760704 tid 1 2020-06-12T02:38:29.013 INFO:tasks.rados.rados.0.smithi107.stdout:6468: delete oid 4 current snap is 0 2020-06-12T02:38:29.013 INFO:tasks.rados.rados.0.smithi107.stdout:6468: done (2 left) 2020-06-12T02:38:29.013 INFO:tasks.rados.rados.0.smithi107.stdout:6469: delete oid 20 current snap is 0 2020-06-12T02:38:29.013 INFO:tasks.rados.rados.0.smithi107.stdout:6467: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:29.013 INFO:tasks.rados.rados.0.smithi107.stdout:6469: done (2 left) 2020-06-12T02:38:29.014 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:38:29.014 INFO:tasks.rados.rados.0.smithi107.stdout:6470: seq_num 6469 ranges {2195456=352256} 2020-06-12T02:38:29.014 INFO:tasks.rados.rados.0.smithi107.stdout:6470: writing smithi10729407-16 from 2195456 to 2547712 tid 1 2020-06-12T02:38:29.015 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:29.015 INFO:tasks.rados.rados.0.smithi107.stdout:6471: seq_num 6470 ranges {0=368640} 2020-06-12T02:38:29.015 INFO:tasks.rados.rados.0.smithi107.stdout:6471: writing smithi10729407-7 from 0 to 368640 tid 1 2020-06-12T02:38:29.016 INFO:tasks.rados.rados.0.smithi107.stdout:6472: delete oid 15 current snap is 0 2020-06-12T02:38:29.016 INFO:tasks.rados.rados.0.smithi107.stdout:6467: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:29.016 INFO:tasks.rados.rados.0.smithi107.stdout:6467: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:29.016 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1163 (ObjNum 6466 snap 0 seq_num 6466) dirty exists 2020-06-12T02:38:29.016 INFO:tasks.rados.rados.0.smithi107.stdout:6467: left oid 14 (ObjNum 6466 snap 0 seq_num 6466) 2020-06-12T02:38:29.017 INFO:tasks.rados.rados.0.smithi107.stdout:6467: done (4 left) 2020-06-12T02:38:29.017 INFO:tasks.rados.rados.0.smithi107.stdout:6472: done (3 left) 2020-06-12T02:38:29.017 INFO:tasks.rados.rados.0.smithi107.stdout:6473: delete oid 19 current snap is 0 2020-06-12T02:38:29.017 INFO:tasks.rados.rados.0.smithi107.stdout:6470: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:29.018 INFO:tasks.rados.rados.0.smithi107.stdout:6470: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:29.018 INFO:tasks.rados.rados.0.smithi107.stdout:6470: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:29.018 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1138 (ObjNum 6469 snap 0 seq_num 6469) dirty exists 2020-06-12T02:38:29.018 INFO:tasks.rados.rados.0.smithi107.stdout:6470: left oid 16 (ObjNum 6469 snap 0 seq_num 6469) 2020-06-12T02:38:29.018 INFO:tasks.rados.rados.0.smithi107.stdout:6471: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:29.019 INFO:tasks.rados.rados.0.smithi107.stdout:6470: done (3 left) 2020-06-12T02:38:29.019 INFO:tasks.rados.rados.0.smithi107.stdout:6473: done (2 left) 2020-06-12T02:38:29.019 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:29.019 INFO:tasks.rados.rados.0.smithi107.stdout:6474: seq_num 6473 ranges {2400256=417792} 2020-06-12T02:38:29.019 INFO:tasks.rados.rados.0.smithi107.stdout:6474: writing smithi10729407-6 from 2400256 to 2818048 tid 1 2020-06-12T02:38:29.020 INFO:tasks.rados.rados.0.smithi107.stdout:6475: delete oid 19 current snap is 0 2020-06-12T02:38:29.020 INFO:tasks.rados.rados.0.smithi107.stdout:6471: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:29.020 INFO:tasks.rados.rados.0.smithi107.stdout:6471: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:29.020 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1217 (ObjNum 6470 snap 0 seq_num 6470) dirty exists 2020-06-12T02:38:29.020 INFO:tasks.rados.rados.0.smithi107.stdout:6471: left oid 7 (ObjNum 6470 snap 0 seq_num 6470) 2020-06-12T02:38:29.020 INFO:tasks.rados.rados.0.smithi107.stdout:6471: done (3 left) 2020-06-12T02:38:29.021 INFO:tasks.rados.rados.0.smithi107.stdout:6475: done (2 left) 2020-06-12T02:38:29.021 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:29.021 INFO:tasks.rados.rados.0.smithi107.stdout:6476: seq_num 6475 ranges {0=450560} 2020-06-12T02:38:29.021 INFO:tasks.rados.rados.0.smithi107.stdout:6476: writing smithi10729407-2 from 0 to 450560 tid 1 2020-06-12T02:38:29.022 INFO:tasks.rados.rados.0.smithi107.stdout:6477: delete oid 4 current snap is 0 2020-06-12T02:38:29.022 INFO:tasks.rados.rados.0.smithi107.stdout:6474: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:29.022 INFO:tasks.rados.rados.0.smithi107.stdout:6474: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:29.022 INFO:tasks.rados.rados.0.smithi107.stdout:6474: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:29.022 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1182 (ObjNum 6473 snap 0 seq_num 6473) dirty exists 2020-06-12T02:38:29.022 INFO:tasks.rados.rados.0.smithi107.stdout:6474: left oid 6 (ObjNum 6473 snap 0 seq_num 6473) 2020-06-12T02:38:29.023 INFO:tasks.rados.rados.0.smithi107.stdout:6474: done (3 left) 2020-06-12T02:38:29.023 INFO:tasks.rados.rados.0.smithi107.stdout:6477: done (2 left) 2020-06-12T02:38:29.023 INFO:tasks.rados.rados.0.smithi107.stdout:6478: delete oid 20 current snap is 0 2020-06-12T02:38:29.023 INFO:tasks.rados.rados.0.smithi107.stdout:6476: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:29.023 INFO:tasks.rados.rados.0.smithi107.stdout:6478: done (2 left) 2020-06-12T02:38:29.024 INFO:tasks.rados.rados.0.smithi107.stdout:6479: delete oid 19 current snap is 0 2020-06-12T02:38:29.024 INFO:tasks.rados.rados.0.smithi107.stdout:6476: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:29.024 INFO:tasks.rados.rados.0.smithi107.stdout:6479: done (2 left) 2020-06-12T02:38:29.024 INFO:tasks.rados.rados.0.smithi107.stdout:6480: delete oid 16 current snap is 0 2020-06-12T02:38:29.024 INFO:tasks.rados.rados.0.smithi107.stdout:6476: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:29.025 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 619 (ObjNum 6475 snap 0 seq_num 6475) dirty exists 2020-06-12T02:38:29.025 INFO:tasks.rados.rados.0.smithi107.stdout:6476: left oid 2 (ObjNum 6475 snap 0 seq_num 6475) 2020-06-12T02:38:29.025 INFO:tasks.rados.rados.0.smithi107.stdout:6476: done (2 left) 2020-06-12T02:38:29.025 INFO:tasks.rados.rados.0.smithi107.stdout:6480: done (1 left) 2020-06-12T02:38:29.025 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:38:29.026 INFO:tasks.rados.rados.0.smithi107.stdout:6481: seq_num 6480 ranges {2170880=532480} 2020-06-12T02:38:29.026 INFO:tasks.rados.rados.0.smithi107.stdout:6481: writing smithi10729407-5 from 2170880 to 2703360 tid 1 2020-06-12T02:38:29.026 INFO:tasks.rados.rados.0.smithi107.stdout:6482: delete oid 4 current snap is 0 2020-06-12T02:38:29.026 INFO:tasks.rados.rados.0.smithi107.stdout:6482: done (2 left) 2020-06-12T02:38:29.026 INFO:tasks.rados.rados.0.smithi107.stdout:6483: delete oid 16 current snap is 0 2020-06-12T02:38:29.027 INFO:tasks.rados.rados.0.smithi107.stdout:6483: done (2 left) 2020-06-12T02:38:29.027 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:29.027 INFO:tasks.rados.rados.0.smithi107.stdout:6484: seq_num 6483 ranges {450560=581632} 2020-06-12T02:38:29.027 INFO:tasks.rados.rados.0.smithi107.stdout:6484: writing smithi10729407-2 from 450560 to 1032192 tid 1 2020-06-12T02:38:29.027 INFO:tasks.rados.rados.0.smithi107.stdout:6485: delete oid 3 current snap is 0 2020-06-12T02:38:29.028 INFO:tasks.rados.rados.0.smithi107.stdout:6481: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:29.028 INFO:tasks.rados.rados.0.smithi107.stdout:6481: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:29.028 INFO:tasks.rados.rados.0.smithi107.stdout:6481: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:29.028 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 655 (ObjNum 6480 snap 0 seq_num 6480) dirty exists 2020-06-12T02:38:29.028 INFO:tasks.rados.rados.0.smithi107.stdout:6481: left oid 5 (ObjNum 6480 snap 0 seq_num 6480) 2020-06-12T02:38:29.029 INFO:tasks.rados.rados.0.smithi107.stdout:6481: done (3 left) 2020-06-12T02:38:29.029 INFO:tasks.rados.rados.0.smithi107.stdout:6485: done (2 left) 2020-06-12T02:38:29.029 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:29.029 INFO:tasks.rados.rados.0.smithi107.stdout:6486: seq_num 6485 ranges {0=614400} 2020-06-12T02:38:29.029 INFO:tasks.rados.rados.0.smithi107.stdout:6486: writing smithi10729407-12 from 0 to 614400 tid 1 2020-06-12T02:38:29.030 INFO:tasks.rados.rados.0.smithi107.stdout:6487: delete oid 19 current snap is 0 2020-06-12T02:38:29.030 INFO:tasks.rados.rados.0.smithi107.stdout:6484: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:29.030 INFO:tasks.rados.rados.0.smithi107.stdout:6484: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:29.030 INFO:tasks.rados.rados.0.smithi107.stdout:6484: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:29.030 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 621 (ObjNum 6483 snap 0 seq_num 6483) dirty exists 2020-06-12T02:38:29.031 INFO:tasks.rados.rados.0.smithi107.stdout:6484: left oid 2 (ObjNum 6483 snap 0 seq_num 6483) 2020-06-12T02:38:29.031 INFO:tasks.rados.rados.0.smithi107.stdout:6486: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:29.031 INFO:tasks.rados.rados.0.smithi107.stdout:6484: done (3 left) 2020-06-12T02:38:29.031 INFO:tasks.rados.rados.0.smithi107.stdout:6487: done (2 left) 2020-06-12T02:38:29.032 INFO:tasks.rados.rados.0.smithi107.stdout:6488: delete oid 14 current snap is 0 2020-06-12T02:38:29.032 INFO:tasks.rados.rados.0.smithi107.stdout:6486: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:29.033 INFO:tasks.rados.rados.0.smithi107.stdout:6486: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:29.033 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1172 (ObjNum 6485 snap 0 seq_num 6485) dirty exists 2020-06-12T02:38:29.033 INFO:tasks.rados.rados.0.smithi107.stdout:6486: left oid 12 (ObjNum 6485 snap 0 seq_num 6485) 2020-06-12T02:38:29.033 INFO:tasks.rados.rados.0.smithi107.stdout:6486: done (2 left) 2020-06-12T02:38:29.034 INFO:tasks.rados.rados.0.smithi107.stdout:6488: done (1 left) 2020-06-12T02:38:29.034 INFO:tasks.rados.rados.0.smithi107.stdout:6489: delete oid 11 current snap is 0 2020-06-12T02:38:29.034 INFO:tasks.rados.rados.0.smithi107.stdout:6489: done (1 left) 2020-06-12T02:38:29.034 INFO:tasks.rados.rados.0.smithi107.stdout:6490: delete oid 20 current snap is 0 2020-06-12T02:38:29.034 INFO:tasks.rados.rados.0.smithi107.stdout:6490: done (1 left) 2020-06-12T02:38:29.035 INFO:tasks.rados.rados.0.smithi107.stdout:6491: delete oid 2 current snap is 0 2020-06-12T02:38:29.035 INFO:tasks.rados.rados.0.smithi107.stdout:6491: done (1 left) 2020-06-12T02:38:29.035 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:38:29.035 INFO:tasks.rados.rados.0.smithi107.stdout:6492: seq_num 6491 ranges {0=720896} 2020-06-12T02:38:29.035 INFO:tasks.rados.rados.0.smithi107.stdout:6492: writing smithi10729407-13 from 0 to 720896 tid 1 2020-06-12T02:38:29.035 INFO:tasks.rados.rados.0.smithi107.stdout:6493: delete oid 17 current snap is 0 2020-06-12T02:38:29.036 INFO:tasks.rados.rados.0.smithi107.stdout:6493: done (2 left) 2020-06-12T02:38:29.036 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:29.036 INFO:tasks.rados.rados.0.smithi107.stdout:6494: seq_num 6493 ranges {0=753664} 2020-06-12T02:38:29.036 INFO:tasks.rados.rados.0.smithi107.stdout:6494: writing smithi10729407-2 from 0 to 753664 tid 1 2020-06-12T02:38:29.036 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:38:29.037 INFO:tasks.rados.rados.0.smithi107.stdout:6495: seq_num 6494 ranges {2818048=770048} 2020-06-12T02:38:29.037 INFO:tasks.rados.rados.0.smithi107.stdout:6495: writing smithi10729407-6 from 2818048 to 3588096 tid 1 2020-06-12T02:38:29.037 INFO:tasks.rados.rados.0.smithi107.stdout:6496: delete oid 12 current snap is 0 2020-06-12T02:38:29.037 INFO:tasks.rados.rados.0.smithi107.stdout:6492: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:29.037 INFO:tasks.rados.rados.0.smithi107.stdout:6492: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:29.038 INFO:tasks.rados.rados.0.smithi107.stdout:6492: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:29.038 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1219 (ObjNum 6491 snap 0 seq_num 6491) dirty exists 2020-06-12T02:38:29.038 INFO:tasks.rados.rados.0.smithi107.stdout:6492: left oid 13 (ObjNum 6491 snap 0 seq_num 6491) 2020-06-12T02:38:29.038 INFO:tasks.rados.rados.0.smithi107.stdout:6494: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:29.038 INFO:tasks.rados.rados.0.smithi107.stdout:6494: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:29.039 INFO:tasks.rados.rados.0.smithi107.stdout:6494: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:29.039 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 624 (ObjNum 6493 snap 0 seq_num 6493) dirty exists 2020-06-12T02:38:29.039 INFO:tasks.rados.rados.0.smithi107.stdout:6494: left oid 2 (ObjNum 6493 snap 0 seq_num 6493) 2020-06-12T02:38:29.039 INFO:tasks.rados.rados.0.smithi107.stdout:6492: done (4 left) 2020-06-12T02:38:29.039 INFO:tasks.rados.rados.0.smithi107.stdout:6494: done (3 left) 2020-06-12T02:38:29.040 INFO:tasks.rados.rados.0.smithi107.stdout:6496: done (2 left) 2020-06-12T02:38:29.040 INFO:tasks.rados.rados.0.smithi107.stdout:6497: delete oid 17 current snap is 0 2020-06-12T02:38:29.040 INFO:tasks.rados.rados.0.smithi107.stdout:6497: done (2 left) 2020-06-12T02:38:29.040 INFO:tasks.rados.rados.0.smithi107.stdout:6498: delete oid 17 current snap is 0 2020-06-12T02:38:29.040 INFO:tasks.rados.rados.0.smithi107.stdout:6495: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:29.041 INFO:tasks.rados.rados.0.smithi107.stdout:6495: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:29.041 INFO:tasks.rados.rados.0.smithi107.stdout:6495: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:29.041 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1184 (ObjNum 6494 snap 0 seq_num 6494) dirty exists 2020-06-12T02:38:29.041 INFO:tasks.rados.rados.0.smithi107.stdout:6495: left oid 6 (ObjNum 6494 snap 0 seq_num 6494) 2020-06-12T02:38:29.041 INFO:tasks.rados.rados.0.smithi107.stdout:6495: done (2 left) 2020-06-12T02:38:29.042 INFO:tasks.rados.rados.0.smithi107.stdout:6498: done (1 left) 2020-06-12T02:38:29.042 INFO:tasks.rados.rados.0.smithi107.stdout:6499: delete oid 18 current snap is 0 2020-06-12T02:38:29.042 INFO:tasks.rados.rados.0.smithi107.stdout:6499: done (1 left) 2020-06-12T02:38:29.042 INFO:tasks.rados.rados.0.smithi107.stdout:6500: delete oid 8 current snap is 0 2020-06-12T02:38:29.042 INFO:tasks.rados.rados.0.smithi107.stdout:6500: done (1 left) 2020-06-12T02:38:29.043 INFO:tasks.rados.rados.0.smithi107.stdout:6501: delete oid 16 current snap is 0 2020-06-12T02:38:29.043 INFO:tasks.rados.rados.0.smithi107.stdout:6501: done (1 left) 2020-06-12T02:38:29.043 INFO:tasks.rados.rados.0.smithi107.stdout:6502: delete oid 4 current snap is 0 2020-06-12T02:38:29.043 INFO:tasks.rados.rados.0.smithi107.stdout:6502: done (1 left) 2020-06-12T02:38:29.043 INFO:tasks.rados.rados.0.smithi107.stdout:6503: delete oid 14 current snap is 0 2020-06-12T02:38:29.043 INFO:tasks.rados.rados.0.smithi107.stdout:6503: done (1 left) 2020-06-12T02:38:29.044 INFO:tasks.rados.rados.0.smithi107.stdout:6504: delete oid 20 current snap is 0 2020-06-12T02:38:29.044 INFO:tasks.rados.rados.0.smithi107.stdout:6504: done (1 left) 2020-06-12T02:38:29.044 INFO:tasks.rados.rados.0.smithi107.stdout:6505: delete oid 15 current snap is 0 2020-06-12T02:38:29.044 INFO:tasks.rados.rados.0.smithi107.stdout:6505: done (1 left) 2020-06-12T02:38:29.044 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:29.045 INFO:tasks.rados.rados.0.smithi107.stdout:6506: seq_num 6505 ranges {0=548864,548864=401408} 2020-06-12T02:38:29.045 INFO:tasks.rados.rados.0.smithi107.stdout:6506: writing smithi10729407-12 from 0 to 548864 tid 1 2020-06-12T02:38:29.045 INFO:tasks.rados.rados.0.smithi107.stdout:6506: writing smithi10729407-12 from 548864 to 950272 tid 2 2020-06-12T02:38:29.045 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:29.045 INFO:tasks.rados.rados.0.smithi107.stdout:6507: seq_num 6506 ranges {0=565248,565248=401408} 2020-06-12T02:38:29.046 INFO:tasks.rados.rados.0.smithi107.stdout:6507: writing smithi10729407-18 from 0 to 565248 tid 1 2020-06-12T02:38:29.046 INFO:tasks.rados.rados.0.smithi107.stdout:6507: writing smithi10729407-18 from 565248 to 966656 tid 2 2020-06-12T02:38:29.046 INFO:tasks.rados.rados.0.smithi107.stdout:6508: delete oid 19 current snap is 0 2020-06-12T02:38:29.046 INFO:tasks.rados.rados.0.smithi107.stdout:6506: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:29.046 INFO:tasks.rados.rados.0.smithi107.stdout:6507: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:29.047 INFO:tasks.rados.rados.0.smithi107.stdout:6506: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:29.047 INFO:tasks.rados.rados.0.smithi107.stdout:6506: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:29.047 INFO:tasks.rados.rados.0.smithi107.stdout:6506: finishing write tid 5 to smithi10729407-12 2020-06-12T02:38:29.047 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1176 (ObjNum 6505 snap 0 seq_num 6505) dirty exists 2020-06-12T02:38:29.048 INFO:tasks.rados.rados.0.smithi107.stdout:6506: left oid 12 (ObjNum 6505 snap 0 seq_num 6505) 2020-06-12T02:38:29.048 INFO:tasks.rados.rados.0.smithi107.stdout:6506: done (3 left) 2020-06-12T02:38:29.048 INFO:tasks.rados.rados.0.smithi107.stdout:6508: done (2 left) 2020-06-12T02:38:29.048 INFO:tasks.rados.rados.0.smithi107.stdout:6509: delete oid 17 current snap is 0 2020-06-12T02:38:29.048 INFO:tasks.rados.rados.0.smithi107.stdout:6507: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:29.049 INFO:tasks.rados.rados.0.smithi107.stdout:6507: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:29.049 INFO:tasks.rados.rados.0.smithi107.stdout:6509: done (2 left) 2020-06-12T02:38:29.049 INFO:tasks.rados.rados.0.smithi107.stdout:6510: delete oid 2 current snap is 0 2020-06-12T02:38:29.049 INFO:tasks.rados.rados.0.smithi107.stdout:6507: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:29.050 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1188 (ObjNum 6506 snap 0 seq_num 6506) dirty exists 2020-06-12T02:38:29.050 INFO:tasks.rados.rados.0.smithi107.stdout:6507: left oid 18 (ObjNum 6506 snap 0 seq_num 6506) 2020-06-12T02:38:29.050 INFO:tasks.rados.rados.0.smithi107.stdout:6507: done (2 left) 2020-06-12T02:38:29.050 INFO:tasks.rados.rados.0.smithi107.stdout:6510: done (1 left) 2020-06-12T02:38:29.050 INFO:tasks.rados.rados.0.smithi107.stdout:6511: delete oid 12 current snap is 0 2020-06-12T02:38:29.051 INFO:tasks.rados.rados.0.smithi107.stdout:6511: done (1 left) 2020-06-12T02:38:29.051 INFO:tasks.rados.rados.0.smithi107.stdout:6512: delete oid 1 current snap is 0 2020-06-12T02:38:29.051 INFO:tasks.rados.rados.0.smithi107.stdout:6512: done (1 left) 2020-06-12T02:38:29.051 INFO:tasks.rados.rados.0.smithi107.stdout:6513: delete oid 13 current snap is 0 2020-06-12T02:38:29.051 INFO:tasks.rados.rados.0.smithi107.stdout:6513: done (1 left) 2020-06-12T02:38:29.052 INFO:tasks.rados.rados.0.smithi107.stdout:6514: delete oid 8 current snap is 0 2020-06-12T02:38:29.052 INFO:tasks.rados.rados.0.smithi107.stdout:6514: done (1 left) 2020-06-12T02:38:29.052 INFO:tasks.rados.rados.0.smithi107.stdout:6515: delete oid 13 current snap is 0 2020-06-12T02:38:29.052 INFO:tasks.rados.rados.0.smithi107.stdout:6515: done (1 left) 2020-06-12T02:38:29.052 INFO:tasks.rados.rados.0.smithi107.stdout:6516: delete oid 16 current snap is 0 2020-06-12T02:38:29.053 INFO:tasks.rados.rados.0.smithi107.stdout:6516: done (1 left) 2020-06-12T02:38:29.053 INFO:tasks.rados.rados.0.smithi107.stdout:6517: delete oid 20 current snap is 0 2020-06-12T02:38:29.053 INFO:tasks.rados.rados.0.smithi107.stdout:6517: done (1 left) 2020-06-12T02:38:29.053 INFO:tasks.rados.rados.0.smithi107.stdout:6518: delete oid 13 current snap is 0 2020-06-12T02:38:29.053 INFO:tasks.rados.rados.0.smithi107.stdout:6518: done (1 left) 2020-06-12T02:38:29.054 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:29.054 INFO:tasks.rados.rados.0.smithi107.stdout:6519: seq_num 6518 ranges {0=770048,770048=401408} 2020-06-12T02:38:29.054 INFO:tasks.rados.rados.0.smithi107.stdout:6519: writing smithi10729407-15 from 0 to 770048 tid 1 2020-06-12T02:38:29.055 INFO:tasks.rados.rados.0.smithi107.stdout:6519: writing smithi10729407-15 from 770048 to 1171456 tid 2 2020-06-12T02:38:29.055 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:29.055 INFO:tasks.rados.rados.0.smithi107.stdout:6520: seq_num 6519 ranges {368640=786432,1155072=401408} 2020-06-12T02:38:29.055 INFO:tasks.rados.rados.0.smithi107.stdout:6520: writing smithi10729407-7 from 368640 to 1155072 tid 1 2020-06-12T02:38:29.055 INFO:tasks.rados.rados.0.smithi107.stdout:6520: writing smithi10729407-7 from 1155072 to 1556480 tid 2 2020-06-12T02:38:29.056 INFO:tasks.rados.rados.0.smithi107.stdout:6521: delete oid 13 current snap is 0 2020-06-12T02:38:29.056 INFO:tasks.rados.rados.0.smithi107.stdout:6519: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:29.056 INFO:tasks.rados.rados.0.smithi107.stdout:6519: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:29.056 INFO:tasks.rados.rados.0.smithi107.stdout:6520: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:29.056 INFO:tasks.rados.rados.0.smithi107.stdout:6519: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:29.057 INFO:tasks.rados.rados.0.smithi107.stdout:6520: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:29.057 INFO:tasks.rados.rados.0.smithi107.stdout:6520: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:29.057 INFO:tasks.rados.rados.0.smithi107.stdout:6520: finishing write tid 5 to smithi10729407-7 2020-06-12T02:38:29.057 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1225 (ObjNum 6519 snap 0 seq_num 6519) dirty exists 2020-06-12T02:38:29.057 INFO:tasks.rados.rados.0.smithi107.stdout:6520: left oid 7 (ObjNum 6519 snap 0 seq_num 6519) 2020-06-12T02:38:29.057 INFO:tasks.rados.rados.0.smithi107.stdout:6519: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:29.058 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1208 (ObjNum 6518 snap 0 seq_num 6518) dirty exists 2020-06-12T02:38:29.058 INFO:tasks.rados.rados.0.smithi107.stdout:6519: left oid 15 (ObjNum 6518 snap 0 seq_num 6518) 2020-06-12T02:38:29.058 INFO:tasks.rados.rados.0.smithi107.stdout:6519: done (3 left) 2020-06-12T02:38:29.058 INFO:tasks.rados.rados.0.smithi107.stdout:6520: done (2 left) 2020-06-12T02:38:29.058 INFO:tasks.rados.rados.0.smithi107.stdout:6521: done (1 left) 2020-06-12T02:38:29.059 INFO:tasks.rados.rados.0.smithi107.stdout:6522: delete oid 15 current snap is 0 2020-06-12T02:38:29.059 INFO:tasks.rados.rados.0.smithi107.stdout:6522: done (1 left) 2020-06-12T02:38:29.059 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:29.059 INFO:tasks.rados.rados.0.smithi107.stdout:6523: seq_num 6522 ranges {0=434176,434176=598016,1032192=204800} 2020-06-12T02:38:29.059 INFO:tasks.rados.rados.0.smithi107.stdout:6523: writing smithi10729407-15 from 0 to 434176 tid 1 2020-06-12T02:38:29.060 INFO:tasks.rados.rados.0.smithi107.stdout:6523: writing smithi10729407-15 from 434176 to 1032192 tid 2 2020-06-12T02:38:29.060 INFO:tasks.rados.rados.0.smithi107.stdout:6523: writing smithi10729407-15 from 1032192 to 1236992 tid 3 2020-06-12T02:38:29.060 INFO:tasks.rados.rados.0.smithi107.stdout:6524: delete oid 9 current snap is 0 2020-06-12T02:38:29.060 INFO:tasks.rados.rados.0.smithi107.stdout:6523: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:29.060 INFO:tasks.rados.rados.0.smithi107.stdout:6524: done (2 left) 2020-06-12T02:38:29.061 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:29.061 INFO:tasks.rados.rados.0.smithi107.stdout:6525: seq_num 6524 ranges {0=466944,466944=409600,876544=393216} 2020-06-12T02:38:29.061 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:28.992+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:29.065 INFO:tasks.rados.rados.0.smithi107.stdout:6525: writing smithi10729407-8 from 0 to 466944 tid 1 2020-06-12T02:38:29.065 INFO:tasks.rados.rados.0.smithi107.stdout:6525: writing smithi10729407-8 from 466944 to 876544 tid 2 2020-06-12T02:38:29.065 INFO:tasks.rados.rados.0.smithi107.stdout:6525: writing smithi10729407-8 from 876544 to 1269760 tid 3 2020-06-12T02:38:29.066 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:38:29.066 INFO:tasks.rados.rados.0.smithi107.stdout:6526: seq_num 6525 ranges {0=483328,483328=696320,1179648=106496} 2020-06-12T02:38:29.066 INFO:tasks.rados.rados.0.smithi107.stdout:6526: writing smithi10729407-1 from 0 to 483328 tid 1 2020-06-12T02:38:29.066 INFO:tasks.rados.rados.0.smithi107.stdout:6526: writing smithi10729407-1 from 483328 to 1179648 tid 2 2020-06-12T02:38:29.066 INFO:tasks.rados.rados.0.smithi107.stdout:6526: writing smithi10729407-1 from 1179648 to 1286144 tid 3 2020-06-12T02:38:29.067 INFO:tasks.rados.rados.0.smithi107.stdout:6527: delete oid 2 current snap is 0 2020-06-12T02:38:29.067 INFO:tasks.rados.rados.0.smithi107.stdout:6523: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:29.067 INFO:tasks.rados.rados.0.smithi107.stdout:6523: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:29.067 INFO:tasks.rados.rados.0.smithi107.stdout:6523: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:29.067 INFO:tasks.rados.rados.0.smithi107.stdout:6527: done (4 left) 2020-06-12T02:38:29.068 INFO:tasks.rados.rados.0.smithi107.stdout:6528: delete oid 3 current snap is 0 2020-06-12T02:38:29.068 INFO:tasks.rados.rados.0.smithi107.stdout:6525: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:29.068 INFO:tasks.rados.rados.0.smithi107.stdout:6525: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:29.068 INFO:tasks.rados.rados.0.smithi107.stdout:6523: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:29.068 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1213 (ObjNum 6522 snap 0 seq_num 6522) dirty exists 2020-06-12T02:38:29.069 INFO:tasks.rados.rados.0.smithi107.stdout:6523: left oid 15 (ObjNum 6522 snap 0 seq_num 6522) 2020-06-12T02:38:29.069 INFO:tasks.rados.rados.0.smithi107.stdout:6526: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:29.069 INFO:tasks.rados.rados.0.smithi107.stdout:6523: done (4 left) 2020-06-12T02:38:29.069 INFO:tasks.rados.rados.0.smithi107.stdout:6528: done (3 left) 2020-06-12T02:38:29.069 INFO:tasks.rados.rados.0.smithi107.stdout:6529: delete oid 20 current snap is 0 2020-06-12T02:38:29.070 INFO:tasks.rados.rados.0.smithi107.stdout:6526: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:29.070 INFO:tasks.rados.rados.0.smithi107.stdout:6525: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:29.070 INFO:tasks.rados.rados.0.smithi107.stdout:6526: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:29.070 INFO:tasks.rados.rados.0.smithi107.stdout:6526: finishing write tid 5 to smithi10729407-1 2020-06-12T02:38:29.070 INFO:tasks.rados.rados.0.smithi107.stdout:6525: finishing write tid 5 to smithi10729407-8 2020-06-12T02:38:29.071 INFO:tasks.rados.rados.0.smithi107.stdout:6529: done (3 left) 2020-06-12T02:38:29.071 INFO:tasks.rados.rados.0.smithi107.stdout:6530: delete oid 16 current snap is 0 2020-06-12T02:38:29.071 INFO:tasks.rados.rados.0.smithi107.stdout:6525: finishing write tid 6 to smithi10729407-8 2020-06-12T02:38:29.071 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1217 (ObjNum 6524 snap 0 seq_num 6524) dirty exists 2020-06-12T02:38:29.071 INFO:tasks.rados.rados.0.smithi107.stdout:6525: left oid 8 (ObjNum 6524 snap 0 seq_num 6524) 2020-06-12T02:38:29.072 INFO:tasks.rados.rados.0.smithi107.stdout:6526: finishing write tid 6 to smithi10729407-1 2020-06-12T02:38:29.072 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1189 (ObjNum 6525 snap 0 seq_num 6525) dirty exists 2020-06-12T02:38:29.072 INFO:tasks.rados.rados.0.smithi107.stdout:6526: left oid 1 (ObjNum 6525 snap 0 seq_num 6525) 2020-06-12T02:38:29.072 INFO:tasks.rados.rados.0.smithi107.stdout:6525: done (3 left) 2020-06-12T02:38:29.072 INFO:tasks.rados.rados.0.smithi107.stdout:6526: done (2 left) 2020-06-12T02:38:29.073 INFO:tasks.rados.rados.0.smithi107.stdout:6530: done (1 left) 2020-06-12T02:38:29.073 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:29.073 INFO:tasks.rados.rados.0.smithi107.stdout:6531: seq_num 6530 ranges {0=573440,573440=516096,1089536=286720} 2020-06-12T02:38:29.073 INFO:tasks.rados.rados.0.smithi107.stdout:6531: writing smithi10729407-19 from 0 to 573440 tid 1 2020-06-12T02:38:29.073 INFO:tasks.rados.rados.0.smithi107.stdout:6531: writing smithi10729407-19 from 573440 to 1089536 tid 2 2020-06-12T02:38:29.074 INFO:tasks.rados.rados.0.smithi107.stdout:6531: writing smithi10729407-19 from 1089536 to 1376256 tid 3 2020-06-12T02:38:29.074 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:29.074 INFO:tasks.rados.rados.0.smithi107.stdout:6532: seq_num 6531 ranges {0=589824,589824=450560,1040384=352256} 2020-06-12T02:38:29.074 INFO:tasks.rados.rados.0.smithi107.stdout:6532: writing smithi10729407-12 from 0 to 589824 tid 1 2020-06-12T02:38:29.074 INFO:tasks.rados.rados.0.smithi107.stdout:6532: writing smithi10729407-12 from 589824 to 1040384 tid 2 2020-06-12T02:38:29.075 INFO:tasks.rados.rados.0.smithi107.stdout:6532: writing smithi10729407-12 from 1040384 to 1392640 tid 3 2020-06-12T02:38:29.075 INFO:tasks.rados.rados.0.smithi107.stdout:6533: delete oid 17 current snap is 0 2020-06-12T02:38:29.075 INFO:tasks.rados.rados.0.smithi107.stdout:6531: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:29.075 INFO:tasks.rados.rados.0.smithi107.stdout:6531: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:29.076 INFO:tasks.rados.rados.0.smithi107.stdout:6531: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:29.076 INFO:tasks.rados.rados.0.smithi107.stdout:6533: done (3 left) 2020-06-12T02:38:29.076 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:38:29.076 INFO:tasks.rados.rados.0.smithi107.stdout:6534: seq_num 6533 ranges {2703360=622592,3325952=622592,3948544=180224} 2020-06-12T02:38:29.077 INFO:tasks.rados.rados.0.smithi107.stdout:6534: writing smithi10729407-5 from 2703360 to 3325952 tid 1 2020-06-12T02:38:29.079 INFO:tasks.rados.rados.0.smithi107.stdout:6534: writing smithi10729407-5 from 3325952 to 3948544 tid 2 2020-06-12T02:38:29.081 INFO:tasks.rados.rados.0.smithi107.stdout:6534: writing smithi10729407-5 from 3948544 to 4128768 tid 3 2020-06-12T02:38:29.081 INFO:tasks.rados.rados.0.smithi107.stdout:6535: delete oid 6 current snap is 0 2020-06-12T02:38:29.081 INFO:tasks.rados.rados.0.smithi107.stdout:6531: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:29.081 INFO:tasks.rados.rados.0.smithi107.stdout:6532: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:29.082 INFO:tasks.rados.rados.0.smithi107.stdout:6531: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:29.082 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1182 (ObjNum 6530 snap 0 seq_num 6530) dirty exists 2020-06-12T02:38:29.082 INFO:tasks.rados.rados.0.smithi107.stdout:6531: left oid 19 (ObjNum 6530 snap 0 seq_num 6530) 2020-06-12T02:38:29.082 INFO:tasks.rados.rados.0.smithi107.stdout:6532: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:29.082 INFO:tasks.rados.rados.0.smithi107.stdout:6532: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:29.083 INFO:tasks.rados.rados.0.smithi107.stdout:6532: finishing write tid 5 to smithi10729407-12 2020-06-12T02:38:29.083 INFO:tasks.rados.rados.0.smithi107.stdout:6532: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:29.083 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1186 (ObjNum 6531 snap 0 seq_num 6531) dirty exists 2020-06-12T02:38:29.083 INFO:tasks.rados.rados.0.smithi107.stdout:6532: left oid 12 (ObjNum 6531 snap 0 seq_num 6531) 2020-06-12T02:38:29.084 INFO:tasks.rados.rados.0.smithi107.stdout:6534: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:29.084 INFO:tasks.rados.rados.0.smithi107.stdout:6531: done (4 left) 2020-06-12T02:38:29.084 INFO:tasks.rados.rados.0.smithi107.stdout:6532: done (3 left) 2020-06-12T02:38:29.084 INFO:tasks.rados.rados.0.smithi107.stdout:6535: done (2 left) 2020-06-12T02:38:29.085 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:29.085 INFO:tasks.rados.rados.0.smithi107.stdout:6536: seq_num 6535 ranges {0=655360,655360=794624,1449984=8192} 2020-06-12T02:38:29.088 INFO:tasks.rados.rados.0.smithi107.stdout:6536: writing smithi10729407-2 from 0 to 655360 tid 1 2020-06-12T02:38:29.093 INFO:tasks.rados.rados.0.smithi107.stdout:6536: writing smithi10729407-2 from 655360 to 1449984 tid 2 2020-06-12T02:38:29.093 INFO:tasks.rados.rados.0.smithi107.stdout:6536: writing smithi10729407-2 from 1449984 to 1458176 tid 3 2020-06-12T02:38:29.093 INFO:tasks.rados.rados.0.smithi107.stdout:6537: delete oid 15 current snap is 0 2020-06-12T02:38:29.094 INFO:tasks.rados.rados.0.smithi107.stdout:6534: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:29.094 INFO:tasks.rados.rados.0.smithi107.stdout:6534: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:29.094 INFO:tasks.rados.rados.0.smithi107.stdout:6534: finishing write tid 5 to smithi10729407-5 2020-06-12T02:38:29.096 INFO:tasks.rados.rados.0.smithi107.stdout:6534: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:29.096 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 659 (ObjNum 6533 snap 0 seq_num 6533) dirty exists 2020-06-12T02:38:29.096 INFO:tasks.rados.rados.0.smithi107.stdout:6534: left oid 5 (ObjNum 6533 snap 0 seq_num 6533) 2020-06-12T02:38:29.100 INFO:tasks.rados.rados.0.smithi107.stdout:6536: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:29.101 INFO:tasks.rados.rados.0.smithi107.stdout:6534: done (3 left) 2020-06-12T02:38:29.101 INFO:tasks.rados.rados.0.smithi107.stdout:6537: done (2 left) 2020-06-12T02:38:29.101 INFO:tasks.rados.rados.0.smithi107.stdout:6538: delete oid 20 current snap is 0 2020-06-12T02:38:29.102 INFO:tasks.rados.rados.0.smithi107.stdout:6536: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:29.104 INFO:tasks.rados.rados.0.smithi107.stdout:6536: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:29.105 INFO:tasks.rados.rados.0.smithi107.stdout:6536: finishing write tid 5 to smithi10729407-2 2020-06-12T02:38:29.106 INFO:tasks.rados.rados.0.smithi107.stdout:6536: finishing write tid 6 to smithi10729407-2 2020-06-12T02:38:29.106 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 630 (ObjNum 6535 snap 0 seq_num 6535) dirty exists 2020-06-12T02:38:29.106 INFO:tasks.rados.rados.0.smithi107.stdout:6536: left oid 2 (ObjNum 6535 snap 0 seq_num 6535) 2020-06-12T02:38:29.107 INFO:tasks.rados.rados.0.smithi107.stdout:6536: done (2 left) 2020-06-12T02:38:29.108 INFO:tasks.rados.rados.0.smithi107.stdout:6538: done (1 left) 2020-06-12T02:38:29.108 INFO:tasks.rados.rados.0.smithi107.stdout:6539: delete oid 20 current snap is 0 2020-06-12T02:38:29.110 INFO:tasks.rados.rados.0.smithi107.stdout:6539: done (1 left) 2020-06-12T02:38:29.110 INFO:tasks.rados.rados.0.smithi107.stdout:6540: delete oid 7 current snap is 0 2020-06-12T02:38:29.112 INFO:tasks.rados.rados.0.smithi107.stdout:6540: done (1 left) 2020-06-12T02:38:29.113 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:29.113 INFO:tasks.rados.rados.0.smithi107.stdout:6541: seq_num 6540 ranges {1392640=737280,2129920=663552,2793472=139264} 2020-06-12T02:38:29.120 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:38:29.117+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:38:29.124 INFO:tasks.rados.rados.0.smithi107.stdout:6541: writing smithi10729407-12 from 1392640 to 2129920 tid 1 2020-06-12T02:38:29.128 INFO:tasks.rados.rados.0.smithi107.stdout:6541: writing smithi10729407-12 from 2129920 to 2793472 tid 2 2020-06-12T02:38:29.136 INFO:tasks.rados.rados.0.smithi107.stdout:6541: writing smithi10729407-12 from 2793472 to 2932736 tid 3 2020-06-12T02:38:29.137 INFO:tasks.rados.rados.0.smithi107.stdout:6542: delete oid 13 current snap is 0 2020-06-12T02:38:29.137 INFO:tasks.rados.rados.0.smithi107.stdout:6541: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:29.137 INFO:tasks.rados.rados.0.smithi107.stdout:6542: done (2 left) 2020-06-12T02:38:29.137 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:29.137 INFO:tasks.rados.rados.0.smithi107.stdout:6543: seq_num 6542 ranges {0=770048,770048=425984,1196032=376832} 2020-06-12T02:38:29.138 INFO:tasks.rados.rados.0.smithi107.stdout:6543: writing smithi10729407-3 from 0 to 770048 tid 1 2020-06-12T02:38:29.139 INFO:tasks.rados.rados.0.smithi107.stdout:6543: writing smithi10729407-3 from 770048 to 1196032 tid 2 2020-06-12T02:38:29.142 INFO:tasks.rados.rados.0.smithi107.stdout:6543: writing smithi10729407-3 from 1196032 to 1572864 tid 3 2020-06-12T02:38:29.142 INFO:tasks.rados.rados.0.smithi107.stdout:6544: delete oid 4 current snap is 0 2020-06-12T02:38:29.142 INFO:tasks.rados.rados.0.smithi107.stdout:6541: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:29.143 INFO:tasks.rados.rados.0.smithi107.stdout:6541: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:29.143 INFO:tasks.rados.rados.0.smithi107.stdout:6541: finishing write tid 5 to smithi10729407-12 2020-06-12T02:38:29.143 INFO:tasks.rados.rados.0.smithi107.stdout:6541: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:29.144 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1190 (ObjNum 6540 snap 0 seq_num 6540) dirty exists 2020-06-12T02:38:29.144 INFO:tasks.rados.rados.0.smithi107.stdout:6541: left oid 12 (ObjNum 6540 snap 0 seq_num 6540) 2020-06-12T02:38:29.146 INFO:tasks.rados.rados.0.smithi107.stdout:6543: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:29.147 INFO:tasks.rados.rados.0.smithi107.stdout:6541: done (3 left) 2020-06-12T02:38:29.147 INFO:tasks.rados.rados.0.smithi107.stdout:6544: done (2 left) 2020-06-12T02:38:29.147 INFO:tasks.rados.rados.0.smithi107.stdout:6545: delete oid 12 current snap is 0 2020-06-12T02:38:29.149 INFO:tasks.rados.rados.0.smithi107.stdout:6545: done (2 left) 2020-06-12T02:38:29.149 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:29.150 INFO:tasks.rados.rados.0.smithi107.stdout:6546: seq_num 6545 ranges {0=417792,417792=483328,901120=409600,1310720=311296} 2020-06-12T02:38:29.152 INFO:tasks.rados.rados.0.smithi107.stdout:6546: writing smithi10729407-13 from 0 to 417792 tid 1 2020-06-12T02:38:29.158 INFO:tasks.rados.rados.0.smithi107.stdout:6546: writing smithi10729407-13 from 417792 to 901120 tid 2 2020-06-12T02:38:29.158 INFO:tasks.rados.rados.0.smithi107.stdout:6546: writing smithi10729407-13 from 901120 to 1310720 tid 3 2020-06-12T02:38:29.159 INFO:tasks.rados.rados.0.smithi107.stdout:6546: writing smithi10729407-13 from 1310720 to 1622016 tid 4 2020-06-12T02:38:29.159 INFO:tasks.rados.rados.0.smithi107.stdout:6547: delete oid 7 current snap is 0 2020-06-12T02:38:29.159 INFO:tasks.rados.rados.0.smithi107.stdout:6543: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:29.160 INFO:tasks.rados.rados.0.smithi107.stdout:6543: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:29.160 INFO:tasks.rados.rados.0.smithi107.stdout:6543: finishing write tid 5 to smithi10729407-3 2020-06-12T02:38:29.160 INFO:tasks.rados.rados.0.smithi107.stdout:6546: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:29.161 INFO:tasks.rados.rados.0.smithi107.stdout:6543: finishing write tid 6 to smithi10729407-3 2020-06-12T02:38:29.161 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1149 (ObjNum 6542 snap 0 seq_num 6542) dirty exists 2020-06-12T02:38:29.161 INFO:tasks.rados.rados.0.smithi107.stdout:6543: left oid 3 (ObjNum 6542 snap 0 seq_num 6542) 2020-06-12T02:38:29.166 INFO:tasks.rados.rados.0.smithi107.stdout:6546: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:29.169 INFO:tasks.rados.rados.0.smithi107.stdout:6546: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:29.170 INFO:tasks.rados.rados.0.smithi107.stdout:6546: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:29.172 INFO:tasks.rados.rados.0.smithi107.stdout:6546: finishing write tid 6 to smithi10729407-13 2020-06-12T02:38:29.175 INFO:tasks.rados.rados.0.smithi107.stdout:6546: finishing write tid 7 to smithi10729407-13 2020-06-12T02:38:29.175 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1233 (ObjNum 6545 snap 0 seq_num 6545) dirty exists 2020-06-12T02:38:29.175 INFO:tasks.rados.rados.0.smithi107.stdout:6546: left oid 13 (ObjNum 6545 snap 0 seq_num 6545) 2020-06-12T02:38:29.176 INFO:tasks.rados.rados.0.smithi107.stdout:6543: done (3 left) 2020-06-12T02:38:29.176 INFO:tasks.rados.rados.0.smithi107.stdout:6546: done (2 left) 2020-06-12T02:38:29.176 INFO:tasks.rados.rados.0.smithi107.stdout:6547: done (1 left) 2020-06-12T02:38:29.176 INFO:tasks.rados.rados.0.smithi107.stdout:6548: delete oid 13 current snap is 0 2020-06-12T02:38:29.180 INFO:tasks.rados.rados.0.smithi107.stdout:6548: done (1 left) 2020-06-12T02:38:29.180 INFO:tasks.rados.rados.0.smithi107.stdout:6549: delete oid 17 current snap is 0 2020-06-12T02:38:29.185 INFO:tasks.rados.rados.0.smithi107.stdout:6549: done (1 left) 2020-06-12T02:38:29.185 INFO:tasks.rados.rados.0.smithi107.stdout:6550: delete oid 15 current snap is 0 2020-06-12T02:38:29.188 INFO:tasks.rados.rados.0.smithi107.stdout:6550: done (1 left) 2020-06-12T02:38:29.188 INFO:tasks.rados.rados.0.smithi107.stdout:6551: delete oid 9 current snap is 0 2020-06-12T02:38:29.190 INFO:tasks.rados.rados.0.smithi107.stdout:6551: done (1 left) 2020-06-12T02:38:29.190 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:29.190 INFO:tasks.rados.rados.0.smithi107.stdout:6552: seq_num 6551 ranges {0=524288,524288=638976,1163264=565248} 2020-06-12T02:38:29.193 INFO:tasks.rados.rados.0.smithi107.stdout:6552: writing smithi10729407-12 from 0 to 524288 tid 1 2020-06-12T02:38:29.197 INFO:tasks.rados.rados.0.smithi107.stdout:6552: writing smithi10729407-12 from 524288 to 1163264 tid 2 2020-06-12T02:38:29.197 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:29.194+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:29.200 INFO:tasks.rados.rados.0.smithi107.stdout:6552: writing smithi10729407-12 from 1163264 to 1728512 tid 3 2020-06-12T02:38:29.200 INFO:tasks.rados.rados.0.smithi107.stdout:6553: delete oid 13 current snap is 0 2020-06-12T02:38:29.200 INFO:tasks.rados.rados.0.smithi107.stdout:6552: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:29.206 INFO:tasks.rados.rados.0.smithi107.stdout:6553: done (2 left) 2020-06-12T02:38:29.206 INFO:tasks.rados.rados.0.smithi107.stdout:6554: delete oid 4 current snap is 0 2020-06-12T02:38:29.209 INFO:tasks.rados.rados.0.smithi107.stdout:6552: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:29.209 INFO:tasks.rados.rados.0.smithi107.stdout:6554: done (2 left) 2020-06-12T02:38:29.210 INFO:tasks.rados.rados.0.smithi107.stdout:6555: delete oid 9 current snap is 0 2020-06-12T02:38:29.210 INFO:tasks.rados.rados.0.smithi107.stdout:6552: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:29.211 INFO:tasks.rados.rados.0.smithi107.stdout:6552: finishing write tid 5 to smithi10729407-12 2020-06-12T02:38:29.212 INFO:tasks.rados.rados.0.smithi107.stdout:6555: done (2 left) 2020-06-12T02:38:29.212 INFO:tasks.rados.rados.0.smithi107.stdout:6556: delete oid 7 current snap is 0 2020-06-12T02:38:29.213 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:29.213 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:29.214 INFO:tasks.rados.rados.0.smithi107.stdout:6552: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:29.214 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1195 (ObjNum 6551 snap 0 seq_num 6551) dirty exists 2020-06-12T02:38:29.214 INFO:tasks.rados.rados.0.smithi107.stdout:6552: left oid 12 (ObjNum 6551 snap 0 seq_num 6551) 2020-06-12T02:38:29.215 INFO:tasks.rados.rados.0.smithi107.stdout:6552: done (2 left) 2020-06-12T02:38:29.215 INFO:tasks.rados.rados.0.smithi107.stdout:6556: done (1 left) 2020-06-12T02:38:29.215 INFO:tasks.rados.rados.0.smithi107.stdout:6557: delete oid 11 current snap is 0 2020-06-12T02:38:29.216 INFO:tasks.rados.rados.0.smithi107.stdout:6557: done (1 left) 2020-06-12T02:38:29.217 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:29.217 INFO:tasks.rados.rados.0.smithi107.stdout:6558: seq_num 6557 ranges {1728512=622592,2351104=794624,3145728=409600} 2020-06-12T02:38:29.222 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-06-12T02:38:29.229 INFO:tasks.rados.rados.0.smithi107.stdout:6558: writing smithi10729407-12 from 1728512 to 2351104 tid 1 2020-06-12T02:38:29.233 INFO:tasks.rados.rados.0.smithi107.stdout:6558: writing smithi10729407-12 from 2351104 to 3145728 tid 2 2020-06-12T02:38:29.236 INFO:tasks.rados.rados.0.smithi107.stdout:6558: writing smithi10729407-12 from 3145728 to 3555328 tid 3 2020-06-12T02:38:29.236 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:29.237 INFO:tasks.rados.rados.0.smithi107.stdout:6558: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:29.237 INFO:tasks.rados.rados.0.smithi107.stdout:6559: seq_num 6558 ranges {0=638976,638976=679936,1318912=524288} 2020-06-12T02:38:29.240 INFO:tasks.rados.rados.0.smithi107.stdout:6559: writing smithi10729407-9 from 0 to 638976 tid 1 2020-06-12T02:38:29.247 INFO:tasks.rados.rados.0.smithi107.stdout:6559: writing smithi10729407-9 from 638976 to 1318912 tid 2 2020-06-12T02:38:29.248 INFO:tasks.rados.rados.0.smithi107.stdout:6559: writing smithi10729407-9 from 1318912 to 1843200 tid 3 2020-06-12T02:38:29.248 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:29.249 INFO:tasks.rados.rados.0.smithi107.stdout:6560: seq_num 6559 ranges {0=655360,655360=565248,1220608=638976} 2020-06-12T02:38:29.252 INFO:tasks.rados.rados.0.smithi107.stdout:6560: writing smithi10729407-4 from 0 to 655360 tid 1 2020-06-12T02:38:29.255 INFO:tasks.rados.rados.0.smithi107.stdout:6560: writing smithi10729407-4 from 655360 to 1220608 tid 2 2020-06-12T02:38:29.259 INFO:tasks.rados.rados.0.smithi107.stdout:6560: writing smithi10729407-4 from 1220608 to 1859584 tid 3 2020-06-12T02:38:29.260 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:38:29.260 INFO:tasks.rados.rados.0.smithi107.stdout:6561: seq_num 6560 ranges {0=671744,671744=450560,1122304=524288,1646592=229376} 2020-06-12T02:38:29.264 INFO:tasks.rados.rados.0.smithi107.stdout:6561: writing smithi10729407-6 from 0 to 671744 tid 1 2020-06-12T02:38:29.267 INFO:tasks.rados.rados.0.smithi107.stdout:6561: writing smithi10729407-6 from 671744 to 1122304 tid 2 2020-06-12T02:38:29.270 INFO:tasks.rados.rados.0.smithi107.stdout:6561: writing smithi10729407-6 from 1122304 to 1646592 tid 3 2020-06-12T02:38:29.272 INFO:tasks.rados.rados.0.smithi107.stdout:6561: writing smithi10729407-6 from 1646592 to 1875968 tid 4 2020-06-12T02:38:29.272 INFO:tasks.rados.rados.0.smithi107.stdout:6562: delete oid 13 current snap is 0 2020-06-12T02:38:29.273 INFO:tasks.rados.rados.0.smithi107.stdout:6558: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:29.273 INFO:tasks.rados.rados.0.smithi107.stdout:6558: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:29.273 INFO:tasks.rados.rados.0.smithi107.stdout:6558: finishing write tid 5 to smithi10729407-12 2020-06-12T02:38:29.273 INFO:tasks.rados.rados.0.smithi107.stdout:6559: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:29.273 INFO:tasks.rados.rados.0.smithi107.stdout:6558: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:29.274 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1199 (ObjNum 6557 snap 0 seq_num 6557) dirty exists 2020-06-12T02:38:29.274 INFO:tasks.rados.rados.0.smithi107.stdout:6558: left oid 12 (ObjNum 6557 snap 0 seq_num 6557) 2020-06-12T02:38:29.274 INFO:tasks.rados.rados.0.smithi107.stdout:6559: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:29.274 INFO:tasks.rados.rados.0.smithi107.stdout:6559: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:29.274 INFO:tasks.rados.rados.0.smithi107.stdout:6559: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:29.274 INFO:tasks.rados.rados.0.smithi107.stdout:6560: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:29.275 INFO:tasks.rados.rados.0.smithi107.stdout:6559: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:29.275 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1195 (ObjNum 6558 snap 0 seq_num 6558) dirty exists 2020-06-12T02:38:29.275 INFO:tasks.rados.rados.0.smithi107.stdout:6559: left oid 9 (ObjNum 6558 snap 0 seq_num 6558) 2020-06-12T02:38:29.275 INFO:tasks.rados.rados.0.smithi107.stdout:6560: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:29.275 INFO:tasks.rados.rados.0.smithi107.stdout:6560: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:29.276 INFO:tasks.rados.rados.0.smithi107.stdout:6560: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:29.276 INFO:tasks.rados.rados.0.smithi107.stdout:6560: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:29.277 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1231 (ObjNum 6559 snap 0 seq_num 6559) dirty exists 2020-06-12T02:38:29.277 INFO:tasks.rados.rados.0.smithi107.stdout:6560: left oid 4 (ObjNum 6559 snap 0 seq_num 6559) 2020-06-12T02:38:29.277 INFO:tasks.rados.rados.0.smithi107.stdout:6558: done (5 left) 2020-06-12T02:38:29.277 INFO:tasks.rados.rados.0.smithi107.stdout:6559: done (4 left) 2020-06-12T02:38:29.278 INFO:tasks.rados.rados.0.smithi107.stdout:6560: done (3 left) 2020-06-12T02:38:29.278 INFO:tasks.rados.rados.0.smithi107.stdout:6562: done (2 left) 2020-06-12T02:38:29.278 INFO:tasks.rados.rados.0.smithi107.stdout:6563: delete oid 3 current snap is 0 2020-06-12T02:38:29.279 INFO:tasks.rados.rados.0.smithi107.stdout:6561: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:29.279 INFO:tasks.rados.rados.0.smithi107.stdout:6563: done (2 left) 2020-06-12T02:38:29.279 INFO:tasks.rados.rados.0.smithi107.stdout:6564: delete oid 1 current snap is 0 2020-06-12T02:38:29.280 INFO:tasks.rados.rados.0.smithi107.stdout:6561: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:29.280 INFO:tasks.rados.rados.0.smithi107.stdout:6561: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:29.282 INFO:tasks.rados.rados.0.smithi107.stdout:6561: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:29.284 INFO:tasks.rados.rados.0.smithi107.stdout:6561: finishing write tid 6 to smithi10729407-6 2020-06-12T02:38:29.285 INFO:tasks.rados.rados.0.smithi107.stdout:6561: finishing write tid 7 to smithi10729407-6 2020-06-12T02:38:29.286 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1195 (ObjNum 6560 snap 0 seq_num 6560) dirty exists 2020-06-12T02:38:29.286 INFO:tasks.rados.rados.0.smithi107.stdout:6561: left oid 6 (ObjNum 6560 snap 0 seq_num 6560) 2020-06-12T02:38:29.287 INFO:tasks.rados.rados.0.smithi107.stdout:6561: done (2 left) 2020-06-12T02:38:29.287 INFO:tasks.rados.rados.0.smithi107.stdout:6564: done (1 left) 2020-06-12T02:38:29.287 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:38:29.287 INFO:tasks.rados.rados.0.smithi107.stdout:6565: seq_num 6564 ranges {0=737280,737280=434176,1171456=499712,1671168=270336} 2020-06-12T02:38:29.291 INFO:tasks.rados.rados.0.smithi107.stdout:6565: writing smithi10729407-16 from 0 to 737280 tid 1 2020-06-12T02:38:29.293 INFO:tasks.rados.rados.0.smithi107.stdout:6565: writing smithi10729407-16 from 737280 to 1171456 tid 2 2020-06-12T02:38:29.297 INFO:tasks.rados.rados.0.smithi107.stdout:6565: writing smithi10729407-16 from 1171456 to 1671168 tid 3 2020-06-12T02:38:29.298 INFO:tasks.rados.rados.0.smithi107.stdout:6565: writing smithi10729407-16 from 1671168 to 1941504 tid 4 2020-06-12T02:38:29.299 INFO:tasks.rados.rados.0.smithi107.stdout:6566: delete oid 12 current snap is 0 2020-06-12T02:38:29.302 INFO:tasks.rados.rados.0.smithi107.stdout:6566: done (2 left) 2020-06-12T02:38:29.302 INFO:tasks.rados.rados.0.smithi107.stdout:6567: delete oid 2 current snap is 0 2020-06-12T02:38:29.304 INFO:tasks.rados.rados.0.smithi107.stdout:6565: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:29.305 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:38:29.302+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:38:29.305 INFO:tasks.rados.rados.0.smithi107.stdout:6565: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:29.306 INFO:tasks.rados.rados.0.smithi107.stdout:6565: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:29.306 INFO:tasks.rados.rados.0.smithi107.stdout:6567: done (2 left) 2020-06-12T02:38:29.307 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:29.307 INFO:tasks.rados.rados.0.smithi107.stdout:6568: seq_num 6567 ranges {0=794624,794624=483328,1277952=720896} 2020-06-12T02:38:29.310 INFO:tasks.rados.rados.0.smithi107.stdout:6568: writing smithi10729407-2 from 0 to 794624 tid 1 2020-06-12T02:38:29.313 INFO:tasks.rados.rados.0.smithi107.stdout:6568: writing smithi10729407-2 from 794624 to 1277952 tid 2 2020-06-12T02:38:29.318 INFO:tasks.rados.rados.0.smithi107.stdout:6568: writing smithi10729407-2 from 1277952 to 1998848 tid 3 2020-06-12T02:38:29.318 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:38:29.318 INFO:tasks.rados.rados.0.smithi107.stdout:6569: seq_num 6568 ranges {0=409600,409600=770048,1179648=483328,1662976=352256} 2020-06-12T02:38:29.320 INFO:tasks.rados.rados.0.smithi107.stdout:6569: writing smithi10729407-1 from 0 to 409600 tid 1 2020-06-12T02:38:29.340 INFO:tasks.rados.rados.0.smithi107.stdout:6569: writing smithi10729407-1 from 409600 to 1179648 tid 2 2020-06-12T02:38:29.341 INFO:tasks.rados.rados.0.smithi107.stdout:6569: writing smithi10729407-1 from 1179648 to 1662976 tid 3 2020-06-12T02:38:29.341 INFO:tasks.rados.rados.0.smithi107.stdout:6569: writing smithi10729407-1 from 1662976 to 2015232 tid 4 2020-06-12T02:38:29.341 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:29.341 INFO:tasks.rados.rados.0.smithi107.stdout:6570: seq_num 6569 ranges {0=425984,425984=704512,1130496=688128,1818624=212992} 2020-06-12T02:38:29.342 INFO:tasks.rados.rados.0.smithi107.stdout:6570: writing smithi10729407-17 from 0 to 425984 tid 1 2020-06-12T02:38:29.342 INFO:tasks.rados.rados.0.smithi107.stdout:6570: writing smithi10729407-17 from 425984 to 1130496 tid 2 2020-06-12T02:38:29.343 INFO:tasks.rados.rados.0.smithi107.stdout:6570: writing smithi10729407-17 from 1130496 to 1818624 tid 3 2020-06-12T02:38:29.343 INFO:tasks.rados.rados.0.smithi107.stdout:6570: writing smithi10729407-17 from 1818624 to 2031616 tid 4 2020-06-12T02:38:29.343 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:29.343 INFO:tasks.rados.rados.0.smithi107.stdout:6571: seq_num 6570 ranges {0=442368,442368=589824,1032192=499712,1531904=516096} 2020-06-12T02:38:29.345 INFO:tasks.rados.rados.0.smithi107.stdout:6571: writing smithi10729407-3 from 0 to 442368 tid 1 2020-06-12T02:38:29.348 INFO:tasks.rados.rados.0.smithi107.stdout:6571: writing smithi10729407-3 from 442368 to 1032192 tid 2 2020-06-12T02:38:29.351 INFO:tasks.rados.rados.0.smithi107.stdout:6571: writing smithi10729407-3 from 1032192 to 1531904 tid 3 2020-06-12T02:38:29.355 INFO:tasks.rados.rados.0.smithi107.stdout:6571: writing smithi10729407-3 from 1531904 to 2048000 tid 4 2020-06-12T02:38:29.355 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:29.355 INFO:tasks.rados.rados.0.smithi107.stdout:6572: seq_num 6571 ranges {1843200=458752,2301952=475136,2777088=704512,3481600=425984} 2020-06-12T02:38:29.368 INFO:tasks.rados.rados.0.smithi107.stdout:6572: writing smithi10729407-9 from 1843200 to 2301952 tid 1 2020-06-12T02:38:29.370 INFO:tasks.rados.rados.0.smithi107.stdout:6572: writing smithi10729407-9 from 2301952 to 2777088 tid 2 2020-06-12T02:38:29.383 INFO:tasks.rados.rados.0.smithi107.stdout:6572: writing smithi10729407-9 from 2777088 to 3481600 tid 3 2020-06-12T02:38:29.384 INFO:tasks.rados.rados.0.smithi107.stdout:6572: writing smithi10729407-9 from 3481600 to 3907584 tid 4 2020-06-12T02:38:29.384 INFO:tasks.rados.rados.0.smithi107.stdout:6573: delete oid 12 current snap is 0 2020-06-12T02:38:29.384 INFO:tasks.rados.rados.0.smithi107.stdout:6565: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:29.384 INFO:tasks.rados.rados.0.smithi107.stdout:6565: finishing write tid 6 to smithi10729407-16 2020-06-12T02:38:29.384 INFO:tasks.rados.rados.0.smithi107.stdout:6565: finishing write tid 7 to smithi10729407-16 2020-06-12T02:38:29.385 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1155 (ObjNum 6564 snap 0 seq_num 6564) dirty exists 2020-06-12T02:38:29.385 INFO:tasks.rados.rados.0.smithi107.stdout:6565: left oid 16 (ObjNum 6564 snap 0 seq_num 6564) 2020-06-12T02:38:29.385 INFO:tasks.rados.rados.0.smithi107.stdout:6568: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:29.385 INFO:tasks.rados.rados.0.smithi107.stdout:6568: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:29.386 INFO:tasks.rados.rados.0.smithi107.stdout:6568: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:29.386 INFO:tasks.rados.rados.0.smithi107.stdout:6568: finishing write tid 5 to smithi10729407-2 2020-06-12T02:38:29.386 INFO:tasks.rados.rados.0.smithi107.stdout:6569: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:29.386 INFO:tasks.rados.rados.0.smithi107.stdout:6568: finishing write tid 6 to smithi10729407-2 2020-06-12T02:38:29.386 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 635 (ObjNum 6567 snap 0 seq_num 6567) dirty exists 2020-06-12T02:38:29.387 INFO:tasks.rados.rados.0.smithi107.stdout:6568: left oid 2 (ObjNum 6567 snap 0 seq_num 6567) 2020-06-12T02:38:29.387 INFO:tasks.rados.rados.0.smithi107.stdout:6569: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:29.387 INFO:tasks.rados.rados.0.smithi107.stdout:6569: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:29.387 INFO:tasks.rados.rados.0.smithi107.stdout:6569: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:29.387 INFO:tasks.rados.rados.0.smithi107.stdout:6569: finishing write tid 6 to smithi10729407-1 2020-06-12T02:38:29.387 INFO:tasks.rados.rados.0.smithi107.stdout:6569: finishing write tid 7 to smithi10729407-1 2020-06-12T02:38:29.388 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1201 (ObjNum 6568 snap 0 seq_num 6568) dirty exists 2020-06-12T02:38:29.388 INFO:tasks.rados.rados.0.smithi107.stdout:6569: left oid 1 (ObjNum 6568 snap 0 seq_num 6568) 2020-06-12T02:38:29.388 INFO:tasks.rados.rados.0.smithi107.stdout:6570: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:29.388 INFO:tasks.rados.rados.0.smithi107.stdout:6570: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:29.388 INFO:tasks.rados.rados.0.smithi107.stdout:6571: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:29.389 INFO:tasks.rados.rados.0.smithi107.stdout:6570: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:29.389 INFO:tasks.rados.rados.0.smithi107.stdout:6570: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:29.389 INFO:tasks.rados.rados.0.smithi107.stdout:6570: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:29.389 INFO:tasks.rados.rados.0.smithi107.stdout:6570: finishing write tid 7 to smithi10729407-17 2020-06-12T02:38:29.389 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1236 (ObjNum 6569 snap 0 seq_num 6569) dirty exists 2020-06-12T02:38:29.390 INFO:tasks.rados.rados.0.smithi107.stdout:6570: left oid 17 (ObjNum 6569 snap 0 seq_num 6569) 2020-06-12T02:38:29.390 INFO:tasks.rados.rados.0.smithi107.stdout:6571: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:29.390 INFO:tasks.rados.rados.0.smithi107.stdout:6571: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:29.390 INFO:tasks.rados.rados.0.smithi107.stdout:6571: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:29.390 INFO:tasks.rados.rados.0.smithi107.stdout:6571: finishing write tid 6 to smithi10729407-3 2020-06-12T02:38:29.390 INFO:tasks.rados.rados.0.smithi107.stdout:6571: finishing write tid 7 to smithi10729407-3 2020-06-12T02:38:29.391 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1160 (ObjNum 6570 snap 0 seq_num 6570) dirty exists 2020-06-12T02:38:29.391 INFO:tasks.rados.rados.0.smithi107.stdout:6571: left oid 3 (ObjNum 6570 snap 0 seq_num 6570) 2020-06-12T02:38:29.391 INFO:tasks.rados.rados.0.smithi107.stdout:6572: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:29.391 INFO:tasks.rados.rados.0.smithi107.stdout:6565: done (7 left) 2020-06-12T02:38:29.391 INFO:tasks.rados.rados.0.smithi107.stdout:6568: done (6 left) 2020-06-12T02:38:29.392 INFO:tasks.rados.rados.0.smithi107.stdout:6569: done (5 left) 2020-06-12T02:38:29.392 INFO:tasks.rados.rados.0.smithi107.stdout:6570: done (4 left) 2020-06-12T02:38:29.392 INFO:tasks.rados.rados.0.smithi107.stdout:6571: done (3 left) 2020-06-12T02:38:29.392 INFO:tasks.rados.rados.0.smithi107.stdout:6573: done (2 left) 2020-06-12T02:38:29.393 INFO:tasks.rados.rados.0.smithi107.stdout:6574: delete oid 6 current snap is 0 2020-06-12T02:38:29.394 INFO:tasks.rados.rados.0.smithi107.stdout:6572: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:29.394 INFO:tasks.rados.rados.0.smithi107.stdout:6572: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:29.394 INFO:tasks.rados.rados.0.smithi107.stdout:6572: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:29.395 INFO:tasks.rados.rados.0.smithi107.stdout:6574: done (2 left) 2020-06-12T02:38:29.395 INFO:tasks.rados.rados.0.smithi107.stdout:6575: delete oid 8 current snap is 0 2020-06-12T02:38:29.395 INFO:tasks.rados.rados.0.smithi107.stdout:6572: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:29.395 INFO:tasks.rados.rados.0.smithi107.stdout:6572: finishing write tid 7 to smithi10729407-9 2020-06-12T02:38:29.395 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1200 (ObjNum 6571 snap 0 seq_num 6571) dirty exists 2020-06-12T02:38:29.396 INFO:tasks.rados.rados.0.smithi107.stdout:6572: left oid 9 (ObjNum 6571 snap 0 seq_num 6571) 2020-06-12T02:38:29.396 INFO:tasks.rados.rados.0.smithi107.stdout:6572: done (2 left) 2020-06-12T02:38:29.396 INFO:tasks.rados.rados.0.smithi107.stdout:6575: done (1 left) 2020-06-12T02:38:29.396 INFO:tasks.rados.rados.0.smithi107.stdout:6576: delete oid 16 current snap is 0 2020-06-12T02:38:29.396 INFO:tasks.rados.rados.0.smithi107.stdout:6576: done (1 left) 2020-06-12T02:38:29.397 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:29.397 INFO:tasks.rados.rados.0.smithi107.stdout:6577: seq_num 6576 ranges {1998848=540672,2539520=696320,3235840=442368,3678208=466944} 2020-06-12T02:38:29.399 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:38:29.396+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:38:29.407 INFO:tasks.rados.rados.0.smithi107.stdout:6577: writing smithi10729407-2 from 1998848 to 2539520 tid 1 2020-06-12T02:38:29.411 INFO:tasks.rados.rados.0.smithi107.stdout:6577: writing smithi10729407-2 from 2539520 to 3235840 tid 2 2020-06-12T02:38:29.414 INFO:tasks.rados.rados.0.smithi107.stdout:6577: writing smithi10729407-2 from 3235840 to 3678208 tid 3 2020-06-12T02:38:29.422 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2020-06-12T02:38:29.422 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2020-06-12T02:38:29.423 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump -f json-pretty 2020-06-12T02:38:29.426 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:29.427 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:29.427 INFO:tasks.rados.rados.0.smithi107.stdout:6577: writing smithi10729407-2 from 3678208 to 4145152 tid 4 2020-06-12T02:38:29.427 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:29.427 INFO:tasks.rados.rados.0.smithi107.stdout:6578: seq_num 6577 ranges {0=557056,557056=630784,1187840=655360,1843200=319488} 2020-06-12T02:38:29.427 INFO:tasks.rados.rados.0.smithi107.stdout:6578: writing smithi10729407-15 from 0 to 557056 tid 1 2020-06-12T02:38:29.428 INFO:tasks.rados.rados.0.smithi107.stdout:6578: writing smithi10729407-15 from 557056 to 1187840 tid 2 2020-06-12T02:38:29.428 INFO:tasks.rados.rados.0.smithi107.stdout:6578: writing smithi10729407-15 from 1187840 to 1843200 tid 3 2020-06-12T02:38:29.430 INFO:tasks.rados.rados.0.smithi107.stdout:6578: writing smithi10729407-15 from 1843200 to 2162688 tid 4 2020-06-12T02:38:29.430 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:38:29.430 INFO:tasks.rados.rados.0.smithi107.stdout:6579: seq_num 6578 ranges {0=573440,573440=516096,1089536=458752,1548288=417792,1966080=212992} 2020-06-12T02:38:29.432 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd unset noscrub 2020-06-12T02:38:29.433 INFO:tasks.rados.rados.0.smithi107.stdout:6579: writing smithi10729407-16 from 0 to 573440 tid 1 2020-06-12T02:38:29.435 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-06-12T02:38:29.440 INFO:tasks.rados.rados.0.smithi107.stdout:6579: writing smithi10729407-16 from 573440 to 1089536 tid 2 2020-06-12T02:38:29.440 INFO:tasks.rados.rados.0.smithi107.stdout:6579: writing smithi10729407-16 from 1089536 to 1548288 tid 3 2020-06-12T02:38:29.442 INFO:tasks.rados.rados.0.smithi107.stdout:6579: writing smithi10729407-16 from 1548288 to 1966080 tid 4 2020-06-12T02:38:29.443 INFO:tasks.rados.rados.0.smithi107.stdout:6579: writing smithi10729407-16 from 1966080 to 2179072 tid 5 2020-06-12T02:38:29.443 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:29.443 INFO:tasks.rados.rados.0.smithi107.stdout:6580: seq_num 6579 ranges {0=589824,589824=802816,1392640=671744,2064384=131072} 2020-06-12T02:38:29.447 INFO:tasks.rados.rados.0.smithi107.stdout:6580: writing smithi10729407-12 from 0 to 589824 tid 1 2020-06-12T02:38:29.452 INFO:tasks.rados.rados.0.smithi107.stdout:6580: writing smithi10729407-12 from 589824 to 1392640 tid 2 2020-06-12T02:38:29.457 INFO:tasks.rados.rados.0.smithi107.stdout:6580: writing smithi10729407-12 from 1392640 to 2064384 tid 3 2020-06-12T02:38:29.459 INFO:tasks.rados.rados.0.smithi107.stdout:6580: writing smithi10729407-12 from 2064384 to 2195456 tid 4 2020-06-12T02:38:29.459 INFO:tasks.rados.rados.0.smithi107.stdout:6581: delete oid 7 current snap is 0 2020-06-12T02:38:29.460 INFO:tasks.rados.rados.0.smithi107.stdout:6577: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:29.460 INFO:tasks.rados.rados.0.smithi107.stdout:6577: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:29.460 INFO:tasks.rados.rados.0.smithi107.stdout:6577: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:29.461 INFO:tasks.rados.rados.0.smithi107.stdout:6577: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:29.461 INFO:tasks.rados.rados.0.smithi107.stdout:6577: finishing write tid 6 to smithi10729407-2 2020-06-12T02:38:29.462 INFO:tasks.rados.rados.0.smithi107.stdout:6577: finishing write tid 7 to smithi10729407-2 2020-06-12T02:38:29.462 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 640 (ObjNum 6576 snap 0 seq_num 6576) dirty exists 2020-06-12T02:38:29.462 INFO:tasks.rados.rados.0.smithi107.stdout:6577: left oid 2 (ObjNum 6576 snap 0 seq_num 6576) 2020-06-12T02:38:29.462 INFO:tasks.rados.rados.0.smithi107.stdout:6578: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:29.463 INFO:tasks.rados.rados.0.smithi107.stdout:6578: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:29.463 INFO:tasks.rados.rados.0.smithi107.stdout:6578: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:29.463 INFO:tasks.rados.rados.0.smithi107.stdout:6578: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:29.464 INFO:tasks.rados.rados.0.smithi107.stdout:6578: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:29.464 INFO:tasks.rados.rados.0.smithi107.stdout:6578: finishing write tid 7 to smithi10729407-15 2020-06-12T02:38:29.464 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1225 (ObjNum 6577 snap 0 seq_num 6577) dirty exists 2020-06-12T02:38:29.465 INFO:tasks.rados.rados.0.smithi107.stdout:6578: left oid 15 (ObjNum 6577 snap 0 seq_num 6577) 2020-06-12T02:38:29.465 INFO:tasks.rados.rados.0.smithi107.stdout:6579: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:29.465 INFO:tasks.rados.rados.0.smithi107.stdout:6579: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:29.465 INFO:tasks.rados.rados.0.smithi107.stdout:6579: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:29.466 INFO:tasks.rados.rados.0.smithi107.stdout:6579: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:29.466 INFO:tasks.rados.rados.0.smithi107.stdout:6579: finishing write tid 5 to smithi10729407-16 2020-06-12T02:38:29.466 INFO:tasks.rados.rados.0.smithi107.stdout:6579: finishing write tid 7 to smithi10729407-16 2020-06-12T02:38:29.466 INFO:tasks.rados.rados.0.smithi107.stdout:6579: finishing write tid 8 to smithi10729407-16 2020-06-12T02:38:29.467 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1167 (ObjNum 6578 snap 0 seq_num 6578) dirty exists 2020-06-12T02:38:29.467 INFO:tasks.rados.rados.0.smithi107.stdout:6579: left oid 16 (ObjNum 6578 snap 0 seq_num 6578) 2020-06-12T02:38:29.467 INFO:tasks.rados.rados.0.smithi107.stdout:6580: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:29.468 INFO:tasks.rados.rados.0.smithi107.stdout:6577: done (5 left) 2020-06-12T02:38:29.469 INFO:tasks.rados.rados.0.smithi107.stdout:6578: done (4 left) 2020-06-12T02:38:29.469 INFO:tasks.rados.rados.0.smithi107.stdout:6579: done (3 left) 2020-06-12T02:38:29.469 INFO:tasks.rados.rados.0.smithi107.stdout:6581: done (2 left) 2020-06-12T02:38:29.469 INFO:tasks.rados.rados.0.smithi107.stdout:6582: delete oid 15 current snap is 0 2020-06-12T02:38:29.470 INFO:tasks.rados.rados.0.smithi107.stdout:6580: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:29.470 INFO:tasks.rados.rados.0.smithi107.stdout:6582: done (2 left) 2020-06-12T02:38:29.470 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:29.470 INFO:tasks.rados.rados.0.smithi107.stdout:6583: seq_num 6582 ranges {966656=638976,1605632=450560,2056192=442368,2498560=425984,2924544=286720} 2020-06-12T02:38:29.473 INFO:tasks.rados.rados.0.smithi107.stdout:6583: writing smithi10729407-18 from 966656 to 1605632 tid 1 2020-06-12T02:38:29.476 INFO:tasks.rados.rados.0.smithi107.stdout:6583: writing smithi10729407-18 from 1605632 to 2056192 tid 2 2020-06-12T02:38:29.479 INFO:tasks.rados.rados.0.smithi107.stdout:6583: writing smithi10729407-18 from 2056192 to 2498560 tid 3 2020-06-12T02:38:29.482 INFO:tasks.rados.rados.0.smithi107.stdout:6583: writing smithi10729407-18 from 2498560 to 2924544 tid 4 2020-06-12T02:38:29.484 INFO:tasks.rados.rados.0.smithi107.stdout:6583: writing smithi10729407-18 from 2924544 to 3211264 tid 5 2020-06-12T02:38:29.485 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:29.485 INFO:tasks.rados.rados.0.smithi107.stdout:6584: seq_num 6583 ranges {0=655360,655360=737280,1392640=647168,2039808=221184} 2020-06-12T02:38:29.489 INFO:tasks.rados.rados.0.smithi107.stdout:6584: writing smithi10729407-14 from 0 to 655360 tid 1 2020-06-12T02:38:29.494 INFO:tasks.rados.rados.0.smithi107.stdout:6584: writing smithi10729407-14 from 655360 to 1392640 tid 2 2020-06-12T02:38:29.498 INFO:tasks.rados.rados.0.smithi107.stdout:6584: writing smithi10729407-14 from 1392640 to 2039808 tid 3 2020-06-12T02:38:29.499 INFO:tasks.rados.rados.0.smithi107.stdout:6584: writing smithi10729407-14 from 2039808 to 2260992 tid 4 2020-06-12T02:38:29.500 INFO:tasks.rados.rados.0.smithi107.stdout:6585: delete oid 8 current snap is 0 2020-06-12T02:38:29.500 INFO:tasks.rados.rados.0.smithi107.stdout:6580: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:29.500 INFO:tasks.rados.rados.0.smithi107.stdout:6580: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:29.500 INFO:tasks.rados.rados.0.smithi107.stdout:6580: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:29.501 INFO:tasks.rados.rados.0.smithi107.stdout:6580: finishing write tid 7 to smithi10729407-12 2020-06-12T02:38:29.501 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1206 (ObjNum 6579 snap 0 seq_num 6579) dirty exists 2020-06-12T02:38:29.501 INFO:tasks.rados.rados.0.smithi107.stdout:6580: left oid 12 (ObjNum 6579 snap 0 seq_num 6579) 2020-06-12T02:38:29.501 INFO:tasks.rados.rados.0.smithi107.stdout:6583: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:29.501 INFO:tasks.rados.rados.0.smithi107.stdout:6583: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:29.501 INFO:tasks.rados.rados.0.smithi107.stdout:6583: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:29.502 INFO:tasks.rados.rados.0.smithi107.stdout:6583: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:29.503 INFO:tasks.rados.rados.0.smithi107.stdout:6583: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:29.504 INFO:tasks.rados.rados.0.smithi107.stdout:6583: finishing write tid 7 to smithi10729407-18 2020-06-12T02:38:29.504 INFO:tasks.rados.rados.0.smithi107.stdout:6584: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:29.504 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:29.497+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:29.505 INFO:tasks.rados.rados.0.smithi107.stdout:6580: done (4 left) 2020-06-12T02:38:29.505 INFO:tasks.rados.rados.0.smithi107.stdout:6585: done (3 left) 2020-06-12T02:38:29.505 INFO:tasks.rados.rados.0.smithi107.stdout:6586: delete oid 2 current snap is 0 2020-06-12T02:38:29.507 INFO:tasks.rados.rados.0.smithi107.stdout:6583: finishing write tid 8 to smithi10729407-18 2020-06-12T02:38:29.508 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1206 (ObjNum 6582 snap 0 seq_num 6582) dirty exists 2020-06-12T02:38:29.508 INFO:tasks.rados.rados.0.smithi107.stdout:6583: left oid 18 (ObjNum 6582 snap 0 seq_num 6582) 2020-06-12T02:38:29.510 INFO:tasks.rados.rados.0.smithi107.stdout:6583: done (3 left) 2020-06-12T02:38:29.510 INFO:tasks.rados.rados.0.smithi107.stdout:6586: done (2 left) 2020-06-12T02:38:29.510 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:29.510 INFO:tasks.rados.rados.0.smithi107.stdout:6587: seq_num 6586 ranges {2048000=712704,2760704=434176,3194880=417792,3612672=475136,4087808=278528} 2020-06-12T02:38:29.525 INFO:tasks.rados.rados.0.smithi107.stdout:6587: writing smithi10729407-3 from 2048000 to 2760704 tid 1 2020-06-12T02:38:29.528 INFO:tasks.rados.rados.0.smithi107.stdout:6587: writing smithi10729407-3 from 2760704 to 3194880 tid 2 2020-06-12T02:38:29.531 INFO:tasks.rados.rados.0.smithi107.stdout:6587: writing smithi10729407-3 from 3194880 to 3612672 tid 3 2020-06-12T02:38:29.543 INFO:tasks.rados.rados.0.smithi107.stdout:6587: writing smithi10729407-3 from 3612672 to 4087808 tid 4 2020-06-12T02:38:29.544 INFO:tasks.rados.rados.0.smithi107.stdout:6587: writing smithi10729407-3 from 4087808 to 4366336 tid 5 2020-06-12T02:38:29.544 INFO:tasks.rados.rados.0.smithi107.stdout:6588: delete oid 15 current snap is 0 2020-06-12T02:38:29.544 INFO:tasks.rados.rados.0.smithi107.stdout:6584: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:29.544 INFO:tasks.rados.rados.0.smithi107.stdout:6584: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:29.544 INFO:tasks.rados.rados.0.smithi107.stdout:6584: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:29.545 INFO:tasks.rados.rados.0.smithi107.stdout:6584: finishing write tid 6 to smithi10729407-14 2020-06-12T02:38:29.545 INFO:tasks.rados.rados.0.smithi107.stdout:6584: finishing write tid 7 to smithi10729407-14 2020-06-12T02:38:29.545 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1172 (ObjNum 6583 snap 0 seq_num 6583) dirty exists 2020-06-12T02:38:29.545 INFO:tasks.rados.rados.0.smithi107.stdout:6584: left oid 14 (ObjNum 6583 snap 0 seq_num 6583) 2020-06-12T02:38:29.546 INFO:tasks.rados.rados.0.smithi107.stdout:6587: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:29.546 INFO:tasks.rados.rados.0.smithi107.stdout:6584: done (3 left) 2020-06-12T02:38:29.546 INFO:tasks.rados.rados.0.smithi107.stdout:6588: done (2 left) 2020-06-12T02:38:29.547 INFO:tasks.rados.rados.0.smithi107.stdout:6589: delete oid 5 current snap is 0 2020-06-12T02:38:29.547 INFO:tasks.rados.rados.0.smithi107.stdout:6589: done (2 left) 2020-06-12T02:38:29.548 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:29.548 INFO:tasks.rados.rados.0.smithi107.stdout:6590: seq_num 6589 ranges {1859584=761856,2621440=491520,3112960=598016,3710976=516096} 2020-06-12T02:38:29.561 INFO:tasks.rados.rados.0.smithi107.stdout:6590: writing smithi10729407-4 from 1859584 to 2621440 tid 1 2020-06-12T02:38:29.563 INFO:tasks.rados.rados.0.smithi107.stdout:6590: writing smithi10729407-4 from 2621440 to 3112960 tid 2 2020-06-12T02:38:29.566 INFO:tasks.rados.rados.0.smithi107.stdout:6590: writing smithi10729407-4 from 3112960 to 3710976 tid 3 2020-06-12T02:38:29.570 INFO:tasks.rados.rados.0.smithi107.stdout:6590: writing smithi10729407-4 from 3710976 to 4227072 tid 4 2020-06-12T02:38:29.570 INFO:tasks.rados.rados.0.smithi107.stdout:6591: delete oid 14 current snap is 0 2020-06-12T02:38:29.571 INFO:tasks.rados.rados.0.smithi107.stdout:6587: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:29.571 INFO:tasks.rados.rados.0.smithi107.stdout:6587: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:29.571 INFO:tasks.rados.rados.0.smithi107.stdout:6587: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:29.571 INFO:tasks.rados.rados.0.smithi107.stdout:6587: finishing write tid 5 to smithi10729407-3 2020-06-12T02:38:29.572 INFO:tasks.rados.rados.0.smithi107.stdout:6587: finishing write tid 7 to smithi10729407-3 2020-06-12T02:38:29.572 INFO:tasks.rados.rados.0.smithi107.stdout:6587: finishing write tid 8 to smithi10729407-3 2020-06-12T02:38:29.572 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1173 (ObjNum 6586 snap 0 seq_num 6586) dirty exists 2020-06-12T02:38:29.572 INFO:tasks.rados.rados.0.smithi107.stdout:6587: left oid 3 (ObjNum 6586 snap 0 seq_num 6586) 2020-06-12T02:38:29.579 INFO:tasks.rados.rados.0.smithi107.stdout:6590: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:29.579 INFO:tasks.rados.rados.0.smithi107.stdout:6587: done (3 left) 2020-06-12T02:38:29.580 INFO:tasks.rados.rados.0.smithi107.stdout:6591: done (2 left) 2020-06-12T02:38:29.580 INFO:tasks.rados.rados.0.smithi107.stdout:6592: delete oid 16 current snap is 0 2020-06-12T02:38:29.581 INFO:tasks.rados.rados.0.smithi107.stdout:6590: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:29.581 INFO:tasks.rados.rados.0.smithi107.stdout:6592: done (2 left) 2020-06-12T02:38:29.581 INFO:tasks.rados.rados.0.smithi107.stdout:6593: delete oid 9 current snap is 0 2020-06-12T02:38:29.581 INFO:tasks.rados.rados.0.smithi107.stdout:6590: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:29.584 INFO:tasks.rados.rados.0.smithi107.stdout:6590: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:29.584 INFO:tasks.rados.rados.0.smithi107.stdout:6590: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:29.586 INFO:tasks.rados.rados.0.smithi107.stdout:6593: done (2 left) 2020-06-12T02:38:29.586 INFO:tasks.rados.rados.0.smithi107.stdout:6594: delete oid 16 current snap is 0 2020-06-12T02:38:29.587 INFO:tasks.rados.rados.0.smithi107.stdout:6590: finishing write tid 7 to smithi10729407-4 2020-06-12T02:38:29.587 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1241 (ObjNum 6589 snap 0 seq_num 6589) dirty exists 2020-06-12T02:38:29.587 INFO:tasks.rados.rados.0.smithi107.stdout:6590: left oid 4 (ObjNum 6589 snap 0 seq_num 6589) 2020-06-12T02:38:29.591 INFO:tasks.rados.rados.0.smithi107.stdout:6590: done (2 left) 2020-06-12T02:38:29.592 INFO:tasks.rados.rados.0.smithi107.stdout:6594: done (1 left) 2020-06-12T02:38:29.592 INFO:tasks.rados.rados.0.smithi107.stdout:6595: delete oid 19 current snap is 0 2020-06-12T02:38:29.597 INFO:tasks.rados.rados.0.smithi107.stdout:6595: done (1 left) 2020-06-12T02:38:29.598 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:29.598 INFO:tasks.rados.rados.0.smithi107.stdout:6596: seq_num 6595 ranges {4366336=57344} 2020-06-12T02:38:29.600 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:29.597+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:29.615 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:29.615 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:29.618 INFO:tasks.rados.rados.0.smithi107.stdout:6596: writing smithi10729407-3 from 4366336 to 4423680 tid 1 2020-06-12T02:38:29.618 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:29.618 INFO:tasks.rados.rados.0.smithi107.stdout:6597: seq_num 6596 ranges {2031616=73728} 2020-06-12T02:38:29.624 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-06-12T02:38:29.630 INFO:tasks.rados.rados.0.smithi107.stdout:6597: writing smithi10729407-17 from 2031616 to 2105344 tid 1 2020-06-12T02:38:29.630 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:38:29.630 INFO:tasks.rados.rados.0.smithi107.stdout:6598: seq_num 6597 ranges {0=90112} 2020-06-12T02:38:29.631 INFO:tasks.rados.rados.0.smithi107.stdout:6598: writing smithi10729407-2 from 0 to 90112 tid 1 2020-06-12T02:38:29.631 INFO:tasks.rados.rados.0.smithi107.stdout:6599: delete oid 6 current snap is 0 2020-06-12T02:38:29.631 INFO:tasks.rados.rados.0.smithi107.stdout:6596: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:29.632 INFO:tasks.rados.rados.0.smithi107.stdout:6596: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:29.632 INFO:tasks.rados.rados.0.smithi107.stdout:6596: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:29.632 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1177 (ObjNum 6595 snap 0 seq_num 6595) dirty exists 2020-06-12T02:38:29.632 INFO:tasks.rados.rados.0.smithi107.stdout:6596: left oid 3 (ObjNum 6595 snap 0 seq_num 6595) 2020-06-12T02:38:29.633 INFO:tasks.rados.rados.0.smithi107.stdout:6597: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:29.633 INFO:tasks.rados.rados.0.smithi107.stdout:6596: done (4 left) 2020-06-12T02:38:29.633 INFO:tasks.rados.rados.0.smithi107.stdout:6599: done (3 left) 2020-06-12T02:38:29.634 INFO:tasks.rados.rados.0.smithi107.stdout:6600: delete oid 4 current snap is 0 2020-06-12T02:38:29.634 INFO:tasks.rados.rados.0.smithi107.stdout:6598: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:29.635 INFO:tasks.rados.rados.0.smithi107.stdout:6597: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:29.636 INFO:tasks.rados.rados.0.smithi107.stdout:6597: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:29.637 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1243 (ObjNum 6596 snap 0 seq_num 6596) dirty exists 2020-06-12T02:38:29.637 INFO:tasks.rados.rados.0.smithi107.stdout:6597: left oid 17 (ObjNum 6596 snap 0 seq_num 6596) 2020-06-12T02:38:29.638 INFO:tasks.rados.rados.0.smithi107.stdout:6598: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:29.638 INFO:tasks.rados.rados.0.smithi107.stdout:6597: done (3 left) 2020-06-12T02:38:29.639 INFO:tasks.rados.rados.0.smithi107.stdout:6600: done (2 left) 2020-06-12T02:38:29.639 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:29.639 INFO:tasks.rados.rados.0.smithi107.stdout:6601: seq_num 6600 ranges {0=139264} 2020-06-12T02:38:29.640 INFO:tasks.rados.rados.0.smithi107.stdout:6601: writing smithi10729407-15 from 0 to 139264 tid 1 2020-06-12T02:38:29.640 INFO:tasks.rados.rados.0.smithi107.stdout:6602: delete oid 4 current snap is 0 2020-06-12T02:38:29.641 INFO:tasks.rados.rados.0.smithi107.stdout:6598: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:29.641 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 643 (ObjNum 6597 snap 0 seq_num 6597) dirty exists 2020-06-12T02:38:29.641 INFO:tasks.rados.rados.0.smithi107.stdout:6598: left oid 2 (ObjNum 6597 snap 0 seq_num 6597) 2020-06-12T02:38:29.646 INFO:tasks.rados.rados.0.smithi107.stdout:6598: done (3 left) 2020-06-12T02:38:29.646 INFO:tasks.rados.rados.0.smithi107.stdout:6602: done (2 left) 2020-06-12T02:38:29.647 INFO:tasks.rados.rados.0.smithi107.stdout:6603: delete oid 19 current snap is 0 2020-06-12T02:38:29.647 INFO:tasks.rados.rados.0.smithi107.stdout:6601: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:29.647 INFO:tasks.rados.rados.0.smithi107.stdout:6601: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:29.648 INFO:tasks.rados.rados.0.smithi107.stdout:6601: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:29.648 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1230 (ObjNum 6600 snap 0 seq_num 6600) dirty exists 2020-06-12T02:38:29.649 INFO:tasks.rados.rados.0.smithi107.stdout:6601: left oid 15 (ObjNum 6600 snap 0 seq_num 6600) 2020-06-12T02:38:29.651 INFO:tasks.rados.rados.0.smithi107.stdout:6601: done (2 left) 2020-06-12T02:38:29.652 INFO:tasks.rados.rados.0.smithi107.stdout:6603: done (1 left) 2020-06-12T02:38:29.652 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:29.652 INFO:tasks.rados.rados.0.smithi107.stdout:6604: seq_num 6603 ranges {0=188416} 2020-06-12T02:38:29.653 INFO:tasks.rados.rados.0.smithi107.stdout:6604: writing smithi10729407-19 from 0 to 188416 tid 1 2020-06-12T02:38:29.653 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:38:29.653 INFO:tasks.rados.rados.0.smithi107.stdout:6605: seq_num 6604 ranges {0=212992} 2020-06-12T02:38:29.654 INFO:tasks.rados.rados.0.smithi107.stdout:6605: writing smithi10729407-16 from 0 to 212992 tid 1 2020-06-12T02:38:29.654 INFO:tasks.rados.rados.0.smithi107.stdout:6606: delete oid 17 current snap is 0 2020-06-12T02:38:29.658 INFO:tasks.rados.rados.0.smithi107.stdout:6604: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:29.658 INFO:tasks.rados.rados.0.smithi107.stdout:6606: done (3 left) 2020-06-12T02:38:29.659 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:38:29.659 INFO:tasks.rados.rados.0.smithi107.stdout:6607: seq_num 6606 ranges {0=245760} 2020-06-12T02:38:29.660 INFO:tasks.rados.rados.0.smithi107.stdout:6607: writing smithi10729407-7 from 0 to 245760 tid 1 2020-06-12T02:38:29.661 INFO:tasks.rados.rados.0.smithi107.stdout:6608: delete oid 17 current snap is 0 2020-06-12T02:38:29.661 INFO:tasks.rados.rados.0.smithi107.stdout:6605: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:29.661 INFO:tasks.rados.rados.0.smithi107.stdout:6605: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:29.663 INFO:tasks.rados.rados.0.smithi107.stdout:6604: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:29.664 INFO:tasks.rados.rados.0.smithi107.stdout:6605: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:29.664 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1179 (ObjNum 6604 snap 0 seq_num 6604) dirty exists 2020-06-12T02:38:29.664 INFO:tasks.rados.rados.0.smithi107.stdout:6605: left oid 16 (ObjNum 6604 snap 0 seq_num 6604) 2020-06-12T02:38:29.665 INFO:tasks.rados.rados.0.smithi107.stdout:6604: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:29.665 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1210 (ObjNum 6603 snap 0 seq_num 6603) dirty exists 2020-06-12T02:38:29.665 INFO:tasks.rados.rados.0.smithi107.stdout:6604: left oid 19 (ObjNum 6603 snap 0 seq_num 6603) 2020-06-12T02:38:29.666 INFO:tasks.rados.rados.0.smithi107.stdout:6607: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:29.668 INFO:tasks.rados.rados.0.smithi107.stdout:6604: done (4 left) 2020-06-12T02:38:29.668 INFO:tasks.rados.rados.0.smithi107.stdout:6605: done (3 left) 2020-06-12T02:38:29.668 INFO:tasks.rados.rados.0.smithi107.stdout:6608: done (2 left) 2020-06-12T02:38:29.668 INFO:tasks.rados.rados.0.smithi107.stdout:6609: delete oid 5 current snap is 0 2020-06-12T02:38:29.669 INFO:tasks.rados.rados.0.smithi107.stdout:6607: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:29.670 INFO:tasks.rados.rados.0.smithi107.stdout:6609: done (2 left) 2020-06-12T02:38:29.670 INFO:tasks.rados.rados.0.smithi107.stdout:6610: delete oid 12 current snap is 0 2020-06-12T02:38:29.673 INFO:tasks.rados.rados.0.smithi107.stdout:6607: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:29.673 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1241 (ObjNum 6606 snap 0 seq_num 6606) dirty exists 2020-06-12T02:38:29.673 INFO:tasks.rados.rados.0.smithi107.stdout:6607: left oid 7 (ObjNum 6606 snap 0 seq_num 6606) 2020-06-12T02:38:29.674 INFO:tasks.rados.rados.0.smithi107.stdout:6607: done (2 left) 2020-06-12T02:38:29.674 INFO:tasks.rados.rados.0.smithi107.stdout:6610: done (1 left) 2020-06-12T02:38:29.674 INFO:tasks.rados.rados.0.smithi107.stdout:6611: delete oid 20 current snap is 0 2020-06-12T02:38:29.675 INFO:tasks.rados.rados.0.smithi107.stdout:6611: done (1 left) 2020-06-12T02:38:29.676 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:38:29.676 INFO:tasks.rados.rados.0.smithi107.stdout:6612: seq_num 6611 ranges {0=327680} 2020-06-12T02:38:29.677 INFO:tasks.rados.rados.0.smithi107.stdout:6612: writing smithi10729407-20 from 0 to 327680 tid 1 2020-06-12T02:38:29.678 INFO:tasks.rados.rados.0.smithi107.stdout:6613: delete oid 9 current snap is 0 2020-06-12T02:38:29.680 INFO:tasks.rados.rados.0.smithi107.stdout:6613: done (2 left) 2020-06-12T02:38:29.680 INFO:tasks.rados.rados.0.smithi107.stdout:6614: delete oid 12 current snap is 0 2020-06-12T02:38:29.682 INFO:tasks.rados.rados.0.smithi107.stdout:6612: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:29.683 INFO:tasks.rados.rados.0.smithi107.stdout:6614: done (2 left) 2020-06-12T02:38:29.683 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:29.683 INFO:tasks.rados.rados.0.smithi107.stdout:6615: seq_num 6614 ranges {139264=376832} 2020-06-12T02:38:29.686 INFO:tasks.rados.rados.0.smithi107.stdout:6615: writing smithi10729407-15 from 139264 to 516096 tid 1 2020-06-12T02:38:29.687 INFO:tasks.rados.rados.0.smithi107.stdout:6616: delete oid 9 current snap is 0 2020-06-12T02:38:29.687 INFO:tasks.rados.rados.0.smithi107.stdout:6612: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:29.687 INFO:tasks.rados.rados.0.smithi107.stdout:6612: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:29.688 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 498 (ObjNum 6611 snap 0 seq_num 6611) dirty exists 2020-06-12T02:38:29.688 INFO:tasks.rados.rados.0.smithi107.stdout:6612: left oid 20 (ObjNum 6611 snap 0 seq_num 6611) 2020-06-12T02:38:29.689 INFO:tasks.rados.rados.0.smithi107.stdout:6612: done (3 left) 2020-06-12T02:38:29.689 INFO:tasks.rados.rados.0.smithi107.stdout:6616: done (2 left) 2020-06-12T02:38:29.689 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:29.689 INFO:tasks.rados.rados.0.smithi107.stdout:6617: seq_num 6616 ranges {188416=409600} 2020-06-12T02:38:29.693 INFO:tasks.rados.rados.0.smithi107.stdout:6617: writing smithi10729407-19 from 188416 to 598016 tid 1 2020-06-12T02:38:29.693 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:29.693 INFO:tasks.rados.rados.0.smithi107.stdout:6618: seq_num 6617 ranges {0=425984} 2020-06-12T02:38:29.696 INFO:tasks.rados.rados.0.smithi107.stdout:6618: writing smithi10729407-8 from 0 to 425984 tid 1 2020-06-12T02:38:29.696 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:29.697 INFO:tasks.rados.rados.0.smithi107.stdout:6619: seq_num 6618 ranges {245760=442368} 2020-06-12T02:38:29.701 INFO:tasks.rados.rados.0.smithi107.stdout:6619: writing smithi10729407-7 from 245760 to 688128 tid 1 2020-06-12T02:38:29.701 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:29.702 INFO:tasks.rados.rados.0.smithi107.stdout:6620: seq_num 6619 ranges {0=458752} 2020-06-12T02:38:29.707 INFO:tasks.rados.rados.0.smithi107.stdout:6620: writing smithi10729407-6 from 0 to 458752 tid 1 2020-06-12T02:38:29.707 INFO:tasks.rados.rados.0.smithi107.stdout:6621: delete oid 4 current snap is 0 2020-06-12T02:38:29.707 INFO:tasks.rados.rados.0.smithi107.stdout:6615: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:29.708 INFO:tasks.rados.rados.0.smithi107.stdout:6615: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:29.708 INFO:tasks.rados.rados.0.smithi107.stdout:6615: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:29.708 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1232 (ObjNum 6614 snap 0 seq_num 6614) dirty exists 2020-06-12T02:38:29.708 INFO:tasks.rados.rados.0.smithi107.stdout:6615: left oid 15 (ObjNum 6614 snap 0 seq_num 6614) 2020-06-12T02:38:29.708 INFO:tasks.rados.rados.0.smithi107.stdout:6617: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:29.708 INFO:tasks.rados.rados.0.smithi107.stdout:6617: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:29.709 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:29.705+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:29.710 INFO:tasks.rados.rados.0.smithi107.stdout:6617: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:29.710 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1214 (ObjNum 6616 snap 0 seq_num 6616) dirty exists 2020-06-12T02:38:29.710 INFO:tasks.rados.rados.0.smithi107.stdout:6617: left oid 19 (ObjNum 6616 snap 0 seq_num 6616) 2020-06-12T02:38:29.711 INFO:tasks.rados.rados.0.smithi107.stdout:6618: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:29.712 INFO:tasks.rados.rados.0.smithi107.stdout:6619: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:29.712 INFO:tasks.rados.rados.0.smithi107.stdout:6618: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:29.712 INFO:tasks.rados.rados.0.smithi107.stdout:6619: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:29.713 INFO:tasks.rados.rados.0.smithi107.stdout:6615: done (6 left) 2020-06-12T02:38:29.713 INFO:tasks.rados.rados.0.smithi107.stdout:6617: done (5 left) 2020-06-12T02:38:29.713 INFO:tasks.rados.rados.0.smithi107.stdout:6621: done (4 left) 2020-06-12T02:38:29.714 INFO:tasks.rados.rados.0.smithi107.stdout:6622: delete oid 4 current snap is 0 2020-06-12T02:38:29.715 INFO:tasks.rados.rados.0.smithi107.stdout:6618: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:29.715 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1234 (ObjNum 6617 snap 0 seq_num 6617) dirty exists 2020-06-12T02:38:29.715 INFO:tasks.rados.rados.0.smithi107.stdout:6618: left oid 8 (ObjNum 6617 snap 0 seq_num 6617) 2020-06-12T02:38:29.715 INFO:tasks.rados.rados.0.smithi107.stdout:6620: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:29.715 INFO:tasks.rados.rados.0.smithi107.stdout:6619: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:29.716 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1243 (ObjNum 6618 snap 0 seq_num 6618) dirty exists 2020-06-12T02:38:29.716 INFO:tasks.rados.rados.0.smithi107.stdout:6619: left oid 7 (ObjNum 6618 snap 0 seq_num 6618) 2020-06-12T02:38:29.717 INFO:tasks.rados.rados.0.smithi107.stdout:6618: done (4 left) 2020-06-12T02:38:29.717 INFO:tasks.rados.rados.0.smithi107.stdout:6619: done (3 left) 2020-06-12T02:38:29.717 INFO:tasks.rados.rados.0.smithi107.stdout:6622: done (2 left) 2020-06-12T02:38:29.717 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:29.718 INFO:tasks.rados.rados.0.smithi107.stdout:6623: seq_num 6622 ranges {4423680=507904} 2020-06-12T02:38:29.749 INFO:tasks.rados.rados.0.smithi107.stdout:6623: writing smithi10729407-3 from 4423680 to 4931584 tid 1 2020-06-12T02:38:29.762 INFO:tasks.rados.rados.0.smithi107.stdout:6624: delete oid 1 current snap is 0 2020-06-12T02:38:29.762 INFO:tasks.rados.rados.0.smithi107.stdout:6620: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:29.763 INFO:tasks.rados.rados.0.smithi107.stdout:6620: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:29.763 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1205 (ObjNum 6619 snap 0 seq_num 6619) dirty exists 2020-06-12T02:38:29.763 INFO:tasks.rados.rados.0.smithi107.stdout:6620: left oid 6 (ObjNum 6619 snap 0 seq_num 6619) 2020-06-12T02:38:29.763 INFO:tasks.rados.rados.0.smithi107.stdout:6620: done (3 left) 2020-06-12T02:38:29.763 INFO:tasks.rados.rados.0.smithi107.stdout:6624: done (2 left) 2020-06-12T02:38:29.764 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:38:29.764 INFO:tasks.rados.rados.0.smithi107.stdout:6625: seq_num 6624 ranges {0=548864} 2020-06-12T02:38:29.765 INFO:tasks.rados.rados.0.smithi107.stdout:6625: writing smithi10729407-11 from 0 to 548864 tid 1 2020-06-12T02:38:29.766 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:29.766 INFO:tasks.rados.rados.0.smithi107.stdout:6626: seq_num 6625 ranges {0=565248} 2020-06-12T02:38:29.766 INFO:tasks.rados.rados.0.smithi107.stdout:6626: writing smithi10729407-17 from 0 to 565248 tid 1 2020-06-12T02:38:29.766 INFO:tasks.rados.rados.0.smithi107.stdout:6627: delete oid 14 current snap is 0 2020-06-12T02:38:29.766 INFO:tasks.rados.rados.0.smithi107.stdout:6623: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:29.767 INFO:tasks.rados.rados.0.smithi107.stdout:6623: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:29.767 INFO:tasks.rados.rados.0.smithi107.stdout:6623: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:29.767 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1181 (ObjNum 6622 snap 0 seq_num 6622) dirty exists 2020-06-12T02:38:29.767 INFO:tasks.rados.rados.0.smithi107.stdout:6623: left oid 3 (ObjNum 6622 snap 0 seq_num 6622) 2020-06-12T02:38:29.767 INFO:tasks.rados.rados.0.smithi107.stdout:6625: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:29.767 INFO:tasks.rados.rados.0.smithi107.stdout:6623: done (4 left) 2020-06-12T02:38:29.768 INFO:tasks.rados.rados.0.smithi107.stdout:6627: done (3 left) 2020-06-12T02:38:29.768 INFO:tasks.rados.rados.0.smithi107.stdout:6628: delete oid 8 current snap is 0 2020-06-12T02:38:29.768 INFO:tasks.rados.rados.0.smithi107.stdout:6626: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:29.768 INFO:tasks.rados.rados.0.smithi107.stdout:6626: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:29.768 INFO:tasks.rados.rados.0.smithi107.stdout:6628: done (3 left) 2020-06-12T02:38:29.769 INFO:tasks.rados.rados.0.smithi107.stdout:6629: delete oid 7 current snap is 0 2020-06-12T02:38:29.771 INFO:tasks.rados.rados.0.smithi107.stdout:6625: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:29.771 INFO:tasks.rados.rados.0.smithi107.stdout:6625: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:29.772 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1176 (ObjNum 6624 snap 0 seq_num 6624) dirty exists 2020-06-12T02:38:29.772 INFO:tasks.rados.rados.0.smithi107.stdout:6625: left oid 11 (ObjNum 6624 snap 0 seq_num 6624) 2020-06-12T02:38:29.772 INFO:tasks.rados.rados.0.smithi107.stdout:6626: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:29.772 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1251 (ObjNum 6625 snap 0 seq_num 6625) dirty exists 2020-06-12T02:38:29.772 INFO:tasks.rados.rados.0.smithi107.stdout:6626: left oid 17 (ObjNum 6625 snap 0 seq_num 6625) 2020-06-12T02:38:29.773 INFO:tasks.rados.rados.0.smithi107.stdout:6625: done (3 left) 2020-06-12T02:38:29.773 INFO:tasks.rados.rados.0.smithi107.stdout:6626: done (2 left) 2020-06-12T02:38:29.773 INFO:tasks.rados.rados.0.smithi107.stdout:6629: done (1 left) 2020-06-12T02:38:29.773 INFO:tasks.rados.rados.0.smithi107.stdout:6630: delete oid 8 current snap is 0 2020-06-12T02:38:29.774 INFO:tasks.rados.rados.0.smithi107.stdout:6630: done (1 left) 2020-06-12T02:38:29.774 INFO:tasks.rados.rados.0.smithi107.stdout:6631: delete oid 13 current snap is 0 2020-06-12T02:38:29.783 INFO:tasks.rados.rados.0.smithi107.stdout:6631: done (1 left) 2020-06-12T02:38:29.783 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:29.783 INFO:tasks.rados.rados.0.smithi107.stdout:6632: seq_num 6631 ranges {0=663552} 2020-06-12T02:38:29.783 INFO:tasks.rados.rados.0.smithi107.stdout:6632: writing smithi10729407-4 from 0 to 663552 tid 1 2020-06-12T02:38:29.783 INFO:tasks.rados.rados.0.smithi107.stdout:6633: delete oid 7 current snap is 0 2020-06-12T02:38:29.784 INFO:tasks.rados.rados.0.smithi107.stdout:6633: done (2 left) 2020-06-12T02:38:29.784 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:29.784 INFO:tasks.rados.rados.0.smithi107.stdout:6634: seq_num 6633 ranges {0=696320} 2020-06-12T02:38:29.791 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:38:29.792 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:29.792 INFO:teuthology.orchestra.run.smithi107.stdout: "epoch": 58, 2020-06-12T02:38:29.792 INFO:teuthology.orchestra.run.smithi107.stdout: "fsid": "2f5030a1-3206-43c9-8471-d10429688d36", 2020-06-12T02:38:29.792 INFO:teuthology.orchestra.run.smithi107.stdout: "created": "2020-06-12T02:37:00.578493+0000", 2020-06-12T02:38:29.793 INFO:teuthology.orchestra.run.smithi107.stdout: "modified": "2020-06-12T02:38:27.403966+0000", 2020-06-12T02:38:29.793 INFO:teuthology.orchestra.run.smithi107.stdout: "last_up_change": "2020-06-12T02:37:53.542920+0000", 2020-06-12T02:38:29.793 INFO:teuthology.orchestra.run.smithi107.stdout: "last_in_change": "2020-06-12T02:37:10.705891+0000", 2020-06-12T02:38:29.793 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-06-12T02:38:29.793 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_num": 5806080, 2020-06-12T02:38:29.794 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_set": [ 2020-06-12T02:38:29.794 INFO:teuthology.orchestra.run.smithi107.stdout: "nodeep-scrub", 2020-06-12T02:38:29.794 INFO:teuthology.orchestra.run.smithi107.stdout: "noscrub", 2020-06-12T02:38:29.794 INFO:teuthology.orchestra.run.smithi107.stdout: "pglog_hardlimit", 2020-06-12T02:38:29.794 INFO:teuthology.orchestra.run.smithi107.stdout: "purged_snapdirs", 2020-06-12T02:38:29.795 INFO:teuthology.orchestra.run.smithi107.stdout: "recovery_deletes", 2020-06-12T02:38:29.796 INFO:teuthology.orchestra.run.smithi107.stdout: "sortbitwise" 2020-06-12T02:38:29.796 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2020-06-12T02:38:29.796 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_version": 5, 2020-06-12T02:38:29.796 INFO:teuthology.orchestra.run.smithi107.stdout: "full_ratio": 0.94999998807907104, 2020-06-12T02:38:29.796 INFO:teuthology.orchestra.run.smithi107.stdout: "backfillfull_ratio": 0.89999997615814209, 2020-06-12T02:38:29.797 INFO:teuthology.orchestra.run.smithi107.stdout: "nearfull_ratio": 0.85000002384185791, 2020-06-12T02:38:29.797 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_snapshot": "", 2020-06-12T02:38:29.797 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_max": 3, 2020-06-12T02:38:29.797 INFO:teuthology.orchestra.run.smithi107.stdout: "max_osd": 8, 2020-06-12T02:38:29.798 INFO:teuthology.orchestra.run.smithi107.stdout: "require_min_compat_client": "jewel", 2020-06-12T02:38:29.799 INFO:teuthology.orchestra.run.smithi107.stdout: "min_compat_client": "jewel", 2020-06-12T02:38:29.799 INFO:teuthology.orchestra.run.smithi107.stdout: "require_osd_release": "octopus", 2020-06-12T02:38:29.799 INFO:teuthology.orchestra.run.smithi107.stdout: "pools": [ 2020-06-12T02:38:29.800 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.800 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 1, 2020-06-12T02:38:29.800 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "device_health_metrics", 2020-06-12T02:38:29.800 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2020-06-12T02:37:03.610773+0000", 2020-06-12T02:38:29.800 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2020-06-12T02:38:29.801 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2020-06-12T02:38:29.801 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2020-06-12T02:38:29.801 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 2, 2020-06-12T02:38:29.801 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 1, 2020-06-12T02:38:29.801 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2020-06-12T02:38:29.802 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2020-06-12T02:38:29.802 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2020-06-12T02:38:29.802 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 1, 2020-06-12T02:38:29.802 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 1, 2020-06-12T02:38:29.802 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 1, 2020-06-12T02:38:29.802 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 1, 2020-06-12T02:38:29.803 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 1, 2020-06-12T02:38:29.803 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2020-06-12T02:38:29.803 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2020-06-12T02:38:29.803 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2020-06-12T02:38:29.803 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2020-06-12T02:38:29.804 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2020-06-12T02:38:29.804 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2020-06-12T02:38:29.804 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2020-06-12T02:38:29.804 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.804 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "57", 2020-06-12T02:38:29.805 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2020-06-12T02:38:29.805 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2020-06-12T02:38:29.805 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2020-06-12T02:38:29.805 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2020-06-12T02:38:29.805 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2020-06-12T02:38:29.806 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2020-06-12T02:38:29.806 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2020-06-12T02:38:29.806 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2020-06-12T02:38:29.806 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2020-06-12T02:38:29.806 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2020-06-12T02:38:29.807 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2020-06-12T02:38:29.807 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2020-06-12T02:38:29.807 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2020-06-12T02:38:29.807 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2020-06-12T02:38:29.807 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2020-06-12T02:38:29.807 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2020-06-12T02:38:29.808 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2020-06-12T02:38:29.808 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2020-06-12T02:38:29.808 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-06-12T02:38:29.808 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-06-12T02:38:29.808 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2020-06-12T02:38:29.809 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2020-06-12T02:38:29.809 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2020-06-12T02:38:29.809 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2020-06-12T02:38:29.809 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2020-06-12T02:38:29.810 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2020-06-12T02:38:29.810 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.810 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2020-06-12T02:38:29.810 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2020-06-12T02:38:29.810 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2020-06-12T02:38:29.811 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2020-06-12T02:38:29.811 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2020-06-12T02:38:29.811 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2020-06-12T02:38:29.811 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2020-06-12T02:38:29.811 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2020-06-12T02:38:29.812 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2020-06-12T02:38:29.812 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2020-06-12T02:38:29.812 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2020-06-12T02:38:29.812 INFO:teuthology.orchestra.run.smithi107.stdout: "options": { 2020-06-12T02:38:29.812 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_min": 1 2020-06-12T02:38:29.813 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.813 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2020-06-12T02:38:29.813 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_devicehealth": {} 2020-06-12T02:38:29.813 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:29.813 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.814 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.814 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 2, 2020-06-12T02:38:29.814 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "rbd", 2020-06-12T02:38:29.816 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2020-06-12T02:37:14.831188+0000", 2020-06-12T02:38:29.816 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2020-06-12T02:38:29.816 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2020-06-12T02:38:29.817 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 1, 2020-06-12T02:38:29.817 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 2, 2020-06-12T02:38:29.817 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 1, 2020-06-12T02:38:29.817 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 0, 2020-06-12T02:38:29.818 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2020-06-12T02:38:29.818 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2020-06-12T02:38:29.818 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 8, 2020-06-12T02:38:29.818 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 8, 2020-06-12T02:38:29.818 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 8, 2020-06-12T02:38:29.818 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 8, 2020-06-12T02:38:29.819 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 8, 2020-06-12T02:38:29.819 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2020-06-12T02:38:29.819 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2020-06-12T02:38:29.819 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2020-06-12T02:38:29.820 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2020-06-12T02:38:29.820 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2020-06-12T02:38:29.820 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2020-06-12T02:38:29.820 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2020-06-12T02:38:29.820 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.821 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "18", 2020-06-12T02:38:29.821 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2020-06-12T02:38:29.821 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2020-06-12T02:38:29.821 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2020-06-12T02:38:29.821 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2020-06-12T02:38:29.822 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2020-06-12T02:38:29.822 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2020-06-12T02:38:29.822 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2020-06-12T02:38:29.822 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2020-06-12T02:38:29.822 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2020-06-12T02:38:29.822 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2020-06-12T02:38:29.823 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2020-06-12T02:38:29.823 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2020-06-12T02:38:29.823 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2020-06-12T02:38:29.823 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2020-06-12T02:38:29.823 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2020-06-12T02:38:29.824 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2020-06-12T02:38:29.824 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2020-06-12T02:38:29.824 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2020-06-12T02:38:29.824 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-06-12T02:38:29.824 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-06-12T02:38:29.824 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2020-06-12T02:38:29.825 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2020-06-12T02:38:29.825 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2020-06-12T02:38:29.825 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "", 2020-06-12T02:38:29.825 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2020-06-12T02:38:29.825 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2020-06-12T02:38:29.826 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.826 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2020-06-12T02:38:29.826 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2020-06-12T02:38:29.826 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2020-06-12T02:38:29.826 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2020-06-12T02:38:29.826 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2020-06-12T02:38:29.827 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2020-06-12T02:38:29.827 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2020-06-12T02:38:29.827 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2020-06-12T02:38:29.827 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 0, 2020-06-12T02:38:29.827 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2020-06-12T02:38:29.828 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": false, 2020-06-12T02:38:29.828 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2020-06-12T02:38:29.828 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2020-06-12T02:38:29.828 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd": {} 2020-06-12T02:38:29.828 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:29.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.829 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.829 INFO:teuthology.orchestra.run.smithi107.stdout: "pool": 3, 2020-06-12T02:38:29.830 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": "unique_pool_0", 2020-06-12T02:38:29.830 INFO:teuthology.orchestra.run.smithi107.stdout: "create_time": "2020-06-12T02:37:25.893619+0000", 2020-06-12T02:38:29.830 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2020-06-12T02:38:29.830 INFO:teuthology.orchestra.run.smithi107.stdout: "flags_names": "hashpspool", 2020-06-12T02:38:29.830 INFO:teuthology.orchestra.run.smithi107.stdout: "type": 3, 2020-06-12T02:38:29.831 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 3, 2020-06-12T02:38:29.831 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": 2, 2020-06-12T02:38:29.831 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_rule": 1, 2020-06-12T02:38:29.831 INFO:teuthology.orchestra.run.smithi107.stdout: "object_hash": 2, 2020-06-12T02:38:29.831 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscale_mode": "off", 2020-06-12T02:38:29.832 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": 16, 2020-06-12T02:38:29.832 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num": 16, 2020-06-12T02:38:29.832 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_placement_num_target": 16, 2020-06-12T02:38:29.832 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_target": 16, 2020-06-12T02:38:29.832 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num_pending": 16, 2020-06-12T02:38:29.832 INFO:teuthology.orchestra.run.smithi107.stdout: "last_pg_merge_meta": { 2020-06-12T02:38:29.833 INFO:teuthology.orchestra.run.smithi107.stdout: "source_pgid": "0.0", 2020-06-12T02:38:29.833 INFO:teuthology.orchestra.run.smithi107.stdout: "ready_epoch": 0, 2020-06-12T02:38:29.833 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_started": 0, 2020-06-12T02:38:29.833 INFO:teuthology.orchestra.run.smithi107.stdout: "last_epoch_clean": 0, 2020-06-12T02:38:29.833 INFO:teuthology.orchestra.run.smithi107.stdout: "source_version": "0'0", 2020-06-12T02:38:29.834 INFO:teuthology.orchestra.run.smithi107.stdout: "target_version": "0'0" 2020-06-12T02:38:29.834 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.834 INFO:teuthology.orchestra.run.smithi107.stdout: "last_change": "24", 2020-06-12T02:38:29.834 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend": "0", 2020-06-12T02:38:29.834 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_prenautilus": "0", 2020-06-12T02:38:29.834 INFO:teuthology.orchestra.run.smithi107.stdout: "last_force_op_resend_preluminous": "0", 2020-06-12T02:38:29.835 INFO:teuthology.orchestra.run.smithi107.stdout: "auid": 0, 2020-06-12T02:38:29.835 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_mode": "selfmanaged", 2020-06-12T02:38:29.835 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_seq": 0, 2020-06-12T02:38:29.835 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_epoch": 0, 2020-06-12T02:38:29.835 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_snaps": [], 2020-06-12T02:38:29.836 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps": "[]", 2020-06-12T02:38:29.836 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_bytes": 0, 2020-06-12T02:38:29.836 INFO:teuthology.orchestra.run.smithi107.stdout: "quota_max_objects": 0, 2020-06-12T02:38:29.836 INFO:teuthology.orchestra.run.smithi107.stdout: "tiers": [], 2020-06-12T02:38:29.836 INFO:teuthology.orchestra.run.smithi107.stdout: "tier_of": -1, 2020-06-12T02:38:29.836 INFO:teuthology.orchestra.run.smithi107.stdout: "read_tier": -1, 2020-06-12T02:38:29.837 INFO:teuthology.orchestra.run.smithi107.stdout: "write_tier": -1, 2020-06-12T02:38:29.837 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_mode": "none", 2020-06-12T02:38:29.837 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_bytes": 0, 2020-06-12T02:38:29.837 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_objects": 0, 2020-06-12T02:38:29.837 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-06-12T02:38:29.838 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-06-12T02:38:29.838 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_target_full_ratio_micro": 800000, 2020-06-12T02:38:29.838 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_flush_age": 0, 2020-06-12T02:38:29.838 INFO:teuthology.orchestra.run.smithi107.stdout: "cache_min_evict_age": 0, 2020-06-12T02:38:29.838 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profile": "teuthologyprofile", 2020-06-12T02:38:29.838 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_params": { 2020-06-12T02:38:29.839 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "none" 2020-06-12T02:38:29.839 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.839 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_period": 0, 2020-06-12T02:38:29.839 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_count": 0, 2020-06-12T02:38:29.839 INFO:teuthology.orchestra.run.smithi107.stdout: "use_gmt_hitset": true, 2020-06-12T02:38:29.840 INFO:teuthology.orchestra.run.smithi107.stdout: "min_read_recency_for_promote": 0, 2020-06-12T02:38:29.840 INFO:teuthology.orchestra.run.smithi107.stdout: "min_write_recency_for_promote": 0, 2020-06-12T02:38:29.840 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_grade_decay_rate": 0, 2020-06-12T02:38:29.840 INFO:teuthology.orchestra.run.smithi107.stdout: "hit_set_search_last_n": 0, 2020-06-12T02:38:29.841 INFO:teuthology.orchestra.run.smithi107.stdout: "grade_table": [], 2020-06-12T02:38:29.841 INFO:teuthology.orchestra.run.smithi107.stdout: "stripe_width": 8192, 2020-06-12T02:38:29.841 INFO:teuthology.orchestra.run.smithi107.stdout: "expected_num_objects": 0, 2020-06-12T02:38:29.841 INFO:teuthology.orchestra.run.smithi107.stdout: "fast_read": true, 2020-06-12T02:38:29.841 INFO:teuthology.orchestra.run.smithi107.stdout: "options": {}, 2020-06-12T02:38:29.842 INFO:teuthology.orchestra.run.smithi107.stdout: "application_metadata": { 2020-06-12T02:38:29.842 INFO:teuthology.orchestra.run.smithi107.stdout: "rados": {} 2020-06-12T02:38:29.842 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:29.842 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:29.842 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2020-06-12T02:38:29.842 INFO:teuthology.orchestra.run.smithi107.stdout: "osds": [ 2020-06-12T02:38:29.843 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.843 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 0, 2020-06-12T02:38:29.843 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "ad4b4814-83e7-4a2e-9b1f-d10406174f7e", 2020-06-12T02:38:29.843 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2020-06-12T02:38:29.843 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2020-06-12T02:38:29.844 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2020-06-12T02:38:29.844 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2020-06-12T02:38:29.844 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2020-06-12T02:38:29.844 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2020-06-12T02:38:29.844 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 13, 2020-06-12T02:38:29.844 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 53, 2020-06-12T02:38:29.845 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2020-06-12T02:38:29.845 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2020-06-12T02:38:29.845 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2020-06-12T02:38:29.845 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:29.846 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.846 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:29.846 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6816", 2020-06-12T02:38:29.846 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26614 2020-06-12T02:38:29.846 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.846 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.847 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:29.847 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6817", 2020-06-12T02:38:29.847 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26614 2020-06-12T02:38:29.847 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:29.847 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:29.848 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.848 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2020-06-12T02:38:29.848 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:29.848 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.849 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:29.850 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6819", 2020-06-12T02:38:29.850 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26614 2020-06-12T02:38:29.850 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.850 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.851 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:29.851 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6821", 2020-06-12T02:38:29.851 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26614 2020-06-12T02:38:29.851 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:29.851 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:29.852 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.852 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2020-06-12T02:38:29.852 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:29.853 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.853 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:29.853 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6827", 2020-06-12T02:38:29.853 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26614 2020-06-12T02:38:29.854 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.854 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.855 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:29.855 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6829", 2020-06-12T02:38:29.856 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26614 2020-06-12T02:38:29.856 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:29.856 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:29.857 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.857 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2020-06-12T02:38:29.857 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:29.857 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.857 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:29.858 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6823", 2020-06-12T02:38:29.858 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26614 2020-06-12T02:38:29.858 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.858 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.858 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:29.858 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6825", 2020-06-12T02:38:29.859 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26614 2020-06-12T02:38:29.859 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:29.859 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:29.859 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.859 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.107:6817/26614", 2020-06-12T02:38:29.860 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.107:6821/26614", 2020-06-12T02:38:29.860 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.107:6829/26614", 2020-06-12T02:38:29.860 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.107:6825/26614", 2020-06-12T02:38:29.860 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2020-06-12T02:38:29.861 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2020-06-12T02:38:29.861 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2020-06-12T02:38:29.861 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:29.861 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.861 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.862 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 1, 2020-06-12T02:38:29.862 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "046adf0d-b9e9-41ad-a7c9-3841f8b8f790", 2020-06-12T02:38:29.862 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2020-06-12T02:38:29.862 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2020-06-12T02:38:29.862 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 0.1898193359375, 2020-06-12T02:38:29.863 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2020-06-12T02:38:29.863 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2020-06-12T02:38:29.863 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2020-06-12T02:38:29.863 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 13, 2020-06-12T02:38:29.863 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 29, 2020-06-12T02:38:29.864 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2020-06-12T02:38:29.864 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2020-06-12T02:38:29.864 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2020-06-12T02:38:29.864 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:29.864 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.864 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:29.865 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6800", 2020-06-12T02:38:29.865 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26611 2020-06-12T02:38:29.865 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.865 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.865 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:29.866 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6801", 2020-06-12T02:38:29.866 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26611 2020-06-12T02:38:29.866 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:29.866 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:29.867 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.867 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2020-06-12T02:38:29.867 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:29.867 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.867 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:29.868 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6802", 2020-06-12T02:38:29.868 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26611 2020-06-12T02:38:29.868 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.868 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.868 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:29.868 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6803", 2020-06-12T02:38:29.869 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26611 2020-06-12T02:38:29.869 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:29.869 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:29.869 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.869 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2020-06-12T02:38:29.870 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:29.870 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.870 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:29.870 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6806", 2020-06-12T02:38:29.870 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26611 2020-06-12T02:38:29.871 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.871 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.871 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:29.872 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6807", 2020-06-12T02:38:29.872 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26611 2020-06-12T02:38:29.872 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:29.872 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:29.872 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.872 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2020-06-12T02:38:29.873 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:29.873 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.873 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:29.873 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6804", 2020-06-12T02:38:29.873 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26611 2020-06-12T02:38:29.874 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.874 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.874 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:29.874 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6805", 2020-06-12T02:38:29.874 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26611 2020-06-12T02:38:29.874 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:29.875 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:29.875 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.875 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.107:6801/26611", 2020-06-12T02:38:29.875 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.107:6803/26611", 2020-06-12T02:38:29.875 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.107:6807/26611", 2020-06-12T02:38:29.876 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.107:6805/26611", 2020-06-12T02:38:29.876 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2020-06-12T02:38:29.876 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2020-06-12T02:38:29.876 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2020-06-12T02:38:29.877 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:29.877 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.877 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.877 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 2, 2020-06-12T02:38:29.878 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "e15da6b7-dc79-4185-a5f9-0f443d8b52b6", 2020-06-12T02:38:29.878 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2020-06-12T02:38:29.878 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2020-06-12T02:38:29.878 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2020-06-12T02:38:29.878 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2020-06-12T02:38:29.879 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2020-06-12T02:38:29.879 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2020-06-12T02:38:29.879 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 13, 2020-06-12T02:38:29.879 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 52, 2020-06-12T02:38:29.879 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2020-06-12T02:38:29.879 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2020-06-12T02:38:29.880 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2020-06-12T02:38:29.880 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:29.880 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.880 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:29.880 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6818", 2020-06-12T02:38:29.881 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26613 2020-06-12T02:38:29.881 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.882 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.882 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:29.883 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6820", 2020-06-12T02:38:29.883 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26613 2020-06-12T02:38:29.883 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:29.883 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:29.883 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.884 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2020-06-12T02:38:29.884 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:29.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.884 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:29.884 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6822", 2020-06-12T02:38:29.884 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26613 2020-06-12T02:38:29.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.885 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.885 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:29.885 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6824", 2020-06-12T02:38:29.885 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26613 2020-06-12T02:38:29.886 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:29.886 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:29.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.886 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2020-06-12T02:38:29.886 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:29.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.887 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:29.887 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6830", 2020-06-12T02:38:29.887 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26613 2020-06-12T02:38:29.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.888 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:29.888 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6831", 2020-06-12T02:38:29.888 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26613 2020-06-12T02:38:29.888 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:29.889 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:29.889 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.892 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2020-06-12T02:38:29.892 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:29.892 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.892 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:29.892 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6826", 2020-06-12T02:38:29.893 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26613 2020-06-12T02:38:29.893 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.893 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.893 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:29.893 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6828", 2020-06-12T02:38:29.894 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26613 2020-06-12T02:38:29.894 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:29.894 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:29.894 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.894 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.107:6820/26613", 2020-06-12T02:38:29.895 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.107:6824/26613", 2020-06-12T02:38:29.895 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.107:6831/26613", 2020-06-12T02:38:29.895 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.107:6828/26613", 2020-06-12T02:38:29.895 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2020-06-12T02:38:29.895 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2020-06-12T02:38:29.895 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2020-06-12T02:38:29.896 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:29.896 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.896 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.896 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 3, 2020-06-12T02:38:29.896 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "843e0dd3-5ce4-46b9-9004-69849048d2bd", 2020-06-12T02:38:29.897 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2020-06-12T02:38:29.897 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2020-06-12T02:38:29.897 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2020-06-12T02:38:29.897 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2020-06-12T02:38:29.897 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2020-06-12T02:38:29.898 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2020-06-12T02:38:29.898 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 13, 2020-06-12T02:38:29.898 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 53, 2020-06-12T02:38:29.898 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2020-06-12T02:38:29.898 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2020-06-12T02:38:29.898 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2020-06-12T02:38:29.899 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:29.899 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.899 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:29.899 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6808", 2020-06-12T02:38:29.899 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26612 2020-06-12T02:38:29.900 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.900 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.900 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:29.900 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6809", 2020-06-12T02:38:29.900 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26612 2020-06-12T02:38:29.900 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:29.901 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:29.901 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.901 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2020-06-12T02:38:29.901 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:29.902 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.902 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:29.902 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6810", 2020-06-12T02:38:29.902 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26612 2020-06-12T02:38:29.903 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.903 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.903 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:29.903 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6811", 2020-06-12T02:38:29.903 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26612 2020-06-12T02:38:29.904 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:29.904 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:29.904 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.904 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2020-06-12T02:38:29.904 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:29.904 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.905 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:29.905 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6814", 2020-06-12T02:38:29.905 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26612 2020-06-12T02:38:29.905 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.905 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.906 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:29.906 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6815", 2020-06-12T02:38:29.906 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26612 2020-06-12T02:38:29.906 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:29.906 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:29.907 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.907 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2020-06-12T02:38:29.907 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:29.907 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.907 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:29.907 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6812", 2020-06-12T02:38:29.908 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26612 2020-06-12T02:38:29.908 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.908 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.908 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:29.908 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6813", 2020-06-12T02:38:29.909 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26612 2020-06-12T02:38:29.909 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:29.909 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:29.909 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.909 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.107:6809/26612", 2020-06-12T02:38:29.909 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.107:6811/26612", 2020-06-12T02:38:29.910 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.107:6815/26612", 2020-06-12T02:38:29.910 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.107:6813/26612", 2020-06-12T02:38:29.910 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2020-06-12T02:38:29.910 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2020-06-12T02:38:29.910 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2020-06-12T02:38:29.911 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:29.911 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.911 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.911 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 4, 2020-06-12T02:38:29.911 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "c066ab09-e031-4b49-9df3-d54cd9f9b580", 2020-06-12T02:38:29.911 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2020-06-12T02:38:29.912 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2020-06-12T02:38:29.912 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2020-06-12T02:38:29.912 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2020-06-12T02:38:29.913 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2020-06-12T02:38:29.913 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2020-06-12T02:38:29.913 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 13, 2020-06-12T02:38:29.913 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 53, 2020-06-12T02:38:29.913 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2020-06-12T02:38:29.914 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2020-06-12T02:38:29.914 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2020-06-12T02:38:29.914 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:29.914 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.914 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:29.914 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6818", 2020-06-12T02:38:29.915 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26685 2020-06-12T02:38:29.915 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.915 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.915 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:29.915 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6820", 2020-06-12T02:38:29.916 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26685 2020-06-12T02:38:29.916 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:29.916 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:29.916 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.916 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2020-06-12T02:38:29.916 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:29.917 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.917 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:29.917 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6822", 2020-06-12T02:38:29.917 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26685 2020-06-12T02:38:29.917 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.918 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.918 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:29.918 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6823", 2020-06-12T02:38:29.918 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26685 2020-06-12T02:38:29.918 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:29.918 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:29.919 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.919 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2020-06-12T02:38:29.919 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:29.919 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.919 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:29.920 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6830", 2020-06-12T02:38:29.920 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26685 2020-06-12T02:38:29.920 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.920 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.920 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:29.920 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6832", 2020-06-12T02:38:29.921 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26685 2020-06-12T02:38:29.921 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:29.921 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:29.921 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.921 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2020-06-12T02:38:29.922 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:29.922 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.922 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:29.922 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6825", 2020-06-12T02:38:29.923 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26685 2020-06-12T02:38:29.923 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.923 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.923 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:29.923 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6828", 2020-06-12T02:38:29.924 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26685 2020-06-12T02:38:29.924 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:29.924 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:29.924 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.924 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.188:6820/26685", 2020-06-12T02:38:29.924 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.188:6823/26685", 2020-06-12T02:38:29.925 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.188:6832/26685", 2020-06-12T02:38:29.925 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.188:6828/26685", 2020-06-12T02:38:29.925 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2020-06-12T02:38:29.925 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2020-06-12T02:38:29.925 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2020-06-12T02:38:29.926 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:29.926 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.926 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.926 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 5, 2020-06-12T02:38:29.926 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "56a24e45-1b1b-4a65-b4b4-57cfab296f65", 2020-06-12T02:38:29.926 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2020-06-12T02:38:29.927 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2020-06-12T02:38:29.927 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2020-06-12T02:38:29.927 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 0, 2020-06-12T02:38:29.927 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2020-06-12T02:38:29.927 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2020-06-12T02:38:29.928 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 13, 2020-06-12T02:38:29.928 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 36, 2020-06-12T02:38:29.928 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2020-06-12T02:38:29.928 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2020-06-12T02:38:29.928 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2020-06-12T02:38:29.928 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:29.929 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.929 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:29.929 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6819", 2020-06-12T02:38:29.929 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26684 2020-06-12T02:38:29.929 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.930 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.930 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:29.930 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6821", 2020-06-12T02:38:29.930 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26684 2020-06-12T02:38:29.930 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:29.930 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:29.931 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.931 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2020-06-12T02:38:29.931 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:29.931 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.931 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:29.932 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6824", 2020-06-12T02:38:29.932 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26684 2020-06-12T02:38:29.932 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.932 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.932 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:29.932 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6826", 2020-06-12T02:38:29.933 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26684 2020-06-12T02:38:29.933 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:29.933 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:29.934 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.934 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2020-06-12T02:38:29.934 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:29.934 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.934 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:29.934 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6831", 2020-06-12T02:38:29.935 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26684 2020-06-12T02:38:29.935 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.935 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.935 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:29.935 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6833", 2020-06-12T02:38:29.936 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26684 2020-06-12T02:38:29.936 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:29.936 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:29.936 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.936 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2020-06-12T02:38:29.936 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:29.937 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.937 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:29.937 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6827", 2020-06-12T02:38:29.937 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26684 2020-06-12T02:38:29.937 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.938 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.938 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:29.938 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6829", 2020-06-12T02:38:29.938 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26684 2020-06-12T02:38:29.938 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:29.938 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:29.939 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.939 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.188:6821/26684", 2020-06-12T02:38:29.939 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.188:6826/26684", 2020-06-12T02:38:29.939 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.188:6833/26684", 2020-06-12T02:38:29.939 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.188:6829/26684", 2020-06-12T02:38:29.939 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2020-06-12T02:38:29.940 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2020-06-12T02:38:29.940 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2020-06-12T02:38:29.940 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:29.940 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.940 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.941 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 6, 2020-06-12T02:38:29.941 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "f207a2a8-b3c7-4fd8-ae90-4c2d7a5c1c38", 2020-06-12T02:38:29.941 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2020-06-12T02:38:29.941 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2020-06-12T02:38:29.941 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2020-06-12T02:38:29.941 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2020-06-12T02:38:29.942 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 0, 2020-06-12T02:38:29.942 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 0, 2020-06-12T02:38:29.942 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 13, 2020-06-12T02:38:29.942 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 53, 2020-06-12T02:38:29.943 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 0, 2020-06-12T02:38:29.943 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2020-06-12T02:38:29.944 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2020-06-12T02:38:29.944 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:29.944 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.944 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:29.944 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6810", 2020-06-12T02:38:29.945 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26682 2020-06-12T02:38:29.945 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.945 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.945 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:29.945 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6811", 2020-06-12T02:38:29.945 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26682 2020-06-12T02:38:29.946 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:29.946 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:29.946 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.946 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2020-06-12T02:38:29.946 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:29.947 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.947 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:29.947 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6812", 2020-06-12T02:38:29.947 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26682 2020-06-12T02:38:29.947 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.948 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.949 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:29.949 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6813", 2020-06-12T02:38:29.949 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26682 2020-06-12T02:38:29.950 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:29.950 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:29.950 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.950 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2020-06-12T02:38:29.950 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:29.951 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.951 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:29.951 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6816", 2020-06-12T02:38:29.952 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26682 2020-06-12T02:38:29.952 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.952 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.953 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:29.953 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6817", 2020-06-12T02:38:29.953 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26682 2020-06-12T02:38:29.953 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:29.953 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:29.954 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.954 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2020-06-12T02:38:29.954 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:29.954 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.954 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:29.955 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6814", 2020-06-12T02:38:29.955 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26682 2020-06-12T02:38:29.955 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.955 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.956 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:29.956 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6815", 2020-06-12T02:38:29.957 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 26682 2020-06-12T02:38:29.957 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:29.957 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:29.957 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.958 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.188:6811/26682", 2020-06-12T02:38:29.958 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.188:6813/26682", 2020-06-12T02:38:29.958 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.188:6817/26682", 2020-06-12T02:38:29.958 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.188:6815/26682", 2020-06-12T02:38:29.958 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2020-06-12T02:38:29.959 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2020-06-12T02:38:29.960 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2020-06-12T02:38:29.961 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:29.961 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.962 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.962 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 7, 2020-06-12T02:38:29.962 INFO:teuthology.orchestra.run.smithi107.stdout: "uuid": "4e5cc44f-af17-47c9-973f-8e87d1bd98db", 2020-06-12T02:38:29.962 INFO:teuthology.orchestra.run.smithi107.stdout: "up": 1, 2020-06-12T02:38:29.962 INFO:teuthology.orchestra.run.smithi107.stdout: "in": 1, 2020-06-12T02:38:29.963 INFO:teuthology.orchestra.run.smithi107.stdout: "weight": 1, 2020-06-12T02:38:29.963 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_affinity": 1, 2020-06-12T02:38:29.963 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_begin": 13, 2020-06-12T02:38:29.963 INFO:teuthology.orchestra.run.smithi107.stdout: "last_clean_end": 18, 2020-06-12T02:38:29.963 INFO:teuthology.orchestra.run.smithi107.stdout: "up_from": 34, 2020-06-12T02:38:29.964 INFO:teuthology.orchestra.run.smithi107.stdout: "up_thru": 50, 2020-06-12T02:38:29.964 INFO:teuthology.orchestra.run.smithi107.stdout: "down_at": 19, 2020-06-12T02:38:29.964 INFO:teuthology.orchestra.run.smithi107.stdout: "lost_at": 0, 2020-06-12T02:38:29.965 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrs": { 2020-06-12T02:38:29.965 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:29.965 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.965 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:29.966 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6802", 2020-06-12T02:38:29.966 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 29046 2020-06-12T02:38:29.966 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.966 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.967 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:29.967 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6803", 2020-06-12T02:38:29.967 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 29046 2020-06-12T02:38:29.967 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:29.967 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:29.968 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.968 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addrs": { 2020-06-12T02:38:29.968 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:29.968 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.968 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:29.968 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6804", 2020-06-12T02:38:29.969 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 29046 2020-06-12T02:38:29.969 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.969 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.969 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:29.969 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6805", 2020-06-12T02:38:29.970 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 29046 2020-06-12T02:38:29.970 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:29.970 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:29.971 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.971 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addrs": { 2020-06-12T02:38:29.971 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:29.972 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.973 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:29.973 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6808", 2020-06-12T02:38:29.973 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 29046 2020-06-12T02:38:29.973 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.974 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.974 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:29.974 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6809", 2020-06-12T02:38:29.974 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 29046 2020-06-12T02:38:29.974 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:29.975 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:29.975 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.975 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addrs": { 2020-06-12T02:38:29.975 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2020-06-12T02:38:29.975 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.976 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2020-06-12T02:38:29.976 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6806", 2020-06-12T02:38:29.976 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 29046 2020-06-12T02:38:29.976 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.977 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.977 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2020-06-12T02:38:29.977 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.188:6807", 2020-06-12T02:38:29.977 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 29046 2020-06-12T02:38:29.977 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:29.977 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:29.978 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.978 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "172.21.15.188:6803/29046", 2020-06-12T02:38:29.978 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "172.21.15.188:6805/29046", 2020-06-12T02:38:29.978 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_back_addr": "172.21.15.188:6809/29046", 2020-06-12T02:38:29.978 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_front_addr": "172.21.15.188:6807/29046", 2020-06-12T02:38:29.979 INFO:teuthology.orchestra.run.smithi107.stdout: "state": [ 2020-06-12T02:38:29.979 INFO:teuthology.orchestra.run.smithi107.stdout: "exists", 2020-06-12T02:38:29.979 INFO:teuthology.orchestra.run.smithi107.stdout: "up" 2020-06-12T02:38:29.979 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:29.979 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:29.980 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2020-06-12T02:38:29.980 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_xinfo": [ 2020-06-12T02:38:29.980 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.980 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 0, 2020-06-12T02:38:29.980 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2020-06-12T02:38:29.980 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2020-06-12T02:38:29.981 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2020-06-12T02:38:29.981 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138292836696063, 2020-06-12T02:38:29.981 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2020-06-12T02:38:29.982 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2020-06-12T02:38:09.980294+0000", 2020-06-12T02:38:29.982 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2020-06-12T02:38:29.982 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.982 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.983 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 1, 2020-06-12T02:38:29.983 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2020-06-12T02:38:29.983 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2020-06-12T02:38:29.983 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2020-06-12T02:38:29.983 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138292836696063, 2020-06-12T02:38:29.983 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2020-06-12T02:38:29.984 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2020-06-12T02:38:09.373534+0000", 2020-06-12T02:38:29.984 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2020-06-12T02:38:29.984 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.984 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.984 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 2, 2020-06-12T02:38:29.985 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2020-06-12T02:38:29.985 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2020-06-12T02:38:29.985 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2020-06-12T02:38:29.985 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138292836696063, 2020-06-12T02:38:29.985 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2020-06-12T02:38:29.985 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2020-06-12T02:38:09.542448+0000", 2020-06-12T02:38:29.986 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2020-06-12T02:38:29.986 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.986 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.986 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 3, 2020-06-12T02:38:29.986 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2020-06-12T02:38:29.987 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2020-06-12T02:38:29.987 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2020-06-12T02:38:29.987 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138292836696063, 2020-06-12T02:38:29.987 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2020-06-12T02:38:29.988 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2020-06-12T02:38:09.528988+0000", 2020-06-12T02:38:29.988 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2020-06-12T02:38:29.988 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.988 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.989 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 4, 2020-06-12T02:38:29.989 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2020-06-12T02:38:29.989 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2020-06-12T02:38:29.989 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2020-06-12T02:38:29.989 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138292836696063, 2020-06-12T02:38:29.989 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2020-06-12T02:38:29.990 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2020-06-12T02:38:08.934675+0000", 2020-06-12T02:38:29.990 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2020-06-12T02:38:29.990 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.990 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.990 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 5, 2020-06-12T02:38:29.991 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2020-06-12T02:38:29.991 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2020-06-12T02:38:29.991 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2020-06-12T02:38:29.991 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138292836696063, 2020-06-12T02:38:29.991 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2020-06-12T02:38:29.991 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2020-06-12T02:38:09.183394+0000", 2020-06-12T02:38:29.992 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2020-06-12T02:38:29.992 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.992 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.992 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 6, 2020-06-12T02:38:29.993 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2020-06-12T02:38:29.993 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2020-06-12T02:38:29.993 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2020-06-12T02:38:29.993 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138292836696063, 2020-06-12T02:38:29.993 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2020-06-12T02:38:29.994 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2020-06-12T02:38:09.953928+0000", 2020-06-12T02:38:29.994 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 0 2020-06-12T02:38:29.994 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.994 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:29.994 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": 7, 2020-06-12T02:38:29.995 INFO:teuthology.orchestra.run.smithi107.stdout: "down_stamp": "0.000000", 2020-06-12T02:38:29.995 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_probability": 0, 2020-06-12T02:38:29.995 INFO:teuthology.orchestra.run.smithi107.stdout: "laggy_interval": 0, 2020-06-12T02:38:29.995 INFO:teuthology.orchestra.run.smithi107.stdout: "features": 4540138292836696063, 2020-06-12T02:38:29.995 INFO:teuthology.orchestra.run.smithi107.stdout: "old_weight": 0, 2020-06-12T02:38:29.995 INFO:teuthology.orchestra.run.smithi107.stdout: "last_purged_snaps_scrub": "2020-06-12T02:38:09.294465+0000", 2020-06-12T02:38:29.996 INFO:teuthology.orchestra.run.smithi107.stdout: "dead_epoch": 19 2020-06-12T02:38:29.996 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:29.996 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2020-06-12T02:38:29.997 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_upmap": [], 2020-06-12T02:38:29.997 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_upmap_items": [], 2020-06-12T02:38:29.997 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_temp": [], 2020-06-12T02:38:29.997 INFO:teuthology.orchestra.run.smithi107.stdout: "primary_temp": [], 2020-06-12T02:38:29.998 INFO:teuthology.orchestra.run.smithi107.stdout: "blacklist": {}, 2020-06-12T02:38:29.998 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_profiles": { 2020-06-12T02:38:29.998 INFO:teuthology.orchestra.run.smithi107.stdout: "default": { 2020-06-12T02:38:29.998 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-failure-domain": "osd", 2020-06-12T02:38:29.998 INFO:teuthology.orchestra.run.smithi107.stdout: "k": "2", 2020-06-12T02:38:29.998 INFO:teuthology.orchestra.run.smithi107.stdout: "m": "1", 2020-06-12T02:38:29.999 INFO:teuthology.orchestra.run.smithi107.stdout: "plugin": "jerasure", 2020-06-12T02:38:29.999 INFO:teuthology.orchestra.run.smithi107.stdout: "ruleset-failure-domain": "osd", 2020-06-12T02:38:29.999 INFO:teuthology.orchestra.run.smithi107.stdout: "technique": "reed_sol_van" 2020-06-12T02:38:29.999 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:29.999 INFO:teuthology.orchestra.run.smithi107.stdout: "teuthologyprofile": { 2020-06-12T02:38:30.000 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-device-class": "", 2020-06-12T02:38:30.000 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-failure-domain": "osd", 2020-06-12T02:38:30.000 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-root": "default", 2020-06-12T02:38:30.000 INFO:teuthology.orchestra.run.smithi107.stdout: "jerasure-per-chunk-alignment": "false", 2020-06-12T02:38:30.000 INFO:teuthology.orchestra.run.smithi107.stdout: "k": "2", 2020-06-12T02:38:30.000 INFO:teuthology.orchestra.run.smithi107.stdout: "m": "1", 2020-06-12T02:38:30.001 INFO:teuthology.orchestra.run.smithi107.stdout: "plugin": "jerasure", 2020-06-12T02:38:30.001 INFO:teuthology.orchestra.run.smithi107.stdout: "ruleset-failure-domain": "osd", 2020-06-12T02:38:30.001 INFO:teuthology.orchestra.run.smithi107.stdout: "technique": "reed_sol_van", 2020-06-12T02:38:30.001 INFO:teuthology.orchestra.run.smithi107.stdout: "w": "8" 2020-06-12T02:38:30.001 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:30.002 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:30.002 INFO:teuthology.orchestra.run.smithi107.stdout: "removed_snaps_queue": [], 2020-06-12T02:38:30.002 INFO:teuthology.orchestra.run.smithi107.stdout: "new_removed_snaps": [], 2020-06-12T02:38:30.002 INFO:teuthology.orchestra.run.smithi107.stdout: "new_purged_snaps": [], 2020-06-12T02:38:30.002 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_node_flags": {}, 2020-06-12T02:38:30.003 INFO:teuthology.orchestra.run.smithi107.stdout: "device_class_flags": {} 2020-06-12T02:38:30.003 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:30.004 INFO:tasks.rados.rados.0.smithi107.stdout:6634: writing smithi10729407-9 from 0 to 696320 tid 1 2020-06-12T02:38:30.004 INFO:tasks.rados.rados.0.smithi107.stdout:6635: delete oid 13 current snap is 0 2020-06-12T02:38:30.004 INFO:tasks.rados.rados.0.smithi107.stdout:6632: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:30.020 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:30.020 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:30.021 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-06-12T02:38:30.021 INFO:tasks.rados.rados.0.smithi107.stdout:6635: done (3 left) 2020-06-12T02:38:30.022 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:30.022 INFO:tasks.rados.rados.0.smithi107.stdout:6636: seq_num 6635 ranges {598016=729088} 2020-06-12T02:38:30.022 INFO:tasks.rados.rados.0.smithi107.stdout:6636: writing smithi10729407-19 from 598016 to 1327104 tid 1 2020-06-12T02:38:30.022 INFO:tasks.rados.rados.0.smithi107.stdout:6637: delete oid 18 current snap is 0 2020-06-12T02:38:30.022 INFO:tasks.rados.rados.0.smithi107.stdout:6632: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:30.023 INFO:tasks.rados.rados.0.smithi107.stdout:6632: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:30.023 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1253 (ObjNum 6631 snap 0 seq_num 6631) dirty exists 2020-06-12T02:38:30.023 INFO:tasks.rados.rados.0.smithi107.stdout:6632: left oid 4 (ObjNum 6631 snap 0 seq_num 6631) 2020-06-12T02:38:30.023 INFO:tasks.rados.rados.0.smithi107.stdout:6634: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:30.023 INFO:tasks.rados.rados.0.smithi107.stdout:6634: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:30.023 INFO:tasks.rados.rados.0.smithi107.stdout:6634: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:30.024 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1211 (ObjNum 6633 snap 0 seq_num 6633) dirty exists 2020-06-12T02:38:30.024 INFO:tasks.rados.rados.0.smithi107.stdout:6634: left oid 9 (ObjNum 6633 snap 0 seq_num 6633) 2020-06-12T02:38:30.024 INFO:tasks.rados.rados.0.smithi107.stdout:6632: done (4 left) 2020-06-12T02:38:30.024 INFO:tasks.rados.rados.0.smithi107.stdout:6634: done (3 left) 2020-06-12T02:38:30.024 INFO:tasks.rados.rados.0.smithi107.stdout:6637: done (2 left) 2020-06-12T02:38:30.025 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:30.025 INFO:tasks.rados.rados.0.smithi107.stdout:6638: seq_num 6637 ranges {212992=761856} 2020-06-12T02:38:30.025 INFO:tasks.rados.rados.0.smithi107.stdout:6638: writing smithi10729407-16 from 212992 to 974848 tid 1 2020-06-12T02:38:30.025 INFO:tasks.rados.rados.0.smithi107.stdout:6639: delete oid 8 current snap is 0 2020-06-12T02:38:30.025 INFO:tasks.rados.rados.0.smithi107.stdout:6636: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:30.025 INFO:tasks.rados.rados.0.smithi107.stdout:6639: done (3 left) 2020-06-12T02:38:30.026 INFO:tasks.rados.rados.0.smithi107.stdout:6640: delete oid 6 current snap is 0 2020-06-12T02:38:30.026 INFO:tasks.rados.rados.0.smithi107.stdout:6636: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:30.026 INFO:tasks.rados.rados.0.smithi107.stdout:6636: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:30.026 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1216 (ObjNum 6635 snap 0 seq_num 6635) dirty exists 2020-06-12T02:38:30.026 INFO:tasks.rados.rados.0.smithi107.stdout:6636: left oid 19 (ObjNum 6635 snap 0 seq_num 6635) 2020-06-12T02:38:30.027 INFO:tasks.rados.rados.0.smithi107.stdout:6636: done (3 left) 2020-06-12T02:38:30.027 INFO:tasks.rados.rados.0.smithi107.stdout:6640: done (2 left) 2020-06-12T02:38:30.027 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:30.027 INFO:tasks.rados.rados.0.smithi107.stdout:6641: seq_num 6640 ranges {696320=409600,1105920=401408} 2020-06-12T02:38:30.028 INFO:tasks.rados.rados.0.smithi107.stdout:6641: writing smithi10729407-9 from 696320 to 1105920 tid 1 2020-06-12T02:38:30.028 INFO:tasks.rados.rados.0.smithi107.stdout:6641: writing smithi10729407-9 from 1105920 to 1507328 tid 2 2020-06-12T02:38:30.028 INFO:tasks.rados.rados.0.smithi107.stdout:6642: delete oid 13 current snap is 0 2020-06-12T02:38:30.028 INFO:tasks.rados.rados.0.smithi107.stdout:6638: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:30.028 INFO:tasks.rados.rados.0.smithi107.stdout:6638: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:30.029 INFO:tasks.rados.rados.0.smithi107.stdout:6638: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:30.029 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1183 (ObjNum 6637 snap 0 seq_num 6637) dirty exists 2020-06-12T02:38:30.029 INFO:tasks.rados.rados.0.smithi107.stdout:6638: left oid 16 (ObjNum 6637 snap 0 seq_num 6637) 2020-06-12T02:38:30.029 INFO:tasks.rados.rados.0.smithi107.stdout:6641: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:30.029 INFO:tasks.rados.rados.0.smithi107.stdout:6638: done (3 left) 2020-06-12T02:38:30.030 INFO:tasks.rados.rados.0.smithi107.stdout:6642: done (2 left) 2020-06-12T02:38:30.030 INFO:tasks.rados.rados.0.smithi107.stdout:6643: delete oid 14 current snap is 0 2020-06-12T02:38:30.030 INFO:tasks.rados.rados.0.smithi107.stdout:6643: done (2 left) 2020-06-12T02:38:30.030 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:38:30.030 INFO:tasks.rados.rados.0.smithi107.stdout:6644: seq_num 6643 ranges {548864=466944,1015808=401408} 2020-06-12T02:38:30.031 INFO:tasks.rados.rados.0.smithi107.stdout:6644: writing smithi10729407-11 from 548864 to 1015808 tid 1 2020-06-12T02:38:30.031 INFO:tasks.rados.rados.0.smithi107.stdout:6644: writing smithi10729407-11 from 1015808 to 1417216 tid 2 2020-06-12T02:38:30.031 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:30.031 INFO:tasks.rados.rados.0.smithi107.stdout:6645: seq_num 6644 ranges {90112=483328,573440=401408} 2020-06-12T02:38:30.031 INFO:tasks.rados.rados.0.smithi107.stdout:6645: writing smithi10729407-2 from 90112 to 573440 tid 1 2020-06-12T02:38:30.031 INFO:tasks.rados.rados.0.smithi107.stdout:6645: writing smithi10729407-2 from 573440 to 974848 tid 2 2020-06-12T02:38:30.032 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:30.032 INFO:tasks.rados.rados.0.smithi107.stdout:6646: seq_num 6645 ranges {0=499712,499712=401408} 2020-06-12T02:38:30.032 INFO:tasks.rados.rados.0.smithi107.stdout:6646: writing smithi10729407-12 from 0 to 499712 tid 1 2020-06-12T02:38:30.032 INFO:tasks.rados.rados.0.smithi107.stdout:6646: writing smithi10729407-12 from 499712 to 901120 tid 2 2020-06-12T02:38:30.032 INFO:tasks.rados.rados.0.smithi107.stdout:6647: delete oid 17 current snap is 0 2020-06-12T02:38:30.033 INFO:tasks.rados.rados.0.smithi107.stdout:6641: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:30.033 INFO:tasks.rados.rados.0.smithi107.stdout:6641: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:30.033 INFO:tasks.rados.rados.0.smithi107.stdout:6641: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:30.033 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1215 (ObjNum 6640 snap 0 seq_num 6640) dirty exists 2020-06-12T02:38:30.033 INFO:tasks.rados.rados.0.smithi107.stdout:6641: left oid 9 (ObjNum 6640 snap 0 seq_num 6640) 2020-06-12T02:38:30.033 INFO:tasks.rados.rados.0.smithi107.stdout:6644: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:30.034 INFO:tasks.rados.rados.0.smithi107.stdout:6644: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:30.034 INFO:tasks.rados.rados.0.smithi107.stdout:6644: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:30.034 INFO:tasks.rados.rados.0.smithi107.stdout:6644: finishing write tid 5 to smithi10729407-11 2020-06-12T02:38:30.034 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1180 (ObjNum 6643 snap 0 seq_num 6643) dirty exists 2020-06-12T02:38:30.034 INFO:tasks.rados.rados.0.smithi107.stdout:6644: left oid 11 (ObjNum 6643 snap 0 seq_num 6643) 2020-06-12T02:38:30.035 INFO:tasks.rados.rados.0.smithi107.stdout:6645: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:30.035 INFO:tasks.rados.rados.0.smithi107.stdout:6645: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:30.035 INFO:tasks.rados.rados.0.smithi107.stdout:6645: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:30.035 INFO:tasks.rados.rados.0.smithi107.stdout:6641: done (5 left) 2020-06-12T02:38:30.035 INFO:tasks.rados.rados.0.smithi107.stdout:6644: done (4 left) 2020-06-12T02:38:30.035 INFO:tasks.rados.rados.0.smithi107.stdout:6647: done (3 left) 2020-06-12T02:38:30.036 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:38:30.036 INFO:tasks.rados.rados.0.smithi107.stdout:6648: seq_num 6647 ranges {327680=532480,860160=401408} 2020-06-12T02:38:30.036 INFO:tasks.rados.rados.0.smithi107.stdout:6648: writing smithi10729407-20 from 327680 to 860160 tid 1 2020-06-12T02:38:30.036 INFO:tasks.rados.rados.0.smithi107.stdout:6648: writing smithi10729407-20 from 860160 to 1261568 tid 2 2020-06-12T02:38:30.036 INFO:tasks.rados.rados.0.smithi107.stdout:6649: delete oid 13 current snap is 0 2020-06-12T02:38:30.037 INFO:tasks.rados.rados.0.smithi107.stdout:6646: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:30.037 INFO:tasks.rados.rados.0.smithi107.stdout:6645: finishing write tid 5 to smithi10729407-2 2020-06-12T02:38:30.037 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 646 (ObjNum 6644 snap 0 seq_num 6644) dirty exists 2020-06-12T02:38:30.037 INFO:tasks.rados.rados.0.smithi107.stdout:6645: left oid 2 (ObjNum 6644 snap 0 seq_num 6644) 2020-06-12T02:38:30.037 INFO:tasks.rados.rados.0.smithi107.stdout:6646: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:30.037 INFO:tasks.rados.rados.0.smithi107.stdout:6646: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:30.038 INFO:tasks.rados.rados.0.smithi107.stdout:6646: finishing write tid 5 to smithi10729407-12 2020-06-12T02:38:30.038 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1219 (ObjNum 6645 snap 0 seq_num 6645) dirty exists 2020-06-12T02:38:30.038 INFO:tasks.rados.rados.0.smithi107.stdout:6646: left oid 12 (ObjNum 6645 snap 0 seq_num 6645) 2020-06-12T02:38:30.039 INFO:tasks.rados.rados.0.smithi107.stdout:6648: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:30.039 INFO:tasks.rados.rados.0.smithi107.stdout:6645: done (4 left) 2020-06-12T02:38:30.039 INFO:tasks.rados.rados.0.smithi107.stdout:6646: done (3 left) 2020-06-12T02:38:30.039 INFO:tasks.rados.rados.0.smithi107.stdout:6649: done (2 left) 2020-06-12T02:38:30.039 INFO:tasks.rados.rados.0.smithi107.stdout:6650: delete oid 19 current snap is 0 2020-06-12T02:38:30.040 INFO:tasks.rados.rados.0.smithi107.stdout:6648: finishing write tid 2 to smithi10729407-20 2020-06-12T02:38:30.040 INFO:tasks.rados.rados.0.smithi107.stdout:6648: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:30.040 INFO:tasks.rados.rados.0.smithi107.stdout:6648: finishing write tid 5 to smithi10729407-20 2020-06-12T02:38:30.040 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 501 (ObjNum 6647 snap 0 seq_num 6647) dirty exists 2020-06-12T02:38:30.040 INFO:tasks.rados.rados.0.smithi107.stdout:6648: left oid 20 (ObjNum 6647 snap 0 seq_num 6647) 2020-06-12T02:38:30.041 INFO:tasks.rados.rados.0.smithi107.stdout:6648: done (2 left) 2020-06-12T02:38:30.041 INFO:tasks.rados.rados.0.smithi107.stdout:6650: done (1 left) 2020-06-12T02:38:30.041 INFO:tasks.rados.rados.0.smithi107.stdout:6651: delete oid 17 current snap is 0 2020-06-12T02:38:30.041 INFO:tasks.rados.rados.0.smithi107.stdout:6651: done (1 left) 2020-06-12T02:38:30.041 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:30.042 INFO:tasks.rados.rados.0.smithi107.stdout:6652: seq_num 6651 ranges {4931584=598016,5529600=401408} 2020-06-12T02:38:30.042 INFO:tasks.rados.rados.0.smithi107.stdout:6652: writing smithi10729407-3 from 4931584 to 5529600 tid 1 2020-06-12T02:38:30.042 INFO:tasks.rados.rados.0.smithi107.stdout:6652: writing smithi10729407-3 from 5529600 to 5931008 tid 2 2020-06-12T02:38:30.042 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:30.042 INFO:tasks.rados.rados.0.smithi107.stdout:6653: seq_num 6652 ranges {0=614400,614400=401408} 2020-06-12T02:38:30.043 INFO:tasks.rados.rados.0.smithi107.stdout:6653: writing smithi10729407-18 from 0 to 614400 tid 1 2020-06-12T02:38:30.043 INFO:tasks.rados.rados.0.smithi107.stdout:6653: writing smithi10729407-18 from 614400 to 1015808 tid 2 2020-06-12T02:38:30.043 INFO:tasks.rados.rados.0.smithi107.stdout:6654: delete oid 19 current snap is 0 2020-06-12T02:38:30.043 INFO:tasks.rados.rados.0.smithi107.stdout:6652: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:30.043 INFO:tasks.rados.rados.0.smithi107.stdout:6652: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:30.043 INFO:tasks.rados.rados.0.smithi107.stdout:6654: done (3 left) 2020-06-12T02:38:30.044 INFO:tasks.rados.rados.0.smithi107.stdout:6655: delete oid 15 current snap is 0 2020-06-12T02:38:30.044 INFO:tasks.rados.rados.0.smithi107.stdout:6652: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:30.044 INFO:tasks.rados.rados.0.smithi107.stdout:6652: finishing write tid 5 to smithi10729407-3 2020-06-12T02:38:30.044 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1186 (ObjNum 6651 snap 0 seq_num 6651) dirty exists 2020-06-12T02:38:30.044 INFO:tasks.rados.rados.0.smithi107.stdout:6652: left oid 3 (ObjNum 6651 snap 0 seq_num 6651) 2020-06-12T02:38:30.045 INFO:tasks.rados.rados.0.smithi107.stdout:6652: done (3 left) 2020-06-12T02:38:30.045 INFO:tasks.rados.rados.0.smithi107.stdout:6655: done (2 left) 2020-06-12T02:38:30.045 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:30.045 INFO:tasks.rados.rados.0.smithi107.stdout:6656: seq_num 6655 ranges {0=663552,663552=401408} 2020-06-12T02:38:30.045 INFO:tasks.rados.rados.0.smithi107.stdout:6656: writing smithi10729407-7 from 0 to 663552 tid 1 2020-06-12T02:38:30.046 INFO:tasks.rados.rados.0.smithi107.stdout:6656: writing smithi10729407-7 from 663552 to 1064960 tid 2 2020-06-12T02:38:30.046 INFO:tasks.rados.rados.0.smithi107.stdout:6653: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:30.046 INFO:tasks.rados.rados.0.smithi107.stdout:6657: delete oid 5 current snap is 0 2020-06-12T02:38:30.046 INFO:tasks.rados.rados.0.smithi107.stdout:6653: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:30.046 INFO:tasks.rados.rados.0.smithi107.stdout:6653: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:30.046 INFO:tasks.rados.rados.0.smithi107.stdout:6653: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:30.047 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1218 (ObjNum 6652 snap 0 seq_num 6652) dirty exists 2020-06-12T02:38:30.047 INFO:tasks.rados.rados.0.smithi107.stdout:6653: left oid 18 (ObjNum 6652 snap 0 seq_num 6652) 2020-06-12T02:38:30.047 INFO:tasks.rados.rados.0.smithi107.stdout:6653: done (3 left) 2020-06-12T02:38:30.047 INFO:tasks.rados.rados.0.smithi107.stdout:6657: done (2 left) 2020-06-12T02:38:30.047 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:30.048 INFO:tasks.rados.rados.0.smithi107.stdout:6658: seq_num 6657 ranges {0=696320,696320=401408} 2020-06-12T02:38:30.048 INFO:tasks.rados.rados.0.smithi107.stdout:6658: writing smithi10729407-17 from 0 to 696320 tid 1 2020-06-12T02:38:30.048 INFO:tasks.rados.rados.0.smithi107.stdout:6658: writing smithi10729407-17 from 696320 to 1097728 tid 2 2020-06-12T02:38:30.048 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:38:30.049 INFO:tasks.rados.rados.0.smithi107.stdout:6659: seq_num 6658 ranges {0=712704,712704=401408} 2020-06-12T02:38:30.049 INFO:tasks.rados.rados.0.smithi107.stdout:6659: writing smithi10729407-1 from 0 to 712704 tid 1 2020-06-12T02:38:30.049 INFO:tasks.rados.rados.0.smithi107.stdout:6659: writing smithi10729407-1 from 712704 to 1114112 tid 2 2020-06-12T02:38:30.049 INFO:tasks.rados.rados.0.smithi107.stdout:6660: delete oid 2 current snap is 0 2020-06-12T02:38:30.050 INFO:tasks.rados.rados.0.smithi107.stdout:6656: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:30.050 INFO:tasks.rados.rados.0.smithi107.stdout:6656: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:30.050 INFO:tasks.rados.rados.0.smithi107.stdout:6656: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:30.050 INFO:tasks.rados.rados.0.smithi107.stdout:6656: finishing write tid 5 to smithi10729407-7 2020-06-12T02:38:30.050 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1252 (ObjNum 6655 snap 0 seq_num 6655) dirty exists 2020-06-12T02:38:30.051 INFO:tasks.rados.rados.0.smithi107.stdout:6656: left oid 7 (ObjNum 6655 snap 0 seq_num 6655) 2020-06-12T02:38:30.051 INFO:tasks.rados.rados.0.smithi107.stdout:6656: done (4 left) 2020-06-12T02:38:30.051 INFO:tasks.rados.rados.0.smithi107.stdout:6660: done (3 left) 2020-06-12T02:38:30.051 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:30.051 INFO:tasks.rados.rados.0.smithi107.stdout:6661: seq_num 6660 ranges {1507328=745472,2252800=401408} 2020-06-12T02:38:30.052 INFO:tasks.rados.rados.0.smithi107.stdout:6661: writing smithi10729407-9 from 1507328 to 2252800 tid 1 2020-06-12T02:38:30.052 INFO:tasks.rados.rados.0.smithi107.stdout:6661: writing smithi10729407-9 from 2252800 to 2654208 tid 2 2020-06-12T02:38:30.052 INFO:tasks.rados.rados.0.smithi107.stdout:6662: delete oid 8 current snap is 0 2020-06-12T02:38:30.052 INFO:tasks.rados.rados.0.smithi107.stdout:6658: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:30.052 INFO:tasks.rados.rados.0.smithi107.stdout:6658: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:30.053 INFO:tasks.rados.rados.0.smithi107.stdout:6658: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:30.053 INFO:tasks.rados.rados.0.smithi107.stdout:6658: finishing write tid 5 to smithi10729407-17 2020-06-12T02:38:30.053 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1258 (ObjNum 6657 snap 0 seq_num 6657) dirty exists 2020-06-12T02:38:30.053 INFO:tasks.rados.rados.0.smithi107.stdout:6658: left oid 17 (ObjNum 6657 snap 0 seq_num 6657) 2020-06-12T02:38:30.054 INFO:tasks.rados.rados.0.smithi107.stdout:6659: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:30.054 INFO:tasks.rados.rados.0.smithi107.stdout:6659: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:30.054 INFO:tasks.rados.rados.0.smithi107.stdout:6659: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:30.054 INFO:tasks.rados.rados.0.smithi107.stdout:6659: finishing write tid 5 to smithi10729407-1 2020-06-12T02:38:30.054 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1210 (ObjNum 6658 snap 0 seq_num 6658) dirty exists 2020-06-12T02:38:30.054 INFO:tasks.rados.rados.0.smithi107.stdout:6659: left oid 1 (ObjNum 6658 snap 0 seq_num 6658) 2020-06-12T02:38:30.055 INFO:tasks.rados.rados.0.smithi107.stdout:6658: done (4 left) 2020-06-12T02:38:30.055 INFO:tasks.rados.rados.0.smithi107.stdout:6659: done (3 left) 2020-06-12T02:38:30.055 INFO:tasks.rados.rados.0.smithi107.stdout:6662: done (2 left) 2020-06-12T02:38:30.055 INFO:tasks.rados.rados.0.smithi107.stdout:6663: delete oid 1 current snap is 0 2020-06-12T02:38:30.055 INFO:tasks.rados.rados.0.smithi107.stdout:6663: done (2 left) 2020-06-12T02:38:30.056 INFO:tasks.rados.rados.0.smithi107.stdout:6664: delete oid 14 current snap is 0 2020-06-12T02:38:30.056 INFO:tasks.rados.rados.0.smithi107.stdout:6661: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:30.056 INFO:tasks.rados.rados.0.smithi107.stdout:6664: done (2 left) 2020-06-12T02:38:30.056 INFO:tasks.rados.rados.0.smithi107.stdout:6665: delete oid 16 current snap is 0 2020-06-12T02:38:30.056 INFO:tasks.rados.rados.0.smithi107.stdout:6661: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:30.056 INFO:tasks.rados.rados.0.smithi107.stdout:6661: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:30.057 INFO:tasks.rados.rados.0.smithi107.stdout:6661: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:30.057 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1221 (ObjNum 6660 snap 0 seq_num 6660) dirty exists 2020-06-12T02:38:30.057 INFO:tasks.rados.rados.0.smithi107.stdout:6661: left oid 9 (ObjNum 6660 snap 0 seq_num 6660) 2020-06-12T02:38:30.057 INFO:tasks.rados.rados.0.smithi107.stdout:6661: done (2 left) 2020-06-12T02:38:30.057 INFO:tasks.rados.rados.0.smithi107.stdout:6665: done (1 left) 2020-06-12T02:38:30.058 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:30.058 INFO:tasks.rados.rados.0.smithi107.stdout:6666: seq_num 6665 ranges {0=434176,434176=598016,1032192=204800} 2020-06-12T02:38:30.058 INFO:tasks.rados.rados.0.smithi107.stdout:6666: writing smithi10729407-15 from 0 to 434176 tid 1 2020-06-12T02:38:30.058 INFO:tasks.rados.rados.0.smithi107.stdout:6666: writing smithi10729407-15 from 434176 to 1032192 tid 2 2020-06-12T02:38:30.059 INFO:tasks.rados.rados.0.smithi107.stdout:6666: writing smithi10729407-15 from 1032192 to 1236992 tid 3 2020-06-12T02:38:30.059 INFO:tasks.rados.rados.0.smithi107.stdout:6667: delete oid 18 current snap is 0 2020-06-12T02:38:30.059 INFO:tasks.rados.rados.0.smithi107.stdout:6666: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:30.059 INFO:tasks.rados.rados.0.smithi107.stdout:6667: done (2 left) 2020-06-12T02:38:30.059 INFO:tasks.rados.rados.0.smithi107.stdout:6668: delete oid 9 current snap is 0 2020-06-12T02:38:30.059 INFO:tasks.rados.rados.0.smithi107.stdout:6668: done (2 left) 2020-06-12T02:38:30.060 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:30.060 INFO:tasks.rados.rados.0.smithi107.stdout:6669: seq_num 6668 ranges {0=483328,483328=704512,1187840=98304} 2020-06-12T02:38:30.060 INFO:tasks.rados.rados.0.smithi107.stdout:6669: writing smithi10729407-18 from 0 to 483328 tid 1 2020-06-12T02:38:30.060 INFO:tasks.rados.rados.0.smithi107.stdout:6669: writing smithi10729407-18 from 483328 to 1187840 tid 2 2020-06-12T02:38:30.060 INFO:tasks.rados.rados.0.smithi107.stdout:6669: writing smithi10729407-18 from 1187840 to 1286144 tid 3 2020-06-12T02:38:30.061 INFO:tasks.rados.rados.0.smithi107.stdout:6670: delete oid 20 current snap is 0 2020-06-12T02:38:30.061 INFO:tasks.rados.rados.0.smithi107.stdout:6666: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:30.061 INFO:tasks.rados.rados.0.smithi107.stdout:6666: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:30.061 INFO:tasks.rados.rados.0.smithi107.stdout:6666: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:30.061 INFO:tasks.rados.rados.0.smithi107.stdout:6666: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:30.062 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1243 (ObjNum 6665 snap 0 seq_num 6665) dirty exists 2020-06-12T02:38:30.062 INFO:tasks.rados.rados.0.smithi107.stdout:6666: left oid 15 (ObjNum 6665 snap 0 seq_num 6665) 2020-06-12T02:38:30.062 INFO:tasks.rados.rados.0.smithi107.stdout:6669: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:30.062 INFO:tasks.rados.rados.0.smithi107.stdout:6666: done (3 left) 2020-06-12T02:38:30.063 INFO:tasks.rados.rados.0.smithi107.stdout:6670: done (2 left) 2020-06-12T02:38:30.063 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:30.063 INFO:tasks.rados.rados.0.smithi107.stdout:6671: seq_num 6670 ranges {0=516096,516096=466944,983040=335872} 2020-06-12T02:38:30.063 INFO:tasks.rados.rados.0.smithi107.stdout:6671: writing smithi10729407-13 from 0 to 516096 tid 1 2020-06-12T02:38:30.063 INFO:tasks.rados.rados.0.smithi107.stdout:6671: writing smithi10729407-13 from 516096 to 983040 tid 2 2020-06-12T02:38:30.063 INFO:tasks.rados.rados.0.smithi107.stdout:6671: writing smithi10729407-13 from 983040 to 1318912 tid 3 2020-06-12T02:38:30.064 INFO:tasks.rados.rados.0.smithi107.stdout:6672: delete oid 8 current snap is 0 2020-06-12T02:38:30.064 INFO:tasks.rados.rados.0.smithi107.stdout:6669: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:30.064 INFO:tasks.rados.rados.0.smithi107.stdout:6669: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:30.064 INFO:tasks.rados.rados.0.smithi107.stdout:6669: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:30.064 INFO:tasks.rados.rados.0.smithi107.stdout:6669: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:30.064 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1227 (ObjNum 6668 snap 0 seq_num 6668) dirty exists 2020-06-12T02:38:30.065 INFO:tasks.rados.rados.0.smithi107.stdout:6669: left oid 18 (ObjNum 6668 snap 0 seq_num 6668) 2020-06-12T02:38:30.065 INFO:tasks.rados.rados.0.smithi107.stdout:6671: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:30.065 INFO:tasks.rados.rados.0.smithi107.stdout:6669: done (3 left) 2020-06-12T02:38:30.065 INFO:tasks.rados.rados.0.smithi107.stdout:6672: done (2 left) 2020-06-12T02:38:30.065 INFO:tasks.rados.rados.0.smithi107.stdout:6673: delete oid 11 current snap is 0 2020-06-12T02:38:30.066 INFO:tasks.rados.rados.0.smithi107.stdout:6673: done (2 left) 2020-06-12T02:38:30.066 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:30.066 INFO:tasks.rados.rados.0.smithi107.stdout:6674: seq_num 6673 ranges {0=565248,565248=524288,1089536=278528} 2020-06-12T02:38:30.066 INFO:tasks.rados.rados.0.smithi107.stdout:6674: writing smithi10729407-19 from 0 to 565248 tid 1 2020-06-12T02:38:30.067 INFO:tasks.thrashosds.thrasher:j is {'epoch': 58, 'fsid': '2f5030a1-3206-43c9-8471-d10429688d36', 'created': '2020-06-12T02:37:00.578493+0000', 'modified': '2020-06-12T02:38:27.403966+0000', 'last_up_change': '2020-06-12T02:37:53.542920+0000', 'last_in_change': '2020-06-12T02:37:10.705891+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'jewel', 'min_compat_client': 'jewel', 'require_osd_release': 'octopus', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-06-12T02:37:03.610773+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-06-12T02:37:14.831188+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'unique_pool_0', 'create_time': '2020-06-12T02:37:25.893619+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 3, 'size': 3, 'min_size': 2, 'crush_rule': 1, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'teuthologyprofile', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': True, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'ad4b4814-83e7-4a2e-9b1f-d10406174f7e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 53, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6816', 'nonce': 26614}, {'type': 'v1', 'addr': '172.21.15.107:6817', 'nonce': 26614}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6819', 'nonce': 26614}, {'type': 'v1', 'addr': '172.21.15.107:6821', 'nonce': 26614}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6827', 'nonce': 26614}, {'type': 'v1', 'addr': '172.21.15.107:6829', 'nonce': 26614}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6823', 'nonce': 26614}, {'type': 'v1', 'addr': '172.21.15.107:6825', 'nonce': 26614}]}, 'public_addr': '172.21.15.107:6817/26614', 'cluster_addr': '172.21.15.107:6821/26614', 'heartbeat_back_addr': '172.21.15.107:6829/26614', 'heartbeat_front_addr': '172.21.15.107:6825/26614', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '046adf0d-b9e9-41ad-a7c9-3841f8b8f790', 'up': 1, 'in': 1, 'weight': 0.1898193359375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 29, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6800', 'nonce': 26611}, {'type': 'v1', 'addr': '172.21.15.107:6801', 'nonce': 26611}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6802', 'nonce': 26611}, {'type': 'v1', 'addr': '172.21.15.107:6803', 'nonce': 26611}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6806', 'nonce': 26611}, {'type': 'v1', 'addr': '172.21.15.107:6807', 'nonce': 26611}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6804', 'nonce': 26611}, {'type': 'v1', 'addr': '172.21.15.107:6805', 'nonce': 26611}]}, 'public_addr': '172.21.15.107:6801/26611', 'cluster_addr': '172.21.15.107:6803/26611', 'heartbeat_back_addr': '172.21.15.107:6807/26611', 'heartbeat_front_addr': '172.21.15.107:6805/26611', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'e15da6b7-dc79-4185-a5f9-0f443d8b52b6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 52, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6818', 'nonce': 26613}, {'type': 'v1', 'addr': '172.21.15.107:6820', 'nonce': 26613}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6822', 'nonce': 26613}, {'type': 'v1', 'addr': '172.21.15.107:6824', 'nonce': 26613}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6830', 'nonce': 26613}, {'type': 'v1', 'addr': '172.21.15.107:6831', 'nonce': 26613}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6826', 'nonce': 26613}, {'type': 'v1', 'addr': '172.21.15.107:6828', 'nonce': 26613}]}, 'public_addr': '172.21.15.107:6820/26613', 'cluster_addr': '172.21.15.107:6824/26613', 'heartbeat_back_addr': '172.21.15.107:6831/26613', 'heartbeat_front_addr': '172.21.15.107:6828/26613', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '843e0dd3-5ce4-46b9-9004-69849048d2bd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 53, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6808', 'nonce': 26612}, {'type': 'v1', 'addr': '172.21.15.107:6809', 'nonce': 26612}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6810', 'nonce': 26612}, {'type': 'v1', 'addr': '172.21.15.107:6811', 'nonce': 26612}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6814', 'nonce': 26612}, {'type': 'v1', 'addr': '172.21.15.107:6815', 'nonce': 26612}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6812', 'nonce': 26612}, {'type': 'v1', 'addr': '172.21.15.107:6813', 'nonce': 26612}]}, 'public_addr': '172.21.15.107:6809/26612', 'cluster_addr': '172.21.15.107:6811/26612', 'heartbeat_back_addr': '172.21.15.107:6815/26612', 'heartbeat_front_addr': '172.21.15.107:6813/26612', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'c066ab09-e031-4b49-9df3-d54cd9f9b580', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 53, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6818', 'nonce': 26685}, {'type': 'v1', 'addr': '172.21.15.188:6820', 'nonce': 26685}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6822', 'nonce': 26685}, {'type': 'v1', 'addr': '172.21.15.188:6823', 'nonce': 26685}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6830', 'nonce': 26685}, {'type': 'v1', 'addr': '172.21.15.188:6832', 'nonce': 26685}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6825', 'nonce': 26685}, {'type': 'v1', 'addr': '172.21.15.188:6828', 'nonce': 26685}]}, 'public_addr': '172.21.15.188:6820/26685', 'cluster_addr': '172.21.15.188:6823/26685', 'heartbeat_back_addr': '172.21.15.188:6832/26685', 'heartbeat_front_addr': '172.21.15.188:6828/26685', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '56a24e45-1b1b-4a65-b4b4-57cfab296f65', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 36, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6819', 'nonce': 26684}, {'type': 'v1', 'addr': '172.21.15.188:6821', 'nonce': 26684}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6824', 'nonce': 26684}, {'type': 'v1', 'addr': '172.21.15.188:6826', 'nonce': 26684}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6831', 'nonce': 26684}, {'type': 'v1', 'addr': '172.21.15.188:6833', 'nonce': 26684}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6827', 'nonce': 26684}, {'type': 'v1', 'addr': '172.21.15.188:6829', 'nonce': 26684}]}, 'public_addr': '172.21.15.188:6821/26684', 'cluster_addr': '172.21.15.188:6826/26684', 'heartbeat_back_addr': '172.21.15.188:6833/26684', 'heartbeat_front_addr': '172.21.15.188:6829/26684', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'f207a2a8-b3c7-4fd8-ae90-4c2d7a5c1c38', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 53, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6810', 'nonce': 26682}, {'type': 'v1', 'addr': '172.21.15.188:6811', 'nonce': 26682}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6812', 'nonce': 26682}, {'type': 'v1', 'addr': '172.21.15.188:6813', 'nonce': 26682}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6816', 'nonce': 26682}, {'type': 'v1', 'addr': '172.21.15.188:6817', 'nonce': 26682}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6814', 'nonce': 26682}, {'type': 'v1', 'addr': '172.21.15.188:6815', 'nonce': 26682}]}, 'public_addr': '172.21.15.188:6811/26682', 'cluster_addr': '172.21.15.188:6813/26682', 'heartbeat_back_addr': '172.21.15.188:6817/26682', 'heartbeat_front_addr': '172.21.15.188:6815/26682', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '4e5cc44f-af17-47c9-973f-8e87d1bd98db', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 18, 'up_from': 34, 'up_thru': 50, 'down_at': 19, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6802', 'nonce': 29046}, {'type': 'v1', 'addr': '172.21.15.188:6803', 'nonce': 29046}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6804', 'nonce': 29046}, {'type': 'v1', 'addr': '172.21.15.188:6805', 'nonce': 29046}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6808', 'nonce': 29046}, {'type': 'v1', 'addr': '172.21.15.188:6809', 'nonce': 29046}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6806', 'nonce': 29046}, {'type': 'v1', 'addr': '172.21.15.188:6807', 'nonce': 29046}]}, 'public_addr': '172.21.15.188:6803/29046', 'cluster_addr': '172.21.15.188:6805/29046', 'heartbeat_back_addr': '172.21.15.188:6809/29046', 'heartbeat_front_addr': '172.21.15.188:6807/29046', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138292836696063, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-06-12T02:38:09.980294+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138292836696063, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-06-12T02:38:09.373534+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138292836696063, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-06-12T02:38:09.542448+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138292836696063, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-06-12T02:38:09.528988+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138292836696063, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-06-12T02:38:08.934675+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138292836696063, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-06-12T02:38:09.183394+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138292836696063, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-06-12T02:38:09.953928+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138292836696063, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-06-12T02:38:09.294465+0000', 'dead_epoch': 19}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_temp': [], 'primary_temp': [], 'blacklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}, 'teuthologyprofile': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}} 2020-06-12T02:38:30.068 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json 2020-06-12T02:38:30.069 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:30.003+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:30.072 INFO:tasks.rados.rados.0.smithi107.stdout:6674: writing smithi10729407-19 from 565248 to 1089536 tid 2 2020-06-12T02:38:30.072 INFO:tasks.rados.rados.0.smithi107.stdout:6674: writing smithi10729407-19 from 1089536 to 1368064 tid 3 2020-06-12T02:38:30.072 INFO:tasks.rados.rados.0.smithi107.stdout:6675: delete oid 20 current snap is 0 2020-06-12T02:38:30.072 INFO:tasks.rados.rados.0.smithi107.stdout:6671: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:30.073 INFO:tasks.rados.rados.0.smithi107.stdout:6671: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:30.073 INFO:tasks.rados.rados.0.smithi107.stdout:6671: finishing write tid 5 to smithi10729407-13 2020-06-12T02:38:30.073 INFO:tasks.rados.rados.0.smithi107.stdout:6671: finishing write tid 6 to smithi10729407-13 2020-06-12T02:38:30.073 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1256 (ObjNum 6670 snap 0 seq_num 6670) dirty exists 2020-06-12T02:38:30.073 INFO:tasks.rados.rados.0.smithi107.stdout:6671: left oid 13 (ObjNum 6670 snap 0 seq_num 6670) 2020-06-12T02:38:30.074 INFO:tasks.rados.rados.0.smithi107.stdout:6671: done (3 left) 2020-06-12T02:38:30.074 INFO:tasks.rados.rados.0.smithi107.stdout:6675: done (2 left) 2020-06-12T02:38:30.074 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:30.074 INFO:tasks.rados.rados.0.smithi107.stdout:6676: seq_num 6675 ranges {1286144=598016,1884160=737280,2621440=65536} 2020-06-12T02:38:30.074 INFO:tasks.rados.rados.0.smithi107.stdout:6676: writing smithi10729407-18 from 1286144 to 1884160 tid 1 2020-06-12T02:38:30.074 INFO:tasks.rados.rados.0.smithi107.stdout:6676: writing smithi10729407-18 from 1884160 to 2621440 tid 2 2020-06-12T02:38:30.075 INFO:tasks.rados.rados.0.smithi107.stdout:6676: writing smithi10729407-18 from 2621440 to 2686976 tid 3 2020-06-12T02:38:30.075 INFO:tasks.rados.rados.0.smithi107.stdout:6677: delete oid 1 current snap is 0 2020-06-12T02:38:30.075 INFO:tasks.rados.rados.0.smithi107.stdout:6674: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:30.075 INFO:tasks.rados.rados.0.smithi107.stdout:6674: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:30.075 INFO:tasks.rados.rados.0.smithi107.stdout:6674: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:30.076 INFO:tasks.rados.rados.0.smithi107.stdout:6674: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:30.076 INFO:tasks.rados.rados.0.smithi107.stdout:6674: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:30.076 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1225 (ObjNum 6673 snap 0 seq_num 6673) dirty exists 2020-06-12T02:38:30.076 INFO:tasks.rados.rados.0.smithi107.stdout:6674: left oid 19 (ObjNum 6673 snap 0 seq_num 6673) 2020-06-12T02:38:30.076 INFO:tasks.rados.rados.0.smithi107.stdout:6674: done (3 left) 2020-06-12T02:38:30.076 INFO:tasks.rados.rados.0.smithi107.stdout:6677: done (2 left) 2020-06-12T02:38:30.077 INFO:tasks.rados.rados.0.smithi107.stdout:6678: delete oid 12 current snap is 0 2020-06-12T02:38:30.077 INFO:tasks.rados.rados.0.smithi107.stdout:6676: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:30.077 INFO:tasks.rados.rados.0.smithi107.stdout:6678: done (2 left) 2020-06-12T02:38:30.077 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:30.077 INFO:tasks.rados.rados.0.smithi107.stdout:6679: seq_num 6678 ranges {0=647168,647168=794624,1441792=8192} 2020-06-12T02:38:30.078 INFO:tasks.rados.rados.0.smithi107.stdout:6679: writing smithi10729407-5 from 0 to 647168 tid 1 2020-06-12T02:38:30.078 INFO:tasks.rados.rados.0.smithi107.stdout:6679: writing smithi10729407-5 from 647168 to 1441792 tid 2 2020-06-12T02:38:30.078 INFO:tasks.rados.rados.0.smithi107.stdout:6679: writing smithi10729407-5 from 1441792 to 1449984 tid 3 2020-06-12T02:38:30.078 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:30.078 INFO:tasks.rados.rados.0.smithi107.stdout:6680: seq_num 6679 ranges {1236992=663552,1900544=679936,2580480=122880} 2020-06-12T02:38:30.078 INFO:tasks.rados.rados.0.smithi107.stdout:6680: writing smithi10729407-15 from 1236992 to 1900544 tid 1 2020-06-12T02:38:30.080 INFO:tasks.rados.rados.0.smithi107.stdout:6680: writing smithi10729407-15 from 1900544 to 2580480 tid 2 2020-06-12T02:38:30.081 INFO:tasks.rados.rados.0.smithi107.stdout:6680: writing smithi10729407-15 from 2580480 to 2703360 tid 3 2020-06-12T02:38:30.081 INFO:tasks.rados.rados.0.smithi107.stdout:6681: delete oid 20 current snap is 0 2020-06-12T02:38:30.081 INFO:tasks.rados.rados.0.smithi107.stdout:6676: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:30.081 INFO:tasks.rados.rados.0.smithi107.stdout:6676: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:30.082 INFO:tasks.rados.rados.0.smithi107.stdout:6676: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:30.082 INFO:tasks.rados.rados.0.smithi107.stdout:6676: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:30.082 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1231 (ObjNum 6675 snap 0 seq_num 6675) dirty exists 2020-06-12T02:38:30.082 INFO:tasks.rados.rados.0.smithi107.stdout:6676: left oid 18 (ObjNum 6675 snap 0 seq_num 6675) 2020-06-12T02:38:30.082 INFO:tasks.rados.rados.0.smithi107.stdout:6679: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:30.082 INFO:tasks.rados.rados.0.smithi107.stdout:6679: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:30.083 INFO:tasks.rados.rados.0.smithi107.stdout:6679: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:30.083 INFO:tasks.rados.rados.0.smithi107.stdout:6679: finishing write tid 5 to smithi10729407-5 2020-06-12T02:38:30.083 INFO:tasks.rados.rados.0.smithi107.stdout:6679: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:30.083 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 666 (ObjNum 6678 snap 0 seq_num 6678) dirty exists 2020-06-12T02:38:30.083 INFO:tasks.rados.rados.0.smithi107.stdout:6679: left oid 5 (ObjNum 6678 snap 0 seq_num 6678) 2020-06-12T02:38:30.084 INFO:tasks.rados.rados.0.smithi107.stdout:6676: done (4 left) 2020-06-12T02:38:30.084 INFO:tasks.rados.rados.0.smithi107.stdout:6679: done (3 left) 2020-06-12T02:38:30.084 INFO:tasks.rados.rados.0.smithi107.stdout:6681: done (2 left) 2020-06-12T02:38:30.084 INFO:tasks.rados.rados.0.smithi107.stdout:6682: delete oid 5 current snap is 0 2020-06-12T02:38:30.084 INFO:tasks.rados.rados.0.smithi107.stdout:6680: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:30.085 INFO:tasks.rados.rados.0.smithi107.stdout:6682: done (2 left) 2020-06-12T02:38:30.085 INFO:tasks.rados.rados.0.smithi107.stdout:6683: delete oid 20 current snap is 0 2020-06-12T02:38:30.090 INFO:tasks.rados.rados.0.smithi107.stdout:6683: done (2 left) 2020-06-12T02:38:30.090 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:38:30.090 INFO:tasks.rados.rados.0.smithi107.stdout:6684: seq_num 6683 ranges {0=737280,737280=663552,1400832=139264} 2020-06-12T02:38:30.094 INFO:tasks.rados.rados.0.smithi107.stdout:6684: writing smithi10729407-5 from 0 to 737280 tid 1 2020-06-12T02:38:30.098 INFO:tasks.rados.rados.0.smithi107.stdout:6684: writing smithi10729407-5 from 737280 to 1400832 tid 2 2020-06-12T02:38:30.099 INFO:tasks.rados.rados.0.smithi107.stdout:6684: writing smithi10729407-5 from 1400832 to 1540096 tid 3 2020-06-12T02:38:30.099 INFO:tasks.rados.rados.0.smithi107.stdout:6685: delete oid 6 current snap is 0 2020-06-12T02:38:30.099 INFO:tasks.rados.rados.0.smithi107.stdout:6680: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:30.100 INFO:tasks.rados.rados.0.smithi107.stdout:6680: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:30.100 INFO:tasks.rados.rados.0.smithi107.stdout:6680: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:30.100 INFO:tasks.rados.rados.0.smithi107.stdout:6680: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:30.100 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1248 (ObjNum 6679 snap 0 seq_num 6679) dirty exists 2020-06-12T02:38:30.100 INFO:tasks.rados.rados.0.smithi107.stdout:6680: left oid 15 (ObjNum 6679 snap 0 seq_num 6679) 2020-06-12T02:38:30.102 INFO:tasks.rados.rados.0.smithi107.stdout:6684: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:30.102 INFO:tasks.rados.rados.0.smithi107.stdout:6680: done (3 left) 2020-06-12T02:38:30.102 INFO:tasks.rados.rados.0.smithi107.stdout:6685: done (2 left) 2020-06-12T02:38:30.103 INFO:tasks.rados.rados.0.smithi107.stdout:6686: delete oid 13 current snap is 0 2020-06-12T02:38:30.106 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:30.103+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:30.109 INFO:tasks.rados.rados.0.smithi107.stdout:6686: done (2 left) 2020-06-12T02:38:30.109 INFO:tasks.rados.rados.0.smithi107.stdout:6687: delete oid 11 current snap is 0 2020-06-12T02:38:30.111 INFO:tasks.rados.rados.0.smithi107.stdout:6684: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:30.112 INFO:tasks.rados.rados.0.smithi107.stdout:6684: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:30.113 INFO:tasks.rados.rados.0.smithi107.stdout:6684: finishing write tid 5 to smithi10729407-5 2020-06-12T02:38:30.113 INFO:tasks.rados.rados.0.smithi107.stdout:6687: done (2 left) 2020-06-12T02:38:30.113 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:38:30.114 INFO:tasks.rados.rados.0.smithi107.stdout:6688: seq_num 6687 ranges {0=802816,802816=606208,1409024=196608} 2020-06-12T02:38:30.117 INFO:tasks.rados.rados.0.smithi107.stdout:6688: writing smithi10729407-16 from 0 to 802816 tid 1 2020-06-12T02:38:30.121 INFO:tasks.rados.rados.0.smithi107.stdout:6688: writing smithi10729407-16 from 802816 to 1409024 tid 2 2020-06-12T02:38:30.123 INFO:tasks.rados.rados.0.smithi107.stdout:6688: writing smithi10729407-16 from 1409024 to 1605632 tid 3 2020-06-12T02:38:30.123 INFO:tasks.rados.rados.0.smithi107.stdout:6689: delete oid 4 current snap is 0 2020-06-12T02:38:30.124 INFO:tasks.rados.rados.0.smithi107.stdout:6684: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:30.124 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 671 (ObjNum 6683 snap 0 seq_num 6683) dirty exists 2020-06-12T02:38:30.124 INFO:tasks.rados.rados.0.smithi107.stdout:6684: left oid 5 (ObjNum 6683 snap 0 seq_num 6683) 2020-06-12T02:38:30.125 INFO:tasks.rados.rados.0.smithi107.stdout:6684: done (3 left) 2020-06-12T02:38:30.125 INFO:tasks.rados.rados.0.smithi107.stdout:6689: done (2 left) 2020-06-12T02:38:30.126 INFO:tasks.rados.rados.0.smithi107.stdout:6690: delete oid 20 current snap is 0 2020-06-12T02:38:30.129 INFO:tasks.rados.rados.0.smithi107.stdout:6690: done (2 left) 2020-06-12T02:38:30.129 INFO:tasks.rados.rados.0.smithi107.stdout:6691: delete oid 18 current snap is 0 2020-06-12T02:38:30.130 INFO:tasks.rados.rados.0.smithi107.stdout:6688: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:30.131 INFO:tasks.rados.rados.0.smithi107.stdout:6688: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:30.131 INFO:tasks.rados.rados.0.smithi107.stdout:6688: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:30.131 INFO:tasks.rados.rados.0.smithi107.stdout:6688: finishing write tid 5 to smithi10729407-16 2020-06-12T02:38:30.132 INFO:tasks.rados.rados.0.smithi107.stdout:6691: done (2 left) 2020-06-12T02:38:30.133 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:38:30.133 INFO:tasks.rados.rados.0.smithi107.stdout:6692: seq_num 6691 ranges {1064960=466944,1531904=589824,2121728=516096,2637824=98304} 2020-06-12T02:38:30.151 INFO:tasks.rados.rados.0.smithi107.stdout:6692: writing smithi10729407-7 from 1064960 to 1531904 tid 1 2020-06-12T02:38:30.151 INFO:tasks.rados.rados.0.smithi107.stdout:6692: writing smithi10729407-7 from 1531904 to 2121728 tid 2 2020-06-12T02:38:30.151 INFO:tasks.rados.rados.0.smithi107.stdout:6692: writing smithi10729407-7 from 2121728 to 2637824 tid 3 2020-06-12T02:38:30.152 INFO:tasks.rados.rados.0.smithi107.stdout:6692: writing smithi10729407-7 from 2637824 to 2736128 tid 4 2020-06-12T02:38:30.152 INFO:tasks.rados.rados.0.smithi107.stdout:6693: delete oid 8 current snap is 0 2020-06-12T02:38:30.152 INFO:tasks.rados.rados.0.smithi107.stdout:6688: finishing write tid 6 to smithi10729407-16 2020-06-12T02:38:30.152 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1191 (ObjNum 6687 snap 0 seq_num 6687) dirty exists 2020-06-12T02:38:30.153 INFO:tasks.rados.rados.0.smithi107.stdout:6688: left oid 16 (ObjNum 6687 snap 0 seq_num 6687) 2020-06-12T02:38:30.153 INFO:tasks.rados.rados.0.smithi107.stdout:6692: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:30.153 INFO:tasks.rados.rados.0.smithi107.stdout:6688: done (3 left) 2020-06-12T02:38:30.153 INFO:tasks.rados.rados.0.smithi107.stdout:6693: done (2 left) 2020-06-12T02:38:30.153 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:30.154 INFO:tasks.rados.rados.0.smithi107.stdout:6694: seq_num 6693 ranges {0=499712,499712=761856,1261568=442368} 2020-06-12T02:38:30.154 INFO:tasks.rados.rados.0.smithi107.stdout:6694: writing smithi10729407-8 from 0 to 499712 tid 1 2020-06-12T02:38:30.158 INFO:tasks.rados.rados.0.smithi107.stdout:6694: writing smithi10729407-8 from 499712 to 1261568 tid 2 2020-06-12T02:38:30.161 INFO:tasks.rados.rados.0.smithi107.stdout:6694: writing smithi10729407-8 from 1261568 to 1703936 tid 3 2020-06-12T02:38:30.161 INFO:tasks.rados.rados.0.smithi107.stdout:6695: delete oid 11 current snap is 0 2020-06-12T02:38:30.161 INFO:tasks.rados.rados.0.smithi107.stdout:6692: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:30.161 INFO:tasks.rados.rados.0.smithi107.stdout:6692: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:30.162 INFO:tasks.rados.rados.0.smithi107.stdout:6692: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:30.162 INFO:tasks.rados.rados.0.smithi107.stdout:6692: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:30.163 INFO:tasks.rados.rados.0.smithi107.stdout:6692: finishing write tid 7 to smithi10729407-7 2020-06-12T02:38:30.163 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1262 (ObjNum 6691 snap 0 seq_num 6691) dirty exists 2020-06-12T02:38:30.164 INFO:tasks.rados.rados.0.smithi107.stdout:6692: left oid 7 (ObjNum 6691 snap 0 seq_num 6691) 2020-06-12T02:38:30.164 INFO:tasks.rados.rados.0.smithi107.stdout:6692: done (3 left) 2020-06-12T02:38:30.164 INFO:tasks.rados.rados.0.smithi107.stdout:6695: done (2 left) 2020-06-12T02:38:30.164 INFO:tasks.rados.rados.0.smithi107.stdout:6696: delete oid 5 current snap is 0 2020-06-12T02:38:30.168 INFO:tasks.rados.rados.0.smithi107.stdout:6696: done (2 left) 2020-06-12T02:38:30.169 INFO:tasks.rados.rados.0.smithi107.stdout:6697: delete oid 20 current snap is 0 2020-06-12T02:38:30.174 INFO:tasks.rados.rados.0.smithi107.stdout:6694: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:30.175 INFO:tasks.rados.rados.0.smithi107.stdout:6697: done (2 left) 2020-06-12T02:38:30.175 INFO:tasks.rados.rados.0.smithi107.stdout:6698: delete oid 4 current snap is 0 2020-06-12T02:38:30.176 INFO:tasks.rados.rados.0.smithi107.stdout:6694: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:30.178 INFO:tasks.rados.rados.0.smithi107.stdout:6694: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:30.179 INFO:tasks.rados.rados.0.smithi107.stdout:6698: done (2 left) 2020-06-12T02:38:30.179 INFO:tasks.rados.rados.0.smithi107.stdout:6699: delete oid 2 current snap is 0 2020-06-12T02:38:30.179 INFO:tasks.rados.rados.0.smithi107.stdout:6694: finishing write tid 5 to smithi10729407-8 2020-06-12T02:38:30.180 INFO:tasks.rados.rados.0.smithi107.stdout:6694: finishing write tid 6 to smithi10729407-8 2020-06-12T02:38:30.181 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1253 (ObjNum 6693 snap 0 seq_num 6693) dirty exists 2020-06-12T02:38:30.181 INFO:tasks.rados.rados.0.smithi107.stdout:6694: left oid 8 (ObjNum 6693 snap 0 seq_num 6693) 2020-06-12T02:38:30.184 INFO:tasks.rados.rados.0.smithi107.stdout:6694: done (2 left) 2020-06-12T02:38:30.184 INFO:tasks.rados.rados.0.smithi107.stdout:6699: done (1 left) 2020-06-12T02:38:30.184 INFO:tasks.rados.rados.0.smithi107.stdout:6700: delete oid 14 current snap is 0 2020-06-12T02:38:30.185 INFO:tasks.rados.rados.0.smithi107.stdout:6700: done (1 left) 2020-06-12T02:38:30.186 INFO:tasks.rados.rados.0.smithi107.stdout:6701: delete oid 5 current snap is 0 2020-06-12T02:38:30.209 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:30.204+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:30.210 INFO:tasks.rados.rados.0.smithi107.stdout:6701: done (1 left) 2020-06-12T02:38:30.210 INFO:tasks.rados.rados.0.smithi107.stdout:6702: delete oid 13 current snap is 0 2020-06-12T02:38:30.213 INFO:tasks.rados.rados.0.smithi107.stdout:6702: done (1 left) 2020-06-12T02:38:30.213 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:38:30.213 INFO:tasks.rados.rados.0.smithi107.stdout:6703: seq_num 6702 ranges {0=655360,655360=565248,1220608=638976} 2020-06-12T02:38:30.216 INFO:tasks.rados.rados.0.smithi107.stdout:6703: writing smithi10729407-1 from 0 to 655360 tid 1 2020-06-12T02:38:30.223 INFO:tasks.rados.rados.0.smithi107.stdout:6703: writing smithi10729407-1 from 655360 to 1220608 tid 2 2020-06-12T02:38:30.224 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:30.224 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:30.225 INFO:tasks.rados.rados.0.smithi107.stdout:6703: writing smithi10729407-1 from 1220608 to 1859584 tid 3 2020-06-12T02:38:30.225 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:30.225 INFO:tasks.rados.rados.0.smithi107.stdout:6704: seq_num 6703 ranges {0=671744,671744=450560,1122304=458752,1581056=294912} 2020-06-12T02:38:30.228 INFO:tasks.rados.rados.0.smithi107.stdout:6704: writing smithi10729407-4 from 0 to 671744 tid 1 2020-06-12T02:38:30.231 INFO:tasks.rados.rados.0.smithi107.stdout:6704: writing smithi10729407-4 from 671744 to 1122304 tid 2 2020-06-12T02:38:30.233 INFO:tasks.rados.rados.0.smithi107.stdout:6704: writing smithi10729407-4 from 1122304 to 1581056 tid 3 2020-06-12T02:38:30.235 INFO:tasks.rados.rados.0.smithi107.stdout:6704: writing smithi10729407-4 from 1581056 to 1875968 tid 4 2020-06-12T02:38:30.235 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:30.236 INFO:tasks.rados.rados.0.smithi107.stdout:6705: seq_num 6704 ranges {0=688128,688128=737280,1425408=466944} 2020-06-12T02:38:30.237 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-06-12T02:38:30.239 INFO:tasks.rados.rados.0.smithi107.stdout:6705: writing smithi10729407-18 from 0 to 688128 tid 1 2020-06-12T02:38:30.244 INFO:tasks.rados.rados.0.smithi107.stdout:6705: writing smithi10729407-18 from 688128 to 1425408 tid 2 2020-06-12T02:38:30.247 INFO:tasks.rados.rados.0.smithi107.stdout:6705: writing smithi10729407-18 from 1425408 to 1892352 tid 3 2020-06-12T02:38:30.248 INFO:tasks.rados.rados.0.smithi107.stdout:6706: delete oid 13 current snap is 0 2020-06-12T02:38:30.248 INFO:tasks.rados.rados.0.smithi107.stdout:6703: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:30.248 INFO:tasks.rados.rados.0.smithi107.stdout:6703: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:30.248 INFO:tasks.rados.rados.0.smithi107.stdout:6703: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:30.248 INFO:tasks.rados.rados.0.smithi107.stdout:6703: finishing write tid 5 to smithi10729407-1 2020-06-12T02:38:30.249 INFO:tasks.rados.rados.0.smithi107.stdout:6704: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:30.249 INFO:tasks.rados.rados.0.smithi107.stdout:6703: finishing write tid 6 to smithi10729407-1 2020-06-12T02:38:30.249 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1217 (ObjNum 6702 snap 0 seq_num 6702) dirty exists 2020-06-12T02:38:30.249 INFO:tasks.rados.rados.0.smithi107.stdout:6703: left oid 1 (ObjNum 6702 snap 0 seq_num 6702) 2020-06-12T02:38:30.249 INFO:tasks.rados.rados.0.smithi107.stdout:6704: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:30.250 INFO:tasks.rados.rados.0.smithi107.stdout:6705: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:30.250 INFO:tasks.rados.rados.0.smithi107.stdout:6703: done (4 left) 2020-06-12T02:38:30.251 INFO:tasks.rados.rados.0.smithi107.stdout:6706: done (3 left) 2020-06-12T02:38:30.251 INFO:tasks.rados.rados.0.smithi107.stdout:6707: delete oid 12 current snap is 0 2020-06-12T02:38:30.251 INFO:tasks.rados.rados.0.smithi107.stdout:6704: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:30.252 INFO:tasks.rados.rados.0.smithi107.stdout:6704: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:30.252 INFO:tasks.rados.rados.0.smithi107.stdout:6704: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:30.254 INFO:tasks.rados.rados.0.smithi107.stdout:6704: finishing write tid 7 to smithi10729407-4 2020-06-12T02:38:30.254 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1265 (ObjNum 6703 snap 0 seq_num 6703) dirty exists 2020-06-12T02:38:30.254 INFO:tasks.rados.rados.0.smithi107.stdout:6704: left oid 4 (ObjNum 6703 snap 0 seq_num 6703) 2020-06-12T02:38:30.254 INFO:tasks.rados.rados.0.smithi107.stdout:6704: done (3 left) 2020-06-12T02:38:30.255 INFO:tasks.rados.rados.0.smithi107.stdout:6707: done (2 left) 2020-06-12T02:38:30.255 INFO:tasks.rados.rados.0.smithi107.stdout:6708: delete oid 17 current snap is 0 2020-06-12T02:38:30.256 INFO:tasks.rados.rados.0.smithi107.stdout:6705: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:30.256 INFO:tasks.rados.rados.0.smithi107.stdout:6708: done (2 left) 2020-06-12T02:38:30.257 INFO:tasks.rados.rados.0.smithi107.stdout:6709: delete oid 20 current snap is 0 2020-06-12T02:38:30.257 INFO:tasks.rados.rados.0.smithi107.stdout:6705: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:30.257 INFO:tasks.rados.rados.0.smithi107.stdout:6705: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:30.259 INFO:tasks.rados.rados.0.smithi107.stdout:6705: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:30.259 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1236 (ObjNum 6704 snap 0 seq_num 6704) dirty exists 2020-06-12T02:38:30.259 INFO:tasks.rados.rados.0.smithi107.stdout:6705: left oid 18 (ObjNum 6704 snap 0 seq_num 6704) 2020-06-12T02:38:30.260 INFO:tasks.rados.rados.0.smithi107.stdout:6705: done (2 left) 2020-06-12T02:38:30.260 INFO:tasks.rados.rados.0.smithi107.stdout:6709: done (1 left) 2020-06-12T02:38:30.260 INFO:tasks.rados.rados.0.smithi107.stdout:6710: delete oid 8 current snap is 0 2020-06-12T02:38:30.262 INFO:tasks.rados.rados.0.smithi107.stdout:6710: done (1 left) 2020-06-12T02:38:30.262 INFO:tasks.rados.rados.0.smithi107.stdout:6711: delete oid 4 current snap is 0 2020-06-12T02:38:30.266 INFO:tasks.rados.rados.0.smithi107.stdout:6711: done (1 left) 2020-06-12T02:38:30.266 INFO:tasks.rados.rados.0.smithi107.stdout:6712: delete oid 11 current snap is 0 2020-06-12T02:38:30.269 INFO:tasks.rados.rados.0.smithi107.stdout:6712: done (1 left) 2020-06-12T02:38:30.269 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:30.269 INFO:tasks.rados.rados.0.smithi107.stdout:6713: seq_num 6712 ranges {0=417792,417792=663552,1081344=630784,1712128=311296} 2020-06-12T02:38:30.271 INFO:tasks.rados.rados.0.smithi107.stdout:6713: writing smithi10729407-4 from 0 to 417792 tid 1 2020-06-12T02:38:30.275 INFO:tasks.rados.rados.0.smithi107.stdout:6713: writing smithi10729407-4 from 417792 to 1081344 tid 2 2020-06-12T02:38:30.279 INFO:tasks.rados.rados.0.smithi107.stdout:6713: writing smithi10729407-4 from 1081344 to 1712128 tid 3 2020-06-12T02:38:30.281 INFO:tasks.rados.rados.0.smithi107.stdout:6713: writing smithi10729407-4 from 1712128 to 2023424 tid 4 2020-06-12T02:38:30.281 INFO:tasks.rados.rados.0.smithi107.stdout:6714: delete oid 14 current snap is 0 2020-06-12T02:38:30.281 INFO:tasks.rados.rados.0.smithi107.stdout:6713: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:30.283 INFO:tasks.rados.rados.0.smithi107.stdout:6714: done (2 left) 2020-06-12T02:38:30.284 INFO:tasks.rados.rados.0.smithi107.stdout:6715: delete oid 15 current snap is 0 2020-06-12T02:38:30.287 INFO:tasks.rados.rados.0.smithi107.stdout:6715: done (2 left) 2020-06-12T02:38:30.288 INFO:tasks.rados.rados.0.smithi107.stdout:6716: delete oid 8 current snap is 0 2020-06-12T02:38:30.290 INFO:tasks.rados.rados.0.smithi107.stdout:6713: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:30.292 INFO:tasks.rados.rados.0.smithi107.stdout:6713: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:30.294 INFO:tasks.rados.rados.0.smithi107.stdout:6716: done (2 left) 2020-06-12T02:38:30.294 INFO:tasks.rados.rados.0.smithi107.stdout:6717: delete oid 16 current snap is 0 2020-06-12T02:38:30.295 INFO:tasks.rados.rados.0.smithi107.stdout:6713: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:30.295 INFO:tasks.rados.rados.0.smithi107.stdout:6713: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:30.295 INFO:tasks.rados.rados.0.smithi107.stdout:6713: finishing write tid 7 to smithi10729407-4 2020-06-12T02:38:30.296 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1272 (ObjNum 6712 snap 0 seq_num 6712) dirty exists 2020-06-12T02:38:30.296 INFO:tasks.rados.rados.0.smithi107.stdout:6713: left oid 4 (ObjNum 6712 snap 0 seq_num 6712) 2020-06-12T02:38:30.296 INFO:tasks.rados.rados.0.smithi107.stdout:6713: done (2 left) 2020-06-12T02:38:30.297 INFO:tasks.rados.rados.0.smithi107.stdout:6717: done (1 left) 2020-06-12T02:38:30.297 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:30.297 INFO:tasks.rados.rados.0.smithi107.stdout:6718: seq_num 6717 ranges {0=499712,499712=532480,1032192=417792,1449984=450560,1900544=204800} 2020-06-12T02:38:30.299 INFO:tasks.rados.rados.0.smithi107.stdout:6718: writing smithi10729407-9 from 0 to 499712 tid 1 2020-06-12T02:38:30.302 INFO:tasks.rados.rados.0.smithi107.stdout:6718: writing smithi10729407-9 from 499712 to 1032192 tid 2 2020-06-12T02:38:30.305 INFO:tasks.rados.rados.0.smithi107.stdout:6718: writing smithi10729407-9 from 1032192 to 1449984 tid 3 2020-06-12T02:38:30.308 INFO:tasks.rados.rados.0.smithi107.stdout:6718: writing smithi10729407-9 from 1449984 to 1900544 tid 4 2020-06-12T02:38:30.308 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:30.305+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:30.309 INFO:tasks.rados.rados.0.smithi107.stdout:6718: writing smithi10729407-9 from 1900544 to 2105344 tid 5 2020-06-12T02:38:30.309 INFO:tasks.rados.rados.0.smithi107.stdout:6719: delete oid 6 current snap is 0 2020-06-12T02:38:30.310 INFO:tasks.rados.rados.0.smithi107.stdout:6718: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:30.312 INFO:tasks.rados.rados.0.smithi107.stdout:6719: done (2 left) 2020-06-12T02:38:30.313 INFO:tasks.rados.rados.0.smithi107.stdout:6720: delete oid 13 current snap is 0 2020-06-12T02:38:30.315 INFO:tasks.rados.rados.0.smithi107.stdout:6720: done (2 left) 2020-06-12T02:38:30.316 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:30.316 INFO:tasks.rados.rados.0.smithi107.stdout:6721: seq_num 6720 ranges {0=548864,548864=581632,1130496=589824,1720320=434176} 2020-06-12T02:38:30.319 INFO:tasks.rados.rados.0.smithi107.stdout:6721: writing smithi10729407-14 from 0 to 548864 tid 1 2020-06-12T02:38:30.323 INFO:tasks.rados.rados.0.smithi107.stdout:6721: writing smithi10729407-14 from 548864 to 1130496 tid 2 2020-06-12T02:38:30.329 INFO:tasks.rados.rados.0.smithi107.stdout:6721: writing smithi10729407-14 from 1130496 to 1720320 tid 3 2020-06-12T02:38:30.329 INFO:tasks.rados.rados.0.smithi107.stdout:6721: writing smithi10729407-14 from 1720320 to 2154496 tid 4 2020-06-12T02:38:30.330 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:38:30.330 INFO:tasks.rados.rados.0.smithi107.stdout:6722: seq_num 6721 ranges {2736128=573440,3309568=516096,3825664=802816,4628480=286720} 2020-06-12T02:38:30.348 INFO:tasks.rados.rados.0.smithi107.stdout:6722: writing smithi10729407-7 from 2736128 to 3309568 tid 1 2020-06-12T02:38:30.351 INFO:tasks.rados.rados.0.smithi107.stdout:6722: writing smithi10729407-7 from 3309568 to 3825664 tid 2 2020-06-12T02:38:30.356 INFO:tasks.rados.rados.0.smithi107.stdout:6722: writing smithi10729407-7 from 3825664 to 4628480 tid 3 2020-06-12T02:38:30.358 INFO:tasks.rados.rados.0.smithi107.stdout:6722: writing smithi10729407-7 from 4628480 to 4915200 tid 4 2020-06-12T02:38:30.358 INFO:tasks.rados.rados.0.smithi107.stdout:6723: delete oid 12 current snap is 0 2020-06-12T02:38:30.358 INFO:tasks.rados.rados.0.smithi107.stdout:6718: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:30.359 INFO:tasks.rados.rados.0.smithi107.stdout:6718: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:30.359 INFO:tasks.rados.rados.0.smithi107.stdout:6718: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:30.359 INFO:tasks.rados.rados.0.smithi107.stdout:6718: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:30.359 INFO:tasks.rados.rados.0.smithi107.stdout:6718: finishing write tid 7 to smithi10729407-9 2020-06-12T02:38:30.359 INFO:tasks.rados.rados.0.smithi107.stdout:6718: finishing write tid 8 to smithi10729407-9 2020-06-12T02:38:30.359 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1242 (ObjNum 6717 snap 0 seq_num 6717) dirty exists 2020-06-12T02:38:30.360 INFO:tasks.rados.rados.0.smithi107.stdout:6718: left oid 9 (ObjNum 6717 snap 0 seq_num 6717) 2020-06-12T02:38:30.360 INFO:tasks.rados.rados.0.smithi107.stdout:6721: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:30.360 INFO:tasks.rados.rados.0.smithi107.stdout:6721: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:30.360 INFO:tasks.rados.rados.0.smithi107.stdout:6721: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:30.360 INFO:tasks.rados.rados.0.smithi107.stdout:6721: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:30.361 INFO:tasks.rados.rados.0.smithi107.stdout:6721: finishing write tid 6 to smithi10729407-14 2020-06-12T02:38:30.361 INFO:tasks.rados.rados.0.smithi107.stdout:6721: finishing write tid 7 to smithi10729407-14 2020-06-12T02:38:30.361 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1192 (ObjNum 6720 snap 0 seq_num 6720) dirty exists 2020-06-12T02:38:30.361 INFO:tasks.rados.rados.0.smithi107.stdout:6721: left oid 14 (ObjNum 6720 snap 0 seq_num 6720) 2020-06-12T02:38:30.362 INFO:tasks.rados.rados.0.smithi107.stdout:6722: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:30.362 INFO:tasks.rados.rados.0.smithi107.stdout:6718: done (4 left) 2020-06-12T02:38:30.362 INFO:tasks.rados.rados.0.smithi107.stdout:6721: done (3 left) 2020-06-12T02:38:30.362 INFO:tasks.rados.rados.0.smithi107.stdout:6723: done (2 left) 2020-06-12T02:38:30.363 INFO:tasks.rados.rados.0.smithi107.stdout:6724: delete oid 13 current snap is 0 2020-06-12T02:38:30.367 INFO:tasks.rados.rados.0.smithi107.stdout:6722: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:30.370 INFO:tasks.rados.rados.0.smithi107.stdout:6722: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:30.370 INFO:tasks.rados.rados.0.smithi107.stdout:6722: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:30.371 INFO:tasks.rados.rados.0.smithi107.stdout:6722: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:30.372 INFO:tasks.rados.rados.0.smithi107.stdout:6724: done (2 left) 2020-06-12T02:38:30.372 INFO:tasks.rados.rados.0.smithi107.stdout:6725: delete oid 14 current snap is 0 2020-06-12T02:38:30.372 INFO:tasks.rados.rados.0.smithi107.stdout:6722: finishing write tid 7 to smithi10729407-7 2020-06-12T02:38:30.373 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1270 (ObjNum 6721 snap 0 seq_num 6721) dirty exists 2020-06-12T02:38:30.373 INFO:tasks.rados.rados.0.smithi107.stdout:6722: left oid 7 (ObjNum 6721 snap 0 seq_num 6721) 2020-06-12T02:38:30.374 INFO:tasks.rados.rados.0.smithi107.stdout:6722: done (2 left) 2020-06-12T02:38:30.374 INFO:tasks.rados.rados.0.smithi107.stdout:6725: done (1 left) 2020-06-12T02:38:30.374 INFO:tasks.rados.rados.0.smithi107.stdout:6726: delete oid 1 current snap is 0 2020-06-12T02:38:30.377 INFO:tasks.rados.rados.0.smithi107.stdout:6726: done (1 left) 2020-06-12T02:38:30.377 INFO:tasks.rados.rados.0.smithi107.stdout:6727: delete oid 18 current snap is 0 2020-06-12T02:38:30.380 INFO:tasks.rados.rados.0.smithi107.stdout:6727: done (1 left) 2020-06-12T02:38:30.380 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:30.380 INFO:tasks.rados.rados.0.smithi107.stdout:6728: seq_num 6727 ranges {0=671744,671744=622592,1294336=745472,2039808=237568} 2020-06-12T02:38:30.384 INFO:tasks.rados.rados.0.smithi107.stdout:6728: writing smithi10729407-17 from 0 to 671744 tid 1 2020-06-12T02:38:30.386 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:30.386 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:30.388 INFO:tasks.rados.rados.0.smithi107.stdout:6728: writing smithi10729407-17 from 671744 to 1294336 tid 2 2020-06-12T02:38:30.399 INFO:tasks.rados.rados.0.smithi107.stdout:6728: writing smithi10729407-17 from 1294336 to 2039808 tid 3 2020-06-12T02:38:30.399 INFO:tasks.rados.rados.0.smithi107.stdout:6728: writing smithi10729407-17 from 2039808 to 2277376 tid 4 2020-06-12T02:38:30.399 INFO:tasks.rados.rados.0.smithi107.stdout:6729: delete oid 12 current snap is 0 2020-06-12T02:38:30.399 INFO:tasks.rados.rados.0.smithi107.stdout:6728: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:30.399 INFO:tasks.rados.rados.0.smithi107.stdout:6729: done (2 left) 2020-06-12T02:38:30.400 INFO:tasks.rados.rados.0.smithi107.stdout:6730: delete oid 6 current snap is 0 2020-06-12T02:38:30.400 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-06-12T02:38:30.401 INFO:tasks.rados.rados.0.smithi107.stdout:6730: done (2 left) 2020-06-12T02:38:30.401 INFO:tasks.rados.rados.0.smithi107.stdout:6731: delete oid 5 current snap is 0 2020-06-12T02:38:30.403 INFO:tasks.rados.rados.0.smithi107.stdout:6731: done (2 left) 2020-06-12T02:38:30.403 INFO:tasks.rados.rados.0.smithi107.stdout:6732: delete oid 11 current snap is 0 2020-06-12T02:38:30.403 INFO:tasks.rados.rados.0.smithi107.stdout:6728: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:30.404 INFO:tasks.rados.rados.0.smithi107.stdout:6728: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:30.404 INFO:tasks.rados.rados.0.smithi107.stdout:6728: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:30.405 INFO:tasks.rados.rados.0.smithi107.stdout:6728: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:30.405 INFO:tasks.rados.rados.0.smithi107.stdout:6732: done (2 left) 2020-06-12T02:38:30.405 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:38:30.406 INFO:tasks.rados.rados.0.smithi107.stdout:6733: seq_num 6732 ranges {0=753664,753664=491520,1245184=532480,1777664=581632} 2020-06-12T02:38:30.409 INFO:tasks.rados.rados.0.smithi107.stdout:6733: writing smithi10729407-20 from 0 to 753664 tid 1 2020-06-12T02:38:30.410 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:30.407+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:30.411 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:38:30.418 INFO:tasks.rados.rados.0.smithi107.stdout:6733: writing smithi10729407-20 from 753664 to 1245184 tid 2 2020-06-12T02:38:30.418 INFO:tasks.rados.rados.0.smithi107.stdout:6733: writing smithi10729407-20 from 1245184 to 1777664 tid 3 2020-06-12T02:38:30.418 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2020-06-12T02:38:30.421 INFO:tasks.rados.rados.0.smithi107.stdout:6733: writing smithi10729407-20 from 1777664 to 2359296 tid 4 2020-06-12T02:38:30.421 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:30.421 INFO:tasks.rados.rados.0.smithi107.stdout:6734: seq_num 6733 ranges {0=770048,770048=778240,1548288=737280,2285568=90112} 2020-06-12T02:38:30.426 INFO:tasks.rados.rados.0.smithi107.stdout:6734: writing smithi10729407-8 from 0 to 770048 tid 1 2020-06-12T02:38:30.428 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":98,"stamp":"2020-06-12T02:38:29.607279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5267456,"num_objects":6,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3813,"num_read_kb":2064,"num_write":8434,"num_write_kb":2401394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":13869056,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11143,"ondisk_log_size":11143,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":66,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8411080,"kb_used_data":22408,"kb_used_omap":25,"kb_used_meta":8388582,"kb_avail":746563640,"statfs":{"total":773094113280,"available":764481167360,"internally_reserved":8589934592,"allocated":22945792,"data_stored":20147716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26157,"internal_metadata":8589908435},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,2,0,0,0,0,0,0,0,0,0,6],"upper_bound":16384},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2097152,"num_objects":-1,"num_object_clones":0,"num_object_copies":-3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":1023,"num_read_kb":557,"num_write":2228,"num_write_kb":632632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001265"},"pg_stats":[{"pgid":"3.f","version":"58'1204","reported_seq":"1476","reported_epoch":"58","state":"active+clean","last_fresh":"2020-06-12T02:38:28.347714+0000","last_change":"2020-06-12T02:38:11.587385+0000","last_active":"2020-06-12T02:38:28.347714+0000","last_peered":"2020-06-12T02:38:28.347714+0000","last_clean":"2020-06-12T02:38:28.347714+0000","last_became_active":"2020-06-12T02:37:57.566633+0000","last_became_peered":"2020-06-12T02:37:57.566633+0000","last_unstale":"2020-06-12T02:38:28.347714+0000","last_undegraded":"2020-06-12T02:38:28.347714+0000","last_fullsized":"2020-06-12T02:38:28.347714+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":1204,"ondisk_log_size":1204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":413,"num_read_kb":213,"num_write":872,"num_write_kb":243656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1531904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"58'648","reported_seq":"825","reported_epoch":"58","state":"active+clean","last_fresh":"2020-06-12T02:38:28.110068+0000","last_change":"2020-06-12T02:38:11.572385+0000","last_active":"2020-06-12T02:38:28.110068+0000","last_peered":"2020-06-12T02:38:28.110068+0000","last_clean":"2020-06-12T02:38:28.110068+0000","last_became_active":"2020-06-12T02:37:57.563869+0000","last_became_peered":"2020-06-12T02:37:57.563869+0000","last_unstale":"2020-06-12T02:38:28.110068+0000","last_undegraded":"2020-06-12T02:38:28.110068+0000","last_fullsized":"2020-06-12T02:38:28.110068+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":648,"ondisk_log_size":648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":215,"num_read_kb":116,"num_write":497,"num_write_kb":141016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1466368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"58'1172","reported_seq":"1457","reported_epoch":"58","state":"active+clean","last_fresh":"2020-06-12T02:38:28.225396+0000","last_change":"2020-06-12T02:38:11.587290+0000","last_active":"2020-06-12T02:38:28.225396+0000","last_peered":"2020-06-12T02:38:28.225396+0000","last_clean":"2020-06-12T02:38:28.225396+0000","last_became_active":"2020-06-12T02:37:57.564464+0000","last_became_peered":"2020-06-12T02:37:57.564464+0000","last_unstale":"2020-06-12T02:38:28.225396+0000","last_undegraded":"2020-06-12T02:38:28.225396+0000","last_fullsized":"2020-06-12T02:38:28.225396+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":1172,"ondisk_log_size":1172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":395,"num_read_kb":214,"num_write":885,"num_write_kb":251584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3710976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"38","reported_epoch":"56","state":"active+clean","last_fresh":"2020-06-12T02:38:21.386345+0000","last_change":"2020-06-12T02:37:42.384656+0000","last_active":"2020-06-12T02:38:21.386345+0000","last_peered":"2020-06-12T02:38:21.386345+0000","last_clean":"2020-06-12T02:38:21.386345+0000","last_became_active":"2020-06-12T02:37:42.384363+0000","last_became_peered":"2020-06-12T02:37:42.384363+0000","last_unstale":"2020-06-12T02:38:21.386345+0000","last_undegraded":"2020-06-12T02:38:21.386345+0000","last_fullsized":"2020-06-12T02:38:21.386345+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"57'1072","reported_seq":"1316","reported_epoch":"57","state":"active+clean","last_fresh":"2020-06-12T02:38:26.697832+0000","last_change":"2020-06-12T02:37:27.902643+0000","last_active":"2020-06-12T02:38:26.697832+0000","last_peered":"2020-06-12T02:38:26.697832+0000","last_clean":"2020-06-12T02:38:26.697832+0000","last_became_active":"2020-06-12T02:37:27.902204+0000","last_became_peered":"2020-06-12T02:37:27.902204+0000","last_unstale":"2020-06-12T02:38:26.697832+0000","last_undegraded":"2020-06-12T02:38:26.697832+0000","last_fullsized":"2020-06-12T02:38:26.697832+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":1072,"ondisk_log_size":1072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2269184,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":364,"num_read_kb":201,"num_write":812,"num_write_kb":230304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"57'1127","reported_seq":"1412","reported_epoch":"57","state":"active+clean","last_fresh":"2020-06-12T02:38:26.692474+0000","last_change":"2020-06-12T02:38:16.369345+0000","last_active":"2020-06-12T02:38:26.692474+0000","last_peered":"2020-06-12T02:38:26.692474+0000","last_clean":"2020-06-12T02:38:26.692474+0000","last_became_active":"2020-06-12T02:38:16.367448+0000","last_became_peered":"2020-06-12T02:38:16.367448+0000","last_unstale":"2020-06-12T02:38:26.692474+0000","last_undegraded":"2020-06-12T02:38:26.692474+0000","last_fullsized":"2020-06-12T02:38:26.692474+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":1127,"ondisk_log_size":1127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1187840,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":402,"num_read_kb":220,"num_write":876,"num_write_kb":252880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"57'1092","reported_seq":"1364","reported_epoch":"57","state":"active+clean","last_fresh":"2020-06-12T02:38:26.651580+0000","last_change":"2020-06-12T02:37:57.567142+0000","last_active":"2020-06-12T02:38:26.651580+0000","last_peered":"2020-06-12T02:38:26.651580+0000","last_clean":"2020-06-12T02:38:26.651580+0000","last_became_active":"2020-06-12T02:37:57.566462+0000","last_became_peered":"2020-06-12T02:37:57.566462+0000","last_unstale":"2020-06-12T02:38:26.651580+0000","last_undegraded":"2020-06-12T02:38:26.651580+0000","last_fullsized":"2020-06-12T02:38:26.651580+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":1092,"ondisk_log_size":1092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1171456,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":372,"num_read_kb":208,"num_write":807,"num_write_kb":223672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"57'579","reported_seq":"764","reported_epoch":"57","state":"active+clean","last_fresh":"2020-06-12T02:38:26.547974+0000","last_change":"2020-06-12T02:38:16.372264+0000","last_active":"2020-06-12T02:38:26.547974+0000","last_peered":"2020-06-12T02:38:26.547974+0000","last_clean":"2020-06-12T02:38:26.547974+0000","last_became_active":"2020-06-12T02:38:16.371502+0000","last_became_peered":"2020-06-12T02:38:16.371502+0000","last_unstale":"2020-06-12T02:38:26.547974+0000","last_undegraded":"2020-06-12T02:38:26.547974+0000","last_fullsized":"2020-06-12T02:38:26.547974+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":579,"ondisk_log_size":579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":188,"num_read_kb":100,"num_write":436,"num_write_kb":127440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2793472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"17","reported_epoch":"56","state":"active+clean","last_fresh":"2020-06-12T02:38:21.404797+0000","last_change":"2020-06-12T02:38:13.661966+0000","last_active":"2020-06-12T02:38:21.404797+0000","last_peered":"2020-06-12T02:38:21.404797+0000","last_clean":"2020-06-12T02:38:21.404797+0000","last_became_active":"2020-06-12T02:38:13.661679+0000","last_became_peered":"2020-06-12T02:38:13.661679+0000","last_unstale":"2020-06-12T02:38:21.404797+0000","last_undegraded":"2020-06-12T02:38:21.404797+0000","last_fullsized":"2020-06-12T02:38:21.404797+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"57'1080","reported_seq":"1322","reported_epoch":"57","state":"active+clean","last_fresh":"2020-06-12T02:38:26.578938+0000","last_change":"2020-06-12T02:37:42.379294+0000","last_active":"2020-06-12T02:38:26.578938+0000","last_peered":"2020-06-12T02:38:26.578938+0000","last_clean":"2020-06-12T02:38:26.578938+0000","last_became_active":"2020-06-12T02:37:42.378796+0000","last_became_peered":"2020-06-12T02:37:42.378796+0000","last_unstale":"2020-06-12T02:38:26.578938+0000","last_undegraded":"2020-06-12T02:38:26.578938+0000","last_fullsized":"2020-06-12T02:38:26.578938+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":1080,"ondisk_log_size":1080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":362,"num_read_kb":198,"num_write":806,"num_write_kb":233216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"15","reported_epoch":"56","state":"active+clean","last_fresh":"2020-06-12T02:38:21.394272+0000","last_change":"2020-06-12T02:38:13.664800+0000","last_active":"2020-06-12T02:38:21.394272+0000","last_peered":"2020-06-12T02:38:21.394272+0000","last_clean":"2020-06-12T02:38:21.394272+0000","last_became_active":"2020-06-12T02:38:13.664384+0000","last_became_peered":"2020-06-12T02:38:13.664384+0000","last_unstale":"2020-06-12T02:38:21.394272+0000","last_undegraded":"2020-06-12T02:38:21.394272+0000","last_fullsized":"2020-06-12T02:38:21.394272+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"57'1164","reported_seq":"1443","reported_epoch":"57","state":"active+clean","last_fresh":"2020-06-12T02:38:26.646126+0000","last_change":"2020-06-12T02:38:16.372259+0000","last_active":"2020-06-12T02:38:26.646126+0000","last_peered":"2020-06-12T02:38:26.646126+0000","last_clean":"2020-06-12T02:38:26.646126+0000","last_became_active":"2020-06-12T02:38:16.371577+0000","last_became_peered":"2020-06-12T02:38:16.371577+0000","last_unstale":"2020-06-12T02:38:26.646126+0000","last_undegraded":"2020-06-12T02:38:26.646126+0000","last_fullsized":"2020-06-12T02:38:26.646126+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":1164,"ondisk_log_size":1164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":638976,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":404,"num_read_kb":212,"num_write":898,"num_write_kb":262712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1572864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"50","reported_epoch":"56","state":"active+clean","last_fresh":"2020-06-12T02:38:21.386495+0000","last_change":"2020-06-12T02:38:21.039695+0000","last_active":"2020-06-12T02:38:21.386495+0000","last_peered":"2020-06-12T02:38:21.386495+0000","last_clean":"2020-06-12T02:38:21.386495+0000","last_became_active":"2020-06-12T02:37:16.764958+0000","last_became_peered":"2020-06-12T02:37:16.764958+0000","last_unstale":"2020-06-12T02:38:21.386495+0000","last_undegraded":"2020-06-12T02:38:21.386495+0000","last_fullsized":"2020-06-12T02:38:21.386495+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:38:21.039625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:38:21.039625+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"52'434","reported_seq":"580","reported_epoch":"56","state":"active+clean+wait","last_fresh":"2020-06-12T02:38:21.404714+0000","last_change":"2020-06-12T02:38:16.372481+0000","last_active":"2020-06-12T02:38:21.404714+0000","last_peered":"2020-06-12T02:38:21.404714+0000","last_clean":"2020-06-12T02:38:21.404714+0000","last_became_active":"2020-06-12T02:38:16.371755+0000","last_became_peered":"2020-06-12T02:38:16.371755+0000","last_unstale":"2020-06-12T02:38:21.404714+0000","last_undegraded":"2020-06-12T02:38:21.404714+0000","last_fullsized":"2020-06-12T02:38:21.404714+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":434,"ondisk_log_size":434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":157,"num_read_kb":83,"num_write":342,"num_write_kb":97328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"40","reported_epoch":"56","state":"active+clean","last_fresh":"2020-06-12T02:38:22.079524+0000","last_change":"2020-06-12T02:38:22.079524+0000","last_active":"2020-06-12T02:38:22.079524+0000","last_peered":"2020-06-12T02:38:22.079524+0000","last_clean":"2020-06-12T02:38:22.079524+0000","last_became_active":"2020-06-12T02:37:42.375902+0000","last_became_peered":"2020-06-12T02:37:42.375902+0000","last_unstale":"2020-06-12T02:38:22.079524+0000","last_undegraded":"2020-06-12T02:38:22.079524+0000","last_fullsized":"2020-06-12T02:38:22.079524+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:38:22.079458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:38:22.079458+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"15'2","reported_seq":"49","reported_epoch":"57","state":"active+clean","last_fresh":"2020-06-12T02:38:24.414406+0000","last_change":"2020-06-12T02:38:24.369083+0000","last_active":"2020-06-12T02:38:24.414406+0000","last_peered":"2020-06-12T02:38:24.414406+0000","last_clean":"2020-06-12T02:38:24.414406+0000","last_became_active":"2020-06-12T02:37:54.558125+0000","last_became_peered":"2020-06-12T02:37:54.558125+0000","last_unstale":"2020-06-12T02:38:24.414406+0000","last_undegraded":"2020-06-12T02:38:24.414406+0000","last_fullsized":"2020-06-12T02:38:24.414406+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":3,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'2","last_scrub_stamp":"2020-06-12T02:38:24.368994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:04.419204+0000","last_clean_scrub_stamp":"2020-06-12T02:38:24.368994+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"57'464","reported_seq":"596","reported_epoch":"57","state":"active+clean","last_fresh":"2020-06-12T02:38:26.421107+0000","last_change":"2020-06-12T02:37:27.898985+0000","last_active":"2020-06-12T02:38:26.421107+0000","last_peered":"2020-06-12T02:38:26.421107+0000","last_clean":"2020-06-12T02:38:26.421107+0000","last_became_active":"2020-06-12T02:37:27.898516+0000","last_became_peered":"2020-06-12T02:37:27.898516+0000","last_unstale":"2020-06-12T02:38:26.421107+0000","last_undegraded":"2020-06-12T02:38:26.421107+0000","last_fullsized":"2020-06-12T02:38:26.421107+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":464,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":165,"num_read_kb":89,"num_write":353,"num_write_kb":98952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"51","reported_epoch":"57","state":"active+clean","last_fresh":"2020-06-12T02:38:24.411528+0000","last_change":"2020-06-12T02:38:13.661120+0000","last_active":"2020-06-12T02:38:24.411528+0000","last_peered":"2020-06-12T02:38:24.411528+0000","last_clean":"2020-06-12T02:38:24.411528+0000","last_became_active":"2020-06-12T02:38:13.660654+0000","last_became_peered":"2020-06-12T02:38:13.660654+0000","last_unstale":"2020-06-12T02:38:24.411528+0000","last_undegraded":"2020-06-12T02:38:24.411528+0000","last_fullsized":"2020-06-12T02:38:24.411528+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"43","reported_epoch":"56","state":"active+clean","last_fresh":"2020-06-12T02:38:21.404761+0000","last_change":"2020-06-12T02:37:27.901368+0000","last_active":"2020-06-12T02:38:21.404761+0000","last_peered":"2020-06-12T02:38:21.404761+0000","last_clean":"2020-06-12T02:38:21.404761+0000","last_became_active":"2020-06-12T02:37:27.900888+0000","last_became_peered":"2020-06-12T02:37:27.900888+0000","last_unstale":"2020-06-12T02:38:21.404761+0000","last_undegraded":"2020-06-12T02:38:21.404761+0000","last_fullsized":"2020-06-12T02:38:21.404761+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"43","reported_epoch":"57","state":"active+clean","last_fresh":"2020-06-12T02:38:24.411537+0000","last_change":"2020-06-12T02:38:05.892563+0000","last_active":"2020-06-12T02:38:24.411537+0000","last_peered":"2020-06-12T02:38:24.411537+0000","last_clean":"2020-06-12T02:38:24.411537+0000","last_became_active":"2020-06-12T02:37:54.557420+0000","last_became_peered":"2020-06-12T02:37:54.557420+0000","last_unstale":"2020-06-12T02:38:24.411537+0000","last_undegraded":"2020-06-12T02:38:24.411537+0000","last_fullsized":"2020-06-12T02:38:24.411537+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"18","reported_epoch":"56","state":"active+clean","last_fresh":"2020-06-12T02:38:21.393369+0000","last_change":"2020-06-12T02:38:13.666985+0000","last_active":"2020-06-12T02:38:21.393369+0000","last_peered":"2020-06-12T02:38:21.393369+0000","last_clean":"2020-06-12T02:38:21.393369+0000","last_became_active":"2020-06-12T02:38:13.666441+0000","last_became_peered":"2020-06-12T02:38:13.666441+0000","last_unstale":"2020-06-12T02:38:21.393369+0000","last_undegraded":"2020-06-12T02:38:21.393369+0000","last_fullsized":"2020-06-12T02:38:21.393369+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"57'1105","reported_seq":"1376","reported_epoch":"57","state":"active+clean","last_fresh":"2020-06-12T02:38:26.610035+0000","last_change":"2020-06-12T02:37:57.567683+0000","last_active":"2020-06-12T02:38:26.610035+0000","last_peered":"2020-06-12T02:38:26.610035+0000","last_clean":"2020-06-12T02:38:26.610035+0000","last_became_active":"2020-06-12T02:37:57.566865+0000","last_became_peered":"2020-06-12T02:37:57.566865+0000","last_unstale":"2020-06-12T02:38:26.610035+0000","last_undegraded":"2020-06-12T02:38:26.610035+0000","last_fullsized":"2020-06-12T02:38:26.610035+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":1105,"ondisk_log_size":1105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":376,"num_read_kb":210,"num_write":848,"num_write_kb":238624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2793472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"17","reported_epoch":"56","state":"active+clean","last_fresh":"2020-06-12T02:38:21.394473+0000","last_change":"2020-06-12T02:38:13.664365+0000","last_active":"2020-06-12T02:38:21.394473+0000","last_peered":"2020-06-12T02:38:21.394473+0000","last_clean":"2020-06-12T02:38:21.394473+0000","last_became_active":"2020-06-12T02:38:13.663999+0000","last_became_peered":"2020-06-12T02:38:13.663999+0000","last_unstale":"2020-06-12T02:38:21.394473+0000","last_undegraded":"2020-06-12T02:38:21.394473+0000","last_fullsized":"2020-06-12T02:38:21.394473+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"62","reported_epoch":"56","state":"active+clean","last_fresh":"2020-06-12T02:38:22.034355+0000","last_change":"2020-06-12T02:38:22.034355+0000","last_active":"2020-06-12T02:38:22.034355+0000","last_peered":"2020-06-12T02:38:22.034355+0000","last_clean":"2020-06-12T02:38:22.034355+0000","last_became_active":"2020-06-12T02:37:54.554986+0000","last_became_peered":"2020-06-12T02:37:54.554986+0000","last_unstale":"2020-06-12T02:38:22.034355+0000","last_undegraded":"2020-06-12T02:38:22.034355+0000","last_fullsized":"2020-06-12T02:38:22.034355+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:38:22.034263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:38:22.034263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"52","reported_epoch":"56","state":"active+clean","last_fresh":"2020-06-12T02:38:21.404809+0000","last_change":"2020-06-12T02:38:13.666662+0000","last_active":"2020-06-12T02:38:21.404809+0000","last_peered":"2020-06-12T02:38:21.404809+0000","last_clean":"2020-06-12T02:38:21.404809+0000","last_became_active":"2020-06-12T02:38:13.666144+0000","last_became_peered":"2020-06-12T02:38:13.666144+0000","last_unstale":"2020-06-12T02:38:21.404809+0000","last_undegraded":"2020-06-12T02:38:21.404809+0000","last_fullsized":"2020-06-12T02:38:21.404809+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":5267456,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3813,"num_read_kb":2064,"num_write":8432,"num_write_kb":2401384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":13869056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12496896,"data_stored":12496896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11141,"ondisk_log_size":11141,"up":48,"acting":48,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26157,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":34,"seq":146028888072,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1052836,"kb_used_data":4252,"kb_used_omap":25,"kb_used_meta":1048550,"kb_avail":93319004,"statfs":{"total":96636764160,"available":95558660096,"internally_reserved":1073741824,"allocated":4354048,"data_stored":3998520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26157,"internal_metadata":1073715667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3640000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.87}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.085}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.123}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91300000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1319999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574866,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050480,"kb_used_data":1896,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321360,"statfs":{"total":96636764160,"available":95561072640,"internally_reserved":1073741824,"allocated":1941504,"data_stored":1592050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.655,"5min":1.655,"15min":1.655},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":1.0329999999999999}]},{"osd":1,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.335,"5min":1.335,"15min":1.335},"last":0.60899999999999999},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.923,"5min":1.923,"15min":1.923},"last":0.75800000000000001}]},{"osd":2,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.01,"15min":1.01},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.6699999999999999,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":0.624},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.653,"5min":1.653,"15min":1.653},"last":1.0049999999999999}]},{"osd":3,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.145,"15min":1.145},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.5649999999999999,"5min":1.5649999999999999,"15min":1.5649999999999999},"last":1.165},{"interface":"front","average":{"1min":1.143,"5min":1.143,"15min":1.143},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.556,"5min":1.556,"15min":1.556},"last":1.24}]},{"osd":4,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.246,"5min":1.246,"15min":1.246},"last":1.073},{"interface":"front","average":{"1min":1.071,"5min":1.071,"15min":1.071},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.673,"5min":1.673,"15min":1.673},"last":1.1359999999999999}]},{"osd":5,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.8120000000000001,"5min":1.8120000000000001,"15min":1.8120000000000001},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.677,"5min":1.677,"15min":1.677},"last":1.256}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3160000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049228,"kb_used_data":644,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322612,"statfs":{"total":96636764160,"available":95562354688,"internally_reserved":1073741824,"allocated":659456,"data_stored":310002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 12 02:38:14 2020","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.548,"5min":1.548,"15min":1.548},"last":1.137},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":2.149,"5min":2.149,"15min":2.149},"last":1.177}]},{"osd":2,"last update":"Fri Jun 12 02:38:14 2020","interfaces":[{"interface":"back","average":{"1min":1,"5min":1,"15min":1},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":3.0009999999999999,"5min":3.0009999999999999,"15min":3.0009999999999999},"last":1.294},{"interface":"front","average":{"1min":1.014,"5min":1.014,"15min":1.014},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":2.4159999999999999,"5min":2.4159999999999999,"15min":2.4159999999999999},"last":0.59199999999999997}]},{"osd":3,"last update":"Fri Jun 12 02:38:14 2020","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":2.133,"5min":2.133,"15min":2.133},"last":1.454},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":2.4609999999999999,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":0.94899999999999995}]},{"osd":4,"last update":"Fri Jun 12 02:38:14 2020","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":2.1749999999999998,"5min":2.1749999999999998,"15min":2.1749999999999998},"last":1.804},{"interface":"front","average":{"1min":1.302,"5min":1.302,"15min":1.302},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":2.2919999999999998,"5min":2.2919999999999998,"15min":2.2919999999999998},"last":1.321}]},{"osd":5,"last update":"Fri Jun 12 02:38:14 2020","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.3200000000000001,"15min":1.3200000000000001},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":2.3090000000000002,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":1.0600000000000001},{"interface":"front","average":{"1min":1.448,"5min":1.448,"15min":1.448},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.835,"5min":2.835,"15min":2.835},"last":1.2030000000000001}]},{"osd":6,"last update":"Fri Jun 12 02:38:14 2020","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.389,"15min":1.389},"min":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"max":{"1min":2.4649999999999999,"5min":2.4649999999999999,"15min":2.4649999999999999},"last":1.405},{"interface":"front","average":{"1min":1.6299999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"max":{"1min":3.1589999999999998,"5min":3.1589999999999998,"15min":3.1589999999999998},"last":1.96}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0389999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574867,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050628,"kb_used_data":2044,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321212,"statfs":{"total":96636764160,"available":95560921088,"internally_reserved":1073741824,"allocated":2093056,"data_stored":1743602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,6],"upper_bound":16384},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jun 12 02:38:16 2020","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.7789999999999999,"5min":1.7789999999999999,"15min":1.7789999999999999},"last":1.135}]},{"osd":2,"last update":"Fri Jun 12 02:38:16 2020","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":2.141,"5min":2.141,"15min":2.141},"last":1.2130000000000001},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.873,"5min":1.873,"15min":1.873},"last":0.94699999999999995}]},{"osd":3,"last update":"Fri Jun 12 02:38:16 2020","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":2.2320000000000002,"5min":2.2320000000000002,"15min":2.2320000000000002},"last":1.242},{"interface":"front","average":{"1min":1.23,"5min":1.23,"15min":1.23},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.8220000000000001,"5min":1.8220000000000001,"15min":1.8220000000000001},"last":1.3620000000000001}]},{"osd":4,"last update":"Fri Jun 12 02:38:16 2020","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.7709999999999999,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":1.2849999999999999},{"interface":"front","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.554,"5min":1.554,"15min":1.554},"last":1.397}]},{"osd":5,"last update":"Fri Jun 12 02:38:16 2020","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.212,"15min":1.212},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":1.6699999999999999,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":0.96699999999999997},{"interface":"front","average":{"1min":1.502,"5min":1.502,"15min":1.502},"min":{"1min":1.006,"5min":1.006,"15min":1.006},"max":{"1min":2.6240000000000001,"5min":2.6240000000000001,"15min":2.6240000000000001},"last":1.49}]},{"osd":6,"last update":"Fri Jun 12 02:38:16 2020","interfaces":[{"interface":"back","average":{"1min":1.5840000000000001,"5min":1.5840000000000001,"15min":1.5840000000000001},"min":{"1min":1.157,"5min":1.157,"15min":1.157},"max":{"1min":3.2440000000000002,"5min":3.2440000000000002,"15min":3.2440000000000002},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.335,"5min":1.335,"15min":1.335},"min":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"max":{"1min":3.105,"5min":3.105,"15min":3.105},"last":1.427}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7529999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1052064,"kb_used_data":3480,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93319776,"statfs":{"total":96636764160,"available":95559450624,"internally_reserved":1073741824,"allocated":3563520,"data_stored":3214066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 12 02:38:10 2020","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.871,"15min":0.871},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.9259999999999999,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":3.0049999999999999,"5min":3.0049999999999999,"15min":3.0049999999999999},"last":1.0589999999999999}]},{"osd":1,"last update":"Fri Jun 12 02:38:10 2020","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.046,"15min":1.046},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":2.21,"5min":2.21,"15min":2.21},"last":0.96199999999999997},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":2.5219999999999998,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":1.8620000000000001}]},{"osd":3,"last update":"Fri Jun 12 02:38:10 2020","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1,"5min":1,"15min":1},"last":0.745},{"interface":"front","average":{"1min":1.145,"5min":1.145,"15min":1.145},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.2759999999999998,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":1.038}]},{"osd":4,"last update":"Fri Jun 12 02:38:10 2020","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.321,"15min":1.321},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":2.6949999999999998,"5min":2.6949999999999998,"15min":2.6949999999999998},"last":2.1800000000000002},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"min":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"max":{"1min":1.7549999999999999,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":1.6870000000000001}]},{"osd":5,"last update":"Fri Jun 12 02:38:10 2020","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"min":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"max":{"1min":2.1419999999999999,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":1.224},{"interface":"front","average":{"1min":1.399,"5min":1.399,"15min":1.399},"min":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"max":{"1min":2.851,"5min":2.851,"15min":2.851},"last":2.5600000000000001}]},{"osd":6,"last update":"Fri Jun 12 02:38:10 2020","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":1.7110000000000001,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":1.3440000000000001},{"interface":"front","average":{"1min":1.462,"5min":1.462,"15min":1.462},"min":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"max":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":1.3280000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.968}]}]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1051912,"kb_used_data":3328,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93319928,"statfs":{"total":96636764160,"available":95559606272,"internally_reserved":1073741824,"allocated":3407872,"data_stored":3058418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 12 02:38:12 2020","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":2.0339999999999998,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":1.3220000000000001},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.9219999999999999,"5min":1.9219999999999999,"15min":1.9219999999999999},"last":2.2480000000000002}]},{"osd":1,"last update":"Fri Jun 12 02:38:12 2020","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.9670000000000001,"5min":1.9670000000000001,"15min":1.9670000000000001},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":2.9969999999999999,"5min":2.9969999999999999,"15min":2.9969999999999999},"last":1.5449999999999999}]},{"osd":2,"last update":"Fri Jun 12 02:38:12 2020","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":4.1079999999999997,"5min":4.1079999999999997,"15min":4.1079999999999997},"last":1.609},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":2.504,"5min":2.504,"15min":2.504},"last":1.7050000000000001}]},{"osd":4,"last update":"Fri Jun 12 02:38:12 2020","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":2.359,"5min":2.359,"15min":2.359},"last":1.9330000000000001},{"interface":"front","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":3.9319999999999999,"5min":3.9319999999999999,"15min":3.9319999999999999},"last":0.70799999999999996}]},{"osd":5,"last update":"Fri Jun 12 02:38:12 2020","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":2.4279999999999999,"5min":2.4279999999999999,"15min":2.4279999999999999},"last":1.0449999999999999},{"interface":"front","average":{"1min":1.452,"5min":1.452,"15min":1.452},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":5.0579999999999998,"5min":5.0579999999999998,"15min":5.0579999999999998},"last":1.859}]},{"osd":6,"last update":"Fri Jun 12 02:38:12 2020","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.29,"15min":1.29},"min":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"max":{"1min":4.0410000000000004,"5min":4.0410000000000004,"15min":4.0410000000000004},"last":2.0859999999999999},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.3620000000000001,"15min":1.3620000000000001},"min":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"max":{"1min":3.0230000000000001,"5min":3.0230000000000001,"15min":3.0230000000000001},"last":1.1699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.996}]}]},{"osd":4,"up_from":13,"seq":55834574865,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050132,"kb_used_data":1548,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321708,"statfs":{"total":96636764160,"available":95561428992,"internally_reserved":1073741824,"allocated":1585152,"data_stored":1235698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.712,"5min":1.712,"15min":1.712},"last":3.7839999999999998},{"interface":"front","average":{"1min":1.02,"5min":1.02,"15min":1.02},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":2.855,"5min":2.855,"15min":2.855},"last":1.034}]},{"osd":1,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.91000000000000003},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.8240000000000001,"5min":1.8240000000000001,"15min":1.8240000000000001},"last":3.9220000000000002}]},{"osd":2,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":3.1480000000000001,"5min":3.1480000000000001,"15min":3.1480000000000001},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"min":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"max":{"1min":3.1259999999999999,"5min":3.1259999999999999,"15min":3.1259999999999999},"last":4.8609999999999998}]},{"osd":3,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.079,"15min":1.079},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":2.411,"5min":2.411,"15min":2.411},"last":1.47},{"interface":"front","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":2.734,"5min":2.734,"15min":2.734},"last":1.1539999999999999}]},{"osd":5,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.04,"15min":1.04},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.6399999999999999,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":4.0739999999999998},{"interface":"front","average":{"1min":1.115,"5min":1.115,"15min":1.115},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"last":1.3300000000000001}]},{"osd":6,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":0.998,"15min":0.998},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.83499999999999996},{"interface":"front","average":{"1min":1.341,"5min":1.341,"15min":1.341},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":2.1850000000000001,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":4.2439999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.3970000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574865,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1053800,"kb_used_data":5216,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93318040,"statfs":{"total":96636764160,"available":95557672960,"internally_reserved":1073741824,"allocated":5341184,"data_stored":4995360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 12 02:38:13 2020","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.456,"5min":1.456,"15min":1.456},"last":1.367},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.597,"5min":1.597,"15min":1.597},"last":0.73599999999999999}]},{"osd":1,"last update":"Fri Jun 12 02:38:13 2020","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":1.258},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":1.6950000000000001}]},{"osd":2,"last update":"Fri Jun 12 02:38:13 2020","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":1.5449999999999999},{"interface":"front","average":{"1min":1.012,"5min":1.012,"15min":1.012},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.4259999999999999,"5min":1.4259999999999999,"15min":1.4259999999999999},"last":1.0289999999999999}]},{"osd":3,"last update":"Fri Jun 12 02:38:13 2020","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":2.004,"5min":2.004,"15min":2.004},"last":0.94899999999999995},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.627,"5min":1.627,"15min":1.627},"last":0.86399999999999999}]},{"osd":4,"last update":"Fri Jun 12 02:38:13 2020","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.997,"15min":0.997},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.796,"5min":1.796,"15min":1.796},"last":1.625},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":1.204}]},{"osd":6,"last update":"Fri Jun 12 02:38:13 2020","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.827,"5min":1.827,"15min":1.827},"last":1.054},{"interface":"front","average":{"1min":1.228,"5min":1.228,"15min":1.228},"min":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"max":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":1.772}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.403}]}]}]}} 2020-06-12T02:38:30.431 INFO:tasks.thrashosds.thrasher:Setting 3.8 to [0, 4, 2, 1, 3, 5] 2020-06-12T02:38:30.431 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.8', '0', '4', '2', '1', '3', '5'] 2020-06-12T02:38:30.432 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd pg-upmap-items 3.8 0 4 2 1 3 5 2020-06-12T02:38:30.432 INFO:tasks.rados.rados.0.smithi107.stdout:6734: writing smithi10729407-8 from 770048 to 1548288 tid 2 2020-06-12T02:38:30.434 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2020-06-12T02:38:30.436 INFO:tasks.rados.rados.0.smithi107.stdout:6734: writing smithi10729407-8 from 1548288 to 2285568 tid 3 2020-06-12T02:38:30.438 INFO:tasks.rados.rados.0.smithi107.stdout:6734: writing smithi10729407-8 from 2285568 to 2375680 tid 4 2020-06-12T02:38:30.438 INFO:tasks.rados.rados.0.smithi107.stdout:6735: delete oid 3 current snap is 0 2020-06-12T02:38:30.438 INFO:tasks.rados.rados.0.smithi107.stdout:6728: finishing write tid 7 to smithi10729407-17 2020-06-12T02:38:30.438 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1277 (ObjNum 6727 snap 0 seq_num 6727) dirty exists 2020-06-12T02:38:30.438 INFO:tasks.rados.rados.0.smithi107.stdout:6728: left oid 17 (ObjNum 6727 snap 0 seq_num 6727) 2020-06-12T02:38:30.439 INFO:tasks.rados.rados.0.smithi107.stdout:6733: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:30.439 INFO:tasks.rados.rados.0.smithi107.stdout:6733: finishing write tid 2 to smithi10729407-20 2020-06-12T02:38:30.439 INFO:tasks.rados.rados.0.smithi107.stdout:6733: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:30.439 INFO:tasks.rados.rados.0.smithi107.stdout:6734: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:30.439 INFO:tasks.rados.rados.0.smithi107.stdout:6733: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:30.442 INFO:tasks.rados.rados.0.smithi107.stdout:6733: finishing write tid 6 to smithi10729407-20 2020-06-12T02:38:30.442 INFO:tasks.rados.rados.0.smithi107.stdout:6733: finishing write tid 7 to smithi10729407-20 2020-06-12T02:38:30.442 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 513 (ObjNum 6732 snap 0 seq_num 6732) dirty exists 2020-06-12T02:38:30.442 INFO:tasks.rados.rados.0.smithi107.stdout:6733: left oid 20 (ObjNum 6732 snap 0 seq_num 6732) 2020-06-12T02:38:30.443 INFO:tasks.rados.rados.0.smithi107.stdout:6728: done (4 left) 2020-06-12T02:38:30.444 INFO:tasks.rados.rados.0.smithi107.stdout:6733: done (3 left) 2020-06-12T02:38:30.444 INFO:tasks.rados.rados.0.smithi107.stdout:6735: done (2 left) 2020-06-12T02:38:30.444 INFO:tasks.rados.rados.0.smithi107.stdout:6736: delete oid 15 current snap is 0 2020-06-12T02:38:30.453 INFO:tasks.rados.rados.0.smithi107.stdout:6734: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:30.454 INFO:tasks.rados.rados.0.smithi107.stdout:6734: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:30.455 INFO:tasks.rados.rados.0.smithi107.stdout:6734: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:30.455 INFO:tasks.rados.rados.0.smithi107.stdout:6734: finishing write tid 6 to smithi10729407-8 2020-06-12T02:38:30.456 INFO:tasks.rados.rados.0.smithi107.stdout:6736: done (2 left) 2020-06-12T02:38:30.456 INFO:tasks.rados.rados.0.smithi107.stdout:6737: delete oid 16 current snap is 0 2020-06-12T02:38:30.458 INFO:tasks.rados.rados.0.smithi107.stdout:6734: finishing write tid 7 to smithi10729407-8 2020-06-12T02:38:30.458 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1261 (ObjNum 6733 snap 0 seq_num 6733) dirty exists 2020-06-12T02:38:30.458 INFO:tasks.rados.rados.0.smithi107.stdout:6734: left oid 8 (ObjNum 6733 snap 0 seq_num 6733) 2020-06-12T02:38:30.459 INFO:tasks.rados.rados.0.smithi107.stdout:6734: done (2 left) 2020-06-12T02:38:30.459 INFO:tasks.rados.rados.0.smithi107.stdout:6737: done (1 left) 2020-06-12T02:38:30.459 INFO:tasks.rados.rados.0.smithi107.stdout:6738: delete oid 15 current snap is 0 2020-06-12T02:38:30.461 INFO:tasks.rados.rados.0.smithi107.stdout:6738: done (1 left) 2020-06-12T02:38:30.461 INFO:tasks.rados.rados.0.smithi107.stdout:6739: delete oid 9 current snap is 0 2020-06-12T02:38:30.464 INFO:tasks.rados.rados.0.smithi107.stdout:6739: done (1 left) 2020-06-12T02:38:30.464 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:30.464 INFO:tasks.rados.rados.0.smithi107.stdout:6740: seq_num 6739 ranges {0=65536} 2020-06-12T02:38:30.465 INFO:tasks.rados.rados.0.smithi107.stdout:6740: writing smithi10729407-11 from 0 to 65536 tid 1 2020-06-12T02:38:30.465 INFO:tasks.rados.rados.0.smithi107.stdout:6741: delete oid 20 current snap is 0 2020-06-12T02:38:30.469 INFO:tasks.rados.rados.0.smithi107.stdout:6741: done (2 left) 2020-06-12T02:38:30.469 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:38:30.469 INFO:tasks.rados.rados.0.smithi107.stdout:6742: seq_num 6741 ranges {0=106496} 2020-06-12T02:38:30.470 INFO:tasks.rados.rados.0.smithi107.stdout:6742: writing smithi10729407-16 from 0 to 106496 tid 1 2020-06-12T02:38:30.470 INFO:tasks.rados.rados.0.smithi107.stdout:6743: delete oid 15 current snap is 0 2020-06-12T02:38:30.472 INFO:tasks.rados.rados.0.smithi107.stdout:6743: done (3 left) 2020-06-12T02:38:30.472 INFO:tasks.rados.rados.0.smithi107.stdout:6744: delete oid 17 current snap is 0 2020-06-12T02:38:30.475 INFO:tasks.rados.rados.0.smithi107.stdout:6740: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:30.476 INFO:tasks.rados.rados.0.smithi107.stdout:6740: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:30.476 INFO:tasks.rados.rados.0.smithi107.stdout:6742: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:30.476 INFO:tasks.rados.rados.0.smithi107.stdout:6744: done (3 left) 2020-06-12T02:38:30.476 INFO:tasks.rados.rados.0.smithi107.stdout:6745: delete oid 13 current snap is 0 2020-06-12T02:38:30.476 INFO:tasks.rados.rados.0.smithi107.stdout:6742: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:30.477 INFO:tasks.rados.rados.0.smithi107.stdout:6740: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:30.478 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1196 (ObjNum 6739 snap 0 seq_num 6739) dirty exists 2020-06-12T02:38:30.478 INFO:tasks.rados.rados.0.smithi107.stdout:6740: left oid 11 (ObjNum 6739 snap 0 seq_num 6739) 2020-06-12T02:38:30.478 INFO:tasks.rados.rados.0.smithi107.stdout:6742: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:30.479 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1196 (ObjNum 6741 snap 0 seq_num 6741) dirty exists 2020-06-12T02:38:30.479 INFO:tasks.rados.rados.0.smithi107.stdout:6742: left oid 16 (ObjNum 6741 snap 0 seq_num 6741) 2020-06-12T02:38:30.480 INFO:tasks.rados.rados.0.smithi107.stdout:6740: done (3 left) 2020-06-12T02:38:30.481 INFO:tasks.rados.rados.0.smithi107.stdout:6742: done (2 left) 2020-06-12T02:38:30.481 INFO:tasks.rados.rados.0.smithi107.stdout:6745: done (1 left) 2020-06-12T02:38:30.481 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:30.481 INFO:tasks.rados.rados.0.smithi107.stdout:6746: seq_num 6745 ranges {2023424=172032} 2020-06-12T02:38:30.493 INFO:tasks.rados.rados.0.smithi107.stdout:6746: writing smithi10729407-4 from 2023424 to 2195456 tid 1 2020-06-12T02:38:30.493 INFO:tasks.rados.rados.0.smithi107.stdout:6747: delete oid 15 current snap is 0 2020-06-12T02:38:30.495 INFO:tasks.rados.rados.0.smithi107.stdout:6747: done (2 left) 2020-06-12T02:38:30.495 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:38:30.496 INFO:tasks.rados.rados.0.smithi107.stdout:6748: seq_num 6747 ranges {0=204800} 2020-06-12T02:38:30.496 INFO:tasks.rados.rados.0.smithi107.stdout:6748: writing smithi10729407-20 from 0 to 204800 tid 1 2020-06-12T02:38:30.497 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:30.497 INFO:tasks.rados.rados.0.smithi107.stdout:6749: seq_num 6748 ranges {2375680=221184} 2020-06-12T02:38:30.510 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:30.507+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:30.511 INFO:tasks.rados.rados.0.smithi107.stdout:6749: writing smithi10729407-8 from 2375680 to 2596864 tid 1 2020-06-12T02:38:30.512 INFO:tasks.rados.rados.0.smithi107.stdout:6750: delete oid 3 current snap is 0 2020-06-12T02:38:30.512 INFO:tasks.rados.rados.0.smithi107.stdout:6746: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:30.512 INFO:tasks.rados.rados.0.smithi107.stdout:6748: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:30.512 INFO:tasks.rados.rados.0.smithi107.stdout:6746: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:30.512 INFO:tasks.rados.rados.0.smithi107.stdout:6746: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:30.513 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1280 (ObjNum 6745 snap 0 seq_num 6745) dirty exists 2020-06-12T02:38:30.513 INFO:tasks.rados.rados.0.smithi107.stdout:6746: left oid 4 (ObjNum 6745 snap 0 seq_num 6745) 2020-06-12T02:38:30.513 INFO:tasks.rados.rados.0.smithi107.stdout:6748: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:30.513 INFO:tasks.rados.rados.0.smithi107.stdout:6748: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:30.513 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 516 (ObjNum 6747 snap 0 seq_num 6747) dirty exists 2020-06-12T02:38:30.514 INFO:tasks.rados.rados.0.smithi107.stdout:6748: left oid 20 (ObjNum 6747 snap 0 seq_num 6747) 2020-06-12T02:38:30.514 INFO:tasks.rados.rados.0.smithi107.stdout:6746: done (4 left) 2020-06-12T02:38:30.514 INFO:tasks.rados.rados.0.smithi107.stdout:6748: done (3 left) 2020-06-12T02:38:30.514 INFO:tasks.rados.rados.0.smithi107.stdout:6750: done (2 left) 2020-06-12T02:38:30.515 INFO:tasks.rados.rados.0.smithi107.stdout:6751: delete oid 9 current snap is 0 2020-06-12T02:38:30.515 INFO:tasks.rados.rados.0.smithi107.stdout:6749: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:30.517 INFO:tasks.rados.rados.0.smithi107.stdout:6749: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:30.518 INFO:tasks.rados.rados.0.smithi107.stdout:6751: done (2 left) 2020-06-12T02:38:30.518 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:30.518 INFO:tasks.rados.rados.0.smithi107.stdout:6752: seq_num 6751 ranges {1368064=270336} 2020-06-12T02:38:30.527 INFO:tasks.rados.rados.0.smithi107.stdout:6752: writing smithi10729407-19 from 1368064 to 1638400 tid 1 2020-06-12T02:38:30.527 INFO:tasks.rados.rados.0.smithi107.stdout:6753: delete oid 18 current snap is 0 2020-06-12T02:38:30.527 INFO:tasks.rados.rados.0.smithi107.stdout:6749: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:30.528 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1267 (ObjNum 6748 snap 0 seq_num 6748) dirty exists 2020-06-12T02:38:30.528 INFO:tasks.rados.rados.0.smithi107.stdout:6749: left oid 8 (ObjNum 6748 snap 0 seq_num 6748) 2020-06-12T02:38:30.531 INFO:tasks.rados.rados.0.smithi107.stdout:6749: done (3 left) 2020-06-12T02:38:30.531 INFO:tasks.rados.rados.0.smithi107.stdout:6753: done (2 left) 2020-06-12T02:38:30.532 INFO:tasks.rados.rados.0.smithi107.stdout:6754: delete oid 13 current snap is 0 2020-06-12T02:38:30.532 INFO:tasks.rados.rados.0.smithi107.stdout:6752: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:30.533 INFO:tasks.rados.rados.0.smithi107.stdout:6752: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:30.533 INFO:tasks.rados.rados.0.smithi107.stdout:6754: done (2 left) 2020-06-12T02:38:30.534 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:38:30.534 INFO:tasks.rados.rados.0.smithi107.stdout:6755: seq_num 6754 ranges {204800=319488} 2020-06-12T02:38:30.537 INFO:tasks.rados.rados.0.smithi107.stdout:6755: writing smithi10729407-20 from 204800 to 524288 tid 1 2020-06-12T02:38:30.537 INFO:tasks.rados.rados.0.smithi107.stdout:6756: delete oid 18 current snap is 0 2020-06-12T02:38:30.537 INFO:tasks.rados.rados.0.smithi107.stdout:6752: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:30.538 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1231 (ObjNum 6751 snap 0 seq_num 6751) dirty exists 2020-06-12T02:38:30.538 INFO:tasks.rados.rados.0.smithi107.stdout:6752: left oid 19 (ObjNum 6751 snap 0 seq_num 6751) 2020-06-12T02:38:30.540 INFO:tasks.rados.rados.0.smithi107.stdout:6752: done (3 left) 2020-06-12T02:38:30.540 INFO:tasks.rados.rados.0.smithi107.stdout:6756: done (2 left) 2020-06-12T02:38:30.540 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:30.541 INFO:tasks.rados.rados.0.smithi107.stdout:6757: seq_num 6756 ranges {0=352256} 2020-06-12T02:38:30.542 INFO:tasks.rados.rados.0.smithi107.stdout:6757: writing smithi10729407-13 from 0 to 352256 tid 1 2020-06-12T02:38:30.542 INFO:tasks.rados.rados.0.smithi107.stdout:6758: delete oid 8 current snap is 0 2020-06-12T02:38:30.543 INFO:tasks.rados.rados.0.smithi107.stdout:6755: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:30.545 INFO:tasks.rados.rados.0.smithi107.stdout:6755: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:30.547 INFO:tasks.rados.rados.0.smithi107.stdout:6755: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:30.547 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 518 (ObjNum 6754 snap 0 seq_num 6754) dirty exists 2020-06-12T02:38:30.547 INFO:tasks.rados.rados.0.smithi107.stdout:6755: left oid 20 (ObjNum 6754 snap 0 seq_num 6754) 2020-06-12T02:38:30.548 INFO:tasks.rados.rados.0.smithi107.stdout:6755: done (3 left) 2020-06-12T02:38:30.548 INFO:tasks.rados.rados.0.smithi107.stdout:6758: done (2 left) 2020-06-12T02:38:30.548 INFO:tasks.rados.rados.0.smithi107.stdout:6759: delete oid 1 current snap is 0 2020-06-12T02:38:30.550 INFO:tasks.rados.rados.0.smithi107.stdout:6757: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:30.550 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:30.551 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:30.551 INFO:tasks.rados.rados.0.smithi107.stdout:6759: done (2 left) 2020-06-12T02:38:30.552 INFO:tasks.rados.rados.0.smithi107.stdout:6760: delete oid 8 current snap is 0 2020-06-12T02:38:30.554 INFO:tasks.rados.rados.0.smithi107.stdout:6757: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:30.555 INFO:tasks.rados.rados.0.smithi107.stdout:6760: done (2 left) 2020-06-12T02:38:30.555 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:30.555 INFO:tasks.rados.rados.0.smithi107.stdout:6761: seq_num 6760 ranges {0=425984} 2020-06-12T02:38:30.558 INFO:tasks.rados.rados.0.smithi107.stdout:6761: writing smithi10729407-2 from 0 to 425984 tid 1 2020-06-12T02:38:30.558 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:30.558 INFO:tasks.rados.rados.0.smithi107.stdout:6762: seq_num 6761 ranges {4915200=442368} 2020-06-12T02:38:30.561 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-06-12T02:38:30.588 INFO:tasks.rados.rados.0.smithi107.stdout:6762: writing smithi10729407-7 from 4915200 to 5357568 tid 1 2020-06-12T02:38:30.588 INFO:tasks.rados.rados.0.smithi107.stdout:6763: delete oid 20 current snap is 0 2020-06-12T02:38:30.588 INFO:tasks.rados.rados.0.smithi107.stdout:6757: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:30.588 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1275 (ObjNum 6756 snap 0 seq_num 6756) dirty exists 2020-06-12T02:38:30.589 INFO:tasks.rados.rados.0.smithi107.stdout:6757: left oid 13 (ObjNum 6756 snap 0 seq_num 6756) 2020-06-12T02:38:30.589 INFO:tasks.rados.rados.0.smithi107.stdout:6761: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:30.589 INFO:tasks.rados.rados.0.smithi107.stdout:6761: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:30.589 INFO:tasks.rados.rados.0.smithi107.stdout:6761: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:30.589 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 650 (ObjNum 6760 snap 0 seq_num 6760) dirty exists 2020-06-12T02:38:30.590 INFO:tasks.rados.rados.0.smithi107.stdout:6761: left oid 2 (ObjNum 6760 snap 0 seq_num 6760) 2020-06-12T02:38:30.590 INFO:tasks.rados.rados.0.smithi107.stdout:6757: done (4 left) 2020-06-12T02:38:30.592 INFO:tasks.rados.rados.0.smithi107.stdout:6761: done (3 left) 2020-06-12T02:38:30.592 INFO:tasks.rados.rados.0.smithi107.stdout:6763: done (2 left) 2020-06-12T02:38:30.592 INFO:tasks.rados.rados.0.smithi107.stdout:6764: delete oid 17 current snap is 0 2020-06-12T02:38:30.593 INFO:tasks.rados.rados.0.smithi107.stdout:6764: done (2 left) 2020-06-12T02:38:30.593 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:30.593 INFO:tasks.rados.rados.0.smithi107.stdout:6765: seq_num 6764 ranges {0=491520} 2020-06-12T02:38:30.595 INFO:tasks.rados.rados.0.smithi107.stdout:6765: writing smithi10729407-15 from 0 to 491520 tid 1 2020-06-12T02:38:30.595 INFO:tasks.rados.rados.0.smithi107.stdout:6766: delete oid 18 current snap is 0 2020-06-12T02:38:30.595 INFO:tasks.rados.rados.0.smithi107.stdout:6762: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:30.595 INFO:tasks.rados.rados.0.smithi107.stdout:6762: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:30.596 INFO:tasks.rados.rados.0.smithi107.stdout:6762: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:30.596 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1277 (ObjNum 6761 snap 0 seq_num 6761) dirty exists 2020-06-12T02:38:30.596 INFO:tasks.rados.rados.0.smithi107.stdout:6762: left oid 7 (ObjNum 6761 snap 0 seq_num 6761) 2020-06-12T02:38:30.597 INFO:tasks.rados.rados.0.smithi107.stdout:6762: done (3 left) 2020-06-12T02:38:30.597 INFO:tasks.rados.rados.0.smithi107.stdout:6766: done (2 left) 2020-06-12T02:38:30.598 INFO:tasks.rados.rados.0.smithi107.stdout:6767: delete oid 16 current snap is 0 2020-06-12T02:38:30.599 INFO:tasks.rados.rados.0.smithi107.stdout:6767: done (2 left) 2020-06-12T02:38:30.600 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:30.600 INFO:tasks.rados.rados.0.smithi107.stdout:6768: seq_num 6767 ranges {1638400=540672} 2020-06-12T02:38:30.611 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:30.608+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:30.612 INFO:tasks.rados.rados.0.smithi107.stdout:6768: writing smithi10729407-19 from 1638400 to 2179072 tid 1 2020-06-12T02:38:30.612 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:38:30.613 INFO:tasks.rados.rados.0.smithi107.stdout:6769: seq_num 6768 ranges {0=557056} 2020-06-12T02:38:30.615 INFO:tasks.rados.rados.0.smithi107.stdout:6769: writing smithi10729407-1 from 0 to 557056 tid 1 2020-06-12T02:38:30.616 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:38:30.616 INFO:tasks.rados.rados.0.smithi107.stdout:6770: seq_num 6769 ranges {0=573440} 2020-06-12T02:38:30.620 INFO:tasks.rados.rados.0.smithi107.stdout:6770: writing smithi10729407-16 from 0 to 573440 tid 1 2020-06-12T02:38:30.620 INFO:tasks.rados.rados.0.smithi107.stdout:6771: delete oid 6 current snap is 0 2020-06-12T02:38:30.620 INFO:tasks.rados.rados.0.smithi107.stdout:6765: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:30.620 INFO:tasks.rados.rados.0.smithi107.stdout:6765: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:30.621 INFO:tasks.rados.rados.0.smithi107.stdout:6765: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:30.621 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1271 (ObjNum 6764 snap 0 seq_num 6764) dirty exists 2020-06-12T02:38:30.621 INFO:tasks.rados.rados.0.smithi107.stdout:6765: left oid 15 (ObjNum 6764 snap 0 seq_num 6764) 2020-06-12T02:38:30.621 INFO:tasks.rados.rados.0.smithi107.stdout:6768: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:30.623 INFO:tasks.rados.rados.0.smithi107.stdout:6769: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:30.625 INFO:tasks.rados.rados.0.smithi107.stdout:6769: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:30.626 INFO:tasks.rados.rados.0.smithi107.stdout:6768: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:30.626 INFO:tasks.rados.rados.0.smithi107.stdout:6765: done (5 left) 2020-06-12T02:38:30.627 INFO:tasks.rados.rados.0.smithi107.stdout:6771: done (4 left) 2020-06-12T02:38:30.627 INFO:tasks.rados.rados.0.smithi107.stdout:6772: delete oid 5 current snap is 0 2020-06-12T02:38:30.627 INFO:tasks.rados.rados.0.smithi107.stdout:6769: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:30.627 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1223 (ObjNum 6768 snap 0 seq_num 6768) dirty exists 2020-06-12T02:38:30.628 INFO:tasks.rados.rados.0.smithi107.stdout:6769: left oid 1 (ObjNum 6768 snap 0 seq_num 6768) 2020-06-12T02:38:30.628 INFO:tasks.rados.rados.0.smithi107.stdout:6768: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:30.629 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1233 (ObjNum 6767 snap 0 seq_num 6767) dirty exists 2020-06-12T02:38:30.629 INFO:tasks.rados.rados.0.smithi107.stdout:6768: left oid 19 (ObjNum 6767 snap 0 seq_num 6767) 2020-06-12T02:38:30.629 INFO:tasks.rados.rados.0.smithi107.stdout:6770: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:30.631 INFO:tasks.rados.rados.0.smithi107.stdout:6768: done (4 left) 2020-06-12T02:38:30.631 INFO:tasks.rados.rados.0.smithi107.stdout:6769: done (3 left) 2020-06-12T02:38:30.631 INFO:tasks.rados.rados.0.smithi107.stdout:6772: done (2 left) 2020-06-12T02:38:30.631 INFO:tasks.rados.rados.0.smithi107.stdout:6773: delete oid 5 current snap is 0 2020-06-12T02:38:30.634 INFO:tasks.rados.rados.0.smithi107.stdout:6770: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:30.635 INFO:tasks.rados.rados.0.smithi107.stdout:6770: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:30.636 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1200 (ObjNum 6769 snap 0 seq_num 6769) dirty exists 2020-06-12T02:38:30.636 INFO:tasks.rados.rados.0.smithi107.stdout:6770: left oid 16 (ObjNum 6769 snap 0 seq_num 6769) 2020-06-12T02:38:30.636 INFO:tasks.rados.rados.0.smithi107.stdout:6770: done (2 left) 2020-06-12T02:38:30.637 INFO:tasks.rados.rados.0.smithi107.stdout:6773: done (1 left) 2020-06-12T02:38:30.637 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:30.637 INFO:tasks.rados.rados.0.smithi107.stdout:6774: seq_num 6773 ranges {0=638976} 2020-06-12T02:38:30.640 INFO:tasks.rados.rados.0.smithi107.stdout:6774: writing smithi10729407-6 from 0 to 638976 tid 1 2020-06-12T02:38:30.640 INFO:tasks.rados.rados.0.smithi107.stdout:6775: delete oid 2 current snap is 0 2020-06-12T02:38:30.650 INFO:tasks.rados.rados.0.smithi107.stdout:6775: done (2 left) 2020-06-12T02:38:30.650 INFO:tasks.rados.rados.0.smithi107.stdout:6776: delete oid 13 current snap is 0 2020-06-12T02:38:30.651 INFO:tasks.rados.rados.0.smithi107.stdout:6776: done (2 left) 2020-06-12T02:38:30.651 INFO:tasks.rados.rados.0.smithi107.stdout:6777: delete oid 1 current snap is 0 2020-06-12T02:38:30.651 INFO:tasks.rados.rados.0.smithi107.stdout:6774: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:30.652 INFO:tasks.rados.rados.0.smithi107.stdout:6777: done (2 left) 2020-06-12T02:38:30.652 INFO:tasks.rados.rados.0.smithi107.stdout:6778: delete oid 3 current snap is 0 2020-06-12T02:38:30.653 INFO:tasks.rados.rados.0.smithi107.stdout:6774: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:30.654 INFO:tasks.rados.rados.0.smithi107.stdout:6774: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:30.655 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1227 (ObjNum 6773 snap 0 seq_num 6773) dirty exists 2020-06-12T02:38:30.655 INFO:tasks.rados.rados.0.smithi107.stdout:6774: left oid 6 (ObjNum 6773 snap 0 seq_num 6773) 2020-06-12T02:38:30.656 INFO:tasks.rados.rados.0.smithi107.stdout:6774: done (2 left) 2020-06-12T02:38:30.656 INFO:tasks.rados.rados.0.smithi107.stdout:6778: done (1 left) 2020-06-12T02:38:30.657 INFO:tasks.rados.rados.0.smithi107.stdout:6779: delete oid 8 current snap is 0 2020-06-12T02:38:30.658 INFO:tasks.rados.rados.0.smithi107.stdout:6779: done (1 left) 2020-06-12T02:38:30.659 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:30.659 INFO:tasks.rados.rados.0.smithi107.stdout:6780: seq_num 6779 ranges {0=745472} 2020-06-12T02:38:30.663 INFO:tasks.rados.rados.0.smithi107.stdout:6780: writing smithi10729407-1 from 0 to 745472 tid 1 2020-06-12T02:38:30.663 INFO:tasks.rados.rados.0.smithi107.stdout:6781: delete oid 6 current snap is 0 2020-06-12T02:38:30.673 INFO:tasks.rados.rados.0.smithi107.stdout:6780: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:30.674 INFO:tasks.rados.rados.0.smithi107.stdout:6781: done (2 left) 2020-06-12T02:38:30.674 INFO:tasks.rados.rados.0.smithi107.stdout:6782: delete oid 8 current snap is 0 2020-06-12T02:38:30.676 INFO:tasks.rados.rados.0.smithi107.stdout:6780: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:30.676 INFO:tasks.rados.rados.0.smithi107.stdout:6782: done (2 left) 2020-06-12T02:38:30.677 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:30.677 INFO:tasks.rados.rados.0.smithi107.stdout:6783: seq_num 6782 ranges {0=794624} 2020-06-12T02:38:30.683 INFO:tasks.rados.rados.0.smithi107.stdout:6783: writing smithi10729407-3 from 0 to 794624 tid 1 2020-06-12T02:38:30.683 INFO:tasks.rados.rados.0.smithi107.stdout:6784: delete oid 15 current snap is 0 2020-06-12T02:38:30.684 INFO:tasks.rados.rados.0.smithi107.stdout:6780: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:30.684 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1230 (ObjNum 6779 snap 0 seq_num 6779) dirty exists 2020-06-12T02:38:30.684 INFO:tasks.rados.rados.0.smithi107.stdout:6780: left oid 1 (ObjNum 6779 snap 0 seq_num 6779) 2020-06-12T02:38:30.684 INFO:tasks.rados.rados.0.smithi107.stdout:4413: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:30.690 INFO:tasks.rados.rados.0.smithi107.stdout:6780: done (3 left) 2020-06-12T02:38:30.690 INFO:tasks.rados.rados.0.smithi107.stdout:6784: done (2 left) 2020-06-12T02:38:30.690 INFO:tasks.rados.rados.0.smithi107.stdout:6785: delete oid 14 current snap is 0 2020-06-12T02:38:30.693 INFO:tasks.rados.rados.0.smithi107.stdout:4413: finishing write tid 2 to smithi10729407-10 2020-06-12T02:38:30.694 INFO:tasks.rados.rados.0.smithi107.stdout:4413: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:30.694 INFO:tasks.rados.rados.0.smithi107.stdout:6785: done (2 left) 2020-06-12T02:38:30.694 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:30.694 INFO:tasks.rados.rados.0.smithi107.stdout:6786: seq_num 6785 ranges {2179072=442368,2621440=401408} 2020-06-12T02:38:30.710 INFO:tasks.rados.rados.0.smithi107.stdout:6786: writing smithi10729407-19 from 2179072 to 2621440 tid 1 2020-06-12T02:38:30.712 INFO:tasks.rados.rados.0.smithi107.stdout:6786: writing smithi10729407-19 from 2621440 to 3022848 tid 2 2020-06-12T02:38:30.713 INFO:tasks.rados.rados.0.smithi107.stdout:6787: delete oid 8 current snap is 0 2020-06-12T02:38:30.713 INFO:tasks.rados.rados.0.smithi107.stdout:4413: finishing write tid 4 to smithi10729407-10 2020-06-12T02:38:30.713 INFO:tasks.rados.rados.0.smithi107.stdout:4413: finishing write tid 6 to smithi10729407-10 2020-06-12T02:38:30.713 INFO:tasks.rados.rados.0.smithi107.stdout:6783: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:30.713 INFO:tasks.rados.rados.0.smithi107.stdout:4413: finishing write tid 7 to smithi10729407-10 2020-06-12T02:38:30.714 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 439 (ObjNum 4412 snap 0 seq_num 4412) dirty exists 2020-06-12T02:38:30.714 INFO:tasks.rados.rados.0.smithi107.stdout:4413: left oid 10 (ObjNum 4412 snap 0 seq_num 4412) 2020-06-12T02:38:30.714 INFO:tasks.rados.rados.0.smithi107.stdout:6783: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:30.714 INFO:tasks.rados.rados.0.smithi107.stdout:6783: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:30.714 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1203 (ObjNum 6782 snap 0 seq_num 6782) dirty exists 2020-06-12T02:38:30.715 INFO:tasks.rados.rados.0.smithi107.stdout:6783: left oid 3 (ObjNum 6782 snap 0 seq_num 6782) 2020-06-12T02:38:30.715 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:30.709+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:30.715 INFO:tasks.rados.rados.0.smithi107.stdout:4413: done (3 left) 2020-06-12T02:38:30.716 INFO:tasks.rados.rados.0.smithi107.stdout:6783: done (2 left) 2020-06-12T02:38:30.716 INFO:tasks.rados.rados.0.smithi107.stdout:6787: done (1 left) 2020-06-12T02:38:30.716 INFO:tasks.rados.rados.0.smithi107.stdout:6788: delete oid 6 current snap is 0 2020-06-12T02:38:30.717 INFO:tasks.rados.rados.0.smithi107.stdout:6786: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:30.723 INFO:tasks.rados.rados.0.smithi107.stdout:6786: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:30.723 INFO:tasks.rados.rados.0.smithi107.stdout:6786: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:30.724 INFO:tasks.rados.rados.0.smithi107.stdout:6788: done (1 left) 2020-06-12T02:38:30.724 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:30.724 INFO:tasks.rados.rados.0.smithi107.stdout:6789: seq_num 6788 ranges {573440=491520,1064960=401408} 2020-06-12T02:38:30.728 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-06-12T02:38:30.729 INFO:tasks.rados.rados.0.smithi107.stdout:6789: writing smithi10729407-16 from 573440 to 1064960 tid 1 2020-06-12T02:38:30.733 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:30.733 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:30.734 INFO:tasks.rados.rados.0.smithi107.stdout:6789: writing smithi10729407-16 from 1064960 to 1466368 tid 2 2020-06-12T02:38:30.734 INFO:tasks.rados.rados.0.smithi107.stdout:6790: delete oid 15 current snap is 0 2020-06-12T02:38:30.734 INFO:tasks.rados.rados.0.smithi107.stdout:6786: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:30.735 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1236 (ObjNum 6785 snap 0 seq_num 6785) dirty exists 2020-06-12T02:38:30.735 INFO:tasks.rados.rados.0.smithi107.stdout:6786: left oid 19 (ObjNum 6785 snap 0 seq_num 6785) 2020-06-12T02:38:30.736 INFO:tasks.rados.rados.0.smithi107.stdout:6789: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:30.737 INFO:tasks.rados.rados.0.smithi107.stdout:6786: done (2 left) 2020-06-12T02:38:30.737 INFO:tasks.rados.rados.0.smithi107.stdout:6790: done (1 left) 2020-06-12T02:38:30.738 INFO:tasks.rados.rados.0.smithi107.stdout:6791: delete oid 7 current snap is 0 2020-06-12T02:38:30.742 INFO:tasks.rados.rados.0.smithi107.stdout:6791: done (1 left) 2020-06-12T02:38:30.742 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:30.742 INFO:tasks.rados.rados.0.smithi107.stdout:6792: seq_num 6791 ranges {0=540672,540672=401408} 2020-06-12T02:38:30.744 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-06-12T02:38:30.746 INFO:tasks.rados.rados.0.smithi107.stdout:6792: writing smithi10729407-5 from 0 to 540672 tid 1 2020-06-12T02:38:30.748 INFO:tasks.rados.rados.0.smithi107.stdout:6792: writing smithi10729407-5 from 540672 to 942080 tid 2 2020-06-12T02:38:30.748 INFO:tasks.rados.rados.0.smithi107.stdout:6793: delete oid 19 current snap is 0 2020-06-12T02:38:30.750 INFO:tasks.rados.rados.0.smithi107.stdout:6789: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:30.751 INFO:tasks.rados.rados.0.smithi107.stdout:6789: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:30.751 INFO:tasks.rados.rados.0.smithi107.stdout:6789: finishing write tid 5 to smithi10729407-16 2020-06-12T02:38:30.751 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1206 (ObjNum 6788 snap 0 seq_num 6788) dirty exists 2020-06-12T02:38:30.751 INFO:tasks.rados.rados.0.smithi107.stdout:6789: left oid 16 (ObjNum 6788 snap 0 seq_num 6788) 2020-06-12T02:38:30.752 INFO:tasks.rados.rados.0.smithi107.stdout:6789: done (2 left) 2020-06-12T02:38:30.752 INFO:tasks.rados.rados.0.smithi107.stdout:6793: done (1 left) 2020-06-12T02:38:30.752 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:38:30.752 INFO:tasks.rados.rados.0.smithi107.stdout:6794: seq_num 6793 ranges {0=573440,573440=401408} 2020-06-12T02:38:30.756 INFO:tasks.rados.rados.0.smithi107.stdout:6794: writing smithi10729407-2 from 0 to 573440 tid 1 2020-06-12T02:38:30.758 INFO:tasks.rados.rados.0.smithi107.stdout:6794: writing smithi10729407-2 from 573440 to 974848 tid 2 2020-06-12T02:38:30.758 INFO:tasks.rados.rados.0.smithi107.stdout:6795: delete oid 12 current snap is 0 2020-06-12T02:38:30.758 INFO:tasks.rados.rados.0.smithi107.stdout:6792: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:30.759 INFO:tasks.rados.rados.0.smithi107.stdout:6792: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:30.759 INFO:tasks.rados.rados.0.smithi107.stdout:6792: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:30.759 INFO:tasks.rados.rados.0.smithi107.stdout:6792: finishing write tid 5 to smithi10729407-5 2020-06-12T02:38:30.759 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 679 (ObjNum 6791 snap 0 seq_num 6791) dirty exists 2020-06-12T02:38:30.759 INFO:tasks.rados.rados.0.smithi107.stdout:6792: left oid 5 (ObjNum 6791 snap 0 seq_num 6791) 2020-06-12T02:38:30.761 INFO:tasks.rados.rados.0.smithi107.stdout:6792: done (2 left) 2020-06-12T02:38:30.761 INFO:tasks.rados.rados.0.smithi107.stdout:6795: done (1 left) 2020-06-12T02:38:30.761 INFO:tasks.rados.rados.0.smithi107.stdout:6796: delete oid 3 current snap is 0 2020-06-12T02:38:30.762 INFO:tasks.rados.rados.0.smithi107.stdout:6794: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:30.764 INFO:tasks.rados.rados.0.smithi107.stdout:6796: done (1 left) 2020-06-12T02:38:30.765 INFO:tasks.rados.rados.0.smithi107.stdout:6797: delete oid 10 current snap is 0 2020-06-12T02:38:30.768 INFO:tasks.rados.rados.0.smithi107.stdout:6797: done (1 left) 2020-06-12T02:38:30.768 INFO:tasks.rados.rados.0.smithi107.stdout:6798: delete oid 6 current snap is 0 2020-06-12T02:38:30.768 INFO:tasks.rados.rados.0.smithi107.stdout:6794: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:30.770 INFO:tasks.rados.rados.0.smithi107.stdout:6794: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:30.771 INFO:tasks.rados.rados.0.smithi107.stdout:6798: done (1 left) 2020-06-12T02:38:30.771 INFO:tasks.rados.rados.0.smithi107.stdout:6799: delete oid 3 current snap is 0 2020-06-12T02:38:30.773 INFO:tasks.rados.rados.0.smithi107.stdout:6794: finishing write tid 5 to smithi10729407-2 2020-06-12T02:38:30.773 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 654 (ObjNum 6793 snap 0 seq_num 6793) dirty exists 2020-06-12T02:38:30.774 INFO:tasks.rados.rados.0.smithi107.stdout:6794: left oid 2 (ObjNum 6793 snap 0 seq_num 6793) 2020-06-12T02:38:30.776 INFO:tasks.rados.rados.0.smithi107.stdout:6794: done (1 left) 2020-06-12T02:38:30.776 INFO:tasks.rados.rados.0.smithi107.stdout:6799: done (0 left) 2020-06-12T02:38:30.777 INFO:tasks.rados.rados.0.smithi107.stdout:6800: delete oid 8 current snap is 0 2020-06-12T02:38:30.779 INFO:tasks.rados.rados.0.smithi107.stdout:6800: done (0 left) 2020-06-12T02:38:30.779 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:30.779 INFO:tasks.rados.rados.0.smithi107.stdout:6801: seq_num 6800 ranges {0=696320,696320=401408} 2020-06-12T02:38:30.783 INFO:tasks.rados.rados.0.smithi107.stdout:6801: writing smithi10729407-18 from 0 to 696320 tid 1 2020-06-12T02:38:30.786 INFO:tasks.rados.rados.0.smithi107.stdout:6801: writing smithi10729407-18 from 696320 to 1097728 tid 2 2020-06-12T02:38:30.786 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:30.787 INFO:tasks.rados.rados.0.smithi107.stdout:6802: seq_num 6801 ranges {0=712704,712704=401408} 2020-06-12T02:38:30.790 INFO:tasks.rados.rados.0.smithi107.stdout:6802: writing smithi10729407-19 from 0 to 712704 tid 1 2020-06-12T02:38:30.793 INFO:tasks.rados.rados.0.smithi107.stdout:6802: writing smithi10729407-19 from 712704 to 1114112 tid 2 2020-06-12T02:38:30.793 INFO:tasks.rados.rados.0.smithi107.stdout:6801: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:30.794 INFO:tasks.rados.rados.0.smithi107.stdout:6803: delete oid 11 current snap is 0 2020-06-12T02:38:30.797 INFO:tasks.rados.rados.0.smithi107.stdout:6803: done (2 left) 2020-06-12T02:38:30.798 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:38:30.798 INFO:tasks.rados.rados.0.smithi107.stdout:6804: seq_num 6803 ranges {0=745472,745472=401408} 2020-06-12T02:38:30.802 INFO:tasks.rados.rados.0.smithi107.stdout:6804: writing smithi10729407-20 from 0 to 745472 tid 1 2020-06-12T02:38:30.805 INFO:tasks.rados.rados.0.smithi107.stdout:6804: writing smithi10729407-20 from 745472 to 1146880 tid 2 2020-06-12T02:38:30.805 INFO:tasks.rados.rados.0.smithi107.stdout:6805: delete oid 6 current snap is 0 2020-06-12T02:38:30.805 INFO:tasks.rados.rados.0.smithi107.stdout:6802: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:30.806 INFO:tasks.rados.rados.0.smithi107.stdout:6801: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:30.806 INFO:tasks.rados.rados.0.smithi107.stdout:6801: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:30.807 INFO:tasks.rados.rados.0.smithi107.stdout:6801: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:30.807 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1251 (ObjNum 6800 snap 0 seq_num 6800) dirty exists 2020-06-12T02:38:30.808 INFO:tasks.rados.rados.0.smithi107.stdout:6801: left oid 18 (ObjNum 6800 snap 0 seq_num 6800) 2020-06-12T02:38:30.809 INFO:tasks.rados.rados.0.smithi107.stdout:6802: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:30.809 INFO:tasks.rados.rados.0.smithi107.stdout:6802: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:30.810 INFO:tasks.rados.rados.0.smithi107.stdout:6802: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:30.810 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1241 (ObjNum 6801 snap 0 seq_num 6801) dirty exists 2020-06-12T02:38:30.810 INFO:tasks.rados.rados.0.smithi107.stdout:6802: left oid 19 (ObjNum 6801 snap 0 seq_num 6801) 2020-06-12T02:38:30.811 INFO:tasks.rados.rados.0.smithi107.stdout:6801: done (3 left) 2020-06-12T02:38:30.811 INFO:tasks.rados.rados.0.smithi107.stdout:6802: done (2 left) 2020-06-12T02:38:30.812 INFO:tasks.rados.rados.0.smithi107.stdout:6805: done (1 left) 2020-06-12T02:38:30.812 INFO:tasks.rados.rados.0.smithi107.stdout:6806: delete oid 8 current snap is 0 2020-06-12T02:38:30.814 INFO:tasks.rados.rados.0.smithi107.stdout:6804: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:30.814 INFO:tasks.rados.rados.0.smithi107.stdout:6804: finishing write tid 2 to smithi10729407-20 2020-06-12T02:38:30.815 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:30.810+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:30.815 INFO:tasks.rados.rados.0.smithi107.stdout:6804: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:30.816 INFO:tasks.rados.rados.0.smithi107.stdout:6806: done (1 left) 2020-06-12T02:38:30.816 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:30.816 INFO:tasks.rados.rados.0.smithi107.stdout:6807: seq_num 6806 ranges {0=794624,794624=401408} 2020-06-12T02:38:30.819 INFO:tasks.rados.rados.0.smithi107.stdout:6807: writing smithi10729407-8 from 0 to 794624 tid 1 2020-06-12T02:38:30.821 INFO:teuthology.orchestra.run.smithi107.stderr:Error EPERM: min_compat_client jewel < luminous, which is required for pg-upmap. Try 'ceph osd set-require-min-compat-client luminous' before using the new interface 2020-06-12T02:38:30.822 INFO:tasks.rados.rados.0.smithi107.stdout:6807: writing smithi10729407-8 from 794624 to 1196032 tid 2 2020-06-12T02:38:30.822 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:30.822 INFO:tasks.rados.rados.0.smithi107.stdout:6808: seq_num 6807 ranges {2195456=409600,2605056=720896,3325952=81920} 2020-06-12T02:38:30.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-06-12T02:38:30.824 INFO:tasks.thrashosds.thrasher:Failed to rm-pg-upmap-items, ignoring 2020-06-12T02:38:30.836 INFO:tasks.rados.rados.0.smithi107.stdout:6808: writing smithi10729407-4 from 2195456 to 2605056 tid 1 2020-06-12T02:38:30.842 INFO:tasks.rados.rados.0.smithi107.stdout:6808: writing smithi10729407-4 from 2605056 to 3325952 tid 2 2020-06-12T02:38:30.842 INFO:tasks.rados.rados.0.smithi107.stdout:6808: writing smithi10729407-4 from 3325952 to 3407872 tid 3 2020-06-12T02:38:30.843 INFO:tasks.rados.rados.0.smithi107.stdout:6809: delete oid 19 current snap is 0 2020-06-12T02:38:30.843 INFO:tasks.rados.rados.0.smithi107.stdout:6804: finishing write tid 5 to smithi10729407-20 2020-06-12T02:38:30.843 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 522 (ObjNum 6803 snap 0 seq_num 6803) dirty exists 2020-06-12T02:38:30.843 INFO:tasks.rados.rados.0.smithi107.stdout:6804: left oid 20 (ObjNum 6803 snap 0 seq_num 6803) 2020-06-12T02:38:30.843 INFO:tasks.rados.rados.0.smithi107.stdout:6807: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:30.844 INFO:tasks.rados.rados.0.smithi107.stdout:6807: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:30.844 INFO:tasks.rados.rados.0.smithi107.stdout:6807: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:30.844 INFO:tasks.rados.rados.0.smithi107.stdout:6807: finishing write tid 5 to smithi10729407-8 2020-06-12T02:38:30.844 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1281 (ObjNum 6806 snap 0 seq_num 6806) dirty exists 2020-06-12T02:38:30.844 INFO:tasks.rados.rados.0.smithi107.stdout:6807: left oid 8 (ObjNum 6806 snap 0 seq_num 6806) 2020-06-12T02:38:30.844 INFO:tasks.rados.rados.0.smithi107.stdout:6808: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:30.845 INFO:tasks.rados.rados.0.smithi107.stdout:6804: done (3 left) 2020-06-12T02:38:30.845 INFO:tasks.rados.rados.0.smithi107.stdout:6807: done (2 left) 2020-06-12T02:38:30.845 INFO:tasks.rados.rados.0.smithi107.stdout:6809: done (1 left) 2020-06-12T02:38:30.846 INFO:tasks.rados.rados.0.smithi107.stdout:6810: delete oid 12 current snap is 0 2020-06-12T02:38:30.848 INFO:tasks.rados.rados.0.smithi107.stdout:6810: done (1 left) 2020-06-12T02:38:30.849 INFO:tasks.rados.rados.0.smithi107.stdout:6811: delete oid 16 current snap is 0 2020-06-12T02:38:30.849 INFO:tasks.rados.rados.0.smithi107.stdout:6808: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:30.850 INFO:tasks.rados.rados.0.smithi107.stdout:6808: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:30.850 INFO:tasks.rados.rados.0.smithi107.stdout:6808: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:30.851 INFO:tasks.rados.rados.0.smithi107.stdout:6811: done (1 left) 2020-06-12T02:38:30.851 INFO:tasks.rados.rados.0.smithi107.stdout:6812: delete oid 19 current snap is 0 2020-06-12T02:38:30.852 INFO:tasks.rados.rados.0.smithi107.stdout:6808: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:30.852 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1285 (ObjNum 6807 snap 0 seq_num 6807) dirty exists 2020-06-12T02:38:30.852 INFO:tasks.rados.rados.0.smithi107.stdout:6808: left oid 4 (ObjNum 6807 snap 0 seq_num 6807) 2020-06-12T02:38:30.854 INFO:tasks.rados.rados.0.smithi107.stdout:6808: done (1 left) 2020-06-12T02:38:30.854 INFO:tasks.rados.rados.0.smithi107.stdout:6812: done (0 left) 2020-06-12T02:38:30.854 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:30.854 INFO:tasks.rados.rados.0.smithi107.stdout:6813: seq_num 6812 ranges {1097728=491520,1589248=589824,2179072=212992} 2020-06-12T02:38:30.863 INFO:tasks.rados.rados.0.smithi107.stdout:6813: writing smithi10729407-18 from 1097728 to 1589248 tid 1 2020-06-12T02:38:30.866 INFO:tasks.rados.rados.0.smithi107.stdout:6813: writing smithi10729407-18 from 1589248 to 2179072 tid 2 2020-06-12T02:38:30.867 INFO:tasks.rados.rados.0.smithi107.stdout:6813: writing smithi10729407-18 from 2179072 to 2392064 tid 3 2020-06-12T02:38:30.868 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:30.868 INFO:tasks.rados.rados.0.smithi107.stdout:6814: seq_num 6813 ranges {0=507904,507904=475136,983040=327680} 2020-06-12T02:38:30.871 INFO:tasks.rados.rados.0.smithi107.stdout:6814: writing smithi10729407-17 from 0 to 507904 tid 1 2020-06-12T02:38:30.874 INFO:tasks.rados.rados.0.smithi107.stdout:6814: writing smithi10729407-17 from 507904 to 983040 tid 2 2020-06-12T02:38:30.876 INFO:tasks.rados.rados.0.smithi107.stdout:6814: writing smithi10729407-17 from 983040 to 1310720 tid 3 2020-06-12T02:38:30.876 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:30.877 INFO:tasks.rados.rados.0.smithi107.stdout:6815: seq_num 6814 ranges {0=524288,524288=753664,1277952=49152} 2020-06-12T02:38:30.879 INFO:tasks.rados.rados.0.smithi107.stdout:6815: writing smithi10729407-9 from 0 to 524288 tid 1 2020-06-12T02:38:30.884 INFO:tasks.rados.rados.0.smithi107.stdout:6815: writing smithi10729407-9 from 524288 to 1277952 tid 2 2020-06-12T02:38:30.885 INFO:tasks.rados.rados.0.smithi107.stdout:6815: writing smithi10729407-9 from 1277952 to 1327104 tid 3 2020-06-12T02:38:30.885 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:38:30.885 INFO:tasks.rados.rados.0.smithi107.stdout:6816: seq_num 6815 ranges {0=540672,540672=638976,1179648=163840} 2020-06-12T02:38:30.888 INFO:tasks.rados.rados.0.smithi107.stdout:6816: writing smithi10729407-10 from 0 to 540672 tid 1 2020-06-12T02:38:30.893 INFO:tasks.rados.rados.0.smithi107.stdout:6816: writing smithi10729407-10 from 540672 to 1179648 tid 2 2020-06-12T02:38:30.897 INFO:tasks.rados.rados.0.smithi107.stdout:6816: writing smithi10729407-10 from 1179648 to 1343488 tid 3 2020-06-12T02:38:30.897 INFO:tasks.rados.rados.0.smithi107.stdout:6813: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:30.897 INFO:tasks.rados.rados.0.smithi107.stdout:6813: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:30.897 INFO:tasks.rados.rados.0.smithi107.stdout:6813: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:30.898 INFO:tasks.rados.rados.0.smithi107.stdout:6814: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:30.898 INFO:tasks.rados.rados.0.smithi107.stdout:6813: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:30.898 INFO:tasks.rados.rados.0.smithi107.stdout:6813: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:30.898 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1255 (ObjNum 6812 snap 0 seq_num 6812) dirty exists 2020-06-12T02:38:30.898 INFO:tasks.rados.rados.0.smithi107.stdout:6813: left oid 18 (ObjNum 6812 snap 0 seq_num 6812) 2020-06-12T02:38:30.898 INFO:tasks.rados.rados.0.smithi107.stdout:6814: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:30.899 INFO:tasks.rados.rados.0.smithi107.stdout:6814: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:30.899 INFO:tasks.rados.rados.0.smithi107.stdout:6815: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:30.899 INFO:tasks.rados.rados.0.smithi107.stdout:6814: finishing write tid 5 to smithi10729407-17 2020-06-12T02:38:30.899 INFO:tasks.rados.rados.0.smithi107.stdout:6814: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:30.899 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1289 (ObjNum 6813 snap 0 seq_num 6813) dirty exists 2020-06-12T02:38:30.900 INFO:tasks.rados.rados.0.smithi107.stdout:6814: left oid 17 (ObjNum 6813 snap 0 seq_num 6813) 2020-06-12T02:38:30.900 INFO:tasks.rados.rados.0.smithi107.stdout:6813: done (3 left) 2020-06-12T02:38:30.900 INFO:tasks.rados.rados.0.smithi107.stdout:6814: done (2 left) 2020-06-12T02:38:30.900 INFO:tasks.rados.rados.0.smithi107.stdout:6817: delete oid 16 current snap is 0 2020-06-12T02:38:30.901 INFO:tasks.rados.rados.0.smithi107.stdout:6815: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:30.901 INFO:tasks.rados.rados.0.smithi107.stdout:6817: done (2 left) 2020-06-12T02:38:30.901 INFO:tasks.rados.rados.0.smithi107.stdout:6818: delete oid 20 current snap is 0 2020-06-12T02:38:30.902 INFO:tasks.rados.rados.0.smithi107.stdout:6815: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:30.904 INFO:tasks.rados.rados.0.smithi107.stdout:6815: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:30.906 INFO:tasks.rados.rados.0.smithi107.stdout:6816: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:30.908 INFO:tasks.rados.rados.0.smithi107.stdout:6818: done (2 left) 2020-06-12T02:38:30.908 INFO:tasks.rados.rados.0.smithi107.stdout:6819: delete oid 2 current snap is 0 2020-06-12T02:38:30.909 INFO:tasks.rados.rados.0.smithi107.stdout:6815: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:30.909 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1259 (ObjNum 6814 snap 0 seq_num 6814) dirty exists 2020-06-12T02:38:30.909 INFO:tasks.rados.rados.0.smithi107.stdout:6815: left oid 9 (ObjNum 6814 snap 0 seq_num 6814) 2020-06-12T02:38:30.910 INFO:tasks.rados.rados.0.smithi107.stdout:6816: finishing write tid 2 to smithi10729407-10 2020-06-12T02:38:30.911 INFO:tasks.rados.rados.0.smithi107.stdout:6816: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:30.911 INFO:tasks.rados.rados.0.smithi107.stdout:6816: finishing write tid 5 to smithi10729407-10 2020-06-12T02:38:30.912 INFO:tasks.rados.rados.0.smithi107.stdout:6815: done (2 left) 2020-06-12T02:38:30.912 INFO:tasks.rados.rados.0.smithi107.stdout:6819: done (1 left) 2020-06-12T02:38:30.912 INFO:tasks.rados.rados.0.smithi107.stdout:6820: delete oid 13 current snap is 0 2020-06-12T02:38:30.913 INFO:tasks.rados.rados.0.smithi107.stdout:6816: finishing write tid 6 to smithi10729407-10 2020-06-12T02:38:30.913 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 444 (ObjNum 6815 snap 0 seq_num 6815) dirty exists 2020-06-12T02:38:30.913 INFO:tasks.rados.rados.0.smithi107.stdout:6816: left oid 10 (ObjNum 6815 snap 0 seq_num 6815) 2020-06-12T02:38:30.915 INFO:tasks.rados.rados.0.smithi107.stdout:6816: done (1 left) 2020-06-12T02:38:30.915 INFO:tasks.rados.rados.0.smithi107.stdout:6820: done (0 left) 2020-06-12T02:38:30.915 INFO:tasks.rados.rados.0.smithi107.stdout:6821: delete oid 9 current snap is 0 2020-06-12T02:38:30.916 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:30.911+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:30.917 INFO:tasks.rados.rados.0.smithi107.stdout:6821: done (0 left) 2020-06-12T02:38:30.917 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:30.917 INFO:tasks.rados.rados.0.smithi107.stdout:6822: seq_num 6821 ranges {942080=647168,1589248=794624,2383872=8192} 2020-06-12T02:38:30.919 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:30.920 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:30.924 INFO:tasks.rados.rados.0.smithi107.stdout:6822: writing smithi10729407-5 from 942080 to 1589248 tid 1 2020-06-12T02:38:30.931 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-06-12T02:38:30.932 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:38:30.932 INFO:tasks.rados.rados.0.smithi107.stdout:6822: writing smithi10729407-5 from 1589248 to 2383872 tid 2 2020-06-12T02:38:30.932 INFO:tasks.rados.rados.0.smithi107.stdout:6822: writing smithi10729407-5 from 2383872 to 2392064 tid 3 2020-06-12T02:38:30.933 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:30.933 INFO:tasks.rados.rados.0.smithi107.stdout:6823: seq_num 6822 ranges {0=663552,663552=679936,1343488=122880} 2020-06-12T02:38:30.934 INFO:tasks.rados.rados.0.smithi107.stdout:6823: writing smithi10729407-13 from 0 to 663552 tid 1 2020-06-12T02:38:30.938 INFO:tasks.rados.rados.0.smithi107.stdout:6823: writing smithi10729407-13 from 663552 to 1343488 tid 2 2020-06-12T02:38:30.939 INFO:tasks.rados.rados.0.smithi107.stdout:6823: writing smithi10729407-13 from 1343488 to 1466368 tid 3 2020-06-12T02:38:30.939 INFO:tasks.rados.rados.0.smithi107.stdout:6824: delete oid 12 current snap is 0 2020-06-12T02:38:30.939 INFO:tasks.rados.rados.0.smithi107.stdout:6822: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:30.940 INFO:tasks.rados.rados.0.smithi107.stdout:6822: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:30.940 INFO:tasks.rados.rados.0.smithi107.stdout:6822: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:30.941 INFO:tasks.rados.rados.0.smithi107.stdout:6822: finishing write tid 5 to smithi10729407-5 2020-06-12T02:38:30.941 INFO:tasks.rados.rados.0.smithi107.stdout:6823: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:30.942 INFO:tasks.rados.rados.0.smithi107.stdout:6824: done (2 left) 2020-06-12T02:38:30.942 INFO:tasks.rados.rados.0.smithi107.stdout:6825: delete oid 8 current snap is 0 2020-06-12T02:38:30.943 INFO:tasks.rados.rados.0.smithi107.stdout:6822: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:30.943 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 683 (ObjNum 6821 snap 0 seq_num 6821) dirty exists 2020-06-12T02:38:30.943 INFO:tasks.rados.rados.0.smithi107.stdout:6822: left oid 5 (ObjNum 6821 snap 0 seq_num 6821) 2020-06-12T02:38:30.946 INFO:tasks.rados.rados.0.smithi107.stdout:6822: done (2 left) 2020-06-12T02:38:30.947 INFO:tasks.rados.rados.0.smithi107.stdout:6825: done (1 left) 2020-06-12T02:38:30.947 INFO:tasks.rados.rados.0.smithi107.stdout:6826: delete oid 18 current snap is 0 2020-06-12T02:38:30.952 INFO:tasks.rados.rados.0.smithi107.stdout:6823: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:30.952 INFO:tasks.rados.rados.0.smithi107.stdout:6826: done (1 left) 2020-06-12T02:38:30.952 INFO:tasks.rados.rados.0.smithi107.stdout:6827: delete oid 4 current snap is 0 2020-06-12T02:38:30.954 INFO:tasks.rados.rados.0.smithi107.stdout:6823: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:30.955 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:38:30.956 INFO:tasks.rados.rados.0.smithi107.stdout:6823: finishing write tid 5 to smithi10729407-13 2020-06-12T02:38:30.958 INFO:tasks.rados.rados.0.smithi107.stdout:6827: done (1 left) 2020-06-12T02:38:30.958 INFO:tasks.rados.rados.0.smithi107.stdout:6828: delete oid 2 current snap is 0 2020-06-12T02:38:30.959 INFO:tasks.rados.rados.0.smithi107.stdout:6823: finishing write tid 6 to smithi10729407-13 2020-06-12T02:38:30.959 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1284 (ObjNum 6822 snap 0 seq_num 6822) dirty exists 2020-06-12T02:38:30.959 INFO:tasks.rados.rados.0.smithi107.stdout:6823: left oid 13 (ObjNum 6822 snap 0 seq_num 6822) 2020-06-12T02:38:30.960 INFO:tasks.rados.rados.0.smithi107.stdout:6823: done (1 left) 2020-06-12T02:38:30.961 INFO:tasks.rados.rados.0.smithi107.stdout:6828: done (0 left) 2020-06-12T02:38:30.961 INFO:tasks.rados.rados.0.smithi107.stdout:6829: delete oid 16 current snap is 0 2020-06-12T02:38:30.964 INFO:tasks.rados.rados.0.smithi107.stdout:6829: done (0 left) 2020-06-12T02:38:30.964 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:30.964 INFO:tasks.rados.rados.0.smithi107.stdout:6830: seq_num 6829 ranges {0=778240,778240=720896,1499136=81920} 2020-06-12T02:38:30.968 INFO:tasks.rados.rados.0.smithi107.stdout:6830: writing smithi10729407-9 from 0 to 778240 tid 1 2020-06-12T02:38:30.975 INFO:tasks.rados.rados.0.smithi107.stdout:6830: writing smithi10729407-9 from 778240 to 1499136 tid 2 2020-06-12T02:38:30.976 INFO:tasks.rados.rados.0.smithi107.stdout:6830: writing smithi10729407-9 from 1499136 to 1581056 tid 3 2020-06-12T02:38:30.976 INFO:tasks.rados.rados.0.smithi107.stdout:6831: delete oid 10 current snap is 0 2020-06-12T02:38:30.976 INFO:tasks.rados.rados.0.smithi107.stdout:6830: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:30.979 INFO:tasks.rados.rados.0.smithi107.stdout:6831: done (1 left) 2020-06-12T02:38:30.979 INFO:tasks.rados.rados.0.smithi107.stdout:6832: delete oid 8 current snap is 0 2020-06-12T02:38:30.982 INFO:tasks.rados.rados.0.smithi107.stdout:6832: done (1 left) 2020-06-12T02:38:30.982 INFO:tasks.rados.rados.0.smithi107.stdout:6833: delete oid 3 current snap is 0 2020-06-12T02:38:30.984 INFO:tasks.rados.rados.0.smithi107.stdout:6830: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:30.985 INFO:tasks.rados.rados.0.smithi107.stdout:6833: done (1 left) 2020-06-12T02:38:30.985 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:30.985 INFO:tasks.rados.rados.0.smithi107.stdout:6834: seq_num 6833 ranges {1310720=442368,1753088=655360,2408448=548864} 2020-06-12T02:38:30.986 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:38:31.001 INFO:tasks.rados.rados.0.smithi107.stdout:6834: writing smithi10729407-17 from 1310720 to 1753088 tid 1 2020-06-12T02:38:31.001 INFO:tasks.rados.rados.0.smithi107.stdout:6834: writing smithi10729407-17 from 1753088 to 2408448 tid 2 2020-06-12T02:38:31.002 INFO:tasks.rados.rados.0.smithi107.stdout:6834: writing smithi10729407-17 from 2408448 to 2957312 tid 3 2020-06-12T02:38:31.002 INFO:tasks.rados.rados.0.smithi107.stdout:6835: delete oid 12 current snap is 0 2020-06-12T02:38:31.003 INFO:tasks.rados.rados.0.smithi107.stdout:6830: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:31.003 INFO:tasks.rados.rados.0.smithi107.stdout:6830: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:31.003 INFO:tasks.rados.rados.0.smithi107.stdout:6830: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:31.003 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1265 (ObjNum 6829 snap 0 seq_num 6829) dirty exists 2020-06-12T02:38:31.003 INFO:tasks.rados.rados.0.smithi107.stdout:6830: left oid 9 (ObjNum 6829 snap 0 seq_num 6829) 2020-06-12T02:38:31.004 INFO:tasks.rados.rados.0.smithi107.stdout:6834: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:31.005 INFO:tasks.rados.rados.0.smithi107.stdout:6830: done (2 left) 2020-06-12T02:38:31.005 INFO:tasks.rados.rados.0.smithi107.stdout:6835: done (1 left) 2020-06-12T02:38:31.005 INFO:tasks.rados.rados.0.smithi107.stdout:6836: delete oid 7 current snap is 0 2020-06-12T02:38:31.008 INFO:tasks.rados.rados.0.smithi107.stdout:6836: done (1 left) 2020-06-12T02:38:31.008 INFO:tasks.rados.rados.0.smithi107.stdout:6837: delete oid 11 current snap is 0 2020-06-12T02:38:31.009 INFO:tasks.rados.rados.0.smithi107.stdout:6834: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:31.009 INFO:tasks.rados.rados.0.smithi107.stdout:6834: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:31.010 INFO:tasks.rados.rados.0.smithi107.stdout:6834: finishing write tid 5 to smithi10729407-17 2020-06-12T02:38:31.011 INFO:tasks.rados.rados.0.smithi107.stdout:6837: done (1 left) 2020-06-12T02:38:31.011 INFO:tasks.rados.rados.0.smithi107.stdout:6838: delete oid 3 current snap is 0 2020-06-12T02:38:31.012 INFO:tasks.rados.rados.0.smithi107.stdout:6834: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:31.012 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1294 (ObjNum 6833 snap 0 seq_num 6833) dirty exists 2020-06-12T02:38:31.012 INFO:tasks.rados.rados.0.smithi107.stdout:6834: left oid 17 (ObjNum 6833 snap 0 seq_num 6833) 2020-06-12T02:38:31.013 INFO:tasks.rados.rados.0.smithi107.stdout:6834: done (1 left) 2020-06-12T02:38:31.013 INFO:tasks.rados.rados.0.smithi107.stdout:6838: done (0 left) 2020-06-12T02:38:31.014 INFO:tasks.rados.rados.0.smithi107.stdout:6839: delete oid 5 current snap is 0 2020-06-12T02:38:31.016 INFO:tasks.rados.rados.0.smithi107.stdout:6839: done (0 left) 2020-06-12T02:38:31.016 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:31.016 INFO:tasks.rados.rados.0.smithi107.stdout:6840: seq_num 6839 ranges {0=548864,548864=409600,958464=647168,1605632=147456} 2020-06-12T02:38:31.017 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:31.013+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:31.019 INFO:tasks.rados.rados.0.smithi107.stdout:6840: writing smithi10729407-4 from 0 to 548864 tid 1 2020-06-12T02:38:31.022 INFO:tasks.rados.rados.0.smithi107.stdout:6840: writing smithi10729407-4 from 548864 to 958464 tid 2 2020-06-12T02:38:31.026 INFO:tasks.rados.rados.0.smithi107.stdout:6840: writing smithi10729407-4 from 958464 to 1605632 tid 3 2020-06-12T02:38:31.027 INFO:tasks.rados.rados.0.smithi107.stdout:6840: writing smithi10729407-4 from 1605632 to 1753088 tid 4 2020-06-12T02:38:31.027 INFO:tasks.rados.rados.0.smithi107.stdout:6841: delete oid 18 current snap is 0 2020-06-12T02:38:31.027 INFO:tasks.rados.rados.0.smithi107.stdout:6840: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:31.028 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:38:31.034 INFO:tasks.rados.rados.0.smithi107.stdout:6840: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:31.035 INFO:tasks.rados.rados.0.smithi107.stdout:6840: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:31.035 INFO:tasks.rados.rados.0.smithi107.stdout:6840: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:31.037 INFO:tasks.rados.rados.0.smithi107.stdout:6840: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:31.037 INFO:tasks.rados.rados.0.smithi107.stdout:6841: done (1 left) 2020-06-12T02:38:31.037 INFO:tasks.rados.rados.0.smithi107.stdout:6842: delete oid 2 current snap is 0 2020-06-12T02:38:31.040 INFO:tasks.rados.rados.0.smithi107.stdout:6840: finishing write tid 7 to smithi10729407-4 2020-06-12T02:38:31.040 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1299 (ObjNum 6839 snap 0 seq_num 6839) dirty exists 2020-06-12T02:38:31.040 INFO:tasks.rados.rados.0.smithi107.stdout:6840: left oid 4 (ObjNum 6839 snap 0 seq_num 6839) 2020-06-12T02:38:31.041 INFO:tasks.rados.rados.0.smithi107.stdout:6840: done (1 left) 2020-06-12T02:38:31.041 INFO:tasks.rados.rados.0.smithi107.stdout:6842: done (0 left) 2020-06-12T02:38:31.041 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:31.041 INFO:tasks.rados.rados.0.smithi107.stdout:6843: seq_num 6842 ranges {0=598016,598016=466944,1064960=417792,1482752=319488} 2020-06-12T02:38:31.044 INFO:tasks.rados.rados.0.smithi107.stdout:6843: writing smithi10729407-19 from 0 to 598016 tid 1 2020-06-12T02:38:31.046 INFO:tasks.rados.rados.0.smithi107.stdout:6843: writing smithi10729407-19 from 598016 to 1064960 tid 2 2020-06-12T02:38:31.049 INFO:tasks.rados.rados.0.smithi107.stdout:6843: writing smithi10729407-19 from 1064960 to 1482752 tid 3 2020-06-12T02:38:31.051 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:38:31.052 INFO:tasks.rados.rados.0.smithi107.stdout:6843: writing smithi10729407-19 from 1482752 to 1802240 tid 4 2020-06-12T02:38:31.052 INFO:tasks.rados.rados.0.smithi107.stdout:6844: delete oid 1 current snap is 0 2020-06-12T02:38:31.052 INFO:tasks.rados.rados.0.smithi107.stdout:6843: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:31.054 INFO:tasks.rados.rados.0.smithi107.stdout:6844: done (1 left) 2020-06-12T02:38:31.054 INFO:tasks.rados.rados.0.smithi107.stdout:6845: delete oid 12 current snap is 0 2020-06-12T02:38:31.059 INFO:tasks.rados.rados.0.smithi107.stdout:6843: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:31.063 INFO:tasks.rados.rados.0.smithi107.stdout:6843: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:31.064 INFO:tasks.rados.rados.0.smithi107.stdout:6843: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:31.064 INFO:tasks.rados.rados.0.smithi107.stdout:6843: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:31.067 INFO:tasks.rados.rados.0.smithi107.stdout:6845: done (1 left) 2020-06-12T02:38:31.067 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:31.067 INFO:tasks.rados.rados.0.smithi107.stdout:6846: seq_num 6845 ranges {0=647168,647168=573440,1220608=630784} 2020-06-12T02:38:31.070 INFO:tasks.rados.rados.0.smithi107.stdout:6846: writing smithi10729407-12 from 0 to 647168 tid 1 2020-06-12T02:38:31.074 INFO:tasks.rados.rados.0.smithi107.stdout:6846: writing smithi10729407-12 from 647168 to 1220608 tid 2 2020-06-12T02:38:31.075 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:38:31.079 INFO:tasks.rados.rados.0.smithi107.stdout:6846: writing smithi10729407-12 from 1220608 to 1851392 tid 3 2020-06-12T02:38:31.079 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:31.079 INFO:tasks.rados.rados.0.smithi107.stdout:6847: seq_num 6846 ranges {0=663552,663552=450560,1114112=753664} 2020-06-12T02:38:31.082 INFO:tasks.rados.rados.0.smithi107.stdout:6847: writing smithi10729407-18 from 0 to 663552 tid 1 2020-06-12T02:38:31.086 INFO:tasks.rados.rados.0.smithi107.stdout:6847: writing smithi10729407-18 from 663552 to 1114112 tid 2 2020-06-12T02:38:31.091 INFO:tasks.rados.rados.0.smithi107.stdout:6847: writing smithi10729407-18 from 1114112 to 1867776 tid 3 2020-06-12T02:38:31.091 INFO:tasks.rados.rados.0.smithi107.stdout:6848: delete oid 1 current snap is 0 2020-06-12T02:38:31.091 INFO:tasks.rados.rados.0.smithi107.stdout:6843: finishing write tid 7 to smithi10729407-19 2020-06-12T02:38:31.091 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1251 (ObjNum 6842 snap 0 seq_num 6842) dirty exists 2020-06-12T02:38:31.092 INFO:tasks.rados.rados.0.smithi107.stdout:6843: left oid 19 (ObjNum 6842 snap 0 seq_num 6842) 2020-06-12T02:38:31.092 INFO:tasks.rados.rados.0.smithi107.stdout:6846: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:31.092 INFO:tasks.rados.rados.0.smithi107.stdout:6846: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:31.092 INFO:tasks.rados.rados.0.smithi107.stdout:6846: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:31.092 INFO:tasks.rados.rados.0.smithi107.stdout:6846: finishing write tid 5 to smithi10729407-12 2020-06-12T02:38:31.093 INFO:tasks.rados.rados.0.smithi107.stdout:6846: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:31.093 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1256 (ObjNum 6845 snap 0 seq_num 6845) dirty exists 2020-06-12T02:38:31.093 INFO:tasks.rados.rados.0.smithi107.stdout:6846: left oid 12 (ObjNum 6845 snap 0 seq_num 6845) 2020-06-12T02:38:31.094 INFO:tasks.rados.rados.0.smithi107.stdout:6843: done (3 left) 2020-06-12T02:38:31.094 INFO:tasks.rados.rados.0.smithi107.stdout:6846: done (2 left) 2020-06-12T02:38:31.094 INFO:tasks.rados.rados.0.smithi107.stdout:6848: done (1 left) 2020-06-12T02:38:31.094 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:38:31.094 INFO:tasks.rados.rados.0.smithi107.stdout:6849: seq_num 6848 ranges {0=696320,696320=622592,1318912=417792,1736704=163840} 2020-06-12T02:38:31.114 INFO:tasks.rados.rados.0.smithi107.stdout:6849: writing smithi10729407-20 from 0 to 696320 tid 1 2020-06-12T02:38:31.115 INFO:tasks.rados.rados.0.smithi107.stdout:6849: writing smithi10729407-20 from 696320 to 1318912 tid 2 2020-06-12T02:38:31.115 INFO:tasks.rados.rados.0.smithi107.stdout:6849: writing smithi10729407-20 from 1318912 to 1736704 tid 3 2020-06-12T02:38:31.115 INFO:tasks.rados.rados.0.smithi107.stdout:6849: writing smithi10729407-20 from 1736704 to 1900544 tid 4 2020-06-12T02:38:31.115 INFO:tasks.rados.rados.0.smithi107.stdout:6850: delete oid 1 current snap is 0 2020-06-12T02:38:31.116 INFO:tasks.rados.rados.0.smithi107.stdout:6847: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:31.116 INFO:tasks.rados.rados.0.smithi107.stdout:6847: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:31.116 INFO:tasks.rados.rados.0.smithi107.stdout:6847: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:31.116 INFO:tasks.rados.rados.0.smithi107.stdout:6847: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:31.116 INFO:tasks.rados.rados.0.smithi107.stdout:6847: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:31.117 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1270 (ObjNum 6846 snap 0 seq_num 6846) dirty exists 2020-06-12T02:38:31.117 INFO:tasks.rados.rados.0.smithi107.stdout:6847: left oid 18 (ObjNum 6846 snap 0 seq_num 6846) 2020-06-12T02:38:31.117 INFO:tasks.rados.rados.0.smithi107.stdout:6849: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:31.117 INFO:tasks.rados.rados.0.smithi107.stdout:6847: done (2 left) 2020-06-12T02:38:31.117 INFO:tasks.rados.rados.0.smithi107.stdout:6850: done (1 left) 2020-06-12T02:38:31.118 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:31.118 INFO:tasks.rados.rados.0.smithi107.stdout:6851: seq_num 6850 ranges {1753088=729088,2482176=442368,2924544=761856} 2020-06-12T02:38:31.118 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:38:31.118 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:31.119 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:38:31.119 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2020-06-12T02:38:31.119 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:31.120 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-06-12T02:38:31.121 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:31.117+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:31.130 INFO:tasks.rados.rados.0.smithi107.stdout:6851: writing smithi10729407-4 from 1753088 to 2482176 tid 1 2020-06-12T02:38:31.131 INFO:tasks.rados.rados.0.smithi107.stdout:6851: writing smithi10729407-4 from 2482176 to 2924544 tid 2 2020-06-12T02:38:31.144 INFO:tasks.rados.rados.0.smithi107.stdout:6851: writing smithi10729407-4 from 2924544 to 3686400 tid 3 2020-06-12T02:38:31.144 INFO:tasks.rados.rados.0.smithi107.stdout:6852: delete oid 8 current snap is 0 2020-06-12T02:38:31.144 INFO:tasks.rados.rados.0.smithi107.stdout:6849: finishing write tid 2 to smithi10729407-20 2020-06-12T02:38:31.145 INFO:tasks.rados.rados.0.smithi107.stdout:6849: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:31.145 INFO:tasks.rados.rados.0.smithi107.stdout:6849: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:31.145 INFO:tasks.rados.rados.0.smithi107.stdout:6849: finishing write tid 6 to smithi10729407-20 2020-06-12T02:38:31.145 INFO:tasks.rados.rados.0.smithi107.stdout:6849: finishing write tid 7 to smithi10729407-20 2020-06-12T02:38:31.145 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 528 (ObjNum 6848 snap 0 seq_num 6848) dirty exists 2020-06-12T02:38:31.146 INFO:tasks.rados.rados.0.smithi107.stdout:6849: left oid 20 (ObjNum 6848 snap 0 seq_num 6848) 2020-06-12T02:38:31.146 INFO:tasks.rados.rados.0.smithi107.stdout:6851: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:31.146 INFO:tasks.rados.rados.0.smithi107.stdout:6849: done (2 left) 2020-06-12T02:38:31.146 INFO:tasks.rados.rados.0.smithi107.stdout:6852: done (1 left) 2020-06-12T02:38:31.146 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:38:31.147 INFO:tasks.rados.rados.0.smithi107.stdout:6853: seq_num 6852 ranges {0=761856,761856=606208,1368064=598016} 2020-06-12T02:38:31.147 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2020-06-12T02:38:31.147 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-06-12T02:38:31.147 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-06-12T02:38:31.148 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-06-12T02:38:31.148 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-06-12T02:38:31.148 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-06-12T02:38:31.148 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-06-12T02:38:31.148 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-06-12T02:38:31.149 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-06-12T02:38:31.150 INFO:tasks.rados.rados.0.smithi107.stdout:6853: writing smithi10729407-10 from 0 to 761856 tid 1 2020-06-12T02:38:31.150 INFO:tasks.rados.rados.0.smithi107.stdout:6853: writing smithi10729407-10 from 761856 to 1368064 tid 2 2020-06-12T02:38:31.154 INFO:tasks.rados.rados.0.smithi107.stdout:6853: writing smithi10729407-10 from 1368064 to 1966080 tid 3 2020-06-12T02:38:31.155 INFO:tasks.rados.rados.0.smithi107.stdout:6854: delete oid 16 current snap is 0 2020-06-12T02:38:31.155 INFO:tasks.rados.rados.0.smithi107.stdout:6851: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:31.211 INFO:tasks.rados.rados.0.smithi107.stdout:6854: done (2 left) 2020-06-12T02:38:31.212 INFO:tasks.rados.rados.0.smithi107.stdout:6855: delete oid 9 current snap is 0 2020-06-12T02:38:31.217 INFO:tasks.rados.rados.0.smithi107.stdout:6851: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:31.217 INFO:tasks.rados.rados.0.smithi107.stdout:6853: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:31.218 INFO:tasks.rados.rados.0.smithi107.stdout:6855: done (2 left) 2020-06-12T02:38:31.218 INFO:tasks.rados.rados.0.smithi107.stdout:6856: delete oid 3 current snap is 0 2020-06-12T02:38:31.219 INFO:tasks.rados.rados.0.smithi107.stdout:6851: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:31.219 INFO:tasks.rados.rados.0.smithi107.stdout:6851: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:31.220 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1303 (ObjNum 6850 snap 0 seq_num 6850) dirty exists 2020-06-12T02:38:31.220 INFO:tasks.rados.rados.0.smithi107.stdout:6851: left oid 4 (ObjNum 6850 snap 0 seq_num 6850) 2020-06-12T02:38:31.222 INFO:tasks.rados.rados.0.smithi107.stdout:6851: done (2 left) 2020-06-12T02:38:31.222 INFO:tasks.rados.rados.0.smithi107.stdout:6856: done (1 left) 2020-06-12T02:38:31.223 INFO:tasks.rados.rados.0.smithi107.stdout:6857: delete oid 11 current snap is 0 2020-06-12T02:38:31.223 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:31.219+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:31.225 INFO:tasks.rados.rados.0.smithi107.stdout:6857: done (1 left) 2020-06-12T02:38:31.225 INFO:tasks.rados.rados.0.smithi107.stdout:6858: delete oid 2 current snap is 0 2020-06-12T02:38:31.227 INFO:tasks.rados.rados.0.smithi107.stdout:6853: finishing write tid 2 to smithi10729407-10 2020-06-12T02:38:31.228 INFO:tasks.rados.rados.0.smithi107.stdout:6853: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:31.228 INFO:tasks.rados.rados.0.smithi107.stdout:6858: done (1 left) 2020-06-12T02:38:31.228 INFO:tasks.rados.rados.0.smithi107.stdout:6859: delete oid 7 current snap is 0 2020-06-12T02:38:31.229 INFO:tasks.rados.rados.0.smithi107.stdout:6853: finishing write tid 5 to smithi10729407-10 2020-06-12T02:38:31.230 INFO:tasks.rados.rados.0.smithi107.stdout:6853: finishing write tid 6 to smithi10729407-10 2020-06-12T02:38:31.230 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 449 (ObjNum 6852 snap 0 seq_num 6852) dirty exists 2020-06-12T02:38:31.231 INFO:tasks.rados.rados.0.smithi107.stdout:6853: left oid 10 (ObjNum 6852 snap 0 seq_num 6852) 2020-06-12T02:38:31.232 INFO:tasks.rados.rados.0.smithi107.stdout:6853: done (1 left) 2020-06-12T02:38:31.232 INFO:tasks.rados.rados.0.smithi107.stdout:6859: done (0 left) 2020-06-12T02:38:31.232 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:31.233 INFO:tasks.rados.rados.0.smithi107.stdout:6860: seq_num 6859 ranges {1867776=483328,2351104=647168,2998272=548864,3547136=409600} 2020-06-12T02:38:31.243 INFO:tasks.rados.rados.0.smithi107.stdout:6860: writing smithi10729407-18 from 1867776 to 2351104 tid 1 2020-06-12T02:38:31.247 INFO:tasks.rados.rados.0.smithi107.stdout:6860: writing smithi10729407-18 from 2351104 to 2998272 tid 2 2020-06-12T02:38:31.251 INFO:tasks.rados.rados.0.smithi107.stdout:6860: writing smithi10729407-18 from 2998272 to 3547136 tid 3 2020-06-12T02:38:31.253 INFO:tasks.rados.rados.0.smithi107.stdout:6860: writing smithi10729407-18 from 3547136 to 3956736 tid 4 2020-06-12T02:38:31.254 INFO:tasks.rados.rados.0.smithi107.stdout:6861: delete oid 20 current snap is 0 2020-06-12T02:38:31.254 INFO:tasks.rados.rados.0.smithi107.stdout:6860: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:31.256 INFO:tasks.rados.rados.0.smithi107.stdout:6861: done (1 left) 2020-06-12T02:38:31.257 INFO:tasks.rados.rados.0.smithi107.stdout:6862: delete oid 10 current snap is 0 2020-06-12T02:38:31.262 INFO:tasks.rados.rados.0.smithi107.stdout:6860: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:31.262 INFO:tasks.rados.rados.0.smithi107.stdout:6862: done (1 left) 2020-06-12T02:38:31.262 INFO:tasks.rados.rados.0.smithi107.stdout:6863: delete oid 19 current snap is 0 2020-06-12T02:38:31.263 INFO:tasks.rados.rados.0.smithi107.stdout:6860: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:31.264 INFO:tasks.rados.rados.0.smithi107.stdout:6860: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:31.264 INFO:tasks.rados.rados.0.smithi107.stdout:6860: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:31.265 INFO:tasks.rados.rados.0.smithi107.stdout:6863: done (1 left) 2020-06-12T02:38:31.265 INFO:tasks.rados.rados.0.smithi107.stdout:6864: delete oid 20 current snap is 0 2020-06-12T02:38:31.266 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:31.267 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:38:31.267 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2020-06-12T02:38:31.268 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2020-06-12T02:38:31.268 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:31.268 INFO:tasks.rados.rados.0.smithi107.stdout:6860: finishing write tid 7 to smithi10729407-18 2020-06-12T02:38:31.269 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1276 (ObjNum 6859 snap 0 seq_num 6859) dirty exists 2020-06-12T02:38:31.269 INFO:tasks.rados.rados.0.smithi107.stdout:6860: left oid 18 (ObjNum 6859 snap 0 seq_num 6859) 2020-06-12T02:38:31.270 INFO:tasks.rados.rados.0.smithi107.stdout:6860: done (1 left) 2020-06-12T02:38:31.270 INFO:tasks.rados.rados.0.smithi107.stdout:6864: done (0 left) 2020-06-12T02:38:31.271 INFO:tasks.rados.rados.0.smithi107.stdout:6865: delete oid 9 current snap is 0 2020-06-12T02:38:31.271 INFO:tasks.rados.rados.0.smithi107.stdout:6865: done (0 left) 2020-06-12T02:38:31.271 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:31.272 INFO:tasks.rados.rados.0.smithi107.stdout:6866: seq_num 6865 ranges {0=581632,581632=802816,1384448=540672,1925120=262144} 2020-06-12T02:38:31.273 INFO:tasks.rados.rados.0.smithi107.stdout:6866: writing smithi10729407-16 from 0 to 581632 tid 1 2020-06-12T02:38:31.294 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-06-12T02:38:31.296 INFO:tasks.rados.rados.0.smithi107.stdout:6866: writing smithi10729407-16 from 581632 to 1384448 tid 2 2020-06-12T02:38:31.296 INFO:tasks.rados.rados.0.smithi107.stdout:6866: writing smithi10729407-16 from 1384448 to 1925120 tid 3 2020-06-12T02:38:31.296 INFO:tasks.rados.rados.0.smithi107.stdout:6866: writing smithi10729407-16 from 1925120 to 2187264 tid 4 2020-06-12T02:38:31.297 INFO:tasks.rados.rados.0.smithi107.stdout:6867: delete oid 3 current snap is 0 2020-06-12T02:38:31.297 INFO:tasks.rados.rados.0.smithi107.stdout:6866: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:31.297 INFO:tasks.rados.rados.0.smithi107.stdout:6866: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:31.298 INFO:tasks.rados.rados.0.smithi107.stdout:6866: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:31.298 INFO:tasks.rados.rados.0.smithi107.stdout:6866: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:31.298 INFO:tasks.rados.rados.0.smithi107.stdout:6866: finishing write tid 6 to smithi10729407-16 2020-06-12T02:38:31.299 INFO:tasks.rados.rados.0.smithi107.stdout:6866: finishing write tid 7 to smithi10729407-16 2020-06-12T02:38:31.300 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1220 (ObjNum 6865 snap 0 seq_num 6865) dirty exists 2020-06-12T02:38:31.300 INFO:tasks.rados.rados.0.smithi107.stdout:6866: left oid 16 (ObjNum 6865 snap 0 seq_num 6865) 2020-06-12T02:38:31.300 INFO:tasks.rados.rados.0.smithi107.stdout:6866: done (1 left) 2020-06-12T02:38:31.301 INFO:tasks.rados.rados.0.smithi107.stdout:6867: done (0 left) 2020-06-12T02:38:31.301 INFO:tasks.rados.rados.0.smithi107.stdout:6868: delete oid 16 current snap is 0 2020-06-12T02:38:31.302 INFO:tasks.rados.rados.0.smithi107.stdout:6868: done (0 left) 2020-06-12T02:38:31.302 INFO:tasks.rados.rados.0.smithi107.stdout:6869: delete oid 6 current snap is 0 2020-06-12T02:38:31.303 INFO:tasks.rados.rados.0.smithi107.stdout:6869: done (0 left) 2020-06-12T02:38:31.304 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:38:31.304 INFO:tasks.rados.rados.0.smithi107.stdout:6870: seq_num 6869 ranges {0=647168,647168=745472,1392640=524288,1916928=335872} 2020-06-12T02:38:31.307 INFO:tasks.rados.rados.0.smithi107.stdout:6870: writing smithi10729407-1 from 0 to 647168 tid 1 2020-06-12T02:38:31.312 INFO:tasks.rados.rados.0.smithi107.stdout:6870: writing smithi10729407-1 from 647168 to 1392640 tid 2 2020-06-12T02:38:31.315 INFO:tasks.rados.rados.0.smithi107.stdout:6870: writing smithi10729407-1 from 1392640 to 1916928 tid 3 2020-06-12T02:38:31.317 INFO:tasks.rados.rados.0.smithi107.stdout:6870: writing smithi10729407-1 from 1916928 to 2252800 tid 4 2020-06-12T02:38:31.317 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:31.317 INFO:tasks.rados.rados.0.smithi107.stdout:6871: seq_num 6870 ranges {1851392=663552,2514944=630784,3145728=729088,3874816=245760} 2020-06-12T02:38:31.323 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:31.320+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:31.331 INFO:tasks.rados.rados.0.smithi107.stdout:6871: writing smithi10729407-12 from 1851392 to 2514944 tid 1 2020-06-12T02:38:31.335 INFO:tasks.rados.rados.0.smithi107.stdout:6871: writing smithi10729407-12 from 2514944 to 3145728 tid 2 2020-06-12T02:38:31.340 INFO:tasks.rados.rados.0.smithi107.stdout:6871: writing smithi10729407-12 from 3145728 to 3874816 tid 3 2020-06-12T02:38:31.342 INFO:tasks.rados.rados.0.smithi107.stdout:6871: writing smithi10729407-12 from 3874816 to 4120576 tid 4 2020-06-12T02:38:31.342 INFO:tasks.rados.rados.0.smithi107.stdout:6872: delete oid 20 current snap is 0 2020-06-12T02:38:31.343 INFO:tasks.rados.rados.0.smithi107.stdout:6870: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:31.343 INFO:tasks.rados.rados.0.smithi107.stdout:6870: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:31.343 INFO:tasks.rados.rados.0.smithi107.stdout:6870: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:31.343 INFO:tasks.rados.rados.0.smithi107.stdout:6870: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:31.343 INFO:tasks.rados.rados.0.smithi107.stdout:6870: finishing write tid 6 to smithi10729407-1 2020-06-12T02:38:31.344 INFO:tasks.rados.rados.0.smithi107.stdout:6870: finishing write tid 7 to smithi10729407-1 2020-06-12T02:38:31.344 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1242 (ObjNum 6869 snap 0 seq_num 6869) dirty exists 2020-06-12T02:38:31.344 INFO:tasks.rados.rados.0.smithi107.stdout:6870: left oid 1 (ObjNum 6869 snap 0 seq_num 6869) 2020-06-12T02:38:31.344 INFO:tasks.rados.rados.0.smithi107.stdout:6871: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:31.345 INFO:tasks.rados.rados.0.smithi107.stdout:6870: done (2 left) 2020-06-12T02:38:31.345 INFO:tasks.rados.rados.0.smithi107.stdout:6872: done (1 left) 2020-06-12T02:38:31.345 INFO:tasks.rados.rados.0.smithi107.stdout:6873: delete oid 16 current snap is 0 2020-06-12T02:38:31.347 INFO:tasks.rados.rados.0.smithi107.stdout:6873: done (1 left) 2020-06-12T02:38:31.347 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:31.347 INFO:tasks.rados.rados.0.smithi107.stdout:6874: seq_num 6873 ranges {0=712704,712704=729088,1441792=507904,1949696=368640} 2020-06-12T02:38:31.351 INFO:tasks.rados.rados.0.smithi107.stdout:6874: writing smithi10729407-14 from 0 to 712704 tid 1 2020-06-12T02:38:31.356 INFO:tasks.rados.rados.0.smithi107.stdout:6874: writing smithi10729407-14 from 712704 to 1441792 tid 2 2020-06-12T02:38:31.359 INFO:tasks.rados.rados.0.smithi107.stdout:6874: writing smithi10729407-14 from 1441792 to 1949696 tid 3 2020-06-12T02:38:31.361 INFO:tasks.rados.rados.0.smithi107.stdout:6874: writing smithi10729407-14 from 1949696 to 2318336 tid 4 2020-06-12T02:38:31.361 INFO:tasks.rados.rados.0.smithi107.stdout:6875: delete oid 17 current snap is 0 2020-06-12T02:38:31.362 INFO:tasks.rados.rados.0.smithi107.stdout:6871: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:31.362 INFO:tasks.rados.rados.0.smithi107.stdout:6871: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:31.362 INFO:tasks.rados.rados.0.smithi107.stdout:6871: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:31.362 INFO:tasks.rados.rados.0.smithi107.stdout:6871: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:31.362 INFO:tasks.rados.rados.0.smithi107.stdout:6871: finishing write tid 7 to smithi10729407-12 2020-06-12T02:38:31.363 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1262 (ObjNum 6870 snap 0 seq_num 6870) dirty exists 2020-06-12T02:38:31.363 INFO:tasks.rados.rados.0.smithi107.stdout:6871: left oid 12 (ObjNum 6870 snap 0 seq_num 6870) 2020-06-12T02:38:31.365 INFO:tasks.rados.rados.0.smithi107.stdout:6871: done (2 left) 2020-06-12T02:38:31.365 INFO:tasks.rados.rados.0.smithi107.stdout:6875: done (1 left) 2020-06-12T02:38:31.365 INFO:tasks.rados.rados.0.smithi107.stdout:6876: delete oid 4 current snap is 0 2020-06-12T02:38:31.368 INFO:tasks.rados.rados.0.smithi107.stdout:6874: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:31.369 INFO:tasks.rados.rados.0.smithi107.stdout:6876: done (1 left) 2020-06-12T02:38:31.370 INFO:tasks.rados.rados.0.smithi107.stdout:6877: delete oid 1 current snap is 0 2020-06-12T02:38:31.370 INFO:tasks.rados.rados.0.smithi107.stdout:6874: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:31.371 INFO:tasks.rados.rados.0.smithi107.stdout:6874: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:31.371 INFO:tasks.rados.rados.0.smithi107.stdout:6874: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:31.373 INFO:tasks.rados.rados.0.smithi107.stdout:6874: finishing write tid 6 to smithi10729407-14 2020-06-12T02:38:31.374 INFO:tasks.rados.rados.0.smithi107.stdout:6877: done (1 left) 2020-06-12T02:38:31.374 INFO:tasks.rados.rados.0.smithi107.stdout:6878: delete oid 6 current snap is 0 2020-06-12T02:38:31.375 INFO:tasks.rados.rados.0.smithi107.stdout:6874: finishing write tid 7 to smithi10729407-14 2020-06-12T02:38:31.376 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1205 (ObjNum 6873 snap 0 seq_num 6873) dirty exists 2020-06-12T02:38:31.376 INFO:tasks.rados.rados.0.smithi107.stdout:6874: left oid 14 (ObjNum 6873 snap 0 seq_num 6873) 2020-06-12T02:38:31.380 INFO:tasks.rados.rados.0.smithi107.stdout:6874: done (1 left) 2020-06-12T02:38:31.381 INFO:tasks.rados.rados.0.smithi107.stdout:6878: done (0 left) 2020-06-12T02:38:31.381 INFO:tasks.rados.rados.0.smithi107.stdout:6879: delete oid 11 current snap is 0 2020-06-12T02:38:31.383 INFO:tasks.rados.rados.0.smithi107.stdout:6879: done (0 left) 2020-06-12T02:38:31.383 INFO:tasks.rados.rados.0.smithi107.stdout:6880: delete oid 11 current snap is 0 2020-06-12T02:38:31.386 INFO:tasks.rados.rados.0.smithi107.stdout:6880: done (0 left) 2020-06-12T02:38:31.386 INFO:tasks.rados.rados.0.smithi107.stdout:6881: delete oid 16 current snap is 0 2020-06-12T02:38:31.389 INFO:tasks.rados.rados.0.smithi107.stdout:6881: done (0 left) 2020-06-12T02:38:31.389 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:31.390 INFO:tasks.rados.rados.0.smithi107.stdout:6882: seq_num 6881 ranges {0=49152} 2020-06-12T02:38:31.390 INFO:tasks.rados.rados.0.smithi107.stdout:6882: writing smithi10729407-19 from 0 to 49152 tid 1 2020-06-12T02:38:31.390 INFO:tasks.rados.rados.0.smithi107.stdout:6883: delete oid 16 current snap is 0 2020-06-12T02:38:31.391 INFO:tasks.rados.rados.0.smithi107.stdout:6883: done (1 left) 2020-06-12T02:38:31.391 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:38:31.391 INFO:tasks.rados.rados.0.smithi107.stdout:6884: seq_num 6883 ranges {0=81920} 2020-06-12T02:38:31.392 INFO:tasks.rados.rados.0.smithi107.stdout:6884: writing smithi10729407-11 from 0 to 81920 tid 1 2020-06-12T02:38:31.392 INFO:tasks.rados.rados.0.smithi107.stdout:6885: delete oid 6 current snap is 0 2020-06-12T02:38:31.393 INFO:tasks.rados.rados.0.smithi107.stdout:6882: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:31.393 INFO:tasks.rados.rados.0.smithi107.stdout:6882: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:31.395 INFO:tasks.rados.rados.0.smithi107.stdout:6882: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:31.395 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1264 (ObjNum 6881 snap 0 seq_num 6881) dirty exists 2020-06-12T02:38:31.396 INFO:tasks.rados.rados.0.smithi107.stdout:6882: left oid 19 (ObjNum 6881 snap 0 seq_num 6881) 2020-06-12T02:38:31.396 INFO:tasks.rados.rados.0.smithi107.stdout:6882: done (2 left) 2020-06-12T02:38:31.396 INFO:tasks.rados.rados.0.smithi107.stdout:6885: done (1 left) 2020-06-12T02:38:31.396 INFO:tasks.rados.rados.0.smithi107.stdout:6886: delete oid 12 current snap is 0 2020-06-12T02:38:31.415 INFO:tasks.rados.rados.0.smithi107.stdout:6884: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:31.415 INFO:tasks.rados.rados.0.smithi107.stdout:6886: done (1 left) 2020-06-12T02:38:31.415 INFO:tasks.rados.rados.0.smithi107.stdout:6887: delete oid 20 current snap is 0 2020-06-12T02:38:31.415 INFO:tasks.rados.rados.0.smithi107.stdout:6887: done (1 left) 2020-06-12T02:38:31.416 INFO:tasks.rados.rados.0.smithi107.stdout:6888: delete oid 16 current snap is 0 2020-06-12T02:38:31.416 INFO:tasks.rados.rados.0.smithi107.stdout:6884: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:31.416 INFO:tasks.rados.rados.0.smithi107.stdout:6888: done (1 left) 2020-06-12T02:38:31.416 INFO:tasks.rados.rados.0.smithi107.stdout:6889: delete oid 16 current snap is 0 2020-06-12T02:38:31.416 INFO:tasks.rados.rados.0.smithi107.stdout:6884: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:31.417 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1209 (ObjNum 6883 snap 0 seq_num 6883) dirty exists 2020-06-12T02:38:31.417 INFO:tasks.rados.rados.0.smithi107.stdout:6884: left oid 11 (ObjNum 6883 snap 0 seq_num 6883) 2020-06-12T02:38:31.417 INFO:tasks.rados.rados.0.smithi107.stdout:6884: done (1 left) 2020-06-12T02:38:31.417 INFO:tasks.rados.rados.0.smithi107.stdout:6889: done (0 left) 2020-06-12T02:38:31.417 INFO:tasks.rados.rados.0.smithi107.stdout:6890: delete oid 17 current snap is 0 2020-06-12T02:38:31.418 INFO:tasks.rados.rados.0.smithi107.stdout:6890: done (0 left) 2020-06-12T02:38:31.418 INFO:tasks.rados.rados.0.smithi107.stdout:6891: delete oid 13 current snap is 0 2020-06-12T02:38:31.418 INFO:tasks.rados.rados.0.smithi107.stdout:6891: done (0 left) 2020-06-12T02:38:31.418 INFO:tasks.rados.rados.0.smithi107.stdout:6892: delete oid 13 current snap is 0 2020-06-12T02:38:31.419 INFO:tasks.rados.rados.0.smithi107.stdout:6892: done (0 left) 2020-06-12T02:38:31.419 INFO:tasks.rados.rados.0.smithi107.stdout:6893: delete oid 19 current snap is 0 2020-06-12T02:38:31.427 INFO:tasks.rados.rados.0.smithi107.stdout:6893: done (0 left) 2020-06-12T02:38:31.427 INFO:tasks.rados.rados.0.smithi107.stdout:6894: delete oid 16 current snap is 0 2020-06-12T02:38:31.427 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:31.424+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:31.428 INFO:tasks.rados.rados.0.smithi107.stdout:6894: done (0 left) 2020-06-12T02:38:31.428 INFO:tasks.rados.rados.0.smithi107.stdout:6895: delete oid 4 current snap is 0 2020-06-12T02:38:31.430 INFO:tasks.rados.rados.0.smithi107.stdout:6895: done (0 left) 2020-06-12T02:38:31.430 INFO:tasks.rados.rados.0.smithi107.stdout:6896: delete oid 19 current snap is 0 2020-06-12T02:38:31.432 INFO:tasks.rados.rados.0.smithi107.stdout:6896: done (0 left) 2020-06-12T02:38:31.432 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:31.433 INFO:tasks.rados.rados.0.smithi107.stdout:6897: seq_num 6896 ranges {81920=303104} 2020-06-12T02:38:31.436 INFO:tasks.rados.rados.0.smithi107.stdout:6897: writing smithi10729407-11 from 81920 to 385024 tid 1 2020-06-12T02:38:31.436 INFO:tasks.rados.rados.0.smithi107.stdout:6898: delete oid 19 current snap is 0 2020-06-12T02:38:31.437 INFO:tasks.rados.rados.0.smithi107.stdout:6898: done (1 left) 2020-06-12T02:38:31.437 INFO:tasks.rados.rados.0.smithi107.stdout:6899: delete oid 17 current snap is 0 2020-06-12T02:38:31.439 INFO:tasks.rados.rados.0.smithi107.stdout:6897: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:31.442 INFO:tasks.rados.rados.0.smithi107.stdout:6897: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:31.442 INFO:tasks.rados.rados.0.smithi107.stdout:6899: done (1 left) 2020-06-12T02:38:31.442 INFO:tasks.rados.rados.0.smithi107.stdout:6900: delete oid 10 current snap is 0 2020-06-12T02:38:31.443 INFO:tasks.rados.rados.0.smithi107.stdout:6897: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:31.444 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1211 (ObjNum 6896 snap 0 seq_num 6896) dirty exists 2020-06-12T02:38:31.444 INFO:tasks.rados.rados.0.smithi107.stdout:6897: left oid 11 (ObjNum 6896 snap 0 seq_num 6896) 2020-06-12T02:38:31.446 INFO:tasks.rados.rados.0.smithi107.stdout:6897: done (1 left) 2020-06-12T02:38:31.447 INFO:tasks.rados.rados.0.smithi107.stdout:6900: done (0 left) 2020-06-12T02:38:31.447 INFO:tasks.rados.rados.0.smithi107.stdout:6901: delete oid 8 current snap is 0 2020-06-12T02:38:31.449 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:31.450 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2020-06-12T02:38:31.450 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2020-06-12T02:38:31.451 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2020-06-12T02:38:31.451 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.451 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:2782 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~696320] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:31.452 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:36.808170+0000", 2020-06-12T02:38:31.452 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 54.637114597999997, 2020-06-12T02:38:31.452 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016972839999999999, 2020-06-12T02:38:31.453 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:31.453 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:31.453 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:31.454 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:31.455 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:31.455 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2782 2020-06-12T02:38:31.455 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.456 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:31.456 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.456 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:31.457 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.808170+0000", 2020-06-12T02:38:31.457 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:31.457 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.458 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.458 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:31.459 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.808170+0000", 2020-06-12T02:38:31.459 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000261778 2020-06-12T02:38:31.460 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.460 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.460 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:31.461 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.808432+0000", 2020-06-12T02:38:31.461 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0920000000000003e-06 2020-06-12T02:38:31.462 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.463 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.463 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:31.464 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.808438+0000", 2020-06-12T02:38:31.465 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2539e-05 2020-06-12T02:38:31.465 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.465 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.466 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:31.466 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.808450+0000", 2020-06-12T02:38:31.466 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1859999999999997e-06 2020-06-12T02:38:31.467 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.467 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.467 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:31.468 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.808457+0000", 2020-06-12T02:38:31.468 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0988000000000001e-05 2020-06-12T02:38:31.468 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.469 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.469 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:31.469 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.808498+0000", 2020-06-12T02:38:31.470 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011723 2020-06-12T02:38:31.470 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.470 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.471 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:31.471 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.808615+0000", 2020-06-12T02:38:31.471 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0015648109999999999 2020-06-12T02:38:31.472 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.472 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.473 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:31.473 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.810180+0000", 2020-06-12T02:38:31.474 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1718999999999994e-05 2020-06-12T02:38:31.474 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.474 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.475 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:31.475 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.810271+0000", 2020-06-12T02:38:31.475 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00082076299999999996 2020-06-12T02:38:31.476 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.476 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.476 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:31.477 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.811092+0000", 2020-06-12T02:38:31.477 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01210689 2020-06-12T02:38:31.477 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.478 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.478 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:31.478 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.823199+0000", 2020-06-12T02:38:31.479 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017554300000000001 2020-06-12T02:38:31.479 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.479 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.480 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:31.480 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.824954+0000", 2020-06-12T02:38:31.480 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000188414 2020-06-12T02:38:31.480 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.481 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.481 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:31.481 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.825143+0000", 2020-06-12T02:38:31.482 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016972839999999999 2020-06-12T02:38:31.482 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:31.482 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:31.483 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:31.483 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.483 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.484 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:2785 3.0s0 3:0dddb690:::smithi10729407-10:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:31.484 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:36.813804+0000", 2020-06-12T02:38:31.484 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 54.631480328000002, 2020-06-12T02:38:31.485 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014100781999999999, 2020-06-12T02:38:31.485 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:31.485 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:31.486 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:31.486 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:31.486 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:31.487 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2785 2020-06-12T02:38:31.487 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.487 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:31.487 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.488 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:31.488 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813804+0000", 2020-06-12T02:38:31.488 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:31.489 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.489 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.489 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:31.490 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813804+0000", 2020-06-12T02:38:31.490 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1301999999999999e-05 2020-06-12T02:38:31.490 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.491 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.491 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:31.491 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813846+0000", 2020-06-12T02:38:31.492 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4530000000000004e-06 2020-06-12T02:38:31.492 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.492 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.493 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:31.493 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813850+0000", 2020-06-12T02:38:31.493 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7940000000000003e-06 2020-06-12T02:38:31.494 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.494 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.494 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:31.494 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813858+0000", 2020-06-12T02:38:31.495 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9450000000000002e-06 2020-06-12T02:38:31.495 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.495 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.496 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:31.496 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813861+0000", 2020-06-12T02:38:31.496 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0084749969999999997 2020-06-12T02:38:31.497 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.497 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.497 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:31.498 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.822336+0000", 2020-06-12T02:38:31.498 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013055400000000001 2020-06-12T02:38:31.498 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.499 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.499 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:31.499 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.822466+0000", 2020-06-12T02:38:31.500 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047301400000000001 2020-06-12T02:38:31.500 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.500 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.500 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:31.501 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.822939+0000", 2020-06-12T02:38:31.501 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086569499999999998 2020-06-12T02:38:31.501 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.502 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.502 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:31.502 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.823805+0000", 2020-06-12T02:38:31.503 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0039911460000000001 2020-06-12T02:38:31.503 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.503 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.504 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:31.504 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.827796+0000", 2020-06-12T02:38:31.504 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010888200000000001 2020-06-12T02:38:31.505 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.505 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.505 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:31.506 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.827905+0000", 2020-06-12T02:38:31.506 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014100781999999999 2020-06-12T02:38:31.506 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:31.506 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:31.507 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:31.507 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.507 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.508 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:2786 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:38:31.508 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:36.813885+0000", 2020-06-12T02:38:31.508 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 54.631399510999998, 2020-06-12T02:38:31.509 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015757724000000001, 2020-06-12T02:38:31.509 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:31.509 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:31.510 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:31.510 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:31.510 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:31.511 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2786 2020-06-12T02:38:31.511 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.511 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:31.511 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.512 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:31.512 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813885+0000", 2020-06-12T02:38:31.512 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:31.513 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.513 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.513 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:31.514 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813885+0000", 2020-06-12T02:38:31.514 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2388999999999999e-05 2020-06-12T02:38:31.514 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.515 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.515 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:31.515 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813907+0000", 2020-06-12T02:38:31.516 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9660000000000001e-06 2020-06-12T02:38:31.516 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.516 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.516 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:31.517 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813910+0000", 2020-06-12T02:38:31.517 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7649999999999993e-06 2020-06-12T02:38:31.517 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.518 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.518 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:31.518 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813918+0000", 2020-06-12T02:38:31.519 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0870000000000001e-06 2020-06-12T02:38:31.519 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.519 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.520 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:31.520 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813920+0000", 2020-06-12T02:38:31.520 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093684819999999992 2020-06-12T02:38:31.521 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.521 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.521 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:31.522 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.823289+0000", 2020-06-12T02:38:31.522 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103643 2020-06-12T02:38:31.522 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.522 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.523 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:31.523 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.823392+0000", 2020-06-12T02:38:31.523 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0050514449999999999 2020-06-12T02:38:31.523 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.523 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.524 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:31.524 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.828444+0000", 2020-06-12T02:38:31.524 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000102824 2020-06-12T02:38:31.524 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.524 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.524 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:31.525 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.828547+0000", 2020-06-12T02:38:31.525 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0010961230000000001 2020-06-12T02:38:31.525 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.525 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.525 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:31.526 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.829643+0000", 2020-06-12T02:38:31.526 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015757724000000001 2020-06-12T02:38:31.526 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:31.526 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:31.526 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:31.527 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.527 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.527 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:3330 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~630784] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:31.527 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:38.603794+0000", 2020-06-12T02:38:31.527 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 52.841490819999997, 2020-06-12T02:38:31.528 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016093917999999999, 2020-06-12T02:38:31.528 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:31.528 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:31.528 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:31.528 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:31.529 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:31.529 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 3330 2020-06-12T02:38:31.529 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.529 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:31.529 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.529 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:31.530 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.603794+0000", 2020-06-12T02:38:31.530 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:31.530 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.530 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.530 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:31.531 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.603794+0000", 2020-06-12T02:38:31.531 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00042303700000000002 2020-06-12T02:38:31.531 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.531 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.531 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:31.531 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.604217+0000", 2020-06-12T02:38:31.532 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9350000000000006e-06 2020-06-12T02:38:31.532 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.532 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.532 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:31.532 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.604227+0000", 2020-06-12T02:38:31.533 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9343000000000001e-05 2020-06-12T02:38:31.533 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.533 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.533 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:31.533 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.604246+0000", 2020-06-12T02:38:31.533 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2369999999999998e-06 2020-06-12T02:38:31.534 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.534 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.534 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:31.534 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.604255+0000", 2020-06-12T02:38:31.534 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1530000000000003e-05 2020-06-12T02:38:31.535 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.535 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.535 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:31.535 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.604307+0000", 2020-06-12T02:38:31.535 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103756 2020-06-12T02:38:31.535 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.536 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.536 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:31.536 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.604411+0000", 2020-06-12T02:38:31.536 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073281400000000003 2020-06-12T02:38:31.536 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.537 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.537 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:31.537 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.605143+0000", 2020-06-12T02:38:31.537 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071483730000000004 2020-06-12T02:38:31.537 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.538 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.538 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:31.538 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.612292+0000", 2020-06-12T02:38:31.538 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0075185879999999997 2020-06-12T02:38:31.539 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.539 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.539 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:31.539 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.619810+0000", 2020-06-12T02:38:31.539 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7305000000000006e-05 2020-06-12T02:38:31.539 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.540 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.540 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:31.540 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.619888+0000", 2020-06-12T02:38:31.540 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016093917999999999 2020-06-12T02:38:31.541 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:31.542 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:31.542 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:31.542 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.543 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.543 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:3335 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:38:31.543 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:38.616040+0000", 2020-06-12T02:38:31.543 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 52.829244080999999, 2020-06-12T02:38:31.543 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015075565000000001, 2020-06-12T02:38:31.544 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:31.544 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:31.544 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:31.544 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:31.544 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:31.545 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 3335 2020-06-12T02:38:31.545 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.545 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:31.545 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.545 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:31.546 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.616040+0000", 2020-06-12T02:38:31.546 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:31.546 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.547 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.547 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:31.547 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.616040+0000", 2020-06-12T02:38:31.548 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1950000000000002e-05 2020-06-12T02:38:31.548 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.548 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.548 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:31.548 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.616062+0000", 2020-06-12T02:38:31.549 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.1370000000000002e-06 2020-06-12T02:38:31.549 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.549 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.549 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:31.549 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.616066+0000", 2020-06-12T02:38:31.549 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5019999999999998e-06 2020-06-12T02:38:31.550 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.550 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.550 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:31.550 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.616072+0000", 2020-06-12T02:38:31.550 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3470000000000001e-06 2020-06-12T02:38:31.551 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.551 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.551 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:31.551 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.616074+0000", 2020-06-12T02:38:31.551 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0067702309999999998 2020-06-12T02:38:31.552 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.552 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.552 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:31.552 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.622845+0000", 2020-06-12T02:38:31.552 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7738000000000005e-05 2020-06-12T02:38:31.552 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.553 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.553 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:31.553 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.622942+0000", 2020-06-12T02:38:31.553 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0050881010000000003 2020-06-12T02:38:31.553 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.554 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.554 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:31.554 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.628031+0000", 2020-06-12T02:38:31.554 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000108171 2020-06-12T02:38:31.554 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.554 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.555 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:31.555 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.628139+0000", 2020-06-12T02:38:31.555 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0029773880000000001 2020-06-12T02:38:31.555 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.555 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.556 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:31.556 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.631116+0000", 2020-06-12T02:38:31.556 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015075565000000001 2020-06-12T02:38:31.557 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:31.557 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:31.557 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:31.557 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.557 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.558 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:4969 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~729088,stat] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-06-12T02:38:31.558 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:43.713430+0000", 2020-06-12T02:38:31.558 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 47.731854617000003, 2020-06-12T02:38:31.558 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.18980771900000001, 2020-06-12T02:38:31.558 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:31.559 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:31.559 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:31.559 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:31.559 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:31.559 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 4969 2020-06-12T02:38:31.559 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.560 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:31.560 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.560 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:31.560 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.713430+0000", 2020-06-12T02:38:31.560 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:31.561 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.561 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.561 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:31.561 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.713430+0000", 2020-06-12T02:38:31.561 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018522800000000001 2020-06-12T02:38:31.562 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.562 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.562 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:31.562 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.713615+0000", 2020-06-12T02:38:31.562 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3590000000000002e-06 2020-06-12T02:38:31.562 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.563 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.563 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:31.563 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.713623+0000", 2020-06-12T02:38:31.563 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0939e-05 2020-06-12T02:38:31.563 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.564 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.564 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:31.564 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.713633+0000", 2020-06-12T02:38:31.564 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3229999999999999e-06 2020-06-12T02:38:31.564 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.565 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.565 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:31.565 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.713638+0000", 2020-06-12T02:38:31.565 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7401000000000002e-05 2020-06-12T02:38:31.566 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.566 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.566 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:31.566 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.713685+0000", 2020-06-12T02:38:31.566 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011488 2020-06-12T02:38:31.567 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.567 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.567 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:31.567 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.713800+0000", 2020-06-12T02:38:31.567 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000662541 2020-06-12T02:38:31.568 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.568 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.568 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:31.568 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.714463+0000", 2020-06-12T02:38:31.568 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043485090000000004 2020-06-12T02:38:31.569 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.569 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.569 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:31.569 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.718811+0000", 2020-06-12T02:38:31.569 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.18428694000000001 2020-06-12T02:38:31.570 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.570 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.570 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:31.570 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.903098+0000", 2020-06-12T02:38:31.570 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013959899999999999 2020-06-12T02:38:31.570 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.571 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.571 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:31.571 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.903238+0000", 2020-06-12T02:38:31.571 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.18980771900000001 2020-06-12T02:38:31.571 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:31.572 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:31.572 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:31.572 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.572 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.572 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:4970 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~401408] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-06-12T02:38:31.573 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:43.716006+0000", 2020-06-12T02:38:31.573 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 47.729278504, 2020-06-12T02:38:31.573 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.19216903699999999, 2020-06-12T02:38:31.573 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:31.573 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:31.574 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:31.574 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:31.574 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:31.574 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 4970 2020-06-12T02:38:31.575 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.575 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:31.575 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.575 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:31.575 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716006+0000", 2020-06-12T02:38:31.575 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:31.576 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.576 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.576 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:31.576 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716006+0000", 2020-06-12T02:38:31.576 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000115757 2020-06-12T02:38:31.577 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.577 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.577 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:31.577 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716122+0000", 2020-06-12T02:38:31.578 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1989999999999997e-06 2020-06-12T02:38:31.578 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.578 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.578 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:31.578 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716127+0000", 2020-06-12T02:38:31.579 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2121e-05 2020-06-12T02:38:31.579 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.579 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.579 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:31.579 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716139+0000", 2020-06-12T02:38:31.579 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.113e-06 2020-06-12T02:38:31.580 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.582 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.582 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:31.582 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716144+0000", 2020-06-12T02:38:31.583 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014413900000000001 2020-06-12T02:38:31.583 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.583 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.583 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:31.584 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.717586+0000", 2020-06-12T02:38:31.584 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0257000000000002e-05 2020-06-12T02:38:31.584 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.584 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.584 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:31.585 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.717656+0000", 2020-06-12T02:38:31.585 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.18596705899999999 2020-06-12T02:38:31.585 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.585 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.585 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:31.586 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.903623+0000", 2020-06-12T02:38:31.586 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7858999999999999e-05 2020-06-12T02:38:31.586 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.586 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.586 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:31.587 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.903721+0000", 2020-06-12T02:38:31.587 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000683536 2020-06-12T02:38:31.587 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.587 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.587 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:31.588 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.904404+0000", 2020-06-12T02:38:31.588 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.003364438 2020-06-12T02:38:31.588 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.588 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.588 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:31.589 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.907769+0000", 2020-06-12T02:38:31.589 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00033094500000000002 2020-06-12T02:38:31.589 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.589 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.589 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:31.590 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.908100+0000", 2020-06-12T02:38:31.590 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5363000000000003e-05 2020-06-12T02:38:31.590 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.590 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.590 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:31.591 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.908175+0000", 2020-06-12T02:38:31.591 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.19216903699999999 2020-06-12T02:38:31.591 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:31.591 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:31.591 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:31.591 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.592 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.592 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:4971 3.0s0 3:0dddb690:::smithi10729407-10:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-06-12T02:38:31.592 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:43.716190+0000", 2020-06-12T02:38:31.592 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 47.729094646, 2020-06-12T02:38:31.592 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.19259003999999999, 2020-06-12T02:38:31.593 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:31.593 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:31.593 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:31.594 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:31.594 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:31.595 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 4971 2020-06-12T02:38:31.595 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.595 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:31.595 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.595 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:31.596 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716190+0000", 2020-06-12T02:38:31.596 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:31.596 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.596 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.596 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:31.597 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716190+0000", 2020-06-12T02:38:31.597 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3192999999999998e-05 2020-06-12T02:38:31.597 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.597 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.597 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:31.598 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716233+0000", 2020-06-12T02:38:31.598 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6279999999999998e-06 2020-06-12T02:38:31.598 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.598 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.598 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:31.599 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716237+0000", 2020-06-12T02:38:31.599 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6619999999999999e-06 2020-06-12T02:38:31.599 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.599 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.599 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:31.600 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716243+0000", 2020-06-12T02:38:31.600 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.495e-06 2020-06-12T02:38:31.600 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.600 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.600 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:31.601 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716247+0000", 2020-06-12T02:38:31.601 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014483740000000001 2020-06-12T02:38:31.601 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.601 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.601 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:31.601 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.717695+0000", 2020-06-12T02:38:31.602 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000108108 2020-06-12T02:38:31.602 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.602 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.602 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:31.602 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.717803+0000", 2020-06-12T02:38:31.603 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.18873436800000001 2020-06-12T02:38:31.603 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.603 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.603 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:31.604 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.906538+0000", 2020-06-12T02:38:31.604 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9384999999999998e-05 2020-06-12T02:38:31.604 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.604 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.606 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:31.606 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.906637+0000", 2020-06-12T02:38:31.606 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00042339299999999999 2020-06-12T02:38:31.606 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.606 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.607 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:31.607 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.907061+0000", 2020-06-12T02:38:31.607 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00071097900000000004 2020-06-12T02:38:31.607 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.607 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.608 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:31.608 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.907772+0000", 2020-06-12T02:38:31.608 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00089641299999999998 2020-06-12T02:38:31.608 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.608 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.609 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:31.609 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.908668+0000", 2020-06-12T02:38:31.609 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112042 2020-06-12T02:38:31.609 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.609 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.610 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:31.610 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.908780+0000", 2020-06-12T02:38:31.610 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.19259003999999999 2020-06-12T02:38:31.611 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:31.612 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:31.612 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:31.612 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.613 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.613 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:4972 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-06-12T02:38:31.613 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:43.716274+0000", 2020-06-12T02:38:31.614 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 47.729010877, 2020-06-12T02:38:31.614 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.193959876, 2020-06-12T02:38:31.614 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:31.614 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:31.614 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:31.615 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:31.615 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:31.615 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 4972 2020-06-12T02:38:31.615 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.615 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:31.616 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.616 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:31.616 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716274+0000", 2020-06-12T02:38:31.616 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:31.616 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.617 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.617 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:31.617 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716274+0000", 2020-06-12T02:38:31.617 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6852999999999999e-05 2020-06-12T02:38:31.617 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.617 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.618 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:31.618 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716301+0000", 2020-06-12T02:38:31.618 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3069999999999998e-06 2020-06-12T02:38:31.618 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.618 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.619 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:31.619 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716304+0000", 2020-06-12T02:38:31.619 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6080000000000001e-06 2020-06-12T02:38:31.619 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.619 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.620 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:31.620 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716310+0000", 2020-06-12T02:38:31.620 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6110000000000001e-06 2020-06-12T02:38:31.620 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.620 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.620 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:31.621 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716313+0000", 2020-06-12T02:38:31.621 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001512918 2020-06-12T02:38:31.621 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.621 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.621 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:31.622 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.717826+0000", 2020-06-12T02:38:31.622 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3773000000000002e-05 2020-06-12T02:38:31.622 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.622 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.622 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:31.622 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.717880+0000", 2020-06-12T02:38:31.623 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.18955132599999999 2020-06-12T02:38:31.623 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.623 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.623 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:31.623 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.907431+0000", 2020-06-12T02:38:31.625 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101158 2020-06-12T02:38:31.625 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.626 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.626 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:31.626 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.907532+0000", 2020-06-12T02:38:31.626 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00160593 2020-06-12T02:38:31.626 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.627 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.627 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:31.627 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.909138+0000", 2020-06-12T02:38:31.627 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8590000000000006e-05 2020-06-12T02:38:31.627 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.627 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.628 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:31.628 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.909207+0000", 2020-06-12T02:38:31.628 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0010268020000000001 2020-06-12T02:38:31.628 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.628 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.629 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:31.629 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.910234+0000", 2020-06-12T02:38:31.629 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.193959876 2020-06-12T02:38:31.629 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:31.629 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:31.629 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:31.630 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.630 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.630 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:5006 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-06-12T02:38:31.630 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:43.975526+0000", 2020-06-12T02:38:31.630 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 47.469758827, 2020-06-12T02:38:31.631 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014759296, 2020-06-12T02:38:31.631 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:31.631 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:31.631 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:31.631 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:31.631 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:31.632 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 5006 2020-06-12T02:38:31.632 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.632 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:31.632 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.632 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:31.633 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.975526+0000", 2020-06-12T02:38:31.633 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:31.633 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.633 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.633 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:31.634 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.975526+0000", 2020-06-12T02:38:31.634 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5868999999999997e-05 2020-06-12T02:38:31.634 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.634 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.634 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:31.635 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.975572+0000", 2020-06-12T02:38:31.635 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1210000000000003e-06 2020-06-12T02:38:31.635 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.635 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.635 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:31.635 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.975578+0000", 2020-06-12T02:38:31.636 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2204999999999999e-05 2020-06-12T02:38:31.636 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.636 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.636 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:31.636 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.975590+0000", 2020-06-12T02:38:31.637 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1289999999999998e-06 2020-06-12T02:38:31.637 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.637 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.637 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:31.637 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.975594+0000", 2020-06-12T02:38:31.637 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.7705e-05 2020-06-12T02:38:31.638 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.638 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.638 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:31.638 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.975622+0000", 2020-06-12T02:38:31.638 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9941999999999997e-05 2020-06-12T02:38:31.639 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.639 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.639 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:31.639 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.975672+0000", 2020-06-12T02:38:31.639 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0057881060000000003 2020-06-12T02:38:31.639 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.640 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.640 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:31.640 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.981460+0000", 2020-06-12T02:38:31.640 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0473999999999997e-05 2020-06-12T02:38:31.640 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.641 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.641 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:31.641 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.981520+0000", 2020-06-12T02:38:31.641 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.007714294 2020-06-12T02:38:31.641 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.642 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.642 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:31.642 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.989235+0000", 2020-06-12T02:38:31.642 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010107600000000001 2020-06-12T02:38:31.642 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.642 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.643 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:31.643 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.989336+0000", 2020-06-12T02:38:31.643 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00094937500000000002 2020-06-12T02:38:31.643 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.643 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.644 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:31.644 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.990285+0000", 2020-06-12T02:38:31.644 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014759296 2020-06-12T02:38:31.644 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:31.644 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:31.644 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:31.645 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.645 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.645 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:7841 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~565248] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-06-12T02:38:31.645 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:51.996595+0000", 2020-06-12T02:38:31.645 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 39.448689338000001, 2020-06-12T02:38:31.646 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015060283000000001, 2020-06-12T02:38:31.646 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:31.646 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:31.646 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:31.646 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:31.647 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:31.647 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 7841 2020-06-12T02:38:31.647 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.647 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:31.647 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.647 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:31.648 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.996595+0000", 2020-06-12T02:38:31.648 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:31.648 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.648 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.648 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:31.649 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.996595+0000", 2020-06-12T02:38:31.649 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015120699999999999 2020-06-12T02:38:31.649 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.649 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.649 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:31.649 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.996746+0000", 2020-06-12T02:38:31.650 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4369999999999998e-06 2020-06-12T02:38:31.650 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.650 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.650 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:31.650 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.996752+0000", 2020-06-12T02:38:31.651 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0850000000000003e-06 2020-06-12T02:38:31.651 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.651 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.651 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:31.651 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.996761+0000", 2020-06-12T02:38:31.651 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2370000000000003e-06 2020-06-12T02:38:31.652 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.652 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.652 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:31.652 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.996765+0000", 2020-06-12T02:38:31.652 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5185999999999998e-05 2020-06-12T02:38:31.653 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.653 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.653 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:31.653 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.996800+0000", 2020-06-12T02:38:31.653 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7265000000000001e-05 2020-06-12T02:38:31.654 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.654 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.654 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:31.654 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.996868+0000", 2020-06-12T02:38:31.654 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0024047539999999998 2020-06-12T02:38:31.654 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.655 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.655 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:31.655 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.999272+0000", 2020-06-12T02:38:31.655 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3034999999999994e-05 2020-06-12T02:38:31.655 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.656 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.656 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:31.656 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.999355+0000", 2020-06-12T02:38:31.656 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049334499999999996 2020-06-12T02:38:31.656 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.656 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.657 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:31.657 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.999849+0000", 2020-06-12T02:38:31.657 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011708226 2020-06-12T02:38:31.657 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.657 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.658 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:31.658 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.011557+0000", 2020-06-12T02:38:31.658 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1449999999999997e-05 2020-06-12T02:38:31.658 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.658 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.658 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:31.659 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.011608+0000", 2020-06-12T02:38:31.659 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7055999999999998e-05 2020-06-12T02:38:31.659 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.659 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.659 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:31.660 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.011656+0000", 2020-06-12T02:38:31.660 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015060283000000001 2020-06-12T02:38:31.660 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:31.660 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:31.660 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:31.660 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.661 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.661 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:7845 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e33)", 2020-06-12T02:38:31.661 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:52.002017+0000", 2020-06-12T02:38:31.661 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 39.443267106999997, 2020-06-12T02:38:31.661 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017208558999999998, 2020-06-12T02:38:31.662 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:31.662 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:31.662 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:31.662 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:31.662 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:31.662 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 7845 2020-06-12T02:38:31.663 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.663 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:31.663 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.663 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:31.663 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.002017+0000", 2020-06-12T02:38:31.664 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:31.664 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.664 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.664 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:31.664 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.002017+0000", 2020-06-12T02:38:31.664 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2636000000000003e-05 2020-06-12T02:38:31.665 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.665 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.665 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:31.665 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.002060+0000", 2020-06-12T02:38:31.665 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5080000000000002e-06 2020-06-12T02:38:31.666 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.666 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.666 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:31.666 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.002067+0000", 2020-06-12T02:38:31.666 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3125000000000001e-05 2020-06-12T02:38:31.666 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.667 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.667 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:31.667 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.002080+0000", 2020-06-12T02:38:31.667 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5560000000000001e-06 2020-06-12T02:38:31.667 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.668 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.668 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:31.668 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.002084+0000", 2020-06-12T02:38:31.668 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0088536750000000001 2020-06-12T02:38:31.668 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.668 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.669 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:31.669 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.010938+0000", 2020-06-12T02:38:31.669 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3282999999999995e-05 2020-06-12T02:38:31.669 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.669 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.670 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:31.670 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.011021+0000", 2020-06-12T02:38:31.670 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0052873970000000001 2020-06-12T02:38:31.670 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.670 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.670 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:31.671 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.016309+0000", 2020-06-12T02:38:31.671 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010365399999999999 2020-06-12T02:38:31.671 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.671 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.671 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:31.672 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.016412+0000", 2020-06-12T02:38:31.672 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002813725 2020-06-12T02:38:31.672 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.672 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.672 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:31.672 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.019226+0000", 2020-06-12T02:38:31.673 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017208558999999998 2020-06-12T02:38:31.673 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:31.673 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:31.673 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:31.673 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.674 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.674 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8809 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~606208] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:31.674 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.707225+0000", 2020-06-12T02:38:31.674 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 19.738059669999998, 2020-06-12T02:38:31.674 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016771206, 2020-06-12T02:38:31.674 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:31.675 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:31.675 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:31.675 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:31.675 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:31.675 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8809 2020-06-12T02:38:31.676 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.676 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:31.676 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.676 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:31.676 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.707225+0000", 2020-06-12T02:38:31.676 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:31.677 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.677 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.677 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:31.677 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.707225+0000", 2020-06-12T02:38:31.677 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035342099999999998 2020-06-12T02:38:31.678 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.678 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.678 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:31.678 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.707578+0000", 2020-06-12T02:38:31.678 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2309999999999996e-06 2020-06-12T02:38:31.679 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.679 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.679 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:31.679 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.707587+0000", 2020-06-12T02:38:31.679 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4343e-05 2020-06-12T02:38:31.679 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.680 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.680 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:31.680 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.707601+0000", 2020-06-12T02:38:31.680 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8460000000000002e-06 2020-06-12T02:38:31.680 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.681 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.681 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:31.681 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.707608+0000", 2020-06-12T02:38:31.681 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7374999999999998e-05 2020-06-12T02:38:31.681 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.681 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.682 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:31.682 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.707675+0000", 2020-06-12T02:38:31.682 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111599 2020-06-12T02:38:31.682 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.682 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.683 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:31.683 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.707787+0000", 2020-06-12T02:38:31.683 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00079845100000000004 2020-06-12T02:38:31.683 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.683 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.683 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:31.684 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.708585+0000", 2020-06-12T02:38:31.684 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010899008999999999 2020-06-12T02:38:31.684 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.684 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.684 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:31.685 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.719484+0000", 2020-06-12T02:38:31.685 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306450000000004 2020-06-12T02:38:31.685 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.685 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.685 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:31.685 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.723915+0000", 2020-06-12T02:38:31.686 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1285999999999994e-05 2020-06-12T02:38:31.686 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.686 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.686 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:31.686 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.723996+0000", 2020-06-12T02:38:31.687 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016771206 2020-06-12T02:38:31.687 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:31.687 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:31.687 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:31.687 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.687 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.688 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.3s1 49/21 ECSubWrite(tid=4109, reqid=client.4512.0:8841, at_version=49'773, trim_to=0'0, roll_forward_to=49'771))", 2020-06-12T02:38:31.688 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.822295+0000", 2020-06-12T02:38:31.688 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 19.622989099000002, 2020-06-12T02:38:31.688 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01733517, 2020-06-12T02:38:31.688 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:31.689 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:31.689 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:31.689 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.689 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:31.689 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822295+0000", 2020-06-12T02:38:31.690 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:31.690 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.690 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.690 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:31.690 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822295+0000", 2020-06-12T02:38:31.690 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00089525499999999997 2020-06-12T02:38:31.691 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.691 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.691 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:31.691 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.823191+0000", 2020-06-12T02:38:31.691 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3315999999999998e-05 2020-06-12T02:38:31.692 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.692 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.692 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:31.692 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.823214+0000", 2020-06-12T02:38:31.692 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7096e-05 2020-06-12T02:38:31.692 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.693 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.693 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:31.693 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.823231+0000", 2020-06-12T02:38:31.693 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6890000000000004e-06 2020-06-12T02:38:31.693 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.694 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.694 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:31.694 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.823239+0000", 2020-06-12T02:38:31.694 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00061031599999999996 2020-06-12T02:38:31.694 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.694 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.695 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:31.695 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.823849+0000", 2020-06-12T02:38:31.695 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4205999999999999e-05 2020-06-12T02:38:31.695 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.695 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.696 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:31.696 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.823863+0000", 2020-06-12T02:38:31.696 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015707789999999999 2020-06-12T02:38:31.696 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.696 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.696 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:31.697 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.839571+0000", 2020-06-12T02:38:31.697 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9502000000000001e-05 2020-06-12T02:38:31.697 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.698 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.698 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:31.698 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.839631+0000", 2020-06-12T02:38:31.698 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01733517 2020-06-12T02:38:31.698 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:31.699 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:31.699 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:31.699 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.699 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.699 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8854 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~409600] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:31.700 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.854691+0000", 2020-06-12T02:38:31.700 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 19.590593216999999, 2020-06-12T02:38:31.700 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01967967, 2020-06-12T02:38:31.700 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:31.700 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:31.701 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:31.701 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:31.701 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:31.701 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8854 2020-06-12T02:38:31.701 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.702 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:31.702 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.702 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:31.702 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.854691+0000", 2020-06-12T02:38:31.702 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:31.702 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.703 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.703 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:31.703 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.854691+0000", 2020-06-12T02:38:31.703 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017767799999999999 2020-06-12T02:38:31.703 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.704 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.704 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:31.704 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.854869+0000", 2020-06-12T02:38:31.704 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8049999999999996e-06 2020-06-12T02:38:31.704 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.704 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.705 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:31.705 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.854878+0000", 2020-06-12T02:38:31.705 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5644000000000001e-05 2020-06-12T02:38:31.705 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.705 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.706 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:31.706 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.854893+0000", 2020-06-12T02:38:31.706 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.584e-06 2020-06-12T02:38:31.706 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.706 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.706 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:31.707 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.854900+0000", 2020-06-12T02:38:31.707 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002416769 2020-06-12T02:38:31.707 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.707 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.707 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:31.708 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.857317+0000", 2020-06-12T02:38:31.708 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000100385 2020-06-12T02:38:31.708 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.708 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.708 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:31.708 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.857417+0000", 2020-06-12T02:38:31.709 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0047283969999999996 2020-06-12T02:38:31.709 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.709 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.709 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:31.709 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.862146+0000", 2020-06-12T02:38:31.710 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1930999999999998e-05 2020-06-12T02:38:31.710 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.710 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.710 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:31.710 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.862238+0000", 2020-06-12T02:38:31.710 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00071479400000000004 2020-06-12T02:38:31.711 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.711 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.711 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:31.711 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.862952+0000", 2020-06-12T02:38:31.711 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011219316999999999 2020-06-12T02:38:31.712 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.712 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.712 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:31.712 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.874172+0000", 2020-06-12T02:38:31.712 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000104137 2020-06-12T02:38:31.712 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.713 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.713 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:31.713 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.874276+0000", 2020-06-12T02:38:31.713 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5229000000000006e-05 2020-06-12T02:38:31.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.714 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:31.714 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.874371+0000", 2020-06-12T02:38:31.714 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01967967 2020-06-12T02:38:31.714 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:31.715 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:31.715 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:31.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.715 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.715 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8855 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~516096] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:31.715 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.858551+0000", 2020-06-12T02:38:31.716 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 19.586733359, 2020-06-12T02:38:31.716 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020039164000000002, 2020-06-12T02:38:31.716 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:31.716 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:31.716 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:31.717 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:31.717 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:31.717 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8855 2020-06-12T02:38:31.717 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.717 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:31.717 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.718 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:31.718 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.858551+0000", 2020-06-12T02:38:31.718 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:31.718 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.718 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.719 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:31.719 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.858551+0000", 2020-06-12T02:38:31.719 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00020537100000000001 2020-06-12T02:38:31.719 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.719 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.719 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:31.720 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.858757+0000", 2020-06-12T02:38:31.720 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4870000000000002e-06 2020-06-12T02:38:31.720 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.720 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.720 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:31.721 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.858765+0000", 2020-06-12T02:38:31.721 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4358e-05 2020-06-12T02:38:31.721 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.721 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.721 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:31.721 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.858779+0000", 2020-06-12T02:38:31.722 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3879999999999997e-06 2020-06-12T02:38:31.722 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.722 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.722 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:31.722 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.858786+0000", 2020-06-12T02:38:31.723 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089100000000001 2020-06-12T02:38:31.723 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.723 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.723 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:31.723 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.859127+0000", 2020-06-12T02:38:31.723 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0877000000000005e-05 2020-06-12T02:38:31.724 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.724 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.724 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:31.724 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.859208+0000", 2020-06-12T02:38:31.724 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0059661269999999999 2020-06-12T02:38:31.725 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.725 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.725 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:31.725 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.865174+0000", 2020-06-12T02:38:31.725 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000102761 2020-06-12T02:38:31.725 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.726 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.726 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:31.726 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.865276+0000", 2020-06-12T02:38:31.726 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00079971700000000003 2020-06-12T02:38:31.726 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.727 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.727 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:31.727 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.866076+0000", 2020-06-12T02:38:31.727 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0083001380000000003 2020-06-12T02:38:31.727 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.727 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.728 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:31.728 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.874376+0000", 2020-06-12T02:38:31.728 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041190660000000002 2020-06-12T02:38:31.728 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.728 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.729 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:31.729 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.878495+0000", 2020-06-12T02:38:31.729 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4982999999999995e-05 2020-06-12T02:38:31.729 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.729 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.729 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:31.730 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.878590+0000", 2020-06-12T02:38:31.730 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020039164000000002 2020-06-12T02:38:31.730 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:31.730 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:31.730 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:31.731 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.731 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.731 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8856 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~442368] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:31.731 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.861853+0000", 2020-06-12T02:38:31.731 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 19.583431910000002, 2020-06-12T02:38:31.731 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022102294000000001, 2020-06-12T02:38:31.732 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:31.732 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:31.732 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:31.732 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:31.732 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:31.733 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8856 2020-06-12T02:38:31.733 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.733 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:31.733 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.733 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:31.734 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.861853+0000", 2020-06-12T02:38:31.734 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:31.734 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.734 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.734 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:31.735 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.861853+0000", 2020-06-12T02:38:31.735 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017153099999999999 2020-06-12T02:38:31.735 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.735 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.735 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:31.735 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.862024+0000", 2020-06-12T02:38:31.736 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0889999999999999e-06 2020-06-12T02:38:31.736 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.736 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.736 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:31.736 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.862029+0000", 2020-06-12T02:38:31.737 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3033999999999999e-05 2020-06-12T02:38:31.737 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.737 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.737 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:31.737 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.862042+0000", 2020-06-12T02:38:31.737 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1349999999999997e-06 2020-06-12T02:38:31.738 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.738 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.738 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:31.738 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.862048+0000", 2020-06-12T02:38:31.738 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0081330410000000006 2020-06-12T02:38:31.739 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.739 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.739 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:31.739 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.870181+0000", 2020-06-12T02:38:31.739 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012252500000000001 2020-06-12T02:38:31.739 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.740 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.740 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:31.740 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.870304+0000", 2020-06-12T02:38:31.740 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000740999 2020-06-12T02:38:31.740 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.741 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.741 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:31.741 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.871045+0000", 2020-06-12T02:38:31.741 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012634975 2020-06-12T02:38:31.741 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.741 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.742 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:31.742 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.883680+0000", 2020-06-12T02:38:31.742 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017667900000000001 2020-06-12T02:38:31.742 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.742 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.742 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:31.743 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.883857+0000", 2020-06-12T02:38:31.743 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8286000000000001e-05 2020-06-12T02:38:31.743 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.743 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.744 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:31.744 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.883955+0000", 2020-06-12T02:38:31.744 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022102294000000001 2020-06-12T02:38:31.744 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:31.744 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:31.745 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:31.745 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.745 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.745 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8857 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~237568] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:31.745 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.863733+0000", 2020-06-12T02:38:31.745 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 19.581551927, 2020-06-12T02:38:31.746 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020470258000000002, 2020-06-12T02:38:31.746 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:31.746 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:31.746 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:31.746 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:31.747 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:31.747 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8857 2020-06-12T02:38:31.747 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.747 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:31.747 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.748 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:31.748 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.863733+0000", 2020-06-12T02:38:31.748 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:31.748 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.748 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.748 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:31.749 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.863733+0000", 2020-06-12T02:38:31.749 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014776000000000001 2020-06-12T02:38:31.749 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.749 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.749 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:31.750 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.863880+0000", 2020-06-12T02:38:31.750 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5790000000000004e-06 2020-06-12T02:38:31.750 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.750 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.750 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:31.750 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.863889+0000", 2020-06-12T02:38:31.751 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6960999999999999e-05 2020-06-12T02:38:31.751 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.751 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.751 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:31.751 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.863906+0000", 2020-06-12T02:38:31.752 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5509999999999996e-06 2020-06-12T02:38:31.752 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.752 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.752 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:31.752 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.863913+0000", 2020-06-12T02:38:31.752 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010569258 2020-06-12T02:38:31.753 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.753 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.753 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:31.753 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.874482+0000", 2020-06-12T02:38:31.753 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011260399999999999 2020-06-12T02:38:31.754 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.754 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.754 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:31.754 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.874594+0000", 2020-06-12T02:38:31.754 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014913820000000001 2020-06-12T02:38:31.754 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.755 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.755 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:31.755 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.876086+0000", 2020-06-12T02:38:31.755 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0078744260000000003 2020-06-12T02:38:31.755 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.756 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.756 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:31.756 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.883960+0000", 2020-06-12T02:38:31.756 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015276999999999999 2020-06-12T02:38:31.756 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.756 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.757 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:31.757 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.884113+0000", 2020-06-12T02:38:31.757 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9967000000000001e-05 2020-06-12T02:38:31.757 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.757 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.758 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:31.758 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.884203+0000", 2020-06-12T02:38:31.758 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020470258000000002 2020-06-12T02:38:31.758 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:31.758 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:31.758 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:31.759 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.759 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.759 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8858 3.0s0 3:0dddb690:::smithi10729407-10:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:31.759 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.863968+0000", 2020-06-12T02:38:31.759 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 19.581316635, 2020-06-12T02:38:31.760 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022225373999999999, 2020-06-12T02:38:31.760 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:31.760 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:31.760 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:31.760 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:31.761 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:31.761 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8858 2020-06-12T02:38:31.761 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.761 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:31.761 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.761 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:31.762 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.863968+0000", 2020-06-12T02:38:31.762 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:31.762 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.762 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.762 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:31.763 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.863968+0000", 2020-06-12T02:38:31.763 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5352000000000006e-05 2020-06-12T02:38:31.763 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.763 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.763 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:31.763 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864043+0000", 2020-06-12T02:38:31.764 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9110000000000004e-06 2020-06-12T02:38:31.764 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.764 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.764 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:31.764 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864050+0000", 2020-06-12T02:38:31.765 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2505e-05 2020-06-12T02:38:31.765 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.765 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.765 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:31.765 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864063+0000", 2020-06-12T02:38:31.765 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0900000000000004e-06 2020-06-12T02:38:31.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.766 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.766 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:31.766 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864068+0000", 2020-06-12T02:38:31.766 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014563938 2020-06-12T02:38:31.767 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.767 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.767 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:31.767 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.878632+0000", 2020-06-12T02:38:31.767 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000106852 2020-06-12T02:38:31.767 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.768 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.768 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:31.768 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.878739+0000", 2020-06-12T02:38:31.768 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002931796 2020-06-12T02:38:31.768 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.768 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.769 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:31.769 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.881670+0000", 2020-06-12T02:38:31.769 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025365280000000001 2020-06-12T02:38:31.769 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.769 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.770 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:31.770 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.884207+0000", 2020-06-12T02:38:31.770 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001895224 2020-06-12T02:38:31.770 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.770 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.770 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:31.771 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.886102+0000", 2020-06-12T02:38:31.771 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1178e-05 2020-06-12T02:38:31.771 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.771 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.771 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:31.772 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.886193+0000", 2020-06-12T02:38:31.772 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022225373999999999 2020-06-12T02:38:31.772 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:31.772 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:31.772 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:31.772 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.773 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.773 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8859 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e49)", 2020-06-12T02:38:31.773 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.864115+0000", 2020-06-12T02:38:31.773 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 19.581169733999999, 2020-06-12T02:38:31.773 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.03325032, 2020-06-12T02:38:31.774 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:31.774 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:31.774 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:31.774 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:31.774 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:31.774 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8859 2020-06-12T02:38:31.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.775 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:31.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.775 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:31.775 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864115+0000", 2020-06-12T02:38:31.776 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:31.776 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.776 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.776 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:31.776 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864115+0000", 2020-06-12T02:38:31.776 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6795999999999997e-05 2020-06-12T02:38:31.777 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.777 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.777 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:31.777 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864162+0000", 2020-06-12T02:38:31.777 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5599999999999999e-06 2020-06-12T02:38:31.778 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.778 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.778 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:31.778 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864168+0000", 2020-06-12T02:38:31.778 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5658000000000002e-05 2020-06-12T02:38:31.778 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.779 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.779 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:31.779 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864184+0000", 2020-06-12T02:38:31.779 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8380000000000001e-06 2020-06-12T02:38:31.779 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.780 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.780 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:31.780 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864189+0000", 2020-06-12T02:38:31.780 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019365752 2020-06-12T02:38:31.780 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.780 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.781 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:31.781 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.883554+0000", 2020-06-12T02:38:31.781 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000118819 2020-06-12T02:38:31.781 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.781 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.782 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:31.782 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.883673+0000", 2020-06-12T02:38:31.782 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0089243440000000007 2020-06-12T02:38:31.782 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.782 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.782 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:31.783 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.892598+0000", 2020-06-12T02:38:31.783 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7429999999999994e-05 2020-06-12T02:38:31.783 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.783 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.783 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:31.784 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.892695+0000", 2020-06-12T02:38:31.784 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004670123 2020-06-12T02:38:31.786 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:31.787 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:31.787 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:31.787 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.897365+0000", 2020-06-12T02:38:31.787 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.03325032 2020-06-12T02:38:31.787 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:31.788 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:31.788 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:31.788 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:31.788 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:31.788 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:31.789 INFO:tasks.rados.rados.0.smithi107.stdout:6901: done (0 left) 2020-06-12T02:38:31.789 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:31.789 INFO:tasks.rados.rados.0.smithi107.stdout:6902: seq_num 6901 ranges {0=385024} 2020-06-12T02:38:31.806 INFO:tasks.rados.rados.0.smithi107.stdout:6902: writing smithi10729407-12 from 0 to 385024 tid 1 2020-06-12T02:38:31.807 INFO:tasks.rados.rados.0.smithi107.stdout:6903: delete oid 6 current snap is 0 2020-06-12T02:38:31.807 INFO:tasks.rados.rados.0.smithi107.stdout:6903: done (1 left) 2020-06-12T02:38:31.807 INFO:tasks.rados.rados.0.smithi107.stdout:6904: delete oid 5 current snap is 0 2020-06-12T02:38:31.807 INFO:tasks.rados.rados.0.smithi107.stdout:6902: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:31.807 INFO:tasks.rados.rados.0.smithi107.stdout:6904: done (1 left) 2020-06-12T02:38:31.808 INFO:tasks.rados.rados.0.smithi107.stdout:6905: delete oid 18 current snap is 0 2020-06-12T02:38:31.808 INFO:tasks.rados.rados.0.smithi107.stdout:6902: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:31.808 INFO:tasks.rados.rados.0.smithi107.stdout:6905: done (1 left) 2020-06-12T02:38:31.808 INFO:tasks.rados.rados.0.smithi107.stdout:6906: delete oid 14 current snap is 0 2020-06-12T02:38:31.808 INFO:tasks.rados.rados.0.smithi107.stdout:6902: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:31.808 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1270 (ObjNum 6901 snap 0 seq_num 6901) dirty exists 2020-06-12T02:38:31.809 INFO:tasks.rados.rados.0.smithi107.stdout:6902: left oid 12 (ObjNum 6901 snap 0 seq_num 6901) 2020-06-12T02:38:31.809 INFO:tasks.rados.rados.0.smithi107.stdout:6902: done (1 left) 2020-06-12T02:38:31.809 INFO:tasks.rados.rados.0.smithi107.stdout:6906: done (0 left) 2020-06-12T02:38:31.809 INFO:tasks.rados.rados.0.smithi107.stdout:6907: delete oid 6 current snap is 0 2020-06-12T02:38:31.809 INFO:tasks.rados.rados.0.smithi107.stdout:6907: done (0 left) 2020-06-12T02:38:31.810 INFO:tasks.rados.rados.0.smithi107.stdout:6908: delete oid 18 current snap is 0 2020-06-12T02:38:31.810 INFO:tasks.rados.rados.0.smithi107.stdout:6908: done (0 left) 2020-06-12T02:38:31.810 INFO:tasks.rados.rados.0.smithi107.stdout:6909: delete oid 20 current snap is 0 2020-06-12T02:38:31.810 INFO:tasks.rados.rados.0.smithi107.stdout:6909: done (0 left) 2020-06-12T02:38:31.810 INFO:tasks.rados.rados.0.smithi107.stdout:6910: delete oid 16 current snap is 0 2020-06-12T02:38:31.811 INFO:tasks.rados.rados.0.smithi107.stdout:6910: done (0 left) 2020-06-12T02:38:31.811 INFO:tasks.rados.rados.0.smithi107.stdout:6911: delete oid 8 current snap is 0 2020-06-12T02:38:31.811 INFO:tasks.rados.rados.0.smithi107.stdout:6911: done (0 left) 2020-06-12T02:38:31.811 INFO:tasks.rados.rados.0.smithi107.stdout:6912: delete oid 3 current snap is 0 2020-06-12T02:38:31.811 INFO:tasks.rados.rados.0.smithi107.stdout:6912: done (0 left) 2020-06-12T02:38:31.811 INFO:tasks.rados.rados.0.smithi107.stdout:6913: delete oid 6 current snap is 0 2020-06-12T02:38:31.812 INFO:tasks.rados.rados.0.smithi107.stdout:6913: done (0 left) 2020-06-12T02:38:31.812 INFO:tasks.rados.rados.0.smithi107.stdout:6914: delete oid 6 current snap is 0 2020-06-12T02:38:31.812 INFO:tasks.rados.rados.0.smithi107.stdout:6914: done (0 left) 2020-06-12T02:38:31.812 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:31.812 INFO:tasks.rados.rados.0.smithi107.stdout:6915: seq_num 6914 ranges {0=606208} 2020-06-12T02:38:31.813 INFO:tasks.rados.rados.0.smithi107.stdout:6915: writing smithi10729407-8 from 0 to 606208 tid 1 2020-06-12T02:38:31.813 INFO:tasks.rados.rados.0.smithi107.stdout:6916: delete oid 17 current snap is 0 2020-06-12T02:38:31.813 INFO:tasks.rados.rados.0.smithi107.stdout:6916: done (1 left) 2020-06-12T02:38:31.813 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:31.813 INFO:tasks.rados.rados.0.smithi107.stdout:6917: seq_num 6916 ranges {0=638976} 2020-06-12T02:38:31.813 INFO:tasks.rados.rados.0.smithi107.stdout:6917: writing smithi10729407-4 from 0 to 638976 tid 1 2020-06-12T02:38:31.814 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:31.814 INFO:tasks.rados.rados.0.smithi107.stdout:6918: seq_num 6917 ranges {0=655360} 2020-06-12T02:38:31.814 INFO:tasks.rados.rados.0.smithi107.stdout:6918: writing smithi10729407-9 from 0 to 655360 tid 1 2020-06-12T02:38:31.814 INFO:tasks.rados.rados.0.smithi107.stdout:6919: delete oid 1 current snap is 0 2020-06-12T02:38:31.814 INFO:tasks.rados.rados.0.smithi107.stdout:6915: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:31.815 INFO:tasks.rados.rados.0.smithi107.stdout:6915: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:31.815 INFO:tasks.rados.rados.0.smithi107.stdout:6915: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:31.815 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1288 (ObjNum 6914 snap 0 seq_num 6914) dirty exists 2020-06-12T02:38:31.815 INFO:tasks.rados.rados.0.smithi107.stdout:6915: left oid 8 (ObjNum 6914 snap 0 seq_num 6914) 2020-06-12T02:38:31.815 INFO:tasks.rados.rados.0.smithi107.stdout:6917: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:31.815 INFO:tasks.rados.rados.0.smithi107.stdout:6917: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:31.816 INFO:tasks.rados.rados.0.smithi107.stdout:6915: done (3 left) 2020-06-12T02:38:31.816 INFO:tasks.rados.rados.0.smithi107.stdout:6919: done (2 left) 2020-06-12T02:38:31.816 INFO:tasks.rados.rados.0.smithi107.stdout:6920: delete oid 7 current snap is 0 2020-06-12T02:38:31.816 INFO:tasks.rados.rados.0.smithi107.stdout:6917: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:31.816 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1311 (ObjNum 6916 snap 0 seq_num 6916) dirty exists 2020-06-12T02:38:31.817 INFO:tasks.rados.rados.0.smithi107.stdout:6917: left oid 4 (ObjNum 6916 snap 0 seq_num 6916) 2020-06-12T02:38:31.817 INFO:tasks.rados.rados.0.smithi107.stdout:6917: done (2 left) 2020-06-12T02:38:31.817 INFO:tasks.rados.rados.0.smithi107.stdout:6920: done (1 left) 2020-06-12T02:38:31.817 INFO:tasks.rados.rados.0.smithi107.stdout:6921: delete oid 12 current snap is 0 2020-06-12T02:38:31.817 INFO:tasks.rados.rados.0.smithi107.stdout:6918: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:31.817 INFO:tasks.rados.rados.0.smithi107.stdout:6918: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:31.818 INFO:tasks.rados.rados.0.smithi107.stdout:6921: done (1 left) 2020-06-12T02:38:31.818 INFO:tasks.rados.rados.0.smithi107.stdout:6922: delete oid 15 current snap is 0 2020-06-12T02:38:31.818 INFO:tasks.rados.rados.0.smithi107.stdout:6918: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:31.818 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1281 (ObjNum 6917 snap 0 seq_num 6917) dirty exists 2020-06-12T02:38:31.818 INFO:tasks.rados.rados.0.smithi107.stdout:6918: left oid 9 (ObjNum 6917 snap 0 seq_num 6917) 2020-06-12T02:38:31.819 INFO:tasks.rados.rados.0.smithi107.stdout:6918: done (1 left) 2020-06-12T02:38:31.819 INFO:tasks.rados.rados.0.smithi107.stdout:6922: done (0 left) 2020-06-12T02:38:31.819 INFO:tasks.rados.rados.0.smithi107.stdout:6923: delete oid 8 current snap is 0 2020-06-12T02:38:31.819 INFO:tasks.rados.rados.0.smithi107.stdout:6923: done (0 left) 2020-06-12T02:38:31.819 INFO:tasks.rados.rados.0.smithi107.stdout:6924: delete oid 3 current snap is 0 2020-06-12T02:38:31.819 INFO:tasks.rados.rados.0.smithi107.stdout:6924: done (0 left) 2020-06-12T02:38:31.820 INFO:tasks.rados.rados.0.smithi107.stdout:6925: delete oid 6 current snap is 0 2020-06-12T02:38:31.820 INFO:tasks.rados.rados.0.smithi107.stdout:6925: done (0 left) 2020-06-12T02:38:31.820 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:31.820 INFO:tasks.rados.rados.0.smithi107.stdout:6926: seq_num 6925 ranges {0=786432} 2020-06-12T02:38:31.820 INFO:tasks.rados.rados.0.smithi107.stdout:6926: writing smithi10729407-19 from 0 to 786432 tid 1 2020-06-12T02:38:31.821 INFO:tasks.rados.rados.0.smithi107.stdout:6927: delete oid 2 current snap is 0 2020-06-12T02:38:31.821 INFO:tasks.rados.rados.0.smithi107.stdout:6927: done (1 left) 2020-06-12T02:38:31.821 INFO:tasks.rados.rados.0.smithi107.stdout:6928: delete oid 18 current snap is 0 2020-06-12T02:38:31.821 INFO:tasks.rados.rados.0.smithi107.stdout:6928: done (1 left) 2020-06-12T02:38:31.821 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:31.821 INFO:tasks.rados.rados.0.smithi107.stdout:6929: seq_num 6928 ranges {0=434176,434176=401408} 2020-06-12T02:38:31.822 INFO:tasks.rados.rados.0.smithi107.stdout:6929: writing smithi10729407-18 from 0 to 434176 tid 1 2020-06-12T02:38:31.822 INFO:tasks.rados.rados.0.smithi107.stdout:6929: writing smithi10729407-18 from 434176 to 835584 tid 2 2020-06-12T02:38:31.822 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:31.822 INFO:tasks.rados.rados.0.smithi107.stdout:6930: seq_num 6929 ranges {0=450560,450560=401408} 2020-06-12T02:38:31.822 INFO:tasks.rados.rados.0.smithi107.stdout:6930: writing smithi10729407-15 from 0 to 450560 tid 1 2020-06-12T02:38:31.823 INFO:tasks.rados.rados.0.smithi107.stdout:6930: writing smithi10729407-15 from 450560 to 851968 tid 2 2020-06-12T02:38:31.823 INFO:tasks.rados.rados.0.smithi107.stdout:6931: delete oid 16 current snap is 0 2020-06-12T02:38:31.823 INFO:tasks.rados.rados.0.smithi107.stdout:6926: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:31.823 INFO:tasks.rados.rados.0.smithi107.stdout:6926: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:31.823 INFO:tasks.rados.rados.0.smithi107.stdout:6929: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:31.823 INFO:tasks.rados.rados.0.smithi107.stdout:6926: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:31.824 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1273 (ObjNum 6925 snap 0 seq_num 6925) dirty exists 2020-06-12T02:38:31.824 INFO:tasks.rados.rados.0.smithi107.stdout:6926: left oid 19 (ObjNum 6925 snap 0 seq_num 6925) 2020-06-12T02:38:31.824 INFO:tasks.rados.rados.0.smithi107.stdout:6929: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:31.824 INFO:tasks.rados.rados.0.smithi107.stdout:6930: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:31.824 INFO:tasks.rados.rados.0.smithi107.stdout:6929: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:31.825 INFO:tasks.rados.rados.0.smithi107.stdout:6926: done (3 left) 2020-06-12T02:38:31.826 INFO:tasks.rados.rados.0.smithi107.stdout:6931: done (2 left) 2020-06-12T02:38:31.826 INFO:tasks.rados.rados.0.smithi107.stdout:6932: delete oid 10 current snap is 0 2020-06-12T02:38:31.826 INFO:tasks.rados.rados.0.smithi107.stdout:6929: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:31.826 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1285 (ObjNum 6928 snap 0 seq_num 6928) dirty exists 2020-06-12T02:38:31.826 INFO:tasks.rados.rados.0.smithi107.stdout:6929: left oid 18 (ObjNum 6928 snap 0 seq_num 6928) 2020-06-12T02:38:31.827 INFO:tasks.rados.rados.0.smithi107.stdout:6929: done (2 left) 2020-06-12T02:38:31.827 INFO:tasks.rados.rados.0.smithi107.stdout:6932: done (1 left) 2020-06-12T02:38:31.828 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:31.828 INFO:tasks.rados.rados.0.smithi107.stdout:6933: seq_num 6932 ranges {786432=499712,1286144=401408} 2020-06-12T02:38:31.828 INFO:tasks.rados.rados.0.smithi107.stdout:6933: writing smithi10729407-19 from 786432 to 1286144 tid 1 2020-06-12T02:38:31.828 INFO:tasks.rados.rados.0.smithi107.stdout:6933: writing smithi10729407-19 from 1286144 to 1687552 tid 2 2020-06-12T02:38:31.829 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:31.829 INFO:tasks.rados.rados.0.smithi107.stdout:6934: seq_num 6933 ranges {0=516096,516096=401408} 2020-06-12T02:38:31.829 INFO:tasks.rados.rados.0.smithi107.stdout:6934: writing smithi10729407-5 from 0 to 516096 tid 1 2020-06-12T02:38:31.829 INFO:tasks.rados.rados.0.smithi107.stdout:6934: writing smithi10729407-5 from 516096 to 917504 tid 2 2020-06-12T02:38:31.829 INFO:tasks.rados.rados.0.smithi107.stdout:6935: delete oid 13 current snap is 0 2020-06-12T02:38:31.830 INFO:tasks.rados.rados.0.smithi107.stdout:6930: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:31.830 INFO:tasks.rados.rados.0.smithi107.stdout:6930: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:31.830 INFO:tasks.rados.rados.0.smithi107.stdout:6930: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:31.830 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1293 (ObjNum 6929 snap 0 seq_num 6929) dirty exists 2020-06-12T02:38:31.830 INFO:tasks.rados.rados.0.smithi107.stdout:6930: left oid 15 (ObjNum 6929 snap 0 seq_num 6929) 2020-06-12T02:38:31.831 INFO:tasks.rados.rados.0.smithi107.stdout:6933: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:31.831 INFO:tasks.rados.rados.0.smithi107.stdout:6933: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:31.831 INFO:tasks.rados.rados.0.smithi107.stdout:6930: done (3 left) 2020-06-12T02:38:31.831 INFO:tasks.rados.rados.0.smithi107.stdout:6935: done (2 left) 2020-06-12T02:38:31.831 INFO:tasks.rados.rados.0.smithi107.stdout:6936: delete oid 1 current snap is 0 2020-06-12T02:38:31.832 INFO:tasks.rados.rados.0.smithi107.stdout:6933: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:31.832 INFO:tasks.rados.rados.0.smithi107.stdout:6933: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:31.832 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1276 (ObjNum 6932 snap 0 seq_num 6932) dirty exists 2020-06-12T02:38:31.832 INFO:tasks.rados.rados.0.smithi107.stdout:6933: left oid 19 (ObjNum 6932 snap 0 seq_num 6932) 2020-06-12T02:38:31.832 INFO:tasks.rados.rados.0.smithi107.stdout:6934: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:31.832 INFO:tasks.rados.rados.0.smithi107.stdout:6934: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:31.833 INFO:tasks.rados.rados.0.smithi107.stdout:6933: done (2 left) 2020-06-12T02:38:31.833 INFO:tasks.rados.rados.0.smithi107.stdout:6936: done (1 left) 2020-06-12T02:38:31.833 INFO:tasks.rados.rados.0.smithi107.stdout:6937: delete oid 19 current snap is 0 2020-06-12T02:38:31.833 INFO:tasks.rados.rados.0.smithi107.stdout:6937: done (1 left) 2020-06-12T02:38:31.833 INFO:tasks.rados.rados.0.smithi107.stdout:6938: delete oid 13 current snap is 0 2020-06-12T02:38:31.834 INFO:tasks.rados.rados.0.smithi107.stdout:6934: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:31.834 INFO:tasks.rados.rados.0.smithi107.stdout:6938: done (1 left) 2020-06-12T02:38:31.834 INFO:tasks.rados.rados.0.smithi107.stdout:6939: delete oid 10 current snap is 0 2020-06-12T02:38:31.834 INFO:tasks.rados.rados.0.smithi107.stdout:6934: finishing write tid 5 to smithi10729407-5 2020-06-12T02:38:31.834 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 688 (ObjNum 6933 snap 0 seq_num 6933) dirty exists 2020-06-12T02:38:31.835 INFO:tasks.rados.rados.0.smithi107.stdout:6934: left oid 5 (ObjNum 6933 snap 0 seq_num 6933) 2020-06-12T02:38:31.835 INFO:tasks.rados.rados.0.smithi107.stdout:6934: done (1 left) 2020-06-12T02:38:31.835 INFO:tasks.rados.rados.0.smithi107.stdout:6939: done (0 left) 2020-06-12T02:38:31.835 INFO:tasks.rados.rados.0.smithi107.stdout:6940: delete oid 1 current snap is 0 2020-06-12T02:38:31.835 INFO:tasks.rados.rados.0.smithi107.stdout:6940: done (0 left) 2020-06-12T02:38:31.835 INFO:tasks.rados.rados.0.smithi107.stdout:6941: delete oid 13 current snap is 0 2020-06-12T02:38:31.836 INFO:tasks.rados.rados.0.smithi107.stdout:6941: done (0 left) 2020-06-12T02:38:31.836 INFO:tasks.rados.rados.0.smithi107.stdout:6942: delete oid 2 current snap is 0 2020-06-12T02:38:31.836 INFO:tasks.rados.rados.0.smithi107.stdout:6942: done (0 left) 2020-06-12T02:38:31.836 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:31.836 INFO:tasks.rados.rados.0.smithi107.stdout:6943: seq_num 6942 ranges {851968=671744,1523712=401408} 2020-06-12T02:38:31.837 INFO:tasks.rados.rados.0.smithi107.stdout:6943: writing smithi10729407-15 from 851968 to 1523712 tid 1 2020-06-12T02:38:31.837 INFO:tasks.rados.rados.0.smithi107.stdout:6943: writing smithi10729407-15 from 1523712 to 1925120 tid 2 2020-06-12T02:38:31.837 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:31.837 INFO:tasks.rados.rados.0.smithi107.stdout:6944: seq_num 6943 ranges {0=688128,688128=401408} 2020-06-12T02:38:31.838 INFO:tasks.rados.rados.0.smithi107.stdout:6944: writing smithi10729407-2 from 0 to 688128 tid 1 2020-06-12T02:38:31.838 INFO:tasks.rados.rados.0.smithi107.stdout:6944: writing smithi10729407-2 from 688128 to 1089536 tid 2 2020-06-12T02:38:31.838 INFO:tasks.rados.rados.0.smithi107.stdout:6945: delete oid 6 current snap is 0 2020-06-12T02:38:31.838 INFO:tasks.rados.rados.0.smithi107.stdout:6943: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:31.838 INFO:tasks.rados.rados.0.smithi107.stdout:6943: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:31.839 INFO:tasks.rados.rados.0.smithi107.stdout:6943: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:31.839 INFO:tasks.rados.rados.0.smithi107.stdout:6943: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:31.839 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1296 (ObjNum 6942 snap 0 seq_num 6942) dirty exists 2020-06-12T02:38:31.839 INFO:tasks.rados.rados.0.smithi107.stdout:6943: left oid 15 (ObjNum 6942 snap 0 seq_num 6942) 2020-06-12T02:38:31.839 INFO:tasks.rados.rados.0.smithi107.stdout:6944: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:31.839 INFO:tasks.rados.rados.0.smithi107.stdout:6943: done (2 left) 2020-06-12T02:38:31.840 INFO:tasks.rados.rados.0.smithi107.stdout:6945: done (1 left) 2020-06-12T02:38:31.840 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:31.840 INFO:tasks.rados.rados.0.smithi107.stdout:6946: seq_num 6945 ranges {655360=720896,1376256=401408} 2020-06-12T02:38:31.840 INFO:tasks.rados.rados.0.smithi107.stdout:6946: writing smithi10729407-9 from 655360 to 1376256 tid 1 2020-06-12T02:38:31.840 INFO:tasks.rados.rados.0.smithi107.stdout:6946: writing smithi10729407-9 from 1376256 to 1777664 tid 2 2020-06-12T02:38:31.841 INFO:tasks.rados.rados.0.smithi107.stdout:6947: delete oid 10 current snap is 0 2020-06-12T02:38:31.841 INFO:tasks.rados.rados.0.smithi107.stdout:6944: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:31.841 INFO:tasks.rados.rados.0.smithi107.stdout:6944: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:31.841 INFO:tasks.rados.rados.0.smithi107.stdout:6944: finishing write tid 5 to smithi10729407-2 2020-06-12T02:38:31.841 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 663 (ObjNum 6943 snap 0 seq_num 6943) dirty exists 2020-06-12T02:38:31.842 INFO:tasks.rados.rados.0.smithi107.stdout:6944: left oid 2 (ObjNum 6943 snap 0 seq_num 6943) 2020-06-12T02:38:31.842 INFO:tasks.rados.rados.0.smithi107.stdout:6944: done (2 left) 2020-06-12T02:38:31.842 INFO:tasks.rados.rados.0.smithi107.stdout:6947: done (1 left) 2020-06-12T02:38:31.842 INFO:tasks.rados.rados.0.smithi107.stdout:6948: delete oid 14 current snap is 0 2020-06-12T02:38:31.842 INFO:tasks.rados.rados.0.smithi107.stdout:6946: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:31.843 INFO:tasks.rados.rados.0.smithi107.stdout:6948: done (1 left) 2020-06-12T02:38:31.843 INFO:tasks.rados.rados.0.smithi107.stdout:6949: delete oid 14 current snap is 0 2020-06-12T02:38:31.843 INFO:tasks.rados.rados.0.smithi107.stdout:6949: done (1 left) 2020-06-12T02:38:31.844 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:31.844 INFO:tasks.rados.rados.0.smithi107.stdout:6950: seq_num 6949 ranges {1925120=786432,2711552=401408} 2020-06-12T02:38:31.844 INFO:tasks.rados.rados.0.smithi107.stdout:6950: writing smithi10729407-15 from 1925120 to 2711552 tid 1 2020-06-12T02:38:31.844 INFO:tasks.rados.rados.0.smithi107.stdout:6950: writing smithi10729407-15 from 2711552 to 3112960 tid 2 2020-06-12T02:38:31.844 INFO:tasks.rados.rados.0.smithi107.stdout:6951: delete oid 4 current snap is 0 2020-06-12T02:38:31.844 INFO:tasks.rados.rados.0.smithi107.stdout:6946: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:31.845 INFO:tasks.rados.rados.0.smithi107.stdout:6946: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:31.845 INFO:tasks.rados.rados.0.smithi107.stdout:6946: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:31.845 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1288 (ObjNum 6945 snap 0 seq_num 6945) dirty exists 2020-06-12T02:38:31.845 INFO:tasks.rados.rados.0.smithi107.stdout:6946: left oid 9 (ObjNum 6945 snap 0 seq_num 6945) 2020-06-12T02:38:31.845 INFO:tasks.rados.rados.0.smithi107.stdout:6946: done (2 left) 2020-06-12T02:38:31.846 INFO:tasks.rados.rados.0.smithi107.stdout:6951: done (1 left) 2020-06-12T02:38:31.846 INFO:tasks.rados.rados.0.smithi107.stdout:6952: delete oid 3 current snap is 0 2020-06-12T02:38:31.846 INFO:tasks.rados.rados.0.smithi107.stdout:6952: done (1 left) 2020-06-12T02:38:31.846 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:31.846 INFO:tasks.rados.rados.0.smithi107.stdout:6953: seq_num 6952 ranges {1777664=434176,2211840=491520,2703360=311296} 2020-06-12T02:38:31.847 INFO:tasks.rados.rados.0.smithi107.stdout:6953: writing smithi10729407-9 from 1777664 to 2211840 tid 1 2020-06-12T02:38:31.847 INFO:tasks.rados.rados.0.smithi107.stdout:6953: writing smithi10729407-9 from 2211840 to 2703360 tid 2 2020-06-12T02:38:31.847 INFO:tasks.rados.rados.0.smithi107.stdout:6953: writing smithi10729407-9 from 2703360 to 3014656 tid 3 2020-06-12T02:38:31.847 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:38:31.847 INFO:tasks.rados.rados.0.smithi107.stdout:6954: seq_num 6953 ranges {0=458752,458752=770048,1228800=32768} 2020-06-12T02:38:31.847 INFO:tasks.rados.rados.0.smithi107.stdout:6954: writing smithi10729407-10 from 0 to 458752 tid 1 2020-06-12T02:38:31.848 INFO:tasks.rados.rados.0.smithi107.stdout:6954: writing smithi10729407-10 from 458752 to 1228800 tid 2 2020-06-12T02:38:31.848 INFO:tasks.rados.rados.0.smithi107.stdout:6954: writing smithi10729407-10 from 1228800 to 1261568 tid 3 2020-06-12T02:38:31.848 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:31.848 INFO:tasks.rados.rados.0.smithi107.stdout:6955: seq_num 6954 ranges {0=475136,475136=655360,1130496=147456} 2020-06-12T02:38:31.848 INFO:tasks.rados.rados.0.smithi107.stdout:6955: writing smithi10729407-17 from 0 to 475136 tid 1 2020-06-12T02:38:31.849 INFO:tasks.rados.rados.0.smithi107.stdout:6955: writing smithi10729407-17 from 475136 to 1130496 tid 2 2020-06-12T02:38:31.849 INFO:tasks.rados.rados.0.smithi107.stdout:6955: writing smithi10729407-17 from 1130496 to 1277952 tid 3 2020-06-12T02:38:31.849 INFO:tasks.rados.rados.0.smithi107.stdout:6956: delete oid 18 current snap is 0 2020-06-12T02:38:31.849 INFO:tasks.rados.rados.0.smithi107.stdout:6950: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:31.849 INFO:tasks.rados.rados.0.smithi107.stdout:6950: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:31.850 INFO:tasks.rados.rados.0.smithi107.stdout:6950: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:31.850 INFO:tasks.rados.rados.0.smithi107.stdout:6950: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:31.850 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1299 (ObjNum 6949 snap 0 seq_num 6949) dirty exists 2020-06-12T02:38:31.850 INFO:tasks.rados.rados.0.smithi107.stdout:6950: left oid 15 (ObjNum 6949 snap 0 seq_num 6949) 2020-06-12T02:38:31.850 INFO:tasks.rados.rados.0.smithi107.stdout:6953: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:31.850 INFO:tasks.rados.rados.0.smithi107.stdout:6953: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:31.851 INFO:tasks.rados.rados.0.smithi107.stdout:6953: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:31.851 INFO:tasks.rados.rados.0.smithi107.stdout:6954: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:31.851 INFO:tasks.rados.rados.0.smithi107.stdout:6953: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:31.851 INFO:tasks.rados.rados.0.smithi107.stdout:6953: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:31.851 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1292 (ObjNum 6952 snap 0 seq_num 6952) dirty exists 2020-06-12T02:38:31.851 INFO:tasks.rados.rados.0.smithi107.stdout:6953: left oid 9 (ObjNum 6952 snap 0 seq_num 6952) 2020-06-12T02:38:31.852 INFO:tasks.rados.rados.0.smithi107.stdout:6955: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:31.852 INFO:tasks.rados.rados.0.smithi107.stdout:6954: finishing write tid 2 to smithi10729407-10 2020-06-12T02:38:31.852 INFO:tasks.rados.rados.0.smithi107.stdout:6954: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:31.852 INFO:tasks.rados.rados.0.smithi107.stdout:6954: finishing write tid 5 to smithi10729407-10 2020-06-12T02:38:31.852 INFO:tasks.rados.rados.0.smithi107.stdout:6954: finishing write tid 6 to smithi10729407-10 2020-06-12T02:38:31.853 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 458 (ObjNum 6953 snap 0 seq_num 6953) dirty exists 2020-06-12T02:38:31.853 INFO:tasks.rados.rados.0.smithi107.stdout:6954: left oid 10 (ObjNum 6953 snap 0 seq_num 6953) 2020-06-12T02:38:31.853 INFO:tasks.rados.rados.0.smithi107.stdout:6950: done (4 left) 2020-06-12T02:38:31.853 INFO:tasks.rados.rados.0.smithi107.stdout:6953: done (3 left) 2020-06-12T02:38:31.853 INFO:tasks.rados.rados.0.smithi107.stdout:6954: done (2 left) 2020-06-12T02:38:31.853 INFO:tasks.rados.rados.0.smithi107.stdout:6956: done (1 left) 2020-06-12T02:38:31.854 INFO:tasks.rados.rados.0.smithi107.stdout:6957: delete oid 1 current snap is 0 2020-06-12T02:38:31.854 INFO:tasks.rados.rados.0.smithi107.stdout:6955: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:31.854 INFO:tasks.rados.rados.0.smithi107.stdout:6955: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:31.854 INFO:tasks.rados.rados.0.smithi107.stdout:6957: done (1 left) 2020-06-12T02:38:31.854 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:38:31.855 INFO:tasks.rados.rados.0.smithi107.stdout:6958: seq_num 6957 ranges {1089536=524288,1613824=761856,2375680=40960} 2020-06-12T02:38:31.855 INFO:tasks.rados.rados.0.smithi107.stdout:6958: writing smithi10729407-2 from 1089536 to 1613824 tid 1 2020-06-12T02:38:31.855 INFO:tasks.rados.rados.0.smithi107.stdout:6958: writing smithi10729407-2 from 1613824 to 2375680 tid 2 2020-06-12T02:38:31.855 INFO:tasks.rados.rados.0.smithi107.stdout:6958: writing smithi10729407-2 from 2375680 to 2416640 tid 3 2020-06-12T02:38:31.855 INFO:tasks.rados.rados.0.smithi107.stdout:6959: delete oid 18 current snap is 0 2020-06-12T02:38:31.856 INFO:tasks.rados.rados.0.smithi107.stdout:6955: finishing write tid 5 to smithi10729407-17 2020-06-12T02:38:31.856 INFO:tasks.rados.rados.0.smithi107.stdout:6955: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:31.857 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1316 (ObjNum 6954 snap 0 seq_num 6954) dirty exists 2020-06-12T02:38:31.857 INFO:tasks.rados.rados.0.smithi107.stdout:6955: left oid 17 (ObjNum 6954 snap 0 seq_num 6954) 2020-06-12T02:38:31.857 INFO:tasks.rados.rados.0.smithi107.stdout:6958: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:31.857 INFO:tasks.rados.rados.0.smithi107.stdout:6955: done (2 left) 2020-06-12T02:38:31.857 INFO:tasks.rados.rados.0.smithi107.stdout:6959: done (1 left) 2020-06-12T02:38:31.858 INFO:tasks.rados.rados.0.smithi107.stdout:6960: delete oid 15 current snap is 0 2020-06-12T02:38:31.858 INFO:tasks.rados.rados.0.smithi107.stdout:6960: done (1 left) 2020-06-12T02:38:31.858 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:31.858 INFO:tasks.rados.rados.0.smithi107.stdout:6961: seq_num 6960 ranges {0=573440,573440=409600,983040=393216} 2020-06-12T02:38:31.858 INFO:tasks.rados.rados.0.smithi107.stdout:6961: writing smithi10729407-6 from 0 to 573440 tid 1 2020-06-12T02:38:31.858 INFO:tasks.rados.rados.0.smithi107.stdout:6961: writing smithi10729407-6 from 573440 to 983040 tid 2 2020-06-12T02:38:31.859 INFO:tasks.rados.rados.0.smithi107.stdout:6961: writing smithi10729407-6 from 983040 to 1376256 tid 3 2020-06-12T02:38:31.859 INFO:tasks.rados.rados.0.smithi107.stdout:6962: delete oid 3 current snap is 0 2020-06-12T02:38:31.859 INFO:tasks.rados.rados.0.smithi107.stdout:6958: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:31.859 INFO:tasks.rados.rados.0.smithi107.stdout:6958: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:31.859 INFO:tasks.rados.rados.0.smithi107.stdout:6958: finishing write tid 5 to smithi10729407-2 2020-06-12T02:38:31.860 INFO:tasks.rados.rados.0.smithi107.stdout:6958: finishing write tid 6 to smithi10729407-2 2020-06-12T02:38:31.860 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 667 (ObjNum 6957 snap 0 seq_num 6957) dirty exists 2020-06-12T02:38:31.860 INFO:tasks.rados.rados.0.smithi107.stdout:6958: left oid 2 (ObjNum 6957 snap 0 seq_num 6957) 2020-06-12T02:38:31.860 INFO:tasks.rados.rados.0.smithi107.stdout:6961: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:31.860 INFO:tasks.rados.rados.0.smithi107.stdout:6958: done (2 left) 2020-06-12T02:38:31.860 INFO:tasks.rados.rados.0.smithi107.stdout:6962: done (1 left) 2020-06-12T02:38:31.861 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:31.861 INFO:tasks.rados.rados.0.smithi107.stdout:6963: seq_num 6962 ranges {0=606208,606208=581632,1187840=221184} 2020-06-12T02:38:31.861 INFO:tasks.rados.rados.0.smithi107.stdout:6963: writing smithi10729407-8 from 0 to 606208 tid 1 2020-06-12T02:38:31.861 INFO:tasks.rados.rados.0.smithi107.stdout:6963: writing smithi10729407-8 from 606208 to 1187840 tid 2 2020-06-12T02:38:31.861 INFO:tasks.rados.rados.0.smithi107.stdout:6963: writing smithi10729407-8 from 1187840 to 1409024 tid 3 2020-06-12T02:38:31.862 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:31.862 INFO:tasks.rados.rados.0.smithi107.stdout:6964: seq_num 6963 ranges {0=622592,622592=466944,1089536=335872} 2020-06-12T02:38:31.862 INFO:tasks.rados.rados.0.smithi107.stdout:6964: writing smithi10729407-15 from 0 to 622592 tid 1 2020-06-12T02:38:31.862 INFO:tasks.rados.rados.0.smithi107.stdout:6964: writing smithi10729407-15 from 622592 to 1089536 tid 2 2020-06-12T02:38:31.862 INFO:tasks.rados.rados.0.smithi107.stdout:6964: writing smithi10729407-15 from 1089536 to 1425408 tid 3 2020-06-12T02:38:31.862 INFO:tasks.rados.rados.0.smithi107.stdout:6965: delete oid 5 current snap is 0 2020-06-12T02:38:31.863 INFO:tasks.rados.rados.0.smithi107.stdout:6961: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:31.863 INFO:tasks.rados.rados.0.smithi107.stdout:6961: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:31.863 INFO:tasks.rados.rados.0.smithi107.stdout:6961: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:31.863 INFO:tasks.rados.rados.0.smithi107.stdout:6963: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:31.863 INFO:tasks.rados.rados.0.smithi107.stdout:6961: finishing write tid 6 to smithi10729407-6 2020-06-12T02:38:31.864 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1259 (ObjNum 6960 snap 0 seq_num 6960) dirty exists 2020-06-12T02:38:31.864 INFO:tasks.rados.rados.0.smithi107.stdout:6961: left oid 6 (ObjNum 6960 snap 0 seq_num 6960) 2020-06-12T02:38:31.864 INFO:tasks.rados.rados.0.smithi107.stdout:6961: done (3 left) 2020-06-12T02:38:31.864 INFO:tasks.rados.rados.0.smithi107.stdout:6965: done (2 left) 2020-06-12T02:38:31.865 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:31.865 INFO:tasks.rados.rados.0.smithi107.stdout:6966: seq_num 6965 ranges {0=655360,655360=688128,1343488=114688} 2020-06-12T02:38:31.865 INFO:tasks.rados.rados.0.smithi107.stdout:6966: writing smithi10729407-18 from 0 to 655360 tid 1 2020-06-12T02:38:31.865 INFO:tasks.rados.rados.0.smithi107.stdout:6966: writing smithi10729407-18 from 655360 to 1343488 tid 2 2020-06-12T02:38:31.865 INFO:tasks.rados.rados.0.smithi107.stdout:6966: writing smithi10729407-18 from 1343488 to 1458176 tid 3 2020-06-12T02:38:31.866 INFO:tasks.rados.rados.0.smithi107.stdout:6967: delete oid 5 current snap is 0 2020-06-12T02:38:31.866 INFO:tasks.rados.rados.0.smithi107.stdout:6963: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:31.866 INFO:tasks.rados.rados.0.smithi107.stdout:6963: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:31.866 INFO:tasks.rados.rados.0.smithi107.stdout:6963: finishing write tid 5 to smithi10729407-8 2020-06-12T02:38:31.866 INFO:tasks.rados.rados.0.smithi107.stdout:6963: finishing write tid 6 to smithi10729407-8 2020-06-12T02:38:31.866 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1304 (ObjNum 6962 snap 0 seq_num 6962) dirty exists 2020-06-12T02:38:31.867 INFO:tasks.rados.rados.0.smithi107.stdout:6963: left oid 8 (ObjNum 6962 snap 0 seq_num 6962) 2020-06-12T02:38:31.867 INFO:tasks.rados.rados.0.smithi107.stdout:6964: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:31.867 INFO:tasks.rados.rados.0.smithi107.stdout:6964: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:31.867 INFO:tasks.rados.rados.0.smithi107.stdout:6963: done (3 left) 2020-06-12T02:38:31.868 INFO:tasks.rados.rados.0.smithi107.stdout:6967: done (2 left) 2020-06-12T02:38:31.868 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:38:31.868 INFO:tasks.rados.rados.0.smithi107.stdout:6968: seq_num 6967 ranges {1376256=688128,2064384=450560,2514944=352256} 2020-06-12T02:38:31.868 INFO:tasks.rados.rados.0.smithi107.stdout:6968: writing smithi10729407-6 from 1376256 to 2064384 tid 1 2020-06-12T02:38:31.868 INFO:tasks.rados.rados.0.smithi107.stdout:6968: writing smithi10729407-6 from 2064384 to 2514944 tid 2 2020-06-12T02:38:31.869 INFO:tasks.rados.rados.0.smithi107.stdout:6968: writing smithi10729407-6 from 2514944 to 2867200 tid 3 2020-06-12T02:38:31.869 INFO:tasks.rados.rados.0.smithi107.stdout:6969: delete oid 19 current snap is 0 2020-06-12T02:38:31.869 INFO:tasks.rados.rados.0.smithi107.stdout:6964: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:31.869 INFO:tasks.rados.rados.0.smithi107.stdout:6964: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:31.869 INFO:tasks.rados.rados.0.smithi107.stdout:6964: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:31.869 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1308 (ObjNum 6963 snap 0 seq_num 6963) dirty exists 2020-06-12T02:38:31.870 INFO:tasks.rados.rados.0.smithi107.stdout:6964: left oid 15 (ObjNum 6963 snap 0 seq_num 6963) 2020-06-12T02:38:31.870 INFO:tasks.rados.rados.0.smithi107.stdout:6966: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:31.870 INFO:tasks.rados.rados.0.smithi107.stdout:6966: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:31.870 INFO:tasks.rados.rados.0.smithi107.stdout:6966: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:31.870 INFO:tasks.rados.rados.0.smithi107.stdout:6966: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:31.871 INFO:tasks.rados.rados.0.smithi107.stdout:6966: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:31.871 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1298 (ObjNum 6965 snap 0 seq_num 6965) dirty exists 2020-06-12T02:38:31.871 INFO:tasks.rados.rados.0.smithi107.stdout:6966: left oid 18 (ObjNum 6965 snap 0 seq_num 6965) 2020-06-12T02:38:31.871 INFO:tasks.rados.rados.0.smithi107.stdout:6964: done (3 left) 2020-06-12T02:38:31.871 INFO:tasks.rados.rados.0.smithi107.stdout:6966: done (2 left) 2020-06-12T02:38:31.872 INFO:tasks.rados.rados.0.smithi107.stdout:6969: done (1 left) 2020-06-12T02:38:31.872 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:31.872 INFO:tasks.rados.rados.0.smithi107.stdout:6970: seq_num 6969 ranges {1425408=720896,2146304=622592,2768896=180224} 2020-06-12T02:38:31.872 INFO:tasks.rados.rados.0.smithi107.stdout:6970: writing smithi10729407-15 from 1425408 to 2146304 tid 1 2020-06-12T02:38:31.872 INFO:tasks.rados.rados.0.smithi107.stdout:6970: writing smithi10729407-15 from 2146304 to 2768896 tid 2 2020-06-12T02:38:31.872 INFO:tasks.rados.rados.0.smithi107.stdout:6970: writing smithi10729407-15 from 2768896 to 2949120 tid 3 2020-06-12T02:38:31.873 INFO:tasks.rados.rados.0.smithi107.stdout:6971: delete oid 1 current snap is 0 2020-06-12T02:38:31.873 INFO:tasks.rados.rados.0.smithi107.stdout:6968: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:31.873 INFO:tasks.rados.rados.0.smithi107.stdout:6968: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:31.873 INFO:tasks.rados.rados.0.smithi107.stdout:6968: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:31.873 INFO:tasks.rados.rados.0.smithi107.stdout:6968: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:31.874 INFO:tasks.rados.rados.0.smithi107.stdout:6968: finishing write tid 6 to smithi10729407-6 2020-06-12T02:38:31.874 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1263 (ObjNum 6967 snap 0 seq_num 6967) dirty exists 2020-06-12T02:38:31.874 INFO:tasks.rados.rados.0.smithi107.stdout:6968: left oid 6 (ObjNum 6967 snap 0 seq_num 6967) 2020-06-12T02:38:31.875 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-06-12T02:38:31.875 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:31.787+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:31.879 INFO:tasks.rados.rados.0.smithi107.stdout:6968: done (2 left) 2020-06-12T02:38:31.879 INFO:tasks.rados.rados.0.smithi107.stdout:6971: done (1 left) 2020-06-12T02:38:31.880 INFO:tasks.rados.rados.0.smithi107.stdout:6972: delete oid 8 current snap is 0 2020-06-12T02:38:31.880 INFO:tasks.rados.rados.0.smithi107.stdout:6970: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:31.880 INFO:tasks.rados.rados.0.smithi107.stdout:6970: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:31.880 INFO:tasks.rados.rados.0.smithi107.stdout:6970: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:31.880 INFO:tasks.rados.rados.0.smithi107.stdout:6970: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:31.881 INFO:tasks.rados.rados.0.smithi107.stdout:6972: done (1 left) 2020-06-12T02:38:31.881 INFO:tasks.rados.rados.0.smithi107.stdout:6973: delete oid 7 current snap is 0 2020-06-12T02:38:31.881 INFO:tasks.rados.rados.0.smithi107.stdout:6970: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:31.881 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1312 (ObjNum 6969 snap 0 seq_num 6969) dirty exists 2020-06-12T02:38:31.881 INFO:tasks.rados.rados.0.smithi107.stdout:6970: left oid 15 (ObjNum 6969 snap 0 seq_num 6969) 2020-06-12T02:38:31.881 INFO:tasks.rados.rados.0.smithi107.stdout:6970: done (1 left) 2020-06-12T02:38:31.882 INFO:tasks.rados.rados.0.smithi107.stdout:6973: done (0 left) 2020-06-12T02:38:31.882 INFO:tasks.rados.rados.0.smithi107.stdout:6974: delete oid 4 current snap is 0 2020-06-12T02:38:31.882 INFO:tasks.rados.rados.0.smithi107.stdout:6974: done (0 left) 2020-06-12T02:38:31.882 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:31.882 INFO:tasks.rados.rados.0.smithi107.stdout:6975: seq_num 6974 ranges {0=409600,409600=491520,901120=622592,1523712=90112} 2020-06-12T02:38:31.883 INFO:tasks.rados.rados.0.smithi107.stdout:6975: writing smithi10729407-8 from 0 to 409600 tid 1 2020-06-12T02:38:31.883 INFO:tasks.rados.rados.0.smithi107.stdout:6975: writing smithi10729407-8 from 409600 to 901120 tid 2 2020-06-12T02:38:31.883 INFO:tasks.rados.rados.0.smithi107.stdout:6975: writing smithi10729407-8 from 901120 to 1523712 tid 3 2020-06-12T02:38:31.883 INFO:tasks.rados.rados.0.smithi107.stdout:6975: writing smithi10729407-8 from 1523712 to 1613824 tid 4 2020-06-12T02:38:31.883 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:31.884 INFO:tasks.rados.rados.0.smithi107.stdout:6976: seq_num 6975 ranges {0=425984,425984=778240,1204224=425984} 2020-06-12T02:38:31.884 INFO:tasks.rados.rados.0.smithi107.stdout:6976: writing smithi10729407-14 from 0 to 425984 tid 1 2020-06-12T02:38:31.884 INFO:tasks.rados.rados.0.smithi107.stdout:6976: writing smithi10729407-14 from 425984 to 1204224 tid 2 2020-06-12T02:38:31.884 INFO:tasks.rados.rados.0.smithi107.stdout:6976: writing smithi10729407-14 from 1204224 to 1630208 tid 3 2020-06-12T02:38:31.884 INFO:tasks.rados.rados.0.smithi107.stdout:6977: delete oid 13 current snap is 0 2020-06-12T02:38:31.884 INFO:tasks.rados.rados.0.smithi107.stdout:6975: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:31.885 INFO:tasks.rados.rados.0.smithi107.stdout:6975: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:31.885 INFO:tasks.rados.rados.0.smithi107.stdout:6975: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:31.885 INFO:tasks.rados.rados.0.smithi107.stdout:6975: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:31.885 INFO:tasks.rados.rados.0.smithi107.stdout:6976: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:31.885 INFO:tasks.rados.rados.0.smithi107.stdout:6975: finishing write tid 6 to smithi10729407-8 2020-06-12T02:38:31.886 INFO:tasks.rados.rados.0.smithi107.stdout:6977: done (2 left) 2020-06-12T02:38:31.886 INFO:tasks.rados.rados.0.smithi107.stdout:6978: delete oid 7 current snap is 0 2020-06-12T02:38:31.886 INFO:tasks.rados.rados.0.smithi107.stdout:6975: finishing write tid 7 to smithi10729407-8 2020-06-12T02:38:31.886 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1318 (ObjNum 6974 snap 0 seq_num 6974) dirty exists 2020-06-12T02:38:31.886 INFO:tasks.rados.rados.0.smithi107.stdout:6975: left oid 8 (ObjNum 6974 snap 0 seq_num 6974) 2020-06-12T02:38:31.886 INFO:tasks.rados.rados.0.smithi107.stdout:6975: done (2 left) 2020-06-12T02:38:31.887 INFO:tasks.rados.rados.0.smithi107.stdout:6978: done (1 left) 2020-06-12T02:38:31.887 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:38:31.887 INFO:tasks.rados.rados.0.smithi107.stdout:6979: seq_num 6978 ranges {1261568=475136,1736704=434176,2170880=606208,2777088=163840} 2020-06-12T02:38:31.887 INFO:tasks.rados.rados.0.smithi107.stdout:6979: writing smithi10729407-10 from 1261568 to 1736704 tid 1 2020-06-12T02:38:31.887 INFO:tasks.rados.rados.0.smithi107.stdout:6979: writing smithi10729407-10 from 1736704 to 2170880 tid 2 2020-06-12T02:38:31.887 INFO:tasks.rados.rados.0.smithi107.stdout:6979: writing smithi10729407-10 from 2170880 to 2777088 tid 3 2020-06-12T02:38:31.888 INFO:tasks.rados.rados.0.smithi107.stdout:6979: writing smithi10729407-10 from 2777088 to 2940928 tid 4 2020-06-12T02:38:31.888 INFO:tasks.rados.rados.0.smithi107.stdout:6980: delete oid 4 current snap is 0 2020-06-12T02:38:31.888 INFO:tasks.rados.rados.0.smithi107.stdout:6976: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:31.888 INFO:tasks.rados.rados.0.smithi107.stdout:6976: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:31.888 INFO:tasks.rados.rados.0.smithi107.stdout:6976: finishing write tid 5 to smithi10729407-14 2020-06-12T02:38:31.889 INFO:tasks.rados.rados.0.smithi107.stdout:6976: finishing write tid 6 to smithi10729407-14 2020-06-12T02:38:31.889 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1218 (ObjNum 6975 snap 0 seq_num 6975) dirty exists 2020-06-12T02:38:31.889 INFO:tasks.rados.rados.0.smithi107.stdout:6976: left oid 14 (ObjNum 6975 snap 0 seq_num 6975) 2020-06-12T02:38:31.889 INFO:tasks.rados.rados.0.smithi107.stdout:6979: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:31.889 INFO:tasks.rados.rados.0.smithi107.stdout:6976: done (2 left) 2020-06-12T02:38:31.889 INFO:tasks.rados.rados.0.smithi107.stdout:6980: done (1 left) 2020-06-12T02:38:31.890 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:31.890 INFO:tasks.rados.rados.0.smithi107.stdout:6981: seq_num 6980 ranges {1277952=507904,1785856=647168,2433024=557056} 2020-06-12T02:38:31.891 INFO:tasks.rados.rados.0.smithi107.stdout:6981: writing smithi10729407-17 from 1277952 to 1785856 tid 1 2020-06-12T02:38:31.891 INFO:tasks.rados.rados.0.smithi107.stdout:6981: writing smithi10729407-17 from 1785856 to 2433024 tid 2 2020-06-12T02:38:31.891 INFO:tasks.rados.rados.0.smithi107.stdout:6981: writing smithi10729407-17 from 2433024 to 2990080 tid 3 2020-06-12T02:38:31.892 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:38:31.892 INFO:tasks.rados.rados.0.smithi107.stdout:6982: seq_num 6981 ranges {385024=524288,909312=532480,1441792=671744} 2020-06-12T02:38:31.893 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:31.889+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:31.894 INFO:tasks.rados.rados.0.smithi107.stdout:6982: writing smithi10729407-11 from 385024 to 909312 tid 1 2020-06-12T02:38:31.897 INFO:tasks.rados.rados.0.smithi107.stdout:6982: writing smithi10729407-11 from 909312 to 1441792 tid 2 2020-06-12T02:38:31.901 INFO:tasks.rados.rados.0.smithi107.stdout:6982: writing smithi10729407-11 from 1441792 to 2113536 tid 3 2020-06-12T02:38:31.902 INFO:tasks.rados.rados.0.smithi107.stdout:6983: delete oid 20 current snap is 0 2020-06-12T02:38:31.902 INFO:tasks.rados.rados.0.smithi107.stdout:6979: finishing write tid 2 to smithi10729407-10 2020-06-12T02:38:31.902 INFO:tasks.rados.rados.0.smithi107.stdout:6979: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:31.902 INFO:tasks.rados.rados.0.smithi107.stdout:6979: finishing write tid 4 to smithi10729407-10 2020-06-12T02:38:31.902 INFO:tasks.rados.rados.0.smithi107.stdout:6979: finishing write tid 6 to smithi10729407-10 2020-06-12T02:38:31.903 INFO:tasks.rados.rados.0.smithi107.stdout:6979: finishing write tid 7 to smithi10729407-10 2020-06-12T02:38:31.903 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 463 (ObjNum 6978 snap 0 seq_num 6978) dirty exists 2020-06-12T02:38:31.903 INFO:tasks.rados.rados.0.smithi107.stdout:6979: left oid 10 (ObjNum 6978 snap 0 seq_num 6978) 2020-06-12T02:38:31.903 INFO:tasks.rados.rados.0.smithi107.stdout:6981: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:31.903 INFO:tasks.rados.rados.0.smithi107.stdout:6981: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:31.903 INFO:tasks.rados.rados.0.smithi107.stdout:6981: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:31.904 INFO:tasks.rados.rados.0.smithi107.stdout:6981: finishing write tid 5 to smithi10729407-17 2020-06-12T02:38:31.904 INFO:tasks.rados.rados.0.smithi107.stdout:6981: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:31.904 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1322 (ObjNum 6980 snap 0 seq_num 6980) dirty exists 2020-06-12T02:38:31.904 INFO:tasks.rados.rados.0.smithi107.stdout:6981: left oid 17 (ObjNum 6980 snap 0 seq_num 6980) 2020-06-12T02:38:31.905 INFO:tasks.rados.rados.0.smithi107.stdout:6979: done (3 left) 2020-06-12T02:38:31.905 INFO:tasks.rados.rados.0.smithi107.stdout:6981: done (2 left) 2020-06-12T02:38:31.905 INFO:tasks.rados.rados.0.smithi107.stdout:6983: done (1 left) 2020-06-12T02:38:31.905 INFO:tasks.rados.rados.0.smithi107.stdout:6984: delete oid 19 current snap is 0 2020-06-12T02:38:31.933 INFO:tasks.rados.rados.0.smithi107.stdout:6984: done (1 left) 2020-06-12T02:38:31.933 INFO:tasks.rados.rados.0.smithi107.stdout:6985: delete oid 4 current snap is 0 2020-06-12T02:38:31.934 INFO:tasks.rados.rados.0.smithi107.stdout:6982: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:31.934 INFO:tasks.rados.rados.0.smithi107.stdout:6985: done (1 left) 2020-06-12T02:38:31.934 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:31.934 INFO:tasks.rados.rados.0.smithi107.stdout:6986: seq_num 6985 ranges {0=589824,589824=466944,1056768=737280} 2020-06-12T02:38:31.937 INFO:tasks.rados.rados.0.smithi107.stdout:6986: writing smithi10729407-16 from 0 to 589824 tid 1 2020-06-12T02:38:31.940 INFO:tasks.rados.rados.0.smithi107.stdout:6986: writing smithi10729407-16 from 589824 to 1056768 tid 2 2020-06-12T02:38:31.945 INFO:tasks.rados.rados.0.smithi107.stdout:6986: writing smithi10729407-16 from 1056768 to 1794048 tid 3 2020-06-12T02:38:31.945 INFO:tasks.rados.rados.0.smithi107.stdout:6987: delete oid 1 current snap is 0 2020-06-12T02:38:31.946 INFO:tasks.rados.rados.0.smithi107.stdout:6982: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:31.946 INFO:tasks.rados.rados.0.smithi107.stdout:6986: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:31.948 INFO:tasks.rados.rados.0.smithi107.stdout:6982: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:31.949 INFO:tasks.rados.rados.0.smithi107.stdout:6982: finishing write tid 5 to smithi10729407-11 2020-06-12T02:38:31.949 INFO:tasks.rados.rados.0.smithi107.stdout:6987: done (2 left) 2020-06-12T02:38:31.949 INFO:tasks.rados.rados.0.smithi107.stdout:6988: delete oid 2 current snap is 0 2020-06-12T02:38:31.951 INFO:tasks.rados.rados.0.smithi107.stdout:6982: finishing write tid 6 to smithi10729407-11 2020-06-12T02:38:31.951 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1222 (ObjNum 6981 snap 0 seq_num 6981) dirty exists 2020-06-12T02:38:31.952 INFO:tasks.rados.rados.0.smithi107.stdout:6982: left oid 11 (ObjNum 6981 snap 0 seq_num 6981) 2020-06-12T02:38:31.953 INFO:tasks.rados.rados.0.smithi107.stdout:6982: done (2 left) 2020-06-12T02:38:31.953 INFO:tasks.rados.rados.0.smithi107.stdout:6988: done (1 left) 2020-06-12T02:38:31.953 INFO:tasks.rados.rados.0.smithi107.stdout:6989: delete oid 6 current snap is 0 2020-06-12T02:38:31.955 INFO:tasks.rados.rados.0.smithi107.stdout:6989: done (1 left) 2020-06-12T02:38:31.956 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:38:31.956 INFO:tasks.rados.rados.0.smithi107.stdout:6990: seq_num 6989 ranges {0=655360,655360=458752,1114112=737280,1851392=8192} 2020-06-12T02:38:31.959 INFO:tasks.rados.rados.0.smithi107.stdout:6990: writing smithi10729407-5 from 0 to 655360 tid 1 2020-06-12T02:38:31.962 INFO:tasks.rados.rados.0.smithi107.stdout:6990: writing smithi10729407-5 from 655360 to 1114112 tid 2 2020-06-12T02:38:31.967 INFO:tasks.rados.rados.0.smithi107.stdout:6990: writing smithi10729407-5 from 1114112 to 1851392 tid 3 2020-06-12T02:38:31.967 INFO:tasks.rados.rados.0.smithi107.stdout:6990: writing smithi10729407-5 from 1851392 to 1859584 tid 4 2020-06-12T02:38:31.967 INFO:tasks.rados.rados.0.smithi107.stdout:6991: delete oid 13 current snap is 0 2020-06-12T02:38:31.968 INFO:tasks.rados.rados.0.smithi107.stdout:6986: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:31.968 INFO:tasks.rados.rados.0.smithi107.stdout:6986: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:31.968 INFO:tasks.rados.rados.0.smithi107.stdout:6986: finishing write tid 5 to smithi10729407-16 2020-06-12T02:38:31.968 INFO:tasks.rados.rados.0.smithi107.stdout:6986: finishing write tid 6 to smithi10729407-16 2020-06-12T02:38:31.968 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1238 (ObjNum 6985 snap 0 seq_num 6985) dirty exists 2020-06-12T02:38:31.969 INFO:tasks.rados.rados.0.smithi107.stdout:6986: left oid 16 (ObjNum 6985 snap 0 seq_num 6985) 2020-06-12T02:38:31.969 INFO:tasks.rados.rados.0.smithi107.stdout:6990: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:31.970 INFO:tasks.rados.rados.0.smithi107.stdout:6986: done (2 left) 2020-06-12T02:38:31.970 INFO:tasks.rados.rados.0.smithi107.stdout:6991: done (1 left) 2020-06-12T02:38:31.970 INFO:tasks.rados.rados.0.smithi107.stdout:6992: delete oid 12 current snap is 0 2020-06-12T02:38:31.973 INFO:tasks.rados.rados.0.smithi107.stdout:6992: done (1 left) 2020-06-12T02:38:31.973 INFO:tasks.rados.rados.0.smithi107.stdout:6993: delete oid 14 current snap is 0 2020-06-12T02:38:31.974 INFO:tasks.rados.rados.0.smithi107.stdout:6990: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:31.975 INFO:tasks.rados.rados.0.smithi107.stdout:6990: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:31.976 INFO:tasks.rados.rados.0.smithi107.stdout:6990: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:31.977 INFO:tasks.rados.rados.0.smithi107.stdout:6990: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:31.977 INFO:tasks.rados.rados.0.smithi107.stdout:6993: done (1 left) 2020-06-12T02:38:31.977 INFO:tasks.rados.rados.0.smithi107.stdout:6994: delete oid 1 current snap is 0 2020-06-12T02:38:31.978 INFO:tasks.rados.rados.0.smithi107.stdout:6990: finishing write tid 7 to smithi10729407-5 2020-06-12T02:38:31.978 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 695 (ObjNum 6989 snap 0 seq_num 6989) dirty exists 2020-06-12T02:38:31.978 INFO:tasks.rados.rados.0.smithi107.stdout:6990: left oid 5 (ObjNum 6989 snap 0 seq_num 6989) 2020-06-12T02:38:31.980 INFO:tasks.rados.rados.0.smithi107.stdout:6990: done (1 left) 2020-06-12T02:38:31.980 INFO:tasks.rados.rados.0.smithi107.stdout:6994: done (0 left) 2020-06-12T02:38:31.981 INFO:tasks.rados.rados.0.smithi107.stdout:6995: delete oid 2 current snap is 0 2020-06-12T02:38:31.982 INFO:tasks.rados.rados.0.smithi107.stdout:6995: done (0 left) 2020-06-12T02:38:31.982 INFO:tasks.rados.rados.0.smithi107.stdout:6996: delete oid 2 current snap is 0 2020-06-12T02:38:31.987 INFO:tasks.rados.rados.0.smithi107.stdout:6996: done (0 left) 2020-06-12T02:38:31.987 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:31.987 INFO:tasks.rados.rados.0.smithi107.stdout:6997: seq_num 6996 ranges {1613824=778240,2392064=499712,2891776=540672,3432448=163840} 2020-06-12T02:38:31.996 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:31.993+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:32.000 INFO:tasks.rados.rados.0.smithi107.stdout:6997: writing smithi10729407-8 from 1613824 to 2392064 tid 1 2020-06-12T02:38:32.003 INFO:tasks.rados.rados.0.smithi107.stdout:6997: writing smithi10729407-8 from 2392064 to 2891776 tid 2 2020-06-12T02:38:32.006 INFO:tasks.rados.rados.0.smithi107.stdout:6997: writing smithi10729407-8 from 2891776 to 3432448 tid 3 2020-06-12T02:38:32.007 INFO:tasks.rados.rados.0.smithi107.stdout:6997: writing smithi10729407-8 from 3432448 to 3596288 tid 4 2020-06-12T02:38:32.007 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:38:32.008 INFO:tasks.rados.rados.0.smithi107.stdout:6998: seq_num 6997 ranges {0=794624,794624=786432,1581056=417792} 2020-06-12T02:38:32.012 INFO:tasks.rados.rados.0.smithi107.stdout:6998: writing smithi10729407-6 from 0 to 794624 tid 1 2020-06-12T02:38:32.016 INFO:tasks.rados.rados.0.smithi107.stdout:6998: writing smithi10729407-6 from 794624 to 1581056 tid 2 2020-06-12T02:38:32.019 INFO:tasks.rados.rados.0.smithi107.stdout:6998: writing smithi10729407-6 from 1581056 to 1998848 tid 3 2020-06-12T02:38:32.019 INFO:tasks.rados.rados.0.smithi107.stdout:6999: delete oid 5 current snap is 0 2020-06-12T02:38:32.020 INFO:tasks.rados.rados.0.smithi107.stdout:6997: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:32.021 INFO:tasks.rados.rados.0.smithi107.stdout:6997: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:32.022 INFO:tasks.rados.rados.0.smithi107.stdout:6997: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:32.022 INFO:tasks.rados.rados.0.smithi107.stdout:6997: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:32.024 INFO:tasks.rados.rados.0.smithi107.stdout:6999: done (2 left) 2020-06-12T02:38:32.024 INFO:tasks.rados.rados.0.smithi107.stdout:7000: delete oid 3 current snap is 0 2020-06-12T02:38:32.028 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:32.028 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:38:32.029 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2020-06-12T02:38:32.029 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:32.029 INFO:tasks.rados.rados.0.smithi107.stdout:7000: done (2 left) 2020-06-12T02:38:32.029 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:32.030 INFO:tasks.rados.rados.0.smithi107.stdout:7001: seq_num 7000 ranges {0=442368,442368=434176,876544=524288,1400832=557056,1957888=90112} 2020-06-12T02:38:32.031 INFO:tasks.rados.rados.0.smithi107.stdout:7001: writing smithi10729407-12 from 0 to 442368 tid 1 2020-06-12T02:38:32.032 INFO:tasks.rados.rados.0.smithi107.stdout:7001: writing smithi10729407-12 from 442368 to 876544 tid 2 2020-06-12T02:38:32.036 INFO:tasks.rados.rados.0.smithi107.stdout:7001: writing smithi10729407-12 from 876544 to 1400832 tid 3 2020-06-12T02:38:32.037 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-06-12T02:38:32.039 INFO:tasks.rados.rados.0.smithi107.stdout:7001: writing smithi10729407-12 from 1400832 to 1957888 tid 4 2020-06-12T02:38:32.040 INFO:tasks.rados.rados.0.smithi107.stdout:7001: writing smithi10729407-12 from 1957888 to 2048000 tid 5 2020-06-12T02:38:32.041 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:32.041 INFO:tasks.rados.rados.0.smithi107.stdout:7002: seq_num 7001 ranges {1794048=458752,2252800=720896,2973696=679936,3653632=204800} 2020-06-12T02:38:32.053 INFO:tasks.rados.rados.0.smithi107.stdout:7002: writing smithi10729407-16 from 1794048 to 2252800 tid 1 2020-06-12T02:38:32.058 INFO:tasks.rados.rados.0.smithi107.stdout:7002: writing smithi10729407-16 from 2252800 to 2973696 tid 2 2020-06-12T02:38:32.062 INFO:tasks.rados.rados.0.smithi107.stdout:7002: writing smithi10729407-16 from 2973696 to 3653632 tid 3 2020-06-12T02:38:32.063 INFO:tasks.rados.rados.0.smithi107.stdout:7002: writing smithi10729407-16 from 3653632 to 3858432 tid 4 2020-06-12T02:38:32.064 INFO:tasks.rados.rados.0.smithi107.stdout:7003: delete oid 10 current snap is 0 2020-06-12T02:38:32.064 INFO:tasks.rados.rados.0.smithi107.stdout:6998: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:32.064 INFO:tasks.rados.rados.0.smithi107.stdout:6997: finishing write tid 6 to smithi10729407-8 2020-06-12T02:38:32.064 INFO:tasks.rados.rados.0.smithi107.stdout:6997: finishing write tid 7 to smithi10729407-8 2020-06-12T02:38:32.064 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1323 (ObjNum 6996 snap 0 seq_num 6996) dirty exists 2020-06-12T02:38:32.065 INFO:tasks.rados.rados.0.smithi107.stdout:6997: left oid 8 (ObjNum 6996 snap 0 seq_num 6996) 2020-06-12T02:38:32.065 INFO:tasks.rados.rados.0.smithi107.stdout:6998: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:32.065 INFO:tasks.rados.rados.0.smithi107.stdout:6998: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:32.065 INFO:tasks.rados.rados.0.smithi107.stdout:6998: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:32.066 INFO:tasks.rados.rados.0.smithi107.stdout:6998: finishing write tid 6 to smithi10729407-6 2020-06-12T02:38:32.066 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1271 (ObjNum 6997 snap 0 seq_num 6997) dirty exists 2020-06-12T02:38:32.066 INFO:tasks.rados.rados.0.smithi107.stdout:6998: left oid 6 (ObjNum 6997 snap 0 seq_num 6997) 2020-06-12T02:38:32.066 INFO:tasks.rados.rados.0.smithi107.stdout:7001: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:32.066 INFO:tasks.rados.rados.0.smithi107.stdout:7001: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:32.067 INFO:tasks.rados.rados.0.smithi107.stdout:7001: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:32.067 INFO:tasks.rados.rados.0.smithi107.stdout:7001: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:32.067 INFO:tasks.rados.rados.0.smithi107.stdout:7001: finishing write tid 5 to smithi10729407-12 2020-06-12T02:38:32.067 INFO:tasks.rados.rados.0.smithi107.stdout:7001: finishing write tid 7 to smithi10729407-12 2020-06-12T02:38:32.067 INFO:tasks.rados.rados.0.smithi107.stdout:7001: finishing write tid 8 to smithi10729407-12 2020-06-12T02:38:32.067 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1286 (ObjNum 7000 snap 0 seq_num 7000) dirty exists 2020-06-12T02:38:32.068 INFO:tasks.rados.rados.0.smithi107.stdout:7001: left oid 12 (ObjNum 7000 snap 0 seq_num 7000) 2020-06-12T02:38:32.068 INFO:tasks.rados.rados.0.smithi107.stdout:7002: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:32.069 INFO:tasks.rados.rados.0.smithi107.stdout:6997: done (4 left) 2020-06-12T02:38:32.069 INFO:tasks.rados.rados.0.smithi107.stdout:6998: done (3 left) 2020-06-12T02:38:32.070 INFO:tasks.rados.rados.0.smithi107.stdout:7001: done (2 left) 2020-06-12T02:38:32.070 INFO:tasks.rados.rados.0.smithi107.stdout:7003: done (1 left) 2020-06-12T02:38:32.070 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:32.070 INFO:tasks.rados.rados.0.smithi107.stdout:7004: seq_num 7003 ranges {0=491520,491520=540672,1032192=696320,1728512=368640} 2020-06-12T02:38:32.071 INFO:tasks.rados.rados.0.smithi107.stdout:7004: writing smithi10729407-19 from 0 to 491520 tid 1 2020-06-12T02:38:32.074 INFO:tasks.rados.rados.0.smithi107.stdout:7004: writing smithi10729407-19 from 491520 to 1032192 tid 2 2020-06-12T02:38:32.088 INFO:tasks.rados.rados.0.smithi107.stdout:7004: writing smithi10729407-19 from 1032192 to 1728512 tid 3 2020-06-12T02:38:32.088 INFO:tasks.rados.rados.0.smithi107.stdout:7004: writing smithi10729407-19 from 1728512 to 2097152 tid 4 2020-06-12T02:38:32.092 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:32.093 INFO:tasks.rados.rados.0.smithi107.stdout:7005: seq_num 7004 ranges {1458176=507904,1966080=417792,2383872=499712,2883584=606208,3489792=81920} 2020-06-12T02:38:32.093 INFO:tasks.rados.rados.0.smithi107.stdout:7005: writing smithi10729407-18 from 1458176 to 1966080 tid 1 2020-06-12T02:38:32.094 INFO:tasks.rados.rados.0.smithi107.stdout:7005: writing smithi10729407-18 from 1966080 to 2383872 tid 2 2020-06-12T02:38:32.098 INFO:tasks.rados.rados.0.smithi107.stdout:7005: writing smithi10729407-18 from 2383872 to 2883584 tid 3 2020-06-12T02:38:32.099 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:32.096+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:32.102 INFO:tasks.rados.rados.0.smithi107.stdout:7005: writing smithi10729407-18 from 2883584 to 3489792 tid 4 2020-06-12T02:38:32.102 INFO:tasks.rados.rados.0.smithi107.stdout:7005: writing smithi10729407-18 from 3489792 to 3571712 tid 5 2020-06-12T02:38:32.102 INFO:tasks.rados.rados.0.smithi107.stdout:7006: delete oid 3 current snap is 0 2020-06-12T02:38:32.102 INFO:tasks.rados.rados.0.smithi107.stdout:7002: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:32.103 INFO:tasks.rados.rados.0.smithi107.stdout:7002: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:32.103 INFO:tasks.rados.rados.0.smithi107.stdout:7002: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:32.103 INFO:tasks.rados.rados.0.smithi107.stdout:7002: finishing write tid 6 to smithi10729407-16 2020-06-12T02:38:32.103 INFO:tasks.rados.rados.0.smithi107.stdout:7002: finishing write tid 7 to smithi10729407-16 2020-06-12T02:38:32.103 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1244 (ObjNum 7001 snap 0 seq_num 7001) dirty exists 2020-06-12T02:38:32.103 INFO:tasks.rados.rados.0.smithi107.stdout:7002: left oid 16 (ObjNum 7001 snap 0 seq_num 7001) 2020-06-12T02:38:32.104 INFO:tasks.rados.rados.0.smithi107.stdout:7004: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:32.104 INFO:tasks.rados.rados.0.smithi107.stdout:7004: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:32.104 INFO:tasks.rados.rados.0.smithi107.stdout:7004: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:32.104 INFO:tasks.rados.rados.0.smithi107.stdout:7004: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:32.104 INFO:tasks.rados.rados.0.smithi107.stdout:7004: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:32.105 INFO:tasks.rados.rados.0.smithi107.stdout:7004: finishing write tid 7 to smithi10729407-19 2020-06-12T02:38:32.105 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1291 (ObjNum 7003 snap 0 seq_num 7003) dirty exists 2020-06-12T02:38:32.105 INFO:tasks.rados.rados.0.smithi107.stdout:7004: left oid 19 (ObjNum 7003 snap 0 seq_num 7003) 2020-06-12T02:38:32.105 INFO:tasks.rados.rados.0.smithi107.stdout:7005: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:32.106 INFO:tasks.rados.rados.0.smithi107.stdout:7002: done (3 left) 2020-06-12T02:38:32.106 INFO:tasks.rados.rados.0.smithi107.stdout:7004: done (2 left) 2020-06-12T02:38:32.106 INFO:tasks.rados.rados.0.smithi107.stdout:7006: done (1 left) 2020-06-12T02:38:32.106 INFO:tasks.rados.rados.0.smithi107.stdout:7007: delete oid 20 current snap is 0 2020-06-12T02:38:32.107 INFO:tasks.rados.rados.0.smithi107.stdout:7007: done (1 left) 2020-06-12T02:38:32.107 INFO:tasks.rados.rados.0.smithi107.stdout:7008: delete oid 3 current snap is 0 2020-06-12T02:38:32.109 INFO:tasks.rados.rados.0.smithi107.stdout:7008: done (1 left) 2020-06-12T02:38:32.110 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:32.110 INFO:tasks.rados.rados.0.smithi107.stdout:7009: seq_num 7008 ranges {2097152=573440,2670592=761856,3432448=483328,3915776=360448} 2020-06-12T02:38:32.124 INFO:tasks.rados.rados.0.smithi107.stdout:7009: writing smithi10729407-19 from 2097152 to 2670592 tid 1 2020-06-12T02:38:32.129 INFO:tasks.rados.rados.0.smithi107.stdout:7009: writing smithi10729407-19 from 2670592 to 3432448 tid 2 2020-06-12T02:38:32.132 INFO:tasks.rados.rados.0.smithi107.stdout:7009: writing smithi10729407-19 from 3432448 to 3915776 tid 3 2020-06-12T02:38:32.135 INFO:tasks.rados.rados.0.smithi107.stdout:7009: writing smithi10729407-19 from 3915776 to 4276224 tid 4 2020-06-12T02:38:32.136 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:32.136 INFO:tasks.rados.rados.0.smithi107.stdout:7010: seq_num 7009 ranges {3014656=589824,3604480=647168,4251648=688128,4939776=270336} 2020-06-12T02:38:32.155 INFO:tasks.rados.rados.0.smithi107.stdout:7010: writing smithi10729407-9 from 3014656 to 3604480 tid 1 2020-06-12T02:38:32.160 INFO:tasks.rados.rados.0.smithi107.stdout:7010: writing smithi10729407-9 from 3604480 to 4251648 tid 2 2020-06-12T02:38:32.164 INFO:tasks.rados.rados.0.smithi107.stdout:7010: writing smithi10729407-9 from 4251648 to 4939776 tid 3 2020-06-12T02:38:32.166 INFO:tasks.rados.rados.0.smithi107.stdout:7010: writing smithi10729407-9 from 4939776 to 5210112 tid 4 2020-06-12T02:38:32.166 INFO:tasks.rados.rados.0.smithi107.stdout:7011: delete oid 6 current snap is 0 2020-06-12T02:38:32.167 INFO:tasks.rados.rados.0.smithi107.stdout:7005: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:32.167 INFO:tasks.rados.rados.0.smithi107.stdout:7005: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:32.167 INFO:tasks.rados.rados.0.smithi107.stdout:7005: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:32.168 INFO:tasks.rados.rados.0.smithi107.stdout:7005: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:32.168 INFO:tasks.rados.rados.0.smithi107.stdout:7005: finishing write tid 7 to smithi10729407-18 2020-06-12T02:38:32.168 INFO:tasks.rados.rados.0.smithi107.stdout:7005: finishing write tid 8 to smithi10729407-18 2020-06-12T02:38:32.169 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1304 (ObjNum 7004 snap 0 seq_num 7004) dirty exists 2020-06-12T02:38:32.169 INFO:tasks.rados.rados.0.smithi107.stdout:7005: left oid 18 (ObjNum 7004 snap 0 seq_num 7004) 2020-06-12T02:38:32.169 INFO:tasks.rados.rados.0.smithi107.stdout:7009: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:32.170 INFO:tasks.rados.rados.0.smithi107.stdout:7009: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:32.170 INFO:tasks.rados.rados.0.smithi107.stdout:7009: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:32.170 INFO:tasks.rados.rados.0.smithi107.stdout:7009: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:32.171 INFO:tasks.rados.rados.0.smithi107.stdout:7009: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:32.171 INFO:tasks.rados.rados.0.smithi107.stdout:7009: finishing write tid 7 to smithi10729407-19 2020-06-12T02:38:32.171 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1296 (ObjNum 7008 snap 0 seq_num 7008) dirty exists 2020-06-12T02:38:32.172 INFO:tasks.rados.rados.0.smithi107.stdout:7009: left oid 19 (ObjNum 7008 snap 0 seq_num 7008) 2020-06-12T02:38:32.172 INFO:tasks.rados.rados.0.smithi107.stdout:7010: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:32.173 INFO:tasks.rados.rados.0.smithi107.stdout:7005: done (3 left) 2020-06-12T02:38:32.173 INFO:tasks.rados.rados.0.smithi107.stdout:7009: done (2 left) 2020-06-12T02:38:32.174 INFO:tasks.rados.rados.0.smithi107.stdout:7011: done (1 left) 2020-06-12T02:38:32.175 INFO:tasks.rados.rados.0.smithi107.stdout:7012: delete oid 8 current snap is 0 2020-06-12T02:38:32.175 INFO:tasks.rados.rados.0.smithi107.stdout:7012: done (1 left) 2020-06-12T02:38:32.175 INFO:tasks.rados.rados.0.smithi107.stdout:7013: delete oid 3 current snap is 0 2020-06-12T02:38:32.176 INFO:tasks.rados.rados.0.smithi107.stdout:7010: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:32.176 INFO:tasks.rados.rados.0.smithi107.stdout:7010: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:32.177 INFO:tasks.rados.rados.0.smithi107.stdout:7010: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:32.178 INFO:tasks.rados.rados.0.smithi107.stdout:7010: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:32.178 INFO:tasks.rados.rados.0.smithi107.stdout:7013: done (1 left) 2020-06-12T02:38:32.178 INFO:tasks.rados.rados.0.smithi107.stdout:7014: delete oid 14 current snap is 0 2020-06-12T02:38:32.180 INFO:tasks.rados.rados.0.smithi107.stdout:7010: finishing write tid 7 to smithi10729407-9 2020-06-12T02:38:32.180 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1309 (ObjNum 7009 snap 0 seq_num 7009) dirty exists 2020-06-12T02:38:32.180 INFO:tasks.rados.rados.0.smithi107.stdout:7010: left oid 9 (ObjNum 7009 snap 0 seq_num 7009) 2020-06-12T02:38:32.181 INFO:tasks.rados.rados.0.smithi107.stdout:7010: done (1 left) 2020-06-12T02:38:32.181 INFO:tasks.rados.rados.0.smithi107.stdout:7014: done (0 left) 2020-06-12T02:38:32.181 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:32.182 INFO:tasks.rados.rados.0.smithi107.stdout:7015: seq_num 7014 ranges {0=679936,679936=516096,1196032=425984,1622016=491520,2113536=172032} 2020-06-12T02:38:32.183 INFO:tasks.rados.rados.0.smithi107.stdout:7015: writing smithi10729407-6 from 0 to 679936 tid 1 2020-06-12T02:38:32.185 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:32.185 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:38:32.186 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2020-06-12T02:38:32.186 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2020-06-12T02:38:32.186 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:32.187 INFO:tasks.rados.rados.0.smithi107.stdout:7015: writing smithi10729407-6 from 679936 to 1196032 tid 2 2020-06-12T02:38:32.189 INFO:tasks.rados.rados.0.smithi107.stdout:7015: writing smithi10729407-6 from 1196032 to 1622016 tid 3 2020-06-12T02:38:32.192 INFO:tasks.rados.rados.0.smithi107.stdout:7015: writing smithi10729407-6 from 1622016 to 2113536 tid 4 2020-06-12T02:38:32.207 INFO:tasks.rados.rados.0.smithi107.stdout:7015: writing smithi10729407-6 from 2113536 to 2285568 tid 5 2020-06-12T02:38:32.208 INFO:tasks.rados.rados.0.smithi107.stdout:7016: delete oid 19 current snap is 0 2020-06-12T02:38:32.208 INFO:tasks.rados.rados.0.smithi107.stdout:7015: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:32.209 INFO:tasks.rados.rados.0.smithi107.stdout:7016: done (1 left) 2020-06-12T02:38:32.209 INFO:tasks.rados.rados.0.smithi107.stdout:7017: delete oid 3 current snap is 0 2020-06-12T02:38:32.209 INFO:tasks.rados.rados.0.smithi107.stdout:7015: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:32.210 INFO:tasks.rados.rados.0.smithi107.stdout:7017: done (1 left) 2020-06-12T02:38:32.210 INFO:tasks.rados.rados.0.smithi107.stdout:7018: delete oid 5 current snap is 0 2020-06-12T02:38:32.210 INFO:tasks.rados.rados.0.smithi107.stdout:7015: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:32.211 INFO:tasks.rados.rados.0.smithi107.stdout:7018: done (1 left) 2020-06-12T02:38:32.211 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:38:32.212 INFO:tasks.rados.rados.0.smithi107.stdout:7019: seq_num 7018 ranges {0=745472,745472=499712,1245184=409600,1654784=548864,2203648=147456} 2020-06-12T02:38:32.213 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-06-12T02:38:32.215 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:32.202+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:32.215 INFO:tasks.rados.rados.0.smithi107.stdout:7019: writing smithi10729407-1 from 0 to 745472 tid 1 2020-06-12T02:38:32.215 INFO:tasks.rados.rados.0.smithi107.stdout:7019: writing smithi10729407-1 from 745472 to 1245184 tid 2 2020-06-12T02:38:32.216 INFO:tasks.rados.rados.0.smithi107.stdout:7019: writing smithi10729407-1 from 1245184 to 1654784 tid 3 2020-06-12T02:38:32.218 INFO:tasks.rados.rados.0.smithi107.stdout:7019: writing smithi10729407-1 from 1654784 to 2203648 tid 4 2020-06-12T02:38:32.219 INFO:tasks.rados.rados.0.smithi107.stdout:7019: writing smithi10729407-1 from 2203648 to 2351104 tid 5 2020-06-12T02:38:32.219 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:32.219 INFO:tasks.rados.rados.0.smithi107.stdout:7020: seq_num 7019 ranges {0=761856,761856=786432,1548288=614400,2162688=204800} 2020-06-12T02:38:32.223 INFO:tasks.rados.rados.0.smithi107.stdout:7020: writing smithi10729407-4 from 0 to 761856 tid 1 2020-06-12T02:38:32.229 INFO:tasks.rados.rados.0.smithi107.stdout:7020: writing smithi10729407-4 from 761856 to 1548288 tid 2 2020-06-12T02:38:32.232 INFO:tasks.rados.rados.0.smithi107.stdout:7020: writing smithi10729407-4 from 1548288 to 2162688 tid 3 2020-06-12T02:38:32.233 INFO:tasks.rados.rados.0.smithi107.stdout:7020: writing smithi10729407-4 from 2162688 to 2367488 tid 4 2020-06-12T02:38:32.234 INFO:tasks.rados.rados.0.smithi107.stdout:7021: delete oid 13 current snap is 0 2020-06-12T02:38:32.234 INFO:tasks.rados.rados.0.smithi107.stdout:7015: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:32.234 INFO:tasks.rados.rados.0.smithi107.stdout:7015: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:32.234 INFO:tasks.rados.rados.0.smithi107.stdout:7015: finishing write tid 7 to smithi10729407-6 2020-06-12T02:38:32.235 INFO:tasks.rados.rados.0.smithi107.stdout:7015: finishing write tid 8 to smithi10729407-6 2020-06-12T02:38:32.235 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1278 (ObjNum 7014 snap 0 seq_num 7014) dirty exists 2020-06-12T02:38:32.235 INFO:tasks.rados.rados.0.smithi107.stdout:7015: left oid 6 (ObjNum 7014 snap 0 seq_num 7014) 2020-06-12T02:38:32.235 INFO:tasks.rados.rados.0.smithi107.stdout:7019: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:32.236 INFO:tasks.rados.rados.0.smithi107.stdout:7019: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:32.236 INFO:tasks.rados.rados.0.smithi107.stdout:7019: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:32.236 INFO:tasks.rados.rados.0.smithi107.stdout:7019: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:32.236 INFO:tasks.rados.rados.0.smithi107.stdout:7019: finishing write tid 5 to smithi10729407-1 2020-06-12T02:38:32.237 INFO:tasks.rados.rados.0.smithi107.stdout:7019: finishing write tid 7 to smithi10729407-1 2020-06-12T02:38:32.237 INFO:tasks.rados.rados.0.smithi107.stdout:7019: finishing write tid 8 to smithi10729407-1 2020-06-12T02:38:32.237 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1284 (ObjNum 7018 snap 0 seq_num 7018) dirty exists 2020-06-12T02:38:32.237 INFO:tasks.rados.rados.0.smithi107.stdout:7019: left oid 1 (ObjNum 7018 snap 0 seq_num 7018) 2020-06-12T02:38:32.238 INFO:tasks.rados.rados.0.smithi107.stdout:7015: done (3 left) 2020-06-12T02:38:32.238 INFO:tasks.rados.rados.0.smithi107.stdout:7019: done (2 left) 2020-06-12T02:38:32.238 INFO:tasks.rados.rados.0.smithi107.stdout:7021: done (1 left) 2020-06-12T02:38:32.239 INFO:tasks.rados.rados.0.smithi107.stdout:7022: delete oid 20 current snap is 0 2020-06-12T02:38:32.241 INFO:tasks.rados.rados.0.smithi107.stdout:7020: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:32.241 INFO:tasks.rados.rados.0.smithi107.stdout:7020: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:32.242 INFO:tasks.rados.rados.0.smithi107.stdout:7022: done (1 left) 2020-06-12T02:38:32.242 INFO:tasks.rados.rados.0.smithi107.stdout:7023: delete oid 15 current snap is 0 2020-06-12T02:38:32.243 INFO:tasks.rados.rados.0.smithi107.stdout:7020: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:32.243 INFO:tasks.rados.rados.0.smithi107.stdout:7020: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:32.244 INFO:tasks.rados.rados.0.smithi107.stdout:7020: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:32.245 INFO:tasks.rados.rados.0.smithi107.stdout:7023: done (1 left) 2020-06-12T02:38:32.245 INFO:tasks.rados.rados.0.smithi107.stdout:7024: delete oid 14 current snap is 0 2020-06-12T02:38:32.249 INFO:tasks.rados.rados.0.smithi107.stdout:7020: finishing write tid 7 to smithi10729407-4 2020-06-12T02:38:32.249 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1328 (ObjNum 7019 snap 0 seq_num 7019) dirty exists 2020-06-12T02:38:32.249 INFO:tasks.rados.rados.0.smithi107.stdout:7020: left oid 4 (ObjNum 7019 snap 0 seq_num 7019) 2020-06-12T02:38:32.250 INFO:tasks.rados.rados.0.smithi107.stdout:7020: done (1 left) 2020-06-12T02:38:32.250 INFO:tasks.rados.rados.0.smithi107.stdout:7024: done (0 left) 2020-06-12T02:38:32.250 INFO:tasks.rados.rados.0.smithi107.stdout:7025: delete oid 12 current snap is 0 2020-06-12T02:38:32.251 INFO:tasks.rados.rados.0.smithi107.stdout:7025: done (0 left) 2020-06-12T02:38:32.251 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:38:32.252 INFO:tasks.rados.rados.0.smithi107.stdout:7026: seq_num 7025 ranges {0=57344} 2020-06-12T02:38:32.252 INFO:tasks.rados.rados.0.smithi107.stdout:7026: writing smithi10729407-7 from 0 to 57344 tid 1 2020-06-12T02:38:32.263 INFO:tasks.rados.rados.0.smithi107.stdout:7027: delete oid 13 current snap is 0 2020-06-12T02:38:32.264 INFO:tasks.rados.rados.0.smithi107.stdout:7026: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:32.265 INFO:tasks.rados.rados.0.smithi107.stdout:7027: done (1 left) 2020-06-12T02:38:32.265 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:38:32.265 INFO:tasks.rados.rados.0.smithi107.stdout:7028: seq_num 7027 ranges {0=90112} 2020-06-12T02:38:32.267 INFO:tasks.rados.rados.0.smithi107.stdout:7028: writing smithi10729407-5 from 0 to 90112 tid 1 2020-06-12T02:38:32.267 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:32.268 INFO:tasks.rados.rados.0.smithi107.stdout:7029: seq_num 7028 ranges {2990080=106496} 2020-06-12T02:38:32.275 INFO:tasks.rados.rados.0.smithi107.stdout:7029: writing smithi10729407-17 from 2990080 to 3096576 tid 1 2020-06-12T02:38:32.275 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:32.275 INFO:tasks.rados.rados.0.smithi107.stdout:7030: seq_num 7029 ranges {0=122880} 2020-06-12T02:38:32.276 INFO:tasks.rados.rados.0.smithi107.stdout:7030: writing smithi10729407-19 from 0 to 122880 tid 1 2020-06-12T02:38:32.276 INFO:tasks.rados.rados.0.smithi107.stdout:7031: delete oid 12 current snap is 0 2020-06-12T02:38:32.277 INFO:tasks.rados.rados.0.smithi107.stdout:7026: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:32.277 INFO:tasks.rados.rados.0.smithi107.stdout:7026: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:32.277 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1300 (ObjNum 7025 snap 0 seq_num 7025) dirty exists 2020-06-12T02:38:32.277 INFO:tasks.rados.rados.0.smithi107.stdout:7026: left oid 7 (ObjNum 7025 snap 0 seq_num 7025) 2020-06-12T02:38:32.277 INFO:tasks.rados.rados.0.smithi107.stdout:7028: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:32.277 INFO:tasks.rados.rados.0.smithi107.stdout:7028: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:32.278 INFO:tasks.rados.rados.0.smithi107.stdout:7028: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:32.278 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 699 (ObjNum 7027 snap 0 seq_num 7027) dirty exists 2020-06-12T02:38:32.278 INFO:tasks.rados.rados.0.smithi107.stdout:7028: left oid 5 (ObjNum 7027 snap 0 seq_num 7027) 2020-06-12T02:38:32.279 INFO:tasks.rados.rados.0.smithi107.stdout:7029: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:32.282 INFO:tasks.rados.rados.0.smithi107.stdout:7030: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:32.283 INFO:tasks.rados.rados.0.smithi107.stdout:7029: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:32.283 INFO:tasks.rados.rados.0.smithi107.stdout:7030: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:32.284 INFO:tasks.rados.rados.0.smithi107.stdout:7026: done (4 left) 2020-06-12T02:38:32.285 INFO:tasks.rados.rados.0.smithi107.stdout:7028: done (3 left) 2020-06-12T02:38:32.285 INFO:tasks.rados.rados.0.smithi107.stdout:7031: done (2 left) 2020-06-12T02:38:32.285 INFO:tasks.rados.rados.0.smithi107.stdout:7032: delete oid 13 current snap is 0 2020-06-12T02:38:32.286 INFO:tasks.rados.rados.0.smithi107.stdout:7029: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:32.286 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1330 (ObjNum 7028 snap 0 seq_num 7028) dirty exists 2020-06-12T02:38:32.286 INFO:tasks.rados.rados.0.smithi107.stdout:7029: left oid 17 (ObjNum 7028 snap 0 seq_num 7028) 2020-06-12T02:38:32.287 INFO:tasks.rados.rados.0.smithi107.stdout:7030: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:32.287 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1300 (ObjNum 7029 snap 0 seq_num 7029) dirty exists 2020-06-12T02:38:32.287 INFO:tasks.rados.rados.0.smithi107.stdout:7030: left oid 19 (ObjNum 7029 snap 0 seq_num 7029) 2020-06-12T02:38:32.289 INFO:tasks.rados.rados.0.smithi107.stdout:7029: done (2 left) 2020-06-12T02:38:32.289 INFO:tasks.rados.rados.0.smithi107.stdout:7030: done (1 left) 2020-06-12T02:38:32.289 INFO:tasks.rados.rados.0.smithi107.stdout:7032: done (0 left) 2020-06-12T02:38:32.290 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:32.290 INFO:tasks.rados.rados.0.smithi107.stdout:7033: seq_num 7032 ranges {3571712=180224} 2020-06-12T02:38:32.307 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:32.304+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:32.309 INFO:tasks.rados.rados.0.smithi107.stdout:7033: writing smithi10729407-18 from 3571712 to 3751936 tid 1 2020-06-12T02:38:32.309 INFO:tasks.rados.rados.0.smithi107.stdout:7034: delete oid 11 current snap is 0 2020-06-12T02:38:32.312 INFO:tasks.rados.rados.0.smithi107.stdout:7034: done (1 left) 2020-06-12T02:38:32.313 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:32.313 INFO:tasks.rados.rados.0.smithi107.stdout:7035: seq_num 7034 ranges {0=212992} 2020-06-12T02:38:32.314 INFO:tasks.rados.rados.0.smithi107.stdout:7035: writing smithi10729407-3 from 0 to 212992 tid 1 2020-06-12T02:38:32.315 INFO:tasks.rados.rados.0.smithi107.stdout:7036: delete oid 10 current snap is 0 2020-06-12T02:38:32.315 INFO:tasks.rados.rados.0.smithi107.stdout:7033: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:32.317 INFO:tasks.rados.rados.0.smithi107.stdout:7036: done (2 left) 2020-06-12T02:38:32.318 INFO:tasks.rados.rados.0.smithi107.stdout:7037: delete oid 19 current snap is 0 2020-06-12T02:38:32.319 INFO:tasks.rados.rados.0.smithi107.stdout:7033: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:32.320 INFO:tasks.rados.rados.0.smithi107.stdout:7035: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:32.320 INFO:tasks.rados.rados.0.smithi107.stdout:7037: done (2 left) 2020-06-12T02:38:32.321 INFO:tasks.rados.rados.0.smithi107.stdout:7038: delete oid 4 current snap is 0 2020-06-12T02:38:32.322 INFO:tasks.rados.rados.0.smithi107.stdout:7033: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:32.322 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1311 (ObjNum 7032 snap 0 seq_num 7032) dirty exists 2020-06-12T02:38:32.322 INFO:tasks.rados.rados.0.smithi107.stdout:7033: left oid 18 (ObjNum 7032 snap 0 seq_num 7032) 2020-06-12T02:38:32.323 INFO:tasks.rados.rados.0.smithi107.stdout:7035: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:32.325 INFO:tasks.rados.rados.0.smithi107.stdout:7035: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:32.325 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1250 (ObjNum 7034 snap 0 seq_num 7034) dirty exists 2020-06-12T02:38:32.325 INFO:tasks.rados.rados.0.smithi107.stdout:7035: left oid 3 (ObjNum 7034 snap 0 seq_num 7034) 2020-06-12T02:38:32.326 INFO:tasks.rados.rados.0.smithi107.stdout:7033: done (2 left) 2020-06-12T02:38:32.326 INFO:tasks.rados.rados.0.smithi107.stdout:7035: done (1 left) 2020-06-12T02:38:32.327 INFO:tasks.rados.rados.0.smithi107.stdout:7038: done (0 left) 2020-06-12T02:38:32.327 INFO:tasks.rados.rados.0.smithi107.stdout:7039: delete oid 12 current snap is 0 2020-06-12T02:38:32.329 INFO:tasks.rados.rados.0.smithi107.stdout:7039: done (0 left) 2020-06-12T02:38:32.330 INFO:tasks.rados.rados.0.smithi107.stdout:7040: delete oid 14 current snap is 0 2020-06-12T02:38:32.331 INFO:tasks.rados.rados.0.smithi107.stdout:7040: done (0 left) 2020-06-12T02:38:32.331 INFO:tasks.rados.rados.0.smithi107.stdout:7041: delete oid 12 current snap is 0 2020-06-12T02:38:32.333 INFO:tasks.rados.rados.0.smithi107.stdout:7041: done (0 left) 2020-06-12T02:38:32.334 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:32.334 INFO:tasks.rados.rados.0.smithi107.stdout:7042: seq_num 7041 ranges {3858432=327680} 2020-06-12T02:38:32.352 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:32.352 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2020-06-12T02:38:32.352 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2020-06-12T02:38:32.352 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2020-06-12T02:38:32.353 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.353 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.9s1 25/21 ECSubWrite(tid=615, reqid=client.4512.0:1285, at_version=25'85, trim_to=0'0, roll_forward_to=25'84))", 2020-06-12T02:38:32.353 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.117566+0000", 2020-06-12T02:38:32.353 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 59.230997834, 2020-06-12T02:38:32.353 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010635947999999999, 2020-06-12T02:38:32.354 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:32.354 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:32.354 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:32.354 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.354 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:32.355 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.117566+0000", 2020-06-12T02:38:32.355 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:32.355 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.355 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.355 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:32.355 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.117566+0000", 2020-06-12T02:38:32.356 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000451781 2020-06-12T02:38:32.356 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.356 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.356 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:32.356 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.118017+0000", 2020-06-12T02:38:32.357 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9958999999999999e-05 2020-06-12T02:38:32.357 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.357 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.357 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:32.357 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.118047+0000", 2020-06-12T02:38:32.358 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.694e-05 2020-06-12T02:38:32.358 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.358 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.358 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:32.358 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.118064+0000", 2020-06-12T02:38:32.358 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7629999999999995e-06 2020-06-12T02:38:32.359 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.360 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.360 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:32.360 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.118074+0000", 2020-06-12T02:38:32.361 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1209000000000003e-05 2020-06-12T02:38:32.361 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.361 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.361 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:32.361 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.118135+0000", 2020-06-12T02:38:32.362 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6164999999999999e-05 2020-06-12T02:38:32.362 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.362 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.362 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:32.362 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.118151+0000", 2020-06-12T02:38:32.363 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0099907599999999996 2020-06-12T02:38:32.363 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.363 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.363 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:32.363 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.128142+0000", 2020-06-12T02:38:32.363 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9370999999999998e-05 2020-06-12T02:38:32.364 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.364 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.364 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:32.364 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.128202+0000", 2020-06-12T02:38:32.364 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010635947999999999 2020-06-12T02:38:32.365 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.365 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:32.365 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.365 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.365 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.365 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s2 26/21 ECSubWrite(tid=408, reqid=client.4512.0:1572, at_version=26'99, trim_to=0'0, roll_forward_to=26'97))", 2020-06-12T02:38:32.366 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.887897+0000", 2020-06-12T02:38:32.366 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 58.460666473000003, 2020-06-12T02:38:32.366 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0098940590000000002, 2020-06-12T02:38:32.366 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:32.366 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:32.367 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:32.367 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.367 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:32.367 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.887897+0000", 2020-06-12T02:38:32.367 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:32.367 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.368 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.368 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:32.368 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.887897+0000", 2020-06-12T02:38:32.368 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052976499999999997 2020-06-12T02:38:32.368 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.369 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.369 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:32.369 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888427+0000", 2020-06-12T02:38:32.369 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1276000000000001e-05 2020-06-12T02:38:32.370 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.370 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.370 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:32.371 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888448+0000", 2020-06-12T02:38:32.371 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7121000000000001e-05 2020-06-12T02:38:32.371 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.371 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.371 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:32.372 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888465+0000", 2020-06-12T02:38:32.372 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0490000000000004e-06 2020-06-12T02:38:32.372 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.372 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.372 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:32.373 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888473+0000", 2020-06-12T02:38:32.373 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6770999999999997e-05 2020-06-12T02:38:32.373 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.373 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.373 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:32.373 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888510+0000", 2020-06-12T02:38:32.374 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7649999999999993e-06 2020-06-12T02:38:32.374 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.374 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.374 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:32.374 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.888518+0000", 2020-06-12T02:38:32.375 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0092201669999999996 2020-06-12T02:38:32.375 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.375 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.375 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:32.375 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.897738+0000", 2020-06-12T02:38:32.376 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3145e-05 2020-06-12T02:38:32.376 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.376 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.376 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:32.377 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.897791+0000", 2020-06-12T02:38:32.377 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0098940590000000002 2020-06-12T02:38:32.377 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.377 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:32.377 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.378 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.378 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.378 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s2 26/21 ECSubWrite(tid=409, reqid=client.4512.0:1573, at_version=26'100, trim_to=0'0, roll_forward_to=26'98))", 2020-06-12T02:38:32.378 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.892374+0000", 2020-06-12T02:38:32.378 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 58.456189471999998, 2020-06-12T02:38:32.378 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014071322000000001, 2020-06-12T02:38:32.379 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:32.379 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:32.379 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:32.379 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.380 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:32.380 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.892374+0000", 2020-06-12T02:38:32.381 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:32.381 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.381 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.381 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:32.382 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.892374+0000", 2020-06-12T02:38:32.382 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047864199999999999 2020-06-12T02:38:32.382 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.382 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.382 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:32.383 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.892853+0000", 2020-06-12T02:38:32.383 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5139000000000001e-05 2020-06-12T02:38:32.385 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.385 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.385 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:32.385 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.892878+0000", 2020-06-12T02:38:32.385 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1543999999999999e-05 2020-06-12T02:38:32.386 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.386 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.386 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:32.386 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.892899+0000", 2020-06-12T02:38:32.386 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8989999999999996e-06 2020-06-12T02:38:32.386 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.387 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.387 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:32.387 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.892909+0000", 2020-06-12T02:38:32.387 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00023009199999999999 2020-06-12T02:38:32.387 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.388 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.388 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:32.388 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.893139+0000", 2020-06-12T02:38:32.388 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0044000000000001e-05 2020-06-12T02:38:32.388 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.389 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.389 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:32.389 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.893149+0000", 2020-06-12T02:38:32.389 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013233405 2020-06-12T02:38:32.389 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.389 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.390 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:32.390 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.906383+0000", 2020-06-12T02:38:32.390 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2557000000000004e-05 2020-06-12T02:38:32.391 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.391 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.391 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:32.391 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.906445+0000", 2020-06-12T02:38:32.392 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014071322000000001 2020-06-12T02:38:32.392 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.394 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:32.394 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.394 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.395 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.395 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s2 26/21 ECSubWrite(tid=410, reqid=client.4512.0:1574, at_version=26'101, trim_to=0'0, roll_forward_to=26'98))", 2020-06-12T02:38:32.395 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.895487+0000", 2020-06-12T02:38:32.396 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 58.453076676999999, 2020-06-12T02:38:32.396 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012904807000000001, 2020-06-12T02:38:32.396 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:32.397 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:32.397 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:32.397 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.397 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:32.397 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895487+0000", 2020-06-12T02:38:32.399 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:32.399 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.400 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.400 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:32.400 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895487+0000", 2020-06-12T02:38:32.401 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8478000000000002e-05 2020-06-12T02:38:32.402 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.402 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.403 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:32.403 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895575+0000", 2020-06-12T02:38:32.403 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8672e-05 2020-06-12T02:38:32.404 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.404 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.404 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:32.404 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895594+0000", 2020-06-12T02:38:32.405 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8028000000000001e-05 2020-06-12T02:38:32.405 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.405 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.405 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:32.406 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895612+0000", 2020-06-12T02:38:32.406 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2689999999999999e-06 2020-06-12T02:38:32.406 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.406 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.407 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:32.407 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895620+0000", 2020-06-12T02:38:32.407 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0038940289999999998 2020-06-12T02:38:32.407 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.408 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.408 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:32.409 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.899514+0000", 2020-06-12T02:38:32.409 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4370000000000001e-06 2020-06-12T02:38:32.409 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.410 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.411 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:32.411 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.899522+0000", 2020-06-12T02:38:32.411 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0088184019999999995 2020-06-12T02:38:32.412 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.412 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.412 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:32.412 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.908340+0000", 2020-06-12T02:38:32.413 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1492000000000001e-05 2020-06-12T02:38:32.413 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.413 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.413 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:32.414 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.908392+0000", 2020-06-12T02:38:32.414 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012904807000000001 2020-06-12T02:38:32.414 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.415 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:32.415 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.415 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.415 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.416 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 27/21 ECSubWrite(tid=170, reqid=client.4512.0:2721, at_version=27'110, trim_to=0'0, roll_forward_to=27'107))", 2020-06-12T02:38:32.417 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:36.654148+0000", 2020-06-12T02:38:32.417 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 55.694415343999999, 2020-06-12T02:38:32.417 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0096132549999999994, 2020-06-12T02:38:32.417 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:32.418 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:32.419 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:32.419 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.420 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:32.420 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.654148+0000", 2020-06-12T02:38:32.420 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:32.420 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.421 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.421 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:32.421 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.654148+0000", 2020-06-12T02:38:32.422 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00027898700000000002 2020-06-12T02:38:32.422 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.422 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.422 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:32.423 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.654427+0000", 2020-06-12T02:38:32.423 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.7942000000000001e-05 2020-06-12T02:38:32.423 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.424 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.424 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:32.424 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.654455+0000", 2020-06-12T02:38:32.424 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8269000000000001e-05 2020-06-12T02:38:32.424 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.425 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.425 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:32.425 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.654473+0000", 2020-06-12T02:38:32.426 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0450000000000003e-06 2020-06-12T02:38:32.426 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.427 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.427 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:32.428 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.654480+0000", 2020-06-12T02:38:32.428 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013121000000000001 2020-06-12T02:38:32.430 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.430 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.430 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:32.431 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.654612+0000", 2020-06-12T02:38:32.431 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.167e-05 2020-06-12T02:38:32.432 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.432 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.432 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:32.433 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.654623+0000", 2020-06-12T02:38:32.433 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090820680000000004 2020-06-12T02:38:32.433 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.433 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.434 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:32.434 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.663705+0000", 2020-06-12T02:38:32.434 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6063999999999997e-05 2020-06-12T02:38:32.434 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.435 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.435 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:32.435 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.663761+0000", 2020-06-12T02:38:32.435 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0096132549999999994 2020-06-12T02:38:32.436 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.440 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:32.441 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.441 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.441 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.442 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 27/21 ECSubWrite(tid=179, reqid=client.4512.0:2781, at_version=0'0, trim_to=0'0, roll_forward_to=27'115))", 2020-06-12T02:38:32.444 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:36.810022+0000", 2020-06-12T02:38:32.444 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 55.538540937, 2020-06-12T02:38:32.445 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093088630000000006, 2020-06-12T02:38:32.445 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:32.445 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:32.445 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:32.446 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.446 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:32.446 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.810022+0000", 2020-06-12T02:38:32.447 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:32.447 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.448 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.449 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:32.450 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.810022+0000", 2020-06-12T02:38:32.450 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0468000000000004e-05 2020-06-12T02:38:32.451 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.451 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.451 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:32.451 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.810093+0000", 2020-06-12T02:38:32.452 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6742999999999999e-05 2020-06-12T02:38:32.452 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.452 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.452 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:32.453 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.810110+0000", 2020-06-12T02:38:32.453 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8359000000000002e-05 2020-06-12T02:38:32.453 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.454 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.454 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:32.454 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.810128+0000", 2020-06-12T02:38:32.454 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8390000000000007e-06 2020-06-12T02:38:32.455 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.455 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.455 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:32.455 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.810136+0000", 2020-06-12T02:38:32.455 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004899558 2020-06-12T02:38:32.456 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.456 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.456 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:32.456 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.815035+0000", 2020-06-12T02:38:32.457 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7008e-05 2020-06-12T02:38:32.457 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.457 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.457 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:32.458 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.815052+0000", 2020-06-12T02:38:32.458 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0042246180000000003 2020-06-12T02:38:32.458 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.458 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.459 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:32.459 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.819277+0000", 2020-06-12T02:38:32.459 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.427e-05 2020-06-12T02:38:32.459 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.460 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.460 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:32.460 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.819331+0000", 2020-06-12T02:38:32.461 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093088630000000006 2020-06-12T02:38:32.461 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.461 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:32.462 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.462 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.462 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.464 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 27/21 ECSubWrite(tid=180, reqid=client.4512.0:2782, at_version=27'116, trim_to=0'0, roll_forward_to=27'115))", 2020-06-12T02:38:32.465 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:36.811166+0000", 2020-06-12T02:38:32.465 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 55.537396972000003, 2020-06-12T02:38:32.465 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013435703, 2020-06-12T02:38:32.466 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:32.466 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:32.466 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:32.467 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.467 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:32.467 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.811166+0000", 2020-06-12T02:38:32.468 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:32.468 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.468 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.469 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:32.469 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.811166+0000", 2020-06-12T02:38:32.469 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015779300000000001 2020-06-12T02:38:32.470 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.470 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.470 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:32.471 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.811324+0000", 2020-06-12T02:38:32.471 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8149000000000001e-05 2020-06-12T02:38:32.472 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.472 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.472 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:32.473 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.811352+0000", 2020-06-12T02:38:32.473 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6880000000000001e-05 2020-06-12T02:38:32.474 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.475 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.475 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:32.476 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.811369+0000", 2020-06-12T02:38:32.476 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1099999999999999e-06 2020-06-12T02:38:32.477 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.477 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.477 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:32.478 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.811375+0000", 2020-06-12T02:38:32.479 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0038906349999999999 2020-06-12T02:38:32.479 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.479 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.480 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:32.480 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.815266+0000", 2020-06-12T02:38:32.480 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0536e-05 2020-06-12T02:38:32.481 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.481 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.481 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:32.482 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.815277+0000", 2020-06-12T02:38:32.482 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0092955610000000008 2020-06-12T02:38:32.482 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.483 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.483 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:32.483 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.824572+0000", 2020-06-12T02:38:32.484 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0039000000000002e-05 2020-06-12T02:38:32.484 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.484 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.485 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:32.485 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.824602+0000", 2020-06-12T02:38:32.485 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013435703 2020-06-12T02:38:32.486 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.486 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:32.486 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.487 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.487 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.487 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 27/21 ECSubWrite(tid=222, reqid=client.4512.0:3331, at_version=27'144, trim_to=0'0, roll_forward_to=27'141))", 2020-06-12T02:38:32.488 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:38.609919+0000", 2020-06-12T02:38:32.488 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 53.738644006999998, 2020-06-12T02:38:32.488 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0096801079999999998, 2020-06-12T02:38:32.489 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:32.489 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:32.489 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:32.490 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.490 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:32.490 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.609919+0000", 2020-06-12T02:38:32.491 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:32.491 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.491 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.492 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:32.492 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.609919+0000", 2020-06-12T02:38:32.492 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00024651799999999999 2020-06-12T02:38:32.493 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.493 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.493 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:32.494 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.610166+0000", 2020-06-12T02:38:32.494 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5100999999999999e-05 2020-06-12T02:38:32.494 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.495 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.495 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:32.495 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.610191+0000", 2020-06-12T02:38:32.498 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3227e-05 2020-06-12T02:38:32.498 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.498 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.499 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:32.499 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.610214+0000", 2020-06-12T02:38:32.499 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8339999999999999e-06 2020-06-12T02:38:32.500 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.500 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.500 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:32.501 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.610222+0000", 2020-06-12T02:38:32.501 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013659169999999999 2020-06-12T02:38:32.501 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.502 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.502 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:32.502 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.611588+0000", 2020-06-12T02:38:32.503 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2160999999999999e-05 2020-06-12T02:38:32.503 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.504 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.504 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:32.504 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.611600+0000", 2020-06-12T02:38:32.505 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.007943795 2020-06-12T02:38:32.505 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.505 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.506 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:32.506 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.619544+0000", 2020-06-12T02:38:32.507 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5554999999999999e-05 2020-06-12T02:38:32.507 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.507 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.507 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:32.508 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.619600+0000", 2020-06-12T02:38:32.508 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0096801079999999998 2020-06-12T02:38:32.508 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.509 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:32.509 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.509 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.510 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.510 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 27/21 ECSubWrite(tid=226, reqid=client.4512.0:3334, at_version=0'0, trim_to=0'0, roll_forward_to=27'147))", 2020-06-12T02:38:32.510 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:38.627654+0000", 2020-06-12T02:38:32.511 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 53.720909220000003, 2020-06-12T02:38:32.511 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01294059, 2020-06-12T02:38:32.511 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:32.512 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:32.512 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:32.512 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.513 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:32.513 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.627654+0000", 2020-06-12T02:38:32.513 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:32.514 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.514 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.514 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:32.515 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.627654+0000", 2020-06-12T02:38:32.515 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6134000000000003e-05 2020-06-12T02:38:32.515 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.515 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.516 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:32.516 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.627690+0000", 2020-06-12T02:38:32.516 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1439999999999992e-06 2020-06-12T02:38:32.517 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.517 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.517 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:32.518 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.627699+0000", 2020-06-12T02:38:32.518 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.59e-05 2020-06-12T02:38:32.519 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.519 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.519 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:32.519 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.627715+0000", 2020-06-12T02:38:32.520 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.602e-06 2020-06-12T02:38:32.520 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.520 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.521 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:32.521 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.627724+0000", 2020-06-12T02:38:32.521 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0059999999999998e-05 2020-06-12T02:38:32.522 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.522 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.522 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:32.522 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.627784+0000", 2020-06-12T02:38:32.522 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1885999999999999e-05 2020-06-12T02:38:32.522 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.523 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.523 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:32.523 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.627796+0000", 2020-06-12T02:38:32.523 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012765789 2020-06-12T02:38:32.523 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.524 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.524 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:32.524 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.640562+0000", 2020-06-12T02:38:32.524 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3074999999999998e-05 2020-06-12T02:38:32.524 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.524 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.525 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:32.525 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.640595+0000", 2020-06-12T02:38:32.525 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01294059 2020-06-12T02:38:32.525 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.525 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:32.526 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.526 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.526 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.526 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s2 30/29 ECSubWrite(tid=1889, reqid=client.4512.0:4661, at_version=30'407, trim_to=0'0, roll_forward_to=30'404))", 2020-06-12T02:38:32.526 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:42.957353+0000", 2020-06-12T02:38:32.527 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 49.391210180999998, 2020-06-12T02:38:32.527 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0095676370000000004, 2020-06-12T02:38:32.527 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:32.527 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:32.527 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:32.527 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.528 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:32.528 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.957353+0000", 2020-06-12T02:38:32.528 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:32.528 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.528 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.529 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:32.529 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.957353+0000", 2020-06-12T02:38:32.529 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8380000000000002e-05 2020-06-12T02:38:32.529 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.529 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.529 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:32.530 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.957392+0000", 2020-06-12T02:38:32.530 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4823e-05 2020-06-12T02:38:32.530 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.530 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.530 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:32.531 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.957406+0000", 2020-06-12T02:38:32.531 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8370000000000003e-06 2020-06-12T02:38:32.531 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.531 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.531 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:32.531 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.957415+0000", 2020-06-12T02:38:32.532 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6629999999999999e-06 2020-06-12T02:38:32.532 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.532 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.532 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:32.532 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.957419+0000", 2020-06-12T02:38:32.533 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002437093 2020-06-12T02:38:32.533 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.533 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.533 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:32.533 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.959856+0000", 2020-06-12T02:38:32.533 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7690000000000002e-06 2020-06-12T02:38:32.534 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.534 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.534 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:32.534 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.959865+0000", 2020-06-12T02:38:32.534 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0070048059999999997 2020-06-12T02:38:32.535 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.535 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.535 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:32.535 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.966870+0000", 2020-06-12T02:38:32.536 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1266e-05 2020-06-12T02:38:32.536 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.536 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.536 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:32.536 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:42.966921+0000", 2020-06-12T02:38:32.537 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0095676370000000004 2020-06-12T02:38:32.537 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.537 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:32.537 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.537 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.537 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.538 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 49/21 ECSubWrite(tid=652, reqid=client.4512.0:8809, at_version=49'402, trim_to=0'0, roll_forward_to=36'401))", 2020-06-12T02:38:32.538 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.708711+0000", 2020-06-12T02:38:32.538 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 20.639852657999999, 2020-06-12T02:38:32.538 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011533180000000001, 2020-06-12T02:38:32.538 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:32.539 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:32.539 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:32.539 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.539 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:32.539 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.708711+0000", 2020-06-12T02:38:32.539 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:32.540 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.540 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.540 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:32.540 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.708711+0000", 2020-06-12T02:38:32.540 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00023433800000000001 2020-06-12T02:38:32.541 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.541 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.541 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:32.541 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.708945+0000", 2020-06-12T02:38:32.541 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5338999999999999e-05 2020-06-12T02:38:32.542 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.542 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.542 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:32.542 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.708970+0000", 2020-06-12T02:38:32.542 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7054e-05 2020-06-12T02:38:32.542 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.543 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.543 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:32.543 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.708987+0000", 2020-06-12T02:38:32.543 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8599999999999993e-06 2020-06-12T02:38:32.543 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.543 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.544 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:32.544 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.708995+0000", 2020-06-12T02:38:32.544 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3531e-05 2020-06-12T02:38:32.544 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.544 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.545 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:32.545 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.709049+0000", 2020-06-12T02:38:32.545 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2737e-05 2020-06-12T02:38:32.545 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.545 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.546 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:32.547 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.709062+0000", 2020-06-12T02:38:32.547 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011127211999999999 2020-06-12T02:38:32.547 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.547 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.547 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:32.548 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.720189+0000", 2020-06-12T02:38:32.548 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5108999999999997e-05 2020-06-12T02:38:32.548 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.548 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.548 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:32.549 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.720244+0000", 2020-06-12T02:38:32.549 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011533180000000001 2020-06-12T02:38:32.549 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.549 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:32.549 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.550 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.550 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.550 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 49/21 ECSubWrite(tid=663, reqid=client.4512.0:8855, at_version=49'409, trim_to=0'0, roll_forward_to=49'407))", 2020-06-12T02:38:32.550 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.866635+0000", 2020-06-12T02:38:32.550 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 20.481928331999999, 2020-06-12T02:38:32.551 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01099082, 2020-06-12T02:38:32.551 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:32.551 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:32.551 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:32.551 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.551 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:32.552 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.866635+0000", 2020-06-12T02:38:32.552 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:32.552 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.552 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.552 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:32.553 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.866635+0000", 2020-06-12T02:38:32.553 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000193139 2020-06-12T02:38:32.553 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.553 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.553 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:32.554 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.866828+0000", 2020-06-12T02:38:32.554 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7207999999999999e-05 2020-06-12T02:38:32.554 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.554 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.554 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:32.554 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.866845+0000", 2020-06-12T02:38:32.555 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4484999999999999e-05 2020-06-12T02:38:32.555 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.555 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.555 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:32.555 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.866860+0000", 2020-06-12T02:38:32.556 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6740000000000002e-06 2020-06-12T02:38:32.556 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.556 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.556 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:32.556 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.866866+0000", 2020-06-12T02:38:32.556 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8099999999999997e-05 2020-06-12T02:38:32.557 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.557 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.557 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:32.557 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.866914+0000", 2020-06-12T02:38:32.557 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1518999999999999e-05 2020-06-12T02:38:32.558 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.558 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.558 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:32.558 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.866925+0000", 2020-06-12T02:38:32.558 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010641576 2020-06-12T02:38:32.558 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.559 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.559 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:32.559 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.877567+0000", 2020-06-12T02:38:32.559 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9119000000000001e-05 2020-06-12T02:38:32.559 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.560 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.560 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:32.560 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.877626+0000", 2020-06-12T02:38:32.560 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01099082 2020-06-12T02:38:32.560 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.560 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:32.561 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.561 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.561 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.561 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.9s1 49/37 ECSubWrite(tid=4126, reqid=client.4512.0:8876, at_version=49'700, trim_to=0'0, roll_forward_to=49'697))", 2020-06-12T02:38:32.561 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.918553+0000", 2020-06-12T02:38:32.562 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 20.430010261, 2020-06-12T02:38:32.562 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011086824, 2020-06-12T02:38:32.562 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:32.562 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:32.562 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:32.563 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.563 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:32.563 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.918553+0000", 2020-06-12T02:38:32.563 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:32.563 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.563 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.564 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:32.564 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.918553+0000", 2020-06-12T02:38:32.564 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00024818900000000001 2020-06-12T02:38:32.564 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.564 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.564 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:32.565 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.918801+0000", 2020-06-12T02:38:32.565 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3191e-05 2020-06-12T02:38:32.565 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.565 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.565 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:32.566 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.918815+0000", 2020-06-12T02:38:32.566 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0339999999999999e-05 2020-06-12T02:38:32.566 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.566 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.566 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:32.567 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.918825+0000", 2020-06-12T02:38:32.567 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2440000000000004e-06 2020-06-12T02:38:32.567 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.567 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.567 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:32.567 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.918829+0000", 2020-06-12T02:38:32.568 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4657000000000003e-05 2020-06-12T02:38:32.568 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.568 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.568 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:32.568 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.918874+0000", 2020-06-12T02:38:32.569 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4171e-05 2020-06-12T02:38:32.569 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.569 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.569 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:32.569 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.918888+0000", 2020-06-12T02:38:32.569 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010692896 2020-06-12T02:38:32.570 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.570 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.570 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:32.570 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.929581+0000", 2020-06-12T02:38:32.570 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9135999999999998e-05 2020-06-12T02:38:32.571 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.571 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.571 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:32.571 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.929640+0000", 2020-06-12T02:38:32.571 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011086824 2020-06-12T02:38:32.571 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.572 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:32.572 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.572 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.572 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.572 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.9s1 49/37 ECSubWrite(tid=4209, reqid=client.4512.0:9129, at_version=49'718, trim_to=0'0, roll_forward_to=49'716))", 2020-06-12T02:38:32.573 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:12.647408+0000", 2020-06-12T02:38:32.573 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 19.701155486000001, 2020-06-12T02:38:32.573 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01161733, 2020-06-12T02:38:32.573 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:32.573 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:32.574 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:32.574 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.574 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:32.574 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.647408+0000", 2020-06-12T02:38:32.574 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:32.574 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.575 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.575 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:32.575 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.647408+0000", 2020-06-12T02:38:32.575 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00022105000000000001 2020-06-12T02:38:32.575 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.576 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.576 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:32.576 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.647629+0000", 2020-06-12T02:38:32.576 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2779e-05 2020-06-12T02:38:32.576 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.576 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.577 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:32.577 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.647652+0000", 2020-06-12T02:38:32.577 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0165000000000001e-05 2020-06-12T02:38:32.577 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.577 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.578 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:32.578 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.647672+0000", 2020-06-12T02:38:32.578 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2609999999999993e-06 2020-06-12T02:38:32.578 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.578 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.578 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:32.579 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.647680+0000", 2020-06-12T02:38:32.579 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2991999999999997e-05 2020-06-12T02:38:32.579 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.579 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.579 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:32.580 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.647733+0000", 2020-06-12T02:38:32.580 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4059e-05 2020-06-12T02:38:32.580 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.580 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.580 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:32.580 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.647747+0000", 2020-06-12T02:38:32.581 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011169695 2020-06-12T02:38:32.581 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.581 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.581 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:32.581 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.658917+0000", 2020-06-12T02:38:32.582 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000108329 2020-06-12T02:38:32.582 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.582 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.582 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:32.582 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.659025+0000", 2020-06-12T02:38:32.582 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01161733 2020-06-12T02:38:32.583 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.583 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:32.583 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.583 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.583 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.584 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.9s1 49/37 ECSubWrite(tid=4210, reqid=client.4512.0:9130, at_version=49'719, trim_to=0'0, roll_forward_to=49'716))", 2020-06-12T02:38:32.584 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:12.653737+0000", 2020-06-12T02:38:32.584 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 19.694826693, 2020-06-12T02:38:32.584 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010763803000000001, 2020-06-12T02:38:32.584 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:32.584 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:32.585 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:32.585 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.585 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:32.585 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.653737+0000", 2020-06-12T02:38:32.585 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:32.586 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.586 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.586 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:32.586 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.653737+0000", 2020-06-12T02:38:32.586 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000464399 2020-06-12T02:38:32.587 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.587 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.587 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:32.587 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.654201+0000", 2020-06-12T02:38:32.587 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3209000000000001e-05 2020-06-12T02:38:32.587 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.588 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.588 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:32.588 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.654224+0000", 2020-06-12T02:38:32.588 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6178e-05 2020-06-12T02:38:32.588 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.589 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.589 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:32.589 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.654241+0000", 2020-06-12T02:38:32.589 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2520000000000005e-06 2020-06-12T02:38:32.589 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.589 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.590 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:32.590 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.654250+0000", 2020-06-12T02:38:32.590 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6178000000000003e-05 2020-06-12T02:38:32.590 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.590 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.591 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:32.591 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.654306+0000", 2020-06-12T02:38:32.591 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4729000000000001e-05 2020-06-12T02:38:32.591 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.591 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.591 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:32.592 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.654321+0000", 2020-06-12T02:38:32.592 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010120298 2020-06-12T02:38:32.592 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.592 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.592 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:32.593 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.664441+0000", 2020-06-12T02:38:32.593 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9559999999999999e-05 2020-06-12T02:38:32.593 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.593 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.593 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:32.593 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.664501+0000", 2020-06-12T02:38:32.594 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010763803000000001 2020-06-12T02:38:32.594 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.594 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:32.594 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.594 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.595 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.595 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.9s1 54/37 ECSubWrite(tid=4356, reqid=client.4512.0:9713, at_version=54'773, trim_to=0'0, roll_forward_to=54'771))", 2020-06-12T02:38:32.595 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:17.079811+0000", 2020-06-12T02:38:32.595 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 15.268752068, 2020-06-12T02:38:32.595 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093890299999999996, 2020-06-12T02:38:32.595 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:32.596 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:32.596 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:32.596 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.596 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:32.596 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.079811+0000", 2020-06-12T02:38:32.597 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:32.597 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.597 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.597 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:32.597 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.079811+0000", 2020-06-12T02:38:32.597 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00033145800000000002 2020-06-12T02:38:32.598 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.598 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.598 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:32.598 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.080143+0000", 2020-06-12T02:38:32.598 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3251000000000002e-05 2020-06-12T02:38:32.599 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.599 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.599 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:32.599 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.080196+0000", 2020-06-12T02:38:32.599 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6387000000000001e-05 2020-06-12T02:38:32.599 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.600 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.600 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:32.600 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.080212+0000", 2020-06-12T02:38:32.600 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9909999999999999e-06 2020-06-12T02:38:32.600 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.601 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.601 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:32.601 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.080218+0000", 2020-06-12T02:38:32.601 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7734999999999999e-05 2020-06-12T02:38:32.601 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.601 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.602 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:32.602 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.080316+0000", 2020-06-12T02:38:32.602 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4931e-05 2020-06-12T02:38:32.602 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.607 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.608 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:32.608 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.080331+0000", 2020-06-12T02:38:32.609 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0088272929999999999 2020-06-12T02:38:32.609 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.609 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.609 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:32.609 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.089158+0000", 2020-06-12T02:38:32.610 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1984000000000003e-05 2020-06-12T02:38:32.610 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.610 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.610 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:32.610 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.089200+0000", 2020-06-12T02:38:32.610 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093890299999999996 2020-06-12T02:38:32.611 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.611 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:32.611 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.611 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.611 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.612 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.9s1 54/37 ECSubWrite(tid=4357, reqid=client.4512.0:9714, at_version=54'774, trim_to=0'0, roll_forward_to=54'771))", 2020-06-12T02:38:32.612 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:17.085009+0000", 2020-06-12T02:38:32.612 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 15.263554134, 2020-06-12T02:38:32.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0096277870000000005, 2020-06-12T02:38:32.612 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:32.613 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:32.613 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:32.613 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.613 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:32.613 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.085009+0000", 2020-06-12T02:38:32.613 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:32.614 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.614 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.614 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:32.614 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.085009+0000", 2020-06-12T02:38:32.615 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00046536100000000002 2020-06-12T02:38:32.615 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.615 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.616 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:32.616 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.085475+0000", 2020-06-12T02:38:32.616 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8621e-05 2020-06-12T02:38:32.616 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.616 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.616 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:32.617 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.085533+0000", 2020-06-12T02:38:32.617 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9032999999999999e-05 2020-06-12T02:38:32.617 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.617 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.617 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:32.618 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.085552+0000", 2020-06-12T02:38:32.618 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7370000000000006e-06 2020-06-12T02:38:32.618 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.618 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.618 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:32.618 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.085560+0000", 2020-06-12T02:38:32.619 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051025599999999995 2020-06-12T02:38:32.619 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.619 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.619 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:32.619 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.086070+0000", 2020-06-12T02:38:32.620 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2047e-05 2020-06-12T02:38:32.620 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.620 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.620 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:32.620 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.086082+0000", 2020-06-12T02:38:32.620 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085020559999999992 2020-06-12T02:38:32.621 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.621 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.621 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:32.621 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.094584+0000", 2020-06-12T02:38:32.621 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2676000000000002e-05 2020-06-12T02:38:32.622 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.622 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.622 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:32.622 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.094637+0000", 2020-06-12T02:38:32.622 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0096277870000000005 2020-06-12T02:38:32.622 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.623 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:32.623 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.623 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.623 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.623 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.9s1 54/37 ECSubWrite(tid=4461, reqid=client.4512.0:10001, at_version=54'808, trim_to=0'0, roll_forward_to=54'806))", 2020-06-12T02:38:32.624 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:17.943598+0000", 2020-06-12T02:38:32.624 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 14.40496585, 2020-06-12T02:38:32.624 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011839257000000001, 2020-06-12T02:38:32.624 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:32.624 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:32.624 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:32.625 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.625 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:32.625 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.943598+0000", 2020-06-12T02:38:32.626 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:32.626 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.627 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.627 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:32.627 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.943598+0000", 2020-06-12T02:38:32.627 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00028123299999999998 2020-06-12T02:38:32.627 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.628 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.628 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:32.628 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.943879+0000", 2020-06-12T02:38:32.628 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6650999999999999e-05 2020-06-12T02:38:32.628 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.629 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.629 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:32.629 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.943905+0000", 2020-06-12T02:38:32.629 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0922000000000001e-05 2020-06-12T02:38:32.629 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.630 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.630 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:32.630 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.943926+0000", 2020-06-12T02:38:32.630 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8770000000000005e-06 2020-06-12T02:38:32.630 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.631 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.631 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:32.631 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.943936+0000", 2020-06-12T02:38:32.631 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4051e-05 2020-06-12T02:38:32.631 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.631 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.632 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:32.632 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.944010+0000", 2020-06-12T02:38:32.632 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3377000000000001e-05 2020-06-12T02:38:32.632 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.632 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.633 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:32.633 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.944024+0000", 2020-06-12T02:38:32.633 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011355466 2020-06-12T02:38:32.633 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.633 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.634 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:32.634 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.955379+0000", 2020-06-12T02:38:32.634 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7679999999999997e-05 2020-06-12T02:38:32.634 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.634 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.634 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:32.635 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.955437+0000", 2020-06-12T02:38:32.635 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011839257000000001 2020-06-12T02:38:32.635 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.635 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:32.635 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.636 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.636 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.636 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s2 55/29 ECSubWrite(tid=5814, reqid=client.4512.0:11251, at_version=55'951, trim_to=0'0, roll_forward_to=55'949))", 2020-06-12T02:38:32.636 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:21.379563+0000", 2020-06-12T02:38:32.636 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 10.969000905, 2020-06-12T02:38:32.636 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.009814092, 2020-06-12T02:38:32.637 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:32.637 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:32.637 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:32.637 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.637 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:32.638 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.379563+0000", 2020-06-12T02:38:32.638 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:32.638 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.638 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.638 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:32.638 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.379563+0000", 2020-06-12T02:38:32.639 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016663600000000001 2020-06-12T02:38:32.639 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.639 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.639 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:32.639 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.379729+0000", 2020-06-12T02:38:32.640 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2211000000000001e-05 2020-06-12T02:38:32.640 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.640 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.640 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:32.641 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.379751+0000", 2020-06-12T02:38:32.641 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5478999999999999e-05 2020-06-12T02:38:32.641 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.641 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.641 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:32.642 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.379767+0000", 2020-06-12T02:38:32.642 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.951e-06 2020-06-12T02:38:32.642 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.642 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.642 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:32.642 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.379774+0000", 2020-06-12T02:38:32.643 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6182999999999998e-05 2020-06-12T02:38:32.643 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.643 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.643 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:32.643 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.379860+0000", 2020-06-12T02:38:32.643 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2079000000000001e-05 2020-06-12T02:38:32.644 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.644 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.644 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:32.644 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.379872+0000", 2020-06-12T02:38:32.644 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0094034170000000007 2020-06-12T02:38:32.645 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.645 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.645 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:32.645 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.389275+0000", 2020-06-12T02:38:32.645 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101136 2020-06-12T02:38:32.646 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.646 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.646 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:32.646 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.389377+0000", 2020-06-12T02:38:32.646 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.009814092 2020-06-12T02:38:32.647 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.647 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:32.647 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.647 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.647 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.647 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 59/53 ECSubWrite(tid=44, reqid=client.4512.0:14904, at_version=0'0, trim_to=0'0, roll_forward_to=59'463))", 2020-06-12T02:38:32.648 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.874670+0000", 2020-06-12T02:38:32.648 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.47389308200000002, 2020-06-12T02:38:32.648 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054588419999999999, 2020-06-12T02:38:32.648 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:32.648 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:32.649 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:32.649 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.649 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:32.649 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.874670+0000", 2020-06-12T02:38:32.649 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:32.649 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.650 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.650 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:32.650 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.874670+0000", 2020-06-12T02:38:32.650 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.7840000000000001e-05 2020-06-12T02:38:32.650 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.650 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.651 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:32.651 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.874698+0000", 2020-06-12T02:38:32.651 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3350000000000007e-06 2020-06-12T02:38:32.651 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.651 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.652 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:32.652 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.874707+0000", 2020-06-12T02:38:32.652 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5979999999999997e-06 2020-06-12T02:38:32.652 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.652 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.652 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:32.653 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.874715+0000", 2020-06-12T02:38:32.653 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1420000000000003e-06 2020-06-12T02:38:32.653 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.653 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.653 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:32.654 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.874719+0000", 2020-06-12T02:38:32.654 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2641000000000001e-05 2020-06-12T02:38:32.654 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.654 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.654 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:32.654 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.874752+0000", 2020-06-12T02:38:32.655 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6160000000000002e-06 2020-06-12T02:38:32.655 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.655 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.655 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:32.655 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.874761+0000", 2020-06-12T02:38:32.656 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054451052 2020-06-12T02:38:32.656 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.656 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.656 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:32.656 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.929212+0000", 2020-06-12T02:38:32.656 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7196e-05 2020-06-12T02:38:32.657 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.657 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.657 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:32.657 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.929259+0000", 2020-06-12T02:38:32.657 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054588419999999999 2020-06-12T02:38:32.658 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.658 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:32.658 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.658 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.658 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:32.658 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:32.659 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd unset nodeep-scrub 2020-06-12T02:38:32.674 INFO:tasks.rados.rados.0.smithi107.stdout:7042: writing smithi10729407-16 from 3858432 to 4186112 tid 1 2020-06-12T02:38:32.675 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:38:32.675 INFO:tasks.rados.rados.0.smithi107.stdout:7043: seq_num 7042 ranges {0=344064} 2020-06-12T02:38:32.675 INFO:tasks.rados.rados.0.smithi107.stdout:7043: writing smithi10729407-10 from 0 to 344064 tid 1 2020-06-12T02:38:32.676 INFO:tasks.rados.rados.0.smithi107.stdout:7044: delete oid 19 current snap is 0 2020-06-12T02:38:32.676 INFO:tasks.rados.rados.0.smithi107.stdout:7042: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:32.676 INFO:tasks.rados.rados.0.smithi107.stdout:7044: done (2 left) 2020-06-12T02:38:32.677 INFO:tasks.rados.rados.0.smithi107.stdout:7045: delete oid 12 current snap is 0 2020-06-12T02:38:32.677 INFO:tasks.rados.rados.0.smithi107.stdout:7043: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:32.677 INFO:tasks.rados.rados.0.smithi107.stdout:7043: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:32.677 INFO:tasks.rados.rados.0.smithi107.stdout:7043: finishing write tid 4 to smithi10729407-10 2020-06-12T02:38:32.678 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 467 (ObjNum 7042 snap 0 seq_num 7042) dirty exists 2020-06-12T02:38:32.678 INFO:tasks.rados.rados.0.smithi107.stdout:7043: left oid 10 (ObjNum 7042 snap 0 seq_num 7042) 2020-06-12T02:38:32.678 INFO:tasks.rados.rados.0.smithi107.stdout:7042: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:32.678 INFO:tasks.rados.rados.0.smithi107.stdout:7043: done (2 left) 2020-06-12T02:38:32.679 INFO:tasks.rados.rados.0.smithi107.stdout:7045: done (1 left) 2020-06-12T02:38:32.679 INFO:tasks.rados.rados.0.smithi107.stdout:7046: delete oid 18 current snap is 0 2020-06-12T02:38:32.679 INFO:tasks.rados.rados.0.smithi107.stdout:7042: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:32.679 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1252 (ObjNum 7041 snap 0 seq_num 7041) dirty exists 2020-06-12T02:38:32.679 INFO:tasks.rados.rados.0.smithi107.stdout:7042: left oid 16 (ObjNum 7041 snap 0 seq_num 7041) 2020-06-12T02:38:32.680 INFO:tasks.rados.rados.0.smithi107.stdout:7042: done (1 left) 2020-06-12T02:38:32.680 INFO:tasks.rados.rados.0.smithi107.stdout:7046: done (0 left) 2020-06-12T02:38:32.680 INFO:tasks.rados.rados.0.smithi107.stdout:7047: delete oid 3 current snap is 0 2020-06-12T02:38:32.680 INFO:tasks.rados.rados.0.smithi107.stdout:7047: done (0 left) 2020-06-12T02:38:32.680 INFO:tasks.rados.rados.0.smithi107.stdout:7048: delete oid 1 current snap is 0 2020-06-12T02:38:32.681 INFO:tasks.rados.rados.0.smithi107.stdout:7048: done (0 left) 2020-06-12T02:38:32.681 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:32.681 INFO:tasks.rados.rados.0.smithi107.stdout:7049: seq_num 7048 ranges {4186112=442368} 2020-06-12T02:38:32.681 INFO:tasks.rados.rados.0.smithi107.stdout:7049: writing smithi10729407-16 from 4186112 to 4628480 tid 1 2020-06-12T02:38:32.681 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:32.682 INFO:tasks.rados.rados.0.smithi107.stdout:7050: seq_num 7049 ranges {0=458752} 2020-06-12T02:38:32.682 INFO:tasks.rados.rados.0.smithi107.stdout:7050: writing smithi10729407-13 from 0 to 458752 tid 1 2020-06-12T02:38:32.682 INFO:tasks.rados.rados.0.smithi107.stdout:7051: delete oid 12 current snap is 0 2020-06-12T02:38:32.682 INFO:tasks.rados.rados.0.smithi107.stdout:7051: done (2 left) 2020-06-12T02:38:32.683 INFO:tasks.rados.rados.0.smithi107.stdout:7052: delete oid 2 current snap is 0 2020-06-12T02:38:32.683 INFO:tasks.rados.rados.0.smithi107.stdout:7050: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:32.683 INFO:tasks.rados.rados.0.smithi107.stdout:7049: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:32.683 INFO:tasks.rados.rados.0.smithi107.stdout:7052: done (2 left) 2020-06-12T02:38:32.683 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:32.684 INFO:tasks.rados.rados.0.smithi107.stdout:7053: seq_num 7052 ranges {3096576=516096} 2020-06-12T02:38:32.684 INFO:tasks.rados.rados.0.smithi107.stdout:7053: writing smithi10729407-17 from 3096576 to 3612672 tid 1 2020-06-12T02:38:32.684 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:32.684 INFO:tasks.rados.rados.0.smithi107.stdout:7054: seq_num 7053 ranges {0=532480} 2020-06-12T02:38:32.684 INFO:tasks.rados.rados.0.smithi107.stdout:7054: writing smithi10729407-1 from 0 to 532480 tid 1 2020-06-12T02:38:32.684 INFO:tasks.rados.rados.0.smithi107.stdout:7055: delete oid 7 current snap is 0 2020-06-12T02:38:32.685 INFO:tasks.rados.rados.0.smithi107.stdout:7049: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:32.685 INFO:tasks.rados.rados.0.smithi107.stdout:7049: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:32.685 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1255 (ObjNum 7048 snap 0 seq_num 7048) dirty exists 2020-06-12T02:38:32.685 INFO:tasks.rados.rados.0.smithi107.stdout:7049: left oid 16 (ObjNum 7048 snap 0 seq_num 7048) 2020-06-12T02:38:32.686 INFO:tasks.rados.rados.0.smithi107.stdout:7050: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:32.686 INFO:tasks.rados.rados.0.smithi107.stdout:7050: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:32.686 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1303 (ObjNum 7049 snap 0 seq_num 7049) dirty exists 2020-06-12T02:38:32.687 INFO:tasks.rados.rados.0.smithi107.stdout:7050: left oid 13 (ObjNum 7049 snap 0 seq_num 7049) 2020-06-12T02:38:32.687 INFO:tasks.rados.rados.0.smithi107.stdout:7053: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:32.687 INFO:tasks.rados.rados.0.smithi107.stdout:7049: done (4 left) 2020-06-12T02:38:32.688 INFO:tasks.rados.rados.0.smithi107.stdout:7050: done (3 left) 2020-06-12T02:38:32.688 INFO:tasks.rados.rados.0.smithi107.stdout:7055: done (2 left) 2020-06-12T02:38:32.688 INFO:tasks.rados.rados.0.smithi107.stdout:7056: delete oid 13 current snap is 0 2020-06-12T02:38:32.689 INFO:tasks.rados.rados.0.smithi107.stdout:7054: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:32.689 INFO:tasks.rados.rados.0.smithi107.stdout:7053: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:32.689 INFO:tasks.rados.rados.0.smithi107.stdout:7056: done (2 left) 2020-06-12T02:38:32.690 INFO:tasks.rados.rados.0.smithi107.stdout:7057: delete oid 6 current snap is 0 2020-06-12T02:38:32.690 INFO:tasks.rados.rados.0.smithi107.stdout:7053: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:32.690 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1333 (ObjNum 7052 snap 0 seq_num 7052) dirty exists 2020-06-12T02:38:32.690 INFO:tasks.rados.rados.0.smithi107.stdout:7053: left oid 17 (ObjNum 7052 snap 0 seq_num 7052) 2020-06-12T02:38:32.691 INFO:tasks.rados.rados.0.smithi107.stdout:7054: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:32.691 INFO:tasks.rados.rados.0.smithi107.stdout:7053: done (2 left) 2020-06-12T02:38:32.691 INFO:tasks.rados.rados.0.smithi107.stdout:7057: done (1 left) 2020-06-12T02:38:32.692 INFO:tasks.rados.rados.0.smithi107.stdout:7058: delete oid 13 current snap is 0 2020-06-12T02:38:32.692 INFO:tasks.rados.rados.0.smithi107.stdout:7058: done (1 left) 2020-06-12T02:38:32.692 INFO:tasks.rados.rados.0.smithi107.stdout:7059: delete oid 10 current snap is 0 2020-06-12T02:38:32.692 INFO:tasks.rados.rados.0.smithi107.stdout:7054: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:32.692 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1287 (ObjNum 7053 snap 0 seq_num 7053) dirty exists 2020-06-12T02:38:32.693 INFO:tasks.rados.rados.0.smithi107.stdout:7054: left oid 1 (ObjNum 7053 snap 0 seq_num 7053) 2020-06-12T02:38:32.693 INFO:tasks.rados.rados.0.smithi107.stdout:7054: done (1 left) 2020-06-12T02:38:32.693 INFO:tasks.rados.rados.0.smithi107.stdout:7059: done (0 left) 2020-06-12T02:38:32.693 INFO:tasks.rados.rados.0.smithi107.stdout:7060: delete oid 9 current snap is 0 2020-06-12T02:38:32.694 INFO:tasks.rados.rados.0.smithi107.stdout:7060: done (0 left) 2020-06-12T02:38:32.694 INFO:tasks.rados.rados.0.smithi107.stdout:7061: delete oid 4 current snap is 0 2020-06-12T02:38:32.694 INFO:tasks.rados.rados.0.smithi107.stdout:7061: done (0 left) 2020-06-12T02:38:32.694 INFO:tasks.rados.rados.0.smithi107.stdout:7062: delete oid 17 current snap is 0 2020-06-12T02:38:32.695 INFO:tasks.rados.rados.0.smithi107.stdout:7062: done (0 left) 2020-06-12T02:38:32.695 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:32.695 INFO:tasks.rados.rados.0.smithi107.stdout:7063: seq_num 7062 ranges {4628480=679936} 2020-06-12T02:38:32.695 INFO:tasks.rados.rados.0.smithi107.stdout:7063: writing smithi10729407-16 from 4628480 to 5308416 tid 1 2020-06-12T02:38:32.695 INFO:tasks.rados.rados.0.smithi107.stdout:7064: delete oid 3 current snap is 0 2020-06-12T02:38:32.696 INFO:tasks.rados.rados.0.smithi107.stdout:7063: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:32.696 INFO:tasks.rados.rados.0.smithi107.stdout:7064: done (1 left) 2020-06-12T02:38:32.696 INFO:tasks.rados.rados.0.smithi107.stdout:7065: delete oid 12 current snap is 0 2020-06-12T02:38:32.696 INFO:tasks.rados.rados.0.smithi107.stdout:7063: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:32.697 INFO:tasks.rados.rados.0.smithi107.stdout:7063: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:32.697 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1258 (ObjNum 7062 snap 0 seq_num 7062) dirty exists 2020-06-12T02:38:32.697 INFO:tasks.rados.rados.0.smithi107.stdout:7063: left oid 16 (ObjNum 7062 snap 0 seq_num 7062) 2020-06-12T02:38:32.697 INFO:tasks.rados.rados.0.smithi107.stdout:7063: done (1 left) 2020-06-12T02:38:32.697 INFO:tasks.rados.rados.0.smithi107.stdout:7065: done (0 left) 2020-06-12T02:38:32.698 INFO:tasks.rados.rados.0.smithi107.stdout:7066: delete oid 4 current snap is 0 2020-06-12T02:38:32.698 INFO:tasks.rados.rados.0.smithi107.stdout:7066: done (0 left) 2020-06-12T02:38:32.698 INFO:tasks.rados.rados.0.smithi107.stdout:7067: delete oid 7 current snap is 0 2020-06-12T02:38:32.699 INFO:tasks.rados.rados.0.smithi107.stdout:7067: done (0 left) 2020-06-12T02:38:32.699 INFO:tasks.rados.rados.0.smithi107.stdout:7068: delete oid 8 current snap is 0 2020-06-12T02:38:32.699 INFO:tasks.rados.rados.0.smithi107.stdout:7068: done (0 left) 2020-06-12T02:38:32.699 INFO:tasks.rados.rados.0.smithi107.stdout:7069: delete oid 4 current snap is 0 2020-06-12T02:38:32.700 INFO:tasks.rados.rados.0.smithi107.stdout:7069: done (0 left) 2020-06-12T02:38:32.700 INFO:tasks.rados.rados.0.smithi107.stdout:7070: delete oid 15 current snap is 0 2020-06-12T02:38:32.700 INFO:tasks.rados.rados.0.smithi107.stdout:7070: done (0 left) 2020-06-12T02:38:32.701 INFO:tasks.rados.rados.0.smithi107.stdout:7071: delete oid 6 current snap is 0 2020-06-12T02:38:32.701 INFO:tasks.rados.rados.0.smithi107.stdout:7071: done (0 left) 2020-06-12T02:38:32.701 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:32.702 INFO:tasks.rados.rados.0.smithi107.stdout:7072: seq_num 7071 ranges {0=434176,434176=401408} 2020-06-12T02:38:32.702 INFO:tasks.rados.rados.0.smithi107.stdout:7072: writing smithi10729407-6 from 0 to 434176 tid 1 2020-06-12T02:38:32.702 INFO:tasks.rados.rados.0.smithi107.stdout:7072: writing smithi10729407-6 from 434176 to 835584 tid 2 2020-06-12T02:38:32.703 INFO:tasks.rados.rados.0.smithi107.stdout:7073: delete oid 10 current snap is 0 2020-06-12T02:38:32.703 INFO:tasks.rados.rados.0.smithi107.stdout:7073: done (1 left) 2020-06-12T02:38:32.703 INFO:tasks.rados.rados.0.smithi107.stdout:7074: delete oid 18 current snap is 0 2020-06-12T02:38:32.703 INFO:tasks.rados.rados.0.smithi107.stdout:7072: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:32.704 INFO:tasks.rados.rados.0.smithi107.stdout:7074: done (1 left) 2020-06-12T02:38:32.704 INFO:tasks.rados.rados.0.smithi107.stdout:7075: delete oid 14 current snap is 0 2020-06-12T02:38:32.704 INFO:tasks.rados.rados.0.smithi107.stdout:7072: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:32.705 INFO:tasks.rados.rados.0.smithi107.stdout:7072: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:32.705 INFO:tasks.rados.rados.0.smithi107.stdout:7075: done (1 left) 2020-06-12T02:38:32.705 INFO:tasks.rados.rados.0.smithi107.stdout:7076: delete oid 9 current snap is 0 2020-06-12T02:38:32.706 INFO:tasks.rados.rados.0.smithi107.stdout:7072: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:32.706 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1292 (ObjNum 7071 snap 0 seq_num 7071) dirty exists 2020-06-12T02:38:32.706 INFO:tasks.rados.rados.0.smithi107.stdout:7072: left oid 6 (ObjNum 7071 snap 0 seq_num 7071) 2020-06-12T02:38:32.707 INFO:tasks.rados.rados.0.smithi107.stdout:7072: done (1 left) 2020-06-12T02:38:32.707 INFO:tasks.rados.rados.0.smithi107.stdout:7076: done (0 left) 2020-06-12T02:38:32.707 INFO:tasks.rados.rados.0.smithi107.stdout:7077: delete oid 5 current snap is 0 2020-06-12T02:38:32.708 INFO:tasks.rados.rados.0.smithi107.stdout:7077: done (0 left) 2020-06-12T02:38:32.708 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:38:32.708 INFO:tasks.rados.rados.0.smithi107.stdout:7078: seq_num 7077 ranges {835584=532480,1368064=401408} 2020-06-12T02:38:32.709 INFO:tasks.rados.rados.0.smithi107.stdout:7078: writing smithi10729407-6 from 835584 to 1368064 tid 1 2020-06-12T02:38:32.709 INFO:tasks.rados.rados.0.smithi107.stdout:7078: writing smithi10729407-6 from 1368064 to 1769472 tid 2 2020-06-12T02:38:32.709 INFO:tasks.rados.rados.0.smithi107.stdout:7079: delete oid 5 current snap is 0 2020-06-12T02:38:32.709 INFO:tasks.rados.rados.0.smithi107.stdout:7079: done (1 left) 2020-06-12T02:38:32.710 INFO:tasks.rados.rados.0.smithi107.stdout:7080: delete oid 3 current snap is 0 2020-06-12T02:38:32.710 INFO:tasks.rados.rados.0.smithi107.stdout:7080: done (1 left) 2020-06-12T02:38:32.710 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:32.711 INFO:tasks.rados.rados.0.smithi107.stdout:7081: seq_num 7080 ranges {0=581632,581632=401408} 2020-06-12T02:38:32.711 INFO:tasks.rados.rados.0.smithi107.stdout:7081: writing smithi10729407-15 from 0 to 581632 tid 1 2020-06-12T02:38:32.711 INFO:tasks.rados.rados.0.smithi107.stdout:7081: writing smithi10729407-15 from 581632 to 983040 tid 2 2020-06-12T02:38:32.712 INFO:tasks.rados.rados.0.smithi107.stdout:7082: delete oid 13 current snap is 0 2020-06-12T02:38:32.712 INFO:tasks.rados.rados.0.smithi107.stdout:7078: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:32.712 INFO:tasks.rados.rados.0.smithi107.stdout:7078: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:32.713 INFO:tasks.rados.rados.0.smithi107.stdout:7078: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:32.713 INFO:tasks.rados.rados.0.smithi107.stdout:7078: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:32.714 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1295 (ObjNum 7077 snap 0 seq_num 7077) dirty exists 2020-06-12T02:38:32.714 INFO:tasks.rados.rados.0.smithi107.stdout:7078: left oid 6 (ObjNum 7077 snap 0 seq_num 7077) 2020-06-12T02:38:32.714 INFO:tasks.rados.rados.0.smithi107.stdout:7078: done (2 left) 2020-06-12T02:38:32.715 INFO:tasks.rados.rados.0.smithi107.stdout:7082: done (1 left) 2020-06-12T02:38:32.715 INFO:tasks.rados.rados.0.smithi107.stdout:7083: delete oid 10 current snap is 0 2020-06-12T02:38:32.715 INFO:tasks.rados.rados.0.smithi107.stdout:7081: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:32.715 INFO:tasks.rados.rados.0.smithi107.stdout:7083: done (1 left) 2020-06-12T02:38:32.716 INFO:tasks.rados.rados.0.smithi107.stdout:7084: delete oid 6 current snap is 0 2020-06-12T02:38:32.716 INFO:tasks.rados.rados.0.smithi107.stdout:7081: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:32.716 INFO:tasks.rados.rados.0.smithi107.stdout:7081: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:32.717 INFO:tasks.rados.rados.0.smithi107.stdout:7081: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:32.717 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1330 (ObjNum 7080 snap 0 seq_num 7080) dirty exists 2020-06-12T02:38:32.717 INFO:tasks.rados.rados.0.smithi107.stdout:7081: left oid 15 (ObjNum 7080 snap 0 seq_num 7080) 2020-06-12T02:38:32.718 INFO:tasks.rados.rados.0.smithi107.stdout:7081: done (1 left) 2020-06-12T02:38:32.718 INFO:tasks.rados.rados.0.smithi107.stdout:7084: done (0 left) 2020-06-12T02:38:32.718 INFO:tasks.rados.rados.0.smithi107.stdout:7085: delete oid 9 current snap is 0 2020-06-12T02:38:32.719 INFO:tasks.rados.rados.0.smithi107.stdout:7085: done (0 left) 2020-06-12T02:38:32.719 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:32.719 INFO:tasks.rados.rados.0.smithi107.stdout:7086: seq_num 7085 ranges {0=663552,663552=401408} 2020-06-12T02:38:32.720 INFO:tasks.rados.rados.0.smithi107.stdout:7086: writing smithi10729407-9 from 0 to 663552 tid 1 2020-06-12T02:38:32.720 INFO:tasks.rados.rados.0.smithi107.stdout:7086: writing smithi10729407-9 from 663552 to 1064960 tid 2 2020-06-12T02:38:32.720 INFO:tasks.rados.rados.0.smithi107.stdout:7087: delete oid 3 current snap is 0 2020-06-12T02:38:32.721 INFO:tasks.rados.rados.0.smithi107.stdout:7087: done (1 left) 2020-06-12T02:38:32.721 INFO:tasks.rados.rados.0.smithi107.stdout:7088: delete oid 5 current snap is 0 2020-06-12T02:38:32.721 INFO:tasks.rados.rados.0.smithi107.stdout:7086: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:32.721 INFO:tasks.rados.rados.0.smithi107.stdout:7088: done (1 left) 2020-06-12T02:38:32.722 INFO:tasks.rados.rados.0.smithi107.stdout:7089: delete oid 6 current snap is 0 2020-06-12T02:38:32.722 INFO:tasks.rados.rados.0.smithi107.stdout:7086: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:32.722 INFO:tasks.rados.rados.0.smithi107.stdout:7086: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:32.723 INFO:tasks.rados.rados.0.smithi107.stdout:7086: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:32.723 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1319 (ObjNum 7085 snap 0 seq_num 7085) dirty exists 2020-06-12T02:38:32.723 INFO:tasks.rados.rados.0.smithi107.stdout:7086: left oid 9 (ObjNum 7085 snap 0 seq_num 7085) 2020-06-12T02:38:32.724 INFO:tasks.rados.rados.0.smithi107.stdout:7086: done (1 left) 2020-06-12T02:38:32.724 INFO:tasks.rados.rados.0.smithi107.stdout:7089: done (0 left) 2020-06-12T02:38:32.724 INFO:tasks.rados.rados.0.smithi107.stdout:7090: delete oid 5 current snap is 0 2020-06-12T02:38:32.725 INFO:tasks.rados.rados.0.smithi107.stdout:7090: done (0 left) 2020-06-12T02:38:32.725 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:32.725 INFO:tasks.rados.rados.0.smithi107.stdout:7091: seq_num 7090 ranges {0=753664,753664=401408} 2020-06-12T02:38:32.726 INFO:tasks.rados.rados.0.smithi107.stdout:7091: writing smithi10729407-8 from 0 to 753664 tid 1 2020-06-12T02:38:32.726 INFO:tasks.rados.rados.0.smithi107.stdout:7091: writing smithi10729407-8 from 753664 to 1155072 tid 2 2020-06-12T02:38:32.726 INFO:tasks.rados.rados.0.smithi107.stdout:7092: delete oid 7 current snap is 0 2020-06-12T02:38:32.726 INFO:tasks.rados.rados.0.smithi107.stdout:7092: done (1 left) 2020-06-12T02:38:32.727 INFO:tasks.rados.rados.0.smithi107.stdout:7093: delete oid 13 current snap is 0 2020-06-12T02:38:32.727 INFO:tasks.rados.rados.0.smithi107.stdout:7093: done (1 left) 2020-06-12T02:38:32.727 INFO:tasks.rados.rados.0.smithi107.stdout:7094: delete oid 3 current snap is 0 2020-06-12T02:38:32.728 INFO:tasks.rados.rados.0.smithi107.stdout:7091: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:32.728 INFO:tasks.rados.rados.0.smithi107.stdout:7094: done (1 left) 2020-06-12T02:38:32.729 INFO:tasks.rados.rados.0.smithi107.stdout:7095: delete oid 15 current snap is 0 2020-06-12T02:38:32.729 INFO:tasks.rados.rados.0.smithi107.stdout:7091: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:32.729 INFO:tasks.rados.rados.0.smithi107.stdout:7091: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:32.729 INFO:tasks.rados.rados.0.smithi107.stdout:7095: done (1 left) 2020-06-12T02:38:32.730 INFO:tasks.rados.rados.0.smithi107.stdout:7096: delete oid 16 current snap is 0 2020-06-12T02:38:32.730 INFO:tasks.rados.rados.0.smithi107.stdout:7091: finishing write tid 5 to smithi10729407-8 2020-06-12T02:38:32.730 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1333 (ObjNum 7090 snap 0 seq_num 7090) dirty exists 2020-06-12T02:38:32.730 INFO:tasks.rados.rados.0.smithi107.stdout:7091: left oid 8 (ObjNum 7090 snap 0 seq_num 7090) 2020-06-12T02:38:32.730 INFO:tasks.rados.rados.0.smithi107.stdout:7091: done (1 left) 2020-06-12T02:38:32.731 INFO:tasks.rados.rados.0.smithi107.stdout:7096: done (0 left) 2020-06-12T02:38:32.731 INFO:tasks.rados.rados.0.smithi107.stdout:7097: delete oid 1 current snap is 0 2020-06-12T02:38:32.731 INFO:tasks.rados.rados.0.smithi107.stdout:7097: done (0 left) 2020-06-12T02:38:32.732 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:32.732 INFO:tasks.rados.rados.0.smithi107.stdout:7098: seq_num 7097 ranges {0=466944,466944=663552,1130496=139264} 2020-06-12T02:38:32.732 INFO:tasks.rados.rados.0.smithi107.stdout:7098: writing smithi10729407-19 from 0 to 466944 tid 1 2020-06-12T02:38:32.732 INFO:tasks.rados.rados.0.smithi107.stdout:7098: writing smithi10729407-19 from 466944 to 1130496 tid 2 2020-06-12T02:38:32.732 INFO:tasks.rados.rados.0.smithi107.stdout:7098: writing smithi10729407-19 from 1130496 to 1269760 tid 3 2020-06-12T02:38:32.732 INFO:tasks.rados.rados.0.smithi107.stdout:7099: delete oid 15 current snap is 0 2020-06-12T02:38:32.733 INFO:tasks.rados.rados.0.smithi107.stdout:7099: done (1 left) 2020-06-12T02:38:32.733 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:32.733 INFO:tasks.rados.rados.0.smithi107.stdout:7100: seq_num 7099 ranges {0=499712,499712=425984,925696=376832} 2020-06-12T02:38:32.733 INFO:tasks.rados.rados.0.smithi107.stdout:7100: writing smithi10729407-15 from 0 to 499712 tid 1 2020-06-12T02:38:32.733 INFO:tasks.rados.rados.0.smithi107.stdout:7100: writing smithi10729407-15 from 499712 to 925696 tid 2 2020-06-12T02:38:32.734 INFO:tasks.rados.rados.0.smithi107.stdout:7100: writing smithi10729407-15 from 925696 to 1302528 tid 3 2020-06-12T02:38:32.734 INFO:tasks.rados.rados.0.smithi107.stdout:7101: delete oid 10 current snap is 0 2020-06-12T02:38:32.734 INFO:tasks.rados.rados.0.smithi107.stdout:7098: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:32.734 INFO:tasks.rados.rados.0.smithi107.stdout:7098: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:32.734 INFO:tasks.rados.rados.0.smithi107.stdout:7098: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:32.735 INFO:tasks.rados.rados.0.smithi107.stdout:7098: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:32.735 INFO:tasks.rados.rados.0.smithi107.stdout:7098: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:32.735 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1312 (ObjNum 7097 snap 0 seq_num 7097) dirty exists 2020-06-12T02:38:32.735 INFO:tasks.rados.rados.0.smithi107.stdout:7098: left oid 19 (ObjNum 7097 snap 0 seq_num 7097) 2020-06-12T02:38:32.735 INFO:tasks.rados.rados.0.smithi107.stdout:7098: done (2 left) 2020-06-12T02:38:32.735 INFO:tasks.rados.rados.0.smithi107.stdout:7101: done (1 left) 2020-06-12T02:38:32.736 INFO:tasks.rados.rados.0.smithi107.stdout:7102: delete oid 11 current snap is 0 2020-06-12T02:38:32.736 INFO:tasks.rados.rados.0.smithi107.stdout:7102: done (1 left) 2020-06-12T02:38:32.736 INFO:tasks.rados.rados.0.smithi107.stdout:7103: delete oid 4 current snap is 0 2020-06-12T02:38:32.736 INFO:tasks.rados.rados.0.smithi107.stdout:7100: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:32.736 INFO:tasks.rados.rados.0.smithi107.stdout:7100: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:32.736 INFO:tasks.rados.rados.0.smithi107.stdout:7100: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:32.737 INFO:tasks.rados.rados.0.smithi107.stdout:7100: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:32.737 INFO:tasks.rados.rados.0.smithi107.stdout:7103: done (1 left) 2020-06-12T02:38:32.737 INFO:tasks.rados.rados.0.smithi107.stdout:7104: delete oid 13 current snap is 0 2020-06-12T02:38:32.737 INFO:tasks.rados.rados.0.smithi107.stdout:7100: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:32.737 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1339 (ObjNum 7099 snap 0 seq_num 7099) dirty exists 2020-06-12T02:38:32.738 INFO:tasks.rados.rados.0.smithi107.stdout:7100: left oid 15 (ObjNum 7099 snap 0 seq_num 7099) 2020-06-12T02:38:32.738 INFO:tasks.rados.rados.0.smithi107.stdout:7100: done (1 left) 2020-06-12T02:38:32.738 INFO:tasks.rados.rados.0.smithi107.stdout:7104: done (0 left) 2020-06-12T02:38:32.738 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:32.738 INFO:tasks.rados.rados.0.smithi107.stdout:7105: seq_num 7104 ranges {0=581632,581632=704512,1286144=98304} 2020-06-12T02:38:32.738 INFO:tasks.rados.rados.0.smithi107.stdout:7105: writing smithi10729407-5 from 0 to 581632 tid 1 2020-06-12T02:38:32.739 INFO:tasks.rados.rados.0.smithi107.stdout:7105: writing smithi10729407-5 from 581632 to 1286144 tid 2 2020-06-12T02:38:32.739 INFO:tasks.rados.rados.0.smithi107.stdout:7105: writing smithi10729407-5 from 1286144 to 1384448 tid 3 2020-06-12T02:38:32.739 INFO:tasks.rados.rados.0.smithi107.stdout:7106: delete oid 8 current snap is 0 2020-06-12T02:38:32.739 INFO:tasks.rados.rados.0.smithi107.stdout:7105: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:32.739 INFO:tasks.rados.rados.0.smithi107.stdout:7106: done (1 left) 2020-06-12T02:38:32.740 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:38:32.740 INFO:tasks.rados.rados.0.smithi107.stdout:7107: seq_num 7106 ranges {0=614400,614400=466944,1081344=335872} 2020-06-12T02:38:32.740 INFO:tasks.rados.rados.0.smithi107.stdout:7107: writing smithi10729407-1 from 0 to 614400 tid 1 2020-06-12T02:38:32.740 INFO:tasks.rados.rados.0.smithi107.stdout:7107: writing smithi10729407-1 from 614400 to 1081344 tid 2 2020-06-12T02:38:32.741 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-06-12T02:38:32.741 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:32.657+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:32.748 INFO:tasks.rados.rados.0.smithi107.stdout:7107: writing smithi10729407-1 from 1081344 to 1417216 tid 3 2020-06-12T02:38:32.748 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:32.749 INFO:tasks.rados.rados.0.smithi107.stdout:7108: seq_num 7107 ranges {0=630784,630784=753664,1384448=49152} 2020-06-12T02:38:32.749 INFO:tasks.rados.rados.0.smithi107.stdout:7108: writing smithi10729407-8 from 0 to 630784 tid 1 2020-06-12T02:38:32.749 INFO:tasks.rados.rados.0.smithi107.stdout:7108: writing smithi10729407-8 from 630784 to 1384448 tid 2 2020-06-12T02:38:32.749 INFO:tasks.rados.rados.0.smithi107.stdout:7108: writing smithi10729407-8 from 1384448 to 1433600 tid 3 2020-06-12T02:38:32.750 INFO:tasks.rados.rados.0.smithi107.stdout:7109: delete oid 2 current snap is 0 2020-06-12T02:38:32.750 INFO:tasks.rados.rados.0.smithi107.stdout:7105: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:32.750 INFO:tasks.rados.rados.0.smithi107.stdout:7105: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:32.751 INFO:tasks.rados.rados.0.smithi107.stdout:7105: finishing write tid 5 to smithi10729407-5 2020-06-12T02:38:32.751 INFO:tasks.rados.rados.0.smithi107.stdout:7105: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:32.751 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 707 (ObjNum 7104 snap 0 seq_num 7104) dirty exists 2020-06-12T02:38:32.751 INFO:tasks.rados.rados.0.smithi107.stdout:7105: left oid 5 (ObjNum 7104 snap 0 seq_num 7104) 2020-06-12T02:38:32.751 INFO:tasks.rados.rados.0.smithi107.stdout:7107: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:32.752 INFO:tasks.rados.rados.0.smithi107.stdout:7107: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:32.752 INFO:tasks.rados.rados.0.smithi107.stdout:7105: done (3 left) 2020-06-12T02:38:32.752 INFO:tasks.rados.rados.0.smithi107.stdout:7109: done (2 left) 2020-06-12T02:38:32.752 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:32.752 INFO:tasks.rados.rados.0.smithi107.stdout:7110: seq_num 7109 ranges {0=671744,671744=573440,1245184=229376} 2020-06-12T02:38:32.753 INFO:tasks.rados.rados.0.smithi107.stdout:7110: writing smithi10729407-13 from 0 to 671744 tid 1 2020-06-12T02:38:32.753 INFO:tasks.rados.rados.0.smithi107.stdout:7110: writing smithi10729407-13 from 671744 to 1245184 tid 2 2020-06-12T02:38:32.754 INFO:tasks.rados.rados.0.smithi107.stdout:7110: writing smithi10729407-13 from 1245184 to 1474560 tid 3 2020-06-12T02:38:32.754 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:32.754 INFO:tasks.rados.rados.0.smithi107.stdout:7111: seq_num 7110 ranges {0=688128,688128=458752,1146880=344064} 2020-06-12T02:38:32.754 INFO:tasks.rados.rados.0.smithi107.stdout:7111: writing smithi10729407-12 from 0 to 688128 tid 1 2020-06-12T02:38:32.755 INFO:tasks.rados.rados.0.smithi107.stdout:7111: writing smithi10729407-12 from 688128 to 1146880 tid 2 2020-06-12T02:38:32.755 INFO:tasks.rados.rados.0.smithi107.stdout:7111: writing smithi10729407-12 from 1146880 to 1490944 tid 3 2020-06-12T02:38:32.755 INFO:tasks.rados.rados.0.smithi107.stdout:7112: delete oid 15 current snap is 0 2020-06-12T02:38:32.755 INFO:tasks.rados.rados.0.smithi107.stdout:7107: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:32.755 INFO:tasks.rados.rados.0.smithi107.stdout:7108: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:32.756 INFO:tasks.rados.rados.0.smithi107.stdout:7107: finishing write tid 5 to smithi10729407-1 2020-06-12T02:38:32.756 INFO:tasks.rados.rados.0.smithi107.stdout:7107: finishing write tid 6 to smithi10729407-1 2020-06-12T02:38:32.756 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1302 (ObjNum 7106 snap 0 seq_num 7106) dirty exists 2020-06-12T02:38:32.756 INFO:tasks.rados.rados.0.smithi107.stdout:7107: left oid 1 (ObjNum 7106 snap 0 seq_num 7106) 2020-06-12T02:38:32.756 INFO:tasks.rados.rados.0.smithi107.stdout:7108: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:32.756 INFO:tasks.rados.rados.0.smithi107.stdout:7108: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:32.757 INFO:tasks.rados.rados.0.smithi107.stdout:7108: finishing write tid 5 to smithi10729407-8 2020-06-12T02:38:32.757 INFO:tasks.rados.rados.0.smithi107.stdout:7110: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:32.757 INFO:tasks.rados.rados.0.smithi107.stdout:7108: finishing write tid 6 to smithi10729407-8 2020-06-12T02:38:32.757 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1344 (ObjNum 7107 snap 0 seq_num 7107) dirty exists 2020-06-12T02:38:32.757 INFO:tasks.rados.rados.0.smithi107.stdout:7108: left oid 8 (ObjNum 7107 snap 0 seq_num 7107) 2020-06-12T02:38:32.758 INFO:tasks.rados.rados.0.smithi107.stdout:7107: done (4 left) 2020-06-12T02:38:32.758 INFO:tasks.rados.rados.0.smithi107.stdout:7108: done (3 left) 2020-06-12T02:38:32.758 INFO:tasks.rados.rados.0.smithi107.stdout:7112: done (2 left) 2020-06-12T02:38:32.758 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:38:32.759 INFO:tasks.rados.rados.0.smithi107.stdout:7113: seq_num 7112 ranges {0=720896,720896=622592,1343488=180224} 2020-06-12T02:38:32.759 INFO:tasks.rados.rados.0.smithi107.stdout:7113: writing smithi10729407-11 from 0 to 720896 tid 1 2020-06-12T02:38:32.759 INFO:tasks.rados.rados.0.smithi107.stdout:7113: writing smithi10729407-11 from 720896 to 1343488 tid 2 2020-06-12T02:38:32.759 INFO:tasks.rados.rados.0.smithi107.stdout:7113: writing smithi10729407-11 from 1343488 to 1523712 tid 3 2020-06-12T02:38:32.760 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:32.760 INFO:tasks.rados.rados.0.smithi107.stdout:7114: seq_num 7113 ranges {0=737280,737280=507904,1245184=294912} 2020-06-12T02:38:32.760 INFO:tasks.rados.rados.0.smithi107.stdout:7114: writing smithi10729407-18 from 0 to 737280 tid 1 2020-06-12T02:38:32.760 INFO:tasks.rados.rados.0.smithi107.stdout:7114: writing smithi10729407-18 from 737280 to 1245184 tid 2 2020-06-12T02:38:32.760 INFO:tasks.rados.rados.0.smithi107.stdout:7114: writing smithi10729407-18 from 1245184 to 1540096 tid 3 2020-06-12T02:38:32.760 INFO:tasks.rados.rados.0.smithi107.stdout:7115: delete oid 8 current snap is 0 2020-06-12T02:38:32.761 INFO:tasks.rados.rados.0.smithi107.stdout:7111: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:32.761 INFO:tasks.rados.rados.0.smithi107.stdout:7111: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:32.761 INFO:tasks.rados.rados.0.smithi107.stdout:7110: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:32.761 INFO:tasks.rados.rados.0.smithi107.stdout:7110: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:32.762 INFO:tasks.rados.rados.0.smithi107.stdout:7111: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:32.762 INFO:tasks.rados.rados.0.smithi107.stdout:7110: finishing write tid 5 to smithi10729407-13 2020-06-12T02:38:32.762 INFO:tasks.rados.rados.0.smithi107.stdout:7111: finishing write tid 5 to smithi10729407-12 2020-06-12T02:38:32.762 INFO:tasks.rados.rados.0.smithi107.stdout:7111: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:32.762 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1316 (ObjNum 7110 snap 0 seq_num 7110) dirty exists 2020-06-12T02:38:32.762 INFO:tasks.rados.rados.0.smithi107.stdout:7111: left oid 12 (ObjNum 7110 snap 0 seq_num 7110) 2020-06-12T02:38:32.763 INFO:tasks.rados.rados.0.smithi107.stdout:7113: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:32.763 INFO:tasks.rados.rados.0.smithi107.stdout:7110: finishing write tid 6 to smithi10729407-13 2020-06-12T02:38:32.763 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1315 (ObjNum 7109 snap 0 seq_num 7109) dirty exists 2020-06-12T02:38:32.763 INFO:tasks.rados.rados.0.smithi107.stdout:7110: left oid 13 (ObjNum 7109 snap 0 seq_num 7109) 2020-06-12T02:38:32.763 INFO:tasks.rados.rados.0.smithi107.stdout:7113: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:32.764 INFO:tasks.rados.rados.0.smithi107.stdout:7110: done (4 left) 2020-06-12T02:38:32.764 INFO:tasks.rados.rados.0.smithi107.stdout:7111: done (3 left) 2020-06-12T02:38:32.764 INFO:tasks.rados.rados.0.smithi107.stdout:7115: done (2 left) 2020-06-12T02:38:32.764 INFO:tasks.rados.rados.0.smithi107.stdout:7116: delete oid 8 current snap is 0 2020-06-12T02:38:32.764 INFO:tasks.rados.rados.0.smithi107.stdout:7114: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:32.764 INFO:tasks.rados.rados.0.smithi107.stdout:7113: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:32.765 INFO:tasks.rados.rados.0.smithi107.stdout:7113: finishing write tid 5 to smithi10729407-11 2020-06-12T02:38:32.765 INFO:tasks.rados.rados.0.smithi107.stdout:7113: finishing write tid 6 to smithi10729407-11 2020-06-12T02:38:32.765 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1233 (ObjNum 7112 snap 0 seq_num 7112) dirty exists 2020-06-12T02:38:32.765 INFO:tasks.rados.rados.0.smithi107.stdout:7113: left oid 11 (ObjNum 7112 snap 0 seq_num 7112) 2020-06-12T02:38:32.765 INFO:tasks.rados.rados.0.smithi107.stdout:7113: done (2 left) 2020-06-12T02:38:32.766 INFO:tasks.rados.rados.0.smithi107.stdout:7116: done (1 left) 2020-06-12T02:38:32.766 INFO:tasks.rados.rados.0.smithi107.stdout:7117: delete oid 11 current snap is 0 2020-06-12T02:38:32.766 INFO:tasks.rados.rados.0.smithi107.stdout:7117: done (1 left) 2020-06-12T02:38:32.766 INFO:tasks.rados.rados.0.smithi107.stdout:7118: delete oid 15 current snap is 0 2020-06-12T02:38:32.766 INFO:tasks.rados.rados.0.smithi107.stdout:7114: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:32.767 INFO:tasks.rados.rados.0.smithi107.stdout:7118: done (1 left) 2020-06-12T02:38:32.767 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:38:32.767 INFO:tasks.rados.rados.0.smithi107.stdout:7119: seq_num 7118 ranges {1384448=417792,1802240=778240,2580480=425984} 2020-06-12T02:38:32.770 INFO:tasks.rados.rados.0.smithi107.stdout:7119: writing smithi10729407-5 from 1384448 to 1802240 tid 1 2020-06-12T02:38:32.770 INFO:tasks.rados.rados.0.smithi107.stdout:7119: writing smithi10729407-5 from 1802240 to 2580480 tid 2 2020-06-12T02:38:32.770 INFO:tasks.rados.rados.0.smithi107.stdout:7119: writing smithi10729407-5 from 2580480 to 3006464 tid 3 2020-06-12T02:38:32.771 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:38:32.771 INFO:tasks.rados.rados.0.smithi107.stdout:7120: seq_num 7119 ranges {0=434176,434176=663552,1097728=540672} 2020-06-12T02:38:32.771 INFO:tasks.rados.rados.0.smithi107.stdout:7120: writing smithi10729407-2 from 0 to 434176 tid 1 2020-06-12T02:38:32.771 INFO:tasks.rados.rados.0.smithi107.stdout:7120: writing smithi10729407-2 from 434176 to 1097728 tid 2 2020-06-12T02:38:32.771 INFO:tasks.rados.rados.0.smithi107.stdout:7120: writing smithi10729407-2 from 1097728 to 1638400 tid 3 2020-06-12T02:38:32.772 INFO:tasks.rados.rados.0.smithi107.stdout:7121: delete oid 11 current snap is 0 2020-06-12T02:38:32.772 INFO:tasks.rados.rados.0.smithi107.stdout:7114: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:32.772 INFO:tasks.rados.rados.0.smithi107.stdout:7114: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:32.772 INFO:tasks.rados.rados.0.smithi107.stdout:7114: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:32.772 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1323 (ObjNum 7113 snap 0 seq_num 7113) dirty exists 2020-06-12T02:38:32.773 INFO:tasks.rados.rados.0.smithi107.stdout:7114: left oid 18 (ObjNum 7113 snap 0 seq_num 7113) 2020-06-12T02:38:32.773 INFO:tasks.rados.rados.0.smithi107.stdout:7119: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:32.773 INFO:tasks.rados.rados.0.smithi107.stdout:7119: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:32.773 INFO:tasks.rados.rados.0.smithi107.stdout:7119: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:32.773 INFO:tasks.rados.rados.0.smithi107.stdout:7119: finishing write tid 5 to smithi10729407-5 2020-06-12T02:38:32.774 INFO:tasks.rados.rados.0.smithi107.stdout:7120: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:32.774 INFO:tasks.rados.rados.0.smithi107.stdout:7119: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:32.774 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 711 (ObjNum 7118 snap 0 seq_num 7118) dirty exists 2020-06-12T02:38:32.774 INFO:tasks.rados.rados.0.smithi107.stdout:7119: left oid 5 (ObjNum 7118 snap 0 seq_num 7118) 2020-06-12T02:38:32.775 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:32.766+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:32.776 INFO:tasks.rados.rados.0.smithi107.stdout:7114: done (3 left) 2020-06-12T02:38:32.776 INFO:tasks.rados.rados.0.smithi107.stdout:7119: done (2 left) 2020-06-12T02:38:32.776 INFO:tasks.rados.rados.0.smithi107.stdout:7121: done (1 left) 2020-06-12T02:38:32.777 INFO:tasks.rados.rados.0.smithi107.stdout:7122: delete oid 9 current snap is 0 2020-06-12T02:38:32.777 INFO:tasks.rados.rados.0.smithi107.stdout:7120: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:32.778 INFO:tasks.rados.rados.0.smithi107.stdout:7120: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:32.778 INFO:tasks.rados.rados.0.smithi107.stdout:7120: finishing write tid 5 to smithi10729407-2 2020-06-12T02:38:32.779 INFO:tasks.rados.rados.0.smithi107.stdout:7122: done (1 left) 2020-06-12T02:38:32.779 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:32.779 INFO:tasks.rados.rados.0.smithi107.stdout:7123: seq_num 7122 ranges {0=483328,483328=720896,1204224=483328} 2020-06-12T02:38:32.781 INFO:tasks.rados.rados.0.smithi107.stdout:7123: writing smithi10729407-4 from 0 to 483328 tid 1 2020-06-12T02:38:32.786 INFO:tasks.rados.rados.0.smithi107.stdout:7123: writing smithi10729407-4 from 483328 to 1204224 tid 2 2020-06-12T02:38:32.790 INFO:tasks.rados.rados.0.smithi107.stdout:7123: writing smithi10729407-4 from 1204224 to 1687552 tid 3 2020-06-12T02:38:32.791 INFO:tasks.rados.rados.0.smithi107.stdout:7124: delete oid 14 current snap is 0 2020-06-12T02:38:32.791 INFO:tasks.rados.rados.0.smithi107.stdout:7120: finishing write tid 6 to smithi10729407-2 2020-06-12T02:38:32.791 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 676 (ObjNum 7119 snap 0 seq_num 7119) dirty exists 2020-06-12T02:38:32.792 INFO:tasks.rados.rados.0.smithi107.stdout:7120: left oid 2 (ObjNum 7119 snap 0 seq_num 7119) 2020-06-12T02:38:32.792 INFO:tasks.rados.rados.0.smithi107.stdout:7123: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:32.795 INFO:tasks.rados.rados.0.smithi107.stdout:7120: done (2 left) 2020-06-12T02:38:32.796 INFO:tasks.rados.rados.0.smithi107.stdout:7124: done (1 left) 2020-06-12T02:38:32.796 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:32.796 INFO:tasks.rados.rados.0.smithi107.stdout:7125: seq_num 7124 ranges {0=516096,516096=532480,1048576=671744} 2020-06-12T02:38:32.798 INFO:tasks.rados.rados.0.smithi107.stdout:7125: writing smithi10729407-15 from 0 to 516096 tid 1 2020-06-12T02:38:32.802 INFO:tasks.rados.rados.0.smithi107.stdout:7125: writing smithi10729407-15 from 516096 to 1048576 tid 2 2020-06-12T02:38:32.806 INFO:tasks.rados.rados.0.smithi107.stdout:7125: writing smithi10729407-15 from 1048576 to 1720320 tid 3 2020-06-12T02:38:32.807 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:32.807 INFO:tasks.rados.rados.0.smithi107.stdout:7126: seq_num 7125 ranges {1540096=532480,2072576=417792,2490368=524288,3014656=262144} 2020-06-12T02:38:32.819 INFO:tasks.rados.rados.0.smithi107.stdout:7126: writing smithi10729407-18 from 1540096 to 2072576 tid 1 2020-06-12T02:38:32.822 INFO:tasks.rados.rados.0.smithi107.stdout:7126: writing smithi10729407-18 from 2072576 to 2490368 tid 2 2020-06-12T02:38:32.824 INFO:tasks.rados.rados.0.smithi107.stdout:7126: writing smithi10729407-18 from 2490368 to 3014656 tid 3 2020-06-12T02:38:32.826 INFO:tasks.rados.rados.0.smithi107.stdout:7126: writing smithi10729407-18 from 3014656 to 3276800 tid 4 2020-06-12T02:38:32.826 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:32.827 INFO:tasks.rados.rados.0.smithi107.stdout:7127: seq_num 7126 ranges {0=548864,548864=704512,1253376=499712} 2020-06-12T02:38:32.828 INFO:tasks.rados.rados.0.smithi107.stdout:7127: writing smithi10729407-7 from 0 to 548864 tid 1 2020-06-12T02:38:32.833 INFO:tasks.rados.rados.0.smithi107.stdout:7127: writing smithi10729407-7 from 548864 to 1253376 tid 2 2020-06-12T02:38:32.852 INFO:tasks.rados.rados.0.smithi107.stdout:7127: writing smithi10729407-7 from 1253376 to 1753088 tid 3 2020-06-12T02:38:32.852 INFO:tasks.rados.rados.0.smithi107.stdout:7128: delete oid 13 current snap is 0 2020-06-12T02:38:32.852 INFO:tasks.rados.rados.0.smithi107.stdout:7123: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:32.853 INFO:tasks.rados.rados.0.smithi107.stdout:7123: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:32.853 INFO:tasks.rados.rados.0.smithi107.stdout:7123: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:32.853 INFO:tasks.rados.rados.0.smithi107.stdout:7123: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:32.854 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1342 (ObjNum 7122 snap 0 seq_num 7122) dirty exists 2020-06-12T02:38:32.854 INFO:tasks.rados.rados.0.smithi107.stdout:7123: left oid 4 (ObjNum 7122 snap 0 seq_num 7122) 2020-06-12T02:38:32.854 INFO:tasks.rados.rados.0.smithi107.stdout:7125: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:32.855 INFO:tasks.rados.rados.0.smithi107.stdout:7125: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:32.855 INFO:tasks.rados.rados.0.smithi107.stdout:7125: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:32.855 INFO:tasks.rados.rados.0.smithi107.stdout:7125: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:32.856 INFO:tasks.rados.rados.0.smithi107.stdout:7125: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:32.856 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1352 (ObjNum 7124 snap 0 seq_num 7124) dirty exists 2020-06-12T02:38:32.856 INFO:tasks.rados.rados.0.smithi107.stdout:7125: left oid 15 (ObjNum 7124 snap 0 seq_num 7124) 2020-06-12T02:38:32.901 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:32.902 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2020-06-12T02:38:32.902 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.902 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15220 3.7s0 3:e653d00f:::smithi10729407-18:head [append 0~532480 in=532480b] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:32.902 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.817582+0000", 2020-06-12T02:38:32.903 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.081751001000000004, 2020-06-12T02:38:32.903 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.081776550000000003, 2020-06-12T02:38:32.903 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:32.903 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:32.903 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:32.903 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:32.904 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:32.904 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15220 2020-06-12T02:38:32.904 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.904 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:32.904 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.905 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:32.905 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.817582+0000", 2020-06-12T02:38:32.905 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:32.905 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.905 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.905 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:32.906 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.817582+0000", 2020-06-12T02:38:32.906 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014160799999999999 2020-06-12T02:38:32.906 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.906 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.906 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:32.907 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.817723+0000", 2020-06-12T02:38:32.907 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6889999999999993e-06 2020-06-12T02:38:32.907 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.907 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.907 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:32.908 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.817732+0000", 2020-06-12T02:38:32.908 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2214e-05 2020-06-12T02:38:32.908 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.908 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.908 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:32.908 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.817744+0000", 2020-06-12T02:38:32.909 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9570000000000002e-06 2020-06-12T02:38:32.909 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.909 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.917 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:32.917 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.817752+0000", 2020-06-12T02:38:32.918 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4947e-05 2020-06-12T02:38:32.918 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.918 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.918 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:32.918 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.817817+0000", 2020-06-12T02:38:32.919 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012504899999999999 2020-06-12T02:38:32.919 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.919 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.919 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:32.919 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.817942+0000", 2020-06-12T02:38:32.920 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00081927200000000001 2020-06-12T02:38:32.920 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.920 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.920 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:32.920 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.818761+0000", 2020-06-12T02:38:32.920 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001179736 2020-06-12T02:38:32.921 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.921 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:32.921 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.921 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.921 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.922 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15221 3.7s0 3:e653d00f:::smithi10729407-18:head [append 0~417792 in=417792b] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:32.922 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.819933+0000", 2020-06-12T02:38:32.922 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.079399437000000003, 2020-06-12T02:38:32.922 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.079474534999999999, 2020-06-12T02:38:32.922 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:32.923 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:32.923 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:32.923 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:32.923 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:32.923 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15221 2020-06-12T02:38:32.923 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.924 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:32.924 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.924 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:32.924 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.819933+0000", 2020-06-12T02:38:32.924 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:32.925 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.925 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.925 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:32.925 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.819933+0000", 2020-06-12T02:38:32.925 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000302949 2020-06-12T02:38:32.925 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.926 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.926 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:32.926 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.820236+0000", 2020-06-12T02:38:32.926 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0984e-05 2020-06-12T02:38:32.926 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.927 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.927 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:32.927 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.820247+0000", 2020-06-12T02:38:32.927 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4894e-05 2020-06-12T02:38:32.927 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.927 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.928 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:32.928 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.820262+0000", 2020-06-12T02:38:32.928 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2409999999999998e-06 2020-06-12T02:38:32.928 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.928 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.928 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:32.929 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.820269+0000", 2020-06-12T02:38:32.929 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0024275859999999998 2020-06-12T02:38:32.929 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.929 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.929 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:32.930 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822697+0000", 2020-06-12T02:38:32.930 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011567599999999999 2020-06-12T02:38:32.930 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.930 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.930 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:32.930 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822813+0000", 2020-06-12T02:38:32.931 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000483714 2020-06-12T02:38:32.931 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.931 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.931 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:32.931 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.823296+0000", 2020-06-12T02:38:32.932 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0033630439999999999 2020-06-12T02:38:32.932 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.932 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:32.932 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.932 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.932 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.933 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 59/37 ECSubWriteReply(tid=7875, last_complete=59'1325, committed=1, applied=1))", 2020-06-12T02:38:32.933 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.822130+0000", 2020-06-12T02:38:32.933 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.077202207999999994, 2020-06-12T02:38:32.933 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.077319046000000002, 2020-06-12T02:38:32.933 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:32.934 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "queued for pg", 2020-06-12T02:38:32.934 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:32.934 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.934 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:32.934 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822130+0000", 2020-06-12T02:38:32.934 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:32.935 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.935 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.935 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:32.935 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822130+0000", 2020-06-12T02:38:32.935 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8309e-05 2020-06-12T02:38:32.936 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.936 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.936 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:32.936 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822189+0000", 2020-06-12T02:38:32.936 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6299999999999993e-06 2020-06-12T02:38:32.936 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.937 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.937 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:32.937 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822198+0000", 2020-06-12T02:38:32.937 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4632e-05 2020-06-12T02:38:32.937 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.938 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.938 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:32.938 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822213+0000", 2020-06-12T02:38:32.939 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8600000000000004e-06 2020-06-12T02:38:32.939 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.939 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.939 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:32.939 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822220+0000", 2020-06-12T02:38:32.940 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9431000000000005e-05 2020-06-12T02:38:32.940 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.940 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:32.940 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.940 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.940 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.941 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 59/37 ECSubWriteReply(tid=7875, last_complete=59'1325, committed=1, applied=1))", 2020-06-12T02:38:32.941 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.822633+0000", 2020-06-12T02:38:32.941 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.076699856999999996, 2020-06-12T02:38:32.941 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.076843214000000007, 2020-06-12T02:38:32.941 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:32.942 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "queued for pg", 2020-06-12T02:38:32.942 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:32.942 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.942 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:32.942 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822633+0000", 2020-06-12T02:38:32.942 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:32.943 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.943 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.943 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:32.943 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822633+0000", 2020-06-12T02:38:32.943 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9357000000000001e-05 2020-06-12T02:38:32.944 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.944 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.944 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:32.944 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822692+0000", 2020-06-12T02:38:32.944 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5350000000000004e-06 2020-06-12T02:38:32.944 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.945 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.945 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:32.945 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822701+0000", 2020-06-12T02:38:32.945 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.243e-05 2020-06-12T02:38:32.945 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.946 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.946 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:32.946 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822713+0000", 2020-06-12T02:38:32.946 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.857e-06 2020-06-12T02:38:32.946 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.946 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.947 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:32.947 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822719+0000", 2020-06-12T02:38:32.947 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6179000000000003e-05 2020-06-12T02:38:32.947 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.947 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:32.948 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.948 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.948 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.948 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15222 3.7s0 3.f00bca67 (undecoded) ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:32.948 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.822885+0000", 2020-06-12T02:38:32.948 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.076447411000000007, 2020-06-12T02:38:32.949 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.076618121999999997, 2020-06-12T02:38:32.949 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:32.949 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "queued for pg", 2020-06-12T02:38:32.949 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:32.949 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:32.950 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:32.950 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15222 2020-06-12T02:38:32.950 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.950 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:32.951 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.951 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:32.951 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822885+0000", 2020-06-12T02:38:32.951 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:32.951 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.951 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.952 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:32.952 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822885+0000", 2020-06-12T02:38:32.952 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019930499999999999 2020-06-12T02:38:32.952 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.953 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.953 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:32.953 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.823085+0000", 2020-06-12T02:38:32.953 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4039999999999995e-06 2020-06-12T02:38:32.953 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.954 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.954 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:32.954 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.823093+0000", 2020-06-12T02:38:32.954 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5169e-05 2020-06-12T02:38:32.954 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.955 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.955 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:32.955 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.823108+0000", 2020-06-12T02:38:32.955 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6379999999999998e-06 2020-06-12T02:38:32.955 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.955 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.956 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:32.956 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.823115+0000", 2020-06-12T02:38:32.956 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000229516 2020-06-12T02:38:32.956 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.956 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:32.957 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.957 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.957 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.957 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15223 3.7s0 3.f00bca67 (undecoded) ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:32.957 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.824390+0000", 2020-06-12T02:38:32.957 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.074942239999999993, 2020-06-12T02:38:32.958 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.075143276999999994, 2020-06-12T02:38:32.958 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:32.958 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "queued for pg", 2020-06-12T02:38:32.958 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:32.958 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:32.958 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:32.959 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15223 2020-06-12T02:38:32.959 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.959 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:32.959 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.959 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:32.960 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824390+0000", 2020-06-12T02:38:32.960 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:32.960 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.960 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.960 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:32.960 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824390+0000", 2020-06-12T02:38:32.961 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014672400000000001 2020-06-12T02:38:32.961 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.961 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.961 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:32.961 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824537+0000", 2020-06-12T02:38:32.962 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2020000000000004e-06 2020-06-12T02:38:32.962 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.962 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.962 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:32.962 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824545+0000", 2020-06-12T02:38:32.962 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5106000000000001e-05 2020-06-12T02:38:32.963 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.963 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.963 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:32.963 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824560+0000", 2020-06-12T02:38:32.963 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2539999999999998e-06 2020-06-12T02:38:32.964 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.964 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.964 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:32.964 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824567+0000", 2020-06-12T02:38:32.964 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000176286 2020-06-12T02:38:32.964 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.965 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:32.965 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.965 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.965 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.965 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15224 3.7s0 3.f00bca67 (undecoded) ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:32.966 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.824617+0000", 2020-06-12T02:38:32.966 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.074715395000000004, 2020-06-12T02:38:32.966 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.074946351999999994, 2020-06-12T02:38:32.966 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:32.966 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "queued for pg", 2020-06-12T02:38:32.966 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:32.967 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:32.967 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:32.967 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15224 2020-06-12T02:38:32.967 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.967 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:32.967 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.968 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:32.968 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824617+0000", 2020-06-12T02:38:32.968 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:32.968 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.968 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.969 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:32.969 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824617+0000", 2020-06-12T02:38:32.969 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3214999999999996e-05 2020-06-12T02:38:32.969 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.969 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.969 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:32.970 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824691+0000", 2020-06-12T02:38:32.970 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4949999999999996e-06 2020-06-12T02:38:32.970 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.970 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.970 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:32.971 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824697+0000", 2020-06-12T02:38:32.971 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1935e-05 2020-06-12T02:38:32.971 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.971 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.971 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:32.971 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824709+0000", 2020-06-12T02:38:32.972 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1459999999999998e-06 2020-06-12T02:38:32.972 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.972 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.972 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:32.972 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824713+0000", 2020-06-12T02:38:32.972 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5791000000000005e-05 2020-06-12T02:38:32.973 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.973 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:32.973 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.973 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.973 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.974 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15225 3.7s0 3.f00bca67 (undecoded) ondisk+read+rwordered+known_if_redirected e59)", 2020-06-12T02:38:32.974 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.824754+0000", 2020-06-12T02:38:32.974 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.074578865999999994, 2020-06-12T02:38:32.974 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.074840156000000005, 2020-06-12T02:38:32.974 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:32.974 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "queued for pg", 2020-06-12T02:38:32.975 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:32.975 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:32.975 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:32.975 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15225 2020-06-12T02:38:32.975 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.976 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:32.976 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.976 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:32.976 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824754+0000", 2020-06-12T02:38:32.976 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:32.976 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.977 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.977 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:32.977 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824754+0000", 2020-06-12T02:38:32.977 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3890999999999997e-05 2020-06-12T02:38:32.977 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.978 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.978 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:32.978 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824798+0000", 2020-06-12T02:38:32.978 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6840000000000001e-06 2020-06-12T02:38:32.978 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.978 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.979 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:32.979 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824803+0000", 2020-06-12T02:38:32.979 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1881e-05 2020-06-12T02:38:32.979 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.979 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.979 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:32.980 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824815+0000", 2020-06-12T02:38:32.980 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1400000000000002e-06 2020-06-12T02:38:32.980 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:32.980 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:32.980 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:32.981 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824819+0000", 2020-06-12T02:38:32.981 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5596e-05 2020-06-12T02:38:32.981 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.981 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:32.981 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.981 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:32.982 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2020-06-12T02:38:32.982 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 8 2020-06-12T02:38:32.982 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:32.987 INFO:tasks.rados.rados.0.smithi107.stdout:7126: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:32.987 INFO:tasks.rados.rados.0.smithi107.stdout:7127: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:32.987 INFO:tasks.rados.rados.0.smithi107.stdout:7127: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:32.988 INFO:tasks.rados.rados.0.smithi107.stdout:7126: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:32.988 INFO:tasks.rados.rados.0.smithi107.stdout:7126: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:32.988 INFO:tasks.rados.rados.0.smithi107.stdout:7127: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:32.988 INFO:tasks.rados.rados.0.smithi107.stdout:7126: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:32.988 INFO:tasks.rados.rados.0.smithi107.stdout:7127: finishing write tid 5 to smithi10729407-7 2020-06-12T02:38:32.989 INFO:tasks.rados.rados.0.smithi107.stdout:7126: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:32.989 INFO:tasks.rados.rados.0.smithi107.stdout:7123: done (4 left) 2020-06-12T02:38:32.989 INFO:tasks.rados.rados.0.smithi107.stdout:7125: done (3 left) 2020-06-12T02:38:32.989 INFO:tasks.rados.rados.0.smithi107.stdout:7128: done (2 left) 2020-06-12T02:38:32.989 INFO:tasks.rados.rados.0.smithi107.stdout:7129: delete oid 9 current snap is 0 2020-06-12T02:38:32.990 INFO:tasks.rados.rados.0.smithi107.stdout:7126: finishing write tid 7 to smithi10729407-18 2020-06-12T02:38:32.990 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1329 (ObjNum 7125 snap 0 seq_num 7125) dirty exists 2020-06-12T02:38:32.990 INFO:tasks.rados.rados.0.smithi107.stdout:7126: left oid 18 (ObjNum 7125 snap 0 seq_num 7125) 2020-06-12T02:38:32.990 INFO:tasks.rados.rados.0.smithi107.stdout:7127: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:32.990 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1319 (ObjNum 7126 snap 0 seq_num 7126) dirty exists 2020-06-12T02:38:32.990 INFO:tasks.rados.rados.0.smithi107.stdout:7127: left oid 7 (ObjNum 7126 snap 0 seq_num 7126) 2020-06-12T02:38:32.991 INFO:tasks.rados.rados.0.smithi107.stdout:7126: done (2 left) 2020-06-12T02:38:32.991 INFO:tasks.rados.rados.0.smithi107.stdout:7127: done (1 left) 2020-06-12T02:38:32.991 INFO:tasks.rados.rados.0.smithi107.stdout:7129: done (0 left) 2020-06-12T02:38:32.991 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:32.991 INFO:tasks.rados.rados.0.smithi107.stdout:7130: seq_num 7129 ranges {0=606208,606208=761856,1368064=442368} 2020-06-12T02:38:32.992 INFO:tasks.rados.rados.0.smithi107.stdout:7130: writing smithi10729407-9 from 0 to 606208 tid 1 2020-06-12T02:38:32.992 INFO:tasks.rados.rados.0.smithi107.stdout:7130: writing smithi10729407-9 from 606208 to 1368064 tid 2 2020-06-12T02:38:32.992 INFO:tasks.rados.rados.0.smithi107.stdout:7130: writing smithi10729407-9 from 1368064 to 1810432 tid 3 2020-06-12T02:38:32.992 INFO:tasks.rados.rados.0.smithi107.stdout:7131: delete oid 11 current snap is 0 2020-06-12T02:38:32.992 INFO:tasks.rados.rados.0.smithi107.stdout:7131: done (1 left) 2020-06-12T02:38:32.992 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:38:32.993 INFO:tasks.rados.rados.0.smithi107.stdout:7132: seq_num 7131 ranges {0=638976,638976=524288,1163264=516096,1679360=163840} 2020-06-12T02:38:32.993 INFO:tasks.rados.rados.0.smithi107.stdout:7132: writing smithi10729407-11 from 0 to 638976 tid 1 2020-06-12T02:38:32.993 INFO:tasks.rados.rados.0.smithi107.stdout:7132: writing smithi10729407-11 from 638976 to 1163264 tid 2 2020-06-12T02:38:32.993 INFO:tasks.rados.rados.0.smithi107.stdout:7132: writing smithi10729407-11 from 1163264 to 1679360 tid 3 2020-06-12T02:38:32.993 INFO:tasks.rados.rados.0.smithi107.stdout:7132: writing smithi10729407-11 from 1679360 to 1843200 tid 4 2020-06-12T02:38:32.994 INFO:tasks.rados.rados.0.smithi107.stdout:7133: delete oid 8 current snap is 0 2020-06-12T02:38:32.994 INFO:tasks.rados.rados.0.smithi107.stdout:7130: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:32.994 INFO:tasks.rados.rados.0.smithi107.stdout:7132: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:32.994 INFO:tasks.rados.rados.0.smithi107.stdout:7130: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:32.994 INFO:tasks.rados.rados.0.smithi107.stdout:7130: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:32.994 INFO:tasks.rados.rados.0.smithi107.stdout:7130: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:32.995 INFO:tasks.rados.rados.0.smithi107.stdout:7133: done (2 left) 2020-06-12T02:38:32.995 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:38:32.995 INFO:tasks.rados.rados.0.smithi107.stdout:7134: seq_num 7133 ranges {0=671744,671744=745472,1417216=458752} 2020-06-12T02:38:32.995 INFO:tasks.rados.rados.0.smithi107.stdout:7134: writing smithi10729407-10 from 0 to 671744 tid 1 2020-06-12T02:38:32.995 INFO:tasks.rados.rados.0.smithi107.stdout:7134: writing smithi10729407-10 from 671744 to 1417216 tid 2 2020-06-12T02:38:32.996 INFO:tasks.rados.rados.0.smithi107.stdout:7134: writing smithi10729407-10 from 1417216 to 1875968 tid 3 2020-06-12T02:38:32.996 INFO:tasks.rados.rados.0.smithi107.stdout:7135: delete oid 6 current snap is 0 2020-06-12T02:38:32.996 INFO:tasks.rados.rados.0.smithi107.stdout:7130: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:32.996 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1334 (ObjNum 7129 snap 0 seq_num 7129) dirty exists 2020-06-12T02:38:32.996 INFO:tasks.rados.rados.0.smithi107.stdout:7130: left oid 9 (ObjNum 7129 snap 0 seq_num 7129) 2020-06-12T02:38:32.996 INFO:tasks.rados.rados.0.smithi107.stdout:7132: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:32.997 INFO:tasks.rados.rados.0.smithi107.stdout:7130: done (3 left) 2020-06-12T02:38:32.997 INFO:tasks.rados.rados.0.smithi107.stdout:7135: done (2 left) 2020-06-12T02:38:32.997 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:32.997 INFO:tasks.rados.rados.0.smithi107.stdout:7136: seq_num 7135 ranges {0=704512,704512=516096,1220608=499712,1720320=188416} 2020-06-12T02:38:32.997 INFO:tasks.rados.rados.0.smithi107.stdout:7136: writing smithi10729407-3 from 0 to 704512 tid 1 2020-06-12T02:38:32.998 INFO:tasks.rados.rados.0.smithi107.stdout:7136: writing smithi10729407-3 from 704512 to 1220608 tid 2 2020-06-12T02:38:32.998 INFO:tasks.rados.rados.0.smithi107.stdout:7136: writing smithi10729407-3 from 1220608 to 1720320 tid 3 2020-06-12T02:38:32.998 INFO:tasks.rados.rados.0.smithi107.stdout:7136: writing smithi10729407-3 from 1720320 to 1908736 tid 4 2020-06-12T02:38:32.998 INFO:tasks.rados.rados.0.smithi107.stdout:7137: delete oid 7 current snap is 0 2020-06-12T02:38:32.998 INFO:tasks.rados.rados.0.smithi107.stdout:7132: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:32.998 INFO:tasks.rados.rados.0.smithi107.stdout:7132: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:32.999 INFO:tasks.rados.rados.0.smithi107.stdout:7132: finishing write tid 6 to smithi10729407-11 2020-06-12T02:38:32.999 INFO:tasks.rados.rados.0.smithi107.stdout:7134: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:32.999 INFO:tasks.rados.rados.0.smithi107.stdout:7134: finishing write tid 2 to smithi10729407-10 2020-06-12T02:38:32.999 INFO:tasks.rados.rados.0.smithi107.stdout:7134: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:32.999 INFO:tasks.rados.rados.0.smithi107.stdout:7132: finishing write tid 7 to smithi10729407-11 2020-06-12T02:38:32.999 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1242 (ObjNum 7131 snap 0 seq_num 7131) dirty exists 2020-06-12T02:38:33.000 INFO:tasks.rados.rados.0.smithi107.stdout:7132: left oid 11 (ObjNum 7131 snap 0 seq_num 7131) 2020-06-12T02:38:33.000 INFO:tasks.rados.rados.0.smithi107.stdout:7134: finishing write tid 5 to smithi10729407-10 2020-06-12T02:38:33.000 INFO:tasks.rados.rados.0.smithi107.stdout:7134: finishing write tid 6 to smithi10729407-10 2020-06-12T02:38:33.000 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 475 (ObjNum 7133 snap 0 seq_num 7133) dirty exists 2020-06-12T02:38:33.000 INFO:tasks.rados.rados.0.smithi107.stdout:7134: left oid 10 (ObjNum 7133 snap 0 seq_num 7133) 2020-06-12T02:38:33.001 INFO:tasks.rados.rados.0.smithi107.stdout:7136: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:33.001 INFO:tasks.rados.rados.0.smithi107.stdout:7132: done (3 left) 2020-06-12T02:38:33.001 INFO:tasks.rados.rados.0.smithi107.stdout:7134: done (2 left) 2020-06-12T02:38:33.001 INFO:tasks.rados.rados.0.smithi107.stdout:7137: done (1 left) 2020-06-12T02:38:33.001 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:38:33.001 INFO:tasks.rados.rados.0.smithi107.stdout:7138: seq_num 7137 ranges {0=737280,737280=679936,1417216=466944,1884160=57344} 2020-06-12T02:38:33.002 INFO:tasks.rados.rados.0.smithi107.stdout:7138: writing smithi10729407-6 from 0 to 737280 tid 1 2020-06-12T02:38:33.002 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:32.867+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:33.002 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-06-12T02:38:33.003 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:32.983+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:33.005 INFO:tasks.rados.rados.0.smithi107.stdout:7138: writing smithi10729407-6 from 737280 to 1417216 tid 2 2020-06-12T02:38:33.005 INFO:tasks.rados.rados.0.smithi107.stdout:7138: writing smithi10729407-6 from 1417216 to 1884160 tid 3 2020-06-12T02:38:33.005 INFO:tasks.rados.rados.0.smithi107.stdout:7138: writing smithi10729407-6 from 1884160 to 1941504 tid 4 2020-06-12T02:38:33.006 INFO:tasks.rados.rados.0.smithi107.stdout:7139: delete oid 1 current snap is 0 2020-06-12T02:38:33.006 INFO:tasks.rados.rados.0.smithi107.stdout:7136: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:33.006 INFO:tasks.rados.rados.0.smithi107.stdout:7136: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:33.006 INFO:tasks.rados.rados.0.smithi107.stdout:7136: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:33.006 INFO:tasks.rados.rados.0.smithi107.stdout:7136: finishing write tid 6 to smithi10729407-3 2020-06-12T02:38:33.007 INFO:tasks.rados.rados.0.smithi107.stdout:7136: finishing write tid 7 to smithi10729407-3 2020-06-12T02:38:33.007 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1267 (ObjNum 7135 snap 0 seq_num 7135) dirty exists 2020-06-12T02:38:33.007 INFO:tasks.rados.rados.0.smithi107.stdout:7136: left oid 3 (ObjNum 7135 snap 0 seq_num 7135) 2020-06-12T02:38:33.007 INFO:tasks.rados.rados.0.smithi107.stdout:7138: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:33.007 INFO:tasks.rados.rados.0.smithi107.stdout:7138: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:33.007 INFO:tasks.rados.rados.0.smithi107.stdout:7138: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:33.008 INFO:tasks.rados.rados.0.smithi107.stdout:7138: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:33.008 INFO:tasks.rados.rados.0.smithi107.stdout:7138: finishing write tid 6 to smithi10729407-6 2020-06-12T02:38:33.008 INFO:tasks.rados.rados.0.smithi107.stdout:7136: done (2 left) 2020-06-12T02:38:33.008 INFO:tasks.rados.rados.0.smithi107.stdout:7139: done (1 left) 2020-06-12T02:38:33.008 INFO:tasks.rados.rados.0.smithi107.stdout:7140: delete oid 19 current snap is 0 2020-06-12T02:38:33.009 INFO:tasks.rados.rados.0.smithi107.stdout:7138: finishing write tid 7 to smithi10729407-6 2020-06-12T02:38:33.009 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1308 (ObjNum 7137 snap 0 seq_num 7137) dirty exists 2020-06-12T02:38:33.010 INFO:tasks.rados.rados.0.smithi107.stdout:7138: left oid 6 (ObjNum 7137 snap 0 seq_num 7137) 2020-06-12T02:38:33.010 INFO:tasks.rados.rados.0.smithi107.stdout:7138: done (1 left) 2020-06-12T02:38:33.010 INFO:tasks.rados.rados.0.smithi107.stdout:7140: done (0 left) 2020-06-12T02:38:33.010 INFO:tasks.rados.rados.0.smithi107.stdout:7141: delete oid 6 current snap is 0 2020-06-12T02:38:33.011 INFO:tasks.rados.rados.0.smithi107.stdout:7141: done (0 left) 2020-06-12T02:38:33.012 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:33.012 INFO:tasks.rados.rados.0.smithi107.stdout:7142: seq_num 7141 ranges {3276800=802816,4079616=671744,4751360=442368,5193728=90112} 2020-06-12T02:38:33.034 INFO:tasks.rados.rados.0.smithi107.stdout:7142: writing smithi10729407-18 from 3276800 to 4079616 tid 1 2020-06-12T02:38:33.044 INFO:tasks.rados.rados.0.smithi107.stdout:7142: writing smithi10729407-18 from 4079616 to 4751360 tid 2 2020-06-12T02:38:33.044 INFO:tasks.rados.rados.0.smithi107.stdout:7142: writing smithi10729407-18 from 4751360 to 5193728 tid 3 2020-06-12T02:38:33.045 INFO:tasks.rados.rados.0.smithi107.stdout:7142: writing smithi10729407-18 from 5193728 to 5283840 tid 4 2020-06-12T02:38:33.045 INFO:tasks.rados.rados.0.smithi107.stdout:7143: delete oid 5 current snap is 0 2020-06-12T02:38:33.046 INFO:tasks.rados.rados.0.smithi107.stdout:7142: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:33.046 INFO:tasks.rados.rados.0.smithi107.stdout:7143: done (1 left) 2020-06-12T02:38:33.047 INFO:tasks.rados.rados.0.smithi107.stdout:7144: delete oid 13 current snap is 0 2020-06-12T02:38:33.062 INFO:tasks.rados.rados.0.smithi107.stdout:7144: done (1 left) 2020-06-12T02:38:33.062 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:38:33.063 INFO:tasks.rados.rados.0.smithi107.stdout:7145: seq_num 7144 ranges {1875968=450560,2326528=720896,3047424=663552,3710976=221184} 2020-06-12T02:38:33.065 INFO:tasks.rados.rados.0.smithi107.stdout:7145: writing smithi10729407-10 from 1875968 to 2326528 tid 1 2020-06-12T02:38:33.068 INFO:tasks.rados.rados.0.smithi107.stdout:7145: writing smithi10729407-10 from 2326528 to 3047424 tid 2 2020-06-12T02:38:33.072 INFO:tasks.rados.rados.0.smithi107.stdout:7145: writing smithi10729407-10 from 3047424 to 3710976 tid 3 2020-06-12T02:38:33.074 INFO:tasks.rados.rados.0.smithi107.stdout:7145: writing smithi10729407-10 from 3710976 to 3932160 tid 4 2020-06-12T02:38:33.074 INFO:tasks.rados.rados.0.smithi107.stdout:7146: delete oid 12 current snap is 0 2020-06-12T02:38:33.074 INFO:tasks.rados.rados.0.smithi107.stdout:7142: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:33.074 INFO:tasks.rados.rados.0.smithi107.stdout:7142: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:33.075 INFO:tasks.rados.rados.0.smithi107.stdout:7142: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:33.075 INFO:tasks.rados.rados.0.smithi107.stdout:7142: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:33.075 INFO:tasks.rados.rados.0.smithi107.stdout:7142: finishing write tid 7 to smithi10729407-18 2020-06-12T02:38:33.075 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1339 (ObjNum 7141 snap 0 seq_num 7141) dirty exists 2020-06-12T02:38:33.075 INFO:tasks.rados.rados.0.smithi107.stdout:7142: left oid 18 (ObjNum 7141 snap 0 seq_num 7141) 2020-06-12T02:38:33.076 INFO:tasks.rados.rados.0.smithi107.stdout:7145: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:33.078 INFO:tasks.rados.rados.0.smithi107.stdout:7142: done (2 left) 2020-06-12T02:38:33.078 INFO:tasks.rados.rados.0.smithi107.stdout:7146: done (1 left) 2020-06-12T02:38:33.079 INFO:tasks.rados.rados.0.smithi107.stdout:7147: delete oid 17 current snap is 0 2020-06-12T02:38:33.084 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:33.081+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:33.084 INFO:tasks.rados.rados.0.smithi107.stdout:7145: finishing write tid 2 to smithi10729407-10 2020-06-12T02:38:33.087 INFO:tasks.rados.rados.0.smithi107.stdout:7147: done (1 left) 2020-06-12T02:38:33.087 INFO:tasks.rados.rados.0.smithi107.stdout:7148: delete oid 19 current snap is 0 2020-06-12T02:38:33.087 INFO:tasks.rados.rados.0.smithi107.stdout:7145: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:33.088 INFO:tasks.rados.rados.0.smithi107.stdout:7145: finishing write tid 4 to smithi10729407-10 2020-06-12T02:38:33.088 INFO:tasks.rados.rados.0.smithi107.stdout:7145: finishing write tid 6 to smithi10729407-10 2020-06-12T02:38:33.089 INFO:tasks.rados.rados.0.smithi107.stdout:7145: finishing write tid 7 to smithi10729407-10 2020-06-12T02:38:33.090 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 480 (ObjNum 7144 snap 0 seq_num 7144) dirty exists 2020-06-12T02:38:33.090 INFO:tasks.rados.rados.0.smithi107.stdout:7145: left oid 10 (ObjNum 7144 snap 0 seq_num 7144) 2020-06-12T02:38:33.091 INFO:tasks.rados.rados.0.smithi107.stdout:7145: done (1 left) 2020-06-12T02:38:33.091 INFO:tasks.rados.rados.0.smithi107.stdout:7148: done (0 left) 2020-06-12T02:38:33.092 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:33.092 INFO:tasks.rados.rados.0.smithi107.stdout:7149: seq_num 7148 ranges {1687552=524288,2211840=712704,2924544=647168,3571712=245760} 2020-06-12T02:38:33.102 INFO:tasks.rados.rados.0.smithi107.stdout:7149: writing smithi10729407-4 from 1687552 to 2211840 tid 1 2020-06-12T02:38:33.112 INFO:tasks.rados.rados.0.smithi107.stdout:7149: writing smithi10729407-4 from 2211840 to 2924544 tid 2 2020-06-12T02:38:33.112 INFO:tasks.rados.rados.0.smithi107.stdout:7149: writing smithi10729407-4 from 2924544 to 3571712 tid 3 2020-06-12T02:38:33.113 INFO:tasks.rados.rados.0.smithi107.stdout:7149: writing smithi10729407-4 from 3571712 to 3817472 tid 4 2020-06-12T02:38:33.113 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:33.114 INFO:tasks.rados.rados.0.smithi107.stdout:7150: seq_num 7149 ranges {0=540672,540672=589824,1130496=802816,1933312=212992} 2020-06-12T02:38:33.115 INFO:tasks.rados.rados.0.smithi107.stdout:7150: writing smithi10729407-12 from 0 to 540672 tid 1 2020-06-12T02:38:33.119 INFO:tasks.rados.rados.0.smithi107.stdout:7150: writing smithi10729407-12 from 540672 to 1130496 tid 2 2020-06-12T02:38:33.125 INFO:tasks.rados.rados.0.smithi107.stdout:7150: writing smithi10729407-12 from 1130496 to 1933312 tid 3 2020-06-12T02:38:33.126 INFO:tasks.rados.rados.0.smithi107.stdout:7150: writing smithi10729407-12 from 1933312 to 2146304 tid 4 2020-06-12T02:38:33.126 INFO:tasks.rados.rados.0.smithi107.stdout:7151: delete oid 2 current snap is 0 2020-06-12T02:38:33.127 INFO:tasks.rados.rados.0.smithi107.stdout:7149: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:33.127 INFO:tasks.rados.rados.0.smithi107.stdout:7149: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:33.127 INFO:tasks.rados.rados.0.smithi107.stdout:7149: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:33.127 INFO:tasks.rados.rados.0.smithi107.stdout:7149: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:33.127 INFO:tasks.rados.rados.0.smithi107.stdout:7149: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:33.134 INFO:tasks.rados.rados.0.smithi107.stdout:7151: done (2 left) 2020-06-12T02:38:33.134 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:33.135 INFO:tasks.rados.rados.0.smithi107.stdout:7152: seq_num 7151 ranges {0=573440,573440=761856,1335296=417792,1753088=425984} 2020-06-12T02:38:33.135 INFO:tasks.rados.rados.0.smithi107.stdout:7152: writing smithi10729407-16 from 0 to 573440 tid 1 2020-06-12T02:38:33.138 INFO:tasks.rados.rados.0.smithi107.stdout:7152: writing smithi10729407-16 from 573440 to 1335296 tid 2 2020-06-12T02:38:33.143 INFO:tasks.rados.rados.0.smithi107.stdout:7152: writing smithi10729407-16 from 1335296 to 1753088 tid 3 2020-06-12T02:38:33.144 INFO:tasks.rados.rados.0.smithi107.stdout:7152: writing smithi10729407-16 from 1753088 to 2179072 tid 4 2020-06-12T02:38:33.144 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:38:33.144 INFO:tasks.rados.rados.0.smithi107.stdout:7153: seq_num 7152 ranges {1843200=589824,2433024=647168,3080192=630784,3710976=327680} 2020-06-12T02:38:33.155 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-06-12T02:38:33.164 INFO:tasks.rados.rados.0.smithi107.stdout:7153: writing smithi10729407-11 from 1843200 to 2433024 tid 1 2020-06-12T02:38:33.165 INFO:tasks.rados.rados.0.smithi107.stdout:7153: writing smithi10729407-11 from 2433024 to 3080192 tid 2 2020-06-12T02:38:33.169 INFO:tasks.rados.rados.0.smithi107.stdout:7153: writing smithi10729407-11 from 3080192 to 3710976 tid 3 2020-06-12T02:38:33.170 INFO:tasks.rados.rados.0.smithi107.stdout:7153: writing smithi10729407-11 from 3710976 to 4038656 tid 4 2020-06-12T02:38:33.170 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:33.171 INFO:tasks.rados.rados.0.smithi107.stdout:7154: seq_num 7153 ranges {0=606208,606208=532480,1138688=786432,1925120=286720} 2020-06-12T02:38:33.171 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:33.172 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:38:33.173 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2020-06-12T02:38:33.173 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2020-06-12T02:38:33.174 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:33.176 INFO:tasks.rados.rados.0.smithi107.stdout:7154: writing smithi10729407-2 from 0 to 606208 tid 1 2020-06-12T02:38:33.183 INFO:tasks.rados.rados.0.smithi107.stdout:7154: writing smithi10729407-2 from 606208 to 1138688 tid 2 2020-06-12T02:38:33.187 INFO:tasks.rados.rados.0.smithi107.stdout:7154: writing smithi10729407-2 from 1138688 to 1925120 tid 3 2020-06-12T02:38:33.187 INFO:tasks.rados.rados.0.smithi107.stdout:7154: writing smithi10729407-2 from 1925120 to 2211840 tid 4 2020-06-12T02:38:33.188 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:33.188 INFO:tasks.rados.rados.0.smithi107.stdout:7155: seq_num 7154 ranges {1810432=622592,2433024=466944,2899968=589824,3489792=507904,3997696=40960} 2020-06-12T02:38:33.191 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-06-12T02:38:33.192 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:33.184+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:33.195 INFO:tasks.rados.rados.0.smithi107.stdout:7155: writing smithi10729407-9 from 1810432 to 2433024 tid 1 2020-06-12T02:38:33.201 INFO:tasks.rados.rados.0.smithi107.stdout:7155: writing smithi10729407-9 from 2433024 to 2899968 tid 2 2020-06-12T02:38:33.202 INFO:tasks.rados.rados.0.smithi107.stdout:7155: writing smithi10729407-9 from 2899968 to 3489792 tid 3 2020-06-12T02:38:33.207 INFO:tasks.rados.rados.0.smithi107.stdout:7155: writing smithi10729407-9 from 3489792 to 3997696 tid 4 2020-06-12T02:38:33.208 INFO:tasks.rados.rados.0.smithi107.stdout:7155: writing smithi10729407-9 from 3997696 to 4038656 tid 5 2020-06-12T02:38:33.208 INFO:tasks.rados.rados.0.smithi107.stdout:7156: delete oid 14 current snap is 0 2020-06-12T02:38:33.208 INFO:tasks.rados.rados.0.smithi107.stdout:7150: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:33.209 INFO:tasks.rados.rados.0.smithi107.stdout:7149: finishing write tid 7 to smithi10729407-4 2020-06-12T02:38:33.209 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1348 (ObjNum 7148 snap 0 seq_num 7148) dirty exists 2020-06-12T02:38:33.209 INFO:tasks.rados.rados.0.smithi107.stdout:7149: left oid 4 (ObjNum 7148 snap 0 seq_num 7148) 2020-06-12T02:38:33.210 INFO:tasks.rados.rados.0.smithi107.stdout:7150: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:33.210 INFO:tasks.rados.rados.0.smithi107.stdout:7150: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:33.210 INFO:tasks.rados.rados.0.smithi107.stdout:7150: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:33.211 INFO:tasks.rados.rados.0.smithi107.stdout:7150: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:33.211 INFO:tasks.rados.rados.0.smithi107.stdout:7152: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:33.211 INFO:tasks.rados.rados.0.smithi107.stdout:7150: finishing write tid 7 to smithi10729407-12 2020-06-12T02:38:33.212 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1324 (ObjNum 7149 snap 0 seq_num 7149) dirty exists 2020-06-12T02:38:33.212 INFO:tasks.rados.rados.0.smithi107.stdout:7150: left oid 12 (ObjNum 7149 snap 0 seq_num 7149) 2020-06-12T02:38:33.212 INFO:tasks.rados.rados.0.smithi107.stdout:7152: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:33.213 INFO:tasks.rados.rados.0.smithi107.stdout:7152: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:33.213 INFO:tasks.rados.rados.0.smithi107.stdout:7152: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:33.214 INFO:tasks.rados.rados.0.smithi107.stdout:7152: finishing write tid 6 to smithi10729407-16 2020-06-12T02:38:33.214 INFO:tasks.rados.rados.0.smithi107.stdout:7152: finishing write tid 7 to smithi10729407-16 2020-06-12T02:38:33.214 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1272 (ObjNum 7151 snap 0 seq_num 7151) dirty exists 2020-06-12T02:38:33.215 INFO:tasks.rados.rados.0.smithi107.stdout:7152: left oid 16 (ObjNum 7151 snap 0 seq_num 7151) 2020-06-12T02:38:33.215 INFO:tasks.rados.rados.0.smithi107.stdout:7153: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:33.215 INFO:tasks.rados.rados.0.smithi107.stdout:7153: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:33.216 INFO:tasks.rados.rados.0.smithi107.stdout:7153: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:33.216 INFO:tasks.rados.rados.0.smithi107.stdout:7153: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:33.216 INFO:tasks.rados.rados.0.smithi107.stdout:7153: finishing write tid 6 to smithi10729407-11 2020-06-12T02:38:33.217 INFO:tasks.rados.rados.0.smithi107.stdout:7153: finishing write tid 7 to smithi10729407-11 2020-06-12T02:38:33.217 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1247 (ObjNum 7152 snap 0 seq_num 7152) dirty exists 2020-06-12T02:38:33.217 INFO:tasks.rados.rados.0.smithi107.stdout:7153: left oid 11 (ObjNum 7152 snap 0 seq_num 7152) 2020-06-12T02:38:33.218 INFO:tasks.rados.rados.0.smithi107.stdout:7154: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:33.218 INFO:tasks.rados.rados.0.smithi107.stdout:7154: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:33.219 INFO:tasks.rados.rados.0.smithi107.stdout:7154: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:33.219 INFO:tasks.rados.rados.0.smithi107.stdout:7154: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:33.219 INFO:tasks.rados.rados.0.smithi107.stdout:7154: finishing write tid 6 to smithi10729407-2 2020-06-12T02:38:33.220 INFO:tasks.rados.rados.0.smithi107.stdout:7154: finishing write tid 7 to smithi10729407-2 2020-06-12T02:38:33.220 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 682 (ObjNum 7153 snap 0 seq_num 7153) dirty exists 2020-06-12T02:38:33.220 INFO:tasks.rados.rados.0.smithi107.stdout:7154: left oid 2 (ObjNum 7153 snap 0 seq_num 7153) 2020-06-12T02:38:33.221 INFO:tasks.rados.rados.0.smithi107.stdout:7155: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:33.222 INFO:tasks.rados.rados.0.smithi107.stdout:7149: done (6 left) 2020-06-12T02:38:33.222 INFO:tasks.rados.rados.0.smithi107.stdout:7150: done (5 left) 2020-06-12T02:38:33.223 INFO:tasks.rados.rados.0.smithi107.stdout:7152: done (4 left) 2020-06-12T02:38:33.223 INFO:tasks.rados.rados.0.smithi107.stdout:7153: done (3 left) 2020-06-12T02:38:33.224 INFO:tasks.rados.rados.0.smithi107.stdout:7154: done (2 left) 2020-06-12T02:38:33.224 INFO:tasks.rados.rados.0.smithi107.stdout:7156: done (1 left) 2020-06-12T02:38:33.224 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:33.225 INFO:tasks.rados.rados.0.smithi107.stdout:7157: seq_num 7156 ranges {4038656=655360,4694016=630784,5324800=606208,5931008=368640} 2020-06-12T02:38:33.237 INFO:tasks.rados.rados.0.smithi107.stdout:7157: writing smithi10729407-11 from 4038656 to 4694016 tid 1 2020-06-12T02:38:33.240 INFO:tasks.rados.rados.0.smithi107.stdout:7157: writing smithi10729407-11 from 4694016 to 5324800 tid 2 2020-06-12T02:38:33.244 INFO:tasks.rados.rados.0.smithi107.stdout:7157: writing smithi10729407-11 from 5324800 to 5931008 tid 3 2020-06-12T02:38:33.246 INFO:tasks.rados.rados.0.smithi107.stdout:7157: writing smithi10729407-11 from 5931008 to 6299648 tid 4 2020-06-12T02:38:33.247 INFO:tasks.rados.rados.0.smithi107.stdout:7158: delete oid 12 current snap is 0 2020-06-12T02:38:33.247 INFO:tasks.rados.rados.0.smithi107.stdout:7155: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:33.247 INFO:tasks.rados.rados.0.smithi107.stdout:7155: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:33.247 INFO:tasks.rados.rados.0.smithi107.stdout:7155: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:33.247 INFO:tasks.rados.rados.0.smithi107.stdout:7155: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:33.248 INFO:tasks.rados.rados.0.smithi107.stdout:7155: finishing write tid 7 to smithi10729407-9 2020-06-12T02:38:33.248 INFO:tasks.rados.rados.0.smithi107.stdout:7155: finishing write tid 8 to smithi10729407-9 2020-06-12T02:38:33.248 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1345 (ObjNum 7154 snap 0 seq_num 7154) dirty exists 2020-06-12T02:38:33.248 INFO:tasks.rados.rados.0.smithi107.stdout:7155: left oid 9 (ObjNum 7154 snap 0 seq_num 7154) 2020-06-12T02:38:33.250 INFO:tasks.rados.rados.0.smithi107.stdout:7155: done (2 left) 2020-06-12T02:38:33.250 INFO:tasks.rados.rados.0.smithi107.stdout:7158: done (1 left) 2020-06-12T02:38:33.250 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:38:33.251 INFO:tasks.rados.rados.0.smithi107.stdout:7159: seq_num 7158 ranges {3932160=688128,4620288=802816,5423104=573440,5996544=229376} 2020-06-12T02:38:33.278 INFO:tasks.rados.rados.0.smithi107.stdout:7159: writing smithi10729407-10 from 3932160 to 4620288 tid 1 2020-06-12T02:38:33.284 INFO:tasks.rados.rados.0.smithi107.stdout:7159: writing smithi10729407-10 from 4620288 to 5423104 tid 2 2020-06-12T02:38:33.289 INFO:tasks.rados.rados.0.smithi107.stdout:7159: writing smithi10729407-10 from 5423104 to 5996544 tid 3 2020-06-12T02:38:33.290 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:33.284+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:33.291 INFO:tasks.rados.rados.0.smithi107.stdout:7159: writing smithi10729407-10 from 5996544 to 6225920 tid 4 2020-06-12T02:38:33.291 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:33.291 INFO:tasks.rados.rados.0.smithi107.stdout:7160: seq_num 7159 ranges {0=704512,704512=688128,1392640=778240,2170880=139264} 2020-06-12T02:38:33.296 INFO:tasks.rados.rados.0.smithi107.stdout:7160: writing smithi10729407-5 from 0 to 704512 tid 1 2020-06-12T02:38:33.319 INFO:tasks.rados.rados.0.smithi107.stdout:7160: writing smithi10729407-5 from 704512 to 1392640 tid 2 2020-06-12T02:38:33.319 INFO:tasks.rados.rados.0.smithi107.stdout:7160: writing smithi10729407-5 from 1392640 to 2170880 tid 3 2020-06-12T02:38:33.320 INFO:tasks.rados.rados.0.smithi107.stdout:7160: writing smithi10729407-5 from 2170880 to 2310144 tid 4 2020-06-12T02:38:33.320 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:33.320 INFO:tasks.rados.rados.0.smithi107.stdout:7161: seq_num 7160 ranges {0=720896,720896=573440,1294336=589824,1884160=442368} 2020-06-12T02:38:33.321 INFO:tasks.rados.rados.0.smithi107.stdout:7161: writing smithi10729407-7 from 0 to 720896 tid 1 2020-06-12T02:38:33.321 INFO:tasks.rados.rados.0.smithi107.stdout:7161: writing smithi10729407-7 from 720896 to 1294336 tid 2 2020-06-12T02:38:33.322 INFO:tasks.rados.rados.0.smithi107.stdout:7161: writing smithi10729407-7 from 1294336 to 1884160 tid 3 2020-06-12T02:38:33.325 INFO:tasks.rados.rados.0.smithi107.stdout:7161: writing smithi10729407-7 from 1884160 to 2326528 tid 4 2020-06-12T02:38:33.326 INFO:tasks.rados.rados.0.smithi107.stdout:7162: delete oid 15 current snap is 0 2020-06-12T02:38:33.326 INFO:tasks.rados.rados.0.smithi107.stdout:7157: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:33.326 INFO:tasks.rados.rados.0.smithi107.stdout:7157: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:33.327 INFO:tasks.rados.rados.0.smithi107.stdout:7157: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:33.327 INFO:tasks.rados.rados.0.smithi107.stdout:7157: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:33.327 INFO:tasks.rados.rados.0.smithi107.stdout:7157: finishing write tid 6 to smithi10729407-11 2020-06-12T02:38:33.328 INFO:tasks.rados.rados.0.smithi107.stdout:7157: finishing write tid 7 to smithi10729407-11 2020-06-12T02:38:33.328 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1253 (ObjNum 7156 snap 0 seq_num 7156) dirty exists 2020-06-12T02:38:33.328 INFO:tasks.rados.rados.0.smithi107.stdout:7157: left oid 11 (ObjNum 7156 snap 0 seq_num 7156) 2020-06-12T02:38:33.328 INFO:tasks.rados.rados.0.smithi107.stdout:7159: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:33.329 INFO:tasks.rados.rados.0.smithi107.stdout:7159: finishing write tid 2 to smithi10729407-10 2020-06-12T02:38:33.329 INFO:tasks.rados.rados.0.smithi107.stdout:7159: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:33.329 INFO:tasks.rados.rados.0.smithi107.stdout:7159: finishing write tid 4 to smithi10729407-10 2020-06-12T02:38:33.330 INFO:tasks.rados.rados.0.smithi107.stdout:7159: finishing write tid 6 to smithi10729407-10 2020-06-12T02:38:33.330 INFO:tasks.rados.rados.0.smithi107.stdout:7159: finishing write tid 7 to smithi10729407-10 2020-06-12T02:38:33.331 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 485 (ObjNum 7158 snap 0 seq_num 7158) dirty exists 2020-06-12T02:38:33.331 INFO:tasks.rados.rados.0.smithi107.stdout:7159: left oid 10 (ObjNum 7158 snap 0 seq_num 7158) 2020-06-12T02:38:33.331 INFO:tasks.rados.rados.0.smithi107.stdout:7160: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:33.332 INFO:tasks.rados.rados.0.smithi107.stdout:7160: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:33.332 INFO:tasks.rados.rados.0.smithi107.stdout:7160: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:33.332 INFO:tasks.rados.rados.0.smithi107.stdout:7160: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:33.333 INFO:tasks.rados.rados.0.smithi107.stdout:7160: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:33.333 INFO:tasks.rados.rados.0.smithi107.stdout:7160: finishing write tid 7 to smithi10729407-5 2020-06-12T02:38:33.333 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 717 (ObjNum 7159 snap 0 seq_num 7159) dirty exists 2020-06-12T02:38:33.334 INFO:tasks.rados.rados.0.smithi107.stdout:7160: left oid 5 (ObjNum 7159 snap 0 seq_num 7159) 2020-06-12T02:38:33.335 INFO:tasks.rados.rados.0.smithi107.stdout:7161: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:33.336 INFO:tasks.rados.rados.0.smithi107.stdout:7161: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:33.336 INFO:tasks.rados.rados.0.smithi107.stdout:7157: done (4 left) 2020-06-12T02:38:33.336 INFO:tasks.rados.rados.0.smithi107.stdout:7159: done (3 left) 2020-06-12T02:38:33.337 INFO:tasks.rados.rados.0.smithi107.stdout:7160: done (2 left) 2020-06-12T02:38:33.338 INFO:tasks.rados.rados.0.smithi107.stdout:7162: done (1 left) 2020-06-12T02:38:33.338 INFO:tasks.rados.rados.0.smithi107.stdout:7163: delete oid 9 current snap is 0 2020-06-12T02:38:33.338 INFO:tasks.rados.rados.0.smithi107.stdout:7161: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:33.339 INFO:tasks.rados.rados.0.smithi107.stdout:7161: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:33.339 INFO:tasks.rados.rados.0.smithi107.stdout:7163: done (1 left) 2020-06-12T02:38:33.339 INFO:tasks.rados.rados.0.smithi107.stdout:7164: delete oid 17 current snap is 0 2020-06-12T02:38:33.339 INFO:tasks.rados.rados.0.smithi107.stdout:7161: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:33.341 INFO:tasks.rados.rados.0.smithi107.stdout:7161: finishing write tid 7 to smithi10729407-7 2020-06-12T02:38:33.341 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1327 (ObjNum 7160 snap 0 seq_num 7160) dirty exists 2020-06-12T02:38:33.344 INFO:tasks.rados.rados.0.smithi107.stdout:7161: left oid 7 (ObjNum 7160 snap 0 seq_num 7160) 2020-06-12T02:38:33.344 INFO:tasks.rados.rados.0.smithi107.stdout:7161: done (1 left) 2020-06-12T02:38:33.345 INFO:tasks.rados.rados.0.smithi107.stdout:7164: done (0 left) 2020-06-12T02:38:33.345 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:33.346 INFO:tasks.rados.rados.0.smithi107.stdout:7165: seq_num 7164 ranges {5283840=786432,6070272=557056,6627328=565248,7192576=483328} 2020-06-12T02:38:33.348 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:38:33.369 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:38:33.370 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:33.371 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2020-06-12T02:38:33.371 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2020-06-12T02:38:33.371 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2020-06-12T02:38:33.371 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.371 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8802 3.bs0 3:d0ebae7d:::smithi10729407-3:head [append 0~573440] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:33.372 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.691981+0000", 2020-06-12T02:38:33.372 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 21.674916585999998, 2020-06-12T02:38:33.372 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.024741973, 2020-06-12T02:38:33.372 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:33.372 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:33.373 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:33.373 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:33.373 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:33.373 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8802 2020-06-12T02:38:33.373 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.374 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:33.374 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.374 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:33.374 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.691981+0000", 2020-06-12T02:38:33.374 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:33.374 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.375 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.375 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:33.375 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.691981+0000", 2020-06-12T02:38:33.375 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000369312 2020-06-12T02:38:33.375 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.376 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.376 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:33.376 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.692350+0000", 2020-06-12T02:38:33.376 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9639999999999998e-06 2020-06-12T02:38:33.376 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.377 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.377 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:33.377 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.692360+0000", 2020-06-12T02:38:33.377 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9335999999999999e-05 2020-06-12T02:38:33.377 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.377 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.378 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:33.378 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.692379+0000", 2020-06-12T02:38:33.378 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8229999999999995e-06 2020-06-12T02:38:33.378 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.378 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.379 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:33.379 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.692387+0000", 2020-06-12T02:38:33.381 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0760999999999997e-05 2020-06-12T02:38:33.382 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.382 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.382 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:33.382 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.692438+0000", 2020-06-12T02:38:33.382 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000113064 2020-06-12T02:38:33.383 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.383 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.383 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:33.383 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.692551+0000", 2020-06-12T02:38:33.383 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00080948800000000003 2020-06-12T02:38:33.384 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.384 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.384 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:33.384 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.693361+0000", 2020-06-12T02:38:33.384 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.023159432000000001 2020-06-12T02:38:33.385 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.385 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.385 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:33.385 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.716520+0000", 2020-06-12T02:38:33.386 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000115021 2020-06-12T02:38:33.386 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.386 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.386 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:33.386 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.716635+0000", 2020-06-12T02:38:33.387 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7771999999999999e-05 2020-06-12T02:38:33.387 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.387 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.387 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:33.388 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.716723+0000", 2020-06-12T02:38:33.388 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.024741973 2020-06-12T02:38:33.388 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:33.388 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:33.389 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:33.389 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.389 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.389 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8806 3.bs0 3:d0ebae7d:::smithi10729407-3:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:33.389 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.700882+0000", 2020-06-12T02:38:33.389 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 21.666015861000002, 2020-06-12T02:38:33.390 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025273714999999999, 2020-06-12T02:38:33.390 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:33.390 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:33.390 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:33.390 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:33.391 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:33.391 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8806 2020-06-12T02:38:33.391 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.391 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:33.391 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.392 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:33.392 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700882+0000", 2020-06-12T02:38:33.392 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:33.392 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.392 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.392 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:33.393 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700882+0000", 2020-06-12T02:38:33.393 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7329000000000004e-05 2020-06-12T02:38:33.393 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.393 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.393 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:33.394 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700969+0000", 2020-06-12T02:38:33.394 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0620000000000001e-06 2020-06-12T02:38:33.394 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.394 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.394 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:33.395 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700977+0000", 2020-06-12T02:38:33.396 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5291e-05 2020-06-12T02:38:33.396 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.396 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.396 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:33.396 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700992+0000", 2020-06-12T02:38:33.397 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.297e-06 2020-06-12T02:38:33.397 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.397 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.397 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:33.397 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.700999+0000", 2020-06-12T02:38:33.398 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015970872000000001 2020-06-12T02:38:33.398 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.398 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.398 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:33.398 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.716969+0000", 2020-06-12T02:38:33.399 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105802 2020-06-12T02:38:33.399 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.399 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.399 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:33.399 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.717075+0000", 2020-06-12T02:38:33.400 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00042746400000000001 2020-06-12T02:38:33.400 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.400 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.400 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:33.400 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.717503+0000", 2020-06-12T02:38:33.400 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013889869999999999 2020-06-12T02:38:33.401 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.401 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.401 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:33.401 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.718892+0000", 2020-06-12T02:38:33.401 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071672740000000004 2020-06-12T02:38:33.402 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.402 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.402 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:33.402 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.726059+0000", 2020-06-12T02:38:33.402 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6336999999999996e-05 2020-06-12T02:38:33.402 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.403 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.403 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:33.403 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.726155+0000", 2020-06-12T02:38:33.403 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025273714999999999 2020-06-12T02:38:33.403 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:33.404 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:33.404 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:33.404 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.404 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.404 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8807 3.bs0 3:d0ebae7d:::smithi10729407-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e49)", 2020-06-12T02:38:33.405 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.701041+0000", 2020-06-12T02:38:33.405 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 21.665856271999999, 2020-06-12T02:38:33.405 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030589063, 2020-06-12T02:38:33.405 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:33.405 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:33.405 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:33.406 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:33.406 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:33.406 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8807 2020-06-12T02:38:33.406 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.406 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:33.407 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.407 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:33.407 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.701041+0000", 2020-06-12T02:38:33.407 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:33.407 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.407 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.408 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:33.408 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.701041+0000", 2020-06-12T02:38:33.408 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6292999999999999e-05 2020-06-12T02:38:33.408 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.408 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.409 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:33.409 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.701087+0000", 2020-06-12T02:38:33.409 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9840000000000004e-06 2020-06-12T02:38:33.409 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.409 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.409 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:33.410 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.701094+0000", 2020-06-12T02:38:33.410 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3713000000000001e-05 2020-06-12T02:38:33.410 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.410 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.410 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:33.411 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.701108+0000", 2020-06-12T02:38:33.411 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6200000000000004e-06 2020-06-12T02:38:33.411 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.411 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.411 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:33.412 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.701114+0000", 2020-06-12T02:38:33.412 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016879646000000002 2020-06-12T02:38:33.412 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.412 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.412 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:33.412 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.717993+0000", 2020-06-12T02:38:33.413 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4378999999999999e-05 2020-06-12T02:38:33.413 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.413 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.413 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:33.413 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.718088+0000", 2020-06-12T02:38:33.414 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0092861850000000006 2020-06-12T02:38:33.414 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.414 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.414 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:33.414 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.727374+0000", 2020-06-12T02:38:33.415 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1155000000000004e-05 2020-06-12T02:38:33.415 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.415 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.415 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:33.415 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.727465+0000", 2020-06-12T02:38:33.415 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004165088 2020-06-12T02:38:33.416 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.416 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.416 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:33.416 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.731630+0000", 2020-06-12T02:38:33.416 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.030589063 2020-06-12T02:38:33.416 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:33.417 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:33.417 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:33.417 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.417 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.417 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:11345 3.2s0 3:4afffc6f:::smithi10729407-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e56)", 2020-06-12T02:38:33.418 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:21.666049+0000", 2020-06-12T02:38:33.418 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 11.700848734999999, 2020-06-12T02:38:33.418 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025091826000000001, 2020-06-12T02:38:33.418 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:33.418 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:33.419 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:33.419 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:33.419 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:33.419 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 11345 2020-06-12T02:38:33.419 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.420 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:33.420 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.420 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:33.420 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.666049+0000", 2020-06-12T02:38:33.420 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:33.420 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.421 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.421 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:33.421 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.666049+0000", 2020-06-12T02:38:33.421 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.1736000000000002e-05 2020-06-12T02:38:33.421 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.422 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.426 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:33.426 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.666080+0000", 2020-06-12T02:38:33.427 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8160000000000004e-06 2020-06-12T02:38:33.427 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.427 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.427 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:33.427 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.666084+0000", 2020-06-12T02:38:33.428 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3100000000000003e-06 2020-06-12T02:38:33.428 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.428 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.428 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:33.428 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.666092+0000", 2020-06-12T02:38:33.428 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9490000000000001e-06 2020-06-12T02:38:33.429 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.429 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.429 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:33.429 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.666094+0000", 2020-06-12T02:38:33.429 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4011999999999995e-05 2020-06-12T02:38:33.430 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.430 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.430 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:33.430 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.666178+0000", 2020-06-12T02:38:33.430 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9424999999999993e-05 2020-06-12T02:38:33.430 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.431 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.431 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:33.431 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.666248+0000", 2020-06-12T02:38:33.431 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011857548000000001 2020-06-12T02:38:33.431 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.432 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.432 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:33.432 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.678105+0000", 2020-06-12T02:38:33.432 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8922999999999998e-05 2020-06-12T02:38:33.432 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.432 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.433 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:33.433 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.678164+0000", 2020-06-12T02:38:33.433 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012052894 2020-06-12T02:38:33.433 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.433 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.434 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:33.434 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.690217+0000", 2020-06-12T02:38:33.434 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6490999999999998e-05 2020-06-12T02:38:33.434 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.434 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.434 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:33.435 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.690274+0000", 2020-06-12T02:38:33.435 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000866722 2020-06-12T02:38:33.435 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.435 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.435 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:33.436 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:21.691141+0000", 2020-06-12T02:38:33.436 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025091826000000001 2020-06-12T02:38:33.436 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:33.436 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:33.436 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:33.436 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.437 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.437 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:13722 3.2s0 3:4afffc6f:::smithi10729407-20:head [stat,delete] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:38:33.437 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:28.421613+0000", 2020-06-12T02:38:33.437 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 4.9452849040000002, 2020-06-12T02:38:33.437 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046123156999999998, 2020-06-12T02:38:33.438 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:33.438 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:33.438 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:33.438 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:33.438 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:33.439 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 13722 2020-06-12T02:38:33.439 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.439 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:33.439 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.439 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:33.439 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:28.421613+0000", 2020-06-12T02:38:33.440 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:33.440 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.440 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.440 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:33.440 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:28.421613+0000", 2020-06-12T02:38:33.440 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.533e-05 2020-06-12T02:38:33.441 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.441 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.441 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:33.441 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:28.421638+0000", 2020-06-12T02:38:33.441 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4389999999999999e-06 2020-06-12T02:38:33.442 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.442 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.442 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:33.442 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:28.421643+0000", 2020-06-12T02:38:33.442 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1055e-05 2020-06-12T02:38:33.442 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.443 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.443 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:33.443 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:28.421654+0000", 2020-06-12T02:38:33.443 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3290000000000004e-06 2020-06-12T02:38:33.443 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.444 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.444 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:33.444 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:28.421659+0000", 2020-06-12T02:38:33.444 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3794000000000002e-05 2020-06-12T02:38:33.444 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.444 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.445 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:33.445 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:28.421692+0000", 2020-06-12T02:38:33.445 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2979999999999995e-05 2020-06-12T02:38:33.445 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.445 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.446 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:33.446 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:28.421775+0000", 2020-06-12T02:38:33.446 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.045960229999999998 2020-06-12T02:38:33.446 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.446 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.446 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:33.447 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:28.467736+0000", 2020-06-12T02:38:33.447 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046123156999999998 2020-06-12T02:38:33.447 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:33.447 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:33.447 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:33.447 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.448 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.448 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14661 3.bs0 3:d09543d2:::smithi10729407-16:head [stat,delete] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:33.448 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.152199+0000", 2020-06-12T02:38:33.448 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 2.2146986399999999, 2020-06-12T02:38:33.448 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.057759869999999998, 2020-06-12T02:38:33.449 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:33.449 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:33.449 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:33.449 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:33.449 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:33.449 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14661 2020-06-12T02:38:33.450 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.450 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:33.450 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.450 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:33.450 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.152199+0000", 2020-06-12T02:38:33.451 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:33.451 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.451 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.451 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:33.451 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.152199+0000", 2020-06-12T02:38:33.452 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6761999999999999e-05 2020-06-12T02:38:33.452 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.452 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.452 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:33.452 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.152226+0000", 2020-06-12T02:38:33.452 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1790000000000008e-06 2020-06-12T02:38:33.453 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.453 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.453 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:33.453 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.152234+0000", 2020-06-12T02:38:33.453 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1841999999999999e-05 2020-06-12T02:38:33.454 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.454 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.454 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:33.454 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.152246+0000", 2020-06-12T02:38:33.454 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4660000000000002e-06 2020-06-12T02:38:33.455 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.455 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.455 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:33.455 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.152253+0000", 2020-06-12T02:38:33.455 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9688000000000003e-05 2020-06-12T02:38:33.455 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.456 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.456 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:33.456 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.152303+0000", 2020-06-12T02:38:33.456 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4331000000000004e-05 2020-06-12T02:38:33.456 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.457 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.457 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:33.457 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.152377+0000", 2020-06-12T02:38:33.457 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.057581602000000003 2020-06-12T02:38:33.457 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.457 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.458 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:33.458 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209959+0000", 2020-06-12T02:38:33.458 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.057759869999999998 2020-06-12T02:38:33.458 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:33.459 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:33.459 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:33.459 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.459 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.459 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14912 3.5s1 3:a446366d:::smithi10729407-11:head [append 0~524288,stat] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:33.460 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.893048+0000", 2020-06-12T02:38:33.460 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 1.473849076, 2020-06-12T02:38:33.460 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038906464000000002, 2020-06-12T02:38:33.460 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:33.460 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:33.460 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:33.461 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:33.461 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:33.461 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14912 2020-06-12T02:38:33.461 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.461 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:33.462 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.462 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:33.462 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.893048+0000", 2020-06-12T02:38:33.462 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:33.462 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.463 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.463 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:33.463 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.893048+0000", 2020-06-12T02:38:33.463 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000144361 2020-06-12T02:38:33.463 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.463 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.464 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:33.464 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.893193+0000", 2020-06-12T02:38:33.464 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7450000000000001e-06 2020-06-12T02:38:33.464 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.464 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.465 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:33.465 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.893201+0000", 2020-06-12T02:38:33.465 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9619e-05 2020-06-12T02:38:33.465 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.465 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.465 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:33.466 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.893221+0000", 2020-06-12T02:38:33.466 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9449999999999993e-06 2020-06-12T02:38:33.466 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.466 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.466 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:33.467 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.893229+0000", 2020-06-12T02:38:33.467 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3192000000000003e-05 2020-06-12T02:38:33.467 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.468 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.468 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:33.468 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.893272+0000", 2020-06-12T02:38:33.468 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6910000000000006e-05 2020-06-12T02:38:33.469 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.469 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.469 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:33.469 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.893359+0000", 2020-06-12T02:38:33.469 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000519824 2020-06-12T02:38:33.470 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.470 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.470 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:33.470 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.893879+0000", 2020-06-12T02:38:33.470 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0031381719999999998 2020-06-12T02:38:33.471 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.471 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.471 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:33.471 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.897017+0000", 2020-06-12T02:38:33.471 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034756033999999998 2020-06-12T02:38:33.471 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.472 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.472 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:33.472 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.931773+0000", 2020-06-12T02:38:33.472 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000181662 2020-06-12T02:38:33.472 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.473 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.473 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:33.473 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.931955+0000", 2020-06-12T02:38:33.473 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038906464000000002 2020-06-12T02:38:33.473 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:33.474 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:33.474 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:33.474 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.474 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.474 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14913 3.5s1 3:a446366d:::smithi10729407-11:head [append 0~532480] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:33.475 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.896342+0000", 2020-06-12T02:38:33.475 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 1.470555802, 2020-06-12T02:38:33.475 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.045786648999999999, 2020-06-12T02:38:33.475 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:33.476 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:33.476 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:33.476 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:33.476 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:33.476 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14913 2020-06-12T02:38:33.476 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.477 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:33.477 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.477 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:33.477 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.896342+0000", 2020-06-12T02:38:33.477 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:33.478 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.478 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.478 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:33.478 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.896342+0000", 2020-06-12T02:38:33.478 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000140648 2020-06-12T02:38:33.478 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.479 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.479 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:33.479 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.896482+0000", 2020-06-12T02:38:33.479 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5550000000000002e-06 2020-06-12T02:38:33.479 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.480 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.480 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:33.480 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.896488+0000", 2020-06-12T02:38:33.480 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3759999999999997e-06 2020-06-12T02:38:33.480 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.480 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.481 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:33.481 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.896497+0000", 2020-06-12T02:38:33.481 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6890000000000001e-06 2020-06-12T02:38:33.481 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.481 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.482 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:33.482 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.896503+0000", 2020-06-12T02:38:33.482 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0099999999999999e-05 2020-06-12T02:38:33.482 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.482 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.482 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:33.483 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.896543+0000", 2020-06-12T02:38:33.483 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109474 2020-06-12T02:38:33.483 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.483 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.483 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:33.484 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.896652+0000", 2020-06-12T02:38:33.484 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.036111747 2020-06-12T02:38:33.484 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.484 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.485 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:33.485 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.932764+0000", 2020-06-12T02:38:33.485 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7319999999999998e-05 2020-06-12T02:38:33.485 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.485 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.485 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:33.486 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.932842+0000", 2020-06-12T02:38:33.486 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00062652699999999999 2020-06-12T02:38:33.486 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.486 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.486 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:33.487 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.933468+0000", 2020-06-12T02:38:33.487 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0084385180000000008 2020-06-12T02:38:33.487 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.487 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.487 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:33.487 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.941907+0000", 2020-06-12T02:38:33.488 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012994699999999999 2020-06-12T02:38:33.488 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.488 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.488 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:33.488 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.942037+0000", 2020-06-12T02:38:33.489 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1748000000000003e-05 2020-06-12T02:38:33.489 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.489 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.489 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:33.489 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.942128+0000", 2020-06-12T02:38:33.489 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.045786648999999999 2020-06-12T02:38:33.490 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:33.490 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:33.490 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:33.490 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.490 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.491 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14914 3.5s1 3:a446366d:::smithi10729407-11:head [append 0~671744] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:33.491 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.900206+0000", 2020-06-12T02:38:33.491 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 1.4666914470000001, 2020-06-12T02:38:33.491 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046728764999999998, 2020-06-12T02:38:33.491 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:33.491 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:33.492 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:33.492 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:33.492 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:33.492 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14914 2020-06-12T02:38:33.492 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.493 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:33.493 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.493 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:33.493 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900206+0000", 2020-06-12T02:38:33.493 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:33.494 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.494 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.494 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:33.494 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900206+0000", 2020-06-12T02:38:33.494 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000241518 2020-06-12T02:38:33.494 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.495 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.495 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:33.495 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900447+0000", 2020-06-12T02:38:33.495 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0219e-05 2020-06-12T02:38:33.496 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.496 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.496 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:33.496 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900458+0000", 2020-06-12T02:38:33.497 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7387000000000001e-05 2020-06-12T02:38:33.497 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.497 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.497 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:33.497 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900475+0000", 2020-06-12T02:38:33.498 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.661e-06 2020-06-12T02:38:33.498 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.498 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.498 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:33.498 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900483+0000", 2020-06-12T02:38:33.498 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4412000000000002e-05 2020-06-12T02:38:33.499 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.499 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.499 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:33.499 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900517+0000", 2020-06-12T02:38:33.499 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7199000000000005e-05 2020-06-12T02:38:33.500 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.500 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.500 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:33.500 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900584+0000", 2020-06-12T02:38:33.500 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.035314096000000003 2020-06-12T02:38:33.501 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.501 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.501 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:33.501 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.935898+0000", 2020-06-12T02:38:33.501 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000117197 2020-06-12T02:38:33.501 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.502 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.502 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:33.502 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.936016+0000", 2020-06-12T02:38:33.502 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012090289999999999 2020-06-12T02:38:33.502 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.503 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.503 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:33.503 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.937225+0000", 2020-06-12T02:38:33.503 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0095344799999999997 2020-06-12T02:38:33.503 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.503 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.504 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:33.504 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.946759+0000", 2020-06-12T02:38:33.504 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4289999999999993e-05 2020-06-12T02:38:33.504 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.504 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.504 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:33.505 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.946853+0000", 2020-06-12T02:38:33.505 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1277000000000001e-05 2020-06-12T02:38:33.505 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.505 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.505 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:33.506 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.946935+0000", 2020-06-12T02:38:33.506 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046728764999999998 2020-06-12T02:38:33.506 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:33.506 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:33.506 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:33.506 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.507 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.507 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14915 3.5s1 3:a446366d:::smithi10729407-11:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:33.507 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.900556+0000", 2020-06-12T02:38:33.507 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 1.4663416840000001, 2020-06-12T02:38:33.507 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.047233896999999997, 2020-06-12T02:38:33.508 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:33.508 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:33.508 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:33.508 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:33.508 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:33.508 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14915 2020-06-12T02:38:33.509 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.509 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:33.509 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.509 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:33.509 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900556+0000", 2020-06-12T02:38:33.510 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:33.510 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.511 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.511 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:33.511 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900556+0000", 2020-06-12T02:38:33.511 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3131000000000001e-05 2020-06-12T02:38:33.512 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.512 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.512 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:33.512 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900639+0000", 2020-06-12T02:38:33.513 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5459999999999992e-06 2020-06-12T02:38:33.513 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.513 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.513 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:33.513 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900647+0000", 2020-06-12T02:38:33.514 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5846e-05 2020-06-12T02:38:33.514 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.514 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.514 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:33.514 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900663+0000", 2020-06-12T02:38:33.514 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6760000000000001e-06 2020-06-12T02:38:33.515 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.515 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.515 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:33.515 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900670+0000", 2020-06-12T02:38:33.515 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2813e-05 2020-06-12T02:38:33.516 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.516 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.516 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:33.516 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900693+0000", 2020-06-12T02:38:33.516 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8223999999999998e-05 2020-06-12T02:38:33.516 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.517 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.517 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:33.517 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900741+0000", 2020-06-12T02:38:33.517 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.041466733999999998 2020-06-12T02:38:33.517 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.518 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.518 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:33.518 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.942208+0000", 2020-06-12T02:38:33.518 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010308 2020-06-12T02:38:33.518 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.518 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.519 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:33.519 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.942311+0000", 2020-06-12T02:38:33.519 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00032335099999999999 2020-06-12T02:38:33.519 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.519 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.520 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:33.520 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.942634+0000", 2020-06-12T02:38:33.520 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043047110000000001 2020-06-12T02:38:33.520 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.520 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.520 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:33.521 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.946939+0000", 2020-06-12T02:38:33.521 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6283999999999999e-05 2020-06-12T02:38:33.521 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.521 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.521 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:33.522 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.946985+0000", 2020-06-12T02:38:33.522 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000804501 2020-06-12T02:38:33.522 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.522 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.522 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:33.522 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.947790+0000", 2020-06-12T02:38:33.523 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.047233896999999997 2020-06-12T02:38:33.523 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:33.523 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:33.523 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:33.523 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.524 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.524 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14916 3.5s1 3:a446366d:::smithi10729407-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e59)", 2020-06-12T02:38:33.524 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.900719+0000", 2020-06-12T02:38:33.524 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 1.4661781030000001, 2020-06-12T02:38:33.524 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049202672000000003, 2020-06-12T02:38:33.524 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:33.525 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:33.525 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:33.525 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:33.525 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:33.525 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14916 2020-06-12T02:38:33.526 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.526 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:33.526 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.526 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:33.526 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900719+0000", 2020-06-12T02:38:33.527 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:33.527 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.527 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.527 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:33.527 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900719+0000", 2020-06-12T02:38:33.527 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2898000000000002e-05 2020-06-12T02:38:33.528 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.528 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.528 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:33.528 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900752+0000", 2020-06-12T02:38:33.528 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8879999999999999e-06 2020-06-12T02:38:33.529 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.529 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.529 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:33.529 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900756+0000", 2020-06-12T02:38:33.529 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.011e-06 2020-06-12T02:38:33.529 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.530 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.530 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:33.530 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900764+0000", 2020-06-12T02:38:33.530 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2880000000000001e-06 2020-06-12T02:38:33.530 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.531 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.531 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:33.531 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900767+0000", 2020-06-12T02:38:33.531 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.145e-05 2020-06-12T02:38:33.531 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.531 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.532 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:33.532 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900789+0000", 2020-06-12T02:38:33.532 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7469000000000002e-05 2020-06-12T02:38:33.532 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.532 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.533 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:33.533 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900836+0000", 2020-06-12T02:38:33.533 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042561179999999997 2020-06-12T02:38:33.533 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.533 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.533 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:33.534 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.943397+0000", 2020-06-12T02:38:33.534 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000106377 2020-06-12T02:38:33.534 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.534 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.534 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:33.535 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.943504+0000", 2020-06-12T02:38:33.535 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043186759999999996 2020-06-12T02:38:33.535 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.535 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.535 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:33.535 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.947823+0000", 2020-06-12T02:38:33.536 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4439999999999997e-05 2020-06-12T02:38:33.536 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.536 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.536 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:33.536 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.947917+0000", 2020-06-12T02:38:33.537 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0020049949999999999 2020-06-12T02:38:33.537 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.537 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.537 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:33.537 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.949922+0000", 2020-06-12T02:38:33.538 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049202672000000003 2020-06-12T02:38:33.538 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:33.538 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:33.538 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:33.538 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.538 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.539 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15220 3.7s0 3:e653d00f:::smithi10729407-18:head [append 0~532480 in=532480b] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:33.539 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.817582+0000", 2020-06-12T02:38:33.539 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.54931571000000001, 2020-06-12T02:38:33.539 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.084332243000000001, 2020-06-12T02:38:33.539 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:33.540 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:33.540 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:33.540 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:33.540 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:33.540 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15220 2020-06-12T02:38:33.540 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.541 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:33.541 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.541 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:33.541 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.817582+0000", 2020-06-12T02:38:33.541 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:33.542 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.542 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.542 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:33.542 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.817582+0000", 2020-06-12T02:38:33.542 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014160799999999999 2020-06-12T02:38:33.542 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.543 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.543 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:33.543 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.817723+0000", 2020-06-12T02:38:33.543 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6889999999999993e-06 2020-06-12T02:38:33.543 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.544 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.544 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:33.544 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.817732+0000", 2020-06-12T02:38:33.544 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2214e-05 2020-06-12T02:38:33.544 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.544 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.545 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:33.545 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.817744+0000", 2020-06-12T02:38:33.545 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9570000000000002e-06 2020-06-12T02:38:33.545 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.545 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.546 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:33.546 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.817752+0000", 2020-06-12T02:38:33.546 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4947e-05 2020-06-12T02:38:33.546 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.546 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.546 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:33.547 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.817817+0000", 2020-06-12T02:38:33.547 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012504899999999999 2020-06-12T02:38:33.547 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.547 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.548 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:33.548 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.817942+0000", 2020-06-12T02:38:33.549 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00081927200000000001 2020-06-12T02:38:33.549 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.549 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.549 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:33.549 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.818761+0000", 2020-06-12T02:38:33.550 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.082918138000000002 2020-06-12T02:38:33.550 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.550 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.550 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:33.550 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901680+0000", 2020-06-12T02:38:33.550 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000150111 2020-06-12T02:38:33.551 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.551 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.551 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:33.551 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901830+0000", 2020-06-12T02:38:33.551 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4258000000000005e-05 2020-06-12T02:38:33.552 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.552 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.552 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:33.552 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901914+0000", 2020-06-12T02:38:33.552 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.084332243000000001 2020-06-12T02:38:33.552 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:33.553 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:33.553 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:33.553 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.553 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.553 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15221 3.7s0 3:e653d00f:::smithi10729407-18:head [append 0~417792 in=417792b] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:33.554 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.819933+0000", 2020-06-12T02:38:33.554 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.54696414599999998, 2020-06-12T02:38:33.554 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091280863000000004, 2020-06-12T02:38:33.554 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:33.554 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:33.555 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:33.555 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:33.555 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:33.555 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15221 2020-06-12T02:38:33.555 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.555 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:33.556 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.556 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:33.556 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.819933+0000", 2020-06-12T02:38:33.556 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:33.556 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.557 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.557 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:33.557 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.819933+0000", 2020-06-12T02:38:33.557 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000302949 2020-06-12T02:38:33.557 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.557 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.558 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:33.558 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.820236+0000", 2020-06-12T02:38:33.558 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0984e-05 2020-06-12T02:38:33.558 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.558 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.559 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:33.559 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.820247+0000", 2020-06-12T02:38:33.559 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4894e-05 2020-06-12T02:38:33.559 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.560 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.560 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:33.560 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.820262+0000", 2020-06-12T02:38:33.560 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2409999999999998e-06 2020-06-12T02:38:33.560 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.560 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.561 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:33.561 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.820269+0000", 2020-06-12T02:38:33.561 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0024275859999999998 2020-06-12T02:38:33.561 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.561 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.562 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:33.562 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822697+0000", 2020-06-12T02:38:33.562 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011567599999999999 2020-06-12T02:38:33.562 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.562 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.563 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:33.563 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822813+0000", 2020-06-12T02:38:33.563 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000483714 2020-06-12T02:38:33.563 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.563 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.564 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:33.564 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.823296+0000", 2020-06-12T02:38:33.564 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.087735321000000005 2020-06-12T02:38:33.564 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.564 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.564 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:33.565 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.911032+0000", 2020-06-12T02:38:33.565 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3463999999999998e-05 2020-06-12T02:38:33.565 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.565 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.565 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:33.566 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.911125+0000", 2020-06-12T02:38:33.566 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9034000000000002e-05 2020-06-12T02:38:33.567 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.567 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.567 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:33.567 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.911214+0000", 2020-06-12T02:38:33.568 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091280863000000004 2020-06-12T02:38:33.569 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:33.569 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:33.569 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:33.570 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.570 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.570 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 59/37 ECSubWriteReply(tid=7875, last_complete=59'1325, committed=1, applied=1))", 2020-06-12T02:38:33.571 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.822130+0000", 2020-06-12T02:38:33.571 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.54476691700000002, 2020-06-12T02:38:33.571 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.079550689999999993, 2020-06-12T02:38:33.572 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:33.572 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:33.572 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:33.573 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.573 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:33.574 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822130+0000", 2020-06-12T02:38:33.574 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:33.574 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.575 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.575 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:33.575 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822130+0000", 2020-06-12T02:38:33.575 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8309e-05 2020-06-12T02:38:33.575 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.576 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.576 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:33.576 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822189+0000", 2020-06-12T02:38:33.576 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6299999999999993e-06 2020-06-12T02:38:33.577 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.577 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.577 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:33.577 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822198+0000", 2020-06-12T02:38:33.578 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4632e-05 2020-06-12T02:38:33.578 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.578 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.578 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:33.579 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822213+0000", 2020-06-12T02:38:33.579 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8600000000000004e-06 2020-06-12T02:38:33.579 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.579 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.579 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:33.580 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822220+0000", 2020-06-12T02:38:33.582 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.079445632000000002 2020-06-12T02:38:33.582 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.582 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.582 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:33.583 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901666+0000", 2020-06-12T02:38:33.583 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5627000000000001e-05 2020-06-12T02:38:33.583 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.583 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.583 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:33.584 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901681+0000", 2020-06-12T02:38:33.584 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.079550689999999993 2020-06-12T02:38:33.584 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:33.584 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:33.584 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:33.585 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.585 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.585 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 59/37 ECSubWriteReply(tid=7875, last_complete=59'1325, committed=1, applied=1))", 2020-06-12T02:38:33.585 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.822633+0000", 2020-06-12T02:38:33.585 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.54426456599999995, 2020-06-12T02:38:33.586 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.079294854999999997, 2020-06-12T02:38:33.586 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:33.586 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:33.586 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:33.586 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.586 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:33.587 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822633+0000", 2020-06-12T02:38:33.587 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:33.587 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.587 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.587 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:33.588 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822633+0000", 2020-06-12T02:38:33.588 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9357000000000001e-05 2020-06-12T02:38:33.588 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.588 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.588 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:33.589 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822692+0000", 2020-06-12T02:38:33.589 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5350000000000004e-06 2020-06-12T02:38:33.589 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.589 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.589 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:33.589 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822701+0000", 2020-06-12T02:38:33.590 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.243e-05 2020-06-12T02:38:33.590 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.590 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.590 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:33.590 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822713+0000", 2020-06-12T02:38:33.591 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.857e-06 2020-06-12T02:38:33.591 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.591 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.591 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:33.591 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822719+0000", 2020-06-12T02:38:33.591 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.079006642000000002 2020-06-12T02:38:33.592 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.592 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.592 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:33.592 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901726+0000", 2020-06-12T02:38:33.592 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000202034 2020-06-12T02:38:33.593 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.593 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.593 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:33.593 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901928+0000", 2020-06-12T02:38:33.593 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.079294854999999997 2020-06-12T02:38:33.594 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:33.594 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:33.594 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:33.594 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.594 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.594 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15222 3.7s0 3:e653d00f:::smithi10729407-18:head [append 0~524288] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:33.595 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.822885+0000", 2020-06-12T02:38:33.595 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.54401211999999999, 2020-06-12T02:38:33.595 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.088458414999999999, 2020-06-12T02:38:33.595 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:33.595 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:33.596 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:33.596 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:33.596 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:33.596 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15222 2020-06-12T02:38:33.596 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.596 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:33.597 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.597 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:33.597 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822885+0000", 2020-06-12T02:38:33.597 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:33.597 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.598 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.598 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:33.598 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822885+0000", 2020-06-12T02:38:33.598 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019930499999999999 2020-06-12T02:38:33.598 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.599 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.599 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:33.599 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.823085+0000", 2020-06-12T02:38:33.599 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4039999999999995e-06 2020-06-12T02:38:33.599 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.599 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.600 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:33.600 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.823093+0000", 2020-06-12T02:38:33.600 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5169e-05 2020-06-12T02:38:33.600 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.600 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.601 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:33.601 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.823108+0000", 2020-06-12T02:38:33.601 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6379999999999998e-06 2020-06-12T02:38:33.601 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.601 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.601 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:33.602 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.823115+0000", 2020-06-12T02:38:33.602 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.078878105000000004 2020-06-12T02:38:33.602 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.602 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.602 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:33.603 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901993+0000", 2020-06-12T02:38:33.603 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000116167 2020-06-12T02:38:33.603 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.603 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.603 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:33.603 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.902109+0000", 2020-06-12T02:38:33.604 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00071338200000000004 2020-06-12T02:38:33.605 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.605 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.605 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:33.605 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.902822+0000", 2020-06-12T02:38:33.605 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0083967999999999994 2020-06-12T02:38:33.606 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.606 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.606 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:33.606 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.911219+0000", 2020-06-12T02:38:33.606 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8220000000000002e-05 2020-06-12T02:38:33.607 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.607 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.607 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:33.607 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.911267+0000", 2020-06-12T02:38:33.607 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6224999999999996e-05 2020-06-12T02:38:33.608 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.608 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.608 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:33.608 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.911344+0000", 2020-06-12T02:38:33.608 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.088458414999999999 2020-06-12T02:38:33.609 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:33.609 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:33.609 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:33.609 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.609 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.610 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15223 3.7s0 3:e653d00f:::smithi10729407-18:head [append 0~262144] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:33.610 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.824390+0000", 2020-06-12T02:38:33.610 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.54250694899999996, 2020-06-12T02:38:33.610 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.088985642000000004, 2020-06-12T02:38:33.610 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:33.611 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:33.611 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:33.611 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:33.611 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:33.611 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15223 2020-06-12T02:38:33.611 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.612 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:33.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.612 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:33.612 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824390+0000", 2020-06-12T02:38:33.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:33.613 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.613 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.613 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:33.613 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824390+0000", 2020-06-12T02:38:33.613 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014672400000000001 2020-06-12T02:38:33.613 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.614 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.614 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:33.614 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824537+0000", 2020-06-12T02:38:33.614 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2020000000000004e-06 2020-06-12T02:38:33.614 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.615 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.615 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:33.615 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824545+0000", 2020-06-12T02:38:33.615 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5106000000000001e-05 2020-06-12T02:38:33.615 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.615 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.616 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:33.616 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824560+0000", 2020-06-12T02:38:33.616 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2539999999999998e-06 2020-06-12T02:38:33.616 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.616 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.617 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:33.617 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824567+0000", 2020-06-12T02:38:33.617 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.082220249999999995 2020-06-12T02:38:33.617 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.617 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.618 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:33.618 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.906787+0000", 2020-06-12T02:38:33.618 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109362 2020-06-12T02:38:33.618 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.618 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.618 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:33.619 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.906896+0000", 2020-06-12T02:38:33.619 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00074103899999999998 2020-06-12T02:38:33.619 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.619 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.619 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:33.620 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.907637+0000", 2020-06-12T02:38:33.620 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0054187489999999996 2020-06-12T02:38:33.620 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.620 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.620 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:33.620 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.913056+0000", 2020-06-12T02:38:33.621 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00026505799999999998 2020-06-12T02:38:33.621 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.621 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.621 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:33.621 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.913321+0000", 2020-06-12T02:38:33.622 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4898000000000002e-05 2020-06-12T02:38:33.622 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.622 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.622 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:33.623 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.913376+0000", 2020-06-12T02:38:33.623 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.088985642000000004 2020-06-12T02:38:33.623 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:33.623 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:33.623 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:33.624 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.624 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.624 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15224 3.7s0 3:e653d00f:::smithi10729407-18:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:33.624 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.824617+0000", 2020-06-12T02:38:33.624 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.54228010400000004, 2020-06-12T02:38:33.624 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.09133223, 2020-06-12T02:38:33.625 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:33.625 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:33.625 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:33.625 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:33.625 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:33.626 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15224 2020-06-12T02:38:33.626 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.626 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:33.631 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.631 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:33.631 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824617+0000", 2020-06-12T02:38:33.631 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:33.632 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.632 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.632 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:33.632 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824617+0000", 2020-06-12T02:38:33.632 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3214999999999996e-05 2020-06-12T02:38:33.632 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.633 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.633 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:33.633 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824691+0000", 2020-06-12T02:38:33.633 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4949999999999996e-06 2020-06-12T02:38:33.633 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.634 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.634 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:33.634 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824697+0000", 2020-06-12T02:38:33.634 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1935e-05 2020-06-12T02:38:33.634 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.635 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.635 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:33.635 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824709+0000", 2020-06-12T02:38:33.635 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1459999999999998e-06 2020-06-12T02:38:33.635 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.635 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.636 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:33.636 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824713+0000", 2020-06-12T02:38:33.636 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.086691905 2020-06-12T02:38:33.636 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.636 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.637 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:33.637 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.911405+0000", 2020-06-12T02:38:33.637 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3385000000000003e-05 2020-06-12T02:38:33.638 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.638 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.638 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:33.639 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.911478+0000", 2020-06-12T02:38:33.639 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00033631999999999998 2020-06-12T02:38:33.639 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.639 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.639 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:33.639 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.911815+0000", 2020-06-12T02:38:33.640 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0037284330000000002 2020-06-12T02:38:33.640 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.640 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.640 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:33.640 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.915543+0000", 2020-06-12T02:38:33.641 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035071100000000002 2020-06-12T02:38:33.641 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.641 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.641 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:33.641 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.915894+0000", 2020-06-12T02:38:33.642 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5685e-05 2020-06-12T02:38:33.642 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.642 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.642 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:33.642 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.915950+0000", 2020-06-12T02:38:33.642 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.09133223 2020-06-12T02:38:33.643 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:33.644 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:33.644 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:33.644 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.644 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.645 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15225 3.7s0 3:e653d00f:::smithi10729407-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e59)", 2020-06-12T02:38:33.645 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.824754+0000", 2020-06-12T02:38:33.645 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.54214357499999999, 2020-06-12T02:38:33.645 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093329925999999994, 2020-06-12T02:38:33.645 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:33.646 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:33.646 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:33.646 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:33.646 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:33.646 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15225 2020-06-12T02:38:33.647 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.647 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:33.647 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.647 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:33.647 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824754+0000", 2020-06-12T02:38:33.647 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:33.648 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.648 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.648 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:33.648 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824754+0000", 2020-06-12T02:38:33.648 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3890999999999997e-05 2020-06-12T02:38:33.649 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.649 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.649 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:33.649 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824798+0000", 2020-06-12T02:38:33.649 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6840000000000001e-06 2020-06-12T02:38:33.650 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.650 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.650 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:33.650 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824803+0000", 2020-06-12T02:38:33.650 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1881e-05 2020-06-12T02:38:33.650 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.651 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.651 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:33.651 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824815+0000", 2020-06-12T02:38:33.651 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1400000000000002e-06 2020-06-12T02:38:33.651 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.652 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.652 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:33.652 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824819+0000", 2020-06-12T02:38:33.652 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.087909733000000004 2020-06-12T02:38:33.652 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.652 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.653 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:33.653 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.912729+0000", 2020-06-12T02:38:33.653 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2892999999999996e-05 2020-06-12T02:38:33.653 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.653 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.654 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:33.654 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.912792+0000", 2020-06-12T02:38:33.654 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0040055159999999998 2020-06-12T02:38:33.654 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.654 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.654 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:33.655 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.916798+0000", 2020-06-12T02:38:33.655 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9858000000000006e-05 2020-06-12T02:38:33.655 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.655 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.655 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:33.656 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.916887+0000", 2020-06-12T02:38:33.656 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00119633 2020-06-12T02:38:33.656 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.656 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.656 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:33.656 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.918084+0000", 2020-06-12T02:38:33.657 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093329925999999994 2020-06-12T02:38:33.658 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:33.658 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:33.658 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:33.658 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.658 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.659 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15237 3.7s0 3:ea271007:::smithi10729407-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e59)", 2020-06-12T02:38:33.659 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.931681+0000", 2020-06-12T02:38:33.659 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.43521637800000001, 2020-06-12T02:38:33.659 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025037660999999999, 2020-06-12T02:38:33.659 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:33.660 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:33.660 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:33.660 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:33.660 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:33.660 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15237 2020-06-12T02:38:33.661 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.661 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:33.661 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.661 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:33.661 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.931681+0000", 2020-06-12T02:38:33.661 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:33.662 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.662 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.662 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:33.662 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.931681+0000", 2020-06-12T02:38:33.662 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.4977000000000002e-05 2020-06-12T02:38:33.663 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.663 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.664 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:33.664 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.931706+0000", 2020-06-12T02:38:33.664 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0599999999999999e-06 2020-06-12T02:38:33.664 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.664 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.665 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:33.665 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.931709+0000", 2020-06-12T02:38:33.665 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3659999999999997e-06 2020-06-12T02:38:33.665 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.665 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.666 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:33.666 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.931715+0000", 2020-06-12T02:38:33.666 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.4159999999999998e-06 2020-06-12T02:38:33.666 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.666 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.666 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:33.667 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.931718+0000", 2020-06-12T02:38:33.667 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010323900000000001 2020-06-12T02:38:33.667 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.667 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.667 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:33.668 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.931821+0000", 2020-06-12T02:38:33.668 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8529000000000002e-05 2020-06-12T02:38:33.668 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.668 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.668 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:33.669 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.931870+0000", 2020-06-12T02:38:33.669 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013885599 2020-06-12T02:38:33.669 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.669 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.669 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:33.669 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.945755+0000", 2020-06-12T02:38:33.670 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2727999999999999e-05 2020-06-12T02:38:33.670 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.670 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.670 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:33.670 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.945828+0000", 2020-06-12T02:38:33.671 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0076938550000000003 2020-06-12T02:38:33.671 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.671 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.671 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:33.671 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.953522+0000", 2020-06-12T02:38:33.671 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000116128 2020-06-12T02:38:33.672 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.672 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.672 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:33.672 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.953638+0000", 2020-06-12T02:38:33.672 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0030807640000000002 2020-06-12T02:38:33.673 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:33.673 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:33.673 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:33.673 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.956719+0000", 2020-06-12T02:38:33.673 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025037660999999999 2020-06-12T02:38:33.674 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:33.674 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:33.674 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:33.674 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:33.674 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:33.674 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:33.675 INFO:tasks.rados.rados.0.smithi107.stdout:7165: writing smithi10729407-18 from 5283840 to 6070272 tid 1 2020-06-12T02:38:33.689 INFO:tasks.rados.rados.0.smithi107.stdout:7165: writing smithi10729407-18 from 6070272 to 6627328 tid 2 2020-06-12T02:38:33.689 INFO:tasks.rados.rados.0.smithi107.stdout:7165: writing smithi10729407-18 from 6627328 to 7192576 tid 3 2020-06-12T02:38:33.690 INFO:tasks.rados.rados.0.smithi107.stdout:7165: writing smithi10729407-18 from 7192576 to 7675904 tid 4 2020-06-12T02:38:33.690 INFO:tasks.rados.rados.0.smithi107.stdout:7166: delete oid 16 current snap is 0 2020-06-12T02:38:33.690 INFO:tasks.rados.rados.0.smithi107.stdout:7165: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:33.690 INFO:tasks.rados.rados.0.smithi107.stdout:7166: done (1 left) 2020-06-12T02:38:33.691 INFO:tasks.rados.rados.0.smithi107.stdout:7167: delete oid 1 current snap is 0 2020-06-12T02:38:33.691 INFO:tasks.rados.rados.0.smithi107.stdout:7167: done (1 left) 2020-06-12T02:38:33.691 INFO:tasks.rados.rados.0.smithi107.stdout:7168: delete oid 14 current snap is 0 2020-06-12T02:38:33.691 INFO:tasks.rados.rados.0.smithi107.stdout:7168: done (1 left) 2020-06-12T02:38:33.691 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:38:33.692 INFO:tasks.rados.rados.0.smithi107.stdout:7169: seq_num 7168 ranges {1908736=57344} 2020-06-12T02:38:33.692 INFO:tasks.rados.rados.0.smithi107.stdout:7169: writing smithi10729407-3 from 1908736 to 1966080 tid 1 2020-06-12T02:38:33.692 INFO:tasks.rados.rados.0.smithi107.stdout:7170: delete oid 14 current snap is 0 2020-06-12T02:38:33.692 INFO:tasks.rados.rados.0.smithi107.stdout:7165: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:33.692 INFO:tasks.rados.rados.0.smithi107.stdout:7165: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:33.693 INFO:tasks.rados.rados.0.smithi107.stdout:7165: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:33.693 INFO:tasks.rados.rados.0.smithi107.stdout:7165: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:33.693 INFO:tasks.rados.rados.0.smithi107.stdout:7165: finishing write tid 7 to smithi10729407-18 2020-06-12T02:38:33.693 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1351 (ObjNum 7164 snap 0 seq_num 7164) dirty exists 2020-06-12T02:38:33.693 INFO:tasks.rados.rados.0.smithi107.stdout:7165: left oid 18 (ObjNum 7164 snap 0 seq_num 7164) 2020-06-12T02:38:33.693 INFO:tasks.rados.rados.0.smithi107.stdout:7165: done (2 left) 2020-06-12T02:38:33.694 INFO:tasks.rados.rados.0.smithi107.stdout:7170: done (1 left) 2020-06-12T02:38:33.694 INFO:tasks.rados.rados.0.smithi107.stdout:7171: delete oid 15 current snap is 0 2020-06-12T02:38:33.694 INFO:tasks.rados.rados.0.smithi107.stdout:7169: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:33.694 INFO:tasks.rados.rados.0.smithi107.stdout:7169: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:33.694 INFO:tasks.rados.rados.0.smithi107.stdout:7169: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:33.695 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1275 (ObjNum 7168 snap 0 seq_num 7168) dirty exists 2020-06-12T02:38:33.695 INFO:tasks.rados.rados.0.smithi107.stdout:7169: left oid 3 (ObjNum 7168 snap 0 seq_num 7168) 2020-06-12T02:38:33.695 INFO:tasks.rados.rados.0.smithi107.stdout:7169: done (1 left) 2020-06-12T02:38:33.695 INFO:tasks.rados.rados.0.smithi107.stdout:7171: done (0 left) 2020-06-12T02:38:33.695 INFO:tasks.rados.rados.0.smithi107.stdout:7172: delete oid 9 current snap is 0 2020-06-12T02:38:33.695 INFO:tasks.rados.rados.0.smithi107.stdout:7172: done (0 left) 2020-06-12T02:38:33.696 INFO:tasks.rados.rados.0.smithi107.stdout:7173: delete oid 17 current snap is 0 2020-06-12T02:38:33.696 INFO:tasks.rados.rados.0.smithi107.stdout:7173: done (0 left) 2020-06-12T02:38:33.696 INFO:tasks.rados.rados.0.smithi107.stdout:7174: delete oid 16 current snap is 0 2020-06-12T02:38:33.696 INFO:tasks.rados.rados.0.smithi107.stdout:7174: done (0 left) 2020-06-12T02:38:33.696 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:33.697 INFO:tasks.rados.rados.0.smithi107.stdout:7175: seq_num 7174 ranges {0=155648} 2020-06-12T02:38:33.697 INFO:tasks.rados.rados.0.smithi107.stdout:7175: writing smithi10729407-12 from 0 to 155648 tid 1 2020-06-12T02:38:33.697 INFO:tasks.rados.rados.0.smithi107.stdout:7176: delete oid 3 current snap is 0 2020-06-12T02:38:33.697 INFO:tasks.rados.rados.0.smithi107.stdout:7176: done (1 left) 2020-06-12T02:38:33.697 INFO:tasks.rados.rados.0.smithi107.stdout:7177: delete oid 10 current snap is 0 2020-06-12T02:38:33.697 INFO:tasks.rados.rados.0.smithi107.stdout:7175: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:33.698 INFO:tasks.rados.rados.0.smithi107.stdout:7177: done (1 left) 2020-06-12T02:38:33.698 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:33.698 INFO:tasks.rados.rados.0.smithi107.stdout:7178: seq_num 7177 ranges {0=204800} 2020-06-12T02:38:33.698 INFO:tasks.rados.rados.0.smithi107.stdout:7178: writing smithi10729407-13 from 0 to 204800 tid 1 2020-06-12T02:38:33.698 INFO:tasks.rados.rados.0.smithi107.stdout:7179: delete oid 8 current snap is 0 2020-06-12T02:38:33.698 INFO:tasks.rados.rados.0.smithi107.stdout:7175: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:33.699 INFO:tasks.rados.rados.0.smithi107.stdout:7175: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:33.699 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1327 (ObjNum 7174 snap 0 seq_num 7174) dirty exists 2020-06-12T02:38:33.699 INFO:tasks.rados.rados.0.smithi107.stdout:7175: left oid 12 (ObjNum 7174 snap 0 seq_num 7174) 2020-06-12T02:38:33.699 INFO:tasks.rados.rados.0.smithi107.stdout:7175: done (2 left) 2020-06-12T02:38:33.699 INFO:tasks.rados.rados.0.smithi107.stdout:7179: done (1 left) 2020-06-12T02:38:33.700 INFO:tasks.rados.rados.0.smithi107.stdout:7180: delete oid 18 current snap is 0 2020-06-12T02:38:33.700 INFO:tasks.rados.rados.0.smithi107.stdout:7178: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:33.700 INFO:tasks.rados.rados.0.smithi107.stdout:7180: done (1 left) 2020-06-12T02:38:33.700 INFO:tasks.rados.rados.0.smithi107.stdout:7181: delete oid 9 current snap is 0 2020-06-12T02:38:33.701 INFO:tasks.rados.rados.0.smithi107.stdout:7178: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:33.701 INFO:tasks.rados.rados.0.smithi107.stdout:7178: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:33.701 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1329 (ObjNum 7177 snap 0 seq_num 7177) dirty exists 2020-06-12T02:38:33.701 INFO:tasks.rados.rados.0.smithi107.stdout:7178: left oid 13 (ObjNum 7177 snap 0 seq_num 7177) 2020-06-12T02:38:33.701 INFO:tasks.rados.rados.0.smithi107.stdout:7178: done (1 left) 2020-06-12T02:38:33.701 INFO:tasks.rados.rados.0.smithi107.stdout:7181: done (0 left) 2020-06-12T02:38:33.702 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:33.702 INFO:tasks.rados.rados.0.smithi107.stdout:7182: seq_num 7181 ranges {2326528=270336} 2020-06-12T02:38:33.702 INFO:tasks.rados.rados.0.smithi107.stdout:7182: writing smithi10729407-7 from 2326528 to 2596864 tid 1 2020-06-12T02:38:33.702 INFO:tasks.rados.rados.0.smithi107.stdout:7183: delete oid 2 current snap is 0 2020-06-12T02:38:33.703 INFO:tasks.rados.rados.0.smithi107.stdout:7183: done (1 left) 2020-06-12T02:38:33.703 INFO:tasks.rados.rados.0.smithi107.stdout:7184: delete oid 3 current snap is 0 2020-06-12T02:38:33.703 INFO:tasks.rados.rados.0.smithi107.stdout:7182: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:33.703 INFO:tasks.rados.rados.0.smithi107.stdout:7184: done (1 left) 2020-06-12T02:38:33.703 INFO:tasks.rados.rados.0.smithi107.stdout:7185: delete oid 9 current snap is 0 2020-06-12T02:38:33.704 INFO:tasks.rados.rados.0.smithi107.stdout:7182: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:33.704 INFO:tasks.rados.rados.0.smithi107.stdout:7182: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:33.704 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1331 (ObjNum 7181 snap 0 seq_num 7181) dirty exists 2020-06-12T02:38:33.704 INFO:tasks.rados.rados.0.smithi107.stdout:7182: left oid 7 (ObjNum 7181 snap 0 seq_num 7181) 2020-06-12T02:38:33.704 INFO:tasks.rados.rados.0.smithi107.stdout:7182: done (1 left) 2020-06-12T02:38:33.705 INFO:tasks.rados.rados.0.smithi107.stdout:7185: done (0 left) 2020-06-12T02:38:33.705 INFO:tasks.rados.rados.0.smithi107.stdout:7186: delete oid 15 current snap is 0 2020-06-12T02:38:33.705 INFO:tasks.rados.rados.0.smithi107.stdout:7186: done (0 left) 2020-06-12T02:38:33.705 INFO:tasks.rados.rados.0.smithi107.stdout:7187: delete oid 12 current snap is 0 2020-06-12T02:38:33.706 INFO:tasks.rados.rados.0.smithi107.stdout:7187: done (0 left) 2020-06-12T02:38:33.706 INFO:tasks.rados.rados.0.smithi107.stdout:7188: delete oid 7 current snap is 0 2020-06-12T02:38:33.706 INFO:tasks.rados.rados.0.smithi107.stdout:7188: done (0 left) 2020-06-12T02:38:33.706 INFO:tasks.rados.rados.0.smithi107.stdout:7189: delete oid 4 current snap is 0 2020-06-12T02:38:33.706 INFO:tasks.rados.rados.0.smithi107.stdout:7189: done (0 left) 2020-06-12T02:38:33.707 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:33.707 INFO:tasks.rados.rados.0.smithi107.stdout:7190: seq_num 7189 ranges {0=409600} 2020-06-12T02:38:33.707 INFO:tasks.rados.rados.0.smithi107.stdout:7190: writing smithi10729407-3 from 0 to 409600 tid 1 2020-06-12T02:38:33.707 INFO:tasks.rados.rados.0.smithi107.stdout:7191: delete oid 13 current snap is 0 2020-06-12T02:38:33.707 INFO:tasks.rados.rados.0.smithi107.stdout:7191: done (1 left) 2020-06-12T02:38:33.708 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:33.708 INFO:tasks.rados.rados.0.smithi107.stdout:7192: seq_num 7191 ranges {0=442368} 2020-06-12T02:38:33.708 INFO:tasks.rados.rados.0.smithi107.stdout:7192: writing smithi10729407-15 from 0 to 442368 tid 1 2020-06-12T02:38:33.708 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:38:33.708 INFO:tasks.rados.rados.0.smithi107.stdout:7190: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:33.708 INFO:tasks.rados.rados.0.smithi107.stdout:7193: seq_num 7192 ranges {0=458752} 2020-06-12T02:38:33.709 INFO:tasks.rados.rados.0.smithi107.stdout:7193: writing smithi10729407-10 from 0 to 458752 tid 1 2020-06-12T02:38:33.709 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:33.709 INFO:tasks.rados.rados.0.smithi107.stdout:7194: seq_num 7193 ranges {0=475136} 2020-06-12T02:38:33.709 INFO:tasks.rados.rados.0.smithi107.stdout:7194: writing smithi10729407-19 from 0 to 475136 tid 1 2020-06-12T02:38:33.709 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:33.710 INFO:tasks.rados.rados.0.smithi107.stdout:7195: seq_num 7194 ranges {0=491520} 2020-06-12T02:38:33.710 INFO:tasks.rados.rados.0.smithi107.stdout:7195: writing smithi10729407-13 from 0 to 491520 tid 1 2020-06-12T02:38:33.710 INFO:tasks.rados.rados.0.smithi107.stdout:7196: delete oid 17 current snap is 0 2020-06-12T02:38:33.710 INFO:tasks.rados.rados.0.smithi107.stdout:7190: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:33.710 INFO:tasks.rados.rados.0.smithi107.stdout:7193: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:33.710 INFO:tasks.rados.rados.0.smithi107.stdout:7192: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:33.711 INFO:tasks.rados.rados.0.smithi107.stdout:7190: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:33.711 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1280 (ObjNum 7189 snap 0 seq_num 7189) dirty exists 2020-06-12T02:38:33.711 INFO:tasks.rados.rados.0.smithi107.stdout:7190: left oid 3 (ObjNum 7189 snap 0 seq_num 7189) 2020-06-12T02:38:33.711 INFO:tasks.rados.rados.0.smithi107.stdout:7193: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:33.711 INFO:tasks.rados.rados.0.smithi107.stdout:7194: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:33.712 INFO:tasks.rados.rados.0.smithi107.stdout:7194: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:33.712 INFO:tasks.rados.rados.0.smithi107.stdout:7190: done (5 left) 2020-06-12T02:38:33.712 INFO:tasks.rados.rados.0.smithi107.stdout:7196: done (4 left) 2020-06-12T02:38:33.712 INFO:tasks.rados.rados.0.smithi107.stdout:7197: delete oid 9 current snap is 0 2020-06-12T02:38:33.712 INFO:tasks.rados.rados.0.smithi107.stdout:7193: finishing write tid 4 to smithi10729407-10 2020-06-12T02:38:33.713 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 488 (ObjNum 7192 snap 0 seq_num 7192) dirty exists 2020-06-12T02:38:33.713 INFO:tasks.rados.rados.0.smithi107.stdout:7193: left oid 10 (ObjNum 7192 snap 0 seq_num 7192) 2020-06-12T02:38:33.713 INFO:tasks.rados.rados.0.smithi107.stdout:7192: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:33.713 INFO:tasks.rados.rados.0.smithi107.stdout:7194: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:33.713 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1330 (ObjNum 7193 snap 0 seq_num 7193) dirty exists 2020-06-12T02:38:33.714 INFO:tasks.rados.rados.0.smithi107.stdout:7194: left oid 19 (ObjNum 7193 snap 0 seq_num 7193) 2020-06-12T02:38:33.714 INFO:tasks.rados.rados.0.smithi107.stdout:7193: done (4 left) 2020-06-12T02:38:33.714 INFO:tasks.rados.rados.0.smithi107.stdout:7194: done (3 left) 2020-06-12T02:38:33.714 INFO:tasks.rados.rados.0.smithi107.stdout:7197: done (2 left) 2020-06-12T02:38:33.714 INFO:tasks.rados.rados.0.smithi107.stdout:7198: delete oid 3 current snap is 0 2020-06-12T02:38:33.715 INFO:tasks.rados.rados.0.smithi107.stdout:7195: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:33.715 INFO:tasks.rados.rados.0.smithi107.stdout:7192: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:33.715 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1359 (ObjNum 7191 snap 0 seq_num 7191) dirty exists 2020-06-12T02:38:33.715 INFO:tasks.rados.rados.0.smithi107.stdout:7192: left oid 15 (ObjNum 7191 snap 0 seq_num 7191) 2020-06-12T02:38:33.715 INFO:tasks.rados.rados.0.smithi107.stdout:7195: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:33.716 INFO:tasks.rados.rados.0.smithi107.stdout:7195: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:33.716 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1335 (ObjNum 7194 snap 0 seq_num 7194) dirty exists 2020-06-12T02:38:33.716 INFO:tasks.rados.rados.0.smithi107.stdout:7195: left oid 13 (ObjNum 7194 snap 0 seq_num 7194) 2020-06-12T02:38:33.716 INFO:tasks.rados.rados.0.smithi107.stdout:7192: done (2 left) 2020-06-12T02:38:33.716 INFO:tasks.rados.rados.0.smithi107.stdout:7195: done (1 left) 2020-06-12T02:38:33.717 INFO:tasks.rados.rados.0.smithi107.stdout:7198: done (0 left) 2020-06-12T02:38:33.717 INFO:tasks.rados.rados.0.smithi107.stdout:7199: delete oid 16 current snap is 0 2020-06-12T02:38:33.717 INFO:tasks.rados.rados.0.smithi107.stdout:7199: done (0 left) 2020-06-12T02:38:33.717 INFO:tasks.rados.rados.0.smithi107.stdout:7200: delete oid 18 current snap is 0 2020-06-12T02:38:33.717 INFO:tasks.rados.rados.0.smithi107.stdout:7200: done (0 left) 2020-06-12T02:38:33.717 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:38:33.718 INFO:tasks.rados.rados.0.smithi107.stdout:7201: seq_num 7200 ranges {2310144=589824} 2020-06-12T02:38:33.718 INFO:tasks.rados.rados.0.smithi107.stdout:7201: writing smithi10729407-5 from 2310144 to 2899968 tid 1 2020-06-12T02:38:33.718 INFO:tasks.rados.rados.0.smithi107.stdout:7202: delete oid 3 current snap is 0 2020-06-12T02:38:33.718 INFO:tasks.rados.rados.0.smithi107.stdout:7202: done (1 left) 2020-06-12T02:38:33.719 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:38:33.719 INFO:tasks.rados.rados.0.smithi107.stdout:7203: seq_num 7202 ranges {0=622592} 2020-06-12T02:38:33.719 INFO:tasks.rados.rados.0.smithi107.stdout:7203: writing smithi10729407-7 from 0 to 622592 tid 1 2020-06-12T02:38:33.719 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:33.719 INFO:tasks.rados.rados.0.smithi107.stdout:7204: seq_num 7203 ranges {0=638976} 2020-06-12T02:38:33.720 INFO:tasks.rados.rados.0.smithi107.stdout:7204: writing smithi10729407-2 from 0 to 638976 tid 1 2020-06-12T02:38:33.720 INFO:tasks.rados.rados.0.smithi107.stdout:7205: delete oid 16 current snap is 0 2020-06-12T02:38:33.720 INFO:tasks.rados.rados.0.smithi107.stdout:7201: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:33.720 INFO:tasks.rados.rados.0.smithi107.stdout:7201: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:33.720 INFO:tasks.rados.rados.0.smithi107.stdout:7201: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:33.720 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 719 (ObjNum 7200 snap 0 seq_num 7200) dirty exists 2020-06-12T02:38:33.721 INFO:tasks.rados.rados.0.smithi107.stdout:7201: left oid 5 (ObjNum 7200 snap 0 seq_num 7200) 2020-06-12T02:38:33.721 INFO:tasks.rados.rados.0.smithi107.stdout:7203: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:33.721 INFO:tasks.rados.rados.0.smithi107.stdout:7201: done (3 left) 2020-06-12T02:38:33.721 INFO:tasks.rados.rados.0.smithi107.stdout:7205: done (2 left) 2020-06-12T02:38:33.721 INFO:tasks.rados.rados.0.smithi107.stdout:7206: delete oid 8 current snap is 0 2020-06-12T02:38:33.722 INFO:tasks.rados.rados.0.smithi107.stdout:7204: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:33.722 INFO:tasks.rados.rados.0.smithi107.stdout:7203: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:33.722 INFO:tasks.rados.rados.0.smithi107.stdout:7203: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:33.722 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1337 (ObjNum 7202 snap 0 seq_num 7202) dirty exists 2020-06-12T02:38:33.722 INFO:tasks.rados.rados.0.smithi107.stdout:7203: left oid 7 (ObjNum 7202 snap 0 seq_num 7202) 2020-06-12T02:38:33.722 INFO:tasks.rados.rados.0.smithi107.stdout:7204: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:33.723 INFO:tasks.rados.rados.0.smithi107.stdout:7203: done (2 left) 2020-06-12T02:38:33.723 INFO:tasks.rados.rados.0.smithi107.stdout:7206: done (1 left) 2020-06-12T02:38:33.723 INFO:tasks.rados.rados.0.smithi107.stdout:7207: delete oid 5 current snap is 0 2020-06-12T02:38:33.723 INFO:tasks.rados.rados.0.smithi107.stdout:7204: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:33.723 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 685 (ObjNum 7203 snap 0 seq_num 7203) dirty exists 2020-06-12T02:38:33.723 INFO:tasks.rados.rados.0.smithi107.stdout:7204: left oid 2 (ObjNum 7203 snap 0 seq_num 7203) 2020-06-12T02:38:33.724 INFO:tasks.rados.rados.0.smithi107.stdout:7204: done (1 left) 2020-06-12T02:38:33.724 INFO:tasks.rados.rados.0.smithi107.stdout:7207: done (0 left) 2020-06-12T02:38:33.724 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:33.724 INFO:tasks.rados.rados.0.smithi107.stdout:7208: seq_num 7207 ranges {0=712704} 2020-06-12T02:38:33.724 INFO:tasks.rados.rados.0.smithi107.stdout:7208: writing smithi10729407-18 from 0 to 712704 tid 1 2020-06-12T02:38:33.725 INFO:tasks.rados.rados.0.smithi107.stdout:7209: delete oid 19 current snap is 0 2020-06-12T02:38:33.725 INFO:tasks.rados.rados.0.smithi107.stdout:7209: done (1 left) 2020-06-12T02:38:33.725 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:33.725 INFO:tasks.rados.rados.0.smithi107.stdout:7210: seq_num 7209 ranges {0=745472} 2020-06-12T02:38:33.725 INFO:tasks.rados.rados.0.smithi107.stdout:7210: writing smithi10729407-17 from 0 to 745472 tid 1 2020-06-12T02:38:33.725 INFO:tasks.rados.rados.0.smithi107.stdout:7211: delete oid 4 current snap is 0 2020-06-12T02:38:33.726 INFO:tasks.rados.rados.0.smithi107.stdout:7208: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:33.726 INFO:tasks.rados.rados.0.smithi107.stdout:7208: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:33.726 INFO:tasks.rados.rados.0.smithi107.stdout:7210: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:33.726 INFO:tasks.rados.rados.0.smithi107.stdout:7208: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:33.726 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1359 (ObjNum 7207 snap 0 seq_num 7207) dirty exists 2020-06-12T02:38:33.727 INFO:tasks.rados.rados.0.smithi107.stdout:7208: left oid 18 (ObjNum 7207 snap 0 seq_num 7207) 2020-06-12T02:38:33.727 INFO:tasks.rados.rados.0.smithi107.stdout:7210: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:33.727 INFO:tasks.rados.rados.0.smithi107.stdout:7208: done (2 left) 2020-06-12T02:38:33.727 INFO:tasks.rados.rados.0.smithi107.stdout:7211: done (1 left) 2020-06-12T02:38:33.727 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:33.727 INFO:tasks.rados.rados.0.smithi107.stdout:7212: seq_num 7211 ranges {0=778240} 2020-06-12T02:38:33.728 INFO:tasks.rados.rados.0.smithi107.stdout:7212: writing smithi10729407-16 from 0 to 778240 tid 1 2020-06-12T02:38:33.728 INFO:tasks.rados.rados.0.smithi107.stdout:7213: delete oid 8 current snap is 0 2020-06-12T02:38:33.728 INFO:tasks.rados.rados.0.smithi107.stdout:7210: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:33.728 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1354 (ObjNum 7209 snap 0 seq_num 7209) dirty exists 2020-06-12T02:38:33.728 INFO:tasks.rados.rados.0.smithi107.stdout:7210: left oid 17 (ObjNum 7209 snap 0 seq_num 7209) 2020-06-12T02:38:33.729 INFO:tasks.rados.rados.0.smithi107.stdout:7210: done (2 left) 2020-06-12T02:38:33.729 INFO:tasks.rados.rados.0.smithi107.stdout:7213: done (1 left) 2020-06-12T02:38:33.729 INFO:tasks.rados.rados.0.smithi107.stdout:7214: delete oid 1 current snap is 0 2020-06-12T02:38:33.729 INFO:tasks.rados.rados.0.smithi107.stdout:7212: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:33.729 INFO:tasks.rados.rados.0.smithi107.stdout:7214: done (1 left) 2020-06-12T02:38:33.729 INFO:tasks.rados.rados.0.smithi107.stdout:7215: delete oid 13 current snap is 0 2020-06-12T02:38:33.730 INFO:tasks.rados.rados.0.smithi107.stdout:7215: done (1 left) 2020-06-12T02:38:33.730 INFO:tasks.rados.rados.0.smithi107.stdout:7216: delete oid 9 current snap is 0 2020-06-12T02:38:33.730 INFO:tasks.rados.rados.0.smithi107.stdout:7212: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:33.730 INFO:tasks.rados.rados.0.smithi107.stdout:7212: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:33.730 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1286 (ObjNum 7211 snap 0 seq_num 7211) dirty exists 2020-06-12T02:38:33.730 INFO:tasks.rados.rados.0.smithi107.stdout:7212: left oid 16 (ObjNum 7211 snap 0 seq_num 7211) 2020-06-12T02:38:33.731 INFO:tasks.rados.rados.0.smithi107.stdout:7212: done (1 left) 2020-06-12T02:38:33.731 INFO:tasks.rados.rados.0.smithi107.stdout:7216: done (0 left) 2020-06-12T02:38:33.731 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:33.731 INFO:tasks.rados.rados.0.smithi107.stdout:7217: seq_num 7216 ranges {712704=458752,1171456=401408} 2020-06-12T02:38:33.731 INFO:tasks.rados.rados.0.smithi107.stdout:7217: writing smithi10729407-18 from 712704 to 1171456 tid 1 2020-06-12T02:38:33.732 INFO:tasks.rados.rados.0.smithi107.stdout:7217: writing smithi10729407-18 from 1171456 to 1572864 tid 2 2020-06-12T02:38:33.732 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:33.732 INFO:tasks.rados.rados.0.smithi107.stdout:7218: seq_num 7217 ranges {6299648=475136,6774784=401408} 2020-06-12T02:38:33.732 INFO:tasks.rados.rados.0.smithi107.stdout:7218: writing smithi10729407-11 from 6299648 to 6774784 tid 1 2020-06-12T02:38:33.732 INFO:tasks.rados.rados.0.smithi107.stdout:7218: writing smithi10729407-11 from 6774784 to 7176192 tid 2 2020-06-12T02:38:33.732 INFO:tasks.rados.rados.0.smithi107.stdout:7219: delete oid 2 current snap is 0 2020-06-12T02:38:33.733 INFO:tasks.rados.rados.0.smithi107.stdout:7217: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:33.733 INFO:tasks.rados.rados.0.smithi107.stdout:7217: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:33.733 INFO:tasks.rados.rados.0.smithi107.stdout:7217: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:33.733 INFO:tasks.rados.rados.0.smithi107.stdout:7217: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:33.733 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1363 (ObjNum 7216 snap 0 seq_num 7216) dirty exists 2020-06-12T02:38:33.734 INFO:tasks.rados.rados.0.smithi107.stdout:7217: left oid 18 (ObjNum 7216 snap 0 seq_num 7216) 2020-06-12T02:38:33.734 INFO:tasks.rados.rados.0.smithi107.stdout:7217: done (2 left) 2020-06-12T02:38:33.734 INFO:tasks.rados.rados.0.smithi107.stdout:7219: done (1 left) 2020-06-12T02:38:33.734 INFO:tasks.rados.rados.0.smithi107.stdout:7220: delete oid 9 current snap is 0 2020-06-12T02:38:33.734 INFO:tasks.rados.rados.0.smithi107.stdout:7218: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:33.734 INFO:tasks.rados.rados.0.smithi107.stdout:7218: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:33.735 INFO:tasks.rados.rados.0.smithi107.stdout:7218: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:33.735 INFO:tasks.rados.rados.0.smithi107.stdout:7218: finishing write tid 5 to smithi10729407-11 2020-06-12T02:38:33.735 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1258 (ObjNum 7217 snap 0 seq_num 7217) dirty exists 2020-06-12T02:38:33.735 INFO:tasks.rados.rados.0.smithi107.stdout:7218: left oid 11 (ObjNum 7217 snap 0 seq_num 7217) 2020-06-12T02:38:33.735 INFO:tasks.rados.rados.0.smithi107.stdout:7218: done (1 left) 2020-06-12T02:38:33.736 INFO:tasks.rados.rados.0.smithi107.stdout:7220: done (0 left) 2020-06-12T02:38:33.736 INFO:tasks.rados.rados.0.smithi107.stdout:7221: delete oid 20 current snap is 0 2020-06-12T02:38:33.736 INFO:tasks.rados.rados.0.smithi107.stdout:7221: done (0 left) 2020-06-12T02:38:33.736 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:33.736 INFO:tasks.rados.rados.0.smithi107.stdout:7222: seq_num 7221 ranges {0=540672,540672=401408} 2020-06-12T02:38:33.736 INFO:tasks.rados.rados.0.smithi107.stdout:7222: writing smithi10729407-8 from 0 to 540672 tid 1 2020-06-12T02:38:33.737 INFO:tasks.rados.rados.0.smithi107.stdout:7222: writing smithi10729407-8 from 540672 to 942080 tid 2 2020-06-12T02:38:33.737 INFO:tasks.rados.rados.0.smithi107.stdout:7223: delete oid 14 current snap is 0 2020-06-12T02:38:33.737 INFO:tasks.rados.rados.0.smithi107.stdout:7223: done (1 left) 2020-06-12T02:38:33.737 INFO:tasks.rados.rados.0.smithi107.stdout:7224: delete oid 16 current snap is 0 2020-06-12T02:38:33.737 INFO:tasks.rados.rados.0.smithi107.stdout:7222: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:33.738 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-06-12T02:38:33.739 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:38:33.739 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:38:33.739 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:38:33.739 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:38:33.740 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:38:33.740 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2020-06-12T02:38:33.740 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2020-06-12T02:38:33.740 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-06-12T02:38:33.741 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-06-12T02:38:33.741 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-06-12T02:38:33.741 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-06-12T02:38:33.741 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-06-12T02:38:33.741 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-06-12T02:38:33.742 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-06-12T02:38:33.742 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-06-12T02:38:33.742 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:33.673+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:33.744 INFO:tasks.rados.rados.0.smithi107.stdout:7224: done (1 left) 2020-06-12T02:38:33.745 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:33.745 INFO:tasks.rados.rados.0.smithi107.stdout:7225: seq_num 7224 ranges {442368=598016,1040384=401408} 2020-06-12T02:38:33.745 INFO:tasks.rados.rados.0.smithi107.stdout:7225: writing smithi10729407-15 from 442368 to 1040384 tid 1 2020-06-12T02:38:33.745 INFO:tasks.rados.rados.0.smithi107.stdout:7225: writing smithi10729407-15 from 1040384 to 1441792 tid 2 2020-06-12T02:38:33.746 INFO:tasks.rados.rados.0.smithi107.stdout:7226: delete oid 3 current snap is 0 2020-06-12T02:38:33.746 INFO:tasks.rados.rados.0.smithi107.stdout:7222: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:33.746 INFO:tasks.rados.rados.0.smithi107.stdout:7222: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:33.746 INFO:tasks.rados.rados.0.smithi107.stdout:7222: finishing write tid 5 to smithi10729407-8 2020-06-12T02:38:33.746 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1364 (ObjNum 7221 snap 0 seq_num 7221) dirty exists 2020-06-12T02:38:33.746 INFO:tasks.rados.rados.0.smithi107.stdout:7222: left oid 8 (ObjNum 7221 snap 0 seq_num 7221) 2020-06-12T02:38:33.747 INFO:tasks.rados.rados.0.smithi107.stdout:7222: done (2 left) 2020-06-12T02:38:33.747 INFO:tasks.rados.rados.0.smithi107.stdout:7226: done (1 left) 2020-06-12T02:38:33.747 INFO:tasks.rados.rados.0.smithi107.stdout:7227: delete oid 5 current snap is 0 2020-06-12T02:38:33.747 INFO:tasks.rados.rados.0.smithi107.stdout:7225: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:33.747 INFO:tasks.rados.rados.0.smithi107.stdout:7227: done (1 left) 2020-06-12T02:38:33.748 INFO:tasks.rados.rados.0.smithi107.stdout:7228: delete oid 17 current snap is 0 2020-06-12T02:38:33.748 INFO:tasks.rados.rados.0.smithi107.stdout:7228: done (1 left) 2020-06-12T02:38:33.748 INFO:tasks.rados.rados.0.smithi107.stdout:7229: delete oid 8 current snap is 0 2020-06-12T02:38:33.748 INFO:tasks.rados.rados.0.smithi107.stdout:7225: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:33.748 INFO:tasks.rados.rados.0.smithi107.stdout:7225: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:33.748 INFO:tasks.rados.rados.0.smithi107.stdout:7225: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:33.749 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1367 (ObjNum 7224 snap 0 seq_num 7224) dirty exists 2020-06-12T02:38:33.749 INFO:tasks.rados.rados.0.smithi107.stdout:7225: left oid 15 (ObjNum 7224 snap 0 seq_num 7224) 2020-06-12T02:38:33.749 INFO:tasks.rados.rados.0.smithi107.stdout:7225: done (1 left) 2020-06-12T02:38:33.749 INFO:tasks.rados.rados.0.smithi107.stdout:7229: done (0 left) 2020-06-12T02:38:33.749 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:38:33.750 INFO:tasks.rados.rados.0.smithi107.stdout:7230: seq_num 7229 ranges {458752=679936,1138688=401408} 2020-06-12T02:38:33.750 INFO:tasks.rados.rados.0.smithi107.stdout:7230: writing smithi10729407-10 from 458752 to 1138688 tid 1 2020-06-12T02:38:33.750 INFO:tasks.rados.rados.0.smithi107.stdout:7230: writing smithi10729407-10 from 1138688 to 1540096 tid 2 2020-06-12T02:38:33.750 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:33.750 INFO:tasks.rados.rados.0.smithi107.stdout:7231: seq_num 7230 ranges {7176192=696320,7872512=401408} 2020-06-12T02:38:33.765 INFO:tasks.rados.rados.0.smithi107.stdout:7231: writing smithi10729407-11 from 7176192 to 7872512 tid 1 2020-06-12T02:38:33.767 INFO:tasks.rados.rados.0.smithi107.stdout:7231: writing smithi10729407-11 from 7872512 to 8273920 tid 2 2020-06-12T02:38:33.768 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:33.768 INFO:tasks.rados.rados.0.smithi107.stdout:7232: seq_num 7231 ranges {0=712704,712704=401408} 2020-06-12T02:38:33.771 INFO:tasks.rados.rados.0.smithi107.stdout:7232: writing smithi10729407-14 from 0 to 712704 tid 1 2020-06-12T02:38:33.774 INFO:tasks.rados.rados.0.smithi107.stdout:7232: writing smithi10729407-14 from 712704 to 1114112 tid 2 2020-06-12T02:38:33.774 INFO:tasks.rados.rados.0.smithi107.stdout:7233: delete oid 17 current snap is 0 2020-06-12T02:38:33.775 INFO:tasks.rados.rados.0.smithi107.stdout:7230: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:33.775 INFO:tasks.rados.rados.0.smithi107.stdout:7230: finishing write tid 2 to smithi10729407-10 2020-06-12T02:38:33.775 INFO:tasks.rados.rados.0.smithi107.stdout:7230: finishing write tid 4 to smithi10729407-10 2020-06-12T02:38:33.775 INFO:tasks.rados.rados.0.smithi107.stdout:7230: finishing write tid 5 to smithi10729407-10 2020-06-12T02:38:33.775 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 491 (ObjNum 7229 snap 0 seq_num 7229) dirty exists 2020-06-12T02:38:33.776 INFO:tasks.rados.rados.0.smithi107.stdout:7230: left oid 10 (ObjNum 7229 snap 0 seq_num 7229) 2020-06-12T02:38:33.777 INFO:tasks.rados.rados.0.smithi107.stdout:7230: done (3 left) 2020-06-12T02:38:33.777 INFO:tasks.rados.rados.0.smithi107.stdout:7233: done (2 left) 2020-06-12T02:38:33.777 INFO:tasks.rados.rados.0.smithi107.stdout:7234: delete oid 10 current snap is 0 2020-06-12T02:38:33.778 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:33.775+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:33.779 INFO:tasks.rados.rados.0.smithi107.stdout:7234: done (2 left) 2020-06-12T02:38:33.779 INFO:tasks.rados.rados.0.smithi107.stdout:7235: delete oid 18 current snap is 0 2020-06-12T02:38:33.783 INFO:tasks.rados.rados.0.smithi107.stdout:7231: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:33.783 INFO:tasks.rados.rados.0.smithi107.stdout:7231: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:33.784 INFO:tasks.rados.rados.0.smithi107.stdout:7231: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:33.784 INFO:tasks.rados.rados.0.smithi107.stdout:7235: done (2 left) 2020-06-12T02:38:33.784 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:33.785 INFO:tasks.rados.rados.0.smithi107.stdout:7236: seq_num 7235 ranges {0=778240,778240=401408} 2020-06-12T02:38:33.789 INFO:tasks.rados.rados.0.smithi107.stdout:7236: writing smithi10729407-9 from 0 to 778240 tid 1 2020-06-12T02:38:33.792 INFO:tasks.rados.rados.0.smithi107.stdout:7236: writing smithi10729407-9 from 778240 to 1179648 tid 2 2020-06-12T02:38:33.792 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:38:33.792 INFO:tasks.rados.rados.0.smithi107.stdout:7237: seq_num 7236 ranges {0=794624,794624=401408} 2020-06-12T02:38:33.796 INFO:tasks.rados.rados.0.smithi107.stdout:7237: writing smithi10729407-20 from 0 to 794624 tid 1 2020-06-12T02:38:33.799 INFO:tasks.rados.rados.0.smithi107.stdout:7237: writing smithi10729407-20 from 794624 to 1196032 tid 2 2020-06-12T02:38:33.800 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:38:33.800 INFO:tasks.rados.rados.0.smithi107.stdout:7238: seq_num 7237 ranges {0=409600,409600=565248,974848=237568} 2020-06-12T02:38:33.802 INFO:tasks.rados.rados.0.smithi107.stdout:7238: writing smithi10729407-16 from 0 to 409600 tid 1 2020-06-12T02:38:33.806 INFO:tasks.rados.rados.0.smithi107.stdout:7238: writing smithi10729407-16 from 409600 to 974848 tid 2 2020-06-12T02:38:33.808 INFO:tasks.rados.rados.0.smithi107.stdout:7238: writing smithi10729407-16 from 974848 to 1212416 tid 3 2020-06-12T02:38:33.809 INFO:tasks.rados.rados.0.smithi107.stdout:7239: delete oid 8 current snap is 0 2020-06-12T02:38:33.809 INFO:tasks.rados.rados.0.smithi107.stdout:7232: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:33.809 INFO:tasks.rados.rados.0.smithi107.stdout:7232: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:33.809 INFO:tasks.rados.rados.0.smithi107.stdout:7231: finishing write tid 5 to smithi10729407-11 2020-06-12T02:38:33.810 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1262 (ObjNum 7230 snap 0 seq_num 7230) dirty exists 2020-06-12T02:38:33.810 INFO:tasks.rados.rados.0.smithi107.stdout:7231: left oid 11 (ObjNum 7230 snap 0 seq_num 7230) 2020-06-12T02:38:33.810 INFO:tasks.rados.rados.0.smithi107.stdout:7232: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:33.810 INFO:tasks.rados.rados.0.smithi107.stdout:7232: finishing write tid 5 to smithi10729407-14 2020-06-12T02:38:33.810 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1265 (ObjNum 7231 snap 0 seq_num 7231) dirty exists 2020-06-12T02:38:33.811 INFO:tasks.rados.rados.0.smithi107.stdout:7232: left oid 14 (ObjNum 7231 snap 0 seq_num 7231) 2020-06-12T02:38:33.811 INFO:tasks.rados.rados.0.smithi107.stdout:7236: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:33.811 INFO:tasks.rados.rados.0.smithi107.stdout:7236: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:33.811 INFO:tasks.rados.rados.0.smithi107.stdout:7236: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:33.811 INFO:tasks.rados.rados.0.smithi107.stdout:7237: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:33.812 INFO:tasks.rados.rados.0.smithi107.stdout:7236: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:33.812 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1368 (ObjNum 7235 snap 0 seq_num 7235) dirty exists 2020-06-12T02:38:33.812 INFO:tasks.rados.rados.0.smithi107.stdout:7236: left oid 9 (ObjNum 7235 snap 0 seq_num 7235) 2020-06-12T02:38:33.813 INFO:tasks.rados.rados.0.smithi107.stdout:7238: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:33.813 INFO:tasks.rados.rados.0.smithi107.stdout:7231: done (5 left) 2020-06-12T02:38:33.813 INFO:tasks.rados.rados.0.smithi107.stdout:7232: done (4 left) 2020-06-12T02:38:33.814 INFO:tasks.rados.rados.0.smithi107.stdout:7236: done (3 left) 2020-06-12T02:38:33.814 INFO:tasks.rados.rados.0.smithi107.stdout:7239: done (2 left) 2020-06-12T02:38:33.814 INFO:tasks.rados.rados.0.smithi107.stdout:7240: delete oid 5 current snap is 0 2020-06-12T02:38:33.814 INFO:tasks.rados.rados.0.smithi107.stdout:7237: finishing write tid 2 to smithi10729407-20 2020-06-12T02:38:33.814 INFO:tasks.rados.rados.0.smithi107.stdout:7237: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:33.815 INFO:tasks.rados.rados.0.smithi107.stdout:7237: finishing write tid 5 to smithi10729407-20 2020-06-12T02:38:33.816 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 540 (ObjNum 7236 snap 0 seq_num 7236) dirty exists 2020-06-12T02:38:33.816 INFO:tasks.rados.rados.0.smithi107.stdout:7237: left oid 20 (ObjNum 7236 snap 0 seq_num 7236) 2020-06-12T02:38:33.816 INFO:tasks.rados.rados.0.smithi107.stdout:7238: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:33.816 INFO:tasks.rados.rados.0.smithi107.stdout:7237: done (2 left) 2020-06-12T02:38:33.817 INFO:tasks.rados.rados.0.smithi107.stdout:7240: done (1 left) 2020-06-12T02:38:33.817 INFO:tasks.rados.rados.0.smithi107.stdout:7241: delete oid 5 current snap is 0 2020-06-12T02:38:33.817 INFO:tasks.rados.rados.0.smithi107.stdout:7238: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:33.817 INFO:tasks.rados.rados.0.smithi107.stdout:7238: finishing write tid 5 to smithi10729407-16 2020-06-12T02:38:33.818 INFO:tasks.rados.rados.0.smithi107.stdout:7238: finishing write tid 6 to smithi10729407-16 2020-06-12T02:38:33.818 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1292 (ObjNum 7237 snap 0 seq_num 7237) dirty exists 2020-06-12T02:38:33.818 INFO:tasks.rados.rados.0.smithi107.stdout:7238: left oid 16 (ObjNum 7237 snap 0 seq_num 7237) 2020-06-12T02:38:33.818 INFO:tasks.rados.rados.0.smithi107.stdout:7238: done (1 left) 2020-06-12T02:38:33.819 INFO:tasks.rados.rados.0.smithi107.stdout:7241: done (0 left) 2020-06-12T02:38:33.819 INFO:tasks.rados.rados.0.smithi107.stdout:7242: delete oid 16 current snap is 0 2020-06-12T02:38:33.819 INFO:tasks.rados.rados.0.smithi107.stdout:7242: done (0 left) 2020-06-12T02:38:33.819 INFO:tasks.rados.rados.0.smithi107.stdout:7243: delete oid 17 current snap is 0 2020-06-12T02:38:33.822 INFO:tasks.rados.rados.0.smithi107.stdout:7243: done (0 left) 2020-06-12T02:38:33.822 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:38:33.822 INFO:tasks.rados.rados.0.smithi107.stdout:7244: seq_num 7243 ranges {8273920=507904,8781824=720896,9502720=81920} 2020-06-12T02:38:33.868 INFO:tasks.rados.rados.0.smithi107.stdout:7244: writing smithi10729407-11 from 8273920 to 8781824 tid 1 2020-06-12T02:38:33.872 INFO:tasks.rados.rados.0.smithi107.stdout:7244: writing smithi10729407-11 from 8781824 to 9502720 tid 2 2020-06-12T02:38:33.873 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:33.873 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:33.874 INFO:tasks.rados.rados.0.smithi107.stdout:7244: writing smithi10729407-11 from 9502720 to 9584640 tid 3 2020-06-12T02:38:33.874 INFO:tasks.rados.rados.0.smithi107.stdout:7245: delete oid 19 current snap is 0 2020-06-12T02:38:33.874 INFO:tasks.rados.rados.0.smithi107.stdout:7244: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:33.875 INFO:tasks.rados.rados.0.smithi107.stdout:7245: done (1 left) 2020-06-12T02:38:33.875 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:38:33.876 INFO:tasks.rados.rados.0.smithi107.stdout:7246: seq_num 7245 ranges {0=548864,548864=483328,1032192=319488} 2020-06-12T02:38:33.878 INFO:tasks.rados.rados.0.smithi107.stdout:7246: writing smithi10729407-13 from 0 to 548864 tid 1 2020-06-12T02:38:33.885 INFO:tasks.rados.rados.0.smithi107.stdout:7246: writing smithi10729407-13 from 548864 to 1032192 tid 2 2020-06-12T02:38:33.886 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-06-12T02:38:33.887 INFO:tasks.rados.rados.0.smithi107.stdout:7246: writing smithi10729407-13 from 1032192 to 1351680 tid 3 2020-06-12T02:38:33.887 INFO:tasks.rados.rados.0.smithi107.stdout:7247: delete oid 14 current snap is 0 2020-06-12T02:38:33.887 INFO:tasks.rados.rados.0.smithi107.stdout:7244: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:33.888 INFO:tasks.rados.rados.0.smithi107.stdout:7244: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:33.888 INFO:tasks.rados.rados.0.smithi107.stdout:7244: finishing write tid 5 to smithi10729407-11 2020-06-12T02:38:33.888 INFO:tasks.rados.rados.0.smithi107.stdout:7244: finishing write tid 6 to smithi10729407-11 2020-06-12T02:38:33.888 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1269 (ObjNum 7243 snap 0 seq_num 7243) dirty exists 2020-06-12T02:38:33.888 INFO:tasks.rados.rados.0.smithi107.stdout:7244: left oid 11 (ObjNum 7243 snap 0 seq_num 7243) 2020-06-12T02:38:33.889 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:33.882+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:33.889 INFO:tasks.rados.rados.0.smithi107.stdout:7244: done (2 left) 2020-06-12T02:38:33.890 INFO:tasks.rados.rados.0.smithi107.stdout:7247: done (1 left) 2020-06-12T02:38:33.890 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:38:33.890 INFO:tasks.rados.rados.0.smithi107.stdout:7248: seq_num 7247 ranges {0=581632,581632=704512,1286144=98304} 2020-06-12T02:38:33.892 INFO:tasks.rados.rados.0.smithi107.stdout:7248: writing smithi10729407-16 from 0 to 581632 tid 1 2020-06-12T02:38:33.895 INFO:tasks.rados.rados.0.smithi107.stdout:7248: writing smithi10729407-16 from 581632 to 1286144 tid 2 2020-06-12T02:38:33.897 INFO:tasks.rados.rados.0.smithi107.stdout:7248: writing smithi10729407-16 from 1286144 to 1384448 tid 3 2020-06-12T02:38:33.897 INFO:tasks.rados.rados.0.smithi107.stdout:7249: delete oid 10 current snap is 0 2020-06-12T02:38:33.897 INFO:tasks.rados.rados.0.smithi107.stdout:7246: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:33.897 INFO:tasks.rados.rados.0.smithi107.stdout:7246: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:33.898 INFO:tasks.rados.rados.0.smithi107.stdout:7246: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:33.898 INFO:tasks.rados.rados.0.smithi107.stdout:7246: finishing write tid 5 to smithi10729407-13 2020-06-12T02:38:33.898 INFO:tasks.rados.rados.0.smithi107.stdout:7246: finishing write tid 6 to smithi10729407-13 2020-06-12T02:38:33.898 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1342 (ObjNum 7245 snap 0 seq_num 7245) dirty exists 2020-06-12T02:38:33.898 INFO:tasks.rados.rados.0.smithi107.stdout:7246: left oid 13 (ObjNum 7245 snap 0 seq_num 7245) 2020-06-12T02:38:33.899 INFO:tasks.rados.rados.0.smithi107.stdout:7246: done (2 left) 2020-06-12T02:38:33.899 INFO:tasks.rados.rados.0.smithi107.stdout:7249: done (1 left) 2020-06-12T02:38:33.899 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:33.900 INFO:tasks.rados.rados.0.smithi107.stdout:7250: seq_num 7249 ranges {0=614400,614400=475136,1089536=327680} 2020-06-12T02:38:33.903 INFO:tasks.rados.rados.0.smithi107.stdout:7250: writing smithi10729407-4 from 0 to 614400 tid 1 2020-06-12T02:38:33.906 INFO:tasks.rados.rados.0.smithi107.stdout:7250: writing smithi10729407-4 from 614400 to 1089536 tid 2 2020-06-12T02:38:33.907 INFO:tasks.rados.rados.0.smithi107.stdout:7250: writing smithi10729407-4 from 1089536 to 1417216 tid 3 2020-06-12T02:38:33.908 INFO:tasks.rados.rados.0.smithi107.stdout:7251: delete oid 2 current snap is 0 2020-06-12T02:38:33.908 INFO:tasks.rados.rados.0.smithi107.stdout:7248: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:33.908 INFO:tasks.rados.rados.0.smithi107.stdout:7248: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:33.909 INFO:tasks.rados.rados.0.smithi107.stdout:7248: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:33.909 INFO:tasks.rados.rados.0.smithi107.stdout:7248: finishing write tid 5 to smithi10729407-16 2020-06-12T02:38:33.910 INFO:tasks.rados.rados.0.smithi107.stdout:7250: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:33.911 INFO:tasks.rados.rados.0.smithi107.stdout:7248: finishing write tid 6 to smithi10729407-16 2020-06-12T02:38:33.911 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1297 (ObjNum 7247 snap 0 seq_num 7247) dirty exists 2020-06-12T02:38:33.911 INFO:tasks.rados.rados.0.smithi107.stdout:7248: left oid 16 (ObjNum 7247 snap 0 seq_num 7247) 2020-06-12T02:38:33.914 INFO:tasks.rados.rados.0.smithi107.stdout:7248: done (2 left) 2020-06-12T02:38:33.915 INFO:tasks.rados.rados.0.smithi107.stdout:7251: done (1 left) 2020-06-12T02:38:33.915 INFO:tasks.rados.rados.0.smithi107.stdout:7252: delete oid 12 current snap is 0 2020-06-12T02:38:33.917 INFO:tasks.rados.rados.0.smithi107.stdout:7250: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:33.917 INFO:tasks.rados.rados.0.smithi107.stdout:7252: done (1 left) 2020-06-12T02:38:33.917 INFO:tasks.rados.rados.0.smithi107.stdout:7253: delete oid 20 current snap is 0 2020-06-12T02:38:33.919 INFO:tasks.rados.rados.0.smithi107.stdout:7250: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:33.922 INFO:tasks.rados.rados.0.smithi107.stdout:7253: done (1 left) 2020-06-12T02:38:33.922 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:33.922 INFO:tasks.rados.rados.0.smithi107.stdout:7254: seq_num 7253 ranges {0=679936,679936=458752,1138688=344064} 2020-06-12T02:38:33.926 INFO:tasks.rados.rados.0.smithi107.stdout:7254: writing smithi10729407-18 from 0 to 679936 tid 1 2020-06-12T02:38:33.928 INFO:tasks.rados.rados.0.smithi107.stdout:7254: writing smithi10729407-18 from 679936 to 1138688 tid 2 2020-06-12T02:38:33.939 INFO:tasks.rados.rados.0.smithi107.stdout:7254: writing smithi10729407-18 from 1138688 to 1482752 tid 3 2020-06-12T02:38:33.939 INFO:tasks.rados.rados.0.smithi107.stdout:7255: delete oid 10 current snap is 0 2020-06-12T02:38:33.940 INFO:tasks.rados.rados.0.smithi107.stdout:7250: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:33.940 INFO:tasks.rados.rados.0.smithi107.stdout:7250: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:33.940 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1362 (ObjNum 7249 snap 0 seq_num 7249) dirty exists 2020-06-12T02:38:33.940 INFO:tasks.rados.rados.0.smithi107.stdout:7250: left oid 4 (ObjNum 7249 snap 0 seq_num 7249) 2020-06-12T02:38:33.940 INFO:tasks.rados.rados.0.smithi107.stdout:7254: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:33.940 INFO:tasks.rados.rados.0.smithi107.stdout:7250: done (2 left) 2020-06-12T02:38:33.941 INFO:tasks.rados.rados.0.smithi107.stdout:7255: done (1 left) 2020-06-12T02:38:33.941 INFO:tasks.rados.rados.0.smithi107.stdout:7256: delete oid 3 current snap is 0 2020-06-12T02:38:33.941 INFO:tasks.rados.rados.0.smithi107.stdout:7256: done (1 left) 2020-06-12T02:38:33.941 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:33.941 INFO:tasks.rados.rados.0.smithi107.stdout:7257: seq_num 7256 ranges {0=729088,729088=516096,1245184=286720} 2020-06-12T02:38:33.949 INFO:tasks.rados.rados.0.smithi107.stdout:7257: writing smithi10729407-12 from 0 to 729088 tid 1 2020-06-12T02:38:33.949 INFO:tasks.rados.rados.0.smithi107.stdout:7257: writing smithi10729407-12 from 729088 to 1245184 tid 2 2020-06-12T02:38:33.949 INFO:tasks.rados.rados.0.smithi107.stdout:7257: writing smithi10729407-12 from 1245184 to 1531904 tid 3 2020-06-12T02:38:33.954 INFO:tasks.rados.rados.0.smithi107.stdout:7258: delete oid 17 current snap is 0 2020-06-12T02:38:33.954 INFO:tasks.rados.rados.0.smithi107.stdout:7254: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:33.954 INFO:tasks.rados.rados.0.smithi107.stdout:7254: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:33.954 INFO:tasks.rados.rados.0.smithi107.stdout:7254: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:33.954 INFO:tasks.rados.rados.0.smithi107.stdout:7254: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:33.955 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1372 (ObjNum 7253 snap 0 seq_num 7253) dirty exists 2020-06-12T02:38:33.955 INFO:tasks.rados.rados.0.smithi107.stdout:7254: left oid 18 (ObjNum 7253 snap 0 seq_num 7253) 2020-06-12T02:38:33.956 INFO:tasks.rados.rados.0.smithi107.stdout:7254: done (2 left) 2020-06-12T02:38:33.962 INFO:tasks.rados.rados.0.smithi107.stdout:7258: done (1 left) 2020-06-12T02:38:33.962 INFO:tasks.rados.rados.0.smithi107.stdout:7259: delete oid 18 current snap is 0 2020-06-12T02:38:33.962 INFO:tasks.rados.rados.0.smithi107.stdout:7257: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:33.962 INFO:tasks.rados.rados.0.smithi107.stdout:7259: done (1 left) 2020-06-12T02:38:33.963 INFO:tasks.rados.rados.0.smithi107.stdout:7260: delete oid 2 current snap is 0 2020-06-12T02:38:33.964 INFO:tasks.rados.rados.0.smithi107.stdout:7260: done (1 left) 2020-06-12T02:38:33.964 INFO:tasks.rados.rados.0.smithi107.stdout:7261: delete oid 8 current snap is 0 2020-06-12T02:38:33.964 INFO:tasks.rados.rados.0.smithi107.stdout:7257: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:33.965 INFO:tasks.rados.rados.0.smithi107.stdout:7261: done (1 left) 2020-06-12T02:38:33.965 INFO:tasks.rados.rados.0.smithi107.stdout:7262: delete oid 7 current snap is 0 2020-06-12T02:38:33.966 INFO:tasks.rados.rados.0.smithi107.stdout:7257: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:33.966 INFO:tasks.rados.rados.0.smithi107.stdout:7257: finishing write tid 5 to smithi10729407-12 2020-06-12T02:38:33.967 INFO:tasks.rados.rados.0.smithi107.stdout:7262: done (1 left) 2020-06-12T02:38:33.967 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:33.967 INFO:tasks.rados.rados.0.smithi107.stdout:7263: seq_num 7262 ranges {9584640=425984,10010624=671744,10682368=516096,11198464=16384} 2020-06-12T02:38:33.980 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:33.977+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:34.019 INFO:tasks.rados.rados.0.smithi107.stdout:7263: writing smithi10729407-11 from 9584640 to 10010624 tid 1 2020-06-12T02:38:34.022 INFO:tasks.rados.rados.0.smithi107.stdout:7263: writing smithi10729407-11 from 10010624 to 10682368 tid 2 2020-06-12T02:38:34.026 INFO:tasks.rados.rados.0.smithi107.stdout:7263: writing smithi10729407-11 from 10682368 to 11198464 tid 3 2020-06-12T02:38:34.026 INFO:tasks.rados.rados.0.smithi107.stdout:7263: writing smithi10729407-11 from 11198464 to 11214848 tid 4 2020-06-12T02:38:34.026 INFO:tasks.rados.rados.0.smithi107.stdout:7257: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:34.027 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1337 (ObjNum 7256 snap 0 seq_num 7256) dirty exists 2020-06-12T02:38:34.027 INFO:tasks.rados.rados.0.smithi107.stdout:7257: left oid 12 (ObjNum 7256 snap 0 seq_num 7256) 2020-06-12T02:38:34.027 INFO:tasks.rados.rados.0.smithi107.stdout:7257: done (1 left) 2020-06-12T02:38:34.027 INFO:tasks.rados.rados.0.smithi107.stdout:7264: delete oid 14 current snap is 0 2020-06-12T02:38:34.028 INFO:tasks.rados.rados.0.smithi107.stdout:7263: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:34.031 INFO:tasks.rados.rados.0.smithi107.stdout:7263: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:34.032 INFO:tasks.rados.rados.0.smithi107.stdout:7263: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:34.033 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:34.033 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:34.035 INFO:tasks.rados.rados.0.smithi107.stdout:7263: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:34.037 INFO:tasks.rados.rados.0.smithi107.stdout:7263: finishing write tid 6 to smithi10729407-11 2020-06-12T02:38:34.038 INFO:tasks.rados.rados.0.smithi107.stdout:7264: done (1 left) 2020-06-12T02:38:34.043 INFO:tasks.rados.rados.0.smithi107.stdout:7265: delete oid 19 current snap is 0 2020-06-12T02:38:34.043 INFO:tasks.rados.rados.0.smithi107.stdout:7263: finishing write tid 7 to smithi10729407-11 2020-06-12T02:38:34.044 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1275 (ObjNum 7262 snap 0 seq_num 7262) dirty exists 2020-06-12T02:38:34.044 INFO:tasks.rados.rados.0.smithi107.stdout:7263: left oid 11 (ObjNum 7262 snap 0 seq_num 7262) 2020-06-12T02:38:34.047 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-06-12T02:38:34.048 INFO:tasks.rados.rados.0.smithi107.stdout:7263: done (1 left) 2020-06-12T02:38:34.049 INFO:tasks.rados.rados.0.smithi107.stdout:7265: done (0 left) 2020-06-12T02:38:34.049 INFO:tasks.rados.rados.0.smithi107.stdout:7266: delete oid 8 current snap is 0 2020-06-12T02:38:34.051 INFO:tasks.rados.rados.0.smithi107.stdout:7266: done (0 left) 2020-06-12T02:38:34.052 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:34.052 INFO:tasks.rados.rados.0.smithi107.stdout:7267: seq_num 7266 ranges {0=499712,499712=606208,1105920=491520,1597440=106496} 2020-06-12T02:38:34.055 INFO:tasks.rados.rados.0.smithi107.stdout:7267: writing smithi10729407-17 from 0 to 499712 tid 1 2020-06-12T02:38:34.058 INFO:tasks.rados.rados.0.smithi107.stdout:7267: writing smithi10729407-17 from 499712 to 1105920 tid 2 2020-06-12T02:38:34.061 INFO:tasks.rados.rados.0.smithi107.stdout:7267: writing smithi10729407-17 from 1105920 to 1597440 tid 3 2020-06-12T02:38:34.061 INFO:tasks.rados.rados.0.smithi107.stdout:7267: writing smithi10729407-17 from 1597440 to 1703936 tid 4 2020-06-12T02:38:34.062 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:38:34.062 INFO:tasks.rados.rados.0.smithi107.stdout:7268: seq_num 7267 ranges {11214848=516096,11730944=491520,12222464=655360,12877824=57344} 2020-06-12T02:38:34.092 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:34.089+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:34.118 INFO:tasks.rados.rados.0.smithi107.stdout:7268: writing smithi10729407-11 from 11214848 to 11730944 tid 1 2020-06-12T02:38:34.121 INFO:tasks.rados.rados.0.smithi107.stdout:7268: writing smithi10729407-11 from 11730944 to 12222464 tid 2 2020-06-12T02:38:34.125 INFO:tasks.rados.rados.0.smithi107.stdout:7268: writing smithi10729407-11 from 12222464 to 12877824 tid 3 2020-06-12T02:38:34.125 INFO:tasks.rados.rados.0.smithi107.stdout:7268: writing smithi10729407-11 from 12877824 to 12935168 tid 4 2020-06-12T02:38:34.126 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:38:34.126 INFO:tasks.rados.rados.0.smithi107.stdout:7269: seq_num 7268 ranges {0=532480,532480=425984,958464=458752,1417216=319488} 2020-06-12T02:38:34.129 INFO:tasks.rados.rados.0.smithi107.stdout:7269: writing smithi10729407-3 from 0 to 532480 tid 1 2020-06-12T02:38:34.132 INFO:tasks.rados.rados.0.smithi107.stdout:7269: writing smithi10729407-3 from 532480 to 958464 tid 2 2020-06-12T02:38:34.135 INFO:tasks.rados.rados.0.smithi107.stdout:7269: writing smithi10729407-3 from 958464 to 1417216 tid 3 2020-06-12T02:38:34.138 INFO:tasks.rados.rados.0.smithi107.stdout:7269: writing smithi10729407-3 from 1417216 to 1736704 tid 4 2020-06-12T02:38:34.138 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:34.138 INFO:tasks.rados.rados.0.smithi107.stdout:7270: seq_num 7269 ranges {0=548864,548864=712704,1261568=491520} 2020-06-12T02:38:34.141 INFO:tasks.rados.rados.0.smithi107.stdout:7270: writing smithi10729407-5 from 0 to 548864 tid 1 2020-06-12T02:38:34.146 INFO:tasks.rados.rados.0.smithi107.stdout:7270: writing smithi10729407-5 from 548864 to 1261568 tid 2 2020-06-12T02:38:34.148 INFO:tasks.rados.rados.0.smithi107.stdout:7270: writing smithi10729407-5 from 1261568 to 1753088 tid 3 2020-06-12T02:38:34.149 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:34.149 INFO:tasks.rados.rados.0.smithi107.stdout:7271: seq_num 7270 ranges {1531904=565248,2097152=589824,2686976=475136,3162112=139264} 2020-06-12T02:38:34.161 INFO:tasks.rados.rados.0.smithi107.stdout:7271: writing smithi10729407-12 from 1531904 to 2097152 tid 1 2020-06-12T02:38:34.164 INFO:tasks.rados.rados.0.smithi107.stdout:7271: writing smithi10729407-12 from 2097152 to 2686976 tid 2 2020-06-12T02:38:34.167 INFO:tasks.rados.rados.0.smithi107.stdout:7271: writing smithi10729407-12 from 2686976 to 3162112 tid 3 2020-06-12T02:38:34.168 INFO:tasks.rados.rados.0.smithi107.stdout:7271: writing smithi10729407-12 from 3162112 to 3301376 tid 4 2020-06-12T02:38:34.168 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:34.168 INFO:tasks.rados.rados.0.smithi107.stdout:7272: seq_num 7271 ranges {1417216=581632,1998848=475136,2473984=679936,3153920=49152} 2020-06-12T02:38:34.179 INFO:tasks.rados.rados.0.smithi107.stdout:7272: writing smithi10729407-4 from 1417216 to 1998848 tid 1 2020-06-12T02:38:34.181 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:34.178+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:34.181 INFO:tasks.rados.rados.0.smithi107.stdout:7272: writing smithi10729407-4 from 1998848 to 2473984 tid 2 2020-06-12T02:38:34.185 INFO:tasks.rados.rados.0.smithi107.stdout:7272: writing smithi10729407-4 from 2473984 to 3153920 tid 3 2020-06-12T02:38:34.187 INFO:tasks.rados.rados.0.smithi107.stdout:7272: writing smithi10729407-4 from 3153920 to 3203072 tid 4 2020-06-12T02:38:34.187 INFO:tasks.rados.rados.0.smithi107.stdout:7273: delete oid 16 current snap is 0 2020-06-12T02:38:34.187 INFO:tasks.rados.rados.0.smithi107.stdout:7267: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:34.187 INFO:tasks.rados.rados.0.smithi107.stdout:7267: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:34.187 INFO:tasks.rados.rados.0.smithi107.stdout:7267: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:34.188 INFO:tasks.rados.rados.0.smithi107.stdout:7267: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:34.188 INFO:tasks.rados.rados.0.smithi107.stdout:7267: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:34.188 INFO:tasks.rados.rados.0.smithi107.stdout:7267: finishing write tid 7 to smithi10729407-17 2020-06-12T02:38:34.188 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1368 (ObjNum 7266 snap 0 seq_num 7266) dirty exists 2020-06-12T02:38:34.188 INFO:tasks.rados.rados.0.smithi107.stdout:7267: left oid 17 (ObjNum 7266 snap 0 seq_num 7266) 2020-06-12T02:38:34.189 INFO:tasks.rados.rados.0.smithi107.stdout:7268: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:34.189 INFO:tasks.rados.rados.0.smithi107.stdout:7269: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:34.189 INFO:tasks.rados.rados.0.smithi107.stdout:7268: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:34.189 INFO:tasks.rados.rados.0.smithi107.stdout:7268: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:34.189 INFO:tasks.rados.rados.0.smithi107.stdout:7268: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:34.190 INFO:tasks.rados.rados.0.smithi107.stdout:7268: finishing write tid 6 to smithi10729407-11 2020-06-12T02:38:34.190 INFO:tasks.rados.rados.0.smithi107.stdout:7268: finishing write tid 7 to smithi10729407-11 2020-06-12T02:38:34.190 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1281 (ObjNum 7267 snap 0 seq_num 7267) dirty exists 2020-06-12T02:38:34.190 INFO:tasks.rados.rados.0.smithi107.stdout:7268: left oid 11 (ObjNum 7267 snap 0 seq_num 7267) 2020-06-12T02:38:34.190 INFO:tasks.rados.rados.0.smithi107.stdout:7269: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:34.190 INFO:tasks.rados.rados.0.smithi107.stdout:7269: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:34.191 INFO:tasks.rados.rados.0.smithi107.stdout:7269: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:34.191 INFO:tasks.rados.rados.0.smithi107.stdout:7269: finishing write tid 6 to smithi10729407-3 2020-06-12T02:38:34.191 INFO:tasks.rados.rados.0.smithi107.stdout:7269: finishing write tid 7 to smithi10729407-3 2020-06-12T02:38:34.191 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1303 (ObjNum 7268 snap 0 seq_num 7268) dirty exists 2020-06-12T02:38:34.191 INFO:tasks.rados.rados.0.smithi107.stdout:7269: left oid 3 (ObjNum 7268 snap 0 seq_num 7268) 2020-06-12T02:38:34.192 INFO:tasks.rados.rados.0.smithi107.stdout:7270: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:34.192 INFO:tasks.rados.rados.0.smithi107.stdout:7270: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:34.192 INFO:tasks.rados.rados.0.smithi107.stdout:7270: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:34.192 INFO:tasks.rados.rados.0.smithi107.stdout:7270: finishing write tid 5 to smithi10729407-5 2020-06-12T02:38:34.192 INFO:tasks.rados.rados.0.smithi107.stdout:7270: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:34.192 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 727 (ObjNum 7269 snap 0 seq_num 7269) dirty exists 2020-06-12T02:38:34.193 INFO:tasks.rados.rados.0.smithi107.stdout:7270: left oid 5 (ObjNum 7269 snap 0 seq_num 7269) 2020-06-12T02:38:34.193 INFO:tasks.rados.rados.0.smithi107.stdout:7271: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:34.193 INFO:tasks.rados.rados.0.smithi107.stdout:7271: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:34.193 INFO:tasks.rados.rados.0.smithi107.stdout:7271: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:34.193 INFO:tasks.rados.rados.0.smithi107.stdout:7271: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:34.194 INFO:tasks.rados.rados.0.smithi107.stdout:7271: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:34.194 INFO:tasks.rados.rados.0.smithi107.stdout:7271: finishing write tid 7 to smithi10729407-12 2020-06-12T02:38:34.194 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1343 (ObjNum 7270 snap 0 seq_num 7270) dirty exists 2020-06-12T02:38:34.194 INFO:tasks.rados.rados.0.smithi107.stdout:7271: left oid 12 (ObjNum 7270 snap 0 seq_num 7270) 2020-06-12T02:38:34.195 INFO:tasks.rados.rados.0.smithi107.stdout:7272: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:34.195 INFO:tasks.rados.rados.0.smithi107.stdout:7267: done (6 left) 2020-06-12T02:38:34.195 INFO:tasks.rados.rados.0.smithi107.stdout:7268: done (5 left) 2020-06-12T02:38:34.196 INFO:tasks.rados.rados.0.smithi107.stdout:7269: done (4 left) 2020-06-12T02:38:34.196 INFO:tasks.rados.rados.0.smithi107.stdout:7270: done (3 left) 2020-06-12T02:38:34.196 INFO:tasks.rados.rados.0.smithi107.stdout:7271: done (2 left) 2020-06-12T02:38:34.196 INFO:tasks.rados.rados.0.smithi107.stdout:7273: done (1 left) 2020-06-12T02:38:34.196 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:34.197 INFO:tasks.rados.rados.0.smithi107.stdout:7274: seq_num 7273 ranges {1441792=614400,2056192=647168,2703360=557056} 2020-06-12T02:38:34.200 INFO:tasks.rados.rados.0.smithi107.stdout:7274: writing smithi10729407-15 from 1441792 to 2056192 tid 1 2020-06-12T02:38:34.203 INFO:tasks.rados.rados.0.smithi107.stdout:7274: writing smithi10729407-15 from 2056192 to 2703360 tid 2 2020-06-12T02:38:34.207 INFO:tasks.rados.rados.0.smithi107.stdout:7274: writing smithi10729407-15 from 2703360 to 3260416 tid 3 2020-06-12T02:38:34.207 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:34.207 INFO:tasks.rados.rados.0.smithi107.stdout:7275: seq_num 7274 ranges {12935168=630784,13565952=532480,14098432=458752,14557184=212992} 2020-06-12T02:38:34.232 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:34.232 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:34.241 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-06-12T02:38:34.277 INFO:tasks.rados.rados.0.smithi107.stdout:7275: writing smithi10729407-11 from 12935168 to 13565952 tid 1 2020-06-12T02:38:34.280 INFO:tasks.rados.rados.0.smithi107.stdout:7275: writing smithi10729407-11 from 13565952 to 14098432 tid 2 2020-06-12T02:38:34.282 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:34.280+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:34.283 INFO:tasks.rados.rados.0.smithi107.stdout:7275: writing smithi10729407-11 from 14098432 to 14557184 tid 3 2020-06-12T02:38:34.285 INFO:tasks.rados.rados.0.smithi107.stdout:7275: writing smithi10729407-11 from 14557184 to 14770176 tid 4 2020-06-12T02:38:34.286 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:34.286 INFO:tasks.rados.rados.0.smithi107.stdout:7276: seq_num 7275 ranges {0=647168,647168=417792,1064960=663552,1728512=122880} 2020-06-12T02:38:34.290 INFO:tasks.rados.rados.0.smithi107.stdout:7276: writing smithi10729407-6 from 0 to 647168 tid 1 2020-06-12T02:38:34.292 INFO:tasks.rados.rados.0.smithi107.stdout:7276: writing smithi10729407-6 from 647168 to 1064960 tid 2 2020-06-12T02:38:34.296 INFO:tasks.rados.rados.0.smithi107.stdout:7276: writing smithi10729407-6 from 1064960 to 1728512 tid 3 2020-06-12T02:38:34.312 INFO:tasks.rados.rados.0.smithi107.stdout:7276: writing smithi10729407-6 from 1728512 to 1851392 tid 4 2020-06-12T02:38:34.313 INFO:tasks.rados.rados.0.smithi107.stdout:7277: delete oid 3 current snap is 0 2020-06-12T02:38:34.313 INFO:tasks.rados.rados.0.smithi107.stdout:7272: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:34.313 INFO:tasks.rados.rados.0.smithi107.stdout:7272: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:34.313 INFO:tasks.rados.rados.0.smithi107.stdout:7272: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:34.314 INFO:tasks.rados.rados.0.smithi107.stdout:7272: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:34.314 INFO:tasks.rados.rados.0.smithi107.stdout:7272: finishing write tid 7 to smithi10729407-4 2020-06-12T02:38:34.314 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1373 (ObjNum 7271 snap 0 seq_num 7271) dirty exists 2020-06-12T02:38:34.314 INFO:tasks.rados.rados.0.smithi107.stdout:7272: left oid 4 (ObjNum 7271 snap 0 seq_num 7271) 2020-06-12T02:38:34.314 INFO:tasks.rados.rados.0.smithi107.stdout:7274: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:34.315 INFO:tasks.rados.rados.0.smithi107.stdout:7274: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:34.315 INFO:tasks.rados.rados.0.smithi107.stdout:7274: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:34.315 INFO:tasks.rados.rados.0.smithi107.stdout:7274: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:34.315 INFO:tasks.rados.rados.0.smithi107.stdout:7274: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:34.315 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1375 (ObjNum 7273 snap 0 seq_num 7273) dirty exists 2020-06-12T02:38:34.316 INFO:tasks.rados.rados.0.smithi107.stdout:7274: left oid 15 (ObjNum 7273 snap 0 seq_num 7273) 2020-06-12T02:38:34.316 INFO:tasks.rados.rados.0.smithi107.stdout:7275: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:34.316 INFO:tasks.rados.rados.0.smithi107.stdout:7275: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:34.316 INFO:tasks.rados.rados.0.smithi107.stdout:7275: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:34.316 INFO:tasks.rados.rados.0.smithi107.stdout:7275: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:34.316 INFO:tasks.rados.rados.0.smithi107.stdout:7275: finishing write tid 6 to smithi10729407-11 2020-06-12T02:38:34.317 INFO:tasks.rados.rados.0.smithi107.stdout:7276: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:34.317 INFO:tasks.rados.rados.0.smithi107.stdout:7275: finishing write tid 7 to smithi10729407-11 2020-06-12T02:38:34.317 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1286 (ObjNum 7274 snap 0 seq_num 7274) dirty exists 2020-06-12T02:38:34.317 INFO:tasks.rados.rados.0.smithi107.stdout:7275: left oid 11 (ObjNum 7274 snap 0 seq_num 7274) 2020-06-12T02:38:34.317 INFO:tasks.rados.rados.0.smithi107.stdout:7272: done (4 left) 2020-06-12T02:38:34.330 INFO:tasks.rados.rados.0.smithi107.stdout:7274: done (3 left) 2020-06-12T02:38:34.330 INFO:tasks.rados.rados.0.smithi107.stdout:7275: done (2 left) 2020-06-12T02:38:34.330 INFO:tasks.rados.rados.0.smithi107.stdout:7277: done (1 left) 2020-06-12T02:38:34.331 INFO:tasks.rados.rados.0.smithi107.stdout:7278: delete oid 12 current snap is 0 2020-06-12T02:38:34.331 INFO:tasks.rados.rados.0.smithi107.stdout:7278: done (1 left) 2020-06-12T02:38:34.331 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:34.331 INFO:tasks.rados.rados.0.smithi107.stdout:7279: seq_num 7278 ranges {3203072=696320,3899392=516096,4415488=434176,4849664=253952} 2020-06-12T02:38:34.332 INFO:tasks.rados.rados.0.smithi107.stdout:7279: writing smithi10729407-4 from 3203072 to 3899392 tid 1 2020-06-12T02:38:34.332 INFO:tasks.rados.rados.0.smithi107.stdout:7279: writing smithi10729407-4 from 3899392 to 4415488 tid 2 2020-06-12T02:38:34.334 INFO:tasks.rados.rados.0.smithi107.stdout:7279: writing smithi10729407-4 from 4415488 to 4849664 tid 3 2020-06-12T02:38:34.336 INFO:tasks.rados.rados.0.smithi107.stdout:7279: writing smithi10729407-4 from 4849664 to 5103616 tid 4 2020-06-12T02:38:34.336 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:34.336 INFO:tasks.rados.rados.0.smithi107.stdout:7280: seq_num 7279 ranges {0=712704,712704=802816,1515520=401408} 2020-06-12T02:38:34.340 INFO:tasks.rados.rados.0.smithi107.stdout:7280: writing smithi10729407-1 from 0 to 712704 tid 1 2020-06-12T02:38:34.345 INFO:tasks.rados.rados.0.smithi107.stdout:7280: writing smithi10729407-1 from 712704 to 1515520 tid 2 2020-06-12T02:38:34.348 INFO:tasks.rados.rados.0.smithi107.stdout:7280: writing smithi10729407-1 from 1515520 to 1916928 tid 3 2020-06-12T02:38:34.348 INFO:tasks.rados.rados.0.smithi107.stdout:7281: delete oid 13 current snap is 0 2020-06-12T02:38:34.348 INFO:tasks.rados.rados.0.smithi107.stdout:7276: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:34.349 INFO:tasks.rados.rados.0.smithi107.stdout:7276: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:34.349 INFO:tasks.rados.rados.0.smithi107.stdout:7276: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:34.349 INFO:tasks.rados.rados.0.smithi107.stdout:7276: finishing write tid 6 to smithi10729407-6 2020-06-12T02:38:34.349 INFO:tasks.rados.rados.0.smithi107.stdout:7276: finishing write tid 7 to smithi10729407-6 2020-06-12T02:38:34.349 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1317 (ObjNum 7275 snap 0 seq_num 7275) dirty exists 2020-06-12T02:38:34.350 INFO:tasks.rados.rados.0.smithi107.stdout:7276: left oid 6 (ObjNum 7275 snap 0 seq_num 7275) 2020-06-12T02:38:34.350 INFO:tasks.rados.rados.0.smithi107.stdout:7279: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:34.350 INFO:tasks.rados.rados.0.smithi107.stdout:7279: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:34.350 INFO:tasks.rados.rados.0.smithi107.stdout:7279: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:34.351 INFO:tasks.rados.rados.0.smithi107.stdout:7279: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:34.351 INFO:tasks.rados.rados.0.smithi107.stdout:7279: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:34.351 INFO:tasks.rados.rados.0.smithi107.stdout:7280: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:34.352 INFO:tasks.rados.rados.0.smithi107.stdout:7279: finishing write tid 7 to smithi10729407-4 2020-06-12T02:38:34.352 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1378 (ObjNum 7278 snap 0 seq_num 7278) dirty exists 2020-06-12T02:38:34.353 INFO:tasks.rados.rados.0.smithi107.stdout:7279: left oid 4 (ObjNum 7278 snap 0 seq_num 7278) 2020-06-12T02:38:34.356 INFO:tasks.rados.rados.0.smithi107.stdout:7276: done (3 left) 2020-06-12T02:38:34.356 INFO:tasks.rados.rados.0.smithi107.stdout:7279: done (2 left) 2020-06-12T02:38:34.356 INFO:tasks.rados.rados.0.smithi107.stdout:7281: done (1 left) 2020-06-12T02:38:34.357 INFO:tasks.rados.rados.0.smithi107.stdout:7282: delete oid 19 current snap is 0 2020-06-12T02:38:34.362 INFO:tasks.rados.rados.0.smithi107.stdout:7280: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:34.362 INFO:tasks.rados.rados.0.smithi107.stdout:7282: done (1 left) 2020-06-12T02:38:34.363 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:34.363 INFO:tasks.rados.rados.0.smithi107.stdout:7283: seq_num 7282 ranges {1179648=770048,1949696=450560,2400256=417792,2818048=335872} 2020-06-12T02:38:34.373 INFO:tasks.rados.rados.0.smithi107.stdout:7283: writing smithi10729407-9 from 1179648 to 1949696 tid 1 2020-06-12T02:38:34.376 INFO:tasks.rados.rados.0.smithi107.stdout:7283: writing smithi10729407-9 from 1949696 to 2400256 tid 2 2020-06-12T02:38:34.378 INFO:tasks.rados.rados.0.smithi107.stdout:7283: writing smithi10729407-9 from 2400256 to 2818048 tid 3 2020-06-12T02:38:34.381 INFO:tasks.rados.rados.0.smithi107.stdout:7283: writing smithi10729407-9 from 2818048 to 3153920 tid 4 2020-06-12T02:38:34.381 INFO:tasks.rados.rados.0.smithi107.stdout:7284: delete oid 19 current snap is 0 2020-06-12T02:38:34.382 INFO:tasks.rados.rados.0.smithi107.stdout:7280: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:34.382 INFO:tasks.rados.rados.0.smithi107.stdout:7280: finishing write tid 5 to smithi10729407-1 2020-06-12T02:38:34.382 INFO:tasks.rados.rados.0.smithi107.stdout:7280: finishing write tid 6 to smithi10729407-1 2020-06-12T02:38:34.383 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1321 (ObjNum 7279 snap 0 seq_num 7279) dirty exists 2020-06-12T02:38:34.383 INFO:tasks.rados.rados.0.smithi107.stdout:7280: left oid 1 (ObjNum 7279 snap 0 seq_num 7279) 2020-06-12T02:38:34.383 INFO:tasks.rados.rados.0.smithi107.stdout:7283: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:34.385 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:34.385 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:34.386 INFO:tasks.rados.rados.0.smithi107.stdout:7280: done (2 left) 2020-06-12T02:38:34.386 INFO:tasks.rados.rados.0.smithi107.stdout:7284: done (1 left) 2020-06-12T02:38:34.387 INFO:tasks.rados.rados.0.smithi107.stdout:7285: delete oid 4 current snap is 0 2020-06-12T02:38:34.387 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:34.382+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:34.388 INFO:tasks.rados.rados.0.smithi107.stdout:7283: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:34.389 INFO:tasks.rados.rados.0.smithi107.stdout:7285: done (1 left) 2020-06-12T02:38:34.389 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:34.390 INFO:tasks.rados.rados.0.smithi107.stdout:7286: seq_num 7285 ranges {1703936=417792,2121728=557056,2678784=589824,3268608=458752} 2020-06-12T02:38:34.393 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-06-12T02:38:34.402 INFO:tasks.rados.rados.0.smithi107.stdout:7286: writing smithi10729407-17 from 1703936 to 2121728 tid 1 2020-06-12T02:38:34.404 INFO:tasks.rados.rados.0.smithi107.stdout:7286: writing smithi10729407-17 from 2121728 to 2678784 tid 2 2020-06-12T02:38:34.408 INFO:tasks.rados.rados.0.smithi107.stdout:7286: writing smithi10729407-17 from 2678784 to 3268608 tid 3 2020-06-12T02:38:34.411 INFO:tasks.rados.rados.0.smithi107.stdout:7286: writing smithi10729407-17 from 3268608 to 3727360 tid 4 2020-06-12T02:38:34.411 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:38:34.411 INFO:tasks.rados.rados.0.smithi107.stdout:7287: seq_num 7286 ranges {0=434176,434176=442368,876544=794624,1671168=368640} 2020-06-12T02:38:34.413 INFO:tasks.rados.rados.0.smithi107.stdout:7287: writing smithi10729407-10 from 0 to 434176 tid 1 2020-06-12T02:38:34.416 INFO:tasks.rados.rados.0.smithi107.stdout:7287: writing smithi10729407-10 from 434176 to 876544 tid 2 2020-06-12T02:38:34.421 INFO:tasks.rados.rados.0.smithi107.stdout:7287: writing smithi10729407-10 from 876544 to 1671168 tid 3 2020-06-12T02:38:34.423 INFO:tasks.rados.rados.0.smithi107.stdout:7287: writing smithi10729407-10 from 1671168 to 2039808 tid 4 2020-06-12T02:38:34.424 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:38:34.424 INFO:tasks.rados.rados.0.smithi107.stdout:7288: seq_num 7287 ranges {0=450560,450560=729088,1179648=606208,1785856=270336} 2020-06-12T02:38:34.426 INFO:tasks.rados.rados.0.smithi107.stdout:7288: writing smithi10729407-13 from 0 to 450560 tid 1 2020-06-12T02:38:34.430 INFO:tasks.rados.rados.0.smithi107.stdout:7288: writing smithi10729407-13 from 450560 to 1179648 tid 2 2020-06-12T02:38:34.434 INFO:tasks.rados.rados.0.smithi107.stdout:7288: writing smithi10729407-13 from 1179648 to 1785856 tid 3 2020-06-12T02:38:34.436 INFO:tasks.rados.rados.0.smithi107.stdout:7288: writing smithi10729407-13 from 1785856 to 2056192 tid 4 2020-06-12T02:38:34.436 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:34.437 INFO:tasks.rados.rados.0.smithi107.stdout:7289: seq_num 7288 ranges {1753088=466944,2220032=614400,2834432=761856,3596288=229376} 2020-06-12T02:38:34.449 INFO:tasks.rados.rados.0.smithi107.stdout:7289: writing smithi10729407-5 from 1753088 to 2220032 tid 1 2020-06-12T02:38:34.455 INFO:tasks.rados.rados.0.smithi107.stdout:7289: writing smithi10729407-5 from 2220032 to 2834432 tid 2 2020-06-12T02:38:34.456 INFO:tasks.rados.rados.0.smithi107.stdout:7289: writing smithi10729407-5 from 2834432 to 3596288 tid 3 2020-06-12T02:38:34.459 INFO:tasks.rados.rados.0.smithi107.stdout:7289: writing smithi10729407-5 from 3596288 to 3825664 tid 4 2020-06-12T02:38:34.459 INFO:tasks.rados.rados.0.smithi107.stdout:7290: delete oid 12 current snap is 0 2020-06-12T02:38:34.459 INFO:tasks.rados.rados.0.smithi107.stdout:7283: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:34.460 INFO:tasks.rados.rados.0.smithi107.stdout:7283: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:34.460 INFO:tasks.rados.rados.0.smithi107.stdout:7283: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:34.460 INFO:tasks.rados.rados.0.smithi107.stdout:7283: finishing write tid 7 to smithi10729407-9 2020-06-12T02:38:34.460 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1378 (ObjNum 7282 snap 0 seq_num 7282) dirty exists 2020-06-12T02:38:34.460 INFO:tasks.rados.rados.0.smithi107.stdout:7283: left oid 9 (ObjNum 7282 snap 0 seq_num 7282) 2020-06-12T02:38:34.461 INFO:tasks.rados.rados.0.smithi107.stdout:7286: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:34.461 INFO:tasks.rados.rados.0.smithi107.stdout:7286: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:34.461 INFO:tasks.rados.rados.0.smithi107.stdout:7286: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:34.461 INFO:tasks.rados.rados.0.smithi107.stdout:7286: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:34.461 INFO:tasks.rados.rados.0.smithi107.stdout:7286: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:34.462 INFO:tasks.rados.rados.0.smithi107.stdout:7287: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:34.462 INFO:tasks.rados.rados.0.smithi107.stdout:7286: finishing write tid 7 to smithi10729407-17 2020-06-12T02:38:34.462 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1384 (ObjNum 7285 snap 0 seq_num 7285) dirty exists 2020-06-12T02:38:34.462 INFO:tasks.rados.rados.0.smithi107.stdout:7286: left oid 17 (ObjNum 7285 snap 0 seq_num 7285) 2020-06-12T02:38:34.463 INFO:tasks.rados.rados.0.smithi107.stdout:7287: finishing write tid 2 to smithi10729407-10 2020-06-12T02:38:34.463 INFO:tasks.rados.rados.0.smithi107.stdout:7287: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:34.463 INFO:tasks.rados.rados.0.smithi107.stdout:7287: finishing write tid 4 to smithi10729407-10 2020-06-12T02:38:34.463 INFO:tasks.rados.rados.0.smithi107.stdout:7287: finishing write tid 6 to smithi10729407-10 2020-06-12T02:38:34.464 INFO:tasks.rados.rados.0.smithi107.stdout:7287: finishing write tid 7 to smithi10729407-10 2020-06-12T02:38:34.464 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 499 (ObjNum 7286 snap 0 seq_num 7286) dirty exists 2020-06-12T02:38:34.464 INFO:tasks.rados.rados.0.smithi107.stdout:7287: left oid 10 (ObjNum 7286 snap 0 seq_num 7286) 2020-06-12T02:38:34.464 INFO:tasks.rados.rados.0.smithi107.stdout:7288: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:34.465 INFO:tasks.rados.rados.0.smithi107.stdout:7288: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:34.465 INFO:tasks.rados.rados.0.smithi107.stdout:7288: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:34.465 INFO:tasks.rados.rados.0.smithi107.stdout:7288: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:34.465 INFO:tasks.rados.rados.0.smithi107.stdout:7288: finishing write tid 6 to smithi10729407-13 2020-06-12T02:38:34.466 INFO:tasks.rados.rados.0.smithi107.stdout:7288: finishing write tid 7 to smithi10729407-13 2020-06-12T02:38:34.466 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1349 (ObjNum 7287 snap 0 seq_num 7287) dirty exists 2020-06-12T02:38:34.466 INFO:tasks.rados.rados.0.smithi107.stdout:7288: left oid 13 (ObjNum 7287 snap 0 seq_num 7287) 2020-06-12T02:38:34.467 INFO:tasks.rados.rados.0.smithi107.stdout:7283: done (5 left) 2020-06-12T02:38:34.468 INFO:tasks.rados.rados.0.smithi107.stdout:7286: done (4 left) 2020-06-12T02:38:34.468 INFO:tasks.rados.rados.0.smithi107.stdout:7287: done (3 left) 2020-06-12T02:38:34.468 INFO:tasks.rados.rados.0.smithi107.stdout:7288: done (2 left) 2020-06-12T02:38:34.468 INFO:tasks.rados.rados.0.smithi107.stdout:7290: done (1 left) 2020-06-12T02:38:34.469 INFO:tasks.rados.rados.0.smithi107.stdout:7291: delete oid 2 current snap is 0 2020-06-12T02:38:34.478 INFO:tasks.rados.rados.0.smithi107.stdout:7291: done (1 left) 2020-06-12T02:38:34.478 INFO:tasks.rados.rados.0.smithi107.stdout:7292: delete oid 13 current snap is 0 2020-06-12T02:38:34.479 INFO:tasks.rados.rados.0.smithi107.stdout:7289: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:34.479 INFO:tasks.rados.rados.0.smithi107.stdout:7289: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:34.479 INFO:tasks.rados.rados.0.smithi107.stdout:7292: done (1 left) 2020-06-12T02:38:34.479 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:34.480 INFO:tasks.rados.rados.0.smithi107.stdout:7293: seq_num 7292 ranges {0=532480,532480=598016,1130496=745472,1875968=262144} 2020-06-12T02:38:34.482 INFO:tasks.rados.rados.0.smithi107.stdout:7293: writing smithi10729407-19 from 0 to 532480 tid 1 2020-06-12T02:38:34.482 INFO:tasks.rados.rados.0.smithi107.stdout:7293: writing smithi10729407-19 from 532480 to 1130496 tid 2 2020-06-12T02:38:34.482 INFO:tasks.rados.rados.0.smithi107.stdout:7293: writing smithi10729407-19 from 1130496 to 1875968 tid 3 2020-06-12T02:38:34.483 INFO:tasks.rados.rados.0.smithi107.stdout:7293: writing smithi10729407-19 from 1875968 to 2138112 tid 4 2020-06-12T02:38:34.483 INFO:tasks.rados.rados.0.smithi107.stdout:7294: delete oid 3 current snap is 0 2020-06-12T02:38:34.483 INFO:tasks.rados.rados.0.smithi107.stdout:7289: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:34.484 INFO:tasks.rados.rados.0.smithi107.stdout:7289: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:34.484 INFO:tasks.rados.rados.0.smithi107.stdout:7289: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:34.484 INFO:tasks.rados.rados.0.smithi107.stdout:7289: finishing write tid 7 to smithi10729407-5 2020-06-12T02:38:34.485 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 732 (ObjNum 7288 snap 0 seq_num 7288) dirty exists 2020-06-12T02:38:34.485 INFO:tasks.rados.rados.0.smithi107.stdout:7289: left oid 5 (ObjNum 7288 snap 0 seq_num 7288) 2020-06-12T02:38:34.485 INFO:tasks.rados.rados.0.smithi107.stdout:7293: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:34.485 INFO:tasks.rados.rados.0.smithi107.stdout:7289: done (2 left) 2020-06-12T02:38:34.486 INFO:tasks.rados.rados.0.smithi107.stdout:7294: done (1 left) 2020-06-12T02:38:34.486 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:38:34.486 INFO:tasks.rados.rados.0.smithi107.stdout:7295: seq_num 7294 ranges {0=565248,565248=770048,1335296=761856,2097152=73728} 2020-06-12T02:38:34.488 INFO:tasks.rados.rados.0.smithi107.stdout:7295: writing smithi10729407-7 from 0 to 565248 tid 1 2020-06-12T02:38:34.490 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:34.485+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:34.490 INFO:tasks.rados.rados.0.smithi107.stdout:7295: writing smithi10729407-7 from 565248 to 1335296 tid 2 2020-06-12T02:38:34.494 INFO:tasks.rados.rados.0.smithi107.stdout:7295: writing smithi10729407-7 from 1335296 to 2097152 tid 3 2020-06-12T02:38:34.495 INFO:tasks.rados.rados.0.smithi107.stdout:7295: writing smithi10729407-7 from 2097152 to 2170880 tid 4 2020-06-12T02:38:34.496 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:34.496 INFO:tasks.rados.rados.0.smithi107.stdout:7296: seq_num 7295 ranges {0=581632,581632=647168,1228800=565248,1794048=393216} 2020-06-12T02:38:34.499 INFO:tasks.rados.rados.0.smithi107.stdout:7296: writing smithi10729407-14 from 0 to 581632 tid 1 2020-06-12T02:38:34.503 INFO:tasks.rados.rados.0.smithi107.stdout:7296: writing smithi10729407-14 from 581632 to 1228800 tid 2 2020-06-12T02:38:34.506 INFO:tasks.rados.rados.0.smithi107.stdout:7296: writing smithi10729407-14 from 1228800 to 1794048 tid 3 2020-06-12T02:38:34.509 INFO:tasks.rados.rados.0.smithi107.stdout:7296: writing smithi10729407-14 from 1794048 to 2187264 tid 4 2020-06-12T02:38:34.509 INFO:tasks.rados.rados.0.smithi107.stdout:7297: delete oid 3 current snap is 0 2020-06-12T02:38:34.510 INFO:tasks.rados.rados.0.smithi107.stdout:7293: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:34.510 INFO:tasks.rados.rados.0.smithi107.stdout:7295: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:34.510 INFO:tasks.rados.rados.0.smithi107.stdout:7293: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:34.511 INFO:tasks.rados.rados.0.smithi107.stdout:7293: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:34.511 INFO:tasks.rados.rados.0.smithi107.stdout:7293: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:34.511 INFO:tasks.rados.rados.0.smithi107.stdout:7293: finishing write tid 7 to smithi10729407-19 2020-06-12T02:38:34.512 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1352 (ObjNum 7292 snap 0 seq_num 7292) dirty exists 2020-06-12T02:38:34.512 INFO:tasks.rados.rados.0.smithi107.stdout:7293: left oid 19 (ObjNum 7292 snap 0 seq_num 7292) 2020-06-12T02:38:34.512 INFO:tasks.rados.rados.0.smithi107.stdout:7295: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:34.513 INFO:tasks.rados.rados.0.smithi107.stdout:7295: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:34.514 INFO:tasks.rados.rados.0.smithi107.stdout:7295: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:34.514 INFO:tasks.rados.rados.0.smithi107.stdout:7296: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:34.514 INFO:tasks.rados.rados.0.smithi107.stdout:7295: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:34.515 INFO:tasks.rados.rados.0.smithi107.stdout:7295: finishing write tid 7 to smithi10729407-7 2020-06-12T02:38:34.516 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1355 (ObjNum 7294 snap 0 seq_num 7294) dirty exists 2020-06-12T02:38:34.516 INFO:tasks.rados.rados.0.smithi107.stdout:7295: left oid 7 (ObjNum 7294 snap 0 seq_num 7294) 2020-06-12T02:38:34.516 INFO:tasks.rados.rados.0.smithi107.stdout:7293: done (3 left) 2020-06-12T02:38:34.516 INFO:tasks.rados.rados.0.smithi107.stdout:7295: done (2 left) 2020-06-12T02:38:34.517 INFO:tasks.rados.rados.0.smithi107.stdout:7297: done (1 left) 2020-06-12T02:38:34.517 INFO:tasks.rados.rados.0.smithi107.stdout:7298: delete oid 18 current snap is 0 2020-06-12T02:38:34.517 INFO:tasks.rados.rados.0.smithi107.stdout:7296: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:34.518 INFO:tasks.rados.rados.0.smithi107.stdout:7296: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:34.520 INFO:tasks.rados.rados.0.smithi107.stdout:7296: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:34.520 INFO:tasks.rados.rados.0.smithi107.stdout:7296: finishing write tid 6 to smithi10729407-14 2020-06-12T02:38:34.520 INFO:tasks.rados.rados.0.smithi107.stdout:7298: done (1 left) 2020-06-12T02:38:34.520 INFO:tasks.rados.rados.0.smithi107.stdout:7299: delete oid 11 current snap is 0 2020-06-12T02:38:34.522 INFO:tasks.rados.rados.0.smithi107.stdout:7296: finishing write tid 7 to smithi10729407-14 2020-06-12T02:38:34.522 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1291 (ObjNum 7295 snap 0 seq_num 7295) dirty exists 2020-06-12T02:38:34.523 INFO:tasks.rados.rados.0.smithi107.stdout:7296: left oid 14 (ObjNum 7295 snap 0 seq_num 7295) 2020-06-12T02:38:34.523 INFO:tasks.rados.rados.0.smithi107.stdout:7296: done (1 left) 2020-06-12T02:38:34.523 INFO:tasks.rados.rados.0.smithi107.stdout:7299: done (0 left) 2020-06-12T02:38:34.524 INFO:tasks.rados.rados.0.smithi107.stdout:7300: delete oid 4 current snap is 0 2020-06-12T02:38:34.526 INFO:tasks.rados.rados.0.smithi107.stdout:7300: done (0 left) 2020-06-12T02:38:34.526 INFO:tasks.rados.rados.0.smithi107.stdout:7301: delete oid 13 current snap is 0 2020-06-12T02:38:34.529 INFO:tasks.rados.rados.0.smithi107.stdout:7301: done (0 left) 2020-06-12T02:38:34.529 INFO:tasks.rados.rados.0.smithi107.stdout:7302: delete oid 17 current snap is 0 2020-06-12T02:38:34.533 INFO:tasks.rados.rados.0.smithi107.stdout:7302: done (0 left) 2020-06-12T02:38:34.533 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:34.533 INFO:tasks.rados.rados.0.smithi107.stdout:7303: seq_num 7302 ranges {0=704512,704512=688128,1392640=720896,2113536=196608} 2020-06-12T02:38:34.536 INFO:tasks.rados.rados.0.smithi107.stdout:7303: writing smithi10729407-16 from 0 to 704512 tid 1 2020-06-12T02:38:34.541 INFO:tasks.rados.rados.0.smithi107.stdout:7303: writing smithi10729407-16 from 704512 to 1392640 tid 2 2020-06-12T02:38:34.546 INFO:tasks.rados.rados.0.smithi107.stdout:7303: writing smithi10729407-16 from 1392640 to 2113536 tid 3 2020-06-12T02:38:34.547 INFO:tasks.rados.rados.0.smithi107.stdout:7303: writing smithi10729407-16 from 2113536 to 2310144 tid 4 2020-06-12T02:38:34.548 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:34.548 INFO:tasks.rados.rados.0.smithi107.stdout:7304: seq_num 7303 ranges {3825664=720896,4546560=573440,5120000=524288,5644288=475136,6119424=32768} 2020-06-12T02:38:34.554 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:34.554 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:34.564 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-06-12T02:38:34.572 INFO:tasks.rados.rados.0.smithi107.stdout:7304: writing smithi10729407-5 from 3825664 to 4546560 tid 1 2020-06-12T02:38:34.575 INFO:tasks.rados.rados.0.smithi107.stdout:7304: writing smithi10729407-5 from 4546560 to 5120000 tid 2 2020-06-12T02:38:34.579 INFO:tasks.rados.rados.0.smithi107.stdout:7304: writing smithi10729407-5 from 5120000 to 5644288 tid 3 2020-06-12T02:38:34.582 INFO:tasks.rados.rados.0.smithi107.stdout:7304: writing smithi10729407-5 from 5644288 to 6119424 tid 4 2020-06-12T02:38:34.583 INFO:tasks.rados.rados.0.smithi107.stdout:7304: writing smithi10729407-5 from 6119424 to 6152192 tid 5 2020-06-12T02:38:34.583 INFO:tasks.rados.rados.0.smithi107.stdout:7305: delete oid 10 current snap is 0 2020-06-12T02:38:34.584 INFO:tasks.rados.rados.0.smithi107.stdout:7303: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:34.584 INFO:tasks.rados.rados.0.smithi107.stdout:7303: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:34.584 INFO:tasks.rados.rados.0.smithi107.stdout:7303: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:34.584 INFO:tasks.rados.rados.0.smithi107.stdout:7303: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:34.585 INFO:tasks.rados.rados.0.smithi107.stdout:7303: finishing write tid 6 to smithi10729407-16 2020-06-12T02:38:34.585 INFO:tasks.rados.rados.0.smithi107.stdout:7303: finishing write tid 7 to smithi10729407-16 2020-06-12T02:38:34.585 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1312 (ObjNum 7302 snap 0 seq_num 7302) dirty exists 2020-06-12T02:38:34.586 INFO:tasks.rados.rados.0.smithi107.stdout:7303: left oid 16 (ObjNum 7302 snap 0 seq_num 7302) 2020-06-12T02:38:34.587 INFO:tasks.rados.rados.0.smithi107.stdout:7304: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:34.587 INFO:tasks.rados.rados.0.smithi107.stdout:7304: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:34.587 INFO:tasks.rados.rados.0.smithi107.stdout:7303: done (2 left) 2020-06-12T02:38:34.588 INFO:tasks.rados.rados.0.smithi107.stdout:7305: done (1 left) 2020-06-12T02:38:34.588 INFO:tasks.rados.rados.0.smithi107.stdout:7306: delete oid 8 current snap is 0 2020-06-12T02:38:34.590 INFO:tasks.rados.rados.0.smithi107.stdout:7306: done (1 left) 2020-06-12T02:38:34.590 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:38:34.590 INFO:tasks.rados.rados.0.smithi107.stdout:7307: seq_num 7306 ranges {0=770048,770048=679936,1449984=696320,2146304=229376} 2020-06-12T02:38:34.591 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:34.587+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:34.592 INFO:tasks.rados.rados.0.smithi107.stdout:7307: writing smithi10729407-2 from 0 to 770048 tid 1 2020-06-12T02:38:34.596 INFO:tasks.rados.rados.0.smithi107.stdout:7307: writing smithi10729407-2 from 770048 to 1449984 tid 2 2020-06-12T02:38:34.600 INFO:tasks.rados.rados.0.smithi107.stdout:7307: writing smithi10729407-2 from 1449984 to 2146304 tid 3 2020-06-12T02:38:34.602 INFO:tasks.rados.rados.0.smithi107.stdout:7307: writing smithi10729407-2 from 2146304 to 2375680 tid 4 2020-06-12T02:38:34.602 INFO:tasks.rados.rados.0.smithi107.stdout:7308: delete oid 17 current snap is 0 2020-06-12T02:38:34.603 INFO:tasks.rados.rados.0.smithi107.stdout:7304: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:34.603 INFO:tasks.rados.rados.0.smithi107.stdout:7304: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:34.603 INFO:tasks.rados.rados.0.smithi107.stdout:7304: finishing write tid 5 to smithi10729407-5 2020-06-12T02:38:34.604 INFO:tasks.rados.rados.0.smithi107.stdout:7304: finishing write tid 7 to smithi10729407-5 2020-06-12T02:38:34.604 INFO:tasks.rados.rados.0.smithi107.stdout:7304: finishing write tid 8 to smithi10729407-5 2020-06-12T02:38:34.604 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 738 (ObjNum 7303 snap 0 seq_num 7303) dirty exists 2020-06-12T02:38:34.605 INFO:tasks.rados.rados.0.smithi107.stdout:7304: left oid 5 (ObjNum 7303 snap 0 seq_num 7303) 2020-06-12T02:38:34.605 INFO:tasks.rados.rados.0.smithi107.stdout:7307: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:34.606 INFO:tasks.rados.rados.0.smithi107.stdout:7304: done (2 left) 2020-06-12T02:38:34.606 INFO:tasks.rados.rados.0.smithi107.stdout:7308: done (1 left) 2020-06-12T02:38:34.607 INFO:tasks.rados.rados.0.smithi107.stdout:7309: delete oid 1 current snap is 0 2020-06-12T02:38:34.610 INFO:tasks.rados.rados.0.smithi107.stdout:7309: done (1 left) 2020-06-12T02:38:34.611 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:34.611 INFO:tasks.rados.rados.0.smithi107.stdout:7310: seq_num 7309 ranges {0=16384} 2020-06-12T02:38:34.611 INFO:tasks.rados.rados.0.smithi107.stdout:7310: writing smithi10729407-11 from 0 to 16384 tid 1 2020-06-12T02:38:34.612 INFO:tasks.rados.rados.0.smithi107.stdout:7311: delete oid 1 current snap is 0 2020-06-12T02:38:34.613 INFO:tasks.rados.rados.0.smithi107.stdout:7307: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:34.613 INFO:tasks.rados.rados.0.smithi107.stdout:7307: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:34.613 INFO:tasks.rados.rados.0.smithi107.stdout:7310: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:34.617 INFO:tasks.rados.rados.0.smithi107.stdout:7311: done (2 left) 2020-06-12T02:38:34.618 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:34.618 INFO:tasks.rados.rados.0.smithi107.stdout:7312: seq_num 7311 ranges {2310144=49152} 2020-06-12T02:38:34.629 INFO:tasks.rados.rados.0.smithi107.stdout:7312: writing smithi10729407-16 from 2310144 to 2359296 tid 1 2020-06-12T02:38:34.629 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:34.629 INFO:tasks.rados.rados.0.smithi107.stdout:7313: seq_num 7312 ranges {0=65536} 2020-06-12T02:38:34.630 INFO:tasks.rados.rados.0.smithi107.stdout:7313: writing smithi10729407-4 from 0 to 65536 tid 1 2020-06-12T02:38:34.630 INFO:tasks.rados.rados.0.smithi107.stdout:7314: delete oid 12 current snap is 0 2020-06-12T02:38:34.630 INFO:tasks.rados.rados.0.smithi107.stdout:7310: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:34.631 INFO:tasks.rados.rados.0.smithi107.stdout:7307: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:34.631 INFO:tasks.rados.rados.0.smithi107.stdout:7307: finishing write tid 6 to smithi10729407-2 2020-06-12T02:38:34.631 INFO:tasks.rados.rados.0.smithi107.stdout:7310: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:34.631 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1294 (ObjNum 7309 snap 0 seq_num 7309) dirty exists 2020-06-12T02:38:34.632 INFO:tasks.rados.rados.0.smithi107.stdout:7310: left oid 11 (ObjNum 7309 snap 0 seq_num 7309) 2020-06-12T02:38:34.632 INFO:tasks.rados.rados.0.smithi107.stdout:7307: finishing write tid 7 to smithi10729407-2 2020-06-12T02:38:34.632 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 694 (ObjNum 7306 snap 0 seq_num 7306) dirty exists 2020-06-12T02:38:34.632 INFO:tasks.rados.rados.0.smithi107.stdout:7307: left oid 2 (ObjNum 7306 snap 0 seq_num 7306) 2020-06-12T02:38:34.633 INFO:tasks.rados.rados.0.smithi107.stdout:7312: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:34.633 INFO:tasks.rados.rados.0.smithi107.stdout:7313: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:34.634 INFO:tasks.rados.rados.0.smithi107.stdout:7307: done (4 left) 2020-06-12T02:38:34.634 INFO:tasks.rados.rados.0.smithi107.stdout:7310: done (3 left) 2020-06-12T02:38:34.634 INFO:tasks.rados.rados.0.smithi107.stdout:7314: done (2 left) 2020-06-12T02:38:34.634 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:38:34.634 INFO:tasks.rados.rados.0.smithi107.stdout:7315: seq_num 7314 ranges {0=98304} 2020-06-12T02:38:34.635 INFO:tasks.rados.rados.0.smithi107.stdout:7315: writing smithi10729407-3 from 0 to 98304 tid 1 2020-06-12T02:38:34.635 INFO:tasks.rados.rados.0.smithi107.stdout:7316: delete oid 18 current snap is 0 2020-06-12T02:38:34.642 INFO:tasks.rados.rados.0.smithi107.stdout:7313: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:34.642 INFO:tasks.rados.rados.0.smithi107.stdout:7312: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:34.642 INFO:tasks.rados.rados.0.smithi107.stdout:7312: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:34.643 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1314 (ObjNum 7311 snap 0 seq_num 7311) dirty exists 2020-06-12T02:38:34.644 INFO:tasks.rados.rados.0.smithi107.stdout:7312: left oid 16 (ObjNum 7311 snap 0 seq_num 7311) 2020-06-12T02:38:34.644 INFO:tasks.rados.rados.0.smithi107.stdout:7313: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:34.644 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1389 (ObjNum 7312 snap 0 seq_num 7312) dirty exists 2020-06-12T02:38:34.644 INFO:tasks.rados.rados.0.smithi107.stdout:7313: left oid 4 (ObjNum 7312 snap 0 seq_num 7312) 2020-06-12T02:38:34.645 INFO:tasks.rados.rados.0.smithi107.stdout:7312: done (3 left) 2020-06-12T02:38:34.645 INFO:tasks.rados.rados.0.smithi107.stdout:7313: done (2 left) 2020-06-12T02:38:34.645 INFO:tasks.rados.rados.0.smithi107.stdout:7316: done (1 left) 2020-06-12T02:38:34.646 INFO:tasks.rados.rados.0.smithi107.stdout:7317: delete oid 16 current snap is 0 2020-06-12T02:38:34.646 INFO:tasks.rados.rados.0.smithi107.stdout:7315: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:34.646 INFO:tasks.rados.rados.0.smithi107.stdout:7315: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:34.647 INFO:tasks.rados.rados.0.smithi107.stdout:7315: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:34.647 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1316 (ObjNum 7314 snap 0 seq_num 7314) dirty exists 2020-06-12T02:38:34.647 INFO:tasks.rados.rados.0.smithi107.stdout:7315: left oid 3 (ObjNum 7314 snap 0 seq_num 7314) 2020-06-12T02:38:34.648 INFO:tasks.rados.rados.0.smithi107.stdout:7315: done (1 left) 2020-06-12T02:38:34.648 INFO:tasks.rados.rados.0.smithi107.stdout:7317: done (0 left) 2020-06-12T02:38:34.648 INFO:tasks.rados.rados.0.smithi107.stdout:7318: delete oid 2 current snap is 0 2020-06-12T02:38:34.648 INFO:tasks.rados.rados.0.smithi107.stdout:7318: done (0 left) 2020-06-12T02:38:34.649 INFO:tasks.rados.rados.0.smithi107.stdout:7319: delete oid 17 current snap is 0 2020-06-12T02:38:34.653 INFO:tasks.rados.rados.0.smithi107.stdout:7319: done (0 left) 2020-06-12T02:38:34.653 INFO:tasks.rados.rados.0.smithi107.stdout:7320: delete oid 1 current snap is 0 2020-06-12T02:38:34.659 INFO:tasks.rados.rados.0.smithi107.stdout:7320: done (0 left) 2020-06-12T02:38:34.660 INFO:tasks.rados.rados.0.smithi107.stdout:7321: delete oid 13 current snap is 0 2020-06-12T02:38:34.660 INFO:tasks.rados.rados.0.smithi107.stdout:7321: done (0 left) 2020-06-12T02:38:34.660 INFO:tasks.rados.rados.0.smithi107.stdout:7322: delete oid 1 current snap is 0 2020-06-12T02:38:34.661 INFO:tasks.rados.rados.0.smithi107.stdout:7322: done (0 left) 2020-06-12T02:38:34.661 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:38:34.661 INFO:tasks.rados.rados.0.smithi107.stdout:7323: seq_num 7322 ranges {0=237568} 2020-06-12T02:38:34.662 INFO:tasks.rados.rados.0.smithi107.stdout:7323: writing smithi10729407-2 from 0 to 237568 tid 1 2020-06-12T02:38:34.662 INFO:tasks.rados.rados.0.smithi107.stdout:7324: delete oid 10 current snap is 0 2020-06-12T02:38:34.664 INFO:tasks.rados.rados.0.smithi107.stdout:7324: done (1 left) 2020-06-12T02:38:34.664 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:38:34.664 INFO:tasks.rados.rados.0.smithi107.stdout:7325: seq_num 7324 ranges {2170880=270336} 2020-06-12T02:38:34.677 INFO:tasks.rados.rados.0.smithi107.stdout:7325: writing smithi10729407-7 from 2170880 to 2441216 tid 1 2020-06-12T02:38:34.678 INFO:tasks.rados.rados.0.smithi107.stdout:7326: delete oid 9 current snap is 0 2020-06-12T02:38:34.678 INFO:tasks.rados.rados.0.smithi107.stdout:7323: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:34.678 INFO:tasks.rados.rados.0.smithi107.stdout:7323: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:34.678 INFO:tasks.rados.rados.0.smithi107.stdout:7323: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:34.679 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 697 (ObjNum 7322 snap 0 seq_num 7322) dirty exists 2020-06-12T02:38:34.679 INFO:tasks.rados.rados.0.smithi107.stdout:7323: left oid 2 (ObjNum 7322 snap 0 seq_num 7322) 2020-06-12T02:38:34.680 INFO:tasks.rados.rados.0.smithi107.stdout:7323: done (2 left) 2020-06-12T02:38:34.680 INFO:tasks.rados.rados.0.smithi107.stdout:7326: done (1 left) 2020-06-12T02:38:34.680 INFO:tasks.rados.rados.0.smithi107.stdout:7327: delete oid 4 current snap is 0 2020-06-12T02:38:34.682 INFO:tasks.rados.rados.0.smithi107.stdout:7325: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:34.683 INFO:tasks.rados.rados.0.smithi107.stdout:7327: done (1 left) 2020-06-12T02:38:34.683 INFO:tasks.rados.rados.0.smithi107.stdout:7328: delete oid 19 current snap is 0 2020-06-12T02:38:34.686 INFO:tasks.rados.rados.0.smithi107.stdout:7328: done (1 left) 2020-06-12T02:38:34.686 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:34.686 INFO:tasks.rados.rados.0.smithi107.stdout:7329: seq_num 7328 ranges {0=335872} 2020-06-12T02:38:34.688 INFO:tasks.rados.rados.0.smithi107.stdout:7329: writing smithi10729407-19 from 0 to 335872 tid 1 2020-06-12T02:38:34.688 INFO:tasks.rados.rados.0.smithi107.stdout:7330: delete oid 5 current snap is 0 2020-06-12T02:38:34.688 INFO:tasks.rados.rados.0.smithi107.stdout:7325: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:34.690 INFO:tasks.rados.rados.0.smithi107.stdout:7325: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:34.691 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1359 (ObjNum 7324 snap 0 seq_num 7324) dirty exists 2020-06-12T02:38:34.691 INFO:tasks.rados.rados.0.smithi107.stdout:7325: left oid 7 (ObjNum 7324 snap 0 seq_num 7324) 2020-06-12T02:38:34.691 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:34.687+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:34.692 INFO:tasks.rados.rados.0.smithi107.stdout:7325: done (2 left) 2020-06-12T02:38:34.692 INFO:tasks.rados.rados.0.smithi107.stdout:7330: done (1 left) 2020-06-12T02:38:34.692 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:34.693 INFO:tasks.rados.rados.0.smithi107.stdout:7331: seq_num 7330 ranges {0=368640} 2020-06-12T02:38:34.694 INFO:tasks.rados.rados.0.smithi107.stdout:7331: writing smithi10729407-12 from 0 to 368640 tid 1 2020-06-12T02:38:34.694 INFO:tasks.rados.rados.0.smithi107.stdout:7332: delete oid 14 current snap is 0 2020-06-12T02:38:34.697 INFO:tasks.rados.rados.0.smithi107.stdout:7332: done (2 left) 2020-06-12T02:38:34.698 INFO:tasks.rados.rados.0.smithi107.stdout:7333: delete oid 6 current snap is 0 2020-06-12T02:38:34.704 INFO:tasks.rados.rados.0.smithi107.stdout:7329: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:34.704 INFO:tasks.rados.rados.0.smithi107.stdout:7333: done (2 left) 2020-06-12T02:38:34.704 INFO:tasks.rados.rados.0.smithi107.stdout:7334: delete oid 4 current snap is 0 2020-06-12T02:38:34.705 INFO:tasks.rados.rados.0.smithi107.stdout:7331: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:34.705 INFO:tasks.rados.rados.0.smithi107.stdout:7331: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:34.705 INFO:tasks.rados.rados.0.smithi107.stdout:7329: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:34.706 INFO:tasks.rados.rados.0.smithi107.stdout:7334: done (2 left) 2020-06-12T02:38:34.706 INFO:tasks.rados.rados.0.smithi107.stdout:7335: delete oid 20 current snap is 0 2020-06-12T02:38:34.707 INFO:tasks.rados.rados.0.smithi107.stdout:7331: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:34.707 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1357 (ObjNum 7330 snap 0 seq_num 7330) dirty exists 2020-06-12T02:38:34.708 INFO:tasks.rados.rados.0.smithi107.stdout:7331: left oid 12 (ObjNum 7330 snap 0 seq_num 7330) 2020-06-12T02:38:34.708 INFO:tasks.rados.rados.0.smithi107.stdout:7329: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:34.708 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1358 (ObjNum 7328 snap 0 seq_num 7328) dirty exists 2020-06-12T02:38:34.709 INFO:tasks.rados.rados.0.smithi107.stdout:7329: left oid 19 (ObjNum 7328 snap 0 seq_num 7328) 2020-06-12T02:38:34.709 INFO:tasks.rados.rados.0.smithi107.stdout:7329: done (2 left) 2020-06-12T02:38:34.709 INFO:tasks.rados.rados.0.smithi107.stdout:7331: done (1 left) 2020-06-12T02:38:34.709 INFO:tasks.rados.rados.0.smithi107.stdout:7335: done (0 left) 2020-06-12T02:38:34.709 INFO:tasks.rados.rados.0.smithi107.stdout:7336: delete oid 10 current snap is 0 2020-06-12T02:38:34.710 INFO:tasks.rados.rados.0.smithi107.stdout:7336: done (0 left) 2020-06-12T02:38:34.710 INFO:tasks.rados.rados.0.smithi107.stdout:7337: delete oid 5 current snap is 0 2020-06-12T02:38:34.716 INFO:tasks.rados.rados.0.smithi107.stdout:7337: done (0 left) 2020-06-12T02:38:34.717 INFO:tasks.rados.rados.0.smithi107.stdout:7338: delete oid 1 current snap is 0 2020-06-12T02:38:34.717 INFO:tasks.rados.rados.0.smithi107.stdout:7338: done (0 left) 2020-06-12T02:38:34.717 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:34.717 INFO:tasks.rados.rados.0.smithi107.stdout:7339: seq_num 7338 ranges {0=499712} 2020-06-12T02:38:34.717 INFO:tasks.rados.rados.0.smithi107.stdout:7339: writing smithi10729407-14 from 0 to 499712 tid 1 2020-06-12T02:38:34.718 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:34.718 INFO:tasks.rados.rados.0.smithi107.stdout:7340: seq_num 7339 ranges {0=516096} 2020-06-12T02:38:34.719 INFO:tasks.rados.rados.0.smithi107.stdout:7340: writing smithi10729407-6 from 0 to 516096 tid 1 2020-06-12T02:38:34.720 INFO:tasks.rados.rados.0.smithi107.stdout:7341: delete oid 4 current snap is 0 2020-06-12T02:38:34.721 INFO:tasks.rados.rados.0.smithi107.stdout:7339: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:34.724 INFO:tasks.rados.rados.0.smithi107.stdout:7341: done (2 left) 2020-06-12T02:38:34.724 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:34.724 INFO:tasks.rados.rados.0.smithi107.stdout:7342: seq_num 7341 ranges {2441216=557056} 2020-06-12T02:38:34.741 INFO:tasks.rados.rados.0.smithi107.stdout:7342: writing smithi10729407-7 from 2441216 to 2998272 tid 1 2020-06-12T02:38:34.741 INFO:tasks.rados.rados.0.smithi107.stdout:7343: delete oid 12 current snap is 0 2020-06-12T02:38:34.741 INFO:tasks.rados.rados.0.smithi107.stdout:7339: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:34.742 INFO:tasks.rados.rados.0.smithi107.stdout:7340: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:34.742 INFO:tasks.rados.rados.0.smithi107.stdout:7339: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:34.742 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1297 (ObjNum 7338 snap 0 seq_num 7338) dirty exists 2020-06-12T02:38:34.742 INFO:tasks.rados.rados.0.smithi107.stdout:7339: left oid 14 (ObjNum 7338 snap 0 seq_num 7338) 2020-06-12T02:38:34.742 INFO:tasks.rados.rados.0.smithi107.stdout:7340: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:34.742 INFO:tasks.rados.rados.0.smithi107.stdout:7340: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:34.743 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1329 (ObjNum 7339 snap 0 seq_num 7339) dirty exists 2020-06-12T02:38:34.743 INFO:tasks.rados.rados.0.smithi107.stdout:7340: left oid 6 (ObjNum 7339 snap 0 seq_num 7339) 2020-06-12T02:38:34.743 INFO:tasks.rados.rados.0.smithi107.stdout:7339: done (3 left) 2020-06-12T02:38:34.744 INFO:tasks.rados.rados.0.smithi107.stdout:7340: done (2 left) 2020-06-12T02:38:34.744 INFO:tasks.rados.rados.0.smithi107.stdout:7343: done (1 left) 2020-06-12T02:38:34.744 INFO:tasks.rados.rados.0.smithi107.stdout:7344: delete oid 11 current snap is 0 2020-06-12T02:38:34.750 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:34.750 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:34.751 INFO:tasks.rados.rados.0.smithi107.stdout:7342: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:34.753 INFO:tasks.rados.rados.0.smithi107.stdout:7342: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:34.754 INFO:tasks.rados.rados.0.smithi107.stdout:7344: done (1 left) 2020-06-12T02:38:34.754 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:38:34.754 INFO:tasks.rados.rados.0.smithi107.stdout:7345: seq_num 7344 ranges {0=606208} 2020-06-12T02:38:34.757 INFO:tasks.rados.rados.0.smithi107.stdout:7345: writing smithi10729407-20 from 0 to 606208 tid 1 2020-06-12T02:38:34.758 INFO:tasks.rados.rados.0.smithi107.stdout:7346: delete oid 2 current snap is 0 2020-06-12T02:38:34.758 INFO:tasks.rados.rados.0.smithi107.stdout:7342: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:34.758 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1361 (ObjNum 7341 snap 0 seq_num 7341) dirty exists 2020-06-12T02:38:34.758 INFO:tasks.rados.rados.0.smithi107.stdout:7342: left oid 7 (ObjNum 7341 snap 0 seq_num 7341) 2020-06-12T02:38:34.760 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-06-12T02:38:34.761 INFO:tasks.rados.rados.0.smithi107.stdout:7342: done (2 left) 2020-06-12T02:38:34.761 INFO:tasks.rados.rados.0.smithi107.stdout:7346: done (1 left) 2020-06-12T02:38:34.761 INFO:tasks.rados.rados.0.smithi107.stdout:7347: delete oid 17 current snap is 0 2020-06-12T02:38:34.768 INFO:tasks.rados.rados.0.smithi107.stdout:7347: done (1 left) 2020-06-12T02:38:34.768 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:34.768 INFO:tasks.rados.rados.0.smithi107.stdout:7348: seq_num 7347 ranges {0=655360} 2020-06-12T02:38:34.772 INFO:tasks.rados.rados.0.smithi107.stdout:7348: writing smithi10729407-17 from 0 to 655360 tid 1 2020-06-12T02:38:34.772 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:38:34.772 INFO:tasks.rados.rados.0.smithi107.stdout:7349: seq_num 7348 ranges {0=671744} 2020-06-12T02:38:34.776 INFO:tasks.rados.rados.0.smithi107.stdout:7349: writing smithi10729407-16 from 0 to 671744 tid 1 2020-06-12T02:38:34.776 INFO:tasks.rados.rados.0.smithi107.stdout:7350: delete oid 18 current snap is 0 2020-06-12T02:38:34.776 INFO:tasks.rados.rados.0.smithi107.stdout:7345: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:34.776 INFO:tasks.rados.rados.0.smithi107.stdout:7345: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:34.777 INFO:tasks.rados.rados.0.smithi107.stdout:7345: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:34.777 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 544 (ObjNum 7344 snap 0 seq_num 7344) dirty exists 2020-06-12T02:38:34.777 INFO:tasks.rados.rados.0.smithi107.stdout:7345: left oid 20 (ObjNum 7344 snap 0 seq_num 7344) 2020-06-12T02:38:34.778 INFO:tasks.rados.rados.0.smithi107.stdout:7348: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:34.782 INFO:tasks.rados.rados.0.smithi107.stdout:7348: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:34.785 INFO:tasks.rados.rados.0.smithi107.stdout:7349: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:34.786 INFO:tasks.rados.rados.0.smithi107.stdout:7348: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:34.786 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1396 (ObjNum 7347 snap 0 seq_num 7347) dirty exists 2020-06-12T02:38:34.786 INFO:tasks.rados.rados.0.smithi107.stdout:7348: left oid 17 (ObjNum 7347 snap 0 seq_num 7347) 2020-06-12T02:38:34.786 INFO:tasks.rados.rados.0.smithi107.stdout:7349: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:34.786 INFO:tasks.rados.rados.0.smithi107.stdout:7345: done (3 left) 2020-06-12T02:38:34.787 INFO:tasks.rados.rados.0.smithi107.stdout:7348: done (2 left) 2020-06-12T02:38:34.787 INFO:tasks.rados.rados.0.smithi107.stdout:7350: done (1 left) 2020-06-12T02:38:34.787 INFO:tasks.rados.rados.0.smithi107.stdout:7351: delete oid 1 current snap is 0 2020-06-12T02:38:34.787 INFO:tasks.rados.rados.0.smithi107.stdout:7349: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:34.788 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1319 (ObjNum 7348 snap 0 seq_num 7348) dirty exists 2020-06-12T02:38:34.788 INFO:tasks.rados.rados.0.smithi107.stdout:7349: left oid 16 (ObjNum 7348 snap 0 seq_num 7348) 2020-06-12T02:38:34.788 INFO:tasks.rados.rados.0.smithi107.stdout:7349: done (1 left) 2020-06-12T02:38:34.788 INFO:tasks.rados.rados.0.smithi107.stdout:7351: done (0 left) 2020-06-12T02:38:34.789 INFO:tasks.rados.rados.0.smithi107.stdout:7352: delete oid 11 current snap is 0 2020-06-12T02:38:34.790 INFO:tasks.rados.rados.0.smithi107.stdout:7352: done (0 left) 2020-06-12T02:38:34.790 INFO:tasks.rados.rados.0.smithi107.stdout:7353: delete oid 1 current snap is 0 2020-06-12T02:38:34.792 INFO:tasks.rados.rados.0.smithi107.stdout:7353: done (0 left) 2020-06-12T02:38:34.792 INFO:tasks.rados.rados.0.smithi107.stdout:7354: delete oid 19 current snap is 0 2020-06-12T02:38:34.793 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:34.789+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:34.794 INFO:tasks.rados.rados.0.smithi107.stdout:7354: done (0 left) 2020-06-12T02:38:34.795 INFO:tasks.rados.rados.0.smithi107.stdout:7355: delete oid 17 current snap is 0 2020-06-12T02:38:34.797 INFO:tasks.rados.rados.0.smithi107.stdout:7355: done (0 left) 2020-06-12T02:38:34.797 INFO:tasks.rados.rados.0.smithi107.stdout:7356: delete oid 7 current snap is 0 2020-06-12T02:38:34.799 INFO:tasks.rados.rados.0.smithi107.stdout:7356: done (0 left) 2020-06-12T02:38:34.799 INFO:tasks.rados.rados.0.smithi107.stdout:7357: delete oid 20 current snap is 0 2020-06-12T02:38:34.801 INFO:tasks.rados.rados.0.smithi107.stdout:7357: done (0 left) 2020-06-12T02:38:34.802 INFO:tasks.rados.rados.0.smithi107.stdout:7358: delete oid 13 current snap is 0 2020-06-12T02:38:34.804 INFO:tasks.rados.rados.0.smithi107.stdout:7358: done (0 left) 2020-06-12T02:38:34.804 INFO:tasks.rados.rados.0.smithi107.stdout:7359: delete oid 5 current snap is 0 2020-06-12T02:38:34.806 INFO:tasks.rados.rados.0.smithi107.stdout:7359: done (0 left) 2020-06-12T02:38:34.806 INFO:tasks.rados.rados.0.smithi107.stdout:7360: delete oid 11 current snap is 0 2020-06-12T02:38:34.808 INFO:tasks.rados.rados.0.smithi107.stdout:7360: done (0 left) 2020-06-12T02:38:34.808 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:34.808 INFO:tasks.rados.rados.0.smithi107.stdout:7361: seq_num 7360 ranges {0=475136,475136=401408} 2020-06-12T02:38:34.811 INFO:tasks.rados.rados.0.smithi107.stdout:7361: writing smithi10729407-8 from 0 to 475136 tid 1 2020-06-12T02:38:34.813 INFO:tasks.rados.rados.0.smithi107.stdout:7361: writing smithi10729407-8 from 475136 to 876544 tid 2 2020-06-12T02:38:34.813 INFO:tasks.rados.rados.0.smithi107.stdout:7362: delete oid 19 current snap is 0 2020-06-12T02:38:34.820 INFO:tasks.rados.rados.0.smithi107.stdout:7362: done (1 left) 2020-06-12T02:38:34.820 INFO:tasks.rados.rados.0.smithi107.stdout:7363: delete oid 5 current snap is 0 2020-06-12T02:38:34.820 INFO:tasks.rados.rados.0.smithi107.stdout:7361: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:34.822 INFO:tasks.rados.rados.0.smithi107.stdout:7361: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:34.823 INFO:tasks.rados.rados.0.smithi107.stdout:7363: done (1 left) 2020-06-12T02:38:34.823 INFO:tasks.rados.rados.0.smithi107.stdout:7364: delete oid 6 current snap is 0 2020-06-12T02:38:34.823 INFO:tasks.rados.rados.0.smithi107.stdout:7361: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:34.826 INFO:tasks.rados.rados.0.smithi107.stdout:7361: finishing write tid 5 to smithi10729407-8 2020-06-12T02:38:34.826 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1379 (ObjNum 7360 snap 0 seq_num 7360) dirty exists 2020-06-12T02:38:34.826 INFO:tasks.rados.rados.0.smithi107.stdout:7361: left oid 8 (ObjNum 7360 snap 0 seq_num 7360) 2020-06-12T02:38:34.826 INFO:tasks.rados.rados.0.smithi107.stdout:7361: done (1 left) 2020-06-12T02:38:34.827 INFO:tasks.rados.rados.0.smithi107.stdout:7364: done (0 left) 2020-06-12T02:38:34.827 INFO:tasks.rados.rados.0.smithi107.stdout:7365: delete oid 16 current snap is 0 2020-06-12T02:38:34.831 INFO:tasks.rados.rados.0.smithi107.stdout:7365: done (0 left) 2020-06-12T02:38:34.831 INFO:tasks.rados.rados.0.smithi107.stdout:7366: delete oid 10 current snap is 0 2020-06-12T02:38:34.833 INFO:tasks.rados.rados.0.smithi107.stdout:7366: done (0 left) 2020-06-12T02:38:34.833 INFO:tasks.rados.rados.0.smithi107.stdout:7367: delete oid 9 current snap is 0 2020-06-12T02:38:34.839 INFO:tasks.rados.rados.0.smithi107.stdout:7367: done (0 left) 2020-06-12T02:38:34.839 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:38:34.839 INFO:tasks.rados.rados.0.smithi107.stdout:7368: seq_num 7367 ranges {0=589824,589824=401408} 2020-06-12T02:38:34.841 INFO:tasks.rados.rados.0.smithi107.stdout:7368: writing smithi10729407-2 from 0 to 589824 tid 1 2020-06-12T02:38:34.844 INFO:tasks.rados.rados.0.smithi107.stdout:7368: writing smithi10729407-2 from 589824 to 991232 tid 2 2020-06-12T02:38:34.844 INFO:tasks.rados.rados.0.smithi107.stdout:7369: delete oid 16 current snap is 0 2020-06-12T02:38:34.846 INFO:tasks.rados.rados.0.smithi107.stdout:7368: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:34.847 INFO:tasks.rados.rados.0.smithi107.stdout:7369: done (1 left) 2020-06-12T02:38:34.847 INFO:tasks.rados.rados.0.smithi107.stdout:7370: delete oid 19 current snap is 0 2020-06-12T02:38:34.851 INFO:tasks.rados.rados.0.smithi107.stdout:7370: done (1 left) 2020-06-12T02:38:34.851 INFO:tasks.rados.rados.0.smithi107.stdout:7371: delete oid 9 current snap is 0 2020-06-12T02:38:34.852 INFO:tasks.rados.rados.0.smithi107.stdout:7368: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:34.852 INFO:tasks.rados.rados.0.smithi107.stdout:7368: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:34.853 INFO:tasks.rados.rados.0.smithi107.stdout:7368: finishing write tid 5 to smithi10729407-2 2020-06-12T02:38:34.853 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 701 (ObjNum 7367 snap 0 seq_num 7367) dirty exists 2020-06-12T02:38:34.854 INFO:tasks.rados.rados.0.smithi107.stdout:7368: left oid 2 (ObjNum 7367 snap 0 seq_num 7367) 2020-06-12T02:38:34.854 INFO:tasks.rados.rados.0.smithi107.stdout:7368: done (1 left) 2020-06-12T02:38:34.854 INFO:tasks.rados.rados.0.smithi107.stdout:7371: done (0 left) 2020-06-12T02:38:34.854 INFO:tasks.rados.rados.0.smithi107.stdout:7372: delete oid 2 current snap is 0 2020-06-12T02:38:34.855 INFO:tasks.rados.rados.0.smithi107.stdout:7372: done (0 left) 2020-06-12T02:38:34.855 INFO:tasks.rados.rados.0.smithi107.stdout:7373: delete oid 15 current snap is 0 2020-06-12T02:38:34.859 INFO:tasks.rados.rados.0.smithi107.stdout:7373: done (0 left) 2020-06-12T02:38:34.859 INFO:tasks.rados.rados.0.smithi107.stdout:7374: delete oid 17 current snap is 0 2020-06-12T02:38:34.867 INFO:tasks.rados.rados.0.smithi107.stdout:7374: done (0 left) 2020-06-12T02:38:34.868 INFO:tasks.rados.rados.0.smithi107.stdout:7375: delete oid 1 current snap is 0 2020-06-12T02:38:34.868 INFO:tasks.rados.rados.0.smithi107.stdout:7375: done (0 left) 2020-06-12T02:38:34.868 INFO:tasks.rados.rados.0.smithi107.stdout:7376: delete oid 1 current snap is 0 2020-06-12T02:38:34.868 INFO:tasks.rados.rados.0.smithi107.stdout:7376: done (0 left) 2020-06-12T02:38:34.869 INFO:tasks.rados.rados.0.smithi107.stdout:7377: delete oid 14 current snap is 0 2020-06-12T02:38:34.880 INFO:tasks.rados.rados.0.smithi107.stdout:7377: done (0 left) 2020-06-12T02:38:34.880 INFO:tasks.rados.rados.0.smithi107.stdout:7378: delete oid 13 current snap is 0 2020-06-12T02:38:34.881 INFO:tasks.rados.rados.0.smithi107.stdout:7378: done (0 left) 2020-06-12T02:38:34.881 INFO:tasks.rados.rados.0.smithi107.stdout:7379: delete oid 3 current snap is 0 2020-06-12T02:38:34.882 INFO:tasks.rados.rados.0.smithi107.stdout:7379: done (0 left) 2020-06-12T02:38:34.882 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:34.882 INFO:tasks.rados.rados.0.smithi107.stdout:7380: seq_num 7379 ranges {0=794624,794624=401408} 2020-06-12T02:38:34.882 INFO:tasks.rados.rados.0.smithi107.stdout:7380: writing smithi10729407-7 from 0 to 794624 tid 1 2020-06-12T02:38:34.883 INFO:tasks.rados.rados.0.smithi107.stdout:7380: writing smithi10729407-7 from 794624 to 1196032 tid 2 2020-06-12T02:38:34.883 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:34.883 INFO:tasks.rados.rados.0.smithi107.stdout:7381: seq_num 7380 ranges {0=409600,409600=565248,974848=237568} 2020-06-12T02:38:34.884 INFO:tasks.rados.rados.0.smithi107.stdout:7381: writing smithi10729407-9 from 0 to 409600 tid 1 2020-06-12T02:38:34.887 INFO:tasks.rados.rados.0.smithi107.stdout:7381: writing smithi10729407-9 from 409600 to 974848 tid 2 2020-06-12T02:38:34.889 INFO:tasks.rados.rados.0.smithi107.stdout:7381: writing smithi10729407-9 from 974848 to 1212416 tid 3 2020-06-12T02:38:34.889 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:34.889 INFO:tasks.rados.rados.0.smithi107.stdout:7382: seq_num 7381 ranges {0=425984,425984=450560,876544=352256} 2020-06-12T02:38:34.891 INFO:tasks.rados.rados.0.smithi107.stdout:7382: writing smithi10729407-4 from 0 to 425984 tid 1 2020-06-12T02:38:34.894 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:34.890+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:34.894 INFO:tasks.rados.rados.0.smithi107.stdout:7382: writing smithi10729407-4 from 425984 to 876544 tid 2 2020-06-12T02:38:34.897 INFO:tasks.rados.rados.0.smithi107.stdout:7382: writing smithi10729407-4 from 876544 to 1228800 tid 3 2020-06-12T02:38:34.897 INFO:tasks.rados.rados.0.smithi107.stdout:7383: delete oid 8 current snap is 0 2020-06-12T02:38:34.898 INFO:tasks.rados.rados.0.smithi107.stdout:7380: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:34.898 INFO:tasks.rados.rados.0.smithi107.stdout:7381: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:34.898 INFO:tasks.rados.rados.0.smithi107.stdout:7380: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:34.898 INFO:tasks.rados.rados.0.smithi107.stdout:7380: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:34.899 INFO:tasks.rados.rados.0.smithi107.stdout:7380: finishing write tid 5 to smithi10729407-7 2020-06-12T02:38:34.899 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1367 (ObjNum 7379 snap 0 seq_num 7379) dirty exists 2020-06-12T02:38:34.899 INFO:tasks.rados.rados.0.smithi107.stdout:7380: left oid 7 (ObjNum 7379 snap 0 seq_num 7379) 2020-06-12T02:38:34.899 INFO:tasks.rados.rados.0.smithi107.stdout:7382: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:34.900 INFO:tasks.rados.rados.0.smithi107.stdout:7381: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:34.900 INFO:tasks.rados.rados.0.smithi107.stdout:7381: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:34.902 INFO:tasks.rados.rados.0.smithi107.stdout:7380: done (3 left) 2020-06-12T02:38:34.902 INFO:tasks.rados.rados.0.smithi107.stdout:7383: done (2 left) 2020-06-12T02:38:34.902 INFO:tasks.rados.rados.0.smithi107.stdout:7384: delete oid 12 current snap is 0 2020-06-12T02:38:34.908 INFO:tasks.rados.rados.0.smithi107.stdout:7384: done (2 left) 2020-06-12T02:38:34.909 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:38:34.909 INFO:tasks.rados.rados.0.smithi107.stdout:7385: seq_num 7384 ranges {0=475136,475136=548864,1024000=253952} 2020-06-12T02:38:34.911 INFO:tasks.rados.rados.0.smithi107.stdout:7385: writing smithi10729407-5 from 0 to 475136 tid 1 2020-06-12T02:38:34.914 INFO:tasks.rados.rados.0.smithi107.stdout:7385: writing smithi10729407-5 from 475136 to 1024000 tid 2 2020-06-12T02:38:34.916 INFO:tasks.rados.rados.0.smithi107.stdout:7385: writing smithi10729407-5 from 1024000 to 1277952 tid 3 2020-06-12T02:38:34.917 INFO:tasks.rados.rados.0.smithi107.stdout:7386: delete oid 7 current snap is 0 2020-06-12T02:38:34.917 INFO:tasks.rados.rados.0.smithi107.stdout:7381: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:34.917 INFO:tasks.rados.rados.0.smithi107.stdout:7382: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:34.917 INFO:tasks.rados.rados.0.smithi107.stdout:7382: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:34.917 INFO:tasks.rados.rados.0.smithi107.stdout:7382: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:34.918 INFO:tasks.rados.rados.0.smithi107.stdout:7381: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:34.918 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1388 (ObjNum 7380 snap 0 seq_num 7380) dirty exists 2020-06-12T02:38:34.918 INFO:tasks.rados.rados.0.smithi107.stdout:7381: left oid 9 (ObjNum 7380 snap 0 seq_num 7380) 2020-06-12T02:38:34.918 INFO:tasks.rados.rados.0.smithi107.stdout:7382: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:34.918 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1402 (ObjNum 7381 snap 0 seq_num 7381) dirty exists 2020-06-12T02:38:34.919 INFO:tasks.rados.rados.0.smithi107.stdout:7382: left oid 4 (ObjNum 7381 snap 0 seq_num 7381) 2020-06-12T02:38:34.920 INFO:tasks.rados.rados.0.smithi107.stdout:7385: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:34.920 INFO:tasks.rados.rados.0.smithi107.stdout:7381: done (3 left) 2020-06-12T02:38:34.920 INFO:tasks.rados.rados.0.smithi107.stdout:7382: done (2 left) 2020-06-12T02:38:34.920 INFO:tasks.rados.rados.0.smithi107.stdout:7386: done (1 left) 2020-06-12T02:38:34.920 INFO:tasks.rados.rados.0.smithi107.stdout:7387: delete oid 3 current snap is 0 2020-06-12T02:38:34.925 INFO:tasks.rados.rados.0.smithi107.stdout:7387: done (1 left) 2020-06-12T02:38:34.926 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:34.926 INFO:tasks.rados.rados.0.smithi107.stdout:7388: seq_num 7387 ranges {0=524288,524288=606208,1130496=196608} 2020-06-12T02:38:34.929 INFO:tasks.rados.rados.0.smithi107.stdout:7388: writing smithi10729407-14 from 0 to 524288 tid 1 2020-06-12T02:38:34.933 INFO:tasks.rados.rados.0.smithi107.stdout:7388: writing smithi10729407-14 from 524288 to 1130496 tid 2 2020-06-12T02:38:34.934 INFO:tasks.rados.rados.0.smithi107.stdout:7388: writing smithi10729407-14 from 1130496 to 1327104 tid 3 2020-06-12T02:38:34.934 INFO:tasks.rados.rados.0.smithi107.stdout:7389: delete oid 9 current snap is 0 2020-06-12T02:38:34.935 INFO:tasks.rados.rados.0.smithi107.stdout:7385: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:34.935 INFO:tasks.rados.rados.0.smithi107.stdout:7385: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:34.935 INFO:tasks.rados.rados.0.smithi107.stdout:7385: finishing write tid 5 to smithi10729407-5 2020-06-12T02:38:34.935 INFO:tasks.rados.rados.0.smithi107.stdout:7385: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:34.935 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 746 (ObjNum 7384 snap 0 seq_num 7384) dirty exists 2020-06-12T02:38:34.936 INFO:tasks.rados.rados.0.smithi107.stdout:7385: left oid 5 (ObjNum 7384 snap 0 seq_num 7384) 2020-06-12T02:38:34.936 INFO:tasks.rados.rados.0.smithi107.stdout:7385: done (2 left) 2020-06-12T02:38:34.937 INFO:tasks.rados.rados.0.smithi107.stdout:7389: done (1 left) 2020-06-12T02:38:34.937 INFO:tasks.rados.rados.0.smithi107.stdout:7390: delete oid 1 current snap is 0 2020-06-12T02:38:34.937 INFO:tasks.rados.rados.0.smithi107.stdout:7388: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:34.944 INFO:tasks.rados.rados.0.smithi107.stdout:7390: done (1 left) 2020-06-12T02:38:34.944 INFO:tasks.rados.rados.0.smithi107.stdout:7391: delete oid 15 current snap is 0 2020-06-12T02:38:34.945 INFO:tasks.rados.rados.0.smithi107.stdout:7388: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:34.946 INFO:tasks.rados.rados.0.smithi107.stdout:7388: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:34.946 INFO:tasks.rados.rados.0.smithi107.stdout:7388: finishing write tid 5 to smithi10729407-14 2020-06-12T02:38:34.947 INFO:tasks.rados.rados.0.smithi107.stdout:7391: done (1 left) 2020-06-12T02:38:34.947 INFO:tasks.rados.rados.0.smithi107.stdout:7392: delete oid 13 current snap is 0 2020-06-12T02:38:34.948 INFO:tasks.rados.rados.0.smithi107.stdout:7388: finishing write tid 6 to smithi10729407-14 2020-06-12T02:38:34.949 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1305 (ObjNum 7387 snap 0 seq_num 7387) dirty exists 2020-06-12T02:38:34.949 INFO:tasks.rados.rados.0.smithi107.stdout:7388: left oid 14 (ObjNum 7387 snap 0 seq_num 7387) 2020-06-12T02:38:34.950 INFO:tasks.rados.rados.0.smithi107.stdout:7388: done (1 left) 2020-06-12T02:38:34.950 INFO:tasks.rados.rados.0.smithi107.stdout:7392: done (0 left) 2020-06-12T02:38:34.950 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:34.951 INFO:tasks.rados.rados.0.smithi107.stdout:7393: seq_num 7392 ranges {0=606208,606208=475136,1081344=327680} 2020-06-12T02:38:34.953 INFO:tasks.rados.rados.0.smithi107.stdout:7393: writing smithi10729407-6 from 0 to 606208 tid 1 2020-06-12T02:38:34.956 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:34.956 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:34.957 INFO:tasks.rados.rados.0.smithi107.stdout:7393: writing smithi10729407-6 from 606208 to 1081344 tid 2 2020-06-12T02:38:34.958 INFO:tasks.rados.rados.0.smithi107.stdout:7393: writing smithi10729407-6 from 1081344 to 1409024 tid 3 2020-06-12T02:38:34.958 INFO:tasks.rados.rados.0.smithi107.stdout:7394: delete oid 1 current snap is 0 2020-06-12T02:38:34.959 INFO:tasks.rados.rados.0.smithi107.stdout:7393: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:34.968 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-06-12T02:38:34.969 INFO:tasks.rados.rados.0.smithi107.stdout:7394: done (1 left) 2020-06-12T02:38:34.970 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:34.970 INFO:tasks.rados.rados.0.smithi107.stdout:7395: seq_num 7394 ranges {0=638976,638976=647168,1286144=155648} 2020-06-12T02:38:34.971 INFO:tasks.rados.rados.0.smithi107.stdout:7395: writing smithi10729407-11 from 0 to 638976 tid 1 2020-06-12T02:38:34.975 INFO:tasks.rados.rados.0.smithi107.stdout:7395: writing smithi10729407-11 from 638976 to 1286144 tid 2 2020-06-12T02:38:34.976 INFO:tasks.rados.rados.0.smithi107.stdout:7395: writing smithi10729407-11 from 1286144 to 1441792 tid 3 2020-06-12T02:38:34.976 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:34.976 INFO:tasks.rados.rados.0.smithi107.stdout:7396: seq_num 7395 ranges {0=655360,655360=532480,1187840=270336} 2020-06-12T02:38:34.981 INFO:tasks.rados.rados.0.smithi107.stdout:7396: writing smithi10729407-12 from 0 to 655360 tid 1 2020-06-12T02:38:34.984 INFO:tasks.rados.rados.0.smithi107.stdout:7396: writing smithi10729407-12 from 655360 to 1187840 tid 2 2020-06-12T02:38:34.986 INFO:tasks.rados.rados.0.smithi107.stdout:7396: writing smithi10729407-12 from 1187840 to 1458176 tid 3 2020-06-12T02:38:34.986 INFO:tasks.rados.rados.0.smithi107.stdout:7397: delete oid 19 current snap is 0 2020-06-12T02:38:34.986 INFO:tasks.rados.rados.0.smithi107.stdout:7393: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:34.987 INFO:tasks.rados.rados.0.smithi107.stdout:7393: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:34.987 INFO:tasks.rados.rados.0.smithi107.stdout:7393: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:34.987 INFO:tasks.rados.rados.0.smithi107.stdout:7393: finishing write tid 6 to smithi10729407-6 2020-06-12T02:38:34.987 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1340 (ObjNum 7392 snap 0 seq_num 7392) dirty exists 2020-06-12T02:38:34.987 INFO:tasks.rados.rados.0.smithi107.stdout:7393: left oid 6 (ObjNum 7392 snap 0 seq_num 7392) 2020-06-12T02:38:34.988 INFO:tasks.rados.rados.0.smithi107.stdout:7395: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:34.988 INFO:tasks.rados.rados.0.smithi107.stdout:7395: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:34.995 INFO:tasks.rados.rados.0.smithi107.stdout:7395: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:34.995 INFO:tasks.rados.rados.0.smithi107.stdout:7395: finishing write tid 5 to smithi10729407-11 2020-06-12T02:38:34.995 INFO:tasks.rados.rados.0.smithi107.stdout:7396: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:34.995 INFO:tasks.rados.rados.0.smithi107.stdout:7395: finishing write tid 6 to smithi10729407-11 2020-06-12T02:38:34.996 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1309 (ObjNum 7394 snap 0 seq_num 7394) dirty exists 2020-06-12T02:38:34.996 INFO:tasks.rados.rados.0.smithi107.stdout:7395: left oid 11 (ObjNum 7394 snap 0 seq_num 7394) 2020-06-12T02:38:34.997 INFO:tasks.rados.rados.0.smithi107.stdout:7393: done (3 left) 2020-06-12T02:38:34.997 INFO:tasks.rados.rados.0.smithi107.stdout:7395: done (2 left) 2020-06-12T02:38:34.997 INFO:tasks.rados.rados.0.smithi107.stdout:7397: done (1 left) 2020-06-12T02:38:34.997 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:38:34.999 INFO:tasks.rados.rados.0.smithi107.stdout:7398: seq_num 7397 ranges {0=688128,688128=696320,1384448=106496} 2020-06-12T02:38:34.999 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:34.994+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:35.000 INFO:tasks.rados.rados.0.smithi107.stdout:7398: writing smithi10729407-16 from 0 to 688128 tid 1 2020-06-12T02:38:35.004 INFO:tasks.rados.rados.0.smithi107.stdout:7398: writing smithi10729407-16 from 688128 to 1384448 tid 2 2020-06-12T02:38:35.004 INFO:tasks.rados.rados.0.smithi107.stdout:7398: writing smithi10729407-16 from 1384448 to 1490944 tid 3 2020-06-12T02:38:35.005 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:35.005 INFO:tasks.rados.rados.0.smithi107.stdout:7399: seq_num 7398 ranges {0=712704,712704=630784,1343488=172032} 2020-06-12T02:38:35.008 INFO:tasks.rados.rados.0.smithi107.stdout:7399: writing smithi10729407-9 from 0 to 712704 tid 1 2020-06-12T02:38:35.013 INFO:tasks.rados.rados.0.smithi107.stdout:7399: writing smithi10729407-9 from 712704 to 1343488 tid 2 2020-06-12T02:38:35.014 INFO:tasks.rados.rados.0.smithi107.stdout:7399: writing smithi10729407-9 from 1343488 to 1515520 tid 3 2020-06-12T02:38:35.014 INFO:tasks.rados.rados.0.smithi107.stdout:7400: delete oid 1 current snap is 0 2020-06-12T02:38:35.014 INFO:tasks.rados.rados.0.smithi107.stdout:7396: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:35.015 INFO:tasks.rados.rados.0.smithi107.stdout:7396: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:35.015 INFO:tasks.rados.rados.0.smithi107.stdout:7396: finishing write tid 5 to smithi10729407-12 2020-06-12T02:38:35.015 INFO:tasks.rados.rados.0.smithi107.stdout:7396: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:35.015 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1368 (ObjNum 7395 snap 0 seq_num 7395) dirty exists 2020-06-12T02:38:35.015 INFO:tasks.rados.rados.0.smithi107.stdout:7396: left oid 12 (ObjNum 7395 snap 0 seq_num 7395) 2020-06-12T02:38:35.015 INFO:tasks.rados.rados.0.smithi107.stdout:7398: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:35.016 INFO:tasks.rados.rados.0.smithi107.stdout:7398: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:35.017 INFO:tasks.rados.rados.0.smithi107.stdout:7398: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:35.017 INFO:tasks.rados.rados.0.smithi107.stdout:7398: finishing write tid 5 to smithi10729407-16 2020-06-12T02:38:35.017 INFO:tasks.rados.rados.0.smithi107.stdout:7399: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:35.017 INFO:tasks.rados.rados.0.smithi107.stdout:7396: done (3 left) 2020-06-12T02:38:35.017 INFO:tasks.rados.rados.0.smithi107.stdout:7400: done (2 left) 2020-06-12T02:38:35.017 INFO:tasks.rados.rados.0.smithi107.stdout:7401: delete oid 14 current snap is 0 2020-06-12T02:38:35.018 INFO:tasks.rados.rados.0.smithi107.stdout:7398: finishing write tid 6 to smithi10729407-16 2020-06-12T02:38:35.018 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1327 (ObjNum 7397 snap 0 seq_num 7397) dirty exists 2020-06-12T02:38:35.019 INFO:tasks.rados.rados.0.smithi107.stdout:7398: left oid 16 (ObjNum 7397 snap 0 seq_num 7397) 2020-06-12T02:38:35.019 INFO:tasks.rados.rados.0.smithi107.stdout:7398: done (2 left) 2020-06-12T02:38:35.019 INFO:tasks.rados.rados.0.smithi107.stdout:7401: done (1 left) 2020-06-12T02:38:35.020 INFO:tasks.rados.rados.0.smithi107.stdout:7402: delete oid 6 current snap is 0 2020-06-12T02:38:35.024 INFO:tasks.rados.rados.0.smithi107.stdout:7402: done (1 left) 2020-06-12T02:38:35.024 INFO:tasks.rados.rados.0.smithi107.stdout:7403: delete oid 7 current snap is 0 2020-06-12T02:38:35.024 INFO:tasks.rados.rados.0.smithi107.stdout:7399: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:35.025 INFO:tasks.rados.rados.0.smithi107.stdout:7399: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:35.026 INFO:tasks.rados.rados.0.smithi107.stdout:7399: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:35.026 INFO:tasks.rados.rados.0.smithi107.stdout:7403: done (1 left) 2020-06-12T02:38:35.026 INFO:tasks.rados.rados.0.smithi107.stdout:7404: delete oid 11 current snap is 0 2020-06-12T02:38:35.027 INFO:tasks.rados.rados.0.smithi107.stdout:7399: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:35.027 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1393 (ObjNum 7398 snap 0 seq_num 7398) dirty exists 2020-06-12T02:38:35.027 INFO:tasks.rados.rados.0.smithi107.stdout:7399: left oid 9 (ObjNum 7398 snap 0 seq_num 7398) 2020-06-12T02:38:35.029 INFO:tasks.rados.rados.0.smithi107.stdout:7399: done (1 left) 2020-06-12T02:38:35.029 INFO:tasks.rados.rados.0.smithi107.stdout:7404: done (0 left) 2020-06-12T02:38:35.029 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:35.029 INFO:tasks.rados.rados.0.smithi107.stdout:7405: seq_num 7404 ranges {0=409600,409600=737280,1146880=466944} 2020-06-12T02:38:35.031 INFO:tasks.rados.rados.0.smithi107.stdout:7405: writing smithi10729407-14 from 0 to 409600 tid 1 2020-06-12T02:38:35.036 INFO:tasks.rados.rados.0.smithi107.stdout:7405: writing smithi10729407-14 from 409600 to 1146880 tid 2 2020-06-12T02:38:35.039 INFO:tasks.rados.rados.0.smithi107.stdout:7405: writing smithi10729407-14 from 1146880 to 1613824 tid 3 2020-06-12T02:38:35.040 INFO:tasks.rados.rados.0.smithi107.stdout:7406: delete oid 20 current snap is 0 2020-06-12T02:38:35.040 INFO:tasks.rados.rados.0.smithi107.stdout:7405: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:35.042 INFO:tasks.rados.rados.0.smithi107.stdout:7406: done (1 left) 2020-06-12T02:38:35.043 INFO:tasks.rados.rados.0.smithi107.stdout:7407: delete oid 1 current snap is 0 2020-06-12T02:38:35.045 INFO:tasks.rados.rados.0.smithi107.stdout:7405: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:35.046 INFO:tasks.rados.rados.0.smithi107.stdout:7405: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:35.047 INFO:tasks.rados.rados.0.smithi107.stdout:7405: finishing write tid 5 to smithi10729407-14 2020-06-12T02:38:35.048 INFO:tasks.rados.rados.0.smithi107.stdout:7407: done (1 left) 2020-06-12T02:38:35.049 INFO:tasks.rados.rados.0.smithi107.stdout:7408: delete oid 12 current snap is 0 2020-06-12T02:38:35.052 INFO:tasks.rados.rados.0.smithi107.stdout:7405: finishing write tid 6 to smithi10729407-14 2020-06-12T02:38:35.053 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1315 (ObjNum 7404 snap 0 seq_num 7404) dirty exists 2020-06-12T02:38:35.053 INFO:tasks.rados.rados.0.smithi107.stdout:7405: left oid 14 (ObjNum 7404 snap 0 seq_num 7404) 2020-06-12T02:38:35.054 INFO:tasks.rados.rados.0.smithi107.stdout:7405: done (1 left) 2020-06-12T02:38:35.054 INFO:tasks.rados.rados.0.smithi107.stdout:7408: done (0 left) 2020-06-12T02:38:35.054 INFO:tasks.rados.rados.0.smithi107.stdout:7409: delete oid 8 current snap is 0 2020-06-12T02:38:35.057 INFO:tasks.rados.rados.0.smithi107.stdout:7409: done (0 left) 2020-06-12T02:38:35.057 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:35.057 INFO:tasks.rados.rados.0.smithi107.stdout:7410: seq_num 7409 ranges {1613824=491520,2105344=606208,2711552=434176,3145728=163840} 2020-06-12T02:38:35.071 INFO:tasks.rados.rados.0.smithi107.stdout:7410: writing smithi10729407-14 from 1613824 to 2105344 tid 1 2020-06-12T02:38:35.072 INFO:tasks.rados.rados.0.smithi107.stdout:7410: writing smithi10729407-14 from 2105344 to 2711552 tid 2 2020-06-12T02:38:35.074 INFO:tasks.rados.rados.0.smithi107.stdout:7410: writing smithi10729407-14 from 2711552 to 3145728 tid 3 2020-06-12T02:38:35.076 INFO:tasks.rados.rados.0.smithi107.stdout:7410: writing smithi10729407-14 from 3145728 to 3309568 tid 4 2020-06-12T02:38:35.076 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:35.076 INFO:tasks.rados.rados.0.smithi107.stdout:7411: seq_num 7410 ranges {0=507904,507904=491520,999424=638976,1638400=73728} 2020-06-12T02:38:35.079 INFO:tasks.rados.rados.0.smithi107.stdout:7411: writing smithi10729407-3 from 0 to 507904 tid 1 2020-06-12T02:38:35.083 INFO:tasks.rados.rados.0.smithi107.stdout:7411: writing smithi10729407-3 from 507904 to 999424 tid 2 2020-06-12T02:38:35.087 INFO:tasks.rados.rados.0.smithi107.stdout:7411: writing smithi10729407-3 from 999424 to 1638400 tid 3 2020-06-12T02:38:35.088 INFO:tasks.rados.rados.0.smithi107.stdout:7411: writing smithi10729407-3 from 1638400 to 1712128 tid 4 2020-06-12T02:38:35.088 INFO:tasks.rados.rados.0.smithi107.stdout:7412: delete oid 4 current snap is 0 2020-06-12T02:38:35.089 INFO:tasks.rados.rados.0.smithi107.stdout:7410: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:35.089 INFO:tasks.rados.rados.0.smithi107.stdout:7410: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:35.089 INFO:tasks.rados.rados.0.smithi107.stdout:7410: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:35.089 INFO:tasks.rados.rados.0.smithi107.stdout:7410: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:35.089 INFO:tasks.rados.rados.0.smithi107.stdout:7411: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:35.090 INFO:tasks.rados.rados.0.smithi107.stdout:7410: finishing write tid 6 to smithi10729407-14 2020-06-12T02:38:35.090 INFO:tasks.rados.rados.0.smithi107.stdout:7410: finishing write tid 7 to smithi10729407-14 2020-06-12T02:38:35.090 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1320 (ObjNum 7409 snap 0 seq_num 7409) dirty exists 2020-06-12T02:38:35.090 INFO:tasks.rados.rados.0.smithi107.stdout:7410: left oid 14 (ObjNum 7409 snap 0 seq_num 7409) 2020-06-12T02:38:35.091 INFO:tasks.rados.rados.0.smithi107.stdout:7410: done (2 left) 2020-06-12T02:38:35.091 INFO:tasks.rados.rados.0.smithi107.stdout:7412: done (1 left) 2020-06-12T02:38:35.091 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:35.092 INFO:tasks.rados.rados.0.smithi107.stdout:7413: seq_num 7412 ranges {0=540672,540672=663552,1204224=540672} 2020-06-12T02:38:35.096 INFO:tasks.rados.rados.0.smithi107.stdout:7413: writing smithi10729407-4 from 0 to 540672 tid 1 2020-06-12T02:38:35.098 INFO:tasks.rados.rados.0.smithi107.stdout:7413: writing smithi10729407-4 from 540672 to 1204224 tid 2 2020-06-12T02:38:35.100 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:35.095+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:35.102 INFO:tasks.rados.rados.0.smithi107.stdout:7413: writing smithi10729407-4 from 1204224 to 1744896 tid 3 2020-06-12T02:38:35.102 INFO:tasks.rados.rados.0.smithi107.stdout:7414: delete oid 14 current snap is 0 2020-06-12T02:38:35.103 INFO:tasks.rados.rados.0.smithi107.stdout:7411: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:35.103 INFO:tasks.rados.rados.0.smithi107.stdout:7411: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:35.103 INFO:tasks.rados.rados.0.smithi107.stdout:7411: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:35.103 INFO:tasks.rados.rados.0.smithi107.stdout:7411: finishing write tid 6 to smithi10729407-3 2020-06-12T02:38:35.103 INFO:tasks.rados.rados.0.smithi107.stdout:7411: finishing write tid 7 to smithi10729407-3 2020-06-12T02:38:35.103 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1332 (ObjNum 7410 snap 0 seq_num 7410) dirty exists 2020-06-12T02:38:35.104 INFO:tasks.rados.rados.0.smithi107.stdout:7411: left oid 3 (ObjNum 7410 snap 0 seq_num 7410) 2020-06-12T02:38:35.104 INFO:tasks.rados.rados.0.smithi107.stdout:7411: done (2 left) 2020-06-12T02:38:35.104 INFO:tasks.rados.rados.0.smithi107.stdout:7414: done (1 left) 2020-06-12T02:38:35.105 INFO:tasks.rados.rados.0.smithi107.stdout:7415: delete oid 7 current snap is 0 2020-06-12T02:38:35.105 INFO:tasks.rados.rados.0.smithi107.stdout:7413: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:35.109 INFO:tasks.rados.rados.0.smithi107.stdout:7415: done (1 left) 2020-06-12T02:38:35.110 INFO:tasks.rados.rados.0.smithi107.stdout:7416: delete oid 16 current snap is 0 2020-06-12T02:38:35.111 INFO:tasks.rados.rados.0.smithi107.stdout:7413: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:35.111 INFO:tasks.rados.rados.0.smithi107.stdout:7413: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:35.121 INFO:tasks.rados.rados.0.smithi107.stdout:7416: done (1 left) 2020-06-12T02:38:35.121 INFO:tasks.rados.rados.0.smithi107.stdout:7417: delete oid 16 current snap is 0 2020-06-12T02:38:35.121 INFO:tasks.rados.rados.0.smithi107.stdout:7413: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:35.122 INFO:tasks.rados.rados.0.smithi107.stdout:7413: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:35.122 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1407 (ObjNum 7412 snap 0 seq_num 7412) dirty exists 2020-06-12T02:38:35.122 INFO:tasks.rados.rados.0.smithi107.stdout:7413: left oid 4 (ObjNum 7412 snap 0 seq_num 7412) 2020-06-12T02:38:35.122 INFO:tasks.rados.rados.0.smithi107.stdout:7413: done (1 left) 2020-06-12T02:38:35.122 INFO:tasks.rados.rados.0.smithi107.stdout:7417: done (0 left) 2020-06-12T02:38:35.123 INFO:tasks.rados.rados.0.smithi107.stdout:7418: delete oid 2 current snap is 0 2020-06-12T02:38:35.123 INFO:tasks.rados.rados.0.smithi107.stdout:7418: done (0 left) 2020-06-12T02:38:35.123 INFO:tasks.rados.rados.0.smithi107.stdout:7419: delete oid 1 current snap is 0 2020-06-12T02:38:35.123 INFO:tasks.rados.rados.0.smithi107.stdout:7419: done (0 left) 2020-06-12T02:38:35.123 INFO:tasks.rados.rados.0.smithi107.stdout:7420: delete oid 14 current snap is 0 2020-06-12T02:38:35.124 INFO:tasks.rados.rados.0.smithi107.stdout:7420: done (0 left) 2020-06-12T02:38:35.124 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:35.124 INFO:tasks.rados.rados.0.smithi107.stdout:7421: seq_num 7420 ranges {1515520=679936,2195456=638976,2834432=565248} 2020-06-12T02:38:35.135 INFO:tasks.rados.rados.0.smithi107.stdout:7421: writing smithi10729407-9 from 1515520 to 2195456 tid 1 2020-06-12T02:38:35.139 INFO:tasks.rados.rados.0.smithi107.stdout:7421: writing smithi10729407-9 from 2195456 to 2834432 tid 2 2020-06-12T02:38:35.143 INFO:tasks.rados.rados.0.smithi107.stdout:7421: writing smithi10729407-9 from 2834432 to 3399680 tid 3 2020-06-12T02:38:35.144 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:35.144 INFO:tasks.rados.rados.0.smithi107.stdout:7422: seq_num 7421 ranges {1277952=696320,1974272=516096,2490368=688128} 2020-06-12T02:38:35.144 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:35.145 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:35.154 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-06-12T02:38:35.155 INFO:tasks.rados.rados.0.smithi107.stdout:7422: writing smithi10729407-5 from 1277952 to 1974272 tid 1 2020-06-12T02:38:35.158 INFO:tasks.rados.rados.0.smithi107.stdout:7422: writing smithi10729407-5 from 1974272 to 2490368 tid 2 2020-06-12T02:38:35.162 INFO:tasks.rados.rados.0.smithi107.stdout:7422: writing smithi10729407-5 from 2490368 to 3178496 tid 3 2020-06-12T02:38:35.162 INFO:tasks.rados.rados.0.smithi107.stdout:7423: delete oid 12 current snap is 0 2020-06-12T02:38:35.163 INFO:tasks.rados.rados.0.smithi107.stdout:7421: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:35.163 INFO:tasks.rados.rados.0.smithi107.stdout:7421: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:35.163 INFO:tasks.rados.rados.0.smithi107.stdout:7421: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:35.163 INFO:tasks.rados.rados.0.smithi107.stdout:7421: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:35.163 INFO:tasks.rados.rados.0.smithi107.stdout:7421: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:35.164 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1397 (ObjNum 7420 snap 0 seq_num 7420) dirty exists 2020-06-12T02:38:35.164 INFO:tasks.rados.rados.0.smithi107.stdout:7421: left oid 9 (ObjNum 7420 snap 0 seq_num 7420) 2020-06-12T02:38:35.166 INFO:tasks.rados.rados.0.smithi107.stdout:7421: done (2 left) 2020-06-12T02:38:35.167 INFO:tasks.rados.rados.0.smithi107.stdout:7423: done (1 left) 2020-06-12T02:38:35.167 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:38:35.167 INFO:tasks.rados.rados.0.smithi107.stdout:7424: seq_num 7423 ranges {1712128=729088,2441216=688128,3129344=516096} 2020-06-12T02:38:35.180 INFO:tasks.rados.rados.0.smithi107.stdout:7424: writing smithi10729407-3 from 1712128 to 2441216 tid 1 2020-06-12T02:38:35.185 INFO:tasks.rados.rados.0.smithi107.stdout:7424: writing smithi10729407-3 from 2441216 to 3129344 tid 2 2020-06-12T02:38:35.189 INFO:tasks.rados.rados.0.smithi107.stdout:7424: writing smithi10729407-3 from 3129344 to 3645440 tid 3 2020-06-12T02:38:35.190 INFO:tasks.rados.rados.0.smithi107.stdout:7425: delete oid 7 current snap is 0 2020-06-12T02:38:35.190 INFO:tasks.rados.rados.0.smithi107.stdout:7422: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:35.190 INFO:tasks.rados.rados.0.smithi107.stdout:7422: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:35.190 INFO:tasks.rados.rados.0.smithi107.stdout:7422: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:35.191 INFO:tasks.rados.rados.0.smithi107.stdout:7422: finishing write tid 5 to smithi10729407-5 2020-06-12T02:38:35.191 INFO:tasks.rados.rados.0.smithi107.stdout:7422: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:35.191 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 750 (ObjNum 7421 snap 0 seq_num 7421) dirty exists 2020-06-12T02:38:35.191 INFO:tasks.rados.rados.0.smithi107.stdout:7422: left oid 5 (ObjNum 7421 snap 0 seq_num 7421) 2020-06-12T02:38:35.193 INFO:tasks.rados.rados.0.smithi107.stdout:7424: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:35.194 INFO:tasks.rados.rados.0.smithi107.stdout:7422: done (2 left) 2020-06-12T02:38:35.194 INFO:tasks.rados.rados.0.smithi107.stdout:7425: done (1 left) 2020-06-12T02:38:35.194 INFO:tasks.rados.rados.0.smithi107.stdout:7426: delete oid 19 current snap is 0 2020-06-12T02:38:35.194 INFO:tasks.rados.rados.0.smithi107.stdout:7424: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:35.195 INFO:tasks.rados.rados.0.smithi107.stdout:7424: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:35.195 INFO:tasks.rados.rados.0.smithi107.stdout:7424: finishing write tid 5 to smithi10729407-3 2020-06-12T02:38:35.196 INFO:tasks.rados.rados.0.smithi107.stdout:7426: done (1 left) 2020-06-12T02:38:35.197 INFO:tasks.rados.rados.0.smithi107.stdout:7427: delete oid 16 current snap is 0 2020-06-12T02:38:35.197 INFO:tasks.rados.rados.0.smithi107.stdout:7424: finishing write tid 6 to smithi10729407-3 2020-06-12T02:38:35.198 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1338 (ObjNum 7423 snap 0 seq_num 7423) dirty exists 2020-06-12T02:38:35.198 INFO:tasks.rados.rados.0.smithi107.stdout:7424: left oid 3 (ObjNum 7423 snap 0 seq_num 7423) 2020-06-12T02:38:35.203 INFO:tasks.rados.rados.0.smithi107.stdout:7424: done (1 left) 2020-06-12T02:38:35.203 INFO:tasks.rados.rados.0.smithi107.stdout:7427: done (0 left) 2020-06-12T02:38:35.203 INFO:tasks.rados.rados.0.smithi107.stdout:7428: delete oid 3 current snap is 0 2020-06-12T02:38:35.204 INFO:tasks.rados.rados.0.smithi107.stdout:7428: done (0 left) 2020-06-12T02:38:35.204 INFO:tasks.rados.rados.0.smithi107.stdout:7429: delete oid 10 current snap is 0 2020-06-12T02:38:35.204 INFO:tasks.rados.rados.0.smithi107.stdout:7429: done (0 left) 2020-06-12T02:38:35.204 INFO:tasks.rados.rados.0.smithi107.stdout:7430: delete oid 15 current snap is 0 2020-06-12T02:38:35.205 INFO:tasks.rados.rados.0.smithi107.stdout:7430: done (0 left) 2020-06-12T02:38:35.205 INFO:tasks.rados.rados.0.smithi107.stdout:7431: delete oid 17 current snap is 0 2020-06-12T02:38:35.207 INFO:tasks.rados.rados.0.smithi107.stdout:7431: done (0 left) 2020-06-12T02:38:35.207 INFO:tasks.rados.rados.0.smithi107.stdout:7432: delete oid 8 current snap is 0 2020-06-12T02:38:35.209 INFO:tasks.rados.rados.0.smithi107.stdout:7432: done (0 left) 2020-06-12T02:38:35.210 INFO:tasks.rados.rados.0.smithi107.stdout:7433: delete oid 6 current snap is 0 2020-06-12T02:38:35.212 INFO:tasks.rados.rados.0.smithi107.stdout:7433: done (0 left) 2020-06-12T02:38:35.212 INFO:tasks.rados.rados.0.smithi107.stdout:7434: delete oid 20 current snap is 0 2020-06-12T02:38:35.214 INFO:tasks.rados.rados.0.smithi107.stdout:7434: done (0 left) 2020-06-12T02:38:35.214 INFO:tasks.rados.rados.0.smithi107.stdout:7435: delete oid 6 current snap is 0 2020-06-12T02:38:35.216 INFO:tasks.rados.rados.0.smithi107.stdout:7435: done (0 left) 2020-06-12T02:38:35.216 INFO:tasks.rados.rados.0.smithi107.stdout:7436: delete oid 1 current snap is 0 2020-06-12T02:38:35.218 INFO:tasks.rados.rados.0.smithi107.stdout:7436: done (0 left) 2020-06-12T02:38:35.219 INFO:tasks.rados.rados.0.smithi107.stdout:7437: delete oid 3 current snap is 0 2020-06-12T02:38:35.220 INFO:tasks.rados.rados.0.smithi107.stdout:7437: done (0 left) 2020-06-12T02:38:35.220 INFO:tasks.rados.rados.0.smithi107.stdout:7438: delete oid 16 current snap is 0 2020-06-12T02:38:35.222 INFO:tasks.rados.rados.0.smithi107.stdout:7438: done (0 left) 2020-06-12T02:38:35.222 INFO:tasks.rados.rados.0.smithi107.stdout:7439: delete oid 4 current snap is 0 2020-06-12T02:38:35.224 INFO:tasks.rados.rados.0.smithi107.stdout:7439: done (0 left) 2020-06-12T02:38:35.225 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:35.225 INFO:tasks.rados.rados.0.smithi107.stdout:7440: seq_num 7439 ranges {0=598016,598016=540672,1138688=663552,1802240=401408} 2020-06-12T02:38:35.228 INFO:tasks.rados.rados.0.smithi107.stdout:7440: writing smithi10729407-1 from 0 to 598016 tid 1 2020-06-12T02:38:35.231 INFO:tasks.rados.rados.0.smithi107.stdout:7440: writing smithi10729407-1 from 598016 to 1138688 tid 2 2020-06-12T02:38:35.235 INFO:tasks.rados.rados.0.smithi107.stdout:7440: writing smithi10729407-1 from 1138688 to 1802240 tid 3 2020-06-12T02:38:35.237 INFO:tasks.rados.rados.0.smithi107.stdout:7440: writing smithi10729407-1 from 1802240 to 2203648 tid 4 2020-06-12T02:38:35.237 INFO:tasks.rados.rados.0.smithi107.stdout:7441: delete oid 8 current snap is 0 2020-06-12T02:38:35.238 INFO:tasks.rados.rados.0.smithi107.stdout:7440: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:35.239 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:38:35.235+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:38:35.242 INFO:tasks.rados.rados.0.smithi107.stdout:7441: done (1 left) 2020-06-12T02:38:35.242 INFO:tasks.rados.rados.0.smithi107.stdout:7442: delete oid 8 current snap is 0 2020-06-12T02:38:35.243 INFO:tasks.rados.rados.0.smithi107.stdout:7440: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:35.245 INFO:tasks.rados.rados.0.smithi107.stdout:7442: done (1 left) 2020-06-12T02:38:35.246 INFO:tasks.rados.rados.0.smithi107.stdout:7443: delete oid 14 current snap is 0 2020-06-12T02:38:35.247 INFO:tasks.rados.rados.0.smithi107.stdout:7440: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:35.248 INFO:tasks.rados.rados.0.smithi107.stdout:7440: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:35.249 INFO:tasks.rados.rados.0.smithi107.stdout:7443: done (1 left) 2020-06-12T02:38:35.250 INFO:tasks.rados.rados.0.smithi107.stdout:7444: delete oid 12 current snap is 0 2020-06-12T02:38:35.250 INFO:tasks.rados.rados.0.smithi107.stdout:7440: finishing write tid 6 to smithi10729407-1 2020-06-12T02:38:35.252 INFO:tasks.rados.rados.0.smithi107.stdout:7440: finishing write tid 7 to smithi10729407-1 2020-06-12T02:38:35.253 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1352 (ObjNum 7439 snap 0 seq_num 7439) dirty exists 2020-06-12T02:38:35.253 INFO:tasks.rados.rados.0.smithi107.stdout:7440: left oid 1 (ObjNum 7439 snap 0 seq_num 7439) 2020-06-12T02:38:35.253 INFO:tasks.rados.rados.0.smithi107.stdout:7440: done (1 left) 2020-06-12T02:38:35.253 INFO:tasks.rados.rados.0.smithi107.stdout:7444: done (0 left) 2020-06-12T02:38:35.253 INFO:tasks.rados.rados.0.smithi107.stdout:7445: delete oid 6 current snap is 0 2020-06-12T02:38:35.254 INFO:tasks.rados.rados.0.smithi107.stdout:7445: done (0 left) 2020-06-12T02:38:35.254 INFO:tasks.rados.rados.0.smithi107.stdout:7446: delete oid 10 current snap is 0 2020-06-12T02:38:35.256 INFO:tasks.rados.rados.0.smithi107.stdout:7446: done (0 left) 2020-06-12T02:38:35.256 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:35.256 INFO:tasks.rados.rados.0.smithi107.stdout:7447: seq_num 7446 ranges {0=712704,712704=581632,1294336=466944,1761280=557056} 2020-06-12T02:38:35.260 INFO:tasks.rados.rados.0.smithi107.stdout:7447: writing smithi10729407-3 from 0 to 712704 tid 1 2020-06-12T02:38:35.263 INFO:tasks.rados.rados.0.smithi107.stdout:7447: writing smithi10729407-3 from 712704 to 1294336 tid 2 2020-06-12T02:38:35.267 INFO:tasks.rados.rados.0.smithi107.stdout:7447: writing smithi10729407-3 from 1294336 to 1761280 tid 3 2020-06-12T02:38:35.270 INFO:tasks.rados.rados.0.smithi107.stdout:7447: writing smithi10729407-3 from 1761280 to 2318336 tid 4 2020-06-12T02:38:35.270 INFO:tasks.rados.rados.0.smithi107.stdout:7448: delete oid 8 current snap is 0 2020-06-12T02:38:35.270 INFO:tasks.rados.rados.0.smithi107.stdout:7447: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:35.273 INFO:tasks.rados.rados.0.smithi107.stdout:7448: done (1 left) 2020-06-12T02:38:35.274 INFO:tasks.rados.rados.0.smithi107.stdout:7449: delete oid 4 current snap is 0 2020-06-12T02:38:35.276 INFO:tasks.rados.rados.0.smithi107.stdout:7449: done (1 left) 2020-06-12T02:38:35.276 INFO:tasks.rados.rados.0.smithi107.stdout:7450: delete oid 4 current snap is 0 2020-06-12T02:38:35.278 INFO:tasks.rados.rados.0.smithi107.stdout:7447: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:35.278 INFO:tasks.rados.rados.0.smithi107.stdout:7447: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:35.279 INFO:tasks.rados.rados.0.smithi107.stdout:7450: done (1 left) 2020-06-12T02:38:35.280 INFO:tasks.rados.rados.0.smithi107.stdout:7451: delete oid 8 current snap is 0 2020-06-12T02:38:35.280 INFO:tasks.rados.rados.0.smithi107.stdout:7447: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:35.280 INFO:tasks.rados.rados.0.smithi107.stdout:7447: finishing write tid 6 to smithi10729407-3 2020-06-12T02:38:35.282 INFO:tasks.rados.rados.0.smithi107.stdout:7447: finishing write tid 7 to smithi10729407-3 2020-06-12T02:38:35.282 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1347 (ObjNum 7446 snap 0 seq_num 7446) dirty exists 2020-06-12T02:38:35.282 INFO:tasks.rados.rados.0.smithi107.stdout:7447: left oid 3 (ObjNum 7446 snap 0 seq_num 7446) 2020-06-12T02:38:35.282 INFO:tasks.rados.rados.0.smithi107.stdout:7447: done (1 left) 2020-06-12T02:38:35.283 INFO:tasks.rados.rados.0.smithi107.stdout:7451: done (0 left) 2020-06-12T02:38:35.283 INFO:tasks.rados.rados.0.smithi107.stdout:7452: delete oid 14 current snap is 0 2020-06-12T02:38:35.285 INFO:tasks.rados.rados.0.smithi107.stdout:7452: done (0 left) 2020-06-12T02:38:35.285 INFO:tasks.rados.rados.0.smithi107.stdout:7453: delete oid 15 current snap is 0 2020-06-12T02:38:35.288 INFO:tasks.rados.rados.0.smithi107.stdout:7453: done (0 left) 2020-06-12T02:38:35.288 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:35.288 INFO:tasks.rados.rados.0.smithi107.stdout:7454: seq_num 7453 ranges {0=24576} 2020-06-12T02:38:35.289 INFO:tasks.rados.rados.0.smithi107.stdout:7454: writing smithi10729407-15 from 0 to 24576 tid 1 2020-06-12T02:38:35.289 INFO:tasks.rados.rados.0.smithi107.stdout:7455: delete oid 4 current snap is 0 2020-06-12T02:38:35.291 INFO:tasks.rados.rados.0.smithi107.stdout:7454: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:35.293 INFO:tasks.rados.rados.0.smithi107.stdout:7455: done (1 left) 2020-06-12T02:38:35.293 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:35.294 INFO:tasks.rados.rados.0.smithi107.stdout:7456: seq_num 7455 ranges {0=57344} 2020-06-12T02:38:35.294 INFO:tasks.rados.rados.0.smithi107.stdout:7456: writing smithi10729407-2 from 0 to 57344 tid 1 2020-06-12T02:38:35.294 INFO:tasks.rados.rados.0.smithi107.stdout:7457: delete oid 10 current snap is 0 2020-06-12T02:38:35.296 INFO:tasks.rados.rados.0.smithi107.stdout:7457: done (2 left) 2020-06-12T02:38:35.296 INFO:tasks.rados.rados.0.smithi107.stdout:7458: delete oid 9 current snap is 0 2020-06-12T02:38:35.297 INFO:tasks.rados.rados.0.smithi107.stdout:7454: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:35.297 INFO:tasks.rados.rados.0.smithi107.stdout:7456: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:35.298 INFO:tasks.rados.rados.0.smithi107.stdout:7456: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:35.299 INFO:tasks.rados.rados.0.smithi107.stdout:7454: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:35.299 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1392 (ObjNum 7453 snap 0 seq_num 7453) dirty exists 2020-06-12T02:38:35.300 INFO:tasks.rados.rados.0.smithi107.stdout:7454: left oid 15 (ObjNum 7453 snap 0 seq_num 7453) 2020-06-12T02:38:35.300 INFO:tasks.rados.rados.0.smithi107.stdout:7454: done (2 left) 2020-06-12T02:38:35.300 INFO:tasks.rados.rados.0.smithi107.stdout:7458: done (1 left) 2020-06-12T02:38:35.300 INFO:tasks.rados.rados.0.smithi107.stdout:7459: delete oid 17 current snap is 0 2020-06-12T02:38:35.301 INFO:tasks.rados.rados.0.smithi107.stdout:7456: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:35.301 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 705 (ObjNum 7455 snap 0 seq_num 7455) dirty exists 2020-06-12T02:38:35.301 INFO:tasks.rados.rados.0.smithi107.stdout:7456: left oid 2 (ObjNum 7455 snap 0 seq_num 7455) 2020-06-12T02:38:35.302 INFO:tasks.rados.rados.0.smithi107.stdout:7456: done (1 left) 2020-06-12T02:38:35.302 INFO:tasks.rados.rados.0.smithi107.stdout:7459: done (0 left) 2020-06-12T02:38:35.302 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:35.303 INFO:tasks.rados.rados.0.smithi107.stdout:7460: seq_num 7459 ranges {0=131072} 2020-06-12T02:38:35.304 INFO:tasks.rados.rados.0.smithi107.stdout:7460: writing smithi10729407-16 from 0 to 131072 tid 1 2020-06-12T02:38:35.304 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:35.304 INFO:tasks.rados.rados.0.smithi107.stdout:7461: seq_num 7460 ranges {0=147456} 2020-06-12T02:38:35.304 INFO:tasks.rados.rados.0.smithi107.stdout:7461: writing smithi10729407-9 from 0 to 147456 tid 1 2020-06-12T02:38:35.304 INFO:tasks.rados.rados.0.smithi107.stdout:7462: delete oid 14 current snap is 0 2020-06-12T02:38:35.305 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:35.300+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:35.307 INFO:tasks.rados.rados.0.smithi107.stdout:7460: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:35.307 INFO:tasks.rados.rados.0.smithi107.stdout:7462: done (2 left) 2020-06-12T02:38:35.308 INFO:tasks.rados.rados.0.smithi107.stdout:7463: delete oid 2 current snap is 0 2020-06-12T02:38:35.309 INFO:tasks.rados.rados.0.smithi107.stdout:7461: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:35.310 INFO:tasks.rados.rados.0.smithi107.stdout:7463: done (2 left) 2020-06-12T02:38:35.310 INFO:tasks.rados.rados.0.smithi107.stdout:7464: delete oid 19 current snap is 0 2020-06-12T02:38:35.312 INFO:tasks.rados.rados.0.smithi107.stdout:7464: done (2 left) 2020-06-12T02:38:35.312 INFO:tasks.rados.rados.0.smithi107.stdout:7465: delete oid 10 current snap is 0 2020-06-12T02:38:35.314 INFO:tasks.rados.rados.0.smithi107.stdout:7460: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:35.314 INFO:tasks.rados.rados.0.smithi107.stdout:7465: done (2 left) 2020-06-12T02:38:35.314 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:38:35.315 INFO:tasks.rados.rados.0.smithi107.stdout:7466: seq_num 7465 ranges {2318336=229376} 2020-06-12T02:38:35.328 INFO:tasks.rados.rados.0.smithi107.stdout:7466: writing smithi10729407-3 from 2318336 to 2547712 tid 1 2020-06-12T02:38:35.329 INFO:tasks.rados.rados.0.smithi107.stdout:7467: delete oid 19 current snap is 0 2020-06-12T02:38:35.329 INFO:tasks.rados.rados.0.smithi107.stdout:7461: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:35.329 INFO:tasks.rados.rados.0.smithi107.stdout:7460: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:35.329 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1349 (ObjNum 7459 snap 0 seq_num 7459) dirty exists 2020-06-12T02:38:35.329 INFO:tasks.rados.rados.0.smithi107.stdout:7460: left oid 16 (ObjNum 7459 snap 0 seq_num 7459) 2020-06-12T02:38:35.330 INFO:tasks.rados.rados.0.smithi107.stdout:7461: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:35.330 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1400 (ObjNum 7460 snap 0 seq_num 7460) dirty exists 2020-06-12T02:38:35.330 INFO:tasks.rados.rados.0.smithi107.stdout:7461: left oid 9 (ObjNum 7460 snap 0 seq_num 7460) 2020-06-12T02:38:35.331 INFO:tasks.rados.rados.0.smithi107.stdout:7460: done (3 left) 2020-06-12T02:38:35.331 INFO:tasks.rados.rados.0.smithi107.stdout:7461: done (2 left) 2020-06-12T02:38:35.331 INFO:tasks.rados.rados.0.smithi107.stdout:7467: done (1 left) 2020-06-12T02:38:35.331 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:35.332 INFO:tasks.rados.rados.0.smithi107.stdout:7468: seq_num 7467 ranges {3178496=262144} 2020-06-12T02:38:35.334 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:35.334 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:35.343 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-06-12T02:38:35.349 INFO:tasks.rados.rados.0.smithi107.stdout:7468: writing smithi10729407-5 from 3178496 to 3440640 tid 1 2020-06-12T02:38:35.349 INFO:tasks.rados.rados.0.smithi107.stdout:7469: delete oid 7 current snap is 0 2020-06-12T02:38:35.349 INFO:tasks.rados.rados.0.smithi107.stdout:7466: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:35.350 INFO:tasks.rados.rados.0.smithi107.stdout:7466: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:35.350 INFO:tasks.rados.rados.0.smithi107.stdout:7466: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:35.350 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1351 (ObjNum 7465 snap 0 seq_num 7465) dirty exists 2020-06-12T02:38:35.350 INFO:tasks.rados.rados.0.smithi107.stdout:7466: left oid 3 (ObjNum 7465 snap 0 seq_num 7465) 2020-06-12T02:38:35.352 INFO:tasks.rados.rados.0.smithi107.stdout:7466: done (2 left) 2020-06-12T02:38:35.352 INFO:tasks.rados.rados.0.smithi107.stdout:7469: done (1 left) 2020-06-12T02:38:35.352 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:35.352 INFO:tasks.rados.rados.0.smithi107.stdout:7470: seq_num 7469 ranges {0=294912} 2020-06-12T02:38:35.354 INFO:tasks.rados.rados.0.smithi107.stdout:7470: writing smithi10729407-14 from 0 to 294912 tid 1 2020-06-12T02:38:35.354 INFO:tasks.rados.rados.0.smithi107.stdout:7471: delete oid 4 current snap is 0 2020-06-12T02:38:35.354 INFO:tasks.rados.rados.0.smithi107.stdout:7468: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:35.354 INFO:tasks.rados.rados.0.smithi107.stdout:7468: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:35.355 INFO:tasks.rados.rados.0.smithi107.stdout:7468: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:35.355 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 752 (ObjNum 7467 snap 0 seq_num 7467) dirty exists 2020-06-12T02:38:35.355 INFO:tasks.rados.rados.0.smithi107.stdout:7468: left oid 5 (ObjNum 7467 snap 0 seq_num 7467) 2020-06-12T02:38:35.356 INFO:tasks.rados.rados.0.smithi107.stdout:7468: done (2 left) 2020-06-12T02:38:35.356 INFO:tasks.rados.rados.0.smithi107.stdout:7471: done (1 left) 2020-06-12T02:38:35.357 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:38:35.358 INFO:tasks.rados.rados.0.smithi107.stdout:7472: seq_num 7471 ranges {0=327680} 2020-06-12T02:38:35.360 INFO:tasks.rados.rados.0.smithi107.stdout:7472: writing smithi10729407-13 from 0 to 327680 tid 1 2020-06-12T02:38:35.361 INFO:tasks.rados.rados.0.smithi107.stdout:7473: delete oid 18 current snap is 0 2020-06-12T02:38:35.361 INFO:tasks.rados.rados.0.smithi107.stdout:7470: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:35.361 INFO:tasks.rados.rados.0.smithi107.stdout:7470: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:35.362 INFO:tasks.rados.rados.0.smithi107.stdout:7470: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:35.362 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1327 (ObjNum 7469 snap 0 seq_num 7469) dirty exists 2020-06-12T02:38:35.362 INFO:tasks.rados.rados.0.smithi107.stdout:7470: left oid 14 (ObjNum 7469 snap 0 seq_num 7469) 2020-06-12T02:38:35.363 INFO:tasks.rados.rados.0.smithi107.stdout:7470: done (2 left) 2020-06-12T02:38:35.363 INFO:tasks.rados.rados.0.smithi107.stdout:7473: done (1 left) 2020-06-12T02:38:35.363 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:38:35.363 INFO:tasks.rados.rados.0.smithi107.stdout:7474: seq_num 7473 ranges {0=360448} 2020-06-12T02:38:35.364 INFO:tasks.rados.rados.0.smithi107.stdout:7474: writing smithi10729407-10 from 0 to 360448 tid 1 2020-06-12T02:38:35.364 INFO:tasks.rados.rados.0.smithi107.stdout:7475: delete oid 11 current snap is 0 2020-06-12T02:38:35.365 INFO:tasks.rados.rados.0.smithi107.stdout:7472: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:35.365 INFO:tasks.rados.rados.0.smithi107.stdout:7472: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:35.366 INFO:tasks.rados.rados.0.smithi107.stdout:7472: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:35.367 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1375 (ObjNum 7471 snap 0 seq_num 7471) dirty exists 2020-06-12T02:38:35.367 INFO:tasks.rados.rados.0.smithi107.stdout:7472: left oid 13 (ObjNum 7471 snap 0 seq_num 7471) 2020-06-12T02:38:35.369 INFO:tasks.rados.rados.0.smithi107.stdout:7472: done (2 left) 2020-06-12T02:38:35.369 INFO:tasks.rados.rados.0.smithi107.stdout:7475: done (1 left) 2020-06-12T02:38:35.370 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:38:35.370 INFO:tasks.rados.rados.0.smithi107.stdout:7476: seq_num 7475 ranges {3440640=393216} 2020-06-12T02:38:35.390 INFO:tasks.rados.rados.0.smithi107.stdout:7476: writing smithi10729407-5 from 3440640 to 3833856 tid 1 2020-06-12T02:38:35.390 INFO:tasks.rados.rados.0.smithi107.stdout:7477: delete oid 15 current snap is 0 2020-06-12T02:38:35.391 INFO:tasks.rados.rados.0.smithi107.stdout:7474: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:35.391 INFO:tasks.rados.rados.0.smithi107.stdout:7474: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:35.391 INFO:tasks.rados.rados.0.smithi107.stdout:7474: finishing write tid 4 to smithi10729407-10 2020-06-12T02:38:35.391 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 509 (ObjNum 7473 snap 0 seq_num 7473) dirty exists 2020-06-12T02:38:35.391 INFO:tasks.rados.rados.0.smithi107.stdout:7474: left oid 10 (ObjNum 7473 snap 0 seq_num 7473) 2020-06-12T02:38:35.392 INFO:tasks.rados.rados.0.smithi107.stdout:7474: done (2 left) 2020-06-12T02:38:35.393 INFO:tasks.rados.rados.0.smithi107.stdout:7477: done (1 left) 2020-06-12T02:38:35.393 INFO:tasks.rados.rados.0.smithi107.stdout:7478: delete oid 17 current snap is 0 2020-06-12T02:38:35.397 INFO:tasks.rados.rados.0.smithi107.stdout:7476: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:35.397 INFO:tasks.rados.rados.0.smithi107.stdout:7478: done (1 left) 2020-06-12T02:38:35.398 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:35.398 INFO:tasks.rados.rados.0.smithi107.stdout:7479: seq_num 7478 ranges {0=450560} 2020-06-12T02:38:35.401 INFO:tasks.rados.rados.0.smithi107.stdout:7479: writing smithi10729407-18 from 0 to 450560 tid 1 2020-06-12T02:38:35.401 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:35.401 INFO:tasks.rados.rados.0.smithi107.stdout:7480: seq_num 7479 ranges {294912=466944} 2020-06-12T02:38:35.405 INFO:tasks.rados.rados.0.smithi107.stdout:7480: writing smithi10729407-14 from 294912 to 761856 tid 1 2020-06-12T02:38:35.405 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:38:35.405 INFO:tasks.rados.rados.0.smithi107.stdout:7481: seq_num 7480 ranges {131072=483328} 2020-06-12T02:38:35.408 INFO:tasks.rados.rados.0.smithi107.stdout:7481: writing smithi10729407-16 from 131072 to 614400 tid 1 2020-06-12T02:38:35.409 INFO:tasks.rados.rados.0.smithi107.stdout:7482: delete oid 9 current snap is 0 2020-06-12T02:38:35.409 INFO:tasks.rados.rados.0.smithi107.stdout:7476: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:35.410 INFO:tasks.rados.rados.0.smithi107.stdout:7476: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:35.410 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 754 (ObjNum 7475 snap 0 seq_num 7475) dirty exists 2020-06-12T02:38:35.410 INFO:tasks.rados.rados.0.smithi107.stdout:7476: left oid 5 (ObjNum 7475 snap 0 seq_num 7475) 2020-06-12T02:38:35.410 INFO:tasks.rados.rados.0.smithi107.stdout:7479: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:35.411 INFO:tasks.rados.rados.0.smithi107.stdout:7479: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:35.412 INFO:tasks.rados.rados.0.smithi107.stdout:7479: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:35.412 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1403 (ObjNum 7478 snap 0 seq_num 7478) dirty exists 2020-06-12T02:38:35.412 INFO:tasks.rados.rados.0.smithi107.stdout:7479: left oid 18 (ObjNum 7478 snap 0 seq_num 7478) 2020-06-12T02:38:35.413 INFO:tasks.rados.rados.0.smithi107.stdout:7480: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:35.415 INFO:tasks.rados.rados.0.smithi107.stdout:7481: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:35.415 INFO:tasks.rados.rados.0.smithi107.stdout:7476: done (4 left) 2020-06-12T02:38:35.416 INFO:tasks.rados.rados.0.smithi107.stdout:7479: done (3 left) 2020-06-12T02:38:35.416 INFO:tasks.rados.rados.0.smithi107.stdout:7482: done (2 left) 2020-06-12T02:38:35.416 INFO:tasks.rados.rados.0.smithi107.stdout:7483: delete oid 3 current snap is 0 2020-06-12T02:38:35.416 INFO:tasks.rados.rados.0.smithi107.stdout:7481: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:35.417 INFO:tasks.rados.rados.0.smithi107.stdout:7480: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:35.418 INFO:tasks.rados.rados.0.smithi107.stdout:7481: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:35.418 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1353 (ObjNum 7480 snap 0 seq_num 7480) dirty exists 2020-06-12T02:38:35.418 INFO:tasks.rados.rados.0.smithi107.stdout:7481: left oid 16 (ObjNum 7480 snap 0 seq_num 7480) 2020-06-12T02:38:35.419 INFO:tasks.rados.rados.0.smithi107.stdout:7481: done (2 left) 2020-06-12T02:38:35.419 INFO:tasks.rados.rados.0.smithi107.stdout:7483: done (1 left) 2020-06-12T02:38:35.420 INFO:tasks.rados.rados.0.smithi107.stdout:7484: delete oid 16 current snap is 0 2020-06-12T02:38:35.420 INFO:tasks.rados.rados.0.smithi107.stdout:7480: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:35.420 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1330 (ObjNum 7479 snap 0 seq_num 7479) dirty exists 2020-06-12T02:38:35.420 INFO:tasks.rados.rados.0.smithi107.stdout:7480: left oid 14 (ObjNum 7479 snap 0 seq_num 7479) 2020-06-12T02:38:35.423 INFO:tasks.rados.rados.0.smithi107.stdout:7480: done (1 left) 2020-06-12T02:38:35.423 INFO:tasks.rados.rados.0.smithi107.stdout:7484: done (0 left) 2020-06-12T02:38:35.423 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:35.423 INFO:tasks.rados.rados.0.smithi107.stdout:7485: seq_num 7484 ranges {761856=548864} 2020-06-12T02:38:35.428 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:35.425+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:35.430 INFO:tasks.rados.rados.0.smithi107.stdout:7485: writing smithi10729407-14 from 761856 to 1310720 tid 1 2020-06-12T02:38:35.430 INFO:tasks.rados.rados.0.smithi107.stdout:7486: delete oid 2 current snap is 0 2020-06-12T02:38:35.432 INFO:tasks.rados.rados.0.smithi107.stdout:7486: done (1 left) 2020-06-12T02:38:35.433 INFO:tasks.rados.rados.0.smithi107.stdout:7487: delete oid 16 current snap is 0 2020-06-12T02:38:35.435 INFO:tasks.rados.rados.0.smithi107.stdout:7487: done (1 left) 2020-06-12T02:38:35.436 INFO:tasks.rados.rados.0.smithi107.stdout:7488: delete oid 19 current snap is 0 2020-06-12T02:38:35.436 INFO:tasks.rados.rados.0.smithi107.stdout:7485: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:35.438 INFO:tasks.rados.rados.0.smithi107.stdout:7488: done (1 left) 2020-06-12T02:38:35.439 INFO:tasks.rados.rados.0.smithi107.stdout:7489: delete oid 1 current snap is 0 2020-06-12T02:38:35.439 INFO:tasks.rados.rados.0.smithi107.stdout:7485: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:35.440 INFO:tasks.rados.rados.0.smithi107.stdout:7485: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:35.440 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1332 (ObjNum 7484 snap 0 seq_num 7484) dirty exists 2020-06-12T02:38:35.440 INFO:tasks.rados.rados.0.smithi107.stdout:7485: left oid 14 (ObjNum 7484 snap 0 seq_num 7484) 2020-06-12T02:38:35.442 INFO:tasks.rados.rados.0.smithi107.stdout:7485: done (1 left) 2020-06-12T02:38:35.443 INFO:tasks.rados.rados.0.smithi107.stdout:7489: done (0 left) 2020-06-12T02:38:35.443 INFO:tasks.rados.rados.0.smithi107.stdout:7490: delete oid 16 current snap is 0 2020-06-12T02:38:35.446 INFO:tasks.rados.rados.0.smithi107.stdout:7490: done (0 left) 2020-06-12T02:38:35.446 INFO:tasks.rados.rados.0.smithi107.stdout:7491: delete oid 8 current snap is 0 2020-06-12T02:38:35.449 INFO:tasks.rados.rados.0.smithi107.stdout:7491: done (0 left) 2020-06-12T02:38:35.449 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:35.449 INFO:tasks.rados.rados.0.smithi107.stdout:7492: seq_num 7491 ranges {0=663552} 2020-06-12T02:38:35.452 INFO:tasks.rados.rados.0.smithi107.stdout:7492: writing smithi10729407-8 from 0 to 663552 tid 1 2020-06-12T02:38:35.452 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:35.453 INFO:tasks.rados.rados.0.smithi107.stdout:7493: seq_num 7492 ranges {0=679936} 2020-06-12T02:38:35.456 INFO:tasks.rados.rados.0.smithi107.stdout:7493: writing smithi10729407-17 from 0 to 679936 tid 1 2020-06-12T02:38:35.456 INFO:tasks.rados.rados.0.smithi107.stdout:7494: delete oid 15 current snap is 0 2020-06-12T02:38:35.465 INFO:tasks.rados.rados.0.smithi107.stdout:7493: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:35.466 INFO:tasks.rados.rados.0.smithi107.stdout:7492: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:35.466 INFO:tasks.rados.rados.0.smithi107.stdout:7493: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:35.467 INFO:tasks.rados.rados.0.smithi107.stdout:7494: done (2 left) 2020-06-12T02:38:35.467 INFO:tasks.rados.rados.0.smithi107.stdout:7495: delete oid 4 current snap is 0 2020-06-12T02:38:35.468 INFO:tasks.rados.rados.0.smithi107.stdout:7493: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:35.469 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1417 (ObjNum 7492 snap 0 seq_num 7492) dirty exists 2020-06-12T02:38:35.469 INFO:tasks.rados.rados.0.smithi107.stdout:7493: left oid 17 (ObjNum 7492 snap 0 seq_num 7492) 2020-06-12T02:38:35.471 INFO:tasks.rados.rados.0.smithi107.stdout:7493: done (2 left) 2020-06-12T02:38:35.472 INFO:tasks.rados.rados.0.smithi107.stdout:7495: done (1 left) 2020-06-12T02:38:35.472 INFO:tasks.rados.rados.0.smithi107.stdout:7496: delete oid 19 current snap is 0 2020-06-12T02:38:35.479 INFO:tasks.rados.rados.0.smithi107.stdout:7496: done (1 left) 2020-06-12T02:38:35.480 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:35.480 INFO:tasks.rados.rados.0.smithi107.stdout:7497: seq_num 7496 ranges {0=753664} 2020-06-12T02:38:35.480 INFO:tasks.rados.rados.0.smithi107.stdout:7497: writing smithi10729407-4 from 0 to 753664 tid 1 2020-06-12T02:38:35.480 INFO:tasks.rados.rados.0.smithi107.stdout:7498: delete oid 20 current snap is 0 2020-06-12T02:38:35.481 INFO:tasks.rados.rados.0.smithi107.stdout:7492: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:35.481 INFO:tasks.rados.rados.0.smithi107.stdout:7492: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:35.481 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1397 (ObjNum 7491 snap 0 seq_num 7491) dirty exists 2020-06-12T02:38:35.481 INFO:tasks.rados.rados.0.smithi107.stdout:7492: left oid 8 (ObjNum 7491 snap 0 seq_num 7491) 2020-06-12T02:38:35.482 INFO:tasks.rados.rados.0.smithi107.stdout:7492: done (2 left) 2020-06-12T02:38:35.482 INFO:tasks.rados.rados.0.smithi107.stdout:7498: done (1 left) 2020-06-12T02:38:35.482 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:35.482 INFO:tasks.rados.rados.0.smithi107.stdout:7499: seq_num 7498 ranges {0=786432} 2020-06-12T02:38:35.485 INFO:tasks.rados.rados.0.smithi107.stdout:7499: writing smithi10729407-6 from 0 to 786432 tid 1 2020-06-12T02:38:35.486 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:35.486 INFO:tasks.rados.rados.0.smithi107.stdout:7500: seq_num 7499 ranges {663552=802816} 2020-06-12T02:38:35.494 INFO:tasks.rados.rados.0.smithi107.stdout:7500: writing smithi10729407-8 from 663552 to 1466368 tid 1 2020-06-12T02:38:35.494 INFO:tasks.rados.rados.0.smithi107.stdout:7501: delete oid 13 current snap is 0 2020-06-12T02:38:35.494 INFO:tasks.rados.rados.0.smithi107.stdout:7497: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:35.495 INFO:tasks.rados.rados.0.smithi107.stdout:7497: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:35.495 INFO:tasks.rados.rados.0.smithi107.stdout:7497: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:35.495 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1420 (ObjNum 7496 snap 0 seq_num 7496) dirty exists 2020-06-12T02:38:35.495 INFO:tasks.rados.rados.0.smithi107.stdout:7497: left oid 4 (ObjNum 7496 snap 0 seq_num 7496) 2020-06-12T02:38:35.495 INFO:tasks.rados.rados.0.smithi107.stdout:7499: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:35.501 INFO:tasks.rados.rados.0.smithi107.stdout:7499: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:35.504 INFO:tasks.rados.rados.0.smithi107.stdout:7497: done (3 left) 2020-06-12T02:38:35.505 INFO:tasks.rados.rados.0.smithi107.stdout:7501: done (2 left) 2020-06-12T02:38:35.505 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:35.505 INFO:tasks.rados.rados.0.smithi107.stdout:7502: seq_num 7501 ranges {0=434176,434176=401408} 2020-06-12T02:38:35.507 INFO:tasks.rados.rados.0.smithi107.stdout:7502: writing smithi10729407-15 from 0 to 434176 tid 1 2020-06-12T02:38:35.507 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:35.504+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:35.509 INFO:tasks.rados.rados.0.smithi107.stdout:7502: writing smithi10729407-15 from 434176 to 835584 tid 2 2020-06-12T02:38:35.509 INFO:tasks.rados.rados.0.smithi107.stdout:7503: delete oid 19 current snap is 0 2020-06-12T02:38:35.509 INFO:tasks.rados.rados.0.smithi107.stdout:7499: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:35.510 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1356 (ObjNum 7498 snap 0 seq_num 7498) dirty exists 2020-06-12T02:38:35.510 INFO:tasks.rados.rados.0.smithi107.stdout:7499: left oid 6 (ObjNum 7498 snap 0 seq_num 7498) 2020-06-12T02:38:35.510 INFO:tasks.rados.rados.0.smithi107.stdout:7500: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:35.510 INFO:tasks.rados.rados.0.smithi107.stdout:7500: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:35.511 INFO:tasks.rados.rados.0.smithi107.stdout:7500: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:35.511 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1399 (ObjNum 7499 snap 0 seq_num 7499) dirty exists 2020-06-12T02:38:35.512 INFO:tasks.rados.rados.0.smithi107.stdout:7500: left oid 8 (ObjNum 7499 snap 0 seq_num 7499) 2020-06-12T02:38:35.513 INFO:tasks.rados.rados.0.smithi107.stdout:7499: done (3 left) 2020-06-12T02:38:35.514 INFO:tasks.rados.rados.0.smithi107.stdout:7500: done (2 left) 2020-06-12T02:38:35.514 INFO:tasks.rados.rados.0.smithi107.stdout:7503: done (1 left) 2020-06-12T02:38:35.514 INFO:tasks.rados.rados.0.smithi107.stdout:7504: delete oid 1 current snap is 0 2020-06-12T02:38:35.517 INFO:tasks.rados.rados.0.smithi107.stdout:7504: done (1 left) 2020-06-12T02:38:35.518 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:38:35.518 INFO:tasks.rados.rados.0.smithi107.stdout:7505: seq_num 7504 ranges {0=483328,483328=401408} 2020-06-12T02:38:35.520 INFO:tasks.rados.rados.0.smithi107.stdout:7505: writing smithi10729407-3 from 0 to 483328 tid 1 2020-06-12T02:38:35.523 INFO:tasks.rados.rados.0.smithi107.stdout:7505: writing smithi10729407-3 from 483328 to 884736 tid 2 2020-06-12T02:38:35.523 INFO:tasks.rados.rados.0.smithi107.stdout:7506: delete oid 8 current snap is 0 2020-06-12T02:38:35.523 INFO:tasks.rados.rados.0.smithi107.stdout:7502: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:35.523 INFO:tasks.rados.rados.0.smithi107.stdout:7502: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:35.524 INFO:tasks.rados.rados.0.smithi107.stdout:7502: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:35.524 INFO:tasks.rados.rados.0.smithi107.stdout:7502: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:35.524 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1402 (ObjNum 7501 snap 0 seq_num 7501) dirty exists 2020-06-12T02:38:35.524 INFO:tasks.rados.rados.0.smithi107.stdout:7502: left oid 15 (ObjNum 7501 snap 0 seq_num 7501) 2020-06-12T02:38:35.525 INFO:tasks.rados.rados.0.smithi107.stdout:7502: done (2 left) 2020-06-12T02:38:35.526 INFO:tasks.rados.rados.0.smithi107.stdout:7506: done (1 left) 2020-06-12T02:38:35.526 INFO:tasks.rados.rados.0.smithi107.stdout:7507: delete oid 16 current snap is 0 2020-06-12T02:38:35.528 INFO:tasks.rados.rados.0.smithi107.stdout:7505: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:35.534 INFO:tasks.rados.rados.0.smithi107.stdout:7505: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:35.535 INFO:tasks.rados.rados.0.smithi107.stdout:7505: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:35.536 INFO:tasks.rados.rados.0.smithi107.stdout:7507: done (1 left) 2020-06-12T02:38:35.536 INFO:tasks.rados.rados.0.smithi107.stdout:7508: delete oid 8 current snap is 0 2020-06-12T02:38:35.536 INFO:tasks.rados.rados.0.smithi107.stdout:7505: finishing write tid 5 to smithi10729407-3 2020-06-12T02:38:35.537 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1360 (ObjNum 7504 snap 0 seq_num 7504) dirty exists 2020-06-12T02:38:35.537 INFO:tasks.rados.rados.0.smithi107.stdout:7505: left oid 3 (ObjNum 7504 snap 0 seq_num 7504) 2020-06-12T02:38:35.538 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:35.539 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:35.539 INFO:tasks.rados.rados.0.smithi107.stdout:7505: done (1 left) 2020-06-12T02:38:35.539 INFO:tasks.rados.rados.0.smithi107.stdout:7508: done (0 left) 2020-06-12T02:38:35.540 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:35.540 INFO:tasks.rados.rados.0.smithi107.stdout:7509: seq_num 7508 ranges {0=548864,548864=401408} 2020-06-12T02:38:35.542 INFO:tasks.rados.rados.0.smithi107.stdout:7509: writing smithi10729407-16 from 0 to 548864 tid 1 2020-06-12T02:38:35.545 INFO:tasks.rados.rados.0.smithi107.stdout:7509: writing smithi10729407-16 from 548864 to 950272 tid 2 2020-06-12T02:38:35.545 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:38:35.545 INFO:tasks.rados.rados.0.smithi107.stdout:7510: seq_num 7509 ranges {0=565248,565248=401408} 2020-06-12T02:38:35.548 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-06-12T02:38:35.548 INFO:tasks.rados.rados.0.smithi107.stdout:7510: writing smithi10729407-2 from 0 to 565248 tid 1 2020-06-12T02:38:35.551 INFO:tasks.rados.rados.0.smithi107.stdout:7510: writing smithi10729407-2 from 565248 to 966656 tid 2 2020-06-12T02:38:35.551 INFO:tasks.rados.rados.0.smithi107.stdout:7511: delete oid 10 current snap is 0 2020-06-12T02:38:35.551 INFO:tasks.rados.rados.0.smithi107.stdout:7509: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:35.553 INFO:tasks.rados.rados.0.smithi107.stdout:7509: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:35.555 INFO:tasks.rados.rados.0.smithi107.stdout:7511: done (2 left) 2020-06-12T02:38:35.556 INFO:tasks.rados.rados.0.smithi107.stdout:7512: delete oid 7 current snap is 0 2020-06-12T02:38:35.556 INFO:tasks.rados.rados.0.smithi107.stdout:7509: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:35.557 INFO:tasks.rados.rados.0.smithi107.stdout:7510: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:35.559 INFO:tasks.rados.rados.0.smithi107.stdout:7509: finishing write tid 5 to smithi10729407-16 2020-06-12T02:38:35.560 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1364 (ObjNum 7508 snap 0 seq_num 7508) dirty exists 2020-06-12T02:38:35.560 INFO:tasks.rados.rados.0.smithi107.stdout:7509: left oid 16 (ObjNum 7508 snap 0 seq_num 7508) 2020-06-12T02:38:35.560 INFO:tasks.rados.rados.0.smithi107.stdout:7509: done (2 left) 2020-06-12T02:38:35.561 INFO:tasks.rados.rados.0.smithi107.stdout:7512: done (1 left) 2020-06-12T02:38:35.561 INFO:tasks.rados.rados.0.smithi107.stdout:7513: delete oid 8 current snap is 0 2020-06-12T02:38:35.563 INFO:tasks.rados.rados.0.smithi107.stdout:7510: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:35.564 INFO:tasks.rados.rados.0.smithi107.stdout:7513: done (1 left) 2020-06-12T02:38:35.564 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:35.564 INFO:tasks.rados.rados.0.smithi107.stdout:7514: seq_num 7513 ranges {450560=630784,1081344=401408} 2020-06-12T02:38:35.569 INFO:tasks.rados.rados.0.smithi107.stdout:7514: writing smithi10729407-18 from 450560 to 1081344 tid 1 2020-06-12T02:38:35.571 INFO:tasks.rados.rados.0.smithi107.stdout:7514: writing smithi10729407-18 from 1081344 to 1482752 tid 2 2020-06-12T02:38:35.572 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:35.572 INFO:tasks.rados.rados.0.smithi107.stdout:7515: seq_num 7514 ranges {0=647168,647168=401408} 2020-06-12T02:38:35.575 INFO:tasks.rados.rados.0.smithi107.stdout:7515: writing smithi10729407-7 from 0 to 647168 tid 1 2020-06-12T02:38:35.578 INFO:tasks.rados.rados.0.smithi107.stdout:7515: writing smithi10729407-7 from 647168 to 1048576 tid 2 2020-06-12T02:38:35.578 INFO:tasks.rados.rados.0.smithi107.stdout:7516: delete oid 13 current snap is 0 2020-06-12T02:38:35.578 INFO:tasks.rados.rados.0.smithi107.stdout:7510: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:35.579 INFO:tasks.rados.rados.0.smithi107.stdout:7510: finishing write tid 5 to smithi10729407-2 2020-06-12T02:38:35.579 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 710 (ObjNum 7509 snap 0 seq_num 7509) dirty exists 2020-06-12T02:38:35.579 INFO:tasks.rados.rados.0.smithi107.stdout:7510: left oid 2 (ObjNum 7509 snap 0 seq_num 7509) 2020-06-12T02:38:35.579 INFO:tasks.rados.rados.0.smithi107.stdout:7514: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:35.580 INFO:tasks.rados.rados.0.smithi107.stdout:7514: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:35.581 INFO:tasks.rados.rados.0.smithi107.stdout:7514: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:35.582 INFO:tasks.rados.rados.0.smithi107.stdout:7514: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:35.582 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1407 (ObjNum 7513 snap 0 seq_num 7513) dirty exists 2020-06-12T02:38:35.582 INFO:tasks.rados.rados.0.smithi107.stdout:7514: left oid 18 (ObjNum 7513 snap 0 seq_num 7513) 2020-06-12T02:38:35.584 INFO:tasks.rados.rados.0.smithi107.stdout:7515: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:35.585 INFO:tasks.rados.rados.0.smithi107.stdout:7515: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:35.586 INFO:tasks.rados.rados.0.smithi107.stdout:7515: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:35.587 INFO:tasks.rados.rados.0.smithi107.stdout:7510: done (3 left) 2020-06-12T02:38:35.587 INFO:tasks.rados.rados.0.smithi107.stdout:7514: done (2 left) 2020-06-12T02:38:35.587 INFO:tasks.rados.rados.0.smithi107.stdout:7516: done (1 left) 2020-06-12T02:38:35.587 INFO:tasks.rados.rados.0.smithi107.stdout:7517: delete oid 15 current snap is 0 2020-06-12T02:38:35.588 INFO:tasks.rados.rados.0.smithi107.stdout:7515: finishing write tid 5 to smithi10729407-7 2020-06-12T02:38:35.588 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1380 (ObjNum 7514 snap 0 seq_num 7514) dirty exists 2020-06-12T02:38:35.588 INFO:tasks.rados.rados.0.smithi107.stdout:7515: left oid 7 (ObjNum 7514 snap 0 seq_num 7514) 2020-06-12T02:38:35.589 INFO:tasks.rados.rados.0.smithi107.stdout:7515: done (1 left) 2020-06-12T02:38:35.590 INFO:tasks.rados.rados.0.smithi107.stdout:7517: done (0 left) 2020-06-12T02:38:35.590 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:35.590 INFO:tasks.rados.rados.0.smithi107.stdout:7518: seq_num 7517 ranges {0=704512,704512=401408} 2020-06-12T02:38:35.594 INFO:tasks.rados.rados.0.smithi107.stdout:7518: writing smithi10729407-1 from 0 to 704512 tid 1 2020-06-12T02:38:35.596 INFO:tasks.rados.rados.0.smithi107.stdout:7518: writing smithi10729407-1 from 704512 to 1105920 tid 2 2020-06-12T02:38:35.596 INFO:tasks.rados.rados.0.smithi107.stdout:7519: delete oid 5 current snap is 0 2020-06-12T02:38:35.600 INFO:tasks.rados.rados.0.smithi107.stdout:7519: done (1 left) 2020-06-12T02:38:35.600 INFO:tasks.rados.rados.0.smithi107.stdout:7520: delete oid 14 current snap is 0 2020-06-12T02:38:35.601 INFO:tasks.rados.rados.0.smithi107.stdout:7518: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:35.604 INFO:tasks.rados.rados.0.smithi107.stdout:7520: done (1 left) 2020-06-12T02:38:35.604 INFO:tasks.rados.rados.0.smithi107.stdout:7521: delete oid 13 current snap is 0 2020-06-12T02:38:35.608 INFO:tasks.rados.rados.0.smithi107.stdout:7518: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:35.608 INFO:tasks.rados.rados.0.smithi107.stdout:7521: done (1 left) 2020-06-12T02:38:35.608 INFO:tasks.rados.rados.0.smithi107.stdout:7522: delete oid 16 current snap is 0 2020-06-12T02:38:35.609 INFO:tasks.rados.rados.0.smithi107.stdout:7518: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:35.609 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:35.605+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:35.610 INFO:tasks.rados.rados.0.smithi107.stdout:7518: finishing write tid 5 to smithi10729407-1 2020-06-12T02:38:35.610 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1360 (ObjNum 7517 snap 0 seq_num 7517) dirty exists 2020-06-12T02:38:35.610 INFO:tasks.rados.rados.0.smithi107.stdout:7518: left oid 1 (ObjNum 7517 snap 0 seq_num 7517) 2020-06-12T02:38:35.611 INFO:tasks.rados.rados.0.smithi107.stdout:7518: done (1 left) 2020-06-12T02:38:35.611 INFO:tasks.rados.rados.0.smithi107.stdout:7522: done (0 left) 2020-06-12T02:38:35.611 INFO:tasks.rados.rados.0.smithi107.stdout:7523: delete oid 3 current snap is 0 2020-06-12T02:38:35.613 INFO:tasks.rados.rados.0.smithi107.stdout:7523: done (0 left) 2020-06-12T02:38:35.614 INFO:tasks.rados.rados.0.smithi107.stdout:7524: delete oid 6 current snap is 0 2020-06-12T02:38:35.616 INFO:tasks.rados.rados.0.smithi107.stdout:7524: done (0 left) 2020-06-12T02:38:35.616 INFO:tasks.rados.rados.0.smithi107.stdout:7525: delete oid 1 current snap is 0 2020-06-12T02:38:35.618 INFO:tasks.rados.rados.0.smithi107.stdout:7525: done (0 left) 2020-06-12T02:38:35.619 INFO:tasks.rados.rados.0.smithi107.stdout:7526: delete oid 2 current snap is 0 2020-06-12T02:38:35.621 INFO:tasks.rados.rados.0.smithi107.stdout:7526: done (0 left) 2020-06-12T02:38:35.622 INFO:tasks.rados.rados.0.smithi107.stdout:7527: delete oid 9 current snap is 0 2020-06-12T02:38:35.625 INFO:tasks.rados.rados.0.smithi107.stdout:7527: done (0 left) 2020-06-12T02:38:35.625 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:35.626 INFO:tasks.rados.rados.0.smithi107.stdout:7528: seq_num 7527 ranges {0=466944,466944=557056,1024000=245760} 2020-06-12T02:38:35.627 INFO:tasks.rados.rados.0.smithi107.stdout:7528: writing smithi10729407-9 from 0 to 466944 tid 1 2020-06-12T02:38:35.631 INFO:tasks.rados.rados.0.smithi107.stdout:7528: writing smithi10729407-9 from 466944 to 1024000 tid 2 2020-06-12T02:38:35.632 INFO:tasks.rados.rados.0.smithi107.stdout:7528: writing smithi10729407-9 from 1024000 to 1269760 tid 3 2020-06-12T02:38:35.633 INFO:tasks.rados.rados.0.smithi107.stdout:7529: delete oid 3 current snap is 0 2020-06-12T02:38:35.633 INFO:tasks.rados.rados.0.smithi107.stdout:7528: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:35.635 INFO:tasks.rados.rados.0.smithi107.stdout:7529: done (1 left) 2020-06-12T02:38:35.635 INFO:tasks.rados.rados.0.smithi107.stdout:7530: delete oid 19 current snap is 0 2020-06-12T02:38:35.643 INFO:tasks.rados.rados.0.smithi107.stdout:7528: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:35.644 INFO:tasks.rados.rados.0.smithi107.stdout:7530: done (1 left) 2020-06-12T02:38:35.644 INFO:tasks.rados.rados.0.smithi107.stdout:7531: delete oid 13 current snap is 0 2020-06-12T02:38:35.644 INFO:tasks.rados.rados.0.smithi107.stdout:7528: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:35.644 INFO:tasks.rados.rados.0.smithi107.stdout:7528: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:35.644 INFO:tasks.rados.rados.0.smithi107.stdout:7531: done (1 left) 2020-06-12T02:38:35.645 INFO:tasks.rados.rados.0.smithi107.stdout:7532: delete oid 4 current snap is 0 2020-06-12T02:38:35.645 INFO:tasks.rados.rados.0.smithi107.stdout:7528: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:35.646 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1412 (ObjNum 7527 snap 0 seq_num 7527) dirty exists 2020-06-12T02:38:35.646 INFO:tasks.rados.rados.0.smithi107.stdout:7528: left oid 9 (ObjNum 7527 snap 0 seq_num 7527) 2020-06-12T02:38:35.649 INFO:tasks.rados.rados.0.smithi107.stdout:7528: done (1 left) 2020-06-12T02:38:35.649 INFO:tasks.rados.rados.0.smithi107.stdout:7532: done (0 left) 2020-06-12T02:38:35.649 INFO:tasks.rados.rados.0.smithi107.stdout:7533: delete oid 20 current snap is 0 2020-06-12T02:38:35.653 INFO:tasks.rados.rados.0.smithi107.stdout:7533: done (0 left) 2020-06-12T02:38:35.653 INFO:tasks.rados.rados.0.smithi107.stdout:7534: delete oid 3 current snap is 0 2020-06-12T02:38:35.660 INFO:tasks.rados.rados.0.smithi107.stdout:7534: done (0 left) 2020-06-12T02:38:35.660 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:35.660 INFO:tasks.rados.rados.0.smithi107.stdout:7535: seq_num 7534 ranges {1269760=589824,1859584=598016,2457600=204800} 2020-06-12T02:38:35.664 INFO:tasks.rados.rados.0.smithi107.stdout:7535: writing smithi10729407-9 from 1269760 to 1859584 tid 1 2020-06-12T02:38:35.668 INFO:tasks.rados.rados.0.smithi107.stdout:7535: writing smithi10729407-9 from 1859584 to 2457600 tid 2 2020-06-12T02:38:35.669 INFO:tasks.rados.rados.0.smithi107.stdout:7535: writing smithi10729407-9 from 2457600 to 2662400 tid 3 2020-06-12T02:38:35.670 INFO:tasks.rados.rados.0.smithi107.stdout:7536: delete oid 10 current snap is 0 2020-06-12T02:38:35.671 INFO:tasks.rados.rados.0.smithi107.stdout:7536: done (1 left) 2020-06-12T02:38:35.672 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:35.672 INFO:tasks.rados.rados.0.smithi107.stdout:7537: seq_num 7536 ranges {0=622592,622592=761856,1384448=40960} 2020-06-12T02:38:35.675 INFO:tasks.rados.rados.0.smithi107.stdout:7537: writing smithi10729407-16 from 0 to 622592 tid 1 2020-06-12T02:38:35.688 INFO:tasks.rados.rados.0.smithi107.stdout:7537: writing smithi10729407-16 from 622592 to 1384448 tid 2 2020-06-12T02:38:35.689 INFO:tasks.rados.rados.0.smithi107.stdout:7537: writing smithi10729407-16 from 1384448 to 1425408 tid 3 2020-06-12T02:38:35.689 INFO:tasks.rados.rados.0.smithi107.stdout:7538: delete oid 19 current snap is 0 2020-06-12T02:38:35.689 INFO:tasks.rados.rados.0.smithi107.stdout:7535: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:35.690 INFO:tasks.rados.rados.0.smithi107.stdout:7535: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:35.690 INFO:tasks.rados.rados.0.smithi107.stdout:7535: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:35.690 INFO:tasks.rados.rados.0.smithi107.stdout:7535: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:35.690 INFO:tasks.rados.rados.0.smithi107.stdout:7535: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:35.690 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1416 (ObjNum 7534 snap 0 seq_num 7534) dirty exists 2020-06-12T02:38:35.691 INFO:tasks.rados.rados.0.smithi107.stdout:7535: left oid 9 (ObjNum 7534 snap 0 seq_num 7534) 2020-06-12T02:38:35.691 INFO:tasks.rados.rados.0.smithi107.stdout:7537: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:35.691 INFO:tasks.rados.rados.0.smithi107.stdout:7535: done (2 left) 2020-06-12T02:38:35.691 INFO:tasks.rados.rados.0.smithi107.stdout:7538: done (1 left) 2020-06-12T02:38:35.691 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:35.692 INFO:tasks.rados.rados.0.smithi107.stdout:7539: seq_num 7538 ranges {0=655360,655360=532480,1187840=270336} 2020-06-12T02:38:35.693 INFO:tasks.rados.rados.0.smithi107.stdout:7539: writing smithi10729407-4 from 0 to 655360 tid 1 2020-06-12T02:38:35.694 INFO:tasks.rados.rados.0.smithi107.stdout:7539: writing smithi10729407-4 from 655360 to 1187840 tid 2 2020-06-12T02:38:35.695 INFO:tasks.rados.rados.0.smithi107.stdout:7539: writing smithi10729407-4 from 1187840 to 1458176 tid 3 2020-06-12T02:38:35.695 INFO:tasks.rados.rados.0.smithi107.stdout:7540: delete oid 2 current snap is 0 2020-06-12T02:38:35.695 INFO:tasks.rados.rados.0.smithi107.stdout:7537: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:35.695 INFO:tasks.rados.rados.0.smithi107.stdout:7537: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:35.696 INFO:tasks.rados.rados.0.smithi107.stdout:7537: finishing write tid 5 to smithi10729407-16 2020-06-12T02:38:35.696 INFO:tasks.rados.rados.0.smithi107.stdout:7537: finishing write tid 6 to smithi10729407-16 2020-06-12T02:38:35.696 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1372 (ObjNum 7536 snap 0 seq_num 7536) dirty exists 2020-06-12T02:38:35.696 INFO:tasks.rados.rados.0.smithi107.stdout:7537: left oid 16 (ObjNum 7536 snap 0 seq_num 7536) 2020-06-12T02:38:35.697 INFO:tasks.rados.rados.0.smithi107.stdout:7539: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:35.697 INFO:tasks.rados.rados.0.smithi107.stdout:7537: done (2 left) 2020-06-12T02:38:35.697 INFO:tasks.rados.rados.0.smithi107.stdout:7540: done (1 left) 2020-06-12T02:38:35.698 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:38:35.698 INFO:tasks.rados.rados.0.smithi107.stdout:7541: seq_num 7540 ranges {0=688128,688128=704512,1392640=98304} 2020-06-12T02:38:35.700 INFO:tasks.rados.rados.0.smithi107.stdout:7541: writing smithi10729407-2 from 0 to 688128 tid 1 2020-06-12T02:38:35.705 INFO:tasks.rados.rados.0.smithi107.stdout:7541: writing smithi10729407-2 from 688128 to 1392640 tid 2 2020-06-12T02:38:35.706 INFO:tasks.rados.rados.0.smithi107.stdout:7541: writing smithi10729407-2 from 1392640 to 1490944 tid 3 2020-06-12T02:38:35.706 INFO:tasks.rados.rados.0.smithi107.stdout:7542: delete oid 14 current snap is 0 2020-06-12T02:38:35.706 INFO:tasks.rados.rados.0.smithi107.stdout:7539: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:35.707 INFO:tasks.rados.rados.0.smithi107.stdout:7539: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:35.708 INFO:tasks.rados.rados.0.smithi107.stdout:7539: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:35.709 INFO:tasks.rados.rados.0.smithi107.stdout:7541: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:35.709 INFO:tasks.rados.rados.0.smithi107.stdout:7542: done (2 left) 2020-06-12T02:38:35.709 INFO:tasks.rados.rados.0.smithi107.stdout:7543: delete oid 3 current snap is 0 2020-06-12T02:38:35.710 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:35.706+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:35.710 INFO:tasks.rados.rados.0.smithi107.stdout:7539: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:35.710 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1425 (ObjNum 7538 snap 0 seq_num 7538) dirty exists 2020-06-12T02:38:35.712 INFO:tasks.rados.rados.0.smithi107.stdout:7539: left oid 4 (ObjNum 7538 snap 0 seq_num 7538) 2020-06-12T02:38:35.712 INFO:tasks.rados.rados.0.smithi107.stdout:7539: done (2 left) 2020-06-12T02:38:35.712 INFO:tasks.rados.rados.0.smithi107.stdout:7543: done (1 left) 2020-06-12T02:38:35.713 INFO:tasks.rados.rados.0.smithi107.stdout:7544: delete oid 8 current snap is 0 2020-06-12T02:38:35.715 INFO:tasks.rados.rados.0.smithi107.stdout:7544: done (1 left) 2020-06-12T02:38:35.715 INFO:tasks.rados.rados.0.smithi107.stdout:7545: delete oid 4 current snap is 0 2020-06-12T02:38:35.717 INFO:tasks.rados.rados.0.smithi107.stdout:7541: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:35.718 INFO:tasks.rados.rados.0.smithi107.stdout:7541: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:35.718 INFO:tasks.rados.rados.0.smithi107.stdout:7541: finishing write tid 5 to smithi10729407-2 2020-06-12T02:38:35.718 INFO:tasks.rados.rados.0.smithi107.stdout:7545: done (1 left) 2020-06-12T02:38:35.719 INFO:tasks.rados.rados.0.smithi107.stdout:7546: delete oid 7 current snap is 0 2020-06-12T02:38:35.720 INFO:tasks.rados.rados.0.smithi107.stdout:7541: finishing write tid 6 to smithi10729407-2 2020-06-12T02:38:35.720 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 716 (ObjNum 7540 snap 0 seq_num 7540) dirty exists 2020-06-12T02:38:35.721 INFO:tasks.rados.rados.0.smithi107.stdout:7541: left oid 2 (ObjNum 7540 snap 0 seq_num 7540) 2020-06-12T02:38:35.722 INFO:tasks.rados.rados.0.smithi107.stdout:7541: done (1 left) 2020-06-12T02:38:35.722 INFO:tasks.rados.rados.0.smithi107.stdout:7546: done (0 left) 2020-06-12T02:38:35.723 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:38:35.723 INFO:tasks.rados.rados.0.smithi107.stdout:7547: seq_num 7546 ranges {0=786432,786432=458752,1245184=344064} 2020-06-12T02:38:35.725 INFO:tasks.rados.rados.0.smithi107.stdout:7547: writing smithi10729407-6 from 0 to 786432 tid 1 2020-06-12T02:38:35.728 INFO:tasks.rados.rados.0.smithi107.stdout:7547: writing smithi10729407-6 from 786432 to 1245184 tid 2 2020-06-12T02:38:35.730 INFO:tasks.rados.rados.0.smithi107.stdout:7547: writing smithi10729407-6 from 1245184 to 1589248 tid 3 2020-06-12T02:38:35.730 INFO:tasks.rados.rados.0.smithi107.stdout:7548: delete oid 14 current snap is 0 2020-06-12T02:38:35.737 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:35.737 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:35.738 INFO:tasks.rados.rados.0.smithi107.stdout:7547: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:35.739 INFO:tasks.rados.rados.0.smithi107.stdout:7548: done (1 left) 2020-06-12T02:38:35.739 INFO:tasks.rados.rados.0.smithi107.stdout:7549: delete oid 19 current snap is 0 2020-06-12T02:38:35.740 INFO:tasks.rados.rados.0.smithi107.stdout:7547: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:35.740 INFO:tasks.rados.rados.0.smithi107.stdout:7547: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:35.741 INFO:tasks.rados.rados.0.smithi107.stdout:7547: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:35.741 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd set noscrub 2020-06-12T02:38:35.742 INFO:tasks.rados.rados.0.smithi107.stdout:7547: finishing write tid 6 to smithi10729407-6 2020-06-12T02:38:35.742 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1366 (ObjNum 7546 snap 0 seq_num 7546) dirty exists 2020-06-12T02:38:35.743 INFO:tasks.rados.rados.0.smithi107.stdout:7547: left oid 6 (ObjNum 7546 snap 0 seq_num 7546) 2020-06-12T02:38:35.743 INFO:tasks.rados.rados.0.smithi107.stdout:7547: done (1 left) 2020-06-12T02:38:35.743 INFO:tasks.rados.rados.0.smithi107.stdout:7549: done (0 left) 2020-06-12T02:38:35.743 INFO:tasks.rados.rados.0.smithi107.stdout:7550: delete oid 1 current snap is 0 2020-06-12T02:38:35.744 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-06-12T02:38:35.746 INFO:tasks.rados.rados.0.smithi107.stdout:7550: done (0 left) 2020-06-12T02:38:35.748 INFO:tasks.rados.rados.0.smithi107.stdout:7551: delete oid 18 current snap is 0 2020-06-12T02:38:35.751 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-06-12T02:38:35.751 INFO:tasks.rados.rados.0.smithi107.stdout:7551: done (0 left) 2020-06-12T02:38:35.751 INFO:tasks.rados.rados.0.smithi107.stdout:7552: delete oid 7 current snap is 0 2020-06-12T02:38:35.752 INFO:tasks.rados.rados.0.smithi107.stdout:7552: done (0 left) 2020-06-12T02:38:35.752 INFO:tasks.rados.rados.0.smithi107.stdout:7553: delete oid 16 current snap is 0 2020-06-12T02:38:35.753 INFO:tasks.rados.rados.0.smithi107.stdout:7553: done (0 left) 2020-06-12T02:38:35.753 INFO:tasks.rados.rados.0.smithi107.stdout:7554: delete oid 16 current snap is 0 2020-06-12T02:38:35.756 INFO:tasks.rados.rados.0.smithi107.stdout:7554: done (0 left) 2020-06-12T02:38:35.756 INFO:tasks.rados.rados.0.smithi107.stdout:7555: delete oid 7 current snap is 0 2020-06-12T02:38:35.759 INFO:tasks.rados.rados.0.smithi107.stdout:7555: done (0 left) 2020-06-12T02:38:35.759 INFO:tasks.rados.rados.0.smithi107.stdout:7556: delete oid 7 current snap is 0 2020-06-12T02:38:35.761 INFO:tasks.rados.rados.0.smithi107.stdout:7556: done (0 left) 2020-06-12T02:38:35.762 INFO:tasks.rados.rados.0.smithi107.stdout:7557: delete oid 14 current snap is 0 2020-06-12T02:38:35.764 INFO:tasks.rados.rados.0.smithi107.stdout:7557: done (0 left) 2020-06-12T02:38:35.764 INFO:tasks.rados.rados.0.smithi107.stdout:7558: delete oid 2 current snap is 0 2020-06-12T02:38:35.766 INFO:tasks.rados.rados.0.smithi107.stdout:7558: done (0 left) 2020-06-12T02:38:35.766 INFO:tasks.rados.rados.0.smithi107.stdout:7559: delete oid 14 current snap is 0 2020-06-12T02:38:35.769 INFO:tasks.rados.rados.0.smithi107.stdout:7559: done (0 left) 2020-06-12T02:38:35.769 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:35.769 INFO:tasks.rados.rados.0.smithi107.stdout:7560: seq_num 7559 ranges {0=606208,606208=655360,1261568=524288,1785856=24576} 2020-06-12T02:38:35.772 INFO:tasks.rados.rados.0.smithi107.stdout:7560: writing smithi10729407-14 from 0 to 606208 tid 1 2020-06-12T02:38:35.777 INFO:tasks.rados.rados.0.smithi107.stdout:7560: writing smithi10729407-14 from 606208 to 1261568 tid 2 2020-06-12T02:38:35.780 INFO:tasks.rados.rados.0.smithi107.stdout:7560: writing smithi10729407-14 from 1261568 to 1785856 tid 3 2020-06-12T02:38:35.780 INFO:tasks.rados.rados.0.smithi107.stdout:7560: writing smithi10729407-14 from 1785856 to 1810432 tid 4 2020-06-12T02:38:35.781 INFO:tasks.rados.rados.0.smithi107.stdout:7561: delete oid 20 current snap is 0 2020-06-12T02:38:35.782 INFO:tasks.rados.rados.0.smithi107.stdout:7560: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:35.786 INFO:tasks.rados.rados.0.smithi107.stdout:7561: done (1 left) 2020-06-12T02:38:35.786 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:35.786 INFO:tasks.rados.rados.0.smithi107.stdout:7562: seq_num 7561 ranges {0=638976,638976=417792,1056768=540672,1597440=245760} 2020-06-12T02:38:35.790 INFO:tasks.rados.rados.0.smithi107.stdout:7562: writing smithi10729407-19 from 0 to 638976 tid 1 2020-06-12T02:38:35.792 INFO:tasks.rados.rados.0.smithi107.stdout:7562: writing smithi10729407-19 from 638976 to 1056768 tid 2 2020-06-12T02:38:35.795 INFO:tasks.rados.rados.0.smithi107.stdout:7562: writing smithi10729407-19 from 1056768 to 1597440 tid 3 2020-06-12T02:38:35.797 INFO:tasks.rados.rados.0.smithi107.stdout:7562: writing smithi10729407-19 from 1597440 to 1843200 tid 4 2020-06-12T02:38:35.797 INFO:tasks.rados.rados.0.smithi107.stdout:7563: delete oid 3 current snap is 0 2020-06-12T02:38:35.798 INFO:tasks.rados.rados.0.smithi107.stdout:7560: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:35.801 INFO:tasks.rados.rados.0.smithi107.stdout:7563: done (2 left) 2020-06-12T02:38:35.801 INFO:tasks.rados.rados.0.smithi107.stdout:7564: delete oid 4 current snap is 0 2020-06-12T02:38:35.801 INFO:tasks.rados.rados.0.smithi107.stdout:7560: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:35.802 INFO:tasks.rados.rados.0.smithi107.stdout:7560: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:35.802 INFO:tasks.rados.rados.0.smithi107.stdout:7560: finishing write tid 6 to smithi10729407-14 2020-06-12T02:38:35.804 INFO:tasks.rados.rados.0.smithi107.stdout:7560: finishing write tid 7 to smithi10729407-14 2020-06-12T02:38:35.804 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1342 (ObjNum 7559 snap 0 seq_num 7559) dirty exists 2020-06-12T02:38:35.804 INFO:tasks.rados.rados.0.smithi107.stdout:7560: left oid 14 (ObjNum 7559 snap 0 seq_num 7559) 2020-06-12T02:38:35.805 INFO:tasks.rados.rados.0.smithi107.stdout:7560: done (2 left) 2020-06-12T02:38:35.805 INFO:tasks.rados.rados.0.smithi107.stdout:7564: done (1 left) 2020-06-12T02:38:35.805 INFO:tasks.rados.rados.0.smithi107.stdout:7565: delete oid 11 current snap is 0 2020-06-12T02:38:35.806 INFO:tasks.rados.rados.0.smithi107.stdout:7562: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:35.806 INFO:tasks.rados.rados.0.smithi107.stdout:7562: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:35.807 INFO:tasks.rados.rados.0.smithi107.stdout:7562: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:35.807 INFO:tasks.rados.rados.0.smithi107.stdout:7565: done (1 left) 2020-06-12T02:38:35.808 INFO:tasks.rados.rados.0.smithi107.stdout:7566: delete oid 4 current snap is 0 2020-06-12T02:38:35.808 INFO:tasks.rados.rados.0.smithi107.stdout:7562: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:35.808 INFO:tasks.rados.rados.0.smithi107.stdout:7562: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:35.810 INFO:tasks.rados.rados.0.smithi107.stdout:7566: done (1 left) 2020-06-12T02:38:35.811 INFO:tasks.rados.rados.0.smithi107.stdout:7567: delete oid 20 current snap is 0 2020-06-12T02:38:35.812 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:35.807+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:35.813 INFO:tasks.rados.rados.0.smithi107.stdout:7562: finishing write tid 7 to smithi10729407-19 2020-06-12T02:38:35.813 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1385 (ObjNum 7561 snap 0 seq_num 7561) dirty exists 2020-06-12T02:38:35.813 INFO:tasks.rados.rados.0.smithi107.stdout:7562: left oid 19 (ObjNum 7561 snap 0 seq_num 7561) 2020-06-12T02:38:35.813 INFO:tasks.rados.rados.0.smithi107.stdout:7562: done (1 left) 2020-06-12T02:38:35.814 INFO:tasks.rados.rados.0.smithi107.stdout:7567: done (0 left) 2020-06-12T02:38:35.814 INFO:tasks.rados.rados.0.smithi107.stdout:7568: delete oid 9 current snap is 0 2020-06-12T02:38:35.815 INFO:tasks.rados.rados.0.smithi107.stdout:7568: done (0 left) 2020-06-12T02:38:35.815 INFO:tasks.rados.rados.0.smithi107.stdout:7569: delete oid 17 current snap is 0 2020-06-12T02:38:35.818 INFO:tasks.rados.rados.0.smithi107.stdout:7569: done (0 left) 2020-06-12T02:38:35.818 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:35.818 INFO:tasks.rados.rados.0.smithi107.stdout:7570: seq_num 7569 ranges {0=770048,770048=745472,1515520=458752} 2020-06-12T02:38:35.824 INFO:tasks.rados.rados.0.smithi107.stdout:7570: writing smithi10729407-18 from 0 to 770048 tid 1 2020-06-12T02:38:35.825 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2020-06-12T02:38:35.825 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2020-06-12T02:38:35.826 INFO:tasks.thrashosds.thrasher:inject_pause on 2 2020-06-12T02:38:35.826 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2020-06-12T02:38:35.826 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2020-06-12T02:38:35.826 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set filestore_inject_stall 3 2020-06-12T02:38:35.830 INFO:tasks.rados.rados.0.smithi107.stdout:7570: writing smithi10729407-18 from 770048 to 1515520 tid 2 2020-06-12T02:38:35.833 INFO:tasks.rados.rados.0.smithi107.stdout:7570: writing smithi10729407-18 from 1515520 to 1974272 tid 3 2020-06-12T02:38:35.833 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:35.833 INFO:tasks.rados.rados.0.smithi107.stdout:7571: seq_num 7570 ranges {0=786432,786432=630784,1417216=573440} 2020-06-12T02:38:35.839 INFO:tasks.rados.rados.0.smithi107.stdout:7571: writing smithi10729407-8 from 0 to 786432 tid 1 2020-06-12T02:38:35.843 INFO:tasks.rados.rados.0.smithi107.stdout:7571: writing smithi10729407-8 from 786432 to 1417216 tid 2 2020-06-12T02:38:35.847 INFO:tasks.rados.rados.0.smithi107.stdout:7571: writing smithi10729407-8 from 1417216 to 1990656 tid 3 2020-06-12T02:38:35.848 INFO:tasks.rados.rados.0.smithi107.stdout:7572: delete oid 15 current snap is 0 2020-06-12T02:38:35.848 INFO:tasks.rados.rados.0.smithi107.stdout:7570: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:35.848 INFO:tasks.rados.rados.0.smithi107.stdout:7570: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:35.848 INFO:tasks.rados.rados.0.smithi107.stdout:7570: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:35.848 INFO:tasks.rados.rados.0.smithi107.stdout:7570: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:35.849 INFO:tasks.rados.rados.0.smithi107.stdout:7570: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:35.849 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1422 (ObjNum 7569 snap 0 seq_num 7569) dirty exists 2020-06-12T02:38:35.849 INFO:tasks.rados.rados.0.smithi107.stdout:7570: left oid 18 (ObjNum 7569 snap 0 seq_num 7569) 2020-06-12T02:38:35.855 INFO:tasks.rados.rados.0.smithi107.stdout:7571: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:35.857 INFO:tasks.rados.rados.0.smithi107.stdout:7571: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:35.858 INFO:tasks.rados.rados.0.smithi107.stdout:7571: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:35.858 INFO:tasks.rados.rados.0.smithi107.stdout:7571: finishing write tid 5 to smithi10729407-8 2020-06-12T02:38:35.859 INFO:tasks.rados.rados.0.smithi107.stdout:7570: done (2 left) 2020-06-12T02:38:35.860 INFO:tasks.rados.rados.0.smithi107.stdout:7572: done (1 left) 2020-06-12T02:38:35.860 INFO:tasks.rados.rados.0.smithi107.stdout:7573: delete oid 11 current snap is 0 2020-06-12T02:38:35.860 INFO:tasks.rados.rados.0.smithi107.stdout:7571: finishing write tid 6 to smithi10729407-8 2020-06-12T02:38:35.861 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1411 (ObjNum 7570 snap 0 seq_num 7570) dirty exists 2020-06-12T02:38:35.861 INFO:tasks.rados.rados.0.smithi107.stdout:7571: left oid 8 (ObjNum 7570 snap 0 seq_num 7570) 2020-06-12T02:38:35.862 INFO:tasks.rados.rados.0.smithi107.stdout:7571: done (1 left) 2020-06-12T02:38:35.863 INFO:tasks.rados.rados.0.smithi107.stdout:7573: done (0 left) 2020-06-12T02:38:35.863 INFO:tasks.rados.rados.0.smithi107.stdout:7574: delete oid 14 current snap is 0 2020-06-12T02:38:35.865 INFO:tasks.rados.rados.0.smithi107.stdout:7574: done (0 left) 2020-06-12T02:38:35.866 INFO:tasks.rados.rados.0.smithi107.stdout:7575: delete oid 8 current snap is 0 2020-06-12T02:38:35.871 INFO:tasks.rados.rados.0.smithi107.stdout:7575: done (0 left) 2020-06-12T02:38:35.871 INFO:tasks.rados.rados.0.smithi107.stdout:7576: delete oid 20 current snap is 0 2020-06-12T02:38:35.876 INFO:tasks.rados.rados.0.smithi107.stdout:7576: done (0 left) 2020-06-12T02:38:35.877 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:35.877 INFO:tasks.rados.rados.0.smithi107.stdout:7577: seq_num 7576 ranges {0=491520,491520=786432,1277952=655360,1933312=163840} 2020-06-12T02:38:35.880 INFO:tasks.rados.rados.0.smithi107.stdout:7577: writing smithi10729407-15 from 0 to 491520 tid 1 2020-06-12T02:38:35.891 INFO:tasks.rados.rados.0.smithi107.stdout:7577: writing smithi10729407-15 from 491520 to 1277952 tid 2 2020-06-12T02:38:35.892 INFO:tasks.rados.rados.0.smithi107.stdout:7577: writing smithi10729407-15 from 1277952 to 1933312 tid 3 2020-06-12T02:38:35.893 INFO:tasks.rados.rados.0.smithi107.stdout:7577: writing smithi10729407-15 from 1933312 to 2097152 tid 4 2020-06-12T02:38:35.893 INFO:tasks.rados.rados.0.smithi107.stdout:7578: delete oid 3 current snap is 0 2020-06-12T02:38:35.894 INFO:tasks.rados.rados.0.smithi107.stdout:7577: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:35.896 INFO:tasks.rados.rados.0.smithi107.stdout:7578: done (1 left) 2020-06-12T02:38:35.897 INFO:tasks.rados.rados.0.smithi107.stdout:7579: delete oid 5 current snap is 0 2020-06-12T02:38:35.900 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:35.901 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:35.901 INFO:tasks.rados.rados.0.smithi107.stdout:7579: done (1 left) 2020-06-12T02:38:35.902 INFO:tasks.rados.rados.0.smithi107.stdout:7580: delete oid 19 current snap is 0 2020-06-12T02:38:35.903 INFO:tasks.rados.rados.0.smithi107.stdout:7577: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:35.904 INFO:tasks.rados.rados.0.smithi107.stdout:7577: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:35.904 INFO:tasks.rados.rados.0.smithi107.stdout:7577: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:35.905 INFO:tasks.rados.rados.0.smithi107.stdout:7577: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:35.905 INFO:tasks.rados.rados.0.smithi107.stdout:7580: done (1 left) 2020-06-12T02:38:35.905 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:35.905 INFO:tasks.rados.rados.0.smithi107.stdout:7581: seq_num 7580 ranges {0=557056,557056=770048,1327104=630784,1957888=204800} 2020-06-12T02:38:35.909 INFO:tasks.rados.rados.0.smithi107.stdout:7581: writing smithi10729407-19 from 0 to 557056 tid 1 2020-06-12T02:38:35.911 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:35.908+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:35.912 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-06-12T02:38:35.915 INFO:tasks.rados.rados.0.smithi107.stdout:7581: writing smithi10729407-19 from 557056 to 1327104 tid 2 2020-06-12T02:38:35.917 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:38:35.919 INFO:tasks.rados.rados.0.smithi107.stdout:7581: writing smithi10729407-19 from 1327104 to 1957888 tid 3 2020-06-12T02:38:35.921 INFO:tasks.rados.rados.0.smithi107.stdout:7581: writing smithi10729407-19 from 1957888 to 2162688 tid 4 2020-06-12T02:38:35.921 INFO:tasks.rados.rados.0.smithi107.stdout:7582: delete oid 9 current snap is 0 2020-06-12T02:38:35.922 INFO:tasks.rados.rados.0.smithi107.stdout:7577: finishing write tid 7 to smithi10729407-15 2020-06-12T02:38:35.922 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1418 (ObjNum 7576 snap 0 seq_num 7576) dirty exists 2020-06-12T02:38:35.922 INFO:tasks.rados.rados.0.smithi107.stdout:7577: left oid 15 (ObjNum 7576 snap 0 seq_num 7576) 2020-06-12T02:38:35.922 INFO:tasks.rados.rados.0.smithi107.stdout:7581: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:35.924 INFO:tasks.rados.rados.0.smithi107.stdout:7577: done (2 left) 2020-06-12T02:38:35.925 INFO:tasks.rados.rados.0.smithi107.stdout:7582: done (1 left) 2020-06-12T02:38:35.925 INFO:tasks.rados.rados.0.smithi107.stdout:7583: delete oid 14 current snap is 0 2020-06-12T02:38:35.929 INFO:tasks.rados.rados.0.smithi107.stdout:7583: done (1 left) 2020-06-12T02:38:35.929 INFO:tasks.rados.rados.0.smithi107.stdout:7584: delete oid 3 current snap is 0 2020-06-12T02:38:35.932 INFO:tasks.rados.rados.0.smithi107.stdout:7581: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:35.933 INFO:tasks.rados.rados.0.smithi107.stdout:7581: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:35.933 INFO:tasks.rados.rados.0.smithi107.stdout:7581: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:35.933 INFO:tasks.rados.rados.0.smithi107.stdout:7581: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:35.934 INFO:tasks.rados.rados.0.smithi107.stdout:7584: done (1 left) 2020-06-12T02:38:35.934 INFO:tasks.rados.rados.0.smithi107.stdout:7585: delete oid 9 current snap is 0 2020-06-12T02:38:35.934 INFO:tasks.rados.rados.0.smithi107.stdout:7581: finishing write tid 7 to smithi10729407-19 2020-06-12T02:38:35.935 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1391 (ObjNum 7580 snap 0 seq_num 7580) dirty exists 2020-06-12T02:38:35.935 INFO:tasks.rados.rados.0.smithi107.stdout:7581: left oid 19 (ObjNum 7580 snap 0 seq_num 7580) 2020-06-12T02:38:35.936 INFO:tasks.rados.rados.0.smithi107.stdout:7581: done (1 left) 2020-06-12T02:38:35.938 INFO:tasks.rados.rados.0.smithi107.stdout:7585: done (0 left) 2020-06-12T02:38:35.938 INFO:tasks.rados.rados.0.smithi107.stdout:7586: delete oid 3 current snap is 0 2020-06-12T02:38:35.939 INFO:tasks.rados.rados.0.smithi107.stdout:7586: done (0 left) 2020-06-12T02:38:35.940 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:38:35.940 INFO:tasks.rados.rados.0.smithi107.stdout:7587: seq_num 7586 ranges {0=655360,655360=475136,1130496=630784,1761280=499712} 2020-06-12T02:38:35.943 INFO:tasks.rados.rados.0.smithi107.stdout:7587: writing smithi10729407-13 from 0 to 655360 tid 1 2020-06-12T02:38:35.946 INFO:tasks.rados.rados.0.smithi107.stdout:7587: writing smithi10729407-13 from 655360 to 1130496 tid 2 2020-06-12T02:38:35.948 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:38:35.950 INFO:tasks.rados.rados.0.smithi107.stdout:7587: writing smithi10729407-13 from 1130496 to 1761280 tid 3 2020-06-12T02:38:35.954 INFO:tasks.rados.rados.0.smithi107.stdout:7587: writing smithi10729407-13 from 1761280 to 2260992 tid 4 2020-06-12T02:38:35.955 INFO:tasks.rados.rados.0.smithi107.stdout:7588: delete oid 10 current snap is 0 2020-06-12T02:38:35.957 INFO:tasks.rados.rados.0.smithi107.stdout:7587: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:35.958 INFO:tasks.rados.rados.0.smithi107.stdout:7588: done (1 left) 2020-06-12T02:38:35.958 INFO:tasks.rados.rados.0.smithi107.stdout:7589: delete oid 16 current snap is 0 2020-06-12T02:38:35.965 INFO:tasks.rados.rados.0.smithi107.stdout:7587: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:35.965 INFO:tasks.rados.rados.0.smithi107.stdout:7587: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:35.965 INFO:tasks.rados.rados.0.smithi107.stdout:7587: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:35.967 INFO:tasks.rados.rados.0.smithi107.stdout:7587: finishing write tid 6 to smithi10729407-13 2020-06-12T02:38:35.967 INFO:tasks.rados.rados.0.smithi107.stdout:7589: done (1 left) 2020-06-12T02:38:35.967 INFO:tasks.rados.rados.0.smithi107.stdout:7590: delete oid 11 current snap is 0 2020-06-12T02:38:35.974 INFO:tasks.rados.rados.0.smithi107.stdout:7587: finishing write tid 7 to smithi10729407-13 2020-06-12T02:38:35.974 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1392 (ObjNum 7586 snap 0 seq_num 7586) dirty exists 2020-06-12T02:38:35.974 INFO:tasks.rados.rados.0.smithi107.stdout:7587: left oid 13 (ObjNum 7586 snap 0 seq_num 7586) 2020-06-12T02:38:35.975 INFO:tasks.rados.rados.0.smithi107.stdout:7587: done (1 left) 2020-06-12T02:38:35.975 INFO:tasks.rados.rados.0.smithi107.stdout:7590: done (0 left) 2020-06-12T02:38:35.975 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:38:35.976 INFO:tasks.rados.rados.0.smithi107.stdout:7591: seq_num 7590 ranges {0=720896,720896=466944,1187840=614400,1802240=524288} 2020-06-12T02:38:35.976 INFO:tasks.rados.rados.0.smithi107.stdout:7591: writing smithi10729407-20 from 0 to 720896 tid 1 2020-06-12T02:38:35.979 INFO:tasks.rados.rados.0.smithi107.stdout:7591: writing smithi10729407-20 from 720896 to 1187840 tid 2 2020-06-12T02:38:35.980 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:38:35.983 INFO:tasks.rados.rados.0.smithi107.stdout:7591: writing smithi10729407-20 from 1187840 to 1802240 tid 3 2020-06-12T02:38:35.987 INFO:tasks.rados.rados.0.smithi107.stdout:7591: writing smithi10729407-20 from 1802240 to 2326528 tid 4 2020-06-12T02:38:35.987 INFO:tasks.rados.rados.0.smithi107.stdout:7592: delete oid 18 current snap is 0 2020-06-12T02:38:35.987 INFO:tasks.rados.rados.0.smithi107.stdout:7591: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:35.994 INFO:tasks.rados.rados.0.smithi107.stdout:7592: done (1 left) 2020-06-12T02:38:35.994 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:35.994 INFO:tasks.rados.rados.0.smithi107.stdout:7593: seq_num 7592 ranges {0=761856,761856=630784,1392640=573440,1966080=401408} 2020-06-12T02:38:35.996 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:35.996 INFO:teuthology.orchestra.run.smithi107.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2020-06-12T02:38:35.996 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:35.998 INFO:tasks.rados.rados.0.smithi107.stdout:7593: writing smithi10729407-16 from 0 to 761856 tid 1 2020-06-12T02:38:36.002 INFO:tasks.rados.rados.0.smithi107.stdout:7593: writing smithi10729407-16 from 761856 to 1392640 tid 2 2020-06-12T02:38:36.006 INFO:tasks.rados.rados.0.smithi107.stdout:7593: writing smithi10729407-16 from 1392640 to 1966080 tid 3 2020-06-12T02:38:36.011 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:36.008+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:36.012 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:38:36.013 INFO:tasks.rados.rados.0.smithi107.stdout:7593: writing smithi10729407-16 from 1966080 to 2367488 tid 4 2020-06-12T02:38:36.013 INFO:tasks.rados.rados.0.smithi107.stdout:7594: delete oid 4 current snap is 0 2020-06-12T02:38:36.013 INFO:tasks.rados.rados.0.smithi107.stdout:7591: finishing write tid 2 to smithi10729407-20 2020-06-12T02:38:36.013 INFO:tasks.rados.rados.0.smithi107.stdout:7591: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:36.014 INFO:tasks.rados.rados.0.smithi107.stdout:7591: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:36.014 INFO:tasks.rados.rados.0.smithi107.stdout:7591: finishing write tid 6 to smithi10729407-20 2020-06-12T02:38:36.014 INFO:tasks.rados.rados.0.smithi107.stdout:7591: finishing write tid 7 to smithi10729407-20 2020-06-12T02:38:36.014 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 557 (ObjNum 7590 snap 0 seq_num 7590) dirty exists 2020-06-12T02:38:36.014 INFO:tasks.rados.rados.0.smithi107.stdout:7591: left oid 20 (ObjNum 7590 snap 0 seq_num 7590) 2020-06-12T02:38:36.015 INFO:tasks.rados.rados.0.smithi107.stdout:7593: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:36.016 INFO:tasks.rados.rados.0.smithi107.stdout:7591: done (2 left) 2020-06-12T02:38:36.016 INFO:tasks.rados.rados.0.smithi107.stdout:7594: done (1 left) 2020-06-12T02:38:36.016 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:36.016 INFO:tasks.rados.rados.0.smithi107.stdout:7595: seq_num 7594 ranges {0=794624,794624=802816,1597440=589824,2187264=212992} 2020-06-12T02:38:36.020 INFO:tasks.rados.rados.0.smithi107.stdout:7595: writing smithi10729407-9 from 0 to 794624 tid 1 2020-06-12T02:38:36.026 INFO:tasks.rados.rados.0.smithi107.stdout:7595: writing smithi10729407-9 from 794624 to 1597440 tid 2 2020-06-12T02:38:36.030 INFO:tasks.rados.rados.0.smithi107.stdout:7595: writing smithi10729407-9 from 1597440 to 2187264 tid 3 2020-06-12T02:38:36.032 INFO:tasks.rados.rados.0.smithi107.stdout:7595: writing smithi10729407-9 from 2187264 to 2400256 tid 4 2020-06-12T02:38:36.032 INFO:tasks.rados.rados.0.smithi107.stdout:7596: delete oid 20 current snap is 0 2020-06-12T02:38:36.037 INFO:tasks.rados.rados.0.smithi107.stdout:7595: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:36.041 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:38:36.042 INFO:tasks.rados.rados.0.smithi107.stdout:7596: done (2 left) 2020-06-12T02:38:36.042 INFO:tasks.rados.rados.0.smithi107.stdout:7597: delete oid 4 current snap is 0 2020-06-12T02:38:36.045 INFO:tasks.rados.rados.0.smithi107.stdout:7593: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:36.046 INFO:tasks.rados.rados.0.smithi107.stdout:7593: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:36.055 INFO:tasks.rados.rados.0.smithi107.stdout:7593: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:36.055 INFO:tasks.rados.rados.0.smithi107.stdout:7597: done (2 left) 2020-06-12T02:38:36.055 INFO:tasks.rados.rados.0.smithi107.stdout:7598: delete oid 14 current snap is 0 2020-06-12T02:38:36.055 INFO:tasks.rados.rados.0.smithi107.stdout:7593: finishing write tid 6 to smithi10729407-16 2020-06-12T02:38:36.056 INFO:tasks.rados.rados.0.smithi107.stdout:7595: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:36.056 INFO:tasks.rados.rados.0.smithi107.stdout:7595: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:36.056 INFO:tasks.rados.rados.0.smithi107.stdout:7593: finishing write tid 7 to smithi10729407-16 2020-06-12T02:38:36.056 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1385 (ObjNum 7592 snap 0 seq_num 7592) dirty exists 2020-06-12T02:38:36.056 INFO:tasks.rados.rados.0.smithi107.stdout:7593: left oid 16 (ObjNum 7592 snap 0 seq_num 7592) 2020-06-12T02:38:36.057 INFO:tasks.rados.rados.0.smithi107.stdout:7595: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:36.057 INFO:tasks.rados.rados.0.smithi107.stdout:7595: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:36.058 INFO:tasks.rados.rados.0.smithi107.stdout:7593: done (2 left) 2020-06-12T02:38:36.058 INFO:tasks.rados.rados.0.smithi107.stdout:7598: done (1 left) 2020-06-12T02:38:36.058 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:36.058 INFO:tasks.rados.rados.0.smithi107.stdout:7599: seq_num 7598 ranges {0=57344} 2020-06-12T02:38:36.059 INFO:tasks.rados.rados.0.smithi107.stdout:7599: writing smithi10729407-4 from 0 to 57344 tid 1 2020-06-12T02:38:36.059 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:38:36.059 INFO:tasks.rados.rados.0.smithi107.stdout:7600: seq_num 7599 ranges {2260992=73728} 2020-06-12T02:38:36.071 INFO:tasks.rados.rados.0.smithi107.stdout:7600: writing smithi10729407-13 from 2260992 to 2334720 tid 1 2020-06-12T02:38:36.071 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:38:36.072 INFO:tasks.rados.rados.0.smithi107.stdout:7601: seq_num 7600 ranges {0=90112} 2020-06-12T02:38:36.072 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:38:36.073 INFO:tasks.rados.rados.0.smithi107.stdout:7601: writing smithi10729407-7 from 0 to 90112 tid 1 2020-06-12T02:38:36.073 INFO:tasks.rados.rados.0.smithi107.stdout:7602: delete oid 18 current snap is 0 2020-06-12T02:38:36.073 INFO:tasks.rados.rados.0.smithi107.stdout:7595: finishing write tid 7 to smithi10729407-9 2020-06-12T02:38:36.073 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1430 (ObjNum 7594 snap 0 seq_num 7594) dirty exists 2020-06-12T02:38:36.074 INFO:tasks.rados.rados.0.smithi107.stdout:7595: left oid 9 (ObjNum 7594 snap 0 seq_num 7594) 2020-06-12T02:38:36.074 INFO:tasks.rados.rados.0.smithi107.stdout:7599: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:36.074 INFO:tasks.rados.rados.0.smithi107.stdout:7599: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:36.074 INFO:tasks.rados.rados.0.smithi107.stdout:7599: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:36.074 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1433 (ObjNum 7598 snap 0 seq_num 7598) dirty exists 2020-06-12T02:38:36.074 INFO:tasks.rados.rados.0.smithi107.stdout:7599: left oid 4 (ObjNum 7598 snap 0 seq_num 7598) 2020-06-12T02:38:36.075 INFO:tasks.rados.rados.0.smithi107.stdout:7600: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:36.077 INFO:tasks.rados.rados.0.smithi107.stdout:7595: done (4 left) 2020-06-12T02:38:36.077 INFO:tasks.rados.rados.0.smithi107.stdout:7599: done (3 left) 2020-06-12T02:38:36.077 INFO:tasks.rados.rados.0.smithi107.stdout:7602: done (2 left) 2020-06-12T02:38:36.078 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:36.078 INFO:tasks.rados.rados.0.smithi107.stdout:7603: seq_num 7602 ranges {0=122880} 2020-06-12T02:38:36.078 INFO:tasks.rados.rados.0.smithi107.stdout:7603: writing smithi10729407-17 from 0 to 122880 tid 1 2020-06-12T02:38:36.079 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:38:36.079 INFO:tasks.rados.rados.0.smithi107.stdout:7604: seq_num 7603 ranges {1589248=139264} 2020-06-12T02:38:36.088 INFO:tasks.rados.rados.0.smithi107.stdout:7604: writing smithi10729407-6 from 1589248 to 1728512 tid 1 2020-06-12T02:38:36.088 INFO:tasks.rados.rados.0.smithi107.stdout:7605: delete oid 16 current snap is 0 2020-06-12T02:38:36.088 INFO:tasks.rados.rados.0.smithi107.stdout:7600: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:36.088 INFO:tasks.rados.rados.0.smithi107.stdout:7600: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:36.089 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1394 (ObjNum 7599 snap 0 seq_num 7599) dirty exists 2020-06-12T02:38:36.089 INFO:tasks.rados.rados.0.smithi107.stdout:7600: left oid 13 (ObjNum 7599 snap 0 seq_num 7599) 2020-06-12T02:38:36.089 INFO:tasks.rados.rados.0.smithi107.stdout:7601: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:36.089 INFO:tasks.rados.rados.0.smithi107.stdout:7603: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:36.089 INFO:tasks.rados.rados.0.smithi107.stdout:7603: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:36.089 INFO:tasks.rados.rados.0.smithi107.stdout:7601: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:36.090 INFO:tasks.rados.rados.0.smithi107.stdout:7603: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:36.090 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1435 (ObjNum 7602 snap 0 seq_num 7602) dirty exists 2020-06-12T02:38:36.090 INFO:tasks.rados.rados.0.smithi107.stdout:7603: left oid 17 (ObjNum 7602 snap 0 seq_num 7602) 2020-06-12T02:38:36.090 INFO:tasks.rados.rados.0.smithi107.stdout:7601: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:36.090 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1396 (ObjNum 7600 snap 0 seq_num 7600) dirty exists 2020-06-12T02:38:36.091 INFO:tasks.rados.rados.0.smithi107.stdout:7601: left oid 7 (ObjNum 7600 snap 0 seq_num 7600) 2020-06-12T02:38:36.093 INFO:tasks.rados.rados.0.smithi107.stdout:7604: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:36.093 INFO:tasks.rados.rados.0.smithi107.stdout:7600: done (4 left) 2020-06-12T02:38:36.093 INFO:tasks.rados.rados.0.smithi107.stdout:7601: done (3 left) 2020-06-12T02:38:36.094 INFO:tasks.rados.rados.0.smithi107.stdout:7603: done (2 left) 2020-06-12T02:38:36.094 INFO:tasks.rados.rados.0.smithi107.stdout:7605: done (1 left) 2020-06-12T02:38:36.094 INFO:tasks.rados.rados.0.smithi107.stdout:7606: delete oid 8 current snap is 0 2020-06-12T02:38:36.097 INFO:tasks.rados.rados.0.smithi107.stdout:7606: done (1 left) 2020-06-12T02:38:36.097 INFO:tasks.rados.rados.0.smithi107.stdout:7607: delete oid 7 current snap is 0 2020-06-12T02:38:36.098 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:38:36.098 INFO:tasks.rados.rados.0.smithi107.stdout:7607: done (1 left) 2020-06-12T02:38:36.099 INFO:tasks.rados.rados.0.smithi107.stdout:7608: delete oid 13 current snap is 0 2020-06-12T02:38:36.099 INFO:tasks.rados.rados.0.smithi107.stdout:7604: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:36.101 INFO:tasks.rados.rados.0.smithi107.stdout:7604: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:36.101 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1369 (ObjNum 7603 snap 0 seq_num 7603) dirty exists 2020-06-12T02:38:36.101 INFO:tasks.rados.rados.0.smithi107.stdout:7604: left oid 6 (ObjNum 7603 snap 0 seq_num 7603) 2020-06-12T02:38:36.107 INFO:tasks.rados.rados.0.smithi107.stdout:7604: done (1 left) 2020-06-12T02:38:36.107 INFO:tasks.rados.rados.0.smithi107.stdout:7608: done (0 left) 2020-06-12T02:38:36.107 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:36.107 INFO:tasks.rados.rados.0.smithi107.stdout:7609: seq_num 7608 ranges {0=221184} 2020-06-12T02:38:36.108 INFO:tasks.rados.rados.0.smithi107.stdout:7609: writing smithi10729407-13 from 0 to 221184 tid 1 2020-06-12T02:38:36.108 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:38:36.108 INFO:tasks.rados.rados.0.smithi107.stdout:7610: seq_num 7609 ranges {0=237568} 2020-06-12T02:38:36.109 INFO:tasks.rados.rados.0.smithi107.stdout:7610: writing smithi10729407-10 from 0 to 237568 tid 1 2020-06-12T02:38:36.110 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:36.110 INFO:tasks.rados.rados.0.smithi107.stdout:7611: seq_num 7610 ranges {0=253952} 2020-06-12T02:38:36.112 INFO:tasks.rados.rados.0.smithi107.stdout:7611: writing smithi10729407-3 from 0 to 253952 tid 1 2020-06-12T02:38:36.113 INFO:tasks.rados.rados.0.smithi107.stdout:7612: delete oid 7 current snap is 0 2020-06-12T02:38:36.113 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:36.110+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:36.114 INFO:tasks.rados.rados.0.smithi107.stdout:7609: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:36.117 INFO:tasks.rados.rados.0.smithi107.stdout:7610: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:36.118 INFO:tasks.rados.rados.0.smithi107.stdout:7610: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:36.118 INFO:tasks.rados.rados.0.smithi107.stdout:7612: done (3 left) 2020-06-12T02:38:36.119 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:38:36.119 INFO:tasks.rados.rados.0.smithi107.stdout:7613: seq_num 7612 ranges {0=294912} 2020-06-12T02:38:36.121 INFO:tasks.rados.rados.0.smithi107.stdout:7613: writing smithi10729407-1 from 0 to 294912 tid 1 2020-06-12T02:38:36.121 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:36.121 INFO:tasks.rados.rados.0.smithi107.stdout:7614: seq_num 7613 ranges {0=311296} 2020-06-12T02:38:36.123 INFO:tasks.rados.rados.0.smithi107.stdout:7614: writing smithi10729407-14 from 0 to 311296 tid 1 2020-06-12T02:38:36.123 INFO:tasks.rados.rados.0.smithi107.stdout:7610: finishing write tid 4 to smithi10729407-10 2020-06-12T02:38:36.123 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 514 (ObjNum 7609 snap 0 seq_num 7609) dirty exists 2020-06-12T02:38:36.124 INFO:tasks.rados.rados.0.smithi107.stdout:7610: left oid 10 (ObjNum 7609 snap 0 seq_num 7609) 2020-06-12T02:38:36.124 INFO:tasks.rados.rados.0.smithi107.stdout:7609: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:36.124 INFO:tasks.rados.rados.0.smithi107.stdout:7611: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:36.124 INFO:tasks.rados.rados.0.smithi107.stdout:7609: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:36.124 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1401 (ObjNum 7608 snap 0 seq_num 7608) dirty exists 2020-06-12T02:38:36.124 INFO:tasks.rados.rados.0.smithi107.stdout:7609: left oid 13 (ObjNum 7608 snap 0 seq_num 7608) 2020-06-12T02:38:36.125 INFO:tasks.rados.rados.0.smithi107.stdout:7609: done (4 left) 2020-06-12T02:38:36.125 INFO:tasks.rados.rados.0.smithi107.stdout:7610: done (3 left) 2020-06-12T02:38:36.125 INFO:tasks.rados.rados.0.smithi107.stdout:7615: delete oid 15 current snap is 0 2020-06-12T02:38:36.128 INFO:tasks.rados.rados.0.smithi107.stdout:7615: done (3 left) 2020-06-12T02:38:36.128 INFO:tasks.rados.rados.0.smithi107.stdout:7616: delete oid 10 current snap is 0 2020-06-12T02:38:36.128 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2020-06-12T02:38:36.129 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:36.129 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2020-06-12T02:38:36.129 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.129 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.bs1 60/21 ECSubWrite(tid=8522, reqid=client.4512.0:16260, at_version=60'1388, trim_to=0'0, roll_forward_to=60'1387))", 2020-06-12T02:38:36.129 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:36.120610+0000", 2020-06-12T02:38:36.130 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.005640445, 2020-06-12T02:38:36.130 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0056691830000000004, 2020-06-12T02:38:36.130 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:36.130 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:36.130 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:36.131 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.131 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:36.131 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.120610+0000", 2020-06-12T02:38:36.131 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:36.131 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.131 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.132 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:36.132 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.120610+0000", 2020-06-12T02:38:36.132 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9927999999999999e-05 2020-06-12T02:38:36.133 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.133 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.133 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:36.133 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.120670+0000", 2020-06-12T02:38:36.134 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8380999999999999e-05 2020-06-12T02:38:36.134 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.134 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.134 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:36.134 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.120688+0000", 2020-06-12T02:38:36.134 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5094e-05 2020-06-12T02:38:36.135 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.135 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.135 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:36.135 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.120703+0000", 2020-06-12T02:38:36.135 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1229999999999997e-06 2020-06-12T02:38:36.136 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.136 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.136 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:36.136 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.120709+0000", 2020-06-12T02:38:36.136 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044584560000000004 2020-06-12T02:38:36.136 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.137 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.137 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:36.137 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.125168+0000", 2020-06-12T02:38:36.137 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2588e-05 2020-06-12T02:38:36.137 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.138 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.138 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:36.138 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.125181+0000", 2020-06-12T02:38:36.138 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045705700000000004 2020-06-12T02:38:36.138 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.138 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:36.139 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.139 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.139 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2020-06-12T02:38:36.139 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 1 2020-06-12T02:38:36.139 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:36.141 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-06-12T02:38:36.141 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-06-12T02:38:36.141 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-06-12T02:38:36.141 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-06-12T02:38:36.142 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-06-12T02:38:36.142 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-06-12T02:38:36.142 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-06-12T02:38:36.142 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-06-12T02:38:36.143 INFO:tasks.rados.rados.0.smithi107.stdout:7616: done (3 left) 2020-06-12T02:38:36.143 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:36.143 INFO:tasks.rados.rados.0.smithi107.stdout:7617: seq_num 7616 ranges {57344=360448} 2020-06-12T02:38:36.144 INFO:tasks.rados.rados.0.smithi107.stdout:7617: writing smithi10729407-4 from 57344 to 417792 tid 1 2020-06-12T02:38:36.144 INFO:tasks.rados.rados.0.smithi107.stdout:7618: delete oid 10 current snap is 0 2020-06-12T02:38:36.144 INFO:tasks.rados.rados.0.smithi107.stdout:7613: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:36.144 INFO:tasks.rados.rados.0.smithi107.stdout:7614: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:36.144 INFO:tasks.rados.rados.0.smithi107.stdout:7613: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:36.144 INFO:tasks.rados.rados.0.smithi107.stdout:7613: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:36.145 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1371 (ObjNum 7612 snap 0 seq_num 7612) dirty exists 2020-06-12T02:38:36.145 INFO:tasks.rados.rados.0.smithi107.stdout:7613: left oid 1 (ObjNum 7612 snap 0 seq_num 7612) 2020-06-12T02:38:36.145 INFO:tasks.rados.rados.0.smithi107.stdout:7613: done (4 left) 2020-06-12T02:38:36.145 INFO:tasks.rados.rados.0.smithi107.stdout:7618: done (3 left) 2020-06-12T02:38:36.145 INFO:tasks.rados.rados.0.smithi107.stdout:7619: delete oid 20 current snap is 0 2020-06-12T02:38:36.146 INFO:tasks.rados.rados.0.smithi107.stdout:7617: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:36.146 INFO:tasks.rados.rados.0.smithi107.stdout:7611: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:36.146 INFO:tasks.rados.rados.0.smithi107.stdout:7611: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:36.146 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1388 (ObjNum 7610 snap 0 seq_num 7610) dirty exists 2020-06-12T02:38:36.146 INFO:tasks.rados.rados.0.smithi107.stdout:7611: left oid 3 (ObjNum 7610 snap 0 seq_num 7610) 2020-06-12T02:38:36.146 INFO:tasks.rados.rados.0.smithi107.stdout:7614: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:36.148 INFO:tasks.rados.rados.0.smithi107.stdout:7617: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:36.149 INFO:tasks.rados.rados.0.smithi107.stdout:7611: done (3 left) 2020-06-12T02:38:36.149 INFO:tasks.rados.rados.0.smithi107.stdout:7619: done (2 left) 2020-06-12T02:38:36.149 INFO:tasks.rados.rados.0.smithi107.stdout:7620: delete oid 15 current snap is 0 2020-06-12T02:38:36.149 INFO:tasks.rados.rados.0.smithi107.stdout:7614: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:36.149 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1350 (ObjNum 7613 snap 0 seq_num 7613) dirty exists 2020-06-12T02:38:36.150 INFO:tasks.rados.rados.0.smithi107.stdout:7614: left oid 14 (ObjNum 7613 snap 0 seq_num 7613) 2020-06-12T02:38:36.150 INFO:tasks.rados.rados.0.smithi107.stdout:7617: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:36.150 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1437 (ObjNum 7616 snap 0 seq_num 7616) dirty exists 2020-06-12T02:38:36.150 INFO:tasks.rados.rados.0.smithi107.stdout:7617: left oid 4 (ObjNum 7616 snap 0 seq_num 7616) 2020-06-12T02:38:36.150 INFO:tasks.rados.rados.0.smithi107.stdout:7614: done (2 left) 2020-06-12T02:38:36.151 INFO:tasks.rados.rados.0.smithi107.stdout:7617: done (1 left) 2020-06-12T02:38:36.151 INFO:tasks.rados.rados.0.smithi107.stdout:7620: done (0 left) 2020-06-12T02:38:36.151 INFO:tasks.rados.rados.0.smithi107.stdout:7621: delete oid 3 current snap is 0 2020-06-12T02:38:36.151 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-06-12T02:38:36.153 INFO:tasks.rados.rados.0.smithi107.stdout:7621: done (0 left) 2020-06-12T02:38:36.154 INFO:tasks.rados.rados.0.smithi107.stdout:7622: delete oid 7 current snap is 0 2020-06-12T02:38:36.155 INFO:tasks.rados.rados.0.smithi107.stdout:7622: done (0 left) 2020-06-12T02:38:36.155 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:36.155 INFO:tasks.rados.rados.0.smithi107.stdout:7623: seq_num 7622 ranges {417792=458752} 2020-06-12T02:38:36.156 INFO:tasks.rados.rados.0.smithi107.stdout:7623: writing smithi10729407-4 from 417792 to 876544 tid 1 2020-06-12T02:38:36.156 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:38:36.156 INFO:tasks.rados.rados.0.smithi107.stdout:7624: seq_num 7623 ranges {0=475136} 2020-06-12T02:38:36.158 INFO:tasks.rados.rados.0.smithi107.stdout:7624: writing smithi10729407-10 from 0 to 475136 tid 1 2020-06-12T02:38:36.158 INFO:tasks.rados.rados.0.smithi107.stdout:7625: delete oid 11 current snap is 0 2020-06-12T02:38:36.163 INFO:tasks.rados.rados.0.smithi107.stdout:7623: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:36.164 INFO:tasks.rados.rados.0.smithi107.stdout:7623: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:36.164 INFO:tasks.rados.rados.0.smithi107.stdout:7625: done (2 left) 2020-06-12T02:38:36.165 INFO:tasks.rados.rados.0.smithi107.stdout:7626: delete oid 5 current snap is 0 2020-06-12T02:38:36.165 INFO:tasks.rados.rados.0.smithi107.stdout:7624: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:36.165 INFO:tasks.rados.rados.0.smithi107.stdout:7623: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:36.166 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1439 (ObjNum 7622 snap 0 seq_num 7622) dirty exists 2020-06-12T02:38:36.166 INFO:tasks.rados.rados.0.smithi107.stdout:7623: left oid 4 (ObjNum 7622 snap 0 seq_num 7622) 2020-06-12T02:38:36.166 INFO:tasks.rados.rados.0.smithi107.stdout:7623: done (2 left) 2020-06-12T02:38:36.167 INFO:tasks.rados.rados.0.smithi107.stdout:7626: done (1 left) 2020-06-12T02:38:36.167 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:36.167 INFO:tasks.rados.rados.0.smithi107.stdout:7627: seq_num 7626 ranges {221184=524288} 2020-06-12T02:38:36.171 INFO:tasks.rados.rados.0.smithi107.stdout:7627: writing smithi10729407-13 from 221184 to 745472 tid 1 2020-06-12T02:38:36.171 INFO:tasks.rados.rados.0.smithi107.stdout:7628: delete oid 6 current snap is 0 2020-06-12T02:38:36.172 INFO:tasks.rados.rados.0.smithi107.stdout:7624: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:36.172 INFO:tasks.rados.rados.0.smithi107.stdout:7624: finishing write tid 4 to smithi10729407-10 2020-06-12T02:38:36.172 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 518 (ObjNum 7623 snap 0 seq_num 7623) dirty exists 2020-06-12T02:38:36.172 INFO:tasks.rados.rados.0.smithi107.stdout:7624: left oid 10 (ObjNum 7623 snap 0 seq_num 7623) 2020-06-12T02:38:36.174 INFO:tasks.rados.rados.0.smithi107.stdout:7624: done (2 left) 2020-06-12T02:38:36.174 INFO:tasks.rados.rados.0.smithi107.stdout:7628: done (1 left) 2020-06-12T02:38:36.175 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:36.175 INFO:tasks.rados.rados.0.smithi107.stdout:7629: seq_num 7628 ranges {0=557056} 2020-06-12T02:38:36.177 INFO:tasks.rados.rados.0.smithi107.stdout:7629: writing smithi10729407-8 from 0 to 557056 tid 1 2020-06-12T02:38:36.178 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:36.178 INFO:tasks.rados.rados.0.smithi107.stdout:7630: seq_num 7629 ranges {294912=573440} 2020-06-12T02:38:36.182 INFO:tasks.rados.rados.0.smithi107.stdout:7630: writing smithi10729407-1 from 294912 to 868352 tid 1 2020-06-12T02:38:36.183 INFO:tasks.rados.rados.0.smithi107.stdout:7631: delete oid 3 current snap is 0 2020-06-12T02:38:36.183 INFO:tasks.rados.rados.0.smithi107.stdout:7627: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:36.184 INFO:tasks.rados.rados.0.smithi107.stdout:7627: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:36.184 INFO:tasks.rados.rados.0.smithi107.stdout:7629: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:36.186 INFO:tasks.rados.rados.0.smithi107.stdout:7631: done (3 left) 2020-06-12T02:38:36.187 INFO:tasks.rados.rados.0.smithi107.stdout:7632: delete oid 3 current snap is 0 2020-06-12T02:38:36.187 INFO:tasks.rados.rados.0.smithi107.stdout:7627: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:36.187 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1404 (ObjNum 7626 snap 0 seq_num 7626) dirty exists 2020-06-12T02:38:36.188 INFO:tasks.rados.rados.0.smithi107.stdout:7627: left oid 13 (ObjNum 7626 snap 0 seq_num 7626) 2020-06-12T02:38:36.189 INFO:tasks.rados.rados.0.smithi107.stdout:7629: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:36.190 INFO:tasks.rados.rados.0.smithi107.stdout:7627: done (3 left) 2020-06-12T02:38:36.190 INFO:tasks.rados.rados.0.smithi107.stdout:7632: done (2 left) 2020-06-12T02:38:36.190 INFO:tasks.rados.rados.0.smithi107.stdout:7633: delete oid 20 current snap is 0 2020-06-12T02:38:36.193 INFO:tasks.rados.rados.0.smithi107.stdout:7629: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:36.193 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1423 (ObjNum 7628 snap 0 seq_num 7628) dirty exists 2020-06-12T02:38:36.193 INFO:tasks.rados.rados.0.smithi107.stdout:7629: left oid 8 (ObjNum 7628 snap 0 seq_num 7628) 2020-06-12T02:38:36.193 INFO:tasks.rados.rados.0.smithi107.stdout:7630: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:36.194 INFO:tasks.rados.rados.0.smithi107.stdout:7629: done (2 left) 2020-06-12T02:38:36.194 INFO:tasks.rados.rados.0.smithi107.stdout:7633: done (1 left) 2020-06-12T02:38:36.194 INFO:tasks.rados.rados.0.smithi107.stdout:7634: delete oid 5 current snap is 0 2020-06-12T02:38:36.197 INFO:tasks.rados.rados.0.smithi107.stdout:7634: done (1 left) 2020-06-12T02:38:36.197 INFO:tasks.rados.rados.0.smithi107.stdout:7635: delete oid 2 current snap is 0 2020-06-12T02:38:36.198 INFO:tasks.rados.rados.0.smithi107.stdout:7630: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:36.199 INFO:tasks.rados.rados.0.smithi107.stdout:7630: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:36.199 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1374 (ObjNum 7629 snap 0 seq_num 7629) dirty exists 2020-06-12T02:38:36.199 INFO:tasks.rados.rados.0.smithi107.stdout:7630: left oid 1 (ObjNum 7629 snap 0 seq_num 7629) 2020-06-12T02:38:36.200 INFO:tasks.rados.rados.0.smithi107.stdout:7630: done (1 left) 2020-06-12T02:38:36.200 INFO:tasks.rados.rados.0.smithi107.stdout:7635: done (0 left) 2020-06-12T02:38:36.201 INFO:tasks.rados.rados.0.smithi107.stdout:7636: delete oid 1 current snap is 0 2020-06-12T02:38:36.202 INFO:tasks.rados.rados.0.smithi107.stdout:7636: done (0 left) 2020-06-12T02:38:36.203 INFO:tasks.rados.rados.0.smithi107.stdout:7637: delete oid 10 current snap is 0 2020-06-12T02:38:36.205 INFO:tasks.rados.rados.0.smithi107.stdout:7637: done (0 left) 2020-06-12T02:38:36.205 INFO:tasks.rados.rados.0.smithi107.stdout:7638: delete oid 4 current snap is 0 2020-06-12T02:38:36.207 INFO:tasks.rados.rados.0.smithi107.stdout:7638: done (0 left) 2020-06-12T02:38:36.207 INFO:tasks.rados.rados.0.smithi107.stdout:7639: delete oid 12 current snap is 0 2020-06-12T02:38:36.211 INFO:tasks.rados.rados.0.smithi107.stdout:7639: done (0 left) 2020-06-12T02:38:36.212 INFO:tasks.rados.rados.0.smithi107.stdout:7640: delete oid 5 current snap is 0 2020-06-12T02:38:36.214 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:36.210+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:36.214 INFO:tasks.rados.rados.0.smithi107.stdout:7640: done (0 left) 2020-06-12T02:38:36.214 INFO:tasks.rados.rados.0.smithi107.stdout:7641: delete oid 2 current snap is 0 2020-06-12T02:38:36.217 INFO:tasks.rados.rados.0.smithi107.stdout:7641: done (0 left) 2020-06-12T02:38:36.217 INFO:tasks.rados.rados.0.smithi107.stdout:7642: delete oid 6 current snap is 0 2020-06-12T02:38:36.219 INFO:tasks.rados.rados.0.smithi107.stdout:7642: done (0 left) 2020-06-12T02:38:36.220 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:36.220 INFO:tasks.rados.rados.0.smithi107.stdout:7643: seq_num 7642 ranges {0=794624} 2020-06-12T02:38:36.224 INFO:tasks.rados.rados.0.smithi107.stdout:7643: writing smithi10729407-15 from 0 to 794624 tid 1 2020-06-12T02:38:36.224 INFO:tasks.rados.rados.0.smithi107.stdout:7644: delete oid 18 current snap is 0 2020-06-12T02:38:36.226 INFO:tasks.rados.rados.0.smithi107.stdout:7644: done (1 left) 2020-06-12T02:38:36.227 INFO:tasks.rados.rados.0.smithi107.stdout:7645: delete oid 11 current snap is 0 2020-06-12T02:38:36.229 INFO:tasks.rados.rados.0.smithi107.stdout:7645: done (1 left) 2020-06-12T02:38:36.229 INFO:tasks.rados.rados.0.smithi107.stdout:7646: delete oid 11 current snap is 0 2020-06-12T02:38:36.230 INFO:tasks.rados.rados.0.smithi107.stdout:7646: done (1 left) 2020-06-12T02:38:36.231 INFO:tasks.rados.rados.0.smithi107.stdout:7647: delete oid 18 current snap is 0 2020-06-12T02:38:36.233 INFO:tasks.rados.rados.0.smithi107.stdout:7647: done (1 left) 2020-06-12T02:38:36.233 INFO:tasks.rados.rados.0.smithi107.stdout:7648: delete oid 17 current snap is 0 2020-06-12T02:38:36.235 INFO:tasks.rados.rados.0.smithi107.stdout:7643: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:36.236 INFO:tasks.rados.rados.0.smithi107.stdout:7643: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:36.236 INFO:tasks.rados.rados.0.smithi107.stdout:7648: done (1 left) 2020-06-12T02:38:36.236 INFO:tasks.rados.rados.0.smithi107.stdout:7649: delete oid 19 current snap is 0 2020-06-12T02:38:36.238 INFO:tasks.rados.rados.0.smithi107.stdout:7643: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:36.238 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1425 (ObjNum 7642 snap 0 seq_num 7642) dirty exists 2020-06-12T02:38:36.239 INFO:tasks.rados.rados.0.smithi107.stdout:7643: left oid 15 (ObjNum 7642 snap 0 seq_num 7642) 2020-06-12T02:38:36.242 INFO:tasks.rados.rados.0.smithi107.stdout:7643: done (1 left) 2020-06-12T02:38:36.242 INFO:tasks.rados.rados.0.smithi107.stdout:7649: done (0 left) 2020-06-12T02:38:36.242 INFO:tasks.rados.rados.0.smithi107.stdout:7650: delete oid 15 current snap is 0 2020-06-12T02:38:36.247 INFO:tasks.rados.rados.0.smithi107.stdout:7650: done (0 left) 2020-06-12T02:38:36.247 INFO:tasks.rados.rados.0.smithi107.stdout:7651: delete oid 8 current snap is 0 2020-06-12T02:38:36.252 INFO:tasks.rados.rados.0.smithi107.stdout:7651: done (0 left) 2020-06-12T02:38:36.252 INFO:tasks.rados.rados.0.smithi107.stdout:7652: delete oid 5 current snap is 0 2020-06-12T02:38:36.257 INFO:tasks.rados.rados.0.smithi107.stdout:7652: done (0 left) 2020-06-12T02:38:36.257 INFO:tasks.rados.rados.0.smithi107.stdout:7653: delete oid 8 current snap is 0 2020-06-12T02:38:36.260 INFO:tasks.rados.rados.0.smithi107.stdout:7653: done (0 left) 2020-06-12T02:38:36.260 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:36.260 INFO:tasks.rados.rados.0.smithi107.stdout:7654: seq_num 7653 ranges {2400256=581632,2981888=401408} 2020-06-12T02:38:36.275 INFO:tasks.rados.rados.0.smithi107.stdout:7654: writing smithi10729407-9 from 2400256 to 2981888 tid 1 2020-06-12T02:38:36.278 INFO:tasks.rados.rados.0.smithi107.stdout:7654: writing smithi10729407-9 from 2981888 to 3383296 tid 2 2020-06-12T02:38:36.278 INFO:tasks.rados.rados.0.smithi107.stdout:7655: delete oid 3 current snap is 0 2020-06-12T02:38:36.281 INFO:tasks.rados.rados.0.smithi107.stdout:7655: done (1 left) 2020-06-12T02:38:36.282 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:36.282 INFO:tasks.rados.rados.0.smithi107.stdout:7656: seq_num 7655 ranges {0=614400,614400=401408} 2020-06-12T02:38:36.285 INFO:tasks.rados.rados.0.smithi107.stdout:7656: writing smithi10729407-16 from 0 to 614400 tid 1 2020-06-12T02:38:36.288 INFO:tasks.rados.rados.0.smithi107.stdout:7656: writing smithi10729407-16 from 614400 to 1015808 tid 2 2020-06-12T02:38:36.288 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:36.288 INFO:tasks.rados.rados.0.smithi107.stdout:7657: seq_num 7656 ranges {745472=630784,1376256=401408} 2020-06-12T02:38:36.293 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:36.293 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:38:36.293 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2020-06-12T02:38:36.294 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2020-06-12T02:38:36.294 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:36.300 INFO:tasks.rados.rados.0.smithi107.stdout:7657: writing smithi10729407-13 from 745472 to 1376256 tid 1 2020-06-12T02:38:36.300 INFO:tasks.rados.rados.0.smithi107.stdout:7657: writing smithi10729407-13 from 1376256 to 1777664 tid 2 2020-06-12T02:38:36.301 INFO:tasks.rados.rados.0.smithi107.stdout:7658: delete oid 1 current snap is 0 2020-06-12T02:38:36.301 INFO:tasks.rados.rados.0.smithi107.stdout:7654: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:36.301 INFO:tasks.rados.rados.0.smithi107.stdout:7654: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:36.301 INFO:tasks.rados.rados.0.smithi107.stdout:7656: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:36.302 INFO:tasks.rados.rados.0.smithi107.stdout:7654: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:36.302 INFO:tasks.rados.rados.0.smithi107.stdout:7654: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:36.302 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1436 (ObjNum 7653 snap 0 seq_num 7653) dirty exists 2020-06-12T02:38:36.302 INFO:tasks.rados.rados.0.smithi107.stdout:7654: left oid 9 (ObjNum 7653 snap 0 seq_num 7653) 2020-06-12T02:38:36.303 INFO:tasks.rados.rados.0.smithi107.stdout:7654: done (3 left) 2020-06-12T02:38:36.303 INFO:tasks.rados.rados.0.smithi107.stdout:7658: done (2 left) 2020-06-12T02:38:36.304 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:36.304 INFO:tasks.rados.rados.0.smithi107.stdout:7659: seq_num 7658 ranges {0=663552,663552=401408} 2020-06-12T02:38:36.305 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-06-12T02:38:36.306 INFO:tasks.rados.rados.0.smithi107.stdout:7659: writing smithi10729407-19 from 0 to 663552 tid 1 2020-06-12T02:38:36.309 INFO:tasks.rados.rados.0.smithi107.stdout:7659: writing smithi10729407-19 from 663552 to 1064960 tid 2 2020-06-12T02:38:36.310 INFO:tasks.rados.rados.0.smithi107.stdout:7660: delete oid 20 current snap is 0 2020-06-12T02:38:36.310 INFO:tasks.rados.rados.0.smithi107.stdout:7656: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:36.310 INFO:tasks.rados.rados.0.smithi107.stdout:7657: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:36.310 INFO:tasks.rados.rados.0.smithi107.stdout:7657: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:36.311 INFO:tasks.rados.rados.0.smithi107.stdout:7656: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:36.311 INFO:tasks.rados.rados.0.smithi107.stdout:7657: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:36.311 INFO:tasks.rados.rados.0.smithi107.stdout:7657: finishing write tid 5 to smithi10729407-13 2020-06-12T02:38:36.311 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1407 (ObjNum 7656 snap 0 seq_num 7656) dirty exists 2020-06-12T02:38:36.312 INFO:tasks.rados.rados.0.smithi107.stdout:7657: left oid 13 (ObjNum 7656 snap 0 seq_num 7656) 2020-06-12T02:38:36.312 INFO:tasks.rados.rados.0.smithi107.stdout:7657: done (3 left) 2020-06-12T02:38:36.313 INFO:tasks.rados.rados.0.smithi107.stdout:7660: done (2 left) 2020-06-12T02:38:36.313 INFO:tasks.rados.rados.0.smithi107.stdout:7661: delete oid 10 current snap is 0 2020-06-12T02:38:36.313 INFO:tasks.rados.rados.0.smithi107.stdout:7656: finishing write tid 5 to smithi10729407-16 2020-06-12T02:38:36.313 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1395 (ObjNum 7655 snap 0 seq_num 7655) dirty exists 2020-06-12T02:38:36.313 INFO:tasks.rados.rados.0.smithi107.stdout:7656: left oid 16 (ObjNum 7655 snap 0 seq_num 7655) 2020-06-12T02:38:36.314 INFO:tasks.rados.rados.0.smithi107.stdout:7659: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:36.315 INFO:tasks.rados.rados.0.smithi107.stdout:7656: done (2 left) 2020-06-12T02:38:36.315 INFO:tasks.rados.rados.0.smithi107.stdout:7661: done (1 left) 2020-06-12T02:38:36.316 INFO:tasks.rados.rados.0.smithi107.stdout:7662: delete oid 14 current snap is 0 2020-06-12T02:38:36.316 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:38:36.312+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:38:36.318 INFO:tasks.rados.rados.0.smithi107.stdout:7659: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:36.318 INFO:tasks.rados.rados.0.smithi107.stdout:7659: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:36.318 INFO:tasks.rados.rados.0.smithi107.stdout:7662: done (1 left) 2020-06-12T02:38:36.318 INFO:tasks.rados.rados.0.smithi107.stdout:7663: delete oid 17 current snap is 0 2020-06-12T02:38:36.320 INFO:tasks.rados.rados.0.smithi107.stdout:7659: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:36.320 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1396 (ObjNum 7658 snap 0 seq_num 7658) dirty exists 2020-06-12T02:38:36.320 INFO:tasks.rados.rados.0.smithi107.stdout:7659: left oid 19 (ObjNum 7658 snap 0 seq_num 7658) 2020-06-12T02:38:36.321 INFO:tasks.rados.rados.0.smithi107.stdout:7659: done (1 left) 2020-06-12T02:38:36.321 INFO:tasks.rados.rados.0.smithi107.stdout:7663: done (0 left) 2020-06-12T02:38:36.321 INFO:tasks.rados.rados.0.smithi107.stdout:7664: delete oid 2 current snap is 0 2020-06-12T02:38:36.323 INFO:tasks.rados.rados.0.smithi107.stdout:7664: done (0 left) 2020-06-12T02:38:36.323 INFO:tasks.rados.rados.0.smithi107.stdout:7665: delete oid 17 current snap is 0 2020-06-12T02:38:36.325 INFO:tasks.rados.rados.0.smithi107.stdout:7665: done (0 left) 2020-06-12T02:38:36.325 INFO:tasks.rados.rados.0.smithi107.stdout:7666: delete oid 9 current snap is 0 2020-06-12T02:38:36.327 INFO:tasks.rados.rados.0.smithi107.stdout:7666: done (0 left) 2020-06-12T02:38:36.328 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:38:36.328 INFO:tasks.rados.rados.0.smithi107.stdout:7667: seq_num 7666 ranges {0=794624,794624=401408} 2020-06-12T02:38:36.332 INFO:tasks.rados.rados.0.smithi107.stdout:7667: writing smithi10729407-7 from 0 to 794624 tid 1 2020-06-12T02:38:36.335 INFO:tasks.rados.rados.0.smithi107.stdout:7667: writing smithi10729407-7 from 794624 to 1196032 tid 2 2020-06-12T02:38:36.335 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:36.336 INFO:tasks.rados.rados.0.smithi107.stdout:7668: seq_num 7667 ranges {0=409600,409600=458752,868352=344064} 2020-06-12T02:38:36.338 INFO:tasks.rados.rados.0.smithi107.stdout:7668: writing smithi10729407-18 from 0 to 409600 tid 1 2020-06-12T02:38:36.341 INFO:tasks.rados.rados.0.smithi107.stdout:7668: writing smithi10729407-18 from 409600 to 868352 tid 2 2020-06-12T02:38:36.343 INFO:tasks.rados.rados.0.smithi107.stdout:7668: writing smithi10729407-18 from 868352 to 1212416 tid 3 2020-06-12T02:38:36.343 INFO:tasks.rados.rados.0.smithi107.stdout:7667: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:36.344 INFO:tasks.rados.rados.0.smithi107.stdout:7669: delete oid 16 current snap is 0 2020-06-12T02:38:36.346 INFO:tasks.rados.rados.0.smithi107.stdout:7667: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:36.347 INFO:tasks.rados.rados.0.smithi107.stdout:7667: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:36.348 INFO:tasks.rados.rados.0.smithi107.stdout:7667: finishing write tid 5 to smithi10729407-7 2020-06-12T02:38:36.349 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1410 (ObjNum 7666 snap 0 seq_num 7666) dirty exists 2020-06-12T02:38:36.349 INFO:tasks.rados.rados.0.smithi107.stdout:7667: left oid 7 (ObjNum 7666 snap 0 seq_num 7666) 2020-06-12T02:38:36.350 INFO:tasks.rados.rados.0.smithi107.stdout:7668: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:36.351 INFO:tasks.rados.rados.0.smithi107.stdout:7667: done (2 left) 2020-06-12T02:38:36.351 INFO:tasks.rados.rados.0.smithi107.stdout:7669: done (1 left) 2020-06-12T02:38:36.351 INFO:tasks.rados.rados.0.smithi107.stdout:7670: delete oid 7 current snap is 0 2020-06-12T02:38:36.352 INFO:tasks.rados.rados.0.smithi107.stdout:7668: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:36.352 INFO:tasks.rados.rados.0.smithi107.stdout:7668: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:36.352 INFO:tasks.rados.rados.0.smithi107.stdout:7668: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:36.353 INFO:tasks.rados.rados.0.smithi107.stdout:7670: done (1 left) 2020-06-12T02:38:36.353 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:36.353 INFO:tasks.rados.rados.0.smithi107.stdout:7671: seq_num 7670 ranges {0=466944,466944=507904,974848=294912} 2020-06-12T02:38:36.356 INFO:tasks.rados.rados.0.smithi107.stdout:7671: writing smithi10729407-6 from 0 to 466944 tid 1 2020-06-12T02:38:36.358 INFO:tasks.rados.rados.0.smithi107.stdout:7671: writing smithi10729407-6 from 466944 to 974848 tid 2 2020-06-12T02:38:36.362 INFO:tasks.rados.rados.0.smithi107.stdout:7671: writing smithi10729407-6 from 974848 to 1269760 tid 3 2020-06-12T02:38:36.362 INFO:tasks.rados.rados.0.smithi107.stdout:7672: delete oid 15 current snap is 0 2020-06-12T02:38:36.362 INFO:tasks.rados.rados.0.smithi107.stdout:7668: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:36.363 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1441 (ObjNum 7667 snap 0 seq_num 7667) dirty exists 2020-06-12T02:38:36.363 INFO:tasks.rados.rados.0.smithi107.stdout:7668: left oid 18 (ObjNum 7667 snap 0 seq_num 7667) 2020-06-12T02:38:36.363 INFO:tasks.rados.rados.0.smithi107.stdout:7671: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:36.364 INFO:tasks.rados.rados.0.smithi107.stdout:7668: done (2 left) 2020-06-12T02:38:36.364 INFO:tasks.rados.rados.0.smithi107.stdout:7672: done (1 left) 2020-06-12T02:38:36.365 INFO:tasks.rados.rados.0.smithi107.stdout:7673: delete oid 7 current snap is 0 2020-06-12T02:38:36.368 INFO:tasks.rados.rados.0.smithi107.stdout:7671: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:36.369 INFO:tasks.rados.rados.0.smithi107.stdout:7673: done (1 left) 2020-06-12T02:38:36.369 INFO:tasks.rados.rados.0.smithi107.stdout:7674: delete oid 9 current snap is 0 2020-06-12T02:38:36.370 INFO:tasks.rados.rados.0.smithi107.stdout:7671: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:36.371 INFO:tasks.rados.rados.0.smithi107.stdout:7671: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:36.372 INFO:tasks.rados.rados.0.smithi107.stdout:7674: done (1 left) 2020-06-12T02:38:36.372 INFO:tasks.rados.rados.0.smithi107.stdout:7675: delete oid 3 current snap is 0 2020-06-12T02:38:36.372 INFO:tasks.rados.rados.0.smithi107.stdout:7671: finishing write tid 6 to smithi10729407-6 2020-06-12T02:38:36.373 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1381 (ObjNum 7670 snap 0 seq_num 7670) dirty exists 2020-06-12T02:38:36.373 INFO:tasks.rados.rados.0.smithi107.stdout:7671: left oid 6 (ObjNum 7670 snap 0 seq_num 7670) 2020-06-12T02:38:36.377 INFO:tasks.rados.rados.0.smithi107.stdout:7671: done (1 left) 2020-06-12T02:38:36.377 INFO:tasks.rados.rados.0.smithi107.stdout:7675: done (0 left) 2020-06-12T02:38:36.377 INFO:tasks.rados.rados.0.smithi107.stdout:7676: delete oid 7 current snap is 0 2020-06-12T02:38:36.385 INFO:tasks.rados.rados.0.smithi107.stdout:7676: done (0 left) 2020-06-12T02:38:36.385 INFO:tasks.rados.rados.0.smithi107.stdout:7677: delete oid 9 current snap is 0 2020-06-12T02:38:36.386 INFO:tasks.rados.rados.0.smithi107.stdout:7677: done (0 left) 2020-06-12T02:38:36.386 INFO:tasks.rados.rados.0.smithi107.stdout:7678: delete oid 16 current snap is 0 2020-06-12T02:38:36.386 INFO:tasks.rados.rados.0.smithi107.stdout:7678: done (0 left) 2020-06-12T02:38:36.387 INFO:tasks.rados.rados.0.smithi107.stdout:7679: delete oid 4 current snap is 0 2020-06-12T02:38:36.389 INFO:tasks.rados.rados.0.smithi107.stdout:7679: done (0 left) 2020-06-12T02:38:36.395 INFO:tasks.rados.rados.0.smithi107.stdout:7680: delete oid 1 current snap is 0 2020-06-12T02:38:36.396 INFO:tasks.rados.rados.0.smithi107.stdout:7680: done (0 left) 2020-06-12T02:38:36.397 INFO:tasks.rados.rados.0.smithi107.stdout:7681: delete oid 7 current snap is 0 2020-06-12T02:38:36.397 INFO:tasks.rados.rados.0.smithi107.stdout:7681: done (0 left) 2020-06-12T02:38:36.397 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:36.398 INFO:tasks.rados.rados.0.smithi107.stdout:7682: seq_num 7681 ranges {0=647168,647168=532480,1179648=270336} 2020-06-12T02:38:36.398 INFO:tasks.rados.rados.0.smithi107.stdout:7682: writing smithi10729407-11 from 0 to 647168 tid 1 2020-06-12T02:38:36.401 INFO:tasks.rados.rados.0.smithi107.stdout:7682: writing smithi10729407-11 from 647168 to 1179648 tid 2 2020-06-12T02:38:36.407 INFO:tasks.rados.rados.0.smithi107.stdout:7682: writing smithi10729407-11 from 1179648 to 1449984 tid 3 2020-06-12T02:38:36.407 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:36.407 INFO:tasks.rados.rados.0.smithi107.stdout:7683: seq_num 7682 ranges {0=663552,663552=417792,1081344=385024} 2020-06-12T02:38:36.408 INFO:tasks.rados.rados.0.smithi107.stdout:7683: writing smithi10729407-12 from 0 to 663552 tid 1 2020-06-12T02:38:36.410 INFO:tasks.rados.rados.0.smithi107.stdout:7683: writing smithi10729407-12 from 663552 to 1081344 tid 2 2020-06-12T02:38:36.413 INFO:tasks.rados.rados.0.smithi107.stdout:7683: writing smithi10729407-12 from 1081344 to 1466368 tid 3 2020-06-12T02:38:36.413 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:38:36.413 INFO:tasks.rados.rados.0.smithi107.stdout:7684: seq_num 7683 ranges {0=679936,679936=704512,1384448=98304} 2020-06-12T02:38:36.415 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:36.413+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:36.417 INFO:tasks.rados.rados.0.smithi107.stdout:7684: writing smithi10729407-5 from 0 to 679936 tid 1 2020-06-12T02:38:36.422 INFO:tasks.rados.rados.0.smithi107.stdout:7684: writing smithi10729407-5 from 679936 to 1384448 tid 2 2020-06-12T02:38:36.423 INFO:tasks.rados.rados.0.smithi107.stdout:7684: writing smithi10729407-5 from 1384448 to 1482752 tid 3 2020-06-12T02:38:36.423 INFO:tasks.rados.rados.0.smithi107.stdout:7685: delete oid 18 current snap is 0 2020-06-12T02:38:36.423 INFO:tasks.rados.rados.0.smithi107.stdout:7682: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:36.423 INFO:tasks.rados.rados.0.smithi107.stdout:7682: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:36.423 INFO:tasks.rados.rados.0.smithi107.stdout:7682: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:36.424 INFO:tasks.rados.rados.0.smithi107.stdout:7682: finishing write tid 5 to smithi10729407-11 2020-06-12T02:38:36.424 INFO:tasks.rados.rados.0.smithi107.stdout:7683: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:36.424 INFO:tasks.rados.rados.0.smithi107.stdout:7683: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:36.424 INFO:tasks.rados.rados.0.smithi107.stdout:7683: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:36.424 INFO:tasks.rados.rados.0.smithi107.stdout:7682: finishing write tid 6 to smithi10729407-11 2020-06-12T02:38:36.424 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1358 (ObjNum 7681 snap 0 seq_num 7681) dirty exists 2020-06-12T02:38:36.425 INFO:tasks.rados.rados.0.smithi107.stdout:7682: left oid 11 (ObjNum 7681 snap 0 seq_num 7681) 2020-06-12T02:38:36.425 INFO:tasks.rados.rados.0.smithi107.stdout:7683: finishing write tid 5 to smithi10729407-12 2020-06-12T02:38:36.426 INFO:tasks.rados.rados.0.smithi107.stdout:7684: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:36.426 INFO:tasks.rados.rados.0.smithi107.stdout:7683: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:36.426 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1400 (ObjNum 7682 snap 0 seq_num 7682) dirty exists 2020-06-12T02:38:36.426 INFO:tasks.rados.rados.0.smithi107.stdout:7683: left oid 12 (ObjNum 7682 snap 0 seq_num 7682) 2020-06-12T02:38:36.426 INFO:tasks.rados.rados.0.smithi107.stdout:7682: done (3 left) 2020-06-12T02:38:36.427 INFO:tasks.rados.rados.0.smithi107.stdout:7683: done (2 left) 2020-06-12T02:38:36.427 INFO:tasks.rados.rados.0.smithi107.stdout:7685: done (1 left) 2020-06-12T02:38:36.427 INFO:tasks.rados.rados.0.smithi107.stdout:7686: delete oid 8 current snap is 0 2020-06-12T02:38:36.432 INFO:tasks.rados.rados.0.smithi107.stdout:7684: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:36.432 INFO:tasks.rados.rados.0.smithi107.stdout:7684: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:36.434 INFO:tasks.rados.rados.0.smithi107.stdout:7686: done (1 left) 2020-06-12T02:38:36.434 INFO:tasks.rados.rados.0.smithi107.stdout:7687: delete oid 16 current snap is 0 2020-06-12T02:38:36.437 INFO:tasks.rados.rados.0.smithi107.stdout:7684: finishing write tid 5 to smithi10729407-5 2020-06-12T02:38:36.437 INFO:tasks.rados.rados.0.smithi107.stdout:7687: done (1 left) 2020-06-12T02:38:36.437 INFO:tasks.rados.rados.0.smithi107.stdout:7688: delete oid 13 current snap is 0 2020-06-12T02:38:36.439 INFO:tasks.rados.rados.0.smithi107.stdout:7684: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:36.439 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 764 (ObjNum 7683 snap 0 seq_num 7683) dirty exists 2020-06-12T02:38:36.439 INFO:tasks.rados.rados.0.smithi107.stdout:7684: left oid 5 (ObjNum 7683 snap 0 seq_num 7683) 2020-06-12T02:38:36.440 INFO:tasks.rados.rados.0.smithi107.stdout:7684: done (1 left) 2020-06-12T02:38:36.440 INFO:tasks.rados.rados.0.smithi107.stdout:7688: done (0 left) 2020-06-12T02:38:36.440 INFO:tasks.rados.rados.0.smithi107.stdout:7689: delete oid 3 current snap is 0 2020-06-12T02:38:36.442 INFO:tasks.rados.rados.0.smithi107.stdout:7689: done (0 left) 2020-06-12T02:38:36.442 INFO:tasks.rados.rados.0.smithi107.stdout:7690: delete oid 2 current snap is 0 2020-06-12T02:38:36.445 INFO:tasks.rados.rados.0.smithi107.stdout:7690: done (0 left) 2020-06-12T02:38:36.446 INFO:tasks.rados.rados.0.smithi107.stdout:7691: delete oid 1 current snap is 0 2020-06-12T02:38:36.449 INFO:tasks.rados.rados.0.smithi107.stdout:7691: done (0 left) 2020-06-12T02:38:36.449 INFO:tasks.rados.rados.0.smithi107.stdout:7692: delete oid 5 current snap is 0 2020-06-12T02:38:36.453 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2020-06-12T02:38:36.456 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:36.456 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2020-06-12T02:38:36.457 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2020-06-12T02:38:36.457 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2020-06-12T02:38:36.457 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.457 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.3s2 25/21 ECSubWrite(tid=588, reqid=client.4512.0:1207, at_version=25'103, trim_to=0'0, roll_forward_to=25'101))", 2020-06-12T02:38:36.457 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:32.861241+0000", 2020-06-12T02:38:36.458 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 63.591346176999998, 2020-06-12T02:38:36.458 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012683748, 2020-06-12T02:38:36.458 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:36.458 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:36.458 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:36.458 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.459 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:36.459 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.861241+0000", 2020-06-12T02:38:36.459 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:36.459 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.459 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.460 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:36.460 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.861241+0000", 2020-06-12T02:38:36.460 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000508951 2020-06-12T02:38:36.460 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.460 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.461 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:36.461 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.861750+0000", 2020-06-12T02:38:36.461 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4369000000000002e-05 2020-06-12T02:38:36.461 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.461 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.461 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:36.462 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.861785+0000", 2020-06-12T02:38:36.462 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.065e-05 2020-06-12T02:38:36.464 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.464 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.464 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:36.464 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.861805+0000", 2020-06-12T02:38:36.465 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3259999999999996e-06 2020-06-12T02:38:36.465 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.465 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.465 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:36.465 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.861814+0000", 2020-06-12T02:38:36.465 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6190999999999998e-05 2020-06-12T02:38:36.466 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.466 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.466 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:36.466 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.861870+0000", 2020-06-12T02:38:36.466 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.222e-05 2020-06-12T02:38:36.467 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.467 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.467 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:36.467 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.861882+0000", 2020-06-12T02:38:36.467 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012007545 2020-06-12T02:38:36.468 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.468 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.468 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:36.468 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.873890+0000", 2020-06-12T02:38:36.468 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5496e-05 2020-06-12T02:38:36.468 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.469 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.469 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:36.469 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:32.873925+0000", 2020-06-12T02:38:36.469 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012683748 2020-06-12T02:38:36.469 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.470 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:36.470 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.470 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.470 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.470 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.7s1 31/21 ECSubWrite(tid=2717, reqid=client.4512.0:6017, at_version=31'497, trim_to=0'0, roll_forward_to=31'496))", 2020-06-12T02:38:36.470 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:46.894385+0000", 2020-06-12T02:38:36.471 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 49.558202264000002, 2020-06-12T02:38:36.471 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011994813, 2020-06-12T02:38:36.471 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:36.471 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:36.471 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:36.472 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.472 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:36.472 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:46.894385+0000", 2020-06-12T02:38:36.472 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:36.472 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.473 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.473 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:36.473 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:46.894385+0000", 2020-06-12T02:38:36.473 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000162886 2020-06-12T02:38:36.474 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.474 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.474 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:36.474 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:46.894548+0000", 2020-06-12T02:38:36.474 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.4870000000000001e-05 2020-06-12T02:38:36.475 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.475 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.475 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:36.475 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:46.894573+0000", 2020-06-12T02:38:36.475 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7784000000000001e-05 2020-06-12T02:38:36.475 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.476 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.476 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:36.476 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:46.894591+0000", 2020-06-12T02:38:36.476 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1179999999999997e-06 2020-06-12T02:38:36.476 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.477 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.477 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:36.477 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:46.894597+0000", 2020-06-12T02:38:36.477 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114572 2020-06-12T02:38:36.477 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.478 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.478 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:36.478 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:46.894712+0000", 2020-06-12T02:38:36.478 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1790000000000001e-05 2020-06-12T02:38:36.478 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.478 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.479 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:36.479 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:46.894723+0000", 2020-06-12T02:38:36.479 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011341054 2020-06-12T02:38:36.479 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.480 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.480 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:36.480 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:46.906064+0000", 2020-06-12T02:38:36.480 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00031573899999999999 2020-06-12T02:38:36.480 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.481 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.481 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:36.481 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:46.906380+0000", 2020-06-12T02:38:36.481 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011994813 2020-06-12T02:38:36.481 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.482 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:36.482 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.482 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.482 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.482 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.3s2 49/21 ECSubWrite(tid=4109, reqid=client.4512.0:8841, at_version=49'773, trim_to=0'0, roll_forward_to=49'771))", 2020-06-12T02:38:36.482 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.822144+0000", 2020-06-12T02:38:36.483 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 24.630443760999999, 2020-06-12T02:38:36.483 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014799214, 2020-06-12T02:38:36.483 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:36.483 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:36.484 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:36.484 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.484 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:36.484 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822144+0000", 2020-06-12T02:38:36.484 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:36.485 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.485 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.485 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:36.485 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822144+0000", 2020-06-12T02:38:36.485 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00061470800000000005 2020-06-12T02:38:36.486 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.486 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.486 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:36.486 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822759+0000", 2020-06-12T02:38:36.487 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.7732000000000001e-05 2020-06-12T02:38:36.487 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.487 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.487 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:36.487 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822786+0000", 2020-06-12T02:38:36.487 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1010999999999999e-05 2020-06-12T02:38:36.488 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.488 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.488 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:36.489 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822807+0000", 2020-06-12T02:38:36.489 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.842e-06 2020-06-12T02:38:36.489 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.489 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.489 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:36.490 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822817+0000", 2020-06-12T02:38:36.490 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00024702300000000002 2020-06-12T02:38:36.490 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.490 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.490 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:36.490 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.823064+0000", 2020-06-12T02:38:36.491 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5207e-05 2020-06-12T02:38:36.491 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.491 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.491 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:36.491 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.823079+0000", 2020-06-12T02:38:36.492 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013806817000000001 2020-06-12T02:38:36.492 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.492 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.492 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:36.492 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.836886+0000", 2020-06-12T02:38:36.492 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6873999999999998e-05 2020-06-12T02:38:36.493 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.493 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.493 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:36.493 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.836943+0000", 2020-06-12T02:38:36.494 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014799214 2020-06-12T02:38:36.495 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.495 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:36.495 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.495 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.496 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.496 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14657 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~606208] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:36.496 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.209225+0000", 2020-06-12T02:38:36.497 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 5.2433625099999999, 2020-06-12T02:38:36.497 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016128905999999998, 2020-06-12T02:38:36.497 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:36.497 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:36.497 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:36.498 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:36.498 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:36.498 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14657 2020-06-12T02:38:36.498 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.498 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:36.499 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.502 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:36.502 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209225+0000", 2020-06-12T02:38:36.503 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:36.503 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.503 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.503 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:36.503 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209225+0000", 2020-06-12T02:38:36.504 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018386699999999999 2020-06-12T02:38:36.504 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.504 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.505 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:36.505 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209409+0000", 2020-06-12T02:38:36.505 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8160000000000001e-06 2020-06-12T02:38:36.505 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.505 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.506 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:36.507 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209414+0000", 2020-06-12T02:38:36.507 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0230000000000004e-06 2020-06-12T02:38:36.507 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.507 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.508 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:36.508 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209421+0000", 2020-06-12T02:38:36.508 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0139999999999999e-06 2020-06-12T02:38:36.508 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.508 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.508 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:36.509 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209424+0000", 2020-06-12T02:38:36.509 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0054798169999999997 2020-06-12T02:38:36.509 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.509 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.509 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:36.510 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.214904+0000", 2020-06-12T02:38:36.510 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123063 2020-06-12T02:38:36.510 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.510 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.510 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:36.511 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215027+0000", 2020-06-12T02:38:36.511 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001159923 2020-06-12T02:38:36.511 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.511 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.511 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:36.512 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.216187+0000", 2020-06-12T02:38:36.512 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2835000000000002e-05 2020-06-12T02:38:36.512 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.512 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.512 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:36.512 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.216270+0000", 2020-06-12T02:38:36.513 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073575799999999998 2020-06-12T02:38:36.513 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.513 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.513 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:36.513 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.217005+0000", 2020-06-12T02:38:36.514 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0082434659999999996 2020-06-12T02:38:36.514 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.514 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.514 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:36.514 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225249+0000", 2020-06-12T02:38:36.515 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0145999999999997e-05 2020-06-12T02:38:36.515 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.516 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.516 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:36.516 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225299+0000", 2020-06-12T02:38:36.516 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5177999999999999e-05 2020-06-12T02:38:36.516 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.516 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.517 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:36.517 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225354+0000", 2020-06-12T02:38:36.517 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016128905999999998 2020-06-12T02:38:36.517 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.517 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:36.518 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.518 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.518 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.518 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14658 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~598016] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:36.518 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.209455+0000", 2020-06-12T02:38:36.519 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 5.2431327559999996, 2020-06-12T02:38:36.519 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016447966000000001, 2020-06-12T02:38:36.519 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:36.519 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:36.519 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:36.519 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:36.520 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:36.520 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14658 2020-06-12T02:38:36.520 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.520 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:36.520 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.521 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:36.521 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209455+0000", 2020-06-12T02:38:36.521 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:36.521 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.521 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.522 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:36.522 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209455+0000", 2020-06-12T02:38:36.522 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037921599999999998 2020-06-12T02:38:36.522 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.523 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.523 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:36.523 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209834+0000", 2020-06-12T02:38:36.523 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8339999999999999e-06 2020-06-12T02:38:36.523 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.524 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.524 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:36.524 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209842+0000", 2020-06-12T02:38:36.524 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2564e-05 2020-06-12T02:38:36.524 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.524 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.525 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:36.525 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209855+0000", 2020-06-12T02:38:36.526 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.135e-06 2020-06-12T02:38:36.526 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.526 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.526 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:36.526 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209860+0000", 2020-06-12T02:38:36.527 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0052530290000000002 2020-06-12T02:38:36.527 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.527 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.527 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:36.527 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215113+0000", 2020-06-12T02:38:36.528 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6229000000000003e-05 2020-06-12T02:38:36.528 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.528 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.528 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:36.528 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215209+0000", 2020-06-12T02:38:36.528 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045765409999999999 2020-06-12T02:38:36.529 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.529 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.529 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:36.529 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.219785+0000", 2020-06-12T02:38:36.529 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000102846 2020-06-12T02:38:36.530 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.530 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.530 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:36.530 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.219888+0000", 2020-06-12T02:38:36.530 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00077088700000000005 2020-06-12T02:38:36.531 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.531 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.531 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:36.531 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.220659+0000", 2020-06-12T02:38:36.531 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046979739999999997 2020-06-12T02:38:36.531 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.532 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.532 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:36.532 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225357+0000", 2020-06-12T02:38:36.532 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00046069600000000002 2020-06-12T02:38:36.532 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.532 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.533 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:36.533 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225818+0000", 2020-06-12T02:38:36.533 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5014999999999998e-05 2020-06-12T02:38:36.533 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.533 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.534 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:36.534 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225903+0000", 2020-06-12T02:38:36.534 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016447966000000001 2020-06-12T02:38:36.534 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.534 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:36.535 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.535 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.535 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.535 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14659 3.0s0 3:0dddb690:::smithi10729407-10:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:36.536 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.209909+0000", 2020-06-12T02:38:36.536 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 5.2426785479999998, 2020-06-12T02:38:36.536 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016618236000000002, 2020-06-12T02:38:36.536 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:36.537 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:36.537 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:36.537 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:36.537 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:36.537 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14659 2020-06-12T02:38:36.538 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.538 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:36.538 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.538 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:36.538 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209909+0000", 2020-06-12T02:38:36.538 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:36.539 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.539 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.539 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:36.539 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209909+0000", 2020-06-12T02:38:36.539 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8085999999999999e-05 2020-06-12T02:38:36.540 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.540 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.540 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:36.540 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209967+0000", 2020-06-12T02:38:36.540 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3169999999999998e-06 2020-06-12T02:38:36.541 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.541 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.541 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:36.541 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209974+0000", 2020-06-12T02:38:36.541 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1246999999999999e-05 2020-06-12T02:38:36.542 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.542 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.542 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:36.542 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209985+0000", 2020-06-12T02:38:36.543 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1439999999999996e-06 2020-06-12T02:38:36.543 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.543 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.543 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:36.543 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209989+0000", 2020-06-12T02:38:36.544 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0052537900000000004 2020-06-12T02:38:36.544 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.544 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.544 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:36.544 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215243+0000", 2020-06-12T02:38:36.544 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6023000000000002e-05 2020-06-12T02:38:36.545 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.545 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.545 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:36.545 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215299+0000", 2020-06-12T02:38:36.546 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0087846910000000007 2020-06-12T02:38:36.546 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.546 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.547 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:36.547 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.224083+0000", 2020-06-12T02:38:36.547 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8504000000000004e-05 2020-06-12T02:38:36.547 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.547 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.548 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:36.548 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.224182+0000", 2020-06-12T02:38:36.548 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044251400000000003 2020-06-12T02:38:36.548 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.548 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.548 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:36.549 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.224624+0000", 2020-06-12T02:38:36.549 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013383290000000001 2020-06-12T02:38:36.549 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.549 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.549 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:36.550 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225963+0000", 2020-06-12T02:38:36.550 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047942699999999999 2020-06-12T02:38:36.550 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.550 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.550 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:36.551 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.226442+0000", 2020-06-12T02:38:36.551 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5164e-05 2020-06-12T02:38:36.551 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.551 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.552 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:36.552 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.226527+0000", 2020-06-12T02:38:36.552 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016618236000000002 2020-06-12T02:38:36.552 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.552 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:36.553 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.553 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.553 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.553 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14660 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e59)", 2020-06-12T02:38:36.553 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.210020+0000", 2020-06-12T02:38:36.553 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 5.2425675700000003, 2020-06-12T02:38:36.554 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017984777, 2020-06-12T02:38:36.554 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:36.554 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:36.554 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:36.554 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:36.555 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:36.555 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14660 2020-06-12T02:38:36.555 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.555 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:36.555 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.556 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:36.556 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210020+0000", 2020-06-12T02:38:36.557 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:36.557 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.558 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.558 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:36.558 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210020+0000", 2020-06-12T02:38:36.559 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2875999999999999e-05 2020-06-12T02:38:36.559 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.559 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.560 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:36.560 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210043+0000", 2020-06-12T02:38:36.560 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2200000000000001e-06 2020-06-12T02:38:36.561 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.561 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.561 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:36.562 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210046+0000", 2020-06-12T02:38:36.562 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7410000000000003e-06 2020-06-12T02:38:36.562 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.563 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.563 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:36.563 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210053+0000", 2020-06-12T02:38:36.564 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5610000000000001e-06 2020-06-12T02:38:36.564 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.564 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.565 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:36.565 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210055+0000", 2020-06-12T02:38:36.565 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0052596409999999998 2020-06-12T02:38:36.566 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.566 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.566 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:36.566 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215315+0000", 2020-06-12T02:38:36.567 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5627999999999997e-05 2020-06-12T02:38:36.568 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.568 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.568 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:36.569 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215371+0000", 2020-06-12T02:38:36.569 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0098069549999999991 2020-06-12T02:38:36.569 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.569 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.569 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:36.570 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225178+0000", 2020-06-12T02:38:36.570 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4530000000000001e-05 2020-06-12T02:38:36.570 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.570 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.570 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:36.571 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225232+0000", 2020-06-12T02:38:36.571 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017497350000000001 2020-06-12T02:38:36.571 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.571 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.571 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:36.571 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.226982+0000", 2020-06-12T02:38:36.572 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8154999999999998e-05 2020-06-12T02:38:36.572 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.572 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.572 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:36.572 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.227080+0000", 2020-06-12T02:38:36.573 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00092473499999999999 2020-06-12T02:38:36.573 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.573 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.573 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:36.573 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.228005+0000", 2020-06-12T02:38:36.573 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017984777 2020-06-12T02:38:36.574 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.574 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:36.574 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.574 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.574 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.575 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14833 3.0s0 3:0dddb690:::smithi10729407-10:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:36.575 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.683734+0000", 2020-06-12T02:38:36.575 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 4.7688536060000004, 2020-06-12T02:38:36.575 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011888449000000001, 2020-06-12T02:38:36.575 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:36.575 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:36.576 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:36.576 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:36.576 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:36.576 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14833 2020-06-12T02:38:36.576 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.577 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:36.577 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.577 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:36.577 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.683734+0000", 2020-06-12T02:38:36.577 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:36.578 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.578 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.578 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:36.579 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.683734+0000", 2020-06-12T02:38:36.579 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8739000000000003e-05 2020-06-12T02:38:36.579 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.579 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.579 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:36.580 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.683773+0000", 2020-06-12T02:38:36.580 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2880000000000001e-06 2020-06-12T02:38:36.580 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.580 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.580 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:36.580 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.683776+0000", 2020-06-12T02:38:36.581 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4930000000000003e-06 2020-06-12T02:38:36.581 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.581 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.581 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:36.581 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.683783+0000", 2020-06-12T02:38:36.582 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3460000000000001e-06 2020-06-12T02:38:36.582 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.582 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.582 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:36.582 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.683785+0000", 2020-06-12T02:38:36.582 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8192000000000003e-05 2020-06-12T02:38:36.583 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.583 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.583 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:36.583 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.683823+0000", 2020-06-12T02:38:36.583 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010585 2020-06-12T02:38:36.584 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.584 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.584 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:36.584 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.683929+0000", 2020-06-12T02:38:36.584 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.008154082 2020-06-12T02:38:36.585 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.585 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.585 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:36.585 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.692083+0000", 2020-06-12T02:38:36.585 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010611 2020-06-12T02:38:36.585 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.586 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.586 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:36.586 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.692189+0000", 2020-06-12T02:38:36.586 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051007299999999999 2020-06-12T02:38:36.586 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.587 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.587 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:36.587 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.692699+0000", 2020-06-12T02:38:36.587 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073264800000000002 2020-06-12T02:38:36.587 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.587 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.588 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:36.588 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.693432+0000", 2020-06-12T02:38:36.588 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0021002239999999999 2020-06-12T02:38:36.589 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.589 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.589 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:36.589 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.695532+0000", 2020-06-12T02:38:36.589 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0403999999999997e-05 2020-06-12T02:38:36.590 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.590 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.590 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:36.590 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.695623+0000", 2020-06-12T02:38:36.590 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011888449000000001 2020-06-12T02:38:36.591 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.591 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:36.591 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.591 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.591 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.591 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14834 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e59)", 2020-06-12T02:38:36.592 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.683808+0000", 2020-06-12T02:38:36.592 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 4.7687792169999996, 2020-06-12T02:38:36.592 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013283056999999999, 2020-06-12T02:38:36.592 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:36.592 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:36.593 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:36.593 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:36.593 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:36.593 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14834 2020-06-12T02:38:36.593 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.593 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:36.594 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.594 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:36.594 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.683808+0000", 2020-06-12T02:38:36.594 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:36.594 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.595 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.595 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:36.595 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.683808+0000", 2020-06-12T02:38:36.595 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2376999999999999e-05 2020-06-12T02:38:36.595 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.596 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.596 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:36.596 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.683831+0000", 2020-06-12T02:38:36.596 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0309999999999999e-06 2020-06-12T02:38:36.596 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.596 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.597 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:36.597 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.683834+0000", 2020-06-12T02:38:36.597 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2779999999999999e-06 2020-06-12T02:38:36.597 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.597 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.598 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:36.598 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.683840+0000", 2020-06-12T02:38:36.598 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.571e-06 2020-06-12T02:38:36.598 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.599 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.599 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:36.599 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.683843+0000", 2020-06-12T02:38:36.599 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000108631 2020-06-12T02:38:36.599 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.600 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.600 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:36.600 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.683951+0000", 2020-06-12T02:38:36.600 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5368000000000003e-05 2020-06-12T02:38:36.600 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.600 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.601 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:36.601 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.684007+0000", 2020-06-12T02:38:36.601 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.009048894 2020-06-12T02:38:36.601 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.601 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.602 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:36.602 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.693056+0000", 2020-06-12T02:38:36.602 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000100839 2020-06-12T02:38:36.602 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.602 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.603 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:36.603 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.693156+0000", 2020-06-12T02:38:36.603 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002839374 2020-06-12T02:38:36.603 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.603 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.603 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:36.604 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.695996+0000", 2020-06-12T02:38:36.604 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7194999999999994e-05 2020-06-12T02:38:36.604 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.604 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.604 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:36.605 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.696093+0000", 2020-06-12T02:38:36.605 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00099849899999999991 2020-06-12T02:38:36.605 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.605 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.605 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:36.605 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.697092+0000", 2020-06-12T02:38:36.606 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013283056999999999 2020-06-12T02:38:36.606 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.606 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:36.606 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.606 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.607 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.607 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.3s2 59/53 ECSubWrite(tid=5970, reqid=client.4512.0:15226, at_version=59'1316, trim_to=0'0, roll_forward_to=59'1315))", 2020-06-12T02:38:36.607 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.829184+0000", 2020-06-12T02:38:36.607 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.6234033540000001, 2020-06-12T02:38:36.607 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.081295588000000002, 2020-06-12T02:38:36.607 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:36.608 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:36.608 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:36.608 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.608 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:36.608 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829184+0000", 2020-06-12T02:38:36.609 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:36.609 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.609 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.609 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:36.610 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829184+0000", 2020-06-12T02:38:36.610 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00023814999999999999 2020-06-12T02:38:36.610 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.610 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.610 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:36.611 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829422+0000", 2020-06-12T02:38:36.611 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9768000000000001e-05 2020-06-12T02:38:36.611 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.611 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.611 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:36.612 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829442+0000", 2020-06-12T02:38:36.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1022000000000001e-05 2020-06-12T02:38:36.612 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.612 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:36.612 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829453+0000", 2020-06-12T02:38:36.613 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2759999999999997e-06 2020-06-12T02:38:36.613 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.613 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.613 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:36.613 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829460+0000", 2020-06-12T02:38:36.614 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7284000000000002e-05 2020-06-12T02:38:36.614 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.614 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.614 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:36.614 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829517+0000", 2020-06-12T02:38:36.614 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7028999999999999e-05 2020-06-12T02:38:36.615 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.615 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.615 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:36.615 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829534+0000", 2020-06-12T02:38:36.615 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.080899873999999997 2020-06-12T02:38:36.616 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.616 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.616 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:36.616 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.910434+0000", 2020-06-12T02:38:36.616 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6184999999999999e-05 2020-06-12T02:38:36.616 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.617 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.617 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:36.617 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.910480+0000", 2020-06-12T02:38:36.617 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.081295588000000002 2020-06-12T02:38:36.617 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.618 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:36.618 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.618 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.618 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.618 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.3s2 59/53 ECSubWrite(tid=5973, reqid=client.4512.0:15229, at_version=59'1319, trim_to=0'0, roll_forward_to=59'1315))", 2020-06-12T02:38:36.618 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.901708+0000", 2020-06-12T02:38:36.619 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.5508794749999999, 2020-06-12T02:38:36.619 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013662957999999999, 2020-06-12T02:38:36.619 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:36.620 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:36.620 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:36.620 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.620 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:36.620 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901708+0000", 2020-06-12T02:38:36.620 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:36.622 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.622 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.622 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:36.623 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901708+0000", 2020-06-12T02:38:36.623 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7360000000000001e-05 2020-06-12T02:38:36.623 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.623 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.623 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:36.624 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901756+0000", 2020-06-12T02:38:36.624 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4608e-05 2020-06-12T02:38:36.624 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.624 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.625 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:36.625 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901770+0000", 2020-06-12T02:38:36.625 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.219e-05 2020-06-12T02:38:36.625 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.625 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.626 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:36.626 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901782+0000", 2020-06-12T02:38:36.626 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.511e-06 2020-06-12T02:38:36.626 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.626 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.627 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:36.627 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901789+0000", 2020-06-12T02:38:36.627 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.008025562 2020-06-12T02:38:36.627 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.628 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.628 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:36.628 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.909814+0000", 2020-06-12T02:38:36.628 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1472e-05 2020-06-12T02:38:36.628 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.628 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.629 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:36.629 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.909826+0000", 2020-06-12T02:38:36.629 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0055013350000000004 2020-06-12T02:38:36.629 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.630 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.630 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:36.630 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.915327+0000", 2020-06-12T02:38:36.630 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.392e-05 2020-06-12T02:38:36.630 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.631 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.631 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:36.631 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.915371+0000", 2020-06-12T02:38:36.631 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013662957999999999 2020-06-12T02:38:36.631 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.631 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:36.632 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.632 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.632 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.632 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.3s2 59/53 ECSubWrite(tid=5974, reqid=client.4512.0:15231, at_version=59'1320, trim_to=0'0, roll_forward_to=59'1315))", 2020-06-12T02:38:36.632 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.901839+0000", 2020-06-12T02:38:36.633 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.5507488770000002, 2020-06-12T02:38:36.633 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013568525, 2020-06-12T02:38:36.633 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:36.633 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:36.633 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:36.634 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.634 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:36.634 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901839+0000", 2020-06-12T02:38:36.634 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:36.634 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.634 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.635 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:36.635 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901839+0000", 2020-06-12T02:38:36.635 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6489999999999997e-05 2020-06-12T02:38:36.635 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.635 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.636 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:36.636 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901885+0000", 2020-06-12T02:38:36.636 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6600999999999999e-05 2020-06-12T02:38:36.636 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.636 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.636 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:36.637 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901902+0000", 2020-06-12T02:38:36.637 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2427e-05 2020-06-12T02:38:36.637 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.637 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.638 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:36.638 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901914+0000", 2020-06-12T02:38:36.638 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1710000000000001e-06 2020-06-12T02:38:36.638 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.638 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.638 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:36.639 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901919+0000", 2020-06-12T02:38:36.639 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0086285540000000001 2020-06-12T02:38:36.639 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.639 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.639 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:36.640 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.910548+0000", 2020-06-12T02:38:36.640 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0939e-05 2020-06-12T02:38:36.640 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.641 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.641 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:36.641 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.910559+0000", 2020-06-12T02:38:36.642 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0048161899999999997 2020-06-12T02:38:36.642 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.642 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.642 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:36.642 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.915375+0000", 2020-06-12T02:38:36.643 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2153000000000002e-05 2020-06-12T02:38:36.643 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.643 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.643 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:36.643 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.915407+0000", 2020-06-12T02:38:36.644 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013568525 2020-06-12T02:38:36.644 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.644 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:36.644 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.644 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.644 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.645 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15246 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~671744] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:36.645 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.953872+0000", 2020-06-12T02:38:36.645 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.4987158109999998, 2020-06-12T02:38:36.645 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015306691000000001, 2020-06-12T02:38:36.646 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:36.646 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:36.646 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:36.646 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:36.646 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:36.647 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15246 2020-06-12T02:38:36.647 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.647 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:36.647 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.647 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:36.647 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.953872+0000", 2020-06-12T02:38:36.648 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:36.648 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.648 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.648 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:36.648 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.953872+0000", 2020-06-12T02:38:36.649 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000232544 2020-06-12T02:38:36.650 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.651 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.651 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:36.651 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954104+0000", 2020-06-12T02:38:36.652 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7513999999999999e-05 2020-06-12T02:38:36.652 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.652 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.652 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:36.652 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954122+0000", 2020-06-12T02:38:36.652 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.844e-05 2020-06-12T02:38:36.653 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.653 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.653 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:36.653 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954140+0000", 2020-06-12T02:38:36.653 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2570000000000007e-06 2020-06-12T02:38:36.654 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.654 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.654 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:36.654 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954149+0000", 2020-06-12T02:38:36.654 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00031049499999999999 2020-06-12T02:38:36.655 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.655 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.655 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:36.656 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954459+0000", 2020-06-12T02:38:36.656 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000118182 2020-06-12T02:38:36.656 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.656 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.656 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:36.657 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954577+0000", 2020-06-12T02:38:36.657 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00071788400000000001 2020-06-12T02:38:36.657 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.657 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.657 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:36.658 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.955295+0000", 2020-06-12T02:38:36.658 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013717859000000001 2020-06-12T02:38:36.658 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.659 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.659 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:36.659 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.969013+0000", 2020-06-12T02:38:36.659 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4895999999999995e-05 2020-06-12T02:38:36.659 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.660 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.660 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:36.660 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.969098+0000", 2020-06-12T02:38:36.660 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0619999999999997e-05 2020-06-12T02:38:36.660 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.660 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.661 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:36.661 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.969179+0000", 2020-06-12T02:38:36.661 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015306691000000001 2020-06-12T02:38:36.661 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.661 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:36.662 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.662 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.662 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.662 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15249 3.0s0 3:0dddb690:::smithi10729407-10:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:36.662 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.962172+0000", 2020-06-12T02:38:36.663 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.490415204, 2020-06-12T02:38:36.663 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011978842, 2020-06-12T02:38:36.663 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:36.663 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:36.663 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:36.663 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:36.664 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:36.664 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15249 2020-06-12T02:38:36.664 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.664 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:36.664 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.665 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:36.665 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.962172+0000", 2020-06-12T02:38:36.665 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:36.666 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.666 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.666 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:36.666 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.962172+0000", 2020-06-12T02:38:36.666 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5105999999999995e-05 2020-06-12T02:38:36.666 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.667 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.667 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:36.667 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.962248+0000", 2020-06-12T02:38:36.667 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6730000000000003e-06 2020-06-12T02:38:36.667 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.668 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.668 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:36.668 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.962254+0000", 2020-06-12T02:38:36.668 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3444e-05 2020-06-12T02:38:36.668 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.668 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.669 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:36.669 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.962268+0000", 2020-06-12T02:38:36.669 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9019999999999998e-06 2020-06-12T02:38:36.669 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.669 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.670 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:36.670 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.962273+0000", 2020-06-12T02:38:36.670 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0058252169999999997 2020-06-12T02:38:36.670 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.670 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.671 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:36.671 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.968098+0000", 2020-06-12T02:38:36.671 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000108826 2020-06-12T02:38:36.671 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.671 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.671 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:36.672 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.968207+0000", 2020-06-12T02:38:36.672 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00045775999999999998 2020-06-12T02:38:36.672 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.672 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.672 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:36.673 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.968664+0000", 2020-06-12T02:38:36.673 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00080255400000000003 2020-06-12T02:38:36.673 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.673 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.673 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:36.673 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.969467+0000", 2020-06-12T02:38:36.674 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045928449999999999 2020-06-12T02:38:36.674 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.674 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.674 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:36.674 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.974060+0000", 2020-06-12T02:38:36.675 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1514999999999994e-05 2020-06-12T02:38:36.675 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.675 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.675 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:36.675 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.974151+0000", 2020-06-12T02:38:36.675 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011978842 2020-06-12T02:38:36.676 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.676 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:36.676 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.676 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.676 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.677 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15250 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e59)", 2020-06-12T02:38:36.677 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.962316+0000", 2020-06-12T02:38:36.677 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.4902716659999999, 2020-06-12T02:38:36.677 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013412125, 2020-06-12T02:38:36.677 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:36.678 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:36.678 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:36.678 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:36.678 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:36.678 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15250 2020-06-12T02:38:36.678 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.679 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:36.679 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.679 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:36.679 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.962316+0000", 2020-06-12T02:38:36.680 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:36.680 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.680 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.680 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:36.680 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.962316+0000", 2020-06-12T02:38:36.681 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9218000000000002e-05 2020-06-12T02:38:36.681 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.681 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.681 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:36.681 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.962365+0000", 2020-06-12T02:38:36.682 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0610000000000001e-06 2020-06-12T02:38:36.682 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.682 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.682 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:36.682 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.962371+0000", 2020-06-12T02:38:36.682 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1848000000000001e-05 2020-06-12T02:38:36.683 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.683 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.683 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:36.683 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.962383+0000", 2020-06-12T02:38:36.683 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0489999999999997e-06 2020-06-12T02:38:36.684 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.684 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.684 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:36.684 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.962387+0000", 2020-06-12T02:38:36.684 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0072499060000000004 2020-06-12T02:38:36.684 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.685 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.685 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:36.685 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.969637+0000", 2020-06-12T02:38:36.685 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2698999999999998e-05 2020-06-12T02:38:36.685 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.686 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.686 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:36.686 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.969690+0000", 2020-06-12T02:38:36.686 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0049581240000000004 2020-06-12T02:38:36.686 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.686 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.687 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:36.687 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.974648+0000", 2020-06-12T02:38:36.687 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101106 2020-06-12T02:38:36.687 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.687 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.688 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:36.688 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.974749+0000", 2020-06-12T02:38:36.688 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0009791139999999999 2020-06-12T02:38:36.688 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.688 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.688 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:36.689 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.975728+0000", 2020-06-12T02:38:36.689 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013412125 2020-06-12T02:38:36.689 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.689 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:36.689 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.690 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.690 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.690 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15277 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~720896] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:36.690 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:33.067109+0000", 2020-06-12T02:38:36.690 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.385478751, 2020-06-12T02:38:36.690 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015909191999999999, 2020-06-12T02:38:36.691 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:36.691 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:36.691 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:36.691 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:36.691 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:36.692 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15277 2020-06-12T02:38:36.692 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.692 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:36.692 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.692 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:36.692 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067109+0000", 2020-06-12T02:38:36.693 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:36.693 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.693 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.693 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:36.693 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067109+0000", 2020-06-12T02:38:36.694 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018431400000000001 2020-06-12T02:38:36.694 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.694 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.694 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:36.694 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067293+0000", 2020-06-12T02:38:36.695 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1790000000000002e-06 2020-06-12T02:38:36.695 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.695 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.695 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:36.695 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067300+0000", 2020-06-12T02:38:36.696 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5259999999999999e-06 2020-06-12T02:38:36.696 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.696 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.696 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:36.697 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067310+0000", 2020-06-12T02:38:36.697 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8179999999999997e-06 2020-06-12T02:38:36.697 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.697 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.697 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:36.698 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067314+0000", 2020-06-12T02:38:36.698 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6529999999999997e-05 2020-06-12T02:38:36.698 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.698 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.698 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:36.699 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067360+0000", 2020-06-12T02:38:36.699 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012724399999999999 2020-06-12T02:38:36.699 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.699 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.700 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:36.700 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067488+0000", 2020-06-12T02:38:36.700 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0010983989999999999 2020-06-12T02:38:36.700 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.700 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.700 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:36.701 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.068586+0000", 2020-06-12T02:38:36.701 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5165000000000001e-05 2020-06-12T02:38:36.701 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.701 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.701 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:36.702 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.068681+0000", 2020-06-12T02:38:36.706 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086510899999999997 2020-06-12T02:38:36.706 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.707 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.707 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:36.707 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.069546+0000", 2020-06-12T02:38:36.707 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013284429 2020-06-12T02:38:36.707 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.707 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.708 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:36.708 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.082831+0000", 2020-06-12T02:38:36.708 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4197000000000006e-05 2020-06-12T02:38:36.708 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.708 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.709 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:36.709 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.082925+0000", 2020-06-12T02:38:36.709 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3282000000000006e-05 2020-06-12T02:38:36.709 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.709 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.710 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:36.710 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.083018+0000", 2020-06-12T02:38:36.710 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015909191999999999 2020-06-12T02:38:36.710 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.710 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:36.711 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.711 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.711 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.711 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15331 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~688128] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:36.711 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:33.276748+0000", 2020-06-12T02:38:36.712 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.1758392390000001, 2020-06-12T02:38:36.712 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016825066999999999, 2020-06-12T02:38:36.712 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:36.712 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:36.712 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:36.712 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:36.713 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:36.713 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15331 2020-06-12T02:38:36.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.713 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:36.713 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.714 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:36.714 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276748+0000", 2020-06-12T02:38:36.714 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:36.714 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.714 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:36.715 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276748+0000", 2020-06-12T02:38:36.715 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016936699999999999 2020-06-12T02:38:36.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.715 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.715 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:36.716 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276918+0000", 2020-06-12T02:38:36.716 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6389999999999998e-06 2020-06-12T02:38:36.716 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.716 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.716 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:36.717 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276926+0000", 2020-06-12T02:38:36.717 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3591000000000001e-05 2020-06-12T02:38:36.717 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.717 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.717 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:36.717 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276940+0000", 2020-06-12T02:38:36.718 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0429999999999996e-06 2020-06-12T02:38:36.718 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.718 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.718 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:36.718 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276949+0000", 2020-06-12T02:38:36.719 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8825000000000001e-05 2020-06-12T02:38:36.719 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.719 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.719 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:36.719 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276998+0000", 2020-06-12T02:38:36.719 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6483000000000005e-05 2020-06-12T02:38:36.720 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.720 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.720 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:36.720 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.277084+0000", 2020-06-12T02:38:36.720 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00057545800000000005 2020-06-12T02:38:36.721 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.721 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.721 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:36.721 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.277660+0000", 2020-06-12T02:38:36.721 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015617126 2020-06-12T02:38:36.721 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.722 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.722 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:36.722 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.293277+0000", 2020-06-12T02:38:36.722 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00021931699999999999 2020-06-12T02:38:36.722 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.723 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.723 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:36.723 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.293496+0000", 2020-06-12T02:38:36.723 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7218000000000005e-05 2020-06-12T02:38:36.723 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.723 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.724 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:36.724 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.293573+0000", 2020-06-12T02:38:36.724 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016825066999999999 2020-06-12T02:38:36.724 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.724 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:36.725 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.725 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.725 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.725 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15336 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e59)", 2020-06-12T02:38:36.725 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:33.289305+0000", 2020-06-12T02:38:36.725 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.1632830969999999, 2020-06-12T02:38:36.726 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01212628, 2020-06-12T02:38:36.726 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:36.726 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:36.726 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:36.726 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:36.727 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:36.727 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15336 2020-06-12T02:38:36.727 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.727 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:36.727 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.727 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:36.728 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.289305+0000", 2020-06-12T02:38:36.728 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:36.728 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.728 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.728 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:36.729 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.289305+0000", 2020-06-12T02:38:36.729 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5797999999999998e-05 2020-06-12T02:38:36.729 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.729 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.729 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:36.730 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.289350+0000", 2020-06-12T02:38:36.730 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4920000000000001e-06 2020-06-12T02:38:36.730 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.730 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.730 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:36.730 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.289356+0000", 2020-06-12T02:38:36.731 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.147e-05 2020-06-12T02:38:36.731 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.731 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.731 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:36.731 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.289367+0000", 2020-06-12T02:38:36.732 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6040000000000003e-06 2020-06-12T02:38:36.732 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.732 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.732 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:36.732 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.289372+0000", 2020-06-12T02:38:36.732 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0079783079999999999 2020-06-12T02:38:36.733 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.733 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.733 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:36.733 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.297350+0000", 2020-06-12T02:38:36.733 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6495e-05 2020-06-12T02:38:36.734 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.734 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.734 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:36.734 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.297407+0000", 2020-06-12T02:38:36.734 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0026554920000000002 2020-06-12T02:38:36.734 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.735 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.735 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:36.735 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.300062+0000", 2020-06-12T02:38:36.735 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4963999999999998e-05 2020-06-12T02:38:36.735 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.736 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.736 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:36.736 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.300117+0000", 2020-06-12T02:38:36.736 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001313657 2020-06-12T02:38:36.736 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.736 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.737 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:36.737 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.301431+0000", 2020-06-12T02:38:36.737 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01212628 2020-06-12T02:38:36.737 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.738 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:36.738 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.738 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.738 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.738 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.3s2 60/53 ECSubWrite(tid=6200, reqid=client.4512.0:16202, at_version=60'1389, trim_to=0'0, roll_forward_to=60'1386))", 2020-06-12T02:38:36.739 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:35.948976+0000", 2020-06-12T02:38:36.739 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.50361165799999996, 2020-06-12T02:38:36.739 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013440881999999999, 2020-06-12T02:38:36.739 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:36.739 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:36.740 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:36.740 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.740 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:36.740 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:35.948976+0000", 2020-06-12T02:38:36.740 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:36.741 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.741 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.741 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:36.741 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:35.948976+0000", 2020-06-12T02:38:36.741 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000267586 2020-06-12T02:38:36.742 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.742 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.742 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:36.742 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:35.949244+0000", 2020-06-12T02:38:36.742 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5032999999999999e-05 2020-06-12T02:38:36.742 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.743 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.743 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:36.743 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:35.949269+0000", 2020-06-12T02:38:36.743 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1072000000000001e-05 2020-06-12T02:38:36.743 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.744 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.744 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:36.744 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:35.949290+0000", 2020-06-12T02:38:36.744 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3470000000000005e-06 2020-06-12T02:38:36.744 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.745 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.745 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:36.745 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:35.949299+0000", 2020-06-12T02:38:36.745 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5457999999999996e-05 2020-06-12T02:38:36.745 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.746 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.746 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:36.746 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:35.949365+0000", 2020-06-12T02:38:36.746 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4459e-05 2020-06-12T02:38:36.746 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.746 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.747 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:36.747 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:35.949379+0000", 2020-06-12T02:38:36.747 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012980017 2020-06-12T02:38:36.747 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.747 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.748 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:36.748 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:35.962359+0000", 2020-06-12T02:38:36.748 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.791e-05 2020-06-12T02:38:36.748 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.748 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.748 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:36.749 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:35.962417+0000", 2020-06-12T02:38:36.749 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013440881999999999 2020-06-12T02:38:36.749 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.749 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:36.749 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.750 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.750 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.750 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.7s1 60/37 ECSubWrite(tid=8504, reqid=client.4512.0:16225, at_version=60'1427, trim_to=0'0, roll_forward_to=60'1425))", 2020-06-12T02:38:36.750 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:36.033154+0000", 2020-06-12T02:38:36.750 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.41943387700000001, 2020-06-12T02:38:36.751 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016768244000000002, 2020-06-12T02:38:36.751 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:36.751 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:36.751 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:36.751 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.751 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:36.752 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033154+0000", 2020-06-12T02:38:36.752 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:36.752 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.752 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.752 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:36.753 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033154+0000", 2020-06-12T02:38:36.754 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000154444 2020-06-12T02:38:36.754 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.754 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.754 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:36.754 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033308+0000", 2020-06-12T02:38:36.755 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8772000000000002e-05 2020-06-12T02:38:36.755 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.755 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.755 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:36.755 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033337+0000", 2020-06-12T02:38:36.756 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6704000000000001e-05 2020-06-12T02:38:36.756 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.756 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.756 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:36.756 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033354+0000", 2020-06-12T02:38:36.756 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6100000000000002e-06 2020-06-12T02:38:36.757 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.757 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.757 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:36.757 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033360+0000", 2020-06-12T02:38:36.757 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000173893 2020-06-12T02:38:36.758 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.758 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.758 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:36.758 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033534+0000", 2020-06-12T02:38:36.758 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1929999999999999e-05 2020-06-12T02:38:36.759 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.759 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.759 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:36.760 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033546+0000", 2020-06-12T02:38:36.760 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016272066000000002 2020-06-12T02:38:36.760 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.760 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.760 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:36.761 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.049818+0000", 2020-06-12T02:38:36.761 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103825 2020-06-12T02:38:36.762 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:36.762 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:36.762 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:36.763 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.049922+0000", 2020-06-12T02:38:36.763 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016768244000000002 2020-06-12T02:38:36.763 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.763 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:36.763 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.764 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:36.764 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:36.764 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:36.779 INFO:tasks.rados.rados.0.smithi107.stdout:7692: done (0 left) 2020-06-12T02:38:36.779 INFO:tasks.rados.rados.0.smithi107.stdout:7693: delete oid 9 current snap is 0 2020-06-12T02:38:36.779 INFO:tasks.rados.rados.0.smithi107.stdout:7693: done (0 left) 2020-06-12T02:38:36.780 INFO:tasks.rados.rados.0.smithi107.stdout:7694: delete oid 11 current snap is 0 2020-06-12T02:38:36.780 INFO:tasks.rados.rados.0.smithi107.stdout:7694: done (0 left) 2020-06-12T02:38:36.780 INFO:tasks.rados.rados.0.smithi107.stdout:7695: delete oid 3 current snap is 0 2020-06-12T02:38:36.780 INFO:tasks.rados.rados.0.smithi107.stdout:7695: done (0 left) 2020-06-12T02:38:36.780 INFO:tasks.rados.rados.0.smithi107.stdout:7696: delete oid 11 current snap is 0 2020-06-12T02:38:36.781 INFO:tasks.rados.rados.0.smithi107.stdout:7696: done (0 left) 2020-06-12T02:38:36.781 INFO:tasks.rados.rados.0.smithi107.stdout:7697: delete oid 6 current snap is 0 2020-06-12T02:38:36.781 INFO:tasks.rados.rados.0.smithi107.stdout:7697: done (0 left) 2020-06-12T02:38:36.781 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:38:36.782 INFO:tasks.rados.rados.0.smithi107.stdout:7698: seq_num 7697 ranges {0=516096,516096=786432,1302528=417792} 2020-06-12T02:38:36.782 INFO:tasks.rados.rados.0.smithi107.stdout:7698: writing smithi10729407-20 from 0 to 516096 tid 1 2020-06-12T02:38:36.782 INFO:tasks.rados.rados.0.smithi107.stdout:7698: writing smithi10729407-20 from 516096 to 1302528 tid 2 2020-06-12T02:38:36.782 INFO:tasks.rados.rados.0.smithi107.stdout:7698: writing smithi10729407-20 from 1302528 to 1720320 tid 3 2020-06-12T02:38:36.782 INFO:tasks.rados.rados.0.smithi107.stdout:7699: delete oid 2 current snap is 0 2020-06-12T02:38:36.783 INFO:tasks.rados.rados.0.smithi107.stdout:7698: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:36.783 INFO:tasks.rados.rados.0.smithi107.stdout:7699: done (1 left) 2020-06-12T02:38:36.783 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:38:36.783 INFO:tasks.rados.rados.0.smithi107.stdout:7700: seq_num 7699 ranges {0=548864,548864=557056,1105920=647168} 2020-06-12T02:38:36.783 INFO:tasks.rados.rados.0.smithi107.stdout:7700: writing smithi10729407-11 from 0 to 548864 tid 1 2020-06-12T02:38:36.784 INFO:tasks.rados.rados.0.smithi107.stdout:7700: writing smithi10729407-11 from 548864 to 1105920 tid 2 2020-06-12T02:38:36.784 INFO:tasks.rados.rados.0.smithi107.stdout:7700: writing smithi10729407-11 from 1105920 to 1753088 tid 3 2020-06-12T02:38:36.784 INFO:tasks.rados.rados.0.smithi107.stdout:7701: delete oid 5 current snap is 0 2020-06-12T02:38:36.784 INFO:tasks.rados.rados.0.smithi107.stdout:7698: finishing write tid 2 to smithi10729407-20 2020-06-12T02:38:36.784 INFO:tasks.rados.rados.0.smithi107.stdout:7698: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:36.784 INFO:tasks.rados.rados.0.smithi107.stdout:7698: finishing write tid 5 to smithi10729407-20 2020-06-12T02:38:36.785 INFO:tasks.rados.rados.0.smithi107.stdout:7698: finishing write tid 6 to smithi10729407-20 2020-06-12T02:38:36.785 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 565 (ObjNum 7697 snap 0 seq_num 7697) dirty exists 2020-06-12T02:38:36.785 INFO:tasks.rados.rados.0.smithi107.stdout:7698: left oid 20 (ObjNum 7697 snap 0 seq_num 7697) 2020-06-12T02:38:36.785 INFO:tasks.rados.rados.0.smithi107.stdout:7700: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:36.785 INFO:tasks.rados.rados.0.smithi107.stdout:7698: done (2 left) 2020-06-12T02:38:36.786 INFO:tasks.rados.rados.0.smithi107.stdout:7701: done (1 left) 2020-06-12T02:38:36.786 INFO:tasks.rados.rados.0.smithi107.stdout:7702: delete oid 14 current snap is 0 2020-06-12T02:38:36.786 INFO:tasks.rados.rados.0.smithi107.stdout:7700: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:36.786 INFO:tasks.rados.rados.0.smithi107.stdout:7700: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:36.786 INFO:tasks.rados.rados.0.smithi107.stdout:7700: finishing write tid 5 to smithi10729407-11 2020-06-12T02:38:36.787 INFO:tasks.rados.rados.0.smithi107.stdout:7700: finishing write tid 6 to smithi10729407-11 2020-06-12T02:38:36.787 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1364 (ObjNum 7699 snap 0 seq_num 7699) dirty exists 2020-06-12T02:38:36.787 INFO:tasks.rados.rados.0.smithi107.stdout:7700: left oid 11 (ObjNum 7699 snap 0 seq_num 7699) 2020-06-12T02:38:36.787 INFO:tasks.rados.rados.0.smithi107.stdout:7700: done (1 left) 2020-06-12T02:38:36.787 INFO:tasks.rados.rados.0.smithi107.stdout:7702: done (0 left) 2020-06-12T02:38:36.788 INFO:tasks.rados.rados.0.smithi107.stdout:7703: delete oid 18 current snap is 0 2020-06-12T02:38:36.788 INFO:tasks.rados.rados.0.smithi107.stdout:7703: done (0 left) 2020-06-12T02:38:36.788 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:38:36.788 INFO:tasks.rados.rados.0.smithi107.stdout:7704: seq_num 7703 ranges {0=614400,614400=540672,1155072=663552} 2020-06-12T02:38:36.788 INFO:tasks.rados.rados.0.smithi107.stdout:7704: writing smithi10729407-2 from 0 to 614400 tid 1 2020-06-12T02:38:36.789 INFO:tasks.rados.rados.0.smithi107.stdout:7704: writing smithi10729407-2 from 614400 to 1155072 tid 2 2020-06-12T02:38:36.789 INFO:tasks.rados.rados.0.smithi107.stdout:7704: writing smithi10729407-2 from 1155072 to 1818624 tid 3 2020-06-12T02:38:36.789 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:36.789 INFO:tasks.rados.rados.0.smithi107.stdout:7705: seq_num 7704 ranges {0=630784,630784=425984,1056768=475136,1531904=303104} 2020-06-12T02:38:36.789 INFO:tasks.rados.rados.0.smithi107.stdout:7705: writing smithi10729407-6 from 0 to 630784 tid 1 2020-06-12T02:38:36.790 INFO:tasks.rados.rados.0.smithi107.stdout:7705: writing smithi10729407-6 from 630784 to 1056768 tid 2 2020-06-12T02:38:36.790 INFO:tasks.rados.rados.0.smithi107.stdout:7705: writing smithi10729407-6 from 1056768 to 1531904 tid 3 2020-06-12T02:38:36.790 INFO:tasks.rados.rados.0.smithi107.stdout:7705: writing smithi10729407-6 from 1531904 to 1835008 tid 4 2020-06-12T02:38:36.790 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:36.790 INFO:tasks.rados.rados.0.smithi107.stdout:7706: seq_num 7705 ranges {0=647168,647168=712704,1359872=491520} 2020-06-12T02:38:36.791 INFO:tasks.rados.rados.0.smithi107.stdout:7706: writing smithi10729407-14 from 0 to 647168 tid 1 2020-06-12T02:38:36.792 INFO:tasks.rados.rados.0.smithi107.stdout:7706: writing smithi10729407-14 from 647168 to 1359872 tid 2 2020-06-12T02:38:36.792 INFO:tasks.rados.rados.0.smithi107.stdout:7706: writing smithi10729407-14 from 1359872 to 1851392 tid 3 2020-06-12T02:38:36.792 INFO:tasks.rados.rados.0.smithi107.stdout:7707: delete oid 18 current snap is 0 2020-06-12T02:38:36.792 INFO:tasks.rados.rados.0.smithi107.stdout:7704: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:36.793 INFO:tasks.rados.rados.0.smithi107.stdout:7704: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:36.794 INFO:tasks.rados.rados.0.smithi107.stdout:7704: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:36.794 INFO:tasks.rados.rados.0.smithi107.stdout:7704: finishing write tid 5 to smithi10729407-2 2020-06-12T02:38:36.794 INFO:tasks.rados.rados.0.smithi107.stdout:7704: finishing write tid 6 to smithi10729407-2 2020-06-12T02:38:36.794 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 726 (ObjNum 7703 snap 0 seq_num 7703) dirty exists 2020-06-12T02:38:36.795 INFO:tasks.rados.rados.0.smithi107.stdout:7704: left oid 2 (ObjNum 7703 snap 0 seq_num 7703) 2020-06-12T02:38:36.795 INFO:tasks.rados.rados.0.smithi107.stdout:7705: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:36.795 INFO:tasks.rados.rados.0.smithi107.stdout:7705: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:36.795 INFO:tasks.rados.rados.0.smithi107.stdout:7705: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:36.795 INFO:tasks.rados.rados.0.smithi107.stdout:7704: done (3 left) 2020-06-12T02:38:36.796 INFO:tasks.rados.rados.0.smithi107.stdout:7707: done (2 left) 2020-06-12T02:38:36.796 INFO:tasks.rados.rados.0.smithi107.stdout:7708: delete oid 10 current snap is 0 2020-06-12T02:38:36.796 INFO:tasks.rados.rados.0.smithi107.stdout:7708: done (2 left) 2020-06-12T02:38:36.796 INFO:tasks.rados.rados.0.smithi107.stdout:7709: delete oid 5 current snap is 0 2020-06-12T02:38:36.796 INFO:tasks.rados.rados.0.smithi107.stdout:7705: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:36.797 INFO:tasks.rados.rados.0.smithi107.stdout:7705: finishing write tid 6 to smithi10729407-6 2020-06-12T02:38:36.797 INFO:tasks.rados.rados.0.smithi107.stdout:7709: done (2 left) 2020-06-12T02:38:36.797 INFO:tasks.rados.rados.0.smithi107.stdout:7710: delete oid 12 current snap is 0 2020-06-12T02:38:36.797 INFO:tasks.rados.rados.0.smithi107.stdout:7706: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:36.797 INFO:tasks.rados.rados.0.smithi107.stdout:7706: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:36.798 INFO:tasks.rados.rados.0.smithi107.stdout:7706: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:36.798 INFO:tasks.rados.rados.0.smithi107.stdout:7705: finishing write tid 7 to smithi10729407-6 2020-06-12T02:38:36.798 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1389 (ObjNum 7704 snap 0 seq_num 7704) dirty exists 2020-06-12T02:38:36.798 INFO:tasks.rados.rados.0.smithi107.stdout:7705: left oid 6 (ObjNum 7704 snap 0 seq_num 7704) 2020-06-12T02:38:36.798 INFO:tasks.rados.rados.0.smithi107.stdout:7706: finishing write tid 5 to smithi10729407-14 2020-06-12T02:38:36.799 INFO:tasks.rados.rados.0.smithi107.stdout:7705: done (2 left) 2020-06-12T02:38:36.799 INFO:tasks.rados.rados.0.smithi107.stdout:7710: done (1 left) 2020-06-12T02:38:36.799 INFO:tasks.rados.rados.0.smithi107.stdout:7711: delete oid 11 current snap is 0 2020-06-12T02:38:36.799 INFO:tasks.rados.rados.0.smithi107.stdout:7706: finishing write tid 6 to smithi10729407-14 2020-06-12T02:38:36.799 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1369 (ObjNum 7705 snap 0 seq_num 7705) dirty exists 2020-06-12T02:38:36.800 INFO:tasks.rados.rados.0.smithi107.stdout:7706: left oid 14 (ObjNum 7705 snap 0 seq_num 7705) 2020-06-12T02:38:36.800 INFO:tasks.rados.rados.0.smithi107.stdout:7706: done (1 left) 2020-06-12T02:38:36.800 INFO:tasks.rados.rados.0.smithi107.stdout:7711: done (0 left) 2020-06-12T02:38:36.800 INFO:tasks.rados.rados.0.smithi107.stdout:7712: delete oid 16 current snap is 0 2020-06-12T02:38:36.800 INFO:tasks.rados.rados.0.smithi107.stdout:7712: done (0 left) 2020-06-12T02:38:36.801 INFO:tasks.rados.rados.0.smithi107.stdout:7713: delete oid 17 current snap is 0 2020-06-12T02:38:36.801 INFO:tasks.rados.rados.0.smithi107.stdout:7713: done (0 left) 2020-06-12T02:38:36.801 INFO:tasks.rados.rados.0.smithi107.stdout:7714: delete oid 18 current snap is 0 2020-06-12T02:38:36.801 INFO:tasks.rados.rados.0.smithi107.stdout:7714: done (0 left) 2020-06-12T02:38:36.801 INFO:tasks.rados.rados.0.smithi107.stdout:7715: delete oid 3 current snap is 0 2020-06-12T02:38:36.802 INFO:tasks.rados.rados.0.smithi107.stdout:7715: done (0 left) 2020-06-12T02:38:36.802 INFO:tasks.rados.rados.0.smithi107.stdout:7716: delete oid 2 current snap is 0 2020-06-12T02:38:36.802 INFO:tasks.rados.rados.0.smithi107.stdout:7716: done (0 left) 2020-06-12T02:38:36.802 INFO:tasks.rados.rados.0.smithi107.stdout:7717: delete oid 3 current snap is 0 2020-06-12T02:38:36.802 INFO:tasks.rados.rados.0.smithi107.stdout:7717: done (0 left) 2020-06-12T02:38:36.803 INFO:tasks.rados.rados.0.smithi107.stdout:7718: delete oid 10 current snap is 0 2020-06-12T02:38:36.803 INFO:tasks.rados.rados.0.smithi107.stdout:7718: done (0 left) 2020-06-12T02:38:36.803 INFO:tasks.rados.rados.0.smithi107.stdout:7719: delete oid 14 current snap is 0 2020-06-12T02:38:36.803 INFO:tasks.rados.rados.0.smithi107.stdout:7719: done (0 left) 2020-06-12T02:38:36.803 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:36.804 INFO:tasks.rados.rados.0.smithi107.stdout:7720: seq_num 7719 ranges {0=483328,483328=786432,1269760=589824,1859584=229376} 2020-06-12T02:38:36.804 INFO:tasks.rados.rados.0.smithi107.stdout:7720: writing smithi10729407-14 from 0 to 483328 tid 1 2020-06-12T02:38:36.804 INFO:tasks.rados.rados.0.smithi107.stdout:7720: writing smithi10729407-14 from 483328 to 1269760 tid 2 2020-06-12T02:38:36.804 INFO:tasks.rados.rados.0.smithi107.stdout:7720: writing smithi10729407-14 from 1269760 to 1859584 tid 3 2020-06-12T02:38:36.804 INFO:tasks.rados.rados.0.smithi107.stdout:7720: writing smithi10729407-14 from 1859584 to 2088960 tid 4 2020-06-12T02:38:36.805 INFO:tasks.rados.rados.0.smithi107.stdout:7721: delete oid 12 current snap is 0 2020-06-12T02:38:36.805 INFO:tasks.rados.rados.0.smithi107.stdout:7720: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:36.805 INFO:tasks.rados.rados.0.smithi107.stdout:7721: done (1 left) 2020-06-12T02:38:36.805 INFO:tasks.rados.rados.0.smithi107.stdout:7722: delete oid 16 current snap is 0 2020-06-12T02:38:36.805 INFO:tasks.rados.rados.0.smithi107.stdout:7722: done (1 left) 2020-06-12T02:38:36.806 INFO:tasks.rados.rados.0.smithi107.stdout:7723: delete oid 5 current snap is 0 2020-06-12T02:38:36.806 INFO:tasks.rados.rados.0.smithi107.stdout:7720: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:36.806 INFO:tasks.rados.rados.0.smithi107.stdout:7720: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:36.806 INFO:tasks.rados.rados.0.smithi107.stdout:7720: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:36.806 INFO:tasks.rados.rados.0.smithi107.stdout:7723: done (1 left) 2020-06-12T02:38:36.807 INFO:tasks.rados.rados.0.smithi107.stdout:7724: delete oid 16 current snap is 0 2020-06-12T02:38:36.807 INFO:tasks.rados.rados.0.smithi107.stdout:7720: finishing write tid 6 to smithi10729407-14 2020-06-12T02:38:36.807 INFO:tasks.rados.rados.0.smithi107.stdout:7720: finishing write tid 7 to smithi10729407-14 2020-06-12T02:38:36.807 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1376 (ObjNum 7719 snap 0 seq_num 7719) dirty exists 2020-06-12T02:38:36.807 INFO:tasks.rados.rados.0.smithi107.stdout:7720: left oid 14 (ObjNum 7719 snap 0 seq_num 7719) 2020-06-12T02:38:36.807 INFO:tasks.rados.rados.0.smithi107.stdout:7720: done (1 left) 2020-06-12T02:38:36.808 INFO:tasks.rados.rados.0.smithi107.stdout:7724: done (0 left) 2020-06-12T02:38:36.808 INFO:tasks.rados.rados.0.smithi107.stdout:7725: delete oid 18 current snap is 0 2020-06-12T02:38:36.808 INFO:tasks.rados.rados.0.smithi107.stdout:7725: done (0 left) 2020-06-12T02:38:36.808 INFO:tasks.rados.rados.0.smithi107.stdout:7726: delete oid 5 current snap is 0 2020-06-12T02:38:36.808 INFO:tasks.rados.rados.0.smithi107.stdout:7726: done (0 left) 2020-06-12T02:38:36.809 INFO:tasks.rados.rados.0.smithi107.stdout:7727: delete oid 9 current snap is 0 2020-06-12T02:38:36.809 INFO:tasks.rados.rados.0.smithi107.stdout:7727: done (0 left) 2020-06-12T02:38:36.809 INFO:tasks.rados.rados.0.smithi107.stdout:7728: delete oid 11 current snap is 0 2020-06-12T02:38:36.809 INFO:tasks.rados.rados.0.smithi107.stdout:7728: done (0 left) 2020-06-12T02:38:36.809 INFO:tasks.rados.rados.0.smithi107.stdout:7729: delete oid 13 current snap is 0 2020-06-12T02:38:36.810 INFO:tasks.rados.rados.0.smithi107.stdout:7729: done (0 left) 2020-06-12T02:38:36.810 INFO:tasks.rados.rados.0.smithi107.stdout:7730: delete oid 19 current snap is 0 2020-06-12T02:38:36.810 INFO:tasks.rados.rados.0.smithi107.stdout:7730: done (0 left) 2020-06-12T02:38:36.810 INFO:tasks.rados.rados.0.smithi107.stdout:7731: delete oid 19 current snap is 0 2020-06-12T02:38:36.810 INFO:tasks.rados.rados.0.smithi107.stdout:7731: done (0 left) 2020-06-12T02:38:36.811 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:36.811 INFO:tasks.rados.rados.0.smithi107.stdout:7732: seq_num 7731 ranges {0=688128,688128=647168,1335296=532480,1867776=425984} 2020-06-12T02:38:36.811 INFO:tasks.rados.rados.0.smithi107.stdout:7732: writing smithi10729407-7 from 0 to 688128 tid 1 2020-06-12T02:38:36.811 INFO:tasks.rados.rados.0.smithi107.stdout:7732: writing smithi10729407-7 from 688128 to 1335296 tid 2 2020-06-12T02:38:36.811 INFO:tasks.rados.rados.0.smithi107.stdout:7732: writing smithi10729407-7 from 1335296 to 1867776 tid 3 2020-06-12T02:38:36.812 INFO:tasks.rados.rados.0.smithi107.stdout:7732: writing smithi10729407-7 from 1867776 to 2293760 tid 4 2020-06-12T02:38:36.812 INFO:tasks.rados.rados.0.smithi107.stdout:7733: delete oid 1 current snap is 0 2020-06-12T02:38:36.812 INFO:tasks.rados.rados.0.smithi107.stdout:7732: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:36.812 INFO:tasks.rados.rados.0.smithi107.stdout:7733: done (1 left) 2020-06-12T02:38:36.812 INFO:tasks.rados.rados.0.smithi107.stdout:7734: delete oid 5 current snap is 0 2020-06-12T02:38:36.813 INFO:tasks.rados.rados.0.smithi107.stdout:7734: done (1 left) 2020-06-12T02:38:36.813 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:36.813 INFO:tasks.rados.rados.0.smithi107.stdout:7735: seq_num 7734 ranges {0=737280,737280=753664,1490944=704512,2195456=147456} 2020-06-12T02:38:36.813 INFO:tasks.rados.rados.0.smithi107.stdout:7735: writing smithi10729407-18 from 0 to 737280 tid 1 2020-06-12T02:38:36.813 INFO:tasks.rados.rados.0.smithi107.stdout:7735: writing smithi10729407-18 from 737280 to 1490944 tid 2 2020-06-12T02:38:36.814 INFO:tasks.rados.rados.0.smithi107.stdout:7735: writing smithi10729407-18 from 1490944 to 2195456 tid 3 2020-06-12T02:38:36.814 INFO:tasks.rados.rados.0.smithi107.stdout:7735: writing smithi10729407-18 from 2195456 to 2342912 tid 4 2020-06-12T02:38:36.814 INFO:tasks.rados.rados.0.smithi107.stdout:7736: delete oid 16 current snap is 0 2020-06-12T02:38:36.814 INFO:tasks.rados.rados.0.smithi107.stdout:7732: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:36.814 INFO:tasks.rados.rados.0.smithi107.stdout:7732: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:36.815 INFO:tasks.rados.rados.0.smithi107.stdout:7732: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:36.815 INFO:tasks.rados.rados.0.smithi107.stdout:7732: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:36.815 INFO:tasks.rados.rados.0.smithi107.stdout:7732: finishing write tid 7 to smithi10729407-7 2020-06-12T02:38:36.815 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1421 (ObjNum 7731 snap 0 seq_num 7731) dirty exists 2020-06-12T02:38:36.815 INFO:tasks.rados.rados.0.smithi107.stdout:7732: left oid 7 (ObjNum 7731 snap 0 seq_num 7731) 2020-06-12T02:38:36.816 INFO:tasks.rados.rados.0.smithi107.stdout:7735: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:36.816 INFO:tasks.rados.rados.0.smithi107.stdout:7732: done (2 left) 2020-06-12T02:38:36.816 INFO:tasks.rados.rados.0.smithi107.stdout:7736: done (1 left) 2020-06-12T02:38:36.816 INFO:tasks.rados.rados.0.smithi107.stdout:7737: delete oid 10 current snap is 0 2020-06-12T02:38:36.816 INFO:tasks.rados.rados.0.smithi107.stdout:7737: done (1 left) 2020-06-12T02:38:36.817 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:36.817 INFO:tasks.rados.rados.0.smithi107.stdout:7738: seq_num 7737 ranges {2088960=786432,2875392=409600,3284992=532480,3817472=663552} 2020-06-12T02:38:36.817 INFO:tasks.rados.rados.0.smithi107.stdout:7738: writing smithi10729407-14 from 2088960 to 2875392 tid 1 2020-06-12T02:38:36.817 INFO:tasks.rados.rados.0.smithi107.stdout:7738: writing smithi10729407-14 from 2875392 to 3284992 tid 2 2020-06-12T02:38:36.818 INFO:tasks.rados.rados.0.smithi107.stdout:7738: writing smithi10729407-14 from 3284992 to 3817472 tid 3 2020-06-12T02:38:36.818 INFO:tasks.rados.rados.0.smithi107.stdout:7738: writing smithi10729407-14 from 3817472 to 4481024 tid 4 2020-06-12T02:38:36.818 INFO:tasks.rados.rados.0.smithi107.stdout:7739: delete oid 16 current snap is 0 2020-06-12T02:38:36.818 INFO:tasks.rados.rados.0.smithi107.stdout:7735: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:36.819 INFO:tasks.rados.rados.0.smithi107.stdout:7735: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:36.819 INFO:tasks.rados.rados.0.smithi107.stdout:7735: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:36.819 INFO:tasks.rados.rados.0.smithi107.stdout:7735: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:36.819 INFO:tasks.rados.rados.0.smithi107.stdout:7735: finishing write tid 7 to smithi10729407-18 2020-06-12T02:38:36.819 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1455 (ObjNum 7734 snap 0 seq_num 7734) dirty exists 2020-06-12T02:38:36.820 INFO:tasks.rados.rados.0.smithi107.stdout:7735: left oid 18 (ObjNum 7734 snap 0 seq_num 7734) 2020-06-12T02:38:36.820 INFO:tasks.rados.rados.0.smithi107.stdout:7738: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:36.820 INFO:tasks.rados.rados.0.smithi107.stdout:7735: done (2 left) 2020-06-12T02:38:36.820 INFO:tasks.rados.rados.0.smithi107.stdout:7739: done (1 left) 2020-06-12T02:38:36.820 INFO:tasks.rados.rados.0.smithi107.stdout:7740: delete oid 13 current snap is 0 2020-06-12T02:38:36.821 INFO:tasks.rados.rados.0.smithi107.stdout:7740: done (1 left) 2020-06-12T02:38:36.821 INFO:tasks.rados.rados.0.smithi107.stdout:7741: delete oid 3 current snap is 0 2020-06-12T02:38:36.821 INFO:tasks.rados.rados.0.smithi107.stdout:7738: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:36.821 INFO:tasks.rados.rados.0.smithi107.stdout:7738: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:36.822 INFO:tasks.rados.rados.0.smithi107.stdout:7738: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:36.822 INFO:tasks.rados.rados.0.smithi107.stdout:7741: done (1 left) 2020-06-12T02:38:36.822 INFO:tasks.rados.rados.0.smithi107.stdout:7742: delete oid 1 current snap is 0 2020-06-12T02:38:36.822 INFO:tasks.rados.rados.0.smithi107.stdout:7738: finishing write tid 6 to smithi10729407-14 2020-06-12T02:38:36.822 INFO:tasks.rados.rados.0.smithi107.stdout:7738: finishing write tid 7 to smithi10729407-14 2020-06-12T02:38:36.823 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1382 (ObjNum 7737 snap 0 seq_num 7737) dirty exists 2020-06-12T02:38:36.823 INFO:tasks.rados.rados.0.smithi107.stdout:7738: left oid 14 (ObjNum 7737 snap 0 seq_num 7737) 2020-06-12T02:38:36.823 INFO:tasks.rados.rados.0.smithi107.stdout:7738: done (1 left) 2020-06-12T02:38:36.823 INFO:tasks.rados.rados.0.smithi107.stdout:7742: done (0 left) 2020-06-12T02:38:36.823 INFO:tasks.rados.rados.0.smithi107.stdout:7743: delete oid 5 current snap is 0 2020-06-12T02:38:36.824 INFO:tasks.rados.rados.0.smithi107.stdout:7743: done (0 left) 2020-06-12T02:38:36.824 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:36.824 INFO:tasks.rados.rados.0.smithi107.stdout:7744: seq_num 7743 ranges {2293760=81920} 2020-06-12T02:38:36.824 INFO:tasks.rados.rados.0.smithi107.stdout:7744: writing smithi10729407-7 from 2293760 to 2375680 tid 1 2020-06-12T02:38:36.824 INFO:tasks.rados.rados.0.smithi107.stdout:7745: delete oid 5 current snap is 0 2020-06-12T02:38:36.825 INFO:tasks.rados.rados.0.smithi107.stdout:7745: done (1 left) 2020-06-12T02:38:36.825 INFO:tasks.rados.rados.0.smithi107.stdout:7746: delete oid 12 current snap is 0 2020-06-12T02:38:36.825 INFO:tasks.rados.rados.0.smithi107.stdout:7744: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:36.825 INFO:tasks.rados.rados.0.smithi107.stdout:7744: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:36.825 INFO:tasks.rados.rados.0.smithi107.stdout:7746: done (1 left) 2020-06-12T02:38:36.826 INFO:tasks.rados.rados.0.smithi107.stdout:7747: delete oid 4 current snap is 0 2020-06-12T02:38:36.826 INFO:tasks.rados.rados.0.smithi107.stdout:7744: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:36.826 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1424 (ObjNum 7743 snap 0 seq_num 7743) dirty exists 2020-06-12T02:38:36.826 INFO:tasks.rados.rados.0.smithi107.stdout:7744: left oid 7 (ObjNum 7743 snap 0 seq_num 7743) 2020-06-12T02:38:36.826 INFO:tasks.rados.rados.0.smithi107.stdout:7744: done (1 left) 2020-06-12T02:38:36.827 INFO:tasks.rados.rados.0.smithi107.stdout:7747: done (0 left) 2020-06-12T02:38:36.827 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:36.827 INFO:tasks.rados.rados.0.smithi107.stdout:7748: seq_num 7747 ranges {0=155648} 2020-06-12T02:38:36.827 INFO:tasks.rados.rados.0.smithi107.stdout:7748: writing smithi10729407-5 from 0 to 155648 tid 1 2020-06-12T02:38:36.827 INFO:tasks.rados.rados.0.smithi107.stdout:7749: delete oid 17 current snap is 0 2020-06-12T02:38:36.827 INFO:tasks.rados.rados.0.smithi107.stdout:7749: done (1 left) 2020-06-12T02:38:36.828 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:36.828 INFO:tasks.rados.rados.0.smithi107.stdout:7750: seq_num 7749 ranges {0=188416} 2020-06-12T02:38:36.829 INFO:tasks.rados.rados.0.smithi107.stdout:7750: writing smithi10729407-4 from 0 to 188416 tid 1 2020-06-12T02:38:36.829 INFO:tasks.rados.rados.0.smithi107.stdout:7751: delete oid 11 current snap is 0 2020-06-12T02:38:36.829 INFO:tasks.rados.rados.0.smithi107.stdout:7748: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:36.830 INFO:tasks.rados.rados.0.smithi107.stdout:7751: done (2 left) 2020-06-12T02:38:36.830 INFO:tasks.rados.rados.0.smithi107.stdout:7752: delete oid 12 current snap is 0 2020-06-12T02:38:36.830 INFO:tasks.rados.rados.0.smithi107.stdout:7748: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:36.830 INFO:tasks.rados.rados.0.smithi107.stdout:7752: done (2 left) 2020-06-12T02:38:36.831 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:38:36.831 INFO:tasks.rados.rados.0.smithi107.stdout:7753: seq_num 7752 ranges {1720320=237568} 2020-06-12T02:38:36.831 INFO:tasks.rados.rados.0.smithi107.stdout:7753: writing smithi10729407-20 from 1720320 to 1957888 tid 1 2020-06-12T02:38:36.831 INFO:tasks.rados.rados.0.smithi107.stdout:7754: delete oid 15 current snap is 0 2020-06-12T02:38:36.831 INFO:tasks.rados.rados.0.smithi107.stdout:7750: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:36.832 INFO:tasks.rados.rados.0.smithi107.stdout:7748: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:36.832 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 774 (ObjNum 7747 snap 0 seq_num 7747) dirty exists 2020-06-12T02:38:36.832 INFO:tasks.rados.rados.0.smithi107.stdout:7748: left oid 5 (ObjNum 7747 snap 0 seq_num 7747) 2020-06-12T02:38:36.832 INFO:tasks.rados.rados.0.smithi107.stdout:7750: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:36.832 INFO:tasks.rados.rados.0.smithi107.stdout:7750: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:36.833 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1449 (ObjNum 7749 snap 0 seq_num 7749) dirty exists 2020-06-12T02:38:36.833 INFO:tasks.rados.rados.0.smithi107.stdout:7750: left oid 4 (ObjNum 7749 snap 0 seq_num 7749) 2020-06-12T02:38:36.833 INFO:tasks.rados.rados.0.smithi107.stdout:7748: done (3 left) 2020-06-12T02:38:36.833 INFO:tasks.rados.rados.0.smithi107.stdout:7750: done (2 left) 2020-06-12T02:38:36.833 INFO:tasks.rados.rados.0.smithi107.stdout:7754: done (1 left) 2020-06-12T02:38:36.834 INFO:tasks.rados.rados.0.smithi107.stdout:7755: delete oid 10 current snap is 0 2020-06-12T02:38:36.834 INFO:tasks.rados.rados.0.smithi107.stdout:7753: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:36.834 INFO:tasks.rados.rados.0.smithi107.stdout:7755: done (1 left) 2020-06-12T02:38:36.834 INFO:tasks.rados.rados.0.smithi107.stdout:7756: delete oid 8 current snap is 0 2020-06-12T02:38:36.834 INFO:tasks.rados.rados.0.smithi107.stdout:7753: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:36.835 INFO:tasks.rados.rados.0.smithi107.stdout:7753: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:36.835 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 567 (ObjNum 7752 snap 0 seq_num 7752) dirty exists 2020-06-12T02:38:36.835 INFO:tasks.rados.rados.0.smithi107.stdout:7753: left oid 20 (ObjNum 7752 snap 0 seq_num 7752) 2020-06-12T02:38:36.835 INFO:tasks.rados.rados.0.smithi107.stdout:7753: done (1 left) 2020-06-12T02:38:36.836 INFO:tasks.rados.rados.0.smithi107.stdout:7756: done (0 left) 2020-06-12T02:38:36.836 INFO:tasks.rados.rados.0.smithi107.stdout:7757: delete oid 16 current snap is 0 2020-06-12T02:38:36.836 INFO:tasks.rados.rados.0.smithi107.stdout:7757: done (0 left) 2020-06-12T02:38:36.836 INFO:tasks.rados.rados.0.smithi107.stdout:7758: delete oid 18 current snap is 0 2020-06-12T02:38:36.836 INFO:tasks.rados.rados.0.smithi107.stdout:7758: done (0 left) 2020-06-12T02:38:36.837 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:36.837 INFO:tasks.rados.rados.0.smithi107.stdout:7759: seq_num 7758 ranges {4481024=335872} 2020-06-12T02:38:36.838 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-06-12T02:38:36.838 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:36.762+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:36.842 INFO:tasks.rados.rados.0.smithi107.stdout:7759: writing smithi10729407-14 from 4481024 to 4816896 tid 1 2020-06-12T02:38:36.843 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:38:36.843 INFO:tasks.rados.rados.0.smithi107.stdout:7760: seq_num 7759 ranges {0=352256} 2020-06-12T02:38:36.843 INFO:tasks.rados.rados.0.smithi107.stdout:7760: writing smithi10729407-10 from 0 to 352256 tid 1 2020-06-12T02:38:36.843 INFO:tasks.rados.rados.0.smithi107.stdout:7761: delete oid 7 current snap is 0 2020-06-12T02:38:36.844 INFO:tasks.rados.rados.0.smithi107.stdout:7761: done (2 left) 2020-06-12T02:38:36.844 INFO:tasks.rados.rados.0.smithi107.stdout:7762: delete oid 3 current snap is 0 2020-06-12T02:38:36.844 INFO:tasks.rados.rados.0.smithi107.stdout:7759: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:36.845 INFO:tasks.rados.rados.0.smithi107.stdout:7760: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:36.845 INFO:tasks.rados.rados.0.smithi107.stdout:7759: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:36.845 INFO:tasks.rados.rados.0.smithi107.stdout:7759: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:36.845 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1385 (ObjNum 7758 snap 0 seq_num 7758) dirty exists 2020-06-12T02:38:36.846 INFO:tasks.rados.rados.0.smithi107.stdout:7759: left oid 14 (ObjNum 7758 snap 0 seq_num 7758) 2020-06-12T02:38:36.846 INFO:tasks.rados.rados.0.smithi107.stdout:7760: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:36.846 INFO:tasks.rados.rados.0.smithi107.stdout:7760: finishing write tid 4 to smithi10729407-10 2020-06-12T02:38:36.846 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 526 (ObjNum 7759 snap 0 seq_num 7759) dirty exists 2020-06-12T02:38:36.846 INFO:tasks.rados.rados.0.smithi107.stdout:7760: left oid 10 (ObjNum 7759 snap 0 seq_num 7759) 2020-06-12T02:38:36.847 INFO:tasks.rados.rados.0.smithi107.stdout:7759: done (2 left) 2020-06-12T02:38:36.847 INFO:tasks.rados.rados.0.smithi107.stdout:7760: done (1 left) 2020-06-12T02:38:36.847 INFO:tasks.rados.rados.0.smithi107.stdout:7762: done (0 left) 2020-06-12T02:38:36.847 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:38:36.847 INFO:tasks.rados.rados.0.smithi107.stdout:7763: seq_num 7762 ranges {0=401408} 2020-06-12T02:38:36.847 INFO:tasks.rados.rados.0.smithi107.stdout:7763: writing smithi10729407-3 from 0 to 401408 tid 1 2020-06-12T02:38:36.848 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:38:36.848 INFO:tasks.rados.rados.0.smithi107.stdout:7764: seq_num 7763 ranges {352256=417792} 2020-06-12T02:38:36.848 INFO:tasks.rados.rados.0.smithi107.stdout:7764: writing smithi10729407-10 from 352256 to 770048 tid 1 2020-06-12T02:38:36.848 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:36.848 INFO:tasks.rados.rados.0.smithi107.stdout:7765: seq_num 7764 ranges {0=434176} 2020-06-12T02:38:36.849 INFO:tasks.rados.rados.0.smithi107.stdout:7765: writing smithi10729407-7 from 0 to 434176 tid 1 2020-06-12T02:38:36.849 INFO:tasks.rados.rados.0.smithi107.stdout:7766: delete oid 12 current snap is 0 2020-06-12T02:38:36.849 INFO:tasks.rados.rados.0.smithi107.stdout:7763: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:36.849 INFO:tasks.rados.rados.0.smithi107.stdout:7763: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:36.849 INFO:tasks.rados.rados.0.smithi107.stdout:7764: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:36.850 INFO:tasks.rados.rados.0.smithi107.stdout:7763: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:36.850 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1413 (ObjNum 7762 snap 0 seq_num 7762) dirty exists 2020-06-12T02:38:36.850 INFO:tasks.rados.rados.0.smithi107.stdout:7763: left oid 3 (ObjNum 7762 snap 0 seq_num 7762) 2020-06-12T02:38:36.850 INFO:tasks.rados.rados.0.smithi107.stdout:7764: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:36.850 INFO:tasks.rados.rados.0.smithi107.stdout:7764: finishing write tid 4 to smithi10729407-10 2020-06-12T02:38:36.851 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 528 (ObjNum 7763 snap 0 seq_num 7763) dirty exists 2020-06-12T02:38:36.851 INFO:tasks.rados.rados.0.smithi107.stdout:7764: left oid 10 (ObjNum 7763 snap 0 seq_num 7763) 2020-06-12T02:38:36.851 INFO:tasks.rados.rados.0.smithi107.stdout:7763: done (3 left) 2020-06-12T02:38:36.851 INFO:tasks.rados.rados.0.smithi107.stdout:7764: done (2 left) 2020-06-12T02:38:36.851 INFO:tasks.rados.rados.0.smithi107.stdout:7766: done (1 left) 2020-06-12T02:38:36.852 INFO:tasks.rados.rados.0.smithi107.stdout:7767: delete oid 19 current snap is 0 2020-06-12T02:38:36.852 INFO:tasks.rados.rados.0.smithi107.stdout:7765: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:36.852 INFO:tasks.rados.rados.0.smithi107.stdout:7765: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:36.852 INFO:tasks.rados.rados.0.smithi107.stdout:7767: done (1 left) 2020-06-12T02:38:36.852 INFO:tasks.rados.rados.0.smithi107.stdout:7768: delete oid 19 current snap is 0 2020-06-12T02:38:36.853 INFO:tasks.rados.rados.0.smithi107.stdout:7765: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:36.853 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1427 (ObjNum 7764 snap 0 seq_num 7764) dirty exists 2020-06-12T02:38:36.853 INFO:tasks.rados.rados.0.smithi107.stdout:7765: left oid 7 (ObjNum 7764 snap 0 seq_num 7764) 2020-06-12T02:38:36.853 INFO:tasks.rados.rados.0.smithi107.stdout:7765: done (1 left) 2020-06-12T02:38:36.854 INFO:tasks.rados.rados.0.smithi107.stdout:7768: done (0 left) 2020-06-12T02:38:36.854 INFO:tasks.rados.rados.0.smithi107.stdout:7769: delete oid 3 current snap is 0 2020-06-12T02:38:36.854 INFO:tasks.rados.rados.0.smithi107.stdout:7769: done (0 left) 2020-06-12T02:38:36.854 INFO:tasks.rados.rados.0.smithi107.stdout:7770: delete oid 9 current snap is 0 2020-06-12T02:38:36.854 INFO:tasks.rados.rados.0.smithi107.stdout:7770: done (0 left) 2020-06-12T02:38:36.854 INFO:tasks.rados.rados.0.smithi107.stdout:7771: delete oid 10 current snap is 0 2020-06-12T02:38:36.855 INFO:tasks.rados.rados.0.smithi107.stdout:7771: done (0 left) 2020-06-12T02:38:36.855 INFO:tasks.rados.rados.0.smithi107.stdout:7772: delete oid 12 current snap is 0 2020-06-12T02:38:36.855 INFO:tasks.rados.rados.0.smithi107.stdout:7772: done (0 left) 2020-06-12T02:38:36.855 INFO:tasks.rados.rados.0.smithi107.stdout:7773: delete oid 15 current snap is 0 2020-06-12T02:38:36.856 INFO:tasks.rados.rados.0.smithi107.stdout:7773: done (0 left) 2020-06-12T02:38:36.856 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:38:36.857 INFO:tasks.rados.rados.0.smithi107.stdout:7774: seq_num 7773 ranges {1957888=589824} 2020-06-12T02:38:36.858 INFO:tasks.rados.rados.0.smithi107.stdout:7774: writing smithi10729407-20 from 1957888 to 2547712 tid 1 2020-06-12T02:38:36.858 INFO:tasks.rados.rados.0.smithi107.stdout:7775: delete oid 6 current snap is 0 2020-06-12T02:38:36.860 INFO:tasks.rados.rados.0.smithi107.stdout:7775: done (1 left) 2020-06-12T02:38:36.860 INFO:tasks.rados.rados.0.smithi107.stdout:7776: delete oid 4 current snap is 0 2020-06-12T02:38:36.865 INFO:tasks.rados.rados.0.smithi107.stdout:7776: done (1 left) 2020-06-12T02:38:36.865 INFO:tasks.rados.rados.0.smithi107.stdout:7777: delete oid 17 current snap is 0 2020-06-12T02:38:36.866 INFO:tasks.rados.rados.0.smithi107.stdout:7774: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:36.866 INFO:tasks.rados.rados.0.smithi107.stdout:7774: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:36.867 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:36.863+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:36.867 INFO:tasks.rados.rados.0.smithi107.stdout:7774: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:36.867 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 569 (ObjNum 7773 snap 0 seq_num 7773) dirty exists 2020-06-12T02:38:36.868 INFO:tasks.rados.rados.0.smithi107.stdout:7774: left oid 20 (ObjNum 7773 snap 0 seq_num 7773) 2020-06-12T02:38:36.868 INFO:tasks.rados.rados.0.smithi107.stdout:7774: done (1 left) 2020-06-12T02:38:36.868 INFO:tasks.rados.rados.0.smithi107.stdout:7777: done (0 left) 2020-06-12T02:38:36.869 INFO:tasks.rados.rados.0.smithi107.stdout:7778: delete oid 19 current snap is 0 2020-06-12T02:38:36.870 INFO:tasks.rados.rados.0.smithi107.stdout:7778: done (0 left) 2020-06-12T02:38:36.871 INFO:tasks.rados.rados.0.smithi107.stdout:7779: delete oid 18 current snap is 0 2020-06-12T02:38:36.873 INFO:tasks.rados.rados.0.smithi107.stdout:7779: done (0 left) 2020-06-12T02:38:36.873 INFO:tasks.rados.rados.0.smithi107.stdout:7780: delete oid 1 current snap is 0 2020-06-12T02:38:36.875 INFO:tasks.rados.rados.0.smithi107.stdout:7780: done (0 left) 2020-06-12T02:38:36.875 INFO:tasks.rados.rados.0.smithi107.stdout:7781: delete oid 7 current snap is 0 2020-06-12T02:38:36.877 INFO:tasks.rados.rados.0.smithi107.stdout:7781: done (0 left) 2020-06-12T02:38:36.877 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:36.877 INFO:tasks.rados.rados.0.smithi107.stdout:7782: seq_num 7781 ranges {0=720896} 2020-06-12T02:38:36.881 INFO:tasks.rados.rados.0.smithi107.stdout:7782: writing smithi10729407-17 from 0 to 720896 tid 1 2020-06-12T02:38:36.882 INFO:tasks.rados.rados.0.smithi107.stdout:7783: delete oid 16 current snap is 0 2020-06-12T02:38:36.884 INFO:tasks.rados.rados.0.smithi107.stdout:7783: done (1 left) 2020-06-12T02:38:36.884 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:36.884 INFO:tasks.rados.rados.0.smithi107.stdout:7784: seq_num 7783 ranges {0=753664} 2020-06-12T02:38:36.889 INFO:tasks.rados.rados.0.smithi107.stdout:7784: writing smithi10729407-12 from 0 to 753664 tid 1 2020-06-12T02:38:36.889 INFO:tasks.rados.rados.0.smithi107.stdout:7785: delete oid 13 current snap is 0 2020-06-12T02:38:36.889 INFO:tasks.rados.rados.0.smithi107.stdout:7782: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:36.890 INFO:tasks.rados.rados.0.smithi107.stdout:7782: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:36.890 INFO:tasks.rados.rados.0.smithi107.stdout:7782: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:36.890 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1453 (ObjNum 7781 snap 0 seq_num 7781) dirty exists 2020-06-12T02:38:36.891 INFO:tasks.rados.rados.0.smithi107.stdout:7782: left oid 17 (ObjNum 7781 snap 0 seq_num 7781) 2020-06-12T02:38:36.893 INFO:tasks.rados.rados.0.smithi107.stdout:7782: done (2 left) 2020-06-12T02:38:36.893 INFO:tasks.rados.rados.0.smithi107.stdout:7785: done (1 left) 2020-06-12T02:38:36.893 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:36.894 INFO:tasks.rados.rados.0.smithi107.stdout:7786: seq_num 7785 ranges {0=786432} 2020-06-12T02:38:36.898 INFO:tasks.rados.rados.0.smithi107.stdout:7786: writing smithi10729407-18 from 0 to 786432 tid 1 2020-06-12T02:38:36.898 INFO:tasks.rados.rados.0.smithi107.stdout:7787: delete oid 15 current snap is 0 2020-06-12T02:38:36.898 INFO:tasks.rados.rados.0.smithi107.stdout:7784: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:36.903 INFO:tasks.rados.rados.0.smithi107.stdout:7784: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:36.911 INFO:tasks.rados.rados.0.smithi107.stdout:7787: done (2 left) 2020-06-12T02:38:36.912 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:36.912 INFO:tasks.rados.rados.0.smithi107.stdout:7788: seq_num 7787 ranges {0=425984,425984=401408} 2020-06-12T02:38:36.914 INFO:tasks.rados.rados.0.smithi107.stdout:7788: writing smithi10729407-7 from 0 to 425984 tid 1 2020-06-12T02:38:36.914 INFO:tasks.rados.rados.0.smithi107.stdout:7788: writing smithi10729407-7 from 425984 to 827392 tid 2 2020-06-12T02:38:36.914 INFO:tasks.rados.rados.0.smithi107.stdout:7789: delete oid 2 current snap is 0 2020-06-12T02:38:36.915 INFO:tasks.rados.rados.0.smithi107.stdout:7784: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:36.915 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1413 (ObjNum 7783 snap 0 seq_num 7783) dirty exists 2020-06-12T02:38:36.915 INFO:tasks.rados.rados.0.smithi107.stdout:7784: left oid 12 (ObjNum 7783 snap 0 seq_num 7783) 2020-06-12T02:38:36.915 INFO:tasks.rados.rados.0.smithi107.stdout:7786: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:36.915 INFO:tasks.rados.rados.0.smithi107.stdout:7786: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:36.916 INFO:tasks.rados.rados.0.smithi107.stdout:7784: done (3 left) 2020-06-12T02:38:36.916 INFO:tasks.rados.rados.0.smithi107.stdout:7789: done (2 left) 2020-06-12T02:38:36.916 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:38:36.916 INFO:tasks.rados.rados.0.smithi107.stdout:7790: seq_num 7789 ranges {0=458752,458752=401408} 2020-06-12T02:38:36.917 INFO:tasks.rados.rados.0.smithi107.stdout:7790: writing smithi10729407-6 from 0 to 458752 tid 1 2020-06-12T02:38:36.918 INFO:tasks.rados.rados.0.smithi107.stdout:7790: writing smithi10729407-6 from 458752 to 860160 tid 2 2020-06-12T02:38:36.918 INFO:tasks.rados.rados.0.smithi107.stdout:7791: delete oid 20 current snap is 0 2020-06-12T02:38:36.918 INFO:tasks.rados.rados.0.smithi107.stdout:7788: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:36.918 INFO:tasks.rados.rados.0.smithi107.stdout:7786: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:36.918 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1460 (ObjNum 7785 snap 0 seq_num 7785) dirty exists 2020-06-12T02:38:36.918 INFO:tasks.rados.rados.0.smithi107.stdout:7786: left oid 18 (ObjNum 7785 snap 0 seq_num 7785) 2020-06-12T02:38:36.919 INFO:tasks.rados.rados.0.smithi107.stdout:7788: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:36.919 INFO:tasks.rados.rados.0.smithi107.stdout:7788: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:36.920 INFO:tasks.rados.rados.0.smithi107.stdout:7788: finishing write tid 5 to smithi10729407-7 2020-06-12T02:38:36.920 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1432 (ObjNum 7787 snap 0 seq_num 7787) dirty exists 2020-06-12T02:38:36.921 INFO:tasks.rados.rados.0.smithi107.stdout:7788: left oid 7 (ObjNum 7787 snap 0 seq_num 7787) 2020-06-12T02:38:36.922 INFO:tasks.rados.rados.0.smithi107.stdout:7786: done (3 left) 2020-06-12T02:38:36.923 INFO:tasks.rados.rados.0.smithi107.stdout:7788: done (2 left) 2020-06-12T02:38:36.924 INFO:tasks.rados.rados.0.smithi107.stdout:7791: done (1 left) 2020-06-12T02:38:36.924 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:36.924 INFO:tasks.rados.rados.0.smithi107.stdout:7792: seq_num 7791 ranges {0=491520,491520=401408} 2020-06-12T02:38:36.925 INFO:tasks.rados.rados.0.smithi107.stdout:7792: writing smithi10729407-13 from 0 to 491520 tid 1 2020-06-12T02:38:36.929 INFO:tasks.rados.rados.0.smithi107.stdout:7792: writing smithi10729407-13 from 491520 to 892928 tid 2 2020-06-12T02:38:36.929 INFO:tasks.rados.rados.0.smithi107.stdout:7793: delete oid 5 current snap is 0 2020-06-12T02:38:36.929 INFO:tasks.rados.rados.0.smithi107.stdout:7790: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:36.929 INFO:tasks.rados.rados.0.smithi107.stdout:7790: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:36.930 INFO:tasks.rados.rados.0.smithi107.stdout:7790: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:36.930 INFO:tasks.rados.rados.0.smithi107.stdout:7790: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:36.931 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1396 (ObjNum 7789 snap 0 seq_num 7789) dirty exists 2020-06-12T02:38:36.931 INFO:tasks.rados.rados.0.smithi107.stdout:7790: left oid 6 (ObjNum 7789 snap 0 seq_num 7789) 2020-06-12T02:38:36.932 INFO:tasks.rados.rados.0.smithi107.stdout:7790: done (2 left) 2020-06-12T02:38:36.932 INFO:tasks.rados.rados.0.smithi107.stdout:7793: done (1 left) 2020-06-12T02:38:36.932 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:36.932 INFO:tasks.rados.rados.0.smithi107.stdout:7794: seq_num 7793 ranges {720896=524288,1245184=401408} 2020-06-12T02:38:36.939 INFO:tasks.rados.rados.0.smithi107.stdout:7794: writing smithi10729407-17 from 720896 to 1245184 tid 1 2020-06-12T02:38:36.941 INFO:tasks.rados.rados.0.smithi107.stdout:7794: writing smithi10729407-17 from 1245184 to 1646592 tid 2 2020-06-12T02:38:36.942 INFO:tasks.rados.rados.0.smithi107.stdout:7795: delete oid 9 current snap is 0 2020-06-12T02:38:36.942 INFO:tasks.rados.rados.0.smithi107.stdout:7792: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:36.942 INFO:tasks.rados.rados.0.smithi107.stdout:7792: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:36.942 INFO:tasks.rados.rados.0.smithi107.stdout:7792: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:36.943 INFO:tasks.rados.rados.0.smithi107.stdout:7792: finishing write tid 5 to smithi10729407-13 2020-06-12T02:38:36.943 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1435 (ObjNum 7791 snap 0 seq_num 7791) dirty exists 2020-06-12T02:38:36.943 INFO:tasks.rados.rados.0.smithi107.stdout:7792: left oid 13 (ObjNum 7791 snap 0 seq_num 7791) 2020-06-12T02:38:36.947 INFO:tasks.rados.rados.0.smithi107.stdout:7794: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:36.948 INFO:tasks.rados.rados.0.smithi107.stdout:7792: done (2 left) 2020-06-12T02:38:36.948 INFO:tasks.rados.rados.0.smithi107.stdout:7795: done (1 left) 2020-06-12T02:38:36.948 INFO:tasks.rados.rados.0.smithi107.stdout:7796: delete oid 10 current snap is 0 2020-06-12T02:38:36.949 INFO:tasks.rados.rados.0.smithi107.stdout:7794: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:36.949 INFO:tasks.rados.rados.0.smithi107.stdout:7794: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:36.949 INFO:tasks.rados.rados.0.smithi107.stdout:7796: done (1 left) 2020-06-12T02:38:36.950 INFO:tasks.rados.rados.0.smithi107.stdout:7797: delete oid 14 current snap is 0 2020-06-12T02:38:36.950 INFO:tasks.rados.rados.0.smithi107.stdout:7794: finishing write tid 5 to smithi10729407-17 2020-06-12T02:38:36.950 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1456 (ObjNum 7793 snap 0 seq_num 7793) dirty exists 2020-06-12T02:38:36.950 INFO:tasks.rados.rados.0.smithi107.stdout:7794: left oid 17 (ObjNum 7793 snap 0 seq_num 7793) 2020-06-12T02:38:36.952 INFO:tasks.rados.rados.0.smithi107.stdout:7794: done (1 left) 2020-06-12T02:38:36.952 INFO:tasks.rados.rados.0.smithi107.stdout:7797: done (0 left) 2020-06-12T02:38:36.953 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:36.953 INFO:tasks.rados.rados.0.smithi107.stdout:7798: seq_num 7797 ranges {786432=589824,1376256=401408} 2020-06-12T02:38:36.959 INFO:tasks.rados.rados.0.smithi107.stdout:7798: writing smithi10729407-18 from 786432 to 1376256 tid 1 2020-06-12T02:38:36.962 INFO:tasks.rados.rados.0.smithi107.stdout:7798: writing smithi10729407-18 from 1376256 to 1777664 tid 2 2020-06-12T02:38:36.962 INFO:tasks.rados.rados.0.smithi107.stdout:7799: delete oid 19 current snap is 0 2020-06-12T02:38:36.964 INFO:tasks.rados.rados.0.smithi107.stdout:7799: done (1 left) 2020-06-12T02:38:36.965 INFO:tasks.rados.rados.0.smithi107.stdout:7800: delete oid 16 current snap is 0 2020-06-12T02:38:36.967 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:36.964+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:36.969 INFO:tasks.rados.rados.0.smithi107.stdout:7798: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:36.970 INFO:tasks.rados.rados.0.smithi107.stdout:7800: done (1 left) 2020-06-12T02:38:36.970 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:36.970 INFO:tasks.rados.rados.0.smithi107.stdout:7801: seq_num 7800 ranges {0=638976,638976=401408} 2020-06-12T02:38:36.974 INFO:tasks.rados.rados.0.smithi107.stdout:7801: writing smithi10729407-5 from 0 to 638976 tid 1 2020-06-12T02:38:36.976 INFO:tasks.rados.rados.0.smithi107.stdout:7801: writing smithi10729407-5 from 638976 to 1040384 tid 2 2020-06-12T02:38:36.976 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:36.976 INFO:tasks.rados.rados.0.smithi107.stdout:7802: seq_num 7801 ranges {1646592=655360,2301952=401408} 2020-06-12T02:38:36.989 INFO:tasks.rados.rados.0.smithi107.stdout:7802: writing smithi10729407-17 from 1646592 to 2301952 tid 1 2020-06-12T02:38:36.991 INFO:tasks.rados.rados.0.smithi107.stdout:7802: writing smithi10729407-17 from 2301952 to 2703360 tid 2 2020-06-12T02:38:36.992 INFO:tasks.rados.rados.0.smithi107.stdout:7803: delete oid 9 current snap is 0 2020-06-12T02:38:36.992 INFO:tasks.rados.rados.0.smithi107.stdout:7798: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:36.992 INFO:tasks.rados.rados.0.smithi107.stdout:7798: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:36.992 INFO:tasks.rados.rados.0.smithi107.stdout:7798: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:36.993 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1464 (ObjNum 7797 snap 0 seq_num 7797) dirty exists 2020-06-12T02:38:36.993 INFO:tasks.rados.rados.0.smithi107.stdout:7798: left oid 18 (ObjNum 7797 snap 0 seq_num 7797) 2020-06-12T02:38:36.993 INFO:tasks.rados.rados.0.smithi107.stdout:7801: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:36.993 INFO:tasks.rados.rados.0.smithi107.stdout:7801: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:36.993 INFO:tasks.rados.rados.0.smithi107.stdout:7801: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:36.993 INFO:tasks.rados.rados.0.smithi107.stdout:7801: finishing write tid 5 to smithi10729407-5 2020-06-12T02:38:36.994 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 778 (ObjNum 7800 snap 0 seq_num 7800) dirty exists 2020-06-12T02:38:36.994 INFO:tasks.rados.rados.0.smithi107.stdout:7801: left oid 5 (ObjNum 7800 snap 0 seq_num 7800) 2020-06-12T02:38:36.997 INFO:tasks.rados.rados.0.smithi107.stdout:7798: done (3 left) 2020-06-12T02:38:36.998 INFO:tasks.rados.rados.0.smithi107.stdout:7801: done (2 left) 2020-06-12T02:38:36.998 INFO:tasks.rados.rados.0.smithi107.stdout:7803: done (1 left) 2020-06-12T02:38:36.998 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:36.999 INFO:tasks.rados.rados.0.smithi107.stdout:7804: seq_num 7803 ranges {0=688128,688128=401408} 2020-06-12T02:38:37.001 INFO:tasks.rados.rados.0.smithi107.stdout:7804: writing smithi10729407-8 from 0 to 688128 tid 1 2020-06-12T02:38:37.002 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:37.002 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:38:37.002 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2020-06-12T02:38:37.003 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:37.004 INFO:tasks.rados.rados.0.smithi107.stdout:7804: writing smithi10729407-8 from 688128 to 1089536 tid 2 2020-06-12T02:38:37.007 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:37.007 INFO:tasks.rados.rados.0.smithi107.stdout:7805: seq_num 7804 ranges {0=704512,704512=401408} 2020-06-12T02:38:37.009 INFO:tasks.rados.rados.0.smithi107.stdout:7805: writing smithi10729407-14 from 0 to 704512 tid 1 2020-06-12T02:38:37.012 INFO:tasks.rados.rados.0.smithi107.stdout:7805: writing smithi10729407-14 from 704512 to 1105920 tid 2 2020-06-12T02:38:37.013 INFO:tasks.rados.rados.0.smithi107.stdout:7806: delete oid 7 current snap is 0 2020-06-12T02:38:37.013 INFO:tasks.rados.rados.0.smithi107.stdout:7802: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:37.013 INFO:tasks.rados.rados.0.smithi107.stdout:7802: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:37.013 INFO:tasks.rados.rados.0.smithi107.stdout:7802: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:37.014 INFO:tasks.rados.rados.0.smithi107.stdout:7802: finishing write tid 5 to smithi10729407-17 2020-06-12T02:38:37.014 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1459 (ObjNum 7801 snap 0 seq_num 7801) dirty exists 2020-06-12T02:38:37.014 INFO:tasks.rados.rados.0.smithi107.stdout:7802: left oid 17 (ObjNum 7801 snap 0 seq_num 7801) 2020-06-12T02:38:37.014 INFO:tasks.rados.rados.0.smithi107.stdout:7804: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:37.015 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-06-12T02:38:37.017 INFO:tasks.rados.rados.0.smithi107.stdout:7804: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:37.017 INFO:tasks.rados.rados.0.smithi107.stdout:7802: done (3 left) 2020-06-12T02:38:37.017 INFO:tasks.rados.rados.0.smithi107.stdout:7806: done (2 left) 2020-06-12T02:38:37.018 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:37.018 INFO:tasks.rados.rados.0.smithi107.stdout:7807: seq_num 7806 ranges {0=745472,745472=401408} 2020-06-12T02:38:37.021 INFO:tasks.rados.rados.0.smithi107.stdout:7807: writing smithi10729407-1 from 0 to 745472 tid 1 2020-06-12T02:38:37.024 INFO:tasks.rados.rados.0.smithi107.stdout:7807: writing smithi10729407-1 from 745472 to 1146880 tid 2 2020-06-12T02:38:37.024 INFO:tasks.rados.rados.0.smithi107.stdout:7808: delete oid 11 current snap is 0 2020-06-12T02:38:37.025 INFO:tasks.rados.rados.0.smithi107.stdout:7804: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:37.025 INFO:tasks.rados.rados.0.smithi107.stdout:7804: finishing write tid 5 to smithi10729407-8 2020-06-12T02:38:37.025 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1437 (ObjNum 7803 snap 0 seq_num 7803) dirty exists 2020-06-12T02:38:37.025 INFO:tasks.rados.rados.0.smithi107.stdout:7804: left oid 8 (ObjNum 7803 snap 0 seq_num 7803) 2020-06-12T02:38:37.026 INFO:tasks.rados.rados.0.smithi107.stdout:7805: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:37.026 INFO:tasks.rados.rados.0.smithi107.stdout:7805: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:37.026 INFO:tasks.rados.rados.0.smithi107.stdout:7805: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:37.027 INFO:tasks.rados.rados.0.smithi107.stdout:7805: finishing write tid 5 to smithi10729407-14 2020-06-12T02:38:37.027 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1389 (ObjNum 7804 snap 0 seq_num 7804) dirty exists 2020-06-12T02:38:37.027 INFO:tasks.rados.rados.0.smithi107.stdout:7805: left oid 14 (ObjNum 7804 snap 0 seq_num 7804) 2020-06-12T02:38:37.028 INFO:tasks.rados.rados.0.smithi107.stdout:7804: done (3 left) 2020-06-12T02:38:37.029 INFO:tasks.rados.rados.0.smithi107.stdout:7805: done (2 left) 2020-06-12T02:38:37.029 INFO:tasks.rados.rados.0.smithi107.stdout:7808: done (1 left) 2020-06-12T02:38:37.030 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:38:37.030 INFO:tasks.rados.rados.0.smithi107.stdout:7809: seq_num 7808 ranges {0=778240,778240=401408} 2020-06-12T02:38:37.032 INFO:tasks.rados.rados.0.smithi107.stdout:7809: writing smithi10729407-20 from 0 to 778240 tid 1 2020-06-12T02:38:37.035 INFO:tasks.rados.rados.0.smithi107.stdout:7809: writing smithi10729407-20 from 778240 to 1179648 tid 2 2020-06-12T02:38:37.035 INFO:tasks.rados.rados.0.smithi107.stdout:7810: delete oid 8 current snap is 0 2020-06-12T02:38:37.035 INFO:tasks.rados.rados.0.smithi107.stdout:7807: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:37.035 INFO:tasks.rados.rados.0.smithi107.stdout:7807: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:37.036 INFO:tasks.rados.rados.0.smithi107.stdout:7807: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:37.036 INFO:tasks.rados.rados.0.smithi107.stdout:7807: finishing write tid 5 to smithi10729407-1 2020-06-12T02:38:37.036 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1399 (ObjNum 7806 snap 0 seq_num 7806) dirty exists 2020-06-12T02:38:37.037 INFO:tasks.rados.rados.0.smithi107.stdout:7807: left oid 1 (ObjNum 7806 snap 0 seq_num 7806) 2020-06-12T02:38:37.041 INFO:tasks.rados.rados.0.smithi107.stdout:7807: done (2 left) 2020-06-12T02:38:37.042 INFO:tasks.rados.rados.0.smithi107.stdout:7810: done (1 left) 2020-06-12T02:38:37.042 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:37.042 INFO:tasks.rados.rados.0.smithi107.stdout:7811: seq_num 7810 ranges {1040384=409600,1449984=458752,1908736=344064} 2020-06-12T02:38:37.049 INFO:tasks.rados.rados.0.smithi107.stdout:7811: writing smithi10729407-5 from 1040384 to 1449984 tid 1 2020-06-12T02:38:37.052 INFO:tasks.rados.rados.0.smithi107.stdout:7811: writing smithi10729407-5 from 1449984 to 1908736 tid 2 2020-06-12T02:38:37.054 INFO:tasks.rados.rados.0.smithi107.stdout:7811: writing smithi10729407-5 from 1908736 to 2252800 tid 3 2020-06-12T02:38:37.054 INFO:tasks.rados.rados.0.smithi107.stdout:7812: delete oid 6 current snap is 0 2020-06-12T02:38:37.055 INFO:tasks.rados.rados.0.smithi107.stdout:7809: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:37.055 INFO:tasks.rados.rados.0.smithi107.stdout:7809: finishing write tid 2 to smithi10729407-20 2020-06-12T02:38:37.055 INFO:tasks.rados.rados.0.smithi107.stdout:7809: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:37.055 INFO:tasks.rados.rados.0.smithi107.stdout:7809: finishing write tid 5 to smithi10729407-20 2020-06-12T02:38:37.055 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 573 (ObjNum 7808 snap 0 seq_num 7808) dirty exists 2020-06-12T02:38:37.056 INFO:tasks.rados.rados.0.smithi107.stdout:7809: left oid 20 (ObjNum 7808 snap 0 seq_num 7808) 2020-06-12T02:38:37.062 INFO:tasks.rados.rados.0.smithi107.stdout:7809: done (2 left) 2020-06-12T02:38:37.062 INFO:tasks.rados.rados.0.smithi107.stdout:7812: done (1 left) 2020-06-12T02:38:37.062 INFO:tasks.rados.rados.0.smithi107.stdout:7813: delete oid 1 current snap is 0 2020-06-12T02:38:37.063 INFO:tasks.rados.rados.0.smithi107.stdout:7811: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:37.063 INFO:tasks.rados.rados.0.smithi107.stdout:7811: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:37.063 INFO:tasks.rados.rados.0.smithi107.stdout:7813: done (1 left) 2020-06-12T02:38:37.064 INFO:tasks.rados.rados.0.smithi107.stdout:7814: delete oid 1 current snap is 0 2020-06-12T02:38:37.064 INFO:tasks.rados.rados.0.smithi107.stdout:7811: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:37.065 INFO:tasks.rados.rados.0.smithi107.stdout:7811: finishing write tid 5 to smithi10729407-5 2020-06-12T02:38:37.066 INFO:tasks.rados.rados.0.smithi107.stdout:7811: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:37.067 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 782 (ObjNum 7810 snap 0 seq_num 7810) dirty exists 2020-06-12T02:38:37.067 INFO:tasks.rados.rados.0.smithi107.stdout:7811: left oid 5 (ObjNum 7810 snap 0 seq_num 7810) 2020-06-12T02:38:37.069 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:37.066+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:37.070 INFO:tasks.rados.rados.0.smithi107.stdout:7811: done (1 left) 2020-06-12T02:38:37.070 INFO:tasks.rados.rados.0.smithi107.stdout:7814: done (0 left) 2020-06-12T02:38:37.070 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:38:37.071 INFO:tasks.rados.rados.0.smithi107.stdout:7815: seq_num 7814 ranges {0=475136,475136=794624,1269760=8192} 2020-06-12T02:38:37.072 INFO:tasks.rados.rados.0.smithi107.stdout:7815: writing smithi10729407-7 from 0 to 475136 tid 1 2020-06-12T02:38:37.077 INFO:tasks.rados.rados.0.smithi107.stdout:7815: writing smithi10729407-7 from 475136 to 1269760 tid 2 2020-06-12T02:38:37.078 INFO:tasks.rados.rados.0.smithi107.stdout:7815: writing smithi10729407-7 from 1269760 to 1277952 tid 3 2020-06-12T02:38:37.078 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:37.078 INFO:tasks.rados.rados.0.smithi107.stdout:7816: seq_num 7815 ranges {1105920=491520,1597440=679936,2277376=122880} 2020-06-12T02:38:37.086 INFO:tasks.rados.rados.0.smithi107.stdout:7816: writing smithi10729407-14 from 1105920 to 1597440 tid 1 2020-06-12T02:38:37.091 INFO:tasks.rados.rados.0.smithi107.stdout:7816: writing smithi10729407-14 from 1597440 to 2277376 tid 2 2020-06-12T02:38:37.092 INFO:tasks.rados.rados.0.smithi107.stdout:7816: writing smithi10729407-14 from 2277376 to 2400256 tid 3 2020-06-12T02:38:37.092 INFO:tasks.rados.rados.0.smithi107.stdout:7817: delete oid 16 current snap is 0 2020-06-12T02:38:37.092 INFO:tasks.rados.rados.0.smithi107.stdout:7815: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:37.093 INFO:tasks.rados.rados.0.smithi107.stdout:7815: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:37.093 INFO:tasks.rados.rados.0.smithi107.stdout:7815: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:37.093 INFO:tasks.rados.rados.0.smithi107.stdout:7815: finishing write tid 5 to smithi10729407-7 2020-06-12T02:38:37.093 INFO:tasks.rados.rados.0.smithi107.stdout:7815: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:37.093 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1440 (ObjNum 7814 snap 0 seq_num 7814) dirty exists 2020-06-12T02:38:37.094 INFO:tasks.rados.rados.0.smithi107.stdout:7815: left oid 7 (ObjNum 7814 snap 0 seq_num 7814) 2020-06-12T02:38:37.097 INFO:tasks.rados.rados.0.smithi107.stdout:7816: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:37.098 INFO:tasks.rados.rados.0.smithi107.stdout:7815: done (2 left) 2020-06-12T02:38:37.098 INFO:tasks.rados.rados.0.smithi107.stdout:7817: done (1 left) 2020-06-12T02:38:37.098 INFO:tasks.rados.rados.0.smithi107.stdout:7818: delete oid 10 current snap is 0 2020-06-12T02:38:37.099 INFO:tasks.rados.rados.0.smithi107.stdout:7818: done (1 left) 2020-06-12T02:38:37.100 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:37.100 INFO:tasks.rados.rados.0.smithi107.stdout:7819: seq_num 7818 ranges {2703360=540672,3244032=786432,4030464=16384} 2020-06-12T02:38:37.117 INFO:tasks.rados.rados.0.smithi107.stdout:7819: writing smithi10729407-17 from 2703360 to 3244032 tid 1 2020-06-12T02:38:37.122 INFO:tasks.rados.rados.0.smithi107.stdout:7819: writing smithi10729407-17 from 3244032 to 4030464 tid 2 2020-06-12T02:38:37.122 INFO:tasks.rados.rados.0.smithi107.stdout:7819: writing smithi10729407-17 from 4030464 to 4046848 tid 3 2020-06-12T02:38:37.122 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:37.123 INFO:tasks.rados.rados.0.smithi107.stdout:7820: seq_num 7819 ranges {0=557056,557056=671744,1228800=131072} 2020-06-12T02:38:37.125 INFO:tasks.rados.rados.0.smithi107.stdout:7820: writing smithi10729407-15 from 0 to 557056 tid 1 2020-06-12T02:38:37.129 INFO:tasks.rados.rados.0.smithi107.stdout:7820: writing smithi10729407-15 from 557056 to 1228800 tid 2 2020-06-12T02:38:37.130 INFO:tasks.rados.rados.0.smithi107.stdout:7820: writing smithi10729407-15 from 1228800 to 1359872 tid 3 2020-06-12T02:38:37.130 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:37.131 INFO:tasks.rados.rados.0.smithi107.stdout:7821: seq_num 7820 ranges {0=573440,573440=548864,1122304=253952} 2020-06-12T02:38:37.133 INFO:tasks.rados.rados.0.smithi107.stdout:7821: writing smithi10729407-4 from 0 to 573440 tid 1 2020-06-12T02:38:37.136 INFO:tasks.rados.rados.0.smithi107.stdout:7821: writing smithi10729407-4 from 573440 to 1122304 tid 2 2020-06-12T02:38:37.138 INFO:tasks.rados.rados.0.smithi107.stdout:7821: writing smithi10729407-4 from 1122304 to 1376256 tid 3 2020-06-12T02:38:37.138 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:37.139 INFO:tasks.rados.rados.0.smithi107.stdout:7822: seq_num 7821 ranges {1777664=589824,2367488=434176,2801664=368640} 2020-06-12T02:38:37.151 INFO:tasks.rados.rados.0.smithi107.stdout:7822: writing smithi10729407-18 from 1777664 to 2367488 tid 1 2020-06-12T02:38:37.154 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:37.154 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:38:37.155 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2020-06-12T02:38:37.155 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2020-06-12T02:38:37.155 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:37.155 INFO:tasks.rados.rados.0.smithi107.stdout:7822: writing smithi10729407-18 from 2367488 to 2801664 tid 2 2020-06-12T02:38:37.157 INFO:tasks.rados.rados.0.smithi107.stdout:7822: writing smithi10729407-18 from 2801664 to 3170304 tid 3 2020-06-12T02:38:37.157 INFO:tasks.rados.rados.0.smithi107.stdout:7823: delete oid 3 current snap is 0 2020-06-12T02:38:37.158 INFO:tasks.rados.rados.0.smithi107.stdout:7816: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:37.158 INFO:tasks.rados.rados.0.smithi107.stdout:7816: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:37.158 INFO:tasks.rados.rados.0.smithi107.stdout:7816: finishing write tid 5 to smithi10729407-14 2020-06-12T02:38:37.158 INFO:tasks.rados.rados.0.smithi107.stdout:7816: finishing write tid 6 to smithi10729407-14 2020-06-12T02:38:37.158 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1394 (ObjNum 7815 snap 0 seq_num 7815) dirty exists 2020-06-12T02:38:37.160 INFO:tasks.rados.rados.0.smithi107.stdout:7816: left oid 14 (ObjNum 7815 snap 0 seq_num 7815) 2020-06-12T02:38:37.160 INFO:tasks.rados.rados.0.smithi107.stdout:7819: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:37.160 INFO:tasks.rados.rados.0.smithi107.stdout:7820: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:37.161 INFO:tasks.rados.rados.0.smithi107.stdout:7819: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:37.161 INFO:tasks.rados.rados.0.smithi107.stdout:7819: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:37.161 INFO:tasks.rados.rados.0.smithi107.stdout:7819: finishing write tid 5 to smithi10729407-17 2020-06-12T02:38:37.163 INFO:tasks.rados.rados.0.smithi107.stdout:7820: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:37.163 INFO:tasks.rados.rados.0.smithi107.stdout:7820: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:37.163 INFO:tasks.rados.rados.0.smithi107.stdout:7820: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:37.163 INFO:tasks.rados.rados.0.smithi107.stdout:7821: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:37.163 INFO:tasks.rados.rados.0.smithi107.stdout:7819: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:37.164 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1463 (ObjNum 7818 snap 0 seq_num 7818) dirty exists 2020-06-12T02:38:37.164 INFO:tasks.rados.rados.0.smithi107.stdout:7819: left oid 17 (ObjNum 7818 snap 0 seq_num 7818) 2020-06-12T02:38:37.164 INFO:tasks.rados.rados.0.smithi107.stdout:7820: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:37.164 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1442 (ObjNum 7819 snap 0 seq_num 7819) dirty exists 2020-06-12T02:38:37.164 INFO:tasks.rados.rados.0.smithi107.stdout:7820: left oid 15 (ObjNum 7819 snap 0 seq_num 7819) 2020-06-12T02:38:37.165 INFO:tasks.rados.rados.0.smithi107.stdout:7821: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:37.165 INFO:tasks.rados.rados.0.smithi107.stdout:7821: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:37.165 INFO:tasks.rados.rados.0.smithi107.stdout:7821: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:37.165 INFO:tasks.rados.rados.0.smithi107.stdout:7821: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:37.166 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1467 (ObjNum 7820 snap 0 seq_num 7820) dirty exists 2020-06-12T02:38:37.166 INFO:tasks.rados.rados.0.smithi107.stdout:7821: left oid 4 (ObjNum 7820 snap 0 seq_num 7820) 2020-06-12T02:38:37.167 INFO:tasks.rados.rados.0.smithi107.stdout:7822: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:37.168 INFO:tasks.rados.rados.0.smithi107.stdout:7816: done (5 left) 2020-06-12T02:38:37.168 INFO:tasks.rados.rados.0.smithi107.stdout:7819: done (4 left) 2020-06-12T02:38:37.168 INFO:tasks.rados.rados.0.smithi107.stdout:7820: done (3 left) 2020-06-12T02:38:37.168 INFO:tasks.rados.rados.0.smithi107.stdout:7821: done (2 left) 2020-06-12T02:38:37.168 INFO:tasks.rados.rados.0.smithi107.stdout:7823: done (1 left) 2020-06-12T02:38:37.169 INFO:tasks.rados.rados.0.smithi107.stdout:7824: delete oid 5 current snap is 0 2020-06-12T02:38:37.169 INFO:tasks.rados.rados.0.smithi107.stdout:7822: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:37.169 INFO:tasks.rados.rados.0.smithi107.stdout:7824: done (1 left) 2020-06-12T02:38:37.169 INFO:tasks.rados.rados.0.smithi107.stdout:7825: delete oid 19 current snap is 0 2020-06-12T02:38:37.170 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-06-12T02:38:37.171 INFO:tasks.rados.rados.0.smithi107.stdout:7822: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:37.172 INFO:tasks.rados.rados.0.smithi107.stdout:7822: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:37.172 INFO:tasks.rados.rados.0.smithi107.stdout:7825: done (1 left) 2020-06-12T02:38:37.172 INFO:tasks.rados.rados.0.smithi107.stdout:7826: delete oid 10 current snap is 0 2020-06-12T02:38:37.172 INFO:tasks.rados.rados.0.smithi107.stdout:7822: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:37.172 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1469 (ObjNum 7821 snap 0 seq_num 7821) dirty exists 2020-06-12T02:38:37.173 INFO:tasks.rados.rados.0.smithi107.stdout:7822: left oid 18 (ObjNum 7821 snap 0 seq_num 7821) 2020-06-12T02:38:37.173 INFO:tasks.rados.rados.0.smithi107.stdout:7822: done (1 left) 2020-06-12T02:38:37.173 INFO:tasks.rados.rados.0.smithi107.stdout:7826: done (0 left) 2020-06-12T02:38:37.174 INFO:tasks.rados.rados.0.smithi107.stdout:7827: delete oid 20 current snap is 0 2020-06-12T02:38:37.174 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:37.168+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:37.175 INFO:tasks.rados.rados.0.smithi107.stdout:7827: done (0 left) 2020-06-12T02:38:37.175 INFO:tasks.rados.rados.0.smithi107.stdout:7828: delete oid 4 current snap is 0 2020-06-12T02:38:37.176 INFO:tasks.rados.rados.0.smithi107.stdout:7828: done (0 left) 2020-06-12T02:38:37.176 INFO:tasks.rados.rados.0.smithi107.stdout:7829: delete oid 19 current snap is 0 2020-06-12T02:38:37.183 INFO:tasks.rados.rados.0.smithi107.stdout:7829: done (0 left) 2020-06-12T02:38:37.183 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:37.184 INFO:tasks.rados.rados.0.smithi107.stdout:7830: seq_num 7829 ranges {0=729088,729088=761856,1490944=40960} 2020-06-12T02:38:37.185 INFO:tasks.rados.rados.0.smithi107.stdout:7830: writing smithi10729407-16 from 0 to 729088 tid 1 2020-06-12T02:38:37.187 INFO:tasks.rados.rados.0.smithi107.stdout:7830: writing smithi10729407-16 from 729088 to 1490944 tid 2 2020-06-12T02:38:37.188 INFO:tasks.rados.rados.0.smithi107.stdout:7830: writing smithi10729407-16 from 1490944 to 1531904 tid 3 2020-06-12T02:38:37.188 INFO:tasks.rados.rados.0.smithi107.stdout:7831: delete oid 14 current snap is 0 2020-06-12T02:38:37.194 INFO:tasks.rados.rados.0.smithi107.stdout:7830: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:37.194 INFO:tasks.rados.rados.0.smithi107.stdout:7831: done (1 left) 2020-06-12T02:38:37.194 INFO:tasks.rados.rados.0.smithi107.stdout:7832: delete oid 2 current snap is 0 2020-06-12T02:38:37.198 INFO:tasks.rados.rados.0.smithi107.stdout:7832: done (1 left) 2020-06-12T02:38:37.198 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:38:37.198 INFO:tasks.rados.rados.0.smithi107.stdout:7833: seq_num 7832 ranges {0=778240,778240=466944,1245184=335872} 2020-06-12T02:38:37.203 INFO:tasks.rados.rados.0.smithi107.stdout:7833: writing smithi10729407-10 from 0 to 778240 tid 1 2020-06-12T02:38:37.206 INFO:tasks.rados.rados.0.smithi107.stdout:7833: writing smithi10729407-10 from 778240 to 1245184 tid 2 2020-06-12T02:38:37.208 INFO:tasks.rados.rados.0.smithi107.stdout:7833: writing smithi10729407-10 from 1245184 to 1581056 tid 3 2020-06-12T02:38:37.208 INFO:tasks.rados.rados.0.smithi107.stdout:7834: delete oid 3 current snap is 0 2020-06-12T02:38:37.209 INFO:tasks.rados.rados.0.smithi107.stdout:7830: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:37.209 INFO:tasks.rados.rados.0.smithi107.stdout:7830: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:37.209 INFO:tasks.rados.rados.0.smithi107.stdout:7830: finishing write tid 5 to smithi10729407-16 2020-06-12T02:38:37.209 INFO:tasks.rados.rados.0.smithi107.stdout:7830: finishing write tid 6 to smithi10729407-16 2020-06-12T02:38:37.210 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1422 (ObjNum 7829 snap 0 seq_num 7829) dirty exists 2020-06-12T02:38:37.210 INFO:tasks.rados.rados.0.smithi107.stdout:7830: left oid 16 (ObjNum 7829 snap 0 seq_num 7829) 2020-06-12T02:38:37.213 INFO:tasks.rados.rados.0.smithi107.stdout:7830: done (2 left) 2020-06-12T02:38:37.213 INFO:tasks.rados.rados.0.smithi107.stdout:7834: done (1 left) 2020-06-12T02:38:37.214 INFO:tasks.rados.rados.0.smithi107.stdout:7835: delete oid 19 current snap is 0 2020-06-12T02:38:37.216 INFO:tasks.rados.rados.0.smithi107.stdout:7833: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:37.217 INFO:tasks.rados.rados.0.smithi107.stdout:7833: finishing write tid 2 to smithi10729407-10 2020-06-12T02:38:37.217 INFO:tasks.rados.rados.0.smithi107.stdout:7833: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:37.219 INFO:tasks.rados.rados.0.smithi107.stdout:7835: done (1 left) 2020-06-12T02:38:37.219 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:37.219 INFO:tasks.rados.rados.0.smithi107.stdout:7836: seq_num 7835 ranges {0=425984,425984=516096,942080=475136,1417216=212992} 2020-06-12T02:38:37.220 INFO:tasks.rados.rados.0.smithi107.stdout:7836: writing smithi10729407-19 from 0 to 425984 tid 1 2020-06-12T02:38:37.223 INFO:tasks.rados.rados.0.smithi107.stdout:7836: writing smithi10729407-19 from 425984 to 942080 tid 2 2020-06-12T02:38:37.226 INFO:tasks.rados.rados.0.smithi107.stdout:7836: writing smithi10729407-19 from 942080 to 1417216 tid 3 2020-06-12T02:38:37.228 INFO:tasks.rados.rados.0.smithi107.stdout:7836: writing smithi10729407-19 from 1417216 to 1630208 tid 4 2020-06-12T02:38:37.228 INFO:tasks.rados.rados.0.smithi107.stdout:7837: delete oid 15 current snap is 0 2020-06-12T02:38:37.228 INFO:tasks.rados.rados.0.smithi107.stdout:7833: finishing write tid 5 to smithi10729407-10 2020-06-12T02:38:37.229 INFO:tasks.rados.rados.0.smithi107.stdout:7833: finishing write tid 6 to smithi10729407-10 2020-06-12T02:38:37.229 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 536 (ObjNum 7832 snap 0 seq_num 7832) dirty exists 2020-06-12T02:38:37.229 INFO:tasks.rados.rados.0.smithi107.stdout:7833: left oid 10 (ObjNum 7832 snap 0 seq_num 7832) 2020-06-12T02:38:37.229 INFO:tasks.rados.rados.0.smithi107.stdout:7836: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:37.231 INFO:tasks.rados.rados.0.smithi107.stdout:7833: done (2 left) 2020-06-12T02:38:37.231 INFO:tasks.rados.rados.0.smithi107.stdout:7837: done (1 left) 2020-06-12T02:38:37.231 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:37.232 INFO:tasks.rados.rados.0.smithi107.stdout:7838: seq_num 7837 ranges {0=458752,458752=688128,1146880=442368,1589248=73728} 2020-06-12T02:38:37.234 INFO:tasks.rados.rados.0.smithi107.stdout:7838: writing smithi10729407-1 from 0 to 458752 tid 1 2020-06-12T02:38:37.238 INFO:tasks.rados.rados.0.smithi107.stdout:7838: writing smithi10729407-1 from 458752 to 1146880 tid 2 2020-06-12T02:38:37.241 INFO:tasks.rados.rados.0.smithi107.stdout:7838: writing smithi10729407-1 from 1146880 to 1589248 tid 3 2020-06-12T02:38:37.242 INFO:tasks.rados.rados.0.smithi107.stdout:7838: writing smithi10729407-1 from 1589248 to 1662976 tid 4 2020-06-12T02:38:37.242 INFO:tasks.rados.rados.0.smithi107.stdout:7839: delete oid 14 current snap is 0 2020-06-12T02:38:37.242 INFO:tasks.rados.rados.0.smithi107.stdout:7836: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:37.242 INFO:tasks.rados.rados.0.smithi107.stdout:7836: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:37.243 INFO:tasks.rados.rados.0.smithi107.stdout:7836: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:37.243 INFO:tasks.rados.rados.0.smithi107.stdout:7836: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:37.243 INFO:tasks.rados.rados.0.smithi107.stdout:7838: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:37.243 INFO:tasks.rados.rados.0.smithi107.stdout:7836: finishing write tid 7 to smithi10729407-19 2020-06-12T02:38:37.244 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1422 (ObjNum 7835 snap 0 seq_num 7835) dirty exists 2020-06-12T02:38:37.244 INFO:tasks.rados.rados.0.smithi107.stdout:7836: left oid 19 (ObjNum 7835 snap 0 seq_num 7835) 2020-06-12T02:38:37.246 INFO:tasks.rados.rados.0.smithi107.stdout:7836: done (2 left) 2020-06-12T02:38:37.246 INFO:tasks.rados.rados.0.smithi107.stdout:7839: done (1 left) 2020-06-12T02:38:37.246 INFO:tasks.rados.rados.0.smithi107.stdout:7840: delete oid 3 current snap is 0 2020-06-12T02:38:37.249 INFO:tasks.rados.rados.0.smithi107.stdout:7840: done (1 left) 2020-06-12T02:38:37.249 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:38:37.250 INFO:tasks.rados.rados.0.smithi107.stdout:7841: seq_num 7840 ranges {1581056=507904,2088960=786432,2875392=417792} 2020-06-12T02:38:37.260 INFO:tasks.rados.rados.0.smithi107.stdout:7841: writing smithi10729407-10 from 1581056 to 2088960 tid 1 2020-06-12T02:38:37.268 INFO:tasks.rados.rados.0.smithi107.stdout:7841: writing smithi10729407-10 from 2088960 to 2875392 tid 2 2020-06-12T02:38:37.268 INFO:tasks.rados.rados.0.smithi107.stdout:7841: writing smithi10729407-10 from 2875392 to 3293184 tid 3 2020-06-12T02:38:37.268 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:37.268 INFO:tasks.rados.rados.0.smithi107.stdout:7842: seq_num 7841 ranges {0=524288,524288=671744,1196032=417792,1613824=114688} 2020-06-12T02:38:37.271 INFO:tasks.rados.rados.0.smithi107.stdout:7842: writing smithi10729407-8 from 0 to 524288 tid 1 2020-06-12T02:38:37.272 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:37.269+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:37.276 INFO:tasks.rados.rados.0.smithi107.stdout:7842: writing smithi10729407-8 from 524288 to 1196032 tid 2 2020-06-12T02:38:37.278 INFO:tasks.rados.rados.0.smithi107.stdout:7842: writing smithi10729407-8 from 1196032 to 1613824 tid 3 2020-06-12T02:38:37.281 INFO:tasks.rados.rados.0.smithi107.stdout:7842: writing smithi10729407-8 from 1613824 to 1728512 tid 4 2020-06-12T02:38:37.281 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:37.281 INFO:tasks.rados.rados.0.smithi107.stdout:7843: seq_num 7842 ranges {0=540672,540672=557056,1097728=630784,1728512=16384} 2020-06-12T02:38:37.282 INFO:tasks.rados.rados.0.smithi107.stdout:7843: writing smithi10729407-15 from 0 to 540672 tid 1 2020-06-12T02:38:37.285 INFO:tasks.rados.rados.0.smithi107.stdout:7843: writing smithi10729407-15 from 540672 to 1097728 tid 2 2020-06-12T02:38:37.289 INFO:tasks.rados.rados.0.smithi107.stdout:7843: writing smithi10729407-15 from 1097728 to 1728512 tid 3 2020-06-12T02:38:37.289 INFO:tasks.rados.rados.0.smithi107.stdout:7843: writing smithi10729407-15 from 1728512 to 1744896 tid 4 2020-06-12T02:38:37.290 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:38:37.290 INFO:tasks.rados.rados.0.smithi107.stdout:7844: seq_num 7843 ranges {0=557056,557056=442368,999424=434176,1433600=327680} 2020-06-12T02:38:37.292 INFO:tasks.rados.rados.0.smithi107.stdout:7844: writing smithi10729407-2 from 0 to 557056 tid 1 2020-06-12T02:38:37.296 INFO:tasks.rados.rados.0.smithi107.stdout:7844: writing smithi10729407-2 from 557056 to 999424 tid 2 2020-06-12T02:38:37.298 INFO:tasks.rados.rados.0.smithi107.stdout:7844: writing smithi10729407-2 from 999424 to 1433600 tid 3 2020-06-12T02:38:37.300 INFO:tasks.rados.rados.0.smithi107.stdout:7844: writing smithi10729407-2 from 1433600 to 1761280 tid 4 2020-06-12T02:38:37.300 INFO:tasks.rados.rados.0.smithi107.stdout:7845: delete oid 13 current snap is 0 2020-06-12T02:38:37.301 INFO:tasks.rados.rados.0.smithi107.stdout:7838: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:37.301 INFO:tasks.rados.rados.0.smithi107.stdout:7838: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:37.301 INFO:tasks.rados.rados.0.smithi107.stdout:7838: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:37.301 INFO:tasks.rados.rados.0.smithi107.stdout:7838: finishing write tid 6 to smithi10729407-1 2020-06-12T02:38:37.302 INFO:tasks.rados.rados.0.smithi107.stdout:7838: finishing write tid 7 to smithi10729407-1 2020-06-12T02:38:37.302 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1407 (ObjNum 7837 snap 0 seq_num 7837) dirty exists 2020-06-12T02:38:37.302 INFO:tasks.rados.rados.0.smithi107.stdout:7838: left oid 1 (ObjNum 7837 snap 0 seq_num 7837) 2020-06-12T02:38:37.302 INFO:tasks.rados.rados.0.smithi107.stdout:7841: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:37.302 INFO:tasks.rados.rados.0.smithi107.stdout:7841: finishing write tid 2 to smithi10729407-10 2020-06-12T02:38:37.319 INFO:tasks.rados.rados.0.smithi107.stdout:7841: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:37.319 INFO:tasks.rados.rados.0.smithi107.stdout:7841: finishing write tid 5 to smithi10729407-10 2020-06-12T02:38:37.320 INFO:tasks.rados.rados.0.smithi107.stdout:7841: finishing write tid 6 to smithi10729407-10 2020-06-12T02:38:37.320 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 540 (ObjNum 7840 snap 0 seq_num 7840) dirty exists 2020-06-12T02:38:37.321 INFO:tasks.rados.rados.0.smithi107.stdout:7841: left oid 10 (ObjNum 7840 snap 0 seq_num 7840) 2020-06-12T02:38:37.321 INFO:tasks.rados.rados.0.smithi107.stdout:7842: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:37.321 INFO:tasks.rados.rados.0.smithi107.stdout:7842: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:37.321 INFO:tasks.rados.rados.0.smithi107.stdout:7842: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:37.322 INFO:tasks.rados.rados.0.smithi107.stdout:7842: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:37.323 INFO:tasks.rados.rados.0.smithi107.stdout:7842: finishing write tid 6 to smithi10729407-8 2020-06-12T02:38:37.323 INFO:tasks.rados.rados.0.smithi107.stdout:7843: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:37.323 INFO:tasks.rados.rados.0.smithi107.stdout:7842: finishing write tid 7 to smithi10729407-8 2020-06-12T02:38:37.323 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1448 (ObjNum 7841 snap 0 seq_num 7841) dirty exists 2020-06-12T02:38:37.324 INFO:tasks.rados.rados.0.smithi107.stdout:7842: left oid 8 (ObjNum 7841 snap 0 seq_num 7841) 2020-06-12T02:38:37.330 INFO:tasks.rados.rados.0.smithi107.stdout:7844: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:37.333 INFO:tasks.rados.rados.0.smithi107.stdout:7838: done (5 left) 2020-06-12T02:38:37.333 INFO:tasks.rados.rados.0.smithi107.stdout:7841: done (4 left) 2020-06-12T02:38:37.334 INFO:tasks.rados.rados.0.smithi107.stdout:7842: done (3 left) 2020-06-12T02:38:37.334 INFO:tasks.rados.rados.0.smithi107.stdout:7845: done (2 left) 2020-06-12T02:38:37.334 INFO:tasks.rados.rados.0.smithi107.stdout:7846: delete oid 1 current snap is 0 2020-06-12T02:38:37.334 INFO:tasks.rados.rados.0.smithi107.stdout:7843: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:37.334 INFO:tasks.rados.rados.0.smithi107.stdout:7844: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:37.335 INFO:tasks.rados.rados.0.smithi107.stdout:7844: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:37.335 INFO:tasks.rados.rados.0.smithi107.stdout:7844: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:37.335 INFO:tasks.rados.rados.0.smithi107.stdout:7844: finishing write tid 6 to smithi10729407-2 2020-06-12T02:38:37.335 INFO:tasks.rados.rados.0.smithi107.stdout:7843: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:37.335 INFO:tasks.rados.rados.0.smithi107.stdout:7843: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:37.336 INFO:tasks.rados.rados.0.smithi107.stdout:7843: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:37.336 INFO:tasks.rados.rados.0.smithi107.stdout:7846: done (2 left) 2020-06-12T02:38:37.336 INFO:tasks.rados.rados.0.smithi107.stdout:7847: delete oid 14 current snap is 0 2020-06-12T02:38:37.336 INFO:tasks.rados.rados.0.smithi107.stdout:7844: finishing write tid 7 to smithi10729407-2 2020-06-12T02:38:37.336 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 734 (ObjNum 7843 snap 0 seq_num 7843) dirty exists 2020-06-12T02:38:37.337 INFO:tasks.rados.rados.0.smithi107.stdout:7844: left oid 2 (ObjNum 7843 snap 0 seq_num 7843) 2020-06-12T02:38:37.337 INFO:tasks.rados.rados.0.smithi107.stdout:7844: done (2 left) 2020-06-12T02:38:37.337 INFO:tasks.rados.rados.0.smithi107.stdout:7847: done (1 left) 2020-06-12T02:38:37.337 INFO:tasks.rados.rados.0.smithi107.stdout:7848: delete oid 5 current snap is 0 2020-06-12T02:38:37.337 INFO:tasks.rados.rados.0.smithi107.stdout:7843: finishing write tid 7 to smithi10729407-15 2020-06-12T02:38:37.338 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1453 (ObjNum 7842 snap 0 seq_num 7842) dirty exists 2020-06-12T02:38:37.338 INFO:tasks.rados.rados.0.smithi107.stdout:7843: left oid 15 (ObjNum 7842 snap 0 seq_num 7842) 2020-06-12T02:38:37.338 INFO:tasks.rados.rados.0.smithi107.stdout:7843: done (1 left) 2020-06-12T02:38:37.338 INFO:tasks.rados.rados.0.smithi107.stdout:7848: done (0 left) 2020-06-12T02:38:37.338 INFO:tasks.rados.rados.0.smithi107.stdout:7849: delete oid 19 current snap is 0 2020-06-12T02:38:37.339 INFO:tasks.rados.rados.0.smithi107.stdout:7849: done (0 left) 2020-06-12T02:38:37.339 INFO:tasks.rados.rados.0.smithi107.stdout:7850: delete oid 11 current snap is 0 2020-06-12T02:38:37.339 INFO:tasks.rados.rados.0.smithi107.stdout:7850: done (0 left) 2020-06-12T02:38:37.339 INFO:tasks.rados.rados.0.smithi107.stdout:7851: delete oid 7 current snap is 0 2020-06-12T02:38:37.339 INFO:tasks.rados.rados.0.smithi107.stdout:7851: done (0 left) 2020-06-12T02:38:37.340 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:37.340 INFO:tasks.rados.rados.0.smithi107.stdout:7852: seq_num 7851 ranges {1744896=696320,2441216=770048,3211264=434176} 2020-06-12T02:38:37.340 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:37.340 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2020-06-12T02:38:37.341 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2020-06-12T02:38:37.341 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2020-06-12T02:38:37.341 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.341 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:2782 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~696320] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:37.341 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:36.808170+0000", 2020-06-12T02:38:37.342 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 60.516015873999997, 2020-06-12T02:38:37.342 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016972839999999999, 2020-06-12T02:38:37.342 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:37.342 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:37.342 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:37.343 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:37.343 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:37.343 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2782 2020-06-12T02:38:37.343 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.343 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:37.344 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.344 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:37.344 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.808170+0000", 2020-06-12T02:38:37.344 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:37.344 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.345 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.345 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:37.345 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.808170+0000", 2020-06-12T02:38:37.345 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000261778 2020-06-12T02:38:37.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.346 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.346 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:37.346 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.808432+0000", 2020-06-12T02:38:37.346 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0920000000000003e-06 2020-06-12T02:38:37.346 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.347 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.347 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:37.347 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.808438+0000", 2020-06-12T02:38:37.347 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2539e-05 2020-06-12T02:38:37.347 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.348 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.348 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:37.348 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.808450+0000", 2020-06-12T02:38:37.348 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1859999999999997e-06 2020-06-12T02:38:37.348 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.349 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.349 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:37.349 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.808457+0000", 2020-06-12T02:38:37.349 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0988000000000001e-05 2020-06-12T02:38:37.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.350 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.350 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:37.350 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.808498+0000", 2020-06-12T02:38:37.350 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011723 2020-06-12T02:38:37.350 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.351 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.351 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:37.351 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.808615+0000", 2020-06-12T02:38:37.351 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0015648109999999999 2020-06-12T02:38:37.351 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.352 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.352 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:37.352 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.810180+0000", 2020-06-12T02:38:37.352 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1718999999999994e-05 2020-06-12T02:38:37.352 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.353 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.353 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:37.353 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.810271+0000", 2020-06-12T02:38:37.353 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00082076299999999996 2020-06-12T02:38:37.353 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.354 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.354 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:37.354 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.811092+0000", 2020-06-12T02:38:37.354 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01210689 2020-06-12T02:38:37.354 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.355 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.355 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:37.355 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.823199+0000", 2020-06-12T02:38:37.355 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017554300000000001 2020-06-12T02:38:37.355 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.356 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.356 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:37.356 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.824954+0000", 2020-06-12T02:38:37.356 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000188414 2020-06-12T02:38:37.356 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.357 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.357 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:37.357 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.825143+0000", 2020-06-12T02:38:37.357 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016972839999999999 2020-06-12T02:38:37.357 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:37.358 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:37.358 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:37.358 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.358 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.358 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:2785 3.0s0 3:0dddb690:::smithi10729407-10:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:37.359 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:36.813804+0000", 2020-06-12T02:38:37.359 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 60.510381604000003, 2020-06-12T02:38:37.359 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014100781999999999, 2020-06-12T02:38:37.359 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:37.359 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:37.360 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:37.360 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:37.360 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:37.360 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2785 2020-06-12T02:38:37.360 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.361 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:37.361 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.361 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:37.361 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813804+0000", 2020-06-12T02:38:37.362 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:37.362 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.362 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.362 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:37.362 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813804+0000", 2020-06-12T02:38:37.363 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1301999999999999e-05 2020-06-12T02:38:37.363 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.363 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.363 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:37.363 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813846+0000", 2020-06-12T02:38:37.364 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4530000000000004e-06 2020-06-12T02:38:37.364 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.364 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.364 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:37.364 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813850+0000", 2020-06-12T02:38:37.365 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7940000000000003e-06 2020-06-12T02:38:37.365 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.365 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.365 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:37.365 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813858+0000", 2020-06-12T02:38:37.366 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9450000000000002e-06 2020-06-12T02:38:37.366 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.366 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.366 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:37.366 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813861+0000", 2020-06-12T02:38:37.367 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0084749969999999997 2020-06-12T02:38:37.367 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.367 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.367 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:37.367 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.822336+0000", 2020-06-12T02:38:37.368 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013055400000000001 2020-06-12T02:38:37.368 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.368 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.368 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:37.368 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.822466+0000", 2020-06-12T02:38:37.369 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047301400000000001 2020-06-12T02:38:37.369 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.369 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.369 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:37.369 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.822939+0000", 2020-06-12T02:38:37.370 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086569499999999998 2020-06-12T02:38:37.370 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.370 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.370 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:37.370 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.823805+0000", 2020-06-12T02:38:37.371 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0039911460000000001 2020-06-12T02:38:37.371 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.371 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.371 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:37.371 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.827796+0000", 2020-06-12T02:38:37.372 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010888200000000001 2020-06-12T02:38:37.372 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.372 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.372 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:37.372 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.827905+0000", 2020-06-12T02:38:37.373 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014100781999999999 2020-06-12T02:38:37.373 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:37.373 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:37.373 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:37.373 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.374 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.374 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:2786 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:38:37.374 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:36.813885+0000", 2020-06-12T02:38:37.374 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 60.510300786999998, 2020-06-12T02:38:37.374 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015757724000000001, 2020-06-12T02:38:37.375 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:37.375 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:37.375 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:37.375 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:37.375 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:37.375 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2786 2020-06-12T02:38:37.376 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.376 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:37.376 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.376 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:37.376 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813885+0000", 2020-06-12T02:38:37.377 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:37.377 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.377 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.377 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:37.377 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813885+0000", 2020-06-12T02:38:37.378 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2388999999999999e-05 2020-06-12T02:38:37.378 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.379 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.379 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:37.380 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813907+0000", 2020-06-12T02:38:37.380 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9660000000000001e-06 2020-06-12T02:38:37.380 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.380 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.380 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:37.381 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813910+0000", 2020-06-12T02:38:37.381 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7649999999999993e-06 2020-06-12T02:38:37.381 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.381 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.381 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:37.382 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813918+0000", 2020-06-12T02:38:37.382 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0870000000000001e-06 2020-06-12T02:38:37.382 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.382 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.382 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:37.383 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813920+0000", 2020-06-12T02:38:37.383 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093684819999999992 2020-06-12T02:38:37.383 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.383 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.383 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:37.384 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.823289+0000", 2020-06-12T02:38:37.384 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103643 2020-06-12T02:38:37.384 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.384 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.384 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:37.385 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.823392+0000", 2020-06-12T02:38:37.385 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0050514449999999999 2020-06-12T02:38:37.385 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.385 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.385 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:37.386 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.828444+0000", 2020-06-12T02:38:37.386 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000102824 2020-06-12T02:38:37.386 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.386 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.386 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:37.387 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.828547+0000", 2020-06-12T02:38:37.387 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0010961230000000001 2020-06-12T02:38:37.387 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.387 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.387 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:37.388 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.829643+0000", 2020-06-12T02:38:37.388 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015757724000000001 2020-06-12T02:38:37.388 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:37.388 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:37.388 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:37.389 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.389 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.389 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:3330 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~630784] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:37.389 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:38.603794+0000", 2020-06-12T02:38:37.389 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 58.720392095999998, 2020-06-12T02:38:37.390 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016093917999999999, 2020-06-12T02:38:37.390 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:37.390 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:37.390 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:37.390 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:37.391 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:37.391 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 3330 2020-06-12T02:38:37.391 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.391 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:37.391 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.392 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:37.392 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.603794+0000", 2020-06-12T02:38:37.392 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:37.392 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.392 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.393 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:37.393 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.603794+0000", 2020-06-12T02:38:37.393 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00042303700000000002 2020-06-12T02:38:37.393 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.393 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.394 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:37.394 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.604217+0000", 2020-06-12T02:38:37.394 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9350000000000006e-06 2020-06-12T02:38:37.394 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.394 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.395 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:37.395 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.604227+0000", 2020-06-12T02:38:37.395 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9343000000000001e-05 2020-06-12T02:38:37.395 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.395 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.396 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:37.396 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.604246+0000", 2020-06-12T02:38:37.396 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2369999999999998e-06 2020-06-12T02:38:37.397 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.397 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.397 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:37.397 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.604255+0000", 2020-06-12T02:38:37.397 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1530000000000003e-05 2020-06-12T02:38:37.398 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.398 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.398 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:37.398 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.604307+0000", 2020-06-12T02:38:37.399 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103756 2020-06-12T02:38:37.399 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.399 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.399 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:37.399 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.604411+0000", 2020-06-12T02:38:37.400 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073281400000000003 2020-06-12T02:38:37.400 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.400 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.400 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:37.401 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.605143+0000", 2020-06-12T02:38:37.401 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071483730000000004 2020-06-12T02:38:37.401 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.401 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.402 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:37.402 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.612292+0000", 2020-06-12T02:38:37.402 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0075185879999999997 2020-06-12T02:38:37.402 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.402 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.403 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:37.403 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.619810+0000", 2020-06-12T02:38:37.403 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7305000000000006e-05 2020-06-12T02:38:37.403 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.403 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.404 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:37.404 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.619888+0000", 2020-06-12T02:38:37.404 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016093917999999999 2020-06-12T02:38:37.404 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:37.404 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:37.405 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:37.405 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.405 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.405 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:3335 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:38:37.405 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:38.616040+0000", 2020-06-12T02:38:37.406 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 58.708145356999999, 2020-06-12T02:38:37.406 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015075565000000001, 2020-06-12T02:38:37.406 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:37.406 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:37.406 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:37.407 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:37.407 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:37.407 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 3335 2020-06-12T02:38:37.407 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.408 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:37.408 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.408 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:37.408 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.616040+0000", 2020-06-12T02:38:37.408 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:37.409 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.409 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.409 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:37.409 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.616040+0000", 2020-06-12T02:38:37.409 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1950000000000002e-05 2020-06-12T02:38:37.410 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.410 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.410 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:37.410 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.616062+0000", 2020-06-12T02:38:37.410 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.1370000000000002e-06 2020-06-12T02:38:37.411 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.411 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.411 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:37.411 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.616066+0000", 2020-06-12T02:38:37.411 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5019999999999998e-06 2020-06-12T02:38:37.412 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.412 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.412 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:37.412 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.616072+0000", 2020-06-12T02:38:37.412 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3470000000000001e-06 2020-06-12T02:38:37.413 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.413 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.413 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:37.413 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.616074+0000", 2020-06-12T02:38:37.413 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0067702309999999998 2020-06-12T02:38:37.414 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.414 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.414 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:37.414 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.622845+0000", 2020-06-12T02:38:37.414 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7738000000000005e-05 2020-06-12T02:38:37.415 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.415 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.415 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:37.415 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.622942+0000", 2020-06-12T02:38:37.415 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0050881010000000003 2020-06-12T02:38:37.416 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.416 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.416 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:37.416 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.628031+0000", 2020-06-12T02:38:37.416 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000108171 2020-06-12T02:38:37.417 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.417 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.417 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:37.417 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.628139+0000", 2020-06-12T02:38:37.417 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0029773880000000001 2020-06-12T02:38:37.418 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.418 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.418 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:37.418 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.631116+0000", 2020-06-12T02:38:37.418 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015075565000000001 2020-06-12T02:38:37.419 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:37.419 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:37.419 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:37.419 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.419 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.420 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:4969 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~729088,stat] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-06-12T02:38:37.420 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:43.713430+0000", 2020-06-12T02:38:37.420 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 53.610755892999997, 2020-06-12T02:38:37.420 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.18980771900000001, 2020-06-12T02:38:37.420 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:37.421 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:37.421 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:37.421 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:37.421 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:37.421 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 4969 2020-06-12T02:38:37.422 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.422 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:37.422 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.422 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:37.422 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.713430+0000", 2020-06-12T02:38:37.423 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:37.423 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.423 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.423 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:37.423 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.713430+0000", 2020-06-12T02:38:37.423 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018522800000000001 2020-06-12T02:38:37.424 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.424 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.424 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:37.424 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.713615+0000", 2020-06-12T02:38:37.424 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3590000000000002e-06 2020-06-12T02:38:37.425 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.425 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.425 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:37.425 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.713623+0000", 2020-06-12T02:38:37.425 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0939e-05 2020-06-12T02:38:37.426 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.426 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.426 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:37.426 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.713633+0000", 2020-06-12T02:38:37.426 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3229999999999999e-06 2020-06-12T02:38:37.427 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.427 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.427 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:37.427 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.713638+0000", 2020-06-12T02:38:37.427 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7401000000000002e-05 2020-06-12T02:38:37.428 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.428 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.428 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:37.428 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.713685+0000", 2020-06-12T02:38:37.428 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011488 2020-06-12T02:38:37.429 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.429 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.429 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:37.429 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.713800+0000", 2020-06-12T02:38:37.429 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000662541 2020-06-12T02:38:37.430 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.430 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.430 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:37.430 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.714463+0000", 2020-06-12T02:38:37.430 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043485090000000004 2020-06-12T02:38:37.431 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.431 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.431 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:37.431 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.718811+0000", 2020-06-12T02:38:37.431 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.18428694000000001 2020-06-12T02:38:37.432 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.432 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.432 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:37.432 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.903098+0000", 2020-06-12T02:38:37.432 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013959899999999999 2020-06-12T02:38:37.433 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.433 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.433 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:37.433 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.903238+0000", 2020-06-12T02:38:37.433 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.18980771900000001 2020-06-12T02:38:37.434 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:37.434 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:37.434 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:37.434 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.434 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.435 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:4970 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~401408] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-06-12T02:38:37.435 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:43.716006+0000", 2020-06-12T02:38:37.435 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 53.60817978, 2020-06-12T02:38:37.435 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.19216903699999999, 2020-06-12T02:38:37.435 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:37.436 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:37.436 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:37.436 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:37.436 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:37.436 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 4970 2020-06-12T02:38:37.437 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.437 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:37.437 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.437 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:37.437 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716006+0000", 2020-06-12T02:38:37.438 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:37.438 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.438 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.438 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:37.438 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716006+0000", 2020-06-12T02:38:37.439 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000115757 2020-06-12T02:38:37.439 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.439 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.439 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:37.439 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716122+0000", 2020-06-12T02:38:37.440 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1989999999999997e-06 2020-06-12T02:38:37.440 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.440 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.440 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:37.440 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716127+0000", 2020-06-12T02:38:37.441 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2121e-05 2020-06-12T02:38:37.441 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.441 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.441 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:37.441 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716139+0000", 2020-06-12T02:38:37.442 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.113e-06 2020-06-12T02:38:37.442 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.442 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.442 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:37.442 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716144+0000", 2020-06-12T02:38:37.442 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014413900000000001 2020-06-12T02:38:37.443 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.443 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.443 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:37.443 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.717586+0000", 2020-06-12T02:38:37.443 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0257000000000002e-05 2020-06-12T02:38:37.444 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.444 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.444 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:37.444 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.717656+0000", 2020-06-12T02:38:37.444 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.18596705899999999 2020-06-12T02:38:37.445 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.445 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.445 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:37.445 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.903623+0000", 2020-06-12T02:38:37.445 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7858999999999999e-05 2020-06-12T02:38:37.446 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.446 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.446 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:37.446 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.903721+0000", 2020-06-12T02:38:37.446 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000683536 2020-06-12T02:38:37.447 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.447 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.447 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:37.447 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.904404+0000", 2020-06-12T02:38:37.447 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.003364438 2020-06-12T02:38:37.448 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.448 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.448 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:37.448 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.907769+0000", 2020-06-12T02:38:37.448 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00033094500000000002 2020-06-12T02:38:37.449 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.449 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.449 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:37.449 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.908100+0000", 2020-06-12T02:38:37.449 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5363000000000003e-05 2020-06-12T02:38:37.450 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.450 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.450 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:37.450 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.908175+0000", 2020-06-12T02:38:37.450 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.19216903699999999 2020-06-12T02:38:37.451 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:37.451 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:37.451 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:37.451 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.451 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.451 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:4971 3.0s0 3:0dddb690:::smithi10729407-10:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-06-12T02:38:37.452 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:43.716190+0000", 2020-06-12T02:38:37.452 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 53.607995922000001, 2020-06-12T02:38:37.452 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.19259003999999999, 2020-06-12T02:38:37.452 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:37.452 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:37.453 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:37.453 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:37.453 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:37.453 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 4971 2020-06-12T02:38:37.453 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.454 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:37.454 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.454 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:37.454 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716190+0000", 2020-06-12T02:38:37.454 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:37.455 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.455 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.455 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:37.455 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716190+0000", 2020-06-12T02:38:37.455 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3192999999999998e-05 2020-06-12T02:38:37.456 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.456 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.456 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:37.456 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716233+0000", 2020-06-12T02:38:37.456 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6279999999999998e-06 2020-06-12T02:38:37.457 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.457 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.457 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:37.457 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716237+0000", 2020-06-12T02:38:37.457 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6619999999999999e-06 2020-06-12T02:38:37.458 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.458 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.458 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:37.458 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716243+0000", 2020-06-12T02:38:37.458 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.495e-06 2020-06-12T02:38:37.459 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.459 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.459 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:37.459 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716247+0000", 2020-06-12T02:38:37.459 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014483740000000001 2020-06-12T02:38:37.460 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.460 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.460 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:37.460 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.717695+0000", 2020-06-12T02:38:37.460 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000108108 2020-06-12T02:38:37.461 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.461 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.461 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:37.461 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.717803+0000", 2020-06-12T02:38:37.461 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.18873436800000001 2020-06-12T02:38:37.462 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.462 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.462 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:37.462 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.906538+0000", 2020-06-12T02:38:37.462 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9384999999999998e-05 2020-06-12T02:38:37.463 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.463 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.463 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:37.463 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.906637+0000", 2020-06-12T02:38:37.463 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00042339299999999999 2020-06-12T02:38:37.464 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.464 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.464 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:37.464 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.907061+0000", 2020-06-12T02:38:37.464 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00071097900000000004 2020-06-12T02:38:37.465 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.465 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.465 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:37.465 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.907772+0000", 2020-06-12T02:38:37.467 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00089641299999999998 2020-06-12T02:38:37.467 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.467 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.467 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:37.468 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.908668+0000", 2020-06-12T02:38:37.468 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112042 2020-06-12T02:38:37.468 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.468 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.468 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:37.468 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.908780+0000", 2020-06-12T02:38:37.469 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.19259003999999999 2020-06-12T02:38:37.469 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:37.469 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:37.469 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:37.469 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.470 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.470 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:4972 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-06-12T02:38:37.470 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:43.716274+0000", 2020-06-12T02:38:37.470 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 53.607912153000001, 2020-06-12T02:38:37.470 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.193959876, 2020-06-12T02:38:37.471 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:37.471 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:37.471 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:37.471 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:37.471 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:37.471 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 4972 2020-06-12T02:38:37.472 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.472 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:37.472 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.472 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:37.472 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716274+0000", 2020-06-12T02:38:37.473 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:37.473 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.473 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.473 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:37.473 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716274+0000", 2020-06-12T02:38:37.474 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6852999999999999e-05 2020-06-12T02:38:37.474 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.474 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.474 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:37.474 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716301+0000", 2020-06-12T02:38:37.475 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3069999999999998e-06 2020-06-12T02:38:37.475 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.475 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.475 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:37.475 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716304+0000", 2020-06-12T02:38:37.475 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6080000000000001e-06 2020-06-12T02:38:37.476 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.476 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.476 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:37.476 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716310+0000", 2020-06-12T02:38:37.476 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6110000000000001e-06 2020-06-12T02:38:37.477 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.477 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.477 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:37.477 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716313+0000", 2020-06-12T02:38:37.477 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001512918 2020-06-12T02:38:37.478 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.478 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.478 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:37.478 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.717826+0000", 2020-06-12T02:38:37.478 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3773000000000002e-05 2020-06-12T02:38:37.478 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.479 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.479 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:37.479 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.717880+0000", 2020-06-12T02:38:37.479 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.18955132599999999 2020-06-12T02:38:37.479 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.480 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.480 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:37.480 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.907431+0000", 2020-06-12T02:38:37.480 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101158 2020-06-12T02:38:37.480 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.480 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.481 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:37.481 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.907532+0000", 2020-06-12T02:38:37.481 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00160593 2020-06-12T02:38:37.481 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.481 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.482 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:37.482 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.909138+0000", 2020-06-12T02:38:37.482 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8590000000000006e-05 2020-06-12T02:38:37.482 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.482 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.483 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:37.483 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.909207+0000", 2020-06-12T02:38:37.483 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0010268020000000001 2020-06-12T02:38:37.483 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.483 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.483 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:37.484 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.910234+0000", 2020-06-12T02:38:37.484 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.193959876 2020-06-12T02:38:37.484 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:37.484 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:37.484 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:37.485 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.485 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.485 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:5006 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-06-12T02:38:37.485 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:43.975526+0000", 2020-06-12T02:38:37.485 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 53.348660103, 2020-06-12T02:38:37.486 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014759296, 2020-06-12T02:38:37.486 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:37.486 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:37.486 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:37.486 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:37.487 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:37.487 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 5006 2020-06-12T02:38:37.487 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.487 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:37.487 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.487 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:37.488 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.975526+0000", 2020-06-12T02:38:37.488 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:37.489 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.489 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.490 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:37.490 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.975526+0000", 2020-06-12T02:38:37.490 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5868999999999997e-05 2020-06-12T02:38:37.490 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.490 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.490 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:37.491 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.975572+0000", 2020-06-12T02:38:37.491 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1210000000000003e-06 2020-06-12T02:38:37.491 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.491 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.491 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:37.492 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.975578+0000", 2020-06-12T02:38:37.492 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2204999999999999e-05 2020-06-12T02:38:37.492 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.492 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.492 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:37.493 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.975590+0000", 2020-06-12T02:38:37.493 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1289999999999998e-06 2020-06-12T02:38:37.493 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.493 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.493 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:37.494 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.975594+0000", 2020-06-12T02:38:37.494 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.7705e-05 2020-06-12T02:38:37.494 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.494 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.494 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:37.495 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.975622+0000", 2020-06-12T02:38:37.495 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9941999999999997e-05 2020-06-12T02:38:37.495 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.495 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.495 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:37.495 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.975672+0000", 2020-06-12T02:38:37.496 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0057881060000000003 2020-06-12T02:38:37.496 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.496 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.496 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:37.496 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.981460+0000", 2020-06-12T02:38:37.497 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0473999999999997e-05 2020-06-12T02:38:37.497 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.497 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.497 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:37.497 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.981520+0000", 2020-06-12T02:38:37.498 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.007714294 2020-06-12T02:38:37.498 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.498 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.498 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:37.498 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.989235+0000", 2020-06-12T02:38:37.498 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010107600000000001 2020-06-12T02:38:37.499 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.499 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.499 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:37.499 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.989336+0000", 2020-06-12T02:38:37.499 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00094937500000000002 2020-06-12T02:38:37.500 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.500 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.500 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:37.500 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.990285+0000", 2020-06-12T02:38:37.500 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014759296 2020-06-12T02:38:37.501 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:37.501 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:37.501 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:37.501 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.501 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.502 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:7841 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~565248] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-06-12T02:38:37.502 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:51.996595+0000", 2020-06-12T02:38:37.502 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 45.327590614000002, 2020-06-12T02:38:37.502 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015060283000000001, 2020-06-12T02:38:37.502 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:37.503 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:37.503 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:37.503 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:37.503 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:37.503 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 7841 2020-06-12T02:38:37.504 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.504 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:37.504 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.504 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:37.505 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.996595+0000", 2020-06-12T02:38:37.505 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:37.505 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.505 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.505 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:37.505 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.996595+0000", 2020-06-12T02:38:37.506 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015120699999999999 2020-06-12T02:38:37.506 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.506 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.506 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:37.506 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.996746+0000", 2020-06-12T02:38:37.507 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4369999999999998e-06 2020-06-12T02:38:37.507 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.507 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.507 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:37.508 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.996752+0000", 2020-06-12T02:38:37.508 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0850000000000003e-06 2020-06-12T02:38:37.509 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.509 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.510 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:37.510 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.996761+0000", 2020-06-12T02:38:37.510 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2370000000000003e-06 2020-06-12T02:38:37.510 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.510 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.511 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:37.511 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.996765+0000", 2020-06-12T02:38:37.511 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5185999999999998e-05 2020-06-12T02:38:37.511 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.511 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.512 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:37.512 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.996800+0000", 2020-06-12T02:38:37.512 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7265000000000001e-05 2020-06-12T02:38:37.512 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.512 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.513 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:37.513 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.996868+0000", 2020-06-12T02:38:37.513 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0024047539999999998 2020-06-12T02:38:37.513 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.513 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.513 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:37.514 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.999272+0000", 2020-06-12T02:38:37.514 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3034999999999994e-05 2020-06-12T02:38:37.514 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.514 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.514 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:37.515 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.999355+0000", 2020-06-12T02:38:37.515 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049334499999999996 2020-06-12T02:38:37.515 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.515 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.515 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:37.516 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.999849+0000", 2020-06-12T02:38:37.516 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011708226 2020-06-12T02:38:37.516 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.516 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.516 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:37.516 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.011557+0000", 2020-06-12T02:38:37.517 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1449999999999997e-05 2020-06-12T02:38:37.517 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.517 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.517 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:37.517 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.011608+0000", 2020-06-12T02:38:37.518 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7055999999999998e-05 2020-06-12T02:38:37.518 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.518 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.518 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:37.518 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.011656+0000", 2020-06-12T02:38:37.519 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015060283000000001 2020-06-12T02:38:37.522 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:37.523 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:37.523 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:37.523 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.523 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.523 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:7845 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e33)", 2020-06-12T02:38:37.524 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:52.002017+0000", 2020-06-12T02:38:37.524 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 45.322168382999998, 2020-06-12T02:38:37.524 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017208558999999998, 2020-06-12T02:38:37.524 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:37.525 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:37.525 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:37.525 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:37.525 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:37.526 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 7845 2020-06-12T02:38:37.526 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.526 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:37.526 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.526 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:37.526 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.002017+0000", 2020-06-12T02:38:37.527 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:37.527 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.527 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.527 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:37.527 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.002017+0000", 2020-06-12T02:38:37.528 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2636000000000003e-05 2020-06-12T02:38:37.528 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.528 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.528 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:37.528 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.002060+0000", 2020-06-12T02:38:37.528 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5080000000000002e-06 2020-06-12T02:38:37.529 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.529 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.529 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:37.529 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.002067+0000", 2020-06-12T02:38:37.529 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3125000000000001e-05 2020-06-12T02:38:37.530 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.530 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.530 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:37.530 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.002080+0000", 2020-06-12T02:38:37.530 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5560000000000001e-06 2020-06-12T02:38:37.531 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.531 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.532 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:37.532 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.002084+0000", 2020-06-12T02:38:37.532 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0088536750000000001 2020-06-12T02:38:37.532 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.532 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.533 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:37.533 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.010938+0000", 2020-06-12T02:38:37.533 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3282999999999995e-05 2020-06-12T02:38:37.533 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.533 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.534 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:37.534 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.011021+0000", 2020-06-12T02:38:37.534 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0052873970000000001 2020-06-12T02:38:37.535 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.535 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.535 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:37.536 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.016309+0000", 2020-06-12T02:38:37.536 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010365399999999999 2020-06-12T02:38:37.536 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.537 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.537 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:37.537 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.016412+0000", 2020-06-12T02:38:37.537 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002813725 2020-06-12T02:38:37.538 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.538 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.538 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:37.538 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.019226+0000", 2020-06-12T02:38:37.538 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017208558999999998 2020-06-12T02:38:37.539 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:37.539 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:37.539 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:37.539 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.539 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.540 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8809 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~606208] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:37.540 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.707225+0000", 2020-06-12T02:38:37.540 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 25.616960945999999, 2020-06-12T02:38:37.540 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016771206, 2020-06-12T02:38:37.541 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:37.541 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:37.542 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:37.542 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:37.543 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:37.543 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8809 2020-06-12T02:38:37.543 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.544 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:37.544 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.544 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:37.544 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.707225+0000", 2020-06-12T02:38:37.545 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:37.545 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.545 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.546 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:37.546 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.707225+0000", 2020-06-12T02:38:37.546 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035342099999999998 2020-06-12T02:38:37.547 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.547 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.547 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:37.548 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.707578+0000", 2020-06-12T02:38:37.548 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2309999999999996e-06 2020-06-12T02:38:37.548 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.549 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.549 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:37.549 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.707587+0000", 2020-06-12T02:38:37.550 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4343e-05 2020-06-12T02:38:37.550 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.550 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.550 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:37.551 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.707601+0000", 2020-06-12T02:38:37.551 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8460000000000002e-06 2020-06-12T02:38:37.551 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.552 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.552 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:37.552 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.707608+0000", 2020-06-12T02:38:37.553 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7374999999999998e-05 2020-06-12T02:38:37.553 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.553 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.554 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:37.554 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.707675+0000", 2020-06-12T02:38:37.555 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111599 2020-06-12T02:38:37.555 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.555 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.555 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:37.556 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.707787+0000", 2020-06-12T02:38:37.556 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00079845100000000004 2020-06-12T02:38:37.556 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.557 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.557 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:37.557 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.708585+0000", 2020-06-12T02:38:37.558 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010899008999999999 2020-06-12T02:38:37.558 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.558 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.559 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:37.559 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.719484+0000", 2020-06-12T02:38:37.559 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306450000000004 2020-06-12T02:38:37.560 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.560 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.560 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:37.560 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.723915+0000", 2020-06-12T02:38:37.561 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1285999999999994e-05 2020-06-12T02:38:37.561 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.561 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.562 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:37.562 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.723996+0000", 2020-06-12T02:38:37.562 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016771206 2020-06-12T02:38:37.563 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:37.563 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:37.563 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:37.564 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.564 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.564 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.3s1 49/21 ECSubWrite(tid=4109, reqid=client.4512.0:8841, at_version=49'773, trim_to=0'0, roll_forward_to=49'771))", 2020-06-12T02:38:37.565 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.822295+0000", 2020-06-12T02:38:37.565 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 25.501890374999999, 2020-06-12T02:38:37.565 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01733517, 2020-06-12T02:38:37.566 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:37.566 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:37.566 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:37.567 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.567 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:37.567 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822295+0000", 2020-06-12T02:38:37.567 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:37.568 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.568 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.568 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:37.569 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822295+0000", 2020-06-12T02:38:37.569 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00089525499999999997 2020-06-12T02:38:37.569 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.570 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.570 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:37.570 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.823191+0000", 2020-06-12T02:38:37.571 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3315999999999998e-05 2020-06-12T02:38:37.571 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.571 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.571 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:37.572 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.823214+0000", 2020-06-12T02:38:37.572 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7096e-05 2020-06-12T02:38:37.572 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.573 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.573 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:37.573 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.823231+0000", 2020-06-12T02:38:37.574 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6890000000000004e-06 2020-06-12T02:38:37.574 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.574 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.575 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:37.575 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.823239+0000", 2020-06-12T02:38:37.575 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00061031599999999996 2020-06-12T02:38:37.576 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.576 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.576 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:37.577 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.823849+0000", 2020-06-12T02:38:37.577 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4205999999999999e-05 2020-06-12T02:38:37.577 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.578 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.578 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:37.578 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.823863+0000", 2020-06-12T02:38:37.578 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015707789999999999 2020-06-12T02:38:37.579 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.579 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.579 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:37.579 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.839571+0000", 2020-06-12T02:38:37.580 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9502000000000001e-05 2020-06-12T02:38:37.580 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.580 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.580 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:37.581 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.839631+0000", 2020-06-12T02:38:37.581 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01733517 2020-06-12T02:38:37.581 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:37.581 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:37.582 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:37.582 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.582 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.582 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8854 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~409600] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:37.583 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.854691+0000", 2020-06-12T02:38:37.583 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 25.469494492999999, 2020-06-12T02:38:37.583 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01967967, 2020-06-12T02:38:37.584 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:37.584 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:37.584 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:37.585 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:37.585 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:37.585 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8854 2020-06-12T02:38:37.585 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.586 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:37.586 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.586 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:37.587 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.854691+0000", 2020-06-12T02:38:37.587 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:37.587 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.587 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.588 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:37.588 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.854691+0000", 2020-06-12T02:38:37.588 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017767799999999999 2020-06-12T02:38:37.588 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.589 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.589 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:37.589 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.854869+0000", 2020-06-12T02:38:37.590 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8049999999999996e-06 2020-06-12T02:38:37.590 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.590 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.591 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:37.591 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.854878+0000", 2020-06-12T02:38:37.591 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5644000000000001e-05 2020-06-12T02:38:37.591 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.592 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.592 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:37.592 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.854893+0000", 2020-06-12T02:38:37.593 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.584e-06 2020-06-12T02:38:37.593 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.593 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.594 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:37.594 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.854900+0000", 2020-06-12T02:38:37.594 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002416769 2020-06-12T02:38:37.594 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.595 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.595 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:37.595 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.857317+0000", 2020-06-12T02:38:37.595 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000100385 2020-06-12T02:38:37.596 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.596 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.596 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:37.596 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.857417+0000", 2020-06-12T02:38:37.597 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0047283969999999996 2020-06-12T02:38:37.597 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.597 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.598 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:37.598 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.862146+0000", 2020-06-12T02:38:37.598 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1930999999999998e-05 2020-06-12T02:38:37.599 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.599 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.599 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:37.600 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.862238+0000", 2020-06-12T02:38:37.600 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00071479400000000004 2020-06-12T02:38:37.600 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.600 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.601 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:37.601 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.862952+0000", 2020-06-12T02:38:37.601 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011219316999999999 2020-06-12T02:38:37.602 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.602 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.602 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:37.602 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.874172+0000", 2020-06-12T02:38:37.602 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000104137 2020-06-12T02:38:37.603 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.603 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.603 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:37.603 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.874276+0000", 2020-06-12T02:38:37.603 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5229000000000006e-05 2020-06-12T02:38:37.604 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.604 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.604 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:37.604 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.874371+0000", 2020-06-12T02:38:37.604 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01967967 2020-06-12T02:38:37.605 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:37.605 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:37.605 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:37.605 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.605 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.605 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8855 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~516096] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:37.606 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.858551+0000", 2020-06-12T02:38:37.606 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 25.465634635000001, 2020-06-12T02:38:37.606 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020039164000000002, 2020-06-12T02:38:37.606 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:37.606 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:37.607 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:37.607 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:37.607 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:37.607 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8855 2020-06-12T02:38:37.607 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.608 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:37.608 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.608 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:37.608 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.858551+0000", 2020-06-12T02:38:37.608 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:37.608 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.609 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.609 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:37.609 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.858551+0000", 2020-06-12T02:38:37.609 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00020537100000000001 2020-06-12T02:38:37.609 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.610 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.610 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:37.610 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.858757+0000", 2020-06-12T02:38:37.610 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4870000000000002e-06 2020-06-12T02:38:37.610 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.611 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.611 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:37.611 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.858765+0000", 2020-06-12T02:38:37.611 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4358e-05 2020-06-12T02:38:37.611 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.611 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.612 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:37.612 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.858779+0000", 2020-06-12T02:38:37.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3879999999999997e-06 2020-06-12T02:38:37.612 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.613 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:37.613 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.858786+0000", 2020-06-12T02:38:37.613 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089100000000001 2020-06-12T02:38:37.613 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.613 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.613 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:37.614 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.859127+0000", 2020-06-12T02:38:37.614 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0877000000000005e-05 2020-06-12T02:38:37.614 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.614 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.614 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:37.615 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.859208+0000", 2020-06-12T02:38:37.615 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0059661269999999999 2020-06-12T02:38:37.615 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.615 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.615 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:37.616 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.865174+0000", 2020-06-12T02:38:37.616 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000102761 2020-06-12T02:38:37.616 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.616 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.616 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:37.616 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.865276+0000", 2020-06-12T02:38:37.617 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00079971700000000003 2020-06-12T02:38:37.617 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.617 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.617 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:37.617 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.866076+0000", 2020-06-12T02:38:37.618 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0083001380000000003 2020-06-12T02:38:37.618 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.618 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.618 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:37.618 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.874376+0000", 2020-06-12T02:38:37.618 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041190660000000002 2020-06-12T02:38:37.619 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.619 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.619 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:37.619 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.878495+0000", 2020-06-12T02:38:37.619 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4982999999999995e-05 2020-06-12T02:38:37.620 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.620 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.620 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:37.620 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.878590+0000", 2020-06-12T02:38:37.620 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020039164000000002 2020-06-12T02:38:37.620 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:37.621 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:37.621 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:37.621 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.621 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.621 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8856 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~442368] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:37.622 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.861853+0000", 2020-06-12T02:38:37.622 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 25.462333185999999, 2020-06-12T02:38:37.622 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022102294000000001, 2020-06-12T02:38:37.622 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:37.622 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:37.622 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:37.623 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:37.623 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:37.623 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8856 2020-06-12T02:38:37.623 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.623 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:37.624 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.624 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:37.624 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.861853+0000", 2020-06-12T02:38:37.624 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:37.625 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.625 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.625 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:37.625 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.861853+0000", 2020-06-12T02:38:37.625 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017153099999999999 2020-06-12T02:38:37.626 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.626 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.626 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:37.626 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.862024+0000", 2020-06-12T02:38:37.626 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0889999999999999e-06 2020-06-12T02:38:37.627 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.627 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.627 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:37.627 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.862029+0000", 2020-06-12T02:38:37.627 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3033999999999999e-05 2020-06-12T02:38:37.627 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.628 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.628 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:37.628 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.862042+0000", 2020-06-12T02:38:37.628 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1349999999999997e-06 2020-06-12T02:38:37.628 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.629 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.629 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:37.629 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.862048+0000", 2020-06-12T02:38:37.629 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0081330410000000006 2020-06-12T02:38:37.629 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.629 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.630 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:37.630 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.870181+0000", 2020-06-12T02:38:37.630 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012252500000000001 2020-06-12T02:38:37.630 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.630 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.631 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:37.631 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.870304+0000", 2020-06-12T02:38:37.631 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000740999 2020-06-12T02:38:37.631 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.631 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.631 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:37.632 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.871045+0000", 2020-06-12T02:38:37.632 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012634975 2020-06-12T02:38:37.632 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.632 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.632 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:37.633 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.883680+0000", 2020-06-12T02:38:37.633 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017667900000000001 2020-06-12T02:38:37.633 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.633 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.633 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:37.633 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.883857+0000", 2020-06-12T02:38:37.634 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8286000000000001e-05 2020-06-12T02:38:37.634 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.634 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.634 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:37.634 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.883955+0000", 2020-06-12T02:38:37.635 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022102294000000001 2020-06-12T02:38:37.635 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:37.635 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:37.635 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:37.636 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.636 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.636 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8857 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~237568] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:37.637 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.863733+0000", 2020-06-12T02:38:37.637 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 25.460453203, 2020-06-12T02:38:37.637 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020470258000000002, 2020-06-12T02:38:37.637 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:37.637 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:37.638 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:37.638 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:37.638 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:37.638 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8857 2020-06-12T02:38:37.638 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.639 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:37.639 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.639 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:37.639 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.863733+0000", 2020-06-12T02:38:37.640 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:37.640 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.640 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.640 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:37.640 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.863733+0000", 2020-06-12T02:38:37.640 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014776000000000001 2020-06-12T02:38:37.641 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.641 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.641 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:37.641 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.863880+0000", 2020-06-12T02:38:37.641 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5790000000000004e-06 2020-06-12T02:38:37.642 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.642 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.642 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:37.642 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.863889+0000", 2020-06-12T02:38:37.642 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6960999999999999e-05 2020-06-12T02:38:37.642 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.643 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.643 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:37.643 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.863906+0000", 2020-06-12T02:38:37.643 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5509999999999996e-06 2020-06-12T02:38:37.643 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.644 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.644 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:37.644 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.863913+0000", 2020-06-12T02:38:37.644 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010569258 2020-06-12T02:38:37.644 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.644 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.645 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:37.645 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.874482+0000", 2020-06-12T02:38:37.645 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011260399999999999 2020-06-12T02:38:37.645 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.645 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.646 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:37.646 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.874594+0000", 2020-06-12T02:38:37.646 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014913820000000001 2020-06-12T02:38:37.646 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.646 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.647 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:37.647 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.876086+0000", 2020-06-12T02:38:37.647 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0078744260000000003 2020-06-12T02:38:37.647 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.647 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.647 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:37.648 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.883960+0000", 2020-06-12T02:38:37.648 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015276999999999999 2020-06-12T02:38:37.648 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.648 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.648 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:37.649 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.884113+0000", 2020-06-12T02:38:37.649 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9967000000000001e-05 2020-06-12T02:38:37.649 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.649 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.649 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:37.649 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.884203+0000", 2020-06-12T02:38:37.650 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020470258000000002 2020-06-12T02:38:37.650 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:37.650 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:37.650 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:37.650 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.651 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.652 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8858 3.0s0 3:0dddb690:::smithi10729407-10:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:37.652 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.863968+0000", 2020-06-12T02:38:37.652 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 25.460217911000001, 2020-06-12T02:38:37.652 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022225373999999999, 2020-06-12T02:38:37.653 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:37.653 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:37.653 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:37.653 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:37.654 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:37.654 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8858 2020-06-12T02:38:37.654 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.655 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:37.655 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.655 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:37.655 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.863968+0000", 2020-06-12T02:38:37.655 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:37.655 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.656 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.656 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:37.656 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.863968+0000", 2020-06-12T02:38:37.656 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5352000000000006e-05 2020-06-12T02:38:37.656 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.657 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.657 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:37.657 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864043+0000", 2020-06-12T02:38:37.657 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9110000000000004e-06 2020-06-12T02:38:37.657 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.658 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.658 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:37.658 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864050+0000", 2020-06-12T02:38:37.658 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2505e-05 2020-06-12T02:38:37.658 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.659 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.659 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:37.659 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864063+0000", 2020-06-12T02:38:37.659 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0900000000000004e-06 2020-06-12T02:38:37.659 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.659 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.660 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:37.660 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864068+0000", 2020-06-12T02:38:37.660 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014563938 2020-06-12T02:38:37.660 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.661 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.661 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:37.661 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.878632+0000", 2020-06-12T02:38:37.662 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000106852 2020-06-12T02:38:37.662 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.662 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.662 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:37.663 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.878739+0000", 2020-06-12T02:38:37.663 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002931796 2020-06-12T02:38:37.663 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.663 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.664 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:37.664 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.881670+0000", 2020-06-12T02:38:37.665 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025365280000000001 2020-06-12T02:38:37.665 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.665 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.666 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:37.666 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.884207+0000", 2020-06-12T02:38:37.666 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001895224 2020-06-12T02:38:37.667 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.667 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.667 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:37.668 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.886102+0000", 2020-06-12T02:38:37.668 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1178e-05 2020-06-12T02:38:37.668 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.668 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.669 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:37.669 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.886193+0000", 2020-06-12T02:38:37.669 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022225373999999999 2020-06-12T02:38:37.670 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:37.670 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:37.673 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:37.674 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.674 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.675 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8859 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e49)", 2020-06-12T02:38:37.675 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.864115+0000", 2020-06-12T02:38:37.675 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 25.46007101, 2020-06-12T02:38:37.676 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.03325032, 2020-06-12T02:38:37.676 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:37.676 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:37.677 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:37.677 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:37.677 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:37.678 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8859 2020-06-12T02:38:37.678 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.678 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:37.679 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.679 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:37.679 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864115+0000", 2020-06-12T02:38:37.680 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:37.680 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.680 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.681 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:37.682 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864115+0000", 2020-06-12T02:38:37.683 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6795999999999997e-05 2020-06-12T02:38:37.683 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.683 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.683 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:37.684 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864162+0000", 2020-06-12T02:38:37.684 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5599999999999999e-06 2020-06-12T02:38:37.684 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.685 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.685 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:37.686 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864168+0000", 2020-06-12T02:38:37.686 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5658000000000002e-05 2020-06-12T02:38:37.686 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.686 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.687 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:37.687 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864184+0000", 2020-06-12T02:38:37.687 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8380000000000001e-06 2020-06-12T02:38:37.688 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.688 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.688 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:37.689 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864189+0000", 2020-06-12T02:38:37.689 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019365752 2020-06-12T02:38:37.690 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.690 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.690 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:37.690 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.883554+0000", 2020-06-12T02:38:37.691 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000118819 2020-06-12T02:38:37.691 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.691 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.691 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:37.692 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.883673+0000", 2020-06-12T02:38:37.692 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0089243440000000007 2020-06-12T02:38:37.694 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.694 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.694 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:37.694 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.892598+0000", 2020-06-12T02:38:37.695 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7429999999999994e-05 2020-06-12T02:38:37.695 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.695 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.695 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:37.695 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.892695+0000", 2020-06-12T02:38:37.695 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004670123 2020-06-12T02:38:37.696 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:37.696 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:37.696 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:37.697 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.897365+0000", 2020-06-12T02:38:37.697 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.03325032 2020-06-12T02:38:37.697 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:37.697 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:37.697 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:37.698 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:37.698 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:37.698 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:37.716 INFO:tasks.rados.rados.0.smithi107.stdout:7852: writing smithi10729407-15 from 1744896 to 2441216 tid 1 2020-06-12T02:38:37.717 INFO:tasks.rados.rados.0.smithi107.stdout:7852: writing smithi10729407-15 from 2441216 to 3211264 tid 2 2020-06-12T02:38:37.717 INFO:tasks.rados.rados.0.smithi107.stdout:7852: writing smithi10729407-15 from 3211264 to 3645440 tid 3 2020-06-12T02:38:37.717 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:37.717 INFO:tasks.rados.rados.0.smithi107.stdout:7853: seq_num 7852 ranges {0=712704,712704=647168,1359872=557056} 2020-06-12T02:38:37.718 INFO:tasks.rados.rados.0.smithi107.stdout:7853: writing smithi10729407-19 from 0 to 712704 tid 1 2020-06-12T02:38:37.718 INFO:tasks.rados.rados.0.smithi107.stdout:7853: writing smithi10729407-19 from 712704 to 1359872 tid 2 2020-06-12T02:38:37.718 INFO:tasks.rados.rados.0.smithi107.stdout:7853: writing smithi10729407-19 from 1359872 to 1916928 tid 3 2020-06-12T02:38:37.719 INFO:tasks.rados.rados.0.smithi107.stdout:7854: delete oid 2 current snap is 0 2020-06-12T02:38:37.719 INFO:tasks.rados.rados.0.smithi107.stdout:7852: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:37.719 INFO:tasks.rados.rados.0.smithi107.stdout:7852: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:37.719 INFO:tasks.rados.rados.0.smithi107.stdout:7852: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:37.719 INFO:tasks.rados.rados.0.smithi107.stdout:7852: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:37.721 INFO:tasks.rados.rados.0.smithi107.stdout:7852: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:37.721 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1457 (ObjNum 7851 snap 0 seq_num 7851) dirty exists 2020-06-12T02:38:37.721 INFO:tasks.rados.rados.0.smithi107.stdout:7852: left oid 15 (ObjNum 7851 snap 0 seq_num 7851) 2020-06-12T02:38:37.721 INFO:tasks.rados.rados.0.smithi107.stdout:7852: done (2 left) 2020-06-12T02:38:37.721 INFO:tasks.rados.rados.0.smithi107.stdout:7854: done (1 left) 2020-06-12T02:38:37.722 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:37.722 INFO:tasks.rados.rados.0.smithi107.stdout:7855: seq_num 7854 ranges {0=745472,745472=466944,1212416=573440,1785856=163840} 2020-06-12T02:38:37.722 INFO:tasks.rados.rados.0.smithi107.stdout:7855: writing smithi10729407-2 from 0 to 745472 tid 1 2020-06-12T02:38:37.722 INFO:tasks.rados.rados.0.smithi107.stdout:7855: writing smithi10729407-2 from 745472 to 1212416 tid 2 2020-06-12T02:38:37.727 INFO:tasks.rados.rados.0.smithi107.stdout:7855: writing smithi10729407-2 from 1212416 to 1785856 tid 3 2020-06-12T02:38:37.727 INFO:tasks.rados.rados.0.smithi107.stdout:7855: writing smithi10729407-2 from 1785856 to 1949696 tid 4 2020-06-12T02:38:37.727 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:37.727 INFO:tasks.rados.rados.0.smithi107.stdout:7856: seq_num 7855 ranges {3170304=761856,3932160=753664,4685824=450560} 2020-06-12T02:38:37.728 INFO:tasks.rados.rados.0.smithi107.stdout:7856: writing smithi10729407-18 from 3170304 to 3932160 tid 1 2020-06-12T02:38:37.728 INFO:tasks.rados.rados.0.smithi107.stdout:7856: writing smithi10729407-18 from 3932160 to 4685824 tid 2 2020-06-12T02:38:37.728 INFO:tasks.rados.rados.0.smithi107.stdout:7856: writing smithi10729407-18 from 4685824 to 5136384 tid 3 2020-06-12T02:38:37.728 INFO:tasks.rados.rados.0.smithi107.stdout:7857: delete oid 20 current snap is 0 2020-06-12T02:38:37.728 INFO:tasks.rados.rados.0.smithi107.stdout:7853: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:37.729 INFO:tasks.rados.rados.0.smithi107.stdout:7853: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:37.729 INFO:tasks.rados.rados.0.smithi107.stdout:7853: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:37.729 INFO:tasks.rados.rados.0.smithi107.stdout:7853: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:37.729 INFO:tasks.rados.rados.0.smithi107.stdout:7853: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:37.729 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1427 (ObjNum 7852 snap 0 seq_num 7852) dirty exists 2020-06-12T02:38:37.730 INFO:tasks.rados.rados.0.smithi107.stdout:7853: left oid 19 (ObjNum 7852 snap 0 seq_num 7852) 2020-06-12T02:38:37.730 INFO:tasks.rados.rados.0.smithi107.stdout:7855: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:37.730 INFO:tasks.rados.rados.0.smithi107.stdout:7855: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:37.730 INFO:tasks.rados.rados.0.smithi107.stdout:7855: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:37.732 INFO:tasks.rados.rados.0.smithi107.stdout:7855: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:37.732 INFO:tasks.rados.rados.0.smithi107.stdout:7855: finishing write tid 6 to smithi10729407-2 2020-06-12T02:38:37.732 INFO:tasks.rados.rados.0.smithi107.stdout:7855: finishing write tid 7 to smithi10729407-2 2020-06-12T02:38:37.732 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 740 (ObjNum 7854 snap 0 seq_num 7854) dirty exists 2020-06-12T02:38:37.733 INFO:tasks.rados.rados.0.smithi107.stdout:7855: left oid 2 (ObjNum 7854 snap 0 seq_num 7854) 2020-06-12T02:38:37.733 INFO:tasks.rados.rados.0.smithi107.stdout:7853: done (3 left) 2020-06-12T02:38:37.733 INFO:tasks.rados.rados.0.smithi107.stdout:7855: done (2 left) 2020-06-12T02:38:37.733 INFO:tasks.rados.rados.0.smithi107.stdout:7857: done (1 left) 2020-06-12T02:38:37.733 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:37.734 INFO:tasks.rados.rados.0.smithi107.stdout:7858: seq_num 7857 ranges {1531904=794624,2326528=524288,2850816=679936} 2020-06-12T02:38:37.734 INFO:tasks.rados.rados.0.smithi107.stdout:7858: writing smithi10729407-16 from 1531904 to 2326528 tid 1 2020-06-12T02:38:37.734 INFO:tasks.rados.rados.0.smithi107.stdout:7858: writing smithi10729407-16 from 2326528 to 2850816 tid 2 2020-06-12T02:38:37.734 INFO:tasks.rados.rados.0.smithi107.stdout:7858: writing smithi10729407-16 from 2850816 to 3530752 tid 3 2020-06-12T02:38:37.734 INFO:tasks.rados.rados.0.smithi107.stdout:7859: delete oid 19 current snap is 0 2020-06-12T02:38:37.735 INFO:tasks.rados.rados.0.smithi107.stdout:7856: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:37.735 INFO:tasks.rados.rados.0.smithi107.stdout:7856: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:37.735 INFO:tasks.rados.rados.0.smithi107.stdout:7856: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:37.735 INFO:tasks.rados.rados.0.smithi107.stdout:7856: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:37.735 INFO:tasks.rados.rados.0.smithi107.stdout:7856: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:37.736 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1473 (ObjNum 7855 snap 0 seq_num 7855) dirty exists 2020-06-12T02:38:37.736 INFO:tasks.rados.rados.0.smithi107.stdout:7856: left oid 18 (ObjNum 7855 snap 0 seq_num 7855) 2020-06-12T02:38:37.736 INFO:tasks.rados.rados.0.smithi107.stdout:7858: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:37.736 INFO:tasks.rados.rados.0.smithi107.stdout:7856: done (2 left) 2020-06-12T02:38:37.736 INFO:tasks.rados.rados.0.smithi107.stdout:7859: done (1 left) 2020-06-12T02:38:37.736 INFO:tasks.rados.rados.0.smithi107.stdout:7860: delete oid 18 current snap is 0 2020-06-12T02:38:37.737 INFO:tasks.rados.rados.0.smithi107.stdout:7860: done (1 left) 2020-06-12T02:38:37.737 INFO:tasks.rados.rados.0.smithi107.stdout:7861: delete oid 13 current snap is 0 2020-06-12T02:38:37.737 INFO:tasks.rados.rados.0.smithi107.stdout:7858: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:37.737 INFO:tasks.rados.rados.0.smithi107.stdout:7861: done (1 left) 2020-06-12T02:38:37.737 INFO:tasks.rados.rados.0.smithi107.stdout:7862: delete oid 18 current snap is 0 2020-06-12T02:38:37.738 INFO:tasks.rados.rados.0.smithi107.stdout:7858: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:37.738 INFO:tasks.rados.rados.0.smithi107.stdout:7858: finishing write tid 5 to smithi10729407-16 2020-06-12T02:38:37.738 INFO:tasks.rados.rados.0.smithi107.stdout:7862: done (1 left) 2020-06-12T02:38:37.738 INFO:tasks.rados.rados.0.smithi107.stdout:7863: delete oid 14 current snap is 0 2020-06-12T02:38:37.739 INFO:tasks.rados.rados.0.smithi107.stdout:7858: finishing write tid 6 to smithi10729407-16 2020-06-12T02:38:37.739 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1428 (ObjNum 7857 snap 0 seq_num 7857) dirty exists 2020-06-12T02:38:37.739 INFO:tasks.rados.rados.0.smithi107.stdout:7858: left oid 16 (ObjNum 7857 snap 0 seq_num 7857) 2020-06-12T02:38:37.739 INFO:tasks.rados.rados.0.smithi107.stdout:7858: done (1 left) 2020-06-12T02:38:37.739 INFO:tasks.rados.rados.0.smithi107.stdout:7863: done (0 left) 2020-06-12T02:38:37.740 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:37.740 INFO:tasks.rados.rados.0.smithi107.stdout:7864: seq_num 7863 ranges {0=499712,499712=679936,1179648=737280,1916928=188416} 2020-06-12T02:38:37.740 INFO:tasks.rados.rados.0.smithi107.stdout:7864: writing smithi10729407-1 from 0 to 499712 tid 1 2020-06-12T02:38:37.740 INFO:tasks.rados.rados.0.smithi107.stdout:7864: writing smithi10729407-1 from 499712 to 1179648 tid 2 2020-06-12T02:38:37.740 INFO:tasks.rados.rados.0.smithi107.stdout:7864: writing smithi10729407-1 from 1179648 to 1916928 tid 3 2020-06-12T02:38:37.740 INFO:tasks.rados.rados.0.smithi107.stdout:7864: writing smithi10729407-1 from 1916928 to 2105344 tid 4 2020-06-12T02:38:37.741 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:38:37.741 INFO:tasks.rados.rados.0.smithi107.stdout:7865: seq_num 7864 ranges {3293184=516096,3809280=557056,4366336=548864,4915200=499712} 2020-06-12T02:38:37.741 INFO:tasks.rados.rados.0.smithi107.stdout:7865: writing smithi10729407-10 from 3293184 to 3809280 tid 1 2020-06-12T02:38:37.741 INFO:tasks.rados.rados.0.smithi107.stdout:7865: writing smithi10729407-10 from 3809280 to 4366336 tid 2 2020-06-12T02:38:37.741 INFO:tasks.rados.rados.0.smithi107.stdout:7865: writing smithi10729407-10 from 4366336 to 4915200 tid 3 2020-06-12T02:38:37.742 INFO:tasks.rados.rados.0.smithi107.stdout:7865: writing smithi10729407-10 from 4915200 to 5414912 tid 4 2020-06-12T02:38:37.742 INFO:tasks.rados.rados.0.smithi107.stdout:7866: delete oid 18 current snap is 0 2020-06-12T02:38:37.742 INFO:tasks.rados.rados.0.smithi107.stdout:7864: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:37.742 INFO:tasks.rados.rados.0.smithi107.stdout:7864: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:37.742 INFO:tasks.rados.rados.0.smithi107.stdout:7864: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:37.742 INFO:tasks.rados.rados.0.smithi107.stdout:7864: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:37.743 INFO:tasks.rados.rados.0.smithi107.stdout:7864: finishing write tid 6 to smithi10729407-1 2020-06-12T02:38:37.743 INFO:tasks.rados.rados.0.smithi107.stdout:7864: finishing write tid 7 to smithi10729407-1 2020-06-12T02:38:37.743 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1413 (ObjNum 7863 snap 0 seq_num 7863) dirty exists 2020-06-12T02:38:37.743 INFO:tasks.rados.rados.0.smithi107.stdout:7864: left oid 1 (ObjNum 7863 snap 0 seq_num 7863) 2020-06-12T02:38:37.743 INFO:tasks.rados.rados.0.smithi107.stdout:7865: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:37.744 INFO:tasks.rados.rados.0.smithi107.stdout:7864: done (2 left) 2020-06-12T02:38:37.744 INFO:tasks.rados.rados.0.smithi107.stdout:7866: done (1 left) 2020-06-12T02:38:37.744 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:37.744 INFO:tasks.rados.rados.0.smithi107.stdout:7867: seq_num 7866 ranges {1728512=548864,2277376=729088,3006464=507904,3514368=368640} 2020-06-12T02:38:37.744 INFO:tasks.rados.rados.0.smithi107.stdout:7867: writing smithi10729407-8 from 1728512 to 2277376 tid 1 2020-06-12T02:38:37.744 INFO:tasks.rados.rados.0.smithi107.stdout:7867: writing smithi10729407-8 from 2277376 to 3006464 tid 2 2020-06-12T02:38:37.745 INFO:tasks.rados.rados.0.smithi107.stdout:7867: writing smithi10729407-8 from 3006464 to 3514368 tid 3 2020-06-12T02:38:37.745 INFO:tasks.rados.rados.0.smithi107.stdout:7867: writing smithi10729407-8 from 3514368 to 3883008 tid 4 2020-06-12T02:38:37.745 INFO:tasks.rados.rados.0.smithi107.stdout:7868: delete oid 16 current snap is 0 2020-06-12T02:38:37.745 INFO:tasks.rados.rados.0.smithi107.stdout:7865: finishing write tid 2 to smithi10729407-10 2020-06-12T02:38:37.745 INFO:tasks.rados.rados.0.smithi107.stdout:7865: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:37.746 INFO:tasks.rados.rados.0.smithi107.stdout:7865: finishing write tid 4 to smithi10729407-10 2020-06-12T02:38:37.746 INFO:tasks.rados.rados.0.smithi107.stdout:7865: finishing write tid 6 to smithi10729407-10 2020-06-12T02:38:37.746 INFO:tasks.rados.rados.0.smithi107.stdout:7865: finishing write tid 7 to smithi10729407-10 2020-06-12T02:38:37.746 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 545 (ObjNum 7864 snap 0 seq_num 7864) dirty exists 2020-06-12T02:38:37.746 INFO:tasks.rados.rados.0.smithi107.stdout:7865: left oid 10 (ObjNum 7864 snap 0 seq_num 7864) 2020-06-12T02:38:37.746 INFO:tasks.rados.rados.0.smithi107.stdout:7867: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:37.747 INFO:tasks.rados.rados.0.smithi107.stdout:7865: done (2 left) 2020-06-12T02:38:37.747 INFO:tasks.rados.rados.0.smithi107.stdout:7868: done (1 left) 2020-06-12T02:38:37.747 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:37.747 INFO:tasks.rados.rados.0.smithi107.stdout:7869: seq_num 7868 ranges {4046848=581632,4628480=499712,5128192=524288,5652480=581632} 2020-06-12T02:38:37.747 INFO:tasks.rados.rados.0.smithi107.stdout:7869: writing smithi10729407-17 from 4046848 to 4628480 tid 1 2020-06-12T02:38:37.748 INFO:tasks.rados.rados.0.smithi107.stdout:7869: writing smithi10729407-17 from 4628480 to 5128192 tid 2 2020-06-12T02:38:37.748 INFO:tasks.rados.rados.0.smithi107.stdout:7869: writing smithi10729407-17 from 5128192 to 5652480 tid 3 2020-06-12T02:38:37.748 INFO:tasks.rados.rados.0.smithi107.stdout:7869: writing smithi10729407-17 from 5652480 to 6234112 tid 4 2020-06-12T02:38:37.748 INFO:tasks.rados.rados.0.smithi107.stdout:7870: delete oid 1 current snap is 0 2020-06-12T02:38:37.748 INFO:tasks.rados.rados.0.smithi107.stdout:7867: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:37.749 INFO:tasks.rados.rados.0.smithi107.stdout:7867: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:37.749 INFO:tasks.rados.rados.0.smithi107.stdout:7867: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:37.749 INFO:tasks.rados.rados.0.smithi107.stdout:7867: finishing write tid 6 to smithi10729407-8 2020-06-12T02:38:37.749 INFO:tasks.rados.rados.0.smithi107.stdout:7867: finishing write tid 7 to smithi10729407-8 2020-06-12T02:38:37.749 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1462 (ObjNum 7866 snap 0 seq_num 7866) dirty exists 2020-06-12T02:38:37.750 INFO:tasks.rados.rados.0.smithi107.stdout:7867: left oid 8 (ObjNum 7866 snap 0 seq_num 7866) 2020-06-12T02:38:37.750 INFO:tasks.rados.rados.0.smithi107.stdout:7869: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:37.750 INFO:tasks.rados.rados.0.smithi107.stdout:7867: done (2 left) 2020-06-12T02:38:37.750 INFO:tasks.rados.rados.0.smithi107.stdout:7870: done (1 left) 2020-06-12T02:38:37.750 INFO:tasks.rados.rados.0.smithi107.stdout:7871: delete oid 10 current snap is 0 2020-06-12T02:38:37.751 INFO:tasks.rados.rados.0.smithi107.stdout:7871: done (1 left) 2020-06-12T02:38:37.751 INFO:tasks.rados.rados.0.smithi107.stdout:7872: delete oid 11 current snap is 0 2020-06-12T02:38:37.752 INFO:tasks.rados.rados.0.smithi107.stdout:7869: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:37.752 INFO:tasks.rados.rados.0.smithi107.stdout:7869: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:37.752 INFO:tasks.rados.rados.0.smithi107.stdout:7869: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:37.752 INFO:tasks.rados.rados.0.smithi107.stdout:7872: done (1 left) 2020-06-12T02:38:37.753 INFO:tasks.rados.rados.0.smithi107.stdout:7873: delete oid 16 current snap is 0 2020-06-12T02:38:37.753 INFO:tasks.rados.rados.0.smithi107.stdout:7869: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:37.753 INFO:tasks.rados.rados.0.smithi107.stdout:7869: finishing write tid 7 to smithi10729407-17 2020-06-12T02:38:37.753 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1473 (ObjNum 7868 snap 0 seq_num 7868) dirty exists 2020-06-12T02:38:37.753 INFO:tasks.rados.rados.0.smithi107.stdout:7869: left oid 17 (ObjNum 7868 snap 0 seq_num 7868) 2020-06-12T02:38:37.754 INFO:tasks.rados.rados.0.smithi107.stdout:7869: done (1 left) 2020-06-12T02:38:37.754 INFO:tasks.rados.rados.0.smithi107.stdout:7873: done (0 left) 2020-06-12T02:38:37.754 INFO:tasks.rados.rados.0.smithi107.stdout:7874: delete oid 18 current snap is 0 2020-06-12T02:38:37.754 INFO:tasks.rados.rados.0.smithi107.stdout:7874: done (0 left) 2020-06-12T02:38:37.754 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:37.754 INFO:tasks.rados.rados.0.smithi107.stdout:7875: seq_num 7874 ranges {0=679936,679936=655360,1335296=475136,1810432=475136} 2020-06-12T02:38:37.755 INFO:tasks.rados.rados.0.smithi107.stdout:7875: writing smithi10729407-9 from 0 to 679936 tid 1 2020-06-12T02:38:37.755 INFO:tasks.rados.rados.0.smithi107.stdout:7875: writing smithi10729407-9 from 679936 to 1335296 tid 2 2020-06-12T02:38:37.755 INFO:tasks.rados.rados.0.smithi107.stdout:7875: writing smithi10729407-9 from 1335296 to 1810432 tid 3 2020-06-12T02:38:37.755 INFO:tasks.rados.rados.0.smithi107.stdout:7875: writing smithi10729407-9 from 1810432 to 2285568 tid 4 2020-06-12T02:38:37.755 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:38:37.756 INFO:tasks.rados.rados.0.smithi107.stdout:7876: seq_num 7875 ranges {0=696320,696320=540672,1236992=679936,1916928=385024} 2020-06-12T02:38:37.756 INFO:tasks.rados.rados.0.smithi107.stdout:7876: writing smithi10729407-16 from 0 to 696320 tid 1 2020-06-12T02:38:37.756 INFO:tasks.rados.rados.0.smithi107.stdout:7876: writing smithi10729407-16 from 696320 to 1236992 tid 2 2020-06-12T02:38:37.756 INFO:tasks.rados.rados.0.smithi107.stdout:7876: writing smithi10729407-16 from 1236992 to 1916928 tid 3 2020-06-12T02:38:37.756 INFO:tasks.rados.rados.0.smithi107.stdout:7876: writing smithi10729407-16 from 1916928 to 2301952 tid 4 2020-06-12T02:38:37.756 INFO:tasks.rados.rados.0.smithi107.stdout:7877: delete oid 17 current snap is 0 2020-06-12T02:38:37.757 INFO:tasks.rados.rados.0.smithi107.stdout:7875: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:37.757 INFO:tasks.rados.rados.0.smithi107.stdout:7875: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:37.757 INFO:tasks.rados.rados.0.smithi107.stdout:7875: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:37.757 INFO:tasks.rados.rados.0.smithi107.stdout:7875: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:37.757 INFO:tasks.rados.rados.0.smithi107.stdout:7875: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:37.758 INFO:tasks.rados.rados.0.smithi107.stdout:7875: finishing write tid 7 to smithi10729407-9 2020-06-12T02:38:37.758 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1482 (ObjNum 7874 snap 0 seq_num 7874) dirty exists 2020-06-12T02:38:37.758 INFO:tasks.rados.rados.0.smithi107.stdout:7875: left oid 9 (ObjNum 7874 snap 0 seq_num 7874) 2020-06-12T02:38:37.758 INFO:tasks.rados.rados.0.smithi107.stdout:7875: done (2 left) 2020-06-12T02:38:37.758 INFO:tasks.rados.rados.0.smithi107.stdout:7877: done (1 left) 2020-06-12T02:38:37.759 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:38:37.759 INFO:tasks.rados.rados.0.smithi107.stdout:7878: seq_num 7877 ranges {0=729088,729088=753664,1482752=647168,2129920=204800} 2020-06-12T02:38:37.759 INFO:tasks.rados.rados.0.smithi107.stdout:7878: writing smithi10729407-11 from 0 to 729088 tid 1 2020-06-12T02:38:37.759 INFO:tasks.rados.rados.0.smithi107.stdout:7878: writing smithi10729407-11 from 729088 to 1482752 tid 2 2020-06-12T02:38:37.759 INFO:tasks.rados.rados.0.smithi107.stdout:7878: writing smithi10729407-11 from 1482752 to 2129920 tid 3 2020-06-12T02:38:37.759 INFO:tasks.rados.rados.0.smithi107.stdout:7878: writing smithi10729407-11 from 2129920 to 2334720 tid 4 2020-06-12T02:38:37.760 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:38:37.760 INFO:tasks.rados.rados.0.smithi107.stdout:7879: seq_num 7878 ranges {0=745472,745472=638976,1384448=450560,1835008=516096} 2020-06-12T02:38:37.760 INFO:tasks.rados.rados.0.smithi107.stdout:7879: writing smithi10729407-6 from 0 to 745472 tid 1 2020-06-12T02:38:37.760 INFO:tasks.rados.rados.0.smithi107.stdout:7879: writing smithi10729407-6 from 745472 to 1384448 tid 2 2020-06-12T02:38:37.760 INFO:tasks.rados.rados.0.smithi107.stdout:7879: writing smithi10729407-6 from 1384448 to 1835008 tid 3 2020-06-12T02:38:37.761 INFO:tasks.rados.rados.0.smithi107.stdout:7879: writing smithi10729407-6 from 1835008 to 2351104 tid 4 2020-06-12T02:38:37.761 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:37.761 INFO:tasks.rados.rados.0.smithi107.stdout:7880: seq_num 7879 ranges {0=761856,761856=524288,1286144=663552,1949696=417792} 2020-06-12T02:38:37.761 INFO:tasks.rados.rados.0.smithi107.stdout:7880: writing smithi10729407-1 from 0 to 761856 tid 1 2020-06-12T02:38:37.761 INFO:tasks.rados.rados.0.smithi107.stdout:7880: writing smithi10729407-1 from 761856 to 1286144 tid 2 2020-06-12T02:38:37.761 INFO:tasks.rados.rados.0.smithi107.stdout:7880: writing smithi10729407-1 from 1286144 to 1949696 tid 3 2020-06-12T02:38:37.762 INFO:tasks.rados.rados.0.smithi107.stdout:7880: writing smithi10729407-1 from 1949696 to 2367488 tid 4 2020-06-12T02:38:37.762 INFO:tasks.rados.rados.0.smithi107.stdout:7881: delete oid 7 current snap is 0 2020-06-12T02:38:37.762 INFO:tasks.rados.rados.0.smithi107.stdout:7876: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:37.762 INFO:tasks.rados.rados.0.smithi107.stdout:7876: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:37.762 INFO:tasks.rados.rados.0.smithi107.stdout:7876: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:37.763 INFO:tasks.rados.rados.0.smithi107.stdout:7876: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:37.763 INFO:tasks.rados.rados.0.smithi107.stdout:7876: finishing write tid 6 to smithi10729407-16 2020-06-12T02:38:37.763 INFO:tasks.rados.rados.0.smithi107.stdout:7878: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:37.763 INFO:tasks.rados.rados.0.smithi107.stdout:7876: finishing write tid 7 to smithi10729407-16 2020-06-12T02:38:37.763 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1435 (ObjNum 7875 snap 0 seq_num 7875) dirty exists 2020-06-12T02:38:37.763 INFO:tasks.rados.rados.0.smithi107.stdout:7876: left oid 16 (ObjNum 7875 snap 0 seq_num 7875) 2020-06-12T02:38:37.764 INFO:tasks.rados.rados.0.smithi107.stdout:7878: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:37.764 INFO:tasks.rados.rados.0.smithi107.stdout:7878: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:37.764 INFO:tasks.rados.rados.0.smithi107.stdout:7878: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:37.764 INFO:tasks.rados.rados.0.smithi107.stdout:7878: finishing write tid 6 to smithi10729407-11 2020-06-12T02:38:37.764 INFO:tasks.rados.rados.0.smithi107.stdout:7878: finishing write tid 7 to smithi10729407-11 2020-06-12T02:38:37.764 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1405 (ObjNum 7877 snap 0 seq_num 7877) dirty exists 2020-06-12T02:38:37.765 INFO:tasks.rados.rados.0.smithi107.stdout:7878: left oid 11 (ObjNum 7877 snap 0 seq_num 7877) 2020-06-12T02:38:37.765 INFO:tasks.rados.rados.0.smithi107.stdout:7879: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:37.765 INFO:tasks.rados.rados.0.smithi107.stdout:7879: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:37.765 INFO:tasks.rados.rados.0.smithi107.stdout:7879: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:37.765 INFO:tasks.rados.rados.0.smithi107.stdout:7879: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:37.766 INFO:tasks.rados.rados.0.smithi107.stdout:7879: finishing write tid 6 to smithi10729407-6 2020-06-12T02:38:37.766 INFO:tasks.rados.rados.0.smithi107.stdout:7879: finishing write tid 7 to smithi10729407-6 2020-06-12T02:38:37.766 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1419 (ObjNum 7878 snap 0 seq_num 7878) dirty exists 2020-06-12T02:38:37.766 INFO:tasks.rados.rados.0.smithi107.stdout:7879: left oid 6 (ObjNum 7878 snap 0 seq_num 7878) 2020-06-12T02:38:37.766 INFO:tasks.rados.rados.0.smithi107.stdout:7880: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:37.767 INFO:tasks.rados.rados.0.smithi107.stdout:7876: done (4 left) 2020-06-12T02:38:37.767 INFO:tasks.rados.rados.0.smithi107.stdout:7878: done (3 left) 2020-06-12T02:38:37.767 INFO:tasks.rados.rados.0.smithi107.stdout:7879: done (2 left) 2020-06-12T02:38:37.767 INFO:tasks.rados.rados.0.smithi107.stdout:7881: done (1 left) 2020-06-12T02:38:37.767 INFO:tasks.rados.rados.0.smithi107.stdout:7882: delete oid 17 current snap is 0 2020-06-12T02:38:37.768 INFO:tasks.rados.rados.0.smithi107.stdout:7882: done (1 left) 2020-06-12T02:38:37.768 INFO:tasks.rados.rados.0.smithi107.stdout:7883: delete oid 9 current snap is 0 2020-06-12T02:38:37.768 INFO:tasks.rados.rados.0.smithi107.stdout:7883: done (1 left) 2020-06-12T02:38:37.768 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:37.768 INFO:tasks.rados.rados.0.smithi107.stdout:7884: seq_num 7883 ranges {0=32768} 2020-06-12T02:38:37.769 INFO:tasks.rados.rados.0.smithi107.stdout:7884: writing smithi10729407-5 from 0 to 32768 tid 1 2020-06-12T02:38:37.769 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:37.769 INFO:tasks.rados.rados.0.smithi107.stdout:7885: seq_num 7884 ranges {0=49152} 2020-06-12T02:38:37.769 INFO:tasks.rados.rados.0.smithi107.stdout:7885: writing smithi10729407-14 from 0 to 49152 tid 1 2020-06-12T02:38:37.769 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:37.769 INFO:tasks.rados.rados.0.smithi107.stdout:7886: seq_num 7885 ranges {0=65536} 2020-06-12T02:38:37.770 INFO:tasks.rados.rados.0.smithi107.stdout:7886: writing smithi10729407-4 from 0 to 65536 tid 1 2020-06-12T02:38:37.770 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:37.770 INFO:tasks.rados.rados.0.smithi107.stdout:7887: seq_num 7886 ranges {0=81920} 2020-06-12T02:38:37.770 INFO:tasks.rados.rados.0.smithi107.stdout:7887: writing smithi10729407-3 from 0 to 81920 tid 1 2020-06-12T02:38:37.770 INFO:tasks.rados.rados.0.smithi107.stdout:7888: delete oid 17 current snap is 0 2020-06-12T02:38:37.771 INFO:tasks.rados.rados.0.smithi107.stdout:7885: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:37.771 INFO:tasks.rados.rados.0.smithi107.stdout:7885: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:37.771 INFO:tasks.rados.rados.0.smithi107.stdout:7880: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:37.771 INFO:tasks.rados.rados.0.smithi107.stdout:7880: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:37.771 INFO:tasks.rados.rados.0.smithi107.stdout:7884: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:37.771 INFO:tasks.rados.rados.0.smithi107.stdout:7884: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:37.772 INFO:tasks.rados.rados.0.smithi107.stdout:7887: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:37.772 INFO:tasks.rados.rados.0.smithi107.stdout:7880: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:37.772 INFO:tasks.rados.rados.0.smithi107.stdout:7886: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:37.772 INFO:tasks.rados.rados.0.smithi107.stdout:7885: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:37.772 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1407 (ObjNum 7884 snap 0 seq_num 7884) dirty exists 2020-06-12T02:38:37.773 INFO:tasks.rados.rados.0.smithi107.stdout:7885: left oid 14 (ObjNum 7884 snap 0 seq_num 7884) 2020-06-12T02:38:37.773 INFO:tasks.rados.rados.0.smithi107.stdout:7884: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:37.773 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 786 (ObjNum 7883 snap 0 seq_num 7883) dirty exists 2020-06-12T02:38:37.773 INFO:tasks.rados.rados.0.smithi107.stdout:7884: left oid 5 (ObjNum 7883 snap 0 seq_num 7883) 2020-06-12T02:38:37.773 INFO:tasks.rados.rados.0.smithi107.stdout:7886: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:37.773 INFO:tasks.rados.rados.0.smithi107.stdout:7880: finishing write tid 6 to smithi10729407-1 2020-06-12T02:38:37.774 INFO:tasks.rados.rados.0.smithi107.stdout:7887: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:37.774 INFO:tasks.rados.rados.0.smithi107.stdout:7884: done (5 left) 2020-06-12T02:38:37.774 INFO:tasks.rados.rados.0.smithi107.stdout:7885: done (4 left) 2020-06-12T02:38:37.774 INFO:tasks.rados.rados.0.smithi107.stdout:7888: done (3 left) 2020-06-12T02:38:37.774 INFO:tasks.rados.rados.0.smithi107.stdout:7889: delete oid 6 current snap is 0 2020-06-12T02:38:37.775 INFO:tasks.rados.rados.0.smithi107.stdout:7886: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:37.775 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1477 (ObjNum 7885 snap 0 seq_num 7885) dirty exists 2020-06-12T02:38:37.775 INFO:tasks.rados.rados.0.smithi107.stdout:7886: left oid 4 (ObjNum 7885 snap 0 seq_num 7885) 2020-06-12T02:38:37.775 INFO:tasks.rados.rados.0.smithi107.stdout:7880: finishing write tid 7 to smithi10729407-1 2020-06-12T02:38:37.775 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1424 (ObjNum 7879 snap 0 seq_num 7879) dirty exists 2020-06-12T02:38:37.775 INFO:tasks.rados.rados.0.smithi107.stdout:7880: left oid 1 (ObjNum 7879 snap 0 seq_num 7879) 2020-06-12T02:38:37.776 INFO:tasks.rados.rados.0.smithi107.stdout:7887: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:37.776 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1437 (ObjNum 7886 snap 0 seq_num 7886) dirty exists 2020-06-12T02:38:37.776 INFO:tasks.rados.rados.0.smithi107.stdout:7887: left oid 3 (ObjNum 7886 snap 0 seq_num 7886) 2020-06-12T02:38:37.776 INFO:tasks.rados.rados.0.smithi107.stdout:7880: done (3 left) 2020-06-12T02:38:37.776 INFO:tasks.rados.rados.0.smithi107.stdout:7886: done (2 left) 2020-06-12T02:38:37.776 INFO:tasks.rados.rados.0.smithi107.stdout:7887: done (1 left) 2020-06-12T02:38:37.777 INFO:tasks.rados.rados.0.smithi107.stdout:7889: done (0 left) 2020-06-12T02:38:37.777 INFO:tasks.rados.rados.0.smithi107.stdout:7890: delete oid 14 current snap is 0 2020-06-12T02:38:37.777 INFO:tasks.rados.rados.0.smithi107.stdout:7890: done (0 left) 2020-06-12T02:38:37.777 INFO:tasks.rados.rados.0.smithi107.stdout:7891: delete oid 7 current snap is 0 2020-06-12T02:38:37.777 INFO:tasks.rados.rados.0.smithi107.stdout:7891: done (0 left) 2020-06-12T02:38:37.778 INFO:tasks.rados.rados.0.smithi107.stdout:7892: delete oid 17 current snap is 0 2020-06-12T02:38:37.778 INFO:tasks.rados.rados.0.smithi107.stdout:7892: done (0 left) 2020-06-12T02:38:37.778 INFO:tasks.rados.rados.0.smithi107.stdout:7893: delete oid 17 current snap is 0 2020-06-12T02:38:37.778 INFO:tasks.rados.rados.0.smithi107.stdout:7893: done (0 left) 2020-06-12T02:38:37.778 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:37.778 INFO:tasks.rados.rados.0.smithi107.stdout:7894: seq_num 7893 ranges {0=196608} 2020-06-12T02:38:37.779 INFO:tasks.rados.rados.0.smithi107.stdout:7894: writing smithi10729407-18 from 0 to 196608 tid 1 2020-06-12T02:38:37.779 INFO:tasks.rados.rados.0.smithi107.stdout:7895: delete oid 5 current snap is 0 2020-06-12T02:38:37.779 INFO:tasks.rados.rados.0.smithi107.stdout:7895: done (1 left) 2020-06-12T02:38:37.779 INFO:tasks.rados.rados.0.smithi107.stdout:7896: delete oid 20 current snap is 0 2020-06-12T02:38:37.779 INFO:tasks.rados.rados.0.smithi107.stdout:7894: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:37.780 INFO:tasks.rados.rados.0.smithi107.stdout:7896: done (1 left) 2020-06-12T02:38:37.780 INFO:tasks.rados.rados.0.smithi107.stdout:7897: delete oid 16 current snap is 0 2020-06-12T02:38:37.780 INFO:tasks.rados.rados.0.smithi107.stdout:7894: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:37.780 INFO:tasks.rados.rados.0.smithi107.stdout:7894: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:37.780 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1485 (ObjNum 7893 snap 0 seq_num 7893) dirty exists 2020-06-12T02:38:37.780 INFO:tasks.rados.rados.0.smithi107.stdout:7894: left oid 18 (ObjNum 7893 snap 0 seq_num 7893) 2020-06-12T02:38:37.781 INFO:tasks.rados.rados.0.smithi107.stdout:7894: done (1 left) 2020-06-12T02:38:37.781 INFO:tasks.rados.rados.0.smithi107.stdout:7897: done (0 left) 2020-06-12T02:38:37.781 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:37.781 INFO:tasks.rados.rados.0.smithi107.stdout:7898: seq_num 7897 ranges {3645440=262144} 2020-06-12T02:38:37.781 INFO:tasks.rados.rados.0.smithi107.stdout:7898: writing smithi10729407-15 from 3645440 to 3907584 tid 1 2020-06-12T02:38:37.781 INFO:tasks.rados.rados.0.smithi107.stdout:7899: delete oid 8 current snap is 0 2020-06-12T02:38:37.782 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:37.696+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:37.782 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-06-12T02:38:37.785 INFO:tasks.rados.rados.0.smithi107.stdout:7898: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:37.785 INFO:tasks.rados.rados.0.smithi107.stdout:7899: done (1 left) 2020-06-12T02:38:37.785 INFO:tasks.rados.rados.0.smithi107.stdout:7900: delete oid 3 current snap is 0 2020-06-12T02:38:37.785 INFO:tasks.rados.rados.0.smithi107.stdout:7898: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:37.786 INFO:tasks.rados.rados.0.smithi107.stdout:7900: done (1 left) 2020-06-12T02:38:37.786 INFO:tasks.rados.rados.0.smithi107.stdout:7901: delete oid 11 current snap is 0 2020-06-12T02:38:37.787 INFO:tasks.rados.rados.0.smithi107.stdout:7898: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:37.787 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1465 (ObjNum 7897 snap 0 seq_num 7897) dirty exists 2020-06-12T02:38:37.788 INFO:tasks.rados.rados.0.smithi107.stdout:7898: left oid 15 (ObjNum 7897 snap 0 seq_num 7897) 2020-06-12T02:38:37.788 INFO:tasks.rados.rados.0.smithi107.stdout:7898: done (1 left) 2020-06-12T02:38:37.788 INFO:tasks.rados.rados.0.smithi107.stdout:7901: done (0 left) 2020-06-12T02:38:37.788 INFO:tasks.rados.rados.0.smithi107.stdout:7902: delete oid 17 current snap is 0 2020-06-12T02:38:37.788 INFO:tasks.rados.rados.0.smithi107.stdout:7902: done (0 left) 2020-06-12T02:38:37.789 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:37.789 INFO:tasks.rados.rados.0.smithi107.stdout:7903: seq_num 7902 ranges {3907584=352256} 2020-06-12T02:38:37.789 INFO:tasks.rados.rados.0.smithi107.stdout:7903: writing smithi10729407-15 from 3907584 to 4259840 tid 1 2020-06-12T02:38:37.789 INFO:tasks.rados.rados.0.smithi107.stdout:7904: delete oid 5 current snap is 0 2020-06-12T02:38:37.790 INFO:tasks.rados.rados.0.smithi107.stdout:7904: done (1 left) 2020-06-12T02:38:37.790 INFO:tasks.rados.rados.0.smithi107.stdout:7905: delete oid 8 current snap is 0 2020-06-12T02:38:37.790 INFO:tasks.rados.rados.0.smithi107.stdout:7903: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:37.790 INFO:tasks.rados.rados.0.smithi107.stdout:7905: done (1 left) 2020-06-12T02:38:37.790 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:38:37.790 INFO:tasks.rados.rados.0.smithi107.stdout:7906: seq_num 7905 ranges {0=401408} 2020-06-12T02:38:37.791 INFO:tasks.rados.rados.0.smithi107.stdout:7906: writing smithi10729407-11 from 0 to 401408 tid 1 2020-06-12T02:38:37.791 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:38:37.791 INFO:tasks.rados.rados.0.smithi107.stdout:7907: seq_num 7906 ranges {0=417792} 2020-06-12T02:38:37.791 INFO:tasks.rados.rados.0.smithi107.stdout:7907: writing smithi10729407-10 from 0 to 417792 tid 1 2020-06-12T02:38:37.791 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:38:37.792 INFO:tasks.rados.rados.0.smithi107.stdout:7908: seq_num 7907 ranges {1949696=434176} 2020-06-12T02:38:37.792 INFO:tasks.rados.rados.0.smithi107.stdout:7908: writing smithi10729407-2 from 1949696 to 2383872 tid 1 2020-06-12T02:38:37.792 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:37.792 INFO:tasks.rados.rados.0.smithi107.stdout:7909: seq_num 7908 ranges {753664=450560} 2020-06-12T02:38:37.794 INFO:tasks.rados.rados.0.smithi107.stdout:7909: writing smithi10729407-12 from 753664 to 1204224 tid 1 2020-06-12T02:38:37.794 INFO:tasks.rados.rados.0.smithi107.stdout:7910: delete oid 19 current snap is 0 2020-06-12T02:38:37.794 INFO:tasks.rados.rados.0.smithi107.stdout:7903: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:37.794 INFO:tasks.rados.rados.0.smithi107.stdout:7903: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:37.795 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1468 (ObjNum 7902 snap 0 seq_num 7902) dirty exists 2020-06-12T02:38:37.795 INFO:tasks.rados.rados.0.smithi107.stdout:7903: left oid 15 (ObjNum 7902 snap 0 seq_num 7902) 2020-06-12T02:38:37.795 INFO:tasks.rados.rados.0.smithi107.stdout:7906: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:37.795 INFO:tasks.rados.rados.0.smithi107.stdout:7907: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:37.795 INFO:tasks.rados.rados.0.smithi107.stdout:7906: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:37.796 INFO:tasks.rados.rados.0.smithi107.stdout:7906: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:37.796 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1411 (ObjNum 7905 snap 0 seq_num 7905) dirty exists 2020-06-12T02:38:37.796 INFO:tasks.rados.rados.0.smithi107.stdout:7906: left oid 11 (ObjNum 7905 snap 0 seq_num 7905) 2020-06-12T02:38:37.796 INFO:tasks.rados.rados.0.smithi107.stdout:7908: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:37.796 INFO:tasks.rados.rados.0.smithi107.stdout:7908: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:37.796 INFO:tasks.rados.rados.0.smithi107.stdout:7908: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:37.797 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 742 (ObjNum 7907 snap 0 seq_num 7907) dirty exists 2020-06-12T02:38:37.797 INFO:tasks.rados.rados.0.smithi107.stdout:7908: left oid 2 (ObjNum 7907 snap 0 seq_num 7907) 2020-06-12T02:38:37.797 INFO:tasks.rados.rados.0.smithi107.stdout:7909: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:37.798 INFO:tasks.rados.rados.0.smithi107.stdout:7909: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:37.799 INFO:tasks.rados.rados.0.smithi107.stdout:7903: done (5 left) 2020-06-12T02:38:37.799 INFO:tasks.rados.rados.0.smithi107.stdout:7906: done (4 left) 2020-06-12T02:38:37.799 INFO:tasks.rados.rados.0.smithi107.stdout:7908: done (3 left) 2020-06-12T02:38:37.799 INFO:tasks.rados.rados.0.smithi107.stdout:7910: done (2 left) 2020-06-12T02:38:37.799 INFO:tasks.rados.rados.0.smithi107.stdout:7911: delete oid 13 current snap is 0 2020-06-12T02:38:37.799 INFO:tasks.rados.rados.0.smithi107.stdout:7909: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:37.800 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1430 (ObjNum 7908 snap 0 seq_num 7908) dirty exists 2020-06-12T02:38:37.800 INFO:tasks.rados.rados.0.smithi107.stdout:7909: left oid 12 (ObjNum 7908 snap 0 seq_num 7908) 2020-06-12T02:38:37.802 INFO:tasks.rados.rados.0.smithi107.stdout:7909: done (2 left) 2020-06-12T02:38:37.802 INFO:tasks.rados.rados.0.smithi107.stdout:7911: done (1 left) 2020-06-12T02:38:37.803 INFO:tasks.rados.rados.0.smithi107.stdout:7912: delete oid 8 current snap is 0 2020-06-12T02:38:37.803 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:37.800+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:37.805 INFO:tasks.rados.rados.0.smithi107.stdout:7912: done (1 left) 2020-06-12T02:38:37.805 INFO:tasks.rados.rados.0.smithi107.stdout:7913: delete oid 2 current snap is 0 2020-06-12T02:38:37.808 INFO:tasks.rados.rados.0.smithi107.stdout:7913: done (1 left) 2020-06-12T02:38:37.808 INFO:tasks.rados.rados.0.smithi107.stdout:7914: delete oid 7 current snap is 0 2020-06-12T02:38:37.810 INFO:tasks.rados.rados.0.smithi107.stdout:7914: done (1 left) 2020-06-12T02:38:37.811 INFO:tasks.rados.rados.0.smithi107.stdout:7915: delete oid 3 current snap is 0 2020-06-12T02:38:37.816 INFO:tasks.rados.rados.0.smithi107.stdout:7915: done (1 left) 2020-06-12T02:38:37.817 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:37.817 INFO:tasks.rados.rados.0.smithi107.stdout:7916: seq_num 7915 ranges {0=565248} 2020-06-12T02:38:37.820 INFO:tasks.rados.rados.0.smithi107.stdout:7916: writing smithi10729407-3 from 0 to 565248 tid 1 2020-06-12T02:38:37.820 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:37.820 INFO:tasks.rados.rados.0.smithi107.stdout:7917: seq_num 7916 ranges {0=581632} 2020-06-12T02:38:37.823 INFO:tasks.rados.rados.0.smithi107.stdout:7917: writing smithi10729407-17 from 0 to 581632 tid 1 2020-06-12T02:38:37.824 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:38:37.824 INFO:tasks.rados.rados.0.smithi107.stdout:7918: seq_num 7917 ranges {0=598016} 2020-06-12T02:38:37.827 INFO:tasks.rados.rados.0.smithi107.stdout:7918: writing smithi10729407-6 from 0 to 598016 tid 1 2020-06-12T02:38:37.828 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:37.828 INFO:tasks.rados.rados.0.smithi107.stdout:7919: seq_num 7918 ranges {1204224=614400} 2020-06-12T02:38:37.838 INFO:tasks.rados.rados.0.smithi107.stdout:7919: writing smithi10729407-12 from 1204224 to 1818624 tid 1 2020-06-12T02:38:37.839 INFO:tasks.rados.rados.0.smithi107.stdout:7920: delete oid 20 current snap is 0 2020-06-12T02:38:37.839 INFO:tasks.rados.rados.0.smithi107.stdout:7916: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:37.839 INFO:tasks.rados.rados.0.smithi107.stdout:7916: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:37.839 INFO:tasks.rados.rados.0.smithi107.stdout:7916: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:37.840 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1442 (ObjNum 7915 snap 0 seq_num 7915) dirty exists 2020-06-12T02:38:37.840 INFO:tasks.rados.rados.0.smithi107.stdout:7916: left oid 3 (ObjNum 7915 snap 0 seq_num 7915) 2020-06-12T02:38:37.840 INFO:tasks.rados.rados.0.smithi107.stdout:7917: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:37.840 INFO:tasks.rados.rados.0.smithi107.stdout:7918: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:37.840 INFO:tasks.rados.rados.0.smithi107.stdout:7918: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:37.841 INFO:tasks.rados.rados.0.smithi107.stdout:7917: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:37.841 INFO:tasks.rados.rados.0.smithi107.stdout:7918: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:37.841 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1427 (ObjNum 7917 snap 0 seq_num 7917) dirty exists 2020-06-12T02:38:37.841 INFO:tasks.rados.rados.0.smithi107.stdout:7918: left oid 6 (ObjNum 7917 snap 0 seq_num 7917) 2020-06-12T02:38:37.842 INFO:tasks.rados.rados.0.smithi107.stdout:7917: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:37.842 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1483 (ObjNum 7916 snap 0 seq_num 7916) dirty exists 2020-06-12T02:38:37.842 INFO:tasks.rados.rados.0.smithi107.stdout:7917: left oid 17 (ObjNum 7916 snap 0 seq_num 7916) 2020-06-12T02:38:37.843 INFO:tasks.rados.rados.0.smithi107.stdout:7916: done (5 left) 2020-06-12T02:38:37.843 INFO:tasks.rados.rados.0.smithi107.stdout:7917: done (4 left) 2020-06-12T02:38:37.843 INFO:tasks.rados.rados.0.smithi107.stdout:7918: done (3 left) 2020-06-12T02:38:37.843 INFO:tasks.rados.rados.0.smithi107.stdout:7920: done (2 left) 2020-06-12T02:38:37.844 INFO:tasks.rados.rados.0.smithi107.stdout:7921: delete oid 7 current snap is 0 2020-06-12T02:38:37.845 INFO:tasks.rados.rados.0.smithi107.stdout:7921: done (2 left) 2020-06-12T02:38:37.845 INFO:tasks.rados.rados.0.smithi107.stdout:7922: delete oid 19 current snap is 0 2020-06-12T02:38:37.845 INFO:tasks.rados.rados.0.smithi107.stdout:7919: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:37.849 INFO:tasks.rados.rados.0.smithi107.stdout:7919: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:37.850 INFO:tasks.rados.rados.0.smithi107.stdout:7919: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:37.851 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1433 (ObjNum 7918 snap 0 seq_num 7918) dirty exists 2020-06-12T02:38:37.851 INFO:tasks.rados.rados.0.smithi107.stdout:7919: left oid 12 (ObjNum 7918 snap 0 seq_num 7918) 2020-06-12T02:38:37.852 INFO:tasks.rados.rados.0.smithi107.stdout:7919: done (2 left) 2020-06-12T02:38:37.852 INFO:tasks.rados.rados.0.smithi107.stdout:7922: done (1 left) 2020-06-12T02:38:37.852 INFO:tasks.rados.rados.0.smithi107.stdout:7923: delete oid 11 current snap is 0 2020-06-12T02:38:37.857 INFO:tasks.rados.rados.0.smithi107.stdout:7923: done (1 left) 2020-06-12T02:38:37.857 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:37.858 INFO:tasks.rados.rados.0.smithi107.stdout:7924: seq_num 7923 ranges {65536=704512} 2020-06-12T02:38:37.868 INFO:tasks.rados.rados.0.smithi107.stdout:7924: writing smithi10729407-4 from 65536 to 770048 tid 1 2020-06-12T02:38:37.868 INFO:tasks.rados.rados.0.smithi107.stdout:7925: delete oid 11 current snap is 0 2020-06-12T02:38:37.869 INFO:tasks.rados.rados.0.smithi107.stdout:7925: done (2 left) 2020-06-12T02:38:37.869 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:38:37.869 INFO:tasks.rados.rados.0.smithi107.stdout:7926: seq_num 7925 ranges {0=737280} 2020-06-12T02:38:37.870 INFO:tasks.rados.rados.0.smithi107.stdout:7926: writing smithi10729407-16 from 0 to 737280 tid 1 2020-06-12T02:38:37.870 INFO:tasks.rados.rados.0.smithi107.stdout:7927: delete oid 19 current snap is 0 2020-06-12T02:38:37.870 INFO:tasks.rados.rados.0.smithi107.stdout:7924: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:37.871 INFO:tasks.rados.rados.0.smithi107.stdout:7924: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:37.887 INFO:tasks.rados.rados.0.smithi107.stdout:7927: done (3 left) 2020-06-12T02:38:37.888 INFO:tasks.rados.rados.0.smithi107.stdout:7928: delete oid 20 current snap is 0 2020-06-12T02:38:37.888 INFO:tasks.rados.rados.0.smithi107.stdout:7924: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:37.888 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1485 (ObjNum 7923 snap 0 seq_num 7923) dirty exists 2020-06-12T02:38:37.889 INFO:tasks.rados.rados.0.smithi107.stdout:7924: left oid 4 (ObjNum 7923 snap 0 seq_num 7923) 2020-06-12T02:38:37.889 INFO:tasks.rados.rados.0.smithi107.stdout:7924: done (3 left) 2020-06-12T02:38:37.889 INFO:tasks.rados.rados.0.smithi107.stdout:7928: done (2 left) 2020-06-12T02:38:37.889 INFO:tasks.rados.rados.0.smithi107.stdout:7929: delete oid 17 current snap is 0 2020-06-12T02:38:37.890 INFO:tasks.rados.rados.0.smithi107.stdout:7926: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:37.890 INFO:tasks.rados.rados.0.smithi107.stdout:7929: done (2 left) 2020-06-12T02:38:37.890 INFO:tasks.rados.rados.0.smithi107.stdout:7930: delete oid 2 current snap is 0 2020-06-12T02:38:37.890 INFO:tasks.rados.rados.0.smithi107.stdout:7926: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:37.891 INFO:tasks.rados.rados.0.smithi107.stdout:7930: done (2 left) 2020-06-12T02:38:37.891 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:37.891 INFO:tasks.rados.rados.0.smithi107.stdout:7931: seq_num 7930 ranges {0=417792,417792=401408} 2020-06-12T02:38:37.891 INFO:tasks.rados.rados.0.smithi107.stdout:7931: writing smithi10729407-7 from 0 to 417792 tid 1 2020-06-12T02:38:37.892 INFO:tasks.rados.rados.0.smithi107.stdout:7931: writing smithi10729407-7 from 417792 to 819200 tid 2 2020-06-12T02:38:37.892 INFO:tasks.rados.rados.0.smithi107.stdout:7932: delete oid 19 current snap is 0 2020-06-12T02:38:37.892 INFO:tasks.rados.rados.0.smithi107.stdout:7926: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:37.892 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1444 (ObjNum 7925 snap 0 seq_num 7925) dirty exists 2020-06-12T02:38:37.893 INFO:tasks.rados.rados.0.smithi107.stdout:7926: left oid 16 (ObjNum 7925 snap 0 seq_num 7925) 2020-06-12T02:38:37.893 INFO:tasks.rados.rados.0.smithi107.stdout:7926: done (3 left) 2020-06-12T02:38:37.894 INFO:tasks.rados.rados.0.smithi107.stdout:7932: done (2 left) 2020-06-12T02:38:37.894 INFO:tasks.rados.rados.0.smithi107.stdout:7933: delete oid 5 current snap is 0 2020-06-12T02:38:37.894 INFO:tasks.rados.rados.0.smithi107.stdout:7931: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:37.895 INFO:tasks.rados.rados.0.smithi107.stdout:7931: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:37.895 INFO:tasks.rados.rados.0.smithi107.stdout:7931: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:37.895 INFO:tasks.rados.rados.0.smithi107.stdout:7933: done (2 left) 2020-06-12T02:38:37.895 INFO:tasks.rados.rados.0.smithi107.stdout:7934: delete oid 3 current snap is 0 2020-06-12T02:38:37.896 INFO:tasks.rados.rados.0.smithi107.stdout:7931: finishing write tid 5 to smithi10729407-7 2020-06-12T02:38:37.896 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1451 (ObjNum 7930 snap 0 seq_num 7930) dirty exists 2020-06-12T02:38:37.897 INFO:tasks.rados.rados.0.smithi107.stdout:7931: left oid 7 (ObjNum 7930 snap 0 seq_num 7930) 2020-06-12T02:38:37.899 INFO:tasks.rados.rados.0.smithi107.stdout:7931: done (2 left) 2020-06-12T02:38:37.899 INFO:tasks.rados.rados.0.smithi107.stdout:7934: done (1 left) 2020-06-12T02:38:37.899 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:37.900 INFO:tasks.rados.rados.0.smithi107.stdout:7935: seq_num 7934 ranges {196608=483328,679936=401408} 2020-06-12T02:38:37.903 INFO:tasks.rados.rados.0.smithi107.stdout:7935: writing smithi10729407-18 from 196608 to 679936 tid 1 2020-06-12T02:38:37.903 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:37.900+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:37.905 INFO:tasks.rados.rados.0.smithi107.stdout:7935: writing smithi10729407-18 from 679936 to 1081344 tid 2 2020-06-12T02:38:37.905 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:37.905 INFO:tasks.rados.rados.0.smithi107.stdout:7936: seq_num 7935 ranges {4259840=499712,4759552=401408} 2020-06-12T02:38:37.927 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:37.928 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:38:37.928 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2020-06-12T02:38:37.928 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:37.931 INFO:tasks.rados.rados.0.smithi107.stdout:7936: writing smithi10729407-15 from 4259840 to 4759552 tid 1 2020-06-12T02:38:37.933 INFO:tasks.rados.rados.0.smithi107.stdout:7936: writing smithi10729407-15 from 4759552 to 5160960 tid 2 2020-06-12T02:38:37.933 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:37.934 INFO:tasks.rados.rados.0.smithi107.stdout:7937: seq_num 7936 ranges {770048=516096,1286144=401408} 2020-06-12T02:38:37.936 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-06-12T02:38:37.940 INFO:tasks.rados.rados.0.smithi107.stdout:7937: writing smithi10729407-4 from 770048 to 1286144 tid 1 2020-06-12T02:38:37.943 INFO:tasks.rados.rados.0.smithi107.stdout:7937: writing smithi10729407-4 from 1286144 to 1687552 tid 2 2020-06-12T02:38:37.943 INFO:tasks.rados.rados.0.smithi107.stdout:7938: delete oid 8 current snap is 0 2020-06-12T02:38:37.943 INFO:tasks.rados.rados.0.smithi107.stdout:7935: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:37.944 INFO:tasks.rados.rados.0.smithi107.stdout:7935: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:37.944 INFO:tasks.rados.rados.0.smithi107.stdout:7935: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:37.944 INFO:tasks.rados.rados.0.smithi107.stdout:7935: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:37.944 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1488 (ObjNum 7934 snap 0 seq_num 7934) dirty exists 2020-06-12T02:38:37.944 INFO:tasks.rados.rados.0.smithi107.stdout:7935: left oid 18 (ObjNum 7934 snap 0 seq_num 7934) 2020-06-12T02:38:37.945 INFO:tasks.rados.rados.0.smithi107.stdout:7936: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:37.945 INFO:tasks.rados.rados.0.smithi107.stdout:7936: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:37.946 INFO:tasks.rados.rados.0.smithi107.stdout:7936: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:37.947 INFO:tasks.rados.rados.0.smithi107.stdout:7935: done (4 left) 2020-06-12T02:38:37.947 INFO:tasks.rados.rados.0.smithi107.stdout:7938: done (3 left) 2020-06-12T02:38:37.948 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:37.948 INFO:tasks.rados.rados.0.smithi107.stdout:7939: seq_num 7938 ranges {1818624=548864,2367488=401408} 2020-06-12T02:38:37.960 INFO:tasks.rados.rados.0.smithi107.stdout:7939: writing smithi10729407-12 from 1818624 to 2367488 tid 1 2020-06-12T02:38:37.962 INFO:tasks.rados.rados.0.smithi107.stdout:7939: writing smithi10729407-12 from 2367488 to 2768896 tid 2 2020-06-12T02:38:37.963 INFO:tasks.rados.rados.0.smithi107.stdout:7940: delete oid 14 current snap is 0 2020-06-12T02:38:37.963 INFO:tasks.rados.rados.0.smithi107.stdout:7936: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:37.963 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1472 (ObjNum 7935 snap 0 seq_num 7935) dirty exists 2020-06-12T02:38:37.963 INFO:tasks.rados.rados.0.smithi107.stdout:7936: left oid 15 (ObjNum 7935 snap 0 seq_num 7935) 2020-06-12T02:38:37.964 INFO:tasks.rados.rados.0.smithi107.stdout:7937: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:37.964 INFO:tasks.rados.rados.0.smithi107.stdout:7937: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:37.964 INFO:tasks.rados.rados.0.smithi107.stdout:7937: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:37.964 INFO:tasks.rados.rados.0.smithi107.stdout:7937: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:37.964 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1489 (ObjNum 7936 snap 0 seq_num 7936) dirty exists 2020-06-12T02:38:37.965 INFO:tasks.rados.rados.0.smithi107.stdout:7937: left oid 4 (ObjNum 7936 snap 0 seq_num 7936) 2020-06-12T02:38:37.965 INFO:tasks.rados.rados.0.smithi107.stdout:7936: done (4 left) 2020-06-12T02:38:37.966 INFO:tasks.rados.rados.0.smithi107.stdout:7937: done (3 left) 2020-06-12T02:38:37.966 INFO:tasks.rados.rados.0.smithi107.stdout:7940: done (2 left) 2020-06-12T02:38:37.966 INFO:tasks.rados.rados.0.smithi107.stdout:7941: delete oid 16 current snap is 0 2020-06-12T02:38:37.966 INFO:tasks.rados.rados.0.smithi107.stdout:7939: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:37.968 INFO:tasks.rados.rados.0.smithi107.stdout:7941: done (2 left) 2020-06-12T02:38:37.968 INFO:tasks.rados.rados.0.smithi107.stdout:7942: delete oid 13 current snap is 0 2020-06-12T02:38:37.973 INFO:tasks.rados.rados.0.smithi107.stdout:7907: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:37.975 INFO:tasks.rados.rados.0.smithi107.stdout:7939: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:37.975 INFO:tasks.rados.rados.0.smithi107.stdout:7942: done (2 left) 2020-06-12T02:38:37.976 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:37.976 INFO:tasks.rados.rados.0.smithi107.stdout:7943: seq_num 7942 ranges {0=622592,622592=401408} 2020-06-12T02:38:37.979 INFO:tasks.rados.rados.0.smithi107.stdout:7943: writing smithi10729407-19 from 0 to 622592 tid 1 2020-06-12T02:38:37.981 INFO:tasks.rados.rados.0.smithi107.stdout:7943: writing smithi10729407-19 from 622592 to 1024000 tid 2 2020-06-12T02:38:37.982 INFO:tasks.rados.rados.0.smithi107.stdout:7944: delete oid 6 current snap is 0 2020-06-12T02:38:37.986 INFO:tasks.rados.rados.0.smithi107.stdout:7939: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:37.986 INFO:tasks.rados.rados.0.smithi107.stdout:7907: finishing write tid 4 to smithi10729407-10 2020-06-12T02:38:37.986 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 548 (ObjNum 7906 snap 0 seq_num 7906) dirty exists 2020-06-12T02:38:37.986 INFO:tasks.rados.rados.0.smithi107.stdout:7907: left oid 10 (ObjNum 7906 snap 0 seq_num 7906) 2020-06-12T02:38:37.987 INFO:tasks.rados.rados.0.smithi107.stdout:7939: finishing write tid 5 to smithi10729407-12 2020-06-12T02:38:37.987 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1439 (ObjNum 7938 snap 0 seq_num 7938) dirty exists 2020-06-12T02:38:37.987 INFO:tasks.rados.rados.0.smithi107.stdout:7939: left oid 12 (ObjNum 7938 snap 0 seq_num 7938) 2020-06-12T02:38:38.003 INFO:tasks.rados.rados.0.smithi107.stdout:7907: done (3 left) 2020-06-12T02:38:38.003 INFO:tasks.rados.rados.0.smithi107.stdout:7939: done (2 left) 2020-06-12T02:38:38.003 INFO:tasks.rados.rados.0.smithi107.stdout:7944: done (1 left) 2020-06-12T02:38:38.003 INFO:tasks.rados.rados.0.smithi107.stdout:7945: delete oid 8 current snap is 0 2020-06-12T02:38:38.004 INFO:tasks.rados.rados.0.smithi107.stdout:7943: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:38.005 INFO:tasks.rados.rados.0.smithi107.stdout:7945: done (1 left) 2020-06-12T02:38:38.006 INFO:tasks.rados.rados.0.smithi107.stdout:7946: delete oid 6 current snap is 0 2020-06-12T02:38:38.006 INFO:tasks.rados.rados.0.smithi107.stdout:7946: done (1 left) 2020-06-12T02:38:38.006 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:38.006 INFO:tasks.rados.rados.0.smithi107.stdout:7947: seq_num 7946 ranges {0=688128,688128=401408} 2020-06-12T02:38:38.006 INFO:tasks.rados.rados.0.smithi107.stdout:7947: writing smithi10729407-17 from 0 to 688128 tid 1 2020-06-12T02:38:38.007 INFO:tasks.rados.rados.0.smithi107.stdout:7947: writing smithi10729407-17 from 688128 to 1089536 tid 2 2020-06-12T02:38:38.007 INFO:tasks.rados.rados.0.smithi107.stdout:7948: delete oid 4 current snap is 0 2020-06-12T02:38:38.007 INFO:tasks.rados.rados.0.smithi107.stdout:7943: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:38.007 INFO:tasks.rados.rados.0.smithi107.stdout:7943: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:38.007 INFO:tasks.rados.rados.0.smithi107.stdout:7943: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:38.007 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1442 (ObjNum 7942 snap 0 seq_num 7942) dirty exists 2020-06-12T02:38:38.008 INFO:tasks.rados.rados.0.smithi107.stdout:7943: left oid 19 (ObjNum 7942 snap 0 seq_num 7942) 2020-06-12T02:38:38.008 INFO:tasks.rados.rados.0.smithi107.stdout:7947: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:38.008 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:38.002+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:38.009 INFO:tasks.rados.rados.0.smithi107.stdout:7943: done (2 left) 2020-06-12T02:38:38.009 INFO:tasks.rados.rados.0.smithi107.stdout:7948: done (1 left) 2020-06-12T02:38:38.009 INFO:tasks.rados.rados.0.smithi107.stdout:7949: delete oid 19 current snap is 0 2020-06-12T02:38:38.009 INFO:tasks.rados.rados.0.smithi107.stdout:7947: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:38.011 INFO:tasks.rados.rados.0.smithi107.stdout:7947: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:38.012 INFO:tasks.rados.rados.0.smithi107.stdout:7947: finishing write tid 5 to smithi10729407-17 2020-06-12T02:38:38.012 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1493 (ObjNum 7946 snap 0 seq_num 7946) dirty exists 2020-06-12T02:38:38.012 INFO:tasks.rados.rados.0.smithi107.stdout:7947: left oid 17 (ObjNum 7946 snap 0 seq_num 7946) 2020-06-12T02:38:38.012 INFO:tasks.rados.rados.0.smithi107.stdout:7947: done (1 left) 2020-06-12T02:38:38.012 INFO:tasks.rados.rados.0.smithi107.stdout:7949: done (0 left) 2020-06-12T02:38:38.013 INFO:tasks.rados.rados.0.smithi107.stdout:7950: delete oid 17 current snap is 0 2020-06-12T02:38:38.019 INFO:tasks.rados.rados.0.smithi107.stdout:7950: done (0 left) 2020-06-12T02:38:38.019 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:38.020 INFO:tasks.rados.rados.0.smithi107.stdout:7951: seq_num 7950 ranges {0=753664,753664=401408} 2020-06-12T02:38:38.021 INFO:tasks.rados.rados.0.smithi107.stdout:7951: writing smithi10729407-5 from 0 to 753664 tid 1 2020-06-12T02:38:38.023 INFO:tasks.rados.rados.0.smithi107.stdout:7951: writing smithi10729407-5 from 753664 to 1155072 tid 2 2020-06-12T02:38:38.023 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:38.024 INFO:tasks.rados.rados.0.smithi107.stdout:7952: seq_num 7951 ranges {0=770048,770048=401408} 2020-06-12T02:38:38.028 INFO:tasks.rados.rados.0.smithi107.stdout:7952: writing smithi10729407-2 from 0 to 770048 tid 1 2020-06-12T02:38:38.031 INFO:tasks.rados.rados.0.smithi107.stdout:7952: writing smithi10729407-2 from 770048 to 1171456 tid 2 2020-06-12T02:38:38.031 INFO:tasks.rados.rados.0.smithi107.stdout:7953: delete oid 14 current snap is 0 2020-06-12T02:38:38.031 INFO:tasks.rados.rados.0.smithi107.stdout:7951: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:38.033 INFO:tasks.rados.rados.0.smithi107.stdout:7951: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:38.033 INFO:tasks.rados.rados.0.smithi107.stdout:7951: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:38.034 INFO:tasks.rados.rados.0.smithi107.stdout:7953: done (2 left) 2020-06-12T02:38:38.035 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:38.040 INFO:tasks.rados.rados.0.smithi107.stdout:7954: seq_num 7953 ranges {0=802816,802816=401408} 2020-06-12T02:38:38.042 INFO:tasks.rados.rados.0.smithi107.stdout:7954: writing smithi10729407-4 from 0 to 802816 tid 1 2020-06-12T02:38:38.043 INFO:tasks.rados.rados.0.smithi107.stdout:7954: writing smithi10729407-4 from 802816 to 1204224 tid 2 2020-06-12T02:38:38.043 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:38.043 INFO:tasks.rados.rados.0.smithi107.stdout:7955: seq_num 7954 ranges {0=417792,417792=745472,1163264=57344} 2020-06-12T02:38:38.045 INFO:tasks.rados.rados.0.smithi107.stdout:7955: writing smithi10729407-19 from 0 to 417792 tid 1 2020-06-12T02:38:38.050 INFO:tasks.rados.rados.0.smithi107.stdout:7955: writing smithi10729407-19 from 417792 to 1163264 tid 2 2020-06-12T02:38:38.051 INFO:tasks.rados.rados.0.smithi107.stdout:7955: writing smithi10729407-19 from 1163264 to 1220608 tid 3 2020-06-12T02:38:38.051 INFO:tasks.rados.rados.0.smithi107.stdout:7956: delete oid 6 current snap is 0 2020-06-12T02:38:38.051 INFO:tasks.rados.rados.0.smithi107.stdout:7951: finishing write tid 5 to smithi10729407-5 2020-06-12T02:38:38.052 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 792 (ObjNum 7950 snap 0 seq_num 7950) dirty exists 2020-06-12T02:38:38.052 INFO:tasks.rados.rados.0.smithi107.stdout:7951: left oid 5 (ObjNum 7950 snap 0 seq_num 7950) 2020-06-12T02:38:38.053 INFO:tasks.rados.rados.0.smithi107.stdout:7952: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:38.053 INFO:tasks.rados.rados.0.smithi107.stdout:7952: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:38.054 INFO:tasks.rados.rados.0.smithi107.stdout:7952: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:38.055 INFO:tasks.rados.rados.0.smithi107.stdout:7952: finishing write tid 5 to smithi10729407-2 2020-06-12T02:38:38.055 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 747 (ObjNum 7951 snap 0 seq_num 7951) dirty exists 2020-06-12T02:38:38.055 INFO:tasks.rados.rados.0.smithi107.stdout:7952: left oid 2 (ObjNum 7951 snap 0 seq_num 7951) 2020-06-12T02:38:38.055 INFO:tasks.rados.rados.0.smithi107.stdout:7954: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:38.055 INFO:tasks.rados.rados.0.smithi107.stdout:7955: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:38.056 INFO:tasks.rados.rados.0.smithi107.stdout:7954: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:38.056 INFO:tasks.rados.rados.0.smithi107.stdout:7951: done (4 left) 2020-06-12T02:38:38.056 INFO:tasks.rados.rados.0.smithi107.stdout:7952: done (3 left) 2020-06-12T02:38:38.057 INFO:tasks.rados.rados.0.smithi107.stdout:7956: done (2 left) 2020-06-12T02:38:38.057 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:38.057 INFO:tasks.rados.rados.0.smithi107.stdout:7957: seq_num 7956 ranges {0=450560,450560=516096,966656=286720} 2020-06-12T02:38:38.058 INFO:tasks.rados.rados.0.smithi107.stdout:7957: writing smithi10729407-9 from 0 to 450560 tid 1 2020-06-12T02:38:38.062 INFO:tasks.rados.rados.0.smithi107.stdout:7957: writing smithi10729407-9 from 450560 to 966656 tid 2 2020-06-12T02:38:38.065 INFO:tasks.rados.rados.0.smithi107.stdout:7957: writing smithi10729407-9 from 966656 to 1253376 tid 3 2020-06-12T02:38:38.065 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:38.065 INFO:tasks.rados.rados.0.smithi107.stdout:7958: seq_num 7957 ranges {2367488=466944,2834432=802816} 2020-06-12T02:38:38.080 INFO:tasks.rados.rados.0.smithi107.stdout:7958: writing smithi10729407-1 from 2367488 to 2834432 tid 1 2020-06-12T02:38:38.081 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:38.081 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:38:38.081 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2020-06-12T02:38:38.081 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2020-06-12T02:38:38.081 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:38.090 INFO:tasks.rados.rados.0.smithi107.stdout:7958: writing smithi10729407-1 from 2834432 to 3637248 tid 2 2020-06-12T02:38:38.090 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:38.090 INFO:tasks.rados.rados.0.smithi107.stdout:7959: seq_num 7958 ranges {0=483328,483328=688128,1171456=114688} 2020-06-12T02:38:38.090 INFO:tasks.rados.rados.0.smithi107.stdout:7959: writing smithi10729407-6 from 0 to 483328 tid 1 2020-06-12T02:38:38.091 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-06-12T02:38:38.092 INFO:tasks.rados.rados.0.smithi107.stdout:7959: writing smithi10729407-6 from 483328 to 1171456 tid 2 2020-06-12T02:38:38.093 INFO:tasks.rados.rados.0.smithi107.stdout:7959: writing smithi10729407-6 from 1171456 to 1286144 tid 3 2020-06-12T02:38:38.093 INFO:tasks.rados.rados.0.smithi107.stdout:7960: delete oid 8 current snap is 0 2020-06-12T02:38:38.093 INFO:tasks.rados.rados.0.smithi107.stdout:7954: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:38.093 INFO:tasks.rados.rados.0.smithi107.stdout:7955: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:38.094 INFO:tasks.rados.rados.0.smithi107.stdout:7954: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:38.095 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1497 (ObjNum 7953 snap 0 seq_num 7953) dirty exists 2020-06-12T02:38:38.096 INFO:tasks.rados.rados.0.smithi107.stdout:7954: left oid 4 (ObjNum 7953 snap 0 seq_num 7953) 2020-06-12T02:38:38.097 INFO:tasks.rados.rados.0.smithi107.stdout:7955: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:38.097 INFO:tasks.rados.rados.0.smithi107.stdout:7955: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:38.097 INFO:tasks.rados.rados.0.smithi107.stdout:7957: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:38.097 INFO:tasks.rados.rados.0.smithi107.stdout:7955: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:38.097 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1447 (ObjNum 7954 snap 0 seq_num 7954) dirty exists 2020-06-12T02:38:38.098 INFO:tasks.rados.rados.0.smithi107.stdout:7955: left oid 19 (ObjNum 7954 snap 0 seq_num 7954) 2020-06-12T02:38:38.098 INFO:tasks.rados.rados.0.smithi107.stdout:7957: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:38.098 INFO:tasks.rados.rados.0.smithi107.stdout:7957: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:38.098 INFO:tasks.rados.rados.0.smithi107.stdout:7957: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:38.098 INFO:tasks.rados.rados.0.smithi107.stdout:7957: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:38.098 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1492 (ObjNum 7956 snap 0 seq_num 7956) dirty exists 2020-06-12T02:38:38.099 INFO:tasks.rados.rados.0.smithi107.stdout:7957: left oid 9 (ObjNum 7956 snap 0 seq_num 7956) 2020-06-12T02:38:38.099 INFO:tasks.rados.rados.0.smithi107.stdout:7958: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:38.100 INFO:tasks.rados.rados.0.smithi107.stdout:7958: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:38.100 INFO:tasks.rados.rados.0.smithi107.stdout:7958: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:38.100 INFO:tasks.rados.rados.0.smithi107.stdout:7954: done (5 left) 2020-06-12T02:38:38.100 INFO:tasks.rados.rados.0.smithi107.stdout:7955: done (4 left) 2020-06-12T02:38:38.101 INFO:tasks.rados.rados.0.smithi107.stdout:7957: done (3 left) 2020-06-12T02:38:38.101 INFO:tasks.rados.rados.0.smithi107.stdout:7960: done (2 left) 2020-06-12T02:38:38.101 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:38.101 INFO:tasks.rados.rados.0.smithi107.stdout:7961: seq_num 7960 ranges {0=524288,524288=499712,1024000=303104} 2020-06-12T02:38:38.102 INFO:tasks.rados.rados.0.smithi107.stdout:7961: writing smithi10729407-17 from 0 to 524288 tid 1 2020-06-12T02:38:38.104 INFO:tasks.rados.rados.0.smithi107.stdout:7961: writing smithi10729407-17 from 524288 to 1024000 tid 2 2020-06-12T02:38:38.107 INFO:tasks.rados.rados.0.smithi107.stdout:7961: writing smithi10729407-17 from 1024000 to 1327104 tid 3 2020-06-12T02:38:38.107 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:38.107 INFO:tasks.rados.rados.0.smithi107.stdout:7962: seq_num 7961 ranges {2768896=540672,3309568=786432,4096000=16384} 2020-06-12T02:38:38.107 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:38.103+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:38.125 INFO:tasks.rados.rados.0.smithi107.stdout:7962: writing smithi10729407-12 from 2768896 to 3309568 tid 1 2020-06-12T02:38:38.130 INFO:tasks.rados.rados.0.smithi107.stdout:7962: writing smithi10729407-12 from 3309568 to 4096000 tid 2 2020-06-12T02:38:38.130 INFO:tasks.rados.rados.0.smithi107.stdout:7962: writing smithi10729407-12 from 4096000 to 4112384 tid 3 2020-06-12T02:38:38.130 INFO:tasks.rados.rados.0.smithi107.stdout:7963: delete oid 9 current snap is 0 2020-06-12T02:38:38.130 INFO:tasks.rados.rados.0.smithi107.stdout:7958: finishing write tid 5 to smithi10729407-1 2020-06-12T02:38:38.130 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1433 (ObjNum 7957 snap 0 seq_num 7957) dirty exists 2020-06-12T02:38:38.131 INFO:tasks.rados.rados.0.smithi107.stdout:7958: left oid 1 (ObjNum 7957 snap 0 seq_num 7957) 2020-06-12T02:38:38.131 INFO:tasks.rados.rados.0.smithi107.stdout:7959: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:38.131 INFO:tasks.rados.rados.0.smithi107.stdout:7959: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:38.131 INFO:tasks.rados.rados.0.smithi107.stdout:7959: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:38.131 INFO:tasks.rados.rados.0.smithi107.stdout:7961: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:38.132 INFO:tasks.rados.rados.0.smithi107.stdout:7959: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:38.132 INFO:tasks.rados.rados.0.smithi107.stdout:7959: finishing write tid 6 to smithi10729407-6 2020-06-12T02:38:38.134 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1437 (ObjNum 7958 snap 0 seq_num 7958) dirty exists 2020-06-12T02:38:38.135 INFO:tasks.rados.rados.0.smithi107.stdout:7959: left oid 6 (ObjNum 7958 snap 0 seq_num 7958) 2020-06-12T02:38:38.135 INFO:tasks.rados.rados.0.smithi107.stdout:7961: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:38.135 INFO:tasks.rados.rados.0.smithi107.stdout:7961: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:38.136 INFO:tasks.rados.rados.0.smithi107.stdout:7961: finishing write tid 5 to smithi10729407-17 2020-06-12T02:38:38.136 INFO:tasks.rados.rados.0.smithi107.stdout:7961: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:38.136 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1501 (ObjNum 7960 snap 0 seq_num 7960) dirty exists 2020-06-12T02:38:38.136 INFO:tasks.rados.rados.0.smithi107.stdout:7961: left oid 17 (ObjNum 7960 snap 0 seq_num 7960) 2020-06-12T02:38:38.137 INFO:tasks.rados.rados.0.smithi107.stdout:7962: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:38.137 INFO:tasks.rados.rados.0.smithi107.stdout:7958: done (4 left) 2020-06-12T02:38:38.137 INFO:tasks.rados.rados.0.smithi107.stdout:7959: done (3 left) 2020-06-12T02:38:38.138 INFO:tasks.rados.rados.0.smithi107.stdout:7961: done (2 left) 2020-06-12T02:38:38.138 INFO:tasks.rados.rados.0.smithi107.stdout:7963: done (1 left) 2020-06-12T02:38:38.138 INFO:tasks.rados.rados.0.smithi107.stdout:7964: delete oid 8 current snap is 0 2020-06-12T02:38:38.138 INFO:tasks.rados.rados.0.smithi107.stdout:7964: done (1 left) 2020-06-12T02:38:38.138 INFO:tasks.rados.rados.0.smithi107.stdout:7965: delete oid 20 current snap is 0 2020-06-12T02:38:38.139 INFO:tasks.rados.rados.0.smithi107.stdout:7965: done (1 left) 2020-06-12T02:38:38.139 INFO:tasks.rados.rados.0.smithi107.stdout:7966: delete oid 4 current snap is 0 2020-06-12T02:38:38.139 INFO:tasks.rados.rados.0.smithi107.stdout:7962: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:38.140 INFO:tasks.rados.rados.0.smithi107.stdout:7962: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:38.144 INFO:tasks.rados.rados.0.smithi107.stdout:7966: done (1 left) 2020-06-12T02:38:38.144 INFO:tasks.rados.rados.0.smithi107.stdout:7967: delete oid 11 current snap is 0 2020-06-12T02:38:38.146 INFO:tasks.rados.rados.0.smithi107.stdout:7962: finishing write tid 5 to smithi10729407-12 2020-06-12T02:38:38.146 INFO:tasks.rados.rados.0.smithi107.stdout:7962: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:38.147 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1451 (ObjNum 7961 snap 0 seq_num 7961) dirty exists 2020-06-12T02:38:38.147 INFO:tasks.rados.rados.0.smithi107.stdout:7962: left oid 12 (ObjNum 7961 snap 0 seq_num 7961) 2020-06-12T02:38:38.147 INFO:tasks.rados.rados.0.smithi107.stdout:7962: done (1 left) 2020-06-12T02:38:38.148 INFO:tasks.rados.rados.0.smithi107.stdout:7967: done (0 left) 2020-06-12T02:38:38.148 INFO:tasks.rados.rados.0.smithi107.stdout:7968: delete oid 10 current snap is 0 2020-06-12T02:38:38.150 INFO:tasks.rados.rados.0.smithi107.stdout:7968: done (0 left) 2020-06-12T02:38:38.150 INFO:tasks.rados.rados.0.smithi107.stdout:7969: delete oid 16 current snap is 0 2020-06-12T02:38:38.155 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-06-12T02:38:38.156 INFO:tasks.rados.rados.0.smithi107.stdout:7969: done (0 left) 2020-06-12T02:38:38.158 INFO:tasks.rados.rados.0.smithi107.stdout:7970: delete oid 19 current snap is 0 2020-06-12T02:38:38.158 INFO:tasks.rados.rados.0.smithi107.stdout:7970: done (0 left) 2020-06-12T02:38:38.158 INFO:tasks.rados.rados.0.smithi107.stdout:7971: delete oid 12 current snap is 0 2020-06-12T02:38:38.161 INFO:tasks.rados.rados.0.smithi107.stdout:7971: done (0 left) 2020-06-12T02:38:38.161 INFO:tasks.rados.rados.0.smithi107.stdout:7972: delete oid 12 current snap is 0 2020-06-12T02:38:38.168 INFO:tasks.rados.rados.0.smithi107.stdout:7972: done (0 left) 2020-06-12T02:38:38.169 INFO:tasks.rados.rados.0.smithi107.stdout:7973: delete oid 9 current snap is 0 2020-06-12T02:38:38.170 INFO:tasks.rados.rados.0.smithi107.stdout:7973: done (0 left) 2020-06-12T02:38:38.170 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:38:38.170 INFO:tasks.rados.rados.0.smithi107.stdout:7974: seq_num 7973 ranges {0=737280,737280=647168,1384448=155648} 2020-06-12T02:38:38.174 INFO:tasks.rados.rados.0.smithi107.stdout:7974: writing smithi10729407-13 from 0 to 737280 tid 1 2020-06-12T02:38:38.178 INFO:tasks.rados.rados.0.smithi107.stdout:7974: writing smithi10729407-13 from 737280 to 1384448 tid 2 2020-06-12T02:38:38.179 INFO:tasks.rados.rados.0.smithi107.stdout:7974: writing smithi10729407-13 from 1384448 to 1540096 tid 3 2020-06-12T02:38:38.180 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:38.180 INFO:tasks.rados.rados.0.smithi107.stdout:7975: seq_num 7974 ranges {0=753664,753664=532480,1286144=270336} 2020-06-12T02:38:38.184 INFO:tasks.rados.rados.0.smithi107.stdout:7975: writing smithi10729407-4 from 0 to 753664 tid 1 2020-06-12T02:38:38.187 INFO:tasks.rados.rados.0.smithi107.stdout:7975: writing smithi10729407-4 from 753664 to 1286144 tid 2 2020-06-12T02:38:38.189 INFO:tasks.rados.rados.0.smithi107.stdout:7975: writing smithi10729407-4 from 1286144 to 1556480 tid 3 2020-06-12T02:38:38.190 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:38:38.191 INFO:tasks.rados.rados.0.smithi107.stdout:7976: seq_num 7975 ranges {0=770048,770048=466944,1236992=335872} 2020-06-12T02:38:38.193 INFO:tasks.rados.rados.0.smithi107.stdout:7976: writing smithi10729407-10 from 0 to 770048 tid 1 2020-06-12T02:38:38.196 INFO:tasks.rados.rados.0.smithi107.stdout:7976: writing smithi10729407-10 from 770048 to 1236992 tid 2 2020-06-12T02:38:38.199 INFO:tasks.rados.rados.0.smithi107.stdout:7976: writing smithi10729407-10 from 1236992 to 1572864 tid 3 2020-06-12T02:38:38.200 INFO:tasks.rados.rados.0.smithi107.stdout:7977: delete oid 19 current snap is 0 2020-06-12T02:38:38.200 INFO:tasks.rados.rados.0.smithi107.stdout:7974: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:38.200 INFO:tasks.rados.rados.0.smithi107.stdout:7974: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:38.200 INFO:tasks.rados.rados.0.smithi107.stdout:7974: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:38.200 INFO:tasks.rados.rados.0.smithi107.stdout:7974: finishing write tid 5 to smithi10729407-13 2020-06-12T02:38:38.201 INFO:tasks.rados.rados.0.smithi107.stdout:7974: finishing write tid 6 to smithi10729407-13 2020-06-12T02:38:38.201 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1456 (ObjNum 7973 snap 0 seq_num 7973) dirty exists 2020-06-12T02:38:38.201 INFO:tasks.rados.rados.0.smithi107.stdout:7974: left oid 13 (ObjNum 7973 snap 0 seq_num 7973) 2020-06-12T02:38:38.201 INFO:tasks.rados.rados.0.smithi107.stdout:7975: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:38.201 INFO:tasks.rados.rados.0.smithi107.stdout:7975: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:38.202 INFO:tasks.rados.rados.0.smithi107.stdout:7975: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:38.202 INFO:tasks.rados.rados.0.smithi107.stdout:7975: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:38.205 INFO:tasks.rados.rados.0.smithi107.stdout:7975: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:38.205 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1506 (ObjNum 7974 snap 0 seq_num 7974) dirty exists 2020-06-12T02:38:38.205 INFO:tasks.rados.rados.0.smithi107.stdout:7975: left oid 4 (ObjNum 7974 snap 0 seq_num 7974) 2020-06-12T02:38:38.205 INFO:tasks.rados.rados.0.smithi107.stdout:7976: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:38.207 INFO:tasks.rados.rados.0.smithi107.stdout:7974: done (3 left) 2020-06-12T02:38:38.207 INFO:tasks.rados.rados.0.smithi107.stdout:7975: done (2 left) 2020-06-12T02:38:38.207 INFO:tasks.rados.rados.0.smithi107.stdout:7977: done (1 left) 2020-06-12T02:38:38.207 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:38.207 INFO:tasks.rados.rados.0.smithi107.stdout:7978: seq_num 7977 ranges {819200=802816,1622016=638976,2260992=163840} 2020-06-12T02:38:38.208 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:38.204+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:38.219 INFO:tasks.rados.rados.0.smithi107.stdout:7978: writing smithi10729407-7 from 819200 to 1622016 tid 1 2020-06-12T02:38:38.220 INFO:tasks.rados.rados.0.smithi107.stdout:7978: writing smithi10729407-7 from 1622016 to 2260992 tid 2 2020-06-12T02:38:38.221 INFO:tasks.rados.rados.0.smithi107.stdout:7978: writing smithi10729407-7 from 2260992 to 2424832 tid 3 2020-06-12T02:38:38.221 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:38:38.221 INFO:tasks.rados.rados.0.smithi107.stdout:7979: seq_num 7978 ranges {3637248=417792,4055040=516096,4571136=409600,4980736=278528} 2020-06-12T02:38:38.243 INFO:tasks.rados.rados.0.smithi107.stdout:7979: writing smithi10729407-1 from 3637248 to 4055040 tid 1 2020-06-12T02:38:38.245 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:38.245 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2020-06-12T02:38:38.245 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2020-06-12T02:38:38.246 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2020-06-12T02:38:38.246 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.246 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.9s1 25/21 ECSubWrite(tid=615, reqid=client.4512.0:1285, at_version=25'85, trim_to=0'0, roll_forward_to=25'84))", 2020-06-12T02:38:38.246 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.117566+0000", 2020-06-12T02:38:38.246 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 65.124263311999997, 2020-06-12T02:38:38.247 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010635947999999999, 2020-06-12T02:38:38.247 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:38.247 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:38.247 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:38.247 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.247 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:38.248 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.117566+0000", 2020-06-12T02:38:38.248 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:38.248 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.248 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.248 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:38.249 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.117566+0000", 2020-06-12T02:38:38.249 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000451781 2020-06-12T02:38:38.249 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.249 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.249 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:38.250 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.118017+0000", 2020-06-12T02:38:38.250 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9958999999999999e-05 2020-06-12T02:38:38.250 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.250 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.250 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:38.250 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.118047+0000", 2020-06-12T02:38:38.251 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.694e-05 2020-06-12T02:38:38.251 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.251 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.251 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:38.251 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.118064+0000", 2020-06-12T02:38:38.252 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7629999999999995e-06 2020-06-12T02:38:38.252 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.252 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.252 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:38.252 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.118074+0000", 2020-06-12T02:38:38.252 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1209000000000003e-05 2020-06-12T02:38:38.253 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.253 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.253 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:38.253 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.118135+0000", 2020-06-12T02:38:38.253 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6164999999999999e-05 2020-06-12T02:38:38.254 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.254 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.254 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:38.254 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.118151+0000", 2020-06-12T02:38:38.254 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0099907599999999996 2020-06-12T02:38:38.254 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.255 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.255 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:38.255 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.128142+0000", 2020-06-12T02:38:38.255 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9370999999999998e-05 2020-06-12T02:38:38.255 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.256 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.256 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:38.256 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.128202+0000", 2020-06-12T02:38:38.256 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010635947999999999 2020-06-12T02:38:38.256 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:38.256 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:38.257 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:38.257 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.257 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.257 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s2 26/21 ECSubWrite(tid=409, reqid=client.4512.0:1573, at_version=26'100, trim_to=0'0, roll_forward_to=26'98))", 2020-06-12T02:38:38.257 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.892374+0000", 2020-06-12T02:38:38.258 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 64.349454949999995, 2020-06-12T02:38:38.258 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014071322000000001, 2020-06-12T02:38:38.258 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:38.258 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:38.258 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:38.259 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.259 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:38.259 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.892374+0000", 2020-06-12T02:38:38.259 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:38.259 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.259 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.260 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:38.260 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.892374+0000", 2020-06-12T02:38:38.260 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047864199999999999 2020-06-12T02:38:38.260 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.261 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.261 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:38.261 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.892853+0000", 2020-06-12T02:38:38.261 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5139000000000001e-05 2020-06-12T02:38:38.261 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.262 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.262 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:38.262 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.892878+0000", 2020-06-12T02:38:38.262 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1543999999999999e-05 2020-06-12T02:38:38.262 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.262 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.263 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:38.263 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.892899+0000", 2020-06-12T02:38:38.263 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8989999999999996e-06 2020-06-12T02:38:38.263 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.263 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.264 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:38.264 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.892909+0000", 2020-06-12T02:38:38.264 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00023009199999999999 2020-06-12T02:38:38.264 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.264 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.264 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:38.265 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.893139+0000", 2020-06-12T02:38:38.265 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0044000000000001e-05 2020-06-12T02:38:38.266 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.266 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.266 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:38.266 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.893149+0000", 2020-06-12T02:38:38.266 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013233405 2020-06-12T02:38:38.267 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.267 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.267 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:38.267 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.906383+0000", 2020-06-12T02:38:38.267 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2557000000000004e-05 2020-06-12T02:38:38.267 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.268 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.268 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:38.268 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.906445+0000", 2020-06-12T02:38:38.268 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014071322000000001 2020-06-12T02:38:38.268 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:38.269 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:38.269 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:38.269 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.269 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.269 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s2 26/21 ECSubWrite(tid=410, reqid=client.4512.0:1574, at_version=26'101, trim_to=0'0, roll_forward_to=26'98))", 2020-06-12T02:38:38.270 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.895487+0000", 2020-06-12T02:38:38.270 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 64.346342155000002, 2020-06-12T02:38:38.270 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012904807000000001, 2020-06-12T02:38:38.270 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:38.270 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:38.270 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:38.271 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.271 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:38.271 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895487+0000", 2020-06-12T02:38:38.271 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:38.271 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.272 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.272 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:38.272 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895487+0000", 2020-06-12T02:38:38.272 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8478000000000002e-05 2020-06-12T02:38:38.272 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.272 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.273 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:38.273 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895575+0000", 2020-06-12T02:38:38.273 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8672e-05 2020-06-12T02:38:38.273 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.273 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.274 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:38.274 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895594+0000", 2020-06-12T02:38:38.274 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8028000000000001e-05 2020-06-12T02:38:38.274 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.274 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.274 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:38.275 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895612+0000", 2020-06-12T02:38:38.275 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2689999999999999e-06 2020-06-12T02:38:38.275 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.275 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.275 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:38.276 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895620+0000", 2020-06-12T02:38:38.276 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0038940289999999998 2020-06-12T02:38:38.276 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.276 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.276 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:38.276 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.899514+0000", 2020-06-12T02:38:38.277 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4370000000000001e-06 2020-06-12T02:38:38.277 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.277 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.277 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:38.277 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.899522+0000", 2020-06-12T02:38:38.278 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0088184019999999995 2020-06-12T02:38:38.278 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.278 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.278 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:38.278 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.908340+0000", 2020-06-12T02:38:38.278 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1492000000000001e-05 2020-06-12T02:38:38.279 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.279 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.279 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:38.279 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.908392+0000", 2020-06-12T02:38:38.279 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012904807000000001 2020-06-12T02:38:38.280 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:38.280 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:38.280 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:38.280 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.280 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.280 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 27/21 ECSubWrite(tid=180, reqid=client.4512.0:2782, at_version=27'116, trim_to=0'0, roll_forward_to=27'115))", 2020-06-12T02:38:38.281 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:36.811166+0000", 2020-06-12T02:38:38.281 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 61.43066245, 2020-06-12T02:38:38.281 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013435703, 2020-06-12T02:38:38.281 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:38.281 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:38.282 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:38.282 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.282 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:38.282 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.811166+0000", 2020-06-12T02:38:38.282 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:38.283 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.283 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.283 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:38.283 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.811166+0000", 2020-06-12T02:38:38.283 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015779300000000001 2020-06-12T02:38:38.283 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.284 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.284 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:38.284 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.811324+0000", 2020-06-12T02:38:38.284 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8149000000000001e-05 2020-06-12T02:38:38.284 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.285 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.285 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:38.285 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.811352+0000", 2020-06-12T02:38:38.285 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6880000000000001e-05 2020-06-12T02:38:38.285 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.286 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.286 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:38.286 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.811369+0000", 2020-06-12T02:38:38.286 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1099999999999999e-06 2020-06-12T02:38:38.286 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.287 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.287 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:38.287 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.811375+0000", 2020-06-12T02:38:38.287 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0038906349999999999 2020-06-12T02:38:38.287 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.287 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.288 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:38.288 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.815266+0000", 2020-06-12T02:38:38.288 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0536e-05 2020-06-12T02:38:38.288 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.288 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.289 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:38.289 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.815277+0000", 2020-06-12T02:38:38.289 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0092955610000000008 2020-06-12T02:38:38.289 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.289 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.290 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:38.290 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.824572+0000", 2020-06-12T02:38:38.290 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0039000000000002e-05 2020-06-12T02:38:38.290 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.290 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.290 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:38.291 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.824602+0000", 2020-06-12T02:38:38.291 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013435703 2020-06-12T02:38:38.291 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:38.291 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:38.291 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:38.292 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.292 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.292 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 27/21 ECSubWrite(tid=226, reqid=client.4512.0:3334, at_version=0'0, trim_to=0'0, roll_forward_to=27'147))", 2020-06-12T02:38:38.292 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:38.627654+0000", 2020-06-12T02:38:38.292 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 59.614174697999999, 2020-06-12T02:38:38.293 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01294059, 2020-06-12T02:38:38.298 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:38.299 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:38.299 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:38.299 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.299 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:38.300 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.627654+0000", 2020-06-12T02:38:38.300 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:38.300 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.300 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.300 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:38.300 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.627654+0000", 2020-06-12T02:38:38.301 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6134000000000003e-05 2020-06-12T02:38:38.301 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.301 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.301 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:38.301 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.627690+0000", 2020-06-12T02:38:38.302 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1439999999999992e-06 2020-06-12T02:38:38.302 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.302 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.302 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:38.302 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.627699+0000", 2020-06-12T02:38:38.303 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.59e-05 2020-06-12T02:38:38.303 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.303 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.303 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:38.303 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.627715+0000", 2020-06-12T02:38:38.304 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.602e-06 2020-06-12T02:38:38.304 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.304 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.304 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:38.304 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.627724+0000", 2020-06-12T02:38:38.305 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0059999999999998e-05 2020-06-12T02:38:38.305 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.305 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.305 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:38.305 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.627784+0000", 2020-06-12T02:38:38.305 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1885999999999999e-05 2020-06-12T02:38:38.306 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.306 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.306 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:38.306 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.627796+0000", 2020-06-12T02:38:38.306 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012765789 2020-06-12T02:38:38.307 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.307 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.307 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:38.307 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.640562+0000", 2020-06-12T02:38:38.307 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3074999999999998e-05 2020-06-12T02:38:38.307 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.308 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.308 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:38.308 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.640595+0000", 2020-06-12T02:38:38.308 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01294059 2020-06-12T02:38:38.308 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:38.309 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:38.309 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:38.309 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.309 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.310 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 49/21 ECSubWrite(tid=652, reqid=client.4512.0:8809, at_version=49'402, trim_to=0'0, roll_forward_to=36'401))", 2020-06-12T02:38:38.310 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.708711+0000", 2020-06-12T02:38:38.310 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 26.533118135999999, 2020-06-12T02:38:38.310 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011533180000000001, 2020-06-12T02:38:38.310 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:38.311 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:38.311 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:38.311 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.311 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:38.311 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.708711+0000", 2020-06-12T02:38:38.311 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:38.312 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.312 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.312 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:38.312 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.708711+0000", 2020-06-12T02:38:38.312 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00023433800000000001 2020-06-12T02:38:38.313 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.313 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.313 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:38.313 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.708945+0000", 2020-06-12T02:38:38.313 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5338999999999999e-05 2020-06-12T02:38:38.313 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.314 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.314 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:38.314 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.708970+0000", 2020-06-12T02:38:38.314 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7054e-05 2020-06-12T02:38:38.314 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.315 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.315 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:38.315 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.708987+0000", 2020-06-12T02:38:38.315 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8599999999999993e-06 2020-06-12T02:38:38.315 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.315 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.316 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:38.316 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.708995+0000", 2020-06-12T02:38:38.316 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3531e-05 2020-06-12T02:38:38.316 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.316 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.317 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:38.317 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.709049+0000", 2020-06-12T02:38:38.317 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2737e-05 2020-06-12T02:38:38.317 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.317 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.317 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:38.318 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.709062+0000", 2020-06-12T02:38:38.318 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011127211999999999 2020-06-12T02:38:38.318 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.318 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.318 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:38.319 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.720189+0000", 2020-06-12T02:38:38.319 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5108999999999997e-05 2020-06-12T02:38:38.319 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.319 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.319 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:38.319 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.720244+0000", 2020-06-12T02:38:38.320 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011533180000000001 2020-06-12T02:38:38.320 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:38.320 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:38.320 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:38.320 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.321 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.321 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 49/21 ECSubWrite(tid=663, reqid=client.4512.0:8855, at_version=49'409, trim_to=0'0, roll_forward_to=49'407))", 2020-06-12T02:38:38.321 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.866635+0000", 2020-06-12T02:38:38.321 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 26.375193809999999, 2020-06-12T02:38:38.321 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01099082, 2020-06-12T02:38:38.321 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:38.322 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:38.322 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:38.322 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.322 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:38.322 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.866635+0000", 2020-06-12T02:38:38.323 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:38.323 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.323 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.323 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:38.323 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.866635+0000", 2020-06-12T02:38:38.323 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000193139 2020-06-12T02:38:38.324 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.324 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.324 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:38.324 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.866828+0000", 2020-06-12T02:38:38.324 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7207999999999999e-05 2020-06-12T02:38:38.325 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.325 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.325 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:38.325 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.866845+0000", 2020-06-12T02:38:38.325 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4484999999999999e-05 2020-06-12T02:38:38.326 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.326 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.326 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:38.326 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.866860+0000", 2020-06-12T02:38:38.326 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6740000000000002e-06 2020-06-12T02:38:38.326 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.327 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.327 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:38.327 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.866866+0000", 2020-06-12T02:38:38.327 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8099999999999997e-05 2020-06-12T02:38:38.327 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.328 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.328 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:38.328 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.866914+0000", 2020-06-12T02:38:38.328 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1518999999999999e-05 2020-06-12T02:38:38.328 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.328 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.329 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:38.329 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.866925+0000", 2020-06-12T02:38:38.329 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010641576 2020-06-12T02:38:38.329 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.329 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.330 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:38.330 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.877567+0000", 2020-06-12T02:38:38.330 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9119000000000001e-05 2020-06-12T02:38:38.330 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.330 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.331 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:38.331 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.877626+0000", 2020-06-12T02:38:38.331 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01099082 2020-06-12T02:38:38.331 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:38.331 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:38.331 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:38.332 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.332 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.332 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.9s1 49/37 ECSubWrite(tid=4126, reqid=client.4512.0:8876, at_version=49'700, trim_to=0'0, roll_forward_to=49'697))", 2020-06-12T02:38:38.332 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.918553+0000", 2020-06-12T02:38:38.332 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 26.323275739, 2020-06-12T02:38:38.333 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011086824, 2020-06-12T02:38:38.333 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:38.333 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:38.333 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:38.334 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.334 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:38.338 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.918553+0000", 2020-06-12T02:38:38.338 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:38.339 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.339 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.339 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:38.339 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.918553+0000", 2020-06-12T02:38:38.339 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00024818900000000001 2020-06-12T02:38:38.339 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.340 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.340 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:38.340 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.918801+0000", 2020-06-12T02:38:38.340 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3191e-05 2020-06-12T02:38:38.340 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.341 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.341 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:38.341 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.918815+0000", 2020-06-12T02:38:38.341 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0339999999999999e-05 2020-06-12T02:38:38.341 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.342 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.342 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:38.342 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.918825+0000", 2020-06-12T02:38:38.342 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2440000000000004e-06 2020-06-12T02:38:38.342 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.342 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.343 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:38.343 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.918829+0000", 2020-06-12T02:38:38.343 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4657000000000003e-05 2020-06-12T02:38:38.343 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.343 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.344 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:38.344 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.918874+0000", 2020-06-12T02:38:38.344 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4171e-05 2020-06-12T02:38:38.344 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.344 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.344 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:38.345 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.918888+0000", 2020-06-12T02:38:38.345 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010692896 2020-06-12T02:38:38.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.345 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.345 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:38.346 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.929581+0000", 2020-06-12T02:38:38.346 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9135999999999998e-05 2020-06-12T02:38:38.346 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.346 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.346 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:38.346 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.929640+0000", 2020-06-12T02:38:38.347 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011086824 2020-06-12T02:38:38.347 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:38.347 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:38.347 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:38.347 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.348 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.348 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.9s1 49/37 ECSubWrite(tid=4209, reqid=client.4512.0:9129, at_version=49'718, trim_to=0'0, roll_forward_to=49'716))", 2020-06-12T02:38:38.348 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:12.647408+0000", 2020-06-12T02:38:38.348 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 25.594420964000001, 2020-06-12T02:38:38.348 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01161733, 2020-06-12T02:38:38.349 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:38.349 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:38.349 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:38.349 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.349 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:38.349 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.647408+0000", 2020-06-12T02:38:38.350 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:38.350 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.350 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.350 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:38.350 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.647408+0000", 2020-06-12T02:38:38.351 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00022105000000000001 2020-06-12T02:38:38.351 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.351 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.351 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:38.351 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.647629+0000", 2020-06-12T02:38:38.351 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2779e-05 2020-06-12T02:38:38.352 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.352 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.352 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:38.352 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.647652+0000", 2020-06-12T02:38:38.352 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0165000000000001e-05 2020-06-12T02:38:38.353 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.354 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.354 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:38.354 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.647672+0000", 2020-06-12T02:38:38.354 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2609999999999993e-06 2020-06-12T02:38:38.355 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.355 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.355 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:38.355 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.647680+0000", 2020-06-12T02:38:38.355 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2991999999999997e-05 2020-06-12T02:38:38.356 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.356 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.356 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:38.356 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.647733+0000", 2020-06-12T02:38:38.356 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4059e-05 2020-06-12T02:38:38.356 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.357 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.357 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:38.357 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.647747+0000", 2020-06-12T02:38:38.357 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011169695 2020-06-12T02:38:38.357 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.358 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.358 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:38.358 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.658917+0000", 2020-06-12T02:38:38.358 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000108329 2020-06-12T02:38:38.358 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.359 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.359 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:38.359 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.659025+0000", 2020-06-12T02:38:38.359 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01161733 2020-06-12T02:38:38.359 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:38.359 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:38.360 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:38.361 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.361 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.361 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.9s1 49/37 ECSubWrite(tid=4210, reqid=client.4512.0:9130, at_version=49'719, trim_to=0'0, roll_forward_to=49'716))", 2020-06-12T02:38:38.361 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:12.653737+0000", 2020-06-12T02:38:38.361 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 25.588092171, 2020-06-12T02:38:38.362 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010763803000000001, 2020-06-12T02:38:38.362 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:38.362 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:38.362 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:38.362 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.363 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:38.363 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.653737+0000", 2020-06-12T02:38:38.363 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:38.363 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.363 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.363 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:38.364 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.653737+0000", 2020-06-12T02:38:38.364 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000464399 2020-06-12T02:38:38.364 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.364 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.364 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:38.365 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.654201+0000", 2020-06-12T02:38:38.365 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3209000000000001e-05 2020-06-12T02:38:38.365 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.365 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.365 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:38.366 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.654224+0000", 2020-06-12T02:38:38.366 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6178e-05 2020-06-12T02:38:38.366 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.366 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.366 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:38.367 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.654241+0000", 2020-06-12T02:38:38.367 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2520000000000005e-06 2020-06-12T02:38:38.367 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.367 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.367 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:38.367 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.654250+0000", 2020-06-12T02:38:38.368 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6178000000000003e-05 2020-06-12T02:38:38.368 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.368 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.368 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:38.368 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.654306+0000", 2020-06-12T02:38:38.369 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4729000000000001e-05 2020-06-12T02:38:38.369 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.369 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.369 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:38.369 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.654321+0000", 2020-06-12T02:38:38.370 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010120298 2020-06-12T02:38:38.370 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.370 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.370 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:38.371 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.664441+0000", 2020-06-12T02:38:38.371 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9559999999999999e-05 2020-06-12T02:38:38.371 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.371 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.371 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:38.372 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.664501+0000", 2020-06-12T02:38:38.372 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010763803000000001 2020-06-12T02:38:38.372 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:38.372 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:38.372 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:38.373 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.373 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.373 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.9s1 54/37 ECSubWrite(tid=4461, reqid=client.4512.0:10001, at_version=54'808, trim_to=0'0, roll_forward_to=54'806))", 2020-06-12T02:38:38.373 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:17.943598+0000", 2020-06-12T02:38:38.373 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 20.298231328, 2020-06-12T02:38:38.374 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011839257000000001, 2020-06-12T02:38:38.374 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:38.374 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:38.374 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:38.374 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.374 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:38.375 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.943598+0000", 2020-06-12T02:38:38.375 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:38.375 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.375 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.375 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:38.376 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.943598+0000", 2020-06-12T02:38:38.376 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00028123299999999998 2020-06-12T02:38:38.376 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.376 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.376 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:38.376 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.943879+0000", 2020-06-12T02:38:38.377 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6650999999999999e-05 2020-06-12T02:38:38.377 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.377 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.377 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:38.377 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.943905+0000", 2020-06-12T02:38:38.378 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0922000000000001e-05 2020-06-12T02:38:38.378 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.378 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.378 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:38.378 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.943926+0000", 2020-06-12T02:38:38.379 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8770000000000005e-06 2020-06-12T02:38:38.379 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.379 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.379 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:38.379 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.943936+0000", 2020-06-12T02:38:38.379 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4051e-05 2020-06-12T02:38:38.380 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.380 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.380 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:38.380 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.944010+0000", 2020-06-12T02:38:38.380 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3377000000000001e-05 2020-06-12T02:38:38.381 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.381 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.381 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:38.382 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.944024+0000", 2020-06-12T02:38:38.382 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011355466 2020-06-12T02:38:38.382 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.382 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.382 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:38.382 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.955379+0000", 2020-06-12T02:38:38.383 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7679999999999997e-05 2020-06-12T02:38:38.383 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.383 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.383 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:38.383 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.955437+0000", 2020-06-12T02:38:38.384 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011839257000000001 2020-06-12T02:38:38.384 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:38.384 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:38.384 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:38.384 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.384 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.385 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 59/53 ECSubWrite(tid=44, reqid=client.4512.0:14904, at_version=0'0, trim_to=0'0, roll_forward_to=59'463))", 2020-06-12T02:38:38.385 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.874670+0000", 2020-06-12T02:38:38.385 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 6.36715856, 2020-06-12T02:38:38.385 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054588419999999999, 2020-06-12T02:38:38.385 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:38.386 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:38.386 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:38.386 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.386 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:38.386 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.874670+0000", 2020-06-12T02:38:38.387 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:38.387 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.387 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.387 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:38.387 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.874670+0000", 2020-06-12T02:38:38.387 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.7840000000000001e-05 2020-06-12T02:38:38.388 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.388 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.388 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:38.388 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.874698+0000", 2020-06-12T02:38:38.388 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3350000000000007e-06 2020-06-12T02:38:38.389 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.389 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.389 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:38.389 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.874707+0000", 2020-06-12T02:38:38.389 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5979999999999997e-06 2020-06-12T02:38:38.389 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.390 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.390 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:38.390 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.874715+0000", 2020-06-12T02:38:38.390 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1420000000000003e-06 2020-06-12T02:38:38.390 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.391 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.391 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:38.391 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.874719+0000", 2020-06-12T02:38:38.391 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2641000000000001e-05 2020-06-12T02:38:38.392 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.392 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.392 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:38.392 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.874752+0000", 2020-06-12T02:38:38.392 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6160000000000002e-06 2020-06-12T02:38:38.393 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.393 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.393 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:38.393 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.874761+0000", 2020-06-12T02:38:38.393 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054451052 2020-06-12T02:38:38.394 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.394 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.394 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:38.394 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.929212+0000", 2020-06-12T02:38:38.394 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7196e-05 2020-06-12T02:38:38.394 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.395 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.395 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:38.395 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.929259+0000", 2020-06-12T02:38:38.395 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054588419999999999 2020-06-12T02:38:38.395 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:38.396 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:38.396 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:38.396 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.396 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.396 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s2 59/29 ECSubWrite(tid=7891, reqid=client.4512.0:15240, at_version=59'1239, trim_to=0'0, roll_forward_to=59'1238))", 2020-06-12T02:38:38.397 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.947393+0000", 2020-06-12T02:38:38.397 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 5.2944354840000001, 2020-06-12T02:38:38.397 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013284153, 2020-06-12T02:38:38.397 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:38.398 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:38.398 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:38.398 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.398 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:38.398 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.947393+0000", 2020-06-12T02:38:38.398 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:38.399 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.399 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.399 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:38.401 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.947393+0000", 2020-06-12T02:38:38.401 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119331 2020-06-12T02:38:38.401 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.401 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.401 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:38.402 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.947513+0000", 2020-06-12T02:38:38.402 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9915000000000002e-05 2020-06-12T02:38:38.402 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.402 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.402 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:38.403 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.947533+0000", 2020-06-12T02:38:38.403 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5891000000000001e-05 2020-06-12T02:38:38.403 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.403 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.403 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:38.403 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.947549+0000", 2020-06-12T02:38:38.404 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6919999999999996e-06 2020-06-12T02:38:38.404 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.404 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.404 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:38.404 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.947555+0000", 2020-06-12T02:38:38.405 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0016197220000000001 2020-06-12T02:38:38.405 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.405 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.405 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:38.405 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.949175+0000", 2020-06-12T02:38:38.406 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4054e-05 2020-06-12T02:38:38.406 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.406 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.406 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:38.406 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.949189+0000", 2020-06-12T02:38:38.407 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011443251999999999 2020-06-12T02:38:38.407 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.407 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.407 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:38.408 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.960632+0000", 2020-06-12T02:38:38.408 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5296000000000001e-05 2020-06-12T02:38:38.408 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.408 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.408 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:38.409 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.960678+0000", 2020-06-12T02:38:38.409 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013284153 2020-06-12T02:38:38.409 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:38.409 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:38.409 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:38.410 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.410 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.410 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s2 59/29 ECSubWrite(tid=7893, reqid=client.4512.0:15241, at_version=59'1240, trim_to=0'0, roll_forward_to=59'1238))", 2020-06-12T02:38:38.410 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.951130+0000", 2020-06-12T02:38:38.410 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 5.2906986949999997, 2020-06-12T02:38:38.411 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017482992999999999, 2020-06-12T02:38:38.411 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:38.411 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:38.411 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:38.411 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.411 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:38.412 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.951130+0000", 2020-06-12T02:38:38.412 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:38.412 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.412 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.412 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:38.413 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.951130+0000", 2020-06-12T02:38:38.413 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000113396 2020-06-12T02:38:38.413 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.413 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.413 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:38.414 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.951244+0000", 2020-06-12T02:38:38.414 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0119000000000002e-05 2020-06-12T02:38:38.414 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.414 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.414 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:38.414 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.951264+0000", 2020-06-12T02:38:38.415 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5778999999999999e-05 2020-06-12T02:38:38.415 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.415 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.415 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:38.415 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.951279+0000", 2020-06-12T02:38:38.416 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.127e-06 2020-06-12T02:38:38.416 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.416 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.416 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:38.416 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.951286+0000", 2020-06-12T02:38:38.416 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086118200000000005 2020-06-12T02:38:38.417 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.417 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.418 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:38.418 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.952147+0000", 2020-06-12T02:38:38.418 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7280000000000007e-06 2020-06-12T02:38:38.418 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.418 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.419 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:38.419 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.952157+0000", 2020-06-12T02:38:38.419 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016405067999999998 2020-06-12T02:38:38.419 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.420 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.420 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:38.420 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.968562+0000", 2020-06-12T02:38:38.420 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1594000000000001e-05 2020-06-12T02:38:38.420 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.420 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.421 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:38.421 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.968613+0000", 2020-06-12T02:38:38.421 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017482992999999999 2020-06-12T02:38:38.421 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:38.421 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:38.422 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:38.422 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.423 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.423 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s2 59/29 ECSubWrite(tid=7895, reqid=client.4512.0:15242, at_version=59'1241, trim_to=0'0, roll_forward_to=59'1238))", 2020-06-12T02:38:38.423 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.954847+0000", 2020-06-12T02:38:38.423 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 5.2869820230000002, 2020-06-12T02:38:38.424 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01380811, 2020-06-12T02:38:38.424 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:38.424 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:38.424 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:38.424 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.424 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:38.425 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954847+0000", 2020-06-12T02:38:38.425 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:38.425 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.425 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.425 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:38.426 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954847+0000", 2020-06-12T02:38:38.426 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1149000000000005e-05 2020-06-12T02:38:38.426 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.427 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.427 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:38.427 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954928+0000", 2020-06-12T02:38:38.428 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2370000000000001e-05 2020-06-12T02:38:38.428 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.428 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.428 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:38.428 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954950+0000", 2020-06-12T02:38:38.429 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6290000000000002e-05 2020-06-12T02:38:38.429 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.430 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.430 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:38.430 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954967+0000", 2020-06-12T02:38:38.430 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1500000000000004e-06 2020-06-12T02:38:38.431 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.431 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.431 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:38.431 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954973+0000", 2020-06-12T02:38:38.432 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0057361110000000003 2020-06-12T02:38:38.432 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.432 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.432 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:38.433 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.960709+0000", 2020-06-12T02:38:38.433 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3400000000000003e-06 2020-06-12T02:38:38.433 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.433 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.434 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:38.434 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.960715+0000", 2020-06-12T02:38:38.434 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0079014319999999999 2020-06-12T02:38:38.434 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.435 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.435 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:38.435 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.968617+0000", 2020-06-12T02:38:38.435 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8268e-05 2020-06-12T02:38:38.435 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.436 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.436 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:38.436 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.968655+0000", 2020-06-12T02:38:38.436 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01380811 2020-06-12T02:38:38.436 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:38.436 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:38.437 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:38.437 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.437 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.437 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s2 59/29 ECSubWrite(tid=7896, reqid=client.4512.0:15243, at_version=59'1242, trim_to=0'0, roll_forward_to=59'1238))", 2020-06-12T02:38:38.437 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.957094+0000", 2020-06-12T02:38:38.438 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 5.2847353190000002, 2020-06-12T02:38:38.438 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011598476, 2020-06-12T02:38:38.438 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:38.438 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:38.438 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:38.439 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.439 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:38.439 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.957094+0000", 2020-06-12T02:38:38.440 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:38.440 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.440 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.440 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:38.440 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.957094+0000", 2020-06-12T02:38:38.441 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0612000000000002e-05 2020-06-12T02:38:38.441 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.442 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.442 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:38.443 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.957154+0000", 2020-06-12T02:38:38.444 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9766e-05 2020-06-12T02:38:38.444 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.444 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.444 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:38.445 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.957174+0000", 2020-06-12T02:38:38.445 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5478999999999999e-05 2020-06-12T02:38:38.445 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.446 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.446 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:38.446 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.957189+0000", 2020-06-12T02:38:38.447 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6460000000000004e-06 2020-06-12T02:38:38.447 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.447 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.448 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:38.448 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.957196+0000", 2020-06-12T02:38:38.448 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0047543350000000002 2020-06-12T02:38:38.448 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.449 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.449 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:38.449 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.961950+0000", 2020-06-12T02:38:38.450 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2198000000000001e-05 2020-06-12T02:38:38.450 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.450 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.451 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:38.451 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.961963+0000", 2020-06-12T02:38:38.451 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0066945809999999998 2020-06-12T02:38:38.452 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.452 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.452 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:38.453 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.968657+0000", 2020-06-12T02:38:38.453 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4858999999999999e-05 2020-06-12T02:38:38.453 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.454 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.454 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:38.454 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.968692+0000", 2020-06-12T02:38:38.455 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011598476 2020-06-12T02:38:38.455 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:38.455 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:38.456 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:38.456 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.456 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.457 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 59/53 ECSubWrite(tid=66, reqid=client.4512.0:15277, at_version=59'477, trim_to=0'0, roll_forward_to=59'476))", 2020-06-12T02:38:38.457 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:33.069631+0000", 2020-06-12T02:38:38.457 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 5.1721981890000004, 2020-06-12T02:38:38.458 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010482168, 2020-06-12T02:38:38.458 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:38.458 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:38.459 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:38.459 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.459 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:38.460 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.069631+0000", 2020-06-12T02:38:38.460 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:38.460 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.460 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.461 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:38.461 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.069631+0000", 2020-06-12T02:38:38.461 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016329300000000001 2020-06-12T02:38:38.462 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.462 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.462 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:38.463 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.069794+0000", 2020-06-12T02:38:38.463 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5128999999999999e-05 2020-06-12T02:38:38.463 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.464 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.464 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:38.464 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.069819+0000", 2020-06-12T02:38:38.465 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0531e-05 2020-06-12T02:38:38.465 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.465 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.466 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:38.466 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.069840+0000", 2020-06-12T02:38:38.466 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1226e-05 2020-06-12T02:38:38.466 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.467 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.467 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:38.467 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.069861+0000", 2020-06-12T02:38:38.468 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1826999999999997e-05 2020-06-12T02:38:38.468 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.468 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.469 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:38.469 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.069913+0000", 2020-06-12T02:38:38.469 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1963e-05 2020-06-12T02:38:38.470 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.470 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.470 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:38.471 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.069925+0000", 2020-06-12T02:38:38.471 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010135310999999999 2020-06-12T02:38:38.471 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.471 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.472 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:38.472 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.080060+0000", 2020-06-12T02:38:38.472 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2887999999999999e-05 2020-06-12T02:38:38.473 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.473 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.473 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:38.474 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.080113+0000", 2020-06-12T02:38:38.474 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010482168 2020-06-12T02:38:38.474 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:38.475 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:38.475 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:38.475 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.475 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.476 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.9s1 59/37 ECSubWrite(tid=5986, reqid=client.4512.0:15292, at_version=59'1321, trim_to=0'0, roll_forward_to=59'1318))", 2020-06-12T02:38:38.476 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:33.119863+0000", 2020-06-12T02:38:38.476 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 5.1219657060000001, 2020-06-12T02:38:38.477 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011994942999999999, 2020-06-12T02:38:38.477 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:38.477 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:38.478 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:38.478 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.478 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:38.479 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.119863+0000", 2020-06-12T02:38:38.479 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:38.479 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.479 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.480 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:38.480 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.119863+0000", 2020-06-12T02:38:38.480 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00029855799999999998 2020-06-12T02:38:38.481 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.481 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.481 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:38.481 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.120162+0000", 2020-06-12T02:38:38.482 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0029000000000001e-05 2020-06-12T02:38:38.482 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.482 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.483 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:38.483 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.120182+0000", 2020-06-12T02:38:38.483 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7958e-05 2020-06-12T02:38:38.483 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.484 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.485 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:38.485 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.120200+0000", 2020-06-12T02:38:38.485 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0509999999999999e-06 2020-06-12T02:38:38.486 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.486 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.486 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:38.487 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.120207+0000", 2020-06-12T02:38:38.487 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000110938 2020-06-12T02:38:38.487 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.487 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.488 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:38.488 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.120318+0000", 2020-06-12T02:38:38.488 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8726999999999999e-05 2020-06-12T02:38:38.489 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.489 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.489 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:38.490 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.120336+0000", 2020-06-12T02:38:38.490 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011476471 2020-06-12T02:38:38.490 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.490 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.491 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:38.491 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.131813+0000", 2020-06-12T02:38:38.491 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5210999999999998e-05 2020-06-12T02:38:38.491 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.492 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.492 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:38.492 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.131858+0000", 2020-06-12T02:38:38.493 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011994942999999999 2020-06-12T02:38:38.493 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:38.493 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:38.493 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:38.494 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.494 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.494 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 59/53 ECSubWrite(tid=72, reqid=client.4512.0:15331, at_version=59'481, trim_to=0'0, roll_forward_to=59'480))", 2020-06-12T02:38:38.495 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:33.277833+0000", 2020-06-12T02:38:38.495 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 4.9639954450000001, 2020-06-12T02:38:38.495 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011615464000000001, 2020-06-12T02:38:38.495 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:38.496 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:38.496 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:38.496 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.497 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:38.497 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.277833+0000", 2020-06-12T02:38:38.497 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:38.497 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.498 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.498 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:38.498 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.277833+0000", 2020-06-12T02:38:38.499 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015142500000000001 2020-06-12T02:38:38.499 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.499 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.499 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:38.500 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.277985+0000", 2020-06-12T02:38:38.500 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9533000000000001e-05 2020-06-12T02:38:38.500 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.501 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.501 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:38.501 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.278004+0000", 2020-06-12T02:38:38.501 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1656e-05 2020-06-12T02:38:38.502 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.502 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.502 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:38.503 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.278016+0000", 2020-06-12T02:38:38.503 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.874e-06 2020-06-12T02:38:38.503 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.503 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.504 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:38.504 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.278022+0000", 2020-06-12T02:38:38.504 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6635000000000002e-05 2020-06-12T02:38:38.505 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.505 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.505 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:38.505 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.278079+0000", 2020-06-12T02:38:38.506 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5098e-05 2020-06-12T02:38:38.506 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.506 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.506 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:38.507 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.278094+0000", 2020-06-12T02:38:38.507 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011301432 2020-06-12T02:38:38.507 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.507 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.508 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:38.508 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.289395+0000", 2020-06-12T02:38:38.508 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3810999999999997e-05 2020-06-12T02:38:38.508 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.509 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.509 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:38.509 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.289449+0000", 2020-06-12T02:38:38.509 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011615464000000001 2020-06-12T02:38:38.510 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:38.510 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:38.510 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:38.510 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.511 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.511 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s2 61/29 ECSubWrite(tid=8614, reqid=client.4512.0:16426, at_version=61'1366, trim_to=0'0, roll_forward_to=61'1364))", 2020-06-12T02:38:38.511 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:36.535477+0000", 2020-06-12T02:38:38.512 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 1.70635172, 2020-06-12T02:38:38.512 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013175393000000001, 2020-06-12T02:38:38.512 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:38.513 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:38.513 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:38.513 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.514 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:38.514 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.535477+0000", 2020-06-12T02:38:38.514 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:38.515 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.515 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.515 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:38.515 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.535477+0000", 2020-06-12T02:38:38.516 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000108959 2020-06-12T02:38:38.516 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.516 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.517 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:38.517 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.535586+0000", 2020-06-12T02:38:38.517 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6935999999999998e-05 2020-06-12T02:38:38.518 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.518 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.518 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:38.519 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.535603+0000", 2020-06-12T02:38:38.519 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1772e-05 2020-06-12T02:38:38.519 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.520 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.520 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:38.520 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.535615+0000", 2020-06-12T02:38:38.521 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6449999999999999e-06 2020-06-12T02:38:38.521 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.521 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.522 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:38.522 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.535621+0000", 2020-06-12T02:38:38.522 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5775000000000001e-05 2020-06-12T02:38:38.523 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.523 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.523 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:38.523 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.535667+0000", 2020-06-12T02:38:38.523 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2561999999999999e-05 2020-06-12T02:38:38.524 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.524 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.524 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:38.524 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.535680+0000", 2020-06-12T02:38:38.524 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012913482 2020-06-12T02:38:38.525 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.525 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.525 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:38.525 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.548593+0000", 2020-06-12T02:38:38.525 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9262000000000003e-05 2020-06-12T02:38:38.526 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:38.526 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:38.526 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:38.526 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.548653+0000", 2020-06-12T02:38:38.526 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013175393000000001 2020-06-12T02:38:38.526 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:38.527 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:38.527 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:38.527 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:38.527 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:38.527 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:38.539 INFO:tasks.rados.rados.0.smithi107.stdout:7979: writing smithi10729407-1 from 4055040 to 4571136 tid 2 2020-06-12T02:38:38.539 INFO:tasks.rados.rados.0.smithi107.stdout:7979: writing smithi10729407-1 from 4571136 to 4980736 tid 3 2020-06-12T02:38:38.540 INFO:tasks.rados.rados.0.smithi107.stdout:7979: writing smithi10729407-1 from 4980736 to 5259264 tid 4 2020-06-12T02:38:38.540 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:38.540 INFO:tasks.rados.rados.0.smithi107.stdout:7980: seq_num 7979 ranges {0=442368,442368=802816,1245184=401408} 2020-06-12T02:38:38.540 INFO:tasks.rados.rados.0.smithi107.stdout:7980: writing smithi10729407-3 from 0 to 442368 tid 1 2020-06-12T02:38:38.540 INFO:tasks.rados.rados.0.smithi107.stdout:7980: writing smithi10729407-3 from 442368 to 1245184 tid 2 2020-06-12T02:38:38.541 INFO:tasks.rados.rados.0.smithi107.stdout:7980: writing smithi10729407-3 from 1245184 to 1646592 tid 3 2020-06-12T02:38:38.541 INFO:tasks.rados.rados.0.smithi107.stdout:7981: delete oid 19 current snap is 0 2020-06-12T02:38:38.541 INFO:tasks.rados.rados.0.smithi107.stdout:7976: finishing write tid 2 to smithi10729407-10 2020-06-12T02:38:38.541 INFO:tasks.rados.rados.0.smithi107.stdout:7976: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:38.541 INFO:tasks.rados.rados.0.smithi107.stdout:7976: finishing write tid 5 to smithi10729407-10 2020-06-12T02:38:38.542 INFO:tasks.rados.rados.0.smithi107.stdout:7976: finishing write tid 6 to smithi10729407-10 2020-06-12T02:38:38.542 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 553 (ObjNum 7975 snap 0 seq_num 7975) dirty exists 2020-06-12T02:38:38.542 INFO:tasks.rados.rados.0.smithi107.stdout:7976: left oid 10 (ObjNum 7975 snap 0 seq_num 7975) 2020-06-12T02:38:38.542 INFO:tasks.rados.rados.0.smithi107.stdout:7978: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:38.543 INFO:tasks.rados.rados.0.smithi107.stdout:7978: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:38.543 INFO:tasks.rados.rados.0.smithi107.stdout:7978: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:38.543 INFO:tasks.rados.rados.0.smithi107.stdout:7978: finishing write tid 5 to smithi10729407-7 2020-06-12T02:38:38.543 INFO:tasks.rados.rados.0.smithi107.stdout:7978: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:38.544 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1460 (ObjNum 7977 snap 0 seq_num 7977) dirty exists 2020-06-12T02:38:38.546 INFO:tasks.rados.rados.0.smithi107.stdout:7978: left oid 7 (ObjNum 7977 snap 0 seq_num 7977) 2020-06-12T02:38:38.546 INFO:tasks.rados.rados.0.smithi107.stdout:7979: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:38.546 INFO:tasks.rados.rados.0.smithi107.stdout:7979: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:38.547 INFO:tasks.rados.rados.0.smithi107.stdout:7979: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:38.547 INFO:tasks.rados.rados.0.smithi107.stdout:7979: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:38.547 INFO:tasks.rados.rados.0.smithi107.stdout:7979: finishing write tid 6 to smithi10729407-1 2020-06-12T02:38:38.547 INFO:tasks.rados.rados.0.smithi107.stdout:7980: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:38.547 INFO:tasks.rados.rados.0.smithi107.stdout:7979: finishing write tid 7 to smithi10729407-1 2020-06-12T02:38:38.547 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1442 (ObjNum 7978 snap 0 seq_num 7978) dirty exists 2020-06-12T02:38:38.548 INFO:tasks.rados.rados.0.smithi107.stdout:7979: left oid 1 (ObjNum 7978 snap 0 seq_num 7978) 2020-06-12T02:38:38.548 INFO:tasks.rados.rados.0.smithi107.stdout:7976: done (4 left) 2020-06-12T02:38:38.548 INFO:tasks.rados.rados.0.smithi107.stdout:7978: done (3 left) 2020-06-12T02:38:38.548 INFO:tasks.rados.rados.0.smithi107.stdout:7979: done (2 left) 2020-06-12T02:38:38.548 INFO:tasks.rados.rados.0.smithi107.stdout:7981: done (1 left) 2020-06-12T02:38:38.549 INFO:tasks.rados.rados.0.smithi107.stdout:7982: delete oid 5 current snap is 0 2020-06-12T02:38:38.549 INFO:tasks.rados.rados.0.smithi107.stdout:7982: done (1 left) 2020-06-12T02:38:38.549 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:38:38.549 INFO:tasks.rados.rados.0.smithi107.stdout:7983: seq_num 7982 ranges {1286144=491520,1777664=458752,2236416=745472} 2020-06-12T02:38:38.549 INFO:tasks.rados.rados.0.smithi107.stdout:7983: writing smithi10729407-6 from 1286144 to 1777664 tid 1 2020-06-12T02:38:38.549 INFO:tasks.rados.rados.0.smithi107.stdout:7983: writing smithi10729407-6 from 1777664 to 2236416 tid 2 2020-06-12T02:38:38.550 INFO:tasks.rados.rados.0.smithi107.stdout:7983: writing smithi10729407-6 from 2236416 to 2981888 tid 3 2020-06-12T02:38:38.550 INFO:tasks.rados.rados.0.smithi107.stdout:7984: delete oid 11 current snap is 0 2020-06-12T02:38:38.550 INFO:tasks.rados.rados.0.smithi107.stdout:7980: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:38.550 INFO:tasks.rados.rados.0.smithi107.stdout:7980: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:38.550 INFO:tasks.rados.rados.0.smithi107.stdout:7980: finishing write tid 5 to smithi10729407-3 2020-06-12T02:38:38.551 INFO:tasks.rados.rados.0.smithi107.stdout:7980: finishing write tid 6 to smithi10729407-3 2020-06-12T02:38:38.551 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1451 (ObjNum 7979 snap 0 seq_num 7979) dirty exists 2020-06-12T02:38:38.551 INFO:tasks.rados.rados.0.smithi107.stdout:7980: left oid 3 (ObjNum 7979 snap 0 seq_num 7979) 2020-06-12T02:38:38.551 INFO:tasks.rados.rados.0.smithi107.stdout:7983: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:38.551 INFO:tasks.rados.rados.0.smithi107.stdout:7983: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:38.551 INFO:tasks.rados.rados.0.smithi107.stdout:7980: done (2 left) 2020-06-12T02:38:38.552 INFO:tasks.rados.rados.0.smithi107.stdout:7984: done (1 left) 2020-06-12T02:38:38.552 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:38:38.552 INFO:tasks.rados.rados.0.smithi107.stdout:7985: seq_num 7984 ranges {1171456=524288,1695744=679936,2375680=524288} 2020-06-12T02:38:38.552 INFO:tasks.rados.rados.0.smithi107.stdout:7985: writing smithi10729407-2 from 1171456 to 1695744 tid 1 2020-06-12T02:38:38.553 INFO:tasks.rados.rados.0.smithi107.stdout:7985: writing smithi10729407-2 from 1695744 to 2375680 tid 2 2020-06-12T02:38:38.553 INFO:tasks.rados.rados.0.smithi107.stdout:7985: writing smithi10729407-2 from 2375680 to 2899968 tid 3 2020-06-12T02:38:38.553 INFO:tasks.rados.rados.0.smithi107.stdout:7986: delete oid 15 current snap is 0 2020-06-12T02:38:38.553 INFO:tasks.rados.rados.0.smithi107.stdout:7983: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:38.553 INFO:tasks.rados.rados.0.smithi107.stdout:7983: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:38.554 INFO:tasks.rados.rados.0.smithi107.stdout:7983: finishing write tid 6 to smithi10729407-6 2020-06-12T02:38:38.554 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1446 (ObjNum 7982 snap 0 seq_num 7982) dirty exists 2020-06-12T02:38:38.554 INFO:tasks.rados.rados.0.smithi107.stdout:7983: left oid 6 (ObjNum 7982 snap 0 seq_num 7982) 2020-06-12T02:38:38.554 INFO:tasks.rados.rados.0.smithi107.stdout:7985: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:38.554 INFO:tasks.rados.rados.0.smithi107.stdout:7983: done (2 left) 2020-06-12T02:38:38.555 INFO:tasks.rados.rados.0.smithi107.stdout:7986: done (1 left) 2020-06-12T02:38:38.555 INFO:tasks.rados.rados.0.smithi107.stdout:7987: delete oid 5 current snap is 0 2020-06-12T02:38:38.555 INFO:tasks.rados.rados.0.smithi107.stdout:7985: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:38.555 INFO:tasks.rados.rados.0.smithi107.stdout:7985: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:38.555 INFO:tasks.rados.rados.0.smithi107.stdout:7985: finishing write tid 5 to smithi10729407-2 2020-06-12T02:38:38.555 INFO:tasks.rados.rados.0.smithi107.stdout:7985: finishing write tid 6 to smithi10729407-2 2020-06-12T02:38:38.556 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 751 (ObjNum 7984 snap 0 seq_num 7984) dirty exists 2020-06-12T02:38:38.556 INFO:tasks.rados.rados.0.smithi107.stdout:7985: left oid 2 (ObjNum 7984 snap 0 seq_num 7984) 2020-06-12T02:38:38.556 INFO:tasks.rados.rados.0.smithi107.stdout:7985: done (1 left) 2020-06-12T02:38:38.556 INFO:tasks.rados.rados.0.smithi107.stdout:7987: done (0 left) 2020-06-12T02:38:38.556 INFO:tasks.rados.rados.0.smithi107.stdout:7988: delete oid 2 current snap is 0 2020-06-12T02:38:38.557 INFO:tasks.rados.rados.0.smithi107.stdout:7988: done (0 left) 2020-06-12T02:38:38.557 INFO:tasks.rados.rados.0.smithi107.stdout:7989: delete oid 13 current snap is 0 2020-06-12T02:38:38.557 INFO:tasks.rados.rados.0.smithi107.stdout:7989: done (0 left) 2020-06-12T02:38:38.557 INFO:tasks.rados.rados.0.smithi107.stdout:7990: delete oid 13 current snap is 0 2020-06-12T02:38:38.557 INFO:tasks.rados.rados.0.smithi107.stdout:7990: done (0 left) 2020-06-12T02:38:38.557 INFO:tasks.rados.rados.0.smithi107.stdout:7991: delete oid 20 current snap is 0 2020-06-12T02:38:38.558 INFO:tasks.rados.rados.0.smithi107.stdout:7991: done (0 left) 2020-06-12T02:38:38.558 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:38:38.558 INFO:tasks.rados.rados.0.smithi107.stdout:7992: seq_num 7991 ranges {0=638976,638976=712704,1351680=491520} 2020-06-12T02:38:38.558 INFO:tasks.rados.rados.0.smithi107.stdout:7992: writing smithi10729407-11 from 0 to 638976 tid 1 2020-06-12T02:38:38.558 INFO:tasks.rados.rados.0.smithi107.stdout:7992: writing smithi10729407-11 from 638976 to 1351680 tid 2 2020-06-12T02:38:38.559 INFO:tasks.rados.rados.0.smithi107.stdout:7992: writing smithi10729407-11 from 1351680 to 1843200 tid 3 2020-06-12T02:38:38.559 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:38:38.559 INFO:tasks.rados.rados.0.smithi107.stdout:7993: seq_num 7992 ranges {0=655360,655360=598016,1253376=606208} 2020-06-12T02:38:38.559 INFO:tasks.rados.rados.0.smithi107.stdout:7993: writing smithi10729407-2 from 0 to 655360 tid 1 2020-06-12T02:38:38.559 INFO:tasks.rados.rados.0.smithi107.stdout:7993: writing smithi10729407-2 from 655360 to 1253376 tid 2 2020-06-12T02:38:38.559 INFO:tasks.rados.rados.0.smithi107.stdout:7993: writing smithi10729407-2 from 1253376 to 1859584 tid 3 2020-06-12T02:38:38.560 INFO:tasks.rados.rados.0.smithi107.stdout:7994: delete oid 16 current snap is 0 2020-06-12T02:38:38.560 INFO:tasks.rados.rados.0.smithi107.stdout:7992: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:38.560 INFO:tasks.rados.rados.0.smithi107.stdout:7993: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:38.560 INFO:tasks.rados.rados.0.smithi107.stdout:7992: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:38.560 INFO:tasks.rados.rados.0.smithi107.stdout:7992: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:38.561 INFO:tasks.rados.rados.0.smithi107.stdout:7992: finishing write tid 5 to smithi10729407-11 2020-06-12T02:38:38.561 INFO:tasks.rados.rados.0.smithi107.stdout:7992: finishing write tid 6 to smithi10729407-11 2020-06-12T02:38:38.561 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1421 (ObjNum 7991 snap 0 seq_num 7991) dirty exists 2020-06-12T02:38:38.561 INFO:tasks.rados.rados.0.smithi107.stdout:7992: left oid 11 (ObjNum 7991 snap 0 seq_num 7991) 2020-06-12T02:38:38.561 INFO:tasks.rados.rados.0.smithi107.stdout:7992: done (2 left) 2020-06-12T02:38:38.561 INFO:tasks.rados.rados.0.smithi107.stdout:7994: done (1 left) 2020-06-12T02:38:38.562 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:38:38.562 INFO:tasks.rados.rados.0.smithi107.stdout:7995: seq_num 7994 ranges {1572864=688128,2260992=770048,3031040=434176} 2020-06-12T02:38:38.562 INFO:tasks.rados.rados.0.smithi107.stdout:7995: writing smithi10729407-10 from 1572864 to 2260992 tid 1 2020-06-12T02:38:38.562 INFO:tasks.rados.rados.0.smithi107.stdout:7995: writing smithi10729407-10 from 2260992 to 3031040 tid 2 2020-06-12T02:38:38.562 INFO:tasks.rados.rados.0.smithi107.stdout:7995: writing smithi10729407-10 from 3031040 to 3465216 tid 3 2020-06-12T02:38:38.563 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:38.563 INFO:tasks.rados.rados.0.smithi107.stdout:7996: seq_num 7995 ranges {0=704512,704512=655360,1359872=540672,1900544=8192} 2020-06-12T02:38:38.563 INFO:tasks.rados.rados.0.smithi107.stdout:7996: writing smithi10729407-13 from 0 to 704512 tid 1 2020-06-12T02:38:38.563 INFO:tasks.rados.rados.0.smithi107.stdout:7996: writing smithi10729407-13 from 704512 to 1359872 tid 2 2020-06-12T02:38:38.563 INFO:tasks.rados.rados.0.smithi107.stdout:7996: writing smithi10729407-13 from 1359872 to 1900544 tid 3 2020-06-12T02:38:38.563 INFO:tasks.rados.rados.0.smithi107.stdout:7996: writing smithi10729407-13 from 1900544 to 1908736 tid 4 2020-06-12T02:38:38.564 INFO:tasks.rados.rados.0.smithi107.stdout:7997: delete oid 1 current snap is 0 2020-06-12T02:38:38.564 INFO:tasks.rados.rados.0.smithi107.stdout:7993: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:38.564 INFO:tasks.rados.rados.0.smithi107.stdout:7993: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:38.564 INFO:tasks.rados.rados.0.smithi107.stdout:7993: finishing write tid 5 to smithi10729407-2 2020-06-12T02:38:38.564 INFO:tasks.rados.rados.0.smithi107.stdout:7993: finishing write tid 6 to smithi10729407-2 2020-06-12T02:38:38.564 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 756 (ObjNum 7992 snap 0 seq_num 7992) dirty exists 2020-06-12T02:38:38.565 INFO:tasks.rados.rados.0.smithi107.stdout:7993: left oid 2 (ObjNum 7992 snap 0 seq_num 7992) 2020-06-12T02:38:38.565 INFO:tasks.rados.rados.0.smithi107.stdout:7995: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:38.565 INFO:tasks.rados.rados.0.smithi107.stdout:7995: finishing write tid 2 to smithi10729407-10 2020-06-12T02:38:38.565 INFO:tasks.rados.rados.0.smithi107.stdout:7995: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:38.565 INFO:tasks.rados.rados.0.smithi107.stdout:7995: finishing write tid 5 to smithi10729407-10 2020-06-12T02:38:38.566 INFO:tasks.rados.rados.0.smithi107.stdout:7995: finishing write tid 6 to smithi10729407-10 2020-06-12T02:38:38.566 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 557 (ObjNum 7994 snap 0 seq_num 7994) dirty exists 2020-06-12T02:38:38.566 INFO:tasks.rados.rados.0.smithi107.stdout:7995: left oid 10 (ObjNum 7994 snap 0 seq_num 7994) 2020-06-12T02:38:38.566 INFO:tasks.rados.rados.0.smithi107.stdout:7996: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:38.566 INFO:tasks.rados.rados.0.smithi107.stdout:7993: done (3 left) 2020-06-12T02:38:38.566 INFO:tasks.rados.rados.0.smithi107.stdout:7995: done (2 left) 2020-06-12T02:38:38.567 INFO:tasks.rados.rados.0.smithi107.stdout:7997: done (1 left) 2020-06-12T02:38:38.567 INFO:tasks.rados.rados.0.smithi107.stdout:7998: delete oid 9 current snap is 0 2020-06-12T02:38:38.567 INFO:tasks.rados.rados.0.smithi107.stdout:7998: done (1 left) 2020-06-12T02:38:38.567 INFO:tasks.rados.rados.0.smithi107.stdout:7999: delete oid 15 current snap is 0 2020-06-12T02:38:38.567 INFO:tasks.rados.rados.0.smithi107.stdout:7999: done (1 left) 2020-06-12T02:38:38.568 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:38.568 INFO:tasks.rados.rados.0.smithi107.stdout:8000: seq_num 7999 ranges {0=778240,778240=638976,1417216=524288,1941504=40960} 2020-06-12T02:38:38.568 INFO:tasks.rados.rados.0.smithi107.stdout:8000: writing smithi10729407-19 from 0 to 778240 tid 1 2020-06-12T02:38:38.568 INFO:tasks.rados.rados.0.smithi107.stdout:8000: writing smithi10729407-19 from 778240 to 1417216 tid 2 2020-06-12T02:38:38.568 INFO:tasks.rados.rados.0.smithi107.stdout:8000: writing smithi10729407-19 from 1417216 to 1941504 tid 3 2020-06-12T02:38:38.568 INFO:tasks.rados.rados.0.smithi107.stdout:8000: writing smithi10729407-19 from 1941504 to 1982464 tid 4 2020-06-12T02:38:38.569 INFO:tasks.rados.rados.0.smithi107.stdout:8001: delete oid 3 current snap is 0 2020-06-12T02:38:38.569 INFO:tasks.rados.rados.0.smithi107.stdout:7996: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:38.569 INFO:tasks.rados.rados.0.smithi107.stdout:7996: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:38.569 INFO:tasks.rados.rados.0.smithi107.stdout:7996: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:38.569 INFO:tasks.rados.rados.0.smithi107.stdout:7996: finishing write tid 6 to smithi10729407-13 2020-06-12T02:38:38.570 INFO:tasks.rados.rados.0.smithi107.stdout:7996: finishing write tid 7 to smithi10729407-13 2020-06-12T02:38:38.570 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1467 (ObjNum 7995 snap 0 seq_num 7995) dirty exists 2020-06-12T02:38:38.570 INFO:tasks.rados.rados.0.smithi107.stdout:7996: left oid 13 (ObjNum 7995 snap 0 seq_num 7995) 2020-06-12T02:38:38.570 INFO:tasks.rados.rados.0.smithi107.stdout:7996: done (2 left) 2020-06-12T02:38:38.570 INFO:tasks.rados.rados.0.smithi107.stdout:8001: done (1 left) 2020-06-12T02:38:38.570 INFO:tasks.rados.rados.0.smithi107.stdout:8002: delete oid 8 current snap is 0 2020-06-12T02:38:38.571 INFO:tasks.rados.rados.0.smithi107.stdout:8002: done (1 left) 2020-06-12T02:38:38.571 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:38:38.571 INFO:tasks.rados.rados.0.smithi107.stdout:8003: seq_num 8002 ranges {3465216=425984,3891200=696320,4587520=696320,5283840=212992} 2020-06-12T02:38:38.571 INFO:tasks.rados.rados.0.smithi107.stdout:8003: writing smithi10729407-10 from 3465216 to 3891200 tid 1 2020-06-12T02:38:38.571 INFO:tasks.rados.rados.0.smithi107.stdout:8003: writing smithi10729407-10 from 3891200 to 4587520 tid 2 2020-06-12T02:38:38.572 INFO:tasks.rados.rados.0.smithi107.stdout:8003: writing smithi10729407-10 from 4587520 to 5283840 tid 3 2020-06-12T02:38:38.572 INFO:tasks.rados.rados.0.smithi107.stdout:8003: writing smithi10729407-10 from 5283840 to 5496832 tid 4 2020-06-12T02:38:38.572 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:38.572 INFO:tasks.rados.rados.0.smithi107.stdout:8004: seq_num 8003 ranges {0=442368,442368=581632,1024000=499712,1523712=524288} 2020-06-12T02:38:38.572 INFO:tasks.rados.rados.0.smithi107.stdout:8004: writing smithi10729407-15 from 0 to 442368 tid 1 2020-06-12T02:38:38.572 INFO:tasks.rados.rados.0.smithi107.stdout:8004: writing smithi10729407-15 from 442368 to 1024000 tid 2 2020-06-12T02:38:38.573 INFO:tasks.rados.rados.0.smithi107.stdout:8004: writing smithi10729407-15 from 1024000 to 1523712 tid 3 2020-06-12T02:38:38.573 INFO:tasks.rados.rados.0.smithi107.stdout:8004: writing smithi10729407-15 from 1523712 to 2048000 tid 4 2020-06-12T02:38:38.573 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:38.573 INFO:tasks.rados.rados.0.smithi107.stdout:8005: seq_num 8004 ranges {0=458752,458752=458752,917504=663552,1581056=483328} 2020-06-12T02:38:38.573 INFO:tasks.rados.rados.0.smithi107.stdout:8005: writing smithi10729407-8 from 0 to 458752 tid 1 2020-06-12T02:38:38.574 INFO:tasks.rados.rados.0.smithi107.stdout:8005: writing smithi10729407-8 from 458752 to 917504 tid 2 2020-06-12T02:38:38.574 INFO:tasks.rados.rados.0.smithi107.stdout:8005: writing smithi10729407-8 from 917504 to 1581056 tid 3 2020-06-12T02:38:38.574 INFO:tasks.rados.rados.0.smithi107.stdout:8005: writing smithi10729407-8 from 1581056 to 2064384 tid 4 2020-06-12T02:38:38.574 INFO:tasks.rados.rados.0.smithi107.stdout:8006: delete oid 3 current snap is 0 2020-06-12T02:38:38.574 INFO:tasks.rados.rados.0.smithi107.stdout:8000: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:38.574 INFO:tasks.rados.rados.0.smithi107.stdout:8000: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:38.575 INFO:tasks.rados.rados.0.smithi107.stdout:8000: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:38.575 INFO:tasks.rados.rados.0.smithi107.stdout:8000: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:38.575 INFO:tasks.rados.rados.0.smithi107.stdout:8000: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:38.575 INFO:tasks.rados.rados.0.smithi107.stdout:8000: finishing write tid 7 to smithi10729407-19 2020-06-12T02:38:38.575 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1461 (ObjNum 7999 snap 0 seq_num 7999) dirty exists 2020-06-12T02:38:38.576 INFO:tasks.rados.rados.0.smithi107.stdout:8000: left oid 19 (ObjNum 7999 snap 0 seq_num 7999) 2020-06-12T02:38:38.576 INFO:tasks.rados.rados.0.smithi107.stdout:8003: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:38.576 INFO:tasks.rados.rados.0.smithi107.stdout:8004: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:38.576 INFO:tasks.rados.rados.0.smithi107.stdout:8003: finishing write tid 2 to smithi10729407-10 2020-06-12T02:38:38.576 INFO:tasks.rados.rados.0.smithi107.stdout:8003: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:38.576 INFO:tasks.rados.rados.0.smithi107.stdout:8003: finishing write tid 4 to smithi10729407-10 2020-06-12T02:38:38.577 INFO:tasks.rados.rados.0.smithi107.stdout:8003: finishing write tid 6 to smithi10729407-10 2020-06-12T02:38:38.577 INFO:tasks.rados.rados.0.smithi107.stdout:8003: finishing write tid 7 to smithi10729407-10 2020-06-12T02:38:38.577 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 562 (ObjNum 8002 snap 0 seq_num 8002) dirty exists 2020-06-12T02:38:38.577 INFO:tasks.rados.rados.0.smithi107.stdout:8003: left oid 10 (ObjNum 8002 snap 0 seq_num 8002) 2020-06-12T02:38:38.577 INFO:tasks.rados.rados.0.smithi107.stdout:8004: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:38.578 INFO:tasks.rados.rados.0.smithi107.stdout:8004: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:38.578 INFO:tasks.rados.rados.0.smithi107.stdout:8000: done (4 left) 2020-06-12T02:38:38.578 INFO:tasks.rados.rados.0.smithi107.stdout:8003: done (3 left) 2020-06-12T02:38:38.578 INFO:tasks.rados.rados.0.smithi107.stdout:8006: done (2 left) 2020-06-12T02:38:38.578 INFO:tasks.rados.rados.0.smithi107.stdout:8007: delete oid 5 current snap is 0 2020-06-12T02:38:38.578 INFO:tasks.rados.rados.0.smithi107.stdout:8004: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:38.579 INFO:tasks.rados.rados.0.smithi107.stdout:8004: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:38.579 INFO:tasks.rados.rados.0.smithi107.stdout:8007: done (2 left) 2020-06-12T02:38:38.579 INFO:tasks.rados.rados.0.smithi107.stdout:8008: delete oid 12 current snap is 0 2020-06-12T02:38:38.579 INFO:tasks.rados.rados.0.smithi107.stdout:8004: finishing write tid 7 to smithi10729407-15 2020-06-12T02:38:38.579 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1484 (ObjNum 8003 snap 0 seq_num 8003) dirty exists 2020-06-12T02:38:38.580 INFO:tasks.rados.rados.0.smithi107.stdout:8004: left oid 15 (ObjNum 8003 snap 0 seq_num 8003) 2020-06-12T02:38:38.580 INFO:tasks.rados.rados.0.smithi107.stdout:8004: done (2 left) 2020-06-12T02:38:38.580 INFO:tasks.rados.rados.0.smithi107.stdout:8008: done (1 left) 2020-06-12T02:38:38.580 INFO:tasks.rados.rados.0.smithi107.stdout:8009: delete oid 7 current snap is 0 2020-06-12T02:38:38.580 INFO:tasks.rados.rados.0.smithi107.stdout:8005: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:38.580 INFO:tasks.rados.rados.0.smithi107.stdout:8009: done (1 left) 2020-06-12T02:38:38.581 INFO:tasks.rados.rados.0.smithi107.stdout:8010: delete oid 20 current snap is 0 2020-06-12T02:38:38.581 INFO:tasks.rados.rados.0.smithi107.stdout:8005: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:38.581 INFO:tasks.rados.rados.0.smithi107.stdout:8005: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:38.581 INFO:tasks.rados.rados.0.smithi107.stdout:8010: done (1 left) 2020-06-12T02:38:38.581 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:38:38.582 INFO:tasks.rados.rados.0.smithi107.stdout:8011: seq_num 8010 ranges {5496832=557056,6053888=614400,6668288=655360,7323648=335872} 2020-06-12T02:38:38.582 INFO:tasks.rados.rados.0.smithi107.stdout:8011: writing smithi10729407-10 from 5496832 to 6053888 tid 1 2020-06-12T02:38:38.582 INFO:tasks.rados.rados.0.smithi107.stdout:8011: writing smithi10729407-10 from 6053888 to 6668288 tid 2 2020-06-12T02:38:38.582 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-06-12T02:38:38.583 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:38:38.583 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:38:38.584 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:38:38.584 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:38:38.584 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:38:38.584 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:38:38.584 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:38:38.585 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2020-06-12T02:38:38.585 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-06-12T02:38:38.585 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-06-12T02:38:38.585 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-06-12T02:38:38.585 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-06-12T02:38:38.586 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-06-12T02:38:38.586 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-06-12T02:38:38.586 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-06-12T02:38:38.586 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-06-12T02:38:38.587 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:38:38.525+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:38:38.589 INFO:tasks.rados.rados.0.smithi107.stdout:8011: writing smithi10729407-10 from 6668288 to 7323648 tid 3 2020-06-12T02:38:38.589 INFO:tasks.rados.rados.0.smithi107.stdout:8011: writing smithi10729407-10 from 7323648 to 7659520 tid 4 2020-06-12T02:38:38.589 INFO:tasks.rados.rados.0.smithi107.stdout:8012: delete oid 12 current snap is 0 2020-06-12T02:38:38.590 INFO:tasks.rados.rados.0.smithi107.stdout:8005: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:38.590 INFO:tasks.rados.rados.0.smithi107.stdout:8005: finishing write tid 6 to smithi10729407-8 2020-06-12T02:38:38.590 INFO:tasks.rados.rados.0.smithi107.stdout:8005: finishing write tid 7 to smithi10729407-8 2020-06-12T02:38:38.590 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1489 (ObjNum 8004 snap 0 seq_num 8004) dirty exists 2020-06-12T02:38:38.590 INFO:tasks.rados.rados.0.smithi107.stdout:8005: left oid 8 (ObjNum 8004 snap 0 seq_num 8004) 2020-06-12T02:38:38.591 INFO:tasks.rados.rados.0.smithi107.stdout:8011: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:38.591 INFO:tasks.rados.rados.0.smithi107.stdout:8005: done (2 left) 2020-06-12T02:38:38.591 INFO:tasks.rados.rados.0.smithi107.stdout:8012: done (1 left) 2020-06-12T02:38:38.591 INFO:tasks.rados.rados.0.smithi107.stdout:8013: delete oid 11 current snap is 0 2020-06-12T02:38:38.591 INFO:tasks.rados.rados.0.smithi107.stdout:8011: finishing write tid 2 to smithi10729407-10 2020-06-12T02:38:38.591 INFO:tasks.rados.rados.0.smithi107.stdout:8013: done (1 left) 2020-06-12T02:38:38.592 INFO:tasks.rados.rados.0.smithi107.stdout:8014: delete oid 19 current snap is 0 2020-06-12T02:38:38.592 INFO:tasks.rados.rados.0.smithi107.stdout:8011: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:38.592 INFO:tasks.rados.rados.0.smithi107.stdout:8011: finishing write tid 4 to smithi10729407-10 2020-06-12T02:38:38.592 INFO:tasks.rados.rados.0.smithi107.stdout:8014: done (1 left) 2020-06-12T02:38:38.592 INFO:tasks.rados.rados.0.smithi107.stdout:8015: delete oid 1 current snap is 0 2020-06-12T02:38:38.593 INFO:tasks.rados.rados.0.smithi107.stdout:8011: finishing write tid 6 to smithi10729407-10 2020-06-12T02:38:38.593 INFO:tasks.rados.rados.0.smithi107.stdout:8011: finishing write tid 7 to smithi10729407-10 2020-06-12T02:38:38.593 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 567 (ObjNum 8010 snap 0 seq_num 8010) dirty exists 2020-06-12T02:38:38.593 INFO:tasks.rados.rados.0.smithi107.stdout:8011: left oid 10 (ObjNum 8010 snap 0 seq_num 8010) 2020-06-12T02:38:38.593 INFO:tasks.rados.rados.0.smithi107.stdout:8011: done (1 left) 2020-06-12T02:38:38.594 INFO:tasks.rados.rados.0.smithi107.stdout:8015: done (0 left) 2020-06-12T02:38:38.594 INFO:tasks.rados.rados.0.smithi107.stdout:8016: delete oid 9 current snap is 0 2020-06-12T02:38:38.594 INFO:tasks.rados.rados.0.smithi107.stdout:8016: done (0 left) 2020-06-12T02:38:38.594 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:38.594 INFO:tasks.rados.rados.0.smithi107.stdout:8017: seq_num 8016 ranges {0=655360,655360=770048,1425408=606208,2031616=229376} 2020-06-12T02:38:38.594 INFO:tasks.rados.rados.0.smithi107.stdout:8017: writing smithi10729407-7 from 0 to 655360 tid 1 2020-06-12T02:38:38.595 INFO:tasks.rados.rados.0.smithi107.stdout:8017: writing smithi10729407-7 from 655360 to 1425408 tid 2 2020-06-12T02:38:38.595 INFO:tasks.rados.rados.0.smithi107.stdout:8017: writing smithi10729407-7 from 1425408 to 2031616 tid 3 2020-06-12T02:38:38.595 INFO:tasks.rados.rados.0.smithi107.stdout:8017: writing smithi10729407-7 from 2031616 to 2260992 tid 4 2020-06-12T02:38:38.595 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:38.595 INFO:tasks.rados.rados.0.smithi107.stdout:8018: seq_num 8017 ranges {1556480=671744,2228224=655360,2883584=409600,3293184=540672} 2020-06-12T02:38:38.595 INFO:tasks.rados.rados.0.smithi107.stdout:8018: writing smithi10729407-4 from 1556480 to 2228224 tid 1 2020-06-12T02:38:38.597 INFO:tasks.rados.rados.0.smithi107.stdout:8018: writing smithi10729407-4 from 2228224 to 2883584 tid 2 2020-06-12T02:38:38.597 INFO:tasks.rados.rados.0.smithi107.stdout:8018: writing smithi10729407-4 from 2883584 to 3293184 tid 3 2020-06-12T02:38:38.597 INFO:tasks.rados.rados.0.smithi107.stdout:8018: writing smithi10729407-4 from 3293184 to 3833856 tid 4 2020-06-12T02:38:38.598 INFO:tasks.rados.rados.0.smithi107.stdout:8019: delete oid 12 current snap is 0 2020-06-12T02:38:38.598 INFO:tasks.rados.rados.0.smithi107.stdout:8017: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:38.598 INFO:tasks.rados.rados.0.smithi107.stdout:8017: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:38.598 INFO:tasks.rados.rados.0.smithi107.stdout:8017: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:38.598 INFO:tasks.rados.rados.0.smithi107.stdout:8017: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:38.599 INFO:tasks.rados.rados.0.smithi107.stdout:8017: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:38.599 INFO:tasks.rados.rados.0.smithi107.stdout:8017: finishing write tid 7 to smithi10729407-7 2020-06-12T02:38:38.599 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1473 (ObjNum 8016 snap 0 seq_num 8016) dirty exists 2020-06-12T02:38:38.599 INFO:tasks.rados.rados.0.smithi107.stdout:8017: left oid 7 (ObjNum 8016 snap 0 seq_num 8016) 2020-06-12T02:38:38.599 INFO:tasks.rados.rados.0.smithi107.stdout:8018: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:38.600 INFO:tasks.rados.rados.0.smithi107.stdout:8017: done (2 left) 2020-06-12T02:38:38.600 INFO:tasks.rados.rados.0.smithi107.stdout:8019: done (1 left) 2020-06-12T02:38:38.601 INFO:tasks.rados.rados.0.smithi107.stdout:8020: delete oid 3 current snap is 0 2020-06-12T02:38:38.602 INFO:tasks.rados.rados.0.smithi107.stdout:8020: done (1 left) 2020-06-12T02:38:38.603 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:38:38.603 INFO:tasks.rados.rados.0.smithi107.stdout:8021: seq_num 8020 ranges {0=729088,729088=712704,1441792=581632,2023424=311296} 2020-06-12T02:38:38.607 INFO:tasks.rados.rados.0.smithi107.stdout:8021: writing smithi10729407-5 from 0 to 729088 tid 1 2020-06-12T02:38:38.611 INFO:tasks.rados.rados.0.smithi107.stdout:8021: writing smithi10729407-5 from 729088 to 1441792 tid 2 2020-06-12T02:38:38.614 INFO:tasks.rados.rados.0.smithi107.stdout:8021: writing smithi10729407-5 from 1441792 to 2023424 tid 3 2020-06-12T02:38:38.616 INFO:tasks.rados.rados.0.smithi107.stdout:8021: writing smithi10729407-5 from 2023424 to 2334720 tid 4 2020-06-12T02:38:38.617 INFO:tasks.rados.rados.0.smithi107.stdout:8022: delete oid 17 current snap is 0 2020-06-12T02:38:38.617 INFO:tasks.rados.rados.0.smithi107.stdout:8018: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:38.617 INFO:tasks.rados.rados.0.smithi107.stdout:8018: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:38.617 INFO:tasks.rados.rados.0.smithi107.stdout:8018: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:38.617 INFO:tasks.rados.rados.0.smithi107.stdout:8018: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:38.618 INFO:tasks.rados.rados.0.smithi107.stdout:8021: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:38.618 INFO:tasks.rados.rados.0.smithi107.stdout:8018: finishing write tid 7 to smithi10729407-4 2020-06-12T02:38:38.618 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1511 (ObjNum 8017 snap 0 seq_num 8017) dirty exists 2020-06-12T02:38:38.618 INFO:tasks.rados.rados.0.smithi107.stdout:8018: left oid 4 (ObjNum 8017 snap 0 seq_num 8017) 2020-06-12T02:38:38.619 INFO:tasks.rados.rados.0.smithi107.stdout:8018: done (2 left) 2020-06-12T02:38:38.620 INFO:tasks.rados.rados.0.smithi107.stdout:8022: done (1 left) 2020-06-12T02:38:38.620 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:38:38.620 INFO:tasks.rados.rados.0.smithi107.stdout:8023: seq_num 8022 ranges {0=761856,761856=532480,1294336=598016,1892352=475136} 2020-06-12T02:38:38.623 INFO:tasks.rados.rados.0.smithi107.stdout:8023: writing smithi10729407-20 from 0 to 761856 tid 1 2020-06-12T02:38:38.627 INFO:tasks.rados.rados.0.smithi107.stdout:8023: writing smithi10729407-20 from 761856 to 1294336 tid 2 2020-06-12T02:38:38.629 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:38.626+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:38.630 INFO:tasks.rados.rados.0.smithi107.stdout:8023: writing smithi10729407-20 from 1294336 to 1892352 tid 3 2020-06-12T02:38:38.634 INFO:tasks.rados.rados.0.smithi107.stdout:8023: writing smithi10729407-20 from 1892352 to 2367488 tid 4 2020-06-12T02:38:38.634 INFO:tasks.rados.rados.0.smithi107.stdout:8024: delete oid 7 current snap is 0 2020-06-12T02:38:38.634 INFO:tasks.rados.rados.0.smithi107.stdout:8021: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:38.635 INFO:tasks.rados.rados.0.smithi107.stdout:8021: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:38.635 INFO:tasks.rados.rados.0.smithi107.stdout:8021: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:38.635 INFO:tasks.rados.rados.0.smithi107.stdout:8021: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:38.636 INFO:tasks.rados.rados.0.smithi107.stdout:8021: finishing write tid 7 to smithi10729407-5 2020-06-12T02:38:38.636 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 800 (ObjNum 8020 snap 0 seq_num 8020) dirty exists 2020-06-12T02:38:38.637 INFO:tasks.rados.rados.0.smithi107.stdout:8021: left oid 5 (ObjNum 8020 snap 0 seq_num 8020) 2020-06-12T02:38:38.638 INFO:tasks.rados.rados.0.smithi107.stdout:8021: done (2 left) 2020-06-12T02:38:38.638 INFO:tasks.rados.rados.0.smithi107.stdout:8024: done (1 left) 2020-06-12T02:38:38.638 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:38.639 INFO:tasks.rados.rados.0.smithi107.stdout:8025: seq_num 8024 ranges {0=794624,794624=696320,1490944=565248,2056192=344064} 2020-06-12T02:38:38.643 INFO:tasks.rados.rados.0.smithi107.stdout:8025: writing smithi10729407-17 from 0 to 794624 tid 1 2020-06-12T02:38:38.647 INFO:tasks.rados.rados.0.smithi107.stdout:8025: writing smithi10729407-17 from 794624 to 1490944 tid 2 2020-06-12T02:38:38.650 INFO:tasks.rados.rados.0.smithi107.stdout:8025: writing smithi10729407-17 from 1490944 to 2056192 tid 3 2020-06-12T02:38:38.652 INFO:tasks.rados.rados.0.smithi107.stdout:8025: writing smithi10729407-17 from 2056192 to 2400256 tid 4 2020-06-12T02:38:38.652 INFO:tasks.rados.rados.0.smithi107.stdout:8026: delete oid 6 current snap is 0 2020-06-12T02:38:38.652 INFO:tasks.rados.rados.0.smithi107.stdout:8023: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:38.653 INFO:tasks.rados.rados.0.smithi107.stdout:8023: finishing write tid 2 to smithi10729407-20 2020-06-12T02:38:38.653 INFO:tasks.rados.rados.0.smithi107.stdout:8023: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:38.653 INFO:tasks.rados.rados.0.smithi107.stdout:8023: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:38.653 INFO:tasks.rados.rados.0.smithi107.stdout:8023: finishing write tid 6 to smithi10729407-20 2020-06-12T02:38:38.653 INFO:tasks.rados.rados.0.smithi107.stdout:8023: finishing write tid 7 to smithi10729407-20 2020-06-12T02:38:38.654 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 586 (ObjNum 8022 snap 0 seq_num 8022) dirty exists 2020-06-12T02:38:38.654 INFO:tasks.rados.rados.0.smithi107.stdout:8023: left oid 20 (ObjNum 8022 snap 0 seq_num 8022) 2020-06-12T02:38:38.654 INFO:tasks.rados.rados.0.smithi107.stdout:8025: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:38.655 INFO:tasks.rados.rados.0.smithi107.stdout:8023: done (2 left) 2020-06-12T02:38:38.655 INFO:tasks.rados.rados.0.smithi107.stdout:8026: done (1 left) 2020-06-12T02:38:38.656 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:38.656 INFO:tasks.rados.rados.0.smithi107.stdout:8027: seq_num 8026 ranges {0=24576} 2020-06-12T02:38:38.656 INFO:tasks.rados.rados.0.smithi107.stdout:8027: writing smithi10729407-14 from 0 to 24576 tid 1 2020-06-12T02:38:38.656 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:38.656 INFO:tasks.rados.rados.0.smithi107.stdout:8028: seq_num 8027 ranges {0=40960} 2020-06-12T02:38:38.657 INFO:tasks.rados.rados.0.smithi107.stdout:8028: writing smithi10729407-6 from 0 to 40960 tid 1 2020-06-12T02:38:38.657 INFO:tasks.rados.rados.0.smithi107.stdout:8029: delete oid 12 current snap is 0 2020-06-12T02:38:38.660 INFO:tasks.rados.rados.0.smithi107.stdout:8027: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:38.660 INFO:tasks.rados.rados.0.smithi107.stdout:8027: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:38.661 INFO:tasks.rados.rados.0.smithi107.stdout:8029: done (3 left) 2020-06-12T02:38:38.661 INFO:tasks.rados.rados.0.smithi107.stdout:8030: delete oid 5 current snap is 0 2020-06-12T02:38:38.663 INFO:tasks.rados.rados.0.smithi107.stdout:8025: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:38.663 INFO:tasks.rados.rados.0.smithi107.stdout:8027: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:38.663 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1424 (ObjNum 8026 snap 0 seq_num 8026) dirty exists 2020-06-12T02:38:38.663 INFO:tasks.rados.rados.0.smithi107.stdout:8027: left oid 14 (ObjNum 8026 snap 0 seq_num 8026) 2020-06-12T02:38:38.663 INFO:tasks.rados.rados.0.smithi107.stdout:8025: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:38.664 INFO:tasks.rados.rados.0.smithi107.stdout:8028: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:38.664 INFO:tasks.rados.rados.0.smithi107.stdout:8025: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:38.668 INFO:tasks.rados.rados.0.smithi107.stdout:8027: done (3 left) 2020-06-12T02:38:38.668 INFO:tasks.rados.rados.0.smithi107.stdout:8030: done (2 left) 2020-06-12T02:38:38.668 INFO:tasks.rados.rados.0.smithi107.stdout:8031: delete oid 18 current snap is 0 2020-06-12T02:38:38.668 INFO:tasks.rados.rados.0.smithi107.stdout:8028: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:38.669 INFO:tasks.rados.rados.0.smithi107.stdout:8025: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:38.670 INFO:tasks.rados.rados.0.smithi107.stdout:8028: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:38.670 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1451 (ObjNum 8027 snap 0 seq_num 8027) dirty exists 2020-06-12T02:38:38.670 INFO:tasks.rados.rados.0.smithi107.stdout:8028: left oid 6 (ObjNum 8027 snap 0 seq_num 8027) 2020-06-12T02:38:38.671 INFO:tasks.rados.rados.0.smithi107.stdout:8025: finishing write tid 7 to smithi10729407-17 2020-06-12T02:38:38.671 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1517 (ObjNum 8024 snap 0 seq_num 8024) dirty exists 2020-06-12T02:38:38.671 INFO:tasks.rados.rados.0.smithi107.stdout:8025: left oid 17 (ObjNum 8024 snap 0 seq_num 8024) 2020-06-12T02:38:38.672 INFO:tasks.rados.rados.0.smithi107.stdout:8025: done (2 left) 2020-06-12T02:38:38.672 INFO:tasks.rados.rados.0.smithi107.stdout:8028: done (1 left) 2020-06-12T02:38:38.673 INFO:tasks.rados.rados.0.smithi107.stdout:8031: done (0 left) 2020-06-12T02:38:38.673 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:38:38.673 INFO:tasks.rados.rados.0.smithi107.stdout:8032: seq_num 8031 ranges {0=106496} 2020-06-12T02:38:38.673 INFO:tasks.rados.rados.0.smithi107.stdout:8032: writing smithi10729407-5 from 0 to 106496 tid 1 2020-06-12T02:38:38.673 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:38:38.674 INFO:tasks.rados.rados.0.smithi107.stdout:8033: seq_num 8032 ranges {0=122880} 2020-06-12T02:38:38.674 INFO:tasks.rados.rados.0.smithi107.stdout:8033: writing smithi10729407-3 from 0 to 122880 tid 1 2020-06-12T02:38:38.674 INFO:tasks.rados.rados.0.smithi107.stdout:8034: delete oid 7 current snap is 0 2020-06-12T02:38:38.676 INFO:tasks.rados.rados.0.smithi107.stdout:8032: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:38.678 INFO:tasks.rados.rados.0.smithi107.stdout:8034: done (2 left) 2020-06-12T02:38:38.678 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:38.678 INFO:tasks.rados.rados.0.smithi107.stdout:8035: seq_num 8034 ranges {2048000=155648} 2020-06-12T02:38:38.690 INFO:tasks.rados.rados.0.smithi107.stdout:8035: writing smithi10729407-15 from 2048000 to 2203648 tid 1 2020-06-12T02:38:38.690 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:38.691 INFO:tasks.rados.rados.0.smithi107.stdout:8036: seq_num 8035 ranges {0=172032} 2020-06-12T02:38:38.691 INFO:tasks.rados.rados.0.smithi107.stdout:8036: writing smithi10729407-9 from 0 to 172032 tid 1 2020-06-12T02:38:38.692 INFO:tasks.rados.rados.0.smithi107.stdout:8037: delete oid 1 current snap is 0 2020-06-12T02:38:38.692 INFO:tasks.rados.rados.0.smithi107.stdout:8033: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:38.692 INFO:tasks.rados.rados.0.smithi107.stdout:8033: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:38.692 INFO:tasks.rados.rados.0.smithi107.stdout:8032: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:38.693 INFO:tasks.rados.rados.0.smithi107.stdout:8033: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:38.693 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1457 (ObjNum 8032 snap 0 seq_num 8032) dirty exists 2020-06-12T02:38:38.693 INFO:tasks.rados.rados.0.smithi107.stdout:8033: left oid 3 (ObjNum 8032 snap 0 seq_num 8032) 2020-06-12T02:38:38.693 INFO:tasks.rados.rados.0.smithi107.stdout:8032: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:38.693 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 803 (ObjNum 8031 snap 0 seq_num 8031) dirty exists 2020-06-12T02:38:38.693 INFO:tasks.rados.rados.0.smithi107.stdout:8032: left oid 5 (ObjNum 8031 snap 0 seq_num 8031) 2020-06-12T02:38:38.694 INFO:tasks.rados.rados.0.smithi107.stdout:8032: done (4 left) 2020-06-12T02:38:38.694 INFO:tasks.rados.rados.0.smithi107.stdout:8033: done (3 left) 2020-06-12T02:38:38.695 INFO:tasks.rados.rados.0.smithi107.stdout:8037: done (2 left) 2020-06-12T02:38:38.695 INFO:tasks.rados.rados.0.smithi107.stdout:8038: delete oid 6 current snap is 0 2020-06-12T02:38:38.695 INFO:tasks.rados.rados.0.smithi107.stdout:8035: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:38.695 INFO:tasks.rados.rados.0.smithi107.stdout:8036: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:38.697 INFO:tasks.rados.rados.0.smithi107.stdout:8035: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:38.697 INFO:tasks.rados.rados.0.smithi107.stdout:8038: done (2 left) 2020-06-12T02:38:38.698 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:38.698 INFO:tasks.rados.rados.0.smithi107.stdout:8039: seq_num 8038 ranges {3833856=229376} 2020-06-12T02:38:38.719 INFO:tasks.rados.rados.0.smithi107.stdout:8039: writing smithi10729407-4 from 3833856 to 4063232 tid 1 2020-06-12T02:38:38.720 INFO:tasks.rados.rados.0.smithi107.stdout:8040: delete oid 12 current snap is 0 2020-06-12T02:38:38.720 INFO:tasks.rados.rados.0.smithi107.stdout:8036: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:38.721 INFO:tasks.rados.rados.0.smithi107.stdout:8035: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:38.721 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1491 (ObjNum 8034 snap 0 seq_num 8034) dirty exists 2020-06-12T02:38:38.721 INFO:tasks.rados.rados.0.smithi107.stdout:8035: left oid 15 (ObjNum 8034 snap 0 seq_num 8034) 2020-06-12T02:38:38.721 INFO:tasks.rados.rados.0.smithi107.stdout:8036: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:38.722 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1499 (ObjNum 8035 snap 0 seq_num 8035) dirty exists 2020-06-12T02:38:38.722 INFO:tasks.rados.rados.0.smithi107.stdout:8036: left oid 9 (ObjNum 8035 snap 0 seq_num 8035) 2020-06-12T02:38:38.723 INFO:tasks.rados.rados.0.smithi107.stdout:8035: done (3 left) 2020-06-12T02:38:38.723 INFO:tasks.rados.rados.0.smithi107.stdout:8036: done (2 left) 2020-06-12T02:38:38.724 INFO:tasks.rados.rados.0.smithi107.stdout:8040: done (1 left) 2020-06-12T02:38:38.724 INFO:tasks.rados.rados.0.smithi107.stdout:8041: delete oid 13 current snap is 0 2020-06-12T02:38:38.727 INFO:tasks.rados.rados.0.smithi107.stdout:8039: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:38.728 INFO:tasks.rados.rados.0.smithi107.stdout:8039: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:38.729 INFO:tasks.rados.rados.0.smithi107.stdout:8039: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:38.729 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1519 (ObjNum 8038 snap 0 seq_num 8038) dirty exists 2020-06-12T02:38:38.729 INFO:tasks.rados.rados.0.smithi107.stdout:8039: left oid 4 (ObjNum 8038 snap 0 seq_num 8038) 2020-06-12T02:38:38.730 INFO:tasks.rados.rados.0.smithi107.stdout:8039: done (1 left) 2020-06-12T02:38:38.730 INFO:tasks.rados.rados.0.smithi107.stdout:8041: done (0 left) 2020-06-12T02:38:38.730 INFO:tasks.rados.rados.0.smithi107.stdout:8042: delete oid 2 current snap is 0 2020-06-12T02:38:38.731 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:38.727+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:38.733 INFO:tasks.rados.rados.0.smithi107.stdout:8042: done (0 left) 2020-06-12T02:38:38.733 INFO:tasks.rados.rados.0.smithi107.stdout:8043: delete oid 3 current snap is 0 2020-06-12T02:38:38.736 INFO:tasks.rados.rados.0.smithi107.stdout:8043: done (0 left) 2020-06-12T02:38:38.736 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:38:38.737 INFO:tasks.rados.rados.0.smithi107.stdout:8044: seq_num 8043 ranges {7659520=311296} 2020-06-12T02:38:38.739 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:38.739 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:38.760 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-06-12T02:38:38.779 INFO:tasks.rados.rados.0.smithi107.stdout:8044: writing smithi10729407-10 from 7659520 to 7970816 tid 1 2020-06-12T02:38:38.779 INFO:tasks.rados.rados.0.smithi107.stdout:8045: delete oid 18 current snap is 0 2020-06-12T02:38:38.782 INFO:tasks.rados.rados.0.smithi107.stdout:8045: done (1 left) 2020-06-12T02:38:38.782 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:38.783 INFO:tasks.rados.rados.0.smithi107.stdout:8046: seq_num 8045 ranges {0=344064} 2020-06-12T02:38:38.785 INFO:tasks.rados.rados.0.smithi107.stdout:8046: writing smithi10729407-13 from 0 to 344064 tid 1 2020-06-12T02:38:38.785 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:38.785 INFO:tasks.rados.rados.0.smithi107.stdout:8047: seq_num 8046 ranges {2064384=360448} 2020-06-12T02:38:38.799 INFO:tasks.rados.rados.0.smithi107.stdout:8047: writing smithi10729407-8 from 2064384 to 2424832 tid 1 2020-06-12T02:38:38.799 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:38.799 INFO:tasks.rados.rados.0.smithi107.stdout:8048: seq_num 8047 ranges {0=376832} 2020-06-12T02:38:38.801 INFO:tasks.rados.rados.0.smithi107.stdout:8048: writing smithi10729407-6 from 0 to 376832 tid 1 2020-06-12T02:38:38.802 INFO:tasks.rados.rados.0.smithi107.stdout:8049: delete oid 19 current snap is 0 2020-06-12T02:38:38.802 INFO:tasks.rados.rados.0.smithi107.stdout:8044: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:38.802 INFO:tasks.rados.rados.0.smithi107.stdout:8044: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:38.802 INFO:tasks.rados.rados.0.smithi107.stdout:8044: finishing write tid 4 to smithi10729407-10 2020-06-12T02:38:38.803 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 569 (ObjNum 8043 snap 0 seq_num 8043) dirty exists 2020-06-12T02:38:38.803 INFO:tasks.rados.rados.0.smithi107.stdout:8044: left oid 10 (ObjNum 8043 snap 0 seq_num 8043) 2020-06-12T02:38:38.803 INFO:tasks.rados.rados.0.smithi107.stdout:8046: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:38.803 INFO:tasks.rados.rados.0.smithi107.stdout:8046: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:38.803 INFO:tasks.rados.rados.0.smithi107.stdout:8046: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:38.803 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1478 (ObjNum 8045 snap 0 seq_num 8045) dirty exists 2020-06-12T02:38:38.804 INFO:tasks.rados.rados.0.smithi107.stdout:8046: left oid 13 (ObjNum 8045 snap 0 seq_num 8045) 2020-06-12T02:38:38.806 INFO:tasks.rados.rados.0.smithi107.stdout:8047: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:38.807 INFO:tasks.rados.rados.0.smithi107.stdout:8044: done (4 left) 2020-06-12T02:38:38.807 INFO:tasks.rados.rados.0.smithi107.stdout:8046: done (3 left) 2020-06-12T02:38:38.808 INFO:tasks.rados.rados.0.smithi107.stdout:8049: done (2 left) 2020-06-12T02:38:38.808 INFO:tasks.rados.rados.0.smithi107.stdout:8050: delete oid 5 current snap is 0 2020-06-12T02:38:38.808 INFO:tasks.rados.rados.0.smithi107.stdout:8048: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:38.809 INFO:tasks.rados.rados.0.smithi107.stdout:8047: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:38.810 INFO:tasks.rados.rados.0.smithi107.stdout:8047: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:38.811 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1493 (ObjNum 8046 snap 0 seq_num 8046) dirty exists 2020-06-12T02:38:38.811 INFO:tasks.rados.rados.0.smithi107.stdout:8047: left oid 8 (ObjNum 8046 snap 0 seq_num 8046) 2020-06-12T02:38:38.814 INFO:tasks.rados.rados.0.smithi107.stdout:8048: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:38.814 INFO:tasks.rados.rados.0.smithi107.stdout:8047: done (2 left) 2020-06-12T02:38:38.814 INFO:tasks.rados.rados.0.smithi107.stdout:8050: done (1 left) 2020-06-12T02:38:38.815 INFO:tasks.rados.rados.0.smithi107.stdout:8051: delete oid 18 current snap is 0 2020-06-12T02:38:38.816 INFO:tasks.rados.rados.0.smithi107.stdout:8048: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:38.816 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1455 (ObjNum 8047 snap 0 seq_num 8047) dirty exists 2020-06-12T02:38:38.816 INFO:tasks.rados.rados.0.smithi107.stdout:8048: left oid 6 (ObjNum 8047 snap 0 seq_num 8047) 2020-06-12T02:38:38.817 INFO:tasks.rados.rados.0.smithi107.stdout:8048: done (1 left) 2020-06-12T02:38:38.817 INFO:tasks.rados.rados.0.smithi107.stdout:8051: done (0 left) 2020-06-12T02:38:38.817 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:38.818 INFO:tasks.rados.rados.0.smithi107.stdout:8052: seq_num 8051 ranges {24576=442368} 2020-06-12T02:38:38.820 INFO:tasks.rados.rados.0.smithi107.stdout:8052: writing smithi10729407-14 from 24576 to 466944 tid 1 2020-06-12T02:38:38.820 INFO:tasks.rados.rados.0.smithi107.stdout:8053: delete oid 13 current snap is 0 2020-06-12T02:38:38.822 INFO:tasks.rados.rados.0.smithi107.stdout:8053: done (1 left) 2020-06-12T02:38:38.822 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:38.822 INFO:tasks.rados.rados.0.smithi107.stdout:8054: seq_num 8053 ranges {0=475136} 2020-06-12T02:38:38.825 INFO:tasks.rados.rados.0.smithi107.stdout:8054: writing smithi10729407-12 from 0 to 475136 tid 1 2020-06-12T02:38:38.826 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:38.826 INFO:tasks.rados.rados.0.smithi107.stdout:8055: seq_num 8054 ranges {0=491520} 2020-06-12T02:38:38.828 INFO:tasks.rados.rados.0.smithi107.stdout:8055: writing smithi10729407-18 from 0 to 491520 tid 1 2020-06-12T02:38:38.829 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:38:38.829 INFO:tasks.rados.rados.0.smithi107.stdout:8056: seq_num 8055 ranges {0=507904} 2020-06-12T02:38:38.831 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:38.828+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:38.832 INFO:tasks.rados.rados.0.smithi107.stdout:8056: writing smithi10729407-7 from 0 to 507904 tid 1 2020-06-12T02:38:38.832 INFO:tasks.rados.rados.0.smithi107.stdout:8057: delete oid 5 current snap is 0 2020-06-12T02:38:38.832 INFO:tasks.rados.rados.0.smithi107.stdout:8052: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:38.832 INFO:tasks.rados.rados.0.smithi107.stdout:8052: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:38.833 INFO:tasks.rados.rados.0.smithi107.stdout:8054: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:38.833 INFO:tasks.rados.rados.0.smithi107.stdout:8052: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:38.834 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1426 (ObjNum 8051 snap 0 seq_num 8051) dirty exists 2020-06-12T02:38:38.834 INFO:tasks.rados.rados.0.smithi107.stdout:8052: left oid 14 (ObjNum 8051 snap 0 seq_num 8051) 2020-06-12T02:38:38.835 INFO:tasks.rados.rados.0.smithi107.stdout:8054: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:38.836 INFO:tasks.rados.rados.0.smithi107.stdout:8052: done (4 left) 2020-06-12T02:38:38.836 INFO:tasks.rados.rados.0.smithi107.stdout:8057: done (3 left) 2020-06-12T02:38:38.836 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:38.837 INFO:tasks.rados.rados.0.smithi107.stdout:8058: seq_num 8057 ranges {376832=548864} 2020-06-12T02:38:38.838 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd set nodeep-scrub 2020-06-12T02:38:38.841 INFO:tasks.rados.rados.0.smithi107.stdout:8058: writing smithi10729407-6 from 376832 to 925696 tid 1 2020-06-12T02:38:38.842 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:38.842 INFO:tasks.rados.rados.0.smithi107.stdout:8059: seq_num 8058 ranges {0=565248} 2020-06-12T02:38:38.845 INFO:tasks.rados.rados.0.smithi107.stdout:8059: writing smithi10729407-5 from 0 to 565248 tid 1 2020-06-12T02:38:38.846 INFO:tasks.rados.rados.0.smithi107.stdout:8060: delete oid 9 current snap is 0 2020-06-12T02:38:38.846 INFO:tasks.rados.rados.0.smithi107.stdout:8055: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:38.846 INFO:tasks.rados.rados.0.smithi107.stdout:8054: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:38.846 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1470 (ObjNum 8053 snap 0 seq_num 8053) dirty exists 2020-06-12T02:38:38.847 INFO:tasks.rados.rados.0.smithi107.stdout:8054: left oid 12 (ObjNum 8053 snap 0 seq_num 8053) 2020-06-12T02:38:38.847 INFO:tasks.rados.rados.0.smithi107.stdout:8056: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:38.847 INFO:tasks.rados.rados.0.smithi107.stdout:8055: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:38.847 INFO:tasks.rados.rados.0.smithi107.stdout:8055: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:38.847 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1503 (ObjNum 8054 snap 0 seq_num 8054) dirty exists 2020-06-12T02:38:38.847 INFO:tasks.rados.rados.0.smithi107.stdout:8055: left oid 18 (ObjNum 8054 snap 0 seq_num 8054) 2020-06-12T02:38:38.848 INFO:tasks.rados.rados.0.smithi107.stdout:8056: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:38.848 INFO:tasks.rados.rados.0.smithi107.stdout:8056: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:38.849 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1481 (ObjNum 8055 snap 0 seq_num 8055) dirty exists 2020-06-12T02:38:38.849 INFO:tasks.rados.rados.0.smithi107.stdout:8056: left oid 7 (ObjNum 8055 snap 0 seq_num 8055) 2020-06-12T02:38:38.849 INFO:tasks.rados.rados.0.smithi107.stdout:8054: done (5 left) 2020-06-12T02:38:38.849 INFO:tasks.rados.rados.0.smithi107.stdout:8055: done (4 left) 2020-06-12T02:38:38.849 INFO:tasks.rados.rados.0.smithi107.stdout:8056: done (3 left) 2020-06-12T02:38:38.849 INFO:tasks.rados.rados.0.smithi107.stdout:8060: done (2 left) 2020-06-12T02:38:38.850 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:38:38.850 INFO:tasks.rados.rados.0.smithi107.stdout:8061: seq_num 8060 ranges {2367488=598016} 2020-06-12T02:38:38.865 INFO:tasks.rados.rados.0.smithi107.stdout:8061: writing smithi10729407-20 from 2367488 to 2965504 tid 1 2020-06-12T02:38:38.866 INFO:tasks.rados.rados.0.smithi107.stdout:8062: delete oid 7 current snap is 0 2020-06-12T02:38:38.866 INFO:tasks.rados.rados.0.smithi107.stdout:8058: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:38.866 INFO:tasks.rados.rados.0.smithi107.stdout:8058: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:38.866 INFO:tasks.rados.rados.0.smithi107.stdout:8059: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:38.867 INFO:tasks.rados.rados.0.smithi107.stdout:8059: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:38.867 INFO:tasks.rados.rados.0.smithi107.stdout:8058: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:38.867 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1457 (ObjNum 8057 snap 0 seq_num 8057) dirty exists 2020-06-12T02:38:38.867 INFO:tasks.rados.rados.0.smithi107.stdout:8058: left oid 6 (ObjNum 8057 snap 0 seq_num 8057) 2020-06-12T02:38:38.867 INFO:tasks.rados.rados.0.smithi107.stdout:8059: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:38.867 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 807 (ObjNum 8058 snap 0 seq_num 8058) dirty exists 2020-06-12T02:38:38.868 INFO:tasks.rados.rados.0.smithi107.stdout:8059: left oid 5 (ObjNum 8058 snap 0 seq_num 8058) 2020-06-12T02:38:38.868 INFO:tasks.rados.rados.0.smithi107.stdout:8058: done (3 left) 2020-06-12T02:38:38.869 INFO:tasks.rados.rados.0.smithi107.stdout:8059: done (2 left) 2020-06-12T02:38:38.869 INFO:tasks.rados.rados.0.smithi107.stdout:8062: done (1 left) 2020-06-12T02:38:38.869 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:38:38.869 INFO:tasks.rados.rados.0.smithi107.stdout:8063: seq_num 8062 ranges {0=630784} 2020-06-12T02:38:38.872 INFO:tasks.rados.rados.0.smithi107.stdout:8063: writing smithi10729407-16 from 0 to 630784 tid 1 2020-06-12T02:38:38.872 INFO:tasks.rados.rados.0.smithi107.stdout:8064: delete oid 12 current snap is 0 2020-06-12T02:38:38.873 INFO:tasks.rados.rados.0.smithi107.stdout:8061: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:38.873 INFO:tasks.rados.rados.0.smithi107.stdout:8061: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:38.873 INFO:tasks.rados.rados.0.smithi107.stdout:8061: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:38.874 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 588 (ObjNum 8060 snap 0 seq_num 8060) dirty exists 2020-06-12T02:38:38.874 INFO:tasks.rados.rados.0.smithi107.stdout:8061: left oid 20 (ObjNum 8060 snap 0 seq_num 8060) 2020-06-12T02:38:38.875 INFO:tasks.rados.rados.0.smithi107.stdout:8061: done (2 left) 2020-06-12T02:38:38.875 INFO:tasks.rados.rados.0.smithi107.stdout:8064: done (1 left) 2020-06-12T02:38:38.875 INFO:tasks.rados.rados.0.smithi107.stdout:8065: delete oid 19 current snap is 0 2020-06-12T02:38:38.878 INFO:tasks.rados.rados.0.smithi107.stdout:8065: done (1 left) 2020-06-12T02:38:38.879 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:38.879 INFO:tasks.rados.rados.0.smithi107.stdout:8066: seq_num 8065 ranges {4063232=679936} 2020-06-12T02:38:38.907 INFO:tasks.rados.rados.0.smithi107.stdout:8066: writing smithi10729407-4 from 4063232 to 4743168 tid 1 2020-06-12T02:38:38.907 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:38.908 INFO:tasks.rados.rados.0.smithi107.stdout:8067: seq_num 8066 ranges {0=696320} 2020-06-12T02:38:38.912 INFO:tasks.rados.rados.0.smithi107.stdout:8067: writing smithi10729407-13 from 0 to 696320 tid 1 2020-06-12T02:38:38.912 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:38.912 INFO:tasks.rados.rados.0.smithi107.stdout:8068: seq_num 8067 ranges {466944=712704} 2020-06-12T02:38:38.918 INFO:tasks.rados.rados.0.smithi107.stdout:8068: writing smithi10729407-14 from 466944 to 1179648 tid 1 2020-06-12T02:38:38.918 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:38:38.919 INFO:tasks.rados.rados.0.smithi107.stdout:8069: seq_num 8068 ranges {0=729088} 2020-06-12T02:38:38.924 INFO:tasks.rados.rados.0.smithi107.stdout:8069: writing smithi10729407-2 from 0 to 729088 tid 1 2020-06-12T02:38:38.924 INFO:tasks.rados.rados.0.smithi107.stdout:8070: delete oid 9 current snap is 0 2020-06-12T02:38:38.924 INFO:tasks.rados.rados.0.smithi107.stdout:8063: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:38.925 INFO:tasks.rados.rados.0.smithi107.stdout:8063: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:38.925 INFO:tasks.rados.rados.0.smithi107.stdout:8063: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:38.925 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1460 (ObjNum 8062 snap 0 seq_num 8062) dirty exists 2020-06-12T02:38:38.926 INFO:tasks.rados.rados.0.smithi107.stdout:8063: left oid 16 (ObjNum 8062 snap 0 seq_num 8062) 2020-06-12T02:38:38.926 INFO:tasks.rados.rados.0.smithi107.stdout:8066: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:38.926 INFO:tasks.rados.rados.0.smithi107.stdout:8066: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:38.927 INFO:tasks.rados.rados.0.smithi107.stdout:8066: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:38.927 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1521 (ObjNum 8065 snap 0 seq_num 8065) dirty exists 2020-06-12T02:38:38.927 INFO:tasks.rados.rados.0.smithi107.stdout:8066: left oid 4 (ObjNum 8065 snap 0 seq_num 8065) 2020-06-12T02:38:38.928 INFO:tasks.rados.rados.0.smithi107.stdout:8067: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:38.928 INFO:tasks.rados.rados.0.smithi107.stdout:8067: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:38.928 INFO:tasks.rados.rados.0.smithi107.stdout:8067: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:38.929 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1484 (ObjNum 8066 snap 0 seq_num 8066) dirty exists 2020-06-12T02:38:38.929 INFO:tasks.rados.rados.0.smithi107.stdout:8067: left oid 13 (ObjNum 8066 snap 0 seq_num 8066) 2020-06-12T02:38:38.934 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:38.935 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:38.936 INFO:tasks.rados.rados.0.smithi107.stdout:8068: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:38.937 INFO:tasks.rados.rados.0.smithi107.stdout:8063: done (5 left) 2020-06-12T02:38:38.937 INFO:tasks.rados.rados.0.smithi107.stdout:8066: done (4 left) 2020-06-12T02:38:38.938 INFO:tasks.rados.rados.0.smithi107.stdout:8067: done (3 left) 2020-06-12T02:38:38.938 INFO:tasks.rados.rados.0.smithi107.stdout:8070: done (2 left) 2020-06-12T02:38:38.938 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:38:38.943 INFO:tasks.rados.rados.0.smithi107.stdout:8071: seq_num 8070 ranges {7970816=761856} 2020-06-12T02:38:38.947 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-06-12T02:38:38.947 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:38.943+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:38.979 INFO:tasks.rados.rados.0.smithi107.stdout:8071: writing smithi10729407-10 from 7970816 to 8732672 tid 1 2020-06-12T02:38:38.980 INFO:tasks.rados.rados.0.smithi107.stdout:8072: delete oid 6 current snap is 0 2020-06-12T02:38:38.981 INFO:tasks.rados.rados.0.smithi107.stdout:8069: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:38.981 INFO:tasks.rados.rados.0.smithi107.stdout:8068: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:38.981 INFO:tasks.rados.rados.0.smithi107.stdout:8068: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:38.982 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1428 (ObjNum 8067 snap 0 seq_num 8067) dirty exists 2020-06-12T02:38:38.983 INFO:tasks.rados.rados.0.smithi107.stdout:8068: left oid 14 (ObjNum 8067 snap 0 seq_num 8067) 2020-06-12T02:38:38.984 INFO:tasks.rados.rados.0.smithi107.stdout:8069: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:38.984 INFO:tasks.rados.rados.0.smithi107.stdout:8069: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:38.984 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 759 (ObjNum 8068 snap 0 seq_num 8068) dirty exists 2020-06-12T02:38:38.984 INFO:tasks.rados.rados.0.smithi107.stdout:8069: left oid 2 (ObjNum 8068 snap 0 seq_num 8068) 2020-06-12T02:38:38.986 INFO:tasks.rados.rados.0.smithi107.stdout:8068: done (3 left) 2020-06-12T02:38:38.986 INFO:tasks.rados.rados.0.smithi107.stdout:8069: done (2 left) 2020-06-12T02:38:38.987 INFO:tasks.rados.rados.0.smithi107.stdout:8072: done (1 left) 2020-06-12T02:38:38.987 INFO:tasks.rados.rados.0.smithi107.stdout:8073: delete oid 16 current snap is 0 2020-06-12T02:38:38.987 INFO:tasks.rados.rados.0.smithi107.stdout:8073: done (1 left) 2020-06-12T02:38:38.988 INFO:tasks.rados.rados.0.smithi107.stdout:8074: delete oid 7 current snap is 0 2020-06-12T02:38:38.988 INFO:tasks.rados.rados.0.smithi107.stdout:8071: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:38.990 INFO:tasks.rados.rados.0.smithi107.stdout:8071: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:38.991 INFO:tasks.rados.rados.0.smithi107.stdout:8074: done (1 left) 2020-06-12T02:38:38.991 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:38.991 INFO:tasks.rados.rados.0.smithi107.stdout:8075: seq_num 8074 ranges {1179648=425984,1605632=401408} 2020-06-12T02:38:38.998 INFO:tasks.rados.rados.0.smithi107.stdout:8075: writing smithi10729407-14 from 1179648 to 1605632 tid 1 2020-06-12T02:38:39.001 INFO:tasks.rados.rados.0.smithi107.stdout:8075: writing smithi10729407-14 from 1605632 to 2007040 tid 2 2020-06-12T02:38:39.002 INFO:tasks.rados.rados.0.smithi107.stdout:8076: delete oid 19 current snap is 0 2020-06-12T02:38:39.002 INFO:tasks.rados.rados.0.smithi107.stdout:8071: finishing write tid 4 to smithi10729407-10 2020-06-12T02:38:39.002 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 571 (ObjNum 8070 snap 0 seq_num 8070) dirty exists 2020-06-12T02:38:39.003 INFO:tasks.rados.rados.0.smithi107.stdout:8071: left oid 10 (ObjNum 8070 snap 0 seq_num 8070) 2020-06-12T02:38:39.006 INFO:tasks.rados.rados.0.smithi107.stdout:8071: done (2 left) 2020-06-12T02:38:39.006 INFO:tasks.rados.rados.0.smithi107.stdout:8076: done (1 left) 2020-06-12T02:38:39.006 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:39.007 INFO:tasks.rados.rados.0.smithi107.stdout:8077: seq_num 8076 ranges {2424832=466944,2891776=401408} 2020-06-12T02:38:39.023 INFO:tasks.rados.rados.0.smithi107.stdout:8077: writing smithi10729407-8 from 2424832 to 2891776 tid 1 2020-06-12T02:38:39.024 INFO:tasks.rados.rados.0.smithi107.stdout:8077: writing smithi10729407-8 from 2891776 to 3293184 tid 2 2020-06-12T02:38:39.025 INFO:tasks.rados.rados.0.smithi107.stdout:8078: delete oid 5 current snap is 0 2020-06-12T02:38:39.025 INFO:tasks.rados.rados.0.smithi107.stdout:8075: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:39.025 INFO:tasks.rados.rados.0.smithi107.stdout:8075: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:39.025 INFO:tasks.rados.rados.0.smithi107.stdout:8075: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:39.025 INFO:tasks.rados.rados.0.smithi107.stdout:8075: finishing write tid 5 to smithi10729407-14 2020-06-12T02:38:39.026 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1431 (ObjNum 8074 snap 0 seq_num 8074) dirty exists 2020-06-12T02:38:39.026 INFO:tasks.rados.rados.0.smithi107.stdout:8075: left oid 14 (ObjNum 8074 snap 0 seq_num 8074) 2020-06-12T02:38:39.026 INFO:tasks.rados.rados.0.smithi107.stdout:8075: done (2 left) 2020-06-12T02:38:39.027 INFO:tasks.rados.rados.0.smithi107.stdout:8078: done (1 left) 2020-06-12T02:38:39.027 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:39.027 INFO:tasks.rados.rados.0.smithi107.stdout:8079: seq_num 8078 ranges {2203648=499712,2703360=401408} 2020-06-12T02:38:39.041 INFO:tasks.rados.rados.0.smithi107.stdout:8079: writing smithi10729407-15 from 2203648 to 2703360 tid 1 2020-06-12T02:38:39.044 INFO:tasks.rados.rados.0.smithi107.stdout:8079: writing smithi10729407-15 from 2703360 to 3104768 tid 2 2020-06-12T02:38:39.044 INFO:tasks.rados.rados.0.smithi107.stdout:8080: delete oid 20 current snap is 0 2020-06-12T02:38:39.045 INFO:tasks.rados.rados.0.smithi107.stdout:8077: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:39.045 INFO:tasks.rados.rados.0.smithi107.stdout:8077: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:39.045 INFO:tasks.rados.rados.0.smithi107.stdout:8077: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:39.045 INFO:tasks.rados.rados.0.smithi107.stdout:8077: finishing write tid 5 to smithi10729407-8 2020-06-12T02:38:39.046 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1496 (ObjNum 8076 snap 0 seq_num 8076) dirty exists 2020-06-12T02:38:39.046 INFO:tasks.rados.rados.0.smithi107.stdout:8077: left oid 8 (ObjNum 8076 snap 0 seq_num 8076) 2020-06-12T02:38:39.048 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:39.045+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:39.050 INFO:tasks.rados.rados.0.smithi107.stdout:8079: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:39.051 INFO:tasks.rados.rados.0.smithi107.stdout:8077: done (2 left) 2020-06-12T02:38:39.051 INFO:tasks.rados.rados.0.smithi107.stdout:8080: done (1 left) 2020-06-12T02:38:39.051 INFO:tasks.rados.rados.0.smithi107.stdout:8081: delete oid 3 current snap is 0 2020-06-12T02:38:39.053 INFO:tasks.rados.rados.0.smithi107.stdout:8079: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:39.054 INFO:tasks.rados.rados.0.smithi107.stdout:8079: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:39.055 INFO:tasks.rados.rados.0.smithi107.stdout:8081: done (1 left) 2020-06-12T02:38:39.055 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:39.055 INFO:tasks.rados.rados.0.smithi107.stdout:8082: seq_num 8081 ranges {0=548864,548864=401408} 2020-06-12T02:38:39.056 INFO:tasks.rados.rados.0.smithi107.stdout:8082: writing smithi10729407-9 from 0 to 548864 tid 1 2020-06-12T02:38:39.060 INFO:tasks.rados.rados.0.smithi107.stdout:8082: writing smithi10729407-9 from 548864 to 950272 tid 2 2020-06-12T02:38:39.060 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:38:39.060 INFO:tasks.rados.rados.0.smithi107.stdout:8083: seq_num 8082 ranges {0=565248,565248=401408} 2020-06-12T02:38:39.063 INFO:tasks.rados.rados.0.smithi107.stdout:8083: writing smithi10729407-7 from 0 to 565248 tid 1 2020-06-12T02:38:39.066 INFO:tasks.rados.rados.0.smithi107.stdout:8083: writing smithi10729407-7 from 565248 to 966656 tid 2 2020-06-12T02:38:39.066 INFO:tasks.rados.rados.0.smithi107.stdout:8084: delete oid 1 current snap is 0 2020-06-12T02:38:39.068 INFO:tasks.rados.rados.0.smithi107.stdout:8079: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:39.068 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1499 (ObjNum 8078 snap 0 seq_num 8078) dirty exists 2020-06-12T02:38:39.068 INFO:tasks.rados.rados.0.smithi107.stdout:8079: left oid 15 (ObjNum 8078 snap 0 seq_num 8078) 2020-06-12T02:38:39.069 INFO:tasks.rados.rados.0.smithi107.stdout:8082: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:39.071 INFO:tasks.rados.rados.0.smithi107.stdout:8082: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:39.071 INFO:tasks.rados.rados.0.smithi107.stdout:8079: done (3 left) 2020-06-12T02:38:39.071 INFO:tasks.rados.rados.0.smithi107.stdout:8084: done (2 left) 2020-06-12T02:38:39.071 INFO:tasks.rados.rados.0.smithi107.stdout:8085: delete oid 1 current snap is 0 2020-06-12T02:38:39.072 INFO:tasks.rados.rados.0.smithi107.stdout:8083: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:39.073 INFO:tasks.rados.rados.0.smithi107.stdout:8082: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:39.073 INFO:tasks.rados.rados.0.smithi107.stdout:8085: done (2 left) 2020-06-12T02:38:39.074 INFO:tasks.rados.rados.0.smithi107.stdout:8086: delete oid 19 current snap is 0 2020-06-12T02:38:39.075 INFO:tasks.rados.rados.0.smithi107.stdout:8082: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:39.076 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1508 (ObjNum 8081 snap 0 seq_num 8081) dirty exists 2020-06-12T02:38:39.076 INFO:tasks.rados.rados.0.smithi107.stdout:8082: left oid 9 (ObjNum 8081 snap 0 seq_num 8081) 2020-06-12T02:38:39.078 INFO:tasks.rados.rados.0.smithi107.stdout:8082: done (2 left) 2020-06-12T02:38:39.078 INFO:tasks.rados.rados.0.smithi107.stdout:8086: done (1 left) 2020-06-12T02:38:39.079 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:39.079 INFO:tasks.rados.rados.0.smithi107.stdout:8087: seq_num 8086 ranges {0=630784,630784=401408} 2020-06-12T02:38:39.082 INFO:tasks.rados.rados.0.smithi107.stdout:8087: writing smithi10729407-5 from 0 to 630784 tid 1 2020-06-12T02:38:39.084 INFO:tasks.rados.rados.0.smithi107.stdout:8087: writing smithi10729407-5 from 630784 to 1032192 tid 2 2020-06-12T02:38:39.084 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:38:39.084 INFO:tasks.rados.rados.0.smithi107.stdout:8088: seq_num 8087 ranges {0=647168,647168=401408} 2020-06-12T02:38:39.087 INFO:tasks.rados.rados.0.smithi107.stdout:8088: writing smithi10729407-3 from 0 to 647168 tid 1 2020-06-12T02:38:39.090 INFO:tasks.rados.rados.0.smithi107.stdout:8088: writing smithi10729407-3 from 647168 to 1048576 tid 2 2020-06-12T02:38:39.091 INFO:tasks.rados.rados.0.smithi107.stdout:8089: delete oid 11 current snap is 0 2020-06-12T02:38:39.091 INFO:tasks.rados.rados.0.smithi107.stdout:8083: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:39.091 INFO:tasks.rados.rados.0.smithi107.stdout:8083: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:39.091 INFO:tasks.rados.rados.0.smithi107.stdout:8083: finishing write tid 5 to smithi10729407-7 2020-06-12T02:38:39.091 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1488 (ObjNum 8082 snap 0 seq_num 8082) dirty exists 2020-06-12T02:38:39.092 INFO:tasks.rados.rados.0.smithi107.stdout:8083: left oid 7 (ObjNum 8082 snap 0 seq_num 8082) 2020-06-12T02:38:39.092 INFO:tasks.rados.rados.0.smithi107.stdout:8087: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:39.093 INFO:tasks.rados.rados.0.smithi107.stdout:8087: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:39.093 INFO:tasks.rados.rados.0.smithi107.stdout:8087: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:39.094 INFO:tasks.rados.rados.0.smithi107.stdout:8083: done (3 left) 2020-06-12T02:38:39.095 INFO:tasks.rados.rados.0.smithi107.stdout:8089: done (2 left) 2020-06-12T02:38:39.095 INFO:tasks.rados.rados.0.smithi107.stdout:8090: delete oid 17 current snap is 0 2020-06-12T02:38:39.095 INFO:tasks.rados.rados.0.smithi107.stdout:8087: finishing write tid 5 to smithi10729407-5 2020-06-12T02:38:39.096 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 811 (ObjNum 8086 snap 0 seq_num 8086) dirty exists 2020-06-12T02:38:39.096 INFO:tasks.rados.rados.0.smithi107.stdout:8087: left oid 5 (ObjNum 8086 snap 0 seq_num 8086) 2020-06-12T02:38:39.097 INFO:tasks.rados.rados.0.smithi107.stdout:8087: done (2 left) 2020-06-12T02:38:39.097 INFO:tasks.rados.rados.0.smithi107.stdout:8090: done (1 left) 2020-06-12T02:38:39.098 INFO:tasks.rados.rados.0.smithi107.stdout:8091: delete oid 2 current snap is 0 2020-06-12T02:38:39.101 INFO:tasks.rados.rados.0.smithi107.stdout:8091: done (1 left) 2020-06-12T02:38:39.101 INFO:tasks.rados.rados.0.smithi107.stdout:8092: delete oid 8 current snap is 0 2020-06-12T02:38:39.102 INFO:tasks.rados.rados.0.smithi107.stdout:8088: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:39.102 INFO:tasks.rados.rados.0.smithi107.stdout:8088: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:39.103 INFO:tasks.rados.rados.0.smithi107.stdout:8088: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:39.104 INFO:tasks.rados.rados.0.smithi107.stdout:8092: done (1 left) 2020-06-12T02:38:39.104 INFO:tasks.rados.rados.0.smithi107.stdout:8093: delete oid 12 current snap is 0 2020-06-12T02:38:39.106 INFO:tasks.rados.rados.0.smithi107.stdout:8088: finishing write tid 5 to smithi10729407-3 2020-06-12T02:38:39.106 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1465 (ObjNum 8087 snap 0 seq_num 8087) dirty exists 2020-06-12T02:38:39.107 INFO:tasks.rados.rados.0.smithi107.stdout:8088: left oid 3 (ObjNum 8087 snap 0 seq_num 8087) 2020-06-12T02:38:39.108 INFO:tasks.rados.rados.0.smithi107.stdout:8088: done (1 left) 2020-06-12T02:38:39.108 INFO:tasks.rados.rados.0.smithi107.stdout:8093: done (0 left) 2020-06-12T02:38:39.109 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:39.109 INFO:tasks.rados.rados.0.smithi107.stdout:8094: seq_num 8093 ranges {0=745472,745472=401408} 2020-06-12T02:38:39.113 INFO:tasks.rados.rados.0.smithi107.stdout:8094: writing smithi10729407-12 from 0 to 745472 tid 1 2020-06-12T02:38:39.115 INFO:tasks.rados.rados.0.smithi107.stdout:8094: writing smithi10729407-12 from 745472 to 1146880 tid 2 2020-06-12T02:38:39.115 INFO:tasks.rados.rados.0.smithi107.stdout:8095: delete oid 13 current snap is 0 2020-06-12T02:38:39.118 INFO:tasks.rados.rados.0.smithi107.stdout:8095: done (1 left) 2020-06-12T02:38:39.118 INFO:tasks.rados.rados.0.smithi107.stdout:8096: delete oid 5 current snap is 0 2020-06-12T02:38:39.119 INFO:tasks.rados.rados.0.smithi107.stdout:8094: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:39.122 INFO:tasks.rados.rados.0.smithi107.stdout:8096: done (1 left) 2020-06-12T02:38:39.122 INFO:tasks.rados.rados.0.smithi107.stdout:8097: delete oid 18 current snap is 0 2020-06-12T02:38:39.125 INFO:tasks.rados.rados.0.smithi107.stdout:8097: done (1 left) 2020-06-12T02:38:39.125 INFO:tasks.rados.rados.0.smithi107.stdout:8098: delete oid 5 current snap is 0 2020-06-12T02:38:39.125 INFO:tasks.rados.rados.0.smithi107.stdout:8094: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:39.130 INFO:tasks.rados.rados.0.smithi107.stdout:8094: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:39.130 INFO:tasks.rados.rados.0.smithi107.stdout:8098: done (1 left) 2020-06-12T02:38:39.130 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:39.131 INFO:tasks.rados.rados.0.smithi107.stdout:8099: seq_num 8098 ranges {0=434176,434176=630784,1064960=172032} 2020-06-12T02:38:39.132 INFO:tasks.rados.rados.0.smithi107.stdout:8099: writing smithi10729407-1 from 0 to 434176 tid 1 2020-06-12T02:38:39.133 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:39.134 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:39.136 INFO:tasks.rados.rados.0.smithi107.stdout:8099: writing smithi10729407-1 from 434176 to 1064960 tid 2 2020-06-12T02:38:39.137 INFO:tasks.rados.rados.0.smithi107.stdout:8099: writing smithi10729407-1 from 1064960 to 1236992 tid 3 2020-06-12T02:38:39.138 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:38:39.138 INFO:tasks.rados.rados.0.smithi107.stdout:8100: seq_num 8099 ranges {1048576=450560,1499136=516096,2015232=286720} 2020-06-12T02:38:39.144 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-06-12T02:38:39.146 INFO:tasks.rados.rados.0.smithi107.stdout:8100: writing smithi10729407-3 from 1048576 to 1499136 tid 1 2020-06-12T02:38:39.151 INFO:tasks.rados.rados.0.smithi107.stdout:8100: writing smithi10729407-3 from 1499136 to 2015232 tid 2 2020-06-12T02:38:39.152 INFO:tasks.rados.rados.0.smithi107.stdout:8100: writing smithi10729407-3 from 2015232 to 2301952 tid 3 2020-06-12T02:38:39.153 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:39.159 INFO:tasks.rados.rados.0.smithi107.stdout:8101: seq_num 8100 ranges {0=466944,466944=802816} 2020-06-12T02:38:39.160 INFO:tasks.rados.rados.0.smithi107.stdout:8101: writing smithi10729407-2 from 0 to 466944 tid 1 2020-06-12T02:38:39.162 INFO:tasks.rados.rados.0.smithi107.stdout:8101: writing smithi10729407-2 from 466944 to 1269760 tid 2 2020-06-12T02:38:39.163 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:38:39.163 INFO:tasks.rados.rados.0.smithi107.stdout:8102: seq_num 8101 ranges {8732672=483328,9216000=688128,9904128=114688} 2020-06-12T02:38:39.189 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:38:39.186+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:38:39.211 INFO:tasks.rados.rados.0.smithi107.stdout:8102: writing smithi10729407-10 from 8732672 to 9216000 tid 1 2020-06-12T02:38:39.216 INFO:tasks.rados.rados.0.smithi107.stdout:8102: writing smithi10729407-10 from 9216000 to 9904128 tid 2 2020-06-12T02:38:39.217 INFO:tasks.rados.rados.0.smithi107.stdout:8102: writing smithi10729407-10 from 9904128 to 10018816 tid 3 2020-06-12T02:38:39.218 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:39.219 INFO:tasks.rados.rados.0.smithi107.stdout:8103: seq_num 8102 ranges {0=499712,499712=573440,1073152=229376} 2020-06-12T02:38:39.219 INFO:tasks.rados.rados.0.smithi107.stdout:8103: writing smithi10729407-19 from 0 to 499712 tid 1 2020-06-12T02:38:39.222 INFO:tasks.rados.rados.0.smithi107.stdout:8103: writing smithi10729407-19 from 499712 to 1073152 tid 2 2020-06-12T02:38:39.223 INFO:tasks.rados.rados.0.smithi107.stdout:8103: writing smithi10729407-19 from 1073152 to 1302528 tid 3 2020-06-12T02:38:39.224 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:38:39.225 INFO:tasks.rados.rados.0.smithi107.stdout:8104: seq_num 8103 ranges {0=516096,516096=458752,974848=344064} 2020-06-12T02:38:39.227 INFO:tasks.rados.rados.0.smithi107.stdout:8104: writing smithi10729407-16 from 0 to 516096 tid 1 2020-06-12T02:38:39.229 INFO:tasks.rados.rados.0.smithi107.stdout:8104: writing smithi10729407-16 from 516096 to 974848 tid 2 2020-06-12T02:38:39.233 INFO:tasks.rados.rados.0.smithi107.stdout:8104: writing smithi10729407-16 from 974848 to 1318912 tid 3 2020-06-12T02:38:39.234 INFO:tasks.rados.rados.0.smithi107.stdout:8105: delete oid 13 current snap is 0 2020-06-12T02:38:39.234 INFO:tasks.rados.rados.0.smithi107.stdout:8094: finishing write tid 5 to smithi10729407-12 2020-06-12T02:38:39.234 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1478 (ObjNum 8093 snap 0 seq_num 8093) dirty exists 2020-06-12T02:38:39.235 INFO:tasks.rados.rados.0.smithi107.stdout:8094: left oid 12 (ObjNum 8093 snap 0 seq_num 8093) 2020-06-12T02:38:39.235 INFO:tasks.rados.rados.0.smithi107.stdout:8099: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:39.235 INFO:tasks.rados.rados.0.smithi107.stdout:8099: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:39.235 INFO:tasks.rados.rados.0.smithi107.stdout:8099: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:39.235 INFO:tasks.rados.rados.0.smithi107.stdout:8099: finishing write tid 5 to smithi10729407-1 2020-06-12T02:38:39.235 INFO:tasks.rados.rados.0.smithi107.stdout:8099: finishing write tid 6 to smithi10729407-1 2020-06-12T02:38:39.236 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1464 (ObjNum 8098 snap 0 seq_num 8098) dirty exists 2020-06-12T02:38:39.236 INFO:tasks.rados.rados.0.smithi107.stdout:8099: left oid 1 (ObjNum 8098 snap 0 seq_num 8098) 2020-06-12T02:38:39.236 INFO:tasks.rados.rados.0.smithi107.stdout:8100: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:39.236 INFO:tasks.rados.rados.0.smithi107.stdout:8100: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:39.236 INFO:tasks.rados.rados.0.smithi107.stdout:8100: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:39.237 INFO:tasks.rados.rados.0.smithi107.stdout:8100: finishing write tid 5 to smithi10729407-3 2020-06-12T02:38:39.237 INFO:tasks.rados.rados.0.smithi107.stdout:8101: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:39.237 INFO:tasks.rados.rados.0.smithi107.stdout:8100: finishing write tid 6 to smithi10729407-3 2020-06-12T02:38:39.237 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1469 (ObjNum 8099 snap 0 seq_num 8099) dirty exists 2020-06-12T02:38:39.237 INFO:tasks.rados.rados.0.smithi107.stdout:8100: left oid 3 (ObjNum 8099 snap 0 seq_num 8099) 2020-06-12T02:38:39.238 INFO:tasks.rados.rados.0.smithi107.stdout:8101: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:39.238 INFO:tasks.rados.rados.0.smithi107.stdout:8101: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:39.238 INFO:tasks.rados.rados.0.smithi107.stdout:8101: finishing write tid 5 to smithi10729407-2 2020-06-12T02:38:39.238 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 763 (ObjNum 8100 snap 0 seq_num 8100) dirty exists 2020-06-12T02:38:39.238 INFO:tasks.rados.rados.0.smithi107.stdout:8101: left oid 2 (ObjNum 8100 snap 0 seq_num 8100) 2020-06-12T02:38:39.238 INFO:tasks.rados.rados.0.smithi107.stdout:8102: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:39.239 INFO:tasks.rados.rados.0.smithi107.stdout:8102: finishing write tid 2 to smithi10729407-10 2020-06-12T02:38:39.239 INFO:tasks.rados.rados.0.smithi107.stdout:8102: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:39.239 INFO:tasks.rados.rados.0.smithi107.stdout:8103: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:39.239 INFO:tasks.rados.rados.0.smithi107.stdout:8102: finishing write tid 5 to smithi10729407-10 2020-06-12T02:38:39.239 INFO:tasks.rados.rados.0.smithi107.stdout:8102: finishing write tid 6 to smithi10729407-10 2020-06-12T02:38:39.240 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 575 (ObjNum 8101 snap 0 seq_num 8101) dirty exists 2020-06-12T02:38:39.240 INFO:tasks.rados.rados.0.smithi107.stdout:8102: left oid 10 (ObjNum 8101 snap 0 seq_num 8101) 2020-06-12T02:38:39.240 INFO:tasks.rados.rados.0.smithi107.stdout:8103: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:39.241 INFO:tasks.rados.rados.0.smithi107.stdout:8103: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:39.242 INFO:tasks.rados.rados.0.smithi107.stdout:8103: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:39.242 INFO:tasks.rados.rados.0.smithi107.stdout:8104: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:39.242 INFO:tasks.rados.rados.0.smithi107.stdout:8094: done (7 left) 2020-06-12T02:38:39.242 INFO:tasks.rados.rados.0.smithi107.stdout:8099: done (6 left) 2020-06-12T02:38:39.242 INFO:tasks.rados.rados.0.smithi107.stdout:8100: done (5 left) 2020-06-12T02:38:39.242 INFO:tasks.rados.rados.0.smithi107.stdout:8101: done (4 left) 2020-06-12T02:38:39.243 INFO:tasks.rados.rados.0.smithi107.stdout:8102: done (3 left) 2020-06-12T02:38:39.243 INFO:tasks.rados.rados.0.smithi107.stdout:8105: done (2 left) 2020-06-12T02:38:39.243 INFO:tasks.rados.rados.0.smithi107.stdout:8106: delete oid 13 current snap is 0 2020-06-12T02:38:39.243 INFO:tasks.rados.rados.0.smithi107.stdout:8104: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:39.243 INFO:tasks.rados.rados.0.smithi107.stdout:8103: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:39.244 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1482 (ObjNum 8102 snap 0 seq_num 8102) dirty exists 2020-06-12T02:38:39.244 INFO:tasks.rados.rados.0.smithi107.stdout:8103: left oid 19 (ObjNum 8102 snap 0 seq_num 8102) 2020-06-12T02:38:39.244 INFO:tasks.rados.rados.0.smithi107.stdout:8104: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:39.244 INFO:tasks.rados.rados.0.smithi107.stdout:8104: finishing write tid 5 to smithi10729407-16 2020-06-12T02:38:39.245 INFO:tasks.rados.rados.0.smithi107.stdout:8103: done (2 left) 2020-06-12T02:38:39.245 INFO:tasks.rados.rados.0.smithi107.stdout:8106: done (1 left) 2020-06-12T02:38:39.245 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:38:39.246 INFO:tasks.rados.rados.0.smithi107.stdout:8107: seq_num 8106 ranges {0=565248,565248=557056,1122304=245760} 2020-06-12T02:38:39.246 INFO:tasks.rados.rados.0.smithi107.stdout:8107: writing smithi10729407-20 from 0 to 565248 tid 1 2020-06-12T02:38:39.248 INFO:tasks.rados.rados.0.smithi107.stdout:8107: writing smithi10729407-20 from 565248 to 1122304 tid 2 2020-06-12T02:38:39.250 INFO:tasks.rados.rados.0.smithi107.stdout:8107: writing smithi10729407-20 from 1122304 to 1368064 tid 3 2020-06-12T02:38:39.250 INFO:tasks.rados.rados.0.smithi107.stdout:8108: delete oid 14 current snap is 0 2020-06-12T02:38:39.251 INFO:tasks.rados.rados.0.smithi107.stdout:8104: finishing write tid 6 to smithi10729407-16 2020-06-12T02:38:39.251 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1473 (ObjNum 8103 snap 0 seq_num 8103) dirty exists 2020-06-12T02:38:39.251 INFO:tasks.rados.rados.0.smithi107.stdout:8104: left oid 16 (ObjNum 8103 snap 0 seq_num 8103) 2020-06-12T02:38:39.251 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:39.247+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:39.252 INFO:tasks.rados.rados.0.smithi107.stdout:8104: done (2 left) 2020-06-12T02:38:39.252 INFO:tasks.rados.rados.0.smithi107.stdout:8108: done (1 left) 2020-06-12T02:38:39.252 INFO:tasks.rados.rados.0.smithi107.stdout:8109: delete oid 9 current snap is 0 2020-06-12T02:38:39.253 INFO:tasks.rados.rados.0.smithi107.stdout:8107: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:39.256 INFO:tasks.rados.rados.0.smithi107.stdout:8107: finishing write tid 2 to smithi10729407-20 2020-06-12T02:38:39.257 INFO:tasks.rados.rados.0.smithi107.stdout:8109: done (1 left) 2020-06-12T02:38:39.257 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:39.257 INFO:tasks.rados.rados.0.smithi107.stdout:8110: seq_num 8109 ranges {0=614400,614400=614400,1228800=188416} 2020-06-12T02:38:39.259 INFO:tasks.rados.rados.0.smithi107.stdout:8110: writing smithi10729407-5 from 0 to 614400 tid 1 2020-06-12T02:38:39.263 INFO:tasks.rados.rados.0.smithi107.stdout:8110: writing smithi10729407-5 from 614400 to 1228800 tid 2 2020-06-12T02:38:39.264 INFO:tasks.rados.rados.0.smithi107.stdout:8110: writing smithi10729407-5 from 1228800 to 1417216 tid 3 2020-06-12T02:38:39.264 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:39.264 INFO:tasks.rados.rados.0.smithi107.stdout:8111: seq_num 8110 ranges {1146880=630784,1777664=499712,2277376=303104} 2020-06-12T02:38:39.274 INFO:tasks.rados.rados.0.smithi107.stdout:8111: writing smithi10729407-12 from 1146880 to 1777664 tid 1 2020-06-12T02:38:39.277 INFO:tasks.rados.rados.0.smithi107.stdout:8111: writing smithi10729407-12 from 1777664 to 2277376 tid 2 2020-06-12T02:38:39.279 INFO:tasks.rados.rados.0.smithi107.stdout:8111: writing smithi10729407-12 from 2277376 to 2580480 tid 3 2020-06-12T02:38:39.280 INFO:tasks.rados.rados.0.smithi107.stdout:8112: delete oid 9 current snap is 0 2020-06-12T02:38:39.280 INFO:tasks.rados.rados.0.smithi107.stdout:8107: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:39.280 INFO:tasks.rados.rados.0.smithi107.stdout:8107: finishing write tid 5 to smithi10729407-20 2020-06-12T02:38:39.280 INFO:tasks.rados.rados.0.smithi107.stdout:8107: finishing write tid 6 to smithi10729407-20 2020-06-12T02:38:39.280 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 593 (ObjNum 8106 snap 0 seq_num 8106) dirty exists 2020-06-12T02:38:39.281 INFO:tasks.rados.rados.0.smithi107.stdout:8107: left oid 20 (ObjNum 8106 snap 0 seq_num 8106) 2020-06-12T02:38:39.281 INFO:tasks.rados.rados.0.smithi107.stdout:8110: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:39.281 INFO:tasks.rados.rados.0.smithi107.stdout:8110: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:39.281 INFO:tasks.rados.rados.0.smithi107.stdout:8110: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:39.281 INFO:tasks.rados.rados.0.smithi107.stdout:8110: finishing write tid 5 to smithi10729407-5 2020-06-12T02:38:39.282 INFO:tasks.rados.rados.0.smithi107.stdout:8110: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:39.282 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 817 (ObjNum 8109 snap 0 seq_num 8109) dirty exists 2020-06-12T02:38:39.282 INFO:tasks.rados.rados.0.smithi107.stdout:8110: left oid 5 (ObjNum 8109 snap 0 seq_num 8109) 2020-06-12T02:38:39.283 INFO:tasks.rados.rados.0.smithi107.stdout:8107: done (3 left) 2020-06-12T02:38:39.283 INFO:tasks.rados.rados.0.smithi107.stdout:8110: done (2 left) 2020-06-12T02:38:39.283 INFO:tasks.rados.rados.0.smithi107.stdout:8112: done (1 left) 2020-06-12T02:38:39.283 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:38:39.283 INFO:tasks.rados.rados.0.smithi107.stdout:8113: seq_num 8112 ranges {1236992=663552,1900544=712704,2613248=90112} 2020-06-12T02:38:39.288 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:39.288 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:39.293 INFO:tasks.rados.rados.0.smithi107.stdout:8113: writing smithi10729407-1 from 1236992 to 1900544 tid 1 2020-06-12T02:38:39.297 INFO:tasks.rados.rados.0.smithi107.stdout:8113: writing smithi10729407-1 from 1900544 to 2613248 tid 2 2020-06-12T02:38:39.298 INFO:tasks.rados.rados.0.smithi107.stdout:8113: writing smithi10729407-1 from 2613248 to 2703360 tid 3 2020-06-12T02:38:39.300 INFO:tasks.rados.rados.0.smithi107.stdout:8114: delete oid 20 current snap is 0 2020-06-12T02:38:39.300 INFO:tasks.rados.rados.0.smithi107.stdout:8111: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:39.300 INFO:tasks.rados.rados.0.smithi107.stdout:8111: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:39.301 INFO:tasks.rados.rados.0.smithi107.stdout:8111: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:39.301 INFO:tasks.rados.rados.0.smithi107.stdout:8111: finishing write tid 5 to smithi10729407-12 2020-06-12T02:38:39.301 INFO:tasks.rados.rados.0.smithi107.stdout:8111: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:39.302 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1486 (ObjNum 8110 snap 0 seq_num 8110) dirty exists 2020-06-12T02:38:39.302 INFO:tasks.rados.rados.0.smithi107.stdout:8111: left oid 12 (ObjNum 8110 snap 0 seq_num 8110) 2020-06-12T02:38:39.304 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-06-12T02:38:39.305 INFO:tasks.rados.rados.0.smithi107.stdout:8111: done (2 left) 2020-06-12T02:38:39.306 INFO:tasks.rados.rados.0.smithi107.stdout:8114: done (1 left) 2020-06-12T02:38:39.306 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:38:39.306 INFO:tasks.rados.rados.0.smithi107.stdout:8115: seq_num 8114 ranges {0=696320,696320=483328,1179648=319488} 2020-06-12T02:38:39.307 INFO:tasks.rados.rados.0.smithi107.stdout:8115: writing smithi10729407-6 from 0 to 696320 tid 1 2020-06-12T02:38:39.308 INFO:tasks.rados.rados.0.smithi107.stdout:8115: writing smithi10729407-6 from 696320 to 1179648 tid 2 2020-06-12T02:38:39.310 INFO:tasks.rados.rados.0.smithi107.stdout:8115: writing smithi10729407-6 from 1179648 to 1499136 tid 3 2020-06-12T02:38:39.311 INFO:tasks.rados.rados.0.smithi107.stdout:8116: delete oid 3 current snap is 0 2020-06-12T02:38:39.311 INFO:tasks.rados.rados.0.smithi107.stdout:8113: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:39.314 INFO:tasks.rados.rados.0.smithi107.stdout:8116: done (2 left) 2020-06-12T02:38:39.314 INFO:tasks.rados.rados.0.smithi107.stdout:8117: delete oid 19 current snap is 0 2020-06-12T02:38:39.315 INFO:tasks.rados.rados.0.smithi107.stdout:8113: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:39.315 INFO:tasks.rados.rados.0.smithi107.stdout:8113: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:39.316 INFO:tasks.rados.rados.0.smithi107.stdout:8113: finishing write tid 5 to smithi10729407-1 2020-06-12T02:38:39.318 INFO:tasks.rados.rados.0.smithi107.stdout:8117: done (2 left) 2020-06-12T02:38:39.318 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:39.319 INFO:tasks.rados.rados.0.smithi107.stdout:8118: seq_num 8117 ranges {0=753664,753664=532480,1286144=270336} 2020-06-12T02:38:39.322 INFO:tasks.rados.rados.0.smithi107.stdout:8118: writing smithi10729407-19 from 0 to 753664 tid 1 2020-06-12T02:38:39.325 INFO:tasks.rados.rados.0.smithi107.stdout:8118: writing smithi10729407-19 from 753664 to 1286144 tid 2 2020-06-12T02:38:39.327 INFO:tasks.rados.rados.0.smithi107.stdout:8118: writing smithi10729407-19 from 1286144 to 1556480 tid 3 2020-06-12T02:38:39.328 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:38:39.328 INFO:tasks.rados.rados.0.smithi107.stdout:8119: seq_num 8118 ranges {1318912=770048,2088960=417792,2506752=385024} 2020-06-12T02:38:39.339 INFO:tasks.rados.rados.0.smithi107.stdout:8119: writing smithi10729407-16 from 1318912 to 2088960 tid 1 2020-06-12T02:38:39.344 INFO:tasks.rados.rados.0.smithi107.stdout:8119: writing smithi10729407-16 from 2088960 to 2506752 tid 2 2020-06-12T02:38:39.345 INFO:tasks.rados.rados.0.smithi107.stdout:8119: writing smithi10729407-16 from 2506752 to 2891776 tid 3 2020-06-12T02:38:39.345 INFO:tasks.rados.rados.0.smithi107.stdout:8120: delete oid 4 current snap is 0 2020-06-12T02:38:39.346 INFO:tasks.rados.rados.0.smithi107.stdout:8115: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:39.346 INFO:tasks.rados.rados.0.smithi107.stdout:8113: finishing write tid 6 to smithi10729407-1 2020-06-12T02:38:39.346 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1468 (ObjNum 8112 snap 0 seq_num 8112) dirty exists 2020-06-12T02:38:39.347 INFO:tasks.rados.rados.0.smithi107.stdout:8113: left oid 1 (ObjNum 8112 snap 0 seq_num 8112) 2020-06-12T02:38:39.347 INFO:tasks.rados.rados.0.smithi107.stdout:8115: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:39.347 INFO:tasks.rados.rados.0.smithi107.stdout:8115: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:39.348 INFO:tasks.rados.rados.0.smithi107.stdout:8115: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:39.348 INFO:tasks.rados.rados.0.smithi107.stdout:8115: finishing write tid 6 to smithi10729407-6 2020-06-12T02:38:39.348 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1472 (ObjNum 8114 snap 0 seq_num 8114) dirty exists 2020-06-12T02:38:39.348 INFO:tasks.rados.rados.0.smithi107.stdout:8115: left oid 6 (ObjNum 8114 snap 0 seq_num 8114) 2020-06-12T02:38:39.349 INFO:tasks.rados.rados.0.smithi107.stdout:8118: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:39.349 INFO:tasks.rados.rados.0.smithi107.stdout:8118: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:39.349 INFO:tasks.rados.rados.0.smithi107.stdout:8118: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:39.349 INFO:tasks.rados.rados.0.smithi107.stdout:8118: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:39.350 INFO:tasks.rados.rados.0.smithi107.stdout:8118: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:39.350 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1491 (ObjNum 8117 snap 0 seq_num 8117) dirty exists 2020-06-12T02:38:39.351 INFO:tasks.rados.rados.0.smithi107.stdout:8118: left oid 19 (ObjNum 8117 snap 0 seq_num 8117) 2020-06-12T02:38:39.353 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:39.348+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:39.353 INFO:tasks.rados.rados.0.smithi107.stdout:8113: done (4 left) 2020-06-12T02:38:39.354 INFO:tasks.rados.rados.0.smithi107.stdout:8115: done (3 left) 2020-06-12T02:38:39.354 INFO:tasks.rados.rados.0.smithi107.stdout:8118: done (2 left) 2020-06-12T02:38:39.354 INFO:tasks.rados.rados.0.smithi107.stdout:8120: done (1 left) 2020-06-12T02:38:39.355 INFO:tasks.rados.rados.0.smithi107.stdout:8121: delete oid 14 current snap is 0 2020-06-12T02:38:39.355 INFO:tasks.rados.rados.0.smithi107.stdout:8121: done (1 left) 2020-06-12T02:38:39.355 INFO:tasks.rados.rados.0.smithi107.stdout:8122: delete oid 8 current snap is 0 2020-06-12T02:38:39.356 INFO:tasks.rados.rados.0.smithi107.stdout:8119: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:39.356 INFO:tasks.rados.rados.0.smithi107.stdout:8119: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:39.356 INFO:tasks.rados.rados.0.smithi107.stdout:8119: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:39.357 INFO:tasks.rados.rados.0.smithi107.stdout:8119: finishing write tid 5 to smithi10729407-16 2020-06-12T02:38:39.357 INFO:tasks.rados.rados.0.smithi107.stdout:8122: done (1 left) 2020-06-12T02:38:39.357 INFO:tasks.rados.rados.0.smithi107.stdout:8123: delete oid 1 current snap is 0 2020-06-12T02:38:39.358 INFO:tasks.rados.rados.0.smithi107.stdout:8119: finishing write tid 6 to smithi10729407-16 2020-06-12T02:38:39.359 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1478 (ObjNum 8118 snap 0 seq_num 8118) dirty exists 2020-06-12T02:38:39.359 INFO:tasks.rados.rados.0.smithi107.stdout:8119: left oid 16 (ObjNum 8118 snap 0 seq_num 8118) 2020-06-12T02:38:39.359 INFO:tasks.rados.rados.0.smithi107.stdout:8119: done (1 left) 2020-06-12T02:38:39.359 INFO:tasks.rados.rados.0.smithi107.stdout:8123: done (0 left) 2020-06-12T02:38:39.360 INFO:tasks.rados.rados.0.smithi107.stdout:8124: delete oid 3 current snap is 0 2020-06-12T02:38:39.360 INFO:tasks.rados.rados.0.smithi107.stdout:8124: done (0 left) 2020-06-12T02:38:39.360 INFO:tasks.rados.rados.0.smithi107.stdout:8125: delete oid 15 current snap is 0 2020-06-12T02:38:39.361 INFO:tasks.rados.rados.0.smithi107.stdout:8125: done (0 left) 2020-06-12T02:38:39.362 INFO:tasks.rados.rados.0.smithi107.stdout:8126: delete oid 8 current snap is 0 2020-06-12T02:38:39.367 INFO:tasks.rados.rados.0.smithi107.stdout:8126: done (0 left) 2020-06-12T02:38:39.368 INFO:tasks.rados.rados.0.smithi107.stdout:8127: delete oid 3 current snap is 0 2020-06-12T02:38:39.370 INFO:tasks.rados.rados.0.smithi107.stdout:8127: done (0 left) 2020-06-12T02:38:39.370 INFO:tasks.rados.rados.0.smithi107.stdout:8128: delete oid 12 current snap is 0 2020-06-12T02:38:39.373 INFO:tasks.rados.rados.0.smithi107.stdout:8128: done (0 left) 2020-06-12T02:38:39.373 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:39.373 INFO:tasks.rados.rados.0.smithi107.stdout:8129: seq_num 8128 ranges {0=532480,532480=565248,1097728=507904,1605632=131072} 2020-06-12T02:38:39.375 INFO:tasks.rados.rados.0.smithi107.stdout:8129: writing smithi10729407-9 from 0 to 532480 tid 1 2020-06-12T02:38:39.379 INFO:tasks.rados.rados.0.smithi107.stdout:8129: writing smithi10729407-9 from 532480 to 1097728 tid 2 2020-06-12T02:38:39.383 INFO:tasks.rados.rados.0.smithi107.stdout:8129: writing smithi10729407-9 from 1097728 to 1605632 tid 3 2020-06-12T02:38:39.383 INFO:tasks.rados.rados.0.smithi107.stdout:8129: writing smithi10729407-9 from 1605632 to 1736704 tid 4 2020-06-12T02:38:39.383 INFO:tasks.rados.rados.0.smithi107.stdout:8130: delete oid 15 current snap is 0 2020-06-12T02:38:39.384 INFO:tasks.rados.rados.0.smithi107.stdout:8129: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:39.387 INFO:tasks.rados.rados.0.smithi107.stdout:8130: done (1 left) 2020-06-12T02:38:39.388 INFO:tasks.rados.rados.0.smithi107.stdout:8131: delete oid 6 current snap is 0 2020-06-12T02:38:39.388 INFO:tasks.rados.rados.0.smithi107.stdout:8131: done (1 left) 2020-06-12T02:38:39.389 INFO:tasks.rados.rados.0.smithi107.stdout:8132: delete oid 7 current snap is 0 2020-06-12T02:38:39.390 INFO:tasks.rados.rados.0.smithi107.stdout:8129: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:39.391 INFO:tasks.rados.rados.0.smithi107.stdout:8129: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:39.394 INFO:tasks.rados.rados.0.smithi107.stdout:8129: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:39.395 INFO:tasks.rados.rados.0.smithi107.stdout:8129: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:39.395 INFO:tasks.rados.rados.0.smithi107.stdout:8132: done (1 left) 2020-06-12T02:38:39.395 INFO:tasks.rados.rados.0.smithi107.stdout:8133: delete oid 14 current snap is 0 2020-06-12T02:38:39.396 INFO:tasks.rados.rados.0.smithi107.stdout:8129: finishing write tid 7 to smithi10729407-9 2020-06-12T02:38:39.397 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1516 (ObjNum 8128 snap 0 seq_num 8128) dirty exists 2020-06-12T02:38:39.397 INFO:tasks.rados.rados.0.smithi107.stdout:8129: left oid 9 (ObjNum 8128 snap 0 seq_num 8128) 2020-06-12T02:38:39.398 INFO:tasks.rados.rados.0.smithi107.stdout:8129: done (1 left) 2020-06-12T02:38:39.399 INFO:tasks.rados.rados.0.smithi107.stdout:8133: done (0 left) 2020-06-12T02:38:39.399 INFO:tasks.rados.rados.0.smithi107.stdout:8134: delete oid 16 current snap is 0 2020-06-12T02:38:39.401 INFO:tasks.rados.rados.0.smithi107.stdout:8134: done (0 left) 2020-06-12T02:38:39.402 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:38:39.402 INFO:tasks.rados.rados.0.smithi107.stdout:8135: seq_num 8134 ranges {0=638976,638976=671744,1310720=499712,1810432=32768} 2020-06-12T02:38:39.405 INFO:tasks.rados.rados.0.smithi107.stdout:8135: writing smithi10729407-1 from 0 to 638976 tid 1 2020-06-12T02:38:39.409 INFO:tasks.rados.rados.0.smithi107.stdout:8135: writing smithi10729407-1 from 638976 to 1310720 tid 2 2020-06-12T02:38:39.412 INFO:tasks.rados.rados.0.smithi107.stdout:8135: writing smithi10729407-1 from 1310720 to 1810432 tid 3 2020-06-12T02:38:39.413 INFO:tasks.rados.rados.0.smithi107.stdout:8135: writing smithi10729407-1 from 1810432 to 1843200 tid 4 2020-06-12T02:38:39.413 INFO:tasks.rados.rados.0.smithi107.stdout:8136: delete oid 4 current snap is 0 2020-06-12T02:38:39.414 INFO:tasks.rados.rados.0.smithi107.stdout:8135: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:39.419 INFO:tasks.rados.rados.0.smithi107.stdout:8136: done (1 left) 2020-06-12T02:38:39.419 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:39.420 INFO:tasks.rados.rados.0.smithi107.stdout:8137: seq_num 8136 ranges {0=671744,671744=483328,1155072=466944,1622016=253952} 2020-06-12T02:38:39.423 INFO:tasks.rados.rados.0.smithi107.stdout:8137: writing smithi10729407-16 from 0 to 671744 tid 1 2020-06-12T02:38:39.427 INFO:tasks.rados.rados.0.smithi107.stdout:8137: writing smithi10729407-16 from 671744 to 1155072 tid 2 2020-06-12T02:38:39.428 INFO:tasks.rados.rados.0.smithi107.stdout:8137: writing smithi10729407-16 from 1155072 to 1622016 tid 3 2020-06-12T02:38:39.431 INFO:tasks.rados.rados.0.smithi107.stdout:8137: writing smithi10729407-16 from 1622016 to 1875968 tid 4 2020-06-12T02:38:39.432 INFO:tasks.rados.rados.0.smithi107.stdout:8138: delete oid 11 current snap is 0 2020-06-12T02:38:39.433 INFO:tasks.rados.rados.0.smithi107.stdout:8135: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:39.433 INFO:tasks.rados.rados.0.smithi107.stdout:8135: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:39.433 INFO:tasks.rados.rados.0.smithi107.stdout:8135: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:39.433 INFO:tasks.rados.rados.0.smithi107.stdout:8135: finishing write tid 6 to smithi10729407-1 2020-06-12T02:38:39.433 INFO:tasks.rados.rados.0.smithi107.stdout:8135: finishing write tid 7 to smithi10729407-1 2020-06-12T02:38:39.434 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1479 (ObjNum 8134 snap 0 seq_num 8134) dirty exists 2020-06-12T02:38:39.434 INFO:tasks.rados.rados.0.smithi107.stdout:8135: left oid 1 (ObjNum 8134 snap 0 seq_num 8134) 2020-06-12T02:38:39.434 INFO:tasks.rados.rados.0.smithi107.stdout:8137: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:39.435 INFO:tasks.rados.rados.0.smithi107.stdout:8135: done (2 left) 2020-06-12T02:38:39.435 INFO:tasks.rados.rados.0.smithi107.stdout:8138: done (1 left) 2020-06-12T02:38:39.435 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:39.435 INFO:tasks.rados.rados.0.smithi107.stdout:8139: seq_num 8138 ranges {0=704512,704512=655360,1359872=483328,1843200=65536} 2020-06-12T02:38:39.437 INFO:tasks.rados.rados.0.smithi107.stdout:8139: writing smithi10729407-17 from 0 to 704512 tid 1 2020-06-12T02:38:39.441 INFO:tasks.rados.rados.0.smithi107.stdout:8139: writing smithi10729407-17 from 704512 to 1359872 tid 2 2020-06-12T02:38:39.444 INFO:tasks.rados.rados.0.smithi107.stdout:8139: writing smithi10729407-17 from 1359872 to 1843200 tid 3 2020-06-12T02:38:39.445 INFO:tasks.rados.rados.0.smithi107.stdout:8139: writing smithi10729407-17 from 1843200 to 1908736 tid 4 2020-06-12T02:38:39.445 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:39.445 INFO:tasks.rados.rados.0.smithi107.stdout:8140: seq_num 8139 ranges {0=720896,720896=540672,1261568=638976,1900544=24576} 2020-06-12T02:38:39.449 INFO:tasks.rados.rados.0.smithi107.stdout:8140: writing smithi10729407-8 from 0 to 720896 tid 1 2020-06-12T02:38:39.452 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:39.449+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:39.453 INFO:tasks.rados.rados.0.smithi107.stdout:8140: writing smithi10729407-8 from 720896 to 1261568 tid 2 2020-06-12T02:38:39.457 INFO:tasks.rados.rados.0.smithi107.stdout:8140: writing smithi10729407-8 from 1261568 to 1900544 tid 3 2020-06-12T02:38:39.458 INFO:tasks.rados.rados.0.smithi107.stdout:8140: writing smithi10729407-8 from 1900544 to 1925120 tid 4 2020-06-12T02:38:39.458 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:39.458 INFO:tasks.rados.rados.0.smithi107.stdout:8141: seq_num 8140 ranges {0=737280,737280=425984,1163264=442368,1605632=335872} 2020-06-12T02:38:39.462 INFO:tasks.rados.rados.0.smithi107.stdout:8141: writing smithi10729407-15 from 0 to 737280 tid 1 2020-06-12T02:38:39.463 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2020-06-12T02:38:39.465 INFO:tasks.rados.rados.0.smithi107.stdout:8141: writing smithi10729407-15 from 737280 to 1163264 tid 2 2020-06-12T02:38:39.467 INFO:tasks.rados.rados.0.smithi107.stdout:8141: writing smithi10729407-15 from 1163264 to 1605632 tid 3 2020-06-12T02:38:39.470 INFO:tasks.rados.rados.0.smithi107.stdout:8141: writing smithi10729407-15 from 1605632 to 1941504 tid 4 2020-06-12T02:38:39.470 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:38:39.471 INFO:tasks.rados.rados.0.smithi107.stdout:8142: seq_num 8141 ranges {10018816=753664,10772480=712704,11485184=491520} 2020-06-12T02:38:39.507 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:39.508 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:39.516 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-06-12T02:38:39.526 INFO:tasks.rados.rados.0.smithi107.stdout:8142: writing smithi10729407-10 from 10018816 to 10772480 tid 1 2020-06-12T02:38:39.530 INFO:tasks.rados.rados.0.smithi107.stdout:8142: writing smithi10729407-10 from 10772480 to 11485184 tid 2 2020-06-12T02:38:39.535 INFO:tasks.rados.rados.0.smithi107.stdout:8142: writing smithi10729407-10 from 11485184 to 11976704 tid 3 2020-06-12T02:38:39.535 INFO:tasks.rados.rados.0.smithi107.stdout:8143: delete oid 1 current snap is 0 2020-06-12T02:38:39.535 INFO:tasks.rados.rados.0.smithi107.stdout:8137: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:39.536 INFO:tasks.rados.rados.0.smithi107.stdout:8137: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:39.536 INFO:tasks.rados.rados.0.smithi107.stdout:8137: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:39.536 INFO:tasks.rados.rados.0.smithi107.stdout:8137: finishing write tid 6 to smithi10729407-16 2020-06-12T02:38:39.537 INFO:tasks.rados.rados.0.smithi107.stdout:8137: finishing write tid 7 to smithi10729407-16 2020-06-12T02:38:39.538 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1486 (ObjNum 8136 snap 0 seq_num 8136) dirty exists 2020-06-12T02:38:39.538 INFO:tasks.rados.rados.0.smithi107.stdout:8137: left oid 16 (ObjNum 8136 snap 0 seq_num 8136) 2020-06-12T02:38:39.538 INFO:tasks.rados.rados.0.smithi107.stdout:8139: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:39.539 INFO:tasks.rados.rados.0.smithi107.stdout:8139: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:39.539 INFO:tasks.rados.rados.0.smithi107.stdout:8140: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:39.539 INFO:tasks.rados.rados.0.smithi107.stdout:8139: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:39.540 INFO:tasks.rados.rados.0.smithi107.stdout:8139: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:39.540 INFO:tasks.rados.rados.0.smithi107.stdout:8139: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:39.540 INFO:tasks.rados.rados.0.smithi107.stdout:8139: finishing write tid 7 to smithi10729407-17 2020-06-12T02:38:39.540 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1529 (ObjNum 8138 snap 0 seq_num 8138) dirty exists 2020-06-12T02:38:39.541 INFO:tasks.rados.rados.0.smithi107.stdout:8139: left oid 17 (ObjNum 8138 snap 0 seq_num 8138) 2020-06-12T02:38:39.541 INFO:tasks.rados.rados.0.smithi107.stdout:8140: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:39.541 INFO:tasks.rados.rados.0.smithi107.stdout:8140: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:39.542 INFO:tasks.rados.rados.0.smithi107.stdout:8140: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:39.542 INFO:tasks.rados.rados.0.smithi107.stdout:8140: finishing write tid 6 to smithi10729407-8 2020-06-12T02:38:39.542 INFO:tasks.rados.rados.0.smithi107.stdout:8140: finishing write tid 7 to smithi10729407-8 2020-06-12T02:38:39.543 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1509 (ObjNum 8139 snap 0 seq_num 8139) dirty exists 2020-06-12T02:38:39.543 INFO:tasks.rados.rados.0.smithi107.stdout:8140: left oid 8 (ObjNum 8139 snap 0 seq_num 8139) 2020-06-12T02:38:39.543 INFO:tasks.rados.rados.0.smithi107.stdout:8141: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:39.544 INFO:tasks.rados.rados.0.smithi107.stdout:8141: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:39.544 INFO:tasks.rados.rados.0.smithi107.stdout:8141: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:39.544 INFO:tasks.rados.rados.0.smithi107.stdout:8141: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:39.545 INFO:tasks.rados.rados.0.smithi107.stdout:8141: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:39.545 INFO:tasks.rados.rados.0.smithi107.stdout:8141: finishing write tid 7 to smithi10729407-15 2020-06-12T02:38:39.545 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1514 (ObjNum 8140 snap 0 seq_num 8140) dirty exists 2020-06-12T02:38:39.545 INFO:tasks.rados.rados.0.smithi107.stdout:8141: left oid 15 (ObjNum 8140 snap 0 seq_num 8140) 2020-06-12T02:38:39.548 INFO:tasks.rados.rados.0.smithi107.stdout:8142: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:39.549 INFO:tasks.rados.rados.0.smithi107.stdout:8137: done (5 left) 2020-06-12T02:38:39.549 INFO:tasks.rados.rados.0.smithi107.stdout:8139: done (4 left) 2020-06-12T02:38:39.549 INFO:tasks.rados.rados.0.smithi107.stdout:8140: done (3 left) 2020-06-12T02:38:39.550 INFO:tasks.rados.rados.0.smithi107.stdout:8141: done (2 left) 2020-06-12T02:38:39.550 INFO:tasks.rados.rados.0.smithi107.stdout:8143: done (1 left) 2020-06-12T02:38:39.550 INFO:tasks.rados.rados.0.smithi107.stdout:8144: delete oid 18 current snap is 0 2020-06-12T02:38:39.551 INFO:tasks.rados.rados.0.smithi107.stdout:8142: finishing write tid 2 to smithi10729407-10 2020-06-12T02:38:39.551 INFO:tasks.rados.rados.0.smithi107.stdout:8142: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:39.551 INFO:tasks.rados.rados.0.smithi107.stdout:8142: finishing write tid 5 to smithi10729407-10 2020-06-12T02:38:39.552 INFO:tasks.rados.rados.0.smithi107.stdout:8142: finishing write tid 6 to smithi10729407-10 2020-06-12T02:38:39.552 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 579 (ObjNum 8141 snap 0 seq_num 8141) dirty exists 2020-06-12T02:38:39.552 INFO:tasks.rados.rados.0.smithi107.stdout:8142: left oid 10 (ObjNum 8141 snap 0 seq_num 8141) 2020-06-12T02:38:39.552 INFO:tasks.rados.rados.0.smithi107.stdout:8142: done (1 left) 2020-06-12T02:38:39.553 INFO:tasks.rados.rados.0.smithi107.stdout:8144: done (0 left) 2020-06-12T02:38:39.553 INFO:tasks.rados.rados.0.smithi107.stdout:8145: delete oid 2 current snap is 0 2020-06-12T02:38:39.554 INFO:tasks.rados.rados.0.smithi107.stdout:8145: done (0 left) 2020-06-12T02:38:39.555 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:39.555 INFO:tasks.rados.rados.0.smithi107.stdout:8146: seq_num 8145 ranges {0=417792,417792=696320,1114112=630784,1744896=278528} 2020-06-12T02:38:39.555 INFO:tasks.rados.rados.0.smithi107.stdout:8146: writing smithi10729407-4 from 0 to 417792 tid 1 2020-06-12T02:38:39.556 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:39.551+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:39.556 INFO:tasks.rados.rados.0.smithi107.stdout:8146: writing smithi10729407-4 from 417792 to 1114112 tid 2 2020-06-12T02:38:39.559 INFO:tasks.rados.rados.0.smithi107.stdout:8146: writing smithi10729407-4 from 1114112 to 1744896 tid 3 2020-06-12T02:38:39.560 INFO:tasks.rados.rados.0.smithi107.stdout:8146: writing smithi10729407-4 from 1744896 to 2023424 tid 4 2020-06-12T02:38:39.561 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:39.561 INFO:tasks.rados.rados.0.smithi107.stdout:8147: seq_num 8146 ranges {0=434176,434176=581632,1015808=442368,1458176=499712,1957888=81920} 2020-06-12T02:38:39.563 INFO:tasks.rados.rados.0.smithi107.stdout:8147: writing smithi10729407-6 from 0 to 434176 tid 1 2020-06-12T02:38:39.567 INFO:tasks.rados.rados.0.smithi107.stdout:8147: writing smithi10729407-6 from 434176 to 1015808 tid 2 2020-06-12T02:38:39.569 INFO:tasks.rados.rados.0.smithi107.stdout:8147: writing smithi10729407-6 from 1015808 to 1458176 tid 3 2020-06-12T02:38:39.572 INFO:tasks.rados.rados.0.smithi107.stdout:8147: writing smithi10729407-6 from 1458176 to 1957888 tid 4 2020-06-12T02:38:39.573 INFO:tasks.rados.rados.0.smithi107.stdout:8147: writing smithi10729407-6 from 1957888 to 2039808 tid 5 2020-06-12T02:38:39.573 INFO:tasks.rados.rados.0.smithi107.stdout:8148: delete oid 17 current snap is 0 2020-06-12T02:38:39.575 INFO:tasks.rados.rados.0.smithi107.stdout:8146: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:39.575 INFO:tasks.rados.rados.0.smithi107.stdout:8146: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:39.575 INFO:tasks.rados.rados.0.smithi107.stdout:8147: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:39.576 INFO:tasks.rados.rados.0.smithi107.stdout:8146: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:39.577 INFO:tasks.rados.rados.0.smithi107.stdout:8146: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:39.577 INFO:tasks.rados.rados.0.smithi107.stdout:8146: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:39.578 INFO:tasks.rados.rados.0.smithi107.stdout:8146: finishing write tid 7 to smithi10729407-4 2020-06-12T02:38:39.586 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1534 (ObjNum 8145 snap 0 seq_num 8145) dirty exists 2020-06-12T02:38:39.586 INFO:tasks.rados.rados.0.smithi107.stdout:8146: left oid 4 (ObjNum 8145 snap 0 seq_num 8145) 2020-06-12T02:38:39.587 INFO:tasks.rados.rados.0.smithi107.stdout:8146: done (2 left) 2020-06-12T02:38:39.587 INFO:tasks.rados.rados.0.smithi107.stdout:8148: done (1 left) 2020-06-12T02:38:39.587 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:39.588 INFO:tasks.rados.rados.0.smithi107.stdout:8149: seq_num 8148 ranges {0=466944,466944=753664,1220608=409600,1630208=442368} 2020-06-12T02:38:39.588 INFO:tasks.rados.rados.0.smithi107.stdout:8149: writing smithi10729407-17 from 0 to 466944 tid 1 2020-06-12T02:38:39.588 INFO:tasks.rados.rados.0.smithi107.stdout:8149: writing smithi10729407-17 from 466944 to 1220608 tid 2 2020-06-12T02:38:39.589 INFO:tasks.rados.rados.0.smithi107.stdout:8149: writing smithi10729407-17 from 1220608 to 1630208 tid 3 2020-06-12T02:38:39.591 INFO:tasks.rados.rados.0.smithi107.stdout:8149: writing smithi10729407-17 from 1630208 to 2072576 tid 4 2020-06-12T02:38:39.591 INFO:tasks.rados.rados.0.smithi107.stdout:8150: delete oid 20 current snap is 0 2020-06-12T02:38:39.591 INFO:tasks.rados.rados.0.smithi107.stdout:8147: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:39.591 INFO:tasks.rados.rados.0.smithi107.stdout:8147: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:39.592 INFO:tasks.rados.rados.0.smithi107.stdout:8147: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:39.592 INFO:tasks.rados.rados.0.smithi107.stdout:8147: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:39.592 INFO:tasks.rados.rados.0.smithi107.stdout:8149: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:39.592 INFO:tasks.rados.rados.0.smithi107.stdout:8147: finishing write tid 7 to smithi10729407-6 2020-06-12T02:38:39.592 INFO:tasks.rados.rados.0.smithi107.stdout:8147: finishing write tid 8 to smithi10729407-6 2020-06-12T02:38:39.593 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1486 (ObjNum 8146 snap 0 seq_num 8146) dirty exists 2020-06-12T02:38:39.593 INFO:tasks.rados.rados.0.smithi107.stdout:8147: left oid 6 (ObjNum 8146 snap 0 seq_num 8146) 2020-06-12T02:38:39.593 INFO:tasks.rados.rados.0.smithi107.stdout:8147: done (2 left) 2020-06-12T02:38:39.593 INFO:tasks.rados.rados.0.smithi107.stdout:8150: done (1 left) 2020-06-12T02:38:39.594 INFO:tasks.rados.rados.0.smithi107.stdout:8151: delete oid 12 current snap is 0 2020-06-12T02:38:39.597 INFO:tasks.rados.rados.0.smithi107.stdout:8151: done (1 left) 2020-06-12T02:38:39.598 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:39.598 INFO:tasks.rados.rados.0.smithi107.stdout:8152: seq_num 8151 ranges {0=516096,516096=450560,966656=581632,1548288=442368,1990656=131072} 2020-06-12T02:38:39.600 INFO:tasks.rados.rados.0.smithi107.stdout:8152: writing smithi10729407-7 from 0 to 516096 tid 1 2020-06-12T02:38:39.603 INFO:tasks.rados.rados.0.smithi107.stdout:8152: writing smithi10729407-7 from 516096 to 966656 tid 2 2020-06-12T02:38:39.606 INFO:tasks.rados.rados.0.smithi107.stdout:8152: writing smithi10729407-7 from 966656 to 1548288 tid 3 2020-06-12T02:38:39.609 INFO:tasks.rados.rados.0.smithi107.stdout:8152: writing smithi10729407-7 from 1548288 to 1990656 tid 4 2020-06-12T02:38:39.610 INFO:tasks.rados.rados.0.smithi107.stdout:8152: writing smithi10729407-7 from 1990656 to 2121728 tid 5 2020-06-12T02:38:39.610 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:39.610 INFO:tasks.rados.rados.0.smithi107.stdout:8153: seq_num 8152 ranges {1556480=532480,2088960=737280,2826240=786432,3612672=81920} 2020-06-12T02:38:39.621 INFO:tasks.rados.rados.0.smithi107.stdout:8153: writing smithi10729407-19 from 1556480 to 2088960 tid 1 2020-06-12T02:38:39.626 INFO:tasks.rados.rados.0.smithi107.stdout:8153: writing smithi10729407-19 from 2088960 to 2826240 tid 2 2020-06-12T02:38:39.631 INFO:tasks.rados.rados.0.smithi107.stdout:8153: writing smithi10729407-19 from 2826240 to 3612672 tid 3 2020-06-12T02:38:39.632 INFO:tasks.rados.rados.0.smithi107.stdout:8153: writing smithi10729407-19 from 3612672 to 3694592 tid 4 2020-06-12T02:38:39.632 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:39.633 INFO:tasks.rados.rados.0.smithi107.stdout:8154: seq_num 8153 ranges {1925120=548864,2473984=622592,3096576=598016,3694592=385024} 2020-06-12T02:38:39.647 INFO:tasks.rados.rados.0.smithi107.stdout:8154: writing smithi10729407-8 from 1925120 to 2473984 tid 1 2020-06-12T02:38:39.658 INFO:tasks.rados.rados.0.smithi107.stdout:8154: writing smithi10729407-8 from 2473984 to 3096576 tid 2 2020-06-12T02:38:39.658 INFO:tasks.rados.rados.0.smithi107.stdout:8154: writing smithi10729407-8 from 3096576 to 3694592 tid 3 2020-06-12T02:38:39.659 INFO:tasks.rados.rados.0.smithi107.stdout:8154: writing smithi10729407-8 from 3694592 to 4079616 tid 4 2020-06-12T02:38:39.659 INFO:tasks.rados.rados.0.smithi107.stdout:8155: delete oid 12 current snap is 0 2020-06-12T02:38:39.660 INFO:tasks.rados.rados.0.smithi107.stdout:8149: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:39.660 INFO:tasks.rados.rados.0.smithi107.stdout:8149: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:39.660 INFO:tasks.rados.rados.0.smithi107.stdout:8149: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:39.660 INFO:tasks.rados.rados.0.smithi107.stdout:8149: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:39.661 INFO:tasks.rados.rados.0.smithi107.stdout:8149: finishing write tid 7 to smithi10729407-17 2020-06-12T02:38:39.661 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1540 (ObjNum 8148 snap 0 seq_num 8148) dirty exists 2020-06-12T02:38:39.661 INFO:tasks.rados.rados.0.smithi107.stdout:8149: left oid 17 (ObjNum 8148 snap 0 seq_num 8148) 2020-06-12T02:38:39.662 INFO:tasks.rados.rados.0.smithi107.stdout:8152: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:39.662 INFO:tasks.rados.rados.0.smithi107.stdout:8152: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:39.662 INFO:tasks.rados.rados.0.smithi107.stdout:8152: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:39.663 INFO:tasks.rados.rados.0.smithi107.stdout:8152: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:39.663 INFO:tasks.rados.rados.0.smithi107.stdout:8152: finishing write tid 5 to smithi10729407-7 2020-06-12T02:38:39.663 INFO:tasks.rados.rados.0.smithi107.stdout:8152: finishing write tid 7 to smithi10729407-7 2020-06-12T02:38:39.664 INFO:tasks.rados.rados.0.smithi107.stdout:8153: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:39.664 INFO:tasks.rados.rados.0.smithi107.stdout:8152: finishing write tid 8 to smithi10729407-7 2020-06-12T02:38:39.664 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1498 (ObjNum 8151 snap 0 seq_num 8151) dirty exists 2020-06-12T02:38:39.665 INFO:tasks.rados.rados.0.smithi107.stdout:8152: left oid 7 (ObjNum 8151 snap 0 seq_num 8151) 2020-06-12T02:38:39.665 INFO:tasks.rados.rados.0.smithi107.stdout:8153: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:39.665 INFO:tasks.rados.rados.0.smithi107.stdout:8153: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:39.666 INFO:tasks.rados.rados.0.smithi107.stdout:8153: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:39.666 INFO:tasks.rados.rados.0.smithi107.stdout:8153: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:39.666 INFO:tasks.rados.rados.0.smithi107.stdout:8153: finishing write tid 7 to smithi10729407-19 2020-06-12T02:38:39.666 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1498 (ObjNum 8152 snap 0 seq_num 8152) dirty exists 2020-06-12T02:38:39.667 INFO:tasks.rados.rados.0.smithi107.stdout:8153: left oid 19 (ObjNum 8152 snap 0 seq_num 8152) 2020-06-12T02:38:39.667 INFO:tasks.rados.rados.0.smithi107.stdout:8154: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:39.667 INFO:tasks.rados.rados.0.smithi107.stdout:8149: done (4 left) 2020-06-12T02:38:39.668 INFO:tasks.rados.rados.0.smithi107.stdout:8152: done (3 left) 2020-06-12T02:38:39.668 INFO:tasks.rados.rados.0.smithi107.stdout:8153: done (2 left) 2020-06-12T02:38:39.668 INFO:tasks.rados.rados.0.smithi107.stdout:8155: done (1 left) 2020-06-12T02:38:39.668 INFO:tasks.rados.rados.0.smithi107.stdout:8156: delete oid 13 current snap is 0 2020-06-12T02:38:39.670 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:39.653+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:39.670 INFO:tasks.rados.rados.0.smithi107.stdout:8156: done (1 left) 2020-06-12T02:38:39.670 INFO:tasks.rados.rados.0.smithi107.stdout:8157: delete oid 4 current snap is 0 2020-06-12T02:38:39.670 INFO:tasks.rados.rados.0.smithi107.stdout:8154: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:39.671 INFO:tasks.rados.rados.0.smithi107.stdout:8154: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:39.671 INFO:tasks.rados.rados.0.smithi107.stdout:8157: done (1 left) 2020-06-12T02:38:39.671 INFO:tasks.rados.rados.0.smithi107.stdout:8158: delete oid 4 current snap is 0 2020-06-12T02:38:39.671 INFO:tasks.rados.rados.0.smithi107.stdout:8154: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:39.671 INFO:tasks.rados.rados.0.smithi107.stdout:8154: finishing write tid 6 to smithi10729407-8 2020-06-12T02:38:39.674 INFO:tasks.rados.rados.0.smithi107.stdout:8154: finishing write tid 7 to smithi10729407-8 2020-06-12T02:38:39.674 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1519 (ObjNum 8153 snap 0 seq_num 8153) dirty exists 2020-06-12T02:38:39.674 INFO:tasks.rados.rados.0.smithi107.stdout:8154: left oid 8 (ObjNum 8153 snap 0 seq_num 8153) 2020-06-12T02:38:39.675 INFO:tasks.rados.rados.0.smithi107.stdout:8154: done (1 left) 2020-06-12T02:38:39.675 INFO:tasks.rados.rados.0.smithi107.stdout:8158: done (0 left) 2020-06-12T02:38:39.675 INFO:tasks.rados.rados.0.smithi107.stdout:8159: delete oid 16 current snap is 0 2020-06-12T02:38:39.676 INFO:tasks.rados.rados.0.smithi107.stdout:8159: done (0 left) 2020-06-12T02:38:39.676 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:38:39.676 INFO:tasks.rados.rados.0.smithi107.stdout:8160: seq_num 8159 ranges {1417216=655360,2072576=778240,2850816=540672,3391488=286720} 2020-06-12T02:38:39.678 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:39.678 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:39.687 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-06-12T02:38:39.688 INFO:tasks.rados.rados.0.smithi107.stdout:8160: writing smithi10729407-5 from 1417216 to 2072576 tid 1 2020-06-12T02:38:39.692 INFO:tasks.rados.rados.0.smithi107.stdout:8160: writing smithi10729407-5 from 2072576 to 2850816 tid 2 2020-06-12T02:38:39.695 INFO:tasks.rados.rados.0.smithi107.stdout:8160: writing smithi10729407-5 from 2850816 to 3391488 tid 3 2020-06-12T02:38:39.697 INFO:tasks.rados.rados.0.smithi107.stdout:8160: writing smithi10729407-5 from 3391488 to 3678208 tid 4 2020-06-12T02:38:39.697 INFO:tasks.rados.rados.0.smithi107.stdout:8161: delete oid 2 current snap is 0 2020-06-12T02:38:39.697 INFO:tasks.rados.rados.0.smithi107.stdout:8160: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:39.699 INFO:tasks.rados.rados.0.smithi107.stdout:8161: done (1 left) 2020-06-12T02:38:39.699 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:38:39.699 INFO:tasks.rados.rados.0.smithi107.stdout:8162: seq_num 8161 ranges {2039808=688128,2727936=548864,3276800=557056,3833856=499712} 2020-06-12T02:38:39.714 INFO:tasks.rados.rados.0.smithi107.stdout:8162: writing smithi10729407-6 from 2039808 to 2727936 tid 1 2020-06-12T02:38:39.717 INFO:tasks.rados.rados.0.smithi107.stdout:8162: writing smithi10729407-6 from 2727936 to 3276800 tid 2 2020-06-12T02:38:39.720 INFO:tasks.rados.rados.0.smithi107.stdout:8162: writing smithi10729407-6 from 3276800 to 3833856 tid 3 2020-06-12T02:38:39.723 INFO:tasks.rados.rados.0.smithi107.stdout:8162: writing smithi10729407-6 from 3833856 to 4333568 tid 4 2020-06-12T02:38:39.724 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:39.724 INFO:tasks.rados.rados.0.smithi107.stdout:8163: seq_num 8162 ranges {0=704512,704512=425984,1130496=761856,1892352=417792} 2020-06-12T02:38:39.727 INFO:tasks.rados.rados.0.smithi107.stdout:8163: writing smithi10729407-14 from 0 to 704512 tid 1 2020-06-12T02:38:39.730 INFO:tasks.rados.rados.0.smithi107.stdout:8163: writing smithi10729407-14 from 704512 to 1130496 tid 2 2020-06-12T02:38:39.736 INFO:tasks.rados.rados.0.smithi107.stdout:8163: writing smithi10729407-14 from 1130496 to 1892352 tid 3 2020-06-12T02:38:39.739 INFO:tasks.rados.rados.0.smithi107.stdout:8163: writing smithi10729407-14 from 1892352 to 2310144 tid 4 2020-06-12T02:38:39.739 INFO:tasks.rados.rados.0.smithi107.stdout:8164: delete oid 7 current snap is 0 2020-06-12T02:38:39.740 INFO:tasks.rados.rados.0.smithi107.stdout:8160: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:39.743 INFO:tasks.rados.rados.0.smithi107.stdout:8160: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:39.743 INFO:tasks.rados.rados.0.smithi107.stdout:8160: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:39.743 INFO:tasks.rados.rados.0.smithi107.stdout:8160: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:39.743 INFO:tasks.rados.rados.0.smithi107.stdout:8160: finishing write tid 7 to smithi10729407-5 2020-06-12T02:38:39.744 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 822 (ObjNum 8159 snap 0 seq_num 8159) dirty exists 2020-06-12T02:38:39.744 INFO:tasks.rados.rados.0.smithi107.stdout:8160: left oid 5 (ObjNum 8159 snap 0 seq_num 8159) 2020-06-12T02:38:39.744 INFO:tasks.rados.rados.0.smithi107.stdout:8162: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:39.744 INFO:tasks.rados.rados.0.smithi107.stdout:8162: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:39.744 INFO:tasks.rados.rados.0.smithi107.stdout:8162: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:39.745 INFO:tasks.rados.rados.0.smithi107.stdout:8162: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:39.745 INFO:tasks.rados.rados.0.smithi107.stdout:8162: finishing write tid 6 to smithi10729407-6 2020-06-12T02:38:39.745 INFO:tasks.rados.rados.0.smithi107.stdout:8162: finishing write tid 7 to smithi10729407-6 2020-06-12T02:38:39.745 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1491 (ObjNum 8161 snap 0 seq_num 8161) dirty exists 2020-06-12T02:38:39.745 INFO:tasks.rados.rados.0.smithi107.stdout:8162: left oid 6 (ObjNum 8161 snap 0 seq_num 8161) 2020-06-12T02:38:39.745 INFO:tasks.rados.rados.0.smithi107.stdout:8163: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:39.746 INFO:tasks.rados.rados.0.smithi107.stdout:8160: done (3 left) 2020-06-12T02:38:39.747 INFO:tasks.rados.rados.0.smithi107.stdout:8162: done (2 left) 2020-06-12T02:38:39.747 INFO:tasks.rados.rados.0.smithi107.stdout:8164: done (1 left) 2020-06-12T02:38:39.747 INFO:tasks.rados.rados.0.smithi107.stdout:8165: delete oid 13 current snap is 0 2020-06-12T02:38:39.747 INFO:tasks.rados.rados.0.smithi107.stdout:8163: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:39.747 INFO:tasks.rados.rados.0.smithi107.stdout:8165: done (1 left) 2020-06-12T02:38:39.748 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:39.748 INFO:tasks.rados.rados.0.smithi107.stdout:8166: seq_num 8165 ranges {0=753664,753664=532480,1286144=532480,1818624=540672} 2020-06-12T02:38:39.750 INFO:tasks.rados.rados.0.smithi107.stdout:8166: writing smithi10729407-18 from 0 to 753664 tid 1 2020-06-12T02:38:39.753 INFO:tasks.rados.rados.0.smithi107.stdout:8166: writing smithi10729407-18 from 753664 to 1286144 tid 2 2020-06-12T02:38:39.757 INFO:tasks.rados.rados.0.smithi107.stdout:8166: writing smithi10729407-18 from 1286144 to 1818624 tid 3 2020-06-12T02:38:39.757 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:39.754+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:39.760 INFO:tasks.rados.rados.0.smithi107.stdout:8166: writing smithi10729407-18 from 1818624 to 2359296 tid 4 2020-06-12T02:38:39.760 INFO:tasks.rados.rados.0.smithi107.stdout:8167: delete oid 8 current snap is 0 2020-06-12T02:38:39.761 INFO:tasks.rados.rados.0.smithi107.stdout:8163: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:39.761 INFO:tasks.rados.rados.0.smithi107.stdout:8163: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:39.761 INFO:tasks.rados.rados.0.smithi107.stdout:8163: finishing write tid 6 to smithi10729407-14 2020-06-12T02:38:39.761 INFO:tasks.rados.rados.0.smithi107.stdout:8163: finishing write tid 7 to smithi10729407-14 2020-06-12T02:38:39.761 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1441 (ObjNum 8162 snap 0 seq_num 8162) dirty exists 2020-06-12T02:38:39.762 INFO:tasks.rados.rados.0.smithi107.stdout:8163: left oid 14 (ObjNum 8162 snap 0 seq_num 8162) 2020-06-12T02:38:39.762 INFO:tasks.rados.rados.0.smithi107.stdout:8166: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:39.763 INFO:tasks.rados.rados.0.smithi107.stdout:8163: done (2 left) 2020-06-12T02:38:39.763 INFO:tasks.rados.rados.0.smithi107.stdout:8167: done (1 left) 2020-06-12T02:38:39.763 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:39.764 INFO:tasks.rados.rados.0.smithi107.stdout:8168: seq_num 8167 ranges {0=786432,786432=704512,1490944=548864,2039808=352256} 2020-06-12T02:38:39.771 INFO:tasks.rados.rados.0.smithi107.stdout:8168: writing smithi10729407-11 from 0 to 786432 tid 1 2020-06-12T02:38:39.772 INFO:tasks.rados.rados.0.smithi107.stdout:8168: writing smithi10729407-11 from 786432 to 1490944 tid 2 2020-06-12T02:38:39.777 INFO:tasks.rados.rados.0.smithi107.stdout:8168: writing smithi10729407-11 from 1490944 to 2039808 tid 3 2020-06-12T02:38:39.779 INFO:tasks.rados.rados.0.smithi107.stdout:8168: writing smithi10729407-11 from 2039808 to 2392064 tid 4 2020-06-12T02:38:39.779 INFO:tasks.rados.rados.0.smithi107.stdout:8169: delete oid 16 current snap is 0 2020-06-12T02:38:39.779 INFO:tasks.rados.rados.0.smithi107.stdout:8166: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:39.779 INFO:tasks.rados.rados.0.smithi107.stdout:8166: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:39.780 INFO:tasks.rados.rados.0.smithi107.stdout:8166: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:39.780 INFO:tasks.rados.rados.0.smithi107.stdout:8166: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:39.780 INFO:tasks.rados.rados.0.smithi107.stdout:8166: finishing write tid 7 to smithi10729407-18 2020-06-12T02:38:39.780 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1522 (ObjNum 8165 snap 0 seq_num 8165) dirty exists 2020-06-12T02:38:39.780 INFO:tasks.rados.rados.0.smithi107.stdout:8166: left oid 18 (ObjNum 8165 snap 0 seq_num 8165) 2020-06-12T02:38:39.781 INFO:tasks.rados.rados.0.smithi107.stdout:8168: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:39.782 INFO:tasks.rados.rados.0.smithi107.stdout:8166: done (2 left) 2020-06-12T02:38:39.782 INFO:tasks.rados.rados.0.smithi107.stdout:8169: done (1 left) 2020-06-12T02:38:39.783 INFO:tasks.rados.rados.0.smithi107.stdout:8170: delete oid 3 current snap is 0 2020-06-12T02:38:39.783 INFO:tasks.rados.rados.0.smithi107.stdout:8168: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:39.785 INFO:tasks.rados.rados.0.smithi107.stdout:8170: done (1 left) 2020-06-12T02:38:39.785 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:38:39.786 INFO:tasks.rados.rados.0.smithi107.stdout:8171: seq_num 8170 ranges {11976704=32768} 2020-06-12T02:38:39.848 INFO:tasks.rados.rados.0.smithi107.stdout:8171: writing smithi10729407-10 from 11976704 to 12009472 tid 1 2020-06-12T02:38:39.848 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:39.848 INFO:tasks.rados.rados.0.smithi107.stdout:8168: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:39.849 INFO:tasks.rados.rados.0.smithi107.stdout:8168: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:39.849 INFO:tasks.rados.rados.0.smithi107.stdout:8168: finishing write tid 6 to smithi10729407-11 2020-06-12T02:38:39.849 INFO:tasks.rados.rados.0.smithi107.stdout:8168: finishing write tid 7 to smithi10729407-11 2020-06-12T02:38:39.849 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1446 (ObjNum 8167 snap 0 seq_num 8167) dirty exists 2020-06-12T02:38:39.849 INFO:tasks.rados.rados.0.smithi107.stdout:8168: left oid 11 (ObjNum 8167 snap 0 seq_num 8167) 2020-06-12T02:38:39.849 INFO:tasks.rados.rados.0.smithi107.stdout:8172: seq_num 8171 ranges {1941504=49152} 2020-06-12T02:38:39.858 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:39.859 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:39.860 INFO:tasks.rados.rados.0.smithi107.stdout:8172: writing smithi10729407-15 from 1941504 to 1990656 tid 1 2020-06-12T02:38:39.860 INFO:tasks.rados.rados.0.smithi107.stdout:8168: done (2 left) 2020-06-12T02:38:39.861 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:38:39.861 INFO:tasks.rados.rados.0.smithi107.stdout:8173: seq_num 8172 ranges {0=65536} 2020-06-12T02:38:39.861 INFO:tasks.rados.rados.0.smithi107.stdout:8173: writing smithi10729407-20 from 0 to 65536 tid 1 2020-06-12T02:38:39.862 INFO:tasks.rados.rados.0.smithi107.stdout:8174: delete oid 7 current snap is 0 2020-06-12T02:38:39.862 INFO:tasks.rados.rados.0.smithi107.stdout:8171: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:39.862 INFO:tasks.rados.rados.0.smithi107.stdout:8171: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:39.862 INFO:tasks.rados.rados.0.smithi107.stdout:8171: finishing write tid 4 to smithi10729407-10 2020-06-12T02:38:39.863 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 581 (ObjNum 8170 snap 0 seq_num 8170) dirty exists 2020-06-12T02:38:39.863 INFO:tasks.rados.rados.0.smithi107.stdout:8171: left oid 10 (ObjNum 8170 snap 0 seq_num 8170) 2020-06-12T02:38:39.863 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:39.856+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:39.864 INFO:tasks.rados.rados.0.smithi107.stdout:8171: done (3 left) 2020-06-12T02:38:39.864 INFO:tasks.rados.rados.0.smithi107.stdout:8174: done (2 left) 2020-06-12T02:38:39.864 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:39.865 INFO:tasks.rados.rados.0.smithi107.stdout:8175: seq_num 8174 ranges {3694592=106496} 2020-06-12T02:38:39.870 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-06-12T02:38:39.883 INFO:tasks.rados.rados.0.smithi107.stdout:8175: writing smithi10729407-19 from 3694592 to 3801088 tid 1 2020-06-12T02:38:39.884 INFO:tasks.rados.rados.0.smithi107.stdout:8176: delete oid 3 current snap is 0 2020-06-12T02:38:39.884 INFO:tasks.rados.rados.0.smithi107.stdout:8173: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:39.884 INFO:tasks.rados.rados.0.smithi107.stdout:8172: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:39.884 INFO:tasks.rados.rados.0.smithi107.stdout:8173: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:39.885 INFO:tasks.rados.rados.0.smithi107.stdout:8172: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:39.885 INFO:tasks.rados.rados.0.smithi107.stdout:8173: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:39.885 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 597 (ObjNum 8172 snap 0 seq_num 8172) dirty exists 2020-06-12T02:38:39.886 INFO:tasks.rados.rados.0.smithi107.stdout:8173: left oid 20 (ObjNum 8172 snap 0 seq_num 8172) 2020-06-12T02:38:39.886 INFO:tasks.rados.rados.0.smithi107.stdout:8172: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:39.887 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1522 (ObjNum 8171 snap 0 seq_num 8171) dirty exists 2020-06-12T02:38:39.887 INFO:tasks.rados.rados.0.smithi107.stdout:8172: left oid 15 (ObjNum 8171 snap 0 seq_num 8171) 2020-06-12T02:38:39.888 INFO:tasks.rados.rados.0.smithi107.stdout:8172: done (3 left) 2020-06-12T02:38:39.889 INFO:tasks.rados.rados.0.smithi107.stdout:8173: done (2 left) 2020-06-12T02:38:39.889 INFO:tasks.rados.rados.0.smithi107.stdout:8176: done (1 left) 2020-06-12T02:38:39.889 INFO:tasks.rados.rados.0.smithi107.stdout:8177: delete oid 7 current snap is 0 2020-06-12T02:38:39.890 INFO:tasks.rados.rados.0.smithi107.stdout:8175: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:39.890 INFO:tasks.rados.rados.0.smithi107.stdout:8177: done (1 left) 2020-06-12T02:38:39.890 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:39.891 INFO:tasks.rados.rados.0.smithi107.stdout:8178: seq_num 8177 ranges {0=155648} 2020-06-12T02:38:39.892 INFO:tasks.rados.rados.0.smithi107.stdout:8178: writing smithi10729407-13 from 0 to 155648 tid 1 2020-06-12T02:38:39.894 INFO:tasks.rados.rados.0.smithi107.stdout:8179: delete oid 16 current snap is 0 2020-06-12T02:38:39.894 INFO:tasks.rados.rados.0.smithi107.stdout:8175: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:39.895 INFO:tasks.rados.rados.0.smithi107.stdout:8175: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:39.895 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1501 (ObjNum 8174 snap 0 seq_num 8174) dirty exists 2020-06-12T02:38:39.895 INFO:tasks.rados.rados.0.smithi107.stdout:8175: left oid 19 (ObjNum 8174 snap 0 seq_num 8174) 2020-06-12T02:38:39.896 INFO:tasks.rados.rados.0.smithi107.stdout:8175: done (2 left) 2020-06-12T02:38:39.897 INFO:tasks.rados.rados.0.smithi107.stdout:8179: done (1 left) 2020-06-12T02:38:39.897 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:39.897 INFO:tasks.rados.rados.0.smithi107.stdout:8180: seq_num 8179 ranges {0=188416} 2020-06-12T02:38:39.898 INFO:tasks.rados.rados.0.smithi107.stdout:8180: writing smithi10729407-12 from 0 to 188416 tid 1 2020-06-12T02:38:39.898 INFO:tasks.rados.rados.0.smithi107.stdout:8181: delete oid 17 current snap is 0 2020-06-12T02:38:39.898 INFO:tasks.rados.rados.0.smithi107.stdout:8178: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:39.899 INFO:tasks.rados.rados.0.smithi107.stdout:8181: done (2 left) 2020-06-12T02:38:39.900 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:38:39.900 INFO:tasks.rados.rados.0.smithi107.stdout:8182: seq_num 8181 ranges {0=221184} 2020-06-12T02:38:39.901 INFO:tasks.rados.rados.0.smithi107.stdout:8182: writing smithi10729407-3 from 0 to 221184 tid 1 2020-06-12T02:38:39.901 INFO:tasks.rados.rados.0.smithi107.stdout:8183: delete oid 20 current snap is 0 2020-06-12T02:38:39.901 INFO:tasks.rados.rados.0.smithi107.stdout:8178: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:39.902 INFO:tasks.rados.rados.0.smithi107.stdout:8180: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:39.903 INFO:tasks.rados.rados.0.smithi107.stdout:8178: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:39.903 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1505 (ObjNum 8177 snap 0 seq_num 8177) dirty exists 2020-06-12T02:38:39.904 INFO:tasks.rados.rados.0.smithi107.stdout:8178: left oid 13 (ObjNum 8177 snap 0 seq_num 8177) 2020-06-12T02:38:39.904 INFO:tasks.rados.rados.0.smithi107.stdout:8180: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:39.906 INFO:tasks.rados.rados.0.smithi107.stdout:8178: done (3 left) 2020-06-12T02:38:39.906 INFO:tasks.rados.rados.0.smithi107.stdout:8183: done (2 left) 2020-06-12T02:38:39.906 INFO:tasks.rados.rados.0.smithi107.stdout:8184: delete oid 19 current snap is 0 2020-06-12T02:38:39.907 INFO:tasks.rados.rados.0.smithi107.stdout:8180: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:39.907 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1503 (ObjNum 8179 snap 0 seq_num 8179) dirty exists 2020-06-12T02:38:39.907 INFO:tasks.rados.rados.0.smithi107.stdout:8180: left oid 12 (ObjNum 8179 snap 0 seq_num 8179) 2020-06-12T02:38:39.909 INFO:tasks.rados.rados.0.smithi107.stdout:8182: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:39.909 INFO:tasks.rados.rados.0.smithi107.stdout:8180: done (2 left) 2020-06-12T02:38:39.909 INFO:tasks.rados.rados.0.smithi107.stdout:8184: done (1 left) 2020-06-12T02:38:39.909 INFO:tasks.rados.rados.0.smithi107.stdout:8185: delete oid 11 current snap is 0 2020-06-12T02:38:39.910 INFO:tasks.rados.rados.0.smithi107.stdout:8182: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:39.911 INFO:tasks.rados.rados.0.smithi107.stdout:8182: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:39.911 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1493 (ObjNum 8181 snap 0 seq_num 8181) dirty exists 2020-06-12T02:38:39.912 INFO:tasks.rados.rados.0.smithi107.stdout:8182: left oid 3 (ObjNum 8181 snap 0 seq_num 8181) 2020-06-12T02:38:39.913 INFO:tasks.rados.rados.0.smithi107.stdout:8182: done (1 left) 2020-06-12T02:38:39.913 INFO:tasks.rados.rados.0.smithi107.stdout:8185: done (0 left) 2020-06-12T02:38:39.913 INFO:tasks.rados.rados.0.smithi107.stdout:8186: delete oid 14 current snap is 0 2020-06-12T02:38:39.917 INFO:tasks.rados.rados.0.smithi107.stdout:8186: done (0 left) 2020-06-12T02:38:39.918 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:39.918 INFO:tasks.rados.rados.0.smithi107.stdout:8187: seq_num 8186 ranges {0=303104} 2020-06-12T02:38:39.919 INFO:tasks.rados.rados.0.smithi107.stdout:8187: writing smithi10729407-8 from 0 to 303104 tid 1 2020-06-12T02:38:39.920 INFO:tasks.rados.rados.0.smithi107.stdout:8188: delete oid 10 current snap is 0 2020-06-12T02:38:39.923 INFO:tasks.rados.rados.0.smithi107.stdout:8188: done (1 left) 2020-06-12T02:38:39.923 INFO:tasks.rados.rados.0.smithi107.stdout:8189: delete oid 18 current snap is 0 2020-06-12T02:38:39.925 INFO:tasks.rados.rados.0.smithi107.stdout:8187: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:39.928 INFO:tasks.rados.rados.0.smithi107.stdout:8189: done (1 left) 2020-06-12T02:38:39.928 INFO:tasks.rados.rados.0.smithi107.stdout:8190: delete oid 9 current snap is 0 2020-06-12T02:38:39.931 INFO:tasks.rados.rados.0.smithi107.stdout:8187: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:39.933 INFO:tasks.rados.rados.0.smithi107.stdout:8187: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:39.934 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1524 (ObjNum 8186 snap 0 seq_num 8186) dirty exists 2020-06-12T02:38:39.934 INFO:tasks.rados.rados.0.smithi107.stdout:8187: left oid 8 (ObjNum 8186 snap 0 seq_num 8186) 2020-06-12T02:38:39.934 INFO:tasks.rados.rados.0.smithi107.stdout:8187: done (1 left) 2020-06-12T02:38:39.935 INFO:tasks.rados.rados.0.smithi107.stdout:8190: done (0 left) 2020-06-12T02:38:39.935 INFO:tasks.rados.rados.0.smithi107.stdout:8191: delete oid 11 current snap is 0 2020-06-12T02:38:39.937 INFO:tasks.rados.rados.0.smithi107.stdout:8191: done (0 left) 2020-06-12T02:38:39.937 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:39.938 INFO:tasks.rados.rados.0.smithi107.stdout:8192: seq_num 8191 ranges {1990656=385024} 2020-06-12T02:38:39.949 INFO:tasks.rados.rados.0.smithi107.stdout:8192: writing smithi10729407-15 from 1990656 to 2375680 tid 1 2020-06-12T02:38:39.950 INFO:tasks.rados.rados.0.smithi107.stdout:8193: delete oid 6 current snap is 0 2020-06-12T02:38:39.952 INFO:tasks.rados.rados.0.smithi107.stdout:8193: done (1 left) 2020-06-12T02:38:39.952 INFO:tasks.rados.rados.0.smithi107.stdout:8194: delete oid 12 current snap is 0 2020-06-12T02:38:39.956 INFO:tasks.rados.rados.0.smithi107.stdout:8192: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:39.959 INFO:tasks.rados.rados.0.smithi107.stdout:8194: done (1 left) 2020-06-12T02:38:39.959 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:39.960 INFO:tasks.rados.rados.0.smithi107.stdout:8195: seq_num 8194 ranges {0=442368} 2020-06-12T02:38:39.960 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:39.956+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:39.961 INFO:tasks.rados.rados.0.smithi107.stdout:8195: writing smithi10729407-12 from 0 to 442368 tid 1 2020-06-12T02:38:39.962 INFO:tasks.rados.rados.0.smithi107.stdout:8196: delete oid 20 current snap is 0 2020-06-12T02:38:39.962 INFO:tasks.rados.rados.0.smithi107.stdout:8192: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:39.968 INFO:tasks.rados.rados.0.smithi107.stdout:8192: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:39.968 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1526 (ObjNum 8191 snap 0 seq_num 8191) dirty exists 2020-06-12T02:38:39.968 INFO:tasks.rados.rados.0.smithi107.stdout:8192: left oid 15 (ObjNum 8191 snap 0 seq_num 8191) 2020-06-12T02:38:39.968 INFO:tasks.rados.rados.0.smithi107.stdout:8192: done (2 left) 2020-06-12T02:38:39.968 INFO:tasks.rados.rados.0.smithi107.stdout:8196: done (1 left) 2020-06-12T02:38:39.969 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:38:39.969 INFO:tasks.rados.rados.0.smithi107.stdout:8197: seq_num 8196 ranges {0=475136} 2020-06-12T02:38:39.970 INFO:tasks.rados.rados.0.smithi107.stdout:8197: writing smithi10729407-16 from 0 to 475136 tid 1 2020-06-12T02:38:39.970 INFO:tasks.rados.rados.0.smithi107.stdout:8198: delete oid 2 current snap is 0 2020-06-12T02:38:39.970 INFO:tasks.rados.rados.0.smithi107.stdout:8195: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:39.970 INFO:tasks.rados.rados.0.smithi107.stdout:8195: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:39.986 INFO:tasks.rados.rados.0.smithi107.stdout:8195: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:39.988 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1507 (ObjNum 8194 snap 0 seq_num 8194) dirty exists 2020-06-12T02:38:39.988 INFO:tasks.rados.rados.0.smithi107.stdout:8195: left oid 12 (ObjNum 8194 snap 0 seq_num 8194) 2020-06-12T02:38:39.989 INFO:tasks.rados.rados.0.smithi107.stdout:8195: done (2 left) 2020-06-12T02:38:39.989 INFO:tasks.rados.rados.0.smithi107.stdout:8198: done (1 left) 2020-06-12T02:38:39.989 INFO:tasks.rados.rados.0.smithi107.stdout:8199: delete oid 10 current snap is 0 2020-06-12T02:38:39.989 INFO:tasks.rados.rados.0.smithi107.stdout:8199: done (1 left) 2020-06-12T02:38:39.989 INFO:tasks.rados.rados.0.smithi107.stdout:8200: delete oid 8 current snap is 0 2020-06-12T02:38:39.990 INFO:tasks.rados.rados.0.smithi107.stdout:8197: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:39.990 INFO:tasks.rados.rados.0.smithi107.stdout:8197: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:39.990 INFO:tasks.rados.rados.0.smithi107.stdout:8197: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:39.990 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1495 (ObjNum 8196 snap 0 seq_num 8196) dirty exists 2020-06-12T02:38:39.991 INFO:tasks.rados.rados.0.smithi107.stdout:8197: left oid 16 (ObjNum 8196 snap 0 seq_num 8196) 2020-06-12T02:38:39.991 INFO:tasks.rados.rados.0.smithi107.stdout:8197: done (1 left) 2020-06-12T02:38:39.991 INFO:tasks.rados.rados.0.smithi107.stdout:8200: done (0 left) 2020-06-12T02:38:39.991 INFO:tasks.rados.rados.0.smithi107.stdout:8201: delete oid 7 current snap is 0 2020-06-12T02:38:39.992 INFO:tasks.rados.rados.0.smithi107.stdout:8201: done (0 left) 2020-06-12T02:38:39.992 INFO:tasks.rados.rados.0.smithi107.stdout:8202: delete oid 15 current snap is 0 2020-06-12T02:38:39.992 INFO:tasks.rados.rados.0.smithi107.stdout:8202: done (0 left) 2020-06-12T02:38:39.992 INFO:tasks.rados.rados.0.smithi107.stdout:8203: delete oid 13 current snap is 0 2020-06-12T02:38:39.992 INFO:tasks.rados.rados.0.smithi107.stdout:8203: done (0 left) 2020-06-12T02:38:39.993 INFO:tasks.rados.rados.0.smithi107.stdout:8204: delete oid 4 current snap is 0 2020-06-12T02:38:39.994 INFO:tasks.rados.rados.0.smithi107.stdout:8204: done (0 left) 2020-06-12T02:38:39.994 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:39.994 INFO:tasks.rados.rados.0.smithi107.stdout:8205: seq_num 8204 ranges {0=606208} 2020-06-12T02:38:39.994 INFO:tasks.rados.rados.0.smithi107.stdout:8205: writing smithi10729407-2 from 0 to 606208 tid 1 2020-06-12T02:38:39.995 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:38:39.995 INFO:tasks.rados.rados.0.smithi107.stdout:8206: seq_num 8205 ranges {0=622592} 2020-06-12T02:38:39.995 INFO:tasks.rados.rados.0.smithi107.stdout:8206: writing smithi10729407-11 from 0 to 622592 tid 1 2020-06-12T02:38:39.996 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:39.996 INFO:tasks.rados.rados.0.smithi107.stdout:8207: seq_num 8206 ranges {475136=638976} 2020-06-12T02:38:40.001 INFO:tasks.rados.rados.0.smithi107.stdout:8207: writing smithi10729407-16 from 475136 to 1114112 tid 1 2020-06-12T02:38:40.001 INFO:tasks.rados.rados.0.smithi107.stdout:8208: delete oid 3 current snap is 0 2020-06-12T02:38:40.002 INFO:tasks.rados.rados.0.smithi107.stdout:8205: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:40.002 INFO:tasks.rados.rados.0.smithi107.stdout:8205: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:40.002 INFO:tasks.rados.rados.0.smithi107.stdout:8206: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:40.002 INFO:tasks.rados.rados.0.smithi107.stdout:8205: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:40.002 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 768 (ObjNum 8204 snap 0 seq_num 8204) dirty exists 2020-06-12T02:38:40.003 INFO:tasks.rados.rados.0.smithi107.stdout:8205: left oid 2 (ObjNum 8204 snap 0 seq_num 8204) 2020-06-12T02:38:40.003 INFO:tasks.rados.rados.0.smithi107.stdout:8206: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:40.005 INFO:tasks.rados.rados.0.smithi107.stdout:8206: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:40.005 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1451 (ObjNum 8205 snap 0 seq_num 8205) dirty exists 2020-06-12T02:38:40.006 INFO:tasks.rados.rados.0.smithi107.stdout:8206: left oid 11 (ObjNum 8205 snap 0 seq_num 8205) 2020-06-12T02:38:40.009 INFO:tasks.rados.rados.0.smithi107.stdout:8207: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:40.013 INFO:tasks.rados.rados.0.smithi107.stdout:8205: done (3 left) 2020-06-12T02:38:40.013 INFO:tasks.rados.rados.0.smithi107.stdout:8206: done (2 left) 2020-06-12T02:38:40.013 INFO:tasks.rados.rados.0.smithi107.stdout:8208: done (1 left) 2020-06-12T02:38:40.013 INFO:tasks.rados.rados.0.smithi107.stdout:8209: delete oid 3 current snap is 0 2020-06-12T02:38:40.014 INFO:tasks.rados.rados.0.smithi107.stdout:8207: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:40.014 INFO:tasks.rados.rados.0.smithi107.stdout:8207: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:40.015 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1498 (ObjNum 8206 snap 0 seq_num 8206) dirty exists 2020-06-12T02:38:40.015 INFO:tasks.rados.rados.0.smithi107.stdout:8207: left oid 16 (ObjNum 8206 snap 0 seq_num 8206) 2020-06-12T02:38:40.017 INFO:tasks.rados.rados.0.smithi107.stdout:8207: done (1 left) 2020-06-12T02:38:40.018 INFO:tasks.rados.rados.0.smithi107.stdout:8209: done (0 left) 2020-06-12T02:38:40.018 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:40.018 INFO:tasks.rados.rados.0.smithi107.stdout:8210: seq_num 8209 ranges {0=688128} 2020-06-12T02:38:40.019 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:40.019 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:40.021 INFO:tasks.rados.rados.0.smithi107.stdout:8210: writing smithi10729407-18 from 0 to 688128 tid 1 2020-06-12T02:38:40.021 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:38:40.021 INFO:tasks.rados.rados.0.smithi107.stdout:8211: seq_num 8210 ranges {0=704512} 2020-06-12T02:38:40.025 INFO:tasks.rados.rados.0.smithi107.stdout:8211: writing smithi10729407-10 from 0 to 704512 tid 1 2020-06-12T02:38:40.025 INFO:tasks.rados.rados.0.smithi107.stdout:8212: delete oid 14 current snap is 0 2020-06-12T02:38:40.029 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-06-12T02:38:40.030 INFO:tasks.rados.rados.0.smithi107.stdout:8210: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:40.031 INFO:tasks.rados.rados.0.smithi107.stdout:8212: done (2 left) 2020-06-12T02:38:40.031 INFO:tasks.rados.rados.0.smithi107.stdout:8213: delete oid 11 current snap is 0 2020-06-12T02:38:40.032 INFO:tasks.rados.rados.0.smithi107.stdout:8211: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:40.035 INFO:tasks.rados.rados.0.smithi107.stdout:8211: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:40.037 INFO:tasks.rados.rados.0.smithi107.stdout:8210: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:40.037 INFO:tasks.rados.rados.0.smithi107.stdout:8211: finishing write tid 4 to smithi10729407-10 2020-06-12T02:38:40.038 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 585 (ObjNum 8210 snap 0 seq_num 8210) dirty exists 2020-06-12T02:38:40.038 INFO:tasks.rados.rados.0.smithi107.stdout:8211: left oid 10 (ObjNum 8210 snap 0 seq_num 8210) 2020-06-12T02:38:40.038 INFO:tasks.rados.rados.0.smithi107.stdout:8211: done (2 left) 2020-06-12T02:38:40.038 INFO:tasks.rados.rados.0.smithi107.stdout:8213: done (1 left) 2020-06-12T02:38:40.038 INFO:tasks.rados.rados.0.smithi107.stdout:8214: delete oid 10 current snap is 0 2020-06-12T02:38:40.039 INFO:tasks.rados.rados.0.smithi107.stdout:8210: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:40.039 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1526 (ObjNum 8209 snap 0 seq_num 8209) dirty exists 2020-06-12T02:38:40.039 INFO:tasks.rados.rados.0.smithi107.stdout:8210: left oid 18 (ObjNum 8209 snap 0 seq_num 8209) 2020-06-12T02:38:40.043 INFO:tasks.rados.rados.0.smithi107.stdout:8210: done (1 left) 2020-06-12T02:38:40.044 INFO:tasks.rados.rados.0.smithi107.stdout:8214: done (0 left) 2020-06-12T02:38:40.044 INFO:tasks.rados.rados.0.smithi107.stdout:8215: delete oid 13 current snap is 0 2020-06-12T02:38:40.049 INFO:tasks.rados.rados.0.smithi107.stdout:8215: done (0 left) 2020-06-12T02:38:40.049 INFO:tasks.rados.rados.0.smithi107.stdout:8216: delete oid 5 current snap is 0 2020-06-12T02:38:40.051 INFO:tasks.rados.rados.0.smithi107.stdout:8216: done (0 left) 2020-06-12T02:38:40.052 INFO:tasks.rados.rados.0.smithi107.stdout:8217: delete oid 5 current snap is 0 2020-06-12T02:38:40.059 INFO:tasks.rados.rados.0.smithi107.stdout:8217: done (0 left) 2020-06-12T02:38:40.059 INFO:tasks.rados.rados.0.smithi107.stdout:8218: delete oid 6 current snap is 0 2020-06-12T02:38:40.061 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:40.059+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:40.062 INFO:tasks.rados.rados.0.smithi107.stdout:8218: done (0 left) 2020-06-12T02:38:40.062 INFO:tasks.rados.rados.0.smithi107.stdout:8219: delete oid 2 current snap is 0 2020-06-12T02:38:40.064 INFO:tasks.rados.rados.0.smithi107.stdout:8219: done (0 left) 2020-06-12T02:38:40.064 INFO:tasks.rados.rados.0.smithi107.stdout:8220: delete oid 4 current snap is 0 2020-06-12T02:38:40.067 INFO:tasks.rados.rados.0.smithi107.stdout:8220: done (0 left) 2020-06-12T02:38:40.067 INFO:tasks.rados.rados.0.smithi107.stdout:8221: delete oid 17 current snap is 0 2020-06-12T02:38:40.070 INFO:tasks.rados.rados.0.smithi107.stdout:8221: done (0 left) 2020-06-12T02:38:40.070 INFO:tasks.rados.rados.0.smithi107.stdout:8222: delete oid 13 current snap is 0 2020-06-12T02:38:40.072 INFO:tasks.rados.rados.0.smithi107.stdout:8222: done (0 left) 2020-06-12T02:38:40.073 INFO:tasks.rados.rados.0.smithi107.stdout:8223: delete oid 18 current snap is 0 2020-06-12T02:38:40.075 INFO:tasks.rados.rados.0.smithi107.stdout:8223: done (0 left) 2020-06-12T02:38:40.075 INFO:tasks.rados.rados.0.smithi107.stdout:8224: delete oid 12 current snap is 0 2020-06-12T02:38:40.078 INFO:tasks.rados.rados.0.smithi107.stdout:8224: done (0 left) 2020-06-12T02:38:40.078 INFO:tasks.rados.rados.0.smithi107.stdout:8225: delete oid 10 current snap is 0 2020-06-12T02:38:40.083 INFO:tasks.rados.rados.0.smithi107.stdout:8225: done (0 left) 2020-06-12T02:38:40.083 INFO:tasks.rados.rados.0.smithi107.stdout:8226: delete oid 16 current snap is 0 2020-06-12T02:38:40.085 INFO:tasks.rados.rados.0.smithi107.stdout:8226: done (0 left) 2020-06-12T02:38:40.085 INFO:tasks.rados.rados.0.smithi107.stdout:8227: delete oid 10 current snap is 0 2020-06-12T02:38:40.087 INFO:tasks.rados.rados.0.smithi107.stdout:8227: done (0 left) 2020-06-12T02:38:40.087 INFO:tasks.rados.rados.0.smithi107.stdout:8228: delete oid 8 current snap is 0 2020-06-12T02:38:40.090 INFO:tasks.rados.rados.0.smithi107.stdout:8228: done (0 left) 2020-06-12T02:38:40.090 INFO:tasks.rados.rados.0.smithi107.stdout:8229: delete oid 6 current snap is 0 2020-06-12T02:38:40.093 INFO:tasks.rados.rados.0.smithi107.stdout:8229: done (0 left) 2020-06-12T02:38:40.093 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:40.093 INFO:tasks.rados.rados.0.smithi107.stdout:8230: seq_num 8229 ranges {0=622592,622592=401408} 2020-06-12T02:38:40.096 INFO:tasks.rados.rados.0.smithi107.stdout:8230: writing smithi10729407-4 from 0 to 622592 tid 1 2020-06-12T02:38:40.098 INFO:tasks.rados.rados.0.smithi107.stdout:8230: writing smithi10729407-4 from 622592 to 1024000 tid 2 2020-06-12T02:38:40.098 INFO:tasks.rados.rados.0.smithi107.stdout:8231: delete oid 14 current snap is 0 2020-06-12T02:38:40.101 INFO:tasks.rados.rados.0.smithi107.stdout:8231: done (1 left) 2020-06-12T02:38:40.101 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:40.101 INFO:tasks.rados.rados.0.smithi107.stdout:8232: seq_num 8231 ranges {0=663552,663552=401408} 2020-06-12T02:38:40.105 INFO:tasks.rados.rados.0.smithi107.stdout:8232: writing smithi10729407-14 from 0 to 663552 tid 1 2020-06-12T02:38:40.107 INFO:tasks.rados.rados.0.smithi107.stdout:8232: writing smithi10729407-14 from 663552 to 1064960 tid 2 2020-06-12T02:38:40.108 INFO:tasks.rados.rados.0.smithi107.stdout:8233: delete oid 17 current snap is 0 2020-06-12T02:38:40.108 INFO:tasks.rados.rados.0.smithi107.stdout:8230: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:40.111 INFO:tasks.rados.rados.0.smithi107.stdout:8230: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:40.111 INFO:tasks.rados.rados.0.smithi107.stdout:8230: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:40.111 INFO:tasks.rados.rados.0.smithi107.stdout:8232: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:40.113 INFO:tasks.rados.rados.0.smithi107.stdout:8233: done (2 left) 2020-06-12T02:38:40.113 INFO:tasks.rados.rados.0.smithi107.stdout:8234: delete oid 1 current snap is 0 2020-06-12T02:38:40.114 INFO:tasks.rados.rados.0.smithi107.stdout:8230: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:40.114 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1549 (ObjNum 8229 snap 0 seq_num 8229) dirty exists 2020-06-12T02:38:40.114 INFO:tasks.rados.rados.0.smithi107.stdout:8230: left oid 4 (ObjNum 8229 snap 0 seq_num 8229) 2020-06-12T02:38:40.117 INFO:tasks.rados.rados.0.smithi107.stdout:8232: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:40.117 INFO:tasks.rados.rados.0.smithi107.stdout:8232: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:40.118 INFO:tasks.rados.rados.0.smithi107.stdout:8230: done (2 left) 2020-06-12T02:38:40.118 INFO:tasks.rados.rados.0.smithi107.stdout:8234: done (1 left) 2020-06-12T02:38:40.118 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:38:40.118 INFO:tasks.rados.rados.0.smithi107.stdout:8235: seq_num 8234 ranges {0=712704,712704=401408} 2020-06-12T02:38:40.122 INFO:tasks.rados.rados.0.smithi107.stdout:8235: writing smithi10729407-10 from 0 to 712704 tid 1 2020-06-12T02:38:40.125 INFO:tasks.rados.rados.0.smithi107.stdout:8235: writing smithi10729407-10 from 712704 to 1114112 tid 2 2020-06-12T02:38:40.125 INFO:tasks.rados.rados.0.smithi107.stdout:8236: delete oid 17 current snap is 0 2020-06-12T02:38:40.125 INFO:tasks.rados.rados.0.smithi107.stdout:8232: finishing write tid 5 to smithi10729407-14 2020-06-12T02:38:40.125 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1457 (ObjNum 8231 snap 0 seq_num 8231) dirty exists 2020-06-12T02:38:40.126 INFO:tasks.rados.rados.0.smithi107.stdout:8232: left oid 14 (ObjNum 8231 snap 0 seq_num 8231) 2020-06-12T02:38:40.128 INFO:tasks.rados.rados.0.smithi107.stdout:8232: done (2 left) 2020-06-12T02:38:40.128 INFO:tasks.rados.rados.0.smithi107.stdout:8236: done (1 left) 2020-06-12T02:38:40.128 INFO:tasks.rados.rados.0.smithi107.stdout:8237: delete oid 18 current snap is 0 2020-06-12T02:38:40.131 INFO:tasks.rados.rados.0.smithi107.stdout:8235: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:40.131 INFO:tasks.rados.rados.0.smithi107.stdout:8237: done (1 left) 2020-06-12T02:38:40.131 INFO:tasks.rados.rados.0.smithi107.stdout:8238: delete oid 6 current snap is 0 2020-06-12T02:38:40.132 INFO:tasks.rados.rados.0.smithi107.stdout:8235: finishing write tid 2 to smithi10729407-10 2020-06-12T02:38:40.132 INFO:tasks.rados.rados.0.smithi107.stdout:8235: finishing write tid 4 to smithi10729407-10 2020-06-12T02:38:40.136 INFO:tasks.rados.rados.0.smithi107.stdout:8235: finishing write tid 5 to smithi10729407-10 2020-06-12T02:38:40.136 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 591 (ObjNum 8234 snap 0 seq_num 8234) dirty exists 2020-06-12T02:38:40.136 INFO:tasks.rados.rados.0.smithi107.stdout:8235: left oid 10 (ObjNum 8234 snap 0 seq_num 8234) 2020-06-12T02:38:40.137 INFO:tasks.rados.rados.0.smithi107.stdout:8235: done (1 left) 2020-06-12T02:38:40.137 INFO:tasks.rados.rados.0.smithi107.stdout:8238: done (0 left) 2020-06-12T02:38:40.137 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:38:40.137 INFO:tasks.rados.rados.0.smithi107.stdout:8239: seq_num 8238 ranges {0=778240,778240=401408} 2020-06-12T02:38:40.138 INFO:tasks.rados.rados.0.smithi107.stdout:8239: writing smithi10729407-5 from 0 to 778240 tid 1 2020-06-12T02:38:40.140 INFO:tasks.rados.rados.0.smithi107.stdout:8239: writing smithi10729407-5 from 778240 to 1179648 tid 2 2020-06-12T02:38:40.140 INFO:tasks.rados.rados.0.smithi107.stdout:8240: delete oid 4 current snap is 0 2020-06-12T02:38:40.143 INFO:tasks.rados.rados.0.smithi107.stdout:8240: done (1 left) 2020-06-12T02:38:40.144 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:40.144 INFO:tasks.rados.rados.0.smithi107.stdout:8241: seq_num 8240 ranges {0=409600,409600=704512,1114112=98304} 2020-06-12T02:38:40.146 INFO:tasks.rados.rados.0.smithi107.stdout:8241: writing smithi10729407-18 from 0 to 409600 tid 1 2020-06-12T02:38:40.151 INFO:tasks.rados.rados.0.smithi107.stdout:8241: writing smithi10729407-18 from 409600 to 1114112 tid 2 2020-06-12T02:38:40.151 INFO:tasks.rados.rados.0.smithi107.stdout:8241: writing smithi10729407-18 from 1114112 to 1212416 tid 3 2020-06-12T02:38:40.152 INFO:tasks.rados.rados.0.smithi107.stdout:8242: delete oid 6 current snap is 0 2020-06-12T02:38:40.152 INFO:tasks.rados.rados.0.smithi107.stdout:8239: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:40.152 INFO:tasks.rados.rados.0.smithi107.stdout:8239: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:40.153 INFO:tasks.rados.rados.0.smithi107.stdout:8241: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:40.154 INFO:tasks.rados.rados.0.smithi107.stdout:8239: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:40.154 INFO:tasks.rados.rados.0.smithi107.stdout:8242: done (2 left) 2020-06-12T02:38:40.155 INFO:tasks.rados.rados.0.smithi107.stdout:8243: delete oid 8 current snap is 0 2020-06-12T02:38:40.157 INFO:tasks.rados.rados.0.smithi107.stdout:8239: finishing write tid 5 to smithi10729407-5 2020-06-12T02:38:40.158 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 827 (ObjNum 8238 snap 0 seq_num 8238) dirty exists 2020-06-12T02:38:40.158 INFO:tasks.rados.rados.0.smithi107.stdout:8239: left oid 5 (ObjNum 8238 snap 0 seq_num 8238) 2020-06-12T02:38:40.158 INFO:tasks.rados.rados.0.smithi107.stdout:8239: done (2 left) 2020-06-12T02:38:40.159 INFO:tasks.rados.rados.0.smithi107.stdout:8243: done (1 left) 2020-06-12T02:38:40.159 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:38:40.159 INFO:tasks.rados.rados.0.smithi107.stdout:8244: seq_num 8243 ranges {1114112=458752,1572864=409600,1982464=393216} 2020-06-12T02:38:40.162 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:40.159+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:40.167 INFO:tasks.rados.rados.0.smithi107.stdout:8244: writing smithi10729407-10 from 1114112 to 1572864 tid 1 2020-06-12T02:38:40.169 INFO:tasks.rados.rados.0.smithi107.stdout:8244: writing smithi10729407-10 from 1572864 to 1982464 tid 2 2020-06-12T02:38:40.171 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:40.171 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:40.172 INFO:tasks.rados.rados.0.smithi107.stdout:8244: writing smithi10729407-10 from 1982464 to 2375680 tid 3 2020-06-12T02:38:40.172 INFO:tasks.rados.rados.0.smithi107.stdout:8245: delete oid 2 current snap is 0 2020-06-12T02:38:40.172 INFO:tasks.rados.rados.0.smithi107.stdout:8241: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:40.172 INFO:tasks.rados.rados.0.smithi107.stdout:8241: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:40.173 INFO:tasks.rados.rados.0.smithi107.stdout:8241: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:40.173 INFO:tasks.rados.rados.0.smithi107.stdout:8241: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:40.173 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1532 (ObjNum 8240 snap 0 seq_num 8240) dirty exists 2020-06-12T02:38:40.173 INFO:tasks.rados.rados.0.smithi107.stdout:8241: left oid 18 (ObjNum 8240 snap 0 seq_num 8240) 2020-06-12T02:38:40.173 INFO:tasks.rados.rados.0.smithi107.stdout:8244: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:40.174 INFO:tasks.rados.rados.0.smithi107.stdout:8241: done (2 left) 2020-06-12T02:38:40.174 INFO:tasks.rados.rados.0.smithi107.stdout:8245: done (1 left) 2020-06-12T02:38:40.174 INFO:tasks.rados.rados.0.smithi107.stdout:8246: delete oid 11 current snap is 0 2020-06-12T02:38:40.178 INFO:tasks.rados.rados.0.smithi107.stdout:8246: done (1 left) 2020-06-12T02:38:40.178 INFO:tasks.rados.rados.0.smithi107.stdout:8247: delete oid 11 current snap is 0 2020-06-12T02:38:40.179 INFO:tasks.rados.rados.0.smithi107.stdout:8244: finishing write tid 2 to smithi10729407-10 2020-06-12T02:38:40.180 INFO:tasks.rados.rados.0.smithi107.stdout:8244: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:40.181 INFO:tasks.rados.rados.0.smithi107.stdout:8244: finishing write tid 5 to smithi10729407-10 2020-06-12T02:38:40.182 INFO:tasks.rados.rados.0.smithi107.stdout:8247: done (1 left) 2020-06-12T02:38:40.182 INFO:tasks.rados.rados.0.smithi107.stdout:8248: delete oid 8 current snap is 0 2020-06-12T02:38:40.183 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-06-12T02:38:40.184 INFO:tasks.rados.rados.0.smithi107.stdout:8244: finishing write tid 6 to smithi10729407-10 2020-06-12T02:38:40.184 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 595 (ObjNum 8243 snap 0 seq_num 8243) dirty exists 2020-06-12T02:38:40.184 INFO:tasks.rados.rados.0.smithi107.stdout:8244: left oid 10 (ObjNum 8243 snap 0 seq_num 8243) 2020-06-12T02:38:40.185 INFO:tasks.rados.rados.0.smithi107.stdout:8244: done (1 left) 2020-06-12T02:38:40.188 INFO:tasks.rados.rados.0.smithi107.stdout:8248: done (0 left) 2020-06-12T02:38:40.189 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:40.189 INFO:tasks.rados.rados.0.smithi107.stdout:8249: seq_num 8248 ranges {0=540672,540672=630784,1171456=172032} 2020-06-12T02:38:40.195 INFO:tasks.rados.rados.0.smithi107.stdout:8249: writing smithi10729407-16 from 0 to 540672 tid 1 2020-06-12T02:38:40.195 INFO:tasks.rados.rados.0.smithi107.stdout:8249: writing smithi10729407-16 from 540672 to 1171456 tid 2 2020-06-12T02:38:40.195 INFO:tasks.rados.rados.0.smithi107.stdout:8249: writing smithi10729407-16 from 1171456 to 1343488 tid 3 2020-06-12T02:38:40.196 INFO:tasks.rados.rados.0.smithi107.stdout:8250: delete oid 20 current snap is 0 2020-06-12T02:38:40.196 INFO:tasks.rados.rados.0.smithi107.stdout:8249: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:40.196 INFO:tasks.rados.rados.0.smithi107.stdout:8250: done (1 left) 2020-06-12T02:38:40.196 INFO:tasks.rados.rados.0.smithi107.stdout:8251: delete oid 17 current snap is 0 2020-06-12T02:38:40.197 INFO:tasks.rados.rados.0.smithi107.stdout:8251: done (1 left) 2020-06-12T02:38:40.197 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:38:40.198 INFO:tasks.rados.rados.0.smithi107.stdout:8252: seq_num 8251 ranges {0=598016,598016=729088,1327104=73728} 2020-06-12T02:38:40.201 INFO:tasks.rados.rados.0.smithi107.stdout:8252: writing smithi10729407-20 from 0 to 598016 tid 1 2020-06-12T02:38:40.206 INFO:tasks.rados.rados.0.smithi107.stdout:8252: writing smithi10729407-20 from 598016 to 1327104 tid 2 2020-06-12T02:38:40.207 INFO:tasks.rados.rados.0.smithi107.stdout:8252: writing smithi10729407-20 from 1327104 to 1400832 tid 3 2020-06-12T02:38:40.207 INFO:tasks.rados.rados.0.smithi107.stdout:8253: delete oid 12 current snap is 0 2020-06-12T02:38:40.207 INFO:tasks.rados.rados.0.smithi107.stdout:8249: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:40.207 INFO:tasks.rados.rados.0.smithi107.stdout:8249: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:40.208 INFO:tasks.rados.rados.0.smithi107.stdout:8249: finishing write tid 5 to smithi10729407-16 2020-06-12T02:38:40.208 INFO:tasks.rados.rados.0.smithi107.stdout:8249: finishing write tid 6 to smithi10729407-16 2020-06-12T02:38:40.208 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1504 (ObjNum 8248 snap 0 seq_num 8248) dirty exists 2020-06-12T02:38:40.208 INFO:tasks.rados.rados.0.smithi107.stdout:8249: left oid 16 (ObjNum 8248 snap 0 seq_num 8248) 2020-06-12T02:38:40.209 INFO:tasks.rados.rados.0.smithi107.stdout:8249: done (2 left) 2020-06-12T02:38:40.209 INFO:tasks.rados.rados.0.smithi107.stdout:8253: done (1 left) 2020-06-12T02:38:40.210 INFO:tasks.rados.rados.0.smithi107.stdout:8254: delete oid 9 current snap is 0 2020-06-12T02:38:40.211 INFO:tasks.rados.rados.0.smithi107.stdout:8252: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:40.212 INFO:tasks.rados.rados.0.smithi107.stdout:8252: finishing write tid 2 to smithi10729407-20 2020-06-12T02:38:40.213 INFO:tasks.rados.rados.0.smithi107.stdout:8252: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:40.213 INFO:tasks.rados.rados.0.smithi107.stdout:8254: done (1 left) 2020-06-12T02:38:40.213 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:40.213 INFO:tasks.rados.rados.0.smithi107.stdout:8255: seq_num 8254 ranges {0=647168,647168=786432,1433600=16384} 2020-06-12T02:38:40.216 INFO:tasks.rados.rados.0.smithi107.stdout:8255: writing smithi10729407-2 from 0 to 647168 tid 1 2020-06-12T02:38:40.221 INFO:tasks.rados.rados.0.smithi107.stdout:8255: writing smithi10729407-2 from 647168 to 1433600 tid 2 2020-06-12T02:38:40.222 INFO:tasks.rados.rados.0.smithi107.stdout:8255: writing smithi10729407-2 from 1433600 to 1449984 tid 3 2020-06-12T02:38:40.222 INFO:tasks.rados.rados.0.smithi107.stdout:8256: delete oid 4 current snap is 0 2020-06-12T02:38:40.222 INFO:tasks.rados.rados.0.smithi107.stdout:8252: finishing write tid 5 to smithi10729407-20 2020-06-12T02:38:40.222 INFO:tasks.rados.rados.0.smithi107.stdout:8252: finishing write tid 6 to smithi10729407-20 2020-06-12T02:38:40.222 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 604 (ObjNum 8251 snap 0 seq_num 8251) dirty exists 2020-06-12T02:38:40.223 INFO:tasks.rados.rados.0.smithi107.stdout:8252: left oid 20 (ObjNum 8251 snap 0 seq_num 8251) 2020-06-12T02:38:40.226 INFO:tasks.rados.rados.0.smithi107.stdout:8252: done (2 left) 2020-06-12T02:38:40.227 INFO:tasks.rados.rados.0.smithi107.stdout:8256: done (1 left) 2020-06-12T02:38:40.228 INFO:tasks.rados.rados.0.smithi107.stdout:8257: delete oid 6 current snap is 0 2020-06-12T02:38:40.229 INFO:tasks.rados.rados.0.smithi107.stdout:8255: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:40.229 INFO:tasks.rados.rados.0.smithi107.stdout:8255: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:40.238 INFO:tasks.rados.rados.0.smithi107.stdout:8255: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:40.238 INFO:tasks.rados.rados.0.smithi107.stdout:8257: done (1 left) 2020-06-12T02:38:40.239 INFO:tasks.rados.rados.0.smithi107.stdout:8258: delete oid 11 current snap is 0 2020-06-12T02:38:40.239 INFO:tasks.rados.rados.0.smithi107.stdout:8255: finishing write tid 5 to smithi10729407-2 2020-06-12T02:38:40.239 INFO:tasks.rados.rados.0.smithi107.stdout:8255: finishing write tid 6 to smithi10729407-2 2020-06-12T02:38:40.239 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 774 (ObjNum 8254 snap 0 seq_num 8254) dirty exists 2020-06-12T02:38:40.239 INFO:tasks.rados.rados.0.smithi107.stdout:8255: left oid 2 (ObjNum 8254 snap 0 seq_num 8254) 2020-06-12T02:38:40.239 INFO:tasks.rados.rados.0.smithi107.stdout:8255: done (1 left) 2020-06-12T02:38:40.240 INFO:tasks.rados.rados.0.smithi107.stdout:8258: done (0 left) 2020-06-12T02:38:40.240 INFO:tasks.rados.rados.0.smithi107.stdout:8259: delete oid 12 current snap is 0 2020-06-12T02:38:40.240 INFO:tasks.rados.rados.0.smithi107.stdout:8259: done (0 left) 2020-06-12T02:38:40.240 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:38:40.240 INFO:tasks.rados.rados.0.smithi107.stdout:8260: seq_num 8259 ranges {0=729088,729088=655360,1384448=147456} 2020-06-12T02:38:40.242 INFO:tasks.rados.rados.0.smithi107.stdout:8260: writing smithi10729407-1 from 0 to 729088 tid 1 2020-06-12T02:38:40.242 INFO:tasks.rados.rados.0.smithi107.stdout:8260: writing smithi10729407-1 from 729088 to 1384448 tid 2 2020-06-12T02:38:40.243 INFO:tasks.rados.rados.0.smithi107.stdout:8260: writing smithi10729407-1 from 1384448 to 1531904 tid 3 2020-06-12T02:38:40.244 INFO:tasks.rados.rados.0.smithi107.stdout:8261: delete oid 9 current snap is 0 2020-06-12T02:38:40.246 INFO:tasks.rados.rados.0.smithi107.stdout:8261: done (1 left) 2020-06-12T02:38:40.246 INFO:tasks.rados.rados.0.smithi107.stdout:8262: delete oid 12 current snap is 0 2020-06-12T02:38:40.250 INFO:tasks.rados.rados.0.smithi107.stdout:8262: done (1 left) 2020-06-12T02:38:40.250 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:40.250 INFO:tasks.rados.rados.0.smithi107.stdout:8263: seq_num 8262 ranges {0=778240,778240=712704,1490944=90112} 2020-06-12T02:38:40.254 INFO:tasks.rados.rados.0.smithi107.stdout:8263: writing smithi10729407-17 from 0 to 778240 tid 1 2020-06-12T02:38:40.258 INFO:tasks.rados.rados.0.smithi107.stdout:8263: writing smithi10729407-17 from 778240 to 1490944 tid 2 2020-06-12T02:38:40.259 INFO:tasks.rados.rados.0.smithi107.stdout:8263: writing smithi10729407-17 from 1490944 to 1581056 tid 3 2020-06-12T02:38:40.259 INFO:tasks.rados.rados.0.smithi107.stdout:8264: delete oid 12 current snap is 0 2020-06-12T02:38:40.260 INFO:tasks.rados.rados.0.smithi107.stdout:8260: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:40.260 INFO:tasks.rados.rados.0.smithi107.stdout:8260: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:40.260 INFO:tasks.rados.rados.0.smithi107.stdout:8260: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:40.260 INFO:tasks.rados.rados.0.smithi107.stdout:8260: finishing write tid 5 to smithi10729407-1 2020-06-12T02:38:40.264 INFO:tasks.rados.rados.0.smithi107.stdout:8264: done (2 left) 2020-06-12T02:38:40.264 INFO:tasks.rados.rados.0.smithi107.stdout:8265: delete oid 7 current snap is 0 2020-06-12T02:38:40.265 INFO:tasks.rados.rados.0.smithi107.stdout:8260: finishing write tid 6 to smithi10729407-1 2020-06-12T02:38:40.265 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1502 (ObjNum 8259 snap 0 seq_num 8259) dirty exists 2020-06-12T02:38:40.265 INFO:tasks.rados.rados.0.smithi107.stdout:8260: left oid 1 (ObjNum 8259 snap 0 seq_num 8259) 2020-06-12T02:38:40.266 INFO:tasks.rados.rados.0.smithi107.stdout:8263: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:40.266 INFO:tasks.rados.rados.0.smithi107.stdout:8260: done (2 left) 2020-06-12T02:38:40.266 INFO:tasks.rados.rados.0.smithi107.stdout:8265: done (1 left) 2020-06-12T02:38:40.267 INFO:tasks.rados.rados.0.smithi107.stdout:8266: delete oid 12 current snap is 0 2020-06-12T02:38:40.269 INFO:tasks.rados.rados.0.smithi107.stdout:8266: done (1 left) 2020-06-12T02:38:40.270 INFO:tasks.rados.rados.0.smithi107.stdout:8267: delete oid 10 current snap is 0 2020-06-12T02:38:40.271 INFO:tasks.rados.rados.0.smithi107.stdout:8267: done (1 left) 2020-06-12T02:38:40.271 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:40.271 INFO:tasks.rados.rados.0.smithi107.stdout:8268: seq_num 8267 ranges {0=458752,458752=581632,1040384=458752,1499136=163840} 2020-06-12T02:38:40.273 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:40.270+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:40.274 INFO:tasks.rados.rados.0.smithi107.stdout:8268: writing smithi10729407-4 from 0 to 458752 tid 1 2020-06-12T02:38:40.277 INFO:tasks.rados.rados.0.smithi107.stdout:8268: writing smithi10729407-4 from 458752 to 1040384 tid 2 2020-06-12T02:38:40.280 INFO:tasks.rados.rados.0.smithi107.stdout:8268: writing smithi10729407-4 from 1040384 to 1499136 tid 3 2020-06-12T02:38:40.281 INFO:tasks.rados.rados.0.smithi107.stdout:8268: writing smithi10729407-4 from 1499136 to 1662976 tid 4 2020-06-12T02:38:40.281 INFO:tasks.rados.rados.0.smithi107.stdout:8269: delete oid 3 current snap is 0 2020-06-12T02:38:40.282 INFO:tasks.rados.rados.0.smithi107.stdout:8263: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:40.282 INFO:tasks.rados.rados.0.smithi107.stdout:8263: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:40.283 INFO:tasks.rados.rados.0.smithi107.stdout:8269: done (2 left) 2020-06-12T02:38:40.284 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:40.284 INFO:tasks.rados.rados.0.smithi107.stdout:8270: seq_num 8269 ranges {0=491520,491520=745472,1236992=458752} 2020-06-12T02:38:40.286 INFO:tasks.rados.rados.0.smithi107.stdout:8270: writing smithi10729407-15 from 0 to 491520 tid 1 2020-06-12T02:38:40.291 INFO:tasks.rados.rados.0.smithi107.stdout:8270: writing smithi10729407-15 from 491520 to 1236992 tid 2 2020-06-12T02:38:40.294 INFO:tasks.rados.rados.0.smithi107.stdout:8270: writing smithi10729407-15 from 1236992 to 1695744 tid 3 2020-06-12T02:38:40.294 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:40.295 INFO:tasks.rados.rados.0.smithi107.stdout:8271: seq_num 8270 ranges {0=516096,516096=630784,1146880=573440} 2020-06-12T02:38:40.296 INFO:tasks.rados.rados.0.smithi107.stdout:8271: writing smithi10729407-9 from 0 to 516096 tid 1 2020-06-12T02:38:40.300 INFO:tasks.rados.rados.0.smithi107.stdout:8271: writing smithi10729407-9 from 516096 to 1146880 tid 2 2020-06-12T02:38:40.303 INFO:tasks.rados.rados.0.smithi107.stdout:8271: writing smithi10729407-9 from 1146880 to 1720320 tid 3 2020-06-12T02:38:40.304 INFO:tasks.rados.rados.0.smithi107.stdout:8272: delete oid 3 current snap is 0 2020-06-12T02:38:40.304 INFO:tasks.rados.rados.0.smithi107.stdout:8263: finishing write tid 5 to smithi10729407-17 2020-06-12T02:38:40.304 INFO:tasks.rados.rados.0.smithi107.stdout:8268: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:40.304 INFO:tasks.rados.rados.0.smithi107.stdout:8263: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:40.304 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1558 (ObjNum 8262 snap 0 seq_num 8262) dirty exists 2020-06-12T02:38:40.305 INFO:tasks.rados.rados.0.smithi107.stdout:8263: left oid 17 (ObjNum 8262 snap 0 seq_num 8262) 2020-06-12T02:38:40.305 INFO:tasks.rados.rados.0.smithi107.stdout:8268: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:40.305 INFO:tasks.rados.rados.0.smithi107.stdout:8268: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:40.305 INFO:tasks.rados.rados.0.smithi107.stdout:8268: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:40.305 INFO:tasks.rados.rados.0.smithi107.stdout:8268: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:40.305 INFO:tasks.rados.rados.0.smithi107.stdout:8270: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:40.319 INFO:tasks.rados.rados.0.smithi107.stdout:8270: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:40.319 INFO:tasks.rados.rados.0.smithi107.stdout:8268: finishing write tid 7 to smithi10729407-4 2020-06-12T02:38:40.319 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1563 (ObjNum 8267 snap 0 seq_num 8267) dirty exists 2020-06-12T02:38:40.319 INFO:tasks.rados.rados.0.smithi107.stdout:8268: left oid 4 (ObjNum 8267 snap 0 seq_num 8267) 2020-06-12T02:38:40.319 INFO:tasks.rados.rados.0.smithi107.stdout:8270: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:40.320 INFO:tasks.rados.rados.0.smithi107.stdout:8271: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:40.320 INFO:tasks.rados.rados.0.smithi107.stdout:8270: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:40.320 INFO:tasks.rados.rados.0.smithi107.stdout:8263: done (4 left) 2020-06-12T02:38:40.320 INFO:tasks.rados.rados.0.smithi107.stdout:8268: done (3 left) 2020-06-12T02:38:40.320 INFO:tasks.rados.rados.0.smithi107.stdout:8272: done (2 left) 2020-06-12T02:38:40.321 INFO:tasks.rados.rados.0.smithi107.stdout:8273: delete oid 20 current snap is 0 2020-06-12T02:38:40.321 INFO:tasks.rados.rados.0.smithi107.stdout:8270: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:40.321 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1535 (ObjNum 8269 snap 0 seq_num 8269) dirty exists 2020-06-12T02:38:40.321 INFO:tasks.rados.rados.0.smithi107.stdout:8270: left oid 15 (ObjNum 8269 snap 0 seq_num 8269) 2020-06-12T02:38:40.321 INFO:tasks.rados.rados.0.smithi107.stdout:8270: done (2 left) 2020-06-12T02:38:40.322 INFO:tasks.rados.rados.0.smithi107.stdout:8273: done (1 left) 2020-06-12T02:38:40.322 INFO:tasks.rados.rados.0.smithi107.stdout:8274: delete oid 6 current snap is 0 2020-06-12T02:38:40.322 INFO:tasks.rados.rados.0.smithi107.stdout:8271: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:40.323 INFO:tasks.rados.rados.0.smithi107.stdout:8271: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:40.323 INFO:tasks.rados.rados.0.smithi107.stdout:8274: done (1 left) 2020-06-12T02:38:40.323 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:40.324 INFO:tasks.rados.rados.0.smithi107.stdout:8275: seq_num 8274 ranges {1695744=581632,2277376=622592,2899968=581632} 2020-06-12T02:38:40.335 INFO:tasks.rados.rados.0.smithi107.stdout:8275: writing smithi10729407-15 from 1695744 to 2277376 tid 1 2020-06-12T02:38:40.338 INFO:tasks.rados.rados.0.smithi107.stdout:8275: writing smithi10729407-15 from 2277376 to 2899968 tid 2 2020-06-12T02:38:40.342 INFO:tasks.rados.rados.0.smithi107.stdout:8275: writing smithi10729407-15 from 2899968 to 3481600 tid 3 2020-06-12T02:38:40.342 INFO:tasks.rados.rados.0.smithi107.stdout:8276: delete oid 4 current snap is 0 2020-06-12T02:38:40.342 INFO:tasks.rados.rados.0.smithi107.stdout:8271: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:40.343 INFO:tasks.rados.rados.0.smithi107.stdout:8271: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:40.343 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1538 (ObjNum 8270 snap 0 seq_num 8270) dirty exists 2020-06-12T02:38:40.343 INFO:tasks.rados.rados.0.smithi107.stdout:8271: left oid 9 (ObjNum 8270 snap 0 seq_num 8270) 2020-06-12T02:38:40.346 INFO:tasks.rados.rados.0.smithi107.stdout:8271: done (2 left) 2020-06-12T02:38:40.346 INFO:tasks.rados.rados.0.smithi107.stdout:8276: done (1 left) 2020-06-12T02:38:40.347 INFO:tasks.rados.rados.0.smithi107.stdout:8277: delete oid 3 current snap is 0 2020-06-12T02:38:40.349 INFO:tasks.rados.rados.0.smithi107.stdout:8277: done (1 left) 2020-06-12T02:38:40.350 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:40.350 INFO:tasks.rados.rados.0.smithi107.stdout:8278: seq_num 8277 ranges {1581056=630784,2211840=671744,2883584=434176,3317760=98304} 2020-06-12T02:38:40.361 INFO:tasks.rados.rados.0.smithi107.stdout:8278: writing smithi10729407-17 from 1581056 to 2211840 tid 1 2020-06-12T02:38:40.363 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:40.360+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:40.365 INFO:tasks.rados.rados.0.smithi107.stdout:8278: writing smithi10729407-17 from 2211840 to 2883584 tid 2 2020-06-12T02:38:40.368 INFO:tasks.rados.rados.0.smithi107.stdout:8278: writing smithi10729407-17 from 2883584 to 3317760 tid 3 2020-06-12T02:38:40.369 INFO:tasks.rados.rados.0.smithi107.stdout:8278: writing smithi10729407-17 from 3317760 to 3416064 tid 4 2020-06-12T02:38:40.369 INFO:tasks.rados.rados.0.smithi107.stdout:8275: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:40.369 INFO:tasks.rados.rados.0.smithi107.stdout:8279: delete oid 12 current snap is 0 2020-06-12T02:38:40.369 INFO:tasks.rados.rados.0.smithi107.stdout:8275: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:40.370 INFO:tasks.rados.rados.0.smithi107.stdout:8275: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:40.370 INFO:tasks.rados.rados.0.smithi107.stdout:8275: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:40.370 INFO:tasks.rados.rados.0.smithi107.stdout:8275: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:40.370 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1539 (ObjNum 8274 snap 0 seq_num 8274) dirty exists 2020-06-12T02:38:40.370 INFO:tasks.rados.rados.0.smithi107.stdout:8275: left oid 15 (ObjNum 8274 snap 0 seq_num 8274) 2020-06-12T02:38:40.371 INFO:tasks.rados.rados.0.smithi107.stdout:8278: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:40.372 INFO:tasks.rados.rados.0.smithi107.stdout:8275: done (2 left) 2020-06-12T02:38:40.372 INFO:tasks.rados.rados.0.smithi107.stdout:8279: done (1 left) 2020-06-12T02:38:40.372 INFO:tasks.rados.rados.0.smithi107.stdout:8280: delete oid 3 current snap is 0 2020-06-12T02:38:40.377 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:40.378 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:40.378 INFO:tasks.rados.rados.0.smithi107.stdout:8280: done (1 left) 2020-06-12T02:38:40.378 INFO:tasks.rados.rados.0.smithi107.stdout:8281: delete oid 6 current snap is 0 2020-06-12T02:38:40.383 INFO:tasks.rados.rados.0.smithi107.stdout:8281: done (1 left) 2020-06-12T02:38:40.383 INFO:tasks.rados.rados.0.smithi107.stdout:8282: delete oid 16 current snap is 0 2020-06-12T02:38:40.383 INFO:tasks.rados.rados.0.smithi107.stdout:8278: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:40.383 INFO:tasks.rados.rados.0.smithi107.stdout:8278: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:40.384 INFO:tasks.rados.rados.0.smithi107.stdout:8278: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:40.384 INFO:tasks.rados.rados.0.smithi107.stdout:8278: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:40.384 INFO:tasks.rados.rados.0.smithi107.stdout:8282: done (1 left) 2020-06-12T02:38:40.385 INFO:tasks.rados.rados.0.smithi107.stdout:8283: delete oid 3 current snap is 0 2020-06-12T02:38:40.386 INFO:tasks.rados.rados.0.smithi107.stdout:8278: finishing write tid 7 to smithi10729407-17 2020-06-12T02:38:40.386 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1569 (ObjNum 8277 snap 0 seq_num 8277) dirty exists 2020-06-12T02:38:40.386 INFO:tasks.rados.rados.0.smithi107.stdout:8278: left oid 17 (ObjNum 8277 snap 0 seq_num 8277) 2020-06-12T02:38:40.387 INFO:tasks.rados.rados.0.smithi107.stdout:8278: done (1 left) 2020-06-12T02:38:40.387 INFO:tasks.rados.rados.0.smithi107.stdout:8283: done (0 left) 2020-06-12T02:38:40.387 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:40.388 INFO:tasks.rados.rados.0.smithi107.stdout:8284: seq_num 8283 ranges {0=729088,729088=425984,1155072=778240} 2020-06-12T02:38:40.389 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-06-12T02:38:40.391 INFO:tasks.rados.rados.0.smithi107.stdout:8284: writing smithi10729407-8 from 0 to 729088 tid 1 2020-06-12T02:38:40.394 INFO:tasks.rados.rados.0.smithi107.stdout:8284: writing smithi10729407-8 from 729088 to 1155072 tid 2 2020-06-12T02:38:40.398 INFO:tasks.rados.rados.0.smithi107.stdout:8284: writing smithi10729407-8 from 1155072 to 1933312 tid 3 2020-06-12T02:38:40.398 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:40.399 INFO:tasks.rados.rados.0.smithi107.stdout:8285: seq_num 8284 ranges {3481600=745472,4227072=712704,4939776=491520} 2020-06-12T02:38:40.420 INFO:tasks.rados.rados.0.smithi107.stdout:8285: writing smithi10729407-15 from 3481600 to 4227072 tid 1 2020-06-12T02:38:40.424 INFO:tasks.rados.rados.0.smithi107.stdout:8285: writing smithi10729407-15 from 4227072 to 4939776 tid 2 2020-06-12T02:38:40.427 INFO:tasks.rados.rados.0.smithi107.stdout:8285: writing smithi10729407-15 from 4939776 to 5431296 tid 3 2020-06-12T02:38:40.428 INFO:tasks.rados.rados.0.smithi107.stdout:8286: delete oid 7 current snap is 0 2020-06-12T02:38:40.428 INFO:tasks.rados.rados.0.smithi107.stdout:8284: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:40.428 INFO:tasks.rados.rados.0.smithi107.stdout:8284: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:40.428 INFO:tasks.rados.rados.0.smithi107.stdout:8284: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:40.428 INFO:tasks.rados.rados.0.smithi107.stdout:8284: finishing write tid 5 to smithi10729407-8 2020-06-12T02:38:40.429 INFO:tasks.rados.rados.0.smithi107.stdout:8284: finishing write tid 6 to smithi10729407-8 2020-06-12T02:38:40.430 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1543 (ObjNum 8283 snap 0 seq_num 8283) dirty exists 2020-06-12T02:38:40.430 INFO:tasks.rados.rados.0.smithi107.stdout:8284: left oid 8 (ObjNum 8283 snap 0 seq_num 8283) 2020-06-12T02:38:40.431 INFO:tasks.rados.rados.0.smithi107.stdout:8284: done (2 left) 2020-06-12T02:38:40.431 INFO:tasks.rados.rados.0.smithi107.stdout:8286: done (1 left) 2020-06-12T02:38:40.431 INFO:tasks.rados.rados.0.smithi107.stdout:8287: delete oid 19 current snap is 0 2020-06-12T02:38:40.433 INFO:tasks.rados.rados.0.smithi107.stdout:8287: done (1 left) 2020-06-12T02:38:40.434 INFO:tasks.rados.rados.0.smithi107.stdout:8288: delete oid 16 current snap is 0 2020-06-12T02:38:40.435 INFO:tasks.rados.rados.0.smithi107.stdout:8285: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:40.437 INFO:tasks.rados.rados.0.smithi107.stdout:8285: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:40.437 INFO:tasks.rados.rados.0.smithi107.stdout:8288: done (1 left) 2020-06-12T02:38:40.437 INFO:tasks.rados.rados.0.smithi107.stdout:8289: delete oid 5 current snap is 0 2020-06-12T02:38:40.437 INFO:tasks.rados.rados.0.smithi107.stdout:8285: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:40.439 INFO:tasks.rados.rados.0.smithi107.stdout:8285: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:40.441 INFO:tasks.rados.rados.0.smithi107.stdout:8285: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:40.441 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1547 (ObjNum 8284 snap 0 seq_num 8284) dirty exists 2020-06-12T02:38:40.441 INFO:tasks.rados.rados.0.smithi107.stdout:8285: left oid 15 (ObjNum 8284 snap 0 seq_num 8284) 2020-06-12T02:38:40.442 INFO:tasks.rados.rados.0.smithi107.stdout:8285: done (1 left) 2020-06-12T02:38:40.442 INFO:tasks.rados.rados.0.smithi107.stdout:8289: done (0 left) 2020-06-12T02:38:40.442 INFO:tasks.rados.rados.0.smithi107.stdout:8290: delete oid 13 current snap is 0 2020-06-12T02:38:40.447 INFO:tasks.rados.rados.0.smithi107.stdout:8290: done (0 left) 2020-06-12T02:38:40.447 INFO:tasks.rados.rados.0.smithi107.stdout:8291: delete oid 10 current snap is 0 2020-06-12T02:38:40.449 INFO:tasks.rados.rados.0.smithi107.stdout:8291: done (0 left) 2020-06-12T02:38:40.449 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:40.449 INFO:tasks.rados.rados.0.smithi107.stdout:8292: seq_num 8291 ranges {0=466944,466944=753664,1220608=745472,1966080=106496} 2020-06-12T02:38:40.451 INFO:tasks.rados.rados.0.smithi107.stdout:8292: writing smithi10729407-12 from 0 to 466944 tid 1 2020-06-12T02:38:40.456 INFO:tasks.rados.rados.0.smithi107.stdout:8292: writing smithi10729407-12 from 466944 to 1220608 tid 2 2020-06-12T02:38:40.460 INFO:tasks.rados.rados.0.smithi107.stdout:8292: writing smithi10729407-12 from 1220608 to 1966080 tid 3 2020-06-12T02:38:40.461 INFO:tasks.rados.rados.0.smithi107.stdout:8292: writing smithi10729407-12 from 1966080 to 2072576 tid 4 2020-06-12T02:38:40.461 INFO:tasks.rados.rados.0.smithi107.stdout:8293: delete oid 19 current snap is 0 2020-06-12T02:38:40.461 INFO:tasks.rados.rados.0.smithi107.stdout:8292: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:40.465 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:40.462+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:40.470 INFO:tasks.rados.rados.0.smithi107.stdout:8292: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:40.473 INFO:tasks.rados.rados.0.smithi107.stdout:8292: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:40.473 INFO:tasks.rados.rados.0.smithi107.stdout:8292: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:40.475 INFO:tasks.rados.rados.0.smithi107.stdout:8292: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:40.478 INFO:tasks.rados.rados.0.smithi107.stdout:8293: done (1 left) 2020-06-12T02:38:40.478 INFO:tasks.rados.rados.0.smithi107.stdout:8294: delete oid 8 current snap is 0 2020-06-12T02:38:40.478 INFO:tasks.rados.rados.0.smithi107.stdout:8292: finishing write tid 7 to smithi10729407-12 2020-06-12T02:38:40.479 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1520 (ObjNum 8291 snap 0 seq_num 8291) dirty exists 2020-06-12T02:38:40.479 INFO:tasks.rados.rados.0.smithi107.stdout:8292: left oid 12 (ObjNum 8291 snap 0 seq_num 8291) 2020-06-12T02:38:40.483 INFO:tasks.rados.rados.0.smithi107.stdout:8292: done (1 left) 2020-06-12T02:38:40.483 INFO:tasks.rados.rados.0.smithi107.stdout:8294: done (0 left) 2020-06-12T02:38:40.484 INFO:tasks.rados.rados.0.smithi107.stdout:8295: delete oid 6 current snap is 0 2020-06-12T02:38:40.488 INFO:tasks.rados.rados.0.smithi107.stdout:8295: done (0 left) 2020-06-12T02:38:40.488 INFO:tasks.rados.rados.0.smithi107.stdout:8296: delete oid 18 current snap is 0 2020-06-12T02:38:40.490 INFO:tasks.rados.rados.0.smithi107.stdout:8296: done (0 left) 2020-06-12T02:38:40.491 INFO:tasks.rados.rados.0.smithi107.stdout:8297: delete oid 5 current snap is 0 2020-06-12T02:38:40.493 INFO:tasks.rados.rados.0.smithi107.stdout:8297: done (0 left) 2020-06-12T02:38:40.493 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:38:40.493 INFO:tasks.rados.rados.0.smithi107.stdout:8298: seq_num 8297 ranges {0=565248,565248=507904,1073152=737280,1810432=360448} 2020-06-12T02:38:40.497 INFO:tasks.rados.rados.0.smithi107.stdout:8298: writing smithi10729407-7 from 0 to 565248 tid 1 2020-06-12T02:38:40.499 INFO:tasks.rados.rados.0.smithi107.stdout:8298: writing smithi10729407-7 from 565248 to 1073152 tid 2 2020-06-12T02:38:40.503 INFO:tasks.rados.rados.0.smithi107.stdout:8298: writing smithi10729407-7 from 1073152 to 1810432 tid 3 2020-06-12T02:38:40.506 INFO:tasks.rados.rados.0.smithi107.stdout:8298: writing smithi10729407-7 from 1810432 to 2170880 tid 4 2020-06-12T02:38:40.506 INFO:tasks.rados.rados.0.smithi107.stdout:8299: delete oid 12 current snap is 0 2020-06-12T02:38:40.506 INFO:tasks.rados.rados.0.smithi107.stdout:8298: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:40.509 INFO:tasks.rados.rados.0.smithi107.stdout:8299: done (1 left) 2020-06-12T02:38:40.509 INFO:tasks.rados.rados.0.smithi107.stdout:8300: delete oid 10 current snap is 0 2020-06-12T02:38:40.511 INFO:tasks.rados.rados.0.smithi107.stdout:8298: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:40.512 INFO:tasks.rados.rados.0.smithi107.stdout:8300: done (1 left) 2020-06-12T02:38:40.512 INFO:tasks.rados.rados.0.smithi107.stdout:8301: delete oid 18 current snap is 0 2020-06-12T02:38:40.514 INFO:tasks.rados.rados.0.smithi107.stdout:8301: done (1 left) 2020-06-12T02:38:40.514 INFO:tasks.rados.rados.0.smithi107.stdout:8302: delete oid 9 current snap is 0 2020-06-12T02:38:40.516 INFO:tasks.rados.rados.0.smithi107.stdout:8302: done (1 left) 2020-06-12T02:38:40.517 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:38:40.517 INFO:tasks.rados.rados.0.smithi107.stdout:8303: seq_num 8302 ranges {1531904=647168,2179072=778240,2957312=524288,3481600=303104} 2020-06-12T02:38:40.528 INFO:tasks.rados.rados.0.smithi107.stdout:8303: writing smithi10729407-1 from 1531904 to 2179072 tid 1 2020-06-12T02:38:40.533 INFO:tasks.rados.rados.0.smithi107.stdout:8303: writing smithi10729407-1 from 2179072 to 2957312 tid 2 2020-06-12T02:38:40.536 INFO:tasks.rados.rados.0.smithi107.stdout:8303: writing smithi10729407-1 from 2957312 to 3481600 tid 3 2020-06-12T02:38:40.538 INFO:tasks.rados.rados.0.smithi107.stdout:8303: writing smithi10729407-1 from 3481600 to 3784704 tid 4 2020-06-12T02:38:40.538 INFO:tasks.rados.rados.0.smithi107.stdout:8304: delete oid 20 current snap is 0 2020-06-12T02:38:40.539 INFO:tasks.rados.rados.0.smithi107.stdout:8298: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:40.539 INFO:tasks.rados.rados.0.smithi107.stdout:8298: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:40.539 INFO:tasks.rados.rados.0.smithi107.stdout:8298: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:40.539 INFO:tasks.rados.rados.0.smithi107.stdout:8298: finishing write tid 7 to smithi10729407-7 2020-06-12T02:38:40.539 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1517 (ObjNum 8297 snap 0 seq_num 8297) dirty exists 2020-06-12T02:38:40.540 INFO:tasks.rados.rados.0.smithi107.stdout:8298: left oid 7 (ObjNum 8297 snap 0 seq_num 8297) 2020-06-12T02:38:40.540 INFO:tasks.rados.rados.0.smithi107.stdout:8298: done (2 left) 2020-06-12T02:38:40.540 INFO:tasks.rados.rados.0.smithi107.stdout:8304: done (1 left) 2020-06-12T02:38:40.540 INFO:tasks.rados.rados.0.smithi107.stdout:8305: delete oid 12 current snap is 0 2020-06-12T02:38:40.543 INFO:tasks.rados.rados.0.smithi107.stdout:8305: done (1 left) 2020-06-12T02:38:40.543 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:40.544 INFO:tasks.rados.rados.0.smithi107.stdout:8306: seq_num 8305 ranges {0=696320,696320=434176,1130496=704512,1835008=466944} 2020-06-12T02:38:40.548 INFO:tasks.rados.rados.0.smithi107.stdout:8306: writing smithi10729407-12 from 0 to 696320 tid 1 2020-06-12T02:38:40.551 INFO:tasks.rados.rados.0.smithi107.stdout:8306: writing smithi10729407-12 from 696320 to 1130496 tid 2 2020-06-12T02:38:40.554 INFO:tasks.rados.rados.0.smithi107.stdout:8306: writing smithi10729407-12 from 1130496 to 1835008 tid 3 2020-06-12T02:38:40.558 INFO:tasks.rados.rados.0.smithi107.stdout:8306: writing smithi10729407-12 from 1835008 to 2301952 tid 4 2020-06-12T02:38:40.558 INFO:tasks.rados.rados.0.smithi107.stdout:8307: delete oid 2 current snap is 0 2020-06-12T02:38:40.558 INFO:tasks.rados.rados.0.smithi107.stdout:8303: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:40.558 INFO:tasks.rados.rados.0.smithi107.stdout:8303: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:40.559 INFO:tasks.rados.rados.0.smithi107.stdout:8303: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:40.559 INFO:tasks.rados.rados.0.smithi107.stdout:8303: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:40.559 INFO:tasks.rados.rados.0.smithi107.stdout:8303: finishing write tid 6 to smithi10729407-1 2020-06-12T02:38:40.559 INFO:tasks.rados.rados.0.smithi107.stdout:8303: finishing write tid 7 to smithi10729407-1 2020-06-12T02:38:40.559 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1510 (ObjNum 8302 snap 0 seq_num 8302) dirty exists 2020-06-12T02:38:40.560 INFO:tasks.rados.rados.0.smithi107.stdout:8303: left oid 1 (ObjNum 8302 snap 0 seq_num 8302) 2020-06-12T02:38:40.560 INFO:tasks.rados.rados.0.smithi107.stdout:8306: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:40.560 INFO:tasks.rados.rados.0.smithi107.stdout:8303: done (2 left) 2020-06-12T02:38:40.561 INFO:tasks.rados.rados.0.smithi107.stdout:8307: done (1 left) 2020-06-12T02:38:40.561 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:38:40.561 INFO:tasks.rados.rados.0.smithi107.stdout:8308: seq_num 8307 ranges {0=729088,729088=598016,1327104=663552,1990656=344064} 2020-06-12T02:38:40.565 INFO:tasks.rados.rados.0.smithi107.stdout:8308: writing smithi10729407-3 from 0 to 729088 tid 1 2020-06-12T02:38:40.565 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:40.562+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:40.568 INFO:tasks.rados.rados.0.smithi107.stdout:8308: writing smithi10729407-3 from 729088 to 1327104 tid 2 2020-06-12T02:38:40.572 INFO:tasks.rados.rados.0.smithi107.stdout:8308: writing smithi10729407-3 from 1327104 to 1990656 tid 3 2020-06-12T02:38:40.574 INFO:tasks.rados.rados.0.smithi107.stdout:8308: writing smithi10729407-3 from 1990656 to 2334720 tid 4 2020-06-12T02:38:40.575 INFO:tasks.rados.rados.0.smithi107.stdout:8309: delete oid 10 current snap is 0 2020-06-12T02:38:40.575 INFO:tasks.rados.rados.0.smithi107.stdout:8306: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:40.575 INFO:tasks.rados.rados.0.smithi107.stdout:8306: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:40.575 INFO:tasks.rados.rados.0.smithi107.stdout:8306: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:40.575 INFO:tasks.rados.rados.0.smithi107.stdout:8306: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:40.576 INFO:tasks.rados.rados.0.smithi107.stdout:8308: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:40.577 INFO:tasks.rados.rados.0.smithi107.stdout:8306: finishing write tid 7 to smithi10729407-12 2020-06-12T02:38:40.577 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1528 (ObjNum 8305 snap 0 seq_num 8305) dirty exists 2020-06-12T02:38:40.577 INFO:tasks.rados.rados.0.smithi107.stdout:8306: left oid 12 (ObjNum 8305 snap 0 seq_num 8305) 2020-06-12T02:38:40.577 INFO:tasks.rados.rados.0.smithi107.stdout:8306: done (2 left) 2020-06-12T02:38:40.578 INFO:tasks.rados.rados.0.smithi107.stdout:8309: done (1 left) 2020-06-12T02:38:40.578 INFO:tasks.rados.rados.0.smithi107.stdout:8310: delete oid 15 current snap is 0 2020-06-12T02:38:40.580 INFO:tasks.rados.rados.0.smithi107.stdout:8310: done (1 left) 2020-06-12T02:38:40.580 INFO:tasks.rados.rados.0.smithi107.stdout:8311: delete oid 7 current snap is 0 2020-06-12T02:38:40.586 INFO:tasks.rados.rados.0.smithi107.stdout:8308: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:40.587 INFO:tasks.rados.rados.0.smithi107.stdout:8311: done (1 left) 2020-06-12T02:38:40.587 INFO:tasks.rados.rados.0.smithi107.stdout:8312: delete oid 11 current snap is 0 2020-06-12T02:38:40.587 INFO:tasks.rados.rados.0.smithi107.stdout:8308: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:40.588 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-06-12T02:38:40.589 INFO:tasks.rados.rados.0.smithi107.stdout:8308: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:40.590 INFO:tasks.rados.rados.0.smithi107.stdout:8308: finishing write tid 6 to smithi10729407-3 2020-06-12T02:38:40.590 INFO:tasks.rados.rados.0.smithi107.stdout:8312: done (1 left) 2020-06-12T02:38:40.590 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:38:40.590 INFO:tasks.rados.rados.0.smithi107.stdout:8313: seq_num 8312 ranges {0=16384} 2020-06-12T02:38:40.591 INFO:tasks.rados.rados.0.smithi107.stdout:8313: writing smithi10729407-20 from 0 to 16384 tid 1 2020-06-12T02:38:40.591 INFO:tasks.rados.rados.0.smithi107.stdout:8314: delete oid 6 current snap is 0 2020-06-12T02:38:40.591 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:40.592 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:40.592 INFO:tasks.rados.rados.0.smithi107.stdout:8308: finishing write tid 7 to smithi10729407-3 2020-06-12T02:38:40.592 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1516 (ObjNum 8307 snap 0 seq_num 8307) dirty exists 2020-06-12T02:38:40.592 INFO:tasks.rados.rados.0.smithi107.stdout:8308: left oid 3 (ObjNum 8307 snap 0 seq_num 8307) 2020-06-12T02:38:40.593 INFO:tasks.rados.rados.0.smithi107.stdout:8308: done (2 left) 2020-06-12T02:38:40.594 INFO:tasks.rados.rados.0.smithi107.stdout:8314: done (1 left) 2020-06-12T02:38:40.594 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:40.594 INFO:tasks.rados.rados.0.smithi107.stdout:8315: seq_num 8314 ranges {0=49152} 2020-06-12T02:38:40.594 INFO:tasks.rados.rados.0.smithi107.stdout:8315: writing smithi10729407-19 from 0 to 49152 tid 1 2020-06-12T02:38:40.594 INFO:tasks.rados.rados.0.smithi107.stdout:8316: delete oid 17 current snap is 0 2020-06-12T02:38:40.595 INFO:tasks.rados.rados.0.smithi107.stdout:8313: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:40.596 INFO:tasks.rados.rados.0.smithi107.stdout:8313: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:40.598 INFO:tasks.rados.rados.0.smithi107.stdout:8313: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:40.599 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 608 (ObjNum 8312 snap 0 seq_num 8312) dirty exists 2020-06-12T02:38:40.599 INFO:tasks.rados.rados.0.smithi107.stdout:8313: left oid 20 (ObjNum 8312 snap 0 seq_num 8312) 2020-06-12T02:38:40.600 INFO:tasks.rados.rados.0.smithi107.stdout:8315: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:40.600 INFO:tasks.rados.rados.0.smithi107.stdout:8315: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:40.600 INFO:tasks.rados.rados.0.smithi107.stdout:8313: done (2 left) 2020-06-12T02:38:40.600 INFO:tasks.rados.rados.0.smithi107.stdout:8316: done (1 left) 2020-06-12T02:38:40.601 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:40.601 INFO:tasks.rados.rados.0.smithi107.stdout:8317: seq_num 8316 ranges {2301952=81920} 2020-06-12T02:38:40.603 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-06-12T02:38:40.614 INFO:tasks.rados.rados.0.smithi107.stdout:8317: writing smithi10729407-12 from 2301952 to 2383872 tid 1 2020-06-12T02:38:40.615 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:40.615 INFO:tasks.rados.rados.0.smithi107.stdout:8318: seq_num 8317 ranges {0=98304} 2020-06-12T02:38:40.615 INFO:tasks.rados.rados.0.smithi107.stdout:8318: writing smithi10729407-2 from 0 to 98304 tid 1 2020-06-12T02:38:40.615 INFO:tasks.rados.rados.0.smithi107.stdout:8319: delete oid 17 current snap is 0 2020-06-12T02:38:40.616 INFO:tasks.rados.rados.0.smithi107.stdout:8315: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:40.616 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1530 (ObjNum 8314 snap 0 seq_num 8314) dirty exists 2020-06-12T02:38:40.617 INFO:tasks.rados.rados.0.smithi107.stdout:8315: left oid 19 (ObjNum 8314 snap 0 seq_num 8314) 2020-06-12T02:38:40.618 INFO:tasks.rados.rados.0.smithi107.stdout:8317: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:40.619 INFO:tasks.rados.rados.0.smithi107.stdout:8318: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:40.619 INFO:tasks.rados.rados.0.smithi107.stdout:8315: done (3 left) 2020-06-12T02:38:40.620 INFO:tasks.rados.rados.0.smithi107.stdout:8319: done (2 left) 2020-06-12T02:38:40.620 INFO:tasks.rados.rados.0.smithi107.stdout:8320: delete oid 10 current snap is 0 2020-06-12T02:38:40.620 INFO:tasks.rados.rados.0.smithi107.stdout:8318: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:40.621 INFO:tasks.rados.rados.0.smithi107.stdout:8318: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:40.621 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 777 (ObjNum 8317 snap 0 seq_num 8317) dirty exists 2020-06-12T02:38:40.621 INFO:tasks.rados.rados.0.smithi107.stdout:8318: left oid 2 (ObjNum 8317 snap 0 seq_num 8317) 2020-06-12T02:38:40.623 INFO:tasks.rados.rados.0.smithi107.stdout:8318: done (2 left) 2020-06-12T02:38:40.623 INFO:tasks.rados.rados.0.smithi107.stdout:8320: done (1 left) 2020-06-12T02:38:40.624 INFO:tasks.rados.rados.0.smithi107.stdout:8321: delete oid 3 current snap is 0 2020-06-12T02:38:40.626 INFO:tasks.rados.rados.0.smithi107.stdout:8317: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:40.627 INFO:tasks.rados.rados.0.smithi107.stdout:8317: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:40.628 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1532 (ObjNum 8316 snap 0 seq_num 8316) dirty exists 2020-06-12T02:38:40.628 INFO:tasks.rados.rados.0.smithi107.stdout:8317: left oid 12 (ObjNum 8316 snap 0 seq_num 8316) 2020-06-12T02:38:40.628 INFO:tasks.rados.rados.0.smithi107.stdout:8317: done (1 left) 2020-06-12T02:38:40.629 INFO:tasks.rados.rados.0.smithi107.stdout:8321: done (0 left) 2020-06-12T02:38:40.629 INFO:tasks.rados.rados.0.smithi107.stdout:8322: delete oid 4 current snap is 0 2020-06-12T02:38:40.634 INFO:tasks.rados.rados.0.smithi107.stdout:8322: done (0 left) 2020-06-12T02:38:40.634 INFO:tasks.rados.rados.0.smithi107.stdout:8323: delete oid 18 current snap is 0 2020-06-12T02:38:40.636 INFO:tasks.rados.rados.0.smithi107.stdout:8323: done (0 left) 2020-06-12T02:38:40.637 INFO:tasks.rados.rados.0.smithi107.stdout:8324: delete oid 16 current snap is 0 2020-06-12T02:38:40.639 INFO:tasks.rados.rados.0.smithi107.stdout:8324: done (0 left) 2020-06-12T02:38:40.639 INFO:tasks.rados.rados.0.smithi107.stdout:8325: delete oid 15 current snap is 0 2020-06-12T02:38:40.642 INFO:tasks.rados.rados.0.smithi107.stdout:8325: done (0 left) 2020-06-12T02:38:40.642 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:40.642 INFO:tasks.rados.rados.0.smithi107.stdout:8326: seq_num 8325 ranges {0=229376} 2020-06-12T02:38:40.643 INFO:tasks.rados.rados.0.smithi107.stdout:8326: writing smithi10729407-8 from 0 to 229376 tid 1 2020-06-12T02:38:40.643 INFO:tasks.rados.rados.0.smithi107.stdout:8327: delete oid 11 current snap is 0 2020-06-12T02:38:40.646 INFO:tasks.rados.rados.0.smithi107.stdout:8327: done (1 left) 2020-06-12T02:38:40.646 INFO:tasks.rados.rados.0.smithi107.stdout:8328: delete oid 19 current snap is 0 2020-06-12T02:38:40.648 INFO:tasks.rados.rados.0.smithi107.stdout:8326: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:40.649 INFO:tasks.rados.rados.0.smithi107.stdout:8328: done (1 left) 2020-06-12T02:38:40.650 INFO:tasks.rados.rados.0.smithi107.stdout:8329: delete oid 19 current snap is 0 2020-06-12T02:38:40.652 INFO:tasks.rados.rados.0.smithi107.stdout:8326: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:40.655 INFO:tasks.rados.rados.0.smithi107.stdout:8329: done (1 left) 2020-06-12T02:38:40.655 INFO:tasks.rados.rados.0.smithi107.stdout:8330: delete oid 9 current snap is 0 2020-06-12T02:38:40.655 INFO:tasks.rados.rados.0.smithi107.stdout:8326: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:40.656 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1552 (ObjNum 8325 snap 0 seq_num 8325) dirty exists 2020-06-12T02:38:40.656 INFO:tasks.rados.rados.0.smithi107.stdout:8326: left oid 8 (ObjNum 8325 snap 0 seq_num 8325) 2020-06-12T02:38:40.657 INFO:tasks.rados.rados.0.smithi107.stdout:8326: done (1 left) 2020-06-12T02:38:40.657 INFO:tasks.rados.rados.0.smithi107.stdout:8330: done (0 left) 2020-06-12T02:38:40.658 INFO:tasks.rados.rados.0.smithi107.stdout:8331: delete oid 8 current snap is 0 2020-06-12T02:38:40.660 INFO:tasks.rados.rados.0.smithi107.stdout:8331: done (0 left) 2020-06-12T02:38:40.660 INFO:tasks.rados.rados.0.smithi107.stdout:8332: delete oid 8 current snap is 0 2020-06-12T02:38:40.666 INFO:tasks.rados.rados.0.smithi107.stdout:8332: done (0 left) 2020-06-12T02:38:40.666 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:40.666 INFO:tasks.rados.rados.0.smithi107.stdout:8333: seq_num 8332 ranges {0=352256} 2020-06-12T02:38:40.667 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:40.662+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:40.669 INFO:tasks.rados.rados.0.smithi107.stdout:8333: writing smithi10729407-4 from 0 to 352256 tid 1 2020-06-12T02:38:40.669 INFO:tasks.rados.rados.0.smithi107.stdout:8334: delete oid 11 current snap is 0 2020-06-12T02:38:40.671 INFO:tasks.rados.rados.0.smithi107.stdout:8334: done (1 left) 2020-06-12T02:38:40.671 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:40.672 INFO:tasks.rados.rados.0.smithi107.stdout:8335: seq_num 8334 ranges {0=385024} 2020-06-12T02:38:40.673 INFO:tasks.rados.rados.0.smithi107.stdout:8335: writing smithi10729407-15 from 0 to 385024 tid 1 2020-06-12T02:38:40.673 INFO:tasks.rados.rados.0.smithi107.stdout:8336: delete oid 13 current snap is 0 2020-06-12T02:38:40.674 INFO:tasks.rados.rados.0.smithi107.stdout:8333: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:40.676 INFO:tasks.rados.rados.0.smithi107.stdout:8336: done (2 left) 2020-06-12T02:38:40.676 INFO:tasks.rados.rados.0.smithi107.stdout:8337: delete oid 9 current snap is 0 2020-06-12T02:38:40.676 INFO:tasks.rados.rados.0.smithi107.stdout:8333: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:40.678 INFO:tasks.rados.rados.0.smithi107.stdout:8333: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:40.679 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1574 (ObjNum 8332 snap 0 seq_num 8332) dirty exists 2020-06-12T02:38:40.679 INFO:tasks.rados.rados.0.smithi107.stdout:8333: left oid 4 (ObjNum 8332 snap 0 seq_num 8332) 2020-06-12T02:38:40.680 INFO:tasks.rados.rados.0.smithi107.stdout:8333: done (2 left) 2020-06-12T02:38:40.680 INFO:tasks.rados.rados.0.smithi107.stdout:8337: done (1 left) 2020-06-12T02:38:40.680 INFO:tasks.rados.rados.0.smithi107.stdout:8338: delete oid 8 current snap is 0 2020-06-12T02:38:40.681 INFO:tasks.rados.rados.0.smithi107.stdout:8335: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:40.685 INFO:tasks.rados.rados.0.smithi107.stdout:8335: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:40.686 INFO:tasks.rados.rados.0.smithi107.stdout:8338: done (1 left) 2020-06-12T02:38:40.686 INFO:tasks.rados.rados.0.smithi107.stdout:8339: delete oid 11 current snap is 0 2020-06-12T02:38:40.687 INFO:tasks.rados.rados.0.smithi107.stdout:8335: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:40.687 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1556 (ObjNum 8334 snap 0 seq_num 8334) dirty exists 2020-06-12T02:38:40.687 INFO:tasks.rados.rados.0.smithi107.stdout:8335: left oid 15 (ObjNum 8334 snap 0 seq_num 8334) 2020-06-12T02:38:40.689 INFO:tasks.rados.rados.0.smithi107.stdout:8335: done (1 left) 2020-06-12T02:38:40.689 INFO:tasks.rados.rados.0.smithi107.stdout:8339: done (0 left) 2020-06-12T02:38:40.690 INFO:tasks.rados.rados.0.smithi107.stdout:8340: delete oid 12 current snap is 0 2020-06-12T02:38:40.691 INFO:tasks.rados.rados.0.smithi107.stdout:8340: done (0 left) 2020-06-12T02:38:40.691 INFO:tasks.rados.rados.0.smithi107.stdout:8341: delete oid 10 current snap is 0 2020-06-12T02:38:40.694 INFO:tasks.rados.rados.0.smithi107.stdout:8341: done (0 left) 2020-06-12T02:38:40.694 INFO:tasks.rados.rados.0.smithi107.stdout:8342: delete oid 19 current snap is 0 2020-06-12T02:38:40.699 INFO:tasks.rados.rados.0.smithi107.stdout:8342: done (0 left) 2020-06-12T02:38:40.699 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:40.699 INFO:tasks.rados.rados.0.smithi107.stdout:8343: seq_num 8342 ranges {352256=516096} 2020-06-12T02:38:40.703 INFO:tasks.rados.rados.0.smithi107.stdout:8343: writing smithi10729407-4 from 352256 to 868352 tid 1 2020-06-12T02:38:40.704 INFO:tasks.rados.rados.0.smithi107.stdout:8344: delete oid 8 current snap is 0 2020-06-12T02:38:40.713 INFO:tasks.rados.rados.0.smithi107.stdout:8344: done (1 left) 2020-06-12T02:38:40.714 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:38:40.714 INFO:tasks.rados.rados.0.smithi107.stdout:8345: seq_num 8344 ranges {0=548864} 2020-06-12T02:38:40.715 INFO:tasks.rados.rados.0.smithi107.stdout:8345: writing smithi10729407-10 from 0 to 548864 tid 1 2020-06-12T02:38:40.715 INFO:tasks.rados.rados.0.smithi107.stdout:8346: delete oid 2 current snap is 0 2020-06-12T02:38:40.715 INFO:tasks.rados.rados.0.smithi107.stdout:8343: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:40.716 INFO:tasks.rados.rados.0.smithi107.stdout:8343: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:40.716 INFO:tasks.rados.rados.0.smithi107.stdout:8343: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:40.716 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1576 (ObjNum 8342 snap 0 seq_num 8342) dirty exists 2020-06-12T02:38:40.716 INFO:tasks.rados.rados.0.smithi107.stdout:8343: left oid 4 (ObjNum 8342 snap 0 seq_num 8342) 2020-06-12T02:38:40.717 INFO:tasks.rados.rados.0.smithi107.stdout:8343: done (2 left) 2020-06-12T02:38:40.717 INFO:tasks.rados.rados.0.smithi107.stdout:8346: done (1 left) 2020-06-12T02:38:40.718 INFO:tasks.rados.rados.0.smithi107.stdout:8347: delete oid 4 current snap is 0 2020-06-12T02:38:40.722 INFO:tasks.rados.rados.0.smithi107.stdout:8345: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:40.722 INFO:tasks.rados.rados.0.smithi107.stdout:8347: done (1 left) 2020-06-12T02:38:40.722 INFO:tasks.rados.rados.0.smithi107.stdout:8348: delete oid 7 current snap is 0 2020-06-12T02:38:40.723 INFO:tasks.rados.rados.0.smithi107.stdout:8345: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:40.724 INFO:tasks.rados.rados.0.smithi107.stdout:8345: finishing write tid 4 to smithi10729407-10 2020-06-12T02:38:40.724 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 603 (ObjNum 8344 snap 0 seq_num 8344) dirty exists 2020-06-12T02:38:40.724 INFO:tasks.rados.rados.0.smithi107.stdout:8345: left oid 10 (ObjNum 8344 snap 0 seq_num 8344) 2020-06-12T02:38:40.725 INFO:tasks.rados.rados.0.smithi107.stdout:8345: done (1 left) 2020-06-12T02:38:40.725 INFO:tasks.rados.rados.0.smithi107.stdout:8348: done (0 left) 2020-06-12T02:38:40.725 INFO:tasks.rados.rados.0.smithi107.stdout:8349: delete oid 6 current snap is 0 2020-06-12T02:38:40.727 INFO:tasks.rados.rados.0.smithi107.stdout:8349: done (0 left) 2020-06-12T02:38:40.727 INFO:tasks.rados.rados.0.smithi107.stdout:8350: delete oid 14 current snap is 0 2020-06-12T02:38:40.730 INFO:tasks.rados.rados.0.smithi107.stdout:8350: done (0 left) 2020-06-12T02:38:40.730 INFO:tasks.rados.rados.0.smithi107.stdout:8351: delete oid 14 current snap is 0 2020-06-12T02:38:40.734 INFO:tasks.rados.rados.0.smithi107.stdout:8351: done (0 left) 2020-06-12T02:38:40.734 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:40.734 INFO:tasks.rados.rados.0.smithi107.stdout:8352: seq_num 8351 ranges {0=671744} 2020-06-12T02:38:40.737 INFO:tasks.rados.rados.0.smithi107.stdout:8352: writing smithi10729407-11 from 0 to 671744 tid 1 2020-06-12T02:38:40.737 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:38:40.737 INFO:tasks.rados.rados.0.smithi107.stdout:8353: seq_num 8352 ranges {0=688128} 2020-06-12T02:38:40.744 INFO:tasks.rados.rados.0.smithi107.stdout:8353: writing smithi10729407-5 from 0 to 688128 tid 1 2020-06-12T02:38:40.744 INFO:tasks.rados.rados.0.smithi107.stdout:8354: delete oid 17 current snap is 0 2020-06-12T02:38:40.745 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:40.746 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:40.746 INFO:tasks.rados.rados.0.smithi107.stdout:8352: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:40.747 INFO:tasks.rados.rados.0.smithi107.stdout:8352: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:40.747 INFO:tasks.rados.rados.0.smithi107.stdout:8354: done (2 left) 2020-06-12T02:38:40.747 INFO:tasks.rados.rados.0.smithi107.stdout:8355: delete oid 10 current snap is 0 2020-06-12T02:38:40.748 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:38:40.750 INFO:tasks.rados.rados.0.smithi107.stdout:8352: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:40.750 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1468 (ObjNum 8351 snap 0 seq_num 8351) dirty exists 2020-06-12T02:38:40.750 INFO:tasks.rados.rados.0.smithi107.stdout:8352: left oid 11 (ObjNum 8351 snap 0 seq_num 8351) 2020-06-12T02:38:40.750 INFO:tasks.rados.rados.0.smithi107.stdout:8353: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:40.751 INFO:tasks.rados.rados.0.smithi107.stdout:8352: done (2 left) 2020-06-12T02:38:40.751 INFO:tasks.rados.rados.0.smithi107.stdout:8355: done (1 left) 2020-06-12T02:38:40.751 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:40.752 INFO:tasks.rados.rados.0.smithi107.stdout:8356: seq_num 8355 ranges {0=737280} 2020-06-12T02:38:40.754 INFO:tasks.rados.rados.0.smithi107.stdout:8356: writing smithi10729407-14 from 0 to 737280 tid 1 2020-06-12T02:38:40.755 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:38:40.755 INFO:tasks.rados.rados.0.smithi107.stdout:8357: seq_num 8356 ranges {0=753664} 2020-06-12T02:38:40.756 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-06-12T02:38:40.758 INFO:tasks.rados.rados.0.smithi107.stdout:8357: writing smithi10729407-6 from 0 to 753664 tid 1 2020-06-12T02:38:40.759 INFO:tasks.rados.rados.0.smithi107.stdout:8358: delete oid 12 current snap is 0 2020-06-12T02:38:40.759 INFO:tasks.rados.rados.0.smithi107.stdout:8353: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:40.759 INFO:tasks.rados.rados.0.smithi107.stdout:8353: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:40.759 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 831 (ObjNum 8352 snap 0 seq_num 8352) dirty exists 2020-06-12T02:38:40.759 INFO:tasks.rados.rados.0.smithi107.stdout:8353: left oid 5 (ObjNum 8352 snap 0 seq_num 8352) 2020-06-12T02:38:40.762 INFO:tasks.rados.rados.0.smithi107.stdout:8353: done (3 left) 2020-06-12T02:38:40.762 INFO:tasks.rados.rados.0.smithi107.stdout:8358: done (2 left) 2020-06-12T02:38:40.762 INFO:tasks.rados.rados.0.smithi107.stdout:8359: delete oid 13 current snap is 0 2020-06-12T02:38:40.763 INFO:tasks.rados.rados.0.smithi107.stdout:8356: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:40.763 INFO:tasks.rados.rados.0.smithi107.stdout:8356: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:40.764 INFO:tasks.rados.rados.0.smithi107.stdout:8356: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:40.764 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1470 (ObjNum 8355 snap 0 seq_num 8355) dirty exists 2020-06-12T02:38:40.765 INFO:tasks.rados.rados.0.smithi107.stdout:8356: left oid 14 (ObjNum 8355 snap 0 seq_num 8355) 2020-06-12T02:38:40.765 INFO:tasks.rados.rados.0.smithi107.stdout:8356: done (2 left) 2020-06-12T02:38:40.765 INFO:tasks.rados.rados.0.smithi107.stdout:8359: done (1 left) 2020-06-12T02:38:40.765 INFO:tasks.rados.rados.0.smithi107.stdout:8360: delete oid 17 current snap is 0 2020-06-12T02:38:40.769 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:40.766+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:40.769 INFO:tasks.rados.rados.0.smithi107.stdout:8360: done (1 left) 2020-06-12T02:38:40.770 INFO:tasks.rados.rados.0.smithi107.stdout:8361: delete oid 5 current snap is 0 2020-06-12T02:38:40.770 INFO:tasks.rados.rados.0.smithi107.stdout:8357: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:40.770 INFO:tasks.rados.rados.0.smithi107.stdout:8357: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:40.772 INFO:tasks.rados.rados.0.smithi107.stdout:8361: done (1 left) 2020-06-12T02:38:40.772 INFO:tasks.rados.rados.0.smithi107.stdout:8362: delete oid 2 current snap is 0 2020-06-12T02:38:40.772 INFO:tasks.rados.rados.0.smithi107.stdout:8357: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:40.773 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1514 (ObjNum 8356 snap 0 seq_num 8356) dirty exists 2020-06-12T02:38:40.773 INFO:tasks.rados.rados.0.smithi107.stdout:8357: left oid 6 (ObjNum 8356 snap 0 seq_num 8356) 2020-06-12T02:38:40.775 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:38:40.776 INFO:tasks.rados.rados.0.smithi107.stdout:8357: done (1 left) 2020-06-12T02:38:40.776 INFO:tasks.rados.rados.0.smithi107.stdout:8362: done (0 left) 2020-06-12T02:38:40.776 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:40.777 INFO:tasks.rados.rados.0.smithi107.stdout:8363: seq_num 8362 ranges {0=450560,450560=401408} 2020-06-12T02:38:40.779 INFO:tasks.rados.rados.0.smithi107.stdout:8363: writing smithi10729407-17 from 0 to 450560 tid 1 2020-06-12T02:38:40.781 INFO:tasks.rados.rados.0.smithi107.stdout:8363: writing smithi10729407-17 from 450560 to 851968 tid 2 2020-06-12T02:38:40.781 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:38:40.781 INFO:tasks.rados.rados.0.smithi107.stdout:8364: seq_num 8363 ranges {0=466944,466944=401408} 2020-06-12T02:38:40.783 INFO:tasks.rados.rados.0.smithi107.stdout:8364: writing smithi10729407-10 from 0 to 466944 tid 1 2020-06-12T02:38:40.786 INFO:tasks.rados.rados.0.smithi107.stdout:8364: writing smithi10729407-10 from 466944 to 868352 tid 2 2020-06-12T02:38:40.787 INFO:tasks.rados.rados.0.smithi107.stdout:8365: delete oid 7 current snap is 0 2020-06-12T02:38:40.789 INFO:tasks.rados.rados.0.smithi107.stdout:8364: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:40.789 INFO:tasks.rados.rados.0.smithi107.stdout:8363: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:40.790 INFO:tasks.rados.rados.0.smithi107.stdout:8363: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:40.790 INFO:tasks.rados.rados.0.smithi107.stdout:8363: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:40.790 INFO:tasks.rados.rados.0.smithi107.stdout:8365: done (2 left) 2020-06-12T02:38:40.791 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:40.791 INFO:tasks.rados.rados.0.smithi107.stdout:8366: seq_num 8365 ranges {0=499712,499712=401408} 2020-06-12T02:38:40.795 INFO:tasks.rados.rados.0.smithi107.stdout:8366: writing smithi10729407-8 from 0 to 499712 tid 1 2020-06-12T02:38:40.796 INFO:tasks.rados.rados.0.smithi107.stdout:8366: writing smithi10729407-8 from 499712 to 901120 tid 2 2020-06-12T02:38:40.796 INFO:tasks.rados.rados.0.smithi107.stdout:8367: delete oid 2 current snap is 0 2020-06-12T02:38:40.796 INFO:tasks.rados.rados.0.smithi107.stdout:8363: finishing write tid 5 to smithi10729407-17 2020-06-12T02:38:40.796 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1582 (ObjNum 8362 snap 0 seq_num 8362) dirty exists 2020-06-12T02:38:40.796 INFO:tasks.rados.rados.0.smithi107.stdout:8363: left oid 17 (ObjNum 8362 snap 0 seq_num 8362) 2020-06-12T02:38:40.797 INFO:tasks.rados.rados.0.smithi107.stdout:8364: finishing write tid 2 to smithi10729407-10 2020-06-12T02:38:40.798 INFO:tasks.rados.rados.0.smithi107.stdout:8364: finishing write tid 4 to smithi10729407-10 2020-06-12T02:38:40.798 INFO:tasks.rados.rados.0.smithi107.stdout:8364: finishing write tid 5 to smithi10729407-10 2020-06-12T02:38:40.798 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 607 (ObjNum 8363 snap 0 seq_num 8363) dirty exists 2020-06-12T02:38:40.798 INFO:tasks.rados.rados.0.smithi107.stdout:8364: left oid 10 (ObjNum 8363 snap 0 seq_num 8363) 2020-06-12T02:38:40.799 INFO:tasks.rados.rados.0.smithi107.stdout:8363: done (3 left) 2020-06-12T02:38:40.800 INFO:tasks.rados.rados.0.smithi107.stdout:8364: done (2 left) 2020-06-12T02:38:40.800 INFO:tasks.rados.rados.0.smithi107.stdout:8367: done (1 left) 2020-06-12T02:38:40.800 INFO:tasks.rados.rados.0.smithi107.stdout:8368: delete oid 4 current snap is 0 2020-06-12T02:38:40.800 INFO:tasks.rados.rados.0.smithi107.stdout:8366: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:40.802 INFO:tasks.rados.rados.0.smithi107.stdout:8368: done (1 left) 2020-06-12T02:38:40.802 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:40.803 INFO:tasks.rados.rados.0.smithi107.stdout:8369: seq_num 8368 ranges {0=557056,557056=401408} 2020-06-12T02:38:40.803 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:38:40.806 INFO:tasks.rados.rados.0.smithi107.stdout:8369: writing smithi10729407-18 from 0 to 557056 tid 1 2020-06-12T02:38:40.808 INFO:tasks.rados.rados.0.smithi107.stdout:8369: writing smithi10729407-18 from 557056 to 958464 tid 2 2020-06-12T02:38:40.808 INFO:tasks.rados.rados.0.smithi107.stdout:8370: delete oid 12 current snap is 0 2020-06-12T02:38:40.808 INFO:tasks.rados.rados.0.smithi107.stdout:8366: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:40.808 INFO:tasks.rados.rados.0.smithi107.stdout:8366: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:40.809 INFO:tasks.rados.rados.0.smithi107.stdout:8366: finishing write tid 5 to smithi10729407-8 2020-06-12T02:38:40.810 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1561 (ObjNum 8365 snap 0 seq_num 8365) dirty exists 2020-06-12T02:38:40.810 INFO:tasks.rados.rados.0.smithi107.stdout:8366: left oid 8 (ObjNum 8365 snap 0 seq_num 8365) 2020-06-12T02:38:40.814 INFO:tasks.rados.rados.0.smithi107.stdout:8366: done (2 left) 2020-06-12T02:38:40.815 INFO:tasks.rados.rados.0.smithi107.stdout:8370: done (1 left) 2020-06-12T02:38:40.815 INFO:tasks.rados.rados.0.smithi107.stdout:8371: delete oid 6 current snap is 0 2020-06-12T02:38:40.815 INFO:tasks.rados.rados.0.smithi107.stdout:8369: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:40.815 INFO:tasks.rados.rados.0.smithi107.stdout:8369: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:40.816 INFO:tasks.rados.rados.0.smithi107.stdout:8369: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:40.816 INFO:tasks.rados.rados.0.smithi107.stdout:8369: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:40.817 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1547 (ObjNum 8368 snap 0 seq_num 8368) dirty exists 2020-06-12T02:38:40.817 INFO:tasks.rados.rados.0.smithi107.stdout:8369: left oid 18 (ObjNum 8368 snap 0 seq_num 8368) 2020-06-12T02:38:40.817 INFO:tasks.rados.rados.0.smithi107.stdout:8369: done (1 left) 2020-06-12T02:38:40.818 INFO:tasks.rados.rados.0.smithi107.stdout:8371: done (0 left) 2020-06-12T02:38:40.818 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:40.818 INFO:tasks.rados.rados.0.smithi107.stdout:8372: seq_num 8371 ranges {958464=606208,1564672=401408} 2020-06-12T02:38:40.826 INFO:tasks.rados.rados.0.smithi107.stdout:8372: writing smithi10729407-18 from 958464 to 1564672 tid 1 2020-06-12T02:38:40.826 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:38:40.830 INFO:tasks.rados.rados.0.smithi107.stdout:8372: writing smithi10729407-18 from 1564672 to 1966080 tid 2 2020-06-12T02:38:40.830 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:40.830 INFO:tasks.rados.rados.0.smithi107.stdout:8373: seq_num 8372 ranges {0=622592,622592=401408} 2020-06-12T02:38:40.833 INFO:tasks.rados.rados.0.smithi107.stdout:8373: writing smithi10729407-9 from 0 to 622592 tid 1 2020-06-12T02:38:40.835 INFO:tasks.rados.rados.0.smithi107.stdout:8373: writing smithi10729407-9 from 622592 to 1024000 tid 2 2020-06-12T02:38:40.835 INFO:tasks.rados.rados.0.smithi107.stdout:8374: delete oid 14 current snap is 0 2020-06-12T02:38:40.837 INFO:tasks.rados.rados.0.smithi107.stdout:8372: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:40.840 INFO:tasks.rados.rados.0.smithi107.stdout:8374: done (2 left) 2020-06-12T02:38:40.840 INFO:tasks.rados.rados.0.smithi107.stdout:8375: delete oid 11 current snap is 0 2020-06-12T02:38:40.845 INFO:tasks.rados.rados.0.smithi107.stdout:8375: done (2 left) 2020-06-12T02:38:40.845 INFO:tasks.rados.rados.0.smithi107.stdout:8376: delete oid 11 current snap is 0 2020-06-12T02:38:40.848 INFO:tasks.rados.rados.0.smithi107.stdout:8373: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:40.848 INFO:tasks.rados.rados.0.smithi107.stdout:8372: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:40.848 INFO:tasks.rados.rados.0.smithi107.stdout:8372: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:40.849 INFO:tasks.rados.rados.0.smithi107.stdout:8372: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:40.849 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1551 (ObjNum 8371 snap 0 seq_num 8371) dirty exists 2020-06-12T02:38:40.850 INFO:tasks.rados.rados.0.smithi107.stdout:8372: left oid 18 (ObjNum 8371 snap 0 seq_num 8371) 2020-06-12T02:38:40.850 INFO:tasks.rados.rados.0.smithi107.stdout:8373: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:40.851 INFO:tasks.rados.rados.0.smithi107.stdout:8373: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:40.851 INFO:tasks.rados.rados.0.smithi107.stdout:8372: done (2 left) 2020-06-12T02:38:40.852 INFO:tasks.rados.rados.0.smithi107.stdout:8376: done (1 left) 2020-06-12T02:38:40.852 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:38:40.852 INFO:tasks.rados.rados.0.smithi107.stdout:8377: seq_num 8376 ranges {0=688128,688128=401408} 2020-06-12T02:38:40.855 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:38:40.856 INFO:tasks.rados.rados.0.smithi107.stdout:8377: writing smithi10729407-6 from 0 to 688128 tid 1 2020-06-12T02:38:40.858 INFO:tasks.rados.rados.0.smithi107.stdout:8377: writing smithi10729407-6 from 688128 to 1089536 tid 2 2020-06-12T02:38:40.858 INFO:tasks.rados.rados.0.smithi107.stdout:8378: delete oid 11 current snap is 0 2020-06-12T02:38:40.858 INFO:tasks.rados.rados.0.smithi107.stdout:8373: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:40.859 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1553 (ObjNum 8372 snap 0 seq_num 8372) dirty exists 2020-06-12T02:38:40.859 INFO:tasks.rados.rados.0.smithi107.stdout:8373: left oid 9 (ObjNum 8372 snap 0 seq_num 8372) 2020-06-12T02:38:40.862 INFO:tasks.rados.rados.0.smithi107.stdout:8373: done (2 left) 2020-06-12T02:38:40.862 INFO:tasks.rados.rados.0.smithi107.stdout:8378: done (1 left) 2020-06-12T02:38:40.863 INFO:tasks.rados.rados.0.smithi107.stdout:8379: delete oid 2 current snap is 0 2020-06-12T02:38:40.864 INFO:tasks.rados.rados.0.smithi107.stdout:8379: done (1 left) 2020-06-12T02:38:40.864 INFO:tasks.rados.rados.0.smithi107.stdout:8380: delete oid 1 current snap is 0 2020-06-12T02:38:40.865 INFO:tasks.rados.rados.0.smithi107.stdout:8377: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:40.867 INFO:tasks.rados.rados.0.smithi107.stdout:8377: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:40.868 INFO:tasks.rados.rados.0.smithi107.stdout:8377: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:40.870 INFO:tasks.rados.rados.0.smithi107.stdout:8377: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:40.870 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1518 (ObjNum 8376 snap 0 seq_num 8376) dirty exists 2020-06-12T02:38:40.870 INFO:tasks.rados.rados.0.smithi107.stdout:8377: left oid 6 (ObjNum 8376 snap 0 seq_num 8376) 2020-06-12T02:38:40.871 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:40.867+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:40.871 INFO:tasks.rados.rados.0.smithi107.stdout:8377: done (1 left) 2020-06-12T02:38:40.872 INFO:tasks.rados.rados.0.smithi107.stdout:8380: done (0 left) 2020-06-12T02:38:40.872 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:40.872 INFO:tasks.rados.rados.0.smithi107.stdout:8381: seq_num 8380 ranges {0=753664,753664=401408} 2020-06-12T02:38:40.876 INFO:tasks.rados.rados.0.smithi107.stdout:8381: writing smithi10729407-14 from 0 to 753664 tid 1 2020-06-12T02:38:40.877 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:38:40.880 INFO:tasks.rados.rados.0.smithi107.stdout:8381: writing smithi10729407-14 from 753664 to 1155072 tid 2 2020-06-12T02:38:40.881 INFO:tasks.rados.rados.0.smithi107.stdout:8382: delete oid 6 current snap is 0 2020-06-12T02:38:40.884 INFO:tasks.rados.rados.0.smithi107.stdout:8381: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:40.885 INFO:tasks.rados.rados.0.smithi107.stdout:8382: done (1 left) 2020-06-12T02:38:40.885 INFO:tasks.rados.rados.0.smithi107.stdout:8383: delete oid 19 current snap is 0 2020-06-12T02:38:40.889 INFO:tasks.rados.rados.0.smithi107.stdout:8383: done (1 left) 2020-06-12T02:38:40.890 INFO:tasks.rados.rados.0.smithi107.stdout:8384: delete oid 12 current snap is 0 2020-06-12T02:38:40.890 INFO:tasks.rados.rados.0.smithi107.stdout:8381: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:40.891 INFO:tasks.rados.rados.0.smithi107.stdout:8381: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:40.892 INFO:tasks.rados.rados.0.smithi107.stdout:8381: finishing write tid 5 to smithi10729407-14 2020-06-12T02:38:40.892 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1477 (ObjNum 8380 snap 0 seq_num 8380) dirty exists 2020-06-12T02:38:40.892 INFO:tasks.rados.rados.0.smithi107.stdout:8381: left oid 14 (ObjNum 8380 snap 0 seq_num 8380) 2020-06-12T02:38:40.892 INFO:tasks.rados.rados.0.smithi107.stdout:8381: done (1 left) 2020-06-12T02:38:40.893 INFO:tasks.rados.rados.0.smithi107.stdout:8384: done (0 left) 2020-06-12T02:38:40.893 INFO:tasks.rados.rados.0.smithi107.stdout:8385: delete oid 5 current snap is 0 2020-06-12T02:38:40.895 INFO:tasks.rados.rados.0.smithi107.stdout:8385: done (0 left) 2020-06-12T02:38:40.895 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:38:40.895 INFO:tasks.rados.rados.0.smithi107.stdout:8386: seq_num 8385 ranges {0=434176,434176=475136,909312=327680} 2020-06-12T02:38:40.896 INFO:tasks.rados.rados.0.smithi107.stdout:8386: writing smithi10729407-16 from 0 to 434176 tid 1 2020-06-12T02:38:40.899 INFO:tasks.rados.rados.0.smithi107.stdout:8386: writing smithi10729407-16 from 434176 to 909312 tid 2 2020-06-12T02:38:40.902 INFO:tasks.rados.rados.0.smithi107.stdout:8386: writing smithi10729407-16 from 909312 to 1236992 tid 3 2020-06-12T02:38:40.902 INFO:tasks.rados.rados.0.smithi107.stdout:8387: delete oid 4 current snap is 0 2020-06-12T02:38:40.903 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:38:40.905 INFO:tasks.rados.rados.0.smithi107.stdout:8387: done (1 left) 2020-06-12T02:38:40.905 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:38:40.906 INFO:tasks.rados.rados.0.smithi107.stdout:8388: seq_num 8387 ranges {0=475136,475136=696320,1171456=106496} 2020-06-12T02:38:40.908 INFO:tasks.rados.rados.0.smithi107.stdout:8388: writing smithi10729407-7 from 0 to 475136 tid 1 2020-06-12T02:38:40.913 INFO:tasks.rados.rados.0.smithi107.stdout:8388: writing smithi10729407-7 from 475136 to 1171456 tid 2 2020-06-12T02:38:40.913 INFO:tasks.rados.rados.0.smithi107.stdout:8388: writing smithi10729407-7 from 1171456 to 1277952 tid 3 2020-06-12T02:38:40.914 INFO:tasks.rados.rados.0.smithi107.stdout:8389: delete oid 5 current snap is 0 2020-06-12T02:38:40.914 INFO:tasks.rados.rados.0.smithi107.stdout:8386: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:40.914 INFO:tasks.rados.rados.0.smithi107.stdout:8386: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:40.915 INFO:tasks.rados.rados.0.smithi107.stdout:8386: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:40.915 INFO:tasks.rados.rados.0.smithi107.stdout:8386: finishing write tid 5 to smithi10729407-16 2020-06-12T02:38:40.915 INFO:tasks.rados.rados.0.smithi107.stdout:8388: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:40.916 INFO:tasks.rados.rados.0.smithi107.stdout:8386: finishing write tid 6 to smithi10729407-16 2020-06-12T02:38:40.916 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1522 (ObjNum 8385 snap 0 seq_num 8385) dirty exists 2020-06-12T02:38:40.916 INFO:tasks.rados.rados.0.smithi107.stdout:8386: left oid 16 (ObjNum 8385 snap 0 seq_num 8385) 2020-06-12T02:38:40.917 INFO:tasks.rados.rados.0.smithi107.stdout:8386: done (2 left) 2020-06-12T02:38:40.917 INFO:tasks.rados.rados.0.smithi107.stdout:8389: done (1 left) 2020-06-12T02:38:40.917 INFO:tasks.rados.rados.0.smithi107.stdout:8390: delete oid 5 current snap is 0 2020-06-12T02:38:40.919 INFO:tasks.rados.rados.0.smithi107.stdout:8390: done (1 left) 2020-06-12T02:38:40.920 INFO:tasks.rados.rados.0.smithi107.stdout:8391: delete oid 17 current snap is 0 2020-06-12T02:38:40.922 INFO:tasks.rados.rados.0.smithi107.stdout:8391: done (1 left) 2020-06-12T02:38:40.923 INFO:tasks.rados.rados.0.smithi107.stdout:8392: delete oid 14 current snap is 0 2020-06-12T02:38:40.924 INFO:tasks.rados.rados.0.smithi107.stdout:8388: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:40.924 INFO:tasks.rados.rados.0.smithi107.stdout:8388: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:40.926 INFO:tasks.rados.rados.0.smithi107.stdout:8388: finishing write tid 5 to smithi10729407-7 2020-06-12T02:38:40.926 INFO:tasks.rados.rados.0.smithi107.stdout:8392: done (1 left) 2020-06-12T02:38:40.926 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:40.926 INFO:tasks.rados.rados.0.smithi107.stdout:8393: seq_num 8392 ranges {0=557056,557056=516096,1073152=286720} 2020-06-12T02:38:40.929 INFO:tasks.rados.rados.0.smithi107.stdout:8393: writing smithi10729407-3 from 0 to 557056 tid 1 2020-06-12T02:38:40.931 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2020-06-12T02:38:40.933 INFO:tasks.rados.rados.0.smithi107.stdout:8393: writing smithi10729407-3 from 557056 to 1073152 tid 2 2020-06-12T02:38:40.934 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-06-12T02:38:40.934 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-06-12T02:38:40.935 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-06-12T02:38:40.935 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-06-12T02:38:40.935 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-06-12T02:38:40.935 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-06-12T02:38:40.935 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-06-12T02:38:40.936 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-06-12T02:38:40.936 INFO:tasks.rados.rados.0.smithi107.stdout:8393: writing smithi10729407-3 from 1073152 to 1359872 tid 3 2020-06-12T02:38:40.936 INFO:tasks.rados.rados.0.smithi107.stdout:8394: delete oid 17 current snap is 0 2020-06-12T02:38:40.937 INFO:tasks.rados.rados.0.smithi107.stdout:8388: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:40.937 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1526 (ObjNum 8387 snap 0 seq_num 8387) dirty exists 2020-06-12T02:38:40.937 INFO:tasks.rados.rados.0.smithi107.stdout:8388: left oid 7 (ObjNum 8387 snap 0 seq_num 8387) 2020-06-12T02:38:40.938 INFO:tasks.rados.rados.0.smithi107.stdout:8393: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:40.939 INFO:tasks.rados.rados.0.smithi107.stdout:8388: done (2 left) 2020-06-12T02:38:40.940 INFO:tasks.rados.rados.0.smithi107.stdout:8394: done (1 left) 2020-06-12T02:38:40.940 INFO:tasks.rados.rados.0.smithi107.stdout:8395: delete oid 8 current snap is 0 2020-06-12T02:38:40.943 INFO:tasks.rados.rados.0.smithi107.stdout:8395: done (1 left) 2020-06-12T02:38:40.943 INFO:tasks.rados.rados.0.smithi107.stdout:8396: delete oid 15 current snap is 0 2020-06-12T02:38:40.947 INFO:tasks.rados.rados.0.smithi107.stdout:8393: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:40.948 INFO:tasks.rados.rados.0.smithi107.stdout:8396: done (1 left) 2020-06-12T02:38:40.948 INFO:tasks.rados.rados.0.smithi107.stdout:8397: delete oid 8 current snap is 0 2020-06-12T02:38:40.949 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:40.949 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:38:40.950 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2020-06-12T02:38:40.950 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:40.950 INFO:tasks.rados.rados.0.smithi107.stdout:8393: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:40.950 INFO:tasks.rados.rados.0.smithi107.stdout:8393: finishing write tid 5 to smithi10729407-3 2020-06-12T02:38:40.951 INFO:tasks.rados.rados.0.smithi107.stdout:8393: finishing write tid 6 to smithi10729407-3 2020-06-12T02:38:40.951 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1526 (ObjNum 8392 snap 0 seq_num 8392) dirty exists 2020-06-12T02:38:40.951 INFO:tasks.rados.rados.0.smithi107.stdout:8393: left oid 3 (ObjNum 8392 snap 0 seq_num 8392) 2020-06-12T02:38:40.952 INFO:tasks.rados.rados.0.smithi107.stdout:8393: done (1 left) 2020-06-12T02:38:40.952 INFO:tasks.rados.rados.0.smithi107.stdout:8397: done (0 left) 2020-06-12T02:38:40.952 INFO:tasks.rados.rados.0.smithi107.stdout:8398: delete oid 9 current snap is 0 2020-06-12T02:38:40.955 INFO:tasks.rados.rados.0.smithi107.stdout:8398: done (0 left) 2020-06-12T02:38:40.955 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:38:40.955 INFO:tasks.rados.rados.0.smithi107.stdout:8399: seq_num 8398 ranges {0=655360,655360=671744,1327104=131072} 2020-06-12T02:38:40.958 INFO:tasks.rados.rados.0.smithi107.stdout:8399: writing smithi10729407-2 from 0 to 655360 tid 1 2020-06-12T02:38:40.959 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-06-12T02:38:40.962 INFO:tasks.rados.rados.0.smithi107.stdout:8399: writing smithi10729407-2 from 655360 to 1327104 tid 2 2020-06-12T02:38:40.963 INFO:tasks.rados.rados.0.smithi107.stdout:8399: writing smithi10729407-2 from 1327104 to 1458176 tid 3 2020-06-12T02:38:40.964 INFO:tasks.rados.rados.0.smithi107.stdout:8400: delete oid 9 current snap is 0 2020-06-12T02:38:40.964 INFO:tasks.rados.rados.0.smithi107.stdout:8399: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:40.968 INFO:tasks.rados.rados.0.smithi107.stdout:8400: done (1 left) 2020-06-12T02:38:40.968 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:40.968 INFO:tasks.rados.rados.0.smithi107.stdout:8401: seq_num 8400 ranges {0=688128,688128=442368,1130496=360448} 2020-06-12T02:38:40.972 INFO:tasks.rados.rados.0.smithi107.stdout:8401: writing smithi10729407-19 from 0 to 688128 tid 1 2020-06-12T02:38:40.975 INFO:tasks.rados.rados.0.smithi107.stdout:8401: writing smithi10729407-19 from 688128 to 1130496 tid 2 2020-06-12T02:38:40.977 INFO:tasks.rados.rados.0.smithi107.stdout:8401: writing smithi10729407-19 from 1130496 to 1490944 tid 3 2020-06-12T02:38:40.977 INFO:tasks.rados.rados.0.smithi107.stdout:8402: delete oid 13 current snap is 0 2020-06-12T02:38:40.978 INFO:tasks.rados.rados.0.smithi107.stdout:8399: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:40.978 INFO:tasks.rados.rados.0.smithi107.stdout:8399: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:40.978 INFO:tasks.rados.rados.0.smithi107.stdout:8399: finishing write tid 5 to smithi10729407-2 2020-06-12T02:38:40.978 INFO:tasks.rados.rados.0.smithi107.stdout:8399: finishing write tid 6 to smithi10729407-2 2020-06-12T02:38:40.978 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 785 (ObjNum 8398 snap 0 seq_num 8398) dirty exists 2020-06-12T02:38:40.979 INFO:tasks.rados.rados.0.smithi107.stdout:8399: left oid 2 (ObjNum 8398 snap 0 seq_num 8398) 2020-06-12T02:38:40.981 INFO:tasks.rados.rados.0.smithi107.stdout:8399: done (2 left) 2020-06-12T02:38:40.981 INFO:tasks.rados.rados.0.smithi107.stdout:8402: done (1 left) 2020-06-12T02:38:40.981 INFO:tasks.rados.rados.0.smithi107.stdout:8403: delete oid 1 current snap is 0 2020-06-12T02:38:40.984 INFO:tasks.rados.rados.0.smithi107.stdout:8401: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:40.984 INFO:tasks.rados.rados.0.smithi107.stdout:8403: done (1 left) 2020-06-12T02:38:40.984 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:40.984 INFO:tasks.rados.rados.0.smithi107.stdout:8404: seq_num 8403 ranges {0=737280,737280=540672,1277952=262144} 2020-06-12T02:38:40.988 INFO:tasks.rados.rados.0.smithi107.stdout:8404: writing smithi10729407-4 from 0 to 737280 tid 1 2020-06-12T02:38:40.991 INFO:tasks.rados.rados.0.smithi107.stdout:8404: writing smithi10729407-4 from 737280 to 1277952 tid 2 2020-06-12T02:38:41.000 INFO:tasks.rados.rados.0.smithi107.stdout:8404: writing smithi10729407-4 from 1277952 to 1540096 tid 3 2020-06-12T02:38:41.000 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:41.000 INFO:tasks.rados.rados.0.smithi107.stdout:8405: seq_num 8404 ranges {1966080=753664,2719744=425984,3145728=376832} 2020-06-12T02:38:41.001 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:40.997+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:41.006 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2020-06-12T02:38:41.007 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2020-06-12T02:38:41.007 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2020-06-12T02:38:41.007 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json 2020-06-12T02:38:41.009 INFO:tasks.rados.rados.0.smithi107.stdout:8405: writing smithi10729407-18 from 1966080 to 2719744 tid 1 2020-06-12T02:38:41.012 INFO:tasks.rados.rados.0.smithi107.stdout:8405: writing smithi10729407-18 from 2719744 to 3145728 tid 2 2020-06-12T02:38:41.015 INFO:tasks.rados.rados.0.smithi107.stdout:8405: writing smithi10729407-18 from 3145728 to 3522560 tid 3 2020-06-12T02:38:41.015 INFO:tasks.rados.rados.0.smithi107.stdout:8406: delete oid 8 current snap is 0 2020-06-12T02:38:41.016 INFO:tasks.rados.rados.0.smithi107.stdout:8401: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:41.016 INFO:tasks.rados.rados.0.smithi107.stdout:8401: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:41.016 INFO:tasks.rados.rados.0.smithi107.stdout:8401: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:41.016 INFO:tasks.rados.rados.0.smithi107.stdout:8401: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:41.016 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1544 (ObjNum 8400 snap 0 seq_num 8400) dirty exists 2020-06-12T02:38:41.016 INFO:tasks.rados.rados.0.smithi107.stdout:8401: left oid 19 (ObjNum 8400 snap 0 seq_num 8400) 2020-06-12T02:38:41.017 INFO:tasks.rados.rados.0.smithi107.stdout:8404: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:41.017 INFO:tasks.rados.rados.0.smithi107.stdout:8404: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:41.017 INFO:tasks.rados.rados.0.smithi107.stdout:8404: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:41.017 INFO:tasks.rados.rados.0.smithi107.stdout:8404: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:41.017 INFO:tasks.rados.rados.0.smithi107.stdout:8404: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:41.018 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1590 (ObjNum 8403 snap 0 seq_num 8403) dirty exists 2020-06-12T02:38:41.018 INFO:tasks.rados.rados.0.smithi107.stdout:8404: left oid 4 (ObjNum 8403 snap 0 seq_num 8403) 2020-06-12T02:38:41.018 INFO:tasks.rados.rados.0.smithi107.stdout:8401: done (3 left) 2020-06-12T02:38:41.018 INFO:tasks.rados.rados.0.smithi107.stdout:8404: done (2 left) 2020-06-12T02:38:41.019 INFO:tasks.rados.rados.0.smithi107.stdout:8406: done (1 left) 2020-06-12T02:38:41.019 INFO:tasks.rados.rados.0.smithi107.stdout:8407: delete oid 8 current snap is 0 2020-06-12T02:38:41.022 INFO:tasks.rados.rados.0.smithi107.stdout:8407: done (1 left) 2020-06-12T02:38:41.022 INFO:tasks.rados.rados.0.smithi107.stdout:8408: delete oid 12 current snap is 0 2020-06-12T02:38:41.023 INFO:tasks.rados.rados.0.smithi107.stdout:8405: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:41.023 INFO:tasks.rados.rados.0.smithi107.stdout:8405: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:41.024 INFO:tasks.rados.rados.0.smithi107.stdout:8405: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:41.025 INFO:tasks.rados.rados.0.smithi107.stdout:8405: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:41.025 INFO:tasks.rados.rados.0.smithi107.stdout:8408: done (1 left) 2020-06-12T02:38:41.025 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:41.025 INFO:tasks.rados.rados.0.smithi107.stdout:8409: seq_num 8408 ranges {0=425984,425984=761856,1187840=434176,1622016=8192} 2020-06-12T02:38:41.028 INFO:tasks.rados.rados.0.smithi107.stdout:8409: writing smithi10729407-5 from 0 to 425984 tid 1 2020-06-12T02:38:41.032 INFO:tasks.rados.rados.0.smithi107.stdout:8409: writing smithi10729407-5 from 425984 to 1187840 tid 2 2020-06-12T02:38:41.035 INFO:tasks.rados.rados.0.smithi107.stdout:8409: writing smithi10729407-5 from 1187840 to 1622016 tid 3 2020-06-12T02:38:41.036 INFO:tasks.rados.rados.0.smithi107.stdout:8409: writing smithi10729407-5 from 1622016 to 1630208 tid 4 2020-06-12T02:38:41.036 INFO:tasks.rados.rados.0.smithi107.stdout:8410: delete oid 14 current snap is 0 2020-06-12T02:38:41.036 INFO:tasks.rados.rados.0.smithi107.stdout:8405: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:41.036 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1559 (ObjNum 8404 snap 0 seq_num 8404) dirty exists 2020-06-12T02:38:41.036 INFO:tasks.rados.rados.0.smithi107.stdout:8405: left oid 18 (ObjNum 8404 snap 0 seq_num 8404) 2020-06-12T02:38:41.037 INFO:tasks.rados.rados.0.smithi107.stdout:8409: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:41.038 INFO:tasks.rados.rados.0.smithi107.stdout:8405: done (2 left) 2020-06-12T02:38:41.038 INFO:tasks.rados.rados.0.smithi107.stdout:8410: done (1 left) 2020-06-12T02:38:41.039 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:41.039 INFO:tasks.rados.rados.0.smithi107.stdout:8411: seq_num 8410 ranges {0=458752,458752=581632,1040384=622592} 2020-06-12T02:38:41.041 INFO:tasks.rados.rados.0.smithi107.stdout:8411: writing smithi10729407-12 from 0 to 458752 tid 1 2020-06-12T02:38:41.046 INFO:tasks.rados.rados.0.smithi107.stdout:8411: writing smithi10729407-12 from 458752 to 1040384 tid 2 2020-06-12T02:38:41.048 INFO:tasks.rados.rados.0.smithi107.stdout:8411: writing smithi10729407-12 from 1040384 to 1662976 tid 3 2020-06-12T02:38:41.049 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:41.049 INFO:tasks.rados.rados.0.smithi107.stdout:8412: seq_num 8411 ranges {0=475136,475136=466944,942080=606208,1548288=131072} 2020-06-12T02:38:41.051 INFO:tasks.rados.rados.0.smithi107.stdout:8412: writing smithi10729407-1 from 0 to 475136 tid 1 2020-06-12T02:38:41.054 INFO:tasks.rados.rados.0.smithi107.stdout:8412: writing smithi10729407-1 from 475136 to 942080 tid 2 2020-06-12T02:38:41.057 INFO:tasks.rados.rados.0.smithi107.stdout:8412: writing smithi10729407-1 from 942080 to 1548288 tid 3 2020-06-12T02:38:41.058 INFO:tasks.rados.rados.0.smithi107.stdout:8412: writing smithi10729407-1 from 1548288 to 1679360 tid 4 2020-06-12T02:38:41.059 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:41.059 INFO:tasks.rados.rados.0.smithi107.stdout:8413: seq_num 8412 ranges {3522560=491520,4014080=753664,4767744=417792,5185536=32768} 2020-06-12T02:38:41.071 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:41.068+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:41.080 INFO:tasks.rados.rados.0.smithi107.stdout:8413: writing smithi10729407-18 from 3522560 to 4014080 tid 1 2020-06-12T02:38:41.086 INFO:tasks.rados.rados.0.smithi107.stdout:8413: writing smithi10729407-18 from 4014080 to 4767744 tid 2 2020-06-12T02:38:41.088 INFO:tasks.rados.rados.0.smithi107.stdout:8413: writing smithi10729407-18 from 4767744 to 5185536 tid 3 2020-06-12T02:38:41.088 INFO:tasks.rados.rados.0.smithi107.stdout:8413: writing smithi10729407-18 from 5185536 to 5218304 tid 4 2020-06-12T02:38:41.089 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:38:41.089 INFO:tasks.rados.rados.0.smithi107.stdout:8414: seq_num 8413 ranges {868352=507904,1376256=638976,2015232=565248} 2020-06-12T02:38:41.097 INFO:tasks.rados.rados.0.smithi107.stdout:8414: writing smithi10729407-10 from 868352 to 1376256 tid 1 2020-06-12T02:38:41.101 INFO:tasks.rados.rados.0.smithi107.stdout:8414: writing smithi10729407-10 from 1376256 to 2015232 tid 2 2020-06-12T02:38:41.104 INFO:tasks.rados.rados.0.smithi107.stdout:8414: writing smithi10729407-10 from 2015232 to 2580480 tid 3 2020-06-12T02:38:41.105 INFO:tasks.rados.rados.0.smithi107.stdout:8415: delete oid 11 current snap is 0 2020-06-12T02:38:41.105 INFO:tasks.rados.rados.0.smithi107.stdout:8409: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:41.105 INFO:tasks.rados.rados.0.smithi107.stdout:8409: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:41.105 INFO:tasks.rados.rados.0.smithi107.stdout:8409: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:41.105 INFO:tasks.rados.rados.0.smithi107.stdout:8409: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:41.106 INFO:tasks.rados.rados.0.smithi107.stdout:8409: finishing write tid 7 to smithi10729407-5 2020-06-12T02:38:41.106 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 840 (ObjNum 8408 snap 0 seq_num 8408) dirty exists 2020-06-12T02:38:41.106 INFO:tasks.rados.rados.0.smithi107.stdout:8409: left oid 5 (ObjNum 8408 snap 0 seq_num 8408) 2020-06-12T02:38:41.106 INFO:tasks.rados.rados.0.smithi107.stdout:8411: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:41.107 INFO:tasks.rados.rados.0.smithi107.stdout:8411: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:41.108 INFO:tasks.rados.rados.0.smithi107.stdout:8411: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:41.109 INFO:tasks.rados.rados.0.smithi107.stdout:8411: finishing write tid 5 to smithi10729407-12 2020-06-12T02:38:41.109 INFO:tasks.rados.rados.0.smithi107.stdout:8412: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:41.109 INFO:tasks.rados.rados.0.smithi107.stdout:8411: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:41.109 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1549 (ObjNum 8410 snap 0 seq_num 8410) dirty exists 2020-06-12T02:38:41.110 INFO:tasks.rados.rados.0.smithi107.stdout:8411: left oid 12 (ObjNum 8410 snap 0 seq_num 8410) 2020-06-12T02:38:41.110 INFO:tasks.rados.rados.0.smithi107.stdout:8412: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:41.110 INFO:tasks.rados.rados.0.smithi107.stdout:8412: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:41.110 INFO:tasks.rados.rados.0.smithi107.stdout:8412: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:41.110 INFO:tasks.rados.rados.0.smithi107.stdout:8412: finishing write tid 6 to smithi10729407-1 2020-06-12T02:38:41.111 INFO:tasks.rados.rados.0.smithi107.stdout:8412: finishing write tid 7 to smithi10729407-1 2020-06-12T02:38:41.113 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1526 (ObjNum 8411 snap 0 seq_num 8411) dirty exists 2020-06-12T02:38:41.115 INFO:tasks.rados.rados.0.smithi107.stdout:8412: left oid 1 (ObjNum 8411 snap 0 seq_num 8411) 2020-06-12T02:38:41.115 INFO:tasks.rados.rados.0.smithi107.stdout:8413: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:41.116 INFO:tasks.rados.rados.0.smithi107.stdout:8413: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:41.116 INFO:tasks.rados.rados.0.smithi107.stdout:8413: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:41.117 INFO:tasks.rados.rados.0.smithi107.stdout:8413: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:41.117 INFO:tasks.rados.rados.0.smithi107.stdout:8413: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:41.117 INFO:tasks.rados.rados.0.smithi107.stdout:8413: finishing write tid 7 to smithi10729407-18 2020-06-12T02:38:41.118 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1564 (ObjNum 8412 snap 0 seq_num 8412) dirty exists 2020-06-12T02:38:41.118 INFO:tasks.rados.rados.0.smithi107.stdout:8413: left oid 18 (ObjNum 8412 snap 0 seq_num 8412) 2020-06-12T02:38:41.118 INFO:tasks.rados.rados.0.smithi107.stdout:8414: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:41.118 INFO:tasks.rados.rados.0.smithi107.stdout:8409: done (5 left) 2020-06-12T02:38:41.119 INFO:tasks.rados.rados.0.smithi107.stdout:8411: done (4 left) 2020-06-12T02:38:41.119 INFO:tasks.rados.rados.0.smithi107.stdout:8412: done (3 left) 2020-06-12T02:38:41.119 INFO:tasks.rados.rados.0.smithi107.stdout:8413: done (2 left) 2020-06-12T02:38:41.119 INFO:tasks.rados.rados.0.smithi107.stdout:8415: done (1 left) 2020-06-12T02:38:41.119 INFO:tasks.rados.rados.0.smithi107.stdout:8416: delete oid 12 current snap is 0 2020-06-12T02:38:41.120 INFO:tasks.rados.rados.0.smithi107.stdout:8414: finishing write tid 2 to smithi10729407-10 2020-06-12T02:38:41.120 INFO:tasks.rados.rados.0.smithi107.stdout:8414: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:41.120 INFO:tasks.rados.rados.0.smithi107.stdout:8414: finishing write tid 5 to smithi10729407-10 2020-06-12T02:38:41.121 INFO:tasks.rados.rados.0.smithi107.stdout:8416: done (1 left) 2020-06-12T02:38:41.121 INFO:tasks.rados.rados.0.smithi107.stdout:8417: delete oid 3 current snap is 0 2020-06-12T02:38:41.121 INFO:tasks.rados.rados.0.smithi107.stdout:8414: finishing write tid 6 to smithi10729407-10 2020-06-12T02:38:41.121 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 611 (ObjNum 8413 snap 0 seq_num 8413) dirty exists 2020-06-12T02:38:41.121 INFO:tasks.rados.rados.0.smithi107.stdout:8414: left oid 10 (ObjNum 8413 snap 0 seq_num 8413) 2020-06-12T02:38:41.122 INFO:tasks.rados.rados.0.smithi107.stdout:8414: done (1 left) 2020-06-12T02:38:41.122 INFO:tasks.rados.rados.0.smithi107.stdout:8417: done (0 left) 2020-06-12T02:38:41.123 INFO:tasks.rados.rados.0.smithi107.stdout:8418: delete oid 9 current snap is 0 2020-06-12T02:38:41.128 INFO:tasks.rados.rados.0.smithi107.stdout:8418: done (0 left) 2020-06-12T02:38:41.129 INFO:tasks.rados.rados.0.smithi107.stdout:8419: delete oid 17 current snap is 0 2020-06-12T02:38:41.130 INFO:tasks.rados.rados.0.smithi107.stdout:8419: done (0 left) 2020-06-12T02:38:41.130 INFO:tasks.rados.rados.0.smithi107.stdout:8420: delete oid 19 current snap is 0 2020-06-12T02:38:41.130 INFO:tasks.rados.rados.0.smithi107.stdout:8420: done (0 left) 2020-06-12T02:38:41.130 INFO:tasks.rados.rados.0.smithi107.stdout:8421: delete oid 5 current snap is 0 2020-06-12T02:38:41.135 INFO:tasks.rados.rados.0.smithi107.stdout:8421: done (0 left) 2020-06-12T02:38:41.136 INFO:tasks.rados.rados.0.smithi107.stdout:8422: delete oid 18 current snap is 0 2020-06-12T02:38:41.138 INFO:tasks.rados.rados.0.smithi107.stdout:8422: done (0 left) 2020-06-12T02:38:41.139 INFO:tasks.rados.rados.0.smithi107.stdout:8423: delete oid 1 current snap is 0 2020-06-12T02:38:41.142 INFO:tasks.rados.rados.0.smithi107.stdout:8423: done (0 left) 2020-06-12T02:38:41.142 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:38:41.142 INFO:tasks.rados.rados.0.smithi107.stdout:8424: seq_num 8423 ranges {1458176=671744,2129920=729088,2859008=475136} 2020-06-12T02:38:41.150 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:41.150 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:38:41.151 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:38:41.151 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:38:41.151 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:41.156 INFO:tasks.rados.rados.0.smithi107.stdout:8424: writing smithi10729407-2 from 1458176 to 2129920 tid 1 2020-06-12T02:38:41.158 INFO:tasks.rados.rados.0.smithi107.stdout:8424: writing smithi10729407-2 from 2129920 to 2859008 tid 2 2020-06-12T02:38:41.160 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-06-12T02:38:41.163 INFO:tasks.rados.rados.0.smithi107.stdout:8424: writing smithi10729407-2 from 2859008 to 3334144 tid 3 2020-06-12T02:38:41.164 INFO:tasks.rados.rados.0.smithi107.stdout:8425: delete oid 3 current snap is 0 2020-06-12T02:38:41.164 INFO:tasks.rados.rados.0.smithi107.stdout:8424: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:41.167 INFO:tasks.rados.rados.0.smithi107.stdout:8425: done (1 left) 2020-06-12T02:38:41.168 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:38:41.168 INFO:tasks.rados.rados.0.smithi107.stdout:8426: seq_num 8425 ranges {2580480=712704,3293184=548864,3842048=516096,4358144=139264} 2020-06-12T02:38:41.172 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:41.169+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:41.183 INFO:tasks.rados.rados.0.smithi107.stdout:8426: writing smithi10729407-10 from 2580480 to 3293184 tid 1 2020-06-12T02:38:41.187 INFO:tasks.rados.rados.0.smithi107.stdout:8426: writing smithi10729407-10 from 3293184 to 3842048 tid 2 2020-06-12T02:38:41.190 INFO:tasks.rados.rados.0.smithi107.stdout:8426: writing smithi10729407-10 from 3842048 to 4358144 tid 3 2020-06-12T02:38:41.191 INFO:tasks.rados.rados.0.smithi107.stdout:8426: writing smithi10729407-10 from 4358144 to 4497408 tid 4 2020-06-12T02:38:41.191 INFO:tasks.rados.rados.0.smithi107.stdout:8427: delete oid 5 current snap is 0 2020-06-12T02:38:41.191 INFO:tasks.rados.rados.0.smithi107.stdout:8424: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:41.192 INFO:tasks.rados.rados.0.smithi107.stdout:8424: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:41.192 INFO:tasks.rados.rados.0.smithi107.stdout:8424: finishing write tid 5 to smithi10729407-2 2020-06-12T02:38:41.192 INFO:tasks.rados.rados.0.smithi107.stdout:8424: finishing write tid 6 to smithi10729407-2 2020-06-12T02:38:41.192 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 789 (ObjNum 8423 snap 0 seq_num 8423) dirty exists 2020-06-12T02:38:41.192 INFO:tasks.rados.rados.0.smithi107.stdout:8424: left oid 2 (ObjNum 8423 snap 0 seq_num 8423) 2020-06-12T02:38:41.193 INFO:tasks.rados.rados.0.smithi107.stdout:8426: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:41.194 INFO:tasks.rados.rados.0.smithi107.stdout:8424: done (2 left) 2020-06-12T02:38:41.194 INFO:tasks.rados.rados.0.smithi107.stdout:8427: done (1 left) 2020-06-12T02:38:41.194 INFO:tasks.rados.rados.0.smithi107.stdout:8428: delete oid 19 current snap is 0 2020-06-12T02:38:41.196 INFO:tasks.rados.rados.0.smithi107.stdout:8428: done (1 left) 2020-06-12T02:38:41.196 INFO:tasks.rados.rados.0.smithi107.stdout:8429: delete oid 7 current snap is 0 2020-06-12T02:38:41.198 INFO:tasks.rados.rados.0.smithi107.stdout:8426: finishing write tid 2 to smithi10729407-10 2020-06-12T02:38:41.200 INFO:tasks.rados.rados.0.smithi107.stdout:8429: done (1 left) 2020-06-12T02:38:41.210 INFO:tasks.rados.rados.0.smithi107.stdout:8430: delete oid 18 current snap is 0 2020-06-12T02:38:41.211 INFO:tasks.rados.rados.0.smithi107.stdout:8426: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:41.212 INFO:tasks.rados.rados.0.smithi107.stdout:8426: finishing write tid 4 to smithi10729407-10 2020-06-12T02:38:41.213 INFO:tasks.rados.rados.0.smithi107.stdout:8426: finishing write tid 6 to smithi10729407-10 2020-06-12T02:38:41.213 INFO:tasks.rados.rados.0.smithi107.stdout:8426: finishing write tid 7 to smithi10729407-10 2020-06-12T02:38:41.213 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 616 (ObjNum 8425 snap 0 seq_num 8425) dirty exists 2020-06-12T02:38:41.215 INFO:tasks.rados.rados.0.smithi107.stdout:8426: left oid 10 (ObjNum 8425 snap 0 seq_num 8425) 2020-06-12T02:38:41.215 INFO:tasks.rados.rados.0.smithi107.stdout:8426: done (1 left) 2020-06-12T02:38:41.215 INFO:tasks.rados.rados.0.smithi107.stdout:8430: done (0 left) 2020-06-12T02:38:41.215 INFO:tasks.rados.rados.0.smithi107.stdout:8431: delete oid 6 current snap is 0 2020-06-12T02:38:41.215 INFO:tasks.rados.rados.0.smithi107.stdout:8431: done (0 left) 2020-06-12T02:38:41.216 INFO:tasks.rados.rados.0.smithi107.stdout:8432: delete oid 9 current snap is 0 2020-06-12T02:38:41.216 INFO:tasks.rados.rados.0.smithi107.stdout:8432: done (0 left) 2020-06-12T02:38:41.216 INFO:tasks.rados.rados.0.smithi107.stdout:8433: delete oid 12 current snap is 0 2020-06-12T02:38:41.217 INFO:tasks.rados.rados.0.smithi107.stdout:8433: done (0 left) 2020-06-12T02:38:41.217 INFO:tasks.rados.rados.0.smithi107.stdout:8434: delete oid 7 current snap is 0 2020-06-12T02:38:41.218 INFO:tasks.rados.rados.0.smithi107.stdout:8434: done (0 left) 2020-06-12T02:38:41.218 INFO:tasks.rados.rados.0.smithi107.stdout:8435: delete oid 19 current snap is 0 2020-06-12T02:38:41.218 INFO:tasks.rados.rados.0.smithi107.stdout:8435: done (0 left) 2020-06-12T02:38:41.218 INFO:tasks.rados.rados.0.smithi107.stdout:8436: delete oid 6 current snap is 0 2020-06-12T02:38:41.221 INFO:tasks.rados.rados.0.smithi107.stdout:8436: done (0 left) 2020-06-12T02:38:41.221 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:38:41.221 INFO:tasks.rados.rados.0.smithi107.stdout:8437: seq_num 8436 ranges {16384=491520,507904=524288,1032192=696320,1728512=385024} 2020-06-12T02:38:41.223 INFO:tasks.rados.rados.0.smithi107.stdout:8437: writing smithi10729407-20 from 16384 to 507904 tid 1 2020-06-12T02:38:41.227 INFO:tasks.rados.rados.0.smithi107.stdout:8437: writing smithi10729407-20 from 507904 to 1032192 tid 2 2020-06-12T02:38:41.233 INFO:tasks.rados.rados.0.smithi107.stdout:8437: writing smithi10729407-20 from 1032192 to 1728512 tid 3 2020-06-12T02:38:41.234 INFO:tasks.rados.rados.0.smithi107.stdout:8437: writing smithi10729407-20 from 1728512 to 2113536 tid 4 2020-06-12T02:38:41.235 INFO:tasks.rados.rados.0.smithi107.stdout:8438: delete oid 16 current snap is 0 2020-06-12T02:38:41.235 INFO:tasks.rados.rados.0.smithi107.stdout:8437: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:41.237 INFO:tasks.rados.rados.0.smithi107.stdout:8437: finishing write tid 2 to smithi10729407-20 2020-06-12T02:38:41.239 INFO:tasks.rados.rados.0.smithi107.stdout:8438: done (1 left) 2020-06-12T02:38:41.239 INFO:tasks.rados.rados.0.smithi107.stdout:8439: delete oid 18 current snap is 0 2020-06-12T02:38:41.241 INFO:tasks.rados.rados.0.smithi107.stdout:8437: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:41.242 INFO:tasks.rados.rados.0.smithi107.stdout:8437: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:41.242 INFO:tasks.rados.rados.0.smithi107.stdout:8437: finishing write tid 6 to smithi10729407-20 2020-06-12T02:38:41.244 INFO:tasks.rados.rados.0.smithi107.stdout:8437: finishing write tid 7 to smithi10729407-20 2020-06-12T02:38:41.244 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 613 (ObjNum 8436 snap 0 seq_num 8436) dirty exists 2020-06-12T02:38:41.244 INFO:tasks.rados.rados.0.smithi107.stdout:8437: left oid 20 (ObjNum 8436 snap 0 seq_num 8436) 2020-06-12T02:38:41.246 INFO:tasks.rados.rados.0.smithi107.stdout:8437: done (1 left) 2020-06-12T02:38:41.247 INFO:tasks.rados.rados.0.smithi107.stdout:8439: done (0 left) 2020-06-12T02:38:41.247 INFO:tasks.rados.rados.0.smithi107.stdout:8440: delete oid 20 current snap is 0 2020-06-12T02:38:41.249 INFO:tasks.rados.rados.0.smithi107.stdout:8440: done (0 left) 2020-06-12T02:38:41.249 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:38:41.250 INFO:tasks.rados.rados.0.smithi107.stdout:8441: seq_num 8440 ranges {0=557056,557056=507904,1064960=679936,1744896=417792} 2020-06-12T02:38:41.258 INFO:tasks.rados.rados.0.smithi107.stdout:8441: writing smithi10729407-20 from 0 to 557056 tid 1 2020-06-12T02:38:41.258 INFO:tasks.rados.rados.0.smithi107.stdout:8441: writing smithi10729407-20 from 557056 to 1064960 tid 2 2020-06-12T02:38:41.259 INFO:tasks.rados.rados.0.smithi107.stdout:8441: writing smithi10729407-20 from 1064960 to 1744896 tid 3 2020-06-12T02:38:41.261 INFO:tasks.rados.rados.0.smithi107.stdout:8441: writing smithi10729407-20 from 1744896 to 2162688 tid 4 2020-06-12T02:38:41.261 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:38:41.262 INFO:tasks.rados.rados.0.smithi107.stdout:8442: seq_num 8441 ranges {0=573440,573440=794624,1368064=483328,1851392=327680} 2020-06-12T02:38:41.264 INFO:tasks.rados.rados.0.smithi107.stdout:8442: writing smithi10729407-11 from 0 to 573440 tid 1 2020-06-12T02:38:41.269 INFO:tasks.rados.rados.0.smithi107.stdout:8442: writing smithi10729407-11 from 573440 to 1368064 tid 2 2020-06-12T02:38:41.272 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:41.270+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:41.273 INFO:tasks.rados.rados.0.smithi107.stdout:8442: writing smithi10729407-11 from 1368064 to 1851392 tid 3 2020-06-12T02:38:41.275 INFO:tasks.rados.rados.0.smithi107.stdout:8442: writing smithi10729407-11 from 1851392 to 2179072 tid 4 2020-06-12T02:38:41.275 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:41.275 INFO:tasks.rados.rados.0.smithi107.stdout:8443: seq_num 8442 ranges {0=589824,589824=679936,1269760=647168,1916928=278528} 2020-06-12T02:38:41.278 INFO:tasks.rados.rados.0.smithi107.stdout:8443: writing smithi10729407-12 from 0 to 589824 tid 1 2020-06-12T02:38:41.283 INFO:tasks.rados.rados.0.smithi107.stdout:8443: writing smithi10729407-12 from 589824 to 1269760 tid 2 2020-06-12T02:38:41.291 INFO:tasks.rados.rados.0.smithi107.stdout:8443: writing smithi10729407-12 from 1269760 to 1916928 tid 3 2020-06-12T02:38:41.292 INFO:tasks.rados.rados.0.smithi107.stdout:8443: writing smithi10729407-12 from 1916928 to 2195456 tid 4 2020-06-12T02:38:41.292 INFO:tasks.rados.rados.0.smithi107.stdout:8444: delete oid 9 current snap is 0 2020-06-12T02:38:41.292 INFO:tasks.rados.rados.0.smithi107.stdout:8441: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:41.292 INFO:tasks.rados.rados.0.smithi107.stdout:8441: finishing write tid 2 to smithi10729407-20 2020-06-12T02:38:41.292 INFO:tasks.rados.rados.0.smithi107.stdout:8441: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:41.293 INFO:tasks.rados.rados.0.smithi107.stdout:8441: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:41.293 INFO:tasks.rados.rados.0.smithi107.stdout:8441: finishing write tid 6 to smithi10729407-20 2020-06-12T02:38:41.293 INFO:tasks.rados.rados.0.smithi107.stdout:8442: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:41.293 INFO:tasks.rados.rados.0.smithi107.stdout:8441: finishing write tid 7 to smithi10729407-20 2020-06-12T02:38:41.293 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 619 (ObjNum 8440 snap 0 seq_num 8440) dirty exists 2020-06-12T02:38:41.294 INFO:tasks.rados.rados.0.smithi107.stdout:8441: left oid 20 (ObjNum 8440 snap 0 seq_num 8440) 2020-06-12T02:38:41.294 INFO:tasks.rados.rados.0.smithi107.stdout:8442: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:41.294 INFO:tasks.rados.rados.0.smithi107.stdout:8443: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:41.294 INFO:tasks.rados.rados.0.smithi107.stdout:8442: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:41.294 INFO:tasks.rados.rados.0.smithi107.stdout:8442: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:41.294 INFO:tasks.rados.rados.0.smithi107.stdout:8442: finishing write tid 6 to smithi10729407-11 2020-06-12T02:38:41.295 INFO:tasks.rados.rados.0.smithi107.stdout:8442: finishing write tid 7 to smithi10729407-11 2020-06-12T02:38:41.295 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1485 (ObjNum 8441 snap 0 seq_num 8441) dirty exists 2020-06-12T02:38:41.295 INFO:tasks.rados.rados.0.smithi107.stdout:8442: left oid 11 (ObjNum 8441 snap 0 seq_num 8441) 2020-06-12T02:38:41.295 INFO:tasks.rados.rados.0.smithi107.stdout:8441: done (3 left) 2020-06-12T02:38:41.296 INFO:tasks.rados.rados.0.smithi107.stdout:8442: done (2 left) 2020-06-12T02:38:41.296 INFO:tasks.rados.rados.0.smithi107.stdout:8444: done (1 left) 2020-06-12T02:38:41.298 INFO:tasks.rados.rados.0.smithi107.stdout:8445: delete oid 14 current snap is 0 2020-06-12T02:38:41.299 INFO:tasks.rados.rados.0.smithi107.stdout:8445: done (1 left) 2020-06-12T02:38:41.299 INFO:tasks.rados.rados.0.smithi107.stdout:8446: delete oid 20 current snap is 0 2020-06-12T02:38:41.299 INFO:tasks.rados.rados.0.smithi107.stdout:8446: done (1 left) 2020-06-12T02:38:41.299 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:38:41.299 INFO:tasks.rados.rados.0.smithi107.stdout:8447: seq_num 8446 ranges {0=663552,663552=614400,1277952=622592,1900544=368640} 2020-06-12T02:38:41.303 INFO:tasks.rados.rados.0.smithi107.stdout:8447: writing smithi10729407-20 from 0 to 663552 tid 1 2020-06-12T02:38:41.307 INFO:tasks.rados.rados.0.smithi107.stdout:8447: writing smithi10729407-20 from 663552 to 1277952 tid 2 2020-06-12T02:38:41.310 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:41.311 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:38:41.311 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:38:41.311 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:38:41.311 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.312 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8464 3.fs0 3.57bc3c1f (undecoded) ondisk+retry+write+known_if_redirected e34)", 2020-06-12T02:38:41.312 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:53.549364+0000", 2020-06-12T02:38:41.312 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 47.757216341000003, 2020-06-12T02:38:41.312 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.003107494, 2020-06-12T02:38:41.312 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:41.313 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:38:41.313 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:41.313 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:41.314 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:41.315 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8464 2020-06-12T02:38:41.315 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.316 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:41.316 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.316 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:41.316 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.549364+0000", 2020-06-12T02:38:41.317 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:41.317 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.317 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.317 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:41.318 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.549364+0000", 2020-06-12T02:38:41.318 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023109609999999998 2020-06-12T02:38:41.318 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.318 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.318 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:41.319 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551675+0000", 2020-06-12T02:38:41.319 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3331e-05 2020-06-12T02:38:41.319 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.319 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.319 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:41.319 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551688+0000", 2020-06-12T02:38:41.320 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4173e-05 2020-06-12T02:38:41.320 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.320 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.320 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:41.320 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551703+0000", 2020-06-12T02:38:41.321 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0454e-05 2020-06-12T02:38:41.321 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.321 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.321 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:41.321 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551713+0000", 2020-06-12T02:38:41.322 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031069050000000001 2020-06-12T02:38:41.322 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.322 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.322 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.322 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554820+0000", 2020-06-12T02:38:41.322 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2820000000000002e-06 2020-06-12T02:38:41.323 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.323 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.323 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2020-06-12T02:38:41.323 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554826+0000", 2020-06-12T02:38:41.323 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99762648499999995 2020-06-12T02:38:41.324 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.324 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.324 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.324 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552453+0000", 2020-06-12T02:38:41.324 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8902999999999998e-05 2020-06-12T02:38:41.324 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.325 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.325 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:41.325 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552472+0000", 2020-06-12T02:38:41.325 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.003107494 2020-06-12T02:38:41.325 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:41.326 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:41.326 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:41.326 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.326 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.326 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8465 3.fs0 3.57bc3c1f (undecoded) ondisk+retry+write+known_if_redirected e34)", 2020-06-12T02:38:41.326 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:53.551774+0000", 2020-06-12T02:38:41.327 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 47.754806705, 2020-06-12T02:38:41.327 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.000741442, 2020-06-12T02:38:41.327 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:41.327 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:38:41.327 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:41.328 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:41.328 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:41.328 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8465 2020-06-12T02:38:41.328 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.328 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:41.328 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.329 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:41.329 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551774+0000", 2020-06-12T02:38:41.329 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:41.329 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.329 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.330 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:41.330 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551774+0000", 2020-06-12T02:38:41.331 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0010760539999999999 2020-06-12T02:38:41.331 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.332 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.332 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:41.332 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552850+0000", 2020-06-12T02:38:41.332 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0139000000000001e-05 2020-06-12T02:38:41.333 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.333 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.333 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:41.333 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552860+0000", 2020-06-12T02:38:41.333 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7646999999999999e-05 2020-06-12T02:38:41.334 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.334 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.334 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:41.334 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552878+0000", 2020-06-12T02:38:41.335 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2916e-05 2020-06-12T02:38:41.335 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.335 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.335 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:41.335 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552890+0000", 2020-06-12T02:38:41.336 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0019574610000000002 2020-06-12T02:38:41.336 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.336 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.336 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.336 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554848+0000", 2020-06-12T02:38:41.336 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0399999999999995e-07 2020-06-12T02:38:41.337 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.339 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.340 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2020-06-12T02:38:41.340 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554849+0000", 2020-06-12T02:38:41.341 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99765586100000003 2020-06-12T02:38:41.341 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.341 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.341 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.341 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552504+0000", 2020-06-12T02:38:41.341 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.066e-05 2020-06-12T02:38:41.342 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.342 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.342 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:41.342 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552515+0000", 2020-06-12T02:38:41.342 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.000741442 2020-06-12T02:38:41.343 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:41.343 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:41.343 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:41.343 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.343 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.344 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8466 3.fs0 3.57bc3c1f (undecoded) ondisk+write+known_if_redirected e34)", 2020-06-12T02:38:41.344 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:53.552953+0000", 2020-06-12T02:38:41.344 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 47.753627424999998, 2020-06-12T02:38:41.344 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99960144299999998, 2020-06-12T02:38:41.344 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:41.344 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:38:41.345 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:41.345 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:41.345 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:41.345 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8466 2020-06-12T02:38:41.345 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.346 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:41.347 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.348 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:41.348 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552953+0000", 2020-06-12T02:38:41.349 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:41.349 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.350 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.350 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:41.350 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552953+0000", 2020-06-12T02:38:41.350 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013847100000000001 2020-06-12T02:38:41.350 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.351 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.351 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:41.351 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553091+0000", 2020-06-12T02:38:41.351 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7190000000000004e-06 2020-06-12T02:38:41.351 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.351 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.352 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:41.352 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553098+0000", 2020-06-12T02:38:41.352 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.171e-05 2020-06-12T02:38:41.352 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.352 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.353 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:41.353 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553110+0000", 2020-06-12T02:38:41.353 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5770000000000001e-06 2020-06-12T02:38:41.353 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.353 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.354 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:41.354 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553115+0000", 2020-06-12T02:38:41.354 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0017512739999999999 2020-06-12T02:38:41.354 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.354 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.354 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.355 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554867+0000", 2020-06-12T02:38:41.355 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1200000000000003e-07 2020-06-12T02:38:41.355 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.355 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.355 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2020-06-12T02:38:41.356 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554867+0000", 2020-06-12T02:38:41.356 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99767707100000003 2020-06-12T02:38:41.356 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.356 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.357 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.357 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552544+0000", 2020-06-12T02:38:41.357 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0009e-05 2020-06-12T02:38:41.357 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.357 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.358 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:41.358 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552554+0000", 2020-06-12T02:38:41.358 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99960144299999998 2020-06-12T02:38:41.358 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:41.358 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:41.358 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:41.359 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.359 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.359 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8467 3.fs0 3.57bc3c1f (undecoded) ondisk+write+known_if_redirected e34)", 2020-06-12T02:38:41.359 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:53.553172+0000", 2020-06-12T02:38:41.359 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 47.753408209, 2020-06-12T02:38:41.360 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99942086200000002, 2020-06-12T02:38:41.360 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:41.360 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:38:41.360 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:41.360 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:41.361 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:41.361 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8467 2020-06-12T02:38:41.361 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.361 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:41.361 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.361 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:41.362 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553172+0000", 2020-06-12T02:38:41.362 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:41.362 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.362 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.362 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:41.363 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553172+0000", 2020-06-12T02:38:41.363 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5913000000000003e-05 2020-06-12T02:38:41.363 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.363 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.364 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:41.364 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553248+0000", 2020-06-12T02:38:41.364 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7740000000000001e-06 2020-06-12T02:38:41.364 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.364 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.365 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:41.365 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553254+0000", 2020-06-12T02:38:41.365 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1494999999999999e-05 2020-06-12T02:38:41.365 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.365 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.365 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:41.366 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553265+0000", 2020-06-12T02:38:41.366 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5170000000000001e-06 2020-06-12T02:38:41.366 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.366 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.366 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:41.367 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553270+0000", 2020-06-12T02:38:41.367 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001616567 2020-06-12T02:38:41.367 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.367 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.367 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.367 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554886+0000", 2020-06-12T02:38:41.368 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7800000000000001e-07 2020-06-12T02:38:41.368 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.368 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.368 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2020-06-12T02:38:41.369 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554887+0000", 2020-06-12T02:38:41.369 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99769652399999997 2020-06-12T02:38:41.369 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.369 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.369 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.370 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552584+0000", 2020-06-12T02:38:41.370 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4939999999999992e-06 2020-06-12T02:38:41.371 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.371 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.371 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:41.371 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552593+0000", 2020-06-12T02:38:41.371 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99942086200000002 2020-06-12T02:38:41.372 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:41.372 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:41.372 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:41.372 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.372 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.373 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8468 3.fs0 3.57bc3c1f (undecoded) ondisk+read+rwordered+known_if_redirected e34)", 2020-06-12T02:38:41.373 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:53.553311+0000", 2020-06-12T02:38:41.373 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 47.753269611, 2020-06-12T02:38:41.373 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99988704799999994, 2020-06-12T02:38:41.373 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:41.374 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:38:41.374 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:41.374 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:41.375 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:41.375 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8468 2020-06-12T02:38:41.375 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.375 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:41.375 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.376 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:41.376 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553311+0000", 2020-06-12T02:38:41.376 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:41.376 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.377 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.377 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:41.377 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553311+0000", 2020-06-12T02:38:41.377 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2050000000000001e-05 2020-06-12T02:38:41.377 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.378 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.378 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:41.378 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553333+0000", 2020-06-12T02:38:41.378 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5489999999999998e-06 2020-06-12T02:38:41.378 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.379 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.379 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:41.379 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553336+0000", 2020-06-12T02:38:41.379 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1620000000000002e-06 2020-06-12T02:38:41.379 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.379 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.380 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:41.380 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553344+0000", 2020-06-12T02:38:41.380 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.807e-06 2020-06-12T02:38:41.380 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.380 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.381 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:41.381 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553346+0000", 2020-06-12T02:38:41.381 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001560575 2020-06-12T02:38:41.381 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.381 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.382 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.382 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554907+0000", 2020-06-12T02:38:41.382 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1429999999999999e-06 2020-06-12T02:38:41.382 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.382 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.382 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2020-06-12T02:38:41.383 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554908+0000", 2020-06-12T02:38:41.383 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99827844600000004 2020-06-12T02:38:41.383 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.383 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.383 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.384 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.553187+0000", 2020-06-12T02:38:41.384 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1316e-05 2020-06-12T02:38:41.385 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.385 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.385 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:41.385 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.553198+0000", 2020-06-12T02:38:41.385 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99988704799999994 2020-06-12T02:38:41.386 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:41.386 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:41.386 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:41.386 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.386 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.386 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:9619 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e54)", 2020-06-12T02:38:41.387 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:16.872925+0000", 2020-06-12T02:38:41.387 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 24.433655487999999, 2020-06-12T02:38:41.387 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020560512, 2020-06-12T02:38:41.387 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:41.387 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:41.388 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:41.388 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:41.388 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:41.388 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 9619 2020-06-12T02:38:41.388 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.389 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:41.389 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.389 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:41.389 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.872925+0000", 2020-06-12T02:38:41.389 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:41.389 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.390 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.390 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:41.390 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.872925+0000", 2020-06-12T02:38:41.390 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4219e-05 2020-06-12T02:38:41.390 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.391 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.391 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:41.391 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.872949+0000", 2020-06-12T02:38:41.391 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2909999999999998e-06 2020-06-12T02:38:41.391 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.392 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.392 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:41.392 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.872954+0000", 2020-06-12T02:38:41.392 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6299999999999998e-06 2020-06-12T02:38:41.392 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.393 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.393 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:41.393 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.872962+0000", 2020-06-12T02:38:41.393 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2010000000000002e-06 2020-06-12T02:38:41.394 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.394 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.394 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:41.394 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.872964+0000", 2020-06-12T02:38:41.395 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045097540000000004 2020-06-12T02:38:41.395 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.395 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.395 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.395 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.877474+0000", 2020-06-12T02:38:41.396 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8949000000000002e-05 2020-06-12T02:38:41.396 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.396 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.396 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:41.396 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.877523+0000", 2020-06-12T02:38:41.396 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.007143132 2020-06-12T02:38:41.397 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.397 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.397 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.397 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.884666+0000", 2020-06-12T02:38:41.397 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3729e-05 2020-06-12T02:38:41.398 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.398 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.398 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:41.398 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.884720+0000", 2020-06-12T02:38:41.398 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0076965799999999997 2020-06-12T02:38:41.399 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.399 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.399 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.399 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.892416+0000", 2020-06-12T02:38:41.399 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3109999999999997e-05 2020-06-12T02:38:41.400 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.400 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.400 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:41.400 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.892480+0000", 2020-06-12T02:38:41.400 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0010059170000000001 2020-06-12T02:38:41.400 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.401 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.401 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:41.401 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:16.893485+0000", 2020-06-12T02:38:41.401 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020560512 2020-06-12T02:38:41.401 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:41.402 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:41.402 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:41.402 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.402 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.402 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11675 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [append 0~753664] snapc 0=[] ondisk+write+known_if_redirected e56)", 2020-06-12T02:38:41.402 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:22.698885+0000", 2020-06-12T02:38:41.403 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 18.607695502999999, 2020-06-12T02:38:41.403 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020548525000000002, 2020-06-12T02:38:41.403 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:41.403 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:41.403 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:41.404 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:41.404 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:41.404 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11675 2020-06-12T02:38:41.404 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.404 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:41.405 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.405 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:41.405 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.698885+0000", 2020-06-12T02:38:41.405 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:41.406 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.406 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.406 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:41.406 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.698885+0000", 2020-06-12T02:38:41.406 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00092031600000000002 2020-06-12T02:38:41.407 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.407 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.407 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:41.407 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.699805+0000", 2020-06-12T02:38:41.407 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4439999999999996e-06 2020-06-12T02:38:41.408 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.411 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.411 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:41.411 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.699812+0000", 2020-06-12T02:38:41.412 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1874e-05 2020-06-12T02:38:41.412 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.412 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.412 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:41.413 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.699824+0000", 2020-06-12T02:38:41.414 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1360000000000002e-06 2020-06-12T02:38:41.414 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.414 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.414 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:41.414 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.699830+0000", 2020-06-12T02:38:41.415 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9334e-05 2020-06-12T02:38:41.415 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.415 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.415 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.415 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.699869+0000", 2020-06-12T02:38:41.416 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1976000000000003e-05 2020-06-12T02:38:41.416 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.416 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.416 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:41.416 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.699951+0000", 2020-06-12T02:38:41.417 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0055547799999999996 2020-06-12T02:38:41.417 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.417 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.418 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.418 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.705506+0000", 2020-06-12T02:38:41.418 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1692000000000005e-05 2020-06-12T02:38:41.418 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.418 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.419 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:41.419 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.705567+0000", 2020-06-12T02:38:41.419 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00065387599999999996 2020-06-12T02:38:41.419 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.419 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.420 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:41.420 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.706221+0000", 2020-06-12T02:38:41.420 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012749887 2020-06-12T02:38:41.420 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.420 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.420 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:41.421 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.718971+0000", 2020-06-12T02:38:41.421 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00042074300000000001 2020-06-12T02:38:41.421 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.421 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.421 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:41.422 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.719392+0000", 2020-06-12T02:38:41.422 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1467000000000001e-05 2020-06-12T02:38:41.422 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.422 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.423 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:41.423 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.719433+0000", 2020-06-12T02:38:41.423 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020548525000000002 2020-06-12T02:38:41.423 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:41.423 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:41.423 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:41.424 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.424 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.425 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11679 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e56)", 2020-06-12T02:38:41.425 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:22.704390+0000", 2020-06-12T02:38:41.425 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 18.602190806999999, 2020-06-12T02:38:41.426 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020541976999999999, 2020-06-12T02:38:41.426 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:41.426 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:41.426 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:41.427 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:41.427 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:41.427 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11679 2020-06-12T02:38:41.427 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.427 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:41.428 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.428 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:41.428 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.704390+0000", 2020-06-12T02:38:41.428 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:41.428 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.428 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.429 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:41.429 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.704390+0000", 2020-06-12T02:38:41.429 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7000999999999999e-05 2020-06-12T02:38:41.429 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.429 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.430 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:41.430 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.704437+0000", 2020-06-12T02:38:41.430 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3609999999999997e-06 2020-06-12T02:38:41.430 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.430 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.430 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:41.431 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.704443+0000", 2020-06-12T02:38:41.431 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1874e-05 2020-06-12T02:38:41.431 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.431 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.431 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:41.432 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.704455+0000", 2020-06-12T02:38:41.432 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.639e-06 2020-06-12T02:38:41.432 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.432 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.432 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:41.433 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.704459+0000", 2020-06-12T02:38:41.433 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3045e-05 2020-06-12T02:38:41.433 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.433 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.433 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.433 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.704493+0000", 2020-06-12T02:38:41.434 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2006e-05 2020-06-12T02:38:41.434 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.434 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.434 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:41.434 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.704585+0000", 2020-06-12T02:38:41.435 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01442094 2020-06-12T02:38:41.435 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.435 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.435 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.435 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.719005+0000", 2020-06-12T02:38:41.435 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9537000000000006e-05 2020-06-12T02:38:41.436 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.436 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.436 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:41.436 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.719105+0000", 2020-06-12T02:38:41.436 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0043688859999999998 2020-06-12T02:38:41.437 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.437 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.437 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.437 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.723474+0000", 2020-06-12T02:38:41.437 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000104456 2020-06-12T02:38:41.438 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.438 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.438 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:41.438 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.723578+0000", 2020-06-12T02:38:41.438 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001353232 2020-06-12T02:38:41.438 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.439 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.439 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:41.439 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.724932+0000", 2020-06-12T02:38:41.439 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020541976999999999 2020-06-12T02:38:41.439 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:41.440 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:41.440 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:41.440 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.440 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.440 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11759 3.ds0 3:b983fd28:::smithi10729407-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e56)", 2020-06-12T02:38:41.441 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:22.914106+0000", 2020-06-12T02:38:41.441 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 18.392474880000002, 2020-06-12T02:38:41.441 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020070629999999999, 2020-06-12T02:38:41.442 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:41.442 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:41.442 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:41.442 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:41.442 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:41.443 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11759 2020-06-12T02:38:41.443 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.443 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:41.443 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.443 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:41.443 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.914106+0000", 2020-06-12T02:38:41.444 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:41.444 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.444 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.444 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:41.444 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.914106+0000", 2020-06-12T02:38:41.445 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3359000000000001e-05 2020-06-12T02:38:41.445 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.445 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.445 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:41.445 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.914129+0000", 2020-06-12T02:38:41.445 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2839999999999998e-06 2020-06-12T02:38:41.446 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.446 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.446 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:41.446 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.914132+0000", 2020-06-12T02:38:41.446 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0999999999999998e-06 2020-06-12T02:38:41.447 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.447 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.447 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:41.447 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.914139+0000", 2020-06-12T02:38:41.447 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4370000000000001e-06 2020-06-12T02:38:41.448 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.448 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.448 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:41.448 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.914142+0000", 2020-06-12T02:38:41.448 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0042189710000000002 2020-06-12T02:38:41.449 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.449 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.449 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.449 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.918361+0000", 2020-06-12T02:38:41.449 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9816999999999997e-05 2020-06-12T02:38:41.449 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.450 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.450 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:41.450 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.918461+0000", 2020-06-12T02:38:41.450 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0092868900000000008 2020-06-12T02:38:41.450 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.451 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.451 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.452 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.927747+0000", 2020-06-12T02:38:41.452 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9674999999999999e-05 2020-06-12T02:38:41.452 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.452 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.453 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:41.453 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.927827+0000", 2020-06-12T02:38:41.453 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0054114660000000002 2020-06-12T02:38:41.453 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.453 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.454 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.454 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.933239+0000", 2020-06-12T02:38:41.454 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011061900000000001 2020-06-12T02:38:41.454 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.454 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.454 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:41.455 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.933349+0000", 2020-06-12T02:38:41.455 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00082701200000000002 2020-06-12T02:38:41.456 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.456 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.456 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:41.456 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.934176+0000", 2020-06-12T02:38:41.456 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020070629999999999 2020-06-12T02:38:41.457 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:41.457 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:41.457 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:41.457 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.457 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.458 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11992 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [append 0~581632] snapc 0=[] ondisk+write+known_if_redirected e56)", 2020-06-12T02:38:41.458 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:23.783061+0000", 2020-06-12T02:38:41.458 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 17.52351943, 2020-06-12T02:38:41.458 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022850993, 2020-06-12T02:38:41.458 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:41.458 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:41.459 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:41.459 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:41.459 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:41.459 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11992 2020-06-12T02:38:41.459 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.460 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:41.460 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.460 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:41.460 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783061+0000", 2020-06-12T02:38:41.460 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:41.461 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.461 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.461 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:41.462 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783061+0000", 2020-06-12T02:38:41.462 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047999300000000001 2020-06-12T02:38:41.462 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.463 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.463 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:41.463 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783541+0000", 2020-06-12T02:38:41.463 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5649999999999998e-06 2020-06-12T02:38:41.463 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.464 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.464 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:41.464 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783548+0000", 2020-06-12T02:38:41.464 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3559e-05 2020-06-12T02:38:41.465 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.465 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.465 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:41.466 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783561+0000", 2020-06-12T02:38:41.466 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3539999999999996e-06 2020-06-12T02:38:41.466 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.466 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.467 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:41.467 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783567+0000", 2020-06-12T02:38:41.467 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001353604 2020-06-12T02:38:41.467 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.467 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.467 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.468 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.784921+0000", 2020-06-12T02:38:41.468 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000111853 2020-06-12T02:38:41.468 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.468 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.468 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:41.469 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.785033+0000", 2020-06-12T02:38:41.469 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051647919999999996 2020-06-12T02:38:41.469 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.469 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.469 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.470 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.790198+0000", 2020-06-12T02:38:41.470 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000110742 2020-06-12T02:38:41.470 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.470 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.470 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:41.471 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.790308+0000", 2020-06-12T02:38:41.471 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00090050099999999995 2020-06-12T02:38:41.471 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.471 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.471 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:41.471 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.791209+0000", 2020-06-12T02:38:41.472 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010078284999999999 2020-06-12T02:38:41.472 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.472 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.472 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:41.472 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.801287+0000", 2020-06-12T02:38:41.473 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045443109999999997 2020-06-12T02:38:41.473 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.473 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.473 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:41.473 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.805832+0000", 2020-06-12T02:38:41.473 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.0433999999999996e-05 2020-06-12T02:38:41.474 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.474 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.474 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:41.474 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.805912+0000", 2020-06-12T02:38:41.474 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022850993 2020-06-12T02:38:41.475 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:41.475 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:41.475 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:41.475 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.475 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.476 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11993 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [append 0~647168] snapc 0=[] ondisk+write+known_if_redirected e56)", 2020-06-12T02:38:41.476 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:23.785990+0000", 2020-06-12T02:38:41.476 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 17.520589986000001, 2020-06-12T02:38:41.477 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.02007219, 2020-06-12T02:38:41.477 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:41.477 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:41.477 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:41.477 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:41.478 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:41.478 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11993 2020-06-12T02:38:41.478 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.478 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:41.478 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.478 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:41.479 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.785990+0000", 2020-06-12T02:38:41.479 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:41.479 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.479 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.479 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:41.480 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.785990+0000", 2020-06-12T02:38:41.481 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00072696100000000004 2020-06-12T02:38:41.481 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.481 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.481 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:41.481 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.786717+0000", 2020-06-12T02:38:41.482 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5780000000000004e-06 2020-06-12T02:38:41.482 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.482 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.482 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:41.482 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.786724+0000", 2020-06-12T02:38:41.483 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0777e-05 2020-06-12T02:38:41.483 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.483 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.483 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:41.483 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.786735+0000", 2020-06-12T02:38:41.484 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7469999999999999e-06 2020-06-12T02:38:41.484 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.484 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.484 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:41.485 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.786741+0000", 2020-06-12T02:38:41.485 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000128975 2020-06-12T02:38:41.485 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.485 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.485 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.486 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.786870+0000", 2020-06-12T02:38:41.486 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000111037 2020-06-12T02:38:41.486 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.486 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.486 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:41.487 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.786982+0000", 2020-06-12T02:38:41.487 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0081893199999999999 2020-06-12T02:38:41.487 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.487 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.487 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.488 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.795171+0000", 2020-06-12T02:38:41.488 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9732000000000003e-05 2020-06-12T02:38:41.488 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.488 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.488 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:41.488 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.795241+0000", 2020-06-12T02:38:41.489 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00059786699999999997 2020-06-12T02:38:41.489 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.489 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.489 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:41.489 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.795838+0000", 2020-06-12T02:38:41.490 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0063341910000000003 2020-06-12T02:38:41.490 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.490 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.490 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:41.490 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.802173+0000", 2020-06-12T02:38:41.491 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0038212929999999999 2020-06-12T02:38:41.491 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.491 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.491 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:41.491 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.805994+0000", 2020-06-12T02:38:41.492 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8712000000000002e-05 2020-06-12T02:38:41.492 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.492 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.492 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:41.492 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.806063+0000", 2020-06-12T02:38:41.492 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.02007219 2020-06-12T02:38:41.493 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:41.493 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:41.493 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:41.493 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.493 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.494 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11994 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected e56)", 2020-06-12T02:38:41.494 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:23.788734+0000", 2020-06-12T02:38:41.494 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 17.517846785, 2020-06-12T02:38:41.494 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024022136999999999, 2020-06-12T02:38:41.494 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:41.495 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:41.495 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:41.495 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:41.495 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:41.496 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11994 2020-06-12T02:38:41.496 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.496 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:41.496 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.496 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:41.497 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.788734+0000", 2020-06-12T02:38:41.497 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:41.497 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.497 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.497 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:41.498 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.788734+0000", 2020-06-12T02:38:41.498 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00034212499999999998 2020-06-12T02:38:41.498 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.498 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.498 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:41.498 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789076+0000", 2020-06-12T02:38:41.499 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.254e-06 2020-06-12T02:38:41.499 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.499 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.499 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:41.499 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789081+0000", 2020-06-12T02:38:41.500 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8249999999999994e-06 2020-06-12T02:38:41.500 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.500 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.500 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:41.500 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789090+0000", 2020-06-12T02:38:41.500 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2259999999999999e-06 2020-06-12T02:38:41.501 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.501 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.501 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:41.501 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789094+0000", 2020-06-12T02:38:41.501 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2913000000000001e-05 2020-06-12T02:38:41.502 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.502 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.502 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.502 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789127+0000", 2020-06-12T02:38:41.502 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2001000000000005e-05 2020-06-12T02:38:41.503 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.503 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.503 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:41.503 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789189+0000", 2020-06-12T02:38:41.503 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0097474550000000004 2020-06-12T02:38:41.503 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.504 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.504 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.504 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.798936+0000", 2020-06-12T02:38:41.504 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9882000000000005e-05 2020-06-12T02:38:41.504 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.505 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.505 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:41.505 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.799016+0000", 2020-06-12T02:38:41.505 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00048691000000000001 2020-06-12T02:38:41.506 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.506 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.506 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:41.506 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.799503+0000", 2020-06-12T02:38:41.506 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002766363 2020-06-12T02:38:41.507 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.507 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.507 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:41.507 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.802270+0000", 2020-06-12T02:38:41.507 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010404759 2020-06-12T02:38:41.508 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.508 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.508 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:41.508 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.812674+0000", 2020-06-12T02:38:41.508 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1423999999999998e-05 2020-06-12T02:38:41.508 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.509 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.509 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:41.509 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.812756+0000", 2020-06-12T02:38:41.509 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024022136999999999 2020-06-12T02:38:41.509 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:41.510 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:41.510 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:41.510 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.510 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.510 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11995 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e56)", 2020-06-12T02:38:41.510 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:23.789128+0000", 2020-06-12T02:38:41.511 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 17.517452338999998, 2020-06-12T02:38:41.511 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031795508, 2020-06-12T02:38:41.511 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:41.511 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:41.511 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:41.512 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:41.512 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:41.512 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11995 2020-06-12T02:38:41.512 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.512 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:41.512 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.513 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:41.513 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789128+0000", 2020-06-12T02:38:41.513 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:41.513 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.514 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.514 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:41.514 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789128+0000", 2020-06-12T02:38:41.515 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4419999999999998e-05 2020-06-12T02:38:41.515 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.515 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.515 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:41.516 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789172+0000", 2020-06-12T02:38:41.516 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7110000000000001e-06 2020-06-12T02:38:41.516 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.516 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.517 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:41.517 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789176+0000", 2020-06-12T02:38:41.517 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5089999999999999e-06 2020-06-12T02:38:41.517 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.517 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.517 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:41.518 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789183+0000", 2020-06-12T02:38:41.518 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.497e-06 2020-06-12T02:38:41.518 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.518 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.518 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:41.519 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789185+0000", 2020-06-12T02:38:41.519 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012172176 2020-06-12T02:38:41.519 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.519 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.519 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.519 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.801357+0000", 2020-06-12T02:38:41.520 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4264000000000006e-05 2020-06-12T02:38:41.520 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.520 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.520 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:41.520 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.801432+0000", 2020-06-12T02:38:41.521 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00029680499999999997 2020-06-12T02:38:41.521 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.521 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.521 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:41.521 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.801728+0000", 2020-06-12T02:38:41.522 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00066278300000000001 2020-06-12T02:38:41.522 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.522 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.522 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:41.522 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.802391+0000", 2020-06-12T02:38:41.522 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018429867999999999 2020-06-12T02:38:41.523 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.523 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.523 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:41.523 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.820821+0000", 2020-06-12T02:38:41.523 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000102475 2020-06-12T02:38:41.524 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.524 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.524 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:41.524 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.820924+0000", 2020-06-12T02:38:41.524 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031795508 2020-06-12T02:38:41.524 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:41.525 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:41.525 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:41.525 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.525 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.525 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11996 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e56)", 2020-06-12T02:38:41.526 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:23.789235+0000", 2020-06-12T02:38:41.526 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 17.517345488, 2020-06-12T02:38:41.526 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.034785745, 2020-06-12T02:38:41.527 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:41.527 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:41.527 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:41.527 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:41.527 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:41.527 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11996 2020-06-12T02:38:41.528 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.528 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:41.528 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.528 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:41.528 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789235+0000", 2020-06-12T02:38:41.529 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:41.529 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.529 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.529 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:41.529 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789235+0000", 2020-06-12T02:38:41.530 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6375000000000001e-05 2020-06-12T02:38:41.530 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.530 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.530 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:41.530 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789261+0000", 2020-06-12T02:38:41.531 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2679999999999999e-06 2020-06-12T02:38:41.531 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.531 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.531 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:41.531 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789265+0000", 2020-06-12T02:38:41.531 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6039999999999998e-06 2020-06-12T02:38:41.532 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.532 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.532 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:41.532 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789271+0000", 2020-06-12T02:38:41.532 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4219999999999999e-06 2020-06-12T02:38:41.533 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.533 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.533 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:41.533 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789274+0000", 2020-06-12T02:38:41.534 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013034463 2020-06-12T02:38:41.534 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.534 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.534 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.534 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.802308+0000", 2020-06-12T02:38:41.535 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8862000000000005e-05 2020-06-12T02:38:41.535 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.535 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.535 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:41.535 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.802387+0000", 2020-06-12T02:38:41.535 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019102297000000001 2020-06-12T02:38:41.536 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.536 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.536 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.537 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.821489+0000", 2020-06-12T02:38:41.537 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8838999999999996e-05 2020-06-12T02:38:41.537 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.537 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.537 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:41.538 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.821558+0000", 2020-06-12T02:38:41.538 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002462615 2020-06-12T02:38:41.538 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.538 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.538 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:41.538 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.824021+0000", 2020-06-12T02:38:41.539 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.034785745 2020-06-12T02:38:41.539 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:41.539 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:41.539 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:41.539 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.540 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.540 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:15005 3.ds0 3:b3445631:::smithi10729407-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e59)", 2020-06-12T02:38:41.540 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:32.192302+0000", 2020-06-12T02:38:41.540 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.1142782530000002, 2020-06-12T02:38:41.540 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024128666, 2020-06-12T02:38:41.540 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:41.541 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:41.541 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:41.541 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:41.549 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:41.550 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 15005 2020-06-12T02:38:41.550 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.550 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:41.550 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.550 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:41.551 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192302+0000", 2020-06-12T02:38:41.551 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:41.551 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.551 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.551 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:41.552 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192302+0000", 2020-06-12T02:38:41.552 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3415999999999999e-05 2020-06-12T02:38:41.552 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.552 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.552 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:41.552 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192346+0000", 2020-06-12T02:38:41.553 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.93e-06 2020-06-12T02:38:41.553 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.553 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.553 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:41.553 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192352+0000", 2020-06-12T02:38:41.554 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3397000000000001e-05 2020-06-12T02:38:41.554 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.554 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.554 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:41.554 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192365+0000", 2020-06-12T02:38:41.555 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3950000000000003e-06 2020-06-12T02:38:41.555 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.555 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.555 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:41.555 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192369+0000", 2020-06-12T02:38:41.555 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010208887999999999 2020-06-12T02:38:41.556 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.556 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.556 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.556 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.202578+0000", 2020-06-12T02:38:41.556 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000105611 2020-06-12T02:38:41.557 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.557 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.557 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:41.558 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.202684+0000", 2020-06-12T02:38:41.558 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0091288959999999992 2020-06-12T02:38:41.558 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.558 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.558 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.558 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.211813+0000", 2020-06-12T02:38:41.559 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000101798 2020-06-12T02:38:41.559 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.559 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.559 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:41.559 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.211914+0000", 2020-06-12T02:38:41.560 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045163349999999998 2020-06-12T02:38:41.560 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.560 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.560 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:41.560 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.216431+0000", 2020-06-12T02:38:41.561 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024128666 2020-06-12T02:38:41.561 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:41.561 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:41.561 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:41.561 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.561 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.562 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:16425 3.ds0 3:b3445631:::smithi10729407-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e61)", 2020-06-12T02:38:41.562 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:36.531070+0000", 2020-06-12T02:38:41.562 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.7755105279999999, 2020-06-12T02:38:41.562 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.02029912, 2020-06-12T02:38:41.562 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:41.563 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:41.563 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:41.563 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:41.563 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:41.563 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 16425 2020-06-12T02:38:41.563 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.564 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:41.564 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.564 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:41.564 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:36.531070+0000", 2020-06-12T02:38:41.564 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:41.565 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.565 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.565 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:41.565 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:36.531070+0000", 2020-06-12T02:38:41.565 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3014999999999999e-05 2020-06-12T02:38:41.565 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.566 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.566 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:41.566 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:36.531093+0000", 2020-06-12T02:38:41.566 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0630000000000002e-06 2020-06-12T02:38:41.566 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.567 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.567 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:41.567 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:36.531096+0000", 2020-06-12T02:38:41.568 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1829999999999999e-06 2020-06-12T02:38:41.568 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.568 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.568 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:41.568 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:36.531102+0000", 2020-06-12T02:38:41.569 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2749999999999998e-06 2020-06-12T02:38:41.569 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.569 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.569 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:41.569 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:36.531104+0000", 2020-06-12T02:38:41.570 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0099790060000000003 2020-06-12T02:38:41.570 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.570 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.570 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.571 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:36.541083+0000", 2020-06-12T02:38:41.571 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7460000000000005e-05 2020-06-12T02:38:41.571 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.571 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.571 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:41.571 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:36.541181+0000", 2020-06-12T02:38:41.572 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0084066269999999998 2020-06-12T02:38:41.572 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.572 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.572 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.572 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:36.549588+0000", 2020-06-12T02:38:41.573 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6845999999999997e-05 2020-06-12T02:38:41.573 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.573 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.573 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:41.573 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:36.549644+0000", 2020-06-12T02:38:41.574 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0017246449999999999 2020-06-12T02:38:41.574 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.574 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.574 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:41.574 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:36.551369+0000", 2020-06-12T02:38:41.574 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.02029912 2020-06-12T02:38:41.575 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:41.575 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:41.575 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:41.575 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.575 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.576 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:16638 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e61)", 2020-06-12T02:38:41.576 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:37.125803+0000", 2020-06-12T02:38:41.576 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.1807774990000004, 2020-06-12T02:38:41.576 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020796752000000002, 2020-06-12T02:38:41.576 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:41.576 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:41.577 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:41.577 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:41.577 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:41.578 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 16638 2020-06-12T02:38:41.578 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.578 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:41.578 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.578 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:41.579 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.125803+0000", 2020-06-12T02:38:41.579 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:41.579 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.579 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.579 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:41.579 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.125803+0000", 2020-06-12T02:38:41.580 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3255999999999999e-05 2020-06-12T02:38:41.580 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.580 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.580 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:41.580 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.125826+0000", 2020-06-12T02:38:41.581 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2380000000000002e-06 2020-06-12T02:38:41.581 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.581 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.581 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:41.581 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.125829+0000", 2020-06-12T02:38:41.582 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2090000000000002e-06 2020-06-12T02:38:41.582 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.582 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.582 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:41.582 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.125836+0000", 2020-06-12T02:38:41.582 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3029999999999998e-06 2020-06-12T02:38:41.583 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.583 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.583 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:41.583 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.125838+0000", 2020-06-12T02:38:41.583 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0075006819999999998 2020-06-12T02:38:41.584 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.584 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.584 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.584 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.133339+0000", 2020-06-12T02:38:41.584 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.224e-05 2020-06-12T02:38:41.584 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.585 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.585 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:41.585 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.133401+0000", 2020-06-12T02:38:41.585 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0055524069999999997 2020-06-12T02:38:41.585 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.586 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.586 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.586 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.138953+0000", 2020-06-12T02:38:41.586 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000111172 2020-06-12T02:38:41.586 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.587 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.587 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:41.587 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.139064+0000", 2020-06-12T02:38:41.587 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0075352450000000003 2020-06-12T02:38:41.587 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.587 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.588 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:41.588 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.146600+0000", 2020-06-12T02:38:41.588 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020796752000000002 2020-06-12T02:38:41.589 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:41.589 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:41.589 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:41.589 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.589 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.590 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:16810 3.ds0 3:b983fd28:::smithi10729407-1:head [append 0~524288] snapc 0=[] ondisk+write+known_if_redirected e61)", 2020-06-12T02:38:41.590 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:37.642363+0000", 2020-06-12T02:38:41.590 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.6642169990000002, 2020-06-12T02:38:41.590 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022969679, 2020-06-12T02:38:41.590 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:41.590 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:41.591 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:41.591 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:41.591 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:41.591 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 16810 2020-06-12T02:38:41.591 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.592 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:41.592 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.592 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:41.592 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.642363+0000", 2020-06-12T02:38:41.592 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:41.592 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.593 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.593 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:41.593 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.642363+0000", 2020-06-12T02:38:41.593 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00069050800000000005 2020-06-12T02:38:41.593 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.594 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.594 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:41.594 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.643054+0000", 2020-06-12T02:38:41.594 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8220000000000007e-06 2020-06-12T02:38:41.594 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.594 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.595 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:41.595 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.643062+0000", 2020-06-12T02:38:41.595 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3742e-05 2020-06-12T02:38:41.595 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.595 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.596 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:41.596 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.643075+0000", 2020-06-12T02:38:41.596 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5910000000000001e-06 2020-06-12T02:38:41.596 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.596 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.596 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:41.597 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.643082+0000", 2020-06-12T02:38:41.597 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0064097879999999996 2020-06-12T02:38:41.597 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.597 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.597 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.598 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.649492+0000", 2020-06-12T02:38:41.598 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8151999999999994e-05 2020-06-12T02:38:41.598 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.599 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.599 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:41.599 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.649560+0000", 2020-06-12T02:38:41.599 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0020294940000000002 2020-06-12T02:38:41.599 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.600 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.600 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.600 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.651590+0000", 2020-06-12T02:38:41.600 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5242999999999997e-05 2020-06-12T02:38:41.600 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.600 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.601 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:41.601 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.651675+0000", 2020-06-12T02:38:41.601 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00069163799999999995 2020-06-12T02:38:41.601 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.601 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.602 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:41.602 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.652366+0000", 2020-06-12T02:38:41.602 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012711642 2020-06-12T02:38:41.602 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.602 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.602 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:41.603 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.665078+0000", 2020-06-12T02:38:41.603 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017519200000000001 2020-06-12T02:38:41.603 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.603 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.603 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:41.604 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.665253+0000", 2020-06-12T02:38:41.604 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9867e-05 2020-06-12T02:38:41.604 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.604 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.604 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:41.604 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.665333+0000", 2020-06-12T02:38:41.605 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022969679 2020-06-12T02:38:41.605 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:41.605 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:41.605 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:41.605 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.606 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.606 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:16813 3.ds0 3:b983fd28:::smithi10729407-1:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e61)", 2020-06-12T02:38:41.606 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:37.649483+0000", 2020-06-12T02:38:41.606 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.6570974060000001, 2020-06-12T02:38:41.606 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020141480999999999, 2020-06-12T02:38:41.607 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:41.607 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:41.607 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:41.607 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:41.607 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:41.607 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 16813 2020-06-12T02:38:41.608 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.608 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:41.608 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.608 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:41.609 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.649483+0000", 2020-06-12T02:38:41.609 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:41.609 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.609 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.610 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:41.610 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.649483+0000", 2020-06-12T02:38:41.610 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2657000000000002e-05 2020-06-12T02:38:41.610 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.610 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.610 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:41.611 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.649526+0000", 2020-06-12T02:38:41.611 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3629999999999999e-06 2020-06-12T02:38:41.611 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.611 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.614 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:41.614 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.649532+0000", 2020-06-12T02:38:41.614 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0813e-05 2020-06-12T02:38:41.615 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.615 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.615 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:41.615 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.649543+0000", 2020-06-12T02:38:41.615 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3780000000000003e-06 2020-06-12T02:38:41.615 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.616 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.616 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:41.616 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.649547+0000", 2020-06-12T02:38:41.616 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00080469199999999997 2020-06-12T02:38:41.616 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.617 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.617 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.617 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.650352+0000", 2020-06-12T02:38:41.617 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.6528999999999997e-05 2020-06-12T02:38:41.617 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.618 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.618 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:41.618 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.650448+0000", 2020-06-12T02:38:41.618 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013415148 2020-06-12T02:38:41.618 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.618 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.619 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.619 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.663864+0000", 2020-06-12T02:38:41.619 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6097999999999997e-05 2020-06-12T02:38:41.620 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.620 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.620 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:41.620 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.663920+0000", 2020-06-12T02:38:41.620 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00024680500000000001 2020-06-12T02:38:41.620 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.621 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.621 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:41.621 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.664166+0000", 2020-06-12T02:38:41.621 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0017621030000000001 2020-06-12T02:38:41.621 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.622 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.622 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:41.622 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.665929+0000", 2020-06-12T02:38:41.622 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036077819999999999 2020-06-12T02:38:41.622 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.623 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.623 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:41.623 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.669536+0000", 2020-06-12T02:38:41.623 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8113000000000002e-05 2020-06-12T02:38:41.623 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.623 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.624 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:41.624 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.669624+0000", 2020-06-12T02:38:41.624 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020141480999999999 2020-06-12T02:38:41.624 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:41.624 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:41.625 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:41.625 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.625 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.625 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:16814 3.ds0 3:b983fd28:::smithi10729407-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e61)", 2020-06-12T02:38:41.625 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:37.649584+0000", 2020-06-12T02:38:41.625 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.6569959600000002, 2020-06-12T02:38:41.626 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022653303, 2020-06-12T02:38:41.626 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:41.626 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:41.626 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:41.626 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:41.627 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:41.627 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 16814 2020-06-12T02:38:41.627 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.627 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:41.627 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.627 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:41.628 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.649584+0000", 2020-06-12T02:38:41.628 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:41.628 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.628 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.628 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:41.629 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.649584+0000", 2020-06-12T02:38:41.629 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1154e-05 2020-06-12T02:38:41.629 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.630 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.630 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:41.630 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.649616+0000", 2020-06-12T02:38:41.630 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.9120000000000003e-06 2020-06-12T02:38:41.630 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.630 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.631 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:41.631 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.649622+0000", 2020-06-12T02:38:41.631 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1705999999999999e-05 2020-06-12T02:38:41.631 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.631 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.632 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:41.632 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.649633+0000", 2020-06-12T02:38:41.632 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2180000000000001e-06 2020-06-12T02:38:41.632 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.632 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.633 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:41.633 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.649637+0000", 2020-06-12T02:38:41.633 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00083813500000000005 2020-06-12T02:38:41.633 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.634 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.634 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.634 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.650476+0000", 2020-06-12T02:38:41.634 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4375000000000005e-05 2020-06-12T02:38:41.634 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.634 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.635 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:41.635 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.650550+0000", 2020-06-12T02:38:41.635 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014339413000000001 2020-06-12T02:38:41.635 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.635 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.636 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.636 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.664889+0000", 2020-06-12T02:38:41.636 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1619999999999994e-05 2020-06-12T02:38:41.636 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.636 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.637 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:41.637 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.664981+0000", 2020-06-12T02:38:41.637 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0054750429999999998 2020-06-12T02:38:41.637 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.637 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.637 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:41.638 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.670456+0000", 2020-06-12T02:38:41.638 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0585000000000001e-05 2020-06-12T02:38:41.638 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.638 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.638 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:41.639 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.670547+0000", 2020-06-12T02:38:41.639 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0016911420000000001 2020-06-12T02:38:41.639 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:41.639 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:41.640 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:41.640 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.672238+0000", 2020-06-12T02:38:41.640 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022653303 2020-06-12T02:38:41.640 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:41.640 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:41.641 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:41.641 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:41.641 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:41.641 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:41.642 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd unset noscrub 2020-06-12T02:38:41.662 INFO:tasks.rados.rados.0.smithi107.stdout:8447: writing smithi10729407-20 from 1277952 to 1900544 tid 3 2020-06-12T02:38:41.662 INFO:tasks.rados.rados.0.smithi107.stdout:8447: writing smithi10729407-20 from 1900544 to 2269184 tid 4 2020-06-12T02:38:41.663 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:41.663 INFO:tasks.rados.rados.0.smithi107.stdout:8448: seq_num 8447 ranges {0=679936,679936=548864,1228800=434176,1662976=598016,2260992=24576} 2020-06-12T02:38:41.663 INFO:tasks.rados.rados.0.smithi107.stdout:8448: writing smithi10729407-15 from 0 to 679936 tid 1 2020-06-12T02:38:41.663 INFO:tasks.rados.rados.0.smithi107.stdout:8448: writing smithi10729407-15 from 679936 to 1228800 tid 2 2020-06-12T02:38:41.663 INFO:tasks.rados.rados.0.smithi107.stdout:8448: writing smithi10729407-15 from 1228800 to 1662976 tid 3 2020-06-12T02:38:41.664 INFO:tasks.rados.rados.0.smithi107.stdout:8448: writing smithi10729407-15 from 1662976 to 2260992 tid 4 2020-06-12T02:38:41.664 INFO:tasks.rados.rados.0.smithi107.stdout:8448: writing smithi10729407-15 from 2260992 to 2285568 tid 5 2020-06-12T02:38:41.664 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:41.664 INFO:tasks.rados.rados.0.smithi107.stdout:8449: seq_num 8448 ranges {0=696320,696320=434176,1130496=638976,1769472=483328,2252800=49152} 2020-06-12T02:38:41.664 INFO:tasks.rados.rados.0.smithi107.stdout:8449: writing smithi10729407-8 from 0 to 696320 tid 1 2020-06-12T02:38:41.665 INFO:tasks.rados.rados.0.smithi107.stdout:8449: writing smithi10729407-8 from 696320 to 1130496 tid 2 2020-06-12T02:38:41.665 INFO:tasks.rados.rados.0.smithi107.stdout:8449: writing smithi10729407-8 from 1130496 to 1769472 tid 3 2020-06-12T02:38:41.665 INFO:tasks.rados.rados.0.smithi107.stdout:8449: writing smithi10729407-8 from 1769472 to 2252800 tid 4 2020-06-12T02:38:41.665 INFO:tasks.rados.rados.0.smithi107.stdout:8449: writing smithi10729407-8 from 2252800 to 2301952 tid 5 2020-06-12T02:38:41.665 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:41.665 INFO:tasks.rados.rados.0.smithi107.stdout:8450: seq_num 8449 ranges {0=712704,712704=720896,1433600=442368,1875968=425984,2301952=16384} 2020-06-12T02:38:41.666 INFO:tasks.rados.rados.0.smithi107.stdout:8450: writing smithi10729407-9 from 0 to 712704 tid 1 2020-06-12T02:38:41.666 INFO:tasks.rados.rados.0.smithi107.stdout:8450: writing smithi10729407-9 from 712704 to 1433600 tid 2 2020-06-12T02:38:41.666 INFO:tasks.rados.rados.0.smithi107.stdout:8450: writing smithi10729407-9 from 1433600 to 1875968 tid 3 2020-06-12T02:38:41.666 INFO:tasks.rados.rados.0.smithi107.stdout:8450: writing smithi10729407-9 from 1875968 to 2301952 tid 4 2020-06-12T02:38:41.666 INFO:tasks.rados.rados.0.smithi107.stdout:8450: writing smithi10729407-9 from 2301952 to 2318336 tid 5 2020-06-12T02:38:41.667 INFO:tasks.rados.rados.0.smithi107.stdout:8451: delete oid 13 current snap is 0 2020-06-12T02:38:41.667 INFO:tasks.rados.rados.0.smithi107.stdout:8443: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:41.667 INFO:tasks.rados.rados.0.smithi107.stdout:8443: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:41.667 INFO:tasks.rados.rados.0.smithi107.stdout:8443: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:41.667 INFO:tasks.rados.rados.0.smithi107.stdout:8443: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:41.668 INFO:tasks.rados.rados.0.smithi107.stdout:8443: finishing write tid 7 to smithi10729407-12 2020-06-12T02:38:41.668 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1559 (ObjNum 8442 snap 0 seq_num 8442) dirty exists 2020-06-12T02:38:41.668 INFO:tasks.rados.rados.0.smithi107.stdout:8443: left oid 12 (ObjNum 8442 snap 0 seq_num 8442) 2020-06-12T02:38:41.668 INFO:tasks.rados.rados.0.smithi107.stdout:8447: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:41.668 INFO:tasks.rados.rados.0.smithi107.stdout:8447: finishing write tid 2 to smithi10729407-20 2020-06-12T02:38:41.669 INFO:tasks.rados.rados.0.smithi107.stdout:8447: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:41.669 INFO:tasks.rados.rados.0.smithi107.stdout:8447: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:41.669 INFO:tasks.rados.rados.0.smithi107.stdout:8448: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:41.669 INFO:tasks.rados.rados.0.smithi107.stdout:8447: finishing write tid 6 to smithi10729407-20 2020-06-12T02:38:41.669 INFO:tasks.rados.rados.0.smithi107.stdout:8447: finishing write tid 7 to smithi10729407-20 2020-06-12T02:38:41.670 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 625 (ObjNum 8446 snap 0 seq_num 8446) dirty exists 2020-06-12T02:38:41.670 INFO:tasks.rados.rados.0.smithi107.stdout:8447: left oid 20 (ObjNum 8446 snap 0 seq_num 8446) 2020-06-12T02:38:41.670 INFO:tasks.rados.rados.0.smithi107.stdout:8448: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:41.671 INFO:tasks.rados.rados.0.smithi107.stdout:8448: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:41.671 INFO:tasks.rados.rados.0.smithi107.stdout:8448: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:41.671 INFO:tasks.rados.rados.0.smithi107.stdout:8448: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:41.671 INFO:tasks.rados.rados.0.smithi107.stdout:8448: finishing write tid 7 to smithi10729407-15 2020-06-12T02:38:41.672 INFO:tasks.rados.rados.0.smithi107.stdout:8449: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:41.672 INFO:tasks.rados.rados.0.smithi107.stdout:8450: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:41.672 INFO:tasks.rados.rados.0.smithi107.stdout:8448: finishing write tid 8 to smithi10729407-15 2020-06-12T02:38:41.672 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1572 (ObjNum 8447 snap 0 seq_num 8447) dirty exists 2020-06-12T02:38:41.672 INFO:tasks.rados.rados.0.smithi107.stdout:8448: left oid 15 (ObjNum 8447 snap 0 seq_num 8447) 2020-06-12T02:38:41.673 INFO:tasks.rados.rados.0.smithi107.stdout:8443: done (5 left) 2020-06-12T02:38:41.673 INFO:tasks.rados.rados.0.smithi107.stdout:8447: done (4 left) 2020-06-12T02:38:41.673 INFO:tasks.rados.rados.0.smithi107.stdout:8448: done (3 left) 2020-06-12T02:38:41.673 INFO:tasks.rados.rados.0.smithi107.stdout:8451: done (2 left) 2020-06-12T02:38:41.673 INFO:tasks.rados.rados.0.smithi107.stdout:8452: delete oid 13 current snap is 0 2020-06-12T02:38:41.673 INFO:tasks.rados.rados.0.smithi107.stdout:8449: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:41.674 INFO:tasks.rados.rados.0.smithi107.stdout:8452: done (2 left) 2020-06-12T02:38:41.674 INFO:tasks.rados.rados.0.smithi107.stdout:8453: delete oid 11 current snap is 0 2020-06-12T02:38:41.674 INFO:tasks.rados.rados.0.smithi107.stdout:8450: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:41.674 INFO:tasks.rados.rados.0.smithi107.stdout:8450: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:41.674 INFO:tasks.rados.rados.0.smithi107.stdout:8453: done (2 left) 2020-06-12T02:38:41.675 INFO:tasks.rados.rados.0.smithi107.stdout:8454: delete oid 5 current snap is 0 2020-06-12T02:38:41.675 INFO:tasks.rados.rados.0.smithi107.stdout:8449: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:41.675 INFO:tasks.rados.rados.0.smithi107.stdout:8450: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:41.675 INFO:tasks.rados.rados.0.smithi107.stdout:8450: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:41.675 INFO:tasks.rados.rados.0.smithi107.stdout:8449: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:41.676 INFO:tasks.rados.rados.0.smithi107.stdout:8449: finishing write tid 5 to smithi10729407-8 2020-06-12T02:38:41.676 INFO:tasks.rados.rados.0.smithi107.stdout:8449: finishing write tid 7 to smithi10729407-8 2020-06-12T02:38:41.676 INFO:tasks.rados.rados.0.smithi107.stdout:8450: finishing write tid 7 to smithi10729407-9 2020-06-12T02:38:41.676 INFO:tasks.rados.rados.0.smithi107.stdout:8454: done (2 left) 2020-06-12T02:38:41.676 INFO:tasks.rados.rados.0.smithi107.stdout:8455: delete oid 18 current snap is 0 2020-06-12T02:38:41.677 INFO:tasks.rados.rados.0.smithi107.stdout:8449: finishing write tid 8 to smithi10729407-8 2020-06-12T02:38:41.677 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1578 (ObjNum 8448 snap 0 seq_num 8448) dirty exists 2020-06-12T02:38:41.677 INFO:tasks.rados.rados.0.smithi107.stdout:8449: left oid 8 (ObjNum 8448 snap 0 seq_num 8448) 2020-06-12T02:38:41.677 INFO:tasks.rados.rados.0.smithi107.stdout:8450: finishing write tid 8 to smithi10729407-9 2020-06-12T02:38:41.677 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1576 (ObjNum 8449 snap 0 seq_num 8449) dirty exists 2020-06-12T02:38:41.678 INFO:tasks.rados.rados.0.smithi107.stdout:8450: left oid 9 (ObjNum 8449 snap 0 seq_num 8449) 2020-06-12T02:38:41.678 INFO:tasks.rados.rados.0.smithi107.stdout:8449: done (2 left) 2020-06-12T02:38:41.678 INFO:tasks.rados.rados.0.smithi107.stdout:8450: done (1 left) 2020-06-12T02:38:41.678 INFO:tasks.rados.rados.0.smithi107.stdout:8455: done (0 left) 2020-06-12T02:38:41.678 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:41.678 INFO:tasks.rados.rados.0.smithi107.stdout:8456: seq_num 8455 ranges {0=8192} 2020-06-12T02:38:41.679 INFO:tasks.rados.rados.0.smithi107.stdout:8456: writing smithi10729407-7 from 0 to 8192 tid 1 2020-06-12T02:38:41.679 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:41.679 INFO:tasks.rados.rados.0.smithi107.stdout:8457: seq_num 8456 ranges {0=24576} 2020-06-12T02:38:41.679 INFO:tasks.rados.rados.0.smithi107.stdout:8457: writing smithi10729407-18 from 0 to 24576 tid 1 2020-06-12T02:38:41.679 INFO:tasks.rados.rados.0.smithi107.stdout:8458: delete oid 6 current snap is 0 2020-06-12T02:38:41.680 INFO:tasks.rados.rados.0.smithi107.stdout:8457: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:41.680 INFO:tasks.rados.rados.0.smithi107.stdout:8456: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:41.680 INFO:tasks.rados.rados.0.smithi107.stdout:8458: done (2 left) 2020-06-12T02:38:41.680 INFO:tasks.rados.rados.0.smithi107.stdout:8459: delete oid 13 current snap is 0 2020-06-12T02:38:41.680 INFO:tasks.rados.rados.0.smithi107.stdout:8456: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:41.680 INFO:tasks.rados.rados.0.smithi107.stdout:8456: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:41.681 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1533 (ObjNum 8455 snap 0 seq_num 8455) dirty exists 2020-06-12T02:38:41.681 INFO:tasks.rados.rados.0.smithi107.stdout:8456: left oid 7 (ObjNum 8455 snap 0 seq_num 8455) 2020-06-12T02:38:41.681 INFO:tasks.rados.rados.0.smithi107.stdout:8457: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:41.682 INFO:tasks.rados.rados.0.smithi107.stdout:8457: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:41.682 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1579 (ObjNum 8456 snap 0 seq_num 8456) dirty exists 2020-06-12T02:38:41.682 INFO:tasks.rados.rados.0.smithi107.stdout:8457: left oid 18 (ObjNum 8456 snap 0 seq_num 8456) 2020-06-12T02:38:41.682 INFO:tasks.rados.rados.0.smithi107.stdout:8456: done (2 left) 2020-06-12T02:38:41.682 INFO:tasks.rados.rados.0.smithi107.stdout:8457: done (1 left) 2020-06-12T02:38:41.683 INFO:tasks.rados.rados.0.smithi107.stdout:8459: done (0 left) 2020-06-12T02:38:41.683 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:38:41.683 INFO:tasks.rados.rados.0.smithi107.stdout:8460: seq_num 8459 ranges {2269184=73728} 2020-06-12T02:38:41.683 INFO:tasks.rados.rados.0.smithi107.stdout:8460: writing smithi10729407-20 from 2269184 to 2342912 tid 1 2020-06-12T02:38:41.683 INFO:tasks.rados.rados.0.smithi107.stdout:8461: delete oid 1 current snap is 0 2020-06-12T02:38:41.684 INFO:tasks.rados.rados.0.smithi107.stdout:8460: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:41.684 INFO:tasks.rados.rados.0.smithi107.stdout:8461: done (1 left) 2020-06-12T02:38:41.684 INFO:tasks.rados.rados.0.smithi107.stdout:8462: delete oid 8 current snap is 0 2020-06-12T02:38:41.684 INFO:tasks.rados.rados.0.smithi107.stdout:8460: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:41.684 INFO:tasks.rados.rados.0.smithi107.stdout:8460: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:41.685 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 627 (ObjNum 8459 snap 0 seq_num 8459) dirty exists 2020-06-12T02:38:41.685 INFO:tasks.rados.rados.0.smithi107.stdout:8460: left oid 20 (ObjNum 8459 snap 0 seq_num 8459) 2020-06-12T02:38:41.685 INFO:tasks.rados.rados.0.smithi107.stdout:8460: done (1 left) 2020-06-12T02:38:41.685 INFO:tasks.rados.rados.0.smithi107.stdout:8462: done (0 left) 2020-06-12T02:38:41.685 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:38:41.686 INFO:tasks.rados.rados.0.smithi107.stdout:8463: seq_num 8462 ranges {0=122880} 2020-06-12T02:38:41.686 INFO:tasks.rados.rados.0.smithi107.stdout:8463: writing smithi10729407-6 from 0 to 122880 tid 1 2020-06-12T02:38:41.686 INFO:tasks.rados.rados.0.smithi107.stdout:8464: delete oid 15 current snap is 0 2020-06-12T02:38:41.686 INFO:tasks.rados.rados.0.smithi107.stdout:8464: done (1 left) 2020-06-12T02:38:41.686 INFO:tasks.rados.rados.0.smithi107.stdout:8465: delete oid 17 current snap is 0 2020-06-12T02:38:41.687 INFO:tasks.rados.rados.0.smithi107.stdout:8463: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:41.687 INFO:tasks.rados.rados.0.smithi107.stdout:8463: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:41.687 INFO:tasks.rados.rados.0.smithi107.stdout:8465: done (1 left) 2020-06-12T02:38:41.687 INFO:tasks.rados.rados.0.smithi107.stdout:8466: delete oid 12 current snap is 0 2020-06-12T02:38:41.687 INFO:tasks.rados.rados.0.smithi107.stdout:8463: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:41.688 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1533 (ObjNum 8462 snap 0 seq_num 8462) dirty exists 2020-06-12T02:38:41.688 INFO:tasks.rados.rados.0.smithi107.stdout:8463: left oid 6 (ObjNum 8462 snap 0 seq_num 8462) 2020-06-12T02:38:41.688 INFO:tasks.rados.rados.0.smithi107.stdout:8463: done (1 left) 2020-06-12T02:38:41.688 INFO:tasks.rados.rados.0.smithi107.stdout:8466: done (0 left) 2020-06-12T02:38:41.688 INFO:tasks.rados.rados.0.smithi107.stdout:8467: delete oid 13 current snap is 0 2020-06-12T02:38:41.688 INFO:tasks.rados.rados.0.smithi107.stdout:8467: done (0 left) 2020-06-12T02:38:41.689 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:41.689 INFO:tasks.rados.rados.0.smithi107.stdout:8468: seq_num 8467 ranges {0=212992} 2020-06-12T02:38:41.689 INFO:tasks.rados.rados.0.smithi107.stdout:8468: writing smithi10729407-8 from 0 to 212992 tid 1 2020-06-12T02:38:41.689 INFO:tasks.rados.rados.0.smithi107.stdout:8469: delete oid 15 current snap is 0 2020-06-12T02:38:41.689 INFO:tasks.rados.rados.0.smithi107.stdout:8468: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:41.690 INFO:tasks.rados.rados.0.smithi107.stdout:8469: done (1 left) 2020-06-12T02:38:41.690 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:41.690 INFO:tasks.rados.rados.0.smithi107.stdout:8470: seq_num 8469 ranges {0=245760} 2020-06-12T02:38:41.690 INFO:tasks.rados.rados.0.smithi107.stdout:8470: writing smithi10729407-14 from 0 to 245760 tid 1 2020-06-12T02:38:41.690 INFO:tasks.rados.rados.0.smithi107.stdout:8471: delete oid 13 current snap is 0 2020-06-12T02:38:41.690 INFO:tasks.rados.rados.0.smithi107.stdout:8468: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:41.691 INFO:tasks.rados.rados.0.smithi107.stdout:8468: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:41.691 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1583 (ObjNum 8467 snap 0 seq_num 8467) dirty exists 2020-06-12T02:38:41.691 INFO:tasks.rados.rados.0.smithi107.stdout:8468: left oid 8 (ObjNum 8467 snap 0 seq_num 8467) 2020-06-12T02:38:41.692 INFO:tasks.rados.rados.0.smithi107.stdout:8468: done (2 left) 2020-06-12T02:38:41.692 INFO:tasks.rados.rados.0.smithi107.stdout:8471: done (1 left) 2020-06-12T02:38:41.692 INFO:tasks.rados.rados.0.smithi107.stdout:8472: delete oid 4 current snap is 0 2020-06-12T02:38:41.692 INFO:tasks.rados.rados.0.smithi107.stdout:8472: done (1 left) 2020-06-12T02:38:41.692 INFO:tasks.rados.rados.0.smithi107.stdout:8473: delete oid 7 current snap is 0 2020-06-12T02:38:41.693 INFO:tasks.rados.rados.0.smithi107.stdout:8470: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:41.693 INFO:tasks.rados.rados.0.smithi107.stdout:8473: done (1 left) 2020-06-12T02:38:41.693 INFO:tasks.rados.rados.0.smithi107.stdout:8474: delete oid 1 current snap is 0 2020-06-12T02:38:41.693 INFO:tasks.rados.rados.0.smithi107.stdout:8470: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:41.693 INFO:tasks.rados.rados.0.smithi107.stdout:8470: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:41.694 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1489 (ObjNum 8469 snap 0 seq_num 8469) dirty exists 2020-06-12T02:38:41.694 INFO:tasks.rados.rados.0.smithi107.stdout:8470: left oid 14 (ObjNum 8469 snap 0 seq_num 8469) 2020-06-12T02:38:41.694 INFO:tasks.rados.rados.0.smithi107.stdout:8470: done (1 left) 2020-06-12T02:38:41.694 INFO:tasks.rados.rados.0.smithi107.stdout:8474: done (0 left) 2020-06-12T02:38:41.694 INFO:tasks.rados.rados.0.smithi107.stdout:8475: delete oid 9 current snap is 0 2020-06-12T02:38:41.694 INFO:tasks.rados.rados.0.smithi107.stdout:8475: done (0 left) 2020-06-12T02:38:41.695 INFO:tasks.rados.rados.0.smithi107.stdout:8476: delete oid 2 current snap is 0 2020-06-12T02:38:41.695 INFO:tasks.rados.rados.0.smithi107.stdout:8476: done (0 left) 2020-06-12T02:38:41.695 INFO:tasks.rados.rados.0.smithi107.stdout:8477: delete oid 9 current snap is 0 2020-06-12T02:38:41.695 INFO:tasks.rados.rados.0.smithi107.stdout:8477: done (0 left) 2020-06-12T02:38:41.695 INFO:tasks.rados.rados.0.smithi107.stdout:8478: delete oid 12 current snap is 0 2020-06-12T02:38:41.696 INFO:tasks.rados.rados.0.smithi107.stdout:8478: done (0 left) 2020-06-12T02:38:41.696 INFO:tasks.rados.rados.0.smithi107.stdout:8479: delete oid 3 current snap is 0 2020-06-12T02:38:41.696 INFO:tasks.rados.rados.0.smithi107.stdout:8479: done (0 left) 2020-06-12T02:38:41.696 INFO:tasks.rados.rados.0.smithi107.stdout:8480: delete oid 16 current snap is 0 2020-06-12T02:38:41.696 INFO:tasks.rados.rados.0.smithi107.stdout:8480: done (0 left) 2020-06-12T02:38:41.696 INFO:tasks.rados.rados.0.smithi107.stdout:8481: delete oid 13 current snap is 0 2020-06-12T02:38:41.697 INFO:tasks.rados.rados.0.smithi107.stdout:8481: done (0 left) 2020-06-12T02:38:41.697 INFO:tasks.rados.rados.0.smithi107.stdout:8482: delete oid 17 current snap is 0 2020-06-12T02:38:41.697 INFO:tasks.rados.rados.0.smithi107.stdout:8482: done (0 left) 2020-06-12T02:38:41.697 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:38:41.697 INFO:tasks.rados.rados.0.smithi107.stdout:8483: seq_num 8482 ranges {122880=458752} 2020-06-12T02:38:41.698 INFO:tasks.rados.rados.0.smithi107.stdout:8483: writing smithi10729407-6 from 122880 to 581632 tid 1 2020-06-12T02:38:41.698 INFO:tasks.rados.rados.0.smithi107.stdout:8484: delete oid 16 current snap is 0 2020-06-12T02:38:41.698 INFO:tasks.rados.rados.0.smithi107.stdout:8484: done (1 left) 2020-06-12T02:38:41.698 INFO:tasks.rados.rados.0.smithi107.stdout:8485: delete oid 11 current snap is 0 2020-06-12T02:38:41.698 INFO:tasks.rados.rados.0.smithi107.stdout:8485: done (1 left) 2020-06-12T02:38:41.698 INFO:tasks.rados.rados.0.smithi107.stdout:8486: delete oid 4 current snap is 0 2020-06-12T02:38:41.699 INFO:tasks.rados.rados.0.smithi107.stdout:8483: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:41.699 INFO:tasks.rados.rados.0.smithi107.stdout:8483: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:41.699 INFO:tasks.rados.rados.0.smithi107.stdout:8486: done (1 left) 2020-06-12T02:38:41.699 INFO:tasks.rados.rados.0.smithi107.stdout:8487: delete oid 11 current snap is 0 2020-06-12T02:38:41.699 INFO:tasks.rados.rados.0.smithi107.stdout:8483: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:41.700 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1536 (ObjNum 8482 snap 0 seq_num 8482) dirty exists 2020-06-12T02:38:41.700 INFO:tasks.rados.rados.0.smithi107.stdout:8483: left oid 6 (ObjNum 8482 snap 0 seq_num 8482) 2020-06-12T02:38:41.700 INFO:tasks.rados.rados.0.smithi107.stdout:8483: done (1 left) 2020-06-12T02:38:41.700 INFO:tasks.rados.rados.0.smithi107.stdout:8487: done (0 left) 2020-06-12T02:38:41.700 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:41.701 INFO:tasks.rados.rados.0.smithi107.stdout:8488: seq_num 8487 ranges {24576=548864} 2020-06-12T02:38:41.701 INFO:tasks.rados.rados.0.smithi107.stdout:8488: writing smithi10729407-18 from 24576 to 573440 tid 1 2020-06-12T02:38:41.701 INFO:tasks.rados.rados.0.smithi107.stdout:8489: delete oid 15 current snap is 0 2020-06-12T02:38:41.701 INFO:tasks.rados.rados.0.smithi107.stdout:8489: done (1 left) 2020-06-12T02:38:41.701 INFO:tasks.rados.rados.0.smithi107.stdout:8490: delete oid 16 current snap is 0 2020-06-12T02:38:41.702 INFO:tasks.rados.rados.0.smithi107.stdout:8490: done (1 left) 2020-06-12T02:38:41.702 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:41.702 INFO:tasks.rados.rados.0.smithi107.stdout:8491: seq_num 8490 ranges {0=598016} 2020-06-12T02:38:41.702 INFO:tasks.rados.rados.0.smithi107.stdout:8491: writing smithi10729407-7 from 0 to 598016 tid 1 2020-06-12T02:38:41.703 INFO:tasks.rados.rados.0.smithi107.stdout:8492: delete oid 13 current snap is 0 2020-06-12T02:38:41.703 INFO:tasks.rados.rados.0.smithi107.stdout:8488: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:41.703 INFO:tasks.rados.rados.0.smithi107.stdout:8488: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:41.703 INFO:tasks.rados.rados.0.smithi107.stdout:8488: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:41.703 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1583 (ObjNum 8487 snap 0 seq_num 8487) dirty exists 2020-06-12T02:38:41.704 INFO:tasks.rados.rados.0.smithi107.stdout:8488: left oid 18 (ObjNum 8487 snap 0 seq_num 8487) 2020-06-12T02:38:41.704 INFO:tasks.rados.rados.0.smithi107.stdout:8491: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:41.704 INFO:tasks.rados.rados.0.smithi107.stdout:8491: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:41.704 INFO:tasks.rados.rados.0.smithi107.stdout:8491: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:41.704 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1540 (ObjNum 8490 snap 0 seq_num 8490) dirty exists 2020-06-12T02:38:41.705 INFO:tasks.rados.rados.0.smithi107.stdout:8491: left oid 7 (ObjNum 8490 snap 0 seq_num 8490) 2020-06-12T02:38:41.705 INFO:tasks.rados.rados.0.smithi107.stdout:8488: done (2 left) 2020-06-12T02:38:41.705 INFO:tasks.rados.rados.0.smithi107.stdout:8491: done (1 left) 2020-06-12T02:38:41.705 INFO:tasks.rados.rados.0.smithi107.stdout:8492: done (0 left) 2020-06-12T02:38:41.705 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:41.705 INFO:tasks.rados.rados.0.smithi107.stdout:8493: seq_num 8492 ranges {0=630784} 2020-06-12T02:38:41.706 INFO:tasks.rados.rados.0.smithi107.stdout:8493: writing smithi10729407-12 from 0 to 630784 tid 1 2020-06-12T02:38:41.706 INFO:tasks.rados.rados.0.smithi107.stdout:8494: delete oid 6 current snap is 0 2020-06-12T02:38:41.706 INFO:tasks.rados.rados.0.smithi107.stdout:8494: done (1 left) 2020-06-12T02:38:41.706 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:41.706 INFO:tasks.rados.rados.0.smithi107.stdout:8495: seq_num 8494 ranges {0=663552} 2020-06-12T02:38:41.707 INFO:tasks.rados.rados.0.smithi107.stdout:8495: writing smithi10729407-9 from 0 to 663552 tid 1 2020-06-12T02:38:41.707 INFO:tasks.rados.rados.0.smithi107.stdout:8496: delete oid 8 current snap is 0 2020-06-12T02:38:41.707 INFO:tasks.rados.rados.0.smithi107.stdout:8493: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:41.707 INFO:tasks.rados.rados.0.smithi107.stdout:8496: done (2 left) 2020-06-12T02:38:41.707 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:38:41.707 INFO:tasks.rados.rados.0.smithi107.stdout:8497: seq_num 8496 ranges {0=696320} 2020-06-12T02:38:41.708 INFO:tasks.rados.rados.0.smithi107.stdout:8497: writing smithi10729407-1 from 0 to 696320 tid 1 2020-06-12T02:38:41.708 INFO:tasks.rados.rados.0.smithi107.stdout:8498: delete oid 11 current snap is 0 2020-06-12T02:38:41.708 INFO:tasks.rados.rados.0.smithi107.stdout:8493: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:41.708 INFO:tasks.rados.rados.0.smithi107.stdout:8495: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:41.708 INFO:tasks.rados.rados.0.smithi107.stdout:8493: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:41.709 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1563 (ObjNum 8492 snap 0 seq_num 8492) dirty exists 2020-06-12T02:38:41.709 INFO:tasks.rados.rados.0.smithi107.stdout:8493: left oid 12 (ObjNum 8492 snap 0 seq_num 8492) 2020-06-12T02:38:41.709 INFO:tasks.rados.rados.0.smithi107.stdout:8495: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:41.709 INFO:tasks.rados.rados.0.smithi107.stdout:8493: done (3 left) 2020-06-12T02:38:41.709 INFO:tasks.rados.rados.0.smithi107.stdout:8498: done (2 left) 2020-06-12T02:38:41.709 INFO:tasks.rados.rados.0.smithi107.stdout:8499: delete oid 18 current snap is 0 2020-06-12T02:38:41.710 INFO:tasks.rados.rados.0.smithi107.stdout:8495: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:41.710 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1585 (ObjNum 8494 snap 0 seq_num 8494) dirty exists 2020-06-12T02:38:41.710 INFO:tasks.rados.rados.0.smithi107.stdout:8495: left oid 9 (ObjNum 8494 snap 0 seq_num 8494) 2020-06-12T02:38:41.710 INFO:tasks.rados.rados.0.smithi107.stdout:8495: done (2 left) 2020-06-12T02:38:41.710 INFO:tasks.rados.rados.0.smithi107.stdout:8499: done (1 left) 2020-06-12T02:38:41.710 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:41.711 INFO:tasks.rados.rados.0.smithi107.stdout:8500: seq_num 8499 ranges {663552=745472} 2020-06-12T02:38:41.711 INFO:tasks.rados.rados.0.smithi107.stdout:8500: writing smithi10729407-9 from 663552 to 1409024 tid 1 2020-06-12T02:38:41.711 INFO:tasks.rados.rados.0.smithi107.stdout:8501: delete oid 10 current snap is 0 2020-06-12T02:38:41.711 INFO:tasks.rados.rados.0.smithi107.stdout:8497: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:41.711 INFO:tasks.rados.rados.0.smithi107.stdout:8497: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:41.712 INFO:tasks.rados.rados.0.smithi107.stdout:8497: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:41.712 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1539 (ObjNum 8496 snap 0 seq_num 8496) dirty exists 2020-06-12T02:38:41.712 INFO:tasks.rados.rados.0.smithi107.stdout:8497: left oid 1 (ObjNum 8496 snap 0 seq_num 8496) 2020-06-12T02:38:41.713 INFO:tasks.rados.rados.0.smithi107.stdout:8497: done (2 left) 2020-06-12T02:38:41.713 INFO:tasks.rados.rados.0.smithi107.stdout:8501: done (1 left) 2020-06-12T02:38:41.713 INFO:tasks.rados.rados.0.smithi107.stdout:8502: delete oid 2 current snap is 0 2020-06-12T02:38:41.713 INFO:tasks.rados.rados.0.smithi107.stdout:8502: done (1 left) 2020-06-12T02:38:41.713 INFO:tasks.rados.rados.0.smithi107.stdout:8503: delete oid 6 current snap is 0 2020-06-12T02:38:41.714 INFO:tasks.rados.rados.0.smithi107.stdout:8500: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:41.714 INFO:tasks.rados.rados.0.smithi107.stdout:8503: done (1 left) 2020-06-12T02:38:41.714 INFO:tasks.rados.rados.0.smithi107.stdout:8504: delete oid 5 current snap is 0 2020-06-12T02:38:41.714 INFO:tasks.rados.rados.0.smithi107.stdout:8504: done (1 left) 2020-06-12T02:38:41.714 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:38:41.714 INFO:tasks.rados.rados.0.smithi107.stdout:8505: seq_num 8504 ranges {0=434176,434176=401408} 2020-06-12T02:38:41.715 INFO:tasks.rados.rados.0.smithi107.stdout:8505: writing smithi10729407-10 from 0 to 434176 tid 1 2020-06-12T02:38:41.715 INFO:tasks.rados.rados.0.smithi107.stdout:8505: writing smithi10729407-10 from 434176 to 835584 tid 2 2020-06-12T02:38:41.715 INFO:tasks.rados.rados.0.smithi107.stdout:8506: delete oid 15 current snap is 0 2020-06-12T02:38:41.715 INFO:tasks.rados.rados.0.smithi107.stdout:8500: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:41.715 INFO:tasks.rados.rados.0.smithi107.stdout:8500: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:41.716 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1588 (ObjNum 8499 snap 0 seq_num 8499) dirty exists 2020-06-12T02:38:41.716 INFO:tasks.rados.rados.0.smithi107.stdout:8500: left oid 9 (ObjNum 8499 snap 0 seq_num 8499) 2020-06-12T02:38:41.716 INFO:tasks.rados.rados.0.smithi107.stdout:8505: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:41.716 INFO:tasks.rados.rados.0.smithi107.stdout:8500: done (2 left) 2020-06-12T02:38:41.716 INFO:tasks.rados.rados.0.smithi107.stdout:8506: done (1 left) 2020-06-12T02:38:41.716 INFO:tasks.rados.rados.0.smithi107.stdout:8507: delete oid 17 current snap is 0 2020-06-12T02:38:41.717 INFO:tasks.rados.rados.0.smithi107.stdout:8507: done (1 left) 2020-06-12T02:38:41.717 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:41.717 INFO:tasks.rados.rados.0.smithi107.stdout:8508: seq_num 8507 ranges {0=483328,483328=401408} 2020-06-12T02:38:41.717 INFO:tasks.rados.rados.0.smithi107.stdout:8508: writing smithi10729407-17 from 0 to 483328 tid 1 2020-06-12T02:38:41.717 INFO:tasks.rados.rados.0.smithi107.stdout:8508: writing smithi10729407-17 from 483328 to 884736 tid 2 2020-06-12T02:38:41.718 INFO:tasks.rados.rados.0.smithi107.stdout:8509: delete oid 6 current snap is 0 2020-06-12T02:38:41.718 INFO:tasks.rados.rados.0.smithi107.stdout:8505: finishing write tid 2 to smithi10729407-10 2020-06-12T02:38:41.718 INFO:tasks.rados.rados.0.smithi107.stdout:8505: finishing write tid 4 to smithi10729407-10 2020-06-12T02:38:41.718 INFO:tasks.rados.rados.0.smithi107.stdout:8505: finishing write tid 5 to smithi10729407-10 2020-06-12T02:38:41.718 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 620 (ObjNum 8504 snap 0 seq_num 8504) dirty exists 2020-06-12T02:38:41.718 INFO:tasks.rados.rados.0.smithi107.stdout:8505: left oid 10 (ObjNum 8504 snap 0 seq_num 8504) 2020-06-12T02:38:41.719 INFO:tasks.rados.rados.0.smithi107.stdout:8505: done (2 left) 2020-06-12T02:38:41.719 INFO:tasks.rados.rados.0.smithi107.stdout:8509: done (1 left) 2020-06-12T02:38:41.719 INFO:tasks.rados.rados.0.smithi107.stdout:8510: delete oid 6 current snap is 0 2020-06-12T02:38:41.719 INFO:tasks.rados.rados.0.smithi107.stdout:8508: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:41.719 INFO:tasks.rados.rados.0.smithi107.stdout:8510: done (1 left) 2020-06-12T02:38:41.719 INFO:tasks.rados.rados.0.smithi107.stdout:8511: delete oid 8 current snap is 0 2020-06-12T02:38:41.720 INFO:tasks.rados.rados.0.smithi107.stdout:8511: done (1 left) 2020-06-12T02:38:41.720 INFO:tasks.rados.rados.0.smithi107.stdout:8512: delete oid 9 current snap is 0 2020-06-12T02:38:41.720 INFO:tasks.rados.rados.0.smithi107.stdout:8508: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:41.720 INFO:tasks.rados.rados.0.smithi107.stdout:8508: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:41.720 INFO:tasks.rados.rados.0.smithi107.stdout:8512: done (1 left) 2020-06-12T02:38:41.721 INFO:tasks.rados.rados.0.smithi107.stdout:8513: delete oid 11 current snap is 0 2020-06-12T02:38:41.721 INFO:tasks.rados.rados.0.smithi107.stdout:8508: finishing write tid 5 to smithi10729407-17 2020-06-12T02:38:41.721 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1599 (ObjNum 8507 snap 0 seq_num 8507) dirty exists 2020-06-12T02:38:41.721 INFO:tasks.rados.rados.0.smithi107.stdout:8508: left oid 17 (ObjNum 8507 snap 0 seq_num 8507) 2020-06-12T02:38:41.721 INFO:tasks.rados.rados.0.smithi107.stdout:8508: done (1 left) 2020-06-12T02:38:41.721 INFO:tasks.rados.rados.0.smithi107.stdout:8513: done (0 left) 2020-06-12T02:38:41.722 INFO:tasks.rados.rados.0.smithi107.stdout:8514: delete oid 1 current snap is 0 2020-06-12T02:38:41.722 INFO:tasks.rados.rados.0.smithi107.stdout:8514: done (0 left) 2020-06-12T02:38:41.722 INFO:tasks.rados.rados.0.smithi107.stdout:8515: delete oid 6 current snap is 0 2020-06-12T02:38:41.723 INFO:tasks.rados.rados.0.smithi107.stdout:8515: done (0 left) 2020-06-12T02:38:41.723 INFO:tasks.rados.rados.0.smithi107.stdout:8516: delete oid 19 current snap is 0 2020-06-12T02:38:41.723 INFO:tasks.rados.rados.0.smithi107.stdout:8516: done (0 left) 2020-06-12T02:38:41.723 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:41.723 INFO:tasks.rados.rados.0.smithi107.stdout:8517: seq_num 8516 ranges {0=630784,630784=401408} 2020-06-12T02:38:41.724 INFO:tasks.rados.rados.0.smithi107.stdout:8517: writing smithi10729407-1 from 0 to 630784 tid 1 2020-06-12T02:38:41.724 INFO:tasks.rados.rados.0.smithi107.stdout:8517: writing smithi10729407-1 from 630784 to 1032192 tid 2 2020-06-12T02:38:41.724 INFO:tasks.rados.rados.0.smithi107.stdout:8518: delete oid 3 current snap is 0 2020-06-12T02:38:41.724 INFO:tasks.rados.rados.0.smithi107.stdout:8518: done (1 left) 2020-06-12T02:38:41.724 INFO:tasks.rados.rados.0.smithi107.stdout:8519: delete oid 8 current snap is 0 2020-06-12T02:38:41.724 INFO:tasks.rados.rados.0.smithi107.stdout:8517: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:41.725 INFO:tasks.rados.rados.0.smithi107.stdout:8519: done (1 left) 2020-06-12T02:38:41.725 INFO:tasks.rados.rados.0.smithi107.stdout:8520: delete oid 8 current snap is 0 2020-06-12T02:38:41.725 INFO:tasks.rados.rados.0.smithi107.stdout:8517: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:41.725 INFO:tasks.rados.rados.0.smithi107.stdout:8517: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:41.725 INFO:tasks.rados.rados.0.smithi107.stdout:8517: finishing write tid 5 to smithi10729407-1 2020-06-12T02:38:41.726 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1547 (ObjNum 8516 snap 0 seq_num 8516) dirty exists 2020-06-12T02:38:41.726 INFO:tasks.rados.rados.0.smithi107.stdout:8517: left oid 1 (ObjNum 8516 snap 0 seq_num 8516) 2020-06-12T02:38:41.726 INFO:tasks.rados.rados.0.smithi107.stdout:8517: done (1 left) 2020-06-12T02:38:41.726 INFO:tasks.rados.rados.0.smithi107.stdout:8520: done (0 left) 2020-06-12T02:38:41.726 INFO:tasks.rados.rados.0.smithi107.stdout:8521: delete oid 14 current snap is 0 2020-06-12T02:38:41.726 INFO:tasks.rados.rados.0.smithi107.stdout:8521: done (0 left) 2020-06-12T02:38:41.727 INFO:tasks.rados.rados.0.smithi107.stdout:8522: delete oid 7 current snap is 0 2020-06-12T02:38:41.727 INFO:tasks.rados.rados.0.smithi107.stdout:8522: done (0 left) 2020-06-12T02:38:41.727 INFO:tasks.rados.rados.0.smithi107.stdout:8523: delete oid 16 current snap is 0 2020-06-12T02:38:41.727 INFO:tasks.rados.rados.0.smithi107.stdout:8523: done (0 left) 2020-06-12T02:38:41.727 INFO:tasks.rados.rados.0.smithi107.stdout:8524: delete oid 12 current snap is 0 2020-06-12T02:38:41.728 INFO:tasks.rados.rados.0.smithi107.stdout:8524: done (0 left) 2020-06-12T02:38:41.728 INFO:tasks.rados.rados.0.smithi107.stdout:8525: delete oid 10 current snap is 0 2020-06-12T02:38:41.728 INFO:tasks.rados.rados.0.smithi107.stdout:8525: done (0 left) 2020-06-12T02:38:41.728 INFO:tasks.rados.rados.0.smithi107.stdout:8526: delete oid 8 current snap is 0 2020-06-12T02:38:41.728 INFO:tasks.rados.rados.0.smithi107.stdout:8526: done (0 left) 2020-06-12T02:38:41.728 INFO:tasks.rados.rados.0.smithi107.stdout:8527: delete oid 6 current snap is 0 2020-06-12T02:38:41.729 INFO:tasks.rados.rados.0.smithi107.stdout:8527: done (0 left) 2020-06-12T02:38:41.729 INFO:tasks.rados.rados.0.smithi107.stdout:8528: delete oid 14 current snap is 0 2020-06-12T02:38:41.729 INFO:tasks.rados.rados.0.smithi107.stdout:8528: done (0 left) 2020-06-12T02:38:41.729 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:41.729 INFO:tasks.rados.rados.0.smithi107.stdout:8529: seq_num 8528 ranges {0=434176,434176=483328,917504=319488} 2020-06-12T02:38:41.730 INFO:tasks.rados.rados.0.smithi107.stdout:8529: writing smithi10729407-3 from 0 to 434176 tid 1 2020-06-12T02:38:41.730 INFO:tasks.rados.rados.0.smithi107.stdout:8529: writing smithi10729407-3 from 434176 to 917504 tid 2 2020-06-12T02:38:41.730 INFO:tasks.rados.rados.0.smithi107.stdout:8529: writing smithi10729407-3 from 917504 to 1236992 tid 3 2020-06-12T02:38:41.730 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:41.730 INFO:tasks.rados.rados.0.smithi107.stdout:8530: seq_num 8529 ranges {0=450560,450560=761856,1212416=40960} 2020-06-12T02:38:41.730 INFO:tasks.rados.rados.0.smithi107.stdout:8530: writing smithi10729407-18 from 0 to 450560 tid 1 2020-06-12T02:38:41.731 INFO:tasks.rados.rados.0.smithi107.stdout:8530: writing smithi10729407-18 from 450560 to 1212416 tid 2 2020-06-12T02:38:41.731 INFO:tasks.rados.rados.0.smithi107.stdout:8530: writing smithi10729407-18 from 1212416 to 1253376 tid 3 2020-06-12T02:38:41.731 INFO:tasks.rados.rados.0.smithi107.stdout:8531: delete oid 20 current snap is 0 2020-06-12T02:38:41.731 INFO:tasks.rados.rados.0.smithi107.stdout:8529: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:41.731 INFO:tasks.rados.rados.0.smithi107.stdout:8530: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:41.732 INFO:tasks.rados.rados.0.smithi107.stdout:8529: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:41.732 INFO:tasks.rados.rados.0.smithi107.stdout:8531: done (2 left) 2020-06-12T02:38:41.732 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:41.732 INFO:tasks.rados.rados.0.smithi107.stdout:8532: seq_num 8531 ranges {884736=483328,1368064=581632,1949696=221184} 2020-06-12T02:38:41.732 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:38:41.733 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":107,"stamp":"2020-06-12T02:38:39.608898+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":16703488,"num_objects":9,"num_object_clones":0,"num_object_copies":25,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":4771,"num_read_kb":2584,"num_write":10607,"num_write_kb":3027426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":13869056,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14066,"ondisk_log_size":14066,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":66,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8407768,"kb_used_data":19096,"kb_used_omap":25,"kb_used_meta":8388582,"kb_avail":746566952,"statfs":{"total":773094113280,"available":764484558848,"internally_reserved":8589934592,"allocated":19554304,"data_stored":16598304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26157,"internal_metadata":8589908435},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7839744,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":586,"num_read_kb":316,"num_write":1332,"num_write_kb":385312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.168368"},"pg_stats":[{"pgid":"3.f","version":"61'1506","reported_seq":"1839","reported_epoch":"61","state":"active+clean","last_fresh":"2020-06-12T02:38:38.199570+0000","last_change":"2020-06-12T02:38:11.587385+0000","last_active":"2020-06-12T02:38:38.199570+0000","last_peered":"2020-06-12T02:38:38.199570+0000","last_clean":"2020-06-12T02:38:38.199570+0000","last_became_active":"2020-06-12T02:37:57.566633+0000","last_became_peered":"2020-06-12T02:37:57.566633+0000","last_unstale":"2020-06-12T02:38:38.199570+0000","last_undegraded":"2020-06-12T02:38:38.199570+0000","last_fullsized":"2020-06-12T02:38:38.199570+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":1506,"ondisk_log_size":1506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2883584,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":514,"num_read_kb":271,"num_write":1103,"num_write_kb":310024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1531904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"61'794","reported_seq":"1001","reported_epoch":"61","state":"active+clean","last_fresh":"2020-06-12T02:38:38.265611+0000","last_change":"2020-06-12T02:38:11.572385+0000","last_active":"2020-06-12T02:38:38.265611+0000","last_peered":"2020-06-12T02:38:38.265611+0000","last_clean":"2020-06-12T02:38:38.265611+0000","last_became_active":"2020-06-12T02:37:57.563869+0000","last_became_peered":"2020-06-12T02:37:57.563869+0000","last_unstale":"2020-06-12T02:38:38.265611+0000","last_undegraded":"2020-06-12T02:38:38.265611+0000","last_fullsized":"2020-06-12T02:38:38.265611+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":794,"ondisk_log_size":794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":262,"num_read_kb":143,"num_write":609,"num_write_kb":173584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1466368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"61'1447","reported_seq":"1783","reported_epoch":"61","state":"active+clean","last_fresh":"2020-06-12T02:38:38.293816+0000","last_change":"2020-06-12T02:38:11.587290+0000","last_active":"2020-06-12T02:38:38.293816+0000","last_peered":"2020-06-12T02:38:38.293816+0000","last_clean":"2020-06-12T02:38:38.293816+0000","last_became_active":"2020-06-12T02:37:57.564464+0000","last_became_peered":"2020-06-12T02:37:57.564464+0000","last_unstale":"2020-06-12T02:38:38.293816+0000","last_undegraded":"2020-06-12T02:38:38.293816+0000","last_fullsized":"2020-06-12T02:38:38.293816+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":1447,"ondisk_log_size":1447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2981888,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":485,"num_read_kb":262,"num_write":1097,"num_write_kb":314192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3710976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"42","reported_epoch":"60","state":"active+clean","last_fresh":"2020-06-12T02:38:33.451024+0000","last_change":"2020-06-12T02:37:42.384656+0000","last_active":"2020-06-12T02:38:33.451024+0000","last_peered":"2020-06-12T02:38:33.451024+0000","last_clean":"2020-06-12T02:38:33.451024+0000","last_became_active":"2020-06-12T02:37:42.384363+0000","last_became_peered":"2020-06-12T02:37:42.384363+0000","last_unstale":"2020-06-12T02:38:33.451024+0000","last_undegraded":"2020-06-12T02:38:33.451024+0000","last_fullsized":"2020-06-12T02:38:33.451024+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"61'1409","reported_seq":"1718","reported_epoch":"61","state":"active+clean","last_fresh":"2020-06-12T02:38:36.697807+0000","last_change":"2020-06-12T02:37:27.902643+0000","last_active":"2020-06-12T02:38:36.697807+0000","last_peered":"2020-06-12T02:38:36.697807+0000","last_clean":"2020-06-12T02:38:36.697807+0000","last_became_active":"2020-06-12T02:37:27.902204+0000","last_became_peered":"2020-06-12T02:37:27.902204+0000","last_unstale":"2020-06-12T02:38:36.697807+0000","last_undegraded":"2020-06-12T02:38:36.697807+0000","last_fullsized":"2020-06-12T02:38:36.697807+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":1409,"ondisk_log_size":1409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":484,"num_read_kb":262,"num_write":1056,"num_write_kb":296120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"60'1430","reported_seq":"1770","reported_epoch":"60","state":"active+clean","last_fresh":"2020-06-12T02:38:36.432854+0000","last_change":"2020-06-12T02:38:16.369345+0000","last_active":"2020-06-12T02:38:36.432854+0000","last_peered":"2020-06-12T02:38:36.432854+0000","last_clean":"2020-06-12T02:38:36.432854+0000","last_became_active":"2020-06-12T02:38:16.367448+0000","last_became_peered":"2020-06-12T02:38:16.367448+0000","last_unstale":"2020-06-12T02:38:36.432854+0000","last_undegraded":"2020-06-12T02:38:36.432854+0000","last_fullsized":"2020-06-12T02:38:36.432854+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":1430,"ondisk_log_size":1430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":500,"num_read_kb":271,"num_write":1088,"num_write_kb":313280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"61'1404","reported_seq":"1736","reported_epoch":"61","state":"active+clean","last_fresh":"2020-06-12T02:38:36.621362+0000","last_change":"2020-06-12T02:37:57.567142+0000","last_active":"2020-06-12T02:38:36.621362+0000","last_peered":"2020-06-12T02:38:36.621362+0000","last_clean":"2020-06-12T02:38:36.621362+0000","last_became_active":"2020-06-12T02:37:57.566462+0000","last_became_peered":"2020-06-12T02:37:57.566462+0000","last_unstale":"2020-06-12T02:38:36.621362+0000","last_undegraded":"2020-06-12T02:38:36.621362+0000","last_fullsized":"2020-06-12T02:38:36.621362+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":1404,"ondisk_log_size":1404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":476,"num_read_kb":264,"num_write":1048,"num_write_kb":291656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"61'727","reported_seq":"943","reported_epoch":"61","state":"active+clean","last_fresh":"2020-06-12T02:38:36.528165+0000","last_change":"2020-06-12T02:38:16.372264+0000","last_active":"2020-06-12T02:38:36.528165+0000","last_peered":"2020-06-12T02:38:36.528165+0000","last_clean":"2020-06-12T02:38:36.528165+0000","last_became_active":"2020-06-12T02:38:16.371502+0000","last_became_peered":"2020-06-12T02:38:16.371502+0000","last_unstale":"2020-06-12T02:38:36.528165+0000","last_undegraded":"2020-06-12T02:38:36.528165+0000","last_fullsized":"2020-06-12T02:38:36.528165+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":727,"ondisk_log_size":727,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":239,"num_read_kb":127,"num_write":550,"num_write_kb":158752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2793472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"24","reported_epoch":"60","state":"active+clean","last_fresh":"2020-06-12T02:38:34.510728+0000","last_change":"2020-06-12T02:38:34.510728+0000","last_active":"2020-06-12T02:38:34.510728+0000","last_peered":"2020-06-12T02:38:34.510728+0000","last_clean":"2020-06-12T02:38:34.510728+0000","last_became_active":"2020-06-12T02:38:13.661679+0000","last_became_peered":"2020-06-12T02:38:13.661679+0000","last_unstale":"2020-06-12T02:38:34.510728+0000","last_undegraded":"2020-06-12T02:38:34.510728+0000","last_fullsized":"2020-06-12T02:38:34.510728+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:38:34.510641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:38:34.510641+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"61'1382","reported_seq":"1680","reported_epoch":"61","state":"active+clean","last_fresh":"2020-06-12T02:38:36.703797+0000","last_change":"2020-06-12T02:37:42.379294+0000","last_active":"2020-06-12T02:38:36.703797+0000","last_peered":"2020-06-12T02:38:36.703797+0000","last_clean":"2020-06-12T02:38:36.703797+0000","last_became_active":"2020-06-12T02:37:42.378796+0000","last_became_peered":"2020-06-12T02:37:42.378796+0000","last_unstale":"2020-06-12T02:38:36.703797+0000","last_undegraded":"2020-06-12T02:38:36.703797+0000","last_fullsized":"2020-06-12T02:38:36.703797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":1382,"ondisk_log_size":1382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4481024,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":455,"num_read_kb":250,"num_write":1037,"num_write_kb":302992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"19","reported_epoch":"60","state":"active+clean","last_fresh":"2020-06-12T02:38:33.479727+0000","last_change":"2020-06-12T02:38:13.664800+0000","last_active":"2020-06-12T02:38:33.479727+0000","last_peered":"2020-06-12T02:38:33.479727+0000","last_clean":"2020-06-12T02:38:33.479727+0000","last_became_active":"2020-06-12T02:38:13.664384+0000","last_became_peered":"2020-06-12T02:38:13.664384+0000","last_unstale":"2020-06-12T02:38:33.479727+0000","last_undegraded":"2020-06-12T02:38:33.479727+0000","last_fullsized":"2020-06-12T02:38:33.479727+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"61'1422","reported_seq":"1750","reported_epoch":"61","state":"active+clean","last_fresh":"2020-06-12T02:38:36.650322+0000","last_change":"2020-06-12T02:38:16.372259+0000","last_active":"2020-06-12T02:38:36.650322+0000","last_peered":"2020-06-12T02:38:36.650322+0000","last_clean":"2020-06-12T02:38:36.650322+0000","last_became_active":"2020-06-12T02:38:16.371577+0000","last_became_peered":"2020-06-12T02:38:16.371577+0000","last_unstale":"2020-06-12T02:38:36.650322+0000","last_undegraded":"2020-06-12T02:38:36.650322+0000","last_fullsized":"2020-06-12T02:38:36.650322+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":1422,"ondisk_log_size":1422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2293760,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":487,"num_read_kb":257,"num_write":1072,"num_write_kb":307824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1572864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"54","reported_epoch":"60","state":"active+clean","last_fresh":"2020-06-12T02:38:33.448179+0000","last_change":"2020-06-12T02:38:21.039695+0000","last_active":"2020-06-12T02:38:33.448179+0000","last_peered":"2020-06-12T02:38:33.448179+0000","last_clean":"2020-06-12T02:38:33.448179+0000","last_became_active":"2020-06-12T02:37:16.764958+0000","last_became_peered":"2020-06-12T02:37:16.764958+0000","last_unstale":"2020-06-12T02:38:33.448179+0000","last_undegraded":"2020-06-12T02:38:33.448179+0000","last_fullsized":"2020-06-12T02:38:33.448179+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:38:21.039625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:38:21.039625+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"61'523","reported_seq":"689","reported_epoch":"61","state":"active+clean","last_fresh":"2020-06-12T02:38:36.581191+0000","last_change":"2020-06-12T02:38:30.671927+0000","last_active":"2020-06-12T02:38:36.581191+0000","last_peered":"2020-06-12T02:38:36.581191+0000","last_clean":"2020-06-12T02:38:36.581191+0000","last_became_active":"2020-06-12T02:38:16.371755+0000","last_became_peered":"2020-06-12T02:38:16.371755+0000","last_unstale":"2020-06-12T02:38:36.581191+0000","last_undegraded":"2020-06-12T02:38:36.581191+0000","last_fullsized":"2020-06-12T02:38:36.581191+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":523,"ondisk_log_size":523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":184,"num_read_kb":98,"num_write":407,"num_write_kb":116248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"44","reported_epoch":"60","state":"active+clean","last_fresh":"2020-06-12T02:38:33.448129+0000","last_change":"2020-06-12T02:38:22.079524+0000","last_active":"2020-06-12T02:38:33.448129+0000","last_peered":"2020-06-12T02:38:33.448129+0000","last_clean":"2020-06-12T02:38:33.448129+0000","last_became_active":"2020-06-12T02:37:42.375902+0000","last_became_peered":"2020-06-12T02:37:42.375902+0000","last_unstale":"2020-06-12T02:38:33.448129+0000","last_undegraded":"2020-06-12T02:38:33.448129+0000","last_fullsized":"2020-06-12T02:38:33.448129+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:38:22.079458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:38:22.079458+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"15'2","reported_seq":"52","reported_epoch":"60","state":"active+clean","last_fresh":"2020-06-12T02:38:33.467975+0000","last_change":"2020-06-12T02:38:24.369083+0000","last_active":"2020-06-12T02:38:33.467975+0000","last_peered":"2020-06-12T02:38:33.467975+0000","last_clean":"2020-06-12T02:38:33.467975+0000","last_became_active":"2020-06-12T02:37:54.558125+0000","last_became_peered":"2020-06-12T02:37:54.558125+0000","last_unstale":"2020-06-12T02:38:33.467975+0000","last_undegraded":"2020-06-12T02:38:33.467975+0000","last_fullsized":"2020-06-12T02:38:33.467975+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":3,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'2","last_scrub_stamp":"2020-06-12T02:38:24.368994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:04.419204+0000","last_clean_scrub_stamp":"2020-06-12T02:38:24.368994+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"61'565","reported_seq":"716","reported_epoch":"61","state":"active+clean","last_fresh":"2020-06-12T02:38:36.487857+0000","last_change":"2020-06-12T02:37:27.898985+0000","last_active":"2020-06-12T02:38:36.487857+0000","last_peered":"2020-06-12T02:38:36.487857+0000","last_clean":"2020-06-12T02:38:36.487857+0000","last_became_active":"2020-06-12T02:37:27.898516+0000","last_became_peered":"2020-06-12T02:37:27.898516+0000","last_unstale":"2020-06-12T02:38:36.487857+0000","last_undegraded":"2020-06-12T02:38:36.487857+0000","last_fullsized":"2020-06-12T02:38:36.487857+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":565,"ondisk_log_size":565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1720320,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":190,"num_read_kb":104,"num_write":409,"num_write_kb":114240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"57","reported_epoch":"60","state":"active+clean","last_fresh":"2020-06-12T02:38:34.394637+0000","last_change":"2020-06-12T02:38:34.394637+0000","last_active":"2020-06-12T02:38:34.394637+0000","last_peered":"2020-06-12T02:38:34.394637+0000","last_clean":"2020-06-12T02:38:34.394637+0000","last_became_active":"2020-06-12T02:38:13.660654+0000","last_became_peered":"2020-06-12T02:38:13.660654+0000","last_unstale":"2020-06-12T02:38:34.394637+0000","last_undegraded":"2020-06-12T02:38:34.394637+0000","last_fullsized":"2020-06-12T02:38:34.394637+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:38:34.394578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:38:34.394578+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"47","reported_epoch":"60","state":"active+clean","last_fresh":"2020-06-12T02:38:33.446878+0000","last_change":"2020-06-12T02:37:27.901368+0000","last_active":"2020-06-12T02:38:33.446878+0000","last_peered":"2020-06-12T02:38:33.446878+0000","last_clean":"2020-06-12T02:38:33.446878+0000","last_became_active":"2020-06-12T02:37:27.900888+0000","last_became_peered":"2020-06-12T02:37:27.900888+0000","last_unstale":"2020-06-12T02:38:33.446878+0000","last_undegraded":"2020-06-12T02:38:33.446878+0000","last_fullsized":"2020-06-12T02:38:33.446878+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"46","reported_epoch":"60","state":"active+clean","last_fresh":"2020-06-12T02:38:33.468045+0000","last_change":"2020-06-12T02:38:05.892563+0000","last_active":"2020-06-12T02:38:33.468045+0000","last_peered":"2020-06-12T02:38:33.468045+0000","last_clean":"2020-06-12T02:38:33.468045+0000","last_became_active":"2020-06-12T02:37:54.557420+0000","last_became_peered":"2020-06-12T02:37:54.557420+0000","last_unstale":"2020-06-12T02:38:33.468045+0000","last_undegraded":"2020-06-12T02:38:33.468045+0000","last_fullsized":"2020-06-12T02:38:33.468045+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"22","reported_epoch":"60","state":"active+clean","last_fresh":"2020-06-12T02:38:33.448251+0000","last_change":"2020-06-12T02:38:13.666985+0000","last_active":"2020-06-12T02:38:33.448251+0000","last_peered":"2020-06-12T02:38:33.448251+0000","last_clean":"2020-06-12T02:38:33.448251+0000","last_became_active":"2020-06-12T02:38:13.666441+0000","last_became_peered":"2020-06-12T02:38:13.666441+0000","last_unstale":"2020-06-12T02:38:33.448251+0000","last_undegraded":"2020-06-12T02:38:33.448251+0000","last_fullsized":"2020-06-12T02:38:33.448251+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"61'1455","reported_seq":"1795","reported_epoch":"61","state":"active+clean","last_fresh":"2020-06-12T02:38:36.671749+0000","last_change":"2020-06-12T02:37:57.567683+0000","last_active":"2020-06-12T02:38:36.671749+0000","last_peered":"2020-06-12T02:38:36.671749+0000","last_clean":"2020-06-12T02:38:36.671749+0000","last_became_active":"2020-06-12T02:37:57.566865+0000","last_became_peered":"2020-06-12T02:37:57.566865+0000","last_unstale":"2020-06-12T02:38:36.671749+0000","last_undegraded":"2020-06-12T02:38:36.671749+0000","last_fullsized":"2020-06-12T02:38:36.671749+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":1455,"ondisk_log_size":1455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2342912,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":495,"num_read_kb":275,"num_write":1129,"num_write_kb":328504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2793472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"21","reported_epoch":"60","state":"active+clean","last_fresh":"2020-06-12T02:38:33.479661+0000","last_change":"2020-06-12T02:38:13.664365+0000","last_active":"2020-06-12T02:38:33.479661+0000","last_peered":"2020-06-12T02:38:33.479661+0000","last_clean":"2020-06-12T02:38:33.479661+0000","last_became_active":"2020-06-12T02:38:13.663999+0000","last_became_peered":"2020-06-12T02:38:13.663999+0000","last_unstale":"2020-06-12T02:38:33.479661+0000","last_undegraded":"2020-06-12T02:38:33.479661+0000","last_fullsized":"2020-06-12T02:38:33.479661+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"66","reported_epoch":"60","state":"active+clean","last_fresh":"2020-06-12T02:38:33.479689+0000","last_change":"2020-06-12T02:38:22.034355+0000","last_active":"2020-06-12T02:38:33.479689+0000","last_peered":"2020-06-12T02:38:33.479689+0000","last_clean":"2020-06-12T02:38:33.479689+0000","last_became_active":"2020-06-12T02:37:54.554986+0000","last_became_peered":"2020-06-12T02:37:54.554986+0000","last_unstale":"2020-06-12T02:38:33.479689+0000","last_undegraded":"2020-06-12T02:38:33.479689+0000","last_fullsized":"2020-06-12T02:38:33.479689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:38:22.034263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:38:22.034263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"56","reported_epoch":"60","state":"active+clean","last_fresh":"2020-06-12T02:38:33.446928+0000","last_change":"2020-06-12T02:38:13.666662+0000","last_active":"2020-06-12T02:38:33.446928+0000","last_peered":"2020-06-12T02:38:33.446928+0000","last_clean":"2020-06-12T02:38:33.446928+0000","last_became_active":"2020-06-12T02:38:13.666144+0000","last_became_peered":"2020-06-12T02:38:13.666144+0000","last_unstale":"2020-06-12T02:38:33.446928+0000","last_undegraded":"2020-06-12T02:38:33.446928+0000","last_fullsized":"2020-06-12T02:38:33.446928+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":16703488,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4771,"num_read_kb":2584,"num_write":10605,"num_write_kb":3027416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":13869056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":22355968,"data_stored":22355968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14064,"ondisk_log_size":14064,"up":48,"acting":48,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26157,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":34,"seq":146028888074,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1054544,"kb_used_data":5960,"kb_used_omap":25,"kb_used_meta":1048550,"kb_avail":93317296,"statfs":{"total":96636764160,"available":95556911104,"internally_reserved":1073741824,"allocated":6103040,"data_stored":5729290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26157,"internal_metadata":1073715667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95699999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3120000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.879}]}]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1051424,"kb_used_data":2840,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93320416,"statfs":{"total":96636764160,"available":95560105984,"internally_reserved":1073741824,"allocated":2908160,"data_stored":2540484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.655,"5min":1.655,"15min":1.655},"last":2.339},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.97199999999999998}]},{"osd":1,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.335,"5min":1.335,"15min":1.335},"last":1.365},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.923,"5min":1.923,"15min":1.923},"last":2.798}]},{"osd":2,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.01,"15min":1.01},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.6699999999999999,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":1.097},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.653,"5min":1.653,"15min":1.653},"last":1.0269999999999999}]},{"osd":3,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.145,"15min":1.145},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.5649999999999999,"5min":1.5649999999999999,"15min":1.5649999999999999},"last":2.887},{"interface":"front","average":{"1min":1.143,"5min":1.143,"15min":1.143},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.556,"5min":1.556,"15min":1.556},"last":1.397}]},{"osd":4,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.246,"5min":1.246,"15min":1.246},"last":2.702},{"interface":"front","average":{"1min":1.071,"5min":1.071,"15min":1.071},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.673,"5min":1.673,"15min":1.673},"last":0.84299999999999997}]},{"osd":5,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.8120000000000001,"5min":1.8120000000000001,"15min":1.8120000000000001},"last":1.212},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.677,"5min":1.677,"15min":1.677},"last":2.9700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2549999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049264,"kb_used_data":680,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322576,"statfs":{"total":96636764160,"available":95562317824,"internally_reserved":1073741824,"allocated":696320,"data_stored":328644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 12 02:38:14 2020","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.548,"5min":1.548,"15min":1.548},"last":1.6359999999999999},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":2.149,"5min":2.149,"15min":2.149},"last":2.5539999999999998}]},{"osd":2,"last update":"Fri Jun 12 02:38:14 2020","interfaces":[{"interface":"back","average":{"1min":1,"5min":1,"15min":1},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":3.0009999999999999,"5min":3.0009999999999999,"15min":3.0009999999999999},"last":1.478},{"interface":"front","average":{"1min":1.014,"5min":1.014,"15min":1.014},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":2.4159999999999999,"5min":2.4159999999999999,"15min":2.4159999999999999},"last":2.7029999999999998}]},{"osd":3,"last update":"Fri Jun 12 02:38:14 2020","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":2.133,"5min":2.133,"15min":2.133},"last":3.1819999999999999},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":2.4609999999999999,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":1.889}]},{"osd":4,"last update":"Fri Jun 12 02:38:14 2020","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":2.1749999999999998,"5min":2.1749999999999998,"15min":2.1749999999999998},"last":3.2290000000000001},{"interface":"front","average":{"1min":1.302,"5min":1.302,"15min":1.302},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":2.2919999999999998,"5min":2.2919999999999998,"15min":2.2919999999999998},"last":2.8730000000000002}]},{"osd":5,"last update":"Fri Jun 12 02:38:14 2020","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.3200000000000001,"15min":1.3200000000000001},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":2.3090000000000002,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":2.0590000000000002},{"interface":"front","average":{"1min":1.448,"5min":1.448,"15min":1.448},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.835,"5min":2.835,"15min":2.835},"last":2.2210000000000001}]},{"osd":6,"last update":"Fri Jun 12 02:38:14 2020","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.389,"15min":1.389},"min":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"max":{"1min":2.4649999999999999,"5min":2.4649999999999999,"15min":2.4649999999999999},"last":2.3849999999999998},{"interface":"front","average":{"1min":1.6299999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"max":{"1min":3.1589999999999998,"5min":3.1589999999999998,"15min":3.1589999999999998},"last":3.1099999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.968}]}]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049276,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322564,"statfs":{"total":96636764160,"available":95562305536,"internally_reserved":1073741824,"allocated":708608,"data_stored":334858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jun 12 02:38:16 2020","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.7789999999999999,"5min":1.7789999999999999,"15min":1.7789999999999999},"last":1.228}]},{"osd":2,"last update":"Fri Jun 12 02:38:16 2020","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":2.141,"5min":2.141,"15min":2.141},"last":1.006},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.873,"5min":1.873,"15min":1.873},"last":1.028}]},{"osd":3,"last update":"Fri Jun 12 02:38:16 2020","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":2.2320000000000002,"5min":2.2320000000000002,"15min":2.2320000000000002},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.23,"5min":1.23,"15min":1.23},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.8220000000000001,"5min":1.8220000000000001,"15min":1.8220000000000001},"last":0.85599999999999998}]},{"osd":4,"last update":"Fri Jun 12 02:38:16 2020","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.7709999999999999,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":1.3680000000000001},{"interface":"front","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.554,"5min":1.554,"15min":1.554},"last":1.073}]},{"osd":5,"last update":"Fri Jun 12 02:38:16 2020","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.212,"15min":1.212},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":1.6699999999999999,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":1.502,"5min":1.502,"15min":1.502},"min":{"1min":1.006,"5min":1.006,"15min":1.006},"max":{"1min":2.6240000000000001,"5min":2.6240000000000001,"15min":2.6240000000000001},"last":1.1830000000000001}]},{"osd":6,"last update":"Fri Jun 12 02:38:16 2020","interfaces":[{"interface":"back","average":{"1min":1.5840000000000001,"5min":1.5840000000000001,"15min":1.5840000000000001},"min":{"1min":1.157,"5min":1.157,"15min":1.157},"max":{"1min":3.2440000000000002,"5min":3.2440000000000002,"15min":3.2440000000000002},"last":1.101},{"interface":"front","average":{"1min":1.335,"5min":1.335,"15min":1.335},"min":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"max":{"1min":3.105,"5min":3.105,"15min":3.105},"last":1.5029999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2070000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574868,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050320,"kb_used_data":1736,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321520,"statfs":{"total":96636764160,"available":95561236480,"internally_reserved":1073741824,"allocated":1777664,"data_stored":1409988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 12 02:38:10 2020","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.871,"15min":0.871},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.9259999999999999,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":3.0049999999999999,"5min":3.0049999999999999,"15min":3.0049999999999999},"last":0.71199999999999997}]},{"osd":1,"last update":"Fri Jun 12 02:38:10 2020","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.046,"15min":1.046},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":2.21,"5min":2.21,"15min":2.21},"last":0.747},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":2.5219999999999998,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":0.88100000000000001}]},{"osd":3,"last update":"Fri Jun 12 02:38:10 2020","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1,"5min":1,"15min":1},"last":0.95499999999999996},{"interface":"front","average":{"1min":1.145,"5min":1.145,"15min":1.145},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.2759999999999998,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":0.90900000000000003}]},{"osd":4,"last update":"Fri Jun 12 02:38:10 2020","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.321,"15min":1.321},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":2.6949999999999998,"5min":2.6949999999999998,"15min":2.6949999999999998},"last":0.61799999999999999},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"min":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"max":{"1min":1.7549999999999999,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":1.044}]},{"osd":5,"last update":"Fri Jun 12 02:38:10 2020","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"min":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"max":{"1min":2.1419999999999999,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":1.0620000000000001},{"interface":"front","average":{"1min":1.399,"5min":1.399,"15min":1.399},"min":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"max":{"1min":2.851,"5min":2.851,"15min":2.851},"last":1.0029999999999999}]},{"osd":6,"last update":"Fri Jun 12 02:38:10 2020","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":1.7110000000000001,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":1.125},{"interface":"front","average":{"1min":1.462,"5min":1.462,"15min":1.462},"min":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"max":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":1.21}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0920000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574868,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050588,"kb_used_data":2004,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321252,"statfs":{"total":96636764160,"available":95560962048,"internally_reserved":1073741824,"allocated":2052096,"data_stored":1684420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 12 02:38:12 2020","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":2.0339999999999998,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.9219999999999999,"5min":1.9219999999999999,"15min":1.9219999999999999},"last":0.74399999999999999}]},{"osd":1,"last update":"Fri Jun 12 02:38:12 2020","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.9670000000000001,"5min":1.9670000000000001,"15min":1.9670000000000001},"last":0.97399999999999998},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":2.9969999999999999,"5min":2.9969999999999999,"15min":2.9969999999999999},"last":0.93700000000000006}]},{"osd":2,"last update":"Fri Jun 12 02:38:12 2020","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":4.1079999999999997,"5min":4.1079999999999997,"15min":4.1079999999999997},"last":0.80600000000000005},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":2.504,"5min":2.504,"15min":2.504},"last":1.2}]},{"osd":4,"last update":"Fri Jun 12 02:38:12 2020","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":2.359,"5min":2.359,"15min":2.359},"last":1.143},{"interface":"front","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":3.9319999999999999,"5min":3.9319999999999999,"15min":3.9319999999999999},"last":1.3999999999999999}]},{"osd":5,"last update":"Fri Jun 12 02:38:12 2020","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":2.4279999999999999,"5min":2.4279999999999999,"15min":2.4279999999999999},"last":1.552},{"interface":"front","average":{"1min":1.452,"5min":1.452,"15min":1.452},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":5.0579999999999998,"5min":5.0579999999999998,"15min":5.0579999999999998},"last":1.361}]},{"osd":6,"last update":"Fri Jun 12 02:38:12 2020","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.29,"15min":1.29},"min":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"max":{"1min":4.0410000000000004,"5min":4.0410000000000004,"15min":4.0410000000000004},"last":1.24},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.3620000000000001,"15min":1.3620000000000001},"min":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"max":{"1min":3.0230000000000001,"5min":3.0230000000000001,"15min":3.0230000000000001},"last":1.673}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5109999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050188,"kb_used_data":1604,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321652,"statfs":{"total":96636764160,"available":95561371648,"internally_reserved":1073741824,"allocated":1642496,"data_stored":1268746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.712,"5min":1.712,"15min":1.712},"last":0.498},{"interface":"front","average":{"1min":1.02,"5min":1.02,"15min":1.02},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":2.855,"5min":2.855,"15min":2.855},"last":1.2150000000000001}]},{"osd":1,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.8240000000000001,"5min":1.8240000000000001,"15min":1.8240000000000001},"last":0.80100000000000005}]},{"osd":2,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":3.1480000000000001,"5min":3.1480000000000001,"15min":3.1480000000000001},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"min":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"max":{"1min":3.1259999999999999,"5min":3.1259999999999999,"15min":3.1259999999999999},"last":1.046}]},{"osd":3,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.079,"15min":1.079},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":2.411,"5min":2.411,"15min":2.411},"last":1.5620000000000001},{"interface":"front","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":2.734,"5min":2.734,"15min":2.734},"last":1.006}]},{"osd":5,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.04,"15min":1.04},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.6399999999999999,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":0.96499999999999997},{"interface":"front","average":{"1min":1.115,"5min":1.115,"15min":1.115},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"last":1.4379999999999999}]},{"osd":6,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":0.998,"15min":0.998},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.755},{"interface":"front","average":{"1min":1.341,"5min":1.341,"15min":1.341},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":2.1850000000000001,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":1.115}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1910000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1052164,"kb_used_data":3580,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93319676,"statfs":{"total":96636764160,"available":95559348224,"internally_reserved":1073741824,"allocated":3665920,"data_stored":3301874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 12 02:38:13 2020","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.456,"5min":1.456,"15min":1.456},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.597,"5min":1.597,"15min":1.597},"last":0.64900000000000002}]},{"osd":1,"last update":"Fri Jun 12 02:38:13 2020","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":1.173},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.83799999999999997}]},{"osd":2,"last update":"Fri Jun 12 02:38:13 2020","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.012,"5min":1.012,"15min":1.012},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.4259999999999999,"5min":1.4259999999999999,"15min":1.4259999999999999},"last":1.1519999999999999}]},{"osd":3,"last update":"Fri Jun 12 02:38:13 2020","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":2.004,"5min":2.004,"15min":2.004},"last":0.877},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.627,"5min":1.627,"15min":1.627},"last":1.26}]},{"osd":4,"last update":"Fri Jun 12 02:38:13 2020","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.997,"15min":0.997},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.796,"5min":1.796,"15min":1.796},"last":1.857},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":1.026}]},{"osd":6,"last update":"Fri Jun 12 02:38:13 2020","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.827,"5min":1.827,"15min":1.827},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.228,"5min":1.228,"15min":1.228},"min":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"max":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":1.1399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8679999999999999}]}]}]}} 2020-06-12T02:38:41.735 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2020-06-12T02:38:41.738 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2020-06-12T02:38:41.739 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 26 2020-06-12T02:38:41.740 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:41.640+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:41.740 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-06-12T02:38:41.746 INFO:tasks.rados.rados.0.smithi107.stdout:8532: writing smithi10729407-17 from 884736 to 1368064 tid 1 2020-06-12T02:38:41.746 INFO:tasks.rados.rados.0.smithi107.stdout:8532: writing smithi10729407-17 from 1368064 to 1949696 tid 2 2020-06-12T02:38:41.746 INFO:tasks.rados.rados.0.smithi107.stdout:8532: writing smithi10729407-17 from 1949696 to 2170880 tid 3 2020-06-12T02:38:41.747 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:41.747 INFO:tasks.rados.rados.0.smithi107.stdout:8533: seq_num 8532 ranges {0=499712,499712=466944,966656=335872} 2020-06-12T02:38:41.748 INFO:tasks.rados.rados.0.smithi107.stdout:8533: writing smithi10729407-19 from 0 to 499712 tid 1 2020-06-12T02:38:41.748 INFO:tasks.rados.rados.0.smithi107.stdout:8533: writing smithi10729407-19 from 499712 to 966656 tid 2 2020-06-12T02:38:41.748 INFO:tasks.rados.rados.0.smithi107.stdout:8533: writing smithi10729407-19 from 966656 to 1302528 tid 3 2020-06-12T02:38:41.748 INFO:tasks.rados.rados.0.smithi107.stdout:8534: delete oid 2 current snap is 0 2020-06-12T02:38:41.749 INFO:tasks.rados.rados.0.smithi107.stdout:8529: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:41.749 INFO:tasks.rados.rados.0.smithi107.stdout:8529: finishing write tid 5 to smithi10729407-3 2020-06-12T02:38:41.749 INFO:tasks.rados.rados.0.smithi107.stdout:8530: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:41.749 INFO:tasks.rados.rados.0.smithi107.stdout:8529: finishing write tid 6 to smithi10729407-3 2020-06-12T02:38:41.749 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1539 (ObjNum 8528 snap 0 seq_num 8528) dirty exists 2020-06-12T02:38:41.749 INFO:tasks.rados.rados.0.smithi107.stdout:8529: left oid 3 (ObjNum 8528 snap 0 seq_num 8528) 2020-06-12T02:38:41.750 INFO:tasks.rados.rados.0.smithi107.stdout:8530: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:41.750 INFO:tasks.rados.rados.0.smithi107.stdout:8530: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:41.750 INFO:tasks.rados.rados.0.smithi107.stdout:8530: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:41.750 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1593 (ObjNum 8529 snap 0 seq_num 8529) dirty exists 2020-06-12T02:38:41.750 INFO:tasks.rados.rados.0.smithi107.stdout:8530: left oid 18 (ObjNum 8529 snap 0 seq_num 8529) 2020-06-12T02:38:41.751 INFO:tasks.rados.rados.0.smithi107.stdout:8532: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:41.751 INFO:tasks.rados.rados.0.smithi107.stdout:8532: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:41.751 INFO:tasks.rados.rados.0.smithi107.stdout:8532: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:41.751 INFO:tasks.rados.rados.0.smithi107.stdout:8533: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:41.751 INFO:tasks.rados.rados.0.smithi107.stdout:8532: finishing write tid 5 to smithi10729407-17 2020-06-12T02:38:41.752 INFO:tasks.rados.rados.0.smithi107.stdout:8529: done (4 left) 2020-06-12T02:38:41.752 INFO:tasks.rados.rados.0.smithi107.stdout:8530: done (3 left) 2020-06-12T02:38:41.752 INFO:tasks.rados.rados.0.smithi107.stdout:8534: done (2 left) 2020-06-12T02:38:41.752 INFO:tasks.rados.rados.0.smithi107.stdout:8535: delete oid 11 current snap is 0 2020-06-12T02:38:41.752 INFO:tasks.rados.rados.0.smithi107.stdout:8532: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:41.752 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1603 (ObjNum 8531 snap 0 seq_num 8531) dirty exists 2020-06-12T02:38:41.753 INFO:tasks.rados.rados.0.smithi107.stdout:8532: left oid 17 (ObjNum 8531 snap 0 seq_num 8531) 2020-06-12T02:38:41.753 INFO:tasks.rados.rados.0.smithi107.stdout:8532: done (2 left) 2020-06-12T02:38:41.753 INFO:tasks.rados.rados.0.smithi107.stdout:8535: done (1 left) 2020-06-12T02:38:41.753 INFO:tasks.rados.rados.0.smithi107.stdout:8536: delete oid 18 current snap is 0 2020-06-12T02:38:41.754 INFO:tasks.rados.rados.0.smithi107.stdout:8533: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:41.754 INFO:tasks.rados.rados.0.smithi107.stdout:8533: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:41.754 INFO:tasks.rados.rados.0.smithi107.stdout:8536: done (1 left) 2020-06-12T02:38:41.754 INFO:tasks.rados.rados.0.smithi107.stdout:8537: delete oid 11 current snap is 0 2020-06-12T02:38:41.755 INFO:tasks.rados.rados.0.smithi107.stdout:8533: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:41.755 INFO:tasks.rados.rados.0.smithi107.stdout:8537: done (1 left) 2020-06-12T02:38:41.755 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:41.755 INFO:tasks.rados.rados.0.smithi107.stdout:8538: seq_num 8537 ranges {1032192=581632,1613824=688128,2301952=114688} 2020-06-12T02:38:41.755 INFO:tasks.rados.rados.0.smithi107.stdout:8538: writing smithi10729407-1 from 1032192 to 1613824 tid 1 2020-06-12T02:38:41.755 INFO:tasks.rados.rados.0.smithi107.stdout:8538: writing smithi10729407-1 from 1613824 to 2301952 tid 2 2020-06-12T02:38:41.756 INFO:tasks.rados.rados.0.smithi107.stdout:8538: writing smithi10729407-1 from 2301952 to 2416640 tid 3 2020-06-12T02:38:41.756 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:41.756 INFO:tasks.rados.rados.0.smithi107.stdout:8539: seq_num 8538 ranges {0=598016,598016=622592,1220608=180224} 2020-06-12T02:38:41.756 INFO:tasks.rados.rados.0.smithi107.stdout:8539: writing smithi10729407-4 from 0 to 598016 tid 1 2020-06-12T02:38:41.756 INFO:tasks.rados.rados.0.smithi107.stdout:8539: writing smithi10729407-4 from 598016 to 1220608 tid 2 2020-06-12T02:38:41.757 INFO:tasks.rados.rados.0.smithi107.stdout:8539: writing smithi10729407-4 from 1220608 to 1400832 tid 3 2020-06-12T02:38:41.757 INFO:tasks.rados.rados.0.smithi107.stdout:8540: delete oid 7 current snap is 0 2020-06-12T02:38:41.757 INFO:tasks.rados.rados.0.smithi107.stdout:8533: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:41.757 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1569 (ObjNum 8532 snap 0 seq_num 8532) dirty exists 2020-06-12T02:38:41.757 INFO:tasks.rados.rados.0.smithi107.stdout:8533: left oid 19 (ObjNum 8532 snap 0 seq_num 8532) 2020-06-12T02:38:41.757 INFO:tasks.rados.rados.0.smithi107.stdout:8538: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:41.758 INFO:tasks.rados.rados.0.smithi107.stdout:8538: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:41.758 INFO:tasks.rados.rados.0.smithi107.stdout:8538: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:41.758 INFO:tasks.rados.rados.0.smithi107.stdout:8538: finishing write tid 5 to smithi10729407-1 2020-06-12T02:38:41.758 INFO:tasks.rados.rados.0.smithi107.stdout:8533: done (3 left) 2020-06-12T02:38:41.758 INFO:tasks.rados.rados.0.smithi107.stdout:8540: done (2 left) 2020-06-12T02:38:41.759 INFO:tasks.rados.rados.0.smithi107.stdout:8541: delete oid 15 current snap is 0 2020-06-12T02:38:41.759 INFO:tasks.rados.rados.0.smithi107.stdout:8538: finishing write tid 6 to smithi10729407-1 2020-06-12T02:38:41.759 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1552 (ObjNum 8537 snap 0 seq_num 8537) dirty exists 2020-06-12T02:38:41.759 INFO:tasks.rados.rados.0.smithi107.stdout:8538: left oid 1 (ObjNum 8537 snap 0 seq_num 8537) 2020-06-12T02:38:41.759 INFO:tasks.rados.rados.0.smithi107.stdout:8538: done (2 left) 2020-06-12T02:38:41.760 INFO:tasks.rados.rados.0.smithi107.stdout:8541: done (1 left) 2020-06-12T02:38:41.760 INFO:tasks.rados.rados.0.smithi107.stdout:8542: delete oid 9 current snap is 0 2020-06-12T02:38:41.760 INFO:tasks.rados.rados.0.smithi107.stdout:8539: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:41.760 INFO:tasks.rados.rados.0.smithi107.stdout:8539: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:41.760 INFO:tasks.rados.rados.0.smithi107.stdout:8539: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:41.760 INFO:tasks.rados.rados.0.smithi107.stdout:8539: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:41.761 INFO:tasks.rados.rados.0.smithi107.stdout:8542: done (1 left) 2020-06-12T02:38:41.761 INFO:tasks.rados.rados.0.smithi107.stdout:8543: delete oid 2 current snap is 0 2020-06-12T02:38:41.761 INFO:tasks.rados.rados.0.smithi107.stdout:8539: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:41.761 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1607 (ObjNum 8538 snap 0 seq_num 8538) dirty exists 2020-06-12T02:38:41.761 INFO:tasks.rados.rados.0.smithi107.stdout:8539: left oid 4 (ObjNum 8538 snap 0 seq_num 8538) 2020-06-12T02:38:41.761 INFO:tasks.rados.rados.0.smithi107.stdout:8539: done (1 left) 2020-06-12T02:38:41.762 INFO:tasks.rados.rados.0.smithi107.stdout:8543: done (0 left) 2020-06-12T02:38:41.762 INFO:tasks.rados.rados.0.smithi107.stdout:8544: delete oid 20 current snap is 0 2020-06-12T02:38:41.762 INFO:tasks.rados.rados.0.smithi107.stdout:8544: done (0 left) 2020-06-12T02:38:41.762 INFO:tasks.rados.rados.0.smithi107.stdout:8545: delete oid 15 current snap is 0 2020-06-12T02:38:41.762 INFO:tasks.rados.rados.0.smithi107.stdout:8545: done (0 left) 2020-06-12T02:38:41.763 INFO:tasks.rados.rados.0.smithi107.stdout:8546: delete oid 13 current snap is 0 2020-06-12T02:38:41.763 INFO:tasks.rados.rados.0.smithi107.stdout:8546: done (0 left) 2020-06-12T02:38:41.763 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:41.763 INFO:tasks.rados.rados.0.smithi107.stdout:8547: seq_num 8546 ranges {0=737280,737280=548864,1286144=253952} 2020-06-12T02:38:41.763 INFO:tasks.rados.rados.0.smithi107.stdout:8547: writing smithi10729407-2 from 0 to 737280 tid 1 2020-06-12T02:38:41.766 INFO:tasks.rados.rados.0.smithi107.stdout:8547: writing smithi10729407-2 from 737280 to 1286144 tid 2 2020-06-12T02:38:41.766 INFO:tasks.rados.rados.0.smithi107.stdout:8547: writing smithi10729407-2 from 1286144 to 1540096 tid 3 2020-06-12T02:38:41.767 INFO:tasks.rados.rados.0.smithi107.stdout:8548: delete oid 3 current snap is 0 2020-06-12T02:38:41.767 INFO:tasks.rados.rados.0.smithi107.stdout:8548: done (1 left) 2020-06-12T02:38:41.767 INFO:tasks.rados.rados.0.smithi107.stdout:8549: delete oid 5 current snap is 0 2020-06-12T02:38:41.767 INFO:tasks.rados.rados.0.smithi107.stdout:8547: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:41.767 INFO:tasks.rados.rados.0.smithi107.stdout:8547: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:41.768 INFO:tasks.rados.rados.0.smithi107.stdout:8549: done (1 left) 2020-06-12T02:38:41.768 INFO:tasks.rados.rados.0.smithi107.stdout:8550: delete oid 4 current snap is 0 2020-06-12T02:38:41.768 INFO:tasks.rados.rados.0.smithi107.stdout:8547: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:41.768 INFO:tasks.rados.rados.0.smithi107.stdout:8547: finishing write tid 5 to smithi10729407-2 2020-06-12T02:38:41.768 INFO:tasks.rados.rados.0.smithi107.stdout:8547: finishing write tid 6 to smithi10729407-2 2020-06-12T02:38:41.768 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 797 (ObjNum 8546 snap 0 seq_num 8546) dirty exists 2020-06-12T02:38:41.769 INFO:tasks.rados.rados.0.smithi107.stdout:8547: left oid 2 (ObjNum 8546 snap 0 seq_num 8546) 2020-06-12T02:38:41.769 INFO:tasks.rados.rados.0.smithi107.stdout:8547: done (1 left) 2020-06-12T02:38:41.769 INFO:tasks.rados.rados.0.smithi107.stdout:8550: done (0 left) 2020-06-12T02:38:41.769 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:41.769 INFO:tasks.rados.rados.0.smithi107.stdout:8551: seq_num 8550 ranges {2170880=802816,2973696=483328,3457024=319488} 2020-06-12T02:38:41.770 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:41.763+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:41.782 INFO:tasks.rados.rados.0.smithi107.stdout:8551: writing smithi10729407-17 from 2170880 to 2973696 tid 1 2020-06-12T02:38:41.785 INFO:tasks.rados.rados.0.smithi107.stdout:8551: writing smithi10729407-17 from 2973696 to 3457024 tid 2 2020-06-12T02:38:41.787 INFO:tasks.rados.rados.0.smithi107.stdout:8551: writing smithi10729407-17 from 3457024 to 3776512 tid 3 2020-06-12T02:38:41.787 INFO:tasks.rados.rados.0.smithi107.stdout:8552: delete oid 4 current snap is 0 2020-06-12T02:38:41.790 INFO:tasks.rados.rados.0.smithi107.stdout:8551: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:41.797 INFO:tasks.rados.rados.0.smithi107.stdout:8552: done (1 left) 2020-06-12T02:38:41.797 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:38:41.798 INFO:tasks.rados.rados.0.smithi107.stdout:8553: seq_num 8552 ranges {0=434176,434176=655360,1089536=532480,1622016=16384} 2020-06-12T02:38:41.800 INFO:tasks.rados.rados.0.smithi107.stdout:8553: writing smithi10729407-3 from 0 to 434176 tid 1 2020-06-12T02:38:41.804 INFO:tasks.rados.rados.0.smithi107.stdout:8553: writing smithi10729407-3 from 434176 to 1089536 tid 2 2020-06-12T02:38:41.808 INFO:tasks.rados.rados.0.smithi107.stdout:8553: writing smithi10729407-3 from 1089536 to 1622016 tid 3 2020-06-12T02:38:41.809 INFO:tasks.rados.rados.0.smithi107.stdout:8553: writing smithi10729407-3 from 1622016 to 1638400 tid 4 2020-06-12T02:38:41.809 INFO:tasks.rados.rados.0.smithi107.stdout:8554: delete oid 8 current snap is 0 2020-06-12T02:38:41.809 INFO:tasks.rados.rados.0.smithi107.stdout:8551: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:41.809 INFO:tasks.rados.rados.0.smithi107.stdout:8551: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:41.810 INFO:tasks.rados.rados.0.smithi107.stdout:8551: finishing write tid 5 to smithi10729407-17 2020-06-12T02:38:41.810 INFO:tasks.rados.rados.0.smithi107.stdout:8551: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:41.810 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1613 (ObjNum 8550 snap 0 seq_num 8550) dirty exists 2020-06-12T02:38:41.810 INFO:tasks.rados.rados.0.smithi107.stdout:8551: left oid 17 (ObjNum 8550 snap 0 seq_num 8550) 2020-06-12T02:38:41.812 INFO:tasks.rados.rados.0.smithi107.stdout:8551: done (2 left) 2020-06-12T02:38:41.812 INFO:tasks.rados.rados.0.smithi107.stdout:8554: done (1 left) 2020-06-12T02:38:41.812 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:41.812 INFO:tasks.rados.rados.0.smithi107.stdout:8555: seq_num 8554 ranges {0=466944,466944=466944,933888=548864,1482752=188416} 2020-06-12T02:38:41.819 INFO:tasks.rados.rados.0.smithi107.stdout:8555: writing smithi10729407-4 from 0 to 466944 tid 1 2020-06-12T02:38:41.819 INFO:tasks.rados.rados.0.smithi107.stdout:8555: writing smithi10729407-4 from 466944 to 933888 tid 2 2020-06-12T02:38:41.823 INFO:tasks.rados.rados.0.smithi107.stdout:8555: writing smithi10729407-4 from 933888 to 1482752 tid 3 2020-06-12T02:38:41.825 INFO:tasks.rados.rados.0.smithi107.stdout:8555: writing smithi10729407-4 from 1482752 to 1671168 tid 4 2020-06-12T02:38:41.825 INFO:tasks.rados.rados.0.smithi107.stdout:8556: delete oid 19 current snap is 0 2020-06-12T02:38:41.825 INFO:tasks.rados.rados.0.smithi107.stdout:8553: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:41.825 INFO:tasks.rados.rados.0.smithi107.stdout:8553: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:41.826 INFO:tasks.rados.rados.0.smithi107.stdout:8553: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:41.826 INFO:tasks.rados.rados.0.smithi107.stdout:8553: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:41.826 INFO:tasks.rados.rados.0.smithi107.stdout:8553: finishing write tid 6 to smithi10729407-3 2020-06-12T02:38:41.829 INFO:tasks.rados.rados.0.smithi107.stdout:8553: finishing write tid 7 to smithi10729407-3 2020-06-12T02:38:41.829 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1545 (ObjNum 8552 snap 0 seq_num 8552) dirty exists 2020-06-12T02:38:41.829 INFO:tasks.rados.rados.0.smithi107.stdout:8553: left oid 3 (ObjNum 8552 snap 0 seq_num 8552) 2020-06-12T02:38:41.830 INFO:tasks.rados.rados.0.smithi107.stdout:8553: done (2 left) 2020-06-12T02:38:41.830 INFO:tasks.rados.rados.0.smithi107.stdout:8556: done (1 left) 2020-06-12T02:38:41.830 INFO:tasks.rados.rados.0.smithi107.stdout:8557: delete oid 1 current snap is 0 2020-06-12T02:38:41.834 INFO:tasks.rados.rados.0.smithi107.stdout:8555: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:41.835 INFO:tasks.rados.rados.0.smithi107.stdout:8555: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:41.835 INFO:tasks.rados.rados.0.smithi107.stdout:8557: done (1 left) 2020-06-12T02:38:41.835 INFO:tasks.rados.rados.0.smithi107.stdout:8558: delete oid 14 current snap is 0 2020-06-12T02:38:41.836 INFO:tasks.rados.rados.0.smithi107.stdout:8555: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:41.836 INFO:tasks.rados.rados.0.smithi107.stdout:8555: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:41.836 INFO:tasks.rados.rados.0.smithi107.stdout:8555: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:41.838 INFO:tasks.rados.rados.0.smithi107.stdout:8555: finishing write tid 7 to smithi10729407-4 2020-06-12T02:38:41.839 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1618 (ObjNum 8554 snap 0 seq_num 8554) dirty exists 2020-06-12T02:38:41.839 INFO:tasks.rados.rados.0.smithi107.stdout:8555: left oid 4 (ObjNum 8554 snap 0 seq_num 8554) 2020-06-12T02:38:41.839 INFO:tasks.rados.rados.0.smithi107.stdout:8555: done (1 left) 2020-06-12T02:38:41.839 INFO:tasks.rados.rados.0.smithi107.stdout:8558: done (0 left) 2020-06-12T02:38:41.839 INFO:tasks.rados.rados.0.smithi107.stdout:8559: delete oid 13 current snap is 0 2020-06-12T02:38:41.841 INFO:tasks.rados.rados.0.smithi107.stdout:8559: done (0 left) 2020-06-12T02:38:41.842 INFO:tasks.rados.rados.0.smithi107.stdout:8560: delete oid 7 current snap is 0 2020-06-12T02:38:41.844 INFO:tasks.rados.rados.0.smithi107.stdout:8560: done (0 left) 2020-06-12T02:38:41.844 INFO:tasks.rados.rados.0.smithi107.stdout:8561: delete oid 13 current snap is 0 2020-06-12T02:38:41.846 INFO:tasks.rados.rados.0.smithi107.stdout:8561: done (0 left) 2020-06-12T02:38:41.846 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:38:41.847 INFO:tasks.rados.rados.0.smithi107.stdout:8562: seq_num 8561 ranges {0=589824,589824=507904,1097728=696320} 2020-06-12T02:38:41.851 INFO:tasks.rados.rados.0.smithi107.stdout:8562: writing smithi10729407-10 from 0 to 589824 tid 1 2020-06-12T02:38:41.856 INFO:tasks.rados.rados.0.smithi107.stdout:8562: writing smithi10729407-10 from 589824 to 1097728 tid 2 2020-06-12T02:38:41.860 INFO:tasks.rados.rados.0.smithi107.stdout:8562: writing smithi10729407-10 from 1097728 to 1794048 tid 3 2020-06-12T02:38:41.860 INFO:tasks.rados.rados.0.smithi107.stdout:8563: delete oid 19 current snap is 0 2020-06-12T02:38:41.862 INFO:tasks.rados.rados.0.smithi107.stdout:8562: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:41.866 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:41.864+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:41.868 INFO:tasks.rados.rados.0.smithi107.stdout:8563: done (1 left) 2020-06-12T02:38:41.868 INFO:tasks.rados.rados.0.smithi107.stdout:8564: delete oid 8 current snap is 0 2020-06-12T02:38:41.868 INFO:tasks.rados.rados.0.smithi107.stdout:8562: finishing write tid 2 to smithi10729407-10 2020-06-12T02:38:41.869 INFO:tasks.rados.rados.0.smithi107.stdout:8562: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:41.870 INFO:tasks.rados.rados.0.smithi107.stdout:8562: finishing write tid 5 to smithi10729407-10 2020-06-12T02:38:41.870 INFO:tasks.rados.rados.0.smithi107.stdout:8564: done (1 left) 2020-06-12T02:38:41.871 INFO:tasks.rados.rados.0.smithi107.stdout:8565: delete oid 18 current snap is 0 2020-06-12T02:38:41.872 INFO:tasks.rados.rados.0.smithi107.stdout:8562: finishing write tid 6 to smithi10729407-10 2020-06-12T02:38:41.872 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 625 (ObjNum 8561 snap 0 seq_num 8561) dirty exists 2020-06-12T02:38:41.872 INFO:tasks.rados.rados.0.smithi107.stdout:8562: left oid 10 (ObjNum 8561 snap 0 seq_num 8561) 2020-06-12T02:38:41.874 INFO:tasks.rados.rados.0.smithi107.stdout:8562: done (1 left) 2020-06-12T02:38:41.874 INFO:tasks.rados.rados.0.smithi107.stdout:8565: done (0 left) 2020-06-12T02:38:41.874 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:41.874 INFO:tasks.rados.rados.0.smithi107.stdout:8566: seq_num 8565 ranges {0=655360,655360=450560,1105920=679936,1785856=73728} 2020-06-12T02:38:41.878 INFO:tasks.rados.rados.0.smithi107.stdout:8566: writing smithi10729407-9 from 0 to 655360 tid 1 2020-06-12T02:38:41.882 INFO:tasks.rados.rados.0.smithi107.stdout:8566: writing smithi10729407-9 from 655360 to 1105920 tid 2 2020-06-12T02:38:41.887 INFO:tasks.rados.rados.0.smithi107.stdout:8566: writing smithi10729407-9 from 1105920 to 1785856 tid 3 2020-06-12T02:38:41.890 INFO:tasks.rados.rados.0.smithi107.stdout:8566: writing smithi10729407-9 from 1785856 to 1859584 tid 4 2020-06-12T02:38:41.891 INFO:tasks.rados.rados.0.smithi107.stdout:8567: delete oid 3 current snap is 0 2020-06-12T02:38:41.891 INFO:tasks.rados.rados.0.smithi107.stdout:8566: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:41.901 INFO:tasks.rados.rados.0.smithi107.stdout:8567: done (1 left) 2020-06-12T02:38:41.902 INFO:tasks.rados.rados.0.smithi107.stdout:8568: delete oid 14 current snap is 0 2020-06-12T02:38:41.903 INFO:tasks.rados.rados.0.smithi107.stdout:8566: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:41.905 INFO:tasks.rados.rados.0.smithi107.stdout:8566: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:41.905 INFO:tasks.rados.rados.0.smithi107.stdout:8568: done (1 left) 2020-06-12T02:38:41.905 INFO:tasks.rados.rados.0.smithi107.stdout:8569: delete oid 7 current snap is 0 2020-06-12T02:38:41.906 INFO:tasks.rados.rados.0.smithi107.stdout:8566: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:41.906 INFO:tasks.rados.rados.0.smithi107.stdout:8566: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:41.908 INFO:tasks.rados.rados.0.smithi107.stdout:8566: finishing write tid 7 to smithi10729407-9 2020-06-12T02:38:41.908 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1601 (ObjNum 8565 snap 0 seq_num 8565) dirty exists 2020-06-12T02:38:41.908 INFO:tasks.rados.rados.0.smithi107.stdout:8566: left oid 9 (ObjNum 8565 snap 0 seq_num 8565) 2020-06-12T02:38:41.921 INFO:tasks.rados.rados.0.smithi107.stdout:8566: done (1 left) 2020-06-12T02:38:41.921 INFO:tasks.rados.rados.0.smithi107.stdout:8569: done (0 left) 2020-06-12T02:38:41.921 INFO:tasks.rados.rados.0.smithi107.stdout:8570: delete oid 4 current snap is 0 2020-06-12T02:38:41.923 INFO:tasks.rados.rados.0.smithi107.stdout:8570: done (0 left) 2020-06-12T02:38:41.924 INFO:tasks.rados.rados.0.smithi107.stdout:8571: delete oid 9 current snap is 0 2020-06-12T02:38:41.927 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:41.927 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:38:41.928 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2020-06-12T02:38:41.928 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:41.928 INFO:tasks.rados.rados.0.smithi107.stdout:8571: done (0 left) 2020-06-12T02:38:41.929 INFO:tasks.rados.rados.0.smithi107.stdout:8572: delete oid 3 current snap is 0 2020-06-12T02:38:41.934 INFO:tasks.rados.rados.0.smithi107.stdout:8572: done (0 left) 2020-06-12T02:38:41.934 INFO:tasks.rados.rados.0.smithi107.stdout:8573: delete oid 5 current snap is 0 2020-06-12T02:38:41.937 INFO:tasks.rados.rados.0.smithi107.stdout:8573: done (0 left) 2020-06-12T02:38:41.937 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:38:41.937 INFO:tasks.rados.rados.0.smithi107.stdout:8574: seq_num 8573 ranges {0=786432,786432=770048,1556480=434176} 2020-06-12T02:38:41.938 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-06-12T02:38:41.943 INFO:tasks.rados.rados.0.smithi107.stdout:8574: writing smithi10729407-5 from 0 to 786432 tid 1 2020-06-12T02:38:41.949 INFO:tasks.rados.rados.0.smithi107.stdout:8574: writing smithi10729407-5 from 786432 to 1556480 tid 2 2020-06-12T02:38:41.953 INFO:tasks.rados.rados.0.smithi107.stdout:8574: writing smithi10729407-5 from 1556480 to 1990656 tid 3 2020-06-12T02:38:41.953 INFO:tasks.rados.rados.0.smithi107.stdout:8575: delete oid 3 current snap is 0 2020-06-12T02:38:41.953 INFO:tasks.rados.rados.0.smithi107.stdout:8574: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:41.956 INFO:tasks.rados.rados.0.smithi107.stdout:8575: done (1 left) 2020-06-12T02:38:41.956 INFO:tasks.rados.rados.0.smithi107.stdout:8576: delete oid 18 current snap is 0 2020-06-12T02:38:41.960 INFO:tasks.rados.rados.0.smithi107.stdout:8576: done (1 left) 2020-06-12T02:38:41.961 INFO:tasks.rados.rados.0.smithi107.stdout:8577: delete oid 18 current snap is 0 2020-06-12T02:38:41.963 INFO:tasks.rados.rados.0.smithi107.stdout:8574: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:41.964 INFO:tasks.rados.rados.0.smithi107.stdout:8577: done (1 left) 2020-06-12T02:38:41.964 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:41.964 INFO:tasks.rados.rados.0.smithi107.stdout:8578: seq_num 8577 ranges {0=450560,450560=761856,1212416=622592,1835008=221184} 2020-06-12T02:38:41.966 INFO:tasks.rados.rados.0.smithi107.stdout:8578: writing smithi10729407-1 from 0 to 450560 tid 1 2020-06-12T02:38:41.967 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:41.964+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:41.971 INFO:tasks.rados.rados.0.smithi107.stdout:8578: writing smithi10729407-1 from 450560 to 1212416 tid 2 2020-06-12T02:38:41.975 INFO:tasks.rados.rados.0.smithi107.stdout:8578: writing smithi10729407-1 from 1212416 to 1835008 tid 3 2020-06-12T02:38:41.977 INFO:tasks.rados.rados.0.smithi107.stdout:8578: writing smithi10729407-1 from 1835008 to 2056192 tid 4 2020-06-12T02:38:41.977 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:38:41.977 INFO:tasks.rados.rados.0.smithi107.stdout:8579: seq_num 8578 ranges {0=466944,466944=647168,1114112=425984,1540096=532480} 2020-06-12T02:38:41.979 INFO:tasks.rados.rados.0.smithi107.stdout:8579: writing smithi10729407-11 from 0 to 466944 tid 1 2020-06-12T02:38:41.984 INFO:tasks.rados.rados.0.smithi107.stdout:8579: writing smithi10729407-11 from 466944 to 1114112 tid 2 2020-06-12T02:38:41.987 INFO:tasks.rados.rados.0.smithi107.stdout:8579: writing smithi10729407-11 from 1114112 to 1540096 tid 3 2020-06-12T02:38:41.991 INFO:tasks.rados.rados.0.smithi107.stdout:8579: writing smithi10729407-11 from 1540096 to 2072576 tid 4 2020-06-12T02:38:41.991 INFO:tasks.rados.rados.0.smithi107.stdout:8580: delete oid 8 current snap is 0 2020-06-12T02:38:41.992 INFO:tasks.rados.rados.0.smithi107.stdout:8574: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:41.992 INFO:tasks.rados.rados.0.smithi107.stdout:8574: finishing write tid 5 to smithi10729407-5 2020-06-12T02:38:41.992 INFO:tasks.rados.rados.0.smithi107.stdout:8574: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:41.992 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 850 (ObjNum 8573 snap 0 seq_num 8573) dirty exists 2020-06-12T02:38:41.992 INFO:tasks.rados.rados.0.smithi107.stdout:8574: left oid 5 (ObjNum 8573 snap 0 seq_num 8573) 2020-06-12T02:38:41.993 INFO:tasks.rados.rados.0.smithi107.stdout:8578: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:41.993 INFO:tasks.rados.rados.0.smithi107.stdout:8578: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:41.993 INFO:tasks.rados.rados.0.smithi107.stdout:8578: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:41.994 INFO:tasks.rados.rados.0.smithi107.stdout:8579: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:41.994 INFO:tasks.rados.rados.0.smithi107.stdout:8578: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:41.994 INFO:tasks.rados.rados.0.smithi107.stdout:8574: done (3 left) 2020-06-12T02:38:41.995 INFO:tasks.rados.rados.0.smithi107.stdout:8580: done (2 left) 2020-06-12T02:38:41.995 INFO:tasks.rados.rados.0.smithi107.stdout:8581: delete oid 9 current snap is 0 2020-06-12T02:38:41.999 INFO:tasks.rados.rados.0.smithi107.stdout:8581: done (2 left) 2020-06-12T02:38:41.999 INFO:tasks.rados.rados.0.smithi107.stdout:8582: delete oid 4 current snap is 0 2020-06-12T02:38:42.001 INFO:tasks.rados.rados.0.smithi107.stdout:8578: finishing write tid 6 to smithi10729407-1 2020-06-12T02:38:42.003 INFO:tasks.rados.rados.0.smithi107.stdout:8578: finishing write tid 7 to smithi10729407-1 2020-06-12T02:38:42.003 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1558 (ObjNum 8577 snap 0 seq_num 8577) dirty exists 2020-06-12T02:38:42.003 INFO:tasks.rados.rados.0.smithi107.stdout:8578: left oid 1 (ObjNum 8577 snap 0 seq_num 8577) 2020-06-12T02:38:42.004 INFO:tasks.rados.rados.0.smithi107.stdout:8578: done (2 left) 2020-06-12T02:38:42.004 INFO:tasks.rados.rados.0.smithi107.stdout:8582: done (1 left) 2020-06-12T02:38:42.005 INFO:tasks.rados.rados.0.smithi107.stdout:8583: delete oid 7 current snap is 0 2020-06-12T02:38:42.006 INFO:tasks.rados.rados.0.smithi107.stdout:8579: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:42.010 INFO:tasks.rados.rados.0.smithi107.stdout:8579: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:42.010 INFO:tasks.rados.rados.0.smithi107.stdout:8579: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:42.011 INFO:tasks.rados.rados.0.smithi107.stdout:8579: finishing write tid 6 to smithi10729407-11 2020-06-12T02:38:42.011 INFO:tasks.rados.rados.0.smithi107.stdout:8583: done (1 left) 2020-06-12T02:38:42.011 INFO:tasks.rados.rados.0.smithi107.stdout:8584: delete oid 4 current snap is 0 2020-06-12T02:38:42.012 INFO:tasks.rados.rados.0.smithi107.stdout:8579: finishing write tid 7 to smithi10729407-11 2020-06-12T02:38:42.013 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1504 (ObjNum 8578 snap 0 seq_num 8578) dirty exists 2020-06-12T02:38:42.013 INFO:tasks.rados.rados.0.smithi107.stdout:8579: left oid 11 (ObjNum 8578 snap 0 seq_num 8578) 2020-06-12T02:38:42.014 INFO:tasks.rados.rados.0.smithi107.stdout:8579: done (1 left) 2020-06-12T02:38:42.014 INFO:tasks.rados.rados.0.smithi107.stdout:8584: done (0 left) 2020-06-12T02:38:42.014 INFO:tasks.rados.rados.0.smithi107.stdout:8585: delete oid 9 current snap is 0 2020-06-12T02:38:42.017 INFO:tasks.rados.rados.0.smithi107.stdout:8585: done (0 left) 2020-06-12T02:38:42.017 INFO:tasks.rados.rados.0.smithi107.stdout:8586: delete oid 8 current snap is 0 2020-06-12T02:38:42.020 INFO:tasks.rados.rados.0.smithi107.stdout:8586: done (0 left) 2020-06-12T02:38:42.020 INFO:tasks.rados.rados.0.smithi107.stdout:8587: delete oid 6 current snap is 0 2020-06-12T02:38:42.026 INFO:tasks.rados.rados.0.smithi107.stdout:8587: done (0 left) 2020-06-12T02:38:42.027 INFO:tasks.rados.rados.0.smithi107.stdout:8588: delete oid 14 current snap is 0 2020-06-12T02:38:42.027 INFO:tasks.rados.rados.0.smithi107.stdout:8588: done (0 left) 2020-06-12T02:38:42.027 INFO:tasks.rados.rados.0.smithi107.stdout:8589: delete oid 10 current snap is 0 2020-06-12T02:38:42.027 INFO:tasks.rados.rados.0.smithi107.stdout:8589: done (0 left) 2020-06-12T02:38:42.028 INFO:tasks.rados.rados.0.smithi107.stdout:8590: delete oid 8 current snap is 0 2020-06-12T02:38:42.030 INFO:tasks.rados.rados.0.smithi107.stdout:8590: done (0 left) 2020-06-12T02:38:42.031 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:42.031 INFO:tasks.rados.rados.0.smithi107.stdout:8591: seq_num 8590 ranges {3776512=671744,4448256=507904,4956160=770048,5726208=327680} 2020-06-12T02:38:42.054 INFO:tasks.rados.rados.0.smithi107.stdout:8591: writing smithi10729407-17 from 3776512 to 4448256 tid 1 2020-06-12T02:38:42.057 INFO:tasks.rados.rados.0.smithi107.stdout:8591: writing smithi10729407-17 from 4448256 to 4956160 tid 2 2020-06-12T02:38:42.062 INFO:tasks.rados.rados.0.smithi107.stdout:8591: writing smithi10729407-17 from 4956160 to 5726208 tid 3 2020-06-12T02:38:42.064 INFO:tasks.rados.rados.0.smithi107.stdout:8591: writing smithi10729407-17 from 5726208 to 6053888 tid 4 2020-06-12T02:38:42.064 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:42.064 INFO:tasks.rados.rados.0.smithi107.stdout:8592: seq_num 8591 ranges {0=688128,688128=442368,1130496=573440,1703936=589824} 2020-06-12T02:38:42.067 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:42.065+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:42.068 INFO:tasks.rados.rados.0.smithi107.stdout:8592: writing smithi10729407-4 from 0 to 688128 tid 1 2020-06-12T02:38:42.071 INFO:tasks.rados.rados.0.smithi107.stdout:8592: writing smithi10729407-4 from 688128 to 1130496 tid 2 2020-06-12T02:38:42.075 INFO:tasks.rados.rados.0.smithi107.stdout:8592: writing smithi10729407-4 from 1130496 to 1703936 tid 3 2020-06-12T02:38:42.079 INFO:tasks.rados.rados.0.smithi107.stdout:8592: writing smithi10729407-4 from 1703936 to 2293760 tid 4 2020-06-12T02:38:42.079 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:42.079 INFO:tasks.rados.rados.0.smithi107.stdout:8593: seq_num 8592 ranges {0=704512,704512=720896,1425408=786432,2211840=98304} 2020-06-12T02:38:42.083 INFO:tasks.rados.rados.0.smithi107.stdout:8593: writing smithi10729407-7 from 0 to 704512 tid 1 2020-06-12T02:38:42.088 INFO:tasks.rados.rados.0.smithi107.stdout:8593: writing smithi10729407-7 from 704512 to 1425408 tid 2 2020-06-12T02:38:42.093 INFO:tasks.rados.rados.0.smithi107.stdout:8593: writing smithi10729407-7 from 1425408 to 2211840 tid 3 2020-06-12T02:38:42.094 INFO:tasks.rados.rados.0.smithi107.stdout:8593: writing smithi10729407-7 from 2211840 to 2310144 tid 4 2020-06-12T02:38:42.094 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:42.094 INFO:tasks.rados.rados.0.smithi107.stdout:8594: seq_num 8593 ranges {0=720896,720896=606208,1327104=540672,1867776=442368,2310144=16384} 2020-06-12T02:38:42.098 INFO:tasks.rados.rados.0.smithi107.stdout:8594: writing smithi10729407-12 from 0 to 720896 tid 1 2020-06-12T02:38:42.099 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:42.099 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:38:42.100 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2020-06-12T02:38:42.100 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2020-06-12T02:38:42.100 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:42.102 INFO:tasks.rados.rados.0.smithi107.stdout:8594: writing smithi10729407-12 from 720896 to 1327104 tid 2 2020-06-12T02:38:42.106 INFO:tasks.rados.rados.0.smithi107.stdout:8594: writing smithi10729407-12 from 1327104 to 1867776 tid 3 2020-06-12T02:38:42.109 INFO:tasks.rados.rados.0.smithi107.stdout:8594: writing smithi10729407-12 from 1867776 to 2310144 tid 4 2020-06-12T02:38:42.109 INFO:tasks.rados.rados.0.smithi107.stdout:8594: writing smithi10729407-12 from 2310144 to 2326528 tid 5 2020-06-12T02:38:42.109 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:42.109 INFO:tasks.rados.rados.0.smithi107.stdout:8595: seq_num 8594 ranges {1540096=737280,2277376=491520,2768896=753664,3522560=360448} 2020-06-12T02:38:42.112 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-06-12T02:38:42.123 INFO:tasks.rados.rados.0.smithi107.stdout:8595: writing smithi10729407-2 from 1540096 to 2277376 tid 1 2020-06-12T02:38:42.126 INFO:tasks.rados.rados.0.smithi107.stdout:8595: writing smithi10729407-2 from 2277376 to 2768896 tid 2 2020-06-12T02:38:42.131 INFO:tasks.rados.rados.0.smithi107.stdout:8595: writing smithi10729407-2 from 2768896 to 3522560 tid 3 2020-06-12T02:38:42.134 INFO:tasks.rados.rados.0.smithi107.stdout:8595: writing smithi10729407-2 from 3522560 to 3883008 tid 4 2020-06-12T02:38:42.134 INFO:tasks.rados.rados.0.smithi107.stdout:8596: delete oid 11 current snap is 0 2020-06-12T02:38:42.135 INFO:tasks.rados.rados.0.smithi107.stdout:8591: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:42.135 INFO:tasks.rados.rados.0.smithi107.stdout:8591: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:42.135 INFO:tasks.rados.rados.0.smithi107.stdout:8591: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:42.135 INFO:tasks.rados.rados.0.smithi107.stdout:8591: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:42.136 INFO:tasks.rados.rados.0.smithi107.stdout:8591: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:42.136 INFO:tasks.rados.rados.0.smithi107.stdout:8592: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:42.136 INFO:tasks.rados.rados.0.smithi107.stdout:8591: finishing write tid 7 to smithi10729407-17 2020-06-12T02:38:42.136 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1626 (ObjNum 8590 snap 0 seq_num 8590) dirty exists 2020-06-12T02:38:42.136 INFO:tasks.rados.rados.0.smithi107.stdout:8591: left oid 17 (ObjNum 8590 snap 0 seq_num 8590) 2020-06-12T02:38:42.137 INFO:tasks.rados.rados.0.smithi107.stdout:8592: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:42.137 INFO:tasks.rados.rados.0.smithi107.stdout:8593: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:42.137 INFO:tasks.rados.rados.0.smithi107.stdout:8592: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:42.137 INFO:tasks.rados.rados.0.smithi107.stdout:8592: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:42.137 INFO:tasks.rados.rados.0.smithi107.stdout:8593: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:42.138 INFO:tasks.rados.rados.0.smithi107.stdout:8592: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:42.138 INFO:tasks.rados.rados.0.smithi107.stdout:8593: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:42.138 INFO:tasks.rados.rados.0.smithi107.stdout:8592: finishing write tid 7 to smithi10729407-4 2020-06-12T02:38:42.138 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1631 (ObjNum 8591 snap 0 seq_num 8591) dirty exists 2020-06-12T02:38:42.138 INFO:tasks.rados.rados.0.smithi107.stdout:8592: left oid 4 (ObjNum 8591 snap 0 seq_num 8591) 2020-06-12T02:38:42.139 INFO:tasks.rados.rados.0.smithi107.stdout:8593: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:42.139 INFO:tasks.rados.rados.0.smithi107.stdout:8593: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:42.139 INFO:tasks.rados.rados.0.smithi107.stdout:8593: finishing write tid 7 to smithi10729407-7 2020-06-12T02:38:42.139 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1554 (ObjNum 8592 snap 0 seq_num 8592) dirty exists 2020-06-12T02:38:42.139 INFO:tasks.rados.rados.0.smithi107.stdout:8593: left oid 7 (ObjNum 8592 snap 0 seq_num 8592) 2020-06-12T02:38:42.139 INFO:tasks.rados.rados.0.smithi107.stdout:8594: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:42.140 INFO:tasks.rados.rados.0.smithi107.stdout:8594: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:42.140 INFO:tasks.rados.rados.0.smithi107.stdout:8594: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:42.140 INFO:tasks.rados.rados.0.smithi107.stdout:8594: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:42.140 INFO:tasks.rados.rados.0.smithi107.stdout:8594: finishing write tid 5 to smithi10729407-12 2020-06-12T02:38:42.140 INFO:tasks.rados.rados.0.smithi107.stdout:8594: finishing write tid 7 to smithi10729407-12 2020-06-12T02:38:42.140 INFO:tasks.rados.rados.0.smithi107.stdout:8594: finishing write tid 8 to smithi10729407-12 2020-06-12T02:38:42.141 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1577 (ObjNum 8593 snap 0 seq_num 8593) dirty exists 2020-06-12T02:38:42.141 INFO:tasks.rados.rados.0.smithi107.stdout:8594: left oid 12 (ObjNum 8593 snap 0 seq_num 8593) 2020-06-12T02:38:42.142 INFO:tasks.rados.rados.0.smithi107.stdout:8595: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:42.142 INFO:tasks.rados.rados.0.smithi107.stdout:8591: done (5 left) 2020-06-12T02:38:42.143 INFO:tasks.rados.rados.0.smithi107.stdout:8592: done (4 left) 2020-06-12T02:38:42.143 INFO:tasks.rados.rados.0.smithi107.stdout:8593: done (3 left) 2020-06-12T02:38:42.143 INFO:tasks.rados.rados.0.smithi107.stdout:8594: done (2 left) 2020-06-12T02:38:42.143 INFO:tasks.rados.rados.0.smithi107.stdout:8596: done (1 left) 2020-06-12T02:38:42.143 INFO:tasks.rados.rados.0.smithi107.stdout:8597: delete oid 8 current snap is 0 2020-06-12T02:38:42.143 INFO:tasks.rados.rados.0.smithi107.stdout:8595: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:42.144 INFO:tasks.rados.rados.0.smithi107.stdout:8595: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:42.144 INFO:tasks.rados.rados.0.smithi107.stdout:8595: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:42.144 INFO:tasks.rados.rados.0.smithi107.stdout:8595: finishing write tid 6 to smithi10729407-2 2020-06-12T02:38:42.145 INFO:tasks.rados.rados.0.smithi107.stdout:8595: finishing write tid 7 to smithi10729407-2 2020-06-12T02:38:42.145 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 802 (ObjNum 8594 snap 0 seq_num 8594) dirty exists 2020-06-12T02:38:42.145 INFO:tasks.rados.rados.0.smithi107.stdout:8595: left oid 2 (ObjNum 8594 snap 0 seq_num 8594) 2020-06-12T02:38:42.146 INFO:tasks.rados.rados.0.smithi107.stdout:8595: done (1 left) 2020-06-12T02:38:42.146 INFO:tasks.rados.rados.0.smithi107.stdout:8597: done (0 left) 2020-06-12T02:38:42.146 INFO:tasks.rados.rados.0.smithi107.stdout:8598: delete oid 3 current snap is 0 2020-06-12T02:38:42.149 INFO:tasks.rados.rados.0.smithi107.stdout:8598: done (0 left) 2020-06-12T02:38:42.149 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:42.149 INFO:tasks.rados.rados.0.smithi107.stdout:8599: seq_num 8598 ranges {2310144=802816,3112960=425984,3538944=729088,4268032=450560} 2020-06-12T02:38:42.168 INFO:tasks.rados.rados.0.smithi107.stdout:8599: writing smithi10729407-7 from 2310144 to 3112960 tid 1 2020-06-12T02:38:42.168 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:42.165+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:42.170 INFO:tasks.rados.rados.0.smithi107.stdout:8599: writing smithi10729407-7 from 3112960 to 3538944 tid 2 2020-06-12T02:38:42.180 INFO:tasks.rados.rados.0.smithi107.stdout:8599: writing smithi10729407-7 from 3538944 to 4268032 tid 3 2020-06-12T02:38:42.180 INFO:tasks.rados.rados.0.smithi107.stdout:8599: writing smithi10729407-7 from 4268032 to 4718592 tid 4 2020-06-12T02:38:42.180 INFO:tasks.rados.rados.0.smithi107.stdout:8600: delete oid 12 current snap is 0 2020-06-12T02:38:42.181 INFO:tasks.rados.rados.0.smithi107.stdout:8599: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:42.181 INFO:tasks.rados.rados.0.smithi107.stdout:8599: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:42.183 INFO:tasks.rados.rados.0.smithi107.stdout:8600: done (1 left) 2020-06-12T02:38:42.183 INFO:tasks.rados.rados.0.smithi107.stdout:8601: delete oid 5 current snap is 0 2020-06-12T02:38:42.186 INFO:tasks.rados.rados.0.smithi107.stdout:8601: done (1 left) 2020-06-12T02:38:42.186 INFO:tasks.rados.rados.0.smithi107.stdout:8602: delete oid 5 current snap is 0 2020-06-12T02:38:42.187 INFO:tasks.rados.rados.0.smithi107.stdout:8599: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:42.189 INFO:tasks.rados.rados.0.smithi107.stdout:8599: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:42.190 INFO:tasks.rados.rados.0.smithi107.stdout:8599: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:42.190 INFO:tasks.rados.rados.0.smithi107.stdout:8599: finishing write tid 7 to smithi10729407-7 2020-06-12T02:38:42.191 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1559 (ObjNum 8598 snap 0 seq_num 8598) dirty exists 2020-06-12T02:38:42.191 INFO:tasks.rados.rados.0.smithi107.stdout:8599: left oid 7 (ObjNum 8598 snap 0 seq_num 8598) 2020-06-12T02:38:42.191 INFO:tasks.rados.rados.0.smithi107.stdout:8599: done (1 left) 2020-06-12T02:38:42.191 INFO:tasks.rados.rados.0.smithi107.stdout:8602: done (0 left) 2020-06-12T02:38:42.192 INFO:tasks.rados.rados.0.smithi107.stdout:8603: delete oid 6 current snap is 0 2020-06-12T02:38:42.193 INFO:tasks.rados.rados.0.smithi107.stdout:8603: done (0 left) 2020-06-12T02:38:42.193 INFO:tasks.rados.rados.0.smithi107.stdout:8604: delete oid 17 current snap is 0 2020-06-12T02:38:42.196 INFO:tasks.rados.rados.0.smithi107.stdout:8604: done (0 left) 2020-06-12T02:38:42.196 INFO:tasks.rados.rados.0.smithi107.stdout:8605: delete oid 6 current snap is 0 2020-06-12T02:38:42.200 INFO:tasks.rados.rados.0.smithi107.stdout:8605: done (0 left) 2020-06-12T02:38:42.200 INFO:tasks.rados.rados.0.smithi107.stdout:8606: delete oid 1 current snap is 0 2020-06-12T02:38:42.202 INFO:tasks.rados.rados.0.smithi107.stdout:8606: done (0 left) 2020-06-12T02:38:42.203 INFO:tasks.rados.rados.0.smithi107.stdout:8607: delete oid 11 current snap is 0 2020-06-12T02:38:42.205 INFO:tasks.rados.rados.0.smithi107.stdout:8607: done (0 left) 2020-06-12T02:38:42.205 INFO:tasks.rados.rados.0.smithi107.stdout:8608: delete oid 10 current snap is 0 2020-06-12T02:38:42.207 INFO:tasks.rados.rados.0.smithi107.stdout:8608: done (0 left) 2020-06-12T02:38:42.207 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:42.207 INFO:tasks.rados.rados.0.smithi107.stdout:8609: seq_num 8608 ranges {0=172032} 2020-06-12T02:38:42.208 INFO:tasks.rados.rados.0.smithi107.stdout:8609: writing smithi10729407-1 from 0 to 172032 tid 1 2020-06-12T02:38:42.208 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:42.208 INFO:tasks.rados.rados.0.smithi107.stdout:8610: seq_num 8609 ranges {0=188416} 2020-06-12T02:38:42.209 INFO:tasks.rados.rados.0.smithi107.stdout:8610: writing smithi10729407-19 from 0 to 188416 tid 1 2020-06-12T02:38:42.209 INFO:tasks.rados.rados.0.smithi107.stdout:8611: delete oid 15 current snap is 0 2020-06-12T02:38:42.211 INFO:tasks.rados.rados.0.smithi107.stdout:8609: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:42.214 INFO:tasks.rados.rados.0.smithi107.stdout:8611: done (2 left) 2020-06-12T02:38:42.214 INFO:tasks.rados.rados.0.smithi107.stdout:8612: delete oid 15 current snap is 0 2020-06-12T02:38:42.215 INFO:tasks.rados.rados.0.smithi107.stdout:8609: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:42.216 INFO:tasks.rados.rados.0.smithi107.stdout:8610: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:42.219 INFO:tasks.rados.rados.0.smithi107.stdout:8609: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:42.219 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1564 (ObjNum 8608 snap 0 seq_num 8608) dirty exists 2020-06-12T02:38:42.219 INFO:tasks.rados.rados.0.smithi107.stdout:8609: left oid 1 (ObjNum 8608 snap 0 seq_num 8608) 2020-06-12T02:38:42.220 INFO:tasks.rados.rados.0.smithi107.stdout:8609: done (2 left) 2020-06-12T02:38:42.220 INFO:tasks.rados.rados.0.smithi107.stdout:8612: done (1 left) 2020-06-12T02:38:42.220 INFO:tasks.rados.rados.0.smithi107.stdout:8613: delete oid 15 current snap is 0 2020-06-12T02:38:42.220 INFO:tasks.rados.rados.0.smithi107.stdout:8610: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:42.221 INFO:tasks.rados.rados.0.smithi107.stdout:8610: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:42.222 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1580 (ObjNum 8609 snap 0 seq_num 8609) dirty exists 2020-06-12T02:38:42.222 INFO:tasks.rados.rados.0.smithi107.stdout:8610: left oid 19 (ObjNum 8609 snap 0 seq_num 8609) 2020-06-12T02:38:42.223 INFO:tasks.rados.rados.0.smithi107.stdout:8610: done (1 left) 2020-06-12T02:38:42.223 INFO:tasks.rados.rados.0.smithi107.stdout:8613: done (0 left) 2020-06-12T02:38:42.223 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:38:42.223 INFO:tasks.rados.rados.0.smithi107.stdout:8614: seq_num 8613 ranges {0=253952} 2020-06-12T02:38:42.224 INFO:tasks.rados.rados.0.smithi107.stdout:8614: writing smithi10729407-5 from 0 to 253952 tid 1 2020-06-12T02:38:42.225 INFO:tasks.rados.rados.0.smithi107.stdout:8615: delete oid 12 current snap is 0 2020-06-12T02:38:42.231 INFO:tasks.rados.rados.0.smithi107.stdout:8615: done (1 left) 2020-06-12T02:38:42.232 INFO:tasks.rados.rados.0.smithi107.stdout:8616: delete oid 15 current snap is 0 2020-06-12T02:38:42.232 INFO:tasks.rados.rados.0.smithi107.stdout:8614: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:42.232 INFO:tasks.rados.rados.0.smithi107.stdout:8616: done (1 left) 2020-06-12T02:38:42.232 INFO:tasks.rados.rados.0.smithi107.stdout:8617: delete oid 15 current snap is 0 2020-06-12T02:38:42.233 INFO:tasks.rados.rados.0.smithi107.stdout:8614: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:42.233 INFO:tasks.rados.rados.0.smithi107.stdout:8617: done (1 left) 2020-06-12T02:38:42.234 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:42.234 INFO:tasks.rados.rados.0.smithi107.stdout:8618: seq_num 8617 ranges {0=319488} 2020-06-12T02:38:42.236 INFO:tasks.rados.rados.0.smithi107.stdout:8618: writing smithi10729407-9 from 0 to 319488 tid 1 2020-06-12T02:38:42.236 INFO:tasks.rados.rados.0.smithi107.stdout:8619: delete oid 17 current snap is 0 2020-06-12T02:38:42.236 INFO:tasks.rados.rados.0.smithi107.stdout:8614: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:42.236 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 854 (ObjNum 8613 snap 0 seq_num 8613) dirty exists 2020-06-12T02:38:42.236 INFO:tasks.rados.rados.0.smithi107.stdout:8614: left oid 5 (ObjNum 8613 snap 0 seq_num 8613) 2020-06-12T02:38:42.240 INFO:tasks.rados.rados.0.smithi107.stdout:8614: done (2 left) 2020-06-12T02:38:42.240 INFO:tasks.rados.rados.0.smithi107.stdout:8619: done (1 left) 2020-06-12T02:38:42.240 INFO:tasks.rados.rados.0.smithi107.stdout:8620: delete oid 5 current snap is 0 2020-06-12T02:38:42.244 INFO:tasks.rados.rados.0.smithi107.stdout:8618: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:42.244 INFO:tasks.rados.rados.0.smithi107.stdout:8618: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:42.244 INFO:tasks.rados.rados.0.smithi107.stdout:8620: done (1 left) 2020-06-12T02:38:42.244 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:42.244 INFO:tasks.rados.rados.0.smithi107.stdout:8621: seq_num 8620 ranges {0=376832} 2020-06-12T02:38:42.245 INFO:tasks.rados.rados.0.smithi107.stdout:8621: writing smithi10729407-13 from 0 to 376832 tid 1 2020-06-12T02:38:42.245 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:42.246 INFO:tasks.rados.rados.0.smithi107.stdout:8622: seq_num 8621 ranges {0=393216} 2020-06-12T02:38:42.246 INFO:tasks.rados.rados.0.smithi107.stdout:8622: writing smithi10729407-8 from 0 to 393216 tid 1 2020-06-12T02:38:42.247 INFO:tasks.rados.rados.0.smithi107.stdout:8623: delete oid 2 current snap is 0 2020-06-12T02:38:42.247 INFO:tasks.rados.rados.0.smithi107.stdout:8618: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:42.247 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1608 (ObjNum 8617 snap 0 seq_num 8617) dirty exists 2020-06-12T02:38:42.247 INFO:tasks.rados.rados.0.smithi107.stdout:8618: left oid 9 (ObjNum 8617 snap 0 seq_num 8617) 2020-06-12T02:38:42.249 INFO:tasks.rados.rados.0.smithi107.stdout:8621: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:42.275 INFO:tasks.rados.rados.0.smithi107.stdout:8618: done (3 left) 2020-06-12T02:38:42.275 INFO:tasks.rados.rados.0.smithi107.stdout:8623: done (2 left) 2020-06-12T02:38:42.275 INFO:tasks.rados.rados.0.smithi107.stdout:8624: delete oid 4 current snap is 0 2020-06-12T02:38:42.276 INFO:tasks.rados.rados.0.smithi107.stdout:8622: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:42.276 INFO:tasks.rados.rados.0.smithi107.stdout:8622: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:42.276 INFO:tasks.rados.rados.0.smithi107.stdout:8621: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:42.276 INFO:tasks.rados.rados.0.smithi107.stdout:8622: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:42.276 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1605 (ObjNum 8621 snap 0 seq_num 8621) dirty exists 2020-06-12T02:38:42.276 INFO:tasks.rados.rados.0.smithi107.stdout:8622: left oid 8 (ObjNum 8621 snap 0 seq_num 8621) 2020-06-12T02:38:42.277 INFO:tasks.rados.rados.0.smithi107.stdout:8621: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:42.277 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1561 (ObjNum 8620 snap 0 seq_num 8620) dirty exists 2020-06-12T02:38:42.277 INFO:tasks.rados.rados.0.smithi107.stdout:8621: left oid 13 (ObjNum 8620 snap 0 seq_num 8620) 2020-06-12T02:38:42.277 INFO:tasks.rados.rados.0.smithi107.stdout:8621: done (2 left) 2020-06-12T02:38:42.277 INFO:tasks.rados.rados.0.smithi107.stdout:8622: done (1 left) 2020-06-12T02:38:42.286 INFO:tasks.rados.rados.0.smithi107.stdout:8624: done (0 left) 2020-06-12T02:38:42.286 INFO:tasks.rados.rados.0.smithi107.stdout:8625: delete oid 2 current snap is 0 2020-06-12T02:38:42.286 INFO:tasks.rados.rados.0.smithi107.stdout:8625: done (0 left) 2020-06-12T02:38:42.286 INFO:tasks.rados.rados.0.smithi107.stdout:8626: delete oid 14 current snap is 0 2020-06-12T02:38:42.287 INFO:tasks.rados.rados.0.smithi107.stdout:8626: done (0 left) 2020-06-12T02:38:42.287 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:42.287 INFO:tasks.rados.rados.0.smithi107.stdout:8627: seq_num 8626 ranges {0=475136} 2020-06-12T02:38:42.287 INFO:tasks.rados.rados.0.smithi107.stdout:8627: writing smithi10729407-12 from 0 to 475136 tid 1 2020-06-12T02:38:42.287 INFO:tasks.rados.rados.0.smithi107.stdout:8628: delete oid 5 current snap is 0 2020-06-12T02:38:42.288 INFO:tasks.rados.rados.0.smithi107.stdout:8628: done (1 left) 2020-06-12T02:38:42.288 INFO:tasks.rados.rados.0.smithi107.stdout:8629: delete oid 18 current snap is 0 2020-06-12T02:38:42.288 INFO:tasks.rados.rados.0.smithi107.stdout:8629: done (1 left) 2020-06-12T02:38:42.288 INFO:tasks.rados.rados.0.smithi107.stdout:8630: delete oid 6 current snap is 0 2020-06-12T02:38:42.289 INFO:tasks.rados.rados.0.smithi107.stdout:8627: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:42.291 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:38:42.267+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:38:42.292 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:42.292 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2020-06-12T02:38:42.292 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2020-06-12T02:38:42.292 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2020-06-12T02:38:42.293 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.293 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.3s2 49/21 ECSubWrite(tid=4109, reqid=client.4512.0:8841, at_version=49'773, trim_to=0'0, roll_forward_to=49'771))", 2020-06-12T02:38:42.293 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.822144+0000", 2020-06-12T02:38:42.293 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 30.450829348999999, 2020-06-12T02:38:42.293 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014799214, 2020-06-12T02:38:42.294 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:42.294 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:42.294 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:42.294 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.294 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:42.294 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822144+0000", 2020-06-12T02:38:42.295 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:42.295 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.295 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.298 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:42.298 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822144+0000", 2020-06-12T02:38:42.298 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00061470800000000005 2020-06-12T02:38:42.299 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.301 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.301 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:42.302 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822759+0000", 2020-06-12T02:38:42.302 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.7732000000000001e-05 2020-06-12T02:38:42.302 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.302 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.302 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:42.303 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822786+0000", 2020-06-12T02:38:42.303 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1010999999999999e-05 2020-06-12T02:38:42.303 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.303 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.305 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:42.305 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822807+0000", 2020-06-12T02:38:42.305 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.842e-06 2020-06-12T02:38:42.305 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.306 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.306 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:42.306 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822817+0000", 2020-06-12T02:38:42.306 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00024702300000000002 2020-06-12T02:38:42.306 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.307 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.307 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:42.307 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.823064+0000", 2020-06-12T02:38:42.307 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5207e-05 2020-06-12T02:38:42.308 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.308 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.308 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:42.308 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.823079+0000", 2020-06-12T02:38:42.308 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013806817000000001 2020-06-12T02:38:42.309 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.309 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.309 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:42.309 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.836886+0000", 2020-06-12T02:38:42.310 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6873999999999998e-05 2020-06-12T02:38:42.310 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.310 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.310 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:42.310 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.836943+0000", 2020-06-12T02:38:42.311 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014799214 2020-06-12T02:38:42.311 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:42.311 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:42.312 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:42.312 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.312 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.312 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14657 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~606208] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:42.312 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.209225+0000", 2020-06-12T02:38:42.313 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 11.063748098, 2020-06-12T02:38:42.313 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016128905999999998, 2020-06-12T02:38:42.313 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:42.313 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:42.313 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:42.313 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:42.314 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:42.314 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14657 2020-06-12T02:38:42.314 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.314 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:42.314 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.315 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:42.315 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209225+0000", 2020-06-12T02:38:42.315 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:42.316 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.316 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.316 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:42.316 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209225+0000", 2020-06-12T02:38:42.316 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018386699999999999 2020-06-12T02:38:42.317 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.317 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.317 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:42.317 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209409+0000", 2020-06-12T02:38:42.317 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8160000000000001e-06 2020-06-12T02:38:42.317 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.318 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.318 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:42.318 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209414+0000", 2020-06-12T02:38:42.318 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0230000000000004e-06 2020-06-12T02:38:42.319 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.319 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.319 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:42.319 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209421+0000", 2020-06-12T02:38:42.320 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0139999999999999e-06 2020-06-12T02:38:42.320 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.320 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.320 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:42.320 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209424+0000", 2020-06-12T02:38:42.321 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0054798169999999997 2020-06-12T02:38:42.321 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.321 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.321 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:42.321 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.214904+0000", 2020-06-12T02:38:42.322 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123063 2020-06-12T02:38:42.322 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.322 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.323 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:42.323 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215027+0000", 2020-06-12T02:38:42.323 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001159923 2020-06-12T02:38:42.323 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.323 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.324 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:42.324 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.216187+0000", 2020-06-12T02:38:42.324 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2835000000000002e-05 2020-06-12T02:38:42.324 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.324 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.325 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:42.325 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.216270+0000", 2020-06-12T02:38:42.325 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073575799999999998 2020-06-12T02:38:42.325 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.325 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.325 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:42.326 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.217005+0000", 2020-06-12T02:38:42.326 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0082434659999999996 2020-06-12T02:38:42.326 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.326 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.326 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:42.327 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225249+0000", 2020-06-12T02:38:42.327 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0145999999999997e-05 2020-06-12T02:38:42.327 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.327 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.327 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:42.327 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225299+0000", 2020-06-12T02:38:42.328 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5177999999999999e-05 2020-06-12T02:38:42.328 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.328 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.328 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:42.328 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225354+0000", 2020-06-12T02:38:42.329 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016128905999999998 2020-06-12T02:38:42.329 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:42.329 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:42.329 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:42.330 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.330 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.330 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14658 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~598016] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:42.330 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.209455+0000", 2020-06-12T02:38:42.330 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 11.063518344, 2020-06-12T02:38:42.331 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016447966000000001, 2020-06-12T02:38:42.331 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:42.332 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:42.332 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:42.332 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:42.332 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:42.333 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14658 2020-06-12T02:38:42.333 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.333 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:42.333 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.333 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:42.333 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209455+0000", 2020-06-12T02:38:42.334 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:42.334 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.334 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.334 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:42.334 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209455+0000", 2020-06-12T02:38:42.335 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037921599999999998 2020-06-12T02:38:42.335 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.335 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.335 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:42.335 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209834+0000", 2020-06-12T02:38:42.336 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8339999999999999e-06 2020-06-12T02:38:42.336 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.336 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.336 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:42.336 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209842+0000", 2020-06-12T02:38:42.336 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2564e-05 2020-06-12T02:38:42.337 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.337 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.337 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:42.337 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209855+0000", 2020-06-12T02:38:42.337 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.135e-06 2020-06-12T02:38:42.338 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.338 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.338 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:42.338 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209860+0000", 2020-06-12T02:38:42.338 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0052530290000000002 2020-06-12T02:38:42.338 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.339 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.339 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:42.339 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215113+0000", 2020-06-12T02:38:42.339 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6229000000000003e-05 2020-06-12T02:38:42.339 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.341 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.341 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:42.341 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215209+0000", 2020-06-12T02:38:42.342 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045765409999999999 2020-06-12T02:38:42.342 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.342 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.342 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:42.342 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.219785+0000", 2020-06-12T02:38:42.343 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000102846 2020-06-12T02:38:42.343 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.343 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.343 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:42.343 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.219888+0000", 2020-06-12T02:38:42.344 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00077088700000000005 2020-06-12T02:38:42.344 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.344 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.344 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:42.344 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.220659+0000", 2020-06-12T02:38:42.344 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046979739999999997 2020-06-12T02:38:42.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.346 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.346 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:42.346 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225357+0000", 2020-06-12T02:38:42.346 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00046069600000000002 2020-06-12T02:38:42.346 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.347 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.347 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:42.347 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225818+0000", 2020-06-12T02:38:42.347 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5014999999999998e-05 2020-06-12T02:38:42.347 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.347 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.348 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:42.348 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225903+0000", 2020-06-12T02:38:42.348 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016447966000000001 2020-06-12T02:38:42.348 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:42.348 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:42.349 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:42.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.349 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.349 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14659 3.0s0 3:0dddb690:::smithi10729407-10:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:42.349 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.209909+0000", 2020-06-12T02:38:42.350 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 11.063064135999999, 2020-06-12T02:38:42.350 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016618236000000002, 2020-06-12T02:38:42.350 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:42.350 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:42.350 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:42.350 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:42.351 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:42.351 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14659 2020-06-12T02:38:42.351 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.352 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:42.352 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.352 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:42.352 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209909+0000", 2020-06-12T02:38:42.352 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:42.353 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.353 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.353 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:42.353 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209909+0000", 2020-06-12T02:38:42.353 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8085999999999999e-05 2020-06-12T02:38:42.353 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.354 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.354 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:42.354 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209967+0000", 2020-06-12T02:38:42.354 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3169999999999998e-06 2020-06-12T02:38:42.354 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.355 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.355 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:42.355 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209974+0000", 2020-06-12T02:38:42.355 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1246999999999999e-05 2020-06-12T02:38:42.355 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.355 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.356 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:42.356 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209985+0000", 2020-06-12T02:38:42.356 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1439999999999996e-06 2020-06-12T02:38:42.356 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.356 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.357 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:42.357 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209989+0000", 2020-06-12T02:38:42.357 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0052537900000000004 2020-06-12T02:38:42.357 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.357 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.358 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:42.358 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215243+0000", 2020-06-12T02:38:42.358 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6023000000000002e-05 2020-06-12T02:38:42.358 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.358 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.358 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:42.359 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215299+0000", 2020-06-12T02:38:42.359 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0087846910000000007 2020-06-12T02:38:42.359 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.359 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.359 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:42.360 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.224083+0000", 2020-06-12T02:38:42.360 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8504000000000004e-05 2020-06-12T02:38:42.360 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.360 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.360 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:42.360 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.224182+0000", 2020-06-12T02:38:42.361 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044251400000000003 2020-06-12T02:38:42.361 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.361 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.361 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:42.362 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.224624+0000", 2020-06-12T02:38:42.362 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013383290000000001 2020-06-12T02:38:42.362 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.362 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.362 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:42.363 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225963+0000", 2020-06-12T02:38:42.363 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047942699999999999 2020-06-12T02:38:42.363 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.363 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.363 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:42.364 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.226442+0000", 2020-06-12T02:38:42.364 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5164e-05 2020-06-12T02:38:42.364 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.365 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.365 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:42.365 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.226527+0000", 2020-06-12T02:38:42.365 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016618236000000002 2020-06-12T02:38:42.365 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:42.366 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:42.366 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:42.366 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.366 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.366 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14660 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e59)", 2020-06-12T02:38:42.367 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.210020+0000", 2020-06-12T02:38:42.367 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 11.062953157999999, 2020-06-12T02:38:42.367 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017984777, 2020-06-12T02:38:42.367 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:42.367 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:42.367 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:42.368 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:42.368 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:42.368 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14660 2020-06-12T02:38:42.368 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.368 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:42.369 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.369 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:42.369 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210020+0000", 2020-06-12T02:38:42.369 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:42.369 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.370 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.370 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:42.370 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210020+0000", 2020-06-12T02:38:42.370 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2875999999999999e-05 2020-06-12T02:38:42.370 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.370 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.371 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:42.371 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210043+0000", 2020-06-12T02:38:42.371 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2200000000000001e-06 2020-06-12T02:38:42.371 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.371 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.372 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:42.372 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210046+0000", 2020-06-12T02:38:42.372 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7410000000000003e-06 2020-06-12T02:38:42.372 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.373 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.373 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:42.373 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210053+0000", 2020-06-12T02:38:42.373 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5610000000000001e-06 2020-06-12T02:38:42.373 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.374 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.374 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:42.374 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210055+0000", 2020-06-12T02:38:42.374 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0052596409999999998 2020-06-12T02:38:42.374 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.374 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.375 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:42.375 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215315+0000", 2020-06-12T02:38:42.375 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5627999999999997e-05 2020-06-12T02:38:42.375 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.375 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.376 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:42.376 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215371+0000", 2020-06-12T02:38:42.376 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0098069549999999991 2020-06-12T02:38:42.376 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.376 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.376 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:42.377 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225178+0000", 2020-06-12T02:38:42.377 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4530000000000001e-05 2020-06-12T02:38:42.377 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.377 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.378 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:42.379 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225232+0000", 2020-06-12T02:38:42.379 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017497350000000001 2020-06-12T02:38:42.379 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.379 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.379 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:42.380 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.226982+0000", 2020-06-12T02:38:42.380 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8154999999999998e-05 2020-06-12T02:38:42.380 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.380 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.380 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:42.381 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.227080+0000", 2020-06-12T02:38:42.381 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00092473499999999999 2020-06-12T02:38:42.381 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.381 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.381 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:42.381 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.228005+0000", 2020-06-12T02:38:42.382 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017984777 2020-06-12T02:38:42.382 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:42.382 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:42.383 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:42.383 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.383 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.383 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.3s2 59/53 ECSubWrite(tid=5970, reqid=client.4512.0:15226, at_version=59'1316, trim_to=0'0, roll_forward_to=59'1315))", 2020-06-12T02:38:42.383 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.829184+0000", 2020-06-12T02:38:42.383 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 9.4437889419999994, 2020-06-12T02:38:42.384 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.081295588000000002, 2020-06-12T02:38:42.384 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:42.384 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:42.384 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:42.384 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.385 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:42.385 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829184+0000", 2020-06-12T02:38:42.385 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:42.385 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.385 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.386 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:42.386 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829184+0000", 2020-06-12T02:38:42.386 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00023814999999999999 2020-06-12T02:38:42.386 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.386 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.387 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:42.387 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829422+0000", 2020-06-12T02:38:42.387 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9768000000000001e-05 2020-06-12T02:38:42.387 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.387 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.388 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:42.388 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829442+0000", 2020-06-12T02:38:42.388 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1022000000000001e-05 2020-06-12T02:38:42.388 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.388 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.390 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:42.390 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829453+0000", 2020-06-12T02:38:42.390 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2759999999999997e-06 2020-06-12T02:38:42.390 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.391 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.391 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:42.391 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829460+0000", 2020-06-12T02:38:42.391 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7284000000000002e-05 2020-06-12T02:38:42.391 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.392 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.392 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:42.392 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829517+0000", 2020-06-12T02:38:42.392 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7028999999999999e-05 2020-06-12T02:38:42.392 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.393 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.393 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:42.393 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829534+0000", 2020-06-12T02:38:42.393 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.080899873999999997 2020-06-12T02:38:42.393 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.394 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.394 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:42.394 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.910434+0000", 2020-06-12T02:38:42.394 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6184999999999999e-05 2020-06-12T02:38:42.394 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.395 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.395 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:42.395 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.910480+0000", 2020-06-12T02:38:42.395 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.081295588000000002 2020-06-12T02:38:42.395 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:42.396 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:42.396 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:42.396 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.397 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.397 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.3s2 59/53 ECSubWrite(tid=5973, reqid=client.4512.0:15229, at_version=59'1319, trim_to=0'0, roll_forward_to=59'1315))", 2020-06-12T02:38:42.397 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.901708+0000", 2020-06-12T02:38:42.398 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 9.3712650629999992, 2020-06-12T02:38:42.398 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013662957999999999, 2020-06-12T02:38:42.398 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:42.398 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:42.398 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:42.399 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.399 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:42.399 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901708+0000", 2020-06-12T02:38:42.399 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:42.399 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.400 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.400 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:42.400 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901708+0000", 2020-06-12T02:38:42.400 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7360000000000001e-05 2020-06-12T02:38:42.400 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.401 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.401 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:42.401 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901756+0000", 2020-06-12T02:38:42.401 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4608e-05 2020-06-12T02:38:42.401 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.402 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.402 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:42.402 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901770+0000", 2020-06-12T02:38:42.402 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.219e-05 2020-06-12T02:38:42.402 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.403 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.403 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:42.403 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901782+0000", 2020-06-12T02:38:42.403 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.511e-06 2020-06-12T02:38:42.403 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.404 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.404 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:42.404 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901789+0000", 2020-06-12T02:38:42.404 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.008025562 2020-06-12T02:38:42.404 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.405 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.405 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:42.405 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.909814+0000", 2020-06-12T02:38:42.405 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1472e-05 2020-06-12T02:38:42.406 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.406 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.406 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:42.407 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.909826+0000", 2020-06-12T02:38:42.407 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0055013350000000004 2020-06-12T02:38:42.407 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.407 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.408 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:42.408 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.915327+0000", 2020-06-12T02:38:42.408 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.392e-05 2020-06-12T02:38:42.408 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.408 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.409 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:42.409 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.915371+0000", 2020-06-12T02:38:42.409 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013662957999999999 2020-06-12T02:38:42.409 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:42.409 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:42.410 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:42.410 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.410 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.410 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.3s2 59/53 ECSubWrite(tid=5974, reqid=client.4512.0:15231, at_version=59'1320, trim_to=0'0, roll_forward_to=59'1315))", 2020-06-12T02:38:42.410 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.901839+0000", 2020-06-12T02:38:42.411 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 9.3711344650000008, 2020-06-12T02:38:42.411 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013568525, 2020-06-12T02:38:42.411 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:42.411 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:42.411 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:42.412 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.412 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:42.412 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901839+0000", 2020-06-12T02:38:42.412 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:42.412 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.413 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.413 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:42.413 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901839+0000", 2020-06-12T02:38:42.413 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6489999999999997e-05 2020-06-12T02:38:42.413 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.414 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.414 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:42.414 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901885+0000", 2020-06-12T02:38:42.414 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6600999999999999e-05 2020-06-12T02:38:42.414 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.415 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.415 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:42.415 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901902+0000", 2020-06-12T02:38:42.415 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2427e-05 2020-06-12T02:38:42.415 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.416 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.416 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:42.416 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901914+0000", 2020-06-12T02:38:42.416 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1710000000000001e-06 2020-06-12T02:38:42.417 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.417 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.417 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:42.417 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901919+0000", 2020-06-12T02:38:42.418 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0086285540000000001 2020-06-12T02:38:42.418 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.418 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.418 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:42.419 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.910548+0000", 2020-06-12T02:38:42.419 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0939e-05 2020-06-12T02:38:42.419 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.419 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.419 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:42.420 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.910559+0000", 2020-06-12T02:38:42.420 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0048161899999999997 2020-06-12T02:38:42.420 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.420 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.420 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:42.421 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.915375+0000", 2020-06-12T02:38:42.421 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2153000000000002e-05 2020-06-12T02:38:42.421 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.421 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.421 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:42.422 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.915407+0000", 2020-06-12T02:38:42.422 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013568525 2020-06-12T02:38:42.422 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:42.422 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:42.423 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:42.423 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.423 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.423 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15246 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~671744] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:42.423 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.953872+0000", 2020-06-12T02:38:42.424 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 9.3191013989999991, 2020-06-12T02:38:42.424 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015306691000000001, 2020-06-12T02:38:42.425 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:42.425 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:42.425 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:42.426 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:42.426 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:42.426 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15246 2020-06-12T02:38:42.426 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.426 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:42.427 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.427 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:42.427 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.953872+0000", 2020-06-12T02:38:42.427 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:42.427 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.427 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.428 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:42.428 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.953872+0000", 2020-06-12T02:38:42.428 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000232544 2020-06-12T02:38:42.428 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.428 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.429 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:42.429 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954104+0000", 2020-06-12T02:38:42.429 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7513999999999999e-05 2020-06-12T02:38:42.429 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.429 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.430 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:42.430 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954122+0000", 2020-06-12T02:38:42.430 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.844e-05 2020-06-12T02:38:42.430 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.430 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.431 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:42.431 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954140+0000", 2020-06-12T02:38:42.431 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2570000000000007e-06 2020-06-12T02:38:42.431 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.431 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.432 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:42.432 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954149+0000", 2020-06-12T02:38:42.434 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00031049499999999999 2020-06-12T02:38:42.435 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.435 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.435 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:42.435 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954459+0000", 2020-06-12T02:38:42.436 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000118182 2020-06-12T02:38:42.436 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.436 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.436 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:42.436 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954577+0000", 2020-06-12T02:38:42.437 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00071788400000000001 2020-06-12T02:38:42.437 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.437 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.437 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:42.437 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.955295+0000", 2020-06-12T02:38:42.438 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013717859000000001 2020-06-12T02:38:42.438 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.438 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.438 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:42.438 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.969013+0000", 2020-06-12T02:38:42.439 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4895999999999995e-05 2020-06-12T02:38:42.439 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.439 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.439 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:42.439 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.969098+0000", 2020-06-12T02:38:42.439 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0619999999999997e-05 2020-06-12T02:38:42.440 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.440 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.440 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:42.440 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.969179+0000", 2020-06-12T02:38:42.441 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015306691000000001 2020-06-12T02:38:42.441 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:42.441 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:42.441 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:42.442 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.443 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.443 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15277 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~720896] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:42.443 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:33.067109+0000", 2020-06-12T02:38:42.443 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 9.2058643389999997, 2020-06-12T02:38:42.444 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015909191999999999, 2020-06-12T02:38:42.444 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:42.444 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:42.444 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:42.445 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:42.445 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:42.445 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15277 2020-06-12T02:38:42.445 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.445 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:42.446 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.446 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:42.446 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067109+0000", 2020-06-12T02:38:42.447 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:42.447 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.447 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.447 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:42.447 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067109+0000", 2020-06-12T02:38:42.448 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018431400000000001 2020-06-12T02:38:42.448 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.448 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.448 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:42.448 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067293+0000", 2020-06-12T02:38:42.448 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1790000000000002e-06 2020-06-12T02:38:42.449 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.449 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.449 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:42.449 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067300+0000", 2020-06-12T02:38:42.449 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5259999999999999e-06 2020-06-12T02:38:42.450 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.450 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.450 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:42.450 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067310+0000", 2020-06-12T02:38:42.450 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8179999999999997e-06 2020-06-12T02:38:42.451 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.451 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.451 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:42.451 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067314+0000", 2020-06-12T02:38:42.451 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6529999999999997e-05 2020-06-12T02:38:42.452 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.452 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.452 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:42.452 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067360+0000", 2020-06-12T02:38:42.452 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012724399999999999 2020-06-12T02:38:42.452 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.453 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.453 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:42.453 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067488+0000", 2020-06-12T02:38:42.455 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0010983989999999999 2020-06-12T02:38:42.455 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.455 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.455 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:42.456 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.068586+0000", 2020-06-12T02:38:42.456 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5165000000000001e-05 2020-06-12T02:38:42.456 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.456 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.456 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:42.457 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.068681+0000", 2020-06-12T02:38:42.457 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086510899999999997 2020-06-12T02:38:42.457 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.457 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.457 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:42.458 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.069546+0000", 2020-06-12T02:38:42.458 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013284429 2020-06-12T02:38:42.458 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.458 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.458 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:42.459 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.082831+0000", 2020-06-12T02:38:42.459 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4197000000000006e-05 2020-06-12T02:38:42.459 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.459 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.459 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:42.460 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.082925+0000", 2020-06-12T02:38:42.460 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3282000000000006e-05 2020-06-12T02:38:42.460 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.460 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.460 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:42.460 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.083018+0000", 2020-06-12T02:38:42.461 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015909191999999999 2020-06-12T02:38:42.461 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:42.461 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:42.461 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:42.461 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.462 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.462 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15331 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~688128] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:42.462 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:33.276748+0000", 2020-06-12T02:38:42.462 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 8.9962248270000007, 2020-06-12T02:38:42.462 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016825066999999999, 2020-06-12T02:38:42.462 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:42.463 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:42.463 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:42.463 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:42.463 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:42.463 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15331 2020-06-12T02:38:42.464 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.464 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:42.464 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.464 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:42.464 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276748+0000", 2020-06-12T02:38:42.464 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:42.465 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.465 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.465 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:42.465 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276748+0000", 2020-06-12T02:38:42.465 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016936699999999999 2020-06-12T02:38:42.466 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.466 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.466 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:42.466 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276918+0000", 2020-06-12T02:38:42.466 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6389999999999998e-06 2020-06-12T02:38:42.467 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.467 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.467 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:42.467 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276926+0000", 2020-06-12T02:38:42.467 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3591000000000001e-05 2020-06-12T02:38:42.467 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.468 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.468 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:42.468 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276940+0000", 2020-06-12T02:38:42.468 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0429999999999996e-06 2020-06-12T02:38:42.468 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.469 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.469 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:42.469 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276949+0000", 2020-06-12T02:38:42.469 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8825000000000001e-05 2020-06-12T02:38:42.469 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.469 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.470 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:42.470 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276998+0000", 2020-06-12T02:38:42.470 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6483000000000005e-05 2020-06-12T02:38:42.470 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.470 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.471 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:42.471 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.277084+0000", 2020-06-12T02:38:42.471 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00057545800000000005 2020-06-12T02:38:42.471 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.471 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.471 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:42.472 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.277660+0000", 2020-06-12T02:38:42.472 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015617126 2020-06-12T02:38:42.472 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.472 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.472 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:42.473 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.293277+0000", 2020-06-12T02:38:42.473 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00021931699999999999 2020-06-12T02:38:42.473 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.473 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.473 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:42.473 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.293496+0000", 2020-06-12T02:38:42.474 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7218000000000005e-05 2020-06-12T02:38:42.474 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.474 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.474 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:42.475 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.293573+0000", 2020-06-12T02:38:42.475 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016825066999999999 2020-06-12T02:38:42.475 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:42.477 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:42.477 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:42.478 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.478 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.478 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.7s1 60/37 ECSubWrite(tid=8504, reqid=client.4512.0:16225, at_version=60'1427, trim_to=0'0, roll_forward_to=60'1425))", 2020-06-12T02:38:42.478 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:36.033154+0000", 2020-06-12T02:38:42.478 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 6.2398194650000001, 2020-06-12T02:38:42.479 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016768244000000002, 2020-06-12T02:38:42.479 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:42.479 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:42.480 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:42.480 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.480 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:42.480 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033154+0000", 2020-06-12T02:38:42.480 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:42.481 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.481 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.481 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:42.481 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033154+0000", 2020-06-12T02:38:42.481 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000154444 2020-06-12T02:38:42.482 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.482 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.482 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:42.482 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033308+0000", 2020-06-12T02:38:42.482 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8772000000000002e-05 2020-06-12T02:38:42.483 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.483 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.483 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:42.483 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033337+0000", 2020-06-12T02:38:42.483 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6704000000000001e-05 2020-06-12T02:38:42.484 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.484 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.484 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:42.484 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033354+0000", 2020-06-12T02:38:42.484 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6100000000000002e-06 2020-06-12T02:38:42.485 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.485 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.485 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:42.485 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033360+0000", 2020-06-12T02:38:42.485 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000173893 2020-06-12T02:38:42.486 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.486 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.486 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:42.486 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033534+0000", 2020-06-12T02:38:42.486 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1929999999999999e-05 2020-06-12T02:38:42.486 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.487 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.487 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:42.487 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033546+0000", 2020-06-12T02:38:42.488 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016272066000000002 2020-06-12T02:38:42.488 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.488 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.488 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:42.488 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.049818+0000", 2020-06-12T02:38:42.489 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103825 2020-06-12T02:38:42.489 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.489 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.489 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:42.490 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.049922+0000", 2020-06-12T02:38:42.490 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016768244000000002 2020-06-12T02:38:42.490 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:42.490 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:42.491 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:42.491 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.491 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.491 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:17973 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~507904] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:38:42.491 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:41.095893+0000", 2020-06-12T02:38:42.492 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 1.177079912, 2020-06-12T02:38:42.492 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013655560000000001, 2020-06-12T02:38:42.492 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:42.492 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:42.492 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:42.493 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:42.493 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:42.493 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 17973 2020-06-12T02:38:42.493 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.493 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:42.494 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.494 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:42.494 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.095893+0000", 2020-06-12T02:38:42.494 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:42.495 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.495 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.495 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:42.495 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.095893+0000", 2020-06-12T02:38:42.495 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018351000000000001 2020-06-12T02:38:42.496 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.496 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.496 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:42.496 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.096077+0000", 2020-06-12T02:38:42.496 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1559999999999995e-06 2020-06-12T02:38:42.497 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.497 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.497 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:42.497 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.096085+0000", 2020-06-12T02:38:42.497 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3764999999999999e-05 2020-06-12T02:38:42.498 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.498 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.499 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:42.499 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.096099+0000", 2020-06-12T02:38:42.499 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0460000000000002e-06 2020-06-12T02:38:42.499 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.500 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.500 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:42.500 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.096105+0000", 2020-06-12T02:38:42.500 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6931000000000002e-05 2020-06-12T02:38:42.501 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.501 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.501 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:42.501 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.096152+0000", 2020-06-12T02:38:42.501 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010251600000000001 2020-06-12T02:38:42.502 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.502 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.502 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:42.502 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.096254+0000", 2020-06-12T02:38:42.502 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072763000000000005 2020-06-12T02:38:42.503 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.503 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.503 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:42.503 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.096982+0000", 2020-06-12T02:38:42.503 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012262115000000001 2020-06-12T02:38:42.504 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.504 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.504 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:42.504 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.109244+0000", 2020-06-12T02:38:42.504 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1791000000000003e-05 2020-06-12T02:38:42.504 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.505 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.505 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:42.505 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.109336+0000", 2020-06-12T02:38:42.505 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0002131 2020-06-12T02:38:42.505 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.506 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.506 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:42.506 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.109549+0000", 2020-06-12T02:38:42.506 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013655560000000001 2020-06-12T02:38:42.506 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:42.506 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:42.507 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:42.507 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.507 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.507 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:17977 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:38:42.507 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:41.103849+0000", 2020-06-12T02:38:42.508 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 1.1691242829999999, 2020-06-12T02:38:42.508 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015465273, 2020-06-12T02:38:42.508 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:42.508 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:42.509 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:42.509 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:42.509 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:42.509 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 17977 2020-06-12T02:38:42.509 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.510 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:42.510 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.510 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:42.510 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.103849+0000", 2020-06-12T02:38:42.511 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:42.511 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.511 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.511 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:42.511 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.103849+0000", 2020-06-12T02:38:42.512 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1254e-05 2020-06-12T02:38:42.512 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.512 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.512 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:42.512 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.103870+0000", 2020-06-12T02:38:42.513 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9869999999999999e-06 2020-06-12T02:38:42.513 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.513 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.513 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:42.513 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.103873+0000", 2020-06-12T02:38:42.513 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6159999999999998e-06 2020-06-12T02:38:42.514 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.514 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.514 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:42.514 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.103880+0000", 2020-06-12T02:38:42.514 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0899999999999999e-06 2020-06-12T02:38:42.515 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.515 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.515 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:42.515 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.103882+0000", 2020-06-12T02:38:42.515 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0066572259999999996 2020-06-12T02:38:42.516 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.516 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.516 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:42.516 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.110539+0000", 2020-06-12T02:38:42.516 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5937000000000003e-05 2020-06-12T02:38:42.517 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.517 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.517 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:42.517 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.110595+0000", 2020-06-12T02:38:42.517 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0066983820000000001 2020-06-12T02:38:42.518 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.518 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.518 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:42.518 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.117293+0000", 2020-06-12T02:38:42.519 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103828 2020-06-12T02:38:42.519 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.519 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.519 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:42.520 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.117397+0000", 2020-06-12T02:38:42.520 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0019169530000000001 2020-06-12T02:38:42.520 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.520 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.521 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:42.521 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.119314+0000", 2020-06-12T02:38:42.521 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015465273 2020-06-12T02:38:42.521 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:42.521 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:42.522 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:42.522 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.522 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.522 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:17998 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:38:42.522 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:41.190236+0000", 2020-06-12T02:38:42.523 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 1.0827372280000001, 2020-06-12T02:38:42.523 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013903766, 2020-06-12T02:38:42.523 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:42.523 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:42.523 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:42.524 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:42.524 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:42.524 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 17998 2020-06-12T02:38:42.524 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.524 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:42.524 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.525 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:42.525 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.190236+0000", 2020-06-12T02:38:42.525 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:42.525 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.526 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.528 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:42.528 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.190236+0000", 2020-06-12T02:38:42.529 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.207e-05 2020-06-12T02:38:42.529 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.529 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.529 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:42.529 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.190258+0000", 2020-06-12T02:38:42.529 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9629999999999998e-06 2020-06-12T02:38:42.530 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.530 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.530 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:42.530 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.190261+0000", 2020-06-12T02:38:42.530 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.409e-06 2020-06-12T02:38:42.531 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.531 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.531 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:42.531 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.190267+0000", 2020-06-12T02:38:42.531 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2189999999999998e-06 2020-06-12T02:38:42.532 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.532 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.532 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:42.532 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.190270+0000", 2020-06-12T02:38:42.532 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0089464690000000003 2020-06-12T02:38:42.532 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.533 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.533 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:42.533 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.199216+0000", 2020-06-12T02:38:42.533 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010023500000000001 2020-06-12T02:38:42.533 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.534 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.534 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:42.534 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.199316+0000", 2020-06-12T02:38:42.534 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0038963800000000001 2020-06-12T02:38:42.534 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.534 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.535 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:42.535 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.203213+0000", 2020-06-12T02:38:42.535 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6963000000000003e-05 2020-06-12T02:38:42.535 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.535 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.536 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:42.536 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.203270+0000", 2020-06-12T02:38:42.536 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00087005800000000005 2020-06-12T02:38:42.536 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.536 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.537 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:42.537 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.204140+0000", 2020-06-12T02:38:42.537 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013903766 2020-06-12T02:38:42.537 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:42.537 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:42.537 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:42.538 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.538 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.538 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.7s1 62/37 ECSubWrite(tid=9513, reqid=client.4512.0:18266, at_version=0'0, trim_to=0'0, roll_forward_to=62'1601))", 2020-06-12T02:38:42.539 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:41.904485+0000", 2020-06-12T02:38:42.539 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.368488131, 2020-06-12T02:38:42.539 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01412036, 2020-06-12T02:38:42.539 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:42.539 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:42.539 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:42.540 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.540 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:42.540 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904485+0000", 2020-06-12T02:38:42.540 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:42.540 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.541 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.541 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:42.541 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904485+0000", 2020-06-12T02:38:42.541 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.345e-05 2020-06-12T02:38:42.541 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.541 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.542 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:42.542 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904519+0000", 2020-06-12T02:38:42.542 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0869999999999999e-05 2020-06-12T02:38:42.542 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.542 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.543 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:42.543 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904529+0000", 2020-06-12T02:38:42.543 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1248e-05 2020-06-12T02:38:42.543 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.543 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.543 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:42.544 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904541+0000", 2020-06-12T02:38:42.544 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0730999999999999e-05 2020-06-12T02:38:42.544 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.544 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.544 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:42.544 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904551+0000", 2020-06-12T02:38:42.545 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0985999999999997e-05 2020-06-12T02:38:42.545 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.545 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.545 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:42.545 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904592+0000", 2020-06-12T02:38:42.546 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2255999999999999e-05 2020-06-12T02:38:42.546 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.546 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.546 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:42.546 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904605+0000", 2020-06-12T02:38:42.547 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013935858000000001 2020-06-12T02:38:42.547 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.547 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.547 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:42.547 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.918541+0000", 2020-06-12T02:38:42.548 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4961000000000004e-05 2020-06-12T02:38:42.548 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.548 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.548 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:42.548 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.918605+0000", 2020-06-12T02:38:42.548 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01412036 2020-06-12T02:38:42.549 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:42.549 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:42.549 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:42.549 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.549 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.550 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "pg_update_log_missing(3.3s2 epoch 62/53 rep_tid 6734 entries 62'1548 (0'0) error 3:ce45c70a:::smithi10729407-7:head by client.4512.0:18270 0.000000 -2 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 62'1546)", 2020-06-12T02:38:42.550 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:41.904668+0000", 2020-06-12T02:38:42.550 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.36830559499999999, 2020-06-12T02:38:42.550 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01430155, 2020-06-12T02:38:42.550 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:42.550 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:42.551 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:42.551 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.551 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:42.551 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904668+0000", 2020-06-12T02:38:42.551 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:42.552 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.552 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.552 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:42.552 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904668+0000", 2020-06-12T02:38:42.552 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1882000000000001e-05 2020-06-12T02:38:42.552 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.553 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.553 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:42.553 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904730+0000", 2020-06-12T02:38:42.553 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0861e-05 2020-06-12T02:38:42.553 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.554 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.554 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:42.554 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904740+0000", 2020-06-12T02:38:42.554 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0406000000000001e-05 2020-06-12T02:38:42.554 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.555 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.555 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:42.555 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904761+0000", 2020-06-12T02:38:42.555 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0339999999999997e-06 2020-06-12T02:38:42.555 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.555 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.556 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:42.556 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904769+0000", 2020-06-12T02:38:42.556 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3888999999999995e-05 2020-06-12T02:38:42.556 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.556 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.557 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:42.557 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904863+0000", 2020-06-12T02:38:42.557 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014106478 2020-06-12T02:38:42.557 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.557 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.557 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:42.558 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.918969+0000", 2020-06-12T02:38:42.558 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01430155 2020-06-12T02:38:42.558 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:42.558 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:42.558 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:42.559 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.559 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.559 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.fs2 62/37 ECSubWrite(tid=3343, reqid=client.4512.0:18307, at_version=62'1623, trim_to=0'0, roll_forward_to=62'1622))", 2020-06-12T02:38:42.559 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:42.062050+0000", 2020-06-12T02:38:42.559 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.21092293100000001, 2020-06-12T02:38:42.559 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019528429999999999, 2020-06-12T02:38:42.560 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:42.560 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:42.560 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:42.560 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.560 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:42.561 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062050+0000", 2020-06-12T02:38:42.561 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:42.561 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.561 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.561 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:42.561 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062050+0000", 2020-06-12T02:38:42.562 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00022828799999999999 2020-06-12T02:38:42.562 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.562 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.562 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:42.562 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062279+0000", 2020-06-12T02:38:42.563 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2115000000000001e-05 2020-06-12T02:38:42.563 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.563 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.563 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:42.563 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062301+0000", 2020-06-12T02:38:42.563 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9301e-05 2020-06-12T02:38:42.564 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.564 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.564 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:42.564 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062320+0000", 2020-06-12T02:38:42.564 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2811e-05 2020-06-12T02:38:42.565 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.565 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.565 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:42.565 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062333+0000", 2020-06-12T02:38:42.565 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8856e-05 2020-06-12T02:38:42.565 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.566 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.566 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:42.566 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062392+0000", 2020-06-12T02:38:42.566 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1246999999999999e-05 2020-06-12T02:38:42.566 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.567 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.567 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:42.567 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062403+0000", 2020-06-12T02:38:42.567 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019095523 2020-06-12T02:38:42.567 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.568 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.568 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:42.568 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.081498+0000", 2020-06-12T02:38:42.569 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0289000000000003e-05 2020-06-12T02:38:42.569 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.569 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.569 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:42.569 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.081579+0000", 2020-06-12T02:38:42.570 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019528429999999999 2020-06-12T02:38:42.570 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:42.570 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:42.570 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:42.570 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.570 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.571 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.fs2 62/37 ECSubWrite(tid=3344, reqid=client.4512.0:18308, at_version=62'1624, trim_to=0'0, roll_forward_to=62'1622))", 2020-06-12T02:38:42.571 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:42.064848+0000", 2020-06-12T02:38:42.571 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.20812494600000001, 2020-06-12T02:38:42.571 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01677273, 2020-06-12T02:38:42.571 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:42.572 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:42.572 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:42.572 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.572 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:42.572 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.064848+0000", 2020-06-12T02:38:42.572 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:42.573 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.573 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.573 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:42.573 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.064848+0000", 2020-06-12T02:38:42.573 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000383655 2020-06-12T02:38:42.574 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.574 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.574 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:42.574 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.065232+0000", 2020-06-12T02:38:42.574 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3065999999999999e-05 2020-06-12T02:38:42.575 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.575 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.575 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:42.575 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.065255+0000", 2020-06-12T02:38:42.575 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7142999999999998e-05 2020-06-12T02:38:42.575 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.576 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.576 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:42.576 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.065272+0000", 2020-06-12T02:38:42.576 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7750000000000004e-06 2020-06-12T02:38:42.576 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.576 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.577 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:42.577 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.065279+0000", 2020-06-12T02:38:42.577 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00091758699999999998 2020-06-12T02:38:42.577 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.577 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.578 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:42.578 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.066197+0000", 2020-06-12T02:38:42.578 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4158999999999999e-05 2020-06-12T02:38:42.578 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.578 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.578 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:42.579 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.066211+0000", 2020-06-12T02:38:42.579 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015373370000000001 2020-06-12T02:38:42.579 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.579 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.579 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:42.580 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.081584+0000", 2020-06-12T02:38:42.580 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6974999999999998e-05 2020-06-12T02:38:42.580 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.580 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.580 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:42.580 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.081621+0000", 2020-06-12T02:38:42.581 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01677273 2020-06-12T02:38:42.581 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:42.581 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:42.581 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:42.581 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.582 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.582 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.3s2 62/53 ECSubWrite(tid=6737, reqid=client.4512.0:18319, at_version=62'1551, trim_to=0'0, roll_forward_to=62'1548))", 2020-06-12T02:38:42.582 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:42.089962+0000", 2020-06-12T02:38:42.582 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.18301130800000001, 2020-06-12T02:38:42.582 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015432937000000001, 2020-06-12T02:38:42.582 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:42.583 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:42.583 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:42.583 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.583 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:42.583 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.089962+0000", 2020-06-12T02:38:42.584 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:42.584 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.584 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.584 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:42.584 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.089962+0000", 2020-06-12T02:38:42.584 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00068051099999999996 2020-06-12T02:38:42.585 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.585 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.585 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:42.585 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.090642+0000", 2020-06-12T02:38:42.585 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8766999999999998e-05 2020-06-12T02:38:42.586 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.586 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.586 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:42.586 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.090661+0000", 2020-06-12T02:38:42.586 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6461e-05 2020-06-12T02:38:42.586 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.587 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.587 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:42.587 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.090678+0000", 2020-06-12T02:38:42.587 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0580000000000003e-06 2020-06-12T02:38:42.587 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.588 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.588 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:42.588 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.090684+0000", 2020-06-12T02:38:42.588 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000104752 2020-06-12T02:38:42.588 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.588 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.589 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:42.589 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.090789+0000", 2020-06-12T02:38:42.589 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0767e-05 2020-06-12T02:38:42.589 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.589 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.590 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:42.590 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.090799+0000", 2020-06-12T02:38:42.590 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014539606 2020-06-12T02:38:42.590 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.590 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.590 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:42.591 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.105339+0000", 2020-06-12T02:38:42.591 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6014999999999998e-05 2020-06-12T02:38:42.591 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:42.591 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:42.591 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:42.591 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.105395+0000", 2020-06-12T02:38:42.592 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015432937000000001 2020-06-12T02:38:42.592 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:42.592 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:42.592 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:42.592 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:42.593 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:42.593 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:42.593 INFO:tasks.rados.rados.0.smithi107.stdout:8630: done (1 left) 2020-06-12T02:38:42.593 INFO:tasks.rados.rados.0.smithi107.stdout:8631: delete oid 8 current snap is 0 2020-06-12T02:38:42.593 INFO:tasks.rados.rados.0.smithi107.stdout:8631: done (1 left) 2020-06-12T02:38:42.594 INFO:tasks.rados.rados.0.smithi107.stdout:8632: delete oid 16 current snap is 0 2020-06-12T02:38:42.594 INFO:tasks.rados.rados.0.smithi107.stdout:8627: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:42.594 INFO:tasks.rados.rados.0.smithi107.stdout:8632: done (1 left) 2020-06-12T02:38:42.594 INFO:tasks.rados.rados.0.smithi107.stdout:8633: delete oid 5 current snap is 0 2020-06-12T02:38:42.594 INFO:tasks.rados.rados.0.smithi107.stdout:8627: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:42.594 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1583 (ObjNum 8626 snap 0 seq_num 8626) dirty exists 2020-06-12T02:38:42.595 INFO:tasks.rados.rados.0.smithi107.stdout:8627: left oid 12 (ObjNum 8626 snap 0 seq_num 8626) 2020-06-12T02:38:42.595 INFO:tasks.rados.rados.0.smithi107.stdout:8627: done (1 left) 2020-06-12T02:38:42.595 INFO:tasks.rados.rados.0.smithi107.stdout:8633: done (0 left) 2020-06-12T02:38:42.595 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:42.595 INFO:tasks.rados.rados.0.smithi107.stdout:8634: seq_num 8633 ranges {0=589824} 2020-06-12T02:38:42.596 INFO:tasks.rados.rados.0.smithi107.stdout:8634: writing smithi10729407-14 from 0 to 589824 tid 1 2020-06-12T02:38:42.596 INFO:tasks.rados.rados.0.smithi107.stdout:8635: delete oid 10 current snap is 0 2020-06-12T02:38:42.596 INFO:tasks.rados.rados.0.smithi107.stdout:8635: done (1 left) 2020-06-12T02:38:42.596 INFO:tasks.rados.rados.0.smithi107.stdout:8636: delete oid 13 current snap is 0 2020-06-12T02:38:42.597 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-06-12T02:38:42.612 INFO:tasks.rados.rados.0.smithi107.stdout:8636: done (1 left) 2020-06-12T02:38:42.612 INFO:tasks.rados.rados.0.smithi107.stdout:8637: delete oid 15 current snap is 0 2020-06-12T02:38:42.613 INFO:tasks.rados.rados.0.smithi107.stdout:8637: done (1 left) 2020-06-12T02:38:42.613 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:42.613 INFO:tasks.rados.rados.0.smithi107.stdout:8638: seq_num 8637 ranges {319488=655360} 2020-06-12T02:38:42.613 INFO:tasks.rados.rados.0.smithi107.stdout:8638: writing smithi10729407-9 from 319488 to 974848 tid 1 2020-06-12T02:38:42.613 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:38:42.614 INFO:tasks.rados.rados.0.smithi107.stdout:8639: seq_num 8638 ranges {0=679936} 2020-06-12T02:38:42.614 INFO:tasks.rados.rados.0.smithi107.stdout:8639: writing smithi10729407-20 from 0 to 679936 tid 1 2020-06-12T02:38:42.614 INFO:tasks.rados.rados.0.smithi107.stdout:8640: delete oid 1 current snap is 0 2020-06-12T02:38:42.614 INFO:tasks.rados.rados.0.smithi107.stdout:8634: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:42.614 INFO:tasks.rados.rados.0.smithi107.stdout:8634: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:42.615 INFO:tasks.rados.rados.0.smithi107.stdout:8634: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:42.615 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1510 (ObjNum 8633 snap 0 seq_num 8633) dirty exists 2020-06-12T02:38:42.615 INFO:tasks.rados.rados.0.smithi107.stdout:8634: left oid 14 (ObjNum 8633 snap 0 seq_num 8633) 2020-06-12T02:38:42.615 INFO:tasks.rados.rados.0.smithi107.stdout:8634: done (3 left) 2020-06-12T02:38:42.615 INFO:tasks.rados.rados.0.smithi107.stdout:8640: done (2 left) 2020-06-12T02:38:42.616 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:38:42.616 INFO:tasks.rados.rados.0.smithi107.stdout:8641: seq_num 8640 ranges {0=712704} 2020-06-12T02:38:42.616 INFO:tasks.rados.rados.0.smithi107.stdout:8641: writing smithi10729407-1 from 0 to 712704 tid 1 2020-06-12T02:38:42.616 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:42.616 INFO:tasks.rados.rados.0.smithi107.stdout:8642: seq_num 8641 ranges {0=729088} 2020-06-12T02:38:42.616 INFO:tasks.rados.rados.0.smithi107.stdout:8642: writing smithi10729407-3 from 0 to 729088 tid 1 2020-06-12T02:38:42.617 INFO:tasks.rados.rados.0.smithi107.stdout:8643: delete oid 12 current snap is 0 2020-06-12T02:38:42.617 INFO:tasks.rados.rados.0.smithi107.stdout:8638: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:42.617 INFO:tasks.rados.rados.0.smithi107.stdout:8638: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:42.617 INFO:tasks.rados.rados.0.smithi107.stdout:8639: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:42.617 INFO:tasks.rados.rados.0.smithi107.stdout:8639: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:42.618 INFO:tasks.rados.rados.0.smithi107.stdout:8638: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:42.618 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1611 (ObjNum 8637 snap 0 seq_num 8637) dirty exists 2020-06-12T02:38:42.618 INFO:tasks.rados.rados.0.smithi107.stdout:8638: left oid 9 (ObjNum 8637 snap 0 seq_num 8637) 2020-06-12T02:38:42.618 INFO:tasks.rados.rados.0.smithi107.stdout:8639: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:42.618 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 631 (ObjNum 8638 snap 0 seq_num 8638) dirty exists 2020-06-12T02:38:42.618 INFO:tasks.rados.rados.0.smithi107.stdout:8639: left oid 20 (ObjNum 8638 snap 0 seq_num 8638) 2020-06-12T02:38:42.619 INFO:tasks.rados.rados.0.smithi107.stdout:8638: done (4 left) 2020-06-12T02:38:42.619 INFO:tasks.rados.rados.0.smithi107.stdout:8639: done (3 left) 2020-06-12T02:38:42.619 INFO:tasks.rados.rados.0.smithi107.stdout:8643: done (2 left) 2020-06-12T02:38:42.619 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:42.619 INFO:tasks.rados.rados.0.smithi107.stdout:8644: seq_num 8643 ranges {188416=761856} 2020-06-12T02:38:42.620 INFO:tasks.rados.rados.0.smithi107.stdout:8644: writing smithi10729407-19 from 188416 to 950272 tid 1 2020-06-12T02:38:42.620 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:42.620 INFO:tasks.rados.rados.0.smithi107.stdout:8645: seq_num 8644 ranges {0=778240} 2020-06-12T02:38:42.620 INFO:tasks.rados.rados.0.smithi107.stdout:8645: writing smithi10729407-15 from 0 to 778240 tid 1 2020-06-12T02:38:42.620 INFO:tasks.rados.rados.0.smithi107.stdout:8646: delete oid 4 current snap is 0 2020-06-12T02:38:42.620 INFO:tasks.rados.rados.0.smithi107.stdout:8641: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:42.621 INFO:tasks.rados.rados.0.smithi107.stdout:8641: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:42.621 INFO:tasks.rados.rados.0.smithi107.stdout:8642: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:42.621 INFO:tasks.rados.rados.0.smithi107.stdout:8641: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:42.621 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1568 (ObjNum 8640 snap 0 seq_num 8640) dirty exists 2020-06-12T02:38:42.621 INFO:tasks.rados.rados.0.smithi107.stdout:8641: left oid 1 (ObjNum 8640 snap 0 seq_num 8640) 2020-06-12T02:38:42.622 INFO:tasks.rados.rados.0.smithi107.stdout:8642: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:42.622 INFO:tasks.rados.rados.0.smithi107.stdout:8642: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:42.622 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1552 (ObjNum 8641 snap 0 seq_num 8641) dirty exists 2020-06-12T02:38:42.622 INFO:tasks.rados.rados.0.smithi107.stdout:8642: left oid 3 (ObjNum 8641 snap 0 seq_num 8641) 2020-06-12T02:38:42.622 INFO:tasks.rados.rados.0.smithi107.stdout:8644: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:42.623 INFO:tasks.rados.rados.0.smithi107.stdout:8644: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:42.623 INFO:tasks.rados.rados.0.smithi107.stdout:8644: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:42.623 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1586 (ObjNum 8643 snap 0 seq_num 8643) dirty exists 2020-06-12T02:38:42.623 INFO:tasks.rados.rados.0.smithi107.stdout:8644: left oid 19 (ObjNum 8643 snap 0 seq_num 8643) 2020-06-12T02:38:42.623 INFO:tasks.rados.rados.0.smithi107.stdout:8645: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:42.623 INFO:tasks.rados.rados.0.smithi107.stdout:8641: done (4 left) 2020-06-12T02:38:42.624 INFO:tasks.rados.rados.0.smithi107.stdout:8642: done (3 left) 2020-06-12T02:38:42.624 INFO:tasks.rados.rados.0.smithi107.stdout:8644: done (2 left) 2020-06-12T02:38:42.624 INFO:tasks.rados.rados.0.smithi107.stdout:8646: done (1 left) 2020-06-12T02:38:42.624 INFO:tasks.rados.rados.0.smithi107.stdout:8647: delete oid 12 current snap is 0 2020-06-12T02:38:42.624 INFO:tasks.rados.rados.0.smithi107.stdout:8645: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:42.624 INFO:tasks.rados.rados.0.smithi107.stdout:8647: done (1 left) 2020-06-12T02:38:42.625 INFO:tasks.rados.rados.0.smithi107.stdout:8648: delete oid 8 current snap is 0 2020-06-12T02:38:42.625 INFO:tasks.rados.rados.0.smithi107.stdout:8645: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:42.625 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1609 (ObjNum 8644 snap 0 seq_num 8644) dirty exists 2020-06-12T02:38:42.625 INFO:tasks.rados.rados.0.smithi107.stdout:8645: left oid 15 (ObjNum 8644 snap 0 seq_num 8644) 2020-06-12T02:38:42.625 INFO:tasks.rados.rados.0.smithi107.stdout:8645: done (1 left) 2020-06-12T02:38:42.626 INFO:tasks.rados.rados.0.smithi107.stdout:8648: done (0 left) 2020-06-12T02:38:42.626 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:42.626 INFO:tasks.rados.rados.0.smithi107.stdout:8649: seq_num 8648 ranges {0=442368,442368=401408} 2020-06-12T02:38:42.626 INFO:tasks.rados.rados.0.smithi107.stdout:8649: writing smithi10729407-6 from 0 to 442368 tid 1 2020-06-12T02:38:42.626 INFO:tasks.rados.rados.0.smithi107.stdout:8649: writing smithi10729407-6 from 442368 to 843776 tid 2 2020-06-12T02:38:42.626 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:42.627 INFO:tasks.rados.rados.0.smithi107.stdout:8650: seq_num 8649 ranges {0=458752,458752=401408} 2020-06-12T02:38:42.627 INFO:tasks.rados.rados.0.smithi107.stdout:8650: writing smithi10729407-11 from 0 to 458752 tid 1 2020-06-12T02:38:42.627 INFO:tasks.rados.rados.0.smithi107.stdout:8650: writing smithi10729407-11 from 458752 to 860160 tid 2 2020-06-12T02:38:42.627 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:42.627 INFO:tasks.rados.rados.0.smithi107.stdout:8651: seq_num 8650 ranges {0=475136,475136=401408} 2020-06-12T02:38:42.628 INFO:tasks.rados.rados.0.smithi107.stdout:8651: writing smithi10729407-18 from 0 to 475136 tid 1 2020-06-12T02:38:42.628 INFO:tasks.rados.rados.0.smithi107.stdout:8651: writing smithi10729407-18 from 475136 to 876544 tid 2 2020-06-12T02:38:42.628 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:38:42.628 INFO:tasks.rados.rados.0.smithi107.stdout:8652: seq_num 8651 ranges {679936=491520,1171456=401408} 2020-06-12T02:38:42.628 INFO:tasks.rados.rados.0.smithi107.stdout:8652: writing smithi10729407-20 from 679936 to 1171456 tid 1 2020-06-12T02:38:42.628 INFO:tasks.rados.rados.0.smithi107.stdout:8652: writing smithi10729407-20 from 1171456 to 1572864 tid 2 2020-06-12T02:38:42.629 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:38:42.629 INFO:tasks.rados.rados.0.smithi107.stdout:8653: seq_num 8652 ranges {712704=507904,1220608=401408} 2020-06-12T02:38:42.629 INFO:tasks.rados.rados.0.smithi107.stdout:8653: writing smithi10729407-1 from 712704 to 1220608 tid 1 2020-06-12T02:38:42.629 INFO:tasks.rados.rados.0.smithi107.stdout:8653: writing smithi10729407-1 from 1220608 to 1622016 tid 2 2020-06-12T02:38:42.629 INFO:tasks.rados.rados.0.smithi107.stdout:8654: delete oid 4 current snap is 0 2020-06-12T02:38:42.630 INFO:tasks.rados.rados.0.smithi107.stdout:8649: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:42.630 INFO:tasks.rados.rados.0.smithi107.stdout:8649: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:42.630 INFO:tasks.rados.rados.0.smithi107.stdout:8649: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:42.630 INFO:tasks.rados.rados.0.smithi107.stdout:8650: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:42.630 INFO:tasks.rados.rados.0.smithi107.stdout:8650: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:42.630 INFO:tasks.rados.rados.0.smithi107.stdout:8650: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:42.631 INFO:tasks.rados.rados.0.smithi107.stdout:8649: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:42.631 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1571 (ObjNum 8648 snap 0 seq_num 8648) dirty exists 2020-06-12T02:38:42.631 INFO:tasks.rados.rados.0.smithi107.stdout:8649: left oid 6 (ObjNum 8648 snap 0 seq_num 8648) 2020-06-12T02:38:42.631 INFO:tasks.rados.rados.0.smithi107.stdout:8651: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:42.631 INFO:tasks.rados.rados.0.smithi107.stdout:8650: finishing write tid 5 to smithi10729407-11 2020-06-12T02:38:42.632 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1513 (ObjNum 8649 snap 0 seq_num 8649) dirty exists 2020-06-12T02:38:42.632 INFO:tasks.rados.rados.0.smithi107.stdout:8650: left oid 11 (ObjNum 8649 snap 0 seq_num 8649) 2020-06-12T02:38:42.632 INFO:tasks.rados.rados.0.smithi107.stdout:8651: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:42.632 INFO:tasks.rados.rados.0.smithi107.stdout:8651: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:42.632 INFO:tasks.rados.rados.0.smithi107.stdout:8652: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:42.632 INFO:tasks.rados.rados.0.smithi107.stdout:8651: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:42.633 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1614 (ObjNum 8650 snap 0 seq_num 8650) dirty exists 2020-06-12T02:38:42.633 INFO:tasks.rados.rados.0.smithi107.stdout:8651: left oid 18 (ObjNum 8650 snap 0 seq_num 8650) 2020-06-12T02:38:42.633 INFO:tasks.rados.rados.0.smithi107.stdout:8652: finishing write tid 2 to smithi10729407-20 2020-06-12T02:38:42.633 INFO:tasks.rados.rados.0.smithi107.stdout:8652: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:42.633 INFO:tasks.rados.rados.0.smithi107.stdout:8652: finishing write tid 5 to smithi10729407-20 2020-06-12T02:38:42.633 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 634 (ObjNum 8651 snap 0 seq_num 8651) dirty exists 2020-06-12T02:38:42.634 INFO:tasks.rados.rados.0.smithi107.stdout:8652: left oid 20 (ObjNum 8651 snap 0 seq_num 8651) 2020-06-12T02:38:42.634 INFO:tasks.rados.rados.0.smithi107.stdout:8649: done (5 left) 2020-06-12T02:38:42.634 INFO:tasks.rados.rados.0.smithi107.stdout:8650: done (4 left) 2020-06-12T02:38:42.634 INFO:tasks.rados.rados.0.smithi107.stdout:8651: done (3 left) 2020-06-12T02:38:42.634 INFO:tasks.rados.rados.0.smithi107.stdout:8652: done (2 left) 2020-06-12T02:38:42.635 INFO:tasks.rados.rados.0.smithi107.stdout:8654: done (1 left) 2020-06-12T02:38:42.635 INFO:tasks.rados.rados.0.smithi107.stdout:8655: delete oid 18 current snap is 0 2020-06-12T02:38:42.635 INFO:tasks.rados.rados.0.smithi107.stdout:8653: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:42.635 INFO:tasks.rados.rados.0.smithi107.stdout:8655: done (1 left) 2020-06-12T02:38:42.635 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:42.635 INFO:tasks.rados.rados.0.smithi107.stdout:8656: seq_num 8655 ranges {778240=557056,1335296=401408} 2020-06-12T02:38:42.636 INFO:tasks.rados.rados.0.smithi107.stdout:8656: writing smithi10729407-15 from 778240 to 1335296 tid 1 2020-06-12T02:38:42.636 INFO:tasks.rados.rados.0.smithi107.stdout:8656: writing smithi10729407-15 from 1335296 to 1736704 tid 2 2020-06-12T02:38:42.636 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:42.636 INFO:tasks.rados.rados.0.smithi107.stdout:8657: seq_num 8656 ranges {860160=573440,1433600=401408} 2020-06-12T02:38:42.636 INFO:tasks.rados.rados.0.smithi107.stdout:8657: writing smithi10729407-11 from 860160 to 1433600 tid 1 2020-06-12T02:38:42.637 INFO:tasks.rados.rados.0.smithi107.stdout:8657: writing smithi10729407-11 from 1433600 to 1835008 tid 2 2020-06-12T02:38:42.637 INFO:tasks.rados.rados.0.smithi107.stdout:8658: delete oid 14 current snap is 0 2020-06-12T02:38:42.637 INFO:tasks.rados.rados.0.smithi107.stdout:8653: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:42.637 INFO:tasks.rados.rados.0.smithi107.stdout:8653: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:42.637 INFO:tasks.rados.rados.0.smithi107.stdout:8653: finishing write tid 5 to smithi10729407-1 2020-06-12T02:38:42.638 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1574 (ObjNum 8652 snap 0 seq_num 8652) dirty exists 2020-06-12T02:38:42.638 INFO:tasks.rados.rados.0.smithi107.stdout:8653: left oid 1 (ObjNum 8652 snap 0 seq_num 8652) 2020-06-12T02:38:42.642 INFO:tasks.rados.rados.0.smithi107.stdout:8656: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:42.642 INFO:tasks.rados.rados.0.smithi107.stdout:8656: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:42.643 INFO:tasks.rados.rados.0.smithi107.stdout:8656: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:42.643 INFO:tasks.rados.rados.0.smithi107.stdout:8656: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:42.643 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1613 (ObjNum 8655 snap 0 seq_num 8655) dirty exists 2020-06-12T02:38:42.643 INFO:tasks.rados.rados.0.smithi107.stdout:8656: left oid 15 (ObjNum 8655 snap 0 seq_num 8655) 2020-06-12T02:38:42.643 INFO:tasks.rados.rados.0.smithi107.stdout:8657: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:42.644 INFO:tasks.rados.rados.0.smithi107.stdout:8657: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:42.644 INFO:tasks.rados.rados.0.smithi107.stdout:8657: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:42.644 INFO:tasks.rados.rados.0.smithi107.stdout:8657: finishing write tid 5 to smithi10729407-11 2020-06-12T02:38:42.644 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1516 (ObjNum 8656 snap 0 seq_num 8656) dirty exists 2020-06-12T02:38:42.644 INFO:tasks.rados.rados.0.smithi107.stdout:8657: left oid 11 (ObjNum 8656 snap 0 seq_num 8656) 2020-06-12T02:38:42.644 INFO:tasks.rados.rados.0.smithi107.stdout:8653: done (3 left) 2020-06-12T02:38:42.645 INFO:tasks.rados.rados.0.smithi107.stdout:8656: done (2 left) 2020-06-12T02:38:42.645 INFO:tasks.rados.rados.0.smithi107.stdout:8657: done (1 left) 2020-06-12T02:38:42.645 INFO:tasks.rados.rados.0.smithi107.stdout:8658: done (0 left) 2020-06-12T02:38:42.645 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:42.645 INFO:tasks.rados.rados.0.smithi107.stdout:8659: seq_num 8658 ranges {974848=614400,1589248=401408} 2020-06-12T02:38:42.646 INFO:tasks.rados.rados.0.smithi107.stdout:8659: writing smithi10729407-9 from 974848 to 1589248 tid 1 2020-06-12T02:38:42.646 INFO:tasks.rados.rados.0.smithi107.stdout:8659: writing smithi10729407-9 from 1589248 to 1990656 tid 2 2020-06-12T02:38:42.646 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:38:42.646 INFO:tasks.rados.rados.0.smithi107.stdout:8660: seq_num 8659 ranges {0=630784,630784=401408} 2020-06-12T02:38:42.646 INFO:tasks.rados.rados.0.smithi107.stdout:8660: writing smithi10729407-13 from 0 to 630784 tid 1 2020-06-12T02:38:42.646 INFO:tasks.rados.rados.0.smithi107.stdout:8660: writing smithi10729407-13 from 630784 to 1032192 tid 2 2020-06-12T02:38:42.647 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:42.647 INFO:tasks.rados.rados.0.smithi107.stdout:8661: seq_num 8660 ranges {0=647168,647168=401408} 2020-06-12T02:38:42.647 INFO:tasks.rados.rados.0.smithi107.stdout:8661: writing smithi10729407-18 from 0 to 647168 tid 1 2020-06-12T02:38:42.647 INFO:tasks.rados.rados.0.smithi107.stdout:8661: writing smithi10729407-18 from 647168 to 1048576 tid 2 2020-06-12T02:38:42.647 INFO:tasks.rados.rados.0.smithi107.stdout:8662: delete oid 20 current snap is 0 2020-06-12T02:38:42.647 INFO:tasks.rados.rados.0.smithi107.stdout:8659: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:42.648 INFO:tasks.rados.rados.0.smithi107.stdout:8659: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:42.648 INFO:tasks.rados.rados.0.smithi107.stdout:8659: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:42.648 INFO:tasks.rados.rados.0.smithi107.stdout:8659: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:42.648 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1618 (ObjNum 8658 snap 0 seq_num 8658) dirty exists 2020-06-12T02:38:42.648 INFO:tasks.rados.rados.0.smithi107.stdout:8659: left oid 9 (ObjNum 8658 snap 0 seq_num 8658) 2020-06-12T02:38:42.649 INFO:tasks.rados.rados.0.smithi107.stdout:8660: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:42.649 INFO:tasks.rados.rados.0.smithi107.stdout:8660: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:42.649 INFO:tasks.rados.rados.0.smithi107.stdout:8660: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:42.649 INFO:tasks.rados.rados.0.smithi107.stdout:8660: finishing write tid 5 to smithi10729407-13 2020-06-12T02:38:42.649 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1565 (ObjNum 8659 snap 0 seq_num 8659) dirty exists 2020-06-12T02:38:42.650 INFO:tasks.rados.rados.0.smithi107.stdout:8660: left oid 13 (ObjNum 8659 snap 0 seq_num 8659) 2020-06-12T02:38:42.650 INFO:tasks.rados.rados.0.smithi107.stdout:8659: done (3 left) 2020-06-12T02:38:42.650 INFO:tasks.rados.rados.0.smithi107.stdout:8660: done (2 left) 2020-06-12T02:38:42.650 INFO:tasks.rados.rados.0.smithi107.stdout:8662: done (1 left) 2020-06-12T02:38:42.650 INFO:tasks.rados.rados.0.smithi107.stdout:8663: delete oid 14 current snap is 0 2020-06-12T02:38:42.650 INFO:tasks.rados.rados.0.smithi107.stdout:8661: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:42.651 INFO:tasks.rados.rados.0.smithi107.stdout:8663: done (1 left) 2020-06-12T02:38:42.651 INFO:tasks.rados.rados.0.smithi107.stdout:8664: delete oid 2 current snap is 0 2020-06-12T02:38:42.651 INFO:tasks.rados.rados.0.smithi107.stdout:8664: done (1 left) 2020-06-12T02:38:42.651 INFO:tasks.rados.rados.0.smithi107.stdout:8665: delete oid 8 current snap is 0 2020-06-12T02:38:42.651 INFO:tasks.rados.rados.0.smithi107.stdout:8661: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:42.652 INFO:tasks.rados.rados.0.smithi107.stdout:8661: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:42.652 INFO:tasks.rados.rados.0.smithi107.stdout:8665: done (1 left) 2020-06-12T02:38:42.652 INFO:tasks.rados.rados.0.smithi107.stdout:8666: delete oid 13 current snap is 0 2020-06-12T02:38:42.652 INFO:tasks.rados.rados.0.smithi107.stdout:8661: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:42.652 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1621 (ObjNum 8660 snap 0 seq_num 8660) dirty exists 2020-06-12T02:38:42.652 INFO:tasks.rados.rados.0.smithi107.stdout:8661: left oid 18 (ObjNum 8660 snap 0 seq_num 8660) 2020-06-12T02:38:42.653 INFO:tasks.rados.rados.0.smithi107.stdout:8661: done (1 left) 2020-06-12T02:38:42.653 INFO:tasks.rados.rados.0.smithi107.stdout:8666: done (0 left) 2020-06-12T02:38:42.653 INFO:tasks.rados.rados.0.smithi107.stdout:8667: delete oid 14 current snap is 0 2020-06-12T02:38:42.653 INFO:tasks.rados.rados.0.smithi107.stdout:8667: done (0 left) 2020-06-12T02:38:42.653 INFO:tasks.rados.rados.0.smithi107.stdout:8668: delete oid 15 current snap is 0 2020-06-12T02:38:42.653 INFO:tasks.rados.rados.0.smithi107.stdout:8668: done (0 left) 2020-06-12T02:38:42.654 INFO:tasks.rados.rados.0.smithi107.stdout:8669: delete oid 2 current snap is 0 2020-06-12T02:38:42.654 INFO:tasks.rados.rados.0.smithi107.stdout:8669: done (0 left) 2020-06-12T02:38:42.654 INFO:tasks.rados.rados.0.smithi107.stdout:8670: delete oid 16 current snap is 0 2020-06-12T02:38:42.654 INFO:tasks.rados.rados.0.smithi107.stdout:8670: done (0 left) 2020-06-12T02:38:42.654 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:42.655 INFO:tasks.rados.rados.0.smithi107.stdout:8671: seq_num 8670 ranges {0=409600,409600=598016,1007616=204800} 2020-06-12T02:38:42.655 INFO:tasks.rados.rados.0.smithi107.stdout:8671: writing smithi10729407-15 from 0 to 409600 tid 1 2020-06-12T02:38:42.655 INFO:tasks.rados.rados.0.smithi107.stdout:8671: writing smithi10729407-15 from 409600 to 1007616 tid 2 2020-06-12T02:38:42.655 INFO:tasks.rados.rados.0.smithi107.stdout:8671: writing smithi10729407-15 from 1007616 to 1212416 tid 3 2020-06-12T02:38:42.655 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:42.655 INFO:tasks.rados.rados.0.smithi107.stdout:8672: seq_num 8671 ranges {0=425984,425984=483328,909312=319488} 2020-06-12T02:38:42.656 INFO:tasks.rados.rados.0.smithi107.stdout:8672: writing smithi10729407-17 from 0 to 425984 tid 1 2020-06-12T02:38:42.656 INFO:tasks.rados.rados.0.smithi107.stdout:8672: writing smithi10729407-17 from 425984 to 909312 tid 2 2020-06-12T02:38:42.656 INFO:tasks.rados.rados.0.smithi107.stdout:8672: writing smithi10729407-17 from 909312 to 1228800 tid 3 2020-06-12T02:38:42.656 INFO:tasks.rados.rados.0.smithi107.stdout:8673: delete oid 13 current snap is 0 2020-06-12T02:38:42.656 INFO:tasks.rados.rados.0.smithi107.stdout:8671: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:42.657 INFO:tasks.rados.rados.0.smithi107.stdout:8671: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:42.657 INFO:tasks.rados.rados.0.smithi107.stdout:8673: done (2 left) 2020-06-12T02:38:42.657 INFO:tasks.rados.rados.0.smithi107.stdout:8674: delete oid 5 current snap is 0 2020-06-12T02:38:42.657 INFO:tasks.rados.rados.0.smithi107.stdout:8671: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:42.657 INFO:tasks.rados.rados.0.smithi107.stdout:8672: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:42.658 INFO:tasks.rados.rados.0.smithi107.stdout:8671: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:42.658 INFO:tasks.rados.rados.0.smithi107.stdout:8672: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:42.658 INFO:tasks.rados.rados.0.smithi107.stdout:8672: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:42.658 INFO:tasks.rados.rados.0.smithi107.stdout:8674: done (2 left) 2020-06-12T02:38:42.658 INFO:tasks.rados.rados.0.smithi107.stdout:8675: delete oid 18 current snap is 0 2020-06-12T02:38:42.658 INFO:tasks.rados.rados.0.smithi107.stdout:8671: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:42.659 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1619 (ObjNum 8670 snap 0 seq_num 8670) dirty exists 2020-06-12T02:38:42.659 INFO:tasks.rados.rados.0.smithi107.stdout:8671: left oid 15 (ObjNum 8670 snap 0 seq_num 8670) 2020-06-12T02:38:42.659 INFO:tasks.rados.rados.0.smithi107.stdout:8672: finishing write tid 5 to smithi10729407-17 2020-06-12T02:38:42.659 INFO:tasks.rados.rados.0.smithi107.stdout:8671: done (2 left) 2020-06-12T02:38:42.659 INFO:tasks.rados.rados.0.smithi107.stdout:8675: done (1 left) 2020-06-12T02:38:42.660 INFO:tasks.rados.rados.0.smithi107.stdout:8676: delete oid 3 current snap is 0 2020-06-12T02:38:42.660 INFO:tasks.rados.rados.0.smithi107.stdout:8672: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:42.660 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1640 (ObjNum 8671 snap 0 seq_num 8671) dirty exists 2020-06-12T02:38:42.660 INFO:tasks.rados.rados.0.smithi107.stdout:8672: left oid 17 (ObjNum 8671 snap 0 seq_num 8671) 2020-06-12T02:38:42.660 INFO:tasks.rados.rados.0.smithi107.stdout:8672: done (1 left) 2020-06-12T02:38:42.660 INFO:tasks.rados.rados.0.smithi107.stdout:8676: done (0 left) 2020-06-12T02:38:42.661 INFO:tasks.rados.rados.0.smithi107.stdout:8677: delete oid 11 current snap is 0 2020-06-12T02:38:42.661 INFO:tasks.rados.rados.0.smithi107.stdout:8677: done (0 left) 2020-06-12T02:38:42.661 INFO:tasks.rados.rados.0.smithi107.stdout:8678: delete oid 14 current snap is 0 2020-06-12T02:38:42.661 INFO:tasks.rados.rados.0.smithi107.stdout:8678: done (0 left) 2020-06-12T02:38:42.661 INFO:tasks.rados.rados.0.smithi107.stdout:8679: delete oid 11 current snap is 0 2020-06-12T02:38:42.662 INFO:tasks.rados.rados.0.smithi107.stdout:8679: done (0 left) 2020-06-12T02:38:42.662 INFO:tasks.rados.rados.0.smithi107.stdout:8680: delete oid 15 current snap is 0 2020-06-12T02:38:42.662 INFO:tasks.rados.rados.0.smithi107.stdout:8680: done (0 left) 2020-06-12T02:38:42.662 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:38:42.662 INFO:tasks.rados.rados.0.smithi107.stdout:8681: seq_num 8680 ranges {1622016=581632,2203648=688128,2891776=114688} 2020-06-12T02:38:42.662 INFO:tasks.rados.rados.0.smithi107.stdout:8681: writing smithi10729407-1 from 1622016 to 2203648 tid 1 2020-06-12T02:38:42.663 INFO:tasks.rados.rados.0.smithi107.stdout:8681: writing smithi10729407-1 from 2203648 to 2891776 tid 2 2020-06-12T02:38:42.663 INFO:tasks.rados.rados.0.smithi107.stdout:8681: writing smithi10729407-1 from 2891776 to 3006464 tid 3 2020-06-12T02:38:42.663 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:42.663 INFO:tasks.rados.rados.0.smithi107.stdout:8682: seq_num 8681 ranges {1990656=598016,2588672=573440,3162112=229376} 2020-06-12T02:38:42.664 INFO:tasks.rados.rados.0.smithi107.stdout:8682: writing smithi10729407-9 from 1990656 to 2588672 tid 1 2020-06-12T02:38:42.664 INFO:tasks.rados.rados.0.smithi107.stdout:8682: writing smithi10729407-9 from 2588672 to 3162112 tid 2 2020-06-12T02:38:42.664 INFO:tasks.rados.rados.0.smithi107.stdout:8682: writing smithi10729407-9 from 3162112 to 3391488 tid 3 2020-06-12T02:38:42.664 INFO:tasks.rados.rados.0.smithi107.stdout:8683: delete oid 7 current snap is 0 2020-06-12T02:38:42.664 INFO:tasks.rados.rados.0.smithi107.stdout:8681: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:42.664 INFO:tasks.rados.rados.0.smithi107.stdout:8681: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:42.665 INFO:tasks.rados.rados.0.smithi107.stdout:8681: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:42.665 INFO:tasks.rados.rados.0.smithi107.stdout:8681: finishing write tid 5 to smithi10729407-1 2020-06-12T02:38:42.665 INFO:tasks.rados.rados.0.smithi107.stdout:8681: finishing write tid 6 to smithi10729407-1 2020-06-12T02:38:42.665 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1578 (ObjNum 8680 snap 0 seq_num 8680) dirty exists 2020-06-12T02:38:42.665 INFO:tasks.rados.rados.0.smithi107.stdout:8681: left oid 1 (ObjNum 8680 snap 0 seq_num 8680) 2020-06-12T02:38:42.666 INFO:tasks.rados.rados.0.smithi107.stdout:8681: done (2 left) 2020-06-12T02:38:42.666 INFO:tasks.rados.rados.0.smithi107.stdout:8683: done (1 left) 2020-06-12T02:38:42.666 INFO:tasks.rados.rados.0.smithi107.stdout:8684: delete oid 17 current snap is 0 2020-06-12T02:38:42.666 INFO:tasks.rados.rados.0.smithi107.stdout:8682: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:42.666 INFO:tasks.rados.rados.0.smithi107.stdout:8682: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:42.666 INFO:tasks.rados.rados.0.smithi107.stdout:8684: done (1 left) 2020-06-12T02:38:42.667 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:42.667 INFO:tasks.rados.rados.0.smithi107.stdout:8685: seq_num 8684 ranges {0=647168,647168=679936,1327104=122880} 2020-06-12T02:38:42.667 INFO:tasks.rados.rados.0.smithi107.stdout:8685: writing smithi10729407-18 from 0 to 647168 tid 1 2020-06-12T02:38:42.667 INFO:tasks.rados.rados.0.smithi107.stdout:8685: writing smithi10729407-18 from 647168 to 1327104 tid 2 2020-06-12T02:38:42.667 INFO:tasks.rados.rados.0.smithi107.stdout:8685: writing smithi10729407-18 from 1327104 to 1449984 tid 3 2020-06-12T02:38:42.668 INFO:tasks.rados.rados.0.smithi107.stdout:8686: delete oid 17 current snap is 0 2020-06-12T02:38:42.668 INFO:tasks.rados.rados.0.smithi107.stdout:8682: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:42.668 INFO:tasks.rados.rados.0.smithi107.stdout:8682: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:42.669 INFO:tasks.rados.rados.0.smithi107.stdout:8682: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:42.669 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1626 (ObjNum 8681 snap 0 seq_num 8681) dirty exists 2020-06-12T02:38:42.669 INFO:tasks.rados.rados.0.smithi107.stdout:8682: left oid 9 (ObjNum 8681 snap 0 seq_num 8681) 2020-06-12T02:38:42.669 INFO:tasks.rados.rados.0.smithi107.stdout:8685: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:42.669 INFO:tasks.rados.rados.0.smithi107.stdout:8682: done (2 left) 2020-06-12T02:38:42.670 INFO:tasks.rados.rados.0.smithi107.stdout:8686: done (1 left) 2020-06-12T02:38:42.670 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:42.670 INFO:tasks.rados.rados.0.smithi107.stdout:8687: seq_num 8686 ranges {843776=679936,1523712=450560,1974272=352256} 2020-06-12T02:38:42.670 INFO:tasks.rados.rados.0.smithi107.stdout:8687: writing smithi10729407-6 from 843776 to 1523712 tid 1 2020-06-12T02:38:42.670 INFO:tasks.rados.rados.0.smithi107.stdout:8687: writing smithi10729407-6 from 1523712 to 1974272 tid 2 2020-06-12T02:38:42.671 INFO:tasks.rados.rados.0.smithi107.stdout:8687: writing smithi10729407-6 from 1974272 to 2326528 tid 3 2020-06-12T02:38:42.671 INFO:tasks.rados.rados.0.smithi107.stdout:8688: delete oid 13 current snap is 0 2020-06-12T02:38:42.671 INFO:tasks.rados.rados.0.smithi107.stdout:8685: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:42.671 INFO:tasks.rados.rados.0.smithi107.stdout:8685: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:42.671 INFO:tasks.rados.rados.0.smithi107.stdout:8685: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:42.672 INFO:tasks.rados.rados.0.smithi107.stdout:8685: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:42.672 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1630 (ObjNum 8684 snap 0 seq_num 8684) dirty exists 2020-06-12T02:38:42.672 INFO:tasks.rados.rados.0.smithi107.stdout:8685: left oid 18 (ObjNum 8684 snap 0 seq_num 8684) 2020-06-12T02:38:42.672 INFO:tasks.rados.rados.0.smithi107.stdout:8687: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:42.672 INFO:tasks.rados.rados.0.smithi107.stdout:8685: done (2 left) 2020-06-12T02:38:42.672 INFO:tasks.rados.rados.0.smithi107.stdout:8688: done (1 left) 2020-06-12T02:38:42.673 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:42.673 INFO:tasks.rados.rados.0.smithi107.stdout:8689: seq_num 8688 ranges {0=712704,712704=614400,1327104=188416} 2020-06-12T02:38:42.673 INFO:tasks.rados.rados.0.smithi107.stdout:8689: writing smithi10729407-14 from 0 to 712704 tid 1 2020-06-12T02:38:42.673 INFO:tasks.rados.rados.0.smithi107.stdout:8689: writing smithi10729407-14 from 712704 to 1327104 tid 2 2020-06-12T02:38:42.673 INFO:teuthology.orchestra.run.smithi107.stderr:set pool 3 pg_num to 26 2020-06-12T02:38:42.674 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2020-06-12T02:38:42.674 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:42.596+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:42.678 INFO:tasks.rados.rados.0.smithi107.stdout:8689: writing smithi10729407-14 from 1327104 to 1515520 tid 3 2020-06-12T02:38:42.679 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:42.679 INFO:tasks.rados.rados.0.smithi107.stdout:8690: seq_num 8689 ranges {0=729088,729088=499712,1228800=303104} 2020-06-12T02:38:42.679 INFO:tasks.rados.rados.0.smithi107.stdout:8690: writing smithi10729407-17 from 0 to 729088 tid 1 2020-06-12T02:38:42.679 INFO:tasks.rados.rados.0.smithi107.stdout:8690: writing smithi10729407-17 from 729088 to 1228800 tid 2 2020-06-12T02:38:42.680 INFO:tasks.rados.rados.0.smithi107.stdout:8690: writing smithi10729407-17 from 1228800 to 1531904 tid 3 2020-06-12T02:38:42.680 INFO:tasks.rados.rados.0.smithi107.stdout:8691: delete oid 13 current snap is 0 2020-06-12T02:38:42.680 INFO:tasks.rados.rados.0.smithi107.stdout:8687: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:42.680 INFO:tasks.rados.rados.0.smithi107.stdout:8687: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:42.680 INFO:tasks.rados.rados.0.smithi107.stdout:8687: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:42.680 INFO:tasks.rados.rados.0.smithi107.stdout:8687: finishing write tid 6 to smithi10729407-6 2020-06-12T02:38:42.681 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1582 (ObjNum 8686 snap 0 seq_num 8686) dirty exists 2020-06-12T02:38:42.681 INFO:tasks.rados.rados.0.smithi107.stdout:8687: left oid 6 (ObjNum 8686 snap 0 seq_num 8686) 2020-06-12T02:38:42.681 INFO:tasks.rados.rados.0.smithi107.stdout:8689: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:42.681 INFO:tasks.rados.rados.0.smithi107.stdout:8689: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:42.681 INFO:tasks.rados.rados.0.smithi107.stdout:8689: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:42.682 INFO:tasks.rados.rados.0.smithi107.stdout:8689: finishing write tid 5 to smithi10729407-14 2020-06-12T02:38:42.682 INFO:tasks.rados.rados.0.smithi107.stdout:8689: finishing write tid 6 to smithi10729407-14 2020-06-12T02:38:42.682 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1526 (ObjNum 8688 snap 0 seq_num 8688) dirty exists 2020-06-12T02:38:42.682 INFO:tasks.rados.rados.0.smithi107.stdout:8689: left oid 14 (ObjNum 8688 snap 0 seq_num 8688) 2020-06-12T02:38:42.682 INFO:tasks.rados.rados.0.smithi107.stdout:8687: done (3 left) 2020-06-12T02:38:42.682 INFO:tasks.rados.rados.0.smithi107.stdout:8689: done (2 left) 2020-06-12T02:38:42.683 INFO:tasks.rados.rados.0.smithi107.stdout:8691: done (1 left) 2020-06-12T02:38:42.683 INFO:tasks.rados.rados.0.smithi107.stdout:8692: delete oid 6 current snap is 0 2020-06-12T02:38:42.683 INFO:tasks.rados.rados.0.smithi107.stdout:8692: done (1 left) 2020-06-12T02:38:42.683 INFO:tasks.rados.rados.0.smithi107.stdout:8693: delete oid 2 current snap is 0 2020-06-12T02:38:42.683 INFO:tasks.rados.rados.0.smithi107.stdout:8690: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:42.684 INFO:tasks.rados.rados.0.smithi107.stdout:8690: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:42.684 INFO:tasks.rados.rados.0.smithi107.stdout:8690: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:42.684 INFO:tasks.rados.rados.0.smithi107.stdout:8690: finishing write tid 5 to smithi10729407-17 2020-06-12T02:38:42.686 INFO:tasks.rados.rados.0.smithi107.stdout:8693: done (1 left) 2020-06-12T02:38:42.686 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:42.686 INFO:tasks.rados.rados.0.smithi107.stdout:8694: seq_num 8693 ranges {0=794624,794624=483328,1277952=319488} 2020-06-12T02:38:42.686 INFO:tasks.rados.rados.0.smithi107.stdout:8694: writing smithi10729407-3 from 0 to 794624 tid 1 2020-06-12T02:38:42.687 INFO:tasks.rados.rados.0.smithi107.stdout:8694: writing smithi10729407-3 from 794624 to 1277952 tid 2 2020-06-12T02:38:42.687 INFO:tasks.rados.rados.0.smithi107.stdout:8694: writing smithi10729407-3 from 1277952 to 1597440 tid 3 2020-06-12T02:38:42.687 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:42.687 INFO:tasks.rados.rados.0.smithi107.stdout:8695: seq_num 8694 ranges {0=409600,409600=770048,1179648=434176} 2020-06-12T02:38:42.687 INFO:tasks.rados.rados.0.smithi107.stdout:8695: writing smithi10729407-15 from 0 to 409600 tid 1 2020-06-12T02:38:42.688 INFO:tasks.rados.rados.0.smithi107.stdout:8695: writing smithi10729407-15 from 409600 to 1179648 tid 2 2020-06-12T02:38:42.688 INFO:tasks.rados.rados.0.smithi107.stdout:8695: writing smithi10729407-15 from 1179648 to 1613824 tid 3 2020-06-12T02:38:42.688 INFO:tasks.rados.rados.0.smithi107.stdout:8696: delete oid 18 current snap is 0 2020-06-12T02:38:42.688 INFO:tasks.rados.rados.0.smithi107.stdout:8690: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:42.688 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1646 (ObjNum 8689 snap 0 seq_num 8689) dirty exists 2020-06-12T02:38:42.689 INFO:tasks.rados.rados.0.smithi107.stdout:8690: left oid 17 (ObjNum 8689 snap 0 seq_num 8689) 2020-06-12T02:38:42.689 INFO:tasks.rados.rados.0.smithi107.stdout:8694: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:42.689 INFO:tasks.rados.rados.0.smithi107.stdout:8695: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:42.689 INFO:tasks.rados.rados.0.smithi107.stdout:8694: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:42.689 INFO:tasks.rados.rados.0.smithi107.stdout:8694: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:42.690 INFO:tasks.rados.rados.0.smithi107.stdout:8694: finishing write tid 5 to smithi10729407-3 2020-06-12T02:38:42.690 INFO:tasks.rados.rados.0.smithi107.stdout:8690: done (3 left) 2020-06-12T02:38:42.690 INFO:tasks.rados.rados.0.smithi107.stdout:8696: done (2 left) 2020-06-12T02:38:42.690 INFO:tasks.rados.rados.0.smithi107.stdout:8697: delete oid 18 current snap is 0 2020-06-12T02:38:42.690 INFO:tasks.rados.rados.0.smithi107.stdout:8694: finishing write tid 6 to smithi10729407-3 2020-06-12T02:38:42.690 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1558 (ObjNum 8693 snap 0 seq_num 8693) dirty exists 2020-06-12T02:38:42.691 INFO:tasks.rados.rados.0.smithi107.stdout:8694: left oid 3 (ObjNum 8693 snap 0 seq_num 8693) 2020-06-12T02:38:42.691 INFO:tasks.rados.rados.0.smithi107.stdout:8694: done (2 left) 2020-06-12T02:38:42.691 INFO:tasks.rados.rados.0.smithi107.stdout:8697: done (1 left) 2020-06-12T02:38:42.691 INFO:tasks.rados.rados.0.smithi107.stdout:8698: delete oid 17 current snap is 0 2020-06-12T02:38:42.691 INFO:tasks.rados.rados.0.smithi107.stdout:8695: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:42.692 INFO:tasks.rados.rados.0.smithi107.stdout:8695: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:42.692 INFO:tasks.rados.rados.0.smithi107.stdout:8698: done (1 left) 2020-06-12T02:38:42.692 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:42.692 INFO:tasks.rados.rados.0.smithi107.stdout:8699: seq_num 8698 ranges {1597440=483328,2080768=761856,2842624=442368} 2020-06-12T02:38:42.692 INFO:tasks.rados.rados.0.smithi107.stdout:8699: writing smithi10729407-3 from 1597440 to 2080768 tid 1 2020-06-12T02:38:42.694 INFO:tasks.rados.rados.0.smithi107.stdout:8699: writing smithi10729407-3 from 2080768 to 2842624 tid 2 2020-06-12T02:38:42.694 INFO:tasks.rados.rados.0.smithi107.stdout:8699: writing smithi10729407-3 from 2842624 to 3284992 tid 3 2020-06-12T02:38:42.695 INFO:tasks.rados.rados.0.smithi107.stdout:8700: delete oid 4 current snap is 0 2020-06-12T02:38:42.695 INFO:tasks.rados.rados.0.smithi107.stdout:8695: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:42.695 INFO:tasks.rados.rados.0.smithi107.stdout:8695: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:42.695 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1624 (ObjNum 8694 snap 0 seq_num 8694) dirty exists 2020-06-12T02:38:42.695 INFO:tasks.rados.rados.0.smithi107.stdout:8695: left oid 15 (ObjNum 8694 snap 0 seq_num 8694) 2020-06-12T02:38:42.696 INFO:tasks.rados.rados.0.smithi107.stdout:8699: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:42.696 INFO:tasks.rados.rados.0.smithi107.stdout:8695: done (2 left) 2020-06-12T02:38:42.696 INFO:tasks.rados.rados.0.smithi107.stdout:8700: done (1 left) 2020-06-12T02:38:42.696 INFO:tasks.rados.rados.0.smithi107.stdout:8701: delete oid 15 current snap is 0 2020-06-12T02:38:42.696 INFO:tasks.rados.rados.0.smithi107.stdout:8701: done (1 left) 2020-06-12T02:38:42.697 INFO:tasks.rados.rados.0.smithi107.stdout:8702: delete oid 13 current snap is 0 2020-06-12T02:38:42.697 INFO:tasks.rados.rados.0.smithi107.stdout:8702: done (1 left) 2020-06-12T02:38:42.698 INFO:tasks.rados.rados.0.smithi107.stdout:8703: delete oid 6 current snap is 0 2020-06-12T02:38:42.698 INFO:tasks.rados.rados.0.smithi107.stdout:8699: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:42.698 INFO:tasks.rados.rados.0.smithi107.stdout:8699: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:42.698 INFO:tasks.rados.rados.0.smithi107.stdout:8703: done (1 left) 2020-06-12T02:38:42.698 INFO:tasks.rados.rados.0.smithi107.stdout:8704: delete oid 16 current snap is 0 2020-06-12T02:38:42.699 INFO:tasks.rados.rados.0.smithi107.stdout:8699: finishing write tid 5 to smithi10729407-3 2020-06-12T02:38:42.700 INFO:tasks.rados.rados.0.smithi107.stdout:8699: finishing write tid 6 to smithi10729407-3 2020-06-12T02:38:42.700 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1562 (ObjNum 8698 snap 0 seq_num 8698) dirty exists 2020-06-12T02:38:42.700 INFO:tasks.rados.rados.0.smithi107.stdout:8699: left oid 3 (ObjNum 8698 snap 0 seq_num 8698) 2020-06-12T02:38:42.701 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:42.697+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:42.702 INFO:tasks.rados.rados.0.smithi107.stdout:8699: done (1 left) 2020-06-12T02:38:42.703 INFO:tasks.rados.rados.0.smithi107.stdout:8704: done (0 left) 2020-06-12T02:38:42.703 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:42.703 INFO:tasks.rados.rados.0.smithi107.stdout:8705: seq_num 8704 ranges {950272=581632,1531904=466944,1998848=638976,2637824=98304} 2020-06-12T02:38:42.711 INFO:tasks.rados.rados.0.smithi107.stdout:8705: writing smithi10729407-19 from 950272 to 1531904 tid 1 2020-06-12T02:38:42.714 INFO:tasks.rados.rados.0.smithi107.stdout:8705: writing smithi10729407-19 from 1531904 to 1998848 tid 2 2020-06-12T02:38:42.717 INFO:tasks.rados.rados.0.smithi107.stdout:8705: writing smithi10729407-19 from 1998848 to 2637824 tid 3 2020-06-12T02:38:42.718 INFO:tasks.rados.rados.0.smithi107.stdout:8705: writing smithi10729407-19 from 2637824 to 2736128 tid 4 2020-06-12T02:38:42.718 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:38:42.718 INFO:tasks.rados.rados.0.smithi107.stdout:8706: seq_num 8705 ranges {3006464=598016,3604480=802816,4407296=401408} 2020-06-12T02:38:42.747 INFO:tasks.rados.rados.0.smithi107.stdout:8706: writing smithi10729407-1 from 3006464 to 3604480 tid 1 2020-06-12T02:38:42.747 INFO:tasks.rados.rados.0.smithi107.stdout:8706: writing smithi10729407-1 from 3604480 to 4407296 tid 2 2020-06-12T02:38:42.747 INFO:tasks.rados.rados.0.smithi107.stdout:8706: writing smithi10729407-1 from 4407296 to 4808704 tid 3 2020-06-12T02:38:42.748 INFO:tasks.rados.rados.0.smithi107.stdout:8707: delete oid 3 current snap is 0 2020-06-12T02:38:42.748 INFO:tasks.rados.rados.0.smithi107.stdout:8705: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:42.748 INFO:tasks.rados.rados.0.smithi107.stdout:8705: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:42.748 INFO:tasks.rados.rados.0.smithi107.stdout:8705: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:42.748 INFO:tasks.rados.rados.0.smithi107.stdout:8705: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:42.749 INFO:tasks.rados.rados.0.smithi107.stdout:8705: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:42.749 INFO:tasks.rados.rados.0.smithi107.stdout:8705: finishing write tid 7 to smithi10729407-19 2020-06-12T02:38:42.749 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1592 (ObjNum 8704 snap 0 seq_num 8704) dirty exists 2020-06-12T02:38:42.749 INFO:tasks.rados.rados.0.smithi107.stdout:8705: left oid 19 (ObjNum 8704 snap 0 seq_num 8704) 2020-06-12T02:38:42.749 INFO:tasks.rados.rados.0.smithi107.stdout:8706: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:42.750 INFO:tasks.rados.rados.0.smithi107.stdout:8705: done (2 left) 2020-06-12T02:38:42.750 INFO:tasks.rados.rados.0.smithi107.stdout:8707: done (1 left) 2020-06-12T02:38:42.750 INFO:tasks.rados.rados.0.smithi107.stdout:8708: delete oid 18 current snap is 0 2020-06-12T02:38:42.752 INFO:tasks.rados.rados.0.smithi107.stdout:8708: done (1 left) 2020-06-12T02:38:42.753 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:42.753 INFO:tasks.rados.rados.0.smithi107.stdout:8709: seq_num 8708 ranges {0=647168,647168=450560,1097728=614400,1712128=139264} 2020-06-12T02:38:42.756 INFO:tasks.rados.rados.0.smithi107.stdout:8709: writing smithi10729407-18 from 0 to 647168 tid 1 2020-06-12T02:38:42.759 INFO:tasks.rados.rados.0.smithi107.stdout:8709: writing smithi10729407-18 from 647168 to 1097728 tid 2 2020-06-12T02:38:42.762 INFO:tasks.rados.rados.0.smithi107.stdout:8709: writing smithi10729407-18 from 1097728 to 1712128 tid 3 2020-06-12T02:38:42.763 INFO:tasks.rados.rados.0.smithi107.stdout:8709: writing smithi10729407-18 from 1712128 to 1851392 tid 4 2020-06-12T02:38:42.764 INFO:tasks.rados.rados.0.smithi107.stdout:8710: delete oid 17 current snap is 0 2020-06-12T02:38:42.764 INFO:tasks.rados.rados.0.smithi107.stdout:8706: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:42.764 INFO:tasks.rados.rados.0.smithi107.stdout:8706: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:42.764 INFO:tasks.rados.rados.0.smithi107.stdout:8706: finishing write tid 5 to smithi10729407-1 2020-06-12T02:38:42.764 INFO:tasks.rados.rados.0.smithi107.stdout:8706: finishing write tid 6 to smithi10729407-1 2020-06-12T02:38:42.765 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1588 (ObjNum 8705 snap 0 seq_num 8705) dirty exists 2020-06-12T02:38:42.765 INFO:tasks.rados.rados.0.smithi107.stdout:8706: left oid 1 (ObjNum 8705 snap 0 seq_num 8705) 2020-06-12T02:38:42.765 INFO:tasks.rados.rados.0.smithi107.stdout:8709: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:42.772 INFO:tasks.rados.rados.0.smithi107.stdout:8706: done (2 left) 2020-06-12T02:38:42.772 INFO:tasks.rados.rados.0.smithi107.stdout:8710: done (1 left) 2020-06-12T02:38:42.772 INFO:tasks.rados.rados.0.smithi107.stdout:8711: delete oid 19 current snap is 0 2020-06-12T02:38:42.773 INFO:tasks.rados.rados.0.smithi107.stdout:8709: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:42.774 INFO:tasks.rados.rados.0.smithi107.stdout:8711: done (1 left) 2020-06-12T02:38:42.774 INFO:tasks.rados.rados.0.smithi107.stdout:8712: delete oid 11 current snap is 0 2020-06-12T02:38:42.774 INFO:tasks.rados.rados.0.smithi107.stdout:8709: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:42.775 INFO:tasks.rados.rados.0.smithi107.stdout:8709: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:42.778 INFO:tasks.rados.rados.0.smithi107.stdout:8712: done (1 left) 2020-06-12T02:38:42.779 INFO:tasks.rados.rados.0.smithi107.stdout:8713: delete oid 12 current snap is 0 2020-06-12T02:38:42.779 INFO:tasks.rados.rados.0.smithi107.stdout:8709: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:42.780 INFO:tasks.rados.rados.0.smithi107.stdout:8709: finishing write tid 7 to smithi10729407-18 2020-06-12T02:38:42.780 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1638 (ObjNum 8708 snap 0 seq_num 8708) dirty exists 2020-06-12T02:38:42.781 INFO:tasks.rados.rados.0.smithi107.stdout:8709: left oid 18 (ObjNum 8708 snap 0 seq_num 8708) 2020-06-12T02:38:42.781 INFO:tasks.rados.rados.0.smithi107.stdout:8709: done (1 left) 2020-06-12T02:38:42.782 INFO:tasks.rados.rados.0.smithi107.stdout:8713: done (0 left) 2020-06-12T02:38:42.782 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:38:42.782 INFO:tasks.rados.rados.0.smithi107.stdout:8714: seq_num 8713 ranges {0=729088,729088=720896,1449984=483328} 2020-06-12T02:38:42.785 INFO:tasks.rados.rados.0.smithi107.stdout:8714: writing smithi10729407-20 from 0 to 729088 tid 1 2020-06-12T02:38:42.789 INFO:tasks.rados.rados.0.smithi107.stdout:8714: writing smithi10729407-20 from 729088 to 1449984 tid 2 2020-06-12T02:38:42.792 INFO:tasks.rados.rados.0.smithi107.stdout:8714: writing smithi10729407-20 from 1449984 to 1933312 tid 3 2020-06-12T02:38:42.793 INFO:tasks.rados.rados.0.smithi107.stdout:8715: delete oid 13 current snap is 0 2020-06-12T02:38:42.793 INFO:tasks.rados.rados.0.smithi107.stdout:8714: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:42.796 INFO:tasks.rados.rados.0.smithi107.stdout:8715: done (1 left) 2020-06-12T02:38:42.796 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:42.797 INFO:tasks.rados.rados.0.smithi107.stdout:8716: seq_num 8715 ranges {0=770048,770048=491520,1261568=417792,1679360=294912} 2020-06-12T02:38:42.801 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:42.798+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:42.803 INFO:tasks.rados.rados.0.smithi107.stdout:8716: writing smithi10729407-19 from 0 to 770048 tid 1 2020-06-12T02:38:42.804 INFO:tasks.rados.rados.0.smithi107.stdout:8716: writing smithi10729407-19 from 770048 to 1261568 tid 2 2020-06-12T02:38:42.806 INFO:tasks.rados.rados.0.smithi107.stdout:8716: writing smithi10729407-19 from 1261568 to 1679360 tid 3 2020-06-12T02:38:42.808 INFO:tasks.rados.rados.0.smithi107.stdout:8716: writing smithi10729407-19 from 1679360 to 1974272 tid 4 2020-06-12T02:38:42.809 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:38:42.810 INFO:tasks.rados.rados.0.smithi107.stdout:8717: seq_num 8716 ranges {0=786432,786432=778240,1564672=425984} 2020-06-12T02:38:42.813 INFO:tasks.rados.rados.0.smithi107.stdout:8717: writing smithi10729407-10 from 0 to 786432 tid 1 2020-06-12T02:38:42.816 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:42.817 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:38:42.817 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2020-06-12T02:38:42.817 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:42.825 INFO:tasks.rados.rados.0.smithi107.stdout:8717: writing smithi10729407-10 from 786432 to 1564672 tid 2 2020-06-12T02:38:42.826 INFO:tasks.rados.rados.0.smithi107.stdout:8717: writing smithi10729407-10 from 1564672 to 1990656 tid 3 2020-06-12T02:38:42.826 INFO:tasks.rados.rados.0.smithi107.stdout:8718: delete oid 18 current snap is 0 2020-06-12T02:38:42.826 INFO:tasks.rados.rados.0.smithi107.stdout:8714: finishing write tid 2 to smithi10729407-20 2020-06-12T02:38:42.827 INFO:tasks.rados.rados.0.smithi107.stdout:8714: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:42.827 INFO:tasks.rados.rados.0.smithi107.stdout:8714: finishing write tid 5 to smithi10729407-20 2020-06-12T02:38:42.827 INFO:tasks.rados.rados.0.smithi107.stdout:8714: finishing write tid 6 to smithi10729407-20 2020-06-12T02:38:42.827 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 639 (ObjNum 8713 snap 0 seq_num 8713) dirty exists 2020-06-12T02:38:42.827 INFO:tasks.rados.rados.0.smithi107.stdout:8714: left oid 20 (ObjNum 8713 snap 0 seq_num 8713) 2020-06-12T02:38:42.827 INFO:tasks.rados.rados.0.smithi107.stdout:8716: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:42.828 INFO:tasks.rados.rados.0.smithi107.stdout:8716: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:42.828 INFO:tasks.rados.rados.0.smithi107.stdout:8716: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:42.828 INFO:tasks.rados.rados.0.smithi107.stdout:8716: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:42.828 INFO:tasks.rados.rados.0.smithi107.stdout:8716: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:42.828 INFO:tasks.rados.rados.0.smithi107.stdout:8717: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:42.829 INFO:tasks.rados.rados.0.smithi107.stdout:8716: finishing write tid 7 to smithi10729407-19 2020-06-12T02:38:42.829 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1599 (ObjNum 8715 snap 0 seq_num 8715) dirty exists 2020-06-12T02:38:42.829 INFO:tasks.rados.rados.0.smithi107.stdout:8716: left oid 19 (ObjNum 8715 snap 0 seq_num 8715) 2020-06-12T02:38:42.830 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-06-12T02:38:42.831 INFO:tasks.rados.rados.0.smithi107.stdout:8714: done (3 left) 2020-06-12T02:38:42.831 INFO:tasks.rados.rados.0.smithi107.stdout:8716: done (2 left) 2020-06-12T02:38:42.831 INFO:tasks.rados.rados.0.smithi107.stdout:8718: done (1 left) 2020-06-12T02:38:42.832 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:42.832 INFO:tasks.rados.rados.0.smithi107.stdout:8719: seq_num 8718 ranges {0=417792,417792=540672,958464=589824,1548288=475136} 2020-06-12T02:38:42.832 INFO:tasks.rados.rados.0.smithi107.stdout:8719: writing smithi10729407-12 from 0 to 417792 tid 1 2020-06-12T02:38:42.834 INFO:tasks.rados.rados.0.smithi107.stdout:8719: writing smithi10729407-12 from 417792 to 958464 tid 2 2020-06-12T02:38:42.838 INFO:tasks.rados.rados.0.smithi107.stdout:8719: writing smithi10729407-12 from 958464 to 1548288 tid 3 2020-06-12T02:38:42.841 INFO:tasks.rados.rados.0.smithi107.stdout:8719: writing smithi10729407-12 from 1548288 to 2023424 tid 4 2020-06-12T02:38:42.841 INFO:tasks.rados.rados.0.smithi107.stdout:8720: delete oid 20 current snap is 0 2020-06-12T02:38:42.842 INFO:tasks.rados.rados.0.smithi107.stdout:8717: finishing write tid 2 to smithi10729407-10 2020-06-12T02:38:42.842 INFO:tasks.rados.rados.0.smithi107.stdout:8717: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:42.842 INFO:tasks.rados.rados.0.smithi107.stdout:8717: finishing write tid 5 to smithi10729407-10 2020-06-12T02:38:42.842 INFO:tasks.rados.rados.0.smithi107.stdout:8717: finishing write tid 6 to smithi10729407-10 2020-06-12T02:38:42.842 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 632 (ObjNum 8716 snap 0 seq_num 8716) dirty exists 2020-06-12T02:38:42.843 INFO:tasks.rados.rados.0.smithi107.stdout:8717: left oid 10 (ObjNum 8716 snap 0 seq_num 8716) 2020-06-12T02:38:42.843 INFO:tasks.rados.rados.0.smithi107.stdout:8719: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:42.843 INFO:tasks.rados.rados.0.smithi107.stdout:8717: done (2 left) 2020-06-12T02:38:42.844 INFO:tasks.rados.rados.0.smithi107.stdout:8720: done (1 left) 2020-06-12T02:38:42.844 INFO:tasks.rados.rados.0.smithi107.stdout:8721: delete oid 16 current snap is 0 2020-06-12T02:38:42.847 INFO:tasks.rados.rados.0.smithi107.stdout:8721: done (1 left) 2020-06-12T02:38:42.847 INFO:tasks.rados.rados.0.smithi107.stdout:8722: delete oid 8 current snap is 0 2020-06-12T02:38:42.848 INFO:tasks.rados.rados.0.smithi107.stdout:8719: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:42.852 INFO:tasks.rados.rados.0.smithi107.stdout:8722: done (1 left) 2020-06-12T02:38:42.852 INFO:tasks.rados.rados.0.smithi107.stdout:8723: delete oid 16 current snap is 0 2020-06-12T02:38:42.853 INFO:tasks.rados.rados.0.smithi107.stdout:8719: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:42.853 INFO:tasks.rados.rados.0.smithi107.stdout:8719: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:42.853 INFO:tasks.rados.rados.0.smithi107.stdout:8719: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:42.855 INFO:tasks.rados.rados.0.smithi107.stdout:8723: done (1 left) 2020-06-12T02:38:42.855 INFO:tasks.rados.rados.0.smithi107.stdout:8724: delete oid 15 current snap is 0 2020-06-12T02:38:42.855 INFO:tasks.rados.rados.0.smithi107.stdout:8719: finishing write tid 7 to smithi10729407-12 2020-06-12T02:38:42.856 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1604 (ObjNum 8718 snap 0 seq_num 8718) dirty exists 2020-06-12T02:38:42.856 INFO:tasks.rados.rados.0.smithi107.stdout:8719: left oid 12 (ObjNum 8718 snap 0 seq_num 8718) 2020-06-12T02:38:42.858 INFO:tasks.rados.rados.0.smithi107.stdout:8719: done (1 left) 2020-06-12T02:38:42.858 INFO:tasks.rados.rados.0.smithi107.stdout:8724: done (0 left) 2020-06-12T02:38:42.858 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:42.858 INFO:tasks.rados.rados.0.smithi107.stdout:8725: seq_num 8724 ranges {1974272=516096,2490368=704512,3194880=540672,3735552=360448} 2020-06-12T02:38:42.872 INFO:tasks.rados.rados.0.smithi107.stdout:8725: writing smithi10729407-19 from 1974272 to 2490368 tid 1 2020-06-12T02:38:42.876 INFO:tasks.rados.rados.0.smithi107.stdout:8725: writing smithi10729407-19 from 2490368 to 3194880 tid 2 2020-06-12T02:38:42.879 INFO:tasks.rados.rados.0.smithi107.stdout:8725: writing smithi10729407-19 from 3194880 to 3735552 tid 3 2020-06-12T02:38:42.881 INFO:tasks.rados.rados.0.smithi107.stdout:8725: writing smithi10729407-19 from 3735552 to 4096000 tid 4 2020-06-12T02:38:42.881 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:38:42.881 INFO:tasks.rados.rados.0.smithi107.stdout:8726: seq_num 8725 ranges {0=532480,532480=581632,1114112=745472,1859584=278528} 2020-06-12T02:38:42.884 INFO:tasks.rados.rados.0.smithi107.stdout:8726: writing smithi10729407-6 from 0 to 532480 tid 1 2020-06-12T02:38:42.888 INFO:tasks.rados.rados.0.smithi107.stdout:8726: writing smithi10729407-6 from 532480 to 1114112 tid 2 2020-06-12T02:38:42.893 INFO:tasks.rados.rados.0.smithi107.stdout:8726: writing smithi10729407-6 from 1114112 to 1859584 tid 3 2020-06-12T02:38:42.895 INFO:tasks.rados.rados.0.smithi107.stdout:8726: writing smithi10729407-6 from 1859584 to 2138112 tid 4 2020-06-12T02:38:42.895 INFO:tasks.rados.rados.0.smithi107.stdout:8727: delete oid 9 current snap is 0 2020-06-12T02:38:42.895 INFO:tasks.rados.rados.0.smithi107.stdout:8725: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:42.895 INFO:tasks.rados.rados.0.smithi107.stdout:8725: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:42.896 INFO:tasks.rados.rados.0.smithi107.stdout:8725: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:42.896 INFO:tasks.rados.rados.0.smithi107.stdout:8725: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:42.896 INFO:tasks.rados.rados.0.smithi107.stdout:8725: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:42.896 INFO:tasks.rados.rados.0.smithi107.stdout:8726: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:42.897 INFO:tasks.rados.rados.0.smithi107.stdout:8725: finishing write tid 7 to smithi10729407-19 2020-06-12T02:38:42.897 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1609 (ObjNum 8724 snap 0 seq_num 8724) dirty exists 2020-06-12T02:38:42.897 INFO:tasks.rados.rados.0.smithi107.stdout:8725: left oid 19 (ObjNum 8724 snap 0 seq_num 8724) 2020-06-12T02:38:42.898 INFO:tasks.rados.rados.0.smithi107.stdout:8725: done (2 left) 2020-06-12T02:38:42.898 INFO:tasks.rados.rados.0.smithi107.stdout:8727: done (1 left) 2020-06-12T02:38:42.898 INFO:tasks.rados.rados.0.smithi107.stdout:8728: delete oid 2 current snap is 0 2020-06-12T02:38:42.902 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:42.899+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:42.902 INFO:tasks.rados.rados.0.smithi107.stdout:8726: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:42.905 INFO:tasks.rados.rados.0.smithi107.stdout:8728: done (1 left) 2020-06-12T02:38:42.905 INFO:tasks.rados.rados.0.smithi107.stdout:8729: delete oid 20 current snap is 0 2020-06-12T02:38:42.906 INFO:tasks.rados.rados.0.smithi107.stdout:8726: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:42.906 INFO:tasks.rados.rados.0.smithi107.stdout:8726: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:42.907 INFO:tasks.rados.rados.0.smithi107.stdout:8726: finishing write tid 6 to smithi10729407-6 2020-06-12T02:38:42.907 INFO:tasks.rados.rados.0.smithi107.stdout:8729: done (1 left) 2020-06-12T02:38:42.907 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:42.907 INFO:tasks.rados.rados.0.smithi107.stdout:8730: seq_num 8729 ranges {0=598016,598016=573440,1171456=729088,1900544=303104} 2020-06-12T02:38:42.910 INFO:tasks.rados.rados.0.smithi107.stdout:8730: writing smithi10729407-13 from 0 to 598016 tid 1 2020-06-12T02:38:42.914 INFO:tasks.rados.rados.0.smithi107.stdout:8730: writing smithi10729407-13 from 598016 to 1171456 tid 2 2020-06-12T02:38:42.923 INFO:tasks.rados.rados.0.smithi107.stdout:8730: writing smithi10729407-13 from 1171456 to 1900544 tid 3 2020-06-12T02:38:42.924 INFO:tasks.rados.rados.0.smithi107.stdout:8730: writing smithi10729407-13 from 1900544 to 2203648 tid 4 2020-06-12T02:38:42.925 INFO:tasks.rados.rados.0.smithi107.stdout:8731: delete oid 14 current snap is 0 2020-06-12T02:38:42.925 INFO:tasks.rados.rados.0.smithi107.stdout:8726: finishing write tid 7 to smithi10729407-6 2020-06-12T02:38:42.925 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1593 (ObjNum 8725 snap 0 seq_num 8725) dirty exists 2020-06-12T02:38:42.925 INFO:tasks.rados.rados.0.smithi107.stdout:8726: left oid 6 (ObjNum 8725 snap 0 seq_num 8725) 2020-06-12T02:38:42.925 INFO:tasks.rados.rados.0.smithi107.stdout:8730: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:42.926 INFO:tasks.rados.rados.0.smithi107.stdout:8726: done (2 left) 2020-06-12T02:38:42.926 INFO:tasks.rados.rados.0.smithi107.stdout:8731: done (1 left) 2020-06-12T02:38:42.927 INFO:tasks.rados.rados.0.smithi107.stdout:8732: delete oid 20 current snap is 0 2020-06-12T02:38:42.927 INFO:tasks.rados.rados.0.smithi107.stdout:8730: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:42.927 INFO:tasks.rados.rados.0.smithi107.stdout:8732: done (1 left) 2020-06-12T02:38:42.927 INFO:tasks.rados.rados.0.smithi107.stdout:8733: delete oid 7 current snap is 0 2020-06-12T02:38:42.929 INFO:tasks.rados.rados.0.smithi107.stdout:8730: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:42.930 INFO:tasks.rados.rados.0.smithi107.stdout:8730: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:42.930 INFO:tasks.rados.rados.0.smithi107.stdout:8730: finishing write tid 6 to smithi10729407-13 2020-06-12T02:38:42.931 INFO:tasks.rados.rados.0.smithi107.stdout:8733: done (1 left) 2020-06-12T02:38:42.931 INFO:tasks.rados.rados.0.smithi107.stdout:8734: delete oid 14 current snap is 0 2020-06-12T02:38:42.932 INFO:tasks.rados.rados.0.smithi107.stdout:8730: finishing write tid 7 to smithi10729407-13 2020-06-12T02:38:42.932 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1577 (ObjNum 8729 snap 0 seq_num 8729) dirty exists 2020-06-12T02:38:42.932 INFO:tasks.rados.rados.0.smithi107.stdout:8730: left oid 13 (ObjNum 8729 snap 0 seq_num 8729) 2020-06-12T02:38:42.933 INFO:tasks.rados.rados.0.smithi107.stdout:8730: done (1 left) 2020-06-12T02:38:42.934 INFO:tasks.rados.rados.0.smithi107.stdout:8734: done (0 left) 2020-06-12T02:38:42.934 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:42.934 INFO:tasks.rados.rados.0.smithi107.stdout:8735: seq_num 8734 ranges {0=679936,679936=794624,1474560=516096,1990656=294912} 2020-06-12T02:38:42.937 INFO:tasks.rados.rados.0.smithi107.stdout:8735: writing smithi10729407-2 from 0 to 679936 tid 1 2020-06-12T02:38:42.942 INFO:tasks.rados.rados.0.smithi107.stdout:8735: writing smithi10729407-2 from 679936 to 1474560 tid 2 2020-06-12T02:38:42.947 INFO:tasks.rados.rados.0.smithi107.stdout:8735: writing smithi10729407-2 from 1474560 to 1990656 tid 3 2020-06-12T02:38:42.948 INFO:tasks.rados.rados.0.smithi107.stdout:8735: writing smithi10729407-2 from 1990656 to 2285568 tid 4 2020-06-12T02:38:42.948 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:38:42.948 INFO:tasks.rados.rados.0.smithi107.stdout:8736: seq_num 8735 ranges {0=704512,704512=679936,1384448=720896,2105344=204800} 2020-06-12T02:38:42.950 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-06-12T02:38:42.951 INFO:tasks.rados.rados.0.smithi107.stdout:8736: writing smithi10729407-3 from 0 to 704512 tid 1 2020-06-12T02:38:42.956 INFO:tasks.rados.rados.0.smithi107.stdout:8736: writing smithi10729407-3 from 704512 to 1384448 tid 2 2020-06-12T02:38:42.961 INFO:tasks.rados.rados.0.smithi107.stdout:8736: writing smithi10729407-3 from 1384448 to 2105344 tid 3 2020-06-12T02:38:42.963 INFO:tasks.rados.rados.0.smithi107.stdout:8736: writing smithi10729407-3 from 2105344 to 2310144 tid 4 2020-06-12T02:38:42.963 INFO:tasks.rados.rados.0.smithi107.stdout:8737: delete oid 20 current snap is 0 2020-06-12T02:38:42.965 INFO:tasks.rados.rados.0.smithi107.stdout:8735: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:42.965 INFO:tasks.rados.rados.0.smithi107.stdout:8735: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:42.965 INFO:tasks.rados.rados.0.smithi107.stdout:8735: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:42.965 INFO:tasks.rados.rados.0.smithi107.stdout:8735: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:42.966 INFO:tasks.rados.rados.0.smithi107.stdout:8735: finishing write tid 6 to smithi10729407-2 2020-06-12T02:38:42.966 INFO:tasks.rados.rados.0.smithi107.stdout:8735: finishing write tid 7 to smithi10729407-2 2020-06-12T02:38:42.966 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 813 (ObjNum 8734 snap 0 seq_num 8734) dirty exists 2020-06-12T02:38:42.966 INFO:tasks.rados.rados.0.smithi107.stdout:8735: left oid 2 (ObjNum 8734 snap 0 seq_num 8734) 2020-06-12T02:38:42.967 INFO:tasks.rados.rados.0.smithi107.stdout:8735: done (2 left) 2020-06-12T02:38:42.967 INFO:tasks.rados.rados.0.smithi107.stdout:8737: done (1 left) 2020-06-12T02:38:42.967 INFO:tasks.rados.rados.0.smithi107.stdout:8738: delete oid 16 current snap is 0 2020-06-12T02:38:42.971 INFO:tasks.rados.rados.0.smithi107.stdout:8736: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:42.972 INFO:tasks.rados.rados.0.smithi107.stdout:8736: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:42.972 INFO:tasks.rados.rados.0.smithi107.stdout:8736: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:42.974 INFO:tasks.rados.rados.0.smithi107.stdout:8736: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:42.974 INFO:tasks.rados.rados.0.smithi107.stdout:8736: finishing write tid 6 to smithi10729407-3 2020-06-12T02:38:42.977 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:42.978 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:38:42.978 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2020-06-12T02:38:42.978 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2020-06-12T02:38:42.978 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:42.979 INFO:tasks.rados.rados.0.smithi107.stdout:8736: finishing write tid 7 to smithi10729407-3 2020-06-12T02:38:42.979 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1571 (ObjNum 8735 snap 0 seq_num 8735) dirty exists 2020-06-12T02:38:42.979 INFO:tasks.rados.rados.0.smithi107.stdout:8736: left oid 3 (ObjNum 8735 snap 0 seq_num 8735) 2020-06-12T02:38:42.980 INFO:tasks.rados.rados.0.smithi107.stdout:8736: done (1 left) 2020-06-12T02:38:42.981 INFO:tasks.rados.rados.0.smithi107.stdout:8738: done (0 left) 2020-06-12T02:38:42.981 INFO:tasks.rados.rados.0.smithi107.stdout:8739: delete oid 15 current snap is 0 2020-06-12T02:38:42.982 INFO:tasks.rados.rados.0.smithi107.stdout:8739: done (0 left) 2020-06-12T02:38:42.982 INFO:tasks.rados.rados.0.smithi107.stdout:8740: delete oid 3 current snap is 0 2020-06-12T02:38:42.983 INFO:tasks.rados.rados.0.smithi107.stdout:8740: done (0 left) 2020-06-12T02:38:42.983 INFO:tasks.rados.rados.0.smithi107.stdout:8741: delete oid 6 current snap is 0 2020-06-12T02:38:42.988 INFO:tasks.rados.rados.0.smithi107.stdout:8741: done (0 left) 2020-06-12T02:38:42.988 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:42.989 INFO:tasks.rados.rados.0.smithi107.stdout:8742: seq_num 8741 ranges {0=802816,802816=434176,1236992=671744,1908736=499712} 2020-06-12T02:38:42.989 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-06-12T02:38:42.991 INFO:tasks.rados.rados.0.smithi107.stdout:8742: writing smithi10729407-9 from 0 to 802816 tid 1 2020-06-12T02:38:42.997 INFO:tasks.rados.rados.0.smithi107.stdout:8742: writing smithi10729407-9 from 802816 to 1236992 tid 2 2020-06-12T02:38:42.998 INFO:tasks.rados.rados.0.smithi107.stdout:8742: writing smithi10729407-9 from 1236992 to 1908736 tid 3 2020-06-12T02:38:43.001 INFO:tasks.rados.rados.0.smithi107.stdout:8742: writing smithi10729407-9 from 1908736 to 2408448 tid 4 2020-06-12T02:38:43.001 INFO:tasks.rados.rados.0.smithi107.stdout:8743: delete oid 3 current snap is 0 2020-06-12T02:38:43.003 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:43.001+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:43.004 INFO:tasks.rados.rados.0.smithi107.stdout:8743: done (1 left) 2020-06-12T02:38:43.004 INFO:tasks.rados.rados.0.smithi107.stdout:8744: delete oid 19 current snap is 0 2020-06-12T02:38:43.004 INFO:tasks.rados.rados.0.smithi107.stdout:8742: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:43.005 INFO:tasks.rados.rados.0.smithi107.stdout:8742: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:43.009 INFO:tasks.rados.rados.0.smithi107.stdout:8744: done (1 left) 2020-06-12T02:38:43.009 INFO:tasks.rados.rados.0.smithi107.stdout:8745: delete oid 12 current snap is 0 2020-06-12T02:38:43.012 INFO:tasks.rados.rados.0.smithi107.stdout:8742: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:43.013 INFO:tasks.rados.rados.0.smithi107.stdout:8742: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:43.013 INFO:tasks.rados.rados.0.smithi107.stdout:8742: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:43.017 INFO:tasks.rados.rados.0.smithi107.stdout:8745: done (1 left) 2020-06-12T02:38:43.017 INFO:tasks.rados.rados.0.smithi107.stdout:8746: delete oid 3 current snap is 0 2020-06-12T02:38:43.017 INFO:tasks.rados.rados.0.smithi107.stdout:8742: finishing write tid 7 to smithi10729407-9 2020-06-12T02:38:43.018 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1645 (ObjNum 8741 snap 0 seq_num 8741) dirty exists 2020-06-12T02:38:43.018 INFO:tasks.rados.rados.0.smithi107.stdout:8742: left oid 9 (ObjNum 8741 snap 0 seq_num 8741) 2020-06-12T02:38:43.020 INFO:tasks.rados.rados.0.smithi107.stdout:8742: done (1 left) 2020-06-12T02:38:43.020 INFO:tasks.rados.rados.0.smithi107.stdout:8746: done (0 left) 2020-06-12T02:38:43.021 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:43.021 INFO:tasks.rados.rados.0.smithi107.stdout:8747: seq_num 8746 ranges {2408448=81920} 2020-06-12T02:38:43.033 INFO:tasks.rados.rados.0.smithi107.stdout:8747: writing smithi10729407-9 from 2408448 to 2490368 tid 1 2020-06-12T02:38:43.034 INFO:tasks.rados.rados.0.smithi107.stdout:8748: delete oid 11 current snap is 0 2020-06-12T02:38:43.038 INFO:tasks.rados.rados.0.smithi107.stdout:8747: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:43.038 INFO:tasks.rados.rados.0.smithi107.stdout:8748: done (1 left) 2020-06-12T02:38:43.038 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:43.038 INFO:tasks.rados.rados.0.smithi107.stdout:8749: seq_num 8748 ranges {0=114688} 2020-06-12T02:38:43.039 INFO:tasks.rados.rados.0.smithi107.stdout:8749: writing smithi10729407-11 from 0 to 114688 tid 1 2020-06-12T02:38:43.039 INFO:tasks.rados.rados.0.smithi107.stdout:8750: delete oid 1 current snap is 0 2020-06-12T02:38:43.040 INFO:tasks.rados.rados.0.smithi107.stdout:8747: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:43.042 INFO:tasks.rados.rados.0.smithi107.stdout:8747: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:43.043 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1647 (ObjNum 8746 snap 0 seq_num 8746) dirty exists 2020-06-12T02:38:43.043 INFO:tasks.rados.rados.0.smithi107.stdout:8747: left oid 9 (ObjNum 8746 snap 0 seq_num 8746) 2020-06-12T02:38:43.043 INFO:tasks.rados.rados.0.smithi107.stdout:8747: done (2 left) 2020-06-12T02:38:43.044 INFO:tasks.rados.rados.0.smithi107.stdout:8750: done (1 left) 2020-06-12T02:38:43.044 INFO:tasks.rados.rados.0.smithi107.stdout:8751: delete oid 20 current snap is 0 2020-06-12T02:38:43.046 INFO:tasks.rados.rados.0.smithi107.stdout:8751: done (1 left) 2020-06-12T02:38:43.046 INFO:tasks.rados.rados.0.smithi107.stdout:8752: delete oid 5 current snap is 0 2020-06-12T02:38:43.048 INFO:tasks.rados.rados.0.smithi107.stdout:8749: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:43.048 INFO:tasks.rados.rados.0.smithi107.stdout:8749: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:43.048 INFO:tasks.rados.rados.0.smithi107.stdout:8752: done (1 left) 2020-06-12T02:38:43.049 INFO:tasks.rados.rados.0.smithi107.stdout:8753: delete oid 8 current snap is 0 2020-06-12T02:38:43.050 INFO:tasks.rados.rados.0.smithi107.stdout:8749: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:43.050 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1532 (ObjNum 8748 snap 0 seq_num 8748) dirty exists 2020-06-12T02:38:43.050 INFO:tasks.rados.rados.0.smithi107.stdout:8749: left oid 11 (ObjNum 8748 snap 0 seq_num 8748) 2020-06-12T02:38:43.056 INFO:tasks.rados.rados.0.smithi107.stdout:8749: done (1 left) 2020-06-12T02:38:43.057 INFO:tasks.rados.rados.0.smithi107.stdout:8753: done (0 left) 2020-06-12T02:38:43.057 INFO:tasks.rados.rados.0.smithi107.stdout:8754: delete oid 13 current snap is 0 2020-06-12T02:38:43.057 INFO:tasks.rados.rados.0.smithi107.stdout:8754: done (0 left) 2020-06-12T02:38:43.057 INFO:tasks.rados.rados.0.smithi107.stdout:8755: delete oid 18 current snap is 0 2020-06-12T02:38:43.057 INFO:tasks.rados.rados.0.smithi107.stdout:8755: done (0 left) 2020-06-12T02:38:43.058 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:43.058 INFO:tasks.rados.rados.0.smithi107.stdout:8756: seq_num 8755 ranges {0=237568} 2020-06-12T02:38:43.061 INFO:tasks.rados.rados.0.smithi107.stdout:8756: writing smithi10729407-1 from 0 to 237568 tid 1 2020-06-12T02:38:43.061 INFO:tasks.rados.rados.0.smithi107.stdout:8757: delete oid 9 current snap is 0 2020-06-12T02:38:43.061 INFO:tasks.rados.rados.0.smithi107.stdout:8757: done (1 left) 2020-06-12T02:38:43.061 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:43.062 INFO:tasks.rados.rados.0.smithi107.stdout:8758: seq_num 8757 ranges {2285568=270336} 2020-06-12T02:38:43.076 INFO:tasks.rados.rados.0.smithi107.stdout:8758: writing smithi10729407-2 from 2285568 to 2555904 tid 1 2020-06-12T02:38:43.077 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:43.077 INFO:tasks.rados.rados.0.smithi107.stdout:8759: seq_num 8758 ranges {0=286720} 2020-06-12T02:38:43.078 INFO:tasks.rados.rados.0.smithi107.stdout:8759: writing smithi10729407-12 from 0 to 286720 tid 1 2020-06-12T02:38:43.079 INFO:tasks.rados.rados.0.smithi107.stdout:8760: delete oid 5 current snap is 0 2020-06-12T02:38:43.079 INFO:tasks.rados.rados.0.smithi107.stdout:8756: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:43.079 INFO:tasks.rados.rados.0.smithi107.stdout:8756: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:43.079 INFO:tasks.rados.rados.0.smithi107.stdout:8756: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:43.079 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1597 (ObjNum 8755 snap 0 seq_num 8755) dirty exists 2020-06-12T02:38:43.080 INFO:tasks.rados.rados.0.smithi107.stdout:8756: left oid 1 (ObjNum 8755 snap 0 seq_num 8755) 2020-06-12T02:38:43.081 INFO:tasks.rados.rados.0.smithi107.stdout:8758: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:43.081 INFO:tasks.rados.rados.0.smithi107.stdout:8758: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:43.082 INFO:tasks.rados.rados.0.smithi107.stdout:8756: done (3 left) 2020-06-12T02:38:43.082 INFO:tasks.rados.rados.0.smithi107.stdout:8760: done (2 left) 2020-06-12T02:38:43.082 INFO:tasks.rados.rados.0.smithi107.stdout:8761: delete oid 1 current snap is 0 2020-06-12T02:38:43.083 INFO:tasks.rados.rados.0.smithi107.stdout:8758: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:43.083 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 815 (ObjNum 8757 snap 0 seq_num 8757) dirty exists 2020-06-12T02:38:43.084 INFO:tasks.rados.rados.0.smithi107.stdout:8758: left oid 2 (ObjNum 8757 snap 0 seq_num 8757) 2020-06-12T02:38:43.085 INFO:tasks.rados.rados.0.smithi107.stdout:8759: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:43.085 INFO:tasks.rados.rados.0.smithi107.stdout:8758: done (2 left) 2020-06-12T02:38:43.085 INFO:tasks.rados.rados.0.smithi107.stdout:8761: done (1 left) 2020-06-12T02:38:43.085 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:43.086 INFO:tasks.rados.rados.0.smithi107.stdout:8762: seq_num 8761 ranges {0=335872} 2020-06-12T02:38:43.087 INFO:tasks.rados.rados.0.smithi107.stdout:8762: writing smithi10729407-19 from 0 to 335872 tid 1 2020-06-12T02:38:43.088 INFO:tasks.rados.rados.0.smithi107.stdout:8763: delete oid 18 current snap is 0 2020-06-12T02:38:43.089 INFO:tasks.rados.rados.0.smithi107.stdout:8763: done (2 left) 2020-06-12T02:38:43.090 INFO:tasks.rados.rados.0.smithi107.stdout:8764: delete oid 17 current snap is 0 2020-06-12T02:38:43.090 INFO:tasks.rados.rados.0.smithi107.stdout:8759: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:43.093 INFO:tasks.rados.rados.0.smithi107.stdout:8762: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:43.093 INFO:tasks.rados.rados.0.smithi107.stdout:8764: done (2 left) 2020-06-12T02:38:43.093 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:43.094 INFO:tasks.rados.rados.0.smithi107.stdout:8765: seq_num 8764 ranges {0=385024} 2020-06-12T02:38:43.096 INFO:tasks.rados.rados.0.smithi107.stdout:8765: writing smithi10729407-16 from 0 to 385024 tid 1 2020-06-12T02:38:43.096 INFO:tasks.rados.rados.0.smithi107.stdout:8766: delete oid 14 current snap is 0 2020-06-12T02:38:43.096 INFO:tasks.rados.rados.0.smithi107.stdout:8759: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:43.096 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1613 (ObjNum 8758 snap 0 seq_num 8758) dirty exists 2020-06-12T02:38:43.097 INFO:tasks.rados.rados.0.smithi107.stdout:8759: left oid 12 (ObjNum 8758 snap 0 seq_num 8758) 2020-06-12T02:38:43.097 INFO:tasks.rados.rados.0.smithi107.stdout:8762: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:43.102 INFO:tasks.rados.rados.0.smithi107.stdout:8762: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:43.102 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1615 (ObjNum 8761 snap 0 seq_num 8761) dirty exists 2020-06-12T02:38:43.103 INFO:tasks.rados.rados.0.smithi107.stdout:8762: left oid 19 (ObjNum 8761 snap 0 seq_num 8761) 2020-06-12T02:38:43.103 INFO:tasks.rados.rados.0.smithi107.stdout:8759: done (3 left) 2020-06-12T02:38:43.103 INFO:tasks.rados.rados.0.smithi107.stdout:8762: done (2 left) 2020-06-12T02:38:43.103 INFO:tasks.rados.rados.0.smithi107.stdout:8766: done (1 left) 2020-06-12T02:38:43.103 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:38:43.104 INFO:tasks.rados.rados.0.smithi107.stdout:8767: seq_num 8766 ranges {1990656=417792} 2020-06-12T02:38:43.106 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:43.103+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:43.114 INFO:tasks.rados.rados.0.smithi107.stdout:8767: writing smithi10729407-10 from 1990656 to 2408448 tid 1 2020-06-12T02:38:43.114 INFO:tasks.rados.rados.0.smithi107.stdout:8768: delete oid 4 current snap is 0 2020-06-12T02:38:43.114 INFO:tasks.rados.rados.0.smithi107.stdout:8765: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:43.114 INFO:tasks.rados.rados.0.smithi107.stdout:8765: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:43.115 INFO:tasks.rados.rados.0.smithi107.stdout:8765: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:43.115 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1577 (ObjNum 8764 snap 0 seq_num 8764) dirty exists 2020-06-12T02:38:43.115 INFO:tasks.rados.rados.0.smithi107.stdout:8765: left oid 16 (ObjNum 8764 snap 0 seq_num 8764) 2020-06-12T02:38:43.119 INFO:tasks.rados.rados.0.smithi107.stdout:8765: done (2 left) 2020-06-12T02:38:43.119 INFO:tasks.rados.rados.0.smithi107.stdout:8768: done (1 left) 2020-06-12T02:38:43.119 INFO:tasks.rados.rados.0.smithi107.stdout:8769: delete oid 12 current snap is 0 2020-06-12T02:38:43.120 INFO:tasks.rados.rados.0.smithi107.stdout:8767: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:43.120 INFO:tasks.rados.rados.0.smithi107.stdout:8767: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:43.121 INFO:tasks.rados.rados.0.smithi107.stdout:8769: done (1 left) 2020-06-12T02:38:43.122 INFO:tasks.rados.rados.0.smithi107.stdout:8770: delete oid 14 current snap is 0 2020-06-12T02:38:43.122 INFO:tasks.rados.rados.0.smithi107.stdout:8767: finishing write tid 4 to smithi10729407-10 2020-06-12T02:38:43.122 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 634 (ObjNum 8766 snap 0 seq_num 8766) dirty exists 2020-06-12T02:38:43.123 INFO:tasks.rados.rados.0.smithi107.stdout:8767: left oid 10 (ObjNum 8766 snap 0 seq_num 8766) 2020-06-12T02:38:43.124 INFO:tasks.rados.rados.0.smithi107.stdout:8767: done (1 left) 2020-06-12T02:38:43.124 INFO:tasks.rados.rados.0.smithi107.stdout:8770: done (0 left) 2020-06-12T02:38:43.125 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:43.125 INFO:tasks.rados.rados.0.smithi107.stdout:8771: seq_num 8770 ranges {335872=483328} 2020-06-12T02:38:43.129 INFO:tasks.rados.rados.0.smithi107.stdout:8771: writing smithi10729407-19 from 335872 to 819200 tid 1 2020-06-12T02:38:43.129 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:43.129 INFO:tasks.rados.rados.0.smithi107.stdout:8772: seq_num 8771 ranges {0=499712} 2020-06-12T02:38:43.131 INFO:tasks.rados.rados.0.smithi107.stdout:8772: writing smithi10729407-5 from 0 to 499712 tid 1 2020-06-12T02:38:43.132 INFO:tasks.rados.rados.0.smithi107.stdout:8773: delete oid 12 current snap is 0 2020-06-12T02:38:43.134 INFO:tasks.rados.rados.0.smithi107.stdout:8771: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:43.136 INFO:tasks.rados.rados.0.smithi107.stdout:8771: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:43.138 INFO:tasks.rados.rados.0.smithi107.stdout:8773: done (2 left) 2020-06-12T02:38:43.138 INFO:tasks.rados.rados.0.smithi107.stdout:8774: delete oid 14 current snap is 0 2020-06-12T02:38:43.139 INFO:tasks.rados.rados.0.smithi107.stdout:8771: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:43.139 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1618 (ObjNum 8770 snap 0 seq_num 8770) dirty exists 2020-06-12T02:38:43.139 INFO:tasks.rados.rados.0.smithi107.stdout:8771: left oid 19 (ObjNum 8770 snap 0 seq_num 8770) 2020-06-12T02:38:43.140 INFO:tasks.rados.rados.0.smithi107.stdout:8772: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:43.140 INFO:tasks.rados.rados.0.smithi107.stdout:8772: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:43.141 INFO:tasks.rados.rados.0.smithi107.stdout:8771: done (2 left) 2020-06-12T02:38:43.141 INFO:tasks.rados.rados.0.smithi107.stdout:8774: done (1 left) 2020-06-12T02:38:43.141 INFO:tasks.rados.rados.0.smithi107.stdout:8775: delete oid 19 current snap is 0 2020-06-12T02:38:43.142 INFO:tasks.rados.rados.0.smithi107.stdout:8772: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:43.142 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 862 (ObjNum 8771 snap 0 seq_num 8771) dirty exists 2020-06-12T02:38:43.142 INFO:tasks.rados.rados.0.smithi107.stdout:8772: left oid 5 (ObjNum 8771 snap 0 seq_num 8771) 2020-06-12T02:38:43.143 INFO:tasks.rados.rados.0.smithi107.stdout:8772: done (1 left) 2020-06-12T02:38:43.144 INFO:tasks.rados.rados.0.smithi107.stdout:8775: done (0 left) 2020-06-12T02:38:43.144 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:38:43.144 INFO:tasks.rados.rados.0.smithi107.stdout:8776: seq_num 8775 ranges {0=573440} 2020-06-12T02:38:43.147 INFO:tasks.rados.rados.0.smithi107.stdout:8776: writing smithi10729407-13 from 0 to 573440 tid 1 2020-06-12T02:38:43.147 INFO:tasks.rados.rados.0.smithi107.stdout:8777: delete oid 4 current snap is 0 2020-06-12T02:38:43.149 INFO:tasks.rados.rados.0.smithi107.stdout:8777: done (1 left) 2020-06-12T02:38:43.150 INFO:tasks.rados.rados.0.smithi107.stdout:8778: delete oid 9 current snap is 0 2020-06-12T02:38:43.153 INFO:tasks.rados.rados.0.smithi107.stdout:8776: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:43.154 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:38:43.154 INFO:tasks.rados.rados.0.smithi107.stdout:8778: done (1 left) 2020-06-12T02:38:43.154 INFO:tasks.rados.rados.0.smithi107.stdout:8779: delete oid 2 current snap is 0 2020-06-12T02:38:43.156 INFO:tasks.rados.rados.0.smithi107.stdout:8779: done (1 left) 2020-06-12T02:38:43.157 INFO:tasks.rados.rados.0.smithi107.stdout:8780: delete oid 17 current snap is 0 2020-06-12T02:38:43.157 INFO:tasks.rados.rados.0.smithi107.stdout:8776: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:43.160 INFO:tasks.rados.rados.0.smithi107.stdout:8780: done (1 left) 2020-06-12T02:38:43.160 INFO:tasks.rados.rados.0.smithi107.stdout:8781: delete oid 8 current snap is 0 2020-06-12T02:38:43.161 INFO:tasks.rados.rados.0.smithi107.stdout:8776: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:43.161 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1581 (ObjNum 8775 snap 0 seq_num 8775) dirty exists 2020-06-12T02:38:43.161 INFO:tasks.rados.rados.0.smithi107.stdout:8776: left oid 13 (ObjNum 8775 snap 0 seq_num 8775) 2020-06-12T02:38:43.163 INFO:tasks.rados.rados.0.smithi107.stdout:8776: done (1 left) 2020-06-12T02:38:43.163 INFO:tasks.rados.rados.0.smithi107.stdout:8781: done (0 left) 2020-06-12T02:38:43.163 INFO:tasks.rados.rados.0.smithi107.stdout:8782: delete oid 18 current snap is 0 2020-06-12T02:38:43.166 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:43.166 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2020-06-12T02:38:43.166 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2020-06-12T02:38:43.166 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2020-06-12T02:38:43.167 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.167 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:2782 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~696320] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:43.167 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:36.808170+0000", 2020-06-12T02:38:43.167 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 66.353661625000001, 2020-06-12T02:38:43.167 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016972839999999999, 2020-06-12T02:38:43.167 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:43.168 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:43.168 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:43.168 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:43.168 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:43.168 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2782 2020-06-12T02:38:43.169 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.169 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:43.169 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.169 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:43.169 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.808170+0000", 2020-06-12T02:38:43.170 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:43.170 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.170 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.170 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:43.170 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.808170+0000", 2020-06-12T02:38:43.170 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000261778 2020-06-12T02:38:43.171 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.171 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.171 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:43.171 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.808432+0000", 2020-06-12T02:38:43.171 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0920000000000003e-06 2020-06-12T02:38:43.172 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.172 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.172 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:43.172 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.808438+0000", 2020-06-12T02:38:43.172 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2539e-05 2020-06-12T02:38:43.172 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.173 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.173 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:43.173 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.808450+0000", 2020-06-12T02:38:43.173 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1859999999999997e-06 2020-06-12T02:38:43.173 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.174 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.174 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:43.174 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.808457+0000", 2020-06-12T02:38:43.174 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0988000000000001e-05 2020-06-12T02:38:43.174 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.174 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.175 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:43.176 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.808498+0000", 2020-06-12T02:38:43.176 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011723 2020-06-12T02:38:43.176 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.177 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.177 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:43.177 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.808615+0000", 2020-06-12T02:38:43.178 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0015648109999999999 2020-06-12T02:38:43.178 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.178 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.178 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:43.178 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.810180+0000", 2020-06-12T02:38:43.179 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1718999999999994e-05 2020-06-12T02:38:43.179 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.179 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.179 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:43.179 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.810271+0000", 2020-06-12T02:38:43.180 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00082076299999999996 2020-06-12T02:38:43.180 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.180 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.180 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:43.180 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.811092+0000", 2020-06-12T02:38:43.180 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01210689 2020-06-12T02:38:43.181 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.181 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.181 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:43.181 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.823199+0000", 2020-06-12T02:38:43.182 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017554300000000001 2020-06-12T02:38:43.182 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.182 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.182 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:43.182 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.824954+0000", 2020-06-12T02:38:43.183 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000188414 2020-06-12T02:38:43.183 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.183 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.183 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:43.183 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.825143+0000", 2020-06-12T02:38:43.183 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016972839999999999 2020-06-12T02:38:43.184 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:43.184 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:43.184 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:43.184 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.184 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.185 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:2785 3.0s0 3:0dddb690:::smithi10729407-10:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:43.185 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:36.813804+0000", 2020-06-12T02:38:43.185 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 66.348027354999999, 2020-06-12T02:38:43.185 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014100781999999999, 2020-06-12T02:38:43.185 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:43.186 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:43.186 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:43.186 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:43.186 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:43.186 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2785 2020-06-12T02:38:43.186 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.187 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:43.187 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.187 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:43.187 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813804+0000", 2020-06-12T02:38:43.187 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:43.188 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.188 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.188 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:43.188 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813804+0000", 2020-06-12T02:38:43.188 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1301999999999999e-05 2020-06-12T02:38:43.188 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.189 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.189 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:43.189 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813846+0000", 2020-06-12T02:38:43.189 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4530000000000004e-06 2020-06-12T02:38:43.189 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.190 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.190 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:43.190 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813850+0000", 2020-06-12T02:38:43.190 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7940000000000003e-06 2020-06-12T02:38:43.190 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.191 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.192 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:43.192 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813858+0000", 2020-06-12T02:38:43.192 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9450000000000002e-06 2020-06-12T02:38:43.192 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.193 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.193 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:43.193 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813861+0000", 2020-06-12T02:38:43.193 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0084749969999999997 2020-06-12T02:38:43.193 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.194 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.194 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:43.194 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.822336+0000", 2020-06-12T02:38:43.194 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013055400000000001 2020-06-12T02:38:43.194 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.195 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.195 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:43.195 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.822466+0000", 2020-06-12T02:38:43.195 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047301400000000001 2020-06-12T02:38:43.195 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.195 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.196 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:43.196 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.822939+0000", 2020-06-12T02:38:43.196 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086569499999999998 2020-06-12T02:38:43.196 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.196 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.197 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:43.197 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.823805+0000", 2020-06-12T02:38:43.197 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0039911460000000001 2020-06-12T02:38:43.197 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.197 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.197 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:43.198 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.827796+0000", 2020-06-12T02:38:43.198 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010888200000000001 2020-06-12T02:38:43.198 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.198 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.198 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:43.199 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.827905+0000", 2020-06-12T02:38:43.199 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014100781999999999 2020-06-12T02:38:43.199 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:43.199 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:43.199 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:43.199 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.200 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.200 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:2786 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:38:43.200 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:36.813885+0000", 2020-06-12T02:38:43.200 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 66.347946538000002, 2020-06-12T02:38:43.200 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015757724000000001, 2020-06-12T02:38:43.201 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:43.201 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:43.201 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:43.201 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:43.201 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:43.202 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2786 2020-06-12T02:38:43.202 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.202 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:43.202 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.202 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:43.202 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813885+0000", 2020-06-12T02:38:43.203 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:43.203 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.203 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.203 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:43.203 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813885+0000", 2020-06-12T02:38:43.204 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2388999999999999e-05 2020-06-12T02:38:43.204 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.204 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.204 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:43.204 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813907+0000", 2020-06-12T02:38:43.205 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9660000000000001e-06 2020-06-12T02:38:43.205 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.205 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.205 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:43.205 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813910+0000", 2020-06-12T02:38:43.205 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7649999999999993e-06 2020-06-12T02:38:43.206 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.206 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.206 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:43.206 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813918+0000", 2020-06-12T02:38:43.206 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0870000000000001e-06 2020-06-12T02:38:43.207 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.207 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.207 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:43.207 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813920+0000", 2020-06-12T02:38:43.207 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093684819999999992 2020-06-12T02:38:43.207 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.208 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.208 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:43.208 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.823289+0000", 2020-06-12T02:38:43.208 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103643 2020-06-12T02:38:43.208 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.209 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.209 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:43.209 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.823392+0000", 2020-06-12T02:38:43.209 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0050514449999999999 2020-06-12T02:38:43.209 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.210 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.210 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:43.213 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.828444+0000", 2020-06-12T02:38:43.214 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000102824 2020-06-12T02:38:43.214 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.214 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.214 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:43.214 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.828547+0000", 2020-06-12T02:38:43.215 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0010961230000000001 2020-06-12T02:38:43.215 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.215 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.215 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:43.215 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.829643+0000", 2020-06-12T02:38:43.216 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015757724000000001 2020-06-12T02:38:43.216 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:43.216 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:43.216 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:43.216 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.216 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.217 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:3330 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~630784] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:38:43.217 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:38.603794+0000", 2020-06-12T02:38:43.217 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 64.558037846999994, 2020-06-12T02:38:43.217 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016093917999999999, 2020-06-12T02:38:43.217 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:43.218 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:43.218 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:43.218 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:43.218 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:43.218 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 3330 2020-06-12T02:38:43.219 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.219 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:43.219 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.219 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:43.219 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.603794+0000", 2020-06-12T02:38:43.219 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:43.220 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.220 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.220 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:43.220 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.603794+0000", 2020-06-12T02:38:43.220 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00042303700000000002 2020-06-12T02:38:43.221 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.221 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.221 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:43.221 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.604217+0000", 2020-06-12T02:38:43.221 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9350000000000006e-06 2020-06-12T02:38:43.221 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.222 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.222 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:43.222 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.604227+0000", 2020-06-12T02:38:43.222 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9343000000000001e-05 2020-06-12T02:38:43.222 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.223 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.223 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:43.223 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.604246+0000", 2020-06-12T02:38:43.223 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2369999999999998e-06 2020-06-12T02:38:43.223 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.223 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.224 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:43.224 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.604255+0000", 2020-06-12T02:38:43.224 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1530000000000003e-05 2020-06-12T02:38:43.224 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.224 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.225 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:43.225 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.604307+0000", 2020-06-12T02:38:43.225 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103756 2020-06-12T02:38:43.225 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.225 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.225 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:43.226 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.604411+0000", 2020-06-12T02:38:43.226 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073281400000000003 2020-06-12T02:38:43.226 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.226 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.226 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:43.227 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.605143+0000", 2020-06-12T02:38:43.227 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071483730000000004 2020-06-12T02:38:43.227 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.227 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.227 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:43.227 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.612292+0000", 2020-06-12T02:38:43.228 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0075185879999999997 2020-06-12T02:38:43.228 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.228 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.228 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:43.228 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.619810+0000", 2020-06-12T02:38:43.229 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7305000000000006e-05 2020-06-12T02:38:43.229 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.229 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.229 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:43.229 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.619888+0000", 2020-06-12T02:38:43.229 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016093917999999999 2020-06-12T02:38:43.230 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:43.230 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:43.230 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:43.230 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.230 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.231 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:3335 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:38:43.231 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:38.616040+0000", 2020-06-12T02:38:43.231 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 64.545791108000003, 2020-06-12T02:38:43.231 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015075565000000001, 2020-06-12T02:38:43.231 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:43.231 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:43.232 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:43.232 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:43.232 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:43.232 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 3335 2020-06-12T02:38:43.232 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.233 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:43.233 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.233 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:43.233 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.616040+0000", 2020-06-12T02:38:43.233 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:43.233 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.234 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.234 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:43.234 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.616040+0000", 2020-06-12T02:38:43.234 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1950000000000002e-05 2020-06-12T02:38:43.234 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.235 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.235 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:43.235 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.616062+0000", 2020-06-12T02:38:43.235 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.1370000000000002e-06 2020-06-12T02:38:43.235 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.235 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.236 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:43.236 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.616066+0000", 2020-06-12T02:38:43.236 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5019999999999998e-06 2020-06-12T02:38:43.236 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.236 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.237 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:43.237 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.616072+0000", 2020-06-12T02:38:43.237 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3470000000000001e-06 2020-06-12T02:38:43.237 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.237 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.237 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:43.238 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.616074+0000", 2020-06-12T02:38:43.238 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0067702309999999998 2020-06-12T02:38:43.238 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.238 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.238 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:43.239 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.622845+0000", 2020-06-12T02:38:43.239 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7738000000000005e-05 2020-06-12T02:38:43.239 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.239 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.239 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:43.239 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.622942+0000", 2020-06-12T02:38:43.240 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0050881010000000003 2020-06-12T02:38:43.240 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.240 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.240 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:43.240 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.628031+0000", 2020-06-12T02:38:43.241 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000108171 2020-06-12T02:38:43.241 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.241 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.241 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:43.241 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.628139+0000", 2020-06-12T02:38:43.241 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0029773880000000001 2020-06-12T02:38:43.242 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.242 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.242 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:43.242 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.631116+0000", 2020-06-12T02:38:43.242 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015075565000000001 2020-06-12T02:38:43.243 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:43.243 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:43.243 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:43.243 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.243 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.243 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:4969 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~729088,stat] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-06-12T02:38:43.244 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:43.713430+0000", 2020-06-12T02:38:43.244 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 59.448401644, 2020-06-12T02:38:43.244 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.18980771900000001, 2020-06-12T02:38:43.244 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:43.244 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:43.245 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:43.245 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:43.245 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:43.245 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 4969 2020-06-12T02:38:43.245 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.245 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:43.246 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.246 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:43.246 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.713430+0000", 2020-06-12T02:38:43.246 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:43.246 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.247 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.247 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:43.247 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.713430+0000", 2020-06-12T02:38:43.247 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018522800000000001 2020-06-12T02:38:43.247 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.247 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.248 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:43.248 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.713615+0000", 2020-06-12T02:38:43.248 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3590000000000002e-06 2020-06-12T02:38:43.248 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.248 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.249 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:43.249 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.713623+0000", 2020-06-12T02:38:43.249 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0939e-05 2020-06-12T02:38:43.249 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.249 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.250 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:43.250 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.713633+0000", 2020-06-12T02:38:43.250 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3229999999999999e-06 2020-06-12T02:38:43.250 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.250 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.250 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:43.251 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.713638+0000", 2020-06-12T02:38:43.251 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7401000000000002e-05 2020-06-12T02:38:43.251 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.251 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.251 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:43.252 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.713685+0000", 2020-06-12T02:38:43.252 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011488 2020-06-12T02:38:43.252 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.252 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.252 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:43.252 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.713800+0000", 2020-06-12T02:38:43.253 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000662541 2020-06-12T02:38:43.253 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.253 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.253 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:43.253 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.714463+0000", 2020-06-12T02:38:43.254 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043485090000000004 2020-06-12T02:38:43.254 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.254 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.254 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:43.254 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.718811+0000", 2020-06-12T02:38:43.254 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.18428694000000001 2020-06-12T02:38:43.255 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.255 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.255 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:43.255 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.903098+0000", 2020-06-12T02:38:43.255 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013959899999999999 2020-06-12T02:38:43.255 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.256 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.258 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:43.259 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.903238+0000", 2020-06-12T02:38:43.259 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.18980771900000001 2020-06-12T02:38:43.259 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:43.259 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:43.260 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:43.260 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.260 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.260 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:4970 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~401408] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-06-12T02:38:43.260 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:43.716006+0000", 2020-06-12T02:38:43.260 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 59.445825530999997, 2020-06-12T02:38:43.261 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.19216903699999999, 2020-06-12T02:38:43.261 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:43.261 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:43.261 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:43.261 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:43.262 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:43.262 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 4970 2020-06-12T02:38:43.262 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.262 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:43.262 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.262 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:43.263 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716006+0000", 2020-06-12T02:38:43.263 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:43.263 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.263 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.263 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:43.264 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716006+0000", 2020-06-12T02:38:43.264 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000115757 2020-06-12T02:38:43.264 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.264 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.264 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:43.264 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716122+0000", 2020-06-12T02:38:43.265 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1989999999999997e-06 2020-06-12T02:38:43.265 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.265 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.265 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:43.265 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716127+0000", 2020-06-12T02:38:43.266 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2121e-05 2020-06-12T02:38:43.266 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.266 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.266 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:43.266 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716139+0000", 2020-06-12T02:38:43.266 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.113e-06 2020-06-12T02:38:43.267 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.267 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.267 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:43.267 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716144+0000", 2020-06-12T02:38:43.267 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014413900000000001 2020-06-12T02:38:43.268 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.268 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.268 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:43.268 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.717586+0000", 2020-06-12T02:38:43.268 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0257000000000002e-05 2020-06-12T02:38:43.268 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.269 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.269 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:43.269 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.717656+0000", 2020-06-12T02:38:43.269 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.18596705899999999 2020-06-12T02:38:43.269 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.270 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.270 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:43.270 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.903623+0000", 2020-06-12T02:38:43.270 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7858999999999999e-05 2020-06-12T02:38:43.270 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.271 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.271 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:43.271 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.903721+0000", 2020-06-12T02:38:43.271 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000683536 2020-06-12T02:38:43.271 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.272 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.272 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:43.272 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.904404+0000", 2020-06-12T02:38:43.272 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.003364438 2020-06-12T02:38:43.272 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.273 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.273 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:43.273 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.907769+0000", 2020-06-12T02:38:43.273 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00033094500000000002 2020-06-12T02:38:43.273 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.274 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.274 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:43.274 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.908100+0000", 2020-06-12T02:38:43.274 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5363000000000003e-05 2020-06-12T02:38:43.274 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.275 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.275 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:43.275 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.908175+0000", 2020-06-12T02:38:43.276 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.19216903699999999 2020-06-12T02:38:43.276 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:43.276 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:43.276 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:43.276 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.276 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.277 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:4971 3.0s0 3:0dddb690:::smithi10729407-10:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-06-12T02:38:43.277 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:43.716190+0000", 2020-06-12T02:38:43.277 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 59.445641672999997, 2020-06-12T02:38:43.277 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.19259003999999999, 2020-06-12T02:38:43.277 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:43.278 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:43.278 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:43.278 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:43.278 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:43.278 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 4971 2020-06-12T02:38:43.278 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.279 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:43.279 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.279 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:43.279 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716190+0000", 2020-06-12T02:38:43.279 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:43.280 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.280 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.280 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:43.280 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716190+0000", 2020-06-12T02:38:43.280 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3192999999999998e-05 2020-06-12T02:38:43.281 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.281 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.281 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:43.281 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716233+0000", 2020-06-12T02:38:43.281 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6279999999999998e-06 2020-06-12T02:38:43.282 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.282 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.282 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:43.282 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716237+0000", 2020-06-12T02:38:43.282 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6619999999999999e-06 2020-06-12T02:38:43.283 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.283 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.283 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:43.283 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716243+0000", 2020-06-12T02:38:43.283 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.495e-06 2020-06-12T02:38:43.284 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.284 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.284 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:43.284 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716247+0000", 2020-06-12T02:38:43.284 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014483740000000001 2020-06-12T02:38:43.284 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.285 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.285 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:43.285 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.717695+0000", 2020-06-12T02:38:43.285 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000108108 2020-06-12T02:38:43.285 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.286 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.286 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:43.286 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.717803+0000", 2020-06-12T02:38:43.286 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.18873436800000001 2020-06-12T02:38:43.286 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.287 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.287 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:43.287 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.906538+0000", 2020-06-12T02:38:43.287 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9384999999999998e-05 2020-06-12T02:38:43.287 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.287 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.288 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:43.288 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.906637+0000", 2020-06-12T02:38:43.288 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00042339299999999999 2020-06-12T02:38:43.288 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.288 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.289 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:43.289 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.907061+0000", 2020-06-12T02:38:43.289 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00071097900000000004 2020-06-12T02:38:43.289 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.289 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.289 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:43.290 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.907772+0000", 2020-06-12T02:38:43.290 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00089641299999999998 2020-06-12T02:38:43.290 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.290 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.290 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:43.291 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.908668+0000", 2020-06-12T02:38:43.291 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112042 2020-06-12T02:38:43.291 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.291 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.291 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:43.291 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.908780+0000", 2020-06-12T02:38:43.292 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.19259003999999999 2020-06-12T02:38:43.292 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:43.292 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:43.292 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:43.292 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.293 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.293 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:4972 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-06-12T02:38:43.293 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:43.716274+0000", 2020-06-12T02:38:43.293 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 59.445557903999998, 2020-06-12T02:38:43.293 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.193959876, 2020-06-12T02:38:43.293 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:43.294 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:43.294 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:43.294 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:43.294 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:43.294 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 4972 2020-06-12T02:38:43.295 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.295 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:43.295 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.295 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:43.295 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716274+0000", 2020-06-12T02:38:43.295 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:43.296 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.296 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.296 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:43.296 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716274+0000", 2020-06-12T02:38:43.296 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6852999999999999e-05 2020-06-12T02:38:43.297 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.297 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.297 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:43.297 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716301+0000", 2020-06-12T02:38:43.297 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3069999999999998e-06 2020-06-12T02:38:43.297 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.298 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.298 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:43.298 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716304+0000", 2020-06-12T02:38:43.298 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6080000000000001e-06 2020-06-12T02:38:43.298 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.299 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.299 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:43.299 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716310+0000", 2020-06-12T02:38:43.299 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6110000000000001e-06 2020-06-12T02:38:43.299 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.299 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.300 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:43.300 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716313+0000", 2020-06-12T02:38:43.300 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001512918 2020-06-12T02:38:43.300 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.300 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.301 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:43.301 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.717826+0000", 2020-06-12T02:38:43.301 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3773000000000002e-05 2020-06-12T02:38:43.301 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.301 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.301 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:43.302 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.717880+0000", 2020-06-12T02:38:43.302 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.18955132599999999 2020-06-12T02:38:43.302 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.302 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.302 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:43.303 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.907431+0000", 2020-06-12T02:38:43.303 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101158 2020-06-12T02:38:43.303 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.303 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.303 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:43.303 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.907532+0000", 2020-06-12T02:38:43.304 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00160593 2020-06-12T02:38:43.304 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.304 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.304 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:43.304 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.909138+0000", 2020-06-12T02:38:43.305 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8590000000000006e-05 2020-06-12T02:38:43.305 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.305 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.305 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:43.305 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.909207+0000", 2020-06-12T02:38:43.305 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0010268020000000001 2020-06-12T02:38:43.306 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.306 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.306 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:43.306 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.910234+0000", 2020-06-12T02:38:43.306 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.193959876 2020-06-12T02:38:43.307 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:43.307 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:43.307 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:43.307 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.307 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.307 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:5006 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-06-12T02:38:43.308 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:43.975526+0000", 2020-06-12T02:38:43.308 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 59.186305853999997, 2020-06-12T02:38:43.308 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014759296, 2020-06-12T02:38:43.308 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:43.308 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:43.309 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:43.309 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:43.309 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:43.309 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 5006 2020-06-12T02:38:43.309 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.310 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:43.310 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.310 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:43.310 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.975526+0000", 2020-06-12T02:38:43.310 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:43.310 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.311 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.311 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:43.311 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.975526+0000", 2020-06-12T02:38:43.311 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5868999999999997e-05 2020-06-12T02:38:43.312 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.312 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.312 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:43.312 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.975572+0000", 2020-06-12T02:38:43.312 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1210000000000003e-06 2020-06-12T02:38:43.313 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.313 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.313 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:43.313 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.975578+0000", 2020-06-12T02:38:43.313 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2204999999999999e-05 2020-06-12T02:38:43.314 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.314 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.314 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:43.314 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.975590+0000", 2020-06-12T02:38:43.315 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1289999999999998e-06 2020-06-12T02:38:43.315 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.315 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.315 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:43.315 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.975594+0000", 2020-06-12T02:38:43.315 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.7705e-05 2020-06-12T02:38:43.316 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.316 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.316 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:43.316 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.975622+0000", 2020-06-12T02:38:43.316 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9941999999999997e-05 2020-06-12T02:38:43.317 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.317 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.317 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:43.317 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.975672+0000", 2020-06-12T02:38:43.317 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0057881060000000003 2020-06-12T02:38:43.317 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.318 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.318 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:43.318 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.981460+0000", 2020-06-12T02:38:43.318 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0473999999999997e-05 2020-06-12T02:38:43.318 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.319 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.319 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:43.319 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.981520+0000", 2020-06-12T02:38:43.319 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.007714294 2020-06-12T02:38:43.319 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.320 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.320 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:43.320 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.989235+0000", 2020-06-12T02:38:43.320 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010107600000000001 2020-06-12T02:38:43.320 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.320 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.321 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:43.321 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.989336+0000", 2020-06-12T02:38:43.321 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00094937500000000002 2020-06-12T02:38:43.321 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.321 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.322 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:43.322 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.990285+0000", 2020-06-12T02:38:43.322 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014759296 2020-06-12T02:38:43.322 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:43.322 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:43.322 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:43.323 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.323 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.323 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:7841 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~565248] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-06-12T02:38:43.323 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:51.996595+0000", 2020-06-12T02:38:43.323 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 51.165236364999998, 2020-06-12T02:38:43.324 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015060283000000001, 2020-06-12T02:38:43.324 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:43.324 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:43.324 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:43.324 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:43.324 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:43.325 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 7841 2020-06-12T02:38:43.325 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.325 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:43.325 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.325 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:43.326 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.996595+0000", 2020-06-12T02:38:43.326 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:43.326 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.326 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.326 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:43.326 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.996595+0000", 2020-06-12T02:38:43.327 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015120699999999999 2020-06-12T02:38:43.327 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.327 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.327 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:43.327 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.996746+0000", 2020-06-12T02:38:43.328 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4369999999999998e-06 2020-06-12T02:38:43.328 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.328 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.328 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:43.328 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.996752+0000", 2020-06-12T02:38:43.328 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0850000000000003e-06 2020-06-12T02:38:43.329 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.329 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.329 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:43.329 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.996761+0000", 2020-06-12T02:38:43.329 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2370000000000003e-06 2020-06-12T02:38:43.330 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.330 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.330 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:43.330 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.996765+0000", 2020-06-12T02:38:43.330 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5185999999999998e-05 2020-06-12T02:38:43.330 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.331 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.331 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:43.331 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.996800+0000", 2020-06-12T02:38:43.331 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7265000000000001e-05 2020-06-12T02:38:43.331 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.332 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.332 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:43.332 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.996868+0000", 2020-06-12T02:38:43.332 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0024047539999999998 2020-06-12T02:38:43.332 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.332 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.333 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:43.333 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.999272+0000", 2020-06-12T02:38:43.333 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3034999999999994e-05 2020-06-12T02:38:43.333 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.333 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.334 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:43.334 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.999355+0000", 2020-06-12T02:38:43.334 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049334499999999996 2020-06-12T02:38:43.334 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.334 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.334 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:43.335 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.999849+0000", 2020-06-12T02:38:43.335 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011708226 2020-06-12T02:38:43.335 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.335 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.335 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:43.336 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.011557+0000", 2020-06-12T02:38:43.336 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1449999999999997e-05 2020-06-12T02:38:43.336 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.336 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.336 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:43.336 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.011608+0000", 2020-06-12T02:38:43.337 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7055999999999998e-05 2020-06-12T02:38:43.337 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.337 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.337 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:43.337 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.011656+0000", 2020-06-12T02:38:43.338 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015060283000000001 2020-06-12T02:38:43.338 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:43.338 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:43.338 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:43.338 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.338 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.339 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:7845 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e33)", 2020-06-12T02:38:43.339 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:52.002017+0000", 2020-06-12T02:38:43.339 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 51.159814134000001, 2020-06-12T02:38:43.339 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017208558999999998, 2020-06-12T02:38:43.339 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:43.340 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:43.340 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:43.340 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:43.340 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:43.340 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 7845 2020-06-12T02:38:43.340 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.341 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:43.341 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.341 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:43.341 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.002017+0000", 2020-06-12T02:38:43.341 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:43.342 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.342 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.342 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:43.342 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.002017+0000", 2020-06-12T02:38:43.342 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2636000000000003e-05 2020-06-12T02:38:43.343 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.343 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.343 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:43.343 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.002060+0000", 2020-06-12T02:38:43.344 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5080000000000002e-06 2020-06-12T02:38:43.344 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.344 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.344 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:43.344 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.002067+0000", 2020-06-12T02:38:43.344 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3125000000000001e-05 2020-06-12T02:38:43.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.345 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.345 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:43.345 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.002080+0000", 2020-06-12T02:38:43.345 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5560000000000001e-06 2020-06-12T02:38:43.346 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.346 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.346 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:43.346 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.002084+0000", 2020-06-12T02:38:43.346 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0088536750000000001 2020-06-12T02:38:43.346 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.347 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.347 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:43.347 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.010938+0000", 2020-06-12T02:38:43.347 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3282999999999995e-05 2020-06-12T02:38:43.347 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.348 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.348 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:43.348 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.011021+0000", 2020-06-12T02:38:43.348 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0052873970000000001 2020-06-12T02:38:43.348 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.348 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.349 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:43.349 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.016309+0000", 2020-06-12T02:38:43.349 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010365399999999999 2020-06-12T02:38:43.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.349 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.350 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:43.350 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.016412+0000", 2020-06-12T02:38:43.350 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002813725 2020-06-12T02:38:43.350 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.350 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.350 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:43.351 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.019226+0000", 2020-06-12T02:38:43.351 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017208558999999998 2020-06-12T02:38:43.351 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:43.351 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:43.351 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:43.352 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.352 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.352 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8809 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~606208] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:43.352 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.707225+0000", 2020-06-12T02:38:43.352 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 31.454606696999999, 2020-06-12T02:38:43.352 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016771206, 2020-06-12T02:38:43.353 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:43.353 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:43.353 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:43.353 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:43.353 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:43.354 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8809 2020-06-12T02:38:43.354 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.354 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:43.354 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.354 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:43.354 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.707225+0000", 2020-06-12T02:38:43.355 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:43.355 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.355 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.355 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:43.355 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.707225+0000", 2020-06-12T02:38:43.356 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035342099999999998 2020-06-12T02:38:43.356 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.356 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.356 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:43.356 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.707578+0000", 2020-06-12T02:38:43.356 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2309999999999996e-06 2020-06-12T02:38:43.357 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.357 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.357 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:43.357 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.707587+0000", 2020-06-12T02:38:43.357 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4343e-05 2020-06-12T02:38:43.358 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.358 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.358 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:43.358 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.707601+0000", 2020-06-12T02:38:43.358 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8460000000000002e-06 2020-06-12T02:38:43.358 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.359 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.359 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:43.359 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.707608+0000", 2020-06-12T02:38:43.359 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7374999999999998e-05 2020-06-12T02:38:43.359 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.360 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.360 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:43.360 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.707675+0000", 2020-06-12T02:38:43.360 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111599 2020-06-12T02:38:43.360 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.360 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.361 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:43.361 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.707787+0000", 2020-06-12T02:38:43.361 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00079845100000000004 2020-06-12T02:38:43.361 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.361 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.362 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:43.362 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.708585+0000", 2020-06-12T02:38:43.362 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010899008999999999 2020-06-12T02:38:43.362 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.362 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.363 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:43.363 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.719484+0000", 2020-06-12T02:38:43.363 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306450000000004 2020-06-12T02:38:43.363 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.363 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.363 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:43.364 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.723915+0000", 2020-06-12T02:38:43.364 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1285999999999994e-05 2020-06-12T02:38:43.364 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.364 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.364 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:43.364 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.723996+0000", 2020-06-12T02:38:43.365 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016771206 2020-06-12T02:38:43.365 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:43.365 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:43.365 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:43.365 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.366 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.366 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.3s1 49/21 ECSubWrite(tid=4109, reqid=client.4512.0:8841, at_version=49'773, trim_to=0'0, roll_forward_to=49'771))", 2020-06-12T02:38:43.366 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.822295+0000", 2020-06-12T02:38:43.366 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 31.339536125999999, 2020-06-12T02:38:43.366 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01733517, 2020-06-12T02:38:43.367 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:43.367 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:43.367 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:43.367 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.367 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:43.367 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822295+0000", 2020-06-12T02:38:43.368 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:43.368 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.368 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.368 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:43.368 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822295+0000", 2020-06-12T02:38:43.368 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00089525499999999997 2020-06-12T02:38:43.369 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.369 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.369 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:43.369 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.823191+0000", 2020-06-12T02:38:43.369 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3315999999999998e-05 2020-06-12T02:38:43.370 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.370 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.370 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:43.370 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.823214+0000", 2020-06-12T02:38:43.370 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7096e-05 2020-06-12T02:38:43.371 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.371 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.371 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:43.371 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.823231+0000", 2020-06-12T02:38:43.372 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6890000000000004e-06 2020-06-12T02:38:43.372 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.372 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.372 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:43.372 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.823239+0000", 2020-06-12T02:38:43.372 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00061031599999999996 2020-06-12T02:38:43.373 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.373 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.373 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:43.373 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.823849+0000", 2020-06-12T02:38:43.373 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4205999999999999e-05 2020-06-12T02:38:43.374 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.374 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.374 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:43.374 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.823863+0000", 2020-06-12T02:38:43.374 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015707789999999999 2020-06-12T02:38:43.375 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.375 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.375 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:43.375 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.839571+0000", 2020-06-12T02:38:43.376 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9502000000000001e-05 2020-06-12T02:38:43.376 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.376 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.376 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:43.376 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.839631+0000", 2020-06-12T02:38:43.376 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01733517 2020-06-12T02:38:43.377 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:43.377 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:43.377 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:43.377 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.377 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.378 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8854 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~409600] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:43.378 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.854691+0000", 2020-06-12T02:38:43.378 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 31.307140243999999, 2020-06-12T02:38:43.378 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01967967, 2020-06-12T02:38:43.378 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:43.379 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:43.379 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:43.379 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:43.379 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:43.379 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8854 2020-06-12T02:38:43.379 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.380 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:43.380 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.380 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:43.380 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.854691+0000", 2020-06-12T02:38:43.380 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:43.380 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.381 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.381 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:43.381 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.854691+0000", 2020-06-12T02:38:43.381 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017767799999999999 2020-06-12T02:38:43.381 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.382 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.382 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:43.382 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.854869+0000", 2020-06-12T02:38:43.382 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8049999999999996e-06 2020-06-12T02:38:43.383 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.383 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.383 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:43.383 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.854878+0000", 2020-06-12T02:38:43.383 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5644000000000001e-05 2020-06-12T02:38:43.384 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.384 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.384 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:43.384 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.854893+0000", 2020-06-12T02:38:43.384 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.584e-06 2020-06-12T02:38:43.384 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.385 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.385 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:43.385 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.854900+0000", 2020-06-12T02:38:43.385 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002416769 2020-06-12T02:38:43.385 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.386 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.386 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:43.386 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.857317+0000", 2020-06-12T02:38:43.386 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000100385 2020-06-12T02:38:43.386 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.386 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.387 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:43.387 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.857417+0000", 2020-06-12T02:38:43.387 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0047283969999999996 2020-06-12T02:38:43.387 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.387 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.388 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:43.388 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.862146+0000", 2020-06-12T02:38:43.388 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1930999999999998e-05 2020-06-12T02:38:43.388 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.388 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.388 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:43.389 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.862238+0000", 2020-06-12T02:38:43.389 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00071479400000000004 2020-06-12T02:38:43.389 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.389 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.389 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:43.390 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.862952+0000", 2020-06-12T02:38:43.390 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011219316999999999 2020-06-12T02:38:43.390 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.390 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.390 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:43.390 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.874172+0000", 2020-06-12T02:38:43.391 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000104137 2020-06-12T02:38:43.391 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.391 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.391 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:43.391 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.874276+0000", 2020-06-12T02:38:43.392 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5229000000000006e-05 2020-06-12T02:38:43.392 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.392 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.392 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:43.392 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.874371+0000", 2020-06-12T02:38:43.392 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01967967 2020-06-12T02:38:43.393 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:43.393 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:43.393 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:43.393 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.393 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.394 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8855 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~516096] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:43.394 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.858551+0000", 2020-06-12T02:38:43.394 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 31.303280386000001, 2020-06-12T02:38:43.394 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020039164000000002, 2020-06-12T02:38:43.394 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:43.395 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:43.395 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:43.395 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:43.395 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:43.395 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8855 2020-06-12T02:38:43.395 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.396 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:43.396 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.396 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:43.396 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.858551+0000", 2020-06-12T02:38:43.396 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:43.397 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.397 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.397 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:43.397 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.858551+0000", 2020-06-12T02:38:43.397 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00020537100000000001 2020-06-12T02:38:43.397 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.398 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.398 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:43.398 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.858757+0000", 2020-06-12T02:38:43.398 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4870000000000002e-06 2020-06-12T02:38:43.398 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.399 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.399 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:43.399 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.858765+0000", 2020-06-12T02:38:43.399 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4358e-05 2020-06-12T02:38:43.399 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.399 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.400 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:43.400 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.858779+0000", 2020-06-12T02:38:43.400 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3879999999999997e-06 2020-06-12T02:38:43.400 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.400 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.400 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:43.401 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.858786+0000", 2020-06-12T02:38:43.401 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089100000000001 2020-06-12T02:38:43.401 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.401 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.401 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:43.402 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.859127+0000", 2020-06-12T02:38:43.402 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0877000000000005e-05 2020-06-12T02:38:43.402 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.402 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.402 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:43.403 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.859208+0000", 2020-06-12T02:38:43.403 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0059661269999999999 2020-06-12T02:38:43.403 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.403 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.403 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:43.403 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.865174+0000", 2020-06-12T02:38:43.404 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000102761 2020-06-12T02:38:43.404 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.404 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.404 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:43.404 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.865276+0000", 2020-06-12T02:38:43.404 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00079971700000000003 2020-06-12T02:38:43.405 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.405 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.405 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:43.405 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.866076+0000", 2020-06-12T02:38:43.405 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0083001380000000003 2020-06-12T02:38:43.406 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.406 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.406 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:43.406 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.874376+0000", 2020-06-12T02:38:43.406 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041190660000000002 2020-06-12T02:38:43.407 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.407 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.407 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:43.407 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.878495+0000", 2020-06-12T02:38:43.407 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4982999999999995e-05 2020-06-12T02:38:43.407 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.408 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.408 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:43.408 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.878590+0000", 2020-06-12T02:38:43.408 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020039164000000002 2020-06-12T02:38:43.408 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:43.409 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:43.409 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:43.409 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.409 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.409 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8856 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~442368] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:43.409 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.861853+0000", 2020-06-12T02:38:43.410 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 31.299978936999999, 2020-06-12T02:38:43.410 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022102294000000001, 2020-06-12T02:38:43.410 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:43.410 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:43.410 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:43.411 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:43.411 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:43.411 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8856 2020-06-12T02:38:43.411 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.411 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:43.411 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.412 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:43.412 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.861853+0000", 2020-06-12T02:38:43.412 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:43.412 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.413 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.413 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:43.413 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.861853+0000", 2020-06-12T02:38:43.413 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017153099999999999 2020-06-12T02:38:43.413 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.414 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.414 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:43.414 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.862024+0000", 2020-06-12T02:38:43.414 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0889999999999999e-06 2020-06-12T02:38:43.414 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.415 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.415 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:43.415 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.862029+0000", 2020-06-12T02:38:43.415 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3033999999999999e-05 2020-06-12T02:38:43.415 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.416 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.416 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:43.416 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.862042+0000", 2020-06-12T02:38:43.416 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1349999999999997e-06 2020-06-12T02:38:43.416 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.417 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.417 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:43.417 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.862048+0000", 2020-06-12T02:38:43.417 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0081330410000000006 2020-06-12T02:38:43.417 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.418 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.418 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:43.418 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.870181+0000", 2020-06-12T02:38:43.418 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012252500000000001 2020-06-12T02:38:43.418 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.418 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.419 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:43.419 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.870304+0000", 2020-06-12T02:38:43.419 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000740999 2020-06-12T02:38:43.419 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.419 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.419 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:43.420 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.871045+0000", 2020-06-12T02:38:43.420 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012634975 2020-06-12T02:38:43.420 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.420 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.420 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:43.421 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.883680+0000", 2020-06-12T02:38:43.421 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017667900000000001 2020-06-12T02:38:43.421 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.421 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.421 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:43.422 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.883857+0000", 2020-06-12T02:38:43.422 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8286000000000001e-05 2020-06-12T02:38:43.422 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.423 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.423 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:43.423 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.883955+0000", 2020-06-12T02:38:43.423 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022102294000000001 2020-06-12T02:38:43.423 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:43.424 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:43.424 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:43.424 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.424 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.424 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8857 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~237568] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:43.425 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.863733+0000", 2020-06-12T02:38:43.425 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 31.298098954, 2020-06-12T02:38:43.425 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020470258000000002, 2020-06-12T02:38:43.425 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:43.425 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:43.425 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:43.426 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:43.426 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:43.426 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8857 2020-06-12T02:38:43.426 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.426 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:43.427 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.427 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:43.427 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.863733+0000", 2020-06-12T02:38:43.427 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:43.427 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.427 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.428 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:43.428 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.863733+0000", 2020-06-12T02:38:43.428 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014776000000000001 2020-06-12T02:38:43.428 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.428 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.429 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:43.429 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.863880+0000", 2020-06-12T02:38:43.429 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5790000000000004e-06 2020-06-12T02:38:43.429 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.429 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.430 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:43.430 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.863889+0000", 2020-06-12T02:38:43.430 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6960999999999999e-05 2020-06-12T02:38:43.430 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.430 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.430 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:43.431 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.863906+0000", 2020-06-12T02:38:43.431 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5509999999999996e-06 2020-06-12T02:38:43.431 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.431 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.431 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:43.432 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.863913+0000", 2020-06-12T02:38:43.432 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010569258 2020-06-12T02:38:43.432 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.432 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.432 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:43.432 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.874482+0000", 2020-06-12T02:38:43.433 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011260399999999999 2020-06-12T02:38:43.433 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.433 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.433 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:43.433 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.874594+0000", 2020-06-12T02:38:43.434 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014913820000000001 2020-06-12T02:38:43.434 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.434 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.434 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:43.434 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.876086+0000", 2020-06-12T02:38:43.434 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0078744260000000003 2020-06-12T02:38:43.435 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.435 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.435 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:43.435 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.883960+0000", 2020-06-12T02:38:43.435 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015276999999999999 2020-06-12T02:38:43.436 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.436 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.436 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:43.436 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.884113+0000", 2020-06-12T02:38:43.436 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9967000000000001e-05 2020-06-12T02:38:43.436 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.437 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.437 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:43.437 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.884203+0000", 2020-06-12T02:38:43.437 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020470258000000002 2020-06-12T02:38:43.437 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:43.438 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:43.438 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:43.438 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.438 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.438 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8858 3.0s0 3:0dddb690:::smithi10729407-10:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:38:43.438 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.863968+0000", 2020-06-12T02:38:43.439 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 31.297863662000001, 2020-06-12T02:38:43.439 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022225373999999999, 2020-06-12T02:38:43.439 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:43.439 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:43.439 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:43.440 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:43.440 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:43.440 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8858 2020-06-12T02:38:43.440 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.440 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:43.440 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.441 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:43.441 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.863968+0000", 2020-06-12T02:38:43.441 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:43.441 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.441 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.442 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:43.442 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.863968+0000", 2020-06-12T02:38:43.442 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5352000000000006e-05 2020-06-12T02:38:43.442 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.442 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.442 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:43.443 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864043+0000", 2020-06-12T02:38:43.443 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9110000000000004e-06 2020-06-12T02:38:43.443 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.443 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.443 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:43.444 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864050+0000", 2020-06-12T02:38:43.444 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2505e-05 2020-06-12T02:38:43.444 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.444 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.444 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:43.444 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864063+0000", 2020-06-12T02:38:43.445 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0900000000000004e-06 2020-06-12T02:38:43.445 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.445 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.445 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:43.445 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864068+0000", 2020-06-12T02:38:43.446 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014563938 2020-06-12T02:38:43.446 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.446 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.446 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:43.446 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.878632+0000", 2020-06-12T02:38:43.446 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000106852 2020-06-12T02:38:43.447 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.447 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.447 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:43.447 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.878739+0000", 2020-06-12T02:38:43.447 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002931796 2020-06-12T02:38:43.447 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.448 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.448 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:43.448 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.881670+0000", 2020-06-12T02:38:43.448 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025365280000000001 2020-06-12T02:38:43.448 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.449 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.449 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:43.449 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.884207+0000", 2020-06-12T02:38:43.449 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001895224 2020-06-12T02:38:43.449 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.449 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.450 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:43.450 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.886102+0000", 2020-06-12T02:38:43.450 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1178e-05 2020-06-12T02:38:43.450 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.450 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.451 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:43.451 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.886193+0000", 2020-06-12T02:38:43.451 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022225373999999999 2020-06-12T02:38:43.451 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:43.451 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:43.451 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:43.452 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.452 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.452 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8859 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e49)", 2020-06-12T02:38:43.452 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.864115+0000", 2020-06-12T02:38:43.452 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 31.297716761, 2020-06-12T02:38:43.453 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.03325032, 2020-06-12T02:38:43.453 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:43.453 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:43.453 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:43.453 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:43.453 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:43.454 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8859 2020-06-12T02:38:43.454 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.454 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:43.454 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.454 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:43.454 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864115+0000", 2020-06-12T02:38:43.455 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:43.455 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.455 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.455 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:43.455 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864115+0000", 2020-06-12T02:38:43.456 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6795999999999997e-05 2020-06-12T02:38:43.456 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.456 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.456 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:43.456 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864162+0000", 2020-06-12T02:38:43.456 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5599999999999999e-06 2020-06-12T02:38:43.457 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.457 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.457 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:43.457 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864168+0000", 2020-06-12T02:38:43.457 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5658000000000002e-05 2020-06-12T02:38:43.458 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.458 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.458 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:43.458 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864184+0000", 2020-06-12T02:38:43.458 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8380000000000001e-06 2020-06-12T02:38:43.458 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.459 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.459 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:43.459 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864189+0000", 2020-06-12T02:38:43.459 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019365752 2020-06-12T02:38:43.459 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.460 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.460 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:43.460 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.883554+0000", 2020-06-12T02:38:43.462 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000118819 2020-06-12T02:38:43.462 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.462 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.463 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:43.463 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.883673+0000", 2020-06-12T02:38:43.463 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0089243440000000007 2020-06-12T02:38:43.463 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.463 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.464 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:43.464 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.892598+0000", 2020-06-12T02:38:43.464 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7429999999999994e-05 2020-06-12T02:38:43.464 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.464 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.464 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:43.465 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.892695+0000", 2020-06-12T02:38:43.465 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004670123 2020-06-12T02:38:43.465 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:43.465 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:43.465 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:43.466 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.897365+0000", 2020-06-12T02:38:43.466 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.03325032 2020-06-12T02:38:43.466 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:43.466 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:43.466 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:43.466 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:43.467 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:43.467 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:43.467 INFO:tasks.rados.rados.0.smithi107.stdout:8782: done (0 left) 2020-06-12T02:38:43.467 INFO:tasks.rados.rados.0.smithi107.stdout:8783: delete oid 12 current snap is 0 2020-06-12T02:38:43.484 INFO:tasks.rados.rados.0.smithi107.stdout:8783: done (0 left) 2020-06-12T02:38:43.485 INFO:tasks.rados.rados.0.smithi107.stdout:8784: delete oid 16 current snap is 0 2020-06-12T02:38:43.485 INFO:tasks.rados.rados.0.smithi107.stdout:8784: done (0 left) 2020-06-12T02:38:43.485 INFO:tasks.rados.rados.0.smithi107.stdout:8785: delete oid 3 current snap is 0 2020-06-12T02:38:43.485 INFO:tasks.rados.rados.0.smithi107.stdout:8785: done (0 left) 2020-06-12T02:38:43.486 INFO:tasks.rados.rados.0.smithi107.stdout:8786: delete oid 20 current snap is 0 2020-06-12T02:38:43.486 INFO:tasks.rados.rados.0.smithi107.stdout:8786: done (0 left) 2020-06-12T02:38:43.486 INFO:tasks.rados.rados.0.smithi107.stdout:8787: delete oid 16 current snap is 0 2020-06-12T02:38:43.486 INFO:tasks.rados.rados.0.smithi107.stdout:8787: done (0 left) 2020-06-12T02:38:43.486 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:43.487 INFO:tasks.rados.rados.0.smithi107.stdout:8788: seq_num 8787 ranges {0=770048} 2020-06-12T02:38:43.487 INFO:tasks.rados.rados.0.smithi107.stdout:8788: writing smithi10729407-17 from 0 to 770048 tid 1 2020-06-12T02:38:43.487 INFO:tasks.rados.rados.0.smithi107.stdout:8789: delete oid 5 current snap is 0 2020-06-12T02:38:43.487 INFO:tasks.rados.rados.0.smithi107.stdout:8789: done (1 left) 2020-06-12T02:38:43.487 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:43.487 INFO:tasks.rados.rados.0.smithi107.stdout:8790: seq_num 8789 ranges {0=802816} 2020-06-12T02:38:43.488 INFO:tasks.rados.rados.0.smithi107.stdout:8790: writing smithi10729407-14 from 0 to 802816 tid 1 2020-06-12T02:38:43.488 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:43.488 INFO:tasks.rados.rados.0.smithi107.stdout:8791: seq_num 8790 ranges {0=417792,417792=401408} 2020-06-12T02:38:43.488 INFO:tasks.rados.rados.0.smithi107.stdout:8791: writing smithi10729407-2 from 0 to 417792 tid 1 2020-06-12T02:38:43.488 INFO:tasks.rados.rados.0.smithi107.stdout:8791: writing smithi10729407-2 from 417792 to 819200 tid 2 2020-06-12T02:38:43.489 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:43.489 INFO:tasks.rados.rados.0.smithi107.stdout:8792: seq_num 8791 ranges {0=434176,434176=401408} 2020-06-12T02:38:43.489 INFO:tasks.rados.rados.0.smithi107.stdout:8792: writing smithi10729407-15 from 0 to 434176 tid 1 2020-06-12T02:38:43.489 INFO:tasks.rados.rados.0.smithi107.stdout:8792: writing smithi10729407-15 from 434176 to 835584 tid 2 2020-06-12T02:38:43.489 INFO:tasks.rados.rados.0.smithi107.stdout:8793: delete oid 8 current snap is 0 2020-06-12T02:38:43.489 INFO:tasks.rados.rados.0.smithi107.stdout:8788: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:43.490 INFO:tasks.rados.rados.0.smithi107.stdout:8788: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:43.490 INFO:tasks.rados.rados.0.smithi107.stdout:8788: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:43.490 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1655 (ObjNum 8787 snap 0 seq_num 8787) dirty exists 2020-06-12T02:38:43.490 INFO:tasks.rados.rados.0.smithi107.stdout:8788: left oid 17 (ObjNum 8787 snap 0 seq_num 8787) 2020-06-12T02:38:43.490 INFO:tasks.rados.rados.0.smithi107.stdout:8790: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:43.491 INFO:tasks.rados.rados.0.smithi107.stdout:8790: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:43.491 INFO:tasks.rados.rados.0.smithi107.stdout:8791: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:43.491 INFO:tasks.rados.rados.0.smithi107.stdout:8790: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:43.491 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1537 (ObjNum 8789 snap 0 seq_num 8789) dirty exists 2020-06-12T02:38:43.491 INFO:tasks.rados.rados.0.smithi107.stdout:8790: left oid 14 (ObjNum 8789 snap 0 seq_num 8789) 2020-06-12T02:38:43.491 INFO:tasks.rados.rados.0.smithi107.stdout:8791: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:43.492 INFO:tasks.rados.rados.0.smithi107.stdout:8791: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:43.492 INFO:tasks.rados.rados.0.smithi107.stdout:8791: finishing write tid 5 to smithi10729407-2 2020-06-12T02:38:43.492 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 819 (ObjNum 8790 snap 0 seq_num 8790) dirty exists 2020-06-12T02:38:43.492 INFO:tasks.rados.rados.0.smithi107.stdout:8791: left oid 2 (ObjNum 8790 snap 0 seq_num 8790) 2020-06-12T02:38:43.492 INFO:tasks.rados.rados.0.smithi107.stdout:8792: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:43.493 INFO:tasks.rados.rados.0.smithi107.stdout:8792: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:43.493 INFO:tasks.rados.rados.0.smithi107.stdout:8792: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:43.493 INFO:tasks.rados.rados.0.smithi107.stdout:8792: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:43.493 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1633 (ObjNum 8791 snap 0 seq_num 8791) dirty exists 2020-06-12T02:38:43.493 INFO:tasks.rados.rados.0.smithi107.stdout:8792: left oid 15 (ObjNum 8791 snap 0 seq_num 8791) 2020-06-12T02:38:43.493 INFO:tasks.rados.rados.0.smithi107.stdout:8788: done (4 left) 2020-06-12T02:38:43.494 INFO:tasks.rados.rados.0.smithi107.stdout:8790: done (3 left) 2020-06-12T02:38:43.494 INFO:tasks.rados.rados.0.smithi107.stdout:8791: done (2 left) 2020-06-12T02:38:43.494 INFO:tasks.rados.rados.0.smithi107.stdout:8792: done (1 left) 2020-06-12T02:38:43.494 INFO:tasks.rados.rados.0.smithi107.stdout:8793: done (0 left) 2020-06-12T02:38:43.494 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:43.495 INFO:tasks.rados.rados.0.smithi107.stdout:8794: seq_num 8793 ranges {0=475136,475136=401408} 2020-06-12T02:38:43.495 INFO:tasks.rados.rados.0.smithi107.stdout:8794: writing smithi10729407-16 from 0 to 475136 tid 1 2020-06-12T02:38:43.495 INFO:tasks.rados.rados.0.smithi107.stdout:8794: writing smithi10729407-16 from 475136 to 876544 tid 2 2020-06-12T02:38:43.495 INFO:tasks.rados.rados.0.smithi107.stdout:8795: delete oid 5 current snap is 0 2020-06-12T02:38:43.495 INFO:tasks.rados.rados.0.smithi107.stdout:8795: done (1 left) 2020-06-12T02:38:43.495 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:43.496 INFO:tasks.rados.rados.0.smithi107.stdout:8796: seq_num 8795 ranges {835584=507904,1343488=401408} 2020-06-12T02:38:43.496 INFO:tasks.rados.rados.0.smithi107.stdout:8796: writing smithi10729407-15 from 835584 to 1343488 tid 1 2020-06-12T02:38:43.496 INFO:tasks.rados.rados.0.smithi107.stdout:8796: writing smithi10729407-15 from 1343488 to 1744896 tid 2 2020-06-12T02:38:43.496 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:38:43.496 INFO:tasks.rados.rados.0.smithi107.stdout:8797: seq_num 8796 ranges {0=524288,524288=401408} 2020-06-12T02:38:43.497 INFO:tasks.rados.rados.0.smithi107.stdout:8797: writing smithi10729407-20 from 0 to 524288 tid 1 2020-06-12T02:38:43.497 INFO:tasks.rados.rados.0.smithi107.stdout:8797: writing smithi10729407-20 from 524288 to 925696 tid 2 2020-06-12T02:38:43.497 INFO:tasks.rados.rados.0.smithi107.stdout:8798: delete oid 13 current snap is 0 2020-06-12T02:38:43.497 INFO:tasks.rados.rados.0.smithi107.stdout:8794: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:43.497 INFO:tasks.rados.rados.0.smithi107.stdout:8794: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:43.497 INFO:tasks.rados.rados.0.smithi107.stdout:8794: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:43.498 INFO:tasks.rados.rados.0.smithi107.stdout:8794: finishing write tid 5 to smithi10729407-16 2020-06-12T02:38:43.498 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1583 (ObjNum 8793 snap 0 seq_num 8793) dirty exists 2020-06-12T02:38:43.498 INFO:tasks.rados.rados.0.smithi107.stdout:8794: left oid 16 (ObjNum 8793 snap 0 seq_num 8793) 2020-06-12T02:38:43.498 INFO:tasks.rados.rados.0.smithi107.stdout:8796: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:43.498 INFO:tasks.rados.rados.0.smithi107.stdout:8794: done (3 left) 2020-06-12T02:38:43.499 INFO:tasks.rados.rados.0.smithi107.stdout:8798: done (2 left) 2020-06-12T02:38:43.499 INFO:tasks.rados.rados.0.smithi107.stdout:8799: delete oid 19 current snap is 0 2020-06-12T02:38:43.499 INFO:tasks.rados.rados.0.smithi107.stdout:8796: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:43.499 INFO:tasks.rados.rados.0.smithi107.stdout:8796: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:43.499 INFO:tasks.rados.rados.0.smithi107.stdout:8799: done (2 left) 2020-06-12T02:38:43.499 INFO:tasks.rados.rados.0.smithi107.stdout:8800: delete oid 13 current snap is 0 2020-06-12T02:38:43.500 INFO:tasks.rados.rados.0.smithi107.stdout:8797: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:43.500 INFO:tasks.rados.rados.0.smithi107.stdout:8797: finishing write tid 2 to smithi10729407-20 2020-06-12T02:38:43.500 INFO:tasks.rados.rados.0.smithi107.stdout:8797: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:43.500 INFO:tasks.rados.rados.0.smithi107.stdout:8796: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:43.500 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1637 (ObjNum 8795 snap 0 seq_num 8795) dirty exists 2020-06-12T02:38:43.500 INFO:tasks.rados.rados.0.smithi107.stdout:8796: left oid 15 (ObjNum 8795 snap 0 seq_num 8795) 2020-06-12T02:38:43.501 INFO:tasks.rados.rados.0.smithi107.stdout:8797: finishing write tid 5 to smithi10729407-20 2020-06-12T02:38:43.501 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 648 (ObjNum 8796 snap 0 seq_num 8796) dirty exists 2020-06-12T02:38:43.501 INFO:tasks.rados.rados.0.smithi107.stdout:8797: left oid 20 (ObjNum 8796 snap 0 seq_num 8796) 2020-06-12T02:38:43.501 INFO:tasks.rados.rados.0.smithi107.stdout:8796: done (2 left) 2020-06-12T02:38:43.501 INFO:tasks.rados.rados.0.smithi107.stdout:8797: done (1 left) 2020-06-12T02:38:43.502 INFO:tasks.rados.rados.0.smithi107.stdout:8800: done (0 left) 2020-06-12T02:38:43.502 INFO:tasks.rados.rados.0.smithi107.stdout:8801: delete oid 9 current snap is 0 2020-06-12T02:38:43.502 INFO:tasks.rados.rados.0.smithi107.stdout:8801: done (0 left) 2020-06-12T02:38:43.502 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:38:43.502 INFO:tasks.rados.rados.0.smithi107.stdout:8802: seq_num 8801 ranges {0=606208,606208=401408} 2020-06-12T02:38:43.502 INFO:tasks.rados.rados.0.smithi107.stdout:8802: writing smithi10729407-6 from 0 to 606208 tid 1 2020-06-12T02:38:43.503 INFO:tasks.rados.rados.0.smithi107.stdout:8802: writing smithi10729407-6 from 606208 to 1007616 tid 2 2020-06-12T02:38:43.503 INFO:tasks.rados.rados.0.smithi107.stdout:8803: delete oid 1 current snap is 0 2020-06-12T02:38:43.503 INFO:tasks.rados.rados.0.smithi107.stdout:8802: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:43.503 INFO:tasks.rados.rados.0.smithi107.stdout:8802: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:43.503 INFO:tasks.rados.rados.0.smithi107.stdout:8802: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:43.504 INFO:tasks.rados.rados.0.smithi107.stdout:8803: done (1 left) 2020-06-12T02:38:43.504 INFO:tasks.rados.rados.0.smithi107.stdout:8804: delete oid 13 current snap is 0 2020-06-12T02:38:43.504 INFO:tasks.rados.rados.0.smithi107.stdout:8802: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:43.504 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1601 (ObjNum 8801 snap 0 seq_num 8801) dirty exists 2020-06-12T02:38:43.504 INFO:tasks.rados.rados.0.smithi107.stdout:8802: left oid 6 (ObjNum 8801 snap 0 seq_num 8801) 2020-06-12T02:38:43.504 INFO:tasks.rados.rados.0.smithi107.stdout:8802: done (1 left) 2020-06-12T02:38:43.505 INFO:tasks.rados.rados.0.smithi107.stdout:8804: done (0 left) 2020-06-12T02:38:43.505 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:43.505 INFO:tasks.rados.rados.0.smithi107.stdout:8805: seq_num 8804 ranges {0=655360,655360=401408} 2020-06-12T02:38:43.505 INFO:tasks.rados.rados.0.smithi107.stdout:8805: writing smithi10729407-4 from 0 to 655360 tid 1 2020-06-12T02:38:43.505 INFO:tasks.rados.rados.0.smithi107.stdout:8805: writing smithi10729407-4 from 655360 to 1056768 tid 2 2020-06-12T02:38:43.506 INFO:tasks.rados.rados.0.smithi107.stdout:8806: delete oid 11 current snap is 0 2020-06-12T02:38:43.506 INFO:tasks.rados.rados.0.smithi107.stdout:8806: done (1 left) 2020-06-12T02:38:43.506 INFO:tasks.rados.rados.0.smithi107.stdout:8807: delete oid 8 current snap is 0 2020-06-12T02:38:43.506 INFO:tasks.rados.rados.0.smithi107.stdout:8805: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:43.506 INFO:tasks.rados.rados.0.smithi107.stdout:8807: done (1 left) 2020-06-12T02:38:43.506 INFO:tasks.rados.rados.0.smithi107.stdout:8808: delete oid 20 current snap is 0 2020-06-12T02:38:43.507 INFO:tasks.rados.rados.0.smithi107.stdout:8808: done (1 left) 2020-06-12T02:38:43.507 INFO:tasks.rados.rados.0.smithi107.stdout:8809: delete oid 17 current snap is 0 2020-06-12T02:38:43.507 INFO:tasks.rados.rados.0.smithi107.stdout:8805: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:43.507 INFO:tasks.rados.rados.0.smithi107.stdout:8805: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:43.507 INFO:tasks.rados.rados.0.smithi107.stdout:8805: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:43.507 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1658 (ObjNum 8804 snap 0 seq_num 8804) dirty exists 2020-06-12T02:38:43.508 INFO:tasks.rados.rados.0.smithi107.stdout:8805: left oid 4 (ObjNum 8804 snap 0 seq_num 8804) 2020-06-12T02:38:43.508 INFO:tasks.rados.rados.0.smithi107.stdout:8805: done (1 left) 2020-06-12T02:38:43.508 INFO:tasks.rados.rados.0.smithi107.stdout:8809: done (0 left) 2020-06-12T02:38:43.508 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:43.508 INFO:tasks.rados.rados.0.smithi107.stdout:8810: seq_num 8809 ranges {0=737280,737280=401408} 2020-06-12T02:38:43.509 INFO:tasks.rados.rados.0.smithi107.stdout:8810: writing smithi10729407-19 from 0 to 737280 tid 1 2020-06-12T02:38:43.509 INFO:tasks.rados.rados.0.smithi107.stdout:8810: writing smithi10729407-19 from 737280 to 1138688 tid 2 2020-06-12T02:38:43.509 INFO:tasks.rados.rados.0.smithi107.stdout:8811: delete oid 20 current snap is 0 2020-06-12T02:38:43.509 INFO:tasks.rados.rados.0.smithi107.stdout:8811: done (1 left) 2020-06-12T02:38:43.509 INFO:tasks.rados.rados.0.smithi107.stdout:8812: delete oid 17 current snap is 0 2020-06-12T02:38:43.509 INFO:tasks.rados.rados.0.smithi107.stdout:8812: done (1 left) 2020-06-12T02:38:43.510 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:43.510 INFO:tasks.rados.rados.0.smithi107.stdout:8813: seq_num 8812 ranges {1744896=794624,2539520=401408} 2020-06-12T02:38:43.510 INFO:tasks.rados.rados.0.smithi107.stdout:8813: writing smithi10729407-15 from 1744896 to 2539520 tid 1 2020-06-12T02:38:43.510 INFO:tasks.rados.rados.0.smithi107.stdout:8813: writing smithi10729407-15 from 2539520 to 2940928 tid 2 2020-06-12T02:38:43.510 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:43.511 INFO:tasks.rados.rados.0.smithi107.stdout:8814: seq_num 8813 ranges {0=409600,409600=598016,1007616=204800} 2020-06-12T02:38:43.511 INFO:tasks.rados.rados.0.smithi107.stdout:8814: writing smithi10729407-18 from 0 to 409600 tid 1 2020-06-12T02:38:43.511 INFO:tasks.rados.rados.0.smithi107.stdout:8814: writing smithi10729407-18 from 409600 to 1007616 tid 2 2020-06-12T02:38:43.511 INFO:tasks.rados.rados.0.smithi107.stdout:8814: writing smithi10729407-18 from 1007616 to 1212416 tid 3 2020-06-12T02:38:43.511 INFO:tasks.rados.rados.0.smithi107.stdout:8815: delete oid 11 current snap is 0 2020-06-12T02:38:43.512 INFO:tasks.rados.rados.0.smithi107.stdout:8810: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:43.512 INFO:tasks.rados.rados.0.smithi107.stdout:8810: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:43.512 INFO:tasks.rados.rados.0.smithi107.stdout:8810: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:43.512 INFO:tasks.rados.rados.0.smithi107.stdout:8810: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:43.512 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1625 (ObjNum 8809 snap 0 seq_num 8809) dirty exists 2020-06-12T02:38:43.513 INFO:tasks.rados.rados.0.smithi107.stdout:8810: left oid 19 (ObjNum 8809 snap 0 seq_num 8809) 2020-06-12T02:38:43.513 INFO:tasks.rados.rados.0.smithi107.stdout:8813: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:43.513 INFO:tasks.rados.rados.0.smithi107.stdout:8813: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:43.513 INFO:tasks.rados.rados.0.smithi107.stdout:8813: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:43.513 INFO:tasks.rados.rados.0.smithi107.stdout:8813: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:43.514 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1641 (ObjNum 8812 snap 0 seq_num 8812) dirty exists 2020-06-12T02:38:43.514 INFO:tasks.rados.rados.0.smithi107.stdout:8813: left oid 15 (ObjNum 8812 snap 0 seq_num 8812) 2020-06-12T02:38:43.514 INFO:tasks.rados.rados.0.smithi107.stdout:8814: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:43.514 INFO:tasks.rados.rados.0.smithi107.stdout:8810: done (3 left) 2020-06-12T02:38:43.514 INFO:tasks.rados.rados.0.smithi107.stdout:8813: done (2 left) 2020-06-12T02:38:43.515 INFO:tasks.rados.rados.0.smithi107.stdout:8815: done (1 left) 2020-06-12T02:38:43.515 INFO:tasks.rados.rados.0.smithi107.stdout:8816: delete oid 1 current snap is 0 2020-06-12T02:38:43.515 INFO:tasks.rados.rados.0.smithi107.stdout:8814: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:43.515 INFO:tasks.rados.rados.0.smithi107.stdout:8814: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:43.516 INFO:tasks.rados.rados.0.smithi107.stdout:8816: done (1 left) 2020-06-12T02:38:43.516 INFO:tasks.rados.rados.0.smithi107.stdout:8817: delete oid 15 current snap is 0 2020-06-12T02:38:43.516 INFO:tasks.rados.rados.0.smithi107.stdout:8814: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:43.516 INFO:tasks.rados.rados.0.smithi107.stdout:8817: done (1 left) 2020-06-12T02:38:43.516 INFO:tasks.rados.rados.0.smithi107.stdout:8818: delete oid 19 current snap is 0 2020-06-12T02:38:43.516 INFO:tasks.rados.rados.0.smithi107.stdout:8814: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:43.517 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1657 (ObjNum 8813 snap 0 seq_num 8813) dirty exists 2020-06-12T02:38:43.517 INFO:tasks.rados.rados.0.smithi107.stdout:8814: left oid 18 (ObjNum 8813 snap 0 seq_num 8813) 2020-06-12T02:38:43.517 INFO:tasks.rados.rados.0.smithi107.stdout:8814: done (1 left) 2020-06-12T02:38:43.517 INFO:tasks.rados.rados.0.smithi107.stdout:8818: done (0 left) 2020-06-12T02:38:43.517 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:43.518 INFO:tasks.rados.rados.0.smithi107.stdout:8819: seq_num 8818 ranges {0=491520,491520=425984,917504=376832} 2020-06-12T02:38:43.518 INFO:tasks.rados.rados.0.smithi107.stdout:8819: writing smithi10729407-7 from 0 to 491520 tid 1 2020-06-12T02:38:43.518 INFO:tasks.rados.rados.0.smithi107.stdout:8819: writing smithi10729407-7 from 491520 to 917504 tid 2 2020-06-12T02:38:43.518 INFO:tasks.rados.rados.0.smithi107.stdout:8819: writing smithi10729407-7 from 917504 to 1294336 tid 3 2020-06-12T02:38:43.518 INFO:tasks.rados.rados.0.smithi107.stdout:8820: delete oid 13 current snap is 0 2020-06-12T02:38:43.518 INFO:tasks.rados.rados.0.smithi107.stdout:8819: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:43.519 INFO:tasks.rados.rados.0.smithi107.stdout:8819: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:43.519 INFO:tasks.rados.rados.0.smithi107.stdout:8819: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:43.519 INFO:tasks.rados.rados.0.smithi107.stdout:8819: finishing write tid 5 to smithi10729407-7 2020-06-12T02:38:43.519 INFO:tasks.rados.rados.0.smithi107.stdout:8819: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:43.519 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1588 (ObjNum 8818 snap 0 seq_num 8818) dirty exists 2020-06-12T02:38:43.520 INFO:tasks.rados.rados.0.smithi107.stdout:8819: left oid 7 (ObjNum 8818 snap 0 seq_num 8818) 2020-06-12T02:38:43.520 INFO:tasks.rados.rados.0.smithi107.stdout:8819: done (1 left) 2020-06-12T02:38:43.520 INFO:tasks.rados.rados.0.smithi107.stdout:8820: done (0 left) 2020-06-12T02:38:43.520 INFO:tasks.rados.rados.0.smithi107.stdout:8821: delete oid 4 current snap is 0 2020-06-12T02:38:43.520 INFO:tasks.rados.rados.0.smithi107.stdout:8821: done (0 left) 2020-06-12T02:38:43.520 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:43.521 INFO:tasks.rados.rados.0.smithi107.stdout:8822: seq_num 8821 ranges {0=540672,540672=524288,1064960=278528} 2020-06-12T02:38:43.521 INFO:tasks.rados.rados.0.smithi107.stdout:8822: writing smithi10729407-13 from 0 to 540672 tid 1 2020-06-12T02:38:43.521 INFO:tasks.rados.rados.0.smithi107.stdout:8822: writing smithi10729407-13 from 540672 to 1064960 tid 2 2020-06-12T02:38:43.521 INFO:tasks.rados.rados.0.smithi107.stdout:8822: writing smithi10729407-13 from 1064960 to 1343488 tid 3 2020-06-12T02:38:43.521 INFO:tasks.rados.rados.0.smithi107.stdout:8823: delete oid 7 current snap is 0 2020-06-12T02:38:43.522 INFO:tasks.rados.rados.0.smithi107.stdout:8822: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:43.522 INFO:tasks.rados.rados.0.smithi107.stdout:8823: done (1 left) 2020-06-12T02:38:43.522 INFO:tasks.rados.rados.0.smithi107.stdout:8824: delete oid 19 current snap is 0 2020-06-12T02:38:43.522 INFO:tasks.rados.rados.0.smithi107.stdout:8822: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:43.522 INFO:tasks.rados.rados.0.smithi107.stdout:8822: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:43.522 INFO:tasks.rados.rados.0.smithi107.stdout:8822: finishing write tid 5 to smithi10729407-13 2020-06-12T02:38:43.523 INFO:tasks.rados.rados.0.smithi107.stdout:8824: done (1 left) 2020-06-12T02:38:43.523 INFO:tasks.rados.rados.0.smithi107.stdout:8825: delete oid 5 current snap is 0 2020-06-12T02:38:43.523 INFO:tasks.rados.rados.0.smithi107.stdout:8822: finishing write tid 6 to smithi10729407-13 2020-06-12T02:38:43.523 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1594 (ObjNum 8821 snap 0 seq_num 8821) dirty exists 2020-06-12T02:38:43.523 INFO:tasks.rados.rados.0.smithi107.stdout:8822: left oid 13 (ObjNum 8821 snap 0 seq_num 8821) 2020-06-12T02:38:43.524 INFO:tasks.rados.rados.0.smithi107.stdout:8822: done (1 left) 2020-06-12T02:38:43.524 INFO:tasks.rados.rados.0.smithi107.stdout:8825: done (0 left) 2020-06-12T02:38:43.524 INFO:tasks.rados.rados.0.smithi107.stdout:8826: delete oid 3 current snap is 0 2020-06-12T02:38:43.524 INFO:tasks.rados.rados.0.smithi107.stdout:8826: done (0 left) 2020-06-12T02:38:43.524 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:43.524 INFO:tasks.rados.rados.0.smithi107.stdout:8827: seq_num 8826 ranges {0=622592,622592=745472,1368064=57344} 2020-06-12T02:38:43.525 INFO:tasks.rados.rados.0.smithi107.stdout:8827: writing smithi10729407-19 from 0 to 622592 tid 1 2020-06-12T02:38:43.525 INFO:tasks.rados.rados.0.smithi107.stdout:8827: writing smithi10729407-19 from 622592 to 1368064 tid 2 2020-06-12T02:38:43.525 INFO:tasks.rados.rados.0.smithi107.stdout:8827: writing smithi10729407-19 from 1368064 to 1425408 tid 3 2020-06-12T02:38:43.525 INFO:tasks.rados.rados.0.smithi107.stdout:8828: delete oid 12 current snap is 0 2020-06-12T02:38:43.525 INFO:tasks.rados.rados.0.smithi107.stdout:8827: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:43.526 INFO:tasks.rados.rados.0.smithi107.stdout:8827: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:43.526 INFO:tasks.rados.rados.0.smithi107.stdout:8827: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:43.526 INFO:tasks.rados.rados.0.smithi107.stdout:8827: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:43.526 INFO:tasks.rados.rados.0.smithi107.stdout:8828: done (1 left) 2020-06-12T02:38:43.526 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:43.526 INFO:tasks.rados.rados.0.smithi107.stdout:8829: seq_num 8828 ranges {0=655360,655360=565248,1220608=237568} 2020-06-12T02:38:43.527 INFO:tasks.rados.rados.0.smithi107.stdout:8829: writing smithi10729407-8 from 0 to 655360 tid 1 2020-06-12T02:38:43.527 INFO:tasks.rados.rados.0.smithi107.stdout:8829: writing smithi10729407-8 from 655360 to 1220608 tid 2 2020-06-12T02:38:43.527 INFO:tasks.rados.rados.0.smithi107.stdout:8829: writing smithi10729407-8 from 1220608 to 1458176 tid 3 2020-06-12T02:38:43.527 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:43.527 INFO:tasks.rados.rados.0.smithi107.stdout:8830: seq_num 8829 ranges {802816=671744,1474560=450560,1925120=352256} 2020-06-12T02:38:43.527 INFO:tasks.rados.rados.0.smithi107.stdout:8830: writing smithi10729407-14 from 802816 to 1474560 tid 1 2020-06-12T02:38:43.528 INFO:tasks.rados.rados.0.smithi107.stdout:8830: writing smithi10729407-14 from 1474560 to 1925120 tid 2 2020-06-12T02:38:43.528 INFO:tasks.rados.rados.0.smithi107.stdout:8830: writing smithi10729407-14 from 1925120 to 2277376 tid 3 2020-06-12T02:38:43.528 INFO:tasks.rados.rados.0.smithi107.stdout:8831: delete oid 16 current snap is 0 2020-06-12T02:38:43.528 INFO:tasks.rados.rados.0.smithi107.stdout:8827: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:43.528 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1631 (ObjNum 8826 snap 0 seq_num 8826) dirty exists 2020-06-12T02:38:43.529 INFO:tasks.rados.rados.0.smithi107.stdout:8827: left oid 19 (ObjNum 8826 snap 0 seq_num 8826) 2020-06-12T02:38:43.529 INFO:tasks.rados.rados.0.smithi107.stdout:8829: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:43.529 INFO:tasks.rados.rados.0.smithi107.stdout:8829: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:43.529 INFO:tasks.rados.rados.0.smithi107.stdout:8829: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:43.529 INFO:tasks.rados.rados.0.smithi107.stdout:8829: finishing write tid 5 to smithi10729407-8 2020-06-12T02:38:43.529 INFO:tasks.rados.rados.0.smithi107.stdout:8830: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:43.530 INFO:tasks.rados.rados.0.smithi107.stdout:8827: done (3 left) 2020-06-12T02:38:43.530 INFO:tasks.rados.rados.0.smithi107.stdout:8831: done (2 left) 2020-06-12T02:38:43.530 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:38:43.530 INFO:tasks.rados.rados.0.smithi107.stdout:8832: seq_num 8831 ranges {0=704512,704512=622592,1327104=180224} 2020-06-12T02:38:43.530 INFO:tasks.rados.rados.0.smithi107.stdout:8832: writing smithi10729407-12 from 0 to 704512 tid 1 2020-06-12T02:38:43.531 INFO:tasks.rados.rados.0.smithi107.stdout:8832: writing smithi10729407-12 from 704512 to 1327104 tid 2 2020-06-12T02:38:43.531 INFO:tasks.rados.rados.0.smithi107.stdout:8832: writing smithi10729407-12 from 1327104 to 1507328 tid 3 2020-06-12T02:38:43.531 INFO:tasks.rados.rados.0.smithi107.stdout:8833: delete oid 13 current snap is 0 2020-06-12T02:38:43.531 INFO:tasks.rados.rados.0.smithi107.stdout:8829: finishing write tid 6 to smithi10729407-8 2020-06-12T02:38:43.531 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1646 (ObjNum 8828 snap 0 seq_num 8828) dirty exists 2020-06-12T02:38:43.531 INFO:tasks.rados.rados.0.smithi107.stdout:8829: left oid 8 (ObjNum 8828 snap 0 seq_num 8828) 2020-06-12T02:38:43.532 INFO:tasks.rados.rados.0.smithi107.stdout:8830: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:43.532 INFO:tasks.rados.rados.0.smithi107.stdout:8830: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:43.532 INFO:tasks.rados.rados.0.smithi107.stdout:8830: finishing write tid 5 to smithi10729407-14 2020-06-12T02:38:43.532 INFO:tasks.rados.rados.0.smithi107.stdout:8830: finishing write tid 6 to smithi10729407-14 2020-06-12T02:38:43.532 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1543 (ObjNum 8829 snap 0 seq_num 8829) dirty exists 2020-06-12T02:38:43.533 INFO:tasks.rados.rados.0.smithi107.stdout:8830: left oid 14 (ObjNum 8829 snap 0 seq_num 8829) 2020-06-12T02:38:43.533 INFO:tasks.rados.rados.0.smithi107.stdout:8829: done (3 left) 2020-06-12T02:38:43.533 INFO:tasks.rados.rados.0.smithi107.stdout:8830: done (2 left) 2020-06-12T02:38:43.533 INFO:tasks.rados.rados.0.smithi107.stdout:8833: done (1 left) 2020-06-12T02:38:43.533 INFO:tasks.rados.rados.0.smithi107.stdout:8834: delete oid 4 current snap is 0 2020-06-12T02:38:43.533 INFO:tasks.rados.rados.0.smithi107.stdout:8832: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:43.534 INFO:tasks.rados.rados.0.smithi107.stdout:8832: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:43.534 INFO:tasks.rados.rados.0.smithi107.stdout:8832: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:43.534 INFO:tasks.rados.rados.0.smithi107.stdout:8834: done (1 left) 2020-06-12T02:38:43.534 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:43.534 INFO:tasks.rados.rados.0.smithi107.stdout:8835: seq_num 8834 ranges {1212416=761856,1974272=720896,2695168=81920} 2020-06-12T02:38:43.534 INFO:tasks.rados.rados.0.smithi107.stdout:8835: writing smithi10729407-18 from 1212416 to 1974272 tid 1 2020-06-12T02:38:43.535 INFO:tasks.rados.rados.0.smithi107.stdout:8835: writing smithi10729407-18 from 1974272 to 2695168 tid 2 2020-06-12T02:38:43.535 INFO:tasks.rados.rados.0.smithi107.stdout:8835: writing smithi10729407-18 from 2695168 to 2777088 tid 3 2020-06-12T02:38:43.535 INFO:tasks.rados.rados.0.smithi107.stdout:8836: delete oid 10 current snap is 0 2020-06-12T02:38:43.535 INFO:tasks.rados.rados.0.smithi107.stdout:8832: finishing write tid 5 to smithi10729407-12 2020-06-12T02:38:43.535 INFO:tasks.rados.rados.0.smithi107.stdout:8832: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:43.536 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1636 (ObjNum 8831 snap 0 seq_num 8831) dirty exists 2020-06-12T02:38:43.536 INFO:tasks.rados.rados.0.smithi107.stdout:8832: left oid 12 (ObjNum 8831 snap 0 seq_num 8831) 2020-06-12T02:38:43.536 INFO:tasks.rados.rados.0.smithi107.stdout:8832: done (2 left) 2020-06-12T02:38:43.536 INFO:tasks.rados.rados.0.smithi107.stdout:8836: done (1 left) 2020-06-12T02:38:43.536 INFO:tasks.rados.rados.0.smithi107.stdout:8837: delete oid 3 current snap is 0 2020-06-12T02:38:43.536 INFO:tasks.rados.rados.0.smithi107.stdout:8835: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:43.537 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-06-12T02:38:43.538 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:38:43.538 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:38:43.538 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:38:43.538 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:38:43.539 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:38:43.539 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:38:43.539 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2020-06-12T02:38:43.539 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-06-12T02:38:43.539 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-06-12T02:38:43.540 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-06-12T02:38:43.540 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-06-12T02:38:43.540 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-06-12T02:38:43.540 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-06-12T02:38:43.540 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-06-12T02:38:43.540 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-06-12T02:38:43.541 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:43.466+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:43.544 INFO:tasks.rados.rados.0.smithi107.stdout:8837: done (1 left) 2020-06-12T02:38:43.544 INFO:tasks.rados.rados.0.smithi107.stdout:8838: delete oid 20 current snap is 0 2020-06-12T02:38:43.545 INFO:tasks.rados.rados.0.smithi107.stdout:8838: done (1 left) 2020-06-12T02:38:43.545 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:43.545 INFO:tasks.rados.rados.0.smithi107.stdout:8839: seq_num 8838 ranges {1507328=425984,1933312=663552,2596864=458752,3055616=81920} 2020-06-12T02:38:43.545 INFO:tasks.rados.rados.0.smithi107.stdout:8839: writing smithi10729407-12 from 1507328 to 1933312 tid 1 2020-06-12T02:38:43.545 INFO:tasks.rados.rados.0.smithi107.stdout:8839: writing smithi10729407-12 from 1933312 to 2596864 tid 2 2020-06-12T02:38:43.546 INFO:tasks.rados.rados.0.smithi107.stdout:8839: writing smithi10729407-12 from 2596864 to 3055616 tid 3 2020-06-12T02:38:43.546 INFO:tasks.rados.rados.0.smithi107.stdout:8839: writing smithi10729407-12 from 3055616 to 3137536 tid 4 2020-06-12T02:38:43.546 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:38:43.546 INFO:tasks.rados.rados.0.smithi107.stdout:8840: seq_num 8839 ranges {0=442368,442368=540672,983040=614400,1597440=49152} 2020-06-12T02:38:43.546 INFO:tasks.rados.rados.0.smithi107.stdout:8840: writing smithi10729407-3 from 0 to 442368 tid 1 2020-06-12T02:38:43.547 INFO:tasks.rados.rados.0.smithi107.stdout:8840: writing smithi10729407-3 from 442368 to 983040 tid 2 2020-06-12T02:38:43.547 INFO:tasks.rados.rados.0.smithi107.stdout:8840: writing smithi10729407-3 from 983040 to 1597440 tid 3 2020-06-12T02:38:43.547 INFO:tasks.rados.rados.0.smithi107.stdout:8840: writing smithi10729407-3 from 1597440 to 1646592 tid 4 2020-06-12T02:38:43.547 INFO:tasks.rados.rados.0.smithi107.stdout:8841: delete oid 14 current snap is 0 2020-06-12T02:38:43.547 INFO:tasks.rados.rados.0.smithi107.stdout:8835: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:43.547 INFO:tasks.rados.rados.0.smithi107.stdout:8835: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:43.548 INFO:tasks.rados.rados.0.smithi107.stdout:8835: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:43.548 INFO:tasks.rados.rados.0.smithi107.stdout:8835: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:43.548 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1661 (ObjNum 8834 snap 0 seq_num 8834) dirty exists 2020-06-12T02:38:43.548 INFO:tasks.rados.rados.0.smithi107.stdout:8835: left oid 18 (ObjNum 8834 snap 0 seq_num 8834) 2020-06-12T02:38:43.548 INFO:tasks.rados.rados.0.smithi107.stdout:8839: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:43.549 INFO:tasks.rados.rados.0.smithi107.stdout:8839: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:43.549 INFO:tasks.rados.rados.0.smithi107.stdout:8839: finishing write tid 3 to smithi10729407-12 2020-06-12T02:38:43.549 INFO:tasks.rados.rados.0.smithi107.stdout:8840: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:43.549 INFO:tasks.rados.rados.0.smithi107.stdout:8839: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:43.550 INFO:tasks.rados.rados.0.smithi107.stdout:8839: finishing write tid 6 to smithi10729407-12 2020-06-12T02:38:43.550 INFO:tasks.rados.rados.0.smithi107.stdout:8839: finishing write tid 7 to smithi10729407-12 2020-06-12T02:38:43.550 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1641 (ObjNum 8838 snap 0 seq_num 8838) dirty exists 2020-06-12T02:38:43.550 INFO:tasks.rados.rados.0.smithi107.stdout:8839: left oid 12 (ObjNum 8838 snap 0 seq_num 8838) 2020-06-12T02:38:43.551 INFO:tasks.rados.rados.0.smithi107.stdout:8835: done (3 left) 2020-06-12T02:38:43.551 INFO:tasks.rados.rados.0.smithi107.stdout:8839: done (2 left) 2020-06-12T02:38:43.551 INFO:tasks.rados.rados.0.smithi107.stdout:8841: done (1 left) 2020-06-12T02:38:43.551 INFO:tasks.rados.rados.0.smithi107.stdout:8842: delete oid 12 current snap is 0 2020-06-12T02:38:43.551 INFO:tasks.rados.rados.0.smithi107.stdout:8840: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:43.551 INFO:tasks.rados.rados.0.smithi107.stdout:8840: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:43.552 INFO:tasks.rados.rados.0.smithi107.stdout:8842: done (1 left) 2020-06-12T02:38:43.552 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:38:43.552 INFO:tasks.rados.rados.0.smithi107.stdout:8843: seq_num 8842 ranges {0=491520,491520=647168,1138688=434176,1572864=122880} 2020-06-12T02:38:43.552 INFO:tasks.rados.rados.0.smithi107.stdout:8843: writing smithi10729407-10 from 0 to 491520 tid 1 2020-06-12T02:38:43.552 INFO:tasks.rados.rados.0.smithi107.stdout:8843: writing smithi10729407-10 from 491520 to 1138688 tid 2 2020-06-12T02:38:43.553 INFO:tasks.rados.rados.0.smithi107.stdout:8843: writing smithi10729407-10 from 1138688 to 1572864 tid 3 2020-06-12T02:38:43.553 INFO:tasks.rados.rados.0.smithi107.stdout:8843: writing smithi10729407-10 from 1572864 to 1695744 tid 4 2020-06-12T02:38:43.553 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:38:43.553 INFO:tasks.rados.rados.0.smithi107.stdout:8844: seq_num 8843 ranges {0=507904,507904=532480,1040384=598016,1638400=73728} 2020-06-12T02:38:43.553 INFO:tasks.rados.rados.0.smithi107.stdout:8844: writing smithi10729407-20 from 0 to 507904 tid 1 2020-06-12T02:38:43.554 INFO:tasks.rados.rados.0.smithi107.stdout:8844: writing smithi10729407-20 from 507904 to 1040384 tid 2 2020-06-12T02:38:43.555 INFO:tasks.rados.rados.0.smithi107.stdout:8844: writing smithi10729407-20 from 1040384 to 1638400 tid 3 2020-06-12T02:38:43.555 INFO:tasks.rados.rados.0.smithi107.stdout:8844: writing smithi10729407-20 from 1638400 to 1712128 tid 4 2020-06-12T02:38:43.555 INFO:tasks.rados.rados.0.smithi107.stdout:8845: delete oid 5 current snap is 0 2020-06-12T02:38:43.555 INFO:tasks.rados.rados.0.smithi107.stdout:8840: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:43.556 INFO:tasks.rados.rados.0.smithi107.stdout:8840: finishing write tid 6 to smithi10729407-3 2020-06-12T02:38:43.556 INFO:tasks.rados.rados.0.smithi107.stdout:8840: finishing write tid 7 to smithi10729407-3 2020-06-12T02:38:43.556 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1591 (ObjNum 8839 snap 0 seq_num 8839) dirty exists 2020-06-12T02:38:43.556 INFO:tasks.rados.rados.0.smithi107.stdout:8840: left oid 3 (ObjNum 8839 snap 0 seq_num 8839) 2020-06-12T02:38:43.556 INFO:tasks.rados.rados.0.smithi107.stdout:8843: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:43.557 INFO:tasks.rados.rados.0.smithi107.stdout:8843: finishing write tid 2 to smithi10729407-10 2020-06-12T02:38:43.557 INFO:tasks.rados.rados.0.smithi107.stdout:8843: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:43.557 INFO:tasks.rados.rados.0.smithi107.stdout:8843: finishing write tid 4 to smithi10729407-10 2020-06-12T02:38:43.557 INFO:tasks.rados.rados.0.smithi107.stdout:8843: finishing write tid 6 to smithi10729407-10 2020-06-12T02:38:43.557 INFO:tasks.rados.rados.0.smithi107.stdout:8844: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:43.558 INFO:tasks.rados.rados.0.smithi107.stdout:8843: finishing write tid 7 to smithi10729407-10 2020-06-12T02:38:43.558 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 640 (ObjNum 8842 snap 0 seq_num 8842) dirty exists 2020-06-12T02:38:43.558 INFO:tasks.rados.rados.0.smithi107.stdout:8843: left oid 10 (ObjNum 8842 snap 0 seq_num 8842) 2020-06-12T02:38:43.559 INFO:tasks.rados.rados.0.smithi107.stdout:8840: done (3 left) 2020-06-12T02:38:43.559 INFO:tasks.rados.rados.0.smithi107.stdout:8843: done (2 left) 2020-06-12T02:38:43.559 INFO:tasks.rados.rados.0.smithi107.stdout:8845: done (1 left) 2020-06-12T02:38:43.560 INFO:tasks.rados.rados.0.smithi107.stdout:8846: delete oid 18 current snap is 0 2020-06-12T02:38:43.560 INFO:tasks.rados.rados.0.smithi107.stdout:8844: finishing write tid 2 to smithi10729407-20 2020-06-12T02:38:43.560 INFO:tasks.rados.rados.0.smithi107.stdout:8846: done (1 left) 2020-06-12T02:38:43.560 INFO:tasks.rados.rados.0.smithi107.stdout:8847: delete oid 7 current snap is 0 2020-06-12T02:38:43.560 INFO:tasks.rados.rados.0.smithi107.stdout:8844: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:43.561 INFO:tasks.rados.rados.0.smithi107.stdout:8844: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:43.561 INFO:tasks.rados.rados.0.smithi107.stdout:8844: finishing write tid 6 to smithi10729407-20 2020-06-12T02:38:43.563 INFO:tasks.rados.rados.0.smithi107.stdout:8844: finishing write tid 7 to smithi10729407-20 2020-06-12T02:38:43.563 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 656 (ObjNum 8843 snap 0 seq_num 8843) dirty exists 2020-06-12T02:38:43.563 INFO:tasks.rados.rados.0.smithi107.stdout:8844: left oid 20 (ObjNum 8843 snap 0 seq_num 8843) 2020-06-12T02:38:43.563 INFO:tasks.rados.rados.0.smithi107.stdout:8844: done (1 left) 2020-06-12T02:38:43.563 INFO:tasks.rados.rados.0.smithi107.stdout:8847: done (0 left) 2020-06-12T02:38:43.564 INFO:tasks.rados.rados.0.smithi107.stdout:8848: delete oid 7 current snap is 0 2020-06-12T02:38:43.565 INFO:tasks.rados.rados.0.smithi107.stdout:8848: done (0 left) 2020-06-12T02:38:43.565 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:43.565 INFO:tasks.rados.rados.0.smithi107.stdout:8849: seq_num 8848 ranges {0=589824,589824=753664,1343488=450560} 2020-06-12T02:38:43.568 INFO:tasks.rados.rados.0.smithi107.stdout:8849: writing smithi10729407-16 from 0 to 589824 tid 1 2020-06-12T02:38:43.569 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:43.566+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:43.574 INFO:tasks.rados.rados.0.smithi107.stdout:8849: writing smithi10729407-16 from 589824 to 1343488 tid 2 2020-06-12T02:38:43.577 INFO:tasks.rados.rados.0.smithi107.stdout:8849: writing smithi10729407-16 from 1343488 to 1794048 tid 3 2020-06-12T02:38:43.577 INFO:tasks.rados.rados.0.smithi107.stdout:8850: delete oid 14 current snap is 0 2020-06-12T02:38:43.577 INFO:tasks.rados.rados.0.smithi107.stdout:8849: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:43.580 INFO:tasks.rados.rados.0.smithi107.stdout:8850: done (1 left) 2020-06-12T02:38:43.580 INFO:tasks.rados.rados.0.smithi107.stdout:8851: delete oid 20 current snap is 0 2020-06-12T02:38:43.583 INFO:tasks.rados.rados.0.smithi107.stdout:8851: done (1 left) 2020-06-12T02:38:43.583 INFO:tasks.rados.rados.0.smithi107.stdout:8852: delete oid 19 current snap is 0 2020-06-12T02:38:43.586 INFO:tasks.rados.rados.0.smithi107.stdout:8849: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:43.587 INFO:tasks.rados.rados.0.smithi107.stdout:8849: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:43.587 INFO:tasks.rados.rados.0.smithi107.stdout:8852: done (1 left) 2020-06-12T02:38:43.587 INFO:tasks.rados.rados.0.smithi107.stdout:8853: delete oid 19 current snap is 0 2020-06-12T02:38:43.588 INFO:tasks.rados.rados.0.smithi107.stdout:8849: finishing write tid 5 to smithi10729407-16 2020-06-12T02:38:43.590 INFO:tasks.rados.rados.0.smithi107.stdout:8849: finishing write tid 6 to smithi10729407-16 2020-06-12T02:38:43.590 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1595 (ObjNum 8848 snap 0 seq_num 8848) dirty exists 2020-06-12T02:38:43.590 INFO:tasks.rados.rados.0.smithi107.stdout:8849: left oid 16 (ObjNum 8848 snap 0 seq_num 8848) 2020-06-12T02:38:43.591 INFO:tasks.rados.rados.0.smithi107.stdout:8849: done (1 left) 2020-06-12T02:38:43.591 INFO:tasks.rados.rados.0.smithi107.stdout:8853: done (0 left) 2020-06-12T02:38:43.592 INFO:tasks.rados.rados.0.smithi107.stdout:8854: delete oid 4 current snap is 0 2020-06-12T02:38:43.593 INFO:tasks.rados.rados.0.smithi107.stdout:8854: done (0 left) 2020-06-12T02:38:43.594 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:43.594 INFO:tasks.rados.rados.0.smithi107.stdout:8855: seq_num 8854 ranges {819200=696320,1515520=507904,2023424=696320} 2020-06-12T02:38:43.602 INFO:tasks.rados.rados.0.smithi107.stdout:8855: writing smithi10729407-2 from 819200 to 1515520 tid 1 2020-06-12T02:38:43.605 INFO:tasks.rados.rados.0.smithi107.stdout:8855: writing smithi10729407-2 from 1515520 to 2023424 tid 2 2020-06-12T02:38:43.610 INFO:tasks.rados.rados.0.smithi107.stdout:8855: writing smithi10729407-2 from 2023424 to 2719744 tid 3 2020-06-12T02:38:43.610 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:43.610 INFO:tasks.rados.rados.0.smithi107.stdout:8856: seq_num 8855 ranges {0=712704,712704=794624,1507328=409600} 2020-06-12T02:38:43.616 INFO:tasks.rados.rados.0.smithi107.stdout:8856: writing smithi10729407-4 from 0 to 712704 tid 1 2020-06-12T02:38:43.620 INFO:tasks.rados.rados.0.smithi107.stdout:8856: writing smithi10729407-4 from 712704 to 1507328 tid 2 2020-06-12T02:38:43.622 INFO:tasks.rados.rados.0.smithi107.stdout:8856: writing smithi10729407-4 from 1507328 to 1916928 tid 3 2020-06-12T02:38:43.623 INFO:tasks.rados.rados.0.smithi107.stdout:8857: delete oid 20 current snap is 0 2020-06-12T02:38:43.623 INFO:tasks.rados.rados.0.smithi107.stdout:8855: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:43.623 INFO:tasks.rados.rados.0.smithi107.stdout:8855: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:43.624 INFO:tasks.rados.rados.0.smithi107.stdout:8855: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:43.624 INFO:tasks.rados.rados.0.smithi107.stdout:8855: finishing write tid 5 to smithi10729407-2 2020-06-12T02:38:43.624 INFO:tasks.rados.rados.0.smithi107.stdout:8855: finishing write tid 6 to smithi10729407-2 2020-06-12T02:38:43.624 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 823 (ObjNum 8854 snap 0 seq_num 8854) dirty exists 2020-06-12T02:38:43.624 INFO:tasks.rados.rados.0.smithi107.stdout:8855: left oid 2 (ObjNum 8854 snap 0 seq_num 8854) 2020-06-12T02:38:43.625 INFO:tasks.rados.rados.0.smithi107.stdout:8856: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:43.625 INFO:tasks.rados.rados.0.smithi107.stdout:8855: done (2 left) 2020-06-12T02:38:43.625 INFO:tasks.rados.rados.0.smithi107.stdout:8857: done (1 left) 2020-06-12T02:38:43.626 INFO:tasks.rados.rados.0.smithi107.stdout:8858: delete oid 18 current snap is 0 2020-06-12T02:38:43.632 INFO:tasks.rados.rados.0.smithi107.stdout:8856: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:43.633 INFO:tasks.rados.rados.0.smithi107.stdout:8858: done (1 left) 2020-06-12T02:38:43.633 INFO:tasks.rados.rados.0.smithi107.stdout:8859: delete oid 7 current snap is 0 2020-06-12T02:38:43.633 INFO:tasks.rados.rados.0.smithi107.stdout:8856: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:43.636 INFO:tasks.rados.rados.0.smithi107.stdout:8856: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:43.637 INFO:tasks.rados.rados.0.smithi107.stdout:8856: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:43.638 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1667 (ObjNum 8855 snap 0 seq_num 8855) dirty exists 2020-06-12T02:38:43.638 INFO:tasks.rados.rados.0.smithi107.stdout:8856: left oid 4 (ObjNum 8855 snap 0 seq_num 8855) 2020-06-12T02:38:43.638 INFO:tasks.rados.rados.0.smithi107.stdout:8856: done (1 left) 2020-06-12T02:38:43.639 INFO:tasks.rados.rados.0.smithi107.stdout:8859: done (0 left) 2020-06-12T02:38:43.639 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:43.639 INFO:tasks.rados.rados.0.smithi107.stdout:8860: seq_num 8859 ranges {0=778240,778240=778240,1556480=425984} 2020-06-12T02:38:43.643 INFO:tasks.rados.rados.0.smithi107.stdout:8860: writing smithi10729407-15 from 0 to 778240 tid 1 2020-06-12T02:38:43.647 INFO:tasks.rados.rados.0.smithi107.stdout:8860: writing smithi10729407-15 from 778240 to 1556480 tid 2 2020-06-12T02:38:43.650 INFO:tasks.rados.rados.0.smithi107.stdout:8860: writing smithi10729407-15 from 1556480 to 1982464 tid 3 2020-06-12T02:38:43.650 INFO:tasks.rados.rados.0.smithi107.stdout:8861: delete oid 12 current snap is 0 2020-06-12T02:38:43.650 INFO:tasks.rados.rados.0.smithi107.stdout:8860: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:43.654 INFO:tasks.rados.rados.0.smithi107.stdout:8861: done (1 left) 2020-06-12T02:38:43.654 INFO:tasks.rados.rados.0.smithi107.stdout:8862: delete oid 9 current snap is 0 2020-06-12T02:38:43.659 INFO:tasks.rados.rados.0.smithi107.stdout:8862: done (1 left) 2020-06-12T02:38:43.660 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:38:43.660 INFO:tasks.rados.rados.0.smithi107.stdout:8863: seq_num 8862 ranges {1794048=425984,2220032=434176,2654208=737280,3391488=434176} 2020-06-12T02:38:43.670 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:43.668+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:43.671 INFO:tasks.rados.rados.0.smithi107.stdout:8863: writing smithi10729407-16 from 1794048 to 2220032 tid 1 2020-06-12T02:38:43.674 INFO:tasks.rados.rados.0.smithi107.stdout:8863: writing smithi10729407-16 from 2220032 to 2654208 tid 2 2020-06-12T02:38:43.679 INFO:tasks.rados.rados.0.smithi107.stdout:8863: writing smithi10729407-16 from 2654208 to 3391488 tid 3 2020-06-12T02:38:43.682 INFO:tasks.rados.rados.0.smithi107.stdout:8863: writing smithi10729407-16 from 3391488 to 3825664 tid 4 2020-06-12T02:38:43.682 INFO:tasks.rados.rados.0.smithi107.stdout:8864: delete oid 19 current snap is 0 2020-06-12T02:38:43.682 INFO:tasks.rados.rados.0.smithi107.stdout:8860: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:43.682 INFO:tasks.rados.rados.0.smithi107.stdout:8860: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:43.683 INFO:tasks.rados.rados.0.smithi107.stdout:8860: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:43.683 INFO:tasks.rados.rados.0.smithi107.stdout:8860: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:43.684 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1650 (ObjNum 8859 snap 0 seq_num 8859) dirty exists 2020-06-12T02:38:43.684 INFO:tasks.rados.rados.0.smithi107.stdout:8860: left oid 15 (ObjNum 8859 snap 0 seq_num 8859) 2020-06-12T02:38:43.684 INFO:tasks.rados.rados.0.smithi107.stdout:8863: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:43.684 INFO:tasks.rados.rados.0.smithi107.stdout:8860: done (2 left) 2020-06-12T02:38:43.685 INFO:tasks.rados.rados.0.smithi107.stdout:8864: done (1 left) 2020-06-12T02:38:43.685 INFO:tasks.rados.rados.0.smithi107.stdout:8865: delete oid 1 current snap is 0 2020-06-12T02:38:43.687 INFO:tasks.rados.rados.0.smithi107.stdout:8865: done (1 left) 2020-06-12T02:38:43.688 INFO:tasks.rados.rados.0.smithi107.stdout:8866: delete oid 13 current snap is 0 2020-06-12T02:38:43.688 INFO:tasks.rados.rados.0.smithi107.stdout:8863: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:43.689 INFO:tasks.rados.rados.0.smithi107.stdout:8863: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:43.689 INFO:tasks.rados.rados.0.smithi107.stdout:8863: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:43.689 INFO:tasks.rados.rados.0.smithi107.stdout:8863: finishing write tid 6 to smithi10729407-16 2020-06-12T02:38:43.690 INFO:tasks.rados.rados.0.smithi107.stdout:8866: done (1 left) 2020-06-12T02:38:43.691 INFO:tasks.rados.rados.0.smithi107.stdout:8867: delete oid 9 current snap is 0 2020-06-12T02:38:43.692 INFO:tasks.rados.rados.0.smithi107.stdout:8863: finishing write tid 7 to smithi10729407-16 2020-06-12T02:38:43.693 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1600 (ObjNum 8862 snap 0 seq_num 8862) dirty exists 2020-06-12T02:38:43.693 INFO:tasks.rados.rados.0.smithi107.stdout:8863: left oid 16 (ObjNum 8862 snap 0 seq_num 8862) 2020-06-12T02:38:43.693 INFO:tasks.rados.rados.0.smithi107.stdout:8863: done (1 left) 2020-06-12T02:38:43.694 INFO:tasks.rados.rados.0.smithi107.stdout:8867: done (0 left) 2020-06-12T02:38:43.694 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:43.694 INFO:tasks.rados.rados.0.smithi107.stdout:8868: seq_num 8867 ranges {1458176=507904,1966080=704512,2670592=475136,3145728=425984} 2020-06-12T02:38:43.695 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:43.695 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:43.706 INFO:tasks.rados.rados.0.smithi107.stdout:8868: writing smithi10729407-8 from 1458176 to 1966080 tid 1 2020-06-12T02:38:43.706 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-06-12T02:38:43.708 INFO:tasks.rados.rados.0.smithi107.stdout:8868: writing smithi10729407-8 from 1966080 to 2670592 tid 2 2020-06-12T02:38:43.711 INFO:tasks.rados.rados.0.smithi107.stdout:8868: writing smithi10729407-8 from 2670592 to 3145728 tid 3 2020-06-12T02:38:43.713 INFO:tasks.rados.rados.0.smithi107.stdout:8868: writing smithi10729407-8 from 3145728 to 3571712 tid 4 2020-06-12T02:38:43.714 INFO:tasks.rados.rados.0.smithi107.stdout:8869: delete oid 2 current snap is 0 2020-06-12T02:38:43.714 INFO:tasks.rados.rados.0.smithi107.stdout:8868: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:43.718 INFO:tasks.rados.rados.0.smithi107.stdout:8869: done (1 left) 2020-06-12T02:38:43.718 INFO:tasks.rados.rados.0.smithi107.stdout:8870: delete oid 10 current snap is 0 2020-06-12T02:38:43.721 INFO:tasks.rados.rados.0.smithi107.stdout:8870: done (1 left) 2020-06-12T02:38:43.722 INFO:tasks.rados.rados.0.smithi107.stdout:8871: delete oid 10 current snap is 0 2020-06-12T02:38:43.724 INFO:tasks.rados.rados.0.smithi107.stdout:8868: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:43.724 INFO:tasks.rados.rados.0.smithi107.stdout:8868: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:43.727 INFO:tasks.rados.rados.0.smithi107.stdout:8871: done (1 left) 2020-06-12T02:38:43.727 INFO:tasks.rados.rados.0.smithi107.stdout:8872: delete oid 14 current snap is 0 2020-06-12T02:38:43.728 INFO:tasks.rados.rados.0.smithi107.stdout:8868: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:43.728 INFO:tasks.rados.rados.0.smithi107.stdout:8868: finishing write tid 6 to smithi10729407-8 2020-06-12T02:38:43.729 INFO:tasks.rados.rados.0.smithi107.stdout:8868: finishing write tid 7 to smithi10729407-8 2020-06-12T02:38:43.729 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1655 (ObjNum 8867 snap 0 seq_num 8867) dirty exists 2020-06-12T02:38:43.730 INFO:tasks.rados.rados.0.smithi107.stdout:8868: left oid 8 (ObjNum 8867 snap 0 seq_num 8867) 2020-06-12T02:38:43.738 INFO:tasks.rados.rados.0.smithi107.stdout:8868: done (1 left) 2020-06-12T02:38:43.739 INFO:tasks.rados.rados.0.smithi107.stdout:8872: done (0 left) 2020-06-12T02:38:43.739 INFO:tasks.rados.rados.0.smithi107.stdout:8873: delete oid 11 current snap is 0 2020-06-12T02:38:43.739 INFO:tasks.rados.rados.0.smithi107.stdout:8873: done (0 left) 2020-06-12T02:38:43.739 INFO:tasks.rados.rados.0.smithi107.stdout:8874: delete oid 10 current snap is 0 2020-06-12T02:38:43.739 INFO:tasks.rados.rados.0.smithi107.stdout:8874: done (0 left) 2020-06-12T02:38:43.740 INFO:tasks.rados.rados.0.smithi107.stdout:8875: delete oid 14 current snap is 0 2020-06-12T02:38:43.740 INFO:tasks.rados.rados.0.smithi107.stdout:8875: done (0 left) 2020-06-12T02:38:43.740 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:43.741 INFO:tasks.rados.rados.0.smithi107.stdout:8876: seq_num 8875 ranges {0=647168,647168=630784,1277952=442368,1720320=532480} 2020-06-12T02:38:43.743 INFO:tasks.rados.rados.0.smithi107.stdout:8876: writing smithi10729407-1 from 0 to 647168 tid 1 2020-06-12T02:38:43.747 INFO:tasks.rados.rados.0.smithi107.stdout:8876: writing smithi10729407-1 from 647168 to 1277952 tid 2 2020-06-12T02:38:43.750 INFO:tasks.rados.rados.0.smithi107.stdout:8876: writing smithi10729407-1 from 1277952 to 1720320 tid 3 2020-06-12T02:38:43.753 INFO:tasks.rados.rados.0.smithi107.stdout:8876: writing smithi10729407-1 from 1720320 to 2252800 tid 4 2020-06-12T02:38:43.753 INFO:tasks.rados.rados.0.smithi107.stdout:8877: delete oid 18 current snap is 0 2020-06-12T02:38:43.753 INFO:tasks.rados.rados.0.smithi107.stdout:8876: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:43.756 INFO:tasks.rados.rados.0.smithi107.stdout:8877: done (1 left) 2020-06-12T02:38:43.757 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:38:43.757 INFO:tasks.rados.rados.0.smithi107.stdout:8878: seq_num 8877 ranges {1646592=679936,2326528=794624,3121152=450560,3571712=360448} 2020-06-12T02:38:43.770 INFO:tasks.rados.rados.0.smithi107.stdout:8878: writing smithi10729407-3 from 1646592 to 2326528 tid 1 2020-06-12T02:38:43.774 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:43.771+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:43.774 INFO:tasks.rados.rados.0.smithi107.stdout:8878: writing smithi10729407-3 from 2326528 to 3121152 tid 2 2020-06-12T02:38:43.777 INFO:tasks.rados.rados.0.smithi107.stdout:8878: writing smithi10729407-3 from 3121152 to 3571712 tid 3 2020-06-12T02:38:43.780 INFO:tasks.rados.rados.0.smithi107.stdout:8878: writing smithi10729407-3 from 3571712 to 3932160 tid 4 2020-06-12T02:38:43.780 INFO:tasks.rados.rados.0.smithi107.stdout:8879: delete oid 17 current snap is 0 2020-06-12T02:38:43.780 INFO:tasks.rados.rados.0.smithi107.stdout:8876: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:43.780 INFO:tasks.rados.rados.0.smithi107.stdout:8876: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:43.781 INFO:tasks.rados.rados.0.smithi107.stdout:8876: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:43.781 INFO:tasks.rados.rados.0.smithi107.stdout:8876: finishing write tid 6 to smithi10729407-1 2020-06-12T02:38:43.781 INFO:tasks.rados.rados.0.smithi107.stdout:8876: finishing write tid 7 to smithi10729407-1 2020-06-12T02:38:43.781 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1609 (ObjNum 8875 snap 0 seq_num 8875) dirty exists 2020-06-12T02:38:43.781 INFO:tasks.rados.rados.0.smithi107.stdout:8876: left oid 1 (ObjNum 8875 snap 0 seq_num 8875) 2020-06-12T02:38:43.783 INFO:tasks.rados.rados.0.smithi107.stdout:8878: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:43.786 INFO:tasks.rados.rados.0.smithi107.stdout:8876: done (2 left) 2020-06-12T02:38:43.786 INFO:tasks.rados.rados.0.smithi107.stdout:8879: done (1 left) 2020-06-12T02:38:43.786 INFO:tasks.rados.rados.0.smithi107.stdout:8880: delete oid 4 current snap is 0 2020-06-12T02:38:43.787 INFO:tasks.rados.rados.0.smithi107.stdout:8878: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:43.788 INFO:tasks.rados.rados.0.smithi107.stdout:8880: done (1 left) 2020-06-12T02:38:43.788 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:43.788 INFO:tasks.rados.rados.0.smithi107.stdout:8881: seq_num 8880 ranges {1007616=729088,1736704=499712,2236416=630784,2867200=475136} 2020-06-12T02:38:43.798 INFO:tasks.rados.rados.0.smithi107.stdout:8881: writing smithi10729407-6 from 1007616 to 1736704 tid 1 2020-06-12T02:38:43.801 INFO:tasks.rados.rados.0.smithi107.stdout:8881: writing smithi10729407-6 from 1736704 to 2236416 tid 2 2020-06-12T02:38:43.804 INFO:tasks.rados.rados.0.smithi107.stdout:8881: writing smithi10729407-6 from 2236416 to 2867200 tid 3 2020-06-12T02:38:43.807 INFO:tasks.rados.rados.0.smithi107.stdout:8881: writing smithi10729407-6 from 2867200 to 3342336 tid 4 2020-06-12T02:38:43.808 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:43.808 INFO:tasks.rados.rados.0.smithi107.stdout:8882: seq_num 8881 ranges {0=745472,745472=786432,1531904=434176,1966080=385024} 2020-06-12T02:38:43.812 INFO:tasks.rados.rados.0.smithi107.stdout:8882: writing smithi10729407-2 from 0 to 745472 tid 1 2020-06-12T02:38:43.817 INFO:tasks.rados.rados.0.smithi107.stdout:8882: writing smithi10729407-2 from 745472 to 1531904 tid 2 2020-06-12T02:38:43.819 INFO:tasks.rados.rados.0.smithi107.stdout:8882: writing smithi10729407-2 from 1531904 to 1966080 tid 3 2020-06-12T02:38:43.822 INFO:tasks.rados.rados.0.smithi107.stdout:8882: writing smithi10729407-2 from 1966080 to 2351104 tid 4 2020-06-12T02:38:43.822 INFO:tasks.rados.rados.0.smithi107.stdout:8883: delete oid 11 current snap is 0 2020-06-12T02:38:43.822 INFO:tasks.rados.rados.0.smithi107.stdout:8878: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:43.822 INFO:tasks.rados.rados.0.smithi107.stdout:8878: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:43.823 INFO:tasks.rados.rados.0.smithi107.stdout:8878: finishing write tid 6 to smithi10729407-3 2020-06-12T02:38:43.823 INFO:tasks.rados.rados.0.smithi107.stdout:8878: finishing write tid 7 to smithi10729407-3 2020-06-12T02:38:43.823 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1605 (ObjNum 8877 snap 0 seq_num 8877) dirty exists 2020-06-12T02:38:43.823 INFO:tasks.rados.rados.0.smithi107.stdout:8878: left oid 3 (ObjNum 8877 snap 0 seq_num 8877) 2020-06-12T02:38:43.823 INFO:tasks.rados.rados.0.smithi107.stdout:8881: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:43.824 INFO:tasks.rados.rados.0.smithi107.stdout:8881: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:43.824 INFO:tasks.rados.rados.0.smithi107.stdout:8881: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:43.824 INFO:tasks.rados.rados.0.smithi107.stdout:8881: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:43.824 INFO:tasks.rados.rados.0.smithi107.stdout:8881: finishing write tid 6 to smithi10729407-6 2020-06-12T02:38:43.825 INFO:tasks.rados.rados.0.smithi107.stdout:8882: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:43.825 INFO:tasks.rados.rados.0.smithi107.stdout:8881: finishing write tid 7 to smithi10729407-6 2020-06-12T02:38:43.825 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1614 (ObjNum 8880 snap 0 seq_num 8880) dirty exists 2020-06-12T02:38:43.825 INFO:tasks.rados.rados.0.smithi107.stdout:8881: left oid 6 (ObjNum 8880 snap 0 seq_num 8880) 2020-06-12T02:38:43.826 INFO:tasks.rados.rados.0.smithi107.stdout:8878: done (3 left) 2020-06-12T02:38:43.826 INFO:tasks.rados.rados.0.smithi107.stdout:8881: done (2 left) 2020-06-12T02:38:43.827 INFO:tasks.rados.rados.0.smithi107.stdout:8883: done (1 left) 2020-06-12T02:38:43.827 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:43.827 INFO:tasks.rados.rados.0.smithi107.stdout:8884: seq_num 8883 ranges {0=778240,778240=548864,1327104=450560,1777664=606208} 2020-06-12T02:38:43.830 INFO:tasks.rados.rados.0.smithi107.stdout:8884: writing smithi10729407-14 from 0 to 778240 tid 1 2020-06-12T02:38:43.833 INFO:tasks.rados.rados.0.smithi107.stdout:8884: writing smithi10729407-14 from 778240 to 1327104 tid 2 2020-06-12T02:38:43.837 INFO:tasks.rados.rados.0.smithi107.stdout:8884: writing smithi10729407-14 from 1327104 to 1777664 tid 3 2020-06-12T02:38:43.841 INFO:tasks.rados.rados.0.smithi107.stdout:8884: writing smithi10729407-14 from 1777664 to 2383872 tid 4 2020-06-12T02:38:43.841 INFO:tasks.rados.rados.0.smithi107.stdout:8885: delete oid 19 current snap is 0 2020-06-12T02:38:43.842 INFO:tasks.rados.rados.0.smithi107.stdout:8882: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:43.842 INFO:tasks.rados.rados.0.smithi107.stdout:8882: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:43.842 INFO:tasks.rados.rados.0.smithi107.stdout:8882: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:43.843 INFO:tasks.rados.rados.0.smithi107.stdout:8882: finishing write tid 6 to smithi10729407-2 2020-06-12T02:38:43.844 INFO:tasks.rados.rados.0.smithi107.stdout:8882: finishing write tid 7 to smithi10729407-2 2020-06-12T02:38:43.844 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 829 (ObjNum 8881 snap 0 seq_num 8881) dirty exists 2020-06-12T02:38:43.844 INFO:tasks.rados.rados.0.smithi107.stdout:8882: left oid 2 (ObjNum 8881 snap 0 seq_num 8881) 2020-06-12T02:38:43.844 INFO:tasks.rados.rados.0.smithi107.stdout:8884: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:43.845 INFO:tasks.rados.rados.0.smithi107.stdout:8882: done (2 left) 2020-06-12T02:38:43.845 INFO:tasks.rados.rados.0.smithi107.stdout:8885: done (1 left) 2020-06-12T02:38:43.845 INFO:tasks.rados.rados.0.smithi107.stdout:8886: delete oid 16 current snap is 0 2020-06-12T02:38:43.848 INFO:tasks.rados.rados.0.smithi107.stdout:8884: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:43.851 INFO:tasks.rados.rados.0.smithi107.stdout:8886: done (1 left) 2020-06-12T02:38:43.851 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:43.851 INFO:tasks.rados.rados.0.smithi107.stdout:8887: seq_num 8886 ranges {3571712=24576} 2020-06-12T02:38:43.854 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:43.854 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:43.864 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-06-12T02:38:43.869 INFO:tasks.rados.rados.0.smithi107.stdout:8887: writing smithi10729407-8 from 3571712 to 3596288 tid 1 2020-06-12T02:38:43.870 INFO:tasks.rados.rados.0.smithi107.stdout:8888: delete oid 12 current snap is 0 2020-06-12T02:38:43.870 INFO:tasks.rados.rados.0.smithi107.stdout:8884: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:43.870 INFO:tasks.rados.rados.0.smithi107.stdout:8884: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:43.870 INFO:tasks.rados.rados.0.smithi107.stdout:8884: finishing write tid 6 to smithi10729407-14 2020-06-12T02:38:43.870 INFO:tasks.rados.rados.0.smithi107.stdout:8884: finishing write tid 7 to smithi10729407-14 2020-06-12T02:38:43.876 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1554 (ObjNum 8883 snap 0 seq_num 8883) dirty exists 2020-06-12T02:38:43.876 INFO:tasks.rados.rados.0.smithi107.stdout:8884: left oid 14 (ObjNum 8883 snap 0 seq_num 8883) 2020-06-12T02:38:43.878 INFO:tasks.rados.rados.0.smithi107.stdout:8884: done (2 left) 2020-06-12T02:38:43.878 INFO:tasks.rados.rados.0.smithi107.stdout:8888: done (1 left) 2020-06-12T02:38:43.878 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:43.879 INFO:tasks.rados.rados.0.smithi107.stdout:8889: seq_num 8888 ranges {0=57344} 2020-06-12T02:38:43.879 INFO:tasks.rados.rados.0.smithi107.stdout:8889: writing smithi10729407-13 from 0 to 57344 tid 1 2020-06-12T02:38:43.879 INFO:tasks.rados.rados.0.smithi107.stdout:8890: delete oid 6 current snap is 0 2020-06-12T02:38:43.880 INFO:tasks.rados.rados.0.smithi107.stdout:8887: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:43.880 INFO:tasks.rados.rados.0.smithi107.stdout:8887: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:43.880 INFO:tasks.rados.rados.0.smithi107.stdout:8889: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:43.880 INFO:tasks.rados.rados.0.smithi107.stdout:8887: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:43.880 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1657 (ObjNum 8886 snap 0 seq_num 8886) dirty exists 2020-06-12T02:38:43.880 INFO:tasks.rados.rados.0.smithi107.stdout:8887: left oid 8 (ObjNum 8886 snap 0 seq_num 8886) 2020-06-12T02:38:43.881 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:43.875+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:43.882 INFO:tasks.rados.rados.0.smithi107.stdout:8887: done (2 left) 2020-06-12T02:38:43.882 INFO:tasks.rados.rados.0.smithi107.stdout:8890: done (1 left) 2020-06-12T02:38:43.882 INFO:tasks.rados.rados.0.smithi107.stdout:8891: delete oid 1 current snap is 0 2020-06-12T02:38:43.882 INFO:tasks.rados.rados.0.smithi107.stdout:8889: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:43.883 INFO:tasks.rados.rados.0.smithi107.stdout:8889: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:43.884 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1601 (ObjNum 8888 snap 0 seq_num 8888) dirty exists 2020-06-12T02:38:43.884 INFO:tasks.rados.rados.0.smithi107.stdout:8889: left oid 13 (ObjNum 8888 snap 0 seq_num 8888) 2020-06-12T02:38:43.884 INFO:tasks.rados.rados.0.smithi107.stdout:8889: done (1 left) 2020-06-12T02:38:43.884 INFO:tasks.rados.rados.0.smithi107.stdout:8891: done (0 left) 2020-06-12T02:38:43.885 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:38:43.885 INFO:tasks.rados.rados.0.smithi107.stdout:8892: seq_num 8891 ranges {0=114688} 2020-06-12T02:38:43.885 INFO:tasks.rados.rados.0.smithi107.stdout:8892: writing smithi10729407-10 from 0 to 114688 tid 1 2020-06-12T02:38:43.886 INFO:tasks.rados.rados.0.smithi107.stdout:8893: delete oid 15 current snap is 0 2020-06-12T02:38:43.887 INFO:tasks.rados.rados.0.smithi107.stdout:8893: done (1 left) 2020-06-12T02:38:43.888 INFO:tasks.rados.rados.0.smithi107.stdout:8894: delete oid 16 current snap is 0 2020-06-12T02:38:43.888 INFO:tasks.rados.rados.0.smithi107.stdout:8892: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:43.891 INFO:tasks.rados.rados.0.smithi107.stdout:8894: done (1 left) 2020-06-12T02:38:43.891 INFO:tasks.rados.rados.0.smithi107.stdout:8895: delete oid 18 current snap is 0 2020-06-12T02:38:43.893 INFO:tasks.rados.rados.0.smithi107.stdout:8895: done (1 left) 2020-06-12T02:38:43.893 INFO:tasks.rados.rados.0.smithi107.stdout:8896: delete oid 19 current snap is 0 2020-06-12T02:38:43.895 INFO:tasks.rados.rados.0.smithi107.stdout:8892: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:43.896 INFO:tasks.rados.rados.0.smithi107.stdout:8896: done (1 left) 2020-06-12T02:38:43.896 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:43.897 INFO:tasks.rados.rados.0.smithi107.stdout:8897: seq_num 8896 ranges {0=196608} 2020-06-12T02:38:43.897 INFO:tasks.rados.rados.0.smithi107.stdout:8897: writing smithi10729407-4 from 0 to 196608 tid 1 2020-06-12T02:38:43.898 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:38:43.898 INFO:tasks.rados.rados.0.smithi107.stdout:8898: seq_num 8897 ranges {0=212992} 2020-06-12T02:38:43.898 INFO:tasks.rados.rados.0.smithi107.stdout:8898: writing smithi10729407-16 from 0 to 212992 tid 1 2020-06-12T02:38:43.899 INFO:tasks.rados.rados.0.smithi107.stdout:8899: delete oid 11 current snap is 0 2020-06-12T02:38:43.899 INFO:tasks.rados.rados.0.smithi107.stdout:8892: finishing write tid 4 to smithi10729407-10 2020-06-12T02:38:43.899 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 645 (ObjNum 8891 snap 0 seq_num 8891) dirty exists 2020-06-12T02:38:43.900 INFO:tasks.rados.rados.0.smithi107.stdout:8892: left oid 10 (ObjNum 8891 snap 0 seq_num 8891) 2020-06-12T02:38:43.901 INFO:tasks.rados.rados.0.smithi107.stdout:8897: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:43.904 INFO:tasks.rados.rados.0.smithi107.stdout:8897: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:43.904 INFO:tasks.rados.rados.0.smithi107.stdout:8892: done (3 left) 2020-06-12T02:38:43.904 INFO:tasks.rados.rados.0.smithi107.stdout:8899: done (2 left) 2020-06-12T02:38:43.904 INFO:tasks.rados.rados.0.smithi107.stdout:8900: delete oid 13 current snap is 0 2020-06-12T02:38:43.905 INFO:tasks.rados.rados.0.smithi107.stdout:8897: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:43.906 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1671 (ObjNum 8896 snap 0 seq_num 8896) dirty exists 2020-06-12T02:38:43.906 INFO:tasks.rados.rados.0.smithi107.stdout:8897: left oid 4 (ObjNum 8896 snap 0 seq_num 8896) 2020-06-12T02:38:43.907 INFO:tasks.rados.rados.0.smithi107.stdout:8898: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:43.907 INFO:tasks.rados.rados.0.smithi107.stdout:8898: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:43.907 INFO:tasks.rados.rados.0.smithi107.stdout:8897: done (2 left) 2020-06-12T02:38:43.908 INFO:tasks.rados.rados.0.smithi107.stdout:8900: done (1 left) 2020-06-12T02:38:43.908 INFO:tasks.rados.rados.0.smithi107.stdout:8901: delete oid 1 current snap is 0 2020-06-12T02:38:43.909 INFO:tasks.rados.rados.0.smithi107.stdout:8898: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:43.909 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1609 (ObjNum 8897 snap 0 seq_num 8897) dirty exists 2020-06-12T02:38:43.909 INFO:tasks.rados.rados.0.smithi107.stdout:8898: left oid 16 (ObjNum 8897 snap 0 seq_num 8897) 2020-06-12T02:38:43.910 INFO:tasks.rados.rados.0.smithi107.stdout:8898: done (1 left) 2020-06-12T02:38:43.911 INFO:tasks.rados.rados.0.smithi107.stdout:8901: done (0 left) 2020-06-12T02:38:43.911 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:43.911 INFO:tasks.rados.rados.0.smithi107.stdout:8902: seq_num 8901 ranges {2383872=278528} 2020-06-12T02:38:43.924 INFO:tasks.rados.rados.0.smithi107.stdout:8902: writing smithi10729407-14 from 2383872 to 2662400 tid 1 2020-06-12T02:38:43.924 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:43.925 INFO:tasks.rados.rados.0.smithi107.stdout:8903: seq_num 8902 ranges {0=294912} 2020-06-12T02:38:43.926 INFO:tasks.rados.rados.0.smithi107.stdout:8903: writing smithi10729407-11 from 0 to 294912 tid 1 2020-06-12T02:38:43.926 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:43.926 INFO:tasks.rados.rados.0.smithi107.stdout:8904: seq_num 8903 ranges {0=311296} 2020-06-12T02:38:43.928 INFO:tasks.rados.rados.0.smithi107.stdout:8904: writing smithi10729407-17 from 0 to 311296 tid 1 2020-06-12T02:38:43.935 INFO:tasks.rados.rados.0.smithi107.stdout:8905: delete oid 3 current snap is 0 2020-06-12T02:38:43.935 INFO:tasks.rados.rados.0.smithi107.stdout:8902: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:43.935 INFO:tasks.rados.rados.0.smithi107.stdout:8902: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:43.936 INFO:tasks.rados.rados.0.smithi107.stdout:8903: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:43.936 INFO:tasks.rados.rados.0.smithi107.stdout:8904: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:43.936 INFO:tasks.rados.rados.0.smithi107.stdout:8903: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:43.937 INFO:tasks.rados.rados.0.smithi107.stdout:8904: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:43.937 INFO:tasks.rados.rados.0.smithi107.stdout:8902: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:43.937 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1557 (ObjNum 8901 snap 0 seq_num 8901) dirty exists 2020-06-12T02:38:43.937 INFO:tasks.rados.rados.0.smithi107.stdout:8902: left oid 14 (ObjNum 8901 snap 0 seq_num 8901) 2020-06-12T02:38:43.938 INFO:tasks.rados.rados.0.smithi107.stdout:8902: done (3 left) 2020-06-12T02:38:43.938 INFO:tasks.rados.rados.0.smithi107.stdout:8905: done (2 left) 2020-06-12T02:38:43.939 INFO:tasks.rados.rados.0.smithi107.stdout:8906: delete oid 12 current snap is 0 2020-06-12T02:38:43.939 INFO:tasks.rados.rados.0.smithi107.stdout:8904: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:43.939 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1673 (ObjNum 8903 snap 0 seq_num 8903) dirty exists 2020-06-12T02:38:43.940 INFO:tasks.rados.rados.0.smithi107.stdout:8904: left oid 17 (ObjNum 8903 snap 0 seq_num 8903) 2020-06-12T02:38:43.940 INFO:tasks.rados.rados.0.smithi107.stdout:8903: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:43.940 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1559 (ObjNum 8902 snap 0 seq_num 8902) dirty exists 2020-06-12T02:38:43.940 INFO:tasks.rados.rados.0.smithi107.stdout:8903: left oid 11 (ObjNum 8902 snap 0 seq_num 8902) 2020-06-12T02:38:43.941 INFO:tasks.rados.rados.0.smithi107.stdout:8903: done (2 left) 2020-06-12T02:38:43.941 INFO:tasks.rados.rados.0.smithi107.stdout:8904: done (1 left) 2020-06-12T02:38:43.941 INFO:tasks.rados.rados.0.smithi107.stdout:8906: done (0 left) 2020-06-12T02:38:43.942 INFO:tasks.rados.rados.0.smithi107.stdout:8907: delete oid 10 current snap is 0 2020-06-12T02:38:43.943 INFO:tasks.rados.rados.0.smithi107.stdout:8907: done (0 left) 2020-06-12T02:38:43.943 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:38:43.945 INFO:tasks.rados.rados.0.smithi107.stdout:8908: seq_num 8907 ranges {212992=376832} 2020-06-12T02:38:43.946 INFO:tasks.rados.rados.0.smithi107.stdout:8908: writing smithi10729407-16 from 212992 to 589824 tid 1 2020-06-12T02:38:43.946 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:43.946 INFO:tasks.rados.rados.0.smithi107.stdout:8909: seq_num 8908 ranges {0=393216} 2020-06-12T02:38:43.950 INFO:tasks.rados.rados.0.smithi107.stdout:8909: writing smithi10729407-6 from 0 to 393216 tid 1 2020-06-12T02:38:43.950 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:43.950 INFO:tasks.rados.rados.0.smithi107.stdout:8910: seq_num 8909 ranges {0=417792} 2020-06-12T02:38:43.951 INFO:tasks.rados.rados.0.smithi107.stdout:8910: writing smithi10729407-9 from 0 to 417792 tid 1 2020-06-12T02:38:43.952 INFO:tasks.rados.rados.0.smithi107.stdout:8911: delete oid 18 current snap is 0 2020-06-12T02:38:43.953 INFO:tasks.rados.rados.0.smithi107.stdout:8909: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:43.954 INFO:tasks.rados.rados.0.smithi107.stdout:8908: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:43.958 INFO:tasks.rados.rados.0.smithi107.stdout:8910: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:43.958 INFO:tasks.rados.rados.0.smithi107.stdout:8908: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:43.963 INFO:tasks.rados.rados.0.smithi107.stdout:8910: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:43.963 INFO:tasks.rados.rados.0.smithi107.stdout:8909: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:43.963 INFO:tasks.rados.rados.0.smithi107.stdout:8908: finishing write tid 4 to smithi10729407-16 2020-06-12T02:38:43.964 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1612 (ObjNum 8907 snap 0 seq_num 8907) dirty exists 2020-06-12T02:38:43.964 INFO:tasks.rados.rados.0.smithi107.stdout:8908: left oid 16 (ObjNum 8907 snap 0 seq_num 8907) 2020-06-12T02:38:43.964 INFO:tasks.rados.rados.0.smithi107.stdout:8908: done (3 left) 2020-06-12T02:38:43.964 INFO:tasks.rados.rados.0.smithi107.stdout:8911: done (2 left) 2020-06-12T02:38:43.965 INFO:tasks.rados.rados.0.smithi107.stdout:8912: delete oid 19 current snap is 0 2020-06-12T02:38:43.965 INFO:tasks.rados.rados.0.smithi107.stdout:8909: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:43.965 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1619 (ObjNum 8908 snap 0 seq_num 8908) dirty exists 2020-06-12T02:38:43.965 INFO:tasks.rados.rados.0.smithi107.stdout:8909: left oid 6 (ObjNum 8908 snap 0 seq_num 8908) 2020-06-12T02:38:43.965 INFO:tasks.rados.rados.0.smithi107.stdout:8910: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:43.966 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1669 (ObjNum 8909 snap 0 seq_num 8909) dirty exists 2020-06-12T02:38:43.966 INFO:tasks.rados.rados.0.smithi107.stdout:8910: left oid 9 (ObjNum 8909 snap 0 seq_num 8909) 2020-06-12T02:38:43.966 INFO:tasks.rados.rados.0.smithi107.stdout:8909: done (2 left) 2020-06-12T02:38:43.967 INFO:tasks.rados.rados.0.smithi107.stdout:8910: done (1 left) 2020-06-12T02:38:43.967 INFO:tasks.rados.rados.0.smithi107.stdout:8912: done (0 left) 2020-06-12T02:38:43.967 INFO:tasks.rados.rados.0.smithi107.stdout:8913: delete oid 17 current snap is 0 2020-06-12T02:38:43.967 INFO:tasks.rados.rados.0.smithi107.stdout:8913: done (0 left) 2020-06-12T02:38:43.967 INFO:tasks.rados.rados.0.smithi107.stdout:8914: delete oid 13 current snap is 0 2020-06-12T02:38:43.972 INFO:tasks.rados.rados.0.smithi107.stdout:8914: done (0 left) 2020-06-12T02:38:43.972 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:43.972 INFO:tasks.rados.rados.0.smithi107.stdout:8915: seq_num 8914 ranges {0=499712} 2020-06-12T02:38:43.972 INFO:tasks.rados.rados.0.smithi107.stdout:8915: writing smithi10729407-7 from 0 to 499712 tid 1 2020-06-12T02:38:43.973 INFO:tasks.rados.rados.0.smithi107.stdout:8916: delete oid 8 current snap is 0 2020-06-12T02:38:43.975 INFO:tasks.rados.rados.0.smithi107.stdout:8916: done (1 left) 2020-06-12T02:38:43.975 INFO:tasks.rados.rados.0.smithi107.stdout:8917: delete oid 2 current snap is 0 2020-06-12T02:38:43.978 INFO:tasks.rados.rados.0.smithi107.stdout:8917: done (1 left) 2020-06-12T02:38:43.979 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:43.979 INFO:tasks.rados.rados.0.smithi107.stdout:8918: seq_num 8917 ranges {0=548864} 2020-06-12T02:38:43.987 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:43.976+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:43.988 INFO:tasks.rados.rados.0.smithi107.stdout:8918: writing smithi10729407-2 from 0 to 548864 tid 1 2020-06-12T02:38:43.988 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:38:43.988 INFO:tasks.rados.rados.0.smithi107.stdout:8919: seq_num 8918 ranges {0=565248} 2020-06-12T02:38:43.988 INFO:tasks.rados.rados.0.smithi107.stdout:8919: writing smithi10729407-20 from 0 to 565248 tid 1 2020-06-12T02:38:43.988 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:43.989 INFO:tasks.rados.rados.0.smithi107.stdout:8920: seq_num 8919 ranges {0=581632} 2020-06-12T02:38:43.989 INFO:tasks.rados.rados.0.smithi107.stdout:8920: writing smithi10729407-15 from 0 to 581632 tid 1 2020-06-12T02:38:43.990 INFO:tasks.rados.rados.0.smithi107.stdout:8921: delete oid 4 current snap is 0 2020-06-12T02:38:43.990 INFO:tasks.rados.rados.0.smithi107.stdout:8915: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:43.990 INFO:tasks.rados.rados.0.smithi107.stdout:8915: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:43.990 INFO:tasks.rados.rados.0.smithi107.stdout:8915: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:43.990 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1605 (ObjNum 8914 snap 0 seq_num 8914) dirty exists 2020-06-12T02:38:43.991 INFO:tasks.rados.rados.0.smithi107.stdout:8915: left oid 7 (ObjNum 8914 snap 0 seq_num 8914) 2020-06-12T02:38:43.991 INFO:tasks.rados.rados.0.smithi107.stdout:8918: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:43.991 INFO:tasks.rados.rados.0.smithi107.stdout:8918: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:43.991 INFO:tasks.rados.rados.0.smithi107.stdout:8918: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:43.992 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 832 (ObjNum 8917 snap 0 seq_num 8917) dirty exists 2020-06-12T02:38:43.992 INFO:tasks.rados.rados.0.smithi107.stdout:8918: left oid 2 (ObjNum 8917 snap 0 seq_num 8917) 2020-06-12T02:38:43.994 INFO:tasks.rados.rados.0.smithi107.stdout:8915: done (4 left) 2020-06-12T02:38:43.994 INFO:tasks.rados.rados.0.smithi107.stdout:8918: done (3 left) 2020-06-12T02:38:43.994 INFO:tasks.rados.rados.0.smithi107.stdout:8921: done (2 left) 2020-06-12T02:38:43.994 INFO:tasks.rados.rados.0.smithi107.stdout:8922: delete oid 19 current snap is 0 2020-06-12T02:38:43.997 INFO:tasks.rados.rados.0.smithi107.stdout:8919: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:43.997 INFO:tasks.rados.rados.0.smithi107.stdout:8919: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:43.998 INFO:tasks.rados.rados.0.smithi107.stdout:8922: done (2 left) 2020-06-12T02:38:43.998 INFO:tasks.rados.rados.0.smithi107.stdout:8923: delete oid 10 current snap is 0 2020-06-12T02:38:43.998 INFO:tasks.rados.rados.0.smithi107.stdout:8920: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:44.000 INFO:tasks.rados.rados.0.smithi107.stdout:8919: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:44.000 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 660 (ObjNum 8918 snap 0 seq_num 8918) dirty exists 2020-06-12T02:38:44.000 INFO:tasks.rados.rados.0.smithi107.stdout:8919: left oid 20 (ObjNum 8918 snap 0 seq_num 8918) 2020-06-12T02:38:44.001 INFO:tasks.rados.rados.0.smithi107.stdout:8919: done (2 left) 2020-06-12T02:38:44.001 INFO:tasks.rados.rados.0.smithi107.stdout:8923: done (1 left) 2020-06-12T02:38:44.001 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:38:44.001 INFO:tasks.rados.rados.0.smithi107.stdout:8924: seq_num 8923 ranges {0=647168} 2020-06-12T02:38:44.004 INFO:tasks.rados.rados.0.smithi107.stdout:8924: writing smithi10729407-13 from 0 to 647168 tid 1 2020-06-12T02:38:44.005 INFO:tasks.rados.rados.0.smithi107.stdout:8925: delete oid 11 current snap is 0 2020-06-12T02:38:44.005 INFO:tasks.rados.rados.0.smithi107.stdout:8920: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:44.005 INFO:tasks.rados.rados.0.smithi107.stdout:8920: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:44.005 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1661 (ObjNum 8919 snap 0 seq_num 8919) dirty exists 2020-06-12T02:38:44.005 INFO:tasks.rados.rados.0.smithi107.stdout:8920: left oid 15 (ObjNum 8919 snap 0 seq_num 8919) 2020-06-12T02:38:44.007 INFO:tasks.rados.rados.0.smithi107.stdout:8920: done (2 left) 2020-06-12T02:38:44.007 INFO:tasks.rados.rados.0.smithi107.stdout:8925: done (1 left) 2020-06-12T02:38:44.007 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:44.007 INFO:tasks.rados.rados.0.smithi107.stdout:8926: seq_num 8925 ranges {417792=679936} 2020-06-12T02:38:44.014 INFO:tasks.rados.rados.0.smithi107.stdout:8926: writing smithi10729407-9 from 417792 to 1097728 tid 1 2020-06-12T02:38:44.014 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:44.014 INFO:tasks.rados.rados.0.smithi107.stdout:8927: seq_num 8926 ranges {0=696320} 2020-06-12T02:38:44.018 INFO:tasks.rados.rados.0.smithi107.stdout:8927: writing smithi10729407-17 from 0 to 696320 tid 1 2020-06-12T02:38:44.018 INFO:tasks.rados.rados.0.smithi107.stdout:8928: delete oid 2 current snap is 0 2020-06-12T02:38:44.018 INFO:tasks.rados.rados.0.smithi107.stdout:8924: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:44.019 INFO:tasks.rados.rados.0.smithi107.stdout:8924: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:44.019 INFO:tasks.rados.rados.0.smithi107.stdout:8924: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:44.019 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1607 (ObjNum 8923 snap 0 seq_num 8923) dirty exists 2020-06-12T02:38:44.019 INFO:tasks.rados.rados.0.smithi107.stdout:8924: left oid 13 (ObjNum 8923 snap 0 seq_num 8923) 2020-06-12T02:38:44.020 INFO:tasks.rados.rados.0.smithi107.stdout:8924: done (3 left) 2020-06-12T02:38:44.021 INFO:tasks.rados.rados.0.smithi107.stdout:8928: done (2 left) 2020-06-12T02:38:44.021 INFO:tasks.rados.rados.0.smithi107.stdout:8929: delete oid 18 current snap is 0 2020-06-12T02:38:44.021 INFO:tasks.rados.rados.0.smithi107.stdout:8926: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:44.022 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:44.022 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:44.026 INFO:tasks.rados.rados.0.smithi107.stdout:8927: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:44.029 INFO:tasks.rados.rados.0.smithi107.stdout:8927: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:44.029 INFO:tasks.rados.rados.0.smithi107.stdout:8926: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:44.030 INFO:tasks.rados.rados.0.smithi107.stdout:8929: done (2 left) 2020-06-12T02:38:44.031 INFO:tasks.rados.rados.0.smithi107.stdout:8930: delete oid 4 current snap is 0 2020-06-12T02:38:44.031 INFO:tasks.rados.rados.0.smithi107.stdout:8926: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:44.032 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1672 (ObjNum 8925 snap 0 seq_num 8925) dirty exists 2020-06-12T02:38:44.032 INFO:tasks.rados.rados.0.smithi107.stdout:8926: left oid 9 (ObjNum 8925 snap 0 seq_num 8925) 2020-06-12T02:38:44.033 INFO:tasks.rados.rados.0.smithi107.stdout:8927: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:44.033 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1677 (ObjNum 8926 snap 0 seq_num 8926) dirty exists 2020-06-12T02:38:44.033 INFO:tasks.rados.rados.0.smithi107.stdout:8927: left oid 17 (ObjNum 8926 snap 0 seq_num 8926) 2020-06-12T02:38:44.034 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-06-12T02:38:44.035 INFO:tasks.rados.rados.0.smithi107.stdout:8926: done (2 left) 2020-06-12T02:38:44.035 INFO:tasks.rados.rados.0.smithi107.stdout:8927: done (1 left) 2020-06-12T02:38:44.035 INFO:tasks.rados.rados.0.smithi107.stdout:8930: done (0 left) 2020-06-12T02:38:44.038 INFO:tasks.rados.rados.0.smithi107.stdout:8931: delete oid 20 current snap is 0 2020-06-12T02:38:44.039 INFO:tasks.rados.rados.0.smithi107.stdout:8931: done (0 left) 2020-06-12T02:38:44.039 INFO:tasks.rados.rados.0.smithi107.stdout:8932: delete oid 6 current snap is 0 2020-06-12T02:38:44.040 INFO:tasks.rados.rados.0.smithi107.stdout:8932: done (0 left) 2020-06-12T02:38:44.040 INFO:tasks.rados.rados.0.smithi107.stdout:8933: delete oid 1 current snap is 0 2020-06-12T02:38:44.045 INFO:tasks.rados.rados.0.smithi107.stdout:8933: done (0 left) 2020-06-12T02:38:44.045 INFO:tasks.rados.rados.0.smithi107.stdout:8934: delete oid 4 current snap is 0 2020-06-12T02:38:44.047 INFO:tasks.rados.rados.0.smithi107.stdout:8934: done (0 left) 2020-06-12T02:38:44.047 INFO:tasks.rados.rados.0.smithi107.stdout:8935: delete oid 20 current snap is 0 2020-06-12T02:38:44.049 INFO:tasks.rados.rados.0.smithi107.stdout:8935: done (0 left) 2020-06-12T02:38:44.049 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:38:44.049 INFO:tasks.rados.rados.0.smithi107.stdout:8936: seq_num 8935 ranges {0=450560,450560=401408} 2020-06-12T02:38:44.052 INFO:tasks.rados.rados.0.smithi107.stdout:8936: writing smithi10729407-1 from 0 to 450560 tid 1 2020-06-12T02:38:44.054 INFO:tasks.rados.rados.0.smithi107.stdout:8936: writing smithi10729407-1 from 450560 to 851968 tid 2 2020-06-12T02:38:44.054 INFO:tasks.rados.rados.0.smithi107.stdout:8937: delete oid 20 current snap is 0 2020-06-12T02:38:44.056 INFO:tasks.rados.rados.0.smithi107.stdout:8937: done (1 left) 2020-06-12T02:38:44.057 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:44.057 INFO:tasks.rados.rados.0.smithi107.stdout:8938: seq_num 8937 ranges {0=483328,483328=401408} 2020-06-12T02:38:44.059 INFO:tasks.rados.rados.0.smithi107.stdout:8938: writing smithi10729407-2 from 0 to 483328 tid 1 2020-06-12T02:38:44.064 INFO:tasks.rados.rados.0.smithi107.stdout:8938: writing smithi10729407-2 from 483328 to 884736 tid 2 2020-06-12T02:38:44.064 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:44.065 INFO:tasks.rados.rados.0.smithi107.stdout:8939: seq_num 8938 ranges {0=499712,499712=401408} 2020-06-12T02:38:44.066 INFO:tasks.rados.rados.0.smithi107.stdout:8939: writing smithi10729407-19 from 0 to 499712 tid 1 2020-06-12T02:38:44.069 INFO:tasks.rados.rados.0.smithi107.stdout:8939: writing smithi10729407-19 from 499712 to 901120 tid 2 2020-06-12T02:38:44.069 INFO:tasks.rados.rados.0.smithi107.stdout:8936: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:44.070 INFO:tasks.rados.rados.0.smithi107.stdout:8940: delete oid 6 current snap is 0 2020-06-12T02:38:44.070 INFO:tasks.rados.rados.0.smithi107.stdout:8936: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:44.070 INFO:tasks.rados.rados.0.smithi107.stdout:8936: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:44.070 INFO:tasks.rados.rados.0.smithi107.stdout:8938: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:44.070 INFO:tasks.rados.rados.0.smithi107.stdout:8936: finishing write tid 5 to smithi10729407-1 2020-06-12T02:38:44.071 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1624 (ObjNum 8935 snap 0 seq_num 8935) dirty exists 2020-06-12T02:38:44.071 INFO:tasks.rados.rados.0.smithi107.stdout:8936: left oid 1 (ObjNum 8935 snap 0 seq_num 8935) 2020-06-12T02:38:44.079 INFO:tasks.rados.rados.0.smithi107.stdout:8938: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:44.079 INFO:tasks.rados.rados.0.smithi107.stdout:8938: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:44.081 INFO:tasks.rados.rados.0.smithi107.stdout:8936: done (3 left) 2020-06-12T02:38:44.081 INFO:tasks.rados.rados.0.smithi107.stdout:8940: done (2 left) 2020-06-12T02:38:44.081 INFO:tasks.rados.rados.0.smithi107.stdout:8941: delete oid 14 current snap is 0 2020-06-12T02:38:44.081 INFO:tasks.rados.rados.0.smithi107.stdout:8938: finishing write tid 5 to smithi10729407-2 2020-06-12T02:38:44.082 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 836 (ObjNum 8937 snap 0 seq_num 8937) dirty exists 2020-06-12T02:38:44.082 INFO:tasks.rados.rados.0.smithi107.stdout:8938: left oid 2 (ObjNum 8937 snap 0 seq_num 8937) 2020-06-12T02:38:44.082 INFO:tasks.rados.rados.0.smithi107.stdout:8938: done (2 left) 2020-06-12T02:38:44.082 INFO:tasks.rados.rados.0.smithi107.stdout:8941: done (1 left) 2020-06-12T02:38:44.082 INFO:tasks.rados.rados.0.smithi107.stdout:8942: delete oid 6 current snap is 0 2020-06-12T02:38:44.083 INFO:tasks.rados.rados.0.smithi107.stdout:8939: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:44.083 INFO:tasks.rados.rados.0.smithi107.stdout:8939: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:44.083 INFO:tasks.rados.rados.0.smithi107.stdout:8939: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:44.083 INFO:tasks.rados.rados.0.smithi107.stdout:8939: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:44.083 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1655 (ObjNum 8938 snap 0 seq_num 8938) dirty exists 2020-06-12T02:38:44.083 INFO:tasks.rados.rados.0.smithi107.stdout:8939: left oid 19 (ObjNum 8938 snap 0 seq_num 8938) 2020-06-12T02:38:44.084 INFO:tasks.rados.rados.0.smithi107.stdout:8939: done (1 left) 2020-06-12T02:38:44.084 INFO:tasks.rados.rados.0.smithi107.stdout:8942: done (0 left) 2020-06-12T02:38:44.084 INFO:tasks.rados.rados.0.smithi107.stdout:8943: delete oid 13 current snap is 0 2020-06-12T02:38:44.085 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:44.077+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:44.085 INFO:tasks.rados.rados.0.smithi107.stdout:8943: done (0 left) 2020-06-12T02:38:44.085 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:44.086 INFO:tasks.rados.rados.0.smithi107.stdout:8944: seq_num 8943 ranges {0=581632,581632=401408} 2020-06-12T02:38:44.086 INFO:tasks.rados.rados.0.smithi107.stdout:8944: writing smithi10729407-8 from 0 to 581632 tid 1 2020-06-12T02:38:44.089 INFO:tasks.rados.rados.0.smithi107.stdout:8944: writing smithi10729407-8 from 581632 to 983040 tid 2 2020-06-12T02:38:44.089 INFO:tasks.rados.rados.0.smithi107.stdout:8945: delete oid 16 current snap is 0 2020-06-12T02:38:44.091 INFO:tasks.rados.rados.0.smithi107.stdout:8945: done (1 left) 2020-06-12T02:38:44.091 INFO:tasks.rados.rados.0.smithi107.stdout:8946: delete oid 17 current snap is 0 2020-06-12T02:38:44.094 INFO:tasks.rados.rados.0.smithi107.stdout:8944: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:44.094 INFO:tasks.rados.rados.0.smithi107.stdout:8946: done (1 left) 2020-06-12T02:38:44.094 INFO:tasks.rados.rados.0.smithi107.stdout:8947: delete oid 14 current snap is 0 2020-06-12T02:38:44.100 INFO:tasks.rados.rados.0.smithi107.stdout:8947: done (1 left) 2020-06-12T02:38:44.100 INFO:tasks.rados.rados.0.smithi107.stdout:8948: delete oid 9 current snap is 0 2020-06-12T02:38:44.102 INFO:tasks.rados.rados.0.smithi107.stdout:8944: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:44.102 INFO:tasks.rados.rados.0.smithi107.stdout:8944: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:44.105 INFO:tasks.rados.rados.0.smithi107.stdout:8948: done (1 left) 2020-06-12T02:38:44.105 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:44.106 INFO:tasks.rados.rados.0.smithi107.stdout:8949: seq_num 8948 ranges {884736=671744,1556480=401408} 2020-06-12T02:38:44.114 INFO:tasks.rados.rados.0.smithi107.stdout:8949: writing smithi10729407-2 from 884736 to 1556480 tid 1 2020-06-12T02:38:44.116 INFO:tasks.rados.rados.0.smithi107.stdout:8949: writing smithi10729407-2 from 1556480 to 1957888 tid 2 2020-06-12T02:38:44.117 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:44.117 INFO:tasks.rados.rados.0.smithi107.stdout:8950: seq_num 8949 ranges {0=688128,688128=401408} 2020-06-12T02:38:44.122 INFO:tasks.rados.rados.0.smithi107.stdout:8950: writing smithi10729407-12 from 0 to 688128 tid 1 2020-06-12T02:38:44.125 INFO:tasks.rados.rados.0.smithi107.stdout:8950: writing smithi10729407-12 from 688128 to 1089536 tid 2 2020-06-12T02:38:44.125 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:44.125 INFO:tasks.rados.rados.0.smithi107.stdout:8951: seq_num 8950 ranges {499712=704512,1204224=401408} 2020-06-12T02:38:44.132 INFO:tasks.rados.rados.0.smithi107.stdout:8951: writing smithi10729407-7 from 499712 to 1204224 tid 1 2020-06-12T02:38:44.135 INFO:tasks.rados.rados.0.smithi107.stdout:8951: writing smithi10729407-7 from 1204224 to 1605632 tid 2 2020-06-12T02:38:44.135 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:44.136 INFO:tasks.rados.rados.0.smithi107.stdout:8952: seq_num 8951 ranges {0=720896,720896=401408} 2020-06-12T02:38:44.140 INFO:tasks.rados.rados.0.smithi107.stdout:8952: writing smithi10729407-17 from 0 to 720896 tid 1 2020-06-12T02:38:44.147 INFO:tasks.rados.rados.0.smithi107.stdout:8952: writing smithi10729407-17 from 720896 to 1122304 tid 2 2020-06-12T02:38:44.148 INFO:tasks.rados.rados.0.smithi107.stdout:8953: delete oid 9 current snap is 0 2020-06-12T02:38:44.148 INFO:tasks.rados.rados.0.smithi107.stdout:8944: finishing write tid 5 to smithi10729407-8 2020-06-12T02:38:44.148 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1664 (ObjNum 8943 snap 0 seq_num 8943) dirty exists 2020-06-12T02:38:44.148 INFO:tasks.rados.rados.0.smithi107.stdout:8944: left oid 8 (ObjNum 8943 snap 0 seq_num 8943) 2020-06-12T02:38:44.148 INFO:tasks.rados.rados.0.smithi107.stdout:8949: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:44.149 INFO:tasks.rados.rados.0.smithi107.stdout:8949: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:44.149 INFO:tasks.rados.rados.0.smithi107.stdout:8949: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:44.149 INFO:tasks.rados.rados.0.smithi107.stdout:8949: finishing write tid 5 to smithi10729407-2 2020-06-12T02:38:44.149 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 839 (ObjNum 8948 snap 0 seq_num 8948) dirty exists 2020-06-12T02:38:44.149 INFO:tasks.rados.rados.0.smithi107.stdout:8949: left oid 2 (ObjNum 8948 snap 0 seq_num 8948) 2020-06-12T02:38:44.149 INFO:tasks.rados.rados.0.smithi107.stdout:8950: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:44.150 INFO:tasks.rados.rados.0.smithi107.stdout:8950: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:44.150 INFO:tasks.rados.rados.0.smithi107.stdout:8950: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:44.150 INFO:tasks.rados.rados.0.smithi107.stdout:8950: finishing write tid 5 to smithi10729407-12 2020-06-12T02:38:44.150 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1658 (ObjNum 8949 snap 0 seq_num 8949) dirty exists 2020-06-12T02:38:44.150 INFO:tasks.rados.rados.0.smithi107.stdout:8950: left oid 12 (ObjNum 8949 snap 0 seq_num 8949) 2020-06-12T02:38:44.151 INFO:tasks.rados.rados.0.smithi107.stdout:8951: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:44.152 INFO:tasks.rados.rados.0.smithi107.stdout:8951: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:44.152 INFO:tasks.rados.rados.0.smithi107.stdout:8951: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:44.152 INFO:tasks.rados.rados.0.smithi107.stdout:8951: finishing write tid 5 to smithi10729407-7 2020-06-12T02:38:44.152 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1611 (ObjNum 8950 snap 0 seq_num 8950) dirty exists 2020-06-12T02:38:44.152 INFO:tasks.rados.rados.0.smithi107.stdout:8951: left oid 7 (ObjNum 8950 snap 0 seq_num 8950) 2020-06-12T02:38:44.153 INFO:tasks.rados.rados.0.smithi107.stdout:8944: done (5 left) 2020-06-12T02:38:44.153 INFO:tasks.rados.rados.0.smithi107.stdout:8949: done (4 left) 2020-06-12T02:38:44.153 INFO:tasks.rados.rados.0.smithi107.stdout:8950: done (3 left) 2020-06-12T02:38:44.153 INFO:tasks.rados.rados.0.smithi107.stdout:8951: done (2 left) 2020-06-12T02:38:44.153 INFO:tasks.rados.rados.0.smithi107.stdout:8953: done (1 left) 2020-06-12T02:38:44.154 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:38:44.154 INFO:tasks.rados.rados.0.smithi107.stdout:8954: seq_num 8953 ranges {0=753664,753664=401408} 2020-06-12T02:38:44.156 INFO:tasks.rados.rados.0.smithi107.stdout:8954: writing smithi10729407-20 from 0 to 753664 tid 1 2020-06-12T02:38:44.163 INFO:tasks.rados.rados.0.smithi107.stdout:8954: writing smithi10729407-20 from 753664 to 1155072 tid 2 2020-06-12T02:38:44.164 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:44.164 INFO:tasks.rados.rados.0.smithi107.stdout:8955: seq_num 8954 ranges {0=770048,770048=401408} 2020-06-12T02:38:44.165 INFO:tasks.rados.rados.0.smithi107.stdout:8955: writing smithi10729407-9 from 0 to 770048 tid 1 2020-06-12T02:38:44.168 INFO:tasks.rados.rados.0.smithi107.stdout:8955: writing smithi10729407-9 from 770048 to 1171456 tid 2 2020-06-12T02:38:44.168 INFO:tasks.rados.rados.0.smithi107.stdout:8956: delete oid 4 current snap is 0 2020-06-12T02:38:44.168 INFO:tasks.rados.rados.0.smithi107.stdout:8952: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:44.168 INFO:tasks.rados.rados.0.smithi107.stdout:8952: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:44.169 INFO:tasks.rados.rados.0.smithi107.stdout:8952: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:44.169 INFO:tasks.rados.rados.0.smithi107.stdout:8952: finishing write tid 5 to smithi10729407-17 2020-06-12T02:38:44.169 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1683 (ObjNum 8951 snap 0 seq_num 8951) dirty exists 2020-06-12T02:38:44.169 INFO:tasks.rados.rados.0.smithi107.stdout:8952: left oid 17 (ObjNum 8951 snap 0 seq_num 8951) 2020-06-12T02:38:44.169 INFO:tasks.rados.rados.0.smithi107.stdout:8954: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:44.170 INFO:tasks.rados.rados.0.smithi107.stdout:8954: finishing write tid 2 to smithi10729407-20 2020-06-12T02:38:44.178 INFO:tasks.rados.rados.0.smithi107.stdout:8954: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:44.179 INFO:tasks.rados.rados.0.smithi107.stdout:8954: finishing write tid 5 to smithi10729407-20 2020-06-12T02:38:44.179 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 666 (ObjNum 8953 snap 0 seq_num 8953) dirty exists 2020-06-12T02:38:44.179 INFO:tasks.rados.rados.0.smithi107.stdout:8954: left oid 20 (ObjNum 8953 snap 0 seq_num 8953) 2020-06-12T02:38:44.179 INFO:tasks.rados.rados.0.smithi107.stdout:8952: done (3 left) 2020-06-12T02:38:44.179 INFO:tasks.rados.rados.0.smithi107.stdout:8954: done (2 left) 2020-06-12T02:38:44.180 INFO:tasks.rados.rados.0.smithi107.stdout:8956: done (1 left) 2020-06-12T02:38:44.180 INFO:tasks.rados.rados.0.smithi107.stdout:8957: delete oid 16 current snap is 0 2020-06-12T02:38:44.180 INFO:tasks.rados.rados.0.smithi107.stdout:8957: done (1 left) 2020-06-12T02:38:44.186 INFO:tasks.rados.rados.0.smithi107.stdout:8958: delete oid 4 current snap is 0 2020-06-12T02:38:44.186 INFO:tasks.rados.rados.0.smithi107.stdout:8955: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:44.186 INFO:tasks.rados.rados.0.smithi107.stdout:8955: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:44.187 INFO:tasks.rados.rados.0.smithi107.stdout:8955: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:44.188 INFO:tasks.rados.rados.0.smithi107.stdout:8958: done (1 left) 2020-06-12T02:38:44.188 INFO:tasks.rados.rados.0.smithi107.stdout:8959: delete oid 16 current snap is 0 2020-06-12T02:38:44.188 INFO:tasks.rados.rados.0.smithi107.stdout:8955: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:44.188 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1678 (ObjNum 8954 snap 0 seq_num 8954) dirty exists 2020-06-12T02:38:44.188 INFO:tasks.rados.rados.0.smithi107.stdout:8955: left oid 9 (ObjNum 8954 snap 0 seq_num 8954) 2020-06-12T02:38:44.189 INFO:tasks.rados.rados.0.smithi107.stdout:8955: done (1 left) 2020-06-12T02:38:44.189 INFO:tasks.rados.rados.0.smithi107.stdout:8959: done (0 left) 2020-06-12T02:38:44.189 INFO:tasks.rados.rados.0.smithi107.stdout:8960: delete oid 14 current snap is 0 2020-06-12T02:38:44.189 INFO:tasks.rados.rados.0.smithi107.stdout:8960: done (0 left) 2020-06-12T02:38:44.189 INFO:tasks.rados.rados.0.smithi107.stdout:8961: delete oid 20 current snap is 0 2020-06-12T02:38:44.190 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:44.185+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:44.190 INFO:tasks.rados.rados.0.smithi107.stdout:8961: done (0 left) 2020-06-12T02:38:44.191 INFO:tasks.rados.rados.0.smithi107.stdout:8962: delete oid 3 current snap is 0 2020-06-12T02:38:44.193 INFO:tasks.rados.rados.0.smithi107.stdout:8962: done (0 left) 2020-06-12T02:38:44.194 INFO:tasks.rados.rados.0.smithi107.stdout:8963: delete oid 19 current snap is 0 2020-06-12T02:38:44.195 INFO:tasks.rados.rados.0.smithi107.stdout:8963: done (0 left) 2020-06-12T02:38:44.195 INFO:tasks.rados.rados.0.smithi107.stdout:8964: delete oid 15 current snap is 0 2020-06-12T02:38:44.200 INFO:tasks.rados.rados.0.smithi107.stdout:8964: done (0 left) 2020-06-12T02:38:44.200 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:38:44.200 INFO:tasks.rados.rados.0.smithi107.stdout:8965: seq_num 8964 ranges {0=532480,532480=532480,1064960=270336} 2020-06-12T02:38:44.201 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:44.201 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:44.202 INFO:tasks.rados.rados.0.smithi107.stdout:8965: writing smithi10729407-6 from 0 to 532480 tid 1 2020-06-12T02:38:44.206 INFO:tasks.rados.rados.0.smithi107.stdout:8965: writing smithi10729407-6 from 532480 to 1064960 tid 2 2020-06-12T02:38:44.208 INFO:tasks.rados.rados.0.smithi107.stdout:8965: writing smithi10729407-6 from 1064960 to 1335296 tid 3 2020-06-12T02:38:44.208 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:38:44.209 INFO:tasks.rados.rados.0.smithi107.stdout:8966: seq_num 8965 ranges {0=548864,548864=409600,958464=393216} 2020-06-12T02:38:44.211 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-06-12T02:38:44.212 INFO:tasks.rados.rados.0.smithi107.stdout:8966: writing smithi10729407-5 from 0 to 548864 tid 1 2020-06-12T02:38:44.218 INFO:tasks.rados.rados.0.smithi107.stdout:8966: writing smithi10729407-5 from 548864 to 958464 tid 2 2020-06-12T02:38:44.218 INFO:tasks.rados.rados.0.smithi107.stdout:8966: writing smithi10729407-5 from 958464 to 1351680 tid 3 2020-06-12T02:38:44.218 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:38:44.218 INFO:tasks.rados.rados.0.smithi107.stdout:8967: seq_num 8966 ranges {0=565248,565248=696320,1261568=106496} 2020-06-12T02:38:44.221 INFO:tasks.rados.rados.0.smithi107.stdout:8967: writing smithi10729407-20 from 0 to 565248 tid 1 2020-06-12T02:38:44.225 INFO:tasks.rados.rados.0.smithi107.stdout:8967: writing smithi10729407-20 from 565248 to 1261568 tid 2 2020-06-12T02:38:44.226 INFO:tasks.rados.rados.0.smithi107.stdout:8967: writing smithi10729407-20 from 1261568 to 1368064 tid 3 2020-06-12T02:38:44.226 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:44.226 INFO:tasks.rados.rados.0.smithi107.stdout:8968: seq_num 8967 ranges {1605632=589824,2195456=581632,2777088=221184} 2020-06-12T02:38:44.240 INFO:tasks.rados.rados.0.smithi107.stdout:8968: writing smithi10729407-7 from 1605632 to 2195456 tid 1 2020-06-12T02:38:44.242 INFO:tasks.rados.rados.0.smithi107.stdout:8968: writing smithi10729407-7 from 2195456 to 2777088 tid 2 2020-06-12T02:38:44.243 INFO:tasks.rados.rados.0.smithi107.stdout:8968: writing smithi10729407-7 from 2777088 to 2998272 tid 3 2020-06-12T02:38:44.244 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:44.244 INFO:tasks.rados.rados.0.smithi107.stdout:8969: seq_num 8968 ranges {0=606208,606208=466944,1073152=335872} 2020-06-12T02:38:44.247 INFO:tasks.rados.rados.0.smithi107.stdout:8969: writing smithi10729407-19 from 0 to 606208 tid 1 2020-06-12T02:38:44.250 INFO:tasks.rados.rados.0.smithi107.stdout:8969: writing smithi10729407-19 from 606208 to 1073152 tid 2 2020-06-12T02:38:44.253 INFO:tasks.rados.rados.0.smithi107.stdout:8969: writing smithi10729407-19 from 1073152 to 1409024 tid 3 2020-06-12T02:38:44.253 INFO:tasks.rados.rados.0.smithi107.stdout:8970: delete oid 9 current snap is 0 2020-06-12T02:38:44.253 INFO:tasks.rados.rados.0.smithi107.stdout:8965: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:44.253 INFO:tasks.rados.rados.0.smithi107.stdout:8965: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:44.253 INFO:tasks.rados.rados.0.smithi107.stdout:8965: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:44.254 INFO:tasks.rados.rados.0.smithi107.stdout:8965: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:44.254 INFO:tasks.rados.rados.0.smithi107.stdout:8966: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:44.254 INFO:tasks.rados.rados.0.smithi107.stdout:8965: finishing write tid 6 to smithi10729407-6 2020-06-12T02:38:44.254 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1630 (ObjNum 8964 snap 0 seq_num 8964) dirty exists 2020-06-12T02:38:44.254 INFO:tasks.rados.rados.0.smithi107.stdout:8965: left oid 6 (ObjNum 8964 snap 0 seq_num 8964) 2020-06-12T02:38:44.255 INFO:tasks.rados.rados.0.smithi107.stdout:8966: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:44.255 INFO:tasks.rados.rados.0.smithi107.stdout:8966: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:44.255 INFO:tasks.rados.rados.0.smithi107.stdout:8966: finishing write tid 5 to smithi10729407-5 2020-06-12T02:38:44.255 INFO:tasks.rados.rados.0.smithi107.stdout:8966: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:44.255 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 870 (ObjNum 8965 snap 0 seq_num 8965) dirty exists 2020-06-12T02:38:44.255 INFO:tasks.rados.rados.0.smithi107.stdout:8966: left oid 5 (ObjNum 8965 snap 0 seq_num 8965) 2020-06-12T02:38:44.256 INFO:tasks.rados.rados.0.smithi107.stdout:8967: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:44.256 INFO:tasks.rados.rados.0.smithi107.stdout:8967: finishing write tid 2 to smithi10729407-20 2020-06-12T02:38:44.256 INFO:tasks.rados.rados.0.smithi107.stdout:8967: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:44.256 INFO:tasks.rados.rados.0.smithi107.stdout:8967: finishing write tid 5 to smithi10729407-20 2020-06-12T02:38:44.256 INFO:tasks.rados.rados.0.smithi107.stdout:8967: finishing write tid 6 to smithi10729407-20 2020-06-12T02:38:44.257 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 671 (ObjNum 8966 snap 0 seq_num 8966) dirty exists 2020-06-12T02:38:44.257 INFO:tasks.rados.rados.0.smithi107.stdout:8967: left oid 20 (ObjNum 8966 snap 0 seq_num 8966) 2020-06-12T02:38:44.257 INFO:tasks.rados.rados.0.smithi107.stdout:8968: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:44.257 INFO:tasks.rados.rados.0.smithi107.stdout:8968: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:44.258 INFO:tasks.rados.rados.0.smithi107.stdout:8968: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:44.258 INFO:tasks.rados.rados.0.smithi107.stdout:8968: finishing write tid 5 to smithi10729407-7 2020-06-12T02:38:44.258 INFO:tasks.rados.rados.0.smithi107.stdout:8968: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:44.259 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1615 (ObjNum 8967 snap 0 seq_num 8967) dirty exists 2020-06-12T02:38:44.259 INFO:tasks.rados.rados.0.smithi107.stdout:8968: left oid 7 (ObjNum 8967 snap 0 seq_num 8967) 2020-06-12T02:38:44.259 INFO:tasks.rados.rados.0.smithi107.stdout:8965: done (5 left) 2020-06-12T02:38:44.259 INFO:tasks.rados.rados.0.smithi107.stdout:8966: done (4 left) 2020-06-12T02:38:44.259 INFO:tasks.rados.rados.0.smithi107.stdout:8967: done (3 left) 2020-06-12T02:38:44.259 INFO:tasks.rados.rados.0.smithi107.stdout:8968: done (2 left) 2020-06-12T02:38:44.260 INFO:tasks.rados.rados.0.smithi107.stdout:8970: done (1 left) 2020-06-12T02:38:44.260 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:44.260 INFO:tasks.rados.rados.0.smithi107.stdout:8971: seq_num 8970 ranges {0=638976,638976=638976,1277952=163840} 2020-06-12T02:38:44.269 INFO:tasks.rados.rados.0.smithi107.stdout:8971: writing smithi10729407-9 from 0 to 638976 tid 1 2020-06-12T02:38:44.270 INFO:tasks.rados.rados.0.smithi107.stdout:8971: writing smithi10729407-9 from 638976 to 1277952 tid 2 2020-06-12T02:38:44.270 INFO:tasks.rados.rados.0.smithi107.stdout:8971: writing smithi10729407-9 from 1277952 to 1441792 tid 3 2020-06-12T02:38:44.271 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:44.271 INFO:tasks.rados.rados.0.smithi107.stdout:8972: seq_num 8971 ranges {0=655360,655360=565248,1220608=237568} 2020-06-12T02:38:44.271 INFO:tasks.rados.rados.0.smithi107.stdout:8972: writing smithi10729407-15 from 0 to 655360 tid 1 2020-06-12T02:38:44.275 INFO:tasks.rados.rados.0.smithi107.stdout:8972: writing smithi10729407-15 from 655360 to 1220608 tid 2 2020-06-12T02:38:44.276 INFO:tasks.rados.rados.0.smithi107.stdout:8972: writing smithi10729407-15 from 1220608 to 1458176 tid 3 2020-06-12T02:38:44.277 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:38:44.277 INFO:tasks.rados.rados.0.smithi107.stdout:8973: seq_num 8972 ranges {0=671744,671744=450560,1122304=352256} 2020-06-12T02:38:44.281 INFO:tasks.rados.rados.0.smithi107.stdout:8973: writing smithi10729407-14 from 0 to 671744 tid 1 2020-06-12T02:38:44.284 INFO:tasks.rados.rados.0.smithi107.stdout:8973: writing smithi10729407-14 from 671744 to 1122304 tid 2 2020-06-12T02:38:44.286 INFO:tasks.rados.rados.0.smithi107.stdout:8973: writing smithi10729407-14 from 1122304 to 1474560 tid 3 2020-06-12T02:38:44.286 INFO:tasks.rados.rados.0.smithi107.stdout:8974: delete oid 7 current snap is 0 2020-06-12T02:38:44.286 INFO:tasks.rados.rados.0.smithi107.stdout:8969: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:44.287 INFO:tasks.rados.rados.0.smithi107.stdout:8969: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:44.287 INFO:tasks.rados.rados.0.smithi107.stdout:8969: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:44.287 INFO:tasks.rados.rados.0.smithi107.stdout:8969: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:44.287 INFO:tasks.rados.rados.0.smithi107.stdout:8969: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:44.287 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1663 (ObjNum 8968 snap 0 seq_num 8968) dirty exists 2020-06-12T02:38:44.288 INFO:tasks.rados.rados.0.smithi107.stdout:8969: left oid 19 (ObjNum 8968 snap 0 seq_num 8968) 2020-06-12T02:38:44.288 INFO:tasks.rados.rados.0.smithi107.stdout:8971: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:44.288 INFO:tasks.rados.rados.0.smithi107.stdout:8971: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:44.288 INFO:tasks.rados.rados.0.smithi107.stdout:8971: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:44.288 INFO:tasks.rados.rados.0.smithi107.stdout:8971: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:44.288 INFO:tasks.rados.rados.0.smithi107.stdout:8972: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:44.289 INFO:tasks.rados.rados.0.smithi107.stdout:8972: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:44.289 INFO:tasks.rados.rados.0.smithi107.stdout:8972: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:44.290 INFO:tasks.rados.rados.0.smithi107.stdout:8972: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:44.290 INFO:tasks.rados.rados.0.smithi107.stdout:8971: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:44.290 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1683 (ObjNum 8970 snap 0 seq_num 8970) dirty exists 2020-06-12T02:38:44.291 INFO:tasks.rados.rados.0.smithi107.stdout:8971: left oid 9 (ObjNum 8970 snap 0 seq_num 8970) 2020-06-12T02:38:44.291 INFO:tasks.rados.rados.0.smithi107.stdout:8972: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:44.292 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1669 (ObjNum 8971 snap 0 seq_num 8971) dirty exists 2020-06-12T02:38:44.292 INFO:tasks.rados.rados.0.smithi107.stdout:8972: left oid 15 (ObjNum 8971 snap 0 seq_num 8971) 2020-06-12T02:38:44.292 INFO:tasks.rados.rados.0.smithi107.stdout:8969: done (4 left) 2020-06-12T02:38:44.292 INFO:tasks.rados.rados.0.smithi107.stdout:8971: done (3 left) 2020-06-12T02:38:44.292 INFO:tasks.rados.rados.0.smithi107.stdout:8972: done (2 left) 2020-06-12T02:38:44.292 INFO:tasks.rados.rados.0.smithi107.stdout:8974: done (1 left) 2020-06-12T02:38:44.293 INFO:tasks.rados.rados.0.smithi107.stdout:8975: delete oid 2 current snap is 0 2020-06-12T02:38:44.293 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:44.288+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:44.294 INFO:tasks.rados.rados.0.smithi107.stdout:8973: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:44.294 INFO:tasks.rados.rados.0.smithi107.stdout:8975: done (1 left) 2020-06-12T02:38:44.294 INFO:tasks.rados.rados.0.smithi107.stdout:8976: delete oid 12 current snap is 0 2020-06-12T02:38:44.296 INFO:tasks.rados.rados.0.smithi107.stdout:8976: done (1 left) 2020-06-12T02:38:44.296 INFO:tasks.rados.rados.0.smithi107.stdout:8977: delete oid 9 current snap is 0 2020-06-12T02:38:44.298 INFO:tasks.rados.rados.0.smithi107.stdout:8973: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:44.298 INFO:tasks.rados.rados.0.smithi107.stdout:8973: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:44.300 INFO:tasks.rados.rados.0.smithi107.stdout:8977: done (1 left) 2020-06-12T02:38:44.301 INFO:tasks.rados.rados.0.smithi107.stdout:8978: delete oid 8 current snap is 0 2020-06-12T02:38:44.302 INFO:tasks.rados.rados.0.smithi107.stdout:8973: finishing write tid 5 to smithi10729407-14 2020-06-12T02:38:44.303 INFO:tasks.rados.rados.0.smithi107.stdout:8978: done (1 left) 2020-06-12T02:38:44.303 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:38:44.304 INFO:tasks.rados.rados.0.smithi107.stdout:8979: seq_num 8978 ranges {851968=770048,1622016=557056,2179072=245760} 2020-06-12T02:38:44.312 INFO:tasks.rados.rados.0.smithi107.stdout:8979: writing smithi10729407-1 from 851968 to 1622016 tid 1 2020-06-12T02:38:44.315 INFO:tasks.rados.rados.0.smithi107.stdout:8979: writing smithi10729407-1 from 1622016 to 2179072 tid 2 2020-06-12T02:38:44.317 INFO:tasks.rados.rados.0.smithi107.stdout:8979: writing smithi10729407-1 from 2179072 to 2424832 tid 3 2020-06-12T02:38:44.317 INFO:tasks.rados.rados.0.smithi107.stdout:8980: delete oid 2 current snap is 0 2020-06-12T02:38:44.317 INFO:tasks.rados.rados.0.smithi107.stdout:8973: finishing write tid 6 to smithi10729407-14 2020-06-12T02:38:44.318 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1567 (ObjNum 8972 snap 0 seq_num 8972) dirty exists 2020-06-12T02:38:44.318 INFO:tasks.rados.rados.0.smithi107.stdout:8973: left oid 14 (ObjNum 8972 snap 0 seq_num 8972) 2020-06-12T02:38:44.319 INFO:tasks.rados.rados.0.smithi107.stdout:8973: done (2 left) 2020-06-12T02:38:44.319 INFO:tasks.rados.rados.0.smithi107.stdout:8980: done (1 left) 2020-06-12T02:38:44.320 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:44.320 INFO:tasks.rados.rados.0.smithi107.stdout:8981: seq_num 8980 ranges {0=802816,802816=778240,1581056=24576} 2020-06-12T02:38:44.324 INFO:tasks.rados.rados.0.smithi107.stdout:8981: writing smithi10729407-8 from 0 to 802816 tid 1 2020-06-12T02:38:44.329 INFO:tasks.rados.rados.0.smithi107.stdout:8981: writing smithi10729407-8 from 802816 to 1581056 tid 2 2020-06-12T02:38:44.329 INFO:tasks.rados.rados.0.smithi107.stdout:8981: writing smithi10729407-8 from 1581056 to 1605632 tid 3 2020-06-12T02:38:44.329 INFO:tasks.rados.rados.0.smithi107.stdout:8982: delete oid 2 current snap is 0 2020-06-12T02:38:44.329 INFO:tasks.rados.rados.0.smithi107.stdout:8979: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:44.330 INFO:tasks.rados.rados.0.smithi107.stdout:8979: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:44.330 INFO:tasks.rados.rados.0.smithi107.stdout:8979: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:44.330 INFO:tasks.rados.rados.0.smithi107.stdout:8979: finishing write tid 5 to smithi10729407-1 2020-06-12T02:38:44.330 INFO:tasks.rados.rados.0.smithi107.stdout:8979: finishing write tid 6 to smithi10729407-1 2020-06-12T02:38:44.331 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1634 (ObjNum 8978 snap 0 seq_num 8978) dirty exists 2020-06-12T02:38:44.331 INFO:tasks.rados.rados.0.smithi107.stdout:8979: left oid 1 (ObjNum 8978 snap 0 seq_num 8978) 2020-06-12T02:38:44.332 INFO:tasks.rados.rados.0.smithi107.stdout:8979: done (2 left) 2020-06-12T02:38:44.332 INFO:tasks.rados.rados.0.smithi107.stdout:8982: done (1 left) 2020-06-12T02:38:44.332 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:38:44.333 INFO:tasks.rados.rados.0.smithi107.stdout:8983: seq_num 8982 ranges {0=434176,434176=548864,983040=557056,1540096=98304} 2020-06-12T02:38:44.334 INFO:tasks.rados.rados.0.smithi107.stdout:8983: writing smithi10729407-7 from 0 to 434176 tid 1 2020-06-12T02:38:44.337 INFO:tasks.rados.rados.0.smithi107.stdout:8983: writing smithi10729407-7 from 434176 to 983040 tid 2 2020-06-12T02:38:44.340 INFO:tasks.rados.rados.0.smithi107.stdout:8983: writing smithi10729407-7 from 983040 to 1540096 tid 3 2020-06-12T02:38:44.341 INFO:tasks.rados.rados.0.smithi107.stdout:8983: writing smithi10729407-7 from 1540096 to 1638400 tid 4 2020-06-12T02:38:44.341 INFO:tasks.rados.rados.0.smithi107.stdout:8984: delete oid 3 current snap is 0 2020-06-12T02:38:44.341 INFO:tasks.rados.rados.0.smithi107.stdout:8981: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:44.342 INFO:tasks.rados.rados.0.smithi107.stdout:8983: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:44.342 INFO:tasks.rados.rados.0.smithi107.stdout:8981: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:44.343 INFO:tasks.rados.rados.0.smithi107.stdout:8981: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:44.344 INFO:tasks.rados.rados.0.smithi107.stdout:8981: finishing write tid 5 to smithi10729407-8 2020-06-12T02:38:44.346 INFO:tasks.rados.rados.0.smithi107.stdout:8981: finishing write tid 6 to smithi10729407-8 2020-06-12T02:38:44.346 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1674 (ObjNum 8980 snap 0 seq_num 8980) dirty exists 2020-06-12T02:38:44.346 INFO:tasks.rados.rados.0.smithi107.stdout:8981: left oid 8 (ObjNum 8980 snap 0 seq_num 8980) 2020-06-12T02:38:44.350 INFO:tasks.rados.rados.0.smithi107.stdout:8983: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:44.351 INFO:tasks.rados.rados.0.smithi107.stdout:8981: done (2 left) 2020-06-12T02:38:44.351 INFO:tasks.rados.rados.0.smithi107.stdout:8984: done (1 left) 2020-06-12T02:38:44.351 INFO:tasks.rados.rados.0.smithi107.stdout:8985: delete oid 5 current snap is 0 2020-06-12T02:38:44.352 INFO:tasks.rados.rados.0.smithi107.stdout:8983: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:44.352 INFO:tasks.rados.rados.0.smithi107.stdout:8983: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:44.354 INFO:tasks.rados.rados.0.smithi107.stdout:8985: done (1 left) 2020-06-12T02:38:44.354 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:38:44.355 INFO:tasks.rados.rados.0.smithi107.stdout:8986: seq_num 8985 ranges {0=483328,483328=598016,1081344=606208} 2020-06-12T02:38:44.357 INFO:tasks.rados.rados.0.smithi107.stdout:8986: writing smithi10729407-16 from 0 to 483328 tid 1 2020-06-12T02:38:44.361 INFO:tasks.rados.rados.0.smithi107.stdout:8986: writing smithi10729407-16 from 483328 to 1081344 tid 2 2020-06-12T02:38:44.362 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:44.362 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:44.365 INFO:tasks.rados.rados.0.smithi107.stdout:8986: writing smithi10729407-16 from 1081344 to 1687552 tid 3 2020-06-12T02:38:44.365 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:38:44.366 INFO:tasks.rados.rados.0.smithi107.stdout:8987: seq_num 8986 ranges {2424832=499712,2924544=483328,3407872=532480,3940352=188416} 2020-06-12T02:38:44.374 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-06-12T02:38:44.382 INFO:tasks.rados.rados.0.smithi107.stdout:8987: writing smithi10729407-1 from 2424832 to 2924544 tid 1 2020-06-12T02:38:44.384 INFO:tasks.rados.rados.0.smithi107.stdout:8987: writing smithi10729407-1 from 2924544 to 3407872 tid 2 2020-06-12T02:38:44.387 INFO:tasks.rados.rados.0.smithi107.stdout:8987: writing smithi10729407-1 from 3407872 to 3940352 tid 3 2020-06-12T02:38:44.389 INFO:tasks.rados.rados.0.smithi107.stdout:8987: writing smithi10729407-1 from 3940352 to 4128768 tid 4 2020-06-12T02:38:44.389 INFO:tasks.rados.rados.0.smithi107.stdout:8988: delete oid 19 current snap is 0 2020-06-12T02:38:44.389 INFO:tasks.rados.rados.0.smithi107.stdout:8983: finishing write tid 6 to smithi10729407-7 2020-06-12T02:38:44.389 INFO:tasks.rados.rados.0.smithi107.stdout:8983: finishing write tid 7 to smithi10729407-7 2020-06-12T02:38:44.389 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1621 (ObjNum 8982 snap 0 seq_num 8982) dirty exists 2020-06-12T02:38:44.390 INFO:tasks.rados.rados.0.smithi107.stdout:8983: left oid 7 (ObjNum 8982 snap 0 seq_num 8982) 2020-06-12T02:38:44.390 INFO:tasks.rados.rados.0.smithi107.stdout:8986: finishing write tid 1 to smithi10729407-16 2020-06-12T02:38:44.390 INFO:tasks.rados.rados.0.smithi107.stdout:8986: finishing write tid 2 to smithi10729407-16 2020-06-12T02:38:44.390 INFO:tasks.rados.rados.0.smithi107.stdout:8986: finishing write tid 3 to smithi10729407-16 2020-06-12T02:38:44.390 INFO:tasks.rados.rados.0.smithi107.stdout:8986: finishing write tid 5 to smithi10729407-16 2020-06-12T02:38:44.391 INFO:tasks.rados.rados.0.smithi107.stdout:8986: finishing write tid 6 to smithi10729407-16 2020-06-12T02:38:44.391 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1621 (ObjNum 8985 snap 0 seq_num 8985) dirty exists 2020-06-12T02:38:44.391 INFO:tasks.rados.rados.0.smithi107.stdout:8986: left oid 16 (ObjNum 8985 snap 0 seq_num 8985) 2020-06-12T02:38:44.392 INFO:tasks.rados.rados.0.smithi107.stdout:8983: done (3 left) 2020-06-12T02:38:44.392 INFO:tasks.rados.rados.0.smithi107.stdout:8986: done (2 left) 2020-06-12T02:38:44.392 INFO:tasks.rados.rados.0.smithi107.stdout:8988: done (1 left) 2020-06-12T02:38:44.392 INFO:tasks.rados.rados.0.smithi107.stdout:8989: delete oid 7 current snap is 0 2020-06-12T02:38:44.393 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:44.389+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:44.396 INFO:tasks.rados.rados.0.smithi107.stdout:8989: done (1 left) 2020-06-12T02:38:44.396 INFO:tasks.rados.rados.0.smithi107.stdout:8990: delete oid 11 current snap is 0 2020-06-12T02:38:44.398 INFO:tasks.rados.rados.0.smithi107.stdout:8987: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:44.398 INFO:tasks.rados.rados.0.smithi107.stdout:8987: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:44.400 INFO:tasks.rados.rados.0.smithi107.stdout:8987: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:44.401 INFO:tasks.rados.rados.0.smithi107.stdout:8987: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:44.401 INFO:tasks.rados.rados.0.smithi107.stdout:8990: done (1 left) 2020-06-12T02:38:44.401 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:44.401 INFO:tasks.rados.rados.0.smithi107.stdout:8991: seq_num 8990 ranges {0=573440,573440=466944,1040384=516096,1556480=221184} 2020-06-12T02:38:44.404 INFO:tasks.rados.rados.0.smithi107.stdout:8991: writing smithi10729407-4 from 0 to 573440 tid 1 2020-06-12T02:38:44.407 INFO:tasks.rados.rados.0.smithi107.stdout:8991: writing smithi10729407-4 from 573440 to 1040384 tid 2 2020-06-12T02:38:44.411 INFO:tasks.rados.rados.0.smithi107.stdout:8991: writing smithi10729407-4 from 1040384 to 1556480 tid 3 2020-06-12T02:38:44.412 INFO:tasks.rados.rados.0.smithi107.stdout:8991: writing smithi10729407-4 from 1556480 to 1777664 tid 4 2020-06-12T02:38:44.412 INFO:tasks.rados.rados.0.smithi107.stdout:8992: delete oid 9 current snap is 0 2020-06-12T02:38:44.413 INFO:tasks.rados.rados.0.smithi107.stdout:8987: finishing write tid 6 to smithi10729407-1 2020-06-12T02:38:44.413 INFO:tasks.rados.rados.0.smithi107.stdout:8987: finishing write tid 7 to smithi10729407-1 2020-06-12T02:38:44.413 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1639 (ObjNum 8986 snap 0 seq_num 8986) dirty exists 2020-06-12T02:38:44.413 INFO:tasks.rados.rados.0.smithi107.stdout:8987: left oid 1 (ObjNum 8986 snap 0 seq_num 8986) 2020-06-12T02:38:44.415 INFO:tasks.rados.rados.0.smithi107.stdout:8991: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:44.415 INFO:tasks.rados.rados.0.smithi107.stdout:8991: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:44.419 INFO:tasks.rados.rados.0.smithi107.stdout:8991: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:44.419 INFO:tasks.rados.rados.0.smithi107.stdout:8987: done (2 left) 2020-06-12T02:38:44.419 INFO:tasks.rados.rados.0.smithi107.stdout:8992: done (1 left) 2020-06-12T02:38:44.419 INFO:tasks.rados.rados.0.smithi107.stdout:8993: delete oid 5 current snap is 0 2020-06-12T02:38:44.420 INFO:tasks.rados.rados.0.smithi107.stdout:8991: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:44.421 INFO:tasks.rados.rados.0.smithi107.stdout:8991: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:44.424 INFO:tasks.rados.rados.0.smithi107.stdout:8991: finishing write tid 7 to smithi10729407-4 2020-06-12T02:38:44.424 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1690 (ObjNum 8990 snap 0 seq_num 8990) dirty exists 2020-06-12T02:38:44.424 INFO:tasks.rados.rados.0.smithi107.stdout:8991: left oid 4 (ObjNum 8990 snap 0 seq_num 8990) 2020-06-12T02:38:44.424 INFO:tasks.rados.rados.0.smithi107.stdout:8991: done (1 left) 2020-06-12T02:38:44.424 INFO:tasks.rados.rados.0.smithi107.stdout:8993: done (0 left) 2020-06-12T02:38:44.425 INFO:tasks.rados.rados.0.smithi107.stdout:8994: delete oid 4 current snap is 0 2020-06-12T02:38:44.429 INFO:tasks.rados.rados.0.smithi107.stdout:8994: done (0 left) 2020-06-12T02:38:44.429 INFO:tasks.rados.rados.0.smithi107.stdout:8995: delete oid 20 current snap is 0 2020-06-12T02:38:44.431 INFO:tasks.rados.rados.0.smithi107.stdout:8995: done (0 left) 2020-06-12T02:38:44.432 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:38:44.432 INFO:tasks.rados.rados.0.smithi107.stdout:8996: seq_num 8995 ranges {0=655360,655360=745472,1400832=458752} 2020-06-12T02:38:44.435 INFO:tasks.rados.rados.0.smithi107.stdout:8996: writing smithi10729407-10 from 0 to 655360 tid 1 2020-06-12T02:38:44.440 INFO:tasks.rados.rados.0.smithi107.stdout:8996: writing smithi10729407-10 from 655360 to 1400832 tid 2 2020-06-12T02:38:44.443 INFO:tasks.rados.rados.0.smithi107.stdout:8996: writing smithi10729407-10 from 1400832 to 1859584 tid 3 2020-06-12T02:38:44.443 INFO:tasks.rados.rados.0.smithi107.stdout:8997: delete oid 12 current snap is 0 2020-06-12T02:38:44.446 INFO:tasks.rados.rados.0.smithi107.stdout:8997: done (1 left) 2020-06-12T02:38:44.446 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:44.446 INFO:tasks.rados.rados.0.smithi107.stdout:8998: seq_num 8997 ranges {0=688128,688128=507904,1196032=696320} 2020-06-12T02:38:44.450 INFO:tasks.rados.rados.0.smithi107.stdout:8998: writing smithi10729407-18 from 0 to 688128 tid 1 2020-06-12T02:38:44.453 INFO:tasks.rados.rados.0.smithi107.stdout:8998: writing smithi10729407-18 from 688128 to 1196032 tid 2 2020-06-12T02:38:44.458 INFO:tasks.rados.rados.0.smithi107.stdout:8998: writing smithi10729407-18 from 1196032 to 1892352 tid 3 2020-06-12T02:38:44.458 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:38:44.458 INFO:tasks.rados.rados.0.smithi107.stdout:8999: seq_num 8998 ranges {0=704512,704512=794624,1499136=409600} 2020-06-12T02:38:44.462 INFO:tasks.rados.rados.0.smithi107.stdout:8999: writing smithi10729407-5 from 0 to 704512 tid 1 2020-06-12T02:38:44.468 INFO:tasks.rados.rados.0.smithi107.stdout:8999: writing smithi10729407-5 from 704512 to 1499136 tid 2 2020-06-12T02:38:44.471 INFO:tasks.rados.rados.0.smithi107.stdout:8999: writing smithi10729407-5 from 1499136 to 1908736 tid 3 2020-06-12T02:38:44.472 INFO:tasks.rados.rados.0.smithi107.stdout:9000: delete oid 20 current snap is 0 2020-06-12T02:38:44.472 INFO:tasks.rados.rados.0.smithi107.stdout:8996: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:44.472 INFO:tasks.rados.rados.0.smithi107.stdout:8996: finishing write tid 2 to smithi10729407-10 2020-06-12T02:38:44.472 INFO:tasks.rados.rados.0.smithi107.stdout:8996: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:44.472 INFO:tasks.rados.rados.0.smithi107.stdout:8996: finishing write tid 5 to smithi10729407-10 2020-06-12T02:38:44.473 INFO:tasks.rados.rados.0.smithi107.stdout:8996: finishing write tid 6 to smithi10729407-10 2020-06-12T02:38:44.473 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 651 (ObjNum 8995 snap 0 seq_num 8995) dirty exists 2020-06-12T02:38:44.473 INFO:tasks.rados.rados.0.smithi107.stdout:8996: left oid 10 (ObjNum 8995 snap 0 seq_num 8995) 2020-06-12T02:38:44.473 INFO:tasks.rados.rados.0.smithi107.stdout:8998: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:44.473 INFO:tasks.rados.rados.0.smithi107.stdout:8998: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:44.474 INFO:tasks.rados.rados.0.smithi107.stdout:8998: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:44.474 INFO:tasks.rados.rados.0.smithi107.stdout:8998: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:44.474 INFO:tasks.rados.rados.0.smithi107.stdout:8999: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:44.474 INFO:tasks.rados.rados.0.smithi107.stdout:8998: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:44.475 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1689 (ObjNum 8997 snap 0 seq_num 8997) dirty exists 2020-06-12T02:38:44.475 INFO:tasks.rados.rados.0.smithi107.stdout:8998: left oid 18 (ObjNum 8997 snap 0 seq_num 8997) 2020-06-12T02:38:44.478 INFO:tasks.rados.rados.0.smithi107.stdout:8996: done (3 left) 2020-06-12T02:38:44.478 INFO:tasks.rados.rados.0.smithi107.stdout:8998: done (2 left) 2020-06-12T02:38:44.479 INFO:tasks.rados.rados.0.smithi107.stdout:9000: done (1 left) 2020-06-12T02:38:44.479 INFO:tasks.rados.rados.0.smithi107.stdout:9001: delete oid 2 current snap is 0 2020-06-12T02:38:44.480 INFO:tasks.rados.rados.0.smithi107.stdout:8999: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:44.480 INFO:tasks.rados.rados.0.smithi107.stdout:8999: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:44.481 INFO:tasks.rados.rados.0.smithi107.stdout:8999: finishing write tid 5 to smithi10729407-5 2020-06-12T02:38:44.482 INFO:tasks.rados.rados.0.smithi107.stdout:8999: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:44.483 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 876 (ObjNum 8998 snap 0 seq_num 8998) dirty exists 2020-06-12T02:38:44.483 INFO:tasks.rados.rados.0.smithi107.stdout:8999: left oid 5 (ObjNum 8998 snap 0 seq_num 8998) 2020-06-12T02:38:44.493 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:44.490+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:44.547 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:44.548 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:44.557 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-06-12T02:38:44.601 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:44.599+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:44.675 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd unset nodeep-scrub 2020-06-12T02:38:44.676 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:44.677 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:44.685 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-06-12T02:38:44.720 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:44.717+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:44.796 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:44.793+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:44.842 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:44.842 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:44.850 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-06-12T02:38:44.902 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:44.897+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:44.966 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:44.966 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:44.974 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-06-12T02:38:45.001 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:44.998+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:45.091 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:45.092 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:45.100 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-06-12T02:38:45.102 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:45.100+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:45.203 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:38:45.201+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:38:45.219 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:45.219 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:45.228 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-06-12T02:38:45.311 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:45.309+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:45.384 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:45.385 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:45.393 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-06-12T02:38:45.434 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:45.432+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:45.502 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2020-06-12T02:38:45.503 INFO:tasks.rados.rados.0.smithi107.stdout:8999: done (1 left) 2020-06-12T02:38:45.503 INFO:tasks.rados.rados.0.smithi107.stdout:9001: done (0 left) 2020-06-12T02:38:45.503 INFO:tasks.rados.rados.0.smithi107.stdout:9002: delete oid 16 current snap is 0 2020-06-12T02:38:45.506 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:45.504+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:45.514 INFO:tasks.rados.rados.0.smithi107.stdout:9002: done (0 left) 2020-06-12T02:38:45.515 INFO:tasks.rados.rados.0.smithi107.stdout:9003: delete oid 18 current snap is 0 2020-06-12T02:38:45.522 INFO:tasks.rados.rados.0.smithi107.stdout:9003: done (0 left) 2020-06-12T02:38:45.522 INFO:tasks.rados.rados.0.smithi107.stdout:9004: delete oid 16 current snap is 0 2020-06-12T02:38:45.527 INFO:tasks.rados.rados.0.smithi107.stdout:9004: done (0 left) 2020-06-12T02:38:45.528 INFO:tasks.rados.rados.0.smithi107.stdout:9005: delete oid 20 current snap is 0 2020-06-12T02:38:45.530 INFO:tasks.rados.rados.0.smithi107.stdout:9005: done (0 left) 2020-06-12T02:38:45.530 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:45.531 INFO:tasks.rados.rados.0.smithi107.stdout:9006: seq_num 9005 ranges {0=417792,417792=434176,851968=679936,1531904=491520} 2020-06-12T02:38:45.532 INFO:tasks.rados.rados.0.smithi107.stdout:9006: writing smithi10729407-11 from 0 to 417792 tid 1 2020-06-12T02:38:45.535 INFO:tasks.rados.rados.0.smithi107.stdout:9006: writing smithi10729407-11 from 417792 to 851968 tid 2 2020-06-12T02:38:45.539 INFO:tasks.rados.rados.0.smithi107.stdout:9006: writing smithi10729407-11 from 851968 to 1531904 tid 3 2020-06-12T02:38:45.542 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-06-12T02:38:45.543 INFO:tasks.rados.rados.0.smithi107.stdout:9006: writing smithi10729407-11 from 1531904 to 2023424 tid 4 2020-06-12T02:38:45.543 INFO:tasks.rados.rados.0.smithi107.stdout:9007: delete oid 5 current snap is 0 2020-06-12T02:38:45.544 INFO:tasks.rados.rados.0.smithi107.stdout:9006: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:45.544 INFO:tasks.rados.rados.0.smithi107.stdout:9006: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:45.548 INFO:tasks.rados.rados.0.smithi107.stdout:9007: done (1 left) 2020-06-12T02:38:45.548 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:45.549 INFO:tasks.rados.rados.0.smithi107.stdout:9008: seq_num 9007 ranges {0=458752,458752=606208,1064960=647168,1712128=352256} 2020-06-12T02:38:45.551 INFO:tasks.rados.rados.0.smithi107.stdout:9008: writing smithi10729407-13 from 0 to 458752 tid 1 2020-06-12T02:38:45.555 INFO:tasks.rados.rados.0.smithi107.stdout:9008: writing smithi10729407-13 from 458752 to 1064960 tid 2 2020-06-12T02:38:45.559 INFO:tasks.rados.rados.0.smithi107.stdout:9008: writing smithi10729407-13 from 1064960 to 1712128 tid 3 2020-06-12T02:38:45.561 INFO:tasks.rados.rados.0.smithi107.stdout:9008: writing smithi10729407-13 from 1712128 to 2064384 tid 4 2020-06-12T02:38:45.561 INFO:tasks.rados.rados.0.smithi107.stdout:9009: delete oid 14 current snap is 0 2020-06-12T02:38:45.562 INFO:tasks.rados.rados.0.smithi107.stdout:9006: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:45.562 INFO:tasks.rados.rados.0.smithi107.stdout:9006: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:45.564 INFO:tasks.rados.rados.0.smithi107.stdout:9006: finishing write tid 6 to smithi10729407-11 2020-06-12T02:38:45.564 INFO:tasks.rados.rados.0.smithi107.stdout:9006: finishing write tid 7 to smithi10729407-11 2020-06-12T02:38:45.565 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1573 (ObjNum 9005 snap 0 seq_num 9005) dirty exists 2020-06-12T02:38:45.565 INFO:tasks.rados.rados.0.smithi107.stdout:9006: left oid 11 (ObjNum 9005 snap 0 seq_num 9005) 2020-06-12T02:38:45.565 INFO:tasks.rados.rados.0.smithi107.stdout:9008: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:45.565 INFO:tasks.rados.rados.0.smithi107.stdout:9006: done (2 left) 2020-06-12T02:38:45.566 INFO:tasks.rados.rados.0.smithi107.stdout:9009: done (1 left) 2020-06-12T02:38:45.566 INFO:tasks.rados.rados.0.smithi107.stdout:9010: delete oid 15 current snap is 0 2020-06-12T02:38:45.569 INFO:tasks.rados.rados.0.smithi107.stdout:9008: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:45.569 INFO:tasks.rados.rados.0.smithi107.stdout:9008: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:45.571 INFO:tasks.rados.rados.0.smithi107.stdout:9010: done (1 left) 2020-06-12T02:38:45.571 INFO:tasks.rados.rados.0.smithi107.stdout:9011: delete oid 8 current snap is 0 2020-06-12T02:38:45.572 INFO:tasks.rados.rados.0.smithi107.stdout:9008: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:45.573 INFO:tasks.rados.rados.0.smithi107.stdout:9008: finishing write tid 6 to smithi10729407-13 2020-06-12T02:38:45.575 INFO:tasks.rados.rados.0.smithi107.stdout:9008: finishing write tid 7 to smithi10729407-13 2020-06-12T02:38:45.575 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1627 (ObjNum 9007 snap 0 seq_num 9007) dirty exists 2020-06-12T02:38:45.576 INFO:tasks.rados.rados.0.smithi107.stdout:9008: left oid 13 (ObjNum 9007 snap 0 seq_num 9007) 2020-06-12T02:38:45.577 INFO:tasks.rados.rados.0.smithi107.stdout:9008: done (1 left) 2020-06-12T02:38:45.577 INFO:tasks.rados.rados.0.smithi107.stdout:9011: done (0 left) 2020-06-12T02:38:45.577 INFO:tasks.rados.rados.0.smithi107.stdout:9012: delete oid 5 current snap is 0 2020-06-12T02:38:45.578 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:45.578 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:45.580 INFO:tasks.rados.rados.0.smithi107.stdout:9012: done (0 left) 2020-06-12T02:38:45.580 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:45.580 INFO:tasks.rados.rados.0.smithi107.stdout:9013: seq_num 9012 ranges {0=540672,540672=475136,1015808=434176,1449984=466944,1916928=229376} 2020-06-12T02:38:45.583 INFO:tasks.rados.rados.0.smithi107.stdout:9013: writing smithi10729407-18 from 0 to 540672 tid 1 2020-06-12T02:38:45.586 INFO:tasks.rados.rados.0.smithi107.stdout:9013: writing smithi10729407-18 from 540672 to 1015808 tid 2 2020-06-12T02:38:45.588 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-06-12T02:38:45.589 INFO:tasks.rados.rados.0.smithi107.stdout:9013: writing smithi10729407-18 from 1015808 to 1449984 tid 3 2020-06-12T02:38:45.592 INFO:tasks.rados.rados.0.smithi107.stdout:9013: writing smithi10729407-18 from 1449984 to 1916928 tid 4 2020-06-12T02:38:45.594 INFO:tasks.rados.rados.0.smithi107.stdout:9013: writing smithi10729407-18 from 1916928 to 2146304 tid 5 2020-06-12T02:38:45.594 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:38:45.594 INFO:tasks.rados.rados.0.smithi107.stdout:9014: seq_num 9013 ranges {2023424=557056,2580480=761856,3342336=638976,3981312=204800} 2020-06-12T02:38:45.608 INFO:tasks.rados.rados.0.smithi107.stdout:9014: writing smithi10729407-11 from 2023424 to 2580480 tid 1 2020-06-12T02:38:45.613 INFO:tasks.rados.rados.0.smithi107.stdout:9014: writing smithi10729407-11 from 2580480 to 3342336 tid 2 2020-06-12T02:38:45.617 INFO:tasks.rados.rados.0.smithi107.stdout:9014: writing smithi10729407-11 from 3342336 to 3981312 tid 3 2020-06-12T02:38:45.619 INFO:tasks.rados.rados.0.smithi107.stdout:9014: writing smithi10729407-11 from 3981312 to 4186112 tid 4 2020-06-12T02:38:45.619 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:45.619 INFO:tasks.rados.rados.0.smithi107.stdout:9015: seq_num 9014 ranges {0=573440,573440=647168,1220608=442368,1662976=516096} 2020-06-12T02:38:45.622 INFO:tasks.rados.rados.0.smithi107.stdout:9015: writing smithi10729407-8 from 0 to 573440 tid 1 2020-06-12T02:38:45.626 INFO:tasks.rados.rados.0.smithi107.stdout:9015: writing smithi10729407-8 from 573440 to 1220608 tid 2 2020-06-12T02:38:45.629 INFO:tasks.rados.rados.0.smithi107.stdout:9015: writing smithi10729407-8 from 1220608 to 1662976 tid 3 2020-06-12T02:38:45.630 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:38:45.627+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:38:45.632 INFO:tasks.rados.rados.0.smithi107.stdout:9015: writing smithi10729407-8 from 1662976 to 2179072 tid 4 2020-06-12T02:38:45.632 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:45.632 INFO:tasks.rados.rados.0.smithi107.stdout:9016: seq_num 9015 ranges {0=589824,589824=532480,1122304=606208,1728512=466944} 2020-06-12T02:38:45.635 INFO:tasks.rados.rados.0.smithi107.stdout:9016: writing smithi10729407-4 from 0 to 589824 tid 1 2020-06-12T02:38:45.638 INFO:tasks.rados.rados.0.smithi107.stdout:9016: writing smithi10729407-4 from 589824 to 1122304 tid 2 2020-06-12T02:38:45.642 INFO:tasks.rados.rados.0.smithi107.stdout:9016: writing smithi10729407-4 from 1122304 to 1728512 tid 3 2020-06-12T02:38:45.645 INFO:tasks.rados.rados.0.smithi107.stdout:9016: writing smithi10729407-4 from 1728512 to 2195456 tid 4 2020-06-12T02:38:45.645 INFO:tasks.rados.rados.0.smithi107.stdout:9017: delete oid 20 current snap is 0 2020-06-12T02:38:45.645 INFO:tasks.rados.rados.0.smithi107.stdout:9013: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:45.645 INFO:tasks.rados.rados.0.smithi107.stdout:9013: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:45.646 INFO:tasks.rados.rados.0.smithi107.stdout:9013: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:45.646 INFO:tasks.rados.rados.0.smithi107.stdout:9013: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:45.646 INFO:tasks.rados.rados.0.smithi107.stdout:9013: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:45.646 INFO:tasks.rados.rados.0.smithi107.stdout:9013: finishing write tid 7 to smithi10729407-18 2020-06-12T02:38:45.646 INFO:tasks.rados.rados.0.smithi107.stdout:9013: finishing write tid 8 to smithi10729407-18 2020-06-12T02:38:45.647 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1696 (ObjNum 9012 snap 0 seq_num 9012) dirty exists 2020-06-12T02:38:45.647 INFO:tasks.rados.rados.0.smithi107.stdout:9013: left oid 18 (ObjNum 9012 snap 0 seq_num 9012) 2020-06-12T02:38:45.647 INFO:tasks.rados.rados.0.smithi107.stdout:9014: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:45.647 INFO:tasks.rados.rados.0.smithi107.stdout:9014: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:45.647 INFO:tasks.rados.rados.0.smithi107.stdout:9014: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:45.647 INFO:tasks.rados.rados.0.smithi107.stdout:9014: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:45.648 INFO:tasks.rados.rados.0.smithi107.stdout:9014: finishing write tid 6 to smithi10729407-11 2020-06-12T02:38:45.648 INFO:tasks.rados.rados.0.smithi107.stdout:9014: finishing write tid 7 to smithi10729407-11 2020-06-12T02:38:45.648 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1578 (ObjNum 9013 snap 0 seq_num 9013) dirty exists 2020-06-12T02:38:45.648 INFO:tasks.rados.rados.0.smithi107.stdout:9014: left oid 11 (ObjNum 9013 snap 0 seq_num 9013) 2020-06-12T02:38:45.648 INFO:tasks.rados.rados.0.smithi107.stdout:9015: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:45.649 INFO:tasks.rados.rados.0.smithi107.stdout:9015: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:45.649 INFO:tasks.rados.rados.0.smithi107.stdout:9015: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:45.649 INFO:tasks.rados.rados.0.smithi107.stdout:9015: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:45.650 INFO:tasks.rados.rados.0.smithi107.stdout:9013: done (4 left) 2020-06-12T02:38:45.650 INFO:tasks.rados.rados.0.smithi107.stdout:9014: done (3 left) 2020-06-12T02:38:45.650 INFO:tasks.rados.rados.0.smithi107.stdout:9017: done (2 left) 2020-06-12T02:38:45.650 INFO:tasks.rados.rados.0.smithi107.stdout:9018: delete oid 18 current snap is 0 2020-06-12T02:38:45.652 INFO:tasks.rados.rados.0.smithi107.stdout:9015: finishing write tid 6 to smithi10729407-8 2020-06-12T02:38:45.654 INFO:tasks.rados.rados.0.smithi107.stdout:9016: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:45.655 INFO:tasks.rados.rados.0.smithi107.stdout:9015: finishing write tid 7 to smithi10729407-8 2020-06-12T02:38:45.656 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1681 (ObjNum 9014 snap 0 seq_num 9014) dirty exists 2020-06-12T02:38:45.656 INFO:tasks.rados.rados.0.smithi107.stdout:9015: left oid 8 (ObjNum 9014 snap 0 seq_num 9014) 2020-06-12T02:38:45.656 INFO:tasks.rados.rados.0.smithi107.stdout:9016: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:45.657 INFO:tasks.rados.rados.0.smithi107.stdout:9016: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:45.658 INFO:tasks.rados.rados.0.smithi107.stdout:9016: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:45.659 INFO:tasks.rados.rados.0.smithi107.stdout:9016: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:45.659 INFO:tasks.rados.rados.0.smithi107.stdout:9015: done (2 left) 2020-06-12T02:38:45.659 INFO:tasks.rados.rados.0.smithi107.stdout:9018: done (1 left) 2020-06-12T02:38:45.659 INFO:tasks.rados.rados.0.smithi107.stdout:9019: delete oid 6 current snap is 0 2020-06-12T02:38:45.660 INFO:tasks.rados.rados.0.smithi107.stdout:9016: finishing write tid 7 to smithi10729407-4 2020-06-12T02:38:45.660 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1696 (ObjNum 9015 snap 0 seq_num 9015) dirty exists 2020-06-12T02:38:45.660 INFO:tasks.rados.rados.0.smithi107.stdout:9016: left oid 4 (ObjNum 9015 snap 0 seq_num 9015) 2020-06-12T02:38:45.661 INFO:tasks.rados.rados.0.smithi107.stdout:9016: done (1 left) 2020-06-12T02:38:45.662 INFO:tasks.rados.rados.0.smithi107.stdout:9019: done (0 left) 2020-06-12T02:38:45.662 INFO:tasks.rados.rados.0.smithi107.stdout:9020: delete oid 8 current snap is 0 2020-06-12T02:38:45.666 INFO:tasks.rados.rados.0.smithi107.stdout:9020: done (0 left) 2020-06-12T02:38:45.666 INFO:tasks.rados.rados.0.smithi107.stdout:9021: delete oid 10 current snap is 0 2020-06-12T02:38:45.669 INFO:tasks.rados.rados.0.smithi107.stdout:9021: done (0 left) 2020-06-12T02:38:45.669 INFO:tasks.rados.rados.0.smithi107.stdout:9022: delete oid 14 current snap is 0 2020-06-12T02:38:45.671 INFO:tasks.rados.rados.0.smithi107.stdout:9022: done (0 left) 2020-06-12T02:38:45.672 INFO:tasks.rados.rados.0.smithi107.stdout:9023: delete oid 15 current snap is 0 2020-06-12T02:38:45.674 INFO:tasks.rados.rados.0.smithi107.stdout:9023: done (0 left) 2020-06-12T02:38:45.675 INFO:tasks.rados.rados.0.smithi107.stdout:9024: delete oid 12 current snap is 0 2020-06-12T02:38:45.677 INFO:tasks.rados.rados.0.smithi107.stdout:9024: done (0 left) 2020-06-12T02:38:45.677 INFO:tasks.rados.rados.0.smithi107.stdout:9025: delete oid 1 current snap is 0 2020-06-12T02:38:45.680 INFO:tasks.rados.rados.0.smithi107.stdout:9025: done (0 left) 2020-06-12T02:38:45.680 INFO:tasks.rados.rados.0.smithi107.stdout:9026: delete oid 12 current snap is 0 2020-06-12T02:38:45.683 INFO:tasks.rados.rados.0.smithi107.stdout:9026: done (0 left) 2020-06-12T02:38:45.683 INFO:tasks.rados.rados.0.smithi107.stdout:9027: delete oid 19 current snap is 0 2020-06-12T02:38:45.686 INFO:tasks.rados.rados.0.smithi107.stdout:9027: done (0 left) 2020-06-12T02:38:45.686 INFO:tasks.rados.rados.0.smithi107.stdout:9028: delete oid 14 current snap is 0 2020-06-12T02:38:45.696 INFO:tasks.rados.rados.0.smithi107.stdout:9028: done (0 left) 2020-06-12T02:38:45.696 INFO:tasks.rados.rados.0.smithi107.stdout:9029: delete oid 18 current snap is 0 2020-06-12T02:38:45.696 INFO:tasks.rados.rados.0.smithi107.stdout:9029: done (0 left) 2020-06-12T02:38:45.697 INFO:tasks.rados.rados.0.smithi107.stdout:9030: delete oid 11 current snap is 0 2020-06-12T02:38:45.697 INFO:tasks.rados.rados.0.smithi107.stdout:9030: done (0 left) 2020-06-12T02:38:45.697 INFO:tasks.rados.rados.0.smithi107.stdout:9031: delete oid 2 current snap is 0 2020-06-12T02:38:45.697 INFO:tasks.rados.rados.0.smithi107.stdout:9031: done (0 left) 2020-06-12T02:38:45.697 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:45.697 INFO:tasks.rados.rados.0.smithi107.stdout:9032: seq_num 9031 ranges {0=57344} 2020-06-12T02:38:45.698 INFO:tasks.rados.rados.0.smithi107.stdout:9032: writing smithi10729407-15 from 0 to 57344 tid 1 2020-06-12T02:38:45.698 INFO:tasks.rados.rados.0.smithi107.stdout:9033: delete oid 2 current snap is 0 2020-06-12T02:38:45.700 INFO:tasks.rados.rados.0.smithi107.stdout:9032: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:45.701 INFO:tasks.rados.rados.0.smithi107.stdout:9032: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:45.701 INFO:tasks.rados.rados.0.smithi107.stdout:9033: done (1 left) 2020-06-12T02:38:45.701 INFO:tasks.rados.rados.0.smithi107.stdout:9034: delete oid 3 current snap is 0 2020-06-12T02:38:45.702 INFO:tasks.rados.rados.0.smithi107.stdout:9032: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:45.703 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1685 (ObjNum 9031 snap 0 seq_num 9031) dirty exists 2020-06-12T02:38:45.703 INFO:tasks.rados.rados.0.smithi107.stdout:9032: left oid 15 (ObjNum 9031 snap 0 seq_num 9031) 2020-06-12T02:38:45.704 INFO:tasks.rados.rados.0.smithi107.stdout:9032: done (1 left) 2020-06-12T02:38:45.704 INFO:tasks.rados.rados.0.smithi107.stdout:9034: done (0 left) 2020-06-12T02:38:45.704 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:38:45.704 INFO:tasks.rados.rados.0.smithi107.stdout:9035: seq_num 9034 ranges {0=106496} 2020-06-12T02:38:45.705 INFO:tasks.rados.rados.0.smithi107.stdout:9035: writing smithi10729407-5 from 0 to 106496 tid 1 2020-06-12T02:38:45.705 INFO:tasks.rados.rados.0.smithi107.stdout:9036: delete oid 11 current snap is 0 2020-06-12T02:38:45.712 INFO:tasks.rados.rados.0.smithi107.stdout:9036: done (1 left) 2020-06-12T02:38:45.712 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:45.713 INFO:tasks.rados.rados.0.smithi107.stdout:9037: seq_num 9036 ranges {0=139264} 2020-06-12T02:38:45.713 INFO:tasks.rados.rados.0.smithi107.stdout:9037: writing smithi10729407-1 from 0 to 139264 tid 1 2020-06-12T02:38:45.713 INFO:tasks.rados.rados.0.smithi107.stdout:9038: delete oid 18 current snap is 0 2020-06-12T02:38:45.713 INFO:tasks.rados.rados.0.smithi107.stdout:9035: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:45.713 INFO:tasks.rados.rados.0.smithi107.stdout:9035: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:45.714 INFO:tasks.rados.rados.0.smithi107.stdout:9038: done (2 left) 2020-06-12T02:38:45.714 INFO:tasks.rados.rados.0.smithi107.stdout:9039: delete oid 16 current snap is 0 2020-06-12T02:38:45.714 INFO:tasks.rados.rados.0.smithi107.stdout:9035: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:45.714 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 880 (ObjNum 9034 snap 0 seq_num 9034) dirty exists 2020-06-12T02:38:45.714 INFO:tasks.rados.rados.0.smithi107.stdout:9035: left oid 5 (ObjNum 9034 snap 0 seq_num 9034) 2020-06-12T02:38:45.715 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:38:45.715 INFO:tasks.rados.rados.0.smithi107.stdout:9037: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:45.716 INFO:tasks.rados.rados.0.smithi107.stdout:9035: done (2 left) 2020-06-12T02:38:45.716 INFO:tasks.rados.rados.0.smithi107.stdout:9039: done (1 left) 2020-06-12T02:38:45.716 INFO:tasks.rados.rados.0.smithi107.stdout:9040: delete oid 12 current snap is 0 2020-06-12T02:38:45.717 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:45.713+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:45.717 INFO:tasks.rados.rados.0.smithi107.stdout:9040: done (1 left) 2020-06-12T02:38:45.718 INFO:tasks.rados.rados.0.smithi107.stdout:9041: delete oid 6 current snap is 0 2020-06-12T02:38:45.719 INFO:tasks.rados.rados.0.smithi107.stdout:9037: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:45.720 INFO:tasks.rados.rados.0.smithi107.stdout:9041: done (1 left) 2020-06-12T02:38:45.721 INFO:tasks.rados.rados.0.smithi107.stdout:9042: delete oid 16 current snap is 0 2020-06-12T02:38:45.721 INFO:tasks.rados.rados.0.smithi107.stdout:9037: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:45.721 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1643 (ObjNum 9036 snap 0 seq_num 9036) dirty exists 2020-06-12T02:38:45.721 INFO:tasks.rados.rados.0.smithi107.stdout:9037: left oid 1 (ObjNum 9036 snap 0 seq_num 9036) 2020-06-12T02:38:45.723 INFO:tasks.rados.rados.0.smithi107.stdout:9037: done (1 left) 2020-06-12T02:38:45.723 INFO:tasks.rados.rados.0.smithi107.stdout:9042: done (0 left) 2020-06-12T02:38:45.723 INFO:tasks.rados.rados.0.smithi107.stdout:9043: delete oid 9 current snap is 0 2020-06-12T02:38:45.725 INFO:tasks.rados.rados.0.smithi107.stdout:9043: done (0 left) 2020-06-12T02:38:45.725 INFO:tasks.rados.rados.0.smithi107.stdout:9044: delete oid 7 current snap is 0 2020-06-12T02:38:45.727 INFO:tasks.rados.rados.0.smithi107.stdout:9044: done (0 left) 2020-06-12T02:38:45.727 INFO:tasks.rados.rados.0.smithi107.stdout:9045: delete oid 11 current snap is 0 2020-06-12T02:38:45.729 INFO:tasks.rados.rados.0.smithi107.stdout:9045: done (0 left) 2020-06-12T02:38:45.729 INFO:tasks.rados.rados.0.smithi107.stdout:9046: delete oid 12 current snap is 0 2020-06-12T02:38:45.731 INFO:tasks.rados.rados.0.smithi107.stdout:9046: done (0 left) 2020-06-12T02:38:45.731 INFO:tasks.rados.rados.0.smithi107.stdout:9047: delete oid 13 current snap is 0 2020-06-12T02:38:45.735 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:38:45.736 INFO:tasks.rados.rados.0.smithi107.stdout:9047: done (0 left) 2020-06-12T02:38:45.736 INFO:tasks.rados.rados.0.smithi107.stdout:9048: delete oid 6 current snap is 0 2020-06-12T02:38:45.739 INFO:tasks.rados.rados.0.smithi107.stdout:9048: done (0 left) 2020-06-12T02:38:45.739 INFO:tasks.rados.rados.0.smithi107.stdout:9049: delete oid 20 current snap is 0 2020-06-12T02:38:45.741 INFO:tasks.rados.rados.0.smithi107.stdout:9049: done (0 left) 2020-06-12T02:38:45.741 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:45.741 INFO:tasks.rados.rados.0.smithi107.stdout:9050: seq_num 9049 ranges {0=360448} 2020-06-12T02:38:45.744 INFO:tasks.rados.rados.0.smithi107.stdout:9050: writing smithi10729407-8 from 0 to 360448 tid 1 2020-06-12T02:38:45.744 INFO:tasks.rados.rados.0.smithi107.stdout:9051: delete oid 1 current snap is 0 2020-06-12T02:38:45.747 INFO:tasks.rados.rados.0.smithi107.stdout:9051: done (1 left) 2020-06-12T02:38:45.747 INFO:tasks.rados.rados.0.smithi107.stdout:9052: delete oid 17 current snap is 0 2020-06-12T02:38:45.749 INFO:tasks.rados.rados.0.smithi107.stdout:9050: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:45.752 INFO:tasks.rados.rados.0.smithi107.stdout:9052: done (1 left) 2020-06-12T02:38:45.752 INFO:tasks.rados.rados.0.smithi107.stdout:9053: delete oid 6 current snap is 0 2020-06-12T02:38:45.759 INFO:tasks.rados.rados.0.smithi107.stdout:9053: done (1 left) 2020-06-12T02:38:45.759 INFO:tasks.rados.rados.0.smithi107.stdout:9054: delete oid 7 current snap is 0 2020-06-12T02:38:45.759 INFO:tasks.rados.rados.0.smithi107.stdout:9050: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:45.760 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:38:45.760 INFO:tasks.rados.rados.0.smithi107.stdout:9050: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:45.760 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1687 (ObjNum 9049 snap 0 seq_num 9049) dirty exists 2020-06-12T02:38:45.761 INFO:tasks.rados.rados.0.smithi107.stdout:9050: left oid 8 (ObjNum 9049 snap 0 seq_num 9049) 2020-06-12T02:38:45.761 INFO:tasks.rados.rados.0.smithi107.stdout:9050: done (1 left) 2020-06-12T02:38:45.761 INFO:tasks.rados.rados.0.smithi107.stdout:9054: done (0 left) 2020-06-12T02:38:45.761 INFO:tasks.rados.rados.0.smithi107.stdout:9055: delete oid 19 current snap is 0 2020-06-12T02:38:45.763 INFO:tasks.rados.rados.0.smithi107.stdout:9055: done (0 left) 2020-06-12T02:38:45.763 INFO:tasks.rados.rados.0.smithi107.stdout:9056: delete oid 10 current snap is 0 2020-06-12T02:38:45.765 INFO:tasks.rados.rados.0.smithi107.stdout:9056: done (0 left) 2020-06-12T02:38:45.765 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:45.765 INFO:tasks.rados.rados.0.smithi107.stdout:9057: seq_num 9056 ranges {0=475136} 2020-06-12T02:38:45.769 INFO:tasks.rados.rados.0.smithi107.stdout:9057: writing smithi10729407-17 from 0 to 475136 tid 1 2020-06-12T02:38:45.769 INFO:tasks.rados.rados.0.smithi107.stdout:9058: delete oid 1 current snap is 0 2020-06-12T02:38:45.772 INFO:tasks.rados.rados.0.smithi107.stdout:9058: done (1 left) 2020-06-12T02:38:45.772 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:45.773 INFO:tasks.rados.rados.0.smithi107.stdout:9059: seq_num 9058 ranges {106496=507904} 2020-06-12T02:38:45.777 INFO:tasks.rados.rados.0.smithi107.stdout:9059: writing smithi10729407-5 from 106496 to 614400 tid 1 2020-06-12T02:38:45.778 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:38:45.778 INFO:tasks.rados.rados.0.smithi107.stdout:9060: seq_num 9059 ranges {0=524288} 2020-06-12T02:38:45.781 INFO:tasks.rados.rados.0.smithi107.stdout:9060: writing smithi10729407-1 from 0 to 524288 tid 1 2020-06-12T02:38:45.781 INFO:tasks.rados.rados.0.smithi107.stdout:9061: delete oid 15 current snap is 0 2020-06-12T02:38:45.782 INFO:tasks.rados.rados.0.smithi107.stdout:9057: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:45.782 INFO:tasks.rados.rados.0.smithi107.stdout:9057: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:45.782 INFO:tasks.rados.rados.0.smithi107.stdout:9057: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:45.782 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1699 (ObjNum 9056 snap 0 seq_num 9056) dirty exists 2020-06-12T02:38:45.782 INFO:tasks.rados.rados.0.smithi107.stdout:9057: left oid 17 (ObjNum 9056 snap 0 seq_num 9056) 2020-06-12T02:38:45.786 INFO:tasks.rados.rados.0.smithi107.stdout:9059: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:45.786 INFO:tasks.rados.rados.0.smithi107.stdout:9059: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:45.788 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:38:45.788 INFO:tasks.rados.rados.0.smithi107.stdout:9057: done (3 left) 2020-06-12T02:38:45.789 INFO:tasks.rados.rados.0.smithi107.stdout:9061: done (2 left) 2020-06-12T02:38:45.789 INFO:tasks.rados.rados.0.smithi107.stdout:9062: delete oid 17 current snap is 0 2020-06-12T02:38:45.790 INFO:tasks.rados.rados.0.smithi107.stdout:9060: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:45.790 INFO:tasks.rados.rados.0.smithi107.stdout:9059: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:45.791 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 882 (ObjNum 9058 snap 0 seq_num 9058) dirty exists 2020-06-12T02:38:45.791 INFO:tasks.rados.rados.0.smithi107.stdout:9059: left oid 5 (ObjNum 9058 snap 0 seq_num 9058) 2020-06-12T02:38:45.791 INFO:tasks.rados.rados.0.smithi107.stdout:9060: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:45.792 INFO:tasks.rados.rados.0.smithi107.stdout:9059: done (2 left) 2020-06-12T02:38:45.793 INFO:tasks.rados.rados.0.smithi107.stdout:9062: done (1 left) 2020-06-12T02:38:45.793 INFO:tasks.rados.rados.0.smithi107.stdout:9063: delete oid 8 current snap is 0 2020-06-12T02:38:45.793 INFO:tasks.rados.rados.0.smithi107.stdout:9060: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:45.793 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1650 (ObjNum 9059 snap 0 seq_num 9059) dirty exists 2020-06-12T02:38:45.794 INFO:tasks.rados.rados.0.smithi107.stdout:9060: left oid 1 (ObjNum 9059 snap 0 seq_num 9059) 2020-06-12T02:38:45.796 INFO:tasks.rados.rados.0.smithi107.stdout:9060: done (1 left) 2020-06-12T02:38:45.796 INFO:tasks.rados.rados.0.smithi107.stdout:9063: done (0 left) 2020-06-12T02:38:45.797 INFO:tasks.rados.rados.0.smithi107.stdout:9064: delete oid 13 current snap is 0 2020-06-12T02:38:45.799 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:45.799 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:45.800 INFO:tasks.rados.rados.0.smithi107.stdout:9064: done (0 left) 2020-06-12T02:38:45.800 INFO:tasks.rados.rados.0.smithi107.stdout:9065: delete oid 6 current snap is 0 2020-06-12T02:38:45.802 INFO:tasks.rados.rados.0.smithi107.stdout:9065: done (0 left) 2020-06-12T02:38:45.802 INFO:tasks.rados.rados.0.smithi107.stdout:9066: delete oid 4 current snap is 0 2020-06-12T02:38:45.804 INFO:tasks.rados.rados.0.smithi107.stdout:9066: done (0 left) 2020-06-12T02:38:45.804 INFO:tasks.rados.rados.0.smithi107.stdout:9067: delete oid 4 current snap is 0 2020-06-12T02:38:45.809 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-06-12T02:38:45.811 INFO:tasks.rados.rados.0.smithi107.stdout:9067: done (0 left) 2020-06-12T02:38:45.811 INFO:tasks.rados.rados.0.smithi107.stdout:9068: delete oid 3 current snap is 0 2020-06-12T02:38:45.811 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:38:45.812 INFO:tasks.rados.rados.0.smithi107.stdout:9068: done (0 left) 2020-06-12T02:38:45.812 INFO:tasks.rados.rados.0.smithi107.stdout:9069: delete oid 17 current snap is 0 2020-06-12T02:38:45.816 INFO:tasks.rados.rados.0.smithi107.stdout:9069: done (0 left) 2020-06-12T02:38:45.817 INFO:tasks.rados.rados.0.smithi107.stdout:9070: delete oid 9 current snap is 0 2020-06-12T02:38:45.831 INFO:tasks.rados.rados.0.smithi107.stdout:9070: done (0 left) 2020-06-12T02:38:45.839 INFO:tasks.rados.rados.0.smithi107.stdout:9071: delete oid 9 current snap is 0 2020-06-12T02:38:45.839 INFO:tasks.rados.rados.0.smithi107.stdout:9071: done (0 left) 2020-06-12T02:38:45.839 INFO:tasks.rados.rados.0.smithi107.stdout:9072: delete oid 5 current snap is 0 2020-06-12T02:38:45.839 INFO:tasks.rados.rados.0.smithi107.stdout:9072: done (0 left) 2020-06-12T02:38:45.839 INFO:tasks.rados.rados.0.smithi107.stdout:9073: delete oid 2 current snap is 0 2020-06-12T02:38:45.840 INFO:tasks.rados.rados.0.smithi107.stdout:9073: done (0 left) 2020-06-12T02:38:45.840 INFO:tasks.rados.rados.0.smithi107.stdout:9074: delete oid 13 current snap is 0 2020-06-12T02:38:45.840 INFO:tasks.rados.rados.0.smithi107.stdout:9074: done (0 left) 2020-06-12T02:38:45.840 INFO:tasks.rados.rados.0.smithi107.stdout:9075: delete oid 14 current snap is 0 2020-06-12T02:38:45.840 INFO:tasks.rados.rados.0.smithi107.stdout:9075: done (0 left) 2020-06-12T02:38:45.841 INFO:tasks.rados.rados.0.smithi107.stdout:9076: delete oid 4 current snap is 0 2020-06-12T02:38:45.842 INFO:tasks.rados.rados.0.smithi107.stdout:9076: done (0 left) 2020-06-12T02:38:45.842 INFO:tasks.rados.rados.0.smithi107.stdout:9077: delete oid 17 current snap is 0 2020-06-12T02:38:45.842 INFO:tasks.rados.rados.0.smithi107.stdout:9077: done (0 left) 2020-06-12T02:38:45.843 INFO:tasks.rados.rados.0.smithi107.stdout:9078: delete oid 12 current snap is 0 2020-06-12T02:38:45.843 INFO:tasks.rados.rados.0.smithi107.stdout:9078: done (0 left) 2020-06-12T02:38:45.843 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:38:45.843 INFO:tasks.rados.rados.0.smithi107.stdout:9079: seq_num 9078 ranges {0=442368,442368=401408} 2020-06-12T02:38:45.844 INFO:tasks.rados.rados.0.smithi107.stdout:9079: writing smithi10729407-6 from 0 to 442368 tid 1 2020-06-12T02:38:45.844 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:38:45.844 INFO:tasks.rados.rados.0.smithi107.stdout:9079: writing smithi10729407-6 from 442368 to 843776 tid 2 2020-06-12T02:38:45.845 INFO:tasks.rados.rados.0.smithi107.stdout:9080: delete oid 5 current snap is 0 2020-06-12T02:38:45.849 INFO:tasks.rados.rados.0.smithi107.stdout:9080: done (1 left) 2020-06-12T02:38:45.849 INFO:tasks.rados.rados.0.smithi107.stdout:9081: delete oid 12 current snap is 0 2020-06-12T02:38:45.849 INFO:tasks.rados.rados.0.smithi107.stdout:9081: done (1 left) 2020-06-12T02:38:45.850 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:38:45.850 INFO:tasks.rados.rados.0.smithi107.stdout:9082: seq_num 9081 ranges {0=491520,491520=401408} 2020-06-12T02:38:45.850 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:38:45.848+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:38:45.860 INFO:tasks.rados.rados.0.smithi107.stdout:9082: writing smithi10729407-5 from 0 to 491520 tid 1 2020-06-12T02:38:45.860 INFO:tasks.rados.rados.0.smithi107.stdout:9082: writing smithi10729407-5 from 491520 to 892928 tid 2 2020-06-12T02:38:45.860 INFO:tasks.rados.rados.0.smithi107.stdout:9083: delete oid 19 current snap is 0 2020-06-12T02:38:45.861 INFO:tasks.rados.rados.0.smithi107.stdout:9079: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:45.861 INFO:tasks.rados.rados.0.smithi107.stdout:9079: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:45.861 INFO:tasks.rados.rados.0.smithi107.stdout:9079: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:45.861 INFO:tasks.rados.rados.0.smithi107.stdout:9079: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:45.861 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1654 (ObjNum 9078 snap 0 seq_num 9078) dirty exists 2020-06-12T02:38:45.862 INFO:tasks.rados.rados.0.smithi107.stdout:9079: left oid 6 (ObjNum 9078 snap 0 seq_num 9078) 2020-06-12T02:38:45.862 INFO:tasks.rados.rados.0.smithi107.stdout:9082: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:45.862 INFO:tasks.rados.rados.0.smithi107.stdout:9079: done (2 left) 2020-06-12T02:38:45.862 INFO:tasks.rados.rados.0.smithi107.stdout:9083: done (1 left) 2020-06-12T02:38:45.862 INFO:tasks.rados.rados.0.smithi107.stdout:9084: delete oid 15 current snap is 0 2020-06-12T02:38:45.864 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:38:45.864 INFO:tasks.rados.rados.0.smithi107.stdout:9084: done (1 left) 2020-06-12T02:38:45.865 INFO:tasks.rados.rados.0.smithi107.stdout:9085: delete oid 2 current snap is 0 2020-06-12T02:38:45.865 INFO:tasks.rados.rados.0.smithi107.stdout:9082: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:45.865 INFO:tasks.rados.rados.0.smithi107.stdout:9082: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:45.869 INFO:tasks.rados.rados.0.smithi107.stdout:9082: finishing write tid 5 to smithi10729407-5 2020-06-12T02:38:45.870 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 887 (ObjNum 9081 snap 0 seq_num 9081) dirty exists 2020-06-12T02:38:45.870 INFO:tasks.rados.rados.0.smithi107.stdout:9082: left oid 5 (ObjNum 9081 snap 0 seq_num 9081) 2020-06-12T02:38:45.870 INFO:tasks.rados.rados.0.smithi107.stdout:9082: done (1 left) 2020-06-12T02:38:45.870 INFO:tasks.rados.rados.0.smithi107.stdout:9085: done (0 left) 2020-06-12T02:38:45.870 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:45.871 INFO:tasks.rados.rados.0.smithi107.stdout:9086: seq_num 9085 ranges {0=565248,565248=401408} 2020-06-12T02:38:45.871 INFO:tasks.rados.rados.0.smithi107.stdout:9086: writing smithi10729407-19 from 0 to 565248 tid 1 2020-06-12T02:38:45.872 INFO:tasks.rados.rados.0.smithi107.stdout:9086: writing smithi10729407-19 from 565248 to 966656 tid 2 2020-06-12T02:38:45.873 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:38:45.873 INFO:tasks.rados.rados.0.smithi107.stdout:9087: seq_num 9086 ranges {0=581632,581632=401408} 2020-06-12T02:38:45.876 INFO:tasks.rados.rados.0.smithi107.stdout:9087: writing smithi10729407-12 from 0 to 581632 tid 1 2020-06-12T02:38:45.878 INFO:tasks.rados.rados.0.smithi107.stdout:9087: writing smithi10729407-12 from 581632 to 983040 tid 2 2020-06-12T02:38:45.879 INFO:tasks.rados.rados.0.smithi107.stdout:9088: delete oid 3 current snap is 0 2020-06-12T02:38:45.879 INFO:tasks.rados.rados.0.smithi107.stdout:9086: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:45.883 INFO:tasks.rados.rados.0.smithi107.stdout:9088: done (2 left) 2020-06-12T02:38:45.884 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:45.884 INFO:tasks.rados.rados.0.smithi107.stdout:9089: seq_num 9088 ranges {0=614400,614400=401408} 2020-06-12T02:38:45.885 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2020-06-12T02:38:45.887 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-06-12T02:38:45.887 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-06-12T02:38:45.887 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-06-12T02:38:45.887 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-06-12T02:38:45.887 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-06-12T02:38:45.888 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-06-12T02:38:45.888 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-06-12T02:38:45.888 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-06-12T02:38:45.888 INFO:tasks.rados.rados.0.smithi107.stdout:9089: writing smithi10729407-14 from 0 to 614400 tid 1 2020-06-12T02:38:45.890 INFO:tasks.rados.rados.0.smithi107.stdout:9089: writing smithi10729407-14 from 614400 to 1015808 tid 2 2020-06-12T02:38:45.891 INFO:tasks.rados.rados.0.smithi107.stdout:9090: delete oid 11 current snap is 0 2020-06-12T02:38:45.891 INFO:tasks.rados.rados.0.smithi107.stdout:9086: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:45.892 INFO:tasks.rados.rados.0.smithi107.stdout:9086: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:45.892 INFO:tasks.rados.rados.0.smithi107.stdout:9086: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:45.892 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1678 (ObjNum 9085 snap 0 seq_num 9085) dirty exists 2020-06-12T02:38:45.892 INFO:tasks.rados.rados.0.smithi107.stdout:9086: left oid 19 (ObjNum 9085 snap 0 seq_num 9085) 2020-06-12T02:38:45.892 INFO:tasks.rados.rados.0.smithi107.stdout:9087: finishing write tid 1 to smithi10729407-12 2020-06-12T02:38:45.894 INFO:tasks.rados.rados.0.smithi107.stdout:9087: finishing write tid 2 to smithi10729407-12 2020-06-12T02:38:45.894 INFO:tasks.rados.rados.0.smithi107.stdout:9087: finishing write tid 4 to smithi10729407-12 2020-06-12T02:38:45.896 INFO:tasks.rados.rados.0.smithi107.stdout:9087: finishing write tid 5 to smithi10729407-12 2020-06-12T02:38:45.897 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1681 (ObjNum 9086 snap 0 seq_num 9086) dirty exists 2020-06-12T02:38:45.897 INFO:tasks.rados.rados.0.smithi107.stdout:9087: left oid 12 (ObjNum 9086 snap 0 seq_num 9086) 2020-06-12T02:38:45.897 INFO:tasks.rados.rados.0.smithi107.stdout:9086: done (3 left) 2020-06-12T02:38:45.897 INFO:tasks.rados.rados.0.smithi107.stdout:9087: done (2 left) 2020-06-12T02:38:45.898 INFO:tasks.rados.rados.0.smithi107.stdout:9090: done (1 left) 2020-06-12T02:38:45.898 INFO:tasks.rados.rados.0.smithi107.stdout:9091: delete oid 1 current snap is 0 2020-06-12T02:38:45.899 INFO:tasks.rados.rados.0.smithi107.stdout:9089: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:45.903 INFO:tasks.rados.rados.0.smithi107.stdout:9089: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:45.904 INFO:tasks.rados.rados.0.smithi107.stdout:9089: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:45.904 INFO:tasks.rados.rados.0.smithi107.stdout:9091: done (1 left) 2020-06-12T02:38:45.904 INFO:tasks.rados.rados.0.smithi107.stdout:9092: delete oid 7 current snap is 0 2020-06-12T02:38:45.906 INFO:tasks.rados.rados.0.smithi107.stdout:9089: finishing write tid 5 to smithi10729407-14 2020-06-12T02:38:45.907 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1575 (ObjNum 9088 snap 0 seq_num 9088) dirty exists 2020-06-12T02:38:45.907 INFO:tasks.rados.rados.0.smithi107.stdout:9089: left oid 14 (ObjNum 9088 snap 0 seq_num 9088) 2020-06-12T02:38:45.907 INFO:tasks.rados.rados.0.smithi107.stdout:9089: done (1 left) 2020-06-12T02:38:45.907 INFO:tasks.rados.rados.0.smithi107.stdout:9092: done (0 left) 2020-06-12T02:38:45.908 INFO:tasks.rados.rados.0.smithi107.stdout:9093: delete oid 5 current snap is 0 2020-06-12T02:38:45.909 INFO:tasks.rados.rados.0.smithi107.stdout:9093: done (0 left) 2020-06-12T02:38:45.909 INFO:tasks.rados.rados.0.smithi107.stdout:9094: delete oid 15 current snap is 0 2020-06-12T02:38:45.915 INFO:tasks.rados.rados.0.smithi107.stdout:9094: done (0 left) 2020-06-12T02:38:45.916 INFO:tasks.rados.rados.0.smithi107.stdout:9095: delete oid 15 current snap is 0 2020-06-12T02:38:45.916 INFO:tasks.rados.rados.0.smithi107.stdout:9095: done (0 left) 2020-06-12T02:38:45.917 INFO:tasks.rados.rados.0.smithi107.stdout:9096: delete oid 9 current snap is 0 2020-06-12T02:38:45.917 INFO:tasks.rados.rados.0.smithi107.stdout:9096: done (0 left) 2020-06-12T02:38:45.918 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:45.918 INFO:tasks.rados.rados.0.smithi107.stdout:9097: seq_num 9096 ranges {0=745472,745472=401408} 2020-06-12T02:38:45.918 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:45.915+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:45.921 INFO:tasks.rados.rados.0.smithi107.stdout:9097: writing smithi10729407-7 from 0 to 745472 tid 1 2020-06-12T02:38:45.923 INFO:tasks.rados.rados.0.smithi107.stdout:9097: writing smithi10729407-7 from 745472 to 1146880 tid 2 2020-06-12T02:38:45.923 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:45.924 INFO:tasks.rados.rados.0.smithi107.stdout:9098: seq_num 9097 ranges {0=761856,761856=401408} 2020-06-12T02:38:45.928 INFO:tasks.rados.rados.0.smithi107.stdout:9098: writing smithi10729407-8 from 0 to 761856 tid 1 2020-06-12T02:38:45.930 INFO:tasks.rados.rados.0.smithi107.stdout:9098: writing smithi10729407-8 from 761856 to 1163264 tid 2 2020-06-12T02:38:45.930 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:45.931 INFO:tasks.rados.rados.0.smithi107.stdout:9099: seq_num 9098 ranges {0=778240,778240=401408} 2020-06-12T02:38:45.934 INFO:tasks.rados.rados.0.smithi107.stdout:9099: writing smithi10729407-5 from 0 to 778240 tid 1 2020-06-12T02:38:45.937 INFO:tasks.rados.rados.0.smithi107.stdout:9099: writing smithi10729407-5 from 778240 to 1179648 tid 2 2020-06-12T02:38:45.937 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:45.937 INFO:tasks.rados.rados.0.smithi107.stdout:9100: seq_num 9099 ranges {0=794624,794624=401408} 2020-06-12T02:38:45.942 INFO:tasks.rados.rados.0.smithi107.stdout:9100: writing smithi10729407-9 from 0 to 794624 tid 1 2020-06-12T02:38:45.945 INFO:tasks.rados.rados.0.smithi107.stdout:9100: writing smithi10729407-9 from 794624 to 1196032 tid 2 2020-06-12T02:38:45.945 INFO:tasks.rados.rados.0.smithi107.stdout:9101: delete oid 20 current snap is 0 2020-06-12T02:38:45.945 INFO:tasks.rados.rados.0.smithi107.stdout:9097: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:45.945 INFO:tasks.rados.rados.0.smithi107.stdout:9097: finishing write tid 2 to smithi10729407-7 2020-06-12T02:38:45.946 INFO:tasks.rados.rados.0.smithi107.stdout:9097: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:45.946 INFO:tasks.rados.rados.0.smithi107.stdout:9097: finishing write tid 5 to smithi10729407-7 2020-06-12T02:38:45.946 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1628 (ObjNum 9096 snap 0 seq_num 9096) dirty exists 2020-06-12T02:38:45.946 INFO:tasks.rados.rados.0.smithi107.stdout:9097: left oid 7 (ObjNum 9096 snap 0 seq_num 9096) 2020-06-12T02:38:45.947 INFO:tasks.rados.rados.0.smithi107.stdout:9098: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:45.947 INFO:tasks.rados.rados.0.smithi107.stdout:9099: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:45.950 INFO:tasks.rados.rados.0.smithi107.stdout:9099: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:45.951 INFO:tasks.rados.rados.0.smithi107.stdout:9099: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:45.951 INFO:tasks.rados.rados.0.smithi107.stdout:9098: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:45.951 INFO:tasks.rados.rados.0.smithi107.stdout:9098: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:45.959 INFO:tasks.rados.rados.0.smithi107.stdout:9097: done (4 left) 2020-06-12T02:38:45.959 INFO:tasks.rados.rados.0.smithi107.stdout:9101: done (3 left) 2020-06-12T02:38:45.959 INFO:tasks.rados.rados.0.smithi107.stdout:9102: delete oid 6 current snap is 0 2020-06-12T02:38:45.960 INFO:tasks.rados.rados.0.smithi107.stdout:9100: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:45.960 INFO:tasks.rados.rados.0.smithi107.stdout:9099: finishing write tid 5 to smithi10729407-5 2020-06-12T02:38:45.960 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 891 (ObjNum 9098 snap 0 seq_num 9098) dirty exists 2020-06-12T02:38:45.961 INFO:tasks.rados.rados.0.smithi107.stdout:9099: left oid 5 (ObjNum 9098 snap 0 seq_num 9098) 2020-06-12T02:38:45.961 INFO:tasks.rados.rados.0.smithi107.stdout:9098: finishing write tid 5 to smithi10729407-8 2020-06-12T02:38:45.962 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1695 (ObjNum 9097 snap 0 seq_num 9097) dirty exists 2020-06-12T02:38:45.962 INFO:tasks.rados.rados.0.smithi107.stdout:9098: left oid 8 (ObjNum 9097 snap 0 seq_num 9097) 2020-06-12T02:38:45.962 INFO:tasks.rados.rados.0.smithi107.stdout:9100: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:45.963 INFO:tasks.rados.rados.0.smithi107.stdout:9098: done (3 left) 2020-06-12T02:38:45.963 INFO:tasks.rados.rados.0.smithi107.stdout:9099: done (2 left) 2020-06-12T02:38:45.963 INFO:tasks.rados.rados.0.smithi107.stdout:9102: done (1 left) 2020-06-12T02:38:45.964 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:45.964 INFO:tasks.rados.rados.0.smithi107.stdout:9103: seq_num 9102 ranges {0=442368,442368=663552,1105920=139264} 2020-06-12T02:38:45.964 INFO:tasks.rados.rados.0.smithi107.stdout:9103: writing smithi10729407-4 from 0 to 442368 tid 1 2020-06-12T02:38:45.965 INFO:tasks.rados.rados.0.smithi107.stdout:9103: writing smithi10729407-4 from 442368 to 1105920 tid 2 2020-06-12T02:38:45.965 INFO:tasks.rados.rados.0.smithi107.stdout:9103: writing smithi10729407-4 from 1105920 to 1245184 tid 3 2020-06-12T02:38:45.966 INFO:tasks.rados.rados.0.smithi107.stdout:9104: delete oid 5 current snap is 0 2020-06-12T02:38:45.966 INFO:tasks.rados.rados.0.smithi107.stdout:9100: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:45.966 INFO:tasks.rados.rados.0.smithi107.stdout:9100: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:45.966 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1696 (ObjNum 9099 snap 0 seq_num 9099) dirty exists 2020-06-12T02:38:45.967 INFO:tasks.rados.rados.0.smithi107.stdout:9100: left oid 9 (ObjNum 9099 snap 0 seq_num 9099) 2020-06-12T02:38:45.967 INFO:tasks.rados.rados.0.smithi107.stdout:9103: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:45.967 INFO:tasks.rados.rados.0.smithi107.stdout:9100: done (2 left) 2020-06-12T02:38:45.968 INFO:tasks.rados.rados.0.smithi107.stdout:9104: done (1 left) 2020-06-12T02:38:45.968 INFO:tasks.rados.rados.0.smithi107.stdout:9105: delete oid 14 current snap is 0 2020-06-12T02:38:45.968 INFO:tasks.rados.rados.0.smithi107.stdout:9105: done (1 left) 2020-06-12T02:38:45.969 INFO:tasks.rados.rados.0.smithi107.stdout:9106: delete oid 1 current snap is 0 2020-06-12T02:38:45.972 INFO:tasks.rados.rados.0.smithi107.stdout:9106: done (1 left) 2020-06-12T02:38:45.972 INFO:tasks.rados.rados.0.smithi107.stdout:9107: delete oid 6 current snap is 0 2020-06-12T02:38:45.974 INFO:tasks.rados.rados.0.smithi107.stdout:9107: done (1 left) 2020-06-12T02:38:45.975 INFO:tasks.rados.rados.0.smithi107.stdout:9108: delete oid 17 current snap is 0 2020-06-12T02:38:45.978 INFO:tasks.rados.rados.0.smithi107.stdout:9103: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:45.979 INFO:tasks.rados.rados.0.smithi107.stdout:9103: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:45.979 INFO:tasks.rados.rados.0.smithi107.stdout:9103: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:45.981 INFO:tasks.rados.rados.0.smithi107.stdout:9103: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:45.982 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1709 (ObjNum 9102 snap 0 seq_num 9102) dirty exists 2020-06-12T02:38:45.983 INFO:tasks.rados.rados.0.smithi107.stdout:9103: left oid 4 (ObjNum 9102 snap 0 seq_num 9102) 2020-06-12T02:38:45.984 INFO:tasks.rados.rados.0.smithi107.stdout:9103: done (1 left) 2020-06-12T02:38:45.984 INFO:tasks.rados.rados.0.smithi107.stdout:9108: done (0 left) 2020-06-12T02:38:45.984 INFO:tasks.rados.rados.0.smithi107.stdout:9109: delete oid 18 current snap is 0 2020-06-12T02:38:45.986 INFO:tasks.rados.rados.0.smithi107.stdout:9109: done (0 left) 2020-06-12T02:38:45.986 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:38:45.986 INFO:tasks.rados.rados.0.smithi107.stdout:9110: seq_num 9109 ranges {0=565248,565248=704512,1269760=98304} 2020-06-12T02:38:45.989 INFO:tasks.rados.rados.0.smithi107.stdout:9110: writing smithi10729407-13 from 0 to 565248 tid 1 2020-06-12T02:38:45.993 INFO:tasks.rados.rados.0.smithi107.stdout:9110: writing smithi10729407-13 from 565248 to 1269760 tid 2 2020-06-12T02:38:45.994 INFO:tasks.rados.rados.0.smithi107.stdout:9110: writing smithi10729407-13 from 1269760 to 1368064 tid 3 2020-06-12T02:38:45.994 INFO:tasks.rados.rados.0.smithi107.stdout:9111: delete oid 20 current snap is 0 2020-06-12T02:38:45.996 INFO:tasks.rados.rados.0.smithi107.stdout:9111: done (1 left) 2020-06-12T02:38:45.996 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:45.997 INFO:tasks.rados.rados.0.smithi107.stdout:9112: seq_num 9111 ranges {0=598016,598016=466944,1064960=335872} 2020-06-12T02:38:46.000 INFO:tasks.rados.rados.0.smithi107.stdout:9112: writing smithi10729407-1 from 0 to 598016 tid 1 2020-06-12T02:38:46.002 INFO:tasks.rados.rados.0.smithi107.stdout:9112: writing smithi10729407-1 from 598016 to 1064960 tid 2 2020-06-12T02:38:46.005 INFO:tasks.rados.rados.0.smithi107.stdout:9112: writing smithi10729407-1 from 1064960 to 1400832 tid 3 2020-06-12T02:38:46.005 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:46.005 INFO:tasks.rados.rados.0.smithi107.stdout:9113: seq_num 9112 ranges {966656=614400,1581056=753664,2334720=49152} 2020-06-12T02:38:46.014 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:46.014 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:38:46.014 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:38:46.014 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:38:46.015 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.015 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:4975 3.as0 3:5732055a:::smithi10729407-8:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e30)", 2020-06-12T02:38:46.015 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:43.721794+0000", 2020-06-12T02:38:46.015 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 62.288228512000003, 2020-06-12T02:38:46.015 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.1825135, 2020-06-12T02:38:46.016 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:46.016 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:46.016 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:46.016 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:46.017 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:46.017 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 4975 2020-06-12T02:38:46.017 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.018 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:46.018 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.018 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:46.018 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721794+0000", 2020-06-12T02:38:46.019 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:46.019 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.019 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.019 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:46.019 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721794+0000", 2020-06-12T02:38:46.020 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4542e-05 2020-06-12T02:38:46.020 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.020 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.022 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:46.022 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721818+0000", 2020-06-12T02:38:46.023 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9169999999999997e-06 2020-06-12T02:38:46.023 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.023 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.023 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:46.023 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721823+0000", 2020-06-12T02:38:46.024 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1184e-05 2020-06-12T02:38:46.024 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.024 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.024 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:46.024 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721834+0000", 2020-06-12T02:38:46.025 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.254e-06 2020-06-12T02:38:46.025 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.025 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.025 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:46.025 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721840+0000", 2020-06-12T02:38:46.026 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1075999999999998e-05 2020-06-12T02:38:46.026 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.026 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.026 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.026 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721881+0000", 2020-06-12T02:38:46.026 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7416999999999995e-05 2020-06-12T02:38:46.027 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.027 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.027 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:46.027 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721948+0000", 2020-06-12T02:38:46.028 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.18235910999999999 2020-06-12T02:38:46.028 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.028 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.028 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:46.029 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.904307+0000", 2020-06-12T02:38:46.029 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.1825135 2020-06-12T02:38:46.029 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.029 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:46.029 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.030 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.030 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.030 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11914 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e56)", 2020-06-12T02:38:46.030 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:23.508790+0000", 2020-06-12T02:38:46.030 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 22.501231659999998, 2020-06-12T02:38:46.030 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026541928999999999, 2020-06-12T02:38:46.031 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:46.031 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:46.031 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:46.031 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:46.031 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:46.032 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11914 2020-06-12T02:38:46.032 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.032 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:46.032 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.032 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:46.033 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508790+0000", 2020-06-12T02:38:46.033 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:46.033 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.033 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.033 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:46.033 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508790+0000", 2020-06-12T02:38:46.034 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0872999999999999e-05 2020-06-12T02:38:46.034 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.034 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.034 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:46.034 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508841+0000", 2020-06-12T02:38:46.035 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9179999999999994e-06 2020-06-12T02:38:46.035 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.035 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.035 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:46.035 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508849+0000", 2020-06-12T02:38:46.035 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5551999999999999e-05 2020-06-12T02:38:46.036 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.036 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.036 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:46.036 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508865+0000", 2020-06-12T02:38:46.036 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0629999999999996e-06 2020-06-12T02:38:46.037 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.037 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.037 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:46.037 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508870+0000", 2020-06-12T02:38:46.037 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010597488 2020-06-12T02:38:46.038 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.038 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.038 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.046 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.519467+0000", 2020-06-12T02:38:46.046 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.6083999999999997e-05 2020-06-12T02:38:46.046 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.046 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.046 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:46.047 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.519563+0000", 2020-06-12T02:38:46.047 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0078077800000000003 2020-06-12T02:38:46.047 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.047 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.047 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.048 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.527371+0000", 2020-06-12T02:38:46.048 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6198999999999993e-05 2020-06-12T02:38:46.051 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.052 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.052 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:46.052 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.527447+0000", 2020-06-12T02:38:46.052 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0078849720000000005 2020-06-12T02:38:46.056 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.056 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.056 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:46.056 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.535332+0000", 2020-06-12T02:38:46.057 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026541928999999999 2020-06-12T02:38:46.057 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.057 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:46.057 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.058 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.058 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.058 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13705 3.as0 3:5cb723d2:::smithi10729407-15:head [append 0~745472,stat] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:38:46.058 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.390499+0000", 2020-06-12T02:38:46.059 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 17.619522967000002, 2020-06-12T02:38:46.059 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.067319320000000002, 2020-06-12T02:38:46.059 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:46.059 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:46.059 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:46.060 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:46.060 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:46.060 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13705 2020-06-12T02:38:46.060 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.060 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:46.060 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.061 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:46.061 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.390499+0000", 2020-06-12T02:38:46.061 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:46.061 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.061 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.062 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:46.062 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.390499+0000", 2020-06-12T02:38:46.062 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00066732599999999999 2020-06-12T02:38:46.062 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.062 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.063 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:46.063 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391167+0000", 2020-06-12T02:38:46.063 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3609999999999997e-06 2020-06-12T02:38:46.063 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.063 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.063 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:46.064 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391173+0000", 2020-06-12T02:38:46.064 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0399e-05 2020-06-12T02:38:46.064 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.064 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.064 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:46.065 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391183+0000", 2020-06-12T02:38:46.065 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6480000000000002e-06 2020-06-12T02:38:46.065 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.065 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.065 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:46.065 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391188+0000", 2020-06-12T02:38:46.066 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4150999999999999e-05 2020-06-12T02:38:46.066 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.066 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.066 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.066 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391222+0000", 2020-06-12T02:38:46.067 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9450999999999996e-05 2020-06-12T02:38:46.067 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.067 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.067 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:46.067 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391292+0000", 2020-06-12T02:38:46.068 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054214300000000003 2020-06-12T02:38:46.068 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.068 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.068 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:46.068 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391834+0000", 2020-06-12T02:38:46.068 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0037312069999999998 2020-06-12T02:38:46.069 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.069 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.069 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:46.070 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.395565+0000", 2020-06-12T02:38:46.070 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001050194 2020-06-12T02:38:46.070 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.070 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.070 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:46.070 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396615+0000", 2020-06-12T02:38:46.071 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.061203439999999998 2020-06-12T02:38:46.071 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.071 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.071 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:46.071 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.457819+0000", 2020-06-12T02:38:46.072 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.067319320000000002 2020-06-12T02:38:46.072 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.072 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:46.072 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.072 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.073 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.073 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13706 3.as0 3:5cb723d2:::smithi10729407-15:head [append 0~434176] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:38:46.073 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.393133+0000", 2020-06-12T02:38:46.073 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 17.616888699, 2020-06-12T02:38:46.073 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.078158561000000001, 2020-06-12T02:38:46.074 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:46.074 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:46.074 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:46.074 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:46.074 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:46.075 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13706 2020-06-12T02:38:46.075 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.075 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:46.075 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.076 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:46.076 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393133+0000", 2020-06-12T02:38:46.076 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:46.076 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.076 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.076 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:46.077 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393133+0000", 2020-06-12T02:38:46.077 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00050415499999999995 2020-06-12T02:38:46.077 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.077 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.077 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:46.078 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393638+0000", 2020-06-12T02:38:46.078 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5010000000000004e-06 2020-06-12T02:38:46.078 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.078 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.078 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:46.079 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393646+0000", 2020-06-12T02:38:46.079 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4183e-05 2020-06-12T02:38:46.079 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.079 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.080 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:46.080 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393660+0000", 2020-06-12T02:38:46.080 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6069999999999996e-06 2020-06-12T02:38:46.080 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.080 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.081 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:46.081 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393667+0000", 2020-06-12T02:38:46.081 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001363569 2020-06-12T02:38:46.081 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.081 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.081 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.082 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.395030+0000", 2020-06-12T02:38:46.082 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012127 2020-06-12T02:38:46.082 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.082 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.082 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:46.083 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.395152+0000", 2020-06-12T02:38:46.083 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.063640503000000001 2020-06-12T02:38:46.083 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.083 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.084 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.084 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458792+0000", 2020-06-12T02:38:46.084 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010636 2020-06-12T02:38:46.084 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.084 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.084 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:46.085 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458899+0000", 2020-06-12T02:38:46.085 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00076074200000000002 2020-06-12T02:38:46.085 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.085 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.085 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:46.086 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459659+0000", 2020-06-12T02:38:46.086 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00772984 2020-06-12T02:38:46.086 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.086 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.086 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:46.087 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.467389+0000", 2020-06-12T02:38:46.087 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003815751 2020-06-12T02:38:46.087 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.087 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.087 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:46.088 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471205+0000", 2020-06-12T02:38:46.088 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.708e-05 2020-06-12T02:38:46.088 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.088 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.088 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:46.089 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471292+0000", 2020-06-12T02:38:46.089 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.078158561000000001 2020-06-12T02:38:46.089 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.089 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:46.090 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.090 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.090 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.090 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13707 3.as0 3:5cb723d2:::smithi10729407-15:head [append 0~565248] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:38:46.091 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.396173+0000", 2020-06-12T02:38:46.091 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 17.613848755999999, 2020-06-12T02:38:46.091 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.075298341000000005, 2020-06-12T02:38:46.091 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:46.091 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:46.092 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:46.092 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:46.092 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:46.093 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13707 2020-06-12T02:38:46.094 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.094 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:46.094 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.094 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:46.094 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396173+0000", 2020-06-12T02:38:46.095 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:46.095 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.095 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.095 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:46.095 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396173+0000", 2020-06-12T02:38:46.096 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.061779787000000003 2020-06-12T02:38:46.096 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.096 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.096 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:46.097 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.457953+0000", 2020-06-12T02:38:46.097 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5830000000000007e-06 2020-06-12T02:38:46.097 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.097 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.097 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:46.098 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.457962+0000", 2020-06-12T02:38:46.098 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3029e-05 2020-06-12T02:38:46.098 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.098 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.098 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:46.099 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.457975+0000", 2020-06-12T02:38:46.099 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.233e-06 2020-06-12T02:38:46.099 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.100 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.100 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:46.100 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.457979+0000", 2020-06-12T02:38:46.100 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0044352510000000003 2020-06-12T02:38:46.100 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.101 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.101 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.101 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.462414+0000", 2020-06-12T02:38:46.103 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012825799999999999 2020-06-12T02:38:46.104 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.104 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.105 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:46.105 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.462543+0000", 2020-06-12T02:38:46.105 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00077227899999999996 2020-06-12T02:38:46.106 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.106 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.106 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:46.106 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.463315+0000", 2020-06-12T02:38:46.106 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0079912390000000007 2020-06-12T02:38:46.107 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.107 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.107 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:46.107 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471306+0000", 2020-06-12T02:38:46.107 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000118674 2020-06-12T02:38:46.107 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.108 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.108 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:46.108 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471425+0000", 2020-06-12T02:38:46.108 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7008000000000001e-05 2020-06-12T02:38:46.108 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.109 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.109 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:46.109 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471472+0000", 2020-06-12T02:38:46.109 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.075298341000000005 2020-06-12T02:38:46.109 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.110 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:46.110 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.110 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.110 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.111 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWriteReply(3.as0 58/53 ECSubWriteReply(tid=1890, last_complete=58'1184, committed=1, applied=1))", 2020-06-12T02:38:46.111 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.396464+0000", 2020-06-12T02:38:46.111 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 17.613557877000002, 2020-06-12T02:38:46.111 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.062264803000000001, 2020-06-12T02:38:46.111 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:46.111 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:38:46.112 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:46.112 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.112 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:46.112 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396464+0000", 2020-06-12T02:38:46.112 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:46.113 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.113 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.113 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:46.113 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396464+0000", 2020-06-12T02:38:46.113 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9425999999999999e-05 2020-06-12T02:38:46.114 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.114 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.114 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:46.120 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396494+0000", 2020-06-12T02:38:46.121 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1610000000000002e-06 2020-06-12T02:38:46.121 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.121 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.122 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:46.123 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396499+0000", 2020-06-12T02:38:46.124 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2789999999999998e-06 2020-06-12T02:38:46.124 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.124 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.124 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:46.124 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396507+0000", 2020-06-12T02:38:46.125 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1389999999999997e-06 2020-06-12T02:38:46.125 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.125 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.125 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:46.125 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396511+0000", 2020-06-12T02:38:46.126 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2199000000000001e-05 2020-06-12T02:38:46.126 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.126 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.126 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.126 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396543+0000", 2020-06-12T02:38:46.127 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.062185599000000001 2020-06-12T02:38:46.127 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.127 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.127 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:46.127 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458729+0000", 2020-06-12T02:38:46.128 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.062264803000000001 2020-06-12T02:38:46.128 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.128 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:46.128 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.128 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.128 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.129 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13710 3.as0 3:5cb723d2:::smithi10729407-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e58)", 2020-06-12T02:38:46.129 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.458457+0000", 2020-06-12T02:38:46.129 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 17.551564707000001, 2020-06-12T02:38:46.129 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028102095000000001, 2020-06-12T02:38:46.129 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:46.130 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:46.130 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:46.130 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:46.130 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:46.131 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13710 2020-06-12T02:38:46.131 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.131 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:46.131 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.131 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:46.132 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458457+0000", 2020-06-12T02:38:46.132 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:46.132 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.132 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.132 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:46.133 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458457+0000", 2020-06-12T02:38:46.133 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7650000000000001e-05 2020-06-12T02:38:46.133 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.133 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.133 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:46.133 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458485+0000", 2020-06-12T02:38:46.134 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3210000000000001e-06 2020-06-12T02:38:46.134 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.134 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.134 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:46.134 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458488+0000", 2020-06-12T02:38:46.135 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5920000000000003e-06 2020-06-12T02:38:46.135 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.135 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.135 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:46.135 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458496+0000", 2020-06-12T02:38:46.136 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8789999999999999e-06 2020-06-12T02:38:46.136 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.136 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.136 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:46.136 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458499+0000", 2020-06-12T02:38:46.136 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013081152 2020-06-12T02:38:46.137 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.137 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.137 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.137 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471580+0000", 2020-06-12T02:38:46.137 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4975999999999994e-05 2020-06-12T02:38:46.138 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.138 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.138 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:46.138 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471665+0000", 2020-06-12T02:38:46.138 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0054254029999999996 2020-06-12T02:38:46.139 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.139 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.139 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.139 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.477090+0000", 2020-06-12T02:38:46.139 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2335000000000005e-05 2020-06-12T02:38:46.139 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.140 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.140 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:46.140 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.477163+0000", 2020-06-12T02:38:46.140 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0093967870000000002 2020-06-12T02:38:46.140 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.141 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.141 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:46.141 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486560+0000", 2020-06-12T02:38:46.141 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028102095000000001 2020-06-12T02:38:46.142 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.142 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:46.142 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.142 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.142 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.143 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13711 3.as0 3:5732055a:::smithi10729407-8:head [append 0~761856] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:38:46.143 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.458525+0000", 2020-06-12T02:38:46.143 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 17.551496934999999, 2020-06-12T02:38:46.143 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.027773760000000002, 2020-06-12T02:38:46.143 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:46.143 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:46.144 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:46.144 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:46.144 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:46.144 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13711 2020-06-12T02:38:46.144 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.145 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:46.145 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.145 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:46.145 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458525+0000", 2020-06-12T02:38:46.145 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:46.146 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.146 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.146 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:46.146 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458525+0000", 2020-06-12T02:38:46.146 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00089965200000000003 2020-06-12T02:38:46.146 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.147 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.147 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:46.147 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459425+0000", 2020-06-12T02:38:46.147 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4749999999999996e-06 2020-06-12T02:38:46.147 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.148 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.148 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:46.148 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459432+0000", 2020-06-12T02:38:46.148 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3043e-05 2020-06-12T02:38:46.148 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.148 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.149 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:46.149 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459445+0000", 2020-06-12T02:38:46.149 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0739999999999998e-06 2020-06-12T02:38:46.149 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.149 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.150 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:46.150 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459451+0000", 2020-06-12T02:38:46.150 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012238733999999999 2020-06-12T02:38:46.150 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.150 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.151 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.151 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471690+0000", 2020-06-12T02:38:46.151 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5640000000000003e-05 2020-06-12T02:38:46.151 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.152 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.152 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:46.152 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471746+0000", 2020-06-12T02:38:46.152 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00067859699999999997 2020-06-12T02:38:46.152 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.152 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.153 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:46.153 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.472424+0000", 2020-06-12T02:38:46.153 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050083230000000003 2020-06-12T02:38:46.153 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.154 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.154 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:46.154 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.477433+0000", 2020-06-12T02:38:46.154 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0087878039999999998 2020-06-12T02:38:46.154 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.154 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.155 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:46.155 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486221+0000", 2020-06-12T02:38:46.155 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8417999999999994e-05 2020-06-12T02:38:46.155 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.155 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.156 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:46.156 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486299+0000", 2020-06-12T02:38:46.156 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.027773760000000002 2020-06-12T02:38:46.156 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.156 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:46.157 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.157 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.157 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.157 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13712 3.as0 3:5732055a:::smithi10729407-8:head [append 0~712704] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:38:46.157 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.459508+0000", 2020-06-12T02:38:46.157 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 17.550513655, 2020-06-12T02:38:46.158 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.033995818999999997, 2020-06-12T02:38:46.158 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:46.158 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:46.158 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:46.159 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:46.159 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:46.159 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13712 2020-06-12T02:38:46.159 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.159 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:46.160 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.160 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:46.160 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459508+0000", 2020-06-12T02:38:46.160 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:46.160 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.161 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.161 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:46.161 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459508+0000", 2020-06-12T02:38:46.161 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00075436999999999998 2020-06-12T02:38:46.162 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.162 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.162 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:46.162 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460263+0000", 2020-06-12T02:38:46.162 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4370000000000001e-06 2020-06-12T02:38:46.162 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.163 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.163 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:46.163 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460270+0000", 2020-06-12T02:38:46.163 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2500000000000001e-05 2020-06-12T02:38:46.163 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.164 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.164 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:46.164 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460283+0000", 2020-06-12T02:38:46.164 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2580000000000003e-06 2020-06-12T02:38:46.164 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.165 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.165 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:46.165 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460288+0000", 2020-06-12T02:38:46.165 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017177595 2020-06-12T02:38:46.165 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.165 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.166 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.166 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.477466+0000", 2020-06-12T02:38:46.166 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00019751600000000001 2020-06-12T02:38:46.166 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.166 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.167 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:46.167 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.477663+0000", 2020-06-12T02:38:46.167 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00088697400000000001 2020-06-12T02:38:46.167 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.167 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.167 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:46.168 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.478550+0000", 2020-06-12T02:38:46.168 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.007578787 2020-06-12T02:38:46.168 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.168 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.168 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:46.169 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486129+0000", 2020-06-12T02:38:46.169 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0072972310000000004 2020-06-12T02:38:46.169 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.169 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.169 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:46.169 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493426+0000", 2020-06-12T02:38:46.170 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8151000000000004e-05 2020-06-12T02:38:46.170 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.170 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.170 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:46.170 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493504+0000", 2020-06-12T02:38:46.171 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.033995818999999997 2020-06-12T02:38:46.171 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.171 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:46.171 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.172 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.172 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.172 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13713 3.as0 3:5732055a:::smithi10729407-8:head [append 0~491520] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:38:46.172 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.460333+0000", 2020-06-12T02:38:46.172 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 17.549689198999999, 2020-06-12T02:38:46.172 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.033390145000000003, 2020-06-12T02:38:46.173 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:46.173 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:46.173 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:46.173 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:46.173 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:46.174 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13713 2020-06-12T02:38:46.174 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.174 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:46.174 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.174 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:46.175 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460333+0000", 2020-06-12T02:38:46.175 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:46.175 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.176 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.176 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:46.176 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460333+0000", 2020-06-12T02:38:46.176 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00051168999999999997 2020-06-12T02:38:46.176 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.177 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.177 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:46.177 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460845+0000", 2020-06-12T02:38:46.177 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2930000000000003e-06 2020-06-12T02:38:46.177 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.178 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.178 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:46.178 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460852+0000", 2020-06-12T02:38:46.178 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3054000000000001e-05 2020-06-12T02:38:46.178 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.178 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.179 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:46.179 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460865+0000", 2020-06-12T02:38:46.179 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6130000000000003e-06 2020-06-12T02:38:46.179 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.179 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.180 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:46.180 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460871+0000", 2020-06-12T02:38:46.180 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025753870000000002 2020-06-12T02:38:46.180 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.180 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.181 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.181 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486624+0000", 2020-06-12T02:38:46.181 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2739999999999998e-05 2020-06-12T02:38:46.181 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.181 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.182 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:46.182 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486697+0000", 2020-06-12T02:38:46.182 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00063578799999999998 2020-06-12T02:38:46.182 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.182 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.183 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:46.183 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.487333+0000", 2020-06-12T02:38:46.183 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0043011610000000004 2020-06-12T02:38:46.183 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.183 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.184 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:46.184 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.491634+0000", 2020-06-12T02:38:46.184 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0020130949999999999 2020-06-12T02:38:46.184 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.184 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.185 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:46.185 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493647+0000", 2020-06-12T02:38:46.185 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5840999999999994e-05 2020-06-12T02:38:46.185 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.185 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.185 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:46.186 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493723+0000", 2020-06-12T02:38:46.186 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.033390145000000003 2020-06-12T02:38:46.186 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.186 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:46.186 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.187 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.187 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.187 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13714 3.as0 3:5732055a:::smithi10729407-8:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:38:46.187 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.460921+0000", 2020-06-12T02:38:46.188 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 17.549100676999998, 2020-06-12T02:38:46.188 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.032981457999999998, 2020-06-12T02:38:46.189 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:46.189 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:46.189 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:46.190 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:46.190 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:46.190 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13714 2020-06-12T02:38:46.190 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.190 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:46.191 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.191 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:46.191 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460921+0000", 2020-06-12T02:38:46.191 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:46.191 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.191 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.192 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:46.192 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460921+0000", 2020-06-12T02:38:46.192 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6996999999999995e-05 2020-06-12T02:38:46.192 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.192 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.193 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:46.193 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460998+0000", 2020-06-12T02:38:46.193 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4529999999999999e-06 2020-06-12T02:38:46.193 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.193 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.194 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:46.194 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461005+0000", 2020-06-12T02:38:46.194 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1722000000000001e-05 2020-06-12T02:38:46.194 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.194 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.194 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:46.195 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461017+0000", 2020-06-12T02:38:46.195 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4839999999999998e-06 2020-06-12T02:38:46.195 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.195 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.196 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:46.196 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461021+0000", 2020-06-12T02:38:46.196 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.029157975999999999 2020-06-12T02:38:46.196 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.196 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.197 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.197 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.490179+0000", 2020-06-12T02:38:46.197 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010966700000000001 2020-06-12T02:38:46.197 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.197 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.198 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:46.198 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.490289+0000", 2020-06-12T02:38:46.198 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00032026800000000002 2020-06-12T02:38:46.198 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.198 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.199 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:46.199 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.490609+0000", 2020-06-12T02:38:46.199 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001102947 2020-06-12T02:38:46.199 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.199 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.199 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:46.200 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.491712+0000", 2020-06-12T02:38:46.200 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021050169999999998 2020-06-12T02:38:46.200 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.200 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.200 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:46.201 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493817+0000", 2020-06-12T02:38:46.201 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5927000000000006e-05 2020-06-12T02:38:46.201 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.201 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.201 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:46.201 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493903+0000", 2020-06-12T02:38:46.202 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.032981457999999998 2020-06-12T02:38:46.202 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.202 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:46.202 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.202 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.203 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.203 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13715 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e58)", 2020-06-12T02:38:46.203 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.461062+0000", 2020-06-12T02:38:46.203 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 17.548960037000001, 2020-06-12T02:38:46.203 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.035208338999999998, 2020-06-12T02:38:46.203 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:46.204 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:46.204 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:46.204 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:46.204 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:46.204 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13715 2020-06-12T02:38:46.205 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.205 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:46.205 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.205 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:46.205 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461062+0000", 2020-06-12T02:38:46.206 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:46.206 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.206 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.206 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:46.206 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461062+0000", 2020-06-12T02:38:46.207 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4839999999999998e-05 2020-06-12T02:38:46.207 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.207 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.207 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:46.207 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461107+0000", 2020-06-12T02:38:46.207 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4859999999999997e-06 2020-06-12T02:38:46.208 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.208 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.208 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:46.208 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461112+0000", 2020-06-12T02:38:46.208 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1742999999999999e-05 2020-06-12T02:38:46.209 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.209 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.209 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:46.209 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461124+0000", 2020-06-12T02:38:46.209 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7299999999999996e-06 2020-06-12T02:38:46.209 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.210 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.210 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:46.210 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461129+0000", 2020-06-12T02:38:46.210 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.030397368000000001 2020-06-12T02:38:46.210 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.211 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.211 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.211 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.491526+0000", 2020-06-12T02:38:46.211 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010076799999999999 2020-06-12T02:38:46.211 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.211 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.212 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:46.212 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.491627+0000", 2020-06-12T02:38:46.212 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031869849999999998 2020-06-12T02:38:46.213 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.213 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.213 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.213 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.494814+0000", 2020-06-12T02:38:46.213 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000108915 2020-06-12T02:38:46.214 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.214 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.214 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:46.214 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.494923+0000", 2020-06-12T02:38:46.214 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001347504 2020-06-12T02:38:46.215 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.215 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.215 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:46.215 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.496270+0000", 2020-06-12T02:38:46.215 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.035208338999999998 2020-06-12T02:38:46.215 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.216 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:46.216 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.216 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.216 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.216 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:16706 3.as0 3:5cb723d2:::smithi10729407-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e61)", 2020-06-12T02:38:46.217 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:37.289063+0000", 2020-06-12T02:38:46.217 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 8.7209590129999999, 2020-06-12T02:38:46.217 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026146086999999998, 2020-06-12T02:38:46.217 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:46.217 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:46.217 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:46.218 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:46.218 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:46.218 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 16706 2020-06-12T02:38:46.218 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.218 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:46.219 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.219 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:46.219 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289063+0000", 2020-06-12T02:38:46.219 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:46.219 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.220 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.220 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:46.220 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289063+0000", 2020-06-12T02:38:46.221 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3776000000000002e-05 2020-06-12T02:38:46.221 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.221 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.222 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:46.222 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289107+0000", 2020-06-12T02:38:46.222 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7039999999999999e-06 2020-06-12T02:38:46.222 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.222 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.223 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:46.223 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289113+0000", 2020-06-12T02:38:46.224 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3259e-05 2020-06-12T02:38:46.224 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.225 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.225 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:46.225 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289126+0000", 2020-06-12T02:38:46.225 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1999999999999996e-06 2020-06-12T02:38:46.226 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.226 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.226 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:46.226 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289130+0000", 2020-06-12T02:38:46.226 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0010912820000000001 2020-06-12T02:38:46.226 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.227 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.227 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.227 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.290221+0000", 2020-06-12T02:38:46.227 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3166e-05 2020-06-12T02:38:46.227 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.228 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.228 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:46.228 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.290265+0000", 2020-06-12T02:38:46.228 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015984187 2020-06-12T02:38:46.228 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.229 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.229 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.229 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.306249+0000", 2020-06-12T02:38:46.229 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4623000000000006e-05 2020-06-12T02:38:46.229 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.230 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.231 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:46.231 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.306333+0000", 2020-06-12T02:38:46.231 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0068573690000000003 2020-06-12T02:38:46.232 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.233 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.233 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.233 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.313191+0000", 2020-06-12T02:38:46.233 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7804000000000002e-05 2020-06-12T02:38:46.233 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.234 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.234 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:46.234 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.313279+0000", 2020-06-12T02:38:46.234 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001930717 2020-06-12T02:38:46.234 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.234 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.235 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:46.235 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.315209+0000", 2020-06-12T02:38:46.235 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026146086999999998 2020-06-12T02:38:46.235 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.235 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:46.236 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.236 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.236 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.236 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18051 3.as0 3:5cb723d2:::smithi10729407-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:38:46.236 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.329078+0000", 2020-06-12T02:38:46.236 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.6809444630000003, 2020-06-12T02:38:46.237 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026465103, 2020-06-12T02:38:46.237 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:46.237 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:46.237 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:46.237 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:46.238 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:46.238 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18051 2020-06-12T02:38:46.238 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.238 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:46.238 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.239 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:46.239 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329078+0000", 2020-06-12T02:38:46.239 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:46.239 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.239 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.239 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:46.240 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329078+0000", 2020-06-12T02:38:46.240 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5392000000000001e-05 2020-06-12T02:38:46.240 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.240 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.240 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:46.241 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329133+0000", 2020-06-12T02:38:46.241 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5329999999999997e-06 2020-06-12T02:38:46.241 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.241 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.241 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:46.241 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329141+0000", 2020-06-12T02:38:46.242 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3459999999999999e-05 2020-06-12T02:38:46.242 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.242 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.242 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:46.242 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329164+0000", 2020-06-12T02:38:46.243 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8170000000000001e-06 2020-06-12T02:38:46.243 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.243 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.243 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:46.243 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329171+0000", 2020-06-12T02:38:46.243 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013005208000000001 2020-06-12T02:38:46.244 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.244 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.244 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.244 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.342176+0000", 2020-06-12T02:38:46.244 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3560999999999998e-05 2020-06-12T02:38:46.245 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.245 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.245 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:46.245 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.342230+0000", 2020-06-12T02:38:46.245 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0080818490000000003 2020-06-12T02:38:46.245 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.246 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.246 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.246 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.350312+0000", 2020-06-12T02:38:46.246 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0167000000000006e-05 2020-06-12T02:38:46.246 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.247 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.247 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:46.247 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.350402+0000", 2020-06-12T02:38:46.247 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051411160000000003 2020-06-12T02:38:46.247 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.247 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.248 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:46.248 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.355543+0000", 2020-06-12T02:38:46.248 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026465103 2020-06-12T02:38:46.248 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.248 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:46.249 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.249 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.249 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.249 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18053 3.as0 3:5732055a:::smithi10729407-8:head [append 0~434176] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:38:46.249 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.334425+0000", 2020-06-12T02:38:46.250 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.6755968960000001, 2020-06-12T02:38:46.250 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.029616703000000001, 2020-06-12T02:38:46.250 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:46.250 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:46.250 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:46.250 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:46.251 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:46.251 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18053 2020-06-12T02:38:46.251 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.251 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:46.251 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.251 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:46.252 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334425+0000", 2020-06-12T02:38:46.252 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:46.252 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.252 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.252 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:46.253 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334425+0000", 2020-06-12T02:38:46.253 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047283900000000002 2020-06-12T02:38:46.253 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.253 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.253 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:46.253 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334898+0000", 2020-06-12T02:38:46.254 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9100000000000004e-06 2020-06-12T02:38:46.254 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.254 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.254 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:46.254 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334903+0000", 2020-06-12T02:38:46.255 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4260000000000003e-06 2020-06-12T02:38:46.255 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.255 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.255 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:46.255 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334911+0000", 2020-06-12T02:38:46.255 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1060000000000002e-06 2020-06-12T02:38:46.256 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.256 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.256 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:46.256 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334916+0000", 2020-06-12T02:38:46.256 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010838316000000001 2020-06-12T02:38:46.257 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.257 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.257 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.257 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.345754+0000", 2020-06-12T02:38:46.257 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000103872 2020-06-12T02:38:46.258 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.258 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.258 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:46.258 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.345858+0000", 2020-06-12T02:38:46.258 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0059798739999999996 2020-06-12T02:38:46.258 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.259 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.259 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.259 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.351838+0000", 2020-06-12T02:38:46.259 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000104724 2020-06-12T02:38:46.259 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.260 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.260 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:46.260 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.351942+0000", 2020-06-12T02:38:46.260 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000572687 2020-06-12T02:38:46.260 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.260 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.261 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:46.261 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.352515+0000", 2020-06-12T02:38:46.261 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002905641 2020-06-12T02:38:46.261 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.261 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.262 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:46.262 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.355421+0000", 2020-06-12T02:38:46.262 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0085358350000000003 2020-06-12T02:38:46.262 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.262 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.262 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:46.263 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.363956+0000", 2020-06-12T02:38:46.263 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5473e-05 2020-06-12T02:38:46.263 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.263 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.263 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:46.264 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.364042+0000", 2020-06-12T02:38:46.264 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.029616703000000001 2020-06-12T02:38:46.264 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.264 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:46.265 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.265 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.265 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.265 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18054 3.as0 3:5732055a:::smithi10729407-8:head [append 0~638976] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:38:46.265 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.338171+0000", 2020-06-12T02:38:46.266 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.6718513929999999, 2020-06-12T02:38:46.266 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.030067120999999999, 2020-06-12T02:38:46.266 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:46.266 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:46.266 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:46.267 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:46.267 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:46.267 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18054 2020-06-12T02:38:46.267 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.268 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:46.268 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.268 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:46.268 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338171+0000", 2020-06-12T02:38:46.268 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:46.269 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.269 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.269 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:46.269 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338171+0000", 2020-06-12T02:38:46.269 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00068138900000000002 2020-06-12T02:38:46.270 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.270 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.270 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:46.270 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338852+0000", 2020-06-12T02:38:46.270 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.767e-06 2020-06-12T02:38:46.270 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.271 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.271 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:46.271 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338858+0000", 2020-06-12T02:38:46.271 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8999999999999995e-06 2020-06-12T02:38:46.271 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.272 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.272 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:46.272 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338867+0000", 2020-06-12T02:38:46.272 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.262e-06 2020-06-12T02:38:46.272 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.272 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.273 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:46.273 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338871+0000", 2020-06-12T02:38:46.273 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0074839060000000002 2020-06-12T02:38:46.273 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.273 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.274 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.274 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346355+0000", 2020-06-12T02:38:46.275 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1435999999999999e-05 2020-06-12T02:38:46.275 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.276 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.276 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:46.276 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346416+0000", 2020-06-12T02:38:46.276 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0092109879999999998 2020-06-12T02:38:46.276 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.277 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.277 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.277 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.355627+0000", 2020-06-12T02:38:46.277 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7355999999999998e-05 2020-06-12T02:38:46.277 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.278 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.278 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:46.278 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.355685+0000", 2020-06-12T02:38:46.278 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00074711999999999999 2020-06-12T02:38:46.278 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.278 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.279 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:46.279 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.356432+0000", 2020-06-12T02:38:46.279 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010659572000000001 2020-06-12T02:38:46.279 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.279 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.280 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:46.280 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.367091+0000", 2020-06-12T02:38:46.280 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0010713210000000001 2020-06-12T02:38:46.280 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.280 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.280 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:46.281 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.368163+0000", 2020-06-12T02:38:46.281 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5104000000000005e-05 2020-06-12T02:38:46.281 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.281 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.281 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:46.282 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.368238+0000", 2020-06-12T02:38:46.282 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.030067120999999999 2020-06-12T02:38:46.282 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.282 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:46.282 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.283 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.283 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.283 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18055 3.as0 3:5732055a:::smithi10729407-8:head [append 0~483328] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:38:46.283 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.340740+0000", 2020-06-12T02:38:46.283 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.6692825439999996, 2020-06-12T02:38:46.283 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028669050000000001, 2020-06-12T02:38:46.284 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:46.284 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:46.284 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:46.284 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:46.284 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:46.285 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18055 2020-06-12T02:38:46.285 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.285 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:46.285 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.285 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:46.286 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.340740+0000", 2020-06-12T02:38:46.286 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:46.286 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.286 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.286 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:46.286 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.340740+0000", 2020-06-12T02:38:46.287 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00045126900000000002 2020-06-12T02:38:46.287 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.287 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.287 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:46.287 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341191+0000", 2020-06-12T02:38:46.288 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5630000000000003e-06 2020-06-12T02:38:46.288 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.288 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.288 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:46.288 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341198+0000", 2020-06-12T02:38:46.288 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4467999999999999e-05 2020-06-12T02:38:46.289 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.289 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.289 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:46.289 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341213+0000", 2020-06-12T02:38:46.289 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7540000000000003e-06 2020-06-12T02:38:46.290 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.290 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.290 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:46.290 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341219+0000", 2020-06-12T02:38:46.290 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0052272680000000002 2020-06-12T02:38:46.291 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.291 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.291 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.291 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346446+0000", 2020-06-12T02:38:46.291 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4742999999999998e-05 2020-06-12T02:38:46.292 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.292 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.292 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:46.292 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346531+0000", 2020-06-12T02:38:46.293 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013203138999999999 2020-06-12T02:38:46.293 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.293 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.293 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.293 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.359734+0000", 2020-06-12T02:38:46.294 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000100277 2020-06-12T02:38:46.294 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.294 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.294 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:46.294 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.359834+0000", 2020-06-12T02:38:46.294 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054998099999999995 2020-06-12T02:38:46.295 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.295 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.295 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:46.295 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.360384+0000", 2020-06-12T02:38:46.295 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0067087179999999998 2020-06-12T02:38:46.296 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.296 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.296 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:46.296 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.367093+0000", 2020-06-12T02:38:46.296 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0022746020000000001 2020-06-12T02:38:46.296 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.297 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.297 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:46.297 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369367+0000", 2020-06-12T02:38:46.297 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1267999999999999e-05 2020-06-12T02:38:46.297 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.298 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.298 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:46.298 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369409+0000", 2020-06-12T02:38:46.298 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028669050000000001 2020-06-12T02:38:46.298 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.298 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:46.299 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.299 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.299 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.299 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18056 3.as0 3:5732055a:::smithi10729407-8:head [append 0~49152] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:38:46.299 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.341270+0000", 2020-06-12T02:38:46.300 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.6687517549999997, 2020-06-12T02:38:46.300 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028350462999999999, 2020-06-12T02:38:46.300 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:46.300 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:46.300 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:46.300 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:46.301 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:46.301 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18056 2020-06-12T02:38:46.301 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.301 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:46.301 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.302 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:46.302 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341270+0000", 2020-06-12T02:38:46.302 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:46.302 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.302 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.303 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:46.303 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341270+0000", 2020-06-12T02:38:46.303 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000114082 2020-06-12T02:38:46.303 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.303 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.303 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:46.304 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341384+0000", 2020-06-12T02:38:46.304 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3289999999999999e-06 2020-06-12T02:38:46.304 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.304 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.304 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:46.305 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341391+0000", 2020-06-12T02:38:46.305 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3339999999999999e-05 2020-06-12T02:38:46.305 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.305 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.305 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:46.305 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341404+0000", 2020-06-12T02:38:46.306 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4e-06 2020-06-12T02:38:46.306 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.306 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.306 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:46.306 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341410+0000", 2020-06-12T02:38:46.307 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051518600000000003 2020-06-12T02:38:46.307 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.307 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.307 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.307 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346561+0000", 2020-06-12T02:38:46.307 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1662999999999994e-05 2020-06-12T02:38:46.308 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.308 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.308 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:46.308 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346643+0000", 2020-06-12T02:38:46.308 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017471605000000001 2020-06-12T02:38:46.308 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.309 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.309 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.309 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.364115+0000", 2020-06-12T02:38:46.309 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011059400000000001 2020-06-12T02:38:46.309 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.310 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.310 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:46.310 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.364225+0000", 2020-06-12T02:38:46.310 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000611262 2020-06-12T02:38:46.310 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.310 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.311 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:46.311 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.364837+0000", 2020-06-12T02:38:46.311 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023767710000000002 2020-06-12T02:38:46.311 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.311 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.312 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:46.312 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.367213+0000", 2020-06-12T02:38:46.312 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023325479999999998 2020-06-12T02:38:46.312 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.312 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.312 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:46.313 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369546+0000", 2020-06-12T02:38:46.313 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5009e-05 2020-06-12T02:38:46.313 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.313 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.313 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:46.314 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369621+0000", 2020-06-12T02:38:46.314 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028350462999999999 2020-06-12T02:38:46.314 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.314 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:46.314 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.315 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.315 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.315 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18057 3.as0 3:5732055a:::smithi10729407-8:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:38:46.315 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.341456+0000", 2020-06-12T02:38:46.315 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.6685665949999997, 2020-06-12T02:38:46.315 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028443018, 2020-06-12T02:38:46.316 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:46.316 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:46.316 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:46.316 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:46.316 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:46.316 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18057 2020-06-12T02:38:46.317 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.317 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:46.317 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.317 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:46.317 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341456+0000", 2020-06-12T02:38:46.318 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:46.318 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.323 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.323 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:46.324 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341456+0000", 2020-06-12T02:38:46.324 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6036000000000003e-05 2020-06-12T02:38:46.324 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.324 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.324 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:46.325 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341522+0000", 2020-06-12T02:38:46.325 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2210000000000002e-06 2020-06-12T02:38:46.325 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.325 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.325 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:46.326 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341528+0000", 2020-06-12T02:38:46.326 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0338e-05 2020-06-12T02:38:46.326 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.326 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.326 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:46.326 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341538+0000", 2020-06-12T02:38:46.327 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8639999999999998e-06 2020-06-12T02:38:46.327 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.327 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.327 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:46.327 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341542+0000", 2020-06-12T02:38:46.328 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051405009999999996 2020-06-12T02:38:46.328 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.328 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.328 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.328 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346683+0000", 2020-06-12T02:38:46.328 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9561999999999995e-05 2020-06-12T02:38:46.329 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.329 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.329 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:46.329 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346762+0000", 2020-06-12T02:38:46.329 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019267197999999999 2020-06-12T02:38:46.330 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.330 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.330 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.330 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.366029+0000", 2020-06-12T02:38:46.330 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2634999999999997e-05 2020-06-12T02:38:46.330 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.331 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.331 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:46.331 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.366122+0000", 2020-06-12T02:38:46.331 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00026907500000000002 2020-06-12T02:38:46.331 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.332 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.332 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:46.332 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.366391+0000", 2020-06-12T02:38:46.332 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0018939409999999999 2020-06-12T02:38:46.332 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.332 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.333 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:46.333 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.368285+0000", 2020-06-12T02:38:46.333 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001549405 2020-06-12T02:38:46.333 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.333 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.334 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:46.334 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369834+0000", 2020-06-12T02:38:46.334 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4241999999999999e-05 2020-06-12T02:38:46.334 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.334 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.334 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:46.335 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369899+0000", 2020-06-12T02:38:46.335 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028443018 2020-06-12T02:38:46.335 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.335 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:46.335 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.336 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.336 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.336 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18058 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:38:46.336 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.341580+0000", 2020-06-12T02:38:46.336 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.6684416879999997, 2020-06-12T02:38:46.336 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.030449559000000001, 2020-06-12T02:38:46.337 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:46.337 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:46.337 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:46.337 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:46.337 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:46.338 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18058 2020-06-12T02:38:46.338 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.338 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:46.338 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.338 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:46.338 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341580+0000", 2020-06-12T02:38:46.339 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:46.339 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.339 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.339 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:46.339 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341580+0000", 2020-06-12T02:38:46.340 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1185e-05 2020-06-12T02:38:46.340 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.340 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.340 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:46.340 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341622+0000", 2020-06-12T02:38:46.340 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1599999999999997e-06 2020-06-12T02:38:46.341 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.341 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.341 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:46.341 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341627+0000", 2020-06-12T02:38:46.341 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3546e-05 2020-06-12T02:38:46.342 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.342 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.342 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:46.342 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341640+0000", 2020-06-12T02:38:46.342 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9779999999999999e-06 2020-06-12T02:38:46.342 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.343 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.343 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:46.343 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341644+0000", 2020-06-12T02:38:46.343 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051433700000000004 2020-06-12T02:38:46.343 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.344 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.344 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.344 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346788+0000", 2020-06-12T02:38:46.344 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1622000000000001e-05 2020-06-12T02:38:46.344 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.344 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.345 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:46.345 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346839+0000", 2020-06-12T02:38:46.345 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020283958000000001 2020-06-12T02:38:46.345 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.345 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.346 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.346 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.367123+0000", 2020-06-12T02:38:46.346 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1997999999999997e-05 2020-06-12T02:38:46.346 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.346 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.346 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:46.347 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.367205+0000", 2020-06-12T02:38:46.347 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036875440000000001 2020-06-12T02:38:46.347 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.347 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.347 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.347 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.370893+0000", 2020-06-12T02:38:46.348 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000100604 2020-06-12T02:38:46.348 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.348 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.348 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:46.348 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.370993+0000", 2020-06-12T02:38:46.349 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0010365940000000001 2020-06-12T02:38:46.349 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.349 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.349 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:46.349 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.372030+0000", 2020-06-12T02:38:46.349 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.030449559000000001 2020-06-12T02:38:46.350 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.350 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:46.350 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.350 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.350 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:46.351 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:46.366 INFO:tasks.rados.rados.0.smithi107.stdout:9113: writing smithi10729407-19 from 966656 to 1581056 tid 1 2020-06-12T02:38:46.366 INFO:tasks.rados.rados.0.smithi107.stdout:9113: writing smithi10729407-19 from 1581056 to 2334720 tid 2 2020-06-12T02:38:46.366 INFO:tasks.rados.rados.0.smithi107.stdout:9113: writing smithi10729407-19 from 2334720 to 2383872 tid 3 2020-06-12T02:38:46.366 INFO:tasks.rados.rados.0.smithi107.stdout:9114: delete oid 20 current snap is 0 2020-06-12T02:38:46.367 INFO:tasks.rados.rados.0.smithi107.stdout:9110: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:46.367 INFO:tasks.rados.rados.0.smithi107.stdout:9110: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:46.367 INFO:tasks.rados.rados.0.smithi107.stdout:9110: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:46.367 INFO:tasks.rados.rados.0.smithi107.stdout:9110: finishing write tid 5 to smithi10729407-13 2020-06-12T02:38:46.367 INFO:tasks.rados.rados.0.smithi107.stdout:9110: finishing write tid 6 to smithi10729407-13 2020-06-12T02:38:46.367 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1634 (ObjNum 9109 snap 0 seq_num 9109) dirty exists 2020-06-12T02:38:46.368 INFO:tasks.rados.rados.0.smithi107.stdout:9110: left oid 13 (ObjNum 9109 snap 0 seq_num 9109) 2020-06-12T02:38:46.368 INFO:tasks.rados.rados.0.smithi107.stdout:9112: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:46.368 INFO:tasks.rados.rados.0.smithi107.stdout:9112: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:46.368 INFO:tasks.rados.rados.0.smithi107.stdout:9112: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:46.368 INFO:tasks.rados.rados.0.smithi107.stdout:9110: done (3 left) 2020-06-12T02:38:46.369 INFO:tasks.rados.rados.0.smithi107.stdout:9114: done (2 left) 2020-06-12T02:38:46.369 INFO:tasks.rados.rados.0.smithi107.stdout:9115: delete oid 16 current snap is 0 2020-06-12T02:38:46.369 INFO:tasks.rados.rados.0.smithi107.stdout:9112: finishing write tid 5 to smithi10729407-1 2020-06-12T02:38:46.369 INFO:tasks.rados.rados.0.smithi107.stdout:9112: finishing write tid 6 to smithi10729407-1 2020-06-12T02:38:46.369 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1662 (ObjNum 9111 snap 0 seq_num 9111) dirty exists 2020-06-12T02:38:46.370 INFO:tasks.rados.rados.0.smithi107.stdout:9112: left oid 1 (ObjNum 9111 snap 0 seq_num 9111) 2020-06-12T02:38:46.370 INFO:tasks.rados.rados.0.smithi107.stdout:9113: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:46.370 INFO:tasks.rados.rados.0.smithi107.stdout:9112: done (2 left) 2020-06-12T02:38:46.370 INFO:tasks.rados.rados.0.smithi107.stdout:9115: done (1 left) 2020-06-12T02:38:46.370 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:46.370 INFO:tasks.rados.rados.0.smithi107.stdout:9116: seq_num 9115 ranges {1368064=663552,2031616=409600,2441216=393216} 2020-06-12T02:38:46.371 INFO:tasks.rados.rados.0.smithi107.stdout:9116: writing smithi10729407-13 from 1368064 to 2031616 tid 1 2020-06-12T02:38:46.372 INFO:tasks.rados.rados.0.smithi107.stdout:9116: writing smithi10729407-13 from 2031616 to 2441216 tid 2 2020-06-12T02:38:46.372 INFO:tasks.rados.rados.0.smithi107.stdout:9116: writing smithi10729407-13 from 2441216 to 2834432 tid 3 2020-06-12T02:38:46.372 INFO:tasks.rados.rados.0.smithi107.stdout:9117: delete oid 3 current snap is 0 2020-06-12T02:38:46.373 INFO:tasks.rados.rados.0.smithi107.stdout:9113: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:46.373 INFO:tasks.rados.rados.0.smithi107.stdout:9113: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:46.373 INFO:tasks.rados.rados.0.smithi107.stdout:9113: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:46.374 INFO:tasks.rados.rados.0.smithi107.stdout:9113: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:46.374 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1685 (ObjNum 9112 snap 0 seq_num 9112) dirty exists 2020-06-12T02:38:46.374 INFO:tasks.rados.rados.0.smithi107.stdout:9113: left oid 19 (ObjNum 9112 snap 0 seq_num 9112) 2020-06-12T02:38:46.374 INFO:tasks.rados.rados.0.smithi107.stdout:9116: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:46.375 INFO:tasks.rados.rados.0.smithi107.stdout:9113: done (2 left) 2020-06-12T02:38:46.375 INFO:tasks.rados.rados.0.smithi107.stdout:9117: done (1 left) 2020-06-12T02:38:46.375 INFO:tasks.rados.rados.0.smithi107.stdout:9118: delete oid 11 current snap is 0 2020-06-12T02:38:46.375 INFO:tasks.rados.rados.0.smithi107.stdout:9118: done (1 left) 2020-06-12T02:38:46.375 INFO:tasks.rados.rados.0.smithi107.stdout:9119: delete oid 1 current snap is 0 2020-06-12T02:38:46.376 INFO:tasks.rados.rados.0.smithi107.stdout:9119: done (1 left) 2020-06-12T02:38:46.376 INFO:tasks.rados.rados.0.smithi107.stdout:9120: delete oid 7 current snap is 0 2020-06-12T02:38:46.376 INFO:tasks.rados.rados.0.smithi107.stdout:9116: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:46.376 INFO:tasks.rados.rados.0.smithi107.stdout:9116: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:46.376 INFO:tasks.rados.rados.0.smithi107.stdout:9116: finishing write tid 5 to smithi10729407-13 2020-06-12T02:38:46.377 INFO:tasks.rados.rados.0.smithi107.stdout:9116: finishing write tid 6 to smithi10729407-13 2020-06-12T02:38:46.377 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1638 (ObjNum 9115 snap 0 seq_num 9115) dirty exists 2020-06-12T02:38:46.377 INFO:tasks.rados.rados.0.smithi107.stdout:9116: left oid 13 (ObjNum 9115 snap 0 seq_num 9115) 2020-06-12T02:38:46.377 INFO:tasks.rados.rados.0.smithi107.stdout:9116: done (1 left) 2020-06-12T02:38:46.378 INFO:tasks.rados.rados.0.smithi107.stdout:9120: done (0 left) 2020-06-12T02:38:46.378 INFO:tasks.rados.rados.0.smithi107.stdout:9121: delete oid 17 current snap is 0 2020-06-12T02:38:46.378 INFO:tasks.rados.rados.0.smithi107.stdout:9121: done (0 left) 2020-06-12T02:38:46.378 INFO:tasks.rados.rados.0.smithi107.stdout:9122: delete oid 3 current snap is 0 2020-06-12T02:38:46.378 INFO:tasks.rados.rados.0.smithi107.stdout:9122: done (0 left) 2020-06-12T02:38:46.379 INFO:tasks.rados.rados.0.smithi107.stdout:9123: delete oid 18 current snap is 0 2020-06-12T02:38:46.379 INFO:tasks.rados.rados.0.smithi107.stdout:9123: done (0 left) 2020-06-12T02:38:46.379 INFO:tasks.rados.rados.0.smithi107.stdout:9124: delete oid 3 current snap is 0 2020-06-12T02:38:46.380 INFO:tasks.rados.rados.0.smithi107.stdout:9124: done (0 left) 2020-06-12T02:38:46.380 INFO:tasks.rados.rados.0.smithi107.stdout:9125: delete oid 5 current snap is 0 2020-06-12T02:38:46.380 INFO:tasks.rados.rados.0.smithi107.stdout:9125: done (0 left) 2020-06-12T02:38:46.380 INFO:tasks.rados.rados.0.smithi107.stdout:9126: delete oid 1 current snap is 0 2020-06-12T02:38:46.380 INFO:tasks.rados.rados.0.smithi107.stdout:9126: done (0 left) 2020-06-12T02:38:46.381 INFO:tasks.rados.rados.0.smithi107.stdout:9127: delete oid 11 current snap is 0 2020-06-12T02:38:46.381 INFO:tasks.rados.rados.0.smithi107.stdout:9127: done (0 left) 2020-06-12T02:38:46.381 INFO:tasks.rados.rados.0.smithi107.stdout:9128: delete oid 18 current snap is 0 2020-06-12T02:38:46.381 INFO:tasks.rados.rados.0.smithi107.stdout:9128: done (0 left) 2020-06-12T02:38:46.381 INFO:tasks.rados.rados.0.smithi107.stdout:9129: delete oid 5 current snap is 0 2020-06-12T02:38:46.382 INFO:tasks.rados.rados.0.smithi107.stdout:9129: done (0 left) 2020-06-12T02:38:46.382 INFO:tasks.rados.rados.0.smithi107.stdout:9130: delete oid 3 current snap is 0 2020-06-12T02:38:46.382 INFO:tasks.rados.rados.0.smithi107.stdout:9130: done (0 left) 2020-06-12T02:38:46.383 INFO:tasks.rados.rados.0.smithi107.stdout:9131: delete oid 17 current snap is 0 2020-06-12T02:38:46.383 INFO:tasks.rados.rados.0.smithi107.stdout:9131: done (0 left) 2020-06-12T02:38:46.383 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:46.383 INFO:tasks.rados.rados.0.smithi107.stdout:9132: seq_num 9131 ranges {2834432=532480,3366912=704512,4071424=483328,4554752=16384} 2020-06-12T02:38:46.384 INFO:tasks.rados.rados.0.smithi107.stdout:9132: writing smithi10729407-13 from 2834432 to 3366912 tid 1 2020-06-12T02:38:46.384 INFO:tasks.rados.rados.0.smithi107.stdout:9132: writing smithi10729407-13 from 3366912 to 4071424 tid 2 2020-06-12T02:38:46.384 INFO:tasks.rados.rados.0.smithi107.stdout:9132: writing smithi10729407-13 from 4071424 to 4554752 tid 3 2020-06-12T02:38:46.385 INFO:tasks.rados.rados.0.smithi107.stdout:9132: writing smithi10729407-13 from 4554752 to 4571136 tid 4 2020-06-12T02:38:46.385 INFO:tasks.rados.rados.0.smithi107.stdout:9133: delete oid 14 current snap is 0 2020-06-12T02:38:46.385 INFO:tasks.rados.rados.0.smithi107.stdout:9132: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:46.385 INFO:tasks.rados.rados.0.smithi107.stdout:9133: done (1 left) 2020-06-12T02:38:46.385 INFO:tasks.rados.rados.0.smithi107.stdout:9134: delete oid 17 current snap is 0 2020-06-12T02:38:46.386 INFO:tasks.rados.rados.0.smithi107.stdout:9134: done (1 left) 2020-06-12T02:38:46.386 INFO:tasks.rados.rados.0.smithi107.stdout:9135: delete oid 20 current snap is 0 2020-06-12T02:38:46.386 INFO:tasks.rados.rados.0.smithi107.stdout:9132: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:46.386 INFO:tasks.rados.rados.0.smithi107.stdout:9132: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:46.387 INFO:tasks.rados.rados.0.smithi107.stdout:9135: done (1 left) 2020-06-12T02:38:46.387 INFO:tasks.rados.rados.0.smithi107.stdout:9136: delete oid 8 current snap is 0 2020-06-12T02:38:46.387 INFO:tasks.rados.rados.0.smithi107.stdout:9132: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:46.387 INFO:tasks.rados.rados.0.smithi107.stdout:9132: finishing write tid 6 to smithi10729407-13 2020-06-12T02:38:46.387 INFO:tasks.rados.rados.0.smithi107.stdout:9132: finishing write tid 7 to smithi10729407-13 2020-06-12T02:38:46.388 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1643 (ObjNum 9131 snap 0 seq_num 9131) dirty exists 2020-06-12T02:38:46.388 INFO:tasks.rados.rados.0.smithi107.stdout:9132: left oid 13 (ObjNum 9131 snap 0 seq_num 9131) 2020-06-12T02:38:46.388 INFO:tasks.rados.rados.0.smithi107.stdout:9132: done (1 left) 2020-06-12T02:38:46.388 INFO:tasks.rados.rados.0.smithi107.stdout:9136: done (0 left) 2020-06-12T02:38:46.389 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:46.389 INFO:tasks.rados.rados.0.smithi107.stdout:9137: seq_num 9136 ranges {0=614400,614400=524288,1138688=671744,1810432=8192} 2020-06-12T02:38:46.389 INFO:tasks.rados.rados.0.smithi107.stdout:9137: writing smithi10729407-17 from 0 to 614400 tid 1 2020-06-12T02:38:46.389 INFO:tasks.rados.rados.0.smithi107.stdout:9137: writing smithi10729407-17 from 614400 to 1138688 tid 2 2020-06-12T02:38:46.389 INFO:tasks.rados.rados.0.smithi107.stdout:9137: writing smithi10729407-17 from 1138688 to 1810432 tid 3 2020-06-12T02:38:46.390 INFO:tasks.rados.rados.0.smithi107.stdout:9137: writing smithi10729407-17 from 1810432 to 1818624 tid 4 2020-06-12T02:38:46.390 INFO:tasks.rados.rados.0.smithi107.stdout:9138: delete oid 12 current snap is 0 2020-06-12T02:38:46.390 INFO:tasks.rados.rados.0.smithi107.stdout:9137: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:46.390 INFO:tasks.rados.rados.0.smithi107.stdout:9138: done (1 left) 2020-06-12T02:38:46.391 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:46.391 INFO:tasks.rados.rados.0.smithi107.stdout:9139: seq_num 9138 ranges {0=647168,647168=696320,1343488=507904} 2020-06-12T02:38:46.391 INFO:tasks.rados.rados.0.smithi107.stdout:9139: writing smithi10729407-18 from 0 to 647168 tid 1 2020-06-12T02:38:46.392 INFO:tasks.rados.rados.0.smithi107.stdout:9139: writing smithi10729407-18 from 647168 to 1343488 tid 2 2020-06-12T02:38:46.392 INFO:tasks.rados.rados.0.smithi107.stdout:9139: writing smithi10729407-18 from 1343488 to 1851392 tid 3 2020-06-12T02:38:46.392 INFO:tasks.rados.rados.0.smithi107.stdout:9140: delete oid 3 current snap is 0 2020-06-12T02:38:46.392 INFO:tasks.rados.rados.0.smithi107.stdout:9137: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:46.392 INFO:tasks.rados.rados.0.smithi107.stdout:9137: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:46.392 INFO:tasks.rados.rados.0.smithi107.stdout:9137: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:46.393 INFO:tasks.rados.rados.0.smithi107.stdout:9137: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:46.393 INFO:tasks.rados.rados.0.smithi107.stdout:9137: finishing write tid 7 to smithi10729407-17 2020-06-12T02:38:46.393 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1718 (ObjNum 9136 snap 0 seq_num 9136) dirty exists 2020-06-12T02:38:46.393 INFO:tasks.rados.rados.0.smithi107.stdout:9137: left oid 17 (ObjNum 9136 snap 0 seq_num 9136) 2020-06-12T02:38:46.393 INFO:tasks.rados.rados.0.smithi107.stdout:9139: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:46.394 INFO:tasks.rados.rados.0.smithi107.stdout:9137: done (2 left) 2020-06-12T02:38:46.394 INFO:tasks.rados.rados.0.smithi107.stdout:9140: done (1 left) 2020-06-12T02:38:46.394 INFO:tasks.rados.rados.0.smithi107.stdout:9141: delete oid 9 current snap is 0 2020-06-12T02:38:46.394 INFO:tasks.rados.rados.0.smithi107.stdout:9139: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:46.394 INFO:tasks.rados.rados.0.smithi107.stdout:9139: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:46.395 INFO:tasks.rados.rados.0.smithi107.stdout:9139: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:46.395 INFO:tasks.rados.rados.0.smithi107.stdout:9141: done (1 left) 2020-06-12T02:38:46.395 INFO:tasks.rados.rados.0.smithi107.stdout:9142: delete oid 7 current snap is 0 2020-06-12T02:38:46.395 INFO:tasks.rados.rados.0.smithi107.stdout:9139: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:46.395 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1706 (ObjNum 9138 snap 0 seq_num 9138) dirty exists 2020-06-12T02:38:46.395 INFO:tasks.rados.rados.0.smithi107.stdout:9139: left oid 18 (ObjNum 9138 snap 0 seq_num 9138) 2020-06-12T02:38:46.396 INFO:tasks.rados.rados.0.smithi107.stdout:9139: done (1 left) 2020-06-12T02:38:46.396 INFO:tasks.rados.rados.0.smithi107.stdout:9142: done (0 left) 2020-06-12T02:38:46.396 INFO:tasks.rados.rados.0.smithi107.stdout:9143: delete oid 9 current snap is 0 2020-06-12T02:38:46.396 INFO:tasks.rados.rados.0.smithi107.stdout:9143: done (0 left) 2020-06-12T02:38:46.396 INFO:tasks.rados.rados.0.smithi107.stdout:9144: delete oid 3 current snap is 0 2020-06-12T02:38:46.397 INFO:tasks.rados.rados.0.smithi107.stdout:9144: done (0 left) 2020-06-12T02:38:46.397 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:38:46.397 INFO:tasks.rados.rados.0.smithi107.stdout:9145: seq_num 9144 ranges {1818624=753664,2572288=450560,3022848=638976,3661824=114688} 2020-06-12T02:38:46.397 INFO:tasks.rados.rados.0.smithi107.stdout:9145: writing smithi10729407-17 from 1818624 to 2572288 tid 1 2020-06-12T02:38:46.397 INFO:tasks.rados.rados.0.smithi107.stdout:9145: writing smithi10729407-17 from 2572288 to 3022848 tid 2 2020-06-12T02:38:46.397 INFO:tasks.rados.rados.0.smithi107.stdout:9145: writing smithi10729407-17 from 3022848 to 3661824 tid 3 2020-06-12T02:38:46.398 INFO:tasks.rados.rados.0.smithi107.stdout:9145: writing smithi10729407-17 from 3661824 to 3776512 tid 4 2020-06-12T02:38:46.398 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:46.398 INFO:tasks.rados.rados.0.smithi107.stdout:9146: seq_num 9145 ranges {0=770048,770048=737280,1507328=442368,1949696=24576} 2020-06-12T02:38:46.398 INFO:tasks.rados.rados.0.smithi107.stdout:9146: writing smithi10729407-9 from 0 to 770048 tid 1 2020-06-12T02:38:46.398 INFO:tasks.rados.rados.0.smithi107.stdout:9146: writing smithi10729407-9 from 770048 to 1507328 tid 2 2020-06-12T02:38:46.399 INFO:tasks.rados.rados.0.smithi107.stdout:9146: writing smithi10729407-9 from 1507328 to 1949696 tid 3 2020-06-12T02:38:46.399 INFO:tasks.rados.rados.0.smithi107.stdout:9146: writing smithi10729407-9 from 1949696 to 1974272 tid 4 2020-06-12T02:38:46.399 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:46.399 INFO:tasks.rados.rados.0.smithi107.stdout:9147: seq_num 9146 ranges {0=786432,786432=671744,1458176=532480} 2020-06-12T02:38:46.399 INFO:tasks.rados.rados.0.smithi107.stdout:9147: writing smithi10729407-6 from 0 to 786432 tid 1 2020-06-12T02:38:46.400 INFO:tasks.rados.rados.0.smithi107.stdout:9147: writing smithi10729407-6 from 786432 to 1458176 tid 2 2020-06-12T02:38:46.400 INFO:tasks.rados.rados.0.smithi107.stdout:9147: writing smithi10729407-6 from 1458176 to 1990656 tid 3 2020-06-12T02:38:46.401 INFO:tasks.rados.rados.0.smithi107.stdout:9148: delete oid 11 current snap is 0 2020-06-12T02:38:46.401 INFO:tasks.rados.rados.0.smithi107.stdout:9145: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:46.401 INFO:tasks.rados.rados.0.smithi107.stdout:9145: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:46.401 INFO:tasks.rados.rados.0.smithi107.stdout:9145: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:46.401 INFO:tasks.rados.rados.0.smithi107.stdout:9145: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:46.402 INFO:tasks.rados.rados.0.smithi107.stdout:9145: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:46.402 INFO:tasks.rados.rados.0.smithi107.stdout:9145: finishing write tid 7 to smithi10729407-17 2020-06-12T02:38:46.402 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1723 (ObjNum 9144 snap 0 seq_num 9144) dirty exists 2020-06-12T02:38:46.402 INFO:tasks.rados.rados.0.smithi107.stdout:9145: left oid 17 (ObjNum 9144 snap 0 seq_num 9144) 2020-06-12T02:38:46.402 INFO:tasks.rados.rados.0.smithi107.stdout:9146: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:46.402 INFO:tasks.rados.rados.0.smithi107.stdout:9146: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:46.403 INFO:tasks.rados.rados.0.smithi107.stdout:9146: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:46.403 INFO:tasks.rados.rados.0.smithi107.stdout:9146: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:46.403 INFO:tasks.rados.rados.0.smithi107.stdout:9146: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:46.403 INFO:tasks.rados.rados.0.smithi107.stdout:9146: finishing write tid 7 to smithi10729407-9 2020-06-12T02:38:46.404 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1703 (ObjNum 9145 snap 0 seq_num 9145) dirty exists 2020-06-12T02:38:46.404 INFO:tasks.rados.rados.0.smithi107.stdout:9146: left oid 9 (ObjNum 9145 snap 0 seq_num 9145) 2020-06-12T02:38:46.404 INFO:tasks.rados.rados.0.smithi107.stdout:9147: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:46.404 INFO:tasks.rados.rados.0.smithi107.stdout:9145: done (3 left) 2020-06-12T02:38:46.404 INFO:tasks.rados.rados.0.smithi107.stdout:9146: done (2 left) 2020-06-12T02:38:46.404 INFO:tasks.rados.rados.0.smithi107.stdout:9148: done (1 left) 2020-06-12T02:38:46.405 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:46.405 INFO:tasks.rados.rados.0.smithi107.stdout:9149: seq_num 9148 ranges {4571136=417792,4988928=442368,5431296=614400,6045696=548864} 2020-06-12T02:38:46.405 INFO:tasks.rados.rados.0.smithi107.stdout:9149: writing smithi10729407-13 from 4571136 to 4988928 tid 1 2020-06-12T02:38:46.405 INFO:tasks.rados.rados.0.smithi107.stdout:9149: writing smithi10729407-13 from 4988928 to 5431296 tid 2 2020-06-12T02:38:46.406 INFO:tasks.rados.rados.0.smithi107.stdout:9149: writing smithi10729407-13 from 5431296 to 6045696 tid 3 2020-06-12T02:38:46.406 INFO:tasks.rados.rados.0.smithi107.stdout:9149: writing smithi10729407-13 from 6045696 to 6594560 tid 4 2020-06-12T02:38:46.406 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:46.406 INFO:tasks.rados.rados.0.smithi107.stdout:9150: seq_num 9149 ranges {2383872=434176,2818048=720896,3538944=425984,3964928=458752} 2020-06-12T02:38:46.406 INFO:tasks.rados.rados.0.smithi107.stdout:9150: writing smithi10729407-19 from 2383872 to 2818048 tid 1 2020-06-12T02:38:46.406 INFO:tasks.rados.rados.0.smithi107.stdout:9150: writing smithi10729407-19 from 2818048 to 3538944 tid 2 2020-06-12T02:38:46.407 INFO:tasks.rados.rados.0.smithi107.stdout:9150: writing smithi10729407-19 from 3538944 to 3964928 tid 3 2020-06-12T02:38:46.407 INFO:tasks.rados.rados.0.smithi107.stdout:9150: writing smithi10729407-19 from 3964928 to 4423680 tid 4 2020-06-12T02:38:46.407 INFO:tasks.rados.rados.0.smithi107.stdout:9151: delete oid 8 current snap is 0 2020-06-12T02:38:46.407 INFO:tasks.rados.rados.0.smithi107.stdout:9147: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:46.408 INFO:tasks.rados.rados.0.smithi107.stdout:9147: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:46.408 INFO:tasks.rados.rados.0.smithi107.stdout:9147: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:46.408 INFO:tasks.rados.rados.0.smithi107.stdout:9147: finishing write tid 6 to smithi10729407-6 2020-06-12T02:38:46.408 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1668 (ObjNum 9146 snap 0 seq_num 9146) dirty exists 2020-06-12T02:38:46.408 INFO:tasks.rados.rados.0.smithi107.stdout:9147: left oid 6 (ObjNum 9146 snap 0 seq_num 9146) 2020-06-12T02:38:46.409 INFO:tasks.rados.rados.0.smithi107.stdout:9149: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:46.409 INFO:tasks.rados.rados.0.smithi107.stdout:9149: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:46.409 INFO:tasks.rados.rados.0.smithi107.stdout:9149: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:46.409 INFO:tasks.rados.rados.0.smithi107.stdout:9149: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:46.410 INFO:tasks.rados.rados.0.smithi107.stdout:9149: finishing write tid 6 to smithi10729407-13 2020-06-12T02:38:46.410 INFO:tasks.rados.rados.0.smithi107.stdout:9149: finishing write tid 7 to smithi10729407-13 2020-06-12T02:38:46.410 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1648 (ObjNum 9148 snap 0 seq_num 9148) dirty exists 2020-06-12T02:38:46.411 INFO:tasks.rados.rados.0.smithi107.stdout:9149: left oid 13 (ObjNum 9148 snap 0 seq_num 9148) 2020-06-12T02:38:46.411 INFO:tasks.rados.rados.0.smithi107.stdout:9150: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:46.411 INFO:tasks.rados.rados.0.smithi107.stdout:9147: done (3 left) 2020-06-12T02:38:46.411 INFO:tasks.rados.rados.0.smithi107.stdout:9149: done (2 left) 2020-06-12T02:38:46.411 INFO:tasks.rados.rados.0.smithi107.stdout:9151: done (1 left) 2020-06-12T02:38:46.412 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:46.412 INFO:tasks.rados.rados.0.smithi107.stdout:9152: seq_num 9151 ranges {3776512=466944,4243456=491520,4734976=786432,5521408=327680} 2020-06-12T02:38:46.412 INFO:tasks.rados.rados.0.smithi107.stdout:9152: writing smithi10729407-17 from 3776512 to 4243456 tid 1 2020-06-12T02:38:46.412 INFO:tasks.rados.rados.0.smithi107.stdout:9152: writing smithi10729407-17 from 4243456 to 4734976 tid 2 2020-06-12T02:38:46.412 INFO:tasks.rados.rados.0.smithi107.stdout:9152: writing smithi10729407-17 from 4734976 to 5521408 tid 3 2020-06-12T02:38:46.413 INFO:tasks.rados.rados.0.smithi107.stdout:9152: writing smithi10729407-17 from 5521408 to 5849088 tid 4 2020-06-12T02:38:46.413 INFO:tasks.rados.rados.0.smithi107.stdout:9153: delete oid 1 current snap is 0 2020-06-12T02:38:46.413 INFO:tasks.rados.rados.0.smithi107.stdout:9150: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:46.413 INFO:tasks.rados.rados.0.smithi107.stdout:9150: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:46.413 INFO:tasks.rados.rados.0.smithi107.stdout:9150: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:46.414 INFO:tasks.rados.rados.0.smithi107.stdout:9150: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:46.414 INFO:tasks.rados.rados.0.smithi107.stdout:9150: finishing write tid 7 to smithi10729407-19 2020-06-12T02:38:46.414 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1691 (ObjNum 9149 snap 0 seq_num 9149) dirty exists 2020-06-12T02:38:46.414 INFO:tasks.rados.rados.0.smithi107.stdout:9150: left oid 19 (ObjNum 9149 snap 0 seq_num 9149) 2020-06-12T02:38:46.414 INFO:tasks.rados.rados.0.smithi107.stdout:9152: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:46.414 INFO:tasks.rados.rados.0.smithi107.stdout:9150: done (2 left) 2020-06-12T02:38:46.415 INFO:tasks.rados.rados.0.smithi107.stdout:9153: done (1 left) 2020-06-12T02:38:46.415 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:46.415 INFO:tasks.rados.rados.0.smithi107.stdout:9154: seq_num 9153 ranges {0=499712,499712=663552,1163264=802816,1966080=139264} 2020-06-12T02:38:46.415 INFO:tasks.rados.rados.0.smithi107.stdout:9154: writing smithi10729407-11 from 0 to 499712 tid 1 2020-06-12T02:38:46.416 INFO:tasks.rados.rados.0.smithi107.stdout:9154: writing smithi10729407-11 from 499712 to 1163264 tid 2 2020-06-12T02:38:46.416 INFO:tasks.rados.rados.0.smithi107.stdout:9154: writing smithi10729407-11 from 1163264 to 1966080 tid 3 2020-06-12T02:38:46.416 INFO:tasks.rados.rados.0.smithi107.stdout:9154: writing smithi10729407-11 from 1966080 to 2105344 tid 4 2020-06-12T02:38:46.416 INFO:tasks.rados.rados.0.smithi107.stdout:9155: delete oid 14 current snap is 0 2020-06-12T02:38:46.416 INFO:tasks.rados.rados.0.smithi107.stdout:9152: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:46.417 INFO:tasks.rados.rados.0.smithi107.stdout:9152: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:46.417 INFO:tasks.rados.rados.0.smithi107.stdout:9152: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:46.417 INFO:tasks.rados.rados.0.smithi107.stdout:9152: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:46.417 INFO:tasks.rados.rados.0.smithi107.stdout:9152: finishing write tid 7 to smithi10729407-17 2020-06-12T02:38:46.417 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1728 (ObjNum 9151 snap 0 seq_num 9151) dirty exists 2020-06-12T02:38:46.418 INFO:tasks.rados.rados.0.smithi107.stdout:9152: left oid 17 (ObjNum 9151 snap 0 seq_num 9151) 2020-06-12T02:38:46.419 INFO:tasks.rados.rados.0.smithi107.stdout:9154: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:46.419 INFO:tasks.rados.rados.0.smithi107.stdout:9154: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:46.419 INFO:tasks.rados.rados.0.smithi107.stdout:9154: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:46.419 INFO:tasks.rados.rados.0.smithi107.stdout:9154: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:46.420 INFO:tasks.rados.rados.0.smithi107.stdout:9154: finishing write tid 6 to smithi10729407-11 2020-06-12T02:38:46.420 INFO:tasks.rados.rados.0.smithi107.stdout:9154: finishing write tid 7 to smithi10729407-11 2020-06-12T02:38:46.420 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1590 (ObjNum 9153 snap 0 seq_num 9153) dirty exists 2020-06-12T02:38:46.420 INFO:tasks.rados.rados.0.smithi107.stdout:9154: left oid 11 (ObjNum 9153 snap 0 seq_num 9153) 2020-06-12T02:38:46.420 INFO:tasks.rados.rados.0.smithi107.stdout:9152: done (2 left) 2020-06-12T02:38:46.420 INFO:tasks.rados.rados.0.smithi107.stdout:9154: done (1 left) 2020-06-12T02:38:46.421 INFO:tasks.rados.rados.0.smithi107.stdout:9155: done (0 left) 2020-06-12T02:38:46.421 INFO:tasks.rados.rados.0.smithi107.stdout:9156: delete oid 12 current snap is 0 2020-06-12T02:38:46.421 INFO:tasks.rados.rados.0.smithi107.stdout:9156: done (0 left) 2020-06-12T02:38:46.421 INFO:tasks.rados.rados.0.smithi107.stdout:9157: delete oid 2 current snap is 0 2020-06-12T02:38:46.421 INFO:tasks.rados.rados.0.smithi107.stdout:9157: done (0 left) 2020-06-12T02:38:46.422 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:38:46.422 INFO:tasks.rados.rados.0.smithi107.stdout:9158: seq_num 9157 ranges {0=565248,565248=647168,1212416=786432,1998848=172032} 2020-06-12T02:38:46.422 INFO:tasks.rados.rados.0.smithi107.stdout:9158: writing smithi10729407-2 from 0 to 565248 tid 1 2020-06-12T02:38:46.422 INFO:tasks.rados.rados.0.smithi107.stdout:9158: writing smithi10729407-2 from 565248 to 1212416 tid 2 2020-06-12T02:38:46.422 INFO:tasks.rados.rados.0.smithi107.stdout:9158: writing smithi10729407-2 from 1212416 to 1998848 tid 3 2020-06-12T02:38:46.423 INFO:tasks.rados.rados.0.smithi107.stdout:9158: writing smithi10729407-2 from 1998848 to 2170880 tid 4 2020-06-12T02:38:46.423 INFO:tasks.rados.rados.0.smithi107.stdout:9159: delete oid 7 current snap is 0 2020-06-12T02:38:46.423 INFO:tasks.rados.rados.0.smithi107.stdout:9158: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:46.423 INFO:tasks.rados.rados.0.smithi107.stdout:9158: finishing write tid 2 to smithi10729407-2 2020-06-12T02:38:46.424 INFO:tasks.rados.rados.0.smithi107.stdout:9159: done (1 left) 2020-06-12T02:38:46.424 INFO:tasks.rados.rados.0.smithi107.stdout:9160: delete oid 14 current snap is 0 2020-06-12T02:38:46.424 INFO:tasks.rados.rados.0.smithi107.stdout:9160: done (1 left) 2020-06-12T02:38:46.424 INFO:tasks.rados.rados.0.smithi107.stdout:9161: delete oid 5 current snap is 0 2020-06-12T02:38:46.425 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:46.349+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:46.425 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-06-12T02:38:46.427 INFO:tasks.rados.rados.0.smithi107.stdout:9158: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:46.427 INFO:tasks.rados.rados.0.smithi107.stdout:9158: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:46.428 INFO:tasks.rados.rados.0.smithi107.stdout:9158: finishing write tid 6 to smithi10729407-2 2020-06-12T02:38:46.428 INFO:tasks.rados.rados.0.smithi107.stdout:9161: done (1 left) 2020-06-12T02:38:46.428 INFO:tasks.rados.rados.0.smithi107.stdout:9162: delete oid 9 current snap is 0 2020-06-12T02:38:46.428 INFO:tasks.rados.rados.0.smithi107.stdout:9158: finishing write tid 7 to smithi10729407-2 2020-06-12T02:38:46.428 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 853 (ObjNum 9157 snap 0 seq_num 9157) dirty exists 2020-06-12T02:38:46.429 INFO:tasks.rados.rados.0.smithi107.stdout:9158: left oid 2 (ObjNum 9157 snap 0 seq_num 9157) 2020-06-12T02:38:46.429 INFO:tasks.rados.rados.0.smithi107.stdout:9158: done (1 left) 2020-06-12T02:38:46.429 INFO:tasks.rados.rados.0.smithi107.stdout:9162: done (0 left) 2020-06-12T02:38:46.429 INFO:tasks.rados.rados.0.smithi107.stdout:9163: delete oid 17 current snap is 0 2020-06-12T02:38:46.429 INFO:tasks.rados.rados.0.smithi107.stdout:9163: done (0 left) 2020-06-12T02:38:46.430 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:38:46.430 INFO:tasks.rados.rados.0.smithi107.stdout:9164: seq_num 9163 ranges {6594560=671744,7266304=802816,8069120=729088,8798208=73728} 2020-06-12T02:38:46.430 INFO:tasks.rados.rados.0.smithi107.stdout:9164: writing smithi10729407-13 from 6594560 to 7266304 tid 1 2020-06-12T02:38:46.430 INFO:tasks.rados.rados.0.smithi107.stdout:9164: writing smithi10729407-13 from 7266304 to 8069120 tid 2 2020-06-12T02:38:46.430 INFO:tasks.rados.rados.0.smithi107.stdout:9164: writing smithi10729407-13 from 8069120 to 8798208 tid 3 2020-06-12T02:38:46.430 INFO:tasks.rados.rados.0.smithi107.stdout:9164: writing smithi10729407-13 from 8798208 to 8871936 tid 4 2020-06-12T02:38:46.431 INFO:tasks.rados.rados.0.smithi107.stdout:9165: delete oid 3 current snap is 0 2020-06-12T02:38:46.431 INFO:tasks.rados.rados.0.smithi107.stdout:9164: finishing write tid 1 to smithi10729407-13 2020-06-12T02:38:46.432 INFO:tasks.rados.rados.0.smithi107.stdout:9165: done (1 left) 2020-06-12T02:38:46.432 INFO:tasks.rados.rados.0.smithi107.stdout:9166: delete oid 8 current snap is 0 2020-06-12T02:38:46.433 INFO:tasks.rados.rados.0.smithi107.stdout:9164: finishing write tid 2 to smithi10729407-13 2020-06-12T02:38:46.433 INFO:tasks.rados.rados.0.smithi107.stdout:9164: finishing write tid 3 to smithi10729407-13 2020-06-12T02:38:46.434 INFO:tasks.rados.rados.0.smithi107.stdout:9164: finishing write tid 4 to smithi10729407-13 2020-06-12T02:38:46.435 INFO:tasks.rados.rados.0.smithi107.stdout:9166: done (1 left) 2020-06-12T02:38:46.435 INFO:tasks.rados.rados.0.smithi107.stdout:9167: delete oid 5 current snap is 0 2020-06-12T02:38:46.436 INFO:tasks.rados.rados.0.smithi107.stdout:9164: finishing write tid 6 to smithi10729407-13 2020-06-12T02:38:46.437 INFO:tasks.rados.rados.0.smithi107.stdout:9164: finishing write tid 7 to smithi10729407-13 2020-06-12T02:38:46.437 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1653 (ObjNum 9163 snap 0 seq_num 9163) dirty exists 2020-06-12T02:38:46.438 INFO:tasks.rados.rados.0.smithi107.stdout:9164: left oid 13 (ObjNum 9163 snap 0 seq_num 9163) 2020-06-12T02:38:46.439 INFO:tasks.rados.rados.0.smithi107.stdout:9164: done (1 left) 2020-06-12T02:38:46.439 INFO:tasks.rados.rados.0.smithi107.stdout:9167: done (0 left) 2020-06-12T02:38:46.439 INFO:tasks.rados.rados.0.smithi107.stdout:9168: delete oid 14 current snap is 0 2020-06-12T02:38:46.441 INFO:tasks.rados.rados.0.smithi107.stdout:9168: done (0 left) 2020-06-12T02:38:46.441 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:46.441 INFO:tasks.rados.rados.0.smithi107.stdout:9169: seq_num 9168 ranges {0=753664,753664=622592,1376256=516096,1892352=466944} 2020-06-12T02:38:46.445 INFO:tasks.rados.rados.0.smithi107.stdout:9169: writing smithi10729407-15 from 0 to 753664 tid 1 2020-06-12T02:38:46.449 INFO:tasks.rados.rados.0.smithi107.stdout:9169: writing smithi10729407-15 from 753664 to 1376256 tid 2 2020-06-12T02:38:46.452 INFO:tasks.rados.rados.0.smithi107.stdout:9169: writing smithi10729407-15 from 1376256 to 1892352 tid 3 2020-06-12T02:38:46.452 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:46.449+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:46.454 INFO:tasks.rados.rados.0.smithi107.stdout:9169: writing smithi10729407-15 from 1892352 to 2359296 tid 4 2020-06-12T02:38:46.455 INFO:tasks.rados.rados.0.smithi107.stdout:9170: delete oid 4 current snap is 0 2020-06-12T02:38:46.455 INFO:tasks.rados.rados.0.smithi107.stdout:9169: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:46.458 INFO:tasks.rados.rados.0.smithi107.stdout:9170: done (1 left) 2020-06-12T02:38:46.458 INFO:tasks.rados.rados.0.smithi107.stdout:9171: delete oid 1 current snap is 0 2020-06-12T02:38:46.466 INFO:tasks.rados.rados.0.smithi107.stdout:9171: done (1 left) 2020-06-12T02:38:46.466 INFO:tasks.rados.rados.0.smithi107.stdout:9172: delete oid 19 current snap is 0 2020-06-12T02:38:46.468 INFO:tasks.rados.rados.0.smithi107.stdout:9169: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:46.470 INFO:tasks.rados.rados.0.smithi107.stdout:9172: done (1 left) 2020-06-12T02:38:46.470 INFO:tasks.rados.rados.0.smithi107.stdout:9173: delete oid 4 current snap is 0 2020-06-12T02:38:46.471 INFO:tasks.rados.rados.0.smithi107.stdout:9169: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:46.474 INFO:tasks.rados.rados.0.smithi107.stdout:9169: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:46.474 INFO:tasks.rados.rados.0.smithi107.stdout:9169: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:46.476 INFO:tasks.rados.rados.0.smithi107.stdout:9169: finishing write tid 7 to smithi10729407-15 2020-06-12T02:38:46.476 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1703 (ObjNum 9168 snap 0 seq_num 9168) dirty exists 2020-06-12T02:38:46.476 INFO:tasks.rados.rados.0.smithi107.stdout:9169: left oid 15 (ObjNum 9168 snap 0 seq_num 9168) 2020-06-12T02:38:46.477 INFO:tasks.rados.rados.0.smithi107.stdout:9169: done (1 left) 2020-06-12T02:38:46.478 INFO:tasks.rados.rados.0.smithi107.stdout:9173: done (0 left) 2020-06-12T02:38:46.478 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:38:46.478 INFO:tasks.rados.rados.0.smithi107.stdout:9174: seq_num 9173 ranges {0=32768} 2020-06-12T02:38:46.478 INFO:tasks.rados.rados.0.smithi107.stdout:9174: writing smithi10729407-1 from 0 to 32768 tid 1 2020-06-12T02:38:46.478 INFO:tasks.rados.rados.0.smithi107.stdout:9175: delete oid 17 current snap is 0 2020-06-12T02:38:46.481 INFO:tasks.rados.rados.0.smithi107.stdout:9174: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:46.481 INFO:tasks.rados.rados.0.smithi107.stdout:9175: done (1 left) 2020-06-12T02:38:46.482 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:46.482 INFO:tasks.rados.rados.0.smithi107.stdout:9176: seq_num 9175 ranges {0=65536} 2020-06-12T02:38:46.482 INFO:tasks.rados.rados.0.smithi107.stdout:9176: writing smithi10729407-3 from 0 to 65536 tid 1 2020-06-12T02:38:46.483 INFO:tasks.rados.rados.0.smithi107.stdout:9177: delete oid 5 current snap is 0 2020-06-12T02:38:46.485 INFO:tasks.rados.rados.0.smithi107.stdout:9176: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:46.486 INFO:tasks.rados.rados.0.smithi107.stdout:9177: done (2 left) 2020-06-12T02:38:46.486 INFO:tasks.rados.rados.0.smithi107.stdout:9178: delete oid 12 current snap is 0 2020-06-12T02:38:46.486 INFO:tasks.rados.rados.0.smithi107.stdout:9174: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:46.488 INFO:tasks.rados.rados.0.smithi107.stdout:9176: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:46.490 INFO:tasks.rados.rados.0.smithi107.stdout:9176: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:46.490 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1638 (ObjNum 9175 snap 0 seq_num 9175) dirty exists 2020-06-12T02:38:46.490 INFO:tasks.rados.rados.0.smithi107.stdout:9176: left oid 3 (ObjNum 9175 snap 0 seq_num 9175) 2020-06-12T02:38:46.491 INFO:tasks.rados.rados.0.smithi107.stdout:9174: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:46.491 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1672 (ObjNum 9173 snap 0 seq_num 9173) dirty exists 2020-06-12T02:38:46.491 INFO:tasks.rados.rados.0.smithi107.stdout:9174: left oid 1 (ObjNum 9173 snap 0 seq_num 9173) 2020-06-12T02:38:46.492 INFO:tasks.rados.rados.0.smithi107.stdout:9174: done (2 left) 2020-06-12T02:38:46.493 INFO:tasks.rados.rados.0.smithi107.stdout:9176: done (1 left) 2020-06-12T02:38:46.493 INFO:tasks.rados.rados.0.smithi107.stdout:9178: done (0 left) 2020-06-12T02:38:46.493 INFO:tasks.rados.rados.0.smithi107.stdout:9179: delete oid 8 current snap is 0 2020-06-12T02:38:46.507 INFO:tasks.rados.rados.0.smithi107.stdout:9179: done (0 left) 2020-06-12T02:38:46.507 INFO:tasks.rados.rados.0.smithi107.stdout:9180: delete oid 15 current snap is 0 2020-06-12T02:38:46.510 INFO:tasks.rados.rados.0.smithi107.stdout:9180: done (0 left) 2020-06-12T02:38:46.511 INFO:tasks.rados.rados.0.smithi107.stdout:9181: delete oid 9 current snap is 0 2020-06-12T02:38:46.554 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:46.551+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:46.574 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:46.575 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:38:46.575 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2020-06-12T02:38:46.576 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:46.583 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-06-12T02:38:46.655 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:46.652+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:46.744 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:46.745 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:38:46.745 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:38:46.745 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:38:46.745 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:46.753 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-06-12T02:38:46.755 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:46.753+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:46.857 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:46.854+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:46.872 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:46.872 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:38:46.873 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:38:46.873 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:38:46.873 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.874 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8464 3.fs0 3.57bc3c1f (undecoded) ondisk+retry+write+known_if_redirected e34)", 2020-06-12T02:38:46.874 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:53.549364+0000", 2020-06-12T02:38:46.874 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 53.318537827, 2020-06-12T02:38:46.875 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.003107494, 2020-06-12T02:38:46.875 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:46.875 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:38:46.876 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:46.876 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:46.876 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:46.877 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8464 2020-06-12T02:38:46.877 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.877 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:46.878 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.878 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:46.887 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.549364+0000", 2020-06-12T02:38:46.888 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:46.888 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.888 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.888 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:46.889 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.549364+0000", 2020-06-12T02:38:46.889 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023109609999999998 2020-06-12T02:38:46.889 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.890 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.890 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:46.890 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551675+0000", 2020-06-12T02:38:46.891 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3331e-05 2020-06-12T02:38:46.891 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.891 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.892 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:46.892 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551688+0000", 2020-06-12T02:38:46.892 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4173e-05 2020-06-12T02:38:46.893 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.893 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.893 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:46.894 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551703+0000", 2020-06-12T02:38:46.894 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0454e-05 2020-06-12T02:38:46.894 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.894 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.895 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:46.895 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551713+0000", 2020-06-12T02:38:46.895 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031069050000000001 2020-06-12T02:38:46.896 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.896 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.896 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.897 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554820+0000", 2020-06-12T02:38:46.897 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2820000000000002e-06 2020-06-12T02:38:46.897 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.898 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.898 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2020-06-12T02:38:46.898 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554826+0000", 2020-06-12T02:38:46.899 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99762648499999995 2020-06-12T02:38:46.899 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.899 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.900 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.901 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552453+0000", 2020-06-12T02:38:46.901 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8902999999999998e-05 2020-06-12T02:38:46.902 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.902 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.902 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:46.902 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552472+0000", 2020-06-12T02:38:46.902 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.003107494 2020-06-12T02:38:46.903 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.903 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:46.903 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.903 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.903 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.904 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8465 3.fs0 3.57bc3c1f (undecoded) ondisk+retry+write+known_if_redirected e34)", 2020-06-12T02:38:46.904 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:53.551774+0000", 2020-06-12T02:38:46.904 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 53.316128190999997, 2020-06-12T02:38:46.904 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.000741442, 2020-06-12T02:38:46.904 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:46.905 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:38:46.905 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:46.905 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:46.905 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:46.905 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8465 2020-06-12T02:38:46.906 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.906 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:46.906 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.906 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:46.907 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551774+0000", 2020-06-12T02:38:46.907 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:46.907 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.907 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.907 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:46.908 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551774+0000", 2020-06-12T02:38:46.908 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0010760539999999999 2020-06-12T02:38:46.908 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.908 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.908 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:46.908 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552850+0000", 2020-06-12T02:38:46.909 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0139000000000001e-05 2020-06-12T02:38:46.909 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.909 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.909 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:46.909 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552860+0000", 2020-06-12T02:38:46.910 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7646999999999999e-05 2020-06-12T02:38:46.910 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.910 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.910 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:46.910 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552878+0000", 2020-06-12T02:38:46.911 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2916e-05 2020-06-12T02:38:46.911 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.911 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.911 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:46.911 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552890+0000", 2020-06-12T02:38:46.911 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0019574610000000002 2020-06-12T02:38:46.912 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.912 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.912 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.912 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554848+0000", 2020-06-12T02:38:46.912 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0399999999999995e-07 2020-06-12T02:38:46.913 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.913 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.913 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2020-06-12T02:38:46.913 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554849+0000", 2020-06-12T02:38:46.913 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99765586100000003 2020-06-12T02:38:46.913 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.914 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.914 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.914 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552504+0000", 2020-06-12T02:38:46.914 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.066e-05 2020-06-12T02:38:46.914 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.915 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.915 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:46.915 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552515+0000", 2020-06-12T02:38:46.915 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.000741442 2020-06-12T02:38:46.915 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.916 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:46.916 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.916 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.916 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.916 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8466 3.fs0 3.57bc3c1f (undecoded) ondisk+write+known_if_redirected e34)", 2020-06-12T02:38:46.916 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:53.552953+0000", 2020-06-12T02:38:46.917 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 53.314948911000002, 2020-06-12T02:38:46.917 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99960144299999998, 2020-06-12T02:38:46.917 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:46.917 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:38:46.917 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:46.918 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:46.918 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:46.918 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8466 2020-06-12T02:38:46.918 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.919 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:46.919 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.919 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:46.919 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552953+0000", 2020-06-12T02:38:46.919 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:46.919 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.920 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.920 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:46.920 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552953+0000", 2020-06-12T02:38:46.920 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013847100000000001 2020-06-12T02:38:46.920 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.921 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.921 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:46.921 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553091+0000", 2020-06-12T02:38:46.921 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7190000000000004e-06 2020-06-12T02:38:46.921 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.922 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.922 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:46.922 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553098+0000", 2020-06-12T02:38:46.922 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.171e-05 2020-06-12T02:38:46.922 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.922 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.923 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:46.923 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553110+0000", 2020-06-12T02:38:46.923 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5770000000000001e-06 2020-06-12T02:38:46.923 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.923 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.924 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:46.924 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553115+0000", 2020-06-12T02:38:46.924 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0017512739999999999 2020-06-12T02:38:46.924 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.924 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.924 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.925 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554867+0000", 2020-06-12T02:38:46.925 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1200000000000003e-07 2020-06-12T02:38:46.925 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.925 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.925 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2020-06-12T02:38:46.926 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554867+0000", 2020-06-12T02:38:46.926 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99767707100000003 2020-06-12T02:38:46.926 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.926 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.926 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.926 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552544+0000", 2020-06-12T02:38:46.927 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0009e-05 2020-06-12T02:38:46.927 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.927 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.927 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:46.927 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552554+0000", 2020-06-12T02:38:46.928 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99960144299999998 2020-06-12T02:38:46.928 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.928 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:46.928 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.928 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.929 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.929 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8467 3.fs0 3.57bc3c1f (undecoded) ondisk+write+known_if_redirected e34)", 2020-06-12T02:38:46.929 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:53.553172+0000", 2020-06-12T02:38:46.929 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 53.314729694999997, 2020-06-12T02:38:46.929 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99942086200000002, 2020-06-12T02:38:46.929 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:46.930 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:38:46.930 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:46.930 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:46.930 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:46.930 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8467 2020-06-12T02:38:46.931 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.931 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:46.935 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.935 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:46.935 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553172+0000", 2020-06-12T02:38:46.936 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:46.936 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.936 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.937 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:46.937 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553172+0000", 2020-06-12T02:38:46.937 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5913000000000003e-05 2020-06-12T02:38:46.937 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.937 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.937 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:46.938 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553248+0000", 2020-06-12T02:38:46.938 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7740000000000001e-06 2020-06-12T02:38:46.938 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.938 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.938 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:46.939 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553254+0000", 2020-06-12T02:38:46.939 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1494999999999999e-05 2020-06-12T02:38:46.939 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.939 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.939 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:46.940 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553265+0000", 2020-06-12T02:38:46.940 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5170000000000001e-06 2020-06-12T02:38:46.940 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.940 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.940 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:46.940 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553270+0000", 2020-06-12T02:38:46.941 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001616567 2020-06-12T02:38:46.941 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.941 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.941 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.941 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554886+0000", 2020-06-12T02:38:46.942 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7800000000000001e-07 2020-06-12T02:38:46.942 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.942 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.942 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2020-06-12T02:38:46.942 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554887+0000", 2020-06-12T02:38:46.942 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99769652399999997 2020-06-12T02:38:46.943 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.943 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.943 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.943 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552584+0000", 2020-06-12T02:38:46.943 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4939999999999992e-06 2020-06-12T02:38:46.944 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.944 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.944 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:46.944 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552593+0000", 2020-06-12T02:38:46.944 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99942086200000002 2020-06-12T02:38:46.944 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.945 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:46.945 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.945 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.945 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.945 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8468 3.fs0 3.57bc3c1f (undecoded) ondisk+read+rwordered+known_if_redirected e34)", 2020-06-12T02:38:46.946 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:53.553311+0000", 2020-06-12T02:38:46.946 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 53.314591096999997, 2020-06-12T02:38:46.946 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99988704799999994, 2020-06-12T02:38:46.946 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:46.946 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:38:46.947 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:46.947 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:46.947 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:46.947 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8468 2020-06-12T02:38:46.947 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.947 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:46.948 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.948 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:46.948 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553311+0000", 2020-06-12T02:38:46.948 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:46.948 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.949 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.949 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:46.949 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553311+0000", 2020-06-12T02:38:46.949 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2050000000000001e-05 2020-06-12T02:38:46.949 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.949 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.950 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:46.950 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553333+0000", 2020-06-12T02:38:46.950 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5489999999999998e-06 2020-06-12T02:38:46.950 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.950 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.951 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:46.951 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553336+0000", 2020-06-12T02:38:46.951 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1620000000000002e-06 2020-06-12T02:38:46.951 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.951 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.951 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:46.952 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553344+0000", 2020-06-12T02:38:46.952 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.807e-06 2020-06-12T02:38:46.952 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.952 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.952 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:46.953 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553346+0000", 2020-06-12T02:38:46.953 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001560575 2020-06-12T02:38:46.953 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.953 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.953 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.953 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554907+0000", 2020-06-12T02:38:46.954 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1429999999999999e-06 2020-06-12T02:38:46.954 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.954 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.954 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2020-06-12T02:38:46.954 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554908+0000", 2020-06-12T02:38:46.955 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99827844600000004 2020-06-12T02:38:46.955 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.955 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.955 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.955 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.553187+0000", 2020-06-12T02:38:46.955 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1316e-05 2020-06-12T02:38:46.956 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.956 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.956 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:46.956 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.553198+0000", 2020-06-12T02:38:46.956 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99988704799999994 2020-06-12T02:38:46.957 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.957 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:46.957 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.957 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.957 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.957 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11992 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [append 0~581632] snapc 0=[] ondisk+write+known_if_redirected e56)", 2020-06-12T02:38:46.958 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:23.783061+0000", 2020-06-12T02:38:46.958 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 23.084840916000001, 2020-06-12T02:38:46.958 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022850993, 2020-06-12T02:38:46.958 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:46.958 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:46.959 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:46.959 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:46.959 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:46.959 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11992 2020-06-12T02:38:46.959 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.960 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:46.960 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.961 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:46.961 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783061+0000", 2020-06-12T02:38:46.961 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:46.961 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.961 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.961 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:46.962 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783061+0000", 2020-06-12T02:38:46.962 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047999300000000001 2020-06-12T02:38:46.962 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.962 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.962 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:46.963 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783541+0000", 2020-06-12T02:38:46.963 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5649999999999998e-06 2020-06-12T02:38:46.963 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.963 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.963 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:46.963 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783548+0000", 2020-06-12T02:38:46.964 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3559e-05 2020-06-12T02:38:46.964 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.964 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.964 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:46.964 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783561+0000", 2020-06-12T02:38:46.965 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3539999999999996e-06 2020-06-12T02:38:46.965 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.965 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.965 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:46.965 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783567+0000", 2020-06-12T02:38:46.966 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001353604 2020-06-12T02:38:46.966 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.966 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.966 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.966 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.784921+0000", 2020-06-12T02:38:46.966 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000111853 2020-06-12T02:38:46.967 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.967 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.967 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:46.967 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.785033+0000", 2020-06-12T02:38:46.968 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051647919999999996 2020-06-12T02:38:46.968 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.968 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.968 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.968 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.790198+0000", 2020-06-12T02:38:46.969 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000110742 2020-06-12T02:38:46.969 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.969 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.969 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:46.969 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.790308+0000", 2020-06-12T02:38:46.970 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00090050099999999995 2020-06-12T02:38:46.970 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.970 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.970 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:46.970 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.791209+0000", 2020-06-12T02:38:46.970 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010078284999999999 2020-06-12T02:38:46.971 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.971 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.971 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:46.971 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.801287+0000", 2020-06-12T02:38:46.971 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045443109999999997 2020-06-12T02:38:46.972 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.972 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.972 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:46.972 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.805832+0000", 2020-06-12T02:38:46.972 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.0433999999999996e-05 2020-06-12T02:38:46.973 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.973 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.973 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:46.973 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.805912+0000", 2020-06-12T02:38:46.973 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022850993 2020-06-12T02:38:46.973 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.974 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:46.974 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.974 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.974 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.975 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11994 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected e56)", 2020-06-12T02:38:46.975 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:23.788734+0000", 2020-06-12T02:38:46.975 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 23.079168271, 2020-06-12T02:38:46.975 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024022136999999999, 2020-06-12T02:38:46.975 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:46.975 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:46.976 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:46.976 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:46.976 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:46.976 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11994 2020-06-12T02:38:46.976 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.977 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:46.977 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.977 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:46.977 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.788734+0000", 2020-06-12T02:38:46.977 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:46.977 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.978 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.978 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:46.978 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.788734+0000", 2020-06-12T02:38:46.978 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00034212499999999998 2020-06-12T02:38:46.978 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.979 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.979 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:46.979 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789076+0000", 2020-06-12T02:38:46.979 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.254e-06 2020-06-12T02:38:46.979 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.979 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.980 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:46.980 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789081+0000", 2020-06-12T02:38:46.980 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8249999999999994e-06 2020-06-12T02:38:46.980 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.980 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.981 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:46.981 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789090+0000", 2020-06-12T02:38:46.981 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2259999999999999e-06 2020-06-12T02:38:46.981 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.981 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.981 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:46.982 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789094+0000", 2020-06-12T02:38:46.982 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2913000000000001e-05 2020-06-12T02:38:46.982 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.982 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.982 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.983 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789127+0000", 2020-06-12T02:38:46.983 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2001000000000005e-05 2020-06-12T02:38:46.983 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.983 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.983 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:46.984 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789189+0000", 2020-06-12T02:38:46.984 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0097474550000000004 2020-06-12T02:38:46.984 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.984 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.984 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.984 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.798936+0000", 2020-06-12T02:38:46.985 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9882000000000005e-05 2020-06-12T02:38:46.985 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.985 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.985 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:46.985 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.799016+0000", 2020-06-12T02:38:46.986 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00048691000000000001 2020-06-12T02:38:46.986 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.986 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.986 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:46.986 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.799503+0000", 2020-06-12T02:38:46.986 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002766363 2020-06-12T02:38:46.987 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.987 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.987 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:46.987 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.802270+0000", 2020-06-12T02:38:46.987 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010404759 2020-06-12T02:38:46.988 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.988 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.988 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:46.988 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.812674+0000", 2020-06-12T02:38:46.988 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1423999999999998e-05 2020-06-12T02:38:46.988 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.989 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.989 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:46.989 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.812756+0000", 2020-06-12T02:38:46.989 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024022136999999999 2020-06-12T02:38:46.989 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.990 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:46.990 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:46.990 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.990 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.990 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11995 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e56)", 2020-06-12T02:38:46.990 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:23.789128+0000", 2020-06-12T02:38:46.991 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 23.078773824999999, 2020-06-12T02:38:46.991 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031795508, 2020-06-12T02:38:46.991 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:46.991 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:46.991 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:46.992 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:46.992 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:46.992 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11995 2020-06-12T02:38:46.992 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.992 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:46.992 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.993 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:46.993 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789128+0000", 2020-06-12T02:38:46.993 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:46.993 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.993 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.994 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:46.994 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789128+0000", 2020-06-12T02:38:46.994 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4419999999999998e-05 2020-06-12T02:38:46.994 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.994 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.994 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:46.995 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789172+0000", 2020-06-12T02:38:46.996 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7110000000000001e-06 2020-06-12T02:38:46.996 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.996 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.996 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:46.996 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789176+0000", 2020-06-12T02:38:46.996 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5089999999999999e-06 2020-06-12T02:38:46.997 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.997 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.997 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:46.997 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789183+0000", 2020-06-12T02:38:46.997 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.497e-06 2020-06-12T02:38:46.998 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.998 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.998 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:46.998 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789185+0000", 2020-06-12T02:38:46.998 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012172176 2020-06-12T02:38:46.999 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:46.999 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:46.999 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:46.999 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.801357+0000", 2020-06-12T02:38:46.999 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4264000000000006e-05 2020-06-12T02:38:46.999 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.000 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.000 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:47.000 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.801432+0000", 2020-06-12T02:38:47.000 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00029680499999999997 2020-06-12T02:38:47.000 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.001 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.001 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:47.001 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.801728+0000", 2020-06-12T02:38:47.001 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00066278300000000001 2020-06-12T02:38:47.001 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.001 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.002 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:47.002 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.802391+0000", 2020-06-12T02:38:47.002 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018429867999999999 2020-06-12T02:38:47.002 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.002 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.003 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:47.003 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.820821+0000", 2020-06-12T02:38:47.003 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000102475 2020-06-12T02:38:47.003 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.003 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.003 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:47.004 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.820924+0000", 2020-06-12T02:38:47.004 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031795508 2020-06-12T02:38:47.004 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:47.004 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:47.004 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:47.004 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.005 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.005 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11996 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e56)", 2020-06-12T02:38:47.005 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:23.789235+0000", 2020-06-12T02:38:47.005 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 23.078666974000001, 2020-06-12T02:38:47.005 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.034785745, 2020-06-12T02:38:47.006 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:47.006 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:47.006 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:47.006 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:47.006 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:47.007 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11996 2020-06-12T02:38:47.007 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.007 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:47.007 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.007 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:47.007 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789235+0000", 2020-06-12T02:38:47.008 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:47.008 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.008 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.008 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:47.008 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789235+0000", 2020-06-12T02:38:47.009 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6375000000000001e-05 2020-06-12T02:38:47.009 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.009 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.009 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:47.009 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789261+0000", 2020-06-12T02:38:47.009 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2679999999999999e-06 2020-06-12T02:38:47.010 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.010 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.010 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:47.010 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789265+0000", 2020-06-12T02:38:47.010 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6039999999999998e-06 2020-06-12T02:38:47.011 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.011 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.011 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:47.011 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789271+0000", 2020-06-12T02:38:47.011 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4219999999999999e-06 2020-06-12T02:38:47.011 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.012 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.012 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:47.012 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789274+0000", 2020-06-12T02:38:47.012 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013034463 2020-06-12T02:38:47.012 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.013 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.013 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:47.013 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.802308+0000", 2020-06-12T02:38:47.013 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8862000000000005e-05 2020-06-12T02:38:47.013 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.013 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.014 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:47.014 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.802387+0000", 2020-06-12T02:38:47.014 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019102297000000001 2020-06-12T02:38:47.014 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.014 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.015 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:47.015 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.821489+0000", 2020-06-12T02:38:47.015 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8838999999999996e-05 2020-06-12T02:38:47.015 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.015 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.016 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:47.016 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.821558+0000", 2020-06-12T02:38:47.016 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002462615 2020-06-12T02:38:47.016 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.016 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.016 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:47.017 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.824021+0000", 2020-06-12T02:38:47.017 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.034785745 2020-06-12T02:38:47.017 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:47.017 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:47.017 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:47.018 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.018 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.018 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:15005 3.ds0 3:b3445631:::smithi10729407-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e59)", 2020-06-12T02:38:47.018 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:32.192302+0000", 2020-06-12T02:38:47.019 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 14.675599739000001, 2020-06-12T02:38:47.019 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024128666, 2020-06-12T02:38:47.019 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:47.019 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:47.019 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:47.020 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:47.020 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:47.020 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 15005 2020-06-12T02:38:47.020 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.020 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:47.020 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.021 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:47.021 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192302+0000", 2020-06-12T02:38:47.021 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:47.021 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.021 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.022 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:47.022 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192302+0000", 2020-06-12T02:38:47.022 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3415999999999999e-05 2020-06-12T02:38:47.022 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.022 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.022 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:47.023 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192346+0000", 2020-06-12T02:38:47.023 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.93e-06 2020-06-12T02:38:47.023 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.023 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.023 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:47.024 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192352+0000", 2020-06-12T02:38:47.024 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3397000000000001e-05 2020-06-12T02:38:47.024 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.024 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.024 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:47.024 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192365+0000", 2020-06-12T02:38:47.025 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3950000000000003e-06 2020-06-12T02:38:47.025 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.025 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.025 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:47.025 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192369+0000", 2020-06-12T02:38:47.026 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010208887999999999 2020-06-12T02:38:47.026 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.026 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.026 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:47.026 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.202578+0000", 2020-06-12T02:38:47.026 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000105611 2020-06-12T02:38:47.027 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.027 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.027 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:47.027 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.202684+0000", 2020-06-12T02:38:47.027 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0091288959999999992 2020-06-12T02:38:47.028 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.028 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.028 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:47.028 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.211813+0000", 2020-06-12T02:38:47.028 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000101798 2020-06-12T02:38:47.028 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.029 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.029 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:47.029 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.211914+0000", 2020-06-12T02:38:47.029 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045163349999999998 2020-06-12T02:38:47.029 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.030 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.030 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:47.031 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.216431+0000", 2020-06-12T02:38:47.031 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024128666 2020-06-12T02:38:47.031 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:47.031 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:47.031 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:47.031 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.032 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.032 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:16810 3.ds0 3:b983fd28:::smithi10729407-1:head [append 0~524288] snapc 0=[] ondisk+write+known_if_redirected e61)", 2020-06-12T02:38:47.032 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:37.642363+0000", 2020-06-12T02:38:47.032 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.2255384849999995, 2020-06-12T02:38:47.032 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022969679, 2020-06-12T02:38:47.033 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:47.033 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:47.033 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:47.033 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:47.033 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:47.034 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 16810 2020-06-12T02:38:47.034 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.034 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:47.034 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.034 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:47.035 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.642363+0000", 2020-06-12T02:38:47.035 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:47.035 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.035 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.035 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:47.036 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.642363+0000", 2020-06-12T02:38:47.036 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00069050800000000005 2020-06-12T02:38:47.036 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.037 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.037 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:47.037 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.643054+0000", 2020-06-12T02:38:47.037 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8220000000000007e-06 2020-06-12T02:38:47.037 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.037 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.038 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:47.038 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.643062+0000", 2020-06-12T02:38:47.038 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3742e-05 2020-06-12T02:38:47.038 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.038 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.039 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:47.039 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.643075+0000", 2020-06-12T02:38:47.039 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5910000000000001e-06 2020-06-12T02:38:47.039 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.039 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.040 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:47.040 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.643082+0000", 2020-06-12T02:38:47.040 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0064097879999999996 2020-06-12T02:38:47.040 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.040 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.040 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:47.041 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.649492+0000", 2020-06-12T02:38:47.041 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8151999999999994e-05 2020-06-12T02:38:47.041 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.041 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.041 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:47.042 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.649560+0000", 2020-06-12T02:38:47.042 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0020294940000000002 2020-06-12T02:38:47.042 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.042 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.042 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:47.042 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.651590+0000", 2020-06-12T02:38:47.043 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5242999999999997e-05 2020-06-12T02:38:47.043 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.043 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.043 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:47.043 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.651675+0000", 2020-06-12T02:38:47.044 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00069163799999999995 2020-06-12T02:38:47.044 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.044 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.044 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:47.044 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.652366+0000", 2020-06-12T02:38:47.044 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012711642 2020-06-12T02:38:47.045 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.045 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.045 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:47.045 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.665078+0000", 2020-06-12T02:38:47.045 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017519200000000001 2020-06-12T02:38:47.046 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.046 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.046 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:47.046 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.665253+0000", 2020-06-12T02:38:47.046 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9867e-05 2020-06-12T02:38:47.047 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.047 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.047 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:47.047 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.665333+0000", 2020-06-12T02:38:47.047 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022969679 2020-06-12T02:38:47.047 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:47.048 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:47.048 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:47.048 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.048 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.048 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18287 3.ds0 3:b983fd28:::smithi10729407-1:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:38:47.049 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.975980+0000", 2020-06-12T02:38:47.049 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.8919216590000003, 2020-06-12T02:38:47.049 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.023612415000000001, 2020-06-12T02:38:47.049 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:47.049 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:47.049 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:47.050 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:47.050 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:47.050 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18287 2020-06-12T02:38:47.050 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.050 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:47.051 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.051 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:47.051 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.975980+0000", 2020-06-12T02:38:47.051 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:47.051 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.051 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.052 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:47.052 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.975980+0000", 2020-06-12T02:38:47.052 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0175999999999999e-05 2020-06-12T02:38:47.052 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.052 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.053 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:47.053 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976070+0000", 2020-06-12T02:38:47.053 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7489999999999998e-06 2020-06-12T02:38:47.053 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.053 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.053 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:47.054 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976078+0000", 2020-06-12T02:38:47.054 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5308999999999999e-05 2020-06-12T02:38:47.054 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.054 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.054 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:47.055 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976093+0000", 2020-06-12T02:38:47.055 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7149999999999997e-06 2020-06-12T02:38:47.055 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.055 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.055 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:47.055 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976098+0000", 2020-06-12T02:38:47.056 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0067110759999999998 2020-06-12T02:38:47.056 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.056 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.056 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:47.056 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.982809+0000", 2020-06-12T02:38:47.057 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011082999999999999 2020-06-12T02:38:47.057 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.057 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.057 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:47.057 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.982920+0000", 2020-06-12T02:38:47.057 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00043570099999999998 2020-06-12T02:38:47.058 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.058 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.058 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:47.058 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.983356+0000", 2020-06-12T02:38:47.058 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00076300800000000002 2020-06-12T02:38:47.059 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.059 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.059 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:47.059 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.984119+0000", 2020-06-12T02:38:47.059 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01538405 2020-06-12T02:38:47.059 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.060 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.060 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:47.060 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.999503+0000", 2020-06-12T02:38:47.060 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9801000000000003e-05 2020-06-12T02:38:47.060 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.061 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.061 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:47.061 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.999593+0000", 2020-06-12T02:38:47.061 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.023612415000000001 2020-06-12T02:38:47.061 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:47.061 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:47.062 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:47.062 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.062 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.062 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18288 3.ds0 3:b983fd28:::smithi10729407-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:38:47.062 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.976151+0000", 2020-06-12T02:38:47.063 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.8917507359999997, 2020-06-12T02:38:47.063 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025323907999999999, 2020-06-12T02:38:47.063 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:47.063 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:47.063 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:47.063 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:47.064 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:47.064 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18288 2020-06-12T02:38:47.064 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.064 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:47.064 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.065 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:47.065 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976151+0000", 2020-06-12T02:38:47.065 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:47.065 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.065 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.066 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:47.066 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976151+0000", 2020-06-12T02:38:47.066 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5615000000000002e-05 2020-06-12T02:38:47.066 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.066 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.066 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:47.067 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976207+0000", 2020-06-12T02:38:47.067 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6399999999999997e-06 2020-06-12T02:38:47.067 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.067 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.067 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:47.068 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976214+0000", 2020-06-12T02:38:47.068 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4823e-05 2020-06-12T02:38:47.068 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.068 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.069 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:47.069 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976229+0000", 2020-06-12T02:38:47.069 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1519999999999997e-06 2020-06-12T02:38:47.069 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.069 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.069 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:47.070 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976235+0000", 2020-06-12T02:38:47.070 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0077693090000000003 2020-06-12T02:38:47.070 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.070 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.070 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:47.071 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.984005+0000", 2020-06-12T02:38:47.071 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000101221 2020-06-12T02:38:47.071 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.071 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.071 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:47.071 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.984106+0000", 2020-06-12T02:38:47.072 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016277713999999999 2020-06-12T02:38:47.072 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.072 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.072 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:47.072 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.000384+0000", 2020-06-12T02:38:47.073 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7938999999999999e-05 2020-06-12T02:38:47.073 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.073 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.073 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:47.073 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.000482+0000", 2020-06-12T02:38:47.073 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00099349500000000006 2020-06-12T02:38:47.074 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.074 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.074 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:47.074 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.001475+0000", 2020-06-12T02:38:47.074 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025323907999999999 2020-06-12T02:38:47.075 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:47.075 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:47.075 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:47.075 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.075 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.075 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18307 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [append 0~507904] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:38:47.076 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.056119+0000", 2020-06-12T02:38:47.076 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.8117828530000004, 2020-06-12T02:38:47.076 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025879849, 2020-06-12T02:38:47.076 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:47.076 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:47.077 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:47.077 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:47.077 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:47.077 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18307 2020-06-12T02:38:47.077 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.077 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:47.078 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.078 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:47.078 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.056119+0000", 2020-06-12T02:38:47.078 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:47.078 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.079 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.079 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:47.079 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.056119+0000", 2020-06-12T02:38:47.079 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00052484099999999996 2020-06-12T02:38:47.079 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.080 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.080 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:47.080 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.056644+0000", 2020-06-12T02:38:47.080 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7149999999999997e-06 2020-06-12T02:38:47.080 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.080 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.081 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:47.081 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.056649+0000", 2020-06-12T02:38:47.081 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0046e-05 2020-06-12T02:38:47.081 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.081 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.082 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:47.082 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.056659+0000", 2020-06-12T02:38:47.082 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.848e-06 2020-06-12T02:38:47.082 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.082 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.082 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:47.083 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.056663+0000", 2020-06-12T02:38:47.083 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024537029999999998 2020-06-12T02:38:47.083 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.083 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.083 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:47.084 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.059117+0000", 2020-06-12T02:38:47.084 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000121261 2020-06-12T02:38:47.084 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.084 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.084 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:47.084 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.059238+0000", 2020-06-12T02:38:47.085 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001916484 2020-06-12T02:38:47.085 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.085 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.085 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:47.085 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.061155+0000", 2020-06-12T02:38:47.086 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9120000000000001e-05 2020-06-12T02:38:47.086 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.086 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.086 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:47.086 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.061234+0000", 2020-06-12T02:38:47.086 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00060383599999999996 2020-06-12T02:38:47.087 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.087 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.087 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:47.087 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.061838+0000", 2020-06-12T02:38:47.087 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010259862999999999 2020-06-12T02:38:47.088 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.088 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.088 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:47.088 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.072098+0000", 2020-06-12T02:38:47.088 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0098176419999999997 2020-06-12T02:38:47.088 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.089 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.089 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:47.089 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.081915+0000", 2020-06-12T02:38:47.089 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3490000000000002e-05 2020-06-12T02:38:47.089 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.090 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.090 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:47.090 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.081999+0000", 2020-06-12T02:38:47.090 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025879849 2020-06-12T02:38:47.090 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:47.090 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:47.091 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:47.091 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.091 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.091 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18311 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:38:47.091 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.063458+0000", 2020-06-12T02:38:47.092 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.8044442170000004, 2020-06-12T02:38:47.092 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.035342152000000002, 2020-06-12T02:38:47.092 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:47.092 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:47.092 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:47.092 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:47.093 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:47.093 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18311 2020-06-12T02:38:47.093 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.093 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:47.093 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.094 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:47.094 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.063458+0000", 2020-06-12T02:38:47.094 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:47.094 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.094 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.094 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:47.095 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.063458+0000", 2020-06-12T02:38:47.095 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3773000000000003e-05 2020-06-12T02:38:47.095 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.095 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.095 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:47.096 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.063501+0000", 2020-06-12T02:38:47.096 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0780000000000001e-06 2020-06-12T02:38:47.096 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.096 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.096 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:47.096 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.063508+0000", 2020-06-12T02:38:47.097 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3020000000000001e-05 2020-06-12T02:38:47.097 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.097 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.097 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:47.097 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.063521+0000", 2020-06-12T02:38:47.098 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4839999999999998e-06 2020-06-12T02:38:47.098 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.098 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.098 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:47.098 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.063525+0000", 2020-06-12T02:38:47.098 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0087769890000000007 2020-06-12T02:38:47.099 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.099 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.099 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:47.099 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.072302+0000", 2020-06-12T02:38:47.099 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3121000000000005e-05 2020-06-12T02:38:47.100 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.100 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.100 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:47.100 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.072395+0000", 2020-06-12T02:38:47.100 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010158355000000001 2020-06-12T02:38:47.100 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.101 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.101 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:47.101 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.082554+0000", 2020-06-12T02:38:47.101 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5016000000000001e-05 2020-06-12T02:38:47.101 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.102 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.102 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:47.102 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.082629+0000", 2020-06-12T02:38:47.102 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016171316000000002 2020-06-12T02:38:47.102 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.103 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.103 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:47.103 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.098800+0000", 2020-06-12T02:38:47.103 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.035342152000000002 2020-06-12T02:38:47.103 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:47.103 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:47.104 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:47.104 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.104 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.104 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18313 3.fs0 3:f83c3dea:::smithi10729407-4:head [append 0~442368] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:38:47.104 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.070249+0000", 2020-06-12T02:38:47.105 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.7976529399999999, 2020-06-12T02:38:47.105 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028853489, 2020-06-12T02:38:47.105 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:47.105 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:47.105 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:47.105 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:47.106 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:47.106 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18313 2020-06-12T02:38:47.106 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.106 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:47.106 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.107 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:47.107 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.070249+0000", 2020-06-12T02:38:47.107 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:47.107 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.107 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.108 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:47.109 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.070249+0000", 2020-06-12T02:38:47.109 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00046159400000000001 2020-06-12T02:38:47.109 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.110 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.110 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:47.110 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.070711+0000", 2020-06-12T02:38:47.111 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2539999999999992e-06 2020-06-12T02:38:47.111 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.111 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.112 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:47.112 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.070719+0000", 2020-06-12T02:38:47.112 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3297e-05 2020-06-12T02:38:47.113 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.113 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.113 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:47.114 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.070732+0000", 2020-06-12T02:38:47.114 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0559999999999999e-06 2020-06-12T02:38:47.114 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.115 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.115 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:47.115 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.070739+0000", 2020-06-12T02:38:47.115 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0052673319999999996 2020-06-12T02:38:47.116 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.116 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.116 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:47.116 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.076006+0000", 2020-06-12T02:38:47.117 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6925999999999997e-05 2020-06-12T02:38:47.117 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.117 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.118 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:47.118 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.076063+0000", 2020-06-12T02:38:47.118 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0076712209999999998 2020-06-12T02:38:47.118 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.118 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.119 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:47.119 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.083735+0000", 2020-06-12T02:38:47.119 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8089000000000002e-05 2020-06-12T02:38:47.119 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.119 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.119 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:47.120 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.083833+0000", 2020-06-12T02:38:47.120 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054620600000000004 2020-06-12T02:38:47.120 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.120 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.120 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:47.121 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.084379+0000", 2020-06-12T02:38:47.121 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013940945999999999 2020-06-12T02:38:47.121 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.121 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.121 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:47.122 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.098320+0000", 2020-06-12T02:38:47.122 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00072638599999999998 2020-06-12T02:38:47.122 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.122 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.123 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:47.123 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.099046+0000", 2020-06-12T02:38:47.123 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6181999999999998e-05 2020-06-12T02:38:47.124 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.124 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.124 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:47.125 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.099102+0000", 2020-06-12T02:38:47.125 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028853489 2020-06-12T02:38:47.125 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:47.125 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:47.125 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:47.126 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.126 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.126 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18314 3.fs0 3:f83c3dea:::smithi10729407-4:head [append 0~573440] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:38:47.126 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.074005+0000", 2020-06-12T02:38:47.126 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.793896921, 2020-06-12T02:38:47.127 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031442581999999997, 2020-06-12T02:38:47.127 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:47.127 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:47.127 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:47.127 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:47.127 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:47.128 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18314 2020-06-12T02:38:47.128 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.129 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:47.129 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.129 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:47.129 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.074005+0000", 2020-06-12T02:38:47.129 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:47.130 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.130 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.130 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:47.130 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.074005+0000", 2020-06-12T02:38:47.130 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00056762000000000002 2020-06-12T02:38:47.131 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.131 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.131 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:47.131 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.074573+0000", 2020-06-12T02:38:47.131 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5400000000000003e-06 2020-06-12T02:38:47.131 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.132 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.132 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:47.132 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.074578+0000", 2020-06-12T02:38:47.132 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1090000000000004e-06 2020-06-12T02:38:47.133 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.133 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.133 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:47.133 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.074587+0000", 2020-06-12T02:38:47.133 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4039999999999997e-06 2020-06-12T02:38:47.142 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.142 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.143 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:47.143 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.074592+0000", 2020-06-12T02:38:47.143 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001538031 2020-06-12T02:38:47.144 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.144 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.144 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:47.145 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.076130+0000", 2020-06-12T02:38:47.145 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5117000000000002e-05 2020-06-12T02:38:47.146 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.146 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.146 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:47.147 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.076175+0000", 2020-06-12T02:38:47.147 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011822262 2020-06-12T02:38:47.147 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.148 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.148 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:47.148 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.087997+0000", 2020-06-12T02:38:47.149 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000112318 2020-06-12T02:38:47.149 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.149 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.150 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:47.150 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.088109+0000", 2020-06-12T02:38:47.150 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00072351100000000003 2020-06-12T02:38:47.151 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.151 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.151 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:47.152 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.088833+0000", 2020-06-12T02:38:47.152 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010307874999999999 2020-06-12T02:38:47.152 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.153 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.153 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:47.153 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.099141+0000", 2020-06-12T02:38:47.154 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.006229605 2020-06-12T02:38:47.154 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.154 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.155 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:47.155 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.105370+0000", 2020-06-12T02:38:47.155 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7189999999999998e-05 2020-06-12T02:38:47.155 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.156 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.156 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:47.156 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.105448+0000", 2020-06-12T02:38:47.157 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031442581999999997 2020-06-12T02:38:47.157 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:47.157 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:47.158 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:47.158 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.158 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.159 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18315 3.fs0 3:f83c3dea:::smithi10729407-4:head [append 0~589824] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:38:47.159 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.077724+0000", 2020-06-12T02:38:47.159 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.7901780619999998, 2020-06-12T02:38:47.160 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.027939051999999999, 2020-06-12T02:38:47.160 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:47.160 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:47.161 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:47.161 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:47.161 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:47.161 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18315 2020-06-12T02:38:47.162 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.162 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:47.162 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.163 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:47.163 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.077724+0000", 2020-06-12T02:38:47.163 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:47.164 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.164 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.164 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:47.165 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.077724+0000", 2020-06-12T02:38:47.165 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00093652299999999998 2020-06-12T02:38:47.165 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.166 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.166 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:47.166 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078660+0000", 2020-06-12T02:38:47.167 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1389999999999995e-06 2020-06-12T02:38:47.167 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.167 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.167 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:47.168 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078669+0000", 2020-06-12T02:38:47.168 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3691e-05 2020-06-12T02:38:47.168 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.169 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.169 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:47.169 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078682+0000", 2020-06-12T02:38:47.170 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0100000000000001e-06 2020-06-12T02:38:47.170 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.170 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.171 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:47.171 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078688+0000", 2020-06-12T02:38:47.171 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1209999999999999e-05 2020-06-12T02:38:47.172 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.172 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.172 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:47.173 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078739+0000", 2020-06-12T02:38:47.173 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000102735 2020-06-12T02:38:47.173 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.174 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.174 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:47.174 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078842+0000", 2020-06-12T02:38:47.174 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01321815 2020-06-12T02:38:47.174 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.175 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.175 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:47.175 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.092060+0000", 2020-06-12T02:38:47.175 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00038608499999999999 2020-06-12T02:38:47.175 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.175 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.176 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:47.176 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.092446+0000", 2020-06-12T02:38:47.176 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00056703299999999999 2020-06-12T02:38:47.176 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.177 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.177 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:47.177 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.093013+0000", 2020-06-12T02:38:47.177 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0061289530000000004 2020-06-12T02:38:47.177 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.178 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.178 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:47.178 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.099142+0000", 2020-06-12T02:38:47.178 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0064547700000000003 2020-06-12T02:38:47.178 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.179 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.179 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:47.179 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.105597+0000", 2020-06-12T02:38:47.179 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5753000000000006e-05 2020-06-12T02:38:47.179 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.179 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.180 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:47.180 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.105663+0000", 2020-06-12T02:38:47.180 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.027939051999999999 2020-06-12T02:38:47.180 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:47.180 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:47.181 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:47.181 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.181 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.181 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18316 3.fs0 3:f83c3dea:::smithi10729407-4:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:38:47.183 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.078752+0000", 2020-06-12T02:38:47.183 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.7891502900000003, 2020-06-12T02:38:47.183 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.029108709999999999, 2020-06-12T02:38:47.183 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:47.183 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:47.184 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:47.184 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:47.184 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:47.184 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18316 2020-06-12T02:38:47.184 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.184 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:47.185 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.185 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:47.185 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078752+0000", 2020-06-12T02:38:47.185 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:47.185 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.186 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.186 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:47.186 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078752+0000", 2020-06-12T02:38:47.186 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8967000000000004e-05 2020-06-12T02:38:47.187 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.187 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.187 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:47.187 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078841+0000", 2020-06-12T02:38:47.187 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3309999999999998e-06 2020-06-12T02:38:47.188 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.188 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.188 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:47.188 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078848+0000", 2020-06-12T02:38:47.188 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.291e-05 2020-06-12T02:38:47.189 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.189 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.189 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:47.189 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078861+0000", 2020-06-12T02:38:47.189 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2800000000000003e-06 2020-06-12T02:38:47.189 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.190 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.190 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:47.190 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078866+0000", 2020-06-12T02:38:47.190 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8543e-05 2020-06-12T02:38:47.191 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.191 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.192 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:47.192 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078905+0000", 2020-06-12T02:38:47.192 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9198999999999998e-05 2020-06-12T02:38:47.192 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.192 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.193 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:47.193 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078994+0000", 2020-06-12T02:38:47.193 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01781437 2020-06-12T02:38:47.193 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.193 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.194 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:47.194 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.096808+0000", 2020-06-12T02:38:47.194 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000115975 2020-06-12T02:38:47.194 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.195 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.195 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:47.195 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.096924+0000", 2020-06-12T02:38:47.195 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00049986300000000002 2020-06-12T02:38:47.195 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.196 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.196 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:47.196 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.097424+0000", 2020-06-12T02:38:47.196 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0017191089999999999 2020-06-12T02:38:47.196 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.197 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.197 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:47.197 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.099143+0000", 2020-06-12T02:38:47.197 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0086177909999999996 2020-06-12T02:38:47.197 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.197 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.198 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:47.198 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.107761+0000", 2020-06-12T02:38:47.198 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9371999999999997e-05 2020-06-12T02:38:47.198 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.198 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.199 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:47.199 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.107860+0000", 2020-06-12T02:38:47.199 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.029108709999999999 2020-06-12T02:38:47.199 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:47.199 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:47.199 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:47.200 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.200 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.200 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18317 3.fs0 3:f83c3dea:::smithi10729407-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:38:47.200 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.078920+0000", 2020-06-12T02:38:47.200 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.7889823199999997, 2020-06-12T02:38:47.201 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031356964000000001, 2020-06-12T02:38:47.201 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:47.201 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:47.201 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:47.201 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:47.202 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:47.202 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18317 2020-06-12T02:38:47.202 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.202 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:47.202 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.202 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:47.203 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078920+0000", 2020-06-12T02:38:47.203 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:47.203 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.203 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.203 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:47.204 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078920+0000", 2020-06-12T02:38:47.204 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6454000000000001e-05 2020-06-12T02:38:47.204 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.204 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.204 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:47.204 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078946+0000", 2020-06-12T02:38:47.205 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8160000000000004e-06 2020-06-12T02:38:47.205 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.205 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.205 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:47.205 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078950+0000", 2020-06-12T02:38:47.206 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6090000000000003e-06 2020-06-12T02:38:47.206 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.206 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.206 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:47.207 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078957+0000", 2020-06-12T02:38:47.207 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8380000000000002e-06 2020-06-12T02:38:47.207 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.208 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.208 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:47.208 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078960+0000", 2020-06-12T02:38:47.209 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8400000000000003e-05 2020-06-12T02:38:47.209 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.209 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.209 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:47.209 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.079019+0000", 2020-06-12T02:38:47.209 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6750999999999998e-05 2020-06-12T02:38:47.210 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.210 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.210 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:47.210 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.079085+0000", 2020-06-12T02:38:47.210 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019272036999999999 2020-06-12T02:38:47.211 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.211 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.211 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:47.211 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.098357+0000", 2020-06-12T02:38:47.211 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4322999999999998e-05 2020-06-12T02:38:47.211 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.212 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.212 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:47.212 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.098452+0000", 2020-06-12T02:38:47.212 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010578025 2020-06-12T02:38:47.212 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.213 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.213 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:47.213 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.109030+0000", 2020-06-12T02:38:47.213 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010395099999999999 2020-06-12T02:38:47.213 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.213 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.214 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:47.214 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.109134+0000", 2020-06-12T02:38:47.214 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00114276 2020-06-12T02:38:47.214 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:47.214 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:47.215 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:47.215 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.110277+0000", 2020-06-12T02:38:47.215 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031356964000000001 2020-06-12T02:38:47.215 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:47.215 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:47.215 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:47.216 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:47.216 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:47.216 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:47.218 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:47.215+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:47.219 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-06-12T02:38:47.318 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:47.315+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:47.339 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:47.340 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2020-06-12T02:38:47.341 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.341 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:19543 3.17s0 3.e008e457 (undecoded) ondisk+write+known_if_redirected e66)", 2020-06-12T02:38:47.341 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:46.509417+0000", 2020-06-12T02:38:47.342 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.82834096400000001, 2020-06-12T02:38:47.342 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.82836061900000002, 2020-06-12T02:38:47.342 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:47.342 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "delayed", 2020-06-12T02:38:47.343 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:47.343 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:47.343 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:47.343 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 19543 2020-06-12T02:38:47.344 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.344 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:47.344 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.344 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:47.345 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:46.509417+0000", 2020-06-12T02:38:47.345 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:47.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.345 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.346 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:47.346 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:46.509417+0000", 2020-06-12T02:38:47.346 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.4377000000000001e-05 2020-06-12T02:38:47.346 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.346 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.347 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:47.347 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:46.509442+0000", 2020-06-12T02:38:47.347 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0889999999999999e-06 2020-06-12T02:38:47.347 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.347 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.348 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:47.348 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:46.509447+0000", 2020-06-12T02:38:47.348 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0669999999999999e-05 2020-06-12T02:38:47.348 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.348 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.349 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:47.349 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:46.509458+0000", 2020-06-12T02:38:47.349 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3920000000000002e-06 2020-06-12T02:38:47.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.349 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.350 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:47.350 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:46.509463+0000", 2020-06-12T02:38:47.350 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6390000000000002e-05 2020-06-12T02:38:47.350 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.350 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.351 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:47.351 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:46.509499+0000", 2020-06-12T02:38:47.351 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.1709999999999998e-06 2020-06-12T02:38:47.352 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.352 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.352 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2020-06-12T02:38:47.352 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:46.509503+0000", 2020-06-12T02:38:47.352 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5088999999999998e-05 2020-06-12T02:38:47.353 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:47.353 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:47.353 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:47.353 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:47.354 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2020-06-12T02:38:47.355 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 1 2020-06-12T02:38:47.355 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:47.356 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-06-12T02:38:47.445 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:47.441+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:47.517 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd set noscrub 2020-06-12T02:38:47.520 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:47.517+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:47.529 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:47.530 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:38:47.530 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2020-06-12T02:38:47.530 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2020-06-12T02:38:47.530 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:47.541 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-06-12T02:38:47.621 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:47.618+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:47.678 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2020-06-12T02:38:47.679 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd reweight 0 1 2020-06-12T02:38:47.705 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:47.705 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2020-06-12T02:38:47.706 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2020-06-12T02:38:47.706 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2020-06-12T02:38:47.706 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.706 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.3s2 49/21 ECSubWrite(tid=4109, reqid=client.4512.0:8841, at_version=49'773, trim_to=0'0, roll_forward_to=49'771))", 2020-06-12T02:38:47.706 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.822144+0000", 2020-06-12T02:38:47.707 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 35.879230726999999, 2020-06-12T02:38:47.707 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014799214, 2020-06-12T02:38:47.707 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:47.707 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:47.707 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:47.708 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.708 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:47.708 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822144+0000", 2020-06-12T02:38:47.708 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:47.708 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.709 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.709 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:47.709 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822144+0000", 2020-06-12T02:38:47.709 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00061470800000000005 2020-06-12T02:38:47.709 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.709 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.710 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:47.710 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822759+0000", 2020-06-12T02:38:47.710 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.7732000000000001e-05 2020-06-12T02:38:47.710 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.710 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.711 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:47.711 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822786+0000", 2020-06-12T02:38:47.711 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1010999999999999e-05 2020-06-12T02:38:47.711 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.711 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.711 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:47.712 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822807+0000", 2020-06-12T02:38:47.712 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.842e-06 2020-06-12T02:38:47.712 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.712 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.712 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:47.713 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822817+0000", 2020-06-12T02:38:47.713 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00024702300000000002 2020-06-12T02:38:47.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.713 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.714 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:47.714 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.823064+0000", 2020-06-12T02:38:47.714 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5207e-05 2020-06-12T02:38:47.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.715 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.715 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:47.722 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.823079+0000", 2020-06-12T02:38:47.722 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013806817000000001 2020-06-12T02:38:47.722 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.722 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.723 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:47.723 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.836886+0000", 2020-06-12T02:38:47.723 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6873999999999998e-05 2020-06-12T02:38:47.723 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.723 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.724 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:47.724 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.836943+0000", 2020-06-12T02:38:47.724 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014799214 2020-06-12T02:38:47.724 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:47.724 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:47.725 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:47.725 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.725 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.725 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14657 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~606208] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:47.725 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.209225+0000", 2020-06-12T02:38:47.725 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 16.492149476000002, 2020-06-12T02:38:47.726 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016128905999999998, 2020-06-12T02:38:47.726 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:47.726 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:47.726 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:47.726 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:47.727 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:47.727 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14657 2020-06-12T02:38:47.727 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.727 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:47.727 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.727 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:47.728 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209225+0000", 2020-06-12T02:38:47.728 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:47.728 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.728 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.728 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:47.729 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209225+0000", 2020-06-12T02:38:47.729 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018386699999999999 2020-06-12T02:38:47.729 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.729 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.729 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:47.730 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209409+0000", 2020-06-12T02:38:47.731 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8160000000000001e-06 2020-06-12T02:38:47.731 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.731 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.732 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:47.732 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209414+0000", 2020-06-12T02:38:47.732 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0230000000000004e-06 2020-06-12T02:38:47.733 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.733 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.733 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:47.733 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209421+0000", 2020-06-12T02:38:47.734 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0139999999999999e-06 2020-06-12T02:38:47.734 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.735 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.735 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:47.735 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209424+0000", 2020-06-12T02:38:47.735 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0054798169999999997 2020-06-12T02:38:47.736 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.737 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.737 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:47.737 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.214904+0000", 2020-06-12T02:38:47.738 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123063 2020-06-12T02:38:47.739 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.739 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.739 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:47.740 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215027+0000", 2020-06-12T02:38:47.740 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001159923 2020-06-12T02:38:47.740 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.740 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.740 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:47.741 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.216187+0000", 2020-06-12T02:38:47.741 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2835000000000002e-05 2020-06-12T02:38:47.741 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.742 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.742 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:47.742 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.216270+0000", 2020-06-12T02:38:47.742 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073575799999999998 2020-06-12T02:38:47.742 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.743 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.743 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:47.743 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.217005+0000", 2020-06-12T02:38:47.743 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0082434659999999996 2020-06-12T02:38:47.743 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.744 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.744 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:47.744 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225249+0000", 2020-06-12T02:38:47.744 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0145999999999997e-05 2020-06-12T02:38:47.744 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.745 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.745 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:47.745 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225299+0000", 2020-06-12T02:38:47.745 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5177999999999999e-05 2020-06-12T02:38:47.745 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.746 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.746 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:47.753 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225354+0000", 2020-06-12T02:38:47.753 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016128905999999998 2020-06-12T02:38:47.753 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:47.754 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:47.754 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:47.754 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.754 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.754 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14658 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~598016] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:47.755 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.209455+0000", 2020-06-12T02:38:47.755 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 16.491919721999999, 2020-06-12T02:38:47.755 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016447966000000001, 2020-06-12T02:38:47.755 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:47.755 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:47.755 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:47.756 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:47.756 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:47.756 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14658 2020-06-12T02:38:47.756 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.756 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:47.757 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.757 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:47.757 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209455+0000", 2020-06-12T02:38:47.757 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:47.757 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.758 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.758 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:47.758 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209455+0000", 2020-06-12T02:38:47.758 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037921599999999998 2020-06-12T02:38:47.758 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.759 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.759 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:47.760 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209834+0000", 2020-06-12T02:38:47.760 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8339999999999999e-06 2020-06-12T02:38:47.760 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.760 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.761 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:47.761 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209842+0000", 2020-06-12T02:38:47.761 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2564e-05 2020-06-12T02:38:47.762 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.762 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.763 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:47.764 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209855+0000", 2020-06-12T02:38:47.764 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.135e-06 2020-06-12T02:38:47.764 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.764 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.765 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:47.765 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209860+0000", 2020-06-12T02:38:47.765 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0052530290000000002 2020-06-12T02:38:47.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.766 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.766 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:47.766 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215113+0000", 2020-06-12T02:38:47.767 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6229000000000003e-05 2020-06-12T02:38:47.767 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.767 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.768 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:47.768 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215209+0000", 2020-06-12T02:38:47.768 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045765409999999999 2020-06-12T02:38:47.769 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.769 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.770 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:47.770 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.219785+0000", 2020-06-12T02:38:47.770 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000102846 2020-06-12T02:38:47.770 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.771 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.771 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:47.771 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.219888+0000", 2020-06-12T02:38:47.772 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00077088700000000005 2020-06-12T02:38:47.772 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.772 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.773 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:47.774 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.220659+0000", 2020-06-12T02:38:47.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046979739999999997 2020-06-12T02:38:47.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.776 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.776 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:47.776 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225357+0000", 2020-06-12T02:38:47.776 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00046069600000000002 2020-06-12T02:38:47.776 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.777 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.777 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:47.777 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225818+0000", 2020-06-12T02:38:47.777 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5014999999999998e-05 2020-06-12T02:38:47.777 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.778 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.778 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:47.778 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225903+0000", 2020-06-12T02:38:47.778 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016447966000000001 2020-06-12T02:38:47.778 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:47.779 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:47.779 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:47.779 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.779 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.779 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14659 3.0s0 3:0dddb690:::smithi10729407-10:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:47.780 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.209909+0000", 2020-06-12T02:38:47.780 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 16.491465514000001, 2020-06-12T02:38:47.780 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016618236000000002, 2020-06-12T02:38:47.780 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:47.780 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:47.781 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:47.781 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:47.781 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:47.781 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14659 2020-06-12T02:38:47.781 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.782 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:47.782 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.782 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:47.782 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209909+0000", 2020-06-12T02:38:47.782 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:47.783 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.783 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.783 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:47.784 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209909+0000", 2020-06-12T02:38:47.784 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8085999999999999e-05 2020-06-12T02:38:47.784 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.784 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.784 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:47.785 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209967+0000", 2020-06-12T02:38:47.786 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3169999999999998e-06 2020-06-12T02:38:47.786 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.786 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.786 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:47.786 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209974+0000", 2020-06-12T02:38:47.787 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1246999999999999e-05 2020-06-12T02:38:47.787 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.787 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.787 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:47.787 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209985+0000", 2020-06-12T02:38:47.788 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1439999999999996e-06 2020-06-12T02:38:47.788 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.788 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.788 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:47.788 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209989+0000", 2020-06-12T02:38:47.789 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0052537900000000004 2020-06-12T02:38:47.789 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.789 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.789 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:47.789 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215243+0000", 2020-06-12T02:38:47.789 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6023000000000002e-05 2020-06-12T02:38:47.790 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.790 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.790 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:47.790 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215299+0000", 2020-06-12T02:38:47.790 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0087846910000000007 2020-06-12T02:38:47.791 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.791 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.791 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:47.791 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.224083+0000", 2020-06-12T02:38:47.791 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8504000000000004e-05 2020-06-12T02:38:47.792 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.792 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.792 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:47.792 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.224182+0000", 2020-06-12T02:38:47.792 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044251400000000003 2020-06-12T02:38:47.792 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.793 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.793 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:47.794 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.224624+0000", 2020-06-12T02:38:47.795 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013383290000000001 2020-06-12T02:38:47.795 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.795 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.795 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:47.796 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225963+0000", 2020-06-12T02:38:47.796 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047942699999999999 2020-06-12T02:38:47.796 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.796 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.796 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:47.797 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.226442+0000", 2020-06-12T02:38:47.797 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5164e-05 2020-06-12T02:38:47.797 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.797 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.797 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:47.798 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.226527+0000", 2020-06-12T02:38:47.798 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016618236000000002 2020-06-12T02:38:47.798 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:47.798 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:47.798 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:47.799 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.799 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.799 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14660 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e59)", 2020-06-12T02:38:47.799 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.210020+0000", 2020-06-12T02:38:47.799 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 16.491354535999999, 2020-06-12T02:38:47.800 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017984777, 2020-06-12T02:38:47.800 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:47.800 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:47.800 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:47.800 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:47.800 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:47.801 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14660 2020-06-12T02:38:47.801 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.801 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:47.801 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.801 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:47.803 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210020+0000", 2020-06-12T02:38:47.803 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:47.803 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.803 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.804 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:47.804 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210020+0000", 2020-06-12T02:38:47.804 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2875999999999999e-05 2020-06-12T02:38:47.804 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.804 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.804 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:47.805 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210043+0000", 2020-06-12T02:38:47.805 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2200000000000001e-06 2020-06-12T02:38:47.805 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.805 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.805 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:47.806 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210046+0000", 2020-06-12T02:38:47.806 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7410000000000003e-06 2020-06-12T02:38:47.806 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.806 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.806 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:47.807 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210053+0000", 2020-06-12T02:38:47.807 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5610000000000001e-06 2020-06-12T02:38:47.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.807 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.807 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:47.808 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210055+0000", 2020-06-12T02:38:47.808 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0052596409999999998 2020-06-12T02:38:47.809 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.809 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.809 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:47.809 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215315+0000", 2020-06-12T02:38:47.810 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5627999999999997e-05 2020-06-12T02:38:47.810 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.810 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.810 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:47.810 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215371+0000", 2020-06-12T02:38:47.811 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0098069549999999991 2020-06-12T02:38:47.811 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.811 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.811 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:47.811 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225178+0000", 2020-06-12T02:38:47.811 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4530000000000001e-05 2020-06-12T02:38:47.812 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.812 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.812 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:47.812 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225232+0000", 2020-06-12T02:38:47.812 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017497350000000001 2020-06-12T02:38:47.813 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.813 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.813 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:47.813 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.226982+0000", 2020-06-12T02:38:47.813 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8154999999999998e-05 2020-06-12T02:38:47.814 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.814 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.814 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:47.814 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.227080+0000", 2020-06-12T02:38:47.814 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00092473499999999999 2020-06-12T02:38:47.814 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.815 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.815 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:47.815 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.228005+0000", 2020-06-12T02:38:47.816 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017984777 2020-06-12T02:38:47.816 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:47.816 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:47.817 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:47.817 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.817 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.817 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.3s2 59/53 ECSubWrite(tid=5970, reqid=client.4512.0:15226, at_version=59'1316, trim_to=0'0, roll_forward_to=59'1315))", 2020-06-12T02:38:47.817 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.829184+0000", 2020-06-12T02:38:47.818 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 14.87219032, 2020-06-12T02:38:47.818 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.081295588000000002, 2020-06-12T02:38:47.818 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:47.818 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:47.818 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:47.819 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.819 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:47.819 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829184+0000", 2020-06-12T02:38:47.819 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:47.819 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.820 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.820 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:47.820 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829184+0000", 2020-06-12T02:38:47.820 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00023814999999999999 2020-06-12T02:38:47.820 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.821 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.822 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:47.822 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829422+0000", 2020-06-12T02:38:47.822 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9768000000000001e-05 2020-06-12T02:38:47.823 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.823 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.823 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:47.824 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829442+0000", 2020-06-12T02:38:47.824 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1022000000000001e-05 2020-06-12T02:38:47.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.826 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.826 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:47.827 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829453+0000", 2020-06-12T02:38:47.827 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2759999999999997e-06 2020-06-12T02:38:47.828 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.828 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.828 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:47.829 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829460+0000", 2020-06-12T02:38:47.829 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7284000000000002e-05 2020-06-12T02:38:47.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.830 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.830 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:47.830 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829517+0000", 2020-06-12T02:38:47.830 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7028999999999999e-05 2020-06-12T02:38:47.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.831 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.831 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:47.831 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829534+0000", 2020-06-12T02:38:47.831 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.080899873999999997 2020-06-12T02:38:47.831 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.832 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.832 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:47.832 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.910434+0000", 2020-06-12T02:38:47.832 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6184999999999999e-05 2020-06-12T02:38:47.832 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.832 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.833 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:47.833 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.910480+0000", 2020-06-12T02:38:47.833 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.081295588000000002 2020-06-12T02:38:47.833 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:47.833 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:47.834 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:47.834 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.834 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.834 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.3s2 59/53 ECSubWrite(tid=5973, reqid=client.4512.0:15229, at_version=59'1319, trim_to=0'0, roll_forward_to=59'1315))", 2020-06-12T02:38:47.834 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.901708+0000", 2020-06-12T02:38:47.835 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 14.799666440999999, 2020-06-12T02:38:47.835 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013662957999999999, 2020-06-12T02:38:47.835 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:47.835 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:47.836 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:47.836 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.836 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:47.836 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901708+0000", 2020-06-12T02:38:47.836 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:47.837 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.837 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.838 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:47.838 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901708+0000", 2020-06-12T02:38:47.838 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7360000000000001e-05 2020-06-12T02:38:47.838 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.838 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.839 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:47.839 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901756+0000", 2020-06-12T02:38:47.839 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4608e-05 2020-06-12T02:38:47.839 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.839 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.840 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:47.841 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901770+0000", 2020-06-12T02:38:47.841 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.219e-05 2020-06-12T02:38:47.841 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.841 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.841 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:47.842 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901782+0000", 2020-06-12T02:38:47.842 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.511e-06 2020-06-12T02:38:47.842 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.842 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.842 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:47.843 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901789+0000", 2020-06-12T02:38:47.843 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.008025562 2020-06-12T02:38:47.843 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.843 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.844 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:47.844 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.909814+0000", 2020-06-12T02:38:47.844 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1472e-05 2020-06-12T02:38:47.844 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.844 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.845 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:47.845 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.909826+0000", 2020-06-12T02:38:47.845 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0055013350000000004 2020-06-12T02:38:47.845 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.845 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.846 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:47.846 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.915327+0000", 2020-06-12T02:38:47.846 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.392e-05 2020-06-12T02:38:47.846 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.847 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.847 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:47.847 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.915371+0000", 2020-06-12T02:38:47.847 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013662957999999999 2020-06-12T02:38:47.847 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:47.848 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:47.848 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:47.848 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.848 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.848 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.3s2 59/53 ECSubWrite(tid=5974, reqid=client.4512.0:15231, at_version=59'1320, trim_to=0'0, roll_forward_to=59'1315))", 2020-06-12T02:38:47.849 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.901839+0000", 2020-06-12T02:38:47.849 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 14.799535842999999, 2020-06-12T02:38:47.849 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013568525, 2020-06-12T02:38:47.849 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:47.849 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:47.850 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:47.851 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.851 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:47.851 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901839+0000", 2020-06-12T02:38:47.852 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:47.852 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.852 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.852 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:47.852 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901839+0000", 2020-06-12T02:38:47.852 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6489999999999997e-05 2020-06-12T02:38:47.853 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.853 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.853 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:47.853 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901885+0000", 2020-06-12T02:38:47.853 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6600999999999999e-05 2020-06-12T02:38:47.854 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.854 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.854 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:47.854 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901902+0000", 2020-06-12T02:38:47.854 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2427e-05 2020-06-12T02:38:47.855 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.855 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.855 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:47.855 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901914+0000", 2020-06-12T02:38:47.855 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1710000000000001e-06 2020-06-12T02:38:47.855 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.856 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.856 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:47.856 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901919+0000", 2020-06-12T02:38:47.856 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0086285540000000001 2020-06-12T02:38:47.856 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.857 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.857 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:47.857 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.910548+0000", 2020-06-12T02:38:47.857 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0939e-05 2020-06-12T02:38:47.857 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.857 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.858 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:47.859 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.910559+0000", 2020-06-12T02:38:47.859 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0048161899999999997 2020-06-12T02:38:47.859 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.859 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.859 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:47.860 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.915375+0000", 2020-06-12T02:38:47.860 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2153000000000002e-05 2020-06-12T02:38:47.860 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.860 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.860 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:47.861 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.915407+0000", 2020-06-12T02:38:47.861 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013568525 2020-06-12T02:38:47.861 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:47.861 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:47.862 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:47.862 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.862 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.862 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15246 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~671744] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:47.863 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.953872+0000", 2020-06-12T02:38:47.863 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 14.747502776999999, 2020-06-12T02:38:47.863 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015306691000000001, 2020-06-12T02:38:47.863 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:47.863 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:47.864 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:47.864 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:47.864 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:47.864 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15246 2020-06-12T02:38:47.864 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.864 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:47.865 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.865 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:47.865 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.953872+0000", 2020-06-12T02:38:47.865 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:47.865 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.866 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.866 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:47.866 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.953872+0000", 2020-06-12T02:38:47.866 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000232544 2020-06-12T02:38:47.866 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.867 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.867 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:47.867 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954104+0000", 2020-06-12T02:38:47.867 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7513999999999999e-05 2020-06-12T02:38:47.867 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.868 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.868 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:47.868 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954122+0000", 2020-06-12T02:38:47.868 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.844e-05 2020-06-12T02:38:47.869 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.869 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.869 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:47.869 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954140+0000", 2020-06-12T02:38:47.870 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2570000000000007e-06 2020-06-12T02:38:47.870 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.870 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.870 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:47.870 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954149+0000", 2020-06-12T02:38:47.871 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00031049499999999999 2020-06-12T02:38:47.872 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.872 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.872 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:47.872 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954459+0000", 2020-06-12T02:38:47.872 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000118182 2020-06-12T02:38:47.873 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.873 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.873 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:47.873 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954577+0000", 2020-06-12T02:38:47.873 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00071788400000000001 2020-06-12T02:38:47.874 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.874 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.877 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:47.878 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.955295+0000", 2020-06-12T02:38:47.878 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013717859000000001 2020-06-12T02:38:47.878 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.878 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.878 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:47.879 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.969013+0000", 2020-06-12T02:38:47.879 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4895999999999995e-05 2020-06-12T02:38:47.879 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.879 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.879 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:47.880 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.969098+0000", 2020-06-12T02:38:47.880 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0619999999999997e-05 2020-06-12T02:38:47.880 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.880 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.880 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:47.881 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.969179+0000", 2020-06-12T02:38:47.881 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015306691000000001 2020-06-12T02:38:47.881 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:47.881 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:47.881 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:47.882 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.882 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.882 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15277 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~720896] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:47.883 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:33.067109+0000", 2020-06-12T02:38:47.883 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 14.634265717, 2020-06-12T02:38:47.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015909191999999999, 2020-06-12T02:38:47.883 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:47.884 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:47.884 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:47.885 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:47.885 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:47.885 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15277 2020-06-12T02:38:47.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.886 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:47.886 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.886 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:47.886 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067109+0000", 2020-06-12T02:38:47.887 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:47.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:47.887 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067109+0000", 2020-06-12T02:38:47.887 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018431400000000001 2020-06-12T02:38:47.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.888 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:47.888 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067293+0000", 2020-06-12T02:38:47.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1790000000000002e-06 2020-06-12T02:38:47.889 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.889 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.889 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:47.889 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067300+0000", 2020-06-12T02:38:47.889 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5259999999999999e-06 2020-06-12T02:38:47.890 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.890 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.890 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:47.890 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067310+0000", 2020-06-12T02:38:47.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8179999999999997e-06 2020-06-12T02:38:47.891 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.891 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.891 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:47.892 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067314+0000", 2020-06-12T02:38:47.892 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6529999999999997e-05 2020-06-12T02:38:47.892 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.893 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.894 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:47.894 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067360+0000", 2020-06-12T02:38:47.894 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012724399999999999 2020-06-12T02:38:47.894 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.894 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.895 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:47.895 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067488+0000", 2020-06-12T02:38:47.895 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0010983989999999999 2020-06-12T02:38:47.895 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.895 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.895 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:47.896 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.068586+0000", 2020-06-12T02:38:47.896 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5165000000000001e-05 2020-06-12T02:38:47.896 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.896 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.897 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:47.897 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.068681+0000", 2020-06-12T02:38:47.897 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086510899999999997 2020-06-12T02:38:47.897 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.897 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.898 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:47.898 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.069546+0000", 2020-06-12T02:38:47.898 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013284429 2020-06-12T02:38:47.898 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.898 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.899 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:47.899 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.082831+0000", 2020-06-12T02:38:47.899 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4197000000000006e-05 2020-06-12T02:38:47.899 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.899 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.900 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:47.900 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.082925+0000", 2020-06-12T02:38:47.900 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3282000000000006e-05 2020-06-12T02:38:47.901 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.901 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.901 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:47.901 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.083018+0000", 2020-06-12T02:38:47.901 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015909191999999999 2020-06-12T02:38:47.902 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:47.902 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:47.902 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:47.902 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.903 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.903 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15331 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~688128] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:47.903 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:33.276748+0000", 2020-06-12T02:38:47.904 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 14.424626204999999, 2020-06-12T02:38:47.904 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016825066999999999, 2020-06-12T02:38:47.904 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:47.904 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:47.904 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:47.905 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:47.905 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:47.905 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15331 2020-06-12T02:38:47.905 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.905 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:47.905 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.906 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:47.906 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276748+0000", 2020-06-12T02:38:47.906 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:47.906 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.906 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.907 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:47.907 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276748+0000", 2020-06-12T02:38:47.907 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016936699999999999 2020-06-12T02:38:47.907 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.907 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.908 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:47.908 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276918+0000", 2020-06-12T02:38:47.908 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6389999999999998e-06 2020-06-12T02:38:47.908 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.908 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.908 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:47.909 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276926+0000", 2020-06-12T02:38:47.909 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3591000000000001e-05 2020-06-12T02:38:47.909 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.909 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.909 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:47.910 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276940+0000", 2020-06-12T02:38:47.910 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0429999999999996e-06 2020-06-12T02:38:47.911 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.911 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.911 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:47.911 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276949+0000", 2020-06-12T02:38:47.911 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8825000000000001e-05 2020-06-12T02:38:47.911 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.912 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.912 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:47.912 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276998+0000", 2020-06-12T02:38:47.912 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6483000000000005e-05 2020-06-12T02:38:47.912 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.913 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.913 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:47.914 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.277084+0000", 2020-06-12T02:38:47.914 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00057545800000000005 2020-06-12T02:38:47.914 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.914 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.915 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:47.915 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.277660+0000", 2020-06-12T02:38:47.915 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015617126 2020-06-12T02:38:47.915 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.916 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.916 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:47.916 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.293277+0000", 2020-06-12T02:38:47.916 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00021931699999999999 2020-06-12T02:38:47.916 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.917 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.917 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:47.917 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.293496+0000", 2020-06-12T02:38:47.917 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7218000000000005e-05 2020-06-12T02:38:47.917 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.917 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.918 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:47.918 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.293573+0000", 2020-06-12T02:38:47.918 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016825066999999999 2020-06-12T02:38:47.918 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:47.918 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:47.919 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:47.919 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.919 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.919 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.7s1 60/37 ECSubWrite(tid=8504, reqid=client.4512.0:16225, at_version=60'1427, trim_to=0'0, roll_forward_to=60'1425))", 2020-06-12T02:38:47.919 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:36.033154+0000", 2020-06-12T02:38:47.919 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 11.668220843, 2020-06-12T02:38:47.920 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016768244000000002, 2020-06-12T02:38:47.920 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:47.921 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:47.921 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:47.921 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.921 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:47.921 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033154+0000", 2020-06-12T02:38:47.922 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:47.922 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.922 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.922 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:47.922 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033154+0000", 2020-06-12T02:38:47.923 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000154444 2020-06-12T02:38:47.923 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.924 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.924 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:47.924 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033308+0000", 2020-06-12T02:38:47.924 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8772000000000002e-05 2020-06-12T02:38:47.925 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.925 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.925 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:47.925 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033337+0000", 2020-06-12T02:38:47.925 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6704000000000001e-05 2020-06-12T02:38:47.925 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.926 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.926 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:47.926 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033354+0000", 2020-06-12T02:38:47.926 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6100000000000002e-06 2020-06-12T02:38:47.926 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.927 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.927 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:47.927 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033360+0000", 2020-06-12T02:38:47.927 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000173893 2020-06-12T02:38:47.927 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.927 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.928 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:47.928 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033534+0000", 2020-06-12T02:38:47.928 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1929999999999999e-05 2020-06-12T02:38:47.928 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.928 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.929 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:47.929 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033546+0000", 2020-06-12T02:38:47.929 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016272066000000002 2020-06-12T02:38:47.929 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.929 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.929 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:47.930 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.049818+0000", 2020-06-12T02:38:47.930 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103825 2020-06-12T02:38:47.930 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.930 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.931 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:47.931 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.049922+0000", 2020-06-12T02:38:47.931 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016768244000000002 2020-06-12T02:38:47.931 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:47.932 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:47.932 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:47.933 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.933 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.933 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:17973 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~507904] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:38:47.933 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:41.095893+0000", 2020-06-12T02:38:47.933 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 6.6054812900000002, 2020-06-12T02:38:47.934 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013655560000000001, 2020-06-12T02:38:47.934 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:47.934 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:47.934 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:47.934 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:47.935 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:47.935 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 17973 2020-06-12T02:38:47.935 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.935 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:47.935 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.936 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:47.936 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.095893+0000", 2020-06-12T02:38:47.936 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:47.936 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.936 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.936 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:47.937 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.095893+0000", 2020-06-12T02:38:47.937 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018351000000000001 2020-06-12T02:38:47.937 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.937 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.937 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:47.938 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.096077+0000", 2020-06-12T02:38:47.938 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1559999999999995e-06 2020-06-12T02:38:47.938 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.938 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.938 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:47.939 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.096085+0000", 2020-06-12T02:38:47.939 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3764999999999999e-05 2020-06-12T02:38:47.939 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.939 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.939 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:47.939 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.096099+0000", 2020-06-12T02:38:47.940 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0460000000000002e-06 2020-06-12T02:38:47.940 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.940 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.940 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:47.941 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.096105+0000", 2020-06-12T02:38:47.941 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6931000000000002e-05 2020-06-12T02:38:47.941 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.941 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.941 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:47.941 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.096152+0000", 2020-06-12T02:38:47.942 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010251600000000001 2020-06-12T02:38:47.942 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.942 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.942 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:47.942 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.096254+0000", 2020-06-12T02:38:47.943 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072763000000000005 2020-06-12T02:38:47.943 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.943 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.943 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:47.944 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.096982+0000", 2020-06-12T02:38:47.944 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012262115000000001 2020-06-12T02:38:47.944 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.944 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.944 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:47.945 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.109244+0000", 2020-06-12T02:38:47.945 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1791000000000003e-05 2020-06-12T02:38:47.945 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.945 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.945 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:47.945 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.109336+0000", 2020-06-12T02:38:47.946 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0002131 2020-06-12T02:38:47.946 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.946 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.946 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:47.946 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.109549+0000", 2020-06-12T02:38:47.947 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013655560000000001 2020-06-12T02:38:47.947 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:47.947 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:47.947 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:47.947 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.948 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.948 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:17977 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:38:47.948 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:41.103849+0000", 2020-06-12T02:38:47.948 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 6.5975256609999997, 2020-06-12T02:38:47.948 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015465273, 2020-06-12T02:38:47.948 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:47.949 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:47.949 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:47.949 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:47.949 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:47.949 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 17977 2020-06-12T02:38:47.950 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.950 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:47.955 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.955 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:47.956 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.103849+0000", 2020-06-12T02:38:47.956 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:47.956 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.956 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.956 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:47.957 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.103849+0000", 2020-06-12T02:38:47.957 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1254e-05 2020-06-12T02:38:47.957 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.957 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.957 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:47.958 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.103870+0000", 2020-06-12T02:38:47.958 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9869999999999999e-06 2020-06-12T02:38:47.958 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.958 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.958 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:47.958 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.103873+0000", 2020-06-12T02:38:47.959 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6159999999999998e-06 2020-06-12T02:38:47.959 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.959 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.959 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:47.959 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.103880+0000", 2020-06-12T02:38:47.960 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0899999999999999e-06 2020-06-12T02:38:47.960 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.960 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.960 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:47.960 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.103882+0000", 2020-06-12T02:38:47.960 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0066572259999999996 2020-06-12T02:38:47.961 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.961 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.961 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:47.961 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.110539+0000", 2020-06-12T02:38:47.961 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5937000000000003e-05 2020-06-12T02:38:47.962 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.962 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.962 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:47.962 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.110595+0000", 2020-06-12T02:38:47.962 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0066983820000000001 2020-06-12T02:38:47.963 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.963 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.963 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:47.963 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.117293+0000", 2020-06-12T02:38:47.963 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103828 2020-06-12T02:38:47.963 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.964 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.964 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:47.964 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.117397+0000", 2020-06-12T02:38:47.964 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0019169530000000001 2020-06-12T02:38:47.964 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.964 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.965 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:47.965 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.119314+0000", 2020-06-12T02:38:47.965 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015465273 2020-06-12T02:38:47.966 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:47.966 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:47.966 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:47.966 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.966 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.967 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:17998 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:38:47.968 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:41.190236+0000", 2020-06-12T02:38:47.968 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 6.5111386060000003, 2020-06-12T02:38:47.968 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013903766, 2020-06-12T02:38:47.968 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:47.968 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:47.969 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:47.969 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:47.969 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:47.969 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 17998 2020-06-12T02:38:47.969 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.970 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:47.970 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.970 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:47.970 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.190236+0000", 2020-06-12T02:38:47.971 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:47.971 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.971 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.971 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:47.971 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.190236+0000", 2020-06-12T02:38:47.972 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.207e-05 2020-06-12T02:38:47.972 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.972 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.972 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:47.972 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.190258+0000", 2020-06-12T02:38:47.972 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9629999999999998e-06 2020-06-12T02:38:47.973 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.973 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.973 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:47.973 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.190261+0000", 2020-06-12T02:38:47.973 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.409e-06 2020-06-12T02:38:47.974 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.974 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.974 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:47.974 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.190267+0000", 2020-06-12T02:38:47.974 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2189999999999998e-06 2020-06-12T02:38:47.974 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.975 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.975 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:47.975 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.190270+0000", 2020-06-12T02:38:47.975 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0089464690000000003 2020-06-12T02:38:47.976 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.976 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.976 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:47.976 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.199216+0000", 2020-06-12T02:38:47.976 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010023500000000001 2020-06-12T02:38:47.977 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.977 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.977 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:47.977 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.199316+0000", 2020-06-12T02:38:47.977 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0038963800000000001 2020-06-12T02:38:47.978 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.978 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.978 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:47.978 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.203213+0000", 2020-06-12T02:38:47.978 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6963000000000003e-05 2020-06-12T02:38:47.978 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.979 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.979 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:47.979 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.203270+0000", 2020-06-12T02:38:47.979 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00087005800000000005 2020-06-12T02:38:47.979 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.980 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.980 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:47.980 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.204140+0000", 2020-06-12T02:38:47.980 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013903766 2020-06-12T02:38:47.980 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:47.980 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:47.981 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:47.981 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.981 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.981 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.7s1 62/37 ECSubWrite(tid=9513, reqid=client.4512.0:18266, at_version=0'0, trim_to=0'0, roll_forward_to=62'1601))", 2020-06-12T02:38:47.981 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:41.904485+0000", 2020-06-12T02:38:47.982 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 5.7968895089999997, 2020-06-12T02:38:47.982 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01412036, 2020-06-12T02:38:47.982 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:47.982 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:47.982 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:47.982 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.983 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:47.983 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904485+0000", 2020-06-12T02:38:47.983 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:47.983 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.983 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.984 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:47.984 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904485+0000", 2020-06-12T02:38:47.984 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.345e-05 2020-06-12T02:38:47.984 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.984 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.984 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:47.985 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904519+0000", 2020-06-12T02:38:47.985 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0869999999999999e-05 2020-06-12T02:38:47.985 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.985 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.985 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:47.986 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904529+0000", 2020-06-12T02:38:47.986 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1248e-05 2020-06-12T02:38:47.986 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.986 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.987 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:47.987 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904541+0000", 2020-06-12T02:38:47.987 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0730999999999999e-05 2020-06-12T02:38:47.987 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.987 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.987 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:47.988 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904551+0000", 2020-06-12T02:38:47.988 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0985999999999997e-05 2020-06-12T02:38:47.988 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.988 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.988 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:47.989 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904592+0000", 2020-06-12T02:38:47.989 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2255999999999999e-05 2020-06-12T02:38:47.989 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.989 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.989 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:47.989 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904605+0000", 2020-06-12T02:38:47.990 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013935858000000001 2020-06-12T02:38:47.990 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.990 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.990 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:47.990 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.918541+0000", 2020-06-12T02:38:47.991 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4961000000000004e-05 2020-06-12T02:38:47.991 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.991 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.991 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:47.991 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.918605+0000", 2020-06-12T02:38:47.991 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01412036 2020-06-12T02:38:47.992 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:47.992 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:47.992 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:47.992 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.992 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.993 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "pg_update_log_missing(3.3s2 epoch 62/53 rep_tid 6734 entries 62'1548 (0'0) error 3:ce45c70a:::smithi10729407-7:head by client.4512.0:18270 0.000000 -2 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 62'1546)", 2020-06-12T02:38:47.993 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:41.904668+0000", 2020-06-12T02:38:47.993 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 5.796706973, 2020-06-12T02:38:47.993 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01430155, 2020-06-12T02:38:47.993 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:47.994 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:47.994 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:47.994 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.994 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:47.994 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904668+0000", 2020-06-12T02:38:47.994 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:47.995 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.995 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.995 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:47.995 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904668+0000", 2020-06-12T02:38:47.995 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1882000000000001e-05 2020-06-12T02:38:47.996 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.996 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.996 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:47.996 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904730+0000", 2020-06-12T02:38:47.997 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0861e-05 2020-06-12T02:38:47.997 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.997 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.997 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:47.997 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904740+0000", 2020-06-12T02:38:47.998 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0406000000000001e-05 2020-06-12T02:38:47.998 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.998 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.998 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:47.998 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904761+0000", 2020-06-12T02:38:47.998 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0339999999999997e-06 2020-06-12T02:38:47.999 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:47.999 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:47.999 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:47.999 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904769+0000", 2020-06-12T02:38:47.999 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3888999999999995e-05 2020-06-12T02:38:48.000 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:48.000 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:48.000 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:48.000 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904863+0000", 2020-06-12T02:38:48.000 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014106478 2020-06-12T02:38:48.001 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:48.001 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:48.001 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:48.001 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.918969+0000", 2020-06-12T02:38:48.001 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01430155 2020-06-12T02:38:48.001 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:48.002 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:48.003 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:48.003 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:48.004 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:48.004 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.fs2 62/37 ECSubWrite(tid=3343, reqid=client.4512.0:18307, at_version=62'1623, trim_to=0'0, roll_forward_to=62'1622))", 2020-06-12T02:38:48.004 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:42.062050+0000", 2020-06-12T02:38:48.004 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 5.639324309, 2020-06-12T02:38:48.004 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019528429999999999, 2020-06-12T02:38:48.005 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:48.005 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:48.005 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:48.005 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:48.005 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:48.006 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062050+0000", 2020-06-12T02:38:48.006 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:48.006 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:48.006 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:48.007 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:48.007 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062050+0000", 2020-06-12T02:38:48.007 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00022828799999999999 2020-06-12T02:38:48.007 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:48.007 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:48.008 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:48.008 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062279+0000", 2020-06-12T02:38:48.008 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2115000000000001e-05 2020-06-12T02:38:48.008 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:48.008 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:48.008 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:48.009 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062301+0000", 2020-06-12T02:38:48.009 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9301e-05 2020-06-12T02:38:48.009 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:48.009 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:48.009 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:48.010 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062320+0000", 2020-06-12T02:38:48.010 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2811e-05 2020-06-12T02:38:48.010 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:48.010 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:48.010 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:48.010 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062333+0000", 2020-06-12T02:38:48.011 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8856e-05 2020-06-12T02:38:48.011 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:48.011 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:48.011 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:48.011 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062392+0000", 2020-06-12T02:38:48.012 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1246999999999999e-05 2020-06-12T02:38:48.012 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:48.012 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:48.012 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:48.012 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062403+0000", 2020-06-12T02:38:48.012 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019095523 2020-06-12T02:38:48.013 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:48.013 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:48.013 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:48.013 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.081498+0000", 2020-06-12T02:38:48.013 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0289000000000003e-05 2020-06-12T02:38:48.014 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:48.014 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:48.014 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:48.014 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.081579+0000", 2020-06-12T02:38:48.014 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019528429999999999 2020-06-12T02:38:48.014 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:48.015 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:48.015 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:48.015 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:48.015 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:48.015 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.fs2 62/37 ECSubWrite(tid=3344, reqid=client.4512.0:18308, at_version=62'1624, trim_to=0'0, roll_forward_to=62'1622))", 2020-06-12T02:38:48.016 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:42.064848+0000", 2020-06-12T02:38:48.016 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 5.6365263240000001, 2020-06-12T02:38:48.016 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01677273, 2020-06-12T02:38:48.016 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:48.016 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:48.017 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:48.017 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:48.017 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:48.017 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.064848+0000", 2020-06-12T02:38:48.017 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:48.018 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:48.018 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:48.018 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:48.018 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.064848+0000", 2020-06-12T02:38:48.018 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000383655 2020-06-12T02:38:48.019 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:48.019 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:48.019 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:48.019 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.065232+0000", 2020-06-12T02:38:48.019 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3065999999999999e-05 2020-06-12T02:38:48.019 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:48.020 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:48.020 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:48.020 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.065255+0000", 2020-06-12T02:38:48.020 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7142999999999998e-05 2020-06-12T02:38:48.020 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:48.021 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:48.021 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:48.021 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.065272+0000", 2020-06-12T02:38:48.021 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7750000000000004e-06 2020-06-12T02:38:48.021 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:48.021 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:48.022 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:48.022 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.065279+0000", 2020-06-12T02:38:48.022 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00091758699999999998 2020-06-12T02:38:48.022 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:48.022 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:48.023 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:48.023 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.066197+0000", 2020-06-12T02:38:48.023 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4158999999999999e-05 2020-06-12T02:38:48.023 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:48.023 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:48.023 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:48.024 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.066211+0000", 2020-06-12T02:38:48.024 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015373370000000001 2020-06-12T02:38:48.024 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:48.024 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:48.025 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:48.025 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.081584+0000", 2020-06-12T02:38:48.025 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6974999999999998e-05 2020-06-12T02:38:48.025 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:48.025 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:48.027 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:48.027 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.081621+0000", 2020-06-12T02:38:48.028 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01677273 2020-06-12T02:38:48.028 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:48.028 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:48.028 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:48.028 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:48.029 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:48.029 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.3s2 62/53 ECSubWrite(tid=6737, reqid=client.4512.0:18319, at_version=62'1551, trim_to=0'0, roll_forward_to=62'1548))", 2020-06-12T02:38:48.029 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:42.089962+0000", 2020-06-12T02:38:48.029 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 5.6114126860000004, 2020-06-12T02:38:48.029 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015432937000000001, 2020-06-12T02:38:48.030 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:48.030 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:48.030 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:48.030 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:48.030 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:48.030 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.089962+0000", 2020-06-12T02:38:48.031 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:48.031 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:48.031 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:48.031 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:48.031 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.089962+0000", 2020-06-12T02:38:48.032 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00068051099999999996 2020-06-12T02:38:48.032 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:48.032 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:48.032 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:48.032 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.090642+0000", 2020-06-12T02:38:48.032 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8766999999999998e-05 2020-06-12T02:38:48.033 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:48.033 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:48.033 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:48.033 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.090661+0000", 2020-06-12T02:38:48.033 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6461e-05 2020-06-12T02:38:48.034 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:48.034 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:48.034 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:48.034 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.090678+0000", 2020-06-12T02:38:48.034 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0580000000000003e-06 2020-06-12T02:38:48.034 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:48.035 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:48.035 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:48.035 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.090684+0000", 2020-06-12T02:38:48.035 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000104752 2020-06-12T02:38:48.035 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:48.036 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:48.036 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:48.036 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.090789+0000", 2020-06-12T02:38:48.036 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0767e-05 2020-06-12T02:38:48.036 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:48.036 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:48.037 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:48.037 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.090799+0000", 2020-06-12T02:38:48.037 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014539606 2020-06-12T02:38:48.037 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:48.038 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:48.038 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:48.038 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.105339+0000", 2020-06-12T02:38:48.038 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6014999999999998e-05 2020-06-12T02:38:48.038 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:48.039 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:48.039 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:48.039 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.105395+0000", 2020-06-12T02:38:48.039 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015432937000000001 2020-06-12T02:38:48.039 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:48.039 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:48.040 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:48.040 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:48.040 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:48.040 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:48.041 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-06-12T02:38:48.043 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-06-12T02:38:48.043 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:48.039+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:48.142 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:38:48.140+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:38:48.203 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:48.203 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:38:48.203 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2020-06-12T02:38:48.204 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:48.212 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-06-12T02:38:48.218 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:38:48.244 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:38:48.245 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:48.241+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:48.260 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:38:48.280 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:38:48.301 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:38:48.324 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:38:48.333 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:48.334 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:48.345 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:38:48.346 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-06-12T02:38:48.369 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2020-06-12T02:38:48.369 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-06-12T02:38:48.369 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-06-12T02:38:48.370 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-06-12T02:38:48.370 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-06-12T02:38:48.370 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-06-12T02:38:48.370 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-06-12T02:38:48.371 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-06-12T02:38:48.371 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-06-12T02:38:48.374 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:48.364+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:48.450 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:38:48.447+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:38:48.486 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:48.487 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:48.496 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-06-12T02:38:48.516 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2020-06-12T02:38:48.516 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.0 to 1 (10000) 2020-06-12T02:38:48.531 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd reweight 1 1 2020-06-12T02:38:48.566 INFO:tasks.rados.rados.0.smithi107.stdout:9181: done (0 left) 2020-06-12T02:38:48.567 INFO:tasks.rados.rados.0.smithi107.stdout:9182: delete oid 13 current snap is 0 2020-06-12T02:38:48.567 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:48.564+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:48.569 INFO:tasks.rados.rados.0.smithi107.stdout:9182: done (0 left) 2020-06-12T02:38:48.569 INFO:tasks.rados.rados.0.smithi107.stdout:9183: delete oid 19 current snap is 0 2020-06-12T02:38:48.576 INFO:tasks.rados.rados.0.smithi107.stdout:9183: done (0 left) 2020-06-12T02:38:48.576 INFO:tasks.rados.rados.0.smithi107.stdout:9184: delete oid 15 current snap is 0 2020-06-12T02:38:48.576 INFO:tasks.rados.rados.0.smithi107.stdout:9184: done (0 left) 2020-06-12T02:38:48.576 INFO:tasks.rados.rados.0.smithi107.stdout:9185: delete oid 13 current snap is 0 2020-06-12T02:38:48.580 INFO:tasks.rados.rados.0.smithi107.stdout:9185: done (0 left) 2020-06-12T02:38:48.581 INFO:tasks.rados.rados.0.smithi107.stdout:9186: delete oid 20 current snap is 0 2020-06-12T02:38:48.583 INFO:tasks.rados.rados.0.smithi107.stdout:9186: done (0 left) 2020-06-12T02:38:48.583 INFO:tasks.rados.rados.0.smithi107.stdout:9187: delete oid 17 current snap is 0 2020-06-12T02:38:48.590 INFO:tasks.rados.rados.0.smithi107.stdout:9187: done (0 left) 2020-06-12T02:38:48.590 INFO:tasks.rados.rados.0.smithi107.stdout:9188: delete oid 7 current snap is 0 2020-06-12T02:38:48.590 INFO:tasks.rados.rados.0.smithi107.stdout:9188: done (0 left) 2020-06-12T02:38:48.591 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:38:48.591 INFO:tasks.rados.rados.0.smithi107.stdout:9189: seq_num 9188 ranges {0=286720} 2020-06-12T02:38:48.591 INFO:tasks.rados.rados.0.smithi107.stdout:9189: writing smithi10729407-10 from 0 to 286720 tid 1 2020-06-12T02:38:48.592 INFO:tasks.rados.rados.0.smithi107.stdout:9190: delete oid 19 current snap is 0 2020-06-12T02:38:48.594 INFO:tasks.rados.rados.0.smithi107.stdout:9190: done (1 left) 2020-06-12T02:38:48.594 INFO:tasks.rados.rados.0.smithi107.stdout:9191: delete oid 2 current snap is 0 2020-06-12T02:38:48.595 INFO:tasks.rados.rados.0.smithi107.stdout:9189: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:48.596 INFO:tasks.rados.rados.0.smithi107.stdout:9191: done (1 left) 2020-06-12T02:38:48.596 INFO:tasks.rados.rados.0.smithi107.stdout:9192: delete oid 5 current snap is 0 2020-06-12T02:38:48.599 INFO:tasks.rados.rados.0.smithi107.stdout:9189: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:48.601 INFO:tasks.rados.rados.0.smithi107.stdout:9192: done (1 left) 2020-06-12T02:38:48.601 INFO:tasks.rados.rados.0.smithi107.stdout:9193: delete oid 15 current snap is 0 2020-06-12T02:38:48.602 INFO:tasks.rados.rados.0.smithi107.stdout:9189: finishing write tid 4 to smithi10729407-10 2020-06-12T02:38:48.603 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 655 (ObjNum 9188 snap 0 seq_num 9188) dirty exists 2020-06-12T02:38:48.603 INFO:tasks.rados.rados.0.smithi107.stdout:9189: left oid 10 (ObjNum 9188 snap 0 seq_num 9188) 2020-06-12T02:38:48.603 INFO:tasks.rados.rados.0.smithi107.stdout:9189: done (1 left) 2020-06-12T02:38:48.604 INFO:tasks.rados.rados.0.smithi107.stdout:9193: done (0 left) 2020-06-12T02:38:48.604 INFO:tasks.rados.rados.0.smithi107.stdout:9194: delete oid 16 current snap is 0 2020-06-12T02:38:48.606 INFO:tasks.rados.rados.0.smithi107.stdout:9194: done (0 left) 2020-06-12T02:38:48.606 INFO:tasks.rados.rados.0.smithi107.stdout:9195: delete oid 14 current snap is 0 2020-06-12T02:38:48.609 INFO:tasks.rados.rados.0.smithi107.stdout:9195: done (0 left) 2020-06-12T02:38:48.609 INFO:tasks.rados.rados.0.smithi107.stdout:9196: delete oid 13 current snap is 0 2020-06-12T02:38:48.611 INFO:tasks.rados.rados.0.smithi107.stdout:9196: done (0 left) 2020-06-12T02:38:48.611 INFO:tasks.rados.rados.0.smithi107.stdout:9197: delete oid 20 current snap is 0 2020-06-12T02:38:48.613 INFO:tasks.rados.rados.0.smithi107.stdout:9197: done (0 left) 2020-06-12T02:38:48.613 INFO:tasks.rados.rados.0.smithi107.stdout:9198: delete oid 10 current snap is 0 2020-06-12T02:38:48.615 INFO:tasks.rados.rados.0.smithi107.stdout:9198: done (0 left) 2020-06-12T02:38:48.615 INFO:tasks.rados.rados.0.smithi107.stdout:9199: delete oid 17 current snap is 0 2020-06-12T02:38:48.618 INFO:tasks.rados.rados.0.smithi107.stdout:9199: done (0 left) 2020-06-12T02:38:48.618 INFO:tasks.rados.rados.0.smithi107.stdout:9200: delete oid 5 current snap is 0 2020-06-12T02:38:48.620 INFO:tasks.rados.rados.0.smithi107.stdout:9200: done (0 left) 2020-06-12T02:38:48.620 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:48.620 INFO:tasks.rados.rados.0.smithi107.stdout:9201: seq_num 9200 ranges {0=491520} 2020-06-12T02:38:48.622 INFO:tasks.rados.rados.0.smithi107.stdout:9201: writing smithi10729407-15 from 0 to 491520 tid 1 2020-06-12T02:38:48.623 INFO:tasks.rados.rados.0.smithi107.stdout:9202: delete oid 14 current snap is 0 2020-06-12T02:38:48.625 INFO:tasks.rados.rados.0.smithi107.stdout:9202: done (1 left) 2020-06-12T02:38:48.625 INFO:tasks.rados.rados.0.smithi107.stdout:9203: delete oid 14 current snap is 0 2020-06-12T02:38:48.627 INFO:tasks.rados.rados.0.smithi107.stdout:9203: done (1 left) 2020-06-12T02:38:48.627 INFO:tasks.rados.rados.0.smithi107.stdout:9204: delete oid 3 current snap is 0 2020-06-12T02:38:48.629 INFO:tasks.rados.rados.0.smithi107.stdout:9201: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:48.629 INFO:tasks.rados.rados.0.smithi107.stdout:9204: done (1 left) 2020-06-12T02:38:48.629 INFO:tasks.rados.rados.0.smithi107.stdout:9205: delete oid 18 current snap is 0 2020-06-12T02:38:48.630 INFO:tasks.rados.rados.0.smithi107.stdout:9201: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:48.631 INFO:tasks.rados.rados.0.smithi107.stdout:9205: done (1 left) 2020-06-12T02:38:48.632 INFO:tasks.rados.rados.0.smithi107.stdout:9206: delete oid 5 current snap is 0 2020-06-12T02:38:48.633 INFO:tasks.rados.rados.0.smithi107.stdout:9201: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:48.633 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1709 (ObjNum 9200 snap 0 seq_num 9200) dirty exists 2020-06-12T02:38:48.633 INFO:tasks.rados.rados.0.smithi107.stdout:9201: left oid 15 (ObjNum 9200 snap 0 seq_num 9200) 2020-06-12T02:38:48.635 INFO:tasks.rados.rados.0.smithi107.stdout:9201: done (1 left) 2020-06-12T02:38:48.635 INFO:tasks.rados.rados.0.smithi107.stdout:9206: done (0 left) 2020-06-12T02:38:48.635 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:48.636 INFO:tasks.rados.rados.0.smithi107.stdout:9207: seq_num 9206 ranges {0=589824} 2020-06-12T02:38:48.639 INFO:tasks.rados.rados.0.smithi107.stdout:9207: writing smithi10729407-18 from 0 to 589824 tid 1 2020-06-12T02:38:48.640 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:48.640 INFO:tasks.rados.rados.0.smithi107.stdout:9208: seq_num 9207 ranges {0=606208} 2020-06-12T02:38:48.644 INFO:tasks.rados.rados.0.smithi107.stdout:9208: writing smithi10729407-19 from 0 to 606208 tid 1 2020-06-12T02:38:48.644 INFO:tasks.rados.rados.0.smithi107.stdout:9209: delete oid 2 current snap is 0 2020-06-12T02:38:48.645 INFO:tasks.rados.rados.0.smithi107.stdout:9207: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:48.648 INFO:tasks.rados.rados.0.smithi107.stdout:9209: done (2 left) 2020-06-12T02:38:48.648 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:38:48.648 INFO:tasks.rados.rados.0.smithi107.stdout:9210: seq_num 9209 ranges {32768=638976} 2020-06-12T02:38:48.651 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:48.651 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:48.654 INFO:tasks.rados.rados.0.smithi107.stdout:9210: writing smithi10729407-1 from 32768 to 671744 tid 1 2020-06-12T02:38:48.654 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:38:48.654 INFO:tasks.rados.rados.0.smithi107.stdout:9211: seq_num 9210 ranges {0=655360} 2020-06-12T02:38:48.659 INFO:tasks.rados.rados.0.smithi107.stdout:9211: writing smithi10729407-10 from 0 to 655360 tid 1 2020-06-12T02:38:48.659 INFO:tasks.rados.rados.0.smithi107.stdout:9212: delete oid 5 current snap is 0 2020-06-12T02:38:48.659 INFO:tasks.rados.rados.0.smithi107.stdout:9207: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:48.659 INFO:tasks.rados.rados.0.smithi107.stdout:9207: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:48.659 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1709 (ObjNum 9206 snap 0 seq_num 9206) dirty exists 2020-06-12T02:38:48.660 INFO:tasks.rados.rados.0.smithi107.stdout:9207: left oid 18 (ObjNum 9206 snap 0 seq_num 9206) 2020-06-12T02:38:48.660 INFO:tasks.rados.rados.0.smithi107.stdout:9208: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:48.660 INFO:tasks.rados.rados.0.smithi107.stdout:9208: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:48.660 INFO:tasks.rados.rados.0.smithi107.stdout:9208: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:48.660 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1698 (ObjNum 9207 snap 0 seq_num 9207) dirty exists 2020-06-12T02:38:48.661 INFO:tasks.rados.rados.0.smithi107.stdout:9208: left oid 19 (ObjNum 9207 snap 0 seq_num 9207) 2020-06-12T02:38:48.662 INFO:tasks.rados.rados.0.smithi107.stdout:9207: done (4 left) 2020-06-12T02:38:48.663 INFO:tasks.rados.rados.0.smithi107.stdout:9208: done (3 left) 2020-06-12T02:38:48.663 INFO:tasks.rados.rados.0.smithi107.stdout:9212: done (2 left) 2020-06-12T02:38:48.663 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:48.663 INFO:tasks.rados.rados.0.smithi107.stdout:9213: seq_num 9212 ranges {2105344=688128} 2020-06-12T02:38:48.663 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:48.659+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:48.664 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-06-12T02:38:48.678 INFO:tasks.rados.rados.0.smithi107.stdout:9213: writing smithi10729407-11 from 2105344 to 2793472 tid 1 2020-06-12T02:38:48.678 INFO:tasks.rados.rados.0.smithi107.stdout:9214: delete oid 7 current snap is 0 2020-06-12T02:38:48.678 INFO:tasks.rados.rados.0.smithi107.stdout:9210: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:48.678 INFO:tasks.rados.rados.0.smithi107.stdout:9210: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:48.678 INFO:tasks.rados.rados.0.smithi107.stdout:9211: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:48.679 INFO:tasks.rados.rados.0.smithi107.stdout:9210: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:48.679 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1674 (ObjNum 9209 snap 0 seq_num 9209) dirty exists 2020-06-12T02:38:48.679 INFO:tasks.rados.rados.0.smithi107.stdout:9210: left oid 1 (ObjNum 9209 snap 0 seq_num 9209) 2020-06-12T02:38:48.679 INFO:tasks.rados.rados.0.smithi107.stdout:9211: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:48.679 INFO:tasks.rados.rados.0.smithi107.stdout:9211: finishing write tid 4 to smithi10729407-10 2020-06-12T02:38:48.680 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 658 (ObjNum 9210 snap 0 seq_num 9210) dirty exists 2020-06-12T02:38:48.680 INFO:tasks.rados.rados.0.smithi107.stdout:9211: left oid 10 (ObjNum 9210 snap 0 seq_num 9210) 2020-06-12T02:38:48.680 INFO:tasks.rados.rados.0.smithi107.stdout:9210: done (3 left) 2020-06-12T02:38:48.685 INFO:tasks.rados.rados.0.smithi107.stdout:9211: done (2 left) 2020-06-12T02:38:48.685 INFO:tasks.rados.rados.0.smithi107.stdout:9214: done (1 left) 2020-06-12T02:38:48.686 INFO:tasks.rados.rados.0.smithi107.stdout:9215: delete oid 18 current snap is 0 2020-06-12T02:38:48.687 INFO:tasks.rados.rados.0.smithi107.stdout:9215: done (1 left) 2020-06-12T02:38:48.687 INFO:tasks.rados.rados.0.smithi107.stdout:9216: delete oid 1 current snap is 0 2020-06-12T02:38:48.687 INFO:tasks.rados.rados.0.smithi107.stdout:9213: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:48.687 INFO:tasks.rados.rados.0.smithi107.stdout:9213: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:48.688 INFO:tasks.rados.rados.0.smithi107.stdout:9216: done (1 left) 2020-06-12T02:38:48.688 INFO:tasks.rados.rados.0.smithi107.stdout:9217: delete oid 5 current snap is 0 2020-06-12T02:38:48.688 INFO:tasks.rados.rados.0.smithi107.stdout:9213: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:48.688 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1592 (ObjNum 9212 snap 0 seq_num 9212) dirty exists 2020-06-12T02:38:48.688 INFO:tasks.rados.rados.0.smithi107.stdout:9213: left oid 11 (ObjNum 9212 snap 0 seq_num 9212) 2020-06-12T02:38:48.689 INFO:tasks.rados.rados.0.smithi107.stdout:9213: done (1 left) 2020-06-12T02:38:48.689 INFO:tasks.rados.rados.0.smithi107.stdout:9217: done (0 left) 2020-06-12T02:38:48.690 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:38:48.690 INFO:tasks.rados.rados.0.smithi107.stdout:9218: seq_num 9217 ranges {655360=770048} 2020-06-12T02:38:48.711 INFO:tasks.rados.rados.0.smithi107.stdout:9218: writing smithi10729407-10 from 655360 to 1425408 tid 1 2020-06-12T02:38:48.711 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:38:48.711 INFO:tasks.rados.rados.0.smithi107.stdout:9219: seq_num 9218 ranges {0=786432} 2020-06-12T02:38:48.712 INFO:tasks.rados.rados.0.smithi107.stdout:9219: writing smithi10729407-2 from 0 to 786432 tid 1 2020-06-12T02:38:48.712 INFO:tasks.rados.rados.0.smithi107.stdout:9220: delete oid 9 current snap is 0 2020-06-12T02:38:48.712 INFO:tasks.rados.rados.0.smithi107.stdout:9218: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:48.712 INFO:tasks.rados.rados.0.smithi107.stdout:9218: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:48.712 INFO:tasks.rados.rados.0.smithi107.stdout:9220: done (2 left) 2020-06-12T02:38:48.713 INFO:tasks.rados.rados.0.smithi107.stdout:9221: delete oid 9 current snap is 0 2020-06-12T02:38:48.713 INFO:tasks.rados.rados.0.smithi107.stdout:9219: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:48.713 INFO:tasks.rados.rados.0.smithi107.stdout:9218: finishing write tid 4 to smithi10729407-10 2020-06-12T02:38:48.714 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 660 (ObjNum 9217 snap 0 seq_num 9217) dirty exists 2020-06-12T02:38:48.714 INFO:tasks.rados.rados.0.smithi107.stdout:9218: left oid 10 (ObjNum 9217 snap 0 seq_num 9217) 2020-06-12T02:38:48.714 INFO:tasks.rados.rados.0.smithi107.stdout:9219: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:48.722 INFO:tasks.rados.rados.0.smithi107.stdout:9218: done (2 left) 2020-06-12T02:38:48.722 INFO:tasks.rados.rados.0.smithi107.stdout:9221: done (1 left) 2020-06-12T02:38:48.722 INFO:tasks.rados.rados.0.smithi107.stdout:9222: delete oid 7 current snap is 0 2020-06-12T02:38:48.723 INFO:tasks.rados.rados.0.smithi107.stdout:9219: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:48.723 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 857 (ObjNum 9218 snap 0 seq_num 9218) dirty exists 2020-06-12T02:38:48.723 INFO:tasks.rados.rados.0.smithi107.stdout:9219: left oid 2 (ObjNum 9218 snap 0 seq_num 9218) 2020-06-12T02:38:48.723 INFO:tasks.rados.rados.0.smithi107.stdout:9219: done (1 left) 2020-06-12T02:38:48.723 INFO:tasks.rados.rados.0.smithi107.stdout:9222: done (0 left) 2020-06-12T02:38:48.724 INFO:tasks.rados.rados.0.smithi107.stdout:9223: delete oid 4 current snap is 0 2020-06-12T02:38:48.724 INFO:tasks.rados.rados.0.smithi107.stdout:9223: done (0 left) 2020-06-12T02:38:48.724 INFO:tasks.rados.rados.0.smithi107.stdout:9224: delete oid 8 current snap is 0 2020-06-12T02:38:48.724 INFO:tasks.rados.rados.0.smithi107.stdout:9224: done (0 left) 2020-06-12T02:38:48.725 INFO:tasks.rados.rados.0.smithi107.stdout:9225: delete oid 20 current snap is 0 2020-06-12T02:38:48.726 INFO:tasks.rados.rados.0.smithi107.stdout:9225: done (0 left) 2020-06-12T02:38:48.727 INFO:tasks.rados.rados.0.smithi107.stdout:9226: delete oid 4 current snap is 0 2020-06-12T02:38:48.729 INFO:tasks.rados.rados.0.smithi107.stdout:9226: done (0 left) 2020-06-12T02:38:48.729 INFO:tasks.rados.rados.0.smithi107.stdout:9227: delete oid 2 current snap is 0 2020-06-12T02:38:48.731 INFO:tasks.rados.rados.0.smithi107.stdout:9227: done (0 left) 2020-06-12T02:38:48.731 INFO:tasks.rados.rados.0.smithi107.stdout:9228: delete oid 10 current snap is 0 2020-06-12T02:38:48.734 INFO:tasks.rados.rados.0.smithi107.stdout:9228: done (0 left) 2020-06-12T02:38:48.734 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:48.735 INFO:tasks.rados.rados.0.smithi107.stdout:9229: seq_num 9228 ranges {0=557056,557056=401408} 2020-06-12T02:38:48.737 INFO:tasks.rados.rados.0.smithi107.stdout:9229: writing smithi10729407-9 from 0 to 557056 tid 1 2020-06-12T02:38:48.740 INFO:tasks.rados.rados.0.smithi107.stdout:9229: writing smithi10729407-9 from 557056 to 958464 tid 2 2020-06-12T02:38:48.740 INFO:tasks.rados.rados.0.smithi107.stdout:9230: delete oid 19 current snap is 0 2020-06-12T02:38:48.743 INFO:tasks.rados.rados.0.smithi107.stdout:9230: done (1 left) 2020-06-12T02:38:48.743 INFO:tasks.rados.rados.0.smithi107.stdout:9231: delete oid 18 current snap is 0 2020-06-12T02:38:48.745 INFO:tasks.rados.rados.0.smithi107.stdout:9229: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:48.751 INFO:tasks.rados.rados.0.smithi107.stdout:9231: done (1 left) 2020-06-12T02:38:48.752 INFO:tasks.rados.rados.0.smithi107.stdout:9232: delete oid 13 current snap is 0 2020-06-12T02:38:48.753 INFO:tasks.rados.rados.0.smithi107.stdout:9229: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:48.753 INFO:tasks.rados.rados.0.smithi107.stdout:9229: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:48.754 INFO:tasks.rados.rados.0.smithi107.stdout:9232: done (1 left) 2020-06-12T02:38:48.754 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:48.754 INFO:tasks.rados.rados.0.smithi107.stdout:9233: seq_num 9232 ranges {0=622592,622592=401408} 2020-06-12T02:38:48.758 INFO:tasks.rados.rados.0.smithi107.stdout:9233: writing smithi10729407-19 from 0 to 622592 tid 1 2020-06-12T02:38:48.762 INFO:tasks.rados.rados.0.smithi107.stdout:9233: writing smithi10729407-19 from 622592 to 1024000 tid 2 2020-06-12T02:38:48.762 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:48.762 INFO:tasks.rados.rados.0.smithi107.stdout:9234: seq_num 9233 ranges {0=638976,638976=401408} 2020-06-12T02:38:48.764 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:48.762+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:48.766 INFO:tasks.rados.rados.0.smithi107.stdout:9234: writing smithi10729407-18 from 0 to 638976 tid 1 2020-06-12T02:38:48.769 INFO:tasks.rados.rados.0.smithi107.stdout:9234: writing smithi10729407-18 from 638976 to 1040384 tid 2 2020-06-12T02:38:48.770 INFO:tasks.rados.rados.0.smithi107.stdout:9235: delete oid 11 current snap is 0 2020-06-12T02:38:48.771 INFO:tasks.rados.rados.0.smithi107.stdout:9229: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:48.771 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1710 (ObjNum 9228 snap 0 seq_num 9228) dirty exists 2020-06-12T02:38:48.772 INFO:tasks.rados.rados.0.smithi107.stdout:9229: left oid 9 (ObjNum 9228 snap 0 seq_num 9228) 2020-06-12T02:38:48.772 INFO:tasks.rados.rados.0.smithi107.stdout:9233: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:48.772 INFO:tasks.rados.rados.0.smithi107.stdout:9233: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:48.773 INFO:tasks.rados.rados.0.smithi107.stdout:9233: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:48.780 INFO:tasks.rados.rados.0.smithi107.stdout:9234: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:48.780 INFO:tasks.rados.rados.0.smithi107.stdout:9229: done (3 left) 2020-06-12T02:38:48.780 INFO:tasks.rados.rados.0.smithi107.stdout:9235: done (2 left) 2020-06-12T02:38:48.781 INFO:tasks.rados.rados.0.smithi107.stdout:9236: delete oid 16 current snap is 0 2020-06-12T02:38:48.781 INFO:tasks.rados.rados.0.smithi107.stdout:9233: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:48.782 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1702 (ObjNum 9232 snap 0 seq_num 9232) dirty exists 2020-06-12T02:38:48.782 INFO:tasks.rados.rados.0.smithi107.stdout:9233: left oid 19 (ObjNum 9232 snap 0 seq_num 9232) 2020-06-12T02:38:48.782 INFO:tasks.rados.rados.0.smithi107.stdout:9233: done (2 left) 2020-06-12T02:38:48.782 INFO:tasks.rados.rados.0.smithi107.stdout:9236: done (1 left) 2020-06-12T02:38:48.782 INFO:tasks.rados.rados.0.smithi107.stdout:9237: delete oid 1 current snap is 0 2020-06-12T02:38:48.783 INFO:tasks.rados.rados.0.smithi107.stdout:9234: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:48.783 INFO:tasks.rados.rados.0.smithi107.stdout:9234: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:48.784 INFO:tasks.rados.rados.0.smithi107.stdout:9237: done (1 left) 2020-06-12T02:38:48.784 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:48.784 INFO:tasks.rados.rados.0.smithi107.stdout:9238: seq_num 9237 ranges {0=704512,704512=401408} 2020-06-12T02:38:48.787 INFO:tasks.rados.rados.0.smithi107.stdout:9238: writing smithi10729407-11 from 0 to 704512 tid 1 2020-06-12T02:38:48.789 INFO:tasks.rados.rados.0.smithi107.stdout:9238: writing smithi10729407-11 from 704512 to 1105920 tid 2 2020-06-12T02:38:48.789 INFO:tasks.rados.rados.0.smithi107.stdout:9239: delete oid 19 current snap is 0 2020-06-12T02:38:48.790 INFO:tasks.rados.rados.0.smithi107.stdout:9234: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:48.790 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1714 (ObjNum 9233 snap 0 seq_num 9233) dirty exists 2020-06-12T02:38:48.790 INFO:tasks.rados.rados.0.smithi107.stdout:9234: left oid 18 (ObjNum 9233 snap 0 seq_num 9233) 2020-06-12T02:38:48.792 INFO:tasks.rados.rados.0.smithi107.stdout:9234: done (2 left) 2020-06-12T02:38:48.792 INFO:tasks.rados.rados.0.smithi107.stdout:9239: done (1 left) 2020-06-12T02:38:48.792 INFO:tasks.rados.rados.0.smithi107.stdout:9240: delete oid 17 current snap is 0 2020-06-12T02:38:48.795 INFO:tasks.rados.rados.0.smithi107.stdout:9238: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:48.795 INFO:tasks.rados.rados.0.smithi107.stdout:9240: done (1 left) 2020-06-12T02:38:48.796 INFO:tasks.rados.rados.0.smithi107.stdout:9241: delete oid 18 current snap is 0 2020-06-12T02:38:48.797 INFO:tasks.rados.rados.0.smithi107.stdout:9238: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:48.798 INFO:tasks.rados.rados.0.smithi107.stdout:9238: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:48.798 INFO:tasks.rados.rados.0.smithi107.stdout:9241: done (1 left) 2020-06-12T02:38:48.798 INFO:tasks.rados.rados.0.smithi107.stdout:9242: delete oid 7 current snap is 0 2020-06-12T02:38:48.802 INFO:tasks.rados.rados.0.smithi107.stdout:9238: finishing write tid 5 to smithi10729407-11 2020-06-12T02:38:48.803 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1596 (ObjNum 9237 snap 0 seq_num 9237) dirty exists 2020-06-12T02:38:48.803 INFO:tasks.rados.rados.0.smithi107.stdout:9238: left oid 11 (ObjNum 9237 snap 0 seq_num 9237) 2020-06-12T02:38:48.803 INFO:tasks.rados.rados.0.smithi107.stdout:9238: done (1 left) 2020-06-12T02:38:48.803 INFO:tasks.rados.rados.0.smithi107.stdout:9242: done (0 left) 2020-06-12T02:38:48.803 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:48.804 INFO:tasks.rados.rados.0.smithi107.stdout:9243: seq_num 9242 ranges {1105920=794624,1900544=401408} 2020-06-12T02:38:48.810 INFO:tasks.rados.rados.0.smithi107.stdout:9243: writing smithi10729407-11 from 1105920 to 1900544 tid 1 2020-06-12T02:38:48.813 INFO:tasks.rados.rados.0.smithi107.stdout:9243: writing smithi10729407-11 from 1900544 to 2301952 tid 2 2020-06-12T02:38:48.813 INFO:tasks.rados.rados.0.smithi107.stdout:9244: delete oid 1 current snap is 0 2020-06-12T02:38:48.820 INFO:tasks.rados.rados.0.smithi107.stdout:9244: done (1 left) 2020-06-12T02:38:48.820 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:48.821 INFO:tasks.rados.rados.0.smithi107.stdout:9245: seq_num 9244 ranges {958464=425984,1384448=729088,2113536=73728} 2020-06-12T02:38:48.828 INFO:tasks.rados.rados.0.smithi107.stdout:9245: writing smithi10729407-9 from 958464 to 1384448 tid 1 2020-06-12T02:38:48.832 INFO:tasks.rados.rados.0.smithi107.stdout:9245: writing smithi10729407-9 from 1384448 to 2113536 tid 2 2020-06-12T02:38:48.833 INFO:tasks.rados.rados.0.smithi107.stdout:9245: writing smithi10729407-9 from 2113536 to 2187264 tid 3 2020-06-12T02:38:48.833 INFO:tasks.rados.rados.0.smithi107.stdout:9243: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:48.833 INFO:tasks.rados.rados.0.smithi107.stdout:9243: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:48.833 INFO:tasks.rados.rados.0.smithi107.stdout:9243: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:48.833 INFO:tasks.rados.rados.0.smithi107.stdout:9243: finishing write tid 5 to smithi10729407-11 2020-06-12T02:38:48.834 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1599 (ObjNum 9242 snap 0 seq_num 9242) dirty exists 2020-06-12T02:38:48.834 INFO:tasks.rados.rados.0.smithi107.stdout:9243: left oid 11 (ObjNum 9242 snap 0 seq_num 9242) 2020-06-12T02:38:48.834 INFO:tasks.rados.rados.0.smithi107.stdout:9243: done (1 left) 2020-06-12T02:38:48.834 INFO:tasks.rados.rados.0.smithi107.stdout:9246: delete oid 7 current snap is 0 2020-06-12T02:38:48.835 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:48.835 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:48.836 INFO:tasks.rados.rados.0.smithi107.stdout:9246: done (1 left) 2020-06-12T02:38:48.836 INFO:tasks.rados.rados.0.smithi107.stdout:9247: delete oid 6 current snap is 0 2020-06-12T02:38:48.837 INFO:tasks.rados.rados.0.smithi107.stdout:9245: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:48.840 INFO:tasks.rados.rados.0.smithi107.stdout:9245: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:48.840 INFO:tasks.rados.rados.0.smithi107.stdout:9247: done (1 left) 2020-06-12T02:38:48.840 INFO:tasks.rados.rados.0.smithi107.stdout:9248: delete oid 18 current snap is 0 2020-06-12T02:38:48.844 INFO:tasks.rados.rados.0.smithi107.stdout:9245: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:48.844 INFO:tasks.rados.rados.0.smithi107.stdout:9248: done (1 left) 2020-06-12T02:38:48.844 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:48.845 INFO:tasks.rados.rados.0.smithi107.stdout:9249: seq_num 9248 ranges {0=491520,491520=720896,1212416=81920} 2020-06-12T02:38:48.847 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-06-12T02:38:48.848 INFO:tasks.rados.rados.0.smithi107.stdout:9249: writing smithi10729407-8 from 0 to 491520 tid 1 2020-06-12T02:38:48.851 INFO:tasks.rados.rados.0.smithi107.stdout:9249: writing smithi10729407-8 from 491520 to 1212416 tid 2 2020-06-12T02:38:48.852 INFO:tasks.rados.rados.0.smithi107.stdout:9249: writing smithi10729407-8 from 1212416 to 1294336 tid 3 2020-06-12T02:38:48.852 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:48.852 INFO:tasks.rados.rados.0.smithi107.stdout:9250: seq_num 9249 ranges {0=507904,507904=606208,1114112=196608} 2020-06-12T02:38:48.854 INFO:tasks.rados.rados.0.smithi107.stdout:9250: writing smithi10729407-3 from 0 to 507904 tid 1 2020-06-12T02:38:48.858 INFO:tasks.rados.rados.0.smithi107.stdout:9250: writing smithi10729407-3 from 507904 to 1114112 tid 2 2020-06-12T02:38:48.868 INFO:tasks.rados.rados.0.smithi107.stdout:9250: writing smithi10729407-3 from 1114112 to 1310720 tid 3 2020-06-12T02:38:48.869 INFO:tasks.rados.rados.0.smithi107.stdout:9245: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:48.870 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:48.870 INFO:tasks.rados.rados.0.smithi107.stdout:9251: seq_num 9250 ranges {0=524288,524288=483328,1007616=319488} 2020-06-12T02:38:48.870 INFO:tasks.rados.rados.0.smithi107.stdout:9251: writing smithi10729407-18 from 0 to 524288 tid 1 2020-06-12T02:38:48.870 INFO:tasks.rados.rados.0.smithi107.stdout:9251: writing smithi10729407-18 from 524288 to 1007616 tid 2 2020-06-12T02:38:48.871 INFO:tasks.rados.rados.0.smithi107.stdout:9251: writing smithi10729407-18 from 1007616 to 1327104 tid 3 2020-06-12T02:38:48.872 INFO:tasks.rados.rados.0.smithi107.stdout:9252: delete oid 10 current snap is 0 2020-06-12T02:38:48.872 INFO:tasks.rados.rados.0.smithi107.stdout:9245: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:48.872 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1714 (ObjNum 9244 snap 0 seq_num 9244) dirty exists 2020-06-12T02:38:48.872 INFO:tasks.rados.rados.0.smithi107.stdout:9245: left oid 9 (ObjNum 9244 snap 0 seq_num 9244) 2020-06-12T02:38:48.872 INFO:tasks.rados.rados.0.smithi107.stdout:9249: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:48.873 INFO:tasks.rados.rados.0.smithi107.stdout:9250: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:48.873 INFO:tasks.rados.rados.0.smithi107.stdout:9249: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:48.873 INFO:tasks.rados.rados.0.smithi107.stdout:9249: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:48.873 INFO:tasks.rados.rados.0.smithi107.stdout:9249: finishing write tid 5 to smithi10729407-8 2020-06-12T02:38:48.873 INFO:tasks.rados.rados.0.smithi107.stdout:9249: finishing write tid 6 to smithi10729407-8 2020-06-12T02:38:48.874 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1714 (ObjNum 9248 snap 0 seq_num 9248) dirty exists 2020-06-12T02:38:48.874 INFO:tasks.rados.rados.0.smithi107.stdout:9249: left oid 8 (ObjNum 9248 snap 0 seq_num 9248) 2020-06-12T02:38:48.875 INFO:tasks.rados.rados.0.smithi107.stdout:9245: done (4 left) 2020-06-12T02:38:48.875 INFO:tasks.rados.rados.0.smithi107.stdout:9249: done (3 left) 2020-06-12T02:38:48.875 INFO:tasks.rados.rados.0.smithi107.stdout:9252: done (2 left) 2020-06-12T02:38:48.875 INFO:tasks.rados.rados.0.smithi107.stdout:9253: delete oid 16 current snap is 0 2020-06-12T02:38:48.876 INFO:tasks.rados.rados.0.smithi107.stdout:9251: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:48.876 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:48.869+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:48.877 INFO:tasks.rados.rados.0.smithi107.stdout:9250: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:48.877 INFO:tasks.rados.rados.0.smithi107.stdout:9250: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:48.883 INFO:tasks.rados.rados.0.smithi107.stdout:9250: finishing write tid 5 to smithi10729407-3 2020-06-12T02:38:48.883 INFO:tasks.rados.rados.0.smithi107.stdout:9250: finishing write tid 6 to smithi10729407-3 2020-06-12T02:38:48.883 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1645 (ObjNum 9249 snap 0 seq_num 9249) dirty exists 2020-06-12T02:38:48.883 INFO:tasks.rados.rados.0.smithi107.stdout:9250: left oid 3 (ObjNum 9249 snap 0 seq_num 9249) 2020-06-12T02:38:48.884 INFO:tasks.rados.rados.0.smithi107.stdout:9250: done (2 left) 2020-06-12T02:38:48.884 INFO:tasks.rados.rados.0.smithi107.stdout:9253: done (1 left) 2020-06-12T02:38:48.884 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:38:48.885 INFO:tasks.rados.rados.0.smithi107.stdout:9254: seq_num 9253 ranges {0=573440,573440=589824,1163264=212992} 2020-06-12T02:38:48.885 INFO:tasks.rados.rados.0.smithi107.stdout:9254: writing smithi10729407-6 from 0 to 573440 tid 1 2020-06-12T02:38:48.887 INFO:tasks.rados.rados.0.smithi107.stdout:9254: writing smithi10729407-6 from 573440 to 1163264 tid 2 2020-06-12T02:38:48.888 INFO:tasks.rados.rados.0.smithi107.stdout:9254: writing smithi10729407-6 from 1163264 to 1376256 tid 3 2020-06-12T02:38:48.888 INFO:tasks.rados.rados.0.smithi107.stdout:9255: delete oid 17 current snap is 0 2020-06-12T02:38:48.889 INFO:tasks.rados.rados.0.smithi107.stdout:9251: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:48.890 INFO:tasks.rados.rados.0.smithi107.stdout:9251: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:48.890 INFO:tasks.rados.rados.0.smithi107.stdout:9251: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:48.890 INFO:tasks.rados.rados.0.smithi107.stdout:9251: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:48.891 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1720 (ObjNum 9250 snap 0 seq_num 9250) dirty exists 2020-06-12T02:38:48.891 INFO:tasks.rados.rados.0.smithi107.stdout:9251: left oid 18 (ObjNum 9250 snap 0 seq_num 9250) 2020-06-12T02:38:48.892 INFO:tasks.rados.rados.0.smithi107.stdout:9251: done (2 left) 2020-06-12T02:38:48.892 INFO:tasks.rados.rados.0.smithi107.stdout:9255: done (1 left) 2020-06-12T02:38:48.892 INFO:tasks.rados.rados.0.smithi107.stdout:9256: delete oid 12 current snap is 0 2020-06-12T02:38:48.893 INFO:tasks.rados.rados.0.smithi107.stdout:9254: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:48.896 INFO:tasks.rados.rados.0.smithi107.stdout:9256: done (1 left) 2020-06-12T02:38:48.896 INFO:tasks.rados.rados.0.smithi107.stdout:9257: delete oid 14 current snap is 0 2020-06-12T02:38:48.897 INFO:tasks.rados.rados.0.smithi107.stdout:9254: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:48.899 INFO:tasks.rados.rados.0.smithi107.stdout:9254: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:48.905 INFO:tasks.rados.rados.0.smithi107.stdout:9254: finishing write tid 5 to smithi10729407-6 2020-06-12T02:38:48.905 INFO:tasks.rados.rados.0.smithi107.stdout:9257: done (1 left) 2020-06-12T02:38:48.905 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:48.906 INFO:tasks.rados.rados.0.smithi107.stdout:9258: seq_num 9257 ranges {0=638976,638976=524288,1163264=278528} 2020-06-12T02:38:48.908 INFO:tasks.rados.rados.0.smithi107.stdout:9258: writing smithi10729407-19 from 0 to 638976 tid 1 2020-06-12T02:38:48.914 INFO:tasks.rados.rados.0.smithi107.stdout:9258: writing smithi10729407-19 from 638976 to 1163264 tid 2 2020-06-12T02:38:48.915 INFO:tasks.rados.rados.0.smithi107.stdout:9258: writing smithi10729407-19 from 1163264 to 1441792 tid 3 2020-06-12T02:38:48.915 INFO:tasks.rados.rados.0.smithi107.stdout:9259: delete oid 20 current snap is 0 2020-06-12T02:38:48.915 INFO:tasks.rados.rados.0.smithi107.stdout:9254: finishing write tid 6 to smithi10729407-6 2020-06-12T02:38:48.915 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1682 (ObjNum 9253 snap 0 seq_num 9253) dirty exists 2020-06-12T02:38:48.915 INFO:tasks.rados.rados.0.smithi107.stdout:9254: left oid 6 (ObjNum 9253 snap 0 seq_num 9253) 2020-06-12T02:38:48.916 INFO:tasks.rados.rados.0.smithi107.stdout:9254: done (2 left) 2020-06-12T02:38:48.917 INFO:tasks.rados.rados.0.smithi107.stdout:9259: done (1 left) 2020-06-12T02:38:48.917 INFO:tasks.rados.rados.0.smithi107.stdout:9260: delete oid 13 current snap is 0 2020-06-12T02:38:48.920 INFO:tasks.rados.rados.0.smithi107.stdout:9260: done (1 left) 2020-06-12T02:38:48.920 INFO:tasks.rados.rados.0.smithi107.stdout:9261: delete oid 2 current snap is 0 2020-06-12T02:38:48.920 INFO:tasks.rados.rados.0.smithi107.stdout:9258: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:48.921 INFO:tasks.rados.rados.0.smithi107.stdout:9258: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:48.923 INFO:tasks.rados.rados.0.smithi107.stdout:9258: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:48.923 INFO:tasks.rados.rados.0.smithi107.stdout:9258: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:48.923 INFO:tasks.rados.rados.0.smithi107.stdout:9261: done (1 left) 2020-06-12T02:38:48.924 INFO:tasks.rados.rados.0.smithi107.stdout:9262: delete oid 14 current snap is 0 2020-06-12T02:38:48.924 INFO:tasks.rados.rados.0.smithi107.stdout:9258: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:48.924 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1708 (ObjNum 9257 snap 0 seq_num 9257) dirty exists 2020-06-12T02:38:48.925 INFO:tasks.rados.rados.0.smithi107.stdout:9258: left oid 19 (ObjNum 9257 snap 0 seq_num 9257) 2020-06-12T02:38:48.926 INFO:tasks.rados.rados.0.smithi107.stdout:9258: done (1 left) 2020-06-12T02:38:48.926 INFO:tasks.rados.rados.0.smithi107.stdout:9262: done (0 left) 2020-06-12T02:38:48.926 INFO:tasks.rados.rados.0.smithi107.stdout:9263: delete oid 2 current snap is 0 2020-06-12T02:38:48.927 INFO:tasks.rados.rados.0.smithi107.stdout:9263: done (0 left) 2020-06-12T02:38:48.928 INFO:tasks.rados.rados.0.smithi107.stdout:9264: delete oid 16 current snap is 0 2020-06-12T02:38:48.930 INFO:tasks.rados.rados.0.smithi107.stdout:9264: done (0 left) 2020-06-12T02:38:48.930 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:48.931 INFO:tasks.rados.rados.0.smithi107.stdout:9265: seq_num 9264 ranges {1327104=761856,2088960=565248,2654208=237568} 2020-06-12T02:38:48.943 INFO:tasks.rados.rados.0.smithi107.stdout:9265: writing smithi10729407-18 from 1327104 to 2088960 tid 1 2020-06-12T02:38:48.948 INFO:tasks.rados.rados.0.smithi107.stdout:9265: writing smithi10729407-18 from 2088960 to 2654208 tid 2 2020-06-12T02:38:48.950 INFO:tasks.rados.rados.0.smithi107.stdout:9265: writing smithi10729407-18 from 2654208 to 2891776 tid 3 2020-06-12T02:38:48.950 INFO:tasks.rados.rados.0.smithi107.stdout:9266: delete oid 17 current snap is 0 2020-06-12T02:38:48.956 INFO:tasks.rados.rados.0.smithi107.stdout:9265: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:48.957 INFO:tasks.rados.rados.0.smithi107.stdout:9266: done (1 left) 2020-06-12T02:38:48.957 INFO:tasks.rados.rados.0.smithi107.stdout:9267: delete oid 8 current snap is 0 2020-06-12T02:38:48.958 INFO:tasks.rados.rados.0.smithi107.stdout:9265: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:48.958 INFO:tasks.rados.rados.0.smithi107.stdout:9265: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:48.959 INFO:tasks.rados.rados.0.smithi107.stdout:9267: done (1 left) 2020-06-12T02:38:48.959 INFO:tasks.rados.rados.0.smithi107.stdout:9268: delete oid 7 current snap is 0 2020-06-12T02:38:48.959 INFO:tasks.rados.rados.0.smithi107.stdout:9265: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:48.961 INFO:tasks.rados.rados.0.smithi107.stdout:9265: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:48.961 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1724 (ObjNum 9264 snap 0 seq_num 9264) dirty exists 2020-06-12T02:38:48.961 INFO:tasks.rados.rados.0.smithi107.stdout:9265: left oid 18 (ObjNum 9264 snap 0 seq_num 9264) 2020-06-12T02:38:48.964 INFO:tasks.rados.rados.0.smithi107.stdout:9265: done (1 left) 2020-06-12T02:38:48.964 INFO:tasks.rados.rados.0.smithi107.stdout:9268: done (0 left) 2020-06-12T02:38:48.964 INFO:tasks.rados.rados.0.smithi107.stdout:9269: delete oid 18 current snap is 0 2020-06-12T02:38:48.965 INFO:tasks.rados.rados.0.smithi107.stdout:9269: done (0 left) 2020-06-12T02:38:48.966 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:48.966 INFO:tasks.rados.rados.0.smithi107.stdout:9270: seq_num 9269 ranges {1310720=442368,1753088=434176,2187264=638976,2826240=131072} 2020-06-12T02:38:48.972 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:48.969+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:48.975 INFO:tasks.rados.rados.0.smithi107.stdout:9270: writing smithi10729407-3 from 1310720 to 1753088 tid 1 2020-06-12T02:38:48.977 INFO:tasks.rados.rados.0.smithi107.stdout:9270: writing smithi10729407-3 from 1753088 to 2187264 tid 2 2020-06-12T02:38:48.984 INFO:tasks.rados.rados.0.smithi107.stdout:9270: writing smithi10729407-3 from 2187264 to 2826240 tid 3 2020-06-12T02:38:48.984 INFO:tasks.rados.rados.0.smithi107.stdout:9270: writing smithi10729407-3 from 2826240 to 2957312 tid 4 2020-06-12T02:38:48.984 INFO:tasks.rados.rados.0.smithi107.stdout:9271: delete oid 8 current snap is 0 2020-06-12T02:38:48.985 INFO:tasks.rados.rados.0.smithi107.stdout:9270: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:48.985 INFO:tasks.rados.rados.0.smithi107.stdout:9271: done (1 left) 2020-06-12T02:38:48.985 INFO:tasks.rados.rados.0.smithi107.stdout:9272: delete oid 6 current snap is 0 2020-06-12T02:38:48.987 INFO:tasks.rados.rados.0.smithi107.stdout:9272: done (1 left) 2020-06-12T02:38:48.988 INFO:tasks.rados.rados.0.smithi107.stdout:9273: delete oid 14 current snap is 0 2020-06-12T02:38:48.990 INFO:tasks.rados.rados.0.smithi107.stdout:9273: done (1 left) 2020-06-12T02:38:48.990 INFO:tasks.rados.rados.0.smithi107.stdout:9274: delete oid 5 current snap is 0 2020-06-12T02:38:48.993 INFO:tasks.rados.rados.0.smithi107.stdout:9270: finishing write tid 2 to smithi10729407-3 2020-06-12T02:38:48.994 INFO:tasks.rados.rados.0.smithi107.stdout:9274: done (1 left) 2020-06-12T02:38:48.994 INFO:tasks.rados.rados.0.smithi107.stdout:9275: delete oid 7 current snap is 0 2020-06-12T02:38:48.995 INFO:tasks.rados.rados.0.smithi107.stdout:9270: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:48.995 INFO:tasks.rados.rados.0.smithi107.stdout:9270: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:48.995 INFO:tasks.rados.rados.0.smithi107.stdout:9270: finishing write tid 6 to smithi10729407-3 2020-06-12T02:38:48.996 INFO:tasks.rados.rados.0.smithi107.stdout:9275: done (1 left) 2020-06-12T02:38:48.996 INFO:tasks.rados.rados.0.smithi107.stdout:9276: delete oid 12 current snap is 0 2020-06-12T02:38:48.997 INFO:tasks.rados.rados.0.smithi107.stdout:9270: finishing write tid 7 to smithi10729407-3 2020-06-12T02:38:48.998 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1652 (ObjNum 9269 snap 0 seq_num 9269) dirty exists 2020-06-12T02:38:48.998 INFO:tasks.rados.rados.0.smithi107.stdout:9270: left oid 3 (ObjNum 9269 snap 0 seq_num 9269) 2020-06-12T02:38:48.998 INFO:tasks.rados.rados.0.smithi107.stdout:9270: done (1 left) 2020-06-12T02:38:48.999 INFO:tasks.rados.rados.0.smithi107.stdout:9276: done (0 left) 2020-06-12T02:38:48.999 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:48.999 INFO:tasks.rados.rados.0.smithi107.stdout:9277: seq_num 9276 ranges {0=557056,557056=475136,1032192=729088} 2020-06-12T02:38:49.001 INFO:tasks.rados.rados.0.smithi107.stdout:9277: writing smithi10729407-4 from 0 to 557056 tid 1 2020-06-12T02:38:49.004 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:49.004 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:49.005 INFO:tasks.rados.rados.0.smithi107.stdout:9277: writing smithi10729407-4 from 557056 to 1032192 tid 2 2020-06-12T02:38:49.009 INFO:tasks.rados.rados.0.smithi107.stdout:9277: writing smithi10729407-4 from 1032192 to 1761280 tid 3 2020-06-12T02:38:49.009 INFO:tasks.rados.rados.0.smithi107.stdout:9278: delete oid 13 current snap is 0 2020-06-12T02:38:49.009 INFO:tasks.rados.rados.0.smithi107.stdout:9277: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:49.011 INFO:tasks.rados.rados.0.smithi107.stdout:9278: done (1 left) 2020-06-12T02:38:49.012 INFO:tasks.rados.rados.0.smithi107.stdout:9279: delete oid 18 current snap is 0 2020-06-12T02:38:49.014 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-06-12T02:38:49.017 INFO:tasks.rados.rados.0.smithi107.stdout:9277: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:49.019 INFO:tasks.rados.rados.0.smithi107.stdout:9277: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:49.019 INFO:tasks.rados.rados.0.smithi107.stdout:9279: done (1 left) 2020-06-12T02:38:49.019 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:49.020 INFO:tasks.rados.rados.0.smithi107.stdout:9280: seq_num 9279 ranges {0=614400,614400=532480,1146880=614400,1761280=57344} 2020-06-12T02:38:49.023 INFO:tasks.rados.rados.0.smithi107.stdout:9280: writing smithi10729407-17 from 0 to 614400 tid 1 2020-06-12T02:38:49.031 INFO:tasks.rados.rados.0.smithi107.stdout:9280: writing smithi10729407-17 from 614400 to 1146880 tid 2 2020-06-12T02:38:49.032 INFO:tasks.rados.rados.0.smithi107.stdout:9280: writing smithi10729407-17 from 1146880 to 1761280 tid 3 2020-06-12T02:38:49.032 INFO:tasks.rados.rados.0.smithi107.stdout:9280: writing smithi10729407-17 from 1761280 to 1818624 tid 4 2020-06-12T02:38:49.032 INFO:tasks.rados.rados.0.smithi107.stdout:9281: delete oid 2 current snap is 0 2020-06-12T02:38:49.032 INFO:tasks.rados.rados.0.smithi107.stdout:9277: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:49.041 INFO:tasks.rados.rados.0.smithi107.stdout:9277: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:49.041 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1743 (ObjNum 9276 snap 0 seq_num 9276) dirty exists 2020-06-12T02:38:49.041 INFO:tasks.rados.rados.0.smithi107.stdout:9277: left oid 4 (ObjNum 9276 snap 0 seq_num 9276) 2020-06-12T02:38:49.041 INFO:tasks.rados.rados.0.smithi107.stdout:9280: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:49.042 INFO:tasks.rados.rados.0.smithi107.stdout:9277: done (2 left) 2020-06-12T02:38:49.042 INFO:tasks.rados.rados.0.smithi107.stdout:9281: done (1 left) 2020-06-12T02:38:49.042 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:49.043 INFO:tasks.rados.rados.0.smithi107.stdout:9282: seq_num 9281 ranges {0=647168,647168=696320,1343488=507904} 2020-06-12T02:38:49.043 INFO:tasks.rados.rados.0.smithi107.stdout:9282: writing smithi10729407-14 from 0 to 647168 tid 1 2020-06-12T02:38:49.043 INFO:tasks.rados.rados.0.smithi107.stdout:9282: writing smithi10729407-14 from 647168 to 1343488 tid 2 2020-06-12T02:38:49.052 INFO:tasks.rados.rados.0.smithi107.stdout:9282: writing smithi10729407-14 from 1343488 to 1851392 tid 3 2020-06-12T02:38:49.052 INFO:tasks.rados.rados.0.smithi107.stdout:9283: delete oid 10 current snap is 0 2020-06-12T02:38:49.052 INFO:tasks.rados.rados.0.smithi107.stdout:9280: finishing write tid 2 to smithi10729407-17 2020-06-12T02:38:49.052 INFO:tasks.rados.rados.0.smithi107.stdout:9280: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:49.053 INFO:tasks.rados.rados.0.smithi107.stdout:9280: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:49.053 INFO:tasks.rados.rados.0.smithi107.stdout:9280: finishing write tid 6 to smithi10729407-17 2020-06-12T02:38:49.053 INFO:tasks.rados.rados.0.smithi107.stdout:9282: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:49.053 INFO:tasks.rados.rados.0.smithi107.stdout:9280: finishing write tid 7 to smithi10729407-17 2020-06-12T02:38:49.053 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1748 (ObjNum 9279 snap 0 seq_num 9279) dirty exists 2020-06-12T02:38:49.054 INFO:tasks.rados.rados.0.smithi107.stdout:9280: left oid 17 (ObjNum 9279 snap 0 seq_num 9279) 2020-06-12T02:38:49.054 INFO:tasks.rados.rados.0.smithi107.stdout:9280: done (2 left) 2020-06-12T02:38:49.054 INFO:tasks.rados.rados.0.smithi107.stdout:9283: done (1 left) 2020-06-12T02:38:49.054 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:49.054 INFO:tasks.rados.rados.0.smithi107.stdout:9284: seq_num 9283 ranges {0=679936,679936=516096,1196032=589824,1785856=98304} 2020-06-12T02:38:49.054 INFO:tasks.rados.rados.0.smithi107.stdout:9284: writing smithi10729407-6 from 0 to 679936 tid 1 2020-06-12T02:38:49.055 INFO:tasks.rados.rados.0.smithi107.stdout:9284: writing smithi10729407-6 from 679936 to 1196032 tid 2 2020-06-12T02:38:49.057 INFO:tasks.rados.rados.0.smithi107.stdout:9284: writing smithi10729407-6 from 1196032 to 1785856 tid 3 2020-06-12T02:38:49.057 INFO:tasks.rados.rados.0.smithi107.stdout:9284: writing smithi10729407-6 from 1785856 to 1884160 tid 4 2020-06-12T02:38:49.058 INFO:tasks.rados.rados.0.smithi107.stdout:9285: delete oid 8 current snap is 0 2020-06-12T02:38:49.058 INFO:tasks.rados.rados.0.smithi107.stdout:9282: finishing write tid 2 to smithi10729407-14 2020-06-12T02:38:49.058 INFO:tasks.rados.rados.0.smithi107.stdout:9282: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:49.059 INFO:tasks.rados.rados.0.smithi107.stdout:9282: finishing write tid 5 to smithi10729407-14 2020-06-12T02:38:49.059 INFO:tasks.rados.rados.0.smithi107.stdout:9284: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:49.061 INFO:tasks.rados.rados.0.smithi107.stdout:9282: finishing write tid 6 to smithi10729407-14 2020-06-12T02:38:49.061 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1590 (ObjNum 9281 snap 0 seq_num 9281) dirty exists 2020-06-12T02:38:49.061 INFO:tasks.rados.rados.0.smithi107.stdout:9282: left oid 14 (ObjNum 9281 snap 0 seq_num 9281) 2020-06-12T02:38:49.061 INFO:tasks.rados.rados.0.smithi107.stdout:9282: done (2 left) 2020-06-12T02:38:49.062 INFO:tasks.rados.rados.0.smithi107.stdout:9285: done (1 left) 2020-06-12T02:38:49.062 INFO:tasks.rados.rados.0.smithi107.stdout:9286: delete oid 5 current snap is 0 2020-06-12T02:38:49.064 INFO:tasks.rados.rados.0.smithi107.stdout:9286: done (1 left) 2020-06-12T02:38:49.065 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:49.065 INFO:tasks.rados.rados.0.smithi107.stdout:9287: seq_num 9286 ranges {2301952=729088,3031040=573440,3604480=630784} 2020-06-12T02:38:49.073 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:49.071+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:49.081 INFO:tasks.rados.rados.0.smithi107.stdout:9287: writing smithi10729407-11 from 2301952 to 3031040 tid 1 2020-06-12T02:38:49.085 INFO:tasks.rados.rados.0.smithi107.stdout:9287: writing smithi10729407-11 from 3031040 to 3604480 tid 2 2020-06-12T02:38:49.089 INFO:tasks.rados.rados.0.smithi107.stdout:9287: writing smithi10729407-11 from 3604480 to 4235264 tid 3 2020-06-12T02:38:49.089 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:49.089 INFO:tasks.rados.rados.0.smithi107.stdout:9288: seq_num 9287 ranges {1441792=745472,2187264=458752,2646016=573440,3219456=172032} 2020-06-12T02:38:49.102 INFO:tasks.rados.rados.0.smithi107.stdout:9288: writing smithi10729407-19 from 1441792 to 2187264 tid 1 2020-06-12T02:38:49.104 INFO:tasks.rados.rados.0.smithi107.stdout:9288: writing smithi10729407-19 from 2187264 to 2646016 tid 2 2020-06-12T02:38:49.107 INFO:tasks.rados.rados.0.smithi107.stdout:9288: writing smithi10729407-19 from 2646016 to 3219456 tid 3 2020-06-12T02:38:49.109 INFO:tasks.rados.rados.0.smithi107.stdout:9288: writing smithi10729407-19 from 3219456 to 3391488 tid 4 2020-06-12T02:38:49.109 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:38:49.109 INFO:tasks.rados.rados.0.smithi107.stdout:9289: seq_num 9288 ranges {2187264=761856,2949120=737280,3686400=466944} 2020-06-12T02:38:49.126 INFO:tasks.rados.rados.0.smithi107.stdout:9289: writing smithi10729407-9 from 2187264 to 2949120 tid 1 2020-06-12T02:38:49.130 INFO:tasks.rados.rados.0.smithi107.stdout:9289: writing smithi10729407-9 from 2949120 to 3686400 tid 2 2020-06-12T02:38:49.139 INFO:tasks.rados.rados.0.smithi107.stdout:9289: writing smithi10729407-9 from 3686400 to 4153344 tid 3 2020-06-12T02:38:49.139 INFO:tasks.rados.rados.0.smithi107.stdout:9290: delete oid 4 current snap is 0 2020-06-12T02:38:49.139 INFO:tasks.rados.rados.0.smithi107.stdout:9284: finishing write tid 2 to smithi10729407-6 2020-06-12T02:38:49.139 INFO:tasks.rados.rados.0.smithi107.stdout:9284: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:49.140 INFO:tasks.rados.rados.0.smithi107.stdout:9284: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:49.140 INFO:tasks.rados.rados.0.smithi107.stdout:9284: finishing write tid 6 to smithi10729407-6 2020-06-12T02:38:49.140 INFO:tasks.rados.rados.0.smithi107.stdout:9284: finishing write tid 7 to smithi10729407-6 2020-06-12T02:38:49.140 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1688 (ObjNum 9283 snap 0 seq_num 9283) dirty exists 2020-06-12T02:38:49.140 INFO:tasks.rados.rados.0.smithi107.stdout:9284: left oid 6 (ObjNum 9283 snap 0 seq_num 9283) 2020-06-12T02:38:49.141 INFO:tasks.rados.rados.0.smithi107.stdout:9287: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:49.141 INFO:tasks.rados.rados.0.smithi107.stdout:9287: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:49.141 INFO:tasks.rados.rados.0.smithi107.stdout:9287: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:49.141 INFO:tasks.rados.rados.0.smithi107.stdout:9287: finishing write tid 5 to smithi10729407-11 2020-06-12T02:38:49.141 INFO:tasks.rados.rados.0.smithi107.stdout:9287: finishing write tid 6 to smithi10729407-11 2020-06-12T02:38:49.142 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1603 (ObjNum 9286 snap 0 seq_num 9286) dirty exists 2020-06-12T02:38:49.142 INFO:tasks.rados.rados.0.smithi107.stdout:9287: left oid 11 (ObjNum 9286 snap 0 seq_num 9286) 2020-06-12T02:38:49.142 INFO:tasks.rados.rados.0.smithi107.stdout:9288: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:49.142 INFO:tasks.rados.rados.0.smithi107.stdout:9288: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:49.142 INFO:tasks.rados.rados.0.smithi107.stdout:9288: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:49.143 INFO:tasks.rados.rados.0.smithi107.stdout:9288: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:49.143 INFO:tasks.rados.rados.0.smithi107.stdout:9288: finishing write tid 6 to smithi10729407-19 2020-06-12T02:38:49.143 INFO:tasks.rados.rados.0.smithi107.stdout:9288: finishing write tid 7 to smithi10729407-19 2020-06-12T02:38:49.143 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1714 (ObjNum 9287 snap 0 seq_num 9287) dirty exists 2020-06-12T02:38:49.143 INFO:tasks.rados.rados.0.smithi107.stdout:9288: left oid 19 (ObjNum 9287 snap 0 seq_num 9287) 2020-06-12T02:38:49.144 INFO:tasks.rados.rados.0.smithi107.stdout:9289: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:49.145 INFO:tasks.rados.rados.0.smithi107.stdout:9289: finishing write tid 2 to smithi10729407-9 2020-06-12T02:38:49.145 INFO:tasks.rados.rados.0.smithi107.stdout:9284: done (4 left) 2020-06-12T02:38:49.145 INFO:tasks.rados.rados.0.smithi107.stdout:9287: done (3 left) 2020-06-12T02:38:49.145 INFO:tasks.rados.rados.0.smithi107.stdout:9288: done (2 left) 2020-06-12T02:38:49.146 INFO:tasks.rados.rados.0.smithi107.stdout:9290: done (1 left) 2020-06-12T02:38:49.146 INFO:tasks.rados.rados.0.smithi107.stdout:9291: delete oid 4 current snap is 0 2020-06-12T02:38:49.146 INFO:tasks.rados.rados.0.smithi107.stdout:9289: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:49.146 INFO:tasks.rados.rados.0.smithi107.stdout:9289: finishing write tid 5 to smithi10729407-9 2020-06-12T02:38:49.147 INFO:tasks.rados.rados.0.smithi107.stdout:9289: finishing write tid 6 to smithi10729407-9 2020-06-12T02:38:49.147 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1718 (ObjNum 9288 snap 0 seq_num 9288) dirty exists 2020-06-12T02:38:49.147 INFO:tasks.rados.rados.0.smithi107.stdout:9289: left oid 9 (ObjNum 9288 snap 0 seq_num 9288) 2020-06-12T02:38:49.148 INFO:tasks.rados.rados.0.smithi107.stdout:9289: done (1 left) 2020-06-12T02:38:49.148 INFO:tasks.rados.rados.0.smithi107.stdout:9291: done (0 left) 2020-06-12T02:38:49.148 INFO:tasks.rados.rados.0.smithi107.stdout:9292: delete oid 18 current snap is 0 2020-06-12T02:38:49.149 INFO:tasks.rados.rados.0.smithi107.stdout:9292: done (0 left) 2020-06-12T02:38:49.149 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:38:49.149 INFO:tasks.rados.rados.0.smithi107.stdout:9293: seq_num 9292 ranges {0=425984,425984=729088,1155072=761856,1916928=114688} 2020-06-12T02:38:49.150 INFO:tasks.rados.rados.0.smithi107.stdout:9293: writing smithi10729407-4 from 0 to 425984 tid 1 2020-06-12T02:38:49.155 INFO:tasks.rados.rados.0.smithi107.stdout:9293: writing smithi10729407-4 from 425984 to 1155072 tid 2 2020-06-12T02:38:49.159 INFO:tasks.rados.rados.0.smithi107.stdout:9293: writing smithi10729407-4 from 1155072 to 1916928 tid 3 2020-06-12T02:38:49.160 INFO:tasks.rados.rados.0.smithi107.stdout:9293: writing smithi10729407-4 from 1916928 to 2031616 tid 4 2020-06-12T02:38:49.160 INFO:tasks.rados.rados.0.smithi107.stdout:9294: delete oid 8 current snap is 0 2020-06-12T02:38:49.160 INFO:tasks.rados.rados.0.smithi107.stdout:9293: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:49.163 INFO:tasks.rados.rados.0.smithi107.stdout:9294: done (1 left) 2020-06-12T02:38:49.163 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:38:49.164 INFO:tasks.rados.rados.0.smithi107.stdout:9295: seq_num 9294 ranges {4235264=458752,4694016=491520,5185536=729088,5914624=385024} 2020-06-12T02:38:49.168 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:49.168 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:49.174 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:49.172+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:49.177 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-06-12T02:38:49.189 INFO:tasks.rados.rados.0.smithi107.stdout:9295: writing smithi10729407-11 from 4235264 to 4694016 tid 1 2020-06-12T02:38:49.192 INFO:tasks.rados.rados.0.smithi107.stdout:9295: writing smithi10729407-11 from 4694016 to 5185536 tid 2 2020-06-12T02:38:49.197 INFO:tasks.rados.rados.0.smithi107.stdout:9295: writing smithi10729407-11 from 5185536 to 5914624 tid 3 2020-06-12T02:38:49.206 INFO:tasks.rados.rados.0.smithi107.stdout:9295: writing smithi10729407-11 from 5914624 to 6299648 tid 4 2020-06-12T02:38:49.207 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:38:49.207 INFO:tasks.rados.rados.0.smithi107.stdout:9296: seq_num 9295 ranges {491520=475136,966656=778240,1744896=532480,2277376=294912} 2020-06-12T02:38:49.207 INFO:tasks.rados.rados.0.smithi107.stdout:9296: writing smithi10729407-15 from 491520 to 966656 tid 1 2020-06-12T02:38:49.215 INFO:tasks.rados.rados.0.smithi107.stdout:9296: writing smithi10729407-15 from 966656 to 1744896 tid 2 2020-06-12T02:38:49.216 INFO:tasks.rados.rados.0.smithi107.stdout:9296: writing smithi10729407-15 from 1744896 to 2277376 tid 3 2020-06-12T02:38:49.216 INFO:tasks.rados.rados.0.smithi107.stdout:9296: writing smithi10729407-15 from 2277376 to 2572288 tid 4 2020-06-12T02:38:49.217 INFO:tasks.rados.rados.0.smithi107.stdout:9297: delete oid 3 current snap is 0 2020-06-12T02:38:49.217 INFO:tasks.rados.rados.0.smithi107.stdout:9293: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:49.217 INFO:tasks.rados.rados.0.smithi107.stdout:9293: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:49.217 INFO:tasks.rados.rados.0.smithi107.stdout:9293: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:49.217 INFO:tasks.rados.rados.0.smithi107.stdout:9293: finishing write tid 6 to smithi10729407-4 2020-06-12T02:38:49.218 INFO:tasks.rados.rados.0.smithi107.stdout:9293: finishing write tid 7 to smithi10729407-4 2020-06-12T02:38:49.218 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1755 (ObjNum 9292 snap 0 seq_num 9292) dirty exists 2020-06-12T02:38:49.218 INFO:tasks.rados.rados.0.smithi107.stdout:9293: left oid 4 (ObjNum 9292 snap 0 seq_num 9292) 2020-06-12T02:38:49.218 INFO:tasks.rados.rados.0.smithi107.stdout:9295: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:49.219 INFO:tasks.rados.rados.0.smithi107.stdout:9295: finishing write tid 2 to smithi10729407-11 2020-06-12T02:38:49.219 INFO:tasks.rados.rados.0.smithi107.stdout:9295: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:49.219 INFO:tasks.rados.rados.0.smithi107.stdout:9295: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:49.219 INFO:tasks.rados.rados.0.smithi107.stdout:9295: finishing write tid 6 to smithi10729407-11 2020-06-12T02:38:49.219 INFO:tasks.rados.rados.0.smithi107.stdout:9296: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:49.220 INFO:tasks.rados.rados.0.smithi107.stdout:9295: finishing write tid 7 to smithi10729407-11 2020-06-12T02:38:49.220 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1608 (ObjNum 9294 snap 0 seq_num 9294) dirty exists 2020-06-12T02:38:49.220 INFO:tasks.rados.rados.0.smithi107.stdout:9295: left oid 11 (ObjNum 9294 snap 0 seq_num 9294) 2020-06-12T02:38:49.220 INFO:tasks.rados.rados.0.smithi107.stdout:9293: done (3 left) 2020-06-12T02:38:49.221 INFO:tasks.rados.rados.0.smithi107.stdout:9295: done (2 left) 2020-06-12T02:38:49.221 INFO:tasks.rados.rados.0.smithi107.stdout:9297: done (1 left) 2020-06-12T02:38:49.221 INFO:tasks.rados.rados.0.smithi107.stdout:9298: delete oid 10 current snap is 0 2020-06-12T02:38:49.228 INFO:tasks.rados.rados.0.smithi107.stdout:9296: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:49.228 INFO:tasks.rados.rados.0.smithi107.stdout:9298: done (1 left) 2020-06-12T02:38:49.228 INFO:tasks.rados.rados.0.smithi107.stdout:9299: delete oid 2 current snap is 0 2020-06-12T02:38:49.229 INFO:tasks.rados.rados.0.smithi107.stdout:9296: finishing write tid 3 to smithi10729407-15 2020-06-12T02:38:49.229 INFO:tasks.rados.rados.0.smithi107.stdout:9296: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:49.229 INFO:tasks.rados.rados.0.smithi107.stdout:9299: done (1 left) 2020-06-12T02:38:49.229 INFO:tasks.rados.rados.0.smithi107.stdout:9300: delete oid 3 current snap is 0 2020-06-12T02:38:49.229 INFO:tasks.rados.rados.0.smithi107.stdout:9296: finishing write tid 6 to smithi10729407-15 2020-06-12T02:38:49.230 INFO:tasks.rados.rados.0.smithi107.stdout:9296: finishing write tid 7 to smithi10729407-15 2020-06-12T02:38:49.230 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1723 (ObjNum 9295 snap 0 seq_num 9295) dirty exists 2020-06-12T02:38:49.230 INFO:tasks.rados.rados.0.smithi107.stdout:9296: left oid 15 (ObjNum 9295 snap 0 seq_num 9295) 2020-06-12T02:38:49.230 INFO:tasks.rados.rados.0.smithi107.stdout:9296: done (1 left) 2020-06-12T02:38:49.230 INFO:tasks.rados.rados.0.smithi107.stdout:9300: done (0 left) 2020-06-12T02:38:49.231 INFO:tasks.rados.rados.0.smithi107.stdout:9301: delete oid 19 current snap is 0 2020-06-12T02:38:49.231 INFO:tasks.rados.rados.0.smithi107.stdout:9301: done (0 left) 2020-06-12T02:38:49.231 INFO:tasks.rados.rados.0.smithi107.stdout:9302: delete oid 19 current snap is 0 2020-06-12T02:38:49.237 INFO:tasks.rados.rados.0.smithi107.stdout:9302: done (0 left) 2020-06-12T02:38:49.238 INFO:tasks.rados.rados.0.smithi107.stdout:9303: delete oid 16 current snap is 0 2020-06-12T02:38:49.241 INFO:tasks.rados.rados.0.smithi107.stdout:9303: done (0 left) 2020-06-12T02:38:49.241 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:38:49.241 INFO:tasks.rados.rados.0.smithi107.stdout:9304: seq_num 9303 ranges {0=614400,614400=704512,1318912=491520,1810432=409600} 2020-06-12T02:38:49.244 INFO:tasks.rados.rados.0.smithi107.stdout:9304: writing smithi10729407-5 from 0 to 614400 tid 1 2020-06-12T02:38:49.248 INFO:tasks.rados.rados.0.smithi107.stdout:9304: writing smithi10729407-5 from 614400 to 1318912 tid 2 2020-06-12T02:38:49.251 INFO:tasks.rados.rados.0.smithi107.stdout:9304: writing smithi10729407-5 from 1318912 to 1810432 tid 3 2020-06-12T02:38:49.254 INFO:tasks.rados.rados.0.smithi107.stdout:9304: writing smithi10729407-5 from 1810432 to 2220032 tid 4 2020-06-12T02:38:49.256 INFO:tasks.rados.rados.0.smithi107.stdout:9305: delete oid 11 current snap is 0 2020-06-12T02:38:49.257 INFO:tasks.rados.rados.0.smithi107.stdout:9304: finishing write tid 1 to smithi10729407-5 2020-06-12T02:38:49.257 INFO:tasks.rados.rados.0.smithi107.stdout:9305: done (1 left) 2020-06-12T02:38:49.258 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:38:49.258 INFO:tasks.rados.rados.0.smithi107.stdout:9306: seq_num 9305 ranges {0=647168,647168=475136,1122304=458752,1581056=450560,2031616=221184} 2020-06-12T02:38:49.260 INFO:tasks.rados.rados.0.smithi107.stdout:9306: writing smithi10729407-19 from 0 to 647168 tid 1 2020-06-12T02:38:49.263 INFO:tasks.rados.rados.0.smithi107.stdout:9306: writing smithi10729407-19 from 647168 to 1122304 tid 2 2020-06-12T02:38:49.266 INFO:tasks.rados.rados.0.smithi107.stdout:9306: writing smithi10729407-19 from 1122304 to 1581056 tid 3 2020-06-12T02:38:49.269 INFO:tasks.rados.rados.0.smithi107.stdout:9306: writing smithi10729407-19 from 1581056 to 2031616 tid 4 2020-06-12T02:38:49.270 INFO:tasks.rados.rados.0.smithi107.stdout:9306: writing smithi10729407-19 from 2031616 to 2252800 tid 5 2020-06-12T02:38:49.271 INFO:tasks.rados.rados.0.smithi107.stdout:9307: delete oid 16 current snap is 0 2020-06-12T02:38:49.271 INFO:tasks.rados.rados.0.smithi107.stdout:9304: finishing write tid 2 to smithi10729407-5 2020-06-12T02:38:49.271 INFO:tasks.rados.rados.0.smithi107.stdout:9304: finishing write tid 3 to smithi10729407-5 2020-06-12T02:38:49.271 INFO:tasks.rados.rados.0.smithi107.stdout:9304: finishing write tid 4 to smithi10729407-5 2020-06-12T02:38:49.272 INFO:tasks.rados.rados.0.smithi107.stdout:9304: finishing write tid 6 to smithi10729407-5 2020-06-12T02:38:49.272 INFO:tasks.rados.rados.0.smithi107.stdout:9306: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:49.272 INFO:tasks.rados.rados.0.smithi107.stdout:9304: finishing write tid 7 to smithi10729407-5 2020-06-12T02:38:49.272 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 909 (ObjNum 9303 snap 0 seq_num 9303) dirty exists 2020-06-12T02:38:49.273 INFO:tasks.rados.rados.0.smithi107.stdout:9304: left oid 5 (ObjNum 9303 snap 0 seq_num 9303) 2020-06-12T02:38:49.275 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:49.272+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:49.275 INFO:tasks.rados.rados.0.smithi107.stdout:9304: done (2 left) 2020-06-12T02:38:49.275 INFO:tasks.rados.rados.0.smithi107.stdout:9307: done (1 left) 2020-06-12T02:38:49.276 INFO:tasks.rados.rados.0.smithi107.stdout:9308: delete oid 8 current snap is 0 2020-06-12T02:38:49.279 INFO:tasks.rados.rados.0.smithi107.stdout:9308: done (1 left) 2020-06-12T02:38:49.279 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:49.279 INFO:tasks.rados.rados.0.smithi107.stdout:9309: seq_num 9308 ranges {0=696320,696320=573440,1269760=679936,1949696=352256} 2020-06-12T02:38:49.282 INFO:tasks.rados.rados.0.smithi107.stdout:9309: writing smithi10729407-18 from 0 to 696320 tid 1 2020-06-12T02:38:49.286 INFO:tasks.rados.rados.0.smithi107.stdout:9309: writing smithi10729407-18 from 696320 to 1269760 tid 2 2020-06-12T02:38:49.290 INFO:tasks.rados.rados.0.smithi107.stdout:9309: writing smithi10729407-18 from 1269760 to 1949696 tid 3 2020-06-12T02:38:49.292 INFO:tasks.rados.rados.0.smithi107.stdout:9309: writing smithi10729407-18 from 1949696 to 2301952 tid 4 2020-06-12T02:38:49.293 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:38:49.293 INFO:tasks.rados.rados.0.smithi107.stdout:9310: seq_num 9309 ranges {0=712704,712704=458752,1171456=442368,1613824=507904,2121728=196608} 2020-06-12T02:38:49.297 INFO:tasks.rados.rados.0.smithi107.stdout:9310: writing smithi10729407-8 from 0 to 712704 tid 1 2020-06-12T02:38:49.300 INFO:tasks.rados.rados.0.smithi107.stdout:9310: writing smithi10729407-8 from 712704 to 1171456 tid 2 2020-06-12T02:38:49.303 INFO:tasks.rados.rados.0.smithi107.stdout:9310: writing smithi10729407-8 from 1171456 to 1613824 tid 3 2020-06-12T02:38:49.306 INFO:tasks.rados.rados.0.smithi107.stdout:9310: writing smithi10729407-8 from 1613824 to 2121728 tid 4 2020-06-12T02:38:49.307 INFO:tasks.rados.rados.0.smithi107.stdout:9310: writing smithi10729407-8 from 2121728 to 2318336 tid 5 2020-06-12T02:38:49.308 INFO:tasks.rados.rados.0.smithi107.stdout:9311: delete oid 11 current snap is 0 2020-06-12T02:38:49.308 INFO:tasks.rados.rados.0.smithi107.stdout:9306: finishing write tid 2 to smithi10729407-19 2020-06-12T02:38:49.308 INFO:tasks.rados.rados.0.smithi107.stdout:9306: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:49.308 INFO:tasks.rados.rados.0.smithi107.stdout:9306: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:49.309 INFO:tasks.rados.rados.0.smithi107.stdout:9306: finishing write tid 5 to smithi10729407-19 2020-06-12T02:38:49.309 INFO:tasks.rados.rados.0.smithi107.stdout:9306: finishing write tid 7 to smithi10729407-19 2020-06-12T02:38:49.309 INFO:tasks.rados.rados.0.smithi107.stdout:9306: finishing write tid 8 to smithi10729407-19 2020-06-12T02:38:49.310 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1722 (ObjNum 9305 snap 0 seq_num 9305) dirty exists 2020-06-12T02:38:49.310 INFO:tasks.rados.rados.0.smithi107.stdout:9306: left oid 19 (ObjNum 9305 snap 0 seq_num 9305) 2020-06-12T02:38:49.310 INFO:tasks.rados.rados.0.smithi107.stdout:9309: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:49.311 INFO:tasks.rados.rados.0.smithi107.stdout:9309: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:49.311 INFO:tasks.rados.rados.0.smithi107.stdout:9309: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:49.311 INFO:tasks.rados.rados.0.smithi107.stdout:9309: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:49.311 INFO:tasks.rados.rados.0.smithi107.stdout:9309: finishing write tid 6 to smithi10729407-18 2020-06-12T02:38:49.311 INFO:tasks.rados.rados.0.smithi107.stdout:9309: finishing write tid 7 to smithi10729407-18 2020-06-12T02:38:49.312 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1732 (ObjNum 9308 snap 0 seq_num 9308) dirty exists 2020-06-12T02:38:49.312 INFO:tasks.rados.rados.0.smithi107.stdout:9309: left oid 18 (ObjNum 9308 snap 0 seq_num 9308) 2020-06-12T02:38:49.312 INFO:tasks.rados.rados.0.smithi107.stdout:9310: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:49.313 INFO:tasks.rados.rados.0.smithi107.stdout:9306: done (3 left) 2020-06-12T02:38:49.313 INFO:tasks.rados.rados.0.smithi107.stdout:9309: done (2 left) 2020-06-12T02:38:49.313 INFO:tasks.rados.rados.0.smithi107.stdout:9311: done (1 left) 2020-06-12T02:38:49.313 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:38:49.314 INFO:tasks.rados.rados.0.smithi107.stdout:9312: seq_num 9311 ranges {0=745472,745472=630784,1376256=458752,1835008=516096} 2020-06-12T02:38:49.320 INFO:tasks.rados.rados.0.smithi107.stdout:9312: writing smithi10729407-20 from 0 to 745472 tid 1 2020-06-12T02:38:49.321 INFO:tasks.rados.rados.0.smithi107.stdout:9312: writing smithi10729407-20 from 745472 to 1376256 tid 2 2020-06-12T02:38:49.321 INFO:tasks.rados.rados.0.smithi107.stdout:9312: writing smithi10729407-20 from 1376256 to 1835008 tid 3 2020-06-12T02:38:49.324 INFO:tasks.rados.rados.0.smithi107.stdout:9312: writing smithi10729407-20 from 1835008 to 2351104 tid 4 2020-06-12T02:38:49.324 INFO:tasks.rados.rados.0.smithi107.stdout:9313: delete oid 6 current snap is 0 2020-06-12T02:38:49.324 INFO:tasks.rados.rados.0.smithi107.stdout:9310: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:49.324 INFO:tasks.rados.rados.0.smithi107.stdout:9310: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:49.325 INFO:tasks.rados.rados.0.smithi107.stdout:9310: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:49.325 INFO:tasks.rados.rados.0.smithi107.stdout:9312: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:49.325 INFO:tasks.rados.rados.0.smithi107.stdout:9310: finishing write tid 5 to smithi10729407-8 2020-06-12T02:38:49.326 INFO:tasks.rados.rados.0.smithi107.stdout:9310: finishing write tid 7 to smithi10729407-8 2020-06-12T02:38:49.327 INFO:tasks.rados.rados.0.smithi107.stdout:9313: done (2 left) 2020-06-12T02:38:49.327 INFO:tasks.rados.rados.0.smithi107.stdout:9314: delete oid 13 current snap is 0 2020-06-12T02:38:49.328 INFO:tasks.rados.rados.0.smithi107.stdout:9310: finishing write tid 8 to smithi10729407-8 2020-06-12T02:38:49.328 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1730 (ObjNum 9309 snap 0 seq_num 9309) dirty exists 2020-06-12T02:38:49.328 INFO:tasks.rados.rados.0.smithi107.stdout:9310: left oid 8 (ObjNum 9309 snap 0 seq_num 9309) 2020-06-12T02:38:49.331 INFO:tasks.rados.rados.0.smithi107.stdout:9310: done (2 left) 2020-06-12T02:38:49.331 INFO:tasks.rados.rados.0.smithi107.stdout:9314: done (1 left) 2020-06-12T02:38:49.331 INFO:tasks.rados.rados.0.smithi107.stdout:9315: delete oid 15 current snap is 0 2020-06-12T02:38:49.333 INFO:tasks.rados.rados.0.smithi107.stdout:9312: finishing write tid 2 to smithi10729407-20 2020-06-12T02:38:49.334 INFO:tasks.rados.rados.0.smithi107.stdout:9315: done (1 left) 2020-06-12T02:38:49.334 INFO:tasks.rados.rados.0.smithi107.stdout:9316: delete oid 10 current snap is 0 2020-06-12T02:38:49.336 INFO:tasks.rados.rados.0.smithi107.stdout:9316: done (1 left) 2020-06-12T02:38:49.336 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:49.337 INFO:tasks.rados.rados.0.smithi107.stdout:9317: seq_num 9316 ranges {2031616=32768} 2020-06-12T02:38:49.341 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:49.342 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:49.348 INFO:tasks.rados.rados.0.smithi107.stdout:9317: writing smithi10729407-4 from 2031616 to 2064384 tid 1 2020-06-12T02:38:49.348 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:49.348 INFO:tasks.rados.rados.0.smithi107.stdout:9318: seq_num 9317 ranges {2252800=49152} 2020-06-12T02:38:49.351 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-06-12T02:38:49.361 INFO:tasks.rados.rados.0.smithi107.stdout:9318: writing smithi10729407-19 from 2252800 to 2301952 tid 1 2020-06-12T02:38:49.361 INFO:tasks.rados.rados.0.smithi107.stdout:9319: delete oid 18 current snap is 0 2020-06-12T02:38:49.361 INFO:tasks.rados.rados.0.smithi107.stdout:9312: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:49.362 INFO:tasks.rados.rados.0.smithi107.stdout:9312: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:49.362 INFO:tasks.rados.rados.0.smithi107.stdout:9312: finishing write tid 6 to smithi10729407-20 2020-06-12T02:38:49.362 INFO:tasks.rados.rados.0.smithi107.stdout:9312: finishing write tid 7 to smithi10729407-20 2020-06-12T02:38:49.362 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 689 (ObjNum 9311 snap 0 seq_num 9311) dirty exists 2020-06-12T02:38:49.362 INFO:tasks.rados.rados.0.smithi107.stdout:9312: left oid 20 (ObjNum 9311 snap 0 seq_num 9311) 2020-06-12T02:38:49.363 INFO:tasks.rados.rados.0.smithi107.stdout:9317: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:49.363 INFO:tasks.rados.rados.0.smithi107.stdout:9317: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:49.363 INFO:tasks.rados.rados.0.smithi107.stdout:9317: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:49.363 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1757 (ObjNum 9316 snap 0 seq_num 9316) dirty exists 2020-06-12T02:38:49.364 INFO:tasks.rados.rados.0.smithi107.stdout:9317: left oid 4 (ObjNum 9316 snap 0 seq_num 9316) 2020-06-12T02:38:49.365 INFO:tasks.rados.rados.0.smithi107.stdout:9312: done (3 left) 2020-06-12T02:38:49.365 INFO:tasks.rados.rados.0.smithi107.stdout:9317: done (2 left) 2020-06-12T02:38:49.365 INFO:tasks.rados.rados.0.smithi107.stdout:9319: done (1 left) 2020-06-12T02:38:49.365 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:38:49.365 INFO:tasks.rados.rados.0.smithi107.stdout:9320: seq_num 9319 ranges {4153344=81920} 2020-06-12T02:38:49.387 INFO:tasks.rados.rados.0.smithi107.stdout:9320: writing smithi10729407-9 from 4153344 to 4235264 tid 1 2020-06-12T02:38:49.388 INFO:tasks.rados.rados.0.smithi107.stdout:9321: delete oid 20 current snap is 0 2020-06-12T02:38:49.389 INFO:tasks.rados.rados.0.smithi107.stdout:9318: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:49.389 INFO:tasks.rados.rados.0.smithi107.stdout:9318: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:49.389 INFO:tasks.rados.rados.0.smithi107.stdout:9318: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:49.389 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1724 (ObjNum 9317 snap 0 seq_num 9317) dirty exists 2020-06-12T02:38:49.389 INFO:tasks.rados.rados.0.smithi107.stdout:9318: left oid 19 (ObjNum 9317 snap 0 seq_num 9317) 2020-06-12T02:38:49.390 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:49.384+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:49.392 INFO:tasks.rados.rados.0.smithi107.stdout:9318: done (2 left) 2020-06-12T02:38:49.392 INFO:tasks.rados.rados.0.smithi107.stdout:9321: done (1 left) 2020-06-12T02:38:49.393 INFO:tasks.rados.rados.0.smithi107.stdout:9322: delete oid 15 current snap is 0 2020-06-12T02:38:49.395 INFO:tasks.rados.rados.0.smithi107.stdout:9320: finishing write tid 1 to smithi10729407-9 2020-06-12T02:38:49.396 INFO:tasks.rados.rados.0.smithi107.stdout:9320: finishing write tid 3 to smithi10729407-9 2020-06-12T02:38:49.396 INFO:tasks.rados.rados.0.smithi107.stdout:9322: done (1 left) 2020-06-12T02:38:49.396 INFO:tasks.rados.rados.0.smithi107.stdout:9323: delete oid 12 current snap is 0 2020-06-12T02:38:49.397 INFO:tasks.rados.rados.0.smithi107.stdout:9320: finishing write tid 4 to smithi10729407-9 2020-06-12T02:38:49.397 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1720 (ObjNum 9319 snap 0 seq_num 9319) dirty exists 2020-06-12T02:38:49.397 INFO:tasks.rados.rados.0.smithi107.stdout:9320: left oid 9 (ObjNum 9319 snap 0 seq_num 9319) 2020-06-12T02:38:49.398 INFO:tasks.rados.rados.0.smithi107.stdout:9320: done (1 left) 2020-06-12T02:38:49.398 INFO:tasks.rados.rados.0.smithi107.stdout:9323: done (0 left) 2020-06-12T02:38:49.399 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:38:49.399 INFO:tasks.rados.rados.0.smithi107.stdout:9324: seq_num 9323 ranges {0=147456} 2020-06-12T02:38:49.399 INFO:tasks.rados.rados.0.smithi107.stdout:9324: writing smithi10729407-20 from 0 to 147456 tid 1 2020-06-12T02:38:49.400 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:49.400 INFO:tasks.rados.rados.0.smithi107.stdout:9325: seq_num 9324 ranges {2318336=163840} 2020-06-12T02:38:49.413 INFO:tasks.rados.rados.0.smithi107.stdout:9325: writing smithi10729407-8 from 2318336 to 2482176 tid 1 2020-06-12T02:38:49.413 INFO:tasks.rados.rados.0.smithi107.stdout:9326: delete oid 14 current snap is 0 2020-06-12T02:38:49.413 INFO:tasks.rados.rados.0.smithi107.stdout:9324: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:49.414 INFO:tasks.rados.rados.0.smithi107.stdout:9324: finishing write tid 3 to smithi10729407-20 2020-06-12T02:38:49.414 INFO:tasks.rados.rados.0.smithi107.stdout:9324: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:49.414 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 692 (ObjNum 9323 snap 0 seq_num 9323) dirty exists 2020-06-12T02:38:49.414 INFO:tasks.rados.rados.0.smithi107.stdout:9324: left oid 20 (ObjNum 9323 snap 0 seq_num 9323) 2020-06-12T02:38:49.416 INFO:tasks.rados.rados.0.smithi107.stdout:9324: done (2 left) 2020-06-12T02:38:49.416 INFO:tasks.rados.rados.0.smithi107.stdout:9326: done (1 left) 2020-06-12T02:38:49.416 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:49.416 INFO:tasks.rados.rados.0.smithi107.stdout:9327: seq_num 9326 ranges {0=196608} 2020-06-12T02:38:49.417 INFO:tasks.rados.rados.0.smithi107.stdout:9327: writing smithi10729407-18 from 0 to 196608 tid 1 2020-06-12T02:38:49.417 INFO:tasks.rados.rados.0.smithi107.stdout:9328: delete oid 17 current snap is 0 2020-06-12T02:38:49.419 INFO:tasks.rados.rados.0.smithi107.stdout:9325: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:49.420 INFO:tasks.rados.rados.0.smithi107.stdout:9325: finishing write tid 3 to smithi10729407-8 2020-06-12T02:38:49.421 INFO:tasks.rados.rados.0.smithi107.stdout:9328: done (2 left) 2020-06-12T02:38:49.421 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:38:49.422 INFO:tasks.rados.rados.0.smithi107.stdout:9329: seq_num 9328 ranges {0=229376} 2020-06-12T02:38:49.422 INFO:tasks.rados.rados.0.smithi107.stdout:9329: writing smithi10729407-11 from 0 to 229376 tid 1 2020-06-12T02:38:49.423 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:38:49.423 INFO:tasks.rados.rados.0.smithi107.stdout:9330: seq_num 9329 ranges {0=245760} 2020-06-12T02:38:49.424 INFO:tasks.rados.rados.0.smithi107.stdout:9330: writing smithi10729407-3 from 0 to 245760 tid 1 2020-06-12T02:38:49.424 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:38:49.424 INFO:tasks.rados.rados.0.smithi107.stdout:9331: seq_num 9330 ranges {0=262144} 2020-06-12T02:38:49.425 INFO:tasks.rados.rados.0.smithi107.stdout:9331: writing smithi10729407-7 from 0 to 262144 tid 1 2020-06-12T02:38:49.425 INFO:tasks.rados.rados.0.smithi107.stdout:9332: delete oid 4 current snap is 0 2020-06-12T02:38:49.426 INFO:tasks.rados.rados.0.smithi107.stdout:9325: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:49.442 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1734 (ObjNum 9324 snap 0 seq_num 9324) dirty exists 2020-06-12T02:38:49.442 INFO:tasks.rados.rados.0.smithi107.stdout:9325: left oid 8 (ObjNum 9324 snap 0 seq_num 9324) 2020-06-12T02:38:49.442 INFO:tasks.rados.rados.0.smithi107.stdout:9327: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:49.442 INFO:tasks.rados.rados.0.smithi107.stdout:9327: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:49.442 INFO:tasks.rados.rados.0.smithi107.stdout:9329: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:49.444 INFO:tasks.rados.rados.0.smithi107.stdout:9327: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:49.445 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1735 (ObjNum 9326 snap 0 seq_num 9326) dirty exists 2020-06-12T02:38:49.445 INFO:tasks.rados.rados.0.smithi107.stdout:9327: left oid 18 (ObjNum 9326 snap 0 seq_num 9326) 2020-06-12T02:38:49.445 INFO:tasks.rados.rados.0.smithi107.stdout:9329: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:49.445 INFO:tasks.rados.rados.0.smithi107.stdout:9329: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:49.445 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1612 (ObjNum 9328 snap 0 seq_num 9328) dirty exists 2020-06-12T02:38:49.446 INFO:tasks.rados.rados.0.smithi107.stdout:9329: left oid 11 (ObjNum 9328 snap 0 seq_num 9328) 2020-06-12T02:38:49.446 INFO:tasks.rados.rados.0.smithi107.stdout:9325: done (5 left) 2020-06-12T02:38:49.446 INFO:tasks.rados.rados.0.smithi107.stdout:9327: done (4 left) 2020-06-12T02:38:49.446 INFO:tasks.rados.rados.0.smithi107.stdout:9329: done (3 left) 2020-06-12T02:38:49.446 INFO:tasks.rados.rados.0.smithi107.stdout:9332: done (2 left) 2020-06-12T02:38:49.446 INFO:tasks.rados.rados.0.smithi107.stdout:9333: delete oid 19 current snap is 0 2020-06-12T02:38:49.447 INFO:tasks.rados.rados.0.smithi107.stdout:9330: finishing write tid 1 to smithi10729407-3 2020-06-12T02:38:49.447 INFO:tasks.rados.rados.0.smithi107.stdout:9331: finishing write tid 1 to smithi10729407-7 2020-06-12T02:38:49.447 INFO:tasks.rados.rados.0.smithi107.stdout:9331: finishing write tid 3 to smithi10729407-7 2020-06-12T02:38:49.447 INFO:tasks.rados.rados.0.smithi107.stdout:9331: finishing write tid 4 to smithi10729407-7 2020-06-12T02:38:49.447 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1640 (ObjNum 9330 snap 0 seq_num 9330) dirty exists 2020-06-12T02:38:49.448 INFO:tasks.rados.rados.0.smithi107.stdout:9331: left oid 7 (ObjNum 9330 snap 0 seq_num 9330) 2020-06-12T02:38:49.448 INFO:tasks.rados.rados.0.smithi107.stdout:9331: done (2 left) 2020-06-12T02:38:49.448 INFO:tasks.rados.rados.0.smithi107.stdout:9333: done (1 left) 2020-06-12T02:38:49.448 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:38:49.448 INFO:tasks.rados.rados.0.smithi107.stdout:9334: seq_num 9333 ranges {196608=311296} 2020-06-12T02:38:49.448 INFO:tasks.rados.rados.0.smithi107.stdout:9334: writing smithi10729407-18 from 196608 to 507904 tid 1 2020-06-12T02:38:49.449 INFO:tasks.rados.rados.0.smithi107.stdout:9335: delete oid 5 current snap is 0 2020-06-12T02:38:49.449 INFO:tasks.rados.rados.0.smithi107.stdout:9330: finishing write tid 3 to smithi10729407-3 2020-06-12T02:38:49.449 INFO:tasks.rados.rados.0.smithi107.stdout:9330: finishing write tid 4 to smithi10729407-3 2020-06-12T02:38:49.449 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1658 (ObjNum 9329 snap 0 seq_num 9329) dirty exists 2020-06-12T02:38:49.449 INFO:tasks.rados.rados.0.smithi107.stdout:9330: left oid 3 (ObjNum 9329 snap 0 seq_num 9329) 2020-06-12T02:38:49.463 INFO:tasks.rados.rados.0.smithi107.stdout:9330: done (2 left) 2020-06-12T02:38:49.464 INFO:tasks.rados.rados.0.smithi107.stdout:9335: done (1 left) 2020-06-12T02:38:49.464 INFO:tasks.rados.rados.0.smithi107.stdout:9336: delete oid 9 current snap is 0 2020-06-12T02:38:49.464 INFO:tasks.rados.rados.0.smithi107.stdout:9334: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:49.464 INFO:tasks.rados.rados.0.smithi107.stdout:9334: finishing write tid 3 to smithi10729407-18 2020-06-12T02:38:49.465 INFO:tasks.rados.rados.0.smithi107.stdout:9334: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:49.465 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1737 (ObjNum 9333 snap 0 seq_num 9333) dirty exists 2020-06-12T02:38:49.465 INFO:tasks.rados.rados.0.smithi107.stdout:9334: left oid 18 (ObjNum 9333 snap 0 seq_num 9333) 2020-06-12T02:38:49.465 INFO:tasks.rados.rados.0.smithi107.stdout:9334: done (1 left) 2020-06-12T02:38:49.465 INFO:tasks.rados.rados.0.smithi107.stdout:9336: done (0 left) 2020-06-12T02:38:49.467 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:38:49.467 INFO:tasks.rados.rados.0.smithi107.stdout:9337: seq_num 9336 ranges {0=368640} 2020-06-12T02:38:49.467 INFO:tasks.rados.rados.0.smithi107.stdout:9337: writing smithi10729407-10 from 0 to 368640 tid 1 2020-06-12T02:38:49.467 INFO:tasks.rados.rados.0.smithi107.stdout:9338: delete oid 14 current snap is 0 2020-06-12T02:38:49.468 INFO:tasks.rados.rados.0.smithi107.stdout:9338: done (1 left) 2020-06-12T02:38:49.468 INFO:tasks.rados.rados.0.smithi107.stdout:9339: delete oid 14 current snap is 0 2020-06-12T02:38:49.468 INFO:tasks.rados.rados.0.smithi107.stdout:9339: done (1 left) 2020-06-12T02:38:49.468 INFO:tasks.rados.rados.0.smithi107.stdout:9340: delete oid 11 current snap is 0 2020-06-12T02:38:49.468 INFO:tasks.rados.rados.0.smithi107.stdout:9337: finishing write tid 1 to smithi10729407-10 2020-06-12T02:38:49.469 INFO:tasks.rados.rados.0.smithi107.stdout:9337: finishing write tid 3 to smithi10729407-10 2020-06-12T02:38:49.469 INFO:tasks.rados.rados.0.smithi107.stdout:9340: done (1 left) 2020-06-12T02:38:49.469 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:38:49.469 INFO:tasks.rados.rados.0.smithi107.stdout:9341: seq_num 9340 ranges {0=434176} 2020-06-12T02:38:49.469 INFO:tasks.rados.rados.0.smithi107.stdout:9341: writing smithi10729407-17 from 0 to 434176 tid 1 2020-06-12T02:38:49.469 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:49.470 INFO:tasks.rados.rados.0.smithi107.stdout:9342: seq_num 9341 ranges {0=450560} 2020-06-12T02:38:49.471 INFO:tasks.rados.rados.0.smithi107.stdout:9342: writing smithi10729407-4 from 0 to 450560 tid 1 2020-06-12T02:38:49.471 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:38:49.472 INFO:tasks.rados.rados.0.smithi107.stdout:9343: seq_num 9342 ranges {0=466944} 2020-06-12T02:38:49.472 INFO:tasks.rados.rados.0.smithi107.stdout:9343: writing smithi10729407-2 from 0 to 466944 tid 1 2020-06-12T02:38:49.472 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:38:49.472 INFO:tasks.rados.rados.0.smithi107.stdout:9344: seq_num 9343 ranges {0=483328} 2020-06-12T02:38:49.472 INFO:tasks.rados.rados.0.smithi107.stdout:9344: writing smithi10729407-14 from 0 to 483328 tid 1 2020-06-12T02:38:49.473 INFO:tasks.rados.rados.0.smithi107.stdout:9345: delete oid 18 current snap is 0 2020-06-12T02:38:49.473 INFO:tasks.rados.rados.0.smithi107.stdout:9337: finishing write tid 4 to smithi10729407-10 2020-06-12T02:38:49.474 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 667 (ObjNum 9336 snap 0 seq_num 9336) dirty exists 2020-06-12T02:38:49.474 INFO:tasks.rados.rados.0.smithi107.stdout:9337: left oid 10 (ObjNum 9336 snap 0 seq_num 9336) 2020-06-12T02:38:49.474 INFO:tasks.rados.rados.0.smithi107.stdout:9341: finishing write tid 1 to smithi10729407-17 2020-06-12T02:38:49.474 INFO:tasks.rados.rados.0.smithi107.stdout:9342: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:49.475 INFO:tasks.rados.rados.0.smithi107.stdout:9342: finishing write tid 3 to smithi10729407-4 2020-06-12T02:38:49.475 INFO:tasks.rados.rados.0.smithi107.stdout:9341: finishing write tid 3 to smithi10729407-17 2020-06-12T02:38:49.476 INFO:tasks.rados.rados.0.smithi107.stdout:9342: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:49.476 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1762 (ObjNum 9341 snap 0 seq_num 9341) dirty exists 2020-06-12T02:38:49.477 INFO:tasks.rados.rados.0.smithi107.stdout:9342: left oid 4 (ObjNum 9341 snap 0 seq_num 9341) 2020-06-12T02:38:49.477 INFO:tasks.rados.rados.0.smithi107.stdout:9341: finishing write tid 4 to smithi10729407-17 2020-06-12T02:38:49.477 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1763 (ObjNum 9340 snap 0 seq_num 9340) dirty exists 2020-06-12T02:38:49.477 INFO:tasks.rados.rados.0.smithi107.stdout:9341: left oid 17 (ObjNum 9340 snap 0 seq_num 9340) 2020-06-12T02:38:49.477 INFO:tasks.rados.rados.0.smithi107.stdout:9343: finishing write tid 1 to smithi10729407-2 2020-06-12T02:38:49.478 INFO:tasks.rados.rados.0.smithi107.stdout:9337: done (5 left) 2020-06-12T02:38:49.478 INFO:tasks.rados.rados.0.smithi107.stdout:9341: done (4 left) 2020-06-12T02:38:49.478 INFO:tasks.rados.rados.0.smithi107.stdout:9342: done (3 left) 2020-06-12T02:38:49.478 INFO:tasks.rados.rados.0.smithi107.stdout:9345: done (2 left) 2020-06-12T02:38:49.478 INFO:tasks.rados.rados.0.smithi107.stdout:9346: delete oid 3 current snap is 0 2020-06-12T02:38:49.479 INFO:tasks.rados.rados.0.smithi107.stdout:9344: finishing write tid 1 to smithi10729407-14 2020-06-12T02:38:49.479 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:49.473+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:49.480 INFO:tasks.rados.rados.0.smithi107.stdout:9343: finishing write tid 3 to smithi10729407-2 2020-06-12T02:38:49.480 INFO:tasks.rados.rados.0.smithi107.stdout:9344: finishing write tid 3 to smithi10729407-14 2020-06-12T02:38:49.480 INFO:tasks.rados.rados.0.smithi107.stdout:9346: done (2 left) 2020-06-12T02:38:49.480 INFO:tasks.rados.rados.0.smithi107.stdout:9347: delete oid 13 current snap is 0 2020-06-12T02:38:49.482 INFO:tasks.rados.rados.0.smithi107.stdout:9343: finishing write tid 4 to smithi10729407-2 2020-06-12T02:38:49.482 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 864 (ObjNum 9342 snap 0 seq_num 9342) dirty exists 2020-06-12T02:38:49.482 INFO:tasks.rados.rados.0.smithi107.stdout:9343: left oid 2 (ObjNum 9342 snap 0 seq_num 9342) 2020-06-12T02:38:49.483 INFO:tasks.rados.rados.0.smithi107.stdout:9344: finishing write tid 4 to smithi10729407-14 2020-06-12T02:38:49.483 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1595 (ObjNum 9343 snap 0 seq_num 9343) dirty exists 2020-06-12T02:38:49.483 INFO:tasks.rados.rados.0.smithi107.stdout:9344: left oid 14 (ObjNum 9343 snap 0 seq_num 9343) 2020-06-12T02:38:49.483 INFO:tasks.rados.rados.0.smithi107.stdout:9343: done (2 left) 2020-06-12T02:38:49.484 INFO:tasks.rados.rados.0.smithi107.stdout:9344: done (1 left) 2020-06-12T02:38:49.484 INFO:tasks.rados.rados.0.smithi107.stdout:9347: done (0 left) 2020-06-12T02:38:49.484 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:38:49.484 INFO:tasks.rados.rados.0.smithi107.stdout:9348: seq_num 9347 ranges {0=548864} 2020-06-12T02:38:49.485 INFO:tasks.rados.rados.0.smithi107.stdout:9348: writing smithi10729407-1 from 0 to 548864 tid 1 2020-06-12T02:38:49.486 INFO:tasks.rados.rados.0.smithi107.stdout:9349: delete oid 9 current snap is 0 2020-06-12T02:38:49.488 INFO:tasks.rados.rados.0.smithi107.stdout:9349: done (1 left) 2020-06-12T02:38:49.488 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:38:49.488 INFO:tasks.rados.rados.0.smithi107.stdout:9350: seq_num 9349 ranges {0=581632} 2020-06-12T02:38:49.492 INFO:tasks.rados.rados.0.smithi107.stdout:9350: writing smithi10729407-19 from 0 to 581632 tid 1 2020-06-12T02:38:49.492 INFO:tasks.rados.rados.0.smithi107.stdout:9351: delete oid 13 current snap is 0 2020-06-12T02:38:49.492 INFO:tasks.rados.rados.0.smithi107.stdout:9348: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:49.496 INFO:tasks.rados.rados.0.smithi107.stdout:9351: done (2 left) 2020-06-12T02:38:49.496 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:38:49.497 INFO:tasks.rados.rados.0.smithi107.stdout:9352: seq_num 9351 ranges {0=614400} 2020-06-12T02:38:49.500 INFO:tasks.rados.rados.0.smithi107.stdout:9352: writing smithi10729407-11 from 0 to 614400 tid 1 2020-06-12T02:38:49.500 INFO:tasks.rados.rados.0.smithi107.stdout:9353: delete oid 8 current snap is 0 2020-06-12T02:38:49.501 INFO:tasks.rados.rados.0.smithi107.stdout:9348: finishing write tid 3 to smithi10729407-1 2020-06-12T02:38:49.501 INFO:tasks.rados.rados.0.smithi107.stdout:9348: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:49.501 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1691 (ObjNum 9347 snap 0 seq_num 9347) dirty exists 2020-06-12T02:38:49.501 INFO:tasks.rados.rados.0.smithi107.stdout:9348: left oid 1 (ObjNum 9347 snap 0 seq_num 9347) 2020-06-12T02:38:49.501 INFO:tasks.rados.rados.0.smithi107.stdout:9350: finishing write tid 1 to smithi10729407-19 2020-06-12T02:38:49.502 INFO:tasks.rados.rados.0.smithi107.stdout:9350: finishing write tid 3 to smithi10729407-19 2020-06-12T02:38:49.504 INFO:tasks.rados.rados.0.smithi107.stdout:9350: finishing write tid 4 to smithi10729407-19 2020-06-12T02:38:49.504 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1728 (ObjNum 9349 snap 0 seq_num 9349) dirty exists 2020-06-12T02:38:49.504 INFO:tasks.rados.rados.0.smithi107.stdout:9350: left oid 19 (ObjNum 9349 snap 0 seq_num 9349) 2020-06-12T02:38:49.505 INFO:tasks.rados.rados.0.smithi107.stdout:9348: done (3 left) 2020-06-12T02:38:49.505 INFO:tasks.rados.rados.0.smithi107.stdout:9350: done (2 left) 2020-06-12T02:38:49.505 INFO:tasks.rados.rados.0.smithi107.stdout:9353: done (1 left) 2020-06-12T02:38:49.505 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:38:49.506 INFO:tasks.rados.rados.0.smithi107.stdout:9354: seq_num 9353 ranges {262144=647168} 2020-06-12T02:38:49.511 INFO:tasks.rados.rados.0.smithi107.stdout:9354: writing smithi10729407-7 from 262144 to 909312 tid 1 2020-06-12T02:38:49.511 INFO:tasks.rados.rados.0.smithi107.stdout:9355: delete oid 13 current snap is 0 2020-06-12T02:38:49.511 INFO:tasks.rados.rados.0.smithi107.stdout:9352: finishing write tid 1 to smithi10729407-11 2020-06-12T02:38:49.526 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:49.526 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:49.526 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.1 to 1 (10000) 2020-06-12T02:38:49.527 INFO:tasks.rados.rados.0.smithi107.stdout:9352: finishing write tid 3 to smithi10729407-11 2020-06-12T02:38:49.531 INFO:tasks.rados.rados.0.smithi107.stdout:9352: finishing write tid 4 to smithi10729407-11 2020-06-12T02:38:49.531 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1615 (ObjNum 9351 snap 0 seq_num 9351) dirty exists 2020-06-12T02:38:49.532 INFO:tasks.rados.rados.0.smithi107.stdout:9352: left oid 11 (ObjNum 9351 snap 0 seq_num 9351) 2020-06-12T02:38:49.534 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd reweight 2 1 2020-06-12T02:38:49.536 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-06-12T02:38:49.583 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:49.580+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:49.653 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:49.654 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:49.662 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-06-12T02:38:49.702 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:49.700+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:49.786 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:49.784+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:49.818 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:49.819 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:49.830 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-06-12T02:38:49.888 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:49.884+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:49.987 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:49.987 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:49.990 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:49.987+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:49.996 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-06-12T02:38:50.091 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:50.088+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:50.153 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:50.154 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:50.162 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-06-12T02:38:50.206 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:50.204+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:50.278 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:50.279 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:50.287 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-06-12T02:38:50.329 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:50.326+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:50.376 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-06-12T02:38:50.381 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.2 to 1 (10000) 2020-06-12T02:38:50.394 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:50.391+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:50.400 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd reweight 3 1 2020-06-12T02:38:50.459 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:50.459 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:50.467 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-06-12T02:38:50.494 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:50.491+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:50.533 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd set nodeep-scrub 2020-06-12T02:38:50.534 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:38:50.556 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:38:50.579 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:38:50.587 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:50.588 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:50.595 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-06-12T02:38:50.596 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:50.594+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:50.599 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:38:50.620 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:38:50.640 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:38:50.662 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:38:50.683 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2020-06-12T02:38:50.684 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-06-12T02:38:50.685 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-06-12T02:38:50.685 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-06-12T02:38:50.685 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-06-12T02:38:50.685 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-06-12T02:38:50.686 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-06-12T02:38:50.686 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-06-12T02:38:50.686 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-06-12T02:38:50.714 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:50.714 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:38:50.715 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:38:50.715 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:38:50.715 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:50.722 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-06-12T02:38:50.736 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:50.733+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:50.802 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:50.795+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:50.840 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:50.841 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:38:50.841 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:38:50.841 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:38:50.842 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.842 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:4975 3.as0 3:5732055a:::smithi10729407-8:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e30)", 2020-06-12T02:38:50.842 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:43.721794+0000", 2020-06-12T02:38:50.842 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 67.114673355999997, 2020-06-12T02:38:50.842 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.1825135, 2020-06-12T02:38:50.843 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:50.843 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:50.843 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:50.843 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:50.844 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:50.844 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 4975 2020-06-12T02:38:50.844 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.844 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:50.844 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.844 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:50.845 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721794+0000", 2020-06-12T02:38:50.845 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:50.845 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.845 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.845 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:50.850 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721794+0000", 2020-06-12T02:38:50.850 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4542e-05 2020-06-12T02:38:50.851 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.851 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.851 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:50.851 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721818+0000", 2020-06-12T02:38:50.851 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9169999999999997e-06 2020-06-12T02:38:50.852 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.852 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.852 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:50.852 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721823+0000", 2020-06-12T02:38:50.852 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1184e-05 2020-06-12T02:38:50.859 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.859 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.859 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:50.860 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721834+0000", 2020-06-12T02:38:50.860 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.254e-06 2020-06-12T02:38:50.860 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.860 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.860 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:50.860 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721840+0000", 2020-06-12T02:38:50.861 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1075999999999998e-05 2020-06-12T02:38:50.861 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.861 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.861 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:50.861 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721881+0000", 2020-06-12T02:38:50.862 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7416999999999995e-05 2020-06-12T02:38:50.862 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.862 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.862 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:50.863 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721948+0000", 2020-06-12T02:38:50.863 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.18235910999999999 2020-06-12T02:38:50.863 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.863 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.863 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:50.863 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.904307+0000", 2020-06-12T02:38:50.864 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.1825135 2020-06-12T02:38:50.864 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:50.865 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:50.865 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:50.865 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.865 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.866 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11914 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e56)", 2020-06-12T02:38:50.867 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:23.508790+0000", 2020-06-12T02:38:50.867 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 27.327676503999999, 2020-06-12T02:38:50.868 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026541928999999999, 2020-06-12T02:38:50.868 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:50.868 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:50.868 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:50.868 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:50.869 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:50.869 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11914 2020-06-12T02:38:50.869 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.869 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:50.869 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.870 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:50.870 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508790+0000", 2020-06-12T02:38:50.870 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:50.870 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.871 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.871 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:50.871 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508790+0000", 2020-06-12T02:38:50.871 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0872999999999999e-05 2020-06-12T02:38:50.871 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.872 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.872 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:50.872 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508841+0000", 2020-06-12T02:38:50.872 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9179999999999994e-06 2020-06-12T02:38:50.872 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.873 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.873 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:50.873 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508849+0000", 2020-06-12T02:38:50.873 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5551999999999999e-05 2020-06-12T02:38:50.873 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.873 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.874 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:50.874 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508865+0000", 2020-06-12T02:38:50.874 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0629999999999996e-06 2020-06-12T02:38:50.874 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.875 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.875 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:50.875 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508870+0000", 2020-06-12T02:38:50.876 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010597488 2020-06-12T02:38:50.876 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.876 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.876 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:50.877 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.519467+0000", 2020-06-12T02:38:50.877 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.6083999999999997e-05 2020-06-12T02:38:50.877 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.877 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.878 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:50.878 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.519563+0000", 2020-06-12T02:38:50.878 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0078077800000000003 2020-06-12T02:38:50.878 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.878 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.878 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:50.879 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.527371+0000", 2020-06-12T02:38:50.879 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6198999999999993e-05 2020-06-12T02:38:50.879 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.879 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.879 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:50.880 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.527447+0000", 2020-06-12T02:38:50.880 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0078849720000000005 2020-06-12T02:38:50.880 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.880 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.880 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:50.881 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.535332+0000", 2020-06-12T02:38:50.881 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026541928999999999 2020-06-12T02:38:50.881 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:50.881 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:50.881 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:50.882 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.882 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.882 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13705 3.as0 3:5cb723d2:::smithi10729407-15:head [append 0~745472,stat] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:38:50.882 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.390499+0000", 2020-06-12T02:38:50.882 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 22.445967810999999, 2020-06-12T02:38:50.882 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.067319320000000002, 2020-06-12T02:38:50.883 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:50.883 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:50.883 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:50.883 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:50.883 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:50.884 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13705 2020-06-12T02:38:50.884 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.884 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:50.884 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.884 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:50.884 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.390499+0000", 2020-06-12T02:38:50.885 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:50.885 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.885 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.885 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:50.885 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.390499+0000", 2020-06-12T02:38:50.886 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00066732599999999999 2020-06-12T02:38:50.886 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.886 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.886 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:50.886 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391167+0000", 2020-06-12T02:38:50.887 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3609999999999997e-06 2020-06-12T02:38:50.887 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.887 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.887 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:50.888 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391173+0000", 2020-06-12T02:38:50.888 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0399e-05 2020-06-12T02:38:50.888 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.888 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.888 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:50.889 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391183+0000", 2020-06-12T02:38:50.890 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6480000000000002e-06 2020-06-12T02:38:50.890 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.890 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.890 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:50.890 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391188+0000", 2020-06-12T02:38:50.891 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4150999999999999e-05 2020-06-12T02:38:50.891 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.891 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.891 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:50.891 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391222+0000", 2020-06-12T02:38:50.892 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9450999999999996e-05 2020-06-12T02:38:50.892 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.892 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.892 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:50.892 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391292+0000", 2020-06-12T02:38:50.892 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054214300000000003 2020-06-12T02:38:50.893 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.893 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.893 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:50.893 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391834+0000", 2020-06-12T02:38:50.893 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0037312069999999998 2020-06-12T02:38:50.894 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.894 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.894 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:50.894 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.395565+0000", 2020-06-12T02:38:50.894 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001050194 2020-06-12T02:38:50.895 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.895 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.895 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:50.895 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396615+0000", 2020-06-12T02:38:50.895 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.061203439999999998 2020-06-12T02:38:50.895 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.896 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.896 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:50.896 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.457819+0000", 2020-06-12T02:38:50.896 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.067319320000000002 2020-06-12T02:38:50.896 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:50.897 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:50.897 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:50.897 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.897 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.897 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13706 3.as0 3:5cb723d2:::smithi10729407-15:head [append 0~434176] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:38:50.898 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.393133+0000", 2020-06-12T02:38:50.898 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 22.443333543000001, 2020-06-12T02:38:50.899 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.078158561000000001, 2020-06-12T02:38:50.900 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:50.900 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:50.900 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:50.901 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:50.901 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:50.901 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13706 2020-06-12T02:38:50.901 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.901 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:50.902 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.902 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:50.902 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393133+0000", 2020-06-12T02:38:50.903 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:50.903 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.903 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.903 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:50.903 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393133+0000", 2020-06-12T02:38:50.904 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00050415499999999995 2020-06-12T02:38:50.904 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.904 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.904 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:50.904 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393638+0000", 2020-06-12T02:38:50.904 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5010000000000004e-06 2020-06-12T02:38:50.905 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.905 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.905 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:50.905 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393646+0000", 2020-06-12T02:38:50.905 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4183e-05 2020-06-12T02:38:50.906 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.906 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.906 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:50.906 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393660+0000", 2020-06-12T02:38:50.906 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6069999999999996e-06 2020-06-12T02:38:50.906 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.907 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.907 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:50.907 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393667+0000", 2020-06-12T02:38:50.907 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001363569 2020-06-12T02:38:50.907 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.908 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.908 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:50.908 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.395030+0000", 2020-06-12T02:38:50.908 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012127 2020-06-12T02:38:50.908 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.908 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.909 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:50.909 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.395152+0000", 2020-06-12T02:38:50.909 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.063640503000000001 2020-06-12T02:38:50.909 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.909 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.910 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:50.910 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458792+0000", 2020-06-12T02:38:50.910 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010636 2020-06-12T02:38:50.910 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.910 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.911 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:50.911 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458899+0000", 2020-06-12T02:38:50.911 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00076074200000000002 2020-06-12T02:38:50.911 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.911 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.911 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:50.912 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459659+0000", 2020-06-12T02:38:50.912 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00772984 2020-06-12T02:38:50.912 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.912 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.912 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:50.913 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.467389+0000", 2020-06-12T02:38:50.913 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003815751 2020-06-12T02:38:50.913 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.913 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.913 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:50.913 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471205+0000", 2020-06-12T02:38:50.914 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.708e-05 2020-06-12T02:38:50.914 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.914 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.914 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:50.914 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471292+0000", 2020-06-12T02:38:50.915 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.078158561000000001 2020-06-12T02:38:50.915 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:50.915 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:50.915 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:50.915 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.915 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.916 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13707 3.as0 3:5cb723d2:::smithi10729407-15:head [append 0~565248] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:38:50.916 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.396173+0000", 2020-06-12T02:38:50.916 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 22.4402936, 2020-06-12T02:38:50.916 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.075298341000000005, 2020-06-12T02:38:50.916 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:50.917 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:50.917 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:50.917 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:50.917 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:50.917 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13707 2020-06-12T02:38:50.917 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.918 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:50.918 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.918 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:50.918 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396173+0000", 2020-06-12T02:38:50.918 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:50.919 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.919 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.919 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:50.919 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396173+0000", 2020-06-12T02:38:50.919 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.061779787000000003 2020-06-12T02:38:50.919 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.920 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.920 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:50.920 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.457953+0000", 2020-06-12T02:38:50.920 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5830000000000007e-06 2020-06-12T02:38:50.920 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.921 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.921 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:50.921 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.457962+0000", 2020-06-12T02:38:50.921 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3029e-05 2020-06-12T02:38:50.921 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.921 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.922 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:50.922 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.457975+0000", 2020-06-12T02:38:50.922 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.233e-06 2020-06-12T02:38:50.922 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.922 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.923 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:50.923 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.457979+0000", 2020-06-12T02:38:50.923 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0044352510000000003 2020-06-12T02:38:50.923 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.923 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.923 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:50.924 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.462414+0000", 2020-06-12T02:38:50.924 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012825799999999999 2020-06-12T02:38:50.924 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.924 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.924 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:50.925 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.462543+0000", 2020-06-12T02:38:50.925 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00077227899999999996 2020-06-12T02:38:50.925 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.925 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.925 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:50.925 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.463315+0000", 2020-06-12T02:38:50.926 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0079912390000000007 2020-06-12T02:38:50.926 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.926 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.926 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:50.926 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471306+0000", 2020-06-12T02:38:50.927 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000118674 2020-06-12T02:38:50.927 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.927 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.927 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:50.927 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471425+0000", 2020-06-12T02:38:50.928 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7008000000000001e-05 2020-06-12T02:38:50.928 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.928 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.928 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:50.928 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471472+0000", 2020-06-12T02:38:50.929 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.075298341000000005 2020-06-12T02:38:50.929 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:50.929 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:50.929 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:50.929 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.930 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.930 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWriteReply(3.as0 58/53 ECSubWriteReply(tid=1890, last_complete=58'1184, committed=1, applied=1))", 2020-06-12T02:38:50.930 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.396464+0000", 2020-06-12T02:38:50.930 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 22.440002720999999, 2020-06-12T02:38:50.930 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.062264803000000001, 2020-06-12T02:38:50.931 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:50.931 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:38:50.931 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:50.931 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.931 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:50.932 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396464+0000", 2020-06-12T02:38:50.932 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:50.932 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.932 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.932 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:50.932 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396464+0000", 2020-06-12T02:38:50.933 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9425999999999999e-05 2020-06-12T02:38:50.933 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.933 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.933 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:50.933 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396494+0000", 2020-06-12T02:38:50.934 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1610000000000002e-06 2020-06-12T02:38:50.934 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.934 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.934 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:50.934 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396499+0000", 2020-06-12T02:38:50.935 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2789999999999998e-06 2020-06-12T02:38:50.935 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.935 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.935 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:50.935 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396507+0000", 2020-06-12T02:38:50.935 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1389999999999997e-06 2020-06-12T02:38:50.936 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.936 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.936 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:50.936 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396511+0000", 2020-06-12T02:38:50.936 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2199000000000001e-05 2020-06-12T02:38:50.937 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.937 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.937 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:50.937 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396543+0000", 2020-06-12T02:38:50.937 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.062185599000000001 2020-06-12T02:38:50.937 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.938 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.938 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:50.938 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458729+0000", 2020-06-12T02:38:50.938 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.062264803000000001 2020-06-12T02:38:50.938 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:50.939 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:50.939 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:50.939 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.939 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.939 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13710 3.as0 3:5cb723d2:::smithi10729407-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e58)", 2020-06-12T02:38:50.940 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.458457+0000", 2020-06-12T02:38:50.940 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 22.378009551000002, 2020-06-12T02:38:50.940 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028102095000000001, 2020-06-12T02:38:50.940 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:50.940 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:50.941 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:50.941 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:50.941 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:50.941 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13710 2020-06-12T02:38:50.941 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.942 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:50.942 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.942 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:50.942 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458457+0000", 2020-06-12T02:38:50.942 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:50.942 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.943 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.943 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:50.943 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458457+0000", 2020-06-12T02:38:50.943 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7650000000000001e-05 2020-06-12T02:38:50.943 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.944 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.944 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:50.944 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458485+0000", 2020-06-12T02:38:50.944 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3210000000000001e-06 2020-06-12T02:38:50.944 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.944 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.945 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:50.945 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458488+0000", 2020-06-12T02:38:50.945 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5920000000000003e-06 2020-06-12T02:38:50.945 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.945 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.946 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:50.946 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458496+0000", 2020-06-12T02:38:50.946 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8789999999999999e-06 2020-06-12T02:38:50.946 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.946 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.947 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:50.947 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458499+0000", 2020-06-12T02:38:50.947 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013081152 2020-06-12T02:38:50.947 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.947 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.947 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:50.948 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471580+0000", 2020-06-12T02:38:50.948 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4975999999999994e-05 2020-06-12T02:38:50.948 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.948 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.948 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:50.948 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471665+0000", 2020-06-12T02:38:50.949 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0054254029999999996 2020-06-12T02:38:50.949 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.949 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.949 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:50.949 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.477090+0000", 2020-06-12T02:38:50.950 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2335000000000005e-05 2020-06-12T02:38:50.950 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.950 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.950 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:50.950 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.477163+0000", 2020-06-12T02:38:50.951 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0093967870000000002 2020-06-12T02:38:50.951 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.951 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.951 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:50.951 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486560+0000", 2020-06-12T02:38:50.951 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028102095000000001 2020-06-12T02:38:50.952 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:50.952 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:50.952 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:50.952 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.952 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.953 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13711 3.as0 3:5732055a:::smithi10729407-8:head [append 0~761856] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:38:50.953 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.458525+0000", 2020-06-12T02:38:50.953 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 22.377941779, 2020-06-12T02:38:50.953 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.027773760000000002, 2020-06-12T02:38:50.953 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:50.954 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:50.954 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:50.955 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:50.955 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:50.955 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13711 2020-06-12T02:38:50.955 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.955 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:50.956 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.956 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:50.956 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458525+0000", 2020-06-12T02:38:50.956 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:50.956 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.957 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.957 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:50.957 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458525+0000", 2020-06-12T02:38:50.957 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00089965200000000003 2020-06-12T02:38:50.957 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.958 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.958 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:50.958 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459425+0000", 2020-06-12T02:38:50.958 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4749999999999996e-06 2020-06-12T02:38:50.958 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.958 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.959 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:50.959 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459432+0000", 2020-06-12T02:38:50.959 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3043e-05 2020-06-12T02:38:50.959 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.959 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.960 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:50.960 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459445+0000", 2020-06-12T02:38:50.960 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0739999999999998e-06 2020-06-12T02:38:50.960 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.960 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.961 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:50.961 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459451+0000", 2020-06-12T02:38:50.961 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012238733999999999 2020-06-12T02:38:50.961 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.961 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.961 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:50.962 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471690+0000", 2020-06-12T02:38:50.962 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5640000000000003e-05 2020-06-12T02:38:50.962 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.962 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.962 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:50.963 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471746+0000", 2020-06-12T02:38:50.963 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00067859699999999997 2020-06-12T02:38:50.963 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.963 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.963 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:50.964 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.472424+0000", 2020-06-12T02:38:50.964 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050083230000000003 2020-06-12T02:38:50.964 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.964 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.965 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:50.965 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.477433+0000", 2020-06-12T02:38:50.965 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0087878039999999998 2020-06-12T02:38:50.965 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.965 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.966 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:50.966 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486221+0000", 2020-06-12T02:38:50.966 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8417999999999994e-05 2020-06-12T02:38:50.966 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.966 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.967 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:50.967 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486299+0000", 2020-06-12T02:38:50.967 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.027773760000000002 2020-06-12T02:38:50.967 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:50.967 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:50.968 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:50.968 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.968 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.968 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13712 3.as0 3:5732055a:::smithi10729407-8:head [append 0~712704] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:38:50.968 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.459508+0000", 2020-06-12T02:38:50.968 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 22.376958499000001, 2020-06-12T02:38:50.969 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.033995818999999997, 2020-06-12T02:38:50.969 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:50.969 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:50.969 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:50.969 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:50.970 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:50.970 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13712 2020-06-12T02:38:50.970 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.970 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:50.971 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.971 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:50.971 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459508+0000", 2020-06-12T02:38:50.971 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:50.971 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.971 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.972 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:50.972 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459508+0000", 2020-06-12T02:38:50.972 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00075436999999999998 2020-06-12T02:38:50.972 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.972 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.973 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:50.973 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460263+0000", 2020-06-12T02:38:50.973 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4370000000000001e-06 2020-06-12T02:38:50.973 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.973 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.974 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:50.974 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460270+0000", 2020-06-12T02:38:50.974 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2500000000000001e-05 2020-06-12T02:38:50.974 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.975 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.975 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:50.975 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460283+0000", 2020-06-12T02:38:50.975 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2580000000000003e-06 2020-06-12T02:38:50.975 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.976 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.976 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:50.976 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460288+0000", 2020-06-12T02:38:50.976 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017177595 2020-06-12T02:38:50.976 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.976 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.977 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:50.977 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.477466+0000", 2020-06-12T02:38:50.977 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00019751600000000001 2020-06-12T02:38:50.977 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.977 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.978 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:50.978 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.477663+0000", 2020-06-12T02:38:50.978 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00088697400000000001 2020-06-12T02:38:50.978 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.978 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.979 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:50.979 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.478550+0000", 2020-06-12T02:38:50.979 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.007578787 2020-06-12T02:38:50.979 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.979 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.979 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:50.980 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486129+0000", 2020-06-12T02:38:50.980 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0072972310000000004 2020-06-12T02:38:50.980 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.980 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.980 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:50.981 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493426+0000", 2020-06-12T02:38:50.981 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8151000000000004e-05 2020-06-12T02:38:50.981 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.981 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.981 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:50.982 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493504+0000", 2020-06-12T02:38:50.982 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.033995818999999997 2020-06-12T02:38:50.982 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:50.982 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:50.982 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:50.982 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.983 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.983 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13713 3.as0 3:5732055a:::smithi10729407-8:head [append 0~491520] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:38:50.983 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.460333+0000", 2020-06-12T02:38:50.983 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 22.376134043, 2020-06-12T02:38:50.983 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.033390145000000003, 2020-06-12T02:38:50.984 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:50.984 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:50.984 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:50.984 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:50.984 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:50.985 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13713 2020-06-12T02:38:50.985 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.985 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:50.985 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.986 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:50.986 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460333+0000", 2020-06-12T02:38:50.986 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:50.986 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.986 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.987 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:50.987 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460333+0000", 2020-06-12T02:38:50.987 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00051168999999999997 2020-06-12T02:38:50.987 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.987 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.988 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:50.988 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460845+0000", 2020-06-12T02:38:50.988 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2930000000000003e-06 2020-06-12T02:38:50.988 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.988 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.988 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:50.989 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460852+0000", 2020-06-12T02:38:50.989 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3054000000000001e-05 2020-06-12T02:38:50.989 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.989 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.989 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:50.990 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460865+0000", 2020-06-12T02:38:50.990 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6130000000000003e-06 2020-06-12T02:38:50.990 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.990 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.990 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:50.991 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460871+0000", 2020-06-12T02:38:50.991 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025753870000000002 2020-06-12T02:38:50.991 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.991 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.991 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:50.991 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486624+0000", 2020-06-12T02:38:50.992 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2739999999999998e-05 2020-06-12T02:38:50.992 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.992 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.992 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:50.992 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486697+0000", 2020-06-12T02:38:50.993 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00063578799999999998 2020-06-12T02:38:50.993 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.993 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.993 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:50.993 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.487333+0000", 2020-06-12T02:38:50.994 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0043011610000000004 2020-06-12T02:38:50.994 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.994 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.994 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:50.994 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.491634+0000", 2020-06-12T02:38:50.994 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0020130949999999999 2020-06-12T02:38:50.995 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.995 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.997 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:50.998 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493647+0000", 2020-06-12T02:38:50.998 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5840999999999994e-05 2020-06-12T02:38:50.998 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:50.998 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:50.998 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:50.999 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493723+0000", 2020-06-12T02:38:50.999 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.033390145000000003 2020-06-12T02:38:50.999 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:50.999 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:50.999 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.000 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.000 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.000 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13714 3.as0 3:5732055a:::smithi10729407-8:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:38:51.000 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.460921+0000", 2020-06-12T02:38:51.000 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 22.375545520999999, 2020-06-12T02:38:51.001 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.032981457999999998, 2020-06-12T02:38:51.001 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:51.001 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:51.001 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:51.001 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:51.001 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:51.002 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13714 2020-06-12T02:38:51.002 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.002 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:51.002 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.002 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:51.003 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460921+0000", 2020-06-12T02:38:51.003 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:51.003 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.003 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.003 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:51.003 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460921+0000", 2020-06-12T02:38:51.004 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6996999999999995e-05 2020-06-12T02:38:51.004 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.004 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.004 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:51.004 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460998+0000", 2020-06-12T02:38:51.005 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4529999999999999e-06 2020-06-12T02:38:51.005 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.005 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.005 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:51.005 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461005+0000", 2020-06-12T02:38:51.006 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1722000000000001e-05 2020-06-12T02:38:51.006 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.006 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.006 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:51.006 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461017+0000", 2020-06-12T02:38:51.006 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4839999999999998e-06 2020-06-12T02:38:51.007 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.007 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.007 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:51.007 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461021+0000", 2020-06-12T02:38:51.008 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.029157975999999999 2020-06-12T02:38:51.008 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.008 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.008 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.008 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.490179+0000", 2020-06-12T02:38:51.009 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010966700000000001 2020-06-12T02:38:51.009 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.009 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.009 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:51.009 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.490289+0000", 2020-06-12T02:38:51.010 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00032026800000000002 2020-06-12T02:38:51.010 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.010 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.010 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:51.010 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.490609+0000", 2020-06-12T02:38:51.011 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001102947 2020-06-12T02:38:51.011 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.011 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.011 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:51.011 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.491712+0000", 2020-06-12T02:38:51.011 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021050169999999998 2020-06-12T02:38:51.012 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.012 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.012 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:51.012 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493817+0000", 2020-06-12T02:38:51.012 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5927000000000006e-05 2020-06-12T02:38:51.013 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.013 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.013 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:51.013 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493903+0000", 2020-06-12T02:38:51.013 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.032981457999999998 2020-06-12T02:38:51.013 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.014 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:51.014 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.014 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.014 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.014 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13715 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e58)", 2020-06-12T02:38:51.015 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.461062+0000", 2020-06-12T02:38:51.015 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 22.375404881000001, 2020-06-12T02:38:51.015 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.035208338999999998, 2020-06-12T02:38:51.015 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:51.015 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:51.016 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:51.016 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:51.016 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:51.016 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13715 2020-06-12T02:38:51.016 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.016 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:51.017 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.017 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:51.017 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461062+0000", 2020-06-12T02:38:51.017 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:51.017 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.018 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.018 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:51.018 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461062+0000", 2020-06-12T02:38:51.018 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4839999999999998e-05 2020-06-12T02:38:51.018 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.019 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.019 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:51.019 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461107+0000", 2020-06-12T02:38:51.019 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4859999999999997e-06 2020-06-12T02:38:51.019 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.020 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.020 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:51.020 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461112+0000", 2020-06-12T02:38:51.020 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1742999999999999e-05 2020-06-12T02:38:51.020 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.020 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.021 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:51.021 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461124+0000", 2020-06-12T02:38:51.021 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7299999999999996e-06 2020-06-12T02:38:51.021 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.021 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.022 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:51.022 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461129+0000", 2020-06-12T02:38:51.022 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.030397368000000001 2020-06-12T02:38:51.022 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.022 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.023 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.023 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.491526+0000", 2020-06-12T02:38:51.023 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010076799999999999 2020-06-12T02:38:51.023 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.023 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.023 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:51.024 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.491627+0000", 2020-06-12T02:38:51.024 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031869849999999998 2020-06-12T02:38:51.024 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.024 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.024 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.025 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.494814+0000", 2020-06-12T02:38:51.025 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000108915 2020-06-12T02:38:51.025 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.025 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.025 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:51.025 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.494923+0000", 2020-06-12T02:38:51.026 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001347504 2020-06-12T02:38:51.026 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.026 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.026 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:51.026 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.496270+0000", 2020-06-12T02:38:51.027 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.035208338999999998 2020-06-12T02:38:51.027 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.027 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:51.028 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.030 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.030 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.030 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:16706 3.as0 3:5cb723d2:::smithi10729407-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e61)", 2020-06-12T02:38:51.030 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:37.289063+0000", 2020-06-12T02:38:51.031 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 13.547403857000001, 2020-06-12T02:38:51.031 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026146086999999998, 2020-06-12T02:38:51.031 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:51.031 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:51.031 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:51.032 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:51.032 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:51.032 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 16706 2020-06-12T02:38:51.032 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.032 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:51.033 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.033 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:51.033 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289063+0000", 2020-06-12T02:38:51.033 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:51.033 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.033 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.034 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:51.034 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289063+0000", 2020-06-12T02:38:51.034 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3776000000000002e-05 2020-06-12T02:38:51.034 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.034 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.035 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:51.035 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289107+0000", 2020-06-12T02:38:51.035 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7039999999999999e-06 2020-06-12T02:38:51.035 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.035 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.035 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:51.036 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289113+0000", 2020-06-12T02:38:51.036 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3259e-05 2020-06-12T02:38:51.036 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.036 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.036 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:51.037 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289126+0000", 2020-06-12T02:38:51.037 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1999999999999996e-06 2020-06-12T02:38:51.037 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.037 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.037 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:51.038 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289130+0000", 2020-06-12T02:38:51.044 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0010912820000000001 2020-06-12T02:38:51.045 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.045 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.045 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.045 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.290221+0000", 2020-06-12T02:38:51.046 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3166e-05 2020-06-12T02:38:51.046 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.046 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.046 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:51.046 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.290265+0000", 2020-06-12T02:38:51.047 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015984187 2020-06-12T02:38:51.047 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.047 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.047 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.047 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.306249+0000", 2020-06-12T02:38:51.048 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4623000000000006e-05 2020-06-12T02:38:51.048 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.048 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.048 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:51.049 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.306333+0000", 2020-06-12T02:38:51.049 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0068573690000000003 2020-06-12T02:38:51.049 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.049 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.049 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.050 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.313191+0000", 2020-06-12T02:38:51.050 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7804000000000002e-05 2020-06-12T02:38:51.050 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.050 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.050 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:51.051 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.313279+0000", 2020-06-12T02:38:51.052 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001930717 2020-06-12T02:38:51.052 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.052 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.052 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:51.053 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.315209+0000", 2020-06-12T02:38:51.053 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026146086999999998 2020-06-12T02:38:51.053 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.053 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:51.053 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.054 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.054 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.054 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18051 3.as0 3:5cb723d2:::smithi10729407-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:38:51.054 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.329078+0000", 2020-06-12T02:38:51.054 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.5073893070000004, 2020-06-12T02:38:51.055 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026465103, 2020-06-12T02:38:51.055 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:51.055 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:51.056 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:51.056 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:51.056 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:51.056 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18051 2020-06-12T02:38:51.056 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.057 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:51.057 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.057 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:51.057 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329078+0000", 2020-06-12T02:38:51.058 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:51.058 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.058 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.058 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:51.058 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329078+0000", 2020-06-12T02:38:51.059 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5392000000000001e-05 2020-06-12T02:38:51.059 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.059 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.059 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:51.059 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329133+0000", 2020-06-12T02:38:51.060 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5329999999999997e-06 2020-06-12T02:38:51.060 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.060 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.060 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:51.060 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329141+0000", 2020-06-12T02:38:51.060 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3459999999999999e-05 2020-06-12T02:38:51.061 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.061 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.061 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:51.061 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329164+0000", 2020-06-12T02:38:51.062 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8170000000000001e-06 2020-06-12T02:38:51.062 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.062 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.062 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:51.062 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329171+0000", 2020-06-12T02:38:51.063 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013005208000000001 2020-06-12T02:38:51.063 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.063 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.063 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.063 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.342176+0000", 2020-06-12T02:38:51.064 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3560999999999998e-05 2020-06-12T02:38:51.064 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.064 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.064 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:51.064 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.342230+0000", 2020-06-12T02:38:51.064 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0080818490000000003 2020-06-12T02:38:51.065 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.065 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.065 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.066 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.350312+0000", 2020-06-12T02:38:51.066 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0167000000000006e-05 2020-06-12T02:38:51.066 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.066 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.067 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:51.067 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.350402+0000", 2020-06-12T02:38:51.067 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051411160000000003 2020-06-12T02:38:51.067 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.067 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.068 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:51.068 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.355543+0000", 2020-06-12T02:38:51.068 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026465103 2020-06-12T02:38:51.068 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.068 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:51.069 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.069 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.069 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.069 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18053 3.as0 3:5732055a:::smithi10729407-8:head [append 0~434176] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:38:51.069 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.334425+0000", 2020-06-12T02:38:51.070 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.5020417399999992, 2020-06-12T02:38:51.070 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.029616703000000001, 2020-06-12T02:38:51.070 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:51.070 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:51.070 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:51.070 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:51.071 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:51.071 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18053 2020-06-12T02:38:51.071 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.071 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:51.071 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.072 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:51.072 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334425+0000", 2020-06-12T02:38:51.072 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:51.072 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.072 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.073 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:51.073 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334425+0000", 2020-06-12T02:38:51.073 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047283900000000002 2020-06-12T02:38:51.073 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.074 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.074 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:51.074 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334898+0000", 2020-06-12T02:38:51.074 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9100000000000004e-06 2020-06-12T02:38:51.074 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.075 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.075 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:51.075 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334903+0000", 2020-06-12T02:38:51.075 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4260000000000003e-06 2020-06-12T02:38:51.075 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.076 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.078 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:51.079 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334911+0000", 2020-06-12T02:38:51.079 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1060000000000002e-06 2020-06-12T02:38:51.079 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.079 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.079 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:51.080 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334916+0000", 2020-06-12T02:38:51.080 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010838316000000001 2020-06-12T02:38:51.080 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.080 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.080 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.080 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.345754+0000", 2020-06-12T02:38:51.081 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000103872 2020-06-12T02:38:51.081 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.081 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.081 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:51.081 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.345858+0000", 2020-06-12T02:38:51.082 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0059798739999999996 2020-06-12T02:38:51.082 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.082 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.082 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.082 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.351838+0000", 2020-06-12T02:38:51.083 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000104724 2020-06-12T02:38:51.083 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.083 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.083 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:51.083 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.351942+0000", 2020-06-12T02:38:51.084 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000572687 2020-06-12T02:38:51.084 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.084 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.084 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:51.084 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.352515+0000", 2020-06-12T02:38:51.085 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002905641 2020-06-12T02:38:51.085 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.085 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.085 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:51.085 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.355421+0000", 2020-06-12T02:38:51.086 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0085358350000000003 2020-06-12T02:38:51.086 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.086 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.087 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:51.087 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.363956+0000", 2020-06-12T02:38:51.087 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5473e-05 2020-06-12T02:38:51.087 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.087 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.088 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:51.088 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.364042+0000", 2020-06-12T02:38:51.088 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.029616703000000001 2020-06-12T02:38:51.088 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.088 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:51.089 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.089 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.089 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.090 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18054 3.as0 3:5732055a:::smithi10729407-8:head [append 0~638976] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:38:51.090 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.338171+0000", 2020-06-12T02:38:51.090 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.4982962369999999, 2020-06-12T02:38:51.090 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.030067120999999999, 2020-06-12T02:38:51.090 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:51.091 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:51.091 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:51.091 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:51.091 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:51.091 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18054 2020-06-12T02:38:51.092 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.092 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:51.092 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.092 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:51.092 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338171+0000", 2020-06-12T02:38:51.093 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:51.093 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.093 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.093 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:51.093 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338171+0000", 2020-06-12T02:38:51.094 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00068138900000000002 2020-06-12T02:38:51.094 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.094 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.094 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:51.094 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338852+0000", 2020-06-12T02:38:51.095 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.767e-06 2020-06-12T02:38:51.095 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.095 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.095 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:51.095 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338858+0000", 2020-06-12T02:38:51.095 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8999999999999995e-06 2020-06-12T02:38:51.096 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.096 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.096 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:51.096 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338867+0000", 2020-06-12T02:38:51.097 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.262e-06 2020-06-12T02:38:51.097 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.097 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.097 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:51.098 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338871+0000", 2020-06-12T02:38:51.098 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0074839060000000002 2020-06-12T02:38:51.098 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.098 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.099 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.099 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346355+0000", 2020-06-12T02:38:51.099 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1435999999999999e-05 2020-06-12T02:38:51.099 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.099 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.100 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:51.100 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346416+0000", 2020-06-12T02:38:51.100 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0092109879999999998 2020-06-12T02:38:51.100 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.100 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.101 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.101 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.355627+0000", 2020-06-12T02:38:51.101 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7355999999999998e-05 2020-06-12T02:38:51.101 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.102 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.102 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:51.102 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.355685+0000", 2020-06-12T02:38:51.102 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00074711999999999999 2020-06-12T02:38:51.103 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.103 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.103 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:51.103 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.356432+0000", 2020-06-12T02:38:51.103 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010659572000000001 2020-06-12T02:38:51.104 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.104 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.104 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:51.104 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.367091+0000", 2020-06-12T02:38:51.104 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0010713210000000001 2020-06-12T02:38:51.104 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.105 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.105 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:51.105 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.368163+0000", 2020-06-12T02:38:51.105 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5104000000000005e-05 2020-06-12T02:38:51.105 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.106 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.106 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:51.106 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.368238+0000", 2020-06-12T02:38:51.106 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.030067120999999999 2020-06-12T02:38:51.106 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.107 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:51.107 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.107 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.108 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.108 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18055 3.as0 3:5732055a:::smithi10729407-8:head [append 0~483328] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:38:51.108 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.340740+0000", 2020-06-12T02:38:51.108 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.4957273880000006, 2020-06-12T02:38:51.108 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028669050000000001, 2020-06-12T02:38:51.109 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:51.109 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:51.109 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:51.109 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:51.109 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:51.109 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18055 2020-06-12T02:38:51.110 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.110 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:51.110 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.110 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:51.110 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.340740+0000", 2020-06-12T02:38:51.111 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:51.111 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.111 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.111 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:51.111 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.340740+0000", 2020-06-12T02:38:51.111 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00045126900000000002 2020-06-12T02:38:51.112 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.112 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.112 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:51.112 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341191+0000", 2020-06-12T02:38:51.112 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5630000000000003e-06 2020-06-12T02:38:51.113 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.113 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.113 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:51.113 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341198+0000", 2020-06-12T02:38:51.113 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4467999999999999e-05 2020-06-12T02:38:51.114 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.114 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.114 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:51.114 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341213+0000", 2020-06-12T02:38:51.114 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7540000000000003e-06 2020-06-12T02:38:51.115 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.115 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.115 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:51.116 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341219+0000", 2020-06-12T02:38:51.116 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0052272680000000002 2020-06-12T02:38:51.117 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.118 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.118 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.118 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346446+0000", 2020-06-12T02:38:51.118 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4742999999999998e-05 2020-06-12T02:38:51.118 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.119 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.119 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:51.119 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346531+0000", 2020-06-12T02:38:51.119 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013203138999999999 2020-06-12T02:38:51.119 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.120 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.120 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.120 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.359734+0000", 2020-06-12T02:38:51.120 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000100277 2020-06-12T02:38:51.120 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.121 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.121 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:51.121 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.359834+0000", 2020-06-12T02:38:51.121 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054998099999999995 2020-06-12T02:38:51.121 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.121 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.122 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:51.122 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.360384+0000", 2020-06-12T02:38:51.122 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0067087179999999998 2020-06-12T02:38:51.122 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.122 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.123 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:51.123 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.367093+0000", 2020-06-12T02:38:51.123 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0022746020000000001 2020-06-12T02:38:51.123 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.123 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.124 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:51.124 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369367+0000", 2020-06-12T02:38:51.124 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1267999999999999e-05 2020-06-12T02:38:51.124 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.124 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.124 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:51.125 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369409+0000", 2020-06-12T02:38:51.125 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028669050000000001 2020-06-12T02:38:51.126 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.126 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:51.126 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.126 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.126 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.127 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18056 3.as0 3:5732055a:::smithi10729407-8:head [append 0~49152] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:38:51.127 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.341270+0000", 2020-06-12T02:38:51.127 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.4951965989999998, 2020-06-12T02:38:51.127 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028350462999999999, 2020-06-12T02:38:51.127 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:51.129 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:51.129 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:51.129 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:51.129 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:51.129 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18056 2020-06-12T02:38:51.130 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.130 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:51.130 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.130 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:51.130 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341270+0000", 2020-06-12T02:38:51.130 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:51.131 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.131 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.131 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:51.131 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341270+0000", 2020-06-12T02:38:51.131 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000114082 2020-06-12T02:38:51.132 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.132 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.132 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:51.132 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341384+0000", 2020-06-12T02:38:51.132 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3289999999999999e-06 2020-06-12T02:38:51.133 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.133 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.133 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:51.133 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341391+0000", 2020-06-12T02:38:51.133 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3339999999999999e-05 2020-06-12T02:38:51.133 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.134 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.134 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:51.134 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341404+0000", 2020-06-12T02:38:51.134 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4e-06 2020-06-12T02:38:51.134 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.135 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.135 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:51.135 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341410+0000", 2020-06-12T02:38:51.135 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051518600000000003 2020-06-12T02:38:51.136 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.136 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.136 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.136 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346561+0000", 2020-06-12T02:38:51.136 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1662999999999994e-05 2020-06-12T02:38:51.137 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.137 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.137 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:51.137 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346643+0000", 2020-06-12T02:38:51.137 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017471605000000001 2020-06-12T02:38:51.137 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.138 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.138 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.138 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.364115+0000", 2020-06-12T02:38:51.138 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011059400000000001 2020-06-12T02:38:51.138 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.139 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.139 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:51.139 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.364225+0000", 2020-06-12T02:38:51.139 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000611262 2020-06-12T02:38:51.139 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.139 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.140 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:51.140 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.364837+0000", 2020-06-12T02:38:51.140 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023767710000000002 2020-06-12T02:38:51.140 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.140 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.141 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:51.141 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.367213+0000", 2020-06-12T02:38:51.141 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023325479999999998 2020-06-12T02:38:51.141 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.141 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.142 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:51.142 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369546+0000", 2020-06-12T02:38:51.142 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5009e-05 2020-06-12T02:38:51.142 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.142 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.142 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:51.143 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369621+0000", 2020-06-12T02:38:51.143 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028350462999999999 2020-06-12T02:38:51.143 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.143 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:51.143 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.144 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.144 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.144 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18057 3.as0 3:5732055a:::smithi10729407-8:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:38:51.144 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.341456+0000", 2020-06-12T02:38:51.144 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.4950114390000007, 2020-06-12T02:38:51.145 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028443018, 2020-06-12T02:38:51.145 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:51.145 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:51.145 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:51.145 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:51.145 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:51.146 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18057 2020-06-12T02:38:51.146 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.146 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:51.146 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.146 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:51.147 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341456+0000", 2020-06-12T02:38:51.147 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:51.147 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.147 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.147 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:51.147 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341456+0000", 2020-06-12T02:38:51.148 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6036000000000003e-05 2020-06-12T02:38:51.148 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.148 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.148 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:51.148 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341522+0000", 2020-06-12T02:38:51.149 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2210000000000002e-06 2020-06-12T02:38:51.149 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.149 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.149 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:51.149 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341528+0000", 2020-06-12T02:38:51.150 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0338e-05 2020-06-12T02:38:51.150 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.150 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.150 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:51.150 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341538+0000", 2020-06-12T02:38:51.150 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8639999999999998e-06 2020-06-12T02:38:51.151 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.151 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.151 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:51.151 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341542+0000", 2020-06-12T02:38:51.151 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051405009999999996 2020-06-12T02:38:51.152 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.152 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.152 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.152 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346683+0000", 2020-06-12T02:38:51.152 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9561999999999995e-05 2020-06-12T02:38:51.153 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.153 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.153 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:51.153 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346762+0000", 2020-06-12T02:38:51.154 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019267197999999999 2020-06-12T02:38:51.154 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.154 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.154 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.154 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.366029+0000", 2020-06-12T02:38:51.155 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2634999999999997e-05 2020-06-12T02:38:51.155 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.155 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.155 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:51.156 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.366122+0000", 2020-06-12T02:38:51.156 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00026907500000000002 2020-06-12T02:38:51.156 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.156 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.157 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:51.157 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.366391+0000", 2020-06-12T02:38:51.157 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0018939409999999999 2020-06-12T02:38:51.157 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.157 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.158 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:51.158 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.368285+0000", 2020-06-12T02:38:51.158 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001549405 2020-06-12T02:38:51.158 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.158 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.159 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:51.159 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369834+0000", 2020-06-12T02:38:51.159 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4241999999999999e-05 2020-06-12T02:38:51.159 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.159 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.160 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:51.160 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369899+0000", 2020-06-12T02:38:51.160 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028443018 2020-06-12T02:38:51.160 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.160 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:51.160 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.161 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.161 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.161 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18058 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:38:51.161 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.341580+0000", 2020-06-12T02:38:51.162 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.4948865320000007, 2020-06-12T02:38:51.162 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.030449559000000001, 2020-06-12T02:38:51.162 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:51.162 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:51.162 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:51.163 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:51.163 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:51.163 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18058 2020-06-12T02:38:51.163 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.163 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:51.164 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.164 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:51.164 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341580+0000", 2020-06-12T02:38:51.164 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:51.164 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.164 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.165 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:51.165 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341580+0000", 2020-06-12T02:38:51.165 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1185e-05 2020-06-12T02:38:51.165 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.165 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.166 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:51.166 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341622+0000", 2020-06-12T02:38:51.166 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1599999999999997e-06 2020-06-12T02:38:51.166 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.166 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.166 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:51.167 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341627+0000", 2020-06-12T02:38:51.167 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3546e-05 2020-06-12T02:38:51.167 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.167 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.167 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:51.168 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341640+0000", 2020-06-12T02:38:51.168 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9779999999999999e-06 2020-06-12T02:38:51.168 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.168 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.168 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:51.168 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341644+0000", 2020-06-12T02:38:51.169 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051433700000000004 2020-06-12T02:38:51.169 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.169 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.169 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.169 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346788+0000", 2020-06-12T02:38:51.170 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1622000000000001e-05 2020-06-12T02:38:51.170 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.170 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.170 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:51.170 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346839+0000", 2020-06-12T02:38:51.171 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020283958000000001 2020-06-12T02:38:51.171 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.171 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.171 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.171 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.367123+0000", 2020-06-12T02:38:51.171 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1997999999999997e-05 2020-06-12T02:38:51.172 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.172 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.172 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:51.172 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.367205+0000", 2020-06-12T02:38:51.172 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036875440000000001 2020-06-12T02:38:51.173 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.173 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.173 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.173 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.370893+0000", 2020-06-12T02:38:51.174 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000100604 2020-06-12T02:38:51.174 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.174 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.174 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:51.174 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.370993+0000", 2020-06-12T02:38:51.175 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0010365940000000001 2020-06-12T02:38:51.175 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.175 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.175 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:51.175 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.372030+0000", 2020-06-12T02:38:51.175 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.030449559000000001 2020-06-12T02:38:51.176 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.176 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:51.176 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.176 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.176 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:51.177 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:51.178 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:51.175+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:51.179 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-06-12T02:38:51.279 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:51.276+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:51.337 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:51.337 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:38:51.337 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2020-06-12T02:38:51.337 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:51.345 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-06-12T02:38:51.380 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:51.377+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:51.397 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2020-06-12T02:38:51.408 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.3 to 1 (10000) 2020-06-12T02:38:51.424 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd reweight 4 1 2020-06-12T02:38:51.426 INFO:tasks.rados.rados.0.smithi107.stdout:9352: done (2 left) 2020-06-12T02:38:51.426 INFO:tasks.rados.rados.0.smithi107.stdout:9355: done (1 left) 2020-06-12T02:38:51.426 INFO:tasks.rados.rados.0.smithi107.stdout:9356: delete oid 5 current snap is 0 2020-06-12T02:38:51.428 INFO:tasks.rados.rados.0.smithi107.stdout:9356: done (1 left) 2020-06-12T02:38:51.428 INFO:tasks.rados.rados.0.smithi107.stdout:9357: delete oid 1 current snap is 0 2020-06-12T02:38:51.431 INFO:tasks.rados.rados.0.smithi107.stdout:9357: done (1 left) 2020-06-12T02:38:51.431 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:38:51.432 INFO:tasks.rados.rados.0.smithi107.stdout:9358: seq_num 9357 ranges {0=720896} 2020-06-12T02:38:51.437 INFO:tasks.rados.rados.0.smithi107.stdout:9358: writing smithi10729407-6 from 0 to 720896 tid 1 2020-06-12T02:38:51.437 INFO:tasks.rados.rados.0.smithi107.stdout:9359: delete oid 13 current snap is 0 2020-06-12T02:38:51.439 INFO:tasks.rados.rados.0.smithi107.stdout:9359: done (2 left) 2020-06-12T02:38:51.439 INFO:tasks.rados.rados.0.smithi107.stdout:9360: delete oid 1 current snap is 0 2020-06-12T02:38:51.445 INFO:tasks.rados.rados.0.smithi107.stdout:9358: finishing write tid 1 to smithi10729407-6 2020-06-12T02:38:51.449 INFO:tasks.rados.rados.0.smithi107.stdout:9360: done (2 left) 2020-06-12T02:38:51.449 INFO:tasks.rados.rados.0.smithi107.stdout:9361: delete oid 20 current snap is 0 2020-06-12T02:38:51.450 INFO:tasks.rados.rados.0.smithi107.stdout:9358: finishing write tid 3 to smithi10729407-6 2020-06-12T02:38:51.459 INFO:tasks.rados.rados.0.smithi107.stdout:9358: finishing write tid 4 to smithi10729407-6 2020-06-12T02:38:51.460 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1695 (ObjNum 9357 snap 0 seq_num 9357) dirty exists 2020-06-12T02:38:51.460 INFO:tasks.rados.rados.0.smithi107.stdout:9358: left oid 6 (ObjNum 9357 snap 0 seq_num 9357) 2020-06-12T02:38:51.460 INFO:tasks.rados.rados.0.smithi107.stdout:9358: done (2 left) 2020-06-12T02:38:51.460 INFO:tasks.rados.rados.0.smithi107.stdout:9361: done (1 left) 2020-06-12T02:38:51.461 INFO:tasks.rados.rados.0.smithi107.stdout:9362: delete oid 11 current snap is 0 2020-06-12T02:38:51.461 INFO:tasks.rados.rados.0.smithi107.stdout:9362: done (1 left) 2020-06-12T02:38:51.461 INFO:tasks.rados.rados.0.smithi107.stdout:9363: delete oid 12 current snap is 0 2020-06-12T02:38:51.461 INFO:tasks.rados.rados.0.smithi107.stdout:9363: done (1 left) 2020-06-12T02:38:51.461 INFO:tasks.rados.rados.0.smithi107.stdout:9364: delete oid 13 current snap is 0 2020-06-12T02:38:51.462 INFO:tasks.rados.rados.0.smithi107.stdout:9364: done (1 left) 2020-06-12T02:38:51.462 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:51.463 INFO:tasks.rados.rados.0.smithi107.stdout:9365: seq_num 9364 ranges {0=434176,434176=401408} 2020-06-12T02:38:51.464 INFO:tasks.rados.rados.0.smithi107.stdout:9365: writing smithi10729407-15 from 0 to 434176 tid 1 2020-06-12T02:38:51.466 INFO:tasks.rados.rados.0.smithi107.stdout:9365: writing smithi10729407-15 from 434176 to 835584 tid 2 2020-06-12T02:38:51.466 INFO:tasks.rados.rados.0.smithi107.stdout:9366: delete oid 8 current snap is 0 2020-06-12T02:38:51.471 INFO:tasks.rados.rados.0.smithi107.stdout:9365: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:51.477 INFO:tasks.rados.rados.0.smithi107.stdout:9366: done (2 left) 2020-06-12T02:38:51.477 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:38:51.478 INFO:tasks.rados.rados.0.smithi107.stdout:9367: seq_num 9366 ranges {0=466944,466944=401408} 2020-06-12T02:38:51.478 INFO:tasks.rados.rados.0.smithi107.stdout:9367: writing smithi10729407-20 from 0 to 466944 tid 1 2020-06-12T02:38:51.485 INFO:tasks.rados.rados.0.smithi107.stdout:9367: writing smithi10729407-20 from 466944 to 868352 tid 2 2020-06-12T02:38:51.485 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:38:51.486 INFO:tasks.rados.rados.0.smithi107.stdout:9368: seq_num 9367 ranges {0=483328,483328=401408} 2020-06-12T02:38:51.486 INFO:tasks.rados.rados.0.smithi107.stdout:9368: writing smithi10729407-8 from 0 to 483328 tid 1 2020-06-12T02:38:51.487 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:38:51.483+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:38:51.487 INFO:tasks.rados.rados.0.smithi107.stdout:9368: writing smithi10729407-8 from 483328 to 884736 tid 2 2020-06-12T02:38:51.487 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:38:51.487 INFO:tasks.rados.rados.0.smithi107.stdout:9369: seq_num 9368 ranges {0=499712,499712=401408} 2020-06-12T02:38:51.489 INFO:tasks.rados.rados.0.smithi107.stdout:9369: writing smithi10729407-1 from 0 to 499712 tid 1 2020-06-12T02:38:51.494 INFO:tasks.rados.rados.0.smithi107.stdout:9369: writing smithi10729407-1 from 499712 to 901120 tid 2 2020-06-12T02:38:51.494 INFO:tasks.rados.rados.0.smithi107.stdout:9370: delete oid 2 current snap is 0 2020-06-12T02:38:51.494 INFO:tasks.rados.rados.0.smithi107.stdout:9365: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:51.495 INFO:tasks.rados.rados.0.smithi107.stdout:9365: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:51.495 INFO:tasks.rados.rados.0.smithi107.stdout:9365: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:51.495 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1739 (ObjNum 9364 snap 0 seq_num 9364) dirty exists 2020-06-12T02:38:51.495 INFO:tasks.rados.rados.0.smithi107.stdout:9365: left oid 15 (ObjNum 9364 snap 0 seq_num 9364) 2020-06-12T02:38:51.495 INFO:tasks.rados.rados.0.smithi107.stdout:9367: finishing write tid 1 to smithi10729407-20 2020-06-12T02:38:51.496 INFO:tasks.rados.rados.0.smithi107.stdout:9367: finishing write tid 2 to smithi10729407-20 2020-06-12T02:38:51.496 INFO:tasks.rados.rados.0.smithi107.stdout:9367: finishing write tid 4 to smithi10729407-20 2020-06-12T02:38:51.496 INFO:tasks.rados.rados.0.smithi107.stdout:9367: finishing write tid 5 to smithi10729407-20 2020-06-12T02:38:51.496 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 696 (ObjNum 9366 snap 0 seq_num 9366) dirty exists 2020-06-12T02:38:51.496 INFO:tasks.rados.rados.0.smithi107.stdout:9367: left oid 20 (ObjNum 9366 snap 0 seq_num 9366) 2020-06-12T02:38:51.497 INFO:tasks.rados.rados.0.smithi107.stdout:9368: finishing write tid 1 to smithi10729407-8 2020-06-12T02:38:51.497 INFO:tasks.rados.rados.0.smithi107.stdout:9365: done (5 left) 2020-06-12T02:38:51.498 INFO:tasks.rados.rados.0.smithi107.stdout:9367: done (4 left) 2020-06-12T02:38:51.498 INFO:tasks.rados.rados.0.smithi107.stdout:9370: done (3 left) 2020-06-12T02:38:51.498 INFO:tasks.rados.rados.0.smithi107.stdout:9371: delete oid 5 current snap is 0 2020-06-12T02:38:51.498 INFO:tasks.rados.rados.0.smithi107.stdout:9368: finishing write tid 2 to smithi10729407-8 2020-06-12T02:38:51.498 INFO:tasks.rados.rados.0.smithi107.stdout:9368: finishing write tid 4 to smithi10729407-8 2020-06-12T02:38:51.506 INFO:tasks.rados.rados.0.smithi107.stdout:9368: finishing write tid 5 to smithi10729407-8 2020-06-12T02:38:51.506 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1742 (ObjNum 9367 snap 0 seq_num 9367) dirty exists 2020-06-12T02:38:51.506 INFO:tasks.rados.rados.0.smithi107.stdout:9368: left oid 8 (ObjNum 9367 snap 0 seq_num 9367) 2020-06-12T02:38:51.507 INFO:tasks.rados.rados.0.smithi107.stdout:9368: done (3 left) 2020-06-12T02:38:51.507 INFO:tasks.rados.rados.0.smithi107.stdout:9371: done (2 left) 2020-06-12T02:38:51.507 INFO:tasks.rados.rados.0.smithi107.stdout:9372: delete oid 9 current snap is 0 2020-06-12T02:38:51.507 INFO:tasks.rados.rados.0.smithi107.stdout:9369: finishing write tid 1 to smithi10729407-1 2020-06-12T02:38:51.507 INFO:tasks.rados.rados.0.smithi107.stdout:9369: finishing write tid 2 to smithi10729407-1 2020-06-12T02:38:51.508 INFO:tasks.rados.rados.0.smithi107.stdout:9372: done (2 left) 2020-06-12T02:38:51.508 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:38:51.508 INFO:tasks.rados.rados.0.smithi107.stdout:9373: seq_num 9372 ranges {835584=565248,1400832=401408} 2020-06-12T02:38:51.514 INFO:tasks.rados.rados.0.smithi107.stdout:9373: writing smithi10729407-15 from 835584 to 1400832 tid 1 2020-06-12T02:38:51.516 INFO:tasks.rados.rados.0.smithi107.stdout:9373: writing smithi10729407-15 from 1400832 to 1802240 tid 2 2020-06-12T02:38:51.516 INFO:tasks.rados.rados.0.smithi107.stdout:9374: delete oid 19 current snap is 0 2020-06-12T02:38:51.517 INFO:tasks.rados.rados.0.smithi107.stdout:9369: finishing write tid 4 to smithi10729407-1 2020-06-12T02:38:51.517 INFO:tasks.rados.rados.0.smithi107.stdout:9369: finishing write tid 5 to smithi10729407-1 2020-06-12T02:38:51.518 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1698 (ObjNum 9368 snap 0 seq_num 9368) dirty exists 2020-06-12T02:38:51.518 INFO:tasks.rados.rados.0.smithi107.stdout:9369: left oid 1 (ObjNum 9368 snap 0 seq_num 9368) 2020-06-12T02:38:51.520 INFO:tasks.rados.rados.0.smithi107.stdout:9369: done (3 left) 2020-06-12T02:38:51.520 INFO:tasks.rados.rados.0.smithi107.stdout:9374: done (2 left) 2020-06-12T02:38:51.520 INFO:tasks.rados.rados.0.smithi107.stdout:9375: delete oid 3 current snap is 0 2020-06-12T02:38:51.521 INFO:tasks.rados.rados.0.smithi107.stdout:9373: finishing write tid 1 to smithi10729407-15 2020-06-12T02:38:51.523 INFO:tasks.rados.rados.0.smithi107.stdout:9375: done (2 left) 2020-06-12T02:38:51.523 INFO:tasks.rados.rados.0.smithi107.stdout:9376: delete oid 6 current snap is 0 2020-06-12T02:38:51.527 INFO:tasks.rados.rados.0.smithi107.stdout:9376: done (2 left) 2020-06-12T02:38:51.527 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:38:51.527 INFO:tasks.rados.rados.0.smithi107.stdout:9377: seq_num 9376 ranges {0=638976,638976=401408} 2020-06-12T02:38:51.530 INFO:tasks.rados.rados.0.smithi107.stdout:9377: writing smithi10729407-18 from 0 to 638976 tid 1 2020-06-12T02:38:51.532 INFO:tasks.rados.rados.0.smithi107.stdout:9377: writing smithi10729407-18 from 638976 to 1040384 tid 2 2020-06-12T02:38:51.533 INFO:tasks.rados.rados.0.smithi107.stdout:9378: delete oid 11 current snap is 0 2020-06-12T02:38:51.533 INFO:tasks.rados.rados.0.smithi107.stdout:9373: finishing write tid 2 to smithi10729407-15 2020-06-12T02:38:51.533 INFO:tasks.rados.rados.0.smithi107.stdout:9373: finishing write tid 4 to smithi10729407-15 2020-06-12T02:38:51.533 INFO:tasks.rados.rados.0.smithi107.stdout:9373: finishing write tid 5 to smithi10729407-15 2020-06-12T02:38:51.533 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1745 (ObjNum 9372 snap 0 seq_num 9372) dirty exists 2020-06-12T02:38:51.534 INFO:tasks.rados.rados.0.smithi107.stdout:9373: left oid 15 (ObjNum 9372 snap 0 seq_num 9372) 2020-06-12T02:38:51.535 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:51.535 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:38:51.535 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:38:51.536 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:38:51.536 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:51.537 INFO:tasks.rados.rados.0.smithi107.stdout:9373: done (3 left) 2020-06-12T02:38:51.537 INFO:tasks.rados.rados.0.smithi107.stdout:9378: done (2 left) 2020-06-12T02:38:51.537 INFO:tasks.rados.rados.0.smithi107.stdout:9379: delete oid 9 current snap is 0 2020-06-12T02:38:51.538 INFO:tasks.rados.rados.0.smithi107.stdout:9377: finishing write tid 1 to smithi10729407-18 2020-06-12T02:38:51.542 INFO:tasks.rados.rados.0.smithi107.stdout:9379: done (2 left) 2020-06-12T02:38:51.542 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:38:51.543 INFO:tasks.rados.rados.0.smithi107.stdout:9380: seq_num 9379 ranges {450560=688128,1138688=401408} 2020-06-12T02:38:51.545 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-06-12T02:38:51.548 INFO:tasks.rados.rados.0.smithi107.stdout:9380: writing smithi10729407-4 from 450560 to 1138688 tid 1 2020-06-12T02:38:51.550 INFO:tasks.rados.rados.0.smithi107.stdout:9380: writing smithi10729407-4 from 1138688 to 1540096 tid 2 2020-06-12T02:38:51.551 INFO:tasks.rados.rados.0.smithi107.stdout:9381: delete oid 16 current snap is 0 2020-06-12T02:38:51.552 INFO:tasks.rados.rados.0.smithi107.stdout:9377: finishing write tid 2 to smithi10729407-18 2020-06-12T02:38:51.552 INFO:tasks.rados.rados.0.smithi107.stdout:9377: finishing write tid 4 to smithi10729407-18 2020-06-12T02:38:51.552 INFO:tasks.rados.rados.0.smithi107.stdout:9377: finishing write tid 5 to smithi10729407-18 2020-06-12T02:38:51.552 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1741 (ObjNum 9376 snap 0 seq_num 9376) dirty exists 2020-06-12T02:38:51.552 INFO:tasks.rados.rados.0.smithi107.stdout:9377: left oid 18 (ObjNum 9376 snap 0 seq_num 9376) 2020-06-12T02:38:51.553 INFO:tasks.rados.rados.0.smithi107.stdout:9377: done (3 left) 2020-06-12T02:38:51.553 INFO:tasks.rados.rados.0.smithi107.stdout:9381: done (2 left) 2020-06-12T02:38:51.553 INFO:tasks.rados.rados.0.smithi107.stdout:9382: delete oid 18 current snap is 0 2020-06-12T02:38:51.557 INFO:tasks.rados.rados.0.smithi107.stdout:9380: finishing write tid 1 to smithi10729407-4 2020-06-12T02:38:51.558 INFO:tasks.rados.rados.0.smithi107.stdout:9382: done (2 left) 2020-06-12T02:38:51.558 INFO:tasks.rados.rados.0.smithi107.stdout:9383: delete oid 20 current snap is 0 2020-06-12T02:38:51.558 INFO:tasks.rados.rados.0.smithi107.stdout:9380: finishing write tid 2 to smithi10729407-4 2020-06-12T02:38:51.559 INFO:tasks.rados.rados.0.smithi107.stdout:9380: finishing write tid 4 to smithi10729407-4 2020-06-12T02:38:51.560 INFO:tasks.rados.rados.0.smithi107.stdout:9380: finishing write tid 5 to smithi10729407-4 2020-06-12T02:38:51.560 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1766 (ObjNum 9379 snap 0 seq_num 9379) dirty exists 2020-06-12T02:38:51.560 INFO:tasks.rados.rados.0.smithi107.stdout:9380: left oid 4 (ObjNum 9379 snap 0 seq_num 9379) 2020-06-12T02:38:51.560 INFO:tasks.rados.rados.0.smithi107.stdout:9380: done (2 left) 2020-06-12T02:38:51.561 INFO:tasks.rados.rados.0.smithi107.stdout:9383: done (1 left) 2020-06-12T02:38:51.562 INFO:tasks.rados.rados.0.smithi107.stdout:9384: delete oid 8 current snap is 0 2020-06-12T02:38:51.563 INFO:tasks.rados.rados.0.smithi107.stdout:9384: done (1 left) 2020-06-12T02:38:51.563 INFO:tasks.rados.rados.0.smithi107.stdout:9385: delete oid 4 current snap is 0 2020-06-12T02:38:51.565 INFO:tasks.rados.rados.0.smithi107.stdout:9385: done (1 left) 2020-06-12T02:38:51.565 INFO:tasks.rados.rados.0.smithi107.stdout:9386: delete oid 14 current snap is 0 2020-06-12T02:38:51.586 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:38:51.583+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:38:51.688 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:51.685+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:51.704 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:51.705 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:38:51.705 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:38:51.705 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:38:51.705 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.705 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8464 3.fs0 3.57bc3c1f (undecoded) ondisk+retry+write+known_if_redirected e34)", 2020-06-12T02:38:51.706 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:53.549364+0000", 2020-06-12T02:38:51.708 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 58.150623283000002, 2020-06-12T02:38:51.708 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.003107494, 2020-06-12T02:38:51.708 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:51.708 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:38:51.709 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:51.709 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:51.709 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:51.709 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8464 2020-06-12T02:38:51.710 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.710 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:51.710 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.710 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:51.710 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.549364+0000", 2020-06-12T02:38:51.711 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:51.711 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.711 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.714 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:51.714 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.549364+0000", 2020-06-12T02:38:51.714 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023109609999999998 2020-06-12T02:38:51.714 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.715 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.715 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:51.715 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551675+0000", 2020-06-12T02:38:51.715 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3331e-05 2020-06-12T02:38:51.715 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.716 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.716 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:51.716 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551688+0000", 2020-06-12T02:38:51.716 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4173e-05 2020-06-12T02:38:51.716 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.717 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.717 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:51.717 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551703+0000", 2020-06-12T02:38:51.717 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0454e-05 2020-06-12T02:38:51.717 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.717 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.718 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:51.718 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551713+0000", 2020-06-12T02:38:51.718 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031069050000000001 2020-06-12T02:38:51.718 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.718 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.719 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.719 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554820+0000", 2020-06-12T02:38:51.719 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2820000000000002e-06 2020-06-12T02:38:51.719 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.719 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.720 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2020-06-12T02:38:51.720 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554826+0000", 2020-06-12T02:38:51.720 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99762648499999995 2020-06-12T02:38:51.720 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.720 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.720 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.721 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552453+0000", 2020-06-12T02:38:51.721 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8902999999999998e-05 2020-06-12T02:38:51.721 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.721 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.721 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:51.722 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552472+0000", 2020-06-12T02:38:51.722 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.003107494 2020-06-12T02:38:51.722 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.722 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:51.722 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.724 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.725 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.725 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8465 3.fs0 3.57bc3c1f (undecoded) ondisk+retry+write+known_if_redirected e34)", 2020-06-12T02:38:51.725 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:53.551774+0000", 2020-06-12T02:38:51.725 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 58.148213646999999, 2020-06-12T02:38:51.726 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.000741442, 2020-06-12T02:38:51.726 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:51.726 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:38:51.726 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:51.726 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:51.727 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:51.727 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8465 2020-06-12T02:38:51.727 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.727 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:51.727 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.728 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:51.728 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551774+0000", 2020-06-12T02:38:51.728 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:51.728 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.728 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.729 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:51.730 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551774+0000", 2020-06-12T02:38:51.730 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0010760539999999999 2020-06-12T02:38:51.730 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.731 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.731 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:51.731 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552850+0000", 2020-06-12T02:38:51.731 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0139000000000001e-05 2020-06-12T02:38:51.732 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.733 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.733 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:51.734 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552860+0000", 2020-06-12T02:38:51.734 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7646999999999999e-05 2020-06-12T02:38:51.734 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.734 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.734 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:51.735 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552878+0000", 2020-06-12T02:38:51.735 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2916e-05 2020-06-12T02:38:51.735 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.735 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.735 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:51.735 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552890+0000", 2020-06-12T02:38:51.736 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0019574610000000002 2020-06-12T02:38:51.736 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.736 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.736 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.736 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554848+0000", 2020-06-12T02:38:51.737 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0399999999999995e-07 2020-06-12T02:38:51.737 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.737 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.737 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2020-06-12T02:38:51.737 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554849+0000", 2020-06-12T02:38:51.738 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99765586100000003 2020-06-12T02:38:51.738 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.738 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.738 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.738 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552504+0000", 2020-06-12T02:38:51.738 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.066e-05 2020-06-12T02:38:51.739 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.739 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.739 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:51.739 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552515+0000", 2020-06-12T02:38:51.739 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.000741442 2020-06-12T02:38:51.740 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.740 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:51.740 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.740 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.740 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.741 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8466 3.fs0 3.57bc3c1f (undecoded) ondisk+write+known_if_redirected e34)", 2020-06-12T02:38:51.741 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:53.552953+0000", 2020-06-12T02:38:51.741 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 58.147034367000003, 2020-06-12T02:38:51.741 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99960144299999998, 2020-06-12T02:38:51.741 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:51.742 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:38:51.742 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:51.742 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:51.742 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:51.743 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8466 2020-06-12T02:38:51.743 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.743 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:51.743 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.743 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:51.743 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552953+0000", 2020-06-12T02:38:51.744 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:51.744 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.744 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.744 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:51.744 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552953+0000", 2020-06-12T02:38:51.745 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013847100000000001 2020-06-12T02:38:51.745 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.745 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.745 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:51.745 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553091+0000", 2020-06-12T02:38:51.745 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7190000000000004e-06 2020-06-12T02:38:51.746 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.746 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.746 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:51.746 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553098+0000", 2020-06-12T02:38:51.746 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.171e-05 2020-06-12T02:38:51.747 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.747 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.747 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:51.747 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553110+0000", 2020-06-12T02:38:51.747 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5770000000000001e-06 2020-06-12T02:38:51.747 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.748 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.748 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:51.748 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553115+0000", 2020-06-12T02:38:51.748 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0017512739999999999 2020-06-12T02:38:51.748 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.749 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.749 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.749 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554867+0000", 2020-06-12T02:38:51.749 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1200000000000003e-07 2020-06-12T02:38:51.749 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.749 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.750 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2020-06-12T02:38:51.750 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554867+0000", 2020-06-12T02:38:51.750 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99767707100000003 2020-06-12T02:38:51.750 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.750 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.751 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.751 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552544+0000", 2020-06-12T02:38:51.751 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0009e-05 2020-06-12T02:38:51.751 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.751 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.752 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:51.752 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552554+0000", 2020-06-12T02:38:51.752 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99960144299999998 2020-06-12T02:38:51.759 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.759 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:51.759 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.760 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.760 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.760 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8467 3.fs0 3.57bc3c1f (undecoded) ondisk+write+known_if_redirected e34)", 2020-06-12T02:38:51.760 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:53.553172+0000", 2020-06-12T02:38:51.760 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 58.146815150999998, 2020-06-12T02:38:51.761 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99942086200000002, 2020-06-12T02:38:51.761 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:51.761 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:38:51.761 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:51.761 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:51.762 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:51.762 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8467 2020-06-12T02:38:51.762 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.762 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:51.762 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.763 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:51.763 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553172+0000", 2020-06-12T02:38:51.763 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:51.763 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.763 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.764 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:51.764 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553172+0000", 2020-06-12T02:38:51.764 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5913000000000003e-05 2020-06-12T02:38:51.764 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.764 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.765 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:51.765 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553248+0000", 2020-06-12T02:38:51.765 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7740000000000001e-06 2020-06-12T02:38:51.765 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.765 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.765 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:51.766 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553254+0000", 2020-06-12T02:38:51.766 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1494999999999999e-05 2020-06-12T02:38:51.766 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.766 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.766 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:51.767 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553265+0000", 2020-06-12T02:38:51.767 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5170000000000001e-06 2020-06-12T02:38:51.767 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.767 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.768 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:51.768 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553270+0000", 2020-06-12T02:38:51.768 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001616567 2020-06-12T02:38:51.768 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.768 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.769 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.769 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554886+0000", 2020-06-12T02:38:51.769 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7800000000000001e-07 2020-06-12T02:38:51.769 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.769 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.770 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2020-06-12T02:38:51.770 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554887+0000", 2020-06-12T02:38:51.770 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99769652399999997 2020-06-12T02:38:51.770 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.770 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.770 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.771 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552584+0000", 2020-06-12T02:38:51.771 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4939999999999992e-06 2020-06-12T02:38:51.771 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.771 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.771 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:51.772 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552593+0000", 2020-06-12T02:38:51.772 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99942086200000002 2020-06-12T02:38:51.772 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.772 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:51.773 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.773 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.773 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.773 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8468 3.fs0 3.57bc3c1f (undecoded) ondisk+read+rwordered+known_if_redirected e34)", 2020-06-12T02:38:51.773 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:53.553311+0000", 2020-06-12T02:38:51.774 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 58.146676552999999, 2020-06-12T02:38:51.774 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99988704799999994, 2020-06-12T02:38:51.774 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:51.774 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:38:51.774 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:51.774 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:51.775 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:51.775 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8468 2020-06-12T02:38:51.775 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.775 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:51.775 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.776 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:51.776 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553311+0000", 2020-06-12T02:38:51.776 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:51.777 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.777 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.777 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:51.778 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553311+0000", 2020-06-12T02:38:51.778 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2050000000000001e-05 2020-06-12T02:38:51.778 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.778 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.779 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:51.779 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553333+0000", 2020-06-12T02:38:51.779 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5489999999999998e-06 2020-06-12T02:38:51.779 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.779 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.780 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:51.780 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553336+0000", 2020-06-12T02:38:51.780 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1620000000000002e-06 2020-06-12T02:38:51.780 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.780 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.781 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:51.781 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553344+0000", 2020-06-12T02:38:51.781 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.807e-06 2020-06-12T02:38:51.781 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.781 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.782 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:51.782 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553346+0000", 2020-06-12T02:38:51.782 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001560575 2020-06-12T02:38:51.782 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.783 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.783 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.786 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554907+0000", 2020-06-12T02:38:51.787 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1429999999999999e-06 2020-06-12T02:38:51.787 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.787 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.787 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2020-06-12T02:38:51.788 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554908+0000", 2020-06-12T02:38:51.788 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99827844600000004 2020-06-12T02:38:51.788 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.788 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.788 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.788 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.553187+0000", 2020-06-12T02:38:51.789 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1316e-05 2020-06-12T02:38:51.789 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.789 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.789 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:51.789 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.553198+0000", 2020-06-12T02:38:51.790 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99988704799999994 2020-06-12T02:38:51.790 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.790 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:51.790 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.790 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.791 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.791 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11992 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [append 0~581632] snapc 0=[] ondisk+write+known_if_redirected e56)", 2020-06-12T02:38:51.791 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:23.783061+0000", 2020-06-12T02:38:51.791 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 27.916926371999999, 2020-06-12T02:38:51.791 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022850993, 2020-06-12T02:38:51.792 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:51.802 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:51.802 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:51.803 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:51.803 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:51.803 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11992 2020-06-12T02:38:51.803 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.803 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:51.804 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.804 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:51.804 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783061+0000", 2020-06-12T02:38:51.804 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:51.805 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.805 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.805 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:51.805 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783061+0000", 2020-06-12T02:38:51.805 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047999300000000001 2020-06-12T02:38:51.806 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.806 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.806 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:51.806 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783541+0000", 2020-06-12T02:38:51.806 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5649999999999998e-06 2020-06-12T02:38:51.807 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.807 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.807 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:51.807 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783548+0000", 2020-06-12T02:38:51.808 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3559e-05 2020-06-12T02:38:51.808 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.808 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.808 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:51.808 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783561+0000", 2020-06-12T02:38:51.809 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3539999999999996e-06 2020-06-12T02:38:51.809 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.809 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.809 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:51.809 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783567+0000", 2020-06-12T02:38:51.810 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001353604 2020-06-12T02:38:51.810 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.810 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.810 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.810 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.784921+0000", 2020-06-12T02:38:51.811 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000111853 2020-06-12T02:38:51.811 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.811 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.811 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:51.811 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.785033+0000", 2020-06-12T02:38:51.812 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051647919999999996 2020-06-12T02:38:51.812 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.812 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.812 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.812 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.790198+0000", 2020-06-12T02:38:51.813 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000110742 2020-06-12T02:38:51.813 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.813 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.813 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:51.814 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.790308+0000", 2020-06-12T02:38:51.814 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00090050099999999995 2020-06-12T02:38:51.814 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.814 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.814 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:51.815 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.791209+0000", 2020-06-12T02:38:51.815 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010078284999999999 2020-06-12T02:38:51.815 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.815 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.815 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:51.816 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.801287+0000", 2020-06-12T02:38:51.816 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045443109999999997 2020-06-12T02:38:51.816 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.816 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.816 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:51.817 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.805832+0000", 2020-06-12T02:38:51.817 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.0433999999999996e-05 2020-06-12T02:38:51.817 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.818 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.818 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:51.818 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.805912+0000", 2020-06-12T02:38:51.818 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022850993 2020-06-12T02:38:51.818 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.819 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:51.819 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.819 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.819 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.820 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11994 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected e56)", 2020-06-12T02:38:51.820 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:23.788734+0000", 2020-06-12T02:38:51.820 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 27.911253726999998, 2020-06-12T02:38:51.820 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024022136999999999, 2020-06-12T02:38:51.820 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:51.821 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:51.821 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:51.821 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:51.821 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:51.822 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11994 2020-06-12T02:38:51.822 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.822 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:51.822 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.822 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:51.823 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.788734+0000", 2020-06-12T02:38:51.823 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:51.824 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.824 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.824 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:51.824 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.788734+0000", 2020-06-12T02:38:51.824 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00034212499999999998 2020-06-12T02:38:51.825 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.825 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.825 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:51.825 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789076+0000", 2020-06-12T02:38:51.825 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.254e-06 2020-06-12T02:38:51.826 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.826 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.826 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:51.826 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789081+0000", 2020-06-12T02:38:51.826 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8249999999999994e-06 2020-06-12T02:38:51.827 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.827 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.827 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:51.827 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789090+0000", 2020-06-12T02:38:51.828 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2259999999999999e-06 2020-06-12T02:38:51.828 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.828 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.828 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:51.828 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789094+0000", 2020-06-12T02:38:51.829 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2913000000000001e-05 2020-06-12T02:38:51.829 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.829 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.829 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.830 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789127+0000", 2020-06-12T02:38:51.830 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2001000000000005e-05 2020-06-12T02:38:51.830 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.830 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.830 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:51.831 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789189+0000", 2020-06-12T02:38:51.831 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0097474550000000004 2020-06-12T02:38:51.831 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.831 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.831 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.832 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.798936+0000", 2020-06-12T02:38:51.832 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9882000000000005e-05 2020-06-12T02:38:51.832 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.832 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.832 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:51.833 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.799016+0000", 2020-06-12T02:38:51.833 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00048691000000000001 2020-06-12T02:38:51.833 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.833 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.834 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:51.834 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.799503+0000", 2020-06-12T02:38:51.834 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002766363 2020-06-12T02:38:51.834 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.834 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.835 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:51.835 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.802270+0000", 2020-06-12T02:38:51.835 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010404759 2020-06-12T02:38:51.835 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.835 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.836 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:51.836 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.812674+0000", 2020-06-12T02:38:51.836 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1423999999999998e-05 2020-06-12T02:38:51.836 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.836 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.837 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:51.837 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.812756+0000", 2020-06-12T02:38:51.837 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024022136999999999 2020-06-12T02:38:51.837 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.838 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:51.838 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.838 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.838 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.839 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11995 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e56)", 2020-06-12T02:38:51.839 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:23.789128+0000", 2020-06-12T02:38:51.839 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 27.910859281, 2020-06-12T02:38:51.839 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031795508, 2020-06-12T02:38:51.839 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:51.840 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:51.840 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:51.840 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:51.840 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:51.840 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11995 2020-06-12T02:38:51.841 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.841 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:51.841 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.841 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:51.841 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789128+0000", 2020-06-12T02:38:51.842 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:51.842 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.842 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.842 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:51.842 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789128+0000", 2020-06-12T02:38:51.843 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4419999999999998e-05 2020-06-12T02:38:51.843 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.843 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.843 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:51.843 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789172+0000", 2020-06-12T02:38:51.844 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7110000000000001e-06 2020-06-12T02:38:51.844 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.844 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.844 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:51.844 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789176+0000", 2020-06-12T02:38:51.845 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5089999999999999e-06 2020-06-12T02:38:51.845 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.845 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.845 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:51.845 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789183+0000", 2020-06-12T02:38:51.846 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.497e-06 2020-06-12T02:38:51.846 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.846 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.846 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:51.846 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789185+0000", 2020-06-12T02:38:51.847 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012172176 2020-06-12T02:38:51.847 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.847 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.847 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.847 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.801357+0000", 2020-06-12T02:38:51.847 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4264000000000006e-05 2020-06-12T02:38:51.848 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.848 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.848 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:51.849 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.801432+0000", 2020-06-12T02:38:51.849 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00029680499999999997 2020-06-12T02:38:51.849 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.849 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.849 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:51.850 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.801728+0000", 2020-06-12T02:38:51.850 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00066278300000000001 2020-06-12T02:38:51.850 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.850 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.850 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:51.851 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.802391+0000", 2020-06-12T02:38:51.851 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018429867999999999 2020-06-12T02:38:51.851 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.851 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.851 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:51.852 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.820821+0000", 2020-06-12T02:38:51.852 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000102475 2020-06-12T02:38:51.852 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.852 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.852 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:51.853 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.820924+0000", 2020-06-12T02:38:51.853 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031795508 2020-06-12T02:38:51.853 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.853 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:51.853 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.854 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.854 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.854 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11996 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e56)", 2020-06-12T02:38:51.854 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:23.789235+0000", 2020-06-12T02:38:51.855 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 27.910752429999999, 2020-06-12T02:38:51.855 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.034785745, 2020-06-12T02:38:51.855 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:51.855 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:51.855 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:51.855 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:51.856 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:51.856 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11996 2020-06-12T02:38:51.856 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.856 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:51.856 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.857 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:51.857 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789235+0000", 2020-06-12T02:38:51.857 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:51.857 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.858 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.858 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:51.858 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789235+0000", 2020-06-12T02:38:51.858 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6375000000000001e-05 2020-06-12T02:38:51.859 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.859 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.859 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:51.859 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789261+0000", 2020-06-12T02:38:51.860 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2679999999999999e-06 2020-06-12T02:38:51.860 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.860 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.860 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:51.860 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789265+0000", 2020-06-12T02:38:51.860 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6039999999999998e-06 2020-06-12T02:38:51.861 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.861 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.861 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:51.861 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789271+0000", 2020-06-12T02:38:51.863 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4219999999999999e-06 2020-06-12T02:38:51.863 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.864 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.864 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:51.865 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789274+0000", 2020-06-12T02:38:51.865 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013034463 2020-06-12T02:38:51.865 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.865 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.865 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.866 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.802308+0000", 2020-06-12T02:38:51.866 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8862000000000005e-05 2020-06-12T02:38:51.866 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.866 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.866 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:51.867 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.802387+0000", 2020-06-12T02:38:51.867 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019102297000000001 2020-06-12T02:38:51.867 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.867 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.868 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.868 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.821489+0000", 2020-06-12T02:38:51.868 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8838999999999996e-05 2020-06-12T02:38:51.868 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.868 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.869 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:51.869 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.821558+0000", 2020-06-12T02:38:51.869 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002462615 2020-06-12T02:38:51.870 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.870 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.870 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:51.870 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.824021+0000", 2020-06-12T02:38:51.870 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.034785745 2020-06-12T02:38:51.871 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.871 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:51.871 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.871 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.871 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.872 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:15005 3.ds0 3:b3445631:::smithi10729407-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e59)", 2020-06-12T02:38:51.872 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:32.192302+0000", 2020-06-12T02:38:51.872 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 19.507685195000001, 2020-06-12T02:38:51.872 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024128666, 2020-06-12T02:38:51.872 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:51.873 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:51.873 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:51.873 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:51.873 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:51.873 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 15005 2020-06-12T02:38:51.874 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.874 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:51.874 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.874 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:51.874 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192302+0000", 2020-06-12T02:38:51.875 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:51.875 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.875 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.875 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:51.875 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192302+0000", 2020-06-12T02:38:51.876 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3415999999999999e-05 2020-06-12T02:38:51.876 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.876 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.876 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:51.876 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192346+0000", 2020-06-12T02:38:51.877 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.93e-06 2020-06-12T02:38:51.877 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.877 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.877 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:51.877 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192352+0000", 2020-06-12T02:38:51.878 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3397000000000001e-05 2020-06-12T02:38:51.878 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.878 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.878 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:51.878 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192365+0000", 2020-06-12T02:38:51.879 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3950000000000003e-06 2020-06-12T02:38:51.879 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.879 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.879 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:51.880 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192369+0000", 2020-06-12T02:38:51.880 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010208887999999999 2020-06-12T02:38:51.880 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.880 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.880 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.881 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.202578+0000", 2020-06-12T02:38:51.881 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000105611 2020-06-12T02:38:51.881 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.881 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.881 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:51.882 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.202684+0000", 2020-06-12T02:38:51.882 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0091288959999999992 2020-06-12T02:38:51.882 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.882 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.882 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.883 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.211813+0000", 2020-06-12T02:38:51.883 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000101798 2020-06-12T02:38:51.883 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.883 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.883 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:51.884 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.211914+0000", 2020-06-12T02:38:51.884 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045163349999999998 2020-06-12T02:38:51.884 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.884 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.884 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:51.885 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.216431+0000", 2020-06-12T02:38:51.885 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024128666 2020-06-12T02:38:51.885 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.885 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:51.885 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.886 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.886 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.886 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:16810 3.ds0 3:b983fd28:::smithi10729407-1:head [append 0~524288] snapc 0=[] ondisk+write+known_if_redirected e61)", 2020-06-12T02:38:51.886 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:37.642363+0000", 2020-06-12T02:38:51.886 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 14.057623940999999, 2020-06-12T02:38:51.887 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022969679, 2020-06-12T02:38:51.887 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:51.887 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:51.887 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:51.887 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:51.888 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:51.888 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 16810 2020-06-12T02:38:51.888 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.888 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:51.888 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.889 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:51.889 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.642363+0000", 2020-06-12T02:38:51.889 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:51.890 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.890 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.890 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:51.890 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.642363+0000", 2020-06-12T02:38:51.891 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00069050800000000005 2020-06-12T02:38:51.891 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.891 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.891 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:51.891 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.643054+0000", 2020-06-12T02:38:51.892 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8220000000000007e-06 2020-06-12T02:38:51.892 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.892 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.892 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:51.892 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.643062+0000", 2020-06-12T02:38:51.893 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3742e-05 2020-06-12T02:38:51.893 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.894 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.894 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:51.895 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.643075+0000", 2020-06-12T02:38:51.895 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5910000000000001e-06 2020-06-12T02:38:51.896 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.896 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.896 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:51.896 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.643082+0000", 2020-06-12T02:38:51.897 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0064097879999999996 2020-06-12T02:38:51.897 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.897 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.897 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.897 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.649492+0000", 2020-06-12T02:38:51.898 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8151999999999994e-05 2020-06-12T02:38:51.898 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.898 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.898 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:51.898 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.649560+0000", 2020-06-12T02:38:51.899 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0020294940000000002 2020-06-12T02:38:51.899 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.899 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.899 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.900 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.651590+0000", 2020-06-12T02:38:51.900 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5242999999999997e-05 2020-06-12T02:38:51.900 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.900 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.901 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:51.901 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.651675+0000", 2020-06-12T02:38:51.901 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00069163799999999995 2020-06-12T02:38:51.901 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.901 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.902 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:51.902 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.652366+0000", 2020-06-12T02:38:51.902 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012711642 2020-06-12T02:38:51.902 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.902 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.903 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:51.903 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.665078+0000", 2020-06-12T02:38:51.903 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017519200000000001 2020-06-12T02:38:51.903 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.904 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.904 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:51.904 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.665253+0000", 2020-06-12T02:38:51.904 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9867e-05 2020-06-12T02:38:51.905 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.905 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.905 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:51.905 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.665333+0000", 2020-06-12T02:38:51.905 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022969679 2020-06-12T02:38:51.906 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.906 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:51.906 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.906 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.906 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.907 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18287 3.ds0 3:b983fd28:::smithi10729407-1:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:38:51.907 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.975980+0000", 2020-06-12T02:38:51.907 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.7240071149999991, 2020-06-12T02:38:51.907 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.023612415000000001, 2020-06-12T02:38:51.907 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:51.908 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:51.908 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:51.908 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:51.908 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:51.908 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18287 2020-06-12T02:38:51.909 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.909 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:51.909 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.909 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:51.909 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.975980+0000", 2020-06-12T02:38:51.910 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:51.910 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.910 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.910 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:51.911 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.975980+0000", 2020-06-12T02:38:51.911 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0175999999999999e-05 2020-06-12T02:38:51.911 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.911 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.911 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:51.912 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976070+0000", 2020-06-12T02:38:51.912 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7489999999999998e-06 2020-06-12T02:38:51.912 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.912 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.912 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:51.913 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976078+0000", 2020-06-12T02:38:51.913 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5308999999999999e-05 2020-06-12T02:38:51.913 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.913 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.913 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:51.914 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976093+0000", 2020-06-12T02:38:51.914 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7149999999999997e-06 2020-06-12T02:38:51.914 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.914 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.914 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:51.915 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976098+0000", 2020-06-12T02:38:51.915 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0067110759999999998 2020-06-12T02:38:51.915 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.915 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.915 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.916 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.982809+0000", 2020-06-12T02:38:51.916 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011082999999999999 2020-06-12T02:38:51.916 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.916 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.916 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:51.917 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.982920+0000", 2020-06-12T02:38:51.917 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00043570099999999998 2020-06-12T02:38:51.917 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.917 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.917 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:51.918 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.983356+0000", 2020-06-12T02:38:51.918 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00076300800000000002 2020-06-12T02:38:51.918 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.918 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.918 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:51.919 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.984119+0000", 2020-06-12T02:38:51.919 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01538405 2020-06-12T02:38:51.919 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.919 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.920 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:51.920 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.999503+0000", 2020-06-12T02:38:51.920 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9801000000000003e-05 2020-06-12T02:38:51.920 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.921 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.921 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:51.921 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.999593+0000", 2020-06-12T02:38:51.921 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.023612415000000001 2020-06-12T02:38:51.921 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.922 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:51.922 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.922 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.922 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.923 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18288 3.ds0 3:b983fd28:::smithi10729407-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:38:51.923 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.976151+0000", 2020-06-12T02:38:51.923 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.7238361920000003, 2020-06-12T02:38:51.923 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025323907999999999, 2020-06-12T02:38:51.924 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:51.924 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:51.924 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:51.924 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:51.925 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:51.925 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18288 2020-06-12T02:38:51.925 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.925 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:51.925 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.926 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:51.926 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976151+0000", 2020-06-12T02:38:51.926 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:51.926 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.926 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.927 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:51.927 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976151+0000", 2020-06-12T02:38:51.927 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5615000000000002e-05 2020-06-12T02:38:51.927 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.927 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.928 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:51.928 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976207+0000", 2020-06-12T02:38:51.928 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6399999999999997e-06 2020-06-12T02:38:51.928 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.928 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.929 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:51.929 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976214+0000", 2020-06-12T02:38:51.929 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4823e-05 2020-06-12T02:38:51.929 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.929 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.930 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:51.930 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976229+0000", 2020-06-12T02:38:51.930 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1519999999999997e-06 2020-06-12T02:38:51.930 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.931 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.931 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:51.931 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976235+0000", 2020-06-12T02:38:51.931 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0077693090000000003 2020-06-12T02:38:51.932 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.932 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.932 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.932 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.984005+0000", 2020-06-12T02:38:51.932 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000101221 2020-06-12T02:38:51.932 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.933 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.933 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:51.933 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.984106+0000", 2020-06-12T02:38:51.933 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016277713999999999 2020-06-12T02:38:51.934 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.934 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.934 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.934 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.000384+0000", 2020-06-12T02:38:51.935 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7938999999999999e-05 2020-06-12T02:38:51.935 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.935 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.935 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:51.935 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.000482+0000", 2020-06-12T02:38:51.936 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00099349500000000006 2020-06-12T02:38:51.936 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.936 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.936 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:51.936 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.001475+0000", 2020-06-12T02:38:51.937 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025323907999999999 2020-06-12T02:38:51.937 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.937 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:51.937 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.937 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.938 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.938 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18307 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [append 0~507904] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:38:51.938 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.056119+0000", 2020-06-12T02:38:51.938 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.6438683090000001, 2020-06-12T02:38:51.938 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025879849, 2020-06-12T02:38:51.939 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:51.939 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:51.939 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:51.939 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:51.939 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:51.940 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18307 2020-06-12T02:38:51.940 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.940 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:51.940 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.940 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:51.941 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.056119+0000", 2020-06-12T02:38:51.941 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:51.941 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.941 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.942 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:51.942 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.056119+0000", 2020-06-12T02:38:51.942 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00052484099999999996 2020-06-12T02:38:51.942 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.943 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.943 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:51.943 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.056644+0000", 2020-06-12T02:38:51.943 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7149999999999997e-06 2020-06-12T02:38:51.943 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.944 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.944 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:51.944 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.056649+0000", 2020-06-12T02:38:51.944 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0046e-05 2020-06-12T02:38:51.944 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.945 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.945 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:51.945 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.056659+0000", 2020-06-12T02:38:51.945 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.848e-06 2020-06-12T02:38:51.945 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.946 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.946 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:51.946 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.056663+0000", 2020-06-12T02:38:51.948 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024537029999999998 2020-06-12T02:38:51.948 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.948 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.948 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.949 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.059117+0000", 2020-06-12T02:38:51.949 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000121261 2020-06-12T02:38:51.949 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.949 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.949 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:51.950 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.059238+0000", 2020-06-12T02:38:51.950 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001916484 2020-06-12T02:38:51.950 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.950 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.952 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.952 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.061155+0000", 2020-06-12T02:38:51.952 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9120000000000001e-05 2020-06-12T02:38:51.953 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.953 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.953 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:51.953 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.061234+0000", 2020-06-12T02:38:51.953 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00060383599999999996 2020-06-12T02:38:51.954 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.954 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.954 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:51.954 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.061838+0000", 2020-06-12T02:38:51.954 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010259862999999999 2020-06-12T02:38:51.955 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.955 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.955 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:51.955 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.072098+0000", 2020-06-12T02:38:51.955 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0098176419999999997 2020-06-12T02:38:51.956 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.956 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.956 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:51.956 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.081915+0000", 2020-06-12T02:38:51.956 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3490000000000002e-05 2020-06-12T02:38:51.957 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.957 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.957 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:51.957 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.081999+0000", 2020-06-12T02:38:51.957 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025879849 2020-06-12T02:38:51.958 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.958 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:51.958 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.958 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.958 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.959 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18311 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:38:51.959 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.063458+0000", 2020-06-12T02:38:51.959 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.6365296730000001, 2020-06-12T02:38:51.959 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.035342152000000002, 2020-06-12T02:38:51.959 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:51.960 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:51.960 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:51.960 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:51.960 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:51.960 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18311 2020-06-12T02:38:51.961 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.961 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:51.961 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.961 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:51.961 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.063458+0000", 2020-06-12T02:38:51.962 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:51.962 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.962 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.963 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:51.964 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.063458+0000", 2020-06-12T02:38:51.964 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3773000000000003e-05 2020-06-12T02:38:51.965 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.965 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.965 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:51.965 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.063501+0000", 2020-06-12T02:38:51.966 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0780000000000001e-06 2020-06-12T02:38:51.966 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.966 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.966 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:51.966 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.063508+0000", 2020-06-12T02:38:51.967 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3020000000000001e-05 2020-06-12T02:38:51.967 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.967 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.967 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:51.967 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.063521+0000", 2020-06-12T02:38:51.968 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4839999999999998e-06 2020-06-12T02:38:51.968 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.968 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.968 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:51.968 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.063525+0000", 2020-06-12T02:38:51.968 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0087769890000000007 2020-06-12T02:38:51.969 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.969 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.969 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.969 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.072302+0000", 2020-06-12T02:38:51.970 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3121000000000005e-05 2020-06-12T02:38:51.970 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.970 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.970 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:51.970 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.072395+0000", 2020-06-12T02:38:51.971 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010158355000000001 2020-06-12T02:38:51.971 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.971 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.971 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.971 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.082554+0000", 2020-06-12T02:38:51.972 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5016000000000001e-05 2020-06-12T02:38:51.972 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.972 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.972 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:51.972 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.082629+0000", 2020-06-12T02:38:51.973 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016171316000000002 2020-06-12T02:38:51.973 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.973 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.973 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:51.973 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.098800+0000", 2020-06-12T02:38:51.974 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.035342152000000002 2020-06-12T02:38:51.974 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.974 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:51.974 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.974 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.975 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.975 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18313 3.fs0 3:f83c3dea:::smithi10729407-4:head [append 0~442368] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:38:51.975 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.070249+0000", 2020-06-12T02:38:51.975 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.6297383960000005, 2020-06-12T02:38:51.975 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028853489, 2020-06-12T02:38:51.976 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:51.976 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:51.976 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:51.976 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:51.976 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:51.977 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18313 2020-06-12T02:38:51.977 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.977 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:51.977 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.977 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:51.978 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.070249+0000", 2020-06-12T02:38:51.978 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:51.978 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.978 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.978 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:51.979 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.070249+0000", 2020-06-12T02:38:51.979 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00046159400000000001 2020-06-12T02:38:51.979 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.979 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.979 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:51.980 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.070711+0000", 2020-06-12T02:38:51.980 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2539999999999992e-06 2020-06-12T02:38:51.980 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.980 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.980 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:51.981 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.070719+0000", 2020-06-12T02:38:51.981 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3297e-05 2020-06-12T02:38:51.981 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.981 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.981 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:51.982 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.070732+0000", 2020-06-12T02:38:51.982 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0559999999999999e-06 2020-06-12T02:38:51.982 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.982 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.982 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:51.983 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.070739+0000", 2020-06-12T02:38:51.983 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0052673319999999996 2020-06-12T02:38:51.983 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.983 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.983 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.984 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.076006+0000", 2020-06-12T02:38:51.984 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6925999999999997e-05 2020-06-12T02:38:51.984 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.984 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.984 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:51.985 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.076063+0000", 2020-06-12T02:38:51.985 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0076712209999999998 2020-06-12T02:38:51.985 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.985 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.985 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:51.986 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.083735+0000", 2020-06-12T02:38:51.986 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8089000000000002e-05 2020-06-12T02:38:51.986 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.986 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.986 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:51.987 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.083833+0000", 2020-06-12T02:38:51.987 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054620600000000004 2020-06-12T02:38:51.987 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.987 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.987 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:51.988 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.084379+0000", 2020-06-12T02:38:51.988 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013940945999999999 2020-06-12T02:38:51.988 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.988 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.988 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:51.989 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.098320+0000", 2020-06-12T02:38:51.989 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00072638599999999998 2020-06-12T02:38:51.989 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.989 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.989 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:51.990 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.099046+0000", 2020-06-12T02:38:51.995 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6181999999999998e-05 2020-06-12T02:38:51.995 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.995 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.996 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:51.996 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.099102+0000", 2020-06-12T02:38:51.996 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028853489 2020-06-12T02:38:51.996 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.996 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:51.997 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:51.997 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.997 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:51.997 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18314 3.fs0 3:f83c3dea:::smithi10729407-4:head [append 0~573440] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:38:51.997 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.074005+0000", 2020-06-12T02:38:51.998 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.6259823769999997, 2020-06-12T02:38:51.998 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031442581999999997, 2020-06-12T02:38:51.998 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:51.998 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:51.998 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:51.999 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:51.999 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:51.999 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18314 2020-06-12T02:38:51.999 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:51.999 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:52.000 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.000 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:52.000 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.074005+0000", 2020-06-12T02:38:52.000 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:52.000 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.001 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.001 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:52.001 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.074005+0000", 2020-06-12T02:38:52.001 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00056762000000000002 2020-06-12T02:38:52.002 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.002 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.002 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:52.002 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.074573+0000", 2020-06-12T02:38:52.002 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5400000000000003e-06 2020-06-12T02:38:52.003 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.003 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.003 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:52.003 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.074578+0000", 2020-06-12T02:38:52.003 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1090000000000004e-06 2020-06-12T02:38:52.004 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.004 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.004 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:52.004 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.074587+0000", 2020-06-12T02:38:52.004 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4039999999999997e-06 2020-06-12T02:38:52.005 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.005 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.005 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:52.005 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.074592+0000", 2020-06-12T02:38:52.005 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001538031 2020-06-12T02:38:52.006 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.006 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.006 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:52.006 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.076130+0000", 2020-06-12T02:38:52.007 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5117000000000002e-05 2020-06-12T02:38:52.007 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.007 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.007 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:52.007 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.076175+0000", 2020-06-12T02:38:52.008 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011822262 2020-06-12T02:38:52.008 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.008 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.008 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:52.008 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.087997+0000", 2020-06-12T02:38:52.009 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000112318 2020-06-12T02:38:52.009 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.009 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.009 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:52.010 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.088109+0000", 2020-06-12T02:38:52.010 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00072351100000000003 2020-06-12T02:38:52.010 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.010 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.010 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:52.011 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.088833+0000", 2020-06-12T02:38:52.011 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010307874999999999 2020-06-12T02:38:52.011 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.011 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.011 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:52.012 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.099141+0000", 2020-06-12T02:38:52.012 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.006229605 2020-06-12T02:38:52.012 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.012 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.012 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:52.013 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.105370+0000", 2020-06-12T02:38:52.013 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7189999999999998e-05 2020-06-12T02:38:52.013 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.013 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.013 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:52.014 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.105448+0000", 2020-06-12T02:38:52.014 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031442581999999997 2020-06-12T02:38:52.014 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:52.014 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:52.014 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:52.015 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.015 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.015 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18315 3.fs0 3:f83c3dea:::smithi10729407-4:head [append 0~589824] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:38:52.015 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.077724+0000", 2020-06-12T02:38:52.015 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.6222635180000005, 2020-06-12T02:38:52.016 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.027939051999999999, 2020-06-12T02:38:52.016 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:52.016 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:52.016 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:52.016 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:52.017 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:52.017 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18315 2020-06-12T02:38:52.017 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.017 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:52.017 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.018 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:52.018 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.077724+0000", 2020-06-12T02:38:52.018 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:52.018 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.018 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.019 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:52.019 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.077724+0000", 2020-06-12T02:38:52.019 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00093652299999999998 2020-06-12T02:38:52.019 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.019 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.019 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:52.020 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078660+0000", 2020-06-12T02:38:52.020 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1389999999999995e-06 2020-06-12T02:38:52.020 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.020 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.020 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:52.021 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078669+0000", 2020-06-12T02:38:52.021 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3691e-05 2020-06-12T02:38:52.021 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.021 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.021 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:52.022 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078682+0000", 2020-06-12T02:38:52.022 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0100000000000001e-06 2020-06-12T02:38:52.022 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.022 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.022 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:52.023 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078688+0000", 2020-06-12T02:38:52.023 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1209999999999999e-05 2020-06-12T02:38:52.023 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.023 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.023 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:52.024 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078739+0000", 2020-06-12T02:38:52.024 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000102735 2020-06-12T02:38:52.024 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.024 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.024 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:52.025 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078842+0000", 2020-06-12T02:38:52.025 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01321815 2020-06-12T02:38:52.025 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.026 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.026 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:52.026 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.092060+0000", 2020-06-12T02:38:52.026 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00038608499999999999 2020-06-12T02:38:52.026 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.027 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.027 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:52.027 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.092446+0000", 2020-06-12T02:38:52.027 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00056703299999999999 2020-06-12T02:38:52.027 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.028 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.028 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:52.028 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.093013+0000", 2020-06-12T02:38:52.028 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0061289530000000004 2020-06-12T02:38:52.028 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.029 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.029 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:52.029 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.099142+0000", 2020-06-12T02:38:52.029 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0064547700000000003 2020-06-12T02:38:52.029 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.030 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.030 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:52.030 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.105597+0000", 2020-06-12T02:38:52.030 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5753000000000006e-05 2020-06-12T02:38:52.030 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.031 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.031 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:52.031 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.105663+0000", 2020-06-12T02:38:52.031 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.027939051999999999 2020-06-12T02:38:52.031 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:52.032 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:52.032 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:52.032 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.032 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.032 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18316 3.fs0 3:f83c3dea:::smithi10729407-4:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:38:52.033 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.078752+0000", 2020-06-12T02:38:52.033 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.621235746, 2020-06-12T02:38:52.033 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.029108709999999999, 2020-06-12T02:38:52.033 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:52.033 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:52.034 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:52.034 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:52.034 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:52.034 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18316 2020-06-12T02:38:52.034 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.035 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:52.035 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.035 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:52.035 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078752+0000", 2020-06-12T02:38:52.036 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:52.036 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.036 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.036 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:52.037 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078752+0000", 2020-06-12T02:38:52.037 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8967000000000004e-05 2020-06-12T02:38:52.037 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.037 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.037 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:52.038 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078841+0000", 2020-06-12T02:38:52.038 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3309999999999998e-06 2020-06-12T02:38:52.038 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.038 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.038 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:52.039 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078848+0000", 2020-06-12T02:38:52.039 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.291e-05 2020-06-12T02:38:52.039 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.039 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.039 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:52.040 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078861+0000", 2020-06-12T02:38:52.040 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2800000000000003e-06 2020-06-12T02:38:52.040 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.040 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.040 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:52.041 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078866+0000", 2020-06-12T02:38:52.041 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8543e-05 2020-06-12T02:38:52.041 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.041 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.041 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:52.042 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078905+0000", 2020-06-12T02:38:52.042 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9198999999999998e-05 2020-06-12T02:38:52.042 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.042 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.042 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:52.043 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078994+0000", 2020-06-12T02:38:52.043 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01781437 2020-06-12T02:38:52.043 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.043 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.043 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:52.044 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.096808+0000", 2020-06-12T02:38:52.044 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000115975 2020-06-12T02:38:52.044 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.044 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.044 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:52.045 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.096924+0000", 2020-06-12T02:38:52.045 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00049986300000000002 2020-06-12T02:38:52.045 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.045 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.045 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:52.046 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.097424+0000", 2020-06-12T02:38:52.046 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0017191089999999999 2020-06-12T02:38:52.046 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.046 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.046 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:52.047 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.099143+0000", 2020-06-12T02:38:52.047 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0086177909999999996 2020-06-12T02:38:52.047 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.047 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.047 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:52.048 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.107761+0000", 2020-06-12T02:38:52.048 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9371999999999997e-05 2020-06-12T02:38:52.048 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.048 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.048 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:52.049 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.107860+0000", 2020-06-12T02:38:52.049 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.029108709999999999 2020-06-12T02:38:52.049 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:52.049 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:52.049 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:52.050 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.050 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.050 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18317 3.fs0 3:f83c3dea:::smithi10729407-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:38:52.050 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.078920+0000", 2020-06-12T02:38:52.050 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.6210677760000003, 2020-06-12T02:38:52.051 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031356964000000001, 2020-06-12T02:38:52.051 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:52.051 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:52.051 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:52.051 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:52.052 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:52.052 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18317 2020-06-12T02:38:52.052 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.052 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:52.052 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.053 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:52.053 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078920+0000", 2020-06-12T02:38:52.053 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:52.053 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.053 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.054 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:52.054 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078920+0000", 2020-06-12T02:38:52.054 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6454000000000001e-05 2020-06-12T02:38:52.054 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.054 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.055 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:52.058 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078946+0000", 2020-06-12T02:38:52.058 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8160000000000004e-06 2020-06-12T02:38:52.058 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.059 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.059 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:52.059 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078950+0000", 2020-06-12T02:38:52.059 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6090000000000003e-06 2020-06-12T02:38:52.059 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.059 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.060 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:52.060 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078957+0000", 2020-06-12T02:38:52.060 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8380000000000002e-06 2020-06-12T02:38:52.060 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.060 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.061 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:52.061 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078960+0000", 2020-06-12T02:38:52.061 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8400000000000003e-05 2020-06-12T02:38:52.061 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.061 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.062 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:52.062 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.079019+0000", 2020-06-12T02:38:52.062 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6750999999999998e-05 2020-06-12T02:38:52.062 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.062 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.062 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:52.063 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.079085+0000", 2020-06-12T02:38:52.063 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019272036999999999 2020-06-12T02:38:52.063 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.063 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.063 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:52.064 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.098357+0000", 2020-06-12T02:38:52.064 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4322999999999998e-05 2020-06-12T02:38:52.064 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.064 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.064 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:52.065 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.098452+0000", 2020-06-12T02:38:52.065 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010578025 2020-06-12T02:38:52.066 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.072 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.072 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:52.072 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.109030+0000", 2020-06-12T02:38:52.072 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010395099999999999 2020-06-12T02:38:52.072 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.073 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.073 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:52.073 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.109134+0000", 2020-06-12T02:38:52.073 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00114276 2020-06-12T02:38:52.073 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:52.074 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:52.074 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:52.074 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.110277+0000", 2020-06-12T02:38:52.074 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031356964000000001 2020-06-12T02:38:52.074 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:52.075 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:52.075 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:52.075 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:52.075 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:52.075 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:52.077 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:52.075+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:52.077 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-06-12T02:38:52.178 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:52.175+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:52.200 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:52.200 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:38:52.201 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2020-06-12T02:38:52.201 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:52.209 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-06-12T02:38:52.293 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:52.290+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:52.365 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:52.366 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:38:52.366 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2020-06-12T02:38:52.366 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2020-06-12T02:38:52.366 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:52.374 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-06-12T02:38:52.380 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:52.378+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:52.399 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.4 to 1 (10000) 2020-06-12T02:38:52.410 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd reweight 5 1 2020-06-12T02:38:52.481 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:52.479+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:52.522 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:52.523 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2020-06-12T02:38:52.523 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2020-06-12T02:38:52.523 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2020-06-12T02:38:52.523 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.523 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.3s2 49/21 ECSubWrite(tid=4109, reqid=client.4512.0:8841, at_version=49'773, trim_to=0'0, roll_forward_to=49'771))", 2020-06-12T02:38:52.524 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.822144+0000", 2020-06-12T02:38:52.524 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 40.696615829999999, 2020-06-12T02:38:52.524 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014799214, 2020-06-12T02:38:52.524 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:52.524 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:52.525 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:52.525 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.525 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:52.525 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822144+0000", 2020-06-12T02:38:52.525 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:52.526 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.526 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.526 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:52.526 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822144+0000", 2020-06-12T02:38:52.527 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00061470800000000005 2020-06-12T02:38:52.527 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.527 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.527 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:52.527 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822759+0000", 2020-06-12T02:38:52.528 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.7732000000000001e-05 2020-06-12T02:38:52.528 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.528 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.528 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:52.528 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822786+0000", 2020-06-12T02:38:52.529 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1010999999999999e-05 2020-06-12T02:38:52.529 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.529 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.529 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:52.529 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822807+0000", 2020-06-12T02:38:52.529 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.842e-06 2020-06-12T02:38:52.530 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.530 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.530 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:52.530 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822817+0000", 2020-06-12T02:38:52.530 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00024702300000000002 2020-06-12T02:38:52.531 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.531 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.531 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:52.534 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.823064+0000", 2020-06-12T02:38:52.534 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5207e-05 2020-06-12T02:38:52.534 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.534 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.534 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:52.535 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.823079+0000", 2020-06-12T02:38:52.535 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013806817000000001 2020-06-12T02:38:52.535 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.535 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.535 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:52.536 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.836886+0000", 2020-06-12T02:38:52.536 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6873999999999998e-05 2020-06-12T02:38:52.536 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.536 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.536 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:52.536 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.836943+0000", 2020-06-12T02:38:52.537 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014799214 2020-06-12T02:38:52.537 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:52.538 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:52.538 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:52.538 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.539 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.539 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14657 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~606208] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:52.539 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.209225+0000", 2020-06-12T02:38:52.539 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 21.309534579000001, 2020-06-12T02:38:52.539 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016128905999999998, 2020-06-12T02:38:52.540 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:52.540 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:52.540 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:52.540 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:52.541 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:52.541 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14657 2020-06-12T02:38:52.542 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.542 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:52.542 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.542 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:52.542 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209225+0000", 2020-06-12T02:38:52.543 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:52.543 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.543 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.543 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:52.544 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209225+0000", 2020-06-12T02:38:52.544 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018386699999999999 2020-06-12T02:38:52.544 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.544 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.544 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:52.545 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209409+0000", 2020-06-12T02:38:52.545 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8160000000000001e-06 2020-06-12T02:38:52.545 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.545 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.545 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:52.546 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209414+0000", 2020-06-12T02:38:52.546 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0230000000000004e-06 2020-06-12T02:38:52.546 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.546 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.546 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:52.547 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209421+0000", 2020-06-12T02:38:52.547 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0139999999999999e-06 2020-06-12T02:38:52.547 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.547 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.547 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:52.547 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209424+0000", 2020-06-12T02:38:52.548 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0054798169999999997 2020-06-12T02:38:52.548 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.548 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.548 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:52.548 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.214904+0000", 2020-06-12T02:38:52.549 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123063 2020-06-12T02:38:52.549 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.549 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.549 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:52.549 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215027+0000", 2020-06-12T02:38:52.550 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001159923 2020-06-12T02:38:52.550 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.550 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.550 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:52.550 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.216187+0000", 2020-06-12T02:38:52.550 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2835000000000002e-05 2020-06-12T02:38:52.551 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.551 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.551 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:52.552 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.216270+0000", 2020-06-12T02:38:52.552 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073575799999999998 2020-06-12T02:38:52.552 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.552 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.553 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:52.553 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.217005+0000", 2020-06-12T02:38:52.553 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0082434659999999996 2020-06-12T02:38:52.553 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.553 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.554 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:52.554 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225249+0000", 2020-06-12T02:38:52.554 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0145999999999997e-05 2020-06-12T02:38:52.554 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.554 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.555 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:52.555 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225299+0000", 2020-06-12T02:38:52.555 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5177999999999999e-05 2020-06-12T02:38:52.555 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.555 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.556 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:52.556 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225354+0000", 2020-06-12T02:38:52.556 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016128905999999998 2020-06-12T02:38:52.556 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:52.556 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:52.556 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:52.557 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.557 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.557 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14658 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~598016] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:52.557 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.209455+0000", 2020-06-12T02:38:52.557 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 21.309304825000002, 2020-06-12T02:38:52.558 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016447966000000001, 2020-06-12T02:38:52.558 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:52.558 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:52.558 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:52.558 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:52.559 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:52.559 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14658 2020-06-12T02:38:52.559 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.559 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:52.560 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.560 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:52.560 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209455+0000", 2020-06-12T02:38:52.560 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:52.560 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.560 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.561 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:52.561 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209455+0000", 2020-06-12T02:38:52.561 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037921599999999998 2020-06-12T02:38:52.561 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.562 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.562 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:52.562 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209834+0000", 2020-06-12T02:38:52.562 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8339999999999999e-06 2020-06-12T02:38:52.563 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.563 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.563 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:52.563 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209842+0000", 2020-06-12T02:38:52.563 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2564e-05 2020-06-12T02:38:52.564 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.564 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.564 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:52.564 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209855+0000", 2020-06-12T02:38:52.564 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.135e-06 2020-06-12T02:38:52.565 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.565 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.565 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:52.565 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209860+0000", 2020-06-12T02:38:52.565 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0052530290000000002 2020-06-12T02:38:52.566 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.566 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.566 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:52.566 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215113+0000", 2020-06-12T02:38:52.566 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6229000000000003e-05 2020-06-12T02:38:52.566 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.567 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.567 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:52.567 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215209+0000", 2020-06-12T02:38:52.567 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045765409999999999 2020-06-12T02:38:52.567 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.568 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.568 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:52.568 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.219785+0000", 2020-06-12T02:38:52.568 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000102846 2020-06-12T02:38:52.568 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.568 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.569 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:52.569 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.219888+0000", 2020-06-12T02:38:52.569 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00077088700000000005 2020-06-12T02:38:52.569 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.569 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.570 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:52.570 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.220659+0000", 2020-06-12T02:38:52.570 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046979739999999997 2020-06-12T02:38:52.570 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.570 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.570 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:52.571 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225357+0000", 2020-06-12T02:38:52.571 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00046069600000000002 2020-06-12T02:38:52.571 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.571 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.571 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:52.572 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225818+0000", 2020-06-12T02:38:52.572 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5014999999999998e-05 2020-06-12T02:38:52.572 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.573 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.573 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:52.573 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225903+0000", 2020-06-12T02:38:52.573 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016447966000000001 2020-06-12T02:38:52.573 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:52.573 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:52.574 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:52.574 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.574 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.574 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14659 3.0s0 3:0dddb690:::smithi10729407-10:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:52.574 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.209909+0000", 2020-06-12T02:38:52.575 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 21.308850617000001, 2020-06-12T02:38:52.575 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016618236000000002, 2020-06-12T02:38:52.575 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:52.575 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:52.575 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:52.576 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:52.576 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:52.576 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14659 2020-06-12T02:38:52.576 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.576 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:52.576 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.577 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:52.577 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209909+0000", 2020-06-12T02:38:52.577 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:52.577 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.577 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.578 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:52.578 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209909+0000", 2020-06-12T02:38:52.578 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8085999999999999e-05 2020-06-12T02:38:52.578 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.578 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.578 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:52.579 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209967+0000", 2020-06-12T02:38:52.579 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3169999999999998e-06 2020-06-12T02:38:52.580 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.580 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.581 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:52.581 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209974+0000", 2020-06-12T02:38:52.581 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1246999999999999e-05 2020-06-12T02:38:52.581 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.581 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.582 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:52.583 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209985+0000", 2020-06-12T02:38:52.583 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1439999999999996e-06 2020-06-12T02:38:52.583 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.583 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.583 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:52.584 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209989+0000", 2020-06-12T02:38:52.584 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0052537900000000004 2020-06-12T02:38:52.584 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.584 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.584 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:52.584 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215243+0000", 2020-06-12T02:38:52.585 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6023000000000002e-05 2020-06-12T02:38:52.585 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.585 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.585 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:52.585 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215299+0000", 2020-06-12T02:38:52.586 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0087846910000000007 2020-06-12T02:38:52.586 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.586 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.586 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:52.586 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.224083+0000", 2020-06-12T02:38:52.586 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8504000000000004e-05 2020-06-12T02:38:52.587 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.587 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.587 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:52.587 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.224182+0000", 2020-06-12T02:38:52.587 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044251400000000003 2020-06-12T02:38:52.588 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.588 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.588 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:52.588 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.224624+0000", 2020-06-12T02:38:52.588 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013383290000000001 2020-06-12T02:38:52.588 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.589 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.589 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:52.589 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225963+0000", 2020-06-12T02:38:52.589 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047942699999999999 2020-06-12T02:38:52.589 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.590 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.590 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:52.590 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.226442+0000", 2020-06-12T02:38:52.590 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5164e-05 2020-06-12T02:38:52.590 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.591 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.591 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:52.591 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.226527+0000", 2020-06-12T02:38:52.591 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016618236000000002 2020-06-12T02:38:52.591 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:52.591 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:52.592 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:52.592 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.592 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.592 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14660 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e59)", 2020-06-12T02:38:52.593 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.210020+0000", 2020-06-12T02:38:52.593 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 21.308739638999999, 2020-06-12T02:38:52.593 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017984777, 2020-06-12T02:38:52.593 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:52.593 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:52.594 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:52.594 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:52.594 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:52.594 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14660 2020-06-12T02:38:52.594 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.595 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:52.595 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.595 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:52.595 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210020+0000", 2020-06-12T02:38:52.595 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:52.595 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.596 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.596 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:52.596 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210020+0000", 2020-06-12T02:38:52.596 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2875999999999999e-05 2020-06-12T02:38:52.596 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.597 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.597 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:52.597 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210043+0000", 2020-06-12T02:38:52.597 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2200000000000001e-06 2020-06-12T02:38:52.597 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.597 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.598 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:52.598 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210046+0000", 2020-06-12T02:38:52.598 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7410000000000003e-06 2020-06-12T02:38:52.598 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.598 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.599 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:52.599 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210053+0000", 2020-06-12T02:38:52.599 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5610000000000001e-06 2020-06-12T02:38:52.599 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.599 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.599 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:52.600 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210055+0000", 2020-06-12T02:38:52.600 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0052596409999999998 2020-06-12T02:38:52.600 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.600 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.600 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:52.601 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215315+0000", 2020-06-12T02:38:52.601 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5627999999999997e-05 2020-06-12T02:38:52.601 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.601 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.601 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:52.601 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215371+0000", 2020-06-12T02:38:52.602 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0098069549999999991 2020-06-12T02:38:52.602 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.602 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.602 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:52.602 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225178+0000", 2020-06-12T02:38:52.603 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4530000000000001e-05 2020-06-12T02:38:52.603 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.603 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.603 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:52.604 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225232+0000", 2020-06-12T02:38:52.604 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017497350000000001 2020-06-12T02:38:52.604 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.604 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.604 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:52.604 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.226982+0000", 2020-06-12T02:38:52.605 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8154999999999998e-05 2020-06-12T02:38:52.605 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.605 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.605 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:52.605 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.227080+0000", 2020-06-12T02:38:52.606 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00092473499999999999 2020-06-12T02:38:52.606 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.606 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.606 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:52.606 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.228005+0000", 2020-06-12T02:38:52.607 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017984777 2020-06-12T02:38:52.607 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:52.607 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:52.607 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:52.607 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.608 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.608 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.3s2 59/53 ECSubWrite(tid=5970, reqid=client.4512.0:15226, at_version=59'1316, trim_to=0'0, roll_forward_to=59'1315))", 2020-06-12T02:38:52.608 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.829184+0000", 2020-06-12T02:38:52.608 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 19.689575423000001, 2020-06-12T02:38:52.608 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.081295588000000002, 2020-06-12T02:38:52.608 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:52.609 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:52.609 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:52.609 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.609 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:52.609 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829184+0000", 2020-06-12T02:38:52.610 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:52.610 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.610 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.610 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:52.610 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829184+0000", 2020-06-12T02:38:52.610 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00023814999999999999 2020-06-12T02:38:52.611 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.611 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.611 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:52.611 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829422+0000", 2020-06-12T02:38:52.611 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9768000000000001e-05 2020-06-12T02:38:52.612 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.612 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:52.612 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829442+0000", 2020-06-12T02:38:52.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1022000000000001e-05 2020-06-12T02:38:52.612 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.613 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.614 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:52.614 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829453+0000", 2020-06-12T02:38:52.614 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2759999999999997e-06 2020-06-12T02:38:52.614 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.614 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.615 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:52.615 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829460+0000", 2020-06-12T02:38:52.615 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7284000000000002e-05 2020-06-12T02:38:52.615 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.615 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.616 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:52.616 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829517+0000", 2020-06-12T02:38:52.616 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7028999999999999e-05 2020-06-12T02:38:52.616 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.616 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.617 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:52.617 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829534+0000", 2020-06-12T02:38:52.618 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.080899873999999997 2020-06-12T02:38:52.618 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.618 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.618 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:52.618 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.910434+0000", 2020-06-12T02:38:52.619 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6184999999999999e-05 2020-06-12T02:38:52.619 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.619 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.619 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:52.619 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.910480+0000", 2020-06-12T02:38:52.619 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.081295588000000002 2020-06-12T02:38:52.620 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:52.620 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:52.620 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:52.620 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.620 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.621 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.3s2 59/53 ECSubWrite(tid=5973, reqid=client.4512.0:15229, at_version=59'1319, trim_to=0'0, roll_forward_to=59'1315))", 2020-06-12T02:38:52.621 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.901708+0000", 2020-06-12T02:38:52.621 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 19.617051543999999, 2020-06-12T02:38:52.621 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013662957999999999, 2020-06-12T02:38:52.621 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:52.622 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:52.622 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:52.622 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.622 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:52.622 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901708+0000", 2020-06-12T02:38:52.622 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:52.623 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.623 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.623 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:52.623 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901708+0000", 2020-06-12T02:38:52.624 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7360000000000001e-05 2020-06-12T02:38:52.624 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.624 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.624 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:52.624 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901756+0000", 2020-06-12T02:38:52.624 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4608e-05 2020-06-12T02:38:52.625 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.625 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.625 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:52.625 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901770+0000", 2020-06-12T02:38:52.625 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.219e-05 2020-06-12T02:38:52.626 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.626 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.626 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:52.626 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901782+0000", 2020-06-12T02:38:52.627 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.511e-06 2020-06-12T02:38:52.627 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.627 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.627 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:52.627 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901789+0000", 2020-06-12T02:38:52.628 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.008025562 2020-06-12T02:38:52.628 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.628 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.628 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:52.628 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.909814+0000", 2020-06-12T02:38:52.629 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1472e-05 2020-06-12T02:38:52.629 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.629 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.629 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:52.629 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.909826+0000", 2020-06-12T02:38:52.630 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0055013350000000004 2020-06-12T02:38:52.630 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.630 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.630 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:52.630 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.915327+0000", 2020-06-12T02:38:52.630 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.392e-05 2020-06-12T02:38:52.631 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.631 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.631 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:52.631 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.915371+0000", 2020-06-12T02:38:52.631 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013662957999999999 2020-06-12T02:38:52.632 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:52.632 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:52.632 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:52.632 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.632 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.632 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.3s2 59/53 ECSubWrite(tid=5974, reqid=client.4512.0:15231, at_version=59'1320, trim_to=0'0, roll_forward_to=59'1315))", 2020-06-12T02:38:52.633 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.901839+0000", 2020-06-12T02:38:52.633 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 19.616920946, 2020-06-12T02:38:52.633 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013568525, 2020-06-12T02:38:52.633 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:52.633 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:52.634 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:52.634 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.634 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:52.634 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901839+0000", 2020-06-12T02:38:52.634 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:52.634 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.635 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.635 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:52.635 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901839+0000", 2020-06-12T02:38:52.635 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6489999999999997e-05 2020-06-12T02:38:52.635 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.636 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.636 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:52.636 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901885+0000", 2020-06-12T02:38:52.636 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6600999999999999e-05 2020-06-12T02:38:52.636 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.636 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.637 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:52.637 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901902+0000", 2020-06-12T02:38:52.637 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2427e-05 2020-06-12T02:38:52.637 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.637 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.638 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:52.638 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901914+0000", 2020-06-12T02:38:52.638 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1710000000000001e-06 2020-06-12T02:38:52.638 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.638 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.638 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:52.639 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901919+0000", 2020-06-12T02:38:52.639 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0086285540000000001 2020-06-12T02:38:52.639 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.639 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.639 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:52.639 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.910548+0000", 2020-06-12T02:38:52.640 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0939e-05 2020-06-12T02:38:52.640 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.640 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.640 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:52.640 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.910559+0000", 2020-06-12T02:38:52.641 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0048161899999999997 2020-06-12T02:38:52.641 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.641 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.641 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:52.641 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.915375+0000", 2020-06-12T02:38:52.642 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2153000000000002e-05 2020-06-12T02:38:52.642 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.643 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.643 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:52.643 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.915407+0000", 2020-06-12T02:38:52.643 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013568525 2020-06-12T02:38:52.643 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:52.644 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:52.644 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:52.644 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.644 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.644 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15246 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~671744] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:52.645 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.953872+0000", 2020-06-12T02:38:52.645 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 19.564887880000001, 2020-06-12T02:38:52.645 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015306691000000001, 2020-06-12T02:38:52.645 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:52.645 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:52.646 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:52.646 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:52.646 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:52.646 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15246 2020-06-12T02:38:52.649 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.649 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:52.650 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.650 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:52.650 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.953872+0000", 2020-06-12T02:38:52.650 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:52.650 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.651 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.651 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:52.651 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.953872+0000", 2020-06-12T02:38:52.651 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000232544 2020-06-12T02:38:52.651 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.652 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.652 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:52.652 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954104+0000", 2020-06-12T02:38:52.652 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7513999999999999e-05 2020-06-12T02:38:52.652 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.653 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.653 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:52.653 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954122+0000", 2020-06-12T02:38:52.653 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.844e-05 2020-06-12T02:38:52.654 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.654 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.654 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:52.654 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954140+0000", 2020-06-12T02:38:52.655 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2570000000000007e-06 2020-06-12T02:38:52.655 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.655 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.655 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:52.655 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954149+0000", 2020-06-12T02:38:52.656 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00031049499999999999 2020-06-12T02:38:52.656 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.656 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.656 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:52.656 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954459+0000", 2020-06-12T02:38:52.656 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000118182 2020-06-12T02:38:52.657 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.657 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.657 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:52.657 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954577+0000", 2020-06-12T02:38:52.657 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00071788400000000001 2020-06-12T02:38:52.658 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.658 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.658 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:52.658 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.955295+0000", 2020-06-12T02:38:52.659 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013717859000000001 2020-06-12T02:38:52.662 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.662 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.663 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:52.663 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.969013+0000", 2020-06-12T02:38:52.663 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4895999999999995e-05 2020-06-12T02:38:52.663 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.664 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.664 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:52.664 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.969098+0000", 2020-06-12T02:38:52.664 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0619999999999997e-05 2020-06-12T02:38:52.664 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.665 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.665 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:52.665 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.969179+0000", 2020-06-12T02:38:52.665 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015306691000000001 2020-06-12T02:38:52.665 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:52.666 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:52.666 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:52.666 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.666 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.666 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15277 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~720896] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:52.667 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:33.067109+0000", 2020-06-12T02:38:52.667 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 19.451650820000001, 2020-06-12T02:38:52.667 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015909191999999999, 2020-06-12T02:38:52.667 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:52.667 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:52.668 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:52.668 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:52.668 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:52.668 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15277 2020-06-12T02:38:52.668 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.669 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:52.669 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.669 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:52.669 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067109+0000", 2020-06-12T02:38:52.672 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:52.672 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.673 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.673 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:52.673 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067109+0000", 2020-06-12T02:38:52.673 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018431400000000001 2020-06-12T02:38:52.673 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.674 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.674 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:52.674 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067293+0000", 2020-06-12T02:38:52.674 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1790000000000002e-06 2020-06-12T02:38:52.674 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.674 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.675 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:52.675 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067300+0000", 2020-06-12T02:38:52.675 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5259999999999999e-06 2020-06-12T02:38:52.675 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.675 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.676 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:52.676 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067310+0000", 2020-06-12T02:38:52.676 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8179999999999997e-06 2020-06-12T02:38:52.676 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.676 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.677 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:52.677 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067314+0000", 2020-06-12T02:38:52.677 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6529999999999997e-05 2020-06-12T02:38:52.677 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.677 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.678 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:52.678 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067360+0000", 2020-06-12T02:38:52.678 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012724399999999999 2020-06-12T02:38:52.678 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.678 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.690 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:52.691 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067488+0000", 2020-06-12T02:38:52.691 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0010983989999999999 2020-06-12T02:38:52.691 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.691 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.691 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:52.692 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.068586+0000", 2020-06-12T02:38:52.692 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5165000000000001e-05 2020-06-12T02:38:52.692 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.692 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.692 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:52.693 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.068681+0000", 2020-06-12T02:38:52.694 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086510899999999997 2020-06-12T02:38:52.694 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.695 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.695 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:52.695 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.069546+0000", 2020-06-12T02:38:52.695 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013284429 2020-06-12T02:38:52.695 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.696 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.696 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:52.696 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.082831+0000", 2020-06-12T02:38:52.696 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4197000000000006e-05 2020-06-12T02:38:52.696 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.697 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.697 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:52.697 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.082925+0000", 2020-06-12T02:38:52.697 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3282000000000006e-05 2020-06-12T02:38:52.697 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.698 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.698 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:52.698 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.083018+0000", 2020-06-12T02:38:52.698 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015909191999999999 2020-06-12T02:38:52.698 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:52.699 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:52.700 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:52.700 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.700 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.700 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15331 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~688128] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:52.701 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:33.276748+0000", 2020-06-12T02:38:52.701 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 19.242011307999999, 2020-06-12T02:38:52.701 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016825066999999999, 2020-06-12T02:38:52.701 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:52.701 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:52.702 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:52.702 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:52.702 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:52.702 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15331 2020-06-12T02:38:52.702 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.703 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:52.703 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.703 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:52.703 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276748+0000", 2020-06-12T02:38:52.703 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:52.703 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.704 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.704 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:52.704 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276748+0000", 2020-06-12T02:38:52.704 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016936699999999999 2020-06-12T02:38:52.704 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.705 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.705 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:52.705 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276918+0000", 2020-06-12T02:38:52.705 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6389999999999998e-06 2020-06-12T02:38:52.705 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.707 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.707 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:52.707 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276926+0000", 2020-06-12T02:38:52.707 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3591000000000001e-05 2020-06-12T02:38:52.707 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.708 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.708 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:52.708 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276940+0000", 2020-06-12T02:38:52.708 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0429999999999996e-06 2020-06-12T02:38:52.708 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.709 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.709 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:52.709 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276949+0000", 2020-06-12T02:38:52.709 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8825000000000001e-05 2020-06-12T02:38:52.709 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.709 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.710 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:52.710 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276998+0000", 2020-06-12T02:38:52.710 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6483000000000005e-05 2020-06-12T02:38:52.710 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.710 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.711 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:52.711 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.277084+0000", 2020-06-12T02:38:52.711 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00057545800000000005 2020-06-12T02:38:52.711 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.711 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.712 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:52.712 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.277660+0000", 2020-06-12T02:38:52.712 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015617126 2020-06-12T02:38:52.712 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.712 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.712 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:52.713 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.293277+0000", 2020-06-12T02:38:52.713 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00021931699999999999 2020-06-12T02:38:52.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.713 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.713 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:52.714 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.293496+0000", 2020-06-12T02:38:52.714 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7218000000000005e-05 2020-06-12T02:38:52.714 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.714 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:52.715 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.293573+0000", 2020-06-12T02:38:52.715 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016825066999999999 2020-06-12T02:38:52.715 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:52.715 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:52.715 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:52.716 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.716 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.716 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.7s1 60/37 ECSubWrite(tid=8504, reqid=client.4512.0:16225, at_version=60'1427, trim_to=0'0, roll_forward_to=60'1425))", 2020-06-12T02:38:52.716 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:36.033154+0000", 2020-06-12T02:38:52.716 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 16.485605946, 2020-06-12T02:38:52.717 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016768244000000002, 2020-06-12T02:38:52.717 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:52.717 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:52.717 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:52.718 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.718 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:52.718 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033154+0000", 2020-06-12T02:38:52.718 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:52.718 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.719 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.719 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:52.719 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033154+0000", 2020-06-12T02:38:52.719 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000154444 2020-06-12T02:38:52.719 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.719 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.720 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:52.720 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033308+0000", 2020-06-12T02:38:52.720 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8772000000000002e-05 2020-06-12T02:38:52.720 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.720 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.721 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:52.721 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033337+0000", 2020-06-12T02:38:52.721 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6704000000000001e-05 2020-06-12T02:38:52.721 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.721 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.722 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:52.723 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033354+0000", 2020-06-12T02:38:52.723 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6100000000000002e-06 2020-06-12T02:38:52.724 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.724 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.724 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:52.724 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033360+0000", 2020-06-12T02:38:52.724 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000173893 2020-06-12T02:38:52.725 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.725 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.725 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:52.725 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033534+0000", 2020-06-12T02:38:52.725 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1929999999999999e-05 2020-06-12T02:38:52.726 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.726 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.726 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:52.726 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033546+0000", 2020-06-12T02:38:52.727 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016272066000000002 2020-06-12T02:38:52.727 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.727 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.727 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:52.728 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.049818+0000", 2020-06-12T02:38:52.728 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103825 2020-06-12T02:38:52.728 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.728 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.729 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:52.729 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.049922+0000", 2020-06-12T02:38:52.729 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016768244000000002 2020-06-12T02:38:52.729 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:52.729 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:52.730 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:52.730 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.730 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.730 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:17973 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~507904] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:38:52.730 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:41.095893+0000", 2020-06-12T02:38:52.730 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 11.422866393, 2020-06-12T02:38:52.731 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013655560000000001, 2020-06-12T02:38:52.731 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:52.731 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:52.731 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:52.731 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:52.732 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:52.732 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 17973 2020-06-12T02:38:52.732 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.732 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:52.732 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.733 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:52.733 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.095893+0000", 2020-06-12T02:38:52.733 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:52.733 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.733 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.733 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:52.734 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.095893+0000", 2020-06-12T02:38:52.734 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018351000000000001 2020-06-12T02:38:52.734 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.734 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.734 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:52.735 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.096077+0000", 2020-06-12T02:38:52.735 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1559999999999995e-06 2020-06-12T02:38:52.735 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.735 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.735 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:52.736 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.096085+0000", 2020-06-12T02:38:52.736 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3764999999999999e-05 2020-06-12T02:38:52.736 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.736 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.736 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:52.736 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.096099+0000", 2020-06-12T02:38:52.737 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0460000000000002e-06 2020-06-12T02:38:52.737 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.737 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.737 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:52.737 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.096105+0000", 2020-06-12T02:38:52.738 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6931000000000002e-05 2020-06-12T02:38:52.738 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.738 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.738 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:52.738 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.096152+0000", 2020-06-12T02:38:52.739 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010251600000000001 2020-06-12T02:38:52.739 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.739 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.739 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:52.739 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.096254+0000", 2020-06-12T02:38:52.740 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072763000000000005 2020-06-12T02:38:52.740 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.740 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.740 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:52.740 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.096982+0000", 2020-06-12T02:38:52.740 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012262115000000001 2020-06-12T02:38:52.741 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.741 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.741 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:52.741 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.109244+0000", 2020-06-12T02:38:52.741 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1791000000000003e-05 2020-06-12T02:38:52.742 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.742 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.742 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:52.742 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.109336+0000", 2020-06-12T02:38:52.742 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0002131 2020-06-12T02:38:52.743 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.743 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.743 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:52.743 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.109549+0000", 2020-06-12T02:38:52.743 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013655560000000001 2020-06-12T02:38:52.744 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:52.744 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:52.744 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:52.744 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.744 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.745 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:17977 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:38:52.745 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:41.103849+0000", 2020-06-12T02:38:52.745 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 11.414910764, 2020-06-12T02:38:52.745 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015465273, 2020-06-12T02:38:52.745 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:52.746 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:52.746 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:52.746 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:52.746 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:52.746 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 17977 2020-06-12T02:38:52.746 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.747 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:52.747 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.747 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:52.747 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.103849+0000", 2020-06-12T02:38:52.747 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:52.748 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.748 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.748 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:52.748 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.103849+0000", 2020-06-12T02:38:52.748 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1254e-05 2020-06-12T02:38:52.749 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.749 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.749 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:52.749 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.103870+0000", 2020-06-12T02:38:52.749 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9869999999999999e-06 2020-06-12T02:38:52.750 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.750 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.750 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:52.750 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.103873+0000", 2020-06-12T02:38:52.750 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6159999999999998e-06 2020-06-12T02:38:52.750 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.751 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.751 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:52.751 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.103880+0000", 2020-06-12T02:38:52.751 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0899999999999999e-06 2020-06-12T02:38:52.751 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.752 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.752 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:52.752 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.103882+0000", 2020-06-12T02:38:52.752 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0066572259999999996 2020-06-12T02:38:52.752 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.753 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.753 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:52.753 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.110539+0000", 2020-06-12T02:38:52.753 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5937000000000003e-05 2020-06-12T02:38:52.753 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.754 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.754 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:52.754 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.110595+0000", 2020-06-12T02:38:52.754 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0066983820000000001 2020-06-12T02:38:52.754 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.755 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.755 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:52.755 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.117293+0000", 2020-06-12T02:38:52.755 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103828 2020-06-12T02:38:52.755 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.756 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.756 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:52.756 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.117397+0000", 2020-06-12T02:38:52.756 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0019169530000000001 2020-06-12T02:38:52.757 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.757 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.757 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:52.757 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.119314+0000", 2020-06-12T02:38:52.757 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015465273 2020-06-12T02:38:52.758 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:52.758 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:52.758 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:52.758 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.758 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.759 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:17998 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:38:52.759 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:41.190236+0000", 2020-06-12T02:38:52.759 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 11.328523709000001, 2020-06-12T02:38:52.759 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013903766, 2020-06-12T02:38:52.759 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:52.760 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:52.760 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:52.760 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:52.760 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:52.760 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 17998 2020-06-12T02:38:52.761 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.762 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:52.762 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.762 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:52.762 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.190236+0000", 2020-06-12T02:38:52.763 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:52.763 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.763 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.763 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:52.763 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.190236+0000", 2020-06-12T02:38:52.764 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.207e-05 2020-06-12T02:38:52.764 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.764 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.765 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:52.765 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.190258+0000", 2020-06-12T02:38:52.765 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9629999999999998e-06 2020-06-12T02:38:52.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.766 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.766 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:52.766 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.190261+0000", 2020-06-12T02:38:52.766 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.409e-06 2020-06-12T02:38:52.767 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.767 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.767 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:52.767 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.190267+0000", 2020-06-12T02:38:52.767 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2189999999999998e-06 2020-06-12T02:38:52.768 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.768 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.768 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:52.768 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.190270+0000", 2020-06-12T02:38:52.768 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0089464690000000003 2020-06-12T02:38:52.769 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.769 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.769 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:52.769 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.199216+0000", 2020-06-12T02:38:52.769 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010023500000000001 2020-06-12T02:38:52.769 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.770 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.770 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:52.771 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.199316+0000", 2020-06-12T02:38:52.771 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0038963800000000001 2020-06-12T02:38:52.771 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.771 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.771 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:52.772 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.203213+0000", 2020-06-12T02:38:52.772 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6963000000000003e-05 2020-06-12T02:38:52.772 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.772 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.772 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:52.773 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.203270+0000", 2020-06-12T02:38:52.773 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00087005800000000005 2020-06-12T02:38:52.773 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.773 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.773 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:52.774 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.204140+0000", 2020-06-12T02:38:52.774 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013903766 2020-06-12T02:38:52.774 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:52.774 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:52.774 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:52.774 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.775 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.7s1 62/37 ECSubWrite(tid=9513, reqid=client.4512.0:18266, at_version=0'0, trim_to=0'0, roll_forward_to=62'1601))", 2020-06-12T02:38:52.775 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:41.904485+0000", 2020-06-12T02:38:52.775 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 10.614274612000001, 2020-06-12T02:38:52.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01412036, 2020-06-12T02:38:52.776 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:52.776 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:52.776 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:52.776 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.776 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:52.776 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904485+0000", 2020-06-12T02:38:52.777 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:52.777 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.777 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.777 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:52.777 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904485+0000", 2020-06-12T02:38:52.778 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.345e-05 2020-06-12T02:38:52.778 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.778 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.778 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:52.778 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904519+0000", 2020-06-12T02:38:52.779 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0869999999999999e-05 2020-06-12T02:38:52.779 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.779 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.780 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:52.780 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904529+0000", 2020-06-12T02:38:52.780 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1248e-05 2020-06-12T02:38:52.780 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.780 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.781 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:52.781 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904541+0000", 2020-06-12T02:38:52.781 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0730999999999999e-05 2020-06-12T02:38:52.781 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.781 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.781 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:52.782 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904551+0000", 2020-06-12T02:38:52.782 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0985999999999997e-05 2020-06-12T02:38:52.782 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.782 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.782 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:52.783 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904592+0000", 2020-06-12T02:38:52.783 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2255999999999999e-05 2020-06-12T02:38:52.783 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.783 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.783 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:52.784 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904605+0000", 2020-06-12T02:38:52.784 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013935858000000001 2020-06-12T02:38:52.784 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.784 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.784 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:52.784 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.918541+0000", 2020-06-12T02:38:52.785 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4961000000000004e-05 2020-06-12T02:38:52.785 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.785 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.785 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:52.785 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.918605+0000", 2020-06-12T02:38:52.786 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01412036 2020-06-12T02:38:52.786 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:52.786 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:52.786 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:52.786 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.787 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.787 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "pg_update_log_missing(3.3s2 epoch 62/53 rep_tid 6734 entries 62'1548 (0'0) error 3:ce45c70a:::smithi10729407-7:head by client.4512.0:18270 0.000000 -2 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 62'1546)", 2020-06-12T02:38:52.787 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:41.904668+0000", 2020-06-12T02:38:52.787 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 10.614092076, 2020-06-12T02:38:52.787 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01430155, 2020-06-12T02:38:52.788 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:52.788 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:52.788 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:52.788 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.789 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:52.789 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904668+0000", 2020-06-12T02:38:52.789 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:52.789 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.790 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.790 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:52.790 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904668+0000", 2020-06-12T02:38:52.790 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1882000000000001e-05 2020-06-12T02:38:52.790 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.790 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.791 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:52.791 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904730+0000", 2020-06-12T02:38:52.791 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0861e-05 2020-06-12T02:38:52.791 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.791 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.792 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:52.792 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904740+0000", 2020-06-12T02:38:52.792 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0406000000000001e-05 2020-06-12T02:38:52.792 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.792 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.793 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:52.793 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904761+0000", 2020-06-12T02:38:52.793 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0339999999999997e-06 2020-06-12T02:38:52.793 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.793 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.794 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:52.794 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904769+0000", 2020-06-12T02:38:52.794 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3888999999999995e-05 2020-06-12T02:38:52.794 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.794 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.795 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:52.795 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904863+0000", 2020-06-12T02:38:52.795 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014106478 2020-06-12T02:38:52.795 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.795 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.795 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:52.796 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.918969+0000", 2020-06-12T02:38:52.796 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01430155 2020-06-12T02:38:52.796 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:52.796 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:52.796 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:52.797 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.797 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.797 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.fs2 62/37 ECSubWrite(tid=3343, reqid=client.4512.0:18307, at_version=62'1623, trim_to=0'0, roll_forward_to=62'1622))", 2020-06-12T02:38:52.798 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:42.062050+0000", 2020-06-12T02:38:52.798 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 10.456709412, 2020-06-12T02:38:52.798 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019528429999999999, 2020-06-12T02:38:52.798 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:52.798 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:52.798 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:52.799 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.799 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:52.799 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062050+0000", 2020-06-12T02:38:52.799 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:52.799 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.800 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.800 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:52.800 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062050+0000", 2020-06-12T02:38:52.800 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00022828799999999999 2020-06-12T02:38:52.800 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.801 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.801 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:52.801 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062279+0000", 2020-06-12T02:38:52.801 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2115000000000001e-05 2020-06-12T02:38:52.801 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.802 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.802 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:52.802 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062301+0000", 2020-06-12T02:38:52.802 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9301e-05 2020-06-12T02:38:52.802 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.802 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.803 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:52.803 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062320+0000", 2020-06-12T02:38:52.803 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2811e-05 2020-06-12T02:38:52.803 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.803 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.804 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:52.804 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062333+0000", 2020-06-12T02:38:52.804 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8856e-05 2020-06-12T02:38:52.804 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.804 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.804 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:52.805 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062392+0000", 2020-06-12T02:38:52.805 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1246999999999999e-05 2020-06-12T02:38:52.805 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.805 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.805 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:52.806 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062403+0000", 2020-06-12T02:38:52.806 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019095523 2020-06-12T02:38:52.806 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.806 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.806 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:52.807 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.081498+0000", 2020-06-12T02:38:52.807 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0289000000000003e-05 2020-06-12T02:38:52.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.807 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.807 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:52.807 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.081579+0000", 2020-06-12T02:38:52.808 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019528429999999999 2020-06-12T02:38:52.808 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:52.808 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:52.808 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:52.809 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.809 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.809 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.fs2 62/37 ECSubWrite(tid=3344, reqid=client.4512.0:18308, at_version=62'1624, trim_to=0'0, roll_forward_to=62'1622))", 2020-06-12T02:38:52.809 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:42.064848+0000", 2020-06-12T02:38:52.809 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 10.453911427, 2020-06-12T02:38:52.810 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01677273, 2020-06-12T02:38:52.810 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:52.810 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:52.810 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:52.810 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.811 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:52.811 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.064848+0000", 2020-06-12T02:38:52.811 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:52.811 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.811 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.812 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:52.812 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.064848+0000", 2020-06-12T02:38:52.812 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000383655 2020-06-12T02:38:52.812 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.812 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.812 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:52.813 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.065232+0000", 2020-06-12T02:38:52.813 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3065999999999999e-05 2020-06-12T02:38:52.813 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.813 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.813 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:52.814 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.065255+0000", 2020-06-12T02:38:52.814 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7142999999999998e-05 2020-06-12T02:38:52.814 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.814 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.814 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:52.814 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.065272+0000", 2020-06-12T02:38:52.815 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7750000000000004e-06 2020-06-12T02:38:52.815 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.815 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.816 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:52.816 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.065279+0000", 2020-06-12T02:38:52.816 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00091758699999999998 2020-06-12T02:38:52.816 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.816 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.817 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:52.817 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.066197+0000", 2020-06-12T02:38:52.817 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4158999999999999e-05 2020-06-12T02:38:52.817 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.817 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.818 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:52.818 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.066211+0000", 2020-06-12T02:38:52.818 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015373370000000001 2020-06-12T02:38:52.818 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.818 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.818 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:52.819 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.081584+0000", 2020-06-12T02:38:52.819 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6974999999999998e-05 2020-06-12T02:38:52.819 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.819 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.819 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:52.820 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.081621+0000", 2020-06-12T02:38:52.820 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01677273 2020-06-12T02:38:52.820 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:52.820 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:52.820 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:52.821 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.821 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.821 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.3s2 62/53 ECSubWrite(tid=6737, reqid=client.4512.0:18319, at_version=62'1551, trim_to=0'0, roll_forward_to=62'1548))", 2020-06-12T02:38:52.821 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:42.089962+0000", 2020-06-12T02:38:52.821 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 10.428797789000001, 2020-06-12T02:38:52.822 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015432937000000001, 2020-06-12T02:38:52.822 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:52.822 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:52.823 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:52.823 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.823 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:52.823 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.089962+0000", 2020-06-12T02:38:52.824 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:52.824 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.824 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.824 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:52.824 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.089962+0000", 2020-06-12T02:38:52.825 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00068051099999999996 2020-06-12T02:38:52.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.825 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.825 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:52.825 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.090642+0000", 2020-06-12T02:38:52.826 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8766999999999998e-05 2020-06-12T02:38:52.826 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.826 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.826 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:52.826 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.090661+0000", 2020-06-12T02:38:52.827 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6461e-05 2020-06-12T02:38:52.827 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.827 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.827 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:52.828 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.090678+0000", 2020-06-12T02:38:52.828 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0580000000000003e-06 2020-06-12T02:38:52.828 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.828 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.828 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:52.829 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.090684+0000", 2020-06-12T02:38:52.829 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000104752 2020-06-12T02:38:52.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.829 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.830 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:52.830 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.090789+0000", 2020-06-12T02:38:52.830 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0767e-05 2020-06-12T02:38:52.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.830 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.831 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:52.831 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.090799+0000", 2020-06-12T02:38:52.831 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014539606 2020-06-12T02:38:52.831 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.831 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.831 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:52.832 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.105339+0000", 2020-06-12T02:38:52.832 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6014999999999998e-05 2020-06-12T02:38:52.832 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:52.833 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:52.833 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:52.833 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.105395+0000", 2020-06-12T02:38:52.833 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015432937000000001 2020-06-12T02:38:52.834 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:52.834 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:52.834 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:52.834 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:52.834 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:52.835 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:52.835 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-06-12T02:38:52.837 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-06-12T02:38:52.838 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:52.834+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:52.937 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:52.935+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:52.998 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:52.999 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:38:52.999 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2020-06-12T02:38:52.999 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:53.008 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-06-12T02:38:53.012 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:38:53.033 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:38:53.043 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:53.037+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:53.062 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:38:53.074 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:38:53.094 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:38:53.115 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:38:53.128 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:53.128 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:53.136 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:38:53.137 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-06-12T02:38:53.157 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2020-06-12T02:38:53.158 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:53.155+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:53.158 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-06-12T02:38:53.159 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-06-12T02:38:53.159 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-06-12T02:38:53.159 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-06-12T02:38:53.159 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-06-12T02:38:53.159 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-06-12T02:38:53.160 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-06-12T02:38:53.160 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-06-12T02:38:53.242 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:53.240+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:53.272 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:53.273 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:53.281 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-06-12T02:38:53.343 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:53.340+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:53.405 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.5 to 1 (10000) 2020-06-12T02:38:53.406 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:53.407 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:53.412 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd unset noscrub 2020-06-12T02:38:53.413 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd reweight 6 1 2020-06-12T02:38:53.429 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-06-12T02:38:53.465 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:53.462+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:53.545 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:53.543+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:53.589 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:53.590 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:53.599 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-06-12T02:38:53.646 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:53.643+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:53.752 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:53.745+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:53.759 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:53.759 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:53.769 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-06-12T02:38:53.851 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:53.847+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:53.887 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:53.888 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:53.896 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-06-12T02:38:53.951 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:38:53.948+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:38:54.052 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:54.050+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:54.052 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:54.053 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:54.061 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-06-12T02:38:54.153 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:54.151+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:54.180 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:54.180 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:54.187 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-06-12T02:38:54.254 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:54.252+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:54.319 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:54.320 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:54.324 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-06-12T02:38:54.367 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:54.363+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:54.456 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:54.454+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:54.483 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:54.484 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:54.491 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-06-12T02:38:54.530 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2020-06-12T02:38:54.532 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.6 to 1 (10000) 2020-06-12T02:38:54.542 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd reweight 7 1 2020-06-12T02:38:54.582 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:54.580+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:54.610 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:54.611 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:54.619 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-06-12T02:38:54.658 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:54.656+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:54.736 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:54.737 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:54.745 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-06-12T02:38:54.789 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:54.787+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:54.860 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:54.858+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:54.861 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:54.861 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:54.870 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-06-12T02:38:54.961 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:54.959+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:55.025 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:55.026 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:55.035 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-06-12T02:38:55.076 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:55.074+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:55.165 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:55.161+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:55.168 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-06-12T02:38:55.192 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:55.193 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:55.207 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-06-12T02:38:55.269 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:55.262+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:55.300 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:38:55.319 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:38:55.351 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:55.352 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:55.352 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-06-12T02:38:55.353 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:38:55.360 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:38:55.365 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:55.363+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:55.382 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:38:55.399 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:38:55.419 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:38:55.439 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2020-06-12T02:38:55.440 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-06-12T02:38:55.440 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-06-12T02:38:55.440 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-06-12T02:38:55.441 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-06-12T02:38:55.441 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-06-12T02:38:55.441 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-06-12T02:38:55.442 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-06-12T02:38:55.442 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-06-12T02:38:55.467 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:55.464+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:55.513 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:55.514 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:38:55.514 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:38:55.514 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:38:55.515 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.515 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:4975 3.as0 3:5732055a:::smithi10729407-8:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e30)", 2020-06-12T02:38:55.515 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:43.721794+0000", 2020-06-12T02:38:55.516 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 71.785033024000001, 2020-06-12T02:38:55.516 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.1825135, 2020-06-12T02:38:55.516 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:55.517 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:55.522 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:55.522 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:55.522 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:55.523 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 4975 2020-06-12T02:38:55.523 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.523 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:55.524 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.524 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:55.524 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721794+0000", 2020-06-12T02:38:55.525 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:55.525 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.525 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.526 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:55.526 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721794+0000", 2020-06-12T02:38:55.526 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4542e-05 2020-06-12T02:38:55.527 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.527 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.527 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:55.528 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721818+0000", 2020-06-12T02:38:55.528 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9169999999999997e-06 2020-06-12T02:38:55.528 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.530 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.530 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:55.530 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721823+0000", 2020-06-12T02:38:55.531 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1184e-05 2020-06-12T02:38:55.531 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.531 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.532 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:55.532 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721834+0000", 2020-06-12T02:38:55.532 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.254e-06 2020-06-12T02:38:55.533 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.533 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.533 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:55.534 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721840+0000", 2020-06-12T02:38:55.534 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1075999999999998e-05 2020-06-12T02:38:55.534 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.535 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.535 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:55.535 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721881+0000", 2020-06-12T02:38:55.536 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7416999999999995e-05 2020-06-12T02:38:55.536 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.536 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.537 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:55.537 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721948+0000", 2020-06-12T02:38:55.537 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.18235910999999999 2020-06-12T02:38:55.538 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.545 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.545 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:55.546 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.904307+0000", 2020-06-12T02:38:55.546 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.1825135 2020-06-12T02:38:55.546 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:55.547 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:55.547 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:55.547 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.548 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.548 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11914 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e56)", 2020-06-12T02:38:55.548 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:23.508790+0000", 2020-06-12T02:38:55.549 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 31.998036171999999, 2020-06-12T02:38:55.549 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026541928999999999, 2020-06-12T02:38:55.549 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:55.550 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:55.550 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:55.550 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:55.551 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:55.551 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11914 2020-06-12T02:38:55.551 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.552 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:55.552 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.552 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:55.553 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508790+0000", 2020-06-12T02:38:55.553 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:55.554 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.555 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.556 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:55.556 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508790+0000", 2020-06-12T02:38:55.557 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0872999999999999e-05 2020-06-12T02:38:55.557 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.557 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.558 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:55.558 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508841+0000", 2020-06-12T02:38:55.558 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9179999999999994e-06 2020-06-12T02:38:55.559 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.559 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.559 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:55.560 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508849+0000", 2020-06-12T02:38:55.560 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5551999999999999e-05 2020-06-12T02:38:55.560 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.561 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.561 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:55.561 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508865+0000", 2020-06-12T02:38:55.562 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0629999999999996e-06 2020-06-12T02:38:55.562 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.562 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.563 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:55.563 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508870+0000", 2020-06-12T02:38:55.563 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010597488 2020-06-12T02:38:55.564 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.564 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.564 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:55.565 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.519467+0000", 2020-06-12T02:38:55.566 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.6083999999999997e-05 2020-06-12T02:38:55.566 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.566 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.567 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:55.567 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.519563+0000", 2020-06-12T02:38:55.567 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0078077800000000003 2020-06-12T02:38:55.567 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.567 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.567 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:55.568 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.527371+0000", 2020-06-12T02:38:55.568 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6198999999999993e-05 2020-06-12T02:38:55.568 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.568 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.568 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:55.569 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.527447+0000", 2020-06-12T02:38:55.569 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0078849720000000005 2020-06-12T02:38:55.569 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.569 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.569 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:55.570 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.535332+0000", 2020-06-12T02:38:55.570 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026541928999999999 2020-06-12T02:38:55.570 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:55.570 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:55.570 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:55.571 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.571 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.571 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13705 3.as0 3:5cb723d2:::smithi10729407-15:head [append 0~745472,stat] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:38:55.571 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.390499+0000", 2020-06-12T02:38:55.571 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 27.116327478999999, 2020-06-12T02:38:55.572 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.067319320000000002, 2020-06-12T02:38:55.572 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:55.572 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:55.572 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:55.572 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:55.572 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:55.573 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13705 2020-06-12T02:38:55.573 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.573 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:55.573 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.573 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:55.574 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.390499+0000", 2020-06-12T02:38:55.574 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:55.574 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.574 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.574 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:55.574 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.390499+0000", 2020-06-12T02:38:55.575 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00066732599999999999 2020-06-12T02:38:55.575 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.575 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.576 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:55.576 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391167+0000", 2020-06-12T02:38:55.576 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3609999999999997e-06 2020-06-12T02:38:55.577 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.577 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.577 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:55.577 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391173+0000", 2020-06-12T02:38:55.577 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0399e-05 2020-06-12T02:38:55.578 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.578 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.578 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:55.578 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391183+0000", 2020-06-12T02:38:55.578 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6480000000000002e-06 2020-06-12T02:38:55.579 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.579 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.579 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:55.579 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391188+0000", 2020-06-12T02:38:55.579 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4150999999999999e-05 2020-06-12T02:38:55.580 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.580 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.580 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:55.580 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391222+0000", 2020-06-12T02:38:55.580 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9450999999999996e-05 2020-06-12T02:38:55.580 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.581 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.581 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:55.581 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391292+0000", 2020-06-12T02:38:55.581 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054214300000000003 2020-06-12T02:38:55.581 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.582 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.582 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:55.582 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391834+0000", 2020-06-12T02:38:55.582 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0037312069999999998 2020-06-12T02:38:55.582 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.582 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.583 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:55.583 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.395565+0000", 2020-06-12T02:38:55.583 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001050194 2020-06-12T02:38:55.583 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.584 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.584 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:55.584 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396615+0000", 2020-06-12T02:38:55.584 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.061203439999999998 2020-06-12T02:38:55.584 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.585 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.585 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:55.585 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.457819+0000", 2020-06-12T02:38:55.585 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.067319320000000002 2020-06-12T02:38:55.585 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:55.586 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:55.586 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:55.586 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.586 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.587 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13706 3.as0 3:5cb723d2:::smithi10729407-15:head [append 0~434176] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:38:55.587 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.393133+0000", 2020-06-12T02:38:55.587 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 27.113693211000001, 2020-06-12T02:38:55.587 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.078158561000000001, 2020-06-12T02:38:55.587 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:55.588 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:55.588 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:55.588 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:55.589 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:55.589 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13706 2020-06-12T02:38:55.589 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.590 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:55.590 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.590 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:55.590 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393133+0000", 2020-06-12T02:38:55.591 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:55.591 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.591 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.591 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:55.591 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393133+0000", 2020-06-12T02:38:55.592 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00050415499999999995 2020-06-12T02:38:55.592 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.592 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.592 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:55.592 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393638+0000", 2020-06-12T02:38:55.593 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5010000000000004e-06 2020-06-12T02:38:55.593 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.593 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.593 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:55.593 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393646+0000", 2020-06-12T02:38:55.594 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4183e-05 2020-06-12T02:38:55.594 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.594 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.594 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:55.594 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393660+0000", 2020-06-12T02:38:55.594 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6069999999999996e-06 2020-06-12T02:38:55.595 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.595 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.595 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:55.595 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393667+0000", 2020-06-12T02:38:55.595 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001363569 2020-06-12T02:38:55.596 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.596 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.596 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:55.596 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.395030+0000", 2020-06-12T02:38:55.596 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012127 2020-06-12T02:38:55.597 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.597 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.597 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:55.597 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.395152+0000", 2020-06-12T02:38:55.597 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.063640503000000001 2020-06-12T02:38:55.597 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.598 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.598 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:55.598 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458792+0000", 2020-06-12T02:38:55.598 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010636 2020-06-12T02:38:55.599 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.599 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.599 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:55.599 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458899+0000", 2020-06-12T02:38:55.600 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00076074200000000002 2020-06-12T02:38:55.600 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.600 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.600 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:55.600 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459659+0000", 2020-06-12T02:38:55.601 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00772984 2020-06-12T02:38:55.601 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.601 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.602 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:55.602 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.467389+0000", 2020-06-12T02:38:55.602 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003815751 2020-06-12T02:38:55.603 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.603 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.603 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:55.603 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471205+0000", 2020-06-12T02:38:55.603 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.708e-05 2020-06-12T02:38:55.604 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.604 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.604 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:55.604 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471292+0000", 2020-06-12T02:38:55.604 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.078158561000000001 2020-06-12T02:38:55.604 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:55.605 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:55.605 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:55.605 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.605 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.605 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13707 3.as0 3:5cb723d2:::smithi10729407-15:head [append 0~565248] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:38:55.606 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.396173+0000", 2020-06-12T02:38:55.606 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 27.110653268, 2020-06-12T02:38:55.606 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.075298341000000005, 2020-06-12T02:38:55.606 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:55.606 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:55.607 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:55.607 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:55.607 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:55.607 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13707 2020-06-12T02:38:55.607 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.607 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:55.608 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.608 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:55.608 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396173+0000", 2020-06-12T02:38:55.608 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:55.608 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.609 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.609 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:55.609 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396173+0000", 2020-06-12T02:38:55.609 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.061779787000000003 2020-06-12T02:38:55.609 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.609 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.610 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:55.610 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.457953+0000", 2020-06-12T02:38:55.610 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5830000000000007e-06 2020-06-12T02:38:55.610 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.610 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.611 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:55.611 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.457962+0000", 2020-06-12T02:38:55.611 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3029e-05 2020-06-12T02:38:55.611 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.611 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.611 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:55.612 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.457975+0000", 2020-06-12T02:38:55.612 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.233e-06 2020-06-12T02:38:55.612 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.612 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.612 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:55.613 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.457979+0000", 2020-06-12T02:38:55.613 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0044352510000000003 2020-06-12T02:38:55.613 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.613 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.613 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:55.613 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.462414+0000", 2020-06-12T02:38:55.614 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012825799999999999 2020-06-12T02:38:55.614 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.614 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.614 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:55.614 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.462543+0000", 2020-06-12T02:38:55.615 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00077227899999999996 2020-06-12T02:38:55.615 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.615 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.615 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:55.615 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.463315+0000", 2020-06-12T02:38:55.616 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0079912390000000007 2020-06-12T02:38:55.616 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.616 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.616 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:55.617 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471306+0000", 2020-06-12T02:38:55.617 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000118674 2020-06-12T02:38:55.617 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.617 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.617 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:55.618 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471425+0000", 2020-06-12T02:38:55.618 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7008000000000001e-05 2020-06-12T02:38:55.618 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.618 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.618 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:55.618 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471472+0000", 2020-06-12T02:38:55.619 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.075298341000000005 2020-06-12T02:38:55.619 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:55.619 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:55.619 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:55.619 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.620 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.620 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWriteReply(3.as0 58/53 ECSubWriteReply(tid=1890, last_complete=58'1184, committed=1, applied=1))", 2020-06-12T02:38:55.620 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.396464+0000", 2020-06-12T02:38:55.620 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 27.110362388999999, 2020-06-12T02:38:55.620 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.062264803000000001, 2020-06-12T02:38:55.620 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:55.621 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:38:55.621 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:55.621 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.621 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:55.621 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396464+0000", 2020-06-12T02:38:55.622 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:55.622 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.622 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.622 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:55.622 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396464+0000", 2020-06-12T02:38:55.623 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9425999999999999e-05 2020-06-12T02:38:55.623 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.623 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.623 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:55.624 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396494+0000", 2020-06-12T02:38:55.624 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1610000000000002e-06 2020-06-12T02:38:55.624 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.624 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.624 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:55.624 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396499+0000", 2020-06-12T02:38:55.625 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2789999999999998e-06 2020-06-12T02:38:55.625 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.625 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.625 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:55.625 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396507+0000", 2020-06-12T02:38:55.626 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1389999999999997e-06 2020-06-12T02:38:55.626 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.626 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.626 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:55.626 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396511+0000", 2020-06-12T02:38:55.627 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2199000000000001e-05 2020-06-12T02:38:55.627 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.627 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.628 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:55.628 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396543+0000", 2020-06-12T02:38:55.628 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.062185599000000001 2020-06-12T02:38:55.628 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.628 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.629 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:55.629 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458729+0000", 2020-06-12T02:38:55.629 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.062264803000000001 2020-06-12T02:38:55.629 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:55.629 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:55.630 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:55.630 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.630 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.630 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13710 3.as0 3:5cb723d2:::smithi10729407-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e58)", 2020-06-12T02:38:55.630 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.458457+0000", 2020-06-12T02:38:55.630 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 27.048369219000001, 2020-06-12T02:38:55.631 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028102095000000001, 2020-06-12T02:38:55.631 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:55.631 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:55.631 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:55.631 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:55.632 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:55.632 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13710 2020-06-12T02:38:55.632 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.632 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:55.632 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.632 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:55.633 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458457+0000", 2020-06-12T02:38:55.633 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:55.633 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.633 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.633 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:55.634 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458457+0000", 2020-06-12T02:38:55.634 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7650000000000001e-05 2020-06-12T02:38:55.634 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.634 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.634 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:55.634 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458485+0000", 2020-06-12T02:38:55.635 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3210000000000001e-06 2020-06-12T02:38:55.635 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.635 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.635 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:55.635 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458488+0000", 2020-06-12T02:38:55.636 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5920000000000003e-06 2020-06-12T02:38:55.636 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.636 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.636 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:55.636 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458496+0000", 2020-06-12T02:38:55.636 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8789999999999999e-06 2020-06-12T02:38:55.637 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.637 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.637 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:55.637 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458499+0000", 2020-06-12T02:38:55.638 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013081152 2020-06-12T02:38:55.638 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.638 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.638 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:55.638 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471580+0000", 2020-06-12T02:38:55.639 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4975999999999994e-05 2020-06-12T02:38:55.639 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.639 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.639 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:55.639 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471665+0000", 2020-06-12T02:38:55.639 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0054254029999999996 2020-06-12T02:38:55.640 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.640 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.640 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:55.640 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.477090+0000", 2020-06-12T02:38:55.640 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2335000000000005e-05 2020-06-12T02:38:55.641 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.641 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.641 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:55.641 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.477163+0000", 2020-06-12T02:38:55.641 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0093967870000000002 2020-06-12T02:38:55.641 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.642 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.642 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:55.642 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486560+0000", 2020-06-12T02:38:55.642 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028102095000000001 2020-06-12T02:38:55.642 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:55.643 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:55.643 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:55.643 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.643 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.643 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13711 3.as0 3:5732055a:::smithi10729407-8:head [append 0~761856] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:38:55.644 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.458525+0000", 2020-06-12T02:38:55.644 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 27.048301447, 2020-06-12T02:38:55.644 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.027773760000000002, 2020-06-12T02:38:55.644 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:55.644 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:55.645 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:55.645 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:55.646 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:55.646 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13711 2020-06-12T02:38:55.646 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.647 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:55.647 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.647 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:55.647 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458525+0000", 2020-06-12T02:38:55.647 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:55.647 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.648 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.648 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:55.648 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458525+0000", 2020-06-12T02:38:55.648 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00089965200000000003 2020-06-12T02:38:55.648 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.649 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.649 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:55.649 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459425+0000", 2020-06-12T02:38:55.649 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4749999999999996e-06 2020-06-12T02:38:55.649 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.649 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.650 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:55.650 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459432+0000", 2020-06-12T02:38:55.650 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3043e-05 2020-06-12T02:38:55.650 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.650 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.651 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:55.651 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459445+0000", 2020-06-12T02:38:55.651 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0739999999999998e-06 2020-06-12T02:38:55.651 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.651 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.651 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:55.652 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459451+0000", 2020-06-12T02:38:55.652 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012238733999999999 2020-06-12T02:38:55.652 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.653 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.653 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:55.653 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471690+0000", 2020-06-12T02:38:55.653 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5640000000000003e-05 2020-06-12T02:38:55.653 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.653 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.654 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:55.654 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471746+0000", 2020-06-12T02:38:55.654 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00067859699999999997 2020-06-12T02:38:55.654 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.654 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.655 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:55.655 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.472424+0000", 2020-06-12T02:38:55.655 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050083230000000003 2020-06-12T02:38:55.655 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.655 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.655 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:55.656 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.477433+0000", 2020-06-12T02:38:55.656 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0087878039999999998 2020-06-12T02:38:55.656 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.656 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.656 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:55.657 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486221+0000", 2020-06-12T02:38:55.657 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8417999999999994e-05 2020-06-12T02:38:55.657 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.657 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.657 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:55.657 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486299+0000", 2020-06-12T02:38:55.658 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.027773760000000002 2020-06-12T02:38:55.658 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:55.658 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:55.658 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:55.658 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.659 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.659 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13712 3.as0 3:5732055a:::smithi10729407-8:head [append 0~712704] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:38:55.659 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.459508+0000", 2020-06-12T02:38:55.659 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 27.047318167, 2020-06-12T02:38:55.659 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.033995818999999997, 2020-06-12T02:38:55.660 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:55.660 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:55.660 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:55.660 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:55.660 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:55.660 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13712 2020-06-12T02:38:55.661 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.661 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:55.661 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.661 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:55.662 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459508+0000", 2020-06-12T02:38:55.662 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:55.662 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.662 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.662 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:55.662 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459508+0000", 2020-06-12T02:38:55.663 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00075436999999999998 2020-06-12T02:38:55.663 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.663 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.663 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:55.663 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460263+0000", 2020-06-12T02:38:55.664 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4370000000000001e-06 2020-06-12T02:38:55.664 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.664 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.664 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:55.664 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460270+0000", 2020-06-12T02:38:55.664 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2500000000000001e-05 2020-06-12T02:38:55.665 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.665 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.665 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:55.665 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460283+0000", 2020-06-12T02:38:55.665 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2580000000000003e-06 2020-06-12T02:38:55.666 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.666 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.666 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:55.666 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460288+0000", 2020-06-12T02:38:55.666 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017177595 2020-06-12T02:38:55.666 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.667 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.667 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:55.667 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.477466+0000", 2020-06-12T02:38:55.667 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00019751600000000001 2020-06-12T02:38:55.667 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.668 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.668 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:55.668 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.477663+0000", 2020-06-12T02:38:55.668 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00088697400000000001 2020-06-12T02:38:55.668 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.668 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.669 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:55.669 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.478550+0000", 2020-06-12T02:38:55.669 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.007578787 2020-06-12T02:38:55.669 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.669 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.670 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:55.670 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486129+0000", 2020-06-12T02:38:55.670 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0072972310000000004 2020-06-12T02:38:55.670 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.670 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.670 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:55.671 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493426+0000", 2020-06-12T02:38:55.671 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8151000000000004e-05 2020-06-12T02:38:55.671 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.671 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.671 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:55.672 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493504+0000", 2020-06-12T02:38:55.672 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.033995818999999997 2020-06-12T02:38:55.672 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:55.673 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:55.673 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:55.673 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.673 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.673 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13713 3.as0 3:5732055a:::smithi10729407-8:head [append 0~491520] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:38:55.674 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.460333+0000", 2020-06-12T02:38:55.674 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 27.046493711, 2020-06-12T02:38:55.674 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.033390145000000003, 2020-06-12T02:38:55.674 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:55.674 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:55.675 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:55.675 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:55.675 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:55.675 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13713 2020-06-12T02:38:55.675 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.675 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:55.676 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.676 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:55.676 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460333+0000", 2020-06-12T02:38:55.676 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:55.676 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.677 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.677 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:55.677 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460333+0000", 2020-06-12T02:38:55.677 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00051168999999999997 2020-06-12T02:38:55.677 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.678 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.678 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:55.678 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460845+0000", 2020-06-12T02:38:55.678 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2930000000000003e-06 2020-06-12T02:38:55.678 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.678 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.679 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:55.679 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460852+0000", 2020-06-12T02:38:55.679 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3054000000000001e-05 2020-06-12T02:38:55.680 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.680 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.680 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:55.680 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460865+0000", 2020-06-12T02:38:55.681 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6130000000000003e-06 2020-06-12T02:38:55.681 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.681 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.681 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:55.681 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460871+0000", 2020-06-12T02:38:55.682 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025753870000000002 2020-06-12T02:38:55.682 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.682 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.682 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:55.682 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486624+0000", 2020-06-12T02:38:55.683 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2739999999999998e-05 2020-06-12T02:38:55.683 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.683 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.683 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:55.683 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486697+0000", 2020-06-12T02:38:55.684 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00063578799999999998 2020-06-12T02:38:55.684 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.684 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.684 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:55.684 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.487333+0000", 2020-06-12T02:38:55.685 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0043011610000000004 2020-06-12T02:38:55.685 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.685 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.685 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:55.685 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.491634+0000", 2020-06-12T02:38:55.686 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0020130949999999999 2020-06-12T02:38:55.686 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.686 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.686 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:55.687 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493647+0000", 2020-06-12T02:38:55.687 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5840999999999994e-05 2020-06-12T02:38:55.687 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.687 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.688 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:55.688 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493723+0000", 2020-06-12T02:38:55.689 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.033390145000000003 2020-06-12T02:38:55.689 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:55.689 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:55.689 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:55.690 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.690 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.690 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13714 3.as0 3:5732055a:::smithi10729407-8:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:38:55.690 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.460921+0000", 2020-06-12T02:38:55.690 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 27.045905188999999, 2020-06-12T02:38:55.691 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.032981457999999998, 2020-06-12T02:38:55.691 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:55.691 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:55.691 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:55.691 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:55.692 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:55.692 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13714 2020-06-12T02:38:55.692 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.692 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:55.692 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.692 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:55.693 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460921+0000", 2020-06-12T02:38:55.693 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:55.693 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.693 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.693 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:55.694 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460921+0000", 2020-06-12T02:38:55.694 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6996999999999995e-05 2020-06-12T02:38:55.694 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.694 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.694 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:55.695 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460998+0000", 2020-06-12T02:38:55.695 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4529999999999999e-06 2020-06-12T02:38:55.695 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.695 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.695 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:55.695 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461005+0000", 2020-06-12T02:38:55.696 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1722000000000001e-05 2020-06-12T02:38:55.696 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.696 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.696 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:55.696 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461017+0000", 2020-06-12T02:38:55.697 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4839999999999998e-06 2020-06-12T02:38:55.697 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.697 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.697 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:55.697 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461021+0000", 2020-06-12T02:38:55.697 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.029157975999999999 2020-06-12T02:38:55.698 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.698 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.698 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:55.698 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.490179+0000", 2020-06-12T02:38:55.698 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010966700000000001 2020-06-12T02:38:55.699 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.699 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.699 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:55.699 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.490289+0000", 2020-06-12T02:38:55.699 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00032026800000000002 2020-06-12T02:38:55.700 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.700 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.700 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:55.700 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.490609+0000", 2020-06-12T02:38:55.700 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001102947 2020-06-12T02:38:55.700 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.701 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.701 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:55.701 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.491712+0000", 2020-06-12T02:38:55.701 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021050169999999998 2020-06-12T02:38:55.701 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.702 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.702 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:55.702 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493817+0000", 2020-06-12T02:38:55.702 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5927000000000006e-05 2020-06-12T02:38:55.702 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.702 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.703 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:55.703 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493903+0000", 2020-06-12T02:38:55.703 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.032981457999999998 2020-06-12T02:38:55.703 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:55.703 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:55.704 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:55.704 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.704 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.704 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13715 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e58)", 2020-06-12T02:38:55.704 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.461062+0000", 2020-06-12T02:38:55.705 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 27.045764549000001, 2020-06-12T02:38:55.705 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.035208338999999998, 2020-06-12T02:38:55.705 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:55.705 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:55.705 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:55.705 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:55.706 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:55.706 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13715 2020-06-12T02:38:55.707 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.707 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:55.707 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.707 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:55.707 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461062+0000", 2020-06-12T02:38:55.707 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:55.708 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.708 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.708 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:55.708 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461062+0000", 2020-06-12T02:38:55.708 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4839999999999998e-05 2020-06-12T02:38:55.709 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.709 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.709 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:55.709 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461107+0000", 2020-06-12T02:38:55.709 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4859999999999997e-06 2020-06-12T02:38:55.709 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.710 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.710 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:55.710 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461112+0000", 2020-06-12T02:38:55.710 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1742999999999999e-05 2020-06-12T02:38:55.710 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.711 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.711 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:55.711 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461124+0000", 2020-06-12T02:38:55.711 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7299999999999996e-06 2020-06-12T02:38:55.711 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.711 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.712 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:55.712 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461129+0000", 2020-06-12T02:38:55.712 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.030397368000000001 2020-06-12T02:38:55.712 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.712 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.713 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:55.713 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.491526+0000", 2020-06-12T02:38:55.713 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010076799999999999 2020-06-12T02:38:55.713 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.713 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.714 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:55.714 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.491627+0000", 2020-06-12T02:38:55.714 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031869849999999998 2020-06-12T02:38:55.714 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.714 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.715 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:55.715 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.494814+0000", 2020-06-12T02:38:55.715 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000108915 2020-06-12T02:38:55.715 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.715 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.715 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:55.716 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.494923+0000", 2020-06-12T02:38:55.718 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001347504 2020-06-12T02:38:55.718 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.718 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.718 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:55.719 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.496270+0000", 2020-06-12T02:38:55.719 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.035208338999999998 2020-06-12T02:38:55.719 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:55.719 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:55.719 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:55.720 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.720 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.720 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:16706 3.as0 3:5cb723d2:::smithi10729407-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e61)", 2020-06-12T02:38:55.720 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:37.289063+0000", 2020-06-12T02:38:55.720 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 18.217763524999999, 2020-06-12T02:38:55.721 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026146086999999998, 2020-06-12T02:38:55.721 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:55.721 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:55.721 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:55.721 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:55.721 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:55.722 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 16706 2020-06-12T02:38:55.722 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.722 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:55.722 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.722 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:55.723 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289063+0000", 2020-06-12T02:38:55.723 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:55.723 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.723 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.723 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:55.724 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289063+0000", 2020-06-12T02:38:55.724 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3776000000000002e-05 2020-06-12T02:38:55.724 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.724 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.724 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:55.724 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289107+0000", 2020-06-12T02:38:55.725 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7039999999999999e-06 2020-06-12T02:38:55.725 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.725 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.725 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:55.725 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289113+0000", 2020-06-12T02:38:55.726 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3259e-05 2020-06-12T02:38:55.726 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.726 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.726 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:55.726 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289126+0000", 2020-06-12T02:38:55.727 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1999999999999996e-06 2020-06-12T02:38:55.727 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.727 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.727 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:55.727 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289130+0000", 2020-06-12T02:38:55.727 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0010912820000000001 2020-06-12T02:38:55.728 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.728 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.728 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:55.728 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.290221+0000", 2020-06-12T02:38:55.729 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3166e-05 2020-06-12T02:38:55.729 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.730 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.730 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:55.730 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.290265+0000", 2020-06-12T02:38:55.730 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015984187 2020-06-12T02:38:55.730 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.731 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.731 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:55.731 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.306249+0000", 2020-06-12T02:38:55.731 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4623000000000006e-05 2020-06-12T02:38:55.731 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.732 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.732 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:55.732 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.306333+0000", 2020-06-12T02:38:55.732 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0068573690000000003 2020-06-12T02:38:55.732 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.732 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.733 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:55.733 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.313191+0000", 2020-06-12T02:38:55.733 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7804000000000002e-05 2020-06-12T02:38:55.733 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.733 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.734 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:55.734 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.313279+0000", 2020-06-12T02:38:55.734 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001930717 2020-06-12T02:38:55.734 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.734 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.735 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:55.735 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.315209+0000", 2020-06-12T02:38:55.735 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026146086999999998 2020-06-12T02:38:55.735 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:55.735 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:55.735 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:55.736 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.736 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.736 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18051 3.as0 3:5cb723d2:::smithi10729407-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:38:55.736 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.329078+0000", 2020-06-12T02:38:55.736 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 14.177748975, 2020-06-12T02:38:55.737 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026465103, 2020-06-12T02:38:55.737 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:55.737 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:55.737 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:55.737 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:55.738 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:55.738 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18051 2020-06-12T02:38:55.738 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.738 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:55.738 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.738 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:55.739 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329078+0000", 2020-06-12T02:38:55.739 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:55.739 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.739 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.739 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:55.740 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329078+0000", 2020-06-12T02:38:55.740 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5392000000000001e-05 2020-06-12T02:38:55.740 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.741 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.741 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:55.741 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329133+0000", 2020-06-12T02:38:55.741 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5329999999999997e-06 2020-06-12T02:38:55.741 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.741 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.742 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:55.742 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329141+0000", 2020-06-12T02:38:55.742 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3459999999999999e-05 2020-06-12T02:38:55.742 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.742 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.743 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:55.743 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329164+0000", 2020-06-12T02:38:55.743 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8170000000000001e-06 2020-06-12T02:38:55.743 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.743 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.743 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:55.744 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329171+0000", 2020-06-12T02:38:55.744 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013005208000000001 2020-06-12T02:38:55.744 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.744 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.744 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:55.745 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.342176+0000", 2020-06-12T02:38:55.745 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3560999999999998e-05 2020-06-12T02:38:55.745 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.745 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.746 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:55.746 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.342230+0000", 2020-06-12T02:38:55.746 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0080818490000000003 2020-06-12T02:38:55.746 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.746 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.747 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:55.747 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.350312+0000", 2020-06-12T02:38:55.747 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0167000000000006e-05 2020-06-12T02:38:55.747 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.747 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.747 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:55.748 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.350402+0000", 2020-06-12T02:38:55.748 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051411160000000003 2020-06-12T02:38:55.748 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.748 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.748 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:55.749 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.355543+0000", 2020-06-12T02:38:55.749 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026465103 2020-06-12T02:38:55.749 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:55.749 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:55.749 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:55.750 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.750 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.750 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18053 3.as0 3:5732055a:::smithi10729407-8:head [append 0~434176] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:38:55.750 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.334425+0000", 2020-06-12T02:38:55.750 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 14.172401408000001, 2020-06-12T02:38:55.750 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.029616703000000001, 2020-06-12T02:38:55.751 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:55.751 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:55.751 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:55.751 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:55.751 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:55.752 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18053 2020-06-12T02:38:55.752 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.752 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:55.752 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.752 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:55.752 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334425+0000", 2020-06-12T02:38:55.753 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:55.753 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.753 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.753 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:55.753 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334425+0000", 2020-06-12T02:38:55.754 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047283900000000002 2020-06-12T02:38:55.754 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.754 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.754 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:55.754 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334898+0000", 2020-06-12T02:38:55.754 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9100000000000004e-06 2020-06-12T02:38:55.755 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.755 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.755 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:55.755 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334903+0000", 2020-06-12T02:38:55.755 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4260000000000003e-06 2020-06-12T02:38:55.756 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.756 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.756 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:55.756 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334911+0000", 2020-06-12T02:38:55.756 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1060000000000002e-06 2020-06-12T02:38:55.756 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.757 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.757 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:55.757 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334916+0000", 2020-06-12T02:38:55.757 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010838316000000001 2020-06-12T02:38:55.757 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.758 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.758 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:55.758 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.345754+0000", 2020-06-12T02:38:55.758 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000103872 2020-06-12T02:38:55.758 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.758 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.759 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:55.759 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.345858+0000", 2020-06-12T02:38:55.759 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0059798739999999996 2020-06-12T02:38:55.759 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.759 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.760 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:55.760 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.351838+0000", 2020-06-12T02:38:55.760 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000104724 2020-06-12T02:38:55.760 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.760 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.760 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:55.761 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.351942+0000", 2020-06-12T02:38:55.761 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000572687 2020-06-12T02:38:55.761 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.761 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.761 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:55.762 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.352515+0000", 2020-06-12T02:38:55.762 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002905641 2020-06-12T02:38:55.762 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.762 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.762 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:55.762 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.355421+0000", 2020-06-12T02:38:55.763 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0085358350000000003 2020-06-12T02:38:55.763 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.763 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.763 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:55.763 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.363956+0000", 2020-06-12T02:38:55.763 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5473e-05 2020-06-12T02:38:55.764 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.764 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.764 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:55.764 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.364042+0000", 2020-06-12T02:38:55.764 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.029616703000000001 2020-06-12T02:38:55.765 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:55.765 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:55.765 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:55.765 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.765 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.765 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18054 3.as0 3:5732055a:::smithi10729407-8:head [append 0~638976] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:38:55.766 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.338171+0000", 2020-06-12T02:38:55.766 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 14.168655905, 2020-06-12T02:38:55.766 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.030067120999999999, 2020-06-12T02:38:55.766 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:55.766 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:55.767 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:55.767 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:55.767 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:55.767 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18054 2020-06-12T02:38:55.767 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.767 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:55.768 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.768 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:55.768 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338171+0000", 2020-06-12T02:38:55.768 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:55.768 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.769 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.769 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:55.770 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338171+0000", 2020-06-12T02:38:55.770 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00068138900000000002 2020-06-12T02:38:55.770 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.770 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.771 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:55.771 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338852+0000", 2020-06-12T02:38:55.771 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.767e-06 2020-06-12T02:38:55.771 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.771 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.772 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:55.772 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338858+0000", 2020-06-12T02:38:55.772 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8999999999999995e-06 2020-06-12T02:38:55.772 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.772 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.773 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:55.773 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338867+0000", 2020-06-12T02:38:55.773 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.262e-06 2020-06-12T02:38:55.773 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.773 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.774 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:55.774 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338871+0000", 2020-06-12T02:38:55.774 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0074839060000000002 2020-06-12T02:38:55.774 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.774 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.774 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:55.775 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346355+0000", 2020-06-12T02:38:55.775 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1435999999999999e-05 2020-06-12T02:38:55.775 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.775 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.775 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:55.776 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346416+0000", 2020-06-12T02:38:55.776 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0092109879999999998 2020-06-12T02:38:55.776 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.776 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.776 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:55.777 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.355627+0000", 2020-06-12T02:38:55.777 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7355999999999998e-05 2020-06-12T02:38:55.777 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.777 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.778 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:55.778 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.355685+0000", 2020-06-12T02:38:55.778 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00074711999999999999 2020-06-12T02:38:55.778 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.779 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.779 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:55.779 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.356432+0000", 2020-06-12T02:38:55.779 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010659572000000001 2020-06-12T02:38:55.779 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.779 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.780 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:55.780 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.367091+0000", 2020-06-12T02:38:55.780 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0010713210000000001 2020-06-12T02:38:55.780 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.780 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.781 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:55.781 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.368163+0000", 2020-06-12T02:38:55.781 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5104000000000005e-05 2020-06-12T02:38:55.781 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.781 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.781 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:55.782 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.368238+0000", 2020-06-12T02:38:55.782 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.030067120999999999 2020-06-12T02:38:55.782 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:55.782 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:55.782 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:55.783 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.783 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.783 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18055 3.as0 3:5732055a:::smithi10729407-8:head [append 0~483328] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:38:55.783 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.340740+0000", 2020-06-12T02:38:55.783 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 14.166087056, 2020-06-12T02:38:55.784 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028669050000000001, 2020-06-12T02:38:55.784 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:55.784 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:55.784 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:55.784 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:55.785 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:55.785 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18055 2020-06-12T02:38:55.785 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.786 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:55.787 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.787 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:55.788 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.340740+0000", 2020-06-12T02:38:55.788 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:55.788 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.788 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.788 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:55.789 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.340740+0000", 2020-06-12T02:38:55.789 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00045126900000000002 2020-06-12T02:38:55.789 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.789 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.789 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:55.790 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341191+0000", 2020-06-12T02:38:55.790 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5630000000000003e-06 2020-06-12T02:38:55.790 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.790 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.790 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:55.791 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341198+0000", 2020-06-12T02:38:55.791 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4467999999999999e-05 2020-06-12T02:38:55.791 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.791 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.791 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:55.791 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341213+0000", 2020-06-12T02:38:55.792 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7540000000000003e-06 2020-06-12T02:38:55.792 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.792 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.792 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:55.792 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341219+0000", 2020-06-12T02:38:55.793 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0052272680000000002 2020-06-12T02:38:55.793 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.793 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.793 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:55.793 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346446+0000", 2020-06-12T02:38:55.793 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4742999999999998e-05 2020-06-12T02:38:55.794 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.794 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.794 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:55.794 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346531+0000", 2020-06-12T02:38:55.794 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013203138999999999 2020-06-12T02:38:55.795 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.795 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.795 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:55.795 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.359734+0000", 2020-06-12T02:38:55.795 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000100277 2020-06-12T02:38:55.795 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.796 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.796 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:55.796 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.359834+0000", 2020-06-12T02:38:55.796 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054998099999999995 2020-06-12T02:38:55.796 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.797 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.797 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:55.797 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.360384+0000", 2020-06-12T02:38:55.797 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0067087179999999998 2020-06-12T02:38:55.797 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.797 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.798 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:55.798 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.367093+0000", 2020-06-12T02:38:55.798 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0022746020000000001 2020-06-12T02:38:55.798 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.798 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.799 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:55.799 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369367+0000", 2020-06-12T02:38:55.799 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1267999999999999e-05 2020-06-12T02:38:55.799 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.799 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.799 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:55.800 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369409+0000", 2020-06-12T02:38:55.800 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028669050000000001 2020-06-12T02:38:55.800 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:55.800 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:55.800 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:55.801 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.801 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.801 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18056 3.as0 3:5732055a:::smithi10729407-8:head [append 0~49152] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:38:55.801 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.341270+0000", 2020-06-12T02:38:55.801 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 14.165556266999999, 2020-06-12T02:38:55.801 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028350462999999999, 2020-06-12T02:38:55.802 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:55.802 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:55.802 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:55.802 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:55.802 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:55.803 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18056 2020-06-12T02:38:55.803 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.803 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:55.803 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.803 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:55.803 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341270+0000", 2020-06-12T02:38:55.804 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:55.804 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.804 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.804 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:55.804 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341270+0000", 2020-06-12T02:38:55.805 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000114082 2020-06-12T02:38:55.805 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.805 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.805 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:55.805 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341384+0000", 2020-06-12T02:38:55.806 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3289999999999999e-06 2020-06-12T02:38:55.806 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.806 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.806 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:55.806 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341391+0000", 2020-06-12T02:38:55.806 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3339999999999999e-05 2020-06-12T02:38:55.807 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.807 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.807 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:55.807 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341404+0000", 2020-06-12T02:38:55.807 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4e-06 2020-06-12T02:38:55.808 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.808 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.808 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:55.808 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341410+0000", 2020-06-12T02:38:55.808 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051518600000000003 2020-06-12T02:38:55.808 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.809 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.809 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:55.809 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346561+0000", 2020-06-12T02:38:55.809 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1662999999999994e-05 2020-06-12T02:38:55.809 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.810 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.810 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:55.810 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346643+0000", 2020-06-12T02:38:55.810 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017471605000000001 2020-06-12T02:38:55.810 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.810 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.811 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:55.811 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.364115+0000", 2020-06-12T02:38:55.811 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011059400000000001 2020-06-12T02:38:55.811 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.811 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.812 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:55.812 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.364225+0000", 2020-06-12T02:38:55.812 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000611262 2020-06-12T02:38:55.812 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.812 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.812 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:55.813 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.364837+0000", 2020-06-12T02:38:55.813 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023767710000000002 2020-06-12T02:38:55.813 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.813 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.813 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:55.814 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.367213+0000", 2020-06-12T02:38:55.814 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023325479999999998 2020-06-12T02:38:55.814 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.814 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.814 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:55.814 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369546+0000", 2020-06-12T02:38:55.815 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5009e-05 2020-06-12T02:38:55.815 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.815 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.815 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:55.815 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369621+0000", 2020-06-12T02:38:55.816 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028350462999999999 2020-06-12T02:38:55.816 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:55.816 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:55.816 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:55.816 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.816 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.817 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18057 3.as0 3:5732055a:::smithi10729407-8:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:38:55.817 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.341456+0000", 2020-06-12T02:38:55.817 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 14.165371107, 2020-06-12T02:38:55.817 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028443018, 2020-06-12T02:38:55.817 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:55.818 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:55.818 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:55.818 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:55.818 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:55.818 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18057 2020-06-12T02:38:55.819 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.819 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:55.819 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.819 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:55.819 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341456+0000", 2020-06-12T02:38:55.819 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:55.820 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.820 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.820 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:55.820 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341456+0000", 2020-06-12T02:38:55.820 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6036000000000003e-05 2020-06-12T02:38:55.820 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.821 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.821 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:55.821 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341522+0000", 2020-06-12T02:38:55.821 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2210000000000002e-06 2020-06-12T02:38:55.821 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.822 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.822 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:55.822 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341528+0000", 2020-06-12T02:38:55.822 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0338e-05 2020-06-12T02:38:55.822 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.822 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.823 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:55.823 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341538+0000", 2020-06-12T02:38:55.823 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8639999999999998e-06 2020-06-12T02:38:55.823 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.823 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.824 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:55.824 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341542+0000", 2020-06-12T02:38:55.824 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051405009999999996 2020-06-12T02:38:55.824 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.824 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.824 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:55.825 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346683+0000", 2020-06-12T02:38:55.825 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9561999999999995e-05 2020-06-12T02:38:55.825 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.825 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.825 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:55.826 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346762+0000", 2020-06-12T02:38:55.826 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019267197999999999 2020-06-12T02:38:55.826 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.826 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.826 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:55.826 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.366029+0000", 2020-06-12T02:38:55.827 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2634999999999997e-05 2020-06-12T02:38:55.827 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.827 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.827 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:55.828 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.366122+0000", 2020-06-12T02:38:55.828 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00026907500000000002 2020-06-12T02:38:55.829 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.829 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.829 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:55.829 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.366391+0000", 2020-06-12T02:38:55.829 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0018939409999999999 2020-06-12T02:38:55.830 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.830 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.830 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:55.830 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.368285+0000", 2020-06-12T02:38:55.830 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001549405 2020-06-12T02:38:55.831 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.831 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.831 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:55.831 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369834+0000", 2020-06-12T02:38:55.831 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4241999999999999e-05 2020-06-12T02:38:55.832 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.832 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.832 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:55.832 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369899+0000", 2020-06-12T02:38:55.832 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028443018 2020-06-12T02:38:55.832 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:55.833 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:55.833 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:55.833 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.833 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.833 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18058 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:38:55.834 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.341580+0000", 2020-06-12T02:38:55.834 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 14.1652462, 2020-06-12T02:38:55.834 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.030449559000000001, 2020-06-12T02:38:55.834 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:55.834 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:55.835 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:55.835 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:55.835 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:55.835 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18058 2020-06-12T02:38:55.835 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.835 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:55.836 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.836 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:55.836 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341580+0000", 2020-06-12T02:38:55.836 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:55.836 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.837 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.837 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:55.837 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341580+0000", 2020-06-12T02:38:55.837 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1185e-05 2020-06-12T02:38:55.837 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.837 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.838 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:55.838 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341622+0000", 2020-06-12T02:38:55.838 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1599999999999997e-06 2020-06-12T02:38:55.838 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.838 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.839 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:55.839 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341627+0000", 2020-06-12T02:38:55.839 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3546e-05 2020-06-12T02:38:55.840 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.840 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.840 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:55.840 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341640+0000", 2020-06-12T02:38:55.840 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9779999999999999e-06 2020-06-12T02:38:55.841 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.841 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.841 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:55.841 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341644+0000", 2020-06-12T02:38:55.841 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051433700000000004 2020-06-12T02:38:55.841 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.842 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.842 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:55.842 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346788+0000", 2020-06-12T02:38:55.842 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1622000000000001e-05 2020-06-12T02:38:55.842 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.843 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.843 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:55.843 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346839+0000", 2020-06-12T02:38:55.843 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020283958000000001 2020-06-12T02:38:55.843 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.843 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.844 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:55.844 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.367123+0000", 2020-06-12T02:38:55.844 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1997999999999997e-05 2020-06-12T02:38:55.844 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.844 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.845 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:55.845 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.367205+0000", 2020-06-12T02:38:55.845 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036875440000000001 2020-06-12T02:38:55.845 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.845 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.846 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:55.846 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.370893+0000", 2020-06-12T02:38:55.846 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000100604 2020-06-12T02:38:55.846 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.846 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.846 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:55.847 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.370993+0000", 2020-06-12T02:38:55.847 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0010365940000000001 2020-06-12T02:38:55.847 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:55.847 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:55.847 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:55.848 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.372030+0000", 2020-06-12T02:38:55.848 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.030449559000000001 2020-06-12T02:38:55.848 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:55.848 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:55.848 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:55.848 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:55.849 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:55.849 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:55.851 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.7 to 1 (10000) 2020-06-12T02:38:55.851 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2020-06-12T02:38:55.851 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json 2020-06-12T02:38:55.852 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-06-12T02:38:55.853 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:55.848+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:55.962 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:55.957+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:56.010 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:56.011 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:38:56.011 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.011 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:19948 3.15s0 3:a9510bda:::smithi10729407-14:head [stat,delete] snapc 0=[] ondisk+write+known_if_redirected e71)", 2020-06-12T02:38:56.011 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:51.564097+0000", 2020-06-12T02:38:56.012 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 4.4447221619999997, 2020-06-12T02:38:56.012 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4447449780000001, 2020-06-12T02:38:56.012 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:56.012 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "delayed", 2020-06-12T02:38:56.012 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:56.012 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:56.013 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:56.013 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 19948 2020-06-12T02:38:56.013 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.013 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:56.013 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.014 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:56.014 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:51.564097+0000", 2020-06-12T02:38:56.014 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:56.014 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.014 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.015 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:56.015 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:51.564097+0000", 2020-06-12T02:38:56.015 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3070000000000001e-05 2020-06-12T02:38:56.015 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.015 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.015 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:56.016 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:51.564120+0000", 2020-06-12T02:38:56.016 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.656e-06 2020-06-12T02:38:56.016 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.016 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.016 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:56.017 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:51.564125+0000", 2020-06-12T02:38:56.017 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.14e-06 2020-06-12T02:38:56.017 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.017 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.017 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:56.018 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:51.564133+0000", 2020-06-12T02:38:56.018 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5240000000000002e-06 2020-06-12T02:38:56.018 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.018 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.018 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:56.018 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:51.564137+0000", 2020-06-12T02:38:56.019 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0881000000000003e-05 2020-06-12T02:38:56.019 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.019 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.019 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:56.019 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:51.564188+0000", 2020-06-12T02:38:56.020 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5320000000000001e-05 2020-06-12T02:38:56.020 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.020 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.020 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for readable", 2020-06-12T02:38:56.020 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:51.564223+0000", 2020-06-12T02:38:56.021 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012659100000000001 2020-06-12T02:38:56.021 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.021 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:56.021 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.021 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.021 INFO:teuthology.orchestra.run.smithi188.stdout: ], 2020-06-12T02:38:56.022 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 1 2020-06-12T02:38:56.022 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:56.023 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-06-12T02:38:56.060 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:56.058+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:56.111 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:38:56.112 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2020-06-12T02:38:56.122 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":128,"stamp":"2020-06-12T02:38:55.611659+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6864896,"num_objects":9,"num_object_clones":0,"num_object_copies":25,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":5274,"num_read_kb":2866,"num_write":11792,"num_write_kb":3359806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":13549568,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25562,"ondisk_log_size":25562,"up":96,"acting":96,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":102,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8401940,"kb_used_data":13268,"kb_used_omap":25,"kb_used_meta":8388582,"kb_avail":746572780,"statfs":{"total":773094113280,"available":764490526720,"internally_reserved":8589934592,"allocated":13586432,"data_stored":10161222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26157,"internal_metadata":8589908435},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-11890688,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1337,"num_read_kb":728,"num_write":2985,"num_write_kb":847060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":2650112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000822"},"pg_stats":[{"pgid":"3.19","version":"63'1666","reported_seq":"17","reported_epoch":"71","state":"active+undersized+remapped","last_fresh":"2020-06-12T02:38:51.398957+0000","last_change":"2020-06-12T02:38:51.398745+0000","last_active":"2020-06-12T02:38:51.398957+0000","last_peered":"2020-06-12T02:38:51.398957+0000","last_clean":"2020-06-12T02:38:44.390059+0000","last_became_active":"2020-06-12T02:38:51.398745+0000","last_became_peered":"2020-06-12T02:38:51.398745+0000","last_unstale":"2020-06-12T02:38:51.398957+0000","last_undegraded":"2020-06-12T02:38:51.398957+0000","last_fullsized":"2020-06-12T02:38:51.390307+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":1666,"ondisk_log_size":1666,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":282,"num_read_kb":155,"num_write":620,"num_write_kb":172356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[2147483647,3,4],"avail_no_missing":["3(1)","1(0)","2(1)","4(0)","4(2)","6(0)","7(2)"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"63'843","reported_seq":"1116","reported_epoch":"71","state":"active+undersized+remapped","last_fresh":"2020-06-12T02:38:51.419916+0000","last_change":"2020-06-12T02:38:51.419752+0000","last_active":"2020-06-12T02:38:51.419916+0000","last_peered":"2020-06-12T02:38:51.419916+0000","last_clean":"2020-06-12T02:38:45.505919+0000","last_became_active":"2020-06-12T02:38:51.419752+0000","last_became_peered":"2020-06-12T02:38:51.419752+0000","last_unstale":"2020-06-12T02:38:51.419916+0000","last_undegraded":"2020-06-12T02:38:51.419916+0000","last_fullsized":"2020-06-12T02:38:51.394729+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":843,"ondisk_log_size":843,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":137,"num_read_kb":75,"num_write":323,"num_write_kb":94224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1396736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,2147483647],"avail_no_missing":["3(0)","0(1)","1(2)","4(1)","7(2)"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"71'1724","reported_seq":"52","reported_epoch":"71","state":"active+clean","last_fresh":"2020-06-12T02:38:51.503665+0000","last_change":"2020-06-12T02:38:50.381078+0000","last_active":"2020-06-12T02:38:51.503665+0000","last_peered":"2020-06-12T02:38:51.503665+0000","last_clean":"2020-06-12T02:38:51.503665+0000","last_became_active":"2020-06-12T02:38:50.380314+0000","last_became_peered":"2020-06-12T02:38:50.380314+0000","last_unstale":"2020-06-12T02:38:51.503665+0000","last_undegraded":"2020-06-12T02:38:51.503665+0000","last_fullsized":"2020-06-12T02:38:51.503665+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":1724,"ondisk_log_size":1724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":946176,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":298,"num_read_kb":166,"num_write":679,"num_write_kb":196988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1396736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"25","reported_epoch":"72","state":"active+clean","last_fresh":"2020-06-12T02:38:52.393478+0000","last_change":"2020-06-12T02:38:50.387232+0000","last_active":"2020-06-12T02:38:52.393478+0000","last_peered":"2020-06-12T02:38:52.393478+0000","last_clean":"2020-06-12T02:38:52.393478+0000","last_became_active":"2020-06-12T02:38:50.386718+0000","last_became_peered":"2020-06-12T02:38:50.386718+0000","last_unstale":"2020-06-12T02:38:52.393478+0000","last_undegraded":"2020-06-12T02:38:52.393478+0000","last_fullsized":"2020-06-12T02:38:52.393478+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"68'1595","reported_seq":"1970","reported_epoch":"72","state":"active+clean+wait","last_fresh":"2020-06-12T02:38:52.396181+0000","last_change":"2020-06-12T02:38:50.388146+0000","last_active":"2020-06-12T02:38:52.396181+0000","last_peered":"2020-06-12T02:38:52.396181+0000","last_clean":"2020-06-12T02:38:52.396181+0000","last_became_active":"2020-06-12T02:38:50.385649+0000","last_became_peered":"2020-06-12T02:38:50.385649+0000","last_unstale":"2020-06-12T02:38:52.396181+0000","last_undegraded":"2020-06-12T02:38:52.396181+0000","last_fullsized":"2020-06-12T02:38:52.396181+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":1595,"ondisk_log_size":1595,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":483328,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":263,"num_read_kb":143,"num_write":592,"num_write_kb":171168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"14","reported_epoch":"70","state":"active+clean","last_fresh":"2020-06-12T02:38:50.373574+0000","last_change":"2020-06-12T02:38:47.508860+0000","last_active":"2020-06-12T02:38:50.373574+0000","last_peered":"2020-06-12T02:38:50.373574+0000","last_clean":"2020-06-12T02:38:50.373574+0000","last_became_active":"2020-06-12T02:38:47.508336+0000","last_became_peered":"2020-06-12T02:38:47.508336+0000","last_unstale":"2020-06-12T02:38:50.373574+0000","last_undegraded":"2020-06-12T02:38:50.373574+0000","last_fullsized":"2020-06-12T02:38:50.373574+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"68'1640","reported_seq":"39","reported_epoch":"72","state":"active+clean+wait","last_fresh":"2020-06-12T02:38:52.393265+0000","last_change":"2020-06-12T02:38:50.386250+0000","last_active":"2020-06-12T02:38:52.393265+0000","last_peered":"2020-06-12T02:38:52.393265+0000","last_clean":"2020-06-12T02:38:52.393265+0000","last_became_active":"2020-06-12T02:38:50.385130+0000","last_became_peered":"2020-06-12T02:38:50.385130+0000","last_unstale":"2020-06-12T02:38:52.393265+0000","last_undegraded":"2020-06-12T02:38:52.393265+0000","last_fullsized":"2020-06-12T02:38:52.393265+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":1640,"ondisk_log_size":1640,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":262144,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":276,"num_read_kb":147,"num_write":614,"num_write_kb":173540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"71'697","reported_seq":"48","reported_epoch":"71","state":"active+clean","last_fresh":"2020-06-12T02:38:51.490105+0000","last_change":"2020-06-12T02:38:50.384388+0000","last_active":"2020-06-12T02:38:51.490105+0000","last_peered":"2020-06-12T02:38:51.490105+0000","last_clean":"2020-06-12T02:38:51.490105+0000","last_became_active":"2020-06-12T02:38:50.383580+0000","last_became_peered":"2020-06-12T02:38:50.383580+0000","last_unstale":"2020-06-12T02:38:51.490105+0000","last_undegraded":"2020-06-12T02:38:51.490105+0000","last_fullsized":"2020-06-12T02:38:51.490105+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'650","last_scrub_stamp":"2020-06-12T02:38:43.307170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:38:43.307170+0000","log_size":697,"ondisk_log_size":697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":118,"num_read_kb":65,"num_write":260,"num_write_kb":71960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"16","reported_epoch":"72","state":"active+clean","last_fresh":"2020-06-12T02:38:52.393043+0000","last_change":"2020-06-12T02:38:47.506911+0000","last_active":"2020-06-12T02:38:52.393043+0000","last_peered":"2020-06-12T02:38:52.393043+0000","last_clean":"2020-06-12T02:38:52.393043+0000","last_became_active":"2020-06-12T02:38:47.506358+0000","last_became_peered":"2020-06-12T02:38:47.506358+0000","last_unstale":"2020-06-12T02:38:52.393043+0000","last_undegraded":"2020-06-12T02:38:52.393043+0000","last_fullsized":"2020-06-12T02:38:52.393043+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"68'667","reported_seq":"46","reported_epoch":"70","state":"active+clean+wait","last_fresh":"2020-06-12T02:38:50.388200+0000","last_change":"2020-06-12T02:38:50.383035+0000","last_active":"2020-06-12T02:38:50.388200+0000","last_peered":"2020-06-12T02:38:50.388200+0000","last_clean":"2020-06-12T02:38:50.388200+0000","last_became_active":"2020-06-12T02:38:50.382197+0000","last_became_peered":"2020-06-12T02:38:50.382197+0000","last_unstale":"2020-06-12T02:38:50.388200+0000","last_undegraded":"2020-06-12T02:38:50.388200+0000","last_fullsized":"2020-06-12T02:38:50.388200+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":667,"ondisk_log_size":667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":368640,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":119,"num_read_kb":66,"num_write":270,"num_write_kb":77464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"71'1767","reported_seq":"2160","reported_epoch":"72","state":"active+clean","last_fresh":"2020-06-12T02:38:52.393887+0000","last_change":"2020-06-12T02:38:46.393841+0000","last_active":"2020-06-12T02:38:52.393887+0000","last_peered":"2020-06-12T02:38:52.393887+0000","last_clean":"2020-06-12T02:38:52.393887+0000","last_became_active":"2020-06-12T02:37:57.566633+0000","last_became_peered":"2020-06-12T02:37:57.566633+0000","last_unstale":"2020-06-12T02:38:52.393887+0000","last_undegraded":"2020-06-12T02:38:52.393887+0000","last_fullsized":"2020-06-12T02:38:52.393887+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1729","last_scrub_stamp":"2020-06-12T02:38:46.393774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:38:46.393774+0000","log_size":1767,"ondisk_log_size":1767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":434176,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":598,"num_read_kb":316,"num_write":1304,"num_write_kb":368976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1531904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"71'912","reported_seq":"1150","reported_epoch":"72","state":"active+clean","last_fresh":"2020-06-12T02:38:52.393598+0000","last_change":"2020-06-12T02:38:11.572385+0000","last_active":"2020-06-12T02:38:52.393598+0000","last_peered":"2020-06-12T02:38:52.393598+0000","last_clean":"2020-06-12T02:38:52.393598+0000","last_became_active":"2020-06-12T02:37:57.563869+0000","last_became_peered":"2020-06-12T02:37:57.563869+0000","last_unstale":"2020-06-12T02:38:52.393598+0000","last_undegraded":"2020-06-12T02:38:52.393598+0000","last_fullsized":"2020-06-12T02:38:52.393598+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":912,"ondisk_log_size":912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":300,"num_read_kb":162,"num_write":688,"num_write_kb":195192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1466368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"71'1699","reported_seq":"2096","reported_epoch":"72","state":"active+clean","last_fresh":"2020-06-12T02:38:52.396596+0000","last_change":"2020-06-12T02:38:11.587290+0000","last_active":"2020-06-12T02:38:52.396596+0000","last_peered":"2020-06-12T02:38:52.396596+0000","last_clean":"2020-06-12T02:38:52.396596+0000","last_became_active":"2020-06-12T02:37:57.564464+0000","last_became_peered":"2020-06-12T02:37:57.564464+0000","last_unstale":"2020-06-12T02:38:52.396596+0000","last_undegraded":"2020-06-12T02:38:52.396596+0000","last_fullsized":"2020-06-12T02:38:52.396596+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":1699,"ondisk_log_size":1699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":901120,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":575,"num_read_kb":311,"num_write":1298,"num_write_kb":369408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3710976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"55","reported_epoch":"70","state":"active+clean","last_fresh":"2020-06-12T02:38:50.372263+0000","last_change":"2020-06-12T02:38:46.288343+0000","last_active":"2020-06-12T02:38:50.372263+0000","last_peered":"2020-06-12T02:38:50.372263+0000","last_clean":"2020-06-12T02:38:50.372263+0000","last_became_active":"2020-06-12T02:37:42.384363+0000","last_became_peered":"2020-06-12T02:37:42.384363+0000","last_unstale":"2020-06-12T02:38:50.372263+0000","last_undegraded":"2020-06-12T02:38:50.372263+0000","last_fullsized":"2020-06-12T02:38:50.372263+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:38:46.288277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:38:46.288277+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"71'1661","reported_seq":"2021","reported_epoch":"71","state":"active+clean","last_fresh":"2020-06-12T02:38:51.521402+0000","last_change":"2020-06-12T02:37:27.902643+0000","last_active":"2020-06-12T02:38:51.521402+0000","last_peered":"2020-06-12T02:38:51.521402+0000","last_clean":"2020-06-12T02:38:51.521402+0000","last_became_active":"2020-06-12T02:37:27.902204+0000","last_became_peered":"2020-06-12T02:37:27.902204+0000","last_unstale":"2020-06-12T02:38:51.521402+0000","last_undegraded":"2020-06-12T02:38:51.521402+0000","last_fullsized":"2020-06-12T02:38:51.521402+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":1661,"ondisk_log_size":1661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":562,"num_read_kb":303,"num_write":1229,"num_write_kb":343448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"71'1746","reported_seq":"2170","reported_epoch":"71","state":"active+undersized+remapped","last_fresh":"2020-06-12T02:38:51.529088+0000","last_change":"2020-06-12T02:38:51.438922+0000","last_active":"2020-06-12T02:38:51.529088+0000","last_peered":"2020-06-12T02:38:51.529088+0000","last_clean":"2020-06-12T02:38:49.502897+0000","last_became_active":"2020-06-12T02:38:51.431610+0000","last_became_peered":"2020-06-12T02:38:51.431610+0000","last_unstale":"2020-06-12T02:38:51.529088+0000","last_undegraded":"2020-06-12T02:38:51.529088+0000","last_fullsized":"2020-06-12T02:38:51.389731+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":1746,"ondisk_log_size":1746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1802240,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":611,"num_read_kb":331,"num_write":1333,"num_write_kb":383192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,2147483647],"avail_no_missing":["6(0)","1(2)","4(1)","5(2)"],"object_location_counts":[{"shards":"4(1),6(0)","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"71'1730","reported_seq":"2139","reported_epoch":"71","state":"active+clean","last_fresh":"2020-06-12T02:38:51.457789+0000","last_change":"2020-06-12T02:38:45.498622+0000","last_active":"2020-06-12T02:38:51.457789+0000","last_peered":"2020-06-12T02:38:51.457789+0000","last_clean":"2020-06-12T02:38:51.457789+0000","last_became_active":"2020-06-12T02:38:45.497443+0000","last_became_peered":"2020-06-12T02:38:45.497443+0000","last_unstale":"2020-06-12T02:38:51.457789+0000","last_undegraded":"2020-06-12T02:38:51.457789+0000","last_fullsized":"2020-06-12T02:38:51.457789+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":1730,"ondisk_log_size":1730,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":301,"num_read_kb":166,"num_write":666,"num_write_kb":185356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"71'865","reported_seq":"1139","reported_epoch":"71","state":"active+undersized+remapped","last_fresh":"2020-06-12T02:38:51.432132+0000","last_change":"2020-06-12T02:38:51.432132+0000","last_active":"2020-06-12T02:38:51.432132+0000","last_peered":"2020-06-12T02:38:51.432132+0000","last_clean":"2020-06-12T02:38:49.480669+0000","last_became_active":"2020-06-12T02:38:51.420632+0000","last_became_peered":"2020-06-12T02:38:51.420632+0000","last_unstale":"2020-06-12T02:38:51.432132+0000","last_undegraded":"2020-06-12T02:38:51.432132+0000","last_fullsized":"2020-06-12T02:38:51.390271+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":865,"ondisk_log_size":865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":143,"num_read_kb":77,"num_write":335,"num_write_kb":97568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1863680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,2147483647,7],"avail_no_missing":["3(0)","1(1)","4(1)","7(2)"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"10","reported_epoch":"70","state":"active+clean","last_fresh":"2020-06-12T02:38:50.380478+0000","last_change":"2020-06-12T02:38:50.377172+0000","last_active":"2020-06-12T02:38:50.380478+0000","last_peered":"2020-06-12T02:38:50.380478+0000","last_clean":"2020-06-12T02:38:50.380478+0000","last_became_active":"2020-06-12T02:38:50.376548+0000","last_became_peered":"2020-06-12T02:38:50.376548+0000","last_unstale":"2020-06-12T02:38:50.380478+0000","last_undegraded":"2020-06-12T02:38:50.380478+0000","last_fullsized":"2020-06-12T02:38:50.380478+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:38:34.510641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:38:34.510641+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"71'1617","reported_seq":"1977","reported_epoch":"71","state":"active+clean","last_fresh":"2020-06-12T02:38:51.452943+0000","last_change":"2020-06-12T02:38:45.496321+0000","last_active":"2020-06-12T02:38:51.452943+0000","last_peered":"2020-06-12T02:38:51.452943+0000","last_clean":"2020-06-12T02:38:51.452943+0000","last_became_active":"2020-06-12T02:38:45.495538+0000","last_became_peered":"2020-06-12T02:38:45.495538+0000","last_unstale":"2020-06-12T02:38:51.452943+0000","last_undegraded":"2020-06-12T02:38:51.452943+0000","last_fullsized":"2020-06-12T02:38:51.452943+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":1617,"ondisk_log_size":1617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":737280,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":273,"num_read_kb":151,"num_write":620,"num_write_kb":181688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"10","reported_epoch":"70","state":"active+clean","last_fresh":"2020-06-12T02:38:50.385378+0000","last_change":"2020-06-12T02:38:50.381973+0000","last_active":"2020-06-12T02:38:50.385378+0000","last_peered":"2020-06-12T02:38:50.385378+0000","last_clean":"2020-06-12T02:38:50.385378+0000","last_became_active":"2020-06-12T02:38:50.381564+0000","last_became_peered":"2020-06-12T02:38:50.381564+0000","last_unstale":"2020-06-12T02:38:50.385378+0000","last_undegraded":"2020-06-12T02:38:50.385378+0000","last_fullsized":"2020-06-12T02:38:50.385378+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:38:44.131122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:38:44.131122+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"71'1665","reported_seq":"2068","reported_epoch":"71","state":"active+undersized+remapped","last_fresh":"2020-06-12T02:38:51.437336+0000","last_change":"2020-06-12T02:38:51.423127+0000","last_active":"2020-06-12T02:38:51.437336+0000","last_peered":"2020-06-12T02:38:51.437336+0000","last_clean":"2020-06-12T02:38:49.494980+0000","last_became_active":"2020-06-12T02:38:51.417442+0000","last_became_peered":"2020-06-12T02:38:51.417442+0000","last_unstale":"2020-06-12T02:38:51.437336+0000","last_undegraded":"2020-06-12T02:38:51.437336+0000","last_fullsized":"2020-06-12T02:38:51.389707+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1653","last_scrub_stamp":"2020-06-12T02:38:46.542505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:38:46.542505+0000","log_size":1665,"ondisk_log_size":1665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,2147483647,0],"avail_no_missing":["4(0)","0(2)","1(1)","5(1)"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"64","reported_epoch":"70","state":"active+clean","last_fresh":"2020-06-12T02:38:50.374517+0000","last_change":"2020-06-12T02:38:21.039695+0000","last_active":"2020-06-12T02:38:50.374517+0000","last_peered":"2020-06-12T02:38:50.374517+0000","last_clean":"2020-06-12T02:38:50.374517+0000","last_became_active":"2020-06-12T02:37:16.764958+0000","last_became_peered":"2020-06-12T02:37:16.764958+0000","last_unstale":"2020-06-12T02:38:50.374517+0000","last_undegraded":"2020-06-12T02:38:50.374517+0000","last_fullsized":"2020-06-12T02:38:50.374517+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:38:21.039625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:38:21.039625+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"63'651","reported_seq":"856","reported_epoch":"71","state":"active+undersized+remapped","last_fresh":"2020-06-12T02:38:51.396177+0000","last_change":"2020-06-12T02:38:51.395897+0000","last_active":"2020-06-12T02:38:51.396177+0000","last_peered":"2020-06-12T02:38:51.396177+0000","last_clean":"2020-06-12T02:38:44.451906+0000","last_became_active":"2020-06-12T02:38:51.395897+0000","last_became_peered":"2020-06-12T02:38:51.395897+0000","last_unstale":"2020-06-12T02:38:51.396177+0000","last_undegraded":"2020-06-12T02:38:51.396177+0000","last_fullsized":"2020-06-12T02:38:51.389031+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":651,"ondisk_log_size":651,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":929792,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":112,"num_read_kb":63,"num_write":259,"num_write_kb":75432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[2147483647,2,6],"avail_no_missing":["2(1)","0(0)","1(0)","6(2)"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"54","reported_epoch":"70","state":"active+clean","last_fresh":"2020-06-12T02:38:50.376233+0000","last_change":"2020-06-12T02:38:22.079524+0000","last_active":"2020-06-12T02:38:50.376233+0000","last_peered":"2020-06-12T02:38:50.376233+0000","last_clean":"2020-06-12T02:38:50.376233+0000","last_became_active":"2020-06-12T02:37:42.375902+0000","last_became_peered":"2020-06-12T02:37:42.375902+0000","last_unstale":"2020-06-12T02:38:50.376233+0000","last_undegraded":"2020-06-12T02:38:50.376233+0000","last_fullsized":"2020-06-12T02:38:50.376233+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:38:22.079458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:38:22.079458+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"15'2","reported_seq":"64","reported_epoch":"72","state":"active+clean","last_fresh":"2020-06-12T02:38:52.394021+0000","last_change":"2020-06-12T02:38:24.369083+0000","last_active":"2020-06-12T02:38:52.394021+0000","last_peered":"2020-06-12T02:38:52.394021+0000","last_clean":"2020-06-12T02:38:52.394021+0000","last_became_active":"2020-06-12T02:37:54.558125+0000","last_became_peered":"2020-06-12T02:37:54.558125+0000","last_unstale":"2020-06-12T02:38:52.394021+0000","last_undegraded":"2020-06-12T02:38:52.394021+0000","last_fullsized":"2020-06-12T02:38:52.394021+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":3,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'2","last_scrub_stamp":"2020-06-12T02:38:24.368994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:04.419204+0000","last_clean_scrub_stamp":"2020-06-12T02:38:24.368994+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"63'673","reported_seq":"870","reported_epoch":"70","state":"active+clean","last_fresh":"2020-06-12T02:38:50.376701+0000","last_change":"2020-06-12T02:38:48.229858+0000","last_active":"2020-06-12T02:38:50.376701+0000","last_peered":"2020-06-12T02:38:50.376701+0000","last_clean":"2020-06-12T02:38:50.376701+0000","last_became_active":"2020-06-12T02:38:45.495198+0000","last_became_peered":"2020-06-12T02:38:45.495198+0000","last_unstale":"2020-06-12T02:38:50.376701+0000","last_undegraded":"2020-06-12T02:38:50.376701+0000","last_fullsized":"2020-06-12T02:38:50.376701+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'673","last_scrub_stamp":"2020-06-12T02:38:48.229769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:38:48.229769+0000","log_size":673,"ondisk_log_size":673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"77","reported_epoch":"72","state":"active+clean","last_fresh":"2020-06-12T02:38:52.393965+0000","last_change":"2020-06-12T02:38:50.386452+0000","last_active":"2020-06-12T02:38:52.393965+0000","last_peered":"2020-06-12T02:38:52.393965+0000","last_clean":"2020-06-12T02:38:52.393965+0000","last_became_active":"2020-06-12T02:38:50.386054+0000","last_became_peered":"2020-06-12T02:38:50.386054+0000","last_unstale":"2020-06-12T02:38:52.393965+0000","last_undegraded":"2020-06-12T02:38:52.393965+0000","last_fullsized":"2020-06-12T02:38:52.393965+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:38:34.394578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:38:34.394578+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"66","reported_epoch":"70","state":"active+clean","last_fresh":"2020-06-12T02:38:50.371190+0000","last_change":"2020-06-12T02:38:45.505097+0000","last_active":"2020-06-12T02:38:50.371190+0000","last_peered":"2020-06-12T02:38:50.371190+0000","last_clean":"2020-06-12T02:38:50.371190+0000","last_became_active":"2020-06-12T02:38:45.499654+0000","last_became_peered":"2020-06-12T02:38:45.499654+0000","last_unstale":"2020-06-12T02:38:50.371190+0000","last_undegraded":"2020-06-12T02:38:50.371190+0000","last_fullsized":"2020-06-12T02:38:50.371190+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"58","reported_epoch":"72","state":"active+clean","last_fresh":"2020-06-12T02:38:52.396482+0000","last_change":"2020-06-12T02:38:05.892563+0000","last_active":"2020-06-12T02:38:52.396482+0000","last_peered":"2020-06-12T02:38:52.396482+0000","last_clean":"2020-06-12T02:38:52.396482+0000","last_became_active":"2020-06-12T02:37:54.557420+0000","last_became_peered":"2020-06-12T02:37:54.557420+0000","last_unstale":"2020-06-12T02:38:52.396482+0000","last_undegraded":"2020-06-12T02:38:52.396482+0000","last_fullsized":"2020-06-12T02:38:52.396482+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"11","reported_epoch":"70","state":"active+clean","last_fresh":"2020-06-12T02:38:50.385790+0000","last_change":"2020-06-12T02:38:50.380210+0000","last_active":"2020-06-12T02:38:50.385790+0000","last_peered":"2020-06-12T02:38:50.385790+0000","last_clean":"2020-06-12T02:38:50.385790+0000","last_became_active":"2020-06-12T02:38:50.379633+0000","last_became_peered":"2020-06-12T02:38:50.379633+0000","last_unstale":"2020-06-12T02:38:50.385790+0000","last_undegraded":"2020-06-12T02:38:50.385790+0000","last_fullsized":"2020-06-12T02:38:50.385790+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:38:46.140457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:38:46.140457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"71'1742","reported_seq":"2157","reported_epoch":"71","state":"active+clean","last_fresh":"2020-06-12T02:38:51.542920+0000","last_change":"2020-06-12T02:38:45.497458+0000","last_active":"2020-06-12T02:38:51.542920+0000","last_peered":"2020-06-12T02:38:51.542920+0000","last_clean":"2020-06-12T02:38:51.542920+0000","last_became_active":"2020-06-12T02:38:45.496715+0000","last_became_peered":"2020-06-12T02:38:45.496715+0000","last_unstale":"2020-06-12T02:38:51.542920+0000","last_undegraded":"2020-06-12T02:38:51.542920+0000","last_fullsized":"2020-06-12T02:38:51.542920+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":1742,"ondisk_log_size":1742,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":306,"num_read_kb":169,"num_write":699,"num_write_kb":201836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1396736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"10","reported_epoch":"70","state":"active+clean","last_fresh":"2020-06-12T02:38:50.380588+0000","last_change":"2020-06-12T02:38:50.376838+0000","last_active":"2020-06-12T02:38:50.380588+0000","last_peered":"2020-06-12T02:38:50.380588+0000","last_clean":"2020-06-12T02:38:50.380588+0000","last_became_active":"2020-06-12T02:38:50.375858+0000","last_became_peered":"2020-06-12T02:38:50.375858+0000","last_unstale":"2020-06-12T02:38:50.380588+0000","last_undegraded":"2020-06-12T02:38:50.380588+0000","last_fullsized":"2020-06-12T02:38:50.380588+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:38:45.158691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:38:45.158691+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"76","reported_epoch":"70","state":"active+clean","last_fresh":"2020-06-12T02:38:50.369520+0000","last_change":"2020-06-12T02:38:22.034355+0000","last_active":"2020-06-12T02:38:50.369520+0000","last_peered":"2020-06-12T02:38:50.369520+0000","last_clean":"2020-06-12T02:38:50.369520+0000","last_became_active":"2020-06-12T02:37:54.554986+0000","last_became_peered":"2020-06-12T02:37:54.554986+0000","last_unstale":"2020-06-12T02:38:50.369520+0000","last_undegraded":"2020-06-12T02:38:50.369520+0000","last_fullsized":"2020-06-12T02:38:50.369520+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:38:22.034263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:38:22.034263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"87","reported_epoch":"70","state":"active+clean","last_fresh":"2020-06-12T02:38:50.390972+0000","last_change":"2020-06-12T02:38:50.386477+0000","last_active":"2020-06-12T02:38:50.390972+0000","last_peered":"2020-06-12T02:38:50.390972+0000","last_clean":"2020-06-12T02:38:50.390972+0000","last_became_active":"2020-06-12T02:38:50.385912+0000","last_became_peered":"2020-06-12T02:38:50.385912+0000","last_unstale":"2020-06-12T02:38:50.390972+0000","last_undegraded":"2020-06-12T02:38:50.390972+0000","last_fullsized":"2020-06-12T02:38:50.390972+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:38:47.662610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:38:47.662610+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":6864896,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5274,"num_read_kb":2866,"num_write":11790,"num_write_kb":3359796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":13549568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6610944,"data_stored":6610944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25560,"ondisk_log_size":25560,"up":78,"acting":78,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26157,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":34,"seq":146028888077,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050424,"kb_used_data":1840,"kb_used_omap":25,"kb_used_meta":1048550,"kb_avail":93321416,"statfs":{"total":96636764160,"available":95561129984,"internally_reserved":1073741824,"allocated":1884160,"data_stored":1446738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26157,"internal_metadata":1073715667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.024}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92800000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.145}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.339}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2270000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574871,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049928,"kb_used_data":1344,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321912,"statfs":{"total":96636764160,"available":95561637888,"internally_reserved":1073741824,"allocated":1376256,"data_stored":950578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.655,"5min":1.655,"15min":1.655},"last":1.0109999999999999},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.879}]},{"osd":1,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.335,"5min":1.335,"15min":1.335},"last":0.67900000000000005},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.923,"5min":1.923,"15min":1.923},"last":1.107}]},{"osd":2,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.01,"15min":1.01},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.6699999999999999,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.653,"5min":1.653,"15min":1.653},"last":0.90200000000000002}]},{"osd":3,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.145,"15min":1.145},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.5649999999999999,"5min":1.5649999999999999,"15min":1.5649999999999999},"last":1.228},{"interface":"front","average":{"1min":1.143,"5min":1.143,"15min":1.143},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.556,"5min":1.556,"15min":1.556},"last":1.2509999999999999}]},{"osd":4,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.246,"5min":1.246,"15min":1.246},"last":1.3600000000000001},{"interface":"front","average":{"1min":1.071,"5min":1.071,"15min":1.071},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.673,"5min":1.673,"15min":1.673},"last":1.387}]},{"osd":5,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.8120000000000001,"5min":1.8120000000000001,"15min":1.8120000000000001},"last":0.998},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.677,"5min":1.677,"15min":1.677},"last":1.645}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5189999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049384,"kb_used_data":800,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322456,"statfs":{"total":96636764160,"available":95562194944,"internally_reserved":1073741824,"allocated":819200,"data_stored":393522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 12 02:38:14 2020","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.548,"5min":1.548,"15min":1.548},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":2.149,"5min":2.149,"15min":2.149},"last":0.88600000000000001}]},{"osd":2,"last update":"Fri Jun 12 02:38:14 2020","interfaces":[{"interface":"back","average":{"1min":1,"5min":1,"15min":1},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":3.0009999999999999,"5min":3.0009999999999999,"15min":3.0009999999999999},"last":0.746},{"interface":"front","average":{"1min":1.014,"5min":1.014,"15min":1.014},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":2.4159999999999999,"5min":2.4159999999999999,"15min":2.4159999999999999},"last":0.62}]},{"osd":3,"last update":"Fri Jun 12 02:38:14 2020","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":2.133,"5min":2.133,"15min":2.133},"last":1.052},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":2.4609999999999999,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":1.1559999999999999}]},{"osd":4,"last update":"Fri Jun 12 02:38:14 2020","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":2.1749999999999998,"5min":2.1749999999999998,"15min":2.1749999999999998},"last":0.95599999999999996},{"interface":"front","average":{"1min":1.302,"5min":1.302,"15min":1.302},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":2.2919999999999998,"5min":2.2919999999999998,"15min":2.2919999999999998},"last":1.1910000000000001}]},{"osd":5,"last update":"Fri Jun 12 02:38:14 2020","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.3200000000000001,"15min":1.3200000000000001},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":2.3090000000000002,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.448,"5min":1.448,"15min":1.448},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.835,"5min":2.835,"15min":2.835},"last":1.0189999999999999}]},{"osd":6,"last update":"Fri Jun 12 02:38:14 2020","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.389,"15min":1.389},"min":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"max":{"1min":2.4649999999999999,"5min":2.4649999999999999,"15min":2.4649999999999999},"last":1.448},{"interface":"front","average":{"1min":1.6299999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"max":{"1min":3.1589999999999998,"5min":3.1589999999999998,"15min":3.1589999999999998},"last":1.123}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.337}]}]},{"osd":0,"up_from":13,"seq":55834574872,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049788,"kb_used_data":1204,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322052,"statfs":{"total":96636764160,"available":95561781248,"internally_reserved":1073741824,"allocated":1232896,"data_stored":801382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jun 12 02:38:16 2020","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.7789999999999999,"5min":1.7789999999999999,"15min":1.7789999999999999},"last":0.92200000000000004}]},{"osd":2,"last update":"Fri Jun 12 02:38:16 2020","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":2.141,"5min":2.141,"15min":2.141},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.873,"5min":1.873,"15min":1.873},"last":0.77000000000000002}]},{"osd":3,"last update":"Fri Jun 12 02:38:16 2020","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":2.2320000000000002,"5min":2.2320000000000002,"15min":2.2320000000000002},"last":0.97599999999999998},{"interface":"front","average":{"1min":1.23,"5min":1.23,"15min":1.23},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.8220000000000001,"5min":1.8220000000000001,"15min":1.8220000000000001},"last":1.056}]},{"osd":4,"last update":"Fri Jun 12 02:38:16 2020","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.7709999999999999,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.554,"5min":1.554,"15min":1.554},"last":1.169}]},{"osd":5,"last update":"Fri Jun 12 02:38:16 2020","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.212,"15min":1.212},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":1.6699999999999999,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":1.079},{"interface":"front","average":{"1min":1.502,"5min":1.502,"15min":1.502},"min":{"1min":1.006,"5min":1.006,"15min":1.006},"max":{"1min":2.6240000000000001,"5min":2.6240000000000001,"15min":2.6240000000000001},"last":1.1870000000000001}]},{"osd":6,"last update":"Fri Jun 12 02:38:16 2020","interfaces":[{"interface":"back","average":{"1min":1.5840000000000001,"5min":1.5840000000000001,"15min":1.5840000000000001},"min":{"1min":1.157,"5min":1.157,"15min":1.157},"max":{"1min":3.2440000000000002,"5min":3.2440000000000002,"15min":3.2440000000000002},"last":1.28},{"interface":"front","average":{"1min":1.335,"5min":1.335,"15min":1.335},"min":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"max":{"1min":3.105,"5min":3.105,"15min":3.105},"last":1.101}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.246}]}]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049968,"kb_used_data":1384,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321872,"statfs":{"total":96636764160,"available":95561596928,"internally_reserved":1073741824,"allocated":1417216,"data_stored":991538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 12 02:38:10 2020","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.871,"15min":0.871},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.9259999999999999,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":3.0049999999999999,"5min":3.0049999999999999,"15min":3.0049999999999999},"last":1.524}]},{"osd":1,"last update":"Fri Jun 12 02:38:10 2020","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.046,"15min":1.046},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":2.21,"5min":2.21,"15min":2.21},"last":1.0049999999999999},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":2.5219999999999998,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":1.708}]},{"osd":3,"last update":"Fri Jun 12 02:38:10 2020","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1,"5min":1,"15min":1},"last":1.97},{"interface":"front","average":{"1min":1.145,"5min":1.145,"15min":1.145},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.2759999999999998,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":2.552}]},{"osd":4,"last update":"Fri Jun 12 02:38:10 2020","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.321,"15min":1.321},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":2.6949999999999998,"5min":2.6949999999999998,"15min":2.6949999999999998},"last":3.5859999999999999},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"min":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"max":{"1min":1.7549999999999999,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":2.113}]},{"osd":5,"last update":"Fri Jun 12 02:38:10 2020","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"min":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"max":{"1min":2.1419999999999999,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":2.2250000000000001},{"interface":"front","average":{"1min":1.399,"5min":1.399,"15min":1.399},"min":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"max":{"1min":2.851,"5min":2.851,"15min":2.851},"last":3.7610000000000001}]},{"osd":6,"last update":"Fri Jun 12 02:38:10 2020","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":1.7110000000000001,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":1.373},{"interface":"front","average":{"1min":1.462,"5min":1.462,"15min":1.462},"min":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"max":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":2.3719999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.9119999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574871,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050488,"kb_used_data":1904,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321352,"statfs":{"total":96636764160,"available":95561064448,"internally_reserved":1073741824,"allocated":1949696,"data_stored":1524018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 12 02:38:12 2020","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":2.0339999999999998,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.9219999999999999,"5min":1.9219999999999999,"15min":1.9219999999999999},"last":0.72799999999999998}]},{"osd":1,"last update":"Fri Jun 12 02:38:12 2020","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.9670000000000001,"5min":1.9670000000000001,"15min":1.9670000000000001},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":2.9969999999999999,"5min":2.9969999999999999,"15min":2.9969999999999999},"last":0.78600000000000003}]},{"osd":2,"last update":"Fri Jun 12 02:38:12 2020","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":4.1079999999999997,"5min":4.1079999999999997,"15min":4.1079999999999997},"last":1.181},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":2.504,"5min":2.504,"15min":2.504},"last":0.92600000000000005}]},{"osd":4,"last update":"Fri Jun 12 02:38:12 2020","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":2.359,"5min":2.359,"15min":2.359},"last":0.90700000000000003},{"interface":"front","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":3.9319999999999999,"5min":3.9319999999999999,"15min":3.9319999999999999},"last":1.3340000000000001}]},{"osd":5,"last update":"Fri Jun 12 02:38:12 2020","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":2.4279999999999999,"5min":2.4279999999999999,"15min":2.4279999999999999},"last":1.411},{"interface":"front","average":{"1min":1.452,"5min":1.452,"15min":1.452},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":5.0579999999999998,"5min":5.0579999999999998,"15min":5.0579999999999998},"last":1.0640000000000001}]},{"osd":6,"last update":"Fri Jun 12 02:38:12 2020","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.29,"15min":1.29},"min":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"max":{"1min":4.0410000000000004,"5min":4.0410000000000004,"15min":4.0410000000000004},"last":1.034},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.3620000000000001,"15min":1.3620000000000001},"min":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"max":{"1min":3.0230000000000001,"5min":3.0230000000000001,"15min":3.0230000000000001},"last":1.4850000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2010000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574870,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1051832,"kb_used_data":3248,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93320008,"statfs":{"total":96636764160,"available":95559688192,"internally_reserved":1073741824,"allocated":3325952,"data_stored":2894438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.712,"5min":1.712,"15min":1.712},"last":1.075},{"interface":"front","average":{"1min":1.02,"5min":1.02,"15min":1.02},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":2.855,"5min":2.855,"15min":2.855},"last":1.6579999999999999}]},{"osd":1,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.8240000000000001,"5min":1.8240000000000001,"15min":1.8240000000000001},"last":0.64400000000000002}]},{"osd":2,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":3.1480000000000001,"5min":3.1480000000000001,"15min":3.1480000000000001},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"min":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"max":{"1min":3.1259999999999999,"5min":3.1259999999999999,"15min":3.1259999999999999},"last":1.284}]},{"osd":3,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.079,"15min":1.079},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":2.411,"5min":2.411,"15min":2.411},"last":0.81399999999999995},{"interface":"front","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":2.734,"5min":2.734,"15min":2.734},"last":1.008}]},{"osd":5,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.04,"15min":1.04},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.6399999999999999,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":0.93999999999999995},{"interface":"front","average":{"1min":1.115,"5min":1.115,"15min":1.115},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"last":1.52}]},{"osd":6,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":0.998,"15min":0.998},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":1.6850000000000001},{"interface":"front","average":{"1min":1.341,"5min":1.341,"15min":1.341},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":2.1850000000000001,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":1.3580000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4950000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574870,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050128,"kb_used_data":1544,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321712,"statfs":{"total":96636764160,"available":95561433088,"internally_reserved":1073741824,"allocated":1581056,"data_stored":1159008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 12 02:38:13 2020","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.456,"5min":1.456,"15min":1.456},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.597,"5min":1.597,"15min":1.597},"last":0.62}]},{"osd":1,"last update":"Fri Jun 12 02:38:13 2020","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.73799999999999999}]},{"osd":2,"last update":"Fri Jun 12 02:38:13 2020","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.82699999999999996},{"interface":"front","average":{"1min":1.012,"5min":1.012,"15min":1.012},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.4259999999999999,"5min":1.4259999999999999,"15min":1.4259999999999999},"last":0.77100000000000002}]},{"osd":3,"last update":"Fri Jun 12 02:38:13 2020","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":2.004,"5min":2.004,"15min":2.004},"last":0.91100000000000003},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.627,"5min":1.627,"15min":1.627},"last":0.94999999999999996}]},{"osd":4,"last update":"Fri Jun 12 02:38:13 2020","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.997,"15min":0.997},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.796,"5min":1.796,"15min":1.796},"last":0.64900000000000002},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":1.0780000000000001}]},{"osd":6,"last update":"Fri Jun 12 02:38:13 2020","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.827,"5min":1.827,"15min":1.827},"last":0.79500000000000004},{"interface":"front","average":{"1min":1.228,"5min":1.228,"15min":1.228},"min":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"max":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":1.012}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1000000000000001}]}]}]}} 2020-06-12T02:38:56.192 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json 2020-06-12T02:38:56.194 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:56.195 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:38:56.195 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:38:56.195 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:38:56.196 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:56.196 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-06-12T02:38:56.197 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:56.192+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:56.302 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:56.296+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:56.316 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:38:56.317 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:38:56.317 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:38:56.318 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:38:56.318 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.318 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8464 3.fs0 3.57bc3c1f (undecoded) ondisk+retry+write+known_if_redirected e34)", 2020-06-12T02:38:56.319 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:53.549364+0000", 2020-06-12T02:38:56.319 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 62.762974489000001, 2020-06-12T02:38:56.319 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.003107494, 2020-06-12T02:38:56.319 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:56.320 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:38:56.320 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:56.320 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:56.321 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:56.321 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8464 2020-06-12T02:38:56.321 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.322 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:56.322 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.322 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:56.323 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.549364+0000", 2020-06-12T02:38:56.323 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:56.323 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.334 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.334 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:56.334 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.549364+0000", 2020-06-12T02:38:56.334 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023109609999999998 2020-06-12T02:38:56.335 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.335 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.335 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:56.335 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551675+0000", 2020-06-12T02:38:56.335 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3331e-05 2020-06-12T02:38:56.335 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.336 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.336 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:56.336 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551688+0000", 2020-06-12T02:38:56.336 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4173e-05 2020-06-12T02:38:56.336 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.337 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.337 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:56.337 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551703+0000", 2020-06-12T02:38:56.337 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0454e-05 2020-06-12T02:38:56.337 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.349 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.349 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:56.349 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551713+0000", 2020-06-12T02:38:56.349 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031069050000000001 2020-06-12T02:38:56.350 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.350 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.351 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:56.351 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554820+0000", 2020-06-12T02:38:56.351 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2820000000000002e-06 2020-06-12T02:38:56.351 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.352 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.352 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2020-06-12T02:38:56.352 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554826+0000", 2020-06-12T02:38:56.352 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99762648499999995 2020-06-12T02:38:56.352 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.352 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.353 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:56.353 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552453+0000", 2020-06-12T02:38:56.353 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8902999999999998e-05 2020-06-12T02:38:56.353 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.353 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.354 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:56.354 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552472+0000", 2020-06-12T02:38:56.354 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.003107494 2020-06-12T02:38:56.354 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.354 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:56.355 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.355 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.355 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.355 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8465 3.fs0 3.57bc3c1f (undecoded) ondisk+retry+write+known_if_redirected e34)", 2020-06-12T02:38:56.355 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:53.551774+0000", 2020-06-12T02:38:56.355 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 62.760564852999998, 2020-06-12T02:38:56.356 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.000741442, 2020-06-12T02:38:56.356 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:56.356 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:38:56.362 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:56.362 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:56.362 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:56.363 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8465 2020-06-12T02:38:56.364 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.364 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:56.364 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.364 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:56.365 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551774+0000", 2020-06-12T02:38:56.365 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:56.365 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.365 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.365 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:56.373 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551774+0000", 2020-06-12T02:38:56.373 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0010760539999999999 2020-06-12T02:38:56.374 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.374 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.374 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:56.374 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552850+0000", 2020-06-12T02:38:56.374 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0139000000000001e-05 2020-06-12T02:38:56.375 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.375 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.375 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:56.375 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552860+0000", 2020-06-12T02:38:56.375 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7646999999999999e-05 2020-06-12T02:38:56.375 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.376 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.376 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:56.376 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552878+0000", 2020-06-12T02:38:56.376 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2916e-05 2020-06-12T02:38:56.376 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.377 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.377 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:56.377 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552890+0000", 2020-06-12T02:38:56.377 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0019574610000000002 2020-06-12T02:38:56.377 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.378 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.378 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:56.378 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554848+0000", 2020-06-12T02:38:56.378 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0399999999999995e-07 2020-06-12T02:38:56.378 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.378 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.379 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2020-06-12T02:38:56.379 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554849+0000", 2020-06-12T02:38:56.379 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99765586100000003 2020-06-12T02:38:56.379 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.379 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.380 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:56.380 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552504+0000", 2020-06-12T02:38:56.380 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.066e-05 2020-06-12T02:38:56.380 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.380 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.380 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:56.381 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552515+0000", 2020-06-12T02:38:56.381 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.000741442 2020-06-12T02:38:56.381 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.381 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:56.381 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.382 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.382 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.382 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8466 3.fs0 3.57bc3c1f (undecoded) ondisk+write+known_if_redirected e34)", 2020-06-12T02:38:56.382 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:53.552953+0000", 2020-06-12T02:38:56.382 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 62.759385573000003, 2020-06-12T02:38:56.382 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99960144299999998, 2020-06-12T02:38:56.383 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:56.383 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:38:56.383 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:56.383 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:56.383 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:56.384 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8466 2020-06-12T02:38:56.384 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.384 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:56.384 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.384 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:56.384 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552953+0000", 2020-06-12T02:38:56.385 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:56.385 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.385 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.385 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:56.385 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552953+0000", 2020-06-12T02:38:56.386 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013847100000000001 2020-06-12T02:38:56.386 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.386 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.386 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:56.386 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553091+0000", 2020-06-12T02:38:56.386 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7190000000000004e-06 2020-06-12T02:38:56.387 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.387 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.387 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:56.387 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553098+0000", 2020-06-12T02:38:56.387 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.171e-05 2020-06-12T02:38:56.388 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.388 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.388 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:56.388 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553110+0000", 2020-06-12T02:38:56.388 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5770000000000001e-06 2020-06-12T02:38:56.388 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.389 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.389 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:56.389 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553115+0000", 2020-06-12T02:38:56.390 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0017512739999999999 2020-06-12T02:38:56.390 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.391 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.391 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:56.391 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554867+0000", 2020-06-12T02:38:56.391 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1200000000000003e-07 2020-06-12T02:38:56.392 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.392 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.392 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2020-06-12T02:38:56.392 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554867+0000", 2020-06-12T02:38:56.392 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99767707100000003 2020-06-12T02:38:56.393 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.393 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.393 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:56.393 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552544+0000", 2020-06-12T02:38:56.393 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0009e-05 2020-06-12T02:38:56.393 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.394 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.394 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:56.394 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552554+0000", 2020-06-12T02:38:56.394 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99960144299999998 2020-06-12T02:38:56.394 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.395 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:56.395 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.395 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.395 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.395 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8467 3.fs0 3.57bc3c1f (undecoded) ondisk+write+known_if_redirected e34)", 2020-06-12T02:38:56.396 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:53.553172+0000", 2020-06-12T02:38:56.396 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 62.759166356999998, 2020-06-12T02:38:56.396 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99942086200000002, 2020-06-12T02:38:56.396 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:56.396 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:38:56.396 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:56.397 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:56.397 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:56.397 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8467 2020-06-12T02:38:56.397 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.397 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:56.398 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.398 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:56.398 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553172+0000", 2020-06-12T02:38:56.398 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:56.398 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.399 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.399 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:56.399 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553172+0000", 2020-06-12T02:38:56.399 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5913000000000003e-05 2020-06-12T02:38:56.399 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.399 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.400 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:56.400 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553248+0000", 2020-06-12T02:38:56.400 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7740000000000001e-06 2020-06-12T02:38:56.400 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.400 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.401 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:56.401 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553254+0000", 2020-06-12T02:38:56.401 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1494999999999999e-05 2020-06-12T02:38:56.401 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.401 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.402 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:56.402 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553265+0000", 2020-06-12T02:38:56.402 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5170000000000001e-06 2020-06-12T02:38:56.402 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.402 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.402 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:56.403 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553270+0000", 2020-06-12T02:38:56.403 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001616567 2020-06-12T02:38:56.403 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.403 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.403 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:56.404 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554886+0000", 2020-06-12T02:38:56.404 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7800000000000001e-07 2020-06-12T02:38:56.404 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.404 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.404 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2020-06-12T02:38:56.404 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554887+0000", 2020-06-12T02:38:56.405 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99769652399999997 2020-06-12T02:38:56.405 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.405 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.405 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:56.405 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552584+0000", 2020-06-12T02:38:56.406 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4939999999999992e-06 2020-06-12T02:38:56.406 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.406 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.406 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:56.406 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552593+0000", 2020-06-12T02:38:56.406 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99942086200000002 2020-06-12T02:38:56.407 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.407 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:56.407 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.407 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.407 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.408 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8468 3.fs0 3.57bc3c1f (undecoded) ondisk+read+rwordered+known_if_redirected e34)", 2020-06-12T02:38:56.408 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:53.553311+0000", 2020-06-12T02:38:56.408 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 62.759027758999999, 2020-06-12T02:38:56.408 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99988704799999994, 2020-06-12T02:38:56.408 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:56.408 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:38:56.409 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:56.409 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:56.409 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:56.409 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8468 2020-06-12T02:38:56.409 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.410 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:56.410 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.410 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:56.410 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553311+0000", 2020-06-12T02:38:56.410 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:56.411 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.411 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.411 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:56.411 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553311+0000", 2020-06-12T02:38:56.411 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2050000000000001e-05 2020-06-12T02:38:56.411 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.412 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.412 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:56.412 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553333+0000", 2020-06-12T02:38:56.412 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5489999999999998e-06 2020-06-12T02:38:56.412 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.413 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.413 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:56.413 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553336+0000", 2020-06-12T02:38:56.413 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1620000000000002e-06 2020-06-12T02:38:56.413 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.413 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.414 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:56.414 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553344+0000", 2020-06-12T02:38:56.414 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.807e-06 2020-06-12T02:38:56.414 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.414 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.415 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:56.415 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553346+0000", 2020-06-12T02:38:56.415 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001560575 2020-06-12T02:38:56.415 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.415 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.415 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:56.416 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554907+0000", 2020-06-12T02:38:56.416 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1429999999999999e-06 2020-06-12T02:38:56.416 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.416 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.416 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2020-06-12T02:38:56.417 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554908+0000", 2020-06-12T02:38:56.417 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99827844600000004 2020-06-12T02:38:56.417 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.417 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.417 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:56.417 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.553187+0000", 2020-06-12T02:38:56.418 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1316e-05 2020-06-12T02:38:56.418 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.418 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.418 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:56.418 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.553198+0000", 2020-06-12T02:38:56.419 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99988704799999994 2020-06-12T02:38:56.419 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.419 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:56.419 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.419 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.419 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.420 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11992 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [append 0~581632] snapc 0=[] ondisk+write+known_if_redirected e56)", 2020-06-12T02:38:56.420 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:23.783061+0000", 2020-06-12T02:38:56.420 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 32.529277577999999, 2020-06-12T02:38:56.420 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022850993, 2020-06-12T02:38:56.420 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:56.421 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:56.421 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:56.421 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:56.421 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:56.421 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11992 2020-06-12T02:38:56.421 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.422 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:56.422 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.422 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:56.422 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783061+0000", 2020-06-12T02:38:56.422 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:56.423 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.423 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.423 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:56.423 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783061+0000", 2020-06-12T02:38:56.423 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047999300000000001 2020-06-12T02:38:56.423 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.424 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.424 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:56.424 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783541+0000", 2020-06-12T02:38:56.424 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5649999999999998e-06 2020-06-12T02:38:56.424 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.425 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.425 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:56.425 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783548+0000", 2020-06-12T02:38:56.425 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3559e-05 2020-06-12T02:38:56.425 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.425 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.426 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:56.426 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783561+0000", 2020-06-12T02:38:56.426 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3539999999999996e-06 2020-06-12T02:38:56.426 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.426 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.427 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:56.427 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783567+0000", 2020-06-12T02:38:56.427 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001353604 2020-06-12T02:38:56.427 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.427 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.428 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:56.428 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.784921+0000", 2020-06-12T02:38:56.428 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000111853 2020-06-12T02:38:56.428 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.428 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.428 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:56.429 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.785033+0000", 2020-06-12T02:38:56.429 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051647919999999996 2020-06-12T02:38:56.429 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.429 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.429 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:56.429 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.790198+0000", 2020-06-12T02:38:56.430 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000110742 2020-06-12T02:38:56.430 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.430 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.430 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:56.430 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.790308+0000", 2020-06-12T02:38:56.431 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00090050099999999995 2020-06-12T02:38:56.431 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.431 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.431 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:56.431 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.791209+0000", 2020-06-12T02:38:56.433 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010078284999999999 2020-06-12T02:38:56.433 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.433 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.434 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:56.434 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.801287+0000", 2020-06-12T02:38:56.434 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045443109999999997 2020-06-12T02:38:56.434 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.435 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.435 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:56.435 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.805832+0000", 2020-06-12T02:38:56.435 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.0433999999999996e-05 2020-06-12T02:38:56.435 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.436 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.436 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:56.436 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.805912+0000", 2020-06-12T02:38:56.436 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022850993 2020-06-12T02:38:56.436 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.437 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:56.437 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.437 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.438 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.438 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11994 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected e56)", 2020-06-12T02:38:56.438 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:23.788734+0000", 2020-06-12T02:38:56.438 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 32.523604933000001, 2020-06-12T02:38:56.438 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024022136999999999, 2020-06-12T02:38:56.439 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:56.439 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:56.439 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:56.439 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:56.439 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:56.439 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11994 2020-06-12T02:38:56.440 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.440 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:56.440 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.440 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:56.440 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.788734+0000", 2020-06-12T02:38:56.441 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:56.441 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.441 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.441 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:56.441 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.788734+0000", 2020-06-12T02:38:56.442 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00034212499999999998 2020-06-12T02:38:56.442 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.443 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.444 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:56.444 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789076+0000", 2020-06-12T02:38:56.444 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.254e-06 2020-06-12T02:38:56.444 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.444 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.445 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:56.445 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789081+0000", 2020-06-12T02:38:56.445 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8249999999999994e-06 2020-06-12T02:38:56.446 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.446 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.446 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:56.446 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789090+0000", 2020-06-12T02:38:56.446 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2259999999999999e-06 2020-06-12T02:38:56.447 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.447 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.447 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:56.447 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789094+0000", 2020-06-12T02:38:56.447 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2913000000000001e-05 2020-06-12T02:38:56.448 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.448 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.448 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:56.449 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789127+0000", 2020-06-12T02:38:56.449 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2001000000000005e-05 2020-06-12T02:38:56.449 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.449 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.449 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:56.452 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789189+0000", 2020-06-12T02:38:56.452 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0097474550000000004 2020-06-12T02:38:56.452 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.452 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.452 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:56.453 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.798936+0000", 2020-06-12T02:38:56.453 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9882000000000005e-05 2020-06-12T02:38:56.453 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.453 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.453 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:56.454 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.799016+0000", 2020-06-12T02:38:56.454 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00048691000000000001 2020-06-12T02:38:56.454 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.454 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.454 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:56.455 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.799503+0000", 2020-06-12T02:38:56.455 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002766363 2020-06-12T02:38:56.455 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.456 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.456 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:56.457 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.802270+0000", 2020-06-12T02:38:56.457 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010404759 2020-06-12T02:38:56.457 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.457 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.457 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:56.458 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.812674+0000", 2020-06-12T02:38:56.458 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1423999999999998e-05 2020-06-12T02:38:56.458 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.458 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.458 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:56.458 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.812756+0000", 2020-06-12T02:38:56.459 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024022136999999999 2020-06-12T02:38:56.459 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.459 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:56.459 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.459 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.460 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.460 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11995 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e56)", 2020-06-12T02:38:56.460 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:23.789128+0000", 2020-06-12T02:38:56.460 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 32.523210487, 2020-06-12T02:38:56.460 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031795508, 2020-06-12T02:38:56.461 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:56.461 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:56.461 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:56.461 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:56.461 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:56.461 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11995 2020-06-12T02:38:56.462 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.462 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:56.462 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.462 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:56.462 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789128+0000", 2020-06-12T02:38:56.463 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:56.463 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.463 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.463 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:56.463 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789128+0000", 2020-06-12T02:38:56.463 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4419999999999998e-05 2020-06-12T02:38:56.464 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.464 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.464 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:56.464 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789172+0000", 2020-06-12T02:38:56.464 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7110000000000001e-06 2020-06-12T02:38:56.465 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.465 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.465 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:56.465 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789176+0000", 2020-06-12T02:38:56.465 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5089999999999999e-06 2020-06-12T02:38:56.466 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.466 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.466 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:56.466 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789183+0000", 2020-06-12T02:38:56.466 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.497e-06 2020-06-12T02:38:56.467 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.467 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.467 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:56.467 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789185+0000", 2020-06-12T02:38:56.467 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012172176 2020-06-12T02:38:56.468 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.468 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.468 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:56.468 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.801357+0000", 2020-06-12T02:38:56.468 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4264000000000006e-05 2020-06-12T02:38:56.468 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.469 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.469 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:56.469 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.801432+0000", 2020-06-12T02:38:56.469 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00029680499999999997 2020-06-12T02:38:56.469 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.470 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.470 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:56.470 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.801728+0000", 2020-06-12T02:38:56.470 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00066278300000000001 2020-06-12T02:38:56.470 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.470 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.471 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:56.471 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.802391+0000", 2020-06-12T02:38:56.471 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018429867999999999 2020-06-12T02:38:56.471 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.471 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.472 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:56.472 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.820821+0000", 2020-06-12T02:38:56.472 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000102475 2020-06-12T02:38:56.472 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.473 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.473 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:56.473 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.820924+0000", 2020-06-12T02:38:56.473 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031795508 2020-06-12T02:38:56.473 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.474 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:56.474 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.474 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.474 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.474 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11996 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e56)", 2020-06-12T02:38:56.474 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:23.789235+0000", 2020-06-12T02:38:56.475 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 32.523103636000002, 2020-06-12T02:38:56.475 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.034785745, 2020-06-12T02:38:56.475 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:56.475 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:56.475 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:56.476 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:56.476 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:56.476 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11996 2020-06-12T02:38:56.476 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.476 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:56.476 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.477 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:56.477 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789235+0000", 2020-06-12T02:38:56.477 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:56.477 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.477 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.478 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:56.478 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789235+0000", 2020-06-12T02:38:56.478 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6375000000000001e-05 2020-06-12T02:38:56.478 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.478 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.479 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:56.479 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789261+0000", 2020-06-12T02:38:56.479 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2679999999999999e-06 2020-06-12T02:38:56.479 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.479 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.479 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:56.480 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789265+0000", 2020-06-12T02:38:56.480 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6039999999999998e-06 2020-06-12T02:38:56.480 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.480 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.480 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:56.481 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789271+0000", 2020-06-12T02:38:56.481 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4219999999999999e-06 2020-06-12T02:38:56.481 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.481 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.481 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:56.481 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789274+0000", 2020-06-12T02:38:56.482 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013034463 2020-06-12T02:38:56.482 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.482 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.482 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:56.482 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.802308+0000", 2020-06-12T02:38:56.483 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8862000000000005e-05 2020-06-12T02:38:56.483 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.483 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.483 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:56.483 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.802387+0000", 2020-06-12T02:38:56.483 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019102297000000001 2020-06-12T02:38:56.484 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.484 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.484 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:56.484 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.821489+0000", 2020-06-12T02:38:56.484 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8838999999999996e-05 2020-06-12T02:38:56.485 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.485 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.485 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:56.485 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.821558+0000", 2020-06-12T02:38:56.485 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002462615 2020-06-12T02:38:56.485 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.486 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.486 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:56.486 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.824021+0000", 2020-06-12T02:38:56.486 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.034785745 2020-06-12T02:38:56.486 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.487 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:56.487 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.487 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.487 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.487 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:15005 3.ds0 3:b3445631:::smithi10729407-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e59)", 2020-06-12T02:38:56.487 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:32.192302+0000", 2020-06-12T02:38:56.488 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 24.120036401, 2020-06-12T02:38:56.488 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024128666, 2020-06-12T02:38:56.488 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:56.488 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:56.488 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:56.489 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:56.489 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:56.489 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 15005 2020-06-12T02:38:56.489 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.489 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:56.490 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.490 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:56.490 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192302+0000", 2020-06-12T02:38:56.490 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:56.490 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.490 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.491 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:56.491 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192302+0000", 2020-06-12T02:38:56.491 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3415999999999999e-05 2020-06-12T02:38:56.491 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.491 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.492 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:56.492 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192346+0000", 2020-06-12T02:38:56.492 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.93e-06 2020-06-12T02:38:56.492 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.492 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.492 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:56.493 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192352+0000", 2020-06-12T02:38:56.493 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3397000000000001e-05 2020-06-12T02:38:56.493 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.493 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.493 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:56.494 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192365+0000", 2020-06-12T02:38:56.494 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3950000000000003e-06 2020-06-12T02:38:56.494 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.494 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.495 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:56.495 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192369+0000", 2020-06-12T02:38:56.495 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010208887999999999 2020-06-12T02:38:56.495 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.495 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.496 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:56.496 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.202578+0000", 2020-06-12T02:38:56.496 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000105611 2020-06-12T02:38:56.496 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.496 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.496 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:56.497 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.202684+0000", 2020-06-12T02:38:56.497 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0091288959999999992 2020-06-12T02:38:56.497 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.497 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.497 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:56.498 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.211813+0000", 2020-06-12T02:38:56.498 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000101798 2020-06-12T02:38:56.498 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.498 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.498 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:56.498 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.211914+0000", 2020-06-12T02:38:56.499 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045163349999999998 2020-06-12T02:38:56.499 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.499 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.499 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:56.499 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.216431+0000", 2020-06-12T02:38:56.500 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024128666 2020-06-12T02:38:56.500 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.500 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:56.500 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.500 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.500 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.501 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:16810 3.ds0 3:b983fd28:::smithi10729407-1:head [append 0~524288] snapc 0=[] ondisk+write+known_if_redirected e61)", 2020-06-12T02:38:56.501 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:37.642363+0000", 2020-06-12T02:38:56.501 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 18.669975146999999, 2020-06-12T02:38:56.501 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022969679, 2020-06-12T02:38:56.501 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:56.502 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:56.502 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:56.502 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:56.502 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:56.503 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 16810 2020-06-12T02:38:56.503 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.504 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:56.504 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.504 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:56.504 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.642363+0000", 2020-06-12T02:38:56.504 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:56.504 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.505 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.505 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:56.505 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.642363+0000", 2020-06-12T02:38:56.505 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00069050800000000005 2020-06-12T02:38:56.505 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.506 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.506 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:56.506 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.643054+0000", 2020-06-12T02:38:56.506 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8220000000000007e-06 2020-06-12T02:38:56.506 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.507 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.507 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:56.507 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.643062+0000", 2020-06-12T02:38:56.507 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3742e-05 2020-06-12T02:38:56.507 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.508 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.508 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:56.508 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.643075+0000", 2020-06-12T02:38:56.508 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5910000000000001e-06 2020-06-12T02:38:56.508 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.508 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.509 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:56.509 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.643082+0000", 2020-06-12T02:38:56.509 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0064097879999999996 2020-06-12T02:38:56.509 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.509 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.510 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:56.510 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.649492+0000", 2020-06-12T02:38:56.510 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8151999999999994e-05 2020-06-12T02:38:56.510 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.510 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.510 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:56.511 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.649560+0000", 2020-06-12T02:38:56.511 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0020294940000000002 2020-06-12T02:38:56.511 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.511 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.511 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:56.512 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.651590+0000", 2020-06-12T02:38:56.512 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5242999999999997e-05 2020-06-12T02:38:56.512 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.512 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.513 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:56.513 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.651675+0000", 2020-06-12T02:38:56.513 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00069163799999999995 2020-06-12T02:38:56.513 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.513 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.514 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:56.514 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.652366+0000", 2020-06-12T02:38:56.514 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012711642 2020-06-12T02:38:56.514 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.514 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.515 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:56.515 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.665078+0000", 2020-06-12T02:38:56.515 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017519200000000001 2020-06-12T02:38:56.516 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.516 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.516 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:56.516 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.665253+0000", 2020-06-12T02:38:56.516 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9867e-05 2020-06-12T02:38:56.517 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.517 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.517 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:56.517 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.665333+0000", 2020-06-12T02:38:56.517 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022969679 2020-06-12T02:38:56.518 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.518 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:56.518 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.518 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.518 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.518 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18287 3.ds0 3:b983fd28:::smithi10729407-1:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:38:56.519 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.975980+0000", 2020-06-12T02:38:56.519 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 14.336358321000001, 2020-06-12T02:38:56.519 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.023612415000000001, 2020-06-12T02:38:56.519 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:56.519 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:56.520 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:56.520 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:56.520 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:56.521 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18287 2020-06-12T02:38:56.521 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.521 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:56.521 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.522 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:56.522 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.975980+0000", 2020-06-12T02:38:56.522 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:56.522 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.522 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.523 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:56.523 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.975980+0000", 2020-06-12T02:38:56.523 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0175999999999999e-05 2020-06-12T02:38:56.523 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.524 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.524 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:56.524 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976070+0000", 2020-06-12T02:38:56.524 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7489999999999998e-06 2020-06-12T02:38:56.524 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.525 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.525 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:56.525 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976078+0000", 2020-06-12T02:38:56.525 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5308999999999999e-05 2020-06-12T02:38:56.525 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.525 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.526 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:56.526 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976093+0000", 2020-06-12T02:38:56.526 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7149999999999997e-06 2020-06-12T02:38:56.526 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.526 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.527 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:56.527 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976098+0000", 2020-06-12T02:38:56.527 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0067110759999999998 2020-06-12T02:38:56.527 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.527 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.528 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:56.528 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.982809+0000", 2020-06-12T02:38:56.528 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011082999999999999 2020-06-12T02:38:56.528 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.528 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.530 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:56.530 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.982920+0000", 2020-06-12T02:38:56.530 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00043570099999999998 2020-06-12T02:38:56.531 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.531 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.531 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:56.531 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.983356+0000", 2020-06-12T02:38:56.531 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00076300800000000002 2020-06-12T02:38:56.532 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.532 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.532 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:56.532 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.984119+0000", 2020-06-12T02:38:56.532 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01538405 2020-06-12T02:38:56.533 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.533 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.533 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:56.533 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.999503+0000", 2020-06-12T02:38:56.533 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9801000000000003e-05 2020-06-12T02:38:56.534 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.534 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.534 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:56.534 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.999593+0000", 2020-06-12T02:38:56.534 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.023612415000000001 2020-06-12T02:38:56.535 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.535 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:56.535 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.535 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.535 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.536 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18288 3.ds0 3:b983fd28:::smithi10729407-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:38:56.536 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.976151+0000", 2020-06-12T02:38:56.536 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 14.336187398, 2020-06-12T02:38:56.536 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025323907999999999, 2020-06-12T02:38:56.536 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:56.537 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:56.537 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:56.537 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:56.537 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:56.538 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18288 2020-06-12T02:38:56.538 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.538 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:56.539 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.539 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:56.539 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976151+0000", 2020-06-12T02:38:56.539 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:56.540 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.540 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.540 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:56.540 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976151+0000", 2020-06-12T02:38:56.540 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5615000000000002e-05 2020-06-12T02:38:56.541 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.541 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.541 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:56.541 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976207+0000", 2020-06-12T02:38:56.541 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6399999999999997e-06 2020-06-12T02:38:56.542 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.542 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.542 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:56.542 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976214+0000", 2020-06-12T02:38:56.543 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4823e-05 2020-06-12T02:38:56.543 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.543 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.543 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:56.543 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976229+0000", 2020-06-12T02:38:56.544 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1519999999999997e-06 2020-06-12T02:38:56.544 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.544 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.544 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:56.544 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976235+0000", 2020-06-12T02:38:56.545 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0077693090000000003 2020-06-12T02:38:56.545 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.545 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.545 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:56.546 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.984005+0000", 2020-06-12T02:38:56.546 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000101221 2020-06-12T02:38:56.546 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.546 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.546 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:56.546 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.984106+0000", 2020-06-12T02:38:56.547 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016277713999999999 2020-06-12T02:38:56.547 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.547 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.547 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:56.547 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.000384+0000", 2020-06-12T02:38:56.548 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7938999999999999e-05 2020-06-12T02:38:56.548 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.548 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.549 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:56.549 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.000482+0000", 2020-06-12T02:38:56.549 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00099349500000000006 2020-06-12T02:38:56.549 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.549 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.550 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:56.550 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.001475+0000", 2020-06-12T02:38:56.550 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025323907999999999 2020-06-12T02:38:56.550 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.551 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:56.551 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.551 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.551 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.552 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18307 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [append 0~507904] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:38:56.552 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.056119+0000", 2020-06-12T02:38:56.552 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 14.256219515, 2020-06-12T02:38:56.552 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025879849, 2020-06-12T02:38:56.552 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:56.553 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:56.553 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:56.553 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:56.553 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:56.553 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18307 2020-06-12T02:38:56.554 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.554 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:56.554 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.555 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:56.555 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.056119+0000", 2020-06-12T02:38:56.555 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:56.555 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.555 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.556 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:56.556 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.056119+0000", 2020-06-12T02:38:56.556 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00052484099999999996 2020-06-12T02:38:56.556 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.556 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.557 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:56.557 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.056644+0000", 2020-06-12T02:38:56.557 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7149999999999997e-06 2020-06-12T02:38:56.557 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.558 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.558 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:56.558 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.056649+0000", 2020-06-12T02:38:56.559 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0046e-05 2020-06-12T02:38:56.559 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.559 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.559 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:56.559 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.056659+0000", 2020-06-12T02:38:56.559 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.848e-06 2020-06-12T02:38:56.560 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.560 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.560 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:56.560 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.056663+0000", 2020-06-12T02:38:56.560 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024537029999999998 2020-06-12T02:38:56.561 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.561 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.561 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:56.561 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.059117+0000", 2020-06-12T02:38:56.561 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000121261 2020-06-12T02:38:56.562 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.563 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.563 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:56.563 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.059238+0000", 2020-06-12T02:38:56.563 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001916484 2020-06-12T02:38:56.563 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.564 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.564 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:56.564 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.061155+0000", 2020-06-12T02:38:56.565 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9120000000000001e-05 2020-06-12T02:38:56.566 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.566 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.566 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:56.566 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.061234+0000", 2020-06-12T02:38:56.567 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00060383599999999996 2020-06-12T02:38:56.567 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.567 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.567 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:56.568 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.061838+0000", 2020-06-12T02:38:56.568 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010259862999999999 2020-06-12T02:38:56.568 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.568 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.568 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:56.568 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.072098+0000", 2020-06-12T02:38:56.569 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0098176419999999997 2020-06-12T02:38:56.569 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.569 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.569 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:56.569 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.081915+0000", 2020-06-12T02:38:56.570 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3490000000000002e-05 2020-06-12T02:38:56.570 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.570 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.577 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:56.577 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.081999+0000", 2020-06-12T02:38:56.577 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025879849 2020-06-12T02:38:56.578 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.578 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:56.578 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.578 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.578 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.579 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18311 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:38:56.579 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.063458+0000", 2020-06-12T02:38:56.579 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 14.248880879, 2020-06-12T02:38:56.579 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.035342152000000002, 2020-06-12T02:38:56.579 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:56.579 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:56.580 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:56.580 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:56.580 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:56.580 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18311 2020-06-12T02:38:56.580 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.581 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:56.581 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.581 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:56.581 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.063458+0000", 2020-06-12T02:38:56.581 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:56.582 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.582 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.582 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:56.582 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.063458+0000", 2020-06-12T02:38:56.582 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3773000000000003e-05 2020-06-12T02:38:56.582 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.583 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.583 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:56.583 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.063501+0000", 2020-06-12T02:38:56.583 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0780000000000001e-06 2020-06-12T02:38:56.583 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.584 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.584 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:56.584 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.063508+0000", 2020-06-12T02:38:56.584 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3020000000000001e-05 2020-06-12T02:38:56.584 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.584 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.585 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:56.585 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.063521+0000", 2020-06-12T02:38:56.585 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4839999999999998e-06 2020-06-12T02:38:56.585 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.585 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.586 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:56.586 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.063525+0000", 2020-06-12T02:38:56.586 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0087769890000000007 2020-06-12T02:38:56.586 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.587 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.587 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:56.587 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.072302+0000", 2020-06-12T02:38:56.588 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3121000000000005e-05 2020-06-12T02:38:56.588 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.588 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.588 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:56.588 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.072395+0000", 2020-06-12T02:38:56.589 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010158355000000001 2020-06-12T02:38:56.589 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.589 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.589 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:56.589 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.082554+0000", 2020-06-12T02:38:56.590 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5016000000000001e-05 2020-06-12T02:38:56.590 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.591 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.591 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:56.591 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.082629+0000", 2020-06-12T02:38:56.591 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016171316000000002 2020-06-12T02:38:56.591 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.592 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.592 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:56.592 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.098800+0000", 2020-06-12T02:38:56.592 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.035342152000000002 2020-06-12T02:38:56.592 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.593 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:56.593 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.593 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.593 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.593 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18313 3.fs0 3:f83c3dea:::smithi10729407-4:head [append 0~442368] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:38:56.594 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.070249+0000", 2020-06-12T02:38:56.594 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 14.242089602, 2020-06-12T02:38:56.594 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028853489, 2020-06-12T02:38:56.594 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:56.594 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:56.595 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:56.595 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:56.595 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:56.595 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18313 2020-06-12T02:38:56.595 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.595 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:56.596 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.596 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:56.596 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.070249+0000", 2020-06-12T02:38:56.596 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:56.596 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.597 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.598 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:56.598 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.070249+0000", 2020-06-12T02:38:56.598 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00046159400000000001 2020-06-12T02:38:56.598 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.598 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.599 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:56.599 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.070711+0000", 2020-06-12T02:38:56.599 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2539999999999992e-06 2020-06-12T02:38:56.599 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.599 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.600 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:56.600 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.070719+0000", 2020-06-12T02:38:56.600 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3297e-05 2020-06-12T02:38:56.600 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.600 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.601 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:56.601 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.070732+0000", 2020-06-12T02:38:56.601 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0559999999999999e-06 2020-06-12T02:38:56.601 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.601 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.602 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:56.602 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.070739+0000", 2020-06-12T02:38:56.602 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0052673319999999996 2020-06-12T02:38:56.602 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.602 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.603 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:56.603 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.076006+0000", 2020-06-12T02:38:56.603 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6925999999999997e-05 2020-06-12T02:38:56.603 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.603 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.603 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:56.604 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.076063+0000", 2020-06-12T02:38:56.604 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0076712209999999998 2020-06-12T02:38:56.604 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.604 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.605 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:56.605 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.083735+0000", 2020-06-12T02:38:56.605 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8089000000000002e-05 2020-06-12T02:38:56.605 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.605 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.606 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:56.606 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.083833+0000", 2020-06-12T02:38:56.606 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054620600000000004 2020-06-12T02:38:56.606 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.606 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.607 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:56.607 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.084379+0000", 2020-06-12T02:38:56.607 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013940945999999999 2020-06-12T02:38:56.608 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.608 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.608 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:56.608 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.098320+0000", 2020-06-12T02:38:56.609 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00072638599999999998 2020-06-12T02:38:56.609 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.609 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.609 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:56.609 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.099046+0000", 2020-06-12T02:38:56.610 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6181999999999998e-05 2020-06-12T02:38:56.610 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.610 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.610 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:56.610 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.099102+0000", 2020-06-12T02:38:56.610 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028853489 2020-06-12T02:38:56.611 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.611 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:56.611 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.611 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.611 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.612 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18314 3.fs0 3:f83c3dea:::smithi10729407-4:head [append 0~573440] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:38:56.612 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.074005+0000", 2020-06-12T02:38:56.612 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 14.238333582999999, 2020-06-12T02:38:56.612 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031442581999999997, 2020-06-12T02:38:56.612 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:56.613 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:56.613 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:56.613 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:56.613 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:56.613 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18314 2020-06-12T02:38:56.613 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.614 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:56.614 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.614 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:56.614 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.074005+0000", 2020-06-12T02:38:56.614 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:56.615 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.615 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.615 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:56.615 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.074005+0000", 2020-06-12T02:38:56.615 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00056762000000000002 2020-06-12T02:38:56.615 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.616 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.616 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:56.616 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.074573+0000", 2020-06-12T02:38:56.616 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5400000000000003e-06 2020-06-12T02:38:56.616 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.617 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.617 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:56.617 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.074578+0000", 2020-06-12T02:38:56.617 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1090000000000004e-06 2020-06-12T02:38:56.617 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.618 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.618 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:56.619 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.074587+0000", 2020-06-12T02:38:56.619 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4039999999999997e-06 2020-06-12T02:38:56.619 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.619 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.619 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:56.620 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.074592+0000", 2020-06-12T02:38:56.620 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001538031 2020-06-12T02:38:56.620 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.620 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.620 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:56.620 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.076130+0000", 2020-06-12T02:38:56.621 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5117000000000002e-05 2020-06-12T02:38:56.621 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.621 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.621 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:56.621 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.076175+0000", 2020-06-12T02:38:56.622 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011822262 2020-06-12T02:38:56.622 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.622 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.622 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:56.622 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.087997+0000", 2020-06-12T02:38:56.622 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000112318 2020-06-12T02:38:56.623 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.623 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.623 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:56.623 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.088109+0000", 2020-06-12T02:38:56.623 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00072351100000000003 2020-06-12T02:38:56.624 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.624 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.624 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:56.624 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.088833+0000", 2020-06-12T02:38:56.624 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010307874999999999 2020-06-12T02:38:56.624 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.625 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.625 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:56.625 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.099141+0000", 2020-06-12T02:38:56.625 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.006229605 2020-06-12T02:38:56.625 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.626 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.626 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:56.626 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.105370+0000", 2020-06-12T02:38:56.626 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7189999999999998e-05 2020-06-12T02:38:56.626 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.627 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.627 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:56.627 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.105448+0000", 2020-06-12T02:38:56.627 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031442581999999997 2020-06-12T02:38:56.627 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.627 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:56.628 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.628 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.629 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.629 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18315 3.fs0 3:f83c3dea:::smithi10729407-4:head [append 0~589824] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:38:56.629 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.077724+0000", 2020-06-12T02:38:56.629 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 14.234614724, 2020-06-12T02:38:56.629 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.027939051999999999, 2020-06-12T02:38:56.630 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:56.630 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:56.630 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:56.630 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:56.630 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:56.631 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18315 2020-06-12T02:38:56.631 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.631 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:56.631 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.631 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:56.631 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.077724+0000", 2020-06-12T02:38:56.632 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:56.632 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.632 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.632 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:56.632 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.077724+0000", 2020-06-12T02:38:56.633 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00093652299999999998 2020-06-12T02:38:56.633 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.633 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.633 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:56.633 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078660+0000", 2020-06-12T02:38:56.634 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1389999999999995e-06 2020-06-12T02:38:56.634 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.634 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.634 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:56.634 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078669+0000", 2020-06-12T02:38:56.634 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3691e-05 2020-06-12T02:38:56.635 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.635 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.635 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:56.635 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078682+0000", 2020-06-12T02:38:56.635 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0100000000000001e-06 2020-06-12T02:38:56.636 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.636 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.636 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:56.636 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078688+0000", 2020-06-12T02:38:56.636 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1209999999999999e-05 2020-06-12T02:38:56.636 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.637 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.637 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:56.637 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078739+0000", 2020-06-12T02:38:56.637 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000102735 2020-06-12T02:38:56.637 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.638 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.638 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:56.639 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078842+0000", 2020-06-12T02:38:56.639 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01321815 2020-06-12T02:38:56.640 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.640 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.640 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:56.640 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.092060+0000", 2020-06-12T02:38:56.640 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00038608499999999999 2020-06-12T02:38:56.641 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.641 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.641 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:56.641 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.092446+0000", 2020-06-12T02:38:56.641 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00056703299999999999 2020-06-12T02:38:56.642 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.642 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.642 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:56.642 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.093013+0000", 2020-06-12T02:38:56.642 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0061289530000000004 2020-06-12T02:38:56.643 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.643 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.643 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:56.643 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.099142+0000", 2020-06-12T02:38:56.643 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0064547700000000003 2020-06-12T02:38:56.644 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.644 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.644 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:56.644 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.105597+0000", 2020-06-12T02:38:56.644 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5753000000000006e-05 2020-06-12T02:38:56.644 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.645 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.645 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:56.645 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.105663+0000", 2020-06-12T02:38:56.645 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.027939051999999999 2020-06-12T02:38:56.645 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.646 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:56.646 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.646 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.646 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.646 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18316 3.fs0 3:f83c3dea:::smithi10729407-4:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:38:56.647 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.078752+0000", 2020-06-12T02:38:56.647 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 14.233586952, 2020-06-12T02:38:56.647 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.029108709999999999, 2020-06-12T02:38:56.647 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:56.647 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:56.647 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:56.648 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:56.648 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:56.648 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18316 2020-06-12T02:38:56.648 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.648 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:56.649 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.649 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:56.649 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078752+0000", 2020-06-12T02:38:56.649 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:56.649 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.650 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.650 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:56.650 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078752+0000", 2020-06-12T02:38:56.650 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8967000000000004e-05 2020-06-12T02:38:56.650 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.651 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.651 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:56.651 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078841+0000", 2020-06-12T02:38:56.651 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3309999999999998e-06 2020-06-12T02:38:56.651 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.651 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.652 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:56.652 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078848+0000", 2020-06-12T02:38:56.652 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.291e-05 2020-06-12T02:38:56.652 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.652 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.653 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:56.653 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078861+0000", 2020-06-12T02:38:56.653 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2800000000000003e-06 2020-06-12T02:38:56.653 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.653 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.653 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:56.654 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078866+0000", 2020-06-12T02:38:56.654 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8543e-05 2020-06-12T02:38:56.654 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.654 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.654 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:56.655 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078905+0000", 2020-06-12T02:38:56.655 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9198999999999998e-05 2020-06-12T02:38:56.655 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.655 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.655 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:56.656 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078994+0000", 2020-06-12T02:38:56.656 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01781437 2020-06-12T02:38:56.656 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.656 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.656 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:56.657 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.096808+0000", 2020-06-12T02:38:56.657 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000115975 2020-06-12T02:38:56.657 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.657 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.657 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:56.657 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.096924+0000", 2020-06-12T02:38:56.658 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00049986300000000002 2020-06-12T02:38:56.658 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.658 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.658 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:38:56.658 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.097424+0000", 2020-06-12T02:38:56.659 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0017191089999999999 2020-06-12T02:38:56.659 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.659 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.659 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:38:56.659 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.099143+0000", 2020-06-12T02:38:56.659 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0086177909999999996 2020-06-12T02:38:56.660 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.660 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.660 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:38:56.660 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.107761+0000", 2020-06-12T02:38:56.660 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9371999999999997e-05 2020-06-12T02:38:56.661 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.661 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.661 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:56.661 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.107860+0000", 2020-06-12T02:38:56.661 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.029108709999999999 2020-06-12T02:38:56.661 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.662 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:56.662 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.662 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.662 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.662 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18317 3.fs0 3:f83c3dea:::smithi10729407-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:38:56.663 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.078920+0000", 2020-06-12T02:38:56.663 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 14.233418982, 2020-06-12T02:38:56.663 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031356964000000001, 2020-06-12T02:38:56.663 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:38:56.663 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:38:56.663 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:38:56.664 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:38:56.664 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:56.664 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18317 2020-06-12T02:38:56.664 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.664 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:38:56.665 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.665 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:38:56.665 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078920+0000", 2020-06-12T02:38:56.665 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:38:56.665 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.666 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.666 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:38:56.666 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078920+0000", 2020-06-12T02:38:56.666 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6454000000000001e-05 2020-06-12T02:38:56.666 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.666 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.667 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:38:56.667 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078946+0000", 2020-06-12T02:38:56.667 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8160000000000004e-06 2020-06-12T02:38:56.667 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.667 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.668 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:38:56.668 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078950+0000", 2020-06-12T02:38:56.668 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6090000000000003e-06 2020-06-12T02:38:56.668 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.668 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.668 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:38:56.669 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078957+0000", 2020-06-12T02:38:56.669 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8380000000000002e-06 2020-06-12T02:38:56.669 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.669 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.669 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:38:56.670 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078960+0000", 2020-06-12T02:38:56.670 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8400000000000003e-05 2020-06-12T02:38:56.670 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.670 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.670 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:56.670 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.079019+0000", 2020-06-12T02:38:56.671 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6750999999999998e-05 2020-06-12T02:38:56.671 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.671 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.671 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:56.671 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.079085+0000", 2020-06-12T02:38:56.672 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019272036999999999 2020-06-12T02:38:56.672 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.672 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.672 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:56.672 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.098357+0000", 2020-06-12T02:38:56.673 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4322999999999998e-05 2020-06-12T02:38:56.673 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.673 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.673 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:56.673 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.098452+0000", 2020-06-12T02:38:56.673 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010578025 2020-06-12T02:38:56.674 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.674 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.674 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:38:56.674 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.109030+0000", 2020-06-12T02:38:56.674 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010395099999999999 2020-06-12T02:38:56.675 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.675 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.675 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:38:56.675 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.109134+0000", 2020-06-12T02:38:56.675 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00114276 2020-06-12T02:38:56.675 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:38:56.676 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:38:56.676 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:38:56.676 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.110277+0000", 2020-06-12T02:38:56.676 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031356964000000001 2020-06-12T02:38:56.676 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.677 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:56.677 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.677 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:38:56.677 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:38:56.677 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:38:56.678 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd unset nodeep-scrub 2020-06-12T02:38:56.681 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:38:56.682 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":128,"stamp":"2020-06-12T02:38:55.611659+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6864896,"num_objects":9,"num_object_clones":0,"num_object_copies":25,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":5274,"num_read_kb":2866,"num_write":11792,"num_write_kb":3359806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":13549568,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25562,"ondisk_log_size":25562,"up":96,"acting":96,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":102,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8401940,"kb_used_data":13268,"kb_used_omap":25,"kb_used_meta":8388582,"kb_avail":746572780,"statfs":{"total":773094113280,"available":764490526720,"internally_reserved":8589934592,"allocated":13586432,"data_stored":10161222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26157,"internal_metadata":8589908435},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-11890688,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1337,"num_read_kb":728,"num_write":2985,"num_write_kb":847060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":2650112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000822"},"pg_stats":[{"pgid":"3.19","version":"63'1666","reported_seq":"17","reported_epoch":"71","state":"active+undersized+remapped","last_fresh":"2020-06-12T02:38:51.398957+0000","last_change":"2020-06-12T02:38:51.398745+0000","last_active":"2020-06-12T02:38:51.398957+0000","last_peered":"2020-06-12T02:38:51.398957+0000","last_clean":"2020-06-12T02:38:44.390059+0000","last_became_active":"2020-06-12T02:38:51.398745+0000","last_became_peered":"2020-06-12T02:38:51.398745+0000","last_unstale":"2020-06-12T02:38:51.398957+0000","last_undegraded":"2020-06-12T02:38:51.398957+0000","last_fullsized":"2020-06-12T02:38:51.390307+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":1666,"ondisk_log_size":1666,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":282,"num_read_kb":155,"num_write":620,"num_write_kb":172356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[2147483647,3,4],"avail_no_missing":["3(1)","1(0)","2(1)","4(0)","4(2)","6(0)","7(2)"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"63'843","reported_seq":"1116","reported_epoch":"71","state":"active+undersized+remapped","last_fresh":"2020-06-12T02:38:51.419916+0000","last_change":"2020-06-12T02:38:51.419752+0000","last_active":"2020-06-12T02:38:51.419916+0000","last_peered":"2020-06-12T02:38:51.419916+0000","last_clean":"2020-06-12T02:38:45.505919+0000","last_became_active":"2020-06-12T02:38:51.419752+0000","last_became_peered":"2020-06-12T02:38:51.419752+0000","last_unstale":"2020-06-12T02:38:51.419916+0000","last_undegraded":"2020-06-12T02:38:51.419916+0000","last_fullsized":"2020-06-12T02:38:51.394729+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":843,"ondisk_log_size":843,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":137,"num_read_kb":75,"num_write":323,"num_write_kb":94224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1396736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,2147483647],"avail_no_missing":["3(0)","0(1)","1(2)","4(1)","7(2)"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"71'1724","reported_seq":"52","reported_epoch":"71","state":"active+clean","last_fresh":"2020-06-12T02:38:51.503665+0000","last_change":"2020-06-12T02:38:50.381078+0000","last_active":"2020-06-12T02:38:51.503665+0000","last_peered":"2020-06-12T02:38:51.503665+0000","last_clean":"2020-06-12T02:38:51.503665+0000","last_became_active":"2020-06-12T02:38:50.380314+0000","last_became_peered":"2020-06-12T02:38:50.380314+0000","last_unstale":"2020-06-12T02:38:51.503665+0000","last_undegraded":"2020-06-12T02:38:51.503665+0000","last_fullsized":"2020-06-12T02:38:51.503665+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":1724,"ondisk_log_size":1724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":946176,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":298,"num_read_kb":166,"num_write":679,"num_write_kb":196988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1396736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"25","reported_epoch":"72","state":"active+clean","last_fresh":"2020-06-12T02:38:52.393478+0000","last_change":"2020-06-12T02:38:50.387232+0000","last_active":"2020-06-12T02:38:52.393478+0000","last_peered":"2020-06-12T02:38:52.393478+0000","last_clean":"2020-06-12T02:38:52.393478+0000","last_became_active":"2020-06-12T02:38:50.386718+0000","last_became_peered":"2020-06-12T02:38:50.386718+0000","last_unstale":"2020-06-12T02:38:52.393478+0000","last_undegraded":"2020-06-12T02:38:52.393478+0000","last_fullsized":"2020-06-12T02:38:52.393478+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"68'1595","reported_seq":"1970","reported_epoch":"72","state":"active+clean+wait","last_fresh":"2020-06-12T02:38:52.396181+0000","last_change":"2020-06-12T02:38:50.388146+0000","last_active":"2020-06-12T02:38:52.396181+0000","last_peered":"2020-06-12T02:38:52.396181+0000","last_clean":"2020-06-12T02:38:52.396181+0000","last_became_active":"2020-06-12T02:38:50.385649+0000","last_became_peered":"2020-06-12T02:38:50.385649+0000","last_unstale":"2020-06-12T02:38:52.396181+0000","last_undegraded":"2020-06-12T02:38:52.396181+0000","last_fullsized":"2020-06-12T02:38:52.396181+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":1595,"ondisk_log_size":1595,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":483328,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":263,"num_read_kb":143,"num_write":592,"num_write_kb":171168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"14","reported_epoch":"70","state":"active+clean","last_fresh":"2020-06-12T02:38:50.373574+0000","last_change":"2020-06-12T02:38:47.508860+0000","last_active":"2020-06-12T02:38:50.373574+0000","last_peered":"2020-06-12T02:38:50.373574+0000","last_clean":"2020-06-12T02:38:50.373574+0000","last_became_active":"2020-06-12T02:38:47.508336+0000","last_became_peered":"2020-06-12T02:38:47.508336+0000","last_unstale":"2020-06-12T02:38:50.373574+0000","last_undegraded":"2020-06-12T02:38:50.373574+0000","last_fullsized":"2020-06-12T02:38:50.373574+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"68'1640","reported_seq":"39","reported_epoch":"72","state":"active+clean+wait","last_fresh":"2020-06-12T02:38:52.393265+0000","last_change":"2020-06-12T02:38:50.386250+0000","last_active":"2020-06-12T02:38:52.393265+0000","last_peered":"2020-06-12T02:38:52.393265+0000","last_clean":"2020-06-12T02:38:52.393265+0000","last_became_active":"2020-06-12T02:38:50.385130+0000","last_became_peered":"2020-06-12T02:38:50.385130+0000","last_unstale":"2020-06-12T02:38:52.393265+0000","last_undegraded":"2020-06-12T02:38:52.393265+0000","last_fullsized":"2020-06-12T02:38:52.393265+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":1640,"ondisk_log_size":1640,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":262144,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":276,"num_read_kb":147,"num_write":614,"num_write_kb":173540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"71'697","reported_seq":"48","reported_epoch":"71","state":"active+clean","last_fresh":"2020-06-12T02:38:51.490105+0000","last_change":"2020-06-12T02:38:50.384388+0000","last_active":"2020-06-12T02:38:51.490105+0000","last_peered":"2020-06-12T02:38:51.490105+0000","last_clean":"2020-06-12T02:38:51.490105+0000","last_became_active":"2020-06-12T02:38:50.383580+0000","last_became_peered":"2020-06-12T02:38:50.383580+0000","last_unstale":"2020-06-12T02:38:51.490105+0000","last_undegraded":"2020-06-12T02:38:51.490105+0000","last_fullsized":"2020-06-12T02:38:51.490105+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'650","last_scrub_stamp":"2020-06-12T02:38:43.307170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:38:43.307170+0000","log_size":697,"ondisk_log_size":697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":118,"num_read_kb":65,"num_write":260,"num_write_kb":71960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"16","reported_epoch":"72","state":"active+clean","last_fresh":"2020-06-12T02:38:52.393043+0000","last_change":"2020-06-12T02:38:47.506911+0000","last_active":"2020-06-12T02:38:52.393043+0000","last_peered":"2020-06-12T02:38:52.393043+0000","last_clean":"2020-06-12T02:38:52.393043+0000","last_became_active":"2020-06-12T02:38:47.506358+0000","last_became_peered":"2020-06-12T02:38:47.506358+0000","last_unstale":"2020-06-12T02:38:52.393043+0000","last_undegraded":"2020-06-12T02:38:52.393043+0000","last_fullsized":"2020-06-12T02:38:52.393043+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"68'667","reported_seq":"46","reported_epoch":"70","state":"active+clean+wait","last_fresh":"2020-06-12T02:38:50.388200+0000","last_change":"2020-06-12T02:38:50.383035+0000","last_active":"2020-06-12T02:38:50.388200+0000","last_peered":"2020-06-12T02:38:50.388200+0000","last_clean":"2020-06-12T02:38:50.388200+0000","last_became_active":"2020-06-12T02:38:50.382197+0000","last_became_peered":"2020-06-12T02:38:50.382197+0000","last_unstale":"2020-06-12T02:38:50.388200+0000","last_undegraded":"2020-06-12T02:38:50.388200+0000","last_fullsized":"2020-06-12T02:38:50.388200+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":667,"ondisk_log_size":667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":368640,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":119,"num_read_kb":66,"num_write":270,"num_write_kb":77464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"71'1767","reported_seq":"2160","reported_epoch":"72","state":"active+clean","last_fresh":"2020-06-12T02:38:52.393887+0000","last_change":"2020-06-12T02:38:46.393841+0000","last_active":"2020-06-12T02:38:52.393887+0000","last_peered":"2020-06-12T02:38:52.393887+0000","last_clean":"2020-06-12T02:38:52.393887+0000","last_became_active":"2020-06-12T02:37:57.566633+0000","last_became_peered":"2020-06-12T02:37:57.566633+0000","last_unstale":"2020-06-12T02:38:52.393887+0000","last_undegraded":"2020-06-12T02:38:52.393887+0000","last_fullsized":"2020-06-12T02:38:52.393887+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1729","last_scrub_stamp":"2020-06-12T02:38:46.393774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:38:46.393774+0000","log_size":1767,"ondisk_log_size":1767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":434176,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":598,"num_read_kb":316,"num_write":1304,"num_write_kb":368976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1531904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"71'912","reported_seq":"1150","reported_epoch":"72","state":"active+clean","last_fresh":"2020-06-12T02:38:52.393598+0000","last_change":"2020-06-12T02:38:11.572385+0000","last_active":"2020-06-12T02:38:52.393598+0000","last_peered":"2020-06-12T02:38:52.393598+0000","last_clean":"2020-06-12T02:38:52.393598+0000","last_became_active":"2020-06-12T02:37:57.563869+0000","last_became_peered":"2020-06-12T02:37:57.563869+0000","last_unstale":"2020-06-12T02:38:52.393598+0000","last_undegraded":"2020-06-12T02:38:52.393598+0000","last_fullsized":"2020-06-12T02:38:52.393598+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":912,"ondisk_log_size":912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":300,"num_read_kb":162,"num_write":688,"num_write_kb":195192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1466368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"71'1699","reported_seq":"2096","reported_epoch":"72","state":"active+clean","last_fresh":"2020-06-12T02:38:52.396596+0000","last_change":"2020-06-12T02:38:11.587290+0000","last_active":"2020-06-12T02:38:52.396596+0000","last_peered":"2020-06-12T02:38:52.396596+0000","last_clean":"2020-06-12T02:38:52.396596+0000","last_became_active":"2020-06-12T02:37:57.564464+0000","last_became_peered":"2020-06-12T02:37:57.564464+0000","last_unstale":"2020-06-12T02:38:52.396596+0000","last_undegraded":"2020-06-12T02:38:52.396596+0000","last_fullsized":"2020-06-12T02:38:52.396596+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":1699,"ondisk_log_size":1699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":901120,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":575,"num_read_kb":311,"num_write":1298,"num_write_kb":369408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3710976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"55","reported_epoch":"70","state":"active+clean","last_fresh":"2020-06-12T02:38:50.372263+0000","last_change":"2020-06-12T02:38:46.288343+0000","last_active":"2020-06-12T02:38:50.372263+0000","last_peered":"2020-06-12T02:38:50.372263+0000","last_clean":"2020-06-12T02:38:50.372263+0000","last_became_active":"2020-06-12T02:37:42.384363+0000","last_became_peered":"2020-06-12T02:37:42.384363+0000","last_unstale":"2020-06-12T02:38:50.372263+0000","last_undegraded":"2020-06-12T02:38:50.372263+0000","last_fullsized":"2020-06-12T02:38:50.372263+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:38:46.288277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:38:46.288277+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"71'1661","reported_seq":"2021","reported_epoch":"71","state":"active+clean","last_fresh":"2020-06-12T02:38:51.521402+0000","last_change":"2020-06-12T02:37:27.902643+0000","last_active":"2020-06-12T02:38:51.521402+0000","last_peered":"2020-06-12T02:38:51.521402+0000","last_clean":"2020-06-12T02:38:51.521402+0000","last_became_active":"2020-06-12T02:37:27.902204+0000","last_became_peered":"2020-06-12T02:37:27.902204+0000","last_unstale":"2020-06-12T02:38:51.521402+0000","last_undegraded":"2020-06-12T02:38:51.521402+0000","last_fullsized":"2020-06-12T02:38:51.521402+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":1661,"ondisk_log_size":1661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":562,"num_read_kb":303,"num_write":1229,"num_write_kb":343448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"71'1746","reported_seq":"2170","reported_epoch":"71","state":"active+undersized+remapped","last_fresh":"2020-06-12T02:38:51.529088+0000","last_change":"2020-06-12T02:38:51.438922+0000","last_active":"2020-06-12T02:38:51.529088+0000","last_peered":"2020-06-12T02:38:51.529088+0000","last_clean":"2020-06-12T02:38:49.502897+0000","last_became_active":"2020-06-12T02:38:51.431610+0000","last_became_peered":"2020-06-12T02:38:51.431610+0000","last_unstale":"2020-06-12T02:38:51.529088+0000","last_undegraded":"2020-06-12T02:38:51.529088+0000","last_fullsized":"2020-06-12T02:38:51.389731+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":1746,"ondisk_log_size":1746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1802240,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":611,"num_read_kb":331,"num_write":1333,"num_write_kb":383192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,2147483647],"avail_no_missing":["6(0)","1(2)","4(1)","5(2)"],"object_location_counts":[{"shards":"4(1),6(0)","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"71'1730","reported_seq":"2139","reported_epoch":"71","state":"active+clean","last_fresh":"2020-06-12T02:38:51.457789+0000","last_change":"2020-06-12T02:38:45.498622+0000","last_active":"2020-06-12T02:38:51.457789+0000","last_peered":"2020-06-12T02:38:51.457789+0000","last_clean":"2020-06-12T02:38:51.457789+0000","last_became_active":"2020-06-12T02:38:45.497443+0000","last_became_peered":"2020-06-12T02:38:45.497443+0000","last_unstale":"2020-06-12T02:38:51.457789+0000","last_undegraded":"2020-06-12T02:38:51.457789+0000","last_fullsized":"2020-06-12T02:38:51.457789+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":1730,"ondisk_log_size":1730,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":301,"num_read_kb":166,"num_write":666,"num_write_kb":185356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"71'865","reported_seq":"1139","reported_epoch":"71","state":"active+undersized+remapped","last_fresh":"2020-06-12T02:38:51.432132+0000","last_change":"2020-06-12T02:38:51.432132+0000","last_active":"2020-06-12T02:38:51.432132+0000","last_peered":"2020-06-12T02:38:51.432132+0000","last_clean":"2020-06-12T02:38:49.480669+0000","last_became_active":"2020-06-12T02:38:51.420632+0000","last_became_peered":"2020-06-12T02:38:51.420632+0000","last_unstale":"2020-06-12T02:38:51.432132+0000","last_undegraded":"2020-06-12T02:38:51.432132+0000","last_fullsized":"2020-06-12T02:38:51.390271+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":865,"ondisk_log_size":865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":143,"num_read_kb":77,"num_write":335,"num_write_kb":97568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1863680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,2147483647,7],"avail_no_missing":["3(0)","1(1)","4(1)","7(2)"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"10","reported_epoch":"70","state":"active+clean","last_fresh":"2020-06-12T02:38:50.380478+0000","last_change":"2020-06-12T02:38:50.377172+0000","last_active":"2020-06-12T02:38:50.380478+0000","last_peered":"2020-06-12T02:38:50.380478+0000","last_clean":"2020-06-12T02:38:50.380478+0000","last_became_active":"2020-06-12T02:38:50.376548+0000","last_became_peered":"2020-06-12T02:38:50.376548+0000","last_unstale":"2020-06-12T02:38:50.380478+0000","last_undegraded":"2020-06-12T02:38:50.380478+0000","last_fullsized":"2020-06-12T02:38:50.380478+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:38:34.510641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:38:34.510641+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"71'1617","reported_seq":"1977","reported_epoch":"71","state":"active+clean","last_fresh":"2020-06-12T02:38:51.452943+0000","last_change":"2020-06-12T02:38:45.496321+0000","last_active":"2020-06-12T02:38:51.452943+0000","last_peered":"2020-06-12T02:38:51.452943+0000","last_clean":"2020-06-12T02:38:51.452943+0000","last_became_active":"2020-06-12T02:38:45.495538+0000","last_became_peered":"2020-06-12T02:38:45.495538+0000","last_unstale":"2020-06-12T02:38:51.452943+0000","last_undegraded":"2020-06-12T02:38:51.452943+0000","last_fullsized":"2020-06-12T02:38:51.452943+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":1617,"ondisk_log_size":1617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":737280,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":273,"num_read_kb":151,"num_write":620,"num_write_kb":181688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"10","reported_epoch":"70","state":"active+clean","last_fresh":"2020-06-12T02:38:50.385378+0000","last_change":"2020-06-12T02:38:50.381973+0000","last_active":"2020-06-12T02:38:50.385378+0000","last_peered":"2020-06-12T02:38:50.385378+0000","last_clean":"2020-06-12T02:38:50.385378+0000","last_became_active":"2020-06-12T02:38:50.381564+0000","last_became_peered":"2020-06-12T02:38:50.381564+0000","last_unstale":"2020-06-12T02:38:50.385378+0000","last_undegraded":"2020-06-12T02:38:50.385378+0000","last_fullsized":"2020-06-12T02:38:50.385378+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:38:44.131122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:38:44.131122+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"71'1665","reported_seq":"2068","reported_epoch":"71","state":"active+undersized+remapped","last_fresh":"2020-06-12T02:38:51.437336+0000","last_change":"2020-06-12T02:38:51.423127+0000","last_active":"2020-06-12T02:38:51.437336+0000","last_peered":"2020-06-12T02:38:51.437336+0000","last_clean":"2020-06-12T02:38:49.494980+0000","last_became_active":"2020-06-12T02:38:51.417442+0000","last_became_peered":"2020-06-12T02:38:51.417442+0000","last_unstale":"2020-06-12T02:38:51.437336+0000","last_undegraded":"2020-06-12T02:38:51.437336+0000","last_fullsized":"2020-06-12T02:38:51.389707+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1653","last_scrub_stamp":"2020-06-12T02:38:46.542505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:38:46.542505+0000","log_size":1665,"ondisk_log_size":1665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,2147483647,0],"avail_no_missing":["4(0)","0(2)","1(1)","5(1)"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"64","reported_epoch":"70","state":"active+clean","last_fresh":"2020-06-12T02:38:50.374517+0000","last_change":"2020-06-12T02:38:21.039695+0000","last_active":"2020-06-12T02:38:50.374517+0000","last_peered":"2020-06-12T02:38:50.374517+0000","last_clean":"2020-06-12T02:38:50.374517+0000","last_became_active":"2020-06-12T02:37:16.764958+0000","last_became_peered":"2020-06-12T02:37:16.764958+0000","last_unstale":"2020-06-12T02:38:50.374517+0000","last_undegraded":"2020-06-12T02:38:50.374517+0000","last_fullsized":"2020-06-12T02:38:50.374517+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:38:21.039625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:38:21.039625+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"63'651","reported_seq":"856","reported_epoch":"71","state":"active+undersized+remapped","last_fresh":"2020-06-12T02:38:51.396177+0000","last_change":"2020-06-12T02:38:51.395897+0000","last_active":"2020-06-12T02:38:51.396177+0000","last_peered":"2020-06-12T02:38:51.396177+0000","last_clean":"2020-06-12T02:38:44.451906+0000","last_became_active":"2020-06-12T02:38:51.395897+0000","last_became_peered":"2020-06-12T02:38:51.395897+0000","last_unstale":"2020-06-12T02:38:51.396177+0000","last_undegraded":"2020-06-12T02:38:51.396177+0000","last_fullsized":"2020-06-12T02:38:51.389031+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":651,"ondisk_log_size":651,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":929792,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":112,"num_read_kb":63,"num_write":259,"num_write_kb":75432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[2147483647,2,6],"avail_no_missing":["2(1)","0(0)","1(0)","6(2)"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"54","reported_epoch":"70","state":"active+clean","last_fresh":"2020-06-12T02:38:50.376233+0000","last_change":"2020-06-12T02:38:22.079524+0000","last_active":"2020-06-12T02:38:50.376233+0000","last_peered":"2020-06-12T02:38:50.376233+0000","last_clean":"2020-06-12T02:38:50.376233+0000","last_became_active":"2020-06-12T02:37:42.375902+0000","last_became_peered":"2020-06-12T02:37:42.375902+0000","last_unstale":"2020-06-12T02:38:50.376233+0000","last_undegraded":"2020-06-12T02:38:50.376233+0000","last_fullsized":"2020-06-12T02:38:50.376233+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:38:22.079458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:38:22.079458+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"15'2","reported_seq":"64","reported_epoch":"72","state":"active+clean","last_fresh":"2020-06-12T02:38:52.394021+0000","last_change":"2020-06-12T02:38:24.369083+0000","last_active":"2020-06-12T02:38:52.394021+0000","last_peered":"2020-06-12T02:38:52.394021+0000","last_clean":"2020-06-12T02:38:52.394021+0000","last_became_active":"2020-06-12T02:37:54.558125+0000","last_became_peered":"2020-06-12T02:37:54.558125+0000","last_unstale":"2020-06-12T02:38:52.394021+0000","last_undegraded":"2020-06-12T02:38:52.394021+0000","last_fullsized":"2020-06-12T02:38:52.394021+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":3,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'2","last_scrub_stamp":"2020-06-12T02:38:24.368994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:04.419204+0000","last_clean_scrub_stamp":"2020-06-12T02:38:24.368994+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"63'673","reported_seq":"870","reported_epoch":"70","state":"active+clean","last_fresh":"2020-06-12T02:38:50.376701+0000","last_change":"2020-06-12T02:38:48.229858+0000","last_active":"2020-06-12T02:38:50.376701+0000","last_peered":"2020-06-12T02:38:50.376701+0000","last_clean":"2020-06-12T02:38:50.376701+0000","last_became_active":"2020-06-12T02:38:45.495198+0000","last_became_peered":"2020-06-12T02:38:45.495198+0000","last_unstale":"2020-06-12T02:38:50.376701+0000","last_undegraded":"2020-06-12T02:38:50.376701+0000","last_fullsized":"2020-06-12T02:38:50.376701+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'673","last_scrub_stamp":"2020-06-12T02:38:48.229769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:38:48.229769+0000","log_size":673,"ondisk_log_size":673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"77","reported_epoch":"72","state":"active+clean","last_fresh":"2020-06-12T02:38:52.393965+0000","last_change":"2020-06-12T02:38:50.386452+0000","last_active":"2020-06-12T02:38:52.393965+0000","last_peered":"2020-06-12T02:38:52.393965+0000","last_clean":"2020-06-12T02:38:52.393965+0000","last_became_active":"2020-06-12T02:38:50.386054+0000","last_became_peered":"2020-06-12T02:38:50.386054+0000","last_unstale":"2020-06-12T02:38:52.393965+0000","last_undegraded":"2020-06-12T02:38:52.393965+0000","last_fullsized":"2020-06-12T02:38:52.393965+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:38:34.394578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:38:34.394578+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"66","reported_epoch":"70","state":"active+clean","last_fresh":"2020-06-12T02:38:50.371190+0000","last_change":"2020-06-12T02:38:45.505097+0000","last_active":"2020-06-12T02:38:50.371190+0000","last_peered":"2020-06-12T02:38:50.371190+0000","last_clean":"2020-06-12T02:38:50.371190+0000","last_became_active":"2020-06-12T02:38:45.499654+0000","last_became_peered":"2020-06-12T02:38:45.499654+0000","last_unstale":"2020-06-12T02:38:50.371190+0000","last_undegraded":"2020-06-12T02:38:50.371190+0000","last_fullsized":"2020-06-12T02:38:50.371190+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"58","reported_epoch":"72","state":"active+clean","last_fresh":"2020-06-12T02:38:52.396482+0000","last_change":"2020-06-12T02:38:05.892563+0000","last_active":"2020-06-12T02:38:52.396482+0000","last_peered":"2020-06-12T02:38:52.396482+0000","last_clean":"2020-06-12T02:38:52.396482+0000","last_became_active":"2020-06-12T02:37:54.557420+0000","last_became_peered":"2020-06-12T02:37:54.557420+0000","last_unstale":"2020-06-12T02:38:52.396482+0000","last_undegraded":"2020-06-12T02:38:52.396482+0000","last_fullsized":"2020-06-12T02:38:52.396482+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:37:15.738781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"11","reported_epoch":"70","state":"active+clean","last_fresh":"2020-06-12T02:38:50.385790+0000","last_change":"2020-06-12T02:38:50.380210+0000","last_active":"2020-06-12T02:38:50.385790+0000","last_peered":"2020-06-12T02:38:50.385790+0000","last_clean":"2020-06-12T02:38:50.385790+0000","last_became_active":"2020-06-12T02:38:50.379633+0000","last_became_peered":"2020-06-12T02:38:50.379633+0000","last_unstale":"2020-06-12T02:38:50.385790+0000","last_undegraded":"2020-06-12T02:38:50.385790+0000","last_fullsized":"2020-06-12T02:38:50.385790+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:38:46.140457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:38:46.140457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"71'1742","reported_seq":"2157","reported_epoch":"71","state":"active+clean","last_fresh":"2020-06-12T02:38:51.542920+0000","last_change":"2020-06-12T02:38:45.497458+0000","last_active":"2020-06-12T02:38:51.542920+0000","last_peered":"2020-06-12T02:38:51.542920+0000","last_clean":"2020-06-12T02:38:51.542920+0000","last_became_active":"2020-06-12T02:38:45.496715+0000","last_became_peered":"2020-06-12T02:38:45.496715+0000","last_unstale":"2020-06-12T02:38:51.542920+0000","last_undegraded":"2020-06-12T02:38:51.542920+0000","last_fullsized":"2020-06-12T02:38:51.542920+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:37:26.883460+0000","log_size":1742,"ondisk_log_size":1742,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":306,"num_read_kb":169,"num_write":699,"num_write_kb":201836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1396736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"10","reported_epoch":"70","state":"active+clean","last_fresh":"2020-06-12T02:38:50.380588+0000","last_change":"2020-06-12T02:38:50.376838+0000","last_active":"2020-06-12T02:38:50.380588+0000","last_peered":"2020-06-12T02:38:50.380588+0000","last_clean":"2020-06-12T02:38:50.380588+0000","last_became_active":"2020-06-12T02:38:50.375858+0000","last_became_peered":"2020-06-12T02:38:50.375858+0000","last_unstale":"2020-06-12T02:38:50.380588+0000","last_undegraded":"2020-06-12T02:38:50.380588+0000","last_fullsized":"2020-06-12T02:38:50.380588+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:38:45.158691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:38:45.158691+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"76","reported_epoch":"70","state":"active+clean","last_fresh":"2020-06-12T02:38:50.369520+0000","last_change":"2020-06-12T02:38:22.034355+0000","last_active":"2020-06-12T02:38:50.369520+0000","last_peered":"2020-06-12T02:38:50.369520+0000","last_clean":"2020-06-12T02:38:50.369520+0000","last_became_active":"2020-06-12T02:37:54.554986+0000","last_became_peered":"2020-06-12T02:37:54.554986+0000","last_unstale":"2020-06-12T02:38:50.369520+0000","last_undegraded":"2020-06-12T02:38:50.369520+0000","last_fullsized":"2020-06-12T02:38:50.369520+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:38:22.034263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:15.738781+0000","last_clean_scrub_stamp":"2020-06-12T02:38:22.034263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"87","reported_epoch":"70","state":"active+clean","last_fresh":"2020-06-12T02:38:50.390972+0000","last_change":"2020-06-12T02:38:50.386477+0000","last_active":"2020-06-12T02:38:50.390972+0000","last_peered":"2020-06-12T02:38:50.390972+0000","last_clean":"2020-06-12T02:38:50.390972+0000","last_became_active":"2020-06-12T02:38:50.385912+0000","last_became_peered":"2020-06-12T02:38:50.385912+0000","last_unstale":"2020-06-12T02:38:50.390972+0000","last_undegraded":"2020-06-12T02:38:50.390972+0000","last_fullsized":"2020-06-12T02:38:50.390972+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-12T02:38:47.662610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-12T02:37:26.883460+0000","last_clean_scrub_stamp":"2020-06-12T02:38:47.662610+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":6864896,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5274,"num_read_kb":2866,"num_write":11790,"num_write_kb":3359796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":13549568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6610944,"data_stored":6610944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25560,"ondisk_log_size":25560,"up":78,"acting":78,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26157,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":34,"seq":146028888077,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050424,"kb_used_data":1840,"kb_used_omap":25,"kb_used_meta":1048550,"kb_avail":93321416,"statfs":{"total":96636764160,"available":95561129984,"internally_reserved":1073741824,"allocated":1884160,"data_stored":1446738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26157,"internal_metadata":1073715667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.024}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92800000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.145}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.339}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2270000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574871,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049928,"kb_used_data":1344,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321912,"statfs":{"total":96636764160,"available":95561637888,"internally_reserved":1073741824,"allocated":1376256,"data_stored":950578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.655,"5min":1.655,"15min":1.655},"last":1.0109999999999999},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.879}]},{"osd":1,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.335,"5min":1.335,"15min":1.335},"last":0.67900000000000005},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.923,"5min":1.923,"15min":1.923},"last":1.107}]},{"osd":2,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.01,"15min":1.01},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.6699999999999999,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.653,"5min":1.653,"15min":1.653},"last":0.90200000000000002}]},{"osd":3,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.145,"15min":1.145},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.5649999999999999,"5min":1.5649999999999999,"15min":1.5649999999999999},"last":1.228},{"interface":"front","average":{"1min":1.143,"5min":1.143,"15min":1.143},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.556,"5min":1.556,"15min":1.556},"last":1.2509999999999999}]},{"osd":4,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.246,"5min":1.246,"15min":1.246},"last":1.3600000000000001},{"interface":"front","average":{"1min":1.071,"5min":1.071,"15min":1.071},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.673,"5min":1.673,"15min":1.673},"last":1.387}]},{"osd":5,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.8120000000000001,"5min":1.8120000000000001,"15min":1.8120000000000001},"last":0.998},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.677,"5min":1.677,"15min":1.677},"last":1.645}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5189999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049384,"kb_used_data":800,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322456,"statfs":{"total":96636764160,"available":95562194944,"internally_reserved":1073741824,"allocated":819200,"data_stored":393522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 12 02:38:14 2020","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.548,"5min":1.548,"15min":1.548},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":2.149,"5min":2.149,"15min":2.149},"last":0.88600000000000001}]},{"osd":2,"last update":"Fri Jun 12 02:38:14 2020","interfaces":[{"interface":"back","average":{"1min":1,"5min":1,"15min":1},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":3.0009999999999999,"5min":3.0009999999999999,"15min":3.0009999999999999},"last":0.746},{"interface":"front","average":{"1min":1.014,"5min":1.014,"15min":1.014},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":2.4159999999999999,"5min":2.4159999999999999,"15min":2.4159999999999999},"last":0.62}]},{"osd":3,"last update":"Fri Jun 12 02:38:14 2020","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":2.133,"5min":2.133,"15min":2.133},"last":1.052},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":2.4609999999999999,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":1.1559999999999999}]},{"osd":4,"last update":"Fri Jun 12 02:38:14 2020","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":2.1749999999999998,"5min":2.1749999999999998,"15min":2.1749999999999998},"last":0.95599999999999996},{"interface":"front","average":{"1min":1.302,"5min":1.302,"15min":1.302},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":2.2919999999999998,"5min":2.2919999999999998,"15min":2.2919999999999998},"last":1.1910000000000001}]},{"osd":5,"last update":"Fri Jun 12 02:38:14 2020","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.3200000000000001,"15min":1.3200000000000001},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":2.3090000000000002,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.448,"5min":1.448,"15min":1.448},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.835,"5min":2.835,"15min":2.835},"last":1.0189999999999999}]},{"osd":6,"last update":"Fri Jun 12 02:38:14 2020","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.389,"15min":1.389},"min":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"max":{"1min":2.4649999999999999,"5min":2.4649999999999999,"15min":2.4649999999999999},"last":1.448},{"interface":"front","average":{"1min":1.6299999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"max":{"1min":3.1589999999999998,"5min":3.1589999999999998,"15min":3.1589999999999998},"last":1.123}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.337}]}]},{"osd":0,"up_from":13,"seq":55834574872,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049788,"kb_used_data":1204,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322052,"statfs":{"total":96636764160,"available":95561781248,"internally_reserved":1073741824,"allocated":1232896,"data_stored":801382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jun 12 02:38:16 2020","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.7789999999999999,"5min":1.7789999999999999,"15min":1.7789999999999999},"last":0.92200000000000004}]},{"osd":2,"last update":"Fri Jun 12 02:38:16 2020","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":2.141,"5min":2.141,"15min":2.141},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.873,"5min":1.873,"15min":1.873},"last":0.77000000000000002}]},{"osd":3,"last update":"Fri Jun 12 02:38:16 2020","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":2.2320000000000002,"5min":2.2320000000000002,"15min":2.2320000000000002},"last":0.97599999999999998},{"interface":"front","average":{"1min":1.23,"5min":1.23,"15min":1.23},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.8220000000000001,"5min":1.8220000000000001,"15min":1.8220000000000001},"last":1.056}]},{"osd":4,"last update":"Fri Jun 12 02:38:16 2020","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.7709999999999999,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.554,"5min":1.554,"15min":1.554},"last":1.169}]},{"osd":5,"last update":"Fri Jun 12 02:38:16 2020","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.212,"15min":1.212},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":1.6699999999999999,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":1.079},{"interface":"front","average":{"1min":1.502,"5min":1.502,"15min":1.502},"min":{"1min":1.006,"5min":1.006,"15min":1.006},"max":{"1min":2.6240000000000001,"5min":2.6240000000000001,"15min":2.6240000000000001},"last":1.1870000000000001}]},{"osd":6,"last update":"Fri Jun 12 02:38:16 2020","interfaces":[{"interface":"back","average":{"1min":1.5840000000000001,"5min":1.5840000000000001,"15min":1.5840000000000001},"min":{"1min":1.157,"5min":1.157,"15min":1.157},"max":{"1min":3.2440000000000002,"5min":3.2440000000000002,"15min":3.2440000000000002},"last":1.28},{"interface":"front","average":{"1min":1.335,"5min":1.335,"15min":1.335},"min":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"max":{"1min":3.105,"5min":3.105,"15min":3.105},"last":1.101}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.246}]}]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049968,"kb_used_data":1384,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321872,"statfs":{"total":96636764160,"available":95561596928,"internally_reserved":1073741824,"allocated":1417216,"data_stored":991538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 12 02:38:10 2020","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.871,"15min":0.871},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.9259999999999999,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":3.0049999999999999,"5min":3.0049999999999999,"15min":3.0049999999999999},"last":1.524}]},{"osd":1,"last update":"Fri Jun 12 02:38:10 2020","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.046,"15min":1.046},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":2.21,"5min":2.21,"15min":2.21},"last":1.0049999999999999},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":2.5219999999999998,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":1.708}]},{"osd":3,"last update":"Fri Jun 12 02:38:10 2020","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1,"5min":1,"15min":1},"last":1.97},{"interface":"front","average":{"1min":1.145,"5min":1.145,"15min":1.145},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.2759999999999998,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":2.552}]},{"osd":4,"last update":"Fri Jun 12 02:38:10 2020","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.321,"15min":1.321},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":2.6949999999999998,"5min":2.6949999999999998,"15min":2.6949999999999998},"last":3.5859999999999999},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"min":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"max":{"1min":1.7549999999999999,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":2.113}]},{"osd":5,"last update":"Fri Jun 12 02:38:10 2020","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"min":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"max":{"1min":2.1419999999999999,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":2.2250000000000001},{"interface":"front","average":{"1min":1.399,"5min":1.399,"15min":1.399},"min":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"max":{"1min":2.851,"5min":2.851,"15min":2.851},"last":3.7610000000000001}]},{"osd":6,"last update":"Fri Jun 12 02:38:10 2020","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":1.7110000000000001,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":1.373},{"interface":"front","average":{"1min":1.462,"5min":1.462,"15min":1.462},"min":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"max":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":2.3719999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.9119999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574871,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050488,"kb_used_data":1904,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321352,"statfs":{"total":96636764160,"available":95561064448,"internally_reserved":1073741824,"allocated":1949696,"data_stored":1524018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 12 02:38:12 2020","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":2.0339999999999998,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.9219999999999999,"5min":1.9219999999999999,"15min":1.9219999999999999},"last":0.72799999999999998}]},{"osd":1,"last update":"Fri Jun 12 02:38:12 2020","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.9670000000000001,"5min":1.9670000000000001,"15min":1.9670000000000001},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":2.9969999999999999,"5min":2.9969999999999999,"15min":2.9969999999999999},"last":0.78600000000000003}]},{"osd":2,"last update":"Fri Jun 12 02:38:12 2020","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":4.1079999999999997,"5min":4.1079999999999997,"15min":4.1079999999999997},"last":1.181},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":2.504,"5min":2.504,"15min":2.504},"last":0.92600000000000005}]},{"osd":4,"last update":"Fri Jun 12 02:38:12 2020","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":2.359,"5min":2.359,"15min":2.359},"last":0.90700000000000003},{"interface":"front","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":3.9319999999999999,"5min":3.9319999999999999,"15min":3.9319999999999999},"last":1.3340000000000001}]},{"osd":5,"last update":"Fri Jun 12 02:38:12 2020","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":2.4279999999999999,"5min":2.4279999999999999,"15min":2.4279999999999999},"last":1.411},{"interface":"front","average":{"1min":1.452,"5min":1.452,"15min":1.452},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":5.0579999999999998,"5min":5.0579999999999998,"15min":5.0579999999999998},"last":1.0640000000000001}]},{"osd":6,"last update":"Fri Jun 12 02:38:12 2020","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.29,"15min":1.29},"min":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"max":{"1min":4.0410000000000004,"5min":4.0410000000000004,"15min":4.0410000000000004},"last":1.034},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.3620000000000001,"15min":1.3620000000000001},"min":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"max":{"1min":3.0230000000000001,"5min":3.0230000000000001,"15min":3.0230000000000001},"last":1.4850000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2010000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574870,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1051832,"kb_used_data":3248,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93320008,"statfs":{"total":96636764160,"available":95559688192,"internally_reserved":1073741824,"allocated":3325952,"data_stored":2894438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.712,"5min":1.712,"15min":1.712},"last":1.075},{"interface":"front","average":{"1min":1.02,"5min":1.02,"15min":1.02},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":2.855,"5min":2.855,"15min":2.855},"last":1.6579999999999999}]},{"osd":1,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.8240000000000001,"5min":1.8240000000000001,"15min":1.8240000000000001},"last":0.64400000000000002}]},{"osd":2,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":3.1480000000000001,"5min":3.1480000000000001,"15min":3.1480000000000001},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"min":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"max":{"1min":3.1259999999999999,"5min":3.1259999999999999,"15min":3.1259999999999999},"last":1.284}]},{"osd":3,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.079,"15min":1.079},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":2.411,"5min":2.411,"15min":2.411},"last":0.81399999999999995},{"interface":"front","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":2.734,"5min":2.734,"15min":2.734},"last":1.008}]},{"osd":5,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.04,"15min":1.04},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.6399999999999999,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":0.93999999999999995},{"interface":"front","average":{"1min":1.115,"5min":1.115,"15min":1.115},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"last":1.52}]},{"osd":6,"last update":"Fri Jun 12 02:38:11 2020","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":0.998,"15min":0.998},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":1.6850000000000001},{"interface":"front","average":{"1min":1.341,"5min":1.341,"15min":1.341},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":2.1850000000000001,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":1.3580000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4950000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574870,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050128,"kb_used_data":1544,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321712,"statfs":{"total":96636764160,"available":95561433088,"internally_reserved":1073741824,"allocated":1581056,"data_stored":1159008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 12 02:38:13 2020","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.456,"5min":1.456,"15min":1.456},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.597,"5min":1.597,"15min":1.597},"last":0.62}]},{"osd":1,"last update":"Fri Jun 12 02:38:13 2020","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.73799999999999999}]},{"osd":2,"last update":"Fri Jun 12 02:38:13 2020","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.82699999999999996},{"interface":"front","average":{"1min":1.012,"5min":1.012,"15min":1.012},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.4259999999999999,"5min":1.4259999999999999,"15min":1.4259999999999999},"last":0.77100000000000002}]},{"osd":3,"last update":"Fri Jun 12 02:38:13 2020","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":2.004,"5min":2.004,"15min":2.004},"last":0.91100000000000003},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.627,"5min":1.627,"15min":1.627},"last":0.94999999999999996}]},{"osd":4,"last update":"Fri Jun 12 02:38:13 2020","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.997,"15min":0.997},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.796,"5min":1.796,"15min":1.796},"last":0.64900000000000002},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":1.0780000000000001}]},{"osd":6,"last update":"Fri Jun 12 02:38:13 2020","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.827,"5min":1.827,"15min":1.827},"last":0.79500000000000004},{"interface":"front","average":{"1min":1.228,"5min":1.228,"15min":1.228},"min":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"max":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":1.012}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1000000000000001}]}]}]}} 2020-06-12T02:38:56.683 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2020-06-12T02:38:56.686 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2020-06-12T02:38:56.686 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:38:56.676+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:38:56.687 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2020-06-12T02:38:56.687 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 1.000000 2020-06-12T02:38:56.687 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd primary-affinity 5 1 2020-06-12T02:38:56.688 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-06-12T02:38:56.787 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:56.784+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:56.849 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:56.849 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:38:56.850 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2020-06-12T02:38:56.850 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:56.859 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-06-12T02:38:56.903 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:56.901+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:56.986 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:56.990 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:38:56.990 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2020-06-12T02:38:56.990 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2020-06-12T02:38:56.990 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:56.994 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:56.991+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:56.995 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-06-12T02:38:57.094 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:57.092+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:57.163 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:57.164 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2020-06-12T02:38:57.164 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2020-06-12T02:38:57.164 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2020-06-12T02:38:57.164 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.164 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.3s2 49/21 ECSubWrite(tid=4109, reqid=client.4512.0:8841, at_version=49'773, trim_to=0'0, roll_forward_to=49'771))", 2020-06-12T02:38:57.165 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.822144+0000", 2020-06-12T02:38:57.165 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 45.336967264000002, 2020-06-12T02:38:57.165 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014799214, 2020-06-12T02:38:57.165 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:57.166 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:57.167 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:57.167 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.167 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:57.168 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822144+0000", 2020-06-12T02:38:57.168 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:57.168 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.169 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.169 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:57.169 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822144+0000", 2020-06-12T02:38:57.170 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00061470800000000005 2020-06-12T02:38:57.170 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.171 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.171 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:57.171 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822759+0000", 2020-06-12T02:38:57.172 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.7732000000000001e-05 2020-06-12T02:38:57.172 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.173 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.173 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:57.173 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822786+0000", 2020-06-12T02:38:57.174 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1010999999999999e-05 2020-06-12T02:38:57.174 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.176 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.176 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:57.177 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822807+0000", 2020-06-12T02:38:57.178 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.842e-06 2020-06-12T02:38:57.178 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.179 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.179 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:57.179 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822817+0000", 2020-06-12T02:38:57.179 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00024702300000000002 2020-06-12T02:38:57.179 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.180 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.180 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:57.180 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.823064+0000", 2020-06-12T02:38:57.180 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5207e-05 2020-06-12T02:38:57.180 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.180 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.181 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:57.181 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.823079+0000", 2020-06-12T02:38:57.181 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013806817000000001 2020-06-12T02:38:57.181 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.181 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.189 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:57.189 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.836886+0000", 2020-06-12T02:38:57.189 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6873999999999998e-05 2020-06-12T02:38:57.189 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.189 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.190 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:57.190 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.836943+0000", 2020-06-12T02:38:57.190 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014799214 2020-06-12T02:38:57.190 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:57.190 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:57.191 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:57.191 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.191 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.191 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14657 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~606208] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:57.191 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.209225+0000", 2020-06-12T02:38:57.192 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 25.949886013, 2020-06-12T02:38:57.192 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016128905999999998, 2020-06-12T02:38:57.193 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:57.193 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:57.193 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:57.193 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:57.193 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:57.194 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14657 2020-06-12T02:38:57.194 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.194 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:57.194 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.194 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:57.195 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209225+0000", 2020-06-12T02:38:57.196 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:57.196 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.196 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.196 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:57.197 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209225+0000", 2020-06-12T02:38:57.197 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018386699999999999 2020-06-12T02:38:57.197 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.197 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.197 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:57.197 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209409+0000", 2020-06-12T02:38:57.198 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8160000000000001e-06 2020-06-12T02:38:57.198 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.198 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.198 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:57.198 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209414+0000", 2020-06-12T02:38:57.199 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0230000000000004e-06 2020-06-12T02:38:57.199 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.199 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.199 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:57.199 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209421+0000", 2020-06-12T02:38:57.200 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0139999999999999e-06 2020-06-12T02:38:57.200 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.200 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.200 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:57.200 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209424+0000", 2020-06-12T02:38:57.200 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0054798169999999997 2020-06-12T02:38:57.201 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.201 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.201 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:57.201 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.214904+0000", 2020-06-12T02:38:57.201 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123063 2020-06-12T02:38:57.202 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.202 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.202 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:57.202 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215027+0000", 2020-06-12T02:38:57.202 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001159923 2020-06-12T02:38:57.203 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.203 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.203 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:57.203 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.216187+0000", 2020-06-12T02:38:57.203 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2835000000000002e-05 2020-06-12T02:38:57.203 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.204 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.204 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:57.204 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.216270+0000", 2020-06-12T02:38:57.204 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073575799999999998 2020-06-12T02:38:57.204 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.205 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.205 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:57.205 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.217005+0000", 2020-06-12T02:38:57.205 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0082434659999999996 2020-06-12T02:38:57.205 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.205 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.206 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:57.206 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225249+0000", 2020-06-12T02:38:57.206 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0145999999999997e-05 2020-06-12T02:38:57.206 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.206 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.207 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:57.207 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225299+0000", 2020-06-12T02:38:57.207 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5177999999999999e-05 2020-06-12T02:38:57.207 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.207 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.208 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:57.208 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225354+0000", 2020-06-12T02:38:57.208 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016128905999999998 2020-06-12T02:38:57.208 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:57.208 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:57.208 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:57.209 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.209 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.209 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14658 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~598016] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:57.209 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.209455+0000", 2020-06-12T02:38:57.209 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 25.949656259000001, 2020-06-12T02:38:57.210 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016447966000000001, 2020-06-12T02:38:57.210 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:57.210 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:57.210 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:57.211 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:57.211 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:57.211 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14658 2020-06-12T02:38:57.211 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.211 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:57.212 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.212 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:57.212 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209455+0000", 2020-06-12T02:38:57.212 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:57.212 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.213 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.213 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:57.213 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209455+0000", 2020-06-12T02:38:57.213 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037921599999999998 2020-06-12T02:38:57.213 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.213 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.214 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:57.214 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209834+0000", 2020-06-12T02:38:57.214 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8339999999999999e-06 2020-06-12T02:38:57.214 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.214 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.215 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:57.215 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209842+0000", 2020-06-12T02:38:57.215 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2564e-05 2020-06-12T02:38:57.216 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.216 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.216 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:57.216 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209855+0000", 2020-06-12T02:38:57.217 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.135e-06 2020-06-12T02:38:57.217 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.217 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.218 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:57.218 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209860+0000", 2020-06-12T02:38:57.219 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0052530290000000002 2020-06-12T02:38:57.219 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.219 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.219 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:57.219 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215113+0000", 2020-06-12T02:38:57.220 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6229000000000003e-05 2020-06-12T02:38:57.220 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.220 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.220 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:57.220 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215209+0000", 2020-06-12T02:38:57.221 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045765409999999999 2020-06-12T02:38:57.221 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.221 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.221 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:57.221 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.219785+0000", 2020-06-12T02:38:57.222 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000102846 2020-06-12T02:38:57.222 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.222 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.222 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:57.222 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.219888+0000", 2020-06-12T02:38:57.223 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00077088700000000005 2020-06-12T02:38:57.223 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.223 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.223 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:57.223 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.220659+0000", 2020-06-12T02:38:57.224 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046979739999999997 2020-06-12T02:38:57.224 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.224 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.224 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:57.224 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225357+0000", 2020-06-12T02:38:57.225 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00046069600000000002 2020-06-12T02:38:57.225 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.225 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.225 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:57.225 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225818+0000", 2020-06-12T02:38:57.225 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5014999999999998e-05 2020-06-12T02:38:57.226 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.226 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.226 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:57.226 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225903+0000", 2020-06-12T02:38:57.226 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016447966000000001 2020-06-12T02:38:57.227 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:57.227 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:57.227 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:57.227 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.227 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.228 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14659 3.0s0 3:0dddb690:::smithi10729407-10:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:57.228 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.209909+0000", 2020-06-12T02:38:57.228 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 25.949202051, 2020-06-12T02:38:57.229 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016618236000000002, 2020-06-12T02:38:57.229 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:57.229 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:57.229 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:57.229 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:57.230 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:57.230 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14659 2020-06-12T02:38:57.230 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.230 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:57.230 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.231 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:57.231 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209909+0000", 2020-06-12T02:38:57.231 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:57.231 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.231 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.232 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:57.232 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209909+0000", 2020-06-12T02:38:57.232 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8085999999999999e-05 2020-06-12T02:38:57.232 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.232 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.233 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:57.233 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209967+0000", 2020-06-12T02:38:57.233 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3169999999999998e-06 2020-06-12T02:38:57.233 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.233 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.234 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:57.234 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209974+0000", 2020-06-12T02:38:57.234 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1246999999999999e-05 2020-06-12T02:38:57.234 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.234 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.235 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:57.235 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209985+0000", 2020-06-12T02:38:57.235 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1439999999999996e-06 2020-06-12T02:38:57.235 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.235 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.236 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:57.236 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209989+0000", 2020-06-12T02:38:57.236 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0052537900000000004 2020-06-12T02:38:57.236 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.236 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.236 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:57.237 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215243+0000", 2020-06-12T02:38:57.237 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6023000000000002e-05 2020-06-12T02:38:57.237 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.237 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.237 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:57.238 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215299+0000", 2020-06-12T02:38:57.238 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0087846910000000007 2020-06-12T02:38:57.238 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.238 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.238 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:57.239 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.224083+0000", 2020-06-12T02:38:57.239 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8504000000000004e-05 2020-06-12T02:38:57.239 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.239 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.239 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:57.239 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.224182+0000", 2020-06-12T02:38:57.240 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044251400000000003 2020-06-12T02:38:57.240 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.240 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.240 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:57.240 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.224624+0000", 2020-06-12T02:38:57.241 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013383290000000001 2020-06-12T02:38:57.241 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.241 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.241 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:57.241 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225963+0000", 2020-06-12T02:38:57.241 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047942699999999999 2020-06-12T02:38:57.242 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.242 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.242 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:57.242 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.226442+0000", 2020-06-12T02:38:57.242 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5164e-05 2020-06-12T02:38:57.243 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.243 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.243 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:57.243 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.226527+0000", 2020-06-12T02:38:57.243 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016618236000000002 2020-06-12T02:38:57.244 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:57.244 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:57.244 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:57.244 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.244 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.245 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14660 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e59)", 2020-06-12T02:38:57.245 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.210020+0000", 2020-06-12T02:38:57.245 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 25.949091073000002, 2020-06-12T02:38:57.245 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017984777, 2020-06-12T02:38:57.246 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:57.246 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:57.246 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:57.246 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:57.246 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:57.247 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14660 2020-06-12T02:38:57.247 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.247 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:57.247 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.247 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:57.247 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210020+0000", 2020-06-12T02:38:57.248 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:57.248 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.248 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.248 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:57.248 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210020+0000", 2020-06-12T02:38:57.249 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2875999999999999e-05 2020-06-12T02:38:57.249 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.249 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.250 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:57.250 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210043+0000", 2020-06-12T02:38:57.250 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2200000000000001e-06 2020-06-12T02:38:57.251 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.251 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.252 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:57.252 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210046+0000", 2020-06-12T02:38:57.252 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7410000000000003e-06 2020-06-12T02:38:57.252 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.253 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.253 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:57.253 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210053+0000", 2020-06-12T02:38:57.253 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5610000000000001e-06 2020-06-12T02:38:57.253 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.254 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.254 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:57.254 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210055+0000", 2020-06-12T02:38:57.254 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0052596409999999998 2020-06-12T02:38:57.254 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.255 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.255 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:57.255 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215315+0000", 2020-06-12T02:38:57.255 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5627999999999997e-05 2020-06-12T02:38:57.255 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.256 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.256 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:57.256 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215371+0000", 2020-06-12T02:38:57.256 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0098069549999999991 2020-06-12T02:38:57.256 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.257 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.257 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:57.257 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225178+0000", 2020-06-12T02:38:57.257 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4530000000000001e-05 2020-06-12T02:38:57.257 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.257 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.258 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:57.258 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225232+0000", 2020-06-12T02:38:57.258 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017497350000000001 2020-06-12T02:38:57.258 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.258 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.259 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:57.259 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.226982+0000", 2020-06-12T02:38:57.259 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8154999999999998e-05 2020-06-12T02:38:57.259 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.259 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.260 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:57.260 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.227080+0000", 2020-06-12T02:38:57.260 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00092473499999999999 2020-06-12T02:38:57.260 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.260 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.261 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:57.261 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.228005+0000", 2020-06-12T02:38:57.261 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017984777 2020-06-12T02:38:57.261 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:57.261 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:57.262 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:57.262 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.262 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.262 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.3s2 59/53 ECSubWrite(tid=5970, reqid=client.4512.0:15226, at_version=59'1316, trim_to=0'0, roll_forward_to=59'1315))", 2020-06-12T02:38:57.262 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.829184+0000", 2020-06-12T02:38:57.262 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 24.329926857, 2020-06-12T02:38:57.263 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.081295588000000002, 2020-06-12T02:38:57.263 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:57.263 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:57.263 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:57.263 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.264 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:57.264 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829184+0000", 2020-06-12T02:38:57.264 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:57.264 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.264 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.265 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:57.265 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829184+0000", 2020-06-12T02:38:57.265 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00023814999999999999 2020-06-12T02:38:57.265 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.265 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.265 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:57.266 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829422+0000", 2020-06-12T02:38:57.266 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9768000000000001e-05 2020-06-12T02:38:57.266 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.266 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.266 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:57.267 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829442+0000", 2020-06-12T02:38:57.267 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1022000000000001e-05 2020-06-12T02:38:57.267 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.267 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.267 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:57.268 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829453+0000", 2020-06-12T02:38:57.268 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2759999999999997e-06 2020-06-12T02:38:57.268 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.268 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.268 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:57.268 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829460+0000", 2020-06-12T02:38:57.269 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7284000000000002e-05 2020-06-12T02:38:57.269 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.269 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.269 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:57.269 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829517+0000", 2020-06-12T02:38:57.270 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7028999999999999e-05 2020-06-12T02:38:57.270 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.270 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.270 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:57.270 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829534+0000", 2020-06-12T02:38:57.270 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.080899873999999997 2020-06-12T02:38:57.271 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.271 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.271 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:57.271 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.910434+0000", 2020-06-12T02:38:57.271 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6184999999999999e-05 2020-06-12T02:38:57.272 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.272 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.272 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:57.272 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.910480+0000", 2020-06-12T02:38:57.272 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.081295588000000002 2020-06-12T02:38:57.273 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:57.273 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:57.273 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:57.273 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.273 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.273 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.3s2 59/53 ECSubWrite(tid=5973, reqid=client.4512.0:15229, at_version=59'1319, trim_to=0'0, roll_forward_to=59'1315))", 2020-06-12T02:38:57.274 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.901708+0000", 2020-06-12T02:38:57.274 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 24.257402977999998, 2020-06-12T02:38:57.274 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013662957999999999, 2020-06-12T02:38:57.274 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:57.274 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:57.275 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:57.275 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.275 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:57.275 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901708+0000", 2020-06-12T02:38:57.276 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:57.276 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.276 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.276 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:57.276 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901708+0000", 2020-06-12T02:38:57.276 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7360000000000001e-05 2020-06-12T02:38:57.277 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.277 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.277 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:57.277 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901756+0000", 2020-06-12T02:38:57.277 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4608e-05 2020-06-12T02:38:57.278 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.278 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.278 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:57.278 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901770+0000", 2020-06-12T02:38:57.278 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.219e-05 2020-06-12T02:38:57.279 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.279 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.279 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:57.279 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901782+0000", 2020-06-12T02:38:57.279 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.511e-06 2020-06-12T02:38:57.279 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.280 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.280 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:57.280 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901789+0000", 2020-06-12T02:38:57.280 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.008025562 2020-06-12T02:38:57.280 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.281 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.281 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:57.281 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.909814+0000", 2020-06-12T02:38:57.281 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1472e-05 2020-06-12T02:38:57.281 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.281 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.282 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:57.282 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.909826+0000", 2020-06-12T02:38:57.282 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0055013350000000004 2020-06-12T02:38:57.282 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.282 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.283 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:57.283 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.915327+0000", 2020-06-12T02:38:57.283 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.392e-05 2020-06-12T02:38:57.283 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.283 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.283 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:57.284 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.915371+0000", 2020-06-12T02:38:57.284 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013662957999999999 2020-06-12T02:38:57.284 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:57.284 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:57.284 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:57.285 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.288 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.288 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.3s2 59/53 ECSubWrite(tid=5974, reqid=client.4512.0:15231, at_version=59'1320, trim_to=0'0, roll_forward_to=59'1315))", 2020-06-12T02:38:57.288 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.901839+0000", 2020-06-12T02:38:57.288 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 24.25727238, 2020-06-12T02:38:57.289 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013568525, 2020-06-12T02:38:57.289 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:57.289 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:57.289 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:57.289 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.290 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:57.290 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901839+0000", 2020-06-12T02:38:57.290 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:57.290 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.290 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.291 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:57.291 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901839+0000", 2020-06-12T02:38:57.291 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6489999999999997e-05 2020-06-12T02:38:57.291 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.291 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.292 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:57.292 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901885+0000", 2020-06-12T02:38:57.292 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6600999999999999e-05 2020-06-12T02:38:57.292 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.292 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.292 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:57.293 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901902+0000", 2020-06-12T02:38:57.293 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2427e-05 2020-06-12T02:38:57.293 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.293 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.293 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:57.294 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901914+0000", 2020-06-12T02:38:57.294 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1710000000000001e-06 2020-06-12T02:38:57.294 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.294 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.294 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:57.298 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901919+0000", 2020-06-12T02:38:57.298 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0086285540000000001 2020-06-12T02:38:57.298 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.298 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.299 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:57.299 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.910548+0000", 2020-06-12T02:38:57.299 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0939e-05 2020-06-12T02:38:57.299 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.299 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.300 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:57.300 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.910559+0000", 2020-06-12T02:38:57.301 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0048161899999999997 2020-06-12T02:38:57.301 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.301 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.301 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:57.302 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.915375+0000", 2020-06-12T02:38:57.302 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2153000000000002e-05 2020-06-12T02:38:57.302 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.302 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.302 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:57.303 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.915407+0000", 2020-06-12T02:38:57.303 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013568525 2020-06-12T02:38:57.303 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:57.303 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:57.303 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:57.304 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.304 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.304 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15246 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~671744] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:57.304 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.953872+0000", 2020-06-12T02:38:57.304 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 24.205239314, 2020-06-12T02:38:57.304 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015306691000000001, 2020-06-12T02:38:57.305 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:57.305 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:57.305 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:57.305 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:57.305 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:57.306 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15246 2020-06-12T02:38:57.306 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.306 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:57.306 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.306 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:57.307 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.953872+0000", 2020-06-12T02:38:57.307 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:57.307 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.307 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.307 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:57.308 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.953872+0000", 2020-06-12T02:38:57.308 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000232544 2020-06-12T02:38:57.308 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.308 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.309 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:57.310 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954104+0000", 2020-06-12T02:38:57.310 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7513999999999999e-05 2020-06-12T02:38:57.310 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.311 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.311 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:57.311 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954122+0000", 2020-06-12T02:38:57.311 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.844e-05 2020-06-12T02:38:57.311 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.312 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.312 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:57.312 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954140+0000", 2020-06-12T02:38:57.312 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2570000000000007e-06 2020-06-12T02:38:57.312 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.312 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.313 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:57.313 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954149+0000", 2020-06-12T02:38:57.313 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00031049499999999999 2020-06-12T02:38:57.313 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.313 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.314 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:57.314 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954459+0000", 2020-06-12T02:38:57.314 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000118182 2020-06-12T02:38:57.314 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.314 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.315 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:57.315 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954577+0000", 2020-06-12T02:38:57.315 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00071788400000000001 2020-06-12T02:38:57.315 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.315 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.315 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:57.316 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.955295+0000", 2020-06-12T02:38:57.316 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013717859000000001 2020-06-12T02:38:57.316 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.316 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.316 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:57.317 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.969013+0000", 2020-06-12T02:38:57.317 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4895999999999995e-05 2020-06-12T02:38:57.317 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.317 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.317 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:57.318 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.969098+0000", 2020-06-12T02:38:57.318 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0619999999999997e-05 2020-06-12T02:38:57.318 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.318 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.318 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:57.319 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.969179+0000", 2020-06-12T02:38:57.319 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015306691000000001 2020-06-12T02:38:57.319 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:57.320 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:57.320 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:57.320 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.320 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.321 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15277 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~720896] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:57.321 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:33.067109+0000", 2020-06-12T02:38:57.321 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 24.092002254000001, 2020-06-12T02:38:57.321 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015909191999999999, 2020-06-12T02:38:57.322 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:57.322 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:57.322 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:57.322 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:57.322 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:57.323 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15277 2020-06-12T02:38:57.323 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.323 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:57.323 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.323 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:57.323 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067109+0000", 2020-06-12T02:38:57.324 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:57.324 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.324 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.324 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:57.324 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067109+0000", 2020-06-12T02:38:57.325 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018431400000000001 2020-06-12T02:38:57.325 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.325 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.325 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:57.325 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067293+0000", 2020-06-12T02:38:57.326 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1790000000000002e-06 2020-06-12T02:38:57.326 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.326 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.326 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:57.326 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067300+0000", 2020-06-12T02:38:57.327 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5259999999999999e-06 2020-06-12T02:38:57.327 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.327 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.327 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:57.327 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067310+0000", 2020-06-12T02:38:57.328 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8179999999999997e-06 2020-06-12T02:38:57.328 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.328 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.328 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:57.328 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067314+0000", 2020-06-12T02:38:57.328 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6529999999999997e-05 2020-06-12T02:38:57.329 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.329 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.329 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:57.329 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067360+0000", 2020-06-12T02:38:57.329 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012724399999999999 2020-06-12T02:38:57.330 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.330 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.330 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:57.330 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067488+0000", 2020-06-12T02:38:57.330 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0010983989999999999 2020-06-12T02:38:57.331 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.331 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.331 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:57.331 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.068586+0000", 2020-06-12T02:38:57.332 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5165000000000001e-05 2020-06-12T02:38:57.332 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.332 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.332 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:57.332 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.068681+0000", 2020-06-12T02:38:57.333 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086510899999999997 2020-06-12T02:38:57.333 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.333 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.333 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:57.333 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.069546+0000", 2020-06-12T02:38:57.334 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013284429 2020-06-12T02:38:57.334 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.334 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.334 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:57.334 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.082831+0000", 2020-06-12T02:38:57.334 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4197000000000006e-05 2020-06-12T02:38:57.335 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.335 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.335 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:57.335 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.082925+0000", 2020-06-12T02:38:57.335 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3282000000000006e-05 2020-06-12T02:38:57.336 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.336 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.336 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:57.336 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.083018+0000", 2020-06-12T02:38:57.336 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015909191999999999 2020-06-12T02:38:57.337 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:57.337 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:57.337 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:57.338 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.338 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.338 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15331 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~688128] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:38:57.338 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:33.276748+0000", 2020-06-12T02:38:57.339 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 23.882362742000002, 2020-06-12T02:38:57.339 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016825066999999999, 2020-06-12T02:38:57.339 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:57.340 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:57.340 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:57.340 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:57.340 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:57.340 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15331 2020-06-12T02:38:57.341 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.341 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:57.341 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.342 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:57.342 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276748+0000", 2020-06-12T02:38:57.342 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:57.342 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.342 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.342 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:57.343 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276748+0000", 2020-06-12T02:38:57.343 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016936699999999999 2020-06-12T02:38:57.343 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.343 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.344 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:57.344 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276918+0000", 2020-06-12T02:38:57.344 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6389999999999998e-06 2020-06-12T02:38:57.344 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.345 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.345 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:57.345 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276926+0000", 2020-06-12T02:38:57.345 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3591000000000001e-05 2020-06-12T02:38:57.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.346 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.346 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:57.346 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276940+0000", 2020-06-12T02:38:57.346 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0429999999999996e-06 2020-06-12T02:38:57.347 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.347 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.347 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:57.347 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276949+0000", 2020-06-12T02:38:57.347 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8825000000000001e-05 2020-06-12T02:38:57.348 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.348 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.348 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:57.348 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276998+0000", 2020-06-12T02:38:57.348 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6483000000000005e-05 2020-06-12T02:38:57.348 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.349 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.349 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:57.349 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.277084+0000", 2020-06-12T02:38:57.349 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00057545800000000005 2020-06-12T02:38:57.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.350 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.350 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:57.350 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.277660+0000", 2020-06-12T02:38:57.350 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015617126 2020-06-12T02:38:57.350 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.351 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.351 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:57.351 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.293277+0000", 2020-06-12T02:38:57.351 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00021931699999999999 2020-06-12T02:38:57.352 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.352 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.352 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:57.353 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.293496+0000", 2020-06-12T02:38:57.353 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7218000000000005e-05 2020-06-12T02:38:57.353 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.353 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.353 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:57.354 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.293573+0000", 2020-06-12T02:38:57.354 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016825066999999999 2020-06-12T02:38:57.354 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:57.354 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:57.354 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:57.355 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.355 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.355 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.7s1 60/37 ECSubWrite(tid=8504, reqid=client.4512.0:16225, at_version=60'1427, trim_to=0'0, roll_forward_to=60'1425))", 2020-06-12T02:38:57.355 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:36.033154+0000", 2020-06-12T02:38:57.355 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 21.125957379999999, 2020-06-12T02:38:57.356 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016768244000000002, 2020-06-12T02:38:57.356 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:57.356 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:57.356 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:57.356 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.356 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:57.358 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033154+0000", 2020-06-12T02:38:57.358 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:57.358 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.358 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.358 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:57.359 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033154+0000", 2020-06-12T02:38:57.360 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000154444 2020-06-12T02:38:57.360 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.361 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.361 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:57.361 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033308+0000", 2020-06-12T02:38:57.361 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8772000000000002e-05 2020-06-12T02:38:57.362 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.362 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.362 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:57.362 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033337+0000", 2020-06-12T02:38:57.362 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6704000000000001e-05 2020-06-12T02:38:57.363 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.363 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.363 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:57.363 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033354+0000", 2020-06-12T02:38:57.363 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6100000000000002e-06 2020-06-12T02:38:57.364 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.364 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.364 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:57.364 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033360+0000", 2020-06-12T02:38:57.364 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000173893 2020-06-12T02:38:57.365 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.367 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.367 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:57.368 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033534+0000", 2020-06-12T02:38:57.368 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1929999999999999e-05 2020-06-12T02:38:57.368 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.368 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.368 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:57.368 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033546+0000", 2020-06-12T02:38:57.369 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016272066000000002 2020-06-12T02:38:57.369 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.369 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.369 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:57.370 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.049818+0000", 2020-06-12T02:38:57.370 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103825 2020-06-12T02:38:57.370 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.370 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.370 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:57.371 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.049922+0000", 2020-06-12T02:38:57.371 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016768244000000002 2020-06-12T02:38:57.371 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:57.371 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:57.371 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:57.371 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.372 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.372 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:17973 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~507904] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:38:57.372 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:41.095893+0000", 2020-06-12T02:38:57.372 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 16.063217826999999, 2020-06-12T02:38:57.372 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013655560000000001, 2020-06-12T02:38:57.373 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:57.373 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:38:57.373 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:57.373 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:57.373 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:57.374 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 17973 2020-06-12T02:38:57.380 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.381 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:57.381 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.381 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:57.381 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.095893+0000", 2020-06-12T02:38:57.381 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:57.382 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.382 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.382 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:57.382 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.095893+0000", 2020-06-12T02:38:57.382 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018351000000000001 2020-06-12T02:38:57.383 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.383 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.383 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:57.383 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.096077+0000", 2020-06-12T02:38:57.383 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1559999999999995e-06 2020-06-12T02:38:57.383 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.384 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.384 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:57.384 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.096085+0000", 2020-06-12T02:38:57.384 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3764999999999999e-05 2020-06-12T02:38:57.384 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.385 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.385 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:57.385 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.096099+0000", 2020-06-12T02:38:57.385 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0460000000000002e-06 2020-06-12T02:38:57.385 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.386 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.386 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:57.386 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.096105+0000", 2020-06-12T02:38:57.386 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6931000000000002e-05 2020-06-12T02:38:57.386 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.387 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.387 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:57.387 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.096152+0000", 2020-06-12T02:38:57.387 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010251600000000001 2020-06-12T02:38:57.388 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.388 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.388 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:57.388 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.096254+0000", 2020-06-12T02:38:57.389 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072763000000000005 2020-06-12T02:38:57.389 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.389 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.389 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:57.389 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.096982+0000", 2020-06-12T02:38:57.390 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012262115000000001 2020-06-12T02:38:57.390 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.390 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.390 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:57.391 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.109244+0000", 2020-06-12T02:38:57.391 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1791000000000003e-05 2020-06-12T02:38:57.391 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.391 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.391 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:38:57.392 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.109336+0000", 2020-06-12T02:38:57.392 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0002131 2020-06-12T02:38:57.392 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.392 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.392 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:57.392 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.109549+0000", 2020-06-12T02:38:57.393 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013655560000000001 2020-06-12T02:38:57.393 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:57.393 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:57.393 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:57.393 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.394 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.394 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:17977 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:38:57.394 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:41.103849+0000", 2020-06-12T02:38:57.394 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 16.055262198000001, 2020-06-12T02:38:57.395 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015465273, 2020-06-12T02:38:57.395 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:57.395 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:57.395 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:57.395 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:57.396 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:57.396 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 17977 2020-06-12T02:38:57.396 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.396 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:57.396 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.397 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:57.397 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.103849+0000", 2020-06-12T02:38:57.397 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:57.397 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.397 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.398 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:57.398 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.103849+0000", 2020-06-12T02:38:57.398 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1254e-05 2020-06-12T02:38:57.398 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.398 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.398 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:57.399 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.103870+0000", 2020-06-12T02:38:57.399 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9869999999999999e-06 2020-06-12T02:38:57.399 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.399 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.400 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:57.406 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.103873+0000", 2020-06-12T02:38:57.408 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6159999999999998e-06 2020-06-12T02:38:57.408 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.408 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.409 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:57.409 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.103880+0000", 2020-06-12T02:38:57.409 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0899999999999999e-06 2020-06-12T02:38:57.409 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.409 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.410 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:57.410 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.103882+0000", 2020-06-12T02:38:57.410 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0066572259999999996 2020-06-12T02:38:57.410 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.410 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.411 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:57.411 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.110539+0000", 2020-06-12T02:38:57.411 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5937000000000003e-05 2020-06-12T02:38:57.411 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.411 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.411 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:57.412 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.110595+0000", 2020-06-12T02:38:57.412 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0066983820000000001 2020-06-12T02:38:57.412 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.412 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.412 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:57.413 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.117293+0000", 2020-06-12T02:38:57.413 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103828 2020-06-12T02:38:57.413 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.413 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.413 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:57.414 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.117397+0000", 2020-06-12T02:38:57.414 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0019169530000000001 2020-06-12T02:38:57.414 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.414 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.414 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:57.414 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.119314+0000", 2020-06-12T02:38:57.415 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015465273 2020-06-12T02:38:57.415 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:57.415 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:57.415 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:57.415 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.416 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.416 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:17998 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:38:57.416 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:41.190236+0000", 2020-06-12T02:38:57.416 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 15.968875143, 2020-06-12T02:38:57.416 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013903766, 2020-06-12T02:38:57.417 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:57.417 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:38:57.417 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:38:57.417 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:38:57.417 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:38:57.417 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 17998 2020-06-12T02:38:57.418 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.418 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:57.418 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.418 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:57.418 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.190236+0000", 2020-06-12T02:38:57.419 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:57.419 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.419 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.419 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:57.419 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.190236+0000", 2020-06-12T02:38:57.419 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.207e-05 2020-06-12T02:38:57.420 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.420 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.420 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:57.420 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.190258+0000", 2020-06-12T02:38:57.420 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9629999999999998e-06 2020-06-12T02:38:57.421 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.421 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.421 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:57.421 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.190261+0000", 2020-06-12T02:38:57.421 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.409e-06 2020-06-12T02:38:57.422 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.422 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.422 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:57.422 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.190267+0000", 2020-06-12T02:38:57.422 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2189999999999998e-06 2020-06-12T02:38:57.422 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.423 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.423 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:57.423 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.190270+0000", 2020-06-12T02:38:57.423 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0089464690000000003 2020-06-12T02:38:57.424 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.424 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.424 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:57.425 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.199216+0000", 2020-06-12T02:38:57.425 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010023500000000001 2020-06-12T02:38:57.425 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.425 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.425 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:38:57.426 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.199316+0000", 2020-06-12T02:38:57.426 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0038963800000000001 2020-06-12T02:38:57.426 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.426 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.426 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:57.427 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.203213+0000", 2020-06-12T02:38:57.427 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6963000000000003e-05 2020-06-12T02:38:57.427 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.427 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.427 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:38:57.427 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.203270+0000", 2020-06-12T02:38:57.428 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00087005800000000005 2020-06-12T02:38:57.428 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.428 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.428 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:57.428 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.204140+0000", 2020-06-12T02:38:57.429 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013903766 2020-06-12T02:38:57.429 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:57.429 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:57.429 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:57.429 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.430 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.430 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.7s1 62/37 ECSubWrite(tid=9513, reqid=client.4512.0:18266, at_version=0'0, trim_to=0'0, roll_forward_to=62'1601))", 2020-06-12T02:38:57.430 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:41.904485+0000", 2020-06-12T02:38:57.430 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 15.254626046, 2020-06-12T02:38:57.430 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01412036, 2020-06-12T02:38:57.430 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:57.431 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:57.431 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:57.431 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.431 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:57.431 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904485+0000", 2020-06-12T02:38:57.432 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:57.432 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.432 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.432 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:57.432 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904485+0000", 2020-06-12T02:38:57.432 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.345e-05 2020-06-12T02:38:57.433 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.433 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.433 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:57.433 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904519+0000", 2020-06-12T02:38:57.433 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0869999999999999e-05 2020-06-12T02:38:57.434 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.434 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.434 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:57.434 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904529+0000", 2020-06-12T02:38:57.434 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1248e-05 2020-06-12T02:38:57.435 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.435 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.435 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:57.435 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904541+0000", 2020-06-12T02:38:57.435 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0730999999999999e-05 2020-06-12T02:38:57.435 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.436 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.436 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:57.436 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904551+0000", 2020-06-12T02:38:57.436 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0985999999999997e-05 2020-06-12T02:38:57.436 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.437 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.437 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:57.437 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904592+0000", 2020-06-12T02:38:57.437 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2255999999999999e-05 2020-06-12T02:38:57.437 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.437 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.438 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:57.438 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904605+0000", 2020-06-12T02:38:57.438 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013935858000000001 2020-06-12T02:38:57.438 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.438 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.439 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:57.439 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.918541+0000", 2020-06-12T02:38:57.439 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4961000000000004e-05 2020-06-12T02:38:57.439 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.439 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.440 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:57.440 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.918605+0000", 2020-06-12T02:38:57.440 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01412036 2020-06-12T02:38:57.440 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:57.440 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:57.440 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:57.441 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.441 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.441 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "pg_update_log_missing(3.3s2 epoch 62/53 rep_tid 6734 entries 62'1548 (0'0) error 3:ce45c70a:::smithi10729407-7:head by client.4512.0:18270 0.000000 -2 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 62'1546)", 2020-06-12T02:38:57.441 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:41.904668+0000", 2020-06-12T02:38:57.441 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 15.25444351, 2020-06-12T02:38:57.442 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01430155, 2020-06-12T02:38:57.442 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:57.442 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:57.442 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:57.442 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.443 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:57.443 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904668+0000", 2020-06-12T02:38:57.443 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:57.443 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.443 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.443 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:57.444 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904668+0000", 2020-06-12T02:38:57.444 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1882000000000001e-05 2020-06-12T02:38:57.444 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.444 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.444 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:57.445 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904730+0000", 2020-06-12T02:38:57.445 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0861e-05 2020-06-12T02:38:57.445 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.445 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.445 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:57.446 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904740+0000", 2020-06-12T02:38:57.446 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0406000000000001e-05 2020-06-12T02:38:57.446 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.446 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.446 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:57.446 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904761+0000", 2020-06-12T02:38:57.447 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0339999999999997e-06 2020-06-12T02:38:57.447 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.447 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.447 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:57.447 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904769+0000", 2020-06-12T02:38:57.448 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3888999999999995e-05 2020-06-12T02:38:57.448 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.448 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.448 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:57.448 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904863+0000", 2020-06-12T02:38:57.448 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014106478 2020-06-12T02:38:57.449 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.449 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.449 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:57.449 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.918969+0000", 2020-06-12T02:38:57.449 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01430155 2020-06-12T02:38:57.450 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:57.450 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:57.450 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:57.450 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.450 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.451 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.fs2 62/37 ECSubWrite(tid=3343, reqid=client.4512.0:18307, at_version=62'1623, trim_to=0'0, roll_forward_to=62'1622))", 2020-06-12T02:38:57.451 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:42.062050+0000", 2020-06-12T02:38:57.451 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 15.097060846, 2020-06-12T02:38:57.451 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019528429999999999, 2020-06-12T02:38:57.451 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:57.452 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:57.452 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:57.456 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.457 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:57.457 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062050+0000", 2020-06-12T02:38:57.457 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:57.457 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.458 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.458 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:57.458 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062050+0000", 2020-06-12T02:38:57.458 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00022828799999999999 2020-06-12T02:38:57.459 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.459 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.459 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:57.459 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062279+0000", 2020-06-12T02:38:57.459 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2115000000000001e-05 2020-06-12T02:38:57.460 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.460 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.460 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:57.460 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062301+0000", 2020-06-12T02:38:57.460 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9301e-05 2020-06-12T02:38:57.461 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.461 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.461 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:57.461 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062320+0000", 2020-06-12T02:38:57.462 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2811e-05 2020-06-12T02:38:57.462 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.462 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.462 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:57.462 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062333+0000", 2020-06-12T02:38:57.463 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8856e-05 2020-06-12T02:38:57.463 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.463 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.463 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:57.464 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062392+0000", 2020-06-12T02:38:57.465 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1246999999999999e-05 2020-06-12T02:38:57.465 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.465 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.465 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:57.466 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062403+0000", 2020-06-12T02:38:57.466 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019095523 2020-06-12T02:38:57.468 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.468 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.468 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:57.469 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.081498+0000", 2020-06-12T02:38:57.469 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0289000000000003e-05 2020-06-12T02:38:57.469 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.469 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.470 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:57.470 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.081579+0000", 2020-06-12T02:38:57.470 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019528429999999999 2020-06-12T02:38:57.470 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:57.470 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:57.471 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:57.471 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.471 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.471 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.fs2 62/37 ECSubWrite(tid=3344, reqid=client.4512.0:18308, at_version=62'1624, trim_to=0'0, roll_forward_to=62'1622))", 2020-06-12T02:38:57.471 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:42.064848+0000", 2020-06-12T02:38:57.472 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 15.094262861000001, 2020-06-12T02:38:57.472 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01677273, 2020-06-12T02:38:57.472 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:57.472 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:57.472 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:57.473 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.473 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:57.473 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.064848+0000", 2020-06-12T02:38:57.473 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:57.473 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.474 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.474 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:57.474 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.064848+0000", 2020-06-12T02:38:57.474 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000383655 2020-06-12T02:38:57.474 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.475 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.475 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:57.475 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.065232+0000", 2020-06-12T02:38:57.475 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3065999999999999e-05 2020-06-12T02:38:57.475 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.475 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.476 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:57.476 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.065255+0000", 2020-06-12T02:38:57.476 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7142999999999998e-05 2020-06-12T02:38:57.476 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.476 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.477 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:57.477 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.065272+0000", 2020-06-12T02:38:57.477 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7750000000000004e-06 2020-06-12T02:38:57.477 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.477 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.478 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:57.481 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.065279+0000", 2020-06-12T02:38:57.481 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00091758699999999998 2020-06-12T02:38:57.482 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.482 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.482 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:57.482 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.066197+0000", 2020-06-12T02:38:57.483 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4158999999999999e-05 2020-06-12T02:38:57.483 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.483 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.483 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:57.483 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.066211+0000", 2020-06-12T02:38:57.483 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015373370000000001 2020-06-12T02:38:57.484 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.484 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.484 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:57.484 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.081584+0000", 2020-06-12T02:38:57.484 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6974999999999998e-05 2020-06-12T02:38:57.485 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.485 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.485 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:57.485 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.081621+0000", 2020-06-12T02:38:57.485 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01677273 2020-06-12T02:38:57.486 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:57.486 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:57.486 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:57.486 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.486 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.487 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.3s2 62/53 ECSubWrite(tid=6737, reqid=client.4512.0:18319, at_version=62'1551, trim_to=0'0, roll_forward_to=62'1548))", 2020-06-12T02:38:57.487 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:42.089962+0000", 2020-06-12T02:38:57.487 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 15.069149223, 2020-06-12T02:38:57.487 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015432937000000001, 2020-06-12T02:38:57.487 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:38:57.487 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:38:57.488 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:38:57.488 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.488 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:38:57.488 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.089962+0000", 2020-06-12T02:38:57.488 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:38:57.489 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.489 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.489 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:38:57.489 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.089962+0000", 2020-06-12T02:38:57.489 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00068051099999999996 2020-06-12T02:38:57.490 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.490 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.490 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:38:57.490 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.090642+0000", 2020-06-12T02:38:57.490 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8766999999999998e-05 2020-06-12T02:38:57.491 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.491 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.491 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:38:57.491 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.090661+0000", 2020-06-12T02:38:57.491 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6461e-05 2020-06-12T02:38:57.492 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.492 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.492 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:38:57.493 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.090678+0000", 2020-06-12T02:38:57.493 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0580000000000003e-06 2020-06-12T02:38:57.493 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.494 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.494 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:38:57.494 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.090684+0000", 2020-06-12T02:38:57.495 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000104752 2020-06-12T02:38:57.495 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.495 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.495 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:38:57.495 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.090789+0000", 2020-06-12T02:38:57.495 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0767e-05 2020-06-12T02:38:57.496 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.496 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.496 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:38:57.496 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.090799+0000", 2020-06-12T02:38:57.496 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014539606 2020-06-12T02:38:57.497 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.497 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.497 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:38:57.497 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.105339+0000", 2020-06-12T02:38:57.497 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6014999999999998e-05 2020-06-12T02:38:57.498 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:38:57.498 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:38:57.498 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:38:57.498 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.105395+0000", 2020-06-12T02:38:57.498 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015432937000000001 2020-06-12T02:38:57.499 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:57.499 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:57.499 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:57.499 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:38:57.499 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:38:57.500 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:57.500 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-06-12T02:38:57.502 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-06-12T02:38:57.503 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:38:57.499+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:38:57.558 INFO:teuthology.orchestra.run.smithi107.stderr:set osd.5 primary-affinity to 1 (8655362) 2020-06-12T02:38:57.559 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2020-06-12T02:38:57.614 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:57.612+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:57.677 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:38:57.677 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:38:57.677 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2020-06-12T02:38:57.678 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:38:57.686 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-06-12T02:38:57.688 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:38:57.704 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:57.702+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:57.709 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:38:57.729 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:38:57.749 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:38:57.770 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:38:57.802 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:38:57.804 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:57.802+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:57.806 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:57.806 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:57.811 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:38:57.819 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-06-12T02:38:57.834 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2020-06-12T02:38:57.835 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-06-12T02:38:57.835 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-06-12T02:38:57.836 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-06-12T02:38:57.836 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-06-12T02:38:57.836 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-06-12T02:38:57.836 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-06-12T02:38:57.836 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-06-12T02:38:57.836 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-06-12T02:38:57.907 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:57.904+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:57.949 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:57.949 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:57.958 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-06-12T02:38:58.010 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:58.007+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:58.075 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:58.076 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:58.084 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-06-12T02:38:58.129 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:58.127+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:58.201 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:58.202 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:58.213 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:38:58.211+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:38:58.214 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-06-12T02:38:58.314 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:58.312+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:58.370 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:58.370 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:58.378 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-06-12T02:38:58.422 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:58.418+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:58.518 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:58.516+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:58.539 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:58.539 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:58.547 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-06-12T02:38:58.624 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:58.616+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:58.703 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:58.703 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:58.712 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-06-12T02:38:58.756 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:38:58.754+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:38:58.819 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:58.817+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:58.828 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:58.829 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:58.837 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-06-12T02:38:58.920 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:58.918+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:58.968 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:58.969 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:58.974 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-06-12T02:38:59.058 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:38:59.056+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:38:59.125 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:59.123+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:59.132 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:59.133 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:59.141 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-06-12T02:38:59.227 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:59.225+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:59.297 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:59.297 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:59.305 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-06-12T02:38:59.328 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:59.326+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:59.425 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:59.426 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:59.428 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:38:59.426+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:38:59.433 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-06-12T02:38:59.529 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:38:59.527+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:38:59.573 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd set noscrub 2020-06-12T02:38:59.593 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:59.593 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:59.598 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-06-12T02:38:59.645 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:59.643+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:59.717 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:59.717 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:59.726 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-06-12T02:38:59.766 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:38:59.764+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:38:59.832 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:38:59.831+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:38:59.848 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-06-12T02:38:59.882 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:38:59.883 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:38:59.891 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-06-12T02:38:59.935 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:38:59.933+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:38:59.981 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:39:00.002 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:39:00.009 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:00.009 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:00.019 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-06-12T02:39:00.021 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:39:00.034 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:39:00.032+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:39:00.040 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:39:00.061 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:39:00.080 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:39:00.100 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:39:00.121 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2020-06-12T02:39:00.122 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-06-12T02:39:00.123 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-06-12T02:39:00.123 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-06-12T02:39:00.123 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-06-12T02:39:00.124 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-06-12T02:39:00.124 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-06-12T02:39:00.124 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-06-12T02:39:00.124 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-06-12T02:39:00.135 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:39:00.133+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:39:00.183 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:39:00.183 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:39:00.183 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:39:00.184 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:39:00.184 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.184 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:4975 3.as0 3:5732055a:::smithi10729407-8:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e30)", 2020-06-12T02:39:00.184 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:43.721794+0000", 2020-06-12T02:39:00.184 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 76.457642273999994, 2020-06-12T02:39:00.185 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.1825135, 2020-06-12T02:39:00.185 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:00.185 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:00.185 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:00.185 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:00.186 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:00.186 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 4975 2020-06-12T02:39:00.186 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.186 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:00.186 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.187 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:00.187 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721794+0000", 2020-06-12T02:39:00.187 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:00.187 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.187 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.188 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:00.188 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721794+0000", 2020-06-12T02:39:00.188 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4542e-05 2020-06-12T02:39:00.188 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.188 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.189 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:00.189 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721818+0000", 2020-06-12T02:39:00.189 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9169999999999997e-06 2020-06-12T02:39:00.189 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.189 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.189 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:00.190 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721823+0000", 2020-06-12T02:39:00.190 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1184e-05 2020-06-12T02:39:00.190 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.190 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.190 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:00.191 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721834+0000", 2020-06-12T02:39:00.191 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.254e-06 2020-06-12T02:39:00.191 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.191 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.191 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:00.191 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721840+0000", 2020-06-12T02:39:00.192 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1075999999999998e-05 2020-06-12T02:39:00.192 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.192 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.192 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:00.192 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721881+0000", 2020-06-12T02:39:00.193 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7416999999999995e-05 2020-06-12T02:39:00.193 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.193 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.193 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:00.194 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721948+0000", 2020-06-12T02:39:00.194 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.18235910999999999 2020-06-12T02:39:00.195 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.195 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.195 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:00.195 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.904307+0000", 2020-06-12T02:39:00.195 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.1825135 2020-06-12T02:39:00.196 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.196 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:00.196 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.196 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.196 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.197 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11914 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e56)", 2020-06-12T02:39:00.197 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:23.508790+0000", 2020-06-12T02:39:00.197 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 36.670645422, 2020-06-12T02:39:00.197 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026541928999999999, 2020-06-12T02:39:00.198 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:00.198 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:00.198 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:00.198 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:00.199 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:00.199 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11914 2020-06-12T02:39:00.199 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.199 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:00.199 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.199 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:00.200 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508790+0000", 2020-06-12T02:39:00.200 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:00.200 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.200 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.200 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:00.201 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508790+0000", 2020-06-12T02:39:00.201 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0872999999999999e-05 2020-06-12T02:39:00.201 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.201 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.201 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:00.202 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508841+0000", 2020-06-12T02:39:00.202 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9179999999999994e-06 2020-06-12T02:39:00.203 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.203 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.203 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:00.203 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508849+0000", 2020-06-12T02:39:00.203 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5551999999999999e-05 2020-06-12T02:39:00.204 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.204 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.204 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:00.204 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508865+0000", 2020-06-12T02:39:00.204 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0629999999999996e-06 2020-06-12T02:39:00.204 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.205 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.205 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:00.205 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508870+0000", 2020-06-12T02:39:00.205 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010597488 2020-06-12T02:39:00.205 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.206 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.206 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:00.206 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.519467+0000", 2020-06-12T02:39:00.206 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.6083999999999997e-05 2020-06-12T02:39:00.206 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.207 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.207 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:00.207 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.519563+0000", 2020-06-12T02:39:00.207 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0078077800000000003 2020-06-12T02:39:00.208 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.208 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.208 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:00.209 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.527371+0000", 2020-06-12T02:39:00.209 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6198999999999993e-05 2020-06-12T02:39:00.209 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.209 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.209 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:00.210 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.527447+0000", 2020-06-12T02:39:00.211 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0078849720000000005 2020-06-12T02:39:00.211 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.211 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.211 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:00.212 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.535332+0000", 2020-06-12T02:39:00.212 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026541928999999999 2020-06-12T02:39:00.212 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.212 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:00.212 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.213 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.213 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.213 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13705 3.as0 3:5cb723d2:::smithi10729407-15:head [append 0~745472,stat] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:39:00.213 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.390499+0000", 2020-06-12T02:39:00.213 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 31.788936729, 2020-06-12T02:39:00.214 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.067319320000000002, 2020-06-12T02:39:00.214 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:00.214 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:00.214 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:00.214 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:00.215 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:00.215 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13705 2020-06-12T02:39:00.215 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.215 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:00.215 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.215 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:00.216 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.390499+0000", 2020-06-12T02:39:00.216 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:00.217 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.217 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.217 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:00.217 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.390499+0000", 2020-06-12T02:39:00.217 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00066732599999999999 2020-06-12T02:39:00.218 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.218 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.218 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:00.218 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391167+0000", 2020-06-12T02:39:00.218 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3609999999999997e-06 2020-06-12T02:39:00.218 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.219 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.219 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:00.219 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391173+0000", 2020-06-12T02:39:00.219 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0399e-05 2020-06-12T02:39:00.219 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.220 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.220 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:00.220 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391183+0000", 2020-06-12T02:39:00.220 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6480000000000002e-06 2020-06-12T02:39:00.220 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.221 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.221 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:00.221 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391188+0000", 2020-06-12T02:39:00.221 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4150999999999999e-05 2020-06-12T02:39:00.221 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.221 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.222 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:00.222 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391222+0000", 2020-06-12T02:39:00.222 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9450999999999996e-05 2020-06-12T02:39:00.222 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.222 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.223 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:00.223 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391292+0000", 2020-06-12T02:39:00.223 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054214300000000003 2020-06-12T02:39:00.223 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.223 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.224 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:00.224 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391834+0000", 2020-06-12T02:39:00.224 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0037312069999999998 2020-06-12T02:39:00.224 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.224 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.224 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:00.225 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.395565+0000", 2020-06-12T02:39:00.225 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001050194 2020-06-12T02:39:00.225 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.225 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.225 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:00.226 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396615+0000", 2020-06-12T02:39:00.226 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.061203439999999998 2020-06-12T02:39:00.226 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.226 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.227 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:00.227 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.457819+0000", 2020-06-12T02:39:00.227 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.067319320000000002 2020-06-12T02:39:00.227 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.227 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:00.228 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.228 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.228 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.228 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13706 3.as0 3:5cb723d2:::smithi10729407-15:head [append 0~434176] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:39:00.228 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.393133+0000", 2020-06-12T02:39:00.229 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 31.786302460999998, 2020-06-12T02:39:00.229 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.078158561000000001, 2020-06-12T02:39:00.229 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:00.229 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:00.229 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:00.230 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:00.230 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:00.230 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13706 2020-06-12T02:39:00.230 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.230 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:00.230 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.231 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:00.231 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393133+0000", 2020-06-12T02:39:00.231 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:00.231 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.231 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.232 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:00.232 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393133+0000", 2020-06-12T02:39:00.232 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00050415499999999995 2020-06-12T02:39:00.232 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.232 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.232 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:00.233 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393638+0000", 2020-06-12T02:39:00.233 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5010000000000004e-06 2020-06-12T02:39:00.233 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.233 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.233 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:00.234 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393646+0000", 2020-06-12T02:39:00.234 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4183e-05 2020-06-12T02:39:00.234 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.234 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.234 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:00.235 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393660+0000", 2020-06-12T02:39:00.235 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6069999999999996e-06 2020-06-12T02:39:00.235 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.235 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.235 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:00.235 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393667+0000", 2020-06-12T02:39:00.236 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001363569 2020-06-12T02:39:00.236 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.236 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.236 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:00.236 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.395030+0000", 2020-06-12T02:39:00.237 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012127 2020-06-12T02:39:00.237 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.237 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.237 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:00.238 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.395152+0000", 2020-06-12T02:39:00.238 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.063640503000000001 2020-06-12T02:39:00.238 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.238 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.238 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:00.239 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458792+0000", 2020-06-12T02:39:00.239 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010636 2020-06-12T02:39:00.239 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.239 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.239 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:00.239 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458899+0000", 2020-06-12T02:39:00.240 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00076074200000000002 2020-06-12T02:39:00.240 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.240 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.240 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:00.240 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459659+0000", 2020-06-12T02:39:00.241 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00772984 2020-06-12T02:39:00.241 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.241 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.241 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:00.241 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.467389+0000", 2020-06-12T02:39:00.241 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003815751 2020-06-12T02:39:00.242 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.242 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.242 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:00.242 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471205+0000", 2020-06-12T02:39:00.242 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.708e-05 2020-06-12T02:39:00.243 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.243 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.243 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:00.243 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471292+0000", 2020-06-12T02:39:00.243 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.078158561000000001 2020-06-12T02:39:00.243 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.244 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:00.244 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.244 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.244 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.244 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13707 3.as0 3:5cb723d2:::smithi10729407-15:head [append 0~565248] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:39:00.245 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.396173+0000", 2020-06-12T02:39:00.245 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 31.783262518000001, 2020-06-12T02:39:00.245 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.075298341000000005, 2020-06-12T02:39:00.245 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:00.245 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:00.246 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:00.246 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:00.246 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:00.246 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13707 2020-06-12T02:39:00.246 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.246 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:00.247 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.247 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:00.247 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396173+0000", 2020-06-12T02:39:00.247 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:00.248 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.248 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.248 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:00.248 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396173+0000", 2020-06-12T02:39:00.248 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.061779787000000003 2020-06-12T02:39:00.249 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.249 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.249 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:00.249 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.457953+0000", 2020-06-12T02:39:00.249 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5830000000000007e-06 2020-06-12T02:39:00.250 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.250 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.250 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:00.250 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.457962+0000", 2020-06-12T02:39:00.250 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3029e-05 2020-06-12T02:39:00.250 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.251 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.256 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:00.256 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.457975+0000", 2020-06-12T02:39:00.257 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.233e-06 2020-06-12T02:39:00.257 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.257 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.258 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:00.258 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.457979+0000", 2020-06-12T02:39:00.258 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0044352510000000003 2020-06-12T02:39:00.259 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.260 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.260 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:00.260 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.462414+0000", 2020-06-12T02:39:00.261 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012825799999999999 2020-06-12T02:39:00.261 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.261 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.261 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:00.261 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.462543+0000", 2020-06-12T02:39:00.262 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00077227899999999996 2020-06-12T02:39:00.262 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.262 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.262 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:00.263 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.463315+0000", 2020-06-12T02:39:00.263 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0079912390000000007 2020-06-12T02:39:00.263 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.263 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.264 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:00.264 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471306+0000", 2020-06-12T02:39:00.264 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000118674 2020-06-12T02:39:00.264 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.264 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.264 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:00.265 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471425+0000", 2020-06-12T02:39:00.265 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7008000000000001e-05 2020-06-12T02:39:00.265 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.265 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.265 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:00.266 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471472+0000", 2020-06-12T02:39:00.266 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.075298341000000005 2020-06-12T02:39:00.266 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.266 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:00.266 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.267 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.267 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.267 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWriteReply(3.as0 58/53 ECSubWriteReply(tid=1890, last_complete=58'1184, committed=1, applied=1))", 2020-06-12T02:39:00.267 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.396464+0000", 2020-06-12T02:39:00.267 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 31.782971638999999, 2020-06-12T02:39:00.268 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.062264803000000001, 2020-06-12T02:39:00.268 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:00.268 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:39:00.268 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:00.269 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.269 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:00.269 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396464+0000", 2020-06-12T02:39:00.269 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:00.270 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.270 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.271 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:00.271 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396464+0000", 2020-06-12T02:39:00.271 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9425999999999999e-05 2020-06-12T02:39:00.272 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.272 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.272 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:00.273 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396494+0000", 2020-06-12T02:39:00.273 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1610000000000002e-06 2020-06-12T02:39:00.273 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.273 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.273 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:00.274 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396499+0000", 2020-06-12T02:39:00.274 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2789999999999998e-06 2020-06-12T02:39:00.274 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.274 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.274 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:00.274 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396507+0000", 2020-06-12T02:39:00.275 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1389999999999997e-06 2020-06-12T02:39:00.275 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.275 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.275 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:00.275 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396511+0000", 2020-06-12T02:39:00.276 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2199000000000001e-05 2020-06-12T02:39:00.276 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.276 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.276 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:00.276 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396543+0000", 2020-06-12T02:39:00.277 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.062185599000000001 2020-06-12T02:39:00.277 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.277 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.277 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:00.277 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458729+0000", 2020-06-12T02:39:00.278 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.062264803000000001 2020-06-12T02:39:00.278 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.278 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:00.278 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.278 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.279 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.279 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13710 3.as0 3:5cb723d2:::smithi10729407-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e58)", 2020-06-12T02:39:00.279 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.458457+0000", 2020-06-12T02:39:00.279 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 31.720978468999999, 2020-06-12T02:39:00.279 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028102095000000001, 2020-06-12T02:39:00.281 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:00.281 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:00.281 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:00.282 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:00.282 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:00.282 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13710 2020-06-12T02:39:00.283 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.283 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:00.283 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.283 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:00.284 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458457+0000", 2020-06-12T02:39:00.284 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:00.284 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.284 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.284 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:00.285 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458457+0000", 2020-06-12T02:39:00.285 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7650000000000001e-05 2020-06-12T02:39:00.285 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.285 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.285 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:00.286 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458485+0000", 2020-06-12T02:39:00.286 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3210000000000001e-06 2020-06-12T02:39:00.286 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.286 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.286 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:00.287 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458488+0000", 2020-06-12T02:39:00.287 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5920000000000003e-06 2020-06-12T02:39:00.287 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.288 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.288 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:00.288 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458496+0000", 2020-06-12T02:39:00.289 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8789999999999999e-06 2020-06-12T02:39:00.289 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.289 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.289 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:00.289 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458499+0000", 2020-06-12T02:39:00.290 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013081152 2020-06-12T02:39:00.290 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.290 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.290 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:00.291 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471580+0000", 2020-06-12T02:39:00.291 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4975999999999994e-05 2020-06-12T02:39:00.291 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.291 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.292 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:00.292 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471665+0000", 2020-06-12T02:39:00.292 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0054254029999999996 2020-06-12T02:39:00.292 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.293 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.293 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:00.293 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.477090+0000", 2020-06-12T02:39:00.294 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2335000000000005e-05 2020-06-12T02:39:00.294 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.294 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.294 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:00.295 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.477163+0000", 2020-06-12T02:39:00.295 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0093967870000000002 2020-06-12T02:39:00.295 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.295 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.296 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:00.296 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486560+0000", 2020-06-12T02:39:00.296 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028102095000000001 2020-06-12T02:39:00.296 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.297 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:00.297 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.297 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.297 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.298 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13711 3.as0 3:5732055a:::smithi10729407-8:head [append 0~761856] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:39:00.298 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.458525+0000", 2020-06-12T02:39:00.298 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 31.720910697000001, 2020-06-12T02:39:00.299 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.027773760000000002, 2020-06-12T02:39:00.299 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:00.299 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:00.300 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:00.300 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:00.300 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:00.301 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13711 2020-06-12T02:39:00.301 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.301 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:00.302 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.302 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:00.302 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458525+0000", 2020-06-12T02:39:00.302 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:00.303 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.303 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.303 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:00.304 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458525+0000", 2020-06-12T02:39:00.304 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00089965200000000003 2020-06-12T02:39:00.304 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.304 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.305 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:00.305 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459425+0000", 2020-06-12T02:39:00.306 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4749999999999996e-06 2020-06-12T02:39:00.306 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.307 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.307 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:00.307 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459432+0000", 2020-06-12T02:39:00.307 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3043e-05 2020-06-12T02:39:00.308 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.308 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.308 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:00.308 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459445+0000", 2020-06-12T02:39:00.308 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0739999999999998e-06 2020-06-12T02:39:00.309 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.309 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.309 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:00.309 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459451+0000", 2020-06-12T02:39:00.309 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012238733999999999 2020-06-12T02:39:00.310 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.310 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.310 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:00.310 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471690+0000", 2020-06-12T02:39:00.310 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5640000000000003e-05 2020-06-12T02:39:00.311 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.311 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.311 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:00.311 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471746+0000", 2020-06-12T02:39:00.311 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00067859699999999997 2020-06-12T02:39:00.312 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.312 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.312 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:00.312 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.472424+0000", 2020-06-12T02:39:00.312 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050083230000000003 2020-06-12T02:39:00.313 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.313 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.313 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:00.313 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.477433+0000", 2020-06-12T02:39:00.313 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0087878039999999998 2020-06-12T02:39:00.313 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.314 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.314 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:00.314 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486221+0000", 2020-06-12T02:39:00.314 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8417999999999994e-05 2020-06-12T02:39:00.314 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.315 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.315 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:00.315 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486299+0000", 2020-06-12T02:39:00.315 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.027773760000000002 2020-06-12T02:39:00.315 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.315 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:00.316 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.316 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.316 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.316 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13712 3.as0 3:5732055a:::smithi10729407-8:head [append 0~712704] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:39:00.316 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.459508+0000", 2020-06-12T02:39:00.317 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 31.719927417000001, 2020-06-12T02:39:00.317 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.033995818999999997, 2020-06-12T02:39:00.317 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:00.317 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:00.317 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:00.318 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:00.318 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:00.318 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13712 2020-06-12T02:39:00.318 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.318 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:00.319 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.319 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:00.319 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459508+0000", 2020-06-12T02:39:00.319 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:00.319 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.319 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.320 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:00.320 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459508+0000", 2020-06-12T02:39:00.320 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00075436999999999998 2020-06-12T02:39:00.320 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.320 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.321 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:00.321 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460263+0000", 2020-06-12T02:39:00.321 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4370000000000001e-06 2020-06-12T02:39:00.321 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.321 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.322 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:00.322 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460270+0000", 2020-06-12T02:39:00.322 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2500000000000001e-05 2020-06-12T02:39:00.322 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.322 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.323 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:00.323 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460283+0000", 2020-06-12T02:39:00.323 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2580000000000003e-06 2020-06-12T02:39:00.323 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.323 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.324 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:00.324 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460288+0000", 2020-06-12T02:39:00.324 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017177595 2020-06-12T02:39:00.325 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.325 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.325 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:00.325 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.477466+0000", 2020-06-12T02:39:00.325 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00019751600000000001 2020-06-12T02:39:00.326 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.326 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.326 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:00.326 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.477663+0000", 2020-06-12T02:39:00.327 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00088697400000000001 2020-06-12T02:39:00.327 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.328 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.328 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:00.328 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.478550+0000", 2020-06-12T02:39:00.329 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.007578787 2020-06-12T02:39:00.329 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.329 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.330 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:00.330 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486129+0000", 2020-06-12T02:39:00.330 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0072972310000000004 2020-06-12T02:39:00.331 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.331 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.332 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:00.333 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493426+0000", 2020-06-12T02:39:00.334 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8151000000000004e-05 2020-06-12T02:39:00.334 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.334 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.334 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:00.335 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493504+0000", 2020-06-12T02:39:00.335 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.033995818999999997 2020-06-12T02:39:00.335 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.335 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:00.335 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.335 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.336 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.336 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13713 3.as0 3:5732055a:::smithi10729407-8:head [append 0~491520] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:39:00.336 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.460333+0000", 2020-06-12T02:39:00.336 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 31.719102961000001, 2020-06-12T02:39:00.336 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.033390145000000003, 2020-06-12T02:39:00.337 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:00.337 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:00.337 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:00.337 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:00.337 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:00.338 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13713 2020-06-12T02:39:00.338 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.338 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:00.338 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.338 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:00.338 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460333+0000", 2020-06-12T02:39:00.339 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:00.339 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.339 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.339 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:00.339 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460333+0000", 2020-06-12T02:39:00.340 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00051168999999999997 2020-06-12T02:39:00.340 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.340 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.340 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:00.340 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460845+0000", 2020-06-12T02:39:00.341 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2930000000000003e-06 2020-06-12T02:39:00.341 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.341 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.341 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:00.341 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460852+0000", 2020-06-12T02:39:00.342 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3054000000000001e-05 2020-06-12T02:39:00.342 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.342 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.342 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:00.342 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460865+0000", 2020-06-12T02:39:00.342 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6130000000000003e-06 2020-06-12T02:39:00.343 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.343 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.343 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:00.343 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460871+0000", 2020-06-12T02:39:00.343 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025753870000000002 2020-06-12T02:39:00.344 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.344 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.344 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:00.344 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486624+0000", 2020-06-12T02:39:00.344 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2739999999999998e-05 2020-06-12T02:39:00.345 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.345 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.345 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:00.345 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486697+0000", 2020-06-12T02:39:00.345 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00063578799999999998 2020-06-12T02:39:00.345 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.346 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.346 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:00.346 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.487333+0000", 2020-06-12T02:39:00.346 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0043011610000000004 2020-06-12T02:39:00.346 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.347 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.347 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:00.347 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.491634+0000", 2020-06-12T02:39:00.347 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0020130949999999999 2020-06-12T02:39:00.347 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.347 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.348 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:00.348 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493647+0000", 2020-06-12T02:39:00.348 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5840999999999994e-05 2020-06-12T02:39:00.348 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.348 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.349 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:00.349 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493723+0000", 2020-06-12T02:39:00.349 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.033390145000000003 2020-06-12T02:39:00.349 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.349 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:00.350 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.350 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.350 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.350 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13714 3.as0 3:5732055a:::smithi10729407-8:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:39:00.350 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.460921+0000", 2020-06-12T02:39:00.350 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 31.718514439, 2020-06-12T02:39:00.351 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.032981457999999998, 2020-06-12T02:39:00.351 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:00.351 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:00.351 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:00.351 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:00.352 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:00.352 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13714 2020-06-12T02:39:00.352 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.352 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:00.352 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.352 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:00.353 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460921+0000", 2020-06-12T02:39:00.353 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:00.353 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.353 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.353 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:00.354 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460921+0000", 2020-06-12T02:39:00.354 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6996999999999995e-05 2020-06-12T02:39:00.354 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.354 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.354 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:00.354 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460998+0000", 2020-06-12T02:39:00.355 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4529999999999999e-06 2020-06-12T02:39:00.355 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.355 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.355 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:00.355 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461005+0000", 2020-06-12T02:39:00.356 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1722000000000001e-05 2020-06-12T02:39:00.356 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.356 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.356 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:00.356 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461017+0000", 2020-06-12T02:39:00.356 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4839999999999998e-06 2020-06-12T02:39:00.357 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.357 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.357 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:00.357 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461021+0000", 2020-06-12T02:39:00.357 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.029157975999999999 2020-06-12T02:39:00.358 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.358 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.358 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:00.358 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.490179+0000", 2020-06-12T02:39:00.358 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010966700000000001 2020-06-12T02:39:00.359 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.359 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.359 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:00.359 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.490289+0000", 2020-06-12T02:39:00.359 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00032026800000000002 2020-06-12T02:39:00.360 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.360 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.360 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:00.361 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.490609+0000", 2020-06-12T02:39:00.361 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001102947 2020-06-12T02:39:00.361 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.361 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.362 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:00.362 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.491712+0000", 2020-06-12T02:39:00.362 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021050169999999998 2020-06-12T02:39:00.362 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.363 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.363 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:00.363 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493817+0000", 2020-06-12T02:39:00.363 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5927000000000006e-05 2020-06-12T02:39:00.364 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.364 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.364 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:00.365 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493903+0000", 2020-06-12T02:39:00.365 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.032981457999999998 2020-06-12T02:39:00.365 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.366 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:00.366 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.366 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.366 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.366 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13715 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e58)", 2020-06-12T02:39:00.367 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.461062+0000", 2020-06-12T02:39:00.367 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 31.718373798999998, 2020-06-12T02:39:00.367 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.035208338999999998, 2020-06-12T02:39:00.367 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:00.367 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:00.368 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:00.368 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:00.368 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:00.368 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13715 2020-06-12T02:39:00.369 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.369 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:00.369 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.369 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:00.369 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461062+0000", 2020-06-12T02:39:00.371 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:00.371 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.372 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.372 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:00.372 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461062+0000", 2020-06-12T02:39:00.373 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4839999999999998e-05 2020-06-12T02:39:00.373 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.373 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.374 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:00.374 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461107+0000", 2020-06-12T02:39:00.374 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4859999999999997e-06 2020-06-12T02:39:00.375 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.375 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.375 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:00.376 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461112+0000", 2020-06-12T02:39:00.376 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1742999999999999e-05 2020-06-12T02:39:00.376 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.377 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.377 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:00.377 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461124+0000", 2020-06-12T02:39:00.378 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7299999999999996e-06 2020-06-12T02:39:00.378 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.378 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.379 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:00.379 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461129+0000", 2020-06-12T02:39:00.380 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.030397368000000001 2020-06-12T02:39:00.380 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.380 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.380 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:00.380 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.491526+0000", 2020-06-12T02:39:00.381 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010076799999999999 2020-06-12T02:39:00.381 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.381 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.381 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:00.382 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.491627+0000", 2020-06-12T02:39:00.382 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031869849999999998 2020-06-12T02:39:00.382 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.382 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.382 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:00.383 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.494814+0000", 2020-06-12T02:39:00.383 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000108915 2020-06-12T02:39:00.383 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.383 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.383 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:00.383 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.494923+0000", 2020-06-12T02:39:00.384 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001347504 2020-06-12T02:39:00.384 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.384 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.384 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:00.384 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.496270+0000", 2020-06-12T02:39:00.385 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.035208338999999998 2020-06-12T02:39:00.385 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.385 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:00.385 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.385 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.386 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.386 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:16706 3.as0 3:5cb723d2:::smithi10729407-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e61)", 2020-06-12T02:39:00.386 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:37.289063+0000", 2020-06-12T02:39:00.386 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 22.890372774999999, 2020-06-12T02:39:00.386 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026146086999999998, 2020-06-12T02:39:00.387 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:00.387 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:00.387 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:00.387 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:00.387 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:00.387 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 16706 2020-06-12T02:39:00.388 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.388 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:00.388 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.389 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:00.389 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289063+0000", 2020-06-12T02:39:00.389 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:00.389 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.389 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.390 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:00.390 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289063+0000", 2020-06-12T02:39:00.390 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3776000000000002e-05 2020-06-12T02:39:00.390 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.391 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.391 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:00.391 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289107+0000", 2020-06-12T02:39:00.391 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7039999999999999e-06 2020-06-12T02:39:00.391 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.392 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.392 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:00.392 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289113+0000", 2020-06-12T02:39:00.392 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3259e-05 2020-06-12T02:39:00.393 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.393 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.394 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:00.394 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289126+0000", 2020-06-12T02:39:00.394 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1999999999999996e-06 2020-06-12T02:39:00.395 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.395 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.395 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:00.395 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289130+0000", 2020-06-12T02:39:00.395 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0010912820000000001 2020-06-12T02:39:00.396 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.396 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.396 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:00.396 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.290221+0000", 2020-06-12T02:39:00.396 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3166e-05 2020-06-12T02:39:00.397 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.397 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.397 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:00.397 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.290265+0000", 2020-06-12T02:39:00.398 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015984187 2020-06-12T02:39:00.398 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.398 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.399 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:00.399 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.306249+0000", 2020-06-12T02:39:00.399 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4623000000000006e-05 2020-06-12T02:39:00.399 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.400 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.400 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:00.400 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.306333+0000", 2020-06-12T02:39:00.402 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0068573690000000003 2020-06-12T02:39:00.402 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.402 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.403 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:00.403 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.313191+0000", 2020-06-12T02:39:00.403 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7804000000000002e-05 2020-06-12T02:39:00.404 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.404 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.404 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:00.405 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.313279+0000", 2020-06-12T02:39:00.405 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001930717 2020-06-12T02:39:00.405 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.406 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.406 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:00.406 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.315209+0000", 2020-06-12T02:39:00.407 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026146086999999998 2020-06-12T02:39:00.408 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.408 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:00.408 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.408 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.409 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.409 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18051 3.as0 3:5cb723d2:::smithi10729407-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:39:00.409 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.329078+0000", 2020-06-12T02:39:00.409 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 18.850358225000001, 2020-06-12T02:39:00.410 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026465103, 2020-06-12T02:39:00.410 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:00.410 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:00.410 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:00.410 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:00.411 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:00.411 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18051 2020-06-12T02:39:00.411 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.411 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:00.411 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.412 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:00.412 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329078+0000", 2020-06-12T02:39:00.412 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:00.412 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.412 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.413 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:00.413 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329078+0000", 2020-06-12T02:39:00.413 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5392000000000001e-05 2020-06-12T02:39:00.413 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.413 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.413 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:00.414 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329133+0000", 2020-06-12T02:39:00.414 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5329999999999997e-06 2020-06-12T02:39:00.414 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.414 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.415 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:00.415 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329141+0000", 2020-06-12T02:39:00.415 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3459999999999999e-05 2020-06-12T02:39:00.416 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.416 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.416 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:00.416 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329164+0000", 2020-06-12T02:39:00.417 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8170000000000001e-06 2020-06-12T02:39:00.417 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.417 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.418 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:00.418 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329171+0000", 2020-06-12T02:39:00.418 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013005208000000001 2020-06-12T02:39:00.418 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.419 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.419 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:00.419 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.342176+0000", 2020-06-12T02:39:00.419 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3560999999999998e-05 2020-06-12T02:39:00.420 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.420 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.420 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:00.420 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.342230+0000", 2020-06-12T02:39:00.420 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0080818490000000003 2020-06-12T02:39:00.421 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.421 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.421 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:00.421 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.350312+0000", 2020-06-12T02:39:00.421 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0167000000000006e-05 2020-06-12T02:39:00.422 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.422 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.422 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:00.422 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.350402+0000", 2020-06-12T02:39:00.423 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051411160000000003 2020-06-12T02:39:00.423 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.423 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.423 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:00.424 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.355543+0000", 2020-06-12T02:39:00.424 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026465103 2020-06-12T02:39:00.424 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.424 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:00.424 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.424 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.425 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.425 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18053 3.as0 3:5732055a:::smithi10729407-8:head [append 0~434176] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:00.425 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.334425+0000", 2020-06-12T02:39:00.425 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 18.845010658, 2020-06-12T02:39:00.425 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.029616703000000001, 2020-06-12T02:39:00.426 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:00.426 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:00.426 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:00.426 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:00.426 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:00.427 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18053 2020-06-12T02:39:00.427 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.427 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:00.427 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.427 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:00.428 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334425+0000", 2020-06-12T02:39:00.428 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:00.428 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.428 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.428 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:00.428 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334425+0000", 2020-06-12T02:39:00.429 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047283900000000002 2020-06-12T02:39:00.429 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.429 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.429 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:00.429 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334898+0000", 2020-06-12T02:39:00.430 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9100000000000004e-06 2020-06-12T02:39:00.430 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.430 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.430 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:00.430 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334903+0000", 2020-06-12T02:39:00.430 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4260000000000003e-06 2020-06-12T02:39:00.431 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.431 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.431 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:00.431 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334911+0000", 2020-06-12T02:39:00.431 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1060000000000002e-06 2020-06-12T02:39:00.432 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.432 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.432 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:00.432 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334916+0000", 2020-06-12T02:39:00.432 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010838316000000001 2020-06-12T02:39:00.433 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.433 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.433 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:00.433 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.345754+0000", 2020-06-12T02:39:00.433 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000103872 2020-06-12T02:39:00.433 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.434 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.434 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:00.434 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.345858+0000", 2020-06-12T02:39:00.434 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0059798739999999996 2020-06-12T02:39:00.435 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.435 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.435 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:00.435 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.351838+0000", 2020-06-12T02:39:00.435 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000104724 2020-06-12T02:39:00.436 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.436 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.436 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:00.436 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.351942+0000", 2020-06-12T02:39:00.436 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000572687 2020-06-12T02:39:00.437 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.437 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.437 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:00.437 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.352515+0000", 2020-06-12T02:39:00.437 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002905641 2020-06-12T02:39:00.438 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.438 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.438 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:00.438 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.355421+0000", 2020-06-12T02:39:00.438 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0085358350000000003 2020-06-12T02:39:00.438 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.439 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.439 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:00.439 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.363956+0000", 2020-06-12T02:39:00.439 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5473e-05 2020-06-12T02:39:00.439 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.440 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.440 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:00.440 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.364042+0000", 2020-06-12T02:39:00.440 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.029616703000000001 2020-06-12T02:39:00.440 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.440 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:00.441 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.441 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.441 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.441 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18054 3.as0 3:5732055a:::smithi10729407-8:head [append 0~638976] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:00.441 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.338171+0000", 2020-06-12T02:39:00.442 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 18.841265154999999, 2020-06-12T02:39:00.442 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.030067120999999999, 2020-06-12T02:39:00.442 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:00.442 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:00.442 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:00.442 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:00.443 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:00.443 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18054 2020-06-12T02:39:00.443 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.443 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:00.443 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.444 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:00.444 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338171+0000", 2020-06-12T02:39:00.444 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:00.444 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.444 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.444 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:00.445 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338171+0000", 2020-06-12T02:39:00.445 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00068138900000000002 2020-06-12T02:39:00.445 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.445 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.445 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:00.446 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338852+0000", 2020-06-12T02:39:00.446 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.767e-06 2020-06-12T02:39:00.446 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.446 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.446 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:00.447 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338858+0000", 2020-06-12T02:39:00.447 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8999999999999995e-06 2020-06-12T02:39:00.447 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.447 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.447 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:00.447 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338867+0000", 2020-06-12T02:39:00.448 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.262e-06 2020-06-12T02:39:00.448 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.448 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.448 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:00.448 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338871+0000", 2020-06-12T02:39:00.449 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0074839060000000002 2020-06-12T02:39:00.449 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.449 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.449 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:00.449 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346355+0000", 2020-06-12T02:39:00.449 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1435999999999999e-05 2020-06-12T02:39:00.450 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.450 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.450 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:00.450 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346416+0000", 2020-06-12T02:39:00.450 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0092109879999999998 2020-06-12T02:39:00.451 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.451 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.451 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:00.451 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.355627+0000", 2020-06-12T02:39:00.451 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7355999999999998e-05 2020-06-12T02:39:00.451 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.452 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.452 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:00.452 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.355685+0000", 2020-06-12T02:39:00.452 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00074711999999999999 2020-06-12T02:39:00.452 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.453 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.453 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:00.453 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.356432+0000", 2020-06-12T02:39:00.453 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010659572000000001 2020-06-12T02:39:00.453 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.454 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.454 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:00.454 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.367091+0000", 2020-06-12T02:39:00.454 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0010713210000000001 2020-06-12T02:39:00.454 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.454 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.455 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:00.455 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.368163+0000", 2020-06-12T02:39:00.455 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5104000000000005e-05 2020-06-12T02:39:00.455 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.455 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.456 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:00.456 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.368238+0000", 2020-06-12T02:39:00.456 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.030067120999999999 2020-06-12T02:39:00.456 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.456 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:00.456 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.457 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.457 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.457 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18055 3.as0 3:5732055a:::smithi10729407-8:head [append 0~483328] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:00.457 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.340740+0000", 2020-06-12T02:39:00.457 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 18.838696305999999, 2020-06-12T02:39:00.458 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028669050000000001, 2020-06-12T02:39:00.458 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:00.458 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:00.458 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:00.458 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:00.459 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:00.459 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18055 2020-06-12T02:39:00.459 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.459 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:00.459 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.459 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:00.460 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.340740+0000", 2020-06-12T02:39:00.460 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:00.460 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.460 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.460 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:00.461 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.340740+0000", 2020-06-12T02:39:00.461 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00045126900000000002 2020-06-12T02:39:00.461 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.461 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.461 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:00.462 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341191+0000", 2020-06-12T02:39:00.462 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5630000000000003e-06 2020-06-12T02:39:00.462 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.462 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.462 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:00.462 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341198+0000", 2020-06-12T02:39:00.463 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4467999999999999e-05 2020-06-12T02:39:00.463 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.463 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.463 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:00.463 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341213+0000", 2020-06-12T02:39:00.464 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7540000000000003e-06 2020-06-12T02:39:00.464 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.464 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.464 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:00.464 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341219+0000", 2020-06-12T02:39:00.464 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0052272680000000002 2020-06-12T02:39:00.465 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.465 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.465 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:00.465 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346446+0000", 2020-06-12T02:39:00.465 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4742999999999998e-05 2020-06-12T02:39:00.466 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.466 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.466 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:00.466 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346531+0000", 2020-06-12T02:39:00.466 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013203138999999999 2020-06-12T02:39:00.467 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.467 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.467 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:00.467 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.359734+0000", 2020-06-12T02:39:00.467 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000100277 2020-06-12T02:39:00.468 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.468 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.468 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:00.468 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.359834+0000", 2020-06-12T02:39:00.468 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054998099999999995 2020-06-12T02:39:00.468 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.469 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.469 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:00.469 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.360384+0000", 2020-06-12T02:39:00.469 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0067087179999999998 2020-06-12T02:39:00.469 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.470 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.470 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:00.470 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.367093+0000", 2020-06-12T02:39:00.470 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0022746020000000001 2020-06-12T02:39:00.470 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.471 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.471 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:00.471 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369367+0000", 2020-06-12T02:39:00.471 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1267999999999999e-05 2020-06-12T02:39:00.471 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.471 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.472 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:00.472 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369409+0000", 2020-06-12T02:39:00.472 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028669050000000001 2020-06-12T02:39:00.472 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.472 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:00.473 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.473 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.473 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.473 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18056 3.as0 3:5732055a:::smithi10729407-8:head [append 0~49152] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:00.473 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.341270+0000", 2020-06-12T02:39:00.474 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 18.838165517, 2020-06-12T02:39:00.474 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028350462999999999, 2020-06-12T02:39:00.474 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:00.474 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:00.474 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:00.474 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:00.475 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:00.475 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18056 2020-06-12T02:39:00.475 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.475 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:00.475 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.476 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:00.477 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341270+0000", 2020-06-12T02:39:00.477 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:00.477 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.477 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.477 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:00.478 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341270+0000", 2020-06-12T02:39:00.478 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000114082 2020-06-12T02:39:00.478 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.478 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.478 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:00.479 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341384+0000", 2020-06-12T02:39:00.479 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3289999999999999e-06 2020-06-12T02:39:00.479 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.479 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.479 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:00.479 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341391+0000", 2020-06-12T02:39:00.480 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3339999999999999e-05 2020-06-12T02:39:00.480 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.480 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.480 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:00.480 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341404+0000", 2020-06-12T02:39:00.481 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4e-06 2020-06-12T02:39:00.481 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.481 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.481 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:00.481 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341410+0000", 2020-06-12T02:39:00.482 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051518600000000003 2020-06-12T02:39:00.482 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.482 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.482 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:00.483 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346561+0000", 2020-06-12T02:39:00.483 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1662999999999994e-05 2020-06-12T02:39:00.483 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.483 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.483 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:00.484 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346643+0000", 2020-06-12T02:39:00.484 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017471605000000001 2020-06-12T02:39:00.484 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.484 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.484 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:00.484 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.364115+0000", 2020-06-12T02:39:00.485 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011059400000000001 2020-06-12T02:39:00.485 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.485 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.485 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:00.485 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.364225+0000", 2020-06-12T02:39:00.486 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000611262 2020-06-12T02:39:00.486 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.486 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.486 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:00.486 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.364837+0000", 2020-06-12T02:39:00.486 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023767710000000002 2020-06-12T02:39:00.487 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.487 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.487 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:00.487 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.367213+0000", 2020-06-12T02:39:00.487 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023325479999999998 2020-06-12T02:39:00.488 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.488 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.488 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:00.488 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369546+0000", 2020-06-12T02:39:00.488 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5009e-05 2020-06-12T02:39:00.489 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.489 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.489 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:00.489 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369621+0000", 2020-06-12T02:39:00.489 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028350462999999999 2020-06-12T02:39:00.489 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.490 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:00.490 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.490 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.490 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.490 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18057 3.as0 3:5732055a:::smithi10729407-8:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:00.491 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.341456+0000", 2020-06-12T02:39:00.491 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 18.837980356999999, 2020-06-12T02:39:00.491 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028443018, 2020-06-12T02:39:00.491 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:00.491 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:00.491 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:00.492 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:00.492 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:00.492 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18057 2020-06-12T02:39:00.492 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.492 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:00.493 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.493 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:00.493 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341456+0000", 2020-06-12T02:39:00.493 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:00.493 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.494 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.494 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:00.494 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341456+0000", 2020-06-12T02:39:00.494 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6036000000000003e-05 2020-06-12T02:39:00.494 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.494 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.495 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:00.495 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341522+0000", 2020-06-12T02:39:00.495 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2210000000000002e-06 2020-06-12T02:39:00.495 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.495 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.496 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:00.496 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341528+0000", 2020-06-12T02:39:00.496 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0338e-05 2020-06-12T02:39:00.496 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.496 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.496 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:00.497 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341538+0000", 2020-06-12T02:39:00.497 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8639999999999998e-06 2020-06-12T02:39:00.497 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.497 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.497 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:00.498 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341542+0000", 2020-06-12T02:39:00.498 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051405009999999996 2020-06-12T02:39:00.498 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.498 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.498 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:00.498 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346683+0000", 2020-06-12T02:39:00.499 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9561999999999995e-05 2020-06-12T02:39:00.499 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.499 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.499 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:00.499 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346762+0000", 2020-06-12T02:39:00.500 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019267197999999999 2020-06-12T02:39:00.500 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.500 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.500 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:00.500 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.366029+0000", 2020-06-12T02:39:00.500 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2634999999999997e-05 2020-06-12T02:39:00.501 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.501 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.501 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:00.501 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.366122+0000", 2020-06-12T02:39:00.501 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00026907500000000002 2020-06-12T02:39:00.502 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.502 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.502 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:00.502 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.366391+0000", 2020-06-12T02:39:00.502 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0018939409999999999 2020-06-12T02:39:00.502 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.503 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.503 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:00.503 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.368285+0000", 2020-06-12T02:39:00.503 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001549405 2020-06-12T02:39:00.503 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.504 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.504 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:00.504 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369834+0000", 2020-06-12T02:39:00.504 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4241999999999999e-05 2020-06-12T02:39:00.504 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.504 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.505 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:00.505 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369899+0000", 2020-06-12T02:39:00.505 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028443018 2020-06-12T02:39:00.505 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.505 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:00.506 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.506 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.506 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.506 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18058 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:39:00.506 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.341580+0000", 2020-06-12T02:39:00.506 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 18.837855449999999, 2020-06-12T02:39:00.507 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.030449559000000001, 2020-06-12T02:39:00.507 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:00.507 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:00.507 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:00.507 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:00.508 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:00.508 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18058 2020-06-12T02:39:00.508 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.508 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:00.508 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.508 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:00.509 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341580+0000", 2020-06-12T02:39:00.509 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:00.509 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.509 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.509 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:00.510 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341580+0000", 2020-06-12T02:39:00.510 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1185e-05 2020-06-12T02:39:00.510 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.510 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.510 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:00.511 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341622+0000", 2020-06-12T02:39:00.511 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1599999999999997e-06 2020-06-12T02:39:00.511 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.511 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.511 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:00.511 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341627+0000", 2020-06-12T02:39:00.512 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3546e-05 2020-06-12T02:39:00.512 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.512 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.512 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:00.512 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341640+0000", 2020-06-12T02:39:00.513 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9779999999999999e-06 2020-06-12T02:39:00.513 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.513 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.513 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:00.513 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341644+0000", 2020-06-12T02:39:00.513 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051433700000000004 2020-06-12T02:39:00.514 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.514 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.514 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:00.514 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346788+0000", 2020-06-12T02:39:00.514 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1622000000000001e-05 2020-06-12T02:39:00.515 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.515 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.515 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:00.515 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346839+0000", 2020-06-12T02:39:00.515 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020283958000000001 2020-06-12T02:39:00.515 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.516 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.516 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:00.516 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.367123+0000", 2020-06-12T02:39:00.516 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1997999999999997e-05 2020-06-12T02:39:00.516 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.517 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.517 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:00.517 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.367205+0000", 2020-06-12T02:39:00.517 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036875440000000001 2020-06-12T02:39:00.517 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.517 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.518 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:00.518 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.370893+0000", 2020-06-12T02:39:00.518 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000100604 2020-06-12T02:39:00.518 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.518 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.519 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:00.519 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.370993+0000", 2020-06-12T02:39:00.519 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0010365940000000001 2020-06-12T02:39:00.519 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.519 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.519 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:00.520 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.372030+0000", 2020-06-12T02:39:00.520 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.030449559000000001 2020-06-12T02:39:00.520 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.520 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:00.520 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.521 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.521 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:00.521 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:39:00.522 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-06-12T02:39:00.523 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:39:00.520+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:39:00.564 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2020-06-12T02:39:00.623 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:39:00.621+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:39:00.684 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:39:00.684 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:39:00.684 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.684 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:19948 3.15s0 3:a9510bda:::smithi10729407-14:head [stat,delete] snapc 0=[] ondisk+write+known_if_redirected e71)", 2020-06-12T02:39:00.685 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:51.564097+0000", 2020-06-12T02:39:00.685 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.1178782330000008, 2020-06-12T02:39:00.685 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1178946540000005, 2020-06-12T02:39:00.685 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:00.685 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "delayed", 2020-06-12T02:39:00.686 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:00.686 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:00.686 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:00.686 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 19948 2020-06-12T02:39:00.686 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.687 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:00.687 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.687 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:00.687 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:51.564097+0000", 2020-06-12T02:39:00.687 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:00.687 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.688 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.688 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:00.688 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:51.564097+0000", 2020-06-12T02:39:00.688 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3070000000000001e-05 2020-06-12T02:39:00.688 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.689 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.689 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:00.689 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:51.564120+0000", 2020-06-12T02:39:00.689 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.656e-06 2020-06-12T02:39:00.689 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.689 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.690 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:00.690 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:51.564125+0000", 2020-06-12T02:39:00.690 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.14e-06 2020-06-12T02:39:00.690 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.690 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.691 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:00.691 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:51.564133+0000", 2020-06-12T02:39:00.691 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5240000000000002e-06 2020-06-12T02:39:00.691 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.691 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.691 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:00.692 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:51.564137+0000", 2020-06-12T02:39:00.692 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0881000000000003e-05 2020-06-12T02:39:00.692 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.692 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.693 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:00.693 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:51.564188+0000", 2020-06-12T02:39:00.693 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5320000000000001e-05 2020-06-12T02:39:00.693 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.693 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.694 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for readable", 2020-06-12T02:39:00.694 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:51.564223+0000", 2020-06-12T02:39:00.694 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012659100000000001 2020-06-12T02:39:00.694 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.694 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:00.695 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.695 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.695 INFO:teuthology.orchestra.run.smithi188.stdout: ], 2020-06-12T02:39:00.695 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 1 2020-06-12T02:39:00.695 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:39:00.697 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-06-12T02:39:00.724 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:39:00.722+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:39:00.812 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:39:00.813 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:39:00.813 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:39:00.814 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:39:00.814 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:39:00.823 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-06-12T02:39:00.826 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:39:00.824+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:39:00.927 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:39:00.925+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:39:00.942 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:39:00.943 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:39:00.943 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:39:00.943 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:39:00.944 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.944 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8464 3.fs0 3.57bc3c1f (undecoded) ondisk+retry+write+known_if_redirected e34)", 2020-06-12T02:39:00.944 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:53.549364+0000", 2020-06-12T02:39:00.945 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 67.388756423000004, 2020-06-12T02:39:00.945 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.003107494, 2020-06-12T02:39:00.945 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:00.946 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:39:00.946 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:00.946 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:00.947 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:00.947 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8464 2020-06-12T02:39:00.947 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.947 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:00.948 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.948 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:00.948 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.549364+0000", 2020-06-12T02:39:00.948 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:00.949 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.950 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.950 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:00.950 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.549364+0000", 2020-06-12T02:39:00.951 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023109609999999998 2020-06-12T02:39:00.951 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.951 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.952 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:00.952 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551675+0000", 2020-06-12T02:39:00.952 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3331e-05 2020-06-12T02:39:00.952 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.953 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.953 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:00.953 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551688+0000", 2020-06-12T02:39:00.954 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4173e-05 2020-06-12T02:39:00.954 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.954 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.955 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:00.955 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551703+0000", 2020-06-12T02:39:00.955 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0454e-05 2020-06-12T02:39:00.956 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.956 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.956 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:00.957 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551713+0000", 2020-06-12T02:39:00.957 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031069050000000001 2020-06-12T02:39:00.957 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.958 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.958 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:00.958 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554820+0000", 2020-06-12T02:39:00.961 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2820000000000002e-06 2020-06-12T02:39:00.962 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.962 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.963 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2020-06-12T02:39:00.963 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554826+0000", 2020-06-12T02:39:00.963 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99762648499999995 2020-06-12T02:39:00.964 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.964 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.964 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:00.965 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552453+0000", 2020-06-12T02:39:00.965 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8902999999999998e-05 2020-06-12T02:39:00.965 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.966 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.966 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:00.966 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552472+0000", 2020-06-12T02:39:00.966 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.003107494 2020-06-12T02:39:00.967 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.967 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:00.967 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.968 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.968 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.968 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8465 3.fs0 3.57bc3c1f (undecoded) ondisk+retry+write+known_if_redirected e34)", 2020-06-12T02:39:00.969 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:53.551774+0000", 2020-06-12T02:39:00.969 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 67.386346786999994, 2020-06-12T02:39:00.969 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.000741442, 2020-06-12T02:39:00.970 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:00.970 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:39:00.970 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:00.971 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:00.971 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:00.971 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8465 2020-06-12T02:39:00.974 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.974 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:00.975 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.975 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:00.975 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551774+0000", 2020-06-12T02:39:00.976 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:00.976 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.976 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.977 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:00.977 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551774+0000", 2020-06-12T02:39:00.977 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0010760539999999999 2020-06-12T02:39:00.978 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.978 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.978 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:00.979 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552850+0000", 2020-06-12T02:39:00.979 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0139000000000001e-05 2020-06-12T02:39:00.979 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.980 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.980 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:00.981 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552860+0000", 2020-06-12T02:39:00.981 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7646999999999999e-05 2020-06-12T02:39:00.981 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.982 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.982 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:00.982 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552878+0000", 2020-06-12T02:39:00.982 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2916e-05 2020-06-12T02:39:00.983 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.983 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.983 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:00.983 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552890+0000", 2020-06-12T02:39:00.984 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0019574610000000002 2020-06-12T02:39:00.984 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.984 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.985 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:00.985 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554848+0000", 2020-06-12T02:39:00.986 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0399999999999995e-07 2020-06-12T02:39:00.986 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.987 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.987 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2020-06-12T02:39:00.987 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554849+0000", 2020-06-12T02:39:00.988 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99765586100000003 2020-06-12T02:39:00.988 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.988 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.989 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:00.989 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552504+0000", 2020-06-12T02:39:00.989 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.066e-05 2020-06-12T02:39:00.990 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.990 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.991 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:00.991 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552515+0000", 2020-06-12T02:39:00.991 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.000741442 2020-06-12T02:39:00.992 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.992 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:00.992 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:00.993 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.993 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.993 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8466 3.fs0 3.57bc3c1f (undecoded) ondisk+write+known_if_redirected e34)", 2020-06-12T02:39:00.993 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:53.552953+0000", 2020-06-12T02:39:00.994 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 67.385167507000006, 2020-06-12T02:39:00.994 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99960144299999998, 2020-06-12T02:39:00.994 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:00.995 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:39:00.995 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:00.995 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:00.996 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:00.996 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8466 2020-06-12T02:39:00.996 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.996 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:00.998 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:00.998 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:00.999 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552953+0000", 2020-06-12T02:39:00.999 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:00.999 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:00.999 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.000 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:01.000 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552953+0000", 2020-06-12T02:39:01.000 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013847100000000001 2020-06-12T02:39:01.001 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.001 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.001 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:01.002 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553091+0000", 2020-06-12T02:39:01.002 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7190000000000004e-06 2020-06-12T02:39:01.002 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.003 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.003 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:01.003 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553098+0000", 2020-06-12T02:39:01.004 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.171e-05 2020-06-12T02:39:01.004 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.004 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.005 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:01.005 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553110+0000", 2020-06-12T02:39:01.006 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5770000000000001e-06 2020-06-12T02:39:01.006 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.007 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.008 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:01.008 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553115+0000", 2020-06-12T02:39:01.008 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0017512739999999999 2020-06-12T02:39:01.009 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.009 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.009 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:01.009 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554867+0000", 2020-06-12T02:39:01.010 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1200000000000003e-07 2020-06-12T02:39:01.010 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.010 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.011 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2020-06-12T02:39:01.011 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554867+0000", 2020-06-12T02:39:01.011 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99767707100000003 2020-06-12T02:39:01.012 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.012 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.012 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:01.012 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552544+0000", 2020-06-12T02:39:01.013 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0009e-05 2020-06-12T02:39:01.013 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.013 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.014 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:01.014 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552554+0000", 2020-06-12T02:39:01.014 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99960144299999998 2020-06-12T02:39:01.015 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:01.015 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:01.015 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:01.016 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.016 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.017 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8467 3.fs0 3.57bc3c1f (undecoded) ondisk+write+known_if_redirected e34)", 2020-06-12T02:39:01.017 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:53.553172+0000", 2020-06-12T02:39:01.018 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 67.384948291000001, 2020-06-12T02:39:01.018 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99942086200000002, 2020-06-12T02:39:01.018 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:01.018 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:39:01.019 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:01.019 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:01.019 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:01.019 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8467 2020-06-12T02:39:01.020 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.020 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:01.020 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.021 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:01.021 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553172+0000", 2020-06-12T02:39:01.022 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:01.022 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.023 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.023 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:01.024 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553172+0000", 2020-06-12T02:39:01.024 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5913000000000003e-05 2020-06-12T02:39:01.024 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.025 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.025 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:01.026 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553248+0000", 2020-06-12T02:39:01.026 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7740000000000001e-06 2020-06-12T02:39:01.026 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.026 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.026 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:01.027 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553254+0000", 2020-06-12T02:39:01.027 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1494999999999999e-05 2020-06-12T02:39:01.027 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.027 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.027 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:01.027 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553265+0000", 2020-06-12T02:39:01.028 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5170000000000001e-06 2020-06-12T02:39:01.028 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.028 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.028 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:01.028 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553270+0000", 2020-06-12T02:39:01.029 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001616567 2020-06-12T02:39:01.029 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.029 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.029 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:01.029 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554886+0000", 2020-06-12T02:39:01.030 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7800000000000001e-07 2020-06-12T02:39:01.030 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.030 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.030 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2020-06-12T02:39:01.030 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554887+0000", 2020-06-12T02:39:01.030 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99769652399999997 2020-06-12T02:39:01.031 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.031 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.031 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:01.031 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552584+0000", 2020-06-12T02:39:01.031 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4939999999999992e-06 2020-06-12T02:39:01.032 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.032 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.032 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:01.033 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552593+0000", 2020-06-12T02:39:01.033 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99942086200000002 2020-06-12T02:39:01.033 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:01.033 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:01.033 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:01.033 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.034 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.034 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8468 3.fs0 3.57bc3c1f (undecoded) ondisk+read+rwordered+known_if_redirected e34)", 2020-06-12T02:39:01.034 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:53.553311+0000", 2020-06-12T02:39:01.034 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 67.384809692999994, 2020-06-12T02:39:01.034 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99988704799999994, 2020-06-12T02:39:01.035 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:01.035 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:39:01.035 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:01.035 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:01.035 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:01.036 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8468 2020-06-12T02:39:01.036 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.036 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:01.036 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.036 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:01.036 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553311+0000", 2020-06-12T02:39:01.037 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:01.037 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.037 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.037 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:01.037 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553311+0000", 2020-06-12T02:39:01.038 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2050000000000001e-05 2020-06-12T02:39:01.038 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.038 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.038 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:01.038 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553333+0000", 2020-06-12T02:39:01.038 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5489999999999998e-06 2020-06-12T02:39:01.039 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.039 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.039 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:01.039 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553336+0000", 2020-06-12T02:39:01.039 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1620000000000002e-06 2020-06-12T02:39:01.040 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.040 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.040 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:01.040 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553344+0000", 2020-06-12T02:39:01.040 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.807e-06 2020-06-12T02:39:01.040 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.041 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.041 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:01.041 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553346+0000", 2020-06-12T02:39:01.041 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001560575 2020-06-12T02:39:01.041 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.042 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.042 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:01.042 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554907+0000", 2020-06-12T02:39:01.043 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1429999999999999e-06 2020-06-12T02:39:01.043 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.043 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.043 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2020-06-12T02:39:01.044 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554908+0000", 2020-06-12T02:39:01.044 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99827844600000004 2020-06-12T02:39:01.045 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.045 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.045 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:01.045 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.553187+0000", 2020-06-12T02:39:01.046 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1316e-05 2020-06-12T02:39:01.046 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.046 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.046 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:01.046 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.553198+0000", 2020-06-12T02:39:01.047 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99988704799999994 2020-06-12T02:39:01.047 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:01.047 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:01.047 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:01.047 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.047 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.048 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11992 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [append 0~581632] snapc 0=[] ondisk+write+known_if_redirected e56)", 2020-06-12T02:39:01.048 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:23.783061+0000", 2020-06-12T02:39:01.048 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 37.155059512000001, 2020-06-12T02:39:01.048 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022850993, 2020-06-12T02:39:01.048 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:01.049 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:01.049 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:01.049 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:01.049 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:01.049 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11992 2020-06-12T02:39:01.049 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.051 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:01.051 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.052 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:01.052 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783061+0000", 2020-06-12T02:39:01.052 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:01.052 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.052 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.054 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:01.054 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783061+0000", 2020-06-12T02:39:01.055 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047999300000000001 2020-06-12T02:39:01.056 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.056 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.056 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:01.056 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783541+0000", 2020-06-12T02:39:01.056 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5649999999999998e-06 2020-06-12T02:39:01.057 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.057 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.057 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:01.057 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783548+0000", 2020-06-12T02:39:01.057 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3559e-05 2020-06-12T02:39:01.057 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.058 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.058 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:01.058 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783561+0000", 2020-06-12T02:39:01.058 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3539999999999996e-06 2020-06-12T02:39:01.058 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.059 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.059 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:01.059 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783567+0000", 2020-06-12T02:39:01.059 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001353604 2020-06-12T02:39:01.059 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.059 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.060 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:01.060 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.784921+0000", 2020-06-12T02:39:01.060 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000111853 2020-06-12T02:39:01.060 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.060 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.061 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:01.061 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.785033+0000", 2020-06-12T02:39:01.061 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051647919999999996 2020-06-12T02:39:01.061 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.061 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.063 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:01.063 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.790198+0000", 2020-06-12T02:39:01.064 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000110742 2020-06-12T02:39:01.064 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.064 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.064 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:01.064 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.790308+0000", 2020-06-12T02:39:01.065 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00090050099999999995 2020-06-12T02:39:01.065 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.065 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.065 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:01.065 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.791209+0000", 2020-06-12T02:39:01.066 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010078284999999999 2020-06-12T02:39:01.066 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.066 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.066 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:01.066 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.801287+0000", 2020-06-12T02:39:01.067 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045443109999999997 2020-06-12T02:39:01.067 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.067 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.067 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:01.067 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.805832+0000", 2020-06-12T02:39:01.068 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.0433999999999996e-05 2020-06-12T02:39:01.068 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.068 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.068 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:01.068 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.805912+0000", 2020-06-12T02:39:01.068 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022850993 2020-06-12T02:39:01.069 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:01.069 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:01.069 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:01.069 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.069 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.077 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11994 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected e56)", 2020-06-12T02:39:01.077 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:23.788734+0000", 2020-06-12T02:39:01.077 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 37.149386866999997, 2020-06-12T02:39:01.078 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024022136999999999, 2020-06-12T02:39:01.078 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:01.078 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:01.078 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:01.078 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:01.079 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:01.079 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11994 2020-06-12T02:39:01.079 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.079 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:01.079 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.080 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:01.080 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.788734+0000", 2020-06-12T02:39:01.080 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:01.080 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.080 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.080 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:01.081 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.788734+0000", 2020-06-12T02:39:01.081 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00034212499999999998 2020-06-12T02:39:01.081 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.081 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.081 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:01.082 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789076+0000", 2020-06-12T02:39:01.082 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.254e-06 2020-06-12T02:39:01.082 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.082 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.082 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:01.083 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789081+0000", 2020-06-12T02:39:01.083 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8249999999999994e-06 2020-06-12T02:39:01.083 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.083 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.083 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:01.083 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789090+0000", 2020-06-12T02:39:01.084 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2259999999999999e-06 2020-06-12T02:39:01.084 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.084 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.084 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:01.084 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789094+0000", 2020-06-12T02:39:01.085 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2913000000000001e-05 2020-06-12T02:39:01.085 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.085 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.085 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:01.086 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789127+0000", 2020-06-12T02:39:01.086 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2001000000000005e-05 2020-06-12T02:39:01.086 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.086 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.086 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:01.087 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789189+0000", 2020-06-12T02:39:01.087 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0097474550000000004 2020-06-12T02:39:01.087 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.088 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.088 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:01.088 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.798936+0000", 2020-06-12T02:39:01.089 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9882000000000005e-05 2020-06-12T02:39:01.089 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.089 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.090 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:01.090 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.799016+0000", 2020-06-12T02:39:01.090 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00048691000000000001 2020-06-12T02:39:01.091 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.092 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.092 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:01.092 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.799503+0000", 2020-06-12T02:39:01.093 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002766363 2020-06-12T02:39:01.093 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.093 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.093 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:01.094 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.802270+0000", 2020-06-12T02:39:01.094 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010404759 2020-06-12T02:39:01.094 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.095 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.095 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:01.095 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.812674+0000", 2020-06-12T02:39:01.096 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1423999999999998e-05 2020-06-12T02:39:01.096 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.097 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.097 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:01.098 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.812756+0000", 2020-06-12T02:39:01.098 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024022136999999999 2020-06-12T02:39:01.098 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:01.099 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:01.099 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:01.099 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.100 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.100 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11995 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e56)", 2020-06-12T02:39:01.100 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:23.789128+0000", 2020-06-12T02:39:01.100 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 37.148992421000003, 2020-06-12T02:39:01.101 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031795508, 2020-06-12T02:39:01.101 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:01.101 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:01.102 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:01.102 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:01.102 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:01.103 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11995 2020-06-12T02:39:01.103 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.103 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:01.103 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.104 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:01.104 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789128+0000", 2020-06-12T02:39:01.104 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:01.104 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.105 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.105 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:01.105 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789128+0000", 2020-06-12T02:39:01.106 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4419999999999998e-05 2020-06-12T02:39:01.106 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.106 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.107 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:01.107 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789172+0000", 2020-06-12T02:39:01.107 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7110000000000001e-06 2020-06-12T02:39:01.107 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.108 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.108 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:01.109 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789176+0000", 2020-06-12T02:39:01.109 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5089999999999999e-06 2020-06-12T02:39:01.109 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.109 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.109 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:01.110 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789183+0000", 2020-06-12T02:39:01.110 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.497e-06 2020-06-12T02:39:01.110 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.110 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.110 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:01.111 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789185+0000", 2020-06-12T02:39:01.111 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012172176 2020-06-12T02:39:01.111 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.111 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.111 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:01.111 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.801357+0000", 2020-06-12T02:39:01.112 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4264000000000006e-05 2020-06-12T02:39:01.113 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.113 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.113 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:01.113 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.801432+0000", 2020-06-12T02:39:01.114 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00029680499999999997 2020-06-12T02:39:01.114 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.114 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.114 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:01.115 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.801728+0000", 2020-06-12T02:39:01.115 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00066278300000000001 2020-06-12T02:39:01.115 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.115 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.115 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:01.115 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.802391+0000", 2020-06-12T02:39:01.116 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018429867999999999 2020-06-12T02:39:01.117 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.117 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.117 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:01.117 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.820821+0000", 2020-06-12T02:39:01.117 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000102475 2020-06-12T02:39:01.118 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.118 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.119 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:01.119 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.820924+0000", 2020-06-12T02:39:01.119 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031795508 2020-06-12T02:39:01.119 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:01.119 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:01.120 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:01.120 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.120 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.120 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11996 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e56)", 2020-06-12T02:39:01.120 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:23.789235+0000", 2020-06-12T02:39:01.120 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 37.148885569999997, 2020-06-12T02:39:01.121 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.034785745, 2020-06-12T02:39:01.121 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:01.121 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:01.121 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:01.122 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:01.122 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:01.122 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11996 2020-06-12T02:39:01.122 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.122 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:01.123 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.123 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:01.123 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789235+0000", 2020-06-12T02:39:01.123 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:01.124 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.124 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.124 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:01.124 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789235+0000", 2020-06-12T02:39:01.124 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6375000000000001e-05 2020-06-12T02:39:01.124 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.125 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.125 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:01.125 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789261+0000", 2020-06-12T02:39:01.125 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2679999999999999e-06 2020-06-12T02:39:01.125 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.126 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.126 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:01.126 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789265+0000", 2020-06-12T02:39:01.126 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6039999999999998e-06 2020-06-12T02:39:01.126 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.127 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.127 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:01.127 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789271+0000", 2020-06-12T02:39:01.127 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4219999999999999e-06 2020-06-12T02:39:01.127 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.127 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.128 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:01.128 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789274+0000", 2020-06-12T02:39:01.128 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013034463 2020-06-12T02:39:01.128 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.129 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.129 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:01.129 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.802308+0000", 2020-06-12T02:39:01.129 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8862000000000005e-05 2020-06-12T02:39:01.130 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.130 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.130 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:01.130 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.802387+0000", 2020-06-12T02:39:01.130 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019102297000000001 2020-06-12T02:39:01.130 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.131 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.131 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:01.131 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.821489+0000", 2020-06-12T02:39:01.131 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8838999999999996e-05 2020-06-12T02:39:01.131 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.132 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.132 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:01.132 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.821558+0000", 2020-06-12T02:39:01.132 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002462615 2020-06-12T02:39:01.132 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.132 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.133 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:01.133 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.824021+0000", 2020-06-12T02:39:01.133 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.034785745 2020-06-12T02:39:01.133 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:01.133 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:01.134 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:01.134 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.142 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.142 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:15005 3.ds0 3:b3445631:::smithi10729407-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e59)", 2020-06-12T02:39:01.142 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:32.192302+0000", 2020-06-12T02:39:01.143 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 28.745818334999999, 2020-06-12T02:39:01.143 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024128666, 2020-06-12T02:39:01.143 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:01.143 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:01.143 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:01.144 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:01.144 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:01.144 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 15005 2020-06-12T02:39:01.144 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.144 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:01.144 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.145 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:01.145 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192302+0000", 2020-06-12T02:39:01.145 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:01.145 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.145 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.146 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:01.146 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192302+0000", 2020-06-12T02:39:01.146 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3415999999999999e-05 2020-06-12T02:39:01.146 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.146 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.147 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:01.147 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192346+0000", 2020-06-12T02:39:01.147 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.93e-06 2020-06-12T02:39:01.147 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.147 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.147 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:01.148 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192352+0000", 2020-06-12T02:39:01.148 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3397000000000001e-05 2020-06-12T02:39:01.148 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.148 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.148 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:01.149 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192365+0000", 2020-06-12T02:39:01.149 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3950000000000003e-06 2020-06-12T02:39:01.149 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.149 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.149 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:01.150 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192369+0000", 2020-06-12T02:39:01.150 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010208887999999999 2020-06-12T02:39:01.150 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.150 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.150 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:01.150 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.202578+0000", 2020-06-12T02:39:01.151 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000105611 2020-06-12T02:39:01.151 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.151 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.151 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:01.151 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.202684+0000", 2020-06-12T02:39:01.152 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0091288959999999992 2020-06-12T02:39:01.152 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.152 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.153 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:01.153 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.211813+0000", 2020-06-12T02:39:01.153 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000101798 2020-06-12T02:39:01.153 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.153 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.153 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:01.154 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.211914+0000", 2020-06-12T02:39:01.154 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045163349999999998 2020-06-12T02:39:01.154 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.154 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.154 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:01.155 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.216431+0000", 2020-06-12T02:39:01.155 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024128666 2020-06-12T02:39:01.155 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:01.155 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:01.156 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:01.156 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.156 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.156 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:16810 3.ds0 3:b983fd28:::smithi10729407-1:head [append 0~524288] snapc 0=[] ondisk+write+known_if_redirected e61)", 2020-06-12T02:39:01.156 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:37.642363+0000", 2020-06-12T02:39:01.156 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 23.295757081000001, 2020-06-12T02:39:01.157 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022969679, 2020-06-12T02:39:01.157 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:01.157 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:01.158 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:01.158 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:01.158 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:01.158 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 16810 2020-06-12T02:39:01.158 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.159 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:01.159 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.159 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:01.159 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.642363+0000", 2020-06-12T02:39:01.159 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:01.159 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.160 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.160 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:01.160 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.642363+0000", 2020-06-12T02:39:01.160 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00069050800000000005 2020-06-12T02:39:01.160 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.161 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.161 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:01.161 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.643054+0000", 2020-06-12T02:39:01.161 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8220000000000007e-06 2020-06-12T02:39:01.161 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.161 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.162 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:01.162 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.643062+0000", 2020-06-12T02:39:01.163 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3742e-05 2020-06-12T02:39:01.163 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.163 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.163 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:01.163 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.643075+0000", 2020-06-12T02:39:01.164 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5910000000000001e-06 2020-06-12T02:39:01.164 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.164 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.164 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:01.164 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.643082+0000", 2020-06-12T02:39:01.165 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0064097879999999996 2020-06-12T02:39:01.165 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.165 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.165 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:01.165 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.649492+0000", 2020-06-12T02:39:01.165 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8151999999999994e-05 2020-06-12T02:39:01.166 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.166 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.166 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:01.166 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.649560+0000", 2020-06-12T02:39:01.166 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0020294940000000002 2020-06-12T02:39:01.167 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.167 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.167 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:01.167 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.651590+0000", 2020-06-12T02:39:01.167 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5242999999999997e-05 2020-06-12T02:39:01.168 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.168 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.168 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:01.168 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.651675+0000", 2020-06-12T02:39:01.168 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00069163799999999995 2020-06-12T02:39:01.168 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.169 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.169 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:01.169 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.652366+0000", 2020-06-12T02:39:01.169 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012711642 2020-06-12T02:39:01.169 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.170 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.170 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:01.170 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.665078+0000", 2020-06-12T02:39:01.170 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017519200000000001 2020-06-12T02:39:01.170 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.170 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.171 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:01.171 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.665253+0000", 2020-06-12T02:39:01.171 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9867e-05 2020-06-12T02:39:01.171 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.172 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.172 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:01.172 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.665333+0000", 2020-06-12T02:39:01.173 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022969679 2020-06-12T02:39:01.173 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:01.173 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:01.173 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:01.174 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.174 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.174 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18287 3.ds0 3:b983fd28:::smithi10729407-1:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:01.174 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.975980+0000", 2020-06-12T02:39:01.174 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 18.962140255000001, 2020-06-12T02:39:01.175 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.023612415000000001, 2020-06-12T02:39:01.175 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:01.175 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:01.175 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:01.175 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:01.176 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:01.176 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18287 2020-06-12T02:39:01.176 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.176 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:01.176 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.176 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:01.177 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.975980+0000", 2020-06-12T02:39:01.177 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:01.177 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.177 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.177 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:01.178 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.975980+0000", 2020-06-12T02:39:01.178 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0175999999999999e-05 2020-06-12T02:39:01.178 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.178 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.178 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:01.178 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976070+0000", 2020-06-12T02:39:01.179 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7489999999999998e-06 2020-06-12T02:39:01.179 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.179 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.179 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:01.179 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976078+0000", 2020-06-12T02:39:01.180 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5308999999999999e-05 2020-06-12T02:39:01.180 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.180 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.180 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:01.180 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976093+0000", 2020-06-12T02:39:01.180 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7149999999999997e-06 2020-06-12T02:39:01.181 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.181 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.181 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:01.181 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976098+0000", 2020-06-12T02:39:01.181 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0067110759999999998 2020-06-12T02:39:01.182 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.182 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.182 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:01.182 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.982809+0000", 2020-06-12T02:39:01.182 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011082999999999999 2020-06-12T02:39:01.183 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.183 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.183 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:01.184 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.982920+0000", 2020-06-12T02:39:01.184 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00043570099999999998 2020-06-12T02:39:01.184 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.184 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.184 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:01.184 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.983356+0000", 2020-06-12T02:39:01.185 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00076300800000000002 2020-06-12T02:39:01.185 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.185 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.185 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:01.185 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.984119+0000", 2020-06-12T02:39:01.186 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01538405 2020-06-12T02:39:01.186 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.186 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.186 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:01.186 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.999503+0000", 2020-06-12T02:39:01.187 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9801000000000003e-05 2020-06-12T02:39:01.187 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.187 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.187 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:01.187 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.999593+0000", 2020-06-12T02:39:01.187 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.023612415000000001 2020-06-12T02:39:01.188 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:01.188 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:01.188 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:01.188 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.188 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.189 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18288 3.ds0 3:b983fd28:::smithi10729407-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:39:01.189 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.976151+0000", 2020-06-12T02:39:01.189 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 18.961969331999999, 2020-06-12T02:39:01.189 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025323907999999999, 2020-06-12T02:39:01.189 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:01.189 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:01.190 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:01.190 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:01.190 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:01.190 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18288 2020-06-12T02:39:01.190 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.191 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:01.191 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.191 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:01.191 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976151+0000", 2020-06-12T02:39:01.191 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:01.192 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.192 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.192 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:01.192 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976151+0000", 2020-06-12T02:39:01.193 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5615000000000002e-05 2020-06-12T02:39:01.193 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.193 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.194 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:01.194 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976207+0000", 2020-06-12T02:39:01.194 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6399999999999997e-06 2020-06-12T02:39:01.194 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.194 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.195 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:01.195 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976214+0000", 2020-06-12T02:39:01.195 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4823e-05 2020-06-12T02:39:01.195 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.195 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.195 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:01.196 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976229+0000", 2020-06-12T02:39:01.196 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1519999999999997e-06 2020-06-12T02:39:01.196 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.196 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.196 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:01.197 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976235+0000", 2020-06-12T02:39:01.197 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0077693090000000003 2020-06-12T02:39:01.197 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.197 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.197 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:01.197 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.984005+0000", 2020-06-12T02:39:01.198 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000101221 2020-06-12T02:39:01.198 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.198 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.198 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:01.198 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.984106+0000", 2020-06-12T02:39:01.199 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016277713999999999 2020-06-12T02:39:01.199 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.199 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.200 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:01.200 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.000384+0000", 2020-06-12T02:39:01.200 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7938999999999999e-05 2020-06-12T02:39:01.200 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.200 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.200 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:01.201 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.000482+0000", 2020-06-12T02:39:01.201 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00099349500000000006 2020-06-12T02:39:01.201 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.201 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.201 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:01.202 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.001475+0000", 2020-06-12T02:39:01.202 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025323907999999999 2020-06-12T02:39:01.202 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:01.202 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:01.202 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:01.203 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.203 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.203 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18307 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [append 0~507904] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:01.203 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.056119+0000", 2020-06-12T02:39:01.203 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 18.882001449000001, 2020-06-12T02:39:01.204 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025879849, 2020-06-12T02:39:01.204 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:01.204 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:01.204 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:01.205 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:01.205 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:01.205 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18307 2020-06-12T02:39:01.205 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.205 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:01.206 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.206 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:01.206 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.056119+0000", 2020-06-12T02:39:01.206 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:01.207 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.207 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.207 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:01.207 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.056119+0000", 2020-06-12T02:39:01.207 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00052484099999999996 2020-06-12T02:39:01.208 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.208 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.208 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:01.208 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.056644+0000", 2020-06-12T02:39:01.208 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7149999999999997e-06 2020-06-12T02:39:01.208 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.209 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.209 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:01.209 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.056649+0000", 2020-06-12T02:39:01.209 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0046e-05 2020-06-12T02:39:01.209 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.210 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.210 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:01.210 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.056659+0000", 2020-06-12T02:39:01.210 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.848e-06 2020-06-12T02:39:01.210 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.210 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.211 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:01.211 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.056663+0000", 2020-06-12T02:39:01.211 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024537029999999998 2020-06-12T02:39:01.211 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.211 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.212 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:01.212 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.059117+0000", 2020-06-12T02:39:01.212 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000121261 2020-06-12T02:39:01.212 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.212 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.212 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:01.213 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.059238+0000", 2020-06-12T02:39:01.213 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001916484 2020-06-12T02:39:01.213 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.213 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.213 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:01.214 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.061155+0000", 2020-06-12T02:39:01.214 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9120000000000001e-05 2020-06-12T02:39:01.214 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.215 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.215 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:01.215 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.061234+0000", 2020-06-12T02:39:01.215 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00060383599999999996 2020-06-12T02:39:01.215 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.216 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.216 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:01.216 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.061838+0000", 2020-06-12T02:39:01.216 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010259862999999999 2020-06-12T02:39:01.216 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.216 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.217 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:01.217 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.072098+0000", 2020-06-12T02:39:01.217 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0098176419999999997 2020-06-12T02:39:01.217 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.217 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.218 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:01.218 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.081915+0000", 2020-06-12T02:39:01.218 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3490000000000002e-05 2020-06-12T02:39:01.218 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.218 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.218 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:01.219 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.081999+0000", 2020-06-12T02:39:01.219 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025879849 2020-06-12T02:39:01.219 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:01.219 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:01.219 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:01.220 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.220 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.220 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18311 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:39:01.220 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.063458+0000", 2020-06-12T02:39:01.221 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 18.874662813, 2020-06-12T02:39:01.221 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.035342152000000002, 2020-06-12T02:39:01.221 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:01.221 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:01.221 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:01.221 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:01.222 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:01.222 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18311 2020-06-12T02:39:01.222 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.222 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:01.222 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.223 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:01.223 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.063458+0000", 2020-06-12T02:39:01.223 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:01.223 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.223 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.223 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:01.224 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.063458+0000", 2020-06-12T02:39:01.224 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3773000000000003e-05 2020-06-12T02:39:01.224 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.225 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.225 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:01.225 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.063501+0000", 2020-06-12T02:39:01.225 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0780000000000001e-06 2020-06-12T02:39:01.225 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.226 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.226 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:01.226 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.063508+0000", 2020-06-12T02:39:01.226 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3020000000000001e-05 2020-06-12T02:39:01.226 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.227 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.227 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:01.227 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.063521+0000", 2020-06-12T02:39:01.227 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4839999999999998e-06 2020-06-12T02:39:01.227 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.227 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.228 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:01.228 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.063525+0000", 2020-06-12T02:39:01.228 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0087769890000000007 2020-06-12T02:39:01.228 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.228 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.229 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:01.229 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.072302+0000", 2020-06-12T02:39:01.229 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3121000000000005e-05 2020-06-12T02:39:01.229 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.229 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.229 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:01.230 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.072395+0000", 2020-06-12T02:39:01.230 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010158355000000001 2020-06-12T02:39:01.230 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.230 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.230 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:01.231 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.082554+0000", 2020-06-12T02:39:01.231 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5016000000000001e-05 2020-06-12T02:39:01.231 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.231 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.231 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:01.231 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.082629+0000", 2020-06-12T02:39:01.232 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016171316000000002 2020-06-12T02:39:01.232 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.232 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.232 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:01.232 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.098800+0000", 2020-06-12T02:39:01.233 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.035342152000000002 2020-06-12T02:39:01.233 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:01.233 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:01.233 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:01.233 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.233 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.234 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18313 3.fs0 3:f83c3dea:::smithi10729407-4:head [append 0~442368] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:01.234 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.070249+0000", 2020-06-12T02:39:01.234 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 18.867871535999999, 2020-06-12T02:39:01.234 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028853489, 2020-06-12T02:39:01.234 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:01.235 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:01.235 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:01.235 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:01.236 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:01.236 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18313 2020-06-12T02:39:01.236 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.236 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:01.236 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.236 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:01.237 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.070249+0000", 2020-06-12T02:39:01.237 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:01.237 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.237 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.237 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:01.238 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.070249+0000", 2020-06-12T02:39:01.238 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00046159400000000001 2020-06-12T02:39:01.238 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.238 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.238 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:01.238 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.070711+0000", 2020-06-12T02:39:01.239 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2539999999999992e-06 2020-06-12T02:39:01.239 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.239 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.239 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:01.239 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.070719+0000", 2020-06-12T02:39:01.240 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3297e-05 2020-06-12T02:39:01.240 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.240 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.240 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:01.240 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.070732+0000", 2020-06-12T02:39:01.240 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0559999999999999e-06 2020-06-12T02:39:01.241 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.241 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.241 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:01.241 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.070739+0000", 2020-06-12T02:39:01.241 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0052673319999999996 2020-06-12T02:39:01.242 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.242 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.242 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:01.242 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.076006+0000", 2020-06-12T02:39:01.242 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6925999999999997e-05 2020-06-12T02:39:01.242 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.243 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.243 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:01.243 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.076063+0000", 2020-06-12T02:39:01.243 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0076712209999999998 2020-06-12T02:39:01.243 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.244 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.244 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:01.244 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.083735+0000", 2020-06-12T02:39:01.244 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8089000000000002e-05 2020-06-12T02:39:01.244 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.245 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.245 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:01.245 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.083833+0000", 2020-06-12T02:39:01.245 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054620600000000004 2020-06-12T02:39:01.246 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.246 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.246 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:01.246 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.084379+0000", 2020-06-12T02:39:01.246 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013940945999999999 2020-06-12T02:39:01.247 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.247 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.247 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:01.247 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.098320+0000", 2020-06-12T02:39:01.247 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00072638599999999998 2020-06-12T02:39:01.248 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.248 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.248 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:01.248 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.099046+0000", 2020-06-12T02:39:01.248 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6181999999999998e-05 2020-06-12T02:39:01.248 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.249 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.249 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:01.249 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.099102+0000", 2020-06-12T02:39:01.249 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028853489 2020-06-12T02:39:01.249 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:01.250 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:01.250 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:01.250 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.250 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.250 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18314 3.fs0 3:f83c3dea:::smithi10729407-4:head [append 0~573440] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:01.251 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.074005+0000", 2020-06-12T02:39:01.251 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 18.864115516999998, 2020-06-12T02:39:01.251 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031442581999999997, 2020-06-12T02:39:01.251 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:01.251 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:01.251 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:01.252 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:01.252 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:01.252 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18314 2020-06-12T02:39:01.252 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.252 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:01.253 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.253 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:01.253 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.074005+0000", 2020-06-12T02:39:01.253 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:01.253 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.253 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.254 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:01.254 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.074005+0000", 2020-06-12T02:39:01.254 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00056762000000000002 2020-06-12T02:39:01.254 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.254 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.255 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:01.255 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.074573+0000", 2020-06-12T02:39:01.255 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5400000000000003e-06 2020-06-12T02:39:01.255 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.256 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.256 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:01.256 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.074578+0000", 2020-06-12T02:39:01.256 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1090000000000004e-06 2020-06-12T02:39:01.256 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.257 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.257 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:01.257 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.074587+0000", 2020-06-12T02:39:01.257 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4039999999999997e-06 2020-06-12T02:39:01.257 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.258 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.258 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:01.258 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.074592+0000", 2020-06-12T02:39:01.258 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001538031 2020-06-12T02:39:01.258 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.258 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.259 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:01.259 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.076130+0000", 2020-06-12T02:39:01.259 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5117000000000002e-05 2020-06-12T02:39:01.259 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.259 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.260 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:01.260 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.076175+0000", 2020-06-12T02:39:01.260 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011822262 2020-06-12T02:39:01.260 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.260 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.260 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:01.261 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.087997+0000", 2020-06-12T02:39:01.261 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000112318 2020-06-12T02:39:01.261 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.261 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.261 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:01.262 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.088109+0000", 2020-06-12T02:39:01.262 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00072351100000000003 2020-06-12T02:39:01.262 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.262 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.262 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:01.263 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.088833+0000", 2020-06-12T02:39:01.263 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010307874999999999 2020-06-12T02:39:01.263 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.263 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.264 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:01.264 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.099141+0000", 2020-06-12T02:39:01.264 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.006229605 2020-06-12T02:39:01.264 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.264 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.265 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:01.265 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.105370+0000", 2020-06-12T02:39:01.265 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7189999999999998e-05 2020-06-12T02:39:01.265 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.265 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.265 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:01.266 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.105448+0000", 2020-06-12T02:39:01.266 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031442581999999997 2020-06-12T02:39:01.266 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:01.267 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:01.267 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:01.267 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.267 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.267 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18315 3.fs0 3:f83c3dea:::smithi10729407-4:head [append 0~589824] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:01.268 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.077724+0000", 2020-06-12T02:39:01.268 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 18.860396657999999, 2020-06-12T02:39:01.268 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.027939051999999999, 2020-06-12T02:39:01.268 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:01.268 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:01.268 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:01.269 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:01.269 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:01.269 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18315 2020-06-12T02:39:01.269 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.269 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:01.270 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.270 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:01.270 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.077724+0000", 2020-06-12T02:39:01.270 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:01.270 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.270 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.271 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:01.271 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.077724+0000", 2020-06-12T02:39:01.271 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00093652299999999998 2020-06-12T02:39:01.271 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.271 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.272 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:01.272 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078660+0000", 2020-06-12T02:39:01.272 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1389999999999995e-06 2020-06-12T02:39:01.272 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.272 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.273 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:01.273 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078669+0000", 2020-06-12T02:39:01.273 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3691e-05 2020-06-12T02:39:01.273 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.273 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.273 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:01.274 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078682+0000", 2020-06-12T02:39:01.281 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0100000000000001e-06 2020-06-12T02:39:01.282 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.282 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.282 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:01.282 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078688+0000", 2020-06-12T02:39:01.282 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1209999999999999e-05 2020-06-12T02:39:01.283 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.283 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.283 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:01.283 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078739+0000", 2020-06-12T02:39:01.283 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000102735 2020-06-12T02:39:01.284 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.284 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.284 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:01.284 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078842+0000", 2020-06-12T02:39:01.284 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01321815 2020-06-12T02:39:01.284 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.285 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.285 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:01.285 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.092060+0000", 2020-06-12T02:39:01.285 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00038608499999999999 2020-06-12T02:39:01.285 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.286 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.286 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:01.286 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.092446+0000", 2020-06-12T02:39:01.286 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00056703299999999999 2020-06-12T02:39:01.286 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.287 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.287 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:01.287 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.093013+0000", 2020-06-12T02:39:01.287 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0061289530000000004 2020-06-12T02:39:01.287 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.287 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.288 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:01.288 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.099142+0000", 2020-06-12T02:39:01.288 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0064547700000000003 2020-06-12T02:39:01.288 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.288 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.289 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:01.289 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.105597+0000", 2020-06-12T02:39:01.289 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5753000000000006e-05 2020-06-12T02:39:01.289 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.289 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.289 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:01.290 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.105663+0000", 2020-06-12T02:39:01.290 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.027939051999999999 2020-06-12T02:39:01.290 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:01.290 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:01.290 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:01.291 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.291 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.291 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18316 3.fs0 3:f83c3dea:::smithi10729407-4:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:01.292 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.078752+0000", 2020-06-12T02:39:01.292 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 18.859368885999999, 2020-06-12T02:39:01.292 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.029108709999999999, 2020-06-12T02:39:01.292 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:01.292 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:01.293 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:01.293 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:01.293 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:01.293 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18316 2020-06-12T02:39:01.293 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.293 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:01.294 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.294 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:01.294 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078752+0000", 2020-06-12T02:39:01.294 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:01.294 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.295 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.295 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:01.295 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078752+0000", 2020-06-12T02:39:01.295 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8967000000000004e-05 2020-06-12T02:39:01.295 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.296 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.296 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:01.296 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078841+0000", 2020-06-12T02:39:01.296 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3309999999999998e-06 2020-06-12T02:39:01.296 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.296 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.297 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:01.297 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078848+0000", 2020-06-12T02:39:01.297 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.291e-05 2020-06-12T02:39:01.297 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.297 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.298 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:01.298 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078861+0000", 2020-06-12T02:39:01.298 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2800000000000003e-06 2020-06-12T02:39:01.298 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.298 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.298 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:01.299 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078866+0000", 2020-06-12T02:39:01.299 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8543e-05 2020-06-12T02:39:01.299 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.299 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.299 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:01.300 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078905+0000", 2020-06-12T02:39:01.300 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9198999999999998e-05 2020-06-12T02:39:01.300 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.300 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.300 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:01.300 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078994+0000", 2020-06-12T02:39:01.301 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01781437 2020-06-12T02:39:01.301 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.301 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.301 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:01.301 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.096808+0000", 2020-06-12T02:39:01.302 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000115975 2020-06-12T02:39:01.302 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.302 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.303 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:01.303 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.096924+0000", 2020-06-12T02:39:01.303 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00049986300000000002 2020-06-12T02:39:01.303 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.303 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.303 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:01.304 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.097424+0000", 2020-06-12T02:39:01.304 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0017191089999999999 2020-06-12T02:39:01.304 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.304 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.304 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:01.305 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.099143+0000", 2020-06-12T02:39:01.305 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0086177909999999996 2020-06-12T02:39:01.305 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.305 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.305 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:01.306 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.107761+0000", 2020-06-12T02:39:01.306 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9371999999999997e-05 2020-06-12T02:39:01.306 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.306 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.306 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:01.307 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.107860+0000", 2020-06-12T02:39:01.307 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.029108709999999999 2020-06-12T02:39:01.307 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:01.307 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:01.308 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:01.308 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.308 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.308 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18317 3.fs0 3:f83c3dea:::smithi10729407-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:39:01.308 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.078920+0000", 2020-06-12T02:39:01.308 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 18.859200915999999, 2020-06-12T02:39:01.309 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031356964000000001, 2020-06-12T02:39:01.309 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:01.309 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:01.309 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:01.309 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:01.310 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:01.310 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18317 2020-06-12T02:39:01.310 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.310 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:01.310 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.310 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:01.311 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078920+0000", 2020-06-12T02:39:01.311 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:01.311 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.311 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.311 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:01.312 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078920+0000", 2020-06-12T02:39:01.312 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6454000000000001e-05 2020-06-12T02:39:01.312 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.313 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.313 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:01.313 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078946+0000", 2020-06-12T02:39:01.313 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8160000000000004e-06 2020-06-12T02:39:01.313 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.313 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.314 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:01.314 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078950+0000", 2020-06-12T02:39:01.314 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6090000000000003e-06 2020-06-12T02:39:01.314 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.314 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.315 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:01.315 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078957+0000", 2020-06-12T02:39:01.315 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8380000000000002e-06 2020-06-12T02:39:01.315 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.315 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.315 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:01.316 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078960+0000", 2020-06-12T02:39:01.316 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8400000000000003e-05 2020-06-12T02:39:01.316 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.316 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.316 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:01.317 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.079019+0000", 2020-06-12T02:39:01.317 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6750999999999998e-05 2020-06-12T02:39:01.317 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.317 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.317 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:01.318 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.079085+0000", 2020-06-12T02:39:01.318 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019272036999999999 2020-06-12T02:39:01.318 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.318 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.318 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:01.318 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.098357+0000", 2020-06-12T02:39:01.319 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4322999999999998e-05 2020-06-12T02:39:01.319 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.319 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.319 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:01.319 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.098452+0000", 2020-06-12T02:39:01.320 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010578025 2020-06-12T02:39:01.320 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.320 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.320 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:01.320 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.109030+0000", 2020-06-12T02:39:01.320 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010395099999999999 2020-06-12T02:39:01.321 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.321 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.321 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:01.321 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.109134+0000", 2020-06-12T02:39:01.321 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00114276 2020-06-12T02:39:01.322 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:01.322 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:01.322 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:01.322 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.110277+0000", 2020-06-12T02:39:01.323 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031356964000000001 2020-06-12T02:39:01.323 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:01.323 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:01.323 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:01.323 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:01.324 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:01.324 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:39:01.325 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:39:01.323+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:39:01.326 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-06-12T02:39:01.426 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:39:01.424+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:39:01.456 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:39:01.457 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:39:01.457 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2020-06-12T02:39:01.457 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:39:01.465 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-06-12T02:39:01.526 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:39:01.524+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:39:01.583 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:39:01.584 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:39:01.584 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2020-06-12T02:39:01.584 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2020-06-12T02:39:01.584 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:39:01.592 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-06-12T02:39:01.629 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:39:01.626+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:39:01.710 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:39:01.711 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2020-06-12T02:39:01.711 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2020-06-12T02:39:01.711 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2020-06-12T02:39:01.711 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.711 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.3s2 49/21 ECSubWrite(tid=4109, reqid=client.4512.0:8841, at_version=49'773, trim_to=0'0, roll_forward_to=49'771))", 2020-06-12T02:39:01.712 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.822144+0000", 2020-06-12T02:39:01.712 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 49.884883615, 2020-06-12T02:39:01.712 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014799214, 2020-06-12T02:39:01.713 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:01.714 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:01.714 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:01.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.714 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:01.715 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822144+0000", 2020-06-12T02:39:01.715 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:01.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.715 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.715 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:01.715 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822144+0000", 2020-06-12T02:39:01.716 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00061470800000000005 2020-06-12T02:39:01.716 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.716 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.716 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:01.716 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822759+0000", 2020-06-12T02:39:01.717 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.7732000000000001e-05 2020-06-12T02:39:01.717 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.717 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.717 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:01.717 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822786+0000", 2020-06-12T02:39:01.718 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1010999999999999e-05 2020-06-12T02:39:01.718 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.718 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.718 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:01.718 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822807+0000", 2020-06-12T02:39:01.718 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.842e-06 2020-06-12T02:39:01.719 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.719 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.719 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:01.719 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822817+0000", 2020-06-12T02:39:01.719 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00024702300000000002 2020-06-12T02:39:01.720 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.720 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.720 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:01.720 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.823064+0000", 2020-06-12T02:39:01.720 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5207e-05 2020-06-12T02:39:01.721 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.721 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.721 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:01.721 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.823079+0000", 2020-06-12T02:39:01.721 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013806817000000001 2020-06-12T02:39:01.721 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.722 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.722 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:01.722 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.836886+0000", 2020-06-12T02:39:01.722 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6873999999999998e-05 2020-06-12T02:39:01.723 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.723 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.723 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:01.723 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.836943+0000", 2020-06-12T02:39:01.723 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014799214 2020-06-12T02:39:01.724 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:01.724 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:01.724 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:01.724 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.724 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.725 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14657 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~606208] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:39:01.725 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.209225+0000", 2020-06-12T02:39:01.725 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 30.497802364000002, 2020-06-12T02:39:01.725 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016128905999999998, 2020-06-12T02:39:01.725 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:01.725 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:01.726 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:01.726 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:01.726 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:01.726 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14657 2020-06-12T02:39:01.726 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.727 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:01.727 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.727 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:01.727 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209225+0000", 2020-06-12T02:39:01.727 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:01.727 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.728 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.728 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:01.728 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209225+0000", 2020-06-12T02:39:01.728 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018386699999999999 2020-06-12T02:39:01.728 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.729 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.729 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:01.729 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209409+0000", 2020-06-12T02:39:01.729 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8160000000000001e-06 2020-06-12T02:39:01.729 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.730 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.730 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:01.730 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209414+0000", 2020-06-12T02:39:01.730 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0230000000000004e-06 2020-06-12T02:39:01.730 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.730 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.731 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:01.731 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209421+0000", 2020-06-12T02:39:01.731 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0139999999999999e-06 2020-06-12T02:39:01.731 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.731 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.732 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:01.732 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209424+0000", 2020-06-12T02:39:01.732 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0054798169999999997 2020-06-12T02:39:01.732 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.732 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.732 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:01.733 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.214904+0000", 2020-06-12T02:39:01.733 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123063 2020-06-12T02:39:01.733 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.733 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.733 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:01.734 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215027+0000", 2020-06-12T02:39:01.734 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001159923 2020-06-12T02:39:01.734 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.734 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.735 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:01.735 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.216187+0000", 2020-06-12T02:39:01.735 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2835000000000002e-05 2020-06-12T02:39:01.735 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.735 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.736 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:01.736 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.216270+0000", 2020-06-12T02:39:01.736 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073575799999999998 2020-06-12T02:39:01.736 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.736 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.737 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:01.737 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.217005+0000", 2020-06-12T02:39:01.737 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0082434659999999996 2020-06-12T02:39:01.737 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.737 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.737 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:01.738 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225249+0000", 2020-06-12T02:39:01.738 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0145999999999997e-05 2020-06-12T02:39:01.738 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.738 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.738 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:39:01.739 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225299+0000", 2020-06-12T02:39:01.739 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5177999999999999e-05 2020-06-12T02:39:01.739 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.739 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.739 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:01.739 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225354+0000", 2020-06-12T02:39:01.740 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016128905999999998 2020-06-12T02:39:01.740 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:01.740 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:01.740 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:01.740 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.741 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.741 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14658 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~598016] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:39:01.741 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.209455+0000", 2020-06-12T02:39:01.741 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 30.497572609999999, 2020-06-12T02:39:01.741 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016447966000000001, 2020-06-12T02:39:01.741 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:01.742 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:01.742 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:01.742 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:01.742 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:01.742 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14658 2020-06-12T02:39:01.743 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.743 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:01.743 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.743 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:01.743 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209455+0000", 2020-06-12T02:39:01.743 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:01.744 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.744 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.744 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:01.744 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209455+0000", 2020-06-12T02:39:01.744 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037921599999999998 2020-06-12T02:39:01.745 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.745 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.745 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:01.745 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209834+0000", 2020-06-12T02:39:01.745 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8339999999999999e-06 2020-06-12T02:39:01.746 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.746 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.746 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:01.746 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209842+0000", 2020-06-12T02:39:01.746 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2564e-05 2020-06-12T02:39:01.746 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.747 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.747 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:01.747 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209855+0000", 2020-06-12T02:39:01.747 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.135e-06 2020-06-12T02:39:01.747 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.747 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.748 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:01.748 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209860+0000", 2020-06-12T02:39:01.748 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0052530290000000002 2020-06-12T02:39:01.748 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.748 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.749 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:01.749 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215113+0000", 2020-06-12T02:39:01.749 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6229000000000003e-05 2020-06-12T02:39:01.749 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.749 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.750 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:01.750 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215209+0000", 2020-06-12T02:39:01.750 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045765409999999999 2020-06-12T02:39:01.750 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.750 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.751 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:01.751 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.219785+0000", 2020-06-12T02:39:01.751 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000102846 2020-06-12T02:39:01.751 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.751 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.751 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:01.752 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.219888+0000", 2020-06-12T02:39:01.752 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00077088700000000005 2020-06-12T02:39:01.752 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.752 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.752 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:01.753 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.220659+0000", 2020-06-12T02:39:01.753 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046979739999999997 2020-06-12T02:39:01.753 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.753 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.753 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:01.754 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225357+0000", 2020-06-12T02:39:01.754 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00046069600000000002 2020-06-12T02:39:01.754 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.754 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.754 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:39:01.754 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225818+0000", 2020-06-12T02:39:01.755 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5014999999999998e-05 2020-06-12T02:39:01.755 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.755 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.755 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:01.755 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225903+0000", 2020-06-12T02:39:01.756 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016447966000000001 2020-06-12T02:39:01.756 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:01.756 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:01.756 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:01.756 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.757 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.757 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14659 3.0s0 3:0dddb690:::smithi10729407-10:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:39:01.757 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.209909+0000", 2020-06-12T02:39:01.757 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 30.497118402000002, 2020-06-12T02:39:01.758 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016618236000000002, 2020-06-12T02:39:01.758 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:01.758 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:01.758 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:01.759 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:01.759 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:01.759 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14659 2020-06-12T02:39:01.759 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.759 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:01.759 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.760 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:01.760 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209909+0000", 2020-06-12T02:39:01.760 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:01.760 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.760 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.761 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:01.761 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209909+0000", 2020-06-12T02:39:01.761 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8085999999999999e-05 2020-06-12T02:39:01.761 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.761 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.762 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:01.762 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209967+0000", 2020-06-12T02:39:01.762 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3169999999999998e-06 2020-06-12T02:39:01.762 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.762 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.762 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:01.763 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209974+0000", 2020-06-12T02:39:01.763 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1246999999999999e-05 2020-06-12T02:39:01.763 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.763 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.763 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:01.764 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209985+0000", 2020-06-12T02:39:01.764 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1439999999999996e-06 2020-06-12T02:39:01.764 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.764 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.764 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:01.764 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209989+0000", 2020-06-12T02:39:01.765 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0052537900000000004 2020-06-12T02:39:01.765 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.765 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.765 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:01.765 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215243+0000", 2020-06-12T02:39:01.766 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6023000000000002e-05 2020-06-12T02:39:01.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.766 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.766 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:01.766 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215299+0000", 2020-06-12T02:39:01.767 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0087846910000000007 2020-06-12T02:39:01.767 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.767 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.767 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:01.767 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.224083+0000", 2020-06-12T02:39:01.767 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8504000000000004e-05 2020-06-12T02:39:01.768 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.768 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.768 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:01.768 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.224182+0000", 2020-06-12T02:39:01.768 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044251400000000003 2020-06-12T02:39:01.769 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.769 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.769 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:01.769 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.224624+0000", 2020-06-12T02:39:01.769 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013383290000000001 2020-06-12T02:39:01.769 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.770 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.770 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:01.770 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225963+0000", 2020-06-12T02:39:01.770 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047942699999999999 2020-06-12T02:39:01.770 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.771 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.771 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:39:01.771 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.226442+0000", 2020-06-12T02:39:01.771 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5164e-05 2020-06-12T02:39:01.771 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.771 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.772 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:01.772 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.226527+0000", 2020-06-12T02:39:01.772 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016618236000000002 2020-06-12T02:39:01.772 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:01.772 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:01.773 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:01.773 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.773 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.773 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14660 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e59)", 2020-06-12T02:39:01.773 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.210020+0000", 2020-06-12T02:39:01.774 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 30.497007424, 2020-06-12T02:39:01.774 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017984777, 2020-06-12T02:39:01.774 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:01.774 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:39:01.774 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:01.774 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:01.775 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:01.775 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14660 2020-06-12T02:39:01.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.775 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:01.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.776 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:01.776 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210020+0000", 2020-06-12T02:39:01.776 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:01.776 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.776 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.776 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:01.777 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210020+0000", 2020-06-12T02:39:01.777 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2875999999999999e-05 2020-06-12T02:39:01.777 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.777 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.777 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:01.778 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210043+0000", 2020-06-12T02:39:01.778 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2200000000000001e-06 2020-06-12T02:39:01.778 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.778 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.778 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:01.778 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210046+0000", 2020-06-12T02:39:01.779 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7410000000000003e-06 2020-06-12T02:39:01.779 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.779 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.779 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:01.779 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210053+0000", 2020-06-12T02:39:01.780 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5610000000000001e-06 2020-06-12T02:39:01.780 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.780 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.780 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:01.780 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210055+0000", 2020-06-12T02:39:01.780 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0052596409999999998 2020-06-12T02:39:01.781 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.781 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.781 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:01.781 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215315+0000", 2020-06-12T02:39:01.781 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5627999999999997e-05 2020-06-12T02:39:01.782 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.782 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.782 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:01.782 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215371+0000", 2020-06-12T02:39:01.782 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0098069549999999991 2020-06-12T02:39:01.783 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.783 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.783 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:01.783 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225178+0000", 2020-06-12T02:39:01.783 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4530000000000001e-05 2020-06-12T02:39:01.783 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.784 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.784 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:01.784 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225232+0000", 2020-06-12T02:39:01.784 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017497350000000001 2020-06-12T02:39:01.784 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.785 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.785 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:01.785 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.226982+0000", 2020-06-12T02:39:01.785 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8154999999999998e-05 2020-06-12T02:39:01.785 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.785 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.786 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:01.786 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.227080+0000", 2020-06-12T02:39:01.786 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00092473499999999999 2020-06-12T02:39:01.786 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.786 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.787 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:01.787 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.228005+0000", 2020-06-12T02:39:01.787 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017984777 2020-06-12T02:39:01.787 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:01.787 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:01.788 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:01.788 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.788 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.788 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.3s2 59/53 ECSubWrite(tid=5970, reqid=client.4512.0:15226, at_version=59'1316, trim_to=0'0, roll_forward_to=59'1315))", 2020-06-12T02:39:01.788 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.829184+0000", 2020-06-12T02:39:01.788 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 28.877843208000002, 2020-06-12T02:39:01.789 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.081295588000000002, 2020-06-12T02:39:01.789 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:01.789 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:01.789 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:01.789 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.790 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:01.790 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829184+0000", 2020-06-12T02:39:01.790 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:01.790 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.790 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.790 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:01.791 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829184+0000", 2020-06-12T02:39:01.791 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00023814999999999999 2020-06-12T02:39:01.791 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.791 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.791 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:01.792 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829422+0000", 2020-06-12T02:39:01.792 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9768000000000001e-05 2020-06-12T02:39:01.792 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.792 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.792 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:01.792 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829442+0000", 2020-06-12T02:39:01.793 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1022000000000001e-05 2020-06-12T02:39:01.793 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.793 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.793 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:01.793 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829453+0000", 2020-06-12T02:39:01.794 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2759999999999997e-06 2020-06-12T02:39:01.794 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.794 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.794 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:01.794 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829460+0000", 2020-06-12T02:39:01.795 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7284000000000002e-05 2020-06-12T02:39:01.795 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.795 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.795 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:01.795 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829517+0000", 2020-06-12T02:39:01.795 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7028999999999999e-05 2020-06-12T02:39:01.796 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.796 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.796 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:01.796 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829534+0000", 2020-06-12T02:39:01.796 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.080899873999999997 2020-06-12T02:39:01.797 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.797 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.797 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:01.797 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.910434+0000", 2020-06-12T02:39:01.797 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6184999999999999e-05 2020-06-12T02:39:01.797 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.798 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.798 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:01.798 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.910480+0000", 2020-06-12T02:39:01.798 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.081295588000000002 2020-06-12T02:39:01.798 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:01.799 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:01.799 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:01.799 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.799 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.799 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.3s2 59/53 ECSubWrite(tid=5973, reqid=client.4512.0:15229, at_version=59'1319, trim_to=0'0, roll_forward_to=59'1315))", 2020-06-12T02:39:01.799 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.901708+0000", 2020-06-12T02:39:01.800 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 28.805319329, 2020-06-12T02:39:01.800 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013662957999999999, 2020-06-12T02:39:01.800 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:01.800 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:01.800 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:01.801 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.801 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:01.801 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901708+0000", 2020-06-12T02:39:01.801 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:01.801 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.802 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.802 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:01.802 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901708+0000", 2020-06-12T02:39:01.802 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7360000000000001e-05 2020-06-12T02:39:01.802 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.803 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.803 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:01.803 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901756+0000", 2020-06-12T02:39:01.803 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4608e-05 2020-06-12T02:39:01.803 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.804 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.804 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:01.804 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901770+0000", 2020-06-12T02:39:01.804 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.219e-05 2020-06-12T02:39:01.804 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.804 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.805 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:01.805 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901782+0000", 2020-06-12T02:39:01.805 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.511e-06 2020-06-12T02:39:01.805 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.805 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.806 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:01.806 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901789+0000", 2020-06-12T02:39:01.806 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.008025562 2020-06-12T02:39:01.806 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.806 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.806 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:01.807 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.909814+0000", 2020-06-12T02:39:01.807 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1472e-05 2020-06-12T02:39:01.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.807 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.807 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:01.808 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.909826+0000", 2020-06-12T02:39:01.808 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0055013350000000004 2020-06-12T02:39:01.808 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.808 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.808 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:01.808 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.915327+0000", 2020-06-12T02:39:01.809 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.392e-05 2020-06-12T02:39:01.809 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.809 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.809 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:01.809 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.915371+0000", 2020-06-12T02:39:01.810 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013662957999999999 2020-06-12T02:39:01.810 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:01.810 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:01.810 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:01.810 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.811 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.811 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.3s2 59/53 ECSubWrite(tid=5974, reqid=client.4512.0:15231, at_version=59'1320, trim_to=0'0, roll_forward_to=59'1315))", 2020-06-12T02:39:01.811 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.901839+0000", 2020-06-12T02:39:01.811 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 28.805188731000001, 2020-06-12T02:39:01.811 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013568525, 2020-06-12T02:39:01.811 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:01.812 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:01.812 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:01.812 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.812 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:01.812 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901839+0000", 2020-06-12T02:39:01.813 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:01.813 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.813 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.813 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:01.813 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901839+0000", 2020-06-12T02:39:01.813 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6489999999999997e-05 2020-06-12T02:39:01.814 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.814 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.814 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:01.814 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901885+0000", 2020-06-12T02:39:01.814 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6600999999999999e-05 2020-06-12T02:39:01.815 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.815 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.815 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:01.815 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901902+0000", 2020-06-12T02:39:01.815 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2427e-05 2020-06-12T02:39:01.815 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.816 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.816 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:01.816 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901914+0000", 2020-06-12T02:39:01.816 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1710000000000001e-06 2020-06-12T02:39:01.816 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.817 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.817 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:01.817 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901919+0000", 2020-06-12T02:39:01.817 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0086285540000000001 2020-06-12T02:39:01.817 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.817 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.818 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:01.818 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.910548+0000", 2020-06-12T02:39:01.818 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0939e-05 2020-06-12T02:39:01.818 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.818 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.819 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:01.819 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.910559+0000", 2020-06-12T02:39:01.819 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0048161899999999997 2020-06-12T02:39:01.819 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.819 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.819 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:01.820 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.915375+0000", 2020-06-12T02:39:01.820 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2153000000000002e-05 2020-06-12T02:39:01.820 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.820 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.820 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:01.821 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.915407+0000", 2020-06-12T02:39:01.821 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013568525 2020-06-12T02:39:01.821 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:01.821 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:01.821 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:01.822 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.822 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.822 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15246 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~671744] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:39:01.822 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.953872+0000", 2020-06-12T02:39:01.822 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 28.753155665000001, 2020-06-12T02:39:01.822 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015306691000000001, 2020-06-12T02:39:01.823 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:01.823 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:01.823 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:01.823 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:01.823 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:01.824 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15246 2020-06-12T02:39:01.824 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.824 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:01.824 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.824 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:01.824 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.953872+0000", 2020-06-12T02:39:01.825 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:01.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.825 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.825 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:01.825 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.953872+0000", 2020-06-12T02:39:01.826 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000232544 2020-06-12T02:39:01.826 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.826 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.826 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:01.826 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954104+0000", 2020-06-12T02:39:01.827 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7513999999999999e-05 2020-06-12T02:39:01.827 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.827 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.827 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:01.827 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954122+0000", 2020-06-12T02:39:01.827 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.844e-05 2020-06-12T02:39:01.828 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.828 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.828 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:01.828 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954140+0000", 2020-06-12T02:39:01.828 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2570000000000007e-06 2020-06-12T02:39:01.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.829 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.829 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:01.829 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954149+0000", 2020-06-12T02:39:01.829 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00031049499999999999 2020-06-12T02:39:01.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.830 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.830 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:01.830 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954459+0000", 2020-06-12T02:39:01.830 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000118182 2020-06-12T02:39:01.831 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.831 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.831 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:01.831 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954577+0000", 2020-06-12T02:39:01.831 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00071788400000000001 2020-06-12T02:39:01.832 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.832 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.832 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:01.832 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.955295+0000", 2020-06-12T02:39:01.832 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013717859000000001 2020-06-12T02:39:01.832 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.833 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.833 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:01.833 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.969013+0000", 2020-06-12T02:39:01.833 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4895999999999995e-05 2020-06-12T02:39:01.833 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.834 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.834 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:39:01.834 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.969098+0000", 2020-06-12T02:39:01.834 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0619999999999997e-05 2020-06-12T02:39:01.835 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.835 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.835 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:01.835 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.969179+0000", 2020-06-12T02:39:01.835 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015306691000000001 2020-06-12T02:39:01.835 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:01.836 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:01.836 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:01.836 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.836 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.836 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15277 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~720896] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:39:01.837 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:33.067109+0000", 2020-06-12T02:39:01.837 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 28.639918604999998, 2020-06-12T02:39:01.837 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015909191999999999, 2020-06-12T02:39:01.837 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:01.837 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:01.838 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:01.838 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:01.838 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:01.838 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15277 2020-06-12T02:39:01.838 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.838 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:01.839 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.839 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:01.839 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067109+0000", 2020-06-12T02:39:01.839 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:01.839 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.840 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.840 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:01.840 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067109+0000", 2020-06-12T02:39:01.840 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018431400000000001 2020-06-12T02:39:01.840 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.840 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.841 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:01.841 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067293+0000", 2020-06-12T02:39:01.841 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1790000000000002e-06 2020-06-12T02:39:01.841 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.841 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.842 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:01.842 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067300+0000", 2020-06-12T02:39:01.842 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5259999999999999e-06 2020-06-12T02:39:01.842 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.842 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.842 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:01.843 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067310+0000", 2020-06-12T02:39:01.843 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8179999999999997e-06 2020-06-12T02:39:01.843 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.843 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.843 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:01.844 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067314+0000", 2020-06-12T02:39:01.844 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6529999999999997e-05 2020-06-12T02:39:01.844 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.844 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.844 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:01.844 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067360+0000", 2020-06-12T02:39:01.845 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012724399999999999 2020-06-12T02:39:01.845 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.845 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.845 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:01.845 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067488+0000", 2020-06-12T02:39:01.846 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0010983989999999999 2020-06-12T02:39:01.846 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.846 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.846 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:01.846 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.068586+0000", 2020-06-12T02:39:01.847 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5165000000000001e-05 2020-06-12T02:39:01.847 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.847 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.847 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:01.847 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.068681+0000", 2020-06-12T02:39:01.847 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086510899999999997 2020-06-12T02:39:01.848 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.848 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.848 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:01.848 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.069546+0000", 2020-06-12T02:39:01.848 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013284429 2020-06-12T02:39:01.849 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.849 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.849 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:01.849 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.082831+0000", 2020-06-12T02:39:01.849 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4197000000000006e-05 2020-06-12T02:39:01.849 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.850 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.850 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:39:01.850 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.082925+0000", 2020-06-12T02:39:01.850 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3282000000000006e-05 2020-06-12T02:39:01.850 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.851 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.851 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:01.851 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.083018+0000", 2020-06-12T02:39:01.851 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015909191999999999 2020-06-12T02:39:01.851 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:01.851 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:01.852 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:01.852 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.852 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.852 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15331 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~688128] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:39:01.852 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:33.276748+0000", 2020-06-12T02:39:01.853 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 28.430279092999999, 2020-06-12T02:39:01.853 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016825066999999999, 2020-06-12T02:39:01.853 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:01.853 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:01.853 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:01.853 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:01.854 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:01.854 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15331 2020-06-12T02:39:01.854 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.854 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:01.854 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.855 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:01.855 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276748+0000", 2020-06-12T02:39:01.855 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:01.855 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.855 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.855 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:01.856 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276748+0000", 2020-06-12T02:39:01.856 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016936699999999999 2020-06-12T02:39:01.856 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.856 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.856 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:01.857 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276918+0000", 2020-06-12T02:39:01.857 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6389999999999998e-06 2020-06-12T02:39:01.857 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.857 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.858 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:01.858 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276926+0000", 2020-06-12T02:39:01.858 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3591000000000001e-05 2020-06-12T02:39:01.858 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.859 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.859 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:01.859 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276940+0000", 2020-06-12T02:39:01.859 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0429999999999996e-06 2020-06-12T02:39:01.859 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.859 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.860 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:01.860 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276949+0000", 2020-06-12T02:39:01.860 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8825000000000001e-05 2020-06-12T02:39:01.860 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.860 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.861 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:01.861 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276998+0000", 2020-06-12T02:39:01.861 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6483000000000005e-05 2020-06-12T02:39:01.861 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.861 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.862 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:01.862 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.277084+0000", 2020-06-12T02:39:01.863 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00057545800000000005 2020-06-12T02:39:01.863 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.863 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.863 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:01.863 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.277660+0000", 2020-06-12T02:39:01.864 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015617126 2020-06-12T02:39:01.864 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.864 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.864 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:01.864 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.293277+0000", 2020-06-12T02:39:01.865 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00021931699999999999 2020-06-12T02:39:01.865 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.865 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.865 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:39:01.865 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.293496+0000", 2020-06-12T02:39:01.865 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7218000000000005e-05 2020-06-12T02:39:01.866 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.866 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.866 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:01.866 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.293573+0000", 2020-06-12T02:39:01.866 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016825066999999999 2020-06-12T02:39:01.867 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:01.867 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:01.867 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:01.867 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.867 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.867 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.7s1 60/37 ECSubWrite(tid=8504, reqid=client.4512.0:16225, at_version=60'1427, trim_to=0'0, roll_forward_to=60'1425))", 2020-06-12T02:39:01.868 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:36.033154+0000", 2020-06-12T02:39:01.868 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 25.673873731, 2020-06-12T02:39:01.868 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016768244000000002, 2020-06-12T02:39:01.868 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:01.868 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:01.869 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:01.869 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.869 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:01.869 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033154+0000", 2020-06-12T02:39:01.869 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:01.870 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.870 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.870 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:01.870 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033154+0000", 2020-06-12T02:39:01.870 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000154444 2020-06-12T02:39:01.870 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.871 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.871 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:01.871 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033308+0000", 2020-06-12T02:39:01.871 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8772000000000002e-05 2020-06-12T02:39:01.871 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.872 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.872 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:01.872 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033337+0000", 2020-06-12T02:39:01.872 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6704000000000001e-05 2020-06-12T02:39:01.872 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.872 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.873 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:01.873 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033354+0000", 2020-06-12T02:39:01.873 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6100000000000002e-06 2020-06-12T02:39:01.873 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.873 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.874 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:01.874 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033360+0000", 2020-06-12T02:39:01.874 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000173893 2020-06-12T02:39:01.874 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.874 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.874 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:01.875 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033534+0000", 2020-06-12T02:39:01.875 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1929999999999999e-05 2020-06-12T02:39:01.875 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.875 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.875 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:01.876 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033546+0000", 2020-06-12T02:39:01.876 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016272066000000002 2020-06-12T02:39:01.876 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.876 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.876 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:01.876 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.049818+0000", 2020-06-12T02:39:01.877 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103825 2020-06-12T02:39:01.877 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.877 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.877 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:01.877 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.049922+0000", 2020-06-12T02:39:01.878 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016768244000000002 2020-06-12T02:39:01.878 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:01.878 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:01.878 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:01.878 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.878 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.879 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:17973 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~507904] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:01.879 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:41.095893+0000", 2020-06-12T02:39:01.879 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 20.611134178, 2020-06-12T02:39:01.879 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013655560000000001, 2020-06-12T02:39:01.879 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:01.880 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:01.880 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:01.880 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:01.880 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:01.880 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 17973 2020-06-12T02:39:01.881 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.881 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:01.881 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.881 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:01.881 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.095893+0000", 2020-06-12T02:39:01.881 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:01.882 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.882 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.882 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:01.882 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.095893+0000", 2020-06-12T02:39:01.882 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018351000000000001 2020-06-12T02:39:01.883 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.883 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:01.883 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.096077+0000", 2020-06-12T02:39:01.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1559999999999995e-06 2020-06-12T02:39:01.883 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.884 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:01.884 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.096085+0000", 2020-06-12T02:39:01.884 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3764999999999999e-05 2020-06-12T02:39:01.884 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.885 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.885 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:01.885 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.096099+0000", 2020-06-12T02:39:01.885 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0460000000000002e-06 2020-06-12T02:39:01.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.885 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.886 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:01.886 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.096105+0000", 2020-06-12T02:39:01.886 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6931000000000002e-05 2020-06-12T02:39:01.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.886 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:01.887 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.096152+0000", 2020-06-12T02:39:01.887 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010251600000000001 2020-06-12T02:39:01.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:01.888 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.096254+0000", 2020-06-12T02:39:01.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00072763000000000005 2020-06-12T02:39:01.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.888 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:01.889 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.096982+0000", 2020-06-12T02:39:01.889 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012262115000000001 2020-06-12T02:39:01.889 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.889 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.889 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:01.889 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.109244+0000", 2020-06-12T02:39:01.890 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1791000000000003e-05 2020-06-12T02:39:01.890 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.890 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.890 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:39:01.890 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.109336+0000", 2020-06-12T02:39:01.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0002131 2020-06-12T02:39:01.891 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.891 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.891 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:01.891 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.109549+0000", 2020-06-12T02:39:01.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013655560000000001 2020-06-12T02:39:01.892 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:01.892 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:01.892 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:01.892 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.892 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.893 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:17977 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:39:01.893 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:41.103849+0000", 2020-06-12T02:39:01.893 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 20.603178548999999, 2020-06-12T02:39:01.893 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015465273, 2020-06-12T02:39:01.893 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:01.894 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:39:01.894 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:01.900 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:01.900 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:01.901 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 17977 2020-06-12T02:39:01.901 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.901 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:01.901 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.901 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:01.902 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.103849+0000", 2020-06-12T02:39:01.902 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:01.902 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.902 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.902 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:01.902 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.103849+0000", 2020-06-12T02:39:01.903 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1254e-05 2020-06-12T02:39:01.903 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.903 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.903 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:01.903 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.103870+0000", 2020-06-12T02:39:01.904 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9869999999999999e-06 2020-06-12T02:39:01.904 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.904 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.904 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:01.904 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.103873+0000", 2020-06-12T02:39:01.904 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6159999999999998e-06 2020-06-12T02:39:01.905 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.905 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.905 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:01.905 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.103880+0000", 2020-06-12T02:39:01.905 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0899999999999999e-06 2020-06-12T02:39:01.906 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.906 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.906 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:01.906 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.103882+0000", 2020-06-12T02:39:01.906 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0066572259999999996 2020-06-12T02:39:01.906 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.907 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.907 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:01.907 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.110539+0000", 2020-06-12T02:39:01.907 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5937000000000003e-05 2020-06-12T02:39:01.907 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.908 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.908 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:01.908 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.110595+0000", 2020-06-12T02:39:01.908 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0066983820000000001 2020-06-12T02:39:01.908 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.909 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.909 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:01.909 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.117293+0000", 2020-06-12T02:39:01.909 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103828 2020-06-12T02:39:01.909 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.909 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.910 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:01.910 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.117397+0000", 2020-06-12T02:39:01.910 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0019169530000000001 2020-06-12T02:39:01.910 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.910 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.911 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:01.911 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.119314+0000", 2020-06-12T02:39:01.911 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015465273 2020-06-12T02:39:01.911 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:01.911 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:01.911 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:01.912 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.912 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.912 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:17998 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:39:01.912 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:41.190236+0000", 2020-06-12T02:39:01.912 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 20.516791494, 2020-06-12T02:39:01.913 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013903766, 2020-06-12T02:39:01.913 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:01.913 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:39:01.913 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:01.913 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:01.913 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:01.914 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 17998 2020-06-12T02:39:01.914 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.914 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:01.914 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.914 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:01.915 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.190236+0000", 2020-06-12T02:39:01.915 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:01.915 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.915 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.915 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:01.915 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.190236+0000", 2020-06-12T02:39:01.916 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.207e-05 2020-06-12T02:39:01.916 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.916 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.916 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:01.916 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.190258+0000", 2020-06-12T02:39:01.917 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9629999999999998e-06 2020-06-12T02:39:01.917 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.917 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.917 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:01.917 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.190261+0000", 2020-06-12T02:39:01.918 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.409e-06 2020-06-12T02:39:01.918 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.918 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.918 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:01.918 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.190267+0000", 2020-06-12T02:39:01.918 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2189999999999998e-06 2020-06-12T02:39:01.919 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.919 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.919 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:01.919 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.190270+0000", 2020-06-12T02:39:01.919 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0089464690000000003 2020-06-12T02:39:01.920 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.920 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.920 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:01.920 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.199216+0000", 2020-06-12T02:39:01.920 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010023500000000001 2020-06-12T02:39:01.920 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.921 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.921 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:01.921 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.199316+0000", 2020-06-12T02:39:01.921 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0038963800000000001 2020-06-12T02:39:01.921 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.922 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.922 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:01.922 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.203213+0000", 2020-06-12T02:39:01.922 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6963000000000003e-05 2020-06-12T02:39:01.922 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.923 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.923 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:01.923 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.203270+0000", 2020-06-12T02:39:01.923 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00087005800000000005 2020-06-12T02:39:01.923 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.923 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.924 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:01.924 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.204140+0000", 2020-06-12T02:39:01.924 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013903766 2020-06-12T02:39:01.924 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:01.924 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:01.925 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:01.925 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.925 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.925 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.7s1 62/37 ECSubWrite(tid=9513, reqid=client.4512.0:18266, at_version=0'0, trim_to=0'0, roll_forward_to=62'1601))", 2020-06-12T02:39:01.925 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:41.904485+0000", 2020-06-12T02:39:01.925 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 19.802542397, 2020-06-12T02:39:01.926 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01412036, 2020-06-12T02:39:01.926 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:01.926 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:01.926 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:01.926 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.927 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:01.927 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904485+0000", 2020-06-12T02:39:01.927 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:01.927 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.927 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.927 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:01.928 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904485+0000", 2020-06-12T02:39:01.928 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.345e-05 2020-06-12T02:39:01.928 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.928 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.928 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:01.929 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904519+0000", 2020-06-12T02:39:01.929 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0869999999999999e-05 2020-06-12T02:39:01.929 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.929 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.929 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:01.929 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904529+0000", 2020-06-12T02:39:01.930 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1248e-05 2020-06-12T02:39:01.930 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.930 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.930 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:01.930 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904541+0000", 2020-06-12T02:39:01.931 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0730999999999999e-05 2020-06-12T02:39:01.931 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.931 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.931 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:01.931 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904551+0000", 2020-06-12T02:39:01.931 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0985999999999997e-05 2020-06-12T02:39:01.932 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.932 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.932 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:01.932 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904592+0000", 2020-06-12T02:39:01.932 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2255999999999999e-05 2020-06-12T02:39:01.933 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.933 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.933 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:01.933 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904605+0000", 2020-06-12T02:39:01.933 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013935858000000001 2020-06-12T02:39:01.934 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.934 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.934 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:01.934 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.918541+0000", 2020-06-12T02:39:01.934 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4961000000000004e-05 2020-06-12T02:39:01.935 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.935 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.935 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:01.935 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.918605+0000", 2020-06-12T02:39:01.936 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01412036 2020-06-12T02:39:01.936 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:01.936 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:01.936 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:01.936 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.936 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.937 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "pg_update_log_missing(3.3s2 epoch 62/53 rep_tid 6734 entries 62'1548 (0'0) error 3:ce45c70a:::smithi10729407-7:head by client.4512.0:18270 0.000000 -2 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 62'1546)", 2020-06-12T02:39:01.937 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:41.904668+0000", 2020-06-12T02:39:01.937 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 19.802359860999999, 2020-06-12T02:39:01.937 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01430155, 2020-06-12T02:39:01.937 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:01.938 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:01.938 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:01.938 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.938 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:01.939 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904668+0000", 2020-06-12T02:39:01.939 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:01.939 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.939 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.939 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:01.939 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904668+0000", 2020-06-12T02:39:01.940 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1882000000000001e-05 2020-06-12T02:39:01.940 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.940 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.940 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:01.940 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904730+0000", 2020-06-12T02:39:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0861e-05 2020-06-12T02:39:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904740+0000", 2020-06-12T02:39:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0406000000000001e-05 2020-06-12T02:39:01.942 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.942 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.942 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:01.942 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904761+0000", 2020-06-12T02:39:01.942 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0339999999999997e-06 2020-06-12T02:39:01.943 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.943 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.943 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:01.943 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904769+0000", 2020-06-12T02:39:01.943 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3888999999999995e-05 2020-06-12T02:39:01.943 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.944 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.944 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:01.944 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.904863+0000", 2020-06-12T02:39:01.944 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014106478 2020-06-12T02:39:01.944 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.945 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.945 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:01.945 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.918969+0000", 2020-06-12T02:39:01.945 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01430155 2020-06-12T02:39:01.945 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:01.945 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:01.946 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:01.946 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.946 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.946 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.fs2 62/37 ECSubWrite(tid=3343, reqid=client.4512.0:18307, at_version=62'1623, trim_to=0'0, roll_forward_to=62'1622))", 2020-06-12T02:39:01.946 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:42.062050+0000", 2020-06-12T02:39:01.947 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 19.644977196999999, 2020-06-12T02:39:01.947 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019528429999999999, 2020-06-12T02:39:01.947 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:01.947 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:01.947 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:01.948 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.948 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:01.948 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062050+0000", 2020-06-12T02:39:01.948 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:01.948 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.948 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.949 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:01.949 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062050+0000", 2020-06-12T02:39:01.949 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00022828799999999999 2020-06-12T02:39:01.949 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.949 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.950 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:01.950 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062279+0000", 2020-06-12T02:39:01.950 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2115000000000001e-05 2020-06-12T02:39:01.950 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.950 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.950 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:01.951 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062301+0000", 2020-06-12T02:39:01.951 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9301e-05 2020-06-12T02:39:01.951 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.951 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.951 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:01.952 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062320+0000", 2020-06-12T02:39:01.952 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2811e-05 2020-06-12T02:39:01.952 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.952 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.952 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:01.952 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062333+0000", 2020-06-12T02:39:01.953 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8856e-05 2020-06-12T02:39:01.953 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.953 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.953 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:01.953 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062392+0000", 2020-06-12T02:39:01.954 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1246999999999999e-05 2020-06-12T02:39:01.954 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.954 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.954 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:01.954 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062403+0000", 2020-06-12T02:39:01.954 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019095523 2020-06-12T02:39:01.955 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.955 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.955 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:01.955 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.081498+0000", 2020-06-12T02:39:01.955 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0289000000000003e-05 2020-06-12T02:39:01.956 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.956 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.956 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:01.956 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.081579+0000", 2020-06-12T02:39:01.956 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019528429999999999 2020-06-12T02:39:01.956 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:01.957 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:01.957 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:01.957 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.957 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.957 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.fs2 62/37 ECSubWrite(tid=3344, reqid=client.4512.0:18308, at_version=62'1624, trim_to=0'0, roll_forward_to=62'1622))", 2020-06-12T02:39:01.958 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:42.064848+0000", 2020-06-12T02:39:01.958 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 19.642179211999999, 2020-06-12T02:39:01.958 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01677273, 2020-06-12T02:39:01.958 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:01.959 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:01.959 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:01.959 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.959 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:01.959 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.064848+0000", 2020-06-12T02:39:01.960 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:01.960 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.960 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.960 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:01.960 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.064848+0000", 2020-06-12T02:39:01.960 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000383655 2020-06-12T02:39:01.961 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.961 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.961 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:01.961 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.065232+0000", 2020-06-12T02:39:01.961 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3065999999999999e-05 2020-06-12T02:39:01.962 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.962 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.962 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:01.962 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.065255+0000", 2020-06-12T02:39:01.962 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7142999999999998e-05 2020-06-12T02:39:01.963 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.963 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.963 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:01.963 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.065272+0000", 2020-06-12T02:39:01.963 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7750000000000004e-06 2020-06-12T02:39:01.963 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.964 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.964 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:01.964 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.065279+0000", 2020-06-12T02:39:01.964 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00091758699999999998 2020-06-12T02:39:01.964 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.964 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.965 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:01.965 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.066197+0000", 2020-06-12T02:39:01.965 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4158999999999999e-05 2020-06-12T02:39:01.965 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.965 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.966 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:01.966 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.066211+0000", 2020-06-12T02:39:01.966 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015373370000000001 2020-06-12T02:39:01.966 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.966 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.967 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:01.967 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.081584+0000", 2020-06-12T02:39:01.967 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6974999999999998e-05 2020-06-12T02:39:01.967 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.967 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.967 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:01.968 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.081621+0000", 2020-06-12T02:39:01.968 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01677273 2020-06-12T02:39:01.968 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:01.968 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:01.968 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:01.968 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.969 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.969 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.3s2 62/53 ECSubWrite(tid=6737, reqid=client.4512.0:18319, at_version=62'1551, trim_to=0'0, roll_forward_to=62'1548))", 2020-06-12T02:39:01.969 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:42.089962+0000", 2020-06-12T02:39:01.969 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 19.617065574000002, 2020-06-12T02:39:01.969 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015432937000000001, 2020-06-12T02:39:01.970 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:01.970 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:01.970 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:01.970 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.970 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:01.971 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.089962+0000", 2020-06-12T02:39:01.971 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:01.971 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.971 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.971 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:01.971 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.089962+0000", 2020-06-12T02:39:01.972 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00068051099999999996 2020-06-12T02:39:01.972 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.972 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.972 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:01.972 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.090642+0000", 2020-06-12T02:39:01.973 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8766999999999998e-05 2020-06-12T02:39:01.973 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.973 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.973 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:01.973 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.090661+0000", 2020-06-12T02:39:01.974 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6461e-05 2020-06-12T02:39:01.974 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.974 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.974 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:01.974 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.090678+0000", 2020-06-12T02:39:01.975 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0580000000000003e-06 2020-06-12T02:39:01.975 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.975 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.975 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:01.975 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.090684+0000", 2020-06-12T02:39:01.976 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000104752 2020-06-12T02:39:01.976 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.976 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.976 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:01.976 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.090789+0000", 2020-06-12T02:39:01.976 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0767e-05 2020-06-12T02:39:01.977 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.977 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.977 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:01.977 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.090799+0000", 2020-06-12T02:39:01.977 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014539606 2020-06-12T02:39:01.978 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.978 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.978 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:01.978 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.105339+0000", 2020-06-12T02:39:01.978 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6014999999999998e-05 2020-06-12T02:39:01.978 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:01.979 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:01.979 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:01.979 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.105395+0000", 2020-06-12T02:39:01.979 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015432937000000001 2020-06-12T02:39:01.979 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:01.980 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:01.980 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:01.980 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:01.980 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:01.980 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:39:01.982 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-06-12T02:39:01.982 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:39:01.980+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:39:02.083 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:39:02.080+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:39:02.132 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-06-12T02:39:02.138 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:39:02.139 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:39:02.139 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2020-06-12T02:39:02.139 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:39:02.148 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-06-12T02:39:02.189 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:39:02.187+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:39:02.268 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:39:02.285 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:39:02.282+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:39:02.291 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:39:02.343 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:02.343 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:02.343 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:39:02.344 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:39:02.345 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-06-12T02:39:02.353 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:39:02.374 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:39:02.385 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:39:02.383+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:39:02.395 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:39:02.416 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2020-06-12T02:39:02.418 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-06-12T02:39:02.418 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-06-12T02:39:02.418 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-06-12T02:39:02.418 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-06-12T02:39:02.419 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-06-12T02:39:02.419 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-06-12T02:39:02.419 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-06-12T02:39:02.419 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-06-12T02:39:02.463 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:02.464 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:02.475 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-06-12T02:39:02.525 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:39:02.515+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:39:02.568 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2020-06-12T02:39:02.569 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2020-06-12T02:39:02.569 INFO:tasks.thrashosds.thrasher:inject_pause on 3 2020-06-12T02:39:02.570 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2020-06-12T02:39:02.570 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2020-06-12T02:39:02.571 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3 2020-06-12T02:39:02.576 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd set nodeep-scrub 2020-06-12T02:39:02.620 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:39:02.618+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:39:02.656 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:02.656 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:02.665 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-06-12T02:39:02.689 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:39:02.686+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:39:02.695 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:39:02.696 INFO:teuthology.orchestra.run.smithi107.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2020-06-12T02:39:02.696 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:39:02.789 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:39:02.787+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:39:02.816 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:02.817 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:02.829 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-06-12T02:39:02.890 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:39:02.888+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:39:02.950 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:02.951 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:02.959 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-06-12T02:39:02.991 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:39:02.989+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:39:03.092 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:39:03.090+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:39:03.122 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:03.122 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:03.134 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-06-12T02:39:03.192 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:39:03.190+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:39:03.290 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:03.291 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:03.293 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:39:03.291+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:39:03.299 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-06-12T02:39:03.394 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:39:03.392+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:39:03.454 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:03.454 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:03.463 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-06-12T02:39:03.495 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:39:03.493+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:39:03.577 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2020-06-12T02:39:03.592 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:03.592 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:03.599 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:39:03.593+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:39:03.601 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-06-12T02:39:03.697 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:39:03.694+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:39:03.763 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:03.763 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:03.773 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-06-12T02:39:03.803 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:39:03.800+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:39:03.903 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:39:03.901+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:39:03.929 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:03.930 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:03.938 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-06-12T02:39:04.025 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:39:04.023+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:39:04.101 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:04.102 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:04.114 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-06-12T02:39:04.115 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:39:04.104+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:39:04.208 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:39:04.205+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:39:04.269 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:04.270 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:04.278 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-06-12T02:39:04.321 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:39:04.316+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:39:04.410 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:39:04.407+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:39:04.428 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-06-12T02:39:04.436 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:04.436 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:04.446 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-06-12T02:39:04.511 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:39:04.509+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:39:04.540 INFO:tasks.rados.rados.0.smithi107.stdout:9386: done (1 left) 2020-06-12T02:39:04.540 INFO:tasks.rados.rados.0.smithi107.stdout:9387: delete oid 4 current snap is 0 2020-06-12T02:39:04.542 INFO:tasks.rados.rados.0.smithi107.stdout:9387: done (1 left) 2020-06-12T02:39:04.542 INFO:tasks.rados.rados.0.smithi107.stdout:9388: delete oid 13 current snap is 0 2020-06-12T02:39:04.543 INFO:tasks.rados.rados.0.smithi107.stdout:9388: done (1 left) 2020-06-12T02:39:04.544 INFO:tasks.rados.rados.0.smithi107.stdout:9389: delete oid 4 current snap is 0 2020-06-12T02:39:04.546 INFO:tasks.rados.rados.0.smithi107.stdout:9389: done (1 left) 2020-06-12T02:39:04.547 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:39:04.547 INFO:tasks.rados.rados.0.smithi107.stdout:9390: seq_num 9389 ranges {0=450560,450560=548864,999424=253952} 2020-06-12T02:39:04.549 INFO:tasks.rados.rados.0.smithi107.stdout:9390: writing smithi10729407-6 from 0 to 450560 tid 1 2020-06-12T02:39:04.552 INFO:tasks.rados.rados.0.smithi107.stdout:9390: writing smithi10729407-6 from 450560 to 999424 tid 2 2020-06-12T02:39:04.554 INFO:tasks.rados.rados.0.smithi107.stdout:9390: writing smithi10729407-6 from 999424 to 1253376 tid 3 2020-06-12T02:39:04.554 INFO:tasks.rados.rados.0.smithi107.stdout:9391: delete oid 10 current snap is 0 2020-06-12T02:39:04.554 INFO:tasks.rados.rados.0.smithi107.stdout:9354: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:04.554 INFO:tasks.rados.rados.0.smithi107.stdout:9354: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:04.556 INFO:tasks.rados.rados.0.smithi107.stdout:9354: finishing write tid 4 to smithi10729407-7 2020-06-12T02:39:04.556 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1642 (ObjNum 9353 snap 0 seq_num 9353) dirty exists 2020-06-12T02:39:04.557 INFO:tasks.rados.rados.0.smithi107.stdout:9354: left oid 7 (ObjNum 9353 snap 0 seq_num 9353) 2020-06-12T02:39:04.559 INFO:tasks.rados.rados.0.smithi107.stdout:9354: done (2 left) 2020-06-12T02:39:04.559 INFO:tasks.rados.rados.0.smithi107.stdout:9391: done (1 left) 2020-06-12T02:39:04.559 INFO:tasks.rados.rados.0.smithi107.stdout:9392: delete oid 8 current snap is 0 2020-06-12T02:39:04.564 INFO:tasks.rados.rados.0.smithi107.stdout:9392: done (1 left) 2020-06-12T02:39:04.564 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:39:04.564 INFO:tasks.rados.rados.0.smithi107.stdout:9393: seq_num 9392 ranges {0=499712,499712=606208,1105920=196608} 2020-06-12T02:39:04.566 INFO:tasks.rados.rados.0.smithi107.stdout:9393: writing smithi10729407-3 from 0 to 499712 tid 1 2020-06-12T02:39:04.570 INFO:tasks.rados.rados.0.smithi107.stdout:9393: writing smithi10729407-3 from 499712 to 1105920 tid 2 2020-06-12T02:39:04.572 INFO:tasks.rados.rados.0.smithi107.stdout:9393: writing smithi10729407-3 from 1105920 to 1302528 tid 3 2020-06-12T02:39:04.573 INFO:tasks.rados.rados.0.smithi107.stdout:9394: delete oid 11 current snap is 0 2020-06-12T02:39:04.573 INFO:tasks.rados.rados.0.smithi107.stdout:9390: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:04.573 INFO:tasks.rados.rados.0.smithi107.stdout:9390: finishing write tid 2 to smithi10729407-6 2020-06-12T02:39:04.573 INFO:tasks.rados.rados.0.smithi107.stdout:9390: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:04.574 INFO:tasks.rados.rados.0.smithi107.stdout:9390: finishing write tid 5 to smithi10729407-6 2020-06-12T02:39:04.574 INFO:tasks.rados.rados.0.smithi107.stdout:9390: finishing write tid 6 to smithi10729407-6 2020-06-12T02:39:04.574 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1703 (ObjNum 9389 snap 0 seq_num 9389) dirty exists 2020-06-12T02:39:04.574 INFO:tasks.rados.rados.0.smithi107.stdout:9390: left oid 6 (ObjNum 9389 snap 0 seq_num 9389) 2020-06-12T02:39:04.575 INFO:tasks.rados.rados.0.smithi107.stdout:9390: done (2 left) 2020-06-12T02:39:04.575 INFO:tasks.rados.rados.0.smithi107.stdout:9394: done (1 left) 2020-06-12T02:39:04.575 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:39:04.576 INFO:tasks.rados.rados.0.smithi107.stdout:9395: seq_num 9394 ranges {0=540672,540672=778240,1318912=24576} 2020-06-12T02:39:04.578 INFO:tasks.rados.rados.0.smithi107.stdout:9395: writing smithi10729407-4 from 0 to 540672 tid 1 2020-06-12T02:39:04.583 INFO:tasks.rados.rados.0.smithi107.stdout:9395: writing smithi10729407-4 from 540672 to 1318912 tid 2 2020-06-12T02:39:04.583 INFO:tasks.rados.rados.0.smithi107.stdout:9395: writing smithi10729407-4 from 1318912 to 1343488 tid 3 2020-06-12T02:39:04.585 INFO:tasks.rados.rados.0.smithi107.stdout:9396: delete oid 13 current snap is 0 2020-06-12T02:39:04.585 INFO:tasks.rados.rados.0.smithi107.stdout:9393: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:04.586 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:39:04.586 INFO:tasks.rados.rados.0.smithi107.stdout:9393: finishing write tid 2 to smithi10729407-3 2020-06-12T02:39:04.591 INFO:tasks.rados.rados.0.smithi107.stdout:9396: done (2 left) 2020-06-12T02:39:04.592 INFO:tasks.rados.rados.0.smithi107.stdout:9397: delete oid 13 current snap is 0 2020-06-12T02:39:04.592 INFO:tasks.rados.rados.0.smithi107.stdout:9393: finishing write tid 3 to smithi10729407-3 2020-06-12T02:39:04.592 INFO:tasks.rados.rados.0.smithi107.stdout:9393: finishing write tid 5 to smithi10729407-3 2020-06-12T02:39:04.593 INFO:tasks.rados.rados.0.smithi107.stdout:9395: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:04.593 INFO:tasks.rados.rados.0.smithi107.stdout:9395: finishing write tid 2 to smithi10729407-4 2020-06-12T02:39:04.594 INFO:tasks.rados.rados.0.smithi107.stdout:9393: finishing write tid 6 to smithi10729407-3 2020-06-12T02:39:04.594 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1665 (ObjNum 9392 snap 0 seq_num 9392) dirty exists 2020-06-12T02:39:04.594 INFO:tasks.rados.rados.0.smithi107.stdout:9393: left oid 3 (ObjNum 9392 snap 0 seq_num 9392) 2020-06-12T02:39:04.595 INFO:tasks.rados.rados.0.smithi107.stdout:9395: finishing write tid 3 to smithi10729407-4 2020-06-12T02:39:04.595 INFO:tasks.rados.rados.0.smithi107.stdout:9395: finishing write tid 5 to smithi10729407-4 2020-06-12T02:39:04.596 INFO:tasks.rados.rados.0.smithi107.stdout:9393: done (2 left) 2020-06-12T02:39:04.596 INFO:tasks.rados.rados.0.smithi107.stdout:9397: done (1 left) 2020-06-12T02:39:04.596 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:39:04.597 INFO:tasks.rados.rados.0.smithi107.stdout:9398: seq_num 9397 ranges {0=589824,589824=475136,1064960=327680} 2020-06-12T02:39:04.599 INFO:tasks.rados.rados.0.smithi107.stdout:9398: writing smithi10729407-2 from 0 to 589824 tid 1 2020-06-12T02:39:04.603 INFO:tasks.rados.rados.0.smithi107.stdout:9398: writing smithi10729407-2 from 589824 to 1064960 tid 2 2020-06-12T02:39:04.605 INFO:tasks.rados.rados.0.smithi107.stdout:9398: writing smithi10729407-2 from 1064960 to 1392640 tid 3 2020-06-12T02:39:04.605 INFO:tasks.rados.rados.0.smithi107.stdout:9399: delete oid 5 current snap is 0 2020-06-12T02:39:04.605 INFO:tasks.rados.rados.0.smithi107.stdout:9395: finishing write tid 6 to smithi10729407-4 2020-06-12T02:39:04.606 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1773 (ObjNum 9394 snap 0 seq_num 9394) dirty exists 2020-06-12T02:39:04.606 INFO:tasks.rados.rados.0.smithi107.stdout:9395: left oid 4 (ObjNum 9394 snap 0 seq_num 9394) 2020-06-12T02:39:04.607 INFO:tasks.rados.rados.0.smithi107.stdout:9395: done (2 left) 2020-06-12T02:39:04.607 INFO:tasks.rados.rados.0.smithi107.stdout:9399: done (1 left) 2020-06-12T02:39:04.607 INFO:tasks.rados.rados.0.smithi107.stdout:9400: delete oid 4 current snap is 0 2020-06-12T02:39:04.608 INFO:tasks.rados.rados.0.smithi107.stdout:9398: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:04.611 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:39:04.611 INFO:tasks.rados.rados.0.smithi107.stdout:9400: done (1 left) 2020-06-12T02:39:04.612 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:39:04.612 INFO:tasks.rados.rados.0.smithi107.stdout:9401: seq_num 9400 ranges {1302528=638976,1941504=532480,2473984=270336} 2020-06-12T02:39:04.618 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:39:04.612+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:39:04.621 INFO:tasks.rados.rados.0.smithi107.stdout:9401: writing smithi10729407-3 from 1302528 to 1941504 tid 1 2020-06-12T02:39:04.625 INFO:tasks.rados.rados.0.smithi107.stdout:9401: writing smithi10729407-3 from 1941504 to 2473984 tid 2 2020-06-12T02:39:04.628 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:04.628 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:04.629 INFO:tasks.rados.rados.0.smithi107.stdout:9401: writing smithi10729407-3 from 2473984 to 2744320 tid 3 2020-06-12T02:39:04.629 INFO:tasks.rados.rados.0.smithi107.stdout:9402: delete oid 9 current snap is 0 2020-06-12T02:39:04.629 INFO:tasks.rados.rados.0.smithi107.stdout:9398: finishing write tid 2 to smithi10729407-2 2020-06-12T02:39:04.629 INFO:tasks.rados.rados.0.smithi107.stdout:9398: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:04.630 INFO:tasks.rados.rados.0.smithi107.stdout:9398: finishing write tid 5 to smithi10729407-2 2020-06-12T02:39:04.630 INFO:tasks.rados.rados.0.smithi107.stdout:9398: finishing write tid 6 to smithi10729407-2 2020-06-12T02:39:04.630 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 869 (ObjNum 9397 snap 0 seq_num 9397) dirty exists 2020-06-12T02:39:04.630 INFO:tasks.rados.rados.0.smithi107.stdout:9398: left oid 2 (ObjNum 9397 snap 0 seq_num 9397) 2020-06-12T02:39:04.630 INFO:tasks.rados.rados.0.smithi107.stdout:9401: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:04.631 INFO:tasks.rados.rados.0.smithi107.stdout:9398: done (2 left) 2020-06-12T02:39:04.631 INFO:tasks.rados.rados.0.smithi107.stdout:9402: done (1 left) 2020-06-12T02:39:04.631 INFO:tasks.rados.rados.0.smithi107.stdout:9403: delete oid 18 current snap is 0 2020-06-12T02:39:04.634 INFO:tasks.rados.rados.0.smithi107.stdout:9403: done (1 left) 2020-06-12T02:39:04.634 INFO:tasks.rados.rados.0.smithi107.stdout:9404: delete oid 16 current snap is 0 2020-06-12T02:39:04.635 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:39:04.638 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-06-12T02:39:04.639 INFO:tasks.rados.rados.0.smithi107.stdout:9401: finishing write tid 2 to smithi10729407-3 2020-06-12T02:39:04.639 INFO:tasks.rados.rados.0.smithi107.stdout:9401: finishing write tid 3 to smithi10729407-3 2020-06-12T02:39:04.640 INFO:tasks.rados.rados.0.smithi107.stdout:9401: finishing write tid 5 to smithi10729407-3 2020-06-12T02:39:04.640 INFO:tasks.rados.rados.0.smithi107.stdout:9404: done (1 left) 2020-06-12T02:39:04.641 INFO:tasks.rados.rados.0.smithi107.stdout:9405: delete oid 4 current snap is 0 2020-06-12T02:39:04.641 INFO:tasks.rados.rados.0.smithi107.stdout:9401: finishing write tid 6 to smithi10729407-3 2020-06-12T02:39:04.641 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1669 (ObjNum 9400 snap 0 seq_num 9400) dirty exists 2020-06-12T02:39:04.641 INFO:tasks.rados.rados.0.smithi107.stdout:9401: left oid 3 (ObjNum 9400 snap 0 seq_num 9400) 2020-06-12T02:39:04.643 INFO:tasks.rados.rados.0.smithi107.stdout:9401: done (1 left) 2020-06-12T02:39:04.644 INFO:tasks.rados.rados.0.smithi107.stdout:9405: done (0 left) 2020-06-12T02:39:04.644 INFO:tasks.rados.rados.0.smithi107.stdout:9406: delete oid 5 current snap is 0 2020-06-12T02:39:04.646 INFO:tasks.rados.rados.0.smithi107.stdout:9406: done (0 left) 2020-06-12T02:39:04.646 INFO:tasks.rados.rados.0.smithi107.stdout:9407: delete oid 15 current snap is 0 2020-06-12T02:39:04.648 INFO:tasks.rados.rados.0.smithi107.stdout:9407: done (0 left) 2020-06-12T02:39:04.648 INFO:tasks.rados.rados.0.smithi107.stdout:9408: delete oid 8 current snap is 0 2020-06-12T02:39:04.651 INFO:tasks.rados.rados.0.smithi107.stdout:9408: done (0 left) 2020-06-12T02:39:04.652 INFO:tasks.rados.rados.0.smithi107.stdout:9409: delete oid 7 current snap is 0 2020-06-12T02:39:04.653 INFO:tasks.rados.rados.0.smithi107.stdout:9409: done (0 left) 2020-06-12T02:39:04.654 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:39:04.654 INFO:tasks.rados.rados.0.smithi107.stdout:9410: seq_num 9409 ranges {0=786432,786432=737280,1523712=65536} 2020-06-12T02:39:04.662 INFO:tasks.rados.rados.0.smithi107.stdout:9410: writing smithi10729407-12 from 0 to 786432 tid 1 2020-06-12T02:39:04.662 INFO:tasks.rados.rados.0.smithi107.stdout:9410: writing smithi10729407-12 from 786432 to 1523712 tid 2 2020-06-12T02:39:04.662 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:39:04.665 INFO:tasks.rados.rados.0.smithi107.stdout:9410: writing smithi10729407-12 from 1523712 to 1589248 tid 3 2020-06-12T02:39:04.665 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:39:04.665 INFO:tasks.rados.rados.0.smithi107.stdout:9411: seq_num 9410 ranges {0=802816,802816=622592,1425408=180224} 2020-06-12T02:39:04.667 INFO:tasks.rados.rados.0.smithi107.stdout:9411: writing smithi10729407-15 from 0 to 802816 tid 1 2020-06-12T02:39:04.670 INFO:tasks.rados.rados.0.smithi107.stdout:9411: writing smithi10729407-15 from 802816 to 1425408 tid 2 2020-06-12T02:39:04.674 INFO:tasks.rados.rados.0.smithi107.stdout:9411: writing smithi10729407-15 from 1425408 to 1605632 tid 3 2020-06-12T02:39:04.756 INFO:tasks.rados.rados.0.smithi107.stdout:9412: delete oid 9 current snap is 0 2020-06-12T02:39:04.756 INFO:tasks.rados.rados.0.smithi107.stdout:9410: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:04.756 INFO:tasks.rados.rados.0.smithi107.stdout:9410: finishing write tid 2 to smithi10729407-12 2020-06-12T02:39:04.756 INFO:tasks.rados.rados.0.smithi107.stdout:9410: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:04.757 INFO:tasks.rados.rados.0.smithi107.stdout:9412: done (2 left) 2020-06-12T02:39:04.757 INFO:tasks.rados.rados.0.smithi107.stdout:9413: delete oid 2 current snap is 0 2020-06-12T02:39:04.757 INFO:tasks.rados.rados.0.smithi107.stdout:9410: finishing write tid 5 to smithi10729407-12 2020-06-12T02:39:04.767 INFO:tasks.rados.rados.0.smithi107.stdout:9413: done (2 left) 2020-06-12T02:39:04.768 INFO:tasks.rados.rados.0.smithi107.stdout:9414: delete oid 18 current snap is 0 2020-06-12T02:39:04.768 INFO:tasks.rados.rados.0.smithi107.stdout:9410: finishing write tid 6 to smithi10729407-12 2020-06-12T02:39:04.768 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1734 (ObjNum 9409 snap 0 seq_num 9409) dirty exists 2020-06-12T02:39:04.768 INFO:tasks.rados.rados.0.smithi107.stdout:9410: left oid 12 (ObjNum 9409 snap 0 seq_num 9409) 2020-06-12T02:39:04.768 INFO:tasks.rados.rados.0.smithi107.stdout:9410: done (2 left) 2020-06-12T02:39:04.769 INFO:tasks.rados.rados.0.smithi107.stdout:9414: done (1 left) 2020-06-12T02:39:04.769 INFO:tasks.rados.rados.0.smithi107.stdout:9415: delete oid 6 current snap is 0 2020-06-12T02:39:04.769 INFO:tasks.rados.rados.0.smithi107.stdout:9411: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:04.769 INFO:tasks.rados.rados.0.smithi107.stdout:9411: finishing write tid 2 to smithi10729407-15 2020-06-12T02:39:04.769 INFO:tasks.rados.rados.0.smithi107.stdout:9411: finishing write tid 3 to smithi10729407-15 2020-06-12T02:39:04.770 INFO:tasks.rados.rados.0.smithi107.stdout:9415: done (1 left) 2020-06-12T02:39:04.770 INFO:tasks.rados.rados.0.smithi107.stdout:9416: delete oid 14 current snap is 0 2020-06-12T02:39:04.770 INFO:tasks.rados.rados.0.smithi107.stdout:9411: finishing write tid 5 to smithi10729407-15 2020-06-12T02:39:04.770 INFO:tasks.rados.rados.0.smithi107.stdout:9411: finishing write tid 6 to smithi10729407-15 2020-06-12T02:39:04.770 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1753 (ObjNum 9410 snap 0 seq_num 9410) dirty exists 2020-06-12T02:39:04.770 INFO:tasks.rados.rados.0.smithi107.stdout:9411: left oid 15 (ObjNum 9410 snap 0 seq_num 9410) 2020-06-12T02:39:04.771 INFO:tasks.rados.rados.0.smithi107.stdout:9411: done (1 left) 2020-06-12T02:39:04.771 INFO:tasks.rados.rados.0.smithi107.stdout:9416: done (0 left) 2020-06-12T02:39:04.771 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:39:04.771 INFO:tasks.rados.rados.0.smithi107.stdout:9417: seq_num 9416 ranges {1589248=507904,2097152=778240,2875392=425984} 2020-06-12T02:39:04.771 INFO:tasks.rados.rados.0.smithi107.stdout:9417: writing smithi10729407-12 from 1589248 to 2097152 tid 1 2020-06-12T02:39:04.772 INFO:tasks.rados.rados.0.smithi107.stdout:9417: writing smithi10729407-12 from 2097152 to 2875392 tid 2 2020-06-12T02:39:04.772 INFO:tasks.rados.rados.0.smithi107.stdout:9417: writing smithi10729407-12 from 2875392 to 3301376 tid 3 2020-06-12T02:39:04.772 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:39:04.772 INFO:tasks.rados.rados.0.smithi107.stdout:9418: seq_num 9417 ranges {2744320=524288,3268608=663552,3932160=540672} 2020-06-12T02:39:04.772 INFO:tasks.rados.rados.0.smithi107.stdout:9418: writing smithi10729407-3 from 2744320 to 3268608 tid 1 2020-06-12T02:39:04.772 INFO:tasks.rados.rados.0.smithi107.stdout:9418: writing smithi10729407-3 from 3268608 to 3932160 tid 2 2020-06-12T02:39:04.777 INFO:tasks.rados.rados.0.smithi107.stdout:9418: writing smithi10729407-3 from 3932160 to 4472832 tid 3 2020-06-12T02:39:04.777 INFO:tasks.rados.rados.0.smithi107.stdout:9419: delete oid 9 current snap is 0 2020-06-12T02:39:04.777 INFO:tasks.rados.rados.0.smithi107.stdout:9417: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:04.778 INFO:tasks.rados.rados.0.smithi107.stdout:9417: finishing write tid 2 to smithi10729407-12 2020-06-12T02:39:04.778 INFO:tasks.rados.rados.0.smithi107.stdout:9417: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:04.780 INFO:tasks.rados.rados.0.smithi107.stdout:9417: finishing write tid 5 to smithi10729407-12 2020-06-12T02:39:04.780 INFO:tasks.rados.rados.0.smithi107.stdout:9417: finishing write tid 6 to smithi10729407-12 2020-06-12T02:39:04.780 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1738 (ObjNum 9416 snap 0 seq_num 9416) dirty exists 2020-06-12T02:39:04.780 INFO:tasks.rados.rados.0.smithi107.stdout:9417: left oid 12 (ObjNum 9416 snap 0 seq_num 9416) 2020-06-12T02:39:04.780 INFO:tasks.rados.rados.0.smithi107.stdout:9418: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:04.781 INFO:tasks.rados.rados.0.smithi107.stdout:9417: done (2 left) 2020-06-12T02:39:04.781 INFO:tasks.rados.rados.0.smithi107.stdout:9419: done (1 left) 2020-06-12T02:39:04.782 INFO:tasks.rados.rados.0.smithi107.stdout:9420: delete oid 12 current snap is 0 2020-06-12T02:39:04.782 INFO:tasks.rados.rados.0.smithi107.stdout:9420: done (1 left) 2020-06-12T02:39:04.782 INFO:tasks.rados.rados.0.smithi107.stdout:9421: delete oid 17 current snap is 0 2020-06-12T02:39:04.783 INFO:tasks.rados.rados.0.smithi107.stdout:9421: done (1 left) 2020-06-12T02:39:04.783 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:39:04.783 INFO:tasks.rados.rados.0.smithi107.stdout:9422: seq_num 9421 ranges {1605632=589824,2195456=647168,2842624=557056} 2020-06-12T02:39:04.784 INFO:tasks.rados.rados.0.smithi107.stdout:9422: writing smithi10729407-15 from 1605632 to 2195456 tid 1 2020-06-12T02:39:04.784 INFO:tasks.rados.rados.0.smithi107.stdout:9422: writing smithi10729407-15 from 2195456 to 2842624 tid 2 2020-06-12T02:39:04.784 INFO:tasks.rados.rados.0.smithi107.stdout:9422: writing smithi10729407-15 from 2842624 to 3399680 tid 3 2020-06-12T02:39:04.784 INFO:tasks.rados.rados.0.smithi107.stdout:9423: delete oid 8 current snap is 0 2020-06-12T02:39:04.784 INFO:tasks.rados.rados.0.smithi107.stdout:9418: finishing write tid 2 to smithi10729407-3 2020-06-12T02:39:04.785 INFO:tasks.rados.rados.0.smithi107.stdout:9418: finishing write tid 3 to smithi10729407-3 2020-06-12T02:39:04.785 INFO:tasks.rados.rados.0.smithi107.stdout:9418: finishing write tid 5 to smithi10729407-3 2020-06-12T02:39:04.785 INFO:tasks.rados.rados.0.smithi107.stdout:9418: finishing write tid 6 to smithi10729407-3 2020-06-12T02:39:04.785 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1674 (ObjNum 9417 snap 0 seq_num 9417) dirty exists 2020-06-12T02:39:04.785 INFO:tasks.rados.rados.0.smithi107.stdout:9418: left oid 3 (ObjNum 9417 snap 0 seq_num 9417) 2020-06-12T02:39:04.785 INFO:tasks.rados.rados.0.smithi107.stdout:9422: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:04.786 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:39:04.794 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:39:04.794 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:39:04.794 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2020-06-12T02:39:04.801 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-06-12T02:39:04.801 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-06-12T02:39:04.801 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-06-12T02:39:04.801 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-06-12T02:39:04.802 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-06-12T02:39:04.802 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-06-12T02:39:04.803 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-06-12T02:39:04.803 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-06-12T02:39:04.808 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:39:04.756+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:39:04.809 INFO:tasks.rados.rados.0.smithi107.stdout:9422: finishing write tid 2 to smithi10729407-15 2020-06-12T02:39:04.809 INFO:tasks.rados.rados.0.smithi107.stdout:9422: finishing write tid 3 to smithi10729407-15 2020-06-12T02:39:04.809 INFO:tasks.rados.rados.0.smithi107.stdout:9422: finishing write tid 5 to smithi10729407-15 2020-06-12T02:39:04.809 INFO:tasks.rados.rados.0.smithi107.stdout:9418: done (2 left) 2020-06-12T02:39:04.810 INFO:tasks.rados.rados.0.smithi107.stdout:9423: done (1 left) 2020-06-12T02:39:04.810 INFO:tasks.rados.rados.0.smithi107.stdout:9424: delete oid 14 current snap is 0 2020-06-12T02:39:04.810 INFO:tasks.rados.rados.0.smithi107.stdout:9422: finishing write tid 6 to smithi10729407-15 2020-06-12T02:39:04.810 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1757 (ObjNum 9421 snap 0 seq_num 9421) dirty exists 2020-06-12T02:39:04.810 INFO:tasks.rados.rados.0.smithi107.stdout:9422: left oid 15 (ObjNum 9421 snap 0 seq_num 9421) 2020-06-12T02:39:04.811 INFO:tasks.rados.rados.0.smithi107.stdout:9422: done (1 left) 2020-06-12T02:39:04.811 INFO:tasks.rados.rados.0.smithi107.stdout:9424: done (0 left) 2020-06-12T02:39:04.811 INFO:tasks.rados.rados.0.smithi107.stdout:9425: delete oid 2 current snap is 0 2020-06-12T02:39:04.811 INFO:tasks.rados.rados.0.smithi107.stdout:9425: done (0 left) 2020-06-12T02:39:04.811 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:39:04.812 INFO:tasks.rados.rados.0.smithi107.stdout:9426: seq_num 9425 ranges {0=655360,655360=589824,1245184=614400} 2020-06-12T02:39:04.812 INFO:tasks.rados.rados.0.smithi107.stdout:9426: writing smithi10729407-13 from 0 to 655360 tid 1 2020-06-12T02:39:04.812 INFO:tasks.rados.rados.0.smithi107.stdout:9426: writing smithi10729407-13 from 655360 to 1245184 tid 2 2020-06-12T02:39:04.812 INFO:tasks.rados.rados.0.smithi107.stdout:9426: writing smithi10729407-13 from 1245184 to 1859584 tid 3 2020-06-12T02:39:04.812 INFO:tasks.rados.rados.0.smithi107.stdout:9427: delete oid 7 current snap is 0 2020-06-12T02:39:04.812 INFO:tasks.rados.rados.0.smithi107.stdout:9426: finishing write tid 1 to smithi10729407-13 2020-06-12T02:39:04.813 INFO:tasks.rados.rados.0.smithi107.stdout:9426: finishing write tid 2 to smithi10729407-13 2020-06-12T02:39:04.813 INFO:tasks.rados.rados.0.smithi107.stdout:9426: finishing write tid 3 to smithi10729407-13 2020-06-12T02:39:04.813 INFO:tasks.rados.rados.0.smithi107.stdout:9427: done (1 left) 2020-06-12T02:39:04.813 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:39:04.813 INFO:tasks.rados.rados.0.smithi107.stdout:9428: seq_num 9427 ranges {0=688128,688128=802816,1490944=401408} 2020-06-12T02:39:04.814 INFO:tasks.rados.rados.0.smithi107.stdout:9428: writing smithi10729407-12 from 0 to 688128 tid 1 2020-06-12T02:39:04.814 INFO:tasks.rados.rados.0.smithi107.stdout:9428: writing smithi10729407-12 from 688128 to 1490944 tid 2 2020-06-12T02:39:04.814 INFO:tasks.rados.rados.0.smithi107.stdout:9428: writing smithi10729407-12 from 1490944 to 1892352 tid 3 2020-06-12T02:39:04.814 INFO:tasks.rados.rados.0.smithi107.stdout:9429: delete oid 3 current snap is 0 2020-06-12T02:39:04.814 INFO:tasks.rados.rados.0.smithi107.stdout:9426: finishing write tid 5 to smithi10729407-13 2020-06-12T02:39:04.815 INFO:tasks.rados.rados.0.smithi107.stdout:9426: finishing write tid 6 to smithi10729407-13 2020-06-12T02:39:04.815 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1672 (ObjNum 9425 snap 0 seq_num 9425) dirty exists 2020-06-12T02:39:04.815 INFO:tasks.rados.rados.0.smithi107.stdout:9426: left oid 13 (ObjNum 9425 snap 0 seq_num 9425) 2020-06-12T02:39:04.816 INFO:tasks.rados.rados.0.smithi107.stdout:9426: done (2 left) 2020-06-12T02:39:04.816 INFO:tasks.rados.rados.0.smithi107.stdout:9429: done (1 left) 2020-06-12T02:39:04.816 INFO:tasks.rados.rados.0.smithi107.stdout:9430: delete oid 6 current snap is 0 2020-06-12T02:39:04.817 INFO:tasks.rados.rados.0.smithi107.stdout:9428: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:04.817 INFO:tasks.rados.rados.0.smithi107.stdout:9430: done (1 left) 2020-06-12T02:39:04.817 INFO:tasks.rados.rados.0.smithi107.stdout:9431: delete oid 5 current snap is 0 2020-06-12T02:39:04.818 INFO:tasks.rados.rados.0.smithi107.stdout:9428: finishing write tid 2 to smithi10729407-12 2020-06-12T02:39:04.819 INFO:tasks.rados.rados.0.smithi107.stdout:9428: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:04.819 INFO:tasks.rados.rados.0.smithi107.stdout:9428: finishing write tid 5 to smithi10729407-12 2020-06-12T02:39:04.819 INFO:tasks.rados.rados.0.smithi107.stdout:9431: done (1 left) 2020-06-12T02:39:04.819 INFO:tasks.rados.rados.0.smithi107.stdout:9432: delete oid 19 current snap is 0 2020-06-12T02:39:04.823 INFO:tasks.rados.rados.0.smithi107.stdout:9432: done (1 left) 2020-06-12T02:39:04.823 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:39:04.824 INFO:tasks.rados.rados.0.smithi107.stdout:9433: seq_num 9432 ranges {0=778240,778240=630784,1409024=475136,1884160=98304} 2020-06-12T02:39:04.830 INFO:tasks.rados.rados.0.smithi107.stdout:9433: writing smithi10729407-20 from 0 to 778240 tid 1 2020-06-12T02:39:04.832 INFO:tasks.rados.rados.0.smithi107.stdout:9433: writing smithi10729407-20 from 778240 to 1409024 tid 2 2020-06-12T02:39:04.836 INFO:tasks.rados.rados.0.smithi107.stdout:9433: writing smithi10729407-20 from 1409024 to 1884160 tid 3 2020-06-12T02:39:04.837 INFO:tasks.rados.rados.0.smithi107.stdout:9433: writing smithi10729407-20 from 1884160 to 1982464 tid 4 2020-06-12T02:39:04.838 INFO:tasks.rados.rados.0.smithi107.stdout:9434: delete oid 3 current snap is 0 2020-06-12T02:39:04.838 INFO:tasks.rados.rados.0.smithi107.stdout:9428: finishing write tid 6 to smithi10729407-12 2020-06-12T02:39:04.839 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1743 (ObjNum 9427 snap 0 seq_num 9427) dirty exists 2020-06-12T02:39:04.839 INFO:tasks.rados.rados.0.smithi107.stdout:9428: left oid 12 (ObjNum 9427 snap 0 seq_num 9427) 2020-06-12T02:39:04.839 INFO:tasks.rados.rados.0.smithi107.stdout:9433: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:04.843 INFO:tasks.rados.rados.0.smithi107.stdout:9428: done (2 left) 2020-06-12T02:39:04.843 INFO:tasks.rados.rados.0.smithi107.stdout:9434: done (1 left) 2020-06-12T02:39:04.844 INFO:tasks.rados.rados.0.smithi107.stdout:9435: delete oid 2 current snap is 0 2020-06-12T02:39:04.844 INFO:tasks.rados.rados.0.smithi107.stdout:9433: finishing write tid 2 to smithi10729407-20 2020-06-12T02:39:04.844 INFO:tasks.rados.rados.0.smithi107.stdout:9433: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:04.846 INFO:tasks.rados.rados.0.smithi107.stdout:9433: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:04.847 INFO:tasks.rados.rados.0.smithi107.stdout:9435: done (1 left) 2020-06-12T02:39:04.847 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:39:04.848 INFO:tasks.rados.rados.0.smithi107.stdout:9436: seq_num 9435 ranges {0=425984,425984=729088,1155072=696320,1851392=180224} 2020-06-12T02:39:04.848 INFO:tasks.rados.rados.0.smithi107.stdout:9436: writing smithi10729407-18 from 0 to 425984 tid 1 2020-06-12T02:39:04.853 INFO:tasks.rados.rados.0.smithi107.stdout:9436: writing smithi10729407-18 from 425984 to 1155072 tid 2 2020-06-12T02:39:04.857 INFO:tasks.rados.rados.0.smithi107.stdout:9436: writing smithi10729407-18 from 1155072 to 1851392 tid 3 2020-06-12T02:39:04.859 INFO:tasks.rados.rados.0.smithi107.stdout:9436: writing smithi10729407-18 from 1851392 to 2031616 tid 4 2020-06-12T02:39:04.859 INFO:tasks.rados.rados.0.smithi107.stdout:9437: delete oid 7 current snap is 0 2020-06-12T02:39:04.860 INFO:tasks.rados.rados.0.smithi107.stdout:9433: finishing write tid 6 to smithi10729407-20 2020-06-12T02:39:04.860 INFO:tasks.rados.rados.0.smithi107.stdout:9433: finishing write tid 7 to smithi10729407-20 2020-06-12T02:39:04.860 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 702 (ObjNum 9432 snap 0 seq_num 9432) dirty exists 2020-06-12T02:39:04.860 INFO:tasks.rados.rados.0.smithi107.stdout:9433: left oid 20 (ObjNum 9432 snap 0 seq_num 9432) 2020-06-12T02:39:04.860 INFO:tasks.rados.rados.0.smithi107.stdout:9436: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:04.861 INFO:tasks.rados.rados.0.smithi107.stdout:9433: done (2 left) 2020-06-12T02:39:04.861 INFO:tasks.rados.rados.0.smithi107.stdout:9437: done (1 left) 2020-06-12T02:39:04.861 INFO:tasks.rados.rados.0.smithi107.stdout:9438: delete oid 3 current snap is 0 2020-06-12T02:39:04.863 INFO:tasks.rados.rados.0.smithi107.stdout:9438: done (1 left) 2020-06-12T02:39:04.863 INFO:tasks.rados.rados.0.smithi107.stdout:9439: delete oid 9 current snap is 0 2020-06-12T02:39:04.865 INFO:tasks.rados.rados.0.smithi107.stdout:9436: finishing write tid 2 to smithi10729407-18 2020-06-12T02:39:04.866 INFO:tasks.rados.rados.0.smithi107.stdout:9436: finishing write tid 3 to smithi10729407-18 2020-06-12T02:39:04.867 INFO:tasks.rados.rados.0.smithi107.stdout:9436: finishing write tid 4 to smithi10729407-18 2020-06-12T02:39:04.867 INFO:tasks.rados.rados.0.smithi107.stdout:9439: done (1 left) 2020-06-12T02:39:04.867 INFO:tasks.rados.rados.0.smithi107.stdout:9440: delete oid 1 current snap is 0 2020-06-12T02:39:04.868 INFO:tasks.rados.rados.0.smithi107.stdout:9436: finishing write tid 6 to smithi10729407-18 2020-06-12T02:39:04.870 INFO:tasks.rados.rados.0.smithi107.stdout:9436: finishing write tid 7 to smithi10729407-18 2020-06-12T02:39:04.870 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1749 (ObjNum 9435 snap 0 seq_num 9435) dirty exists 2020-06-12T02:39:04.870 INFO:tasks.rados.rados.0.smithi107.stdout:9436: left oid 18 (ObjNum 9435 snap 0 seq_num 9435) 2020-06-12T02:39:04.870 INFO:tasks.rados.rados.0.smithi107.stdout:9436: done (1 left) 2020-06-12T02:39:04.871 INFO:tasks.rados.rados.0.smithi107.stdout:9440: done (0 left) 2020-06-12T02:39:04.871 INFO:tasks.rados.rados.0.smithi107.stdout:9441: delete oid 4 current snap is 0 2020-06-12T02:39:04.872 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:39:04.870+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:39:04.873 INFO:tasks.rados.rados.0.smithi107.stdout:9441: done (0 left) 2020-06-12T02:39:04.873 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:39:04.873 INFO:tasks.rados.rados.0.smithi107.stdout:9442: seq_num 9441 ranges {0=524288,524288=434176,958464=647168,1605632=417792,2023424=106496} 2020-06-12T02:39:04.875 INFO:tasks.rados.rados.0.smithi107.stdout:9442: writing smithi10729407-5 from 0 to 524288 tid 1 2020-06-12T02:39:04.880 INFO:tasks.rados.rados.0.smithi107.stdout:9442: writing smithi10729407-5 from 524288 to 958464 tid 2 2020-06-12T02:39:04.883 INFO:tasks.rados.rados.0.smithi107.stdout:9442: writing smithi10729407-5 from 958464 to 1605632 tid 3 2020-06-12T02:39:04.885 INFO:tasks.rados.rados.0.smithi107.stdout:9442: writing smithi10729407-5 from 1605632 to 2023424 tid 4 2020-06-12T02:39:04.886 INFO:tasks.rados.rados.0.smithi107.stdout:9442: writing smithi10729407-5 from 2023424 to 2129920 tid 5 2020-06-12T02:39:04.886 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:39:04.887 INFO:tasks.rados.rados.0.smithi107.stdout:9443: seq_num 9442 ranges {0=540672,540672=720896,1261568=450560,1712128=434176} 2020-06-12T02:39:04.889 INFO:tasks.rados.rados.0.smithi107.stdout:9443: writing smithi10729407-11 from 0 to 540672 tid 1 2020-06-12T02:39:04.893 INFO:tasks.rados.rados.0.smithi107.stdout:9443: writing smithi10729407-11 from 540672 to 1261568 tid 2 2020-06-12T02:39:04.897 INFO:tasks.rados.rados.0.smithi107.stdout:9443: writing smithi10729407-11 from 1261568 to 1712128 tid 3 2020-06-12T02:39:04.900 INFO:tasks.rados.rados.0.smithi107.stdout:9443: writing smithi10729407-11 from 1712128 to 2146304 tid 4 2020-06-12T02:39:04.900 INFO:tasks.rados.rados.0.smithi107.stdout:9444: delete oid 16 current snap is 0 2020-06-12T02:39:04.901 INFO:tasks.rados.rados.0.smithi107.stdout:9442: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:04.901 INFO:tasks.rados.rados.0.smithi107.stdout:9442: finishing write tid 2 to smithi10729407-5 2020-06-12T02:39:04.901 INFO:tasks.rados.rados.0.smithi107.stdout:9442: finishing write tid 3 to smithi10729407-5 2020-06-12T02:39:04.901 INFO:tasks.rados.rados.0.smithi107.stdout:9442: finishing write tid 4 to smithi10729407-5 2020-06-12T02:39:04.902 INFO:tasks.rados.rados.0.smithi107.stdout:9442: finishing write tid 5 to smithi10729407-5 2020-06-12T02:39:04.902 INFO:tasks.rados.rados.0.smithi107.stdout:9443: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:04.902 INFO:tasks.rados.rados.0.smithi107.stdout:9442: finishing write tid 7 to smithi10729407-5 2020-06-12T02:39:04.903 INFO:tasks.rados.rados.0.smithi107.stdout:9442: finishing write tid 8 to smithi10729407-5 2020-06-12T02:39:04.903 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 921 (ObjNum 9441 snap 0 seq_num 9441) dirty exists 2020-06-12T02:39:04.903 INFO:tasks.rados.rados.0.smithi107.stdout:9442: left oid 5 (ObjNum 9441 snap 0 seq_num 9441) 2020-06-12T02:39:04.904 INFO:tasks.rados.rados.0.smithi107.stdout:9442: done (2 left) 2020-06-12T02:39:04.904 INFO:tasks.rados.rados.0.smithi107.stdout:9444: done (1 left) 2020-06-12T02:39:04.904 INFO:tasks.rados.rados.0.smithi107.stdout:9445: delete oid 7 current snap is 0 2020-06-12T02:39:04.906 INFO:tasks.rados.rados.0.smithi107.stdout:9445: done (1 left) 2020-06-12T02:39:04.906 INFO:tasks.rados.rados.0.smithi107.stdout:9446: delete oid 20 current snap is 0 2020-06-12T02:39:04.906 INFO:tasks.rados.rados.0.smithi107.stdout:9443: finishing write tid 2 to smithi10729407-11 2020-06-12T02:39:04.911 INFO:tasks.rados.rados.0.smithi107.stdout:9443: finishing write tid 3 to smithi10729407-11 2020-06-12T02:39:04.911 INFO:tasks.rados.rados.0.smithi107.stdout:9446: done (1 left) 2020-06-12T02:39:04.912 INFO:tasks.rados.rados.0.smithi107.stdout:9447: delete oid 9 current snap is 0 2020-06-12T02:39:04.912 INFO:tasks.rados.rados.0.smithi107.stdout:9443: finishing write tid 4 to smithi10729407-11 2020-06-12T02:39:04.912 INFO:tasks.rados.rados.0.smithi107.stdout:9443: finishing write tid 6 to smithi10729407-11 2020-06-12T02:39:04.913 INFO:tasks.rados.rados.0.smithi107.stdout:9447: done (1 left) 2020-06-12T02:39:04.913 INFO:tasks.rados.rados.0.smithi107.stdout:9448: delete oid 13 current snap is 0 2020-06-12T02:39:04.913 INFO:tasks.rados.rados.0.smithi107.stdout:9443: finishing write tid 7 to smithi10729407-11 2020-06-12T02:39:04.914 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1623 (ObjNum 9442 snap 0 seq_num 9442) dirty exists 2020-06-12T02:39:04.914 INFO:tasks.rados.rados.0.smithi107.stdout:9443: left oid 11 (ObjNum 9442 snap 0 seq_num 9442) 2020-06-12T02:39:04.914 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:39:04.915 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:39:04.915 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:39:04.915 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:39:04.915 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:39:04.916 INFO:tasks.rados.rados.0.smithi107.stdout:9443: done (1 left) 2020-06-12T02:39:04.916 INFO:tasks.rados.rados.0.smithi107.stdout:9448: done (0 left) 2020-06-12T02:39:04.916 INFO:tasks.rados.rados.0.smithi107.stdout:9449: delete oid 18 current snap is 0 2020-06-12T02:39:04.918 INFO:tasks.rados.rados.0.smithi107.stdout:9449: done (0 left) 2020-06-12T02:39:04.918 INFO:tasks.rados.rados.0.smithi107.stdout:9450: delete oid 3 current snap is 0 2020-06-12T02:39:04.924 INFO:tasks.rados.rados.0.smithi107.stdout:9450: done (0 left) 2020-06-12T02:39:04.924 INFO:tasks.rados.rados.0.smithi107.stdout:9451: delete oid 1 current snap is 0 2020-06-12T02:39:04.924 INFO:tasks.rados.rados.0.smithi107.stdout:9451: done (0 left) 2020-06-12T02:39:04.925 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:39:04.925 INFO:tasks.rados.rados.0.smithi107.stdout:9452: seq_num 9451 ranges {3399680=696320,4096000=581632,4677632=622592,5300224=401408} 2020-06-12T02:39:04.926 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-06-12T02:39:04.946 INFO:tasks.rados.rados.0.smithi107.stdout:9452: writing smithi10729407-15 from 3399680 to 4096000 tid 1 2020-06-12T02:39:04.949 INFO:tasks.rados.rados.0.smithi107.stdout:9452: writing smithi10729407-15 from 4096000 to 4677632 tid 2 2020-06-12T02:39:04.952 INFO:tasks.rados.rados.0.smithi107.stdout:9452: writing smithi10729407-15 from 4677632 to 5300224 tid 3 2020-06-12T02:39:04.956 INFO:tasks.rados.rados.0.smithi107.stdout:9452: writing smithi10729407-15 from 5300224 to 5701632 tid 4 2020-06-12T02:39:04.956 INFO:tasks.rados.rados.0.smithi107.stdout:9453: delete oid 8 current snap is 0 2020-06-12T02:39:04.957 INFO:tasks.rados.rados.0.smithi107.stdout:9452: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:04.962 INFO:tasks.rados.rados.0.smithi107.stdout:9452: finishing write tid 2 to smithi10729407-15 2020-06-12T02:39:04.969 INFO:tasks.rados.rados.0.smithi107.stdout:9452: finishing write tid 3 to smithi10729407-15 2020-06-12T02:39:04.969 INFO:tasks.rados.rados.0.smithi107.stdout:9452: finishing write tid 4 to smithi10729407-15 2020-06-12T02:39:04.969 INFO:tasks.rados.rados.0.smithi107.stdout:9452: finishing write tid 6 to smithi10729407-15 2020-06-12T02:39:04.970 INFO:tasks.rados.rados.0.smithi107.stdout:9452: finishing write tid 7 to smithi10729407-15 2020-06-12T02:39:04.970 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1763 (ObjNum 9451 snap 0 seq_num 9451) dirty exists 2020-06-12T02:39:04.971 INFO:tasks.rados.rados.0.smithi107.stdout:9452: left oid 15 (ObjNum 9451 snap 0 seq_num 9451) 2020-06-12T02:39:04.971 INFO:tasks.rados.rados.0.smithi107.stdout:9452: done (1 left) 2020-06-12T02:39:04.971 INFO:tasks.rados.rados.0.smithi107.stdout:9453: done (0 left) 2020-06-12T02:39:04.971 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:39:04.971 INFO:tasks.rados.rados.0.smithi107.stdout:9454: seq_num 9453 ranges {0=729088,729088=745472,1474560=589824,2064384=270336} 2020-06-12T02:39:04.973 INFO:tasks.rados.rados.0.smithi107.stdout:9454: writing smithi10729407-13 from 0 to 729088 tid 1 2020-06-12T02:39:04.978 INFO:tasks.rados.rados.0.smithi107.stdout:9454: writing smithi10729407-13 from 729088 to 1474560 tid 2 2020-06-12T02:39:04.982 INFO:tasks.rados.rados.0.smithi107.stdout:9454: writing smithi10729407-13 from 1474560 to 2064384 tid 3 2020-06-12T02:39:04.985 INFO:tasks.rados.rados.0.smithi107.stdout:9454: writing smithi10729407-13 from 2064384 to 2334720 tid 4 2020-06-12T02:39:04.985 INFO:tasks.rados.rados.0.smithi107.stdout:9455: delete oid 10 current snap is 0 2020-06-12T02:39:04.985 INFO:tasks.rados.rados.0.smithi107.stdout:9454: finishing write tid 1 to smithi10729407-13 2020-06-12T02:39:04.986 INFO:tasks.rados.rados.0.smithi107.stdout:9455: done (1 left) 2020-06-12T02:39:04.986 INFO:tasks.rados.rados.0.smithi107.stdout:9456: delete oid 18 current snap is 0 2020-06-12T02:39:04.988 INFO:tasks.rados.rados.0.smithi107.stdout:9456: done (1 left) 2020-06-12T02:39:04.988 INFO:tasks.rados.rados.0.smithi107.stdout:9457: delete oid 4 current snap is 0 2020-06-12T02:39:04.992 INFO:tasks.rados.rados.0.smithi107.stdout:9457: done (1 left) 2020-06-12T02:39:04.992 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:39:04.993 INFO:tasks.rados.rados.0.smithi107.stdout:9458: seq_num 9457 ranges {0=794624,794624=688128,1482752=565248,2048000=352256} 2020-06-12T02:39:05.000 INFO:tasks.rados.rados.0.smithi107.stdout:9458: writing smithi10729407-2 from 0 to 794624 tid 1 2020-06-12T02:39:05.001 INFO:tasks.rados.rados.0.smithi107.stdout:9458: writing smithi10729407-2 from 794624 to 1482752 tid 2 2020-06-12T02:39:05.005 INFO:tasks.rados.rados.0.smithi107.stdout:9458: writing smithi10729407-2 from 1482752 to 2048000 tid 3 2020-06-12T02:39:05.007 INFO:tasks.rados.rados.0.smithi107.stdout:9458: writing smithi10729407-2 from 2048000 to 2400256 tid 4 2020-06-12T02:39:05.008 INFO:tasks.rados.rados.0.smithi107.stdout:9459: delete oid 12 current snap is 0 2020-06-12T02:39:05.008 INFO:tasks.rados.rados.0.smithi107.stdout:9454: finishing write tid 2 to smithi10729407-13 2020-06-12T02:39:05.008 INFO:tasks.rados.rados.0.smithi107.stdout:9454: finishing write tid 3 to smithi10729407-13 2020-06-12T02:39:05.008 INFO:tasks.rados.rados.0.smithi107.stdout:9454: finishing write tid 4 to smithi10729407-13 2020-06-12T02:39:05.010 INFO:tasks.rados.rados.0.smithi107.stdout:9454: finishing write tid 6 to smithi10729407-13 2020-06-12T02:39:05.010 INFO:tasks.rados.rados.0.smithi107.stdout:9454: finishing write tid 7 to smithi10729407-13 2020-06-12T02:39:05.010 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1678 (ObjNum 9453 snap 0 seq_num 9453) dirty exists 2020-06-12T02:39:05.010 INFO:tasks.rados.rados.0.smithi107.stdout:9454: left oid 13 (ObjNum 9453 snap 0 seq_num 9453) 2020-06-12T02:39:05.011 INFO:tasks.rados.rados.0.smithi107.stdout:9458: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:05.011 INFO:tasks.rados.rados.0.smithi107.stdout:9454: done (2 left) 2020-06-12T02:39:05.011 INFO:tasks.rados.rados.0.smithi107.stdout:9459: done (1 left) 2020-06-12T02:39:05.011 INFO:tasks.rados.rados.0.smithi107.stdout:9460: delete oid 11 current snap is 0 2020-06-12T02:39:05.014 INFO:tasks.rados.rados.0.smithi107.stdout:9460: done (1 left) 2020-06-12T02:39:05.014 INFO:tasks.rados.rados.0.smithi107.stdout:9461: delete oid 7 current snap is 0 2020-06-12T02:39:05.016 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:39:05.013+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:39:05.017 INFO:tasks.rados.rados.0.smithi107.stdout:9458: finishing write tid 2 to smithi10729407-2 2020-06-12T02:39:05.018 INFO:tasks.rados.rados.0.smithi107.stdout:9461: done (1 left) 2020-06-12T02:39:05.018 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:39:05.018 INFO:tasks.rados.rados.0.smithi107.stdout:9462: seq_num 9461 ranges {0=57344} 2020-06-12T02:39:05.018 INFO:tasks.rados.rados.0.smithi107.stdout:9462: writing smithi10729407-11 from 0 to 57344 tid 1 2020-06-12T02:39:05.018 INFO:tasks.rados.rados.0.smithi107.stdout:9463: delete oid 6 current snap is 0 2020-06-12T02:39:05.019 INFO:tasks.rados.rados.0.smithi107.stdout:9458: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:05.019 INFO:tasks.rados.rados.0.smithi107.stdout:9458: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:05.020 INFO:tasks.rados.rados.0.smithi107.stdout:9458: finishing write tid 6 to smithi10729407-2 2020-06-12T02:39:05.022 INFO:tasks.rados.rados.0.smithi107.stdout:9458: finishing write tid 7 to smithi10729407-2 2020-06-12T02:39:05.024 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 877 (ObjNum 9457 snap 0 seq_num 9457) dirty exists 2020-06-12T02:39:05.024 INFO:tasks.rados.rados.0.smithi107.stdout:9458: left oid 2 (ObjNum 9457 snap 0 seq_num 9457) 2020-06-12T02:39:05.025 INFO:tasks.rados.rados.0.smithi107.stdout:9462: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:05.025 INFO:tasks.rados.rados.0.smithi107.stdout:9458: done (2 left) 2020-06-12T02:39:05.025 INFO:tasks.rados.rados.0.smithi107.stdout:9463: done (1 left) 2020-06-12T02:39:05.026 INFO:tasks.rados.rados.0.smithi107.stdout:9464: delete oid 3 current snap is 0 2020-06-12T02:39:05.027 INFO:tasks.rados.rados.0.smithi107.stdout:9462: finishing write tid 3 to smithi10729407-11 2020-06-12T02:39:05.027 INFO:tasks.rados.rados.0.smithi107.stdout:9464: done (1 left) 2020-06-12T02:39:05.027 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:39:05.027 INFO:tasks.rados.rados.0.smithi107.stdout:9465: seq_num 9464 ranges {0=106496} 2020-06-12T02:39:05.028 INFO:tasks.rados.rados.0.smithi107.stdout:9465: writing smithi10729407-8 from 0 to 106496 tid 1 2020-06-12T02:39:05.028 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:39:05.028 INFO:tasks.rados.rados.0.smithi107.stdout:9466: seq_num 9465 ranges {0=122880} 2020-06-12T02:39:05.028 INFO:tasks.rados.rados.0.smithi107.stdout:9466: writing smithi10729407-17 from 0 to 122880 tid 1 2020-06-12T02:39:05.028 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:39:05.029 INFO:tasks.rados.rados.0.smithi107.stdout:9467: seq_num 9466 ranges {0=139264} 2020-06-12T02:39:05.029 INFO:tasks.rados.rados.0.smithi107.stdout:9467: writing smithi10729407-9 from 0 to 139264 tid 1 2020-06-12T02:39:05.030 INFO:tasks.rados.rados.0.smithi107.stdout:9468: delete oid 14 current snap is 0 2020-06-12T02:39:05.030 INFO:tasks.rados.rados.0.smithi107.stdout:9462: finishing write tid 4 to smithi10729407-11 2020-06-12T02:39:05.030 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1626 (ObjNum 9461 snap 0 seq_num 9461) dirty exists 2020-06-12T02:39:05.030 INFO:tasks.rados.rados.0.smithi107.stdout:9462: left oid 11 (ObjNum 9461 snap 0 seq_num 9461) 2020-06-12T02:39:05.034 INFO:tasks.rados.rados.0.smithi107.stdout:9467: finishing write tid 1 to smithi10729407-9 2020-06-12T02:39:05.034 INFO:tasks.rados.rados.0.smithi107.stdout:9465: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:05.034 INFO:tasks.rados.rados.0.smithi107.stdout:9467: finishing write tid 3 to smithi10729407-9 2020-06-12T02:39:05.034 INFO:tasks.rados.rados.0.smithi107.stdout:9465: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:05.034 INFO:tasks.rados.rados.0.smithi107.stdout:9462: done (4 left) 2020-06-12T02:39:05.035 INFO:tasks.rados.rados.0.smithi107.stdout:9468: done (3 left) 2020-06-12T02:39:05.035 INFO:tasks.rados.rados.0.smithi107.stdout:9469: delete oid 19 current snap is 0 2020-06-12T02:39:05.035 INFO:tasks.rados.rados.0.smithi107.stdout:9466: finishing write tid 1 to smithi10729407-17 2020-06-12T02:39:05.036 INFO:tasks.rados.rados.0.smithi107.stdout:9467: finishing write tid 4 to smithi10729407-9 2020-06-12T02:39:05.037 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1731 (ObjNum 9466 snap 0 seq_num 9466) dirty exists 2020-06-12T02:39:05.037 INFO:tasks.rados.rados.0.smithi107.stdout:9467: left oid 9 (ObjNum 9466 snap 0 seq_num 9466) 2020-06-12T02:39:05.038 INFO:tasks.rados.rados.0.smithi107.stdout:9465: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:05.038 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1766 (ObjNum 9464 snap 0 seq_num 9464) dirty exists 2020-06-12T02:39:05.038 INFO:tasks.rados.rados.0.smithi107.stdout:9465: left oid 8 (ObjNum 9464 snap 0 seq_num 9464) 2020-06-12T02:39:05.039 INFO:tasks.rados.rados.0.smithi107.stdout:9466: finishing write tid 3 to smithi10729407-17 2020-06-12T02:39:05.039 INFO:tasks.rados.rados.0.smithi107.stdout:9465: done (3 left) 2020-06-12T02:39:05.039 INFO:tasks.rados.rados.0.smithi107.stdout:9467: done (2 left) 2020-06-12T02:39:05.039 INFO:tasks.rados.rados.0.smithi107.stdout:9469: done (1 left) 2020-06-12T02:39:05.039 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:39:05.040 INFO:tasks.rados.rados.0.smithi107.stdout:9470: seq_num 9469 ranges {0=188416} 2020-06-12T02:39:05.040 INFO:tasks.rados.rados.0.smithi107.stdout:9470: writing smithi10729407-12 from 0 to 188416 tid 1 2020-06-12T02:39:05.041 INFO:tasks.rados.rados.0.smithi107.stdout:9471: delete oid 16 current snap is 0 2020-06-12T02:39:05.041 INFO:tasks.rados.rados.0.smithi107.stdout:9466: finishing write tid 4 to smithi10729407-17 2020-06-12T02:39:05.041 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1780 (ObjNum 9465 snap 0 seq_num 9465) dirty exists 2020-06-12T02:39:05.041 INFO:tasks.rados.rados.0.smithi107.stdout:9466: left oid 17 (ObjNum 9465 snap 0 seq_num 9465) 2020-06-12T02:39:05.042 INFO:tasks.rados.rados.0.smithi107.stdout:9466: done (2 left) 2020-06-12T02:39:05.042 INFO:tasks.rados.rados.0.smithi107.stdout:9471: done (1 left) 2020-06-12T02:39:05.042 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:39:05.042 INFO:tasks.rados.rados.0.smithi107.stdout:9472: seq_num 9471 ranges {122880=229376} 2020-06-12T02:39:05.044 INFO:tasks.rados.rados.0.smithi107.stdout:9472: writing smithi10729407-17 from 122880 to 352256 tid 1 2020-06-12T02:39:05.044 INFO:tasks.rados.rados.0.smithi107.stdout:9473: delete oid 16 current snap is 0 2020-06-12T02:39:05.044 INFO:tasks.rados.rados.0.smithi107.stdout:9470: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:05.047 INFO:tasks.rados.rados.0.smithi107.stdout:9470: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:05.047 INFO:tasks.rados.rados.0.smithi107.stdout:9473: done (2 left) 2020-06-12T02:39:05.047 INFO:tasks.rados.rados.0.smithi107.stdout:9474: delete oid 7 current snap is 0 2020-06-12T02:39:05.049 INFO:tasks.rados.rados.0.smithi107.stdout:9470: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:05.049 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1748 (ObjNum 9469 snap 0 seq_num 9469) dirty exists 2020-06-12T02:39:05.049 INFO:tasks.rados.rados.0.smithi107.stdout:9470: left oid 12 (ObjNum 9469 snap 0 seq_num 9469) 2020-06-12T02:39:05.049 INFO:tasks.rados.rados.0.smithi107.stdout:9472: finishing write tid 1 to smithi10729407-17 2020-06-12T02:39:05.050 INFO:tasks.rados.rados.0.smithi107.stdout:9470: done (2 left) 2020-06-12T02:39:05.050 INFO:tasks.rados.rados.0.smithi107.stdout:9474: done (1 left) 2020-06-12T02:39:05.051 INFO:tasks.rados.rados.0.smithi107.stdout:9475: delete oid 16 current snap is 0 2020-06-12T02:39:05.053 INFO:tasks.rados.rados.0.smithi107.stdout:9475: done (1 left) 2020-06-12T02:39:05.054 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:39:05.054 INFO:tasks.rados.rados.0.smithi107.stdout:9476: seq_num 9475 ranges {2400256=294912} 2020-06-12T02:39:05.068 INFO:tasks.rados.rados.0.smithi107.stdout:9476: writing smithi10729407-2 from 2400256 to 2695168 tid 1 2020-06-12T02:39:05.068 INFO:tasks.rados.rados.0.smithi107.stdout:9477: delete oid 8 current snap is 0 2020-06-12T02:39:05.068 INFO:tasks.rados.rados.0.smithi107.stdout:9472: finishing write tid 3 to smithi10729407-17 2020-06-12T02:39:05.068 INFO:tasks.rados.rados.0.smithi107.stdout:9472: finishing write tid 4 to smithi10729407-17 2020-06-12T02:39:05.069 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1782 (ObjNum 9471 snap 0 seq_num 9471) dirty exists 2020-06-12T02:39:05.069 INFO:tasks.rados.rados.0.smithi107.stdout:9472: left oid 17 (ObjNum 9471 snap 0 seq_num 9471) 2020-06-12T02:39:05.070 INFO:tasks.rados.rados.0.smithi107.stdout:9472: done (2 left) 2020-06-12T02:39:05.071 INFO:tasks.rados.rados.0.smithi107.stdout:9477: done (1 left) 2020-06-12T02:39:05.071 INFO:tasks.rados.rados.0.smithi107.stdout:9478: delete oid 14 current snap is 0 2020-06-12T02:39:05.072 INFO:tasks.rados.rados.0.smithi107.stdout:9476: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:05.074 INFO:tasks.rados.rados.0.smithi107.stdout:9478: done (1 left) 2020-06-12T02:39:05.075 INFO:tasks.rados.rados.0.smithi107.stdout:9479: delete oid 16 current snap is 0 2020-06-12T02:39:05.075 INFO:tasks.rados.rados.0.smithi107.stdout:9476: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:05.076 INFO:tasks.rados.rados.0.smithi107.stdout:9476: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:05.077 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 879 (ObjNum 9475 snap 0 seq_num 9475) dirty exists 2020-06-12T02:39:05.077 INFO:tasks.rados.rados.0.smithi107.stdout:9476: left oid 2 (ObjNum 9475 snap 0 seq_num 9475) 2020-06-12T02:39:05.077 INFO:tasks.rados.rados.0.smithi107.stdout:9476: done (1 left) 2020-06-12T02:39:05.077 INFO:tasks.rados.rados.0.smithi107.stdout:9479: done (0 left) 2020-06-12T02:39:05.078 INFO:tasks.rados.rados.0.smithi107.stdout:9480: delete oid 9 current snap is 0 2020-06-12T02:39:05.078 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:39:05.075+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:39:05.079 INFO:tasks.rados.rados.0.smithi107.stdout:9480: done (0 left) 2020-06-12T02:39:05.079 INFO:tasks.rados.rados.0.smithi107.stdout:9481: delete oid 1 current snap is 0 2020-06-12T02:39:05.082 INFO:tasks.rados.rados.0.smithi107.stdout:9481: done (0 left) 2020-06-12T02:39:05.083 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:39:05.083 INFO:tasks.rados.rados.0.smithi107.stdout:9482: seq_num 9481 ranges {2695168=393216} 2020-06-12T02:39:05.098 INFO:tasks.rados.rados.0.smithi107.stdout:9482: writing smithi10729407-2 from 2695168 to 3088384 tid 1 2020-06-12T02:39:05.099 INFO:tasks.rados.rados.0.smithi107.stdout:9483: delete oid 4 current snap is 0 2020-06-12T02:39:05.102 INFO:tasks.rados.rados.0.smithi107.stdout:9483: done (1 left) 2020-06-12T02:39:05.102 INFO:tasks.rados.rados.0.smithi107.stdout:9484: delete oid 6 current snap is 0 2020-06-12T02:39:05.104 INFO:tasks.rados.rados.0.smithi107.stdout:9482: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:05.105 INFO:tasks.rados.rados.0.smithi107.stdout:9484: done (1 left) 2020-06-12T02:39:05.105 INFO:tasks.rados.rados.0.smithi107.stdout:9485: delete oid 8 current snap is 0 2020-06-12T02:39:05.105 INFO:tasks.rados.rados.0.smithi107.stdout:9482: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:05.106 INFO:tasks.rados.rados.0.smithi107.stdout:9482: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:05.107 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 881 (ObjNum 9481 snap 0 seq_num 9481) dirty exists 2020-06-12T02:39:05.107 INFO:tasks.rados.rados.0.smithi107.stdout:9482: left oid 2 (ObjNum 9481 snap 0 seq_num 9481) 2020-06-12T02:39:05.108 INFO:tasks.rados.rados.0.smithi107.stdout:9482: done (1 left) 2020-06-12T02:39:05.108 INFO:tasks.rados.rados.0.smithi107.stdout:9485: done (0 left) 2020-06-12T02:39:05.109 INFO:tasks.rados.rados.0.smithi107.stdout:9486: delete oid 15 current snap is 0 2020-06-12T02:39:05.111 INFO:tasks.rados.rados.0.smithi107.stdout:9486: done (0 left) 2020-06-12T02:39:05.111 INFO:tasks.rados.rados.0.smithi107.stdout:9487: delete oid 11 current snap is 0 2020-06-12T02:39:05.114 INFO:tasks.rados.rados.0.smithi107.stdout:9487: done (0 left) 2020-06-12T02:39:05.114 INFO:tasks.rados.rados.0.smithi107.stdout:9488: delete oid 6 current snap is 0 2020-06-12T02:39:05.115 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:39:05.115 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:39:05.115 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:39:05.116 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:39:05.116 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.116 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:4975 3.as0 3:5732055a:::smithi10729407-8:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e30)", 2020-06-12T02:39:05.116 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:43.721794+0000", 2020-06-12T02:39:05.117 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 81.378728799000001, 2020-06-12T02:39:05.117 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.1825135, 2020-06-12T02:39:05.117 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:05.118 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:05.118 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:05.118 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:05.118 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:05.119 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 4975 2020-06-12T02:39:05.119 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.119 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:05.120 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.120 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:05.120 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721794+0000", 2020-06-12T02:39:05.121 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:05.121 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.121 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.121 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:05.122 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721794+0000", 2020-06-12T02:39:05.122 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4542e-05 2020-06-12T02:39:05.122 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.124 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.124 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:05.125 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721818+0000", 2020-06-12T02:39:05.125 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9169999999999997e-06 2020-06-12T02:39:05.125 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.125 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.126 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:05.126 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721823+0000", 2020-06-12T02:39:05.126 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1184e-05 2020-06-12T02:39:05.127 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.127 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.127 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:05.127 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721834+0000", 2020-06-12T02:39:05.128 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.254e-06 2020-06-12T02:39:05.128 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.128 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.129 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:05.129 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721840+0000", 2020-06-12T02:39:05.129 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1075999999999998e-05 2020-06-12T02:39:05.129 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.131 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.131 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:05.131 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721881+0000", 2020-06-12T02:39:05.131 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7416999999999995e-05 2020-06-12T02:39:05.132 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.132 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.133 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:05.133 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721948+0000", 2020-06-12T02:39:05.133 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.18235910999999999 2020-06-12T02:39:05.133 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.134 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.134 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:05.134 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.904307+0000", 2020-06-12T02:39:05.134 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.1825135 2020-06-12T02:39:05.135 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:05.135 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:05.135 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:05.135 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.136 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.136 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11914 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e56)", 2020-06-12T02:39:05.136 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:23.508790+0000", 2020-06-12T02:39:05.136 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 41.591731947, 2020-06-12T02:39:05.137 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026541928999999999, 2020-06-12T02:39:05.137 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:05.137 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:05.137 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:05.138 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:05.138 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:05.138 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11914 2020-06-12T02:39:05.138 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.139 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:05.139 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.139 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:05.139 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508790+0000", 2020-06-12T02:39:05.139 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:05.140 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.140 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.140 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:05.141 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508790+0000", 2020-06-12T02:39:05.141 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0872999999999999e-05 2020-06-12T02:39:05.141 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.141 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.142 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:05.142 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508841+0000", 2020-06-12T02:39:05.142 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9179999999999994e-06 2020-06-12T02:39:05.142 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.143 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.143 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:05.143 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508849+0000", 2020-06-12T02:39:05.143 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5551999999999999e-05 2020-06-12T02:39:05.144 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.144 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.144 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:05.144 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508865+0000", 2020-06-12T02:39:05.145 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0629999999999996e-06 2020-06-12T02:39:05.145 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.145 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.146 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:05.146 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508870+0000", 2020-06-12T02:39:05.146 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010597488 2020-06-12T02:39:05.146 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.147 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.147 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:05.147 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.519467+0000", 2020-06-12T02:39:05.148 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.6083999999999997e-05 2020-06-12T02:39:05.148 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.148 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.148 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:05.149 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.519563+0000", 2020-06-12T02:39:05.149 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0078077800000000003 2020-06-12T02:39:05.149 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.149 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.150 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:05.150 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.527371+0000", 2020-06-12T02:39:05.150 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6198999999999993e-05 2020-06-12T02:39:05.150 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.150 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.151 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:05.151 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.527447+0000", 2020-06-12T02:39:05.151 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0078849720000000005 2020-06-12T02:39:05.151 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.152 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.153 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:05.153 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.535332+0000", 2020-06-12T02:39:05.153 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026541928999999999 2020-06-12T02:39:05.154 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:05.154 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:05.155 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:05.155 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.156 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.156 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13705 3.as0 3:5cb723d2:::smithi10729407-15:head [append 0~745472,stat] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:39:05.156 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.390499+0000", 2020-06-12T02:39:05.157 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 36.710023253999999, 2020-06-12T02:39:05.157 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.067319320000000002, 2020-06-12T02:39:05.157 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:05.157 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:05.158 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:05.158 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:05.158 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:05.158 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13705 2020-06-12T02:39:05.159 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.159 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:05.159 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.159 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:05.160 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.390499+0000", 2020-06-12T02:39:05.160 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:05.160 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.160 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.160 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:05.161 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.390499+0000", 2020-06-12T02:39:05.161 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00066732599999999999 2020-06-12T02:39:05.161 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.161 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.162 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:05.163 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391167+0000", 2020-06-12T02:39:05.163 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3609999999999997e-06 2020-06-12T02:39:05.163 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.163 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.163 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:05.164 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391173+0000", 2020-06-12T02:39:05.164 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0399e-05 2020-06-12T02:39:05.164 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.164 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.164 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:05.165 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391183+0000", 2020-06-12T02:39:05.165 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6480000000000002e-06 2020-06-12T02:39:05.165 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.165 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.165 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:05.166 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391188+0000", 2020-06-12T02:39:05.166 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4150999999999999e-05 2020-06-12T02:39:05.166 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.166 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.166 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:05.166 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391222+0000", 2020-06-12T02:39:05.167 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9450999999999996e-05 2020-06-12T02:39:05.167 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.167 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.167 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:05.171 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391292+0000", 2020-06-12T02:39:05.171 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054214300000000003 2020-06-12T02:39:05.172 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.172 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.172 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:05.172 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391834+0000", 2020-06-12T02:39:05.173 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0037312069999999998 2020-06-12T02:39:05.173 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.173 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.173 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:05.173 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.395565+0000", 2020-06-12T02:39:05.174 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001050194 2020-06-12T02:39:05.174 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.174 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.174 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:05.174 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396615+0000", 2020-06-12T02:39:05.175 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.061203439999999998 2020-06-12T02:39:05.175 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.175 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.175 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:05.175 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.457819+0000", 2020-06-12T02:39:05.176 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.067319320000000002 2020-06-12T02:39:05.176 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:05.176 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:05.176 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:05.176 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.176 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.177 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13706 3.as0 3:5cb723d2:::smithi10729407-15:head [append 0~434176] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:39:05.177 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.393133+0000", 2020-06-12T02:39:05.177 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 36.707388985999998, 2020-06-12T02:39:05.177 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.078158561000000001, 2020-06-12T02:39:05.177 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:05.178 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:05.178 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:05.178 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:05.178 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:05.178 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13706 2020-06-12T02:39:05.179 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.179 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:05.179 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.179 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:05.179 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393133+0000", 2020-06-12T02:39:05.179 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:05.180 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.180 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.180 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:05.180 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393133+0000", 2020-06-12T02:39:05.180 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00050415499999999995 2020-06-12T02:39:05.181 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.181 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.181 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:05.181 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393638+0000", 2020-06-12T02:39:05.181 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5010000000000004e-06 2020-06-12T02:39:05.182 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.182 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.182 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:05.182 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393646+0000", 2020-06-12T02:39:05.182 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4183e-05 2020-06-12T02:39:05.182 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.183 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.183 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:05.183 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393660+0000", 2020-06-12T02:39:05.183 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6069999999999996e-06 2020-06-12T02:39:05.183 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.184 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.184 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:05.184 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393667+0000", 2020-06-12T02:39:05.184 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001363569 2020-06-12T02:39:05.184 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.184 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.185 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:05.185 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.395030+0000", 2020-06-12T02:39:05.185 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012127 2020-06-12T02:39:05.185 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.185 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.186 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:05.186 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.395152+0000", 2020-06-12T02:39:05.187 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.063640503000000001 2020-06-12T02:39:05.187 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.187 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.187 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:05.187 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458792+0000", 2020-06-12T02:39:05.188 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010636 2020-06-12T02:39:05.188 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.188 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.188 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:05.188 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458899+0000", 2020-06-12T02:39:05.189 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00076074200000000002 2020-06-12T02:39:05.189 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.189 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.189 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:05.189 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459659+0000", 2020-06-12T02:39:05.190 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00772984 2020-06-12T02:39:05.190 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.190 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.190 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:05.190 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.467389+0000", 2020-06-12T02:39:05.191 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003815751 2020-06-12T02:39:05.191 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.191 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.191 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:05.191 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471205+0000", 2020-06-12T02:39:05.192 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.708e-05 2020-06-12T02:39:05.192 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.192 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.192 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:05.192 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471292+0000", 2020-06-12T02:39:05.193 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.078158561000000001 2020-06-12T02:39:05.193 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:05.193 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:05.193 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:05.193 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.194 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.194 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13707 3.as0 3:5cb723d2:::smithi10729407-15:head [append 0~565248] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:39:05.194 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.396173+0000", 2020-06-12T02:39:05.194 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 36.704349043000001, 2020-06-12T02:39:05.194 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.075298341000000005, 2020-06-12T02:39:05.195 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:05.195 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:05.195 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:05.195 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:05.195 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:05.195 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13707 2020-06-12T02:39:05.196 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.196 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:05.196 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.196 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:05.196 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396173+0000", 2020-06-12T02:39:05.197 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:05.197 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.197 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.197 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:05.197 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396173+0000", 2020-06-12T02:39:05.198 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.061779787000000003 2020-06-12T02:39:05.198 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.198 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.198 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:05.198 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.457953+0000", 2020-06-12T02:39:05.198 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5830000000000007e-06 2020-06-12T02:39:05.199 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.200 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.200 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:05.200 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.457962+0000", 2020-06-12T02:39:05.201 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3029e-05 2020-06-12T02:39:05.201 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.201 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.201 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:05.202 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.457975+0000", 2020-06-12T02:39:05.202 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.233e-06 2020-06-12T02:39:05.202 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.203 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.203 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:05.203 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.457979+0000", 2020-06-12T02:39:05.203 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0044352510000000003 2020-06-12T02:39:05.204 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.205 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.205 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:05.205 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.462414+0000", 2020-06-12T02:39:05.205 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012825799999999999 2020-06-12T02:39:05.206 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.206 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.206 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:05.206 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.462543+0000", 2020-06-12T02:39:05.207 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00077227899999999996 2020-06-12T02:39:05.207 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.208 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.208 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:05.208 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.463315+0000", 2020-06-12T02:39:05.208 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0079912390000000007 2020-06-12T02:39:05.209 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.209 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.210 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:05.210 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471306+0000", 2020-06-12T02:39:05.211 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000118674 2020-06-12T02:39:05.211 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.211 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.211 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:05.212 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471425+0000", 2020-06-12T02:39:05.213 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7008000000000001e-05 2020-06-12T02:39:05.213 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.214 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.214 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:05.214 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471472+0000", 2020-06-12T02:39:05.214 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.075298341000000005 2020-06-12T02:39:05.214 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:05.215 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:05.215 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:05.215 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.215 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.215 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWriteReply(3.as0 58/53 ECSubWriteReply(tid=1890, last_complete=58'1184, committed=1, applied=1))", 2020-06-12T02:39:05.216 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.396464+0000", 2020-06-12T02:39:05.216 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 36.704058164000003, 2020-06-12T02:39:05.216 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.062264803000000001, 2020-06-12T02:39:05.216 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:05.216 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:39:05.217 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:05.217 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.217 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:05.217 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396464+0000", 2020-06-12T02:39:05.217 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:05.218 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.218 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.218 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:05.218 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396464+0000", 2020-06-12T02:39:05.218 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9425999999999999e-05 2020-06-12T02:39:05.219 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.219 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.219 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:05.219 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396494+0000", 2020-06-12T02:39:05.219 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1610000000000002e-06 2020-06-12T02:39:05.220 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.220 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.220 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:05.220 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396499+0000", 2020-06-12T02:39:05.220 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2789999999999998e-06 2020-06-12T02:39:05.221 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.221 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.221 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:05.222 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396507+0000", 2020-06-12T02:39:05.222 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1389999999999997e-06 2020-06-12T02:39:05.222 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.222 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.223 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:05.223 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396511+0000", 2020-06-12T02:39:05.223 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2199000000000001e-05 2020-06-12T02:39:05.223 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.223 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.224 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:05.224 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396543+0000", 2020-06-12T02:39:05.224 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.062185599000000001 2020-06-12T02:39:05.224 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.224 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.225 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:05.225 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458729+0000", 2020-06-12T02:39:05.225 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.062264803000000001 2020-06-12T02:39:05.225 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:05.225 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:05.226 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:05.226 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.226 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.226 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13710 3.as0 3:5cb723d2:::smithi10729407-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e58)", 2020-06-12T02:39:05.226 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.458457+0000", 2020-06-12T02:39:05.227 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 36.642064994000002, 2020-06-12T02:39:05.227 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028102095000000001, 2020-06-12T02:39:05.227 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:05.227 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:05.227 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:05.227 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:05.228 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:05.228 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13710 2020-06-12T02:39:05.228 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.228 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:05.228 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.229 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:05.229 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458457+0000", 2020-06-12T02:39:05.229 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:05.229 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.229 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.230 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:05.230 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458457+0000", 2020-06-12T02:39:05.230 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7650000000000001e-05 2020-06-12T02:39:05.230 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.230 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.230 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:05.231 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458485+0000", 2020-06-12T02:39:05.231 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3210000000000001e-06 2020-06-12T02:39:05.231 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.231 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.232 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:05.232 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458488+0000", 2020-06-12T02:39:05.232 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5920000000000003e-06 2020-06-12T02:39:05.233 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.233 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.233 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:05.233 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458496+0000", 2020-06-12T02:39:05.233 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8789999999999999e-06 2020-06-12T02:39:05.234 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.234 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.234 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:05.234 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458499+0000", 2020-06-12T02:39:05.234 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013081152 2020-06-12T02:39:05.234 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.235 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.235 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:05.235 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471580+0000", 2020-06-12T02:39:05.235 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4975999999999994e-05 2020-06-12T02:39:05.235 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.236 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.236 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:05.236 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471665+0000", 2020-06-12T02:39:05.236 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0054254029999999996 2020-06-12T02:39:05.236 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.236 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.237 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:05.237 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.477090+0000", 2020-06-12T02:39:05.237 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2335000000000005e-05 2020-06-12T02:39:05.237 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.237 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.238 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:05.238 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.477163+0000", 2020-06-12T02:39:05.238 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0093967870000000002 2020-06-12T02:39:05.238 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.238 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.239 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:05.239 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486560+0000", 2020-06-12T02:39:05.239 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028102095000000001 2020-06-12T02:39:05.239 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:05.239 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:05.239 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:05.240 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.240 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.240 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13711 3.as0 3:5732055a:::smithi10729407-8:head [append 0~761856] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:39:05.240 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.458525+0000", 2020-06-12T02:39:05.240 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 36.641997222000001, 2020-06-12T02:39:05.241 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.027773760000000002, 2020-06-12T02:39:05.241 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:05.241 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:05.241 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:05.241 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:05.241 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:05.242 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13711 2020-06-12T02:39:05.242 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.243 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:05.243 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.243 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:05.243 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458525+0000", 2020-06-12T02:39:05.243 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:05.244 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.244 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.244 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:05.244 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458525+0000", 2020-06-12T02:39:05.244 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00089965200000000003 2020-06-12T02:39:05.244 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.245 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.245 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:05.245 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459425+0000", 2020-06-12T02:39:05.245 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4749999999999996e-06 2020-06-12T02:39:05.245 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.246 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.246 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:05.246 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459432+0000", 2020-06-12T02:39:05.246 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3043e-05 2020-06-12T02:39:05.246 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.247 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.247 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:05.247 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459445+0000", 2020-06-12T02:39:05.247 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0739999999999998e-06 2020-06-12T02:39:05.247 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.247 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.248 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:05.248 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459451+0000", 2020-06-12T02:39:05.248 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012238733999999999 2020-06-12T02:39:05.248 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.248 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.248 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:05.249 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471690+0000", 2020-06-12T02:39:05.249 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5640000000000003e-05 2020-06-12T02:39:05.249 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.249 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.249 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:05.250 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471746+0000", 2020-06-12T02:39:05.250 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00067859699999999997 2020-06-12T02:39:05.250 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.250 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.250 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:05.251 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.472424+0000", 2020-06-12T02:39:05.251 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050083230000000003 2020-06-12T02:39:05.251 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.251 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.251 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:05.251 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.477433+0000", 2020-06-12T02:39:05.252 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0087878039999999998 2020-06-12T02:39:05.252 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.252 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.253 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:05.253 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486221+0000", 2020-06-12T02:39:05.253 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8417999999999994e-05 2020-06-12T02:39:05.253 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.254 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.254 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:05.254 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486299+0000", 2020-06-12T02:39:05.254 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.027773760000000002 2020-06-12T02:39:05.254 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:05.254 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:05.255 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:05.255 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.255 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.255 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13712 3.as0 3:5732055a:::smithi10729407-8:head [append 0~712704] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:39:05.255 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.459508+0000", 2020-06-12T02:39:05.256 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 36.641013942000001, 2020-06-12T02:39:05.256 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.033995818999999997, 2020-06-12T02:39:05.256 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:05.256 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:05.256 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:05.257 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:05.257 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:05.257 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13712 2020-06-12T02:39:05.257 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.257 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:05.257 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.258 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:05.258 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459508+0000", 2020-06-12T02:39:05.258 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:05.258 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.258 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.259 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:05.259 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459508+0000", 2020-06-12T02:39:05.259 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00075436999999999998 2020-06-12T02:39:05.259 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.259 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.259 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:05.260 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460263+0000", 2020-06-12T02:39:05.260 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4370000000000001e-06 2020-06-12T02:39:05.260 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.260 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.260 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:05.261 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460270+0000", 2020-06-12T02:39:05.261 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2500000000000001e-05 2020-06-12T02:39:05.261 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.261 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.261 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:05.262 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460283+0000", 2020-06-12T02:39:05.262 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2580000000000003e-06 2020-06-12T02:39:05.262 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.262 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.262 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:05.263 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460288+0000", 2020-06-12T02:39:05.263 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017177595 2020-06-12T02:39:05.263 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.264 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.264 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:05.264 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.477466+0000", 2020-06-12T02:39:05.264 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00019751600000000001 2020-06-12T02:39:05.264 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.265 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.265 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:05.265 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.477663+0000", 2020-06-12T02:39:05.265 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00088697400000000001 2020-06-12T02:39:05.265 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.266 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.266 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:05.266 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.478550+0000", 2020-06-12T02:39:05.266 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.007578787 2020-06-12T02:39:05.266 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.267 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.267 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:05.267 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486129+0000", 2020-06-12T02:39:05.267 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0072972310000000004 2020-06-12T02:39:05.267 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.267 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.268 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:05.268 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493426+0000", 2020-06-12T02:39:05.268 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8151000000000004e-05 2020-06-12T02:39:05.268 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.268 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.269 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:05.269 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493504+0000", 2020-06-12T02:39:05.269 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.033995818999999997 2020-06-12T02:39:05.269 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:05.269 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:05.269 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:05.270 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.270 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.270 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13713 3.as0 3:5732055a:::smithi10729407-8:head [append 0~491520] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:39:05.270 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.460333+0000", 2020-06-12T02:39:05.270 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 36.640189485999997, 2020-06-12T02:39:05.271 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.033390145000000003, 2020-06-12T02:39:05.271 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:05.271 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:05.271 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:05.271 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:05.272 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:05.272 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13713 2020-06-12T02:39:05.272 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.272 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:05.272 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.272 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:05.273 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460333+0000", 2020-06-12T02:39:05.273 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:05.273 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.274 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.274 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:05.274 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460333+0000", 2020-06-12T02:39:05.274 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00051168999999999997 2020-06-12T02:39:05.274 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.275 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.275 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:05.275 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460845+0000", 2020-06-12T02:39:05.275 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2930000000000003e-06 2020-06-12T02:39:05.275 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.275 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.276 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:05.276 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460852+0000", 2020-06-12T02:39:05.276 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3054000000000001e-05 2020-06-12T02:39:05.276 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.276 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.277 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:05.277 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460865+0000", 2020-06-12T02:39:05.277 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6130000000000003e-06 2020-06-12T02:39:05.277 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.277 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.277 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:05.278 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460871+0000", 2020-06-12T02:39:05.278 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025753870000000002 2020-06-12T02:39:05.278 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.278 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.278 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:05.279 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486624+0000", 2020-06-12T02:39:05.279 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2739999999999998e-05 2020-06-12T02:39:05.279 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.279 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.279 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:05.280 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486697+0000", 2020-06-12T02:39:05.280 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00063578799999999998 2020-06-12T02:39:05.280 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.280 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.280 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:05.280 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.487333+0000", 2020-06-12T02:39:05.281 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0043011610000000004 2020-06-12T02:39:05.281 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.281 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.281 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:05.281 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.491634+0000", 2020-06-12T02:39:05.282 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0020130949999999999 2020-06-12T02:39:05.282 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.282 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.282 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:05.283 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493647+0000", 2020-06-12T02:39:05.283 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5840999999999994e-05 2020-06-12T02:39:05.283 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.284 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.285 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:05.285 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493723+0000", 2020-06-12T02:39:05.285 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.033390145000000003 2020-06-12T02:39:05.285 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:05.285 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:05.286 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:05.286 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.286 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.286 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13714 3.as0 3:5732055a:::smithi10729407-8:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:39:05.286 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.460921+0000", 2020-06-12T02:39:05.287 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 36.639600964000003, 2020-06-12T02:39:05.287 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.032981457999999998, 2020-06-12T02:39:05.287 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:05.287 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:05.287 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:05.288 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:05.288 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:05.288 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13714 2020-06-12T02:39:05.288 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.288 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:05.289 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.289 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:05.289 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460921+0000", 2020-06-12T02:39:05.289 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:05.289 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.290 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.290 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:05.290 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460921+0000", 2020-06-12T02:39:05.290 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6996999999999995e-05 2020-06-12T02:39:05.290 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.290 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.291 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:05.291 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460998+0000", 2020-06-12T02:39:05.291 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4529999999999999e-06 2020-06-12T02:39:05.291 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.291 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.292 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:05.292 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461005+0000", 2020-06-12T02:39:05.292 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1722000000000001e-05 2020-06-12T02:39:05.292 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.292 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.293 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:05.293 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461017+0000", 2020-06-12T02:39:05.293 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4839999999999998e-06 2020-06-12T02:39:05.293 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.293 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.294 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:05.294 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461021+0000", 2020-06-12T02:39:05.294 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.029157975999999999 2020-06-12T02:39:05.294 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.294 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.295 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:05.295 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.490179+0000", 2020-06-12T02:39:05.304 INFO:tasks.rados.rados.0.smithi107.stdout:9488: done (0 left) 2020-06-12T02:39:05.304 INFO:tasks.rados.rados.0.smithi107.stdout:9489: delete oid 8 current snap is 0 2020-06-12T02:39:05.305 INFO:tasks.rados.rados.0.smithi107.stdout:9489: done (0 left) 2020-06-12T02:39:05.305 INFO:tasks.rados.rados.0.smithi107.stdout:9490: delete oid 17 current snap is 0 2020-06-12T02:39:05.306 INFO:tasks.rados.rados.0.smithi107.stdout:9490: done (0 left) 2020-06-12T02:39:05.306 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:39:05.306 INFO:tasks.rados.rados.0.smithi107.stdout:9491: seq_num 9490 ranges {0=548864} 2020-06-12T02:39:05.306 INFO:tasks.rados.rados.0.smithi107.stdout:9491: writing smithi10729407-17 from 0 to 548864 tid 1 2020-06-12T02:39:05.307 INFO:tasks.rados.rados.0.smithi107.stdout:9492: delete oid 14 current snap is 0 2020-06-12T02:39:05.307 INFO:tasks.rados.rados.0.smithi107.stdout:9492: done (1 left) 2020-06-12T02:39:05.307 INFO:tasks.rados.rados.0.smithi107.stdout:9493: delete oid 5 current snap is 0 2020-06-12T02:39:05.307 INFO:tasks.rados.rados.0.smithi107.stdout:9491: finishing write tid 1 to smithi10729407-17 2020-06-12T02:39:05.307 INFO:tasks.rados.rados.0.smithi107.stdout:9493: done (1 left) 2020-06-12T02:39:05.308 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:39:05.308 INFO:tasks.rados.rados.0.smithi107.stdout:9494: seq_num 9493 ranges {0=598016} 2020-06-12T02:39:05.308 INFO:tasks.rados.rados.0.smithi107.stdout:9494: writing smithi10729407-4 from 0 to 598016 tid 1 2020-06-12T02:39:05.308 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:39:05.308 INFO:tasks.rados.rados.0.smithi107.stdout:9495: seq_num 9494 ranges {3088384=614400} 2020-06-12T02:39:05.308 INFO:tasks.rados.rados.0.smithi107.stdout:9495: writing smithi10729407-2 from 3088384 to 3702784 tid 1 2020-06-12T02:39:05.309 INFO:tasks.rados.rados.0.smithi107.stdout:9496: delete oid 10 current snap is 0 2020-06-12T02:39:05.309 INFO:tasks.rados.rados.0.smithi107.stdout:9491: finishing write tid 3 to smithi10729407-17 2020-06-12T02:39:05.309 INFO:tasks.rados.rados.0.smithi107.stdout:9491: finishing write tid 4 to smithi10729407-17 2020-06-12T02:39:05.309 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1786 (ObjNum 9490 snap 0 seq_num 9490) dirty exists 2020-06-12T02:39:05.309 INFO:tasks.rados.rados.0.smithi107.stdout:9491: left oid 17 (ObjNum 9490 snap 0 seq_num 9490) 2020-06-12T02:39:05.310 INFO:tasks.rados.rados.0.smithi107.stdout:9494: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:05.310 INFO:tasks.rados.rados.0.smithi107.stdout:9494: finishing write tid 3 to smithi10729407-4 2020-06-12T02:39:05.310 INFO:tasks.rados.rados.0.smithi107.stdout:9494: finishing write tid 4 to smithi10729407-4 2020-06-12T02:39:05.310 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1788 (ObjNum 9493 snap 0 seq_num 9493) dirty exists 2020-06-12T02:39:05.310 INFO:tasks.rados.rados.0.smithi107.stdout:9494: left oid 4 (ObjNum 9493 snap 0 seq_num 9493) 2020-06-12T02:39:05.311 INFO:tasks.rados.rados.0.smithi107.stdout:9491: done (3 left) 2020-06-12T02:39:05.311 INFO:tasks.rados.rados.0.smithi107.stdout:9494: done (2 left) 2020-06-12T02:39:05.311 INFO:tasks.rados.rados.0.smithi107.stdout:9496: done (1 left) 2020-06-12T02:39:05.311 INFO:tasks.rados.rados.0.smithi107.stdout:9497: delete oid 14 current snap is 0 2020-06-12T02:39:05.311 INFO:tasks.rados.rados.0.smithi107.stdout:9497: done (1 left) 2020-06-12T02:39:05.311 INFO:tasks.rados.rados.0.smithi107.stdout:9498: delete oid 1 current snap is 0 2020-06-12T02:39:05.312 INFO:tasks.rados.rados.0.smithi107.stdout:9495: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:05.312 INFO:tasks.rados.rados.0.smithi107.stdout:9498: done (1 left) 2020-06-12T02:39:05.312 INFO:tasks.rados.rados.0.smithi107.stdout:9499: delete oid 9 current snap is 0 2020-06-12T02:39:05.312 INFO:tasks.rados.rados.0.smithi107.stdout:9495: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:05.312 INFO:tasks.rados.rados.0.smithi107.stdout:9499: done (1 left) 2020-06-12T02:39:05.313 INFO:tasks.rados.rados.0.smithi107.stdout:9500: delete oid 12 current snap is 0 2020-06-12T02:39:05.313 INFO:tasks.rados.rados.0.smithi107.stdout:9495: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:05.313 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 883 (ObjNum 9494 snap 0 seq_num 9494) dirty exists 2020-06-12T02:39:05.313 INFO:tasks.rados.rados.0.smithi107.stdout:9495: left oid 2 (ObjNum 9494 snap 0 seq_num 9494) 2020-06-12T02:39:05.313 INFO:tasks.rados.rados.0.smithi107.stdout:9495: done (1 left) 2020-06-12T02:39:05.314 INFO:tasks.rados.rados.0.smithi107.stdout:9500: done (0 left) 2020-06-12T02:39:05.314 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:39:05.314 INFO:tasks.rados.rados.0.smithi107.stdout:9501: seq_num 9500 ranges {0=712704} 2020-06-12T02:39:05.314 INFO:tasks.rados.rados.0.smithi107.stdout:9501: writing smithi10729407-16 from 0 to 712704 tid 1 2020-06-12T02:39:05.314 INFO:tasks.rados.rados.0.smithi107.stdout:9502: delete oid 13 current snap is 0 2020-06-12T02:39:05.314 INFO:tasks.rados.rados.0.smithi107.stdout:9502: done (1 left) 2020-06-12T02:39:05.315 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:39:05.315 INFO:tasks.rados.rados.0.smithi107.stdout:9503: seq_num 9502 ranges {0=745472} 2020-06-12T02:39:05.315 INFO:tasks.rados.rados.0.smithi107.stdout:9503: writing smithi10729407-14 from 0 to 745472 tid 1 2020-06-12T02:39:05.315 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:39:05.315 INFO:tasks.rados.rados.0.smithi107.stdout:9504: seq_num 9503 ranges {0=761856} 2020-06-12T02:39:05.316 INFO:tasks.rados.rados.0.smithi107.stdout:9504: writing smithi10729407-3 from 0 to 761856 tid 1 2020-06-12T02:39:05.316 INFO:tasks.rados.rados.0.smithi107.stdout:9505: delete oid 6 current snap is 0 2020-06-12T02:39:05.316 INFO:tasks.rados.rados.0.smithi107.stdout:9501: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:05.316 INFO:tasks.rados.rados.0.smithi107.stdout:9501: finishing write tid 3 to smithi10729407-16 2020-06-12T02:39:05.316 INFO:tasks.rados.rados.0.smithi107.stdout:9501: finishing write tid 4 to smithi10729407-16 2020-06-12T02:39:05.317 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1686 (ObjNum 9500 snap 0 seq_num 9500) dirty exists 2020-06-12T02:39:05.317 INFO:tasks.rados.rados.0.smithi107.stdout:9501: left oid 16 (ObjNum 9500 snap 0 seq_num 9500) 2020-06-12T02:39:05.317 INFO:tasks.rados.rados.0.smithi107.stdout:9503: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:05.318 INFO:tasks.rados.rados.0.smithi107.stdout:9503: finishing write tid 3 to smithi10729407-14 2020-06-12T02:39:05.318 INFO:tasks.rados.rados.0.smithi107.stdout:9503: finishing write tid 4 to smithi10729407-14 2020-06-12T02:39:05.318 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1604 (ObjNum 9502 snap 0 seq_num 9502) dirty exists 2020-06-12T02:39:05.318 INFO:tasks.rados.rados.0.smithi107.stdout:9503: left oid 14 (ObjNum 9502 snap 0 seq_num 9502) 2020-06-12T02:39:05.318 INFO:tasks.rados.rados.0.smithi107.stdout:9501: done (3 left) 2020-06-12T02:39:05.319 INFO:tasks.rados.rados.0.smithi107.stdout:9503: done (2 left) 2020-06-12T02:39:05.319 INFO:tasks.rados.rados.0.smithi107.stdout:9505: done (1 left) 2020-06-12T02:39:05.319 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:39:05.319 INFO:tasks.rados.rados.0.smithi107.stdout:9506: seq_num 9505 ranges {712704=794624} 2020-06-12T02:39:05.319 INFO:tasks.rados.rados.0.smithi107.stdout:9506: writing smithi10729407-16 from 712704 to 1507328 tid 1 2020-06-12T02:39:05.320 INFO:tasks.rados.rados.0.smithi107.stdout:9507: delete oid 4 current snap is 0 2020-06-12T02:39:05.320 INFO:tasks.rados.rados.0.smithi107.stdout:9504: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:05.320 INFO:tasks.rados.rados.0.smithi107.stdout:9507: done (2 left) 2020-06-12T02:39:05.320 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:39:05.320 INFO:tasks.rados.rados.0.smithi107.stdout:9508: seq_num 9507 ranges {0=425984,425984=401408} 2020-06-12T02:39:05.320 INFO:tasks.rados.rados.0.smithi107.stdout:9508: writing smithi10729407-10 from 0 to 425984 tid 1 2020-06-12T02:39:05.321 INFO:tasks.rados.rados.0.smithi107.stdout:9508: writing smithi10729407-10 from 425984 to 827392 tid 2 2020-06-12T02:39:05.321 INFO:tasks.rados.rados.0.smithi107.stdout:9509: delete oid 6 current snap is 0 2020-06-12T02:39:05.321 INFO:tasks.rados.rados.0.smithi107.stdout:9504: finishing write tid 3 to smithi10729407-3 2020-06-12T02:39:05.321 INFO:tasks.rados.rados.0.smithi107.stdout:9504: finishing write tid 4 to smithi10729407-3 2020-06-12T02:39:05.321 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1688 (ObjNum 9503 snap 0 seq_num 9503) dirty exists 2020-06-12T02:39:05.322 INFO:tasks.rados.rados.0.smithi107.stdout:9504: left oid 3 (ObjNum 9503 snap 0 seq_num 9503) 2020-06-12T02:39:05.322 INFO:tasks.rados.rados.0.smithi107.stdout:9506: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:05.322 INFO:tasks.rados.rados.0.smithi107.stdout:9506: finishing write tid 3 to smithi10729407-16 2020-06-12T02:39:05.322 INFO:tasks.rados.rados.0.smithi107.stdout:9508: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:05.322 INFO:tasks.rados.rados.0.smithi107.stdout:9504: done (3 left) 2020-06-12T02:39:05.322 INFO:tasks.rados.rados.0.smithi107.stdout:9509: done (2 left) 2020-06-12T02:39:05.323 INFO:tasks.rados.rados.0.smithi107.stdout:9510: delete oid 20 current snap is 0 2020-06-12T02:39:05.323 INFO:tasks.rados.rados.0.smithi107.stdout:9506: finishing write tid 4 to smithi10729407-16 2020-06-12T02:39:05.323 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1690 (ObjNum 9505 snap 0 seq_num 9505) dirty exists 2020-06-12T02:39:05.323 INFO:tasks.rados.rados.0.smithi107.stdout:9506: left oid 16 (ObjNum 9505 snap 0 seq_num 9505) 2020-06-12T02:39:05.323 INFO:tasks.rados.rados.0.smithi107.stdout:9506: done (2 left) 2020-06-12T02:39:05.324 INFO:tasks.rados.rados.0.smithi107.stdout:9510: done (1 left) 2020-06-12T02:39:05.324 INFO:tasks.rados.rados.0.smithi107.stdout:9511: delete oid 19 current snap is 0 2020-06-12T02:39:05.324 INFO:tasks.rados.rados.0.smithi107.stdout:9508: finishing write tid 2 to smithi10729407-10 2020-06-12T02:39:05.324 INFO:tasks.rados.rados.0.smithi107.stdout:9511: done (1 left) 2020-06-12T02:39:05.324 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:39:05.324 INFO:tasks.rados.rados.0.smithi107.stdout:9512: seq_num 9511 ranges {745472=499712,1245184=401408} 2020-06-12T02:39:05.325 INFO:tasks.rados.rados.0.smithi107.stdout:9512: writing smithi10729407-14 from 745472 to 1245184 tid 1 2020-06-12T02:39:05.325 INFO:tasks.rados.rados.0.smithi107.stdout:9512: writing smithi10729407-14 from 1245184 to 1646592 tid 2 2020-06-12T02:39:05.325 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:39:05.325 INFO:tasks.rados.rados.0.smithi107.stdout:9513: seq_num 9512 ranges {0=516096,516096=401408} 2020-06-12T02:39:05.325 INFO:tasks.rados.rados.0.smithi107.stdout:9513: writing smithi10729407-8 from 0 to 516096 tid 1 2020-06-12T02:39:05.326 INFO:tasks.rados.rados.0.smithi107.stdout:9513: writing smithi10729407-8 from 516096 to 917504 tid 2 2020-06-12T02:39:05.326 INFO:tasks.rados.rados.0.smithi107.stdout:9514: delete oid 17 current snap is 0 2020-06-12T02:39:05.326 INFO:tasks.rados.rados.0.smithi107.stdout:9508: finishing write tid 4 to smithi10729407-10 2020-06-12T02:39:05.326 INFO:tasks.rados.rados.0.smithi107.stdout:9508: finishing write tid 5 to smithi10729407-10 2020-06-12T02:39:05.326 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 673 (ObjNum 9507 snap 0 seq_num 9507) dirty exists 2020-06-12T02:39:05.326 INFO:tasks.rados.rados.0.smithi107.stdout:9508: left oid 10 (ObjNum 9507 snap 0 seq_num 9507) 2020-06-12T02:39:05.327 INFO:tasks.rados.rados.0.smithi107.stdout:9512: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:05.327 INFO:tasks.rados.rados.0.smithi107.stdout:9508: done (3 left) 2020-06-12T02:39:05.327 INFO:tasks.rados.rados.0.smithi107.stdout:9514: done (2 left) 2020-06-12T02:39:05.327 INFO:tasks.rados.rados.0.smithi107.stdout:9515: delete oid 6 current snap is 0 2020-06-12T02:39:05.327 INFO:tasks.rados.rados.0.smithi107.stdout:9512: finishing write tid 2 to smithi10729407-14 2020-06-12T02:39:05.328 INFO:tasks.rados.rados.0.smithi107.stdout:9512: finishing write tid 4 to smithi10729407-14 2020-06-12T02:39:05.328 INFO:tasks.rados.rados.0.smithi107.stdout:9513: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:05.328 INFO:tasks.rados.rados.0.smithi107.stdout:9512: finishing write tid 5 to smithi10729407-14 2020-06-12T02:39:05.328 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1607 (ObjNum 9511 snap 0 seq_num 9511) dirty exists 2020-06-12T02:39:05.328 INFO:tasks.rados.rados.0.smithi107.stdout:9512: left oid 14 (ObjNum 9511 snap 0 seq_num 9511) 2020-06-12T02:39:05.328 INFO:tasks.rados.rados.0.smithi107.stdout:9512: done (2 left) 2020-06-12T02:39:05.329 INFO:tasks.rados.rados.0.smithi107.stdout:9515: done (1 left) 2020-06-12T02:39:05.329 INFO:tasks.rados.rados.0.smithi107.stdout:9516: delete oid 1 current snap is 0 2020-06-12T02:39:05.329 INFO:tasks.rados.rados.0.smithi107.stdout:9516: done (1 left) 2020-06-12T02:39:05.329 INFO:tasks.rados.rados.0.smithi107.stdout:9517: delete oid 7 current snap is 0 2020-06-12T02:39:05.329 INFO:tasks.rados.rados.0.smithi107.stdout:9513: finishing write tid 2 to smithi10729407-8 2020-06-12T02:39:05.330 INFO:tasks.rados.rados.0.smithi107.stdout:9513: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:05.330 INFO:tasks.rados.rados.0.smithi107.stdout:9513: finishing write tid 5 to smithi10729407-8 2020-06-12T02:39:05.330 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1773 (ObjNum 9512 snap 0 seq_num 9512) dirty exists 2020-06-12T02:39:05.330 INFO:tasks.rados.rados.0.smithi107.stdout:9513: left oid 8 (ObjNum 9512 snap 0 seq_num 9512) 2020-06-12T02:39:05.330 INFO:tasks.rados.rados.0.smithi107.stdout:9513: done (1 left) 2020-06-12T02:39:05.331 INFO:tasks.rados.rados.0.smithi107.stdout:9517: done (0 left) 2020-06-12T02:39:05.332 INFO:tasks.rados.rados.0.smithi107.stdout:9518: delete oid 1 current snap is 0 2020-06-12T02:39:05.332 INFO:tasks.rados.rados.0.smithi107.stdout:9518: done (0 left) 2020-06-12T02:39:05.333 INFO:tasks.rados.rados.0.smithi107.stdout:9519: delete oid 9 current snap is 0 2020-06-12T02:39:05.333 INFO:tasks.rados.rados.0.smithi107.stdout:9519: done (0 left) 2020-06-12T02:39:05.333 INFO:tasks.rados.rados.0.smithi107.stdout:9520: delete oid 17 current snap is 0 2020-06-12T02:39:05.333 INFO:tasks.rados.rados.0.smithi107.stdout:9520: done (0 left) 2020-06-12T02:39:05.333 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:39:05.334 INFO:tasks.rados.rados.0.smithi107.stdout:9521: seq_num 9520 ranges {1507328=647168,2154496=401408} 2020-06-12T02:39:05.334 INFO:tasks.rados.rados.0.smithi107.stdout:9521: writing smithi10729407-16 from 1507328 to 2154496 tid 1 2020-06-12T02:39:05.334 INFO:tasks.rados.rados.0.smithi107.stdout:9521: writing smithi10729407-16 from 2154496 to 2555904 tid 2 2020-06-12T02:39:05.334 INFO:tasks.rados.rados.0.smithi107.stdout:9522: delete oid 8 current snap is 0 2020-06-12T02:39:05.334 INFO:tasks.rados.rados.0.smithi107.stdout:9522: done (1 left) 2020-06-12T02:39:05.335 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:39:05.335 INFO:tasks.rados.rados.0.smithi107.stdout:9523: seq_num 9522 ranges {1646592=679936,2326528=401408} 2020-06-12T02:39:05.335 INFO:tasks.rados.rados.0.smithi107.stdout:9523: writing smithi10729407-14 from 1646592 to 2326528 tid 1 2020-06-12T02:39:05.335 INFO:tasks.rados.rados.0.smithi107.stdout:9523: writing smithi10729407-14 from 2326528 to 2727936 tid 2 2020-06-12T02:39:05.335 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:39:05.336 INFO:tasks.rados.rados.0.smithi107.stdout:9524: seq_num 9523 ranges {827392=696320,1523712=401408} 2020-06-12T02:39:05.336 INFO:tasks.rados.rados.0.smithi107.stdout:9524: writing smithi10729407-10 from 827392 to 1523712 tid 1 2020-06-12T02:39:05.336 INFO:tasks.rados.rados.0.smithi107.stdout:9524: writing smithi10729407-10 from 1523712 to 1925120 tid 2 2020-06-12T02:39:05.337 INFO:tasks.rados.rados.0.smithi107.stdout:9525: delete oid 2 current snap is 0 2020-06-12T02:39:05.337 INFO:tasks.rados.rados.0.smithi107.stdout:9521: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:05.337 INFO:tasks.rados.rados.0.smithi107.stdout:9521: finishing write tid 2 to smithi10729407-16 2020-06-12T02:39:05.337 INFO:tasks.rados.rados.0.smithi107.stdout:9521: finishing write tid 4 to smithi10729407-16 2020-06-12T02:39:05.337 INFO:tasks.rados.rados.0.smithi107.stdout:9521: finishing write tid 5 to smithi10729407-16 2020-06-12T02:39:05.337 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1693 (ObjNum 9520 snap 0 seq_num 9520) dirty exists 2020-06-12T02:39:05.338 INFO:tasks.rados.rados.0.smithi107.stdout:9521: left oid 16 (ObjNum 9520 snap 0 seq_num 9520) 2020-06-12T02:39:05.338 INFO:tasks.rados.rados.0.smithi107.stdout:9523: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:05.338 INFO:tasks.rados.rados.0.smithi107.stdout:9523: finishing write tid 2 to smithi10729407-14 2020-06-12T02:39:05.338 INFO:tasks.rados.rados.0.smithi107.stdout:9523: finishing write tid 4 to smithi10729407-14 2020-06-12T02:39:05.338 INFO:tasks.rados.rados.0.smithi107.stdout:9523: finishing write tid 5 to smithi10729407-14 2020-06-12T02:39:05.339 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1610 (ObjNum 9522 snap 0 seq_num 9522) dirty exists 2020-06-12T02:39:05.339 INFO:tasks.rados.rados.0.smithi107.stdout:9523: left oid 14 (ObjNum 9522 snap 0 seq_num 9522) 2020-06-12T02:39:05.339 INFO:tasks.rados.rados.0.smithi107.stdout:9521: done (3 left) 2020-06-12T02:39:05.339 INFO:tasks.rados.rados.0.smithi107.stdout:9523: done (2 left) 2020-06-12T02:39:05.339 INFO:tasks.rados.rados.0.smithi107.stdout:9525: done (1 left) 2020-06-12T02:39:05.340 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:39:05.340 INFO:tasks.rados.rados.0.smithi107.stdout:9526: seq_num 9525 ranges {0=729088,729088=401408} 2020-06-12T02:39:05.340 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010966700000000001 2020-06-12T02:39:05.340 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.340 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.341 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:05.341 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.490289+0000", 2020-06-12T02:39:05.341 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00032026800000000002 2020-06-12T02:39:05.342 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.342 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.342 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:05.342 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.490609+0000", 2020-06-12T02:39:05.342 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001102947 2020-06-12T02:39:05.343 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.343 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.343 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:05.343 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.491712+0000", 2020-06-12T02:39:05.343 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021050169999999998 2020-06-12T02:39:05.344 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.344 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.344 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:05.344 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493817+0000", 2020-06-12T02:39:05.344 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5927000000000006e-05 2020-06-12T02:39:05.344 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.345 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.345 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:05.345 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493903+0000", 2020-06-12T02:39:05.345 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.032981457999999998 2020-06-12T02:39:05.346 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:05.346 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:05.350 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:05.350 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.350 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.350 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13715 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e58)", 2020-06-12T02:39:05.350 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.461062+0000", 2020-06-12T02:39:05.351 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 36.639460323999998, 2020-06-12T02:39:05.351 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.035208338999999998, 2020-06-12T02:39:05.351 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:05.351 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:05.351 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:05.352 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:05.352 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:05.352 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13715 2020-06-12T02:39:05.352 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.352 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:05.353 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.353 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:05.353 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461062+0000", 2020-06-12T02:39:05.353 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:05.353 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.354 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.354 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:05.354 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461062+0000", 2020-06-12T02:39:05.354 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4839999999999998e-05 2020-06-12T02:39:05.354 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.356 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.357 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:05.357 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461107+0000", 2020-06-12T02:39:05.357 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4859999999999997e-06 2020-06-12T02:39:05.357 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.357 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.358 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:05.358 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461112+0000", 2020-06-12T02:39:05.359 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1742999999999999e-05 2020-06-12T02:39:05.360 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.361 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.361 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:05.361 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461124+0000", 2020-06-12T02:39:05.361 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7299999999999996e-06 2020-06-12T02:39:05.361 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.362 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.362 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:05.362 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461129+0000", 2020-06-12T02:39:05.362 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.030397368000000001 2020-06-12T02:39:05.362 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.363 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.363 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:05.363 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.491526+0000", 2020-06-12T02:39:05.363 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010076799999999999 2020-06-12T02:39:05.363 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.364 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.364 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:05.364 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.491627+0000", 2020-06-12T02:39:05.364 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031869849999999998 2020-06-12T02:39:05.364 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.364 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.365 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:05.365 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.494814+0000", 2020-06-12T02:39:05.365 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000108915 2020-06-12T02:39:05.365 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.365 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.367 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:05.367 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.494923+0000", 2020-06-12T02:39:05.367 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001347504 2020-06-12T02:39:05.367 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.367 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.368 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:05.368 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.496270+0000", 2020-06-12T02:39:05.368 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.035208338999999998 2020-06-12T02:39:05.368 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:05.368 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:05.369 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:05.369 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.369 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.369 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:16706 3.as0 3:5cb723d2:::smithi10729407-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e61)", 2020-06-12T02:39:05.369 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:37.289063+0000", 2020-06-12T02:39:05.370 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 27.811459299999999, 2020-06-12T02:39:05.370 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026146086999999998, 2020-06-12T02:39:05.375 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:05.375 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:05.375 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:05.376 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:05.376 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:05.376 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 16706 2020-06-12T02:39:05.376 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.376 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:05.377 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.377 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:05.377 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289063+0000", 2020-06-12T02:39:05.377 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:05.377 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.378 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.378 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:05.378 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289063+0000", 2020-06-12T02:39:05.378 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3776000000000002e-05 2020-06-12T02:39:05.379 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.379 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.379 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:05.379 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289107+0000", 2020-06-12T02:39:05.379 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7039999999999999e-06 2020-06-12T02:39:05.379 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.380 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.380 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:05.380 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289113+0000", 2020-06-12T02:39:05.380 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3259e-05 2020-06-12T02:39:05.380 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.381 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.381 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:05.381 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289126+0000", 2020-06-12T02:39:05.381 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1999999999999996e-06 2020-06-12T02:39:05.381 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.382 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.382 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:05.382 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289130+0000", 2020-06-12T02:39:05.382 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0010912820000000001 2020-06-12T02:39:05.382 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.383 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.383 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:05.383 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.290221+0000", 2020-06-12T02:39:05.383 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3166e-05 2020-06-12T02:39:05.385 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.386 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.386 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:05.386 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.290265+0000", 2020-06-12T02:39:05.386 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015984187 2020-06-12T02:39:05.387 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.387 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.387 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:05.387 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.306249+0000", 2020-06-12T02:39:05.387 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4623000000000006e-05 2020-06-12T02:39:05.388 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.388 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.388 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:05.388 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.306333+0000", 2020-06-12T02:39:05.388 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0068573690000000003 2020-06-12T02:39:05.389 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.389 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.389 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:05.389 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.313191+0000", 2020-06-12T02:39:05.390 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7804000000000002e-05 2020-06-12T02:39:05.390 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.391 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.391 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:05.392 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.313279+0000", 2020-06-12T02:39:05.392 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001930717 2020-06-12T02:39:05.393 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.393 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.393 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:05.394 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.315209+0000", 2020-06-12T02:39:05.394 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026146086999999998 2020-06-12T02:39:05.394 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:05.394 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:05.395 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:05.395 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.396 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.396 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18051 3.as0 3:5cb723d2:::smithi10729407-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:39:05.397 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.329078+0000", 2020-06-12T02:39:05.397 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 23.771444750000001, 2020-06-12T02:39:05.397 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026465103, 2020-06-12T02:39:05.398 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:05.398 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:05.398 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:05.399 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:05.400 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:05.400 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18051 2020-06-12T02:39:05.400 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.401 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:05.401 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.402 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:05.402 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329078+0000", 2020-06-12T02:39:05.403 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:05.404 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.404 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.404 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:05.405 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329078+0000", 2020-06-12T02:39:05.405 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5392000000000001e-05 2020-06-12T02:39:05.406 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.406 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.406 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:05.406 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329133+0000", 2020-06-12T02:39:05.407 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5329999999999997e-06 2020-06-12T02:39:05.407 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.407 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.407 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:05.408 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329141+0000", 2020-06-12T02:39:05.408 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3459999999999999e-05 2020-06-12T02:39:05.408 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.409 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.409 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:05.409 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329164+0000", 2020-06-12T02:39:05.409 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8170000000000001e-06 2020-06-12T02:39:05.409 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.410 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.410 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:05.411 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329171+0000", 2020-06-12T02:39:05.411 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013005208000000001 2020-06-12T02:39:05.411 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.412 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.412 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:05.412 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.342176+0000", 2020-06-12T02:39:05.413 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3560999999999998e-05 2020-06-12T02:39:05.413 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.413 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.414 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:05.414 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.342230+0000", 2020-06-12T02:39:05.414 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0080818490000000003 2020-06-12T02:39:05.414 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.415 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.415 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:05.415 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.350312+0000", 2020-06-12T02:39:05.416 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0167000000000006e-05 2020-06-12T02:39:05.416 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.416 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.417 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:05.417 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.350402+0000", 2020-06-12T02:39:05.418 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051411160000000003 2020-06-12T02:39:05.418 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.419 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.419 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:05.419 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.355543+0000", 2020-06-12T02:39:05.419 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026465103 2020-06-12T02:39:05.420 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:05.420 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:05.420 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:05.421 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.421 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.421 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18053 3.as0 3:5732055a:::smithi10729407-8:head [append 0~434176] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:05.422 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.334425+0000", 2020-06-12T02:39:05.423 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 23.766097182999999, 2020-06-12T02:39:05.423 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.029616703000000001, 2020-06-12T02:39:05.423 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:05.424 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:05.424 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:05.424 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:05.425 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:05.425 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18053 2020-06-12T02:39:05.425 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.426 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:05.426 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.427 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:05.427 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334425+0000", 2020-06-12T02:39:05.428 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:05.428 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.428 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.429 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:05.429 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334425+0000", 2020-06-12T02:39:05.430 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047283900000000002 2020-06-12T02:39:05.430 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.430 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.431 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:05.431 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334898+0000", 2020-06-12T02:39:05.431 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9100000000000004e-06 2020-06-12T02:39:05.432 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.432 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.432 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:05.433 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334903+0000", 2020-06-12T02:39:05.433 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4260000000000003e-06 2020-06-12T02:39:05.433 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.434 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.434 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:05.434 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334911+0000", 2020-06-12T02:39:05.435 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1060000000000002e-06 2020-06-12T02:39:05.435 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.435 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.436 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:05.436 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334916+0000", 2020-06-12T02:39:05.437 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010838316000000001 2020-06-12T02:39:05.437 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.437 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.437 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:05.438 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.345754+0000", 2020-06-12T02:39:05.438 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000103872 2020-06-12T02:39:05.438 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.439 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.439 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:05.439 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.345858+0000", 2020-06-12T02:39:05.439 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0059798739999999996 2020-06-12T02:39:05.440 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.440 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.440 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:05.440 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.351838+0000", 2020-06-12T02:39:05.441 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000104724 2020-06-12T02:39:05.441 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.441 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.441 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:05.441 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.351942+0000", 2020-06-12T02:39:05.442 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000572687 2020-06-12T02:39:05.442 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.442 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.442 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:05.443 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.352515+0000", 2020-06-12T02:39:05.443 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002905641 2020-06-12T02:39:05.443 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.444 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.444 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:05.445 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.355421+0000", 2020-06-12T02:39:05.445 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0085358350000000003 2020-06-12T02:39:05.445 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.446 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.446 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:05.447 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.363956+0000", 2020-06-12T02:39:05.447 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5473e-05 2020-06-12T02:39:05.448 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.448 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.449 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:05.449 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.364042+0000", 2020-06-12T02:39:05.449 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.029616703000000001 2020-06-12T02:39:05.449 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:05.450 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:05.451 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:05.451 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.451 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.452 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18054 3.as0 3:5732055a:::smithi10729407-8:head [append 0~638976] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:05.452 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.338171+0000", 2020-06-12T02:39:05.452 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 23.762351679999998, 2020-06-12T02:39:05.452 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.030067120999999999, 2020-06-12T02:39:05.452 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:05.453 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:05.453 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:05.453 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:05.453 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:05.453 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18054 2020-06-12T02:39:05.454 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.454 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:05.454 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.454 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:05.454 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338171+0000", 2020-06-12T02:39:05.455 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:05.455 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.455 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.455 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:05.455 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338171+0000", 2020-06-12T02:39:05.455 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00068138900000000002 2020-06-12T02:39:05.456 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.456 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.456 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:05.456 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338852+0000", 2020-06-12T02:39:05.456 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.767e-06 2020-06-12T02:39:05.457 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.457 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.457 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:05.457 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338858+0000", 2020-06-12T02:39:05.457 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8999999999999995e-06 2020-06-12T02:39:05.458 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.458 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.458 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:05.458 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338867+0000", 2020-06-12T02:39:05.458 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.262e-06 2020-06-12T02:39:05.459 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.459 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.459 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:05.459 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338871+0000", 2020-06-12T02:39:05.459 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0074839060000000002 2020-06-12T02:39:05.460 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.460 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.460 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:05.460 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346355+0000", 2020-06-12T02:39:05.460 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1435999999999999e-05 2020-06-12T02:39:05.461 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.461 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.461 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:05.461 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346416+0000", 2020-06-12T02:39:05.461 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0092109879999999998 2020-06-12T02:39:05.462 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.462 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.462 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:05.462 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.355627+0000", 2020-06-12T02:39:05.462 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7355999999999998e-05 2020-06-12T02:39:05.463 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.463 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.463 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:05.463 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.355685+0000", 2020-06-12T02:39:05.463 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00074711999999999999 2020-06-12T02:39:05.463 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.464 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.464 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:05.464 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.356432+0000", 2020-06-12T02:39:05.464 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010659572000000001 2020-06-12T02:39:05.464 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.465 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.465 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:05.465 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.367091+0000", 2020-06-12T02:39:05.465 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0010713210000000001 2020-06-12T02:39:05.465 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.466 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.466 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:05.466 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.368163+0000", 2020-06-12T02:39:05.466 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5104000000000005e-05 2020-06-12T02:39:05.466 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.467 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.467 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:05.467 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.368238+0000", 2020-06-12T02:39:05.467 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.030067120999999999 2020-06-12T02:39:05.467 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:05.467 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:05.468 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:05.468 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.468 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.468 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18055 3.as0 3:5732055a:::smithi10729407-8:head [append 0~483328] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:05.468 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.340740+0000", 2020-06-12T02:39:05.469 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 23.759782830999999, 2020-06-12T02:39:05.469 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028669050000000001, 2020-06-12T02:39:05.469 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:05.469 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:05.469 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:05.470 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:05.470 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:05.470 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18055 2020-06-12T02:39:05.470 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.470 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:05.470 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.471 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:05.471 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.340740+0000", 2020-06-12T02:39:05.471 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:05.471 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.471 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.472 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:05.472 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.340740+0000", 2020-06-12T02:39:05.472 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00045126900000000002 2020-06-12T02:39:05.472 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.472 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.472 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:05.473 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341191+0000", 2020-06-12T02:39:05.473 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5630000000000003e-06 2020-06-12T02:39:05.473 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.473 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.475 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:05.475 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341198+0000", 2020-06-12T02:39:05.475 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4467999999999999e-05 2020-06-12T02:39:05.475 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.476 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.476 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:05.476 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341213+0000", 2020-06-12T02:39:05.476 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7540000000000003e-06 2020-06-12T02:39:05.476 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.477 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.477 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:05.477 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341219+0000", 2020-06-12T02:39:05.477 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0052272680000000002 2020-06-12T02:39:05.477 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.478 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.478 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:05.478 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346446+0000", 2020-06-12T02:39:05.478 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4742999999999998e-05 2020-06-12T02:39:05.478 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.479 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.479 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:05.479 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346531+0000", 2020-06-12T02:39:05.479 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013203138999999999 2020-06-12T02:39:05.479 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.480 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.480 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:05.480 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.359734+0000", 2020-06-12T02:39:05.480 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000100277 2020-06-12T02:39:05.480 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.480 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.481 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:05.481 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.359834+0000", 2020-06-12T02:39:05.481 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054998099999999995 2020-06-12T02:39:05.481 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.481 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.482 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:05.482 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.360384+0000", 2020-06-12T02:39:05.482 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0067087179999999998 2020-06-12T02:39:05.482 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.482 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.482 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:05.483 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.367093+0000", 2020-06-12T02:39:05.486 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0022746020000000001 2020-06-12T02:39:05.486 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.486 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.486 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:05.486 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369367+0000", 2020-06-12T02:39:05.487 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1267999999999999e-05 2020-06-12T02:39:05.487 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.487 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.487 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:05.487 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369409+0000", 2020-06-12T02:39:05.488 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028669050000000001 2020-06-12T02:39:05.488 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:05.488 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:05.488 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:05.488 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.489 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.489 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18056 3.as0 3:5732055a:::smithi10729407-8:head [append 0~49152] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:05.489 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.341270+0000", 2020-06-12T02:39:05.489 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 23.759252042, 2020-06-12T02:39:05.489 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028350462999999999, 2020-06-12T02:39:05.490 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:05.490 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:05.490 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:05.490 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:05.490 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:05.491 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18056 2020-06-12T02:39:05.491 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.491 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:05.491 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.491 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:05.491 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341270+0000", 2020-06-12T02:39:05.492 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:05.492 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.492 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.492 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:05.492 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341270+0000", 2020-06-12T02:39:05.493 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000114082 2020-06-12T02:39:05.493 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.493 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.493 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:05.493 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341384+0000", 2020-06-12T02:39:05.494 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3289999999999999e-06 2020-06-12T02:39:05.494 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.494 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.494 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:05.494 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341391+0000", 2020-06-12T02:39:05.494 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3339999999999999e-05 2020-06-12T02:39:05.495 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.495 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.495 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:05.495 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341404+0000", 2020-06-12T02:39:05.495 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4e-06 2020-06-12T02:39:05.496 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.496 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.496 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:05.496 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341410+0000", 2020-06-12T02:39:05.496 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051518600000000003 2020-06-12T02:39:05.496 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.497 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.497 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:05.497 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346561+0000", 2020-06-12T02:39:05.497 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1662999999999994e-05 2020-06-12T02:39:05.497 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.498 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.498 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:05.498 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346643+0000", 2020-06-12T02:39:05.498 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017471605000000001 2020-06-12T02:39:05.498 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.499 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.499 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:05.499 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.364115+0000", 2020-06-12T02:39:05.499 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011059400000000001 2020-06-12T02:39:05.499 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.500 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.500 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:05.500 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.364225+0000", 2020-06-12T02:39:05.500 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000611262 2020-06-12T02:39:05.500 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.500 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.501 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:05.501 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.364837+0000", 2020-06-12T02:39:05.501 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023767710000000002 2020-06-12T02:39:05.501 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.501 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.502 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:05.502 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.367213+0000", 2020-06-12T02:39:05.502 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023325479999999998 2020-06-12T02:39:05.502 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.502 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.503 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:05.503 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369546+0000", 2020-06-12T02:39:05.503 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5009e-05 2020-06-12T02:39:05.503 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.503 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.504 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:05.504 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369621+0000", 2020-06-12T02:39:05.504 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028350462999999999 2020-06-12T02:39:05.504 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:05.504 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:05.504 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:05.505 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.505 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.505 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18057 3.as0 3:5732055a:::smithi10729407-8:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:05.505 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.341456+0000", 2020-06-12T02:39:05.505 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 23.759066881999999, 2020-06-12T02:39:05.506 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028443018, 2020-06-12T02:39:05.506 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:05.506 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:05.506 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:05.506 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:05.506 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:05.507 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18057 2020-06-12T02:39:05.507 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.507 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:05.507 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.507 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:05.508 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341456+0000", 2020-06-12T02:39:05.508 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:05.508 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.508 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.508 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:05.508 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341456+0000", 2020-06-12T02:39:05.509 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6036000000000003e-05 2020-06-12T02:39:05.509 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.509 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.509 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:05.509 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341522+0000", 2020-06-12T02:39:05.510 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2210000000000002e-06 2020-06-12T02:39:05.510 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.510 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.510 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:05.511 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341528+0000", 2020-06-12T02:39:05.511 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0338e-05 2020-06-12T02:39:05.511 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.511 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.511 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:05.512 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341538+0000", 2020-06-12T02:39:05.512 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8639999999999998e-06 2020-06-12T02:39:05.512 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.512 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.513 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:05.513 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341542+0000", 2020-06-12T02:39:05.513 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051405009999999996 2020-06-12T02:39:05.513 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.514 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.514 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:05.514 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346683+0000", 2020-06-12T02:39:05.514 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9561999999999995e-05 2020-06-12T02:39:05.514 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.514 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.515 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:05.515 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346762+0000", 2020-06-12T02:39:05.515 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019267197999999999 2020-06-12T02:39:05.515 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.515 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.516 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:05.516 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.366029+0000", 2020-06-12T02:39:05.516 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2634999999999997e-05 2020-06-12T02:39:05.516 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.516 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.517 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:05.517 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.366122+0000", 2020-06-12T02:39:05.517 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00026907500000000002 2020-06-12T02:39:05.517 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.517 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.518 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:05.518 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.366391+0000", 2020-06-12T02:39:05.518 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0018939409999999999 2020-06-12T02:39:05.518 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.518 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.519 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:05.519 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.368285+0000", 2020-06-12T02:39:05.519 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001549405 2020-06-12T02:39:05.519 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.520 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.520 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:05.520 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369834+0000", 2020-06-12T02:39:05.521 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4241999999999999e-05 2020-06-12T02:39:05.521 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.521 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.521 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:05.521 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369899+0000", 2020-06-12T02:39:05.522 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028443018 2020-06-12T02:39:05.522 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:05.522 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:05.522 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:05.522 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.522 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.523 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18058 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:39:05.523 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.341580+0000", 2020-06-12T02:39:05.523 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 23.758941974999999, 2020-06-12T02:39:05.523 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.030449559000000001, 2020-06-12T02:39:05.523 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:05.524 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:05.524 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:05.524 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:05.524 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:05.524 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18058 2020-06-12T02:39:05.525 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.525 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:05.525 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.525 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:05.525 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341580+0000", 2020-06-12T02:39:05.525 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:05.526 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.526 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.526 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:05.526 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341580+0000", 2020-06-12T02:39:05.526 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1185e-05 2020-06-12T02:39:05.527 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.527 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.527 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:05.527 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341622+0000", 2020-06-12T02:39:05.527 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1599999999999997e-06 2020-06-12T02:39:05.527 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.528 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.528 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:05.528 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341627+0000", 2020-06-12T02:39:05.528 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3546e-05 2020-06-12T02:39:05.528 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.529 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.529 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:05.530 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341640+0000", 2020-06-12T02:39:05.530 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9779999999999999e-06 2020-06-12T02:39:05.530 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.531 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.531 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:05.531 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341644+0000", 2020-06-12T02:39:05.531 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051433700000000004 2020-06-12T02:39:05.531 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.531 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.532 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:05.532 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346788+0000", 2020-06-12T02:39:05.532 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1622000000000001e-05 2020-06-12T02:39:05.532 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.532 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.533 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:05.533 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346839+0000", 2020-06-12T02:39:05.533 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020283958000000001 2020-06-12T02:39:05.533 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.533 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.534 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:05.534 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.367123+0000", 2020-06-12T02:39:05.534 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1997999999999997e-05 2020-06-12T02:39:05.534 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.534 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.535 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:05.535 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.367205+0000", 2020-06-12T02:39:05.535 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036875440000000001 2020-06-12T02:39:05.535 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.535 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.535 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:05.536 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.370893+0000", 2020-06-12T02:39:05.536 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000100604 2020-06-12T02:39:05.536 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.536 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.536 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:05.537 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.370993+0000", 2020-06-12T02:39:05.537 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0010365940000000001 2020-06-12T02:39:05.537 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:05.537 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:05.537 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:05.538 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.372030+0000", 2020-06-12T02:39:05.538 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.030449559000000001 2020-06-12T02:39:05.538 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:05.538 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:05.538 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:05.538 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:05.539 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:05.539 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:39:05.540 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-06-12T02:39:05.540 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:39:05.294+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:39:05.551 INFO:tasks.rados.rados.0.smithi107.stdout:9526: writing smithi10729407-11 from 0 to 729088 tid 1 2020-06-12T02:39:05.551 INFO:tasks.rados.rados.0.smithi107.stdout:9526: writing smithi10729407-11 from 729088 to 1130496 tid 2 2020-06-12T02:39:05.551 INFO:tasks.rados.rados.0.smithi107.stdout:9527: delete oid 17 current snap is 0 2020-06-12T02:39:05.551 INFO:tasks.rados.rados.0.smithi107.stdout:9524: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:05.552 INFO:tasks.rados.rados.0.smithi107.stdout:9524: finishing write tid 2 to smithi10729407-10 2020-06-12T02:39:05.552 INFO:tasks.rados.rados.0.smithi107.stdout:9524: finishing write tid 4 to smithi10729407-10 2020-06-12T02:39:05.552 INFO:tasks.rados.rados.0.smithi107.stdout:9524: finishing write tid 5 to smithi10729407-10 2020-06-12T02:39:05.552 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 676 (ObjNum 9523 snap 0 seq_num 9523) dirty exists 2020-06-12T02:39:05.552 INFO:tasks.rados.rados.0.smithi107.stdout:9524: left oid 10 (ObjNum 9523 snap 0 seq_num 9523) 2020-06-12T02:39:05.553 INFO:tasks.rados.rados.0.smithi107.stdout:9524: done (2 left) 2020-06-12T02:39:05.553 INFO:tasks.rados.rados.0.smithi107.stdout:9527: done (1 left) 2020-06-12T02:39:05.553 INFO:tasks.rados.rados.0.smithi107.stdout:9528: delete oid 8 current snap is 0 2020-06-12T02:39:05.553 INFO:tasks.rados.rados.0.smithi107.stdout:9526: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:05.553 INFO:tasks.rados.rados.0.smithi107.stdout:9528: done (1 left) 2020-06-12T02:39:05.554 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:39:05.554 INFO:tasks.rados.rados.0.smithi107.stdout:9529: seq_num 9528 ranges {761856=778240,1540096=401408} 2020-06-12T02:39:05.554 INFO:tasks.rados.rados.0.smithi107.stdout:9529: writing smithi10729407-3 from 761856 to 1540096 tid 1 2020-06-12T02:39:05.554 INFO:tasks.rados.rados.0.smithi107.stdout:9529: writing smithi10729407-3 from 1540096 to 1941504 tid 2 2020-06-12T02:39:05.555 INFO:tasks.rados.rados.0.smithi107.stdout:9530: delete oid 1 current snap is 0 2020-06-12T02:39:05.555 INFO:tasks.rados.rados.0.smithi107.stdout:9526: finishing write tid 2 to smithi10729407-11 2020-06-12T02:39:05.555 INFO:tasks.rados.rados.0.smithi107.stdout:9526: finishing write tid 4 to smithi10729407-11 2020-06-12T02:39:05.555 INFO:tasks.rados.rados.0.smithi107.stdout:9526: finishing write tid 5 to smithi10729407-11 2020-06-12T02:39:05.555 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1630 (ObjNum 9525 snap 0 seq_num 9525) dirty exists 2020-06-12T02:39:05.556 INFO:tasks.rados.rados.0.smithi107.stdout:9526: left oid 11 (ObjNum 9525 snap 0 seq_num 9525) 2020-06-12T02:39:05.556 INFO:tasks.rados.rados.0.smithi107.stdout:9526: done (2 left) 2020-06-12T02:39:05.556 INFO:tasks.rados.rados.0.smithi107.stdout:9530: done (1 left) 2020-06-12T02:39:05.556 INFO:tasks.rados.rados.0.smithi107.stdout:9531: delete oid 14 current snap is 0 2020-06-12T02:39:05.556 INFO:tasks.rados.rados.0.smithi107.stdout:9529: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:05.557 INFO:tasks.rados.rados.0.smithi107.stdout:9531: done (1 left) 2020-06-12T02:39:05.557 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:39:05.557 INFO:tasks.rados.rados.0.smithi107.stdout:9532: seq_num 9531 ranges {0=434176,434176=622592,1056768=180224} 2020-06-12T02:39:05.557 INFO:tasks.rados.rados.0.smithi107.stdout:9532: writing smithi10729407-19 from 0 to 434176 tid 1 2020-06-12T02:39:05.557 INFO:tasks.rados.rados.0.smithi107.stdout:9532: writing smithi10729407-19 from 434176 to 1056768 tid 2 2020-06-12T02:39:05.557 INFO:tasks.rados.rados.0.smithi107.stdout:9532: writing smithi10729407-19 from 1056768 to 1236992 tid 3 2020-06-12T02:39:05.558 INFO:tasks.rados.rados.0.smithi107.stdout:9533: delete oid 6 current snap is 0 2020-06-12T02:39:05.558 INFO:tasks.rados.rados.0.smithi107.stdout:9529: finishing write tid 2 to smithi10729407-3 2020-06-12T02:39:05.558 INFO:tasks.rados.rados.0.smithi107.stdout:9529: finishing write tid 4 to smithi10729407-3 2020-06-12T02:39:05.558 INFO:tasks.rados.rados.0.smithi107.stdout:9532: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:05.558 INFO:tasks.rados.rados.0.smithi107.stdout:9529: finishing write tid 5 to smithi10729407-3 2020-06-12T02:39:05.559 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1696 (ObjNum 9528 snap 0 seq_num 9528) dirty exists 2020-06-12T02:39:05.559 INFO:tasks.rados.rados.0.smithi107.stdout:9529: left oid 3 (ObjNum 9528 snap 0 seq_num 9528) 2020-06-12T02:39:05.559 INFO:tasks.rados.rados.0.smithi107.stdout:9529: done (2 left) 2020-06-12T02:39:05.559 INFO:tasks.rados.rados.0.smithi107.stdout:9533: done (1 left) 2020-06-12T02:39:05.559 INFO:tasks.rados.rados.0.smithi107.stdout:9534: delete oid 11 current snap is 0 2020-06-12T02:39:05.560 INFO:tasks.rados.rados.0.smithi107.stdout:9534: done (1 left) 2020-06-12T02:39:05.560 INFO:tasks.rados.rados.0.smithi107.stdout:9535: delete oid 5 current snap is 0 2020-06-12T02:39:05.560 INFO:tasks.rados.rados.0.smithi107.stdout:9532: finishing write tid 2 to smithi10729407-19 2020-06-12T02:39:05.560 INFO:tasks.rados.rados.0.smithi107.stdout:9532: finishing write tid 3 to smithi10729407-19 2020-06-12T02:39:05.560 INFO:tasks.rados.rados.0.smithi107.stdout:9532: finishing write tid 5 to smithi10729407-19 2020-06-12T02:39:05.560 INFO:tasks.rados.rados.0.smithi107.stdout:9532: finishing write tid 6 to smithi10729407-19 2020-06-12T02:39:05.561 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1754 (ObjNum 9531 snap 0 seq_num 9531) dirty exists 2020-06-12T02:39:05.561 INFO:tasks.rados.rados.0.smithi107.stdout:9532: left oid 19 (ObjNum 9531 snap 0 seq_num 9531) 2020-06-12T02:39:05.561 INFO:tasks.rados.rados.0.smithi107.stdout:9532: done (1 left) 2020-06-12T02:39:05.561 INFO:tasks.rados.rados.0.smithi107.stdout:9535: done (0 left) 2020-06-12T02:39:05.561 INFO:tasks.rados.rados.0.smithi107.stdout:9536: delete oid 19 current snap is 0 2020-06-12T02:39:05.562 INFO:tasks.rados.rados.0.smithi107.stdout:9536: done (0 left) 2020-06-12T02:39:05.562 INFO:tasks.rados.rados.0.smithi107.stdout:9537: delete oid 11 current snap is 0 2020-06-12T02:39:05.562 INFO:tasks.rados.rados.0.smithi107.stdout:9537: done (0 left) 2020-06-12T02:39:05.562 INFO:tasks.rados.rados.0.smithi107.stdout:9538: delete oid 11 current snap is 0 2020-06-12T02:39:05.562 INFO:tasks.rados.rados.0.smithi107.stdout:9538: done (0 left) 2020-06-12T02:39:05.562 INFO:tasks.rados.rados.0.smithi107.stdout:9539: delete oid 19 current snap is 0 2020-06-12T02:39:05.563 INFO:tasks.rados.rados.0.smithi107.stdout:9539: done (0 left) 2020-06-12T02:39:05.563 INFO:tasks.rados.rados.0.smithi107.stdout:9540: delete oid 19 current snap is 0 2020-06-12T02:39:05.563 INFO:tasks.rados.rados.0.smithi107.stdout:9540: done (0 left) 2020-06-12T02:39:05.563 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:39:05.563 INFO:tasks.rados.rados.0.smithi107.stdout:9541: seq_num 9540 ranges {0=581632,581632=434176,1015808=368640} 2020-06-12T02:39:05.564 INFO:tasks.rados.rados.0.smithi107.stdout:9541: writing smithi10729407-9 from 0 to 581632 tid 1 2020-06-12T02:39:05.564 INFO:tasks.rados.rados.0.smithi107.stdout:9541: writing smithi10729407-9 from 581632 to 1015808 tid 2 2020-06-12T02:39:05.564 INFO:tasks.rados.rados.0.smithi107.stdout:9541: writing smithi10729407-9 from 1015808 to 1384448 tid 3 2020-06-12T02:39:05.564 INFO:tasks.rados.rados.0.smithi107.stdout:9542: delete oid 8 current snap is 0 2020-06-12T02:39:05.564 INFO:tasks.rados.rados.0.smithi107.stdout:9542: done (1 left) 2020-06-12T02:39:05.565 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:39:05.565 INFO:tasks.rados.rados.0.smithi107.stdout:9543: seq_num 9542 ranges {1941504=614400,2555904=647168,3203072=155648} 2020-06-12T02:39:05.565 INFO:tasks.rados.rados.0.smithi107.stdout:9543: writing smithi10729407-3 from 1941504 to 2555904 tid 1 2020-06-12T02:39:05.565 INFO:tasks.rados.rados.0.smithi107.stdout:9543: writing smithi10729407-3 from 2555904 to 3203072 tid 2 2020-06-12T02:39:05.565 INFO:tasks.rados.rados.0.smithi107.stdout:9543: writing smithi10729407-3 from 3203072 to 3358720 tid 3 2020-06-12T02:39:05.565 INFO:tasks.rados.rados.0.smithi107.stdout:9544: delete oid 15 current snap is 0 2020-06-12T02:39:05.566 INFO:tasks.rados.rados.0.smithi107.stdout:9541: finishing write tid 1 to smithi10729407-9 2020-06-12T02:39:05.566 INFO:tasks.rados.rados.0.smithi107.stdout:9541: finishing write tid 2 to smithi10729407-9 2020-06-12T02:39:05.566 INFO:tasks.rados.rados.0.smithi107.stdout:9541: finishing write tid 3 to smithi10729407-9 2020-06-12T02:39:05.566 INFO:tasks.rados.rados.0.smithi107.stdout:9541: finishing write tid 5 to smithi10729407-9 2020-06-12T02:39:05.566 INFO:tasks.rados.rados.0.smithi107.stdout:9541: finishing write tid 6 to smithi10729407-9 2020-06-12T02:39:05.567 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1738 (ObjNum 9540 snap 0 seq_num 9540) dirty exists 2020-06-12T02:39:05.567 INFO:tasks.rados.rados.0.smithi107.stdout:9541: left oid 9 (ObjNum 9540 snap 0 seq_num 9540) 2020-06-12T02:39:05.567 INFO:tasks.rados.rados.0.smithi107.stdout:9541: done (2 left) 2020-06-12T02:39:05.567 INFO:tasks.rados.rados.0.smithi107.stdout:9544: done (1 left) 2020-06-12T02:39:05.567 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:39:05.568 INFO:tasks.rados.rados.0.smithi107.stdout:9545: seq_num 9544 ranges {0=647168,647168=417792,1064960=385024} 2020-06-12T02:39:05.568 INFO:tasks.rados.rados.0.smithi107.stdout:9545: writing smithi10729407-12 from 0 to 647168 tid 1 2020-06-12T02:39:05.568 INFO:tasks.rados.rados.0.smithi107.stdout:9545: writing smithi10729407-12 from 647168 to 1064960 tid 2 2020-06-12T02:39:05.568 INFO:tasks.rados.rados.0.smithi107.stdout:9545: writing smithi10729407-12 from 1064960 to 1449984 tid 3 2020-06-12T02:39:05.568 INFO:tasks.rados.rados.0.smithi107.stdout:9546: delete oid 6 current snap is 0 2020-06-12T02:39:05.569 INFO:tasks.rados.rados.0.smithi107.stdout:9543: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:05.569 INFO:tasks.rados.rados.0.smithi107.stdout:9543: finishing write tid 2 to smithi10729407-3 2020-06-12T02:39:05.569 INFO:tasks.rados.rados.0.smithi107.stdout:9543: finishing write tid 3 to smithi10729407-3 2020-06-12T02:39:05.569 INFO:tasks.rados.rados.0.smithi107.stdout:9543: finishing write tid 5 to smithi10729407-3 2020-06-12T02:39:05.569 INFO:tasks.rados.rados.0.smithi107.stdout:9543: finishing write tid 6 to smithi10729407-3 2020-06-12T02:39:05.570 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1700 (ObjNum 9542 snap 0 seq_num 9542) dirty exists 2020-06-12T02:39:05.570 INFO:tasks.rados.rados.0.smithi107.stdout:9543: left oid 3 (ObjNum 9542 snap 0 seq_num 9542) 2020-06-12T02:39:05.570 INFO:tasks.rados.rados.0.smithi107.stdout:9543: done (2 left) 2020-06-12T02:39:05.571 INFO:tasks.rados.rados.0.smithi107.stdout:9546: done (1 left) 2020-06-12T02:39:05.571 INFO:tasks.rados.rados.0.smithi107.stdout:9547: delete oid 19 current snap is 0 2020-06-12T02:39:05.571 INFO:tasks.rados.rados.0.smithi107.stdout:9545: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:05.571 INFO:tasks.rados.rados.0.smithi107.stdout:9545: finishing write tid 2 to smithi10729407-12 2020-06-12T02:39:05.571 INFO:tasks.rados.rados.0.smithi107.stdout:9545: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:05.572 INFO:tasks.rados.rados.0.smithi107.stdout:9545: finishing write tid 5 to smithi10729407-12 2020-06-12T02:39:05.572 INFO:tasks.rados.rados.0.smithi107.stdout:9545: finishing write tid 6 to smithi10729407-12 2020-06-12T02:39:05.572 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1761 (ObjNum 9544 snap 0 seq_num 9544) dirty exists 2020-06-12T02:39:05.572 INFO:tasks.rados.rados.0.smithi107.stdout:9545: left oid 12 (ObjNum 9544 snap 0 seq_num 9544) 2020-06-12T02:39:05.572 INFO:tasks.rados.rados.0.smithi107.stdout:9545: done (1 left) 2020-06-12T02:39:05.572 INFO:tasks.rados.rados.0.smithi107.stdout:9547: done (0 left) 2020-06-12T02:39:05.573 INFO:tasks.rados.rados.0.smithi107.stdout:9548: delete oid 12 current snap is 0 2020-06-12T02:39:05.573 INFO:tasks.rados.rados.0.smithi107.stdout:9548: done (0 left) 2020-06-12T02:39:05.573 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:39:05.573 INFO:tasks.rados.rados.0.smithi107.stdout:9549: seq_num 9548 ranges {2555904=720896,3276800=753664,4030464=49152} 2020-06-12T02:39:05.573 INFO:tasks.rados.rados.0.smithi107.stdout:9549: writing smithi10729407-16 from 2555904 to 3276800 tid 1 2020-06-12T02:39:05.574 INFO:tasks.rados.rados.0.smithi107.stdout:9549: writing smithi10729407-16 from 3276800 to 4030464 tid 2 2020-06-12T02:39:05.579 INFO:tasks.rados.rados.0.smithi107.stdout:9549: writing smithi10729407-16 from 4030464 to 4079616 tid 3 2020-06-12T02:39:05.579 INFO:tasks.rados.rados.0.smithi107.stdout:9550: delete oid 3 current snap is 0 2020-06-12T02:39:05.579 INFO:tasks.rados.rados.0.smithi107.stdout:9549: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:05.580 INFO:tasks.rados.rados.0.smithi107.stdout:9550: done (1 left) 2020-06-12T02:39:05.580 INFO:tasks.rados.rados.0.smithi107.stdout:9551: delete oid 13 current snap is 0 2020-06-12T02:39:05.580 INFO:tasks.rados.rados.0.smithi107.stdout:9551: done (1 left) 2020-06-12T02:39:05.580 INFO:tasks.rados.rados.0.smithi107.stdout:9552: delete oid 20 current snap is 0 2020-06-12T02:39:05.580 INFO:tasks.rados.rados.0.smithi107.stdout:9549: finishing write tid 2 to smithi10729407-16 2020-06-12T02:39:05.581 INFO:tasks.rados.rados.0.smithi107.stdout:9549: finishing write tid 3 to smithi10729407-16 2020-06-12T02:39:05.581 INFO:tasks.rados.rados.0.smithi107.stdout:9549: finishing write tid 5 to smithi10729407-16 2020-06-12T02:39:05.581 INFO:tasks.rados.rados.0.smithi107.stdout:9552: done (1 left) 2020-06-12T02:39:05.581 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:39:05.581 INFO:tasks.rados.rados.0.smithi107.stdout:9553: seq_num 9552 ranges {0=786432,786432=745472,1531904=57344} 2020-06-12T02:39:05.582 INFO:tasks.rados.rados.0.smithi107.stdout:9553: writing smithi10729407-17 from 0 to 786432 tid 1 2020-06-12T02:39:05.582 INFO:tasks.rados.rados.0.smithi107.stdout:9553: writing smithi10729407-17 from 786432 to 1531904 tid 2 2020-06-12T02:39:05.582 INFO:tasks.rados.rados.0.smithi107.stdout:9553: writing smithi10729407-17 from 1531904 to 1589248 tid 3 2020-06-12T02:39:05.582 INFO:tasks.rados.rados.0.smithi107.stdout:9554: delete oid 14 current snap is 0 2020-06-12T02:39:05.582 INFO:tasks.rados.rados.0.smithi107.stdout:9549: finishing write tid 6 to smithi10729407-16 2020-06-12T02:39:05.583 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1705 (ObjNum 9548 snap 0 seq_num 9548) dirty exists 2020-06-12T02:39:05.583 INFO:tasks.rados.rados.0.smithi107.stdout:9549: left oid 16 (ObjNum 9548 snap 0 seq_num 9548) 2020-06-12T02:39:05.583 INFO:tasks.rados.rados.0.smithi107.stdout:9553: finishing write tid 1 to smithi10729407-17 2020-06-12T02:39:05.583 INFO:tasks.rados.rados.0.smithi107.stdout:9549: done (2 left) 2020-06-12T02:39:05.583 INFO:tasks.rados.rados.0.smithi107.stdout:9554: done (1 left) 2020-06-12T02:39:05.583 INFO:tasks.rados.rados.0.smithi107.stdout:9555: delete oid 8 current snap is 0 2020-06-12T02:39:05.584 INFO:tasks.rados.rados.0.smithi107.stdout:9555: done (1 left) 2020-06-12T02:39:05.584 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:39:05.584 INFO:tasks.rados.rados.0.smithi107.stdout:9556: seq_num 9555 ranges {1925120=434176,2359296=794624,3153920=409600} 2020-06-12T02:39:05.584 INFO:tasks.rados.rados.0.smithi107.stdout:9556: writing smithi10729407-10 from 1925120 to 2359296 tid 1 2020-06-12T02:39:05.584 INFO:tasks.rados.rados.0.smithi107.stdout:9556: writing smithi10729407-10 from 2359296 to 3153920 tid 2 2020-06-12T02:39:05.585 INFO:tasks.rados.rados.0.smithi107.stdout:9556: writing smithi10729407-10 from 3153920 to 3563520 tid 3 2020-06-12T02:39:05.585 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:39:05.585 INFO:tasks.rados.rados.0.smithi107.stdout:9557: seq_num 9556 ranges {1384448=450560,1835008=679936,2514944=524288} 2020-06-12T02:39:05.585 INFO:tasks.rados.rados.0.smithi107.stdout:9557: writing smithi10729407-9 from 1384448 to 1835008 tid 1 2020-06-12T02:39:05.585 INFO:tasks.rados.rados.0.smithi107.stdout:9557: writing smithi10729407-9 from 1835008 to 2514944 tid 2 2020-06-12T02:39:05.586 INFO:tasks.rados.rados.0.smithi107.stdout:9557: writing smithi10729407-9 from 2514944 to 3039232 tid 3 2020-06-12T02:39:05.586 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:39:05.586 INFO:tasks.rados.rados.0.smithi107.stdout:9558: seq_num 9557 ranges {0=466944,466944=614400,1081344=524288,1605632=65536} 2020-06-12T02:39:05.586 INFO:tasks.rados.rados.0.smithi107.stdout:9558: writing smithi10729407-7 from 0 to 466944 tid 1 2020-06-12T02:39:05.586 INFO:tasks.rados.rados.0.smithi107.stdout:9558: writing smithi10729407-7 from 466944 to 1081344 tid 2 2020-06-12T02:39:05.586 INFO:tasks.rados.rados.0.smithi107.stdout:9558: writing smithi10729407-7 from 1081344 to 1605632 tid 3 2020-06-12T02:39:05.587 INFO:tasks.rados.rados.0.smithi107.stdout:9558: writing smithi10729407-7 from 1605632 to 1671168 tid 4 2020-06-12T02:39:05.587 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:39:05.587 INFO:tasks.rados.rados.0.smithi107.stdout:9559: seq_num 9558 ranges {0=483328,483328=499712,983040=704512} 2020-06-12T02:39:05.587 INFO:tasks.rados.rados.0.smithi107.stdout:9559: writing smithi10729407-18 from 0 to 483328 tid 1 2020-06-12T02:39:05.587 INFO:tasks.rados.rados.0.smithi107.stdout:9559: writing smithi10729407-18 from 483328 to 983040 tid 2 2020-06-12T02:39:05.588 INFO:tasks.rados.rados.0.smithi107.stdout:9559: writing smithi10729407-18 from 983040 to 1687552 tid 3 2020-06-12T02:39:05.588 INFO:tasks.rados.rados.0.smithi107.stdout:9560: delete oid 6 current snap is 0 2020-06-12T02:39:05.588 INFO:tasks.rados.rados.0.smithi107.stdout:9553: finishing write tid 2 to smithi10729407-17 2020-06-12T02:39:05.588 INFO:tasks.rados.rados.0.smithi107.stdout:9553: finishing write tid 3 to smithi10729407-17 2020-06-12T02:39:05.588 INFO:tasks.rados.rados.0.smithi107.stdout:9553: finishing write tid 5 to smithi10729407-17 2020-06-12T02:39:05.589 INFO:tasks.rados.rados.0.smithi107.stdout:9553: finishing write tid 6 to smithi10729407-17 2020-06-12T02:39:05.589 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1796 (ObjNum 9552 snap 0 seq_num 9552) dirty exists 2020-06-12T02:39:05.589 INFO:tasks.rados.rados.0.smithi107.stdout:9553: left oid 17 (ObjNum 9552 snap 0 seq_num 9552) 2020-06-12T02:39:05.589 INFO:tasks.rados.rados.0.smithi107.stdout:9556: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:05.590 INFO:tasks.rados.rados.0.smithi107.stdout:9556: finishing write tid 2 to smithi10729407-10 2020-06-12T02:39:05.590 INFO:tasks.rados.rados.0.smithi107.stdout:9556: finishing write tid 3 to smithi10729407-10 2020-06-12T02:39:05.590 INFO:tasks.rados.rados.0.smithi107.stdout:9556: finishing write tid 5 to smithi10729407-10 2020-06-12T02:39:05.590 INFO:tasks.rados.rados.0.smithi107.stdout:9556: finishing write tid 6 to smithi10729407-10 2020-06-12T02:39:05.590 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 680 (ObjNum 9555 snap 0 seq_num 9555) dirty exists 2020-06-12T02:39:05.591 INFO:tasks.rados.rados.0.smithi107.stdout:9556: left oid 10 (ObjNum 9555 snap 0 seq_num 9555) 2020-06-12T02:39:05.591 INFO:tasks.rados.rados.0.smithi107.stdout:9557: finishing write tid 1 to smithi10729407-9 2020-06-12T02:39:05.591 INFO:tasks.rados.rados.0.smithi107.stdout:9557: finishing write tid 2 to smithi10729407-9 2020-06-12T02:39:05.591 INFO:tasks.rados.rados.0.smithi107.stdout:9557: finishing write tid 3 to smithi10729407-9 2020-06-12T02:39:05.591 INFO:tasks.rados.rados.0.smithi107.stdout:9557: finishing write tid 5 to smithi10729407-9 2020-06-12T02:39:05.591 INFO:tasks.rados.rados.0.smithi107.stdout:9558: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:05.592 INFO:tasks.rados.rados.0.smithi107.stdout:9557: finishing write tid 6 to smithi10729407-9 2020-06-12T02:39:05.592 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1742 (ObjNum 9556 snap 0 seq_num 9556) dirty exists 2020-06-12T02:39:05.592 INFO:tasks.rados.rados.0.smithi107.stdout:9557: left oid 9 (ObjNum 9556 snap 0 seq_num 9556) 2020-06-12T02:39:05.592 INFO:tasks.rados.rados.0.smithi107.stdout:9558: finishing write tid 2 to smithi10729407-7 2020-06-12T02:39:05.592 INFO:tasks.rados.rados.0.smithi107.stdout:9558: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:05.593 INFO:tasks.rados.rados.0.smithi107.stdout:9558: finishing write tid 4 to smithi10729407-7 2020-06-12T02:39:05.593 INFO:tasks.rados.rados.0.smithi107.stdout:9558: finishing write tid 6 to smithi10729407-7 2020-06-12T02:39:05.593 INFO:tasks.rados.rados.0.smithi107.stdout:9553: done (5 left) 2020-06-12T02:39:05.593 INFO:tasks.rados.rados.0.smithi107.stdout:9556: done (4 left) 2020-06-12T02:39:05.593 INFO:tasks.rados.rados.0.smithi107.stdout:9557: done (3 left) 2020-06-12T02:39:05.593 INFO:tasks.rados.rados.0.smithi107.stdout:9560: done (2 left) 2020-06-12T02:39:05.594 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:39:05.594 INFO:tasks.rados.rados.0.smithi107.stdout:9561: seq_num 9560 ranges {0=516096,516096=663552,1179648=540672} 2020-06-12T02:39:05.594 INFO:tasks.rados.rados.0.smithi107.stdout:9561: writing smithi10729407-12 from 0 to 516096 tid 1 2020-06-12T02:39:05.594 INFO:tasks.rados.rados.0.smithi107.stdout:9561: writing smithi10729407-12 from 516096 to 1179648 tid 2 2020-06-12T02:39:05.594 INFO:tasks.rados.rados.0.smithi107.stdout:9561: writing smithi10729407-12 from 1179648 to 1720320 tid 3 2020-06-12T02:39:05.595 INFO:tasks.rados.rados.0.smithi107.stdout:9562: delete oid 17 current snap is 0 2020-06-12T02:39:05.595 INFO:tasks.rados.rados.0.smithi107.stdout:9558: finishing write tid 7 to smithi10729407-7 2020-06-12T02:39:05.595 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1654 (ObjNum 9557 snap 0 seq_num 9557) dirty exists 2020-06-12T02:39:05.595 INFO:tasks.rados.rados.0.smithi107.stdout:9558: left oid 7 (ObjNum 9557 snap 0 seq_num 9557) 2020-06-12T02:39:05.595 INFO:tasks.rados.rados.0.smithi107.stdout:9559: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:05.595 INFO:tasks.rados.rados.0.smithi107.stdout:9559: finishing write tid 2 to smithi10729407-18 2020-06-12T02:39:05.596 INFO:tasks.rados.rados.0.smithi107.stdout:9559: finishing write tid 3 to smithi10729407-18 2020-06-12T02:39:05.596 INFO:tasks.rados.rados.0.smithi107.stdout:9559: finishing write tid 5 to smithi10729407-18 2020-06-12T02:39:05.596 INFO:tasks.rados.rados.0.smithi107.stdout:9559: finishing write tid 6 to smithi10729407-18 2020-06-12T02:39:05.596 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1755 (ObjNum 9558 snap 0 seq_num 9558) dirty exists 2020-06-12T02:39:05.596 INFO:tasks.rados.rados.0.smithi107.stdout:9559: left oid 18 (ObjNum 9558 snap 0 seq_num 9558) 2020-06-12T02:39:05.597 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:39:05.539+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:39:05.597 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd unset noscrub 2020-06-12T02:39:05.601 INFO:tasks.rados.rados.0.smithi107.stdout:9561: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:05.601 INFO:tasks.rados.rados.0.smithi107.stdout:9558: done (3 left) 2020-06-12T02:39:05.601 INFO:tasks.rados.rados.0.smithi107.stdout:9559: done (2 left) 2020-06-12T02:39:05.602 INFO:tasks.rados.rados.0.smithi107.stdout:9562: done (1 left) 2020-06-12T02:39:05.602 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:39:05.602 INFO:tasks.rados.rados.0.smithi107.stdout:9563: seq_num 9562 ranges {0=548864,548864=434176,983040=712704,1695744=57344} 2020-06-12T02:39:05.602 INFO:tasks.rados.rados.0.smithi107.stdout:9563: writing smithi10729407-3 from 0 to 548864 tid 1 2020-06-12T02:39:05.602 INFO:tasks.rados.rados.0.smithi107.stdout:9563: writing smithi10729407-3 from 548864 to 983040 tid 2 2020-06-12T02:39:05.603 INFO:tasks.rados.rados.0.smithi107.stdout:9563: writing smithi10729407-3 from 983040 to 1695744 tid 3 2020-06-12T02:39:05.603 INFO:tasks.rados.rados.0.smithi107.stdout:9563: writing smithi10729407-3 from 1695744 to 1753088 tid 4 2020-06-12T02:39:05.603 INFO:tasks.rados.rados.0.smithi107.stdout:9564: delete oid 18 current snap is 0 2020-06-12T02:39:05.603 INFO:tasks.rados.rados.0.smithi107.stdout:9561: finishing write tid 2 to smithi10729407-12 2020-06-12T02:39:05.603 INFO:tasks.rados.rados.0.smithi107.stdout:9561: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:05.603 INFO:tasks.rados.rados.0.smithi107.stdout:9561: finishing write tid 5 to smithi10729407-12 2020-06-12T02:39:05.604 INFO:tasks.rados.rados.0.smithi107.stdout:9561: finishing write tid 6 to smithi10729407-12 2020-06-12T02:39:05.604 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1767 (ObjNum 9560 snap 0 seq_num 9560) dirty exists 2020-06-12T02:39:05.604 INFO:tasks.rados.rados.0.smithi107.stdout:9561: left oid 12 (ObjNum 9560 snap 0 seq_num 9560) 2020-06-12T02:39:05.604 INFO:tasks.rados.rados.0.smithi107.stdout:9561: done (2 left) 2020-06-12T02:39:05.604 INFO:tasks.rados.rados.0.smithi107.stdout:9564: done (1 left) 2020-06-12T02:39:05.605 INFO:tasks.rados.rados.0.smithi107.stdout:9565: delete oid 9 current snap is 0 2020-06-12T02:39:05.605 INFO:tasks.rados.rados.0.smithi107.stdout:9563: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:05.605 INFO:tasks.rados.rados.0.smithi107.stdout:9563: finishing write tid 2 to smithi10729407-3 2020-06-12T02:39:05.605 INFO:tasks.rados.rados.0.smithi107.stdout:9563: finishing write tid 3 to smithi10729407-3 2020-06-12T02:39:05.605 INFO:tasks.rados.rados.0.smithi107.stdout:9565: done (1 left) 2020-06-12T02:39:05.605 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:39:05.606 INFO:tasks.rados.rados.0.smithi107.stdout:9566: seq_num 9565 ranges {0=598016,598016=540672,1138688=491520,1630208=172032} 2020-06-12T02:39:05.606 INFO:tasks.rados.rados.0.smithi107.stdout:9566: writing smithi10729407-6 from 0 to 598016 tid 1 2020-06-12T02:39:05.606 INFO:tasks.rados.rados.0.smithi107.stdout:9566: writing smithi10729407-6 from 598016 to 1138688 tid 2 2020-06-12T02:39:05.606 INFO:tasks.rados.rados.0.smithi107.stdout:9566: writing smithi10729407-6 from 1138688 to 1630208 tid 3 2020-06-12T02:39:05.606 INFO:tasks.rados.rados.0.smithi107.stdout:9566: writing smithi10729407-6 from 1630208 to 1802240 tid 4 2020-06-12T02:39:05.607 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:39:05.607 INFO:tasks.rados.rados.0.smithi107.stdout:9567: seq_num 9566 ranges {0=614400,614400=417792,1032192=696320,1728512=90112} 2020-06-12T02:39:05.607 INFO:tasks.rados.rados.0.smithi107.stdout:9567: writing smithi10729407-15 from 0 to 614400 tid 1 2020-06-12T02:39:05.607 INFO:tasks.rados.rados.0.smithi107.stdout:9567: writing smithi10729407-15 from 614400 to 1032192 tid 2 2020-06-12T02:39:05.607 INFO:tasks.rados.rados.0.smithi107.stdout:9567: writing smithi10729407-15 from 1032192 to 1728512 tid 3 2020-06-12T02:39:05.607 INFO:tasks.rados.rados.0.smithi107.stdout:9567: writing smithi10729407-15 from 1728512 to 1818624 tid 4 2020-06-12T02:39:05.608 INFO:tasks.rados.rados.0.smithi107.stdout:9568: delete oid 18 current snap is 0 2020-06-12T02:39:05.608 INFO:tasks.rados.rados.0.smithi107.stdout:9563: finishing write tid 4 to smithi10729407-3 2020-06-12T02:39:05.608 INFO:tasks.rados.rados.0.smithi107.stdout:9563: finishing write tid 6 to smithi10729407-3 2020-06-12T02:39:05.608 INFO:tasks.rados.rados.0.smithi107.stdout:9563: finishing write tid 7 to smithi10729407-3 2020-06-12T02:39:05.608 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1710 (ObjNum 9562 snap 0 seq_num 9562) dirty exists 2020-06-12T02:39:05.609 INFO:tasks.rados.rados.0.smithi107.stdout:9563: left oid 3 (ObjNum 9562 snap 0 seq_num 9562) 2020-06-12T02:39:05.609 INFO:tasks.rados.rados.0.smithi107.stdout:9566: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:05.609 INFO:tasks.rados.rados.0.smithi107.stdout:9566: finishing write tid 2 to smithi10729407-6 2020-06-12T02:39:05.609 INFO:tasks.rados.rados.0.smithi107.stdout:9566: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:05.609 INFO:tasks.rados.rados.0.smithi107.stdout:9566: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:05.609 INFO:tasks.rados.rados.0.smithi107.stdout:9566: finishing write tid 6 to smithi10729407-6 2020-06-12T02:39:05.610 INFO:tasks.rados.rados.0.smithi107.stdout:9567: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:05.610 INFO:tasks.rados.rados.0.smithi107.stdout:9566: finishing write tid 7 to smithi10729407-6 2020-06-12T02:39:05.610 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1726 (ObjNum 9565 snap 0 seq_num 9565) dirty exists 2020-06-12T02:39:05.610 INFO:tasks.rados.rados.0.smithi107.stdout:9566: left oid 6 (ObjNum 9565 snap 0 seq_num 9565) 2020-06-12T02:39:05.610 INFO:tasks.rados.rados.0.smithi107.stdout:9563: done (3 left) 2020-06-12T02:39:05.611 INFO:tasks.rados.rados.0.smithi107.stdout:9566: done (2 left) 2020-06-12T02:39:05.611 INFO:tasks.rados.rados.0.smithi107.stdout:9568: done (1 left) 2020-06-12T02:39:05.611 INFO:tasks.rados.rados.0.smithi107.stdout:9569: delete oid 13 current snap is 0 2020-06-12T02:39:05.611 INFO:tasks.rados.rados.0.smithi107.stdout:9569: done (1 left) 2020-06-12T02:39:05.611 INFO:tasks.rados.rados.0.smithi107.stdout:9570: delete oid 6 current snap is 0 2020-06-12T02:39:05.613 INFO:tasks.rados.rados.0.smithi107.stdout:9570: done (1 left) 2020-06-12T02:39:05.613 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:39:05.613 INFO:tasks.rados.rados.0.smithi107.stdout:9571: seq_num 9570 ranges {0=688128,688128=761856,1449984=442368} 2020-06-12T02:39:05.613 INFO:tasks.rados.rados.0.smithi107.stdout:9571: writing smithi10729407-13 from 0 to 688128 tid 1 2020-06-12T02:39:05.614 INFO:tasks.rados.rados.0.smithi107.stdout:9571: writing smithi10729407-13 from 688128 to 1449984 tid 2 2020-06-12T02:39:05.616 INFO:tasks.rados.rados.0.smithi107.stdout:9571: writing smithi10729407-13 from 1449984 to 1892352 tid 3 2020-06-12T02:39:05.616 INFO:tasks.rados.rados.0.smithi107.stdout:9572: delete oid 9 current snap is 0 2020-06-12T02:39:05.616 INFO:tasks.rados.rados.0.smithi107.stdout:9567: finishing write tid 2 to smithi10729407-15 2020-06-12T02:39:05.617 INFO:tasks.rados.rados.0.smithi107.stdout:9567: finishing write tid 3 to smithi10729407-15 2020-06-12T02:39:05.617 INFO:tasks.rados.rados.0.smithi107.stdout:9567: finishing write tid 4 to smithi10729407-15 2020-06-12T02:39:05.617 INFO:tasks.rados.rados.0.smithi107.stdout:9567: finishing write tid 6 to smithi10729407-15 2020-06-12T02:39:05.619 INFO:tasks.rados.rados.0.smithi107.stdout:9567: finishing write tid 7 to smithi10729407-15 2020-06-12T02:39:05.619 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1783 (ObjNum 9566 snap 0 seq_num 9566) dirty exists 2020-06-12T02:39:05.619 INFO:tasks.rados.rados.0.smithi107.stdout:9567: left oid 15 (ObjNum 9566 snap 0 seq_num 9566) 2020-06-12T02:39:05.620 INFO:tasks.rados.rados.0.smithi107.stdout:9567: done (2 left) 2020-06-12T02:39:05.620 INFO:tasks.rados.rados.0.smithi107.stdout:9572: done (1 left) 2020-06-12T02:39:05.620 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:39:05.620 INFO:tasks.rados.rados.0.smithi107.stdout:9573: seq_num 9572 ranges {0=720896,720896=581632,1302528=622592} 2020-06-12T02:39:05.624 INFO:tasks.rados.rados.0.smithi107.stdout:9573: writing smithi10729407-2 from 0 to 720896 tid 1 2020-06-12T02:39:05.628 INFO:tasks.rados.rados.0.smithi107.stdout:9573: writing smithi10729407-2 from 720896 to 1302528 tid 2 2020-06-12T02:39:05.632 INFO:tasks.rados.rados.0.smithi107.stdout:9573: writing smithi10729407-2 from 1302528 to 1925120 tid 3 2020-06-12T02:39:05.632 INFO:tasks.rados.rados.0.smithi107.stdout:9574: delete oid 14 current snap is 0 2020-06-12T02:39:05.632 INFO:tasks.rados.rados.0.smithi107.stdout:9571: finishing write tid 1 to smithi10729407-13 2020-06-12T02:39:05.633 INFO:tasks.rados.rados.0.smithi107.stdout:9571: finishing write tid 2 to smithi10729407-13 2020-06-12T02:39:05.633 INFO:tasks.rados.rados.0.smithi107.stdout:9571: finishing write tid 3 to smithi10729407-13 2020-06-12T02:39:05.633 INFO:tasks.rados.rados.0.smithi107.stdout:9571: finishing write tid 5 to smithi10729407-13 2020-06-12T02:39:05.633 INFO:tasks.rados.rados.0.smithi107.stdout:9571: finishing write tid 6 to smithi10729407-13 2020-06-12T02:39:05.634 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1685 (ObjNum 9570 snap 0 seq_num 9570) dirty exists 2020-06-12T02:39:05.634 INFO:tasks.rados.rados.0.smithi107.stdout:9571: left oid 13 (ObjNum 9570 snap 0 seq_num 9570) 2020-06-12T02:39:05.634 INFO:tasks.rados.rados.0.smithi107.stdout:9573: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:05.635 INFO:tasks.rados.rados.0.smithi107.stdout:9571: done (2 left) 2020-06-12T02:39:05.635 INFO:tasks.rados.rados.0.smithi107.stdout:9574: done (1 left) 2020-06-12T02:39:05.635 INFO:tasks.rados.rados.0.smithi107.stdout:9575: delete oid 15 current snap is 0 2020-06-12T02:39:05.639 INFO:tasks.rados.rados.0.smithi107.stdout:9575: done (1 left) 2020-06-12T02:39:05.640 INFO:tasks.rados.rados.0.smithi107.stdout:9576: delete oid 5 current snap is 0 2020-06-12T02:39:05.641 INFO:tasks.rados.rados.0.smithi107.stdout:9576: done (1 left) 2020-06-12T02:39:05.641 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:39:05.642 INFO:tasks.rados.rados.0.smithi107.stdout:9577: seq_num 9576 ranges {0=786432,786432=516096,1302528=622592,1925120=65536} 2020-06-12T02:39:05.646 INFO:tasks.rados.rados.0.smithi107.stdout:9577: writing smithi10729407-9 from 0 to 786432 tid 1 2020-06-12T02:39:05.649 INFO:tasks.rados.rados.0.smithi107.stdout:9577: writing smithi10729407-9 from 786432 to 1302528 tid 2 2020-06-12T02:39:05.653 INFO:tasks.rados.rados.0.smithi107.stdout:9577: writing smithi10729407-9 from 1302528 to 1925120 tid 3 2020-06-12T02:39:05.654 INFO:tasks.rados.rados.0.smithi107.stdout:9577: writing smithi10729407-9 from 1925120 to 1990656 tid 4 2020-06-12T02:39:05.654 INFO:tasks.rados.rados.0.smithi107.stdout:9578: delete oid 1 current snap is 0 2020-06-12T02:39:05.654 INFO:tasks.rados.rados.0.smithi107.stdout:9573: finishing write tid 2 to smithi10729407-2 2020-06-12T02:39:05.654 INFO:tasks.rados.rados.0.smithi107.stdout:9573: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:05.655 INFO:tasks.rados.rados.0.smithi107.stdout:9573: finishing write tid 5 to smithi10729407-2 2020-06-12T02:39:05.655 INFO:tasks.rados.rados.0.smithi107.stdout:9573: finishing write tid 6 to smithi10729407-2 2020-06-12T02:39:05.655 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 888 (ObjNum 9572 snap 0 seq_num 9572) dirty exists 2020-06-12T02:39:05.655 INFO:tasks.rados.rados.0.smithi107.stdout:9573: left oid 2 (ObjNum 9572 snap 0 seq_num 9572) 2020-06-12T02:39:05.656 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:39:05.653+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:39:05.656 INFO:tasks.rados.rados.0.smithi107.stdout:9573: done (2 left) 2020-06-12T02:39:05.656 INFO:tasks.rados.rados.0.smithi107.stdout:9578: done (1 left) 2020-06-12T02:39:05.657 INFO:tasks.rados.rados.0.smithi107.stdout:9579: delete oid 5 current snap is 0 2020-06-12T02:39:05.659 INFO:tasks.rados.rados.0.smithi107.stdout:9579: done (1 left) 2020-06-12T02:39:05.659 INFO:tasks.rados.rados.0.smithi107.stdout:9580: delete oid 18 current snap is 0 2020-06-12T02:39:05.662 INFO:tasks.rados.rados.0.smithi107.stdout:9577: finishing write tid 1 to smithi10729407-9 2020-06-12T02:39:05.664 INFO:tasks.rados.rados.0.smithi107.stdout:9577: finishing write tid 2 to smithi10729407-9 2020-06-12T02:39:05.665 INFO:tasks.rados.rados.0.smithi107.stdout:9577: finishing write tid 3 to smithi10729407-9 2020-06-12T02:39:05.666 INFO:tasks.rados.rados.0.smithi107.stdout:9577: finishing write tid 4 to smithi10729407-9 2020-06-12T02:39:05.668 INFO:tasks.rados.rados.0.smithi107.stdout:9580: done (1 left) 2020-06-12T02:39:05.668 INFO:tasks.rados.rados.0.smithi107.stdout:9581: delete oid 19 current snap is 0 2020-06-12T02:39:05.669 INFO:tasks.rados.rados.0.smithi107.stdout:9577: finishing write tid 6 to smithi10729407-9 2020-06-12T02:39:05.672 INFO:tasks.rados.rados.0.smithi107.stdout:9577: finishing write tid 7 to smithi10729407-9 2020-06-12T02:39:05.672 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1749 (ObjNum 9576 snap 0 seq_num 9576) dirty exists 2020-06-12T02:39:05.672 INFO:tasks.rados.rados.0.smithi107.stdout:9577: left oid 9 (ObjNum 9576 snap 0 seq_num 9576) 2020-06-12T02:39:05.673 INFO:tasks.rados.rados.0.smithi107.stdout:9577: done (1 left) 2020-06-12T02:39:05.673 INFO:tasks.rados.rados.0.smithi107.stdout:9581: done (0 left) 2020-06-12T02:39:05.674 INFO:tasks.rados.rados.0.smithi107.stdout:9582: delete oid 13 current snap is 0 2020-06-12T02:39:05.675 INFO:tasks.rados.rados.0.smithi107.stdout:9582: done (0 left) 2020-06-12T02:39:05.676 INFO:tasks.rados.rados.0.smithi107.stdout:9583: delete oid 16 current snap is 0 2020-06-12T02:39:05.679 INFO:tasks.rados.rados.0.smithi107.stdout:9583: done (0 left) 2020-06-12T02:39:05.679 INFO:tasks.rados.rados.0.smithi107.stdout:9584: delete oid 10 current snap is 0 2020-06-12T02:39:05.681 INFO:tasks.rados.rados.0.smithi107.stdout:9584: done (0 left) 2020-06-12T02:39:05.681 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:39:05.682 INFO:tasks.rados.rados.0.smithi107.stdout:9585: seq_num 9584 ranges {1925120=516096,2441216=442368,2883584=581632,3465216=499712,3964928=81920} 2020-06-12T02:39:05.696 INFO:tasks.rados.rados.0.smithi107.stdout:9585: writing smithi10729407-2 from 1925120 to 2441216 tid 1 2020-06-12T02:39:05.705 INFO:tasks.rados.rados.0.smithi107.stdout:9585: writing smithi10729407-2 from 2441216 to 2883584 tid 2 2020-06-12T02:39:05.706 INFO:tasks.rados.rados.0.smithi107.stdout:9585: writing smithi10729407-2 from 2883584 to 3465216 tid 3 2020-06-12T02:39:05.707 INFO:tasks.rados.rados.0.smithi107.stdout:9585: writing smithi10729407-2 from 3465216 to 3964928 tid 4 2020-06-12T02:39:05.708 INFO:tasks.rados.rados.0.smithi107.stdout:9585: writing smithi10729407-2 from 3964928 to 4046848 tid 5 2020-06-12T02:39:05.708 INFO:tasks.rados.rados.0.smithi107.stdout:9586: delete oid 10 current snap is 0 2020-06-12T02:39:05.709 INFO:tasks.rados.rados.0.smithi107.stdout:9585: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:05.709 INFO:tasks.rados.rados.0.smithi107.stdout:9585: finishing write tid 2 to smithi10729407-2 2020-06-12T02:39:05.710 INFO:tasks.rados.rados.0.smithi107.stdout:9586: done (1 left) 2020-06-12T02:39:05.711 INFO:tasks.rados.rados.0.smithi107.stdout:9587: delete oid 9 current snap is 0 2020-06-12T02:39:05.711 INFO:tasks.rados.rados.0.smithi107.stdout:9585: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:05.714 INFO:tasks.rados.rados.0.smithi107.stdout:9587: done (1 left) 2020-06-12T02:39:05.714 INFO:tasks.rados.rados.0.smithi107.stdout:9588: delete oid 12 current snap is 0 2020-06-12T02:39:05.715 INFO:tasks.rados.rados.0.smithi107.stdout:9585: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:05.715 INFO:tasks.rados.rados.0.smithi107.stdout:9585: finishing write tid 5 to smithi10729407-2 2020-06-12T02:39:05.716 INFO:tasks.rados.rados.0.smithi107.stdout:9585: finishing write tid 7 to smithi10729407-2 2020-06-12T02:39:05.718 INFO:tasks.rados.rados.0.smithi107.stdout:9588: done (1 left) 2020-06-12T02:39:05.718 INFO:tasks.rados.rados.0.smithi107.stdout:9589: delete oid 18 current snap is 0 2020-06-12T02:39:05.718 INFO:tasks.rados.rados.0.smithi107.stdout:9585: finishing write tid 8 to smithi10729407-2 2020-06-12T02:39:05.719 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 894 (ObjNum 9584 snap 0 seq_num 9584) dirty exists 2020-06-12T02:39:05.719 INFO:tasks.rados.rados.0.smithi107.stdout:9585: left oid 2 (ObjNum 9584 snap 0 seq_num 9584) 2020-06-12T02:39:05.720 INFO:tasks.rados.rados.0.smithi107.stdout:9585: done (1 left) 2020-06-12T02:39:05.721 INFO:tasks.rados.rados.0.smithi107.stdout:9589: done (0 left) 2020-06-12T02:39:05.721 INFO:tasks.rados.rados.0.smithi107.stdout:9590: delete oid 17 current snap is 0 2020-06-12T02:39:05.723 INFO:tasks.rados.rados.0.smithi107.stdout:9590: done (0 left) 2020-06-12T02:39:05.724 INFO:tasks.rados.rados.0.smithi107.stdout:9591: delete oid 1 current snap is 0 2020-06-12T02:39:05.726 INFO:tasks.rados.rados.0.smithi107.stdout:9591: done (0 left) 2020-06-12T02:39:05.726 INFO:tasks.rados.rados.0.smithi107.stdout:9592: delete oid 3 current snap is 0 2020-06-12T02:39:05.728 INFO:tasks.rados.rados.0.smithi107.stdout:9592: done (0 left) 2020-06-12T02:39:05.729 INFO:tasks.rados.rados.0.smithi107.stdout:9593: delete oid 13 current snap is 0 2020-06-12T02:39:05.733 INFO:tasks.rados.rados.0.smithi107.stdout:9593: done (0 left) 2020-06-12T02:39:05.734 INFO:tasks.rados.rados.0.smithi107.stdout:9594: delete oid 18 current snap is 0 2020-06-12T02:39:05.736 INFO:tasks.rados.rados.0.smithi107.stdout:9594: done (0 left) 2020-06-12T02:39:05.736 INFO:tasks.rados.rados.0.smithi107.stdout:9595: delete oid 11 current snap is 0 2020-06-12T02:39:05.738 INFO:tasks.rados.rados.0.smithi107.stdout:9595: done (0 left) 2020-06-12T02:39:05.738 INFO:tasks.rados.rados.0.smithi107.stdout:9596: delete oid 12 current snap is 0 2020-06-12T02:39:05.742 INFO:tasks.rados.rados.0.smithi107.stdout:9596: done (0 left) 2020-06-12T02:39:05.743 INFO:tasks.rados.rados.0.smithi107.stdout:9597: delete oid 18 current snap is 0 2020-06-12T02:39:05.743 INFO:tasks.rados.rados.0.smithi107.stdout:9597: done (0 left) 2020-06-12T02:39:05.743 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:39:05.744 INFO:tasks.rados.rados.0.smithi107.stdout:9598: seq_num 9597 ranges {0=737280,737280=638976,1376256=729088,2105344=237568} 2020-06-12T02:39:05.744 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:39:05.744 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:39:05.745 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2020-06-12T02:39:05.745 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:39:05.747 INFO:tasks.rados.rados.0.smithi107.stdout:9598: writing smithi10729407-13 from 0 to 737280 tid 1 2020-06-12T02:39:05.747 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:39:05.745+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:39:05.751 INFO:tasks.rados.rados.0.smithi107.stdout:9598: writing smithi10729407-13 from 737280 to 1376256 tid 2 2020-06-12T02:39:05.753 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-06-12T02:39:05.756 INFO:tasks.rados.rados.0.smithi107.stdout:9598: writing smithi10729407-13 from 1376256 to 2105344 tid 3 2020-06-12T02:39:05.758 INFO:tasks.rados.rados.0.smithi107.stdout:9598: writing smithi10729407-13 from 2105344 to 2342912 tid 4 2020-06-12T02:39:05.758 INFO:tasks.rados.rados.0.smithi107.stdout:9599: delete oid 12 current snap is 0 2020-06-12T02:39:05.759 INFO:tasks.rados.rados.0.smithi107.stdout:9598: finishing write tid 1 to smithi10729407-13 2020-06-12T02:39:05.761 INFO:tasks.rados.rados.0.smithi107.stdout:9599: done (1 left) 2020-06-12T02:39:05.762 INFO:tasks.rados.rados.0.smithi107.stdout:9600: delete oid 20 current snap is 0 2020-06-12T02:39:05.766 INFO:tasks.rados.rados.0.smithi107.stdout:9600: done (1 left) 2020-06-12T02:39:05.766 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:39:05.766 INFO:tasks.rados.rados.0.smithi107.stdout:9601: seq_num 9600 ranges {0=786432,786432=688128,1474560=507904,1982464=409600} 2020-06-12T02:39:05.770 INFO:tasks.rados.rados.0.smithi107.stdout:9601: writing smithi10729407-20 from 0 to 786432 tid 1 2020-06-12T02:39:05.775 INFO:tasks.rados.rados.0.smithi107.stdout:9601: writing smithi10729407-20 from 786432 to 1474560 tid 2 2020-06-12T02:39:05.778 INFO:tasks.rados.rados.0.smithi107.stdout:9601: writing smithi10729407-20 from 1474560 to 1982464 tid 3 2020-06-12T02:39:05.784 INFO:tasks.rados.rados.0.smithi107.stdout:9601: writing smithi10729407-20 from 1982464 to 2392064 tid 4 2020-06-12T02:39:05.784 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:39:05.784 INFO:tasks.rados.rados.0.smithi107.stdout:9602: seq_num 9601 ranges {0=802816,802816=573440,1376256=712704,2088960=319488} 2020-06-12T02:39:05.786 INFO:tasks.rados.rados.0.smithi107.stdout:9602: writing smithi10729407-14 from 0 to 802816 tid 1 2020-06-12T02:39:05.790 INFO:tasks.rados.rados.0.smithi107.stdout:9602: writing smithi10729407-14 from 802816 to 1376256 tid 2 2020-06-12T02:39:05.794 INFO:tasks.rados.rados.0.smithi107.stdout:9602: writing smithi10729407-14 from 1376256 to 2088960 tid 3 2020-06-12T02:39:05.797 INFO:tasks.rados.rados.0.smithi107.stdout:9602: writing smithi10729407-14 from 2088960 to 2408448 tid 4 2020-06-12T02:39:05.797 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:39:05.798 INFO:tasks.rados.rados.0.smithi107.stdout:9603: seq_num 9602 ranges {0=16384} 2020-06-12T02:39:05.798 INFO:tasks.rados.rados.0.smithi107.stdout:9603: writing smithi10729407-10 from 0 to 16384 tid 1 2020-06-12T02:39:05.799 INFO:tasks.rados.rados.0.smithi107.stdout:9604: delete oid 2 current snap is 0 2020-06-12T02:39:05.799 INFO:tasks.rados.rados.0.smithi107.stdout:9598: finishing write tid 2 to smithi10729407-13 2020-06-12T02:39:05.799 INFO:tasks.rados.rados.0.smithi107.stdout:9598: finishing write tid 3 to smithi10729407-13 2020-06-12T02:39:05.800 INFO:tasks.rados.rados.0.smithi107.stdout:9598: finishing write tid 4 to smithi10729407-13 2020-06-12T02:39:05.800 INFO:tasks.rados.rados.0.smithi107.stdout:9598: finishing write tid 6 to smithi10729407-13 2020-06-12T02:39:05.801 INFO:tasks.rados.rados.0.smithi107.stdout:9598: finishing write tid 7 to smithi10729407-13 2020-06-12T02:39:05.801 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1692 (ObjNum 9597 snap 0 seq_num 9597) dirty exists 2020-06-12T02:39:05.812 INFO:tasks.rados.rados.0.smithi107.stdout:9598: left oid 13 (ObjNum 9597 snap 0 seq_num 9597) 2020-06-12T02:39:05.813 INFO:tasks.rados.rados.0.smithi107.stdout:9601: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:05.813 INFO:tasks.rados.rados.0.smithi107.stdout:9601: finishing write tid 2 to smithi10729407-20 2020-06-12T02:39:05.822 INFO:tasks.rados.rados.0.smithi107.stdout:9601: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:05.822 INFO:tasks.rados.rados.0.smithi107.stdout:9601: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:05.822 INFO:tasks.rados.rados.0.smithi107.stdout:9601: finishing write tid 6 to smithi10729407-20 2020-06-12T02:39:05.823 INFO:tasks.rados.rados.0.smithi107.stdout:9601: finishing write tid 7 to smithi10729407-20 2020-06-12T02:39:05.823 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 711 (ObjNum 9600 snap 0 seq_num 9600) dirty exists 2020-06-12T02:39:05.823 INFO:tasks.rados.rados.0.smithi107.stdout:9601: left oid 20 (ObjNum 9600 snap 0 seq_num 9600) 2020-06-12T02:39:05.823 INFO:tasks.rados.rados.0.smithi107.stdout:9602: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:05.841 INFO:tasks.rados.rados.0.smithi107.stdout:9598: done (4 left) 2020-06-12T02:39:05.841 INFO:tasks.rados.rados.0.smithi107.stdout:9601: done (3 left) 2020-06-12T02:39:05.841 INFO:tasks.rados.rados.0.smithi107.stdout:9604: done (2 left) 2020-06-12T02:39:05.841 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:39:05.842 INFO:tasks.rados.rados.0.smithi107.stdout:9605: seq_num 9604 ranges {0=49152} 2020-06-12T02:39:05.842 INFO:tasks.rados.rados.0.smithi107.stdout:9605: writing smithi10729407-17 from 0 to 49152 tid 1 2020-06-12T02:39:05.842 INFO:tasks.rados.rados.0.smithi107.stdout:9606: delete oid 19 current snap is 0 2020-06-12T02:39:05.842 INFO:tasks.rados.rados.0.smithi107.stdout:9603: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:05.842 INFO:tasks.rados.rados.0.smithi107.stdout:9603: finishing write tid 3 to smithi10729407-10 2020-06-12T02:39:05.843 INFO:tasks.rados.rados.0.smithi107.stdout:9603: finishing write tid 4 to smithi10729407-10 2020-06-12T02:39:05.843 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 684 (ObjNum 9602 snap 0 seq_num 9602) dirty exists 2020-06-12T02:39:05.843 INFO:tasks.rados.rados.0.smithi107.stdout:9603: left oid 10 (ObjNum 9602 snap 0 seq_num 9602) 2020-06-12T02:39:05.843 INFO:tasks.rados.rados.0.smithi107.stdout:9602: finishing write tid 2 to smithi10729407-14 2020-06-12T02:39:05.843 INFO:tasks.rados.rados.0.smithi107.stdout:9605: finishing write tid 1 to smithi10729407-17 2020-06-12T02:39:05.844 INFO:tasks.rados.rados.0.smithi107.stdout:9605: finishing write tid 3 to smithi10729407-17 2020-06-12T02:39:05.844 INFO:tasks.rados.rados.0.smithi107.stdout:9603: done (3 left) 2020-06-12T02:39:05.844 INFO:tasks.rados.rados.0.smithi107.stdout:9606: done (2 left) 2020-06-12T02:39:05.844 INFO:tasks.rados.rados.0.smithi107.stdout:9607: delete oid 9 current snap is 0 2020-06-12T02:39:05.845 INFO:tasks.rados.rados.0.smithi107.stdout:9607: done (2 left) 2020-06-12T02:39:05.845 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:39:05.845 INFO:tasks.rados.rados.0.smithi107.stdout:9608: seq_num 9607 ranges {2392064=106496} 2020-06-12T02:39:05.845 INFO:tasks.rados.rados.0.smithi107.stdout:9608: writing smithi10729407-20 from 2392064 to 2498560 tid 1 2020-06-12T02:39:05.846 INFO:tasks.rados.rados.0.smithi107.stdout:9609: delete oid 9 current snap is 0 2020-06-12T02:39:05.846 INFO:tasks.rados.rados.0.smithi107.stdout:9602: finishing write tid 3 to smithi10729407-14 2020-06-12T02:39:05.846 INFO:tasks.rados.rados.0.smithi107.stdout:9602: finishing write tid 4 to smithi10729407-14 2020-06-12T02:39:05.846 INFO:tasks.rados.rados.0.smithi107.stdout:9602: finishing write tid 6 to smithi10729407-14 2020-06-12T02:39:05.846 INFO:tasks.rados.rados.0.smithi107.stdout:9605: finishing write tid 4 to smithi10729407-17 2020-06-12T02:39:05.846 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1800 (ObjNum 9604 snap 0 seq_num 9604) dirty exists 2020-06-12T02:39:05.847 INFO:tasks.rados.rados.0.smithi107.stdout:9605: left oid 17 (ObjNum 9604 snap 0 seq_num 9604) 2020-06-12T02:39:05.847 INFO:tasks.rados.rados.0.smithi107.stdout:9602: finishing write tid 7 to smithi10729407-14 2020-06-12T02:39:05.847 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1618 (ObjNum 9601 snap 0 seq_num 9601) dirty exists 2020-06-12T02:39:05.847 INFO:tasks.rados.rados.0.smithi107.stdout:9602: left oid 14 (ObjNum 9601 snap 0 seq_num 9601) 2020-06-12T02:39:05.847 INFO:tasks.rados.rados.0.smithi107.stdout:9602: done (3 left) 2020-06-12T02:39:05.848 INFO:tasks.rados.rados.0.smithi107.stdout:9605: done (2 left) 2020-06-12T02:39:05.848 INFO:tasks.rados.rados.0.smithi107.stdout:9609: done (1 left) 2020-06-12T02:39:05.848 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:39:05.848 INFO:tasks.rados.rados.0.smithi107.stdout:9610: seq_num 9609 ranges {0=139264} 2020-06-12T02:39:05.848 INFO:tasks.rados.rados.0.smithi107.stdout:9610: writing smithi10729407-16 from 0 to 139264 tid 1 2020-06-12T02:39:05.849 INFO:tasks.rados.rados.0.smithi107.stdout:9611: delete oid 4 current snap is 0 2020-06-12T02:39:05.849 INFO:tasks.rados.rados.0.smithi107.stdout:9608: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:05.849 INFO:tasks.rados.rados.0.smithi107.stdout:9608: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:05.849 INFO:tasks.rados.rados.0.smithi107.stdout:9608: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:05.849 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 713 (ObjNum 9607 snap 0 seq_num 9607) dirty exists 2020-06-12T02:39:05.849 INFO:tasks.rados.rados.0.smithi107.stdout:9608: left oid 20 (ObjNum 9607 snap 0 seq_num 9607) 2020-06-12T02:39:05.850 INFO:tasks.rados.rados.0.smithi107.stdout:9610: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:05.850 INFO:tasks.rados.rados.0.smithi107.stdout:9608: done (2 left) 2020-06-12T02:39:05.850 INFO:tasks.rados.rados.0.smithi107.stdout:9611: done (1 left) 2020-06-12T02:39:05.850 INFO:tasks.rados.rados.0.smithi107.stdout:9612: delete oid 12 current snap is 0 2020-06-12T02:39:05.850 INFO:tasks.rados.rados.0.smithi107.stdout:9610: finishing write tid 3 to smithi10729407-16 2020-06-12T02:39:05.851 INFO:tasks.rados.rados.0.smithi107.stdout:9612: done (1 left) 2020-06-12T02:39:05.851 INFO:tasks.rados.rados.0.smithi107.stdout:9613: delete oid 20 current snap is 0 2020-06-12T02:39:05.851 INFO:tasks.rados.rados.0.smithi107.stdout:9610: finishing write tid 4 to smithi10729407-16 2020-06-12T02:39:05.851 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1714 (ObjNum 9609 snap 0 seq_num 9609) dirty exists 2020-06-12T02:39:05.851 INFO:tasks.rados.rados.0.smithi107.stdout:9610: left oid 16 (ObjNum 9609 snap 0 seq_num 9609) 2020-06-12T02:39:05.852 INFO:tasks.rados.rados.0.smithi107.stdout:9610: done (1 left) 2020-06-12T02:39:05.852 INFO:tasks.rados.rados.0.smithi107.stdout:9613: done (0 left) 2020-06-12T02:39:05.852 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:39:05.852 INFO:tasks.rados.rados.0.smithi107.stdout:9614: seq_num 9613 ranges {0=204800} 2020-06-12T02:39:05.852 INFO:tasks.rados.rados.0.smithi107.stdout:9614: writing smithi10729407-20 from 0 to 204800 tid 1 2020-06-12T02:39:05.853 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:39:05.853 INFO:tasks.rados.rados.0.smithi107.stdout:9615: seq_num 9614 ranges {0=221184} 2020-06-12T02:39:05.853 INFO:tasks.rados.rados.0.smithi107.stdout:9615: writing smithi10729407-19 from 0 to 221184 tid 1 2020-06-12T02:39:05.853 INFO:tasks.rados.rados.0.smithi107.stdout:9616: delete oid 3 current snap is 0 2020-06-12T02:39:05.855 INFO:tasks.rados.rados.0.smithi107.stdout:9616: done (2 left) 2020-06-12T02:39:05.855 INFO:tasks.rados.rados.0.smithi107.stdout:9617: delete oid 17 current snap is 0 2020-06-12T02:39:05.855 INFO:tasks.rados.rados.0.smithi107.stdout:9614: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:05.856 INFO:tasks.rados.rados.0.smithi107.stdout:9614: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:05.856 INFO:tasks.rados.rados.0.smithi107.stdout:9617: done (2 left) 2020-06-12T02:39:05.856 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:39:05.856 INFO:tasks.rados.rados.0.smithi107.stdout:9618: seq_num 9617 ranges {139264=270336} 2020-06-12T02:39:05.856 INFO:tasks.rados.rados.0.smithi107.stdout:9618: writing smithi10729407-16 from 139264 to 409600 tid 1 2020-06-12T02:39:05.857 INFO:tasks.rados.rados.0.smithi107.stdout:9619: delete oid 5 current snap is 0 2020-06-12T02:39:05.857 INFO:tasks.rados.rados.0.smithi107.stdout:9615: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:05.857 INFO:tasks.rados.rados.0.smithi107.stdout:9614: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:05.858 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 716 (ObjNum 9613 snap 0 seq_num 9613) dirty exists 2020-06-12T02:39:05.858 INFO:tasks.rados.rados.0.smithi107.stdout:9614: left oid 20 (ObjNum 9613 snap 0 seq_num 9613) 2020-06-12T02:39:05.860 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:39:05.852+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:39:05.860 INFO:tasks.rados.rados.0.smithi107.stdout:9615: finishing write tid 3 to smithi10729407-19 2020-06-12T02:39:05.860 INFO:tasks.rados.rados.0.smithi107.stdout:9614: done (3 left) 2020-06-12T02:39:05.860 INFO:tasks.rados.rados.0.smithi107.stdout:9619: done (2 left) 2020-06-12T02:39:05.861 INFO:tasks.rados.rados.0.smithi107.stdout:9620: delete oid 3 current snap is 0 2020-06-12T02:39:05.861 INFO:tasks.rados.rados.0.smithi107.stdout:9618: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:05.861 INFO:tasks.rados.rados.0.smithi107.stdout:9618: finishing write tid 3 to smithi10729407-16 2020-06-12T02:39:05.861 INFO:tasks.rados.rados.0.smithi107.stdout:9615: finishing write tid 4 to smithi10729407-19 2020-06-12T02:39:05.861 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1775 (ObjNum 9614 snap 0 seq_num 9614) dirty exists 2020-06-12T02:39:05.862 INFO:tasks.rados.rados.0.smithi107.stdout:9615: left oid 19 (ObjNum 9614 snap 0 seq_num 9614) 2020-06-12T02:39:05.862 INFO:tasks.rados.rados.0.smithi107.stdout:9618: finishing write tid 4 to smithi10729407-16 2020-06-12T02:39:05.862 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1717 (ObjNum 9617 snap 0 seq_num 9617) dirty exists 2020-06-12T02:39:05.862 INFO:tasks.rados.rados.0.smithi107.stdout:9618: left oid 16 (ObjNum 9617 snap 0 seq_num 9617) 2020-06-12T02:39:05.863 INFO:tasks.rados.rados.0.smithi107.stdout:9615: done (2 left) 2020-06-12T02:39:05.863 INFO:tasks.rados.rados.0.smithi107.stdout:9618: done (1 left) 2020-06-12T02:39:05.863 INFO:tasks.rados.rados.0.smithi107.stdout:9620: done (0 left) 2020-06-12T02:39:05.863 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:39:05.864 INFO:tasks.rados.rados.0.smithi107.stdout:9621: seq_num 9620 ranges {221184=319488} 2020-06-12T02:39:05.864 INFO:tasks.rados.rados.0.smithi107.stdout:9621: writing smithi10729407-19 from 221184 to 540672 tid 1 2020-06-12T02:39:05.865 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:39:05.865 INFO:tasks.rados.rados.0.smithi107.stdout:9622: seq_num 9621 ranges {204800=335872} 2020-06-12T02:39:05.867 INFO:tasks.rados.rados.0.smithi107.stdout:9622: writing smithi10729407-20 from 204800 to 540672 tid 1 2020-06-12T02:39:05.867 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:39:05.867 INFO:tasks.rados.rados.0.smithi107.stdout:9623: seq_num 9622 ranges {0=352256} 2020-06-12T02:39:05.869 INFO:tasks.rados.rados.0.smithi107.stdout:9623: writing smithi10729407-11 from 0 to 352256 tid 1 2020-06-12T02:39:05.870 INFO:tasks.rados.rados.0.smithi107.stdout:9624: delete oid 9 current snap is 0 2020-06-12T02:39:05.870 INFO:tasks.rados.rados.0.smithi107.stdout:9621: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:05.874 INFO:tasks.rados.rados.0.smithi107.stdout:9622: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:05.875 INFO:tasks.rados.rados.0.smithi107.stdout:9624: done (3 left) 2020-06-12T02:39:05.875 INFO:tasks.rados.rados.0.smithi107.stdout:9625: delete oid 4 current snap is 0 2020-06-12T02:39:05.875 INFO:tasks.rados.rados.0.smithi107.stdout:9621: finishing write tid 3 to smithi10729407-19 2020-06-12T02:39:05.876 INFO:tasks.rados.rados.0.smithi107.stdout:9622: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:05.878 INFO:tasks.rados.rados.0.smithi107.stdout:9623: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:05.878 INFO:tasks.rados.rados.0.smithi107.stdout:9623: finishing write tid 3 to smithi10729407-11 2020-06-12T02:39:05.879 INFO:tasks.rados.rados.0.smithi107.stdout:9622: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:05.879 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 718 (ObjNum 9621 snap 0 seq_num 9621) dirty exists 2020-06-12T02:39:05.879 INFO:tasks.rados.rados.0.smithi107.stdout:9622: left oid 20 (ObjNum 9621 snap 0 seq_num 9621) 2020-06-12T02:39:05.879 INFO:tasks.rados.rados.0.smithi107.stdout:9621: finishing write tid 4 to smithi10729407-19 2020-06-12T02:39:05.879 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1777 (ObjNum 9620 snap 0 seq_num 9620) dirty exists 2020-06-12T02:39:05.880 INFO:tasks.rados.rados.0.smithi107.stdout:9621: left oid 19 (ObjNum 9620 snap 0 seq_num 9620) 2020-06-12T02:39:05.880 INFO:tasks.rados.rados.0.smithi107.stdout:9621: done (3 left) 2020-06-12T02:39:05.880 INFO:tasks.rados.rados.0.smithi107.stdout:9622: done (2 left) 2020-06-12T02:39:05.880 INFO:tasks.rados.rados.0.smithi107.stdout:9625: done (1 left) 2020-06-12T02:39:05.880 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:39:05.881 INFO:tasks.rados.rados.0.smithi107.stdout:9626: seq_num 9625 ranges {0=401408} 2020-06-12T02:39:05.881 INFO:tasks.rados.rados.0.smithi107.stdout:9626: writing smithi10729407-9 from 0 to 401408 tid 1 2020-06-12T02:39:05.882 INFO:tasks.rados.rados.0.smithi107.stdout:9627: delete oid 12 current snap is 0 2020-06-12T02:39:05.882 INFO:tasks.rados.rados.0.smithi107.stdout:9623: finishing write tid 4 to smithi10729407-11 2020-06-12T02:39:05.882 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1636 (ObjNum 9622 snap 0 seq_num 9622) dirty exists 2020-06-12T02:39:05.882 INFO:tasks.rados.rados.0.smithi107.stdout:9623: left oid 11 (ObjNum 9622 snap 0 seq_num 9622) 2020-06-12T02:39:05.884 INFO:tasks.rados.rados.0.smithi107.stdout:9623: done (2 left) 2020-06-12T02:39:05.884 INFO:tasks.rados.rados.0.smithi107.stdout:9627: done (1 left) 2020-06-12T02:39:05.884 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:39:05.885 INFO:tasks.rados.rados.0.smithi107.stdout:9628: seq_num 9627 ranges {0=442368} 2020-06-12T02:39:05.887 INFO:tasks.rados.rados.0.smithi107.stdout:9628: writing smithi10729407-3 from 0 to 442368 tid 1 2020-06-12T02:39:05.887 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:39:05.887 INFO:tasks.rados.rados.0.smithi107.stdout:9629: seq_num 9628 ranges {0=458752} 2020-06-12T02:39:05.890 INFO:tasks.rados.rados.0.smithi107.stdout:9629: writing smithi10729407-8 from 0 to 458752 tid 1 2020-06-12T02:39:05.890 INFO:tasks.rados.rados.0.smithi107.stdout:9630: delete oid 14 current snap is 0 2020-06-12T02:39:05.891 INFO:tasks.rados.rados.0.smithi107.stdout:9626: finishing write tid 1 to smithi10729407-9 2020-06-12T02:39:05.891 INFO:tasks.rados.rados.0.smithi107.stdout:9626: finishing write tid 3 to smithi10729407-9 2020-06-12T02:39:05.891 INFO:tasks.rados.rados.0.smithi107.stdout:9626: finishing write tid 4 to smithi10729407-9 2020-06-12T02:39:05.892 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1755 (ObjNum 9625 snap 0 seq_num 9625) dirty exists 2020-06-12T02:39:05.892 INFO:tasks.rados.rados.0.smithi107.stdout:9626: left oid 9 (ObjNum 9625 snap 0 seq_num 9625) 2020-06-12T02:39:05.895 INFO:tasks.rados.rados.0.smithi107.stdout:9628: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:05.896 INFO:tasks.rados.rados.0.smithi107.stdout:9628: finishing write tid 3 to smithi10729407-3 2020-06-12T02:39:05.896 INFO:tasks.rados.rados.0.smithi107.stdout:9628: finishing write tid 4 to smithi10729407-3 2020-06-12T02:39:05.897 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1720 (ObjNum 9627 snap 0 seq_num 9627) dirty exists 2020-06-12T02:39:05.897 INFO:tasks.rados.rados.0.smithi107.stdout:9628: left oid 3 (ObjNum 9627 snap 0 seq_num 9627) 2020-06-12T02:39:05.897 INFO:tasks.rados.rados.0.smithi107.stdout:9626: done (3 left) 2020-06-12T02:39:05.898 INFO:tasks.rados.rados.0.smithi107.stdout:9628: done (2 left) 2020-06-12T02:39:05.898 INFO:tasks.rados.rados.0.smithi107.stdout:9630: done (1 left) 2020-06-12T02:39:05.898 INFO:tasks.rados.rados.0.smithi107.stdout:9631: delete oid 3 current snap is 0 2020-06-12T02:39:05.899 INFO:tasks.rados.rados.0.smithi107.stdout:9629: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:05.900 INFO:tasks.rados.rados.0.smithi107.stdout:9629: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:05.902 INFO:tasks.rados.rados.0.smithi107.stdout:9631: done (1 left) 2020-06-12T02:39:05.902 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:39:05.902 INFO:tasks.rados.rados.0.smithi107.stdout:9632: seq_num 9631 ranges {1671168=507904} 2020-06-12T02:39:05.914 INFO:tasks.rados.rados.0.smithi107.stdout:9632: writing smithi10729407-7 from 1671168 to 2179072 tid 1 2020-06-12T02:39:05.914 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:39:05.915 INFO:tasks.rados.rados.0.smithi107.stdout:9633: seq_num 9632 ranges {0=524288} 2020-06-12T02:39:05.917 INFO:tasks.rados.rados.0.smithi107.stdout:9633: writing smithi10729407-1 from 0 to 524288 tid 1 2020-06-12T02:39:05.917 INFO:tasks.rados.rados.0.smithi107.stdout:9634: delete oid 16 current snap is 0 2020-06-12T02:39:05.917 INFO:tasks.rados.rados.0.smithi107.stdout:9629: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:05.918 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1786 (ObjNum 9628 snap 0 seq_num 9628) dirty exists 2020-06-12T02:39:05.918 INFO:tasks.rados.rados.0.smithi107.stdout:9629: left oid 8 (ObjNum 9628 snap 0 seq_num 9628) 2020-06-12T02:39:05.921 INFO:tasks.rados.rados.0.smithi107.stdout:9629: done (3 left) 2020-06-12T02:39:05.921 INFO:tasks.rados.rados.0.smithi107.stdout:9634: done (2 left) 2020-06-12T02:39:05.921 INFO:tasks.rados.rados.0.smithi107.stdout:9635: delete oid 12 current snap is 0 2020-06-12T02:39:05.923 INFO:tasks.rados.rados.0.smithi107.stdout:9632: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:05.924 INFO:tasks.rados.rados.0.smithi107.stdout:9635: done (2 left) 2020-06-12T02:39:05.924 INFO:tasks.rados.rados.0.smithi107.stdout:9636: delete oid 3 current snap is 0 2020-06-12T02:39:05.925 INFO:tasks.rados.rados.0.smithi107.stdout:9633: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:05.927 INFO:tasks.rados.rados.0.smithi107.stdout:9636: done (2 left) 2020-06-12T02:39:05.927 INFO:tasks.rados.rados.0.smithi107.stdout:9637: delete oid 3 current snap is 0 2020-06-12T02:39:05.928 INFO:tasks.rados.rados.0.smithi107.stdout:9632: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:05.929 INFO:tasks.rados.rados.0.smithi107.stdout:9633: finishing write tid 3 to smithi10729407-1 2020-06-12T02:39:05.930 INFO:tasks.rados.rados.0.smithi107.stdout:9632: finishing write tid 4 to smithi10729407-7 2020-06-12T02:39:05.930 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1656 (ObjNum 9631 snap 0 seq_num 9631) dirty exists 2020-06-12T02:39:05.931 INFO:tasks.rados.rados.0.smithi107.stdout:9632: left oid 7 (ObjNum 9631 snap 0 seq_num 9631) 2020-06-12T02:39:05.931 INFO:tasks.rados.rados.0.smithi107.stdout:9632: done (2 left) 2020-06-12T02:39:05.931 INFO:tasks.rados.rados.0.smithi107.stdout:9637: done (1 left) 2020-06-12T02:39:05.931 INFO:tasks.rados.rados.0.smithi107.stdout:9638: delete oid 2 current snap is 0 2020-06-12T02:39:05.932 INFO:tasks.rados.rados.0.smithi107.stdout:9633: finishing write tid 4 to smithi10729407-1 2020-06-12T02:39:05.932 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1731 (ObjNum 9632 snap 0 seq_num 9632) dirty exists 2020-06-12T02:39:05.932 INFO:tasks.rados.rados.0.smithi107.stdout:9633: left oid 1 (ObjNum 9632 snap 0 seq_num 9632) 2020-06-12T02:39:05.933 INFO:tasks.rados.rados.0.smithi107.stdout:9633: done (1 left) 2020-06-12T02:39:05.933 INFO:tasks.rados.rados.0.smithi107.stdout:9638: done (0 left) 2020-06-12T02:39:05.933 INFO:tasks.rados.rados.0.smithi107.stdout:9639: delete oid 6 current snap is 0 2020-06-12T02:39:05.935 INFO:tasks.rados.rados.0.smithi107.stdout:9639: done (0 left) 2020-06-12T02:39:05.935 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:39:05.935 INFO:tasks.rados.rados.0.smithi107.stdout:9640: seq_num 9639 ranges {0=638976} 2020-06-12T02:39:05.939 INFO:tasks.rados.rados.0.smithi107.stdout:9640: writing smithi10729407-6 from 0 to 638976 tid 1 2020-06-12T02:39:05.939 INFO:tasks.rados.rados.0.smithi107.stdout:9641: delete oid 4 current snap is 0 2020-06-12T02:39:05.942 INFO:tasks.rados.rados.0.smithi107.stdout:9641: done (1 left) 2020-06-12T02:39:05.942 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:39:05.943 INFO:tasks.rados.rados.0.smithi107.stdout:9642: seq_num 9641 ranges {0=671744} 2020-06-12T02:39:05.945 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:39:05.945 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:39:05.945 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:39:05.946 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:39:05.946 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:39:05.946 INFO:tasks.rados.rados.0.smithi107.stdout:9642: writing smithi10729407-4 from 0 to 671744 tid 1 2020-06-12T02:39:05.947 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:39:05.947 INFO:tasks.rados.rados.0.smithi107.stdout:9643: seq_num 9642 ranges {0=688128} 2020-06-12T02:39:05.950 INFO:tasks.rados.rados.0.smithi107.stdout:9643: writing smithi10729407-3 from 0 to 688128 tid 1 2020-06-12T02:39:05.951 INFO:tasks.rados.rados.0.smithi107.stdout:9644: delete oid 9 current snap is 0 2020-06-12T02:39:05.951 INFO:tasks.rados.rados.0.smithi107.stdout:9640: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:05.951 INFO:tasks.rados.rados.0.smithi107.stdout:9640: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:05.952 INFO:tasks.rados.rados.0.smithi107.stdout:9640: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:05.952 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1734 (ObjNum 9639 snap 0 seq_num 9639) dirty exists 2020-06-12T02:39:05.952 INFO:tasks.rados.rados.0.smithi107.stdout:9640: left oid 6 (ObjNum 9639 snap 0 seq_num 9639) 2020-06-12T02:39:05.956 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:39:05.954+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:39:05.956 INFO:tasks.rados.rados.0.smithi107.stdout:9640: done (3 left) 2020-06-12T02:39:05.956 INFO:tasks.rados.rados.0.smithi107.stdout:9644: done (2 left) 2020-06-12T02:39:05.956 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:39:05.957 INFO:tasks.rados.rados.0.smithi107.stdout:9645: seq_num 9644 ranges {0=720896} 2020-06-12T02:39:05.958 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-06-12T02:39:05.961 INFO:tasks.rados.rados.0.smithi107.stdout:9645: writing smithi10729407-18 from 0 to 720896 tid 1 2020-06-12T02:39:05.961 INFO:tasks.rados.rados.0.smithi107.stdout:9646: delete oid 10 current snap is 0 2020-06-12T02:39:05.961 INFO:tasks.rados.rados.0.smithi107.stdout:9642: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:05.961 INFO:tasks.rados.rados.0.smithi107.stdout:9642: finishing write tid 3 to smithi10729407-4 2020-06-12T02:39:05.961 INFO:tasks.rados.rados.0.smithi107.stdout:9642: finishing write tid 4 to smithi10729407-4 2020-06-12T02:39:05.962 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1806 (ObjNum 9641 snap 0 seq_num 9641) dirty exists 2020-06-12T02:39:05.962 INFO:tasks.rados.rados.0.smithi107.stdout:9642: left oid 4 (ObjNum 9641 snap 0 seq_num 9641) 2020-06-12T02:39:05.962 INFO:tasks.rados.rados.0.smithi107.stdout:9643: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:05.966 INFO:tasks.rados.rados.0.smithi107.stdout:9643: finishing write tid 3 to smithi10729407-3 2020-06-12T02:39:05.967 INFO:tasks.rados.rados.0.smithi107.stdout:9642: done (3 left) 2020-06-12T02:39:05.967 INFO:tasks.rados.rados.0.smithi107.stdout:9646: done (2 left) 2020-06-12T02:39:05.967 INFO:tasks.rados.rados.0.smithi107.stdout:9647: delete oid 8 current snap is 0 2020-06-12T02:39:05.969 INFO:tasks.rados.rados.0.smithi107.stdout:9645: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:05.969 INFO:tasks.rados.rados.0.smithi107.stdout:9643: finishing write tid 4 to smithi10729407-3 2020-06-12T02:39:05.969 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1726 (ObjNum 9642 snap 0 seq_num 9642) dirty exists 2020-06-12T02:39:05.970 INFO:tasks.rados.rados.0.smithi107.stdout:9643: left oid 3 (ObjNum 9642 snap 0 seq_num 9642) 2020-06-12T02:39:05.970 INFO:tasks.rados.rados.0.smithi107.stdout:9643: done (2 left) 2020-06-12T02:39:05.970 INFO:tasks.rados.rados.0.smithi107.stdout:9647: done (1 left) 2020-06-12T02:39:05.970 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:39:05.970 INFO:tasks.rados.rados.0.smithi107.stdout:9648: seq_num 9647 ranges {352256=778240} 2020-06-12T02:39:05.976 INFO:tasks.rados.rados.0.smithi107.stdout:9648: writing smithi10729407-11 from 352256 to 1130496 tid 1 2020-06-12T02:39:05.977 INFO:tasks.rados.rados.0.smithi107.stdout:9649: delete oid 3 current snap is 0 2020-06-12T02:39:05.978 INFO:tasks.rados.rados.0.smithi107.stdout:9645: finishing write tid 3 to smithi10729407-18 2020-06-12T02:39:05.978 INFO:tasks.rados.rados.0.smithi107.stdout:9645: finishing write tid 4 to smithi10729407-18 2020-06-12T02:39:05.979 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1763 (ObjNum 9644 snap 0 seq_num 9644) dirty exists 2020-06-12T02:39:05.979 INFO:tasks.rados.rados.0.smithi107.stdout:9645: left oid 18 (ObjNum 9644 snap 0 seq_num 9644) 2020-06-12T02:39:05.979 INFO:tasks.rados.rados.0.smithi107.stdout:9645: done (2 left) 2020-06-12T02:39:05.980 INFO:tasks.rados.rados.0.smithi107.stdout:9649: done (1 left) 2020-06-12T02:39:05.980 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:39:05.980 INFO:tasks.rados.rados.0.smithi107.stdout:9650: seq_num 9649 ranges {671744=409600,1081344=401408} 2020-06-12T02:39:05.988 INFO:tasks.rados.rados.0.smithi107.stdout:9650: writing smithi10729407-4 from 671744 to 1081344 tid 1 2020-06-12T02:39:05.989 INFO:tasks.rados.rados.0.smithi107.stdout:9650: writing smithi10729407-4 from 1081344 to 1482752 tid 2 2020-06-12T02:39:05.989 INFO:tasks.rados.rados.0.smithi107.stdout:9651: delete oid 15 current snap is 0 2020-06-12T02:39:05.990 INFO:tasks.rados.rados.0.smithi107.stdout:9648: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:05.990 INFO:tasks.rados.rados.0.smithi107.stdout:9648: finishing write tid 3 to smithi10729407-11 2020-06-12T02:39:05.990 INFO:tasks.rados.rados.0.smithi107.stdout:9648: finishing write tid 4 to smithi10729407-11 2020-06-12T02:39:05.991 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1638 (ObjNum 9647 snap 0 seq_num 9647) dirty exists 2020-06-12T02:39:05.991 INFO:tasks.rados.rados.0.smithi107.stdout:9648: left oid 11 (ObjNum 9647 snap 0 seq_num 9647) 2020-06-12T02:39:05.992 INFO:tasks.rados.rados.0.smithi107.stdout:9648: done (2 left) 2020-06-12T02:39:05.992 INFO:tasks.rados.rados.0.smithi107.stdout:9651: done (1 left) 2020-06-12T02:39:05.992 INFO:tasks.rados.rados.0.smithi107.stdout:9652: delete oid 16 current snap is 0 2020-06-12T02:39:05.994 INFO:tasks.rados.rados.0.smithi107.stdout:9650: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:05.995 INFO:tasks.rados.rados.0.smithi107.stdout:9650: finishing write tid 2 to smithi10729407-4 2020-06-12T02:39:05.996 INFO:tasks.rados.rados.0.smithi107.stdout:9652: done (1 left) 2020-06-12T02:39:05.996 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:39:05.996 INFO:tasks.rados.rados.0.smithi107.stdout:9653: seq_num 9652 ranges {0=458752,458752=401408} 2020-06-12T02:39:05.998 INFO:tasks.rados.rados.0.smithi107.stdout:9653: writing smithi10729407-17 from 0 to 458752 tid 1 2020-06-12T02:39:06.001 INFO:tasks.rados.rados.0.smithi107.stdout:9653: writing smithi10729407-17 from 458752 to 860160 tid 2 2020-06-12T02:39:06.001 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:39:06.001 INFO:tasks.rados.rados.0.smithi107.stdout:9654: seq_num 9653 ranges {0=475136,475136=401408} 2020-06-12T02:39:06.006 INFO:tasks.rados.rados.0.smithi107.stdout:9654: writing smithi10729407-8 from 0 to 475136 tid 1 2020-06-12T02:39:06.006 INFO:tasks.rados.rados.0.smithi107.stdout:9654: writing smithi10729407-8 from 475136 to 876544 tid 2 2020-06-12T02:39:06.006 INFO:tasks.rados.rados.0.smithi107.stdout:9655: delete oid 10 current snap is 0 2020-06-12T02:39:06.007 INFO:tasks.rados.rados.0.smithi107.stdout:9650: finishing write tid 4 to smithi10729407-4 2020-06-12T02:39:06.008 INFO:tasks.rados.rados.0.smithi107.stdout:9653: finishing write tid 1 to smithi10729407-17 2020-06-12T02:39:06.009 INFO:tasks.rados.rados.0.smithi107.stdout:9653: finishing write tid 2 to smithi10729407-17 2020-06-12T02:39:06.010 INFO:tasks.rados.rados.0.smithi107.stdout:9650: finishing write tid 5 to smithi10729407-4 2020-06-12T02:39:06.010 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1809 (ObjNum 9649 snap 0 seq_num 9649) dirty exists 2020-06-12T02:39:06.010 INFO:tasks.rados.rados.0.smithi107.stdout:9650: left oid 4 (ObjNum 9649 snap 0 seq_num 9649) 2020-06-12T02:39:06.013 INFO:tasks.rados.rados.0.smithi107.stdout:9650: done (3 left) 2020-06-12T02:39:06.013 INFO:tasks.rados.rados.0.smithi107.stdout:9655: done (2 left) 2020-06-12T02:39:06.013 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:39:06.013 INFO:tasks.rados.rados.0.smithi107.stdout:9656: seq_num 9655 ranges {720896=507904,1228800=401408} 2020-06-12T02:39:06.020 INFO:tasks.rados.rados.0.smithi107.stdout:9656: writing smithi10729407-18 from 720896 to 1228800 tid 1 2020-06-12T02:39:06.023 INFO:tasks.rados.rados.0.smithi107.stdout:9656: writing smithi10729407-18 from 1228800 to 1630208 tid 2 2020-06-12T02:39:06.023 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:39:06.023 INFO:tasks.rados.rados.0.smithi107.stdout:9657: seq_num 9656 ranges {540672=524288,1064960=401408} 2020-06-12T02:39:06.029 INFO:tasks.rados.rados.0.smithi107.stdout:9657: writing smithi10729407-19 from 540672 to 1064960 tid 1 2020-06-12T02:39:06.032 INFO:tasks.rados.rados.0.smithi107.stdout:9657: writing smithi10729407-19 from 1064960 to 1466368 tid 2 2020-06-12T02:39:06.032 INFO:tasks.rados.rados.0.smithi107.stdout:9658: delete oid 7 current snap is 0 2020-06-12T02:39:06.032 INFO:tasks.rados.rados.0.smithi107.stdout:9653: finishing write tid 4 to smithi10729407-17 2020-06-12T02:39:06.033 INFO:tasks.rados.rados.0.smithi107.stdout:9654: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:06.033 INFO:tasks.rados.rados.0.smithi107.stdout:9654: finishing write tid 2 to smithi10729407-8 2020-06-12T02:39:06.033 INFO:tasks.rados.rados.0.smithi107.stdout:9653: finishing write tid 5 to smithi10729407-17 2020-06-12T02:39:06.033 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1812 (ObjNum 9652 snap 0 seq_num 9652) dirty exists 2020-06-12T02:39:06.034 INFO:tasks.rados.rados.0.smithi107.stdout:9653: left oid 17 (ObjNum 9652 snap 0 seq_num 9652) 2020-06-12T02:39:06.034 INFO:tasks.rados.rados.0.smithi107.stdout:9654: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:06.034 INFO:tasks.rados.rados.0.smithi107.stdout:9654: finishing write tid 5 to smithi10729407-8 2020-06-12T02:39:06.034 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1791 (ObjNum 9653 snap 0 seq_num 9653) dirty exists 2020-06-12T02:39:06.034 INFO:tasks.rados.rados.0.smithi107.stdout:9654: left oid 8 (ObjNum 9653 snap 0 seq_num 9653) 2020-06-12T02:39:06.035 INFO:tasks.rados.rados.0.smithi107.stdout:9656: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:06.035 INFO:tasks.rados.rados.0.smithi107.stdout:9656: finishing write tid 2 to smithi10729407-18 2020-06-12T02:39:06.036 INFO:tasks.rados.rados.0.smithi107.stdout:9656: finishing write tid 4 to smithi10729407-18 2020-06-12T02:39:06.036 INFO:tasks.rados.rados.0.smithi107.stdout:9653: done (4 left) 2020-06-12T02:39:06.036 INFO:tasks.rados.rados.0.smithi107.stdout:9654: done (3 left) 2020-06-12T02:39:06.036 INFO:tasks.rados.rados.0.smithi107.stdout:9658: done (2 left) 2020-06-12T02:39:06.037 INFO:tasks.rados.rados.0.smithi107.stdout:9659: delete oid 4 current snap is 0 2020-06-12T02:39:06.037 INFO:tasks.rados.rados.0.smithi107.stdout:9656: finishing write tid 5 to smithi10729407-18 2020-06-12T02:39:06.037 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1766 (ObjNum 9655 snap 0 seq_num 9655) dirty exists 2020-06-12T02:39:06.037 INFO:tasks.rados.rados.0.smithi107.stdout:9656: left oid 18 (ObjNum 9655 snap 0 seq_num 9655) 2020-06-12T02:39:06.039 INFO:tasks.rados.rados.0.smithi107.stdout:9657: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:06.040 INFO:tasks.rados.rados.0.smithi107.stdout:9656: done (2 left) 2020-06-12T02:39:06.040 INFO:tasks.rados.rados.0.smithi107.stdout:9659: done (1 left) 2020-06-12T02:39:06.040 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:39:06.041 INFO:tasks.rados.rados.0.smithi107.stdout:9660: seq_num 9659 ranges {0=573440,573440=401408} 2020-06-12T02:39:06.043 INFO:tasks.rados.rados.0.smithi107.stdout:9660: writing smithi10729407-9 from 0 to 573440 tid 1 2020-06-12T02:39:06.045 INFO:tasks.rados.rados.0.smithi107.stdout:9660: writing smithi10729407-9 from 573440 to 974848 tid 2 2020-06-12T02:39:06.046 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:39:06.046 INFO:tasks.rados.rados.0.smithi107.stdout:9661: seq_num 9660 ranges {0=589824,589824=401408} 2020-06-12T02:39:06.049 INFO:tasks.rados.rados.0.smithi107.stdout:9661: writing smithi10729407-2 from 0 to 589824 tid 1 2020-06-12T02:39:06.053 INFO:tasks.rados.rados.0.smithi107.stdout:9661: writing smithi10729407-2 from 589824 to 991232 tid 2 2020-06-12T02:39:06.053 INFO:tasks.rados.rados.0.smithi107.stdout:9662: delete oid 17 current snap is 0 2020-06-12T02:39:06.054 INFO:tasks.rados.rados.0.smithi107.stdout:9657: finishing write tid 2 to smithi10729407-19 2020-06-12T02:39:06.054 INFO:tasks.rados.rados.0.smithi107.stdout:9657: finishing write tid 4 to smithi10729407-19 2020-06-12T02:39:06.054 INFO:tasks.rados.rados.0.smithi107.stdout:9657: finishing write tid 5 to smithi10729407-19 2020-06-12T02:39:06.054 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1782 (ObjNum 9656 snap 0 seq_num 9656) dirty exists 2020-06-12T02:39:06.054 INFO:tasks.rados.rados.0.smithi107.stdout:9657: left oid 19 (ObjNum 9656 snap 0 seq_num 9656) 2020-06-12T02:39:06.055 INFO:tasks.rados.rados.0.smithi107.stdout:9660: finishing write tid 1 to smithi10729407-9 2020-06-12T02:39:06.056 INFO:tasks.rados.rados.0.smithi107.stdout:9660: finishing write tid 2 to smithi10729407-9 2020-06-12T02:39:06.056 INFO:tasks.rados.rados.0.smithi107.stdout:9660: finishing write tid 4 to smithi10729407-9 2020-06-12T02:39:06.056 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:39:06.054+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:39:06.057 INFO:tasks.rados.rados.0.smithi107.stdout:9660: finishing write tid 5 to smithi10729407-9 2020-06-12T02:39:06.057 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1759 (ObjNum 9659 snap 0 seq_num 9659) dirty exists 2020-06-12T02:39:06.057 INFO:tasks.rados.rados.0.smithi107.stdout:9660: left oid 9 (ObjNum 9659 snap 0 seq_num 9659) 2020-06-12T02:39:06.058 INFO:tasks.rados.rados.0.smithi107.stdout:9661: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:06.058 INFO:tasks.rados.rados.0.smithi107.stdout:9657: done (3 left) 2020-06-12T02:39:06.059 INFO:tasks.rados.rados.0.smithi107.stdout:9660: done (2 left) 2020-06-12T02:39:06.059 INFO:tasks.rados.rados.0.smithi107.stdout:9662: done (1 left) 2020-06-12T02:39:06.060 INFO:tasks.rados.rados.0.smithi107.stdout:9663: delete oid 10 current snap is 0 2020-06-12T02:39:06.061 INFO:tasks.rados.rados.0.smithi107.stdout:9663: done (1 left) 2020-06-12T02:39:06.062 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:39:06.063 INFO:tasks.rados.rados.0.smithi107.stdout:9664: seq_num 9663 ranges {1130496=638976,1769472=401408} 2020-06-12T02:39:06.083 INFO:tasks.rados.rados.0.smithi107.stdout:9664: writing smithi10729407-11 from 1130496 to 1769472 tid 1 2020-06-12T02:39:06.083 INFO:tasks.rados.rados.0.smithi107.stdout:9664: writing smithi10729407-11 from 1769472 to 2170880 tid 2 2020-06-12T02:39:06.084 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:39:06.084 INFO:tasks.rados.rados.0.smithi107.stdout:9665: seq_num 9664 ranges {0=663552,663552=401408} 2020-06-12T02:39:06.084 INFO:tasks.rados.rados.0.smithi107.stdout:9665: writing smithi10729407-5 from 0 to 663552 tid 1 2020-06-12T02:39:06.084 INFO:tasks.rados.rados.0.smithi107.stdout:9665: writing smithi10729407-5 from 663552 to 1064960 tid 2 2020-06-12T02:39:06.085 INFO:tasks.rados.rados.0.smithi107.stdout:9666: delete oid 9 current snap is 0 2020-06-12T02:39:06.085 INFO:tasks.rados.rados.0.smithi107.stdout:9661: finishing write tid 2 to smithi10729407-2 2020-06-12T02:39:06.085 INFO:tasks.rados.rados.0.smithi107.stdout:9661: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:06.085 INFO:tasks.rados.rados.0.smithi107.stdout:9661: finishing write tid 5 to smithi10729407-2 2020-06-12T02:39:06.086 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 899 (ObjNum 9660 snap 0 seq_num 9660) dirty exists 2020-06-12T02:39:06.086 INFO:tasks.rados.rados.0.smithi107.stdout:9661: left oid 2 (ObjNum 9660 snap 0 seq_num 9660) 2020-06-12T02:39:06.086 INFO:tasks.rados.rados.0.smithi107.stdout:9664: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:06.087 INFO:tasks.rados.rados.0.smithi107.stdout:9661: done (3 left) 2020-06-12T02:39:06.087 INFO:tasks.rados.rados.0.smithi107.stdout:9666: done (2 left) 2020-06-12T02:39:06.087 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:39:06.088 INFO:tasks.rados.rados.0.smithi107.stdout:9667: seq_num 9666 ranges {524288=696320,1220608=401408} 2020-06-12T02:39:06.093 INFO:tasks.rados.rados.0.smithi107.stdout:9667: writing smithi10729407-1 from 524288 to 1220608 tid 1 2020-06-12T02:39:06.096 INFO:tasks.rados.rados.0.smithi107.stdout:9667: writing smithi10729407-1 from 1220608 to 1622016 tid 2 2020-06-12T02:39:06.096 INFO:tasks.rados.rados.0.smithi107.stdout:9668: delete oid 17 current snap is 0 2020-06-12T02:39:06.096 INFO:tasks.rados.rados.0.smithi107.stdout:9665: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:06.097 INFO:tasks.rados.rados.0.smithi107.stdout:9664: finishing write tid 2 to smithi10729407-11 2020-06-12T02:39:06.097 INFO:tasks.rados.rados.0.smithi107.stdout:9664: finishing write tid 4 to smithi10729407-11 2020-06-12T02:39:06.097 INFO:tasks.rados.rados.0.smithi107.stdout:9665: finishing write tid 2 to smithi10729407-5 2020-06-12T02:39:06.097 INFO:tasks.rados.rados.0.smithi107.stdout:9665: finishing write tid 4 to smithi10729407-5 2020-06-12T02:39:06.098 INFO:tasks.rados.rados.0.smithi107.stdout:9664: finishing write tid 5 to smithi10729407-11 2020-06-12T02:39:06.098 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1641 (ObjNum 9663 snap 0 seq_num 9663) dirty exists 2020-06-12T02:39:06.098 INFO:tasks.rados.rados.0.smithi107.stdout:9664: left oid 11 (ObjNum 9663 snap 0 seq_num 9663) 2020-06-12T02:39:06.098 INFO:tasks.rados.rados.0.smithi107.stdout:9665: finishing write tid 5 to smithi10729407-5 2020-06-12T02:39:06.098 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 929 (ObjNum 9664 snap 0 seq_num 9664) dirty exists 2020-06-12T02:39:06.099 INFO:tasks.rados.rados.0.smithi107.stdout:9665: left oid 5 (ObjNum 9664 snap 0 seq_num 9664) 2020-06-12T02:39:06.105 INFO:tasks.rados.rados.0.smithi107.stdout:9664: done (3 left) 2020-06-12T02:39:06.105 INFO:tasks.rados.rados.0.smithi107.stdout:9665: done (2 left) 2020-06-12T02:39:06.105 INFO:tasks.rados.rados.0.smithi107.stdout:9668: done (1 left) 2020-06-12T02:39:06.105 INFO:tasks.rados.rados.0.smithi107.stdout:9669: delete oid 2 current snap is 0 2020-06-12T02:39:06.106 INFO:tasks.rados.rados.0.smithi107.stdout:9667: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:06.106 INFO:tasks.rados.rados.0.smithi107.stdout:9669: done (1 left) 2020-06-12T02:39:06.106 INFO:tasks.rados.rados.0.smithi107.stdout:9670: delete oid 9 current snap is 0 2020-06-12T02:39:06.113 INFO:tasks.rados.rados.0.smithi107.stdout:9670: done (1 left) 2020-06-12T02:39:06.113 INFO:tasks.rados.rados.0.smithi107.stdout:9671: delete oid 19 current snap is 0 2020-06-12T02:39:06.113 INFO:tasks.rados.rados.0.smithi107.stdout:9667: finishing write tid 2 to smithi10729407-1 2020-06-12T02:39:06.114 INFO:tasks.rados.rados.0.smithi107.stdout:9667: finishing write tid 4 to smithi10729407-1 2020-06-12T02:39:06.114 INFO:tasks.rados.rados.0.smithi107.stdout:9671: done (1 left) 2020-06-12T02:39:06.114 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:39:06.115 INFO:tasks.rados.rados.0.smithi107.stdout:9672: seq_num 9671 ranges {0=778240,778240=401408} 2020-06-12T02:39:06.115 INFO:tasks.rados.rados.0.smithi107.stdout:9672: writing smithi10729407-14 from 0 to 778240 tid 1 2020-06-12T02:39:06.117 INFO:tasks.rados.rados.0.smithi107.stdout:9672: writing smithi10729407-14 from 778240 to 1179648 tid 2 2020-06-12T02:39:06.118 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:39:06.118 INFO:tasks.rados.rados.0.smithi107.stdout:9673: seq_num 9672 ranges {0=794624,794624=401408} 2020-06-12T02:39:06.121 INFO:tasks.rados.rados.0.smithi107.stdout:9673: writing smithi10729407-2 from 0 to 794624 tid 1 2020-06-12T02:39:06.124 INFO:tasks.rados.rados.0.smithi107.stdout:9673: writing smithi10729407-2 from 794624 to 1196032 tid 2 2020-06-12T02:39:06.124 INFO:tasks.rados.rados.0.smithi107.stdout:9674: delete oid 3 current snap is 0 2020-06-12T02:39:06.125 INFO:tasks.rados.rados.0.smithi107.stdout:9667: finishing write tid 5 to smithi10729407-1 2020-06-12T02:39:06.125 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1737 (ObjNum 9666 snap 0 seq_num 9666) dirty exists 2020-06-12T02:39:06.125 INFO:tasks.rados.rados.0.smithi107.stdout:9667: left oid 1 (ObjNum 9666 snap 0 seq_num 9666) 2020-06-12T02:39:06.126 INFO:tasks.rados.rados.0.smithi107.stdout:9672: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:06.127 INFO:tasks.rados.rados.0.smithi107.stdout:9667: done (3 left) 2020-06-12T02:39:06.127 INFO:tasks.rados.rados.0.smithi107.stdout:9674: done (2 left) 2020-06-12T02:39:06.128 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:39:06.128 INFO:tasks.rados.rados.0.smithi107.stdout:9675: seq_num 9674 ranges {2170880=425984,2596864=622592,3219456=180224} 2020-06-12T02:39:06.142 INFO:tasks.rados.rados.0.smithi107.stdout:9675: writing smithi10729407-11 from 2170880 to 2596864 tid 1 2020-06-12T02:39:06.145 INFO:tasks.rados.rados.0.smithi107.stdout:9675: writing smithi10729407-11 from 2596864 to 3219456 tid 2 2020-06-12T02:39:06.147 INFO:tasks.rados.rados.0.smithi107.stdout:9675: writing smithi10729407-11 from 3219456 to 3399680 tid 3 2020-06-12T02:39:06.147 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:39:06.147 INFO:tasks.rados.rados.0.smithi107.stdout:9676: seq_num 9675 ranges {0=442368,442368=507904,950272=294912} 2020-06-12T02:39:06.150 INFO:tasks.rados.rados.0.smithi107.stdout:9676: writing smithi10729407-19 from 0 to 442368 tid 1 2020-06-12T02:39:06.153 INFO:tasks.rados.rados.0.smithi107.stdout:9676: writing smithi10729407-19 from 442368 to 950272 tid 2 2020-06-12T02:39:06.155 INFO:tasks.rados.rados.0.smithi107.stdout:9676: writing smithi10729407-19 from 950272 to 1245184 tid 3 2020-06-12T02:39:06.156 INFO:tasks.rados.rados.0.smithi107.stdout:9677: delete oid 8 current snap is 0 2020-06-12T02:39:06.156 INFO:tasks.rados.rados.0.smithi107.stdout:9672: finishing write tid 2 to smithi10729407-14 2020-06-12T02:39:06.156 INFO:tasks.rados.rados.0.smithi107.stdout:9672: finishing write tid 4 to smithi10729407-14 2020-06-12T02:39:06.156 INFO:tasks.rados.rados.0.smithi107.stdout:9672: finishing write tid 5 to smithi10729407-14 2020-06-12T02:39:06.157 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1622 (ObjNum 9671 snap 0 seq_num 9671) dirty exists 2020-06-12T02:39:06.157 INFO:tasks.rados.rados.0.smithi107.stdout:9672: left oid 14 (ObjNum 9671 snap 0 seq_num 9671) 2020-06-12T02:39:06.157 INFO:tasks.rados.rados.0.smithi107.stdout:9673: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:06.157 INFO:tasks.rados.rados.0.smithi107.stdout:9673: finishing write tid 2 to smithi10729407-2 2020-06-12T02:39:06.157 INFO:tasks.rados.rados.0.smithi107.stdout:9673: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:06.158 INFO:tasks.rados.rados.0.smithi107.stdout:9673: finishing write tid 5 to smithi10729407-2 2020-06-12T02:39:06.158 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 903 (ObjNum 9672 snap 0 seq_num 9672) dirty exists 2020-06-12T02:39:06.158 INFO:tasks.rados.rados.0.smithi107.stdout:9673: left oid 2 (ObjNum 9672 snap 0 seq_num 9672) 2020-06-12T02:39:06.158 INFO:tasks.rados.rados.0.smithi107.stdout:9675: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:06.158 INFO:tasks.rados.rados.0.smithi107.stdout:9675: finishing write tid 2 to smithi10729407-11 2020-06-12T02:39:06.158 INFO:tasks.rados.rados.0.smithi107.stdout:9675: finishing write tid 3 to smithi10729407-11 2020-06-12T02:39:06.159 INFO:tasks.rados.rados.0.smithi107.stdout:9675: finishing write tid 5 to smithi10729407-11 2020-06-12T02:39:06.159 INFO:tasks.rados.rados.0.smithi107.stdout:9676: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:06.160 INFO:tasks.rados.rados.0.smithi107.stdout:9675: finishing write tid 6 to smithi10729407-11 2020-06-12T02:39:06.160 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1645 (ObjNum 9674 snap 0 seq_num 9674) dirty exists 2020-06-12T02:39:06.160 INFO:tasks.rados.rados.0.smithi107.stdout:9675: left oid 11 (ObjNum 9674 snap 0 seq_num 9674) 2020-06-12T02:39:06.160 INFO:tasks.rados.rados.0.smithi107.stdout:9672: done (4 left) 2020-06-12T02:39:06.161 INFO:tasks.rados.rados.0.smithi107.stdout:9673: done (3 left) 2020-06-12T02:39:06.161 INFO:tasks.rados.rados.0.smithi107.stdout:9675: done (2 left) 2020-06-12T02:39:06.161 INFO:tasks.rados.rados.0.smithi107.stdout:9677: done (1 left) 2020-06-12T02:39:06.161 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:39:06.161 INFO:tasks.rados.rados.0.smithi107.stdout:9678: seq_num 9677 ranges {2342912=475136,2818048=679936,3497984=122880} 2020-06-12T02:39:06.162 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:39:06.159+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:39:06.165 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:39:06.166 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:39:06.166 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:39:06.166 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:39:06.166 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.166 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8464 3.fs0 3.57bc3c1f (undecoded) ondisk+retry+write+known_if_redirected e34)", 2020-06-12T02:39:06.167 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:53.549364+0000", 2020-06-12T02:39:06.167 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 72.611512262000005, 2020-06-12T02:39:06.167 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.003107494, 2020-06-12T02:39:06.167 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:06.168 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:39:06.168 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:06.168 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:06.168 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:06.168 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8464 2020-06-12T02:39:06.169 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.169 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:06.169 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.169 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:06.169 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.549364+0000", 2020-06-12T02:39:06.169 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:06.170 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.170 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.170 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:06.170 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.549364+0000", 2020-06-12T02:39:06.170 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023109609999999998 2020-06-12T02:39:06.171 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.171 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.171 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:06.171 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551675+0000", 2020-06-12T02:39:06.171 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3331e-05 2020-06-12T02:39:06.172 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.172 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.172 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:06.172 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551688+0000", 2020-06-12T02:39:06.172 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4173e-05 2020-06-12T02:39:06.172 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.173 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.173 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:06.173 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551703+0000", 2020-06-12T02:39:06.173 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0454e-05 2020-06-12T02:39:06.173 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.174 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.174 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:06.174 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551713+0000", 2020-06-12T02:39:06.174 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031069050000000001 2020-06-12T02:39:06.174 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.174 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.175 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:06.175 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554820+0000", 2020-06-12T02:39:06.175 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2820000000000002e-06 2020-06-12T02:39:06.175 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.175 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.176 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2020-06-12T02:39:06.176 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554826+0000", 2020-06-12T02:39:06.176 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99762648499999995 2020-06-12T02:39:06.176 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.176 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.176 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:06.178 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552453+0000", 2020-06-12T02:39:06.178 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8902999999999998e-05 2020-06-12T02:39:06.178 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.178 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.179 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:06.179 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552472+0000", 2020-06-12T02:39:06.179 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.003107494 2020-06-12T02:39:06.179 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:06.179 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:06.179 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:06.180 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.180 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.180 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8465 3.fs0 3.57bc3c1f (undecoded) ondisk+retry+write+known_if_redirected e34)", 2020-06-12T02:39:06.180 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:53.551774+0000", 2020-06-12T02:39:06.180 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 72.609102625999995, 2020-06-12T02:39:06.181 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.000741442, 2020-06-12T02:39:06.181 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:06.181 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:39:06.181 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:06.181 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:06.182 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:06.182 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8465 2020-06-12T02:39:06.182 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.182 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:06.182 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.182 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:06.183 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551774+0000", 2020-06-12T02:39:06.183 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:06.183 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.183 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.183 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:06.184 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551774+0000", 2020-06-12T02:39:06.184 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0010760539999999999 2020-06-12T02:39:06.184 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.184 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.184 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:06.184 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552850+0000", 2020-06-12T02:39:06.185 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0139000000000001e-05 2020-06-12T02:39:06.185 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.185 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.185 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:06.186 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552860+0000", 2020-06-12T02:39:06.186 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7646999999999999e-05 2020-06-12T02:39:06.186 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.186 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.186 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:06.187 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552878+0000", 2020-06-12T02:39:06.191 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2916e-05 2020-06-12T02:39:06.191 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.192 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.192 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:06.192 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552890+0000", 2020-06-12T02:39:06.192 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0019574610000000002 2020-06-12T02:39:06.192 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.193 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.193 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:06.193 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554848+0000", 2020-06-12T02:39:06.193 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0399999999999995e-07 2020-06-12T02:39:06.193 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.193 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.194 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2020-06-12T02:39:06.194 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554849+0000", 2020-06-12T02:39:06.194 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99765586100000003 2020-06-12T02:39:06.194 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.194 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.195 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:06.195 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552504+0000", 2020-06-12T02:39:06.195 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.066e-05 2020-06-12T02:39:06.195 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.195 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.196 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:06.196 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552515+0000", 2020-06-12T02:39:06.196 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.000741442 2020-06-12T02:39:06.196 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:06.196 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:06.196 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:06.197 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.197 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.197 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8466 3.fs0 3.57bc3c1f (undecoded) ondisk+write+known_if_redirected e34)", 2020-06-12T02:39:06.198 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:53.552953+0000", 2020-06-12T02:39:06.198 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 72.607923346000007, 2020-06-12T02:39:06.198 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99960144299999998, 2020-06-12T02:39:06.198 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:06.198 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:39:06.199 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:06.199 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:06.199 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:06.199 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8466 2020-06-12T02:39:06.199 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.199 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:06.200 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.200 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:06.200 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552953+0000", 2020-06-12T02:39:06.201 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:06.201 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.201 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.201 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:06.202 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552953+0000", 2020-06-12T02:39:06.202 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013847100000000001 2020-06-12T02:39:06.202 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.202 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.203 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:06.203 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553091+0000", 2020-06-12T02:39:06.203 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7190000000000004e-06 2020-06-12T02:39:06.203 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.203 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.203 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:06.204 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553098+0000", 2020-06-12T02:39:06.204 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.171e-05 2020-06-12T02:39:06.204 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.204 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.204 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:06.205 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553110+0000", 2020-06-12T02:39:06.205 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5770000000000001e-06 2020-06-12T02:39:06.205 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.205 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.205 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:06.205 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553115+0000", 2020-06-12T02:39:06.206 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0017512739999999999 2020-06-12T02:39:06.206 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.206 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.206 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:06.206 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554867+0000", 2020-06-12T02:39:06.207 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1200000000000003e-07 2020-06-12T02:39:06.207 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.207 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.207 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2020-06-12T02:39:06.207 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554867+0000", 2020-06-12T02:39:06.208 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99767707100000003 2020-06-12T02:39:06.208 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.208 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.208 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:06.208 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552544+0000", 2020-06-12T02:39:06.208 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0009e-05 2020-06-12T02:39:06.209 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.209 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.209 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:06.209 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552554+0000", 2020-06-12T02:39:06.209 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99960144299999998 2020-06-12T02:39:06.210 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:06.210 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:06.210 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:06.210 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.210 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.210 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8467 3.fs0 3.57bc3c1f (undecoded) ondisk+write+known_if_redirected e34)", 2020-06-12T02:39:06.211 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:53.553172+0000", 2020-06-12T02:39:06.211 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 72.607704130000002, 2020-06-12T02:39:06.211 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99942086200000002, 2020-06-12T02:39:06.211 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:06.211 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:39:06.212 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:06.212 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:06.212 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:06.212 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8467 2020-06-12T02:39:06.212 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.213 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:06.213 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.213 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:06.213 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553172+0000", 2020-06-12T02:39:06.213 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:06.213 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.214 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.214 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:06.214 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553172+0000", 2020-06-12T02:39:06.214 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5913000000000003e-05 2020-06-12T02:39:06.214 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.215 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.215 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:06.215 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553248+0000", 2020-06-12T02:39:06.215 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7740000000000001e-06 2020-06-12T02:39:06.215 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.215 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.216 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:06.216 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553254+0000", 2020-06-12T02:39:06.216 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1494999999999999e-05 2020-06-12T02:39:06.216 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.216 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.217 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:06.217 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553265+0000", 2020-06-12T02:39:06.217 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5170000000000001e-06 2020-06-12T02:39:06.217 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.217 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.217 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:06.218 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553270+0000", 2020-06-12T02:39:06.218 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001616567 2020-06-12T02:39:06.218 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.218 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.218 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:06.219 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554886+0000", 2020-06-12T02:39:06.219 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7800000000000001e-07 2020-06-12T02:39:06.219 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.221 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.221 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2020-06-12T02:39:06.221 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554887+0000", 2020-06-12T02:39:06.221 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99769652399999997 2020-06-12T02:39:06.222 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.222 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.222 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:06.222 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552584+0000", 2020-06-12T02:39:06.223 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4939999999999992e-06 2020-06-12T02:39:06.223 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.223 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.223 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:06.223 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552593+0000", 2020-06-12T02:39:06.223 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99942086200000002 2020-06-12T02:39:06.224 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:06.224 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:06.224 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:06.224 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.224 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.225 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8468 3.fs0 3.57bc3c1f (undecoded) ondisk+read+rwordered+known_if_redirected e34)", 2020-06-12T02:39:06.225 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:53.553311+0000", 2020-06-12T02:39:06.225 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 72.607565531999995, 2020-06-12T02:39:06.225 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99988704799999994, 2020-06-12T02:39:06.225 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:06.226 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:39:06.226 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:06.226 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:06.226 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:06.226 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8468 2020-06-12T02:39:06.226 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.227 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:06.227 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.227 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:06.227 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553311+0000", 2020-06-12T02:39:06.227 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:06.228 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.228 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.228 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:06.228 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553311+0000", 2020-06-12T02:39:06.228 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2050000000000001e-05 2020-06-12T02:39:06.228 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.229 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.229 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:06.229 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553333+0000", 2020-06-12T02:39:06.229 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5489999999999998e-06 2020-06-12T02:39:06.229 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.230 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.230 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:06.230 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553336+0000", 2020-06-12T02:39:06.230 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1620000000000002e-06 2020-06-12T02:39:06.230 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.231 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.231 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:06.231 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553344+0000", 2020-06-12T02:39:06.231 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.807e-06 2020-06-12T02:39:06.231 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.231 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.232 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:06.232 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553346+0000", 2020-06-12T02:39:06.232 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001560575 2020-06-12T02:39:06.232 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.232 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.232 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:06.233 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554907+0000", 2020-06-12T02:39:06.233 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1429999999999999e-06 2020-06-12T02:39:06.233 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.233 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.233 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2020-06-12T02:39:06.234 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554908+0000", 2020-06-12T02:39:06.234 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99827844600000004 2020-06-12T02:39:06.234 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.234 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.234 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:06.235 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.553187+0000", 2020-06-12T02:39:06.235 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1316e-05 2020-06-12T02:39:06.235 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.235 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.235 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:06.235 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.553198+0000", 2020-06-12T02:39:06.236 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99988704799999994 2020-06-12T02:39:06.236 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:06.236 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:06.236 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:06.237 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.237 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.237 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11992 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [append 0~581632] snapc 0=[] ondisk+write+known_if_redirected e56)", 2020-06-12T02:39:06.237 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:23.783061+0000", 2020-06-12T02:39:06.237 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 42.377815351000002, 2020-06-12T02:39:06.238 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022850993, 2020-06-12T02:39:06.238 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:06.238 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:06.238 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:06.238 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:06.238 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:06.239 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11992 2020-06-12T02:39:06.239 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.239 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:06.239 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.240 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:06.240 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783061+0000", 2020-06-12T02:39:06.240 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:06.240 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.241 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.241 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:06.242 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783061+0000", 2020-06-12T02:39:06.242 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047999300000000001 2020-06-12T02:39:06.242 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.242 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.242 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:06.242 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783541+0000", 2020-06-12T02:39:06.243 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5649999999999998e-06 2020-06-12T02:39:06.243 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.244 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.244 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:06.244 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783548+0000", 2020-06-12T02:39:06.244 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3559e-05 2020-06-12T02:39:06.245 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.245 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.245 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:06.245 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783561+0000", 2020-06-12T02:39:06.245 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3539999999999996e-06 2020-06-12T02:39:06.246 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.246 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.246 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:06.246 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783567+0000", 2020-06-12T02:39:06.247 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001353604 2020-06-12T02:39:06.247 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.247 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.247 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:06.247 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.784921+0000", 2020-06-12T02:39:06.248 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000111853 2020-06-12T02:39:06.248 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.248 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.248 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:06.248 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.785033+0000", 2020-06-12T02:39:06.249 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051647919999999996 2020-06-12T02:39:06.249 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.249 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.249 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:06.249 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.790198+0000", 2020-06-12T02:39:06.250 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000110742 2020-06-12T02:39:06.250 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.250 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.250 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:06.250 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.790308+0000", 2020-06-12T02:39:06.251 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00090050099999999995 2020-06-12T02:39:06.251 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.251 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.251 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:06.251 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.791209+0000", 2020-06-12T02:39:06.252 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010078284999999999 2020-06-12T02:39:06.252 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.252 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.253 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:06.253 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.801287+0000", 2020-06-12T02:39:06.253 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045443109999999997 2020-06-12T02:39:06.253 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.254 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.254 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:06.254 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.805832+0000", 2020-06-12T02:39:06.254 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.0433999999999996e-05 2020-06-12T02:39:06.255 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.255 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.255 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:06.255 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.805912+0000", 2020-06-12T02:39:06.255 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022850993 2020-06-12T02:39:06.256 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:06.256 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:06.256 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:06.256 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.256 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.257 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11994 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected e56)", 2020-06-12T02:39:06.257 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:23.788734+0000", 2020-06-12T02:39:06.257 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 42.372142705999998, 2020-06-12T02:39:06.257 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024022136999999999, 2020-06-12T02:39:06.257 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:06.258 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:06.258 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:06.258 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:06.258 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:06.258 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11994 2020-06-12T02:39:06.259 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.259 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:06.259 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.259 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:06.260 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.788734+0000", 2020-06-12T02:39:06.260 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:06.260 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.260 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.261 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:06.261 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.788734+0000", 2020-06-12T02:39:06.261 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00034212499999999998 2020-06-12T02:39:06.261 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.261 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.262 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:06.262 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789076+0000", 2020-06-12T02:39:06.262 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.254e-06 2020-06-12T02:39:06.262 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.262 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.263 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:06.263 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789081+0000", 2020-06-12T02:39:06.263 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8249999999999994e-06 2020-06-12T02:39:06.263 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.263 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.264 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:06.264 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789090+0000", 2020-06-12T02:39:06.264 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2259999999999999e-06 2020-06-12T02:39:06.264 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.264 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.265 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:06.265 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789094+0000", 2020-06-12T02:39:06.265 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2913000000000001e-05 2020-06-12T02:39:06.265 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.265 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.266 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:06.266 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789127+0000", 2020-06-12T02:39:06.266 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2001000000000005e-05 2020-06-12T02:39:06.266 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.266 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.267 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:06.267 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789189+0000", 2020-06-12T02:39:06.267 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0097474550000000004 2020-06-12T02:39:06.267 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.267 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.268 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:06.268 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.798936+0000", 2020-06-12T02:39:06.268 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9882000000000005e-05 2020-06-12T02:39:06.268 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.268 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.269 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:06.269 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.799016+0000", 2020-06-12T02:39:06.269 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00048691000000000001 2020-06-12T02:39:06.269 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.269 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.270 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:06.270 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.799503+0000", 2020-06-12T02:39:06.270 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002766363 2020-06-12T02:39:06.270 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.270 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.271 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:06.271 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.802270+0000", 2020-06-12T02:39:06.271 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010404759 2020-06-12T02:39:06.271 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.271 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.272 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:06.272 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.812674+0000", 2020-06-12T02:39:06.272 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1423999999999998e-05 2020-06-12T02:39:06.272 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.272 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.273 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:06.273 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.812756+0000", 2020-06-12T02:39:06.273 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024022136999999999 2020-06-12T02:39:06.273 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:06.273 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:06.274 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:06.274 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.274 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.274 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11995 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e56)", 2020-06-12T02:39:06.274 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:23.789128+0000", 2020-06-12T02:39:06.275 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 42.371748259999997, 2020-06-12T02:39:06.275 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031795508, 2020-06-12T02:39:06.275 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:06.276 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:06.276 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:06.276 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:06.276 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:06.277 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11995 2020-06-12T02:39:06.277 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.277 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:06.277 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.277 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:06.278 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789128+0000", 2020-06-12T02:39:06.278 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:06.279 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.279 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.279 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:06.279 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789128+0000", 2020-06-12T02:39:06.279 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4419999999999998e-05 2020-06-12T02:39:06.280 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.280 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.280 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:06.280 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789172+0000", 2020-06-12T02:39:06.280 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7110000000000001e-06 2020-06-12T02:39:06.281 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.281 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.281 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:06.281 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789176+0000", 2020-06-12T02:39:06.281 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5089999999999999e-06 2020-06-12T02:39:06.282 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.282 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.282 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:06.282 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789183+0000", 2020-06-12T02:39:06.282 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.497e-06 2020-06-12T02:39:06.283 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.283 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.283 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:06.283 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789185+0000", 2020-06-12T02:39:06.283 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012172176 2020-06-12T02:39:06.284 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.284 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.284 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:06.284 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.801357+0000", 2020-06-12T02:39:06.284 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4264000000000006e-05 2020-06-12T02:39:06.285 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.285 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.285 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:06.285 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.801432+0000", 2020-06-12T02:39:06.285 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00029680499999999997 2020-06-12T02:39:06.286 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.286 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.286 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:06.286 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.801728+0000", 2020-06-12T02:39:06.286 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00066278300000000001 2020-06-12T02:39:06.287 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.287 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.287 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:06.287 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.802391+0000", 2020-06-12T02:39:06.287 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018429867999999999 2020-06-12T02:39:06.288 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.288 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.288 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:06.288 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.820821+0000", 2020-06-12T02:39:06.288 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000102475 2020-06-12T02:39:06.289 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.289 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.289 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:06.289 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.820924+0000", 2020-06-12T02:39:06.289 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031795508 2020-06-12T02:39:06.290 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:06.290 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:06.290 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:06.290 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.290 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.291 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11996 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e56)", 2020-06-12T02:39:06.291 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:23.789235+0000", 2020-06-12T02:39:06.291 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 42.371641408999999, 2020-06-12T02:39:06.291 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.034785745, 2020-06-12T02:39:06.292 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:06.293 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:06.293 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:06.293 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:06.293 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:06.294 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11996 2020-06-12T02:39:06.294 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.294 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:06.294 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.294 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:06.295 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789235+0000", 2020-06-12T02:39:06.295 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:06.295 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.295 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.295 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:06.295 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789235+0000", 2020-06-12T02:39:06.296 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6375000000000001e-05 2020-06-12T02:39:06.296 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.296 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.296 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:06.296 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789261+0000", 2020-06-12T02:39:06.297 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2679999999999999e-06 2020-06-12T02:39:06.297 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.297 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.297 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:06.297 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789265+0000", 2020-06-12T02:39:06.298 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6039999999999998e-06 2020-06-12T02:39:06.299 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.299 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.299 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:06.300 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789271+0000", 2020-06-12T02:39:06.300 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4219999999999999e-06 2020-06-12T02:39:06.300 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.300 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.300 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:06.301 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789274+0000", 2020-06-12T02:39:06.301 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013034463 2020-06-12T02:39:06.301 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.301 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.301 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:06.302 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.802308+0000", 2020-06-12T02:39:06.302 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8862000000000005e-05 2020-06-12T02:39:06.302 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.303 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.303 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:06.303 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.802387+0000", 2020-06-12T02:39:06.303 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019102297000000001 2020-06-12T02:39:06.303 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.304 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.304 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:06.304 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.821489+0000", 2020-06-12T02:39:06.304 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8838999999999996e-05 2020-06-12T02:39:06.304 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.305 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.305 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:06.305 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.821558+0000", 2020-06-12T02:39:06.305 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002462615 2020-06-12T02:39:06.305 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.305 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.306 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:06.306 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.824021+0000", 2020-06-12T02:39:06.306 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.034785745 2020-06-12T02:39:06.306 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:06.306 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:06.307 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:06.307 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.307 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.307 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:15005 3.ds0 3:b3445631:::smithi10729407-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e59)", 2020-06-12T02:39:06.307 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:32.192302+0000", 2020-06-12T02:39:06.308 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 33.968574173999997, 2020-06-12T02:39:06.308 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024128666, 2020-06-12T02:39:06.308 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:06.308 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:06.309 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:06.309 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:06.309 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:06.310 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 15005 2020-06-12T02:39:06.310 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.310 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:06.310 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.310 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:06.311 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192302+0000", 2020-06-12T02:39:06.311 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:06.311 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.311 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.311 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:06.311 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192302+0000", 2020-06-12T02:39:06.312 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3415999999999999e-05 2020-06-12T02:39:06.312 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.312 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.312 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:06.312 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192346+0000", 2020-06-12T02:39:06.313 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.93e-06 2020-06-12T02:39:06.313 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.313 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.313 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:06.314 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192352+0000", 2020-06-12T02:39:06.314 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3397000000000001e-05 2020-06-12T02:39:06.314 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.314 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.314 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:06.315 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192365+0000", 2020-06-12T02:39:06.315 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3950000000000003e-06 2020-06-12T02:39:06.315 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.315 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.315 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:06.316 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192369+0000", 2020-06-12T02:39:06.316 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010208887999999999 2020-06-12T02:39:06.316 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.316 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.316 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:06.316 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.202578+0000", 2020-06-12T02:39:06.317 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000105611 2020-06-12T02:39:06.317 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.317 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.317 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:06.317 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.202684+0000", 2020-06-12T02:39:06.318 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0091288959999999992 2020-06-12T02:39:06.318 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.318 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.318 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:06.319 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.211813+0000", 2020-06-12T02:39:06.319 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000101798 2020-06-12T02:39:06.319 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.320 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.320 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:06.321 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.211914+0000", 2020-06-12T02:39:06.321 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045163349999999998 2020-06-12T02:39:06.321 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.321 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.322 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:06.322 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.216431+0000", 2020-06-12T02:39:06.322 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024128666 2020-06-12T02:39:06.322 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:06.322 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:06.323 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:06.323 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.323 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.323 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:16810 3.ds0 3:b983fd28:::smithi10729407-1:head [append 0~524288] snapc 0=[] ondisk+write+known_if_redirected e61)", 2020-06-12T02:39:06.324 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:37.642363+0000", 2020-06-12T02:39:06.324 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 28.518512919999999, 2020-06-12T02:39:06.324 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022969679, 2020-06-12T02:39:06.324 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:06.324 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:06.325 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:06.325 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:06.325 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:06.325 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 16810 2020-06-12T02:39:06.326 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.326 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:06.326 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.326 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:06.326 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.642363+0000", 2020-06-12T02:39:06.327 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:06.327 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.327 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.327 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:06.327 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.642363+0000", 2020-06-12T02:39:06.328 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00069050800000000005 2020-06-12T02:39:06.328 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.328 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.328 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:06.328 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.643054+0000", 2020-06-12T02:39:06.329 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8220000000000007e-06 2020-06-12T02:39:06.329 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.329 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.329 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:06.329 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.643062+0000", 2020-06-12T02:39:06.330 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3742e-05 2020-06-12T02:39:06.330 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.330 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.330 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:06.330 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.643075+0000", 2020-06-12T02:39:06.331 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5910000000000001e-06 2020-06-12T02:39:06.331 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.331 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.331 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:06.331 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.643082+0000", 2020-06-12T02:39:06.332 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0064097879999999996 2020-06-12T02:39:06.332 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.332 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.333 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:06.333 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.649492+0000", 2020-06-12T02:39:06.333 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8151999999999994e-05 2020-06-12T02:39:06.333 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.333 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.334 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:06.334 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.649560+0000", 2020-06-12T02:39:06.334 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0020294940000000002 2020-06-12T02:39:06.334 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.334 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.335 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:06.335 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.651590+0000", 2020-06-12T02:39:06.335 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5242999999999997e-05 2020-06-12T02:39:06.335 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.335 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.336 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:06.336 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.651675+0000", 2020-06-12T02:39:06.336 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00069163799999999995 2020-06-12T02:39:06.336 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.336 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.337 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:06.337 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.652366+0000", 2020-06-12T02:39:06.337 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012711642 2020-06-12T02:39:06.337 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.337 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.337 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:06.338 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.665078+0000", 2020-06-12T02:39:06.338 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017519200000000001 2020-06-12T02:39:06.338 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.338 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.338 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:06.339 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.665253+0000", 2020-06-12T02:39:06.339 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9867e-05 2020-06-12T02:39:06.339 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.339 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.339 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:06.340 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.665333+0000", 2020-06-12T02:39:06.340 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022969679 2020-06-12T02:39:06.340 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:06.340 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:06.340 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:06.341 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.341 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.341 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18287 3.ds0 3:b983fd28:::smithi10729407-1:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:06.342 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.975980+0000", 2020-06-12T02:39:06.342 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 24.184896093999999, 2020-06-12T02:39:06.342 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.023612415000000001, 2020-06-12T02:39:06.342 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:06.342 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:06.343 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:06.343 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:06.343 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:06.343 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18287 2020-06-12T02:39:06.343 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.344 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:06.344 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.344 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:06.344 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.975980+0000", 2020-06-12T02:39:06.344 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:06.345 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.345 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.345 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:06.345 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.975980+0000", 2020-06-12T02:39:06.345 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0175999999999999e-05 2020-06-12T02:39:06.346 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.346 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.346 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:06.346 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976070+0000", 2020-06-12T02:39:06.346 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7489999999999998e-06 2020-06-12T02:39:06.347 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.347 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.347 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:06.347 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976078+0000", 2020-06-12T02:39:06.348 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5308999999999999e-05 2020-06-12T02:39:06.348 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.348 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.348 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:06.348 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976093+0000", 2020-06-12T02:39:06.348 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7149999999999997e-06 2020-06-12T02:39:06.349 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.349 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.349 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:06.349 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976098+0000", 2020-06-12T02:39:06.350 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0067110759999999998 2020-06-12T02:39:06.350 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.350 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.350 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:06.351 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.982809+0000", 2020-06-12T02:39:06.351 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011082999999999999 2020-06-12T02:39:06.351 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.351 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.352 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:06.352 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.982920+0000", 2020-06-12T02:39:06.352 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00043570099999999998 2020-06-12T02:39:06.352 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.352 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.352 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:06.353 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.983356+0000", 2020-06-12T02:39:06.353 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00076300800000000002 2020-06-12T02:39:06.353 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.353 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.353 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:06.354 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.984119+0000", 2020-06-12T02:39:06.354 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01538405 2020-06-12T02:39:06.354 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.354 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.354 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:06.355 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.999503+0000", 2020-06-12T02:39:06.355 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9801000000000003e-05 2020-06-12T02:39:06.355 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.355 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.355 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:06.356 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.999593+0000", 2020-06-12T02:39:06.356 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.023612415000000001 2020-06-12T02:39:06.356 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:06.356 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:06.356 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:06.357 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.357 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.357 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18288 3.ds0 3:b983fd28:::smithi10729407-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:39:06.357 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.976151+0000", 2020-06-12T02:39:06.357 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 24.184725171, 2020-06-12T02:39:06.358 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025323907999999999, 2020-06-12T02:39:06.358 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:06.358 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:06.358 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:06.358 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:06.359 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:06.359 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18288 2020-06-12T02:39:06.359 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.360 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:06.360 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.360 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:06.360 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976151+0000", 2020-06-12T02:39:06.360 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:06.361 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.361 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.361 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:06.361 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976151+0000", 2020-06-12T02:39:06.361 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5615000000000002e-05 2020-06-12T02:39:06.361 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.362 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.362 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:06.362 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976207+0000", 2020-06-12T02:39:06.362 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6399999999999997e-06 2020-06-12T02:39:06.362 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.363 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.363 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:06.363 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976214+0000", 2020-06-12T02:39:06.363 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4823e-05 2020-06-12T02:39:06.363 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.364 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.364 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:06.364 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976229+0000", 2020-06-12T02:39:06.364 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1519999999999997e-06 2020-06-12T02:39:06.364 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.365 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.365 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:06.365 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976235+0000", 2020-06-12T02:39:06.365 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0077693090000000003 2020-06-12T02:39:06.365 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.366 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.366 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:06.366 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.984005+0000", 2020-06-12T02:39:06.366 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000101221 2020-06-12T02:39:06.366 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.367 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.367 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:06.367 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.984106+0000", 2020-06-12T02:39:06.367 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016277713999999999 2020-06-12T02:39:06.367 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.367 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.368 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:06.368 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.000384+0000", 2020-06-12T02:39:06.368 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7938999999999999e-05 2020-06-12T02:39:06.368 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.368 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.369 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:06.369 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.000482+0000", 2020-06-12T02:39:06.369 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00099349500000000006 2020-06-12T02:39:06.369 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.370 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.370 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:06.370 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.001475+0000", 2020-06-12T02:39:06.370 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025323907999999999 2020-06-12T02:39:06.370 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:06.371 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:06.371 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:06.371 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.371 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.371 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18307 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [append 0~507904] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:06.372 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.056119+0000", 2020-06-12T02:39:06.372 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 24.104757287999998, 2020-06-12T02:39:06.372 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025879849, 2020-06-12T02:39:06.372 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:06.372 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:06.373 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:06.373 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:06.373 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:06.373 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18307 2020-06-12T02:39:06.373 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.374 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:06.374 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.374 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:06.374 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.056119+0000", 2020-06-12T02:39:06.374 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:06.375 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.375 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.375 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:06.375 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.056119+0000", 2020-06-12T02:39:06.375 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00052484099999999996 2020-06-12T02:39:06.376 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.376 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.376 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:06.376 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.056644+0000", 2020-06-12T02:39:06.376 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7149999999999997e-06 2020-06-12T02:39:06.377 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.377 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.377 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:06.377 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.056649+0000", 2020-06-12T02:39:06.377 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0046e-05 2020-06-12T02:39:06.378 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.378 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.378 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:06.378 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.056659+0000", 2020-06-12T02:39:06.378 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.848e-06 2020-06-12T02:39:06.379 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.379 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.379 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:06.379 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.056663+0000", 2020-06-12T02:39:06.379 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024537029999999998 2020-06-12T02:39:06.380 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.380 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.380 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:06.380 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.059117+0000", 2020-06-12T02:39:06.380 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000121261 2020-06-12T02:39:06.381 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.381 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.381 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:06.381 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.059238+0000", 2020-06-12T02:39:06.382 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001916484 2020-06-12T02:39:06.382 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.382 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.382 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:06.382 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.061155+0000", 2020-06-12T02:39:06.383 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9120000000000001e-05 2020-06-12T02:39:06.383 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.383 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.383 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:06.383 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.061234+0000", 2020-06-12T02:39:06.384 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00060383599999999996 2020-06-12T02:39:06.384 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.384 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.384 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:06.385 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.061838+0000", 2020-06-12T02:39:06.385 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010259862999999999 2020-06-12T02:39:06.385 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.385 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.386 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:06.386 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.072098+0000", 2020-06-12T02:39:06.386 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0098176419999999997 2020-06-12T02:39:06.386 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.386 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.387 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:06.387 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.081915+0000", 2020-06-12T02:39:06.387 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3490000000000002e-05 2020-06-12T02:39:06.387 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.387 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.388 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:06.388 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.081999+0000", 2020-06-12T02:39:06.388 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025879849 2020-06-12T02:39:06.388 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:06.388 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:06.389 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:06.389 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.389 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.389 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18311 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:39:06.389 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.063458+0000", 2020-06-12T02:39:06.390 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 24.097418652000002, 2020-06-12T02:39:06.390 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.035342152000000002, 2020-06-12T02:39:06.390 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:06.390 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:06.391 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:06.391 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:06.391 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:06.391 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18311 2020-06-12T02:39:06.391 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.395 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:06.395 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.395 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:06.395 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.063458+0000", 2020-06-12T02:39:06.396 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:06.396 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.396 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.396 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:06.397 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.063458+0000", 2020-06-12T02:39:06.397 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3773000000000003e-05 2020-06-12T02:39:06.397 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.397 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.397 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:06.398 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.063501+0000", 2020-06-12T02:39:06.398 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0780000000000001e-06 2020-06-12T02:39:06.398 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.398 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.398 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:06.399 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.063508+0000", 2020-06-12T02:39:06.399 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3020000000000001e-05 2020-06-12T02:39:06.399 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.399 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.400 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:06.400 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.063521+0000", 2020-06-12T02:39:06.400 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4839999999999998e-06 2020-06-12T02:39:06.400 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.400 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.401 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:06.401 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.063525+0000", 2020-06-12T02:39:06.401 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0087769890000000007 2020-06-12T02:39:06.401 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.401 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.402 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:06.402 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.072302+0000", 2020-06-12T02:39:06.402 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3121000000000005e-05 2020-06-12T02:39:06.402 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.402 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.403 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:06.403 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.072395+0000", 2020-06-12T02:39:06.403 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010158355000000001 2020-06-12T02:39:06.403 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.403 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.403 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:06.404 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.082554+0000", 2020-06-12T02:39:06.404 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5016000000000001e-05 2020-06-12T02:39:06.404 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.406 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.406 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:06.406 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.082629+0000", 2020-06-12T02:39:06.406 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016171316000000002 2020-06-12T02:39:06.407 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.407 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.407 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:06.407 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.098800+0000", 2020-06-12T02:39:06.407 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.035342152000000002 2020-06-12T02:39:06.408 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:06.408 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:06.408 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:06.408 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.408 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.409 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18313 3.fs0 3:f83c3dea:::smithi10729407-4:head [append 0~442368] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:06.409 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.070249+0000", 2020-06-12T02:39:06.409 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 24.090627375, 2020-06-12T02:39:06.409 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028853489, 2020-06-12T02:39:06.409 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:06.409 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:06.410 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:06.410 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:06.410 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:06.410 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18313 2020-06-12T02:39:06.410 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.411 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:06.411 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.411 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:06.411 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.070249+0000", 2020-06-12T02:39:06.411 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:06.412 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.412 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.412 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:06.412 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.070249+0000", 2020-06-12T02:39:06.412 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00046159400000000001 2020-06-12T02:39:06.412 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.413 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.413 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:06.413 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.070711+0000", 2020-06-12T02:39:06.413 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2539999999999992e-06 2020-06-12T02:39:06.413 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.414 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.414 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:06.414 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.070719+0000", 2020-06-12T02:39:06.414 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3297e-05 2020-06-12T02:39:06.414 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.415 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.415 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:06.415 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.070732+0000", 2020-06-12T02:39:06.415 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0559999999999999e-06 2020-06-12T02:39:06.415 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.415 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.416 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:06.416 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.070739+0000", 2020-06-12T02:39:06.416 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0052673319999999996 2020-06-12T02:39:06.416 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.416 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.417 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:06.417 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.076006+0000", 2020-06-12T02:39:06.417 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6925999999999997e-05 2020-06-12T02:39:06.417 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.417 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.417 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:06.418 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.076063+0000", 2020-06-12T02:39:06.418 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0076712209999999998 2020-06-12T02:39:06.418 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.418 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.418 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:06.419 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.083735+0000", 2020-06-12T02:39:06.419 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8089000000000002e-05 2020-06-12T02:39:06.419 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.419 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.419 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:06.420 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.083833+0000", 2020-06-12T02:39:06.420 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054620600000000004 2020-06-12T02:39:06.420 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.420 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.420 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:06.420 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.084379+0000", 2020-06-12T02:39:06.421 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013940945999999999 2020-06-12T02:39:06.421 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.421 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.421 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:06.421 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.098320+0000", 2020-06-12T02:39:06.422 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00072638599999999998 2020-06-12T02:39:06.422 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.422 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.422 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:06.422 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.099046+0000", 2020-06-12T02:39:06.423 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6181999999999998e-05 2020-06-12T02:39:06.423 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.423 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.423 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:06.423 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.099102+0000", 2020-06-12T02:39:06.423 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028853489 2020-06-12T02:39:06.424 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:06.424 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:06.424 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:06.424 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.424 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.425 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18314 3.fs0 3:f83c3dea:::smithi10729407-4:head [append 0~573440] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:06.425 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.074005+0000", 2020-06-12T02:39:06.425 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 24.086871356, 2020-06-12T02:39:06.425 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031442581999999997, 2020-06-12T02:39:06.425 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:06.426 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:06.426 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:06.426 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:06.426 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:06.426 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18314 2020-06-12T02:39:06.426 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.427 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:06.427 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.427 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:06.427 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.074005+0000", 2020-06-12T02:39:06.427 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:06.428 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.428 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.428 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:06.428 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.074005+0000", 2020-06-12T02:39:06.428 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00056762000000000002 2020-06-12T02:39:06.428 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.429 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.429 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:06.429 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.074573+0000", 2020-06-12T02:39:06.429 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5400000000000003e-06 2020-06-12T02:39:06.429 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.430 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.430 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:06.430 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.074578+0000", 2020-06-12T02:39:06.431 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1090000000000004e-06 2020-06-12T02:39:06.431 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.431 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.431 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:06.431 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.074587+0000", 2020-06-12T02:39:06.432 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4039999999999997e-06 2020-06-12T02:39:06.432 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.432 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.432 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:06.432 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.074592+0000", 2020-06-12T02:39:06.432 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001538031 2020-06-12T02:39:06.433 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.433 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.433 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:06.433 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.076130+0000", 2020-06-12T02:39:06.433 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5117000000000002e-05 2020-06-12T02:39:06.434 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.434 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.434 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:06.434 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.076175+0000", 2020-06-12T02:39:06.434 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011822262 2020-06-12T02:39:06.434 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.435 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.435 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:06.435 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.087997+0000", 2020-06-12T02:39:06.435 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000112318 2020-06-12T02:39:06.435 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.436 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.436 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:06.436 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.088109+0000", 2020-06-12T02:39:06.436 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00072351100000000003 2020-06-12T02:39:06.436 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.436 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.437 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:06.437 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.088833+0000", 2020-06-12T02:39:06.438 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010307874999999999 2020-06-12T02:39:06.438 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.438 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.439 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:06.439 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.099141+0000", 2020-06-12T02:39:06.439 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.006229605 2020-06-12T02:39:06.439 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.440 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.440 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:06.440 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.105370+0000", 2020-06-12T02:39:06.440 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7189999999999998e-05 2020-06-12T02:39:06.440 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.441 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.441 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:06.441 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.105448+0000", 2020-06-12T02:39:06.441 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031442581999999997 2020-06-12T02:39:06.441 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:06.442 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:06.442 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:06.442 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.443 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.443 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18315 3.fs0 3:f83c3dea:::smithi10729407-4:head [append 0~589824] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:06.443 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.077724+0000", 2020-06-12T02:39:06.443 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 24.083152497, 2020-06-12T02:39:06.443 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.027939051999999999, 2020-06-12T02:39:06.444 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:06.444 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:06.444 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:06.444 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:06.444 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:06.444 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18315 2020-06-12T02:39:06.445 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.445 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:06.445 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.445 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:06.445 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.077724+0000", 2020-06-12T02:39:06.446 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:06.446 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.446 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.446 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:06.446 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.077724+0000", 2020-06-12T02:39:06.447 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00093652299999999998 2020-06-12T02:39:06.447 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.447 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.447 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:06.447 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078660+0000", 2020-06-12T02:39:06.447 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1389999999999995e-06 2020-06-12T02:39:06.448 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.448 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.448 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:06.448 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078669+0000", 2020-06-12T02:39:06.448 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3691e-05 2020-06-12T02:39:06.449 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.449 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.449 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:06.449 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078682+0000", 2020-06-12T02:39:06.449 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0100000000000001e-06 2020-06-12T02:39:06.449 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.450 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.450 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:06.450 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078688+0000", 2020-06-12T02:39:06.450 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1209999999999999e-05 2020-06-12T02:39:06.450 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.451 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.451 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:06.451 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078739+0000", 2020-06-12T02:39:06.451 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000102735 2020-06-12T02:39:06.451 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.451 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.452 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:06.452 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078842+0000", 2020-06-12T02:39:06.452 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01321815 2020-06-12T02:39:06.452 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.452 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.453 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:06.453 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.092060+0000", 2020-06-12T02:39:06.453 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00038608499999999999 2020-06-12T02:39:06.453 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.453 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.454 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:06.454 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.092446+0000", 2020-06-12T02:39:06.454 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00056703299999999999 2020-06-12T02:39:06.454 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.455 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.455 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:06.455 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.093013+0000", 2020-06-12T02:39:06.455 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0061289530000000004 2020-06-12T02:39:06.455 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.455 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.456 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:06.456 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.099142+0000", 2020-06-12T02:39:06.456 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0064547700000000003 2020-06-12T02:39:06.456 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.456 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.457 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:06.457 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.105597+0000", 2020-06-12T02:39:06.457 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5753000000000006e-05 2020-06-12T02:39:06.457 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.457 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.458 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:06.458 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.105663+0000", 2020-06-12T02:39:06.458 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.027939051999999999 2020-06-12T02:39:06.458 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:06.458 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:06.459 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:06.459 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.459 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.459 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18316 3.fs0 3:f83c3dea:::smithi10729407-4:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:06.460 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.078752+0000", 2020-06-12T02:39:06.461 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 24.082124725, 2020-06-12T02:39:06.462 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.029108709999999999, 2020-06-12T02:39:06.462 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:06.462 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:06.462 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:06.463 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:06.463 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:06.463 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18316 2020-06-12T02:39:06.463 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.463 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:06.464 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.464 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:06.464 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078752+0000", 2020-06-12T02:39:06.464 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:06.465 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.465 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.465 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:06.465 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078752+0000", 2020-06-12T02:39:06.466 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8967000000000004e-05 2020-06-12T02:39:06.466 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.466 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.466 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:06.466 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078841+0000", 2020-06-12T02:39:06.466 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3309999999999998e-06 2020-06-12T02:39:06.467 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.467 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.467 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:06.468 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078848+0000", 2020-06-12T02:39:06.468 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.291e-05 2020-06-12T02:39:06.468 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.469 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.469 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:06.469 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078861+0000", 2020-06-12T02:39:06.469 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2800000000000003e-06 2020-06-12T02:39:06.470 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.470 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.470 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:06.470 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078866+0000", 2020-06-12T02:39:06.471 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8543e-05 2020-06-12T02:39:06.471 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.471 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.471 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:06.472 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078905+0000", 2020-06-12T02:39:06.472 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9198999999999998e-05 2020-06-12T02:39:06.472 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.472 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.472 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:06.473 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078994+0000", 2020-06-12T02:39:06.473 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01781437 2020-06-12T02:39:06.473 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.473 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.473 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:06.474 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.096808+0000", 2020-06-12T02:39:06.474 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000115975 2020-06-12T02:39:06.474 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.474 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.475 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:06.475 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.096924+0000", 2020-06-12T02:39:06.475 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00049986300000000002 2020-06-12T02:39:06.475 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.476 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.476 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:06.476 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.097424+0000", 2020-06-12T02:39:06.476 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0017191089999999999 2020-06-12T02:39:06.476 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.477 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.477 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:06.477 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.099143+0000", 2020-06-12T02:39:06.477 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0086177909999999996 2020-06-12T02:39:06.477 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.478 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.478 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:06.478 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.107761+0000", 2020-06-12T02:39:06.479 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9371999999999997e-05 2020-06-12T02:39:06.479 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.479 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.479 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:06.480 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.107860+0000", 2020-06-12T02:39:06.480 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.029108709999999999 2020-06-12T02:39:06.480 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:06.480 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:06.481 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:06.481 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.481 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.481 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18317 3.fs0 3:f83c3dea:::smithi10729407-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:39:06.482 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.078920+0000", 2020-06-12T02:39:06.482 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 24.081956755, 2020-06-12T02:39:06.482 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031356964000000001, 2020-06-12T02:39:06.482 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:06.482 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:06.483 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:06.483 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:06.483 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:06.483 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18317 2020-06-12T02:39:06.483 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.484 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:06.484 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.484 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:06.484 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078920+0000", 2020-06-12T02:39:06.484 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:06.485 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.485 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.485 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:06.485 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078920+0000", 2020-06-12T02:39:06.485 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6454000000000001e-05 2020-06-12T02:39:06.486 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.486 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.486 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:06.486 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078946+0000", 2020-06-12T02:39:06.486 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8160000000000004e-06 2020-06-12T02:39:06.487 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.487 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.487 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:06.487 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078950+0000", 2020-06-12T02:39:06.487 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6090000000000003e-06 2020-06-12T02:39:06.488 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.488 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.488 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:06.489 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078957+0000", 2020-06-12T02:39:06.489 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8380000000000002e-06 2020-06-12T02:39:06.489 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.490 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.490 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:06.490 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078960+0000", 2020-06-12T02:39:06.490 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8400000000000003e-05 2020-06-12T02:39:06.490 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.491 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.491 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:06.491 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.079019+0000", 2020-06-12T02:39:06.491 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6750999999999998e-05 2020-06-12T02:39:06.491 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.492 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.492 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:06.492 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.079085+0000", 2020-06-12T02:39:06.492 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019272036999999999 2020-06-12T02:39:06.492 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.493 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.493 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:06.493 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.098357+0000", 2020-06-12T02:39:06.493 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4322999999999998e-05 2020-06-12T02:39:06.494 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.494 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.494 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:06.494 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.098452+0000", 2020-06-12T02:39:06.495 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010578025 2020-06-12T02:39:06.495 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.495 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.495 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:06.495 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.109030+0000", 2020-06-12T02:39:06.495 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010395099999999999 2020-06-12T02:39:06.496 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.496 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.496 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:06.496 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.109134+0000", 2020-06-12T02:39:06.497 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00114276 2020-06-12T02:39:06.497 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:06.497 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:06.498 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:06.498 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.110277+0000", 2020-06-12T02:39:06.498 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031356964000000001 2020-06-12T02:39:06.498 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:06.499 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:06.499 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:06.499 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:06.500 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:06.500 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:39:06.523 INFO:tasks.rados.rados.0.smithi107.stdout:9678: writing smithi10729407-13 from 2342912 to 2818048 tid 1 2020-06-12T02:39:06.524 INFO:tasks.rados.rados.0.smithi107.stdout:9678: writing smithi10729407-13 from 2818048 to 3497984 tid 2 2020-06-12T02:39:06.524 INFO:tasks.rados.rados.0.smithi107.stdout:9678: writing smithi10729407-13 from 3497984 to 3620864 tid 3 2020-06-12T02:39:06.525 INFO:tasks.rados.rados.0.smithi107.stdout:9679: delete oid 2 current snap is 0 2020-06-12T02:39:06.525 INFO:tasks.rados.rados.0.smithi107.stdout:9676: finishing write tid 2 to smithi10729407-19 2020-06-12T02:39:06.525 INFO:tasks.rados.rados.0.smithi107.stdout:9676: finishing write tid 3 to smithi10729407-19 2020-06-12T02:39:06.525 INFO:tasks.rados.rados.0.smithi107.stdout:9676: finishing write tid 5 to smithi10729407-19 2020-06-12T02:39:06.526 INFO:tasks.rados.rados.0.smithi107.stdout:9676: finishing write tid 6 to smithi10729407-19 2020-06-12T02:39:06.526 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1787 (ObjNum 9675 snap 0 seq_num 9675) dirty exists 2020-06-12T02:39:06.526 INFO:tasks.rados.rados.0.smithi107.stdout:9676: left oid 19 (ObjNum 9675 snap 0 seq_num 9675) 2020-06-12T02:39:06.526 INFO:tasks.rados.rados.0.smithi107.stdout:9676: done (2 left) 2020-06-12T02:39:06.526 INFO:tasks.rados.rados.0.smithi107.stdout:9679: done (1 left) 2020-06-12T02:39:06.527 INFO:tasks.rados.rados.0.smithi107.stdout:9680: delete oid 18 current snap is 0 2020-06-12T02:39:06.528 INFO:tasks.rados.rados.0.smithi107.stdout:9678: finishing write tid 1 to smithi10729407-13 2020-06-12T02:39:06.528 INFO:tasks.rados.rados.0.smithi107.stdout:9678: finishing write tid 2 to smithi10729407-13 2020-06-12T02:39:06.528 INFO:tasks.rados.rados.0.smithi107.stdout:9678: finishing write tid 3 to smithi10729407-13 2020-06-12T02:39:06.528 INFO:tasks.rados.rados.0.smithi107.stdout:9678: finishing write tid 5 to smithi10729407-13 2020-06-12T02:39:06.529 INFO:tasks.rados.rados.0.smithi107.stdout:9680: done (1 left) 2020-06-12T02:39:06.529 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:39:06.529 INFO:tasks.rados.rados.0.smithi107.stdout:9681: seq_num 9680 ranges {540672=524288,1064960=778240,1843200=24576} 2020-06-12T02:39:06.529 INFO:tasks.rados.rados.0.smithi107.stdout:9681: writing smithi10729407-20 from 540672 to 1064960 tid 1 2020-06-12T02:39:06.529 INFO:tasks.rados.rados.0.smithi107.stdout:9681: writing smithi10729407-20 from 1064960 to 1843200 tid 2 2020-06-12T02:39:06.530 INFO:tasks.rados.rados.0.smithi107.stdout:9681: writing smithi10729407-20 from 1843200 to 1867776 tid 3 2020-06-12T02:39:06.530 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:39:06.530 INFO:tasks.rados.rados.0.smithi107.stdout:9682: seq_num 9681 ranges {1622016=540672,2162688=663552,2826240=139264} 2020-06-12T02:39:06.531 INFO:tasks.rados.rados.0.smithi107.stdout:9682: writing smithi10729407-1 from 1622016 to 2162688 tid 1 2020-06-12T02:39:06.531 INFO:tasks.rados.rados.0.smithi107.stdout:9682: writing smithi10729407-1 from 2162688 to 2826240 tid 2 2020-06-12T02:39:06.531 INFO:tasks.rados.rados.0.smithi107.stdout:9682: writing smithi10729407-1 from 2826240 to 2965504 tid 3 2020-06-12T02:39:06.531 INFO:tasks.rados.rados.0.smithi107.stdout:9683: delete oid 12 current snap is 0 2020-06-12T02:39:06.531 INFO:tasks.rados.rados.0.smithi107.stdout:9678: finishing write tid 6 to smithi10729407-13 2020-06-12T02:39:06.532 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1696 (ObjNum 9677 snap 0 seq_num 9677) dirty exists 2020-06-12T02:39:06.532 INFO:tasks.rados.rados.0.smithi107.stdout:9678: left oid 13 (ObjNum 9677 snap 0 seq_num 9677) 2020-06-12T02:39:06.532 INFO:tasks.rados.rados.0.smithi107.stdout:9681: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:06.532 INFO:tasks.rados.rados.0.smithi107.stdout:9681: finishing write tid 2 to smithi10729407-20 2020-06-12T02:39:06.532 INFO:tasks.rados.rados.0.smithi107.stdout:9681: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:06.533 INFO:tasks.rados.rados.0.smithi107.stdout:9681: finishing write tid 5 to smithi10729407-20 2020-06-12T02:39:06.533 INFO:tasks.rados.rados.0.smithi107.stdout:9681: finishing write tid 6 to smithi10729407-20 2020-06-12T02:39:06.533 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 722 (ObjNum 9680 snap 0 seq_num 9680) dirty exists 2020-06-12T02:39:06.534 INFO:tasks.rados.rados.0.smithi107.stdout:9681: left oid 20 (ObjNum 9680 snap 0 seq_num 9680) 2020-06-12T02:39:06.534 INFO:tasks.rados.rados.0.smithi107.stdout:9682: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:06.534 INFO:tasks.rados.rados.0.smithi107.stdout:9678: done (3 left) 2020-06-12T02:39:06.534 INFO:tasks.rados.rados.0.smithi107.stdout:9681: done (2 left) 2020-06-12T02:39:06.535 INFO:tasks.rados.rados.0.smithi107.stdout:9683: done (1 left) 2020-06-12T02:39:06.535 INFO:tasks.rados.rados.0.smithi107.stdout:9684: delete oid 5 current snap is 0 2020-06-12T02:39:06.535 INFO:tasks.rados.rados.0.smithi107.stdout:9684: done (1 left) 2020-06-12T02:39:06.535 INFO:tasks.rados.rados.0.smithi107.stdout:9685: delete oid 17 current snap is 0 2020-06-12T02:39:06.536 INFO:tasks.rados.rados.0.smithi107.stdout:9682: finishing write tid 2 to smithi10729407-1 2020-06-12T02:39:06.536 INFO:tasks.rados.rados.0.smithi107.stdout:9685: done (1 left) 2020-06-12T02:39:06.536 INFO:tasks.rados.rados.0.smithi107.stdout:9686: delete oid 4 current snap is 0 2020-06-12T02:39:06.536 INFO:tasks.rados.rados.0.smithi107.stdout:9682: finishing write tid 3 to smithi10729407-1 2020-06-12T02:39:06.536 INFO:tasks.rados.rados.0.smithi107.stdout:9682: finishing write tid 5 to smithi10729407-1 2020-06-12T02:39:06.537 INFO:tasks.rados.rados.0.smithi107.stdout:9682: finishing write tid 6 to smithi10729407-1 2020-06-12T02:39:06.537 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1741 (ObjNum 9681 snap 0 seq_num 9681) dirty exists 2020-06-12T02:39:06.537 INFO:tasks.rados.rados.0.smithi107.stdout:9682: left oid 1 (ObjNum 9681 snap 0 seq_num 9681) 2020-06-12T02:39:06.537 INFO:tasks.rados.rados.0.smithi107.stdout:9682: done (1 left) 2020-06-12T02:39:06.537 INFO:tasks.rados.rados.0.smithi107.stdout:9686: done (0 left) 2020-06-12T02:39:06.538 INFO:tasks.rados.rados.0.smithi107.stdout:9687: delete oid 19 current snap is 0 2020-06-12T02:39:06.538 INFO:tasks.rados.rados.0.smithi107.stdout:9687: done (0 left) 2020-06-12T02:39:06.538 INFO:tasks.rados.rados.0.smithi107.stdout:9688: delete oid 9 current snap is 0 2020-06-12T02:39:06.538 INFO:tasks.rados.rados.0.smithi107.stdout:9688: done (0 left) 2020-06-12T02:39:06.539 INFO:tasks.rados.rados.0.smithi107.stdout:9689: delete oid 16 current snap is 0 2020-06-12T02:39:06.539 INFO:tasks.rados.rados.0.smithi107.stdout:9689: done (0 left) 2020-06-12T02:39:06.539 INFO:tasks.rados.rados.0.smithi107.stdout:9690: delete oid 2 current snap is 0 2020-06-12T02:39:06.539 INFO:tasks.rados.rados.0.smithi107.stdout:9690: done (0 left) 2020-06-12T02:39:06.539 INFO:tasks.rados.rados.0.smithi107.stdout:9691: delete oid 3 current snap is 0 2020-06-12T02:39:06.540 INFO:tasks.rados.rados.0.smithi107.stdout:9691: done (0 left) 2020-06-12T02:39:06.540 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:39:06.540 INFO:tasks.rados.rados.0.smithi107.stdout:9692: seq_num 9691 ranges {0=712704,712704=761856,1474560=40960} 2020-06-12T02:39:06.541 INFO:tasks.rados.rados.0.smithi107.stdout:9692: writing smithi10729407-19 from 0 to 712704 tid 1 2020-06-12T02:39:06.541 INFO:tasks.rados.rados.0.smithi107.stdout:9692: writing smithi10729407-19 from 712704 to 1474560 tid 2 2020-06-12T02:39:06.541 INFO:tasks.rados.rados.0.smithi107.stdout:9692: writing smithi10729407-19 from 1474560 to 1515520 tid 3 2020-06-12T02:39:06.541 INFO:tasks.rados.rados.0.smithi107.stdout:9693: delete oid 5 current snap is 0 2020-06-12T02:39:06.541 INFO:tasks.rados.rados.0.smithi107.stdout:9693: done (1 left) 2020-06-12T02:39:06.542 INFO:tasks.rados.rados.0.smithi107.stdout:9694: delete oid 8 current snap is 0 2020-06-12T02:39:06.543 INFO:tasks.rados.rados.0.smithi107.stdout:9694: done (1 left) 2020-06-12T02:39:06.543 INFO:tasks.rados.rados.0.smithi107.stdout:9695: delete oid 17 current snap is 0 2020-06-12T02:39:06.543 INFO:tasks.rados.rados.0.smithi107.stdout:9692: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:06.543 INFO:tasks.rados.rados.0.smithi107.stdout:9692: finishing write tid 2 to smithi10729407-19 2020-06-12T02:39:06.543 INFO:tasks.rados.rados.0.smithi107.stdout:9692: finishing write tid 3 to smithi10729407-19 2020-06-12T02:39:06.544 INFO:tasks.rados.rados.0.smithi107.stdout:9692: finishing write tid 5 to smithi10729407-19 2020-06-12T02:39:06.544 INFO:tasks.rados.rados.0.smithi107.stdout:9695: done (1 left) 2020-06-12T02:39:06.544 INFO:tasks.rados.rados.0.smithi107.stdout:9696: delete oid 4 current snap is 0 2020-06-12T02:39:06.544 INFO:tasks.rados.rados.0.smithi107.stdout:9692: finishing write tid 6 to smithi10729407-19 2020-06-12T02:39:06.544 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1793 (ObjNum 9691 snap 0 seq_num 9691) dirty exists 2020-06-12T02:39:06.545 INFO:tasks.rados.rados.0.smithi107.stdout:9692: left oid 19 (ObjNum 9691 snap 0 seq_num 9691) 2020-06-12T02:39:06.545 INFO:tasks.rados.rados.0.smithi107.stdout:9692: done (1 left) 2020-06-12T02:39:06.545 INFO:tasks.rados.rados.0.smithi107.stdout:9696: done (0 left) 2020-06-12T02:39:06.545 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:39:06.545 INFO:tasks.rados.rados.0.smithi107.stdout:9697: seq_num 9696 ranges {0=794624,794624=630784,1425408=172032} 2020-06-12T02:39:06.546 INFO:tasks.rados.rados.0.smithi107.stdout:9697: writing smithi10729407-10 from 0 to 794624 tid 1 2020-06-12T02:39:06.546 INFO:tasks.rados.rados.0.smithi107.stdout:9697: writing smithi10729407-10 from 794624 to 1425408 tid 2 2020-06-12T02:39:06.546 INFO:tasks.rados.rados.0.smithi107.stdout:9697: writing smithi10729407-10 from 1425408 to 1597440 tid 3 2020-06-12T02:39:06.546 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:39:06.547 INFO:tasks.rados.rados.0.smithi107.stdout:9698: seq_num 9697 ranges {638976=409600,1048576=516096,1564672=688128} 2020-06-12T02:39:06.547 INFO:tasks.rados.rados.0.smithi107.stdout:9698: writing smithi10729407-6 from 638976 to 1048576 tid 1 2020-06-12T02:39:06.547 INFO:tasks.rados.rados.0.smithi107.stdout:9698: writing smithi10729407-6 from 1048576 to 1564672 tid 2 2020-06-12T02:39:06.547 INFO:tasks.rados.rados.0.smithi107.stdout:9698: writing smithi10729407-6 from 1564672 to 2252800 tid 3 2020-06-12T02:39:06.548 INFO:tasks.rados.rados.0.smithi107.stdout:9699: delete oid 20 current snap is 0 2020-06-12T02:39:06.548 INFO:tasks.rados.rados.0.smithi107.stdout:9697: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:06.548 INFO:tasks.rados.rados.0.smithi107.stdout:9697: finishing write tid 2 to smithi10729407-10 2020-06-12T02:39:06.548 INFO:tasks.rados.rados.0.smithi107.stdout:9697: finishing write tid 3 to smithi10729407-10 2020-06-12T02:39:06.548 INFO:tasks.rados.rados.0.smithi107.stdout:9697: finishing write tid 5 to smithi10729407-10 2020-06-12T02:39:06.549 INFO:tasks.rados.rados.0.smithi107.stdout:9697: finishing write tid 6 to smithi10729407-10 2020-06-12T02:39:06.549 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 691 (ObjNum 9696 snap 0 seq_num 9696) dirty exists 2020-06-12T02:39:06.549 INFO:tasks.rados.rados.0.smithi107.stdout:9697: left oid 10 (ObjNum 9696 snap 0 seq_num 9696) 2020-06-12T02:39:06.550 INFO:tasks.rados.rados.0.smithi107.stdout:9697: done (2 left) 2020-06-12T02:39:06.550 INFO:tasks.rados.rados.0.smithi107.stdout:9699: done (1 left) 2020-06-12T02:39:06.550 INFO:tasks.rados.rados.0.smithi107.stdout:9700: delete oid 2 current snap is 0 2020-06-12T02:39:06.550 INFO:tasks.rados.rados.0.smithi107.stdout:9700: done (1 left) 2020-06-12T02:39:06.550 INFO:tasks.rados.rados.0.smithi107.stdout:9701: delete oid 7 current snap is 0 2020-06-12T02:39:06.551 INFO:tasks.rados.rados.0.smithi107.stdout:9698: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:06.551 INFO:tasks.rados.rados.0.smithi107.stdout:9698: finishing write tid 2 to smithi10729407-6 2020-06-12T02:39:06.552 INFO:tasks.rados.rados.0.smithi107.stdout:9698: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:06.552 INFO:tasks.rados.rados.0.smithi107.stdout:9701: done (1 left) 2020-06-12T02:39:06.552 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:39:06.552 INFO:tasks.rados.rados.0.smithi107.stdout:9702: seq_num 9701 ranges {2965504=475136,3440640=499712,3940352=671744,4612096=32768} 2020-06-12T02:39:06.553 INFO:tasks.rados.rados.0.smithi107.stdout:9702: writing smithi10729407-1 from 2965504 to 3440640 tid 1 2020-06-12T02:39:06.553 INFO:tasks.rados.rados.0.smithi107.stdout:9702: writing smithi10729407-1 from 3440640 to 3940352 tid 2 2020-06-12T02:39:06.553 INFO:tasks.rados.rados.0.smithi107.stdout:9702: writing smithi10729407-1 from 3940352 to 4612096 tid 3 2020-06-12T02:39:06.553 INFO:tasks.rados.rados.0.smithi107.stdout:9702: writing smithi10729407-1 from 4612096 to 4644864 tid 4 2020-06-12T02:39:06.554 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:39:06.554 INFO:tasks.rados.rados.0.smithi107.stdout:9703: seq_num 9702 ranges {0=491520,491520=786432,1277952=417792} 2020-06-12T02:39:06.554 INFO:tasks.rados.rados.0.smithi107.stdout:9703: writing smithi10729407-3 from 0 to 491520 tid 1 2020-06-12T02:39:06.554 INFO:tasks.rados.rados.0.smithi107.stdout:9703: writing smithi10729407-3 from 491520 to 1277952 tid 2 2020-06-12T02:39:06.554 INFO:tasks.rados.rados.0.smithi107.stdout:9703: writing smithi10729407-3 from 1277952 to 1695744 tid 3 2020-06-12T02:39:06.555 INFO:tasks.rados.rados.0.smithi107.stdout:9704: delete oid 2 current snap is 0 2020-06-12T02:39:06.555 INFO:tasks.rados.rados.0.smithi107.stdout:9698: finishing write tid 5 to smithi10729407-6 2020-06-12T02:39:06.555 INFO:tasks.rados.rados.0.smithi107.stdout:9698: finishing write tid 6 to smithi10729407-6 2020-06-12T02:39:06.555 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1745 (ObjNum 9697 snap 0 seq_num 9697) dirty exists 2020-06-12T02:39:06.555 INFO:tasks.rados.rados.0.smithi107.stdout:9698: left oid 6 (ObjNum 9697 snap 0 seq_num 9697) 2020-06-12T02:39:06.556 INFO:tasks.rados.rados.0.smithi107.stdout:9702: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:06.556 INFO:tasks.rados.rados.0.smithi107.stdout:9702: finishing write tid 2 to smithi10729407-1 2020-06-12T02:39:06.556 INFO:tasks.rados.rados.0.smithi107.stdout:9702: finishing write tid 3 to smithi10729407-1 2020-06-12T02:39:06.556 INFO:tasks.rados.rados.0.smithi107.stdout:9702: finishing write tid 4 to smithi10729407-1 2020-06-12T02:39:06.556 INFO:tasks.rados.rados.0.smithi107.stdout:9702: finishing write tid 6 to smithi10729407-1 2020-06-12T02:39:06.557 INFO:tasks.rados.rados.0.smithi107.stdout:9703: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:06.557 INFO:tasks.rados.rados.0.smithi107.stdout:9702: finishing write tid 7 to smithi10729407-1 2020-06-12T02:39:06.557 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1750 (ObjNum 9701 snap 0 seq_num 9701) dirty exists 2020-06-12T02:39:06.557 INFO:tasks.rados.rados.0.smithi107.stdout:9702: left oid 1 (ObjNum 9701 snap 0 seq_num 9701) 2020-06-12T02:39:06.557 INFO:tasks.rados.rados.0.smithi107.stdout:9698: done (3 left) 2020-06-12T02:39:06.558 INFO:tasks.rados.rados.0.smithi107.stdout:9702: done (2 left) 2020-06-12T02:39:06.558 INFO:tasks.rados.rados.0.smithi107.stdout:9704: done (1 left) 2020-06-12T02:39:06.558 INFO:tasks.rados.rados.0.smithi107.stdout:9705: delete oid 9 current snap is 0 2020-06-12T02:39:06.558 INFO:tasks.rados.rados.0.smithi107.stdout:9705: done (1 left) 2020-06-12T02:39:06.558 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:39:06.559 INFO:tasks.rados.rados.0.smithi107.stdout:9706: seq_num 9705 ranges {0=548864,548864=434176,983040=655360,1638400=114688} 2020-06-12T02:39:06.559 INFO:tasks.rados.rados.0.smithi107.stdout:9706: writing smithi10729407-15 from 0 to 548864 tid 1 2020-06-12T02:39:06.559 INFO:tasks.rados.rados.0.smithi107.stdout:9706: writing smithi10729407-15 from 548864 to 983040 tid 2 2020-06-12T02:39:06.559 INFO:tasks.rados.rados.0.smithi107.stdout:9706: writing smithi10729407-15 from 983040 to 1638400 tid 3 2020-06-12T02:39:06.559 INFO:tasks.rados.rados.0.smithi107.stdout:9706: writing smithi10729407-15 from 1638400 to 1753088 tid 4 2020-06-12T02:39:06.560 INFO:tasks.rados.rados.0.smithi107.stdout:9707: delete oid 19 current snap is 0 2020-06-12T02:39:06.560 INFO:tasks.rados.rados.0.smithi107.stdout:9703: finishing write tid 2 to smithi10729407-3 2020-06-12T02:39:06.560 INFO:tasks.rados.rados.0.smithi107.stdout:9703: finishing write tid 3 to smithi10729407-3 2020-06-12T02:39:06.560 INFO:tasks.rados.rados.0.smithi107.stdout:9703: finishing write tid 5 to smithi10729407-3 2020-06-12T02:39:06.560 INFO:tasks.rados.rados.0.smithi107.stdout:9703: finishing write tid 6 to smithi10729407-3 2020-06-12T02:39:06.561 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1735 (ObjNum 9702 snap 0 seq_num 9702) dirty exists 2020-06-12T02:39:06.561 INFO:tasks.rados.rados.0.smithi107.stdout:9703: left oid 3 (ObjNum 9702 snap 0 seq_num 9702) 2020-06-12T02:39:06.561 INFO:tasks.rados.rados.0.smithi107.stdout:9706: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:06.562 INFO:tasks.rados.rados.0.smithi107.stdout:9703: done (2 left) 2020-06-12T02:39:06.562 INFO:tasks.rados.rados.0.smithi107.stdout:9707: done (1 left) 2020-06-12T02:39:06.563 INFO:tasks.rados.rados.0.smithi107.stdout:9708: delete oid 16 current snap is 0 2020-06-12T02:39:06.563 INFO:tasks.rados.rados.0.smithi107.stdout:9708: done (1 left) 2020-06-12T02:39:06.563 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:39:06.563 INFO:tasks.rados.rados.0.smithi107.stdout:9709: seq_num 9708 ranges {0=598016,598016=491520,1089536=425984,1515520=286720} 2020-06-12T02:39:06.563 INFO:tasks.rados.rados.0.smithi107.stdout:9709: writing smithi10729407-2 from 0 to 598016 tid 1 2020-06-12T02:39:06.563 INFO:tasks.rados.rados.0.smithi107.stdout:9709: writing smithi10729407-2 from 598016 to 1089536 tid 2 2020-06-12T02:39:06.564 INFO:tasks.rados.rados.0.smithi107.stdout:9709: writing smithi10729407-2 from 1089536 to 1515520 tid 3 2020-06-12T02:39:06.564 INFO:tasks.rados.rados.0.smithi107.stdout:9709: writing smithi10729407-2 from 1515520 to 1802240 tid 4 2020-06-12T02:39:06.564 INFO:tasks.rados.rados.0.smithi107.stdout:9710: delete oid 8 current snap is 0 2020-06-12T02:39:06.564 INFO:tasks.rados.rados.0.smithi107.stdout:9706: finishing write tid 2 to smithi10729407-15 2020-06-12T02:39:06.564 INFO:tasks.rados.rados.0.smithi107.stdout:9706: finishing write tid 3 to smithi10729407-15 2020-06-12T02:39:06.565 INFO:tasks.rados.rados.0.smithi107.stdout:9706: finishing write tid 4 to smithi10729407-15 2020-06-12T02:39:06.565 INFO:tasks.rados.rados.0.smithi107.stdout:9706: finishing write tid 6 to smithi10729407-15 2020-06-12T02:39:06.565 INFO:tasks.rados.rados.0.smithi107.stdout:9706: finishing write tid 7 to smithi10729407-15 2020-06-12T02:39:06.565 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1798 (ObjNum 9705 snap 0 seq_num 9705) dirty exists 2020-06-12T02:39:06.565 INFO:tasks.rados.rados.0.smithi107.stdout:9706: left oid 15 (ObjNum 9705 snap 0 seq_num 9705) 2020-06-12T02:39:06.566 INFO:tasks.rados.rados.0.smithi107.stdout:9709: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:06.566 INFO:tasks.rados.rados.0.smithi107.stdout:9706: done (2 left) 2020-06-12T02:39:06.566 INFO:tasks.rados.rados.0.smithi107.stdout:9710: done (1 left) 2020-06-12T02:39:06.566 INFO:tasks.rados.rados.0.smithi107.stdout:9711: delete oid 10 current snap is 0 2020-06-12T02:39:06.566 INFO:tasks.rados.rados.0.smithi107.stdout:9709: finishing write tid 2 to smithi10729407-2 2020-06-12T02:39:06.567 INFO:tasks.rados.rados.0.smithi107.stdout:9711: done (1 left) 2020-06-12T02:39:06.567 INFO:tasks.rados.rados.0.smithi107.stdout:9712: delete oid 7 current snap is 0 2020-06-12T02:39:06.567 INFO:tasks.rados.rados.0.smithi107.stdout:9709: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:06.567 INFO:tasks.rados.rados.0.smithi107.stdout:9712: done (1 left) 2020-06-12T02:39:06.567 INFO:tasks.rados.rados.0.smithi107.stdout:9713: delete oid 9 current snap is 0 2020-06-12T02:39:06.567 INFO:tasks.rados.rados.0.smithi107.stdout:9709: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:06.568 INFO:tasks.rados.rados.0.smithi107.stdout:9709: finishing write tid 6 to smithi10729407-2 2020-06-12T02:39:06.568 INFO:tasks.rados.rados.0.smithi107.stdout:9709: finishing write tid 7 to smithi10729407-2 2020-06-12T02:39:06.568 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 912 (ObjNum 9708 snap 0 seq_num 9708) dirty exists 2020-06-12T02:39:06.568 INFO:tasks.rados.rados.0.smithi107.stdout:9709: left oid 2 (ObjNum 9708 snap 0 seq_num 9708) 2020-06-12T02:39:06.570 INFO:tasks.rados.rados.0.smithi107.stdout:9709: done (1 left) 2020-06-12T02:39:06.570 INFO:tasks.rados.rados.0.smithi107.stdout:9713: done (0 left) 2020-06-12T02:39:06.571 INFO:tasks.rados.rados.0.smithi107.stdout:9714: delete oid 7 current snap is 0 2020-06-12T02:39:06.571 INFO:tasks.rados.rados.0.smithi107.stdout:9714: done (0 left) 2020-06-12T02:39:06.571 INFO:tasks.rados.rados.0.smithi107.stdout:9715: delete oid 10 current snap is 0 2020-06-12T02:39:06.571 INFO:tasks.rados.rados.0.smithi107.stdout:9715: done (0 left) 2020-06-12T02:39:06.572 INFO:tasks.rados.rados.0.smithi107.stdout:9716: delete oid 7 current snap is 0 2020-06-12T02:39:06.572 INFO:tasks.rados.rados.0.smithi107.stdout:9716: done (0 left) 2020-06-12T02:39:06.572 INFO:tasks.rados.rados.0.smithi107.stdout:9717: delete oid 18 current snap is 0 2020-06-12T02:39:06.572 INFO:tasks.rados.rados.0.smithi107.stdout:9717: done (0 left) 2020-06-12T02:39:06.573 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:39:06.573 INFO:tasks.rados.rados.0.smithi107.stdout:9718: seq_num 9717 ranges {0=745472,745472=753664,1499136=450560} 2020-06-12T02:39:06.573 INFO:tasks.rados.rados.0.smithi107.stdout:9718: writing smithi10729407-8 from 0 to 745472 tid 1 2020-06-12T02:39:06.573 INFO:tasks.rados.rados.0.smithi107.stdout:9718: writing smithi10729407-8 from 745472 to 1499136 tid 2 2020-06-12T02:39:06.574 INFO:tasks.rados.rados.0.smithi107.stdout:9718: writing smithi10729407-8 from 1499136 to 1949696 tid 3 2020-06-12T02:39:06.574 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:39:06.574 INFO:tasks.rados.rados.0.smithi107.stdout:9719: seq_num 9718 ranges {0=761856,761856=630784,1392640=573440} 2020-06-12T02:39:06.574 INFO:tasks.rados.rados.0.smithi107.stdout:9719: writing smithi10729407-17 from 0 to 761856 tid 1 2020-06-12T02:39:06.575 INFO:tasks.rados.rados.0.smithi107.stdout:9719: writing smithi10729407-17 from 761856 to 1392640 tid 2 2020-06-12T02:39:06.575 INFO:tasks.rados.rados.0.smithi107.stdout:9719: writing smithi10729407-17 from 1392640 to 1966080 tid 3 2020-06-12T02:39:06.575 INFO:tasks.rados.rados.0.smithi107.stdout:9720: delete oid 19 current snap is 0 2020-06-12T02:39:06.575 INFO:tasks.rados.rados.0.smithi107.stdout:9718: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:06.576 INFO:tasks.rados.rados.0.smithi107.stdout:9718: finishing write tid 2 to smithi10729407-8 2020-06-12T02:39:06.577 INFO:tasks.rados.rados.0.smithi107.stdout:9718: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:06.577 INFO:tasks.rados.rados.0.smithi107.stdout:9718: finishing write tid 5 to smithi10729407-8 2020-06-12T02:39:06.577 INFO:tasks.rados.rados.0.smithi107.stdout:9719: finishing write tid 1 to smithi10729407-17 2020-06-12T02:39:06.578 INFO:tasks.rados.rados.0.smithi107.stdout:9718: finishing write tid 6 to smithi10729407-8 2020-06-12T02:39:06.578 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1803 (ObjNum 9717 snap 0 seq_num 9717) dirty exists 2020-06-12T02:39:06.578 INFO:tasks.rados.rados.0.smithi107.stdout:9718: left oid 8 (ObjNum 9717 snap 0 seq_num 9717) 2020-06-12T02:39:06.579 INFO:tasks.rados.rados.0.smithi107.stdout:9718: done (2 left) 2020-06-12T02:39:06.579 INFO:tasks.rados.rados.0.smithi107.stdout:9720: done (1 left) 2020-06-12T02:39:06.579 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:39:06.579 INFO:tasks.rados.rados.0.smithi107.stdout:9721: seq_num 9720 ranges {4644864=794624,5439488=802816,6242304=401408} 2020-06-12T02:39:06.580 INFO:tasks.rados.rados.0.smithi107.stdout:9721: writing smithi10729407-1 from 4644864 to 5439488 tid 1 2020-06-12T02:39:06.580 INFO:tasks.rados.rados.0.smithi107.stdout:9721: writing smithi10729407-1 from 5439488 to 6242304 tid 2 2020-06-12T02:39:06.580 INFO:tasks.rados.rados.0.smithi107.stdout:9721: writing smithi10729407-1 from 6242304 to 6643712 tid 3 2020-06-12T02:39:06.580 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:39:06.581 INFO:tasks.rados.rados.0.smithi107.stdout:9722: seq_num 9721 ranges {0=409600,409600=688128,1097728=573440,1671168=344064} 2020-06-12T02:39:06.581 INFO:tasks.rados.rados.0.smithi107.stdout:9722: writing smithi10729407-20 from 0 to 409600 tid 1 2020-06-12T02:39:06.582 INFO:tasks.rados.rados.0.smithi107.stdout:9722: writing smithi10729407-20 from 409600 to 1097728 tid 2 2020-06-12T02:39:06.582 INFO:tasks.rados.rados.0.smithi107.stdout:9722: writing smithi10729407-20 from 1097728 to 1671168 tid 3 2020-06-12T02:39:06.582 INFO:tasks.rados.rados.0.smithi107.stdout:9722: writing smithi10729407-20 from 1671168 to 2015232 tid 4 2020-06-12T02:39:06.582 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:39:06.583 INFO:tasks.rados.rados.0.smithi107.stdout:9723: seq_num 9722 ranges {0=434176,434176=573440,1007616=737280,1744896=294912} 2020-06-12T02:39:06.583 INFO:tasks.rados.rados.0.smithi107.stdout:9723: writing smithi10729407-19 from 0 to 434176 tid 1 2020-06-12T02:39:06.583 INFO:tasks.rados.rados.0.smithi107.stdout:9723: writing smithi10729407-19 from 434176 to 1007616 tid 2 2020-06-12T02:39:06.583 INFO:tasks.rados.rados.0.smithi107.stdout:9723: writing smithi10729407-19 from 1007616 to 1744896 tid 3 2020-06-12T02:39:06.584 INFO:tasks.rados.rados.0.smithi107.stdout:9723: writing smithi10729407-19 from 1744896 to 2039808 tid 4 2020-06-12T02:39:06.584 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:39:06.584 INFO:tasks.rados.rados.0.smithi107.stdout:9724: seq_num 9723 ranges {0=450560,450560=458752,909312=540672,1449984=581632,2031616=24576} 2020-06-12T02:39:06.584 INFO:tasks.rados.rados.0.smithi107.stdout:9724: writing smithi10729407-7 from 0 to 450560 tid 1 2020-06-12T02:39:06.585 INFO:tasks.rados.rados.0.smithi107.stdout:9724: writing smithi10729407-7 from 450560 to 909312 tid 2 2020-06-12T02:39:06.585 INFO:tasks.rados.rados.0.smithi107.stdout:9724: writing smithi10729407-7 from 909312 to 1449984 tid 3 2020-06-12T02:39:06.585 INFO:tasks.rados.rados.0.smithi107.stdout:9724: writing smithi10729407-7 from 1449984 to 2031616 tid 4 2020-06-12T02:39:06.586 INFO:tasks.rados.rados.0.smithi107.stdout:9724: writing smithi10729407-7 from 2031616 to 2056192 tid 5 2020-06-12T02:39:06.586 INFO:tasks.rados.rados.0.smithi107.stdout:9725: delete oid 11 current snap is 0 2020-06-12T02:39:06.586 INFO:tasks.rados.rados.0.smithi107.stdout:9719: finishing write tid 2 to smithi10729407-17 2020-06-12T02:39:06.587 INFO:tasks.rados.rados.0.smithi107.stdout:9719: finishing write tid 3 to smithi10729407-17 2020-06-12T02:39:06.587 INFO:tasks.rados.rados.0.smithi107.stdout:9719: finishing write tid 5 to smithi10729407-17 2020-06-12T02:39:06.588 INFO:tasks.rados.rados.0.smithi107.stdout:9719: finishing write tid 6 to smithi10729407-17 2020-06-12T02:39:06.588 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1823 (ObjNum 9718 snap 0 seq_num 9718) dirty exists 2020-06-12T02:39:06.588 INFO:tasks.rados.rados.0.smithi107.stdout:9719: left oid 17 (ObjNum 9718 snap 0 seq_num 9718) 2020-06-12T02:39:06.588 INFO:tasks.rados.rados.0.smithi107.stdout:9721: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:06.589 INFO:tasks.rados.rados.0.smithi107.stdout:9721: finishing write tid 2 to smithi10729407-1 2020-06-12T02:39:06.589 INFO:tasks.rados.rados.0.smithi107.stdout:9721: finishing write tid 3 to smithi10729407-1 2020-06-12T02:39:06.589 INFO:tasks.rados.rados.0.smithi107.stdout:9721: finishing write tid 5 to smithi10729407-1 2020-06-12T02:39:06.589 INFO:tasks.rados.rados.0.smithi107.stdout:9722: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:06.590 INFO:tasks.rados.rados.0.smithi107.stdout:9721: finishing write tid 6 to smithi10729407-1 2020-06-12T02:39:06.590 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1754 (ObjNum 9720 snap 0 seq_num 9720) dirty exists 2020-06-12T02:39:06.590 INFO:tasks.rados.rados.0.smithi107.stdout:9721: left oid 1 (ObjNum 9720 snap 0 seq_num 9720) 2020-06-12T02:39:06.590 INFO:tasks.rados.rados.0.smithi107.stdout:9722: finishing write tid 2 to smithi10729407-20 2020-06-12T02:39:06.591 INFO:tasks.rados.rados.0.smithi107.stdout:9722: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:06.591 INFO:tasks.rados.rados.0.smithi107.stdout:9722: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:06.591 INFO:tasks.rados.rados.0.smithi107.stdout:9722: finishing write tid 6 to smithi10729407-20 2020-06-12T02:39:06.591 INFO:tasks.rados.rados.0.smithi107.stdout:9723: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:06.592 INFO:tasks.rados.rados.0.smithi107.stdout:9723: finishing write tid 2 to smithi10729407-19 2020-06-12T02:39:06.592 INFO:tasks.rados.rados.0.smithi107.stdout:9722: finishing write tid 7 to smithi10729407-20 2020-06-12T02:39:06.592 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 728 (ObjNum 9721 snap 0 seq_num 9721) dirty exists 2020-06-12T02:39:06.592 INFO:tasks.rados.rados.0.smithi107.stdout:9722: left oid 20 (ObjNum 9721 snap 0 seq_num 9721) 2020-06-12T02:39:06.592 INFO:tasks.rados.rados.0.smithi107.stdout:9723: finishing write tid 3 to smithi10729407-19 2020-06-12T02:39:06.593 INFO:tasks.rados.rados.0.smithi107.stdout:9723: finishing write tid 4 to smithi10729407-19 2020-06-12T02:39:06.593 INFO:tasks.rados.rados.0.smithi107.stdout:9723: finishing write tid 6 to smithi10729407-19 2020-06-12T02:39:06.593 INFO:tasks.rados.rados.0.smithi107.stdout:9724: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:06.593 INFO:tasks.rados.rados.0.smithi107.stdout:9719: done (5 left) 2020-06-12T02:39:06.594 INFO:tasks.rados.rados.0.smithi107.stdout:9721: done (4 left) 2020-06-12T02:39:06.594 INFO:tasks.rados.rados.0.smithi107.stdout:9722: done (3 left) 2020-06-12T02:39:06.594 INFO:tasks.rados.rados.0.smithi107.stdout:9725: done (2 left) 2020-06-12T02:39:06.594 INFO:tasks.rados.rados.0.smithi107.stdout:9726: delete oid 15 current snap is 0 2020-06-12T02:39:06.594 INFO:tasks.rados.rados.0.smithi107.stdout:9723: finishing write tid 7 to smithi10729407-19 2020-06-12T02:39:06.595 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1800 (ObjNum 9722 snap 0 seq_num 9722) dirty exists 2020-06-12T02:39:06.595 INFO:tasks.rados.rados.0.smithi107.stdout:9723: left oid 19 (ObjNum 9722 snap 0 seq_num 9722) 2020-06-12T02:39:06.595 INFO:tasks.rados.rados.0.smithi107.stdout:9723: done (2 left) 2020-06-12T02:39:06.595 INFO:tasks.rados.rados.0.smithi107.stdout:9726: done (1 left) 2020-06-12T02:39:06.595 INFO:tasks.rados.rados.0.smithi107.stdout:9727: delete oid 8 current snap is 0 2020-06-12T02:39:06.596 INFO:tasks.rados.rados.0.smithi107.stdout:9724: finishing write tid 2 to smithi10729407-7 2020-06-12T02:39:06.596 INFO:tasks.rados.rados.0.smithi107.stdout:9727: done (1 left) 2020-06-12T02:39:06.596 INFO:tasks.rados.rados.0.smithi107.stdout:9728: delete oid 11 current snap is 0 2020-06-12T02:39:06.596 INFO:tasks.rados.rados.0.smithi107.stdout:9728: done (1 left) 2020-06-12T02:39:06.596 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:39:06.597 INFO:tasks.rados.rados.0.smithi107.stdout:9729: seq_num 9728 ranges {1802240=532480,2334720=729088,3063808=729088,3792896=147456} 2020-06-12T02:39:06.597 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2020-06-12T02:39:06.598 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:39:06.499+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:39:06.603 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-06-12T02:39:06.608 INFO:tasks.rados.rados.0.smithi107.stdout:9729: writing smithi10729407-2 from 1802240 to 2334720 tid 1 2020-06-12T02:39:06.609 INFO:tasks.rados.rados.0.smithi107.stdout:9729: writing smithi10729407-2 from 2334720 to 3063808 tid 2 2020-06-12T02:39:06.609 INFO:tasks.rados.rados.0.smithi107.stdout:9729: writing smithi10729407-2 from 3063808 to 3792896 tid 3 2020-06-12T02:39:06.610 INFO:tasks.rados.rados.0.smithi107.stdout:9729: writing smithi10729407-2 from 3792896 to 3940352 tid 4 2020-06-12T02:39:06.610 INFO:tasks.rados.rados.0.smithi107.stdout:9730: delete oid 16 current snap is 0 2020-06-12T02:39:06.610 INFO:tasks.rados.rados.0.smithi107.stdout:9724: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:06.611 INFO:tasks.rados.rados.0.smithi107.stdout:9724: finishing write tid 4 to smithi10729407-7 2020-06-12T02:39:06.611 INFO:tasks.rados.rados.0.smithi107.stdout:9724: finishing write tid 5 to smithi10729407-7 2020-06-12T02:39:06.611 INFO:tasks.rados.rados.0.smithi107.stdout:9724: finishing write tid 7 to smithi10729407-7 2020-06-12T02:39:06.611 INFO:tasks.rados.rados.0.smithi107.stdout:9724: finishing write tid 8 to smithi10729407-7 2020-06-12T02:39:06.612 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1667 (ObjNum 9723 snap 0 seq_num 9723) dirty exists 2020-06-12T02:39:06.612 INFO:tasks.rados.rados.0.smithi107.stdout:9724: left oid 7 (ObjNum 9723 snap 0 seq_num 9723) 2020-06-12T02:39:06.612 INFO:tasks.rados.rados.0.smithi107.stdout:9729: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:06.612 INFO:tasks.rados.rados.0.smithi107.stdout:9724: done (2 left) 2020-06-12T02:39:06.612 INFO:tasks.rados.rados.0.smithi107.stdout:9730: done (1 left) 2020-06-12T02:39:06.613 INFO:tasks.rados.rados.0.smithi107.stdout:9731: delete oid 4 current snap is 0 2020-06-12T02:39:06.613 INFO:tasks.rados.rados.0.smithi107.stdout:9729: finishing write tid 2 to smithi10729407-2 2020-06-12T02:39:06.613 INFO:tasks.rados.rados.0.smithi107.stdout:9729: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:06.613 INFO:tasks.rados.rados.0.smithi107.stdout:9731: done (1 left) 2020-06-12T02:39:06.614 INFO:tasks.rados.rados.0.smithi107.stdout:9732: delete oid 10 current snap is 0 2020-06-12T02:39:06.614 INFO:tasks.rados.rados.0.smithi107.stdout:9729: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:06.614 INFO:tasks.rados.rados.0.smithi107.stdout:9729: finishing write tid 6 to smithi10729407-2 2020-06-12T02:39:06.614 INFO:tasks.rados.rados.0.smithi107.stdout:9732: done (1 left) 2020-06-12T02:39:06.614 INFO:tasks.rados.rados.0.smithi107.stdout:9733: delete oid 4 current snap is 0 2020-06-12T02:39:06.615 INFO:tasks.rados.rados.0.smithi107.stdout:9729: finishing write tid 7 to smithi10729407-2 2020-06-12T02:39:06.615 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 917 (ObjNum 9728 snap 0 seq_num 9728) dirty exists 2020-06-12T02:39:06.615 INFO:tasks.rados.rados.0.smithi107.stdout:9729: left oid 2 (ObjNum 9728 snap 0 seq_num 9728) 2020-06-12T02:39:06.615 INFO:tasks.rados.rados.0.smithi107.stdout:9729: done (1 left) 2020-06-12T02:39:06.616 INFO:tasks.rados.rados.0.smithi107.stdout:9733: done (0 left) 2020-06-12T02:39:06.616 INFO:tasks.rados.rados.0.smithi107.stdout:9734: delete oid 18 current snap is 0 2020-06-12T02:39:06.616 INFO:tasks.rados.rados.0.smithi107.stdout:9734: done (0 left) 2020-06-12T02:39:06.616 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:39:06.617 INFO:tasks.rados.rados.0.smithi107.stdout:9735: seq_num 9734 ranges {0=630784,630784=483328,1114112=720896,1835008=401408} 2020-06-12T02:39:06.617 INFO:tasks.rados.rados.0.smithi107.stdout:9735: writing smithi10729407-11 from 0 to 630784 tid 1 2020-06-12T02:39:06.617 INFO:tasks.rados.rados.0.smithi107.stdout:9735: writing smithi10729407-11 from 630784 to 1114112 tid 2 2020-06-12T02:39:06.618 INFO:tasks.rados.rados.0.smithi107.stdout:9735: writing smithi10729407-11 from 1114112 to 1835008 tid 3 2020-06-12T02:39:06.618 INFO:tasks.rados.rados.0.smithi107.stdout:9735: writing smithi10729407-11 from 1835008 to 2236416 tid 4 2020-06-12T02:39:06.618 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:39:06.619 INFO:tasks.rados.rados.0.smithi107.stdout:9736: seq_num 9735 ranges {2056192=647168,2703360=770048,3473408=483328,3956736=352256} 2020-06-12T02:39:06.619 INFO:tasks.rados.rados.0.smithi107.stdout:9736: writing smithi10729407-7 from 2056192 to 2703360 tid 1 2020-06-12T02:39:06.619 INFO:tasks.rados.rados.0.smithi107.stdout:9736: writing smithi10729407-7 from 2703360 to 3473408 tid 2 2020-06-12T02:39:06.619 INFO:tasks.rados.rados.0.smithi107.stdout:9736: writing smithi10729407-7 from 3473408 to 3956736 tid 3 2020-06-12T02:39:06.619 INFO:tasks.rados.rados.0.smithi107.stdout:9736: writing smithi10729407-7 from 3956736 to 4308992 tid 4 2020-06-12T02:39:06.620 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:39:06.620 INFO:tasks.rados.rados.0.smithi107.stdout:9737: seq_num 9736 ranges {0=663552,663552=655360,1318912=688128,2007040=262144} 2020-06-12T02:39:06.620 INFO:tasks.rados.rados.0.smithi107.stdout:9737: writing smithi10729407-16 from 0 to 663552 tid 1 2020-06-12T02:39:06.620 INFO:tasks.rados.rados.0.smithi107.stdout:9737: writing smithi10729407-16 from 663552 to 1318912 tid 2 2020-06-12T02:39:06.621 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:39:06.603+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:39:06.623 INFO:tasks.rados.rados.0.smithi107.stdout:9737: writing smithi10729407-16 from 1318912 to 2007040 tid 3 2020-06-12T02:39:06.624 INFO:tasks.rados.rados.0.smithi107.stdout:9737: writing smithi10729407-16 from 2007040 to 2269184 tid 4 2020-06-12T02:39:06.624 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:39:06.625 INFO:tasks.rados.rados.0.smithi107.stdout:9735: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:06.625 INFO:tasks.rados.rados.0.smithi107.stdout:9735: finishing write tid 2 to smithi10729407-11 2020-06-12T02:39:06.625 INFO:tasks.rados.rados.0.smithi107.stdout:9735: finishing write tid 3 to smithi10729407-11 2020-06-12T02:39:06.625 INFO:tasks.rados.rados.0.smithi107.stdout:9735: finishing write tid 4 to smithi10729407-11 2020-06-12T02:39:06.626 INFO:tasks.rados.rados.0.smithi107.stdout:9735: finishing write tid 6 to smithi10729407-11 2020-06-12T02:39:06.626 INFO:tasks.rados.rados.0.smithi107.stdout:9735: finishing write tid 7 to smithi10729407-11 2020-06-12T02:39:06.627 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1652 (ObjNum 9734 snap 0 seq_num 9734) dirty exists 2020-06-12T02:39:06.627 INFO:tasks.rados.rados.0.smithi107.stdout:9735: left oid 11 (ObjNum 9734 snap 0 seq_num 9734) 2020-06-12T02:39:06.627 INFO:tasks.rados.rados.0.smithi107.stdout:9736: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:06.627 INFO:tasks.rados.rados.0.smithi107.stdout:9736: finishing write tid 2 to smithi10729407-7 2020-06-12T02:39:06.628 INFO:tasks.rados.rados.0.smithi107.stdout:9736: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:06.628 INFO:tasks.rados.rados.0.smithi107.stdout:9736: finishing write tid 4 to smithi10729407-7 2020-06-12T02:39:06.628 INFO:tasks.rados.rados.0.smithi107.stdout:9736: finishing write tid 6 to smithi10729407-7 2020-06-12T02:39:06.628 INFO:tasks.rados.rados.0.smithi107.stdout:9736: finishing write tid 7 to smithi10729407-7 2020-06-12T02:39:06.628 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1672 (ObjNum 9735 snap 0 seq_num 9735) dirty exists 2020-06-12T02:39:06.629 INFO:tasks.rados.rados.0.smithi107.stdout:9736: left oid 7 (ObjNum 9735 snap 0 seq_num 9735) 2020-06-12T02:39:06.629 INFO:tasks.rados.rados.0.smithi107.stdout:9737: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:06.629 INFO:tasks.rados.rados.0.smithi107.stdout:9738: seq_num 9737 ranges {0=679936,679936=532480,1212416=499712,1712128=516096,2228224=57344} 2020-06-12T02:39:06.629 INFO:tasks.rados.rados.0.smithi107.stdout:9738: writing smithi10729407-10 from 0 to 679936 tid 1 2020-06-12T02:39:06.630 INFO:tasks.rados.rados.0.smithi107.stdout:9738: writing smithi10729407-10 from 679936 to 1212416 tid 2 2020-06-12T02:39:06.632 INFO:tasks.rados.rados.0.smithi107.stdout:9738: writing smithi10729407-10 from 1212416 to 1712128 tid 3 2020-06-12T02:39:06.632 INFO:tasks.rados.rados.0.smithi107.stdout:9738: writing smithi10729407-10 from 1712128 to 2228224 tid 4 2020-06-12T02:39:06.632 INFO:tasks.rados.rados.0.smithi107.stdout:9738: writing smithi10729407-10 from 2228224 to 2285568 tid 5 2020-06-12T02:39:06.632 INFO:tasks.rados.rados.0.smithi107.stdout:9737: finishing write tid 2 to smithi10729407-16 2020-06-12T02:39:06.633 INFO:tasks.rados.rados.0.smithi107.stdout:9735: done (3 left) 2020-06-12T02:39:06.633 INFO:tasks.rados.rados.0.smithi107.stdout:9736: done (2 left) 2020-06-12T02:39:06.633 INFO:tasks.rados.rados.0.smithi107.stdout:9739: delete oid 7 current snap is 0 2020-06-12T02:39:06.633 INFO:tasks.rados.rados.0.smithi107.stdout:9738: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:06.634 INFO:tasks.rados.rados.0.smithi107.stdout:9737: finishing write tid 3 to smithi10729407-16 2020-06-12T02:39:06.635 INFO:tasks.rados.rados.0.smithi107.stdout:9738: finishing write tid 2 to smithi10729407-10 2020-06-12T02:39:06.635 INFO:tasks.rados.rados.0.smithi107.stdout:9737: finishing write tid 4 to smithi10729407-16 2020-06-12T02:39:06.635 INFO:tasks.rados.rados.0.smithi107.stdout:9739: done (2 left) 2020-06-12T02:39:06.636 INFO:tasks.rados.rados.0.smithi107.stdout:9740: delete oid 15 current snap is 0 2020-06-12T02:39:06.636 INFO:tasks.rados.rados.0.smithi107.stdout:9737: finishing write tid 6 to smithi10729407-16 2020-06-12T02:39:06.636 INFO:tasks.rados.rados.0.smithi107.stdout:9738: finishing write tid 3 to smithi10729407-10 2020-06-12T02:39:06.637 INFO:tasks.rados.rados.0.smithi107.stdout:9738: finishing write tid 4 to smithi10729407-10 2020-06-12T02:39:06.637 INFO:tasks.rados.rados.0.smithi107.stdout:9738: finishing write tid 5 to smithi10729407-10 2020-06-12T02:39:06.637 INFO:tasks.rados.rados.0.smithi107.stdout:9738: finishing write tid 7 to smithi10729407-10 2020-06-12T02:39:06.638 INFO:tasks.rados.rados.0.smithi107.stdout:9737: finishing write tid 7 to smithi10729407-16 2020-06-12T02:39:06.638 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1742 (ObjNum 9736 snap 0 seq_num 9736) dirty exists 2020-06-12T02:39:06.638 INFO:tasks.rados.rados.0.smithi107.stdout:9737: left oid 16 (ObjNum 9736 snap 0 seq_num 9736) 2020-06-12T02:39:06.639 INFO:tasks.rados.rados.0.smithi107.stdout:9738: finishing write tid 8 to smithi10729407-10 2020-06-12T02:39:06.639 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 700 (ObjNum 9737 snap 0 seq_num 9737) dirty exists 2020-06-12T02:39:06.640 INFO:tasks.rados.rados.0.smithi107.stdout:9738: left oid 10 (ObjNum 9737 snap 0 seq_num 9737) 2020-06-12T02:39:06.640 INFO:tasks.rados.rados.0.smithi107.stdout:9737: done (2 left) 2020-06-12T02:39:06.641 INFO:tasks.rados.rados.0.smithi107.stdout:9738: done (1 left) 2020-06-12T02:39:06.641 INFO:tasks.rados.rados.0.smithi107.stdout:9740: done (0 left) 2020-06-12T02:39:06.641 INFO:tasks.rados.rados.0.smithi107.stdout:9741: delete oid 10 current snap is 0 2020-06-12T02:39:06.642 INFO:tasks.rados.rados.0.smithi107.stdout:9741: done (0 left) 2020-06-12T02:39:06.642 INFO:tasks.rados.rados.0.smithi107.stdout:9742: delete oid 5 current snap is 0 2020-06-12T02:39:06.645 INFO:tasks.rados.rados.0.smithi107.stdout:9742: done (0 left) 2020-06-12T02:39:06.645 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:39:06.645 INFO:tasks.rados.rados.0.smithi107.stdout:9743: seq_num 9742 ranges {0=770048,770048=409600,1179648=688128,1867776=507904} 2020-06-12T02:39:06.649 INFO:tasks.rados.rados.0.smithi107.stdout:9743: writing smithi10729407-4 from 0 to 770048 tid 1 2020-06-12T02:39:06.652 INFO:tasks.rados.rados.0.smithi107.stdout:9743: writing smithi10729407-4 from 770048 to 1179648 tid 2 2020-06-12T02:39:06.656 INFO:tasks.rados.rados.0.smithi107.stdout:9743: writing smithi10729407-4 from 1179648 to 1867776 tid 3 2020-06-12T02:39:06.659 INFO:tasks.rados.rados.0.smithi107.stdout:9743: writing smithi10729407-4 from 1867776 to 2375680 tid 4 2020-06-12T02:39:06.659 INFO:tasks.rados.rados.0.smithi107.stdout:9744: delete oid 13 current snap is 0 2020-06-12T02:39:06.659 INFO:tasks.rados.rados.0.smithi107.stdout:9743: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:06.661 INFO:tasks.rados.rados.0.smithi107.stdout:9744: done (1 left) 2020-06-12T02:39:06.662 INFO:tasks.rados.rados.0.smithi107.stdout:9745: delete oid 20 current snap is 0 2020-06-12T02:39:06.664 INFO:tasks.rados.rados.0.smithi107.stdout:9743: finishing write tid 2 to smithi10729407-4 2020-06-12T02:39:06.665 INFO:tasks.rados.rados.0.smithi107.stdout:9745: done (1 left) 2020-06-12T02:39:06.665 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:39:06.665 INFO:tasks.rados.rados.0.smithi107.stdout:9746: seq_num 9745 ranges {0=16384} 2020-06-12T02:39:06.665 INFO:tasks.rados.rados.0.smithi107.stdout:9746: writing smithi10729407-20 from 0 to 16384 tid 1 2020-06-12T02:39:06.666 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:39:06.666 INFO:tasks.rados.rados.0.smithi107.stdout:9747: seq_num 9746 ranges {2252800=32768} 2020-06-12T02:39:06.678 INFO:tasks.rados.rados.0.smithi107.stdout:9747: writing smithi10729407-6 from 2252800 to 2285568 tid 1 2020-06-12T02:39:06.678 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:39:06.678 INFO:tasks.rados.rados.0.smithi107.stdout:9748: seq_num 9747 ranges {6643712=49152} 2020-06-12T02:39:06.711 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:39:06.704+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:39:06.718 INFO:tasks.rados.rados.0.smithi107.stdout:9748: writing smithi10729407-1 from 6643712 to 6692864 tid 1 2020-06-12T02:39:06.719 INFO:tasks.rados.rados.0.smithi107.stdout:9749: delete oid 3 current snap is 0 2020-06-12T02:39:06.719 INFO:tasks.rados.rados.0.smithi107.stdout:9743: finishing write tid 3 to smithi10729407-4 2020-06-12T02:39:06.719 INFO:tasks.rados.rados.0.smithi107.stdout:9746: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:06.720 INFO:tasks.rados.rados.0.smithi107.stdout:9743: finishing write tid 4 to smithi10729407-4 2020-06-12T02:39:06.720 INFO:tasks.rados.rados.0.smithi107.stdout:9743: finishing write tid 6 to smithi10729407-4 2020-06-12T02:39:06.720 INFO:tasks.rados.rados.0.smithi107.stdout:9743: finishing write tid 7 to smithi10729407-4 2020-06-12T02:39:06.720 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1830 (ObjNum 9742 snap 0 seq_num 9742) dirty exists 2020-06-12T02:39:06.720 INFO:tasks.rados.rados.0.smithi107.stdout:9743: left oid 4 (ObjNum 9742 snap 0 seq_num 9742) 2020-06-12T02:39:06.721 INFO:tasks.rados.rados.0.smithi107.stdout:9746: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:06.721 INFO:tasks.rados.rados.0.smithi107.stdout:9746: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:06.721 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 731 (ObjNum 9745 snap 0 seq_num 9745) dirty exists 2020-06-12T02:39:06.721 INFO:tasks.rados.rados.0.smithi107.stdout:9746: left oid 20 (ObjNum 9745 snap 0 seq_num 9745) 2020-06-12T02:39:06.721 INFO:tasks.rados.rados.0.smithi107.stdout:9747: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:06.726 INFO:tasks.rados.rados.0.smithi107.stdout:9747: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:06.727 INFO:tasks.rados.rados.0.smithi107.stdout:9747: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:06.727 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1756 (ObjNum 9746 snap 0 seq_num 9746) dirty exists 2020-06-12T02:39:06.727 INFO:tasks.rados.rados.0.smithi107.stdout:9747: left oid 6 (ObjNum 9746 snap 0 seq_num 9746) 2020-06-12T02:39:06.727 INFO:tasks.rados.rados.0.smithi107.stdout:9743: done (4 left) 2020-06-12T02:39:06.728 INFO:tasks.rados.rados.0.smithi107.stdout:9746: done (3 left) 2020-06-12T02:39:06.728 INFO:tasks.rados.rados.0.smithi107.stdout:9747: done (2 left) 2020-06-12T02:39:06.728 INFO:tasks.rados.rados.0.smithi107.stdout:9749: done (1 left) 2020-06-12T02:39:06.728 INFO:tasks.rados.rados.0.smithi107.stdout:9750: delete oid 16 current snap is 0 2020-06-12T02:39:06.729 INFO:tasks.rados.rados.0.smithi107.stdout:9748: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:06.730 INFO:tasks.rados.rados.0.smithi107.stdout:9748: finishing write tid 3 to smithi10729407-1 2020-06-12T02:39:06.730 INFO:tasks.rados.rados.0.smithi107.stdout:9750: done (1 left) 2020-06-12T02:39:06.730 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:39:06.731 INFO:tasks.rados.rados.0.smithi107.stdout:9751: seq_num 9750 ranges {1966080=98304} 2020-06-12T02:39:06.734 INFO:tasks.rados.rados.0.smithi107.stdout:9751: writing smithi10729407-17 from 1966080 to 2064384 tid 1 2020-06-12T02:39:06.735 INFO:tasks.rados.rados.0.smithi107.stdout:9752: delete oid 8 current snap is 0 2020-06-12T02:39:06.735 INFO:tasks.rados.rados.0.smithi107.stdout:9748: finishing write tid 4 to smithi10729407-1 2020-06-12T02:39:06.736 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1758 (ObjNum 9747 snap 0 seq_num 9747) dirty exists 2020-06-12T02:39:06.737 INFO:tasks.rados.rados.0.smithi107.stdout:9748: left oid 1 (ObjNum 9747 snap 0 seq_num 9747) 2020-06-12T02:39:06.739 INFO:tasks.rados.rados.0.smithi107.stdout:9748: done (2 left) 2020-06-12T02:39:06.739 INFO:tasks.rados.rados.0.smithi107.stdout:9752: done (1 left) 2020-06-12T02:39:06.739 INFO:tasks.rados.rados.0.smithi107.stdout:9753: delete oid 5 current snap is 0 2020-06-12T02:39:06.740 INFO:tasks.rados.rados.0.smithi107.stdout:9751: finishing write tid 1 to smithi10729407-17 2020-06-12T02:39:06.740 INFO:tasks.rados.rados.0.smithi107.stdout:9751: finishing write tid 3 to smithi10729407-17 2020-06-12T02:39:06.742 INFO:tasks.rados.rados.0.smithi107.stdout:9753: done (1 left) 2020-06-12T02:39:06.742 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:39:06.742 INFO:tasks.rados.rados.0.smithi107.stdout:9754: seq_num 9753 ranges {0=147456} 2020-06-12T02:39:06.743 INFO:tasks.rados.rados.0.smithi107.stdout:9754: writing smithi10729407-8 from 0 to 147456 tid 1 2020-06-12T02:39:06.743 INFO:tasks.rados.rados.0.smithi107.stdout:9755: delete oid 4 current snap is 0 2020-06-12T02:39:06.743 INFO:tasks.rados.rados.0.smithi107.stdout:9751: finishing write tid 4 to smithi10729407-17 2020-06-12T02:39:06.744 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1832 (ObjNum 9750 snap 0 seq_num 9750) dirty exists 2020-06-12T02:39:06.744 INFO:tasks.rados.rados.0.smithi107.stdout:9751: left oid 17 (ObjNum 9750 snap 0 seq_num 9750) 2020-06-12T02:39:06.745 INFO:tasks.rados.rados.0.smithi107.stdout:9751: done (2 left) 2020-06-12T02:39:06.745 INFO:tasks.rados.rados.0.smithi107.stdout:9755: done (1 left) 2020-06-12T02:39:06.745 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:39:06.746 INFO:tasks.rados.rados.0.smithi107.stdout:9756: seq_num 9755 ranges {3940352=180224} 2020-06-12T02:39:06.762 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:39:06.762 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:39:06.762 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2020-06-12T02:39:06.763 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:39:06.766 INFO:tasks.rados.rados.0.smithi107.stdout:9756: writing smithi10729407-2 from 3940352 to 4120576 tid 1 2020-06-12T02:39:06.766 INFO:tasks.rados.rados.0.smithi107.stdout:9757: delete oid 5 current snap is 0 2020-06-12T02:39:06.766 INFO:tasks.rados.rados.0.smithi107.stdout:9754: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:06.767 INFO:tasks.rados.rados.0.smithi107.stdout:9754: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:06.767 INFO:tasks.rados.rados.0.smithi107.stdout:9754: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:06.767 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1809 (ObjNum 9753 snap 0 seq_num 9753) dirty exists 2020-06-12T02:39:06.767 INFO:tasks.rados.rados.0.smithi107.stdout:9754: left oid 8 (ObjNum 9753 snap 0 seq_num 9753) 2020-06-12T02:39:06.768 INFO:tasks.rados.rados.0.smithi107.stdout:9754: done (2 left) 2020-06-12T02:39:06.768 INFO:tasks.rados.rados.0.smithi107.stdout:9757: done (1 left) 2020-06-12T02:39:06.768 INFO:tasks.rados.rados.0.smithi107.stdout:9758: delete oid 7 current snap is 0 2020-06-12T02:39:06.770 INFO:tasks.rados.rados.0.smithi107.stdout:9758: done (1 left) 2020-06-12T02:39:06.770 INFO:tasks.rados.rados.0.smithi107.stdout:9759: delete oid 8 current snap is 0 2020-06-12T02:39:06.772 INFO:tasks.rados.rados.0.smithi107.stdout:9756: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:06.772 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-06-12T02:39:06.773 INFO:tasks.rados.rados.0.smithi107.stdout:9759: done (1 left) 2020-06-12T02:39:06.774 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:39:06.774 INFO:tasks.rados.rados.0.smithi107.stdout:9760: seq_num 9759 ranges {0=245760} 2020-06-12T02:39:06.775 INFO:tasks.rados.rados.0.smithi107.stdout:9760: writing smithi10729407-8 from 0 to 245760 tid 1 2020-06-12T02:39:06.776 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:39:06.776 INFO:tasks.rados.rados.0.smithi107.stdout:9761: seq_num 9760 ranges {0=262144} 2020-06-12T02:39:06.777 INFO:tasks.rados.rados.0.smithi107.stdout:9761: writing smithi10729407-15 from 0 to 262144 tid 1 2020-06-12T02:39:06.777 INFO:tasks.rados.rados.0.smithi107.stdout:9762: delete oid 16 current snap is 0 2020-06-12T02:39:06.777 INFO:tasks.rados.rados.0.smithi107.stdout:9756: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:06.778 INFO:tasks.rados.rados.0.smithi107.stdout:9756: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:06.778 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 919 (ObjNum 9755 snap 0 seq_num 9755) dirty exists 2020-06-12T02:39:06.778 INFO:tasks.rados.rados.0.smithi107.stdout:9756: left oid 2 (ObjNum 9755 snap 0 seq_num 9755) 2020-06-12T02:39:06.779 INFO:tasks.rados.rados.0.smithi107.stdout:9756: done (3 left) 2020-06-12T02:39:06.779 INFO:tasks.rados.rados.0.smithi107.stdout:9762: done (2 left) 2020-06-12T02:39:06.780 INFO:tasks.rados.rados.0.smithi107.stdout:9763: delete oid 18 current snap is 0 2020-06-12T02:39:06.782 INFO:tasks.rados.rados.0.smithi107.stdout:9760: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:06.782 INFO:tasks.rados.rados.0.smithi107.stdout:9763: done (2 left) 2020-06-12T02:39:06.782 INFO:tasks.rados.rados.0.smithi107.stdout:9764: delete oid 17 current snap is 0 2020-06-12T02:39:06.783 INFO:tasks.rados.rados.0.smithi107.stdout:9761: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:06.790 INFO:tasks.rados.rados.0.smithi107.stdout:9761: finishing write tid 3 to smithi10729407-15 2020-06-12T02:39:06.791 INFO:tasks.rados.rados.0.smithi107.stdout:9761: finishing write tid 4 to smithi10729407-15 2020-06-12T02:39:06.791 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1813 (ObjNum 9760 snap 0 seq_num 9760) dirty exists 2020-06-12T02:39:06.791 INFO:tasks.rados.rados.0.smithi107.stdout:9761: left oid 15 (ObjNum 9760 snap 0 seq_num 9760) 2020-06-12T02:39:06.791 INFO:tasks.rados.rados.0.smithi107.stdout:9761: done (2 left) 2020-06-12T02:39:06.791 INFO:tasks.rados.rados.0.smithi107.stdout:9764: done (1 left) 2020-06-12T02:39:06.792 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:39:06.792 INFO:tasks.rados.rados.0.smithi107.stdout:9765: seq_num 9764 ranges {262144=335872} 2020-06-12T02:39:06.792 INFO:tasks.rados.rados.0.smithi107.stdout:9765: writing smithi10729407-15 from 262144 to 598016 tid 1 2020-06-12T02:39:06.792 INFO:tasks.rados.rados.0.smithi107.stdout:9766: delete oid 5 current snap is 0 2020-06-12T02:39:06.792 INFO:tasks.rados.rados.0.smithi107.stdout:9760: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:06.793 INFO:tasks.rados.rados.0.smithi107.stdout:9766: done (2 left) 2020-06-12T02:39:06.793 INFO:tasks.rados.rados.0.smithi107.stdout:9767: delete oid 12 current snap is 0 2020-06-12T02:39:06.794 INFO:tasks.rados.rados.0.smithi107.stdout:9760: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:06.794 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1814 (ObjNum 9759 snap 0 seq_num 9759) dirty exists 2020-06-12T02:39:06.794 INFO:tasks.rados.rados.0.smithi107.stdout:9760: left oid 8 (ObjNum 9759 snap 0 seq_num 9759) 2020-06-12T02:39:06.797 INFO:tasks.rados.rados.0.smithi107.stdout:9765: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:06.798 INFO:tasks.rados.rados.0.smithi107.stdout:9760: done (2 left) 2020-06-12T02:39:06.798 INFO:tasks.rados.rados.0.smithi107.stdout:9767: done (1 left) 2020-06-12T02:39:06.798 INFO:tasks.rados.rados.0.smithi107.stdout:9768: delete oid 13 current snap is 0 2020-06-12T02:39:06.799 INFO:tasks.rados.rados.0.smithi107.stdout:9765: finishing write tid 3 to smithi10729407-15 2020-06-12T02:39:06.799 INFO:tasks.rados.rados.0.smithi107.stdout:9765: finishing write tid 4 to smithi10729407-15 2020-06-12T02:39:06.800 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1816 (ObjNum 9764 snap 0 seq_num 9764) dirty exists 2020-06-12T02:39:06.800 INFO:tasks.rados.rados.0.smithi107.stdout:9765: left oid 15 (ObjNum 9764 snap 0 seq_num 9764) 2020-06-12T02:39:06.801 INFO:tasks.rados.rados.0.smithi107.stdout:9765: done (1 left) 2020-06-12T02:39:06.801 INFO:tasks.rados.rados.0.smithi107.stdout:9768: done (0 left) 2020-06-12T02:39:06.802 INFO:tasks.rados.rados.0.smithi107.stdout:9769: delete oid 12 current snap is 0 2020-06-12T02:39:06.807 INFO:tasks.rados.rados.0.smithi107.stdout:9769: done (0 left) 2020-06-12T02:39:06.807 INFO:tasks.rados.rados.0.smithi107.stdout:9770: delete oid 9 current snap is 0 2020-06-12T02:39:06.807 INFO:tasks.rados.rados.0.smithi107.stdout:9770: done (0 left) 2020-06-12T02:39:06.807 INFO:tasks.rados.rados.0.smithi107.stdout:9771: delete oid 3 current snap is 0 2020-06-12T02:39:06.808 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:39:06.805+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:39:06.808 INFO:tasks.rados.rados.0.smithi107.stdout:9771: done (0 left) 2020-06-12T02:39:06.809 INFO:tasks.rados.rados.0.smithi107.stdout:9772: delete oid 3 current snap is 0 2020-06-12T02:39:06.810 INFO:tasks.rados.rados.0.smithi107.stdout:9772: done (0 left) 2020-06-12T02:39:06.811 INFO:tasks.rados.rados.0.smithi107.stdout:9773: delete oid 3 current snap is 0 2020-06-12T02:39:06.813 INFO:tasks.rados.rados.0.smithi107.stdout:9773: done (0 left) 2020-06-12T02:39:06.813 INFO:tasks.rados.rados.0.smithi107.stdout:9774: delete oid 7 current snap is 0 2020-06-12T02:39:06.815 INFO:tasks.rados.rados.0.smithi107.stdout:9774: done (0 left) 2020-06-12T02:39:06.815 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:39:06.815 INFO:tasks.rados.rados.0.smithi107.stdout:9775: seq_num 9774 ranges {2039808=499712} 2020-06-12T02:39:06.817 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-06-12T02:39:06.830 INFO:tasks.rados.rados.0.smithi107.stdout:9775: writing smithi10729407-19 from 2039808 to 2539520 tid 1 2020-06-12T02:39:06.831 INFO:tasks.rados.rados.0.smithi107.stdout:9776: delete oid 17 current snap is 0 2020-06-12T02:39:06.834 INFO:tasks.rados.rados.0.smithi107.stdout:9776: done (1 left) 2020-06-12T02:39:06.834 INFO:tasks.rados.rados.0.smithi107.stdout:9777: delete oid 13 current snap is 0 2020-06-12T02:39:06.837 INFO:tasks.rados.rados.0.smithi107.stdout:9775: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:06.837 INFO:tasks.rados.rados.0.smithi107.stdout:9777: done (1 left) 2020-06-12T02:39:06.837 INFO:tasks.rados.rados.0.smithi107.stdout:9778: delete oid 9 current snap is 0 2020-06-12T02:39:06.838 INFO:tasks.rados.rados.0.smithi107.stdout:9775: finishing write tid 3 to smithi10729407-19 2020-06-12T02:39:06.839 INFO:tasks.rados.rados.0.smithi107.stdout:9775: finishing write tid 4 to smithi10729407-19 2020-06-12T02:39:06.840 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1804 (ObjNum 9774 snap 0 seq_num 9774) dirty exists 2020-06-12T02:39:06.840 INFO:tasks.rados.rados.0.smithi107.stdout:9775: left oid 19 (ObjNum 9774 snap 0 seq_num 9774) 2020-06-12T02:39:06.840 INFO:tasks.rados.rados.0.smithi107.stdout:9775: done (1 left) 2020-06-12T02:39:06.840 INFO:tasks.rados.rados.0.smithi107.stdout:9778: done (0 left) 2020-06-12T02:39:06.840 INFO:tasks.rados.rados.0.smithi107.stdout:9779: delete oid 8 current snap is 0 2020-06-12T02:39:06.842 INFO:tasks.rados.rados.0.smithi107.stdout:9779: done (0 left) 2020-06-12T02:39:06.843 INFO:tasks.rados.rados.0.smithi107.stdout:9780: delete oid 19 current snap is 0 2020-06-12T02:39:06.844 INFO:tasks.rados.rados.0.smithi107.stdout:9780: done (0 left) 2020-06-12T02:39:06.844 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:39:06.844 INFO:tasks.rados.rados.0.smithi107.stdout:9781: seq_num 9780 ranges {2236416=598016} 2020-06-12T02:39:06.860 INFO:tasks.rados.rados.0.smithi107.stdout:9781: writing smithi10729407-11 from 2236416 to 2834432 tid 1 2020-06-12T02:39:06.860 INFO:tasks.rados.rados.0.smithi107.stdout:9782: delete oid 17 current snap is 0 2020-06-12T02:39:06.866 INFO:tasks.rados.rados.0.smithi107.stdout:9782: done (1 left) 2020-06-12T02:39:06.867 INFO:tasks.rados.rados.0.smithi107.stdout:9783: delete oid 19 current snap is 0 2020-06-12T02:39:06.867 INFO:tasks.rados.rados.0.smithi107.stdout:9781: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:06.869 INFO:tasks.rados.rados.0.smithi107.stdout:9783: done (1 left) 2020-06-12T02:39:06.869 INFO:tasks.rados.rados.0.smithi107.stdout:9784: delete oid 10 current snap is 0 2020-06-12T02:39:06.870 INFO:tasks.rados.rados.0.smithi107.stdout:9781: finishing write tid 3 to smithi10729407-11 2020-06-12T02:39:06.871 INFO:tasks.rados.rados.0.smithi107.stdout:9781: finishing write tid 4 to smithi10729407-11 2020-06-12T02:39:06.871 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1654 (ObjNum 9780 snap 0 seq_num 9780) dirty exists 2020-06-12T02:39:06.871 INFO:tasks.rados.rados.0.smithi107.stdout:9781: left oid 11 (ObjNum 9780 snap 0 seq_num 9780) 2020-06-12T02:39:06.872 INFO:tasks.rados.rados.0.smithi107.stdout:9781: done (1 left) 2020-06-12T02:39:06.872 INFO:tasks.rados.rados.0.smithi107.stdout:9784: done (0 left) 2020-06-12T02:39:06.872 INFO:tasks.rados.rados.0.smithi107.stdout:9785: delete oid 9 current snap is 0 2020-06-12T02:39:06.873 INFO:tasks.rados.rados.0.smithi107.stdout:9785: done (0 left) 2020-06-12T02:39:06.874 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:39:06.878 INFO:tasks.rados.rados.0.smithi107.stdout:9786: seq_num 9785 ranges {16384=688128} 2020-06-12T02:39:06.879 INFO:tasks.rados.rados.0.smithi107.stdout:9786: writing smithi10729407-20 from 16384 to 704512 tid 1 2020-06-12T02:39:06.879 INFO:tasks.rados.rados.0.smithi107.stdout:9787: delete oid 9 current snap is 0 2020-06-12T02:39:06.882 INFO:tasks.rados.rados.0.smithi107.stdout:9787: done (1 left) 2020-06-12T02:39:06.882 INFO:tasks.rados.rados.0.smithi107.stdout:9788: delete oid 10 current snap is 0 2020-06-12T02:39:06.883 INFO:tasks.rados.rados.0.smithi107.stdout:9788: done (1 left) 2020-06-12T02:39:06.884 INFO:tasks.rados.rados.0.smithi107.stdout:9789: delete oid 5 current snap is 0 2020-06-12T02:39:06.884 INFO:tasks.rados.rados.0.smithi107.stdout:9786: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:06.886 INFO:tasks.rados.rados.0.smithi107.stdout:9789: done (1 left) 2020-06-12T02:39:06.886 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:39:06.886 INFO:tasks.rados.rados.0.smithi107.stdout:9790: seq_num 9789 ranges {0=753664} 2020-06-12T02:39:06.890 INFO:tasks.rados.rados.0.smithi107.stdout:9790: writing smithi10729407-13 from 0 to 753664 tid 1 2020-06-12T02:39:06.891 INFO:tasks.rados.rados.0.smithi107.stdout:9791: delete oid 16 current snap is 0 2020-06-12T02:39:06.891 INFO:tasks.rados.rados.0.smithi107.stdout:9786: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:06.891 INFO:tasks.rados.rados.0.smithi107.stdout:9786: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:06.892 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 733 (ObjNum 9785 snap 0 seq_num 9785) dirty exists 2020-06-12T02:39:06.892 INFO:tasks.rados.rados.0.smithi107.stdout:9786: left oid 20 (ObjNum 9785 snap 0 seq_num 9785) 2020-06-12T02:39:06.902 INFO:tasks.rados.rados.0.smithi107.stdout:9790: finishing write tid 1 to smithi10729407-13 2020-06-12T02:39:06.902 INFO:tasks.rados.rados.0.smithi107.stdout:9786: done (2 left) 2020-06-12T02:39:06.903 INFO:tasks.rados.rados.0.smithi107.stdout:9791: done (1 left) 2020-06-12T02:39:06.903 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:39:06.903 INFO:tasks.rados.rados.0.smithi107.stdout:9792: seq_num 9791 ranges {2834432=786432} 2020-06-12T02:39:06.908 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:39:06.906+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:39:06.920 INFO:tasks.rados.rados.0.smithi107.stdout:9792: writing smithi10729407-11 from 2834432 to 3620864 tid 1 2020-06-12T02:39:06.920 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:39:06.920 INFO:tasks.rados.rados.0.smithi107.stdout:9793: seq_num 9792 ranges {0=802816} 2020-06-12T02:39:06.927 INFO:tasks.rados.rados.0.smithi107.stdout:9793: writing smithi10729407-19 from 0 to 802816 tid 1 2020-06-12T02:39:06.927 INFO:tasks.rados.rados.0.smithi107.stdout:9794: delete oid 2 current snap is 0 2020-06-12T02:39:06.927 INFO:tasks.rados.rados.0.smithi107.stdout:9790: finishing write tid 3 to smithi10729407-13 2020-06-12T02:39:06.927 INFO:tasks.rados.rados.0.smithi107.stdout:9790: finishing write tid 4 to smithi10729407-13 2020-06-12T02:39:06.928 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1701 (ObjNum 9789 snap 0 seq_num 9789) dirty exists 2020-06-12T02:39:06.928 INFO:tasks.rados.rados.0.smithi107.stdout:9790: left oid 13 (ObjNum 9789 snap 0 seq_num 9789) 2020-06-12T02:39:06.928 INFO:tasks.rados.rados.0.smithi107.stdout:9790: done (3 left) 2020-06-12T02:39:06.929 INFO:tasks.rados.rados.0.smithi107.stdout:9794: done (2 left) 2020-06-12T02:39:06.929 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:39:06.929 INFO:tasks.rados.rados.0.smithi107.stdout:9795: seq_num 9794 ranges {1179648=434176,1613824=401408} 2020-06-12T02:39:06.937 INFO:tasks.rados.rados.0.smithi107.stdout:9795: writing smithi10729407-14 from 1179648 to 1613824 tid 1 2020-06-12T02:39:06.940 INFO:tasks.rados.rados.0.smithi107.stdout:9795: writing smithi10729407-14 from 1613824 to 2015232 tid 2 2020-06-12T02:39:06.941 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:39:06.941 INFO:tasks.rados.rados.0.smithi107.stdout:9796: seq_num 9795 ranges {0=450560,450560=401408} 2020-06-12T02:39:06.942 INFO:tasks.rados.rados.0.smithi107.stdout:9796: writing smithi10729407-16 from 0 to 450560 tid 1 2020-06-12T02:39:06.945 INFO:tasks.rados.rados.0.smithi107.stdout:9796: writing smithi10729407-16 from 450560 to 851968 tid 2 2020-06-12T02:39:06.945 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:39:06.945 INFO:tasks.rados.rados.0.smithi107.stdout:9797: seq_num 9796 ranges {0=466944,466944=401408} 2020-06-12T02:39:06.948 INFO:tasks.rados.rados.0.smithi107.stdout:9797: writing smithi10729407-5 from 0 to 466944 tid 1 2020-06-12T02:39:06.954 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:39:06.954 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:39:06.954 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2020-06-12T02:39:06.954 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2020-06-12T02:39:06.954 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:39:06.955 INFO:tasks.rados.rados.0.smithi107.stdout:9797: writing smithi10729407-5 from 466944 to 868352 tid 2 2020-06-12T02:39:06.955 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:39:06.955 INFO:tasks.rados.rados.0.smithi107.stdout:9798: seq_num 9797 ranges {598016=483328,1081344=401408} 2020-06-12T02:39:06.957 INFO:tasks.rados.rados.0.smithi107.stdout:9798: writing smithi10729407-15 from 598016 to 1081344 tid 1 2020-06-12T02:39:06.962 INFO:tasks.rados.rados.0.smithi107.stdout:9798: writing smithi10729407-15 from 1081344 to 1482752 tid 2 2020-06-12T02:39:06.962 INFO:tasks.rados.rados.0.smithi107.stdout:9799: delete oid 8 current snap is 0 2020-06-12T02:39:06.962 INFO:tasks.rados.rados.0.smithi107.stdout:9792: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:06.962 INFO:tasks.rados.rados.0.smithi107.stdout:9793: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:06.963 INFO:tasks.rados.rados.0.smithi107.stdout:9792: finishing write tid 3 to smithi10729407-11 2020-06-12T02:39:06.963 INFO:tasks.rados.rados.0.smithi107.stdout:9792: finishing write tid 4 to smithi10729407-11 2020-06-12T02:39:06.963 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1656 (ObjNum 9791 snap 0 seq_num 9791) dirty exists 2020-06-12T02:39:06.963 INFO:tasks.rados.rados.0.smithi107.stdout:9792: left oid 11 (ObjNum 9791 snap 0 seq_num 9791) 2020-06-12T02:39:06.963 INFO:tasks.rados.rados.0.smithi107.stdout:9793: finishing write tid 3 to smithi10729407-19 2020-06-12T02:39:06.963 INFO:tasks.rados.rados.0.smithi107.stdout:9793: finishing write tid 4 to smithi10729407-19 2020-06-12T02:39:06.964 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1808 (ObjNum 9792 snap 0 seq_num 9792) dirty exists 2020-06-12T02:39:06.964 INFO:tasks.rados.rados.0.smithi107.stdout:9793: left oid 19 (ObjNum 9792 snap 0 seq_num 9792) 2020-06-12T02:39:06.964 INFO:tasks.rados.rados.0.smithi107.stdout:9796: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:06.964 INFO:tasks.rados.rados.0.smithi107.stdout:9795: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:06.964 INFO:tasks.rados.rados.0.smithi107.stdout:9795: finishing write tid 2 to smithi10729407-14 2020-06-12T02:39:06.965 INFO:tasks.rados.rados.0.smithi107.stdout:9795: finishing write tid 4 to smithi10729407-14 2020-06-12T02:39:06.965 INFO:tasks.rados.rados.0.smithi107.stdout:9797: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:06.965 INFO:tasks.rados.rados.0.smithi107.stdout:9795: finishing write tid 5 to smithi10729407-14 2020-06-12T02:39:06.965 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1625 (ObjNum 9794 snap 0 seq_num 9794) dirty exists 2020-06-12T02:39:06.965 INFO:tasks.rados.rados.0.smithi107.stdout:9795: left oid 14 (ObjNum 9794 snap 0 seq_num 9794) 2020-06-12T02:39:06.966 INFO:tasks.rados.rados.0.smithi107.stdout:9796: finishing write tid 2 to smithi10729407-16 2020-06-12T02:39:06.966 INFO:tasks.rados.rados.0.smithi107.stdout:9796: finishing write tid 4 to smithi10729407-16 2020-06-12T02:39:06.966 INFO:tasks.rados.rados.0.smithi107.stdout:9796: finishing write tid 5 to smithi10729407-16 2020-06-12T02:39:06.966 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1752 (ObjNum 9795 snap 0 seq_num 9795) dirty exists 2020-06-12T02:39:06.966 INFO:tasks.rados.rados.0.smithi107.stdout:9796: left oid 16 (ObjNum 9795 snap 0 seq_num 9795) 2020-06-12T02:39:06.968 INFO:tasks.rados.rados.0.smithi107.stdout:9797: finishing write tid 2 to smithi10729407-5 2020-06-12T02:39:06.968 INFO:tasks.rados.rados.0.smithi107.stdout:9797: finishing write tid 4 to smithi10729407-5 2020-06-12T02:39:06.968 INFO:tasks.rados.rados.0.smithi107.stdout:9797: finishing write tid 5 to smithi10729407-5 2020-06-12T02:39:06.968 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 939 (ObjNum 9796 snap 0 seq_num 9796) dirty exists 2020-06-12T02:39:06.968 INFO:tasks.rados.rados.0.smithi107.stdout:9797: left oid 5 (ObjNum 9796 snap 0 seq_num 9796) 2020-06-12T02:39:06.969 INFO:tasks.rados.rados.0.smithi107.stdout:9798: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:06.969 INFO:tasks.rados.rados.0.smithi107.stdout:9798: finishing write tid 2 to smithi10729407-15 2020-06-12T02:39:06.969 INFO:tasks.rados.rados.0.smithi107.stdout:9798: finishing write tid 4 to smithi10729407-15 2020-06-12T02:39:06.970 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-06-12T02:39:06.971 INFO:tasks.rados.rados.0.smithi107.stdout:9798: finishing write tid 5 to smithi10729407-15 2020-06-12T02:39:06.972 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1820 (ObjNum 9797 snap 0 seq_num 9797) dirty exists 2020-06-12T02:39:06.972 INFO:tasks.rados.rados.0.smithi107.stdout:9798: left oid 15 (ObjNum 9797 snap 0 seq_num 9797) 2020-06-12T02:39:06.972 INFO:tasks.rados.rados.0.smithi107.stdout:9792: done (6 left) 2020-06-12T02:39:06.972 INFO:tasks.rados.rados.0.smithi107.stdout:9793: done (5 left) 2020-06-12T02:39:06.972 INFO:tasks.rados.rados.0.smithi107.stdout:9795: done (4 left) 2020-06-12T02:39:06.973 INFO:tasks.rados.rados.0.smithi107.stdout:9796: done (3 left) 2020-06-12T02:39:06.973 INFO:tasks.rados.rados.0.smithi107.stdout:9797: done (2 left) 2020-06-12T02:39:06.973 INFO:tasks.rados.rados.0.smithi107.stdout:9798: done (1 left) 2020-06-12T02:39:06.973 INFO:tasks.rados.rados.0.smithi107.stdout:9799: done (0 left) 2020-06-12T02:39:06.973 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:39:06.974 INFO:tasks.rados.rados.0.smithi107.stdout:9800: seq_num 9799 ranges {0=516096,516096=401408} 2020-06-12T02:39:06.974 INFO:tasks.rados.rados.0.smithi107.stdout:9800: writing smithi10729407-9 from 0 to 516096 tid 1 2020-06-12T02:39:06.975 INFO:tasks.rados.rados.0.smithi107.stdout:9800: writing smithi10729407-9 from 516096 to 917504 tid 2 2020-06-12T02:39:06.976 INFO:tasks.rados.rados.0.smithi107.stdout:9801: delete oid 7 current snap is 0 2020-06-12T02:39:06.979 INFO:tasks.rados.rados.0.smithi107.stdout:9801: done (1 left) 2020-06-12T02:39:06.980 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:39:06.980 INFO:tasks.rados.rados.0.smithi107.stdout:9802: seq_num 9801 ranges {0=557056,557056=401408} 2020-06-12T02:39:06.982 INFO:tasks.rados.rados.0.smithi107.stdout:9802: writing smithi10729407-17 from 0 to 557056 tid 1 2020-06-12T02:39:06.984 INFO:tasks.rados.rados.0.smithi107.stdout:9802: writing smithi10729407-17 from 557056 to 958464 tid 2 2020-06-12T02:39:06.984 INFO:tasks.rados.rados.0.smithi107.stdout:9803: delete oid 12 current snap is 0 2020-06-12T02:39:06.985 INFO:tasks.rados.rados.0.smithi107.stdout:9800: finishing write tid 1 to smithi10729407-9 2020-06-12T02:39:06.989 INFO:tasks.rados.rados.0.smithi107.stdout:9800: finishing write tid 2 to smithi10729407-9 2020-06-12T02:39:06.989 INFO:tasks.rados.rados.0.smithi107.stdout:9800: finishing write tid 4 to smithi10729407-9 2020-06-12T02:39:06.990 INFO:tasks.rados.rados.0.smithi107.stdout:9803: done (2 left) 2020-06-12T02:39:06.990 INFO:tasks.rados.rados.0.smithi107.stdout:9804: delete oid 2 current snap is 0 2020-06-12T02:39:06.990 INFO:tasks.rados.rados.0.smithi107.stdout:9800: finishing write tid 5 to smithi10729407-9 2020-06-12T02:39:06.991 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1771 (ObjNum 9799 snap 0 seq_num 9799) dirty exists 2020-06-12T02:39:06.991 INFO:tasks.rados.rados.0.smithi107.stdout:9800: left oid 9 (ObjNum 9799 snap 0 seq_num 9799) 2020-06-12T02:39:06.992 INFO:tasks.rados.rados.0.smithi107.stdout:9800: done (2 left) 2020-06-12T02:39:06.992 INFO:tasks.rados.rados.0.smithi107.stdout:9804: done (1 left) 2020-06-12T02:39:06.992 INFO:tasks.rados.rados.0.smithi107.stdout:9805: delete oid 11 current snap is 0 2020-06-12T02:39:06.994 INFO:tasks.rados.rados.0.smithi107.stdout:9802: finishing write tid 1 to smithi10729407-17 2020-06-12T02:39:06.994 INFO:tasks.rados.rados.0.smithi107.stdout:9802: finishing write tid 2 to smithi10729407-17 2020-06-12T02:39:06.994 INFO:tasks.rados.rados.0.smithi107.stdout:9802: finishing write tid 4 to smithi10729407-17 2020-06-12T02:39:06.995 INFO:tasks.rados.rados.0.smithi107.stdout:9805: done (1 left) 2020-06-12T02:39:06.995 INFO:tasks.rados.rados.0.smithi107.stdout:9806: delete oid 10 current snap is 0 2020-06-12T02:39:06.996 INFO:tasks.rados.rados.0.smithi107.stdout:9806: done (1 left) 2020-06-12T02:39:06.997 INFO:tasks.rados.rados.0.smithi107.stdout:9807: delete oid 18 current snap is 0 2020-06-12T02:39:06.997 INFO:tasks.rados.rados.0.smithi107.stdout:9802: finishing write tid 5 to smithi10729407-17 2020-06-12T02:39:06.997 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1839 (ObjNum 9801 snap 0 seq_num 9801) dirty exists 2020-06-12T02:39:06.998 INFO:tasks.rados.rados.0.smithi107.stdout:9802: left oid 17 (ObjNum 9801 snap 0 seq_num 9801) 2020-06-12T02:39:07.003 INFO:tasks.rados.rados.0.smithi107.stdout:9802: done (1 left) 2020-06-12T02:39:07.003 INFO:tasks.rados.rados.0.smithi107.stdout:9807: done (0 left) 2020-06-12T02:39:07.004 INFO:tasks.rados.rados.0.smithi107.stdout:9808: delete oid 2 current snap is 0 2020-06-12T02:39:07.004 INFO:tasks.rados.rados.0.smithi107.stdout:9808: done (0 left) 2020-06-12T02:39:07.004 INFO:tasks.rados.rados.0.smithi107.stdout:9809: delete oid 19 current snap is 0 2020-06-12T02:39:07.005 INFO:tasks.rados.rados.0.smithi107.stdout:9809: done (0 left) 2020-06-12T02:39:07.006 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:39:07.006 INFO:tasks.rados.rados.0.smithi107.stdout:9810: seq_num 9809 ranges {0=688128,688128=401408} 2020-06-12T02:39:07.009 INFO:tasks.rados.rados.0.smithi107.stdout:9810: writing smithi10729407-7 from 0 to 688128 tid 1 2020-06-12T02:39:07.010 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:39:07.007+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:39:07.018 INFO:tasks.rados.rados.0.smithi107.stdout:9810: writing smithi10729407-7 from 688128 to 1089536 tid 2 2020-06-12T02:39:07.018 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:39:07.018 INFO:tasks.rados.rados.0.smithi107.stdout:9811: seq_num 9810 ranges {2285568=704512,2990080=401408} 2020-06-12T02:39:07.019 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:39:07.050 INFO:tasks.rados.rados.0.smithi107.stdout:9811: writing smithi10729407-6 from 2285568 to 2990080 tid 1 2020-06-12T02:39:07.050 INFO:tasks.rados.rados.0.smithi107.stdout:9811: writing smithi10729407-6 from 2990080 to 3391488 tid 2 2020-06-12T02:39:07.051 INFO:tasks.rados.rados.0.smithi107.stdout:9812: delete oid 5 current snap is 0 2020-06-12T02:39:07.051 INFO:tasks.rados.rados.0.smithi107.stdout:9810: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:07.051 INFO:tasks.rados.rados.0.smithi107.stdout:9810: finishing write tid 2 to smithi10729407-7 2020-06-12T02:39:07.051 INFO:tasks.rados.rados.0.smithi107.stdout:9810: finishing write tid 4 to smithi10729407-7 2020-06-12T02:39:07.052 INFO:tasks.rados.rados.0.smithi107.stdout:9810: finishing write tid 5 to smithi10729407-7 2020-06-12T02:39:07.052 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1679 (ObjNum 9809 snap 0 seq_num 9809) dirty exists 2020-06-12T02:39:07.052 INFO:tasks.rados.rados.0.smithi107.stdout:9810: left oid 7 (ObjNum 9809 snap 0 seq_num 9809) 2020-06-12T02:39:07.052 INFO:tasks.rados.rados.0.smithi107.stdout:9810: done (2 left) 2020-06-12T02:39:07.052 INFO:tasks.rados.rados.0.smithi107.stdout:9812: done (1 left) 2020-06-12T02:39:07.052 INFO:tasks.rados.rados.0.smithi107.stdout:9813: delete oid 10 current snap is 0 2020-06-12T02:39:07.053 INFO:tasks.rados.rados.0.smithi107.stdout:9811: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:07.053 INFO:tasks.rados.rados.0.smithi107.stdout:9811: finishing write tid 2 to smithi10729407-6 2020-06-12T02:39:07.053 INFO:tasks.rados.rados.0.smithi107.stdout:9813: done (1 left) 2020-06-12T02:39:07.053 INFO:tasks.rados.rados.0.smithi107.stdout:9814: delete oid 7 current snap is 0 2020-06-12T02:39:07.053 INFO:tasks.rados.rados.0.smithi107.stdout:9811: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:07.054 INFO:tasks.rados.rados.0.smithi107.stdout:9811: finishing write tid 5 to smithi10729407-6 2020-06-12T02:39:07.054 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1761 (ObjNum 9810 snap 0 seq_num 9810) dirty exists 2020-06-12T02:39:07.054 INFO:tasks.rados.rados.0.smithi107.stdout:9811: left oid 6 (ObjNum 9810 snap 0 seq_num 9810) 2020-06-12T02:39:07.054 INFO:tasks.rados.rados.0.smithi107.stdout:9811: done (1 left) 2020-06-12T02:39:07.054 INFO:tasks.rados.rados.0.smithi107.stdout:9814: done (0 left) 2020-06-12T02:39:07.055 INFO:tasks.rados.rados.0.smithi107.stdout:9815: delete oid 13 current snap is 0 2020-06-12T02:39:07.055 INFO:tasks.rados.rados.0.smithi107.stdout:9815: done (0 left) 2020-06-12T02:39:07.055 INFO:tasks.rados.rados.0.smithi107.stdout:9816: delete oid 19 current snap is 0 2020-06-12T02:39:07.055 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:39:07.056 INFO:tasks.rados.rados.0.smithi107.stdout:9816: done (0 left) 2020-06-12T02:39:07.057 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:39:07.057 INFO:tasks.rados.rados.0.smithi107.stdout:9817: seq_num 9816 ranges {0=802816,802816=401408} 2020-06-12T02:39:07.057 INFO:tasks.rados.rados.0.smithi107.stdout:9817: writing smithi10729407-4 from 0 to 802816 tid 1 2020-06-12T02:39:07.057 INFO:tasks.rados.rados.0.smithi107.stdout:9817: writing smithi10729407-4 from 802816 to 1204224 tid 2 2020-06-12T02:39:07.057 INFO:tasks.rados.rados.0.smithi107.stdout:9818: delete oid 18 current snap is 0 2020-06-12T02:39:07.062 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:39:07.062 INFO:tasks.rados.rados.0.smithi107.stdout:9817: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:07.062 INFO:tasks.rados.rados.0.smithi107.stdout:9818: done (1 left) 2020-06-12T02:39:07.063 INFO:tasks.rados.rados.0.smithi107.stdout:9819: delete oid 9 current snap is 0 2020-06-12T02:39:07.063 INFO:tasks.rados.rados.0.smithi107.stdout:9817: finishing write tid 2 to smithi10729407-4 2020-06-12T02:39:07.065 INFO:tasks.rados.rados.0.smithi107.stdout:9817: finishing write tid 4 to smithi10729407-4 2020-06-12T02:39:07.067 INFO:tasks.rados.rados.0.smithi107.stdout:9817: finishing write tid 5 to smithi10729407-4 2020-06-12T02:39:07.067 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1842 (ObjNum 9816 snap 0 seq_num 9816) dirty exists 2020-06-12T02:39:07.067 INFO:tasks.rados.rados.0.smithi107.stdout:9817: left oid 4 (ObjNum 9816 snap 0 seq_num 9816) 2020-06-12T02:39:07.068 INFO:tasks.rados.rados.0.smithi107.stdout:9817: done (1 left) 2020-06-12T02:39:07.068 INFO:tasks.rados.rados.0.smithi107.stdout:9819: done (0 left) 2020-06-12T02:39:07.068 INFO:tasks.rados.rados.0.smithi107.stdout:9820: delete oid 12 current snap is 0 2020-06-12T02:39:07.070 INFO:tasks.rados.rados.0.smithi107.stdout:9820: done (0 left) 2020-06-12T02:39:07.071 INFO:tasks.rados.rados.0.smithi107.stdout:9821: delete oid 13 current snap is 0 2020-06-12T02:39:07.073 INFO:tasks.rados.rados.0.smithi107.stdout:9821: done (0 left) 2020-06-12T02:39:07.073 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:39:07.074 INFO:tasks.rados.rados.0.smithi107.stdout:9822: seq_num 9821 ranges {0=491520,491520=565248,1056768=237568} 2020-06-12T02:39:07.076 INFO:tasks.rados.rados.0.smithi107.stdout:9822: writing smithi10729407-18 from 0 to 491520 tid 1 2020-06-12T02:39:07.079 INFO:tasks.rados.rados.0.smithi107.stdout:9822: writing smithi10729407-18 from 491520 to 1056768 tid 2 2020-06-12T02:39:07.081 INFO:tasks.rados.rados.0.smithi107.stdout:9822: writing smithi10729407-18 from 1056768 to 1294336 tid 3 2020-06-12T02:39:07.081 INFO:tasks.rados.rados.0.smithi107.stdout:9823: delete oid 10 current snap is 0 2020-06-12T02:39:07.082 INFO:tasks.rados.rados.0.smithi107.stdout:9822: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:07.084 INFO:tasks.rados.rados.0.smithi107.stdout:9823: done (1 left) 2020-06-12T02:39:07.084 INFO:tasks.rados.rados.0.smithi107.stdout:9824: delete oid 20 current snap is 0 2020-06-12T02:39:07.094 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:39:07.095 INFO:tasks.rados.rados.0.smithi107.stdout:9824: done (1 left) 2020-06-12T02:39:07.095 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:39:07.095 INFO:tasks.rados.rados.0.smithi107.stdout:9825: seq_num 9824 ranges {0=540672,540672=671744,1212416=131072} 2020-06-12T02:39:07.095 INFO:tasks.rados.rados.0.smithi107.stdout:9825: writing smithi10729407-10 from 0 to 540672 tid 1 2020-06-12T02:39:07.097 INFO:tasks.rados.rados.0.smithi107.stdout:9825: writing smithi10729407-10 from 540672 to 1212416 tid 2 2020-06-12T02:39:07.099 INFO:tasks.rados.rados.0.smithi107.stdout:9825: writing smithi10729407-10 from 1212416 to 1343488 tid 3 2020-06-12T02:39:07.099 INFO:tasks.rados.rados.0.smithi107.stdout:9826: delete oid 8 current snap is 0 2020-06-12T02:39:07.099 INFO:tasks.rados.rados.0.smithi107.stdout:9822: finishing write tid 2 to smithi10729407-18 2020-06-12T02:39:07.099 INFO:tasks.rados.rados.0.smithi107.stdout:9822: finishing write tid 3 to smithi10729407-18 2020-06-12T02:39:07.100 INFO:tasks.rados.rados.0.smithi107.stdout:9822: finishing write tid 5 to smithi10729407-18 2020-06-12T02:39:07.100 INFO:tasks.rados.rados.0.smithi107.stdout:9822: finishing write tid 6 to smithi10729407-18 2020-06-12T02:39:07.100 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1776 (ObjNum 9821 snap 0 seq_num 9821) dirty exists 2020-06-12T02:39:07.100 INFO:tasks.rados.rados.0.smithi107.stdout:9822: left oid 18 (ObjNum 9821 snap 0 seq_num 9821) 2020-06-12T02:39:07.101 INFO:tasks.rados.rados.0.smithi107.stdout:9825: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:07.103 INFO:tasks.rados.rados.0.smithi107.stdout:9822: done (2 left) 2020-06-12T02:39:07.103 INFO:tasks.rados.rados.0.smithi107.stdout:9826: done (1 left) 2020-06-12T02:39:07.103 INFO:tasks.rados.rados.0.smithi107.stdout:9827: delete oid 6 current snap is 0 2020-06-12T02:39:07.106 INFO:tasks.rados.rados.0.smithi107.stdout:9827: done (1 left) 2020-06-12T02:39:07.107 INFO:tasks.rados.rados.0.smithi107.stdout:9828: delete oid 13 current snap is 0 2020-06-12T02:39:07.109 INFO:tasks.rados.rados.0.smithi107.stdout:9825: finishing write tid 2 to smithi10729407-10 2020-06-12T02:39:07.110 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:39:07.107+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:39:07.110 INFO:tasks.rados.rados.0.smithi107.stdout:9828: done (1 left) 2020-06-12T02:39:07.110 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:39:07.111 INFO:tasks.rados.rados.0.smithi107.stdout:9829: seq_num 9828 ranges {1482752=606208,2088960=606208,2695168=196608} 2020-06-12T02:39:07.116 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:39:07.123 INFO:tasks.rados.rados.0.smithi107.stdout:9829: writing smithi10729407-15 from 1482752 to 2088960 tid 1 2020-06-12T02:39:07.127 INFO:tasks.rados.rados.0.smithi107.stdout:9829: writing smithi10729407-15 from 2088960 to 2695168 tid 2 2020-06-12T02:39:07.130 INFO:tasks.rados.rados.0.smithi107.stdout:9829: writing smithi10729407-15 from 2695168 to 2891776 tid 3 2020-06-12T02:39:07.130 INFO:tasks.rados.rados.0.smithi107.stdout:9830: delete oid 16 current snap is 0 2020-06-12T02:39:07.131 INFO:tasks.rados.rados.0.smithi107.stdout:9825: finishing write tid 3 to smithi10729407-10 2020-06-12T02:39:07.131 INFO:tasks.rados.rados.0.smithi107.stdout:9825: finishing write tid 5 to smithi10729407-10 2020-06-12T02:39:07.131 INFO:tasks.rados.rados.0.smithi107.stdout:9825: finishing write tid 6 to smithi10729407-10 2020-06-12T02:39:07.131 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 710 (ObjNum 9824 snap 0 seq_num 9824) dirty exists 2020-06-12T02:39:07.131 INFO:tasks.rados.rados.0.smithi107.stdout:9825: left oid 10 (ObjNum 9824 snap 0 seq_num 9824) 2020-06-12T02:39:07.132 INFO:tasks.rados.rados.0.smithi107.stdout:9829: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:07.132 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:07.132 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:07.133 INFO:tasks.rados.rados.0.smithi107.stdout:9825: done (2 left) 2020-06-12T02:39:07.133 INFO:tasks.rados.rados.0.smithi107.stdout:9830: done (1 left) 2020-06-12T02:39:07.133 INFO:tasks.rados.rados.0.smithi107.stdout:9831: delete oid 4 current snap is 0 2020-06-12T02:39:07.144 INFO:tasks.rados.rados.0.smithi107.stdout:9831: done (1 left) 2020-06-12T02:39:07.144 INFO:tasks.rados.rados.0.smithi107.stdout:9832: delete oid 2 current snap is 0 2020-06-12T02:39:07.145 INFO:tasks.rados.rados.0.smithi107.stdout:9829: finishing write tid 2 to smithi10729407-15 2020-06-12T02:39:07.145 INFO:tasks.rados.rados.0.smithi107.stdout:9829: finishing write tid 3 to smithi10729407-15 2020-06-12T02:39:07.145 INFO:tasks.rados.rados.0.smithi107.stdout:9829: finishing write tid 5 to smithi10729407-15 2020-06-12T02:39:07.146 INFO:tasks.rados.rados.0.smithi107.stdout:9832: done (1 left) 2020-06-12T02:39:07.146 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:39:07.146 INFO:tasks.rados.rados.0.smithi107.stdout:9833: seq_num 9832 ranges {0=671744,671744=589824,1261568=212992} 2020-06-12T02:39:07.147 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:39:07.147 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-06-12T02:39:07.148 INFO:tasks.rados.rados.0.smithi107.stdout:9833: writing smithi10729407-4 from 0 to 671744 tid 1 2020-06-12T02:39:07.149 INFO:tasks.rados.rados.0.smithi107.stdout:9833: writing smithi10729407-4 from 671744 to 1261568 tid 2 2020-06-12T02:39:07.150 INFO:tasks.rados.rados.0.smithi107.stdout:9833: writing smithi10729407-4 from 1261568 to 1474560 tid 3 2020-06-12T02:39:07.151 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:39:07.151 INFO:tasks.rados.rados.0.smithi107.stdout:9834: seq_num 9833 ranges {0=688128,688128=475136,1163264=327680} 2020-06-12T02:39:07.154 INFO:tasks.rados.rados.0.smithi107.stdout:9834: writing smithi10729407-3 from 0 to 688128 tid 1 2020-06-12T02:39:07.158 INFO:tasks.rados.rados.0.smithi107.stdout:9834: writing smithi10729407-3 from 688128 to 1163264 tid 2 2020-06-12T02:39:07.160 INFO:tasks.rados.rados.0.smithi107.stdout:9834: writing smithi10729407-3 from 1163264 to 1490944 tid 3 2020-06-12T02:39:07.160 INFO:tasks.rados.rados.0.smithi107.stdout:9835: delete oid 16 current snap is 0 2020-06-12T02:39:07.160 INFO:tasks.rados.rados.0.smithi107.stdout:9829: finishing write tid 6 to smithi10729407-15 2020-06-12T02:39:07.160 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1826 (ObjNum 9828 snap 0 seq_num 9828) dirty exists 2020-06-12T02:39:07.161 INFO:tasks.rados.rados.0.smithi107.stdout:9829: left oid 15 (ObjNum 9828 snap 0 seq_num 9828) 2020-06-12T02:39:07.161 INFO:tasks.rados.rados.0.smithi107.stdout:9833: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:07.161 INFO:tasks.rados.rados.0.smithi107.stdout:9833: finishing write tid 2 to smithi10729407-4 2020-06-12T02:39:07.162 INFO:tasks.rados.rados.0.smithi107.stdout:9834: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:07.162 INFO:tasks.rados.rados.0.smithi107.stdout:9833: finishing write tid 3 to smithi10729407-4 2020-06-12T02:39:07.163 INFO:tasks.rados.rados.0.smithi107.stdout:9833: finishing write tid 5 to smithi10729407-4 2020-06-12T02:39:07.164 INFO:tasks.rados.rados.0.smithi107.stdout:9833: finishing write tid 6 to smithi10729407-4 2020-06-12T02:39:07.164 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1847 (ObjNum 9832 snap 0 seq_num 9832) dirty exists 2020-06-12T02:39:07.165 INFO:tasks.rados.rados.0.smithi107.stdout:9833: left oid 4 (ObjNum 9832 snap 0 seq_num 9832) 2020-06-12T02:39:07.170 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:39:07.170 INFO:tasks.rados.rados.0.smithi107.stdout:9829: done (3 left) 2020-06-12T02:39:07.171 INFO:tasks.rados.rados.0.smithi107.stdout:9833: done (2 left) 2020-06-12T02:39:07.171 INFO:tasks.rados.rados.0.smithi107.stdout:9835: done (1 left) 2020-06-12T02:39:07.171 INFO:tasks.rados.rados.0.smithi107.stdout:9836: delete oid 18 current snap is 0 2020-06-12T02:39:07.171 INFO:tasks.rados.rados.0.smithi107.stdout:9834: finishing write tid 2 to smithi10729407-3 2020-06-12T02:39:07.172 INFO:tasks.rados.rados.0.smithi107.stdout:9834: finishing write tid 3 to smithi10729407-3 2020-06-12T02:39:07.173 INFO:tasks.rados.rados.0.smithi107.stdout:9836: done (1 left) 2020-06-12T02:39:07.173 INFO:tasks.rados.rados.0.smithi107.stdout:9837: delete oid 7 current snap is 0 2020-06-12T02:39:07.176 INFO:tasks.rados.rados.0.smithi107.stdout:9834: finishing write tid 5 to smithi10729407-3 2020-06-12T02:39:07.176 INFO:tasks.rados.rados.0.smithi107.stdout:9837: done (1 left) 2020-06-12T02:39:07.176 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:39:07.176 INFO:tasks.rados.rados.0.smithi107.stdout:9838: seq_num 9837 ranges {0=753664,753664=417792,1171456=385024} 2020-06-12T02:39:07.181 INFO:tasks.rados.rados.0.smithi107.stdout:9838: writing smithi10729407-16 from 0 to 753664 tid 1 2020-06-12T02:39:07.184 INFO:tasks.rados.rados.0.smithi107.stdout:9838: writing smithi10729407-16 from 753664 to 1171456 tid 2 2020-06-12T02:39:07.189 INFO:tasks.rados.rados.0.smithi107.stdout:9838: writing smithi10729407-16 from 1171456 to 1556480 tid 3 2020-06-12T02:39:07.189 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:39:07.190 INFO:tasks.rados.rados.0.smithi107.stdout:9839: seq_num 9838 ranges {0=770048,770048=745472,1515520=57344} 2020-06-12T02:39:07.193 INFO:tasks.rados.rados.0.smithi107.stdout:9839: writing smithi10729407-18 from 0 to 770048 tid 1 2020-06-12T02:39:07.199 INFO:tasks.rados.rados.0.smithi107.stdout:9839: writing smithi10729407-18 from 770048 to 1515520 tid 2 2020-06-12T02:39:07.200 INFO:tasks.rados.rados.0.smithi107.stdout:9839: writing smithi10729407-18 from 1515520 to 1572864 tid 3 2020-06-12T02:39:07.200 INFO:tasks.rados.rados.0.smithi107.stdout:9840: delete oid 1 current snap is 0 2020-06-12T02:39:07.200 INFO:tasks.rados.rados.0.smithi107.stdout:9834: finishing write tid 6 to smithi10729407-3 2020-06-12T02:39:07.201 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1758 (ObjNum 9833 snap 0 seq_num 9833) dirty exists 2020-06-12T02:39:07.201 INFO:tasks.rados.rados.0.smithi107.stdout:9834: left oid 3 (ObjNum 9833 snap 0 seq_num 9833) 2020-06-12T02:39:07.201 INFO:tasks.rados.rados.0.smithi107.stdout:9838: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:07.201 INFO:tasks.rados.rados.0.smithi107.stdout:9838: finishing write tid 2 to smithi10729407-16 2020-06-12T02:39:07.201 INFO:tasks.rados.rados.0.smithi107.stdout:9838: finishing write tid 3 to smithi10729407-16 2020-06-12T02:39:07.202 INFO:tasks.rados.rados.0.smithi107.stdout:9838: finishing write tid 5 to smithi10729407-16 2020-06-12T02:39:07.210 INFO:tasks.rados.rados.0.smithi107.stdout:9838: finishing write tid 6 to smithi10729407-16 2020-06-12T02:39:07.210 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1762 (ObjNum 9837 snap 0 seq_num 9837) dirty exists 2020-06-12T02:39:07.210 INFO:tasks.rados.rados.0.smithi107.stdout:9838: left oid 16 (ObjNum 9837 snap 0 seq_num 9837) 2020-06-12T02:39:07.210 INFO:tasks.rados.rados.0.smithi107.stdout:9834: done (3 left) 2020-06-12T02:39:07.211 INFO:tasks.rados.rados.0.smithi107.stdout:9838: done (2 left) 2020-06-12T02:39:07.211 INFO:tasks.rados.rados.0.smithi107.stdout:9840: done (1 left) 2020-06-12T02:39:07.211 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:39:07.211 INFO:tasks.rados.rados.0.smithi107.stdout:9841: seq_num 9840 ranges {0=409600,409600=516096,925696=630784,1556480=57344} 2020-06-12T02:39:07.212 INFO:tasks.rados.rados.0.smithi107.stdout:9841: writing smithi10729407-7 from 0 to 409600 tid 1 2020-06-12T02:39:07.212 INFO:tasks.rados.rados.0.smithi107.stdout:9841: writing smithi10729407-7 from 409600 to 925696 tid 2 2020-06-12T02:39:07.212 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2020-06-12T02:39:07.212 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-06-12T02:39:07.213 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-06-12T02:39:07.213 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-06-12T02:39:07.213 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-06-12T02:39:07.213 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-06-12T02:39:07.213 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-06-12T02:39:07.214 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-06-12T02:39:07.214 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-06-12T02:39:07.216 INFO:tasks.rados.rados.0.smithi107.stdout:9841: writing smithi10729407-7 from 925696 to 1556480 tid 3 2020-06-12T02:39:07.224 INFO:tasks.rados.rados.0.smithi107.stdout:9841: writing smithi10729407-7 from 1556480 to 1613824 tid 4 2020-06-12T02:39:07.225 INFO:tasks.rados.rados.0.smithi107.stdout:9842: delete oid 4 current snap is 0 2020-06-12T02:39:07.225 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:39:07.213+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:39:07.226 INFO:tasks.rados.rados.0.smithi107.stdout:9841: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:07.227 INFO:tasks.rados.rados.0.smithi107.stdout:9842: done (2 left) 2020-06-12T02:39:07.227 INFO:tasks.rados.rados.0.smithi107.stdout:9843: delete oid 20 current snap is 0 2020-06-12T02:39:07.228 INFO:tasks.rados.rados.0.smithi107.stdout:9841: finishing write tid 2 to smithi10729407-7 2020-06-12T02:39:07.228 INFO:tasks.rados.rados.0.smithi107.stdout:9841: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:07.229 INFO:tasks.rados.rados.0.smithi107.stdout:9841: finishing write tid 4 to smithi10729407-7 2020-06-12T02:39:07.229 INFO:tasks.rados.rados.0.smithi107.stdout:9841: finishing write tid 6 to smithi10729407-7 2020-06-12T02:39:07.230 INFO:tasks.rados.rados.0.smithi107.stdout:9841: finishing write tid 7 to smithi10729407-7 2020-06-12T02:39:07.231 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1686 (ObjNum 9840 snap 0 seq_num 9840) dirty exists 2020-06-12T02:39:07.231 INFO:tasks.rados.rados.0.smithi107.stdout:9841: left oid 7 (ObjNum 9840 snap 0 seq_num 9840) 2020-06-12T02:39:07.232 INFO:tasks.rados.rados.0.smithi107.stdout:9839: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:07.287 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:07.288 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:07.288 INFO:tasks.rados.rados.0.smithi107.stdout:9841: done (2 left) 2020-06-12T02:39:07.289 INFO:tasks.rados.rados.0.smithi107.stdout:9843: done (1 left) 2020-06-12T02:39:07.289 INFO:tasks.rados.rados.0.smithi107.stdout:9844: delete oid 6 current snap is 0 2020-06-12T02:39:07.292 INFO:tasks.rados.rados.0.smithi107.stdout:9844: done (1 left) 2020-06-12T02:39:07.292 INFO:tasks.rados.rados.0.smithi107.stdout:9845: delete oid 16 current snap is 0 2020-06-12T02:39:07.294 INFO:tasks.rados.rados.0.smithi107.stdout:9845: done (1 left) 2020-06-12T02:39:07.295 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:39:07.295 INFO:tasks.rados.rados.0.smithi107.stdout:9846: seq_num 9845 ranges {1613824=491520,2105344=737280,2842624=466944} 2020-06-12T02:39:07.298 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-06-12T02:39:07.310 INFO:tasks.rados.rados.0.smithi107.stdout:9846: writing smithi10729407-7 from 1613824 to 2105344 tid 1 2020-06-12T02:39:07.311 INFO:tasks.rados.rados.0.smithi107.stdout:9846: writing smithi10729407-7 from 2105344 to 2842624 tid 2 2020-06-12T02:39:07.314 INFO:tasks.rados.rados.0.smithi107.stdout:9846: writing smithi10729407-7 from 2842624 to 3309568 tid 3 2020-06-12T02:39:07.314 INFO:tasks.rados.rados.0.smithi107.stdout:9847: delete oid 17 current snap is 0 2020-06-12T02:39:07.314 INFO:tasks.rados.rados.0.smithi107.stdout:9839: finishing write tid 2 to smithi10729407-18 2020-06-12T02:39:07.315 INFO:tasks.rados.rados.0.smithi107.stdout:9839: finishing write tid 3 to smithi10729407-18 2020-06-12T02:39:07.315 INFO:tasks.rados.rados.0.smithi107.stdout:9839: finishing write tid 5 to smithi10729407-18 2020-06-12T02:39:07.315 INFO:tasks.rados.rados.0.smithi107.stdout:9839: finishing write tid 6 to smithi10729407-18 2020-06-12T02:39:07.315 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1781 (ObjNum 9838 snap 0 seq_num 9838) dirty exists 2020-06-12T02:39:07.316 INFO:tasks.rados.rados.0.smithi107.stdout:9839: left oid 18 (ObjNum 9838 snap 0 seq_num 9838) 2020-06-12T02:39:07.316 INFO:tasks.rados.rados.0.smithi107.stdout:9846: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:07.317 INFO:tasks.rados.rados.0.smithi107.stdout:9839: done (2 left) 2020-06-12T02:39:07.317 INFO:tasks.rados.rados.0.smithi107.stdout:9847: done (1 left) 2020-06-12T02:39:07.317 INFO:tasks.rados.rados.0.smithi107.stdout:9848: delete oid 8 current snap is 0 2020-06-12T02:39:07.317 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:39:07.315+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:39:07.322 INFO:tasks.rados.rados.0.smithi107.stdout:9848: done (1 left) 2020-06-12T02:39:07.322 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:39:07.323 INFO:tasks.rados.rados.0.smithi107.stdout:9849: seq_num 9848 ranges {0=540672,540672=442368,983040=589824,1572864=172032} 2020-06-12T02:39:07.329 INFO:tasks.rados.rados.0.smithi107.stdout:9849: writing smithi10729407-8 from 0 to 540672 tid 1 2020-06-12T02:39:07.330 INFO:tasks.rados.rados.0.smithi107.stdout:9849: writing smithi10729407-8 from 540672 to 983040 tid 2 2020-06-12T02:39:07.333 INFO:tasks.rados.rados.0.smithi107.stdout:9849: writing smithi10729407-8 from 983040 to 1572864 tid 3 2020-06-12T02:39:07.333 INFO:tasks.rados.rados.0.smithi107.stdout:9849: writing smithi10729407-8 from 1572864 to 1744896 tid 4 2020-06-12T02:39:07.334 INFO:tasks.rados.rados.0.smithi107.stdout:9850: delete oid 3 current snap is 0 2020-06-12T02:39:07.334 INFO:tasks.rados.rados.0.smithi107.stdout:9846: finishing write tid 2 to smithi10729407-7 2020-06-12T02:39:07.334 INFO:tasks.rados.rados.0.smithi107.stdout:9846: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:07.334 INFO:tasks.rados.rados.0.smithi107.stdout:9846: finishing write tid 5 to smithi10729407-7 2020-06-12T02:39:07.335 INFO:tasks.rados.rados.0.smithi107.stdout:9846: finishing write tid 6 to smithi10729407-7 2020-06-12T02:39:07.335 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1690 (ObjNum 9845 snap 0 seq_num 9845) dirty exists 2020-06-12T02:39:07.335 INFO:tasks.rados.rados.0.smithi107.stdout:9846: left oid 7 (ObjNum 9845 snap 0 seq_num 9845) 2020-06-12T02:39:07.335 INFO:tasks.rados.rados.0.smithi107.stdout:9849: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:07.336 INFO:tasks.rados.rados.0.smithi107.stdout:9846: done (2 left) 2020-06-12T02:39:07.336 INFO:tasks.rados.rados.0.smithi107.stdout:9850: done (1 left) 2020-06-12T02:39:07.336 INFO:tasks.rados.rados.0.smithi107.stdout:9851: delete oid 4 current snap is 0 2020-06-12T02:39:07.340 INFO:tasks.rados.rados.0.smithi107.stdout:9851: done (1 left) 2020-06-12T02:39:07.340 INFO:tasks.rados.rados.0.smithi107.stdout:9852: delete oid 2 current snap is 0 2020-06-12T02:39:07.341 INFO:tasks.rados.rados.0.smithi107.stdout:9852: done (1 left) 2020-06-12T02:39:07.342 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:39:07.342 INFO:tasks.rados.rados.0.smithi107.stdout:9853: seq_num 9852 ranges {0=606208,606208=778240,1384448=425984} 2020-06-12T02:39:07.345 INFO:tasks.rados.rados.0.smithi107.stdout:9853: writing smithi10729407-6 from 0 to 606208 tid 1 2020-06-12T02:39:07.349 INFO:tasks.rados.rados.0.smithi107.stdout:9853: writing smithi10729407-6 from 606208 to 1384448 tid 2 2020-06-12T02:39:07.354 INFO:tasks.rados.rados.0.smithi107.stdout:9853: writing smithi10729407-6 from 1384448 to 1810432 tid 3 2020-06-12T02:39:07.354 INFO:tasks.rados.rados.0.smithi107.stdout:9854: delete oid 17 current snap is 0 2020-06-12T02:39:07.355 INFO:tasks.rados.rados.0.smithi107.stdout:9849: finishing write tid 2 to smithi10729407-8 2020-06-12T02:39:07.355 INFO:tasks.rados.rados.0.smithi107.stdout:9849: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:07.355 INFO:tasks.rados.rados.0.smithi107.stdout:9849: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:07.355 INFO:tasks.rados.rados.0.smithi107.stdout:9849: finishing write tid 6 to smithi10729407-8 2020-06-12T02:39:07.355 INFO:tasks.rados.rados.0.smithi107.stdout:9849: finishing write tid 7 to smithi10729407-8 2020-06-12T02:39:07.356 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1832 (ObjNum 9848 snap 0 seq_num 9848) dirty exists 2020-06-12T02:39:07.356 INFO:tasks.rados.rados.0.smithi107.stdout:9849: left oid 8 (ObjNum 9848 snap 0 seq_num 9848) 2020-06-12T02:39:07.356 INFO:tasks.rados.rados.0.smithi107.stdout:9853: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:07.357 INFO:tasks.rados.rados.0.smithi107.stdout:9849: done (2 left) 2020-06-12T02:39:07.357 INFO:tasks.rados.rados.0.smithi107.stdout:9854: done (1 left) 2020-06-12T02:39:07.357 INFO:tasks.rados.rados.0.smithi107.stdout:9855: delete oid 17 current snap is 0 2020-06-12T02:39:07.359 INFO:tasks.rados.rados.0.smithi107.stdout:9855: done (1 left) 2020-06-12T02:39:07.359 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:39:07.359 INFO:tasks.rados.rados.0.smithi107.stdout:9856: seq_num 9855 ranges {0=655360,655360=483328,1138688=720896} 2020-06-12T02:39:07.370 INFO:tasks.rados.rados.0.smithi107.stdout:9856: writing smithi10729407-11 from 0 to 655360 tid 1 2020-06-12T02:39:07.371 INFO:tasks.rados.rados.0.smithi107.stdout:9856: writing smithi10729407-11 from 655360 to 1138688 tid 2 2020-06-12T02:39:07.371 INFO:tasks.rados.rados.0.smithi107.stdout:9856: writing smithi10729407-11 from 1138688 to 1859584 tid 3 2020-06-12T02:39:07.371 INFO:tasks.rados.rados.0.smithi107.stdout:9857: delete oid 8 current snap is 0 2020-06-12T02:39:07.372 INFO:tasks.rados.rados.0.smithi107.stdout:9853: finishing write tid 2 to smithi10729407-6 2020-06-12T02:39:07.372 INFO:tasks.rados.rados.0.smithi107.stdout:9853: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:07.372 INFO:tasks.rados.rados.0.smithi107.stdout:9853: finishing write tid 5 to smithi10729407-6 2020-06-12T02:39:07.372 INFO:tasks.rados.rados.0.smithi107.stdout:9853: finishing write tid 6 to smithi10729407-6 2020-06-12T02:39:07.372 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1768 (ObjNum 9852 snap 0 seq_num 9852) dirty exists 2020-06-12T02:39:07.374 INFO:tasks.rados.rados.0.smithi107.stdout:9853: left oid 6 (ObjNum 9852 snap 0 seq_num 9852) 2020-06-12T02:39:07.375 INFO:tasks.rados.rados.0.smithi107.stdout:9856: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:07.376 INFO:tasks.rados.rados.0.smithi107.stdout:9853: done (2 left) 2020-06-12T02:39:07.376 INFO:tasks.rados.rados.0.smithi107.stdout:9857: done (1 left) 2020-06-12T02:39:07.376 INFO:tasks.rados.rados.0.smithi107.stdout:9858: delete oid 17 current snap is 0 2020-06-12T02:39:07.381 INFO:tasks.rados.rados.0.smithi107.stdout:9858: done (1 left) 2020-06-12T02:39:07.381 INFO:tasks.rados.rados.0.smithi107.stdout:9859: delete oid 16 current snap is 0 2020-06-12T02:39:07.382 INFO:tasks.rados.rados.0.smithi107.stdout:9856: finishing write tid 2 to smithi10729407-11 2020-06-12T02:39:07.385 INFO:tasks.rados.rados.0.smithi107.stdout:9859: done (1 left) 2020-06-12T02:39:07.385 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:39:07.386 INFO:tasks.rados.rados.0.smithi107.stdout:9860: seq_num 9859 ranges {0=729088,729088=417792,1146880=729088,1875968=57344} 2020-06-12T02:39:07.389 INFO:tasks.rados.rados.0.smithi107.stdout:9860: writing smithi10729407-12 from 0 to 729088 tid 1 2020-06-12T02:39:07.393 INFO:tasks.rados.rados.0.smithi107.stdout:9860: writing smithi10729407-12 from 729088 to 1146880 tid 2 2020-06-12T02:39:07.395 INFO:tasks.rados.rados.0.smithi107.stdout:9860: writing smithi10729407-12 from 1146880 to 1875968 tid 3 2020-06-12T02:39:07.396 INFO:tasks.rados.rados.0.smithi107.stdout:9860: writing smithi10729407-12 from 1875968 to 1933312 tid 4 2020-06-12T02:39:07.397 INFO:tasks.rados.rados.0.smithi107.stdout:9861: delete oid 14 current snap is 0 2020-06-12T02:39:07.397 INFO:tasks.rados.rados.0.smithi107.stdout:9856: finishing write tid 3 to smithi10729407-11 2020-06-12T02:39:07.397 INFO:tasks.rados.rados.0.smithi107.stdout:9856: finishing write tid 5 to smithi10729407-11 2020-06-12T02:39:07.397 INFO:tasks.rados.rados.0.smithi107.stdout:9856: finishing write tid 6 to smithi10729407-11 2020-06-12T02:39:07.397 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1661 (ObjNum 9855 snap 0 seq_num 9855) dirty exists 2020-06-12T02:39:07.398 INFO:tasks.rados.rados.0.smithi107.stdout:9856: left oid 11 (ObjNum 9855 snap 0 seq_num 9855) 2020-06-12T02:39:07.398 INFO:tasks.rados.rados.0.smithi107.stdout:9860: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:07.400 INFO:tasks.rados.rados.0.smithi107.stdout:9856: done (2 left) 2020-06-12T02:39:07.400 INFO:tasks.rados.rados.0.smithi107.stdout:9861: done (1 left) 2020-06-12T02:39:07.400 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:39:07.400 INFO:tasks.rados.rados.0.smithi107.stdout:9862: seq_num 9861 ranges {0=761856,761856=638976,1400832=565248} 2020-06-12T02:39:07.405 INFO:tasks.rados.rados.0.smithi107.stdout:9862: writing smithi10729407-20 from 0 to 761856 tid 1 2020-06-12T02:39:07.409 INFO:tasks.rados.rados.0.smithi107.stdout:9862: writing smithi10729407-20 from 761856 to 1400832 tid 2 2020-06-12T02:39:07.413 INFO:tasks.rados.rados.0.smithi107.stdout:9862: writing smithi10729407-20 from 1400832 to 1966080 tid 3 2020-06-12T02:39:07.413 INFO:tasks.rados.rados.0.smithi107.stdout:9863: delete oid 6 current snap is 0 2020-06-12T02:39:07.414 INFO:tasks.rados.rados.0.smithi107.stdout:9860: finishing write tid 2 to smithi10729407-12 2020-06-12T02:39:07.414 INFO:tasks.rados.rados.0.smithi107.stdout:9860: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:07.414 INFO:tasks.rados.rados.0.smithi107.stdout:9860: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:07.415 INFO:tasks.rados.rados.0.smithi107.stdout:9860: finishing write tid 6 to smithi10729407-12 2020-06-12T02:39:07.415 INFO:tasks.rados.rados.0.smithi107.stdout:9860: finishing write tid 7 to smithi10729407-12 2020-06-12T02:39:07.415 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1817 (ObjNum 9859 snap 0 seq_num 9859) dirty exists 2020-06-12T02:39:07.415 INFO:tasks.rados.rados.0.smithi107.stdout:9860: left oid 12 (ObjNum 9859 snap 0 seq_num 9859) 2020-06-12T02:39:07.416 INFO:tasks.rados.rados.0.smithi107.stdout:9860: done (2 left) 2020-06-12T02:39:07.417 INFO:tasks.rados.rados.0.smithi107.stdout:9863: done (1 left) 2020-06-12T02:39:07.417 INFO:tasks.rados.rados.0.smithi107.stdout:9864: delete oid 13 current snap is 0 2020-06-12T02:39:07.417 INFO:tasks.rados.rados.0.smithi107.stdout:9862: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:07.425 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:39:07.421+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:39:07.426 INFO:tasks.rados.rados.0.smithi107.stdout:9864: done (1 left) 2020-06-12T02:39:07.426 INFO:tasks.rados.rados.0.smithi107.stdout:9865: delete oid 7 current snap is 0 2020-06-12T02:39:07.426 INFO:tasks.rados.rados.0.smithi107.stdout:9862: finishing write tid 2 to smithi10729407-20 2020-06-12T02:39:07.427 INFO:tasks.rados.rados.0.smithi107.stdout:9862: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:07.427 INFO:tasks.rados.rados.0.smithi107.stdout:9862: finishing write tid 5 to smithi10729407-20 2020-06-12T02:39:07.427 INFO:tasks.rados.rados.0.smithi107.stdout:9865: done (1 left) 2020-06-12T02:39:07.427 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:39:07.428 INFO:tasks.rados.rados.0.smithi107.stdout:9866: seq_num 9865 ranges {0=425984,425984=573440,999424=720896,1720320=311296} 2020-06-12T02:39:07.428 INFO:tasks.rados.rados.0.smithi107.stdout:9866: writing smithi10729407-16 from 0 to 425984 tid 1 2020-06-12T02:39:07.430 INFO:tasks.rados.rados.0.smithi107.stdout:9866: writing smithi10729407-16 from 425984 to 999424 tid 2 2020-06-12T02:39:07.434 INFO:tasks.rados.rados.0.smithi107.stdout:9866: writing smithi10729407-16 from 999424 to 1720320 tid 3 2020-06-12T02:39:07.436 INFO:tasks.rados.rados.0.smithi107.stdout:9866: writing smithi10729407-16 from 1720320 to 2031616 tid 4 2020-06-12T02:39:07.437 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:39:07.437 INFO:tasks.rados.rados.0.smithi107.stdout:9867: seq_num 9866 ranges {0=442368,442368=458752,901120=483328,1384448=663552} 2020-06-12T02:39:07.440 INFO:tasks.rados.rados.0.smithi107.stdout:9867: writing smithi10729407-9 from 0 to 442368 tid 1 2020-06-12T02:39:07.443 INFO:tasks.rados.rados.0.smithi107.stdout:9867: writing smithi10729407-9 from 442368 to 901120 tid 2 2020-06-12T02:39:07.447 INFO:tasks.rados.rados.0.smithi107.stdout:9867: writing smithi10729407-9 from 901120 to 1384448 tid 3 2020-06-12T02:39:07.452 INFO:tasks.rados.rados.0.smithi107.stdout:9867: writing smithi10729407-9 from 1384448 to 2048000 tid 4 2020-06-12T02:39:07.452 INFO:tasks.rados.rados.0.smithi107.stdout:9868: delete oid 15 current snap is 0 2020-06-12T02:39:07.452 INFO:tasks.rados.rados.0.smithi107.stdout:9862: finishing write tid 6 to smithi10729407-20 2020-06-12T02:39:07.452 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 739 (ObjNum 9861 snap 0 seq_num 9861) dirty exists 2020-06-12T02:39:07.453 INFO:tasks.rados.rados.0.smithi107.stdout:9862: left oid 20 (ObjNum 9861 snap 0 seq_num 9861) 2020-06-12T02:39:07.453 INFO:tasks.rados.rados.0.smithi107.stdout:9866: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:07.453 INFO:tasks.rados.rados.0.smithi107.stdout:9866: finishing write tid 2 to smithi10729407-16 2020-06-12T02:39:07.453 INFO:tasks.rados.rados.0.smithi107.stdout:9866: finishing write tid 3 to smithi10729407-16 2020-06-12T02:39:07.453 INFO:tasks.rados.rados.0.smithi107.stdout:9867: finishing write tid 1 to smithi10729407-9 2020-06-12T02:39:07.454 INFO:tasks.rados.rados.0.smithi107.stdout:9866: finishing write tid 4 to smithi10729407-16 2020-06-12T02:39:07.454 INFO:tasks.rados.rados.0.smithi107.stdout:9866: finishing write tid 6 to smithi10729407-16 2020-06-12T02:39:07.454 INFO:tasks.rados.rados.0.smithi107.stdout:9866: finishing write tid 7 to smithi10729407-16 2020-06-12T02:39:07.455 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1770 (ObjNum 9865 snap 0 seq_num 9865) dirty exists 2020-06-12T02:39:07.455 INFO:tasks.rados.rados.0.smithi107.stdout:9866: left oid 16 (ObjNum 9865 snap 0 seq_num 9865) 2020-06-12T02:39:07.455 INFO:tasks.rados.rados.0.smithi107.stdout:9862: done (3 left) 2020-06-12T02:39:07.455 INFO:tasks.rados.rados.0.smithi107.stdout:9866: done (2 left) 2020-06-12T02:39:07.455 INFO:tasks.rados.rados.0.smithi107.stdout:9868: done (1 left) 2020-06-12T02:39:07.456 INFO:tasks.rados.rados.0.smithi107.stdout:9869: delete oid 6 current snap is 0 2020-06-12T02:39:07.458 INFO:tasks.rados.rados.0.smithi107.stdout:9867: finishing write tid 2 to smithi10729407-9 2020-06-12T02:39:07.459 INFO:tasks.rados.rados.0.smithi107.stdout:9867: finishing write tid 3 to smithi10729407-9 2020-06-12T02:39:07.459 INFO:tasks.rados.rados.0.smithi107.stdout:9867: finishing write tid 4 to smithi10729407-9 2020-06-12T02:39:07.460 INFO:tasks.rados.rados.0.smithi107.stdout:9869: done (1 left) 2020-06-12T02:39:07.461 INFO:tasks.rados.rados.0.smithi107.stdout:9870: delete oid 2 current snap is 0 2020-06-12T02:39:07.461 INFO:tasks.rados.rados.0.smithi107.stdout:9867: finishing write tid 6 to smithi10729407-9 2020-06-12T02:39:07.461 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:07.462 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:07.464 INFO:tasks.rados.rados.0.smithi107.stdout:9867: finishing write tid 7 to smithi10729407-9 2020-06-12T02:39:07.464 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1777 (ObjNum 9866 snap 0 seq_num 9866) dirty exists 2020-06-12T02:39:07.464 INFO:tasks.rados.rados.0.smithi107.stdout:9867: left oid 9 (ObjNum 9866 snap 0 seq_num 9866) 2020-06-12T02:39:07.466 INFO:tasks.rados.rados.0.smithi107.stdout:9867: done (1 left) 2020-06-12T02:39:07.466 INFO:tasks.rados.rados.0.smithi107.stdout:9870: done (0 left) 2020-06-12T02:39:07.466 INFO:tasks.rados.rados.0.smithi107.stdout:9871: delete oid 9 current snap is 0 2020-06-12T02:39:07.467 INFO:tasks.rados.rados.0.smithi107.stdout:9871: done (0 left) 2020-06-12T02:39:07.467 INFO:tasks.rados.rados.0.smithi107.stdout:9872: delete oid 13 current snap is 0 2020-06-12T02:39:07.473 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-06-12T02:39:07.474 INFO:tasks.rados.rados.0.smithi107.stdout:9872: done (0 left) 2020-06-12T02:39:07.474 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:39:07.474 INFO:tasks.rados.rados.0.smithi107.stdout:9873: seq_num 9872 ranges {0=540672,540672=614400,1155072=475136,1630208=516096} 2020-06-12T02:39:07.476 INFO:tasks.rados.rados.0.smithi107.stdout:9873: writing smithi10729407-14 from 0 to 540672 tid 1 2020-06-12T02:39:07.479 INFO:tasks.rados.rados.0.smithi107.stdout:9873: writing smithi10729407-14 from 540672 to 1155072 tid 2 2020-06-12T02:39:07.482 INFO:tasks.rados.rados.0.smithi107.stdout:9873: writing smithi10729407-14 from 1155072 to 1630208 tid 3 2020-06-12T02:39:07.486 INFO:tasks.rados.rados.0.smithi107.stdout:9873: writing smithi10729407-14 from 1630208 to 2146304 tid 4 2020-06-12T02:39:07.486 INFO:tasks.rados.rados.0.smithi107.stdout:9874: delete oid 7 current snap is 0 2020-06-12T02:39:07.487 INFO:tasks.rados.rados.0.smithi107.stdout:9873: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:07.487 INFO:tasks.rados.rados.0.smithi107.stdout:9873: finishing write tid 2 to smithi10729407-14 2020-06-12T02:39:07.489 INFO:tasks.rados.rados.0.smithi107.stdout:9874: done (1 left) 2020-06-12T02:39:07.489 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:39:07.489 INFO:tasks.rados.rados.0.smithi107.stdout:9875: seq_num 9874 ranges {2031616=573440,2605056=786432,3391488=442368,3833856=376832} 2020-06-12T02:39:07.503 INFO:tasks.rados.rados.0.smithi107.stdout:9875: writing smithi10729407-16 from 2031616 to 2605056 tid 1 2020-06-12T02:39:07.509 INFO:tasks.rados.rados.0.smithi107.stdout:9875: writing smithi10729407-16 from 2605056 to 3391488 tid 2 2020-06-12T02:39:07.514 INFO:tasks.rados.rados.0.smithi107.stdout:9875: writing smithi10729407-16 from 3391488 to 3833856 tid 3 2020-06-12T02:39:07.515 INFO:tasks.rados.rados.0.smithi107.stdout:9875: writing smithi10729407-16 from 3833856 to 4210688 tid 4 2020-06-12T02:39:07.515 INFO:tasks.rados.rados.0.smithi107.stdout:9876: delete oid 18 current snap is 0 2020-06-12T02:39:07.515 INFO:tasks.rados.rados.0.smithi107.stdout:9873: finishing write tid 3 to smithi10729407-14 2020-06-12T02:39:07.515 INFO:tasks.rados.rados.0.smithi107.stdout:9873: finishing write tid 4 to smithi10729407-14 2020-06-12T02:39:07.516 INFO:tasks.rados.rados.0.smithi107.stdout:9873: finishing write tid 6 to smithi10729407-14 2020-06-12T02:39:07.516 INFO:tasks.rados.rados.0.smithi107.stdout:9873: finishing write tid 7 to smithi10729407-14 2020-06-12T02:39:07.516 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1631 (ObjNum 9872 snap 0 seq_num 9872) dirty exists 2020-06-12T02:39:07.516 INFO:tasks.rados.rados.0.smithi107.stdout:9873: left oid 14 (ObjNum 9872 snap 0 seq_num 9872) 2020-06-12T02:39:07.516 INFO:tasks.rados.rados.0.smithi107.stdout:9875: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:07.517 INFO:tasks.rados.rados.0.smithi107.stdout:9873: done (2 left) 2020-06-12T02:39:07.517 INFO:tasks.rados.rados.0.smithi107.stdout:9876: done (1 left) 2020-06-12T02:39:07.517 INFO:tasks.rados.rados.0.smithi107.stdout:9877: delete oid 20 current snap is 0 2020-06-12T02:39:07.526 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:39:07.523+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:39:07.526 INFO:tasks.rados.rados.0.smithi107.stdout:9877: done (1 left) 2020-06-12T02:39:07.527 INFO:tasks.rados.rados.0.smithi107.stdout:9878: delete oid 7 current snap is 0 2020-06-12T02:39:07.529 INFO:tasks.rados.rados.0.smithi107.stdout:9878: done (1 left) 2020-06-12T02:39:07.530 INFO:tasks.rados.rados.0.smithi107.stdout:9879: delete oid 1 current snap is 0 2020-06-12T02:39:07.530 INFO:tasks.rados.rados.0.smithi107.stdout:9875: finishing write tid 2 to smithi10729407-16 2020-06-12T02:39:07.531 INFO:tasks.rados.rados.0.smithi107.stdout:9875: finishing write tid 3 to smithi10729407-16 2020-06-12T02:39:07.533 INFO:tasks.rados.rados.0.smithi107.stdout:9875: finishing write tid 4 to smithi10729407-16 2020-06-12T02:39:07.533 INFO:tasks.rados.rados.0.smithi107.stdout:9875: finishing write tid 6 to smithi10729407-16 2020-06-12T02:39:07.533 INFO:tasks.rados.rados.0.smithi107.stdout:9879: done (1 left) 2020-06-12T02:39:07.534 INFO:tasks.rados.rados.0.smithi107.stdout:9880: delete oid 19 current snap is 0 2020-06-12T02:39:07.534 INFO:tasks.rados.rados.0.smithi107.stdout:9875: finishing write tid 7 to smithi10729407-16 2020-06-12T02:39:07.535 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1775 (ObjNum 9874 snap 0 seq_num 9874) dirty exists 2020-06-12T02:39:07.535 INFO:tasks.rados.rados.0.smithi107.stdout:9875: left oid 16 (ObjNum 9874 snap 0 seq_num 9874) 2020-06-12T02:39:07.538 INFO:tasks.rados.rados.0.smithi107.stdout:9875: done (1 left) 2020-06-12T02:39:07.539 INFO:tasks.rados.rados.0.smithi107.stdout:9880: done (0 left) 2020-06-12T02:39:07.539 INFO:tasks.rados.rados.0.smithi107.stdout:9881: delete oid 11 current snap is 0 2020-06-12T02:39:07.539 INFO:tasks.rados.rados.0.smithi107.stdout:9881: done (0 left) 2020-06-12T02:39:07.539 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:39:07.539 INFO:tasks.rados.rados.0.smithi107.stdout:9882: seq_num 9881 ranges {0=696320,696320=425984,1122304=647168,1769472=532480} 2020-06-12T02:39:07.545 INFO:tasks.rados.rados.0.smithi107.stdout:9882: writing smithi10729407-1 from 0 to 696320 tid 1 2020-06-12T02:39:07.546 INFO:tasks.rados.rados.0.smithi107.stdout:9882: writing smithi10729407-1 from 696320 to 1122304 tid 2 2020-06-12T02:39:07.548 INFO:tasks.rados.rados.0.smithi107.stdout:9882: writing smithi10729407-1 from 1122304 to 1769472 tid 3 2020-06-12T02:39:07.551 INFO:tasks.rados.rados.0.smithi107.stdout:9882: writing smithi10729407-1 from 1769472 to 2301952 tid 4 2020-06-12T02:39:07.552 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:39:07.552 INFO:tasks.rados.rados.0.smithi107.stdout:9883: seq_num 9882 ranges {0=712704,712704=712704,1425408=802816,2228224=90112} 2020-06-12T02:39:07.557 INFO:tasks.rados.rados.0.smithi107.stdout:9883: writing smithi10729407-2 from 0 to 712704 tid 1 2020-06-12T02:39:07.562 INFO:tasks.rados.rados.0.smithi107.stdout:9883: writing smithi10729407-2 from 712704 to 1425408 tid 2 2020-06-12T02:39:07.567 INFO:tasks.rados.rados.0.smithi107.stdout:9883: writing smithi10729407-2 from 1425408 to 2228224 tid 3 2020-06-12T02:39:07.568 INFO:tasks.rados.rados.0.smithi107.stdout:9883: writing smithi10729407-2 from 2228224 to 2318336 tid 4 2020-06-12T02:39:07.568 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:39:07.568 INFO:tasks.rados.rados.0.smithi107.stdout:9884: seq_num 9883 ranges {0=729088,729088=589824,1318912=606208,1925120=409600} 2020-06-12T02:39:07.571 INFO:tasks.rados.rados.0.smithi107.stdout:9884: writing smithi10729407-5 from 0 to 729088 tid 1 2020-06-12T02:39:07.575 INFO:tasks.rados.rados.0.smithi107.stdout:9884: writing smithi10729407-5 from 729088 to 1318912 tid 2 2020-06-12T02:39:07.579 INFO:tasks.rados.rados.0.smithi107.stdout:9884: writing smithi10729407-5 from 1318912 to 1925120 tid 3 2020-06-12T02:39:07.582 INFO:tasks.rados.rados.0.smithi107.stdout:9884: writing smithi10729407-5 from 1925120 to 2334720 tid 4 2020-06-12T02:39:07.582 INFO:tasks.rados.rados.0.smithi107.stdout:9885: delete oid 13 current snap is 0 2020-06-12T02:39:07.583 INFO:tasks.rados.rados.0.smithi107.stdout:9882: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:07.583 INFO:tasks.rados.rados.0.smithi107.stdout:9882: finishing write tid 2 to smithi10729407-1 2020-06-12T02:39:07.583 INFO:tasks.rados.rados.0.smithi107.stdout:9882: finishing write tid 3 to smithi10729407-1 2020-06-12T02:39:07.583 INFO:tasks.rados.rados.0.smithi107.stdout:9882: finishing write tid 4 to smithi10729407-1 2020-06-12T02:39:07.583 INFO:tasks.rados.rados.0.smithi107.stdout:9882: finishing write tid 6 to smithi10729407-1 2020-06-12T02:39:07.584 INFO:tasks.rados.rados.0.smithi107.stdout:9883: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:07.584 INFO:tasks.rados.rados.0.smithi107.stdout:9882: finishing write tid 7 to smithi10729407-1 2020-06-12T02:39:07.584 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1776 (ObjNum 9881 snap 0 seq_num 9881) dirty exists 2020-06-12T02:39:07.584 INFO:tasks.rados.rados.0.smithi107.stdout:9882: left oid 1 (ObjNum 9881 snap 0 seq_num 9881) 2020-06-12T02:39:07.584 INFO:tasks.rados.rados.0.smithi107.stdout:9883: finishing write tid 2 to smithi10729407-2 2020-06-12T02:39:07.585 INFO:tasks.rados.rados.0.smithi107.stdout:9883: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:07.585 INFO:tasks.rados.rados.0.smithi107.stdout:9883: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:07.585 INFO:tasks.rados.rados.0.smithi107.stdout:9883: finishing write tid 6 to smithi10729407-2 2020-06-12T02:39:07.585 INFO:tasks.rados.rados.0.smithi107.stdout:9883: finishing write tid 7 to smithi10729407-2 2020-06-12T02:39:07.585 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 930 (ObjNum 9882 snap 0 seq_num 9882) dirty exists 2020-06-12T02:39:07.585 INFO:tasks.rados.rados.0.smithi107.stdout:9883: left oid 2 (ObjNum 9882 snap 0 seq_num 9882) 2020-06-12T02:39:07.586 INFO:tasks.rados.rados.0.smithi107.stdout:9882: done (3 left) 2020-06-12T02:39:07.586 INFO:tasks.rados.rados.0.smithi107.stdout:9883: done (2 left) 2020-06-12T02:39:07.586 INFO:tasks.rados.rados.0.smithi107.stdout:9885: done (1 left) 2020-06-12T02:39:07.587 INFO:tasks.rados.rados.0.smithi107.stdout:9886: delete oid 2 current snap is 0 2020-06-12T02:39:07.589 INFO:tasks.rados.rados.0.smithi107.stdout:9884: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:07.589 INFO:tasks.rados.rados.0.smithi107.stdout:9884: finishing write tid 2 to smithi10729407-5 2020-06-12T02:39:07.589 INFO:tasks.rados.rados.0.smithi107.stdout:9884: finishing write tid 3 to smithi10729407-5 2020-06-12T02:39:07.590 INFO:tasks.rados.rados.0.smithi107.stdout:9886: done (1 left) 2020-06-12T02:39:07.590 INFO:tasks.rados.rados.0.smithi107.stdout:9887: delete oid 19 current snap is 0 2020-06-12T02:39:07.591 INFO:tasks.rados.rados.0.smithi107.stdout:9884: finishing write tid 4 to smithi10729407-5 2020-06-12T02:39:07.592 INFO:tasks.rados.rados.0.smithi107.stdout:9884: finishing write tid 6 to smithi10729407-5 2020-06-12T02:39:07.593 INFO:tasks.rados.rados.0.smithi107.stdout:9887: done (1 left) 2020-06-12T02:39:07.593 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:39:07.593 INFO:tasks.rados.rados.0.smithi107.stdout:9888: seq_num 9887 ranges {0=794624,794624=581632,1376256=589824,1966080=417792,2383872=16384} 2020-06-12T02:39:07.599 INFO:tasks.rados.rados.0.smithi107.stdout:9888: writing smithi10729407-6 from 0 to 794624 tid 1 2020-06-12T02:39:07.605 INFO:tasks.rados.rados.0.smithi107.stdout:9888: writing smithi10729407-6 from 794624 to 1376256 tid 2 2020-06-12T02:39:07.608 INFO:tasks.rados.rados.0.smithi107.stdout:9888: writing smithi10729407-6 from 1376256 to 1966080 tid 3 2020-06-12T02:39:07.611 INFO:tasks.rados.rados.0.smithi107.stdout:9888: writing smithi10729407-6 from 1966080 to 2383872 tid 4 2020-06-12T02:39:07.611 INFO:tasks.rados.rados.0.smithi107.stdout:9888: writing smithi10729407-6 from 2383872 to 2400256 tid 5 2020-06-12T02:39:07.612 INFO:tasks.rados.rados.0.smithi107.stdout:9889: delete oid 8 current snap is 0 2020-06-12T02:39:07.612 INFO:tasks.rados.rados.0.smithi107.stdout:9884: finishing write tid 7 to smithi10729407-5 2020-06-12T02:39:07.612 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 945 (ObjNum 9883 snap 0 seq_num 9883) dirty exists 2020-06-12T02:39:07.613 INFO:tasks.rados.rados.0.smithi107.stdout:9884: left oid 5 (ObjNum 9883 snap 0 seq_num 9883) 2020-06-12T02:39:07.613 INFO:tasks.rados.rados.0.smithi107.stdout:9888: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:07.614 INFO:tasks.rados.rados.0.smithi107.stdout:9884: done (2 left) 2020-06-12T02:39:07.614 INFO:tasks.rados.rados.0.smithi107.stdout:9889: done (1 left) 2020-06-12T02:39:07.615 INFO:tasks.rados.rados.0.smithi107.stdout:9890: delete oid 19 current snap is 0 2020-06-12T02:39:07.619 INFO:tasks.rados.rados.0.smithi107.stdout:9888: finishing write tid 2 to smithi10729407-6 2020-06-12T02:39:07.620 INFO:tasks.rados.rados.0.smithi107.stdout:9890: done (1 left) 2020-06-12T02:39:07.620 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:39:07.620 INFO:tasks.rados.rados.0.smithi107.stdout:9891: seq_num 9890 ranges {0=40960} 2020-06-12T02:39:07.621 INFO:tasks.rados.rados.0.smithi107.stdout:9891: writing smithi10729407-4 from 0 to 40960 tid 1 2020-06-12T02:39:07.621 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:39:07.621 INFO:tasks.rados.rados.0.smithi107.stdout:9892: seq_num 9891 ranges {0=57344} 2020-06-12T02:39:07.622 INFO:tasks.rados.rados.0.smithi107.stdout:9892: writing smithi10729407-2 from 0 to 57344 tid 1 2020-06-12T02:39:07.622 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:39:07.622 INFO:tasks.rados.rados.0.smithi107.stdout:9893: seq_num 9892 ranges {1933312=73728} 2020-06-12T02:39:07.631 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:07.631 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:07.631 INFO:tasks.rados.rados.0.smithi107.stdout:9893: writing smithi10729407-12 from 1933312 to 2007040 tid 1 2020-06-12T02:39:07.632 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:39:07.632 INFO:tasks.rados.rados.0.smithi107.stdout:9894: seq_num 9893 ranges {0=90112} 2020-06-12T02:39:07.632 INFO:tasks.rados.rados.0.smithi107.stdout:9894: writing smithi10729407-20 from 0 to 90112 tid 1 2020-06-12T02:39:07.632 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:39:07.633 INFO:tasks.rados.rados.0.smithi107.stdout:9895: seq_num 9894 ranges {2334720=106496} 2020-06-12T02:39:07.633 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:39:07.625+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:39:07.639 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-06-12T02:39:07.644 INFO:tasks.rados.rados.0.smithi107.stdout:9895: writing smithi10729407-5 from 2334720 to 2441216 tid 1 2020-06-12T02:39:07.645 INFO:tasks.rados.rados.0.smithi107.stdout:9896: delete oid 9 current snap is 0 2020-06-12T02:39:07.645 INFO:tasks.rados.rados.0.smithi107.stdout:9888: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:07.648 INFO:tasks.rados.rados.0.smithi107.stdout:9888: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:07.648 INFO:tasks.rados.rados.0.smithi107.stdout:9888: finishing write tid 5 to smithi10729407-6 2020-06-12T02:39:07.649 INFO:tasks.rados.rados.0.smithi107.stdout:9891: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:07.649 INFO:tasks.rados.rados.0.smithi107.stdout:9891: finishing write tid 3 to smithi10729407-4 2020-06-12T02:39:07.649 INFO:tasks.rados.rados.0.smithi107.stdout:9892: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:07.649 INFO:tasks.rados.rados.0.smithi107.stdout:9892: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:07.650 INFO:tasks.rados.rados.0.smithi107.stdout:9888: finishing write tid 7 to smithi10729407-6 2020-06-12T02:39:07.650 INFO:tasks.rados.rados.0.smithi107.stdout:9891: finishing write tid 4 to smithi10729407-4 2020-06-12T02:39:07.650 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1855 (ObjNum 9890 snap 0 seq_num 9890) dirty exists 2020-06-12T02:39:07.650 INFO:tasks.rados.rados.0.smithi107.stdout:9891: left oid 4 (ObjNum 9890 snap 0 seq_num 9890) 2020-06-12T02:39:07.650 INFO:tasks.rados.rados.0.smithi107.stdout:9888: finishing write tid 8 to smithi10729407-6 2020-06-12T02:39:07.651 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1782 (ObjNum 9887 snap 0 seq_num 9887) dirty exists 2020-06-12T02:39:07.651 INFO:tasks.rados.rados.0.smithi107.stdout:9888: left oid 6 (ObjNum 9887 snap 0 seq_num 9887) 2020-06-12T02:39:07.651 INFO:tasks.rados.rados.0.smithi107.stdout:9892: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:07.651 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 933 (ObjNum 9891 snap 0 seq_num 9891) dirty exists 2020-06-12T02:39:07.652 INFO:tasks.rados.rados.0.smithi107.stdout:9892: left oid 2 (ObjNum 9891 snap 0 seq_num 9891) 2020-06-12T02:39:07.652 INFO:tasks.rados.rados.0.smithi107.stdout:9893: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:07.652 INFO:tasks.rados.rados.0.smithi107.stdout:9893: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:07.652 INFO:tasks.rados.rados.0.smithi107.stdout:9893: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:07.653 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1822 (ObjNum 9892 snap 0 seq_num 9892) dirty exists 2020-06-12T02:39:07.653 INFO:tasks.rados.rados.0.smithi107.stdout:9893: left oid 12 (ObjNum 9892 snap 0 seq_num 9892) 2020-06-12T02:39:07.653 INFO:tasks.rados.rados.0.smithi107.stdout:9894: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:07.654 INFO:tasks.rados.rados.0.smithi107.stdout:9894: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:07.655 INFO:tasks.rados.rados.0.smithi107.stdout:9894: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:07.655 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 742 (ObjNum 9893 snap 0 seq_num 9893) dirty exists 2020-06-12T02:39:07.655 INFO:tasks.rados.rados.0.smithi107.stdout:9894: left oid 20 (ObjNum 9893 snap 0 seq_num 9893) 2020-06-12T02:39:07.656 INFO:tasks.rados.rados.0.smithi107.stdout:9888: done (6 left) 2020-06-12T02:39:07.657 INFO:tasks.rados.rados.0.smithi107.stdout:9891: done (5 left) 2020-06-12T02:39:07.657 INFO:tasks.rados.rados.0.smithi107.stdout:9892: done (4 left) 2020-06-12T02:39:07.657 INFO:tasks.rados.rados.0.smithi107.stdout:9893: done (3 left) 2020-06-12T02:39:07.657 INFO:tasks.rados.rados.0.smithi107.stdout:9894: done (2 left) 2020-06-12T02:39:07.658 INFO:tasks.rados.rados.0.smithi107.stdout:9896: done (1 left) 2020-06-12T02:39:07.658 INFO:tasks.rados.rados.0.smithi107.stdout:9897: delete oid 3 current snap is 0 2020-06-12T02:39:07.658 INFO:tasks.rados.rados.0.smithi107.stdout:9895: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:07.658 INFO:tasks.rados.rados.0.smithi107.stdout:9897: done (1 left) 2020-06-12T02:39:07.658 INFO:tasks.rados.rados.0.smithi107.stdout:9898: delete oid 16 current snap is 0 2020-06-12T02:39:07.659 INFO:tasks.rados.rados.0.smithi107.stdout:9895: finishing write tid 3 to smithi10729407-5 2020-06-12T02:39:07.659 INFO:tasks.rados.rados.0.smithi107.stdout:9895: finishing write tid 4 to smithi10729407-5 2020-06-12T02:39:07.660 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 947 (ObjNum 9894 snap 0 seq_num 9894) dirty exists 2020-06-12T02:39:07.660 INFO:tasks.rados.rados.0.smithi107.stdout:9895: left oid 5 (ObjNum 9894 snap 0 seq_num 9894) 2020-06-12T02:39:07.660 INFO:tasks.rados.rados.0.smithi107.stdout:9895: done (1 left) 2020-06-12T02:39:07.660 INFO:tasks.rados.rados.0.smithi107.stdout:9898: done (0 left) 2020-06-12T02:39:07.660 INFO:tasks.rados.rados.0.smithi107.stdout:9899: delete oid 16 current snap is 0 2020-06-12T02:39:07.661 INFO:tasks.rados.rados.0.smithi107.stdout:9899: done (0 left) 2020-06-12T02:39:07.662 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:39:07.662 INFO:tasks.rados.rados.0.smithi107.stdout:9900: seq_num 9899 ranges {2007040=196608} 2020-06-12T02:39:07.671 INFO:tasks.rados.rados.0.smithi107.stdout:9900: writing smithi10729407-12 from 2007040 to 2203648 tid 1 2020-06-12T02:39:07.672 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:39:07.672 INFO:tasks.rados.rados.0.smithi107.stdout:9901: seq_num 9900 ranges {2400256=212992} 2020-06-12T02:39:07.686 INFO:tasks.rados.rados.0.smithi107.stdout:9901: writing smithi10729407-6 from 2400256 to 2613248 tid 1 2020-06-12T02:39:07.686 INFO:tasks.rados.rados.0.smithi107.stdout:9902: delete oid 9 current snap is 0 2020-06-12T02:39:07.687 INFO:tasks.rados.rados.0.smithi107.stdout:9900: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:07.687 INFO:tasks.rados.rados.0.smithi107.stdout:9900: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:07.687 INFO:tasks.rados.rados.0.smithi107.stdout:9900: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:07.687 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1824 (ObjNum 9899 snap 0 seq_num 9899) dirty exists 2020-06-12T02:39:07.687 INFO:tasks.rados.rados.0.smithi107.stdout:9900: left oid 12 (ObjNum 9899 snap 0 seq_num 9899) 2020-06-12T02:39:07.688 INFO:tasks.rados.rados.0.smithi107.stdout:9900: done (2 left) 2020-06-12T02:39:07.688 INFO:tasks.rados.rados.0.smithi107.stdout:9902: done (1 left) 2020-06-12T02:39:07.689 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:39:07.689 INFO:tasks.rados.rados.0.smithi107.stdout:9903: seq_num 9902 ranges {40960=245760} 2020-06-12T02:39:07.690 INFO:tasks.rados.rados.0.smithi107.stdout:9903: writing smithi10729407-4 from 40960 to 286720 tid 1 2020-06-12T02:39:07.690 INFO:tasks.rados.rados.0.smithi107.stdout:9904: delete oid 1 current snap is 0 2020-06-12T02:39:07.691 INFO:tasks.rados.rados.0.smithi107.stdout:9901: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:07.691 INFO:tasks.rados.rados.0.smithi107.stdout:9901: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:07.693 INFO:tasks.rados.rados.0.smithi107.stdout:9901: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:07.694 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1784 (ObjNum 9900 snap 0 seq_num 9900) dirty exists 2020-06-12T02:39:07.694 INFO:tasks.rados.rados.0.smithi107.stdout:9901: left oid 6 (ObjNum 9900 snap 0 seq_num 9900) 2020-06-12T02:39:07.695 INFO:tasks.rados.rados.0.smithi107.stdout:9901: done (2 left) 2020-06-12T02:39:07.695 INFO:tasks.rados.rados.0.smithi107.stdout:9904: done (1 left) 2020-06-12T02:39:07.695 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:39:07.696 INFO:tasks.rados.rados.0.smithi107.stdout:9905: seq_num 9904 ranges {2146304=278528} 2020-06-12T02:39:07.708 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2020-06-12T02:39:07.708 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2020-06-12T02:39:07.708 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2020-06-12T02:39:07.708 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd reweight-by-utilization 110 1.0 2020-06-12T02:39:07.710 INFO:tasks.rados.rados.0.smithi107.stdout:9905: writing smithi10729407-14 from 2146304 to 2424832 tid 1 2020-06-12T02:39:07.710 INFO:tasks.rados.rados.0.smithi107.stdout:9906: delete oid 12 current snap is 0 2020-06-12T02:39:07.710 INFO:tasks.rados.rados.0.smithi107.stdout:9903: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:07.711 INFO:tasks.rados.rados.0.smithi107.stdout:9903: finishing write tid 3 to smithi10729407-4 2020-06-12T02:39:07.711 INFO:tasks.rados.rados.0.smithi107.stdout:9903: finishing write tid 4 to smithi10729407-4 2020-06-12T02:39:07.711 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1857 (ObjNum 9902 snap 0 seq_num 9902) dirty exists 2020-06-12T02:39:07.711 INFO:tasks.rados.rados.0.smithi107.stdout:9903: left oid 4 (ObjNum 9902 snap 0 seq_num 9902) 2020-06-12T02:39:07.713 INFO:tasks.rados.rados.0.smithi107.stdout:9903: done (2 left) 2020-06-12T02:39:07.713 INFO:tasks.rados.rados.0.smithi107.stdout:9906: done (1 left) 2020-06-12T02:39:07.713 INFO:tasks.rados.rados.0.smithi107.stdout:9907: delete oid 2 current snap is 0 2020-06-12T02:39:07.714 INFO:tasks.rados.rados.0.smithi107.stdout:9905: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:07.715 INFO:tasks.rados.rados.0.smithi107.stdout:9907: done (1 left) 2020-06-12T02:39:07.715 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:39:07.715 INFO:tasks.rados.rados.0.smithi107.stdout:9908: seq_num 9907 ranges {0=327680} 2020-06-12T02:39:07.716 INFO:tasks.rados.rados.0.smithi107.stdout:9908: writing smithi10729407-18 from 0 to 327680 tid 1 2020-06-12T02:39:07.717 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:39:07.717 INFO:tasks.rados.rados.0.smithi107.stdout:9909: seq_num 9908 ranges {0=344064} 2020-06-12T02:39:07.719 INFO:tasks.rados.rados.0.smithi107.stdout:9909: writing smithi10729407-8 from 0 to 344064 tid 1 2020-06-12T02:39:07.719 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:39:07.719 INFO:tasks.rados.rados.0.smithi107.stdout:9910: seq_num 9909 ranges {2441216=360448} 2020-06-12T02:39:07.728 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:39:07.726+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:39:07.735 INFO:tasks.rados.rados.0.smithi107.stdout:9910: writing smithi10729407-5 from 2441216 to 2801664 tid 1 2020-06-12T02:39:07.735 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:39:07.735 INFO:tasks.rados.rados.0.smithi107.stdout:9911: seq_num 9910 ranges {0=376832} 2020-06-12T02:39:07.737 INFO:tasks.rados.rados.0.smithi107.stdout:9911: writing smithi10729407-12 from 0 to 376832 tid 1 2020-06-12T02:39:07.737 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:39:07.737 INFO:tasks.rados.rados.0.smithi107.stdout:9912: seq_num 9911 ranges {0=393216} 2020-06-12T02:39:07.739 INFO:tasks.rados.rados.0.smithi107.stdout:9912: writing smithi10729407-13 from 0 to 393216 tid 1 2020-06-12T02:39:07.740 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:39:07.740 INFO:tasks.rados.rados.0.smithi107.stdout:9913: seq_num 9912 ranges {0=409600} 2020-06-12T02:39:07.742 INFO:tasks.rados.rados.0.smithi107.stdout:9913: writing smithi10729407-17 from 0 to 409600 tid 1 2020-06-12T02:39:07.742 INFO:tasks.rados.rados.0.smithi107.stdout:9914: delete oid 7 current snap is 0 2020-06-12T02:39:07.743 INFO:tasks.rados.rados.0.smithi107.stdout:9905: finishing write tid 3 to smithi10729407-14 2020-06-12T02:39:07.743 INFO:tasks.rados.rados.0.smithi107.stdout:9908: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:07.744 INFO:tasks.rados.rados.0.smithi107.stdout:9905: finishing write tid 4 to smithi10729407-14 2020-06-12T02:39:07.745 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1633 (ObjNum 9904 snap 0 seq_num 9904) dirty exists 2020-06-12T02:39:07.745 INFO:tasks.rados.rados.0.smithi107.stdout:9905: left oid 14 (ObjNum 9904 snap 0 seq_num 9904) 2020-06-12T02:39:07.745 INFO:tasks.rados.rados.0.smithi107.stdout:9908: finishing write tid 3 to smithi10729407-18 2020-06-12T02:39:07.745 INFO:tasks.rados.rados.0.smithi107.stdout:9909: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:07.745 INFO:tasks.rados.rados.0.smithi107.stdout:9909: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:07.745 INFO:tasks.rados.rados.0.smithi107.stdout:9908: finishing write tid 4 to smithi10729407-18 2020-06-12T02:39:07.746 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1784 (ObjNum 9907 snap 0 seq_num 9907) dirty exists 2020-06-12T02:39:07.746 INFO:tasks.rados.rados.0.smithi107.stdout:9908: left oid 18 (ObjNum 9907 snap 0 seq_num 9907) 2020-06-12T02:39:07.746 INFO:tasks.rados.rados.0.smithi107.stdout:9909: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:07.746 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1837 (ObjNum 9908 snap 0 seq_num 9908) dirty exists 2020-06-12T02:39:07.746 INFO:tasks.rados.rados.0.smithi107.stdout:9909: left oid 8 (ObjNum 9908 snap 0 seq_num 9908) 2020-06-12T02:39:07.747 INFO:tasks.rados.rados.0.smithi107.stdout:9910: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:07.747 INFO:tasks.rados.rados.0.smithi107.stdout:9910: finishing write tid 3 to smithi10729407-5 2020-06-12T02:39:07.747 INFO:tasks.rados.rados.0.smithi107.stdout:9911: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:07.748 INFO:tasks.rados.rados.0.smithi107.stdout:9910: finishing write tid 4 to smithi10729407-5 2020-06-12T02:39:07.748 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 949 (ObjNum 9909 snap 0 seq_num 9909) dirty exists 2020-06-12T02:39:07.748 INFO:tasks.rados.rados.0.smithi107.stdout:9910: left oid 5 (ObjNum 9909 snap 0 seq_num 9909) 2020-06-12T02:39:07.749 INFO:tasks.rados.rados.0.smithi107.stdout:9912: finishing write tid 1 to smithi10729407-13 2020-06-12T02:39:07.749 INFO:tasks.rados.rados.0.smithi107.stdout:9912: finishing write tid 3 to smithi10729407-13 2020-06-12T02:39:07.749 INFO:tasks.rados.rados.0.smithi107.stdout:9911: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:07.750 INFO:tasks.rados.rados.0.smithi107.stdout:9905: done (7 left) 2020-06-12T02:39:07.750 INFO:tasks.rados.rados.0.smithi107.stdout:9908: done (6 left) 2020-06-12T02:39:07.750 INFO:tasks.rados.rados.0.smithi107.stdout:9909: done (5 left) 2020-06-12T02:39:07.750 INFO:tasks.rados.rados.0.smithi107.stdout:9910: done (4 left) 2020-06-12T02:39:07.750 INFO:tasks.rados.rados.0.smithi107.stdout:9914: done (3 left) 2020-06-12T02:39:07.751 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:39:07.751 INFO:tasks.rados.rados.0.smithi107.stdout:9915: seq_num 9914 ranges {344064=442368} 2020-06-12T02:39:07.754 INFO:tasks.rados.rados.0.smithi107.stdout:9915: writing smithi10729407-8 from 344064 to 786432 tid 1 2020-06-12T02:39:07.755 INFO:tasks.rados.rados.0.smithi107.stdout:9916: delete oid 4 current snap is 0 2020-06-12T02:39:07.755 INFO:tasks.rados.rados.0.smithi107.stdout:9912: finishing write tid 4 to smithi10729407-13 2020-06-12T02:39:07.755 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1709 (ObjNum 9911 snap 0 seq_num 9911) dirty exists 2020-06-12T02:39:07.755 INFO:tasks.rados.rados.0.smithi107.stdout:9912: left oid 13 (ObjNum 9911 snap 0 seq_num 9911) 2020-06-12T02:39:07.755 INFO:tasks.rados.rados.0.smithi107.stdout:9913: finishing write tid 1 to smithi10729407-17 2020-06-12T02:39:07.756 INFO:tasks.rados.rados.0.smithi107.stdout:9911: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:07.756 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1827 (ObjNum 9910 snap 0 seq_num 9910) dirty exists 2020-06-12T02:39:07.756 INFO:tasks.rados.rados.0.smithi107.stdout:9911: left oid 12 (ObjNum 9910 snap 0 seq_num 9910) 2020-06-12T02:39:07.756 INFO:tasks.rados.rados.0.smithi107.stdout:9913: finishing write tid 3 to smithi10729407-17 2020-06-12T02:39:07.756 INFO:tasks.rados.rados.0.smithi107.stdout:9913: finishing write tid 4 to smithi10729407-17 2020-06-12T02:39:07.757 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1859 (ObjNum 9912 snap 0 seq_num 9912) dirty exists 2020-06-12T02:39:07.757 INFO:tasks.rados.rados.0.smithi107.stdout:9913: left oid 17 (ObjNum 9912 snap 0 seq_num 9912) 2020-06-12T02:39:07.757 INFO:tasks.rados.rados.0.smithi107.stdout:9911: done (4 left) 2020-06-12T02:39:07.758 INFO:tasks.rados.rados.0.smithi107.stdout:9912: done (3 left) 2020-06-12T02:39:07.758 INFO:tasks.rados.rados.0.smithi107.stdout:9913: done (2 left) 2020-06-12T02:39:07.758 INFO:tasks.rados.rados.0.smithi107.stdout:9916: done (1 left) 2020-06-12T02:39:07.758 INFO:tasks.rados.rados.0.smithi107.stdout:9917: delete oid 9 current snap is 0 2020-06-12T02:39:07.759 INFO:tasks.rados.rados.0.smithi107.stdout:9917: done (1 left) 2020-06-12T02:39:07.759 INFO:tasks.rados.rados.0.smithi107.stdout:9918: delete oid 13 current snap is 0 2020-06-12T02:39:07.760 INFO:tasks.rados.rados.0.smithi107.stdout:9915: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:07.761 INFO:tasks.rados.rados.0.smithi107.stdout:9915: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:07.762 INFO:tasks.rados.rados.0.smithi107.stdout:9918: done (1 left) 2020-06-12T02:39:07.762 INFO:tasks.rados.rados.0.smithi107.stdout:9919: delete oid 6 current snap is 0 2020-06-12T02:39:07.763 INFO:tasks.rados.rados.0.smithi107.stdout:9915: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:07.764 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1839 (ObjNum 9914 snap 0 seq_num 9914) dirty exists 2020-06-12T02:39:07.764 INFO:tasks.rados.rados.0.smithi107.stdout:9915: left oid 8 (ObjNum 9914 snap 0 seq_num 9914) 2020-06-12T02:39:07.764 INFO:tasks.rados.rados.0.smithi107.stdout:9915: done (1 left) 2020-06-12T02:39:07.765 INFO:tasks.rados.rados.0.smithi107.stdout:9919: done (0 left) 2020-06-12T02:39:07.765 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:39:07.765 INFO:tasks.rados.rados.0.smithi107.stdout:9920: seq_num 9919 ranges {0=532480} 2020-06-12T02:39:07.768 INFO:tasks.rados.rados.0.smithi107.stdout:9920: writing smithi10729407-2 from 0 to 532480 tid 1 2020-06-12T02:39:07.768 INFO:tasks.rados.rados.0.smithi107.stdout:9921: delete oid 14 current snap is 0 2020-06-12T02:39:07.771 INFO:tasks.rados.rados.0.smithi107.stdout:9921: done (1 left) 2020-06-12T02:39:07.771 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:39:07.771 INFO:tasks.rados.rados.0.smithi107.stdout:9922: seq_num 9921 ranges {409600=565248} 2020-06-12T02:39:07.777 INFO:tasks.rados.rados.0.smithi107.stdout:9922: writing smithi10729407-17 from 409600 to 974848 tid 1 2020-06-12T02:39:07.777 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:39:07.778 INFO:tasks.rados.rados.0.smithi107.stdout:9923: seq_num 9922 ranges {1343488=581632} 2020-06-12T02:39:07.788 INFO:tasks.rados.rados.0.smithi107.stdout:9923: writing smithi10729407-10 from 1343488 to 1925120 tid 1 2020-06-12T02:39:07.789 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:39:07.789 INFO:tasks.rados.rados.0.smithi107.stdout:9924: seq_num 9923 ranges {0=598016} 2020-06-12T02:39:07.792 INFO:tasks.rados.rados.0.smithi107.stdout:9924: writing smithi10729407-14 from 0 to 598016 tid 1 2020-06-12T02:39:07.792 INFO:tasks.rados.rados.0.smithi107.stdout:9925: delete oid 1 current snap is 0 2020-06-12T02:39:07.793 INFO:tasks.rados.rados.0.smithi107.stdout:9920: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:07.793 INFO:tasks.rados.rados.0.smithi107.stdout:9920: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:07.793 INFO:tasks.rados.rados.0.smithi107.stdout:9922: finishing write tid 1 to smithi10729407-17 2020-06-12T02:39:07.793 INFO:tasks.rados.rados.0.smithi107.stdout:9920: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:07.793 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 936 (ObjNum 9919 snap 0 seq_num 9919) dirty exists 2020-06-12T02:39:07.794 INFO:tasks.rados.rados.0.smithi107.stdout:9920: left oid 2 (ObjNum 9919 snap 0 seq_num 9919) 2020-06-12T02:39:07.794 INFO:tasks.rados.rados.0.smithi107.stdout:9922: finishing write tid 3 to smithi10729407-17 2020-06-12T02:39:07.794 INFO:tasks.rados.rados.0.smithi107.stdout:9922: finishing write tid 4 to smithi10729407-17 2020-06-12T02:39:07.794 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1862 (ObjNum 9921 snap 0 seq_num 9921) dirty exists 2020-06-12T02:39:07.794 INFO:tasks.rados.rados.0.smithi107.stdout:9922: left oid 17 (ObjNum 9921 snap 0 seq_num 9921) 2020-06-12T02:39:07.795 INFO:tasks.rados.rados.0.smithi107.stdout:9923: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:07.795 INFO:tasks.rados.rados.0.smithi107.stdout:9923: finishing write tid 3 to smithi10729407-10 2020-06-12T02:39:07.797 INFO:tasks.rados.rados.0.smithi107.stdout:9923: finishing write tid 4 to smithi10729407-10 2020-06-12T02:39:07.797 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 712 (ObjNum 9922 snap 0 seq_num 9922) dirty exists 2020-06-12T02:39:07.797 INFO:tasks.rados.rados.0.smithi107.stdout:9923: left oid 10 (ObjNum 9922 snap 0 seq_num 9922) 2020-06-12T02:39:07.798 INFO:tasks.rados.rados.0.smithi107.stdout:9924: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:07.799 INFO:tasks.rados.rados.0.smithi107.stdout:9924: finishing write tid 3 to smithi10729407-14 2020-06-12T02:39:07.801 INFO:tasks.rados.rados.0.smithi107.stdout:9920: done (4 left) 2020-06-12T02:39:07.801 INFO:tasks.rados.rados.0.smithi107.stdout:9922: done (3 left) 2020-06-12T02:39:07.802 INFO:tasks.rados.rados.0.smithi107.stdout:9923: done (2 left) 2020-06-12T02:39:07.802 INFO:tasks.rados.rados.0.smithi107.stdout:9925: done (1 left) 2020-06-12T02:39:07.802 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:39:07.802 INFO:tasks.rados.rados.0.smithi107.stdout:9926: seq_num 9925 ranges {786432=630784} 2020-06-12T02:39:07.805 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:07.805 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:07.808 INFO:tasks.rados.rados.0.smithi107.stdout:9926: writing smithi10729407-8 from 786432 to 1417216 tid 1 2020-06-12T02:39:07.809 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:39:07.809 INFO:tasks.rados.rados.0.smithi107.stdout:9927: seq_num 9926 ranges {532480=647168} 2020-06-12T02:39:07.815 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-06-12T02:39:07.816 INFO:tasks.rados.rados.0.smithi107.stdout:9927: writing smithi10729407-2 from 532480 to 1179648 tid 1 2020-06-12T02:39:07.817 INFO:tasks.rados.rados.0.smithi107.stdout:9928: delete oid 4 current snap is 0 2020-06-12T02:39:07.817 INFO:tasks.rados.rados.0.smithi107.stdout:9924: finishing write tid 4 to smithi10729407-14 2020-06-12T02:39:07.817 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1636 (ObjNum 9923 snap 0 seq_num 9923) dirty exists 2020-06-12T02:39:07.817 INFO:tasks.rados.rados.0.smithi107.stdout:9924: left oid 14 (ObjNum 9923 snap 0 seq_num 9923) 2020-06-12T02:39:07.825 INFO:tasks.rados.rados.0.smithi107.stdout:9926: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:07.825 INFO:tasks.rados.rados.0.smithi107.stdout:9924: done (3 left) 2020-06-12T02:39:07.826 INFO:tasks.rados.rados.0.smithi107.stdout:9928: done (2 left) 2020-06-12T02:39:07.826 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:39:07.827 INFO:tasks.rados.rados.0.smithi107.stdout:9929: seq_num 9928 ranges {90112=679936} 2020-06-12T02:39:07.827 INFO:tasks.rados.rados.0.smithi107.stdout:9929: writing smithi10729407-20 from 90112 to 770048 tid 1 2020-06-12T02:39:07.827 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:39:07.827 INFO:tasks.rados.rados.0.smithi107.stdout:9930: seq_num 9929 ranges {0=696320} 2020-06-12T02:39:07.834 INFO:tasks.rados.rados.0.smithi107.stdout:9930: writing smithi10729407-6 from 0 to 696320 tid 1 2020-06-12T02:39:07.835 INFO:tasks.rados.rados.0.smithi107.stdout:9931: delete oid 19 current snap is 0 2020-06-12T02:39:07.835 INFO:tasks.rados.rados.0.smithi107.stdout:9926: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:07.835 INFO:tasks.rados.rados.0.smithi107.stdout:9927: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:07.836 INFO:tasks.rados.rados.0.smithi107.stdout:9926: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:07.836 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1841 (ObjNum 9925 snap 0 seq_num 9925) dirty exists 2020-06-12T02:39:07.836 INFO:tasks.rados.rados.0.smithi107.stdout:9926: left oid 8 (ObjNum 9925 snap 0 seq_num 9925) 2020-06-12T02:39:07.837 INFO:tasks.rados.rados.0.smithi107.stdout:9927: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:07.837 INFO:tasks.rados.rados.0.smithi107.stdout:9926: done (4 left) 2020-06-12T02:39:07.837 INFO:tasks.rados.rados.0.smithi107.stdout:9931: done (3 left) 2020-06-12T02:39:07.838 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:39:07.838 INFO:tasks.rados.rados.0.smithi107.stdout:9932: seq_num 9931 ranges {1417216=729088} 2020-06-12T02:39:07.838 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:39:07.831+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:39:07.847 INFO:tasks.rados.rados.0.smithi107.stdout:9932: writing smithi10729407-8 from 1417216 to 2146304 tid 1 2020-06-12T02:39:07.847 INFO:tasks.rados.rados.0.smithi107.stdout:9933: delete oid 11 current snap is 0 2020-06-12T02:39:07.848 INFO:tasks.rados.rados.0.smithi107.stdout:9927: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:07.848 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 938 (ObjNum 9926 snap 0 seq_num 9926) dirty exists 2020-06-12T02:39:07.848 INFO:tasks.rados.rados.0.smithi107.stdout:9927: left oid 2 (ObjNum 9926 snap 0 seq_num 9926) 2020-06-12T02:39:07.849 INFO:tasks.rados.rados.0.smithi107.stdout:9929: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:07.849 INFO:tasks.rados.rados.0.smithi107.stdout:9930: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:07.849 INFO:tasks.rados.rados.0.smithi107.stdout:9929: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:07.850 INFO:tasks.rados.rados.0.smithi107.stdout:9930: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:07.850 INFO:tasks.rados.rados.0.smithi107.stdout:9929: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:07.850 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 744 (ObjNum 9928 snap 0 seq_num 9928) dirty exists 2020-06-12T02:39:07.851 INFO:tasks.rados.rados.0.smithi107.stdout:9929: left oid 20 (ObjNum 9928 snap 0 seq_num 9928) 2020-06-12T02:39:07.851 INFO:tasks.rados.rados.0.smithi107.stdout:9930: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:07.851 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1789 (ObjNum 9929 snap 0 seq_num 9929) dirty exists 2020-06-12T02:39:07.852 INFO:tasks.rados.rados.0.smithi107.stdout:9930: left oid 6 (ObjNum 9929 snap 0 seq_num 9929) 2020-06-12T02:39:07.852 INFO:tasks.rados.rados.0.smithi107.stdout:9927: done (4 left) 2020-06-12T02:39:07.859 INFO:tasks.rados.rados.0.smithi107.stdout:9929: done (3 left) 2020-06-12T02:39:07.860 INFO:tasks.rados.rados.0.smithi107.stdout:9930: done (2 left) 2020-06-12T02:39:07.860 INFO:tasks.rados.rados.0.smithi107.stdout:9933: done (1 left) 2020-06-12T02:39:07.860 INFO:tasks.rados.rados.0.smithi107.stdout:9934: delete oid 11 current snap is 0 2020-06-12T02:39:07.861 INFO:tasks.rados.rados.0.smithi107.stdout:9934: done (1 left) 2020-06-12T02:39:07.861 INFO:tasks.rados.rados.0.smithi107.stdout:9935: delete oid 15 current snap is 0 2020-06-12T02:39:07.863 INFO:tasks.rados.rados.0.smithi107.stdout:9932: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:07.866 INFO:tasks.rados.rados.0.smithi107.stdout:9932: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:07.866 INFO:tasks.rados.rados.0.smithi107.stdout:9932: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:07.867 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1843 (ObjNum 9931 snap 0 seq_num 9931) dirty exists 2020-06-12T02:39:07.867 INFO:tasks.rados.rados.0.smithi107.stdout:9932: left oid 8 (ObjNum 9931 snap 0 seq_num 9931) 2020-06-12T02:39:07.867 INFO:tasks.rados.rados.0.smithi107.stdout:9932: done (1 left) 2020-06-12T02:39:07.867 INFO:tasks.rados.rados.0.smithi107.stdout:9935: done (0 left) 2020-06-12T02:39:07.868 INFO:tasks.rados.rados.0.smithi107.stdout:9936: delete oid 15 current snap is 0 2020-06-12T02:39:07.868 INFO:tasks.rados.rados.0.smithi107.stdout:9936: done (0 left) 2020-06-12T02:39:07.868 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:39:07.868 INFO:tasks.rados.rados.0.smithi107.stdout:9937: seq_num 9936 ranges {598016=417792,1015808=401408} 2020-06-12T02:39:07.869 INFO:tasks.rados.rados.0.smithi107.stdout:9937: writing smithi10729407-14 from 598016 to 1015808 tid 1 2020-06-12T02:39:07.869 INFO:tasks.rados.rados.0.smithi107.stdout:9937: writing smithi10729407-14 from 1015808 to 1417216 tid 2 2020-06-12T02:39:07.869 INFO:tasks.rados.rados.0.smithi107.stdout:9938: delete oid 15 current snap is 0 2020-06-12T02:39:07.872 INFO:tasks.rados.rados.0.smithi107.stdout:9938: done (1 left) 2020-06-12T02:39:07.873 INFO:tasks.rados.rados.0.smithi107.stdout:9939: delete oid 12 current snap is 0 2020-06-12T02:39:07.874 INFO:tasks.rados.rados.0.smithi107.stdout:9937: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:07.875 INFO:tasks.rados.rados.0.smithi107.stdout:9937: finishing write tid 2 to smithi10729407-14 2020-06-12T02:39:07.876 INFO:tasks.rados.rados.0.smithi107.stdout:9937: finishing write tid 4 to smithi10729407-14 2020-06-12T02:39:07.876 INFO:tasks.rados.rados.0.smithi107.stdout:9939: done (1 left) 2020-06-12T02:39:07.876 INFO:tasks.rados.rados.0.smithi107.stdout:9940: delete oid 20 current snap is 0 2020-06-12T02:39:07.878 INFO:tasks.rados.rados.0.smithi107.stdout:9937: finishing write tid 5 to smithi10729407-14 2020-06-12T02:39:07.878 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1639 (ObjNum 9936 snap 0 seq_num 9936) dirty exists 2020-06-12T02:39:07.878 INFO:tasks.rados.rados.0.smithi107.stdout:9937: left oid 14 (ObjNum 9936 snap 0 seq_num 9936) 2020-06-12T02:39:07.879 INFO:tasks.rados.rados.0.smithi107.stdout:9937: done (1 left) 2020-06-12T02:39:07.879 INFO:tasks.rados.rados.0.smithi107.stdout:9940: done (0 left) 2020-06-12T02:39:07.880 INFO:tasks.rados.rados.0.smithi107.stdout:9941: delete oid 12 current snap is 0 2020-06-12T02:39:07.880 INFO:tasks.rados.rados.0.smithi107.stdout:9941: done (0 left) 2020-06-12T02:39:07.882 INFO:tasks.rados.rados.0.smithi107.stdout:9942: delete oid 18 current snap is 0 2020-06-12T02:39:07.883 INFO:tasks.rados.rados.0.smithi107.stdout:9942: done (0 left) 2020-06-12T02:39:07.883 INFO:tasks.rados.rados.0.smithi107.stdout:9943: delete oid 14 current snap is 0 2020-06-12T02:39:07.888 INFO:tasks.rados.rados.0.smithi107.stdout:9943: done (0 left) 2020-06-12T02:39:07.889 INFO:tasks.rados.rados.0.smithi107.stdout:9944: delete oid 4 current snap is 0 2020-06-12T02:39:07.892 INFO:tasks.rados.rados.0.smithi107.stdout:9944: done (0 left) 2020-06-12T02:39:07.892 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:39:07.892 INFO:tasks.rados.rados.0.smithi107.stdout:9945: seq_num 9944 ranges {696320=548864,1245184=401408} 2020-06-12T02:39:07.898 INFO:tasks.rados.rados.0.smithi107.stdout:9945: writing smithi10729407-6 from 696320 to 1245184 tid 1 2020-06-12T02:39:07.900 INFO:tasks.rados.rados.0.smithi107.stdout:9945: writing smithi10729407-6 from 1245184 to 1646592 tid 2 2020-06-12T02:39:07.901 INFO:tasks.rados.rados.0.smithi107.stdout:9946: delete oid 2 current snap is 0 2020-06-12T02:39:07.906 INFO:tasks.rados.rados.0.smithi107.stdout:9946: done (1 left) 2020-06-12T02:39:07.906 INFO:tasks.rados.rados.0.smithi107.stdout:9947: delete oid 10 current snap is 0 2020-06-12T02:39:07.908 INFO:tasks.rados.rados.0.smithi107.stdout:9945: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:07.909 INFO:tasks.rados.rados.0.smithi107.stdout:9945: finishing write tid 2 to smithi10729407-6 2020-06-12T02:39:07.911 INFO:tasks.rados.rados.0.smithi107.stdout:9945: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:07.911 INFO:tasks.rados.rados.0.smithi107.stdout:9947: done (1 left) 2020-06-12T02:39:07.912 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:39:07.913 INFO:tasks.rados.rados.0.smithi107.stdout:9948: seq_num 9947 ranges {0=598016,598016=401408} 2020-06-12T02:39:07.913 INFO:tasks.rados.rados.0.smithi107.stdout:9948: writing smithi10729407-10 from 0 to 598016 tid 1 2020-06-12T02:39:07.916 INFO:tasks.rados.rados.0.smithi107.stdout:9948: writing smithi10729407-10 from 598016 to 999424 tid 2 2020-06-12T02:39:07.916 INFO:tasks.rados.rados.0.smithi107.stdout:9949: delete oid 2 current snap is 0 2020-06-12T02:39:07.916 INFO:tasks.rados.rados.0.smithi107.stdout:9945: finishing write tid 5 to smithi10729407-6 2020-06-12T02:39:07.916 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1792 (ObjNum 9944 snap 0 seq_num 9944) dirty exists 2020-06-12T02:39:07.917 INFO:tasks.rados.rados.0.smithi107.stdout:9945: left oid 6 (ObjNum 9944 snap 0 seq_num 9944) 2020-06-12T02:39:07.918 INFO:tasks.rados.rados.0.smithi107.stdout:9945: done (2 left) 2020-06-12T02:39:07.918 INFO:tasks.rados.rados.0.smithi107.stdout:9949: done (1 left) 2020-06-12T02:39:07.925 INFO:tasks.rados.rados.0.smithi107.stdout:9950: delete oid 15 current snap is 0 2020-06-12T02:39:07.927 INFO:tasks.rados.rados.0.smithi107.stdout:9948: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:07.927 INFO:tasks.rados.rados.0.smithi107.stdout:9950: done (1 left) 2020-06-12T02:39:07.928 INFO:tasks.rados.rados.0.smithi107.stdout:9951: delete oid 13 current snap is 0 2020-06-12T02:39:07.928 INFO:tasks.rados.rados.0.smithi107.stdout:9951: done (1 left) 2020-06-12T02:39:07.929 INFO:tasks.rados.rados.0.smithi107.stdout:9952: delete oid 20 current snap is 0 2020-06-12T02:39:07.929 INFO:tasks.rados.rados.0.smithi107.stdout:9948: finishing write tid 2 to smithi10729407-10 2020-06-12T02:39:07.930 INFO:tasks.rados.rados.0.smithi107.stdout:9948: finishing write tid 4 to smithi10729407-10 2020-06-12T02:39:07.930 INFO:tasks.rados.rados.0.smithi107.stdout:9952: done (1 left) 2020-06-12T02:39:07.930 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:39:07.931 INFO:tasks.rados.rados.0.smithi107.stdout:9953: seq_num 9952 ranges {0=679936,679936=401408} 2020-06-12T02:39:07.932 INFO:tasks.rados.rados.0.smithi107.stdout:9953: writing smithi10729407-20 from 0 to 679936 tid 1 2020-06-12T02:39:07.935 INFO:tasks.rados.rados.0.smithi107.stdout:9953: writing smithi10729407-20 from 679936 to 1081344 tid 2 2020-06-12T02:39:07.935 INFO:tasks.rados.rados.0.smithi107.stdout:9954: delete oid 2 current snap is 0 2020-06-12T02:39:07.935 INFO:tasks.rados.rados.0.smithi107.stdout:9948: finishing write tid 5 to smithi10729407-10 2020-06-12T02:39:07.936 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 716 (ObjNum 9947 snap 0 seq_num 9947) dirty exists 2020-06-12T02:39:07.936 INFO:tasks.rados.rados.0.smithi107.stdout:9948: left oid 10 (ObjNum 9947 snap 0 seq_num 9947) 2020-06-12T02:39:07.936 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:39:07.932+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:39:07.937 INFO:tasks.rados.rados.0.smithi107.stdout:9953: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:07.944 INFO:tasks.rados.rados.0.smithi107.stdout:9948: done (2 left) 2020-06-12T02:39:07.945 INFO:tasks.rados.rados.0.smithi107.stdout:9954: done (1 left) 2020-06-12T02:39:07.945 INFO:tasks.rados.rados.0.smithi107.stdout:9955: delete oid 17 current snap is 0 2020-06-12T02:39:07.946 INFO:tasks.rados.rados.0.smithi107.stdout:9955: done (1 left) 2020-06-12T02:39:07.946 INFO:tasks.rados.rados.0.smithi107.stdout:9956: delete oid 3 current snap is 0 2020-06-12T02:39:07.946 INFO:tasks.rados.rados.0.smithi107.stdout:9953: finishing write tid 2 to smithi10729407-20 2020-06-12T02:39:07.947 INFO:tasks.rados.rados.0.smithi107.stdout:9953: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:07.948 INFO:tasks.rados.rados.0.smithi107.stdout:9956: done (1 left) 2020-06-12T02:39:07.948 INFO:tasks.rados.rados.0.smithi107.stdout:9957: delete oid 18 current snap is 0 2020-06-12T02:39:07.949 INFO:tasks.rados.rados.0.smithi107.stdout:9953: finishing write tid 5 to smithi10729407-20 2020-06-12T02:39:07.949 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 749 (ObjNum 9952 snap 0 seq_num 9952) dirty exists 2020-06-12T02:39:07.949 INFO:tasks.rados.rados.0.smithi107.stdout:9953: left oid 20 (ObjNum 9952 snap 0 seq_num 9952) 2020-06-12T02:39:07.950 INFO:tasks.rados.rados.0.smithi107.stdout:9953: done (1 left) 2020-06-12T02:39:07.950 INFO:tasks.rados.rados.0.smithi107.stdout:9957: done (0 left) 2020-06-12T02:39:07.951 INFO:tasks.rados.rados.0.smithi107.stdout:9958: delete oid 7 current snap is 0 2020-06-12T02:39:07.952 INFO:tasks.rados.rados.0.smithi107.stdout:9958: done (0 left) 2020-06-12T02:39:07.952 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:39:07.952 INFO:tasks.rados.rados.0.smithi107.stdout:9959: seq_num 9958 ranges {0=786432,786432=401408} 2020-06-12T02:39:07.954 INFO:tasks.rados.rados.0.smithi107.stdout:9959: writing smithi10729407-19 from 0 to 786432 tid 1 2020-06-12T02:39:07.956 INFO:tasks.rados.rados.0.smithi107.stdout:9959: writing smithi10729407-19 from 786432 to 1187840 tid 2 2020-06-12T02:39:07.957 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:39:07.957 INFO:tasks.rados.rados.0.smithi107.stdout:9960: seq_num 9959 ranges {0=802816,802816=401408} 2020-06-12T02:39:07.961 INFO:tasks.rados.rados.0.smithi107.stdout:9960: writing smithi10729407-1 from 0 to 802816 tid 1 2020-06-12T02:39:07.964 INFO:tasks.rados.rados.0.smithi107.stdout:9960: writing smithi10729407-1 from 802816 to 1204224 tid 2 2020-06-12T02:39:07.964 INFO:tasks.rados.rados.0.smithi107.stdout:9961: delete oid 5 current snap is 0 2020-06-12T02:39:07.965 INFO:tasks.rados.rados.0.smithi107.stdout:9959: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:07.965 INFO:tasks.rados.rados.0.smithi107.stdout:9959: finishing write tid 2 to smithi10729407-19 2020-06-12T02:39:07.966 INFO:tasks.rados.rados.0.smithi107.stdout:9959: finishing write tid 4 to smithi10729407-19 2020-06-12T02:39:07.969 INFO:tasks.rados.rados.0.smithi107.stdout:9959: finishing write tid 5 to smithi10729407-19 2020-06-12T02:39:07.970 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1833 (ObjNum 9958 snap 0 seq_num 9958) dirty exists 2020-06-12T02:39:07.970 INFO:tasks.rados.rados.0.smithi107.stdout:9959: left oid 19 (ObjNum 9958 snap 0 seq_num 9958) 2020-06-12T02:39:07.971 INFO:tasks.rados.rados.0.smithi107.stdout:9959: done (2 left) 2020-06-12T02:39:07.971 INFO:tasks.rados.rados.0.smithi107.stdout:9961: done (1 left) 2020-06-12T02:39:07.971 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:39:07.972 INFO:tasks.rados.rados.0.smithi107.stdout:9962: seq_num 9961 ranges {1646592=434176,2080768=516096,2596864=286720} 2020-06-12T02:39:07.974 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:07.974 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:07.981 INFO:tasks.rados.rados.0.smithi107.stdout:9962: writing smithi10729407-6 from 1646592 to 2080768 tid 1 2020-06-12T02:39:07.984 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-06-12T02:39:07.985 INFO:tasks.rados.rados.0.smithi107.stdout:9962: writing smithi10729407-6 from 2080768 to 2596864 tid 2 2020-06-12T02:39:07.987 INFO:tasks.rados.rados.0.smithi107.stdout:9962: writing smithi10729407-6 from 2596864 to 2883584 tid 3 2020-06-12T02:39:07.987 INFO:tasks.rados.rados.0.smithi107.stdout:9963: delete oid 10 current snap is 0 2020-06-12T02:39:07.987 INFO:tasks.rados.rados.0.smithi107.stdout:9960: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:07.987 INFO:tasks.rados.rados.0.smithi107.stdout:9960: finishing write tid 2 to smithi10729407-1 2020-06-12T02:39:07.987 INFO:tasks.rados.rados.0.smithi107.stdout:9960: finishing write tid 4 to smithi10729407-1 2020-06-12T02:39:07.988 INFO:tasks.rados.rados.0.smithi107.stdout:9960: finishing write tid 5 to smithi10729407-1 2020-06-12T02:39:07.988 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1795 (ObjNum 9959 snap 0 seq_num 9959) dirty exists 2020-06-12T02:39:07.988 INFO:tasks.rados.rados.0.smithi107.stdout:9960: left oid 1 (ObjNum 9959 snap 0 seq_num 9959) 2020-06-12T02:39:07.988 INFO:tasks.rados.rados.0.smithi107.stdout:9962: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:07.989 INFO:tasks.rados.rados.0.smithi107.stdout:9960: done (2 left) 2020-06-12T02:39:07.989 INFO:tasks.rados.rados.0.smithi107.stdout:9963: done (1 left) 2020-06-12T02:39:07.989 INFO:tasks.rados.rados.0.smithi107.stdout:9964: delete oid 10 current snap is 0 2020-06-12T02:39:07.992 INFO:tasks.rados.rados.0.smithi107.stdout:9964: done (1 left) 2020-06-12T02:39:07.992 INFO:tasks.rados.rados.0.smithi107.stdout:9965: delete oid 20 current snap is 0 2020-06-12T02:39:07.995 INFO:tasks.rados.rados.0.smithi107.stdout:9962: finishing write tid 2 to smithi10729407-6 2020-06-12T02:39:07.995 INFO:tasks.rados.rados.0.smithi107.stdout:9962: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:07.997 INFO:tasks.rados.rados.0.smithi107.stdout:9965: done (1 left) 2020-06-12T02:39:07.998 INFO:tasks.rados.rados.0.smithi107.stdout:9966: delete oid 12 current snap is 0 2020-06-12T02:39:07.999 INFO:tasks.rados.rados.0.smithi107.stdout:9962: finishing write tid 5 to smithi10729407-6 2020-06-12T02:39:07.999 INFO:tasks.rados.rados.0.smithi107.stdout:9966: done (1 left) 2020-06-12T02:39:08.000 INFO:tasks.rados.rados.0.smithi107.stdout:9967: delete oid 9 current snap is 0 2020-06-12T02:39:08.000 INFO:tasks.rados.rados.0.smithi107.stdout:9962: finishing write tid 6 to smithi10729407-6 2020-06-12T02:39:08.001 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1799 (ObjNum 9961 snap 0 seq_num 9961) dirty exists 2020-06-12T02:39:08.001 INFO:tasks.rados.rados.0.smithi107.stdout:9962: left oid 6 (ObjNum 9961 snap 0 seq_num 9961) 2020-06-12T02:39:08.001 INFO:tasks.rados.rados.0.smithi107.stdout:9962: done (1 left) 2020-06-12T02:39:08.002 INFO:tasks.rados.rados.0.smithi107.stdout:9967: done (0 left) 2020-06-12T02:39:08.002 INFO:tasks.rados.rados.0.smithi107.stdout:9968: delete oid 2 current snap is 0 2020-06-12T02:39:08.003 INFO:tasks.rados.rados.0.smithi107.stdout:9968: done (0 left) 2020-06-12T02:39:08.004 INFO:tasks.rados.rados.0.smithi107.stdout:9969: delete oid 19 current snap is 0 2020-06-12T02:39:08.006 INFO:tasks.rados.rados.0.smithi107.stdout:9969: done (0 left) 2020-06-12T02:39:08.006 INFO:tasks.rados.rados.0.smithi107.stdout:9970: delete oid 15 current snap is 0 2020-06-12T02:39:08.009 INFO:tasks.rados.rados.0.smithi107.stdout:9970: done (0 left) 2020-06-12T02:39:08.010 INFO:tasks.rados.rados.0.smithi107.stdout:9971: delete oid 14 current snap is 0 2020-06-12T02:39:08.013 INFO:tasks.rados.rados.0.smithi107.stdout:9971: done (0 left) 2020-06-12T02:39:08.013 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:39:08.013 INFO:tasks.rados.rados.0.smithi107.stdout:9972: seq_num 9971 ranges {2883584=598016,3481600=606208,4087808=196608} 2020-06-12T02:39:08.032 INFO:tasks.rados.rados.0.smithi107.stdout:9972: writing smithi10729407-6 from 2883584 to 3481600 tid 1 2020-06-12T02:39:08.035 INFO:tasks.rados.rados.0.smithi107.stdout:9972: writing smithi10729407-6 from 3481600 to 4087808 tid 2 2020-06-12T02:39:08.036 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:39:08.033+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:39:08.036 INFO:tasks.rados.rados.0.smithi107.stdout:9972: writing smithi10729407-6 from 4087808 to 4284416 tid 3 2020-06-12T02:39:08.037 INFO:tasks.rados.rados.0.smithi107.stdout:9973: delete oid 13 current snap is 0 2020-06-12T02:39:08.039 INFO:tasks.rados.rados.0.smithi107.stdout:9973: done (1 left) 2020-06-12T02:39:08.039 INFO:tasks.rados.rados.0.smithi107.stdout:9974: delete oid 4 current snap is 0 2020-06-12T02:39:08.043 INFO:tasks.rados.rados.0.smithi107.stdout:9974: done (1 left) 2020-06-12T02:39:08.044 INFO:tasks.rados.rados.0.smithi107.stdout:9975: delete oid 9 current snap is 0 2020-06-12T02:39:08.046 INFO:tasks.rados.rados.0.smithi107.stdout:9975: done (1 left) 2020-06-12T02:39:08.046 INFO:tasks.rados.rados.0.smithi107.stdout:9976: delete oid 11 current snap is 0 2020-06-12T02:39:08.047 INFO:tasks.rados.rados.0.smithi107.stdout:9972: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:08.047 INFO:tasks.rados.rados.0.smithi107.stdout:9972: finishing write tid 2 to smithi10729407-6 2020-06-12T02:39:08.048 INFO:tasks.rados.rados.0.smithi107.stdout:9972: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:08.048 INFO:tasks.rados.rados.0.smithi107.stdout:9972: finishing write tid 5 to smithi10729407-6 2020-06-12T02:39:08.048 INFO:tasks.rados.rados.0.smithi107.stdout:9976: done (1 left) 2020-06-12T02:39:08.049 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:39:08.049 INFO:tasks.rados.rados.0.smithi107.stdout:9977: seq_num 9976 ranges {0=688128,688128=483328,1171456=319488} 2020-06-12T02:39:08.052 INFO:tasks.rados.rados.0.smithi107.stdout:9977: writing smithi10729407-15 from 0 to 688128 tid 1 2020-06-12T02:39:08.055 INFO:tasks.rados.rados.0.smithi107.stdout:9977: writing smithi10729407-15 from 688128 to 1171456 tid 2 2020-06-12T02:39:08.057 INFO:tasks.rados.rados.0.smithi107.stdout:9977: writing smithi10729407-15 from 1171456 to 1490944 tid 3 2020-06-12T02:39:08.057 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:39:08.058 INFO:tasks.rados.rados.0.smithi107.stdout:9978: seq_num 9977 ranges {0=704512,704512=761856,1466368=40960} 2020-06-12T02:39:08.058 INFO:teuthology.orchestra.run.smithi107.stdout:moved 0 / 96 (0%) 2020-06-12T02:39:08.058 INFO:teuthology.orchestra.run.smithi107.stdout:avg 12 2020-06-12T02:39:08.059 INFO:teuthology.orchestra.run.smithi107.stdout:stddev 2.95804 -> 2.95804 (expected baseline 3.24037) 2020-06-12T02:39:08.059 INFO:teuthology.orchestra.run.smithi107.stdout:min osd.2 with 6 -> 6 pgs (0.5 -> 0.5 * mean) 2020-06-12T02:39:08.059 INFO:teuthology.orchestra.run.smithi107.stdout:max osd.7 with 16 -> 16 pgs (1.33333 -> 1.33333 * mean) 2020-06-12T02:39:08.059 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:39:08.059 INFO:teuthology.orchestra.run.smithi107.stdout:oload 110 2020-06-12T02:39:08.060 INFO:teuthology.orchestra.run.smithi107.stdout:max_change 1 2020-06-12T02:39:08.060 INFO:teuthology.orchestra.run.smithi107.stdout:max_change_osds 4 2020-06-12T02:39:08.060 INFO:teuthology.orchestra.run.smithi107.stdout:average_utilization 0.0112 2020-06-12T02:39:08.060 INFO:teuthology.orchestra.run.smithi107.stdout:overload_utilization 0.0123 2020-06-12T02:39:08.061 INFO:tasks.rados.rados.0.smithi107.stdout:9978: writing smithi10729407-18 from 0 to 704512 tid 1 2020-06-12T02:39:08.075 INFO:teuthology.orchestra.run.smithi107.stderr:no change 2020-06-12T02:39:08.075 INFO:tasks.rados.rados.0.smithi107.stdout:9978: writing smithi10729407-18 from 704512 to 1466368 tid 2 2020-06-12T02:39:08.075 INFO:tasks.rados.rados.0.smithi107.stdout:9978: writing smithi10729407-18 from 1466368 to 1507328 tid 3 2020-06-12T02:39:08.075 INFO:tasks.rados.rados.0.smithi107.stdout:9979: delete oid 2 current snap is 0 2020-06-12T02:39:08.076 INFO:tasks.rados.rados.0.smithi107.stdout:9972: finishing write tid 6 to smithi10729407-6 2020-06-12T02:39:08.076 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1803 (ObjNum 9971 snap 0 seq_num 9971) dirty exists 2020-06-12T02:39:08.076 INFO:tasks.rados.rados.0.smithi107.stdout:9972: left oid 6 (ObjNum 9971 snap 0 seq_num 9971) 2020-06-12T02:39:08.076 INFO:tasks.rados.rados.0.smithi107.stdout:9977: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:08.076 INFO:tasks.rados.rados.0.smithi107.stdout:9977: finishing write tid 2 to smithi10729407-15 2020-06-12T02:39:08.076 INFO:tasks.rados.rados.0.smithi107.stdout:9977: finishing write tid 3 to smithi10729407-15 2020-06-12T02:39:08.077 INFO:tasks.rados.rados.0.smithi107.stdout:9977: finishing write tid 5 to smithi10729407-15 2020-06-12T02:39:08.077 INFO:tasks.rados.rados.0.smithi107.stdout:9977: finishing write tid 6 to smithi10729407-15 2020-06-12T02:39:08.077 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1852 (ObjNum 9976 snap 0 seq_num 9976) dirty exists 2020-06-12T02:39:08.077 INFO:tasks.rados.rados.0.smithi107.stdout:9977: left oid 15 (ObjNum 9976 snap 0 seq_num 9976) 2020-06-12T02:39:08.077 INFO:tasks.rados.rados.0.smithi107.stdout:9972: done (3 left) 2020-06-12T02:39:08.078 INFO:tasks.rados.rados.0.smithi107.stdout:9977: done (2 left) 2020-06-12T02:39:08.078 INFO:tasks.rados.rados.0.smithi107.stdout:9979: done (1 left) 2020-06-12T02:39:08.078 INFO:tasks.rados.rados.0.smithi107.stdout:9980: delete oid 17 current snap is 0 2020-06-12T02:39:08.079 INFO:tasks.rados.rados.0.smithi107.stdout:9978: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:08.080 INFO:tasks.rados.rados.0.smithi107.stdout:9978: finishing write tid 2 to smithi10729407-18 2020-06-12T02:39:08.080 INFO:tasks.rados.rados.0.smithi107.stdout:9978: finishing write tid 3 to smithi10729407-18 2020-06-12T02:39:08.080 INFO:tasks.rados.rados.0.smithi107.stdout:9980: done (1 left) 2020-06-12T02:39:08.080 INFO:tasks.rados.rados.0.smithi107.stdout:9981: delete oid 20 current snap is 0 2020-06-12T02:39:08.081 INFO:tasks.rados.rados.0.smithi107.stdout:9978: finishing write tid 5 to smithi10729407-18 2020-06-12T02:39:08.081 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'reweight-by-utilization'} 2020-06-12T02:39:08.082 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd reweight-by-utilization 110 0.05 2020-06-12T02:39:08.083 INFO:tasks.rados.rados.0.smithi107.stdout:9978: finishing write tid 6 to smithi10729407-18 2020-06-12T02:39:08.084 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1790 (ObjNum 9977 snap 0 seq_num 9977) dirty exists 2020-06-12T02:39:08.084 INFO:tasks.rados.rados.0.smithi107.stdout:9978: left oid 18 (ObjNum 9977 snap 0 seq_num 9977) 2020-06-12T02:39:08.085 INFO:tasks.rados.rados.0.smithi107.stdout:9978: done (1 left) 2020-06-12T02:39:08.085 INFO:tasks.rados.rados.0.smithi107.stdout:9981: done (0 left) 2020-06-12T02:39:08.085 INFO:tasks.rados.rados.0.smithi107.stdout:9982: delete oid 20 current snap is 0 2020-06-12T02:39:08.086 INFO:tasks.rados.rados.0.smithi107.stdout:9982: done (0 left) 2020-06-12T02:39:08.086 INFO:tasks.rados.rados.0.smithi107.stdout:9983: delete oid 19 current snap is 0 2020-06-12T02:39:08.089 INFO:tasks.rados.rados.0.smithi107.stdout:9983: done (0 left) 2020-06-12T02:39:08.089 INFO:tasks.rados.rados.0.smithi107.stdout:9984: delete oid 17 current snap is 0 2020-06-12T02:39:08.091 INFO:tasks.rados.rados.0.smithi107.stdout:9984: done (0 left) 2020-06-12T02:39:08.091 INFO:tasks.rados.rados.0.smithi107.stdout:9985: delete oid 6 current snap is 0 2020-06-12T02:39:08.094 INFO:tasks.rados.rados.0.smithi107.stdout:9985: done (0 left) 2020-06-12T02:39:08.094 INFO:tasks.rados.rados.0.smithi107.stdout:9986: delete oid 15 current snap is 0 2020-06-12T02:39:08.099 INFO:tasks.rados.rados.0.smithi107.stdout:9986: done (0 left) 2020-06-12T02:39:08.099 INFO:tasks.rados.rados.0.smithi107.stdout:9987: delete oid 12 current snap is 0 2020-06-12T02:39:08.100 INFO:tasks.rados.rados.0.smithi107.stdout:9987: done (0 left) 2020-06-12T02:39:08.100 INFO:tasks.rados.rados.0.smithi107.stdout:9988: delete oid 6 current snap is 0 2020-06-12T02:39:08.103 INFO:tasks.rados.rados.0.smithi107.stdout:9988: done (0 left) 2020-06-12T02:39:08.104 INFO:tasks.rados.rados.0.smithi107.stdout:9989: delete oid 8 current snap is 0 2020-06-12T02:39:08.107 INFO:tasks.rados.rados.0.smithi107.stdout:9989: done (0 left) 2020-06-12T02:39:08.107 INFO:tasks.rados.rados.0.smithi107.stdout:9990: delete oid 14 current snap is 0 2020-06-12T02:39:08.109 INFO:tasks.rados.rados.0.smithi107.stdout:9990: done (0 left) 2020-06-12T02:39:08.109 INFO:tasks.rados.rados.0.smithi107.stdout:9991: delete oid 11 current snap is 0 2020-06-12T02:39:08.110 INFO:tasks.rados.rados.0.smithi107.stdout:9991: done (0 left) 2020-06-12T02:39:08.110 INFO:tasks.rados.rados.0.smithi107.stdout:9992: delete oid 14 current snap is 0 2020-06-12T02:39:08.112 INFO:tasks.rados.rados.0.smithi107.stdout:9992: done (0 left) 2020-06-12T02:39:08.112 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:39:08.112 INFO:tasks.rados.rados.0.smithi107.stdout:9993: seq_num 9992 ranges {0=548864,548864=729088,1277952=475136} 2020-06-12T02:39:08.115 INFO:tasks.rados.rados.0.smithi107.stdout:9993: writing smithi10729407-5 from 0 to 548864 tid 1 2020-06-12T02:39:08.119 INFO:tasks.rados.rados.0.smithi107.stdout:9993: writing smithi10729407-5 from 548864 to 1277952 tid 2 2020-06-12T02:39:08.122 INFO:tasks.rados.rados.0.smithi107.stdout:9993: writing smithi10729407-5 from 1277952 to 1753088 tid 3 2020-06-12T02:39:08.122 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:39:08.122 INFO:tasks.rados.rados.0.smithi107.stdout:9994: seq_num 9993 ranges {0=565248,565248=614400,1179648=499712,1679360=90112} 2020-06-12T02:39:08.125 INFO:tasks.rados.rados.0.smithi107.stdout:9994: writing smithi10729407-17 from 0 to 565248 tid 1 2020-06-12T02:39:08.136 INFO:tasks.rados.rados.0.smithi107.stdout:9994: writing smithi10729407-17 from 565248 to 1179648 tid 2 2020-06-12T02:39:08.136 INFO:tasks.rados.rados.0.smithi107.stdout:9994: writing smithi10729407-17 from 1179648 to 1679360 tid 3 2020-06-12T02:39:08.136 INFO:tasks.rados.rados.0.smithi107.stdout:9994: writing smithi10729407-17 from 1679360 to 1769472 tid 4 2020-06-12T02:39:08.136 INFO:tasks.rados.rados.0.smithi107.stdout:9995: delete oid 9 current snap is 0 2020-06-12T02:39:08.137 INFO:tasks.rados.rados.0.smithi107.stdout:9993: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:08.137 INFO:tasks.rados.rados.0.smithi107.stdout:9993: finishing write tid 2 to smithi10729407-5 2020-06-12T02:39:08.137 INFO:tasks.rados.rados.0.smithi107.stdout:9993: finishing write tid 3 to smithi10729407-5 2020-06-12T02:39:08.137 INFO:tasks.rados.rados.0.smithi107.stdout:9993: finishing write tid 5 to smithi10729407-5 2020-06-12T02:39:08.138 INFO:tasks.rados.rados.0.smithi107.stdout:9994: finishing write tid 1 to smithi10729407-17 2020-06-12T02:39:08.138 INFO:tasks.rados.rados.0.smithi107.stdout:9993: finishing write tid 6 to smithi10729407-5 2020-06-12T02:39:08.138 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 954 (ObjNum 9992 snap 0 seq_num 9992) dirty exists 2020-06-12T02:39:08.138 INFO:tasks.rados.rados.0.smithi107.stdout:9993: left oid 5 (ObjNum 9992 snap 0 seq_num 9992) 2020-06-12T02:39:08.138 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:08.139 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:08.140 INFO:tasks.rados.rados.0.smithi107.stdout:9993: done (2 left) 2020-06-12T02:39:08.140 INFO:tasks.rados.rados.0.smithi107.stdout:9995: done (1 left) 2020-06-12T02:39:08.140 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:39:08.140 INFO:tasks.rados.rados.0.smithi107.stdout:9996: seq_num 9995 ranges {0=606208,606208=786432,1392640=417792} 2020-06-12T02:39:08.140 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:39:08.138+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:39:08.142 INFO:tasks.rados.rados.0.smithi107.stdout:9996: writing smithi10729407-11 from 0 to 606208 tid 1 2020-06-12T02:39:08.144 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-06-12T02:39:08.147 INFO:tasks.rados.rados.0.smithi107.stdout:9996: writing smithi10729407-11 from 606208 to 1392640 tid 2 2020-06-12T02:39:08.150 INFO:tasks.rados.rados.0.smithi107.stdout:9996: writing smithi10729407-11 from 1392640 to 1810432 tid 3 2020-06-12T02:39:08.151 INFO:tasks.rados.rados.0.smithi107.stdout:9997: delete oid 5 current snap is 0 2020-06-12T02:39:08.151 INFO:tasks.rados.rados.0.smithi107.stdout:9994: finishing write tid 2 to smithi10729407-17 2020-06-12T02:39:08.152 INFO:tasks.rados.rados.0.smithi107.stdout:9994: finishing write tid 3 to smithi10729407-17 2020-06-12T02:39:08.152 INFO:tasks.rados.rados.0.smithi107.stdout:9994: finishing write tid 4 to smithi10729407-17 2020-06-12T02:39:08.152 INFO:tasks.rados.rados.0.smithi107.stdout:9994: finishing write tid 6 to smithi10729407-17 2020-06-12T02:39:08.153 INFO:tasks.rados.rados.0.smithi107.stdout:9994: finishing write tid 7 to smithi10729407-17 2020-06-12T02:39:08.153 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1873 (ObjNum 9993 snap 0 seq_num 9993) dirty exists 2020-06-12T02:39:08.153 INFO:tasks.rados.rados.0.smithi107.stdout:9994: left oid 17 (ObjNum 9993 snap 0 seq_num 9993) 2020-06-12T02:39:08.154 INFO:tasks.rados.rados.0.smithi107.stdout:9994: done (2 left) 2020-06-12T02:39:08.154 INFO:tasks.rados.rados.0.smithi107.stdout:9997: done (1 left) 2020-06-12T02:39:08.154 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:39:08.154 INFO:tasks.rados.rados.0.smithi107.stdout:9998: seq_num 9997 ranges {0=638976,638976=548864,1187840=475136,1662976=180224} 2020-06-12T02:39:08.157 INFO:tasks.rados.rados.0.smithi107.stdout:9998: writing smithi10729407-20 from 0 to 638976 tid 1 2020-06-12T02:39:08.161 INFO:tasks.rados.rados.0.smithi107.stdout:9998: writing smithi10729407-20 from 638976 to 1187840 tid 2 2020-06-12T02:39:08.164 INFO:tasks.rados.rados.0.smithi107.stdout:9998: writing smithi10729407-20 from 1187840 to 1662976 tid 3 2020-06-12T02:39:08.165 INFO:tasks.rados.rados.0.smithi107.stdout:9998: writing smithi10729407-20 from 1662976 to 1843200 tid 4 2020-06-12T02:39:08.166 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:39:08.166 INFO:tasks.rados.rados.0.smithi107.stdout:9999: seq_num 9998 ranges {0=655360,655360=483328,1138688=688128,1826816=32768} 2020-06-12T02:39:08.171 INFO:tasks.rados.rados.0.smithi107.stdout:9999: writing smithi10729407-4 from 0 to 655360 tid 1 2020-06-12T02:39:08.172 INFO:tasks.rados.rados.0.smithi107.stdout:9999: writing smithi10729407-4 from 655360 to 1138688 tid 2 2020-06-12T02:39:08.177 INFO:tasks.rados.rados.0.smithi107.stdout:9999: writing smithi10729407-4 from 1138688 to 1826816 tid 3 2020-06-12T02:39:08.178 INFO:tasks.rados.rados.0.smithi107.stdout:9999: writing smithi10729407-4 from 1826816 to 1859584 tid 4 2020-06-12T02:39:08.179 INFO:tasks.rados.rados.0.smithi107.stdout:10000: delete oid 12 current snap is 0 2020-06-12T02:39:08.179 INFO:tasks.rados.rados.0.smithi107.stdout:9996: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:08.179 INFO:tasks.rados.rados.0.smithi107.stdout:9996: finishing write tid 2 to smithi10729407-11 2020-06-12T02:39:08.179 INFO:tasks.rados.rados.0.smithi107.stdout:9996: finishing write tid 3 to smithi10729407-11 2020-06-12T02:39:08.179 INFO:tasks.rados.rados.0.smithi107.stdout:9996: finishing write tid 5 to smithi10729407-11 2020-06-12T02:39:08.180 INFO:tasks.rados.rados.0.smithi107.stdout:9996: finishing write tid 6 to smithi10729407-11 2020-06-12T02:39:08.180 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1670 (ObjNum 9995 snap 0 seq_num 9995) dirty exists 2020-06-12T02:39:08.180 INFO:tasks.rados.rados.0.smithi107.stdout:9996: left oid 11 (ObjNum 9995 snap 0 seq_num 9995) 2020-06-12T02:39:08.180 INFO:tasks.rados.rados.0.smithi107.stdout:9998: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:08.180 INFO:tasks.rados.rados.0.smithi107.stdout:9998: finishing write tid 2 to smithi10729407-20 2020-06-12T02:39:08.180 INFO:tasks.rados.rados.0.smithi107.stdout:9998: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:08.181 INFO:tasks.rados.rados.0.smithi107.stdout:9998: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:08.181 INFO:tasks.rados.rados.0.smithi107.stdout:9998: finishing write tid 6 to smithi10729407-20 2020-06-12T02:39:08.181 INFO:tasks.rados.rados.0.smithi107.stdout:9998: finishing write tid 7 to smithi10729407-20 2020-06-12T02:39:08.182 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 757 (ObjNum 9997 snap 0 seq_num 9997) dirty exists 2020-06-12T02:39:08.182 INFO:tasks.rados.rados.0.smithi107.stdout:9998: left oid 20 (ObjNum 9997 snap 0 seq_num 9997) 2020-06-12T02:39:08.182 INFO:tasks.rados.rados.0.smithi107.stdout:9996: done (3 left) 2020-06-12T02:39:08.183 INFO:tasks.rados.rados.0.smithi107.stdout:9998: done (2 left) 2020-06-12T02:39:08.183 INFO:tasks.rados.rados.0.smithi107.stdout:10000: done (1 left) 2020-06-12T02:39:08.183 INFO:tasks.rados.rados.0.smithi107.stdout:10001: delete oid 6 current snap is 0 2020-06-12T02:39:08.184 INFO:tasks.rados.rados.0.smithi107.stdout:10001: done (1 left) 2020-06-12T02:39:08.184 INFO:tasks.rados.rados.0.smithi107.stdout:10002: delete oid 9 current snap is 0 2020-06-12T02:39:08.188 INFO:tasks.rados.rados.0.smithi107.stdout:10002: done (1 left) 2020-06-12T02:39:08.188 INFO:tasks.rados.rados.0.smithi107.stdout:10003: delete oid 14 current snap is 0 2020-06-12T02:39:08.189 INFO:tasks.rados.rados.0.smithi107.stdout:9999: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:08.189 INFO:tasks.rados.rados.0.smithi107.stdout:9999: finishing write tid 2 to smithi10729407-4 2020-06-12T02:39:08.189 INFO:tasks.rados.rados.0.smithi107.stdout:9999: finishing write tid 3 to smithi10729407-4 2020-06-12T02:39:08.190 INFO:tasks.rados.rados.0.smithi107.stdout:9999: finishing write tid 4 to smithi10729407-4 2020-06-12T02:39:08.195 INFO:tasks.rados.rados.0.smithi107.stdout:10003: done (1 left) 2020-06-12T02:39:08.195 INFO:tasks.rados.rados.0.smithi107.stdout:10004: delete oid 9 current snap is 0 2020-06-12T02:39:08.195 INFO:tasks.rados.rados.0.smithi107.stdout:9999: finishing write tid 6 to smithi10729407-4 2020-06-12T02:39:08.196 INFO:tasks.rados.rados.0.smithi107.stdout:10004: done (1 left) 2020-06-12T02:39:08.196 INFO:tasks.rados.rados.0.smithi107.stdout:10005: delete oid 17 current snap is 0 2020-06-12T02:39:08.199 INFO:tasks.rados.rados.0.smithi107.stdout:9999: finishing write tid 7 to smithi10729407-4 2020-06-12T02:39:08.199 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1878 (ObjNum 9998 snap 0 seq_num 9998) dirty exists 2020-06-12T02:39:08.199 INFO:tasks.rados.rados.0.smithi107.stdout:9999: left oid 4 (ObjNum 9998 snap 0 seq_num 9998) 2020-06-12T02:39:08.199 INFO:tasks.rados.rados.0.smithi107.stdout:9999: done (1 left) 2020-06-12T02:39:08.199 INFO:tasks.rados.rados.0.smithi107.stdout:10005: done (0 left) 2020-06-12T02:39:08.200 INFO:tasks.rados.rados.0.smithi107.stdout:10006: delete oid 9 current snap is 0 2020-06-12T02:39:08.201 INFO:tasks.rados.rados.0.smithi107.stdout:10006: done (0 left) 2020-06-12T02:39:08.201 INFO:tasks.rados.rados.0.smithi107.stdout:10007: delete oid 11 current snap is 0 2020-06-12T02:39:08.205 INFO:tasks.rados.rados.0.smithi107.stdout:10007: done (0 left) 2020-06-12T02:39:08.205 INFO:tasks.rados.rados.0.smithi107.stdout:10008: delete oid 1 current snap is 0 2020-06-12T02:39:08.206 INFO:tasks.rados.rados.0.smithi107.stdout:10008: done (0 left) 2020-06-12T02:39:08.206 INFO:tasks.rados.rados.0.smithi107.stdout:10009: delete oid 15 current snap is 0 2020-06-12T02:39:08.209 INFO:tasks.rados.rados.0.smithi107.stdout:10009: done (0 left) 2020-06-12T02:39:08.209 INFO:tasks.rados.rados.0.smithi107.stdout:10010: delete oid 11 current snap is 0 2020-06-12T02:39:08.217 INFO:tasks.rados.rados.0.smithi107.stdout:10010: done (0 left) 2020-06-12T02:39:08.218 INFO:tasks.rados.rados.0.smithi107.stdout:10011: delete oid 18 current snap is 0 2020-06-12T02:39:08.218 INFO:tasks.rados.rados.0.smithi107.stdout:10011: done (0 left) 2020-06-12T02:39:08.218 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:39:08.218 INFO:tasks.rados.rados.0.smithi107.stdout:10012: seq_num 10011 ranges {0=466944,466944=630784,1097728=434176,1531904=540672} 2020-06-12T02:39:08.219 INFO:tasks.rados.rados.0.smithi107.stdout:10012: writing smithi10729407-3 from 0 to 466944 tid 1 2020-06-12T02:39:08.221 INFO:tasks.rados.rados.0.smithi107.stdout:10012: writing smithi10729407-3 from 466944 to 1097728 tid 2 2020-06-12T02:39:08.223 INFO:tasks.rados.rados.0.smithi107.stdout:10012: writing smithi10729407-3 from 1097728 to 1531904 tid 3 2020-06-12T02:39:08.227 INFO:tasks.rados.rados.0.smithi107.stdout:10012: writing smithi10729407-3 from 1531904 to 2072576 tid 4 2020-06-12T02:39:08.227 INFO:tasks.rados.rados.0.smithi107.stdout:10013: delete oid 1 current snap is 0 2020-06-12T02:39:08.230 INFO:tasks.rados.rados.0.smithi107.stdout:10013: done (1 left) 2020-06-12T02:39:08.231 INFO:tasks.rados.rados.0.smithi107.stdout:10014: delete oid 17 current snap is 0 2020-06-12T02:39:08.232 INFO:tasks.rados.rados.0.smithi107.stdout:10012: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:08.236 INFO:tasks.rados.rados.0.smithi107.stdout:10014: done (1 left) 2020-06-12T02:39:08.236 INFO:tasks.rados.rados.0.smithi107.stdout:10015: delete oid 19 current snap is 0 2020-06-12T02:39:08.243 INFO:tasks.rados.rados.0.smithi107.stdout:10012: finishing write tid 2 to smithi10729407-3 2020-06-12T02:39:08.243 INFO:tasks.rados.rados.0.smithi107.stdout:10012: finishing write tid 3 to smithi10729407-3 2020-06-12T02:39:08.244 INFO:tasks.rados.rados.0.smithi107.stdout:10012: finishing write tid 4 to smithi10729407-3 2020-06-12T02:39:08.244 INFO:tasks.rados.rados.0.smithi107.stdout:10015: done (1 left) 2020-06-12T02:39:08.244 INFO:tasks.rados.rados.0.smithi107.stdout:10016: delete oid 19 current snap is 0 2020-06-12T02:39:08.244 INFO:tasks.rados.rados.0.smithi107.stdout:10012: finishing write tid 6 to smithi10729407-3 2020-06-12T02:39:08.245 INFO:tasks.rados.rados.0.smithi107.stdout:10012: finishing write tid 7 to smithi10729407-3 2020-06-12T02:39:08.245 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1784 (ObjNum 10011 snap 0 seq_num 10011) dirty exists 2020-06-12T02:39:08.245 INFO:tasks.rados.rados.0.smithi107.stdout:10012: left oid 3 (ObjNum 10011 snap 0 seq_num 10011) 2020-06-12T02:39:08.245 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:39:08.242+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:39:08.246 INFO:tasks.rados.rados.0.smithi107.stdout:10012: done (1 left) 2020-06-12T02:39:08.246 INFO:tasks.rados.rados.0.smithi107.stdout:10016: done (0 left) 2020-06-12T02:39:08.246 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:39:08.247 INFO:tasks.rados.rados.0.smithi107.stdout:10017: seq_num 10016 ranges {0=557056,557056=499712,1056768=622592,1679360=483328} 2020-06-12T02:39:08.248 INFO:tasks.rados.rados.0.smithi107.stdout:10017: writing smithi10729407-9 from 0 to 557056 tid 1 2020-06-12T02:39:08.252 INFO:tasks.rados.rados.0.smithi107.stdout:10017: writing smithi10729407-9 from 557056 to 1056768 tid 2 2020-06-12T02:39:08.258 INFO:tasks.rados.rados.0.smithi107.stdout:10017: writing smithi10729407-9 from 1056768 to 1679360 tid 3 2020-06-12T02:39:08.260 INFO:tasks.rados.rados.0.smithi107.stdout:10017: writing smithi10729407-9 from 1679360 to 2162688 tid 4 2020-06-12T02:39:08.260 INFO:tasks.rados.rados.0.smithi107.stdout:10018: delete oid 8 current snap is 0 2020-06-12T02:39:08.261 INFO:tasks.rados.rados.0.smithi107.stdout:10017: finishing write tid 1 to smithi10729407-9 2020-06-12T02:39:08.263 INFO:tasks.rados.rados.0.smithi107.stdout:10018: done (1 left) 2020-06-12T02:39:08.264 INFO:tasks.rados.rados.0.smithi107.stdout:10019: delete oid 2 current snap is 0 2020-06-12T02:39:08.264 INFO:tasks.rados.rados.0.smithi107.stdout:10017: finishing write tid 2 to smithi10729407-9 2020-06-12T02:39:08.266 INFO:tasks.rados.rados.0.smithi107.stdout:10019: done (1 left) 2020-06-12T02:39:08.266 INFO:tasks.rados.rados.0.smithi107.stdout:10020: delete oid 7 current snap is 0 2020-06-12T02:39:08.269 INFO:tasks.rados.rados.0.smithi107.stdout:10017: finishing write tid 3 to smithi10729407-9 2020-06-12T02:39:08.269 INFO:tasks.rados.rados.0.smithi107.stdout:10017: finishing write tid 4 to smithi10729407-9 2020-06-12T02:39:08.270 INFO:tasks.rados.rados.0.smithi107.stdout:10020: done (1 left) 2020-06-12T02:39:08.270 INFO:tasks.rados.rados.0.smithi107.stdout:10021: delete oid 20 current snap is 0 2020-06-12T02:39:08.270 INFO:tasks.rados.rados.0.smithi107.stdout:10017: finishing write tid 6 to smithi10729407-9 2020-06-12T02:39:08.272 INFO:tasks.rados.rados.0.smithi107.stdout:10017: finishing write tid 7 to smithi10729407-9 2020-06-12T02:39:08.272 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1792 (ObjNum 10016 snap 0 seq_num 10016) dirty exists 2020-06-12T02:39:08.272 INFO:tasks.rados.rados.0.smithi107.stdout:10017: left oid 9 (ObjNum 10016 snap 0 seq_num 10016) 2020-06-12T02:39:08.273 INFO:tasks.rados.rados.0.smithi107.stdout:10017: done (1 left) 2020-06-12T02:39:08.273 INFO:tasks.rados.rados.0.smithi107.stdout:10021: done (0 left) 2020-06-12T02:39:08.273 INFO:tasks.rados.rados.0.smithi107.stdout:10022: delete oid 20 current snap is 0 2020-06-12T02:39:08.275 INFO:tasks.rados.rados.0.smithi107.stdout:10022: done (0 left) 2020-06-12T02:39:08.275 INFO:tasks.rados.rados.0.smithi107.stdout:10023: delete oid 2 current snap is 0 2020-06-12T02:39:08.278 INFO:tasks.rados.rados.0.smithi107.stdout:10023: done (0 left) 2020-06-12T02:39:08.278 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:39:08.278 INFO:tasks.rados.rados.0.smithi107.stdout:10024: seq_num 10023 ranges {0=671744,671744=540672,1212416=778240,1990656=286720} 2020-06-12T02:39:08.283 INFO:tasks.rados.rados.0.smithi107.stdout:10024: writing smithi10729407-16 from 0 to 671744 tid 1 2020-06-12T02:39:08.287 INFO:tasks.rados.rados.0.smithi107.stdout:10024: writing smithi10729407-16 from 671744 to 1212416 tid 2 2020-06-12T02:39:08.293 INFO:tasks.rados.rados.0.smithi107.stdout:10024: writing smithi10729407-16 from 1212416 to 1990656 tid 3 2020-06-12T02:39:08.295 INFO:tasks.rados.rados.0.smithi107.stdout:10024: writing smithi10729407-16 from 1990656 to 2277376 tid 4 2020-06-12T02:39:08.295 INFO:tasks.rados.rados.0.smithi107.stdout:10025: delete oid 14 current snap is 0 2020-06-12T02:39:08.296 INFO:tasks.rados.rados.0.smithi107.stdout:10024: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:08.297 INFO:tasks.rados.rados.0.smithi107.stdout:10025: done (1 left) 2020-06-12T02:39:08.298 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:39:08.298 INFO:tasks.rados.rados.0.smithi107.stdout:10026: seq_num 10025 ranges {0=704512,704512=712704,1417216=737280,2154496=155648} 2020-06-12T02:39:08.303 INFO:tasks.rados.rados.0.smithi107.stdout:10026: writing smithi10729407-6 from 0 to 704512 tid 1 2020-06-12T02:39:08.308 INFO:tasks.rados.rados.0.smithi107.stdout:10026: writing smithi10729407-6 from 704512 to 1417216 tid 2 2020-06-12T02:39:08.314 INFO:tasks.rados.rados.0.smithi107.stdout:10026: writing smithi10729407-6 from 1417216 to 2154496 tid 3 2020-06-12T02:39:08.315 INFO:tasks.rados.rados.0.smithi107.stdout:10026: writing smithi10729407-6 from 2154496 to 2310144 tid 4 2020-06-12T02:39:08.315 INFO:tasks.rados.rados.0.smithi107.stdout:10027: delete oid 3 current snap is 0 2020-06-12T02:39:08.315 INFO:tasks.rados.rados.0.smithi107.stdout:10024: finishing write tid 2 to smithi10729407-16 2020-06-12T02:39:08.316 INFO:tasks.rados.rados.0.smithi107.stdout:10024: finishing write tid 3 to smithi10729407-16 2020-06-12T02:39:08.316 INFO:tasks.rados.rados.0.smithi107.stdout:10024: finishing write tid 4 to smithi10729407-16 2020-06-12T02:39:08.316 INFO:tasks.rados.rados.0.smithi107.stdout:10024: finishing write tid 6 to smithi10729407-16 2020-06-12T02:39:08.316 INFO:tasks.rados.rados.0.smithi107.stdout:10024: finishing write tid 7 to smithi10729407-16 2020-06-12T02:39:08.316 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1789 (ObjNum 10023 snap 0 seq_num 10023) dirty exists 2020-06-12T02:39:08.317 INFO:tasks.rados.rados.0.smithi107.stdout:10024: left oid 16 (ObjNum 10023 snap 0 seq_num 10023) 2020-06-12T02:39:08.317 INFO:tasks.rados.rados.0.smithi107.stdout:10026: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:08.318 INFO:tasks.rados.rados.0.smithi107.stdout:10026: finishing write tid 2 to smithi10729407-6 2020-06-12T02:39:08.322 INFO:tasks.rados.rados.0.smithi107.stdout:10024: done (2 left) 2020-06-12T02:39:08.322 INFO:tasks.rados.rados.0.smithi107.stdout:10027: done (1 left) 2020-06-12T02:39:08.323 INFO:tasks.rados.rados.0.smithi107.stdout:10028: delete oid 17 current snap is 0 2020-06-12T02:39:08.323 INFO:tasks.rados.rados.0.smithi107.stdout:10026: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:08.325 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:08.325 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:08.326 INFO:tasks.rados.rados.0.smithi107.stdout:10026: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:08.326 INFO:tasks.rados.rados.0.smithi107.stdout:10026: finishing write tid 6 to smithi10729407-6 2020-06-12T02:39:08.326 INFO:tasks.rados.rados.0.smithi107.stdout:10028: done (1 left) 2020-06-12T02:39:08.326 INFO:tasks.rados.rados.0.smithi107.stdout:10029: delete oid 4 current snap is 0 2020-06-12T02:39:08.327 INFO:tasks.rados.rados.0.smithi107.stdout:10026: finishing write tid 7 to smithi10729407-6 2020-06-12T02:39:08.327 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1813 (ObjNum 10025 snap 0 seq_num 10025) dirty exists 2020-06-12T02:39:08.327 INFO:tasks.rados.rados.0.smithi107.stdout:10026: left oid 6 (ObjNum 10025 snap 0 seq_num 10025) 2020-06-12T02:39:08.330 INFO:tasks.rados.rados.0.smithi107.stdout:10026: done (1 left) 2020-06-12T02:39:08.330 INFO:tasks.rados.rados.0.smithi107.stdout:10029: done (0 left) 2020-06-12T02:39:08.330 INFO:tasks.rados.rados.0.smithi107.stdout:10030: delete oid 17 current snap is 0 2020-06-12T02:39:08.336 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-06-12T02:39:08.338 INFO:tasks.rados.rados.0.smithi107.stdout:10030: done (0 left) 2020-06-12T02:39:08.338 INFO:tasks.rados.rados.0.smithi107.stdout:10031: delete oid 2 current snap is 0 2020-06-12T02:39:08.339 INFO:tasks.rados.rados.0.smithi107.stdout:10031: done (0 left) 2020-06-12T02:39:08.339 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:39:08.340 INFO:tasks.rados.rados.0.smithi107.stdout:10032: seq_num 10031 ranges {0=802816,802816=466944,1269760=737280,2007040=401408} 2020-06-12T02:39:08.342 INFO:tasks.rados.rados.0.smithi107.stdout:10032: writing smithi10729407-19 from 0 to 802816 tid 1 2020-06-12T02:39:08.344 INFO:tasks.rados.rados.0.smithi107.stdout:10032: writing smithi10729407-19 from 802816 to 1269760 tid 2 2020-06-12T02:39:08.345 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:39:08.343+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:39:08.349 INFO:tasks.rados.rados.0.smithi107.stdout:10032: writing smithi10729407-19 from 1269760 to 2007040 tid 3 2020-06-12T02:39:08.352 INFO:tasks.rados.rados.0.smithi107.stdout:10032: writing smithi10729407-19 from 2007040 to 2408448 tid 4 2020-06-12T02:39:08.352 INFO:tasks.rados.rados.0.smithi107.stdout:10033: delete oid 20 current snap is 0 2020-06-12T02:39:08.352 INFO:tasks.rados.rados.0.smithi107.stdout:10032: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:08.355 INFO:tasks.rados.rados.0.smithi107.stdout:10033: done (1 left) 2020-06-12T02:39:08.356 INFO:tasks.rados.rados.0.smithi107.stdout:10034: delete oid 14 current snap is 0 2020-06-12T02:39:08.358 INFO:tasks.rados.rados.0.smithi107.stdout:10034: done (1 left) 2020-06-12T02:39:08.358 INFO:tasks.rados.rados.0.smithi107.stdout:10035: delete oid 16 current snap is 0 2020-06-12T02:39:08.362 INFO:tasks.rados.rados.0.smithi107.stdout:10035: done (1 left) 2020-06-12T02:39:08.363 INFO:tasks.rados.rados.0.smithi107.stdout:10036: delete oid 7 current snap is 0 2020-06-12T02:39:08.365 INFO:tasks.rados.rados.0.smithi107.stdout:10032: finishing write tid 2 to smithi10729407-19 2020-06-12T02:39:08.365 INFO:tasks.rados.rados.0.smithi107.stdout:10032: finishing write tid 3 to smithi10729407-19 2020-06-12T02:39:08.366 INFO:tasks.rados.rados.0.smithi107.stdout:10036: done (1 left) 2020-06-12T02:39:08.366 INFO:tasks.rados.rados.0.smithi107.stdout:10037: delete oid 16 current snap is 0 2020-06-12T02:39:08.366 INFO:tasks.rados.rados.0.smithi107.stdout:10032: finishing write tid 4 to smithi10729407-19 2020-06-12T02:39:08.366 INFO:tasks.rados.rados.0.smithi107.stdout:10032: finishing write tid 6 to smithi10729407-19 2020-06-12T02:39:08.368 INFO:tasks.rados.rados.0.smithi107.stdout:10032: finishing write tid 7 to smithi10729407-19 2020-06-12T02:39:08.368 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1845 (ObjNum 10031 snap 0 seq_num 10031) dirty exists 2020-06-12T02:39:08.368 INFO:tasks.rados.rados.0.smithi107.stdout:10032: left oid 19 (ObjNum 10031 snap 0 seq_num 10031) 2020-06-12T02:39:08.369 INFO:tasks.rados.rados.0.smithi107.stdout:10032: done (1 left) 2020-06-12T02:39:08.370 INFO:tasks.rados.rados.0.smithi107.stdout:10037: done (0 left) 2020-06-12T02:39:08.370 INFO:tasks.rados.rados.0.smithi107.stdout:10038: delete oid 19 current snap is 0 2020-06-12T02:39:08.374 INFO:tasks.rados.rados.0.smithi107.stdout:10038: done (0 left) 2020-06-12T02:39:08.374 INFO:tasks.rados.rados.0.smithi107.stdout:10039: delete oid 12 current snap is 0 2020-06-12T02:39:08.380 INFO:tasks.rados.rados.0.smithi107.stdout:10039: done (0 left) 2020-06-12T02:39:08.380 INFO:tasks.rados.rados.0.smithi107.stdout:10040: delete oid 9 current snap is 0 2020-06-12T02:39:08.383 INFO:tasks.rados.rados.0.smithi107.stdout:10040: done (0 left) 2020-06-12T02:39:08.383 INFO:tasks.rados.rados.0.smithi107.stdout:10041: delete oid 18 current snap is 0 2020-06-12T02:39:08.389 INFO:tasks.rados.rados.0.smithi107.stdout:10041: done (0 left) 2020-06-12T02:39:08.389 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:39:08.389 INFO:tasks.rados.rados.0.smithi107.stdout:10042: seq_num 10041 ranges {0=172032} 2020-06-12T02:39:08.390 INFO:tasks.rados.rados.0.smithi107.stdout:10042: writing smithi10729407-2 from 0 to 172032 tid 1 2020-06-12T02:39:08.390 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:39:08.390 INFO:tasks.rados.rados.0.smithi107.stdout:10043: seq_num 10042 ranges {0=188416} 2020-06-12T02:39:08.391 INFO:tasks.rados.rados.0.smithi107.stdout:10043: writing smithi10729407-15 from 0 to 188416 tid 1 2020-06-12T02:39:08.391 INFO:tasks.rados.rados.0.smithi107.stdout:10044: delete oid 11 current snap is 0 2020-06-12T02:39:08.395 INFO:tasks.rados.rados.0.smithi107.stdout:10044: done (2 left) 2020-06-12T02:39:08.395 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:39:08.395 INFO:tasks.rados.rados.0.smithi107.stdout:10045: seq_num 10044 ranges {0=221184} 2020-06-12T02:39:08.398 INFO:tasks.rados.rados.0.smithi107.stdout:10045: writing smithi10729407-14 from 0 to 221184 tid 1 2020-06-12T02:39:08.398 INFO:tasks.rados.rados.0.smithi107.stdout:10046: delete oid 6 current snap is 0 2020-06-12T02:39:08.398 INFO:tasks.rados.rados.0.smithi107.stdout:10042: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:08.398 INFO:tasks.rados.rados.0.smithi107.stdout:10043: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:08.399 INFO:tasks.rados.rados.0.smithi107.stdout:10043: finishing write tid 3 to smithi10729407-15 2020-06-12T02:39:08.400 INFO:tasks.rados.rados.0.smithi107.stdout:10042: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:08.402 INFO:tasks.rados.rados.0.smithi107.stdout:10043: finishing write tid 4 to smithi10729407-15 2020-06-12T02:39:08.402 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1858 (ObjNum 10042 snap 0 seq_num 10042) dirty exists 2020-06-12T02:39:08.402 INFO:tasks.rados.rados.0.smithi107.stdout:10043: left oid 15 (ObjNum 10042 snap 0 seq_num 10042) 2020-06-12T02:39:08.403 INFO:tasks.rados.rados.0.smithi107.stdout:10045: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:08.403 INFO:tasks.rados.rados.0.smithi107.stdout:10042: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:08.403 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 948 (ObjNum 10041 snap 0 seq_num 10041) dirty exists 2020-06-12T02:39:08.404 INFO:tasks.rados.rados.0.smithi107.stdout:10042: left oid 2 (ObjNum 10041 snap 0 seq_num 10041) 2020-06-12T02:39:08.404 INFO:tasks.rados.rados.0.smithi107.stdout:10042: done (3 left) 2020-06-12T02:39:08.404 INFO:tasks.rados.rados.0.smithi107.stdout:10043: done (2 left) 2020-06-12T02:39:08.404 INFO:tasks.rados.rados.0.smithi107.stdout:10046: done (1 left) 2020-06-12T02:39:08.404 INFO:tasks.rados.rados.0.smithi107.stdout:10047: delete oid 9 current snap is 0 2020-06-12T02:39:08.405 INFO:tasks.rados.rados.0.smithi107.stdout:10045: finishing write tid 3 to smithi10729407-14 2020-06-12T02:39:08.406 INFO:tasks.rados.rados.0.smithi107.stdout:10045: finishing write tid 4 to smithi10729407-14 2020-06-12T02:39:08.407 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1648 (ObjNum 10044 snap 0 seq_num 10044) dirty exists 2020-06-12T02:39:08.407 INFO:tasks.rados.rados.0.smithi107.stdout:10045: left oid 14 (ObjNum 10044 snap 0 seq_num 10044) 2020-06-12T02:39:08.407 INFO:tasks.rados.rados.0.smithi107.stdout:10045: done (1 left) 2020-06-12T02:39:08.407 INFO:tasks.rados.rados.0.smithi107.stdout:10047: done (0 left) 2020-06-12T02:39:08.408 INFO:tasks.rados.rados.0.smithi107.stdout:10048: delete oid 19 current snap is 0 2020-06-12T02:39:08.410 INFO:tasks.rados.rados.0.smithi107.stdout:10048: done (0 left) 2020-06-12T02:39:08.410 INFO:tasks.rados.rados.0.smithi107.stdout:10049: delete oid 17 current snap is 0 2020-06-12T02:39:08.413 INFO:tasks.rados.rados.0.smithi107.stdout:10049: done (0 left) 2020-06-12T02:39:08.413 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:39:08.413 INFO:tasks.rados.rados.0.smithi107.stdout:10050: seq_num 10049 ranges {0=303104} 2020-06-12T02:39:08.414 INFO:tasks.rados.rados.0.smithi107.stdout:10050: writing smithi10729407-7 from 0 to 303104 tid 1 2020-06-12T02:39:08.415 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:39:08.415 INFO:tasks.rados.rados.0.smithi107.stdout:10051: seq_num 10050 ranges {0=319488} 2020-06-12T02:39:08.416 INFO:tasks.rados.rados.0.smithi107.stdout:10051: writing smithi10729407-4 from 0 to 319488 tid 1 2020-06-12T02:39:08.416 INFO:tasks.rados.rados.0.smithi107.stdout:10052: delete oid 17 current snap is 0 2020-06-12T02:39:08.418 INFO:tasks.rados.rados.0.smithi107.stdout:10050: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:08.422 INFO:tasks.rados.rados.0.smithi107.stdout:10050: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:08.423 INFO:tasks.rados.rados.0.smithi107.stdout:10051: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:08.427 INFO:tasks.rados.rados.0.smithi107.stdout:10050: finishing write tid 4 to smithi10729407-7 2020-06-12T02:39:08.428 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1699 (ObjNum 10049 snap 0 seq_num 10049) dirty exists 2020-06-12T02:39:08.428 INFO:tasks.rados.rados.0.smithi107.stdout:10050: left oid 7 (ObjNum 10049 snap 0 seq_num 10049) 2020-06-12T02:39:08.428 INFO:tasks.rados.rados.0.smithi107.stdout:10051: finishing write tid 3 to smithi10729407-4 2020-06-12T02:39:08.429 INFO:tasks.rados.rados.0.smithi107.stdout:10051: finishing write tid 4 to smithi10729407-4 2020-06-12T02:39:08.429 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1886 (ObjNum 10050 snap 0 seq_num 10050) dirty exists 2020-06-12T02:39:08.429 INFO:tasks.rados.rados.0.smithi107.stdout:10051: left oid 4 (ObjNum 10050 snap 0 seq_num 10050) 2020-06-12T02:39:08.431 INFO:tasks.rados.rados.0.smithi107.stdout:10050: done (2 left) 2020-06-12T02:39:08.431 INFO:tasks.rados.rados.0.smithi107.stdout:10051: done (1 left) 2020-06-12T02:39:08.431 INFO:tasks.rados.rados.0.smithi107.stdout:10052: done (0 left) 2020-06-12T02:39:08.431 INFO:tasks.rados.rados.0.smithi107.stdout:10053: delete oid 8 current snap is 0 2020-06-12T02:39:08.435 INFO:tasks.rados.rados.0.smithi107.stdout:10053: done (0 left) 2020-06-12T02:39:08.435 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:39:08.435 INFO:tasks.rados.rados.0.smithi107.stdout:10054: seq_num 10053 ranges {0=376832} 2020-06-12T02:39:08.437 INFO:tasks.rados.rados.0.smithi107.stdout:10054: writing smithi10729407-12 from 0 to 376832 tid 1 2020-06-12T02:39:08.437 INFO:tasks.rados.rados.0.smithi107.stdout:10055: delete oid 20 current snap is 0 2020-06-12T02:39:08.439 INFO:tasks.rados.rados.0.smithi107.stdout:10055: done (1 left) 2020-06-12T02:39:08.439 INFO:tasks.rados.rados.0.smithi107.stdout:10056: delete oid 7 current snap is 0 2020-06-12T02:39:08.441 INFO:teuthology.orchestra.run.smithi107.stdout:moved 0 / 96 (0%) 2020-06-12T02:39:08.441 INFO:teuthology.orchestra.run.smithi107.stdout:avg 12 2020-06-12T02:39:08.441 INFO:teuthology.orchestra.run.smithi107.stdout:stddev 2.95804 -> 2.95804 (expected baseline 3.24037) 2020-06-12T02:39:08.441 INFO:teuthology.orchestra.run.smithi107.stdout:min osd.2 with 6 -> 6 pgs (0.5 -> 0.5 * mean) 2020-06-12T02:39:08.441 INFO:teuthology.orchestra.run.smithi107.stdout:max osd.7 with 16 -> 16 pgs (1.33333 -> 1.33333 * mean) 2020-06-12T02:39:08.442 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:39:08.442 INFO:teuthology.orchestra.run.smithi107.stdout:oload 110 2020-06-12T02:39:08.442 INFO:teuthology.orchestra.run.smithi107.stdout:max_change 0.05 2020-06-12T02:39:08.446 INFO:teuthology.orchestra.run.smithi107.stdout:max_change_osds 4 2020-06-12T02:39:08.446 INFO:teuthology.orchestra.run.smithi107.stdout:average_utilization 0.0112 2020-06-12T02:39:08.447 INFO:teuthology.orchestra.run.smithi107.stdout:overload_utilization 0.0123 2020-06-12T02:39:08.449 INFO:tasks.rados.rados.0.smithi107.stdout:10056: done (1 left) 2020-06-12T02:39:08.449 INFO:tasks.rados.rados.0.smithi107.stdout:10057: delete oid 19 current snap is 0 2020-06-12T02:39:08.449 INFO:tasks.rados.rados.0.smithi107.stdout:10054: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:08.449 INFO:tasks.rados.rados.0.smithi107.stdout:10054: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:08.450 INFO:tasks.rados.rados.0.smithi107.stdout:10054: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:08.450 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1850 (ObjNum 10053 snap 0 seq_num 10053) dirty exists 2020-06-12T02:39:08.450 INFO:tasks.rados.rados.0.smithi107.stdout:10054: left oid 12 (ObjNum 10053 snap 0 seq_num 10053) 2020-06-12T02:39:08.450 INFO:tasks.rados.rados.0.smithi107.stdout:10054: done (1 left) 2020-06-12T02:39:08.450 INFO:tasks.rados.rados.0.smithi107.stdout:10057: done (0 left) 2020-06-12T02:39:08.451 INFO:tasks.rados.rados.0.smithi107.stdout:10058: delete oid 18 current snap is 0 2020-06-12T02:39:08.451 INFO:teuthology.orchestra.run.smithi107.stderr:no change 2020-06-12T02:39:08.451 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:39:08.446+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:39:08.452 INFO:tasks.rados.rados.0.smithi107.stdout:10058: done (0 left) 2020-06-12T02:39:08.452 INFO:tasks.rados.rados.0.smithi107.stdout:10059: delete oid 18 current snap is 0 2020-06-12T02:39:08.453 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2020-06-12T02:39:08.454 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd reweight-by-utilization 110 3.0 2020-06-12T02:39:08.454 INFO:tasks.rados.rados.0.smithi107.stdout:10059: done (0 left) 2020-06-12T02:39:08.455 INFO:tasks.rados.rados.0.smithi107.stdout:10060: delete oid 3 current snap is 0 2020-06-12T02:39:08.456 INFO:tasks.rados.rados.0.smithi107.stdout:10060: done (0 left) 2020-06-12T02:39:08.457 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:39:08.457 INFO:tasks.rados.rados.0.smithi107.stdout:10061: seq_num 10060 ranges {0=491520} 2020-06-12T02:39:08.460 INFO:tasks.rados.rados.0.smithi107.stdout:10061: writing smithi10729407-5 from 0 to 491520 tid 1 2020-06-12T02:39:08.460 INFO:tasks.rados.rados.0.smithi107.stdout:10062: delete oid 4 current snap is 0 2020-06-12T02:39:08.463 INFO:tasks.rados.rados.0.smithi107.stdout:10062: done (1 left) 2020-06-12T02:39:08.463 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:39:08.463 INFO:tasks.rados.rados.0.smithi107.stdout:10063: seq_num 10062 ranges {0=524288} 2020-06-12T02:39:08.466 INFO:tasks.rados.rados.0.smithi107.stdout:10063: writing smithi10729407-20 from 0 to 524288 tid 1 2020-06-12T02:39:08.466 INFO:tasks.rados.rados.0.smithi107.stdout:10064: delete oid 13 current snap is 0 2020-06-12T02:39:08.468 INFO:tasks.rados.rados.0.smithi107.stdout:10061: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:08.468 INFO:tasks.rados.rados.0.smithi107.stdout:10061: finishing write tid 3 to smithi10729407-5 2020-06-12T02:39:08.469 INFO:tasks.rados.rados.0.smithi107.stdout:10061: finishing write tid 4 to smithi10729407-5 2020-06-12T02:39:08.469 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 957 (ObjNum 10060 snap 0 seq_num 10060) dirty exists 2020-06-12T02:39:08.470 INFO:tasks.rados.rados.0.smithi107.stdout:10061: left oid 5 (ObjNum 10060 snap 0 seq_num 10060) 2020-06-12T02:39:08.472 INFO:tasks.rados.rados.0.smithi107.stdout:10061: done (2 left) 2020-06-12T02:39:08.472 INFO:tasks.rados.rados.0.smithi107.stdout:10064: done (1 left) 2020-06-12T02:39:08.473 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:39:08.473 INFO:tasks.rados.rados.0.smithi107.stdout:10065: seq_num 10064 ranges {0=557056} 2020-06-12T02:39:08.475 INFO:tasks.rados.rados.0.smithi107.stdout:10065: writing smithi10729407-8 from 0 to 557056 tid 1 2020-06-12T02:39:08.476 INFO:tasks.rados.rados.0.smithi107.stdout:10066: delete oid 14 current snap is 0 2020-06-12T02:39:08.476 INFO:tasks.rados.rados.0.smithi107.stdout:10063: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:08.476 INFO:tasks.rados.rados.0.smithi107.stdout:10063: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:08.476 INFO:tasks.rados.rados.0.smithi107.stdout:10063: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:08.477 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 763 (ObjNum 10062 snap 0 seq_num 10062) dirty exists 2020-06-12T02:39:08.477 INFO:tasks.rados.rados.0.smithi107.stdout:10063: left oid 20 (ObjNum 10062 snap 0 seq_num 10062) 2020-06-12T02:39:08.477 INFO:tasks.rados.rados.0.smithi107.stdout:10063: done (2 left) 2020-06-12T02:39:08.478 INFO:tasks.rados.rados.0.smithi107.stdout:10066: done (1 left) 2020-06-12T02:39:08.478 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:39:08.478 INFO:tasks.rados.rados.0.smithi107.stdout:10067: seq_num 10066 ranges {0=589824} 2020-06-12T02:39:08.482 INFO:tasks.rados.rados.0.smithi107.stdout:10067: writing smithi10729407-10 from 0 to 589824 tid 1 2020-06-12T02:39:08.482 INFO:tasks.rados.rados.0.smithi107.stdout:10068: delete oid 17 current snap is 0 2020-06-12T02:39:08.488 INFO:tasks.rados.rados.0.smithi107.stdout:10065: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:08.488 INFO:tasks.rados.rados.0.smithi107.stdout:10065: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:08.489 INFO:tasks.rados.rados.0.smithi107.stdout:10068: done (2 left) 2020-06-12T02:39:08.489 INFO:tasks.rados.rados.0.smithi107.stdout:10069: delete oid 11 current snap is 0 2020-06-12T02:39:08.489 INFO:tasks.rados.rados.0.smithi107.stdout:10067: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:08.491 INFO:tasks.rados.rados.0.smithi107.stdout:10065: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:08.491 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1861 (ObjNum 10064 snap 0 seq_num 10064) dirty exists 2020-06-12T02:39:08.491 INFO:tasks.rados.rados.0.smithi107.stdout:10065: left oid 8 (ObjNum 10064 snap 0 seq_num 10064) 2020-06-12T02:39:08.492 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:08.492 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:08.493 INFO:tasks.rados.rados.0.smithi107.stdout:10065: done (2 left) 2020-06-12T02:39:08.493 INFO:tasks.rados.rados.0.smithi107.stdout:10069: done (1 left) 2020-06-12T02:39:08.493 INFO:tasks.rados.rados.0.smithi107.stdout:10070: delete oid 2 current snap is 0 2020-06-12T02:39:08.494 INFO:tasks.rados.rados.0.smithi107.stdout:10067: finishing write tid 3 to smithi10729407-10 2020-06-12T02:39:08.495 INFO:tasks.rados.rados.0.smithi107.stdout:10067: finishing write tid 4 to smithi10729407-10 2020-06-12T02:39:08.495 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 720 (ObjNum 10066 snap 0 seq_num 10066) dirty exists 2020-06-12T02:39:08.495 INFO:tasks.rados.rados.0.smithi107.stdout:10067: left oid 10 (ObjNum 10066 snap 0 seq_num 10066) 2020-06-12T02:39:08.496 INFO:tasks.rados.rados.0.smithi107.stdout:10067: done (1 left) 2020-06-12T02:39:08.496 INFO:tasks.rados.rados.0.smithi107.stdout:10070: done (0 left) 2020-06-12T02:39:08.496 INFO:tasks.rados.rados.0.smithi107.stdout:10071: delete oid 11 current snap is 0 2020-06-12T02:39:08.499 INFO:tasks.rados.rados.0.smithi107.stdout:10071: done (0 left) 2020-06-12T02:39:08.499 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:39:08.499 INFO:tasks.rados.rados.0.smithi107.stdout:10072: seq_num 10071 ranges {557056=679936} 2020-06-12T02:39:08.502 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-06-12T02:39:08.505 INFO:tasks.rados.rados.0.smithi107.stdout:10072: writing smithi10729407-8 from 557056 to 1236992 tid 1 2020-06-12T02:39:08.506 INFO:tasks.rados.rados.0.smithi107.stdout:10073: delete oid 6 current snap is 0 2020-06-12T02:39:08.509 INFO:tasks.rados.rados.0.smithi107.stdout:10073: done (1 left) 2020-06-12T02:39:08.509 INFO:tasks.rados.rados.0.smithi107.stdout:10074: delete oid 18 current snap is 0 2020-06-12T02:39:08.512 INFO:tasks.rados.rados.0.smithi107.stdout:10074: done (1 left) 2020-06-12T02:39:08.512 INFO:tasks.rados.rados.0.smithi107.stdout:10075: delete oid 18 current snap is 0 2020-06-12T02:39:08.514 INFO:tasks.rados.rados.0.smithi107.stdout:10072: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:08.517 INFO:tasks.rados.rados.0.smithi107.stdout:10072: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:08.517 INFO:tasks.rados.rados.0.smithi107.stdout:10075: done (1 left) 2020-06-12T02:39:08.517 INFO:tasks.rados.rados.0.smithi107.stdout:10076: delete oid 9 current snap is 0 2020-06-12T02:39:08.518 INFO:tasks.rados.rados.0.smithi107.stdout:10072: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:08.518 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1863 (ObjNum 10071 snap 0 seq_num 10071) dirty exists 2020-06-12T02:39:08.518 INFO:tasks.rados.rados.0.smithi107.stdout:10072: left oid 8 (ObjNum 10071 snap 0 seq_num 10071) 2020-06-12T02:39:08.519 INFO:tasks.rados.rados.0.smithi107.stdout:10072: done (1 left) 2020-06-12T02:39:08.519 INFO:tasks.rados.rados.0.smithi107.stdout:10076: done (0 left) 2020-06-12T02:39:08.519 INFO:tasks.rados.rados.0.smithi107.stdout:10077: delete oid 2 current snap is 0 2020-06-12T02:39:08.522 INFO:tasks.rados.rados.0.smithi107.stdout:10077: done (0 left) 2020-06-12T02:39:08.522 INFO:tasks.rados.rados.0.smithi107.stdout:10078: delete oid 4 current snap is 0 2020-06-12T02:39:08.524 INFO:tasks.rados.rados.0.smithi107.stdout:10078: done (0 left) 2020-06-12T02:39:08.525 INFO:tasks.rados.rados.0.smithi107.stdout:10079: delete oid 9 current snap is 0 2020-06-12T02:39:08.526 INFO:tasks.rados.rados.0.smithi107.stdout:10079: done (0 left) 2020-06-12T02:39:08.527 INFO:tasks.rados.rados.0.smithi107.stdout:10080: delete oid 13 current snap is 0 2020-06-12T02:39:08.534 INFO:tasks.rados.rados.0.smithi107.stdout:10080: done (0 left) 2020-06-12T02:39:08.535 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:39:08.535 INFO:tasks.rados.rados.0.smithi107.stdout:10081: seq_num 10080 ranges {188416=425984,614400=401408} 2020-06-12T02:39:08.535 INFO:tasks.rados.rados.0.smithi107.stdout:10081: writing smithi10729407-15 from 188416 to 614400 tid 1 2020-06-12T02:39:08.535 INFO:tasks.rados.rados.0.smithi107.stdout:10081: writing smithi10729407-15 from 614400 to 1015808 tid 2 2020-06-12T02:39:08.535 INFO:tasks.rados.rados.0.smithi107.stdout:10082: delete oid 11 current snap is 0 2020-06-12T02:39:08.537 INFO:tasks.rados.rados.0.smithi107.stdout:10082: done (1 left) 2020-06-12T02:39:08.537 INFO:tasks.rados.rados.0.smithi107.stdout:10083: delete oid 4 current snap is 0 2020-06-12T02:39:08.538 INFO:tasks.rados.rados.0.smithi107.stdout:10081: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:08.540 INFO:tasks.rados.rados.0.smithi107.stdout:10083: done (1 left) 2020-06-12T02:39:08.540 INFO:tasks.rados.rados.0.smithi107.stdout:10084: delete oid 5 current snap is 0 2020-06-12T02:39:08.543 INFO:tasks.rados.rados.0.smithi107.stdout:10084: done (1 left) 2020-06-12T02:39:08.543 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:39:08.543 INFO:tasks.rados.rados.0.smithi107.stdout:10085: seq_num 10084 ranges {0=491520,491520=401408} 2020-06-12T02:39:08.546 INFO:tasks.rados.rados.0.smithi107.stdout:10085: writing smithi10729407-18 from 0 to 491520 tid 1 2020-06-12T02:39:08.549 INFO:tasks.rados.rados.0.smithi107.stdout:10085: writing smithi10729407-18 from 491520 to 892928 tid 2 2020-06-12T02:39:08.549 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:39:08.549 INFO:tasks.rados.rados.0.smithi107.stdout:10086: seq_num 10085 ranges {1236992=507904,1744896=401408} 2020-06-12T02:39:08.550 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:39:08.547+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:39:08.559 INFO:tasks.rados.rados.0.smithi107.stdout:10086: writing smithi10729407-8 from 1236992 to 1744896 tid 1 2020-06-12T02:39:08.561 INFO:tasks.rados.rados.0.smithi107.stdout:10086: writing smithi10729407-8 from 1744896 to 2146304 tid 2 2020-06-12T02:39:08.562 INFO:tasks.rados.rados.0.smithi107.stdout:10087: delete oid 5 current snap is 0 2020-06-12T02:39:08.562 INFO:tasks.rados.rados.0.smithi107.stdout:10085: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:08.562 INFO:tasks.rados.rados.0.smithi107.stdout:10081: finishing write tid 2 to smithi10729407-15 2020-06-12T02:39:08.562 INFO:tasks.rados.rados.0.smithi107.stdout:10081: finishing write tid 4 to smithi10729407-15 2020-06-12T02:39:08.563 INFO:tasks.rados.rados.0.smithi107.stdout:10081: finishing write tid 5 to smithi10729407-15 2020-06-12T02:39:08.563 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1866 (ObjNum 10080 snap 0 seq_num 10080) dirty exists 2020-06-12T02:39:08.563 INFO:tasks.rados.rados.0.smithi107.stdout:10081: left oid 15 (ObjNum 10080 snap 0 seq_num 10080) 2020-06-12T02:39:08.563 INFO:tasks.rados.rados.0.smithi107.stdout:10085: finishing write tid 2 to smithi10729407-18 2020-06-12T02:39:08.563 INFO:tasks.rados.rados.0.smithi107.stdout:10085: finishing write tid 4 to smithi10729407-18 2020-06-12T02:39:08.564 INFO:tasks.rados.rados.0.smithi107.stdout:10085: finishing write tid 5 to smithi10729407-18 2020-06-12T02:39:08.564 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1799 (ObjNum 10084 snap 0 seq_num 10084) dirty exists 2020-06-12T02:39:08.565 INFO:tasks.rados.rados.0.smithi107.stdout:10085: left oid 18 (ObjNum 10084 snap 0 seq_num 10084) 2020-06-12T02:39:08.566 INFO:tasks.rados.rados.0.smithi107.stdout:10081: done (3 left) 2020-06-12T02:39:08.566 INFO:tasks.rados.rados.0.smithi107.stdout:10085: done (2 left) 2020-06-12T02:39:08.566 INFO:tasks.rados.rados.0.smithi107.stdout:10087: done (1 left) 2020-06-12T02:39:08.566 INFO:tasks.rados.rados.0.smithi107.stdout:10088: delete oid 10 current snap is 0 2020-06-12T02:39:08.567 INFO:tasks.rados.rados.0.smithi107.stdout:10088: done (1 left) 2020-06-12T02:39:08.567 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:39:08.567 INFO:tasks.rados.rados.0.smithi107.stdout:10089: seq_num 10088 ranges {0=557056,557056=401408} 2020-06-12T02:39:08.570 INFO:tasks.rados.rados.0.smithi107.stdout:10089: writing smithi10729407-5 from 0 to 557056 tid 1 2020-06-12T02:39:08.573 INFO:tasks.rados.rados.0.smithi107.stdout:10089: writing smithi10729407-5 from 557056 to 958464 tid 2 2020-06-12T02:39:08.573 INFO:tasks.rados.rados.0.smithi107.stdout:10090: delete oid 4 current snap is 0 2020-06-12T02:39:08.573 INFO:tasks.rados.rados.0.smithi107.stdout:10086: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:08.573 INFO:tasks.rados.rados.0.smithi107.stdout:10086: finishing write tid 2 to smithi10729407-8 2020-06-12T02:39:08.575 INFO:tasks.rados.rados.0.smithi107.stdout:10086: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:08.576 INFO:tasks.rados.rados.0.smithi107.stdout:10090: done (2 left) 2020-06-12T02:39:08.577 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:39:08.577 INFO:tasks.rados.rados.0.smithi107.stdout:10091: seq_num 10090 ranges {0=598016,598016=401408} 2020-06-12T02:39:08.580 INFO:tasks.rados.rados.0.smithi107.stdout:10091: writing smithi10729407-6 from 0 to 598016 tid 1 2020-06-12T02:39:08.583 INFO:tasks.rados.rados.0.smithi107.stdout:10091: writing smithi10729407-6 from 598016 to 999424 tid 2 2020-06-12T02:39:08.583 INFO:tasks.rados.rados.0.smithi107.stdout:10092: delete oid 9 current snap is 0 2020-06-12T02:39:08.583 INFO:tasks.rados.rados.0.smithi107.stdout:10086: finishing write tid 5 to smithi10729407-8 2020-06-12T02:39:08.583 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1869 (ObjNum 10085 snap 0 seq_num 10085) dirty exists 2020-06-12T02:39:08.584 INFO:tasks.rados.rados.0.smithi107.stdout:10086: left oid 8 (ObjNum 10085 snap 0 seq_num 10085) 2020-06-12T02:39:08.584 INFO:tasks.rados.rados.0.smithi107.stdout:10089: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:08.584 INFO:tasks.rados.rados.0.smithi107.stdout:10089: finishing write tid 2 to smithi10729407-5 2020-06-12T02:39:08.585 INFO:tasks.rados.rados.0.smithi107.stdout:10089: finishing write tid 4 to smithi10729407-5 2020-06-12T02:39:08.586 INFO:tasks.rados.rados.0.smithi107.stdout:10086: done (3 left) 2020-06-12T02:39:08.586 INFO:tasks.rados.rados.0.smithi107.stdout:10092: done (2 left) 2020-06-12T02:39:08.587 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:39:08.587 INFO:tasks.rados.rados.0.smithi107.stdout:10093: seq_num 10092 ranges {892928=630784,1523712=401408} 2020-06-12T02:39:08.595 INFO:tasks.rados.rados.0.smithi107.stdout:10093: writing smithi10729407-18 from 892928 to 1523712 tid 1 2020-06-12T02:39:08.597 INFO:tasks.rados.rados.0.smithi107.stdout:10093: writing smithi10729407-18 from 1523712 to 1925120 tid 2 2020-06-12T02:39:08.597 INFO:tasks.rados.rados.0.smithi107.stdout:10094: delete oid 3 current snap is 0 2020-06-12T02:39:08.598 INFO:tasks.rados.rados.0.smithi107.stdout:10089: finishing write tid 5 to smithi10729407-5 2020-06-12T02:39:08.598 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 962 (ObjNum 10088 snap 0 seq_num 10088) dirty exists 2020-06-12T02:39:08.598 INFO:tasks.rados.rados.0.smithi107.stdout:10089: left oid 5 (ObjNum 10088 snap 0 seq_num 10088) 2020-06-12T02:39:08.598 INFO:tasks.rados.rados.0.smithi107.stdout:10091: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:08.598 INFO:tasks.rados.rados.0.smithi107.stdout:10091: finishing write tid 2 to smithi10729407-6 2020-06-12T02:39:08.599 INFO:tasks.rados.rados.0.smithi107.stdout:10091: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:08.599 INFO:tasks.rados.rados.0.smithi107.stdout:10091: finishing write tid 5 to smithi10729407-6 2020-06-12T02:39:08.599 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1818 (ObjNum 10090 snap 0 seq_num 10090) dirty exists 2020-06-12T02:39:08.599 INFO:tasks.rados.rados.0.smithi107.stdout:10091: left oid 6 (ObjNum 10090 snap 0 seq_num 10090) 2020-06-12T02:39:08.601 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd unset nodeep-scrub 2020-06-12T02:39:08.602 INFO:tasks.rados.rados.0.smithi107.stdout:10089: done (3 left) 2020-06-12T02:39:08.602 INFO:tasks.rados.rados.0.smithi107.stdout:10091: done (2 left) 2020-06-12T02:39:08.603 INFO:tasks.rados.rados.0.smithi107.stdout:10094: done (1 left) 2020-06-12T02:39:08.603 INFO:tasks.rados.rados.0.smithi107.stdout:10095: delete oid 9 current snap is 0 2020-06-12T02:39:08.603 INFO:tasks.rados.rados.0.smithi107.stdout:10093: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:08.612 INFO:tasks.rados.rados.0.smithi107.stdout:10095: done (1 left) 2020-06-12T02:39:08.612 INFO:tasks.rados.rados.0.smithi107.stdout:10096: delete oid 14 current snap is 0 2020-06-12T02:39:08.613 INFO:tasks.rados.rados.0.smithi107.stdout:10093: finishing write tid 2 to smithi10729407-18 2020-06-12T02:39:08.613 INFO:tasks.rados.rados.0.smithi107.stdout:10093: finishing write tid 4 to smithi10729407-18 2020-06-12T02:39:08.614 INFO:tasks.rados.rados.0.smithi107.stdout:10096: done (1 left) 2020-06-12T02:39:08.614 INFO:tasks.rados.rados.0.smithi107.stdout:10097: delete oid 7 current snap is 0 2020-06-12T02:39:08.614 INFO:tasks.rados.rados.0.smithi107.stdout:10093: finishing write tid 5 to smithi10729407-18 2020-06-12T02:39:08.614 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1802 (ObjNum 10092 snap 0 seq_num 10092) dirty exists 2020-06-12T02:39:08.615 INFO:tasks.rados.rados.0.smithi107.stdout:10093: left oid 18 (ObjNum 10092 snap 0 seq_num 10092) 2020-06-12T02:39:08.615 INFO:tasks.rados.rados.0.smithi107.stdout:10093: done (1 left) 2020-06-12T02:39:08.616 INFO:tasks.rados.rados.0.smithi107.stdout:10097: done (0 left) 2020-06-12T02:39:08.616 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:39:08.616 INFO:tasks.rados.rados.0.smithi107.stdout:10098: seq_num 10097 ranges {524288=712704,1236992=401408} 2020-06-12T02:39:08.623 INFO:tasks.rados.rados.0.smithi107.stdout:10098: writing smithi10729407-20 from 524288 to 1236992 tid 1 2020-06-12T02:39:08.625 INFO:tasks.rados.rados.0.smithi107.stdout:10098: writing smithi10729407-20 from 1236992 to 1638400 tid 2 2020-06-12T02:39:08.625 INFO:tasks.rados.rados.0.smithi107.stdout:10099: delete oid 10 current snap is 0 2020-06-12T02:39:08.628 INFO:tasks.rados.rados.0.smithi107.stdout:10099: done (1 left) 2020-06-12T02:39:08.629 INFO:tasks.rados.rados.0.smithi107.stdout:10100: delete oid 17 current snap is 0 2020-06-12T02:39:08.630 INFO:tasks.rados.rados.0.smithi107.stdout:10098: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:08.632 INFO:tasks.rados.rados.0.smithi107.stdout:10100: done (1 left) 2020-06-12T02:39:08.632 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:39:08.633 INFO:tasks.rados.rados.0.smithi107.stdout:10101: seq_num 10100 ranges {958464=761856,1720320=401408} 2020-06-12T02:39:08.642 INFO:tasks.rados.rados.0.smithi107.stdout:10101: writing smithi10729407-5 from 958464 to 1720320 tid 1 2020-06-12T02:39:08.644 INFO:tasks.rados.rados.0.smithi107.stdout:10101: writing smithi10729407-5 from 1720320 to 2121728 tid 2 2020-06-12T02:39:08.645 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:39:08.645 INFO:tasks.rados.rados.0.smithi107.stdout:10102: seq_num 10101 ranges {376832=778240,1155072=401408} 2020-06-12T02:39:08.652 INFO:tasks.rados.rados.0.smithi107.stdout:10102: writing smithi10729407-12 from 376832 to 1155072 tid 1 2020-06-12T02:39:08.652 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:39:08.650+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:39:08.655 INFO:tasks.rados.rados.0.smithi107.stdout:10102: writing smithi10729407-12 from 1155072 to 1556480 tid 2 2020-06-12T02:39:08.655 INFO:tasks.rados.rados.0.smithi107.stdout:10103: delete oid 18 current snap is 0 2020-06-12T02:39:08.658 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:08.658 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:08.660 INFO:tasks.rados.rados.0.smithi107.stdout:10098: finishing write tid 2 to smithi10729407-20 2020-06-12T02:39:08.661 INFO:tasks.rados.rados.0.smithi107.stdout:10098: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:08.662 INFO:tasks.rados.rados.0.smithi107.stdout:10098: finishing write tid 5 to smithi10729407-20 2020-06-12T02:39:08.662 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 766 (ObjNum 10097 snap 0 seq_num 10097) dirty exists 2020-06-12T02:39:08.662 INFO:tasks.rados.rados.0.smithi107.stdout:10098: left oid 20 (ObjNum 10097 snap 0 seq_num 10097) 2020-06-12T02:39:08.663 INFO:tasks.rados.rados.0.smithi107.stdout:10101: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:08.663 INFO:tasks.rados.rados.0.smithi107.stdout:10101: finishing write tid 2 to smithi10729407-5 2020-06-12T02:39:08.663 INFO:tasks.rados.rados.0.smithi107.stdout:10101: finishing write tid 4 to smithi10729407-5 2020-06-12T02:39:08.664 INFO:tasks.rados.rados.0.smithi107.stdout:10101: finishing write tid 5 to smithi10729407-5 2020-06-12T02:39:08.664 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 965 (ObjNum 10100 snap 0 seq_num 10100) dirty exists 2020-06-12T02:39:08.664 INFO:tasks.rados.rados.0.smithi107.stdout:10101: left oid 5 (ObjNum 10100 snap 0 seq_num 10100) 2020-06-12T02:39:08.664 INFO:tasks.rados.rados.0.smithi107.stdout:10098: done (3 left) 2020-06-12T02:39:08.665 INFO:tasks.rados.rados.0.smithi107.stdout:10101: done (2 left) 2020-06-12T02:39:08.665 INFO:tasks.rados.rados.0.smithi107.stdout:10103: done (1 left) 2020-06-12T02:39:08.665 INFO:tasks.rados.rados.0.smithi107.stdout:10104: delete oid 13 current snap is 0 2020-06-12T02:39:08.666 INFO:tasks.rados.rados.0.smithi107.stdout:10102: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:08.666 INFO:tasks.rados.rados.0.smithi107.stdout:10104: done (1 left) 2020-06-12T02:39:08.666 INFO:tasks.rados.rados.0.smithi107.stdout:10105: delete oid 18 current snap is 0 2020-06-12T02:39:08.669 INFO:tasks.rados.rados.0.smithi107.stdout:10102: finishing write tid 2 to smithi10729407-12 2020-06-12T02:39:08.669 INFO:tasks.rados.rados.0.smithi107.stdout:10102: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:08.670 INFO:tasks.rados.rados.0.smithi107.stdout:10105: done (1 left) 2020-06-12T02:39:08.673 INFO:tasks.rados.rados.0.smithi107.stdout:10106: delete oid 9 current snap is 0 2020-06-12T02:39:08.674 INFO:tasks.rados.rados.0.smithi107.stdout:10102: finishing write tid 5 to smithi10729407-12 2020-06-12T02:39:08.675 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1854 (ObjNum 10101 snap 0 seq_num 10101) dirty exists 2020-06-12T02:39:08.675 INFO:tasks.rados.rados.0.smithi107.stdout:10102: left oid 12 (ObjNum 10101 snap 0 seq_num 10101) 2020-06-12T02:39:08.676 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-06-12T02:39:08.677 INFO:tasks.rados.rados.0.smithi107.stdout:10102: done (1 left) 2020-06-12T02:39:08.677 INFO:tasks.rados.rados.0.smithi107.stdout:10106: done (0 left) 2020-06-12T02:39:08.678 INFO:tasks.rados.rados.0.smithi107.stdout:10107: delete oid 14 current snap is 0 2020-06-12T02:39:08.678 INFO:tasks.rados.rados.0.smithi107.stdout:10107: done (0 left) 2020-06-12T02:39:08.678 INFO:tasks.rados.rados.0.smithi107.stdout:10108: delete oid 13 current snap is 0 2020-06-12T02:39:08.678 INFO:tasks.rados.rados.0.smithi107.stdout:10108: done (0 left) 2020-06-12T02:39:08.688 INFO:tasks.rados.rados.0.smithi107.stdout:10109: delete oid 16 current snap is 0 2020-06-12T02:39:08.689 INFO:tasks.rados.rados.0.smithi107.stdout:10109: done (0 left) 2020-06-12T02:39:08.689 INFO:tasks.rados.rados.0.smithi107.stdout:10110: delete oid 15 current snap is 0 2020-06-12T02:39:08.690 INFO:tasks.rados.rados.0.smithi107.stdout:10110: done (0 left) 2020-06-12T02:39:08.690 INFO:tasks.rados.rados.0.smithi107.stdout:10111: delete oid 1 current snap is 0 2020-06-12T02:39:08.690 INFO:tasks.rados.rados.0.smithi107.stdout:10111: done (0 left) 2020-06-12T02:39:08.690 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:39:08.690 INFO:tasks.rados.rados.0.smithi107.stdout:10112: seq_num 10111 ranges {0=548864,548864=507904,1056768=294912} 2020-06-12T02:39:08.691 INFO:tasks.rados.rados.0.smithi107.stdout:10112: writing smithi10729407-7 from 0 to 548864 tid 1 2020-06-12T02:39:08.693 INFO:tasks.rados.rados.0.smithi107.stdout:10112: writing smithi10729407-7 from 548864 to 1056768 tid 2 2020-06-12T02:39:08.693 INFO:tasks.rados.rados.0.smithi107.stdout:10112: writing smithi10729407-7 from 1056768 to 1351680 tid 3 2020-06-12T02:39:08.694 INFO:tasks.rados.rados.0.smithi107.stdout:10113: delete oid 6 current snap is 0 2020-06-12T02:39:08.694 INFO:tasks.rados.rados.0.smithi107.stdout:10112: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:08.696 INFO:tasks.rados.rados.0.smithi107.stdout:10113: done (1 left) 2020-06-12T02:39:08.696 INFO:tasks.rados.rados.0.smithi107.stdout:10114: delete oid 2 current snap is 0 2020-06-12T02:39:08.699 INFO:tasks.rados.rados.0.smithi107.stdout:10114: done (1 left) 2020-06-12T02:39:08.699 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:39:08.700 INFO:tasks.rados.rados.0.smithi107.stdout:10115: seq_num 10114 ranges {0=598016,598016=614400,1212416=188416} 2020-06-12T02:39:08.702 INFO:tasks.rados.rados.0.smithi107.stdout:10115: writing smithi10729407-1 from 0 to 598016 tid 1 2020-06-12T02:39:08.706 INFO:tasks.rados.rados.0.smithi107.stdout:10115: writing smithi10729407-1 from 598016 to 1212416 tid 2 2020-06-12T02:39:08.708 INFO:tasks.rados.rados.0.smithi107.stdout:10115: writing smithi10729407-1 from 1212416 to 1400832 tid 3 2020-06-12T02:39:08.708 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:39:08.708 INFO:tasks.rados.rados.0.smithi107.stdout:10116: seq_num 10115 ranges {0=614400,614400=499712,1114112=303104} 2020-06-12T02:39:08.711 INFO:tasks.rados.rados.0.smithi107.stdout:10116: writing smithi10729407-15 from 0 to 614400 tid 1 2020-06-12T02:39:08.716 INFO:tasks.rados.rados.0.smithi107.stdout:10116: writing smithi10729407-15 from 614400 to 1114112 tid 2 2020-06-12T02:39:08.717 INFO:tasks.rados.rados.0.smithi107.stdout:10116: writing smithi10729407-15 from 1114112 to 1417216 tid 3 2020-06-12T02:39:08.717 INFO:tasks.rados.rados.0.smithi107.stdout:10117: delete oid 9 current snap is 0 2020-06-12T02:39:08.718 INFO:tasks.rados.rados.0.smithi107.stdout:10112: finishing write tid 2 to smithi10729407-7 2020-06-12T02:39:08.718 INFO:tasks.rados.rados.0.smithi107.stdout:10112: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:08.718 INFO:tasks.rados.rados.0.smithi107.stdout:10112: finishing write tid 5 to smithi10729407-7 2020-06-12T02:39:08.718 INFO:tasks.rados.rados.0.smithi107.stdout:10112: finishing write tid 6 to smithi10729407-7 2020-06-12T02:39:08.719 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1705 (ObjNum 10111 snap 0 seq_num 10111) dirty exists 2020-06-12T02:39:08.719 INFO:tasks.rados.rados.0.smithi107.stdout:10112: left oid 7 (ObjNum 10111 snap 0 seq_num 10111) 2020-06-12T02:39:08.719 INFO:tasks.rados.rados.0.smithi107.stdout:10115: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:08.719 INFO:tasks.rados.rados.0.smithi107.stdout:10115: finishing write tid 2 to smithi10729407-1 2020-06-12T02:39:08.720 INFO:tasks.rados.rados.0.smithi107.stdout:10112: done (3 left) 2020-06-12T02:39:08.720 INFO:tasks.rados.rados.0.smithi107.stdout:10117: done (2 left) 2020-06-12T02:39:08.720 INFO:tasks.rados.rados.0.smithi107.stdout:10118: delete oid 12 current snap is 0 2020-06-12T02:39:08.721 INFO:tasks.rados.rados.0.smithi107.stdout:10115: finishing write tid 3 to smithi10729407-1 2020-06-12T02:39:08.721 INFO:tasks.rados.rados.0.smithi107.stdout:10115: finishing write tid 5 to smithi10729407-1 2020-06-12T02:39:08.724 INFO:tasks.rados.rados.0.smithi107.stdout:10118: done (2 left) 2020-06-12T02:39:08.724 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:39:08.724 INFO:tasks.rados.rados.0.smithi107.stdout:10119: seq_num 10118 ranges {0=663552,663552=548864,1212416=253952} 2020-06-12T02:39:08.727 INFO:tasks.rados.rados.0.smithi107.stdout:10119: writing smithi10729407-18 from 0 to 663552 tid 1 2020-06-12T02:39:08.731 INFO:tasks.rados.rados.0.smithi107.stdout:10119: writing smithi10729407-18 from 663552 to 1212416 tid 2 2020-06-12T02:39:08.734 INFO:tasks.rados.rados.0.smithi107.stdout:10119: writing smithi10729407-18 from 1212416 to 1466368 tid 3 2020-06-12T02:39:08.734 INFO:tasks.rados.rados.0.smithi107.stdout:10120: delete oid 10 current snap is 0 2020-06-12T02:39:08.734 INFO:tasks.rados.rados.0.smithi107.stdout:10116: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:08.734 INFO:tasks.rados.rados.0.smithi107.stdout:10115: finishing write tid 6 to smithi10729407-1 2020-06-12T02:39:08.735 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1824 (ObjNum 10114 snap 0 seq_num 10114) dirty exists 2020-06-12T02:39:08.735 INFO:tasks.rados.rados.0.smithi107.stdout:10115: left oid 1 (ObjNum 10114 snap 0 seq_num 10114) 2020-06-12T02:39:08.735 INFO:tasks.rados.rados.0.smithi107.stdout:10116: finishing write tid 2 to smithi10729407-15 2020-06-12T02:39:08.735 INFO:tasks.rados.rados.0.smithi107.stdout:10116: finishing write tid 3 to smithi10729407-15 2020-06-12T02:39:08.736 INFO:tasks.rados.rados.0.smithi107.stdout:10116: finishing write tid 5 to smithi10729407-15 2020-06-12T02:39:08.736 INFO:tasks.rados.rados.0.smithi107.stdout:10116: finishing write tid 6 to smithi10729407-15 2020-06-12T02:39:08.736 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1874 (ObjNum 10115 snap 0 seq_num 10115) dirty exists 2020-06-12T02:39:08.736 INFO:tasks.rados.rados.0.smithi107.stdout:10116: left oid 15 (ObjNum 10115 snap 0 seq_num 10115) 2020-06-12T02:39:08.737 INFO:tasks.rados.rados.0.smithi107.stdout:10115: done (3 left) 2020-06-12T02:39:08.737 INFO:tasks.rados.rados.0.smithi107.stdout:10116: done (2 left) 2020-06-12T02:39:08.738 INFO:tasks.rados.rados.0.smithi107.stdout:10120: done (1 left) 2020-06-12T02:39:08.738 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:39:08.738 INFO:tasks.rados.rados.0.smithi107.stdout:10121: seq_num 10120 ranges {1417216=696320,2113536=770048,2883584=32768} 2020-06-12T02:39:08.758 INFO:tasks.rados.rados.0.smithi107.stdout:10121: writing smithi10729407-15 from 1417216 to 2113536 tid 1 2020-06-12T02:39:08.759 INFO:tasks.rados.rados.0.smithi107.stdout:10121: writing smithi10729407-15 from 2113536 to 2883584 tid 2 2020-06-12T02:39:08.759 INFO:tasks.rados.rados.0.smithi107.stdout:10121: writing smithi10729407-15 from 2883584 to 2916352 tid 3 2020-06-12T02:39:08.759 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:39:08.759 INFO:tasks.rados.rados.0.smithi107.stdout:10122: seq_num 10121 ranges {1638400=712704,2351104=655360,3006464=147456} 2020-06-12T02:39:08.760 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:39:08.750+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:39:08.765 INFO:tasks.rados.rados.0.smithi107.stdout:10122: writing smithi10729407-20 from 1638400 to 2351104 tid 1 2020-06-12T02:39:08.770 INFO:tasks.rados.rados.0.smithi107.stdout:10122: writing smithi10729407-20 from 2351104 to 3006464 tid 2 2020-06-12T02:39:08.771 INFO:tasks.rados.rados.0.smithi107.stdout:10122: writing smithi10729407-20 from 3006464 to 3153920 tid 3 2020-06-12T02:39:08.771 INFO:tasks.rados.rados.0.smithi107.stdout:10123: delete oid 5 current snap is 0 2020-06-12T02:39:08.771 INFO:tasks.rados.rados.0.smithi107.stdout:10119: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:08.772 INFO:tasks.rados.rados.0.smithi107.stdout:10119: finishing write tid 2 to smithi10729407-18 2020-06-12T02:39:08.772 INFO:tasks.rados.rados.0.smithi107.stdout:10119: finishing write tid 3 to smithi10729407-18 2020-06-12T02:39:08.772 INFO:tasks.rados.rados.0.smithi107.stdout:10119: finishing write tid 5 to smithi10729407-18 2020-06-12T02:39:08.772 INFO:tasks.rados.rados.0.smithi107.stdout:10119: finishing write tid 6 to smithi10729407-18 2020-06-12T02:39:08.772 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1808 (ObjNum 10118 snap 0 seq_num 10118) dirty exists 2020-06-12T02:39:08.773 INFO:tasks.rados.rados.0.smithi107.stdout:10119: left oid 18 (ObjNum 10118 snap 0 seq_num 10118) 2020-06-12T02:39:08.773 INFO:tasks.rados.rados.0.smithi107.stdout:10121: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:08.773 INFO:tasks.rados.rados.0.smithi107.stdout:10121: finishing write tid 2 to smithi10729407-15 2020-06-12T02:39:08.773 INFO:tasks.rados.rados.0.smithi107.stdout:10121: finishing write tid 3 to smithi10729407-15 2020-06-12T02:39:08.773 INFO:tasks.rados.rados.0.smithi107.stdout:10121: finishing write tid 5 to smithi10729407-15 2020-06-12T02:39:08.774 INFO:tasks.rados.rados.0.smithi107.stdout:10121: finishing write tid 6 to smithi10729407-15 2020-06-12T02:39:08.774 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1878 (ObjNum 10120 snap 0 seq_num 10120) dirty exists 2020-06-12T02:39:08.774 INFO:tasks.rados.rados.0.smithi107.stdout:10121: left oid 15 (ObjNum 10120 snap 0 seq_num 10120) 2020-06-12T02:39:08.775 INFO:tasks.rados.rados.0.smithi107.stdout:10122: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:08.775 INFO:tasks.rados.rados.0.smithi107.stdout:10119: done (3 left) 2020-06-12T02:39:08.775 INFO:tasks.rados.rados.0.smithi107.stdout:10121: done (2 left) 2020-06-12T02:39:08.775 INFO:tasks.rados.rados.0.smithi107.stdout:10123: done (1 left) 2020-06-12T02:39:08.775 INFO:tasks.rados.rados.0.smithi107.stdout:10124: delete oid 16 current snap is 0 2020-06-12T02:39:08.782 INFO:tasks.rados.rados.0.smithi107.stdout:10124: done (1 left) 2020-06-12T02:39:08.782 INFO:tasks.rados.rados.0.smithi107.stdout:10125: delete oid 9 current snap is 0 2020-06-12T02:39:08.783 INFO:tasks.rados.rados.0.smithi107.stdout:10122: finishing write tid 2 to smithi10729407-20 2020-06-12T02:39:08.784 INFO:tasks.rados.rados.0.smithi107.stdout:10125: done (1 left) 2020-06-12T02:39:08.784 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:39:08.784 INFO:tasks.rados.rados.0.smithi107.stdout:10126: seq_num 10125 ranges {0=778240,778240=589824,1368064=212992} 2020-06-12T02:39:08.789 INFO:tasks.rados.rados.0.smithi107.stdout:10126: writing smithi10729407-12 from 0 to 778240 tid 1 2020-06-12T02:39:08.791 INFO:tasks.rados.rados.0.smithi107.stdout:10126: writing smithi10729407-12 from 778240 to 1368064 tid 2 2020-06-12T02:39:08.793 INFO:tasks.rados.rados.0.smithi107.stdout:10126: writing smithi10729407-12 from 1368064 to 1581056 tid 3 2020-06-12T02:39:08.793 INFO:tasks.rados.rados.0.smithi107.stdout:10127: delete oid 7 current snap is 0 2020-06-12T02:39:08.793 INFO:tasks.rados.rados.0.smithi107.stdout:10122: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:08.793 INFO:tasks.rados.rados.0.smithi107.stdout:10122: finishing write tid 5 to smithi10729407-20 2020-06-12T02:39:08.794 INFO:tasks.rados.rados.0.smithi107.stdout:10122: finishing write tid 6 to smithi10729407-20 2020-06-12T02:39:08.794 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 770 (ObjNum 10121 snap 0 seq_num 10121) dirty exists 2020-06-12T02:39:08.794 INFO:tasks.rados.rados.0.smithi107.stdout:10122: left oid 20 (ObjNum 10121 snap 0 seq_num 10121) 2020-06-12T02:39:08.796 INFO:tasks.rados.rados.0.smithi107.stdout:10122: done (2 left) 2020-06-12T02:39:08.796 INFO:tasks.rados.rados.0.smithi107.stdout:10127: done (1 left) 2020-06-12T02:39:08.796 INFO:tasks.rados.rados.0.smithi107.stdout:10128: delete oid 16 current snap is 0 2020-06-12T02:39:08.798 INFO:tasks.rados.rados.0.smithi107.stdout:10126: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:08.800 INFO:tasks.rados.rados.0.smithi107.stdout:10128: done (1 left) 2020-06-12T02:39:08.800 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:39:08.800 INFO:tasks.rados.rados.0.smithi107.stdout:10129: seq_num 10128 ranges {3153920=425984,3579904=696320,4276224=475136,4751360=32768} 2020-06-12T02:39:08.810 INFO:teuthology.orchestra.run.smithi107.stdout:moved 0 / 96 (0%) 2020-06-12T02:39:08.810 INFO:teuthology.orchestra.run.smithi107.stdout:avg 12 2020-06-12T02:39:08.811 INFO:teuthology.orchestra.run.smithi107.stdout:stddev 2.95804 -> 2.95804 (expected baseline 3.24037) 2020-06-12T02:39:08.811 INFO:teuthology.orchestra.run.smithi107.stdout:min osd.2 with 6 -> 6 pgs (0.5 -> 0.5 * mean) 2020-06-12T02:39:08.811 INFO:teuthology.orchestra.run.smithi107.stdout:max osd.7 with 16 -> 16 pgs (1.33333 -> 1.33333 * mean) 2020-06-12T02:39:08.811 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:39:08.812 INFO:teuthology.orchestra.run.smithi107.stdout:oload 110 2020-06-12T02:39:08.812 INFO:teuthology.orchestra.run.smithi107.stdout:max_change 3 2020-06-12T02:39:08.812 INFO:teuthology.orchestra.run.smithi107.stdout:max_change_osds 4 2020-06-12T02:39:08.812 INFO:teuthology.orchestra.run.smithi107.stdout:average_utilization 0.0112 2020-06-12T02:39:08.812 INFO:teuthology.orchestra.run.smithi107.stdout:overload_utilization 0.0123 2020-06-12T02:39:08.813 INFO:teuthology.orchestra.run.smithi107.stderr:no change 2020-06-12T02:39:08.819 INFO:tasks.rados.rados.0.smithi107.stdout:10129: writing smithi10729407-20 from 3153920 to 3579904 tid 1 2020-06-12T02:39:08.823 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2020-06-12T02:39:08.823 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd test-reweight-by-utilization 110 1.0 2020-06-12T02:39:08.825 INFO:tasks.rados.rados.0.smithi107.stdout:10129: writing smithi10729407-20 from 3579904 to 4276224 tid 2 2020-06-12T02:39:08.827 INFO:tasks.rados.rados.0.smithi107.stdout:10129: writing smithi10729407-20 from 4276224 to 4751360 tid 3 2020-06-12T02:39:08.828 INFO:tasks.rados.rados.0.smithi107.stdout:10129: writing smithi10729407-20 from 4751360 to 4784128 tid 4 2020-06-12T02:39:08.828 INFO:tasks.rados.rados.0.smithi107.stdout:10130: delete oid 13 current snap is 0 2020-06-12T02:39:08.829 INFO:tasks.rados.rados.0.smithi107.stdout:10126: finishing write tid 2 to smithi10729407-12 2020-06-12T02:39:08.829 INFO:tasks.rados.rados.0.smithi107.stdout:10126: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:08.829 INFO:tasks.rados.rados.0.smithi107.stdout:10126: finishing write tid 5 to smithi10729407-12 2020-06-12T02:39:08.831 INFO:tasks.rados.rados.0.smithi107.stdout:10126: finishing write tid 6 to smithi10729407-12 2020-06-12T02:39:08.831 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1859 (ObjNum 10125 snap 0 seq_num 10125) dirty exists 2020-06-12T02:39:08.831 INFO:tasks.rados.rados.0.smithi107.stdout:10126: left oid 12 (ObjNum 10125 snap 0 seq_num 10125) 2020-06-12T02:39:08.832 INFO:tasks.rados.rados.0.smithi107.stdout:10129: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:08.832 INFO:tasks.rados.rados.0.smithi107.stdout:10126: done (2 left) 2020-06-12T02:39:08.832 INFO:tasks.rados.rados.0.smithi107.stdout:10130: done (1 left) 2020-06-12T02:39:08.833 INFO:tasks.rados.rados.0.smithi107.stdout:10131: delete oid 16 current snap is 0 2020-06-12T02:39:08.836 INFO:tasks.rados.rados.0.smithi107.stdout:10131: done (1 left) 2020-06-12T02:39:08.836 INFO:tasks.rados.rados.0.smithi107.stdout:10132: delete oid 6 current snap is 0 2020-06-12T02:39:08.836 INFO:tasks.rados.rados.0.smithi107.stdout:10129: finishing write tid 2 to smithi10729407-20 2020-06-12T02:39:08.836 INFO:tasks.rados.rados.0.smithi107.stdout:10129: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:08.837 INFO:tasks.rados.rados.0.smithi107.stdout:10129: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:08.838 INFO:tasks.rados.rados.0.smithi107.stdout:10129: finishing write tid 6 to smithi10729407-20 2020-06-12T02:39:08.839 INFO:tasks.rados.rados.0.smithi107.stdout:10132: done (1 left) 2020-06-12T02:39:08.839 INFO:tasks.rados.rados.0.smithi107.stdout:10133: delete oid 9 current snap is 0 2020-06-12T02:39:08.840 INFO:tasks.rados.rados.0.smithi107.stdout:10129: finishing write tid 7 to smithi10729407-20 2020-06-12T02:39:08.840 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 775 (ObjNum 10128 snap 0 seq_num 10128) dirty exists 2020-06-12T02:39:08.840 INFO:tasks.rados.rados.0.smithi107.stdout:10129: left oid 20 (ObjNum 10128 snap 0 seq_num 10128) 2020-06-12T02:39:08.841 INFO:tasks.rados.rados.0.smithi107.stdout:10129: done (1 left) 2020-06-12T02:39:08.842 INFO:tasks.rados.rados.0.smithi107.stdout:10133: done (0 left) 2020-06-12T02:39:08.842 INFO:tasks.rados.rados.0.smithi107.stdout:10134: delete oid 20 current snap is 0 2020-06-12T02:39:08.843 INFO:tasks.rados.rados.0.smithi107.stdout:10134: done (0 left) 2020-06-12T02:39:08.844 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:39:08.844 INFO:tasks.rados.rados.0.smithi107.stdout:10135: seq_num 10134 ranges {0=532480,532480=450560,983040=466944,1449984=286720} 2020-06-12T02:39:08.844 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:08.845 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:08.847 INFO:tasks.rados.rados.0.smithi107.stdout:10135: writing smithi10729407-19 from 0 to 532480 tid 1 2020-06-12T02:39:08.860 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-06-12T02:39:08.861 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:39:08.857+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:39:08.861 INFO:tasks.rados.rados.0.smithi107.stdout:10135: writing smithi10729407-19 from 532480 to 983040 tid 2 2020-06-12T02:39:08.862 INFO:tasks.rados.rados.0.smithi107.stdout:10135: writing smithi10729407-19 from 983040 to 1449984 tid 3 2020-06-12T02:39:08.862 INFO:tasks.rados.rados.0.smithi107.stdout:10135: writing smithi10729407-19 from 1449984 to 1736704 tid 4 2020-06-12T02:39:08.862 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:39:08.862 INFO:tasks.rados.rados.0.smithi107.stdout:10136: seq_num 10135 ranges {0=548864,548864=737280,1286144=466944} 2020-06-12T02:39:08.862 INFO:tasks.rados.rados.0.smithi107.stdout:10136: writing smithi10729407-3 from 0 to 548864 tid 1 2020-06-12T02:39:08.866 INFO:tasks.rados.rados.0.smithi107.stdout:10136: writing smithi10729407-3 from 548864 to 1286144 tid 2 2020-06-12T02:39:08.867 INFO:tasks.rados.rados.0.smithi107.stdout:10136: writing smithi10729407-3 from 1286144 to 1753088 tid 3 2020-06-12T02:39:08.867 INFO:tasks.rados.rados.0.smithi107.stdout:10135: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:08.868 INFO:tasks.rados.rados.0.smithi107.stdout:10135: finishing write tid 2 to smithi10729407-19 2020-06-12T02:39:08.868 INFO:tasks.rados.rados.0.smithi107.stdout:10135: finishing write tid 3 to smithi10729407-19 2020-06-12T02:39:08.868 INFO:tasks.rados.rados.0.smithi107.stdout:10135: finishing write tid 4 to smithi10729407-19 2020-06-12T02:39:08.868 INFO:tasks.rados.rados.0.smithi107.stdout:10135: finishing write tid 6 to smithi10729407-19 2020-06-12T02:39:08.868 INFO:tasks.rados.rados.0.smithi107.stdout:10135: finishing write tid 7 to smithi10729407-19 2020-06-12T02:39:08.869 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1864 (ObjNum 10134 snap 0 seq_num 10134) dirty exists 2020-06-12T02:39:08.869 INFO:tasks.rados.rados.0.smithi107.stdout:10135: left oid 19 (ObjNum 10134 snap 0 seq_num 10134) 2020-06-12T02:39:08.869 INFO:tasks.rados.rados.0.smithi107.stdout:10135: done (1 left) 2020-06-12T02:39:08.869 INFO:tasks.rados.rados.0.smithi107.stdout:10137: delete oid 14 current snap is 0 2020-06-12T02:39:08.870 INFO:tasks.rados.rados.0.smithi107.stdout:10137: done (1 left) 2020-06-12T02:39:08.870 INFO:tasks.rados.rados.0.smithi107.stdout:10138: delete oid 5 current snap is 0 2020-06-12T02:39:08.872 INFO:tasks.rados.rados.0.smithi107.stdout:10138: done (1 left) 2020-06-12T02:39:08.873 INFO:tasks.rados.rados.0.smithi107.stdout:10139: delete oid 8 current snap is 0 2020-06-12T02:39:08.875 INFO:tasks.rados.rados.0.smithi107.stdout:10139: done (1 left) 2020-06-12T02:39:08.875 INFO:tasks.rados.rados.0.smithi107.stdout:10140: delete oid 2 current snap is 0 2020-06-12T02:39:08.877 INFO:tasks.rados.rados.0.smithi107.stdout:10136: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:08.878 INFO:tasks.rados.rados.0.smithi107.stdout:10140: done (1 left) 2020-06-12T02:39:08.878 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:39:08.878 INFO:tasks.rados.rados.0.smithi107.stdout:10136: finishing write tid 2 to smithi10729407-3 2020-06-12T02:39:08.878 INFO:tasks.rados.rados.0.smithi107.stdout:10141: seq_num 10140 ranges {0=630784,630784=557056,1187840=417792,1605632=229376} 2020-06-12T02:39:08.882 INFO:tasks.rados.rados.0.smithi107.stdout:10141: writing smithi10729407-20 from 0 to 630784 tid 1 2020-06-12T02:39:08.885 INFO:tasks.rados.rados.0.smithi107.stdout:10141: writing smithi10729407-20 from 630784 to 1187840 tid 2 2020-06-12T02:39:08.893 INFO:tasks.rados.rados.0.smithi107.stdout:10141: writing smithi10729407-20 from 1187840 to 1605632 tid 3 2020-06-12T02:39:08.894 INFO:tasks.rados.rados.0.smithi107.stdout:10141: writing smithi10729407-20 from 1605632 to 1835008 tid 4 2020-06-12T02:39:08.894 INFO:tasks.rados.rados.0.smithi107.stdout:10142: delete oid 14 current snap is 0 2020-06-12T02:39:08.894 INFO:tasks.rados.rados.0.smithi107.stdout:10136: finishing write tid 3 to smithi10729407-3 2020-06-12T02:39:08.894 INFO:tasks.rados.rados.0.smithi107.stdout:10136: finishing write tid 5 to smithi10729407-3 2020-06-12T02:39:08.895 INFO:tasks.rados.rados.0.smithi107.stdout:10136: finishing write tid 6 to smithi10729407-3 2020-06-12T02:39:08.895 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1802 (ObjNum 10135 snap 0 seq_num 10135) dirty exists 2020-06-12T02:39:08.895 INFO:tasks.rados.rados.0.smithi107.stdout:10136: left oid 3 (ObjNum 10135 snap 0 seq_num 10135) 2020-06-12T02:39:08.895 INFO:tasks.rados.rados.0.smithi107.stdout:10136: done (2 left) 2020-06-12T02:39:08.895 INFO:tasks.rados.rados.0.smithi107.stdout:10142: done (1 left) 2020-06-12T02:39:08.896 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:39:08.896 INFO:tasks.rados.rados.0.smithi107.stdout:10143: seq_num 10142 ranges {0=663552,663552=770048,1433600=425984,1859584=8192} 2020-06-12T02:39:08.897 INFO:tasks.rados.rados.0.smithi107.stdout:10143: writing smithi10729407-4 from 0 to 663552 tid 1 2020-06-12T02:39:08.903 INFO:tasks.rados.rados.0.smithi107.stdout:10143: writing smithi10729407-4 from 663552 to 1433600 tid 2 2020-06-12T02:39:08.907 INFO:tasks.rados.rados.0.smithi107.stdout:10143: writing smithi10729407-4 from 1433600 to 1859584 tid 3 2020-06-12T02:39:08.907 INFO:tasks.rados.rados.0.smithi107.stdout:10143: writing smithi10729407-4 from 1859584 to 1867776 tid 4 2020-06-12T02:39:08.907 INFO:tasks.rados.rados.0.smithi107.stdout:10144: delete oid 14 current snap is 0 2020-06-12T02:39:08.908 INFO:tasks.rados.rados.0.smithi107.stdout:10141: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:08.908 INFO:tasks.rados.rados.0.smithi107.stdout:10141: finishing write tid 2 to smithi10729407-20 2020-06-12T02:39:08.908 INFO:tasks.rados.rados.0.smithi107.stdout:10141: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:08.908 INFO:tasks.rados.rados.0.smithi107.stdout:10141: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:08.908 INFO:tasks.rados.rados.0.smithi107.stdout:10141: finishing write tid 6 to smithi10729407-20 2020-06-12T02:39:08.909 INFO:tasks.rados.rados.0.smithi107.stdout:10141: finishing write tid 7 to smithi10729407-20 2020-06-12T02:39:08.909 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 781 (ObjNum 10140 snap 0 seq_num 10140) dirty exists 2020-06-12T02:39:08.909 INFO:tasks.rados.rados.0.smithi107.stdout:10141: left oid 20 (ObjNum 10140 snap 0 seq_num 10140) 2020-06-12T02:39:08.909 INFO:tasks.rados.rados.0.smithi107.stdout:10143: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:08.910 INFO:tasks.rados.rados.0.smithi107.stdout:10141: done (2 left) 2020-06-12T02:39:08.910 INFO:tasks.rados.rados.0.smithi107.stdout:10144: done (1 left) 2020-06-12T02:39:08.910 INFO:tasks.rados.rados.0.smithi107.stdout:10145: delete oid 6 current snap is 0 2020-06-12T02:39:08.914 INFO:tasks.rados.rados.0.smithi107.stdout:10145: done (1 left) 2020-06-12T02:39:08.915 INFO:tasks.rados.rados.0.smithi107.stdout:10146: delete oid 19 current snap is 0 2020-06-12T02:39:08.915 INFO:tasks.rados.rados.0.smithi107.stdout:10143: finishing write tid 2 to smithi10729407-4 2020-06-12T02:39:08.916 INFO:tasks.rados.rados.0.smithi107.stdout:10143: finishing write tid 3 to smithi10729407-4 2020-06-12T02:39:08.917 INFO:tasks.rados.rados.0.smithi107.stdout:10143: finishing write tid 4 to smithi10729407-4 2020-06-12T02:39:08.917 INFO:tasks.rados.rados.0.smithi107.stdout:10143: finishing write tid 6 to smithi10729407-4 2020-06-12T02:39:08.918 INFO:tasks.rados.rados.0.smithi107.stdout:10146: done (1 left) 2020-06-12T02:39:08.919 INFO:tasks.rados.rados.0.smithi107.stdout:10147: delete oid 5 current snap is 0 2020-06-12T02:39:08.921 INFO:tasks.rados.rados.0.smithi107.stdout:10143: finishing write tid 7 to smithi10729407-4 2020-06-12T02:39:08.921 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1898 (ObjNum 10142 snap 0 seq_num 10142) dirty exists 2020-06-12T02:39:08.921 INFO:tasks.rados.rados.0.smithi107.stdout:10143: left oid 4 (ObjNum 10142 snap 0 seq_num 10142) 2020-06-12T02:39:08.922 INFO:tasks.rados.rados.0.smithi107.stdout:10143: done (1 left) 2020-06-12T02:39:08.922 INFO:tasks.rados.rados.0.smithi107.stdout:10147: done (0 left) 2020-06-12T02:39:08.922 INFO:tasks.rados.rados.0.smithi107.stdout:10148: delete oid 5 current snap is 0 2020-06-12T02:39:08.924 INFO:tasks.rados.rados.0.smithi107.stdout:10148: done (0 left) 2020-06-12T02:39:08.924 INFO:tasks.rados.rados.0.smithi107.stdout:10149: delete oid 4 current snap is 0 2020-06-12T02:39:08.927 INFO:tasks.rados.rados.0.smithi107.stdout:10149: done (0 left) 2020-06-12T02:39:08.928 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:39:08.928 INFO:tasks.rados.rados.0.smithi107.stdout:10150: seq_num 10149 ranges {1753088=786432,2539520=761856,3301376=442368} 2020-06-12T02:39:08.943 INFO:tasks.rados.rados.0.smithi107.stdout:10150: writing smithi10729407-3 from 1753088 to 2539520 tid 1 2020-06-12T02:39:08.949 INFO:tasks.rados.rados.0.smithi107.stdout:10150: writing smithi10729407-3 from 2539520 to 3301376 tid 2 2020-06-12T02:39:08.952 INFO:tasks.rados.rados.0.smithi107.stdout:10150: writing smithi10729407-3 from 3301376 to 3743744 tid 3 2020-06-12T02:39:08.953 INFO:tasks.rados.rados.0.smithi107.stdout:10151: delete oid 17 current snap is 0 2020-06-12T02:39:08.953 INFO:tasks.rados.rados.0.smithi107.stdout:10150: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:08.956 INFO:tasks.rados.rados.0.smithi107.stdout:10151: done (1 left) 2020-06-12T02:39:08.957 INFO:tasks.rados.rados.0.smithi107.stdout:10152: delete oid 14 current snap is 0 2020-06-12T02:39:08.959 INFO:tasks.rados.rados.0.smithi107.stdout:10152: done (1 left) 2020-06-12T02:39:08.959 INFO:tasks.rados.rados.0.smithi107.stdout:10153: delete oid 8 current snap is 0 2020-06-12T02:39:08.962 INFO:tasks.rados.rados.0.smithi107.stdout:10153: done (1 left) 2020-06-12T02:39:08.962 INFO:tasks.rados.rados.0.smithi107.stdout:10154: delete oid 1 current snap is 0 2020-06-12T02:39:08.962 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:39:08.960+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:39:08.964 INFO:tasks.rados.rados.0.smithi107.stdout:10150: finishing write tid 2 to smithi10729407-3 2020-06-12T02:39:08.965 INFO:tasks.rados.rados.0.smithi107.stdout:10150: finishing write tid 3 to smithi10729407-3 2020-06-12T02:39:08.965 INFO:tasks.rados.rados.0.smithi107.stdout:10150: finishing write tid 5 to smithi10729407-3 2020-06-12T02:39:08.969 INFO:tasks.rados.rados.0.smithi107.stdout:10154: done (1 left) 2020-06-12T02:39:08.969 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:39:08.969 INFO:tasks.rados.rados.0.smithi107.stdout:10155: seq_num 10154 ranges {0=466944,466944=638976,1105920=770048,1875968=196608} 2020-06-12T02:39:08.970 INFO:tasks.rados.rados.0.smithi107.stdout:10155: writing smithi10729407-11 from 0 to 466944 tid 1 2020-06-12T02:39:08.974 INFO:tasks.rados.rados.0.smithi107.stdout:10155: writing smithi10729407-11 from 466944 to 1105920 tid 2 2020-06-12T02:39:08.978 INFO:tasks.rados.rados.0.smithi107.stdout:10155: writing smithi10729407-11 from 1105920 to 1875968 tid 3 2020-06-12T02:39:08.979 INFO:tasks.rados.rados.0.smithi107.stdout:10155: writing smithi10729407-11 from 1875968 to 2072576 tid 4 2020-06-12T02:39:08.980 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:39:08.980 INFO:tasks.rados.rados.0.smithi107.stdout:10156: seq_num 10155 ranges {0=483328,483328=516096,999424=581632,1581056=507904} 2020-06-12T02:39:08.982 INFO:tasks.rados.rados.0.smithi107.stdout:10156: writing smithi10729407-6 from 0 to 483328 tid 1 2020-06-12T02:39:08.986 INFO:tasks.rados.rados.0.smithi107.stdout:10156: writing smithi10729407-6 from 483328 to 999424 tid 2 2020-06-12T02:39:08.990 INFO:tasks.rados.rados.0.smithi107.stdout:10156: writing smithi10729407-6 from 999424 to 1581056 tid 3 2020-06-12T02:39:08.994 INFO:tasks.rados.rados.0.smithi107.stdout:10156: writing smithi10729407-6 from 1581056 to 2088960 tid 4 2020-06-12T02:39:08.994 INFO:tasks.rados.rados.0.smithi107.stdout:10157: delete oid 16 current snap is 0 2020-06-12T02:39:08.994 INFO:tasks.rados.rados.0.smithi107.stdout:10150: finishing write tid 6 to smithi10729407-3 2020-06-12T02:39:08.995 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1806 (ObjNum 10149 snap 0 seq_num 10149) dirty exists 2020-06-12T02:39:08.995 INFO:tasks.rados.rados.0.smithi107.stdout:10150: left oid 3 (ObjNum 10149 snap 0 seq_num 10149) 2020-06-12T02:39:08.995 INFO:tasks.rados.rados.0.smithi107.stdout:10155: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:08.995 INFO:tasks.rados.rados.0.smithi107.stdout:10155: finishing write tid 2 to smithi10729407-11 2020-06-12T02:39:08.995 INFO:tasks.rados.rados.0.smithi107.stdout:10155: finishing write tid 3 to smithi10729407-11 2020-06-12T02:39:08.996 INFO:tasks.rados.rados.0.smithi107.stdout:10155: finishing write tid 4 to smithi10729407-11 2020-06-12T02:39:08.996 INFO:tasks.rados.rados.0.smithi107.stdout:10156: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:08.997 INFO:tasks.rados.rados.0.smithi107.stdout:10155: finishing write tid 6 to smithi10729407-11 2020-06-12T02:39:08.998 INFO:tasks.rados.rados.0.smithi107.stdout:10155: finishing write tid 7 to smithi10729407-11 2020-06-12T02:39:08.998 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1681 (ObjNum 10154 snap 0 seq_num 10154) dirty exists 2020-06-12T02:39:08.998 INFO:tasks.rados.rados.0.smithi107.stdout:10155: left oid 11 (ObjNum 10154 snap 0 seq_num 10154) 2020-06-12T02:39:08.999 INFO:tasks.rados.rados.0.smithi107.stdout:10150: done (3 left) 2020-06-12T02:39:08.999 INFO:tasks.rados.rados.0.smithi107.stdout:10155: done (2 left) 2020-06-12T02:39:08.999 INFO:tasks.rados.rados.0.smithi107.stdout:10157: done (1 left) 2020-06-12T02:39:08.999 INFO:tasks.rados.rados.0.smithi107.stdout:10158: delete oid 10 current snap is 0 2020-06-12T02:39:08.999 INFO:tasks.rados.rados.0.smithi107.stdout:10158: done (1 left) 2020-06-12T02:39:09.000 INFO:tasks.rados.rados.0.smithi107.stdout:10159: delete oid 2 current snap is 0 2020-06-12T02:39:09.002 INFO:tasks.rados.rados.0.smithi107.stdout:10159: done (1 left) 2020-06-12T02:39:09.003 INFO:tasks.rados.rados.0.smithi107.stdout:10160: delete oid 17 current snap is 0 2020-06-12T02:39:09.004 INFO:tasks.rados.rados.0.smithi107.stdout:10156: finishing write tid 2 to smithi10729407-6 2020-06-12T02:39:09.006 INFO:tasks.rados.rados.0.smithi107.stdout:10156: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:09.006 INFO:tasks.rados.rados.0.smithi107.stdout:10160: done (1 left) 2020-06-12T02:39:09.006 INFO:tasks.rados.rados.0.smithi107.stdout:10161: delete oid 15 current snap is 0 2020-06-12T02:39:09.007 INFO:tasks.rados.rados.0.smithi107.stdout:10156: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:09.007 INFO:tasks.rados.rados.0.smithi107.stdout:10156: finishing write tid 6 to smithi10729407-6 2020-06-12T02:39:09.008 INFO:tasks.rados.rados.0.smithi107.stdout:10161: done (1 left) 2020-06-12T02:39:09.009 INFO:tasks.rados.rados.0.smithi107.stdout:10162: delete oid 7 current snap is 0 2020-06-12T02:39:09.011 INFO:tasks.rados.rados.0.smithi107.stdout:10156: finishing write tid 7 to smithi10729407-6 2020-06-12T02:39:09.011 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1832 (ObjNum 10155 snap 0 seq_num 10155) dirty exists 2020-06-12T02:39:09.011 INFO:tasks.rados.rados.0.smithi107.stdout:10156: left oid 6 (ObjNum 10155 snap 0 seq_num 10155) 2020-06-12T02:39:09.012 INFO:tasks.rados.rados.0.smithi107.stdout:10156: done (1 left) 2020-06-12T02:39:09.013 INFO:tasks.rados.rados.0.smithi107.stdout:10162: done (0 left) 2020-06-12T02:39:09.013 INFO:tasks.rados.rados.0.smithi107.stdout:10163: delete oid 20 current snap is 0 2020-06-12T02:39:09.015 INFO:tasks.rados.rados.0.smithi107.stdout:10163: done (0 left) 2020-06-12T02:39:09.015 INFO:tasks.rados.rados.0.smithi107.stdout:10164: delete oid 15 current snap is 0 2020-06-12T02:39:09.018 INFO:tasks.rados.rados.0.smithi107.stdout:10164: done (0 left) 2020-06-12T02:39:09.018 INFO:tasks.rados.rados.0.smithi107.stdout:10165: delete oid 19 current snap is 0 2020-06-12T02:39:09.020 INFO:tasks.rados.rados.0.smithi107.stdout:10165: done (0 left) 2020-06-12T02:39:09.021 INFO:tasks.rados.rados.0.smithi107.stdout:10166: delete oid 2 current snap is 0 2020-06-12T02:39:09.022 INFO:tasks.rados.rados.0.smithi107.stdout:10166: done (0 left) 2020-06-12T02:39:09.023 INFO:tasks.rados.rados.0.smithi107.stdout:10167: delete oid 14 current snap is 0 2020-06-12T02:39:09.025 INFO:tasks.rados.rados.0.smithi107.stdout:10167: done (0 left) 2020-06-12T02:39:09.027 INFO:tasks.rados.rados.0.smithi107.stdout:10168: delete oid 15 current snap is 0 2020-06-12T02:39:09.027 INFO:tasks.rados.rados.0.smithi107.stdout:10168: done (0 left) 2020-06-12T02:39:09.028 INFO:tasks.rados.rados.0.smithi107.stdout:10169: delete oid 16 current snap is 0 2020-06-12T02:39:09.030 INFO:tasks.rados.rados.0.smithi107.stdout:10169: done (0 left) 2020-06-12T02:39:09.030 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:39:09.030 INFO:tasks.rados.rados.0.smithi107.stdout:10170: seq_num 10169 ranges {0=720896,720896=598016,1318912=483328,1802240=524288} 2020-06-12T02:39:09.034 INFO:tasks.rados.rados.0.smithi107.stdout:10170: writing smithi10729407-15 from 0 to 720896 tid 1 2020-06-12T02:39:09.040 INFO:tasks.rados.rados.0.smithi107.stdout:10170: writing smithi10729407-15 from 720896 to 1318912 tid 2 2020-06-12T02:39:09.040 INFO:tasks.rados.rados.0.smithi107.stdout:10170: writing smithi10729407-15 from 1318912 to 1802240 tid 3 2020-06-12T02:39:09.043 INFO:tasks.rados.rados.0.smithi107.stdout:10170: writing smithi10729407-15 from 1802240 to 2326528 tid 4 2020-06-12T02:39:09.044 INFO:tasks.rados.rados.0.smithi107.stdout:10171: delete oid 8 current snap is 0 2020-06-12T02:39:09.044 INFO:tasks.rados.rados.0.smithi107.stdout:10170: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:09.048 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:09.048 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:09.053 INFO:tasks.rados.rados.0.smithi107.stdout:10170: finishing write tid 2 to smithi10729407-15 2020-06-12T02:39:09.053 INFO:tasks.rados.rados.0.smithi107.stdout:10170: finishing write tid 3 to smithi10729407-15 2020-06-12T02:39:09.068 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-06-12T02:39:09.068 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:39:09.064+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:39:09.069 INFO:tasks.rados.rados.0.smithi107.stdout:10170: finishing write tid 4 to smithi10729407-15 2020-06-12T02:39:09.069 INFO:tasks.rados.rados.0.smithi107.stdout:10170: finishing write tid 6 to smithi10729407-15 2020-06-12T02:39:09.069 INFO:tasks.rados.rados.0.smithi107.stdout:10171: done (1 left) 2020-06-12T02:39:09.069 INFO:tasks.rados.rados.0.smithi107.stdout:10172: delete oid 7 current snap is 0 2020-06-12T02:39:09.070 INFO:tasks.rados.rados.0.smithi107.stdout:10170: finishing write tid 7 to smithi10729407-15 2020-06-12T02:39:09.070 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1888 (ObjNum 10169 snap 0 seq_num 10169) dirty exists 2020-06-12T02:39:09.070 INFO:tasks.rados.rados.0.smithi107.stdout:10170: left oid 15 (ObjNum 10169 snap 0 seq_num 10169) 2020-06-12T02:39:09.070 INFO:tasks.rados.rados.0.smithi107.stdout:10170: done (1 left) 2020-06-12T02:39:09.070 INFO:tasks.rados.rados.0.smithi107.stdout:10172: done (0 left) 2020-06-12T02:39:09.071 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:39:09.071 INFO:tasks.rados.rados.0.smithi107.stdout:10173: seq_num 10172 ranges {0=770048,770048=704512,1474560=704512,2179072=196608} 2020-06-12T02:39:09.071 INFO:tasks.rados.rados.0.smithi107.stdout:10173: writing smithi10729407-17 from 0 to 770048 tid 1 2020-06-12T02:39:09.071 INFO:tasks.rados.rados.0.smithi107.stdout:10173: writing smithi10729407-17 from 770048 to 1474560 tid 2 2020-06-12T02:39:09.072 INFO:tasks.rados.rados.0.smithi107.stdout:10173: writing smithi10729407-17 from 1474560 to 2179072 tid 3 2020-06-12T02:39:09.078 INFO:tasks.rados.rados.0.smithi107.stdout:10173: writing smithi10729407-17 from 2179072 to 2375680 tid 4 2020-06-12T02:39:09.078 INFO:tasks.rados.rados.0.smithi107.stdout:10174: delete oid 4 current snap is 0 2020-06-12T02:39:09.079 INFO:tasks.rados.rados.0.smithi107.stdout:10173: finishing write tid 1 to smithi10729407-17 2020-06-12T02:39:09.080 INFO:tasks.rados.rados.0.smithi107.stdout:10173: finishing write tid 2 to smithi10729407-17 2020-06-12T02:39:09.080 INFO:tasks.rados.rados.0.smithi107.stdout:10173: finishing write tid 3 to smithi10729407-17 2020-06-12T02:39:09.082 INFO:tasks.rados.rados.0.smithi107.stdout:10173: finishing write tid 4 to smithi10729407-17 2020-06-12T02:39:09.083 INFO:tasks.rados.rados.0.smithi107.stdout:10173: finishing write tid 6 to smithi10729407-17 2020-06-12T02:39:09.084 INFO:tasks.rados.rados.0.smithi107.stdout:10174: done (1 left) 2020-06-12T02:39:09.084 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:39:09.084 INFO:tasks.rados.rados.0.smithi107.stdout:10175: seq_num 10174 ranges {1466368=802816,2269184=466944,2736128=671744,3407872=466944} 2020-06-12T02:39:09.098 INFO:tasks.rados.rados.0.smithi107.stdout:10175: writing smithi10729407-18 from 1466368 to 2269184 tid 1 2020-06-12T02:39:09.099 INFO:tasks.rados.rados.0.smithi107.stdout:10175: writing smithi10729407-18 from 2269184 to 2736128 tid 2 2020-06-12T02:39:09.104 INFO:tasks.rados.rados.0.smithi107.stdout:10175: writing smithi10729407-18 from 2736128 to 3407872 tid 3 2020-06-12T02:39:09.108 INFO:tasks.rados.rados.0.smithi107.stdout:10175: writing smithi10729407-18 from 3407872 to 3874816 tid 4 2020-06-12T02:39:09.108 INFO:tasks.rados.rados.0.smithi107.stdout:10176: delete oid 1 current snap is 0 2020-06-12T02:39:09.108 INFO:tasks.rados.rados.0.smithi107.stdout:10173: finishing write tid 7 to smithi10729407-17 2020-06-12T02:39:09.109 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1906 (ObjNum 10172 snap 0 seq_num 10172) dirty exists 2020-06-12T02:39:09.109 INFO:tasks.rados.rados.0.smithi107.stdout:10173: left oid 17 (ObjNum 10172 snap 0 seq_num 10172) 2020-06-12T02:39:09.109 INFO:tasks.rados.rados.0.smithi107.stdout:10175: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:09.111 INFO:tasks.rados.rados.0.smithi107.stdout:10173: done (2 left) 2020-06-12T02:39:09.111 INFO:tasks.rados.rados.0.smithi107.stdout:10176: done (1 left) 2020-06-12T02:39:09.111 INFO:tasks.rados.rados.0.smithi107.stdout:10177: delete oid 8 current snap is 0 2020-06-12T02:39:09.115 INFO:tasks.rados.rados.0.smithi107.stdout:10177: done (1 left) 2020-06-12T02:39:09.115 INFO:tasks.rados.rados.0.smithi107.stdout:10178: delete oid 12 current snap is 0 2020-06-12T02:39:09.117 INFO:tasks.rados.rados.0.smithi107.stdout:10175: finishing write tid 2 to smithi10729407-18 2020-06-12T02:39:09.117 INFO:tasks.rados.rados.0.smithi107.stdout:10175: finishing write tid 3 to smithi10729407-18 2020-06-12T02:39:09.118 INFO:tasks.rados.rados.0.smithi107.stdout:10178: done (1 left) 2020-06-12T02:39:09.119 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:39:09.119 INFO:tasks.rados.rados.0.smithi107.stdout:10179: seq_num 10178 ranges {0=65536} 2020-06-12T02:39:09.119 INFO:tasks.rados.rados.0.smithi107.stdout:10179: writing smithi10729407-8 from 0 to 65536 tid 1 2020-06-12T02:39:09.119 INFO:tasks.rados.rados.0.smithi107.stdout:10180: delete oid 10 current snap is 0 2020-06-12T02:39:09.120 INFO:tasks.rados.rados.0.smithi107.stdout:10175: finishing write tid 4 to smithi10729407-18 2020-06-12T02:39:09.120 INFO:tasks.rados.rados.0.smithi107.stdout:10175: finishing write tid 6 to smithi10729407-18 2020-06-12T02:39:09.120 INFO:tasks.rados.rados.0.smithi107.stdout:10175: finishing write tid 7 to smithi10729407-18 2020-06-12T02:39:09.120 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1813 (ObjNum 10174 snap 0 seq_num 10174) dirty exists 2020-06-12T02:39:09.121 INFO:tasks.rados.rados.0.smithi107.stdout:10175: left oid 18 (ObjNum 10174 snap 0 seq_num 10174) 2020-06-12T02:39:09.122 INFO:tasks.rados.rados.0.smithi107.stdout:10175: done (2 left) 2020-06-12T02:39:09.122 INFO:tasks.rados.rados.0.smithi107.stdout:10180: done (1 left) 2020-06-12T02:39:09.123 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:39:09.123 INFO:tasks.rados.rados.0.smithi107.stdout:10181: seq_num 10180 ranges {0=98304} 2020-06-12T02:39:09.123 INFO:tasks.rados.rados.0.smithi107.stdout:10181: writing smithi10729407-13 from 0 to 98304 tid 1 2020-06-12T02:39:09.124 INFO:tasks.rados.rados.0.smithi107.stdout:10182: delete oid 12 current snap is 0 2020-06-12T02:39:09.124 INFO:tasks.rados.rados.0.smithi107.stdout:10179: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:09.124 INFO:tasks.rados.rados.0.smithi107.stdout:10179: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:09.126 INFO:tasks.rados.rados.0.smithi107.stdout:10181: finishing write tid 1 to smithi10729407-13 2020-06-12T02:39:09.127 INFO:tasks.rados.rados.0.smithi107.stdout:10179: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:09.127 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1892 (ObjNum 10178 snap 0 seq_num 10178) dirty exists 2020-06-12T02:39:09.127 INFO:tasks.rados.rados.0.smithi107.stdout:10179: left oid 8 (ObjNum 10178 snap 0 seq_num 10178) 2020-06-12T02:39:09.128 INFO:tasks.rados.rados.0.smithi107.stdout:10179: done (2 left) 2020-06-12T02:39:09.128 INFO:tasks.rados.rados.0.smithi107.stdout:10182: done (1 left) 2020-06-12T02:39:09.128 INFO:tasks.rados.rados.0.smithi107.stdout:10183: delete oid 8 current snap is 0 2020-06-12T02:39:09.129 INFO:tasks.rados.rados.0.smithi107.stdout:10181: finishing write tid 3 to smithi10729407-13 2020-06-12T02:39:09.130 INFO:tasks.rados.rados.0.smithi107.stdout:10181: finishing write tid 4 to smithi10729407-13 2020-06-12T02:39:09.131 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1719 (ObjNum 10180 snap 0 seq_num 10180) dirty exists 2020-06-12T02:39:09.131 INFO:tasks.rados.rados.0.smithi107.stdout:10181: left oid 13 (ObjNum 10180 snap 0 seq_num 10180) 2020-06-12T02:39:09.131 INFO:tasks.rados.rados.0.smithi107.stdout:10181: done (1 left) 2020-06-12T02:39:09.131 INFO:tasks.rados.rados.0.smithi107.stdout:10183: done (0 left) 2020-06-12T02:39:09.131 INFO:tasks.rados.rados.0.smithi107.stdout:10184: delete oid 19 current snap is 0 2020-06-12T02:39:09.134 INFO:tasks.rados.rados.0.smithi107.stdout:10184: done (0 left) 2020-06-12T02:39:09.134 INFO:tasks.rados.rados.0.smithi107.stdout:10185: delete oid 17 current snap is 0 2020-06-12T02:39:09.136 INFO:tasks.rados.rados.0.smithi107.stdout:10185: done (0 left) 2020-06-12T02:39:09.136 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:39:09.136 INFO:tasks.rados.rados.0.smithi107.stdout:10186: seq_num 10185 ranges {0=180224} 2020-06-12T02:39:09.137 INFO:tasks.rados.rados.0.smithi107.stdout:10186: writing smithi10729407-19 from 0 to 180224 tid 1 2020-06-12T02:39:09.137 INFO:tasks.rados.rados.0.smithi107.stdout:10187: delete oid 3 current snap is 0 2020-06-12T02:39:09.142 INFO:tasks.rados.rados.0.smithi107.stdout:10187: done (1 left) 2020-06-12T02:39:09.142 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:39:09.142 INFO:tasks.rados.rados.0.smithi107.stdout:10188: seq_num 10187 ranges {0=221184} 2020-06-12T02:39:09.144 INFO:tasks.rados.rados.0.smithi107.stdout:10188: writing smithi10729407-3 from 0 to 221184 tid 1 2020-06-12T02:39:09.144 INFO:tasks.rados.rados.0.smithi107.stdout:10189: delete oid 16 current snap is 0 2020-06-12T02:39:09.144 INFO:tasks.rados.rados.0.smithi107.stdout:10186: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:09.145 INFO:tasks.rados.rados.0.smithi107.stdout:10186: finishing write tid 3 to smithi10729407-19 2020-06-12T02:39:09.147 INFO:tasks.rados.rados.0.smithi107.stdout:10186: finishing write tid 4 to smithi10729407-19 2020-06-12T02:39:09.147 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1871 (ObjNum 10185 snap 0 seq_num 10185) dirty exists 2020-06-12T02:39:09.148 INFO:tasks.rados.rados.0.smithi107.stdout:10186: left oid 19 (ObjNum 10185 snap 0 seq_num 10185) 2020-06-12T02:39:09.149 INFO:tasks.rados.rados.0.smithi107.stdout:10188: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:09.153 INFO:tasks.rados.rados.0.smithi107.stdout:10188: finishing write tid 3 to smithi10729407-3 2020-06-12T02:39:09.154 INFO:tasks.rados.rados.0.smithi107.stdout:10188: finishing write tid 4 to smithi10729407-3 2020-06-12T02:39:09.154 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1811 (ObjNum 10187 snap 0 seq_num 10187) dirty exists 2020-06-12T02:39:09.154 INFO:tasks.rados.rados.0.smithi107.stdout:10188: left oid 3 (ObjNum 10187 snap 0 seq_num 10187) 2020-06-12T02:39:09.154 INFO:tasks.rados.rados.0.smithi107.stdout:10186: done (2 left) 2020-06-12T02:39:09.155 INFO:tasks.rados.rados.0.smithi107.stdout:10188: done (1 left) 2020-06-12T02:39:09.155 INFO:tasks.rados.rados.0.smithi107.stdout:10189: done (0 left) 2020-06-12T02:39:09.155 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:39:09.155 INFO:tasks.rados.rados.0.smithi107.stdout:10190: seq_num 10189 ranges {98304=253952} 2020-06-12T02:39:09.156 INFO:tasks.rados.rados.0.smithi107.stdout:10190: writing smithi10729407-13 from 98304 to 352256 tid 1 2020-06-12T02:39:09.156 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:39:09.156 INFO:tasks.rados.rados.0.smithi107.stdout:10191: seq_num 10190 ranges {0=270336} 2020-06-12T02:39:09.157 INFO:tasks.rados.rados.0.smithi107.stdout:10191: writing smithi10729407-20 from 0 to 270336 tid 1 2020-06-12T02:39:09.157 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:39:09.157 INFO:tasks.rados.rados.0.smithi107.stdout:10192: seq_num 10191 ranges {180224=286720} 2020-06-12T02:39:09.161 INFO:tasks.rados.rados.0.smithi107.stdout:10192: writing smithi10729407-19 from 180224 to 466944 tid 1 2020-06-12T02:39:09.161 INFO:tasks.rados.rados.0.smithi107.stdout:10193: delete oid 3 current snap is 0 2020-06-12T02:39:09.161 INFO:tasks.rados.rados.0.smithi107.stdout:10190: finishing write tid 1 to smithi10729407-13 2020-06-12T02:39:09.162 INFO:tasks.rados.rados.0.smithi107.stdout:10190: finishing write tid 3 to smithi10729407-13 2020-06-12T02:39:09.165 INFO:tasks.rados.rados.0.smithi107.stdout:10191: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:09.165 INFO:tasks.rados.rados.0.smithi107.stdout:10192: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:09.165 INFO:tasks.rados.rados.0.smithi107.stdout:10191: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:09.165 INFO:tasks.rados.rados.0.smithi107.stdout:10190: finishing write tid 4 to smithi10729407-13 2020-06-12T02:39:09.166 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1721 (ObjNum 10189 snap 0 seq_num 10189) dirty exists 2020-06-12T02:39:09.166 INFO:tasks.rados.rados.0.smithi107.stdout:10190: left oid 13 (ObjNum 10189 snap 0 seq_num 10189) 2020-06-12T02:39:09.166 INFO:tasks.rados.rados.0.smithi107.stdout:10190: done (3 left) 2020-06-12T02:39:09.166 INFO:tasks.rados.rados.0.smithi107.stdout:10193: done (2 left) 2020-06-12T02:39:09.167 INFO:tasks.rados.rados.0.smithi107.stdout:10194: delete oid 13 current snap is 0 2020-06-12T02:39:09.167 INFO:tasks.rados.rados.0.smithi107.stdout:10191: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:09.168 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 784 (ObjNum 10190 snap 0 seq_num 10190) dirty exists 2020-06-12T02:39:09.168 INFO:tasks.rados.rados.0.smithi107.stdout:10191: left oid 20 (ObjNum 10190 snap 0 seq_num 10190) 2020-06-12T02:39:09.169 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:39:09.166+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:39:09.169 INFO:tasks.rados.rados.0.smithi107.stdout:10192: finishing write tid 3 to smithi10729407-19 2020-06-12T02:39:09.169 INFO:tasks.rados.rados.0.smithi107.stdout:10191: done (2 left) 2020-06-12T02:39:09.170 INFO:tasks.rados.rados.0.smithi107.stdout:10194: done (1 left) 2020-06-12T02:39:09.170 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:39:09.170 INFO:tasks.rados.rados.0.smithi107.stdout:10195: seq_num 10194 ranges {0=335872} 2020-06-12T02:39:09.171 INFO:tasks.rados.rados.0.smithi107.stdout:10195: writing smithi10729407-10 from 0 to 335872 tid 1 2020-06-12T02:39:09.171 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:39:09.171 INFO:tasks.rados.rados.0.smithi107.stdout:10196: seq_num 10195 ranges {0=352256} 2020-06-12T02:39:09.174 INFO:tasks.rados.rados.0.smithi107.stdout:10196: writing smithi10729407-16 from 0 to 352256 tid 1 2020-06-12T02:39:09.174 INFO:tasks.rados.rados.0.smithi107.stdout:10197: delete oid 15 current snap is 0 2020-06-12T02:39:09.174 INFO:tasks.rados.rados.0.smithi107.stdout:10192: finishing write tid 4 to smithi10729407-19 2020-06-12T02:39:09.174 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1873 (ObjNum 10191 snap 0 seq_num 10191) dirty exists 2020-06-12T02:39:09.175 INFO:tasks.rados.rados.0.smithi107.stdout:10192: left oid 19 (ObjNum 10191 snap 0 seq_num 10191) 2020-06-12T02:39:09.177 INFO:tasks.rados.rados.0.smithi107.stdout:10195: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:09.177 INFO:tasks.rados.rados.0.smithi107.stdout:10192: done (3 left) 2020-06-12T02:39:09.177 INFO:tasks.rados.rados.0.smithi107.stdout:10197: done (2 left) 2020-06-12T02:39:09.178 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:39:09.178 INFO:tasks.rados.rados.0.smithi107.stdout:10198: seq_num 10197 ranges {2088960=385024} 2020-06-12T02:39:09.191 INFO:tasks.rados.rados.0.smithi107.stdout:10198: writing smithi10729407-6 from 2088960 to 2473984 tid 1 2020-06-12T02:39:09.191 INFO:tasks.rados.rados.0.smithi107.stdout:10199: delete oid 15 current snap is 0 2020-06-12T02:39:09.192 INFO:tasks.rados.rados.0.smithi107.stdout:10195: finishing write tid 3 to smithi10729407-10 2020-06-12T02:39:09.192 INFO:tasks.rados.rados.0.smithi107.stdout:10196: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:09.192 INFO:tasks.rados.rados.0.smithi107.stdout:10195: finishing write tid 4 to smithi10729407-10 2020-06-12T02:39:09.192 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 727 (ObjNum 10194 snap 0 seq_num 10194) dirty exists 2020-06-12T02:39:09.192 INFO:tasks.rados.rados.0.smithi107.stdout:10195: left oid 10 (ObjNum 10194 snap 0 seq_num 10194) 2020-06-12T02:39:09.193 INFO:tasks.rados.rados.0.smithi107.stdout:10196: finishing write tid 3 to smithi10729407-16 2020-06-12T02:39:09.193 INFO:tasks.rados.rados.0.smithi107.stdout:10196: finishing write tid 4 to smithi10729407-16 2020-06-12T02:39:09.193 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1815 (ObjNum 10195 snap 0 seq_num 10195) dirty exists 2020-06-12T02:39:09.193 INFO:tasks.rados.rados.0.smithi107.stdout:10196: left oid 16 (ObjNum 10195 snap 0 seq_num 10195) 2020-06-12T02:39:09.195 INFO:teuthology.orchestra.run.smithi107.stdout:moved 0 / 96 (0%) 2020-06-12T02:39:09.198 INFO:teuthology.orchestra.run.smithi107.stdout:avg 12 2020-06-12T02:39:09.198 INFO:teuthology.orchestra.run.smithi107.stdout:stddev 2.95804 -> 2.95804 (expected baseline 3.24037) 2020-06-12T02:39:09.199 INFO:teuthology.orchestra.run.smithi107.stdout:min osd.2 with 6 -> 6 pgs (0.5 -> 0.5 * mean) 2020-06-12T02:39:09.199 INFO:teuthology.orchestra.run.smithi107.stdout:max osd.7 with 16 -> 16 pgs (1.33333 -> 1.33333 * mean) 2020-06-12T02:39:09.199 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:39:09.199 INFO:teuthology.orchestra.run.smithi107.stdout:oload 110 2020-06-12T02:39:09.199 INFO:teuthology.orchestra.run.smithi107.stdout:max_change 1 2020-06-12T02:39:09.200 INFO:teuthology.orchestra.run.smithi107.stdout:max_change_osds 4 2020-06-12T02:39:09.200 INFO:teuthology.orchestra.run.smithi107.stdout:average_utilization 0.0112 2020-06-12T02:39:09.200 INFO:teuthology.orchestra.run.smithi107.stdout:overload_utilization 0.0123 2020-06-12T02:39:09.200 INFO:tasks.rados.rados.0.smithi107.stdout:10195: done (3 left) 2020-06-12T02:39:09.200 INFO:tasks.rados.rados.0.smithi107.stdout:10196: done (2 left) 2020-06-12T02:39:09.201 INFO:tasks.rados.rados.0.smithi107.stdout:10199: done (1 left) 2020-06-12T02:39:09.201 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:39:09.201 INFO:tasks.rados.rados.0.smithi107.stdout:10200: seq_num 10199 ranges {0=417792} 2020-06-12T02:39:09.202 INFO:tasks.rados.rados.0.smithi107.stdout:10200: writing smithi10729407-12 from 0 to 417792 tid 1 2020-06-12T02:39:09.202 INFO:tasks.rados.rados.0.smithi107.stdout:10201: delete oid 2 current snap is 0 2020-06-12T02:39:09.202 INFO:tasks.rados.rados.0.smithi107.stdout:10198: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:09.202 INFO:tasks.rados.rados.0.smithi107.stdout:10201: done (2 left) 2020-06-12T02:39:09.203 INFO:tasks.rados.rados.0.smithi107.stdout:10202: delete oid 20 current snap is 0 2020-06-12T02:39:09.203 INFO:tasks.rados.rados.0.smithi107.stdout:10200: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:09.203 INFO:teuthology.orchestra.run.smithi107.stderr:no change 2020-06-12T02:39:09.204 INFO:tasks.rados.rados.0.smithi107.stdout:10198: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:09.204 INFO:tasks.rados.rados.0.smithi107.stdout:10198: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:09.204 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1835 (ObjNum 10197 snap 0 seq_num 10197) dirty exists 2020-06-12T02:39:09.204 INFO:tasks.rados.rados.0.smithi107.stdout:10198: left oid 6 (ObjNum 10197 snap 0 seq_num 10197) 2020-06-12T02:39:09.205 INFO:tasks.rados.rados.0.smithi107.stdout:10198: done (2 left) 2020-06-12T02:39:09.205 INFO:tasks.rados.rados.0.smithi107.stdout:10202: done (1 left) 2020-06-12T02:39:09.205 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:39:09.205 INFO:tasks.rados.rados.0.smithi107.stdout:10203: seq_num 10202 ranges {335872=466944} 2020-06-12T02:39:09.208 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2020-06-12T02:39:09.208 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd test-reweight-by-utilization 1000 1.0 2020-06-12T02:39:09.210 INFO:tasks.rados.rados.0.smithi107.stdout:10203: writing smithi10729407-10 from 335872 to 802816 tid 1 2020-06-12T02:39:09.210 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:39:09.210 INFO:tasks.rados.rados.0.smithi107.stdout:10204: seq_num 10203 ranges {0=483328} 2020-06-12T02:39:09.212 INFO:tasks.rados.rados.0.smithi107.stdout:10204: writing smithi10729407-7 from 0 to 483328 tid 1 2020-06-12T02:39:09.213 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:39:09.213 INFO:tasks.rados.rados.0.smithi107.stdout:10205: seq_num 10204 ranges {0=499712} 2020-06-12T02:39:09.216 INFO:tasks.rados.rados.0.smithi107.stdout:10205: writing smithi10729407-17 from 0 to 499712 tid 1 2020-06-12T02:39:09.216 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:39:09.216 INFO:tasks.rados.rados.0.smithi107.stdout:10206: seq_num 10205 ranges {3874816=516096} 2020-06-12T02:39:09.226 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-06-12T02:39:09.241 INFO:tasks.rados.rados.0.smithi107.stdout:10206: writing smithi10729407-18 from 3874816 to 4390912 tid 1 2020-06-12T02:39:09.241 INFO:tasks.rados.rados.0.smithi107.stdout:10207: delete oid 1 current snap is 0 2020-06-12T02:39:09.241 INFO:tasks.rados.rados.0.smithi107.stdout:10200: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:09.242 INFO:tasks.rados.rados.0.smithi107.stdout:10200: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:09.242 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1875 (ObjNum 10199 snap 0 seq_num 10199) dirty exists 2020-06-12T02:39:09.242 INFO:tasks.rados.rados.0.smithi107.stdout:10200: left oid 12 (ObjNum 10199 snap 0 seq_num 10199) 2020-06-12T02:39:09.242 INFO:tasks.rados.rados.0.smithi107.stdout:10203: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:09.242 INFO:tasks.rados.rados.0.smithi107.stdout:10203: finishing write tid 3 to smithi10729407-10 2020-06-12T02:39:09.243 INFO:tasks.rados.rados.0.smithi107.stdout:10203: finishing write tid 4 to smithi10729407-10 2020-06-12T02:39:09.243 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 729 (ObjNum 10202 snap 0 seq_num 10202) dirty exists 2020-06-12T02:39:09.243 INFO:tasks.rados.rados.0.smithi107.stdout:10203: left oid 10 (ObjNum 10202 snap 0 seq_num 10202) 2020-06-12T02:39:09.243 INFO:tasks.rados.rados.0.smithi107.stdout:10204: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:09.243 INFO:tasks.rados.rados.0.smithi107.stdout:10205: finishing write tid 1 to smithi10729407-17 2020-06-12T02:39:09.244 INFO:tasks.rados.rados.0.smithi107.stdout:10204: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:09.244 INFO:tasks.rados.rados.0.smithi107.stdout:10205: finishing write tid 3 to smithi10729407-17 2020-06-12T02:39:09.244 INFO:tasks.rados.rados.0.smithi107.stdout:10204: finishing write tid 4 to smithi10729407-7 2020-06-12T02:39:09.244 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1710 (ObjNum 10203 snap 0 seq_num 10203) dirty exists 2020-06-12T02:39:09.244 INFO:tasks.rados.rados.0.smithi107.stdout:10204: left oid 7 (ObjNum 10203 snap 0 seq_num 10203) 2020-06-12T02:39:09.244 INFO:tasks.rados.rados.0.smithi107.stdout:10205: finishing write tid 4 to smithi10729407-17 2020-06-12T02:39:09.245 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1910 (ObjNum 10204 snap 0 seq_num 10204) dirty exists 2020-06-12T02:39:09.245 INFO:tasks.rados.rados.0.smithi107.stdout:10205: left oid 17 (ObjNum 10204 snap 0 seq_num 10204) 2020-06-12T02:39:09.245 INFO:tasks.rados.rados.0.smithi107.stdout:10200: done (5 left) 2020-06-12T02:39:09.246 INFO:tasks.rados.rados.0.smithi107.stdout:10203: done (4 left) 2020-06-12T02:39:09.246 INFO:tasks.rados.rados.0.smithi107.stdout:10204: done (3 left) 2020-06-12T02:39:09.246 INFO:tasks.rados.rados.0.smithi107.stdout:10205: done (2 left) 2020-06-12T02:39:09.246 INFO:tasks.rados.rados.0.smithi107.stdout:10207: done (1 left) 2020-06-12T02:39:09.246 INFO:tasks.rados.rados.0.smithi107.stdout:10208: delete oid 17 current snap is 0 2020-06-12T02:39:09.247 INFO:tasks.rados.rados.0.smithi107.stdout:10208: done (1 left) 2020-06-12T02:39:09.247 INFO:tasks.rados.rados.0.smithi107.stdout:10209: delete oid 17 current snap is 0 2020-06-12T02:39:09.247 INFO:tasks.rados.rados.0.smithi107.stdout:10206: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:09.250 INFO:tasks.rados.rados.0.smithi107.stdout:10209: done (1 left) 2020-06-12T02:39:09.250 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:39:09.250 INFO:tasks.rados.rados.0.smithi107.stdout:10210: seq_num 10209 ranges {483328=589824} 2020-06-12T02:39:09.256 INFO:tasks.rados.rados.0.smithi107.stdout:10210: writing smithi10729407-7 from 483328 to 1073152 tid 1 2020-06-12T02:39:09.256 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:39:09.257 INFO:tasks.rados.rados.0.smithi107.stdout:10211: seq_num 10210 ranges {0=606208} 2020-06-12T02:39:09.260 INFO:tasks.rados.rados.0.smithi107.stdout:10211: writing smithi10729407-17 from 0 to 606208 tid 1 2020-06-12T02:39:09.260 INFO:tasks.rados.rados.0.smithi107.stdout:10212: delete oid 6 current snap is 0 2020-06-12T02:39:09.260 INFO:tasks.rados.rados.0.smithi107.stdout:10206: finishing write tid 3 to smithi10729407-18 2020-06-12T02:39:09.260 INFO:tasks.rados.rados.0.smithi107.stdout:10206: finishing write tid 4 to smithi10729407-18 2020-06-12T02:39:09.260 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1815 (ObjNum 10205 snap 0 seq_num 10205) dirty exists 2020-06-12T02:39:09.261 INFO:tasks.rados.rados.0.smithi107.stdout:10206: left oid 18 (ObjNum 10205 snap 0 seq_num 10205) 2020-06-12T02:39:09.261 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:09.261 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:09.270 INFO:tasks.rados.rados.0.smithi107.stdout:10206: done (3 left) 2020-06-12T02:39:09.270 INFO:tasks.rados.rados.0.smithi107.stdout:10212: done (2 left) 2020-06-12T02:39:09.270 INFO:tasks.rados.rados.0.smithi107.stdout:10213: delete oid 14 current snap is 0 2020-06-12T02:39:09.271 INFO:tasks.rados.rados.0.smithi107.stdout:10210: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:09.271 INFO:tasks.rados.rados.0.smithi107.stdout:10210: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:09.271 INFO:tasks.rados.rados.0.smithi107.stdout:10211: finishing write tid 1 to smithi10729407-17 2020-06-12T02:39:09.271 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:39:09.268+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:39:09.272 INFO:tasks.rados.rados.0.smithi107.stdout:10210: finishing write tid 4 to smithi10729407-7 2020-06-12T02:39:09.272 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1712 (ObjNum 10209 snap 0 seq_num 10209) dirty exists 2020-06-12T02:39:09.273 INFO:tasks.rados.rados.0.smithi107.stdout:10210: left oid 7 (ObjNum 10209 snap 0 seq_num 10209) 2020-06-12T02:39:09.273 INFO:tasks.rados.rados.0.smithi107.stdout:10210: done (2 left) 2020-06-12T02:39:09.273 INFO:tasks.rados.rados.0.smithi107.stdout:10213: done (1 left) 2020-06-12T02:39:09.273 INFO:tasks.rados.rados.0.smithi107.stdout:10214: delete oid 19 current snap is 0 2020-06-12T02:39:09.273 INFO:tasks.rados.rados.0.smithi107.stdout:10211: finishing write tid 3 to smithi10729407-17 2020-06-12T02:39:09.275 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-06-12T02:39:09.276 INFO:tasks.rados.rados.0.smithi107.stdout:10211: finishing write tid 4 to smithi10729407-17 2020-06-12T02:39:09.276 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1914 (ObjNum 10210 snap 0 seq_num 10210) dirty exists 2020-06-12T02:39:09.276 INFO:tasks.rados.rados.0.smithi107.stdout:10211: left oid 17 (ObjNum 10210 snap 0 seq_num 10210) 2020-06-12T02:39:09.276 INFO:tasks.rados.rados.0.smithi107.stdout:10211: done (1 left) 2020-06-12T02:39:09.276 INFO:tasks.rados.rados.0.smithi107.stdout:10214: done (0 left) 2020-06-12T02:39:09.277 INFO:tasks.rados.rados.0.smithi107.stdout:10215: delete oid 19 current snap is 0 2020-06-12T02:39:09.279 INFO:tasks.rados.rados.0.smithi107.stdout:10215: done (0 left) 2020-06-12T02:39:09.279 INFO:tasks.rados.rados.0.smithi107.stdout:10216: delete oid 16 current snap is 0 2020-06-12T02:39:09.281 INFO:tasks.rados.rados.0.smithi107.stdout:10216: done (0 left) 2020-06-12T02:39:09.281 INFO:tasks.rados.rados.0.smithi107.stdout:10217: delete oid 2 current snap is 0 2020-06-12T02:39:09.285 INFO:tasks.rados.rados.0.smithi107.stdout:10217: done (0 left) 2020-06-12T02:39:09.285 INFO:tasks.rados.rados.0.smithi107.stdout:10218: delete oid 17 current snap is 0 2020-06-12T02:39:09.286 INFO:tasks.rados.rados.0.smithi107.stdout:10218: done (0 left) 2020-06-12T02:39:09.286 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:39:09.286 INFO:tasks.rados.rados.0.smithi107.stdout:10219: seq_num 10218 ranges {0=737280} 2020-06-12T02:39:09.290 INFO:tasks.rados.rados.0.smithi107.stdout:10219: writing smithi10729407-14 from 0 to 737280 tid 1 2020-06-12T02:39:09.290 INFO:tasks.rados.rados.0.smithi107.stdout:10220: delete oid 20 current snap is 0 2020-06-12T02:39:09.292 INFO:tasks.rados.rados.0.smithi107.stdout:10220: done (1 left) 2020-06-12T02:39:09.292 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:39:09.293 INFO:tasks.rados.rados.0.smithi107.stdout:10221: seq_num 10220 ranges {0=770048} 2020-06-12T02:39:09.297 INFO:tasks.rados.rados.0.smithi107.stdout:10221: writing smithi10729407-4 from 0 to 770048 tid 1 2020-06-12T02:39:09.297 INFO:tasks.rados.rados.0.smithi107.stdout:10222: delete oid 13 current snap is 0 2020-06-12T02:39:09.298 INFO:tasks.rados.rados.0.smithi107.stdout:10219: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:09.298 INFO:tasks.rados.rados.0.smithi107.stdout:10219: finishing write tid 3 to smithi10729407-14 2020-06-12T02:39:09.299 INFO:tasks.rados.rados.0.smithi107.stdout:10219: finishing write tid 4 to smithi10729407-14 2020-06-12T02:39:09.299 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1659 (ObjNum 10218 snap 0 seq_num 10218) dirty exists 2020-06-12T02:39:09.299 INFO:tasks.rados.rados.0.smithi107.stdout:10219: left oid 14 (ObjNum 10218 snap 0 seq_num 10218) 2020-06-12T02:39:09.300 INFO:tasks.rados.rados.0.smithi107.stdout:10219: done (2 left) 2020-06-12T02:39:09.300 INFO:tasks.rados.rados.0.smithi107.stdout:10222: done (1 left) 2020-06-12T02:39:09.300 INFO:tasks.rados.rados.0.smithi107.stdout:10223: delete oid 6 current snap is 0 2020-06-12T02:39:09.303 INFO:tasks.rados.rados.0.smithi107.stdout:10223: done (1 left) 2020-06-12T02:39:09.304 INFO:tasks.rados.rados.0.smithi107.stdout:10224: delete oid 20 current snap is 0 2020-06-12T02:39:09.305 INFO:tasks.rados.rados.0.smithi107.stdout:10221: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:09.306 INFO:tasks.rados.rados.0.smithi107.stdout:10221: finishing write tid 3 to smithi10729407-4 2020-06-12T02:39:09.308 INFO:tasks.rados.rados.0.smithi107.stdout:10224: done (1 left) 2020-06-12T02:39:09.308 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:39:09.309 INFO:tasks.rados.rados.0.smithi107.stdout:10225: seq_num 10224 ranges {0=434176,434176=401408} 2020-06-12T02:39:09.310 INFO:tasks.rados.rados.0.smithi107.stdout:10225: writing smithi10729407-16 from 0 to 434176 tid 1 2020-06-12T02:39:09.313 INFO:tasks.rados.rados.0.smithi107.stdout:10225: writing smithi10729407-16 from 434176 to 835584 tid 2 2020-06-12T02:39:09.313 INFO:tasks.rados.rados.0.smithi107.stdout:10226: delete oid 10 current snap is 0 2020-06-12T02:39:09.314 INFO:tasks.rados.rados.0.smithi107.stdout:10221: finishing write tid 4 to smithi10729407-4 2020-06-12T02:39:09.314 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1917 (ObjNum 10220 snap 0 seq_num 10220) dirty exists 2020-06-12T02:39:09.314 INFO:tasks.rados.rados.0.smithi107.stdout:10221: left oid 4 (ObjNum 10220 snap 0 seq_num 10220) 2020-06-12T02:39:09.319 INFO:tasks.rados.rados.0.smithi107.stdout:10221: done (2 left) 2020-06-12T02:39:09.319 INFO:tasks.rados.rados.0.smithi107.stdout:10226: done (1 left) 2020-06-12T02:39:09.320 INFO:tasks.rados.rados.0.smithi107.stdout:10227: delete oid 5 current snap is 0 2020-06-12T02:39:09.320 INFO:tasks.rados.rados.0.smithi107.stdout:10225: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:09.321 INFO:tasks.rados.rados.0.smithi107.stdout:10225: finishing write tid 2 to smithi10729407-16 2020-06-12T02:39:09.321 INFO:tasks.rados.rados.0.smithi107.stdout:10225: finishing write tid 4 to smithi10729407-16 2020-06-12T02:39:09.323 INFO:tasks.rados.rados.0.smithi107.stdout:10227: done (1 left) 2020-06-12T02:39:09.323 INFO:tasks.rados.rados.0.smithi107.stdout:10228: delete oid 1 current snap is 0 2020-06-12T02:39:09.324 INFO:tasks.rados.rados.0.smithi107.stdout:10225: finishing write tid 5 to smithi10729407-16 2020-06-12T02:39:09.324 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1819 (ObjNum 10224 snap 0 seq_num 10224) dirty exists 2020-06-12T02:39:09.324 INFO:tasks.rados.rados.0.smithi107.stdout:10225: left oid 16 (ObjNum 10224 snap 0 seq_num 10224) 2020-06-12T02:39:09.325 INFO:tasks.rados.rados.0.smithi107.stdout:10225: done (1 left) 2020-06-12T02:39:09.325 INFO:tasks.rados.rados.0.smithi107.stdout:10228: done (0 left) 2020-06-12T02:39:09.325 INFO:tasks.rados.rados.0.smithi107.stdout:10229: delete oid 5 current snap is 0 2020-06-12T02:39:09.327 INFO:tasks.rados.rados.0.smithi107.stdout:10229: done (0 left) 2020-06-12T02:39:09.327 INFO:tasks.rados.rados.0.smithi107.stdout:10230: delete oid 2 current snap is 0 2020-06-12T02:39:09.329 INFO:tasks.rados.rados.0.smithi107.stdout:10230: done (0 left) 2020-06-12T02:39:09.329 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:39:09.329 INFO:tasks.rados.rados.0.smithi107.stdout:10231: seq_num 10230 ranges {0=540672,540672=401408} 2020-06-12T02:39:09.332 INFO:tasks.rados.rados.0.smithi107.stdout:10231: writing smithi10729407-19 from 0 to 540672 tid 1 2020-06-12T02:39:09.334 INFO:tasks.rados.rados.0.smithi107.stdout:10231: writing smithi10729407-19 from 540672 to 942080 tid 2 2020-06-12T02:39:09.334 INFO:tasks.rados.rados.0.smithi107.stdout:10232: delete oid 14 current snap is 0 2020-06-12T02:39:09.337 INFO:tasks.rados.rados.0.smithi107.stdout:10232: done (1 left) 2020-06-12T02:39:09.337 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:39:09.338 INFO:tasks.rados.rados.0.smithi107.stdout:10233: seq_num 10232 ranges {0=573440,573440=401408} 2020-06-12T02:39:09.341 INFO:tasks.rados.rados.0.smithi107.stdout:10233: writing smithi10729407-14 from 0 to 573440 tid 1 2020-06-12T02:39:09.343 INFO:tasks.rados.rados.0.smithi107.stdout:10233: writing smithi10729407-14 from 573440 to 974848 tid 2 2020-06-12T02:39:09.343 INFO:tasks.rados.rados.0.smithi107.stdout:10234: delete oid 8 current snap is 0 2020-06-12T02:39:09.344 INFO:tasks.rados.rados.0.smithi107.stdout:10231: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:09.344 INFO:tasks.rados.rados.0.smithi107.stdout:10231: finishing write tid 2 to smithi10729407-19 2020-06-12T02:39:09.344 INFO:tasks.rados.rados.0.smithi107.stdout:10231: finishing write tid 4 to smithi10729407-19 2020-06-12T02:39:09.348 INFO:tasks.rados.rados.0.smithi107.stdout:10234: done (2 left) 2020-06-12T02:39:09.349 INFO:tasks.rados.rados.0.smithi107.stdout:10235: delete oid 16 current snap is 0 2020-06-12T02:39:09.349 INFO:tasks.rados.rados.0.smithi107.stdout:10231: finishing write tid 5 to smithi10729407-19 2020-06-12T02:39:09.349 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1880 (ObjNum 10230 snap 0 seq_num 10230) dirty exists 2020-06-12T02:39:09.350 INFO:tasks.rados.rados.0.smithi107.stdout:10231: left oid 19 (ObjNum 10230 snap 0 seq_num 10230) 2020-06-12T02:39:09.350 INFO:tasks.rados.rados.0.smithi107.stdout:10233: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:09.351 INFO:tasks.rados.rados.0.smithi107.stdout:10231: done (2 left) 2020-06-12T02:39:09.351 INFO:tasks.rados.rados.0.smithi107.stdout:10235: done (1 left) 2020-06-12T02:39:09.352 INFO:tasks.rados.rados.0.smithi107.stdout:10236: delete oid 15 current snap is 0 2020-06-12T02:39:09.355 INFO:tasks.rados.rados.0.smithi107.stdout:10236: done (1 left) 2020-06-12T02:39:09.355 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:39:09.355 INFO:tasks.rados.rados.0.smithi107.stdout:10237: seq_num 10236 ranges {0=638976,638976=401408} 2020-06-12T02:39:09.360 INFO:tasks.rados.rados.0.smithi107.stdout:10237: writing smithi10729407-3 from 0 to 638976 tid 1 2020-06-12T02:39:09.363 INFO:tasks.rados.rados.0.smithi107.stdout:10237: writing smithi10729407-3 from 638976 to 1040384 tid 2 2020-06-12T02:39:09.364 INFO:tasks.rados.rados.0.smithi107.stdout:10238: delete oid 1 current snap is 0 2020-06-12T02:39:09.364 INFO:tasks.rados.rados.0.smithi107.stdout:10233: finishing write tid 2 to smithi10729407-14 2020-06-12T02:39:09.364 INFO:tasks.rados.rados.0.smithi107.stdout:10233: finishing write tid 4 to smithi10729407-14 2020-06-12T02:39:09.364 INFO:tasks.rados.rados.0.smithi107.stdout:10233: finishing write tid 5 to smithi10729407-14 2020-06-12T02:39:09.365 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1663 (ObjNum 10232 snap 0 seq_num 10232) dirty exists 2020-06-12T02:39:09.365 INFO:tasks.rados.rados.0.smithi107.stdout:10233: left oid 14 (ObjNum 10232 snap 0 seq_num 10232) 2020-06-12T02:39:09.366 INFO:tasks.rados.rados.0.smithi107.stdout:10233: done (2 left) 2020-06-12T02:39:09.367 INFO:tasks.rados.rados.0.smithi107.stdout:10238: done (1 left) 2020-06-12T02:39:09.367 INFO:tasks.rados.rados.0.smithi107.stdout:10239: delete oid 5 current snap is 0 2020-06-12T02:39:09.367 INFO:tasks.rados.rados.0.smithi107.stdout:10237: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:09.370 INFO:tasks.rados.rados.0.smithi107.stdout:10239: done (1 left) 2020-06-12T02:39:09.370 INFO:tasks.rados.rados.0.smithi107.stdout:10240: delete oid 7 current snap is 0 2020-06-12T02:39:09.372 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:39:09.370+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:39:09.372 INFO:tasks.rados.rados.0.smithi107.stdout:10240: done (1 left) 2020-06-12T02:39:09.373 INFO:tasks.rados.rados.0.smithi107.stdout:10241: delete oid 14 current snap is 0 2020-06-12T02:39:09.374 INFO:tasks.rados.rados.0.smithi107.stdout:10237: finishing write tid 2 to smithi10729407-3 2020-06-12T02:39:09.375 INFO:tasks.rados.rados.0.smithi107.stdout:10237: finishing write tid 4 to smithi10729407-3 2020-06-12T02:39:09.377 INFO:tasks.rados.rados.0.smithi107.stdout:10237: finishing write tid 5 to smithi10729407-3 2020-06-12T02:39:09.378 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1823 (ObjNum 10236 snap 0 seq_num 10236) dirty exists 2020-06-12T02:39:09.378 INFO:tasks.rados.rados.0.smithi107.stdout:10237: left oid 3 (ObjNum 10236 snap 0 seq_num 10236) 2020-06-12T02:39:09.378 INFO:tasks.rados.rados.0.smithi107.stdout:10237: done (1 left) 2020-06-12T02:39:09.378 INFO:tasks.rados.rados.0.smithi107.stdout:10241: done (0 left) 2020-06-12T02:39:09.378 INFO:tasks.rados.rados.0.smithi107.stdout:10242: delete oid 11 current snap is 0 2020-06-12T02:39:09.380 INFO:tasks.rados.rados.0.smithi107.stdout:10242: done (0 left) 2020-06-12T02:39:09.381 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:39:09.381 INFO:tasks.rados.rados.0.smithi107.stdout:10243: seq_num 10242 ranges {1040384=737280,1777664=401408} 2020-06-12T02:39:09.390 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2020-06-12T02:39:09.390 INFO:tasks.rados.rados.0.smithi107.stdout:10243: writing smithi10729407-3 from 1040384 to 1777664 tid 1 2020-06-12T02:39:09.394 INFO:tasks.rados.rados.0.smithi107.stdout:10243: writing smithi10729407-3 from 1777664 to 2179072 tid 2 2020-06-12T02:39:09.394 INFO:tasks.rados.rados.0.smithi107.stdout:10244: delete oid 1 current snap is 0 2020-06-12T02:39:09.397 INFO:tasks.rados.rados.0.smithi107.stdout:10244: done (1 left) 2020-06-12T02:39:09.397 INFO:tasks.rados.rados.0.smithi107.stdout:10245: delete oid 9 current snap is 0 2020-06-12T02:39:09.409 INFO:tasks.rados.rados.0.smithi107.stdout:10245: done (1 left) 2020-06-12T02:39:09.409 INFO:tasks.rados.rados.0.smithi107.stdout:10246: delete oid 5 current snap is 0 2020-06-12T02:39:09.411 INFO:tasks.rados.rados.0.smithi107.stdout:10243: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:09.412 INFO:tasks.rados.rados.0.smithi107.stdout:10243: finishing write tid 2 to smithi10729407-3 2020-06-12T02:39:09.412 INFO:tasks.rados.rados.0.smithi107.stdout:10243: finishing write tid 4 to smithi10729407-3 2020-06-12T02:39:09.412 INFO:tasks.rados.rados.0.smithi107.stdout:10246: done (1 left) 2020-06-12T02:39:09.412 INFO:tasks.rados.rados.0.smithi107.stdout:10247: delete oid 1 current snap is 0 2020-06-12T02:39:09.414 INFO:tasks.rados.rados.0.smithi107.stdout:10243: finishing write tid 5 to smithi10729407-3 2020-06-12T02:39:09.414 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1826 (ObjNum 10242 snap 0 seq_num 10242) dirty exists 2020-06-12T02:39:09.414 INFO:tasks.rados.rados.0.smithi107.stdout:10243: left oid 3 (ObjNum 10242 snap 0 seq_num 10242) 2020-06-12T02:39:09.416 INFO:tasks.rados.rados.0.smithi107.stdout:10243: done (1 left) 2020-06-12T02:39:09.416 INFO:tasks.rados.rados.0.smithi107.stdout:10247: done (0 left) 2020-06-12T02:39:09.416 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:39:09.416 INFO:tasks.rados.rados.0.smithi107.stdout:10248: seq_num 10247 ranges {0=425984,425984=475136,901120=327680} 2020-06-12T02:39:09.417 INFO:tasks.rados.rados.0.smithi107.stdout:10248: writing smithi10729407-2 from 0 to 425984 tid 1 2020-06-12T02:39:09.420 INFO:tasks.rados.rados.0.smithi107.stdout:10248: writing smithi10729407-2 from 425984 to 901120 tid 2 2020-06-12T02:39:09.422 INFO:tasks.rados.rados.0.smithi107.stdout:10248: writing smithi10729407-2 from 901120 to 1228800 tid 3 2020-06-12T02:39:09.423 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:39:09.423 INFO:tasks.rados.rados.0.smithi107.stdout:10249: seq_num 10248 ranges {0=442368,442368=761856,1204224=40960} 2020-06-12T02:39:09.423 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:09.424 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:09.425 INFO:tasks.rados.rados.0.smithi107.stdout:10249: writing smithi10729407-1 from 0 to 442368 tid 1 2020-06-12T02:39:09.431 INFO:tasks.rados.rados.0.smithi107.stdout:10249: writing smithi10729407-1 from 442368 to 1204224 tid 2 2020-06-12T02:39:09.431 INFO:tasks.rados.rados.0.smithi107.stdout:10249: writing smithi10729407-1 from 1204224 to 1245184 tid 3 2020-06-12T02:39:09.432 INFO:tasks.rados.rados.0.smithi107.stdout:10250: delete oid 17 current snap is 0 2020-06-12T02:39:09.432 INFO:tasks.rados.rados.0.smithi107.stdout:10248: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:09.434 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:39:09.435 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-06-12T02:39:09.437 INFO:tasks.rados.rados.0.smithi107.stdout:10248: finishing write tid 2 to smithi10729407-2 2020-06-12T02:39:09.437 INFO:tasks.rados.rados.0.smithi107.stdout:10248: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:09.437 INFO:tasks.rados.rados.0.smithi107.stdout:10248: finishing write tid 5 to smithi10729407-2 2020-06-12T02:39:09.438 INFO:tasks.rados.rados.0.smithi107.stdout:10250: done (2 left) 2020-06-12T02:39:09.438 INFO:tasks.rados.rados.0.smithi107.stdout:10251: delete oid 12 current snap is 0 2020-06-12T02:39:09.439 INFO:tasks.rados.rados.0.smithi107.stdout:10248: finishing write tid 6 to smithi10729407-2 2020-06-12T02:39:09.439 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 961 (ObjNum 10247 snap 0 seq_num 10247) dirty exists 2020-06-12T02:39:09.440 INFO:tasks.rados.rados.0.smithi107.stdout:10248: left oid 2 (ObjNum 10247 snap 0 seq_num 10247) 2020-06-12T02:39:09.441 INFO:tasks.rados.rados.0.smithi107.stdout:10248: done (2 left) 2020-06-12T02:39:09.441 INFO:tasks.rados.rados.0.smithi107.stdout:10251: done (1 left) 2020-06-12T02:39:09.441 INFO:tasks.rados.rados.0.smithi107.stdout:10252: delete oid 13 current snap is 0 2020-06-12T02:39:09.443 INFO:tasks.rados.rados.0.smithi107.stdout:10252: done (1 left) 2020-06-12T02:39:09.443 INFO:tasks.rados.rados.0.smithi107.stdout:10253: delete oid 2 current snap is 0 2020-06-12T02:39:09.444 INFO:tasks.rados.rados.0.smithi107.stdout:10249: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:09.444 INFO:tasks.rados.rados.0.smithi107.stdout:10249: finishing write tid 2 to smithi10729407-1 2020-06-12T02:39:09.444 INFO:tasks.rados.rados.0.smithi107.stdout:10249: finishing write tid 3 to smithi10729407-1 2020-06-12T02:39:09.446 INFO:tasks.rados.rados.0.smithi107.stdout:10249: finishing write tid 5 to smithi10729407-1 2020-06-12T02:39:09.448 INFO:tasks.rados.rados.0.smithi107.stdout:10253: done (1 left) 2020-06-12T02:39:09.448 INFO:tasks.rados.rados.0.smithi107.stdout:10254: delete oid 16 current snap is 0 2020-06-12T02:39:09.449 INFO:tasks.rados.rados.0.smithi107.stdout:10249: finishing write tid 6 to smithi10729407-1 2020-06-12T02:39:09.449 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1846 (ObjNum 10248 snap 0 seq_num 10248) dirty exists 2020-06-12T02:39:09.449 INFO:tasks.rados.rados.0.smithi107.stdout:10249: left oid 1 (ObjNum 10248 snap 0 seq_num 10248) 2020-06-12T02:39:09.451 INFO:tasks.rados.rados.0.smithi107.stdout:10249: done (1 left) 2020-06-12T02:39:09.452 INFO:tasks.rados.rados.0.smithi107.stdout:10254: done (0 left) 2020-06-12T02:39:09.452 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:39:09.452 INFO:tasks.rados.rados.0.smithi107.stdout:10255: seq_num 10254 ranges {0=540672,540672=516096,1056768=286720} 2020-06-12T02:39:09.454 INFO:tasks.rados.rados.0.smithi107.stdout:10255: writing smithi10729407-7 from 0 to 540672 tid 1 2020-06-12T02:39:09.458 INFO:tasks.rados.rados.0.smithi107.stdout:10255: writing smithi10729407-7 from 540672 to 1056768 tid 2 2020-06-12T02:39:09.467 INFO:tasks.rados.rados.0.smithi107.stdout:10255: writing smithi10729407-7 from 1056768 to 1343488 tid 3 2020-06-12T02:39:09.470 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:39:09.471 INFO:tasks.rados.rados.0.smithi107.stdout:10256: seq_num 10255 ranges {0=557056,557056=802816} 2020-06-12T02:39:09.471 INFO:tasks.rados.rados.0.smithi107.stdout:10256: writing smithi10729407-13 from 0 to 557056 tid 1 2020-06-12T02:39:09.471 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:39:09.472 INFO:tasks.rados.rados.0.smithi107.stdout:10256: writing smithi10729407-13 from 557056 to 1359872 tid 2 2020-06-12T02:39:09.472 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:39:09.472 INFO:tasks.rados.rados.0.smithi107.stdout:10257: seq_num 10256 ranges {0=573440,573440=729088,1302528=73728} 2020-06-12T02:39:09.473 INFO:tasks.rados.rados.0.smithi107.stdout:10257: writing smithi10729407-14 from 0 to 573440 tid 1 2020-06-12T02:39:09.476 INFO:tasks.rados.rados.0.smithi107.stdout:10257: writing smithi10729407-14 from 573440 to 1302528 tid 2 2020-06-12T02:39:09.477 INFO:tasks.rados.rados.0.smithi107.stdout:10257: writing smithi10729407-14 from 1302528 to 1376256 tid 3 2020-06-12T02:39:09.477 INFO:tasks.rados.rados.0.smithi107.stdout:10258: delete oid 2 current snap is 0 2020-06-12T02:39:09.478 INFO:tasks.rados.rados.0.smithi107.stdout:10255: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:09.478 INFO:tasks.rados.rados.0.smithi107.stdout:10255: finishing write tid 2 to smithi10729407-7 2020-06-12T02:39:09.478 INFO:tasks.rados.rados.0.smithi107.stdout:10256: finishing write tid 1 to smithi10729407-13 2020-06-12T02:39:09.479 INFO:tasks.rados.rados.0.smithi107.stdout:10255: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:09.479 INFO:tasks.rados.rados.0.smithi107.stdout:10255: finishing write tid 5 to smithi10729407-7 2020-06-12T02:39:09.480 INFO:tasks.rados.rados.0.smithi107.stdout:10258: done (3 left) 2020-06-12T02:39:09.481 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:39:09.481 INFO:tasks.rados.rados.0.smithi107.stdout:10259: seq_num 10258 ranges {0=606208,606208=499712,1105920=303104} 2020-06-12T02:39:09.483 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:39:09.481+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:39:09.484 INFO:tasks.rados.rados.0.smithi107.stdout:10259: writing smithi10729407-6 from 0 to 606208 tid 1 2020-06-12T02:39:09.487 INFO:tasks.rados.rados.0.smithi107.stdout:10259: writing smithi10729407-6 from 606208 to 1105920 tid 2 2020-06-12T02:39:09.490 INFO:tasks.rados.rados.0.smithi107.stdout:10259: writing smithi10729407-6 from 1105920 to 1409024 tid 3 2020-06-12T02:39:09.490 INFO:tasks.rados.rados.0.smithi107.stdout:10260: delete oid 20 current snap is 0 2020-06-12T02:39:09.490 INFO:tasks.rados.rados.0.smithi107.stdout:10255: finishing write tid 6 to smithi10729407-7 2020-06-12T02:39:09.490 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1717 (ObjNum 10254 snap 0 seq_num 10254) dirty exists 2020-06-12T02:39:09.490 INFO:tasks.rados.rados.0.smithi107.stdout:10255: left oid 7 (ObjNum 10254 snap 0 seq_num 10254) 2020-06-12T02:39:09.491 INFO:tasks.rados.rados.0.smithi107.stdout:10256: finishing write tid 2 to smithi10729407-13 2020-06-12T02:39:09.491 INFO:tasks.rados.rados.0.smithi107.stdout:10256: finishing write tid 4 to smithi10729407-13 2020-06-12T02:39:09.491 INFO:tasks.rados.rados.0.smithi107.stdout:10256: finishing write tid 5 to smithi10729407-13 2020-06-12T02:39:09.492 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1727 (ObjNum 10255 snap 0 seq_num 10255) dirty exists 2020-06-12T02:39:09.492 INFO:tasks.rados.rados.0.smithi107.stdout:10256: left oid 13 (ObjNum 10255 snap 0 seq_num 10255) 2020-06-12T02:39:09.492 INFO:tasks.rados.rados.0.smithi107.stdout:10257: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:09.492 INFO:tasks.rados.rados.0.smithi107.stdout:10257: finishing write tid 2 to smithi10729407-14 2020-06-12T02:39:09.492 INFO:tasks.rados.rados.0.smithi107.stdout:10257: finishing write tid 3 to smithi10729407-14 2020-06-12T02:39:09.493 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:39:09.493 INFO:tasks.rados.rados.0.smithi107.stdout:10255: done (4 left) 2020-06-12T02:39:09.494 INFO:tasks.rados.rados.0.smithi107.stdout:10256: done (3 left) 2020-06-12T02:39:09.494 INFO:tasks.rados.rados.0.smithi107.stdout:10260: done (2 left) 2020-06-12T02:39:09.494 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:39:09.494 INFO:tasks.rados.rados.0.smithi107.stdout:10261: seq_num 10260 ranges {0=638976,638976=671744,1310720=131072} 2020-06-12T02:39:09.496 INFO:tasks.rados.rados.0.smithi107.stdout:10261: writing smithi10729407-5 from 0 to 638976 tid 1 2020-06-12T02:39:09.500 INFO:tasks.rados.rados.0.smithi107.stdout:10261: writing smithi10729407-5 from 638976 to 1310720 tid 2 2020-06-12T02:39:09.501 INFO:tasks.rados.rados.0.smithi107.stdout:10261: writing smithi10729407-5 from 1310720 to 1441792 tid 3 2020-06-12T02:39:09.501 INFO:tasks.rados.rados.0.smithi107.stdout:10262: delete oid 1 current snap is 0 2020-06-12T02:39:09.502 INFO:tasks.rados.rados.0.smithi107.stdout:10257: finishing write tid 5 to smithi10729407-14 2020-06-12T02:39:09.502 INFO:tasks.rados.rados.0.smithi107.stdout:10259: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:09.502 INFO:tasks.rados.rados.0.smithi107.stdout:10257: finishing write tid 6 to smithi10729407-14 2020-06-12T02:39:09.502 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1668 (ObjNum 10256 snap 0 seq_num 10256) dirty exists 2020-06-12T02:39:09.502 INFO:tasks.rados.rados.0.smithi107.stdout:10257: left oid 14 (ObjNum 10256 snap 0 seq_num 10256) 2020-06-12T02:39:09.503 INFO:tasks.rados.rados.0.smithi107.stdout:10259: finishing write tid 2 to smithi10729407-6 2020-06-12T02:39:09.506 INFO:tasks.rados.rados.0.smithi107.stdout:10259: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:09.509 INFO:tasks.rados.rados.0.smithi107.stdout:10259: finishing write tid 5 to smithi10729407-6 2020-06-12T02:39:09.509 INFO:tasks.rados.rados.0.smithi107.stdout:10257: done (3 left) 2020-06-12T02:39:09.509 INFO:tasks.rados.rados.0.smithi107.stdout:10262: done (2 left) 2020-06-12T02:39:09.509 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:39:09.510 INFO:tasks.rados.rados.0.smithi107.stdout:10263: seq_num 10262 ranges {0=671744,671744=434176,1105920=368640} 2020-06-12T02:39:09.513 INFO:tasks.rados.rados.0.smithi107.stdout:10263: writing smithi10729407-2 from 0 to 671744 tid 1 2020-06-12T02:39:09.517 INFO:tasks.rados.rados.0.smithi107.stdout:10263: writing smithi10729407-2 from 671744 to 1105920 tid 2 2020-06-12T02:39:09.520 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:39:09.520 INFO:tasks.rados.rados.0.smithi107.stdout:10263: writing smithi10729407-2 from 1105920 to 1474560 tid 3 2020-06-12T02:39:09.520 INFO:tasks.rados.rados.0.smithi107.stdout:10264: delete oid 10 current snap is 0 2020-06-12T02:39:09.520 INFO:tasks.rados.rados.0.smithi107.stdout:10259: finishing write tid 6 to smithi10729407-6 2020-06-12T02:39:09.521 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1850 (ObjNum 10258 snap 0 seq_num 10258) dirty exists 2020-06-12T02:39:09.521 INFO:tasks.rados.rados.0.smithi107.stdout:10259: left oid 6 (ObjNum 10258 snap 0 seq_num 10258) 2020-06-12T02:39:09.521 INFO:tasks.rados.rados.0.smithi107.stdout:10261: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:09.521 INFO:tasks.rados.rados.0.smithi107.stdout:10261: finishing write tid 2 to smithi10729407-5 2020-06-12T02:39:09.521 INFO:tasks.rados.rados.0.smithi107.stdout:10261: finishing write tid 3 to smithi10729407-5 2020-06-12T02:39:09.522 INFO:tasks.rados.rados.0.smithi107.stdout:10261: finishing write tid 5 to smithi10729407-5 2020-06-12T02:39:09.522 INFO:tasks.rados.rados.0.smithi107.stdout:10261: finishing write tid 6 to smithi10729407-5 2020-06-12T02:39:09.522 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 977 (ObjNum 10260 snap 0 seq_num 10260) dirty exists 2020-06-12T02:39:09.522 INFO:tasks.rados.rados.0.smithi107.stdout:10261: left oid 5 (ObjNum 10260 snap 0 seq_num 10260) 2020-06-12T02:39:09.523 INFO:tasks.rados.rados.0.smithi107.stdout:10259: done (3 left) 2020-06-12T02:39:09.523 INFO:tasks.rados.rados.0.smithi107.stdout:10261: done (2 left) 2020-06-12T02:39:09.523 INFO:tasks.rados.rados.0.smithi107.stdout:10264: done (1 left) 2020-06-12T02:39:09.523 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:39:09.524 INFO:tasks.rados.rados.0.smithi107.stdout:10265: seq_num 10264 ranges {770048=704512,1474560=655360,2129920=147456} 2020-06-12T02:39:09.531 INFO:tasks.rados.rados.0.smithi107.stdout:10265: writing smithi10729407-4 from 770048 to 1474560 tid 1 2020-06-12T02:39:09.535 INFO:tasks.rados.rados.0.smithi107.stdout:10265: writing smithi10729407-4 from 1474560 to 2129920 tid 2 2020-06-12T02:39:09.536 INFO:tasks.rados.rados.0.smithi107.stdout:10265: writing smithi10729407-4 from 2129920 to 2277376 tid 3 2020-06-12T02:39:09.536 INFO:tasks.rados.rados.0.smithi107.stdout:10266: delete oid 8 current snap is 0 2020-06-12T02:39:09.536 INFO:tasks.rados.rados.0.smithi107.stdout:10263: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:09.536 INFO:tasks.rados.rados.0.smithi107.stdout:10263: finishing write tid 2 to smithi10729407-2 2020-06-12T02:39:09.536 INFO:tasks.rados.rados.0.smithi107.stdout:10263: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:09.537 INFO:tasks.rados.rados.0.smithi107.stdout:10263: finishing write tid 5 to smithi10729407-2 2020-06-12T02:39:09.537 INFO:tasks.rados.rados.0.smithi107.stdout:10263: finishing write tid 6 to smithi10729407-2 2020-06-12T02:39:09.537 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 967 (ObjNum 10262 snap 0 seq_num 10262) dirty exists 2020-06-12T02:39:09.537 INFO:tasks.rados.rados.0.smithi107.stdout:10263: left oid 2 (ObjNum 10262 snap 0 seq_num 10262) 2020-06-12T02:39:09.539 INFO:tasks.rados.rados.0.smithi107.stdout:10263: done (2 left) 2020-06-12T02:39:09.539 INFO:tasks.rados.rados.0.smithi107.stdout:10266: done (1 left) 2020-06-12T02:39:09.540 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:39:09.540 INFO:tasks.rados.rados.0.smithi107.stdout:10267: seq_num 10266 ranges {0=745472,745472=425984,1171456=376832} 2020-06-12T02:39:09.544 INFO:tasks.rados.rados.0.smithi107.stdout:10267: writing smithi10729407-12 from 0 to 745472 tid 1 2020-06-12T02:39:09.547 INFO:tasks.rados.rados.0.smithi107.stdout:10267: writing smithi10729407-12 from 745472 to 1171456 tid 2 2020-06-12T02:39:09.550 INFO:tasks.rados.rados.0.smithi107.stdout:10267: writing smithi10729407-12 from 1171456 to 1548288 tid 3 2020-06-12T02:39:09.550 INFO:tasks.rados.rados.0.smithi107.stdout:10268: delete oid 10 current snap is 0 2020-06-12T02:39:09.550 INFO:tasks.rados.rados.0.smithi107.stdout:10265: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:09.550 INFO:tasks.rados.rados.0.smithi107.stdout:10265: finishing write tid 2 to smithi10729407-4 2020-06-12T02:39:09.550 INFO:tasks.rados.rados.0.smithi107.stdout:10265: finishing write tid 3 to smithi10729407-4 2020-06-12T02:39:09.551 INFO:tasks.rados.rados.0.smithi107.stdout:10265: finishing write tid 5 to smithi10729407-4 2020-06-12T02:39:09.552 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:39:09.552 INFO:tasks.rados.rados.0.smithi107.stdout:10265: finishing write tid 6 to smithi10729407-4 2020-06-12T02:39:09.552 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1922 (ObjNum 10264 snap 0 seq_num 10264) dirty exists 2020-06-12T02:39:09.553 INFO:tasks.rados.rados.0.smithi107.stdout:10265: left oid 4 (ObjNum 10264 snap 0 seq_num 10264) 2020-06-12T02:39:09.553 INFO:tasks.rados.rados.0.smithi107.stdout:10265: done (2 left) 2020-06-12T02:39:09.553 INFO:tasks.rados.rados.0.smithi107.stdout:10268: done (1 left) 2020-06-12T02:39:09.553 INFO:tasks.rados.rados.0.smithi107.stdout:10269: delete oid 17 current snap is 0 2020-06-12T02:39:09.556 INFO:tasks.rados.rados.0.smithi107.stdout:10269: done (1 left) 2020-06-12T02:39:09.556 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:39:09.556 INFO:tasks.rados.rados.0.smithi107.stdout:10270: seq_num 10269 ranges {0=794624,794624=475136,1269760=327680} 2020-06-12T02:39:09.566 INFO:tasks.rados.rados.0.smithi107.stdout:10270: writing smithi10729407-15 from 0 to 794624 tid 1 2020-06-12T02:39:09.576 INFO:tasks.rados.rados.0.smithi107.stdout:10270: writing smithi10729407-15 from 794624 to 1269760 tid 2 2020-06-12T02:39:09.577 INFO:tasks.rados.rados.0.smithi107.stdout:10270: writing smithi10729407-15 from 1269760 to 1597440 tid 3 2020-06-12T02:39:09.578 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:39:09.578 INFO:tasks.rados.rados.0.smithi107.stdout:10271: seq_num 10270 ranges {1343488=409600,1753088=761856,2514944=442368} 2020-06-12T02:39:09.578 INFO:tasks.rados.rados.0.smithi107.stdout:10271: writing smithi10729407-7 from 1343488 to 1753088 tid 1 2020-06-12T02:39:09.578 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:39:09.575+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:39:09.580 INFO:tasks.rados.rados.0.smithi107.stdout:10271: writing smithi10729407-7 from 1753088 to 2514944 tid 2 2020-06-12T02:39:09.583 INFO:tasks.rados.rados.0.smithi107.stdout:10271: writing smithi10729407-7 from 2514944 to 2957312 tid 3 2020-06-12T02:39:09.583 INFO:tasks.rados.rados.0.smithi107.stdout:10272: delete oid 5 current snap is 0 2020-06-12T02:39:09.583 INFO:tasks.rados.rados.0.smithi107.stdout:10267: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:09.583 INFO:tasks.rados.rados.0.smithi107.stdout:10267: finishing write tid 2 to smithi10729407-12 2020-06-12T02:39:09.584 INFO:tasks.rados.rados.0.smithi107.stdout:10267: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:09.584 INFO:tasks.rados.rados.0.smithi107.stdout:10267: finishing write tid 5 to smithi10729407-12 2020-06-12T02:39:09.584 INFO:tasks.rados.rados.0.smithi107.stdout:10267: finishing write tid 6 to smithi10729407-12 2020-06-12T02:39:09.584 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1885 (ObjNum 10266 snap 0 seq_num 10266) dirty exists 2020-06-12T02:39:09.584 INFO:tasks.rados.rados.0.smithi107.stdout:10267: left oid 12 (ObjNum 10266 snap 0 seq_num 10266) 2020-06-12T02:39:09.585 INFO:tasks.rados.rados.0.smithi107.stdout:10270: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:09.585 INFO:tasks.rados.rados.0.smithi107.stdout:10270: finishing write tid 2 to smithi10729407-15 2020-06-12T02:39:09.585 INFO:tasks.rados.rados.0.smithi107.stdout:10270: finishing write tid 3 to smithi10729407-15 2020-06-12T02:39:09.585 INFO:tasks.rados.rados.0.smithi107.stdout:10270: finishing write tid 5 to smithi10729407-15 2020-06-12T02:39:09.586 INFO:tasks.rados.rados.0.smithi107.stdout:10271: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:09.586 INFO:tasks.rados.rados.0.smithi107.stdout:10270: finishing write tid 6 to smithi10729407-15 2020-06-12T02:39:09.587 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1902 (ObjNum 10269 snap 0 seq_num 10269) dirty exists 2020-06-12T02:39:09.587 INFO:tasks.rados.rados.0.smithi107.stdout:10270: left oid 15 (ObjNum 10269 snap 0 seq_num 10269) 2020-06-12T02:39:09.592 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:39:09.594 INFO:tasks.rados.rados.0.smithi107.stdout:10267: done (3 left) 2020-06-12T02:39:09.594 INFO:tasks.rados.rados.0.smithi107.stdout:10270: done (2 left) 2020-06-12T02:39:09.594 INFO:tasks.rados.rados.0.smithi107.stdout:10272: done (1 left) 2020-06-12T02:39:09.594 INFO:tasks.rados.rados.0.smithi107.stdout:10273: delete oid 5 current snap is 0 2020-06-12T02:39:09.596 INFO:tasks.rados.rados.0.smithi107.stdout:10271: finishing write tid 2 to smithi10729407-7 2020-06-12T02:39:09.596 INFO:tasks.rados.rados.0.smithi107.stdout:10271: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:09.597 INFO:tasks.rados.rados.0.smithi107.stdout:10271: finishing write tid 5 to smithi10729407-7 2020-06-12T02:39:09.598 INFO:tasks.rados.rados.0.smithi107.stdout:10273: done (1 left) 2020-06-12T02:39:09.598 INFO:tasks.rados.rados.0.smithi107.stdout:10274: delete oid 3 current snap is 0 2020-06-12T02:39:09.599 INFO:tasks.rados.rados.0.smithi107.stdout:10271: finishing write tid 6 to smithi10729407-7 2020-06-12T02:39:09.599 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1721 (ObjNum 10270 snap 0 seq_num 10270) dirty exists 2020-06-12T02:39:09.599 INFO:tasks.rados.rados.0.smithi107.stdout:10271: left oid 7 (ObjNum 10270 snap 0 seq_num 10270) 2020-06-12T02:39:09.600 INFO:teuthology.orchestra.run.smithi107.stdout:moved 0 / 96 (0%) 2020-06-12T02:39:09.600 INFO:teuthology.orchestra.run.smithi107.stdout:avg 12 2020-06-12T02:39:09.600 INFO:teuthology.orchestra.run.smithi107.stdout:stddev 2.95804 -> 2.95804 (expected baseline 3.24037) 2020-06-12T02:39:09.600 INFO:teuthology.orchestra.run.smithi107.stdout:min osd.2 with 6 -> 6 pgs (0.5 -> 0.5 * mean) 2020-06-12T02:39:09.600 INFO:teuthology.orchestra.run.smithi107.stdout:max osd.7 with 16 -> 16 pgs (1.33333 -> 1.33333 * mean) 2020-06-12T02:39:09.601 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:39:09.601 INFO:teuthology.orchestra.run.smithi107.stdout:oload 1000 2020-06-12T02:39:09.601 INFO:teuthology.orchestra.run.smithi107.stdout:max_change 1 2020-06-12T02:39:09.601 INFO:teuthology.orchestra.run.smithi107.stdout:max_change_osds 4 2020-06-12T02:39:09.602 INFO:teuthology.orchestra.run.smithi107.stdout:average_utilization 0.0112 2020-06-12T02:39:09.602 INFO:teuthology.orchestra.run.smithi107.stdout:overload_utilization 0.1115 2020-06-12T02:39:09.602 INFO:tasks.rados.rados.0.smithi107.stdout:10271: done (1 left) 2020-06-12T02:39:09.602 INFO:tasks.rados.rados.0.smithi107.stdout:10274: done (0 left) 2020-06-12T02:39:09.603 INFO:tasks.rados.rados.0.smithi107.stdout:10275: delete oid 19 current snap is 0 2020-06-12T02:39:09.603 INFO:teuthology.orchestra.run.smithi107.stderr:no change 2020-06-12T02:39:09.604 INFO:tasks.rados.rados.0.smithi107.stdout:10275: done (0 left) 2020-06-12T02:39:09.604 INFO:tasks.rados.rados.0.smithi107.stdout:10276: delete oid 8 current snap is 0 2020-06-12T02:39:09.608 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:39:09.609 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:39:09.609 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:39:09.609 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:39:09.609 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:39:09.610 INFO:tasks.rados.rados.0.smithi107.stdout:10276: done (0 left) 2020-06-12T02:39:09.610 INFO:tasks.rados.rados.0.smithi107.stdout:10277: delete oid 11 current snap is 0 2020-06-12T02:39:09.611 INFO:tasks.rados.rados.0.smithi107.stdout:10277: done (0 left) 2020-06-12T02:39:09.612 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:39:09.612 INFO:tasks.rados.rados.0.smithi107.stdout:10278: seq_num 10277 ranges {0=524288,524288=802816,1327104=401408} 2020-06-12T02:39:09.614 INFO:tasks.rados.rados.0.smithi107.stdout:10278: writing smithi10729407-8 from 0 to 524288 tid 1 2020-06-12T02:39:09.618 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-06-12T02:39:09.619 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:39:09.619 INFO:tasks.rados.rados.0.smithi107.stdout:10278: writing smithi10729407-8 from 524288 to 1327104 tid 2 2020-06-12T02:39:09.630 INFO:tasks.rados.rados.0.smithi107.stdout:10278: writing smithi10729407-8 from 1327104 to 1728512 tid 3 2020-06-12T02:39:09.631 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:39:09.631 INFO:tasks.rados.rados.0.smithi107.stdout:10279: seq_num 10278 ranges {0=540672,540672=688128,1228800=516096} 2020-06-12T02:39:09.631 INFO:tasks.rados.rados.0.smithi107.stdout:10279: writing smithi10729407-3 from 0 to 540672 tid 1 2020-06-12T02:39:09.631 INFO:tasks.rados.rados.0.smithi107.stdout:10279: writing smithi10729407-3 from 540672 to 1228800 tid 2 2020-06-12T02:39:09.633 INFO:tasks.rados.rados.0.smithi107.stdout:10279: writing smithi10729407-3 from 1228800 to 1744896 tid 3 2020-06-12T02:39:09.633 INFO:tasks.rados.rados.0.smithi107.stdout:10280: delete oid 2 current snap is 0 2020-06-12T02:39:09.633 INFO:tasks.rados.rados.0.smithi107.stdout:10278: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:09.634 INFO:tasks.rados.rados.0.smithi107.stdout:10278: finishing write tid 2 to smithi10729407-8 2020-06-12T02:39:09.634 INFO:tasks.rados.rados.0.smithi107.stdout:10279: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:09.634 INFO:tasks.rados.rados.0.smithi107.stdout:10278: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:09.634 INFO:tasks.rados.rados.0.smithi107.stdout:10278: finishing write tid 5 to smithi10729407-8 2020-06-12T02:39:09.637 INFO:tasks.rados.rados.0.smithi107.stdout:10278: finishing write tid 6 to smithi10729407-8 2020-06-12T02:39:09.637 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1907 (ObjNum 10277 snap 0 seq_num 10277) dirty exists 2020-06-12T02:39:09.637 INFO:tasks.rados.rados.0.smithi107.stdout:10278: left oid 8 (ObjNum 10277 snap 0 seq_num 10277) 2020-06-12T02:39:09.638 INFO:tasks.rados.rados.0.smithi107.stdout:10278: done (2 left) 2020-06-12T02:39:09.638 INFO:tasks.rados.rados.0.smithi107.stdout:10280: done (1 left) 2020-06-12T02:39:09.638 INFO:tasks.rados.rados.0.smithi107.stdout:10281: delete oid 17 current snap is 0 2020-06-12T02:39:09.643 INFO:tasks.rados.rados.0.smithi107.stdout:10281: done (1 left) 2020-06-12T02:39:09.643 INFO:tasks.rados.rados.0.smithi107.stdout:10282: delete oid 20 current snap is 0 2020-06-12T02:39:09.644 INFO:tasks.rados.rados.0.smithi107.stdout:10279: finishing write tid 2 to smithi10729407-3 2020-06-12T02:39:09.645 INFO:tasks.rados.rados.0.smithi107.stdout:10279: finishing write tid 3 to smithi10729407-3 2020-06-12T02:39:09.646 INFO:tasks.rados.rados.0.smithi107.stdout:10279: finishing write tid 5 to smithi10729407-3 2020-06-12T02:39:09.647 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2020-06-12T02:39:09.650 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-06-12T02:39:09.650 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-06-12T02:39:09.650 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-06-12T02:39:09.651 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-06-12T02:39:09.651 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-06-12T02:39:09.651 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-06-12T02:39:09.651 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-06-12T02:39:09.651 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-06-12T02:39:09.652 INFO:tasks.rados.rados.0.smithi107.stdout:10282: done (1 left) 2020-06-12T02:39:09.652 INFO:tasks.rados.rados.0.smithi107.stdout:10283: delete oid 14 current snap is 0 2020-06-12T02:39:09.653 INFO:tasks.rados.rados.0.smithi107.stdout:10279: finishing write tid 6 to smithi10729407-3 2020-06-12T02:39:09.653 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1832 (ObjNum 10278 snap 0 seq_num 10278) dirty exists 2020-06-12T02:39:09.654 INFO:tasks.rados.rados.0.smithi107.stdout:10279: left oid 3 (ObjNum 10278 snap 0 seq_num 10278) 2020-06-12T02:39:09.654 INFO:tasks.rados.rados.0.smithi107.stdout:10279: done (1 left) 2020-06-12T02:39:09.654 INFO:tasks.rados.rados.0.smithi107.stdout:10283: done (0 left) 2020-06-12T02:39:09.654 INFO:tasks.rados.rados.0.smithi107.stdout:10284: delete oid 15 current snap is 0 2020-06-12T02:39:09.655 INFO:tasks.rados.rados.0.smithi107.stdout:10284: done (0 left) 2020-06-12T02:39:09.655 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:39:09.655 INFO:tasks.rados.rados.0.smithi107.stdout:10285: seq_num 10284 ranges {0=647168,647168=442368,1089536=557056,1646592=204800} 2020-06-12T02:39:09.659 INFO:tasks.rados.rados.0.smithi107.stdout:10285: writing smithi10729407-10 from 0 to 647168 tid 1 2020-06-12T02:39:09.661 INFO:tasks.rados.rados.0.smithi107.stdout:10285: writing smithi10729407-10 from 647168 to 1089536 tid 2 2020-06-12T02:39:09.664 INFO:tasks.rados.rados.0.smithi107.stdout:10285: writing smithi10729407-10 from 1089536 to 1646592 tid 3 2020-06-12T02:39:09.665 INFO:tasks.rados.rados.0.smithi107.stdout:10285: writing smithi10729407-10 from 1646592 to 1851392 tid 4 2020-06-12T02:39:09.666 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:39:09.666 INFO:tasks.rados.rados.0.smithi107.stdout:10286: seq_num 10285 ranges {0=663552,663552=729088,1392640=475136} 2020-06-12T02:39:09.669 INFO:tasks.rados.rados.0.smithi107.stdout:10286: writing smithi10729407-14 from 0 to 663552 tid 1 2020-06-12T02:39:09.675 INFO:tasks.rados.rados.0.smithi107.stdout:10286: writing smithi10729407-14 from 663552 to 1392640 tid 2 2020-06-12T02:39:09.678 INFO:tasks.rados.rados.0.smithi107.stdout:10286: writing smithi10729407-14 from 1392640 to 1867776 tid 3 2020-06-12T02:39:09.678 INFO:tasks.rados.rados.0.smithi107.stdout:10287: delete oid 1 current snap is 0 2020-06-12T02:39:09.678 INFO:tasks.rados.rados.0.smithi107.stdout:10285: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:09.678 INFO:tasks.rados.rados.0.smithi107.stdout:10285: finishing write tid 2 to smithi10729407-10 2020-06-12T02:39:09.679 INFO:tasks.rados.rados.0.smithi107.stdout:10285: finishing write tid 3 to smithi10729407-10 2020-06-12T02:39:09.683 INFO:tasks.rados.rados.0.smithi107.stdout:10285: finishing write tid 4 to smithi10729407-10 2020-06-12T02:39:09.683 INFO:tasks.rados.rados.0.smithi107.stdout:10285: finishing write tid 6 to smithi10729407-10 2020-06-12T02:39:09.684 INFO:tasks.rados.rados.0.smithi107.stdout:10285: finishing write tid 7 to smithi10729407-10 2020-06-12T02:39:09.684 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 737 (ObjNum 10284 snap 0 seq_num 10284) dirty exists 2020-06-12T02:39:09.684 INFO:tasks.rados.rados.0.smithi107.stdout:10285: left oid 10 (ObjNum 10284 snap 0 seq_num 10284) 2020-06-12T02:39:09.686 INFO:tasks.rados.rados.0.smithi107.stdout:10286: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:09.688 INFO:tasks.rados.rados.0.smithi107.stdout:10286: finishing write tid 2 to smithi10729407-14 2020-06-12T02:39:09.688 INFO:tasks.rados.rados.0.smithi107.stdout:10286: finishing write tid 3 to smithi10729407-14 2020-06-12T02:39:09.689 INFO:tasks.rados.rados.0.smithi107.stdout:10286: finishing write tid 5 to smithi10729407-14 2020-06-12T02:39:09.691 INFO:tasks.rados.rados.0.smithi107.stdout:10285: done (2 left) 2020-06-12T02:39:09.692 INFO:tasks.rados.rados.0.smithi107.stdout:10287: done (1 left) 2020-06-12T02:39:09.692 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:39:09.692 INFO:tasks.rados.rados.0.smithi107.stdout:10288: seq_num 10287 ranges {0=696320,696320=540672,1236992=663552} 2020-06-12T02:39:09.695 INFO:tasks.rados.rados.0.smithi107.stdout:10288: writing smithi10729407-1 from 0 to 696320 tid 1 2020-06-12T02:39:09.698 INFO:tasks.rados.rados.0.smithi107.stdout:10288: writing smithi10729407-1 from 696320 to 1236992 tid 2 2020-06-12T02:39:09.703 INFO:tasks.rados.rados.0.smithi107.stdout:10288: writing smithi10729407-1 from 1236992 to 1900544 tid 3 2020-06-12T02:39:09.703 INFO:tasks.rados.rados.0.smithi107.stdout:10289: delete oid 4 current snap is 0 2020-06-12T02:39:09.703 INFO:tasks.rados.rados.0.smithi107.stdout:10286: finishing write tid 6 to smithi10729407-14 2020-06-12T02:39:09.703 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1673 (ObjNum 10285 snap 0 seq_num 10285) dirty exists 2020-06-12T02:39:09.703 INFO:tasks.rados.rados.0.smithi107.stdout:10286: left oid 14 (ObjNum 10285 snap 0 seq_num 10285) 2020-06-12T02:39:09.704 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:39:09.701+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:39:09.706 INFO:tasks.rados.rados.0.smithi107.stdout:10286: done (2 left) 2020-06-12T02:39:09.707 INFO:tasks.rados.rados.0.smithi107.stdout:10289: done (1 left) 2020-06-12T02:39:09.707 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:39:09.707 INFO:tasks.rados.rados.0.smithi107.stdout:10290: seq_num 10289 ranges {0=729088,729088=712704,1441792=491520} 2020-06-12T02:39:09.711 INFO:tasks.rados.rados.0.smithi107.stdout:10290: writing smithi10729407-20 from 0 to 729088 tid 1 2020-06-12T02:39:09.714 INFO:tasks.rados.rados.0.smithi107.stdout:10290: writing smithi10729407-20 from 729088 to 1441792 tid 2 2020-06-12T02:39:09.717 INFO:tasks.rados.rados.0.smithi107.stdout:10290: writing smithi10729407-20 from 1441792 to 1933312 tid 3 2020-06-12T02:39:09.718 INFO:tasks.rados.rados.0.smithi107.stdout:10291: delete oid 18 current snap is 0 2020-06-12T02:39:09.718 INFO:tasks.rados.rados.0.smithi107.stdout:10288: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:09.718 INFO:tasks.rados.rados.0.smithi107.stdout:10288: finishing write tid 2 to smithi10729407-1 2020-06-12T02:39:09.718 INFO:tasks.rados.rados.0.smithi107.stdout:10288: finishing write tid 3 to smithi10729407-1 2020-06-12T02:39:09.718 INFO:tasks.rados.rados.0.smithi107.stdout:10288: finishing write tid 5 to smithi10729407-1 2020-06-12T02:39:09.720 INFO:tasks.rados.rados.0.smithi107.stdout:10288: finishing write tid 6 to smithi10729407-1 2020-06-12T02:39:09.721 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1856 (ObjNum 10287 snap 0 seq_num 10287) dirty exists 2020-06-12T02:39:09.721 INFO:tasks.rados.rados.0.smithi107.stdout:10288: left oid 1 (ObjNum 10287 snap 0 seq_num 10287) 2020-06-12T02:39:09.728 INFO:tasks.rados.rados.0.smithi107.stdout:10288: done (2 left) 2020-06-12T02:39:09.728 INFO:tasks.rados.rados.0.smithi107.stdout:10291: done (1 left) 2020-06-12T02:39:09.728 INFO:tasks.rados.rados.0.smithi107.stdout:10292: delete oid 16 current snap is 0 2020-06-12T02:39:09.733 INFO:tasks.rados.rados.0.smithi107.stdout:10292: done (1 left) 2020-06-12T02:39:09.733 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:39:09.733 INFO:tasks.rados.rados.0.smithi107.stdout:10293: seq_num 10292 ranges {1548288=778240,2326528=770048,3096576=434176} 2020-06-12T02:39:09.747 INFO:tasks.rados.rados.0.smithi107.stdout:10293: writing smithi10729407-12 from 1548288 to 2326528 tid 1 2020-06-12T02:39:09.751 INFO:tasks.rados.rados.0.smithi107.stdout:10293: writing smithi10729407-12 from 2326528 to 3096576 tid 2 2020-06-12T02:39:09.753 INFO:tasks.rados.rados.0.smithi107.stdout:10293: writing smithi10729407-12 from 3096576 to 3530752 tid 3 2020-06-12T02:39:09.754 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:39:09.754 INFO:tasks.rados.rados.0.smithi107.stdout:10294: seq_num 10293 ranges {1851392=794624,2646016=655360,3301376=548864} 2020-06-12T02:39:09.769 INFO:tasks.rados.rados.0.smithi107.stdout:10294: writing smithi10729407-10 from 1851392 to 2646016 tid 1 2020-06-12T02:39:09.772 INFO:tasks.rados.rados.0.smithi107.stdout:10294: writing smithi10729407-10 from 2646016 to 3301376 tid 2 2020-06-12T02:39:09.778 INFO:tasks.rados.rados.0.smithi107.stdout:10294: writing smithi10729407-10 from 3301376 to 3850240 tid 3 2020-06-12T02:39:09.778 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:39:09.779 INFO:tasks.rados.rados.0.smithi107.stdout:10295: seq_num 10294 ranges {0=409600,409600=581632,991232=532480,1523712=491520} 2020-06-12T02:39:09.779 INFO:tasks.rados.rados.0.smithi107.stdout:10295: writing smithi10729407-11 from 0 to 409600 tid 1 2020-06-12T02:39:09.780 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:39:09.777+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:39:09.782 INFO:tasks.rados.rados.0.smithi107.stdout:10295: writing smithi10729407-11 from 409600 to 991232 tid 2 2020-06-12T02:39:09.786 INFO:tasks.rados.rados.0.smithi107.stdout:10295: writing smithi10729407-11 from 991232 to 1523712 tid 3 2020-06-12T02:39:09.790 INFO:tasks.rados.rados.0.smithi107.stdout:10295: writing smithi10729407-11 from 1523712 to 2015232 tid 4 2020-06-12T02:39:09.790 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:39:09.790 INFO:tasks.rados.rados.0.smithi107.stdout:10296: seq_num 10295 ranges {1867776=425984,2293760=466944,2760704=696320,3457024=442368} 2020-06-12T02:39:09.802 INFO:tasks.rados.rados.0.smithi107.stdout:10296: writing smithi10729407-14 from 1867776 to 2293760 tid 1 2020-06-12T02:39:09.805 INFO:tasks.rados.rados.0.smithi107.stdout:10296: writing smithi10729407-14 from 2293760 to 2760704 tid 2 2020-06-12T02:39:09.810 INFO:tasks.rados.rados.0.smithi107.stdout:10296: writing smithi10729407-14 from 2760704 to 3457024 tid 3 2020-06-12T02:39:09.813 INFO:tasks.rados.rados.0.smithi107.stdout:10296: writing smithi10729407-14 from 3457024 to 3899392 tid 4 2020-06-12T02:39:09.813 INFO:tasks.rados.rados.0.smithi107.stdout:10297: delete oid 15 current snap is 0 2020-06-12T02:39:09.814 INFO:tasks.rados.rados.0.smithi107.stdout:10290: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:09.814 INFO:tasks.rados.rados.0.smithi107.stdout:10290: finishing write tid 2 to smithi10729407-20 2020-06-12T02:39:09.814 INFO:tasks.rados.rados.0.smithi107.stdout:10290: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:09.814 INFO:tasks.rados.rados.0.smithi107.stdout:10290: finishing write tid 5 to smithi10729407-20 2020-06-12T02:39:09.814 INFO:tasks.rados.rados.0.smithi107.stdout:10290: finishing write tid 6 to smithi10729407-20 2020-06-12T02:39:09.815 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 793 (ObjNum 10289 snap 0 seq_num 10289) dirty exists 2020-06-12T02:39:09.815 INFO:tasks.rados.rados.0.smithi107.stdout:10290: left oid 20 (ObjNum 10289 snap 0 seq_num 10289) 2020-06-12T02:39:09.815 INFO:tasks.rados.rados.0.smithi107.stdout:10293: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:09.815 INFO:tasks.rados.rados.0.smithi107.stdout:10293: finishing write tid 2 to smithi10729407-12 2020-06-12T02:39:09.815 INFO:tasks.rados.rados.0.smithi107.stdout:10293: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:09.816 INFO:tasks.rados.rados.0.smithi107.stdout:10293: finishing write tid 5 to smithi10729407-12 2020-06-12T02:39:09.816 INFO:tasks.rados.rados.0.smithi107.stdout:10293: finishing write tid 6 to smithi10729407-12 2020-06-12T02:39:09.816 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1890 (ObjNum 10292 snap 0 seq_num 10292) dirty exists 2020-06-12T02:39:09.816 INFO:tasks.rados.rados.0.smithi107.stdout:10293: left oid 12 (ObjNum 10292 snap 0 seq_num 10292) 2020-06-12T02:39:09.816 INFO:tasks.rados.rados.0.smithi107.stdout:10294: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:09.817 INFO:tasks.rados.rados.0.smithi107.stdout:10295: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:09.817 INFO:tasks.rados.rados.0.smithi107.stdout:10294: finishing write tid 2 to smithi10729407-10 2020-06-12T02:39:09.817 INFO:tasks.rados.rados.0.smithi107.stdout:10294: finishing write tid 3 to smithi10729407-10 2020-06-12T02:39:09.818 INFO:tasks.rados.rados.0.smithi107.stdout:10294: finishing write tid 5 to smithi10729407-10 2020-06-12T02:39:09.818 INFO:tasks.rados.rados.0.smithi107.stdout:10294: finishing write tid 6 to smithi10729407-10 2020-06-12T02:39:09.818 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 741 (ObjNum 10293 snap 0 seq_num 10293) dirty exists 2020-06-12T02:39:09.818 INFO:tasks.rados.rados.0.smithi107.stdout:10294: left oid 10 (ObjNum 10293 snap 0 seq_num 10293) 2020-06-12T02:39:09.818 INFO:tasks.rados.rados.0.smithi107.stdout:10295: finishing write tid 2 to smithi10729407-11 2020-06-12T02:39:09.818 INFO:tasks.rados.rados.0.smithi107.stdout:10295: finishing write tid 3 to smithi10729407-11 2020-06-12T02:39:09.819 INFO:tasks.rados.rados.0.smithi107.stdout:10295: finishing write tid 4 to smithi10729407-11 2020-06-12T02:39:09.819 INFO:tasks.rados.rados.0.smithi107.stdout:10295: finishing write tid 6 to smithi10729407-11 2020-06-12T02:39:09.819 INFO:tasks.rados.rados.0.smithi107.stdout:10295: finishing write tid 7 to smithi10729407-11 2020-06-12T02:39:09.819 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1688 (ObjNum 10294 snap 0 seq_num 10294) dirty exists 2020-06-12T02:39:09.819 INFO:tasks.rados.rados.0.smithi107.stdout:10295: left oid 11 (ObjNum 10294 snap 0 seq_num 10294) 2020-06-12T02:39:09.820 INFO:tasks.rados.rados.0.smithi107.stdout:10296: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:09.820 INFO:tasks.rados.rados.0.smithi107.stdout:10296: finishing write tid 2 to smithi10729407-14 2020-06-12T02:39:09.821 INFO:tasks.rados.rados.0.smithi107.stdout:10290: done (5 left) 2020-06-12T02:39:09.822 INFO:tasks.rados.rados.0.smithi107.stdout:10293: done (4 left) 2020-06-12T02:39:09.823 INFO:tasks.rados.rados.0.smithi107.stdout:10294: done (3 left) 2020-06-12T02:39:09.823 INFO:tasks.rados.rados.0.smithi107.stdout:10295: done (2 left) 2020-06-12T02:39:09.824 INFO:tasks.rados.rados.0.smithi107.stdout:10297: done (1 left) 2020-06-12T02:39:09.824 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:39:09.824 INFO:tasks.rados.rados.0.smithi107.stdout:10298: seq_num 10297 ranges {0=458752,458752=638976,1097728=712704,1810432=253952} 2020-06-12T02:39:09.824 INFO:tasks.rados.rados.0.smithi107.stdout:10298: writing smithi10729407-19 from 0 to 458752 tid 1 2020-06-12T02:39:09.826 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:39:09.826 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:39:09.826 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:39:09.827 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:39:09.827 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.827 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1206 3.5s0 3:a9510bda:::smithi10729407-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:39:09.827 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.856688+0000", 2020-06-12T02:39:09.827 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 96.954392561999995, 2020-06-12T02:39:09.828 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019116942000000001, 2020-06-12T02:39:09.828 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:09.828 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:09.828 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:09.830 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:09.830 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:09.830 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1206 2020-06-12T02:39:09.830 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.831 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:09.831 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.832 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:09.833 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856688+0000", 2020-06-12T02:39:09.833 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:09.834 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.834 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.835 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:09.836 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856688+0000", 2020-06-12T02:39:09.836 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4355e-05 2020-06-12T02:39:09.836 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.836 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.837 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:09.837 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856713+0000", 2020-06-12T02:39:09.837 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0850000000000001e-06 2020-06-12T02:39:09.837 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.838 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.838 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:09.838 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856716+0000", 2020-06-12T02:39:09.841 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4509999999999997e-06 2020-06-12T02:39:09.841 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.841 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.842 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:09.842 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856722+0000", 2020-06-12T02:39:09.842 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2879999999999999e-06 2020-06-12T02:39:09.843 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.843 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.843 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:09.846 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856724+0000", 2020-06-12T02:39:09.846 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0057507449999999998 2020-06-12T02:39:09.846 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.847 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.847 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:09.847 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.862475+0000", 2020-06-12T02:39:09.847 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5759000000000003e-05 2020-06-12T02:39:09.847 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.848 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.848 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:09.848 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.862571+0000", 2020-06-12T02:39:09.848 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012199018000000001 2020-06-12T02:39:09.848 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.849 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.849 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:09.849 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.874770+0000", 2020-06-12T02:39:09.849 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7260999999999999e-05 2020-06-12T02:39:09.850 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.850 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.850 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:09.850 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.874827+0000", 2020-06-12T02:39:09.850 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00097798000000000004 2020-06-12T02:39:09.851 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.851 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.851 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:09.851 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.875805+0000", 2020-06-12T02:39:09.851 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019116942000000001 2020-06-12T02:39:09.852 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:09.852 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:09.852 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:09.852 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.852 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.852 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1572 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~745472] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-06-12T02:39:09.853 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.886439+0000", 2020-06-12T02:39:09.853 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 95.924641844000007, 2020-06-12T02:39:09.853 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021442571000000001, 2020-06-12T02:39:09.853 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:09.854 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:09.854 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:09.854 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:09.854 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:09.854 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1572 2020-06-12T02:39:09.854 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.855 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:09.855 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.855 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:09.855 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.886439+0000", 2020-06-12T02:39:09.856 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:09.856 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.856 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.857 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:09.857 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.886439+0000", 2020-06-12T02:39:09.857 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00073027599999999997 2020-06-12T02:39:09.857 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.857 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.858 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:09.858 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887169+0000", 2020-06-12T02:39:09.858 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.49e-06 2020-06-12T02:39:09.858 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.858 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.859 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:09.859 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887175+0000", 2020-06-12T02:39:09.859 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7899999999999994e-06 2020-06-12T02:39:09.859 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.859 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.860 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:09.860 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887185+0000", 2020-06-12T02:39:09.860 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.42e-06 2020-06-12T02:39:09.860 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.860 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.860 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:09.861 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887189+0000", 2020-06-12T02:39:09.861 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1619999999999999e-05 2020-06-12T02:39:09.861 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.861 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.861 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:09.862 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887221+0000", 2020-06-12T02:39:09.862 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9720000000000002e-05 2020-06-12T02:39:09.862 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.862 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.862 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:09.863 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887300+0000", 2020-06-12T02:39:09.863 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047354100000000001 2020-06-12T02:39:09.863 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.863 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.863 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:09.864 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887774+0000", 2020-06-12T02:39:09.864 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0030649750000000002 2020-06-12T02:39:09.864 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.864 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.864 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:09.865 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.890839+0000", 2020-06-12T02:39:09.865 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016995321000000001 2020-06-12T02:39:09.865 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.865 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.865 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:09.867 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.907834+0000", 2020-06-12T02:39:09.867 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7417999999999999e-05 2020-06-12T02:39:09.867 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.867 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.868 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:09.868 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.907882+0000", 2020-06-12T02:39:09.868 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021442571000000001 2020-06-12T02:39:09.868 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:09.868 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:09.869 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:09.869 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.869 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.869 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1573 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~712704] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-06-12T02:39:09.869 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.890888+0000", 2020-06-12T02:39:09.870 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 95.920193108999996, 2020-06-12T02:39:09.870 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018868001999999998, 2020-06-12T02:39:09.870 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:09.870 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:09.870 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:09.871 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:09.871 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:09.871 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1573 2020-06-12T02:39:09.871 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.871 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:09.872 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.872 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:09.872 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.890888+0000", 2020-06-12T02:39:09.872 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:09.872 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.873 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.873 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:09.873 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.890888+0000", 2020-06-12T02:39:09.873 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000683605 2020-06-12T02:39:09.873 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.874 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.874 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:09.874 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891571+0000", 2020-06-12T02:39:09.874 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4999999999999999e-06 2020-06-12T02:39:09.874 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.875 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.875 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:09.875 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891577+0000", 2020-06-12T02:39:09.876 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.171e-05 2020-06-12T02:39:09.877 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.877 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.877 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:09.878 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891589+0000", 2020-06-12T02:39:09.878 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7650000000000001e-06 2020-06-12T02:39:09.878 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.878 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.878 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:09.879 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891593+0000", 2020-06-12T02:39:09.879 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6689000000000001e-05 2020-06-12T02:39:09.879 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.879 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.879 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:09.880 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891630+0000", 2020-06-12T02:39:09.880 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9877999999999997e-05 2020-06-12T02:39:09.882 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.882 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.883 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:09.884 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891710+0000", 2020-06-12T02:39:09.884 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053367200000000001 2020-06-12T02:39:09.884 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.884 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.885 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:09.885 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.892244+0000", 2020-06-12T02:39:09.885 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004927196 2020-06-12T02:39:09.885 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.886 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.886 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:09.886 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.897171+0000", 2020-06-12T02:39:09.886 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012496419 2020-06-12T02:39:09.887 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.888 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.888 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:09.889 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.909667+0000", 2020-06-12T02:39:09.889 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8567999999999996e-05 2020-06-12T02:39:09.889 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.889 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.890 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:09.890 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.909756+0000", 2020-06-12T02:39:09.890 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018868001999999998 2020-06-12T02:39:09.890 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:09.890 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:09.890 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:09.891 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.891 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.891 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1576 3.5s0 3:a446366d:::smithi10729407-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e26)", 2020-06-12T02:39:09.891 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.891955+0000", 2020-06-12T02:39:09.891 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 95.919125868999998, 2020-06-12T02:39:09.892 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019893246999999999, 2020-06-12T02:39:09.892 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:09.892 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:09.892 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:09.892 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:09.893 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:09.893 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1576 2020-06-12T02:39:09.893 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.893 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:09.893 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.894 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:09.894 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891955+0000", 2020-06-12T02:39:09.894 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:09.894 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.894 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.895 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:09.895 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891955+0000", 2020-06-12T02:39:09.895 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2067999999999999e-05 2020-06-12T02:39:09.895 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.895 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.895 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:09.896 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891977+0000", 2020-06-12T02:39:09.896 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.18e-06 2020-06-12T02:39:09.896 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.896 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.896 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:09.897 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891980+0000", 2020-06-12T02:39:09.897 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3629999999999999e-06 2020-06-12T02:39:09.897 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.897 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.898 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:09.898 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891987+0000", 2020-06-12T02:39:09.898 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.181e-06 2020-06-12T02:39:09.898 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.898 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.899 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:09.899 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891989+0000", 2020-06-12T02:39:09.899 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050959120000000002 2020-06-12T02:39:09.899 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.899 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.900 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:09.900 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.897085+0000", 2020-06-12T02:39:09.900 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1089999999999998e-05 2020-06-12T02:39:09.900 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.900 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.900 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:09.901 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.897166+0000", 2020-06-12T02:39:09.901 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013400235 2020-06-12T02:39:09.901 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.901 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.901 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:09.902 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.910566+0000", 2020-06-12T02:39:09.902 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000104981 2020-06-12T02:39:09.902 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.902 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.902 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:09.903 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.910671+0000", 2020-06-12T02:39:09.903 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001177237 2020-06-12T02:39:09.903 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.903 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.904 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:09.904 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.911848+0000", 2020-06-12T02:39:09.904 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019893246999999999 2020-06-12T02:39:09.904 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:09.904 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:09.905 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:09.905 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.905 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.905 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2913 3.5s0 3:a9510bda:::smithi10729407-14:head [append 0~770048,stat] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:39:09.905 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.070354+0000", 2020-06-12T02:39:09.906 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 92.740727067999998, 2020-06-12T02:39:09.906 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60690798599999995, 2020-06-12T02:39:09.906 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:09.906 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:09.906 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:09.907 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:09.907 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:09.907 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2913 2020-06-12T02:39:09.907 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.907 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:09.908 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.908 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:09.908 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.070354+0000", 2020-06-12T02:39:09.908 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:09.908 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.909 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.909 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:09.909 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.070354+0000", 2020-06-12T02:39:09.909 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00098478900000000002 2020-06-12T02:39:09.909 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.909 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.910 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:09.910 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071339+0000", 2020-06-12T02:39:09.910 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8349999999999993e-06 2020-06-12T02:39:09.910 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.910 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.911 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:09.911 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071347+0000", 2020-06-12T02:39:09.911 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0749e-05 2020-06-12T02:39:09.911 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.911 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.911 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:09.912 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071358+0000", 2020-06-12T02:39:09.912 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9369999999999997e-06 2020-06-12T02:39:09.912 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.912 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.912 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:09.913 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071362+0000", 2020-06-12T02:39:09.913 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0426e-05 2020-06-12T02:39:09.913 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.913 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.913 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:09.914 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071423+0000", 2020-06-12T02:39:09.914 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4129999999999998e-05 2020-06-12T02:39:09.914 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.914 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.914 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:09.914 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071487+0000", 2020-06-12T02:39:09.915 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053752599999999998 2020-06-12T02:39:09.915 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.915 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.915 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:09.915 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072024+0000", 2020-06-12T02:39:09.916 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036162579999999998 2020-06-12T02:39:09.916 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.916 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.916 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:09.916 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075640+0000", 2020-06-12T02:39:09.916 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60152201999999999 2020-06-12T02:39:09.917 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.917 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.917 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:09.917 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677162+0000", 2020-06-12T02:39:09.917 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9315999999999996e-05 2020-06-12T02:39:09.918 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.918 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.918 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:09.918 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677262+0000", 2020-06-12T02:39:09.918 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60690798599999995 2020-06-12T02:39:09.919 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:09.919 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:09.919 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:09.919 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.919 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.919 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2914 3.5s0 3:a9510bda:::smithi10729407-14:head [append 0~401408] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:39:09.920 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.072409+0000", 2020-06-12T02:39:09.920 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 92.738671624000006, 2020-06-12T02:39:09.920 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.61956801100000003, 2020-06-12T02:39:09.920 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:09.920 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:09.921 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:09.921 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:09.921 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:09.921 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2914 2020-06-12T02:39:09.921 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.922 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:09.922 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.922 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:09.922 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072409+0000", 2020-06-12T02:39:09.922 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:09.922 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.923 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.923 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:09.923 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072409+0000", 2020-06-12T02:39:09.923 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000446317 2020-06-12T02:39:09.924 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.924 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.924 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:09.924 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072856+0000", 2020-06-12T02:39:09.924 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2140000000000004e-06 2020-06-12T02:39:09.925 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.925 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.925 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:09.925 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072861+0000", 2020-06-12T02:39:09.925 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8710000000000003e-06 2020-06-12T02:39:09.926 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.926 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.926 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:09.926 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072870+0000", 2020-06-12T02:39:09.926 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2270000000000004e-06 2020-06-12T02:39:09.927 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.927 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.927 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:09.927 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072874+0000", 2020-06-12T02:39:09.927 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024649960000000001 2020-06-12T02:39:09.927 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.928 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.928 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:09.928 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075339+0000", 2020-06-12T02:39:09.928 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9179999999999996e-05 2020-06-12T02:39:09.928 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.929 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.929 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:09.929 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075438+0000", 2020-06-12T02:39:09.929 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60189281400000005 2020-06-12T02:39:09.929 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.930 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.930 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:09.930 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677331+0000", 2020-06-12T02:39:09.930 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012602800000000001 2020-06-12T02:39:09.930 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.930 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.931 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:09.931 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677457+0000", 2020-06-12T02:39:09.931 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00088167699999999998 2020-06-12T02:39:09.931 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.931 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.932 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:09.932 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.678339+0000", 2020-06-12T02:39:09.932 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004312995 2020-06-12T02:39:09.932 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.932 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.932 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:09.933 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.682652+0000", 2020-06-12T02:39:09.933 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0092869140000000003 2020-06-12T02:39:09.933 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.933 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.933 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:09.934 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691938+0000", 2020-06-12T02:39:09.934 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8778e-05 2020-06-12T02:39:09.934 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.934 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.934 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:09.934 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691977+0000", 2020-06-12T02:39:09.935 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.61956801100000003 2020-06-12T02:39:09.935 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:09.935 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:09.935 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:09.935 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.936 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.936 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2915 3.5s0 3:a9510bda:::smithi10729407-14:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:39:09.936 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.072912+0000", 2020-06-12T02:39:09.936 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 92.738168884000004, 2020-06-12T02:39:09.936 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.6192204, 2020-06-12T02:39:09.937 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:09.937 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:09.937 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:09.937 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:09.937 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:09.937 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2915 2020-06-12T02:39:09.938 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.938 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:09.938 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.938 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:09.938 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072912+0000", 2020-06-12T02:39:09.939 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:09.939 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.939 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.939 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:09.939 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072912+0000", 2020-06-12T02:39:09.940 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9201e-05 2020-06-12T02:39:09.940 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.940 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.940 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:09.940 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072951+0000", 2020-06-12T02:39:09.940 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5480000000000002e-06 2020-06-12T02:39:09.941 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.941 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.941 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:09.941 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072955+0000", 2020-06-12T02:39:09.941 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3879999999999997e-06 2020-06-12T02:39:09.942 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.942 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.942 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:09.942 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072961+0000", 2020-06-12T02:39:09.942 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6589999999999999e-06 2020-06-12T02:39:09.942 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.943 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.943 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:09.943 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072964+0000", 2020-06-12T02:39:09.943 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025221039999999998 2020-06-12T02:39:09.943 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.944 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.944 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:09.944 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075486+0000", 2020-06-12T02:39:09.944 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5038999999999998e-05 2020-06-12T02:39:09.944 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.944 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.945 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:09.945 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075561+0000", 2020-06-12T02:39:09.945 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60452397499999999 2020-06-12T02:39:09.945 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.945 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.946 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:09.946 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680085+0000", 2020-06-12T02:39:09.946 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1398999999999999e-05 2020-06-12T02:39:09.946 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.946 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.946 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:09.947 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680156+0000", 2020-06-12T02:39:09.947 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000274465 2020-06-12T02:39:09.947 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.947 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.947 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:09.948 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680431+0000", 2020-06-12T02:39:09.948 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025071999999999998 2020-06-12T02:39:09.948 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.948 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.948 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:09.949 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.682938+0000", 2020-06-12T02:39:09.949 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0091116449999999998 2020-06-12T02:39:09.949 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.949 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.949 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:09.949 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692050+0000", 2020-06-12T02:39:09.950 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2776999999999997e-05 2020-06-12T02:39:09.950 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.950 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.950 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:09.950 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692132+0000", 2020-06-12T02:39:09.951 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.6192204 2020-06-12T02:39:09.951 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:09.951 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:09.951 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:09.951 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.951 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.952 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2916 3.5s0 3:a9510bda:::smithi10729407-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:39:09.952 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.072986+0000", 2020-06-12T02:39:09.952 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 92.738094645000004, 2020-06-12T02:39:09.952 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.62063941099999997, 2020-06-12T02:39:09.952 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:09.953 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:09.953 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:09.953 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:09.953 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:09.953 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2916 2020-06-12T02:39:09.953 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.954 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:09.954 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.954 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:09.954 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072986+0000", 2020-06-12T02:39:09.954 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:09.955 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.955 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.955 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:09.955 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072986+0000", 2020-06-12T02:39:09.955 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2101000000000001e-05 2020-06-12T02:39:09.956 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.956 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.956 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:09.956 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073008+0000", 2020-06-12T02:39:09.956 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1549999999999999e-06 2020-06-12T02:39:09.956 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.957 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.957 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:09.957 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073011+0000", 2020-06-12T02:39:09.957 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2360000000000001e-06 2020-06-12T02:39:09.957 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.958 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.958 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:09.958 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073018+0000", 2020-06-12T02:39:09.958 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2790000000000001e-06 2020-06-12T02:39:09.958 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.959 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.959 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:09.959 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073020+0000", 2020-06-12T02:39:09.959 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025637670000000002 2020-06-12T02:39:09.959 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.959 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.960 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:09.960 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075584+0000", 2020-06-12T02:39:09.960 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1814999999999998e-05 2020-06-12T02:39:09.960 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.960 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.961 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:09.961 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075636+0000", 2020-06-12T02:39:09.961 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60499943199999995 2020-06-12T02:39:09.961 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.961 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.962 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:09.962 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680635+0000", 2020-06-12T02:39:09.962 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0968000000000002e-05 2020-06-12T02:39:09.962 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.963 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.963 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:09.963 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680706+0000", 2020-06-12T02:39:09.963 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011777101999999999 2020-06-12T02:39:09.963 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.964 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.964 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:09.964 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692483+0000", 2020-06-12T02:39:09.964 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5890999999999995e-05 2020-06-12T02:39:09.964 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.964 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.965 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:09.965 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692569+0000", 2020-06-12T02:39:09.965 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001056665 2020-06-12T02:39:09.965 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.965 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.966 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:09.966 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.693626+0000", 2020-06-12T02:39:09.966 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.62063941099999997 2020-06-12T02:39:09.966 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:09.966 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:09.966 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:09.967 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.967 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.967 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2922 3.ds1 3:b983fd28:::smithi10729407-1:head [append 0~466944] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:39:09.967 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.097192+0000", 2020-06-12T02:39:09.967 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 92.713889277999996, 2020-06-12T02:39:09.968 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58672644699999998, 2020-06-12T02:39:09.968 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:09.968 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:09.968 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:09.968 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:09.968 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:09.969 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2922 2020-06-12T02:39:09.969 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.969 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:09.969 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.969 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:09.970 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097192+0000", 2020-06-12T02:39:09.970 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:09.970 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.970 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.970 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:09.971 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097192+0000", 2020-06-12T02:39:09.971 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047206100000000002 2020-06-12T02:39:09.971 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.971 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.971 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:09.971 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097664+0000", 2020-06-12T02:39:09.972 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8359999999999999e-06 2020-06-12T02:39:09.972 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.972 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.972 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:09.972 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097668+0000", 2020-06-12T02:39:09.973 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0815999999999999e-05 2020-06-12T02:39:09.973 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.974 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.974 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:09.974 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097679+0000", 2020-06-12T02:39:09.974 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.028e-06 2020-06-12T02:39:09.974 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.975 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.975 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:09.975 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097684+0000", 2020-06-12T02:39:09.975 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3932999999999999e-05 2020-06-12T02:39:09.975 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.976 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.976 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:09.976 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097748+0000", 2020-06-12T02:39:09.976 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2214999999999999e-05 2020-06-12T02:39:09.976 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.976 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.977 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:09.977 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097820+0000", 2020-06-12T02:39:09.977 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00041583800000000003 2020-06-12T02:39:09.977 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.977 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.978 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:09.978 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.098236+0000", 2020-06-12T02:39:09.978 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0020339960000000002 2020-06-12T02:39:09.978 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.978 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.978 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:09.979 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.100270+0000", 2020-06-12T02:39:09.979 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58355657299999997 2020-06-12T02:39:09.979 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.979 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.979 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:09.980 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.683827+0000", 2020-06-12T02:39:09.980 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1150999999999995e-05 2020-06-12T02:39:09.980 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.980 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.980 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:09.980 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.683918+0000", 2020-06-12T02:39:09.981 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58672644699999998 2020-06-12T02:39:09.981 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:09.981 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:09.981 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:09.981 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.982 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.982 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2923 3.ds1 3:b983fd28:::smithi10729407-1:head [append 0~614400] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:39:09.982 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.100504+0000", 2020-06-12T02:39:09.982 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 92.710576637000003, 2020-06-12T02:39:09.982 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58853946999999995, 2020-06-12T02:39:09.983 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:09.983 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:09.983 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:09.983 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:09.983 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:09.983 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2923 2020-06-12T02:39:09.984 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.984 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:09.984 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.984 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:09.984 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.100504+0000", 2020-06-12T02:39:09.985 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:09.985 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.985 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.985 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:09.985 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.100504+0000", 2020-06-12T02:39:09.985 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00068395399999999996 2020-06-12T02:39:09.986 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.986 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.986 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:09.986 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101188+0000", 2020-06-12T02:39:09.986 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4050000000000003e-06 2020-06-12T02:39:09.987 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.987 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.987 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:09.987 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101196+0000", 2020-06-12T02:39:09.987 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.328e-05 2020-06-12T02:39:09.988 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.988 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.988 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:09.989 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101209+0000", 2020-06-12T02:39:09.989 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3419999999999996e-06 2020-06-12T02:39:09.989 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.989 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.990 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:09.990 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101215+0000", 2020-06-12T02:39:09.990 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1389000000000005e-05 2020-06-12T02:39:09.990 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.991 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.991 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:09.991 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101277+0000", 2020-06-12T02:39:09.991 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7724000000000004e-05 2020-06-12T02:39:09.991 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.991 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.992 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:09.992 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101344+0000", 2020-06-12T02:39:09.992 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047852499999999999 2020-06-12T02:39:09.992 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.992 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.993 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:09.993 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101823+0000", 2020-06-12T02:39:09.993 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0039255510000000002 2020-06-12T02:39:09.993 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.993 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.994 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:09.994 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105748+0000", 2020-06-12T02:39:09.994 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58321340099999996 2020-06-12T02:39:09.994 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.994 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.994 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:09.995 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.688962+0000", 2020-06-12T02:39:09.995 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1898999999999996e-05 2020-06-12T02:39:09.995 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.995 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.995 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:09.996 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689044+0000", 2020-06-12T02:39:09.996 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58853946999999995 2020-06-12T02:39:09.997 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:09.997 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:09.998 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:09.998 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:09.998 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:09.998 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2924 3.ds1 3:b983fd28:::smithi10729407-1:head [append 0~188416] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:39:09.998 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.101633+0000", 2020-06-12T02:39:09.999 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 92.709447643000004, 2020-06-12T02:39:09.999 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58758259199999996, 2020-06-12T02:39:09.999 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:09.999 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:09.999 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:10.000 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:10.000 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:10.000 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2924 2020-06-12T02:39:10.000 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.000 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:10.001 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.001 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:10.001 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101633+0000", 2020-06-12T02:39:10.001 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:10.001 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.001 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.002 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:10.002 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101633+0000", 2020-06-12T02:39:10.002 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00018775399999999999 2020-06-12T02:39:10.002 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.002 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.003 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:10.003 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101821+0000", 2020-06-12T02:39:10.003 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7669999999999998e-06 2020-06-12T02:39:10.003 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.003 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.004 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:10.004 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101828+0000", 2020-06-12T02:39:10.004 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2716e-05 2020-06-12T02:39:10.004 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.004 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.004 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:10.005 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101840+0000", 2020-06-12T02:39:10.005 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.215e-06 2020-06-12T02:39:10.005 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.005 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.006 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:10.006 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101846+0000", 2020-06-12T02:39:10.006 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021214250000000001 2020-06-12T02:39:10.006 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.006 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.007 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:10.007 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.103967+0000", 2020-06-12T02:39:10.007 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1018000000000003e-05 2020-06-12T02:39:10.007 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.007 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.008 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:10.008 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.104038+0000", 2020-06-12T02:39:10.008 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000333364 2020-06-12T02:39:10.008 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.008 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.008 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:10.009 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.104371+0000", 2020-06-12T02:39:10.009 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001378546 2020-06-12T02:39:10.009 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.009 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.009 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:10.010 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105750+0000", 2020-06-12T02:39:10.010 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58338272899999999 2020-06-12T02:39:10.010 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.010 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.010 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:10.011 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689133+0000", 2020-06-12T02:39:10.012 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3058000000000003e-05 2020-06-12T02:39:10.012 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.012 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.012 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:10.013 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689216+0000", 2020-06-12T02:39:10.013 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58758259199999996 2020-06-12T02:39:10.013 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.013 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:10.013 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.014 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.014 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.014 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2925 3.ds1 3:b983fd28:::smithi10729407-1:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:39:10.014 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.101899+0000", 2020-06-12T02:39:10.014 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 92.709181978000004, 2020-06-12T02:39:10.015 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58747810899999997, 2020-06-12T02:39:10.015 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:10.015 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:10.015 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:10.015 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:10.015 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:10.016 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2925 2020-06-12T02:39:10.016 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.016 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:10.016 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.016 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:10.017 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101899+0000", 2020-06-12T02:39:10.017 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:10.017 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.017 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.017 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:10.018 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101899+0000", 2020-06-12T02:39:10.018 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2906999999999998e-05 2020-06-12T02:39:10.018 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.018 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.018 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:10.018 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101972+0000", 2020-06-12T02:39:10.019 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1789999999999996e-06 2020-06-12T02:39:10.019 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.019 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.019 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:10.019 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101978+0000", 2020-06-12T02:39:10.020 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1358e-05 2020-06-12T02:39:10.020 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.020 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.020 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:10.020 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101989+0000", 2020-06-12T02:39:10.020 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.031e-06 2020-06-12T02:39:10.021 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.021 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.021 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:10.021 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101993+0000", 2020-06-12T02:39:10.021 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033218990000000001 2020-06-12T02:39:10.022 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.022 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.022 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:10.022 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105315+0000", 2020-06-12T02:39:10.022 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6324999999999999e-05 2020-06-12T02:39:10.023 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.023 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.023 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:10.023 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105382+0000", 2020-06-12T02:39:10.023 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000216624 2020-06-12T02:39:10.023 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.024 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.024 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:10.024 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105598+0000", 2020-06-12T02:39:10.024 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001236586 2020-06-12T02:39:10.024 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.025 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.025 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:10.025 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.106835+0000", 2020-06-12T02:39:10.026 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58245509200000001 2020-06-12T02:39:10.026 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.026 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.027 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:10.027 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689290+0000", 2020-06-12T02:39:10.027 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7107999999999993e-05 2020-06-12T02:39:10.027 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.027 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.028 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:10.028 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689377+0000", 2020-06-12T02:39:10.028 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58747810899999997 2020-06-12T02:39:10.028 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.028 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:10.028 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.029 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.029 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.029 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2926 3.ds1 3:b983fd28:::smithi10729407-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:39:10.029 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.102036+0000", 2020-06-12T02:39:10.029 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 92.709044993999996, 2020-06-12T02:39:10.030 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58868048699999997, 2020-06-12T02:39:10.030 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:10.030 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:10.030 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:10.031 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:10.031 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:10.031 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2926 2020-06-12T02:39:10.031 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.031 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:10.032 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.032 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:10.032 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102036+0000", 2020-06-12T02:39:10.032 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:10.032 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.033 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.033 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:10.033 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102036+0000", 2020-06-12T02:39:10.033 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3786999999999999e-05 2020-06-12T02:39:10.033 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.034 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.034 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:10.034 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102080+0000", 2020-06-12T02:39:10.034 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7660000000000004e-06 2020-06-12T02:39:10.034 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.035 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.035 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:10.035 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102085+0000", 2020-06-12T02:39:10.035 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3145e-05 2020-06-12T02:39:10.035 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.036 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.036 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:10.036 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102099+0000", 2020-06-12T02:39:10.036 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2440000000000004e-06 2020-06-12T02:39:10.037 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.038 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.038 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:10.038 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102103+0000", 2020-06-12T02:39:10.039 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036826990000000002 2020-06-12T02:39:10.039 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.039 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.039 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:10.039 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105786+0000", 2020-06-12T02:39:10.039 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.6774000000000006e-05 2020-06-12T02:39:10.040 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.040 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.040 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:10.040 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105882+0000", 2020-06-12T02:39:10.040 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58352084900000001 2020-06-12T02:39:10.041 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.041 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.041 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:10.041 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689403+0000", 2020-06-12T02:39:10.041 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3788999999999995e-05 2020-06-12T02:39:10.042 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.042 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.042 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:10.042 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689467+0000", 2020-06-12T02:39:10.042 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001249434 2020-06-12T02:39:10.042 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.043 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.043 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:10.043 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690716+0000", 2020-06-12T02:39:10.043 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58868048699999997 2020-06-12T02:39:10.043 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.044 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:10.044 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.044 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.044 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.044 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2950 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~598016] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:39:10.045 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.162562+0000", 2020-06-12T02:39:10.045 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 92.648518620000004, 2020-06-12T02:39:10.045 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52780611399999999, 2020-06-12T02:39:10.045 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:10.046 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:10.046 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:10.046 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:10.047 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:10.047 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2950 2020-06-12T02:39:10.047 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.048 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:10.048 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.049 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:10.050 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.162562+0000", 2020-06-12T02:39:10.050 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:10.050 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.050 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.050 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:10.051 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.162562+0000", 2020-06-12T02:39:10.051 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00070718099999999998 2020-06-12T02:39:10.051 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.052 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.052 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:10.052 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163269+0000", 2020-06-12T02:39:10.052 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2029999999999997e-06 2020-06-12T02:39:10.052 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.052 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.053 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:10.053 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163276+0000", 2020-06-12T02:39:10.053 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2734999999999999e-05 2020-06-12T02:39:10.053 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.053 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.054 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:10.054 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163288+0000", 2020-06-12T02:39:10.054 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.404e-06 2020-06-12T02:39:10.054 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.055 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.055 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:10.055 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163295+0000", 2020-06-12T02:39:10.055 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5784000000000006e-05 2020-06-12T02:39:10.055 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.055 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.056 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:10.056 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163361+0000", 2020-06-12T02:39:10.056 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5318e-05 2020-06-12T02:39:10.057 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.057 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.058 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:10.058 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163436+0000", 2020-06-12T02:39:10.058 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054801100000000005 2020-06-12T02:39:10.058 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.058 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.058 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:10.059 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163984+0000", 2020-06-12T02:39:10.059 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.007023077 2020-06-12T02:39:10.059 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.059 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.059 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:10.060 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171007+0000", 2020-06-12T02:39:10.060 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51932051199999996 2020-06-12T02:39:10.060 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.060 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.061 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:10.061 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690327+0000", 2020-06-12T02:39:10.061 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0889000000000001e-05 2020-06-12T02:39:10.061 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.061 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.062 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:10.062 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690368+0000", 2020-06-12T02:39:10.062 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52780611399999999 2020-06-12T02:39:10.062 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.062 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:10.062 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.063 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.063 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.063 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2951 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~532480] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:39:10.063 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.165470+0000", 2020-06-12T02:39:10.064 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 92.645611012000003, 2020-06-12T02:39:10.064 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52507917900000001, 2020-06-12T02:39:10.064 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:10.064 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:10.064 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:10.065 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:10.066 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:10.066 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2951 2020-06-12T02:39:10.066 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.066 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:10.067 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.067 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:10.067 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.165470+0000", 2020-06-12T02:39:10.067 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:10.067 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.068 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.068 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:10.068 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.165470+0000", 2020-06-12T02:39:10.068 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00062231600000000004 2020-06-12T02:39:10.069 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.069 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.069 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:10.069 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166092+0000", 2020-06-12T02:39:10.070 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1610000000000002e-06 2020-06-12T02:39:10.070 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.070 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.071 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:10.071 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166097+0000", 2020-06-12T02:39:10.071 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7780000000000002e-06 2020-06-12T02:39:10.071 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.071 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.072 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:10.072 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166107+0000", 2020-06-12T02:39:10.072 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7169999999999999e-06 2020-06-12T02:39:10.072 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.072 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.073 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:10.073 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166112+0000", 2020-06-12T02:39:10.073 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4792000000000002e-05 2020-06-12T02:39:10.073 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.073 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.074 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:10.074 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166147+0000", 2020-06-12T02:39:10.074 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9336e-05 2020-06-12T02:39:10.074 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.074 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.075 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:10.075 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166236+0000", 2020-06-12T02:39:10.075 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00067747200000000004 2020-06-12T02:39:10.075 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.076 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.076 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:10.076 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166913+0000", 2020-06-12T02:39:10.076 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0047718669999999999 2020-06-12T02:39:10.077 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.077 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.078 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:10.078 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171685+0000", 2020-06-12T02:39:10.078 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51881990700000002 2020-06-12T02:39:10.079 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.079 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.079 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:10.079 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690505+0000", 2020-06-12T02:39:10.080 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3832999999999999e-05 2020-06-12T02:39:10.080 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.080 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.080 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:10.080 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690549+0000", 2020-06-12T02:39:10.081 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52507917900000001 2020-06-12T02:39:10.081 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.081 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:10.081 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.081 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.081 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.082 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2952 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~270336] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:39:10.082 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.166891+0000", 2020-06-12T02:39:10.082 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 92.644189362999995, 2020-06-12T02:39:10.082 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52426417000000003, 2020-06-12T02:39:10.086 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:10.087 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:10.087 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:10.087 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:10.087 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:10.087 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2952 2020-06-12T02:39:10.088 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.088 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:10.088 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.088 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:10.088 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166891+0000", 2020-06-12T02:39:10.089 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:10.089 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.089 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.089 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:10.089 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166891+0000", 2020-06-12T02:39:10.089 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00031878599999999999 2020-06-12T02:39:10.090 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.090 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.090 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:10.090 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167210+0000", 2020-06-12T02:39:10.090 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2360000000000004e-06 2020-06-12T02:39:10.091 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.091 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.091 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:10.091 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167219+0000", 2020-06-12T02:39:10.091 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3298e-05 2020-06-12T02:39:10.092 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.092 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.092 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:10.092 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167232+0000", 2020-06-12T02:39:10.092 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5849999999999999e-06 2020-06-12T02:39:10.092 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.093 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.093 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:10.094 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167237+0000", 2020-06-12T02:39:10.094 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0018755740000000001 2020-06-12T02:39:10.094 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.095 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.095 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:10.095 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.169113+0000", 2020-06-12T02:39:10.095 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000104768 2020-06-12T02:39:10.095 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.096 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.096 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:10.096 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.169218+0000", 2020-06-12T02:39:10.097 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00057389799999999999 2020-06-12T02:39:10.097 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.097 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.097 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:10.097 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.169792+0000", 2020-06-12T02:39:10.098 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0019333989999999999 2020-06-12T02:39:10.098 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.099 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.100 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:10.100 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171725+0000", 2020-06-12T02:39:10.100 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51938508699999997 2020-06-12T02:39:10.100 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.101 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.101 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:10.101 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691110+0000", 2020-06-12T02:39:10.101 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5538999999999999e-05 2020-06-12T02:39:10.101 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.102 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.102 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:10.102 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691156+0000", 2020-06-12T02:39:10.102 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52426417000000003 2020-06-12T02:39:10.102 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.103 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:10.103 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.103 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.103 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.103 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2953 3.5s0 3:a446366d:::smithi10729407-11:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:39:10.103 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.167293+0000", 2020-06-12T02:39:10.104 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 92.643787915999994, 2020-06-12T02:39:10.104 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52428350999999995, 2020-06-12T02:39:10.104 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:10.104 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:10.104 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:10.105 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:10.105 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:10.105 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2953 2020-06-12T02:39:10.105 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.105 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:10.106 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.106 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:10.106 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167293+0000", 2020-06-12T02:39:10.106 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:10.106 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.106 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.107 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:10.107 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167293+0000", 2020-06-12T02:39:10.107 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3880000000000004e-05 2020-06-12T02:39:10.107 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.107 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.108 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:10.108 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167367+0000", 2020-06-12T02:39:10.109 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.319e-06 2020-06-12T02:39:10.109 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.109 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.109 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:10.109 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167373+0000", 2020-06-12T02:39:10.110 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1719000000000001e-05 2020-06-12T02:39:10.110 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.110 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.110 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:10.110 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167385+0000", 2020-06-12T02:39:10.111 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4580000000000003e-06 2020-06-12T02:39:10.111 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.111 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.111 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:10.111 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167389+0000", 2020-06-12T02:39:10.111 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036341110000000002 2020-06-12T02:39:10.112 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.112 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.112 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:10.112 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171023+0000", 2020-06-12T02:39:10.112 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9418000000000005e-05 2020-06-12T02:39:10.113 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.113 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.113 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:10.113 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171093+0000", 2020-06-12T02:39:10.113 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00029087900000000002 2020-06-12T02:39:10.114 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.114 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.114 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:10.115 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171384+0000", 2020-06-12T02:39:10.115 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000489347 2020-06-12T02:39:10.115 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.116 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.116 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:10.116 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171873+0000", 2020-06-12T02:39:10.116 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51963691300000003 2020-06-12T02:39:10.117 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.117 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.118 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:10.118 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691510+0000", 2020-06-12T02:39:10.118 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6465999999999997e-05 2020-06-12T02:39:10.118 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.119 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.119 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:10.119 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691576+0000", 2020-06-12T02:39:10.119 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52428350999999995 2020-06-12T02:39:10.119 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.119 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:10.120 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.120 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.120 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.120 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2954 3.5s0 3:a446366d:::smithi10729407-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:39:10.120 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.167432+0000", 2020-06-12T02:39:10.121 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 92.643649167000007, 2020-06-12T02:39:10.121 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52568652699999996, 2020-06-12T02:39:10.121 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:10.121 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:10.121 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:10.122 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:10.122 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:10.122 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2954 2020-06-12T02:39:10.122 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.122 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:10.123 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.123 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:10.123 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167432+0000", 2020-06-12T02:39:10.123 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:10.123 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.123 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.124 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:10.124 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167432+0000", 2020-06-12T02:39:10.124 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3556000000000001e-05 2020-06-12T02:39:10.124 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.124 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.125 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:10.125 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167475+0000", 2020-06-12T02:39:10.125 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7350000000000001e-06 2020-06-12T02:39:10.125 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.125 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.125 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:10.126 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167481+0000", 2020-06-12T02:39:10.126 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3128e-05 2020-06-12T02:39:10.126 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.126 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.126 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:10.128 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167494+0000", 2020-06-12T02:39:10.128 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1579999999999998e-06 2020-06-12T02:39:10.128 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.128 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.129 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:10.129 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167498+0000", 2020-06-12T02:39:10.129 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004082174 2020-06-12T02:39:10.129 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.129 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.130 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:10.130 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171580+0000", 2020-06-12T02:39:10.130 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8954000000000002e-05 2020-06-12T02:39:10.130 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.130 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.131 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:10.131 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171679+0000", 2020-06-12T02:39:10.131 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51993235100000001 2020-06-12T02:39:10.131 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.132 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.132 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:10.132 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691612+0000", 2020-06-12T02:39:10.133 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4329000000000001e-05 2020-06-12T02:39:10.133 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.133 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.133 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:10.133 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691666+0000", 2020-06-12T02:39:10.134 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001452142 2020-06-12T02:39:10.135 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.135 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.135 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:10.136 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.693118+0000", 2020-06-12T02:39:10.137 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52568652699999996 2020-06-12T02:39:10.137 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.137 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:10.137 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.137 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.138 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.138 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2959 3.ds1 3:b3445631:::smithi10729407-6:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e27)", 2020-06-12T02:39:10.138 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.176248+0000", 2020-06-12T02:39:10.138 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 92.634832426000003, 2020-06-12T02:39:10.138 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51549877099999997, 2020-06-12T02:39:10.139 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:10.139 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:10.139 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:10.139 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:10.139 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:10.139 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2959 2020-06-12T02:39:10.140 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.140 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:10.140 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.140 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:10.141 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176248+0000", 2020-06-12T02:39:10.141 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:10.141 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.141 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.142 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:10.142 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176248+0000", 2020-06-12T02:39:10.142 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5120999999999999e-05 2020-06-12T02:39:10.142 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.143 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.143 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:10.144 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176274+0000", 2020-06-12T02:39:10.144 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9189999999999998e-06 2020-06-12T02:39:10.144 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.144 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.145 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:10.145 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176278+0000", 2020-06-12T02:39:10.145 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6339e-05 2020-06-12T02:39:10.145 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.146 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.146 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:10.146 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176305+0000", 2020-06-12T02:39:10.146 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9089999999999999e-06 2020-06-12T02:39:10.146 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.147 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.147 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:10.147 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176310+0000", 2020-06-12T02:39:10.147 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6095999999999999e-05 2020-06-12T02:39:10.147 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.147 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.148 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:10.148 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176376+0000", 2020-06-12T02:39:10.148 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8590999999999994e-05 2020-06-12T02:39:10.148 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.148 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.149 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:10.149 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176444+0000", 2020-06-12T02:39:10.149 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00024346400000000001 2020-06-12T02:39:10.150 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.151 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.151 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:10.151 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176688+0000", 2020-06-12T02:39:10.151 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00044220899999999999 2020-06-12T02:39:10.151 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.152 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.152 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:10.152 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.177130+0000", 2020-06-12T02:39:10.152 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51452984400000001 2020-06-12T02:39:10.152 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.153 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.153 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:10.153 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691660+0000", 2020-06-12T02:39:10.153 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7279000000000002e-05 2020-06-12T02:39:10.153 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.153 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.154 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:10.154 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691747+0000", 2020-06-12T02:39:10.154 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51549877099999997 2020-06-12T02:39:10.154 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.154 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:10.155 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.155 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.155 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.155 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:3032 3.ds1 3:b983fd28:::smithi10729407-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:39:10.156 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.864289+0000", 2020-06-12T02:39:10.156 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 91.946791508000004, 2020-06-12T02:39:10.156 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019572216999999999, 2020-06-12T02:39:10.156 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:10.156 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:10.157 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:10.157 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:10.157 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:10.157 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 3032 2020-06-12T02:39:10.157 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.158 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:10.158 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.158 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:10.158 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864289+0000", 2020-06-12T02:39:10.158 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:10.158 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.159 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.159 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:10.159 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864289+0000", 2020-06-12T02:39:10.159 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5701999999999997e-05 2020-06-12T02:39:10.159 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.160 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.160 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:10.160 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864335+0000", 2020-06-12T02:39:10.160 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0910000000000004e-06 2020-06-12T02:39:10.160 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.161 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.161 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:10.161 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864342+0000", 2020-06-12T02:39:10.161 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2301e-05 2020-06-12T02:39:10.162 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.162 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.162 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:10.162 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864354+0000", 2020-06-12T02:39:10.162 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5199999999999997e-06 2020-06-12T02:39:10.162 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.163 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.163 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:10.163 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864360+0000", 2020-06-12T02:39:10.163 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00049180999999999997 2020-06-12T02:39:10.163 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.164 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.164 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:10.164 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864852+0000", 2020-06-12T02:39:10.164 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6725000000000003e-05 2020-06-12T02:39:10.164 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.165 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.165 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:10.165 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864908+0000", 2020-06-12T02:39:10.165 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0089687010000000008 2020-06-12T02:39:10.166 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.166 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.166 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:10.166 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.873877+0000", 2020-06-12T02:39:10.166 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3382999999999999e-05 2020-06-12T02:39:10.167 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.167 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.167 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:10.167 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.873941+0000", 2020-06-12T02:39:10.167 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.006408927 2020-06-12T02:39:10.167 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.168 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.168 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:10.168 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.880350+0000", 2020-06-12T02:39:10.169 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3905e-05 2020-06-12T02:39:10.170 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.170 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.170 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:10.171 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.880403+0000", 2020-06-12T02:39:10.171 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0034581519999999999 2020-06-12T02:39:10.171 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.171 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.172 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:10.172 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.883862+0000", 2020-06-12T02:39:10.172 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019572216999999999 2020-06-12T02:39:10.172 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.173 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:10.173 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.173 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.173 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:10.173 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:39:10.174 INFO:tasks.rados.rados.0.smithi107.stdout:10298: writing smithi10729407-19 from 458752 to 1097728 tid 2 2020-06-12T02:39:10.188 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-06-12T02:39:10.189 INFO:tasks.rados.rados.0.smithi107.stdout:10298: writing smithi10729407-19 from 1097728 to 1810432 tid 3 2020-06-12T02:39:10.191 INFO:tasks.rados.rados.0.smithi107.stdout:10298: writing smithi10729407-19 from 1810432 to 2064384 tid 4 2020-06-12T02:39:10.191 INFO:tasks.rados.rados.0.smithi107.stdout:10299: delete oid 5 current snap is 0 2020-06-12T02:39:10.192 INFO:tasks.rados.rados.0.smithi107.stdout:10296: finishing write tid 3 to smithi10729407-14 2020-06-12T02:39:10.192 INFO:tasks.rados.rados.0.smithi107.stdout:10296: finishing write tid 4 to smithi10729407-14 2020-06-12T02:39:10.192 INFO:tasks.rados.rados.0.smithi107.stdout:10296: finishing write tid 6 to smithi10729407-14 2020-06-12T02:39:10.192 INFO:tasks.rados.rados.0.smithi107.stdout:10296: finishing write tid 7 to smithi10729407-14 2020-06-12T02:39:10.192 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1678 (ObjNum 10295 snap 0 seq_num 10295) dirty exists 2020-06-12T02:39:10.193 INFO:tasks.rados.rados.0.smithi107.stdout:10296: left oid 14 (ObjNum 10295 snap 0 seq_num 10295) 2020-06-12T02:39:10.193 INFO:tasks.rados.rados.0.smithi107.stdout:10298: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:10.193 INFO:tasks.rados.rados.0.smithi107.stdout:10296: done (2 left) 2020-06-12T02:39:10.193 INFO:tasks.rados.rados.0.smithi107.stdout:10299: done (1 left) 2020-06-12T02:39:10.193 INFO:tasks.rados.rados.0.smithi107.stdout:10300: delete oid 13 current snap is 0 2020-06-12T02:39:10.194 INFO:tasks.rados.rados.0.smithi107.stdout:10300: done (1 left) 2020-06-12T02:39:10.194 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:39:10.194 INFO:tasks.rados.rados.0.smithi107.stdout:10301: seq_num 10300 ranges {0=507904,507904=688128,1196032=483328,1679360=434176} 2020-06-12T02:39:10.194 INFO:tasks.rados.rados.0.smithi107.stdout:10301: writing smithi10729407-16 from 0 to 507904 tid 1 2020-06-12T02:39:10.195 INFO:tasks.rados.rados.0.smithi107.stdout:10301: writing smithi10729407-16 from 507904 to 1196032 tid 2 2020-06-12T02:39:10.195 INFO:tasks.rados.rados.0.smithi107.stdout:10301: writing smithi10729407-16 from 1196032 to 1679360 tid 3 2020-06-12T02:39:10.195 INFO:tasks.rados.rados.0.smithi107.stdout:10301: writing smithi10729407-16 from 1679360 to 2113536 tid 4 2020-06-12T02:39:10.195 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:39:10.197 INFO:tasks.rados.rados.0.smithi107.stdout:10302: seq_num 10301 ranges {2957312=524288,3481600=573440,4055040=688128,4743168=344064} 2020-06-12T02:39:10.197 INFO:tasks.rados.rados.0.smithi107.stdout:10302: writing smithi10729407-7 from 2957312 to 3481600 tid 1 2020-06-12T02:39:10.197 INFO:tasks.rados.rados.0.smithi107.stdout:10302: writing smithi10729407-7 from 3481600 to 4055040 tid 2 2020-06-12T02:39:10.197 INFO:tasks.rados.rados.0.smithi107.stdout:10302: writing smithi10729407-7 from 4055040 to 4743168 tid 3 2020-06-12T02:39:10.197 INFO:tasks.rados.rados.0.smithi107.stdout:10302: writing smithi10729407-7 from 4743168 to 5087232 tid 4 2020-06-12T02:39:10.198 INFO:tasks.rados.rados.0.smithi107.stdout:10303: delete oid 2 current snap is 0 2020-06-12T02:39:10.198 INFO:tasks.rados.rados.0.smithi107.stdout:10298: finishing write tid 2 to smithi10729407-19 2020-06-12T02:39:10.198 INFO:tasks.rados.rados.0.smithi107.stdout:10298: finishing write tid 3 to smithi10729407-19 2020-06-12T02:39:10.198 INFO:tasks.rados.rados.0.smithi107.stdout:10298: finishing write tid 4 to smithi10729407-19 2020-06-12T02:39:10.198 INFO:tasks.rados.rados.0.smithi107.stdout:10298: finishing write tid 6 to smithi10729407-19 2020-06-12T02:39:10.199 INFO:tasks.rados.rados.0.smithi107.stdout:10298: finishing write tid 7 to smithi10729407-19 2020-06-12T02:39:10.199 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1895 (ObjNum 10297 snap 0 seq_num 10297) dirty exists 2020-06-12T02:39:10.199 INFO:tasks.rados.rados.0.smithi107.stdout:10298: left oid 19 (ObjNum 10297 snap 0 seq_num 10297) 2020-06-12T02:39:10.199 INFO:tasks.rados.rados.0.smithi107.stdout:10301: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:10.199 INFO:tasks.rados.rados.0.smithi107.stdout:10301: finishing write tid 2 to smithi10729407-16 2020-06-12T02:39:10.200 INFO:tasks.rados.rados.0.smithi107.stdout:10301: finishing write tid 3 to smithi10729407-16 2020-06-12T02:39:10.200 INFO:tasks.rados.rados.0.smithi107.stdout:10301: finishing write tid 4 to smithi10729407-16 2020-06-12T02:39:10.200 INFO:tasks.rados.rados.0.smithi107.stdout:10301: finishing write tid 6 to smithi10729407-16 2020-06-12T02:39:10.200 INFO:tasks.rados.rados.0.smithi107.stdout:10301: finishing write tid 7 to smithi10729407-16 2020-06-12T02:39:10.200 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1838 (ObjNum 10300 snap 0 seq_num 10300) dirty exists 2020-06-12T02:39:10.201 INFO:tasks.rados.rados.0.smithi107.stdout:10301: left oid 16 (ObjNum 10300 snap 0 seq_num 10300) 2020-06-12T02:39:10.201 INFO:tasks.rados.rados.0.smithi107.stdout:10302: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:10.201 INFO:tasks.rados.rados.0.smithi107.stdout:10298: done (3 left) 2020-06-12T02:39:10.201 INFO:tasks.rados.rados.0.smithi107.stdout:10301: done (2 left) 2020-06-12T02:39:10.201 INFO:tasks.rados.rados.0.smithi107.stdout:10303: done (1 left) 2020-06-12T02:39:10.202 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:39:10.202 INFO:tasks.rados.rados.0.smithi107.stdout:10304: seq_num 10303 ranges {0=565248,565248=794624,1359872=704512,2064384=106496} 2020-06-12T02:39:10.202 INFO:tasks.rados.rados.0.smithi107.stdout:10304: writing smithi10729407-18 from 0 to 565248 tid 1 2020-06-12T02:39:10.202 INFO:tasks.rados.rados.0.smithi107.stdout:10304: writing smithi10729407-18 from 565248 to 1359872 tid 2 2020-06-12T02:39:10.202 INFO:tasks.rados.rados.0.smithi107.stdout:10304: writing smithi10729407-18 from 1359872 to 2064384 tid 3 2020-06-12T02:39:10.203 INFO:tasks.rados.rados.0.smithi107.stdout:10304: writing smithi10729407-18 from 2064384 to 2170880 tid 4 2020-06-12T02:39:10.203 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:39:10.203 INFO:tasks.rados.rados.0.smithi107.stdout:10305: seq_num 10304 ranges {0=581632,581632=679936,1261568=458752,1720320=466944} 2020-06-12T02:39:10.203 INFO:tasks.rados.rados.0.smithi107.stdout:10305: writing smithi10729407-17 from 0 to 581632 tid 1 2020-06-12T02:39:10.203 INFO:tasks.rados.rados.0.smithi107.stdout:10305: writing smithi10729407-17 from 581632 to 1261568 tid 2 2020-06-12T02:39:10.203 INFO:tasks.rados.rados.0.smithi107.stdout:10305: writing smithi10729407-17 from 1261568 to 1720320 tid 3 2020-06-12T02:39:10.204 INFO:tasks.rados.rados.0.smithi107.stdout:10305: writing smithi10729407-17 from 1720320 to 2187264 tid 4 2020-06-12T02:39:10.204 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:39:10.204 INFO:tasks.rados.rados.0.smithi107.stdout:10306: seq_num 10305 ranges {3530752=598016,4128768=565248,4694016=671744,5365760=368640} 2020-06-12T02:39:10.204 INFO:tasks.rados.rados.0.smithi107.stdout:10306: writing smithi10729407-12 from 3530752 to 4128768 tid 1 2020-06-12T02:39:10.204 INFO:tasks.rados.rados.0.smithi107.stdout:10306: writing smithi10729407-12 from 4128768 to 4694016 tid 2 2020-06-12T02:39:10.205 INFO:tasks.rados.rados.0.smithi107.stdout:10306: writing smithi10729407-12 from 4694016 to 5365760 tid 3 2020-06-12T02:39:10.205 INFO:tasks.rados.rados.0.smithi107.stdout:10306: writing smithi10729407-12 from 5365760 to 5734400 tid 4 2020-06-12T02:39:10.205 INFO:tasks.rados.rados.0.smithi107.stdout:10307: delete oid 2 current snap is 0 2020-06-12T02:39:10.205 INFO:tasks.rados.rados.0.smithi107.stdout:10302: finishing write tid 2 to smithi10729407-7 2020-06-12T02:39:10.205 INFO:tasks.rados.rados.0.smithi107.stdout:10302: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:10.205 INFO:tasks.rados.rados.0.smithi107.stdout:10302: finishing write tid 4 to smithi10729407-7 2020-06-12T02:39:10.206 INFO:tasks.rados.rados.0.smithi107.stdout:10302: finishing write tid 6 to smithi10729407-7 2020-06-12T02:39:10.206 INFO:tasks.rados.rados.0.smithi107.stdout:10304: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:10.206 INFO:tasks.rados.rados.0.smithi107.stdout:10302: finishing write tid 7 to smithi10729407-7 2020-06-12T02:39:10.206 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1726 (ObjNum 10301 snap 0 seq_num 10301) dirty exists 2020-06-12T02:39:10.206 INFO:tasks.rados.rados.0.smithi107.stdout:10302: left oid 7 (ObjNum 10301 snap 0 seq_num 10301) 2020-06-12T02:39:10.207 INFO:tasks.rados.rados.0.smithi107.stdout:10304: finishing write tid 2 to smithi10729407-18 2020-06-12T02:39:10.207 INFO:tasks.rados.rados.0.smithi107.stdout:10304: finishing write tid 3 to smithi10729407-18 2020-06-12T02:39:10.207 INFO:tasks.rados.rados.0.smithi107.stdout:10305: finishing write tid 1 to smithi10729407-17 2020-06-12T02:39:10.207 INFO:tasks.rados.rados.0.smithi107.stdout:10304: finishing write tid 4 to smithi10729407-18 2020-06-12T02:39:10.207 INFO:tasks.rados.rados.0.smithi107.stdout:10305: finishing write tid 2 to smithi10729407-17 2020-06-12T02:39:10.208 INFO:tasks.rados.rados.0.smithi107.stdout:10304: finishing write tid 6 to smithi10729407-18 2020-06-12T02:39:10.208 INFO:tasks.rados.rados.0.smithi107.stdout:10305: finishing write tid 3 to smithi10729407-17 2020-06-12T02:39:10.208 INFO:tasks.rados.rados.0.smithi107.stdout:10305: finishing write tid 4 to smithi10729407-17 2020-06-12T02:39:10.208 INFO:tasks.rados.rados.0.smithi107.stdout:10305: finishing write tid 6 to smithi10729407-17 2020-06-12T02:39:10.208 INFO:tasks.rados.rados.0.smithi107.stdout:10304: finishing write tid 7 to smithi10729407-18 2020-06-12T02:39:10.208 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1821 (ObjNum 10303 snap 0 seq_num 10303) dirty exists 2020-06-12T02:39:10.209 INFO:tasks.rados.rados.0.smithi107.stdout:10304: left oid 18 (ObjNum 10303 snap 0 seq_num 10303) 2020-06-12T02:39:10.209 INFO:tasks.rados.rados.0.smithi107.stdout:10305: finishing write tid 7 to smithi10729407-17 2020-06-12T02:39:10.209 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1930 (ObjNum 10304 snap 0 seq_num 10304) dirty exists 2020-06-12T02:39:10.209 INFO:tasks.rados.rados.0.smithi107.stdout:10305: left oid 17 (ObjNum 10304 snap 0 seq_num 10304) 2020-06-12T02:39:10.209 INFO:tasks.rados.rados.0.smithi107.stdout:10302: done (4 left) 2020-06-12T02:39:10.210 INFO:tasks.rados.rados.0.smithi107.stdout:10304: done (3 left) 2020-06-12T02:39:10.210 INFO:tasks.rados.rados.0.smithi107.stdout:10305: done (2 left) 2020-06-12T02:39:10.210 INFO:tasks.rados.rados.0.smithi107.stdout:10307: done (1 left) 2020-06-12T02:39:10.210 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:39:10.210 INFO:tasks.rados.rados.0.smithi107.stdout:10308: seq_num 10307 ranges {1409024=630784,2039808=729088,2768896=688128,3457024=188416} 2020-06-12T02:39:10.210 INFO:tasks.rados.rados.0.smithi107.stdout:10308: writing smithi10729407-6 from 1409024 to 2039808 tid 1 2020-06-12T02:39:10.211 INFO:tasks.rados.rados.0.smithi107.stdout:10308: writing smithi10729407-6 from 2039808 to 2768896 tid 2 2020-06-12T02:39:10.211 INFO:tasks.rados.rados.0.smithi107.stdout:10308: writing smithi10729407-6 from 2768896 to 3457024 tid 3 2020-06-12T02:39:10.211 INFO:tasks.rados.rados.0.smithi107.stdout:10308: writing smithi10729407-6 from 3457024 to 3645440 tid 4 2020-06-12T02:39:10.211 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:39:10.211 INFO:tasks.rados.rados.0.smithi107.stdout:10309: seq_num 10308 ranges {5087232=647168,5734400=614400,6348800=442368,6791168=548864} 2020-06-12T02:39:10.211 INFO:tasks.rados.rados.0.smithi107.stdout:10309: writing smithi10729407-7 from 5087232 to 5734400 tid 1 2020-06-12T02:39:10.212 INFO:tasks.rados.rados.0.smithi107.stdout:10309: writing smithi10729407-7 from 5734400 to 6348800 tid 2 2020-06-12T02:39:10.212 INFO:tasks.rados.rados.0.smithi107.stdout:10309: writing smithi10729407-7 from 6348800 to 6791168 tid 3 2020-06-12T02:39:10.212 INFO:tasks.rados.rados.0.smithi107.stdout:10309: writing smithi10729407-7 from 6791168 to 7340032 tid 4 2020-06-12T02:39:10.212 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:39:10.213 INFO:tasks.rados.rados.0.smithi107.stdout:10310: seq_num 10309 ranges {0=663552,663552=548864,1212416=647168,1859584=409600} 2020-06-12T02:39:10.213 INFO:tasks.rados.rados.0.smithi107.stdout:10310: writing smithi10729407-13 from 0 to 663552 tid 1 2020-06-12T02:39:10.213 INFO:tasks.rados.rados.0.smithi107.stdout:10310: writing smithi10729407-13 from 663552 to 1212416 tid 2 2020-06-12T02:39:10.214 INFO:tasks.rados.rados.0.smithi107.stdout:10310: writing smithi10729407-13 from 1212416 to 1859584 tid 3 2020-06-12T02:39:10.215 INFO:tasks.rados.rados.0.smithi107.stdout:10310: writing smithi10729407-13 from 1859584 to 2269184 tid 4 2020-06-12T02:39:10.215 INFO:tasks.rados.rados.0.smithi107.stdout:10311: delete oid 1 current snap is 0 2020-06-12T02:39:10.215 INFO:tasks.rados.rados.0.smithi107.stdout:10306: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:10.215 INFO:tasks.rados.rados.0.smithi107.stdout:10306: finishing write tid 2 to smithi10729407-12 2020-06-12T02:39:10.215 INFO:tasks.rados.rados.0.smithi107.stdout:10306: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:10.216 INFO:tasks.rados.rados.0.smithi107.stdout:10306: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:10.216 INFO:tasks.rados.rados.0.smithi107.stdout:10306: finishing write tid 6 to smithi10729407-12 2020-06-12T02:39:10.216 INFO:tasks.rados.rados.0.smithi107.stdout:10306: finishing write tid 7 to smithi10729407-12 2020-06-12T02:39:10.216 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1900 (ObjNum 10305 snap 0 seq_num 10305) dirty exists 2020-06-12T02:39:10.216 INFO:tasks.rados.rados.0.smithi107.stdout:10306: left oid 12 (ObjNum 10305 snap 0 seq_num 10305) 2020-06-12T02:39:10.217 INFO:tasks.rados.rados.0.smithi107.stdout:10308: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:10.217 INFO:tasks.rados.rados.0.smithi107.stdout:10308: finishing write tid 2 to smithi10729407-6 2020-06-12T02:39:10.217 INFO:tasks.rados.rados.0.smithi107.stdout:10308: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:10.217 INFO:tasks.rados.rados.0.smithi107.stdout:10308: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:10.217 INFO:tasks.rados.rados.0.smithi107.stdout:10308: finishing write tid 6 to smithi10729407-6 2020-06-12T02:39:10.218 INFO:tasks.rados.rados.0.smithi107.stdout:10308: finishing write tid 7 to smithi10729407-6 2020-06-12T02:39:10.218 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1861 (ObjNum 10307 snap 0 seq_num 10307) dirty exists 2020-06-12T02:39:10.218 INFO:tasks.rados.rados.0.smithi107.stdout:10308: left oid 6 (ObjNum 10307 snap 0 seq_num 10307) 2020-06-12T02:39:10.218 INFO:tasks.rados.rados.0.smithi107.stdout:10309: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:10.218 INFO:tasks.rados.rados.0.smithi107.stdout:10309: finishing write tid 2 to smithi10729407-7 2020-06-12T02:39:10.218 INFO:tasks.rados.rados.0.smithi107.stdout:10309: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:10.219 INFO:tasks.rados.rados.0.smithi107.stdout:10309: finishing write tid 4 to smithi10729407-7 2020-06-12T02:39:10.219 INFO:tasks.rados.rados.0.smithi107.stdout:10309: finishing write tid 6 to smithi10729407-7 2020-06-12T02:39:10.219 INFO:tasks.rados.rados.0.smithi107.stdout:10309: finishing write tid 7 to smithi10729407-7 2020-06-12T02:39:10.219 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1731 (ObjNum 10308 snap 0 seq_num 10308) dirty exists 2020-06-12T02:39:10.219 INFO:tasks.rados.rados.0.smithi107.stdout:10309: left oid 7 (ObjNum 10308 snap 0 seq_num 10308) 2020-06-12T02:39:10.219 INFO:tasks.rados.rados.0.smithi107.stdout:10310: finishing write tid 1 to smithi10729407-13 2020-06-12T02:39:10.220 INFO:tasks.rados.rados.0.smithi107.stdout:10306: done (4 left) 2020-06-12T02:39:10.220 INFO:tasks.rados.rados.0.smithi107.stdout:10308: done (3 left) 2020-06-12T02:39:10.220 INFO:tasks.rados.rados.0.smithi107.stdout:10309: done (2 left) 2020-06-12T02:39:10.220 INFO:tasks.rados.rados.0.smithi107.stdout:10311: done (1 left) 2020-06-12T02:39:10.220 INFO:tasks.rados.rados.0.smithi107.stdout:10312: delete oid 10 current snap is 0 2020-06-12T02:39:10.221 INFO:tasks.rados.rados.0.smithi107.stdout:10312: done (1 left) 2020-06-12T02:39:10.221 INFO:tasks.rados.rados.0.smithi107.stdout:10313: delete oid 12 current snap is 0 2020-06-12T02:39:10.221 INFO:tasks.rados.rados.0.smithi107.stdout:10313: done (1 left) 2020-06-12T02:39:10.221 INFO:tasks.rados.rados.0.smithi107.stdout:10314: delete oid 20 current snap is 0 2020-06-12T02:39:10.221 INFO:tasks.rados.rados.0.smithi107.stdout:10310: finishing write tid 2 to smithi10729407-13 2020-06-12T02:39:10.221 INFO:tasks.rados.rados.0.smithi107.stdout:10310: finishing write tid 3 to smithi10729407-13 2020-06-12T02:39:10.222 INFO:tasks.rados.rados.0.smithi107.stdout:10310: finishing write tid 4 to smithi10729407-13 2020-06-12T02:39:10.222 INFO:tasks.rados.rados.0.smithi107.stdout:10310: finishing write tid 6 to smithi10729407-13 2020-06-12T02:39:10.222 INFO:tasks.rados.rados.0.smithi107.stdout:10314: done (1 left) 2020-06-12T02:39:10.222 INFO:tasks.rados.rados.0.smithi107.stdout:10315: delete oid 16 current snap is 0 2020-06-12T02:39:10.222 INFO:tasks.rados.rados.0.smithi107.stdout:10310: finishing write tid 7 to smithi10729407-13 2020-06-12T02:39:10.223 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1733 (ObjNum 10309 snap 0 seq_num 10309) dirty exists 2020-06-12T02:39:10.223 INFO:tasks.rados.rados.0.smithi107.stdout:10310: left oid 13 (ObjNum 10309 snap 0 seq_num 10309) 2020-06-12T02:39:10.223 INFO:tasks.rados.rados.0.smithi107.stdout:10310: done (1 left) 2020-06-12T02:39:10.223 INFO:tasks.rados.rados.0.smithi107.stdout:10315: done (0 left) 2020-06-12T02:39:10.223 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:39:10.223 INFO:tasks.rados.rados.0.smithi107.stdout:10316: seq_num 10315 ranges {3645440=761856,4407296=655360,5062656=647168,5709824=303104} 2020-06-12T02:39:10.224 INFO:tasks.rados.rados.0.smithi107.stdout:10316: writing smithi10729407-6 from 3645440 to 4407296 tid 1 2020-06-12T02:39:10.224 INFO:tasks.rados.rados.0.smithi107.stdout:10316: writing smithi10729407-6 from 4407296 to 5062656 tid 2 2020-06-12T02:39:10.224 INFO:tasks.rados.rados.0.smithi107.stdout:10316: writing smithi10729407-6 from 5062656 to 5709824 tid 3 2020-06-12T02:39:10.224 INFO:tasks.rados.rados.0.smithi107.stdout:10316: writing smithi10729407-6 from 5709824 to 6012928 tid 4 2020-06-12T02:39:10.224 INFO:tasks.rados.rados.0.smithi107.stdout:10317: delete oid 17 current snap is 0 2020-06-12T02:39:10.225 INFO:tasks.rados.rados.0.smithi107.stdout:10316: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:10.225 INFO:tasks.rados.rados.0.smithi107.stdout:10317: done (1 left) 2020-06-12T02:39:10.225 INFO:tasks.rados.rados.0.smithi107.stdout:10318: delete oid 11 current snap is 0 2020-06-12T02:39:10.225 INFO:tasks.rados.rados.0.smithi107.stdout:10318: done (1 left) 2020-06-12T02:39:10.225 INFO:tasks.rados.rados.0.smithi107.stdout:10319: delete oid 16 current snap is 0 2020-06-12T02:39:10.226 INFO:tasks.rados.rados.0.smithi107.stdout:10316: finishing write tid 2 to smithi10729407-6 2020-06-12T02:39:10.226 INFO:tasks.rados.rados.0.smithi107.stdout:10319: done (1 left) 2020-06-12T02:39:10.227 INFO:tasks.rados.rados.0.smithi107.stdout:10320: delete oid 11 current snap is 0 2020-06-12T02:39:10.227 INFO:tasks.rados.rados.0.smithi107.stdout:10316: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:10.227 INFO:tasks.rados.rados.0.smithi107.stdout:10316: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:10.227 INFO:tasks.rados.rados.0.smithi107.stdout:10320: done (1 left) 2020-06-12T02:39:10.228 INFO:tasks.rados.rados.0.smithi107.stdout:10321: delete oid 16 current snap is 0 2020-06-12T02:39:10.228 INFO:tasks.rados.rados.0.smithi107.stdout:10316: finishing write tid 6 to smithi10729407-6 2020-06-12T02:39:10.228 INFO:tasks.rados.rados.0.smithi107.stdout:10316: finishing write tid 7 to smithi10729407-6 2020-06-12T02:39:10.228 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1867 (ObjNum 10315 snap 0 seq_num 10315) dirty exists 2020-06-12T02:39:10.228 INFO:tasks.rados.rados.0.smithi107.stdout:10316: left oid 6 (ObjNum 10315 snap 0 seq_num 10315) 2020-06-12T02:39:10.229 INFO:tasks.rados.rados.0.smithi107.stdout:10316: done (1 left) 2020-06-12T02:39:10.229 INFO:tasks.rados.rados.0.smithi107.stdout:10321: done (0 left) 2020-06-12T02:39:10.229 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:39:10.229 INFO:tasks.rados.rados.0.smithi107.stdout:10322: seq_num 10321 ranges {0=57344} 2020-06-12T02:39:10.229 INFO:tasks.rados.rados.0.smithi107.stdout:10322: writing smithi10729407-2 from 0 to 57344 tid 1 2020-06-12T02:39:10.230 INFO:tasks.rados.rados.0.smithi107.stdout:10323: delete oid 6 current snap is 0 2020-06-12T02:39:10.230 INFO:tasks.rados.rados.0.smithi107.stdout:10323: done (1 left) 2020-06-12T02:39:10.230 INFO:tasks.rados.rados.0.smithi107.stdout:10324: delete oid 19 current snap is 0 2020-06-12T02:39:10.230 INFO:tasks.rados.rados.0.smithi107.stdout:10322: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:10.231 INFO:tasks.rados.rados.0.smithi107.stdout:10322: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:10.231 INFO:tasks.rados.rados.0.smithi107.stdout:10324: done (1 left) 2020-06-12T02:39:10.231 INFO:tasks.rados.rados.0.smithi107.stdout:10325: delete oid 13 current snap is 0 2020-06-12T02:39:10.231 INFO:tasks.rados.rados.0.smithi107.stdout:10322: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:10.231 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 972 (ObjNum 10321 snap 0 seq_num 10321) dirty exists 2020-06-12T02:39:10.232 INFO:tasks.rados.rados.0.smithi107.stdout:10322: left oid 2 (ObjNum 10321 snap 0 seq_num 10321) 2020-06-12T02:39:10.232 INFO:tasks.rados.rados.0.smithi107.stdout:10322: done (1 left) 2020-06-12T02:39:10.232 INFO:tasks.rados.rados.0.smithi107.stdout:10325: done (0 left) 2020-06-12T02:39:10.232 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:39:10.232 INFO:tasks.rados.rados.0.smithi107.stdout:10326: seq_num 10325 ranges {0=131072} 2020-06-12T02:39:10.232 INFO:tasks.rados.rados.0.smithi107.stdout:10326: writing smithi10729407-16 from 0 to 131072 tid 1 2020-06-12T02:39:10.233 INFO:tasks.rados.rados.0.smithi107.stdout:10327: delete oid 5 current snap is 0 2020-06-12T02:39:10.233 INFO:tasks.rados.rados.0.smithi107.stdout:10327: done (1 left) 2020-06-12T02:39:10.233 INFO:tasks.rados.rados.0.smithi107.stdout:10328: delete oid 2 current snap is 0 2020-06-12T02:39:10.234 INFO:tasks.rados.rados.0.smithi107.stdout:10328: done (1 left) 2020-06-12T02:39:10.234 INFO:tasks.rados.rados.0.smithi107.stdout:10329: delete oid 6 current snap is 0 2020-06-12T02:39:10.234 INFO:tasks.rados.rados.0.smithi107.stdout:10326: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:10.234 INFO:tasks.rados.rados.0.smithi107.stdout:10329: done (1 left) 2020-06-12T02:39:10.234 INFO:tasks.rados.rados.0.smithi107.stdout:10330: delete oid 18 current snap is 0 2020-06-12T02:39:10.234 INFO:tasks.rados.rados.0.smithi107.stdout:10326: finishing write tid 3 to smithi10729407-16 2020-06-12T02:39:10.235 INFO:tasks.rados.rados.0.smithi107.stdout:10326: finishing write tid 4 to smithi10729407-16 2020-06-12T02:39:10.235 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1843 (ObjNum 10325 snap 0 seq_num 10325) dirty exists 2020-06-12T02:39:10.235 INFO:tasks.rados.rados.0.smithi107.stdout:10326: left oid 16 (ObjNum 10325 snap 0 seq_num 10325) 2020-06-12T02:39:10.235 INFO:tasks.rados.rados.0.smithi107.stdout:10326: done (1 left) 2020-06-12T02:39:10.235 INFO:tasks.rados.rados.0.smithi107.stdout:10330: done (0 left) 2020-06-12T02:39:10.236 INFO:tasks.rados.rados.0.smithi107.stdout:10331: delete oid 10 current snap is 0 2020-06-12T02:39:10.236 INFO:tasks.rados.rados.0.smithi107.stdout:10331: done (0 left) 2020-06-12T02:39:10.236 INFO:tasks.rados.rados.0.smithi107.stdout:10332: delete oid 4 current snap is 0 2020-06-12T02:39:10.236 INFO:tasks.rados.rados.0.smithi107.stdout:10332: done (0 left) 2020-06-12T02:39:10.236 INFO:tasks.rados.rados.0.smithi107.stdout:10333: delete oid 9 current snap is 0 2020-06-12T02:39:10.236 INFO:tasks.rados.rados.0.smithi107.stdout:10333: done (0 left) 2020-06-12T02:39:10.237 INFO:tasks.rados.rados.0.smithi107.stdout:10334: delete oid 20 current snap is 0 2020-06-12T02:39:10.237 INFO:tasks.rados.rados.0.smithi107.stdout:10334: done (0 left) 2020-06-12T02:39:10.237 INFO:tasks.rados.rados.0.smithi107.stdout:10335: delete oid 11 current snap is 0 2020-06-12T02:39:10.237 INFO:tasks.rados.rados.0.smithi107.stdout:10335: done (0 left) 2020-06-12T02:39:10.237 INFO:tasks.rados.rados.0.smithi107.stdout:10336: delete oid 10 current snap is 0 2020-06-12T02:39:10.238 INFO:tasks.rados.rados.0.smithi107.stdout:10336: done (0 left) 2020-06-12T02:39:10.238 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:39:10.238 INFO:tasks.rados.rados.0.smithi107.stdout:10337: seq_num 10336 ranges {0=311296} 2020-06-12T02:39:10.238 INFO:tasks.rados.rados.0.smithi107.stdout:10337: writing smithi10729407-18 from 0 to 311296 tid 1 2020-06-12T02:39:10.238 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:39:10.238 INFO:tasks.rados.rados.0.smithi107.stdout:10338: seq_num 10337 ranges {0=327680} 2020-06-12T02:39:10.239 INFO:tasks.rados.rados.0.smithi107.stdout:10338: writing smithi10729407-2 from 0 to 327680 tid 1 2020-06-12T02:39:10.239 INFO:tasks.rados.rados.0.smithi107.stdout:10339: delete oid 9 current snap is 0 2020-06-12T02:39:10.239 INFO:tasks.rados.rados.0.smithi107.stdout:10337: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:10.239 INFO:tasks.rados.rados.0.smithi107.stdout:10338: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:10.239 INFO:tasks.rados.rados.0.smithi107.stdout:10339: done (2 left) 2020-06-12T02:39:10.239 INFO:tasks.rados.rados.0.smithi107.stdout:10340: delete oid 5 current snap is 0 2020-06-12T02:39:10.240 INFO:tasks.rados.rados.0.smithi107.stdout:10338: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:10.240 INFO:tasks.rados.rados.0.smithi107.stdout:10337: finishing write tid 3 to smithi10729407-18 2020-06-12T02:39:10.240 INFO:tasks.rados.rados.0.smithi107.stdout:10340: done (2 left) 2020-06-12T02:39:10.240 INFO:tasks.rados.rados.0.smithi107.stdout:10341: delete oid 7 current snap is 0 2020-06-12T02:39:10.240 INFO:tasks.rados.rados.0.smithi107.stdout:10338: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:10.241 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 975 (ObjNum 10337 snap 0 seq_num 10337) dirty exists 2020-06-12T02:39:10.241 INFO:tasks.rados.rados.0.smithi107.stdout:10338: left oid 2 (ObjNum 10337 snap 0 seq_num 10337) 2020-06-12T02:39:10.241 INFO:tasks.rados.rados.0.smithi107.stdout:10337: finishing write tid 4 to smithi10729407-18 2020-06-12T02:39:10.241 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1824 (ObjNum 10336 snap 0 seq_num 10336) dirty exists 2020-06-12T02:39:10.241 INFO:tasks.rados.rados.0.smithi107.stdout:10337: left oid 18 (ObjNum 10336 snap 0 seq_num 10336) 2020-06-12T02:39:10.241 INFO:tasks.rados.rados.0.smithi107.stdout:10337: done (2 left) 2020-06-12T02:39:10.242 INFO:tasks.rados.rados.0.smithi107.stdout:10338: done (1 left) 2020-06-12T02:39:10.242 INFO:tasks.rados.rados.0.smithi107.stdout:10341: done (0 left) 2020-06-12T02:39:10.242 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:39:10.242 INFO:tasks.rados.rados.0.smithi107.stdout:10342: seq_num 10341 ranges {0=393216} 2020-06-12T02:39:10.242 INFO:tasks.rados.rados.0.smithi107.stdout:10342: writing smithi10729407-7 from 0 to 393216 tid 1 2020-06-12T02:39:10.243 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:39:10.243 INFO:tasks.rados.rados.0.smithi107.stdout:10343: seq_num 10342 ranges {0=417792} 2020-06-12T02:39:10.243 INFO:tasks.rados.rados.0.smithi107.stdout:10343: writing smithi10729407-1 from 0 to 417792 tid 1 2020-06-12T02:39:10.243 INFO:tasks.rados.rados.0.smithi107.stdout:10344: delete oid 20 current snap is 0 2020-06-12T02:39:10.244 INFO:tasks.rados.rados.0.smithi107.stdout:10344: done (2 left) 2020-06-12T02:39:10.244 INFO:tasks.rados.rados.0.smithi107.stdout:10345: delete oid 4 current snap is 0 2020-06-12T02:39:10.244 INFO:tasks.rados.rados.0.smithi107.stdout:10342: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:10.244 INFO:tasks.rados.rados.0.smithi107.stdout:10343: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:10.244 INFO:tasks.rados.rados.0.smithi107.stdout:10343: finishing write tid 3 to smithi10729407-1 2020-06-12T02:39:10.244 INFO:tasks.rados.rados.0.smithi107.stdout:10345: done (2 left) 2020-06-12T02:39:10.245 INFO:tasks.rados.rados.0.smithi107.stdout:10346: delete oid 12 current snap is 0 2020-06-12T02:39:10.245 INFO:tasks.rados.rados.0.smithi107.stdout:10343: finishing write tid 4 to smithi10729407-1 2020-06-12T02:39:10.245 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1871 (ObjNum 10342 snap 0 seq_num 10342) dirty exists 2020-06-12T02:39:10.245 INFO:tasks.rados.rados.0.smithi107.stdout:10343: left oid 1 (ObjNum 10342 snap 0 seq_num 10342) 2020-06-12T02:39:10.245 INFO:tasks.rados.rados.0.smithi107.stdout:10342: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:10.246 INFO:tasks.rados.rados.0.smithi107.stdout:10342: finishing write tid 4 to smithi10729407-7 2020-06-12T02:39:10.246 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1734 (ObjNum 10341 snap 0 seq_num 10341) dirty exists 2020-06-12T02:39:10.246 INFO:tasks.rados.rados.0.smithi107.stdout:10342: left oid 7 (ObjNum 10341 snap 0 seq_num 10341) 2020-06-12T02:39:10.246 INFO:tasks.rados.rados.0.smithi107.stdout:10342: done (2 left) 2020-06-12T02:39:10.246 INFO:tasks.rados.rados.0.smithi107.stdout:10343: done (1 left) 2020-06-12T02:39:10.246 INFO:tasks.rados.rados.0.smithi107.stdout:10346: done (0 left) 2020-06-12T02:39:10.247 INFO:tasks.rados.rados.0.smithi107.stdout:10347: delete oid 6 current snap is 0 2020-06-12T02:39:10.247 INFO:tasks.rados.rados.0.smithi107.stdout:10347: done (0 left) 2020-06-12T02:39:10.247 INFO:tasks.rados.rados.0.smithi107.stdout:10348: delete oid 8 current snap is 0 2020-06-12T02:39:10.247 INFO:tasks.rados.rados.0.smithi107.stdout:10348: done (0 left) 2020-06-12T02:39:10.247 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:39:10.247 INFO:tasks.rados.rados.0.smithi107.stdout:10349: seq_num 10348 ranges {1744896=516096} 2020-06-12T02:39:10.248 INFO:tasks.rados.rados.0.smithi107.stdout:10349: writing smithi10729407-3 from 1744896 to 2260992 tid 1 2020-06-12T02:39:10.248 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:39:10.248 INFO:tasks.rados.rados.0.smithi107.stdout:10350: seq_num 10349 ranges {0=532480} 2020-06-12T02:39:10.248 INFO:tasks.rados.rados.0.smithi107.stdout:10350: writing smithi10729407-11 from 0 to 532480 tid 1 2020-06-12T02:39:10.248 INFO:tasks.rados.rados.0.smithi107.stdout:10351: delete oid 7 current snap is 0 2020-06-12T02:39:10.249 INFO:tasks.rados.rados.0.smithi107.stdout:10349: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:10.249 INFO:tasks.rados.rados.0.smithi107.stdout:10351: done (2 left) 2020-06-12T02:39:10.249 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:39:10.249 INFO:tasks.rados.rados.0.smithi107.stdout:10352: seq_num 10351 ranges {0=565248} 2020-06-12T02:39:10.249 INFO:tasks.rados.rados.0.smithi107.stdout:10352: writing smithi10729407-15 from 0 to 565248 tid 1 2020-06-12T02:39:10.249 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:39:10.250 INFO:tasks.rados.rados.0.smithi107.stdout:10353: seq_num 10352 ranges {0=581632} 2020-06-12T02:39:10.250 INFO:tasks.rados.rados.0.smithi107.stdout:10353: writing smithi10729407-12 from 0 to 581632 tid 1 2020-06-12T02:39:10.250 INFO:tasks.rados.rados.0.smithi107.stdout:10354: delete oid 7 current snap is 0 2020-06-12T02:39:10.250 INFO:tasks.rados.rados.0.smithi107.stdout:10349: finishing write tid 3 to smithi10729407-3 2020-06-12T02:39:10.250 INFO:tasks.rados.rados.0.smithi107.stdout:10350: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:10.251 INFO:tasks.rados.rados.0.smithi107.stdout:10349: finishing write tid 4 to smithi10729407-3 2020-06-12T02:39:10.251 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1845 (ObjNum 10348 snap 0 seq_num 10348) dirty exists 2020-06-12T02:39:10.251 INFO:tasks.rados.rados.0.smithi107.stdout:10349: left oid 3 (ObjNum 10348 snap 0 seq_num 10348) 2020-06-12T02:39:10.251 INFO:tasks.rados.rados.0.smithi107.stdout:10350: finishing write tid 3 to smithi10729407-11 2020-06-12T02:39:10.251 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:39:10.173+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:39:10.256 INFO:tasks.rados.rados.0.smithi107.stdout:10350: finishing write tid 4 to smithi10729407-11 2020-06-12T02:39:10.256 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1693 (ObjNum 10349 snap 0 seq_num 10349) dirty exists 2020-06-12T02:39:10.256 INFO:tasks.rados.rados.0.smithi107.stdout:10350: left oid 11 (ObjNum 10349 snap 0 seq_num 10349) 2020-06-12T02:39:10.256 INFO:tasks.rados.rados.0.smithi107.stdout:10352: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:10.256 INFO:tasks.rados.rados.0.smithi107.stdout:10349: done (4 left) 2020-06-12T02:39:10.257 INFO:tasks.rados.rados.0.smithi107.stdout:10350: done (3 left) 2020-06-12T02:39:10.257 INFO:tasks.rados.rados.0.smithi107.stdout:10354: done (2 left) 2020-06-12T02:39:10.257 INFO:tasks.rados.rados.0.smithi107.stdout:10355: delete oid 11 current snap is 0 2020-06-12T02:39:10.257 INFO:tasks.rados.rados.0.smithi107.stdout:10355: done (2 left) 2020-06-12T02:39:10.257 INFO:tasks.rados.rados.0.smithi107.stdout:10356: delete oid 4 current snap is 0 2020-06-12T02:39:10.258 INFO:tasks.rados.rados.0.smithi107.stdout:10353: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:10.258 INFO:tasks.rados.rados.0.smithi107.stdout:10352: finishing write tid 3 to smithi10729407-15 2020-06-12T02:39:10.258 INFO:tasks.rados.rados.0.smithi107.stdout:10352: finishing write tid 4 to smithi10729407-15 2020-06-12T02:39:10.258 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1912 (ObjNum 10351 snap 0 seq_num 10351) dirty exists 2020-06-12T02:39:10.258 INFO:tasks.rados.rados.0.smithi107.stdout:10352: left oid 15 (ObjNum 10351 snap 0 seq_num 10351) 2020-06-12T02:39:10.259 INFO:tasks.rados.rados.0.smithi107.stdout:10352: done (2 left) 2020-06-12T02:39:10.259 INFO:tasks.rados.rados.0.smithi107.stdout:10356: done (1 left) 2020-06-12T02:39:10.259 INFO:tasks.rados.rados.0.smithi107.stdout:10357: delete oid 10 current snap is 0 2020-06-12T02:39:10.259 INFO:tasks.rados.rados.0.smithi107.stdout:10353: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:10.259 INFO:tasks.rados.rados.0.smithi107.stdout:10357: done (1 left) 2020-06-12T02:39:10.259 INFO:tasks.rados.rados.0.smithi107.stdout:10358: delete oid 2 current snap is 0 2020-06-12T02:39:10.260 INFO:tasks.rados.rados.0.smithi107.stdout:10353: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:10.260 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1905 (ObjNum 10352 snap 0 seq_num 10352) dirty exists 2020-06-12T02:39:10.260 INFO:tasks.rados.rados.0.smithi107.stdout:10353: left oid 12 (ObjNum 10352 snap 0 seq_num 10352) 2020-06-12T02:39:10.260 INFO:tasks.rados.rados.0.smithi107.stdout:10353: done (1 left) 2020-06-12T02:39:10.260 INFO:tasks.rados.rados.0.smithi107.stdout:10358: done (0 left) 2020-06-12T02:39:10.261 INFO:tasks.rados.rados.0.smithi107.stdout:10359: delete oid 16 current snap is 0 2020-06-12T02:39:10.261 INFO:tasks.rados.rados.0.smithi107.stdout:10359: done (0 left) 2020-06-12T02:39:10.261 INFO:tasks.rados.rados.0.smithi107.stdout:10360: delete oid 14 current snap is 0 2020-06-12T02:39:10.261 INFO:tasks.rados.rados.0.smithi107.stdout:10360: done (0 left) 2020-06-12T02:39:10.261 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:39:10.261 INFO:tasks.rados.rados.0.smithi107.stdout:10361: seq_num 10360 ranges {0=712704} 2020-06-12T02:39:10.262 INFO:tasks.rados.rados.0.smithi107.stdout:10361: writing smithi10729407-20 from 0 to 712704 tid 1 2020-06-12T02:39:10.262 INFO:tasks.rados.rados.0.smithi107.stdout:10362: delete oid 14 current snap is 0 2020-06-12T02:39:10.262 INFO:tasks.rados.rados.0.smithi107.stdout:10362: done (1 left) 2020-06-12T02:39:10.262 INFO:tasks.rados.rados.0.smithi107.stdout:10363: delete oid 1 current snap is 0 2020-06-12T02:39:10.262 INFO:tasks.rados.rados.0.smithi107.stdout:10363: done (1 left) 2020-06-12T02:39:10.263 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:39:10.263 INFO:tasks.rados.rados.0.smithi107.stdout:10364: seq_num 10363 ranges {0=770048} 2020-06-12T02:39:10.263 INFO:tasks.rados.rados.0.smithi107.stdout:10364: writing smithi10729407-9 from 0 to 770048 tid 1 2020-06-12T02:39:10.263 INFO:tasks.rados.rados.0.smithi107.stdout:10365: delete oid 6 current snap is 0 2020-06-12T02:39:10.263 INFO:tasks.rados.rados.0.smithi107.stdout:10361: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:10.264 INFO:tasks.rados.rados.0.smithi107.stdout:10361: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:10.264 INFO:tasks.rados.rados.0.smithi107.stdout:10361: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:10.264 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 798 (ObjNum 10360 snap 0 seq_num 10360) dirty exists 2020-06-12T02:39:10.264 INFO:tasks.rados.rados.0.smithi107.stdout:10361: left oid 20 (ObjNum 10360 snap 0 seq_num 10360) 2020-06-12T02:39:10.264 INFO:tasks.rados.rados.0.smithi107.stdout:10361: done (2 left) 2020-06-12T02:39:10.265 INFO:tasks.rados.rados.0.smithi107.stdout:10365: done (1 left) 2020-06-12T02:39:10.265 INFO:tasks.rados.rados.0.smithi107.stdout:10366: delete oid 13 current snap is 0 2020-06-12T02:39:10.265 INFO:tasks.rados.rados.0.smithi107.stdout:10366: done (1 left) 2020-06-12T02:39:10.266 INFO:tasks.rados.rados.0.smithi107.stdout:10367: delete oid 6 current snap is 0 2020-06-12T02:39:10.266 INFO:tasks.rados.rados.0.smithi107.stdout:10364: finishing write tid 1 to smithi10729407-9 2020-06-12T02:39:10.266 INFO:tasks.rados.rados.0.smithi107.stdout:10364: finishing write tid 3 to smithi10729407-9 2020-06-12T02:39:10.266 INFO:tasks.rados.rados.0.smithi107.stdout:10367: done (1 left) 2020-06-12T02:39:10.266 INFO:tasks.rados.rados.0.smithi107.stdout:10368: delete oid 19 current snap is 0 2020-06-12T02:39:10.266 INFO:tasks.rados.rados.0.smithi107.stdout:10364: finishing write tid 4 to smithi10729407-9 2020-06-12T02:39:10.267 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1807 (ObjNum 10363 snap 0 seq_num 10363) dirty exists 2020-06-12T02:39:10.267 INFO:tasks.rados.rados.0.smithi107.stdout:10364: left oid 9 (ObjNum 10363 snap 0 seq_num 10363) 2020-06-12T02:39:10.267 INFO:tasks.rados.rados.0.smithi107.stdout:10364: done (1 left) 2020-06-12T02:39:10.267 INFO:tasks.rados.rados.0.smithi107.stdout:10368: done (0 left) 2020-06-12T02:39:10.267 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:39:10.268 INFO:tasks.rados.rados.0.smithi107.stdout:10369: seq_num 10368 ranges {311296=450560,761856=401408} 2020-06-12T02:39:10.268 INFO:tasks.rados.rados.0.smithi107.stdout:10369: writing smithi10729407-18 from 311296 to 761856 tid 1 2020-06-12T02:39:10.268 INFO:tasks.rados.rados.0.smithi107.stdout:10369: writing smithi10729407-18 from 761856 to 1163264 tid 2 2020-06-12T02:39:10.268 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:39:10.268 INFO:tasks.rados.rados.0.smithi107.stdout:10370: seq_num 10369 ranges {0=466944,466944=401408} 2020-06-12T02:39:10.268 INFO:tasks.rados.rados.0.smithi107.stdout:10370: writing smithi10729407-8 from 0 to 466944 tid 1 2020-06-12T02:39:10.269 INFO:tasks.rados.rados.0.smithi107.stdout:10370: writing smithi10729407-8 from 466944 to 868352 tid 2 2020-06-12T02:39:10.269 INFO:tasks.rados.rados.0.smithi107.stdout:10371: delete oid 3 current snap is 0 2020-06-12T02:39:10.269 INFO:tasks.rados.rados.0.smithi107.stdout:10369: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:10.269 INFO:tasks.rados.rados.0.smithi107.stdout:10369: finishing write tid 2 to smithi10729407-18 2020-06-12T02:39:10.271 INFO:tasks.rados.rados.0.smithi107.stdout:10369: finishing write tid 4 to smithi10729407-18 2020-06-12T02:39:10.271 INFO:tasks.rados.rados.0.smithi107.stdout:10371: done (2 left) 2020-06-12T02:39:10.271 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:39:10.272 INFO:tasks.rados.rados.0.smithi107.stdout:10372: seq_num 10371 ranges {0=499712,499712=401408} 2020-06-12T02:39:10.272 INFO:tasks.rados.rados.0.smithi107.stdout:10372: writing smithi10729407-3 from 0 to 499712 tid 1 2020-06-12T02:39:10.272 INFO:tasks.rados.rados.0.smithi107.stdout:10372: writing smithi10729407-3 from 499712 to 901120 tid 2 2020-06-12T02:39:10.272 INFO:tasks.rados.rados.0.smithi107.stdout:10373: delete oid 10 current snap is 0 2020-06-12T02:39:10.272 INFO:tasks.rados.rados.0.smithi107.stdout:10369: finishing write tid 5 to smithi10729407-18 2020-06-12T02:39:10.273 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1827 (ObjNum 10368 snap 0 seq_num 10368) dirty exists 2020-06-12T02:39:10.273 INFO:tasks.rados.rados.0.smithi107.stdout:10369: left oid 18 (ObjNum 10368 snap 0 seq_num 10368) 2020-06-12T02:39:10.273 INFO:tasks.rados.rados.0.smithi107.stdout:10370: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:10.273 INFO:tasks.rados.rados.0.smithi107.stdout:10370: finishing write tid 2 to smithi10729407-8 2020-06-12T02:39:10.273 INFO:tasks.rados.rados.0.smithi107.stdout:10370: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:10.274 INFO:tasks.rados.rados.0.smithi107.stdout:10370: finishing write tid 5 to smithi10729407-8 2020-06-12T02:39:10.274 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1915 (ObjNum 10369 snap 0 seq_num 10369) dirty exists 2020-06-12T02:39:10.274 INFO:tasks.rados.rados.0.smithi107.stdout:10370: left oid 8 (ObjNum 10369 snap 0 seq_num 10369) 2020-06-12T02:39:10.274 INFO:tasks.rados.rados.0.smithi107.stdout:10369: done (3 left) 2020-06-12T02:39:10.275 INFO:tasks.rados.rados.0.smithi107.stdout:10370: done (2 left) 2020-06-12T02:39:10.275 INFO:tasks.rados.rados.0.smithi107.stdout:10373: done (1 left) 2020-06-12T02:39:10.275 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:39:10.275 INFO:tasks.rados.rados.0.smithi107.stdout:10374: seq_num 10373 ranges {565248=532480,1097728=401408} 2020-06-12T02:39:10.277 INFO:tasks.rados.rados.0.smithi107.stdout:10374: writing smithi10729407-15 from 565248 to 1097728 tid 1 2020-06-12T02:39:10.277 INFO:tasks.rados.rados.0.smithi107.stdout:10374: writing smithi10729407-15 from 1097728 to 1499136 tid 2 2020-06-12T02:39:10.277 INFO:tasks.rados.rados.0.smithi107.stdout:10375: delete oid 18 current snap is 0 2020-06-12T02:39:10.277 INFO:tasks.rados.rados.0.smithi107.stdout:10372: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:10.277 INFO:tasks.rados.rados.0.smithi107.stdout:10372: finishing write tid 2 to smithi10729407-3 2020-06-12T02:39:10.278 INFO:tasks.rados.rados.0.smithi107.stdout:10372: finishing write tid 4 to smithi10729407-3 2020-06-12T02:39:10.278 INFO:tasks.rados.rados.0.smithi107.stdout:10372: finishing write tid 5 to smithi10729407-3 2020-06-12T02:39:10.278 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1850 (ObjNum 10371 snap 0 seq_num 10371) dirty exists 2020-06-12T02:39:10.278 INFO:tasks.rados.rados.0.smithi107.stdout:10372: left oid 3 (ObjNum 10371 snap 0 seq_num 10371) 2020-06-12T02:39:10.279 INFO:tasks.rados.rados.0.smithi107.stdout:10372: done (2 left) 2020-06-12T02:39:10.279 INFO:tasks.rados.rados.0.smithi107.stdout:10375: done (1 left) 2020-06-12T02:39:10.279 INFO:tasks.rados.rados.0.smithi107.stdout:10376: delete oid 14 current snap is 0 2020-06-12T02:39:10.279 INFO:tasks.rados.rados.0.smithi107.stdout:10376: done (1 left) 2020-06-12T02:39:10.280 INFO:tasks.rados.rados.0.smithi107.stdout:10377: delete oid 18 current snap is 0 2020-06-12T02:39:10.281 INFO:tasks.rados.rados.0.smithi107.stdout:10377: done (1 left) 2020-06-12T02:39:10.282 INFO:tasks.rados.rados.0.smithi107.stdout:10378: delete oid 6 current snap is 0 2020-06-12T02:39:10.283 INFO:tasks.rados.rados.0.smithi107.stdout:10374: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:10.285 INFO:tasks.rados.rados.0.smithi107.stdout:10374: finishing write tid 2 to smithi10729407-15 2020-06-12T02:39:10.285 INFO:tasks.rados.rados.0.smithi107.stdout:10378: done (1 left) 2020-06-12T02:39:10.286 INFO:tasks.rados.rados.0.smithi107.stdout:10379: delete oid 13 current snap is 0 2020-06-12T02:39:10.286 INFO:tasks.rados.rados.0.smithi107.stdout:10374: finishing write tid 4 to smithi10729407-15 2020-06-12T02:39:10.287 INFO:tasks.rados.rados.0.smithi107.stdout:10379: done (1 left) 2020-06-12T02:39:10.287 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:39:10.288 INFO:tasks.rados.rados.0.smithi107.stdout:10380: seq_num 10379 ranges {712704=630784,1343488=401408} 2020-06-12T02:39:10.294 INFO:tasks.rados.rados.0.smithi107.stdout:10380: writing smithi10729407-20 from 712704 to 1343488 tid 1 2020-06-12T02:39:10.298 INFO:tasks.rados.rados.0.smithi107.stdout:10380: writing smithi10729407-20 from 1343488 to 1744896 tid 2 2020-06-12T02:39:10.298 INFO:tasks.rados.rados.0.smithi107.stdout:10381: delete oid 8 current snap is 0 2020-06-12T02:39:10.298 INFO:tasks.rados.rados.0.smithi107.stdout:10374: finishing write tid 5 to smithi10729407-15 2020-06-12T02:39:10.298 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1918 (ObjNum 10373 snap 0 seq_num 10373) dirty exists 2020-06-12T02:39:10.299 INFO:tasks.rados.rados.0.smithi107.stdout:10374: left oid 15 (ObjNum 10373 snap 0 seq_num 10373) 2020-06-12T02:39:10.300 INFO:tasks.rados.rados.0.smithi107.stdout:10374: done (2 left) 2020-06-12T02:39:10.300 INFO:tasks.rados.rados.0.smithi107.stdout:10381: done (1 left) 2020-06-12T02:39:10.300 INFO:tasks.rados.rados.0.smithi107.stdout:10382: delete oid 18 current snap is 0 2020-06-12T02:39:10.302 INFO:tasks.rados.rados.0.smithi107.stdout:10380: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:10.303 INFO:tasks.rados.rados.0.smithi107.stdout:10382: done (1 left) 2020-06-12T02:39:10.303 INFO:tasks.rados.rados.0.smithi107.stdout:10383: delete oid 6 current snap is 0 2020-06-12T02:39:10.308 INFO:tasks.rados.rados.0.smithi107.stdout:10383: done (1 left) 2020-06-12T02:39:10.308 INFO:tasks.rados.rados.0.smithi107.stdout:10384: delete oid 12 current snap is 0 2020-06-12T02:39:10.309 INFO:tasks.rados.rados.0.smithi107.stdout:10380: finishing write tid 2 to smithi10729407-20 2020-06-12T02:39:10.309 INFO:tasks.rados.rados.0.smithi107.stdout:10380: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:10.310 INFO:tasks.rados.rados.0.smithi107.stdout:10380: finishing write tid 5 to smithi10729407-20 2020-06-12T02:39:10.310 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 801 (ObjNum 10379 snap 0 seq_num 10379) dirty exists 2020-06-12T02:39:10.311 INFO:tasks.rados.rados.0.smithi107.stdout:10380: left oid 20 (ObjNum 10379 snap 0 seq_num 10379) 2020-06-12T02:39:10.311 INFO:tasks.rados.rados.0.smithi107.stdout:10380: done (1 left) 2020-06-12T02:39:10.311 INFO:tasks.rados.rados.0.smithi107.stdout:10384: done (0 left) 2020-06-12T02:39:10.312 INFO:tasks.rados.rados.0.smithi107.stdout:10385: delete oid 6 current snap is 0 2020-06-12T02:39:10.314 INFO:tasks.rados.rados.0.smithi107.stdout:10385: done (0 left) 2020-06-12T02:39:10.314 INFO:tasks.rados.rados.0.smithi107.stdout:10386: delete oid 3 current snap is 0 2020-06-12T02:39:10.316 INFO:tasks.rados.rados.0.smithi107.stdout:10386: done (0 left) 2020-06-12T02:39:10.317 INFO:tasks.rados.rados.0.smithi107.stdout:10387: delete oid 19 current snap is 0 2020-06-12T02:39:10.317 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:39:10.315+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:39:10.319 INFO:tasks.rados.rados.0.smithi107.stdout:10387: done (0 left) 2020-06-12T02:39:10.319 INFO:tasks.rados.rados.0.smithi107.stdout:10388: delete oid 17 current snap is 0 2020-06-12T02:39:10.322 INFO:tasks.rados.rados.0.smithi107.stdout:10388: done (0 left) 2020-06-12T02:39:10.322 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:39:10.322 INFO:tasks.rados.rados.0.smithi107.stdout:10389: seq_num 10388 ranges {0=786432,786432=401408} 2020-06-12T02:39:10.326 INFO:tasks.rados.rados.0.smithi107.stdout:10389: writing smithi10729407-10 from 0 to 786432 tid 1 2020-06-12T02:39:10.328 INFO:tasks.rados.rados.0.smithi107.stdout:10389: writing smithi10729407-10 from 786432 to 1187840 tid 2 2020-06-12T02:39:10.329 INFO:tasks.rados.rados.0.smithi107.stdout:10390: delete oid 1 current snap is 0 2020-06-12T02:39:10.332 INFO:tasks.rados.rados.0.smithi107.stdout:10390: done (1 left) 2020-06-12T02:39:10.332 INFO:tasks.rados.rados.0.smithi107.stdout:10391: delete oid 17 current snap is 0 2020-06-12T02:39:10.335 INFO:tasks.rados.rados.0.smithi107.stdout:10391: done (1 left) 2020-06-12T02:39:10.336 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:39:10.336 INFO:tasks.rados.rados.0.smithi107.stdout:10392: seq_num 10391 ranges {1499136=434176,1933312=761856,2695168=40960} 2020-06-12T02:39:10.346 INFO:tasks.rados.rados.0.smithi107.stdout:10392: writing smithi10729407-15 from 1499136 to 1933312 tid 1 2020-06-12T02:39:10.351 INFO:tasks.rados.rados.0.smithi107.stdout:10392: writing smithi10729407-15 from 1933312 to 2695168 tid 2 2020-06-12T02:39:10.352 INFO:tasks.rados.rados.0.smithi107.stdout:10392: writing smithi10729407-15 from 2695168 to 2736128 tid 3 2020-06-12T02:39:10.352 INFO:tasks.rados.rados.0.smithi107.stdout:10393: delete oid 5 current snap is 0 2020-06-12T02:39:10.352 INFO:tasks.rados.rados.0.smithi107.stdout:10389: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:10.352 INFO:tasks.rados.rados.0.smithi107.stdout:10389: finishing write tid 2 to smithi10729407-10 2020-06-12T02:39:10.353 INFO:tasks.rados.rados.0.smithi107.stdout:10389: finishing write tid 4 to smithi10729407-10 2020-06-12T02:39:10.353 INFO:tasks.rados.rados.0.smithi107.stdout:10389: finishing write tid 5 to smithi10729407-10 2020-06-12T02:39:10.353 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 749 (ObjNum 10388 snap 0 seq_num 10388) dirty exists 2020-06-12T02:39:10.353 INFO:tasks.rados.rados.0.smithi107.stdout:10389: left oid 10 (ObjNum 10388 snap 0 seq_num 10388) 2020-06-12T02:39:10.354 INFO:tasks.rados.rados.0.smithi107.stdout:10392: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:10.354 INFO:tasks.rados.rados.0.smithi107.stdout:10389: done (2 left) 2020-06-12T02:39:10.354 INFO:tasks.rados.rados.0.smithi107.stdout:10393: done (1 left) 2020-06-12T02:39:10.355 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:39:10.355 INFO:tasks.rados.rados.0.smithi107.stdout:10394: seq_num 10393 ranges {770048=466944,1236992=581632,1818624=221184} 2020-06-12T02:39:10.360 INFO:tasks.rados.rados.0.smithi107.stdout:10394: writing smithi10729407-9 from 770048 to 1236992 tid 1 2020-06-12T02:39:10.364 INFO:tasks.rados.rados.0.smithi107.stdout:10394: writing smithi10729407-9 from 1236992 to 1818624 tid 2 2020-06-12T02:39:10.366 INFO:tasks.rados.rados.0.smithi107.stdout:10394: writing smithi10729407-9 from 1818624 to 2039808 tid 3 2020-06-12T02:39:10.366 INFO:tasks.rados.rados.0.smithi107.stdout:10392: finishing write tid 2 to smithi10729407-15 2020-06-12T02:39:10.366 INFO:tasks.rados.rados.0.smithi107.stdout:10392: finishing write tid 3 to smithi10729407-15 2020-06-12T02:39:10.366 INFO:tasks.rados.rados.0.smithi107.stdout:10395: delete oid 4 current snap is 0 2020-06-12T02:39:10.367 INFO:tasks.rados.rados.0.smithi107.stdout:10392: finishing write tid 5 to smithi10729407-15 2020-06-12T02:39:10.369 INFO:tasks.rados.rados.0.smithi107.stdout:10392: finishing write tid 6 to smithi10729407-15 2020-06-12T02:39:10.370 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1923 (ObjNum 10391 snap 0 seq_num 10391) dirty exists 2020-06-12T02:39:10.370 INFO:tasks.rados.rados.0.smithi107.stdout:10392: left oid 15 (ObjNum 10391 snap 0 seq_num 10391) 2020-06-12T02:39:10.372 INFO:tasks.rados.rados.0.smithi107.stdout:10394: finishing write tid 1 to smithi10729407-9 2020-06-12T02:39:10.373 INFO:tasks.rados.rados.0.smithi107.stdout:10394: finishing write tid 2 to smithi10729407-9 2020-06-12T02:39:10.373 INFO:tasks.rados.rados.0.smithi107.stdout:10394: finishing write tid 3 to smithi10729407-9 2020-06-12T02:39:10.374 INFO:tasks.rados.rados.0.smithi107.stdout:10392: done (2 left) 2020-06-12T02:39:10.374 INFO:tasks.rados.rados.0.smithi107.stdout:10395: done (1 left) 2020-06-12T02:39:10.374 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:39:10.374 INFO:tasks.rados.rados.0.smithi107.stdout:10396: seq_num 10395 ranges {1744896=499712,2244608=745472,2990080=57344} 2020-06-12T02:39:10.377 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:39:10.374+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:39:10.386 INFO:tasks.rados.rados.0.smithi107.stdout:10396: writing smithi10729407-20 from 1744896 to 2244608 tid 1 2020-06-12T02:39:10.391 INFO:tasks.rados.rados.0.smithi107.stdout:10396: writing smithi10729407-20 from 2244608 to 2990080 tid 2 2020-06-12T02:39:10.391 INFO:tasks.rados.rados.0.smithi107.stdout:10396: writing smithi10729407-20 from 2990080 to 3047424 tid 3 2020-06-12T02:39:10.391 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:39:10.392 INFO:tasks.rados.rados.0.smithi107.stdout:10397: seq_num 10396 ranges {0=516096,516096=630784,1146880=172032} 2020-06-12T02:39:10.394 INFO:tasks.rados.rados.0.smithi107.stdout:10397: writing smithi10729407-1 from 0 to 516096 tid 1 2020-06-12T02:39:10.396 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:39:10.396 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:39:10.396 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2020-06-12T02:39:10.396 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:39:10.399 INFO:tasks.rados.rados.0.smithi107.stdout:10397: writing smithi10729407-1 from 516096 to 1146880 tid 2 2020-06-12T02:39:10.400 INFO:tasks.rados.rados.0.smithi107.stdout:10397: writing smithi10729407-1 from 1146880 to 1318912 tid 3 2020-06-12T02:39:10.400 INFO:tasks.rados.rados.0.smithi107.stdout:10398: delete oid 19 current snap is 0 2020-06-12T02:39:10.400 INFO:tasks.rados.rados.0.smithi107.stdout:10394: finishing write tid 5 to smithi10729407-9 2020-06-12T02:39:10.400 INFO:tasks.rados.rados.0.smithi107.stdout:10394: finishing write tid 6 to smithi10729407-9 2020-06-12T02:39:10.401 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1811 (ObjNum 10393 snap 0 seq_num 10393) dirty exists 2020-06-12T02:39:10.401 INFO:tasks.rados.rados.0.smithi107.stdout:10394: left oid 9 (ObjNum 10393 snap 0 seq_num 10393) 2020-06-12T02:39:10.401 INFO:tasks.rados.rados.0.smithi107.stdout:10396: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:10.401 INFO:tasks.rados.rados.0.smithi107.stdout:10396: finishing write tid 2 to smithi10729407-20 2020-06-12T02:39:10.403 INFO:tasks.rados.rados.0.smithi107.stdout:10396: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:10.404 INFO:tasks.rados.rados.0.smithi107.stdout:10396: finishing write tid 5 to smithi10729407-20 2020-06-12T02:39:10.407 INFO:tasks.rados.rados.0.smithi107.stdout:10396: finishing write tid 6 to smithi10729407-20 2020-06-12T02:39:10.407 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 805 (ObjNum 10395 snap 0 seq_num 10395) dirty exists 2020-06-12T02:39:10.407 INFO:tasks.rados.rados.0.smithi107.stdout:10396: left oid 20 (ObjNum 10395 snap 0 seq_num 10395) 2020-06-12T02:39:10.409 INFO:tasks.rados.rados.0.smithi107.stdout:10394: done (3 left) 2020-06-12T02:39:10.410 INFO:tasks.rados.rados.0.smithi107.stdout:10396: done (2 left) 2020-06-12T02:39:10.410 INFO:tasks.rados.rados.0.smithi107.stdout:10398: done (1 left) 2020-06-12T02:39:10.410 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:39:10.411 INFO:tasks.rados.rados.0.smithi107.stdout:10399: seq_num 10398 ranges {2039808=548864,2588672=802816} 2020-06-12T02:39:10.412 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-06-12T02:39:10.424 INFO:tasks.rados.rados.0.smithi107.stdout:10399: writing smithi10729407-9 from 2039808 to 2588672 tid 1 2020-06-12T02:39:10.429 INFO:tasks.rados.rados.0.smithi107.stdout:10399: writing smithi10729407-9 from 2588672 to 3391488 tid 2 2020-06-12T02:39:10.429 INFO:tasks.rados.rados.0.smithi107.stdout:10400: delete oid 2 current snap is 0 2020-06-12T02:39:10.429 INFO:tasks.rados.rados.0.smithi107.stdout:10397: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:10.430 INFO:tasks.rados.rados.0.smithi107.stdout:10397: finishing write tid 2 to smithi10729407-1 2020-06-12T02:39:10.430 INFO:tasks.rados.rados.0.smithi107.stdout:10397: finishing write tid 3 to smithi10729407-1 2020-06-12T02:39:10.430 INFO:tasks.rados.rados.0.smithi107.stdout:10397: finishing write tid 5 to smithi10729407-1 2020-06-12T02:39:10.430 INFO:tasks.rados.rados.0.smithi107.stdout:10397: finishing write tid 6 to smithi10729407-1 2020-06-12T02:39:10.430 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1883 (ObjNum 10396 snap 0 seq_num 10396) dirty exists 2020-06-12T02:39:10.431 INFO:tasks.rados.rados.0.smithi107.stdout:10397: left oid 1 (ObjNum 10396 snap 0 seq_num 10396) 2020-06-12T02:39:10.431 INFO:tasks.rados.rados.0.smithi107.stdout:10399: finishing write tid 1 to smithi10729407-9 2020-06-12T02:39:10.431 INFO:tasks.rados.rados.0.smithi107.stdout:10397: done (2 left) 2020-06-12T02:39:10.431 INFO:tasks.rados.rados.0.smithi107.stdout:10400: done (1 left) 2020-06-12T02:39:10.432 INFO:tasks.rados.rados.0.smithi107.stdout:10401: delete oid 5 current snap is 0 2020-06-12T02:39:10.441 INFO:tasks.rados.rados.0.smithi107.stdout:10401: done (1 left) 2020-06-12T02:39:10.441 INFO:tasks.rados.rados.0.smithi107.stdout:10402: delete oid 8 current snap is 0 2020-06-12T02:39:10.441 INFO:tasks.rados.rados.0.smithi107.stdout:10399: finishing write tid 2 to smithi10729407-9 2020-06-12T02:39:10.441 INFO:tasks.rados.rados.0.smithi107.stdout:10399: finishing write tid 4 to smithi10729407-9 2020-06-12T02:39:10.441 INFO:tasks.rados.rados.0.smithi107.stdout:10399: finishing write tid 5 to smithi10729407-9 2020-06-12T02:39:10.442 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1814 (ObjNum 10398 snap 0 seq_num 10398) dirty exists 2020-06-12T02:39:10.442 INFO:tasks.rados.rados.0.smithi107.stdout:10399: left oid 9 (ObjNum 10398 snap 0 seq_num 10398) 2020-06-12T02:39:10.442 INFO:tasks.rados.rados.0.smithi107.stdout:10399: done (1 left) 2020-06-12T02:39:10.442 INFO:tasks.rados.rados.0.smithi107.stdout:10402: done (0 left) 2020-06-12T02:39:10.442 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:39:10.443 INFO:tasks.rados.rados.0.smithi107.stdout:10403: seq_num 10402 ranges {1318912=622592,1941504=786432,2727936=16384} 2020-06-12T02:39:10.452 INFO:tasks.rados.rados.0.smithi107.stdout:10403: writing smithi10729407-1 from 1318912 to 1941504 tid 1 2020-06-12T02:39:10.453 INFO:tasks.rados.rados.0.smithi107.stdout:10403: writing smithi10729407-1 from 1941504 to 2727936 tid 2 2020-06-12T02:39:10.453 INFO:tasks.rados.rados.0.smithi107.stdout:10403: writing smithi10729407-1 from 2727936 to 2744320 tid 3 2020-06-12T02:39:10.453 INFO:tasks.rados.rados.0.smithi107.stdout:10404: delete oid 5 current snap is 0 2020-06-12T02:39:10.456 INFO:tasks.rados.rados.0.smithi107.stdout:10404: done (1 left) 2020-06-12T02:39:10.456 INFO:tasks.rados.rados.0.smithi107.stdout:10405: delete oid 3 current snap is 0 2020-06-12T02:39:10.458 INFO:tasks.rados.rados.0.smithi107.stdout:10405: done (1 left) 2020-06-12T02:39:10.459 INFO:tasks.rados.rados.0.smithi107.stdout:10406: delete oid 11 current snap is 0 2020-06-12T02:39:10.461 INFO:tasks.rados.rados.0.smithi107.stdout:10403: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:10.461 INFO:tasks.rados.rados.0.smithi107.stdout:10406: done (1 left) 2020-06-12T02:39:10.462 INFO:tasks.rados.rados.0.smithi107.stdout:10407: delete oid 10 current snap is 0 2020-06-12T02:39:10.463 INFO:tasks.rados.rados.0.smithi107.stdout:10403: finishing write tid 2 to smithi10729407-1 2020-06-12T02:39:10.463 INFO:tasks.rados.rados.0.smithi107.stdout:10403: finishing write tid 3 to smithi10729407-1 2020-06-12T02:39:10.464 INFO:tasks.rados.rados.0.smithi107.stdout:10403: finishing write tid 5 to smithi10729407-1 2020-06-12T02:39:10.466 INFO:tasks.rados.rados.0.smithi107.stdout:10403: finishing write tid 6 to smithi10729407-1 2020-06-12T02:39:10.466 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1887 (ObjNum 10402 snap 0 seq_num 10402) dirty exists 2020-06-12T02:39:10.466 INFO:tasks.rados.rados.0.smithi107.stdout:10403: left oid 1 (ObjNum 10402 snap 0 seq_num 10402) 2020-06-12T02:39:10.467 INFO:tasks.rados.rados.0.smithi107.stdout:10403: done (1 left) 2020-06-12T02:39:10.467 INFO:tasks.rados.rados.0.smithi107.stdout:10407: done (0 left) 2020-06-12T02:39:10.468 INFO:tasks.rados.rados.0.smithi107.stdout:10408: delete oid 3 current snap is 0 2020-06-12T02:39:10.471 INFO:tasks.rados.rados.0.smithi107.stdout:10408: done (0 left) 2020-06-12T02:39:10.471 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:39:10.471 INFO:tasks.rados.rados.0.smithi107.stdout:10409: seq_num 10408 ranges {0=720896,720896=540672,1261568=262144} 2020-06-12T02:39:10.475 INFO:tasks.rados.rados.0.smithi107.stdout:10409: writing smithi10729407-11 from 0 to 720896 tid 1 2020-06-12T02:39:10.477 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:39:10.475+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:39:10.478 INFO:tasks.rados.rados.0.smithi107.stdout:10409: writing smithi10729407-11 from 720896 to 1261568 tid 2 2020-06-12T02:39:10.480 INFO:tasks.rados.rados.0.smithi107.stdout:10409: writing smithi10729407-11 from 1261568 to 1523712 tid 3 2020-06-12T02:39:10.480 INFO:tasks.rados.rados.0.smithi107.stdout:10410: delete oid 12 current snap is 0 2020-06-12T02:39:10.482 INFO:tasks.rados.rados.0.smithi107.stdout:10409: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:10.483 INFO:tasks.rados.rados.0.smithi107.stdout:10410: done (1 left) 2020-06-12T02:39:10.483 INFO:tasks.rados.rados.0.smithi107.stdout:10411: delete oid 19 current snap is 0 2020-06-12T02:39:10.486 INFO:tasks.rados.rados.0.smithi107.stdout:10411: done (1 left) 2020-06-12T02:39:10.486 INFO:tasks.rados.rados.0.smithi107.stdout:10412: delete oid 6 current snap is 0 2020-06-12T02:39:10.490 INFO:tasks.rados.rados.0.smithi107.stdout:10409: finishing write tid 2 to smithi10729407-11 2020-06-12T02:39:10.491 INFO:tasks.rados.rados.0.smithi107.stdout:10409: finishing write tid 3 to smithi10729407-11 2020-06-12T02:39:10.491 INFO:tasks.rados.rados.0.smithi107.stdout:10409: finishing write tid 5 to smithi10729407-11 2020-06-12T02:39:10.492 INFO:tasks.rados.rados.0.smithi107.stdout:10412: done (1 left) 2020-06-12T02:39:10.492 INFO:tasks.rados.rados.0.smithi107.stdout:10413: delete oid 3 current snap is 0 2020-06-12T02:39:10.494 INFO:tasks.rados.rados.0.smithi107.stdout:10409: finishing write tid 6 to smithi10729407-11 2020-06-12T02:39:10.494 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1699 (ObjNum 10408 snap 0 seq_num 10408) dirty exists 2020-06-12T02:39:10.494 INFO:tasks.rados.rados.0.smithi107.stdout:10409: left oid 11 (ObjNum 10408 snap 0 seq_num 10408) 2020-06-12T02:39:10.499 INFO:tasks.rados.rados.0.smithi107.stdout:10409: done (1 left) 2020-06-12T02:39:10.499 INFO:tasks.rados.rados.0.smithi107.stdout:10413: done (0 left) 2020-06-12T02:39:10.499 INFO:tasks.rados.rados.0.smithi107.stdout:10414: delete oid 20 current snap is 0 2020-06-12T02:39:10.500 INFO:tasks.rados.rados.0.smithi107.stdout:10414: done (0 left) 2020-06-12T02:39:10.500 INFO:tasks.rados.rados.0.smithi107.stdout:10415: delete oid 13 current snap is 0 2020-06-12T02:39:10.502 INFO:tasks.rados.rados.0.smithi107.stdout:10415: done (0 left) 2020-06-12T02:39:10.503 INFO:tasks.rados.rados.0.smithi107.stdout:10416: delete oid 14 current snap is 0 2020-06-12T02:39:10.504 INFO:tasks.rados.rados.0.smithi107.stdout:10416: done (0 left) 2020-06-12T02:39:10.504 INFO:tasks.rados.rados.0.smithi107.stdout:10417: delete oid 5 current snap is 0 2020-06-12T02:39:10.506 INFO:tasks.rados.rados.0.smithi107.stdout:10417: done (0 left) 2020-06-12T02:39:10.507 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:39:10.507 INFO:tasks.rados.rados.0.smithi107.stdout:10418: seq_num 10417 ranges {0=466944,466944=753664,1220608=450560} 2020-06-12T02:39:10.509 INFO:tasks.rados.rados.0.smithi107.stdout:10418: writing smithi10729407-8 from 0 to 466944 tid 1 2020-06-12T02:39:10.513 INFO:tasks.rados.rados.0.smithi107.stdout:10418: writing smithi10729407-8 from 466944 to 1220608 tid 2 2020-06-12T02:39:10.516 INFO:tasks.rados.rados.0.smithi107.stdout:10418: writing smithi10729407-8 from 1220608 to 1671168 tid 3 2020-06-12T02:39:10.516 INFO:tasks.rados.rados.0.smithi107.stdout:10419: delete oid 17 current snap is 0 2020-06-12T02:39:10.520 INFO:tasks.rados.rados.0.smithi107.stdout:10419: done (1 left) 2020-06-12T02:39:10.520 INFO:tasks.rados.rados.0.smithi107.stdout:10420: delete oid 10 current snap is 0 2020-06-12T02:39:10.521 INFO:tasks.rados.rados.0.smithi107.stdout:10418: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:10.522 INFO:tasks.rados.rados.0.smithi107.stdout:10420: done (1 left) 2020-06-12T02:39:10.522 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:39:10.523 INFO:tasks.rados.rados.0.smithi107.stdout:10421: seq_num 10420 ranges {2744320=524288,3268608=802816,4071424=401408} 2020-06-12T02:39:10.540 INFO:tasks.rados.rados.0.smithi107.stdout:10421: writing smithi10729407-1 from 2744320 to 3268608 tid 1 2020-06-12T02:39:10.545 INFO:tasks.rados.rados.0.smithi107.stdout:10421: writing smithi10729407-1 from 3268608 to 4071424 tid 2 2020-06-12T02:39:10.547 INFO:tasks.rados.rados.0.smithi107.stdout:10421: writing smithi10729407-1 from 4071424 to 4472832 tid 3 2020-06-12T02:39:10.547 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:39:10.547 INFO:tasks.rados.rados.0.smithi107.stdout:10422: seq_num 10421 ranges {0=540672,540672=688128,1228800=516096} 2020-06-12T02:39:10.550 INFO:tasks.rados.rados.0.smithi107.stdout:10422: writing smithi10729407-10 from 0 to 540672 tid 1 2020-06-12T02:39:10.554 INFO:tasks.rados.rados.0.smithi107.stdout:10422: writing smithi10729407-10 from 540672 to 1228800 tid 2 2020-06-12T02:39:10.557 INFO:tasks.rados.rados.0.smithi107.stdout:10422: writing smithi10729407-10 from 1228800 to 1744896 tid 3 2020-06-12T02:39:10.557 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:39:10.558 INFO:tasks.rados.rados.0.smithi107.stdout:10423: seq_num 10422 ranges {0=557056,557056=573440,1130496=630784} 2020-06-12T02:39:10.560 INFO:tasks.rados.rados.0.smithi107.stdout:10423: writing smithi10729407-16 from 0 to 557056 tid 1 2020-06-12T02:39:10.564 INFO:tasks.rados.rados.0.smithi107.stdout:10423: writing smithi10729407-16 from 557056 to 1130496 tid 2 2020-06-12T02:39:10.568 INFO:tasks.rados.rados.0.smithi107.stdout:10423: writing smithi10729407-16 from 1130496 to 1761280 tid 3 2020-06-12T02:39:10.569 INFO:tasks.rados.rados.0.smithi107.stdout:10424: delete oid 3 current snap is 0 2020-06-12T02:39:10.569 INFO:tasks.rados.rados.0.smithi107.stdout:10418: finishing write tid 2 to smithi10729407-8 2020-06-12T02:39:10.569 INFO:tasks.rados.rados.0.smithi107.stdout:10418: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:10.569 INFO:tasks.rados.rados.0.smithi107.stdout:10418: finishing write tid 5 to smithi10729407-8 2020-06-12T02:39:10.570 INFO:tasks.rados.rados.0.smithi107.stdout:10418: finishing write tid 6 to smithi10729407-8 2020-06-12T02:39:10.570 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1928 (ObjNum 10417 snap 0 seq_num 10417) dirty exists 2020-06-12T02:39:10.570 INFO:tasks.rados.rados.0.smithi107.stdout:10418: left oid 8 (ObjNum 10417 snap 0 seq_num 10417) 2020-06-12T02:39:10.570 INFO:tasks.rados.rados.0.smithi107.stdout:10421: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:10.570 INFO:tasks.rados.rados.0.smithi107.stdout:10421: finishing write tid 2 to smithi10729407-1 2020-06-12T02:39:10.570 INFO:tasks.rados.rados.0.smithi107.stdout:10421: finishing write tid 3 to smithi10729407-1 2020-06-12T02:39:10.571 INFO:tasks.rados.rados.0.smithi107.stdout:10421: finishing write tid 5 to smithi10729407-1 2020-06-12T02:39:10.571 INFO:tasks.rados.rados.0.smithi107.stdout:10421: finishing write tid 6 to smithi10729407-1 2020-06-12T02:39:10.571 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1892 (ObjNum 10420 snap 0 seq_num 10420) dirty exists 2020-06-12T02:39:10.571 INFO:tasks.rados.rados.0.smithi107.stdout:10421: left oid 1 (ObjNum 10420 snap 0 seq_num 10420) 2020-06-12T02:39:10.571 INFO:tasks.rados.rados.0.smithi107.stdout:10422: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:10.572 INFO:tasks.rados.rados.0.smithi107.stdout:10423: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:10.572 INFO:tasks.rados.rados.0.smithi107.stdout:10422: finishing write tid 2 to smithi10729407-10 2020-06-12T02:39:10.573 INFO:tasks.rados.rados.0.smithi107.stdout:10422: finishing write tid 3 to smithi10729407-10 2020-06-12T02:39:10.573 INFO:tasks.rados.rados.0.smithi107.stdout:10422: finishing write tid 5 to smithi10729407-10 2020-06-12T02:39:10.573 INFO:tasks.rados.rados.0.smithi107.stdout:10422: finishing write tid 6 to smithi10729407-10 2020-06-12T02:39:10.573 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 755 (ObjNum 10421 snap 0 seq_num 10421) dirty exists 2020-06-12T02:39:10.573 INFO:tasks.rados.rados.0.smithi107.stdout:10422: left oid 10 (ObjNum 10421 snap 0 seq_num 10421) 2020-06-12T02:39:10.578 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:39:10.575+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:39:10.580 INFO:tasks.rados.rados.0.smithi107.stdout:10423: finishing write tid 2 to smithi10729407-16 2020-06-12T02:39:10.581 INFO:tasks.rados.rados.0.smithi107.stdout:10423: finishing write tid 3 to smithi10729407-16 2020-06-12T02:39:10.581 INFO:tasks.rados.rados.0.smithi107.stdout:10423: finishing write tid 5 to smithi10729407-16 2020-06-12T02:39:10.582 INFO:tasks.rados.rados.0.smithi107.stdout:10418: done (4 left) 2020-06-12T02:39:10.582 INFO:tasks.rados.rados.0.smithi107.stdout:10421: done (3 left) 2020-06-12T02:39:10.582 INFO:tasks.rados.rados.0.smithi107.stdout:10422: done (2 left) 2020-06-12T02:39:10.582 INFO:tasks.rados.rados.0.smithi107.stdout:10424: done (1 left) 2020-06-12T02:39:10.583 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:39:10.583 INFO:tasks.rados.rados.0.smithi107.stdout:10425: seq_num 10424 ranges {0=589824,589824=794624,1384448=409600} 2020-06-12T02:39:10.585 INFO:tasks.rados.rados.0.smithi107.stdout:10425: writing smithi10729407-3 from 0 to 589824 tid 1 2020-06-12T02:39:10.590 INFO:tasks.rados.rados.0.smithi107.stdout:10425: writing smithi10729407-3 from 589824 to 1384448 tid 2 2020-06-12T02:39:10.592 INFO:tasks.rados.rados.0.smithi107.stdout:10425: writing smithi10729407-3 from 1384448 to 1794048 tid 3 2020-06-12T02:39:10.593 INFO:tasks.rados.rados.0.smithi107.stdout:10426: delete oid 4 current snap is 0 2020-06-12T02:39:10.593 INFO:tasks.rados.rados.0.smithi107.stdout:10423: finishing write tid 6 to smithi10729407-16 2020-06-12T02:39:10.593 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1858 (ObjNum 10422 snap 0 seq_num 10422) dirty exists 2020-06-12T02:39:10.593 INFO:tasks.rados.rados.0.smithi107.stdout:10423: left oid 16 (ObjNum 10422 snap 0 seq_num 10422) 2020-06-12T02:39:10.594 INFO:tasks.rados.rados.0.smithi107.stdout:10425: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:10.598 INFO:tasks.rados.rados.0.smithi107.stdout:10423: done (2 left) 2020-06-12T02:39:10.598 INFO:tasks.rados.rados.0.smithi107.stdout:10426: done (1 left) 2020-06-12T02:39:10.598 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:39:10.599 INFO:tasks.rados.rados.0.smithi107.stdout:10427: seq_num 10426 ranges {0=622592,622592=565248,1187840=638976} 2020-06-12T02:39:10.600 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:39:10.600 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:39:10.600 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:39:10.601 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:39:10.601 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:39:10.602 INFO:tasks.rados.rados.0.smithi107.stdout:10427: writing smithi10729407-12 from 0 to 622592 tid 1 2020-06-12T02:39:10.605 INFO:tasks.rados.rados.0.smithi107.stdout:10427: writing smithi10729407-12 from 622592 to 1187840 tid 2 2020-06-12T02:39:10.610 INFO:tasks.rados.rados.0.smithi107.stdout:10427: writing smithi10729407-12 from 1187840 to 1826816 tid 3 2020-06-12T02:39:10.611 INFO:tasks.rados.rados.0.smithi107.stdout:10428: delete oid 16 current snap is 0 2020-06-12T02:39:10.611 INFO:tasks.rados.rados.0.smithi107.stdout:10425: finishing write tid 2 to smithi10729407-3 2020-06-12T02:39:10.611 INFO:tasks.rados.rados.0.smithi107.stdout:10425: finishing write tid 3 to smithi10729407-3 2020-06-12T02:39:10.611 INFO:tasks.rados.rados.0.smithi107.stdout:10425: finishing write tid 5 to smithi10729407-3 2020-06-12T02:39:10.613 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-06-12T02:39:10.613 INFO:tasks.rados.rados.0.smithi107.stdout:10427: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:10.614 INFO:tasks.rados.rados.0.smithi107.stdout:10425: finishing write tid 6 to smithi10729407-3 2020-06-12T02:39:10.614 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1863 (ObjNum 10424 snap 0 seq_num 10424) dirty exists 2020-06-12T02:39:10.614 INFO:tasks.rados.rados.0.smithi107.stdout:10425: left oid 3 (ObjNum 10424 snap 0 seq_num 10424) 2020-06-12T02:39:10.614 INFO:tasks.rados.rados.0.smithi107.stdout:10425: done (2 left) 2020-06-12T02:39:10.614 INFO:tasks.rados.rados.0.smithi107.stdout:10428: done (1 left) 2020-06-12T02:39:10.615 INFO:tasks.rados.rados.0.smithi107.stdout:10429: delete oid 10 current snap is 0 2020-06-12T02:39:10.615 INFO:tasks.rados.rados.0.smithi107.stdout:10429: done (1 left) 2020-06-12T02:39:10.615 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:39:10.616 INFO:tasks.rados.rados.0.smithi107.stdout:10430: seq_num 10429 ranges {2736128=671744,3407872=614400,4022272=516096,4538368=73728} 2020-06-12T02:39:10.634 INFO:tasks.rados.rados.0.smithi107.stdout:10430: writing smithi10729407-15 from 2736128 to 3407872 tid 1 2020-06-12T02:39:10.637 INFO:tasks.rados.rados.0.smithi107.stdout:10430: writing smithi10729407-15 from 3407872 to 4022272 tid 2 2020-06-12T02:39:10.640 INFO:tasks.rados.rados.0.smithi107.stdout:10430: writing smithi10729407-15 from 4022272 to 4538368 tid 3 2020-06-12T02:39:10.641 INFO:tasks.rados.rados.0.smithi107.stdout:10430: writing smithi10729407-15 from 4538368 to 4612096 tid 4 2020-06-12T02:39:10.641 INFO:tasks.rados.rados.0.smithi107.stdout:10431: delete oid 6 current snap is 0 2020-06-12T02:39:10.641 INFO:tasks.rados.rados.0.smithi107.stdout:10427: finishing write tid 2 to smithi10729407-12 2020-06-12T02:39:10.642 INFO:tasks.rados.rados.0.smithi107.stdout:10427: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:10.642 INFO:tasks.rados.rados.0.smithi107.stdout:10427: finishing write tid 5 to smithi10729407-12 2020-06-12T02:39:10.642 INFO:tasks.rados.rados.0.smithi107.stdout:10427: finishing write tid 6 to smithi10729407-12 2020-06-12T02:39:10.642 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1915 (ObjNum 10426 snap 0 seq_num 10426) dirty exists 2020-06-12T02:39:10.642 INFO:tasks.rados.rados.0.smithi107.stdout:10427: left oid 12 (ObjNum 10426 snap 0 seq_num 10426) 2020-06-12T02:39:10.643 INFO:tasks.rados.rados.0.smithi107.stdout:10430: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:10.652 INFO:tasks.rados.rados.0.smithi107.stdout:10430: finishing write tid 2 to smithi10729407-15 2020-06-12T02:39:10.652 INFO:tasks.rados.rados.0.smithi107.stdout:10430: finishing write tid 3 to smithi10729407-15 2020-06-12T02:39:10.652 INFO:tasks.rados.rados.0.smithi107.stdout:10430: finishing write tid 4 to smithi10729407-15 2020-06-12T02:39:10.652 INFO:tasks.rados.rados.0.smithi107.stdout:10430: finishing write tid 6 to smithi10729407-15 2020-06-12T02:39:10.653 INFO:tasks.rados.rados.0.smithi107.stdout:10430: finishing write tid 7 to smithi10729407-15 2020-06-12T02:39:10.654 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1933 (ObjNum 10429 snap 0 seq_num 10429) dirty exists 2020-06-12T02:39:10.654 INFO:tasks.rados.rados.0.smithi107.stdout:10430: left oid 15 (ObjNum 10429 snap 0 seq_num 10429) 2020-06-12T02:39:10.662 INFO:tasks.rados.rados.0.smithi107.stdout:10427: done (2 left) 2020-06-12T02:39:10.663 INFO:tasks.rados.rados.0.smithi107.stdout:10430: done (1 left) 2020-06-12T02:39:10.663 INFO:tasks.rados.rados.0.smithi107.stdout:10431: done (0 left) 2020-06-12T02:39:10.663 INFO:tasks.rados.rados.0.smithi107.stdout:10432: delete oid 20 current snap is 0 2020-06-12T02:39:10.666 INFO:tasks.rados.rados.0.smithi107.stdout:10432: done (0 left) 2020-06-12T02:39:10.666 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:39:10.666 INFO:tasks.rados.rados.0.smithi107.stdout:10433: seq_num 10432 ranges {0=720896,720896=720896,1441792=483328} 2020-06-12T02:39:10.669 INFO:tasks.rados.rados.0.smithi107.stdout:10433: writing smithi10729407-4 from 0 to 720896 tid 1 2020-06-12T02:39:10.674 INFO:tasks.rados.rados.0.smithi107.stdout:10433: writing smithi10729407-4 from 720896 to 1441792 tid 2 2020-06-12T02:39:10.676 INFO:tasks.rados.rados.0.smithi107.stdout:10433: writing smithi10729407-4 from 1441792 to 1925120 tid 3 2020-06-12T02:39:10.676 INFO:tasks.rados.rados.0.smithi107.stdout:10434: delete oid 3 current snap is 0 2020-06-12T02:39:10.680 INFO:tasks.rados.rados.0.smithi107.stdout:10433: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:10.680 INFO:tasks.rados.rados.0.smithi107.stdout:10434: done (1 left) 2020-06-12T02:39:10.680 INFO:tasks.rados.rados.0.smithi107.stdout:10435: delete oid 5 current snap is 0 2020-06-12T02:39:10.683 INFO:tasks.rados.rados.0.smithi107.stdout:10435: done (1 left) 2020-06-12T02:39:10.683 INFO:tasks.rados.rados.0.smithi107.stdout:10436: delete oid 1 current snap is 0 2020-06-12T02:39:10.685 INFO:tasks.rados.rados.0.smithi107.stdout:10433: finishing write tid 2 to smithi10729407-4 2020-06-12T02:39:10.686 INFO:tasks.rados.rados.0.smithi107.stdout:10436: done (1 left) 2020-06-12T02:39:10.686 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:39:10.687 INFO:tasks.rados.rados.0.smithi107.stdout:10437: seq_num 10436 ranges {1826816=786432,2613248=655360,3268608=548864} 2020-06-12T02:39:10.696 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:39:10.694+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:39:10.702 INFO:tasks.rados.rados.0.smithi107.stdout:10437: writing smithi10729407-12 from 1826816 to 2613248 tid 1 2020-06-12T02:39:10.704 INFO:tasks.rados.rados.0.smithi107.stdout:10437: writing smithi10729407-12 from 2613248 to 3268608 tid 2 2020-06-12T02:39:10.710 INFO:tasks.rados.rados.0.smithi107.stdout:10437: writing smithi10729407-12 from 3268608 to 3817472 tid 3 2020-06-12T02:39:10.710 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:39:10.710 INFO:tasks.rados.rados.0.smithi107.stdout:10438: seq_num 10437 ranges {0=802816,802816=540672,1343488=475136,1818624=188416} 2020-06-12T02:39:10.712 INFO:tasks.rados.rados.0.smithi107.stdout:10438: writing smithi10729407-6 from 0 to 802816 tid 1 2020-06-12T02:39:10.716 INFO:tasks.rados.rados.0.smithi107.stdout:10438: writing smithi10729407-6 from 802816 to 1343488 tid 2 2020-06-12T02:39:10.718 INFO:tasks.rados.rados.0.smithi107.stdout:10438: writing smithi10729407-6 from 1343488 to 1818624 tid 3 2020-06-12T02:39:10.720 INFO:tasks.rados.rados.0.smithi107.stdout:10438: writing smithi10729407-6 from 1818624 to 2007040 tid 4 2020-06-12T02:39:10.720 INFO:tasks.rados.rados.0.smithi107.stdout:10439: delete oid 9 current snap is 0 2020-06-12T02:39:10.720 INFO:tasks.rados.rados.0.smithi107.stdout:10433: finishing write tid 3 to smithi10729407-4 2020-06-12T02:39:10.720 INFO:tasks.rados.rados.0.smithi107.stdout:10433: finishing write tid 5 to smithi10729407-4 2020-06-12T02:39:10.721 INFO:tasks.rados.rados.0.smithi107.stdout:10433: finishing write tid 6 to smithi10729407-4 2020-06-12T02:39:10.721 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1943 (ObjNum 10432 snap 0 seq_num 10432) dirty exists 2020-06-12T02:39:10.721 INFO:tasks.rados.rados.0.smithi107.stdout:10433: left oid 4 (ObjNum 10432 snap 0 seq_num 10432) 2020-06-12T02:39:10.721 INFO:tasks.rados.rados.0.smithi107.stdout:10437: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:10.723 INFO:tasks.rados.rados.0.smithi107.stdout:10437: finishing write tid 2 to smithi10729407-12 2020-06-12T02:39:10.724 INFO:tasks.rados.rados.0.smithi107.stdout:10433: done (3 left) 2020-06-12T02:39:10.724 INFO:tasks.rados.rados.0.smithi107.stdout:10439: done (2 left) 2020-06-12T02:39:10.724 INFO:tasks.rados.rados.0.smithi107.stdout:10440: delete oid 9 current snap is 0 2020-06-12T02:39:10.725 INFO:tasks.rados.rados.0.smithi107.stdout:10437: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:10.725 INFO:tasks.rados.rados.0.smithi107.stdout:10437: finishing write tid 5 to smithi10729407-12 2020-06-12T02:39:10.729 INFO:tasks.rados.rados.0.smithi107.stdout:10440: done (2 left) 2020-06-12T02:39:10.730 INFO:tasks.rados.rados.0.smithi107.stdout:10441: delete oid 4 current snap is 0 2020-06-12T02:39:10.730 INFO:tasks.rados.rados.0.smithi107.stdout:10437: finishing write tid 6 to smithi10729407-12 2020-06-12T02:39:10.730 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1919 (ObjNum 10436 snap 0 seq_num 10436) dirty exists 2020-06-12T02:39:10.730 INFO:tasks.rados.rados.0.smithi107.stdout:10437: left oid 12 (ObjNum 10436 snap 0 seq_num 10436) 2020-06-12T02:39:10.733 INFO:tasks.rados.rados.0.smithi107.stdout:10437: done (2 left) 2020-06-12T02:39:10.733 INFO:tasks.rados.rados.0.smithi107.stdout:10441: done (1 left) 2020-06-12T02:39:10.734 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:39:10.734 INFO:tasks.rados.rados.0.smithi107.stdout:10442: seq_num 10441 ranges {0=475136,475136=524288,999424=450560,1449984=466944,1916928=163840} 2020-06-12T02:39:10.736 INFO:tasks.rados.rados.0.smithi107.stdout:10442: writing smithi10729407-13 from 0 to 475136 tid 1 2020-06-12T02:39:10.739 INFO:tasks.rados.rados.0.smithi107.stdout:10442: writing smithi10729407-13 from 475136 to 999424 tid 2 2020-06-12T02:39:10.742 INFO:tasks.rados.rados.0.smithi107.stdout:10442: writing smithi10729407-13 from 999424 to 1449984 tid 3 2020-06-12T02:39:10.744 INFO:tasks.rados.rados.0.smithi107.stdout:10442: writing smithi10729407-13 from 1449984 to 1916928 tid 4 2020-06-12T02:39:10.746 INFO:tasks.rados.rados.0.smithi107.stdout:10442: writing smithi10729407-13 from 1916928 to 2080768 tid 5 2020-06-12T02:39:10.746 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:39:10.746 INFO:tasks.rados.rados.0.smithi107.stdout:10443: seq_num 10442 ranges {1523712=491520,2015232=409600,2424832=663552,3088384=532480} 2020-06-12T02:39:10.757 INFO:tasks.rados.rados.0.smithi107.stdout:10443: writing smithi10729407-11 from 1523712 to 2015232 tid 1 2020-06-12T02:39:10.759 INFO:tasks.rados.rados.0.smithi107.stdout:10443: writing smithi10729407-11 from 2015232 to 2424832 tid 2 2020-06-12T02:39:10.764 INFO:tasks.rados.rados.0.smithi107.stdout:10443: writing smithi10729407-11 from 2424832 to 3088384 tid 3 2020-06-12T02:39:10.768 INFO:tasks.rados.rados.0.smithi107.stdout:10443: writing smithi10729407-11 from 3088384 to 3620864 tid 4 2020-06-12T02:39:10.769 INFO:tasks.rados.rados.0.smithi107.stdout:10444: delete oid 12 current snap is 0 2020-06-12T02:39:10.769 INFO:tasks.rados.rados.0.smithi107.stdout:10438: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:10.769 INFO:tasks.rados.rados.0.smithi107.stdout:10438: finishing write tid 2 to smithi10729407-6 2020-06-12T02:39:10.769 INFO:tasks.rados.rados.0.smithi107.stdout:10438: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:10.769 INFO:tasks.rados.rados.0.smithi107.stdout:10438: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:10.770 INFO:tasks.rados.rados.0.smithi107.stdout:10438: finishing write tid 6 to smithi10729407-6 2020-06-12T02:39:10.770 INFO:tasks.rados.rados.0.smithi107.stdout:10438: finishing write tid 7 to smithi10729407-6 2020-06-12T02:39:10.770 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1899 (ObjNum 10437 snap 0 seq_num 10437) dirty exists 2020-06-12T02:39:10.770 INFO:tasks.rados.rados.0.smithi107.stdout:10438: left oid 6 (ObjNum 10437 snap 0 seq_num 10437) 2020-06-12T02:39:10.770 INFO:tasks.rados.rados.0.smithi107.stdout:10442: finishing write tid 1 to smithi10729407-13 2020-06-12T02:39:10.771 INFO:tasks.rados.rados.0.smithi107.stdout:10442: finishing write tid 2 to smithi10729407-13 2020-06-12T02:39:10.771 INFO:tasks.rados.rados.0.smithi107.stdout:10442: finishing write tid 3 to smithi10729407-13 2020-06-12T02:39:10.771 INFO:tasks.rados.rados.0.smithi107.stdout:10442: finishing write tid 4 to smithi10729407-13 2020-06-12T02:39:10.771 INFO:tasks.rados.rados.0.smithi107.stdout:10442: finishing write tid 5 to smithi10729407-13 2020-06-12T02:39:10.771 INFO:tasks.rados.rados.0.smithi107.stdout:10442: finishing write tid 7 to smithi10729407-13 2020-06-12T02:39:10.771 INFO:tasks.rados.rados.0.smithi107.stdout:10442: finishing write tid 8 to smithi10729407-13 2020-06-12T02:39:10.772 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1743 (ObjNum 10441 snap 0 seq_num 10441) dirty exists 2020-06-12T02:39:10.772 INFO:tasks.rados.rados.0.smithi107.stdout:10442: left oid 13 (ObjNum 10441 snap 0 seq_num 10441) 2020-06-12T02:39:10.772 INFO:tasks.rados.rados.0.smithi107.stdout:10443: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:10.773 INFO:tasks.rados.rados.0.smithi107.stdout:10438: done (3 left) 2020-06-12T02:39:10.773 INFO:tasks.rados.rados.0.smithi107.stdout:10442: done (2 left) 2020-06-12T02:39:10.773 INFO:tasks.rados.rados.0.smithi107.stdout:10444: done (1 left) 2020-06-12T02:39:10.773 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:39:10.773 INFO:tasks.rados.rados.0.smithi107.stdout:10445: seq_num 10444 ranges {0=524288,524288=581632,1105920=630784,1736704=393216} 2020-06-12T02:39:10.775 INFO:tasks.rados.rados.0.smithi107.stdout:10445: writing smithi10729407-20 from 0 to 524288 tid 1 2020-06-12T02:39:10.779 INFO:tasks.rados.rados.0.smithi107.stdout:10445: writing smithi10729407-20 from 524288 to 1105920 tid 2 2020-06-12T02:39:10.781 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:39:10.779+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:39:10.783 INFO:tasks.rados.rados.0.smithi107.stdout:10445: writing smithi10729407-20 from 1105920 to 1736704 tid 3 2020-06-12T02:39:10.786 INFO:tasks.rados.rados.0.smithi107.stdout:10445: writing smithi10729407-20 from 1736704 to 2129920 tid 4 2020-06-12T02:39:10.786 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:39:10.786 INFO:tasks.rados.rados.0.smithi107.stdout:10446: seq_num 10445 ranges {0=540672,540672=466944,1007616=434176,1441792=524288,1966080=180224} 2020-06-12T02:39:10.789 INFO:tasks.rados.rados.0.smithi107.stdout:10446: writing smithi10729407-5 from 0 to 540672 tid 1 2020-06-12T02:39:10.792 INFO:tasks.rados.rados.0.smithi107.stdout:10446: writing smithi10729407-5 from 540672 to 1007616 tid 2 2020-06-12T02:39:10.795 INFO:tasks.rados.rados.0.smithi107.stdout:10446: writing smithi10729407-5 from 1007616 to 1441792 tid 3 2020-06-12T02:39:10.798 INFO:tasks.rados.rados.0.smithi107.stdout:10446: writing smithi10729407-5 from 1441792 to 1966080 tid 4 2020-06-12T02:39:10.799 INFO:tasks.rados.rados.0.smithi107.stdout:10446: writing smithi10729407-5 from 1966080 to 2146304 tid 5 2020-06-12T02:39:10.800 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:39:10.800 INFO:tasks.rados.rados.0.smithi107.stdout:10447: seq_num 10446 ranges {2080768=557056,2637824=794624,3432448=638976,4071424=172032} 2020-06-12T02:39:10.810 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:39:10.811 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:39:10.811 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:39:10.811 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:39:10.811 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.811 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:4975 3.as0 3:5732055a:::smithi10729407-8:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e30)", 2020-06-12T02:39:10.812 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:43.721794+0000", 2020-06-12T02:39:10.812 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 87.084538449999997, 2020-06-12T02:39:10.812 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.1825135, 2020-06-12T02:39:10.812 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:10.812 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:10.812 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:10.813 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:10.813 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:10.813 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 4975 2020-06-12T02:39:10.813 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.814 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:10.814 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.814 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:10.814 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721794+0000", 2020-06-12T02:39:10.814 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:10.814 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.815 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.815 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:10.815 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721794+0000", 2020-06-12T02:39:10.815 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4542e-05 2020-06-12T02:39:10.815 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.816 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.816 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:10.816 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721818+0000", 2020-06-12T02:39:10.816 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9169999999999997e-06 2020-06-12T02:39:10.816 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.817 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.817 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:10.817 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721823+0000", 2020-06-12T02:39:10.817 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1184e-05 2020-06-12T02:39:10.817 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.817 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.818 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:10.818 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721834+0000", 2020-06-12T02:39:10.818 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.254e-06 2020-06-12T02:39:10.818 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.818 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.819 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:10.819 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721840+0000", 2020-06-12T02:39:10.819 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1075999999999998e-05 2020-06-12T02:39:10.819 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.819 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.819 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:10.820 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721881+0000", 2020-06-12T02:39:10.820 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7416999999999995e-05 2020-06-12T02:39:10.820 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.820 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.820 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:10.821 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721948+0000", 2020-06-12T02:39:10.821 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.18235910999999999 2020-06-12T02:39:10.821 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.821 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.822 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:10.822 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.904307+0000", 2020-06-12T02:39:10.822 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.1825135 2020-06-12T02:39:10.823 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.824 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:10.824 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.826 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.827 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.827 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11914 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e56)", 2020-06-12T02:39:10.827 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:23.508790+0000", 2020-06-12T02:39:10.827 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 47.297541598000002, 2020-06-12T02:39:10.828 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026541928999999999, 2020-06-12T02:39:10.828 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:10.829 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:10.829 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:10.829 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:10.829 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:10.829 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11914 2020-06-12T02:39:10.830 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.830 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:10.830 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.830 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:10.831 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508790+0000", 2020-06-12T02:39:10.831 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:10.831 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.831 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.832 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:10.833 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508790+0000", 2020-06-12T02:39:10.833 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0872999999999999e-05 2020-06-12T02:39:10.834 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.834 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.834 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:10.834 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508841+0000", 2020-06-12T02:39:10.834 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9179999999999994e-06 2020-06-12T02:39:10.835 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.835 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.835 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:10.835 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508849+0000", 2020-06-12T02:39:10.835 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5551999999999999e-05 2020-06-12T02:39:10.836 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.836 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.836 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:10.836 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508865+0000", 2020-06-12T02:39:10.836 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0629999999999996e-06 2020-06-12T02:39:10.836 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.837 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.837 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:10.837 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508870+0000", 2020-06-12T02:39:10.837 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010597488 2020-06-12T02:39:10.838 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.838 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.838 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:10.838 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.519467+0000", 2020-06-12T02:39:10.839 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.6083999999999997e-05 2020-06-12T02:39:10.839 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.839 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.839 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:10.839 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.519563+0000", 2020-06-12T02:39:10.840 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0078077800000000003 2020-06-12T02:39:10.840 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.840 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.840 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:10.840 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.527371+0000", 2020-06-12T02:39:10.841 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6198999999999993e-05 2020-06-12T02:39:10.841 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.841 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.841 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:10.841 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.527447+0000", 2020-06-12T02:39:10.842 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0078849720000000005 2020-06-12T02:39:10.842 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.842 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.842 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:10.843 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.535332+0000", 2020-06-12T02:39:10.844 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026541928999999999 2020-06-12T02:39:10.844 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.844 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:10.844 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.845 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.845 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.845 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13705 3.as0 3:5cb723d2:::smithi10729407-15:head [append 0~745472,stat] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:39:10.845 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.390499+0000", 2020-06-12T02:39:10.845 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 42.415832905000002, 2020-06-12T02:39:10.846 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.067319320000000002, 2020-06-12T02:39:10.846 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:10.846 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:10.846 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:10.846 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:10.847 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:10.847 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13705 2020-06-12T02:39:10.847 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.847 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:10.847 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.847 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:10.848 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.390499+0000", 2020-06-12T02:39:10.848 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:10.848 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.848 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.848 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:10.849 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.390499+0000", 2020-06-12T02:39:10.849 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00066732599999999999 2020-06-12T02:39:10.849 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.849 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.849 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:10.849 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391167+0000", 2020-06-12T02:39:10.850 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3609999999999997e-06 2020-06-12T02:39:10.850 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.850 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.850 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:10.850 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391173+0000", 2020-06-12T02:39:10.851 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0399e-05 2020-06-12T02:39:10.851 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.851 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.851 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:10.851 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391183+0000", 2020-06-12T02:39:10.852 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6480000000000002e-06 2020-06-12T02:39:10.852 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.852 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.852 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:10.853 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391188+0000", 2020-06-12T02:39:10.853 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4150999999999999e-05 2020-06-12T02:39:10.853 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.853 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.853 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:10.853 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391222+0000", 2020-06-12T02:39:10.854 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9450999999999996e-05 2020-06-12T02:39:10.854 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.854 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.854 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:10.854 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391292+0000", 2020-06-12T02:39:10.855 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054214300000000003 2020-06-12T02:39:10.855 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.855 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.855 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:10.855 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391834+0000", 2020-06-12T02:39:10.856 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0037312069999999998 2020-06-12T02:39:10.856 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.856 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.856 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:10.856 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.395565+0000", 2020-06-12T02:39:10.856 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001050194 2020-06-12T02:39:10.857 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.858 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.858 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:10.858 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396615+0000", 2020-06-12T02:39:10.858 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.061203439999999998 2020-06-12T02:39:10.859 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.859 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.859 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:10.859 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.457819+0000", 2020-06-12T02:39:10.859 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.067319320000000002 2020-06-12T02:39:10.860 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.860 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:10.860 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.860 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.860 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.861 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13706 3.as0 3:5cb723d2:::smithi10729407-15:head [append 0~434176] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:39:10.861 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.393133+0000", 2020-06-12T02:39:10.861 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 42.413198637000001, 2020-06-12T02:39:10.861 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.078158561000000001, 2020-06-12T02:39:10.861 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:10.862 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:10.862 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:10.862 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:10.862 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:10.863 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13706 2020-06-12T02:39:10.863 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.863 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:10.863 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.863 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:10.864 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393133+0000", 2020-06-12T02:39:10.864 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:10.864 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.864 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.864 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:10.864 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393133+0000", 2020-06-12T02:39:10.865 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00050415499999999995 2020-06-12T02:39:10.865 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.865 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.865 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:10.865 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393638+0000", 2020-06-12T02:39:10.866 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5010000000000004e-06 2020-06-12T02:39:10.866 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.866 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.866 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:10.866 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393646+0000", 2020-06-12T02:39:10.867 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4183e-05 2020-06-12T02:39:10.867 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.867 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.867 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:10.867 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393660+0000", 2020-06-12T02:39:10.867 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6069999999999996e-06 2020-06-12T02:39:10.868 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.868 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.868 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:10.868 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393667+0000", 2020-06-12T02:39:10.868 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001363569 2020-06-12T02:39:10.869 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.869 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.869 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:10.869 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.395030+0000", 2020-06-12T02:39:10.869 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012127 2020-06-12T02:39:10.870 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.870 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.870 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:10.870 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.395152+0000", 2020-06-12T02:39:10.870 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.063640503000000001 2020-06-12T02:39:10.870 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.871 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.871 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:10.871 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458792+0000", 2020-06-12T02:39:10.871 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010636 2020-06-12T02:39:10.871 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.872 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.872 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:10.872 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458899+0000", 2020-06-12T02:39:10.872 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00076074200000000002 2020-06-12T02:39:10.872 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.873 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.873 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:10.873 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459659+0000", 2020-06-12T02:39:10.873 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00772984 2020-06-12T02:39:10.873 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.873 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.874 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:10.874 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.467389+0000", 2020-06-12T02:39:10.874 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003815751 2020-06-12T02:39:10.874 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.874 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.875 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:10.875 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471205+0000", 2020-06-12T02:39:10.875 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.708e-05 2020-06-12T02:39:10.875 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.875 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.876 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:10.876 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471292+0000", 2020-06-12T02:39:10.876 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.078158561000000001 2020-06-12T02:39:10.876 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.876 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:10.876 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.877 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.877 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.877 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13707 3.as0 3:5cb723d2:::smithi10729407-15:head [append 0~565248] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:39:10.877 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.396173+0000", 2020-06-12T02:39:10.877 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 42.410158694000003, 2020-06-12T02:39:10.878 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.075298341000000005, 2020-06-12T02:39:10.878 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:10.878 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:10.878 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:10.878 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:10.879 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:10.879 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13707 2020-06-12T02:39:10.879 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.879 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:10.879 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.879 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:10.880 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396173+0000", 2020-06-12T02:39:10.880 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:10.880 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.880 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.880 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:10.881 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396173+0000", 2020-06-12T02:39:10.881 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.061779787000000003 2020-06-12T02:39:10.881 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.881 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.881 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:10.882 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.457953+0000", 2020-06-12T02:39:10.882 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5830000000000007e-06 2020-06-12T02:39:10.882 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.882 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.882 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:10.883 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.457962+0000", 2020-06-12T02:39:10.883 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3029e-05 2020-06-12T02:39:10.883 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.883 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.883 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:10.883 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.457975+0000", 2020-06-12T02:39:10.884 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.233e-06 2020-06-12T02:39:10.884 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.884 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.884 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:10.884 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.457979+0000", 2020-06-12T02:39:10.885 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0044352510000000003 2020-06-12T02:39:10.885 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.885 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.886 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:10.886 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.462414+0000", 2020-06-12T02:39:10.886 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012825799999999999 2020-06-12T02:39:10.886 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.886 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.887 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:10.887 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.462543+0000", 2020-06-12T02:39:10.887 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00077227899999999996 2020-06-12T02:39:10.887 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.887 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.888 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:10.888 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.463315+0000", 2020-06-12T02:39:10.888 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0079912390000000007 2020-06-12T02:39:10.888 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.888 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.888 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:10.889 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471306+0000", 2020-06-12T02:39:10.889 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000118674 2020-06-12T02:39:10.889 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.889 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.889 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:10.890 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471425+0000", 2020-06-12T02:39:10.890 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7008000000000001e-05 2020-06-12T02:39:10.890 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.890 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.890 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:10.891 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471472+0000", 2020-06-12T02:39:10.891 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.075298341000000005 2020-06-12T02:39:10.891 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.891 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:10.891 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.891 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.892 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.892 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWriteReply(3.as0 58/53 ECSubWriteReply(tid=1890, last_complete=58'1184, committed=1, applied=1))", 2020-06-12T02:39:10.892 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.396464+0000", 2020-06-12T02:39:10.892 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 42.409867814999998, 2020-06-12T02:39:10.892 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.062264803000000001, 2020-06-12T02:39:10.893 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:10.893 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:39:10.893 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:10.893 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.894 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:10.894 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396464+0000", 2020-06-12T02:39:10.894 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:10.894 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.894 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.895 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:10.895 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396464+0000", 2020-06-12T02:39:10.895 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9425999999999999e-05 2020-06-12T02:39:10.895 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.895 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.895 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:10.896 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396494+0000", 2020-06-12T02:39:10.896 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1610000000000002e-06 2020-06-12T02:39:10.896 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.896 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.896 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:10.897 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396499+0000", 2020-06-12T02:39:10.897 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2789999999999998e-06 2020-06-12T02:39:10.897 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.897 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.898 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:10.898 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396507+0000", 2020-06-12T02:39:10.899 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1389999999999997e-06 2020-06-12T02:39:10.899 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.900 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.900 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:10.900 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396511+0000", 2020-06-12T02:39:10.900 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2199000000000001e-05 2020-06-12T02:39:10.900 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.901 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.901 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:10.901 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396543+0000", 2020-06-12T02:39:10.901 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.062185599000000001 2020-06-12T02:39:10.901 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.901 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.902 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:10.902 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458729+0000", 2020-06-12T02:39:10.902 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.062264803000000001 2020-06-12T02:39:10.902 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.902 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:10.903 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.903 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.903 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.903 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13710 3.as0 3:5cb723d2:::smithi10729407-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e58)", 2020-06-12T02:39:10.903 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.458457+0000", 2020-06-12T02:39:10.904 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 42.347874644999997, 2020-06-12T02:39:10.904 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028102095000000001, 2020-06-12T02:39:10.904 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:10.904 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:10.904 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:10.905 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:10.905 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:10.905 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13710 2020-06-12T02:39:10.905 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.905 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:10.906 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.906 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:10.906 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458457+0000", 2020-06-12T02:39:10.906 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:10.906 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.907 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.907 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:10.907 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458457+0000", 2020-06-12T02:39:10.907 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7650000000000001e-05 2020-06-12T02:39:10.907 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.907 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.908 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:10.908 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458485+0000", 2020-06-12T02:39:10.908 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3210000000000001e-06 2020-06-12T02:39:10.908 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.908 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.909 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:10.909 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458488+0000", 2020-06-12T02:39:10.909 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5920000000000003e-06 2020-06-12T02:39:10.909 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.909 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.909 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:10.910 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458496+0000", 2020-06-12T02:39:10.910 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8789999999999999e-06 2020-06-12T02:39:10.910 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.910 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.910 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:10.911 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458499+0000", 2020-06-12T02:39:10.911 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013081152 2020-06-12T02:39:10.911 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.911 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.911 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:10.912 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471580+0000", 2020-06-12T02:39:10.912 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4975999999999994e-05 2020-06-12T02:39:10.912 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.912 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.912 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:10.912 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471665+0000", 2020-06-12T02:39:10.913 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0054254029999999996 2020-06-12T02:39:10.913 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.913 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.913 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:10.913 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.477090+0000", 2020-06-12T02:39:10.914 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2335000000000005e-05 2020-06-12T02:39:10.914 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.914 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.914 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:10.914 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.477163+0000", 2020-06-12T02:39:10.915 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0093967870000000002 2020-06-12T02:39:10.915 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.915 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.915 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:10.915 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486560+0000", 2020-06-12T02:39:10.916 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028102095000000001 2020-06-12T02:39:10.916 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.916 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:10.916 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.916 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.916 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.917 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13711 3.as0 3:5732055a:::smithi10729407-8:head [append 0~761856] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:39:10.917 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.458525+0000", 2020-06-12T02:39:10.917 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 42.347806873000003, 2020-06-12T02:39:10.917 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.027773760000000002, 2020-06-12T02:39:10.917 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:10.918 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:10.918 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:10.918 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:10.918 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:10.918 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13711 2020-06-12T02:39:10.919 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.919 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:10.919 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.919 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:10.919 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458525+0000", 2020-06-12T02:39:10.920 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:10.920 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.920 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.920 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:10.920 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458525+0000", 2020-06-12T02:39:10.920 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00089965200000000003 2020-06-12T02:39:10.921 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.921 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.921 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:10.921 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459425+0000", 2020-06-12T02:39:10.921 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4749999999999996e-06 2020-06-12T02:39:10.922 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.922 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.922 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:10.922 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459432+0000", 2020-06-12T02:39:10.922 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3043e-05 2020-06-12T02:39:10.923 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.923 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.923 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:10.923 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459445+0000", 2020-06-12T02:39:10.923 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0739999999999998e-06 2020-06-12T02:39:10.924 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.924 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.924 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:10.924 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459451+0000", 2020-06-12T02:39:10.925 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012238733999999999 2020-06-12T02:39:10.925 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.925 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.925 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:10.925 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471690+0000", 2020-06-12T02:39:10.925 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5640000000000003e-05 2020-06-12T02:39:10.926 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.926 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.926 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:10.926 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471746+0000", 2020-06-12T02:39:10.926 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00067859699999999997 2020-06-12T02:39:10.927 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.927 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.927 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:10.927 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.472424+0000", 2020-06-12T02:39:10.927 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050083230000000003 2020-06-12T02:39:10.927 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.928 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.928 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:10.928 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.477433+0000", 2020-06-12T02:39:10.928 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0087878039999999998 2020-06-12T02:39:10.928 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.929 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.929 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:10.929 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486221+0000", 2020-06-12T02:39:10.929 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8417999999999994e-05 2020-06-12T02:39:10.929 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.929 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.930 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:10.930 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486299+0000", 2020-06-12T02:39:10.930 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.027773760000000002 2020-06-12T02:39:10.930 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.930 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:10.931 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.931 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.931 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.932 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13712 3.as0 3:5732055a:::smithi10729407-8:head [append 0~712704] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:39:10.932 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.459508+0000", 2020-06-12T02:39:10.932 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 42.346823593000003, 2020-06-12T02:39:10.933 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.033995818999999997, 2020-06-12T02:39:10.933 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:10.933 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:10.933 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:10.934 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:10.934 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:10.934 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13712 2020-06-12T02:39:10.934 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.934 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:10.935 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.935 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:10.935 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459508+0000", 2020-06-12T02:39:10.935 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:10.935 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.936 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.936 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:10.936 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459508+0000", 2020-06-12T02:39:10.936 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00075436999999999998 2020-06-12T02:39:10.936 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.937 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.937 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:10.937 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460263+0000", 2020-06-12T02:39:10.937 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4370000000000001e-06 2020-06-12T02:39:10.937 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.937 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.938 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:10.938 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460270+0000", 2020-06-12T02:39:10.938 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2500000000000001e-05 2020-06-12T02:39:10.938 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.938 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.939 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:10.939 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460283+0000", 2020-06-12T02:39:10.939 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2580000000000003e-06 2020-06-12T02:39:10.939 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.939 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.940 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:10.940 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460288+0000", 2020-06-12T02:39:10.940 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017177595 2020-06-12T02:39:10.940 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.940 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.941 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:10.941 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.477466+0000", 2020-06-12T02:39:10.941 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00019751600000000001 2020-06-12T02:39:10.941 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.942 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.942 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:10.942 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.477663+0000", 2020-06-12T02:39:10.942 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00088697400000000001 2020-06-12T02:39:10.942 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.943 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.943 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:10.943 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.478550+0000", 2020-06-12T02:39:10.943 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.007578787 2020-06-12T02:39:10.943 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.944 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.944 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:10.944 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486129+0000", 2020-06-12T02:39:10.944 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0072972310000000004 2020-06-12T02:39:10.944 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.944 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.945 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:10.945 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493426+0000", 2020-06-12T02:39:10.945 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8151000000000004e-05 2020-06-12T02:39:10.945 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.945 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.946 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:10.946 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493504+0000", 2020-06-12T02:39:10.946 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.033995818999999997 2020-06-12T02:39:10.946 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.946 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:10.947 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.947 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.947 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.947 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13713 3.as0 3:5732055a:::smithi10729407-8:head [append 0~491520] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:39:10.947 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.460333+0000", 2020-06-12T02:39:10.947 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 42.345999137, 2020-06-12T02:39:10.948 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.033390145000000003, 2020-06-12T02:39:10.948 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:10.948 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:10.948 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:10.948 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:10.949 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:10.949 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13713 2020-06-12T02:39:10.949 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.949 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:10.949 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.949 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:10.950 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460333+0000", 2020-06-12T02:39:10.950 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:10.950 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.950 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.950 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:10.951 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460333+0000", 2020-06-12T02:39:10.951 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00051168999999999997 2020-06-12T02:39:10.951 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.951 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.951 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:10.951 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460845+0000", 2020-06-12T02:39:10.952 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2930000000000003e-06 2020-06-12T02:39:10.952 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.952 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.952 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:10.952 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460852+0000", 2020-06-12T02:39:10.953 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3054000000000001e-05 2020-06-12T02:39:10.953 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.953 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.953 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:10.953 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460865+0000", 2020-06-12T02:39:10.953 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6130000000000003e-06 2020-06-12T02:39:10.954 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.954 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.954 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:10.954 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460871+0000", 2020-06-12T02:39:10.954 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025753870000000002 2020-06-12T02:39:10.955 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.955 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.955 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:10.955 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486624+0000", 2020-06-12T02:39:10.955 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2739999999999998e-05 2020-06-12T02:39:10.956 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.956 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.956 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:10.956 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486697+0000", 2020-06-12T02:39:10.956 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00063578799999999998 2020-06-12T02:39:10.956 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.957 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.957 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:10.957 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.487333+0000", 2020-06-12T02:39:10.957 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0043011610000000004 2020-06-12T02:39:10.957 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.958 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.958 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:10.958 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.491634+0000", 2020-06-12T02:39:10.958 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0020130949999999999 2020-06-12T02:39:10.958 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.958 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.959 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:10.959 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493647+0000", 2020-06-12T02:39:10.959 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5840999999999994e-05 2020-06-12T02:39:10.959 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.959 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.960 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:10.960 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493723+0000", 2020-06-12T02:39:10.960 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.033390145000000003 2020-06-12T02:39:10.960 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.960 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:10.961 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.961 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.961 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.961 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13714 3.as0 3:5732055a:::smithi10729407-8:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:39:10.961 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.460921+0000", 2020-06-12T02:39:10.962 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 42.345410614999999, 2020-06-12T02:39:10.962 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.032981457999999998, 2020-06-12T02:39:10.962 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:10.962 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:10.962 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:10.962 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:10.963 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:10.963 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13714 2020-06-12T02:39:10.963 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.963 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:10.963 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.964 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:10.964 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460921+0000", 2020-06-12T02:39:10.964 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:10.964 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.965 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.965 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:10.965 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460921+0000", 2020-06-12T02:39:10.965 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6996999999999995e-05 2020-06-12T02:39:10.966 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.966 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.966 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:10.966 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460998+0000", 2020-06-12T02:39:10.966 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4529999999999999e-06 2020-06-12T02:39:10.967 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.967 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.967 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:10.967 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461005+0000", 2020-06-12T02:39:10.967 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1722000000000001e-05 2020-06-12T02:39:10.967 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.968 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.968 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:10.968 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461017+0000", 2020-06-12T02:39:10.969 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4839999999999998e-06 2020-06-12T02:39:10.969 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.969 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.970 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:10.970 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461021+0000", 2020-06-12T02:39:10.970 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.029157975999999999 2020-06-12T02:39:10.970 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.971 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.971 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:10.971 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.490179+0000", 2020-06-12T02:39:10.971 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010966700000000001 2020-06-12T02:39:10.971 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.972 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.972 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:10.972 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.490289+0000", 2020-06-12T02:39:10.972 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00032026800000000002 2020-06-12T02:39:10.972 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.973 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.973 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:10.973 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.490609+0000", 2020-06-12T02:39:10.973 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001102947 2020-06-12T02:39:10.973 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.974 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.974 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:10.974 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.491712+0000", 2020-06-12T02:39:10.974 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021050169999999998 2020-06-12T02:39:10.974 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.974 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.975 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:10.975 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493817+0000", 2020-06-12T02:39:10.975 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5927000000000006e-05 2020-06-12T02:39:10.975 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.975 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.976 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:10.976 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493903+0000", 2020-06-12T02:39:10.976 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.032981457999999998 2020-06-12T02:39:10.976 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.976 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:10.976 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.977 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.977 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.977 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13715 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e58)", 2020-06-12T02:39:10.977 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.461062+0000", 2020-06-12T02:39:10.977 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 42.345269975000001, 2020-06-12T02:39:10.978 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.035208338999999998, 2020-06-12T02:39:10.978 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:10.978 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:10.978 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:10.978 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:10.979 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:10.979 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13715 2020-06-12T02:39:10.979 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.979 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:10.979 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.980 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:10.980 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461062+0000", 2020-06-12T02:39:10.980 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:10.981 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.981 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.981 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:10.981 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461062+0000", 2020-06-12T02:39:10.982 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4839999999999998e-05 2020-06-12T02:39:10.982 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.982 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.982 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:10.982 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461107+0000", 2020-06-12T02:39:10.983 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4859999999999997e-06 2020-06-12T02:39:10.983 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.983 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.983 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:10.983 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461112+0000", 2020-06-12T02:39:10.983 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1742999999999999e-05 2020-06-12T02:39:10.984 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.984 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.984 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:10.984 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461124+0000", 2020-06-12T02:39:10.984 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7299999999999996e-06 2020-06-12T02:39:10.985 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.985 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.985 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:10.985 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461129+0000", 2020-06-12T02:39:10.985 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.030397368000000001 2020-06-12T02:39:10.986 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.986 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.986 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:10.986 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.491526+0000", 2020-06-12T02:39:10.986 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010076799999999999 2020-06-12T02:39:10.986 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.987 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.987 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:10.987 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.491627+0000", 2020-06-12T02:39:10.987 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031869849999999998 2020-06-12T02:39:10.987 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.988 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.988 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:10.988 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.494814+0000", 2020-06-12T02:39:10.988 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000108915 2020-06-12T02:39:10.988 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.988 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.989 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:10.989 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.494923+0000", 2020-06-12T02:39:10.989 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001347504 2020-06-12T02:39:10.989 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.989 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.990 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:10.990 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.496270+0000", 2020-06-12T02:39:10.990 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.035208338999999998 2020-06-12T02:39:10.990 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.990 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:10.990 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:10.991 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.991 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.991 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:16706 3.as0 3:5cb723d2:::smithi10729407-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e61)", 2020-06-12T02:39:10.991 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:37.289063+0000", 2020-06-12T02:39:10.991 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 33.517268950999998, 2020-06-12T02:39:10.992 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026146086999999998, 2020-06-12T02:39:10.992 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:10.992 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:10.992 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:10.992 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:10.993 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:10.993 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 16706 2020-06-12T02:39:10.994 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.994 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:10.994 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.994 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:10.995 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289063+0000", 2020-06-12T02:39:10.995 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:10.995 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.995 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.995 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:10.996 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289063+0000", 2020-06-12T02:39:10.996 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3776000000000002e-05 2020-06-12T02:39:10.996 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.996 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.996 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:10.996 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289107+0000", 2020-06-12T02:39:10.997 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7039999999999999e-06 2020-06-12T02:39:10.997 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.997 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.997 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:10.997 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289113+0000", 2020-06-12T02:39:10.998 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3259e-05 2020-06-12T02:39:10.998 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.998 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.998 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:10.998 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289126+0000", 2020-06-12T02:39:10.999 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1999999999999996e-06 2020-06-12T02:39:10.999 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:10.999 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:10.999 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:10.999 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289130+0000", 2020-06-12T02:39:10.999 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0010912820000000001 2020-06-12T02:39:11.000 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.000 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.000 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.000 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.290221+0000", 2020-06-12T02:39:11.000 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3166e-05 2020-06-12T02:39:11.001 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.001 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.001 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:11.001 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.290265+0000", 2020-06-12T02:39:11.001 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015984187 2020-06-12T02:39:11.001 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.002 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.002 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.002 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.306249+0000", 2020-06-12T02:39:11.002 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4623000000000006e-05 2020-06-12T02:39:11.003 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.003 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.003 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:11.003 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.306333+0000", 2020-06-12T02:39:11.003 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0068573690000000003 2020-06-12T02:39:11.003 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.004 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.004 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.004 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.313191+0000", 2020-06-12T02:39:11.004 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7804000000000002e-05 2020-06-12T02:39:11.004 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.005 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.005 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:11.005 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.313279+0000", 2020-06-12T02:39:11.005 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001930717 2020-06-12T02:39:11.005 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.005 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.006 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:11.006 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.315209+0000", 2020-06-12T02:39:11.006 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026146086999999998 2020-06-12T02:39:11.006 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.006 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:11.007 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.007 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.007 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.007 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18051 3.as0 3:5cb723d2:::smithi10729407-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:39:11.008 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.329078+0000", 2020-06-12T02:39:11.008 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 29.477254401, 2020-06-12T02:39:11.009 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026465103, 2020-06-12T02:39:11.009 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:11.009 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:11.009 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:11.010 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:11.010 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:11.010 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18051 2020-06-12T02:39:11.010 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.010 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:11.011 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.011 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:11.011 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329078+0000", 2020-06-12T02:39:11.011 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:11.011 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.012 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.012 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:11.012 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329078+0000", 2020-06-12T02:39:11.012 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5392000000000001e-05 2020-06-12T02:39:11.012 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.012 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.013 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:11.013 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329133+0000", 2020-06-12T02:39:11.013 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5329999999999997e-06 2020-06-12T02:39:11.013 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.013 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.014 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:11.014 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329141+0000", 2020-06-12T02:39:11.014 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3459999999999999e-05 2020-06-12T02:39:11.014 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.014 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.014 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:11.015 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329164+0000", 2020-06-12T02:39:11.015 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8170000000000001e-06 2020-06-12T02:39:11.015 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.015 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.015 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:11.016 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329171+0000", 2020-06-12T02:39:11.016 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013005208000000001 2020-06-12T02:39:11.016 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.016 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.016 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.017 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.342176+0000", 2020-06-12T02:39:11.017 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3560999999999998e-05 2020-06-12T02:39:11.017 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.017 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.017 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:11.017 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.342230+0000", 2020-06-12T02:39:11.018 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0080818490000000003 2020-06-12T02:39:11.018 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.018 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.018 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.018 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.350312+0000", 2020-06-12T02:39:11.019 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0167000000000006e-05 2020-06-12T02:39:11.019 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.019 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.019 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:11.019 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.350402+0000", 2020-06-12T02:39:11.019 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051411160000000003 2020-06-12T02:39:11.020 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.020 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.020 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:11.020 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.355543+0000", 2020-06-12T02:39:11.020 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026465103 2020-06-12T02:39:11.021 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.021 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:11.021 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.021 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.021 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.022 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18053 3.as0 3:5732055a:::smithi10729407-8:head [append 0~434176] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:11.022 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.334425+0000", 2020-06-12T02:39:11.022 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 29.471906833999999, 2020-06-12T02:39:11.022 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.029616703000000001, 2020-06-12T02:39:11.022 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:11.023 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:11.023 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:11.023 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:11.023 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:11.023 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18053 2020-06-12T02:39:11.024 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.024 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:11.024 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.024 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:11.024 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334425+0000", 2020-06-12T02:39:11.024 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:11.025 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.025 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.025 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:11.025 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334425+0000", 2020-06-12T02:39:11.025 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047283900000000002 2020-06-12T02:39:11.026 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.026 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.026 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:11.026 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334898+0000", 2020-06-12T02:39:11.026 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9100000000000004e-06 2020-06-12T02:39:11.026 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.027 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.027 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:11.027 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334903+0000", 2020-06-12T02:39:11.027 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4260000000000003e-06 2020-06-12T02:39:11.027 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.028 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.028 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:11.028 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334911+0000", 2020-06-12T02:39:11.028 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1060000000000002e-06 2020-06-12T02:39:11.028 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.028 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.029 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:11.029 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334916+0000", 2020-06-12T02:39:11.029 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010838316000000001 2020-06-12T02:39:11.029 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.029 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.030 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.030 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.345754+0000", 2020-06-12T02:39:11.030 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000103872 2020-06-12T02:39:11.030 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.030 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.031 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:11.031 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.345858+0000", 2020-06-12T02:39:11.031 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0059798739999999996 2020-06-12T02:39:11.031 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.031 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.031 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.032 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.351838+0000", 2020-06-12T02:39:11.032 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000104724 2020-06-12T02:39:11.033 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.033 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.033 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:11.033 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.351942+0000", 2020-06-12T02:39:11.033 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000572687 2020-06-12T02:39:11.034 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.034 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.034 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:11.034 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.352515+0000", 2020-06-12T02:39:11.034 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002905641 2020-06-12T02:39:11.035 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.035 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.035 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:11.035 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.355421+0000", 2020-06-12T02:39:11.035 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0085358350000000003 2020-06-12T02:39:11.036 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.036 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.036 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:11.036 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.363956+0000", 2020-06-12T02:39:11.036 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5473e-05 2020-06-12T02:39:11.037 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.037 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.037 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:11.037 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.364042+0000", 2020-06-12T02:39:11.037 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.029616703000000001 2020-06-12T02:39:11.038 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.038 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:11.038 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.038 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.038 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.038 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18054 3.as0 3:5732055a:::smithi10729407-8:head [append 0~638976] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:11.039 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.338171+0000", 2020-06-12T02:39:11.039 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 29.468161331000001, 2020-06-12T02:39:11.039 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.030067120999999999, 2020-06-12T02:39:11.039 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:11.039 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:11.040 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:11.040 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:11.040 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:11.040 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18054 2020-06-12T02:39:11.040 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.041 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:11.041 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.041 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:11.041 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338171+0000", 2020-06-12T02:39:11.042 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:11.042 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.042 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.042 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:11.042 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338171+0000", 2020-06-12T02:39:11.043 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00068138900000000002 2020-06-12T02:39:11.043 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.043 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.043 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:11.043 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338852+0000", 2020-06-12T02:39:11.044 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.767e-06 2020-06-12T02:39:11.044 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.044 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.044 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:11.044 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338858+0000", 2020-06-12T02:39:11.044 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8999999999999995e-06 2020-06-12T02:39:11.045 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.045 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.045 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:11.045 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338867+0000", 2020-06-12T02:39:11.045 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.262e-06 2020-06-12T02:39:11.046 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.046 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.046 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:11.046 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338871+0000", 2020-06-12T02:39:11.046 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0074839060000000002 2020-06-12T02:39:11.047 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.047 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.047 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.047 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346355+0000", 2020-06-12T02:39:11.047 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1435999999999999e-05 2020-06-12T02:39:11.048 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.048 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.048 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:11.048 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346416+0000", 2020-06-12T02:39:11.048 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0092109879999999998 2020-06-12T02:39:11.049 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.049 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.049 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.049 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.355627+0000", 2020-06-12T02:39:11.049 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7355999999999998e-05 2020-06-12T02:39:11.049 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.050 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.050 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:11.050 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.355685+0000", 2020-06-12T02:39:11.050 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00074711999999999999 2020-06-12T02:39:11.050 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.051 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.051 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:11.051 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.356432+0000", 2020-06-12T02:39:11.051 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010659572000000001 2020-06-12T02:39:11.051 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.052 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.052 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:11.052 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.367091+0000", 2020-06-12T02:39:11.052 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0010713210000000001 2020-06-12T02:39:11.052 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.052 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.053 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:11.053 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.368163+0000", 2020-06-12T02:39:11.053 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5104000000000005e-05 2020-06-12T02:39:11.053 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.053 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.054 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:11.054 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.368238+0000", 2020-06-12T02:39:11.054 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.030067120999999999 2020-06-12T02:39:11.054 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.054 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:11.054 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.055 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.055 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.055 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18055 3.as0 3:5732055a:::smithi10729407-8:head [append 0~483328] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:11.055 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.340740+0000", 2020-06-12T02:39:11.055 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 29.465592482000002, 2020-06-12T02:39:11.056 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028669050000000001, 2020-06-12T02:39:11.056 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:11.056 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:11.056 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:11.056 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:11.057 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:11.057 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18055 2020-06-12T02:39:11.057 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.057 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:11.057 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.057 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:11.058 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.340740+0000", 2020-06-12T02:39:11.058 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:11.058 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.058 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.058 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:11.059 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.340740+0000", 2020-06-12T02:39:11.059 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00045126900000000002 2020-06-12T02:39:11.059 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.059 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.059 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:11.059 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341191+0000", 2020-06-12T02:39:11.060 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5630000000000003e-06 2020-06-12T02:39:11.060 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.060 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.060 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:11.060 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341198+0000", 2020-06-12T02:39:11.061 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4467999999999999e-05 2020-06-12T02:39:11.061 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.061 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.061 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:11.062 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341213+0000", 2020-06-12T02:39:11.062 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7540000000000003e-06 2020-06-12T02:39:11.062 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.062 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.062 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:11.062 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341219+0000", 2020-06-12T02:39:11.063 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0052272680000000002 2020-06-12T02:39:11.063 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.063 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.063 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.064 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346446+0000", 2020-06-12T02:39:11.064 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4742999999999998e-05 2020-06-12T02:39:11.064 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.064 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.064 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:11.065 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346531+0000", 2020-06-12T02:39:11.065 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013203138999999999 2020-06-12T02:39:11.065 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.065 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.065 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.066 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.359734+0000", 2020-06-12T02:39:11.066 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000100277 2020-06-12T02:39:11.066 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.066 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.066 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:11.066 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.359834+0000", 2020-06-12T02:39:11.067 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054998099999999995 2020-06-12T02:39:11.067 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.067 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.067 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:11.067 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.360384+0000", 2020-06-12T02:39:11.068 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0067087179999999998 2020-06-12T02:39:11.068 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.068 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.068 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:11.068 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.367093+0000", 2020-06-12T02:39:11.068 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0022746020000000001 2020-06-12T02:39:11.069 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.069 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.069 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:11.069 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369367+0000", 2020-06-12T02:39:11.069 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1267999999999999e-05 2020-06-12T02:39:11.070 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.070 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.070 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:11.070 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369409+0000", 2020-06-12T02:39:11.070 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028669050000000001 2020-06-12T02:39:11.071 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.071 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:11.071 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.071 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.071 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.071 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18056 3.as0 3:5732055a:::smithi10729407-8:head [append 0~49152] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:11.072 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.341270+0000", 2020-06-12T02:39:11.073 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 29.465061692999999, 2020-06-12T02:39:11.073 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028350462999999999, 2020-06-12T02:39:11.073 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:11.073 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:11.073 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:11.074 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:11.074 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:11.074 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18056 2020-06-12T02:39:11.074 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.074 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:11.074 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.075 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:11.075 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341270+0000", 2020-06-12T02:39:11.075 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:11.075 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.075 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.076 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:11.076 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341270+0000", 2020-06-12T02:39:11.076 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000114082 2020-06-12T02:39:11.076 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.076 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.077 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:11.077 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341384+0000", 2020-06-12T02:39:11.077 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3289999999999999e-06 2020-06-12T02:39:11.077 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.077 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.077 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:11.078 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341391+0000", 2020-06-12T02:39:11.078 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3339999999999999e-05 2020-06-12T02:39:11.078 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.078 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.079 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:11.079 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341404+0000", 2020-06-12T02:39:11.079 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4e-06 2020-06-12T02:39:11.079 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.080 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.080 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:11.080 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341410+0000", 2020-06-12T02:39:11.080 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051518600000000003 2020-06-12T02:39:11.080 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.080 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.081 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.081 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346561+0000", 2020-06-12T02:39:11.081 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1662999999999994e-05 2020-06-12T02:39:11.081 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.081 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.082 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:11.082 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346643+0000", 2020-06-12T02:39:11.082 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017471605000000001 2020-06-12T02:39:11.082 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.082 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.083 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.083 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.364115+0000", 2020-06-12T02:39:11.083 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011059400000000001 2020-06-12T02:39:11.083 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.083 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.084 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:11.084 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.364225+0000", 2020-06-12T02:39:11.084 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000611262 2020-06-12T02:39:11.084 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.084 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.084 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:11.085 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.364837+0000", 2020-06-12T02:39:11.085 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023767710000000002 2020-06-12T02:39:11.085 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.085 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.085 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:11.086 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.367213+0000", 2020-06-12T02:39:11.086 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023325479999999998 2020-06-12T02:39:11.086 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.086 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.086 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:11.087 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369546+0000", 2020-06-12T02:39:11.087 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5009e-05 2020-06-12T02:39:11.088 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.088 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.088 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:11.088 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369621+0000", 2020-06-12T02:39:11.088 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028350462999999999 2020-06-12T02:39:11.089 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.089 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:11.089 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.089 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.089 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.090 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18057 3.as0 3:5732055a:::smithi10729407-8:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:11.090 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.341456+0000", 2020-06-12T02:39:11.090 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 29.464876533000002, 2020-06-12T02:39:11.090 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028443018, 2020-06-12T02:39:11.090 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:11.091 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:11.091 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:11.091 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:11.091 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:11.091 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18057 2020-06-12T02:39:11.092 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.092 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:11.092 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.092 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:11.092 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341456+0000", 2020-06-12T02:39:11.092 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:11.093 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.093 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.093 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:11.093 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341456+0000", 2020-06-12T02:39:11.093 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6036000000000003e-05 2020-06-12T02:39:11.094 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.094 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.094 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:11.094 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341522+0000", 2020-06-12T02:39:11.094 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2210000000000002e-06 2020-06-12T02:39:11.095 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.095 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.095 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:11.095 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341528+0000", 2020-06-12T02:39:11.095 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0338e-05 2020-06-12T02:39:11.095 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.096 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.096 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:11.096 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341538+0000", 2020-06-12T02:39:11.096 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8639999999999998e-06 2020-06-12T02:39:11.096 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.097 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.097 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:11.097 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341542+0000", 2020-06-12T02:39:11.097 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051405009999999996 2020-06-12T02:39:11.097 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.098 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.098 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.098 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346683+0000", 2020-06-12T02:39:11.098 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9561999999999995e-05 2020-06-12T02:39:11.098 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.098 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.099 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:11.099 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346762+0000", 2020-06-12T02:39:11.099 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019267197999999999 2020-06-12T02:39:11.099 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.099 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.100 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.100 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.366029+0000", 2020-06-12T02:39:11.100 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2634999999999997e-05 2020-06-12T02:39:11.100 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.100 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.100 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:11.101 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.366122+0000", 2020-06-12T02:39:11.101 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00026907500000000002 2020-06-12T02:39:11.101 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.101 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.101 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:11.102 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.366391+0000", 2020-06-12T02:39:11.102 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0018939409999999999 2020-06-12T02:39:11.102 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.102 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.102 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:11.103 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.368285+0000", 2020-06-12T02:39:11.103 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001549405 2020-06-12T02:39:11.103 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.103 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.103 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:11.103 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369834+0000", 2020-06-12T02:39:11.104 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4241999999999999e-05 2020-06-12T02:39:11.104 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.104 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.104 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:11.104 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369899+0000", 2020-06-12T02:39:11.105 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028443018 2020-06-12T02:39:11.105 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.105 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:11.105 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.105 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.105 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.106 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18058 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:39:11.106 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.341580+0000", 2020-06-12T02:39:11.106 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 29.464751626000002, 2020-06-12T02:39:11.106 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.030449559000000001, 2020-06-12T02:39:11.106 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:11.107 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:11.107 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:11.107 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:11.107 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:11.107 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18058 2020-06-12T02:39:11.107 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.108 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:11.108 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.108 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:11.108 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341580+0000", 2020-06-12T02:39:11.108 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:11.109 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.109 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.109 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:11.109 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341580+0000", 2020-06-12T02:39:11.109 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1185e-05 2020-06-12T02:39:11.110 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.110 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.110 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:11.110 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341622+0000", 2020-06-12T02:39:11.110 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1599999999999997e-06 2020-06-12T02:39:11.110 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.111 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.111 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:11.111 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341627+0000", 2020-06-12T02:39:11.111 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3546e-05 2020-06-12T02:39:11.111 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.112 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.112 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:11.112 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341640+0000", 2020-06-12T02:39:11.112 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9779999999999999e-06 2020-06-12T02:39:11.113 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.113 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.113 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:11.113 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341644+0000", 2020-06-12T02:39:11.113 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051433700000000004 2020-06-12T02:39:11.114 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.118 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.119 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.119 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346788+0000", 2020-06-12T02:39:11.119 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1622000000000001e-05 2020-06-12T02:39:11.119 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.120 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.120 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:11.120 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346839+0000", 2020-06-12T02:39:11.120 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020283958000000001 2020-06-12T02:39:11.120 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.121 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.121 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.121 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.367123+0000", 2020-06-12T02:39:11.121 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1997999999999997e-05 2020-06-12T02:39:11.121 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.122 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.122 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:11.122 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.367205+0000", 2020-06-12T02:39:11.122 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036875440000000001 2020-06-12T02:39:11.122 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.122 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.123 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.123 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.370893+0000", 2020-06-12T02:39:11.123 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000100604 2020-06-12T02:39:11.123 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.123 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.124 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:11.124 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.370993+0000", 2020-06-12T02:39:11.124 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0010365940000000001 2020-06-12T02:39:11.124 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.124 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.124 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:11.125 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.372030+0000", 2020-06-12T02:39:11.125 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.030449559000000001 2020-06-12T02:39:11.125 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.125 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:11.125 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.126 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.126 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:11.126 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:39:11.142 INFO:tasks.rados.rados.0.smithi107.stdout:10447: writing smithi10729407-13 from 2080768 to 2637824 tid 1 2020-06-12T02:39:11.142 INFO:tasks.rados.rados.0.smithi107.stdout:10447: writing smithi10729407-13 from 2637824 to 3432448 tid 2 2020-06-12T02:39:11.143 INFO:tasks.rados.rados.0.smithi107.stdout:10447: writing smithi10729407-13 from 3432448 to 4071424 tid 3 2020-06-12T02:39:11.143 INFO:tasks.rados.rados.0.smithi107.stdout:10447: writing smithi10729407-13 from 4071424 to 4243456 tid 4 2020-06-12T02:39:11.143 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:39:11.143 INFO:tasks.rados.rados.0.smithi107.stdout:10448: seq_num 10447 ranges {0=573440,573440=679936,1253376=802816,2056192=122880} 2020-06-12T02:39:11.143 INFO:tasks.rados.rados.0.smithi107.stdout:10448: writing smithi10729407-14 from 0 to 573440 tid 1 2020-06-12T02:39:11.144 INFO:tasks.rados.rados.0.smithi107.stdout:10448: writing smithi10729407-14 from 573440 to 1253376 tid 2 2020-06-12T02:39:11.144 INFO:tasks.rados.rados.0.smithi107.stdout:10448: writing smithi10729407-14 from 1253376 to 2056192 tid 3 2020-06-12T02:39:11.144 INFO:tasks.rados.rados.0.smithi107.stdout:10448: writing smithi10729407-14 from 2056192 to 2179072 tid 4 2020-06-12T02:39:11.144 INFO:tasks.rados.rados.0.smithi107.stdout:10449: delete oid 1 current snap is 0 2020-06-12T02:39:11.144 INFO:tasks.rados.rados.0.smithi107.stdout:10443: finishing write tid 2 to smithi10729407-11 2020-06-12T02:39:11.144 INFO:tasks.rados.rados.0.smithi107.stdout:10443: finishing write tid 3 to smithi10729407-11 2020-06-12T02:39:11.145 INFO:tasks.rados.rados.0.smithi107.stdout:10443: finishing write tid 4 to smithi10729407-11 2020-06-12T02:39:11.145 INFO:tasks.rados.rados.0.smithi107.stdout:10443: finishing write tid 6 to smithi10729407-11 2020-06-12T02:39:11.145 INFO:tasks.rados.rados.0.smithi107.stdout:10443: finishing write tid 7 to smithi10729407-11 2020-06-12T02:39:11.145 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1704 (ObjNum 10442 snap 0 seq_num 10442) dirty exists 2020-06-12T02:39:11.145 INFO:tasks.rados.rados.0.smithi107.stdout:10443: left oid 11 (ObjNum 10442 snap 0 seq_num 10442) 2020-06-12T02:39:11.146 INFO:tasks.rados.rados.0.smithi107.stdout:10445: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:11.146 INFO:tasks.rados.rados.0.smithi107.stdout:10445: finishing write tid 2 to smithi10729407-20 2020-06-12T02:39:11.146 INFO:tasks.rados.rados.0.smithi107.stdout:10445: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:11.146 INFO:tasks.rados.rados.0.smithi107.stdout:10445: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:11.146 INFO:tasks.rados.rados.0.smithi107.stdout:10445: finishing write tid 6 to smithi10729407-20 2020-06-12T02:39:11.147 INFO:tasks.rados.rados.0.smithi107.stdout:10445: finishing write tid 7 to smithi10729407-20 2020-06-12T02:39:11.147 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 812 (ObjNum 10444 snap 0 seq_num 10444) dirty exists 2020-06-12T02:39:11.147 INFO:tasks.rados.rados.0.smithi107.stdout:10445: left oid 20 (ObjNum 10444 snap 0 seq_num 10444) 2020-06-12T02:39:11.147 INFO:tasks.rados.rados.0.smithi107.stdout:10446: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:11.147 INFO:tasks.rados.rados.0.smithi107.stdout:10446: finishing write tid 2 to smithi10729407-5 2020-06-12T02:39:11.147 INFO:tasks.rados.rados.0.smithi107.stdout:10446: finishing write tid 3 to smithi10729407-5 2020-06-12T02:39:11.148 INFO:tasks.rados.rados.0.smithi107.stdout:10446: finishing write tid 4 to smithi10729407-5 2020-06-12T02:39:11.148 INFO:tasks.rados.rados.0.smithi107.stdout:10446: finishing write tid 5 to smithi10729407-5 2020-06-12T02:39:11.148 INFO:tasks.rados.rados.0.smithi107.stdout:10446: finishing write tid 7 to smithi10729407-5 2020-06-12T02:39:11.148 INFO:tasks.rados.rados.0.smithi107.stdout:10446: finishing write tid 8 to smithi10729407-5 2020-06-12T02:39:11.148 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 993 (ObjNum 10445 snap 0 seq_num 10445) dirty exists 2020-06-12T02:39:11.149 INFO:tasks.rados.rados.0.smithi107.stdout:10446: left oid 5 (ObjNum 10445 snap 0 seq_num 10445) 2020-06-12T02:39:11.149 INFO:tasks.rados.rados.0.smithi107.stdout:10447: finishing write tid 1 to smithi10729407-13 2020-06-12T02:39:11.149 INFO:tasks.rados.rados.0.smithi107.stdout:10447: finishing write tid 2 to smithi10729407-13 2020-06-12T02:39:11.149 INFO:tasks.rados.rados.0.smithi107.stdout:10447: finishing write tid 3 to smithi10729407-13 2020-06-12T02:39:11.149 INFO:tasks.rados.rados.0.smithi107.stdout:10447: finishing write tid 4 to smithi10729407-13 2020-06-12T02:39:11.149 INFO:tasks.rados.rados.0.smithi107.stdout:10447: finishing write tid 6 to smithi10729407-13 2020-06-12T02:39:11.150 INFO:tasks.rados.rados.0.smithi107.stdout:10447: finishing write tid 7 to smithi10729407-13 2020-06-12T02:39:11.150 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1748 (ObjNum 10446 snap 0 seq_num 10446) dirty exists 2020-06-12T02:39:11.150 INFO:tasks.rados.rados.0.smithi107.stdout:10447: left oid 13 (ObjNum 10446 snap 0 seq_num 10446) 2020-06-12T02:39:11.150 INFO:tasks.rados.rados.0.smithi107.stdout:10448: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:11.150 INFO:tasks.rados.rados.0.smithi107.stdout:10448: finishing write tid 2 to smithi10729407-14 2020-06-12T02:39:11.151 INFO:tasks.rados.rados.0.smithi107.stdout:10448: finishing write tid 3 to smithi10729407-14 2020-06-12T02:39:11.151 INFO:tasks.rados.rados.0.smithi107.stdout:10448: finishing write tid 4 to smithi10729407-14 2020-06-12T02:39:11.151 INFO:tasks.rados.rados.0.smithi107.stdout:10448: finishing write tid 6 to smithi10729407-14 2020-06-12T02:39:11.151 INFO:tasks.rados.rados.0.smithi107.stdout:10448: finishing write tid 7 to smithi10729407-14 2020-06-12T02:39:11.151 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1687 (ObjNum 10447 snap 0 seq_num 10447) dirty exists 2020-06-12T02:39:11.151 INFO:tasks.rados.rados.0.smithi107.stdout:10448: left oid 14 (ObjNum 10447 snap 0 seq_num 10447) 2020-06-12T02:39:11.152 INFO:tasks.rados.rados.0.smithi107.stdout:10443: done (5 left) 2020-06-12T02:39:11.152 INFO:tasks.rados.rados.0.smithi107.stdout:10445: done (4 left) 2020-06-12T02:39:11.152 INFO:tasks.rados.rados.0.smithi107.stdout:10446: done (3 left) 2020-06-12T02:39:11.152 INFO:tasks.rados.rados.0.smithi107.stdout:10447: done (2 left) 2020-06-12T02:39:11.152 INFO:tasks.rados.rados.0.smithi107.stdout:10448: done (1 left) 2020-06-12T02:39:11.153 INFO:tasks.rados.rados.0.smithi107.stdout:10449: done (0 left) 2020-06-12T02:39:11.153 INFO:tasks.rados.rados.0.smithi107.stdout:10450: delete oid 17 current snap is 0 2020-06-12T02:39:11.153 INFO:tasks.rados.rados.0.smithi107.stdout:10450: done (0 left) 2020-06-12T02:39:11.153 INFO:tasks.rados.rados.0.smithi107.stdout:10451: delete oid 10 current snap is 0 2020-06-12T02:39:11.153 INFO:tasks.rados.rados.0.smithi107.stdout:10451: done (0 left) 2020-06-12T02:39:11.153 INFO:tasks.rados.rados.0.smithi107.stdout:10452: delete oid 14 current snap is 0 2020-06-12T02:39:11.154 INFO:tasks.rados.rados.0.smithi107.stdout:10452: done (0 left) 2020-06-12T02:39:11.154 INFO:tasks.rados.rados.0.smithi107.stdout:10453: delete oid 9 current snap is 0 2020-06-12T02:39:11.154 INFO:tasks.rados.rados.0.smithi107.stdout:10453: done (0 left) 2020-06-12T02:39:11.154 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:39:11.154 INFO:tasks.rados.rados.0.smithi107.stdout:10454: seq_num 10453 ranges {0=671744,671744=786432,1458176=794624,2252800=24576} 2020-06-12T02:39:11.155 INFO:tasks.rados.rados.0.smithi107.stdout:10454: writing smithi10729407-3 from 0 to 671744 tid 1 2020-06-12T02:39:11.155 INFO:tasks.rados.rados.0.smithi107.stdout:10454: writing smithi10729407-3 from 671744 to 1458176 tid 2 2020-06-12T02:39:11.155 INFO:tasks.rados.rados.0.smithi107.stdout:10454: writing smithi10729407-3 from 1458176 to 2252800 tid 3 2020-06-12T02:39:11.155 INFO:tasks.rados.rados.0.smithi107.stdout:10454: writing smithi10729407-3 from 2252800 to 2277376 tid 4 2020-06-12T02:39:11.155 INFO:tasks.rados.rados.0.smithi107.stdout:10455: delete oid 9 current snap is 0 2020-06-12T02:39:11.155 INFO:tasks.rados.rados.0.smithi107.stdout:10454: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:11.156 INFO:tasks.rados.rados.0.smithi107.stdout:10455: done (1 left) 2020-06-12T02:39:11.156 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:39:11.156 INFO:tasks.rados.rados.0.smithi107.stdout:10456: seq_num 10455 ranges {0=704512,704512=606208,1310720=761856,2072576=237568} 2020-06-12T02:39:11.156 INFO:tasks.rados.rados.0.smithi107.stdout:10456: writing smithi10729407-7 from 0 to 704512 tid 1 2020-06-12T02:39:11.156 INFO:tasks.rados.rados.0.smithi107.stdout:10456: writing smithi10729407-7 from 704512 to 1310720 tid 2 2020-06-12T02:39:11.157 INFO:tasks.rados.rados.0.smithi107.stdout:10456: writing smithi10729407-7 from 1310720 to 2072576 tid 3 2020-06-12T02:39:11.157 INFO:tasks.rados.rados.0.smithi107.stdout:10456: writing smithi10729407-7 from 2072576 to 2310144 tid 4 2020-06-12T02:39:11.157 INFO:tasks.rados.rados.0.smithi107.stdout:10457: delete oid 13 current snap is 0 2020-06-12T02:39:11.157 INFO:tasks.rados.rados.0.smithi107.stdout:10456: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:11.157 INFO:tasks.rados.rados.0.smithi107.stdout:10454: finishing write tid 2 to smithi10729407-3 2020-06-12T02:39:11.157 INFO:tasks.rados.rados.0.smithi107.stdout:10454: finishing write tid 3 to smithi10729407-3 2020-06-12T02:39:11.158 INFO:tasks.rados.rados.0.smithi107.stdout:10454: finishing write tid 4 to smithi10729407-3 2020-06-12T02:39:11.158 INFO:tasks.rados.rados.0.smithi107.stdout:10454: finishing write tid 6 to smithi10729407-3 2020-06-12T02:39:11.158 INFO:tasks.rados.rados.0.smithi107.stdout:10457: done (2 left) 2020-06-12T02:39:11.158 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:39:11.158 INFO:tasks.rados.rados.0.smithi107.stdout:10458: seq_num 10457 ranges {0=737280,737280=778240,1515520=778240,2293760=49152} 2020-06-12T02:39:11.159 INFO:tasks.rados.rados.0.smithi107.stdout:10458: writing smithi10729407-10 from 0 to 737280 tid 1 2020-06-12T02:39:11.159 INFO:tasks.rados.rados.0.smithi107.stdout:10458: writing smithi10729407-10 from 737280 to 1515520 tid 2 2020-06-12T02:39:11.159 INFO:tasks.rados.rados.0.smithi107.stdout:10458: writing smithi10729407-10 from 1515520 to 2293760 tid 3 2020-06-12T02:39:11.159 INFO:tasks.rados.rados.0.smithi107.stdout:10458: writing smithi10729407-10 from 2293760 to 2342912 tid 4 2020-06-12T02:39:11.159 INFO:tasks.rados.rados.0.smithi107.stdout:10459: delete oid 19 current snap is 0 2020-06-12T02:39:11.159 INFO:tasks.rados.rados.0.smithi107.stdout:10454: finishing write tid 7 to smithi10729407-3 2020-06-12T02:39:11.160 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1870 (ObjNum 10453 snap 0 seq_num 10453) dirty exists 2020-06-12T02:39:11.160 INFO:tasks.rados.rados.0.smithi107.stdout:10454: left oid 3 (ObjNum 10453 snap 0 seq_num 10453) 2020-06-12T02:39:11.160 INFO:tasks.rados.rados.0.smithi107.stdout:10456: finishing write tid 2 to smithi10729407-7 2020-06-12T02:39:11.160 INFO:tasks.rados.rados.0.smithi107.stdout:10456: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:11.160 INFO:tasks.rados.rados.0.smithi107.stdout:10456: finishing write tid 4 to smithi10729407-7 2020-06-12T02:39:11.161 INFO:tasks.rados.rados.0.smithi107.stdout:10456: finishing write tid 6 to smithi10729407-7 2020-06-12T02:39:11.161 INFO:tasks.rados.rados.0.smithi107.stdout:10456: finishing write tid 7 to smithi10729407-7 2020-06-12T02:39:11.161 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1741 (ObjNum 10455 snap 0 seq_num 10455) dirty exists 2020-06-12T02:39:11.161 INFO:tasks.rados.rados.0.smithi107.stdout:10456: left oid 7 (ObjNum 10455 snap 0 seq_num 10455) 2020-06-12T02:39:11.161 INFO:tasks.rados.rados.0.smithi107.stdout:10458: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:11.161 INFO:tasks.rados.rados.0.smithi107.stdout:10458: finishing write tid 2 to smithi10729407-10 2020-06-12T02:39:11.162 INFO:tasks.rados.rados.0.smithi107.stdout:10454: done (3 left) 2020-06-12T02:39:11.162 INFO:tasks.rados.rados.0.smithi107.stdout:10456: done (2 left) 2020-06-12T02:39:11.162 INFO:tasks.rados.rados.0.smithi107.stdout:10459: done (1 left) 2020-06-12T02:39:11.162 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:39:11.162 INFO:tasks.rados.rados.0.smithi107.stdout:10460: seq_num 10459 ranges {1671168=778240,2449408=540672,2990080=745472,3735552=319488} 2020-06-12T02:39:11.163 INFO:tasks.rados.rados.0.smithi107.stdout:10460: writing smithi10729407-8 from 1671168 to 2449408 tid 1 2020-06-12T02:39:11.163 INFO:tasks.rados.rados.0.smithi107.stdout:10460: writing smithi10729407-8 from 2449408 to 2990080 tid 2 2020-06-12T02:39:11.163 INFO:tasks.rados.rados.0.smithi107.stdout:10460: writing smithi10729407-8 from 2990080 to 3735552 tid 3 2020-06-12T02:39:11.163 INFO:tasks.rados.rados.0.smithi107.stdout:10460: writing smithi10729407-8 from 3735552 to 4055040 tid 4 2020-06-12T02:39:11.163 INFO:tasks.rados.rados.0.smithi107.stdout:10461: delete oid 19 current snap is 0 2020-06-12T02:39:11.163 INFO:tasks.rados.rados.0.smithi107.stdout:10458: finishing write tid 3 to smithi10729407-10 2020-06-12T02:39:11.164 INFO:tasks.rados.rados.0.smithi107.stdout:10458: finishing write tid 4 to smithi10729407-10 2020-06-12T02:39:11.164 INFO:tasks.rados.rados.0.smithi107.stdout:10458: finishing write tid 6 to smithi10729407-10 2020-06-12T02:39:11.164 INFO:tasks.rados.rados.0.smithi107.stdout:10458: finishing write tid 7 to smithi10729407-10 2020-06-12T02:39:11.164 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 762 (ObjNum 10457 snap 0 seq_num 10457) dirty exists 2020-06-12T02:39:11.164 INFO:tasks.rados.rados.0.smithi107.stdout:10458: left oid 10 (ObjNum 10457 snap 0 seq_num 10457) 2020-06-12T02:39:11.165 INFO:tasks.rados.rados.0.smithi107.stdout:10458: done (2 left) 2020-06-12T02:39:11.165 INFO:tasks.rados.rados.0.smithi107.stdout:10461: done (1 left) 2020-06-12T02:39:11.165 INFO:tasks.rados.rados.0.smithi107.stdout:10462: delete oid 19 current snap is 0 2020-06-12T02:39:11.165 INFO:tasks.rados.rados.0.smithi107.stdout:10460: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:11.165 INFO:tasks.rados.rados.0.smithi107.stdout:10462: done (1 left) 2020-06-12T02:39:11.165 INFO:tasks.rados.rados.0.smithi107.stdout:10463: delete oid 13 current snap is 0 2020-06-12T02:39:11.166 INFO:tasks.rados.rados.0.smithi107.stdout:10460: finishing write tid 2 to smithi10729407-8 2020-06-12T02:39:11.166 INFO:tasks.rados.rados.0.smithi107.stdout:10460: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:11.166 INFO:tasks.rados.rados.0.smithi107.stdout:10460: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:11.166 INFO:tasks.rados.rados.0.smithi107.stdout:10463: done (1 left) 2020-06-12T02:39:11.166 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:39:11.167 INFO:tasks.rados.rados.0.smithi107.stdout:10464: seq_num 10463 ranges {0=40960} 2020-06-12T02:39:11.167 INFO:tasks.rados.rados.0.smithi107.stdout:10464: writing smithi10729407-13 from 0 to 40960 tid 1 2020-06-12T02:39:11.167 INFO:tasks.rados.rados.0.smithi107.stdout:10465: delete oid 1 current snap is 0 2020-06-12T02:39:11.167 INFO:tasks.rados.rados.0.smithi107.stdout:10460: finishing write tid 6 to smithi10729407-8 2020-06-12T02:39:11.167 INFO:tasks.rados.rados.0.smithi107.stdout:10465: done (2 left) 2020-06-12T02:39:11.167 INFO:tasks.rados.rados.0.smithi107.stdout:10466: delete oid 15 current snap is 0 2020-06-12T02:39:11.168 INFO:tasks.rados.rados.0.smithi107.stdout:10464: finishing write tid 1 to smithi10729407-13 2020-06-12T02:39:11.168 INFO:tasks.rados.rados.0.smithi107.stdout:10464: finishing write tid 3 to smithi10729407-13 2020-06-12T02:39:11.168 INFO:tasks.rados.rados.0.smithi107.stdout:10464: finishing write tid 4 to smithi10729407-13 2020-06-12T02:39:11.168 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1752 (ObjNum 10463 snap 0 seq_num 10463) dirty exists 2020-06-12T02:39:11.168 INFO:tasks.rados.rados.0.smithi107.stdout:10464: left oid 13 (ObjNum 10463 snap 0 seq_num 10463) 2020-06-12T02:39:11.169 INFO:tasks.rados.rados.0.smithi107.stdout:10460: finishing write tid 7 to smithi10729407-8 2020-06-12T02:39:11.169 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1938 (ObjNum 10459 snap 0 seq_num 10459) dirty exists 2020-06-12T02:39:11.169 INFO:tasks.rados.rados.0.smithi107.stdout:10460: left oid 8 (ObjNum 10459 snap 0 seq_num 10459) 2020-06-12T02:39:11.169 INFO:tasks.rados.rados.0.smithi107.stdout:10460: done (2 left) 2020-06-12T02:39:11.169 INFO:tasks.rados.rados.0.smithi107.stdout:10464: done (1 left) 2020-06-12T02:39:11.169 INFO:tasks.rados.rados.0.smithi107.stdout:10466: done (0 left) 2020-06-12T02:39:11.170 INFO:tasks.rados.rados.0.smithi107.stdout:10467: delete oid 7 current snap is 0 2020-06-12T02:39:11.170 INFO:tasks.rados.rados.0.smithi107.stdout:10467: done (0 left) 2020-06-12T02:39:11.170 INFO:tasks.rados.rados.0.smithi107.stdout:10468: delete oid 20 current snap is 0 2020-06-12T02:39:11.170 INFO:tasks.rados.rados.0.smithi107.stdout:10468: done (0 left) 2020-06-12T02:39:11.170 INFO:tasks.rados.rados.0.smithi107.stdout:10469: delete oid 7 current snap is 0 2020-06-12T02:39:11.171 INFO:tasks.rados.rados.0.smithi107.stdout:10469: done (0 left) 2020-06-12T02:39:11.171 INFO:tasks.rados.rados.0.smithi107.stdout:10470: delete oid 1 current snap is 0 2020-06-12T02:39:11.171 INFO:tasks.rados.rados.0.smithi107.stdout:10470: done (0 left) 2020-06-12T02:39:11.171 INFO:tasks.rados.rados.0.smithi107.stdout:10471: delete oid 1 current snap is 0 2020-06-12T02:39:11.171 INFO:tasks.rados.rados.0.smithi107.stdout:10471: done (0 left) 2020-06-12T02:39:11.171 INFO:tasks.rados.rados.0.smithi107.stdout:10472: delete oid 11 current snap is 0 2020-06-12T02:39:11.172 INFO:tasks.rados.rados.0.smithi107.stdout:10472: done (0 left) 2020-06-12T02:39:11.172 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:39:11.172 INFO:tasks.rados.rados.0.smithi107.stdout:10473: seq_num 10472 ranges {0=188416} 2020-06-12T02:39:11.172 INFO:tasks.rados.rados.0.smithi107.stdout:10473: writing smithi10729407-12 from 0 to 188416 tid 1 2020-06-12T02:39:11.172 INFO:tasks.rados.rados.0.smithi107.stdout:10474: delete oid 17 current snap is 0 2020-06-12T02:39:11.172 INFO:tasks.rados.rados.0.smithi107.stdout:10474: done (1 left) 2020-06-12T02:39:11.173 INFO:tasks.rados.rados.0.smithi107.stdout:10475: delete oid 3 current snap is 0 2020-06-12T02:39:11.173 INFO:tasks.rados.rados.0.smithi107.stdout:10473: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:11.173 INFO:tasks.rados.rados.0.smithi107.stdout:10475: done (1 left) 2020-06-12T02:39:11.173 INFO:tasks.rados.rados.0.smithi107.stdout:10476: delete oid 4 current snap is 0 2020-06-12T02:39:11.173 INFO:tasks.rados.rados.0.smithi107.stdout:10473: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:11.174 INFO:tasks.rados.rados.0.smithi107.stdout:10473: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:11.174 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1925 (ObjNum 10472 snap 0 seq_num 10472) dirty exists 2020-06-12T02:39:11.174 INFO:tasks.rados.rados.0.smithi107.stdout:10473: left oid 12 (ObjNum 10472 snap 0 seq_num 10472) 2020-06-12T02:39:11.174 INFO:tasks.rados.rados.0.smithi107.stdout:10473: done (1 left) 2020-06-12T02:39:11.174 INFO:tasks.rados.rados.0.smithi107.stdout:10476: done (0 left) 2020-06-12T02:39:11.175 INFO:tasks.rados.rados.0.smithi107.stdout:10477: delete oid 3 current snap is 0 2020-06-12T02:39:11.175 INFO:tasks.rados.rados.0.smithi107.stdout:10477: done (0 left) 2020-06-12T02:39:11.175 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:39:11.175 INFO:tasks.rados.rados.0.smithi107.stdout:10478: seq_num 10477 ranges {0=270336} 2020-06-12T02:39:11.175 INFO:tasks.rados.rados.0.smithi107.stdout:10478: writing smithi10729407-4 from 0 to 270336 tid 1 2020-06-12T02:39:11.175 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:39:11.176 INFO:tasks.rados.rados.0.smithi107.stdout:10479: seq_num 10478 ranges {188416=294912} 2020-06-12T02:39:11.176 INFO:tasks.rados.rados.0.smithi107.stdout:10479: writing smithi10729407-12 from 188416 to 483328 tid 1 2020-06-12T02:39:11.176 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:39:11.176 INFO:tasks.rados.rados.0.smithi107.stdout:10480: seq_num 10479 ranges {4055040=311296} 2020-06-12T02:39:11.176 INFO:tasks.rados.rados.0.smithi107.stdout:10480: writing smithi10729407-8 from 4055040 to 4366336 tid 1 2020-06-12T02:39:11.176 INFO:tasks.rados.rados.0.smithi107.stdout:10481: delete oid 19 current snap is 0 2020-06-12T02:39:11.177 INFO:tasks.rados.rados.0.smithi107.stdout:10478: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:11.177 INFO:tasks.rados.rados.0.smithi107.stdout:10479: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:11.177 INFO:tasks.rados.rados.0.smithi107.stdout:10478: finishing write tid 3 to smithi10729407-4 2020-06-12T02:39:11.177 INFO:tasks.rados.rados.0.smithi107.stdout:10478: finishing write tid 4 to smithi10729407-4 2020-06-12T02:39:11.177 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1949 (ObjNum 10477 snap 0 seq_num 10477) dirty exists 2020-06-12T02:39:11.178 INFO:tasks.rados.rados.0.smithi107.stdout:10478: left oid 4 (ObjNum 10477 snap 0 seq_num 10477) 2020-06-12T02:39:11.178 INFO:tasks.rados.rados.0.smithi107.stdout:10479: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:11.178 INFO:tasks.rados.rados.0.smithi107.stdout:10479: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:11.178 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1927 (ObjNum 10478 snap 0 seq_num 10478) dirty exists 2020-06-12T02:39:11.178 INFO:tasks.rados.rados.0.smithi107.stdout:10479: left oid 12 (ObjNum 10478 snap 0 seq_num 10478) 2020-06-12T02:39:11.178 INFO:tasks.rados.rados.0.smithi107.stdout:10478: done (3 left) 2020-06-12T02:39:11.179 INFO:tasks.rados.rados.0.smithi107.stdout:10479: done (2 left) 2020-06-12T02:39:11.179 INFO:tasks.rados.rados.0.smithi107.stdout:10481: done (1 left) 2020-06-12T02:39:11.179 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:39:11.179 INFO:tasks.rados.rados.0.smithi107.stdout:10482: seq_num 10481 ranges {2146304=344064} 2020-06-12T02:39:11.179 INFO:tasks.rados.rados.0.smithi107.stdout:10482: writing smithi10729407-5 from 2146304 to 2490368 tid 1 2020-06-12T02:39:11.180 INFO:tasks.rados.rados.0.smithi107.stdout:10483: delete oid 15 current snap is 0 2020-06-12T02:39:11.180 INFO:tasks.rados.rados.0.smithi107.stdout:10480: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:11.180 INFO:tasks.rados.rados.0.smithi107.stdout:10480: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:11.180 INFO:tasks.rados.rados.0.smithi107.stdout:10480: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:11.180 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1941 (ObjNum 10479 snap 0 seq_num 10479) dirty exists 2020-06-12T02:39:11.180 INFO:tasks.rados.rados.0.smithi107.stdout:10480: left oid 8 (ObjNum 10479 snap 0 seq_num 10479) 2020-06-12T02:39:11.181 INFO:tasks.rados.rados.0.smithi107.stdout:10480: done (2 left) 2020-06-12T02:39:11.181 INFO:tasks.rados.rados.0.smithi107.stdout:10483: done (1 left) 2020-06-12T02:39:11.181 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:39:11.181 INFO:tasks.rados.rados.0.smithi107.stdout:10484: seq_num 10483 ranges {0=376832} 2020-06-12T02:39:11.181 INFO:tasks.rados.rados.0.smithi107.stdout:10484: writing smithi10729407-16 from 0 to 376832 tid 1 2020-06-12T02:39:11.182 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:39:11.182 INFO:tasks.rados.rados.0.smithi107.stdout:10485: seq_num 10484 ranges {0=393216} 2020-06-12T02:39:11.182 INFO:tasks.rados.rados.0.smithi107.stdout:10485: writing smithi10729407-2 from 0 to 393216 tid 1 2020-06-12T02:39:11.182 INFO:tasks.rados.rados.0.smithi107.stdout:10486: delete oid 8 current snap is 0 2020-06-12T02:39:11.182 INFO:tasks.rados.rados.0.smithi107.stdout:10482: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:11.182 INFO:tasks.rados.rados.0.smithi107.stdout:10482: finishing write tid 3 to smithi10729407-5 2020-06-12T02:39:11.183 INFO:tasks.rados.rados.0.smithi107.stdout:10484: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:11.183 INFO:tasks.rados.rados.0.smithi107.stdout:10482: finishing write tid 4 to smithi10729407-5 2020-06-12T02:39:11.183 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 995 (ObjNum 10481 snap 0 seq_num 10481) dirty exists 2020-06-12T02:39:11.183 INFO:tasks.rados.rados.0.smithi107.stdout:10482: left oid 5 (ObjNum 10481 snap 0 seq_num 10481) 2020-06-12T02:39:11.183 INFO:tasks.rados.rados.0.smithi107.stdout:10482: done (3 left) 2020-06-12T02:39:11.184 INFO:tasks.rados.rados.0.smithi107.stdout:10486: done (2 left) 2020-06-12T02:39:11.184 INFO:tasks.rados.rados.0.smithi107.stdout:10487: delete oid 8 current snap is 0 2020-06-12T02:39:11.184 INFO:tasks.rados.rados.0.smithi107.stdout:10484: finishing write tid 3 to smithi10729407-16 2020-06-12T02:39:11.184 INFO:tasks.rados.rados.0.smithi107.stdout:10484: finishing write tid 4 to smithi10729407-16 2020-06-12T02:39:11.184 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1874 (ObjNum 10483 snap 0 seq_num 10483) dirty exists 2020-06-12T02:39:11.184 INFO:tasks.rados.rados.0.smithi107.stdout:10484: left oid 16 (ObjNum 10483 snap 0 seq_num 10483) 2020-06-12T02:39:11.185 INFO:tasks.rados.rados.0.smithi107.stdout:10485: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:11.185 INFO:tasks.rados.rados.0.smithi107.stdout:10485: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:11.185 INFO:tasks.rados.rados.0.smithi107.stdout:10484: done (2 left) 2020-06-12T02:39:11.185 INFO:tasks.rados.rados.0.smithi107.stdout:10487: done (1 left) 2020-06-12T02:39:11.185 INFO:tasks.rados.rados.0.smithi107.stdout:10488: delete oid 15 current snap is 0 2020-06-12T02:39:11.186 INFO:tasks.rados.rados.0.smithi107.stdout:10485: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:11.186 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 979 (ObjNum 10484 snap 0 seq_num 10484) dirty exists 2020-06-12T02:39:11.186 INFO:tasks.rados.rados.0.smithi107.stdout:10485: left oid 2 (ObjNum 10484 snap 0 seq_num 10484) 2020-06-12T02:39:11.186 INFO:tasks.rados.rados.0.smithi107.stdout:10485: done (1 left) 2020-06-12T02:39:11.186 INFO:tasks.rados.rados.0.smithi107.stdout:10488: done (0 left) 2020-06-12T02:39:11.186 INFO:tasks.rados.rados.0.smithi107.stdout:10489: delete oid 11 current snap is 0 2020-06-12T02:39:11.187 INFO:tasks.rados.rados.0.smithi107.stdout:10489: done (0 left) 2020-06-12T02:39:11.187 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:39:11.187 INFO:tasks.rados.rados.0.smithi107.stdout:10490: seq_num 10489 ranges {0=475136} 2020-06-12T02:39:11.187 INFO:tasks.rados.rados.0.smithi107.stdout:10490: writing smithi10729407-17 from 0 to 475136 tid 1 2020-06-12T02:39:11.187 INFO:tasks.rados.rados.0.smithi107.stdout:10491: delete oid 4 current snap is 0 2020-06-12T02:39:11.188 INFO:tasks.rados.rados.0.smithi107.stdout:10490: finishing write tid 1 to smithi10729407-17 2020-06-12T02:39:11.188 INFO:tasks.rados.rados.0.smithi107.stdout:10491: done (1 left) 2020-06-12T02:39:11.188 INFO:tasks.rados.rados.0.smithi107.stdout:10492: delete oid 18 current snap is 0 2020-06-12T02:39:11.188 INFO:tasks.rados.rados.0.smithi107.stdout:10490: finishing write tid 3 to smithi10729407-17 2020-06-12T02:39:11.188 INFO:tasks.rados.rados.0.smithi107.stdout:10492: done (1 left) 2020-06-12T02:39:11.188 INFO:tasks.rados.rados.0.smithi107.stdout:10493: delete oid 1 current snap is 0 2020-06-12T02:39:11.189 INFO:tasks.rados.rados.0.smithi107.stdout:10490: finishing write tid 4 to smithi10729407-17 2020-06-12T02:39:11.189 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1952 (ObjNum 10489 snap 0 seq_num 10489) dirty exists 2020-06-12T02:39:11.189 INFO:tasks.rados.rados.0.smithi107.stdout:10490: left oid 17 (ObjNum 10489 snap 0 seq_num 10489) 2020-06-12T02:39:11.189 INFO:tasks.rados.rados.0.smithi107.stdout:10490: done (1 left) 2020-06-12T02:39:11.189 INFO:tasks.rados.rados.0.smithi107.stdout:10493: done (0 left) 2020-06-12T02:39:11.190 INFO:tasks.rados.rados.0.smithi107.stdout:10494: delete oid 18 current snap is 0 2020-06-12T02:39:11.190 INFO:tasks.rados.rados.0.smithi107.stdout:10494: done (0 left) 2020-06-12T02:39:11.190 INFO:tasks.rados.rados.0.smithi107.stdout:10495: delete oid 6 current snap is 0 2020-06-12T02:39:11.190 INFO:tasks.rados.rados.0.smithi107.stdout:10495: done (0 left) 2020-06-12T02:39:11.190 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:39:11.190 INFO:tasks.rados.rados.0.smithi107.stdout:10496: seq_num 10495 ranges {0=573440} 2020-06-12T02:39:11.191 INFO:tasks.rados.rados.0.smithi107.stdout:10496: writing smithi10729407-18 from 0 to 573440 tid 1 2020-06-12T02:39:11.191 INFO:tasks.rados.rados.0.smithi107.stdout:10497: delete oid 5 current snap is 0 2020-06-12T02:39:11.191 INFO:tasks.rados.rados.0.smithi107.stdout:10497: done (1 left) 2020-06-12T02:39:11.191 INFO:tasks.rados.rados.0.smithi107.stdout:10498: delete oid 5 current snap is 0 2020-06-12T02:39:11.191 INFO:tasks.rados.rados.0.smithi107.stdout:10496: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:11.191 INFO:tasks.rados.rados.0.smithi107.stdout:10496: finishing write tid 3 to smithi10729407-18 2020-06-12T02:39:11.192 INFO:tasks.rados.rados.0.smithi107.stdout:10498: done (1 left) 2020-06-12T02:39:11.192 INFO:tasks.rados.rados.0.smithi107.stdout:10499: delete oid 2 current snap is 0 2020-06-12T02:39:11.192 INFO:tasks.rados.rados.0.smithi107.stdout:10499: done (1 left) 2020-06-12T02:39:11.192 INFO:tasks.rados.rados.0.smithi107.stdout:10500: delete oid 14 current snap is 0 2020-06-12T02:39:11.192 INFO:tasks.rados.rados.0.smithi107.stdout:10496: finishing write tid 4 to smithi10729407-18 2020-06-12T02:39:11.193 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1834 (ObjNum 10495 snap 0 seq_num 10495) dirty exists 2020-06-12T02:39:11.193 INFO:tasks.rados.rados.0.smithi107.stdout:10496: left oid 18 (ObjNum 10495 snap 0 seq_num 10495) 2020-06-12T02:39:11.193 INFO:tasks.rados.rados.0.smithi107.stdout:10496: done (1 left) 2020-06-12T02:39:11.193 INFO:tasks.rados.rados.0.smithi107.stdout:10500: done (0 left) 2020-06-12T02:39:11.193 INFO:tasks.rados.rados.0.smithi107.stdout:10501: delete oid 6 current snap is 0 2020-06-12T02:39:11.193 INFO:tasks.rados.rados.0.smithi107.stdout:10501: done (0 left) 2020-06-12T02:39:11.194 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:39:11.194 INFO:tasks.rados.rados.0.smithi107.stdout:10502: seq_num 10501 ranges {0=679936} 2020-06-12T02:39:11.194 INFO:tasks.rados.rados.0.smithi107.stdout:10502: writing smithi10729407-6 from 0 to 679936 tid 1 2020-06-12T02:39:11.194 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:39:11.194 INFO:tasks.rados.rados.0.smithi107.stdout:10503: seq_num 10502 ranges {0=696320} 2020-06-12T02:39:11.195 INFO:tasks.rados.rados.0.smithi107.stdout:10503: writing smithi10729407-8 from 0 to 696320 tid 1 2020-06-12T02:39:11.195 INFO:tasks.rados.rados.0.smithi107.stdout:10504: delete oid 3 current snap is 0 2020-06-12T02:39:11.195 INFO:tasks.rados.rados.0.smithi107.stdout:10502: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:11.195 INFO:tasks.rados.rados.0.smithi107.stdout:10504: done (2 left) 2020-06-12T02:39:11.195 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:39:11.195 INFO:tasks.rados.rados.0.smithi107.stdout:10505: seq_num 10504 ranges {0=729088} 2020-06-12T02:39:11.196 INFO:tasks.rados.rados.0.smithi107.stdout:10505: writing smithi10729407-2 from 0 to 729088 tid 1 2020-06-12T02:39:11.196 INFO:tasks.rados.rados.0.smithi107.stdout:10506: delete oid 15 current snap is 0 2020-06-12T02:39:11.196 INFO:tasks.rados.rados.0.smithi107.stdout:10502: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:11.196 INFO:tasks.rados.rados.0.smithi107.stdout:10502: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:11.196 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1908 (ObjNum 10501 snap 0 seq_num 10501) dirty exists 2020-06-12T02:39:11.197 INFO:tasks.rados.rados.0.smithi107.stdout:10502: left oid 6 (ObjNum 10501 snap 0 seq_num 10501) 2020-06-12T02:39:11.197 INFO:tasks.rados.rados.0.smithi107.stdout:10503: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:11.197 INFO:tasks.rados.rados.0.smithi107.stdout:10503: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:11.197 INFO:tasks.rados.rados.0.smithi107.stdout:10502: done (3 left) 2020-06-12T02:39:11.197 INFO:tasks.rados.rados.0.smithi107.stdout:10506: done (2 left) 2020-06-12T02:39:11.197 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:39:11.198 INFO:tasks.rados.rados.0.smithi107.stdout:10507: seq_num 10506 ranges {0=761856} 2020-06-12T02:39:11.198 INFO:tasks.rados.rados.0.smithi107.stdout:10507: writing smithi10729407-9 from 0 to 761856 tid 1 2020-06-12T02:39:11.198 INFO:tasks.rados.rados.0.smithi107.stdout:10508: delete oid 4 current snap is 0 2020-06-12T02:39:11.198 INFO:tasks.rados.rados.0.smithi107.stdout:10503: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:11.198 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1947 (ObjNum 10502 snap 0 seq_num 10502) dirty exists 2020-06-12T02:39:11.199 INFO:tasks.rados.rados.0.smithi107.stdout:10503: left oid 8 (ObjNum 10502 snap 0 seq_num 10502) 2020-06-12T02:39:11.199 INFO:tasks.rados.rados.0.smithi107.stdout:10505: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:11.199 INFO:tasks.rados.rados.0.smithi107.stdout:10505: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:11.199 INFO:tasks.rados.rados.0.smithi107.stdout:10505: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:11.199 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 982 (ObjNum 10504 snap 0 seq_num 10504) dirty exists 2020-06-12T02:39:11.199 INFO:tasks.rados.rados.0.smithi107.stdout:10505: left oid 2 (ObjNum 10504 snap 0 seq_num 10504) 2020-06-12T02:39:11.200 INFO:tasks.rados.rados.0.smithi107.stdout:10507: finishing write tid 1 to smithi10729407-9 2020-06-12T02:39:11.200 INFO:tasks.rados.rados.0.smithi107.stdout:10503: done (3 left) 2020-06-12T02:39:11.200 INFO:tasks.rados.rados.0.smithi107.stdout:10505: done (2 left) 2020-06-12T02:39:11.200 INFO:tasks.rados.rados.0.smithi107.stdout:10508: done (1 left) 2020-06-12T02:39:11.200 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:39:11.200 INFO:tasks.rados.rados.0.smithi107.stdout:10509: seq_num 10508 ranges {483328=794624} 2020-06-12T02:39:11.201 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-06-12T02:39:11.202 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:39:11.125+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:39:11.206 INFO:tasks.rados.rados.0.smithi107.stdout:10509: writing smithi10729407-12 from 483328 to 1277952 tid 1 2020-06-12T02:39:11.206 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:39:11.206 INFO:tasks.rados.rados.0.smithi107.stdout:10510: seq_num 10509 ranges {0=409600,409600=401408} 2020-06-12T02:39:11.207 INFO:tasks.rados.rados.0.smithi107.stdout:10510: writing smithi10729407-4 from 0 to 409600 tid 1 2020-06-12T02:39:11.207 INFO:tasks.rados.rados.0.smithi107.stdout:10510: writing smithi10729407-4 from 409600 to 811008 tid 2 2020-06-12T02:39:11.207 INFO:tasks.rados.rados.0.smithi107.stdout:10511: delete oid 13 current snap is 0 2020-06-12T02:39:11.207 INFO:tasks.rados.rados.0.smithi107.stdout:10507: finishing write tid 3 to smithi10729407-9 2020-06-12T02:39:11.207 INFO:tasks.rados.rados.0.smithi107.stdout:10507: finishing write tid 4 to smithi10729407-9 2020-06-12T02:39:11.208 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1820 (ObjNum 10506 snap 0 seq_num 10506) dirty exists 2020-06-12T02:39:11.208 INFO:tasks.rados.rados.0.smithi107.stdout:10507: left oid 9 (ObjNum 10506 snap 0 seq_num 10506) 2020-06-12T02:39:11.208 INFO:tasks.rados.rados.0.smithi107.stdout:10507: done (3 left) 2020-06-12T02:39:11.208 INFO:tasks.rados.rados.0.smithi107.stdout:10511: done (2 left) 2020-06-12T02:39:11.208 INFO:tasks.rados.rados.0.smithi107.stdout:10512: delete oid 3 current snap is 0 2020-06-12T02:39:11.209 INFO:tasks.rados.rados.0.smithi107.stdout:10509: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:11.209 INFO:tasks.rados.rados.0.smithi107.stdout:10510: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:11.209 INFO:tasks.rados.rados.0.smithi107.stdout:10509: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:11.209 INFO:tasks.rados.rados.0.smithi107.stdout:10510: finishing write tid 2 to smithi10729407-4 2020-06-12T02:39:11.209 INFO:tasks.rados.rados.0.smithi107.stdout:10512: done (2 left) 2020-06-12T02:39:11.209 INFO:tasks.rados.rados.0.smithi107.stdout:10513: delete oid 3 current snap is 0 2020-06-12T02:39:11.210 INFO:tasks.rados.rados.0.smithi107.stdout:10510: finishing write tid 4 to smithi10729407-4 2020-06-12T02:39:11.210 INFO:tasks.rados.rados.0.smithi107.stdout:10509: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:11.210 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1930 (ObjNum 10508 snap 0 seq_num 10508) dirty exists 2020-06-12T02:39:11.210 INFO:tasks.rados.rados.0.smithi107.stdout:10509: left oid 12 (ObjNum 10508 snap 0 seq_num 10508) 2020-06-12T02:39:11.210 INFO:tasks.rados.rados.0.smithi107.stdout:10510: finishing write tid 5 to smithi10729407-4 2020-06-12T02:39:11.211 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1956 (ObjNum 10509 snap 0 seq_num 10509) dirty exists 2020-06-12T02:39:11.211 INFO:tasks.rados.rados.0.smithi107.stdout:10510: left oid 4 (ObjNum 10509 snap 0 seq_num 10509) 2020-06-12T02:39:11.211 INFO:tasks.rados.rados.0.smithi107.stdout:10509: done (2 left) 2020-06-12T02:39:11.211 INFO:tasks.rados.rados.0.smithi107.stdout:10510: done (1 left) 2020-06-12T02:39:11.211 INFO:tasks.rados.rados.0.smithi107.stdout:10513: done (0 left) 2020-06-12T02:39:11.211 INFO:tasks.rados.rados.0.smithi107.stdout:10514: delete oid 11 current snap is 0 2020-06-12T02:39:11.212 INFO:tasks.rados.rados.0.smithi107.stdout:10514: done (0 left) 2020-06-12T02:39:11.212 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:39:11.212 INFO:tasks.rados.rados.0.smithi107.stdout:10515: seq_num 10514 ranges {0=491520,491520=401408} 2020-06-12T02:39:11.212 INFO:tasks.rados.rados.0.smithi107.stdout:10515: writing smithi10729407-20 from 0 to 491520 tid 1 2020-06-12T02:39:11.212 INFO:tasks.rados.rados.0.smithi107.stdout:10515: writing smithi10729407-20 from 491520 to 892928 tid 2 2020-06-12T02:39:11.213 INFO:tasks.rados.rados.0.smithi107.stdout:10516: delete oid 5 current snap is 0 2020-06-12T02:39:11.213 INFO:tasks.rados.rados.0.smithi107.stdout:10515: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:11.213 INFO:tasks.rados.rados.0.smithi107.stdout:10516: done (1 left) 2020-06-12T02:39:11.213 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:39:11.214 INFO:tasks.rados.rados.0.smithi107.stdout:10517: seq_num 10516 ranges {1277952=532480,1810432=401408} 2020-06-12T02:39:11.214 INFO:tasks.rados.rados.0.smithi107.stdout:10517: writing smithi10729407-12 from 1277952 to 1810432 tid 1 2020-06-12T02:39:11.214 INFO:tasks.rados.rados.0.smithi107.stdout:10517: writing smithi10729407-12 from 1810432 to 2211840 tid 2 2020-06-12T02:39:11.214 INFO:tasks.rados.rados.0.smithi107.stdout:10518: delete oid 10 current snap is 0 2020-06-12T02:39:11.214 INFO:tasks.rados.rados.0.smithi107.stdout:10515: finishing write tid 2 to smithi10729407-20 2020-06-12T02:39:11.215 INFO:tasks.rados.rados.0.smithi107.stdout:10515: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:11.215 INFO:tasks.rados.rados.0.smithi107.stdout:10515: finishing write tid 5 to smithi10729407-20 2020-06-12T02:39:11.215 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 816 (ObjNum 10514 snap 0 seq_num 10514) dirty exists 2020-06-12T02:39:11.215 INFO:tasks.rados.rados.0.smithi107.stdout:10515: left oid 20 (ObjNum 10514 snap 0 seq_num 10514) 2020-06-12T02:39:11.215 INFO:tasks.rados.rados.0.smithi107.stdout:10515: done (2 left) 2020-06-12T02:39:11.215 INFO:tasks.rados.rados.0.smithi107.stdout:10518: done (1 left) 2020-06-12T02:39:11.216 INFO:tasks.rados.rados.0.smithi107.stdout:10519: delete oid 4 current snap is 0 2020-06-12T02:39:11.216 INFO:tasks.rados.rados.0.smithi107.stdout:10517: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:11.216 INFO:tasks.rados.rados.0.smithi107.stdout:10519: done (1 left) 2020-06-12T02:39:11.216 INFO:tasks.rados.rados.0.smithi107.stdout:10520: delete oid 16 current snap is 0 2020-06-12T02:39:11.216 INFO:tasks.rados.rados.0.smithi107.stdout:10520: done (1 left) 2020-06-12T02:39:11.217 INFO:tasks.rados.rados.0.smithi107.stdout:10521: delete oid 5 current snap is 0 2020-06-12T02:39:11.217 INFO:tasks.rados.rados.0.smithi107.stdout:10517: finishing write tid 2 to smithi10729407-12 2020-06-12T02:39:11.217 INFO:tasks.rados.rados.0.smithi107.stdout:10517: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:11.217 INFO:tasks.rados.rados.0.smithi107.stdout:10521: done (1 left) 2020-06-12T02:39:11.217 INFO:tasks.rados.rados.0.smithi107.stdout:10522: delete oid 3 current snap is 0 2020-06-12T02:39:11.217 INFO:tasks.rados.rados.0.smithi107.stdout:10517: finishing write tid 5 to smithi10729407-12 2020-06-12T02:39:11.218 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1933 (ObjNum 10516 snap 0 seq_num 10516) dirty exists 2020-06-12T02:39:11.218 INFO:tasks.rados.rados.0.smithi107.stdout:10517: left oid 12 (ObjNum 10516 snap 0 seq_num 10516) 2020-06-12T02:39:11.218 INFO:tasks.rados.rados.0.smithi107.stdout:10517: done (1 left) 2020-06-12T02:39:11.218 INFO:tasks.rados.rados.0.smithi107.stdout:10522: done (0 left) 2020-06-12T02:39:11.218 INFO:tasks.rados.rados.0.smithi107.stdout:10523: delete oid 17 current snap is 0 2020-06-12T02:39:11.219 INFO:tasks.rados.rados.0.smithi107.stdout:10523: done (0 left) 2020-06-12T02:39:11.219 INFO:tasks.rados.rados.0.smithi107.stdout:10524: delete oid 2 current snap is 0 2020-06-12T02:39:11.219 INFO:tasks.rados.rados.0.smithi107.stdout:10524: done (0 left) 2020-06-12T02:39:11.219 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:39:11.219 INFO:tasks.rados.rados.0.smithi107.stdout:10525: seq_num 10524 ranges {892928=663552,1556480=401408} 2020-06-12T02:39:11.221 INFO:tasks.rados.rados.0.smithi107.stdout:10525: writing smithi10729407-20 from 892928 to 1556480 tid 1 2020-06-12T02:39:11.221 INFO:tasks.rados.rados.0.smithi107.stdout:10525: writing smithi10729407-20 from 1556480 to 1957888 tid 2 2020-06-12T02:39:11.221 INFO:tasks.rados.rados.0.smithi107.stdout:10526: delete oid 12 current snap is 0 2020-06-12T02:39:11.221 INFO:tasks.rados.rados.0.smithi107.stdout:10526: done (1 left) 2020-06-12T02:39:11.221 INFO:tasks.rados.rados.0.smithi107.stdout:10527: delete oid 11 current snap is 0 2020-06-12T02:39:11.222 INFO:tasks.rados.rados.0.smithi107.stdout:10525: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:11.222 INFO:tasks.rados.rados.0.smithi107.stdout:10525: finishing write tid 2 to smithi10729407-20 2020-06-12T02:39:11.222 INFO:tasks.rados.rados.0.smithi107.stdout:10527: done (1 left) 2020-06-12T02:39:11.222 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:39:11.222 INFO:tasks.rados.rados.0.smithi107.stdout:10528: seq_num 10527 ranges {679936=712704,1392640=401408} 2020-06-12T02:39:11.228 INFO:tasks.rados.rados.0.smithi107.stdout:10528: writing smithi10729407-6 from 679936 to 1392640 tid 1 2020-06-12T02:39:11.229 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:39:11.226+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:39:11.230 INFO:tasks.rados.rados.0.smithi107.stdout:10528: writing smithi10729407-6 from 1392640 to 1794048 tid 2 2020-06-12T02:39:11.231 INFO:tasks.rados.rados.0.smithi107.stdout:10529: delete oid 10 current snap is 0 2020-06-12T02:39:11.231 INFO:tasks.rados.rados.0.smithi107.stdout:10525: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:11.231 INFO:tasks.rados.rados.0.smithi107.stdout:10525: finishing write tid 5 to smithi10729407-20 2020-06-12T02:39:11.231 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 819 (ObjNum 10524 snap 0 seq_num 10524) dirty exists 2020-06-12T02:39:11.232 INFO:tasks.rados.rados.0.smithi107.stdout:10525: left oid 20 (ObjNum 10524 snap 0 seq_num 10524) 2020-06-12T02:39:11.232 INFO:tasks.rados.rados.0.smithi107.stdout:10525: done (2 left) 2020-06-12T02:39:11.233 INFO:tasks.rados.rados.0.smithi107.stdout:10529: done (1 left) 2020-06-12T02:39:11.233 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:39:11.233 INFO:tasks.rados.rados.0.smithi107.stdout:10530: seq_num 10529 ranges {0=745472,745472=401408} 2020-06-12T02:39:11.237 INFO:tasks.rados.rados.0.smithi107.stdout:10530: writing smithi10729407-1 from 0 to 745472 tid 1 2020-06-12T02:39:11.240 INFO:tasks.rados.rados.0.smithi107.stdout:10530: writing smithi10729407-1 from 745472 to 1146880 tid 2 2020-06-12T02:39:11.240 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:39:11.240 INFO:tasks.rados.rados.0.smithi107.stdout:10531: seq_num 10530 ranges {0=761856,761856=401408} 2020-06-12T02:39:11.245 INFO:tasks.rados.rados.0.smithi107.stdout:10531: writing smithi10729407-3 from 0 to 761856 tid 1 2020-06-12T02:39:11.248 INFO:tasks.rados.rados.0.smithi107.stdout:10531: writing smithi10729407-3 from 761856 to 1163264 tid 2 2020-06-12T02:39:11.248 INFO:tasks.rados.rados.0.smithi107.stdout:10532: delete oid 7 current snap is 0 2020-06-12T02:39:11.248 INFO:tasks.rados.rados.0.smithi107.stdout:10528: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:11.248 INFO:tasks.rados.rados.0.smithi107.stdout:10528: finishing write tid 2 to smithi10729407-6 2020-06-12T02:39:11.248 INFO:tasks.rados.rados.0.smithi107.stdout:10528: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:11.249 INFO:tasks.rados.rados.0.smithi107.stdout:10528: finishing write tid 5 to smithi10729407-6 2020-06-12T02:39:11.249 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1911 (ObjNum 10527 snap 0 seq_num 10527) dirty exists 2020-06-12T02:39:11.249 INFO:tasks.rados.rados.0.smithi107.stdout:10528: left oid 6 (ObjNum 10527 snap 0 seq_num 10527) 2020-06-12T02:39:11.249 INFO:tasks.rados.rados.0.smithi107.stdout:10530: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:11.253 INFO:tasks.rados.rados.0.smithi107.stdout:10530: finishing write tid 2 to smithi10729407-1 2020-06-12T02:39:11.256 INFO:tasks.rados.rados.0.smithi107.stdout:10531: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:11.257 INFO:tasks.rados.rados.0.smithi107.stdout:10530: finishing write tid 4 to smithi10729407-1 2020-06-12T02:39:11.257 INFO:tasks.rados.rados.0.smithi107.stdout:10528: done (3 left) 2020-06-12T02:39:11.257 INFO:tasks.rados.rados.0.smithi107.stdout:10532: done (2 left) 2020-06-12T02:39:11.258 INFO:tasks.rados.rados.0.smithi107.stdout:10533: delete oid 2 current snap is 0 2020-06-12T02:39:11.258 INFO:tasks.rados.rados.0.smithi107.stdout:10531: finishing write tid 2 to smithi10729407-3 2020-06-12T02:39:11.258 INFO:tasks.rados.rados.0.smithi107.stdout:10531: finishing write tid 4 to smithi10729407-3 2020-06-12T02:39:11.259 INFO:tasks.rados.rados.0.smithi107.stdout:10530: finishing write tid 5 to smithi10729407-1 2020-06-12T02:39:11.259 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1914 (ObjNum 10529 snap 0 seq_num 10529) dirty exists 2020-06-12T02:39:11.260 INFO:tasks.rados.rados.0.smithi107.stdout:10530: left oid 1 (ObjNum 10529 snap 0 seq_num 10529) 2020-06-12T02:39:11.260 INFO:tasks.rados.rados.0.smithi107.stdout:10530: done (2 left) 2020-06-12T02:39:11.260 INFO:tasks.rados.rados.0.smithi107.stdout:10533: done (1 left) 2020-06-12T02:39:11.260 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:39:11.261 INFO:tasks.rados.rados.0.smithi107.stdout:10534: seq_num 10533 ranges {0=409600,409600=483328,892928=319488} 2020-06-12T02:39:11.266 INFO:tasks.rados.rados.0.smithi107.stdout:10534: writing smithi10729407-17 from 0 to 409600 tid 1 2020-06-12T02:39:11.267 INFO:tasks.rados.rados.0.smithi107.stdout:10534: writing smithi10729407-17 from 409600 to 892928 tid 2 2020-06-12T02:39:11.267 INFO:tasks.rados.rados.0.smithi107.stdout:10534: writing smithi10729407-17 from 892928 to 1212416 tid 3 2020-06-12T02:39:11.267 INFO:tasks.rados.rados.0.smithi107.stdout:10535: delete oid 12 current snap is 0 2020-06-12T02:39:11.267 INFO:tasks.rados.rados.0.smithi107.stdout:10531: finishing write tid 5 to smithi10729407-3 2020-06-12T02:39:11.267 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1882 (ObjNum 10530 snap 0 seq_num 10530) dirty exists 2020-06-12T02:39:11.268 INFO:tasks.rados.rados.0.smithi107.stdout:10531: left oid 3 (ObjNum 10530 snap 0 seq_num 10530) 2020-06-12T02:39:11.268 INFO:tasks.rados.rados.0.smithi107.stdout:10534: finishing write tid 1 to smithi10729407-17 2020-06-12T02:39:11.272 INFO:tasks.rados.rados.0.smithi107.stdout:10531: done (2 left) 2020-06-12T02:39:11.273 INFO:tasks.rados.rados.0.smithi107.stdout:10535: done (1 left) 2020-06-12T02:39:11.273 INFO:tasks.rados.rados.0.smithi107.stdout:10536: delete oid 8 current snap is 0 2020-06-12T02:39:11.273 INFO:tasks.rados.rados.0.smithi107.stdout:10536: done (1 left) 2020-06-12T02:39:11.273 INFO:tasks.rados.rados.0.smithi107.stdout:10537: delete oid 2 current snap is 0 2020-06-12T02:39:11.276 INFO:tasks.rados.rados.0.smithi107.stdout:10537: done (1 left) 2020-06-12T02:39:11.276 INFO:tasks.rados.rados.0.smithi107.stdout:10538: delete oid 7 current snap is 0 2020-06-12T02:39:11.276 INFO:tasks.rados.rados.0.smithi107.stdout:10534: finishing write tid 2 to smithi10729407-17 2020-06-12T02:39:11.279 INFO:tasks.rados.rados.0.smithi107.stdout:10534: finishing write tid 3 to smithi10729407-17 2020-06-12T02:39:11.279 INFO:tasks.rados.rados.0.smithi107.stdout:10538: done (1 left) 2020-06-12T02:39:11.280 INFO:tasks.rados.rados.0.smithi107.stdout:10539: delete oid 8 current snap is 0 2020-06-12T02:39:11.280 INFO:tasks.rados.rados.0.smithi107.stdout:10534: finishing write tid 5 to smithi10729407-17 2020-06-12T02:39:11.282 INFO:tasks.rados.rados.0.smithi107.stdout:10534: finishing write tid 6 to smithi10729407-17 2020-06-12T02:39:11.283 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1962 (ObjNum 10533 snap 0 seq_num 10533) dirty exists 2020-06-12T02:39:11.283 INFO:tasks.rados.rados.0.smithi107.stdout:10534: left oid 17 (ObjNum 10533 snap 0 seq_num 10533) 2020-06-12T02:39:11.283 INFO:tasks.rados.rados.0.smithi107.stdout:10534: done (1 left) 2020-06-12T02:39:11.283 INFO:tasks.rados.rados.0.smithi107.stdout:10539: done (0 left) 2020-06-12T02:39:11.284 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:39:11.284 INFO:tasks.rados.rados.0.smithi107.stdout:10540: seq_num 10539 ranges {0=516096,516096=638976,1155072=163840} 2020-06-12T02:39:11.285 INFO:tasks.rados.rados.0.smithi107.stdout:10540: writing smithi10729407-7 from 0 to 516096 tid 1 2020-06-12T02:39:11.289 INFO:tasks.rados.rados.0.smithi107.stdout:10540: writing smithi10729407-7 from 516096 to 1155072 tid 2 2020-06-12T02:39:11.289 INFO:tasks.rados.rados.0.smithi107.stdout:10540: writing smithi10729407-7 from 1155072 to 1318912 tid 3 2020-06-12T02:39:11.290 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:39:11.290 INFO:tasks.rados.rados.0.smithi107.stdout:10541: seq_num 10540 ranges {1212416=532480,1744896=516096,2260992=286720} 2020-06-12T02:39:11.299 INFO:tasks.rados.rados.0.smithi107.stdout:10541: writing smithi10729407-17 from 1212416 to 1744896 tid 1 2020-06-12T02:39:11.302 INFO:tasks.rados.rados.0.smithi107.stdout:10541: writing smithi10729407-17 from 1744896 to 2260992 tid 2 2020-06-12T02:39:11.304 INFO:tasks.rados.rados.0.smithi107.stdout:10541: writing smithi10729407-17 from 2260992 to 2547712 tid 3 2020-06-12T02:39:11.304 INFO:tasks.rados.rados.0.smithi107.stdout:10542: delete oid 18 current snap is 0 2020-06-12T02:39:11.304 INFO:tasks.rados.rados.0.smithi107.stdout:10540: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:11.304 INFO:tasks.rados.rados.0.smithi107.stdout:10540: finishing write tid 2 to smithi10729407-7 2020-06-12T02:39:11.305 INFO:tasks.rados.rados.0.smithi107.stdout:10540: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:11.305 INFO:tasks.rados.rados.0.smithi107.stdout:10540: finishing write tid 5 to smithi10729407-7 2020-06-12T02:39:11.307 INFO:tasks.rados.rados.0.smithi107.stdout:10540: finishing write tid 6 to smithi10729407-7 2020-06-12T02:39:11.307 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1749 (ObjNum 10539 snap 0 seq_num 10539) dirty exists 2020-06-12T02:39:11.307 INFO:tasks.rados.rados.0.smithi107.stdout:10540: left oid 7 (ObjNum 10539 snap 0 seq_num 10539) 2020-06-12T02:39:11.312 INFO:tasks.rados.rados.0.smithi107.stdout:10541: finishing write tid 1 to smithi10729407-17 2020-06-12T02:39:11.314 INFO:tasks.rados.rados.0.smithi107.stdout:10540: done (2 left) 2020-06-12T02:39:11.315 INFO:tasks.rados.rados.0.smithi107.stdout:10542: done (1 left) 2020-06-12T02:39:11.315 INFO:tasks.rados.rados.0.smithi107.stdout:10543: delete oid 14 current snap is 0 2020-06-12T02:39:11.315 INFO:tasks.rados.rados.0.smithi107.stdout:10541: finishing write tid 2 to smithi10729407-17 2020-06-12T02:39:11.315 INFO:tasks.rados.rados.0.smithi107.stdout:10541: finishing write tid 3 to smithi10729407-17 2020-06-12T02:39:11.316 INFO:tasks.rados.rados.0.smithi107.stdout:10541: finishing write tid 5 to smithi10729407-17 2020-06-12T02:39:11.316 INFO:tasks.rados.rados.0.smithi107.stdout:10543: done (1 left) 2020-06-12T02:39:11.316 INFO:tasks.rados.rados.0.smithi107.stdout:10544: delete oid 10 current snap is 0 2020-06-12T02:39:11.317 INFO:tasks.rados.rados.0.smithi107.stdout:10541: finishing write tid 6 to smithi10729407-17 2020-06-12T02:39:11.318 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1966 (ObjNum 10540 snap 0 seq_num 10540) dirty exists 2020-06-12T02:39:11.318 INFO:tasks.rados.rados.0.smithi107.stdout:10541: left oid 17 (ObjNum 10540 snap 0 seq_num 10540) 2020-06-12T02:39:11.322 INFO:tasks.rados.rados.0.smithi107.stdout:10541: done (1 left) 2020-06-12T02:39:11.323 INFO:tasks.rados.rados.0.smithi107.stdout:10544: done (0 left) 2020-06-12T02:39:11.323 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:39:11.323 INFO:tasks.rados.rados.0.smithi107.stdout:10545: seq_num 10544 ranges {0=598016,598016=458752,1056768=344064} 2020-06-12T02:39:11.323 INFO:tasks.rados.rados.0.smithi107.stdout:10545: writing smithi10729407-19 from 0 to 598016 tid 1 2020-06-12T02:39:11.325 INFO:tasks.rados.rados.0.smithi107.stdout:10545: writing smithi10729407-19 from 598016 to 1056768 tid 2 2020-06-12T02:39:11.327 INFO:tasks.rados.rados.0.smithi107.stdout:10545: writing smithi10729407-19 from 1056768 to 1400832 tid 3 2020-06-12T02:39:11.328 INFO:tasks.rados.rados.0.smithi107.stdout:10546: delete oid 20 current snap is 0 2020-06-12T02:39:11.329 INFO:tasks.rados.rados.0.smithi107.stdout:10545: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:11.330 INFO:tasks.rados.rados.0.smithi107.stdout:10546: done (1 left) 2020-06-12T02:39:11.330 INFO:tasks.rados.rados.0.smithi107.stdout:10547: delete oid 7 current snap is 0 2020-06-12T02:39:11.330 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:39:11.327+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:39:11.335 INFO:tasks.rados.rados.0.smithi107.stdout:10547: done (1 left) 2020-06-12T02:39:11.336 INFO:tasks.rados.rados.0.smithi107.stdout:10548: delete oid 7 current snap is 0 2020-06-12T02:39:11.341 INFO:tasks.rados.rados.0.smithi107.stdout:10545: finishing write tid 2 to smithi10729407-19 2020-06-12T02:39:11.341 INFO:tasks.rados.rados.0.smithi107.stdout:10545: finishing write tid 3 to smithi10729407-19 2020-06-12T02:39:11.344 INFO:tasks.rados.rados.0.smithi107.stdout:10548: done (1 left) 2020-06-12T02:39:11.344 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:39:11.344 INFO:tasks.rados.rados.0.smithi107.stdout:10549: seq_num 10548 ranges {0=663552,663552=442368,1105920=360448} 2020-06-12T02:39:11.347 INFO:tasks.rados.rados.0.smithi107.stdout:10549: writing smithi10729407-16 from 0 to 663552 tid 1 2020-06-12T02:39:11.349 INFO:tasks.rados.rados.0.smithi107.stdout:10549: writing smithi10729407-16 from 663552 to 1105920 tid 2 2020-06-12T02:39:11.352 INFO:tasks.rados.rados.0.smithi107.stdout:10549: writing smithi10729407-16 from 1105920 to 1466368 tid 3 2020-06-12T02:39:11.352 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:39:11.353 INFO:tasks.rados.rados.0.smithi107.stdout:10550: seq_num 10549 ranges {0=679936,679936=729088,1409024=73728} 2020-06-12T02:39:11.356 INFO:tasks.rados.rados.0.smithi107.stdout:10550: writing smithi10729407-2 from 0 to 679936 tid 1 2020-06-12T02:39:11.361 INFO:tasks.rados.rados.0.smithi107.stdout:10550: writing smithi10729407-2 from 679936 to 1409024 tid 2 2020-06-12T02:39:11.362 INFO:tasks.rados.rados.0.smithi107.stdout:10550: writing smithi10729407-2 from 1409024 to 1482752 tid 3 2020-06-12T02:39:11.362 INFO:tasks.rados.rados.0.smithi107.stdout:10551: delete oid 15 current snap is 0 2020-06-12T02:39:11.362 INFO:tasks.rados.rados.0.smithi107.stdout:10545: finishing write tid 5 to smithi10729407-19 2020-06-12T02:39:11.363 INFO:tasks.rados.rados.0.smithi107.stdout:10545: finishing write tid 6 to smithi10729407-19 2020-06-12T02:39:11.363 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1939 (ObjNum 10544 snap 0 seq_num 10544) dirty exists 2020-06-12T02:39:11.363 INFO:tasks.rados.rados.0.smithi107.stdout:10545: left oid 19 (ObjNum 10544 snap 0 seq_num 10544) 2020-06-12T02:39:11.363 INFO:tasks.rados.rados.0.smithi107.stdout:10549: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:11.363 INFO:tasks.rados.rados.0.smithi107.stdout:10549: finishing write tid 2 to smithi10729407-16 2020-06-12T02:39:11.364 INFO:tasks.rados.rados.0.smithi107.stdout:10549: finishing write tid 3 to smithi10729407-16 2020-06-12T02:39:11.365 INFO:tasks.rados.rados.0.smithi107.stdout:10549: finishing write tid 5 to smithi10729407-16 2020-06-12T02:39:11.365 INFO:tasks.rados.rados.0.smithi107.stdout:10545: done (3 left) 2020-06-12T02:39:11.365 INFO:tasks.rados.rados.0.smithi107.stdout:10551: done (2 left) 2020-06-12T02:39:11.365 INFO:tasks.rados.rados.0.smithi107.stdout:10552: delete oid 4 current snap is 0 2020-06-12T02:39:11.366 INFO:tasks.rados.rados.0.smithi107.stdout:10549: finishing write tid 6 to smithi10729407-16 2020-06-12T02:39:11.366 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1886 (ObjNum 10548 snap 0 seq_num 10548) dirty exists 2020-06-12T02:39:11.366 INFO:tasks.rados.rados.0.smithi107.stdout:10549: left oid 16 (ObjNum 10548 snap 0 seq_num 10548) 2020-06-12T02:39:11.369 INFO:tasks.rados.rados.0.smithi107.stdout:10550: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:11.370 INFO:tasks.rados.rados.0.smithi107.stdout:10549: done (2 left) 2020-06-12T02:39:11.370 INFO:tasks.rados.rados.0.smithi107.stdout:10552: done (1 left) 2020-06-12T02:39:11.370 INFO:tasks.rados.rados.0.smithi107.stdout:10553: delete oid 10 current snap is 0 2020-06-12T02:39:11.371 INFO:tasks.rados.rados.0.smithi107.stdout:10550: finishing write tid 2 to smithi10729407-2 2020-06-12T02:39:11.371 INFO:tasks.rados.rados.0.smithi107.stdout:10550: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:11.372 INFO:tasks.rados.rados.0.smithi107.stdout:10550: finishing write tid 5 to smithi10729407-2 2020-06-12T02:39:11.372 INFO:tasks.rados.rados.0.smithi107.stdout:10553: done (1 left) 2020-06-12T02:39:11.372 INFO:tasks.rados.rados.0.smithi107.stdout:10554: delete oid 7 current snap is 0 2020-06-12T02:39:11.373 INFO:tasks.rados.rados.0.smithi107.stdout:10550: finishing write tid 6 to smithi10729407-2 2020-06-12T02:39:11.373 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 989 (ObjNum 10549 snap 0 seq_num 10549) dirty exists 2020-06-12T02:39:11.373 INFO:tasks.rados.rados.0.smithi107.stdout:10550: left oid 2 (ObjNum 10549 snap 0 seq_num 10549) 2020-06-12T02:39:11.374 INFO:tasks.rados.rados.0.smithi107.stdout:10550: done (1 left) 2020-06-12T02:39:11.374 INFO:tasks.rados.rados.0.smithi107.stdout:10554: done (0 left) 2020-06-12T02:39:11.375 INFO:tasks.rados.rados.0.smithi107.stdout:10555: delete oid 20 current snap is 0 2020-06-12T02:39:11.376 INFO:tasks.rados.rados.0.smithi107.stdout:10555: done (0 left) 2020-06-12T02:39:11.376 INFO:tasks.rados.rados.0.smithi107.stdout:10556: delete oid 15 current snap is 0 2020-06-12T02:39:11.378 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:39:11.379 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:39:11.379 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2020-06-12T02:39:11.379 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:39:11.379 INFO:tasks.rados.rados.0.smithi107.stdout:10556: done (0 left) 2020-06-12T02:39:11.380 INFO:tasks.rados.rados.0.smithi107.stdout:10557: delete oid 14 current snap is 0 2020-06-12T02:39:11.381 INFO:tasks.rados.rados.0.smithi107.stdout:10557: done (0 left) 2020-06-12T02:39:11.382 INFO:tasks.rados.rados.0.smithi107.stdout:10558: delete oid 5 current snap is 0 2020-06-12T02:39:11.383 INFO:tasks.rados.rados.0.smithi107.stdout:10558: done (0 left) 2020-06-12T02:39:11.384 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:39:11.384 INFO:tasks.rados.rados.0.smithi107.stdout:10559: seq_num 10558 ranges {0=434176,434176=540672,974848=491520,1466368=172032} 2020-06-12T02:39:11.386 INFO:tasks.rados.rados.0.smithi107.stdout:10559: writing smithi10729407-4 from 0 to 434176 tid 1 2020-06-12T02:39:11.389 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-06-12T02:39:11.389 INFO:tasks.rados.rados.0.smithi107.stdout:10559: writing smithi10729407-4 from 434176 to 974848 tid 2 2020-06-12T02:39:11.392 INFO:tasks.rados.rados.0.smithi107.stdout:10559: writing smithi10729407-4 from 974848 to 1466368 tid 3 2020-06-12T02:39:11.393 INFO:tasks.rados.rados.0.smithi107.stdout:10559: writing smithi10729407-4 from 1466368 to 1638400 tid 4 2020-06-12T02:39:11.394 INFO:tasks.rados.rados.0.smithi107.stdout:10560: delete oid 2 current snap is 0 2020-06-12T02:39:11.408 INFO:tasks.rados.rados.0.smithi107.stdout:10559: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:11.408 INFO:tasks.rados.rados.0.smithi107.stdout:10560: done (1 left) 2020-06-12T02:39:11.409 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:39:11.409 INFO:tasks.rados.rados.0.smithi107.stdout:10561: seq_num 10560 ranges {0=466944,466944=753664,1220608=450560} 2020-06-12T02:39:11.409 INFO:tasks.rados.rados.0.smithi107.stdout:10561: writing smithi10729407-14 from 0 to 466944 tid 1 2020-06-12T02:39:11.409 INFO:tasks.rados.rados.0.smithi107.stdout:10561: writing smithi10729407-14 from 466944 to 1220608 tid 2 2020-06-12T02:39:11.409 INFO:tasks.rados.rados.0.smithi107.stdout:10561: writing smithi10729407-14 from 1220608 to 1671168 tid 3 2020-06-12T02:39:11.410 INFO:tasks.rados.rados.0.smithi107.stdout:10562: delete oid 13 current snap is 0 2020-06-12T02:39:11.410 INFO:tasks.rados.rados.0.smithi107.stdout:10559: finishing write tid 2 to smithi10729407-4 2020-06-12T02:39:11.410 INFO:tasks.rados.rados.0.smithi107.stdout:10559: finishing write tid 3 to smithi10729407-4 2020-06-12T02:39:11.410 INFO:tasks.rados.rados.0.smithi107.stdout:10559: finishing write tid 4 to smithi10729407-4 2020-06-12T02:39:11.410 INFO:tasks.rados.rados.0.smithi107.stdout:10559: finishing write tid 6 to smithi10729407-4 2020-06-12T02:39:11.411 INFO:tasks.rados.rados.0.smithi107.stdout:10561: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:11.411 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd set noscrub 2020-06-12T02:39:11.412 INFO:tasks.rados.rados.0.smithi107.stdout:10562: done (2 left) 2020-06-12T02:39:11.412 INFO:tasks.rados.rados.0.smithi107.stdout:10563: delete oid 13 current snap is 0 2020-06-12T02:39:11.415 INFO:tasks.rados.rados.0.smithi107.stdout:10559: finishing write tid 7 to smithi10729407-4 2020-06-12T02:39:11.415 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1972 (ObjNum 10558 snap 0 seq_num 10558) dirty exists 2020-06-12T02:39:11.415 INFO:tasks.rados.rados.0.smithi107.stdout:10559: left oid 4 (ObjNum 10558 snap 0 seq_num 10558) 2020-06-12T02:39:11.415 INFO:tasks.rados.rados.0.smithi107.stdout:10559: done (2 left) 2020-06-12T02:39:11.416 INFO:tasks.rados.rados.0.smithi107.stdout:10563: done (1 left) 2020-06-12T02:39:11.416 INFO:tasks.rados.rados.0.smithi107.stdout:10564: delete oid 8 current snap is 0 2020-06-12T02:39:11.418 INFO:tasks.rados.rados.0.smithi107.stdout:10564: done (1 left) 2020-06-12T02:39:11.418 INFO:tasks.rados.rados.0.smithi107.stdout:10565: delete oid 8 current snap is 0 2020-06-12T02:39:11.419 INFO:tasks.rados.rados.0.smithi107.stdout:10561: finishing write tid 2 to smithi10729407-14 2020-06-12T02:39:11.420 INFO:tasks.rados.rados.0.smithi107.stdout:10561: finishing write tid 3 to smithi10729407-14 2020-06-12T02:39:11.420 INFO:tasks.rados.rados.0.smithi107.stdout:10561: finishing write tid 5 to smithi10729407-14 2020-06-12T02:39:11.421 INFO:tasks.rados.rados.0.smithi107.stdout:10565: done (1 left) 2020-06-12T02:39:11.422 INFO:tasks.rados.rados.0.smithi107.stdout:10566: delete oid 8 current snap is 0 2020-06-12T02:39:11.422 INFO:tasks.rados.rados.0.smithi107.stdout:10561: finishing write tid 6 to smithi10729407-14 2020-06-12T02:39:11.423 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1695 (ObjNum 10560 snap 0 seq_num 10560) dirty exists 2020-06-12T02:39:11.423 INFO:tasks.rados.rados.0.smithi107.stdout:10561: left oid 14 (ObjNum 10560 snap 0 seq_num 10560) 2020-06-12T02:39:11.427 INFO:tasks.rados.rados.0.smithi107.stdout:10561: done (1 left) 2020-06-12T02:39:11.427 INFO:tasks.rados.rados.0.smithi107.stdout:10566: done (0 left) 2020-06-12T02:39:11.427 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:39:11.428 INFO:tasks.rados.rados.0.smithi107.stdout:10567: seq_num 10566 ranges {0=565248,565248=458752,1024000=458752,1482752=286720} 2020-06-12T02:39:11.430 INFO:tasks.rados.rados.0.smithi107.stdout:10567: writing smithi10729407-11 from 0 to 565248 tid 1 2020-06-12T02:39:11.430 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:39:11.428+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:39:11.432 INFO:tasks.rados.rados.0.smithi107.stdout:10567: writing smithi10729407-11 from 565248 to 1024000 tid 2 2020-06-12T02:39:11.435 INFO:tasks.rados.rados.0.smithi107.stdout:10567: writing smithi10729407-11 from 1024000 to 1482752 tid 3 2020-06-12T02:39:11.437 INFO:tasks.rados.rados.0.smithi107.stdout:10567: writing smithi10729407-11 from 1482752 to 1769472 tid 4 2020-06-12T02:39:11.437 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:39:11.438 INFO:tasks.rados.rados.0.smithi107.stdout:10568: seq_num 10567 ranges {2547712=581632,3129344=745472,3874816=458752} 2020-06-12T02:39:11.463 INFO:tasks.rados.rados.0.smithi107.stdout:10568: writing smithi10729407-17 from 2547712 to 3129344 tid 1 2020-06-12T02:39:11.463 INFO:tasks.rados.rados.0.smithi107.stdout:10568: writing smithi10729407-17 from 3129344 to 3874816 tid 2 2020-06-12T02:39:11.463 INFO:tasks.rados.rados.0.smithi107.stdout:10568: writing smithi10729407-17 from 3874816 to 4333568 tid 3 2020-06-12T02:39:11.464 INFO:tasks.rados.rados.0.smithi107.stdout:10569: delete oid 8 current snap is 0 2020-06-12T02:39:11.464 INFO:tasks.rados.rados.0.smithi107.stdout:10567: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:11.464 INFO:tasks.rados.rados.0.smithi107.stdout:10567: finishing write tid 2 to smithi10729407-11 2020-06-12T02:39:11.464 INFO:tasks.rados.rados.0.smithi107.stdout:10567: finishing write tid 3 to smithi10729407-11 2020-06-12T02:39:11.464 INFO:tasks.rados.rados.0.smithi107.stdout:10567: finishing write tid 4 to smithi10729407-11 2020-06-12T02:39:11.464 INFO:tasks.rados.rados.0.smithi107.stdout:10567: finishing write tid 6 to smithi10729407-11 2020-06-12T02:39:11.465 INFO:tasks.rados.rados.0.smithi107.stdout:10567: finishing write tid 7 to smithi10729407-11 2020-06-12T02:39:11.465 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1713 (ObjNum 10566 snap 0 seq_num 10566) dirty exists 2020-06-12T02:39:11.465 INFO:tasks.rados.rados.0.smithi107.stdout:10567: left oid 11 (ObjNum 10566 snap 0 seq_num 10566) 2020-06-12T02:39:11.466 INFO:tasks.rados.rados.0.smithi107.stdout:10568: finishing write tid 1 to smithi10729407-17 2020-06-12T02:39:11.473 INFO:tasks.rados.rados.0.smithi107.stdout:10567: done (2 left) 2020-06-12T02:39:11.473 INFO:tasks.rados.rados.0.smithi107.stdout:10569: done (1 left) 2020-06-12T02:39:11.474 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:39:11.474 INFO:tasks.rados.rados.0.smithi107.stdout:10570: seq_num 10569 ranges {0=614400,614400=565248,1179648=630784,1810432=8192} 2020-06-12T02:39:11.474 INFO:tasks.rados.rados.0.smithi107.stdout:10570: writing smithi10729407-5 from 0 to 614400 tid 1 2020-06-12T02:39:11.475 INFO:tasks.rados.rados.0.smithi107.stdout:10570: writing smithi10729407-5 from 614400 to 1179648 tid 2 2020-06-12T02:39:11.480 INFO:tasks.rados.rados.0.smithi107.stdout:10570: writing smithi10729407-5 from 1179648 to 1810432 tid 3 2020-06-12T02:39:11.480 INFO:tasks.rados.rados.0.smithi107.stdout:10570: writing smithi10729407-5 from 1810432 to 1818624 tid 4 2020-06-12T02:39:11.480 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:39:11.481 INFO:tasks.rados.rados.0.smithi107.stdout:10571: seq_num 10570 ranges {761856=630784,1392640=450560,1843200=434176,2277376=319488} 2020-06-12T02:39:11.494 INFO:tasks.rados.rados.0.smithi107.stdout:10571: writing smithi10729407-9 from 761856 to 1392640 tid 1 2020-06-12T02:39:11.495 INFO:tasks.rados.rados.0.smithi107.stdout:10571: writing smithi10729407-9 from 1392640 to 1843200 tid 2 2020-06-12T02:39:11.495 INFO:tasks.rados.rados.0.smithi107.stdout:10571: writing smithi10729407-9 from 1843200 to 2277376 tid 3 2020-06-12T02:39:11.496 INFO:tasks.rados.rados.0.smithi107.stdout:10571: writing smithi10729407-9 from 2277376 to 2596864 tid 4 2020-06-12T02:39:11.497 INFO:tasks.rados.rados.0.smithi107.stdout:10572: delete oid 8 current snap is 0 2020-06-12T02:39:11.497 INFO:tasks.rados.rados.0.smithi107.stdout:10568: finishing write tid 2 to smithi10729407-17 2020-06-12T02:39:11.497 INFO:tasks.rados.rados.0.smithi107.stdout:10568: finishing write tid 3 to smithi10729407-17 2020-06-12T02:39:11.497 INFO:tasks.rados.rados.0.smithi107.stdout:10568: finishing write tid 5 to smithi10729407-17 2020-06-12T02:39:11.498 INFO:tasks.rados.rados.0.smithi107.stdout:10570: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:11.498 INFO:tasks.rados.rados.0.smithi107.stdout:10568: finishing write tid 6 to smithi10729407-17 2020-06-12T02:39:11.498 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1976 (ObjNum 10567 snap 0 seq_num 10567) dirty exists 2020-06-12T02:39:11.498 INFO:tasks.rados.rados.0.smithi107.stdout:10568: left oid 17 (ObjNum 10567 snap 0 seq_num 10567) 2020-06-12T02:39:11.498 INFO:tasks.rados.rados.0.smithi107.stdout:10570: finishing write tid 2 to smithi10729407-5 2020-06-12T02:39:11.499 INFO:tasks.rados.rados.0.smithi107.stdout:10570: finishing write tid 3 to smithi10729407-5 2020-06-12T02:39:11.499 INFO:tasks.rados.rados.0.smithi107.stdout:10570: finishing write tid 4 to smithi10729407-5 2020-06-12T02:39:11.499 INFO:tasks.rados.rados.0.smithi107.stdout:10570: finishing write tid 6 to smithi10729407-5 2020-06-12T02:39:11.499 INFO:tasks.rados.rados.0.smithi107.stdout:10571: finishing write tid 1 to smithi10729407-9 2020-06-12T02:39:11.499 INFO:tasks.rados.rados.0.smithi107.stdout:10570: finishing write tid 7 to smithi10729407-5 2020-06-12T02:39:11.499 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 1005 (ObjNum 10569 snap 0 seq_num 10569) dirty exists 2020-06-12T02:39:11.500 INFO:tasks.rados.rados.0.smithi107.stdout:10570: left oid 5 (ObjNum 10569 snap 0 seq_num 10569) 2020-06-12T02:39:11.500 INFO:tasks.rados.rados.0.smithi107.stdout:10568: done (3 left) 2020-06-12T02:39:11.500 INFO:tasks.rados.rados.0.smithi107.stdout:10570: done (2 left) 2020-06-12T02:39:11.501 INFO:tasks.rados.rados.0.smithi107.stdout:10572: done (1 left) 2020-06-12T02:39:11.501 INFO:tasks.rados.rados.0.smithi107.stdout:10573: delete oid 20 current snap is 0 2020-06-12T02:39:11.502 INFO:tasks.rados.rados.0.smithi107.stdout:10571: finishing write tid 2 to smithi10729407-9 2020-06-12T02:39:11.502 INFO:tasks.rados.rados.0.smithi107.stdout:10573: done (1 left) 2020-06-12T02:39:11.503 INFO:tasks.rados.rados.0.smithi107.stdout:10574: delete oid 10 current snap is 0 2020-06-12T02:39:11.504 INFO:tasks.rados.rados.0.smithi107.stdout:10574: done (1 left) 2020-06-12T02:39:11.504 INFO:tasks.rados.rados.0.smithi107.stdout:10575: delete oid 16 current snap is 0 2020-06-12T02:39:11.505 INFO:tasks.rados.rados.0.smithi107.stdout:10571: finishing write tid 3 to smithi10729407-9 2020-06-12T02:39:11.505 INFO:tasks.rados.rados.0.smithi107.stdout:10571: finishing write tid 4 to smithi10729407-9 2020-06-12T02:39:11.508 INFO:tasks.rados.rados.0.smithi107.stdout:10575: done (1 left) 2020-06-12T02:39:11.508 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:39:11.508 INFO:tasks.rados.rados.0.smithi107.stdout:10576: seq_num 10575 ranges {0=720896,720896=720896,1441792=483328} 2020-06-12T02:39:11.513 INFO:tasks.rados.rados.0.smithi107.stdout:10576: writing smithi10729407-7 from 0 to 720896 tid 1 2020-06-12T02:39:11.517 INFO:tasks.rados.rados.0.smithi107.stdout:10576: writing smithi10729407-7 from 720896 to 1441792 tid 2 2020-06-12T02:39:11.520 INFO:tasks.rados.rados.0.smithi107.stdout:10576: writing smithi10729407-7 from 1441792 to 1925120 tid 3 2020-06-12T02:39:11.520 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:39:11.520 INFO:tasks.rados.rados.0.smithi107.stdout:10577: seq_num 10576 ranges {1818624=737280,2555904=606208,3162112=434176,3596288=163840} 2020-06-12T02:39:11.530 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:39:11.529+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:39:11.534 INFO:tasks.rados.rados.0.smithi107.stdout:10577: writing smithi10729407-5 from 1818624 to 2555904 tid 1 2020-06-12T02:39:11.537 INFO:tasks.rados.rados.0.smithi107.stdout:10577: writing smithi10729407-5 from 2555904 to 3162112 tid 2 2020-06-12T02:39:11.539 INFO:tasks.rados.rados.0.smithi107.stdout:10577: writing smithi10729407-5 from 3162112 to 3596288 tid 3 2020-06-12T02:39:11.541 INFO:tasks.rados.rados.0.smithi107.stdout:10577: writing smithi10729407-5 from 3596288 to 3760128 tid 4 2020-06-12T02:39:11.541 INFO:tasks.rados.rados.0.smithi107.stdout:10578: delete oid 16 current snap is 0 2020-06-12T02:39:11.541 INFO:tasks.rados.rados.0.smithi107.stdout:10571: finishing write tid 6 to smithi10729407-9 2020-06-12T02:39:11.541 INFO:tasks.rados.rados.0.smithi107.stdout:10571: finishing write tid 7 to smithi10729407-9 2020-06-12T02:39:11.542 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1825 (ObjNum 10570 snap 0 seq_num 10570) dirty exists 2020-06-12T02:39:11.542 INFO:tasks.rados.rados.0.smithi107.stdout:10571: left oid 9 (ObjNum 10570 snap 0 seq_num 10570) 2020-06-12T02:39:11.542 INFO:tasks.rados.rados.0.smithi107.stdout:10576: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:11.542 INFO:tasks.rados.rados.0.smithi107.stdout:10576: finishing write tid 2 to smithi10729407-7 2020-06-12T02:39:11.542 INFO:tasks.rados.rados.0.smithi107.stdout:10576: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:11.543 INFO:tasks.rados.rados.0.smithi107.stdout:10576: finishing write tid 5 to smithi10729407-7 2020-06-12T02:39:11.543 INFO:tasks.rados.rados.0.smithi107.stdout:10576: finishing write tid 6 to smithi10729407-7 2020-06-12T02:39:11.543 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1756 (ObjNum 10575 snap 0 seq_num 10575) dirty exists 2020-06-12T02:39:11.543 INFO:tasks.rados.rados.0.smithi107.stdout:10576: left oid 7 (ObjNum 10575 snap 0 seq_num 10575) 2020-06-12T02:39:11.544 INFO:tasks.rados.rados.0.smithi107.stdout:10571: done (3 left) 2020-06-12T02:39:11.544 INFO:tasks.rados.rados.0.smithi107.stdout:10576: done (2 left) 2020-06-12T02:39:11.544 INFO:tasks.rados.rados.0.smithi107.stdout:10578: done (1 left) 2020-06-12T02:39:11.545 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:39:11.545 INFO:tasks.rados.rados.0.smithi107.stdout:10579: seq_num 10578 ranges {0=770048,770048=770048,1540096=434176} 2020-06-12T02:39:11.549 INFO:tasks.rados.rados.0.smithi107.stdout:10579: writing smithi10729407-20 from 0 to 770048 tid 1 2020-06-12T02:39:11.553 INFO:tasks.rados.rados.0.smithi107.stdout:10579: writing smithi10729407-20 from 770048 to 1540096 tid 2 2020-06-12T02:39:11.556 INFO:tasks.rados.rados.0.smithi107.stdout:10579: writing smithi10729407-20 from 1540096 to 1974272 tid 3 2020-06-12T02:39:11.556 INFO:tasks.rados.rados.0.smithi107.stdout:10580: delete oid 15 current snap is 0 2020-06-12T02:39:11.557 INFO:tasks.rados.rados.0.smithi107.stdout:10577: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:11.557 INFO:tasks.rados.rados.0.smithi107.stdout:10577: finishing write tid 2 to smithi10729407-5 2020-06-12T02:39:11.557 INFO:tasks.rados.rados.0.smithi107.stdout:10577: finishing write tid 3 to smithi10729407-5 2020-06-12T02:39:11.557 INFO:tasks.rados.rados.0.smithi107.stdout:10577: finishing write tid 4 to smithi10729407-5 2020-06-12T02:39:11.558 INFO:tasks.rados.rados.0.smithi107.stdout:10577: finishing write tid 6 to smithi10729407-5 2020-06-12T02:39:11.558 INFO:tasks.rados.rados.0.smithi107.stdout:10577: finishing write tid 7 to smithi10729407-5 2020-06-12T02:39:11.558 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 1010 (ObjNum 10576 snap 0 seq_num 10576) dirty exists 2020-06-12T02:39:11.558 INFO:tasks.rados.rados.0.smithi107.stdout:10577: left oid 5 (ObjNum 10576 snap 0 seq_num 10576) 2020-06-12T02:39:11.559 INFO:tasks.rados.rados.0.smithi107.stdout:10579: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:11.559 INFO:tasks.rados.rados.0.smithi107.stdout:10577: done (2 left) 2020-06-12T02:39:11.559 INFO:tasks.rados.rados.0.smithi107.stdout:10580: done (1 left) 2020-06-12T02:39:11.560 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:39:11.560 INFO:tasks.rados.rados.0.smithi107.stdout:10581: seq_num 10580 ranges {0=802816,802816=540672,1343488=409600,1753088=253952} 2020-06-12T02:39:11.565 INFO:tasks.rados.rados.0.smithi107.stdout:10581: writing smithi10729407-12 from 0 to 802816 tid 1 2020-06-12T02:39:11.573 INFO:tasks.rados.rados.0.smithi107.stdout:10581: writing smithi10729407-12 from 802816 to 1343488 tid 2 2020-06-12T02:39:11.573 INFO:tasks.rados.rados.0.smithi107.stdout:10581: writing smithi10729407-12 from 1343488 to 1753088 tid 3 2020-06-12T02:39:11.578 INFO:tasks.rados.rados.0.smithi107.stdout:10581: writing smithi10729407-12 from 1753088 to 2007040 tid 4 2020-06-12T02:39:11.578 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:39:11.578 INFO:tasks.rados.rados.0.smithi107.stdout:10582: seq_num 10581 ranges {1769472=417792,2187264=425984,2613248=622592,3235840=434176,3670016=122880} 2020-06-12T02:39:11.582 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:39:11.582 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:39:11.582 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:39:11.582 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:39:11.583 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:39:11.586 INFO:tasks.rados.rados.0.smithi107.stdout:10582: writing smithi10729407-11 from 1769472 to 2187264 tid 1 2020-06-12T02:39:11.588 INFO:tasks.rados.rados.0.smithi107.stdout:10582: writing smithi10729407-11 from 2187264 to 2613248 tid 2 2020-06-12T02:39:11.592 INFO:tasks.rados.rados.0.smithi107.stdout:10582: writing smithi10729407-11 from 2613248 to 3235840 tid 3 2020-06-12T02:39:11.593 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-06-12T02:39:11.595 INFO:tasks.rados.rados.0.smithi107.stdout:10582: writing smithi10729407-11 from 3235840 to 3670016 tid 4 2020-06-12T02:39:11.597 INFO:tasks.rados.rados.0.smithi107.stdout:10582: writing smithi10729407-11 from 3670016 to 3792896 tid 5 2020-06-12T02:39:11.597 INFO:tasks.rados.rados.0.smithi107.stdout:10583: delete oid 18 current snap is 0 2020-06-12T02:39:11.597 INFO:tasks.rados.rados.0.smithi107.stdout:10579: finishing write tid 2 to smithi10729407-20 2020-06-12T02:39:11.597 INFO:tasks.rados.rados.0.smithi107.stdout:10579: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:11.598 INFO:tasks.rados.rados.0.smithi107.stdout:10579: finishing write tid 5 to smithi10729407-20 2020-06-12T02:39:11.598 INFO:tasks.rados.rados.0.smithi107.stdout:10581: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:11.598 INFO:tasks.rados.rados.0.smithi107.stdout:10579: finishing write tid 6 to smithi10729407-20 2020-06-12T02:39:11.598 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 826 (ObjNum 10578 snap 0 seq_num 10578) dirty exists 2020-06-12T02:39:11.599 INFO:tasks.rados.rados.0.smithi107.stdout:10579: left oid 20 (ObjNum 10578 snap 0 seq_num 10578) 2020-06-12T02:39:11.599 INFO:tasks.rados.rados.0.smithi107.stdout:10581: finishing write tid 2 to smithi10729407-12 2020-06-12T02:39:11.599 INFO:tasks.rados.rados.0.smithi107.stdout:10581: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:11.599 INFO:tasks.rados.rados.0.smithi107.stdout:10581: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:11.599 INFO:tasks.rados.rados.0.smithi107.stdout:10581: finishing write tid 6 to smithi10729407-12 2020-06-12T02:39:11.599 INFO:tasks.rados.rados.0.smithi107.stdout:10581: finishing write tid 7 to smithi10729407-12 2020-06-12T02:39:11.600 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1944 (ObjNum 10580 snap 0 seq_num 10580) dirty exists 2020-06-12T02:39:11.600 INFO:tasks.rados.rados.0.smithi107.stdout:10581: left oid 12 (ObjNum 10580 snap 0 seq_num 10580) 2020-06-12T02:39:11.600 INFO:tasks.rados.rados.0.smithi107.stdout:10582: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:11.601 INFO:tasks.rados.rados.0.smithi107.stdout:10579: done (3 left) 2020-06-12T02:39:11.601 INFO:tasks.rados.rados.0.smithi107.stdout:10581: done (2 left) 2020-06-12T02:39:11.601 INFO:tasks.rados.rados.0.smithi107.stdout:10583: done (1 left) 2020-06-12T02:39:11.602 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:39:11.602 INFO:tasks.rados.rados.0.smithi107.stdout:10584: seq_num 10583 ranges {0=450560,450560=647168,1097728=581632,1679360=376832} 2020-06-12T02:39:11.602 INFO:tasks.rados.rados.0.smithi107.stdout:10584: writing smithi10729407-8 from 0 to 450560 tid 1 2020-06-12T02:39:11.606 INFO:tasks.rados.rados.0.smithi107.stdout:10584: writing smithi10729407-8 from 450560 to 1097728 tid 2 2020-06-12T02:39:11.610 INFO:tasks.rados.rados.0.smithi107.stdout:10584: writing smithi10729407-8 from 1097728 to 1679360 tid 3 2020-06-12T02:39:11.612 INFO:tasks.rados.rados.0.smithi107.stdout:10584: writing smithi10729407-8 from 1679360 to 2056192 tid 4 2020-06-12T02:39:11.612 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:39:11.613 INFO:tasks.rados.rados.0.smithi107.stdout:10585: seq_num 10584 ranges {1400832=466944,1867776=532480,2400256=794624,3194880=278528} 2020-06-12T02:39:11.622 INFO:tasks.rados.rados.0.smithi107.stdout:10585: writing smithi10729407-19 from 1400832 to 1867776 tid 1 2020-06-12T02:39:11.625 INFO:tasks.rados.rados.0.smithi107.stdout:10585: writing smithi10729407-19 from 1867776 to 2400256 tid 2 2020-06-12T02:39:11.630 INFO:tasks.rados.rados.0.smithi107.stdout:10585: writing smithi10729407-19 from 2400256 to 3194880 tid 3 2020-06-12T02:39:11.632 INFO:tasks.rados.rados.0.smithi107.stdout:10585: writing smithi10729407-19 from 3194880 to 3473408 tid 4 2020-06-12T02:39:11.632 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:39:11.633 INFO:tasks.rados.rados.0.smithi107.stdout:10586: seq_num 10585 ranges {0=483328,483328=409600,892928=598016,1490944=491520,1982464=106496} 2020-06-12T02:39:11.634 INFO:tasks.rados.rados.0.smithi107.stdout:10586: writing smithi10729407-15 from 0 to 483328 tid 1 2020-06-12T02:39:11.637 INFO:tasks.rados.rados.0.smithi107.stdout:10586: writing smithi10729407-15 from 483328 to 892928 tid 2 2020-06-12T02:39:11.640 INFO:tasks.rados.rados.0.smithi107.stdout:10586: writing smithi10729407-15 from 892928 to 1490944 tid 3 2020-06-12T02:39:11.643 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:39:11.641+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:39:11.643 INFO:tasks.rados.rados.0.smithi107.stdout:10586: writing smithi10729407-15 from 1490944 to 1982464 tid 4 2020-06-12T02:39:11.645 INFO:tasks.rados.rados.0.smithi107.stdout:10586: writing smithi10729407-15 from 1982464 to 2088960 tid 5 2020-06-12T02:39:11.645 INFO:tasks.rados.rados.0.smithi107.stdout:10587: delete oid 14 current snap is 0 2020-06-12T02:39:11.645 INFO:tasks.rados.rados.0.smithi107.stdout:10582: finishing write tid 2 to smithi10729407-11 2020-06-12T02:39:11.646 INFO:tasks.rados.rados.0.smithi107.stdout:10582: finishing write tid 3 to smithi10729407-11 2020-06-12T02:39:11.646 INFO:tasks.rados.rados.0.smithi107.stdout:10582: finishing write tid 4 to smithi10729407-11 2020-06-12T02:39:11.646 INFO:tasks.rados.rados.0.smithi107.stdout:10582: finishing write tid 5 to smithi10729407-11 2020-06-12T02:39:11.646 INFO:tasks.rados.rados.0.smithi107.stdout:10584: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:11.647 INFO:tasks.rados.rados.0.smithi107.stdout:10582: finishing write tid 7 to smithi10729407-11 2020-06-12T02:39:11.647 INFO:tasks.rados.rados.0.smithi107.stdout:10582: finishing write tid 8 to smithi10729407-11 2020-06-12T02:39:11.647 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1719 (ObjNum 10581 snap 0 seq_num 10581) dirty exists 2020-06-12T02:39:11.648 INFO:tasks.rados.rados.0.smithi107.stdout:10582: left oid 11 (ObjNum 10581 snap 0 seq_num 10581) 2020-06-12T02:39:11.648 INFO:tasks.rados.rados.0.smithi107.stdout:10584: finishing write tid 2 to smithi10729407-8 2020-06-12T02:39:11.648 INFO:tasks.rados.rados.0.smithi107.stdout:10584: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:11.648 INFO:tasks.rados.rados.0.smithi107.stdout:10584: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:11.648 INFO:tasks.rados.rados.0.smithi107.stdout:10584: finishing write tid 6 to smithi10729407-8 2020-06-12T02:39:11.649 INFO:tasks.rados.rados.0.smithi107.stdout:10584: finishing write tid 7 to smithi10729407-8 2020-06-12T02:39:11.649 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1963 (ObjNum 10583 snap 0 seq_num 10583) dirty exists 2020-06-12T02:39:11.649 INFO:tasks.rados.rados.0.smithi107.stdout:10584: left oid 8 (ObjNum 10583 snap 0 seq_num 10583) 2020-06-12T02:39:11.649 INFO:tasks.rados.rados.0.smithi107.stdout:10585: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:11.649 INFO:tasks.rados.rados.0.smithi107.stdout:10586: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:11.650 INFO:tasks.rados.rados.0.smithi107.stdout:10585: finishing write tid 2 to smithi10729407-19 2020-06-12T02:39:11.650 INFO:tasks.rados.rados.0.smithi107.stdout:10585: finishing write tid 3 to smithi10729407-19 2020-06-12T02:39:11.650 INFO:tasks.rados.rados.0.smithi107.stdout:10582: done (4 left) 2020-06-12T02:39:11.651 INFO:tasks.rados.rados.0.smithi107.stdout:10584: done (3 left) 2020-06-12T02:39:11.651 INFO:tasks.rados.rados.0.smithi107.stdout:10587: done (2 left) 2020-06-12T02:39:11.651 INFO:tasks.rados.rados.0.smithi107.stdout:10588: delete oid 4 current snap is 0 2020-06-12T02:39:11.652 INFO:tasks.rados.rados.0.smithi107.stdout:10585: finishing write tid 4 to smithi10729407-19 2020-06-12T02:39:11.652 INFO:tasks.rados.rados.0.smithi107.stdout:10585: finishing write tid 6 to smithi10729407-19 2020-06-12T02:39:11.653 INFO:tasks.rados.rados.0.smithi107.stdout:10588: done (2 left) 2020-06-12T02:39:11.654 INFO:tasks.rados.rados.0.smithi107.stdout:10589: delete oid 8 current snap is 0 2020-06-12T02:39:11.657 INFO:tasks.rados.rados.0.smithi107.stdout:10585: finishing write tid 7 to smithi10729407-19 2020-06-12T02:39:11.657 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1949 (ObjNum 10584 snap 0 seq_num 10584) dirty exists 2020-06-12T02:39:11.658 INFO:tasks.rados.rados.0.smithi107.stdout:10585: left oid 19 (ObjNum 10584 snap 0 seq_num 10584) 2020-06-12T02:39:11.658 INFO:tasks.rados.rados.0.smithi107.stdout:10586: finishing write tid 2 to smithi10729407-15 2020-06-12T02:39:11.660 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-06-12T02:39:11.661 INFO:tasks.rados.rados.0.smithi107.stdout:10586: finishing write tid 3 to smithi10729407-15 2020-06-12T02:39:11.661 INFO:tasks.rados.rados.0.smithi107.stdout:10586: finishing write tid 4 to smithi10729407-15 2020-06-12T02:39:11.664 INFO:tasks.rados.rados.0.smithi107.stdout:10586: finishing write tid 5 to smithi10729407-15 2020-06-12T02:39:11.666 INFO:tasks.rados.rados.0.smithi107.stdout:10586: finishing write tid 7 to smithi10729407-15 2020-06-12T02:39:11.667 INFO:tasks.rados.rados.0.smithi107.stdout:10585: done (2 left) 2020-06-12T02:39:11.668 INFO:tasks.rados.rados.0.smithi107.stdout:10589: done (1 left) 2020-06-12T02:39:11.668 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:39:11.668 INFO:tasks.rados.rados.0.smithi107.stdout:10590: seq_num 10589 ranges {0=548864,548864=753664,1302528=581632,1884160=270336} 2020-06-12T02:39:11.670 INFO:tasks.rados.rados.0.smithi107.stdout:10590: writing smithi10729407-16 from 0 to 548864 tid 1 2020-06-12T02:39:11.676 INFO:tasks.rados.rados.0.smithi107.stdout:10590: writing smithi10729407-16 from 548864 to 1302528 tid 2 2020-06-12T02:39:11.680 INFO:tasks.rados.rados.0.smithi107.stdout:10590: writing smithi10729407-16 from 1302528 to 1884160 tid 3 2020-06-12T02:39:11.682 INFO:tasks.rados.rados.0.smithi107.stdout:10590: writing smithi10729407-16 from 1884160 to 2154496 tid 4 2020-06-12T02:39:11.682 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:39:11.683 INFO:tasks.rados.rados.0.smithi107.stdout:10586: finishing write tid 8 to smithi10729407-15 2020-06-12T02:39:11.683 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1969 (ObjNum 10585 snap 0 seq_num 10585) dirty exists 2020-06-12T02:39:11.683 INFO:tasks.rados.rados.0.smithi107.stdout:10586: left oid 15 (ObjNum 10585 snap 0 seq_num 10585) 2020-06-12T02:39:11.683 INFO:tasks.rados.rados.0.smithi107.stdout:10590: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:11.684 INFO:tasks.rados.rados.0.smithi107.stdout:10591: seq_num 10590 ranges {1925120=565248,2490368=638976,3129344=737280,3866624=229376} 2020-06-12T02:39:11.700 INFO:tasks.rados.rados.0.smithi107.stdout:10591: writing smithi10729407-7 from 1925120 to 2490368 tid 1 2020-06-12T02:39:11.701 INFO:tasks.rados.rados.0.smithi107.stdout:10591: writing smithi10729407-7 from 2490368 to 3129344 tid 2 2020-06-12T02:39:11.705 INFO:tasks.rados.rados.0.smithi107.stdout:10591: writing smithi10729407-7 from 3129344 to 3866624 tid 3 2020-06-12T02:39:11.707 INFO:tasks.rados.rados.0.smithi107.stdout:10591: writing smithi10729407-7 from 3866624 to 4096000 tid 4 2020-06-12T02:39:11.707 INFO:tasks.rados.rados.0.smithi107.stdout:10586: done (2 left) 2020-06-12T02:39:11.707 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:39:11.708 INFO:tasks.rados.rados.0.smithi107.stdout:10592: seq_num 10591 ranges {0=581632,581632=565248,1146880=548864,1695744=491520} 2020-06-12T02:39:11.710 INFO:tasks.rados.rados.0.smithi107.stdout:10592: writing smithi10729407-14 from 0 to 581632 tid 1 2020-06-12T02:39:11.713 INFO:tasks.rados.rados.0.smithi107.stdout:10592: writing smithi10729407-14 from 581632 to 1146880 tid 2 2020-06-12T02:39:11.717 INFO:tasks.rados.rados.0.smithi107.stdout:10592: writing smithi10729407-14 from 1146880 to 1695744 tid 3 2020-06-12T02:39:11.723 INFO:tasks.rados.rados.0.smithi107.stdout:10592: writing smithi10729407-14 from 1695744 to 2187264 tid 4 2020-06-12T02:39:11.723 INFO:tasks.rados.rados.0.smithi107.stdout:10593: delete oid 20 current snap is 0 2020-06-12T02:39:11.723 INFO:tasks.rados.rados.0.smithi107.stdout:10590: finishing write tid 2 to smithi10729407-16 2020-06-12T02:39:11.724 INFO:tasks.rados.rados.0.smithi107.stdout:10590: finishing write tid 3 to smithi10729407-16 2020-06-12T02:39:11.724 INFO:tasks.rados.rados.0.smithi107.stdout:10590: finishing write tid 4 to smithi10729407-16 2020-06-12T02:39:11.724 INFO:tasks.rados.rados.0.smithi107.stdout:10590: finishing write tid 6 to smithi10729407-16 2020-06-12T02:39:11.724 INFO:tasks.rados.rados.0.smithi107.stdout:10590: finishing write tid 7 to smithi10729407-16 2020-06-12T02:39:11.734 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1893 (ObjNum 10589 snap 0 seq_num 10589) dirty exists 2020-06-12T02:39:11.734 INFO:tasks.rados.rados.0.smithi107.stdout:10590: left oid 16 (ObjNum 10589 snap 0 seq_num 10589) 2020-06-12T02:39:11.734 INFO:tasks.rados.rados.0.smithi107.stdout:10591: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:11.734 INFO:tasks.rados.rados.0.smithi107.stdout:10591: finishing write tid 2 to smithi10729407-7 2020-06-12T02:39:11.735 INFO:tasks.rados.rados.0.smithi107.stdout:10591: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:11.735 INFO:tasks.rados.rados.0.smithi107.stdout:10592: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:11.735 INFO:tasks.rados.rados.0.smithi107.stdout:10591: finishing write tid 4 to smithi10729407-7 2020-06-12T02:39:11.735 INFO:tasks.rados.rados.0.smithi107.stdout:10591: finishing write tid 6 to smithi10729407-7 2020-06-12T02:39:11.737 INFO:tasks.rados.rados.0.smithi107.stdout:10591: finishing write tid 7 to smithi10729407-7 2020-06-12T02:39:11.737 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1761 (ObjNum 10590 snap 0 seq_num 10590) dirty exists 2020-06-12T02:39:11.737 INFO:tasks.rados.rados.0.smithi107.stdout:10591: left oid 7 (ObjNum 10590 snap 0 seq_num 10590) 2020-06-12T02:39:11.737 INFO:tasks.rados.rados.0.smithi107.stdout:10590: done (3 left) 2020-06-12T02:39:11.746 INFO:tasks.rados.rados.0.smithi107.stdout:10591: done (2 left) 2020-06-12T02:39:11.746 INFO:tasks.rados.rados.0.smithi107.stdout:10593: done (1 left) 2020-06-12T02:39:11.750 INFO:tasks.rados.rados.0.smithi107.stdout:10594: delete oid 15 current snap is 0 2020-06-12T02:39:11.750 INFO:tasks.rados.rados.0.smithi107.stdout:10594: done (1 left) 2020-06-12T02:39:11.750 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:39:11.750 INFO:tasks.rados.rados.0.smithi107.stdout:10595: seq_num 10594 ranges {0=638976,638976=622592,1261568=720896,1982464=262144} 2020-06-12T02:39:11.750 INFO:tasks.rados.rados.0.smithi107.stdout:10595: writing smithi10729407-15 from 0 to 638976 tid 1 2020-06-12T02:39:11.752 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:39:11.734+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:39:11.752 INFO:tasks.rados.rados.0.smithi107.stdout:10595: writing smithi10729407-15 from 638976 to 1261568 tid 2 2020-06-12T02:39:11.752 INFO:tasks.rados.rados.0.smithi107.stdout:10595: writing smithi10729407-15 from 1261568 to 1982464 tid 3 2020-06-12T02:39:11.752 INFO:tasks.rados.rados.0.smithi107.stdout:10595: writing smithi10729407-15 from 1982464 to 2244608 tid 4 2020-06-12T02:39:11.752 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:39:11.753 INFO:tasks.rados.rados.0.smithi107.stdout:10596: seq_num 10595 ranges {3792896=655360,4448256=507904,4956160=524288,5480448=425984,5906432=147456} 2020-06-12T02:39:11.765 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:39:11.765 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:39:11.765 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:39:11.766 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:39:11.766 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.766 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8464 3.fs0 3.57bc3c1f (undecoded) ondisk+retry+write+known_if_redirected e34)", 2020-06-12T02:39:11.766 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:53.549364+0000", 2020-06-12T02:39:11.766 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 78.207459929999999, 2020-06-12T02:39:11.767 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.003107494, 2020-06-12T02:39:11.767 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:11.767 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:39:11.767 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:11.767 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:11.768 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:11.768 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8464 2020-06-12T02:39:11.768 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.768 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:11.768 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.769 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:11.769 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.549364+0000", 2020-06-12T02:39:11.769 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:11.769 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.769 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.769 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:11.770 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.549364+0000", 2020-06-12T02:39:11.770 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023109609999999998 2020-06-12T02:39:11.770 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.770 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.770 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:11.771 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551675+0000", 2020-06-12T02:39:11.771 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3331e-05 2020-06-12T02:39:11.771 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.771 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.771 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:11.772 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551688+0000", 2020-06-12T02:39:11.772 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4173e-05 2020-06-12T02:39:11.772 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.772 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.772 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:11.772 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551703+0000", 2020-06-12T02:39:11.773 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0454e-05 2020-06-12T02:39:11.773 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.773 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.773 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:11.773 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551713+0000", 2020-06-12T02:39:11.777 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031069050000000001 2020-06-12T02:39:11.778 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.778 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.778 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.778 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554820+0000", 2020-06-12T02:39:11.778 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2820000000000002e-06 2020-06-12T02:39:11.779 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.779 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.779 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2020-06-12T02:39:11.779 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554826+0000", 2020-06-12T02:39:11.779 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99762648499999995 2020-06-12T02:39:11.780 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.780 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.780 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.780 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552453+0000", 2020-06-12T02:39:11.780 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.8902999999999998e-05 2020-06-12T02:39:11.780 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.781 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.781 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:11.782 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552472+0000", 2020-06-12T02:39:11.782 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.003107494 2020-06-12T02:39:11.782 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.782 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:11.782 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.783 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.783 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.783 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8465 3.fs0 3.57bc3c1f (undecoded) ondisk+retry+write+known_if_redirected e34)", 2020-06-12T02:39:11.783 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:53.551774+0000", 2020-06-12T02:39:11.783 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 78.205050294000003, 2020-06-12T02:39:11.784 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.000741442, 2020-06-12T02:39:11.784 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:11.785 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:39:11.785 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:11.785 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:11.785 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:11.786 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8465 2020-06-12T02:39:11.786 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.786 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:11.786 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.786 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:11.787 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551774+0000", 2020-06-12T02:39:11.787 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:11.787 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.787 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.787 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:11.787 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.551774+0000", 2020-06-12T02:39:11.788 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0010760539999999999 2020-06-12T02:39:11.788 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.788 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.788 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:11.788 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552850+0000", 2020-06-12T02:39:11.789 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0139000000000001e-05 2020-06-12T02:39:11.789 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.789 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.789 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:11.789 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552860+0000", 2020-06-12T02:39:11.790 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.7646999999999999e-05 2020-06-12T02:39:11.790 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.790 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.790 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:11.790 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552878+0000", 2020-06-12T02:39:11.790 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2916e-05 2020-06-12T02:39:11.791 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.791 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.791 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:11.791 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552890+0000", 2020-06-12T02:39:11.791 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0019574610000000002 2020-06-12T02:39:11.792 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.792 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.792 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.792 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554848+0000", 2020-06-12T02:39:11.792 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0399999999999995e-07 2020-06-12T02:39:11.793 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.793 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.793 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2020-06-12T02:39:11.793 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554849+0000", 2020-06-12T02:39:11.793 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99765586100000003 2020-06-12T02:39:11.793 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.794 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.794 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.794 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552504+0000", 2020-06-12T02:39:11.795 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.066e-05 2020-06-12T02:39:11.795 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.795 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.796 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:11.796 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552515+0000", 2020-06-12T02:39:11.796 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.000741442 2020-06-12T02:39:11.796 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.796 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:11.796 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.797 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.797 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.797 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8466 3.fs0 3.57bc3c1f (undecoded) ondisk+write+known_if_redirected e34)", 2020-06-12T02:39:11.797 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:53.552953+0000", 2020-06-12T02:39:11.797 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 78.203871014000001, 2020-06-12T02:39:11.798 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99960144299999998, 2020-06-12T02:39:11.798 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:11.798 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:39:11.798 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:11.798 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:11.799 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:11.799 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8466 2020-06-12T02:39:11.799 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.799 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:11.799 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.799 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:11.800 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552953+0000", 2020-06-12T02:39:11.800 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:11.800 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.800 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.800 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:11.801 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.552953+0000", 2020-06-12T02:39:11.801 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00013847100000000001 2020-06-12T02:39:11.801 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.801 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.801 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:11.802 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553091+0000", 2020-06-12T02:39:11.802 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7190000000000004e-06 2020-06-12T02:39:11.802 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.802 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.802 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:11.802 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553098+0000", 2020-06-12T02:39:11.803 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.171e-05 2020-06-12T02:39:11.803 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.803 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.803 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:11.803 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553110+0000", 2020-06-12T02:39:11.804 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5770000000000001e-06 2020-06-12T02:39:11.804 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.804 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.804 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:11.804 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553115+0000", 2020-06-12T02:39:11.804 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0017512739999999999 2020-06-12T02:39:11.805 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.805 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.805 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.805 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554867+0000", 2020-06-12T02:39:11.806 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1200000000000003e-07 2020-06-12T02:39:11.806 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.806 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.806 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2020-06-12T02:39:11.806 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554867+0000", 2020-06-12T02:39:11.807 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99767707100000003 2020-06-12T02:39:11.807 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.807 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.807 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.807 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552544+0000", 2020-06-12T02:39:11.808 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0009e-05 2020-06-12T02:39:11.808 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.808 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.808 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:11.808 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552554+0000", 2020-06-12T02:39:11.808 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99960144299999998 2020-06-12T02:39:11.809 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.809 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:11.809 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.809 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.810 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.810 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8467 3.fs0 3.57bc3c1f (undecoded) ondisk+write+known_if_redirected e34)", 2020-06-12T02:39:11.810 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:53.553172+0000", 2020-06-12T02:39:11.810 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 78.203651797999996, 2020-06-12T02:39:11.810 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99942086200000002, 2020-06-12T02:39:11.811 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:11.811 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:39:11.811 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:11.811 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:11.811 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:11.811 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8467 2020-06-12T02:39:11.812 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.812 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:11.812 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.812 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:11.812 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553172+0000", 2020-06-12T02:39:11.813 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:11.813 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.813 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.813 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:11.813 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553172+0000", 2020-06-12T02:39:11.813 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5913000000000003e-05 2020-06-12T02:39:11.814 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.814 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.814 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:11.814 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553248+0000", 2020-06-12T02:39:11.814 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7740000000000001e-06 2020-06-12T02:39:11.815 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.815 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.815 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:11.815 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553254+0000", 2020-06-12T02:39:11.816 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1494999999999999e-05 2020-06-12T02:39:11.816 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.816 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.816 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:11.816 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553265+0000", 2020-06-12T02:39:11.817 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5170000000000001e-06 2020-06-12T02:39:11.817 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.817 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.817 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:11.817 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553270+0000", 2020-06-12T02:39:11.818 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001616567 2020-06-12T02:39:11.818 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.818 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.818 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.818 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554886+0000", 2020-06-12T02:39:11.819 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7800000000000001e-07 2020-06-12T02:39:11.819 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.819 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.819 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2020-06-12T02:39:11.819 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554887+0000", 2020-06-12T02:39:11.819 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99769652399999997 2020-06-12T02:39:11.820 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.820 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.820 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.820 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552584+0000", 2020-06-12T02:39:11.820 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4939999999999992e-06 2020-06-12T02:39:11.821 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.821 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.821 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:11.821 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.552593+0000", 2020-06-12T02:39:11.821 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99942086200000002 2020-06-12T02:39:11.822 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.822 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:11.822 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.822 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.822 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.823 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:8468 3.fs0 3.57bc3c1f (undecoded) ondisk+read+rwordered+known_if_redirected e34)", 2020-06-12T02:39:11.824 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:53.553311+0000", 2020-06-12T02:39:11.825 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 78.203513200000003, 2020-06-12T02:39:11.825 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99988704799999994, 2020-06-12T02:39:11.825 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:11.825 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:39:11.826 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:11.826 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:11.826 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:11.826 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 8468 2020-06-12T02:39:11.827 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.827 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:11.827 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.827 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:11.827 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553311+0000", 2020-06-12T02:39:11.827 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:11.828 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.828 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.828 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:11.828 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553311+0000", 2020-06-12T02:39:11.828 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2050000000000001e-05 2020-06-12T02:39:11.829 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.829 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.829 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:11.829 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553333+0000", 2020-06-12T02:39:11.829 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5489999999999998e-06 2020-06-12T02:39:11.830 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.830 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.830 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:11.830 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553336+0000", 2020-06-12T02:39:11.830 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1620000000000002e-06 2020-06-12T02:39:11.831 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.831 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.831 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:11.831 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553344+0000", 2020-06-12T02:39:11.831 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.807e-06 2020-06-12T02:39:11.831 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.832 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.832 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:11.832 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.553346+0000", 2020-06-12T02:39:11.832 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001560575 2020-06-12T02:39:11.832 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.833 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.833 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.833 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554907+0000", 2020-06-12T02:39:11.833 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1429999999999999e-06 2020-06-12T02:39:11.833 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.834 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.834 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2020-06-12T02:39:11.834 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:53.554908+0000", 2020-06-12T02:39:11.834 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99827844600000004 2020-06-12T02:39:11.834 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.835 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.835 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.835 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.553187+0000", 2020-06-12T02:39:11.835 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1316e-05 2020-06-12T02:39:11.835 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.836 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.836 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:11.836 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:54.553198+0000", 2020-06-12T02:39:11.836 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.99988704799999994 2020-06-12T02:39:11.836 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.836 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:11.837 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.837 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.837 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.837 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11992 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [append 0~581632] snapc 0=[] ondisk+write+known_if_redirected e56)", 2020-06-12T02:39:11.837 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:23.783061+0000", 2020-06-12T02:39:11.838 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 47.973763019000003, 2020-06-12T02:39:11.838 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022850993, 2020-06-12T02:39:11.840 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:11.840 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:11.840 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:11.840 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:11.840 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:11.841 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11992 2020-06-12T02:39:11.841 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.841 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:11.841 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.842 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:11.842 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783061+0000", 2020-06-12T02:39:11.842 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:11.842 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.843 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.843 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:11.843 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783061+0000", 2020-06-12T02:39:11.843 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047999300000000001 2020-06-12T02:39:11.844 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.844 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.844 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:11.844 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783541+0000", 2020-06-12T02:39:11.845 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5649999999999998e-06 2020-06-12T02:39:11.845 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.845 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.845 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:11.846 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783548+0000", 2020-06-12T02:39:11.846 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3559e-05 2020-06-12T02:39:11.846 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.847 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.847 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:11.847 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783561+0000", 2020-06-12T02:39:11.847 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3539999999999996e-06 2020-06-12T02:39:11.847 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.848 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.848 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:11.848 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.783567+0000", 2020-06-12T02:39:11.848 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001353604 2020-06-12T02:39:11.849 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.849 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.849 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.850 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.784921+0000", 2020-06-12T02:39:11.850 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000111853 2020-06-12T02:39:11.850 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.850 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.850 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:11.851 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.785033+0000", 2020-06-12T02:39:11.851 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051647919999999996 2020-06-12T02:39:11.851 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.851 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.852 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.852 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.790198+0000", 2020-06-12T02:39:11.852 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000110742 2020-06-12T02:39:11.852 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.852 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.853 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:11.853 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.790308+0000", 2020-06-12T02:39:11.853 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00090050099999999995 2020-06-12T02:39:11.853 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.853 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.854 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:11.854 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.791209+0000", 2020-06-12T02:39:11.854 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010078284999999999 2020-06-12T02:39:11.854 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.854 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.855 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:11.855 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.801287+0000", 2020-06-12T02:39:11.855 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045443109999999997 2020-06-12T02:39:11.855 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.855 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.856 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:11.856 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.805832+0000", 2020-06-12T02:39:11.856 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.0433999999999996e-05 2020-06-12T02:39:11.857 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.857 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.857 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:11.857 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.805912+0000", 2020-06-12T02:39:11.857 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022850993 2020-06-12T02:39:11.858 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.858 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:11.858 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.858 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.858 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.859 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11994 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected e56)", 2020-06-12T02:39:11.859 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:23.788734+0000", 2020-06-12T02:39:11.859 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 47.968090373999999, 2020-06-12T02:39:11.860 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024022136999999999, 2020-06-12T02:39:11.860 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:11.860 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:11.860 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:11.860 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:11.861 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:11.861 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11994 2020-06-12T02:39:11.861 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.861 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:11.861 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.862 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:11.862 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.788734+0000", 2020-06-12T02:39:11.862 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:11.862 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.862 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.863 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:11.863 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.788734+0000", 2020-06-12T02:39:11.863 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00034212499999999998 2020-06-12T02:39:11.863 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.863 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.864 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:11.865 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789076+0000", 2020-06-12T02:39:11.865 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.254e-06 2020-06-12T02:39:11.866 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.866 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.866 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:11.866 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789081+0000", 2020-06-12T02:39:11.866 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8249999999999994e-06 2020-06-12T02:39:11.867 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.867 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.867 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:11.867 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789090+0000", 2020-06-12T02:39:11.867 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2259999999999999e-06 2020-06-12T02:39:11.868 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.868 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.868 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:11.868 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789094+0000", 2020-06-12T02:39:11.868 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2913000000000001e-05 2020-06-12T02:39:11.869 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.869 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.869 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.869 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789127+0000", 2020-06-12T02:39:11.869 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2001000000000005e-05 2020-06-12T02:39:11.870 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.870 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.870 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:11.870 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789189+0000", 2020-06-12T02:39:11.870 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0097474550000000004 2020-06-12T02:39:11.871 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.871 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.871 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.871 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.798936+0000", 2020-06-12T02:39:11.871 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9882000000000005e-05 2020-06-12T02:39:11.871 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.872 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.872 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:11.872 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.799016+0000", 2020-06-12T02:39:11.872 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00048691000000000001 2020-06-12T02:39:11.872 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.873 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.873 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:11.873 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.799503+0000", 2020-06-12T02:39:11.873 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002766363 2020-06-12T02:39:11.874 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.874 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.874 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:11.874 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.802270+0000", 2020-06-12T02:39:11.874 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010404759 2020-06-12T02:39:11.875 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.875 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.875 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:11.875 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.812674+0000", 2020-06-12T02:39:11.876 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1423999999999998e-05 2020-06-12T02:39:11.876 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.876 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.876 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:11.876 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.812756+0000", 2020-06-12T02:39:11.877 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024022136999999999 2020-06-12T02:39:11.877 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.877 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:11.877 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.878 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.878 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.878 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11995 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e56)", 2020-06-12T02:39:11.878 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:23.789128+0000", 2020-06-12T02:39:11.878 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 47.967695927999998, 2020-06-12T02:39:11.879 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031795508, 2020-06-12T02:39:11.879 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:11.879 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:11.879 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:11.879 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:11.879 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:11.880 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11995 2020-06-12T02:39:11.880 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.880 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:11.880 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.880 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:11.881 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789128+0000", 2020-06-12T02:39:11.881 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:11.881 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.881 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.881 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:11.882 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789128+0000", 2020-06-12T02:39:11.882 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4419999999999998e-05 2020-06-12T02:39:11.882 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.882 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.882 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:11.882 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789172+0000", 2020-06-12T02:39:11.883 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7110000000000001e-06 2020-06-12T02:39:11.883 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.883 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.883 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:11.884 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789176+0000", 2020-06-12T02:39:11.884 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5089999999999999e-06 2020-06-12T02:39:11.884 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.884 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.884 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:11.885 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789183+0000", 2020-06-12T02:39:11.885 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.497e-06 2020-06-12T02:39:11.885 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.885 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.886 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:11.886 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789185+0000", 2020-06-12T02:39:11.886 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012172176 2020-06-12T02:39:11.886 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.887 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.887 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.887 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.801357+0000", 2020-06-12T02:39:11.887 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4264000000000006e-05 2020-06-12T02:39:11.887 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.887 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.888 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:11.888 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.801432+0000", 2020-06-12T02:39:11.888 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00029680499999999997 2020-06-12T02:39:11.888 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.888 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.889 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:11.889 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.801728+0000", 2020-06-12T02:39:11.889 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00066278300000000001 2020-06-12T02:39:11.889 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.889 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.890 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:11.890 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.802391+0000", 2020-06-12T02:39:11.890 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018429867999999999 2020-06-12T02:39:11.890 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.890 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.890 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:11.891 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.820821+0000", 2020-06-12T02:39:11.891 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000102475 2020-06-12T02:39:11.891 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.891 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.891 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:11.892 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.820924+0000", 2020-06-12T02:39:11.892 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031795508 2020-06-12T02:39:11.892 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.892 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:11.892 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.892 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.893 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.893 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11996 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e56)", 2020-06-12T02:39:11.893 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:23.789235+0000", 2020-06-12T02:39:11.893 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 47.967589077, 2020-06-12T02:39:11.893 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.034785745, 2020-06-12T02:39:11.894 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:11.894 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:11.894 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:11.895 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:11.895 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:11.895 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11996 2020-06-12T02:39:11.895 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.895 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:11.895 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.896 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:11.896 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789235+0000", 2020-06-12T02:39:11.896 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:11.897 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.897 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.897 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:11.897 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789235+0000", 2020-06-12T02:39:11.897 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6375000000000001e-05 2020-06-12T02:39:11.897 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.898 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.898 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:11.898 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789261+0000", 2020-06-12T02:39:11.898 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2679999999999999e-06 2020-06-12T02:39:11.898 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.899 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.899 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:11.899 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789265+0000", 2020-06-12T02:39:11.899 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6039999999999998e-06 2020-06-12T02:39:11.899 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.900 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.900 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:11.900 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789271+0000", 2020-06-12T02:39:11.900 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4219999999999999e-06 2020-06-12T02:39:11.900 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.900 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.901 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:11.901 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.789274+0000", 2020-06-12T02:39:11.901 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013034463 2020-06-12T02:39:11.901 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.901 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.902 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.902 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.802308+0000", 2020-06-12T02:39:11.902 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8862000000000005e-05 2020-06-12T02:39:11.902 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.902 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.903 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:11.903 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.802387+0000", 2020-06-12T02:39:11.903 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019102297000000001 2020-06-12T02:39:11.903 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.903 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.903 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.904 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.821489+0000", 2020-06-12T02:39:11.904 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8838999999999996e-05 2020-06-12T02:39:11.904 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.904 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.905 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:11.905 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.821558+0000", 2020-06-12T02:39:11.905 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002462615 2020-06-12T02:39:11.905 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.905 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.906 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:11.906 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.824021+0000", 2020-06-12T02:39:11.906 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.034785745 2020-06-12T02:39:11.908 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.908 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:11.908 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.908 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.908 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.909 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:15005 3.ds0 3:b3445631:::smithi10729407-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e59)", 2020-06-12T02:39:11.909 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:32.192302+0000", 2020-06-12T02:39:11.909 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 39.564521841999998, 2020-06-12T02:39:11.909 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024128666, 2020-06-12T02:39:11.909 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:11.910 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:11.910 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:11.910 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:11.910 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:11.910 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 15005 2020-06-12T02:39:11.911 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.911 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:11.911 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.911 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:11.911 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192302+0000", 2020-06-12T02:39:11.912 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:11.912 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.912 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.912 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:11.912 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192302+0000", 2020-06-12T02:39:11.912 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3415999999999999e-05 2020-06-12T02:39:11.913 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.913 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.913 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:11.913 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192346+0000", 2020-06-12T02:39:11.913 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.93e-06 2020-06-12T02:39:11.914 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.914 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.914 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:11.915 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192352+0000", 2020-06-12T02:39:11.915 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3397000000000001e-05 2020-06-12T02:39:11.915 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.915 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.915 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:11.915 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192365+0000", 2020-06-12T02:39:11.916 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3950000000000003e-06 2020-06-12T02:39:11.916 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.916 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.916 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:11.916 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.192369+0000", 2020-06-12T02:39:11.917 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010208887999999999 2020-06-12T02:39:11.917 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.917 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.917 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.917 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.202578+0000", 2020-06-12T02:39:11.918 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000105611 2020-06-12T02:39:11.918 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.918 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.918 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:11.918 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.202684+0000", 2020-06-12T02:39:11.918 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0091288959999999992 2020-06-12T02:39:11.919 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.919 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.919 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.919 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.211813+0000", 2020-06-12T02:39:11.919 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000101798 2020-06-12T02:39:11.920 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.920 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.920 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:11.920 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.211914+0000", 2020-06-12T02:39:11.920 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0045163349999999998 2020-06-12T02:39:11.920 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.921 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.921 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:11.921 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.216431+0000", 2020-06-12T02:39:11.921 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024128666 2020-06-12T02:39:11.921 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.922 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:11.922 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.922 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.922 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.922 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:16810 3.ds0 3:b983fd28:::smithi10729407-1:head [append 0~524288] snapc 0=[] ondisk+write+known_if_redirected e61)", 2020-06-12T02:39:11.923 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:37.642363+0000", 2020-06-12T02:39:11.923 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.114460588, 2020-06-12T02:39:11.923 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022969679, 2020-06-12T02:39:11.923 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:11.923 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:11.924 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:11.924 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:11.924 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:11.924 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 16810 2020-06-12T02:39:11.924 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.924 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:11.925 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.925 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:11.925 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.642363+0000", 2020-06-12T02:39:11.925 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:11.925 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.926 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.926 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:11.926 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.642363+0000", 2020-06-12T02:39:11.926 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00069050800000000005 2020-06-12T02:39:11.926 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.927 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.927 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:11.927 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.643054+0000", 2020-06-12T02:39:11.927 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8220000000000007e-06 2020-06-12T02:39:11.927 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.927 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.928 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:11.928 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.643062+0000", 2020-06-12T02:39:11.928 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3742e-05 2020-06-12T02:39:11.928 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.928 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.929 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:11.929 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.643075+0000", 2020-06-12T02:39:11.929 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5910000000000001e-06 2020-06-12T02:39:11.929 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.929 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.929 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:11.930 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.643082+0000", 2020-06-12T02:39:11.935 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0064097879999999996 2020-06-12T02:39:11.935 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.935 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.935 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.935 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.649492+0000", 2020-06-12T02:39:11.936 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8151999999999994e-05 2020-06-12T02:39:11.936 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.936 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.936 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:11.936 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.649560+0000", 2020-06-12T02:39:11.936 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0020294940000000002 2020-06-12T02:39:11.937 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.937 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.937 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.937 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.651590+0000", 2020-06-12T02:39:11.937 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5242999999999997e-05 2020-06-12T02:39:11.938 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.938 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.938 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:11.938 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.651675+0000", 2020-06-12T02:39:11.938 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00069163799999999995 2020-06-12T02:39:11.939 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.939 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.939 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:11.939 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.652366+0000", 2020-06-12T02:39:11.939 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012711642 2020-06-12T02:39:11.940 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.940 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.940 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:11.940 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.665078+0000", 2020-06-12T02:39:11.940 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00017519200000000001 2020-06-12T02:39:11.940 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.941 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.941 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:11.941 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.665253+0000", 2020-06-12T02:39:11.941 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9867e-05 2020-06-12T02:39:11.941 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.942 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.942 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:11.942 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.665333+0000", 2020-06-12T02:39:11.942 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022969679 2020-06-12T02:39:11.942 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.943 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:11.943 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.943 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.943 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.943 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18287 3.ds0 3:b983fd28:::smithi10729407-1:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:11.943 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.975980+0000", 2020-06-12T02:39:11.944 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 29.780843762, 2020-06-12T02:39:11.944 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.023612415000000001, 2020-06-12T02:39:11.944 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:11.945 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:11.945 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:11.945 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:11.945 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:11.945 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18287 2020-06-12T02:39:11.945 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.946 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:11.946 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.946 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:11.946 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.975980+0000", 2020-06-12T02:39:11.946 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:11.947 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.947 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.947 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:11.947 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.975980+0000", 2020-06-12T02:39:11.947 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0175999999999999e-05 2020-06-12T02:39:11.948 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.948 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.948 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:11.948 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976070+0000", 2020-06-12T02:39:11.948 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7489999999999998e-06 2020-06-12T02:39:11.948 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.949 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.949 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:11.949 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976078+0000", 2020-06-12T02:39:11.949 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5308999999999999e-05 2020-06-12T02:39:11.949 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.950 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.950 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:11.950 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976093+0000", 2020-06-12T02:39:11.950 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7149999999999997e-06 2020-06-12T02:39:11.950 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.951 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.951 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:11.951 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976098+0000", 2020-06-12T02:39:11.951 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0067110759999999998 2020-06-12T02:39:11.952 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.952 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.952 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.952 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.982809+0000", 2020-06-12T02:39:11.952 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011082999999999999 2020-06-12T02:39:11.952 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.953 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.953 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:11.953 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.982920+0000", 2020-06-12T02:39:11.953 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00043570099999999998 2020-06-12T02:39:11.953 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.954 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.954 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:11.954 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.983356+0000", 2020-06-12T02:39:11.954 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00076300800000000002 2020-06-12T02:39:11.954 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.955 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.955 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:11.955 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.984119+0000", 2020-06-12T02:39:11.955 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01538405 2020-06-12T02:39:11.955 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.955 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.956 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:11.956 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.999503+0000", 2020-06-12T02:39:11.956 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9801000000000003e-05 2020-06-12T02:39:11.956 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.956 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.957 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:11.957 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.999593+0000", 2020-06-12T02:39:11.957 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.023612415000000001 2020-06-12T02:39:11.957 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.957 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:11.957 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.958 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.958 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.958 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18288 3.ds0 3:b983fd28:::smithi10729407-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:39:11.958 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.976151+0000", 2020-06-12T02:39:11.958 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 29.780672839000001, 2020-06-12T02:39:11.959 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025323907999999999, 2020-06-12T02:39:11.959 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:11.959 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:11.959 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:11.959 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:11.960 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:11.960 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18288 2020-06-12T02:39:11.960 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.960 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:11.960 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.960 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:11.961 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976151+0000", 2020-06-12T02:39:11.961 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:11.961 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.961 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.961 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:11.962 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976151+0000", 2020-06-12T02:39:11.962 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5615000000000002e-05 2020-06-12T02:39:11.962 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.962 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.962 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:11.962 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976207+0000", 2020-06-12T02:39:11.963 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6399999999999997e-06 2020-06-12T02:39:11.963 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.963 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.963 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:11.963 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976214+0000", 2020-06-12T02:39:11.964 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4823e-05 2020-06-12T02:39:11.964 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.964 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.964 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:11.964 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976229+0000", 2020-06-12T02:39:11.965 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1519999999999997e-06 2020-06-12T02:39:11.965 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.965 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.965 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:11.965 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.976235+0000", 2020-06-12T02:39:11.965 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0077693090000000003 2020-06-12T02:39:11.966 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.966 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.966 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.966 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.984005+0000", 2020-06-12T02:39:11.966 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000101221 2020-06-12T02:39:11.967 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.967 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.967 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:11.967 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.984106+0000", 2020-06-12T02:39:11.967 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016277713999999999 2020-06-12T02:39:11.967 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.968 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.968 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.968 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.000384+0000", 2020-06-12T02:39:11.968 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7938999999999999e-05 2020-06-12T02:39:11.968 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.969 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.969 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:11.969 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.000482+0000", 2020-06-12T02:39:11.969 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00099349500000000006 2020-06-12T02:39:11.969 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.969 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.970 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:11.970 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.001475+0000", 2020-06-12T02:39:11.970 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025323907999999999 2020-06-12T02:39:11.970 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.970 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:11.971 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.971 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.971 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.971 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18307 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [append 0~507904] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:11.971 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.056119+0000", 2020-06-12T02:39:11.971 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 29.700704955999999, 2020-06-12T02:39:11.972 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025879849, 2020-06-12T02:39:11.972 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:11.972 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:11.972 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:11.972 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:11.973 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:11.973 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18307 2020-06-12T02:39:11.973 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.973 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:11.973 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.974 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:11.974 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.056119+0000", 2020-06-12T02:39:11.974 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:11.974 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.974 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.974 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:11.975 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.056119+0000", 2020-06-12T02:39:11.975 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00052484099999999996 2020-06-12T02:39:11.975 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.975 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.975 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:11.976 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.056644+0000", 2020-06-12T02:39:11.976 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7149999999999997e-06 2020-06-12T02:39:11.976 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.976 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.976 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:11.977 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.056649+0000", 2020-06-12T02:39:11.977 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0046e-05 2020-06-12T02:39:11.977 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.977 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.977 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:11.977 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.056659+0000", 2020-06-12T02:39:11.978 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.848e-06 2020-06-12T02:39:11.978 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.978 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.978 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:11.978 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.056663+0000", 2020-06-12T02:39:11.979 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024537029999999998 2020-06-12T02:39:11.979 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.979 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.979 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.979 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.059117+0000", 2020-06-12T02:39:11.979 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000121261 2020-06-12T02:39:11.980 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.980 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.980 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:11.980 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.059238+0000", 2020-06-12T02:39:11.980 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001916484 2020-06-12T02:39:11.981 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.981 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.981 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.981 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.061155+0000", 2020-06-12T02:39:11.981 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9120000000000001e-05 2020-06-12T02:39:11.982 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.982 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.982 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:11.982 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.061234+0000", 2020-06-12T02:39:11.982 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00060383599999999996 2020-06-12T02:39:11.982 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.983 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.983 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:11.983 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.061838+0000", 2020-06-12T02:39:11.983 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010259862999999999 2020-06-12T02:39:11.983 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.984 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.984 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:11.984 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.072098+0000", 2020-06-12T02:39:11.984 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0098176419999999997 2020-06-12T02:39:11.984 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.984 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.985 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:11.985 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.081915+0000", 2020-06-12T02:39:11.985 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3490000000000002e-05 2020-06-12T02:39:11.985 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.985 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.986 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:11.986 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.081999+0000", 2020-06-12T02:39:11.986 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025879849 2020-06-12T02:39:11.986 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.986 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:11.987 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.987 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.987 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.987 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18311 3.fs0 3:fc7e0bf7:::smithi10729407-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:39:11.987 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.063458+0000", 2020-06-12T02:39:11.987 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 29.693366319999999, 2020-06-12T02:39:11.988 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.035342152000000002, 2020-06-12T02:39:11.988 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:11.988 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:11.988 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:11.988 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:11.989 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:11.989 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18311 2020-06-12T02:39:11.989 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.989 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:11.989 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.989 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:11.990 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.063458+0000", 2020-06-12T02:39:11.990 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:11.990 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.990 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.990 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:11.991 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.063458+0000", 2020-06-12T02:39:11.991 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3773000000000003e-05 2020-06-12T02:39:11.991 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.991 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.991 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:11.991 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.063501+0000", 2020-06-12T02:39:11.992 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0780000000000001e-06 2020-06-12T02:39:11.992 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.992 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.992 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:11.992 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.063508+0000", 2020-06-12T02:39:11.993 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3020000000000001e-05 2020-06-12T02:39:11.993 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.993 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.993 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:11.993 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.063521+0000", 2020-06-12T02:39:11.994 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4839999999999998e-06 2020-06-12T02:39:11.994 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.994 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.994 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:11.994 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.063525+0000", 2020-06-12T02:39:11.994 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0087769890000000007 2020-06-12T02:39:11.995 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.995 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.995 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.995 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.072302+0000", 2020-06-12T02:39:11.995 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.3121000000000005e-05 2020-06-12T02:39:11.996 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.996 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.996 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:11.996 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.072395+0000", 2020-06-12T02:39:11.996 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010158355000000001 2020-06-12T02:39:11.996 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.997 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.997 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:11.997 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.082554+0000", 2020-06-12T02:39:11.997 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5016000000000001e-05 2020-06-12T02:39:11.997 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.998 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.998 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:11.998 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.082629+0000", 2020-06-12T02:39:11.998 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016171316000000002 2020-06-12T02:39:11.998 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:11.998 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:11.999 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:11.999 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.098800+0000", 2020-06-12T02:39:11.999 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.035342152000000002 2020-06-12T02:39:11.999 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:11.999 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:12.000 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:12.000 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.000 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.000 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18313 3.fs0 3:f83c3dea:::smithi10729407-4:head [append 0~442368] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:12.000 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.070249+0000", 2020-06-12T02:39:12.000 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 29.686575043000001, 2020-06-12T02:39:12.001 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028853489, 2020-06-12T02:39:12.001 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:12.001 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:12.001 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:12.001 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:12.002 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:12.002 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18313 2020-06-12T02:39:12.002 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.002 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:12.002 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.003 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:12.003 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.070249+0000", 2020-06-12T02:39:12.003 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:12.003 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.003 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.004 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:12.004 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.070249+0000", 2020-06-12T02:39:12.004 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00046159400000000001 2020-06-12T02:39:12.004 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.004 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.004 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:12.005 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.070711+0000", 2020-06-12T02:39:12.005 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2539999999999992e-06 2020-06-12T02:39:12.005 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.005 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.005 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:12.006 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.070719+0000", 2020-06-12T02:39:12.006 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3297e-05 2020-06-12T02:39:12.006 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.006 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.006 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:12.006 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.070732+0000", 2020-06-12T02:39:12.007 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0559999999999999e-06 2020-06-12T02:39:12.007 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.007 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.007 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:12.007 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.070739+0000", 2020-06-12T02:39:12.008 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0052673319999999996 2020-06-12T02:39:12.008 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.008 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.008 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:12.008 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.076006+0000", 2020-06-12T02:39:12.008 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6925999999999997e-05 2020-06-12T02:39:12.009 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.009 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.009 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:12.009 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.076063+0000", 2020-06-12T02:39:12.009 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0076712209999999998 2020-06-12T02:39:12.010 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.010 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.010 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:12.010 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.083735+0000", 2020-06-12T02:39:12.010 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8089000000000002e-05 2020-06-12T02:39:12.011 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.011 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.011 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:12.011 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.083833+0000", 2020-06-12T02:39:12.011 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054620600000000004 2020-06-12T02:39:12.011 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.012 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.012 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:12.012 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.084379+0000", 2020-06-12T02:39:12.012 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013940945999999999 2020-06-12T02:39:12.012 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.013 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.013 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:12.013 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.098320+0000", 2020-06-12T02:39:12.013 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00072638599999999998 2020-06-12T02:39:12.013 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.014 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.014 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:12.014 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.099046+0000", 2020-06-12T02:39:12.014 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6181999999999998e-05 2020-06-12T02:39:12.015 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.015 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.015 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:12.015 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.099102+0000", 2020-06-12T02:39:12.015 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028853489 2020-06-12T02:39:12.016 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:12.016 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:12.016 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:12.016 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.016 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.016 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18314 3.fs0 3:f83c3dea:::smithi10729407-4:head [append 0~573440] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:12.017 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.074005+0000", 2020-06-12T02:39:12.017 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 29.682819024, 2020-06-12T02:39:12.017 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031442581999999997, 2020-06-12T02:39:12.017 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:12.017 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:12.018 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:12.018 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:12.018 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:12.018 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18314 2020-06-12T02:39:12.018 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.019 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:12.019 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.019 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:12.019 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.074005+0000", 2020-06-12T02:39:12.019 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:12.019 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.020 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.020 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:12.020 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.074005+0000", 2020-06-12T02:39:12.020 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00056762000000000002 2020-06-12T02:39:12.020 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.021 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.021 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:12.021 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.074573+0000", 2020-06-12T02:39:12.021 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5400000000000003e-06 2020-06-12T02:39:12.022 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.022 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.022 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:12.023 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.074578+0000", 2020-06-12T02:39:12.023 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1090000000000004e-06 2020-06-12T02:39:12.023 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.023 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.023 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:12.024 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.074587+0000", 2020-06-12T02:39:12.024 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4039999999999997e-06 2020-06-12T02:39:12.024 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.024 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.024 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:12.024 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.074592+0000", 2020-06-12T02:39:12.025 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001538031 2020-06-12T02:39:12.025 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.025 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.025 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:12.025 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.076130+0000", 2020-06-12T02:39:12.026 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5117000000000002e-05 2020-06-12T02:39:12.026 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.026 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.026 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:12.026 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.076175+0000", 2020-06-12T02:39:12.027 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011822262 2020-06-12T02:39:12.027 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.027 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.027 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:12.027 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.087997+0000", 2020-06-12T02:39:12.027 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000112318 2020-06-12T02:39:12.028 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.028 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.028 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:12.028 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.088109+0000", 2020-06-12T02:39:12.028 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00072351100000000003 2020-06-12T02:39:12.029 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.029 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.029 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:12.029 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.088833+0000", 2020-06-12T02:39:12.029 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010307874999999999 2020-06-12T02:39:12.029 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.030 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.030 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:12.030 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.099141+0000", 2020-06-12T02:39:12.030 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.006229605 2020-06-12T02:39:12.030 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.031 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.031 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:12.031 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.105370+0000", 2020-06-12T02:39:12.031 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7189999999999998e-05 2020-06-12T02:39:12.031 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.032 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.032 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:12.032 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.105448+0000", 2020-06-12T02:39:12.032 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031442581999999997 2020-06-12T02:39:12.032 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:12.032 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:12.033 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:12.033 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.033 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.033 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18315 3.fs0 3:f83c3dea:::smithi10729407-4:head [append 0~589824] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:12.033 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.077724+0000", 2020-06-12T02:39:12.034 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 29.679100165000001, 2020-06-12T02:39:12.034 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.027939051999999999, 2020-06-12T02:39:12.034 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:12.034 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:12.034 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:12.035 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:12.035 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:12.035 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18315 2020-06-12T02:39:12.036 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.036 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:12.036 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.036 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:12.036 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.077724+0000", 2020-06-12T02:39:12.037 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:12.037 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.037 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.037 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:12.037 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.077724+0000", 2020-06-12T02:39:12.038 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00093652299999999998 2020-06-12T02:39:12.038 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.038 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.038 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:12.038 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078660+0000", 2020-06-12T02:39:12.038 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1389999999999995e-06 2020-06-12T02:39:12.039 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.039 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.039 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:12.039 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078669+0000", 2020-06-12T02:39:12.039 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3691e-05 2020-06-12T02:39:12.040 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.040 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.040 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:12.040 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078682+0000", 2020-06-12T02:39:12.040 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0100000000000001e-06 2020-06-12T02:39:12.040 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.041 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.041 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:12.041 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078688+0000", 2020-06-12T02:39:12.041 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1209999999999999e-05 2020-06-12T02:39:12.041 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.042 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.042 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:12.042 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078739+0000", 2020-06-12T02:39:12.042 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000102735 2020-06-12T02:39:12.042 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.042 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.043 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:12.043 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078842+0000", 2020-06-12T02:39:12.043 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01321815 2020-06-12T02:39:12.043 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.043 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.044 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:12.044 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.092060+0000", 2020-06-12T02:39:12.044 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00038608499999999999 2020-06-12T02:39:12.045 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.045 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.045 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:12.045 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.092446+0000", 2020-06-12T02:39:12.045 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00056703299999999999 2020-06-12T02:39:12.046 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.046 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.046 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:12.046 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.093013+0000", 2020-06-12T02:39:12.046 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0061289530000000004 2020-06-12T02:39:12.047 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.047 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.047 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:12.047 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.099142+0000", 2020-06-12T02:39:12.047 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0064547700000000003 2020-06-12T02:39:12.048 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.048 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.048 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:12.048 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.105597+0000", 2020-06-12T02:39:12.048 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5753000000000006e-05 2020-06-12T02:39:12.048 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.049 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.049 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:12.049 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.105663+0000", 2020-06-12T02:39:12.049 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.027939051999999999 2020-06-12T02:39:12.049 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:12.050 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:12.050 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:12.050 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.050 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.050 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18316 3.fs0 3:f83c3dea:::smithi10729407-4:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:12.051 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.078752+0000", 2020-06-12T02:39:12.051 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 29.678072393000001, 2020-06-12T02:39:12.051 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.029108709999999999, 2020-06-12T02:39:12.051 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:12.051 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:12.052 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:12.052 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:12.052 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:12.053 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18316 2020-06-12T02:39:12.053 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.053 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:12.053 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.053 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:12.054 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078752+0000", 2020-06-12T02:39:12.054 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:12.054 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.054 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.054 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:12.054 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078752+0000", 2020-06-12T02:39:12.055 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8967000000000004e-05 2020-06-12T02:39:12.055 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.055 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.055 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:12.055 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078841+0000", 2020-06-12T02:39:12.056 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3309999999999998e-06 2020-06-12T02:39:12.056 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.056 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.056 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:12.056 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078848+0000", 2020-06-12T02:39:12.056 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.291e-05 2020-06-12T02:39:12.057 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.057 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.057 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:12.057 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078861+0000", 2020-06-12T02:39:12.057 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2800000000000003e-06 2020-06-12T02:39:12.058 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.058 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.058 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:12.058 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078866+0000", 2020-06-12T02:39:12.058 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8543e-05 2020-06-12T02:39:12.059 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.059 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.059 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:12.059 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078905+0000", 2020-06-12T02:39:12.059 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9198999999999998e-05 2020-06-12T02:39:12.059 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.060 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.060 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:12.060 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078994+0000", 2020-06-12T02:39:12.060 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.01781437 2020-06-12T02:39:12.060 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.061 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.061 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:12.061 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.096808+0000", 2020-06-12T02:39:12.061 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000115975 2020-06-12T02:39:12.061 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.062 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.062 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:12.062 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.096924+0000", 2020-06-12T02:39:12.062 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00049986300000000002 2020-06-12T02:39:12.062 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.062 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.063 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:12.063 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.097424+0000", 2020-06-12T02:39:12.063 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0017191089999999999 2020-06-12T02:39:12.063 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.063 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.064 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:12.064 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.099143+0000", 2020-06-12T02:39:12.064 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0086177909999999996 2020-06-12T02:39:12.064 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.064 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.064 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:12.065 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.107761+0000", 2020-06-12T02:39:12.065 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9371999999999997e-05 2020-06-12T02:39:12.065 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.065 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.065 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:12.066 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.107860+0000", 2020-06-12T02:39:12.066 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.029108709999999999 2020-06-12T02:39:12.066 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:12.066 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:12.066 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:12.067 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.067 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.067 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18317 3.fs0 3:f83c3dea:::smithi10729407-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:39:12.067 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.078920+0000", 2020-06-12T02:39:12.067 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 29.677904423000001, 2020-06-12T02:39:12.067 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031356964000000001, 2020-06-12T02:39:12.068 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:12.068 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:12.068 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:12.068 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:12.068 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:12.069 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18317 2020-06-12T02:39:12.069 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.069 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:12.069 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.069 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:12.069 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078920+0000", 2020-06-12T02:39:12.070 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:12.070 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.070 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.070 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:12.070 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078920+0000", 2020-06-12T02:39:12.071 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6454000000000001e-05 2020-06-12T02:39:12.071 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.071 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.071 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:12.071 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078946+0000", 2020-06-12T02:39:12.071 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8160000000000004e-06 2020-06-12T02:39:12.072 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.072 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.072 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:12.072 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078950+0000", 2020-06-12T02:39:12.072 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6090000000000003e-06 2020-06-12T02:39:12.073 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.073 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.073 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:12.073 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078957+0000", 2020-06-12T02:39:12.073 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8380000000000002e-06 2020-06-12T02:39:12.073 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.074 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.074 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:12.074 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.078960+0000", 2020-06-12T02:39:12.074 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8400000000000003e-05 2020-06-12T02:39:12.074 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.075 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.075 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:12.075 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.079019+0000", 2020-06-12T02:39:12.075 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6750999999999998e-05 2020-06-12T02:39:12.075 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.076 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.076 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:12.076 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.079085+0000", 2020-06-12T02:39:12.076 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019272036999999999 2020-06-12T02:39:12.076 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.076 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.077 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:12.077 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.098357+0000", 2020-06-12T02:39:12.077 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4322999999999998e-05 2020-06-12T02:39:12.077 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.077 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.078 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:12.078 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.098452+0000", 2020-06-12T02:39:12.078 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010578025 2020-06-12T02:39:12.078 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.078 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.078 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:12.079 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.109030+0000", 2020-06-12T02:39:12.079 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010395099999999999 2020-06-12T02:39:12.079 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.079 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.079 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:12.080 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.109134+0000", 2020-06-12T02:39:12.080 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00114276 2020-06-12T02:39:12.080 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:12.080 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:12.080 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:12.080 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.110277+0000", 2020-06-12T02:39:12.081 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.031356964000000001 2020-06-12T02:39:12.081 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:12.081 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:12.081 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:12.081 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:12.082 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:12.082 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:39:12.093 INFO:tasks.rados.rados.0.smithi107.stdout:10596: writing smithi10729407-11 from 3792896 to 4448256 tid 1 2020-06-12T02:39:12.094 INFO:tasks.rados.rados.0.smithi107.stdout:10596: writing smithi10729407-11 from 4448256 to 4956160 tid 2 2020-06-12T02:39:12.094 INFO:tasks.rados.rados.0.smithi107.stdout:10596: writing smithi10729407-11 from 4956160 to 5480448 tid 3 2020-06-12T02:39:12.094 INFO:tasks.rados.rados.0.smithi107.stdout:10596: writing smithi10729407-11 from 5480448 to 5906432 tid 4 2020-06-12T02:39:12.094 INFO:tasks.rados.rados.0.smithi107.stdout:10596: writing smithi10729407-11 from 5906432 to 6053888 tid 5 2020-06-12T02:39:12.095 INFO:tasks.rados.rados.0.smithi107.stdout:10597: delete oid 8 current snap is 0 2020-06-12T02:39:12.095 INFO:tasks.rados.rados.0.smithi107.stdout:10592: finishing write tid 2 to smithi10729407-14 2020-06-12T02:39:12.095 INFO:tasks.rados.rados.0.smithi107.stdout:10592: finishing write tid 3 to smithi10729407-14 2020-06-12T02:39:12.095 INFO:tasks.rados.rados.0.smithi107.stdout:10592: finishing write tid 4 to smithi10729407-14 2020-06-12T02:39:12.095 INFO:tasks.rados.rados.0.smithi107.stdout:10592: finishing write tid 6 to smithi10729407-14 2020-06-12T02:39:12.096 INFO:tasks.rados.rados.0.smithi107.stdout:10592: finishing write tid 7 to smithi10729407-14 2020-06-12T02:39:12.096 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1701 (ObjNum 10591 snap 0 seq_num 10591) dirty exists 2020-06-12T02:39:12.096 INFO:tasks.rados.rados.0.smithi107.stdout:10592: left oid 14 (ObjNum 10591 snap 0 seq_num 10591) 2020-06-12T02:39:12.096 INFO:tasks.rados.rados.0.smithi107.stdout:10595: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:12.096 INFO:tasks.rados.rados.0.smithi107.stdout:10595: finishing write tid 2 to smithi10729407-15 2020-06-12T02:39:12.097 INFO:tasks.rados.rados.0.smithi107.stdout:10595: finishing write tid 3 to smithi10729407-15 2020-06-12T02:39:12.097 INFO:tasks.rados.rados.0.smithi107.stdout:10595: finishing write tid 4 to smithi10729407-15 2020-06-12T02:39:12.097 INFO:tasks.rados.rados.0.smithi107.stdout:10595: finishing write tid 6 to smithi10729407-15 2020-06-12T02:39:12.097 INFO:tasks.rados.rados.0.smithi107.stdout:10595: finishing write tid 7 to smithi10729407-15 2020-06-12T02:39:12.097 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1976 (ObjNum 10594 snap 0 seq_num 10594) dirty exists 2020-06-12T02:39:12.097 INFO:tasks.rados.rados.0.smithi107.stdout:10595: left oid 15 (ObjNum 10594 snap 0 seq_num 10594) 2020-06-12T02:39:12.098 INFO:tasks.rados.rados.0.smithi107.stdout:10592: done (3 left) 2020-06-12T02:39:12.098 INFO:tasks.rados.rados.0.smithi107.stdout:10595: done (2 left) 2020-06-12T02:39:12.098 INFO:tasks.rados.rados.0.smithi107.stdout:10597: done (1 left) 2020-06-12T02:39:12.098 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:39:12.098 INFO:tasks.rados.rados.0.smithi107.stdout:10598: seq_num 10597 ranges {4333568=688128,5021696=671744,5693440=540672,6234112=393216} 2020-06-12T02:39:12.099 INFO:tasks.rados.rados.0.smithi107.stdout:10598: writing smithi10729407-17 from 4333568 to 5021696 tid 1 2020-06-12T02:39:12.099 INFO:tasks.rados.rados.0.smithi107.stdout:10598: writing smithi10729407-17 from 5021696 to 5693440 tid 2 2020-06-12T02:39:12.099 INFO:tasks.rados.rados.0.smithi107.stdout:10598: writing smithi10729407-17 from 5693440 to 6234112 tid 3 2020-06-12T02:39:12.099 INFO:tasks.rados.rados.0.smithi107.stdout:10598: writing smithi10729407-17 from 6234112 to 6627328 tid 4 2020-06-12T02:39:12.099 INFO:tasks.rados.rados.0.smithi107.stdout:10599: delete oid 1 current snap is 0 2020-06-12T02:39:12.100 INFO:tasks.rados.rados.0.smithi107.stdout:10596: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:12.100 INFO:tasks.rados.rados.0.smithi107.stdout:10596: finishing write tid 2 to smithi10729407-11 2020-06-12T02:39:12.100 INFO:tasks.rados.rados.0.smithi107.stdout:10596: finishing write tid 3 to smithi10729407-11 2020-06-12T02:39:12.100 INFO:tasks.rados.rados.0.smithi107.stdout:10596: finishing write tid 4 to smithi10729407-11 2020-06-12T02:39:12.100 INFO:tasks.rados.rados.0.smithi107.stdout:10596: finishing write tid 5 to smithi10729407-11 2020-06-12T02:39:12.100 INFO:tasks.rados.rados.0.smithi107.stdout:10596: finishing write tid 7 to smithi10729407-11 2020-06-12T02:39:12.101 INFO:tasks.rados.rados.0.smithi107.stdout:10596: finishing write tid 8 to smithi10729407-11 2020-06-12T02:39:12.101 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1725 (ObjNum 10595 snap 0 seq_num 10595) dirty exists 2020-06-12T02:39:12.101 INFO:tasks.rados.rados.0.smithi107.stdout:10596: left oid 11 (ObjNum 10595 snap 0 seq_num 10595) 2020-06-12T02:39:12.101 INFO:tasks.rados.rados.0.smithi107.stdout:10598: finishing write tid 1 to smithi10729407-17 2020-06-12T02:39:12.101 INFO:tasks.rados.rados.0.smithi107.stdout:10596: done (2 left) 2020-06-12T02:39:12.102 INFO:tasks.rados.rados.0.smithi107.stdout:10599: done (1 left) 2020-06-12T02:39:12.102 INFO:tasks.rados.rados.0.smithi107.stdout:10600: delete oid 13 current snap is 0 2020-06-12T02:39:12.102 INFO:tasks.rados.rados.0.smithi107.stdout:10600: done (1 left) 2020-06-12T02:39:12.102 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:39:12.102 INFO:tasks.rados.rados.0.smithi107.stdout:10601: seq_num 10600 ranges {0=737280,737280=778240,1515520=712704,2228224=114688} 2020-06-12T02:39:12.103 INFO:tasks.rados.rados.0.smithi107.stdout:10601: writing smithi10729407-10 from 0 to 737280 tid 1 2020-06-12T02:39:12.103 INFO:tasks.rados.rados.0.smithi107.stdout:10601: writing smithi10729407-10 from 737280 to 1515520 tid 2 2020-06-12T02:39:12.103 INFO:tasks.rados.rados.0.smithi107.stdout:10601: writing smithi10729407-10 from 1515520 to 2228224 tid 3 2020-06-12T02:39:12.103 INFO:tasks.rados.rados.0.smithi107.stdout:10601: writing smithi10729407-10 from 2228224 to 2342912 tid 4 2020-06-12T02:39:12.103 INFO:tasks.rados.rados.0.smithi107.stdout:10602: delete oid 8 current snap is 0 2020-06-12T02:39:12.103 INFO:tasks.rados.rados.0.smithi107.stdout:10598: finishing write tid 2 to smithi10729407-17 2020-06-12T02:39:12.104 INFO:tasks.rados.rados.0.smithi107.stdout:10598: finishing write tid 3 to smithi10729407-17 2020-06-12T02:39:12.104 INFO:tasks.rados.rados.0.smithi107.stdout:10598: finishing write tid 4 to smithi10729407-17 2020-06-12T02:39:12.104 INFO:tasks.rados.rados.0.smithi107.stdout:10598: finishing write tid 6 to smithi10729407-17 2020-06-12T02:39:12.104 INFO:tasks.rados.rados.0.smithi107.stdout:10598: finishing write tid 7 to smithi10729407-17 2020-06-12T02:39:12.104 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1982 (ObjNum 10597 snap 0 seq_num 10597) dirty exists 2020-06-12T02:39:12.105 INFO:tasks.rados.rados.0.smithi107.stdout:10598: left oid 17 (ObjNum 10597 snap 0 seq_num 10597) 2020-06-12T02:39:12.105 INFO:tasks.rados.rados.0.smithi107.stdout:10598: done (2 left) 2020-06-12T02:39:12.105 INFO:tasks.rados.rados.0.smithi107.stdout:10602: done (1 left) 2020-06-12T02:39:12.105 INFO:tasks.rados.rados.0.smithi107.stdout:10603: delete oid 12 current snap is 0 2020-06-12T02:39:12.105 INFO:tasks.rados.rados.0.smithi107.stdout:10601: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:12.106 INFO:tasks.rados.rados.0.smithi107.stdout:10601: finishing write tid 2 to smithi10729407-10 2020-06-12T02:39:12.106 INFO:tasks.rados.rados.0.smithi107.stdout:10601: finishing write tid 3 to smithi10729407-10 2020-06-12T02:39:12.106 INFO:tasks.rados.rados.0.smithi107.stdout:10601: finishing write tid 4 to smithi10729407-10 2020-06-12T02:39:12.106 INFO:tasks.rados.rados.0.smithi107.stdout:10603: done (1 left) 2020-06-12T02:39:12.106 INFO:tasks.rados.rados.0.smithi107.stdout:10604: delete oid 6 current snap is 0 2020-06-12T02:39:12.106 INFO:tasks.rados.rados.0.smithi107.stdout:10601: finishing write tid 6 to smithi10729407-10 2020-06-12T02:39:12.107 INFO:tasks.rados.rados.0.smithi107.stdout:10604: done (1 left) 2020-06-12T02:39:12.107 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:39:12.107 INFO:tasks.rados.rados.0.smithi107.stdout:10605: seq_num 10604 ranges {0=802816,802816=712704,1515520=696320,2211840=196608} 2020-06-12T02:39:12.107 INFO:tasks.rados.rados.0.smithi107.stdout:10605: writing smithi10729407-18 from 0 to 802816 tid 1 2020-06-12T02:39:12.107 INFO:tasks.rados.rados.0.smithi107.stdout:10605: writing smithi10729407-18 from 802816 to 1515520 tid 2 2020-06-12T02:39:12.108 INFO:tasks.rados.rados.0.smithi107.stdout:10605: writing smithi10729407-18 from 1515520 to 2211840 tid 3 2020-06-12T02:39:12.108 INFO:tasks.rados.rados.0.smithi107.stdout:10605: writing smithi10729407-18 from 2211840 to 2408448 tid 4 2020-06-12T02:39:12.108 INFO:tasks.rados.rados.0.smithi107.stdout:10606: delete oid 15 current snap is 0 2020-06-12T02:39:12.108 INFO:tasks.rados.rados.0.smithi107.stdout:10601: finishing write tid 7 to smithi10729407-10 2020-06-12T02:39:12.108 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 772 (ObjNum 10600 snap 0 seq_num 10600) dirty exists 2020-06-12T02:39:12.108 INFO:tasks.rados.rados.0.smithi107.stdout:10601: left oid 10 (ObjNum 10600 snap 0 seq_num 10600) 2020-06-12T02:39:12.109 INFO:tasks.rados.rados.0.smithi107.stdout:10605: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:12.109 INFO:tasks.rados.rados.0.smithi107.stdout:10601: done (2 left) 2020-06-12T02:39:12.109 INFO:tasks.rados.rados.0.smithi107.stdout:10606: done (1 left) 2020-06-12T02:39:12.109 INFO:tasks.rados.rados.0.smithi107.stdout:10607: delete oid 17 current snap is 0 2020-06-12T02:39:12.109 INFO:tasks.rados.rados.0.smithi107.stdout:10607: done (1 left) 2020-06-12T02:39:12.110 INFO:tasks.rados.rados.0.smithi107.stdout:10608: delete oid 12 current snap is 0 2020-06-12T02:39:12.110 INFO:tasks.rados.rados.0.smithi107.stdout:10605: finishing write tid 2 to smithi10729407-18 2020-06-12T02:39:12.110 INFO:tasks.rados.rados.0.smithi107.stdout:10605: finishing write tid 3 to smithi10729407-18 2020-06-12T02:39:12.110 INFO:tasks.rados.rados.0.smithi107.stdout:10608: done (1 left) 2020-06-12T02:39:12.110 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:39:12.111 INFO:tasks.rados.rados.0.smithi107.stdout:10609: seq_num 10608 ranges {2596864=65536} 2020-06-12T02:39:12.111 INFO:tasks.rados.rados.0.smithi107.stdout:10609: writing smithi10729407-9 from 2596864 to 2662400 tid 1 2020-06-12T02:39:12.111 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:39:12.111 INFO:tasks.rados.rados.0.smithi107.stdout:10610: seq_num 10609 ranges {3760128=81920} 2020-06-12T02:39:12.111 INFO:tasks.rados.rados.0.smithi107.stdout:10610: writing smithi10729407-5 from 3760128 to 3842048 tid 1 2020-06-12T02:39:12.111 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:39:12.112 INFO:tasks.rados.rados.0.smithi107.stdout:10611: seq_num 10610 ranges {6053888=98304} 2020-06-12T02:39:12.112 INFO:tasks.rados.rados.0.smithi107.stdout:10611: writing smithi10729407-11 from 6053888 to 6152192 tid 1 2020-06-12T02:39:12.112 INFO:tasks.rados.rados.0.smithi107.stdout:10612: delete oid 6 current snap is 0 2020-06-12T02:39:12.112 INFO:tasks.rados.rados.0.smithi107.stdout:10605: finishing write tid 4 to smithi10729407-18 2020-06-12T02:39:12.112 INFO:tasks.rados.rados.0.smithi107.stdout:10605: finishing write tid 6 to smithi10729407-18 2020-06-12T02:39:12.113 INFO:tasks.rados.rados.0.smithi107.stdout:10605: finishing write tid 7 to smithi10729407-18 2020-06-12T02:39:12.113 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1841 (ObjNum 10604 snap 0 seq_num 10604) dirty exists 2020-06-12T02:39:12.113 INFO:tasks.rados.rados.0.smithi107.stdout:10605: left oid 18 (ObjNum 10604 snap 0 seq_num 10604) 2020-06-12T02:39:12.113 INFO:tasks.rados.rados.0.smithi107.stdout:10609: finishing write tid 1 to smithi10729407-9 2020-06-12T02:39:12.113 INFO:tasks.rados.rados.0.smithi107.stdout:10609: finishing write tid 3 to smithi10729407-9 2020-06-12T02:39:12.113 INFO:tasks.rados.rados.0.smithi107.stdout:10609: finishing write tid 4 to smithi10729407-9 2020-06-12T02:39:12.114 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1827 (ObjNum 10608 snap 0 seq_num 10608) dirty exists 2020-06-12T02:39:12.114 INFO:tasks.rados.rados.0.smithi107.stdout:10609: left oid 9 (ObjNum 10608 snap 0 seq_num 10608) 2020-06-12T02:39:12.114 INFO:tasks.rados.rados.0.smithi107.stdout:10610: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:12.114 INFO:tasks.rados.rados.0.smithi107.stdout:10610: finishing write tid 3 to smithi10729407-5 2020-06-12T02:39:12.114 INFO:tasks.rados.rados.0.smithi107.stdout:10610: finishing write tid 4 to smithi10729407-5 2020-06-12T02:39:12.115 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 1012 (ObjNum 10609 snap 0 seq_num 10609) dirty exists 2020-06-12T02:39:12.115 INFO:tasks.rados.rados.0.smithi107.stdout:10610: left oid 5 (ObjNum 10609 snap 0 seq_num 10609) 2020-06-12T02:39:12.115 INFO:tasks.rados.rados.0.smithi107.stdout:10611: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:12.115 INFO:tasks.rados.rados.0.smithi107.stdout:10605: done (4 left) 2020-06-12T02:39:12.116 INFO:tasks.rados.rados.0.smithi107.stdout:10609: done (3 left) 2020-06-12T02:39:12.116 INFO:tasks.rados.rados.0.smithi107.stdout:10610: done (2 left) 2020-06-12T02:39:12.116 INFO:tasks.rados.rados.0.smithi107.stdout:10612: done (1 left) 2020-06-12T02:39:12.116 INFO:tasks.rados.rados.0.smithi107.stdout:10613: delete oid 13 current snap is 0 2020-06-12T02:39:12.116 INFO:tasks.rados.rados.0.smithi107.stdout:10611: finishing write tid 3 to smithi10729407-11 2020-06-12T02:39:12.117 INFO:tasks.rados.rados.0.smithi107.stdout:10611: finishing write tid 4 to smithi10729407-11 2020-06-12T02:39:12.117 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1727 (ObjNum 10610 snap 0 seq_num 10610) dirty exists 2020-06-12T02:39:12.117 INFO:tasks.rados.rados.0.smithi107.stdout:10611: left oid 11 (ObjNum 10610 snap 0 seq_num 10610) 2020-06-12T02:39:12.117 INFO:tasks.rados.rados.0.smithi107.stdout:10611: done (1 left) 2020-06-12T02:39:12.117 INFO:tasks.rados.rados.0.smithi107.stdout:10613: done (0 left) 2020-06-12T02:39:12.118 INFO:tasks.rados.rados.0.smithi107.stdout:10614: delete oid 16 current snap is 0 2020-06-12T02:39:12.118 INFO:tasks.rados.rados.0.smithi107.stdout:10614: done (0 left) 2020-06-12T02:39:12.118 INFO:tasks.rados.rados.0.smithi107.stdout:10615: delete oid 17 current snap is 0 2020-06-12T02:39:12.118 INFO:tasks.rados.rados.0.smithi107.stdout:10615: done (0 left) 2020-06-12T02:39:12.118 INFO:tasks.rados.rados.0.smithi107.stdout:10616: delete oid 17 current snap is 0 2020-06-12T02:39:12.118 INFO:tasks.rados.rados.0.smithi107.stdout:10616: done (0 left) 2020-06-12T02:39:12.119 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:39:12.119 INFO:tasks.rados.rados.0.smithi107.stdout:10617: seq_num 10616 ranges {0=204800} 2020-06-12T02:39:12.119 INFO:tasks.rados.rados.0.smithi107.stdout:10617: writing smithi10729407-12 from 0 to 204800 tid 1 2020-06-12T02:39:12.119 INFO:tasks.rados.rados.0.smithi107.stdout:10618: delete oid 7 current snap is 0 2020-06-12T02:39:12.119 INFO:tasks.rados.rados.0.smithi107.stdout:10618: done (1 left) 2020-06-12T02:39:12.120 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:39:12.120 INFO:tasks.rados.rados.0.smithi107.stdout:10619: seq_num 10618 ranges {6152192=237568} 2020-06-12T02:39:12.120 INFO:tasks.rados.rados.0.smithi107.stdout:10619: writing smithi10729407-11 from 6152192 to 6389760 tid 1 2020-06-12T02:39:12.120 INFO:tasks.rados.rados.0.smithi107.stdout:10620: delete oid 14 current snap is 0 2020-06-12T02:39:12.120 INFO:tasks.rados.rados.0.smithi107.stdout:10617: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:12.120 INFO:tasks.rados.rados.0.smithi107.stdout:10617: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:12.121 INFO:tasks.rados.rados.0.smithi107.stdout:10617: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:12.121 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1953 (ObjNum 10616 snap 0 seq_num 10616) dirty exists 2020-06-12T02:39:12.121 INFO:tasks.rados.rados.0.smithi107.stdout:10617: left oid 12 (ObjNum 10616 snap 0 seq_num 10616) 2020-06-12T02:39:12.121 INFO:tasks.rados.rados.0.smithi107.stdout:10617: done (2 left) 2020-06-12T02:39:12.121 INFO:tasks.rados.rados.0.smithi107.stdout:10620: done (1 left) 2020-06-12T02:39:12.122 INFO:tasks.rados.rados.0.smithi107.stdout:10621: delete oid 8 current snap is 0 2020-06-12T02:39:12.122 INFO:tasks.rados.rados.0.smithi107.stdout:10619: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:12.122 INFO:tasks.rados.rados.0.smithi107.stdout:10619: finishing write tid 3 to smithi10729407-11 2020-06-12T02:39:12.122 INFO:tasks.rados.rados.0.smithi107.stdout:10619: finishing write tid 4 to smithi10729407-11 2020-06-12T02:39:12.122 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1729 (ObjNum 10618 snap 0 seq_num 10618) dirty exists 2020-06-12T02:39:12.122 INFO:tasks.rados.rados.0.smithi107.stdout:10619: left oid 11 (ObjNum 10618 snap 0 seq_num 10618) 2020-06-12T02:39:12.123 INFO:tasks.rados.rados.0.smithi107.stdout:10619: done (1 left) 2020-06-12T02:39:12.123 INFO:tasks.rados.rados.0.smithi107.stdout:10621: done (0 left) 2020-06-12T02:39:12.123 INFO:tasks.rados.rados.0.smithi107.stdout:10622: delete oid 7 current snap is 0 2020-06-12T02:39:12.123 INFO:tasks.rados.rados.0.smithi107.stdout:10622: done (0 left) 2020-06-12T02:39:12.123 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:39:12.124 INFO:tasks.rados.rados.0.smithi107.stdout:10623: seq_num 10622 ranges {0=303104} 2020-06-12T02:39:12.124 INFO:tasks.rados.rados.0.smithi107.stdout:10623: writing smithi10729407-17 from 0 to 303104 tid 1 2020-06-12T02:39:12.124 INFO:tasks.rados.rados.0.smithi107.stdout:10624: delete oid 2 current snap is 0 2020-06-12T02:39:12.124 INFO:tasks.rados.rados.0.smithi107.stdout:10624: done (1 left) 2020-06-12T02:39:12.124 INFO:tasks.rados.rados.0.smithi107.stdout:10625: delete oid 20 current snap is 0 2020-06-12T02:39:12.124 INFO:tasks.rados.rados.0.smithi107.stdout:10625: done (1 left) 2020-06-12T02:39:12.125 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:39:12.125 INFO:tasks.rados.rados.0.smithi107.stdout:10626: seq_num 10625 ranges {2342912=352256} 2020-06-12T02:39:12.125 INFO:tasks.rados.rados.0.smithi107.stdout:10626: writing smithi10729407-10 from 2342912 to 2695168 tid 1 2020-06-12T02:39:12.125 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:39:12.125 INFO:tasks.rados.rados.0.smithi107.stdout:10627: seq_num 10626 ranges {0=368640} 2020-06-12T02:39:12.126 INFO:tasks.rados.rados.0.smithi107.stdout:10627: writing smithi10729407-8 from 0 to 368640 tid 1 2020-06-12T02:39:12.126 INFO:tasks.rados.rados.0.smithi107.stdout:10628: delete oid 18 current snap is 0 2020-06-12T02:39:12.126 INFO:tasks.rados.rados.0.smithi107.stdout:10623: finishing write tid 1 to smithi10729407-17 2020-06-12T02:39:12.126 INFO:tasks.rados.rados.0.smithi107.stdout:10623: finishing write tid 3 to smithi10729407-17 2020-06-12T02:39:12.126 INFO:tasks.rados.rados.0.smithi107.stdout:10623: finishing write tid 4 to smithi10729407-17 2020-06-12T02:39:12.127 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 1987 (ObjNum 10622 snap 0 seq_num 10622) dirty exists 2020-06-12T02:39:12.127 INFO:tasks.rados.rados.0.smithi107.stdout:10623: left oid 17 (ObjNum 10622 snap 0 seq_num 10622) 2020-06-12T02:39:12.127 INFO:tasks.rados.rados.0.smithi107.stdout:10626: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:12.127 INFO:tasks.rados.rados.0.smithi107.stdout:10623: done (3 left) 2020-06-12T02:39:12.127 INFO:tasks.rados.rados.0.smithi107.stdout:10628: done (2 left) 2020-06-12T02:39:12.127 INFO:tasks.rados.rados.0.smithi107.stdout:10629: delete oid 6 current snap is 0 2020-06-12T02:39:12.128 INFO:tasks.rados.rados.0.smithi107.stdout:10626: finishing write tid 3 to smithi10729407-10 2020-06-12T02:39:12.128 INFO:tasks.rados.rados.0.smithi107.stdout:10627: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:12.128 INFO:tasks.rados.rados.0.smithi107.stdout:10626: finishing write tid 4 to smithi10729407-10 2020-06-12T02:39:12.128 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 774 (ObjNum 10625 snap 0 seq_num 10625) dirty exists 2020-06-12T02:39:12.128 INFO:tasks.rados.rados.0.smithi107.stdout:10626: left oid 10 (ObjNum 10625 snap 0 seq_num 10625) 2020-06-12T02:39:12.129 INFO:tasks.rados.rados.0.smithi107.stdout:10626: done (2 left) 2020-06-12T02:39:12.129 INFO:tasks.rados.rados.0.smithi107.stdout:10629: done (1 left) 2020-06-12T02:39:12.129 INFO:tasks.rados.rados.0.smithi107.stdout:10630: delete oid 19 current snap is 0 2020-06-12T02:39:12.129 INFO:tasks.rados.rados.0.smithi107.stdout:10627: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:12.129 INFO:tasks.rados.rados.0.smithi107.stdout:10627: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:12.129 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1982 (ObjNum 10626 snap 0 seq_num 10626) dirty exists 2020-06-12T02:39:12.130 INFO:tasks.rados.rados.0.smithi107.stdout:10627: left oid 8 (ObjNum 10626 snap 0 seq_num 10626) 2020-06-12T02:39:12.130 INFO:tasks.rados.rados.0.smithi107.stdout:10627: done (1 left) 2020-06-12T02:39:12.130 INFO:tasks.rados.rados.0.smithi107.stdout:10630: done (0 left) 2020-06-12T02:39:12.130 INFO:tasks.rados.rados.0.smithi107.stdout:10631: delete oid 8 current snap is 0 2020-06-12T02:39:12.130 INFO:tasks.rados.rados.0.smithi107.stdout:10631: done (0 left) 2020-06-12T02:39:12.131 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:39:12.131 INFO:tasks.rados.rados.0.smithi107.stdout:10632: seq_num 10631 ranges {0=450560} 2020-06-12T02:39:12.131 INFO:tasks.rados.rados.0.smithi107.stdout:10632: writing smithi10729407-19 from 0 to 450560 tid 1 2020-06-12T02:39:12.131 INFO:tasks.rados.rados.0.smithi107.stdout:10633: delete oid 12 current snap is 0 2020-06-12T02:39:12.131 INFO:tasks.rados.rados.0.smithi107.stdout:10632: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:12.131 INFO:tasks.rados.rados.0.smithi107.stdout:10632: finishing write tid 3 to smithi10729407-19 2020-06-12T02:39:12.132 INFO:tasks.rados.rados.0.smithi107.stdout:10633: done (1 left) 2020-06-12T02:39:12.132 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:39:12.132 INFO:tasks.rados.rados.0.smithi107.stdout:10634: seq_num 10633 ranges {0=491520} 2020-06-12T02:39:12.132 INFO:tasks.rados.rados.0.smithi107.stdout:10634: writing smithi10729407-16 from 0 to 491520 tid 1 2020-06-12T02:39:12.132 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:39:12.133 INFO:tasks.rados.rados.0.smithi107.stdout:10635: seq_num 10634 ranges {0=507904} 2020-06-12T02:39:12.133 INFO:tasks.rados.rados.0.smithi107.stdout:10635: writing smithi10729407-14 from 0 to 507904 tid 1 2020-06-12T02:39:12.133 INFO:tasks.rados.rados.0.smithi107.stdout:10636: delete oid 7 current snap is 0 2020-06-12T02:39:12.133 INFO:tasks.rados.rados.0.smithi107.stdout:10632: finishing write tid 4 to smithi10729407-19 2020-06-12T02:39:12.133 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1956 (ObjNum 10631 snap 0 seq_num 10631) dirty exists 2020-06-12T02:39:12.133 INFO:tasks.rados.rados.0.smithi107.stdout:10632: left oid 19 (ObjNum 10631 snap 0 seq_num 10631) 2020-06-12T02:39:12.134 INFO:tasks.rados.rados.0.smithi107.stdout:10634: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:12.134 INFO:tasks.rados.rados.0.smithi107.stdout:10632: done (3 left) 2020-06-12T02:39:12.134 INFO:tasks.rados.rados.0.smithi107.stdout:10636: done (2 left) 2020-06-12T02:39:12.134 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:39:12.134 INFO:tasks.rados.rados.0.smithi107.stdout:10637: seq_num 10636 ranges {3842048=540672} 2020-06-12T02:39:12.135 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:39:12.135 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:39:12.135 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:39:12.135 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:39:12.135 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:39:12.136 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:39:12.136 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:39:12.136 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2020-06-12T02:39:12.137 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-06-12T02:39:12.137 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-06-12T02:39:12.137 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-06-12T02:39:12.137 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-06-12T02:39:12.137 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-06-12T02:39:12.138 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-06-12T02:39:12.138 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-06-12T02:39:12.138 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-06-12T02:39:12.138 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:39:12.081+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:39:12.139 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-06-12T02:39:12.141 INFO:tasks.rados.rados.0.smithi107.stdout:10637: writing smithi10729407-5 from 3842048 to 4382720 tid 1 2020-06-12T02:39:12.142 INFO:tasks.rados.rados.0.smithi107.stdout:10638: delete oid 19 current snap is 0 2020-06-12T02:39:12.142 INFO:tasks.rados.rados.0.smithi107.stdout:10634: finishing write tid 3 to smithi10729407-16 2020-06-12T02:39:12.142 INFO:tasks.rados.rados.0.smithi107.stdout:10635: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:12.142 INFO:tasks.rados.rados.0.smithi107.stdout:10635: finishing write tid 3 to smithi10729407-14 2020-06-12T02:39:12.142 INFO:tasks.rados.rados.0.smithi107.stdout:10634: finishing write tid 4 to smithi10729407-16 2020-06-12T02:39:12.143 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1896 (ObjNum 10633 snap 0 seq_num 10633) dirty exists 2020-06-12T02:39:12.143 INFO:tasks.rados.rados.0.smithi107.stdout:10634: left oid 16 (ObjNum 10633 snap 0 seq_num 10633) 2020-06-12T02:39:12.143 INFO:tasks.rados.rados.0.smithi107.stdout:10635: finishing write tid 4 to smithi10729407-14 2020-06-12T02:39:12.143 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1704 (ObjNum 10634 snap 0 seq_num 10634) dirty exists 2020-06-12T02:39:12.143 INFO:tasks.rados.rados.0.smithi107.stdout:10635: left oid 14 (ObjNum 10634 snap 0 seq_num 10634) 2020-06-12T02:39:12.144 INFO:tasks.rados.rados.0.smithi107.stdout:10634: done (3 left) 2020-06-12T02:39:12.144 INFO:tasks.rados.rados.0.smithi107.stdout:10635: done (2 left) 2020-06-12T02:39:12.144 INFO:tasks.rados.rados.0.smithi107.stdout:10638: done (1 left) 2020-06-12T02:39:12.144 INFO:tasks.rados.rados.0.smithi107.stdout:10639: delete oid 20 current snap is 0 2020-06-12T02:39:12.144 INFO:tasks.rados.rados.0.smithi107.stdout:10639: done (1 left) 2020-06-12T02:39:12.145 INFO:tasks.rados.rados.0.smithi107.stdout:10640: delete oid 13 current snap is 0 2020-06-12T02:39:12.145 INFO:tasks.rados.rados.0.smithi107.stdout:10637: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:12.145 INFO:tasks.rados.rados.0.smithi107.stdout:10637: finishing write tid 3 to smithi10729407-5 2020-06-12T02:39:12.145 INFO:tasks.rados.rados.0.smithi107.stdout:10640: done (1 left) 2020-06-12T02:39:12.145 INFO:tasks.rados.rados.0.smithi107.stdout:10641: delete oid 2 current snap is 0 2020-06-12T02:39:12.146 INFO:tasks.rados.rados.0.smithi107.stdout:10637: finishing write tid 4 to smithi10729407-5 2020-06-12T02:39:12.146 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 1014 (ObjNum 10636 snap 0 seq_num 10636) dirty exists 2020-06-12T02:39:12.146 INFO:tasks.rados.rados.0.smithi107.stdout:10637: left oid 5 (ObjNum 10636 snap 0 seq_num 10636) 2020-06-12T02:39:12.146 INFO:tasks.rados.rados.0.smithi107.stdout:10637: done (1 left) 2020-06-12T02:39:12.146 INFO:tasks.rados.rados.0.smithi107.stdout:10641: done (0 left) 2020-06-12T02:39:12.147 INFO:tasks.rados.rados.0.smithi107.stdout:10642: delete oid 5 current snap is 0 2020-06-12T02:39:12.147 INFO:tasks.rados.rados.0.smithi107.stdout:10642: done (0 left) 2020-06-12T02:39:12.147 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:39:12.147 INFO:tasks.rados.rados.0.smithi107.stdout:10643: seq_num 10642 ranges {0=638976} 2020-06-12T02:39:12.147 INFO:tasks.rados.rados.0.smithi107.stdout:10643: writing smithi10729407-2 from 0 to 638976 tid 1 2020-06-12T02:39:12.147 INFO:tasks.rados.rados.0.smithi107.stdout:10644: delete oid 19 current snap is 0 2020-06-12T02:39:12.148 INFO:tasks.rados.rados.0.smithi107.stdout:10644: done (1 left) 2020-06-12T02:39:12.148 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:39:12.148 INFO:tasks.rados.rados.0.smithi107.stdout:10645: seq_num 10644 ranges {491520=671744} 2020-06-12T02:39:12.148 INFO:tasks.rados.rados.0.smithi107.stdout:10645: writing smithi10729407-16 from 491520 to 1163264 tid 1 2020-06-12T02:39:12.148 INFO:tasks.rados.rados.0.smithi107.stdout:10646: delete oid 14 current snap is 0 2020-06-12T02:39:12.149 INFO:tasks.rados.rados.0.smithi107.stdout:10643: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:12.149 INFO:tasks.rados.rados.0.smithi107.stdout:10646: done (2 left) 2020-06-12T02:39:12.149 INFO:tasks.rados.rados.0.smithi107.stdout:10647: delete oid 6 current snap is 0 2020-06-12T02:39:12.149 INFO:tasks.rados.rados.0.smithi107.stdout:10643: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:12.150 INFO:tasks.rados.rados.0.smithi107.stdout:10643: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:12.151 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 994 (ObjNum 10642 snap 0 seq_num 10642) dirty exists 2020-06-12T02:39:12.151 INFO:tasks.rados.rados.0.smithi107.stdout:10643: left oid 2 (ObjNum 10642 snap 0 seq_num 10642) 2020-06-12T02:39:12.151 INFO:tasks.rados.rados.0.smithi107.stdout:10643: done (2 left) 2020-06-12T02:39:12.151 INFO:tasks.rados.rados.0.smithi107.stdout:10647: done (1 left) 2020-06-12T02:39:12.151 INFO:tasks.rados.rados.0.smithi107.stdout:10648: delete oid 2 current snap is 0 2020-06-12T02:39:12.151 INFO:tasks.rados.rados.0.smithi107.stdout:10648: done (1 left) 2020-06-12T02:39:12.152 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:39:12.152 INFO:tasks.rados.rados.0.smithi107.stdout:10649: seq_num 10648 ranges {0=737280} 2020-06-12T02:39:12.152 INFO:tasks.rados.rados.0.smithi107.stdout:10649: writing smithi10729407-12 from 0 to 737280 tid 1 2020-06-12T02:39:12.152 INFO:tasks.rados.rados.0.smithi107.stdout:10650: delete oid 10 current snap is 0 2020-06-12T02:39:12.153 INFO:tasks.rados.rados.0.smithi107.stdout:10645: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:12.153 INFO:tasks.rados.rados.0.smithi107.stdout:10645: finishing write tid 3 to smithi10729407-16 2020-06-12T02:39:12.154 INFO:tasks.rados.rados.0.smithi107.stdout:10645: finishing write tid 4 to smithi10729407-16 2020-06-12T02:39:12.154 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1898 (ObjNum 10644 snap 0 seq_num 10644) dirty exists 2020-06-12T02:39:12.154 INFO:tasks.rados.rados.0.smithi107.stdout:10645: left oid 16 (ObjNum 10644 snap 0 seq_num 10644) 2020-06-12T02:39:12.154 INFO:tasks.rados.rados.0.smithi107.stdout:10645: done (2 left) 2020-06-12T02:39:12.155 INFO:tasks.rados.rados.0.smithi107.stdout:10650: done (1 left) 2020-06-12T02:39:12.155 INFO:tasks.rados.rados.0.smithi107.stdout:10651: delete oid 6 current snap is 0 2020-06-12T02:39:12.157 INFO:tasks.rados.rados.0.smithi107.stdout:10651: done (1 left) 2020-06-12T02:39:12.157 INFO:tasks.rados.rados.0.smithi107.stdout:10652: delete oid 16 current snap is 0 2020-06-12T02:39:12.158 INFO:tasks.rados.rados.0.smithi107.stdout:10649: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:12.162 INFO:tasks.rados.rados.0.smithi107.stdout:10652: done (1 left) 2020-06-12T02:39:12.162 INFO:tasks.rados.rados.0.smithi107.stdout:10653: delete oid 14 current snap is 0 2020-06-12T02:39:12.162 INFO:tasks.rados.rados.0.smithi107.stdout:10649: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:12.164 INFO:tasks.rados.rados.0.smithi107.stdout:10649: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:12.164 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1961 (ObjNum 10648 snap 0 seq_num 10648) dirty exists 2020-06-12T02:39:12.165 INFO:tasks.rados.rados.0.smithi107.stdout:10649: left oid 12 (ObjNum 10648 snap 0 seq_num 10648) 2020-06-12T02:39:12.165 INFO:tasks.rados.rados.0.smithi107.stdout:10649: done (1 left) 2020-06-12T02:39:12.165 INFO:tasks.rados.rados.0.smithi107.stdout:10653: done (0 left) 2020-06-12T02:39:12.166 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:39:12.166 INFO:tasks.rados.rados.0.smithi107.stdout:10654: seq_num 10653 ranges {0=425984,425984=401408} 2020-06-12T02:39:12.167 INFO:tasks.rados.rados.0.smithi107.stdout:10654: writing smithi10729407-8 from 0 to 425984 tid 1 2020-06-12T02:39:12.170 INFO:tasks.rados.rados.0.smithi107.stdout:10654: writing smithi10729407-8 from 425984 to 827392 tid 2 2020-06-12T02:39:12.170 INFO:tasks.rados.rados.0.smithi107.stdout:10655: delete oid 6 current snap is 0 2020-06-12T02:39:12.173 INFO:tasks.rados.rados.0.smithi107.stdout:10655: done (1 left) 2020-06-12T02:39:12.173 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:39:12.173 INFO:tasks.rados.rados.0.smithi107.stdout:10656: seq_num 10655 ranges {0=458752,458752=401408} 2020-06-12T02:39:12.176 INFO:tasks.rados.rados.0.smithi107.stdout:10656: writing smithi10729407-5 from 0 to 458752 tid 1 2020-06-12T02:39:12.178 INFO:tasks.rados.rados.0.smithi107.stdout:10656: writing smithi10729407-5 from 458752 to 860160 tid 2 2020-06-12T02:39:12.178 INFO:tasks.rados.rados.0.smithi107.stdout:10657: delete oid 1 current snap is 0 2020-06-12T02:39:12.178 INFO:tasks.rados.rados.0.smithi107.stdout:10654: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:12.180 INFO:tasks.rados.rados.0.smithi107.stdout:10654: finishing write tid 2 to smithi10729407-8 2020-06-12T02:39:12.181 INFO:tasks.rados.rados.0.smithi107.stdout:10656: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:12.182 INFO:tasks.rados.rados.0.smithi107.stdout:10657: done (2 left) 2020-06-12T02:39:12.182 INFO:tasks.rados.rados.0.smithi107.stdout:10658: delete oid 4 current snap is 0 2020-06-12T02:39:12.182 INFO:tasks.rados.rados.0.smithi107.stdout:10654: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:12.184 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:39:12.183+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:39:12.185 INFO:tasks.rados.rados.0.smithi107.stdout:10654: finishing write tid 5 to smithi10729407-8 2020-06-12T02:39:12.185 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1986 (ObjNum 10653 snap 0 seq_num 10653) dirty exists 2020-06-12T02:39:12.186 INFO:tasks.rados.rados.0.smithi107.stdout:10654: left oid 8 (ObjNum 10653 snap 0 seq_num 10653) 2020-06-12T02:39:12.187 INFO:tasks.rados.rados.0.smithi107.stdout:10654: done (2 left) 2020-06-12T02:39:12.187 INFO:tasks.rados.rados.0.smithi107.stdout:10658: done (1 left) 2020-06-12T02:39:12.187 INFO:tasks.rados.rados.0.smithi107.stdout:10659: delete oid 1 current snap is 0 2020-06-12T02:39:12.190 INFO:tasks.rados.rados.0.smithi107.stdout:10656: finishing write tid 2 to smithi10729407-5 2020-06-12T02:39:12.191 INFO:tasks.rados.rados.0.smithi107.stdout:10659: done (1 left) 2020-06-12T02:39:12.191 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:39:12.191 INFO:tasks.rados.rados.0.smithi107.stdout:10660: seq_num 10659 ranges {0=524288,524288=401408} 2020-06-12T02:39:12.193 INFO:tasks.rados.rados.0.smithi107.stdout:10660: writing smithi10729407-20 from 0 to 524288 tid 1 2020-06-12T02:39:12.196 INFO:tasks.rados.rados.0.smithi107.stdout:10660: writing smithi10729407-20 from 524288 to 925696 tid 2 2020-06-12T02:39:12.196 INFO:tasks.rados.rados.0.smithi107.stdout:10661: delete oid 18 current snap is 0 2020-06-12T02:39:12.196 INFO:tasks.rados.rados.0.smithi107.stdout:10656: finishing write tid 4 to smithi10729407-5 2020-06-12T02:39:12.197 INFO:tasks.rados.rados.0.smithi107.stdout:10656: finishing write tid 5 to smithi10729407-5 2020-06-12T02:39:12.197 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 1018 (ObjNum 10655 snap 0 seq_num 10655) dirty exists 2020-06-12T02:39:12.197 INFO:tasks.rados.rados.0.smithi107.stdout:10656: left oid 5 (ObjNum 10655 snap 0 seq_num 10655) 2020-06-12T02:39:12.198 INFO:tasks.rados.rados.0.smithi107.stdout:10656: done (2 left) 2020-06-12T02:39:12.199 INFO:tasks.rados.rados.0.smithi107.stdout:10661: done (1 left) 2020-06-12T02:39:12.199 INFO:tasks.rados.rados.0.smithi107.stdout:10662: delete oid 18 current snap is 0 2020-06-12T02:39:12.202 INFO:tasks.rados.rados.0.smithi107.stdout:10660: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:12.202 INFO:tasks.rados.rados.0.smithi107.stdout:10660: finishing write tid 2 to smithi10729407-20 2020-06-12T02:39:12.204 INFO:tasks.rados.rados.0.smithi107.stdout:10662: done (1 left) 2020-06-12T02:39:12.204 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:39:12.205 INFO:tasks.rados.rados.0.smithi107.stdout:10663: seq_num 10662 ranges {0=573440,573440=401408} 2020-06-12T02:39:12.210 INFO:tasks.rados.rados.0.smithi107.stdout:10663: writing smithi10729407-6 from 0 to 573440 tid 1 2020-06-12T02:39:12.213 INFO:tasks.rados.rados.0.smithi107.stdout:10663: writing smithi10729407-6 from 573440 to 974848 tid 2 2020-06-12T02:39:12.214 INFO:tasks.rados.rados.0.smithi107.stdout:10664: delete oid 18 current snap is 0 2020-06-12T02:39:12.214 INFO:tasks.rados.rados.0.smithi107.stdout:10660: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:12.214 INFO:tasks.rados.rados.0.smithi107.stdout:10660: finishing write tid 5 to smithi10729407-20 2020-06-12T02:39:12.214 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 832 (ObjNum 10659 snap 0 seq_num 10659) dirty exists 2020-06-12T02:39:12.214 INFO:tasks.rados.rados.0.smithi107.stdout:10660: left oid 20 (ObjNum 10659 snap 0 seq_num 10659) 2020-06-12T02:39:12.215 INFO:tasks.rados.rados.0.smithi107.stdout:10660: done (2 left) 2020-06-12T02:39:12.215 INFO:tasks.rados.rados.0.smithi107.stdout:10664: done (1 left) 2020-06-12T02:39:12.216 INFO:tasks.rados.rados.0.smithi107.stdout:10665: delete oid 8 current snap is 0 2020-06-12T02:39:12.216 INFO:tasks.rados.rados.0.smithi107.stdout:10663: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:12.216 INFO:tasks.rados.rados.0.smithi107.stdout:10665: done (1 left) 2020-06-12T02:39:12.216 INFO:tasks.rados.rados.0.smithi107.stdout:10666: delete oid 16 current snap is 0 2020-06-12T02:39:12.219 INFO:tasks.rados.rados.0.smithi107.stdout:10666: done (1 left) 2020-06-12T02:39:12.219 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:39:12.219 INFO:tasks.rados.rados.0.smithi107.stdout:10667: seq_num 10666 ranges {0=638976,638976=401408} 2020-06-12T02:39:12.222 INFO:tasks.rados.rados.0.smithi107.stdout:10667: writing smithi10729407-14 from 0 to 638976 tid 1 2020-06-12T02:39:12.225 INFO:tasks.rados.rados.0.smithi107.stdout:10667: writing smithi10729407-14 from 638976 to 1040384 tid 2 2020-06-12T02:39:12.225 INFO:tasks.rados.rados.0.smithi107.stdout:10668: delete oid 18 current snap is 0 2020-06-12T02:39:12.225 INFO:tasks.rados.rados.0.smithi107.stdout:10663: finishing write tid 2 to smithi10729407-6 2020-06-12T02:39:12.225 INFO:tasks.rados.rados.0.smithi107.stdout:10663: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:12.226 INFO:tasks.rados.rados.0.smithi107.stdout:10663: finishing write tid 5 to smithi10729407-6 2020-06-12T02:39:12.226 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1926 (ObjNum 10662 snap 0 seq_num 10662) dirty exists 2020-06-12T02:39:12.226 INFO:tasks.rados.rados.0.smithi107.stdout:10663: left oid 6 (ObjNum 10662 snap 0 seq_num 10662) 2020-06-12T02:39:12.227 INFO:tasks.rados.rados.0.smithi107.stdout:10663: done (2 left) 2020-06-12T02:39:12.227 INFO:tasks.rados.rados.0.smithi107.stdout:10668: done (1 left) 2020-06-12T02:39:12.228 INFO:tasks.rados.rados.0.smithi107.stdout:10669: delete oid 4 current snap is 0 2020-06-12T02:39:12.228 INFO:tasks.rados.rados.0.smithi107.stdout:10667: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:12.231 INFO:tasks.rados.rados.0.smithi107.stdout:10669: done (1 left) 2020-06-12T02:39:12.231 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:39:12.231 INFO:tasks.rados.rados.0.smithi107.stdout:10670: seq_num 10669 ranges {1163264=688128,1851392=401408} 2020-06-12T02:39:12.241 INFO:tasks.rados.rados.0.smithi107.stdout:10670: writing smithi10729407-3 from 1163264 to 1851392 tid 1 2020-06-12T02:39:12.244 INFO:tasks.rados.rados.0.smithi107.stdout:10670: writing smithi10729407-3 from 1851392 to 2252800 tid 2 2020-06-12T02:39:12.244 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:39:12.245 INFO:tasks.rados.rados.0.smithi107.stdout:10671: seq_num 10670 ranges {0=704512,704512=401408} 2020-06-12T02:39:12.249 INFO:tasks.rados.rados.0.smithi107.stdout:10671: writing smithi10729407-16 from 0 to 704512 tid 1 2020-06-12T02:39:12.252 INFO:tasks.rados.rados.0.smithi107.stdout:10671: writing smithi10729407-16 from 704512 to 1105920 tid 2 2020-06-12T02:39:12.252 INFO:tasks.rados.rados.0.smithi107.stdout:10672: delete oid 1 current snap is 0 2020-06-12T02:39:12.252 INFO:tasks.rados.rados.0.smithi107.stdout:10667: finishing write tid 2 to smithi10729407-14 2020-06-12T02:39:12.252 INFO:tasks.rados.rados.0.smithi107.stdout:10667: finishing write tid 4 to smithi10729407-14 2020-06-12T02:39:12.252 INFO:tasks.rados.rados.0.smithi107.stdout:10667: finishing write tid 5 to smithi10729407-14 2020-06-12T02:39:12.253 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1709 (ObjNum 10666 snap 0 seq_num 10666) dirty exists 2020-06-12T02:39:12.253 INFO:tasks.rados.rados.0.smithi107.stdout:10667: left oid 14 (ObjNum 10666 snap 0 seq_num 10666) 2020-06-12T02:39:12.254 INFO:tasks.rados.rados.0.smithi107.stdout:10670: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:12.255 INFO:tasks.rados.rados.0.smithi107.stdout:10667: done (3 left) 2020-06-12T02:39:12.255 INFO:tasks.rados.rados.0.smithi107.stdout:10672: done (2 left) 2020-06-12T02:39:12.255 INFO:tasks.rados.rados.0.smithi107.stdout:10673: delete oid 1 current snap is 0 2020-06-12T02:39:12.257 INFO:tasks.rados.rados.0.smithi107.stdout:10673: done (2 left) 2020-06-12T02:39:12.257 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:39:12.257 INFO:tasks.rados.rados.0.smithi107.stdout:10674: seq_num 10673 ranges {0=761856,761856=401408} 2020-06-12T02:39:12.262 INFO:tasks.rados.rados.0.smithi107.stdout:10674: writing smithi10729407-8 from 0 to 761856 tid 1 2020-06-12T02:39:12.264 INFO:tasks.rados.rados.0.smithi107.stdout:10674: writing smithi10729407-8 from 761856 to 1163264 tid 2 2020-06-12T02:39:12.265 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:39:12.265 INFO:tasks.rados.rados.0.smithi107.stdout:10675: seq_num 10674 ranges {0=778240,778240=401408} 2020-06-12T02:39:12.270 INFO:tasks.rados.rados.0.smithi107.stdout:10675: writing smithi10729407-15 from 0 to 778240 tid 1 2020-06-12T02:39:12.272 INFO:tasks.rados.rados.0.smithi107.stdout:10675: writing smithi10729407-15 from 778240 to 1179648 tid 2 2020-06-12T02:39:12.273 INFO:tasks.rados.rados.0.smithi107.stdout:10676: delete oid 4 current snap is 0 2020-06-12T02:39:12.273 INFO:tasks.rados.rados.0.smithi107.stdout:10671: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:12.273 INFO:tasks.rados.rados.0.smithi107.stdout:10670: finishing write tid 2 to smithi10729407-3 2020-06-12T02:39:12.273 INFO:tasks.rados.rados.0.smithi107.stdout:10670: finishing write tid 4 to smithi10729407-3 2020-06-12T02:39:12.273 INFO:tasks.rados.rados.0.smithi107.stdout:10671: finishing write tid 2 to smithi10729407-16 2020-06-12T02:39:12.274 INFO:tasks.rados.rados.0.smithi107.stdout:10670: finishing write tid 5 to smithi10729407-3 2020-06-12T02:39:12.274 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1904 (ObjNum 10669 snap 0 seq_num 10669) dirty exists 2020-06-12T02:39:12.274 INFO:tasks.rados.rados.0.smithi107.stdout:10670: left oid 3 (ObjNum 10669 snap 0 seq_num 10669) 2020-06-12T02:39:12.274 INFO:tasks.rados.rados.0.smithi107.stdout:10671: finishing write tid 4 to smithi10729407-16 2020-06-12T02:39:12.276 INFO:tasks.rados.rados.0.smithi107.stdout:10671: finishing write tid 5 to smithi10729407-16 2020-06-12T02:39:12.276 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1906 (ObjNum 10670 snap 0 seq_num 10670) dirty exists 2020-06-12T02:39:12.276 INFO:tasks.rados.rados.0.smithi107.stdout:10671: left oid 16 (ObjNum 10670 snap 0 seq_num 10670) 2020-06-12T02:39:12.277 INFO:tasks.rados.rados.0.smithi107.stdout:10674: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:12.279 INFO:tasks.rados.rados.0.smithi107.stdout:10670: done (4 left) 2020-06-12T02:39:12.279 INFO:tasks.rados.rados.0.smithi107.stdout:10671: done (3 left) 2020-06-12T02:39:12.279 INFO:tasks.rados.rados.0.smithi107.stdout:10676: done (2 left) 2020-06-12T02:39:12.280 INFO:tasks.rados.rados.0.smithi107.stdout:10677: delete oid 5 current snap is 0 2020-06-12T02:39:12.282 INFO:tasks.rados.rados.0.smithi107.stdout:10674: finishing write tid 2 to smithi10729407-8 2020-06-12T02:39:12.282 INFO:tasks.rados.rados.0.smithi107.stdout:10674: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:12.282 INFO:tasks.rados.rados.0.smithi107.stdout:10675: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:12.283 INFO:tasks.rados.rados.0.smithi107.stdout:10674: finishing write tid 5 to smithi10729407-8 2020-06-12T02:39:12.283 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1990 (ObjNum 10673 snap 0 seq_num 10673) dirty exists 2020-06-12T02:39:12.283 INFO:tasks.rados.rados.0.smithi107.stdout:10674: left oid 8 (ObjNum 10673 snap 0 seq_num 10673) 2020-06-12T02:39:12.284 INFO:tasks.rados.rados.0.smithi107.stdout:10675: finishing write tid 2 to smithi10729407-15 2020-06-12T02:39:12.284 INFO:tasks.rados.rados.0.smithi107.stdout:10674: done (2 left) 2020-06-12T02:39:12.284 INFO:tasks.rados.rados.0.smithi107.stdout:10677: done (1 left) 2020-06-12T02:39:12.285 INFO:tasks.rados.rados.0.smithi107.stdout:10678: delete oid 9 current snap is 0 2020-06-12T02:39:12.285 INFO:tasks.rados.rados.0.smithi107.stdout:10675: finishing write tid 4 to smithi10729407-15 2020-06-12T02:39:12.286 INFO:tasks.rados.rados.0.smithi107.stdout:10675: finishing write tid 5 to smithi10729407-15 2020-06-12T02:39:12.286 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 1993 (ObjNum 10674 snap 0 seq_num 10674) dirty exists 2020-06-12T02:39:12.287 INFO:tasks.rados.rados.0.smithi107.stdout:10675: left oid 15 (ObjNum 10674 snap 0 seq_num 10674) 2020-06-12T02:39:12.287 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:39:12.285+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:39:12.288 INFO:tasks.rados.rados.0.smithi107.stdout:10675: done (1 left) 2020-06-12T02:39:12.288 INFO:tasks.rados.rados.0.smithi107.stdout:10678: done (0 left) 2020-06-12T02:39:12.288 INFO:tasks.rados.rados.0.smithi107.stdout:10679: delete oid 12 current snap is 0 2020-06-12T02:39:12.291 INFO:tasks.rados.rados.0.smithi107.stdout:10679: done (0 left) 2020-06-12T02:39:12.291 INFO:tasks.rados.rados.0.smithi107.stdout:10680: delete oid 6 current snap is 0 2020-06-12T02:39:12.293 INFO:tasks.rados.rados.0.smithi107.stdout:10680: done (0 left) 2020-06-12T02:39:12.293 INFO:tasks.rados.rados.0.smithi107.stdout:10681: delete oid 18 current snap is 0 2020-06-12T02:39:12.295 INFO:tasks.rados.rados.0.smithi107.stdout:10681: done (0 left) 2020-06-12T02:39:12.295 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:39:12.296 INFO:tasks.rados.rados.0.smithi107.stdout:10682: seq_num 10681 ranges {0=491520,491520=704512,1196032=98304} 2020-06-12T02:39:12.298 INFO:tasks.rados.rados.0.smithi107.stdout:10682: writing smithi10729407-6 from 0 to 491520 tid 1 2020-06-12T02:39:12.298 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:12.298 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:12.302 INFO:tasks.rados.rados.0.smithi107.stdout:10682: writing smithi10729407-6 from 491520 to 1196032 tid 2 2020-06-12T02:39:12.303 INFO:tasks.rados.rados.0.smithi107.stdout:10682: writing smithi10729407-6 from 1196032 to 1294336 tid 3 2020-06-12T02:39:12.303 INFO:tasks.rados.rados.0.smithi107.stdout:10683: delete oid 18 current snap is 0 2020-06-12T02:39:12.303 INFO:tasks.rados.rados.0.smithi107.stdout:10682: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:12.305 INFO:tasks.rados.rados.0.smithi107.stdout:10683: done (1 left) 2020-06-12T02:39:12.305 INFO:tasks.rados.rados.0.smithi107.stdout:10684: delete oid 13 current snap is 0 2020-06-12T02:39:12.308 INFO:tasks.rados.rados.0.smithi107.stdout:10684: done (1 left) 2020-06-12T02:39:12.308 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:39:12.309 INFO:tasks.rados.rados.0.smithi107.stdout:10685: seq_num 10684 ranges {0=540672,540672=409600,950272=393216} 2020-06-12T02:39:12.309 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-06-12T02:39:12.312 INFO:tasks.rados.rados.0.smithi107.stdout:10685: writing smithi10729407-18 from 0 to 540672 tid 1 2020-06-12T02:39:12.315 INFO:tasks.rados.rados.0.smithi107.stdout:10685: writing smithi10729407-18 from 540672 to 950272 tid 2 2020-06-12T02:39:12.317 INFO:tasks.rados.rados.0.smithi107.stdout:10685: writing smithi10729407-18 from 950272 to 1343488 tid 3 2020-06-12T02:39:12.318 INFO:tasks.rados.rados.0.smithi107.stdout:10686: delete oid 2 current snap is 0 2020-06-12T02:39:12.318 INFO:tasks.rados.rados.0.smithi107.stdout:10682: finishing write tid 2 to smithi10729407-6 2020-06-12T02:39:12.318 INFO:tasks.rados.rados.0.smithi107.stdout:10682: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:12.318 INFO:tasks.rados.rados.0.smithi107.stdout:10682: finishing write tid 5 to smithi10729407-6 2020-06-12T02:39:12.318 INFO:tasks.rados.rados.0.smithi107.stdout:10682: finishing write tid 6 to smithi10729407-6 2020-06-12T02:39:12.319 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1933 (ObjNum 10681 snap 0 seq_num 10681) dirty exists 2020-06-12T02:39:12.319 INFO:tasks.rados.rados.0.smithi107.stdout:10682: left oid 6 (ObjNum 10681 snap 0 seq_num 10681) 2020-06-12T02:39:12.319 INFO:tasks.rados.rados.0.smithi107.stdout:10685: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:12.320 INFO:tasks.rados.rados.0.smithi107.stdout:10682: done (2 left) 2020-06-12T02:39:12.320 INFO:tasks.rados.rados.0.smithi107.stdout:10686: done (1 left) 2020-06-12T02:39:12.320 INFO:tasks.rados.rados.0.smithi107.stdout:10687: delete oid 17 current snap is 0 2020-06-12T02:39:12.323 INFO:tasks.rados.rados.0.smithi107.stdout:10687: done (1 left) 2020-06-12T02:39:12.323 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:39:12.323 INFO:tasks.rados.rados.0.smithi107.stdout:10688: seq_num 10687 ranges {0=589824,589824=458752,1048576=344064} 2020-06-12T02:39:12.326 INFO:tasks.rados.rados.0.smithi107.stdout:10688: writing smithi10729407-5 from 0 to 589824 tid 1 2020-06-12T02:39:12.329 INFO:tasks.rados.rados.0.smithi107.stdout:10688: writing smithi10729407-5 from 589824 to 1048576 tid 2 2020-06-12T02:39:12.331 INFO:tasks.rados.rados.0.smithi107.stdout:10688: writing smithi10729407-5 from 1048576 to 1392640 tid 3 2020-06-12T02:39:12.332 INFO:tasks.rados.rados.0.smithi107.stdout:10689: delete oid 20 current snap is 0 2020-06-12T02:39:12.333 INFO:tasks.rados.rados.0.smithi107.stdout:10685: finishing write tid 2 to smithi10729407-18 2020-06-12T02:39:12.333 INFO:tasks.rados.rados.0.smithi107.stdout:10685: finishing write tid 3 to smithi10729407-18 2020-06-12T02:39:12.333 INFO:tasks.rados.rados.0.smithi107.stdout:10685: finishing write tid 5 to smithi10729407-18 2020-06-12T02:39:12.333 INFO:tasks.rados.rados.0.smithi107.stdout:10685: finishing write tid 6 to smithi10729407-18 2020-06-12T02:39:12.334 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1852 (ObjNum 10684 snap 0 seq_num 10684) dirty exists 2020-06-12T02:39:12.334 INFO:tasks.rados.rados.0.smithi107.stdout:10685: left oid 18 (ObjNum 10684 snap 0 seq_num 10684) 2020-06-12T02:39:12.335 INFO:tasks.rados.rados.0.smithi107.stdout:10688: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:12.337 INFO:tasks.rados.rados.0.smithi107.stdout:10685: done (2 left) 2020-06-12T02:39:12.338 INFO:tasks.rados.rados.0.smithi107.stdout:10689: done (1 left) 2020-06-12T02:39:12.338 INFO:tasks.rados.rados.0.smithi107.stdout:10690: delete oid 16 current snap is 0 2020-06-12T02:39:12.341 INFO:tasks.rados.rados.0.smithi107.stdout:10688: finishing write tid 2 to smithi10729407-5 2020-06-12T02:39:12.341 INFO:tasks.rados.rados.0.smithi107.stdout:10690: done (1 left) 2020-06-12T02:39:12.342 INFO:tasks.rados.rados.0.smithi107.stdout:10691: delete oid 11 current snap is 0 2020-06-12T02:39:12.343 INFO:tasks.rados.rados.0.smithi107.stdout:10688: finishing write tid 3 to smithi10729407-5 2020-06-12T02:39:12.347 INFO:tasks.rados.rados.0.smithi107.stdout:10691: done (1 left) 2020-06-12T02:39:12.347 INFO:tasks.rados.rados.0.smithi107.stdout:10692: delete oid 10 current snap is 0 2020-06-12T02:39:12.347 INFO:tasks.rados.rados.0.smithi107.stdout:10688: finishing write tid 5 to smithi10729407-5 2020-06-12T02:39:12.347 INFO:tasks.rados.rados.0.smithi107.stdout:10688: finishing write tid 6 to smithi10729407-5 2020-06-12T02:39:12.348 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 1023 (ObjNum 10687 snap 0 seq_num 10687) dirty exists 2020-06-12T02:39:12.348 INFO:tasks.rados.rados.0.smithi107.stdout:10688: left oid 5 (ObjNum 10687 snap 0 seq_num 10687) 2020-06-12T02:39:12.348 INFO:tasks.rados.rados.0.smithi107.stdout:10688: done (1 left) 2020-06-12T02:39:12.349 INFO:tasks.rados.rados.0.smithi107.stdout:10692: done (0 left) 2020-06-12T02:39:12.349 INFO:tasks.rados.rados.0.smithi107.stdout:10693: delete oid 10 current snap is 0 2020-06-12T02:39:12.351 INFO:tasks.rados.rados.0.smithi107.stdout:10693: done (0 left) 2020-06-12T02:39:12.351 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:39:12.351 INFO:tasks.rados.rados.0.smithi107.stdout:10694: seq_num 10693 ranges {0=696320,696320=614400,1310720=188416} 2020-06-12T02:39:12.354 INFO:tasks.rados.rados.0.smithi107.stdout:10694: writing smithi10729407-12 from 0 to 696320 tid 1 2020-06-12T02:39:12.357 INFO:tasks.rados.rados.0.smithi107.stdout:10694: writing smithi10729407-12 from 696320 to 1310720 tid 2 2020-06-12T02:39:12.358 INFO:tasks.rados.rados.0.smithi107.stdout:10694: writing smithi10729407-12 from 1310720 to 1499136 tid 3 2020-06-12T02:39:12.359 INFO:tasks.rados.rados.0.smithi107.stdout:10695: delete oid 6 current snap is 0 2020-06-12T02:39:12.366 INFO:tasks.rados.rados.0.smithi107.stdout:10694: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:12.366 INFO:tasks.rados.rados.0.smithi107.stdout:10695: done (1 left) 2020-06-12T02:39:12.367 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:39:12.367 INFO:tasks.rados.rados.0.smithi107.stdout:10696: seq_num 10695 ranges {1392640=729088,2121728=786432,2908160=16384} 2020-06-12T02:39:12.376 INFO:tasks.rados.rados.0.smithi107.stdout:10696: writing smithi10729407-5 from 1392640 to 2121728 tid 1 2020-06-12T02:39:12.381 INFO:tasks.rados.rados.0.smithi107.stdout:10696: writing smithi10729407-5 from 2121728 to 2908160 tid 2 2020-06-12T02:39:12.381 INFO:tasks.rados.rados.0.smithi107.stdout:10696: writing smithi10729407-5 from 2908160 to 2924544 tid 3 2020-06-12T02:39:12.381 INFO:tasks.rados.rados.0.smithi107.stdout:10697: delete oid 14 current snap is 0 2020-06-12T02:39:12.381 INFO:tasks.rados.rados.0.smithi107.stdout:10694: finishing write tid 2 to smithi10729407-12 2020-06-12T02:39:12.382 INFO:tasks.rados.rados.0.smithi107.stdout:10694: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:12.382 INFO:tasks.rados.rados.0.smithi107.stdout:10694: finishing write tid 5 to smithi10729407-12 2020-06-12T02:39:12.382 INFO:tasks.rados.rados.0.smithi107.stdout:10694: finishing write tid 6 to smithi10729407-12 2020-06-12T02:39:12.382 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1966 (ObjNum 10693 snap 0 seq_num 10693) dirty exists 2020-06-12T02:39:12.382 INFO:tasks.rados.rados.0.smithi107.stdout:10694: left oid 12 (ObjNum 10693 snap 0 seq_num 10693) 2020-06-12T02:39:12.383 INFO:tasks.rados.rados.0.smithi107.stdout:10696: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:12.384 INFO:tasks.rados.rados.0.smithi107.stdout:10694: done (2 left) 2020-06-12T02:39:12.384 INFO:tasks.rados.rados.0.smithi107.stdout:10697: done (1 left) 2020-06-12T02:39:12.384 INFO:tasks.rados.rados.0.smithi107.stdout:10698: delete oid 16 current snap is 0 2020-06-12T02:39:12.387 INFO:tasks.rados.rados.0.smithi107.stdout:10698: done (1 left) 2020-06-12T02:39:12.388 INFO:tasks.rados.rados.0.smithi107.stdout:10699: delete oid 7 current snap is 0 2020-06-12T02:39:12.388 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:39:12.386+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:39:12.392 INFO:tasks.rados.rados.0.smithi107.stdout:10699: done (1 left) 2020-06-12T02:39:12.392 INFO:tasks.rados.rados.0.smithi107.stdout:10700: delete oid 16 current snap is 0 2020-06-12T02:39:12.392 INFO:tasks.rados.rados.0.smithi107.stdout:10696: finishing write tid 2 to smithi10729407-5 2020-06-12T02:39:12.393 INFO:tasks.rados.rados.0.smithi107.stdout:10696: finishing write tid 3 to smithi10729407-5 2020-06-12T02:39:12.393 INFO:tasks.rados.rados.0.smithi107.stdout:10696: finishing write tid 5 to smithi10729407-5 2020-06-12T02:39:12.394 INFO:tasks.rados.rados.0.smithi107.stdout:10700: done (1 left) 2020-06-12T02:39:12.394 INFO:tasks.rados.rados.0.smithi107.stdout:10701: delete oid 15 current snap is 0 2020-06-12T02:39:12.394 INFO:tasks.rados.rados.0.smithi107.stdout:10696: finishing write tid 6 to smithi10729407-5 2020-06-12T02:39:12.394 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 1027 (ObjNum 10695 snap 0 seq_num 10695) dirty exists 2020-06-12T02:39:12.394 INFO:tasks.rados.rados.0.smithi107.stdout:10696: left oid 5 (ObjNum 10695 snap 0 seq_num 10695) 2020-06-12T02:39:12.396 INFO:tasks.rados.rados.0.smithi107.stdout:10696: done (1 left) 2020-06-12T02:39:12.396 INFO:tasks.rados.rados.0.smithi107.stdout:10701: done (0 left) 2020-06-12T02:39:12.396 INFO:tasks.rados.rados.0.smithi107.stdout:10702: delete oid 7 current snap is 0 2020-06-12T02:39:12.400 INFO:tasks.rados.rados.0.smithi107.stdout:10702: done (0 left) 2020-06-12T02:39:12.400 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:39:12.400 INFO:tasks.rados.rados.0.smithi107.stdout:10703: seq_num 10702 ranges {1343488=442368,1785856=425984,2211840=638976,2850816=139264} 2020-06-12T02:39:12.410 INFO:tasks.rados.rados.0.smithi107.stdout:10703: writing smithi10729407-18 from 1343488 to 1785856 tid 1 2020-06-12T02:39:12.413 INFO:tasks.rados.rados.0.smithi107.stdout:10703: writing smithi10729407-18 from 1785856 to 2211840 tid 2 2020-06-12T02:39:12.417 INFO:tasks.rados.rados.0.smithi107.stdout:10703: writing smithi10729407-18 from 2211840 to 2850816 tid 3 2020-06-12T02:39:12.419 INFO:tasks.rados.rados.0.smithi107.stdout:10703: writing smithi10729407-18 from 2850816 to 2990080 tid 4 2020-06-12T02:39:12.420 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:39:12.420 INFO:tasks.rados.rados.0.smithi107.stdout:10704: seq_num 10703 ranges {1163264=458752,1622016=712704,2334720=450560,2785280=40960} 2020-06-12T02:39:12.422 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2020-06-12T02:39:12.428 INFO:tasks.rados.rados.0.smithi107.stdout:10704: writing smithi10729407-8 from 1163264 to 1622016 tid 1 2020-06-12T02:39:12.433 INFO:tasks.rados.rados.0.smithi107.stdout:10704: writing smithi10729407-8 from 1622016 to 2334720 tid 2 2020-06-12T02:39:12.436 INFO:tasks.rados.rados.0.smithi107.stdout:10704: writing smithi10729407-8 from 2334720 to 2785280 tid 3 2020-06-12T02:39:12.437 INFO:tasks.rados.rados.0.smithi107.stdout:10704: writing smithi10729407-8 from 2785280 to 2826240 tid 4 2020-06-12T02:39:12.437 INFO:tasks.rados.rados.0.smithi107.stdout:10705: delete oid 13 current snap is 0 2020-06-12T02:39:12.437 INFO:tasks.rados.rados.0.smithi107.stdout:10703: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:12.438 INFO:tasks.rados.rados.0.smithi107.stdout:10703: finishing write tid 2 to smithi10729407-18 2020-06-12T02:39:12.438 INFO:tasks.rados.rados.0.smithi107.stdout:10703: finishing write tid 3 to smithi10729407-18 2020-06-12T02:39:12.438 INFO:tasks.rados.rados.0.smithi107.stdout:10703: finishing write tid 4 to smithi10729407-18 2020-06-12T02:39:12.438 INFO:tasks.rados.rados.0.smithi107.stdout:10703: finishing write tid 6 to smithi10729407-18 2020-06-12T02:39:12.439 INFO:tasks.rados.rados.0.smithi107.stdout:10703: finishing write tid 7 to smithi10729407-18 2020-06-12T02:39:12.439 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1857 (ObjNum 10702 snap 0 seq_num 10702) dirty exists 2020-06-12T02:39:12.439 INFO:tasks.rados.rados.0.smithi107.stdout:10703: left oid 18 (ObjNum 10702 snap 0 seq_num 10702) 2020-06-12T02:39:12.439 INFO:tasks.rados.rados.0.smithi107.stdout:10703: done (2 left) 2020-06-12T02:39:12.440 INFO:tasks.rados.rados.0.smithi107.stdout:10705: done (1 left) 2020-06-12T02:39:12.440 INFO:tasks.rados.rados.0.smithi107.stdout:10706: delete oid 11 current snap is 0 2020-06-12T02:39:12.441 INFO:tasks.rados.rados.0.smithi107.stdout:10704: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:12.442 INFO:tasks.rados.rados.0.smithi107.stdout:10706: done (1 left) 2020-06-12T02:39:12.442 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:39:12.442 INFO:tasks.rados.rados.0.smithi107.stdout:10707: seq_num 10706 ranges {0=507904,507904=409600,917504=622592,1540096=172032} 2020-06-12T02:39:12.445 INFO:tasks.rados.rados.0.smithi107.stdout:10707: writing smithi10729407-19 from 0 to 507904 tid 1 2020-06-12T02:39:12.448 INFO:tasks.rados.rados.0.smithi107.stdout:10707: writing smithi10729407-19 from 507904 to 917504 tid 2 2020-06-12T02:39:12.453 INFO:tasks.rados.rados.0.smithi107.stdout:10707: writing smithi10729407-19 from 917504 to 1540096 tid 3 2020-06-12T02:39:12.454 INFO:tasks.rados.rados.0.smithi107.stdout:10707: writing smithi10729407-19 from 1540096 to 1712128 tid 4 2020-06-12T02:39:12.455 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:39:12.455 INFO:tasks.rados.rados.0.smithi107.stdout:10708: seq_num 10707 ranges {0=524288,524288=696320,1220608=425984,1646592=81920} 2020-06-12T02:39:12.458 INFO:tasks.rados.rados.0.smithi107.stdout:10708: writing smithi10729407-9 from 0 to 524288 tid 1 2020-06-12T02:39:12.463 INFO:tasks.rados.rados.0.smithi107.stdout:10708: writing smithi10729407-9 from 524288 to 1220608 tid 2 2020-06-12T02:39:12.467 INFO:tasks.rados.rados.0.smithi107.stdout:10708: writing smithi10729407-9 from 1220608 to 1646592 tid 3 2020-06-12T02:39:12.467 INFO:tasks.rados.rados.0.smithi107.stdout:10708: writing smithi10729407-9 from 1646592 to 1728512 tid 4 2020-06-12T02:39:12.468 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:39:12.468 INFO:tasks.rados.rados.0.smithi107.stdout:10709: seq_num 10708 ranges {2924544=540672,3465216=581632,4046848=622592} 2020-06-12T02:39:12.476 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:12.476 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:12.487 INFO:tasks.rados.rados.0.smithi107.stdout:10709: writing smithi10729407-5 from 2924544 to 3465216 tid 1 2020-06-12T02:39:12.488 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-06-12T02:39:12.491 INFO:tasks.rados.rados.0.smithi107.stdout:10709: writing smithi10729407-5 from 3465216 to 4046848 tid 2 2020-06-12T02:39:12.491 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:39:12.489+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:39:12.495 INFO:tasks.rados.rados.0.smithi107.stdout:10709: writing smithi10729407-5 from 4046848 to 4669440 tid 3 2020-06-12T02:39:12.495 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:39:12.496 INFO:tasks.rados.rados.0.smithi107.stdout:10710: seq_num 10709 ranges {0=557056,557056=466944,1024000=737280} 2020-06-12T02:39:12.499 INFO:tasks.rados.rados.0.smithi107.stdout:10710: writing smithi10729407-4 from 0 to 557056 tid 1 2020-06-12T02:39:12.501 INFO:tasks.rados.rados.0.smithi107.stdout:10710: writing smithi10729407-4 from 557056 to 1024000 tid 2 2020-06-12T02:39:12.506 INFO:tasks.rados.rados.0.smithi107.stdout:10710: writing smithi10729407-4 from 1024000 to 1761280 tid 3 2020-06-12T02:39:12.507 INFO:tasks.rados.rados.0.smithi107.stdout:10711: delete oid 2 current snap is 0 2020-06-12T02:39:12.507 INFO:tasks.rados.rados.0.smithi107.stdout:10704: finishing write tid 2 to smithi10729407-8 2020-06-12T02:39:12.507 INFO:tasks.rados.rados.0.smithi107.stdout:10704: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:12.507 INFO:tasks.rados.rados.0.smithi107.stdout:10704: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:12.507 INFO:tasks.rados.rados.0.smithi107.stdout:10704: finishing write tid 6 to smithi10729407-8 2020-06-12T02:39:12.508 INFO:tasks.rados.rados.0.smithi107.stdout:10704: finishing write tid 7 to smithi10729407-8 2020-06-12T02:39:12.508 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 1999 (ObjNum 10703 snap 0 seq_num 10703) dirty exists 2020-06-12T02:39:12.508 INFO:tasks.rados.rados.0.smithi107.stdout:10704: left oid 8 (ObjNum 10703 snap 0 seq_num 10703) 2020-06-12T02:39:12.508 INFO:tasks.rados.rados.0.smithi107.stdout:10707: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:12.509 INFO:tasks.rados.rados.0.smithi107.stdout:10707: finishing write tid 2 to smithi10729407-19 2020-06-12T02:39:12.509 INFO:tasks.rados.rados.0.smithi107.stdout:10707: finishing write tid 3 to smithi10729407-19 2020-06-12T02:39:12.509 INFO:tasks.rados.rados.0.smithi107.stdout:10707: finishing write tid 4 to smithi10729407-19 2020-06-12T02:39:12.509 INFO:tasks.rados.rados.0.smithi107.stdout:10707: finishing write tid 6 to smithi10729407-19 2020-06-12T02:39:12.509 INFO:tasks.rados.rados.0.smithi107.stdout:10708: finishing write tid 1 to smithi10729407-9 2020-06-12T02:39:12.510 INFO:tasks.rados.rados.0.smithi107.stdout:10707: finishing write tid 7 to smithi10729407-19 2020-06-12T02:39:12.510 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1971 (ObjNum 10706 snap 0 seq_num 10706) dirty exists 2020-06-12T02:39:12.510 INFO:tasks.rados.rados.0.smithi107.stdout:10707: left oid 19 (ObjNum 10706 snap 0 seq_num 10706) 2020-06-12T02:39:12.510 INFO:tasks.rados.rados.0.smithi107.stdout:10708: finishing write tid 2 to smithi10729407-9 2020-06-12T02:39:12.510 INFO:tasks.rados.rados.0.smithi107.stdout:10708: finishing write tid 3 to smithi10729407-9 2020-06-12T02:39:12.510 INFO:tasks.rados.rados.0.smithi107.stdout:10708: finishing write tid 4 to smithi10729407-9 2020-06-12T02:39:12.511 INFO:tasks.rados.rados.0.smithi107.stdout:10708: finishing write tid 6 to smithi10729407-9 2020-06-12T02:39:12.511 INFO:tasks.rados.rados.0.smithi107.stdout:10708: finishing write tid 7 to smithi10729407-9 2020-06-12T02:39:12.511 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1833 (ObjNum 10707 snap 0 seq_num 10707) dirty exists 2020-06-12T02:39:12.511 INFO:tasks.rados.rados.0.smithi107.stdout:10708: left oid 9 (ObjNum 10707 snap 0 seq_num 10707) 2020-06-12T02:39:12.511 INFO:tasks.rados.rados.0.smithi107.stdout:10709: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:12.512 INFO:tasks.rados.rados.0.smithi107.stdout:10709: finishing write tid 2 to smithi10729407-5 2020-06-12T02:39:12.512 INFO:tasks.rados.rados.0.smithi107.stdout:10709: finishing write tid 3 to smithi10729407-5 2020-06-12T02:39:12.512 INFO:tasks.rados.rados.0.smithi107.stdout:10709: finishing write tid 5 to smithi10729407-5 2020-06-12T02:39:12.513 INFO:tasks.rados.rados.0.smithi107.stdout:10709: finishing write tid 6 to smithi10729407-5 2020-06-12T02:39:12.513 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 1031 (ObjNum 10708 snap 0 seq_num 10708) dirty exists 2020-06-12T02:39:12.513 INFO:tasks.rados.rados.0.smithi107.stdout:10709: left oid 5 (ObjNum 10708 snap 0 seq_num 10708) 2020-06-12T02:39:12.514 INFO:tasks.rados.rados.0.smithi107.stdout:10704: done (5 left) 2020-06-12T02:39:12.514 INFO:tasks.rados.rados.0.smithi107.stdout:10707: done (4 left) 2020-06-12T02:39:12.514 INFO:tasks.rados.rados.0.smithi107.stdout:10708: done (3 left) 2020-06-12T02:39:12.514 INFO:tasks.rados.rados.0.smithi107.stdout:10709: done (2 left) 2020-06-12T02:39:12.515 INFO:tasks.rados.rados.0.smithi107.stdout:10711: done (1 left) 2020-06-12T02:39:12.515 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:39:12.515 INFO:tasks.rados.rados.0.smithi107.stdout:10712: seq_num 10711 ranges {0=598016,598016=630784,1228800=409600,1638400=163840} 2020-06-12T02:39:12.515 INFO:tasks.rados.rados.0.smithi107.stdout:10712: writing smithi10729407-2 from 0 to 598016 tid 1 2020-06-12T02:39:12.517 INFO:tasks.rados.rados.0.smithi107.stdout:10712: writing smithi10729407-2 from 598016 to 1228800 tid 2 2020-06-12T02:39:12.520 INFO:tasks.rados.rados.0.smithi107.stdout:10712: writing smithi10729407-2 from 1228800 to 1638400 tid 3 2020-06-12T02:39:12.521 INFO:tasks.rados.rados.0.smithi107.stdout:10712: writing smithi10729407-2 from 1638400 to 1802240 tid 4 2020-06-12T02:39:12.521 INFO:tasks.rados.rados.0.smithi107.stdout:10713: delete oid 13 current snap is 0 2020-06-12T02:39:12.521 INFO:tasks.rados.rados.0.smithi107.stdout:10710: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:12.521 INFO:tasks.rados.rados.0.smithi107.stdout:10710: finishing write tid 2 to smithi10729407-4 2020-06-12T02:39:12.522 INFO:tasks.rados.rados.0.smithi107.stdout:10710: finishing write tid 3 to smithi10729407-4 2020-06-12T02:39:12.522 INFO:tasks.rados.rados.0.smithi107.stdout:10710: finishing write tid 5 to smithi10729407-4 2020-06-12T02:39:12.523 INFO:tasks.rados.rados.0.smithi107.stdout:10713: done (2 left) 2020-06-12T02:39:12.524 INFO:tasks.rados.rados.0.smithi107.stdout:10714: delete oid 18 current snap is 0 2020-06-12T02:39:12.524 INFO:tasks.rados.rados.0.smithi107.stdout:10710: finishing write tid 6 to smithi10729407-4 2020-06-12T02:39:12.524 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 1995 (ObjNum 10709 snap 0 seq_num 10709) dirty exists 2020-06-12T02:39:12.524 INFO:tasks.rados.rados.0.smithi107.stdout:10710: left oid 4 (ObjNum 10709 snap 0 seq_num 10709) 2020-06-12T02:39:12.526 INFO:tasks.rados.rados.0.smithi107.stdout:10712: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:12.529 INFO:tasks.rados.rados.0.smithi107.stdout:10710: done (2 left) 2020-06-12T02:39:12.529 INFO:tasks.rados.rados.0.smithi107.stdout:10714: done (1 left) 2020-06-12T02:39:12.529 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:39:12.529 INFO:tasks.rados.rados.0.smithi107.stdout:10715: seq_num 10714 ranges {0=647168,647168=737280,1384448=466944} 2020-06-12T02:39:12.533 INFO:tasks.rados.rados.0.smithi107.stdout:10715: writing smithi10729407-7 from 0 to 647168 tid 1 2020-06-12T02:39:12.537 INFO:tasks.rados.rados.0.smithi107.stdout:10715: writing smithi10729407-7 from 647168 to 1384448 tid 2 2020-06-12T02:39:12.540 INFO:tasks.rados.rados.0.smithi107.stdout:10715: writing smithi10729407-7 from 1384448 to 1851392 tid 3 2020-06-12T02:39:12.541 INFO:tasks.rados.rados.0.smithi107.stdout:10716: delete oid 3 current snap is 0 2020-06-12T02:39:12.541 INFO:tasks.rados.rados.0.smithi107.stdout:10712: finishing write tid 2 to smithi10729407-2 2020-06-12T02:39:12.541 INFO:tasks.rados.rados.0.smithi107.stdout:10712: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:12.541 INFO:tasks.rados.rados.0.smithi107.stdout:10712: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:12.541 INFO:tasks.rados.rados.0.smithi107.stdout:10712: finishing write tid 6 to smithi10729407-2 2020-06-12T02:39:12.542 INFO:tasks.rados.rados.0.smithi107.stdout:10712: finishing write tid 7 to smithi10729407-2 2020-06-12T02:39:12.542 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 1002 (ObjNum 10711 snap 0 seq_num 10711) dirty exists 2020-06-12T02:39:12.542 INFO:tasks.rados.rados.0.smithi107.stdout:10712: left oid 2 (ObjNum 10711 snap 0 seq_num 10711) 2020-06-12T02:39:12.542 INFO:tasks.rados.rados.0.smithi107.stdout:10715: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:12.543 INFO:tasks.rados.rados.0.smithi107.stdout:10712: done (2 left) 2020-06-12T02:39:12.543 INFO:tasks.rados.rados.0.smithi107.stdout:10716: done (1 left) 2020-06-12T02:39:12.543 INFO:tasks.rados.rados.0.smithi107.stdout:10717: delete oid 2 current snap is 0 2020-06-12T02:39:12.559 INFO:tasks.rados.rados.0.smithi107.stdout:10717: done (1 left) 2020-06-12T02:39:12.559 INFO:tasks.rados.rados.0.smithi107.stdout:10718: delete oid 19 current snap is 0 2020-06-12T02:39:12.559 INFO:tasks.rados.rados.0.smithi107.stdout:10715: finishing write tid 2 to smithi10729407-7 2020-06-12T02:39:12.560 INFO:tasks.rados.rados.0.smithi107.stdout:10718: done (1 left) 2020-06-12T02:39:12.560 INFO:tasks.rados.rados.0.smithi107.stdout:10719: delete oid 1 current snap is 0 2020-06-12T02:39:12.560 INFO:tasks.rados.rados.0.smithi107.stdout:10715: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:12.560 INFO:tasks.rados.rados.0.smithi107.stdout:10715: finishing write tid 5 to smithi10729407-7 2020-06-12T02:39:12.560 INFO:tasks.rados.rados.0.smithi107.stdout:10715: finishing write tid 6 to smithi10729407-7 2020-06-12T02:39:12.560 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1770 (ObjNum 10714 snap 0 seq_num 10714) dirty exists 2020-06-12T02:39:12.561 INFO:tasks.rados.rados.0.smithi107.stdout:10715: left oid 7 (ObjNum 10714 snap 0 seq_num 10714) 2020-06-12T02:39:12.561 INFO:tasks.rados.rados.0.smithi107.stdout:10715: done (1 left) 2020-06-12T02:39:12.561 INFO:tasks.rados.rados.0.smithi107.stdout:10719: done (0 left) 2020-06-12T02:39:12.561 INFO:tasks.rados.rados.0.smithi107.stdout:10720: delete oid 18 current snap is 0 2020-06-12T02:39:12.561 INFO:tasks.rados.rados.0.smithi107.stdout:10720: done (0 left) 2020-06-12T02:39:12.562 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:39:12.562 INFO:tasks.rados.rados.0.smithi107.stdout:10721: seq_num 10720 ranges {0=745472,745472=491520,1236992=573440,1810432=139264} 2020-06-12T02:39:12.562 INFO:tasks.rados.rados.0.smithi107.stdout:10721: writing smithi10729407-17 from 0 to 745472 tid 1 2020-06-12T02:39:12.564 INFO:tasks.rados.rados.0.smithi107.stdout:10721: writing smithi10729407-17 from 745472 to 1236992 tid 2 2020-06-12T02:39:12.567 INFO:tasks.rados.rados.0.smithi107.stdout:10721: writing smithi10729407-17 from 1236992 to 1810432 tid 3 2020-06-12T02:39:12.568 INFO:tasks.rados.rados.0.smithi107.stdout:10721: writing smithi10729407-17 from 1810432 to 1949696 tid 4 2020-06-12T02:39:12.569 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:39:12.569 INFO:tasks.rados.rados.0.smithi107.stdout:10722: seq_num 10721 ranges {1499136=761856,2260992=778240,3039232=425984} 2020-06-12T02:39:12.581 INFO:tasks.rados.rados.0.smithi107.stdout:10722: writing smithi10729407-12 from 1499136 to 2260992 tid 1 2020-06-12T02:39:12.586 INFO:tasks.rados.rados.0.smithi107.stdout:10722: writing smithi10729407-12 from 2260992 to 3039232 tid 2 2020-06-12T02:39:12.588 INFO:tasks.rados.rados.0.smithi107.stdout:10722: writing smithi10729407-12 from 3039232 to 3465216 tid 3 2020-06-12T02:39:12.589 INFO:tasks.rados.rados.0.smithi107.stdout:10723: delete oid 18 current snap is 0 2020-06-12T02:39:12.589 INFO:tasks.rados.rados.0.smithi107.stdout:10721: finishing write tid 1 to smithi10729407-17 2020-06-12T02:39:12.589 INFO:tasks.rados.rados.0.smithi107.stdout:10721: finishing write tid 2 to smithi10729407-17 2020-06-12T02:39:12.589 INFO:tasks.rados.rados.0.smithi107.stdout:10721: finishing write tid 3 to smithi10729407-17 2020-06-12T02:39:12.590 INFO:tasks.rados.rados.0.smithi107.stdout:10721: finishing write tid 4 to smithi10729407-17 2020-06-12T02:39:12.590 INFO:tasks.rados.rados.0.smithi107.stdout:10721: finishing write tid 6 to smithi10729407-17 2020-06-12T02:39:12.590 INFO:tasks.rados.rados.0.smithi107.stdout:10721: finishing write tid 7 to smithi10729407-17 2020-06-12T02:39:12.590 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 2000 (ObjNum 10720 snap 0 seq_num 10720) dirty exists 2020-06-12T02:39:12.590 INFO:tasks.rados.rados.0.smithi107.stdout:10721: left oid 17 (ObjNum 10720 snap 0 seq_num 10720) 2020-06-12T02:39:12.591 INFO:tasks.rados.rados.0.smithi107.stdout:10722: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:12.591 INFO:tasks.rados.rados.0.smithi107.stdout:10721: done (2 left) 2020-06-12T02:39:12.592 INFO:tasks.rados.rados.0.smithi107.stdout:10723: done (1 left) 2020-06-12T02:39:12.592 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:39:12.592 INFO:tasks.rados.rados.0.smithi107.stdout:10724: seq_num 10723 ranges {0=794624,794624=548864,1343488=655360} 2020-06-12T02:39:12.593 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:39:12.589+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:39:12.600 INFO:tasks.rados.rados.0.smithi107.stdout:10724: writing smithi10729407-18 from 0 to 794624 tid 1 2020-06-12T02:39:12.600 INFO:tasks.rados.rados.0.smithi107.stdout:10724: writing smithi10729407-18 from 794624 to 1343488 tid 2 2020-06-12T02:39:12.605 INFO:tasks.rados.rados.0.smithi107.stdout:10724: writing smithi10729407-18 from 1343488 to 1998848 tid 3 2020-06-12T02:39:12.605 INFO:tasks.rados.rados.0.smithi107.stdout:10725: delete oid 3 current snap is 0 2020-06-12T02:39:12.605 INFO:tasks.rados.rados.0.smithi107.stdout:10722: finishing write tid 2 to smithi10729407-12 2020-06-12T02:39:12.605 INFO:tasks.rados.rados.0.smithi107.stdout:10722: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:12.606 INFO:tasks.rados.rados.0.smithi107.stdout:10722: finishing write tid 5 to smithi10729407-12 2020-06-12T02:39:12.606 INFO:tasks.rados.rados.0.smithi107.stdout:10722: finishing write tid 6 to smithi10729407-12 2020-06-12T02:39:12.606 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1976 (ObjNum 10721 snap 0 seq_num 10721) dirty exists 2020-06-12T02:39:12.606 INFO:tasks.rados.rados.0.smithi107.stdout:10722: left oid 12 (ObjNum 10721 snap 0 seq_num 10721) 2020-06-12T02:39:12.607 INFO:tasks.rados.rados.0.smithi107.stdout:10724: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:12.608 INFO:tasks.rados.rados.0.smithi107.stdout:10722: done (2 left) 2020-06-12T02:39:12.608 INFO:tasks.rados.rados.0.smithi107.stdout:10725: done (1 left) 2020-06-12T02:39:12.608 INFO:tasks.rados.rados.0.smithi107.stdout:10726: delete oid 16 current snap is 0 2020-06-12T02:39:12.612 INFO:tasks.rados.rados.0.smithi107.stdout:10726: done (1 left) 2020-06-12T02:39:12.613 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:39:12.613 INFO:tasks.rados.rados.0.smithi107.stdout:10727: seq_num 10726 ranges {1949696=442368,2392064=598016,2990080=524288,3514368=483328} 2020-06-12T02:39:12.626 INFO:tasks.rados.rados.0.smithi107.stdout:10727: writing smithi10729407-17 from 1949696 to 2392064 tid 1 2020-06-12T02:39:12.629 INFO:tasks.rados.rados.0.smithi107.stdout:10727: writing smithi10729407-17 from 2392064 to 2990080 tid 2 2020-06-12T02:39:12.632 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:12.632 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:12.633 INFO:tasks.rados.rados.0.smithi107.stdout:10727: writing smithi10729407-17 from 2990080 to 3514368 tid 3 2020-06-12T02:39:12.636 INFO:tasks.rados.rados.0.smithi107.stdout:10727: writing smithi10729407-17 from 3514368 to 3997696 tid 4 2020-06-12T02:39:12.637 INFO:tasks.rados.rados.0.smithi107.stdout:10728: delete oid 2 current snap is 0 2020-06-12T02:39:12.637 INFO:tasks.rados.rados.0.smithi107.stdout:10724: finishing write tid 2 to smithi10729407-18 2020-06-12T02:39:12.637 INFO:tasks.rados.rados.0.smithi107.stdout:10724: finishing write tid 3 to smithi10729407-18 2020-06-12T02:39:12.637 INFO:tasks.rados.rados.0.smithi107.stdout:10724: finishing write tid 5 to smithi10729407-18 2020-06-12T02:39:12.638 INFO:tasks.rados.rados.0.smithi107.stdout:10724: finishing write tid 6 to smithi10729407-18 2020-06-12T02:39:12.638 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1864 (ObjNum 10723 snap 0 seq_num 10723) dirty exists 2020-06-12T02:39:12.638 INFO:tasks.rados.rados.0.smithi107.stdout:10724: left oid 18 (ObjNum 10723 snap 0 seq_num 10723) 2020-06-12T02:39:12.638 INFO:tasks.rados.rados.0.smithi107.stdout:10727: finishing write tid 1 to smithi10729407-17 2020-06-12T02:39:12.639 INFO:tasks.rados.rados.0.smithi107.stdout:10724: done (2 left) 2020-06-12T02:39:12.639 INFO:tasks.rados.rados.0.smithi107.stdout:10728: done (1 left) 2020-06-12T02:39:12.640 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:39:12.640 INFO:tasks.rados.rados.0.smithi107.stdout:10729: seq_num 10728 ranges {1728512=475136,2203648=417792,2621440=540672,3162112=622592,3784704=24576} 2020-06-12T02:39:12.643 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-06-12T02:39:12.651 INFO:tasks.rados.rados.0.smithi107.stdout:10729: writing smithi10729407-9 from 1728512 to 2203648 tid 1 2020-06-12T02:39:12.653 INFO:tasks.rados.rados.0.smithi107.stdout:10729: writing smithi10729407-9 from 2203648 to 2621440 tid 2 2020-06-12T02:39:12.657 INFO:tasks.rados.rados.0.smithi107.stdout:10729: writing smithi10729407-9 from 2621440 to 3162112 tid 3 2020-06-12T02:39:12.661 INFO:tasks.rados.rados.0.smithi107.stdout:10729: writing smithi10729407-9 from 3162112 to 3784704 tid 4 2020-06-12T02:39:12.661 INFO:tasks.rados.rados.0.smithi107.stdout:10729: writing smithi10729407-9 from 3784704 to 3809280 tid 5 2020-06-12T02:39:12.661 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:39:12.662 INFO:tasks.rados.rados.0.smithi107.stdout:10730: seq_num 10729 ranges {0=491520,491520=704512,1196032=696320,1892352=204800} 2020-06-12T02:39:12.664 INFO:tasks.rados.rados.0.smithi107.stdout:10730: writing smithi10729407-16 from 0 to 491520 tid 1 2020-06-12T02:39:12.668 INFO:tasks.rados.rados.0.smithi107.stdout:10730: writing smithi10729407-16 from 491520 to 1196032 tid 2 2020-06-12T02:39:12.673 INFO:tasks.rados.rados.0.smithi107.stdout:10730: writing smithi10729407-16 from 1196032 to 1892352 tid 3 2020-06-12T02:39:12.675 INFO:tasks.rados.rados.0.smithi107.stdout:10730: writing smithi10729407-16 from 1892352 to 2097152 tid 4 2020-06-12T02:39:12.676 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:39:12.676 INFO:tasks.rados.rados.0.smithi107.stdout:10731: seq_num 10730 ranges {0=516096,516096=581632,1097728=499712,1597440=450560,2048000=73728} 2020-06-12T02:39:12.679 INFO:tasks.rados.rados.0.smithi107.stdout:10731: writing smithi10729407-10 from 0 to 516096 tid 1 2020-06-12T02:39:12.681 INFO:tasks.rados.rados.0.smithi107.stdout:10731: writing smithi10729407-10 from 516096 to 1097728 tid 2 2020-06-12T02:39:12.685 INFO:tasks.rados.rados.0.smithi107.stdout:10731: writing smithi10729407-10 from 1097728 to 1597440 tid 3 2020-06-12T02:39:12.688 INFO:tasks.rados.rados.0.smithi107.stdout:10731: writing smithi10729407-10 from 1597440 to 2048000 tid 4 2020-06-12T02:39:12.689 INFO:tasks.rados.rados.0.smithi107.stdout:10731: writing smithi10729407-10 from 2048000 to 2121728 tid 5 2020-06-12T02:39:12.689 INFO:tasks.rados.rados.0.smithi107.stdout:10732: delete oid 8 current snap is 0 2020-06-12T02:39:12.690 INFO:tasks.rados.rados.0.smithi107.stdout:10727: finishing write tid 2 to smithi10729407-17 2020-06-12T02:39:12.690 INFO:tasks.rados.rados.0.smithi107.stdout:10727: finishing write tid 3 to smithi10729407-17 2020-06-12T02:39:12.690 INFO:tasks.rados.rados.0.smithi107.stdout:10727: finishing write tid 4 to smithi10729407-17 2020-06-12T02:39:12.690 INFO:tasks.rados.rados.0.smithi107.stdout:10727: finishing write tid 6 to smithi10729407-17 2020-06-12T02:39:12.690 INFO:tasks.rados.rados.0.smithi107.stdout:10727: finishing write tid 7 to smithi10729407-17 2020-06-12T02:39:12.691 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 2005 (ObjNum 10726 snap 0 seq_num 10726) dirty exists 2020-06-12T02:39:12.691 INFO:tasks.rados.rados.0.smithi107.stdout:10727: left oid 17 (ObjNum 10726 snap 0 seq_num 10726) 2020-06-12T02:39:12.691 INFO:tasks.rados.rados.0.smithi107.stdout:10729: finishing write tid 1 to smithi10729407-9 2020-06-12T02:39:12.691 INFO:tasks.rados.rados.0.smithi107.stdout:10729: finishing write tid 2 to smithi10729407-9 2020-06-12T02:39:12.691 INFO:tasks.rados.rados.0.smithi107.stdout:10729: finishing write tid 3 to smithi10729407-9 2020-06-12T02:39:12.691 INFO:tasks.rados.rados.0.smithi107.stdout:10729: finishing write tid 4 to smithi10729407-9 2020-06-12T02:39:12.692 INFO:tasks.rados.rados.0.smithi107.stdout:10729: finishing write tid 5 to smithi10729407-9 2020-06-12T02:39:12.692 INFO:tasks.rados.rados.0.smithi107.stdout:10729: finishing write tid 7 to smithi10729407-9 2020-06-12T02:39:12.692 INFO:tasks.rados.rados.0.smithi107.stdout:10730: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:12.692 INFO:tasks.rados.rados.0.smithi107.stdout:10729: finishing write tid 8 to smithi10729407-9 2020-06-12T02:39:12.693 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1839 (ObjNum 10728 snap 0 seq_num 10728) dirty exists 2020-06-12T02:39:12.693 INFO:tasks.rados.rados.0.smithi107.stdout:10729: left oid 9 (ObjNum 10728 snap 0 seq_num 10728) 2020-06-12T02:39:12.693 INFO:tasks.rados.rados.0.smithi107.stdout:10731: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:12.694 INFO:tasks.rados.rados.0.smithi107.stdout:10730: finishing write tid 2 to smithi10729407-16 2020-06-12T02:39:12.694 INFO:tasks.rados.rados.0.smithi107.stdout:10730: finishing write tid 3 to smithi10729407-16 2020-06-12T02:39:12.695 INFO:tasks.rados.rados.0.smithi107.stdout:10730: finishing write tid 4 to smithi10729407-16 2020-06-12T02:39:12.695 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:39:12.692+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:39:12.696 INFO:tasks.rados.rados.0.smithi107.stdout:10727: done (4 left) 2020-06-12T02:39:12.696 INFO:tasks.rados.rados.0.smithi107.stdout:10729: done (3 left) 2020-06-12T02:39:12.696 INFO:tasks.rados.rados.0.smithi107.stdout:10732: done (2 left) 2020-06-12T02:39:12.696 INFO:tasks.rados.rados.0.smithi107.stdout:10733: delete oid 14 current snap is 0 2020-06-12T02:39:12.696 INFO:tasks.rados.rados.0.smithi107.stdout:10731: finishing write tid 2 to smithi10729407-10 2020-06-12T02:39:12.698 INFO:tasks.rados.rados.0.smithi107.stdout:10730: finishing write tid 6 to smithi10729407-16 2020-06-12T02:39:12.698 INFO:tasks.rados.rados.0.smithi107.stdout:10731: finishing write tid 3 to smithi10729407-10 2020-06-12T02:39:12.699 INFO:tasks.rados.rados.0.smithi107.stdout:10730: finishing write tid 7 to smithi10729407-16 2020-06-12T02:39:12.699 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1917 (ObjNum 10729 snap 0 seq_num 10729) dirty exists 2020-06-12T02:39:12.699 INFO:tasks.rados.rados.0.smithi107.stdout:10730: left oid 16 (ObjNum 10729 snap 0 seq_num 10729) 2020-06-12T02:39:12.699 INFO:tasks.rados.rados.0.smithi107.stdout:10730: done (2 left) 2020-06-12T02:39:12.700 INFO:tasks.rados.rados.0.smithi107.stdout:10733: done (1 left) 2020-06-12T02:39:12.700 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:39:12.700 INFO:tasks.rados.rados.0.smithi107.stdout:10734: seq_num 10733 ranges {1851392=565248,2416640=638976,3055616=679936,3735552=286720} 2020-06-12T02:39:12.712 INFO:tasks.rados.rados.0.smithi107.stdout:10734: writing smithi10729407-7 from 1851392 to 2416640 tid 1 2020-06-12T02:39:12.716 INFO:tasks.rados.rados.0.smithi107.stdout:10734: writing smithi10729407-7 from 2416640 to 3055616 tid 2 2020-06-12T02:39:12.720 INFO:tasks.rados.rados.0.smithi107.stdout:10734: writing smithi10729407-7 from 3055616 to 3735552 tid 3 2020-06-12T02:39:12.722 INFO:tasks.rados.rados.0.smithi107.stdout:10734: writing smithi10729407-7 from 3735552 to 4022272 tid 4 2020-06-12T02:39:12.722 INFO:tasks.rados.rados.0.smithi107.stdout:10735: delete oid 17 current snap is 0 2020-06-12T02:39:12.722 INFO:tasks.rados.rados.0.smithi107.stdout:10731: finishing write tid 4 to smithi10729407-10 2020-06-12T02:39:12.722 INFO:tasks.rados.rados.0.smithi107.stdout:10731: finishing write tid 5 to smithi10729407-10 2020-06-12T02:39:12.722 INFO:tasks.rados.rados.0.smithi107.stdout:10731: finishing write tid 7 to smithi10729407-10 2020-06-12T02:39:12.723 INFO:tasks.rados.rados.0.smithi107.stdout:10731: finishing write tid 8 to smithi10729407-10 2020-06-12T02:39:12.723 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 783 (ObjNum 10730 snap 0 seq_num 10730) dirty exists 2020-06-12T02:39:12.723 INFO:tasks.rados.rados.0.smithi107.stdout:10731: left oid 10 (ObjNum 10730 snap 0 seq_num 10730) 2020-06-12T02:39:12.726 INFO:tasks.rados.rados.0.smithi107.stdout:10734: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:12.729 INFO:tasks.rados.rados.0.smithi107.stdout:10731: done (2 left) 2020-06-12T02:39:12.729 INFO:tasks.rados.rados.0.smithi107.stdout:10735: done (1 left) 2020-06-12T02:39:12.729 INFO:tasks.rados.rados.0.smithi107.stdout:10736: delete oid 17 current snap is 0 2020-06-12T02:39:12.733 INFO:tasks.rados.rados.0.smithi107.stdout:10734: finishing write tid 2 to smithi10729407-7 2020-06-12T02:39:12.733 INFO:tasks.rados.rados.0.smithi107.stdout:10734: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:12.734 INFO:tasks.rados.rados.0.smithi107.stdout:10734: finishing write tid 4 to smithi10729407-7 2020-06-12T02:39:12.734 INFO:tasks.rados.rados.0.smithi107.stdout:10734: finishing write tid 6 to smithi10729407-7 2020-06-12T02:39:12.734 INFO:tasks.rados.rados.0.smithi107.stdout:10734: finishing write tid 7 to smithi10729407-7 2020-06-12T02:39:12.735 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1775 (ObjNum 10733 snap 0 seq_num 10733) dirty exists 2020-06-12T02:39:12.735 INFO:tasks.rados.rados.0.smithi107.stdout:10734: left oid 7 (ObjNum 10733 snap 0 seq_num 10733) 2020-06-12T02:39:12.737 INFO:tasks.rados.rados.0.smithi107.stdout:10734: done (1 left) 2020-06-12T02:39:12.737 INFO:tasks.rados.rados.0.smithi107.stdout:10736: done (0 left) 2020-06-12T02:39:12.738 INFO:tasks.rados.rados.0.smithi107.stdout:10737: delete oid 7 current snap is 0 2020-06-12T02:39:12.739 INFO:tasks.rados.rados.0.smithi107.stdout:10737: done (0 left) 2020-06-12T02:39:12.739 INFO:tasks.rados.rados.0.smithi107.stdout:10738: delete oid 10 current snap is 0 2020-06-12T02:39:12.743 INFO:tasks.rados.rados.0.smithi107.stdout:10738: done (0 left) 2020-06-12T02:39:12.743 INFO:tasks.rados.rados.0.smithi107.stdout:10739: delete oid 6 current snap is 0 2020-06-12T02:39:12.746 INFO:tasks.rados.rados.0.smithi107.stdout:10739: done (0 left) 2020-06-12T02:39:12.746 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:39:12.746 INFO:tasks.rados.rados.0.smithi107.stdout:10740: seq_num 10739 ranges {1761280=663552,2424832=794624,3219456=671744,3891200=139264} 2020-06-12T02:39:12.766 INFO:tasks.rados.rados.0.smithi107.stdout:10740: writing smithi10729407-4 from 1761280 to 2424832 tid 1 2020-06-12T02:39:12.766 INFO:tasks.rados.rados.0.smithi107.stdout:10740: writing smithi10729407-4 from 2424832 to 3219456 tid 2 2020-06-12T02:39:12.768 INFO:tasks.rados.rados.0.smithi107.stdout:10740: writing smithi10729407-4 from 3219456 to 3891200 tid 3 2020-06-12T02:39:12.769 INFO:tasks.rados.rados.0.smithi107.stdout:10740: writing smithi10729407-4 from 3891200 to 4030464 tid 4 2020-06-12T02:39:12.769 INFO:tasks.rados.rados.0.smithi107.stdout:10741: delete oid 14 current snap is 0 2020-06-12T02:39:12.769 INFO:tasks.rados.rados.0.smithi107.stdout:10740: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:12.772 INFO:tasks.rados.rados.0.smithi107.stdout:10741: done (1 left) 2020-06-12T02:39:12.772 INFO:tasks.rados.rados.0.smithi107.stdout:10742: delete oid 18 current snap is 0 2020-06-12T02:39:12.776 INFO:tasks.rados.rados.0.smithi107.stdout:10740: finishing write tid 2 to smithi10729407-4 2020-06-12T02:39:12.777 INFO:tasks.rados.rados.0.smithi107.stdout:10740: finishing write tid 3 to smithi10729407-4 2020-06-12T02:39:12.778 INFO:tasks.rados.rados.0.smithi107.stdout:10740: finishing write tid 4 to smithi10729407-4 2020-06-12T02:39:12.778 INFO:tasks.rados.rados.0.smithi107.stdout:10742: done (1 left) 2020-06-12T02:39:12.778 INFO:tasks.rados.rados.0.smithi107.stdout:10743: delete oid 16 current snap is 0 2020-06-12T02:39:12.780 INFO:tasks.rados.rados.0.smithi107.stdout:10740: finishing write tid 6 to smithi10729407-4 2020-06-12T02:39:12.782 INFO:tasks.rados.rados.0.smithi107.stdout:10740: finishing write tid 7 to smithi10729407-4 2020-06-12T02:39:12.783 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 2012 (ObjNum 10739 snap 0 seq_num 10739) dirty exists 2020-06-12T02:39:12.783 INFO:tasks.rados.rados.0.smithi107.stdout:10740: left oid 4 (ObjNum 10739 snap 0 seq_num 10739) 2020-06-12T02:39:12.783 INFO:tasks.rados.rados.0.smithi107.stdout:10740: done (1 left) 2020-06-12T02:39:12.783 INFO:tasks.rados.rados.0.smithi107.stdout:10743: done (0 left) 2020-06-12T02:39:12.784 INFO:tasks.rados.rados.0.smithi107.stdout:10744: delete oid 17 current snap is 0 2020-06-12T02:39:12.788 INFO:tasks.rados.rados.0.smithi107.stdout:10744: done (0 left) 2020-06-12T02:39:12.788 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:39:12.788 INFO:tasks.rados.rados.0.smithi107.stdout:10745: seq_num 10744 ranges {0=745472,745472=663552,1409024=458752,1867776=483328} 2020-06-12T02:39:12.792 INFO:tasks.rados.rados.0.smithi107.stdout:10745: writing smithi10729407-15 from 0 to 745472 tid 1 2020-06-12T02:39:12.795 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:39:12.794+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:39:12.796 INFO:tasks.rados.rados.0.smithi107.stdout:10745: writing smithi10729407-15 from 745472 to 1409024 tid 2 2020-06-12T02:39:12.797 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:12.797 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:12.799 INFO:tasks.rados.rados.0.smithi107.stdout:10745: writing smithi10729407-15 from 1409024 to 1867776 tid 3 2020-06-12T02:39:12.802 INFO:tasks.rados.rados.0.smithi107.stdout:10745: writing smithi10729407-15 from 1867776 to 2351104 tid 4 2020-06-12T02:39:12.802 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:39:12.802 INFO:tasks.rados.rados.0.smithi107.stdout:10746: seq_num 10745 ranges {0=761856,761856=548864,1310720=614400,1925120=442368} 2020-06-12T02:39:12.806 INFO:tasks.rados.rados.0.smithi107.stdout:10746: writing smithi10729407-2 from 0 to 761856 tid 1 2020-06-12T02:39:12.807 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-06-12T02:39:12.810 INFO:tasks.rados.rados.0.smithi107.stdout:10746: writing smithi10729407-2 from 761856 to 1310720 tid 2 2020-06-12T02:39:12.814 INFO:tasks.rados.rados.0.smithi107.stdout:10746: writing smithi10729407-2 from 1310720 to 1925120 tid 3 2020-06-12T02:39:12.817 INFO:tasks.rados.rados.0.smithi107.stdout:10746: writing smithi10729407-2 from 1925120 to 2367488 tid 4 2020-06-12T02:39:12.817 INFO:tasks.rados.rados.0.smithi107.stdout:10747: delete oid 8 current snap is 0 2020-06-12T02:39:12.817 INFO:tasks.rados.rados.0.smithi107.stdout:10745: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:12.817 INFO:tasks.rados.rados.0.smithi107.stdout:10745: finishing write tid 2 to smithi10729407-15 2020-06-12T02:39:12.818 INFO:tasks.rados.rados.0.smithi107.stdout:10745: finishing write tid 3 to smithi10729407-15 2020-06-12T02:39:12.818 INFO:tasks.rados.rados.0.smithi107.stdout:10745: finishing write tid 4 to smithi10729407-15 2020-06-12T02:39:12.818 INFO:tasks.rados.rados.0.smithi107.stdout:10745: finishing write tid 6 to smithi10729407-15 2020-06-12T02:39:12.819 INFO:tasks.rados.rados.0.smithi107.stdout:10746: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:12.819 INFO:tasks.rados.rados.0.smithi107.stdout:10745: finishing write tid 7 to smithi10729407-15 2020-06-12T02:39:12.819 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 2005 (ObjNum 10744 snap 0 seq_num 10744) dirty exists 2020-06-12T02:39:12.819 INFO:tasks.rados.rados.0.smithi107.stdout:10745: left oid 15 (ObjNum 10744 snap 0 seq_num 10744) 2020-06-12T02:39:12.824 INFO:tasks.rados.rados.0.smithi107.stdout:10745: done (2 left) 2020-06-12T02:39:12.824 INFO:tasks.rados.rados.0.smithi107.stdout:10747: done (1 left) 2020-06-12T02:39:12.824 INFO:tasks.rados.rados.0.smithi107.stdout:10748: delete oid 11 current snap is 0 2020-06-12T02:39:12.824 INFO:tasks.rados.rados.0.smithi107.stdout:10748: done (1 left) 2020-06-12T02:39:12.825 INFO:tasks.rados.rados.0.smithi107.stdout:10749: delete oid 7 current snap is 0 2020-06-12T02:39:12.825 INFO:tasks.rados.rados.0.smithi107.stdout:10746: finishing write tid 2 to smithi10729407-2 2020-06-12T02:39:12.825 INFO:tasks.rados.rados.0.smithi107.stdout:10749: done (1 left) 2020-06-12T02:39:12.826 INFO:tasks.rados.rados.0.smithi107.stdout:10750: delete oid 20 current snap is 0 2020-06-12T02:39:12.826 INFO:tasks.rados.rados.0.smithi107.stdout:10746: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:12.826 INFO:tasks.rados.rados.0.smithi107.stdout:10746: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:12.827 INFO:tasks.rados.rados.0.smithi107.stdout:10746: finishing write tid 6 to smithi10729407-2 2020-06-12T02:39:12.827 INFO:tasks.rados.rados.0.smithi107.stdout:10750: done (1 left) 2020-06-12T02:39:12.828 INFO:tasks.rados.rados.0.smithi107.stdout:10751: delete oid 20 current snap is 0 2020-06-12T02:39:12.828 INFO:tasks.rados.rados.0.smithi107.stdout:10746: finishing write tid 7 to smithi10729407-2 2020-06-12T02:39:12.828 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 1009 (ObjNum 10745 snap 0 seq_num 10745) dirty exists 2020-06-12T02:39:12.829 INFO:tasks.rados.rados.0.smithi107.stdout:10746: left oid 2 (ObjNum 10745 snap 0 seq_num 10745) 2020-06-12T02:39:12.829 INFO:tasks.rados.rados.0.smithi107.stdout:10746: done (1 left) 2020-06-12T02:39:12.830 INFO:tasks.rados.rados.0.smithi107.stdout:10751: done (0 left) 2020-06-12T02:39:12.830 INFO:tasks.rados.rados.0.smithi107.stdout:10752: delete oid 12 current snap is 0 2020-06-12T02:39:12.832 INFO:tasks.rados.rados.0.smithi107.stdout:10752: done (0 left) 2020-06-12T02:39:12.832 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:39:12.832 INFO:tasks.rados.rados.0.smithi107.stdout:10753: seq_num 10752 ranges {0=81920} 2020-06-12T02:39:12.833 INFO:tasks.rados.rados.0.smithi107.stdout:10753: writing smithi10729407-10 from 0 to 81920 tid 1 2020-06-12T02:39:12.833 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:39:12.833 INFO:tasks.rados.rados.0.smithi107.stdout:10754: seq_num 10753 ranges {3809280=98304} 2020-06-12T02:39:12.855 INFO:tasks.rados.rados.0.smithi107.stdout:10754: writing smithi10729407-9 from 3809280 to 3907584 tid 1 2020-06-12T02:39:12.855 INFO:tasks.rados.rados.0.smithi107.stdout:10755: delete oid 15 current snap is 0 2020-06-12T02:39:12.855 INFO:tasks.rados.rados.0.smithi107.stdout:10753: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:12.855 INFO:tasks.rados.rados.0.smithi107.stdout:10753: finishing write tid 3 to smithi10729407-10 2020-06-12T02:39:12.856 INFO:tasks.rados.rados.0.smithi107.stdout:10753: finishing write tid 4 to smithi10729407-10 2020-06-12T02:39:12.856 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 786 (ObjNum 10752 snap 0 seq_num 10752) dirty exists 2020-06-12T02:39:12.856 INFO:tasks.rados.rados.0.smithi107.stdout:10753: left oid 10 (ObjNum 10752 snap 0 seq_num 10752) 2020-06-12T02:39:12.857 INFO:tasks.rados.rados.0.smithi107.stdout:10754: finishing write tid 1 to smithi10729407-9 2020-06-12T02:39:12.857 INFO:tasks.rados.rados.0.smithi107.stdout:10753: done (2 left) 2020-06-12T02:39:12.858 INFO:tasks.rados.rados.0.smithi107.stdout:10755: done (1 left) 2020-06-12T02:39:12.858 INFO:tasks.rados.rados.0.smithi107.stdout:10756: delete oid 4 current snap is 0 2020-06-12T02:39:12.860 INFO:tasks.rados.rados.0.smithi107.stdout:10754: finishing write tid 3 to smithi10729407-9 2020-06-12T02:39:12.861 INFO:tasks.rados.rados.0.smithi107.stdout:10756: done (1 left) 2020-06-12T02:39:12.861 INFO:tasks.rados.rados.0.smithi107.stdout:10757: delete oid 3 current snap is 0 2020-06-12T02:39:12.861 INFO:tasks.rados.rados.0.smithi107.stdout:10754: finishing write tid 4 to smithi10729407-9 2020-06-12T02:39:12.862 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1841 (ObjNum 10753 snap 0 seq_num 10753) dirty exists 2020-06-12T02:39:12.862 INFO:tasks.rados.rados.0.smithi107.stdout:10754: left oid 9 (ObjNum 10753 snap 0 seq_num 10753) 2020-06-12T02:39:12.864 INFO:tasks.rados.rados.0.smithi107.stdout:10754: done (1 left) 2020-06-12T02:39:12.864 INFO:tasks.rados.rados.0.smithi107.stdout:10757: done (0 left) 2020-06-12T02:39:12.864 INFO:tasks.rados.rados.0.smithi107.stdout:10758: delete oid 3 current snap is 0 2020-06-12T02:39:12.866 INFO:tasks.rados.rados.0.smithi107.stdout:10758: done (0 left) 2020-06-12T02:39:12.866 INFO:tasks.rados.rados.0.smithi107.stdout:10759: delete oid 11 current snap is 0 2020-06-12T02:39:12.869 INFO:tasks.rados.rados.0.smithi107.stdout:10759: done (0 left) 2020-06-12T02:39:12.869 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:39:12.869 INFO:tasks.rados.rados.0.smithi107.stdout:10760: seq_num 10759 ranges {0=196608} 2020-06-12T02:39:12.870 INFO:tasks.rados.rados.0.smithi107.stdout:10760: writing smithi10729407-19 from 0 to 196608 tid 1 2020-06-12T02:39:12.870 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:39:12.871 INFO:tasks.rados.rados.0.smithi107.stdout:10761: seq_num 10760 ranges {3907584=212992} 2020-06-12T02:39:12.893 INFO:tasks.rados.rados.0.smithi107.stdout:10761: writing smithi10729407-9 from 3907584 to 4120576 tid 1 2020-06-12T02:39:12.893 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:39:12.893 INFO:tasks.rados.rados.0.smithi107.stdout:10762: seq_num 10761 ranges {0=229376} 2020-06-12T02:39:12.894 INFO:tasks.rados.rados.0.smithi107.stdout:10762: writing smithi10729407-12 from 0 to 229376 tid 1 2020-06-12T02:39:12.895 INFO:tasks.rados.rados.0.smithi107.stdout:10763: delete oid 3 current snap is 0 2020-06-12T02:39:12.895 INFO:tasks.rados.rados.0.smithi107.stdout:10760: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:12.895 INFO:tasks.rados.rados.0.smithi107.stdout:10760: finishing write tid 3 to smithi10729407-19 2020-06-12T02:39:12.895 INFO:tasks.rados.rados.0.smithi107.stdout:10760: finishing write tid 4 to smithi10729407-19 2020-06-12T02:39:12.896 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1979 (ObjNum 10759 snap 0 seq_num 10759) dirty exists 2020-06-12T02:39:12.896 INFO:tasks.rados.rados.0.smithi107.stdout:10760: left oid 19 (ObjNum 10759 snap 0 seq_num 10759) 2020-06-12T02:39:12.898 INFO:tasks.rados.rados.0.smithi107.stdout:10761: finishing write tid 1 to smithi10729407-9 2020-06-12T02:39:12.898 INFO:tasks.rados.rados.0.smithi107.stdout:10760: done (3 left) 2020-06-12T02:39:12.898 INFO:tasks.rados.rados.0.smithi107.stdout:10763: done (2 left) 2020-06-12T02:39:12.898 INFO:tasks.rados.rados.0.smithi107.stdout:10764: delete oid 7 current snap is 0 2020-06-12T02:39:12.899 INFO:tasks.rados.rados.0.smithi107.stdout:10761: finishing write tid 3 to smithi10729407-9 2020-06-12T02:39:12.899 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:39:12.895+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:39:12.899 INFO:tasks.rados.rados.0.smithi107.stdout:10762: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:12.900 INFO:tasks.rados.rados.0.smithi107.stdout:10761: finishing write tid 4 to smithi10729407-9 2020-06-12T02:39:12.900 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1843 (ObjNum 10760 snap 0 seq_num 10760) dirty exists 2020-06-12T02:39:12.900 INFO:tasks.rados.rados.0.smithi107.stdout:10761: left oid 9 (ObjNum 10760 snap 0 seq_num 10760) 2020-06-12T02:39:12.910 INFO:tasks.rados.rados.0.smithi107.stdout:10762: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:12.910 INFO:tasks.rados.rados.0.smithi107.stdout:10761: done (2 left) 2020-06-12T02:39:12.910 INFO:tasks.rados.rados.0.smithi107.stdout:10764: done (1 left) 2020-06-12T02:39:12.911 INFO:tasks.rados.rados.0.smithi107.stdout:10765: delete oid 3 current snap is 0 2020-06-12T02:39:12.915 INFO:tasks.rados.rados.0.smithi107.stdout:10762: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:12.915 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1981 (ObjNum 10761 snap 0 seq_num 10761) dirty exists 2020-06-12T02:39:12.916 INFO:tasks.rados.rados.0.smithi107.stdout:10762: left oid 12 (ObjNum 10761 snap 0 seq_num 10761) 2020-06-12T02:39:12.916 INFO:tasks.rados.rados.0.smithi107.stdout:10762: done (1 left) 2020-06-12T02:39:12.916 INFO:tasks.rados.rados.0.smithi107.stdout:10765: done (0 left) 2020-06-12T02:39:12.916 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:39:12.917 INFO:tasks.rados.rados.0.smithi107.stdout:10766: seq_num 10765 ranges {0=294912} 2020-06-12T02:39:12.918 INFO:tasks.rados.rados.0.smithi107.stdout:10766: writing smithi10729407-11 from 0 to 294912 tid 1 2020-06-12T02:39:12.918 INFO:tasks.rados.rados.0.smithi107.stdout:10767: delete oid 8 current snap is 0 2020-06-12T02:39:12.918 INFO:tasks.rados.rados.0.smithi107.stdout:10767: done (1 left) 2020-06-12T02:39:12.918 INFO:tasks.rados.rados.0.smithi107.stdout:10768: delete oid 10 current snap is 0 2020-06-12T02:39:12.919 INFO:tasks.rados.rados.0.smithi107.stdout:10768: done (1 left) 2020-06-12T02:39:12.920 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:39:12.920 INFO:tasks.rados.rados.0.smithi107.stdout:10769: seq_num 10768 ranges {229376=352256} 2020-06-12T02:39:12.920 INFO:tasks.rados.rados.0.smithi107.stdout:10769: writing smithi10729407-12 from 229376 to 581632 tid 1 2020-06-12T02:39:12.920 INFO:tasks.rados.rados.0.smithi107.stdout:10770: delete oid 3 current snap is 0 2020-06-12T02:39:12.920 INFO:tasks.rados.rados.0.smithi107.stdout:10766: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:12.921 INFO:tasks.rados.rados.0.smithi107.stdout:10766: finishing write tid 3 to smithi10729407-11 2020-06-12T02:39:12.921 INFO:tasks.rados.rados.0.smithi107.stdout:10766: finishing write tid 4 to smithi10729407-11 2020-06-12T02:39:12.921 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1735 (ObjNum 10765 snap 0 seq_num 10765) dirty exists 2020-06-12T02:39:12.921 INFO:tasks.rados.rados.0.smithi107.stdout:10766: left oid 11 (ObjNum 10765 snap 0 seq_num 10765) 2020-06-12T02:39:12.921 INFO:tasks.rados.rados.0.smithi107.stdout:10766: done (2 left) 2020-06-12T02:39:12.922 INFO:tasks.rados.rados.0.smithi107.stdout:10770: done (1 left) 2020-06-12T02:39:12.922 INFO:tasks.rados.rados.0.smithi107.stdout:10771: delete oid 5 current snap is 0 2020-06-12T02:39:12.922 INFO:tasks.rados.rados.0.smithi107.stdout:10769: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:12.923 INFO:tasks.rados.rados.0.smithi107.stdout:10769: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:12.930 INFO:tasks.rados.rados.0.smithi107.stdout:10769: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:12.931 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1983 (ObjNum 10768 snap 0 seq_num 10768) dirty exists 2020-06-12T02:39:12.931 INFO:tasks.rados.rados.0.smithi107.stdout:10769: left oid 12 (ObjNum 10768 snap 0 seq_num 10768) 2020-06-12T02:39:12.931 INFO:tasks.rados.rados.0.smithi107.stdout:10769: done (1 left) 2020-06-12T02:39:12.931 INFO:tasks.rados.rados.0.smithi107.stdout:10771: done (0 left) 2020-06-12T02:39:12.931 INFO:tasks.rados.rados.0.smithi107.stdout:10772: delete oid 15 current snap is 0 2020-06-12T02:39:12.932 INFO:tasks.rados.rados.0.smithi107.stdout:10772: done (0 left) 2020-06-12T02:39:12.932 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:39:12.932 INFO:tasks.rados.rados.0.smithi107.stdout:10773: seq_num 10772 ranges {0=417792} 2020-06-12T02:39:12.932 INFO:tasks.rados.rados.0.smithi107.stdout:10773: writing smithi10729407-5 from 0 to 417792 tid 1 2020-06-12T02:39:12.932 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:39:12.933 INFO:tasks.rados.rados.0.smithi107.stdout:10774: seq_num 10773 ranges {0=434176} 2020-06-12T02:39:12.933 INFO:tasks.rados.rados.0.smithi107.stdout:10774: writing smithi10729407-14 from 0 to 434176 tid 1 2020-06-12T02:39:12.933 INFO:tasks.rados.rados.0.smithi107.stdout:10775: delete oid 15 current snap is 0 2020-06-12T02:39:12.935 INFO:tasks.rados.rados.0.smithi107.stdout:10773: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:12.940 INFO:tasks.rados.rados.0.smithi107.stdout:10773: finishing write tid 3 to smithi10729407-5 2020-06-12T02:39:12.940 INFO:tasks.rados.rados.0.smithi107.stdout:10775: done (2 left) 2020-06-12T02:39:12.941 INFO:tasks.rados.rados.0.smithi107.stdout:10776: delete oid 4 current snap is 0 2020-06-12T02:39:12.942 INFO:tasks.rados.rados.0.smithi107.stdout:10773: finishing write tid 4 to smithi10729407-5 2020-06-12T02:39:12.943 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 1034 (ObjNum 10772 snap 0 seq_num 10772) dirty exists 2020-06-12T02:39:12.943 INFO:tasks.rados.rados.0.smithi107.stdout:10773: left oid 5 (ObjNum 10772 snap 0 seq_num 10772) 2020-06-12T02:39:12.943 INFO:tasks.rados.rados.0.smithi107.stdout:10774: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:12.944 INFO:tasks.rados.rados.0.smithi107.stdout:10774: finishing write tid 3 to smithi10729407-14 2020-06-12T02:39:12.944 INFO:tasks.rados.rados.0.smithi107.stdout:10773: done (2 left) 2020-06-12T02:39:12.944 INFO:tasks.rados.rados.0.smithi107.stdout:10776: done (1 left) 2020-06-12T02:39:12.944 INFO:tasks.rados.rados.0.smithi107.stdout:10777: delete oid 3 current snap is 0 2020-06-12T02:39:12.945 INFO:tasks.rados.rados.0.smithi107.stdout:10774: finishing write tid 4 to smithi10729407-14 2020-06-12T02:39:12.945 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1714 (ObjNum 10773 snap 0 seq_num 10773) dirty exists 2020-06-12T02:39:12.945 INFO:tasks.rados.rados.0.smithi107.stdout:10774: left oid 14 (ObjNum 10773 snap 0 seq_num 10773) 2020-06-12T02:39:12.949 INFO:tasks.rados.rados.0.smithi107.stdout:10774: done (1 left) 2020-06-12T02:39:12.949 INFO:tasks.rados.rados.0.smithi107.stdout:10777: done (0 left) 2020-06-12T02:39:12.949 INFO:tasks.rados.rados.0.smithi107.stdout:10778: delete oid 11 current snap is 0 2020-06-12T02:39:12.951 INFO:tasks.rados.rados.0.smithi107.stdout:10778: done (0 left) 2020-06-12T02:39:12.951 INFO:tasks.rados.rados.0.smithi107.stdout:10779: delete oid 15 current snap is 0 2020-06-12T02:39:12.954 INFO:tasks.rados.rados.0.smithi107.stdout:10779: done (0 left) 2020-06-12T02:39:12.954 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:39:12.954 INFO:tasks.rados.rados.0.smithi107.stdout:10780: seq_num 10779 ranges {196608=532480} 2020-06-12T02:39:12.963 INFO:tasks.rados.rados.0.smithi107.stdout:10780: writing smithi10729407-19 from 196608 to 729088 tid 1 2020-06-12T02:39:12.963 INFO:tasks.rados.rados.0.smithi107.stdout:10781: delete oid 14 current snap is 0 2020-06-12T02:39:12.963 INFO:tasks.rados.rados.0.smithi107.stdout:10781: done (1 left) 2020-06-12T02:39:12.964 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:39:12.964 INFO:tasks.rados.rados.0.smithi107.stdout:10782: seq_num 10781 ranges {581632=565248} 2020-06-12T02:39:12.964 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:12.964 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:12.968 INFO:tasks.rados.rados.0.smithi107.stdout:10782: writing smithi10729407-12 from 581632 to 1146880 tid 1 2020-06-12T02:39:12.969 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:39:12.969 INFO:tasks.rados.rados.0.smithi107.stdout:10783: seq_num 10782 ranges {0=581632} 2020-06-12T02:39:12.973 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-06-12T02:39:12.973 INFO:tasks.rados.rados.0.smithi107.stdout:10783: writing smithi10729407-17 from 0 to 581632 tid 1 2020-06-12T02:39:12.974 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:39:12.974 INFO:tasks.rados.rados.0.smithi107.stdout:10784: seq_num 10783 ranges {417792=598016} 2020-06-12T02:39:12.978 INFO:tasks.rados.rados.0.smithi107.stdout:10784: writing smithi10729407-5 from 417792 to 1015808 tid 1 2020-06-12T02:39:12.979 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:39:12.979 INFO:tasks.rados.rados.0.smithi107.stdout:10785: seq_num 10784 ranges {0=614400} 2020-06-12T02:39:12.982 INFO:tasks.rados.rados.0.smithi107.stdout:10785: writing smithi10729407-1 from 0 to 614400 tid 1 2020-06-12T02:39:12.983 INFO:tasks.rados.rados.0.smithi107.stdout:10786: delete oid 13 current snap is 0 2020-06-12T02:39:12.983 INFO:tasks.rados.rados.0.smithi107.stdout:10780: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:12.983 INFO:tasks.rados.rados.0.smithi107.stdout:10780: finishing write tid 3 to smithi10729407-19 2020-06-12T02:39:12.983 INFO:tasks.rados.rados.0.smithi107.stdout:10780: finishing write tid 4 to smithi10729407-19 2020-06-12T02:39:12.984 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1985 (ObjNum 10779 snap 0 seq_num 10779) dirty exists 2020-06-12T02:39:12.984 INFO:tasks.rados.rados.0.smithi107.stdout:10780: left oid 19 (ObjNum 10779 snap 0 seq_num 10779) 2020-06-12T02:39:12.984 INFO:tasks.rados.rados.0.smithi107.stdout:10782: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:12.984 INFO:tasks.rados.rados.0.smithi107.stdout:10782: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:12.984 INFO:tasks.rados.rados.0.smithi107.stdout:10783: finishing write tid 1 to smithi10729407-17 2020-06-12T02:39:12.985 INFO:tasks.rados.rados.0.smithi107.stdout:10782: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:12.985 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1987 (ObjNum 10781 snap 0 seq_num 10781) dirty exists 2020-06-12T02:39:12.985 INFO:tasks.rados.rados.0.smithi107.stdout:10782: left oid 12 (ObjNum 10781 snap 0 seq_num 10781) 2020-06-12T02:39:12.986 INFO:tasks.rados.rados.0.smithi107.stdout:10783: finishing write tid 3 to smithi10729407-17 2020-06-12T02:39:12.986 INFO:tasks.rados.rados.0.smithi107.stdout:10783: finishing write tid 4 to smithi10729407-17 2020-06-12T02:39:12.986 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 2017 (ObjNum 10782 snap 0 seq_num 10782) dirty exists 2020-06-12T02:39:12.986 INFO:tasks.rados.rados.0.smithi107.stdout:10783: left oid 17 (ObjNum 10782 snap 0 seq_num 10782) 2020-06-12T02:39:12.986 INFO:tasks.rados.rados.0.smithi107.stdout:10784: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:12.988 INFO:tasks.rados.rados.0.smithi107.stdout:10780: done (5 left) 2020-06-12T02:39:12.988 INFO:tasks.rados.rados.0.smithi107.stdout:10782: done (4 left) 2020-06-12T02:39:12.988 INFO:tasks.rados.rados.0.smithi107.stdout:10783: done (3 left) 2020-06-12T02:39:12.988 INFO:tasks.rados.rados.0.smithi107.stdout:10786: done (2 left) 2020-06-12T02:39:12.988 INFO:tasks.rados.rados.0.smithi107.stdout:10787: delete oid 13 current snap is 0 2020-06-12T02:39:12.991 INFO:tasks.rados.rados.0.smithi107.stdout:10785: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:12.991 INFO:tasks.rados.rados.0.smithi107.stdout:10784: finishing write tid 3 to smithi10729407-5 2020-06-12T02:39:12.992 INFO:tasks.rados.rados.0.smithi107.stdout:10787: done (2 left) 2020-06-12T02:39:12.992 INFO:tasks.rados.rados.0.smithi107.stdout:10788: delete oid 10 current snap is 0 2020-06-12T02:39:12.995 INFO:tasks.rados.rados.0.smithi107.stdout:10784: finishing write tid 4 to smithi10729407-5 2020-06-12T02:39:12.995 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 1036 (ObjNum 10783 snap 0 seq_num 10783) dirty exists 2020-06-12T02:39:12.995 INFO:tasks.rados.rados.0.smithi107.stdout:10784: left oid 5 (ObjNum 10783 snap 0 seq_num 10783) 2020-06-12T02:39:12.995 INFO:tasks.rados.rados.0.smithi107.stdout:10784: done (2 left) 2020-06-12T02:39:12.996 INFO:tasks.rados.rados.0.smithi107.stdout:10788: done (1 left) 2020-06-12T02:39:12.996 INFO:tasks.rados.rados.0.smithi107.stdout:10789: delete oid 18 current snap is 0 2020-06-12T02:39:12.996 INFO:tasks.rados.rados.0.smithi107.stdout:10785: finishing write tid 3 to smithi10729407-1 2020-06-12T02:39:12.997 INFO:tasks.rados.rados.0.smithi107.stdout:10785: finishing write tid 4 to smithi10729407-1 2020-06-12T02:39:12.997 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1938 (ObjNum 10784 snap 0 seq_num 10784) dirty exists 2020-06-12T02:39:12.997 INFO:tasks.rados.rados.0.smithi107.stdout:10785: left oid 1 (ObjNum 10784 snap 0 seq_num 10784) 2020-06-12T02:39:12.999 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:39:12.996+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:39:13.000 INFO:tasks.rados.rados.0.smithi107.stdout:10785: done (1 left) 2020-06-12T02:39:13.000 INFO:tasks.rados.rados.0.smithi107.stdout:10789: done (0 left) 2020-06-12T02:39:13.000 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:39:13.000 INFO:tasks.rados.rados.0.smithi107.stdout:10790: seq_num 10789 ranges {729088=704512} 2020-06-12T02:39:13.008 INFO:tasks.rados.rados.0.smithi107.stdout:10790: writing smithi10729407-19 from 729088 to 1433600 tid 1 2020-06-12T02:39:13.008 INFO:tasks.rados.rados.0.smithi107.stdout:10791: delete oid 7 current snap is 0 2020-06-12T02:39:13.010 INFO:tasks.rados.rados.0.smithi107.stdout:10791: done (1 left) 2020-06-12T02:39:13.010 INFO:tasks.rados.rados.0.smithi107.stdout:10792: delete oid 4 current snap is 0 2020-06-12T02:39:13.013 INFO:tasks.rados.rados.0.smithi107.stdout:10792: done (1 left) 2020-06-12T02:39:13.013 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:39:13.013 INFO:tasks.rados.rados.0.smithi107.stdout:10793: seq_num 10792 ranges {0=753664} 2020-06-12T02:39:13.018 INFO:tasks.rados.rados.0.smithi107.stdout:10793: writing smithi10729407-7 from 0 to 753664 tid 1 2020-06-12T02:39:13.018 INFO:tasks.rados.rados.0.smithi107.stdout:10794: delete oid 12 current snap is 0 2020-06-12T02:39:13.018 INFO:tasks.rados.rados.0.smithi107.stdout:10790: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:13.020 INFO:tasks.rados.rados.0.smithi107.stdout:10790: finishing write tid 3 to smithi10729407-19 2020-06-12T02:39:13.024 INFO:tasks.rados.rados.0.smithi107.stdout:10790: finishing write tid 4 to smithi10729407-19 2020-06-12T02:39:13.024 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 1989 (ObjNum 10789 snap 0 seq_num 10789) dirty exists 2020-06-12T02:39:13.024 INFO:tasks.rados.rados.0.smithi107.stdout:10790: left oid 19 (ObjNum 10789 snap 0 seq_num 10789) 2020-06-12T02:39:13.025 INFO:tasks.rados.rados.0.smithi107.stdout:10790: done (2 left) 2020-06-12T02:39:13.025 INFO:tasks.rados.rados.0.smithi107.stdout:10794: done (1 left) 2020-06-12T02:39:13.025 INFO:tasks.rados.rados.0.smithi107.stdout:10795: delete oid 1 current snap is 0 2020-06-12T02:39:13.026 INFO:tasks.rados.rados.0.smithi107.stdout:10793: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:13.027 INFO:tasks.rados.rados.0.smithi107.stdout:10795: done (1 left) 2020-06-12T02:39:13.028 INFO:tasks.rados.rados.0.smithi107.stdout:10796: delete oid 18 current snap is 0 2020-06-12T02:39:13.030 INFO:tasks.rados.rados.0.smithi107.stdout:10793: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:13.033 INFO:tasks.rados.rados.0.smithi107.stdout:10796: done (1 left) 2020-06-12T02:39:13.033 INFO:tasks.rados.rados.0.smithi107.stdout:10797: delete oid 8 current snap is 0 2020-06-12T02:39:13.033 INFO:tasks.rados.rados.0.smithi107.stdout:10793: finishing write tid 4 to smithi10729407-7 2020-06-12T02:39:13.033 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1781 (ObjNum 10792 snap 0 seq_num 10792) dirty exists 2020-06-12T02:39:13.034 INFO:tasks.rados.rados.0.smithi107.stdout:10793: left oid 7 (ObjNum 10792 snap 0 seq_num 10792) 2020-06-12T02:39:13.035 INFO:tasks.rados.rados.0.smithi107.stdout:10793: done (1 left) 2020-06-12T02:39:13.036 INFO:tasks.rados.rados.0.smithi107.stdout:10797: done (0 left) 2020-06-12T02:39:13.036 INFO:tasks.rados.rados.0.smithi107.stdout:10798: delete oid 4 current snap is 0 2020-06-12T02:39:13.038 INFO:tasks.rados.rados.0.smithi107.stdout:10798: done (0 left) 2020-06-12T02:39:13.038 INFO:tasks.rados.rados.0.smithi107.stdout:10799: delete oid 6 current snap is 0 2020-06-12T02:39:13.041 INFO:tasks.rados.rados.0.smithi107.stdout:10799: done (0 left) 2020-06-12T02:39:13.041 INFO:tasks.rados.rados.0.smithi107.stdout:10800: delete oid 10 current snap is 0 2020-06-12T02:39:13.043 INFO:tasks.rados.rados.0.smithi107.stdout:10800: done (0 left) 2020-06-12T02:39:13.043 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:39:13.043 INFO:tasks.rados.rados.0.smithi107.stdout:10801: seq_num 10800 ranges {0=483328,483328=401408} 2020-06-12T02:39:13.046 INFO:tasks.rados.rados.0.smithi107.stdout:10801: writing smithi10729407-1 from 0 to 483328 tid 1 2020-06-12T02:39:13.047 INFO:tasks.rados.rados.0.smithi107.stdout:10801: writing smithi10729407-1 from 483328 to 884736 tid 2 2020-06-12T02:39:13.048 INFO:tasks.rados.rados.0.smithi107.stdout:10802: delete oid 20 current snap is 0 2020-06-12T02:39:13.050 INFO:tasks.rados.rados.0.smithi107.stdout:10802: done (1 left) 2020-06-12T02:39:13.050 INFO:tasks.rados.rados.0.smithi107.stdout:10803: delete oid 13 current snap is 0 2020-06-12T02:39:13.050 INFO:tasks.rados.rados.0.smithi107.stdout:10801: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:13.052 INFO:tasks.rados.rados.0.smithi107.stdout:10803: done (1 left) 2020-06-12T02:39:13.052 INFO:tasks.rados.rados.0.smithi107.stdout:10804: delete oid 16 current snap is 0 2020-06-12T02:39:13.057 INFO:tasks.rados.rados.0.smithi107.stdout:10804: done (1 left) 2020-06-12T02:39:13.057 INFO:tasks.rados.rados.0.smithi107.stdout:10805: delete oid 5 current snap is 0 2020-06-12T02:39:13.058 INFO:tasks.rados.rados.0.smithi107.stdout:10801: finishing write tid 2 to smithi10729407-1 2020-06-12T02:39:13.058 INFO:tasks.rados.rados.0.smithi107.stdout:10805: done (1 left) 2020-06-12T02:39:13.058 INFO:tasks.rados.rados.0.smithi107.stdout:10806: delete oid 7 current snap is 0 2020-06-12T02:39:13.059 INFO:tasks.rados.rados.0.smithi107.stdout:10801: finishing write tid 4 to smithi10729407-1 2020-06-12T02:39:13.060 INFO:tasks.rados.rados.0.smithi107.stdout:10801: finishing write tid 5 to smithi10729407-1 2020-06-12T02:39:13.060 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1943 (ObjNum 10800 snap 0 seq_num 10800) dirty exists 2020-06-12T02:39:13.061 INFO:tasks.rados.rados.0.smithi107.stdout:10801: left oid 1 (ObjNum 10800 snap 0 seq_num 10800) 2020-06-12T02:39:13.061 INFO:tasks.rados.rados.0.smithi107.stdout:10801: done (1 left) 2020-06-12T02:39:13.061 INFO:tasks.rados.rados.0.smithi107.stdout:10806: done (0 left) 2020-06-12T02:39:13.062 INFO:tasks.rados.rados.0.smithi107.stdout:10807: delete oid 9 current snap is 0 2020-06-12T02:39:13.063 INFO:tasks.rados.rados.0.smithi107.stdout:10807: done (0 left) 2020-06-12T02:39:13.063 INFO:tasks.rados.rados.0.smithi107.stdout:10808: delete oid 14 current snap is 0 2020-06-12T02:39:13.065 INFO:tasks.rados.rados.0.smithi107.stdout:10808: done (0 left) 2020-06-12T02:39:13.066 INFO:tasks.rados.rados.0.smithi107.stdout:10809: delete oid 5 current snap is 0 2020-06-12T02:39:13.069 INFO:tasks.rados.rados.0.smithi107.stdout:10809: done (0 left) 2020-06-12T02:39:13.069 INFO:tasks.rados.rados.0.smithi107.stdout:10810: delete oid 13 current snap is 0 2020-06-12T02:39:13.072 INFO:tasks.rados.rados.0.smithi107.stdout:10810: done (0 left) 2020-06-12T02:39:13.072 INFO:tasks.rados.rados.0.smithi107.stdout:10811: delete oid 15 current snap is 0 2020-06-12T02:39:13.074 INFO:tasks.rados.rados.0.smithi107.stdout:10811: done (0 left) 2020-06-12T02:39:13.075 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:39:13.075 INFO:tasks.rados.rados.0.smithi107.stdout:10812: seq_num 10811 ranges {884736=671744,1556480=401408} 2020-06-12T02:39:13.083 INFO:tasks.rados.rados.0.smithi107.stdout:10812: writing smithi10729407-1 from 884736 to 1556480 tid 1 2020-06-12T02:39:13.085 INFO:tasks.rados.rados.0.smithi107.stdout:10812: writing smithi10729407-1 from 1556480 to 1957888 tid 2 2020-06-12T02:39:13.085 INFO:tasks.rados.rados.0.smithi107.stdout:10813: delete oid 13 current snap is 0 2020-06-12T02:39:13.088 INFO:tasks.rados.rados.0.smithi107.stdout:10813: done (1 left) 2020-06-12T02:39:13.088 INFO:tasks.rados.rados.0.smithi107.stdout:10814: delete oid 14 current snap is 0 2020-06-12T02:39:13.092 INFO:tasks.rados.rados.0.smithi107.stdout:10812: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:13.093 INFO:tasks.rados.rados.0.smithi107.stdout:10812: finishing write tid 2 to smithi10729407-1 2020-06-12T02:39:13.095 INFO:tasks.rados.rados.0.smithi107.stdout:10812: finishing write tid 4 to smithi10729407-1 2020-06-12T02:39:13.095 INFO:tasks.rados.rados.0.smithi107.stdout:10814: done (1 left) 2020-06-12T02:39:13.095 INFO:tasks.rados.rados.0.smithi107.stdout:10815: delete oid 3 current snap is 0 2020-06-12T02:39:13.097 INFO:tasks.rados.rados.0.smithi107.stdout:10812: finishing write tid 5 to smithi10729407-1 2020-06-12T02:39:13.097 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1946 (ObjNum 10811 snap 0 seq_num 10811) dirty exists 2020-06-12T02:39:13.097 INFO:tasks.rados.rados.0.smithi107.stdout:10812: left oid 1 (ObjNum 10811 snap 0 seq_num 10811) 2020-06-12T02:39:13.098 INFO:tasks.rados.rados.0.smithi107.stdout:10812: done (1 left) 2020-06-12T02:39:13.098 INFO:tasks.rados.rados.0.smithi107.stdout:10815: done (0 left) 2020-06-12T02:39:13.098 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:39:13.099 INFO:tasks.rados.rados.0.smithi107.stdout:10816: seq_num 10815 ranges {0=737280,737280=401408} 2020-06-12T02:39:13.100 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:39:13.098+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:39:13.102 INFO:tasks.rados.rados.0.smithi107.stdout:10816: writing smithi10729407-4 from 0 to 737280 tid 1 2020-06-12T02:39:13.105 INFO:tasks.rados.rados.0.smithi107.stdout:10816: writing smithi10729407-4 from 737280 to 1138688 tid 2 2020-06-12T02:39:13.105 INFO:tasks.rados.rados.0.smithi107.stdout:10817: delete oid 14 current snap is 0 2020-06-12T02:39:13.107 INFO:tasks.rados.rados.0.smithi107.stdout:10817: done (1 left) 2020-06-12T02:39:13.107 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:39:13.107 INFO:tasks.rados.rados.0.smithi107.stdout:10818: seq_num 10817 ranges {0=770048,770048=401408} 2020-06-12T02:39:13.112 INFO:tasks.rados.rados.0.smithi107.stdout:10818: writing smithi10729407-5 from 0 to 770048 tid 1 2020-06-12T02:39:13.114 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:13.115 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:13.115 INFO:tasks.rados.rados.0.smithi107.stdout:10818: writing smithi10729407-5 from 770048 to 1171456 tid 2 2020-06-12T02:39:13.115 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:39:13.115 INFO:tasks.rados.rados.0.smithi107.stdout:10819: seq_num 10818 ranges {0=786432,786432=401408} 2020-06-12T02:39:13.119 INFO:tasks.rados.rados.0.smithi107.stdout:10819: writing smithi10729407-3 from 0 to 786432 tid 1 2020-06-12T02:39:13.122 INFO:tasks.rados.rados.0.smithi107.stdout:10819: writing smithi10729407-3 from 786432 to 1187840 tid 2 2020-06-12T02:39:13.122 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:39:13.122 INFO:tasks.rados.rados.0.smithi107.stdout:10820: seq_num 10819 ranges {0=802816,802816=401408} 2020-06-12T02:39:13.126 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-06-12T02:39:13.127 INFO:tasks.rados.rados.0.smithi107.stdout:10820: writing smithi10729407-13 from 0 to 802816 tid 1 2020-06-12T02:39:13.130 INFO:tasks.rados.rados.0.smithi107.stdout:10820: writing smithi10729407-13 from 802816 to 1204224 tid 2 2020-06-12T02:39:13.131 INFO:tasks.rados.rados.0.smithi107.stdout:10821: delete oid 18 current snap is 0 2020-06-12T02:39:13.131 INFO:tasks.rados.rados.0.smithi107.stdout:10816: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:13.131 INFO:tasks.rados.rados.0.smithi107.stdout:10816: finishing write tid 2 to smithi10729407-4 2020-06-12T02:39:13.131 INFO:tasks.rados.rados.0.smithi107.stdout:10816: finishing write tid 4 to smithi10729407-4 2020-06-12T02:39:13.132 INFO:tasks.rados.rados.0.smithi107.stdout:10816: finishing write tid 5 to smithi10729407-4 2020-06-12T02:39:13.132 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 2022 (ObjNum 10815 snap 0 seq_num 10815) dirty exists 2020-06-12T02:39:13.132 INFO:tasks.rados.rados.0.smithi107.stdout:10816: left oid 4 (ObjNum 10815 snap 0 seq_num 10815) 2020-06-12T02:39:13.132 INFO:tasks.rados.rados.0.smithi107.stdout:10818: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:13.132 INFO:tasks.rados.rados.0.smithi107.stdout:10818: finishing write tid 2 to smithi10729407-5 2020-06-12T02:39:13.133 INFO:tasks.rados.rados.0.smithi107.stdout:10818: finishing write tid 4 to smithi10729407-5 2020-06-12T02:39:13.133 INFO:tasks.rados.rados.0.smithi107.stdout:10818: finishing write tid 5 to smithi10729407-5 2020-06-12T02:39:13.133 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 1041 (ObjNum 10817 snap 0 seq_num 10817) dirty exists 2020-06-12T02:39:13.133 INFO:tasks.rados.rados.0.smithi107.stdout:10818: left oid 5 (ObjNum 10817 snap 0 seq_num 10817) 2020-06-12T02:39:13.134 INFO:tasks.rados.rados.0.smithi107.stdout:10819: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:13.135 INFO:tasks.rados.rados.0.smithi107.stdout:10816: done (4 left) 2020-06-12T02:39:13.135 INFO:tasks.rados.rados.0.smithi107.stdout:10818: done (3 left) 2020-06-12T02:39:13.136 INFO:tasks.rados.rados.0.smithi107.stdout:10821: done (2 left) 2020-06-12T02:39:13.136 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:39:13.136 INFO:tasks.rados.rados.0.smithi107.stdout:10822: seq_num 10821 ranges {1171456=434176,1605632=655360,2260992=147456} 2020-06-12T02:39:13.144 INFO:tasks.rados.rados.0.smithi107.stdout:10822: writing smithi10729407-5 from 1171456 to 1605632 tid 1 2020-06-12T02:39:13.149 INFO:tasks.rados.rados.0.smithi107.stdout:10822: writing smithi10729407-5 from 1605632 to 2260992 tid 2 2020-06-12T02:39:13.150 INFO:tasks.rados.rados.0.smithi107.stdout:10822: writing smithi10729407-5 from 2260992 to 2408448 tid 3 2020-06-12T02:39:13.150 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:39:13.150 INFO:tasks.rados.rados.0.smithi107.stdout:10823: seq_num 10822 ranges {581632=450560,1032192=540672,1572864=262144} 2020-06-12T02:39:13.155 INFO:tasks.rados.rados.0.smithi107.stdout:10823: writing smithi10729407-17 from 581632 to 1032192 tid 1 2020-06-12T02:39:13.159 INFO:tasks.rados.rados.0.smithi107.stdout:10823: writing smithi10729407-17 from 1032192 to 1572864 tid 2 2020-06-12T02:39:13.161 INFO:tasks.rados.rados.0.smithi107.stdout:10823: writing smithi10729407-17 from 1572864 to 1835008 tid 3 2020-06-12T02:39:13.162 INFO:tasks.rados.rados.0.smithi107.stdout:10824: delete oid 1 current snap is 0 2020-06-12T02:39:13.162 INFO:tasks.rados.rados.0.smithi107.stdout:10819: finishing write tid 2 to smithi10729407-3 2020-06-12T02:39:13.162 INFO:tasks.rados.rados.0.smithi107.stdout:10819: finishing write tid 4 to smithi10729407-3 2020-06-12T02:39:13.162 INFO:tasks.rados.rados.0.smithi107.stdout:10820: finishing write tid 1 to smithi10729407-13 2020-06-12T02:39:13.163 INFO:tasks.rados.rados.0.smithi107.stdout:10819: finishing write tid 5 to smithi10729407-3 2020-06-12T02:39:13.163 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1929 (ObjNum 10818 snap 0 seq_num 10818) dirty exists 2020-06-12T02:39:13.163 INFO:tasks.rados.rados.0.smithi107.stdout:10819: left oid 3 (ObjNum 10818 snap 0 seq_num 10818) 2020-06-12T02:39:13.163 INFO:tasks.rados.rados.0.smithi107.stdout:10820: finishing write tid 2 to smithi10729407-13 2020-06-12T02:39:13.163 INFO:tasks.rados.rados.0.smithi107.stdout:10820: finishing write tid 4 to smithi10729407-13 2020-06-12T02:39:13.164 INFO:tasks.rados.rados.0.smithi107.stdout:10820: finishing write tid 5 to smithi10729407-13 2020-06-12T02:39:13.164 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1769 (ObjNum 10819 snap 0 seq_num 10819) dirty exists 2020-06-12T02:39:13.164 INFO:tasks.rados.rados.0.smithi107.stdout:10820: left oid 13 (ObjNum 10819 snap 0 seq_num 10819) 2020-06-12T02:39:13.167 INFO:tasks.rados.rados.0.smithi107.stdout:10822: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:13.167 INFO:tasks.rados.rados.0.smithi107.stdout:10822: finishing write tid 2 to smithi10729407-5 2020-06-12T02:39:13.167 INFO:tasks.rados.rados.0.smithi107.stdout:10822: finishing write tid 3 to smithi10729407-5 2020-06-12T02:39:13.167 INFO:tasks.rados.rados.0.smithi107.stdout:10822: finishing write tid 5 to smithi10729407-5 2020-06-12T02:39:13.170 INFO:tasks.rados.rados.0.smithi107.stdout:10822: finishing write tid 6 to smithi10729407-5 2020-06-12T02:39:13.170 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 1045 (ObjNum 10821 snap 0 seq_num 10821) dirty exists 2020-06-12T02:39:13.171 INFO:tasks.rados.rados.0.smithi107.stdout:10822: left oid 5 (ObjNum 10821 snap 0 seq_num 10821) 2020-06-12T02:39:13.171 INFO:tasks.rados.rados.0.smithi107.stdout:10823: finishing write tid 1 to smithi10729407-17 2020-06-12T02:39:13.171 INFO:tasks.rados.rados.0.smithi107.stdout:10819: done (4 left) 2020-06-12T02:39:13.171 INFO:tasks.rados.rados.0.smithi107.stdout:10820: done (3 left) 2020-06-12T02:39:13.171 INFO:tasks.rados.rados.0.smithi107.stdout:10822: done (2 left) 2020-06-12T02:39:13.171 INFO:tasks.rados.rados.0.smithi107.stdout:10824: done (1 left) 2020-06-12T02:39:13.172 INFO:tasks.rados.rados.0.smithi107.stdout:10825: delete oid 5 current snap is 0 2020-06-12T02:39:13.172 INFO:tasks.rados.rados.0.smithi107.stdout:10825: done (1 left) 2020-06-12T02:39:13.172 INFO:tasks.rados.rados.0.smithi107.stdout:10826: delete oid 19 current snap is 0 2020-06-12T02:39:13.174 INFO:tasks.rados.rados.0.smithi107.stdout:10826: done (1 left) 2020-06-12T02:39:13.174 INFO:tasks.rados.rados.0.smithi107.stdout:10827: delete oid 20 current snap is 0 2020-06-12T02:39:13.176 INFO:tasks.rados.rados.0.smithi107.stdout:10823: finishing write tid 2 to smithi10729407-17 2020-06-12T02:39:13.179 INFO:tasks.rados.rados.0.smithi107.stdout:10827: done (1 left) 2020-06-12T02:39:13.180 INFO:tasks.rados.rados.0.smithi107.stdout:10828: delete oid 10 current snap is 0 2020-06-12T02:39:13.180 INFO:tasks.rados.rados.0.smithi107.stdout:10823: finishing write tid 3 to smithi10729407-17 2020-06-12T02:39:13.180 INFO:tasks.rados.rados.0.smithi107.stdout:10823: finishing write tid 5 to smithi10729407-17 2020-06-12T02:39:13.181 INFO:tasks.rados.rados.0.smithi107.stdout:10823: finishing write tid 6 to smithi10729407-17 2020-06-12T02:39:13.182 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 2026 (ObjNum 10822 snap 0 seq_num 10822) dirty exists 2020-06-12T02:39:13.182 INFO:tasks.rados.rados.0.smithi107.stdout:10823: left oid 17 (ObjNum 10822 snap 0 seq_num 10822) 2020-06-12T02:39:13.182 INFO:tasks.rados.rados.0.smithi107.stdout:10823: done (1 left) 2020-06-12T02:39:13.183 INFO:tasks.rados.rados.0.smithi107.stdout:10828: done (0 left) 2020-06-12T02:39:13.183 INFO:tasks.rados.rados.0.smithi107.stdout:10829: delete oid 7 current snap is 0 2020-06-12T02:39:13.184 INFO:tasks.rados.rados.0.smithi107.stdout:10829: done (0 left) 2020-06-12T02:39:13.185 INFO:tasks.rados.rados.0.smithi107.stdout:10830: delete oid 8 current snap is 0 2020-06-12T02:39:13.187 INFO:tasks.rados.rados.0.smithi107.stdout:10830: done (0 left) 2020-06-12T02:39:13.187 INFO:tasks.rados.rados.0.smithi107.stdout:10831: delete oid 10 current snap is 0 2020-06-12T02:39:13.189 INFO:tasks.rados.rados.0.smithi107.stdout:10831: done (0 left) 2020-06-12T02:39:13.189 INFO:tasks.rados.rados.0.smithi107.stdout:10832: delete oid 20 current snap is 0 2020-06-12T02:39:13.190 INFO:tasks.rados.rados.0.smithi107.stdout:10832: done (0 left) 2020-06-12T02:39:13.191 INFO:tasks.rados.rados.0.smithi107.stdout:10833: delete oid 1 current snap is 0 2020-06-12T02:39:13.193 INFO:tasks.rados.rados.0.smithi107.stdout:10833: done (0 left) 2020-06-12T02:39:13.193 INFO:tasks.rados.rados.0.smithi107.stdout:10834: delete oid 5 current snap is 0 2020-06-12T02:39:13.195 INFO:tasks.rados.rados.0.smithi107.stdout:10834: done (0 left) 2020-06-12T02:39:13.196 INFO:tasks.rados.rados.0.smithi107.stdout:10835: delete oid 7 current snap is 0 2020-06-12T02:39:13.197 INFO:tasks.rados.rados.0.smithi107.stdout:10835: done (0 left) 2020-06-12T02:39:13.198 INFO:tasks.rados.rados.0.smithi107.stdout:10836: delete oid 14 current snap is 0 2020-06-12T02:39:13.199 INFO:tasks.rados.rados.0.smithi107.stdout:10836: done (0 left) 2020-06-12T02:39:13.200 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:39:13.200 INFO:tasks.rados.rados.0.smithi107.stdout:10837: seq_num 10836 ranges {0=688128,688128=622592,1310720=180224} 2020-06-12T02:39:13.202 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:39:13.200+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:39:13.203 INFO:tasks.rados.rados.0.smithi107.stdout:10837: writing smithi10729407-14 from 0 to 688128 tid 1 2020-06-12T02:39:13.207 INFO:tasks.rados.rados.0.smithi107.stdout:10837: writing smithi10729407-14 from 688128 to 1310720 tid 2 2020-06-12T02:39:13.208 INFO:tasks.rados.rados.0.smithi107.stdout:10837: writing smithi10729407-14 from 1310720 to 1490944 tid 3 2020-06-12T02:39:13.208 INFO:tasks.rados.rados.0.smithi107.stdout:10838: delete oid 19 current snap is 0 2020-06-12T02:39:13.211 INFO:tasks.rados.rados.0.smithi107.stdout:10838: done (1 left) 2020-06-12T02:39:13.212 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:39:13.212 INFO:tasks.rados.rados.0.smithi107.stdout:10839: seq_num 10838 ranges {0=720896,720896=786432,1507328=16384} 2020-06-12T02:39:13.216 INFO:tasks.rados.rados.0.smithi107.stdout:10839: writing smithi10729407-12 from 0 to 720896 tid 1 2020-06-12T02:39:13.221 INFO:tasks.rados.rados.0.smithi107.stdout:10839: writing smithi10729407-12 from 720896 to 1507328 tid 2 2020-06-12T02:39:13.221 INFO:tasks.rados.rados.0.smithi107.stdout:10839: writing smithi10729407-12 from 1507328 to 1523712 tid 3 2020-06-12T02:39:13.221 INFO:tasks.rados.rados.0.smithi107.stdout:10840: delete oid 3 current snap is 0 2020-06-12T02:39:13.221 INFO:tasks.rados.rados.0.smithi107.stdout:10837: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:13.222 INFO:tasks.rados.rados.0.smithi107.stdout:10837: finishing write tid 2 to smithi10729407-14 2020-06-12T02:39:13.222 INFO:tasks.rados.rados.0.smithi107.stdout:10837: finishing write tid 3 to smithi10729407-14 2020-06-12T02:39:13.222 INFO:tasks.rados.rados.0.smithi107.stdout:10837: finishing write tid 5 to smithi10729407-14 2020-06-12T02:39:13.222 INFO:tasks.rados.rados.0.smithi107.stdout:10837: finishing write tid 6 to smithi10729407-14 2020-06-12T02:39:13.222 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1723 (ObjNum 10836 snap 0 seq_num 10836) dirty exists 2020-06-12T02:39:13.223 INFO:tasks.rados.rados.0.smithi107.stdout:10837: left oid 14 (ObjNum 10836 snap 0 seq_num 10836) 2020-06-12T02:39:13.223 INFO:tasks.rados.rados.0.smithi107.stdout:10839: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:13.224 INFO:tasks.rados.rados.0.smithi107.stdout:10837: done (2 left) 2020-06-12T02:39:13.224 INFO:tasks.rados.rados.0.smithi107.stdout:10840: done (1 left) 2020-06-12T02:39:13.225 INFO:tasks.rados.rados.0.smithi107.stdout:10841: delete oid 1 current snap is 0 2020-06-12T02:39:13.228 INFO:tasks.rados.rados.0.smithi107.stdout:10841: done (1 left) 2020-06-12T02:39:13.228 INFO:tasks.rados.rados.0.smithi107.stdout:10842: delete oid 17 current snap is 0 2020-06-12T02:39:13.231 INFO:tasks.rados.rados.0.smithi107.stdout:10839: finishing write tid 2 to smithi10729407-12 2020-06-12T02:39:13.232 INFO:tasks.rados.rados.0.smithi107.stdout:10839: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:13.236 INFO:tasks.rados.rados.0.smithi107.stdout:10842: done (1 left) 2020-06-12T02:39:13.236 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:39:13.236 INFO:tasks.rados.rados.0.smithi107.stdout:10843: seq_num 10842 ranges {2367488=786432,3153920=778240,3932160=24576} 2020-06-12T02:39:13.253 INFO:tasks.rados.rados.0.smithi107.stdout:10843: writing smithi10729407-2 from 2367488 to 3153920 tid 1 2020-06-12T02:39:13.258 INFO:tasks.rados.rados.0.smithi107.stdout:10843: writing smithi10729407-2 from 3153920 to 3932160 tid 2 2020-06-12T02:39:13.258 INFO:tasks.rados.rados.0.smithi107.stdout:10843: writing smithi10729407-2 from 3932160 to 3956736 tid 3 2020-06-12T02:39:13.259 INFO:tasks.rados.rados.0.smithi107.stdout:10844: delete oid 4 current snap is 0 2020-06-12T02:39:13.259 INFO:tasks.rados.rados.0.smithi107.stdout:10839: finishing write tid 5 to smithi10729407-12 2020-06-12T02:39:13.259 INFO:tasks.rados.rados.0.smithi107.stdout:10839: finishing write tid 6 to smithi10729407-12 2020-06-12T02:39:13.259 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 1996 (ObjNum 10838 snap 0 seq_num 10838) dirty exists 2020-06-12T02:39:13.260 INFO:tasks.rados.rados.0.smithi107.stdout:10839: left oid 12 (ObjNum 10838 snap 0 seq_num 10838) 2020-06-12T02:39:13.261 INFO:tasks.rados.rados.0.smithi107.stdout:10843: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:13.261 INFO:tasks.rados.rados.0.smithi107.stdout:10839: done (2 left) 2020-06-12T02:39:13.261 INFO:tasks.rados.rados.0.smithi107.stdout:10844: done (1 left) 2020-06-12T02:39:13.261 INFO:tasks.rados.rados.0.smithi107.stdout:10845: delete oid 10 current snap is 0 2020-06-12T02:39:13.266 INFO:tasks.rados.rados.0.smithi107.stdout:10845: done (1 left) 2020-06-12T02:39:13.266 INFO:tasks.rados.rados.0.smithi107.stdout:10846: delete oid 18 current snap is 0 2020-06-12T02:39:13.269 INFO:tasks.rados.rados.0.smithi107.stdout:10846: done (1 left) 2020-06-12T02:39:13.269 INFO:tasks.rados.rados.0.smithi107.stdout:10847: delete oid 8 current snap is 0 2020-06-12T02:39:13.270 INFO:tasks.rados.rados.0.smithi107.stdout:10843: finishing write tid 2 to smithi10729407-2 2020-06-12T02:39:13.271 INFO:tasks.rados.rados.0.smithi107.stdout:10843: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:13.272 INFO:tasks.rados.rados.0.smithi107.stdout:10847: done (1 left) 2020-06-12T02:39:13.272 INFO:tasks.rados.rados.0.smithi107.stdout:10848: delete oid 12 current snap is 0 2020-06-12T02:39:13.272 INFO:tasks.rados.rados.0.smithi107.stdout:10843: finishing write tid 5 to smithi10729407-2 2020-06-12T02:39:13.274 INFO:tasks.rados.rados.0.smithi107.stdout:10843: finishing write tid 6 to smithi10729407-2 2020-06-12T02:39:13.274 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 1013 (ObjNum 10842 snap 0 seq_num 10842) dirty exists 2020-06-12T02:39:13.274 INFO:tasks.rados.rados.0.smithi107.stdout:10843: left oid 2 (ObjNum 10842 snap 0 seq_num 10842) 2020-06-12T02:39:13.275 INFO:tasks.rados.rados.0.smithi107.stdout:10843: done (1 left) 2020-06-12T02:39:13.284 INFO:tasks.rados.rados.0.smithi107.stdout:10848: done (0 left) 2020-06-12T02:39:13.284 INFO:tasks.rados.rados.0.smithi107.stdout:10849: delete oid 11 current snap is 0 2020-06-12T02:39:13.284 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:13.284 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:13.286 INFO:tasks.rados.rados.0.smithi107.stdout:10849: done (0 left) 2020-06-12T02:39:13.286 INFO:tasks.rados.rados.0.smithi107.stdout:10850: delete oid 12 current snap is 0 2020-06-12T02:39:13.286 INFO:tasks.rados.rados.0.smithi107.stdout:10850: done (0 left) 2020-06-12T02:39:13.286 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:39:13.287 INFO:tasks.rados.rados.0.smithi107.stdout:10851: seq_num 10850 ranges {0=524288,524288=696320,1220608=507904} 2020-06-12T02:39:13.287 INFO:tasks.rados.rados.0.smithi107.stdout:10851: writing smithi10729407-5 from 0 to 524288 tid 1 2020-06-12T02:39:13.287 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-06-12T02:39:13.288 INFO:tasks.rados.rados.0.smithi107.stdout:10851: writing smithi10729407-5 from 524288 to 1220608 tid 2 2020-06-12T02:39:13.290 INFO:tasks.rados.rados.0.smithi107.stdout:10851: writing smithi10729407-5 from 1220608 to 1728512 tid 3 2020-06-12T02:39:13.290 INFO:tasks.rados.rados.0.smithi107.stdout:10852: delete oid 12 current snap is 0 2020-06-12T02:39:13.291 INFO:tasks.rados.rados.0.smithi107.stdout:10851: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:13.293 INFO:tasks.rados.rados.0.smithi107.stdout:10852: done (1 left) 2020-06-12T02:39:13.294 INFO:tasks.rados.rados.0.smithi107.stdout:10853: delete oid 6 current snap is 0 2020-06-12T02:39:13.296 INFO:tasks.rados.rados.0.smithi107.stdout:10853: done (1 left) 2020-06-12T02:39:13.297 INFO:tasks.rados.rados.0.smithi107.stdout:10854: delete oid 12 current snap is 0 2020-06-12T02:39:13.297 INFO:tasks.rados.rados.0.smithi107.stdout:10851: finishing write tid 2 to smithi10729407-5 2020-06-12T02:39:13.297 INFO:tasks.rados.rados.0.smithi107.stdout:10851: finishing write tid 3 to smithi10729407-5 2020-06-12T02:39:13.298 INFO:tasks.rados.rados.0.smithi107.stdout:10851: finishing write tid 5 to smithi10729407-5 2020-06-12T02:39:13.300 INFO:tasks.rados.rados.0.smithi107.stdout:10851: finishing write tid 6 to smithi10729407-5 2020-06-12T02:39:13.300 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 1051 (ObjNum 10850 snap 0 seq_num 10850) dirty exists 2020-06-12T02:39:13.300 INFO:tasks.rados.rados.0.smithi107.stdout:10851: left oid 5 (ObjNum 10850 snap 0 seq_num 10850) 2020-06-12T02:39:13.301 INFO:tasks.rados.rados.0.smithi107.stdout:10851: done (1 left) 2020-06-12T02:39:13.301 INFO:tasks.rados.rados.0.smithi107.stdout:10854: done (0 left) 2020-06-12T02:39:13.302 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:39:13.302 INFO:tasks.rados.rados.0.smithi107.stdout:10855: seq_num 10854 ranges {0=589824,589824=638976,1228800=565248} 2020-06-12T02:39:13.304 INFO:tasks.rados.rados.0.smithi107.stdout:10855: writing smithi10729407-11 from 0 to 589824 tid 1 2020-06-12T02:39:13.304 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:39:13.301+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:39:13.307 INFO:tasks.rados.rados.0.smithi107.stdout:10855: writing smithi10729407-11 from 589824 to 1228800 tid 2 2020-06-12T02:39:13.311 INFO:tasks.rados.rados.0.smithi107.stdout:10855: writing smithi10729407-11 from 1228800 to 1794048 tid 3 2020-06-12T02:39:13.312 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:39:13.312 INFO:tasks.rados.rados.0.smithi107.stdout:10856: seq_num 10855 ranges {3956736=606208,4562944=524288,5087232=557056,5644288=122880} 2020-06-12T02:39:13.337 INFO:tasks.rados.rados.0.smithi107.stdout:10856: writing smithi10729407-2 from 3956736 to 4562944 tid 1 2020-06-12T02:39:13.340 INFO:tasks.rados.rados.0.smithi107.stdout:10856: writing smithi10729407-2 from 4562944 to 5087232 tid 2 2020-06-12T02:39:13.344 INFO:tasks.rados.rados.0.smithi107.stdout:10856: writing smithi10729407-2 from 5087232 to 5644288 tid 3 2020-06-12T02:39:13.345 INFO:tasks.rados.rados.0.smithi107.stdout:10856: writing smithi10729407-2 from 5644288 to 5767168 tid 4 2020-06-12T02:39:13.346 INFO:tasks.rados.rados.0.smithi107.stdout:10857: delete oid 10 current snap is 0 2020-06-12T02:39:13.346 INFO:tasks.rados.rados.0.smithi107.stdout:10855: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:13.346 INFO:tasks.rados.rados.0.smithi107.stdout:10855: finishing write tid 2 to smithi10729407-11 2020-06-12T02:39:13.346 INFO:tasks.rados.rados.0.smithi107.stdout:10855: finishing write tid 3 to smithi10729407-11 2020-06-12T02:39:13.347 INFO:tasks.rados.rados.0.smithi107.stdout:10855: finishing write tid 5 to smithi10729407-11 2020-06-12T02:39:13.347 INFO:tasks.rados.rados.0.smithi107.stdout:10855: finishing write tid 6 to smithi10729407-11 2020-06-12T02:39:13.347 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1741 (ObjNum 10854 snap 0 seq_num 10854) dirty exists 2020-06-12T02:39:13.347 INFO:tasks.rados.rados.0.smithi107.stdout:10855: left oid 11 (ObjNum 10854 snap 0 seq_num 10854) 2020-06-12T02:39:13.347 INFO:tasks.rados.rados.0.smithi107.stdout:10856: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:13.348 INFO:tasks.rados.rados.0.smithi107.stdout:10855: done (2 left) 2020-06-12T02:39:13.348 INFO:tasks.rados.rados.0.smithi107.stdout:10857: done (1 left) 2020-06-12T02:39:13.348 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:39:13.349 INFO:tasks.rados.rados.0.smithi107.stdout:10858: seq_num 10857 ranges {1794048=638976,2433024=737280,3170304=466944} 2020-06-12T02:39:13.361 INFO:tasks.rados.rados.0.smithi107.stdout:10858: writing smithi10729407-11 from 1794048 to 2433024 tid 1 2020-06-12T02:39:13.365 INFO:tasks.rados.rados.0.smithi107.stdout:10858: writing smithi10729407-11 from 2433024 to 3170304 tid 2 2020-06-12T02:39:13.373 INFO:tasks.rados.rados.0.smithi107.stdout:10858: writing smithi10729407-11 from 3170304 to 3637248 tid 3 2020-06-12T02:39:13.374 INFO:tasks.rados.rados.0.smithi107.stdout:10859: delete oid 6 current snap is 0 2020-06-12T02:39:13.374 INFO:tasks.rados.rados.0.smithi107.stdout:10856: finishing write tid 2 to smithi10729407-2 2020-06-12T02:39:13.374 INFO:tasks.rados.rados.0.smithi107.stdout:10856: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:13.374 INFO:tasks.rados.rados.0.smithi107.stdout:10856: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:13.375 INFO:tasks.rados.rados.0.smithi107.stdout:10856: finishing write tid 6 to smithi10729407-2 2020-06-12T02:39:13.375 INFO:tasks.rados.rados.0.smithi107.stdout:10856: finishing write tid 7 to smithi10729407-2 2020-06-12T02:39:13.375 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 1018 (ObjNum 10855 snap 0 seq_num 10855) dirty exists 2020-06-12T02:39:13.375 INFO:tasks.rados.rados.0.smithi107.stdout:10856: left oid 2 (ObjNum 10855 snap 0 seq_num 10855) 2020-06-12T02:39:13.376 INFO:tasks.rados.rados.0.smithi107.stdout:10856: done (2 left) 2020-06-12T02:39:13.376 INFO:tasks.rados.rados.0.smithi107.stdout:10859: done (1 left) 2020-06-12T02:39:13.376 INFO:tasks.rados.rados.0.smithi107.stdout:10860: delete oid 15 current snap is 0 2020-06-12T02:39:13.376 INFO:tasks.rados.rados.0.smithi107.stdout:10858: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:13.377 INFO:tasks.rados.rados.0.smithi107.stdout:10858: finishing write tid 2 to smithi10729407-11 2020-06-12T02:39:13.384 INFO:tasks.rados.rados.0.smithi107.stdout:10860: done (1 left) 2020-06-12T02:39:13.384 INFO:tasks.rados.rados.0.smithi107.stdout:10861: delete oid 4 current snap is 0 2020-06-12T02:39:13.384 INFO:tasks.rados.rados.0.smithi107.stdout:10858: finishing write tid 3 to smithi10729407-11 2020-06-12T02:39:13.384 INFO:tasks.rados.rados.0.smithi107.stdout:10858: finishing write tid 5 to smithi10729407-11 2020-06-12T02:39:13.384 INFO:tasks.rados.rados.0.smithi107.stdout:10858: finishing write tid 6 to smithi10729407-11 2020-06-12T02:39:13.385 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1745 (ObjNum 10857 snap 0 seq_num 10857) dirty exists 2020-06-12T02:39:13.385 INFO:tasks.rados.rados.0.smithi107.stdout:10858: left oid 11 (ObjNum 10857 snap 0 seq_num 10857) 2020-06-12T02:39:13.385 INFO:tasks.rados.rados.0.smithi107.stdout:10858: done (1 left) 2020-06-12T02:39:13.385 INFO:tasks.rados.rados.0.smithi107.stdout:10861: done (0 left) 2020-06-12T02:39:13.385 INFO:tasks.rados.rados.0.smithi107.stdout:10862: delete oid 18 current snap is 0 2020-06-12T02:39:13.386 INFO:tasks.rados.rados.0.smithi107.stdout:10862: done (0 left) 2020-06-12T02:39:13.386 INFO:tasks.rados.rados.0.smithi107.stdout:10863: delete oid 9 current snap is 0 2020-06-12T02:39:13.388 INFO:tasks.rados.rados.0.smithi107.stdout:10863: done (0 left) 2020-06-12T02:39:13.388 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:39:13.389 INFO:tasks.rados.rados.0.smithi107.stdout:10864: seq_num 10863 ranges {0=737280,737280=442368,1179648=516096,1695744=245760} 2020-06-12T02:39:13.392 INFO:tasks.rados.rados.0.smithi107.stdout:10864: writing smithi10729407-6 from 0 to 737280 tid 1 2020-06-12T02:39:13.395 INFO:tasks.rados.rados.0.smithi107.stdout:10864: writing smithi10729407-6 from 737280 to 1179648 tid 2 2020-06-12T02:39:13.398 INFO:tasks.rados.rados.0.smithi107.stdout:10864: writing smithi10729407-6 from 1179648 to 1695744 tid 3 2020-06-12T02:39:13.400 INFO:tasks.rados.rados.0.smithi107.stdout:10864: writing smithi10729407-6 from 1695744 to 1941504 tid 4 2020-06-12T02:39:13.400 INFO:tasks.rados.rados.0.smithi107.stdout:10865: delete oid 4 current snap is 0 2020-06-12T02:39:13.400 INFO:tasks.rados.rados.0.smithi107.stdout:10864: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:13.403 INFO:tasks.rados.rados.0.smithi107.stdout:10865: done (1 left) 2020-06-12T02:39:13.403 INFO:tasks.rados.rados.0.smithi107.stdout:10866: delete oid 10 current snap is 0 2020-06-12T02:39:13.406 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:39:13.403+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:39:13.407 INFO:tasks.rados.rados.0.smithi107.stdout:10866: done (1 left) 2020-06-12T02:39:13.408 INFO:tasks.rados.rados.0.smithi107.stdout:10867: delete oid 19 current snap is 0 2020-06-12T02:39:13.408 INFO:tasks.rados.rados.0.smithi107.stdout:10864: finishing write tid 2 to smithi10729407-6 2020-06-12T02:39:13.410 INFO:tasks.rados.rados.0.smithi107.stdout:10867: done (1 left) 2020-06-12T02:39:13.411 INFO:tasks.rados.rados.0.smithi107.stdout:10868: delete oid 8 current snap is 0 2020-06-12T02:39:13.411 INFO:tasks.rados.rados.0.smithi107.stdout:10864: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:13.412 INFO:tasks.rados.rados.0.smithi107.stdout:10864: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:13.414 INFO:tasks.rados.rados.0.smithi107.stdout:10868: done (1 left) 2020-06-12T02:39:13.414 INFO:tasks.rados.rados.0.smithi107.stdout:10869: delete oid 12 current snap is 0 2020-06-12T02:39:13.414 INFO:tasks.rados.rados.0.smithi107.stdout:10864: finishing write tid 6 to smithi10729407-6 2020-06-12T02:39:13.416 INFO:tasks.rados.rados.0.smithi107.stdout:10864: finishing write tid 7 to smithi10729407-6 2020-06-12T02:39:13.416 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1956 (ObjNum 10863 snap 0 seq_num 10863) dirty exists 2020-06-12T02:39:13.417 INFO:tasks.rados.rados.0.smithi107.stdout:10864: left oid 6 (ObjNum 10863 snap 0 seq_num 10863) 2020-06-12T02:39:13.417 INFO:tasks.rados.rados.0.smithi107.stdout:10864: done (1 left) 2020-06-12T02:39:13.418 INFO:tasks.rados.rados.0.smithi107.stdout:10869: done (0 left) 2020-06-12T02:39:13.418 INFO:tasks.rados.rados.0.smithi107.stdout:10870: delete oid 14 current snap is 0 2020-06-12T02:39:13.422 INFO:tasks.rados.rados.0.smithi107.stdout:10870: done (0 left) 2020-06-12T02:39:13.422 INFO:tasks.rados.rados.0.smithi107.stdout:10871: delete oid 1 current snap is 0 2020-06-12T02:39:13.425 INFO:tasks.rados.rados.0.smithi107.stdout:10871: done (0 left) 2020-06-12T02:39:13.425 INFO:tasks.rados.rados.0.smithi107.stdout:10872: delete oid 2 current snap is 0 2020-06-12T02:39:13.427 INFO:tasks.rados.rados.0.smithi107.stdout:10872: done (0 left) 2020-06-12T02:39:13.427 INFO:tasks.rados.rados.0.smithi107.stdout:10873: delete oid 12 current snap is 0 2020-06-12T02:39:13.429 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:13.429 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:13.430 INFO:tasks.rados.rados.0.smithi107.stdout:10873: done (0 left) 2020-06-12T02:39:13.430 INFO:tasks.rados.rados.0.smithi107.stdout:10874: delete oid 18 current snap is 0 2020-06-12T02:39:13.432 INFO:tasks.rados.rados.0.smithi107.stdout:10874: done (0 left) 2020-06-12T02:39:13.433 INFO:tasks.rados.rados.0.smithi107.stdout:10875: delete oid 11 current snap is 0 2020-06-12T02:39:13.437 INFO:tasks.rados.rados.0.smithi107.stdout:10875: done (0 left) 2020-06-12T02:39:13.437 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:39:13.438 INFO:tasks.rados.rados.0.smithi107.stdout:10876: seq_num 10875 ranges {0=540672,540672=761856,1302528=458752,1761280=385024} 2020-06-12T02:39:13.440 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-06-12T02:39:13.441 INFO:tasks.rados.rados.0.smithi107.stdout:10876: writing smithi10729407-12 from 0 to 540672 tid 1 2020-06-12T02:39:13.445 INFO:tasks.rados.rados.0.smithi107.stdout:10876: writing smithi10729407-12 from 540672 to 1302528 tid 2 2020-06-12T02:39:13.448 INFO:tasks.rados.rados.0.smithi107.stdout:10876: writing smithi10729407-12 from 1302528 to 1761280 tid 3 2020-06-12T02:39:13.450 INFO:tasks.rados.rados.0.smithi107.stdout:10876: writing smithi10729407-12 from 1761280 to 2146304 tid 4 2020-06-12T02:39:13.451 INFO:tasks.rados.rados.0.smithi107.stdout:10877: delete oid 20 current snap is 0 2020-06-12T02:39:13.454 INFO:tasks.rados.rados.0.smithi107.stdout:10877: done (1 left) 2020-06-12T02:39:13.454 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:39:13.454 INFO:tasks.rados.rados.0.smithi107.stdout:10878: seq_num 10877 ranges {0=573440,573440=524288,1097728=425984,1523712=589824,2113536=65536} 2020-06-12T02:39:13.457 INFO:tasks.rados.rados.0.smithi107.stdout:10878: writing smithi10729407-17 from 0 to 573440 tid 1 2020-06-12T02:39:13.460 INFO:tasks.rados.rados.0.smithi107.stdout:10878: writing smithi10729407-17 from 573440 to 1097728 tid 2 2020-06-12T02:39:13.463 INFO:tasks.rados.rados.0.smithi107.stdout:10878: writing smithi10729407-17 from 1097728 to 1523712 tid 3 2020-06-12T02:39:13.466 INFO:tasks.rados.rados.0.smithi107.stdout:10878: writing smithi10729407-17 from 1523712 to 2113536 tid 4 2020-06-12T02:39:13.467 INFO:tasks.rados.rados.0.smithi107.stdout:10878: writing smithi10729407-17 from 2113536 to 2179072 tid 5 2020-06-12T02:39:13.467 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:39:13.467 INFO:tasks.rados.rados.0.smithi107.stdout:10879: seq_num 10878 ranges {0=589824,589824=409600,999424=630784,1630208=524288,2154496=40960} 2020-06-12T02:39:13.470 INFO:tasks.rados.rados.0.smithi107.stdout:10879: writing smithi10729407-10 from 0 to 589824 tid 1 2020-06-12T02:39:13.472 INFO:tasks.rados.rados.0.smithi107.stdout:10879: writing smithi10729407-10 from 589824 to 999424 tid 2 2020-06-12T02:39:13.476 INFO:tasks.rados.rados.0.smithi107.stdout:10879: writing smithi10729407-10 from 999424 to 1630208 tid 3 2020-06-12T02:39:13.479 INFO:tasks.rados.rados.0.smithi107.stdout:10879: writing smithi10729407-10 from 1630208 to 2154496 tid 4 2020-06-12T02:39:13.480 INFO:tasks.rados.rados.0.smithi107.stdout:10879: writing smithi10729407-10 from 2154496 to 2195456 tid 5 2020-06-12T02:39:13.480 INFO:tasks.rados.rados.0.smithi107.stdout:10880: delete oid 6 current snap is 0 2020-06-12T02:39:13.480 INFO:tasks.rados.rados.0.smithi107.stdout:10876: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:13.481 INFO:tasks.rados.rados.0.smithi107.stdout:10876: finishing write tid 2 to smithi10729407-12 2020-06-12T02:39:13.481 INFO:tasks.rados.rados.0.smithi107.stdout:10876: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:13.481 INFO:tasks.rados.rados.0.smithi107.stdout:10876: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:13.481 INFO:tasks.rados.rados.0.smithi107.stdout:10876: finishing write tid 6 to smithi10729407-12 2020-06-12T02:39:13.482 INFO:tasks.rados.rados.0.smithi107.stdout:10876: finishing write tid 7 to smithi10729407-12 2020-06-12T02:39:13.482 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 2008 (ObjNum 10875 snap 0 seq_num 10875) dirty exists 2020-06-12T02:39:13.482 INFO:tasks.rados.rados.0.smithi107.stdout:10876: left oid 12 (ObjNum 10875 snap 0 seq_num 10875) 2020-06-12T02:39:13.483 INFO:tasks.rados.rados.0.smithi107.stdout:10878: finishing write tid 1 to smithi10729407-17 2020-06-12T02:39:13.484 INFO:tasks.rados.rados.0.smithi107.stdout:10878: finishing write tid 2 to smithi10729407-17 2020-06-12T02:39:13.484 INFO:tasks.rados.rados.0.smithi107.stdout:10878: finishing write tid 3 to smithi10729407-17 2020-06-12T02:39:13.484 INFO:tasks.rados.rados.0.smithi107.stdout:10878: finishing write tid 4 to smithi10729407-17 2020-06-12T02:39:13.484 INFO:tasks.rados.rados.0.smithi107.stdout:10878: finishing write tid 5 to smithi10729407-17 2020-06-12T02:39:13.485 INFO:tasks.rados.rados.0.smithi107.stdout:10879: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:13.485 INFO:tasks.rados.rados.0.smithi107.stdout:10878: finishing write tid 7 to smithi10729407-17 2020-06-12T02:39:13.486 INFO:tasks.rados.rados.0.smithi107.stdout:10878: finishing write tid 8 to smithi10729407-17 2020-06-12T02:39:13.486 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 2036 (ObjNum 10877 snap 0 seq_num 10877) dirty exists 2020-06-12T02:39:13.486 INFO:tasks.rados.rados.0.smithi107.stdout:10878: left oid 17 (ObjNum 10877 snap 0 seq_num 10877) 2020-06-12T02:39:13.486 INFO:tasks.rados.rados.0.smithi107.stdout:10876: done (3 left) 2020-06-12T02:39:13.487 INFO:tasks.rados.rados.0.smithi107.stdout:10878: done (2 left) 2020-06-12T02:39:13.487 INFO:tasks.rados.rados.0.smithi107.stdout:10880: done (1 left) 2020-06-12T02:39:13.487 INFO:tasks.rados.rados.0.smithi107.stdout:10881: delete oid 12 current snap is 0 2020-06-12T02:39:13.490 INFO:tasks.rados.rados.0.smithi107.stdout:10881: done (1 left) 2020-06-12T02:39:13.490 INFO:tasks.rados.rados.0.smithi107.stdout:10882: delete oid 20 current snap is 0 2020-06-12T02:39:13.490 INFO:tasks.rados.rados.0.smithi107.stdout:10879: finishing write tid 2 to smithi10729407-10 2020-06-12T02:39:13.491 INFO:tasks.rados.rados.0.smithi107.stdout:10879: finishing write tid 3 to smithi10729407-10 2020-06-12T02:39:13.492 INFO:tasks.rados.rados.0.smithi107.stdout:10879: finishing write tid 4 to smithi10729407-10 2020-06-12T02:39:13.492 INFO:tasks.rados.rados.0.smithi107.stdout:10879: finishing write tid 5 to smithi10729407-10 2020-06-12T02:39:13.494 INFO:tasks.rados.rados.0.smithi107.stdout:10879: finishing write tid 7 to smithi10729407-10 2020-06-12T02:39:13.495 INFO:tasks.rados.rados.0.smithi107.stdout:10882: done (1 left) 2020-06-12T02:39:13.495 INFO:tasks.rados.rados.0.smithi107.stdout:10883: delete oid 4 current snap is 0 2020-06-12T02:39:13.496 INFO:tasks.rados.rados.0.smithi107.stdout:10879: finishing write tid 8 to smithi10729407-10 2020-06-12T02:39:13.496 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 800 (ObjNum 10878 snap 0 seq_num 10878) dirty exists 2020-06-12T02:39:13.497 INFO:tasks.rados.rados.0.smithi107.stdout:10879: left oid 10 (ObjNum 10878 snap 0 seq_num 10878) 2020-06-12T02:39:13.500 INFO:tasks.rados.rados.0.smithi107.stdout:10879: done (1 left) 2020-06-12T02:39:13.501 INFO:tasks.rados.rados.0.smithi107.stdout:10883: done (0 left) 2020-06-12T02:39:13.501 INFO:tasks.rados.rados.0.smithi107.stdout:10884: delete oid 2 current snap is 0 2020-06-12T02:39:13.502 INFO:tasks.rados.rados.0.smithi107.stdout:10884: done (0 left) 2020-06-12T02:39:13.502 INFO:tasks.rados.rados.0.smithi107.stdout:10885: delete oid 12 current snap is 0 2020-06-12T02:39:13.507 INFO:tasks.rados.rados.0.smithi107.stdout:10885: done (0 left) 2020-06-12T02:39:13.508 INFO:tasks.rados.rados.0.smithi107.stdout:10886: delete oid 16 current snap is 0 2020-06-12T02:39:13.508 INFO:tasks.rados.rados.0.smithi107.stdout:10886: done (0 left) 2020-06-12T02:39:13.508 INFO:tasks.rados.rados.0.smithi107.stdout:10887: delete oid 9 current snap is 0 2020-06-12T02:39:13.509 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:39:13.505+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:39:13.509 INFO:tasks.rados.rados.0.smithi107.stdout:10887: done (0 left) 2020-06-12T02:39:13.509 INFO:tasks.rados.rados.0.smithi107.stdout:10888: delete oid 11 current snap is 0 2020-06-12T02:39:13.511 INFO:tasks.rados.rados.0.smithi107.stdout:10888: done (0 left) 2020-06-12T02:39:13.511 INFO:tasks.rados.rados.0.smithi107.stdout:10889: delete oid 11 current snap is 0 2020-06-12T02:39:13.514 INFO:tasks.rados.rados.0.smithi107.stdout:10889: done (0 left) 2020-06-12T02:39:13.514 INFO:tasks.rados.rados.0.smithi107.stdout:10890: delete oid 13 current snap is 0 2020-06-12T02:39:13.516 INFO:tasks.rados.rados.0.smithi107.stdout:10890: done (0 left) 2020-06-12T02:39:13.517 INFO:tasks.rados.rados.0.smithi107.stdout:10891: delete oid 17 current snap is 0 2020-06-12T02:39:13.519 INFO:tasks.rados.rados.0.smithi107.stdout:10891: done (0 left) 2020-06-12T02:39:13.519 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:39:13.519 INFO:tasks.rados.rados.0.smithi107.stdout:10892: seq_num 10891 ranges {0=8192} 2020-06-12T02:39:13.520 INFO:tasks.rados.rados.0.smithi107.stdout:10892: writing smithi10729407-19 from 0 to 8192 tid 1 2020-06-12T02:39:13.520 INFO:tasks.rados.rados.0.smithi107.stdout:10893: delete oid 6 current snap is 0 2020-06-12T02:39:13.528 INFO:tasks.rados.rados.0.smithi107.stdout:10892: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:13.529 INFO:tasks.rados.rados.0.smithi107.stdout:10893: done (1 left) 2020-06-12T02:39:13.529 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:39:13.529 INFO:tasks.rados.rados.0.smithi107.stdout:10894: seq_num 10893 ranges {0=40960} 2020-06-12T02:39:13.529 INFO:tasks.rados.rados.0.smithi107.stdout:10894: writing smithi10729407-12 from 0 to 40960 tid 1 2020-06-12T02:39:13.529 INFO:tasks.rados.rados.0.smithi107.stdout:10895: delete oid 2 current snap is 0 2020-06-12T02:39:13.530 INFO:tasks.rados.rados.0.smithi107.stdout:10892: finishing write tid 3 to smithi10729407-19 2020-06-12T02:39:13.530 INFO:tasks.rados.rados.0.smithi107.stdout:10895: done (2 left) 2020-06-12T02:39:13.530 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:39:13.530 INFO:tasks.rados.rados.0.smithi107.stdout:10896: seq_num 10895 ranges {0=73728} 2020-06-12T02:39:13.530 INFO:tasks.rados.rados.0.smithi107.stdout:10896: writing smithi10729407-2 from 0 to 73728 tid 1 2020-06-12T02:39:13.531 INFO:tasks.rados.rados.0.smithi107.stdout:10897: delete oid 10 current snap is 0 2020-06-12T02:39:13.531 INFO:tasks.rados.rados.0.smithi107.stdout:10892: finishing write tid 4 to smithi10729407-19 2020-06-12T02:39:13.531 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 2012 (ObjNum 10891 snap 0 seq_num 10891) dirty exists 2020-06-12T02:39:13.531 INFO:tasks.rados.rados.0.smithi107.stdout:10892: left oid 19 (ObjNum 10891 snap 0 seq_num 10891) 2020-06-12T02:39:13.531 INFO:tasks.rados.rados.0.smithi107.stdout:10894: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:13.532 INFO:tasks.rados.rados.0.smithi107.stdout:10896: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:13.532 INFO:tasks.rados.rados.0.smithi107.stdout:10892: done (3 left) 2020-06-12T02:39:13.532 INFO:tasks.rados.rados.0.smithi107.stdout:10897: done (2 left) 2020-06-12T02:39:13.532 INFO:tasks.rados.rados.0.smithi107.stdout:10898: delete oid 20 current snap is 0 2020-06-12T02:39:13.533 INFO:tasks.rados.rados.0.smithi107.stdout:10896: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:13.534 INFO:tasks.rados.rados.0.smithi107.stdout:10894: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:13.534 INFO:tasks.rados.rados.0.smithi107.stdout:10898: done (2 left) 2020-06-12T02:39:13.534 INFO:tasks.rados.rados.0.smithi107.stdout:10899: delete oid 11 current snap is 0 2020-06-12T02:39:13.538 INFO:tasks.rados.rados.0.smithi107.stdout:10894: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:13.538 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 2014 (ObjNum 10893 snap 0 seq_num 10893) dirty exists 2020-06-12T02:39:13.539 INFO:tasks.rados.rados.0.smithi107.stdout:10894: left oid 12 (ObjNum 10893 snap 0 seq_num 10893) 2020-06-12T02:39:13.539 INFO:tasks.rados.rados.0.smithi107.stdout:10896: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:13.539 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 1023 (ObjNum 10895 snap 0 seq_num 10895) dirty exists 2020-06-12T02:39:13.539 INFO:tasks.rados.rados.0.smithi107.stdout:10896: left oid 2 (ObjNum 10895 snap 0 seq_num 10895) 2020-06-12T02:39:13.542 INFO:tasks.rados.rados.0.smithi107.stdout:10894: done (2 left) 2020-06-12T02:39:13.545 INFO:tasks.rados.rados.0.smithi107.stdout:10896: done (1 left) 2020-06-12T02:39:13.545 INFO:tasks.rados.rados.0.smithi107.stdout:10899: done (0 left) 2020-06-12T02:39:13.545 INFO:tasks.rados.rados.0.smithi107.stdout:10900: delete oid 3 current snap is 0 2020-06-12T02:39:13.546 INFO:tasks.rados.rados.0.smithi107.stdout:10900: done (0 left) 2020-06-12T02:39:13.546 INFO:tasks.rados.rados.0.smithi107.stdout:10901: delete oid 11 current snap is 0 2020-06-12T02:39:13.548 INFO:tasks.rados.rados.0.smithi107.stdout:10901: done (0 left) 2020-06-12T02:39:13.549 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:39:13.549 INFO:tasks.rados.rados.0.smithi107.stdout:10902: seq_num 10901 ranges {0=172032} 2020-06-12T02:39:13.549 INFO:tasks.rados.rados.0.smithi107.stdout:10902: writing smithi10729407-3 from 0 to 172032 tid 1 2020-06-12T02:39:13.549 INFO:tasks.rados.rados.0.smithi107.stdout:10903: delete oid 13 current snap is 0 2020-06-12T02:39:13.551 INFO:tasks.rados.rados.0.smithi107.stdout:10903: done (1 left) 2020-06-12T02:39:13.554 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:39:13.554 INFO:tasks.rados.rados.0.smithi107.stdout:10904: seq_num 10903 ranges {0=204800} 2020-06-12T02:39:13.556 INFO:tasks.rados.rados.0.smithi107.stdout:10904: writing smithi10729407-20 from 0 to 204800 tid 1 2020-06-12T02:39:13.556 INFO:tasks.rados.rados.0.smithi107.stdout:10905: delete oid 18 current snap is 0 2020-06-12T02:39:13.556 INFO:tasks.rados.rados.0.smithi107.stdout:10902: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:13.557 INFO:tasks.rados.rados.0.smithi107.stdout:10905: done (2 left) 2020-06-12T02:39:13.557 INFO:tasks.rados.rados.0.smithi107.stdout:10906: delete oid 12 current snap is 0 2020-06-12T02:39:13.557 INFO:tasks.rados.rados.0.smithi107.stdout:10902: finishing write tid 3 to smithi10729407-3 2020-06-12T02:39:13.559 INFO:tasks.rados.rados.0.smithi107.stdout:10902: finishing write tid 4 to smithi10729407-3 2020-06-12T02:39:13.559 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1934 (ObjNum 10901 snap 0 seq_num 10901) dirty exists 2020-06-12T02:39:13.559 INFO:tasks.rados.rados.0.smithi107.stdout:10902: left oid 3 (ObjNum 10901 snap 0 seq_num 10901) 2020-06-12T02:39:13.559 INFO:tasks.rados.rados.0.smithi107.stdout:10904: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:13.560 INFO:tasks.rados.rados.0.smithi107.stdout:10904: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:13.560 INFO:tasks.rados.rados.0.smithi107.stdout:10902: done (2 left) 2020-06-12T02:39:13.561 INFO:tasks.rados.rados.0.smithi107.stdout:10906: done (1 left) 2020-06-12T02:39:13.561 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:39:13.561 INFO:tasks.rados.rados.0.smithi107.stdout:10907: seq_num 10906 ranges {0=262144} 2020-06-12T02:39:13.562 INFO:tasks.rados.rados.0.smithi107.stdout:10907: writing smithi10729407-16 from 0 to 262144 tid 1 2020-06-12T02:39:13.562 INFO:tasks.rados.rados.0.smithi107.stdout:10908: delete oid 12 current snap is 0 2020-06-12T02:39:13.562 INFO:tasks.rados.rados.0.smithi107.stdout:10904: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:13.562 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 843 (ObjNum 10903 snap 0 seq_num 10903) dirty exists 2020-06-12T02:39:13.563 INFO:tasks.rados.rados.0.smithi107.stdout:10904: left oid 20 (ObjNum 10903 snap 0 seq_num 10903) 2020-06-12T02:39:13.565 INFO:tasks.rados.rados.0.smithi107.stdout:10904: done (2 left) 2020-06-12T02:39:13.565 INFO:tasks.rados.rados.0.smithi107.stdout:10908: done (1 left) 2020-06-12T02:39:13.565 INFO:tasks.rados.rados.0.smithi107.stdout:10909: delete oid 15 current snap is 0 2020-06-12T02:39:13.566 INFO:tasks.rados.rados.0.smithi107.stdout:10907: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:13.567 INFO:tasks.rados.rados.0.smithi107.stdout:10907: finishing write tid 3 to smithi10729407-16 2020-06-12T02:39:13.568 INFO:tasks.rados.rados.0.smithi107.stdout:10909: done (1 left) 2020-06-12T02:39:13.569 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:39:13.569 INFO:tasks.rados.rados.0.smithi107.stdout:10910: seq_num 10909 ranges {8192=311296} 2020-06-12T02:39:13.570 INFO:tasks.rados.rados.0.smithi107.stdout:10910: writing smithi10729407-19 from 8192 to 319488 tid 1 2020-06-12T02:39:13.570 INFO:tasks.rados.rados.0.smithi107.stdout:10911: delete oid 2 current snap is 0 2020-06-12T02:39:13.571 INFO:tasks.rados.rados.0.smithi107.stdout:10907: finishing write tid 4 to smithi10729407-16 2020-06-12T02:39:13.571 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1936 (ObjNum 10906 snap 0 seq_num 10906) dirty exists 2020-06-12T02:39:13.571 INFO:tasks.rados.rados.0.smithi107.stdout:10907: left oid 16 (ObjNum 10906 snap 0 seq_num 10906) 2020-06-12T02:39:13.572 INFO:tasks.rados.rados.0.smithi107.stdout:10907: done (2 left) 2020-06-12T02:39:13.572 INFO:tasks.rados.rados.0.smithi107.stdout:10911: done (1 left) 2020-06-12T02:39:13.573 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:39:13.573 INFO:tasks.rados.rados.0.smithi107.stdout:10912: seq_num 10911 ranges {0=344064} 2020-06-12T02:39:13.574 INFO:tasks.rados.rados.0.smithi107.stdout:10912: writing smithi10729407-2 from 0 to 344064 tid 1 2020-06-12T02:39:13.575 INFO:tasks.rados.rados.0.smithi107.stdout:10913: delete oid 15 current snap is 0 2020-06-12T02:39:13.579 INFO:tasks.rados.rados.0.smithi107.stdout:10910: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:13.580 INFO:tasks.rados.rados.0.smithi107.stdout:10913: done (2 left) 2020-06-12T02:39:13.580 INFO:tasks.rados.rados.0.smithi107.stdout:10914: delete oid 1 current snap is 0 2020-06-12T02:39:13.580 INFO:tasks.rados.rados.0.smithi107.stdout:10910: finishing write tid 3 to smithi10729407-19 2020-06-12T02:39:13.580 INFO:tasks.rados.rados.0.smithi107.stdout:10910: finishing write tid 4 to smithi10729407-19 2020-06-12T02:39:13.581 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 2018 (ObjNum 10909 snap 0 seq_num 10909) dirty exists 2020-06-12T02:39:13.581 INFO:tasks.rados.rados.0.smithi107.stdout:10910: left oid 19 (ObjNum 10909 snap 0 seq_num 10909) 2020-06-12T02:39:13.581 INFO:tasks.rados.rados.0.smithi107.stdout:10912: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:13.585 INFO:tasks.rados.rados.0.smithi107.stdout:10912: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:13.585 INFO:tasks.rados.rados.0.smithi107.stdout:10910: done (2 left) 2020-06-12T02:39:13.585 INFO:tasks.rados.rados.0.smithi107.stdout:10914: done (1 left) 2020-06-12T02:39:13.585 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:39:13.586 INFO:tasks.rados.rados.0.smithi107.stdout:10915: seq_num 10914 ranges {0=393216} 2020-06-12T02:39:13.590 INFO:tasks.rados.rados.0.smithi107.stdout:10915: writing smithi10729407-17 from 0 to 393216 tid 1 2020-06-12T02:39:13.590 INFO:tasks.rados.rados.0.smithi107.stdout:10916: delete oid 7 current snap is 0 2020-06-12T02:39:13.590 INFO:tasks.rados.rados.0.smithi107.stdout:10912: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:13.590 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 1026 (ObjNum 10911 snap 0 seq_num 10911) dirty exists 2020-06-12T02:39:13.591 INFO:tasks.rados.rados.0.smithi107.stdout:10912: left oid 2 (ObjNum 10911 snap 0 seq_num 10911) 2020-06-12T02:39:13.591 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:13.591 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:13.592 INFO:tasks.rados.rados.0.smithi107.stdout:10912: done (2 left) 2020-06-12T02:39:13.592 INFO:tasks.rados.rados.0.smithi107.stdout:10916: done (1 left) 2020-06-12T02:39:13.592 INFO:tasks.rados.rados.0.smithi107.stdout:10917: delete oid 2 current snap is 0 2020-06-12T02:39:13.592 INFO:tasks.rados.rados.0.smithi107.stdout:10917: done (1 left) 2020-06-12T02:39:13.592 INFO:tasks.rados.rados.0.smithi107.stdout:10918: delete oid 11 current snap is 0 2020-06-12T02:39:13.593 INFO:tasks.rados.rados.0.smithi107.stdout:10915: finishing write tid 1 to smithi10729407-17 2020-06-12T02:39:13.596 INFO:tasks.rados.rados.0.smithi107.stdout:10918: done (1 left) 2020-06-12T02:39:13.596 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:39:13.597 INFO:tasks.rados.rados.0.smithi107.stdout:10919: seq_num 10918 ranges {0=458752} 2020-06-12T02:39:13.599 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-06-12T02:39:13.599 INFO:tasks.rados.rados.0.smithi107.stdout:10919: writing smithi10729407-7 from 0 to 458752 tid 1 2020-06-12T02:39:13.599 INFO:tasks.rados.rados.0.smithi107.stdout:10920: delete oid 4 current snap is 0 2020-06-12T02:39:13.600 INFO:tasks.rados.rados.0.smithi107.stdout:10915: finishing write tid 3 to smithi10729407-17 2020-06-12T02:39:13.600 INFO:tasks.rados.rados.0.smithi107.stdout:10915: finishing write tid 4 to smithi10729407-17 2020-06-12T02:39:13.600 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 2040 (ObjNum 10914 snap 0 seq_num 10914) dirty exists 2020-06-12T02:39:13.600 INFO:tasks.rados.rados.0.smithi107.stdout:10915: left oid 17 (ObjNum 10914 snap 0 seq_num 10914) 2020-06-12T02:39:13.602 INFO:tasks.rados.rados.0.smithi107.stdout:10915: done (2 left) 2020-06-12T02:39:13.602 INFO:tasks.rados.rados.0.smithi107.stdout:10920: done (1 left) 2020-06-12T02:39:13.602 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:39:13.603 INFO:tasks.rados.rados.0.smithi107.stdout:10921: seq_num 10920 ranges {0=491520} 2020-06-12T02:39:13.605 INFO:tasks.rados.rados.0.smithi107.stdout:10921: writing smithi10729407-12 from 0 to 491520 tid 1 2020-06-12T02:39:13.605 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:39:13.605 INFO:tasks.rados.rados.0.smithi107.stdout:10922: seq_num 10921 ranges {0=507904} 2020-06-12T02:39:13.608 INFO:tasks.rados.rados.0.smithi107.stdout:10922: writing smithi10729407-8 from 0 to 507904 tid 1 2020-06-12T02:39:13.609 INFO:tasks.rados.rados.0.smithi107.stdout:10923: delete oid 15 current snap is 0 2020-06-12T02:39:13.609 INFO:tasks.rados.rados.0.smithi107.stdout:10919: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:13.609 INFO:tasks.rados.rados.0.smithi107.stdout:10919: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:13.609 INFO:tasks.rados.rados.0.smithi107.stdout:10919: finishing write tid 4 to smithi10729407-7 2020-06-12T02:39:13.609 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1787 (ObjNum 10918 snap 0 seq_num 10918) dirty exists 2020-06-12T02:39:13.610 INFO:tasks.rados.rados.0.smithi107.stdout:10919: left oid 7 (ObjNum 10918 snap 0 seq_num 10918) 2020-06-12T02:39:13.610 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:39:13.605+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:39:13.611 INFO:tasks.rados.rados.0.smithi107.stdout:10921: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:13.615 INFO:tasks.rados.rados.0.smithi107.stdout:10921: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:13.617 INFO:tasks.rados.rados.0.smithi107.stdout:10922: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:13.618 INFO:tasks.rados.rados.0.smithi107.stdout:10921: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:13.618 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 2020 (ObjNum 10920 snap 0 seq_num 10920) dirty exists 2020-06-12T02:39:13.619 INFO:tasks.rados.rados.0.smithi107.stdout:10921: left oid 12 (ObjNum 10920 snap 0 seq_num 10920) 2020-06-12T02:39:13.619 INFO:tasks.rados.rados.0.smithi107.stdout:10922: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:13.619 INFO:tasks.rados.rados.0.smithi107.stdout:10919: done (3 left) 2020-06-12T02:39:13.620 INFO:tasks.rados.rados.0.smithi107.stdout:10921: done (2 left) 2020-06-12T02:39:13.620 INFO:tasks.rados.rados.0.smithi107.stdout:10923: done (1 left) 2020-06-12T02:39:13.620 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:39:13.620 INFO:tasks.rados.rados.0.smithi107.stdout:10924: seq_num 10923 ranges {0=548864} 2020-06-12T02:39:13.622 INFO:tasks.rados.rados.0.smithi107.stdout:10924: writing smithi10729407-14 from 0 to 548864 tid 1 2020-06-12T02:39:13.622 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:39:13.623 INFO:tasks.rados.rados.0.smithi107.stdout:10925: seq_num 10924 ranges {0=565248} 2020-06-12T02:39:13.626 INFO:tasks.rados.rados.0.smithi107.stdout:10925: writing smithi10729407-18 from 0 to 565248 tid 1 2020-06-12T02:39:13.626 INFO:tasks.rados.rados.0.smithi107.stdout:10926: delete oid 5 current snap is 0 2020-06-12T02:39:13.626 INFO:tasks.rados.rados.0.smithi107.stdout:10922: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:13.626 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 2021 (ObjNum 10921 snap 0 seq_num 10921) dirty exists 2020-06-12T02:39:13.626 INFO:tasks.rados.rados.0.smithi107.stdout:10922: left oid 8 (ObjNum 10921 snap 0 seq_num 10921) 2020-06-12T02:39:13.628 INFO:tasks.rados.rados.0.smithi107.stdout:10924: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:13.628 INFO:tasks.rados.rados.0.smithi107.stdout:10922: done (3 left) 2020-06-12T02:39:13.628 INFO:tasks.rados.rados.0.smithi107.stdout:10926: done (2 left) 2020-06-12T02:39:13.629 INFO:tasks.rados.rados.0.smithi107.stdout:10927: delete oid 17 current snap is 0 2020-06-12T02:39:13.631 INFO:tasks.rados.rados.0.smithi107.stdout:10924: finishing write tid 3 to smithi10729407-14 2020-06-12T02:39:13.633 INFO:tasks.rados.rados.0.smithi107.stdout:10924: finishing write tid 4 to smithi10729407-14 2020-06-12T02:39:13.633 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1726 (ObjNum 10923 snap 0 seq_num 10923) dirty exists 2020-06-12T02:39:13.634 INFO:tasks.rados.rados.0.smithi107.stdout:10924: left oid 14 (ObjNum 10923 snap 0 seq_num 10923) 2020-06-12T02:39:13.634 INFO:tasks.rados.rados.0.smithi107.stdout:10925: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:13.636 INFO:tasks.rados.rados.0.smithi107.stdout:10925: finishing write tid 3 to smithi10729407-18 2020-06-12T02:39:13.636 INFO:tasks.rados.rados.0.smithi107.stdout:10924: done (2 left) 2020-06-12T02:39:13.637 INFO:tasks.rados.rados.0.smithi107.stdout:10927: done (1 left) 2020-06-12T02:39:13.637 INFO:tasks.rados.rados.0.smithi107.stdout:10928: delete oid 8 current snap is 0 2020-06-12T02:39:13.638 INFO:tasks.rados.rados.0.smithi107.stdout:10925: finishing write tid 4 to smithi10729407-18 2020-06-12T02:39:13.639 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1874 (ObjNum 10924 snap 0 seq_num 10924) dirty exists 2020-06-12T02:39:13.639 INFO:tasks.rados.rados.0.smithi107.stdout:10925: left oid 18 (ObjNum 10924 snap 0 seq_num 10924) 2020-06-12T02:39:13.642 INFO:tasks.rados.rados.0.smithi107.stdout:10925: done (1 left) 2020-06-12T02:39:13.642 INFO:tasks.rados.rados.0.smithi107.stdout:10928: done (0 left) 2020-06-12T02:39:13.642 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:39:13.643 INFO:tasks.rados.rados.0.smithi107.stdout:10929: seq_num 10928 ranges {0=630784} 2020-06-12T02:39:13.646 INFO:tasks.rados.rados.0.smithi107.stdout:10929: writing smithi10729407-10 from 0 to 630784 tid 1 2020-06-12T02:39:13.646 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:39:13.646 INFO:tasks.rados.rados.0.smithi107.stdout:10930: seq_num 10929 ranges {0=647168} 2020-06-12T02:39:13.649 INFO:tasks.rados.rados.0.smithi107.stdout:10930: writing smithi10729407-1 from 0 to 647168 tid 1 2020-06-12T02:39:13.650 INFO:tasks.rados.rados.0.smithi107.stdout:10931: delete oid 6 current snap is 0 2020-06-12T02:39:13.650 INFO:tasks.rados.rados.0.smithi107.stdout:10929: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:13.654 INFO:tasks.rados.rados.0.smithi107.stdout:10929: finishing write tid 3 to smithi10729407-10 2020-06-12T02:39:13.657 INFO:tasks.rados.rados.0.smithi107.stdout:10929: finishing write tid 4 to smithi10729407-10 2020-06-12T02:39:13.657 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 803 (ObjNum 10928 snap 0 seq_num 10928) dirty exists 2020-06-12T02:39:13.657 INFO:tasks.rados.rados.0.smithi107.stdout:10929: left oid 10 (ObjNum 10928 snap 0 seq_num 10928) 2020-06-12T02:39:13.658 INFO:tasks.rados.rados.0.smithi107.stdout:10930: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:13.660 INFO:tasks.rados.rados.0.smithi107.stdout:10929: done (2 left) 2020-06-12T02:39:13.660 INFO:tasks.rados.rados.0.smithi107.stdout:10931: done (1 left) 2020-06-12T02:39:13.661 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:39:13.661 INFO:tasks.rados.rados.0.smithi107.stdout:10932: seq_num 10931 ranges {204800=679936} 2020-06-12T02:39:13.665 INFO:tasks.rados.rados.0.smithi107.stdout:10932: writing smithi10729407-20 from 204800 to 884736 tid 1 2020-06-12T02:39:13.666 INFO:tasks.rados.rados.0.smithi107.stdout:10933: delete oid 4 current snap is 0 2020-06-12T02:39:13.666 INFO:tasks.rados.rados.0.smithi107.stdout:10930: finishing write tid 3 to smithi10729407-1 2020-06-12T02:39:13.666 INFO:tasks.rados.rados.0.smithi107.stdout:10930: finishing write tid 4 to smithi10729407-1 2020-06-12T02:39:13.666 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1963 (ObjNum 10929 snap 0 seq_num 10929) dirty exists 2020-06-12T02:39:13.667 INFO:tasks.rados.rados.0.smithi107.stdout:10930: left oid 1 (ObjNum 10929 snap 0 seq_num 10929) 2020-06-12T02:39:13.668 INFO:tasks.rados.rados.0.smithi107.stdout:10930: done (2 left) 2020-06-12T02:39:13.668 INFO:tasks.rados.rados.0.smithi107.stdout:10933: done (1 left) 2020-06-12T02:39:13.669 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:39:13.669 INFO:tasks.rados.rados.0.smithi107.stdout:10934: seq_num 10933 ranges {548864=712704} 2020-06-12T02:39:13.693 INFO:tasks.rados.rados.0.smithi107.stdout:10934: writing smithi10729407-14 from 548864 to 1261568 tid 1 2020-06-12T02:39:13.693 INFO:tasks.rados.rados.0.smithi107.stdout:10935: delete oid 15 current snap is 0 2020-06-12T02:39:13.693 INFO:tasks.rados.rados.0.smithi107.stdout:10932: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:13.693 INFO:tasks.rados.rados.0.smithi107.stdout:10932: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:13.694 INFO:tasks.rados.rados.0.smithi107.stdout:10932: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:13.694 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 845 (ObjNum 10931 snap 0 seq_num 10931) dirty exists 2020-06-12T02:39:13.694 INFO:tasks.rados.rados.0.smithi107.stdout:10932: left oid 20 (ObjNum 10931 snap 0 seq_num 10931) 2020-06-12T02:39:13.694 INFO:tasks.rados.rados.0.smithi107.stdout:10932: done (2 left) 2020-06-12T02:39:13.694 INFO:tasks.rados.rados.0.smithi107.stdout:10935: done (1 left) 2020-06-12T02:39:13.695 INFO:tasks.rados.rados.0.smithi107.stdout:10936: delete oid 13 current snap is 0 2020-06-12T02:39:13.695 INFO:tasks.rados.rados.0.smithi107.stdout:10936: done (1 left) 2020-06-12T02:39:13.695 INFO:tasks.rados.rados.0.smithi107.stdout:10937: delete oid 2 current snap is 0 2020-06-12T02:39:13.695 INFO:tasks.rados.rados.0.smithi107.stdout:10937: done (1 left) 2020-06-12T02:39:13.695 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:39:13.696 INFO:tasks.rados.rados.0.smithi107.stdout:10938: seq_num 10937 ranges {458752=778240} 2020-06-12T02:39:13.696 INFO:tasks.rados.rados.0.smithi107.stdout:10938: writing smithi10729407-7 from 458752 to 1236992 tid 1 2020-06-12T02:39:13.696 INFO:tasks.rados.rados.0.smithi107.stdout:10939: delete oid 4 current snap is 0 2020-06-12T02:39:13.696 INFO:tasks.rados.rados.0.smithi107.stdout:10934: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:13.696 INFO:tasks.rados.rados.0.smithi107.stdout:10934: finishing write tid 3 to smithi10729407-14 2020-06-12T02:39:13.696 INFO:tasks.rados.rados.0.smithi107.stdout:10934: finishing write tid 4 to smithi10729407-14 2020-06-12T02:39:13.697 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1728 (ObjNum 10933 snap 0 seq_num 10933) dirty exists 2020-06-12T02:39:13.697 INFO:tasks.rados.rados.0.smithi107.stdout:10934: left oid 14 (ObjNum 10933 snap 0 seq_num 10933) 2020-06-12T02:39:13.697 INFO:tasks.rados.rados.0.smithi107.stdout:10934: done (2 left) 2020-06-12T02:39:13.698 INFO:tasks.rados.rados.0.smithi107.stdout:10939: done (1 left) 2020-06-12T02:39:13.698 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:39:13.698 INFO:tasks.rados.rados.0.smithi107.stdout:10940: seq_num 10939 ranges {0=409600,409600=401408} 2020-06-12T02:39:13.698 INFO:tasks.rados.rados.0.smithi107.stdout:10940: writing smithi10729407-15 from 0 to 409600 tid 1 2020-06-12T02:39:13.700 INFO:tasks.rados.rados.0.smithi107.stdout:10940: writing smithi10729407-15 from 409600 to 811008 tid 2 2020-06-12T02:39:13.700 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:39:13.700 INFO:tasks.rados.rados.0.smithi107.stdout:10941: seq_num 10940 ranges {0=425984,425984=401408} 2020-06-12T02:39:13.702 INFO:tasks.rados.rados.0.smithi107.stdout:10941: writing smithi10729407-13 from 0 to 425984 tid 1 2020-06-12T02:39:13.705 INFO:tasks.rados.rados.0.smithi107.stdout:10941: writing smithi10729407-13 from 425984 to 827392 tid 2 2020-06-12T02:39:13.705 INFO:tasks.rados.rados.0.smithi107.stdout:10942: delete oid 11 current snap is 0 2020-06-12T02:39:13.706 INFO:tasks.rados.rados.0.smithi107.stdout:10938: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:13.709 INFO:tasks.ceph.osd.7.smithi188.stderr:2020-06-12T02:39:13.706+0000 7f2353737700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 29041) UID: 0 2020-06-12T02:39:13.710 INFO:tasks.rados.rados.0.smithi107.stdout:10940: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:13.710 INFO:tasks.rados.rados.0.smithi107.stdout:10940: finishing write tid 2 to smithi10729407-15 2020-06-12T02:39:13.710 INFO:tasks.rados.rados.0.smithi107.stdout:10940: finishing write tid 4 to smithi10729407-15 2020-06-12T02:39:13.710 INFO:tasks.rados.rados.0.smithi107.stdout:10938: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:13.710 INFO:tasks.rados.rados.0.smithi107.stdout:10938: finishing write tid 4 to smithi10729407-7 2020-06-12T02:39:13.711 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1789 (ObjNum 10937 snap 0 seq_num 10937) dirty exists 2020-06-12T02:39:13.711 INFO:tasks.rados.rados.0.smithi107.stdout:10938: left oid 7 (ObjNum 10937 snap 0 seq_num 10937) 2020-06-12T02:39:13.711 INFO:tasks.rados.rados.0.smithi107.stdout:10938: done (3 left) 2020-06-12T02:39:13.711 INFO:tasks.rados.rados.0.smithi107.stdout:10942: done (2 left) 2020-06-12T02:39:13.711 INFO:tasks.rados.rados.0.smithi107.stdout:10943: delete oid 3 current snap is 0 2020-06-12T02:39:13.712 INFO:tasks.rados.rados.0.smithi107.stdout:10940: finishing write tid 5 to smithi10729407-15 2020-06-12T02:39:13.712 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 2027 (ObjNum 10939 snap 0 seq_num 10939) dirty exists 2020-06-12T02:39:13.712 INFO:tasks.rados.rados.0.smithi107.stdout:10940: left oid 15 (ObjNum 10939 snap 0 seq_num 10939) 2020-06-12T02:39:13.713 INFO:tasks.rados.rados.0.smithi107.stdout:10941: finishing write tid 1 to smithi10729407-13 2020-06-12T02:39:13.713 INFO:tasks.rados.rados.0.smithi107.stdout:10941: finishing write tid 2 to smithi10729407-13 2020-06-12T02:39:13.714 INFO:tasks.rados.rados.0.smithi107.stdout:10941: finishing write tid 4 to smithi10729407-13 2020-06-12T02:39:13.714 INFO:tasks.rados.rados.0.smithi107.stdout:10941: finishing write tid 5 to smithi10729407-13 2020-06-12T02:39:13.714 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1775 (ObjNum 10940 snap 0 seq_num 10940) dirty exists 2020-06-12T02:39:13.714 INFO:tasks.rados.rados.0.smithi107.stdout:10941: left oid 13 (ObjNum 10940 snap 0 seq_num 10940) 2020-06-12T02:39:13.715 INFO:tasks.rados.rados.0.smithi107.stdout:10940: done (2 left) 2020-06-12T02:39:13.715 INFO:tasks.rados.rados.0.smithi107.stdout:10941: done (1 left) 2020-06-12T02:39:13.715 INFO:tasks.rados.rados.0.smithi107.stdout:10943: done (0 left) 2020-06-12T02:39:13.715 INFO:tasks.rados.rados.0.smithi107.stdout:10944: delete oid 3 current snap is 0 2020-06-12T02:39:13.723 INFO:tasks.rados.rados.0.smithi107.stdout:10944: done (0 left) 2020-06-12T02:39:13.724 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:39:13.724 INFO:tasks.rados.rados.0.smithi107.stdout:10945: seq_num 10944 ranges {319488=499712,819200=401408} 2020-06-12T02:39:13.728 INFO:tasks.rados.rados.0.smithi107.stdout:10945: writing smithi10729407-19 from 319488 to 819200 tid 1 2020-06-12T02:39:13.730 INFO:tasks.rados.rados.0.smithi107.stdout:10945: writing smithi10729407-19 from 819200 to 1220608 tid 2 2020-06-12T02:39:13.730 INFO:tasks.rados.rados.0.smithi107.stdout:10946: delete oid 9 current snap is 0 2020-06-12T02:39:13.732 INFO:tasks.rados.rados.0.smithi107.stdout:10946: done (1 left) 2020-06-12T02:39:13.733 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:39:13.733 INFO:tasks.rados.rados.0.smithi107.stdout:10947: seq_num 10946 ranges {0=532480,532480=401408} 2020-06-12T02:39:13.736 INFO:tasks.rados.rados.0.smithi107.stdout:10947: writing smithi10729407-4 from 0 to 532480 tid 1 2020-06-12T02:39:13.739 INFO:tasks.rados.rados.0.smithi107.stdout:10947: writing smithi10729407-4 from 532480 to 933888 tid 2 2020-06-12T02:39:13.739 INFO:tasks.rados.rados.0.smithi107.stdout:10948: delete oid 6 current snap is 0 2020-06-12T02:39:13.739 INFO:tasks.rados.rados.0.smithi107.stdout:10945: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:13.739 INFO:tasks.rados.rados.0.smithi107.stdout:10945: finishing write tid 2 to smithi10729407-19 2020-06-12T02:39:13.740 INFO:tasks.rados.rados.0.smithi107.stdout:10945: finishing write tid 4 to smithi10729407-19 2020-06-12T02:39:13.740 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:13.741 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:13.741 INFO:tasks.rados.rados.0.smithi107.stdout:10945: finishing write tid 5 to smithi10729407-19 2020-06-12T02:39:13.741 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 2023 (ObjNum 10944 snap 0 seq_num 10944) dirty exists 2020-06-12T02:39:13.741 INFO:tasks.rados.rados.0.smithi107.stdout:10945: left oid 19 (ObjNum 10944 snap 0 seq_num 10944) 2020-06-12T02:39:13.742 INFO:tasks.rados.rados.0.smithi107.stdout:10947: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:13.743 INFO:tasks.rados.rados.0.smithi107.stdout:10945: done (2 left) 2020-06-12T02:39:13.743 INFO:tasks.rados.rados.0.smithi107.stdout:10948: done (1 left) 2020-06-12T02:39:13.743 INFO:tasks.rados.rados.0.smithi107.stdout:10949: delete oid 10 current snap is 0 2020-06-12T02:39:13.745 INFO:tasks.rados.rados.0.smithi107.stdout:10949: done (1 left) 2020-06-12T02:39:13.745 INFO:tasks.rados.rados.0.smithi107.stdout:10950: delete oid 5 current snap is 0 2020-06-12T02:39:13.749 INFO:tasks.rados.rados.0.smithi107.stdout:10947: finishing write tid 2 to smithi10729407-4 2020-06-12T02:39:13.749 INFO:tasks.rados.rados.0.smithi107.stdout:10950: done (1 left) 2020-06-12T02:39:13.750 INFO:tasks.rados.rados.0.smithi107.stdout:10951: delete oid 3 current snap is 0 2020-06-12T02:39:13.751 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-06-12T02:39:13.751 INFO:tasks.rados.rados.0.smithi107.stdout:10947: finishing write tid 4 to smithi10729407-4 2020-06-12T02:39:13.753 INFO:tasks.rados.rados.0.smithi107.stdout:10951: done (1 left) 2020-06-12T02:39:13.754 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:39:13.754 INFO:tasks.rados.rados.0.smithi107.stdout:10952: seq_num 10951 ranges {1220608=614400,1835008=401408} 2020-06-12T02:39:13.764 INFO:tasks.rados.rados.0.smithi107.stdout:10952: writing smithi10729407-19 from 1220608 to 1835008 tid 1 2020-06-12T02:39:13.766 INFO:tasks.rados.rados.0.smithi107.stdout:10952: writing smithi10729407-19 from 1835008 to 2236416 tid 2 2020-06-12T02:39:13.766 INFO:tasks.rados.rados.0.smithi107.stdout:10953: delete oid 5 current snap is 0 2020-06-12T02:39:13.767 INFO:tasks.rados.rados.0.smithi107.stdout:10947: finishing write tid 5 to smithi10729407-4 2020-06-12T02:39:13.767 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 2047 (ObjNum 10946 snap 0 seq_num 10946) dirty exists 2020-06-12T02:39:13.767 INFO:tasks.rados.rados.0.smithi107.stdout:10947: left oid 4 (ObjNum 10946 snap 0 seq_num 10946) 2020-06-12T02:39:13.769 INFO:tasks.rados.rados.0.smithi107.stdout:10947: done (2 left) 2020-06-12T02:39:13.770 INFO:tasks.rados.rados.0.smithi107.stdout:10953: done (1 left) 2020-06-12T02:39:13.770 INFO:tasks.rados.rados.0.smithi107.stdout:10954: delete oid 10 current snap is 0 2020-06-12T02:39:13.770 INFO:tasks.rados.rados.0.smithi107.stdout:10952: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:13.771 INFO:tasks.rados.rados.0.smithi107.stdout:10954: done (1 left) 2020-06-12T02:39:13.771 INFO:tasks.rados.rados.0.smithi107.stdout:10955: delete oid 10 current snap is 0 2020-06-12T02:39:13.773 INFO:tasks.rados.rados.0.smithi107.stdout:10955: done (1 left) 2020-06-12T02:39:13.774 INFO:tasks.rados.rados.0.smithi107.stdout:10956: delete oid 9 current snap is 0 2020-06-12T02:39:13.776 INFO:tasks.rados.rados.0.smithi107.stdout:10956: done (1 left) 2020-06-12T02:39:13.776 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:39:13.781 INFO:tasks.rados.rados.0.smithi107.stdout:10957: seq_num 10956 ranges {647168=696320,1343488=401408} 2020-06-12T02:39:13.783 INFO:tasks.rados.rados.0.smithi107.stdout:10957: writing smithi10729407-1 from 647168 to 1343488 tid 1 2020-06-12T02:39:13.786 INFO:tasks.rados.rados.0.smithi107.stdout:10957: writing smithi10729407-1 from 1343488 to 1744896 tid 2 2020-06-12T02:39:13.786 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:39:13.786 INFO:tasks.rados.rados.0.smithi107.stdout:10958: seq_num 10957 ranges {0=712704,712704=401408} 2020-06-12T02:39:13.790 INFO:tasks.rados.rados.0.smithi107.stdout:10958: writing smithi10729407-17 from 0 to 712704 tid 1 2020-06-12T02:39:13.793 INFO:tasks.rados.rados.0.smithi107.stdout:10958: writing smithi10729407-17 from 712704 to 1114112 tid 2 2020-06-12T02:39:13.793 INFO:tasks.rados.rados.0.smithi107.stdout:10959: delete oid 5 current snap is 0 2020-06-12T02:39:13.794 INFO:tasks.rados.rados.0.smithi107.stdout:10952: finishing write tid 2 to smithi10729407-19 2020-06-12T02:39:13.794 INFO:tasks.rados.rados.0.smithi107.stdout:10952: finishing write tid 4 to smithi10729407-19 2020-06-12T02:39:13.794 INFO:tasks.rados.rados.0.smithi107.stdout:10952: finishing write tid 5 to smithi10729407-19 2020-06-12T02:39:13.794 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 2026 (ObjNum 10951 snap 0 seq_num 10951) dirty exists 2020-06-12T02:39:13.794 INFO:tasks.rados.rados.0.smithi107.stdout:10952: left oid 19 (ObjNum 10951 snap 0 seq_num 10951) 2020-06-12T02:39:13.795 INFO:tasks.rados.rados.0.smithi107.stdout:10957: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:13.795 INFO:tasks.rados.rados.0.smithi107.stdout:10957: finishing write tid 2 to smithi10729407-1 2020-06-12T02:39:13.795 INFO:tasks.rados.rados.0.smithi107.stdout:10957: finishing write tid 4 to smithi10729407-1 2020-06-12T02:39:13.796 INFO:tasks.rados.rados.0.smithi107.stdout:10952: done (3 left) 2020-06-12T02:39:13.797 INFO:tasks.rados.rados.0.smithi107.stdout:10959: done (2 left) 2020-06-12T02:39:13.797 INFO:tasks.rados.rados.0.smithi107.stdout:10960: delete oid 3 current snap is 0 2020-06-12T02:39:13.798 INFO:tasks.rados.rados.0.smithi107.stdout:10957: finishing write tid 5 to smithi10729407-1 2020-06-12T02:39:13.798 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1967 (ObjNum 10956 snap 0 seq_num 10956) dirty exists 2020-06-12T02:39:13.798 INFO:tasks.rados.rados.0.smithi107.stdout:10957: left oid 1 (ObjNum 10956 snap 0 seq_num 10956) 2020-06-12T02:39:13.801 INFO:tasks.rados.rados.0.smithi107.stdout:10958: finishing write tid 1 to smithi10729407-17 2020-06-12T02:39:13.801 INFO:tasks.rados.rados.0.smithi107.stdout:10957: done (2 left) 2020-06-12T02:39:13.801 INFO:tasks.rados.rados.0.smithi107.stdout:10960: done (1 left) 2020-06-12T02:39:13.802 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:39:13.802 INFO:tasks.rados.rados.0.smithi107.stdout:10961: seq_num 10960 ranges {0=761856,761856=401408} 2020-06-12T02:39:13.805 INFO:tasks.rados.rados.0.smithi107.stdout:10961: writing smithi10729407-11 from 0 to 761856 tid 1 2020-06-12T02:39:13.809 INFO:tasks.rados.rados.0.smithi107.stdout:10961: writing smithi10729407-11 from 761856 to 1163264 tid 2 2020-06-12T02:39:13.810 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:39:13.810 INFO:tasks.rados.rados.0.smithi107.stdout:10962: seq_num 10961 ranges {262144=778240,1040384=401408} 2020-06-12T02:39:13.810 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:39:13.806+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:39:13.814 INFO:tasks.rados.rados.0.smithi107.stdout:10962: writing smithi10729407-16 from 262144 to 1040384 tid 1 2020-06-12T02:39:13.817 INFO:tasks.rados.rados.0.smithi107.stdout:10962: writing smithi10729407-16 from 1040384 to 1441792 tid 2 2020-06-12T02:39:13.817 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:39:13.818 INFO:tasks.rados.rados.0.smithi107.stdout:10963: seq_num 10962 ranges {0=802816,802816=401408} 2020-06-12T02:39:13.822 INFO:tasks.rados.rados.0.smithi107.stdout:10963: writing smithi10729407-5 from 0 to 802816 tid 1 2020-06-12T02:39:13.834 INFO:tasks.rados.rados.0.smithi107.stdout:10963: writing smithi10729407-5 from 802816 to 1204224 tid 2 2020-06-12T02:39:13.834 INFO:tasks.rados.rados.0.smithi107.stdout:10964: delete oid 7 current snap is 0 2020-06-12T02:39:13.834 INFO:tasks.rados.rados.0.smithi107.stdout:10958: finishing write tid 2 to smithi10729407-17 2020-06-12T02:39:13.835 INFO:tasks.rados.rados.0.smithi107.stdout:10958: finishing write tid 4 to smithi10729407-17 2020-06-12T02:39:13.835 INFO:tasks.rados.rados.0.smithi107.stdout:10958: finishing write tid 5 to smithi10729407-17 2020-06-12T02:39:13.835 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 2050 (ObjNum 10957 snap 0 seq_num 10957) dirty exists 2020-06-12T02:39:13.835 INFO:tasks.rados.rados.0.smithi107.stdout:10958: left oid 17 (ObjNum 10957 snap 0 seq_num 10957) 2020-06-12T02:39:13.835 INFO:tasks.rados.rados.0.smithi107.stdout:10961: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:13.836 INFO:tasks.rados.rados.0.smithi107.stdout:10961: finishing write tid 2 to smithi10729407-11 2020-06-12T02:39:13.836 INFO:tasks.rados.rados.0.smithi107.stdout:10961: finishing write tid 4 to smithi10729407-11 2020-06-12T02:39:13.836 INFO:tasks.rados.rados.0.smithi107.stdout:10961: finishing write tid 5 to smithi10729407-11 2020-06-12T02:39:13.836 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1755 (ObjNum 10960 snap 0 seq_num 10960) dirty exists 2020-06-12T02:39:13.836 INFO:tasks.rados.rados.0.smithi107.stdout:10961: left oid 11 (ObjNum 10960 snap 0 seq_num 10960) 2020-06-12T02:39:13.837 INFO:tasks.rados.rados.0.smithi107.stdout:10962: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:13.837 INFO:tasks.rados.rados.0.smithi107.stdout:10962: finishing write tid 2 to smithi10729407-16 2020-06-12T02:39:13.837 INFO:tasks.rados.rados.0.smithi107.stdout:10962: finishing write tid 4 to smithi10729407-16 2020-06-12T02:39:13.837 INFO:tasks.rados.rados.0.smithi107.stdout:10962: finishing write tid 5 to smithi10729407-16 2020-06-12T02:39:13.838 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1943 (ObjNum 10961 snap 0 seq_num 10961) dirty exists 2020-06-12T02:39:13.838 INFO:tasks.rados.rados.0.smithi107.stdout:10962: left oid 16 (ObjNum 10961 snap 0 seq_num 10961) 2020-06-12T02:39:13.838 INFO:tasks.rados.rados.0.smithi107.stdout:10958: done (4 left) 2020-06-12T02:39:13.838 INFO:tasks.rados.rados.0.smithi107.stdout:10961: done (3 left) 2020-06-12T02:39:13.838 INFO:tasks.rados.rados.0.smithi107.stdout:10962: done (2 left) 2020-06-12T02:39:13.839 INFO:tasks.rados.rados.0.smithi107.stdout:10964: done (1 left) 2020-06-12T02:39:13.839 INFO:tasks.rados.rados.0.smithi107.stdout:10965: delete oid 17 current snap is 0 2020-06-12T02:39:13.839 INFO:tasks.rados.rados.0.smithi107.stdout:10963: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:13.840 INFO:tasks.rados.rados.0.smithi107.stdout:10965: done (1 left) 2020-06-12T02:39:13.840 INFO:tasks.rados.rados.0.smithi107.stdout:10966: delete oid 18 current snap is 0 2020-06-12T02:39:13.841 INFO:tasks.rados.rados.0.smithi107.stdout:10963: finishing write tid 2 to smithi10729407-5 2020-06-12T02:39:13.841 INFO:tasks.rados.rados.0.smithi107.stdout:10963: finishing write tid 4 to smithi10729407-5 2020-06-12T02:39:13.841 INFO:tasks.rados.rados.0.smithi107.stdout:10966: done (1 left) 2020-06-12T02:39:13.841 INFO:tasks.rados.rados.0.smithi107.stdout:10967: delete oid 12 current snap is 0 2020-06-12T02:39:13.842 INFO:tasks.rados.rados.0.smithi107.stdout:10963: finishing write tid 5 to smithi10729407-5 2020-06-12T02:39:13.842 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 1058 (ObjNum 10962 snap 0 seq_num 10962) dirty exists 2020-06-12T02:39:13.842 INFO:tasks.rados.rados.0.smithi107.stdout:10963: left oid 5 (ObjNum 10962 snap 0 seq_num 10962) 2020-06-12T02:39:13.844 INFO:tasks.rados.rados.0.smithi107.stdout:10963: done (1 left) 2020-06-12T02:39:13.844 INFO:tasks.rados.rados.0.smithi107.stdout:10967: done (0 left) 2020-06-12T02:39:13.844 INFO:tasks.rados.rados.0.smithi107.stdout:10968: delete oid 20 current snap is 0 2020-06-12T02:39:13.845 INFO:tasks.rados.rados.0.smithi107.stdout:10968: done (0 left) 2020-06-12T02:39:13.845 INFO:tasks.rados.rados.0.smithi107.stdout:10969: delete oid 14 current snap is 0 2020-06-12T02:39:13.846 INFO:tasks.rados.rados.0.smithi107.stdout:10969: done (0 left) 2020-06-12T02:39:13.846 INFO:tasks.rados.rados.0.smithi107.stdout:10970: delete oid 13 current snap is 0 2020-06-12T02:39:13.848 INFO:tasks.rados.rados.0.smithi107.stdout:10970: done (0 left) 2020-06-12T02:39:13.848 INFO:tasks.rados.rados.0.smithi107.stdout:10971: delete oid 13 current snap is 0 2020-06-12T02:39:13.851 INFO:tasks.rados.rados.0.smithi107.stdout:10971: done (0 left) 2020-06-12T02:39:13.852 INFO:tasks.rados.rados.0.smithi107.stdout:10972: delete oid 10 current snap is 0 2020-06-12T02:39:13.853 INFO:tasks.rados.rados.0.smithi107.stdout:10972: done (0 left) 2020-06-12T02:39:13.854 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:39:13.854 INFO:tasks.rados.rados.0.smithi107.stdout:10973: seq_num 10972 ranges {1204224=565248,1769472=581632,2351104=221184} 2020-06-12T02:39:13.863 INFO:tasks.rados.rados.0.smithi107.stdout:10973: writing smithi10729407-5 from 1204224 to 1769472 tid 1 2020-06-12T02:39:13.867 INFO:tasks.rados.rados.0.smithi107.stdout:10973: writing smithi10729407-5 from 1769472 to 2351104 tid 2 2020-06-12T02:39:13.868 INFO:tasks.rados.rados.0.smithi107.stdout:10973: writing smithi10729407-5 from 2351104 to 2572288 tid 3 2020-06-12T02:39:13.868 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:39:13.868 INFO:tasks.rados.rados.0.smithi107.stdout:10974: seq_num 10973 ranges {0=581632,581632=466944,1048576=335872} 2020-06-12T02:39:13.872 INFO:tasks.rados.rados.0.smithi107.stdout:10974: writing smithi10729407-2 from 0 to 581632 tid 1 2020-06-12T02:39:13.876 INFO:tasks.rados.rados.0.smithi107.stdout:10974: writing smithi10729407-2 from 581632 to 1048576 tid 2 2020-06-12T02:39:13.877 INFO:tasks.rados.rados.0.smithi107.stdout:10974: writing smithi10729407-2 from 1048576 to 1384448 tid 3 2020-06-12T02:39:13.877 INFO:tasks.rados.rados.0.smithi107.stdout:10975: delete oid 11 current snap is 0 2020-06-12T02:39:13.878 INFO:tasks.rados.rados.0.smithi107.stdout:10973: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:13.878 INFO:tasks.rados.rados.0.smithi107.stdout:10973: finishing write tid 2 to smithi10729407-5 2020-06-12T02:39:13.878 INFO:tasks.rados.rados.0.smithi107.stdout:10973: finishing write tid 3 to smithi10729407-5 2020-06-12T02:39:13.878 INFO:tasks.rados.rados.0.smithi107.stdout:10973: finishing write tid 5 to smithi10729407-5 2020-06-12T02:39:13.879 INFO:tasks.rados.rados.0.smithi107.stdout:10973: finishing write tid 6 to smithi10729407-5 2020-06-12T02:39:13.879 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 1062 (ObjNum 10972 snap 0 seq_num 10972) dirty exists 2020-06-12T02:39:13.879 INFO:tasks.rados.rados.0.smithi107.stdout:10973: left oid 5 (ObjNum 10972 snap 0 seq_num 10972) 2020-06-12T02:39:13.880 INFO:tasks.rados.rados.0.smithi107.stdout:10973: done (2 left) 2020-06-12T02:39:13.880 INFO:tasks.rados.rados.0.smithi107.stdout:10975: done (1 left) 2020-06-12T02:39:13.880 INFO:tasks.rados.rados.0.smithi107.stdout:10976: delete oid 10 current snap is 0 2020-06-12T02:39:13.882 INFO:tasks.rados.rados.0.smithi107.stdout:10974: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:13.883 INFO:tasks.rados.rados.0.smithi107.stdout:10976: done (1 left) 2020-06-12T02:39:13.883 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:39:13.883 INFO:tasks.rados.rados.0.smithi107.stdout:10977: seq_num 10976 ranges {933888=630784,1564672=524288,2088960=278528} 2020-06-12T02:39:13.891 INFO:tasks.rados.rados.0.smithi107.stdout:10977: writing smithi10729407-4 from 933888 to 1564672 tid 1 2020-06-12T02:39:13.895 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:13.895 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:13.895 INFO:tasks.rados.rados.0.smithi107.stdout:10977: writing smithi10729407-4 from 1564672 to 2088960 tid 2 2020-06-12T02:39:13.898 INFO:tasks.rados.rados.0.smithi107.stdout:10977: writing smithi10729407-4 from 2088960 to 2367488 tid 3 2020-06-12T02:39:13.898 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:39:13.899 INFO:tasks.rados.rados.0.smithi107.stdout:10978: seq_num 10977 ranges {0=647168,647168=802816} 2020-06-12T02:39:13.901 INFO:tasks.rados.rados.0.smithi107.stdout:10978: writing smithi10729407-14 from 0 to 647168 tid 1 2020-06-12T02:39:13.905 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-06-12T02:39:13.908 INFO:tasks.rados.rados.0.smithi107.stdout:10978: writing smithi10729407-14 from 647168 to 1449984 tid 2 2020-06-12T02:39:13.908 INFO:tasks.rados.rados.0.smithi107.stdout:10979: delete oid 15 current snap is 0 2020-06-12T02:39:13.908 INFO:tasks.rados.rados.0.smithi107.stdout:10974: finishing write tid 2 to smithi10729407-2 2020-06-12T02:39:13.908 INFO:tasks.rados.rados.0.smithi107.stdout:10974: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:13.908 INFO:tasks.rados.rados.0.smithi107.stdout:10974: finishing write tid 5 to smithi10729407-2 2020-06-12T02:39:13.909 INFO:tasks.rados.rados.0.smithi107.stdout:10974: finishing write tid 6 to smithi10729407-2 2020-06-12T02:39:13.909 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 1032 (ObjNum 10973 snap 0 seq_num 10973) dirty exists 2020-06-12T02:39:13.909 INFO:tasks.rados.rados.0.smithi107.stdout:10974: left oid 2 (ObjNum 10973 snap 0 seq_num 10973) 2020-06-12T02:39:13.909 INFO:tasks.rados.rados.0.smithi107.stdout:10977: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:13.909 INFO:tasks.rados.rados.0.smithi107.stdout:10977: finishing write tid 2 to smithi10729407-4 2020-06-12T02:39:13.910 INFO:tasks.rados.rados.0.smithi107.stdout:10977: finishing write tid 3 to smithi10729407-4 2020-06-12T02:39:13.911 INFO:tasks.rados.rados.0.smithi107.stdout:10977: finishing write tid 5 to smithi10729407-4 2020-06-12T02:39:13.911 INFO:tasks.rados.rados.0.smithi107.stdout:10974: done (3 left) 2020-06-12T02:39:13.911 INFO:tasks.rados.rados.0.smithi107.stdout:10979: done (2 left) 2020-06-12T02:39:13.911 INFO:tasks.rados.rados.0.smithi107.stdout:10980: delete oid 10 current snap is 0 2020-06-12T02:39:13.912 INFO:tasks.rados.rados.0.smithi107.stdout:10978: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:13.913 INFO:tasks.rados.rados.0.smithi107.stdout:10977: finishing write tid 6 to smithi10729407-4 2020-06-12T02:39:13.913 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 2055 (ObjNum 10976 snap 0 seq_num 10976) dirty exists 2020-06-12T02:39:13.913 INFO:tasks.rados.rados.0.smithi107.stdout:10977: left oid 4 (ObjNum 10976 snap 0 seq_num 10976) 2020-06-12T02:39:13.914 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:39:13.909+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:39:13.915 INFO:tasks.rados.rados.0.smithi107.stdout:10977: done (2 left) 2020-06-12T02:39:13.915 INFO:tasks.rados.rados.0.smithi107.stdout:10980: done (1 left) 2020-06-12T02:39:13.916 INFO:tasks.rados.rados.0.smithi107.stdout:10981: delete oid 4 current snap is 0 2020-06-12T02:39:13.918 INFO:tasks.rados.rados.0.smithi107.stdout:10981: done (1 left) 2020-06-12T02:39:13.920 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:39:13.920 INFO:tasks.rados.rados.0.smithi107.stdout:10982: seq_num 10981 ranges {0=720896,720896=794624,1515520=8192} 2020-06-12T02:39:13.923 INFO:tasks.rados.rados.0.smithi107.stdout:10982: writing smithi10729407-17 from 0 to 720896 tid 1 2020-06-12T02:39:13.928 INFO:tasks.rados.rados.0.smithi107.stdout:10982: writing smithi10729407-17 from 720896 to 1515520 tid 2 2020-06-12T02:39:13.929 INFO:tasks.rados.rados.0.smithi107.stdout:10982: writing smithi10729407-17 from 1515520 to 1523712 tid 3 2020-06-12T02:39:13.929 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:39:13.929 INFO:tasks.rados.rados.0.smithi107.stdout:10983: seq_num 10982 ranges {0=737280,737280=679936,1417216=122880} 2020-06-12T02:39:13.932 INFO:tasks.rados.rados.0.smithi107.stdout:10983: writing smithi10729407-6 from 0 to 737280 tid 1 2020-06-12T02:39:13.937 INFO:tasks.rados.rados.0.smithi107.stdout:10983: writing smithi10729407-6 from 737280 to 1417216 tid 2 2020-06-12T02:39:13.938 INFO:tasks.rados.rados.0.smithi107.stdout:10983: writing smithi10729407-6 from 1417216 to 1540096 tid 3 2020-06-12T02:39:13.938 INFO:tasks.rados.rados.0.smithi107.stdout:10984: delete oid 10 current snap is 0 2020-06-12T02:39:13.938 INFO:tasks.rados.rados.0.smithi107.stdout:10978: finishing write tid 2 to smithi10729407-14 2020-06-12T02:39:13.939 INFO:tasks.rados.rados.0.smithi107.stdout:10978: finishing write tid 4 to smithi10729407-14 2020-06-12T02:39:13.939 INFO:tasks.rados.rados.0.smithi107.stdout:10978: finishing write tid 5 to smithi10729407-14 2020-06-12T02:39:13.939 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1732 (ObjNum 10977 snap 0 seq_num 10977) dirty exists 2020-06-12T02:39:13.939 INFO:tasks.rados.rados.0.smithi107.stdout:10978: left oid 14 (ObjNum 10977 snap 0 seq_num 10977) 2020-06-12T02:39:13.939 INFO:tasks.rados.rados.0.smithi107.stdout:10982: finishing write tid 1 to smithi10729407-17 2020-06-12T02:39:13.940 INFO:tasks.rados.rados.0.smithi107.stdout:10978: done (3 left) 2020-06-12T02:39:13.941 INFO:tasks.rados.rados.0.smithi107.stdout:10984: done (2 left) 2020-06-12T02:39:13.941 INFO:tasks.rados.rados.0.smithi107.stdout:10985: delete oid 3 current snap is 0 2020-06-12T02:39:13.941 INFO:tasks.rados.rados.0.smithi107.stdout:10982: finishing write tid 2 to smithi10729407-17 2020-06-12T02:39:13.941 INFO:tasks.rados.rados.0.smithi107.stdout:10982: finishing write tid 3 to smithi10729407-17 2020-06-12T02:39:13.941 INFO:tasks.rados.rados.0.smithi107.stdout:10983: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:13.942 INFO:tasks.rados.rados.0.smithi107.stdout:10982: finishing write tid 5 to smithi10729407-17 2020-06-12T02:39:13.944 INFO:tasks.rados.rados.0.smithi107.stdout:10982: finishing write tid 6 to smithi10729407-17 2020-06-12T02:39:13.945 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 2060 (ObjNum 10981 snap 0 seq_num 10981) dirty exists 2020-06-12T02:39:13.945 INFO:tasks.rados.rados.0.smithi107.stdout:10982: left oid 17 (ObjNum 10981 snap 0 seq_num 10981) 2020-06-12T02:39:13.945 INFO:tasks.rados.rados.0.smithi107.stdout:10982: done (2 left) 2020-06-12T02:39:13.946 INFO:tasks.rados.rados.0.smithi107.stdout:10985: done (1 left) 2020-06-12T02:39:13.946 INFO:tasks.rados.rados.0.smithi107.stdout:10986: delete oid 2 current snap is 0 2020-06-12T02:39:13.950 INFO:tasks.rados.rados.0.smithi107.stdout:10983: finishing write tid 2 to smithi10729407-6 2020-06-12T02:39:13.951 INFO:tasks.rados.rados.0.smithi107.stdout:10986: done (1 left) 2020-06-12T02:39:13.952 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:39:13.952 INFO:tasks.rados.rados.0.smithi107.stdout:10987: seq_num 10986 ranges {2236416=802816,3039232=663552,3702784=139264} 2020-06-12T02:39:13.967 INFO:tasks.rados.rados.0.smithi107.stdout:10987: writing smithi10729407-19 from 2236416 to 3039232 tid 1 2020-06-12T02:39:13.975 INFO:tasks.rados.rados.0.smithi107.stdout:10987: writing smithi10729407-19 from 3039232 to 3702784 tid 2 2020-06-12T02:39:13.975 INFO:tasks.rados.rados.0.smithi107.stdout:10987: writing smithi10729407-19 from 3702784 to 3842048 tid 3 2020-06-12T02:39:13.975 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:39:13.976 INFO:tasks.rados.rados.0.smithi107.stdout:10983: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:13.976 INFO:tasks.rados.rados.0.smithi107.stdout:10988: seq_num 10987 ranges {0=417792,417792=548864,966656=655360} 2020-06-12T02:39:13.976 INFO:tasks.rados.rados.0.smithi107.stdout:10988: writing smithi10729407-4 from 0 to 417792 tid 1 2020-06-12T02:39:13.978 INFO:tasks.rados.rados.0.smithi107.stdout:10988: writing smithi10729407-4 from 417792 to 966656 tid 2 2020-06-12T02:39:13.982 INFO:tasks.rados.rados.0.smithi107.stdout:10988: writing smithi10729407-4 from 966656 to 1622016 tid 3 2020-06-12T02:39:13.982 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:39:13.982 INFO:tasks.rados.rados.0.smithi107.stdout:10989: seq_num 10988 ranges {0=434176,434176=434176,868352=516096,1384448=253952} 2020-06-12T02:39:13.987 INFO:tasks.rados.rados.0.smithi107.stdout:10989: writing smithi10729407-15 from 0 to 434176 tid 1 2020-06-12T02:39:13.988 INFO:tasks.rados.rados.0.smithi107.stdout:10989: writing smithi10729407-15 from 434176 to 868352 tid 2 2020-06-12T02:39:13.991 INFO:tasks.rados.rados.0.smithi107.stdout:10989: writing smithi10729407-15 from 868352 to 1384448 tid 3 2020-06-12T02:39:13.993 INFO:tasks.rados.rados.0.smithi107.stdout:10989: writing smithi10729407-15 from 1384448 to 1638400 tid 4 2020-06-12T02:39:13.993 INFO:tasks.rados.rados.0.smithi107.stdout:10990: delete oid 3 current snap is 0 2020-06-12T02:39:13.993 INFO:tasks.rados.rados.0.smithi107.stdout:10983: finishing write tid 5 to smithi10729407-6 2020-06-12T02:39:13.993 INFO:tasks.rados.rados.0.smithi107.stdout:10983: finishing write tid 6 to smithi10729407-6 2020-06-12T02:39:13.994 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1971 (ObjNum 10982 snap 0 seq_num 10982) dirty exists 2020-06-12T02:39:13.994 INFO:tasks.rados.rados.0.smithi107.stdout:10983: left oid 6 (ObjNum 10982 snap 0 seq_num 10982) 2020-06-12T02:39:13.994 INFO:tasks.rados.rados.0.smithi107.stdout:10987: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:13.994 INFO:tasks.rados.rados.0.smithi107.stdout:10988: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:13.994 INFO:tasks.rados.rados.0.smithi107.stdout:10987: finishing write tid 2 to smithi10729407-19 2020-06-12T02:39:13.995 INFO:tasks.rados.rados.0.smithi107.stdout:10987: finishing write tid 3 to smithi10729407-19 2020-06-12T02:39:13.995 INFO:tasks.rados.rados.0.smithi107.stdout:10987: finishing write tid 5 to smithi10729407-19 2020-06-12T02:39:13.995 INFO:tasks.rados.rados.0.smithi107.stdout:10987: finishing write tid 6 to smithi10729407-19 2020-06-12T02:39:13.995 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 2031 (ObjNum 10986 snap 0 seq_num 10986) dirty exists 2020-06-12T02:39:13.995 INFO:tasks.rados.rados.0.smithi107.stdout:10987: left oid 19 (ObjNum 10986 snap 0 seq_num 10986) 2020-06-12T02:39:13.996 INFO:tasks.rados.rados.0.smithi107.stdout:10989: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:13.997 INFO:tasks.rados.rados.0.smithi107.stdout:10988: finishing write tid 2 to smithi10729407-4 2020-06-12T02:39:13.997 INFO:tasks.rados.rados.0.smithi107.stdout:10988: finishing write tid 3 to smithi10729407-4 2020-06-12T02:39:13.998 INFO:tasks.rados.rados.0.smithi107.stdout:10983: done (4 left) 2020-06-12T02:39:13.998 INFO:tasks.rados.rados.0.smithi107.stdout:10987: done (3 left) 2020-06-12T02:39:13.998 INFO:tasks.rados.rados.0.smithi107.stdout:10990: done (2 left) 2020-06-12T02:39:13.998 INFO:tasks.rados.rados.0.smithi107.stdout:10991: delete oid 3 current snap is 0 2020-06-12T02:39:13.998 INFO:tasks.rados.rados.0.smithi107.stdout:10988: finishing write tid 5 to smithi10729407-4 2020-06-12T02:39:13.999 INFO:tasks.rados.rados.0.smithi107.stdout:10988: finishing write tid 6 to smithi10729407-4 2020-06-12T02:39:13.999 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 2064 (ObjNum 10987 snap 0 seq_num 10987) dirty exists 2020-06-12T02:39:14.000 INFO:tasks.rados.rados.0.smithi107.stdout:10988: left oid 4 (ObjNum 10987 snap 0 seq_num 10987) 2020-06-12T02:39:14.000 INFO:tasks.rados.rados.0.smithi107.stdout:10988: done (2 left) 2020-06-12T02:39:14.000 INFO:tasks.rados.rados.0.smithi107.stdout:10991: done (1 left) 2020-06-12T02:39:14.000 INFO:tasks.rados.rados.0.smithi107.stdout:10992: delete oid 1 current snap is 0 2020-06-12T02:39:14.001 INFO:tasks.rados.rados.0.smithi107.stdout:10989: finishing write tid 2 to smithi10729407-15 2020-06-12T02:39:14.002 INFO:tasks.rados.rados.0.smithi107.stdout:10992: done (1 left) 2020-06-12T02:39:14.002 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:39:14.002 INFO:tasks.rados.rados.0.smithi107.stdout:10993: seq_num 10992 ranges {1449984=499712,1949696=770048,2719744=434176} 2020-06-12T02:39:14.012 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:39:14.010+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:39:14.013 INFO:tasks.rados.rados.0.smithi107.stdout:10993: writing smithi10729407-14 from 1449984 to 1949696 tid 1 2020-06-12T02:39:14.017 INFO:tasks.rados.rados.0.smithi107.stdout:10993: writing smithi10729407-14 from 1949696 to 2719744 tid 2 2020-06-12T02:39:14.020 INFO:tasks.rados.rados.0.smithi107.stdout:10993: writing smithi10729407-14 from 2719744 to 3153920 tid 3 2020-06-12T02:39:14.020 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:39:14.021 INFO:tasks.rados.rados.0.smithi107.stdout:10994: seq_num 10993 ranges {0=516096,516096=655360,1171456=548864} 2020-06-12T02:39:14.023 INFO:tasks.rados.rados.0.smithi107.stdout:10994: writing smithi10729407-1 from 0 to 516096 tid 1 2020-06-12T02:39:14.026 INFO:tasks.rados.rados.0.smithi107.stdout:10994: writing smithi10729407-1 from 516096 to 1171456 tid 2 2020-06-12T02:39:14.030 INFO:tasks.rados.rados.0.smithi107.stdout:10994: writing smithi10729407-1 from 1171456 to 1720320 tid 3 2020-06-12T02:39:14.030 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:39:14.030 INFO:tasks.rados.rados.0.smithi107.stdout:10995: seq_num 10994 ranges {0=532480,532480=589824,1122304=507904,1630208=106496} 2020-06-12T02:39:14.033 INFO:tasks.rados.rados.0.smithi107.stdout:10995: writing smithi10729407-10 from 0 to 532480 tid 1 2020-06-12T02:39:14.037 INFO:tasks.rados.rados.0.smithi107.stdout:10995: writing smithi10729407-10 from 532480 to 1122304 tid 2 2020-06-12T02:39:14.039 INFO:tasks.rados.rados.0.smithi107.stdout:10995: writing smithi10729407-10 from 1122304 to 1630208 tid 3 2020-06-12T02:39:14.040 INFO:tasks.rados.rados.0.smithi107.stdout:10995: writing smithi10729407-10 from 1630208 to 1736704 tid 4 2020-06-12T02:39:14.040 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:39:14.041 INFO:tasks.rados.rados.0.smithi107.stdout:10996: seq_num 10995 ranges {1441792=548864,1990656=475136,2465792=671744,3137536=57344} 2020-06-12T02:39:14.051 INFO:tasks.rados.rados.0.smithi107.stdout:10996: writing smithi10729407-16 from 1441792 to 1990656 tid 1 2020-06-12T02:39:14.061 INFO:tasks.rados.rados.0.smithi107.stdout:10996: writing smithi10729407-16 from 1990656 to 2465792 tid 2 2020-06-12T02:39:14.062 INFO:tasks.rados.rados.0.smithi107.stdout:10996: writing smithi10729407-16 from 2465792 to 3137536 tid 3 2020-06-12T02:39:14.062 INFO:tasks.rados.rados.0.smithi107.stdout:10996: writing smithi10729407-16 from 3137536 to 3194880 tid 4 2020-06-12T02:39:14.062 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:39:14.062 INFO:tasks.rados.rados.0.smithi107.stdout:10997: seq_num 10996 ranges {0=565248,565248=753664,1318912=450560} 2020-06-12T02:39:14.063 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:14.063 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:14.063 INFO:tasks.rados.rados.0.smithi107.stdout:10997: writing smithi10729407-8 from 0 to 565248 tid 1 2020-06-12T02:39:14.066 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-06-12T02:39:14.069 INFO:tasks.rados.rados.0.smithi107.stdout:10997: writing smithi10729407-8 from 565248 to 1318912 tid 2 2020-06-12T02:39:14.072 INFO:tasks.rados.rados.0.smithi107.stdout:10997: writing smithi10729407-8 from 1318912 to 1769472 tid 3 2020-06-12T02:39:14.072 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:39:14.072 INFO:tasks.rados.rados.0.smithi107.stdout:10998: seq_num 10997 ranges {0=581632,581632=638976,1220608=565248} 2020-06-12T02:39:14.076 INFO:tasks.rados.rados.0.smithi107.stdout:10998: writing smithi10729407-11 from 0 to 581632 tid 1 2020-06-12T02:39:14.080 INFO:tasks.rados.rados.0.smithi107.stdout:10998: writing smithi10729407-11 from 581632 to 1220608 tid 2 2020-06-12T02:39:14.083 INFO:tasks.rados.rados.0.smithi107.stdout:10998: writing smithi10729407-11 from 1220608 to 1785856 tid 3 2020-06-12T02:39:14.083 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:39:14.084 INFO:tasks.rados.rados.0.smithi107.stdout:10999: seq_num 10998 ranges {0=598016,598016=524288,1122304=491520,1613824=188416} 2020-06-12T02:39:14.087 INFO:tasks.rados.rados.0.smithi107.stdout:10999: writing smithi10729407-18 from 0 to 598016 tid 1 2020-06-12T02:39:14.091 INFO:tasks.rados.rados.0.smithi107.stdout:10999: writing smithi10729407-18 from 598016 to 1122304 tid 2 2020-06-12T02:39:14.094 INFO:tasks.rados.rados.0.smithi107.stdout:10999: writing smithi10729407-18 from 1122304 to 1613824 tid 3 2020-06-12T02:39:14.095 INFO:tasks.rados.rados.0.smithi107.stdout:10999: writing smithi10729407-18 from 1613824 to 1802240 tid 4 2020-06-12T02:39:14.096 INFO:tasks.rados.rados.0.smithi107.stdout: waiting on 8 2020-06-12T02:39:14.096 INFO:tasks.rados.rados.0.smithi107.stdout:10989: finishing write tid 3 to smithi10729407-15 2020-06-12T02:39:14.096 INFO:tasks.rados.rados.0.smithi107.stdout:10989: finishing write tid 4 to smithi10729407-15 2020-06-12T02:39:14.096 INFO:tasks.rados.rados.0.smithi107.stdout:10989: finishing write tid 6 to smithi10729407-15 2020-06-12T02:39:14.096 INFO:tasks.rados.rados.0.smithi107.stdout:10989: finishing write tid 7 to smithi10729407-15 2020-06-12T02:39:14.097 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 2033 (ObjNum 10988 snap 0 seq_num 10988) dirty exists 2020-06-12T02:39:14.097 INFO:tasks.rados.rados.0.smithi107.stdout:10989: left oid 15 (ObjNum 10988 snap 0 seq_num 10988) 2020-06-12T02:39:14.097 INFO:tasks.rados.rados.0.smithi107.stdout:10993: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:14.097 INFO:tasks.rados.rados.0.smithi107.stdout:10989: done (7 left) 2020-06-12T02:39:14.097 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:39:14.098 INFO:tasks.rados.rados.0.smithi107.stdout:11000: seq_num 10999 ranges {0=614400,614400=409600,1024000=647168,1671168=147456} 2020-06-12T02:39:14.099 INFO:tasks.rados.rados.0.smithi107.stdout:11000: writing smithi10729407-7 from 0 to 614400 tid 1 2020-06-12T02:39:14.102 INFO:tasks.rados.rados.0.smithi107.stdout:11000: writing smithi10729407-7 from 614400 to 1024000 tid 2 2020-06-12T02:39:14.106 INFO:tasks.rados.rados.0.smithi107.stdout:11000: writing smithi10729407-7 from 1024000 to 1671168 tid 3 2020-06-12T02:39:14.108 INFO:tasks.rados.rados.0.smithi107.stdout:11000: writing smithi10729407-7 from 1671168 to 1818624 tid 4 2020-06-12T02:39:14.108 INFO:tasks.rados.rados.0.smithi107.stdout: waiting on 8 2020-06-12T02:39:14.109 INFO:tasks.rados.rados.0.smithi107.stdout:10993: finishing write tid 2 to smithi10729407-14 2020-06-12T02:39:14.109 INFO:tasks.rados.rados.0.smithi107.stdout:10993: finishing write tid 3 to smithi10729407-14 2020-06-12T02:39:14.109 INFO:tasks.rados.rados.0.smithi107.stdout:10993: finishing write tid 5 to smithi10729407-14 2020-06-12T02:39:14.109 INFO:tasks.rados.rados.0.smithi107.stdout:10994: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:14.110 INFO:tasks.rados.rados.0.smithi107.stdout:10995: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:14.110 INFO:tasks.rados.rados.0.smithi107.stdout:10993: finishing write tid 6 to smithi10729407-14 2020-06-12T02:39:14.110 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1736 (ObjNum 10992 snap 0 seq_num 10992) dirty exists 2020-06-12T02:39:14.110 INFO:tasks.rados.rados.0.smithi107.stdout:10993: left oid 14 (ObjNum 10992 snap 0 seq_num 10992) 2020-06-12T02:39:14.111 INFO:tasks.rados.rados.0.smithi107.stdout:10994: finishing write tid 2 to smithi10729407-1 2020-06-12T02:39:14.111 INFO:tasks.rados.rados.0.smithi107.stdout:10994: finishing write tid 3 to smithi10729407-1 2020-06-12T02:39:14.111 INFO:tasks.rados.rados.0.smithi107.stdout:10994: finishing write tid 5 to smithi10729407-1 2020-06-12T02:39:14.111 INFO:tasks.rados.rados.0.smithi107.stdout:10994: finishing write tid 6 to smithi10729407-1 2020-06-12T02:39:14.112 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1976 (ObjNum 10993 snap 0 seq_num 10993) dirty exists 2020-06-12T02:39:14.112 INFO:tasks.rados.rados.0.smithi107.stdout:10994: left oid 1 (ObjNum 10993 snap 0 seq_num 10993) 2020-06-12T02:39:14.112 INFO:tasks.rados.rados.0.smithi107.stdout:10995: finishing write tid 2 to smithi10729407-10 2020-06-12T02:39:14.112 INFO:tasks.rados.rados.0.smithi107.stdout:10995: finishing write tid 3 to smithi10729407-10 2020-06-12T02:39:14.113 INFO:tasks.rados.rados.0.smithi107.stdout:10993: done (7 left) 2020-06-12T02:39:14.113 INFO:tasks.rados.rados.0.smithi107.stdout:10994: done (6 left) 2020-06-12T02:39:14.113 INFO:tasks.rados.rados.0.smithi107.stdout:11001: delete oid 4 current snap is 0 2020-06-12T02:39:14.113 INFO:tasks.rados.rados.0.smithi107.stdout:10995: finishing write tid 4 to smithi10729407-10 2020-06-12T02:39:14.113 INFO:tasks.rados.rados.0.smithi107.stdout:10995: finishing write tid 6 to smithi10729407-10 2020-06-12T02:39:14.114 INFO:tasks.rados.rados.0.smithi107.stdout:10995: finishing write tid 7 to smithi10729407-10 2020-06-12T02:39:14.114 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 815 (ObjNum 10994 snap 0 seq_num 10994) dirty exists 2020-06-12T02:39:14.114 INFO:tasks.rados.rados.0.smithi107.stdout:10995: left oid 10 (ObjNum 10994 snap 0 seq_num 10994) 2020-06-12T02:39:14.114 INFO:tasks.rados.rados.0.smithi107.stdout:10996: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:14.114 INFO:tasks.rados.rados.0.smithi107.stdout:10997: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:14.115 INFO:tasks.rados.rados.0.smithi107.stdout:10996: finishing write tid 2 to smithi10729407-16 2020-06-12T02:39:14.115 INFO:tasks.rados.rados.0.smithi107.stdout:10996: finishing write tid 3 to smithi10729407-16 2020-06-12T02:39:14.115 INFO:tasks.rados.rados.0.smithi107.stdout:10996: finishing write tid 4 to smithi10729407-16 2020-06-12T02:39:14.115 INFO:tasks.rados.rados.0.smithi107.stdout:10996: finishing write tid 6 to smithi10729407-16 2020-06-12T02:39:14.115 INFO:tasks.rados.rados.0.smithi107.stdout:10996: finishing write tid 7 to smithi10729407-16 2020-06-12T02:39:14.115 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1951 (ObjNum 10995 snap 0 seq_num 10995) dirty exists 2020-06-12T02:39:14.116 INFO:tasks.rados.rados.0.smithi107.stdout:10996: left oid 16 (ObjNum 10995 snap 0 seq_num 10995) 2020-06-12T02:39:14.116 INFO:tasks.rados.rados.0.smithi107.stdout:10997: finishing write tid 2 to smithi10729407-8 2020-06-12T02:39:14.116 INFO:tasks.rados.rados.0.smithi107.stdout:10997: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:14.116 INFO:tasks.rados.rados.0.smithi107.stdout:10997: finishing write tid 5 to smithi10729407-8 2020-06-12T02:39:14.116 INFO:tasks.rados.rados.0.smithi107.stdout:10998: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:14.117 INFO:tasks.rados.rados.0.smithi107.stdout:10997: finishing write tid 6 to smithi10729407-8 2020-06-12T02:39:14.117 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 2037 (ObjNum 10996 snap 0 seq_num 10996) dirty exists 2020-06-12T02:39:14.117 INFO:tasks.rados.rados.0.smithi107.stdout:10997: left oid 8 (ObjNum 10996 snap 0 seq_num 10996) 2020-06-12T02:39:14.117 INFO:tasks.rados.rados.0.smithi107.stdout:10998: finishing write tid 2 to smithi10729407-11 2020-06-12T02:39:14.117 INFO:tasks.rados.rados.0.smithi107.stdout:10999: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:14.117 INFO:tasks.rados.rados.0.smithi107.stdout:10998: finishing write tid 3 to smithi10729407-11 2020-06-12T02:39:14.118 INFO:tasks.rados.rados.0.smithi107.stdout:10998: finishing write tid 5 to smithi10729407-11 2020-06-12T02:39:14.118 INFO:tasks.rados.rados.0.smithi107.stdout:10998: finishing write tid 6 to smithi10729407-11 2020-06-12T02:39:14.118 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1760 (ObjNum 10997 snap 0 seq_num 10997) dirty exists 2020-06-12T02:39:14.118 INFO:tasks.rados.rados.0.smithi107.stdout:10998: left oid 11 (ObjNum 10997 snap 0 seq_num 10997) 2020-06-12T02:39:14.118 INFO:tasks.rados.rados.0.smithi107.stdout:10999: finishing write tid 2 to smithi10729407-18 2020-06-12T02:39:14.119 INFO:tasks.rados.rados.0.smithi107.stdout:10999: finishing write tid 3 to smithi10729407-18 2020-06-12T02:39:14.120 INFO:tasks.rados.rados.0.smithi107.stdout:10999: finishing write tid 4 to smithi10729407-18 2020-06-12T02:39:14.120 INFO:tasks.rados.rados.0.smithi107.stdout:10995: done (6 left) 2020-06-12T02:39:14.120 INFO:tasks.rados.rados.0.smithi107.stdout:10996: done (5 left) 2020-06-12T02:39:14.120 INFO:tasks.rados.rados.0.smithi107.stdout:10997: done (4 left) 2020-06-12T02:39:14.121 INFO:tasks.rados.rados.0.smithi107.stdout:10998: done (3 left) 2020-06-12T02:39:14.121 INFO:tasks.rados.rados.0.smithi107.stdout:11001: done (2 left) 2020-06-12T02:39:14.121 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:39:14.121 INFO:tasks.rados.rados.0.smithi107.stdout:11002: seq_num 11001 ranges {0=655360,655360=630784,1286144=573440} 2020-06-12T02:39:14.122 INFO:tasks.rados.rados.0.smithi107.stdout:11002: writing smithi10729407-4 from 0 to 655360 tid 1 2020-06-12T02:39:14.122 INFO:tasks.rados.rados.0.smithi107.stdout:11002: writing smithi10729407-4 from 655360 to 1286144 tid 2 2020-06-12T02:39:14.123 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:39:14.117+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:39:14.123 INFO:tasks.rados.rados.0.smithi107.stdout:11002: writing smithi10729407-4 from 1286144 to 1859584 tid 3 2020-06-12T02:39:14.123 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:39:14.123 INFO:tasks.rados.rados.0.smithi107.stdout:11003: seq_num 11002 ranges {3842048=671744,4513792=507904,5021696=475136,5496832=221184} 2020-06-12T02:39:14.139 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-06-12T02:39:14.147 INFO:tasks.rados.rados.0.smithi107.stdout:11003: writing smithi10729407-19 from 3842048 to 4513792 tid 1 2020-06-12T02:39:14.150 INFO:tasks.rados.rados.0.smithi107.stdout:11003: writing smithi10729407-19 from 4513792 to 5021696 tid 2 2020-06-12T02:39:14.153 INFO:tasks.rados.rados.0.smithi107.stdout:11003: writing smithi10729407-19 from 5021696 to 5496832 tid 3 2020-06-12T02:39:14.154 INFO:tasks.rados.rados.0.smithi107.stdout:11003: writing smithi10729407-19 from 5496832 to 5718016 tid 4 2020-06-12T02:39:14.155 INFO:tasks.rados.rados.0.smithi107.stdout:11004: delete oid 5 current snap is 0 2020-06-12T02:39:14.155 INFO:tasks.rados.rados.0.smithi107.stdout:10999: finishing write tid 6 to smithi10729407-18 2020-06-12T02:39:14.155 INFO:tasks.rados.rados.0.smithi107.stdout:10999: finishing write tid 7 to smithi10729407-18 2020-06-12T02:39:14.155 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1880 (ObjNum 10998 snap 0 seq_num 10998) dirty exists 2020-06-12T02:39:14.155 INFO:tasks.rados.rados.0.smithi107.stdout:10999: left oid 18 (ObjNum 10998 snap 0 seq_num 10998) 2020-06-12T02:39:14.156 INFO:tasks.rados.rados.0.smithi107.stdout:11000: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:14.156 INFO:tasks.rados.rados.0.smithi107.stdout:11000: finishing write tid 2 to smithi10729407-7 2020-06-12T02:39:14.156 INFO:tasks.rados.rados.0.smithi107.stdout:11000: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:14.156 INFO:tasks.rados.rados.0.smithi107.stdout:11002: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:14.156 INFO:tasks.rados.rados.0.smithi107.stdout:11000: finishing write tid 4 to smithi10729407-7 2020-06-12T02:39:14.156 INFO:tasks.rados.rados.0.smithi107.stdout:11000: finishing write tid 6 to smithi10729407-7 2020-06-12T02:39:14.157 INFO:tasks.rados.rados.0.smithi107.stdout:11000: finishing write tid 7 to smithi10729407-7 2020-06-12T02:39:14.157 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1795 (ObjNum 10999 snap 0 seq_num 10999) dirty exists 2020-06-12T02:39:14.157 INFO:tasks.rados.rados.0.smithi107.stdout:11000: left oid 7 (ObjNum 10999 snap 0 seq_num 10999) 2020-06-12T02:39:14.157 INFO:tasks.rados.rados.0.smithi107.stdout:11002: finishing write tid 2 to smithi10729407-4 2020-06-12T02:39:14.157 INFO:tasks.rados.rados.0.smithi107.stdout:11002: finishing write tid 3 to smithi10729407-4 2020-06-12T02:39:14.158 INFO:tasks.rados.rados.0.smithi107.stdout:11002: finishing write tid 5 to smithi10729407-4 2020-06-12T02:39:14.158 INFO:tasks.rados.rados.0.smithi107.stdout:11002: finishing write tid 6 to smithi10729407-4 2020-06-12T02:39:14.158 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 2069 (ObjNum 11001 snap 0 seq_num 11001) dirty exists 2020-06-12T02:39:14.158 INFO:tasks.rados.rados.0.smithi107.stdout:11002: left oid 4 (ObjNum 11001 snap 0 seq_num 11001) 2020-06-12T02:39:14.158 INFO:tasks.rados.rados.0.smithi107.stdout:11003: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:14.159 INFO:tasks.rados.rados.0.smithi107.stdout:10999: done (4 left) 2020-06-12T02:39:14.159 INFO:tasks.rados.rados.0.smithi107.stdout:11000: done (3 left) 2020-06-12T02:39:14.159 INFO:tasks.rados.rados.0.smithi107.stdout:11002: done (2 left) 2020-06-12T02:39:14.160 INFO:tasks.rados.rados.0.smithi107.stdout:11004: done (1 left) 2020-06-12T02:39:14.160 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:39:14.160 INFO:tasks.rados.rados.0.smithi107.stdout:11005: seq_num 11004 ranges {0=704512,704512=679936,1384448=442368,1826816=81920} 2020-06-12T02:39:14.161 INFO:tasks.rados.rados.0.smithi107.stdout:11005: writing smithi10729407-2 from 0 to 704512 tid 1 2020-06-12T02:39:14.166 INFO:tasks.rados.rados.0.smithi107.stdout:11005: writing smithi10729407-2 from 704512 to 1384448 tid 2 2020-06-12T02:39:14.169 INFO:tasks.rados.rados.0.smithi107.stdout:11005: writing smithi10729407-2 from 1384448 to 1826816 tid 3 2020-06-12T02:39:14.169 INFO:tasks.rados.rados.0.smithi107.stdout:11005: writing smithi10729407-2 from 1826816 to 1908736 tid 4 2020-06-12T02:39:14.170 INFO:tasks.rados.rados.0.smithi107.stdout:11006: delete oid 8 current snap is 0 2020-06-12T02:39:14.170 INFO:tasks.rados.rados.0.smithi107.stdout:11003: finishing write tid 2 to smithi10729407-19 2020-06-12T02:39:14.170 INFO:tasks.rados.rados.0.smithi107.stdout:11003: finishing write tid 3 to smithi10729407-19 2020-06-12T02:39:14.170 INFO:tasks.rados.rados.0.smithi107.stdout:11003: finishing write tid 4 to smithi10729407-19 2020-06-12T02:39:14.170 INFO:tasks.rados.rados.0.smithi107.stdout:11005: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:14.171 INFO:tasks.rados.rados.0.smithi107.stdout:11003: finishing write tid 6 to smithi10729407-19 2020-06-12T02:39:14.172 INFO:tasks.rados.rados.0.smithi107.stdout:11003: finishing write tid 7 to smithi10729407-19 2020-06-12T02:39:14.173 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 2036 (ObjNum 11002 snap 0 seq_num 11002) dirty exists 2020-06-12T02:39:14.173 INFO:tasks.rados.rados.0.smithi107.stdout:11003: left oid 19 (ObjNum 11002 snap 0 seq_num 11002) 2020-06-12T02:39:14.176 INFO:tasks.rados.rados.0.smithi107.stdout:11003: done (2 left) 2020-06-12T02:39:14.177 INFO:tasks.rados.rados.0.smithi107.stdout:11006: done (1 left) 2020-06-12T02:39:14.177 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:39:14.177 INFO:tasks.rados.rados.0.smithi107.stdout:11007: seq_num 11006 ranges {0=737280,737280=450560,1187840=450560,1638400=303104} 2020-06-12T02:39:14.181 INFO:tasks.rados.rados.0.smithi107.stdout:11007: writing smithi10729407-5 from 0 to 737280 tid 1 2020-06-12T02:39:14.185 INFO:tasks.rados.rados.0.smithi107.stdout:11007: writing smithi10729407-5 from 737280 to 1187840 tid 2 2020-06-12T02:39:14.187 INFO:tasks.rados.rados.0.smithi107.stdout:11007: writing smithi10729407-5 from 1187840 to 1638400 tid 3 2020-06-12T02:39:14.189 INFO:tasks.rados.rados.0.smithi107.stdout:11007: writing smithi10729407-5 from 1638400 to 1941504 tid 4 2020-06-12T02:39:14.189 INFO:tasks.rados.rados.0.smithi107.stdout:11008: delete oid 6 current snap is 0 2020-06-12T02:39:14.189 INFO:tasks.rados.rados.0.smithi107.stdout:11005: finishing write tid 2 to smithi10729407-2 2020-06-12T02:39:14.190 INFO:tasks.rados.rados.0.smithi107.stdout:11005: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:14.190 INFO:tasks.rados.rados.0.smithi107.stdout:11005: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:14.190 INFO:tasks.rados.rados.0.smithi107.stdout:11005: finishing write tid 6 to smithi10729407-2 2020-06-12T02:39:14.190 INFO:tasks.rados.rados.0.smithi107.stdout:11005: finishing write tid 7 to smithi10729407-2 2020-06-12T02:39:14.190 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 1038 (ObjNum 11004 snap 0 seq_num 11004) dirty exists 2020-06-12T02:39:14.191 INFO:tasks.rados.rados.0.smithi107.stdout:11005: left oid 2 (ObjNum 11004 snap 0 seq_num 11004) 2020-06-12T02:39:14.191 INFO:tasks.rados.rados.0.smithi107.stdout:11007: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:14.200 INFO:tasks.rados.rados.0.smithi107.stdout:11005: done (2 left) 2020-06-12T02:39:14.200 INFO:tasks.rados.rados.0.smithi107.stdout:11008: done (1 left) 2020-06-12T02:39:14.201 INFO:tasks.rados.rados.0.smithi107.stdout:11009: delete oid 6 current snap is 0 2020-06-12T02:39:14.201 INFO:tasks.rados.rados.0.smithi107.stdout:11009: done (1 left) 2020-06-12T02:39:14.201 INFO:tasks.rados.rados.0.smithi107.stdout:11010: delete oid 4 current snap is 0 2020-06-12T02:39:14.201 INFO:tasks.rados.rados.0.smithi107.stdout:11007: finishing write tid 2 to smithi10729407-5 2020-06-12T02:39:14.202 INFO:tasks.rados.rados.0.smithi107.stdout:11007: finishing write tid 3 to smithi10729407-5 2020-06-12T02:39:14.202 INFO:tasks.rados.rados.0.smithi107.stdout:11007: finishing write tid 4 to smithi10729407-5 2020-06-12T02:39:14.203 INFO:tasks.rados.rados.0.smithi107.stdout:11007: finishing write tid 6 to smithi10729407-5 2020-06-12T02:39:14.206 INFO:tasks.rados.rados.0.smithi107.stdout:11010: done (1 left) 2020-06-12T02:39:14.206 INFO:tasks.rados.rados.0.smithi107.stdout:11011: delete oid 11 current snap is 0 2020-06-12T02:39:14.206 INFO:tasks.rados.rados.0.smithi107.stdout:11011: done (1 left) 2020-06-12T02:39:14.207 INFO:tasks.rados.rados.0.smithi107.stdout:11012: delete oid 11 current snap is 0 2020-06-12T02:39:14.207 INFO:tasks.rados.rados.0.smithi107.stdout:11007: finishing write tid 7 to smithi10729407-5 2020-06-12T02:39:14.208 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 1068 (ObjNum 11006 snap 0 seq_num 11006) dirty exists 2020-06-12T02:39:14.208 INFO:tasks.rados.rados.0.smithi107.stdout:11007: left oid 5 (ObjNum 11006 snap 0 seq_num 11006) 2020-06-12T02:39:14.209 INFO:tasks.rados.rados.0.smithi107.stdout:11007: done (1 left) 2020-06-12T02:39:14.209 INFO:tasks.rados.rados.0.smithi107.stdout:11012: done (0 left) 2020-06-12T02:39:14.209 INFO:tasks.rados.rados.0.smithi107.stdout:11013: delete oid 4 current snap is 0 2020-06-12T02:39:14.211 INFO:tasks.rados.rados.0.smithi107.stdout:11013: done (0 left) 2020-06-12T02:39:14.211 INFO:tasks.rados.rados.0.smithi107.stdout:11014: delete oid 20 current snap is 0 2020-06-12T02:39:14.213 INFO:tasks.rados.rados.0.smithi107.stdout:11014: done (0 left) 2020-06-12T02:39:14.213 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:39:14.214 INFO:tasks.rados.rados.0.smithi107.stdout:11015: seq_num 11014 ranges {1818624=466944,2285568=778240,3063808=417792,3481600=409600} 2020-06-12T02:39:14.220 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:39:14.217+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:39:14.225 INFO:tasks.rados.rados.0.smithi107.stdout:11015: writing smithi10729407-7 from 1818624 to 2285568 tid 1 2020-06-12T02:39:14.229 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:14.229 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:14.230 INFO:tasks.rados.rados.0.smithi107.stdout:11015: writing smithi10729407-7 from 2285568 to 3063808 tid 2 2020-06-12T02:39:14.232 INFO:tasks.rados.rados.0.smithi107.stdout:11015: writing smithi10729407-7 from 3063808 to 3481600 tid 3 2020-06-12T02:39:14.235 INFO:tasks.rados.rados.0.smithi107.stdout:11015: writing smithi10729407-7 from 3481600 to 3891200 tid 4 2020-06-12T02:39:14.235 INFO:tasks.rados.rados.0.smithi107.stdout:11016: delete oid 5 current snap is 0 2020-06-12T02:39:14.235 INFO:tasks.rados.rados.0.smithi107.stdout:11015: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:14.239 INFO:tasks.rados.rados.0.smithi107.stdout:11016: done (1 left) 2020-06-12T02:39:14.240 INFO:tasks.rados.rados.0.smithi107.stdout:11017: delete oid 14 current snap is 0 2020-06-12T02:39:14.244 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-06-12T02:39:14.245 INFO:tasks.rados.rados.0.smithi107.stdout:11015: finishing write tid 2 to smithi10729407-7 2020-06-12T02:39:14.246 INFO:tasks.rados.rados.0.smithi107.stdout:11017: done (1 left) 2020-06-12T02:39:14.246 INFO:tasks.rados.rados.0.smithi107.stdout:11018: delete oid 15 current snap is 0 2020-06-12T02:39:14.246 INFO:tasks.rados.rados.0.smithi107.stdout:11015: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:14.248 INFO:tasks.rados.rados.0.smithi107.stdout:11015: finishing write tid 4 to smithi10729407-7 2020-06-12T02:39:14.249 INFO:tasks.rados.rados.0.smithi107.stdout:11015: finishing write tid 6 to smithi10729407-7 2020-06-12T02:39:14.252 INFO:tasks.rados.rados.0.smithi107.stdout:11015: finishing write tid 7 to smithi10729407-7 2020-06-12T02:39:14.252 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1800 (ObjNum 11014 snap 0 seq_num 11014) dirty exists 2020-06-12T02:39:14.252 INFO:tasks.rados.rados.0.smithi107.stdout:11015: left oid 7 (ObjNum 11014 snap 0 seq_num 11014) 2020-06-12T02:39:14.254 INFO:tasks.rados.rados.0.smithi107.stdout:11015: done (1 left) 2020-06-12T02:39:14.255 INFO:tasks.rados.rados.0.smithi107.stdout:11018: done (0 left) 2020-06-12T02:39:14.255 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:39:14.256 INFO:tasks.rados.rados.0.smithi107.stdout:11019: seq_num 11018 ranges {0=532480,532480=761856,1294336=794624,2088960=49152} 2020-06-12T02:39:14.258 INFO:tasks.rados.rados.0.smithi107.stdout:11019: writing smithi10729407-6 from 0 to 532480 tid 1 2020-06-12T02:39:14.262 INFO:tasks.rados.rados.0.smithi107.stdout:11019: writing smithi10729407-6 from 532480 to 1294336 tid 2 2020-06-12T02:39:14.267 INFO:tasks.rados.rados.0.smithi107.stdout:11019: writing smithi10729407-6 from 1294336 to 2088960 tid 3 2020-06-12T02:39:14.268 INFO:tasks.rados.rados.0.smithi107.stdout:11019: writing smithi10729407-6 from 2088960 to 2138112 tid 4 2020-06-12T02:39:14.268 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:39:14.268 INFO:tasks.rados.rados.0.smithi107.stdout:11020: seq_num 11019 ranges {0=548864,548864=647168,1196032=606208,1802240=352256} 2020-06-12T02:39:14.271 INFO:tasks.rados.rados.0.smithi107.stdout:11020: writing smithi10729407-8 from 0 to 548864 tid 1 2020-06-12T02:39:14.275 INFO:tasks.rados.rados.0.smithi107.stdout:11020: writing smithi10729407-8 from 548864 to 1196032 tid 2 2020-06-12T02:39:14.279 INFO:tasks.rados.rados.0.smithi107.stdout:11020: writing smithi10729407-8 from 1196032 to 1802240 tid 3 2020-06-12T02:39:14.282 INFO:tasks.rados.rados.0.smithi107.stdout:11020: writing smithi10729407-8 from 1802240 to 2154496 tid 4 2020-06-12T02:39:14.282 INFO:tasks.rados.rados.0.smithi107.stdout:11021: delete oid 5 current snap is 0 2020-06-12T02:39:14.283 INFO:tasks.rados.rados.0.smithi107.stdout:11019: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:14.283 INFO:tasks.rados.rados.0.smithi107.stdout:11019: finishing write tid 2 to smithi10729407-6 2020-06-12T02:39:14.283 INFO:tasks.rados.rados.0.smithi107.stdout:11019: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:14.283 INFO:tasks.rados.rados.0.smithi107.stdout:11019: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:14.283 INFO:tasks.rados.rados.0.smithi107.stdout:11019: finishing write tid 6 to smithi10729407-6 2020-06-12T02:39:14.284 INFO:tasks.rados.rados.0.smithi107.stdout:11019: finishing write tid 7 to smithi10729407-6 2020-06-12T02:39:14.284 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1983 (ObjNum 11018 snap 0 seq_num 11018) dirty exists 2020-06-12T02:39:14.284 INFO:tasks.rados.rados.0.smithi107.stdout:11019: left oid 6 (ObjNum 11018 snap 0 seq_num 11018) 2020-06-12T02:39:14.284 INFO:tasks.rados.rados.0.smithi107.stdout:11019: done (2 left) 2020-06-12T02:39:14.285 INFO:tasks.rados.rados.0.smithi107.stdout:11021: done (1 left) 2020-06-12T02:39:14.285 INFO:tasks.rados.rados.0.smithi107.stdout:11022: delete oid 7 current snap is 0 2020-06-12T02:39:14.286 INFO:tasks.rados.rados.0.smithi107.stdout:11020: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:14.287 INFO:tasks.rados.rados.0.smithi107.stdout:11020: finishing write tid 2 to smithi10729407-8 2020-06-12T02:39:14.289 INFO:tasks.rados.rados.0.smithi107.stdout:11022: done (1 left) 2020-06-12T02:39:14.289 INFO:tasks.rados.rados.0.smithi107.stdout:11023: delete oid 20 current snap is 0 2020-06-12T02:39:14.291 INFO:tasks.rados.rados.0.smithi107.stdout:11023: done (1 left) 2020-06-12T02:39:14.291 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:39:14.291 INFO:tasks.rados.rados.0.smithi107.stdout:11024: seq_num 11023 ranges {0=622592,622592=581632,1204224=581632,1785856=434176,2220032=8192} 2020-06-12T02:39:14.295 INFO:tasks.rados.rados.0.smithi107.stdout:11024: writing smithi10729407-12 from 0 to 622592 tid 1 2020-06-12T02:39:14.299 INFO:tasks.rados.rados.0.smithi107.stdout:11024: writing smithi10729407-12 from 622592 to 1204224 tid 2 2020-06-12T02:39:14.302 INFO:tasks.rados.rados.0.smithi107.stdout:11024: writing smithi10729407-12 from 1204224 to 1785856 tid 3 2020-06-12T02:39:14.305 INFO:tasks.rados.rados.0.smithi107.stdout:11024: writing smithi10729407-12 from 1785856 to 2220032 tid 4 2020-06-12T02:39:14.306 INFO:tasks.rados.rados.0.smithi107.stdout:11024: writing smithi10729407-12 from 2220032 to 2228224 tid 5 2020-06-12T02:39:14.306 INFO:tasks.rados.rados.0.smithi107.stdout:11025: delete oid 9 current snap is 0 2020-06-12T02:39:14.306 INFO:tasks.rados.rados.0.smithi107.stdout:11020: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:14.306 INFO:tasks.rados.rados.0.smithi107.stdout:11020: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:14.306 INFO:tasks.rados.rados.0.smithi107.stdout:11020: finishing write tid 6 to smithi10729407-8 2020-06-12T02:39:14.307 INFO:tasks.rados.rados.0.smithi107.stdout:11020: finishing write tid 7 to smithi10729407-8 2020-06-12T02:39:14.307 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 2044 (ObjNum 11019 snap 0 seq_num 11019) dirty exists 2020-06-12T02:39:14.307 INFO:tasks.rados.rados.0.smithi107.stdout:11020: left oid 8 (ObjNum 11019 snap 0 seq_num 11019) 2020-06-12T02:39:14.307 INFO:tasks.rados.rados.0.smithi107.stdout:11024: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:14.308 INFO:tasks.rados.rados.0.smithi107.stdout:11020: done (2 left) 2020-06-12T02:39:14.308 INFO:tasks.rados.rados.0.smithi107.stdout:11025: done (1 left) 2020-06-12T02:39:14.309 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:39:14.309 INFO:tasks.rados.rados.0.smithi107.stdout:11026: seq_num 11025 ranges {1720320=655360,2375680=802816,3178496=548864,3727360=253952} 2020-06-12T02:39:14.320 INFO:tasks.rados.rados.0.smithi107.stdout:11026: writing smithi10729407-1 from 1720320 to 2375680 tid 1 2020-06-12T02:39:14.325 INFO:tasks.rados.rados.0.smithi107.stdout:11026: writing smithi10729407-1 from 2375680 to 3178496 tid 2 2020-06-12T02:39:14.327 INFO:tasks.rados.rados.0.smithi107.stdout:11026: writing smithi10729407-1 from 3178496 to 3727360 tid 3 2020-06-12T02:39:14.329 INFO:tasks.rados.rados.0.smithi107.stdout:11026: writing smithi10729407-1 from 3727360 to 3981312 tid 4 2020-06-12T02:39:14.329 INFO:tasks.rados.rados.0.smithi107.stdout:11027: delete oid 4 current snap is 0 2020-06-12T02:39:14.330 INFO:tasks.rados.rados.0.smithi107.stdout:11024: finishing write tid 2 to smithi10729407-12 2020-06-12T02:39:14.330 INFO:tasks.rados.rados.0.smithi107.stdout:11024: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:14.330 INFO:tasks.rados.rados.0.smithi107.stdout:11024: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:14.330 INFO:tasks.rados.rados.0.smithi107.stdout:11024: finishing write tid 5 to smithi10729407-12 2020-06-12T02:39:14.331 INFO:tasks.rados.rados.0.smithi107.stdout:11024: finishing write tid 7 to smithi10729407-12 2020-06-12T02:39:14.331 INFO:tasks.rados.rados.0.smithi107.stdout:11024: finishing write tid 8 to smithi10729407-12 2020-06-12T02:39:14.331 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 2042 (ObjNum 11023 snap 0 seq_num 11023) dirty exists 2020-06-12T02:39:14.331 INFO:tasks.rados.rados.0.smithi107.stdout:11024: left oid 12 (ObjNum 11023 snap 0 seq_num 11023) 2020-06-12T02:39:14.332 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:39:14.329+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:39:14.333 INFO:tasks.rados.rados.0.smithi107.stdout:11024: done (2 left) 2020-06-12T02:39:14.333 INFO:tasks.rados.rados.0.smithi107.stdout:11027: done (1 left) 2020-06-12T02:39:14.333 INFO:tasks.rados.rados.0.smithi107.stdout:11028: delete oid 3 current snap is 0 2020-06-12T02:39:14.338 INFO:tasks.rados.rados.0.smithi107.stdout:11028: done (1 left) 2020-06-12T02:39:14.338 INFO:tasks.rados.rados.0.smithi107.stdout:11029: delete oid 2 current snap is 0 2020-06-12T02:39:14.338 INFO:tasks.rados.rados.0.smithi107.stdout:11026: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:14.339 INFO:tasks.rados.rados.0.smithi107.stdout:11026: finishing write tid 2 to smithi10729407-1 2020-06-12T02:39:14.341 INFO:tasks.rados.rados.0.smithi107.stdout:11026: finishing write tid 3 to smithi10729407-1 2020-06-12T02:39:14.341 INFO:tasks.rados.rados.0.smithi107.stdout:11026: finishing write tid 4 to smithi10729407-1 2020-06-12T02:39:14.341 INFO:tasks.rados.rados.0.smithi107.stdout:11029: done (1 left) 2020-06-12T02:39:14.342 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:39:14.342 INFO:tasks.rados.rados.0.smithi107.stdout:11030: seq_num 11029 ranges {0=720896,720896=737280,1458176=532480,1990656=335872} 2020-06-12T02:39:14.345 INFO:tasks.rados.rados.0.smithi107.stdout:11030: writing smithi10729407-13 from 0 to 720896 tid 1 2020-06-12T02:39:14.350 INFO:tasks.rados.rados.0.smithi107.stdout:11030: writing smithi10729407-13 from 720896 to 1458176 tid 2 2020-06-12T02:39:14.356 INFO:tasks.rados.rados.0.smithi107.stdout:11030: writing smithi10729407-13 from 1458176 to 1990656 tid 3 2020-06-12T02:39:14.356 INFO:tasks.rados.rados.0.smithi107.stdout:11030: writing smithi10729407-13 from 1990656 to 2326528 tid 4 2020-06-12T02:39:14.356 INFO:tasks.rados.rados.0.smithi107.stdout:11031: delete oid 15 current snap is 0 2020-06-12T02:39:14.357 INFO:tasks.rados.rados.0.smithi107.stdout:11026: finishing write tid 6 to smithi10729407-1 2020-06-12T02:39:14.357 INFO:tasks.rados.rados.0.smithi107.stdout:11026: finishing write tid 7 to smithi10729407-1 2020-06-12T02:39:14.357 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1988 (ObjNum 11025 snap 0 seq_num 11025) dirty exists 2020-06-12T02:39:14.357 INFO:tasks.rados.rados.0.smithi107.stdout:11026: left oid 1 (ObjNum 11025 snap 0 seq_num 11025) 2020-06-12T02:39:14.358 INFO:tasks.rados.rados.0.smithi107.stdout:11030: finishing write tid 1 to smithi10729407-13 2020-06-12T02:39:14.359 INFO:tasks.rados.rados.0.smithi107.stdout:11026: done (2 left) 2020-06-12T02:39:14.359 INFO:tasks.rados.rados.0.smithi107.stdout:11031: done (1 left) 2020-06-12T02:39:14.359 INFO:tasks.rados.rados.0.smithi107.stdout:11032: delete oid 15 current snap is 0 2020-06-12T02:39:14.362 INFO:tasks.rados.rados.0.smithi107.stdout:11032: done (1 left) 2020-06-12T02:39:14.362 INFO:tasks.rados.rados.0.smithi107.stdout:11033: delete oid 3 current snap is 0 2020-06-12T02:39:14.372 INFO:tasks.rados.rados.0.smithi107.stdout:11030: finishing write tid 2 to smithi10729407-13 2020-06-12T02:39:14.373 INFO:tasks.rados.rados.0.smithi107.stdout:11030: finishing write tid 3 to smithi10729407-13 2020-06-12T02:39:14.373 INFO:tasks.rados.rados.0.smithi107.stdout:11030: finishing write tid 4 to smithi10729407-13 2020-06-12T02:39:14.373 INFO:tasks.rados.rados.0.smithi107.stdout:11030: finishing write tid 6 to smithi10729407-13 2020-06-12T02:39:14.373 INFO:tasks.rados.rados.0.smithi107.stdout:11030: finishing write tid 7 to smithi10729407-13 2020-06-12T02:39:14.374 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1782 (ObjNum 11029 snap 0 seq_num 11029) dirty exists 2020-06-12T02:39:14.374 INFO:tasks.rados.rados.0.smithi107.stdout:11030: left oid 13 (ObjNum 11029 snap 0 seq_num 11029) 2020-06-12T02:39:14.374 INFO:tasks.rados.rados.0.smithi107.stdout:11030: done (1 left) 2020-06-12T02:39:14.375 INFO:tasks.rados.rados.0.smithi107.stdout:11033: done (0 left) 2020-06-12T02:39:14.375 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:39:14.375 INFO:tasks.rados.rados.0.smithi107.stdout:11034: seq_num 11033 ranges {2228224=786432,3014656=729088,3743744=507904,4251648=368640} 2020-06-12T02:39:14.382 INFO:tasks.rados.rados.0.smithi107.stdout:11034: writing smithi10729407-12 from 2228224 to 3014656 tid 1 2020-06-12T02:39:14.387 INFO:tasks.rados.rados.0.smithi107.stdout:11034: writing smithi10729407-12 from 3014656 to 3743744 tid 2 2020-06-12T02:39:14.389 INFO:tasks.rados.rados.0.smithi107.stdout:11034: writing smithi10729407-12 from 3743744 to 4251648 tid 3 2020-06-12T02:39:14.395 INFO:tasks.rados.rados.0.smithi107.stdout:11034: writing smithi10729407-12 from 4251648 to 4620288 tid 4 2020-06-12T02:39:14.396 INFO:tasks.rados.rados.0.smithi107.stdout:11035: delete oid 14 current snap is 0 2020-06-12T02:39:14.397 INFO:tasks.rados.rados.0.smithi107.stdout:11034: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:14.397 INFO:tasks.rados.rados.0.smithi107.stdout:11035: done (1 left) 2020-06-12T02:39:14.397 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:39:14.397 INFO:tasks.rados.rados.0.smithi107.stdout:11036: seq_num 11035 ranges {3194880=16384} 2020-06-12T02:39:14.412 INFO:tasks.rados.rados.0.smithi107.stdout:11036: writing smithi10729407-16 from 3194880 to 3211264 tid 1 2020-06-12T02:39:14.412 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:39:14.413 INFO:tasks.rados.rados.0.smithi107.stdout:11034: finishing write tid 2 to smithi10729407-12 2020-06-12T02:39:14.413 INFO:tasks.rados.rados.0.smithi107.stdout:11034: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:14.413 INFO:tasks.rados.rados.0.smithi107.stdout:11034: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:14.413 INFO:tasks.rados.rados.0.smithi107.stdout:11037: seq_num 11036 ranges {3981312=32768} 2020-06-12T02:39:14.421 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:39:14.419+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:39:14.434 INFO:tasks.rados.rados.0.smithi107.stdout:11037: writing smithi10729407-1 from 3981312 to 4014080 tid 1 2020-06-12T02:39:14.435 INFO:tasks.rados.rados.0.smithi107.stdout:11038: delete oid 13 current snap is 0 2020-06-12T02:39:14.435 INFO:tasks.rados.rados.0.smithi107.stdout:11034: finishing write tid 6 to smithi10729407-12 2020-06-12T02:39:14.435 INFO:tasks.rados.rados.0.smithi107.stdout:11034: finishing write tid 7 to smithi10729407-12 2020-06-12T02:39:14.435 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 2047 (ObjNum 11033 snap 0 seq_num 11033) dirty exists 2020-06-12T02:39:14.435 INFO:tasks.rados.rados.0.smithi107.stdout:11034: left oid 12 (ObjNum 11033 snap 0 seq_num 11033) 2020-06-12T02:39:14.436 INFO:tasks.rados.rados.0.smithi107.stdout:11036: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:14.436 INFO:tasks.rados.rados.0.smithi107.stdout:11036: finishing write tid 3 to smithi10729407-16 2020-06-12T02:39:14.436 INFO:tasks.rados.rados.0.smithi107.stdout:11036: finishing write tid 4 to smithi10729407-16 2020-06-12T02:39:14.436 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1955 (ObjNum 11035 snap 0 seq_num 11035) dirty exists 2020-06-12T02:39:14.437 INFO:tasks.rados.rados.0.smithi107.stdout:11036: left oid 16 (ObjNum 11035 snap 0 seq_num 11035) 2020-06-12T02:39:14.437 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd set nodeep-scrub 2020-06-12T02:39:14.439 INFO:tasks.rados.rados.0.smithi107.stdout:11034: done (3 left) 2020-06-12T02:39:14.439 INFO:tasks.rados.rados.0.smithi107.stdout:11036: done (2 left) 2020-06-12T02:39:14.439 INFO:tasks.rados.rados.0.smithi107.stdout:11038: done (1 left) 2020-06-12T02:39:14.439 INFO:tasks.rados.rados.0.smithi107.stdout:11039: delete oid 10 current snap is 0 2020-06-12T02:39:14.440 INFO:tasks.rados.rados.0.smithi107.stdout:11037: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:14.440 INFO:tasks.rados.rados.0.smithi107.stdout:11039: done (1 left) 2020-06-12T02:39:14.440 INFO:tasks.rados.rados.0.smithi107.stdout:11040: delete oid 2 current snap is 0 2020-06-12T02:39:14.445 INFO:tasks.rados.rados.0.smithi107.stdout:11040: done (1 left) 2020-06-12T02:39:14.446 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:39:14.446 INFO:tasks.rados.rados.0.smithi107.stdout:11041: seq_num 11040 ranges {0=106496} 2020-06-12T02:39:14.447 INFO:tasks.rados.rados.0.smithi107.stdout:11041: writing smithi10729407-5 from 0 to 106496 tid 1 2020-06-12T02:39:14.447 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:39:14.447 INFO:tasks.rados.rados.0.smithi107.stdout:11042: seq_num 11041 ranges {2154496=122880} 2020-06-12T02:39:14.449 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:14.450 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:14.459 INFO:tasks.rados.rados.0.smithi107.stdout:11042: writing smithi10729407-8 from 2154496 to 2277376 tid 1 2020-06-12T02:39:14.459 INFO:tasks.rados.rados.0.smithi107.stdout:11043: delete oid 12 current snap is 0 2020-06-12T02:39:14.460 INFO:tasks.rados.rados.0.smithi107.stdout:11037: finishing write tid 3 to smithi10729407-1 2020-06-12T02:39:14.460 INFO:tasks.rados.rados.0.smithi107.stdout:11037: finishing write tid 4 to smithi10729407-1 2020-06-12T02:39:14.460 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1990 (ObjNum 11036 snap 0 seq_num 11036) dirty exists 2020-06-12T02:39:14.460 INFO:tasks.rados.rados.0.smithi107.stdout:11037: left oid 1 (ObjNum 11036 snap 0 seq_num 11036) 2020-06-12T02:39:14.460 INFO:tasks.rados.rados.0.smithi107.stdout:11041: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:14.461 INFO:tasks.rados.rados.0.smithi107.stdout:11041: finishing write tid 3 to smithi10729407-5 2020-06-12T02:39:14.461 INFO:tasks.rados.rados.0.smithi107.stdout:11041: finishing write tid 4 to smithi10729407-5 2020-06-12T02:39:14.461 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 1072 (ObjNum 11040 snap 0 seq_num 11040) dirty exists 2020-06-12T02:39:14.461 INFO:tasks.rados.rados.0.smithi107.stdout:11041: left oid 5 (ObjNum 11040 snap 0 seq_num 11040) 2020-06-12T02:39:14.463 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-06-12T02:39:14.463 INFO:tasks.rados.rados.0.smithi107.stdout:11037: done (3 left) 2020-06-12T02:39:14.464 INFO:tasks.rados.rados.0.smithi107.stdout:11041: done (2 left) 2020-06-12T02:39:14.464 INFO:tasks.rados.rados.0.smithi107.stdout:11043: done (1 left) 2020-06-12T02:39:14.464 INFO:tasks.rados.rados.0.smithi107.stdout:11044: delete oid 19 current snap is 0 2020-06-12T02:39:14.464 INFO:tasks.rados.rados.0.smithi107.stdout:11042: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:14.467 INFO:tasks.rados.rados.0.smithi107.stdout:11042: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:14.469 INFO:tasks.rados.rados.0.smithi107.stdout:11042: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:14.469 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 2048 (ObjNum 11041 snap 0 seq_num 11041) dirty exists 2020-06-12T02:39:14.470 INFO:tasks.rados.rados.0.smithi107.stdout:11042: left oid 8 (ObjNum 11041 snap 0 seq_num 11041) 2020-06-12T02:39:14.472 INFO:tasks.rados.rados.0.smithi107.stdout:11042: done (1 left) 2020-06-12T02:39:14.472 INFO:tasks.rados.rados.0.smithi107.stdout:11044: done (0 left) 2020-06-12T02:39:14.472 INFO:tasks.rados.rados.0.smithi107.stdout:11045: delete oid 17 current snap is 0 2020-06-12T02:39:14.475 INFO:tasks.rados.rados.0.smithi107.stdout:11045: done (0 left) 2020-06-12T02:39:14.476 INFO:tasks.rados.rados.0.smithi107.stdout:11046: delete oid 4 current snap is 0 2020-06-12T02:39:14.481 INFO:tasks.rados.rados.0.smithi107.stdout:11046: done (0 left) 2020-06-12T02:39:14.482 INFO:tasks.rados.rados.0.smithi107.stdout:11047: delete oid 17 current snap is 0 2020-06-12T02:39:14.484 INFO:tasks.rados.rados.0.smithi107.stdout:11047: done (0 left) 2020-06-12T02:39:14.484 INFO:tasks.rados.rados.0.smithi107.stdout:11048: delete oid 11 current snap is 0 2020-06-12T02:39:14.486 INFO:tasks.rados.rados.0.smithi107.stdout:11048: done (0 left) 2020-06-12T02:39:14.487 INFO:tasks.rados.rados.0.smithi107.stdout:11049: delete oid 15 current snap is 0 2020-06-12T02:39:14.490 INFO:tasks.rados.rados.0.smithi107.stdout:11049: done (0 left) 2020-06-12T02:39:14.490 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:39:14.490 INFO:tasks.rados.rados.0.smithi107.stdout:11050: seq_num 11049 ranges {106496=253952} 2020-06-12T02:39:14.492 INFO:tasks.rados.rados.0.smithi107.stdout:11050: writing smithi10729407-5 from 106496 to 360448 tid 1 2020-06-12T02:39:14.492 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:39:14.492 INFO:tasks.rados.rados.0.smithi107.stdout:11051: seq_num 11050 ranges {2277376=270336} 2020-06-12T02:39:14.497 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:39:14.506 INFO:tasks.rados.rados.0.smithi107.stdout:11051: writing smithi10729407-8 from 2277376 to 2547712 tid 1 2020-06-12T02:39:14.506 INFO:tasks.rados.rados.0.smithi107.stdout:11052: delete oid 13 current snap is 0 2020-06-12T02:39:14.506 INFO:tasks.rados.rados.0.smithi107.stdout:11050: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:14.507 INFO:tasks.rados.rados.0.smithi107.stdout:11050: finishing write tid 3 to smithi10729407-5 2020-06-12T02:39:14.507 INFO:tasks.rados.rados.0.smithi107.stdout:11050: finishing write tid 4 to smithi10729407-5 2020-06-12T02:39:14.507 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 1074 (ObjNum 11049 snap 0 seq_num 11049) dirty exists 2020-06-12T02:39:14.507 INFO:tasks.rados.rados.0.smithi107.stdout:11050: left oid 5 (ObjNum 11049 snap 0 seq_num 11049) 2020-06-12T02:39:14.508 INFO:tasks.rados.rados.0.smithi107.stdout:11050: done (2 left) 2020-06-12T02:39:14.508 INFO:tasks.rados.rados.0.smithi107.stdout:11052: done (1 left) 2020-06-12T02:39:14.509 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:39:14.509 INFO:tasks.rados.rados.0.smithi107.stdout:11053: seq_num 11052 ranges {0=303104} 2020-06-12T02:39:14.510 INFO:tasks.rados.rados.0.smithi107.stdout:11053: writing smithi10729407-15 from 0 to 303104 tid 1 2020-06-12T02:39:14.510 INFO:tasks.rados.rados.0.smithi107.stdout:11054: delete oid 11 current snap is 0 2020-06-12T02:39:14.514 INFO:tasks.rados.rados.0.smithi107.stdout:11054: done (2 left) 2020-06-12T02:39:14.514 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:39:14.514 INFO:tasks.rados.rados.0.smithi107.stdout:11055: seq_num 11054 ranges {0=335872} 2020-06-12T02:39:14.516 INFO:tasks.rados.rados.0.smithi107.stdout:11055: writing smithi10729407-4 from 0 to 335872 tid 1 2020-06-12T02:39:14.516 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:39:14.516 INFO:tasks.rados.rados.0.smithi107.stdout:11056: seq_num 11055 ranges {1802240=352256} 2020-06-12T02:39:14.520 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:39:14.528 INFO:tasks.rados.rados.0.smithi107.stdout:11056: writing smithi10729407-18 from 1802240 to 2154496 tid 1 2020-06-12T02:39:14.528 INFO:tasks.rados.rados.0.smithi107.stdout:11057: delete oid 6 current snap is 0 2020-06-12T02:39:14.529 INFO:tasks.rados.rados.0.smithi107.stdout:11051: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:14.529 INFO:tasks.rados.rados.0.smithi107.stdout:11051: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:14.529 INFO:tasks.rados.rados.0.smithi107.stdout:11053: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:14.529 INFO:tasks.rados.rados.0.smithi107.stdout:11051: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:14.529 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 2051 (ObjNum 11050 snap 0 seq_num 11050) dirty exists 2020-06-12T02:39:14.530 INFO:tasks.rados.rados.0.smithi107.stdout:11051: left oid 8 (ObjNum 11050 snap 0 seq_num 11050) 2020-06-12T02:39:14.530 INFO:tasks.rados.rados.0.smithi107.stdout:11053: finishing write tid 3 to smithi10729407-15 2020-06-12T02:39:14.530 INFO:tasks.rados.rados.0.smithi107.stdout:11053: finishing write tid 4 to smithi10729407-15 2020-06-12T02:39:14.531 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 2053 (ObjNum 11052 snap 0 seq_num 11052) dirty exists 2020-06-12T02:39:14.531 INFO:tasks.rados.rados.0.smithi107.stdout:11053: left oid 15 (ObjNum 11052 snap 0 seq_num 11052) 2020-06-12T02:39:14.531 INFO:tasks.rados.rados.0.smithi107.stdout:11055: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:14.531 INFO:tasks.rados.rados.0.smithi107.stdout:11055: finishing write tid 3 to smithi10729407-4 2020-06-12T02:39:14.531 INFO:tasks.rados.rados.0.smithi107.stdout:11055: finishing write tid 4 to smithi10729407-4 2020-06-12T02:39:14.532 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 2077 (ObjNum 11054 snap 0 seq_num 11054) dirty exists 2020-06-12T02:39:14.532 INFO:tasks.rados.rados.0.smithi107.stdout:11055: left oid 4 (ObjNum 11054 snap 0 seq_num 11054) 2020-06-12T02:39:14.532 INFO:tasks.rados.rados.0.smithi107.stdout:11051: done (4 left) 2020-06-12T02:39:14.533 INFO:tasks.rados.rados.0.smithi107.stdout:11053: done (3 left) 2020-06-12T02:39:14.533 INFO:tasks.rados.rados.0.smithi107.stdout:11055: done (2 left) 2020-06-12T02:39:14.533 INFO:tasks.rados.rados.0.smithi107.stdout:11057: done (1 left) 2020-06-12T02:39:14.533 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:39:14.533 INFO:tasks.rados.rados.0.smithi107.stdout:11058: seq_num 11057 ranges {0=385024} 2020-06-12T02:39:14.534 INFO:tasks.rados.rados.0.smithi107.stdout:11058: writing smithi10729407-11 from 0 to 385024 tid 1 2020-06-12T02:39:14.534 INFO:tasks.rados.rados.0.smithi107.stdout:11059: delete oid 2 current snap is 0 2020-06-12T02:39:14.535 INFO:tasks.rados.rados.0.smithi107.stdout:11056: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:14.537 INFO:tasks.rados.rados.0.smithi107.stdout:11059: done (2 left) 2020-06-12T02:39:14.538 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:39:14.538 INFO:tasks.rados.rados.0.smithi107.stdout:11060: seq_num 11059 ranges {0=425984} 2020-06-12T02:39:14.540 INFO:tasks.rados.rados.0.smithi107.stdout:11060: writing smithi10729407-3 from 0 to 425984 tid 1 2020-06-12T02:39:14.541 INFO:tasks.rados.rados.0.smithi107.stdout:11061: delete oid 4 current snap is 0 2020-06-12T02:39:14.541 INFO:tasks.rados.rados.0.smithi107.stdout:11056: finishing write tid 3 to smithi10729407-18 2020-06-12T02:39:14.542 INFO:tasks.rados.rados.0.smithi107.stdout:11058: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:14.545 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:39:14.543+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:39:14.546 INFO:tasks.rados.rados.0.smithi107.stdout:11058: finishing write tid 3 to smithi10729407-11 2020-06-12T02:39:14.546 INFO:tasks.rados.rados.0.smithi107.stdout:11056: finishing write tid 4 to smithi10729407-18 2020-06-12T02:39:14.547 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1882 (ObjNum 11055 snap 0 seq_num 11055) dirty exists 2020-06-12T02:39:14.547 INFO:tasks.rados.rados.0.smithi107.stdout:11056: left oid 18 (ObjNum 11055 snap 0 seq_num 11055) 2020-06-12T02:39:14.547 INFO:tasks.rados.rados.0.smithi107.stdout:11058: finishing write tid 4 to smithi10729407-11 2020-06-12T02:39:14.548 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1766 (ObjNum 11057 snap 0 seq_num 11057) dirty exists 2020-06-12T02:39:14.548 INFO:tasks.rados.rados.0.smithi107.stdout:11058: left oid 11 (ObjNum 11057 snap 0 seq_num 11057) 2020-06-12T02:39:14.548 INFO:tasks.rados.rados.0.smithi107.stdout:11056: done (3 left) 2020-06-12T02:39:14.549 INFO:tasks.rados.rados.0.smithi107.stdout:11058: done (2 left) 2020-06-12T02:39:14.549 INFO:tasks.rados.rados.0.smithi107.stdout:11061: done (1 left) 2020-06-12T02:39:14.549 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:39:14.549 INFO:tasks.rados.rados.0.smithi107.stdout:11062: seq_num 11061 ranges {0=458752} 2020-06-12T02:39:14.550 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:39:14.551 INFO:tasks.rados.rados.0.smithi107.stdout:11062: writing smithi10729407-12 from 0 to 458752 tid 1 2020-06-12T02:39:14.551 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:39:14.552 INFO:tasks.rados.rados.0.smithi107.stdout:11063: seq_num 11062 ranges {0=475136} 2020-06-12T02:39:14.566 INFO:tasks.rados.rados.0.smithi107.stdout:11063: writing smithi10729407-6 from 0 to 475136 tid 1 2020-06-12T02:39:14.567 INFO:tasks.rados.rados.0.smithi107.stdout:11064: delete oid 5 current snap is 0 2020-06-12T02:39:14.567 INFO:tasks.rados.rados.0.smithi107.stdout:11060: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:14.567 INFO:tasks.rados.rados.0.smithi107.stdout:11060: finishing write tid 3 to smithi10729407-3 2020-06-12T02:39:14.567 INFO:tasks.rados.rados.0.smithi107.stdout:11062: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:14.567 INFO:tasks.rados.rados.0.smithi107.stdout:11060: finishing write tid 4 to smithi10729407-3 2020-06-12T02:39:14.568 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1957 (ObjNum 11059 snap 0 seq_num 11059) dirty exists 2020-06-12T02:39:14.568 INFO:tasks.rados.rados.0.smithi107.stdout:11060: left oid 3 (ObjNum 11059 snap 0 seq_num 11059) 2020-06-12T02:39:14.568 INFO:tasks.rados.rados.0.smithi107.stdout:11063: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:14.569 INFO:tasks.rados.rados.0.smithi107.stdout:11060: done (3 left) 2020-06-12T02:39:14.569 INFO:tasks.rados.rados.0.smithi107.stdout:11064: done (2 left) 2020-06-12T02:39:14.569 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:39:14.570 INFO:tasks.rados.rados.0.smithi107.stdout:11065: seq_num 11064 ranges {0=507904} 2020-06-12T02:39:14.570 INFO:tasks.rados.rados.0.smithi107.stdout:11065: writing smithi10729407-17 from 0 to 507904 tid 1 2020-06-12T02:39:14.570 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:39:14.570 INFO:tasks.rados.rados.0.smithi107.stdout:11066: seq_num 11065 ranges {4014080=524288} 2020-06-12T02:39:14.580 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:39:14.593 INFO:tasks.rados.rados.0.smithi107.stdout:11066: writing smithi10729407-1 from 4014080 to 4538368 tid 1 2020-06-12T02:39:14.594 INFO:tasks.rados.rados.0.smithi107.stdout:11067: delete oid 8 current snap is 0 2020-06-12T02:39:14.594 INFO:tasks.rados.rados.0.smithi107.stdout:11062: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:14.595 INFO:tasks.rados.rados.0.smithi107.stdout:11063: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:14.595 INFO:tasks.rados.rados.0.smithi107.stdout:11062: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:14.595 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 2051 (ObjNum 11061 snap 0 seq_num 11061) dirty exists 2020-06-12T02:39:14.595 INFO:tasks.rados.rados.0.smithi107.stdout:11062: left oid 12 (ObjNum 11061 snap 0 seq_num 11061) 2020-06-12T02:39:14.595 INFO:tasks.rados.rados.0.smithi107.stdout:11063: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:14.596 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 1993 (ObjNum 11062 snap 0 seq_num 11062) dirty exists 2020-06-12T02:39:14.596 INFO:tasks.rados.rados.0.smithi107.stdout:11063: left oid 6 (ObjNum 11062 snap 0 seq_num 11062) 2020-06-12T02:39:14.596 INFO:tasks.rados.rados.0.smithi107.stdout:11065: finishing write tid 1 to smithi10729407-17 2020-06-12T02:39:14.596 INFO:tasks.rados.rados.0.smithi107.stdout:11065: finishing write tid 3 to smithi10729407-17 2020-06-12T02:39:14.596 INFO:tasks.rados.rados.0.smithi107.stdout:11065: finishing write tid 4 to smithi10729407-17 2020-06-12T02:39:14.597 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 2080 (ObjNum 11064 snap 0 seq_num 11064) dirty exists 2020-06-12T02:39:14.597 INFO:tasks.rados.rados.0.smithi107.stdout:11065: left oid 17 (ObjNum 11064 snap 0 seq_num 11064) 2020-06-12T02:39:14.597 INFO:tasks.rados.rados.0.smithi107.stdout:11062: done (4 left) 2020-06-12T02:39:14.598 INFO:tasks.rados.rados.0.smithi107.stdout:11063: done (3 left) 2020-06-12T02:39:14.598 INFO:tasks.rados.rados.0.smithi107.stdout:11065: done (2 left) 2020-06-12T02:39:14.598 INFO:tasks.rados.rados.0.smithi107.stdout:11067: done (1 left) 2020-06-12T02:39:14.598 INFO:tasks.rados.rados.0.smithi107.stdout:11068: delete oid 9 current snap is 0 2020-06-12T02:39:14.604 INFO:tasks.rados.rados.0.smithi107.stdout:11068: done (1 left) 2020-06-12T02:39:14.604 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:39:14.604 INFO:tasks.rados.rados.0.smithi107.stdout:11069: seq_num 11068 ranges {458752=573440} 2020-06-12T02:39:14.605 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:39:14.608 INFO:tasks.rados.rados.0.smithi107.stdout:11069: writing smithi10729407-12 from 458752 to 1032192 tid 1 2020-06-12T02:39:14.608 INFO:tasks.rados.rados.0.smithi107.stdout:11070: delete oid 20 current snap is 0 2020-06-12T02:39:14.608 INFO:tasks.rados.rados.0.smithi107.stdout:11066: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:14.609 INFO:tasks.rados.rados.0.smithi107.stdout:11066: finishing write tid 3 to smithi10729407-1 2020-06-12T02:39:14.609 INFO:tasks.rados.rados.0.smithi107.stdout:11066: finishing write tid 4 to smithi10729407-1 2020-06-12T02:39:14.609 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 1995 (ObjNum 11065 snap 0 seq_num 11065) dirty exists 2020-06-12T02:39:14.609 INFO:tasks.rados.rados.0.smithi107.stdout:11066: left oid 1 (ObjNum 11065 snap 0 seq_num 11065) 2020-06-12T02:39:14.610 INFO:tasks.rados.rados.0.smithi107.stdout:11066: done (2 left) 2020-06-12T02:39:14.610 INFO:tasks.rados.rados.0.smithi107.stdout:11070: done (1 left) 2020-06-12T02:39:14.610 INFO:tasks.rados.rados.0.smithi107.stdout:11071: delete oid 1 current snap is 0 2020-06-12T02:39:14.613 INFO:tasks.rados.rados.0.smithi107.stdout:11071: done (1 left) 2020-06-12T02:39:14.613 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:39:14.613 INFO:tasks.rados.rados.0.smithi107.stdout:11072: seq_num 11071 ranges {0=622592} 2020-06-12T02:39:14.614 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2020-06-12T02:39:14.615 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2020-06-12T02:39:14.615 INFO:tasks.thrashosds.thrasher:Killing osd 7, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2020-06-12T02:39:14.616 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2020-06-12T02:39:14.616 INFO:teuthology.orchestra.run:waiting for 300 2020-06-12T02:39:14.617 INFO:tasks.rados.rados.0.smithi107.stdout:11072: writing smithi10729407-20 from 0 to 622592 tid 1 2020-06-12T02:39:14.617 INFO:tasks.rados.rados.0.smithi107.stdout:11073: delete oid 7 current snap is 0 2020-06-12T02:39:14.617 INFO:tasks.rados.rados.0.smithi107.stdout:11069: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:14.624 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:39:14.621+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:39:14.647 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:39:14.647 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:39:14.648 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:39:14.648 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:39:14.648 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.648 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11604 3.3s0 3:ce45c70a:::smithi10729407-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e56)", 2020-06-12T02:39:14.649 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:22.436304+0000", 2020-06-12T02:39:14.649 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 52.207265157000002, 2020-06-12T02:39:14.649 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026739028000000001, 2020-06-12T02:39:14.649 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:14.649 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:14.650 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:14.650 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:14.650 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:14.650 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11604 2020-06-12T02:39:14.650 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.650 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:14.651 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.651 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:14.651 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.436304+0000", 2020-06-12T02:39:14.651 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:14.651 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.652 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.652 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:14.652 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.436304+0000", 2020-06-12T02:39:14.652 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0182e-05 2020-06-12T02:39:14.652 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.652 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.653 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:14.653 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.436335+0000", 2020-06-12T02:39:14.653 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2780000000000002e-06 2020-06-12T02:39:14.653 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.653 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.654 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:14.654 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.436338+0000", 2020-06-12T02:39:14.654 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5050000000000004e-06 2020-06-12T02:39:14.654 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.654 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.655 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:14.655 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.436344+0000", 2020-06-12T02:39:14.655 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6189999999999998e-06 2020-06-12T02:39:14.655 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.655 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.656 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:14.656 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.436347+0000", 2020-06-12T02:39:14.656 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1742000000000002e-05 2020-06-12T02:39:14.656 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.656 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.656 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:14.657 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.436379+0000", 2020-06-12T02:39:14.657 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5516000000000001e-05 2020-06-12T02:39:14.657 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.657 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.657 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:14.658 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.436444+0000", 2020-06-12T02:39:14.658 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012671909 2020-06-12T02:39:14.658 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.658 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.658 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:14.658 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.449116+0000", 2020-06-12T02:39:14.659 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5255999999999998e-05 2020-06-12T02:39:14.659 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.659 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.659 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:14.659 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.449201+0000", 2020-06-12T02:39:14.660 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0065621330000000004 2020-06-12T02:39:14.660 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.660 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.660 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:14.660 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.455764+0000", 2020-06-12T02:39:14.660 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000107519 2020-06-12T02:39:14.661 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.661 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.661 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:14.661 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.455871+0000", 2020-06-12T02:39:14.661 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0071723689999999996 2020-06-12T02:39:14.662 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.662 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.662 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:14.662 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.463043+0000", 2020-06-12T02:39:14.662 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026739028000000001 2020-06-12T02:39:14.662 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:14.663 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:14.663 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:14.663 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.663 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.663 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11605 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~712704] snapc 0=[] ondisk+write+known_if_redirected e56)", 2020-06-12T02:39:14.664 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:22.440426+0000", 2020-06-12T02:39:14.664 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 52.203143474000001, 2020-06-12T02:39:14.664 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022109515, 2020-06-12T02:39:14.664 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:14.664 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:14.665 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:14.665 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:14.665 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:14.665 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11605 2020-06-12T02:39:14.665 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.665 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:14.666 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.666 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:14.666 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.440426+0000", 2020-06-12T02:39:14.666 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:14.666 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.667 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.667 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:14.667 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.440426+0000", 2020-06-12T02:39:14.667 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00086067700000000001 2020-06-12T02:39:14.667 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.667 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.668 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:14.668 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.441287+0000", 2020-06-12T02:39:14.668 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1980000000000007e-06 2020-06-12T02:39:14.668 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.669 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.669 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:14.669 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.441296+0000", 2020-06-12T02:39:14.669 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6762999999999999e-05 2020-06-12T02:39:14.670 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.670 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.670 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:14.670 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.441313+0000", 2020-06-12T02:39:14.670 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2409999999999998e-06 2020-06-12T02:39:14.671 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.671 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.671 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:14.671 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.441320+0000", 2020-06-12T02:39:14.671 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0079076760000000006 2020-06-12T02:39:14.671 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.672 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.672 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:14.672 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.449228+0000", 2020-06-12T02:39:14.672 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2406999999999998e-05 2020-06-12T02:39:14.672 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.673 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.673 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:14.673 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.449320+0000", 2020-06-12T02:39:14.673 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00067351200000000003 2020-06-12T02:39:14.673 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.673 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.674 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:14.674 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.449994+0000", 2020-06-12T02:39:14.674 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0061543329999999997 2020-06-12T02:39:14.674 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.674 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.675 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:14.675 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.456148+0000", 2020-06-12T02:39:14.675 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0063118419999999998 2020-06-12T02:39:14.675 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.675 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.675 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:14.676 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.462460+0000", 2020-06-12T02:39:14.676 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5865999999999995e-05 2020-06-12T02:39:14.676 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.676 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.676 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:14.677 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.462536+0000", 2020-06-12T02:39:14.677 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022109515 2020-06-12T02:39:14.677 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:14.677 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:14.678 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:14.678 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.678 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.678 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11888 3.9s0 3:93ea0cad:::smithi10729407-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e56)", 2020-06-12T02:39:14.679 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:23.435701+0000", 2020-06-12T02:39:14.679 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 51.207868300000001, 2020-06-12T02:39:14.679 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020390334, 2020-06-12T02:39:14.679 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:14.679 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:14.679 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:14.680 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:14.680 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:14.680 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11888 2020-06-12T02:39:14.680 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.680 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:14.681 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.681 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:14.681 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.435701+0000", 2020-06-12T02:39:14.681 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:14.681 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.681 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.682 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:14.682 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.435701+0000", 2020-06-12T02:39:14.682 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1142000000000005e-05 2020-06-12T02:39:14.682 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.682 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.683 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:14.683 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.435762+0000", 2020-06-12T02:39:14.683 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1990000000000006e-06 2020-06-12T02:39:14.683 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.683 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.683 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:14.684 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.435771+0000", 2020-06-12T02:39:14.684 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0879000000000001e-05 2020-06-12T02:39:14.684 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.684 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.684 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:14.685 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.435792+0000", 2020-06-12T02:39:14.685 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9399999999999996e-06 2020-06-12T02:39:14.685 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.685 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.685 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:14.686 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.435798+0000", 2020-06-12T02:39:14.686 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010338498 2020-06-12T02:39:14.686 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.686 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.686 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:14.686 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.446137+0000", 2020-06-12T02:39:14.687 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8717999999999999e-05 2020-06-12T02:39:14.687 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.687 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.687 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:14.687 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.446236+0000", 2020-06-12T02:39:14.688 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0067264899999999999 2020-06-12T02:39:14.688 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.688 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.688 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:14.688 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.452962+0000", 2020-06-12T02:39:14.688 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000104836 2020-06-12T02:39:14.689 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.689 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.689 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:14.689 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.453067+0000", 2020-06-12T02:39:14.689 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0030246320000000002 2020-06-12T02:39:14.690 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.690 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.690 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:14.690 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.456092+0000", 2020-06-12T02:39:14.690 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020390334 2020-06-12T02:39:14.691 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:14.691 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:14.691 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:14.691 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.691 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.691 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:14918 3.9s0 3:93ea0cad:::smithi10729407-19:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e59)", 2020-06-12T02:39:14.692 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:31.902764+0000", 2020-06-12T02:39:14.692 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 42.740806081000002, 2020-06-12T02:39:14.692 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.027517911999999999, 2020-06-12T02:39:14.692 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:14.692 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:14.693 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:14.693 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:14.693 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:14.693 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 14918 2020-06-12T02:39:14.693 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.693 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:14.694 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.694 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:14.694 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:31.902764+0000", 2020-06-12T02:39:14.694 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:14.694 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.695 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.695 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:14.695 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:31.902764+0000", 2020-06-12T02:39:14.695 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8561e-05 2020-06-12T02:39:14.695 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.695 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.696 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:14.696 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:31.902792+0000", 2020-06-12T02:39:14.696 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8930000000000004e-06 2020-06-12T02:39:14.696 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.696 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.697 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:14.697 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:31.902797+0000", 2020-06-12T02:39:14.697 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2152e-05 2020-06-12T02:39:14.697 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.697 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.697 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:14.698 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:31.902809+0000", 2020-06-12T02:39:14.698 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4589999999999997e-06 2020-06-12T02:39:14.698 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.698 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.698 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:14.699 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:31.902815+0000", 2020-06-12T02:39:14.699 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6838999999999997e-05 2020-06-12T02:39:14.699 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.699 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.700 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:14.700 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:31.902851+0000", 2020-06-12T02:39:14.700 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7704000000000002e-05 2020-06-12T02:39:14.700 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.700 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.701 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:14.701 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:31.902909+0000", 2020-06-12T02:39:14.701 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.027372304 2020-06-12T02:39:14.701 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.701 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.702 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:14.702 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:31.930281+0000", 2020-06-12T02:39:14.702 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.027517911999999999 2020-06-12T02:39:14.702 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:14.702 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:14.703 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:14.703 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.703 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.703 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:15173 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~573440] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:39:14.703 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:32.691770+0000", 2020-06-12T02:39:14.704 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 41.951799813000001, 2020-06-12T02:39:14.704 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022201624999999999, 2020-06-12T02:39:14.704 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:14.704 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:14.704 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:14.704 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:14.705 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:14.706 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 15173 2020-06-12T02:39:14.706 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.706 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:14.707 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.707 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:14.707 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.691770+0000", 2020-06-12T02:39:14.708 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:14.708 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.708 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.709 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:14.709 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.691770+0000", 2020-06-12T02:39:14.709 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00055849100000000004 2020-06-12T02:39:14.709 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.710 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.710 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:14.710 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.692328+0000", 2020-06-12T02:39:14.711 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6500000000000002e-06 2020-06-12T02:39:14.711 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.711 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.712 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:14.712 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.692337+0000", 2020-06-12T02:39:14.712 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.467e-05 2020-06-12T02:39:14.712 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.713 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.713 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:14.713 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.692352+0000", 2020-06-12T02:39:14.713 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6599999999999998e-06 2020-06-12T02:39:14.714 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.714 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.714 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:14.715 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.692358+0000", 2020-06-12T02:39:14.715 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023564900000000001 2020-06-12T02:39:14.715 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.715 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.716 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:14.716 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.694715+0000", 2020-06-12T02:39:14.716 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8064000000000002e-05 2020-06-12T02:39:14.717 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.717 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.718 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:14.718 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.694813+0000", 2020-06-12T02:39:14.719 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0044087759999999997 2020-06-12T02:39:14.719 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.719 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.719 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:14.720 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.699222+0000", 2020-06-12T02:39:14.720 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9463999999999998e-05 2020-06-12T02:39:14.720 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.721 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.721 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:14.721 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.699291+0000", 2020-06-12T02:39:14.722 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00097042899999999997 2020-06-12T02:39:14.722 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.722 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.723 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:14.723 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.700261+0000", 2020-06-12T02:39:14.723 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0059618650000000002 2020-06-12T02:39:14.723 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.724 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.724 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:14.724 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.706223+0000", 2020-06-12T02:39:14.724 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.007674775 2020-06-12T02:39:14.725 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.725 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.725 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:14.725 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.713898+0000", 2020-06-12T02:39:14.726 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3291e-05 2020-06-12T02:39:14.726 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.726 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.727 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:14.727 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.713971+0000", 2020-06-12T02:39:14.727 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022201624999999999 2020-06-12T02:39:14.728 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:14.728 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:14.728 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:14.728 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.728 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.728 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:15174 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~229376] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:39:14.729 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:32.692967+0000", 2020-06-12T02:39:14.729 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 41.950602967999998, 2020-06-12T02:39:14.729 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021149365999999999, 2020-06-12T02:39:14.729 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:14.729 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:14.730 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:14.730 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:14.730 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:14.730 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 15174 2020-06-12T02:39:14.730 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.731 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:14.731 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.731 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:14.731 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.692967+0000", 2020-06-12T02:39:14.731 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:14.732 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.732 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.732 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:14.732 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.692967+0000", 2020-06-12T02:39:14.732 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00034681000000000002 2020-06-12T02:39:14.732 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.733 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.733 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:14.733 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.693313+0000", 2020-06-12T02:39:14.733 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2090000000000002e-06 2020-06-12T02:39:14.733 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.734 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.734 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:14.734 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.693320+0000", 2020-06-12T02:39:14.734 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.164e-05 2020-06-12T02:39:14.734 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.735 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.735 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:14.735 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.693331+0000", 2020-06-12T02:39:14.735 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.357e-06 2020-06-12T02:39:14.735 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.735 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.736 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:14.736 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.693336+0000", 2020-06-12T02:39:14.736 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0015410910000000001 2020-06-12T02:39:14.736 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.736 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.737 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:14.737 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.694877+0000", 2020-06-12T02:39:14.737 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7024000000000002e-05 2020-06-12T02:39:14.737 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.737 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.738 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:14.738 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.694934+0000", 2020-06-12T02:39:14.738 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0088149830000000002 2020-06-12T02:39:14.738 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.739 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.739 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:14.739 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.703749+0000", 2020-06-12T02:39:14.739 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014228600000000001 2020-06-12T02:39:14.739 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.739 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.740 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:14.740 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.703891+0000", 2020-06-12T02:39:14.740 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00069168500000000004 2020-06-12T02:39:14.740 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.740 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.741 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:14.741 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.704583+0000", 2020-06-12T02:39:14.741 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0034036829999999998 2020-06-12T02:39:14.741 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.741 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.742 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:14.742 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.707986+0000", 2020-06-12T02:39:14.742 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0060828699999999998 2020-06-12T02:39:14.742 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.742 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.742 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:14.743 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.714069+0000", 2020-06-12T02:39:14.743 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6727999999999997e-05 2020-06-12T02:39:14.743 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.743 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.743 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:14.744 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.714116+0000", 2020-06-12T02:39:14.744 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021149365999999999 2020-06-12T02:39:14.744 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:14.744 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:14.744 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:14.744 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.745 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.745 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:15175 3.3s0 3:c2ca77da:::smithi10729407-13:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:39:14.745 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:32.693372+0000", 2020-06-12T02:39:14.745 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 41.950197355, 2020-06-12T02:39:14.745 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.023275062999999999, 2020-06-12T02:39:14.746 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:14.746 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:14.746 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:14.747 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:14.747 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:14.747 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 15175 2020-06-12T02:39:14.747 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.747 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:14.748 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.748 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:14.748 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.693372+0000", 2020-06-12T02:39:14.748 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:14.748 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.749 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.749 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:14.749 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.693372+0000", 2020-06-12T02:39:14.749 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5139999999999998e-05 2020-06-12T02:39:14.749 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.749 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.750 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:14.750 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.693417+0000", 2020-06-12T02:39:14.750 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8190000000000002e-06 2020-06-12T02:39:14.750 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.750 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.751 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:14.751 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.693421+0000", 2020-06-12T02:39:14.751 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0336000000000001e-05 2020-06-12T02:39:14.751 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.751 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.751 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:14.752 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.693432+0000", 2020-06-12T02:39:14.752 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4650000000000001e-06 2020-06-12T02:39:14.752 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.752 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.753 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:14.753 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.693434+0000", 2020-06-12T02:39:14.753 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001515984 2020-06-12T02:39:14.753 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.753 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.754 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:14.754 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.694950+0000", 2020-06-12T02:39:14.754 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1379999999999998e-05 2020-06-12T02:39:14.754 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.754 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.754 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:14.755 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.695011+0000", 2020-06-12T02:39:14.755 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011265878999999999 2020-06-12T02:39:14.755 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.755 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.755 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:14.756 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.706277+0000", 2020-06-12T02:39:14.756 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2454999999999997e-05 2020-06-12T02:39:14.756 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.756 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.756 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:14.756 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.706340+0000", 2020-06-12T02:39:14.757 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00059463099999999998 2020-06-12T02:39:14.757 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.757 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.757 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:14.757 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.706934+0000", 2020-06-12T02:39:14.758 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0019727239999999999 2020-06-12T02:39:14.758 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.758 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.758 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:14.758 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.708907+0000", 2020-06-12T02:39:14.758 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0076545620000000002 2020-06-12T02:39:14.759 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.759 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.759 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:14.759 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.716562+0000", 2020-06-12T02:39:14.760 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5687999999999997e-05 2020-06-12T02:39:14.760 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.760 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.760 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:14.760 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.716647+0000", 2020-06-12T02:39:14.760 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.023275062999999999 2020-06-12T02:39:14.761 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:14.762 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:14.762 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:14.762 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.762 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.763 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:15176 3.3s0 3:c2ca77da:::smithi10729407-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e59)", 2020-06-12T02:39:14.763 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:32.693459+0000", 2020-06-12T02:39:14.763 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 41.950110273, 2020-06-12T02:39:14.763 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025713850999999999, 2020-06-12T02:39:14.763 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:14.763 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:14.764 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:14.764 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:14.764 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:14.764 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 15176 2020-06-12T02:39:14.765 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.765 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:14.765 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.765 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:14.766 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.693459+0000", 2020-06-12T02:39:14.766 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:14.766 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.768 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.768 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:14.768 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.693459+0000", 2020-06-12T02:39:14.768 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7639000000000001e-05 2020-06-12T02:39:14.768 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.769 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.769 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:14.769 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.693487+0000", 2020-06-12T02:39:14.769 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2799999999999999e-06 2020-06-12T02:39:14.770 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.770 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.770 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:14.770 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.693490+0000", 2020-06-12T02:39:14.770 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5519999999999999e-06 2020-06-12T02:39:14.771 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.771 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.771 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:14.771 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.693498+0000", 2020-06-12T02:39:14.771 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2759999999999999e-06 2020-06-12T02:39:14.771 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.772 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.772 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:14.772 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.693500+0000", 2020-06-12T02:39:14.772 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0015322269999999999 2020-06-12T02:39:14.772 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.773 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.773 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:14.773 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.695032+0000", 2020-06-12T02:39:14.773 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9116999999999996e-05 2020-06-12T02:39:14.773 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.773 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.774 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:14.774 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.695101+0000", 2020-06-12T02:39:14.774 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012542372 2020-06-12T02:39:14.774 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.774 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.775 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:14.775 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.707644+0000", 2020-06-12T02:39:14.775 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9744000000000001e-05 2020-06-12T02:39:14.775 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.775 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.775 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:14.776 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.707724+0000", 2020-06-12T02:39:14.776 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0097828480000000002 2020-06-12T02:39:14.776 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.776 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.776 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:14.777 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.717506+0000", 2020-06-12T02:39:14.777 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1787999999999996e-05 2020-06-12T02:39:14.777 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.777 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.777 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:14.778 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.717598+0000", 2020-06-12T02:39:14.778 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0015750079999999999 2020-06-12T02:39:14.778 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.778 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.778 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:14.778 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.719173+0000", 2020-06-12T02:39:14.779 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025713850999999999 2020-06-12T02:39:14.779 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:14.779 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:14.779 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:14.780 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.780 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.780 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:15226 3.3s0 3:ce45c70a:::smithi10729407-7:head [append 0~548864] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:39:14.781 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:32.827734+0000", 2020-06-12T02:39:14.781 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 41.815835399999997, 2020-06-12T02:39:14.781 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.083147429999999994, 2020-06-12T02:39:14.781 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:14.781 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:14.782 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:14.782 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:14.782 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:14.782 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 15226 2020-06-12T02:39:14.782 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.782 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:14.783 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.783 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:14.783 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.827734+0000", 2020-06-12T02:39:14.783 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:14.783 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.784 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.785 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:14.785 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.827734+0000", 2020-06-12T02:39:14.785 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000451643 2020-06-12T02:39:14.785 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.786 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.786 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:14.786 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.828186+0000", 2020-06-12T02:39:14.786 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2770000000000002e-06 2020-06-12T02:39:14.787 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.787 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.787 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:14.787 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.828192+0000", 2020-06-12T02:39:14.787 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1668000000000001e-05 2020-06-12T02:39:14.788 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.788 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.788 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:14.788 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.828204+0000", 2020-06-12T02:39:14.788 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5350000000000004e-06 2020-06-12T02:39:14.789 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.789 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.789 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:14.789 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.828209+0000", 2020-06-12T02:39:14.789 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7537000000000002e-05 2020-06-12T02:39:14.789 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.790 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.790 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:14.790 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.828257+0000", 2020-06-12T02:39:14.790 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000100552 2020-06-12T02:39:14.791 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.791 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.791 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:14.791 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.828357+0000", 2020-06-12T02:39:14.791 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00070831400000000004 2020-06-12T02:39:14.792 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.792 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.792 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:14.792 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.829066+0000", 2020-06-12T02:39:14.792 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00321163 2020-06-12T02:39:14.793 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.793 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.793 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:14.794 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.832277+0000", 2020-06-12T02:39:14.794 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.078530193999999998 2020-06-12T02:39:14.794 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.794 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.794 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:14.795 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.910808+0000", 2020-06-12T02:39:14.795 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4079999999999995e-05 2020-06-12T02:39:14.795 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.795 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.795 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:14.796 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.910882+0000", 2020-06-12T02:39:14.796 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.083147429999999994 2020-06-12T02:39:14.796 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:14.796 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:14.796 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:14.797 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.797 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.797 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:15227 3.3s0 3:ce45c70a:::smithi10729407-7:head [append 0~704512] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:39:14.797 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:32.831798+0000", 2020-06-12T02:39:14.798 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 41.811771669000002, 2020-06-12T02:39:14.798 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.079221454999999996, 2020-06-12T02:39:14.798 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:14.798 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:14.799 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:14.799 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:14.799 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:14.799 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 15227 2020-06-12T02:39:14.799 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.799 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:14.800 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.800 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:14.800 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.831798+0000", 2020-06-12T02:39:14.800 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:14.800 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.801 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.801 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:14.801 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.831798+0000", 2020-06-12T02:39:14.801 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00061004699999999995 2020-06-12T02:39:14.801 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.806 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.807 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:14.807 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.832408+0000", 2020-06-12T02:39:14.807 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0799999999999995e-06 2020-06-12T02:39:14.807 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.808 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.808 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:14.808 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.832417+0000", 2020-06-12T02:39:14.808 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3973e-05 2020-06-12T02:39:14.808 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.809 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.809 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:14.809 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.832431+0000", 2020-06-12T02:39:14.809 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3990000000000001e-06 2020-06-12T02:39:14.809 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.810 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.810 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:14.810 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.832437+0000", 2020-06-12T02:39:14.810 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0188000000000002e-05 2020-06-12T02:39:14.811 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.811 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.811 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:14.811 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.832478+0000", 2020-06-12T02:39:14.811 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000106051 2020-06-12T02:39:14.811 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.812 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.812 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:14.812 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.832584+0000", 2020-06-12T02:39:14.812 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00067036799999999996 2020-06-12T02:39:14.812 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.813 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.813 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:14.813 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.833254+0000", 2020-06-12T02:39:14.813 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0076593140000000004 2020-06-12T02:39:14.813 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.813 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.814 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:14.814 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.840913+0000", 2020-06-12T02:39:14.814 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.070062532999999996 2020-06-12T02:39:14.814 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.815 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.815 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:14.815 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.910976+0000", 2020-06-12T02:39:14.815 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3501999999999998e-05 2020-06-12T02:39:14.816 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.816 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.816 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:14.816 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.911019+0000", 2020-06-12T02:39:14.817 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.079221454999999996 2020-06-12T02:39:14.817 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:14.817 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:14.818 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:14.818 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.818 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.819 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:15228 3.3s0 3:ce45c70a:::smithi10729407-7:head [append 0~499712] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:39:14.819 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:32.834638+0000", 2020-06-12T02:39:14.819 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 41.808931913000002, 2020-06-12T02:39:14.819 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.07850733, 2020-06-12T02:39:14.820 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:14.820 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:14.820 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:14.821 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:14.821 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:14.821 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 15228 2020-06-12T02:39:14.821 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.822 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:14.822 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.822 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:14.823 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.834638+0000", 2020-06-12T02:39:14.823 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:14.823 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.824 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.824 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:14.824 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.834638+0000", 2020-06-12T02:39:14.825 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00041470899999999998 2020-06-12T02:39:14.825 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.825 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.826 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:14.826 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835052+0000", 2020-06-12T02:39:14.826 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1869999999999996e-06 2020-06-12T02:39:14.827 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.827 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.827 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:14.828 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835058+0000", 2020-06-12T02:39:14.828 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8429999999999999e-06 2020-06-12T02:39:14.828 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.829 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.829 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:14.829 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835066+0000", 2020-06-12T02:39:14.829 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1910000000000002e-06 2020-06-12T02:39:14.830 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.830 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.830 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:14.831 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835071+0000", 2020-06-12T02:39:14.831 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001686218 2020-06-12T02:39:14.831 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.832 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.832 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:14.832 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.836757+0000", 2020-06-12T02:39:14.832 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000126668 2020-06-12T02:39:14.833 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.833 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.833 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:14.833 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.836884+0000", 2020-06-12T02:39:14.834 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00067049399999999999 2020-06-12T02:39:14.834 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.834 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.835 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:14.835 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.837554+0000", 2020-06-12T02:39:14.835 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0034814799999999999 2020-06-12T02:39:14.836 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.836 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.836 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:14.837 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.841035+0000", 2020-06-12T02:39:14.837 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.072068473999999993 2020-06-12T02:39:14.837 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.838 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.838 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:14.838 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.913104+0000", 2020-06-12T02:39:14.839 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1066000000000003e-05 2020-06-12T02:39:14.839 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.840 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.840 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:14.840 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.913145+0000", 2020-06-12T02:39:14.841 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.07850733 2020-06-12T02:39:14.841 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:14.841 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:14.842 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:14.842 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.843 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.843 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:15229 3.3s0 3:ce45c70a:::smithi10729407-7:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:39:14.843 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:32.835098+0000", 2020-06-12T02:39:14.844 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 41.808471246000003, 2020-06-12T02:39:14.844 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.080709318000000002, 2020-06-12T02:39:14.844 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:14.844 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:14.845 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:14.845 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:14.845 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:14.846 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 15229 2020-06-12T02:39:14.846 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.846 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:14.847 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.847 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:14.847 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835098+0000", 2020-06-12T02:39:14.848 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:14.848 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.849 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.849 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:14.849 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835098+0000", 2020-06-12T02:39:14.850 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6427999999999997e-05 2020-06-12T02:39:14.850 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.851 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.851 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:14.852 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835135+0000", 2020-06-12T02:39:14.852 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2619999999999999e-06 2020-06-12T02:39:14.852 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.853 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.853 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:14.853 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835138+0000", 2020-06-12T02:39:14.854 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0209999999999996e-06 2020-06-12T02:39:14.854 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.854 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.854 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:14.854 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835144+0000", 2020-06-12T02:39:14.855 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.464e-06 2020-06-12T02:39:14.855 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.855 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.855 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:14.855 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835147+0000", 2020-06-12T02:39:14.856 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004835799 2020-06-12T02:39:14.856 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.856 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.856 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:14.856 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.839982+0000", 2020-06-12T02:39:14.857 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7294999999999998e-05 2020-06-12T02:39:14.857 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.857 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.857 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:14.857 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.840060+0000", 2020-06-12T02:39:14.857 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00028398300000000002 2020-06-12T02:39:14.858 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.858 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.858 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:14.858 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.840344+0000", 2020-06-12T02:39:14.858 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0017394960000000001 2020-06-12T02:39:14.859 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.859 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.859 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:14.859 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.842083+0000", 2020-06-12T02:39:14.859 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.073668073000000001 2020-06-12T02:39:14.860 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.860 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.860 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:14.860 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.915751+0000", 2020-06-12T02:39:14.860 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6496999999999998e-05 2020-06-12T02:39:14.861 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.861 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.861 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:14.861 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.915808+0000", 2020-06-12T02:39:14.861 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.080709318000000002 2020-06-12T02:39:14.862 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:14.862 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:14.862 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:14.862 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.862 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.863 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:15230 3.3s0 3:ce45c70a:::smithi10729407-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e59)", 2020-06-12T02:39:14.863 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:32.835168+0000", 2020-06-12T02:39:14.863 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 41.808401617000001, 2020-06-12T02:39:14.863 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.083254480000000006, 2020-06-12T02:39:14.863 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:14.864 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:14.864 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:14.864 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:14.864 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:14.865 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 15230 2020-06-12T02:39:14.865 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.865 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:14.865 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.866 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:14.866 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835168+0000", 2020-06-12T02:39:14.866 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:14.866 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.866 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.867 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:14.867 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835168+0000", 2020-06-12T02:39:14.867 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1753999999999999e-05 2020-06-12T02:39:14.868 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.868 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.868 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:14.868 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835190+0000", 2020-06-12T02:39:14.869 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9560000000000002e-06 2020-06-12T02:39:14.869 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.869 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.869 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:14.869 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835193+0000", 2020-06-12T02:39:14.870 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2999999999999998e-06 2020-06-12T02:39:14.870 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.870 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.870 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:14.870 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835199+0000", 2020-06-12T02:39:14.871 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3300000000000001e-06 2020-06-12T02:39:14.871 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.871 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.871 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:14.871 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835201+0000", 2020-06-12T02:39:14.872 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0057324740000000004 2020-06-12T02:39:14.872 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.872 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.872 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:14.872 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.840934+0000", 2020-06-12T02:39:14.872 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4407000000000006e-05 2020-06-12T02:39:14.873 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.873 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.873 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:14.873 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.841028+0000", 2020-06-12T02:39:14.873 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.074811723999999996 2020-06-12T02:39:14.874 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.874 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.874 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:14.874 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.915840+0000", 2020-06-12T02:39:14.874 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0995999999999996e-05 2020-06-12T02:39:14.874 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.875 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.875 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:14.875 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.915911+0000", 2020-06-12T02:39:14.875 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025115390000000001 2020-06-12T02:39:14.875 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.876 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.876 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:14.876 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.918422+0000", 2020-06-12T02:39:14.876 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.083254480000000006 2020-06-12T02:39:14.876 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:14.876 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:14.877 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:14.877 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.877 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.877 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:15231 3.3s0 3:c2ca77da:::smithi10729407-13:head [stat,delete] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:39:14.877 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:32.835222+0000", 2020-06-12T02:39:14.878 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 41.808347458999997, 2020-06-12T02:39:14.878 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.081008726000000003, 2020-06-12T02:39:14.878 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:14.878 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:14.878 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:14.878 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:14.879 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:14.879 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 15231 2020-06-12T02:39:14.879 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.879 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:14.879 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.880 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:14.880 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835222+0000", 2020-06-12T02:39:14.880 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:14.880 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.880 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.880 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:14.881 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835222+0000", 2020-06-12T02:39:14.881 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1980999999999998e-05 2020-06-12T02:39:14.881 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.881 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.881 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:14.882 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835244+0000", 2020-06-12T02:39:14.882 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9179999999999998e-06 2020-06-12T02:39:14.882 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.882 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.882 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:14.882 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835247+0000", 2020-06-12T02:39:14.883 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8449999999999999e-06 2020-06-12T02:39:14.883 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.883 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.883 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:14.883 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835253+0000", 2020-06-12T02:39:14.884 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2809999999999998e-06 2020-06-12T02:39:14.884 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.884 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.884 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:14.884 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835255+0000", 2020-06-12T02:39:14.884 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0058042370000000003 2020-06-12T02:39:14.885 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.885 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.885 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:14.885 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.841059+0000", 2020-06-12T02:39:14.885 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5568000000000004e-05 2020-06-12T02:39:14.886 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.886 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.886 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:14.886 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.841155+0000", 2020-06-12T02:39:14.886 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00045573899999999998 2020-06-12T02:39:14.886 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.887 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.887 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:14.887 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.841611+0000", 2020-06-12T02:39:14.887 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0022931090000000002 2020-06-12T02:39:14.887 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.888 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.888 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:14.888 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.843904+0000", 2020-06-12T02:39:14.888 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.072291870999999994 2020-06-12T02:39:14.888 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.888 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.889 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:14.889 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.916196+0000", 2020-06-12T02:39:14.889 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5176999999999999e-05 2020-06-12T02:39:14.889 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.889 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.890 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:14.890 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.916231+0000", 2020-06-12T02:39:14.890 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.081008726000000003 2020-06-12T02:39:14.890 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:14.890 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:14.890 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:14.891 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.891 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.891 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18318 3.3s0 3:ce45c70a:::smithi10729407-7:head [append 0~704512] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:14.891 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.082536+0000", 2020-06-12T02:39:14.891 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 32.561033362000003, 2020-06-12T02:39:14.892 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020384850999999999, 2020-06-12T02:39:14.892 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:14.892 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:14.892 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:14.892 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:14.893 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:14.893 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18318 2020-06-12T02:39:14.893 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.893 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:14.893 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.893 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:14.894 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.082536+0000", 2020-06-12T02:39:14.894 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:14.894 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.894 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.894 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:14.895 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.082536+0000", 2020-06-12T02:39:14.895 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00088830099999999998 2020-06-12T02:39:14.895 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.895 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.895 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:14.895 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.083425+0000", 2020-06-12T02:39:14.896 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0345e-05 2020-06-12T02:39:14.896 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.896 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.896 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:14.896 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.083435+0000", 2020-06-12T02:39:14.896 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6554000000000001e-05 2020-06-12T02:39:14.897 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.897 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.897 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:14.898 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.083451+0000", 2020-06-12T02:39:14.898 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5069999999999997e-06 2020-06-12T02:39:14.898 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.898 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.899 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:14.899 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.083458+0000", 2020-06-12T02:39:14.899 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9159e-05 2020-06-12T02:39:14.899 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.899 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.900 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:14.900 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.083487+0000", 2020-06-12T02:39:14.900 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7061000000000001e-05 2020-06-12T02:39:14.900 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.900 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.900 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:14.901 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.083544+0000", 2020-06-12T02:39:14.901 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00075509600000000002 2020-06-12T02:39:14.901 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.901 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.901 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:14.902 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.084299+0000", 2020-06-12T02:39:14.902 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0092259279999999996 2020-06-12T02:39:14.902 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.902 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.902 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:14.902 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.093525+0000", 2020-06-12T02:39:14.903 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0093184189999999997 2020-06-12T02:39:14.903 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.903 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.903 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:14.903 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.102844+0000", 2020-06-12T02:39:14.904 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7480999999999999e-05 2020-06-12T02:39:14.904 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.904 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.904 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:14.904 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.102921+0000", 2020-06-12T02:39:14.904 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020384850999999999 2020-06-12T02:39:14.905 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:14.905 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:14.905 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:14.905 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.905 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.906 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18323 3.3s0 3:ce45c70a:::smithi10729407-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:39:14.906 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.095481+0000", 2020-06-12T02:39:14.906 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 32.548088253000003, 2020-06-12T02:39:14.906 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021801823000000001, 2020-06-12T02:39:14.906 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:14.907 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:14.907 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:14.907 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:14.907 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:14.907 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18323 2020-06-12T02:39:14.907 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.908 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:14.908 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.908 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:14.908 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.095481+0000", 2020-06-12T02:39:14.908 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:14.909 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.909 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.909 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:14.909 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.095481+0000", 2020-06-12T02:39:14.909 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3367e-05 2020-06-12T02:39:14.909 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.910 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.910 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:14.910 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.095525+0000", 2020-06-12T02:39:14.910 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4779999999999996e-06 2020-06-12T02:39:14.910 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.911 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.912 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:14.912 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.095531+0000", 2020-06-12T02:39:14.912 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3572e-05 2020-06-12T02:39:14.913 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.913 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.913 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:14.913 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.095545+0000", 2020-06-12T02:39:14.913 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.391e-06 2020-06-12T02:39:14.914 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.914 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.914 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:14.914 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.095549+0000", 2020-06-12T02:39:14.915 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0087690979999999995 2020-06-12T02:39:14.915 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.915 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.915 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:14.916 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.104318+0000", 2020-06-12T02:39:14.916 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4034000000000001e-05 2020-06-12T02:39:14.916 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.916 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.917 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:14.917 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.104412+0000", 2020-06-12T02:39:14.917 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011331465000000001 2020-06-12T02:39:14.918 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.918 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.918 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:14.918 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.115744+0000", 2020-06-12T02:39:14.918 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9160000000000006e-05 2020-06-12T02:39:14.919 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.919 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.919 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:14.919 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.115843+0000", 2020-06-12T02:39:14.919 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001440258 2020-06-12T02:39:14.919 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.920 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.920 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:14.921 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.117283+0000", 2020-06-12T02:39:14.921 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021801823000000001 2020-06-12T02:39:14.921 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:14.921 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:14.921 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:14.922 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.922 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.922 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18324 3.9s0 3:977e4b9d:::smithi10729407-12:head [append 0~720896] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:14.922 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.097735+0000", 2020-06-12T02:39:14.922 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 32.545834904000003, 2020-06-12T02:39:14.923 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021826338000000001, 2020-06-12T02:39:14.923 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:14.923 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:14.923 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:14.924 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:14.924 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:14.925 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18324 2020-06-12T02:39:14.925 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.925 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:14.925 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.926 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:14.926 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.097735+0000", 2020-06-12T02:39:14.926 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:14.926 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.926 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.927 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:14.927 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.097735+0000", 2020-06-12T02:39:14.927 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0022114930000000001 2020-06-12T02:39:14.928 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.928 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.928 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:14.928 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.099946+0000", 2020-06-12T02:39:14.929 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2009999999999999e-06 2020-06-12T02:39:14.929 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.929 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.930 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:14.930 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.099954+0000", 2020-06-12T02:39:14.930 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3570999999999999e-05 2020-06-12T02:39:14.930 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.930 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.931 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:14.931 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.099968+0000", 2020-06-12T02:39:14.931 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2600000000000002e-06 2020-06-12T02:39:14.931 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.931 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.932 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:14.932 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.099974+0000", 2020-06-12T02:39:14.932 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1767999999999997e-05 2020-06-12T02:39:14.932 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.932 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.932 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:14.933 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.100016+0000", 2020-06-12T02:39:14.933 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5339000000000004e-05 2020-06-12T02:39:14.933 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.933 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.933 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:14.934 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.100101+0000", 2020-06-12T02:39:14.934 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000738665 2020-06-12T02:39:14.934 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.934 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.935 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:14.935 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.100840+0000", 2020-06-12T02:39:14.936 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018628308999999999 2020-06-12T02:39:14.936 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.936 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.936 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:14.936 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.119468+0000", 2020-06-12T02:39:14.937 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7867000000000001e-05 2020-06-12T02:39:14.937 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.937 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.937 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:14.937 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.119516+0000", 2020-06-12T02:39:14.938 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4864999999999998e-05 2020-06-12T02:39:14.938 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.938 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.938 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:14.938 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.119561+0000", 2020-06-12T02:39:14.939 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021826338000000001 2020-06-12T02:39:14.939 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:14.939 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:14.939 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:14.939 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.940 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.940 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:22367 3.9s0 3:93ea0cad:::smithi10729407-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e82)", 2020-06-12T02:39:14.940 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:39:11.326381+0000", 2020-06-12T02:39:14.940 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.3171884669999998, 2020-06-12T02:39:14.940 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020596637000000001, 2020-06-12T02:39:14.941 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:14.941 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:14.941 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:14.941 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:14.942 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:14.942 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 22367 2020-06-12T02:39:14.942 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.942 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:14.942 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.942 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:14.943 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.326381+0000", 2020-06-12T02:39:14.943 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:14.943 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.943 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.943 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:14.944 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.326381+0000", 2020-06-12T02:39:14.944 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0082000000000001e-05 2020-06-12T02:39:14.944 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.944 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.944 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:14.944 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.326411+0000", 2020-06-12T02:39:14.945 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.318e-06 2020-06-12T02:39:14.945 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.945 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.945 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:14.946 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.326416+0000", 2020-06-12T02:39:14.946 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9789999999999996e-06 2020-06-12T02:39:14.947 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.947 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.947 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:14.947 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.326423+0000", 2020-06-12T02:39:14.947 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.559e-06 2020-06-12T02:39:14.947 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.948 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.948 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:14.948 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.326425+0000", 2020-06-12T02:39:14.948 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00097856399999999991 2020-06-12T02:39:14.948 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.949 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.949 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:14.949 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.327404+0000", 2020-06-12T02:39:14.949 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8221999999999997e-05 2020-06-12T02:39:14.949 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.950 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.950 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:14.950 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.327492+0000", 2020-06-12T02:39:14.950 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010781778000000001 2020-06-12T02:39:14.950 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.951 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.951 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:14.951 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.338274+0000", 2020-06-12T02:39:14.951 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.393e-05 2020-06-12T02:39:14.952 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.952 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.952 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:14.952 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.338328+0000", 2020-06-12T02:39:14.952 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0046059100000000004 2020-06-12T02:39:14.952 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.953 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.953 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:14.953 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.342933+0000", 2020-06-12T02:39:14.953 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7420000000000003e-05 2020-06-12T02:39:14.953 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.954 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.954 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:14.954 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.342991+0000", 2020-06-12T02:39:14.954 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003986875 2020-06-12T02:39:14.954 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.955 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.955 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:14.955 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.346978+0000", 2020-06-12T02:39:14.955 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020596637000000001 2020-06-12T02:39:14.955 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:14.955 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:14.956 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:14.956 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.956 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.956 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:22473 3.9s0 3:93ea0cad:::smithi10729407-19:head [append 0~532480] snapc 0=[] ondisk+write+known_if_redirected e83)", 2020-06-12T02:39:14.957 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:39:11.624591+0000", 2020-06-12T02:39:14.957 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.018979023, 2020-06-12T02:39:14.958 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021165435999999999, 2020-06-12T02:39:14.958 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:14.958 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:14.958 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:14.958 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:14.959 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:14.959 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 22473 2020-06-12T02:39:14.959 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.959 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:14.959 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.959 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:14.960 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.624591+0000", 2020-06-12T02:39:14.960 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:14.960 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.960 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.960 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:14.961 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.624591+0000", 2020-06-12T02:39:14.961 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000582954 2020-06-12T02:39:14.961 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.962 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.962 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:14.962 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.625174+0000", 2020-06-12T02:39:14.962 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.0399999999999993e-06 2020-06-12T02:39:14.962 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.963 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.963 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:14.963 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.625182+0000", 2020-06-12T02:39:14.963 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3997e-05 2020-06-12T02:39:14.963 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.963 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.964 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:14.964 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.625196+0000", 2020-06-12T02:39:14.964 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1809999999999998e-06 2020-06-12T02:39:14.964 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.964 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.965 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:14.965 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.625202+0000", 2020-06-12T02:39:14.965 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00075445600000000003 2020-06-12T02:39:14.965 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.965 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.965 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:14.966 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.625956+0000", 2020-06-12T02:39:14.966 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000109559 2020-06-12T02:39:14.966 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.966 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.967 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:14.967 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.626066+0000", 2020-06-12T02:39:14.967 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0044162940000000003 2020-06-12T02:39:14.967 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.967 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.968 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:14.968 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.630482+0000", 2020-06-12T02:39:14.968 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2767000000000003e-05 2020-06-12T02:39:14.968 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.968 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.969 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:14.969 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.630555+0000", 2020-06-12T02:39:14.969 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00068772099999999997 2020-06-12T02:39:14.969 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.969 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.970 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:14.970 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.631243+0000", 2020-06-12T02:39:14.970 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0085930759999999998 2020-06-12T02:39:14.970 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.970 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.970 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:14.971 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.639836+0000", 2020-06-12T02:39:14.971 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0058491230000000003 2020-06-12T02:39:14.971 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.971 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.972 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:14.972 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.645685+0000", 2020-06-12T02:39:14.972 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1267999999999996e-05 2020-06-12T02:39:14.972 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.972 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.973 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:14.973 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.645756+0000", 2020-06-12T02:39:14.973 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021165435999999999 2020-06-12T02:39:14.973 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:14.973 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:14.974 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:14.974 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.974 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.975 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:22477 3.9s0 3:93ea0cad:::smithi10729407-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e83)", 2020-06-12T02:39:14.975 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:39:11.631206+0000", 2020-06-12T02:39:14.975 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 3.0123631909999999, 2020-06-12T02:39:14.976 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024571365000000001, 2020-06-12T02:39:14.976 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:14.976 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:14.977 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:14.977 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:14.977 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:14.977 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 22477 2020-06-12T02:39:14.977 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.978 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:14.978 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.978 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:14.978 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.631206+0000", 2020-06-12T02:39:14.978 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:14.979 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.979 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.979 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:14.979 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.631206+0000", 2020-06-12T02:39:14.979 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4255000000000001e-05 2020-06-12T02:39:14.979 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.980 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.980 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:14.980 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.631231+0000", 2020-06-12T02:39:14.980 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0460000000000001e-06 2020-06-12T02:39:14.980 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.981 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.981 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:14.981 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.631234+0000", 2020-06-12T02:39:14.981 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5699999999999998e-06 2020-06-12T02:39:14.982 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.982 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.982 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:14.982 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.631240+0000", 2020-06-12T02:39:14.982 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1739999999999999e-06 2020-06-12T02:39:14.983 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.983 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.983 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:14.983 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.631242+0000", 2020-06-12T02:39:14.983 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014604799 2020-06-12T02:39:14.984 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.984 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.984 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:14.984 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.645847+0000", 2020-06-12T02:39:14.984 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1564999999999996e-05 2020-06-12T02:39:14.984 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.985 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.985 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:14.985 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.645929+0000", 2020-06-12T02:39:14.985 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0061438930000000001 2020-06-12T02:39:14.985 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.986 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.986 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:14.986 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.652073+0000", 2020-06-12T02:39:14.986 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010995800000000001 2020-06-12T02:39:14.986 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.986 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.987 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:14.987 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.652183+0000", 2020-06-12T02:39:14.987 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0035951049999999999 2020-06-12T02:39:14.988 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:14.988 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:14.988 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:14.988 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.655778+0000", 2020-06-12T02:39:14.988 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024571365000000001 2020-06-12T02:39:14.989 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:14.989 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:14.989 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:14.989 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:14.990 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:14.990 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:39:14.994 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:39:14.994 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:39:14.995 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:39:14.989+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:39:14.995 INFO:tasks.ceph.osd.7:Stopped 2020-06-12T02:39:14.995 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.7 2020-06-12T02:39:14.996 INFO:tasks.thrashosds.thrasher:remote for osd 7 is ubuntu@smithi188.front.sepia.ceph.com 2020-06-12T02:39:14.996 INFO:teuthology.orchestra.run.smithi188:> type ceph-objectstore-tool 2020-06-12T02:39:14.997 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-06-12T02:39:15.049 INFO:teuthology.orchestra.run.smithi188.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2020-06-12T02:39:15.049 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-objectstore-tool --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op list-pgs 2020-06-12T02:39:15.098 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:39:15.095+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:39:15.159 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:39:15.160 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:39:15.160 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2020-06-12T02:39:15.160 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:39:15.171 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-06-12T02:39:15.198 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:39:15.196+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:39:15.300 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:39:15.297+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:39:15.306 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:39:15.306 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:39:15.307 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:39:15.307 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:39:15.307 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:39:15.316 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-06-12T02:39:15.401 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:39:15.399+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:39:15.448 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:39:15.449 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:39:15.449 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:39:15.449 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:39:15.449 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.449 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1206 3.5s0 3:a9510bda:::smithi10729407-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e25)", 2020-06-12T02:39:15.450 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:32.856688+0000", 2020-06-12T02:39:15.450 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 102.58400631400001, 2020-06-12T02:39:15.450 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019116942000000001, 2020-06-12T02:39:15.450 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:15.450 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:15.451 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:15.451 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:15.451 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:15.451 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1206 2020-06-12T02:39:15.451 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.451 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:15.452 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.452 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:15.452 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856688+0000", 2020-06-12T02:39:15.452 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:15.452 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.453 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.453 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:15.453 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856688+0000", 2020-06-12T02:39:15.453 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4355e-05 2020-06-12T02:39:15.453 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.454 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.454 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:15.454 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856713+0000", 2020-06-12T02:39:15.454 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0850000000000001e-06 2020-06-12T02:39:15.454 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.454 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.455 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:15.455 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856716+0000", 2020-06-12T02:39:15.455 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4509999999999997e-06 2020-06-12T02:39:15.455 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.455 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.456 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:15.456 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856722+0000", 2020-06-12T02:39:15.456 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2879999999999999e-06 2020-06-12T02:39:15.456 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.456 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.457 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:15.457 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.856724+0000", 2020-06-12T02:39:15.457 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0057507449999999998 2020-06-12T02:39:15.461 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.461 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.461 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:15.462 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.862475+0000", 2020-06-12T02:39:15.462 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5759000000000003e-05 2020-06-12T02:39:15.462 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.462 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.462 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:15.463 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.862571+0000", 2020-06-12T02:39:15.463 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012199018000000001 2020-06-12T02:39:15.463 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.463 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.463 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:15.464 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.874770+0000", 2020-06-12T02:39:15.464 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7260999999999999e-05 2020-06-12T02:39:15.464 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.464 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.464 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:15.464 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.874827+0000", 2020-06-12T02:39:15.465 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00097798000000000004 2020-06-12T02:39:15.465 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.465 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.465 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:15.465 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:32.875805+0000", 2020-06-12T02:39:15.466 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019116942000000001 2020-06-12T02:39:15.466 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.466 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:15.466 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.466 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.467 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.467 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1572 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~745472] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-06-12T02:39:15.467 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.886439+0000", 2020-06-12T02:39:15.478 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 101.554255596, 2020-06-12T02:39:15.478 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021442571000000001, 2020-06-12T02:39:15.479 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:15.479 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:15.479 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:15.479 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:15.479 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:15.480 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1572 2020-06-12T02:39:15.480 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.480 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:15.480 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.480 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:15.481 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.886439+0000", 2020-06-12T02:39:15.481 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:15.481 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.481 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.481 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:15.482 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.886439+0000", 2020-06-12T02:39:15.482 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00073027599999999997 2020-06-12T02:39:15.482 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.482 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.482 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:15.482 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887169+0000", 2020-06-12T02:39:15.483 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.49e-06 2020-06-12T02:39:15.483 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.483 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.483 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:15.484 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887175+0000", 2020-06-12T02:39:15.484 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7899999999999994e-06 2020-06-12T02:39:15.484 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.484 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.485 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:15.485 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887185+0000", 2020-06-12T02:39:15.485 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.42e-06 2020-06-12T02:39:15.486 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.486 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.486 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:15.486 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887189+0000", 2020-06-12T02:39:15.486 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1619999999999999e-05 2020-06-12T02:39:15.486 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.487 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.487 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:15.487 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887221+0000", 2020-06-12T02:39:15.487 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9720000000000002e-05 2020-06-12T02:39:15.487 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.488 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.488 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:15.488 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887300+0000", 2020-06-12T02:39:15.489 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047354100000000001 2020-06-12T02:39:15.489 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.489 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.489 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:15.489 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.887774+0000", 2020-06-12T02:39:15.490 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0030649750000000002 2020-06-12T02:39:15.490 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.490 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.490 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:15.491 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.890839+0000", 2020-06-12T02:39:15.491 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.016995321000000001 2020-06-12T02:39:15.491 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.491 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.491 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:15.492 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.907834+0000", 2020-06-12T02:39:15.492 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7417999999999999e-05 2020-06-12T02:39:15.492 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.492 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.492 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:15.492 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.907882+0000", 2020-06-12T02:39:15.493 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021442571000000001 2020-06-12T02:39:15.493 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.493 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:15.493 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.493 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.494 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.494 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1573 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~712704] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-06-12T02:39:15.494 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.890888+0000", 2020-06-12T02:39:15.495 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 101.54980686099999, 2020-06-12T02:39:15.495 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018868001999999998, 2020-06-12T02:39:15.495 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:15.496 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:15.496 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:15.496 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:15.497 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:15.497 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1573 2020-06-12T02:39:15.497 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.497 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:15.497 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.498 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:15.498 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.890888+0000", 2020-06-12T02:39:15.498 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:15.498 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.499 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.499 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:15.499 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.890888+0000", 2020-06-12T02:39:15.499 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000683605 2020-06-12T02:39:15.499 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.499 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.500 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:15.500 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891571+0000", 2020-06-12T02:39:15.500 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4999999999999999e-06 2020-06-12T02:39:15.500 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.500 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.501 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:15.501 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891577+0000", 2020-06-12T02:39:15.501 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.171e-05 2020-06-12T02:39:15.502 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.502 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.502 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:15.502 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891589+0000", 2020-06-12T02:39:15.503 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7650000000000001e-06 2020-06-12T02:39:15.503 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.503 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.503 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:15.504 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891593+0000", 2020-06-12T02:39:15.504 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6689000000000001e-05 2020-06-12T02:39:15.504 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.505 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.505 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:15.505 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891630+0000", 2020-06-12T02:39:15.505 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9877999999999997e-05 2020-06-12T02:39:15.505 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.505 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.506 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:15.506 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891710+0000", 2020-06-12T02:39:15.506 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053367200000000001 2020-06-12T02:39:15.506 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.506 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.507 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:15.507 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.892244+0000", 2020-06-12T02:39:15.507 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004927196 2020-06-12T02:39:15.507 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.507 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.508 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:15.508 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.897171+0000", 2020-06-12T02:39:15.508 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012496419 2020-06-12T02:39:15.508 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.509 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.509 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:15.509 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.909667+0000", 2020-06-12T02:39:15.509 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8567999999999996e-05 2020-06-12T02:39:15.509 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.510 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.510 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:15.510 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.909756+0000", 2020-06-12T02:39:15.510 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018868001999999998 2020-06-12T02:39:15.510 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.510 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:15.511 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.511 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.511 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.511 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:1576 3.5s0 3:a446366d:::smithi10729407-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e26)", 2020-06-12T02:39:15.511 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:33.891955+0000", 2020-06-12T02:39:15.512 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 101.548739621, 2020-06-12T02:39:15.512 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019893246999999999, 2020-06-12T02:39:15.512 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:15.512 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:15.512 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:15.512 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:15.513 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:15.513 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 1576 2020-06-12T02:39:15.513 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.513 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:15.513 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.514 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:15.514 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891955+0000", 2020-06-12T02:39:15.514 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:15.514 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.515 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.515 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:15.515 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891955+0000", 2020-06-12T02:39:15.515 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2067999999999999e-05 2020-06-12T02:39:15.515 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.516 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.516 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:15.516 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891977+0000", 2020-06-12T02:39:15.516 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.18e-06 2020-06-12T02:39:15.516 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.517 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.517 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:15.517 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891980+0000", 2020-06-12T02:39:15.517 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3629999999999999e-06 2020-06-12T02:39:15.517 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.517 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.518 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:15.518 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891987+0000", 2020-06-12T02:39:15.518 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.181e-06 2020-06-12T02:39:15.518 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.519 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.519 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:15.519 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.891989+0000", 2020-06-12T02:39:15.519 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050959120000000002 2020-06-12T02:39:15.519 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.520 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.520 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:15.520 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.897085+0000", 2020-06-12T02:39:15.523 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1089999999999998e-05 2020-06-12T02:39:15.523 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.523 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.524 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:15.524 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.897166+0000", 2020-06-12T02:39:15.524 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013400235 2020-06-12T02:39:15.524 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.524 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.525 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:15.525 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.910566+0000", 2020-06-12T02:39:15.525 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000104981 2020-06-12T02:39:15.525 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.525 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.526 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:15.526 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.910671+0000", 2020-06-12T02:39:15.526 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001177237 2020-06-12T02:39:15.526 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.526 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.526 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:15.527 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:33.911848+0000", 2020-06-12T02:39:15.527 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019893246999999999 2020-06-12T02:39:15.527 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.527 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:15.527 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.528 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.528 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.528 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2913 3.5s0 3:a9510bda:::smithi10729407-14:head [append 0~770048,stat] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:39:15.528 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.070354+0000", 2020-06-12T02:39:15.528 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 98.370340819999996, 2020-06-12T02:39:15.529 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60690798599999995, 2020-06-12T02:39:15.529 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:15.529 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:15.529 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:15.529 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:15.530 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:15.530 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2913 2020-06-12T02:39:15.530 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.530 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:15.531 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.531 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:15.531 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.070354+0000", 2020-06-12T02:39:15.531 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:15.531 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.532 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.532 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:15.532 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.070354+0000", 2020-06-12T02:39:15.532 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00098478900000000002 2020-06-12T02:39:15.532 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.532 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.533 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:15.533 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071339+0000", 2020-06-12T02:39:15.533 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8349999999999993e-06 2020-06-12T02:39:15.533 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.534 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.534 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:15.534 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071347+0000", 2020-06-12T02:39:15.534 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0749e-05 2020-06-12T02:39:15.534 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.535 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.535 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:15.535 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071358+0000", 2020-06-12T02:39:15.535 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9369999999999997e-06 2020-06-12T02:39:15.535 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.535 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.536 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:15.536 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071362+0000", 2020-06-12T02:39:15.536 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0426e-05 2020-06-12T02:39:15.536 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.536 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.537 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:15.537 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071423+0000", 2020-06-12T02:39:15.537 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4129999999999998e-05 2020-06-12T02:39:15.537 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.537 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.537 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:15.538 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.071487+0000", 2020-06-12T02:39:15.538 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00053752599999999998 2020-06-12T02:39:15.538 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.538 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.538 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:15.539 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072024+0000", 2020-06-12T02:39:15.539 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036162579999999998 2020-06-12T02:39:15.539 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.539 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.539 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:15.539 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075640+0000", 2020-06-12T02:39:15.540 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60152201999999999 2020-06-12T02:39:15.540 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.540 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.541 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:15.541 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677162+0000", 2020-06-12T02:39:15.541 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9315999999999996e-05 2020-06-12T02:39:15.541 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.541 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.542 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:15.542 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677262+0000", 2020-06-12T02:39:15.542 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60690798599999995 2020-06-12T02:39:15.542 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.542 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:15.542 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.543 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.543 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.543 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2914 3.5s0 3:a9510bda:::smithi10729407-14:head [append 0~401408] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:39:15.543 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.072409+0000", 2020-06-12T02:39:15.543 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 98.368285376000003, 2020-06-12T02:39:15.544 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.61956801100000003, 2020-06-12T02:39:15.544 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:15.544 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:15.544 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:15.544 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:15.544 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:15.545 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2914 2020-06-12T02:39:15.545 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.545 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:15.545 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.545 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:15.546 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072409+0000", 2020-06-12T02:39:15.546 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:15.546 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.546 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.546 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:15.547 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072409+0000", 2020-06-12T02:39:15.547 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000446317 2020-06-12T02:39:15.547 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.547 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.547 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:15.547 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072856+0000", 2020-06-12T02:39:15.548 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2140000000000004e-06 2020-06-12T02:39:15.548 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.548 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.548 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:15.548 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072861+0000", 2020-06-12T02:39:15.549 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8710000000000003e-06 2020-06-12T02:39:15.549 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.549 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.549 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:15.549 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072870+0000", 2020-06-12T02:39:15.549 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2270000000000004e-06 2020-06-12T02:39:15.550 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.550 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.550 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:15.550 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072874+0000", 2020-06-12T02:39:15.551 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0024649960000000001 2020-06-12T02:39:15.551 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.551 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.551 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:15.552 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075339+0000", 2020-06-12T02:39:15.552 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9179999999999996e-05 2020-06-12T02:39:15.552 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.552 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.553 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:15.553 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075438+0000", 2020-06-12T02:39:15.553 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60189281400000005 2020-06-12T02:39:15.553 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.554 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.554 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:15.554 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677331+0000", 2020-06-12T02:39:15.554 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012602800000000001 2020-06-12T02:39:15.554 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.555 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.555 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:15.555 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.677457+0000", 2020-06-12T02:39:15.555 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00088167699999999998 2020-06-12T02:39:15.555 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.555 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.556 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:15.556 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.678339+0000", 2020-06-12T02:39:15.556 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004312995 2020-06-12T02:39:15.556 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.556 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.557 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:15.557 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.682652+0000", 2020-06-12T02:39:15.557 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0092869140000000003 2020-06-12T02:39:15.557 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.557 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.557 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:15.558 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691938+0000", 2020-06-12T02:39:15.558 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8778e-05 2020-06-12T02:39:15.558 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.558 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.558 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:15.559 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691977+0000", 2020-06-12T02:39:15.559 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.61956801100000003 2020-06-12T02:39:15.559 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.559 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:15.559 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.559 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.560 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.560 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2915 3.5s0 3:a9510bda:::smithi10729407-14:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:39:15.560 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.072912+0000", 2020-06-12T02:39:15.560 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 98.367782636000001, 2020-06-12T02:39:15.560 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.6192204, 2020-06-12T02:39:15.562 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:15.562 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:15.562 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:15.562 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:15.563 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:15.563 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2915 2020-06-12T02:39:15.563 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.563 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:15.563 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.564 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:15.564 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072912+0000", 2020-06-12T02:39:15.564 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:15.564 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.564 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.565 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:15.565 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072912+0000", 2020-06-12T02:39:15.565 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9201e-05 2020-06-12T02:39:15.565 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.565 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.566 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:15.566 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072951+0000", 2020-06-12T02:39:15.566 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5480000000000002e-06 2020-06-12T02:39:15.566 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.566 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.567 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:15.567 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072955+0000", 2020-06-12T02:39:15.567 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3879999999999997e-06 2020-06-12T02:39:15.567 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.567 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.568 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:15.568 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072961+0000", 2020-06-12T02:39:15.568 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6589999999999999e-06 2020-06-12T02:39:15.568 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.568 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.569 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:15.570 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072964+0000", 2020-06-12T02:39:15.570 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025221039999999998 2020-06-12T02:39:15.570 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.570 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.571 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:15.571 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075486+0000", 2020-06-12T02:39:15.571 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5038999999999998e-05 2020-06-12T02:39:15.571 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.571 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.572 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:15.572 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075561+0000", 2020-06-12T02:39:15.572 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60452397499999999 2020-06-12T02:39:15.572 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.572 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.573 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:15.573 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680085+0000", 2020-06-12T02:39:15.573 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1398999999999999e-05 2020-06-12T02:39:15.573 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.573 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.574 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:15.574 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680156+0000", 2020-06-12T02:39:15.574 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000274465 2020-06-12T02:39:15.574 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.574 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.575 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:15.575 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680431+0000", 2020-06-12T02:39:15.575 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025071999999999998 2020-06-12T02:39:15.575 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.575 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.576 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:15.576 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.682938+0000", 2020-06-12T02:39:15.576 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0091116449999999998 2020-06-12T02:39:15.576 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.576 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.577 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:15.577 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692050+0000", 2020-06-12T02:39:15.577 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2776999999999997e-05 2020-06-12T02:39:15.577 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.577 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.577 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:15.578 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692132+0000", 2020-06-12T02:39:15.578 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.6192204 2020-06-12T02:39:15.578 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.578 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:15.578 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.579 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.579 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.579 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2916 3.5s0 3:a9510bda:::smithi10729407-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:39:15.579 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.072986+0000", 2020-06-12T02:39:15.579 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 98.367708397000001, 2020-06-12T02:39:15.579 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.62063941099999997, 2020-06-12T02:39:15.580 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:15.580 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:15.580 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:15.580 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:15.580 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:15.581 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2916 2020-06-12T02:39:15.581 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.581 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:15.581 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.581 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:15.581 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072986+0000", 2020-06-12T02:39:15.582 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:15.582 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.582 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.582 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:15.582 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.072986+0000", 2020-06-12T02:39:15.583 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2101000000000001e-05 2020-06-12T02:39:15.583 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.583 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.583 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:15.583 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073008+0000", 2020-06-12T02:39:15.584 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1549999999999999e-06 2020-06-12T02:39:15.584 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.584 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.584 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:15.584 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073011+0000", 2020-06-12T02:39:15.584 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2360000000000001e-06 2020-06-12T02:39:15.585 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.585 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.585 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:15.585 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073018+0000", 2020-06-12T02:39:15.585 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2790000000000001e-06 2020-06-12T02:39:15.586 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.586 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.586 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:15.586 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.073020+0000", 2020-06-12T02:39:15.586 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025637670000000002 2020-06-12T02:39:15.587 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.587 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.587 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:15.587 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075584+0000", 2020-06-12T02:39:15.587 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1814999999999998e-05 2020-06-12T02:39:15.587 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.588 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.588 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:15.588 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.075636+0000", 2020-06-12T02:39:15.588 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.60499943199999995 2020-06-12T02:39:15.588 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.589 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.589 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:15.589 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680635+0000", 2020-06-12T02:39:15.589 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0968000000000002e-05 2020-06-12T02:39:15.589 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.589 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.590 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:15.590 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.680706+0000", 2020-06-12T02:39:15.590 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011777101999999999 2020-06-12T02:39:15.590 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.590 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.591 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:15.591 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692483+0000", 2020-06-12T02:39:15.591 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5890999999999995e-05 2020-06-12T02:39:15.591 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.591 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.591 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:15.592 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.692569+0000", 2020-06-12T02:39:15.592 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001056665 2020-06-12T02:39:15.592 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.592 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.592 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:15.593 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.693626+0000", 2020-06-12T02:39:15.593 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.62063941099999997 2020-06-12T02:39:15.593 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.593 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:15.593 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.594 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.594 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.594 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2922 3.ds1 3:b983fd28:::smithi10729407-1:head [append 0~466944] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:39:15.594 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.097192+0000", 2020-06-12T02:39:15.594 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 98.343503029999994, 2020-06-12T02:39:15.595 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58672644699999998, 2020-06-12T02:39:15.595 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:15.595 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:15.595 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:15.595 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:15.596 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:15.596 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2922 2020-06-12T02:39:15.596 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.596 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:15.596 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.596 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:15.597 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097192+0000", 2020-06-12T02:39:15.597 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:15.597 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.597 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.597 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:15.598 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097192+0000", 2020-06-12T02:39:15.598 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047206100000000002 2020-06-12T02:39:15.598 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.598 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.598 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:15.599 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097664+0000", 2020-06-12T02:39:15.599 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8359999999999999e-06 2020-06-12T02:39:15.599 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.599 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.599 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:15.600 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097668+0000", 2020-06-12T02:39:15.600 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0815999999999999e-05 2020-06-12T02:39:15.600 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.600 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.600 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:15.600 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097679+0000", 2020-06-12T02:39:15.601 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.028e-06 2020-06-12T02:39:15.601 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.601 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.601 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:15.601 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097684+0000", 2020-06-12T02:39:15.602 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3932999999999999e-05 2020-06-12T02:39:15.602 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.602 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.602 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:15.602 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097748+0000", 2020-06-12T02:39:15.603 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2214999999999999e-05 2020-06-12T02:39:15.603 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.603 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.603 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:15.603 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.097820+0000", 2020-06-12T02:39:15.603 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00041583800000000003 2020-06-12T02:39:15.604 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.604 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.604 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:15.604 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.098236+0000", 2020-06-12T02:39:15.604 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0020339960000000002 2020-06-12T02:39:15.605 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.605 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.605 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:15.605 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.100270+0000", 2020-06-12T02:39:15.605 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58355657299999997 2020-06-12T02:39:15.606 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.606 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.606 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:15.606 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.683827+0000", 2020-06-12T02:39:15.606 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1150999999999995e-05 2020-06-12T02:39:15.606 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.607 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.607 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:15.607 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.683918+0000", 2020-06-12T02:39:15.607 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58672644699999998 2020-06-12T02:39:15.607 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.608 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:15.608 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.608 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.608 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.608 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2923 3.ds1 3:b983fd28:::smithi10729407-1:head [append 0~614400] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:39:15.608 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.100504+0000", 2020-06-12T02:39:15.609 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 98.340190389, 2020-06-12T02:39:15.609 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58853946999999995, 2020-06-12T02:39:15.609 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:15.609 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:15.609 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:15.610 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:15.610 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:15.610 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2923 2020-06-12T02:39:15.610 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.610 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:15.610 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.611 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:15.611 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.100504+0000", 2020-06-12T02:39:15.611 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:15.611 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.611 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.612 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:15.612 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.100504+0000", 2020-06-12T02:39:15.612 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00068395399999999996 2020-06-12T02:39:15.612 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.612 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.613 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:15.613 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101188+0000", 2020-06-12T02:39:15.613 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4050000000000003e-06 2020-06-12T02:39:15.613 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.614 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.614 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:15.614 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101196+0000", 2020-06-12T02:39:15.614 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.328e-05 2020-06-12T02:39:15.614 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.615 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.615 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:15.615 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101209+0000", 2020-06-12T02:39:15.615 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3419999999999996e-06 2020-06-12T02:39:15.615 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.615 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.616 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:15.616 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101215+0000", 2020-06-12T02:39:15.616 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1389000000000005e-05 2020-06-12T02:39:15.616 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.616 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.617 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:15.617 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101277+0000", 2020-06-12T02:39:15.617 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7724000000000004e-05 2020-06-12T02:39:15.617 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.617 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.617 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:15.618 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101344+0000", 2020-06-12T02:39:15.618 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047852499999999999 2020-06-12T02:39:15.618 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.618 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.618 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:15.619 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101823+0000", 2020-06-12T02:39:15.619 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0039255510000000002 2020-06-12T02:39:15.619 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.619 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.619 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:15.620 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105748+0000", 2020-06-12T02:39:15.620 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58321340099999996 2020-06-12T02:39:15.620 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.620 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.620 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:15.621 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.688962+0000", 2020-06-12T02:39:15.621 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1898999999999996e-05 2020-06-12T02:39:15.621 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.622 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.622 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:15.622 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689044+0000", 2020-06-12T02:39:15.622 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58853946999999995 2020-06-12T02:39:15.622 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.622 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:15.623 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.623 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.623 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.626 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2924 3.ds1 3:b983fd28:::smithi10729407-1:head [append 0~188416] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:39:15.626 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.101633+0000", 2020-06-12T02:39:15.626 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 98.339061395000002, 2020-06-12T02:39:15.627 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58758259199999996, 2020-06-12T02:39:15.627 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:15.627 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:15.627 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:15.627 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:15.627 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:15.628 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2924 2020-06-12T02:39:15.628 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.628 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:15.628 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.628 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:15.629 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101633+0000", 2020-06-12T02:39:15.629 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:15.629 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.629 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.629 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:15.630 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101633+0000", 2020-06-12T02:39:15.630 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00018775399999999999 2020-06-12T02:39:15.630 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.630 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.630 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:15.631 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101821+0000", 2020-06-12T02:39:15.631 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7669999999999998e-06 2020-06-12T02:39:15.631 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.631 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.631 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:15.631 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101828+0000", 2020-06-12T02:39:15.632 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2716e-05 2020-06-12T02:39:15.632 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.632 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.632 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:15.632 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101840+0000", 2020-06-12T02:39:15.633 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.215e-06 2020-06-12T02:39:15.633 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.633 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.633 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:15.633 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101846+0000", 2020-06-12T02:39:15.633 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021214250000000001 2020-06-12T02:39:15.634 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.634 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.634 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:15.634 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.103967+0000", 2020-06-12T02:39:15.634 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1018000000000003e-05 2020-06-12T02:39:15.635 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.635 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.635 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:15.635 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.104038+0000", 2020-06-12T02:39:15.635 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000333364 2020-06-12T02:39:15.635 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.636 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.636 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:15.636 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.104371+0000", 2020-06-12T02:39:15.636 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001378546 2020-06-12T02:39:15.636 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.637 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.637 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:15.637 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105750+0000", 2020-06-12T02:39:15.637 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58338272899999999 2020-06-12T02:39:15.637 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.638 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.638 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:15.638 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689133+0000", 2020-06-12T02:39:15.638 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.3058000000000003e-05 2020-06-12T02:39:15.638 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.638 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.639 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:15.639 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689216+0000", 2020-06-12T02:39:15.639 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58758259199999996 2020-06-12T02:39:15.639 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.639 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:15.640 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.640 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.640 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.640 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2925 3.ds1 3:b983fd28:::smithi10729407-1:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:39:15.640 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.101899+0000", 2020-06-12T02:39:15.641 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 98.338795730000001, 2020-06-12T02:39:15.641 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58747810899999997, 2020-06-12T02:39:15.641 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:15.641 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:15.641 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:15.642 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:15.642 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:15.642 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2925 2020-06-12T02:39:15.642 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.642 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:15.642 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.643 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:15.643 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101899+0000", 2020-06-12T02:39:15.643 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:15.643 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.643 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.644 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:15.644 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101899+0000", 2020-06-12T02:39:15.644 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2906999999999998e-05 2020-06-12T02:39:15.644 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.644 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.645 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:15.645 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101972+0000", 2020-06-12T02:39:15.645 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1789999999999996e-06 2020-06-12T02:39:15.645 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.645 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.646 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:15.646 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101978+0000", 2020-06-12T02:39:15.646 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1358e-05 2020-06-12T02:39:15.646 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.646 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.647 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:15.647 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101989+0000", 2020-06-12T02:39:15.647 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.031e-06 2020-06-12T02:39:15.647 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.647 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.647 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:15.648 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.101993+0000", 2020-06-12T02:39:15.648 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0033218990000000001 2020-06-12T02:39:15.648 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.648 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.649 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:15.649 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105315+0000", 2020-06-12T02:39:15.649 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6324999999999999e-05 2020-06-12T02:39:15.649 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.649 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.649 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:15.650 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105382+0000", 2020-06-12T02:39:15.650 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000216624 2020-06-12T02:39:15.650 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.650 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.650 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:15.651 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105598+0000", 2020-06-12T02:39:15.651 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001236586 2020-06-12T02:39:15.651 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.651 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.651 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:15.651 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.106835+0000", 2020-06-12T02:39:15.652 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58245509200000001 2020-06-12T02:39:15.652 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.652 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.652 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:15.652 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689290+0000", 2020-06-12T02:39:15.653 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7107999999999993e-05 2020-06-12T02:39:15.653 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.653 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.654 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:15.654 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689377+0000", 2020-06-12T02:39:15.654 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58747810899999997 2020-06-12T02:39:15.654 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.654 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:15.654 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.655 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.655 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.655 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2926 3.ds1 3:b983fd28:::smithi10729407-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:39:15.655 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.102036+0000", 2020-06-12T02:39:15.655 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 98.338658745999993, 2020-06-12T02:39:15.656 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58868048699999997, 2020-06-12T02:39:15.656 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:15.656 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:15.656 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:15.656 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:15.656 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:15.657 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2926 2020-06-12T02:39:15.657 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.657 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:15.657 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.657 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:15.658 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102036+0000", 2020-06-12T02:39:15.658 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:15.658 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.658 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.658 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:15.659 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102036+0000", 2020-06-12T02:39:15.659 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3786999999999999e-05 2020-06-12T02:39:15.659 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.660 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.660 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:15.660 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102080+0000", 2020-06-12T02:39:15.660 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7660000000000004e-06 2020-06-12T02:39:15.660 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.661 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.661 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:15.661 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102085+0000", 2020-06-12T02:39:15.661 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3145e-05 2020-06-12T02:39:15.661 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.662 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.662 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:15.662 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102099+0000", 2020-06-12T02:39:15.662 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.2440000000000004e-06 2020-06-12T02:39:15.662 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.663 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.663 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:15.663 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.102103+0000", 2020-06-12T02:39:15.663 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036826990000000002 2020-06-12T02:39:15.663 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.663 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.664 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:15.664 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105786+0000", 2020-06-12T02:39:15.664 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.6774000000000006e-05 2020-06-12T02:39:15.664 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.664 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.665 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:15.665 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.105882+0000", 2020-06-12T02:39:15.665 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58352084900000001 2020-06-12T02:39:15.665 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.665 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.666 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:15.666 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689403+0000", 2020-06-12T02:39:15.666 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3788999999999995e-05 2020-06-12T02:39:15.666 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.666 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.667 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:15.667 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.689467+0000", 2020-06-12T02:39:15.667 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001249434 2020-06-12T02:39:15.667 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.667 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.668 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:15.668 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690716+0000", 2020-06-12T02:39:15.668 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.58868048699999997 2020-06-12T02:39:15.668 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.668 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:15.669 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.669 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.669 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.669 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2950 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~598016] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:39:15.669 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.162562+0000", 2020-06-12T02:39:15.670 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 98.278132372000002, 2020-06-12T02:39:15.670 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52780611399999999, 2020-06-12T02:39:15.670 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:15.670 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:15.670 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:15.670 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:15.671 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:15.671 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2950 2020-06-12T02:39:15.671 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.671 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:15.671 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.671 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:15.672 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.162562+0000", 2020-06-12T02:39:15.672 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:15.672 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.672 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.672 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:15.673 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.162562+0000", 2020-06-12T02:39:15.673 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00070718099999999998 2020-06-12T02:39:15.673 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.673 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.673 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:15.674 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163269+0000", 2020-06-12T02:39:15.674 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2029999999999997e-06 2020-06-12T02:39:15.674 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.674 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.674 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:15.675 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163276+0000", 2020-06-12T02:39:15.675 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2734999999999999e-05 2020-06-12T02:39:15.675 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.675 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.675 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:15.675 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163288+0000", 2020-06-12T02:39:15.676 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.404e-06 2020-06-12T02:39:15.676 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.676 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.676 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:15.676 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163295+0000", 2020-06-12T02:39:15.677 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5784000000000006e-05 2020-06-12T02:39:15.677 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.677 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.677 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:15.677 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163361+0000", 2020-06-12T02:39:15.677 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5318e-05 2020-06-12T02:39:15.678 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.678 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.678 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:15.678 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163436+0000", 2020-06-12T02:39:15.678 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054801100000000005 2020-06-12T02:39:15.679 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.679 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.679 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:15.679 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.163984+0000", 2020-06-12T02:39:15.679 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.007023077 2020-06-12T02:39:15.679 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.680 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.680 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:15.680 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171007+0000", 2020-06-12T02:39:15.680 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51932051199999996 2020-06-12T02:39:15.680 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.680 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.681 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:15.681 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690327+0000", 2020-06-12T02:39:15.681 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0889000000000001e-05 2020-06-12T02:39:15.681 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.681 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.682 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:15.682 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690368+0000", 2020-06-12T02:39:15.682 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52780611399999999 2020-06-12T02:39:15.682 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.682 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:15.683 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.683 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.683 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.683 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2951 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~532480] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:39:15.683 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.165470+0000", 2020-06-12T02:39:15.683 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 98.275224764000001, 2020-06-12T02:39:15.684 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52507917900000001, 2020-06-12T02:39:15.684 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:15.684 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:15.684 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:15.684 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:15.684 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:15.685 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2951 2020-06-12T02:39:15.685 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.685 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:15.685 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.685 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:15.686 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.165470+0000", 2020-06-12T02:39:15.686 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:15.686 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.686 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.686 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:15.687 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.165470+0000", 2020-06-12T02:39:15.687 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00062231600000000004 2020-06-12T02:39:15.687 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.687 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.687 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:15.687 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166092+0000", 2020-06-12T02:39:15.688 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1610000000000002e-06 2020-06-12T02:39:15.688 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.688 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.688 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:15.688 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166097+0000", 2020-06-12T02:39:15.689 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7780000000000002e-06 2020-06-12T02:39:15.689 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.689 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.689 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:15.689 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166107+0000", 2020-06-12T02:39:15.689 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7169999999999999e-06 2020-06-12T02:39:15.690 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.690 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.690 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:15.690 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166112+0000", 2020-06-12T02:39:15.690 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4792000000000002e-05 2020-06-12T02:39:15.691 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.691 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.691 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:15.691 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166147+0000", 2020-06-12T02:39:15.691 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.9336e-05 2020-06-12T02:39:15.691 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.692 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.692 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:15.692 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166236+0000", 2020-06-12T02:39:15.692 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00067747200000000004 2020-06-12T02:39:15.692 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.692 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.693 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:15.693 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166913+0000", 2020-06-12T02:39:15.693 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0047718669999999999 2020-06-12T02:39:15.693 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.693 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.694 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:15.694 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171685+0000", 2020-06-12T02:39:15.694 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51881990700000002 2020-06-12T02:39:15.694 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.694 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.694 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:15.695 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690505+0000", 2020-06-12T02:39:15.695 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3832999999999999e-05 2020-06-12T02:39:15.695 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.695 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.695 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:15.696 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.690549+0000", 2020-06-12T02:39:15.696 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52507917900000001 2020-06-12T02:39:15.696 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.696 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:15.696 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.696 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.697 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.697 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2952 3.5s0 3:a446366d:::smithi10729407-11:head [append 0~270336] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:39:15.697 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.166891+0000", 2020-06-12T02:39:15.697 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 98.273803115000007, 2020-06-12T02:39:15.697 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52426417000000003, 2020-06-12T02:39:15.698 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:15.698 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:15.698 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:15.698 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:15.698 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:15.699 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2952 2020-06-12T02:39:15.699 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.699 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:15.699 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.699 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:15.699 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166891+0000", 2020-06-12T02:39:15.700 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:15.700 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.700 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.700 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:15.700 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.166891+0000", 2020-06-12T02:39:15.701 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00031878599999999999 2020-06-12T02:39:15.701 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.701 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.701 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:15.701 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167210+0000", 2020-06-12T02:39:15.701 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2360000000000004e-06 2020-06-12T02:39:15.702 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.702 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.702 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:15.702 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167219+0000", 2020-06-12T02:39:15.702 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3298e-05 2020-06-12T02:39:15.703 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.703 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.703 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:15.703 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167232+0000", 2020-06-12T02:39:15.703 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5849999999999999e-06 2020-06-12T02:39:15.703 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.704 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.704 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:15.704 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167237+0000", 2020-06-12T02:39:15.704 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0018755740000000001 2020-06-12T02:39:15.704 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.705 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.705 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:15.705 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.169113+0000", 2020-06-12T02:39:15.705 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000104768 2020-06-12T02:39:15.705 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.705 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.706 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:15.706 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.169218+0000", 2020-06-12T02:39:15.706 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00057389799999999999 2020-06-12T02:39:15.706 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.706 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.707 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:15.707 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.169792+0000", 2020-06-12T02:39:15.707 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0019333989999999999 2020-06-12T02:39:15.707 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.707 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.707 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:15.708 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171725+0000", 2020-06-12T02:39:15.708 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51938508699999997 2020-06-12T02:39:15.708 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.708 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.708 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:15.709 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691110+0000", 2020-06-12T02:39:15.709 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5538999999999999e-05 2020-06-12T02:39:15.709 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.709 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.709 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:15.709 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691156+0000", 2020-06-12T02:39:15.710 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52426417000000003 2020-06-12T02:39:15.710 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.710 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:15.710 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.710 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.711 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.711 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2953 3.5s0 3:a446366d:::smithi10729407-11:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:39:15.711 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.167293+0000", 2020-06-12T02:39:15.711 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 98.273401668000005, 2020-06-12T02:39:15.711 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52428350999999995, 2020-06-12T02:39:15.712 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:15.712 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:15.712 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:15.712 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:15.712 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:15.712 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2953 2020-06-12T02:39:15.713 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.713 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:15.713 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.713 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:15.713 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167293+0000", 2020-06-12T02:39:15.714 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:15.714 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.714 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.714 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:15.714 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167293+0000", 2020-06-12T02:39:15.714 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3880000000000004e-05 2020-06-12T02:39:15.715 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.715 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.715 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:15.715 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167367+0000", 2020-06-12T02:39:15.715 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.319e-06 2020-06-12T02:39:15.716 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.716 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.716 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:15.716 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167373+0000", 2020-06-12T02:39:15.716 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1719000000000001e-05 2020-06-12T02:39:15.716 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.717 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.717 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:15.717 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167385+0000", 2020-06-12T02:39:15.717 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4580000000000003e-06 2020-06-12T02:39:15.717 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.718 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.718 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:15.718 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167389+0000", 2020-06-12T02:39:15.718 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036341110000000002 2020-06-12T02:39:15.718 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.718 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.719 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:15.719 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171023+0000", 2020-06-12T02:39:15.719 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9418000000000005e-05 2020-06-12T02:39:15.719 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.719 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.720 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:15.720 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171093+0000", 2020-06-12T02:39:15.720 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00029087900000000002 2020-06-12T02:39:15.720 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.720 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.720 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:15.721 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171384+0000", 2020-06-12T02:39:15.721 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000489347 2020-06-12T02:39:15.721 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.721 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.721 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:15.722 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171873+0000", 2020-06-12T02:39:15.722 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51963691300000003 2020-06-12T02:39:15.722 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.722 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.722 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:15.722 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691510+0000", 2020-06-12T02:39:15.723 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6465999999999997e-05 2020-06-12T02:39:15.723 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.723 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.723 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:15.724 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691576+0000", 2020-06-12T02:39:15.724 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52428350999999995 2020-06-12T02:39:15.724 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.724 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:15.724 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.724 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.725 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.725 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2954 3.5s0 3:a446366d:::smithi10729407-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:39:15.725 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.167432+0000", 2020-06-12T02:39:15.725 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 98.273262919000004, 2020-06-12T02:39:15.725 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52568652699999996, 2020-06-12T02:39:15.726 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:15.726 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:15.726 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:15.726 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:15.726 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:15.726 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2954 2020-06-12T02:39:15.727 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.727 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:15.727 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.727 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:15.727 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167432+0000", 2020-06-12T02:39:15.728 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:15.728 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.728 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.728 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:15.728 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167432+0000", 2020-06-12T02:39:15.728 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3556000000000001e-05 2020-06-12T02:39:15.729 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.729 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.729 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:15.729 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167475+0000", 2020-06-12T02:39:15.729 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7350000000000001e-06 2020-06-12T02:39:15.730 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.730 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.730 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:15.730 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167481+0000", 2020-06-12T02:39:15.730 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3128e-05 2020-06-12T02:39:15.730 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.731 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.731 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:15.731 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167494+0000", 2020-06-12T02:39:15.731 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1579999999999998e-06 2020-06-12T02:39:15.731 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.732 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.732 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:15.732 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.167498+0000", 2020-06-12T02:39:15.732 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004082174 2020-06-12T02:39:15.732 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.732 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.733 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:15.733 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171580+0000", 2020-06-12T02:39:15.733 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8954000000000002e-05 2020-06-12T02:39:15.733 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.733 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.734 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:15.734 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.171679+0000", 2020-06-12T02:39:15.734 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51993235100000001 2020-06-12T02:39:15.734 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.734 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.735 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:15.735 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691612+0000", 2020-06-12T02:39:15.735 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4329000000000001e-05 2020-06-12T02:39:15.735 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.736 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.736 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:15.736 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691666+0000", 2020-06-12T02:39:15.737 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001452142 2020-06-12T02:39:15.737 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.737 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.737 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:15.737 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.693118+0000", 2020-06-12T02:39:15.738 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.52568652699999996 2020-06-12T02:39:15.738 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.738 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:15.738 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.738 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.739 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.739 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:2959 3.ds1 3:b3445631:::smithi10729407-6:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e27)", 2020-06-12T02:39:15.739 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.176248+0000", 2020-06-12T02:39:15.739 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 98.264446178, 2020-06-12T02:39:15.739 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51549877099999997, 2020-06-12T02:39:15.739 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:15.740 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:15.740 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:15.740 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:15.740 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:15.740 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 2959 2020-06-12T02:39:15.741 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.741 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:15.741 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.741 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:15.741 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176248+0000", 2020-06-12T02:39:15.741 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:15.742 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.742 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.742 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:15.742 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176248+0000", 2020-06-12T02:39:15.743 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.5120999999999999e-05 2020-06-12T02:39:15.743 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.743 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.743 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:15.743 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176274+0000", 2020-06-12T02:39:15.744 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9189999999999998e-06 2020-06-12T02:39:15.744 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.744 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.744 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:15.744 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176278+0000", 2020-06-12T02:39:15.744 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6339e-05 2020-06-12T02:39:15.745 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.745 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.745 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:15.745 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176305+0000", 2020-06-12T02:39:15.745 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9089999999999999e-06 2020-06-12T02:39:15.746 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.746 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.746 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:15.746 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176310+0000", 2020-06-12T02:39:15.746 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6095999999999999e-05 2020-06-12T02:39:15.747 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.747 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.747 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:15.747 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176376+0000", 2020-06-12T02:39:15.747 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8590999999999994e-05 2020-06-12T02:39:15.748 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.748 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.748 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:15.748 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176444+0000", 2020-06-12T02:39:15.748 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00024346400000000001 2020-06-12T02:39:15.748 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.749 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.749 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:15.749 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.176688+0000", 2020-06-12T02:39:15.749 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00044220899999999999 2020-06-12T02:39:15.749 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.750 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.750 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:15.750 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.177130+0000", 2020-06-12T02:39:15.750 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51452984400000001 2020-06-12T02:39:15.750 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.750 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.751 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:15.751 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691660+0000", 2020-06-12T02:39:15.751 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7279000000000002e-05 2020-06-12T02:39:15.751 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.751 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.752 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:15.752 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.691747+0000", 2020-06-12T02:39:15.752 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.51549877099999997 2020-06-12T02:39:15.752 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.752 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:15.753 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.753 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.753 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.753 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:3032 3.ds1 3:b983fd28:::smithi10729407-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:39:15.753 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:37.864289+0000", 2020-06-12T02:39:15.754 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 97.576405260000001, 2020-06-12T02:39:15.754 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019572216999999999, 2020-06-12T02:39:15.754 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:15.754 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:15.755 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:15.755 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:15.755 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:15.755 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 3032 2020-06-12T02:39:15.755 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.756 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:15.756 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.756 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:15.756 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864289+0000", 2020-06-12T02:39:15.756 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:15.756 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.757 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.757 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:15.757 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864289+0000", 2020-06-12T02:39:15.757 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5701999999999997e-05 2020-06-12T02:39:15.757 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.758 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.758 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:15.758 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864335+0000", 2020-06-12T02:39:15.758 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0910000000000004e-06 2020-06-12T02:39:15.758 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.759 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.759 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:15.759 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864342+0000", 2020-06-12T02:39:15.759 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2301e-05 2020-06-12T02:39:15.759 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.760 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.760 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:15.760 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864354+0000", 2020-06-12T02:39:15.760 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5199999999999997e-06 2020-06-12T02:39:15.761 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.761 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.761 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:15.761 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864360+0000", 2020-06-12T02:39:15.761 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00049180999999999997 2020-06-12T02:39:15.762 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.762 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.762 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:15.762 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864852+0000", 2020-06-12T02:39:15.762 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6725000000000003e-05 2020-06-12T02:39:15.762 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.763 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.763 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:15.763 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.864908+0000", 2020-06-12T02:39:15.763 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0089687010000000008 2020-06-12T02:39:15.763 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.764 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.764 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:15.764 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.873877+0000", 2020-06-12T02:39:15.764 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3382999999999999e-05 2020-06-12T02:39:15.764 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.764 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.765 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:15.765 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.873941+0000", 2020-06-12T02:39:15.765 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.006408927 2020-06-12T02:39:15.765 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.765 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.766 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:15.766 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.880350+0000", 2020-06-12T02:39:15.766 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3905e-05 2020-06-12T02:39:15.766 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.766 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.766 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:15.767 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.880403+0000", 2020-06-12T02:39:15.767 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0034581519999999999 2020-06-12T02:39:15.767 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.767 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.767 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:15.768 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:37.883862+0000", 2020-06-12T02:39:15.768 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019572216999999999 2020-06-12T02:39:15.768 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.768 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:15.768 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.768 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.769 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:15.769 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:39:15.769 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2020-06-12T02:39:15.771 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:39:15.768+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:39:15.771 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-06-12T02:39:15.872 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:39:15.869+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:39:15.933 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:39:15.934 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:39:15.934 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.934 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:23519 3.13s1 3.50e3a273 (undecoded) ondisk+retry+write+known_if_redirected e85)", 2020-06-12T02:39:15.934 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:39:15.427313+0000", 2020-06-12T02:39:15.934 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.50430854300000005, 2020-06-12T02:39:15.935 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.50433441899999998, 2020-06-12T02:39:15.935 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:15.935 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "delayed", 2020-06-12T02:39:15.935 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:15.935 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:15.936 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:15.936 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 23519 2020-06-12T02:39:15.936 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.936 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:15.936 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.937 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:15.937 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:15.427313+0000", 2020-06-12T02:39:15.937 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:15.937 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.937 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.937 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:15.938 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:15.427313+0000", 2020-06-12T02:39:15.938 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0633999999999997e-05 2020-06-12T02:39:15.938 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.938 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.938 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:15.939 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:15.427364+0000", 2020-06-12T02:39:15.939 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.182e-05 2020-06-12T02:39:15.939 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.939 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.939 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:15.939 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:15.427376+0000", 2020-06-12T02:39:15.940 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0910000000000001e-05 2020-06-12T02:39:15.940 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.940 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.940 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:15.940 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:15.427397+0000", 2020-06-12T02:39:15.941 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1287999999999999e-05 2020-06-12T02:39:15.941 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.941 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.941 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:15.941 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:15.427408+0000", 2020-06-12T02:39:15.942 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3909999999999997e-05 2020-06-12T02:39:15.942 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.942 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.943 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:15.943 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:15.427462+0000", 2020-06-12T02:39:15.943 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.956e-06 2020-06-12T02:39:15.943 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.943 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.944 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "op must wait for map", 2020-06-12T02:39:15.944 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:15.427469+0000", 2020-06-12T02:39:15.944 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0092460760000000006 2020-06-12T02:39:15.945 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.945 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.945 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:15.945 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:15.436715+0000", 2020-06-12T02:39:15.945 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2189999999999998e-06 2020-06-12T02:39:15.945 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:15.946 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:15.946 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2020-06-12T02:39:15.946 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:15.436717+0000", 2020-06-12T02:39:15.946 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0094038130000000004 2020-06-12T02:39:15.946 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.947 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:15.947 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.947 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:15.947 INFO:teuthology.orchestra.run.smithi188.stdout: ], 2020-06-12T02:39:15.947 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 1 2020-06-12T02:39:15.947 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:39:15.949 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-06-12T02:39:15.992 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:39:15.990+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:39:16.074 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:39:16.071+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:39:16.084 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:39:16.085 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:39:16.085 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:39:16.085 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:39:16.085 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:39:16.092 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-06-12T02:39:16.186 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:39:16.178+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:39:16.258 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:39:16.258 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:39:16.258 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:39:16.259 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:39:16.259 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.259 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:4975 3.as0 3:5732055a:::smithi10729407-8:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e30)", 2020-06-12T02:39:16.260 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:37:43.721794+0000", 2020-06-12T02:39:16.261 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 92.532184572999995, 2020-06-12T02:39:16.261 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.1825135, 2020-06-12T02:39:16.261 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:16.262 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:16.262 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:16.262 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:16.263 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:16.264 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 4975 2020-06-12T02:39:16.264 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.264 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:16.265 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.265 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:16.265 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721794+0000", 2020-06-12T02:39:16.266 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:16.266 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.266 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.267 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:16.267 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721794+0000", 2020-06-12T02:39:16.267 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4542e-05 2020-06-12T02:39:16.268 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.268 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.281 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:16.281 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721818+0000", 2020-06-12T02:39:16.281 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9169999999999997e-06 2020-06-12T02:39:16.281 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.282 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.282 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:16.282 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721823+0000", 2020-06-12T02:39:16.282 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1184e-05 2020-06-12T02:39:16.283 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.283 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.283 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:16.283 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721834+0000", 2020-06-12T02:39:16.283 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.254e-06 2020-06-12T02:39:16.284 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.284 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.284 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:16.284 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721840+0000", 2020-06-12T02:39:16.284 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1075999999999998e-05 2020-06-12T02:39:16.285 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.285 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.285 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:16.285 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721881+0000", 2020-06-12T02:39:16.285 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7416999999999995e-05 2020-06-12T02:39:16.286 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.286 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.286 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:16.286 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.721948+0000", 2020-06-12T02:39:16.286 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.18235910999999999 2020-06-12T02:39:16.287 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.287 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.287 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:16.287 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:37:43.904307+0000", 2020-06-12T02:39:16.287 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.1825135 2020-06-12T02:39:16.288 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:16.288 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:16.288 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:16.289 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.289 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.289 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11914 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e56)", 2020-06-12T02:39:16.289 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:23.508790+0000", 2020-06-12T02:39:16.289 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 52.745187721000001, 2020-06-12T02:39:16.290 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026541928999999999, 2020-06-12T02:39:16.290 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:16.290 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:16.290 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:16.290 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:16.291 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:16.291 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11914 2020-06-12T02:39:16.291 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.291 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:16.291 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.292 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:16.292 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508790+0000", 2020-06-12T02:39:16.292 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:16.292 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.292 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.293 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:16.293 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508790+0000", 2020-06-12T02:39:16.293 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0872999999999999e-05 2020-06-12T02:39:16.293 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.293 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.294 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:16.294 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508841+0000", 2020-06-12T02:39:16.294 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9179999999999994e-06 2020-06-12T02:39:16.294 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.294 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.295 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:16.295 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508849+0000", 2020-06-12T02:39:16.295 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.5551999999999999e-05 2020-06-12T02:39:16.295 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.295 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.295 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:16.296 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508865+0000", 2020-06-12T02:39:16.296 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0629999999999996e-06 2020-06-12T02:39:16.296 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.296 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.296 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:16.297 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.508870+0000", 2020-06-12T02:39:16.297 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010597488 2020-06-12T02:39:16.297 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.297 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.297 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:16.298 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.519467+0000", 2020-06-12T02:39:16.298 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.6083999999999997e-05 2020-06-12T02:39:16.298 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.299 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.299 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:16.299 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.519563+0000", 2020-06-12T02:39:16.299 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0078077800000000003 2020-06-12T02:39:16.299 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.300 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.300 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:16.301 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.527371+0000", 2020-06-12T02:39:16.301 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6198999999999993e-05 2020-06-12T02:39:16.302 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.302 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.302 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:16.302 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.527447+0000", 2020-06-12T02:39:16.302 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0078849720000000005 2020-06-12T02:39:16.303 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.303 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.303 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:16.303 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:23.535332+0000", 2020-06-12T02:39:16.303 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026541928999999999 2020-06-12T02:39:16.304 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:16.304 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:16.304 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:16.304 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.304 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.304 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13705 3.as0 3:5cb723d2:::smithi10729407-15:head [append 0~745472,stat] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:39:16.305 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.390499+0000", 2020-06-12T02:39:16.305 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 47.863479028, 2020-06-12T02:39:16.306 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.067319320000000002, 2020-06-12T02:39:16.306 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:16.307 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:16.307 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:16.307 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:16.307 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:16.307 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13705 2020-06-12T02:39:16.308 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.308 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:16.308 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.308 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:16.308 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.390499+0000", 2020-06-12T02:39:16.309 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:16.309 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.309 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.309 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:16.309 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.390499+0000", 2020-06-12T02:39:16.310 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00066732599999999999 2020-06-12T02:39:16.310 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.311 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.311 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:16.311 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391167+0000", 2020-06-12T02:39:16.311 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3609999999999997e-06 2020-06-12T02:39:16.311 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.313 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.313 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:16.314 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391173+0000", 2020-06-12T02:39:16.314 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0399e-05 2020-06-12T02:39:16.314 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.315 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.315 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:16.315 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391183+0000", 2020-06-12T02:39:16.316 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6480000000000002e-06 2020-06-12T02:39:16.316 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.316 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.316 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:16.317 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391188+0000", 2020-06-12T02:39:16.317 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4150999999999999e-05 2020-06-12T02:39:16.317 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.317 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.318 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:16.318 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391222+0000", 2020-06-12T02:39:16.318 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9450999999999996e-05 2020-06-12T02:39:16.318 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.318 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.319 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:16.319 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391292+0000", 2020-06-12T02:39:16.319 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054214300000000003 2020-06-12T02:39:16.319 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.320 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.320 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:16.321 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.391834+0000", 2020-06-12T02:39:16.321 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0037312069999999998 2020-06-12T02:39:16.322 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.322 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.323 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:16.323 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.395565+0000", 2020-06-12T02:39:16.323 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001050194 2020-06-12T02:39:16.324 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.324 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.324 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:16.325 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396615+0000", 2020-06-12T02:39:16.325 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.061203439999999998 2020-06-12T02:39:16.325 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.326 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.326 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:16.327 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.457819+0000", 2020-06-12T02:39:16.327 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.067319320000000002 2020-06-12T02:39:16.328 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:16.328 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:16.329 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:16.329 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.330 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.330 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13706 3.as0 3:5cb723d2:::smithi10729407-15:head [append 0~434176] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:39:16.330 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.393133+0000", 2020-06-12T02:39:16.330 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 47.860844759999999, 2020-06-12T02:39:16.331 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.078158561000000001, 2020-06-12T02:39:16.331 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:16.331 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:16.331 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:16.331 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:16.331 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:16.332 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13706 2020-06-12T02:39:16.332 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.332 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:16.332 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.332 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:16.333 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393133+0000", 2020-06-12T02:39:16.333 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:16.333 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.333 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.333 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:16.334 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393133+0000", 2020-06-12T02:39:16.334 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00050415499999999995 2020-06-12T02:39:16.334 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.334 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.334 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:16.335 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393638+0000", 2020-06-12T02:39:16.335 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5010000000000004e-06 2020-06-12T02:39:16.335 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.335 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.335 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:16.336 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393646+0000", 2020-06-12T02:39:16.336 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4183e-05 2020-06-12T02:39:16.336 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.336 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.337 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:16.337 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393660+0000", 2020-06-12T02:39:16.337 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6069999999999996e-06 2020-06-12T02:39:16.337 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.337 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.338 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:16.338 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.393667+0000", 2020-06-12T02:39:16.338 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001363569 2020-06-12T02:39:16.338 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.338 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.339 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:16.339 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.395030+0000", 2020-06-12T02:39:16.339 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012127 2020-06-12T02:39:16.339 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.339 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.339 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:16.340 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.395152+0000", 2020-06-12T02:39:16.340 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.063640503000000001 2020-06-12T02:39:16.340 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.340 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.340 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:16.341 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458792+0000", 2020-06-12T02:39:16.341 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010636 2020-06-12T02:39:16.341 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.341 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.341 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:16.342 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458899+0000", 2020-06-12T02:39:16.342 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00076074200000000002 2020-06-12T02:39:16.342 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.342 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.342 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:16.342 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459659+0000", 2020-06-12T02:39:16.343 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00772984 2020-06-12T02:39:16.343 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.343 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.343 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:16.343 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.467389+0000", 2020-06-12T02:39:16.344 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.003815751 2020-06-12T02:39:16.344 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.344 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.344 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:16.344 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471205+0000", 2020-06-12T02:39:16.344 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.708e-05 2020-06-12T02:39:16.345 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.345 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.345 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:16.345 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471292+0000", 2020-06-12T02:39:16.345 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.078158561000000001 2020-06-12T02:39:16.346 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:16.347 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:16.347 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:16.347 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.347 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.348 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13707 3.as0 3:5cb723d2:::smithi10729407-15:head [append 0~565248] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:39:16.348 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.396173+0000", 2020-06-12T02:39:16.348 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 47.857804817000002, 2020-06-12T02:39:16.348 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.075298341000000005, 2020-06-12T02:39:16.348 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:16.349 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:16.349 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:16.349 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:16.349 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:16.349 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13707 2020-06-12T02:39:16.350 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.350 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:16.350 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.350 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:16.350 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396173+0000", 2020-06-12T02:39:16.351 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:16.351 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.351 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.351 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:16.351 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396173+0000", 2020-06-12T02:39:16.351 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.061779787000000003 2020-06-12T02:39:16.352 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.352 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.352 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:16.352 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.457953+0000", 2020-06-12T02:39:16.352 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5830000000000007e-06 2020-06-12T02:39:16.353 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.353 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.353 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:16.353 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.457962+0000", 2020-06-12T02:39:16.353 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3029e-05 2020-06-12T02:39:16.354 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.354 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.354 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:16.354 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.457975+0000", 2020-06-12T02:39:16.354 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.233e-06 2020-06-12T02:39:16.354 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.355 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.355 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:16.355 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.457979+0000", 2020-06-12T02:39:16.355 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0044352510000000003 2020-06-12T02:39:16.355 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.356 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.356 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:16.356 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.462414+0000", 2020-06-12T02:39:16.356 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00012825799999999999 2020-06-12T02:39:16.356 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.356 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.357 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:16.357 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.462543+0000", 2020-06-12T02:39:16.357 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00077227899999999996 2020-06-12T02:39:16.357 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.357 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.358 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:16.358 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.463315+0000", 2020-06-12T02:39:16.358 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0079912390000000007 2020-06-12T02:39:16.358 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.358 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.358 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:16.359 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471306+0000", 2020-06-12T02:39:16.359 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000118674 2020-06-12T02:39:16.359 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.359 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.359 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:16.360 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471425+0000", 2020-06-12T02:39:16.360 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7008000000000001e-05 2020-06-12T02:39:16.360 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.360 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.360 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:16.360 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471472+0000", 2020-06-12T02:39:16.361 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.075298341000000005 2020-06-12T02:39:16.361 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:16.361 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:16.361 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:16.361 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.362 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.362 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWriteReply(3.as0 58/53 ECSubWriteReply(tid=1890, last_complete=58'1184, committed=1, applied=1))", 2020-06-12T02:39:16.362 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.396464+0000", 2020-06-12T02:39:16.362 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 47.857513937999997, 2020-06-12T02:39:16.362 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.062264803000000001, 2020-06-12T02:39:16.363 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:16.363 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:39:16.363 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:16.363 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.363 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:16.363 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396464+0000", 2020-06-12T02:39:16.364 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:16.364 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.364 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.364 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:16.364 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396464+0000", 2020-06-12T02:39:16.365 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9425999999999999e-05 2020-06-12T02:39:16.365 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.365 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.365 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:16.365 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396494+0000", 2020-06-12T02:39:16.365 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1610000000000002e-06 2020-06-12T02:39:16.366 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.366 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.366 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:16.366 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396499+0000", 2020-06-12T02:39:16.366 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2789999999999998e-06 2020-06-12T02:39:16.367 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.367 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.367 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:16.367 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396507+0000", 2020-06-12T02:39:16.367 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1389999999999997e-06 2020-06-12T02:39:16.368 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.368 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.368 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:16.368 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396511+0000", 2020-06-12T02:39:16.368 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2199000000000001e-05 2020-06-12T02:39:16.368 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.369 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.369 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:16.369 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.396543+0000", 2020-06-12T02:39:16.369 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.062185599000000001 2020-06-12T02:39:16.369 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.370 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.370 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:16.370 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458729+0000", 2020-06-12T02:39:16.370 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.062264803000000001 2020-06-12T02:39:16.370 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:16.370 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:16.371 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:16.371 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.371 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.371 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13710 3.as0 3:5cb723d2:::smithi10729407-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e58)", 2020-06-12T02:39:16.371 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.458457+0000", 2020-06-12T02:39:16.372 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 47.795520768000003, 2020-06-12T02:39:16.372 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028102095000000001, 2020-06-12T02:39:16.372 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:16.372 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:16.372 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:16.372 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:16.373 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:16.373 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13710 2020-06-12T02:39:16.373 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.373 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:16.373 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.374 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:16.374 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458457+0000", 2020-06-12T02:39:16.374 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:16.374 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.374 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.374 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:16.375 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458457+0000", 2020-06-12T02:39:16.375 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7650000000000001e-05 2020-06-12T02:39:16.375 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.375 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.375 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:16.376 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458485+0000", 2020-06-12T02:39:16.376 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.3210000000000001e-06 2020-06-12T02:39:16.376 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.376 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.376 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:16.376 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458488+0000", 2020-06-12T02:39:16.377 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5920000000000003e-06 2020-06-12T02:39:16.377 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.377 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.377 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:16.377 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458496+0000", 2020-06-12T02:39:16.378 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8789999999999999e-06 2020-06-12T02:39:16.378 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.378 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.378 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:16.378 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458499+0000", 2020-06-12T02:39:16.378 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013081152 2020-06-12T02:39:16.379 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.379 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.379 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:16.379 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471580+0000", 2020-06-12T02:39:16.379 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4975999999999994e-05 2020-06-12T02:39:16.380 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.380 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.380 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:16.380 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471665+0000", 2020-06-12T02:39:16.380 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0054254029999999996 2020-06-12T02:39:16.381 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.381 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.381 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:16.381 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.477090+0000", 2020-06-12T02:39:16.381 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2335000000000005e-05 2020-06-12T02:39:16.382 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.382 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.382 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:16.382 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.477163+0000", 2020-06-12T02:39:16.382 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0093967870000000002 2020-06-12T02:39:16.383 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.383 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.383 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:16.383 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486560+0000", 2020-06-12T02:39:16.383 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028102095000000001 2020-06-12T02:39:16.383 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:16.384 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:16.384 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:16.384 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.384 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.384 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13711 3.as0 3:5732055a:::smithi10729407-8:head [append 0~761856] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:39:16.385 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.458525+0000", 2020-06-12T02:39:16.385 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 47.795452996000002, 2020-06-12T02:39:16.385 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.027773760000000002, 2020-06-12T02:39:16.385 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:16.385 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:16.386 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:16.386 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:16.386 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:16.386 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13711 2020-06-12T02:39:16.386 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.386 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:16.387 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.387 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:16.387 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458525+0000", 2020-06-12T02:39:16.387 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:16.388 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.388 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.388 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:16.388 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.458525+0000", 2020-06-12T02:39:16.388 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00089965200000000003 2020-06-12T02:39:16.389 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.389 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.389 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:16.389 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459425+0000", 2020-06-12T02:39:16.390 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4749999999999996e-06 2020-06-12T02:39:16.391 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.391 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.391 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:16.391 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459432+0000", 2020-06-12T02:39:16.391 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3043e-05 2020-06-12T02:39:16.392 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.392 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.392 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:16.392 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459445+0000", 2020-06-12T02:39:16.392 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0739999999999998e-06 2020-06-12T02:39:16.393 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.393 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.393 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:16.393 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459451+0000", 2020-06-12T02:39:16.393 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012238733999999999 2020-06-12T02:39:16.394 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.394 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.394 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:16.394 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471690+0000", 2020-06-12T02:39:16.394 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5640000000000003e-05 2020-06-12T02:39:16.394 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.395 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.395 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:16.395 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.471746+0000", 2020-06-12T02:39:16.395 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00067859699999999997 2020-06-12T02:39:16.395 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.396 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.396 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:16.396 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.472424+0000", 2020-06-12T02:39:16.396 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0050083230000000003 2020-06-12T02:39:16.396 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.397 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.397 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:16.397 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.477433+0000", 2020-06-12T02:39:16.397 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0087878039999999998 2020-06-12T02:39:16.397 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.398 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.398 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:16.398 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486221+0000", 2020-06-12T02:39:16.398 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8417999999999994e-05 2020-06-12T02:39:16.398 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.398 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.399 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:16.399 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486299+0000", 2020-06-12T02:39:16.399 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.027773760000000002 2020-06-12T02:39:16.399 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:16.399 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:16.400 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:16.400 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.400 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.400 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13712 3.as0 3:5732055a:::smithi10729407-8:head [append 0~712704] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:39:16.400 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.459508+0000", 2020-06-12T02:39:16.401 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 47.794469716000002, 2020-06-12T02:39:16.401 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.033995818999999997, 2020-06-12T02:39:16.401 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:16.401 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:16.401 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:16.402 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:16.402 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:16.402 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13712 2020-06-12T02:39:16.402 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.402 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:16.402 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.403 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:16.403 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459508+0000", 2020-06-12T02:39:16.403 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:16.403 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.403 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.404 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:16.404 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.459508+0000", 2020-06-12T02:39:16.404 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00075436999999999998 2020-06-12T02:39:16.404 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.404 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.405 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:16.405 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460263+0000", 2020-06-12T02:39:16.405 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4370000000000001e-06 2020-06-12T02:39:16.405 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.405 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.406 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:16.406 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460270+0000", 2020-06-12T02:39:16.406 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2500000000000001e-05 2020-06-12T02:39:16.406 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.406 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.406 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:16.407 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460283+0000", 2020-06-12T02:39:16.407 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.2580000000000003e-06 2020-06-12T02:39:16.407 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.407 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.407 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:16.408 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460288+0000", 2020-06-12T02:39:16.408 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017177595 2020-06-12T02:39:16.408 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.408 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.408 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:16.408 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.477466+0000", 2020-06-12T02:39:16.409 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00019751600000000001 2020-06-12T02:39:16.409 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.409 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.409 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:16.409 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.477663+0000", 2020-06-12T02:39:16.410 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00088697400000000001 2020-06-12T02:39:16.410 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.410 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.410 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:16.410 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.478550+0000", 2020-06-12T02:39:16.411 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.007578787 2020-06-12T02:39:16.411 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.411 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.411 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:16.411 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486129+0000", 2020-06-12T02:39:16.412 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0072972310000000004 2020-06-12T02:39:16.412 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.412 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.412 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:16.412 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493426+0000", 2020-06-12T02:39:16.412 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.8151000000000004e-05 2020-06-12T02:39:16.413 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.413 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.413 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:16.413 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493504+0000", 2020-06-12T02:39:16.414 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.033995818999999997 2020-06-12T02:39:16.414 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:16.414 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:16.414 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:16.415 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.415 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.415 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13713 3.as0 3:5732055a:::smithi10729407-8:head [append 0~491520] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:39:16.415 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.460333+0000", 2020-06-12T02:39:16.415 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 47.793645259999998, 2020-06-12T02:39:16.415 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.033390145000000003, 2020-06-12T02:39:16.416 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:16.416 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:16.416 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:16.416 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:16.416 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:16.417 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13713 2020-06-12T02:39:16.417 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.417 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:16.417 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.417 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:16.418 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460333+0000", 2020-06-12T02:39:16.418 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:16.418 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.418 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.418 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:16.418 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460333+0000", 2020-06-12T02:39:16.419 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00051168999999999997 2020-06-12T02:39:16.419 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.419 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.419 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:16.419 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460845+0000", 2020-06-12T02:39:16.420 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2930000000000003e-06 2020-06-12T02:39:16.420 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.420 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.420 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:16.420 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460852+0000", 2020-06-12T02:39:16.420 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3054000000000001e-05 2020-06-12T02:39:16.421 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.421 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.421 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:16.421 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460865+0000", 2020-06-12T02:39:16.422 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6130000000000003e-06 2020-06-12T02:39:16.422 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.422 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.423 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:16.423 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460871+0000", 2020-06-12T02:39:16.423 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025753870000000002 2020-06-12T02:39:16.423 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.423 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.424 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:16.424 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486624+0000", 2020-06-12T02:39:16.424 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2739999999999998e-05 2020-06-12T02:39:16.424 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.424 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.425 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:16.425 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.486697+0000", 2020-06-12T02:39:16.425 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00063578799999999998 2020-06-12T02:39:16.425 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.425 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.426 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:16.426 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.487333+0000", 2020-06-12T02:39:16.426 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0043011610000000004 2020-06-12T02:39:16.426 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.426 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.426 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:16.427 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.491634+0000", 2020-06-12T02:39:16.427 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0020130949999999999 2020-06-12T02:39:16.427 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.427 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.427 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:16.428 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493647+0000", 2020-06-12T02:39:16.428 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5840999999999994e-05 2020-06-12T02:39:16.428 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.428 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.428 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:16.428 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493723+0000", 2020-06-12T02:39:16.429 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.033390145000000003 2020-06-12T02:39:16.429 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:16.430 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:16.430 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:16.430 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.430 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.430 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13714 3.as0 3:5732055a:::smithi10729407-8:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:39:16.431 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.460921+0000", 2020-06-12T02:39:16.431 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 47.793056737999997, 2020-06-12T02:39:16.431 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.032981457999999998, 2020-06-12T02:39:16.431 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:16.431 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:16.432 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:16.432 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:16.432 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:16.432 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13714 2020-06-12T02:39:16.432 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.432 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:16.433 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.433 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:16.433 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460921+0000", 2020-06-12T02:39:16.433 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:16.433 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.434 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.434 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:16.434 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460921+0000", 2020-06-12T02:39:16.434 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.6996999999999995e-05 2020-06-12T02:39:16.435 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.435 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.435 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:16.435 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.460998+0000", 2020-06-12T02:39:16.435 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4529999999999999e-06 2020-06-12T02:39:16.436 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.436 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.436 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:16.436 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461005+0000", 2020-06-12T02:39:16.436 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1722000000000001e-05 2020-06-12T02:39:16.436 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.437 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.437 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:16.437 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461017+0000", 2020-06-12T02:39:16.437 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4839999999999998e-06 2020-06-12T02:39:16.437 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.438 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.438 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:16.438 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461021+0000", 2020-06-12T02:39:16.438 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.029157975999999999 2020-06-12T02:39:16.438 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.438 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.439 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:16.439 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.490179+0000", 2020-06-12T02:39:16.439 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010966700000000001 2020-06-12T02:39:16.439 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.439 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.440 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:16.440 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.490289+0000", 2020-06-12T02:39:16.440 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00032026800000000002 2020-06-12T02:39:16.440 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.440 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.441 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:16.441 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.490609+0000", 2020-06-12T02:39:16.441 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001102947 2020-06-12T02:39:16.441 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.441 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.441 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:16.442 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.491712+0000", 2020-06-12T02:39:16.442 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0021050169999999998 2020-06-12T02:39:16.442 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.442 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.443 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:16.443 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493817+0000", 2020-06-12T02:39:16.444 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5927000000000006e-05 2020-06-12T02:39:16.444 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.445 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.445 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:16.445 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.493903+0000", 2020-06-12T02:39:16.445 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.032981457999999998 2020-06-12T02:39:16.446 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:16.446 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:16.447 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:16.447 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.447 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.447 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:13715 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e58)", 2020-06-12T02:39:16.447 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:28.461062+0000", 2020-06-12T02:39:16.448 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 47.792916097999999, 2020-06-12T02:39:16.448 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.035208338999999998, 2020-06-12T02:39:16.450 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:16.450 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:16.450 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:16.451 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:16.451 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:16.451 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 13715 2020-06-12T02:39:16.451 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.451 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:16.452 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.452 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:16.452 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461062+0000", 2020-06-12T02:39:16.453 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:16.454 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.454 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.454 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:16.455 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461062+0000", 2020-06-12T02:39:16.455 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4839999999999998e-05 2020-06-12T02:39:16.455 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.455 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.455 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:16.456 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461107+0000", 2020-06-12T02:39:16.456 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4859999999999997e-06 2020-06-12T02:39:16.456 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.456 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.456 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:16.457 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461112+0000", 2020-06-12T02:39:16.457 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1742999999999999e-05 2020-06-12T02:39:16.457 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.457 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.457 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:16.458 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461124+0000", 2020-06-12T02:39:16.458 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7299999999999996e-06 2020-06-12T02:39:16.458 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.458 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.458 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:16.458 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.461129+0000", 2020-06-12T02:39:16.459 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.030397368000000001 2020-06-12T02:39:16.459 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.459 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.459 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:16.459 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.491526+0000", 2020-06-12T02:39:16.460 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010076799999999999 2020-06-12T02:39:16.460 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.460 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.460 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:16.460 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.491627+0000", 2020-06-12T02:39:16.461 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0031869849999999998 2020-06-12T02:39:16.461 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.462 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.462 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:16.462 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.494814+0000", 2020-06-12T02:39:16.462 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000108915 2020-06-12T02:39:16.462 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.462 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.463 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:16.463 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.494923+0000", 2020-06-12T02:39:16.463 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001347504 2020-06-12T02:39:16.463 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.463 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.464 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:16.464 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:28.496270+0000", 2020-06-12T02:39:16.464 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.035208338999999998 2020-06-12T02:39:16.464 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:16.464 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:16.465 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:16.465 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.465 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.465 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:16706 3.as0 3:5cb723d2:::smithi10729407-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e61)", 2020-06-12T02:39:16.465 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:37.289063+0000", 2020-06-12T02:39:16.466 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 38.964915073999997, 2020-06-12T02:39:16.466 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026146086999999998, 2020-06-12T02:39:16.466 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:16.467 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:16.467 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:16.467 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:16.468 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:16.468 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 16706 2020-06-12T02:39:16.468 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.468 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:16.468 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.469 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:16.469 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289063+0000", 2020-06-12T02:39:16.469 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:16.469 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.469 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.470 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:16.470 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289063+0000", 2020-06-12T02:39:16.470 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3776000000000002e-05 2020-06-12T02:39:16.470 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.470 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.470 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:16.471 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289107+0000", 2020-06-12T02:39:16.471 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7039999999999999e-06 2020-06-12T02:39:16.471 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.471 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.471 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:16.472 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289113+0000", 2020-06-12T02:39:16.472 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3259e-05 2020-06-12T02:39:16.472 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.472 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.472 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:16.473 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289126+0000", 2020-06-12T02:39:16.473 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1999999999999996e-06 2020-06-12T02:39:16.473 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.473 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.473 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:16.473 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.289130+0000", 2020-06-12T02:39:16.474 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0010912820000000001 2020-06-12T02:39:16.474 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.474 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.474 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:16.474 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.290221+0000", 2020-06-12T02:39:16.475 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3166e-05 2020-06-12T02:39:16.475 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.475 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.475 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:16.475 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.290265+0000", 2020-06-12T02:39:16.475 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.015984187 2020-06-12T02:39:16.476 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.476 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.476 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:16.476 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.306249+0000", 2020-06-12T02:39:16.476 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4623000000000006e-05 2020-06-12T02:39:16.477 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.477 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.477 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:16.477 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.306333+0000", 2020-06-12T02:39:16.477 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0068573690000000003 2020-06-12T02:39:16.478 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.478 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.478 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:16.478 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.313191+0000", 2020-06-12T02:39:16.478 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.7804000000000002e-05 2020-06-12T02:39:16.479 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.479 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.479 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:16.479 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.313279+0000", 2020-06-12T02:39:16.479 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001930717 2020-06-12T02:39:16.479 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.480 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.480 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:16.480 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:37.315209+0000", 2020-06-12T02:39:16.480 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026146086999999998 2020-06-12T02:39:16.480 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:16.481 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:16.481 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:16.481 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.481 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.481 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18051 3.as0 3:5cb723d2:::smithi10729407-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:39:16.481 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.329078+0000", 2020-06-12T02:39:16.482 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.924900524000002, 2020-06-12T02:39:16.482 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026465103, 2020-06-12T02:39:16.482 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:16.482 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:16.483 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:16.483 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:16.483 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:16.484 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18051 2020-06-12T02:39:16.484 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.484 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:16.485 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.485 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:16.485 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329078+0000", 2020-06-12T02:39:16.485 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:16.485 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.486 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.486 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:16.486 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329078+0000", 2020-06-12T02:39:16.486 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5392000000000001e-05 2020-06-12T02:39:16.486 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.487 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.487 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:16.487 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329133+0000", 2020-06-12T02:39:16.487 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5329999999999997e-06 2020-06-12T02:39:16.487 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.488 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.488 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:16.488 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329141+0000", 2020-06-12T02:39:16.488 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3459999999999999e-05 2020-06-12T02:39:16.488 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.488 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.489 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:16.489 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329164+0000", 2020-06-12T02:39:16.489 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.8170000000000001e-06 2020-06-12T02:39:16.489 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.489 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.490 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:16.490 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.329171+0000", 2020-06-12T02:39:16.490 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013005208000000001 2020-06-12T02:39:16.490 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.490 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.491 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:16.491 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.342176+0000", 2020-06-12T02:39:16.491 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.3560999999999998e-05 2020-06-12T02:39:16.491 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.491 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.492 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:16.492 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.342230+0000", 2020-06-12T02:39:16.492 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0080818490000000003 2020-06-12T02:39:16.492 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.493 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.493 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:16.493 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.350312+0000", 2020-06-12T02:39:16.494 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0167000000000006e-05 2020-06-12T02:39:16.494 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.494 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.494 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:16.494 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.350402+0000", 2020-06-12T02:39:16.495 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051411160000000003 2020-06-12T02:39:16.495 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.495 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.495 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:16.495 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.355543+0000", 2020-06-12T02:39:16.496 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026465103 2020-06-12T02:39:16.496 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:16.496 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:16.496 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:16.496 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.497 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.497 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18053 3.as0 3:5732055a:::smithi10729407-8:head [append 0~434176] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:16.497 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.334425+0000", 2020-06-12T02:39:16.497 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.919552957000001, 2020-06-12T02:39:16.497 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.029616703000000001, 2020-06-12T02:39:16.498 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:16.498 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:16.498 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:16.498 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:16.498 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:16.498 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18053 2020-06-12T02:39:16.499 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.499 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:16.499 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.499 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:16.499 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334425+0000", 2020-06-12T02:39:16.500 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:16.500 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.500 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.500 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:16.500 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334425+0000", 2020-06-12T02:39:16.500 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00047283900000000002 2020-06-12T02:39:16.501 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.501 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.501 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:16.501 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334898+0000", 2020-06-12T02:39:16.501 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.9100000000000004e-06 2020-06-12T02:39:16.502 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.502 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.502 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:16.502 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334903+0000", 2020-06-12T02:39:16.502 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4260000000000003e-06 2020-06-12T02:39:16.503 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.503 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.503 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:16.503 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334911+0000", 2020-06-12T02:39:16.503 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1060000000000002e-06 2020-06-12T02:39:16.503 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.504 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.504 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:16.504 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.334916+0000", 2020-06-12T02:39:16.504 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010838316000000001 2020-06-12T02:39:16.504 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.505 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.505 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:16.505 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.345754+0000", 2020-06-12T02:39:16.505 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000103872 2020-06-12T02:39:16.505 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.505 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.506 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:16.506 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.345858+0000", 2020-06-12T02:39:16.506 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0059798739999999996 2020-06-12T02:39:16.506 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.506 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.507 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:16.507 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.351838+0000", 2020-06-12T02:39:16.507 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000104724 2020-06-12T02:39:16.507 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.507 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.507 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:16.508 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.351942+0000", 2020-06-12T02:39:16.508 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000572687 2020-06-12T02:39:16.508 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.508 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.508 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:16.509 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.352515+0000", 2020-06-12T02:39:16.509 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002905641 2020-06-12T02:39:16.509 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.509 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.509 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:16.510 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.355421+0000", 2020-06-12T02:39:16.510 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0085358350000000003 2020-06-12T02:39:16.510 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.511 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.511 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:16.512 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.363956+0000", 2020-06-12T02:39:16.512 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5473e-05 2020-06-12T02:39:16.512 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.512 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.513 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:16.513 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.364042+0000", 2020-06-12T02:39:16.513 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.029616703000000001 2020-06-12T02:39:16.513 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:16.513 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:16.514 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:16.514 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.514 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.515 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18054 3.as0 3:5732055a:::smithi10729407-8:head [append 0~638976] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:16.515 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.338171+0000", 2020-06-12T02:39:16.515 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.915807454000003, 2020-06-12T02:39:16.515 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.030067120999999999, 2020-06-12T02:39:16.515 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:16.516 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:16.516 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:16.516 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:16.516 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:16.517 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18054 2020-06-12T02:39:16.517 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.518 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:16.518 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.519 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:16.519 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338171+0000", 2020-06-12T02:39:16.519 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:16.519 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.519 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.519 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:16.520 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338171+0000", 2020-06-12T02:39:16.520 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00068138900000000002 2020-06-12T02:39:16.520 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.520 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.520 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:16.521 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338852+0000", 2020-06-12T02:39:16.521 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.767e-06 2020-06-12T02:39:16.521 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.521 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.521 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:16.522 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338858+0000", 2020-06-12T02:39:16.522 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8999999999999995e-06 2020-06-12T02:39:16.522 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.522 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.522 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:16.523 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338867+0000", 2020-06-12T02:39:16.523 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.262e-06 2020-06-12T02:39:16.523 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.523 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.523 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:16.524 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.338871+0000", 2020-06-12T02:39:16.524 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0074839060000000002 2020-06-12T02:39:16.524 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.525 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.525 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:16.525 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346355+0000", 2020-06-12T02:39:16.525 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1435999999999999e-05 2020-06-12T02:39:16.525 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.526 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.526 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:16.526 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346416+0000", 2020-06-12T02:39:16.526 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0092109879999999998 2020-06-12T02:39:16.526 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.527 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.527 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:16.527 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.355627+0000", 2020-06-12T02:39:16.527 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7355999999999998e-05 2020-06-12T02:39:16.527 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.528 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.528 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:16.528 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.355685+0000", 2020-06-12T02:39:16.528 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00074711999999999999 2020-06-12T02:39:16.528 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.529 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.529 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:16.529 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.356432+0000", 2020-06-12T02:39:16.529 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.010659572000000001 2020-06-12T02:39:16.529 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.529 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.530 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:16.530 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.367091+0000", 2020-06-12T02:39:16.530 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0010713210000000001 2020-06-12T02:39:16.530 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.532 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.532 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:16.532 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.368163+0000", 2020-06-12T02:39:16.533 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5104000000000005e-05 2020-06-12T02:39:16.533 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.533 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.533 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:16.535 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.368238+0000", 2020-06-12T02:39:16.535 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.030067120999999999 2020-06-12T02:39:16.535 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:16.535 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:16.536 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:16.536 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.536 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.536 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18055 3.as0 3:5732055a:::smithi10729407-8:head [append 0~483328] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:16.536 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.340740+0000", 2020-06-12T02:39:16.537 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.913238604999997, 2020-06-12T02:39:16.537 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028669050000000001, 2020-06-12T02:39:16.537 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:16.537 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:16.537 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:16.538 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:16.538 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:16.538 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18055 2020-06-12T02:39:16.538 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.538 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:16.539 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.539 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:16.539 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.340740+0000", 2020-06-12T02:39:16.539 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:16.539 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.539 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.540 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:16.540 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.340740+0000", 2020-06-12T02:39:16.540 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00045126900000000002 2020-06-12T02:39:16.540 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.540 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.541 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:16.541 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341191+0000", 2020-06-12T02:39:16.541 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5630000000000003e-06 2020-06-12T02:39:16.541 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.541 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.542 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:16.542 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341198+0000", 2020-06-12T02:39:16.542 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.4467999999999999e-05 2020-06-12T02:39:16.542 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.542 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.543 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:16.543 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341213+0000", 2020-06-12T02:39:16.543 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7540000000000003e-06 2020-06-12T02:39:16.543 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.543 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.543 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:16.544 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341219+0000", 2020-06-12T02:39:16.544 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0052272680000000002 2020-06-12T02:39:16.544 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.544 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.544 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:16.545 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346446+0000", 2020-06-12T02:39:16.545 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.4742999999999998e-05 2020-06-12T02:39:16.545 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.545 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.546 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:16.546 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346531+0000", 2020-06-12T02:39:16.546 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.013203138999999999 2020-06-12T02:39:16.546 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.546 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.547 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:16.547 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.359734+0000", 2020-06-12T02:39:16.547 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000100277 2020-06-12T02:39:16.547 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.547 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.547 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:16.548 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.359834+0000", 2020-06-12T02:39:16.548 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00054998099999999995 2020-06-12T02:39:16.548 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.548 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.548 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:16.549 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.360384+0000", 2020-06-12T02:39:16.549 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0067087179999999998 2020-06-12T02:39:16.549 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.549 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.549 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:16.550 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.367093+0000", 2020-06-12T02:39:16.550 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0022746020000000001 2020-06-12T02:39:16.550 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.550 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.550 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:16.550 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369367+0000", 2020-06-12T02:39:16.551 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1267999999999999e-05 2020-06-12T02:39:16.551 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.551 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.551 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:16.551 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369409+0000", 2020-06-12T02:39:16.552 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028669050000000001 2020-06-12T02:39:16.552 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:16.552 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:16.552 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:16.552 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.552 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.553 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18056 3.as0 3:5732055a:::smithi10729407-8:head [append 0~49152] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:16.553 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.341270+0000", 2020-06-12T02:39:16.553 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.912707816000001, 2020-06-12T02:39:16.553 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028350462999999999, 2020-06-12T02:39:16.553 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:16.554 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:16.554 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:16.554 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:16.554 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:16.554 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18056 2020-06-12T02:39:16.554 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.555 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:16.555 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.555 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:16.555 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341270+0000", 2020-06-12T02:39:16.556 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:16.556 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.556 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.556 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:16.556 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341270+0000", 2020-06-12T02:39:16.557 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000114082 2020-06-12T02:39:16.557 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.557 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.557 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:16.557 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341384+0000", 2020-06-12T02:39:16.558 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3289999999999999e-06 2020-06-12T02:39:16.558 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.558 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.559 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:16.559 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341391+0000", 2020-06-12T02:39:16.559 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3339999999999999e-05 2020-06-12T02:39:16.559 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.560 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.560 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:16.560 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341404+0000", 2020-06-12T02:39:16.560 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4e-06 2020-06-12T02:39:16.560 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.560 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.561 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:16.561 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341410+0000", 2020-06-12T02:39:16.561 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051518600000000003 2020-06-12T02:39:16.561 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.561 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.562 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:16.562 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346561+0000", 2020-06-12T02:39:16.562 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1662999999999994e-05 2020-06-12T02:39:16.562 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.562 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.563 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:16.563 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346643+0000", 2020-06-12T02:39:16.563 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.017471605000000001 2020-06-12T02:39:16.563 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.563 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.563 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:16.564 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.364115+0000", 2020-06-12T02:39:16.564 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00011059400000000001 2020-06-12T02:39:16.564 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.564 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.564 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:16.565 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.364225+0000", 2020-06-12T02:39:16.565 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000611262 2020-06-12T02:39:16.565 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.565 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.566 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:16.566 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.364837+0000", 2020-06-12T02:39:16.566 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023767710000000002 2020-06-12T02:39:16.566 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.566 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.567 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:16.567 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.367213+0000", 2020-06-12T02:39:16.567 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023325479999999998 2020-06-12T02:39:16.567 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.567 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.568 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:16.568 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369546+0000", 2020-06-12T02:39:16.568 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5009e-05 2020-06-12T02:39:16.568 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.568 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.568 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:16.569 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369621+0000", 2020-06-12T02:39:16.569 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028350462999999999 2020-06-12T02:39:16.569 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:16.569 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:16.569 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:16.570 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.570 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.570 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18057 3.as0 3:5732055a:::smithi10729407-8:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:16.570 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.341456+0000", 2020-06-12T02:39:16.570 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.912522656, 2020-06-12T02:39:16.570 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028443018, 2020-06-12T02:39:16.571 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:16.571 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:16.571 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:16.571 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:16.571 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:16.572 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18057 2020-06-12T02:39:16.572 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.572 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:16.572 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.572 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:16.572 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341456+0000", 2020-06-12T02:39:16.573 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:16.573 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.573 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.573 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:16.573 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341456+0000", 2020-06-12T02:39:16.574 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6036000000000003e-05 2020-06-12T02:39:16.574 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.574 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.574 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:16.574 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341522+0000", 2020-06-12T02:39:16.575 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2210000000000002e-06 2020-06-12T02:39:16.575 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.575 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.575 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:16.575 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341528+0000", 2020-06-12T02:39:16.576 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0338e-05 2020-06-12T02:39:16.576 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.576 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.576 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:16.576 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341538+0000", 2020-06-12T02:39:16.577 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8639999999999998e-06 2020-06-12T02:39:16.577 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.578 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.578 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:16.578 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341542+0000", 2020-06-12T02:39:16.579 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051405009999999996 2020-06-12T02:39:16.579 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.579 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.579 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:16.580 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346683+0000", 2020-06-12T02:39:16.580 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9561999999999995e-05 2020-06-12T02:39:16.580 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.580 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.580 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:16.581 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346762+0000", 2020-06-12T02:39:16.581 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.019267197999999999 2020-06-12T02:39:16.581 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.581 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.582 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:16.582 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.366029+0000", 2020-06-12T02:39:16.582 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2634999999999997e-05 2020-06-12T02:39:16.582 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.583 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.583 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:16.583 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.366122+0000", 2020-06-12T02:39:16.583 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00026907500000000002 2020-06-12T02:39:16.583 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.584 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.584 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:16.584 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.366391+0000", 2020-06-12T02:39:16.584 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0018939409999999999 2020-06-12T02:39:16.584 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.585 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.585 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:16.585 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.368285+0000", 2020-06-12T02:39:16.585 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001549405 2020-06-12T02:39:16.585 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.586 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.586 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:16.586 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369834+0000", 2020-06-12T02:39:16.586 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4241999999999999e-05 2020-06-12T02:39:16.586 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.587 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.587 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:16.587 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.369899+0000", 2020-06-12T02:39:16.587 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.028443018 2020-06-12T02:39:16.587 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:16.588 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:16.588 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:16.588 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.588 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.588 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18058 3.as0 3:5732055a:::smithi10729407-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:39:16.589 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:41.341580+0000", 2020-06-12T02:39:16.589 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 34.912397749, 2020-06-12T02:39:16.589 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.030449559000000001, 2020-06-12T02:39:16.589 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:16.590 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:16.590 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:16.590 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:16.590 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:16.591 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18058 2020-06-12T02:39:16.591 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.591 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:16.591 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.592 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:16.592 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341580+0000", 2020-06-12T02:39:16.592 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:16.592 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.592 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.592 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:16.593 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341580+0000", 2020-06-12T02:39:16.593 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1185e-05 2020-06-12T02:39:16.593 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.593 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.593 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:16.594 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341622+0000", 2020-06-12T02:39:16.594 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1599999999999997e-06 2020-06-12T02:39:16.594 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.594 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.594 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:16.595 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341627+0000", 2020-06-12T02:39:16.595 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3546e-05 2020-06-12T02:39:16.595 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.595 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.595 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:16.596 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341640+0000", 2020-06-12T02:39:16.596 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.9779999999999999e-06 2020-06-12T02:39:16.596 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.596 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.596 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:16.597 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.341644+0000", 2020-06-12T02:39:16.597 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0051433700000000004 2020-06-12T02:39:16.597 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.597 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.597 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:16.598 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346788+0000", 2020-06-12T02:39:16.598 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1622000000000001e-05 2020-06-12T02:39:16.598 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.598 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.599 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:16.599 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.346839+0000", 2020-06-12T02:39:16.599 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.020283958000000001 2020-06-12T02:39:16.599 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.599 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.600 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:16.600 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.367123+0000", 2020-06-12T02:39:16.600 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1997999999999997e-05 2020-06-12T02:39:16.600 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.600 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.601 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:16.601 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.367205+0000", 2020-06-12T02:39:16.601 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0036875440000000001 2020-06-12T02:39:16.601 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.601 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.602 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:16.602 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.370893+0000", 2020-06-12T02:39:16.602 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000100604 2020-06-12T02:39:16.602 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.602 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.603 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:16.603 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.370993+0000", 2020-06-12T02:39:16.603 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0010365940000000001 2020-06-12T02:39:16.603 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:16.604 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:16.604 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:16.604 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:41.372030+0000", 2020-06-12T02:39:16.604 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.030449559000000001 2020-06-12T02:39:16.605 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:16.605 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:16.606 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:16.606 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:16.606 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:16.606 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:39:16.615 INFO:tasks.rados.rados.0.smithi107.stdout:11072: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:16.616 INFO:tasks.rados.rados.0.smithi107.stdout:11072: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:16.616 INFO:tasks.rados.rados.0.smithi107.stdout:11072: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:16.616 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 851 (ObjNum 11071 snap 0 seq_num 11071) dirty exists 2020-06-12T02:39:16.616 INFO:tasks.rados.rados.0.smithi107.stdout:11072: left oid 20 (ObjNum 11071 snap 0 seq_num 11071) 2020-06-12T02:39:16.617 INFO:tasks.rados.rados.0.smithi107.stdout:11069: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:16.617 INFO:tasks.rados.rados.0.smithi107.stdout:11072: done (2 left) 2020-06-12T02:39:16.617 INFO:tasks.rados.rados.0.smithi107.stdout:11073: done (1 left) 2020-06-12T02:39:16.617 INFO:tasks.rados.rados.0.smithi107.stdout:11074: delete oid 14 current snap is 0 2020-06-12T02:39:16.617 INFO:tasks.rados.rados.0.smithi107.stdout:11069: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:16.617 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 2053 (ObjNum 11068 snap 0 seq_num 11068) dirty exists 2020-06-12T02:39:16.618 INFO:tasks.rados.rados.0.smithi107.stdout:11069: left oid 12 (ObjNum 11068 snap 0 seq_num 11068) 2020-06-12T02:39:16.618 INFO:tasks.rados.rados.0.smithi107.stdout:11069: done (1 left) 2020-06-12T02:39:16.618 INFO:tasks.rados.rados.0.smithi107.stdout:11074: done (0 left) 2020-06-12T02:39:16.618 INFO:tasks.rados.rados.0.smithi107.stdout:11075: delete oid 12 current snap is 0 2020-06-12T02:39:16.618 INFO:tasks.rados.rados.0.smithi107.stdout:11075: done (0 left) 2020-06-12T02:39:16.619 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:39:16.619 INFO:tasks.rados.rados.0.smithi107.stdout:11076: seq_num 11075 ranges {0=688128} 2020-06-12T02:39:16.619 INFO:tasks.rados.rados.0.smithi107.stdout:11076: writing smithi10729407-13 from 0 to 688128 tid 1 2020-06-12T02:39:16.619 INFO:tasks.rados.rados.0.smithi107.stdout:11077: delete oid 14 current snap is 0 2020-06-12T02:39:16.619 INFO:tasks.rados.rados.0.smithi107.stdout:11077: done (1 left) 2020-06-12T02:39:16.620 INFO:tasks.rados.rados.0.smithi107.stdout:11078: delete oid 12 current snap is 0 2020-06-12T02:39:16.620 INFO:tasks.rados.rados.0.smithi107.stdout:11078: done (1 left) 2020-06-12T02:39:16.620 INFO:tasks.rados.rados.0.smithi107.stdout:11079: delete oid 15 current snap is 0 2020-06-12T02:39:16.620 INFO:tasks.rados.rados.0.smithi107.stdout:11076: finishing write tid 1 to smithi10729407-13 2020-06-12T02:39:16.620 INFO:tasks.rados.rados.0.smithi107.stdout:11079: done (1 left) 2020-06-12T02:39:16.620 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:39:16.621 INFO:tasks.rados.rados.0.smithi107.stdout:11080: seq_num 11079 ranges {385024=761856} 2020-06-12T02:39:16.621 INFO:tasks.rados.rados.0.smithi107.stdout:11080: writing smithi10729407-11 from 385024 to 1146880 tid 1 2020-06-12T02:39:16.622 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:39:16.622 INFO:tasks.rados.rados.0.smithi107.stdout:11081: seq_num 11080 ranges {0=778240} 2020-06-12T02:39:16.622 INFO:tasks.rados.rados.0.smithi107.stdout:11081: writing smithi10729407-2 from 0 to 778240 tid 1 2020-06-12T02:39:16.623 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:39:16.623 INFO:tasks.rados.rados.0.smithi107.stdout:11082: seq_num 11081 ranges {2154496=794624} 2020-06-12T02:39:16.623 INFO:tasks.rados.rados.0.smithi107.stdout:11082: writing smithi10729407-18 from 2154496 to 2949120 tid 1 2020-06-12T02:39:16.623 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:39:16.624 INFO:tasks.rados.rados.0.smithi107.stdout:11083: seq_num 11082 ranges {0=409600,409600=401408} 2020-06-12T02:39:16.624 INFO:tasks.rados.rados.0.smithi107.stdout:11083: writing smithi10729407-8 from 0 to 409600 tid 1 2020-06-12T02:39:16.624 INFO:tasks.rados.rados.0.smithi107.stdout:11083: writing smithi10729407-8 from 409600 to 811008 tid 2 2020-06-12T02:39:16.624 INFO:tasks.rados.rados.0.smithi107.stdout:11084: delete oid 20 current snap is 0 2020-06-12T02:39:16.624 INFO:tasks.rados.rados.0.smithi107.stdout:11076: finishing write tid 3 to smithi10729407-13 2020-06-12T02:39:16.625 INFO:tasks.rados.rados.0.smithi107.stdout:11076: finishing write tid 4 to smithi10729407-13 2020-06-12T02:39:16.625 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1786 (ObjNum 11075 snap 0 seq_num 11075) dirty exists 2020-06-12T02:39:16.625 INFO:tasks.rados.rados.0.smithi107.stdout:11076: left oid 13 (ObjNum 11075 snap 0 seq_num 11075) 2020-06-12T02:39:16.625 INFO:tasks.rados.rados.0.smithi107.stdout:11080: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:16.626 INFO:tasks.rados.rados.0.smithi107.stdout:11080: finishing write tid 3 to smithi10729407-11 2020-06-12T02:39:16.626 INFO:tasks.rados.rados.0.smithi107.stdout:11081: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:16.626 INFO:tasks.rados.rados.0.smithi107.stdout:11080: finishing write tid 4 to smithi10729407-11 2020-06-12T02:39:16.626 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1768 (ObjNum 11079 snap 0 seq_num 11079) dirty exists 2020-06-12T02:39:16.626 INFO:tasks.rados.rados.0.smithi107.stdout:11080: left oid 11 (ObjNum 11079 snap 0 seq_num 11079) 2020-06-12T02:39:16.626 INFO:tasks.rados.rados.0.smithi107.stdout:11081: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:16.627 INFO:tasks.rados.rados.0.smithi107.stdout:11081: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:16.627 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 1043 (ObjNum 11080 snap 0 seq_num 11080) dirty exists 2020-06-12T02:39:16.627 INFO:tasks.rados.rados.0.smithi107.stdout:11081: left oid 2 (ObjNum 11080 snap 0 seq_num 11080) 2020-06-12T02:39:16.627 INFO:tasks.rados.rados.0.smithi107.stdout:11076: done (5 left) 2020-06-12T02:39:16.627 INFO:tasks.rados.rados.0.smithi107.stdout:11080: done (4 left) 2020-06-12T02:39:16.628 INFO:tasks.rados.rados.0.smithi107.stdout:11081: done (3 left) 2020-06-12T02:39:16.628 INFO:tasks.rados.rados.0.smithi107.stdout:11084: done (2 left) 2020-06-12T02:39:16.628 INFO:tasks.rados.rados.0.smithi107.stdout:11085: delete oid 1 current snap is 0 2020-06-12T02:39:16.628 INFO:tasks.rados.rados.0.smithi107.stdout:11082: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:16.628 INFO:tasks.rados.rados.0.smithi107.stdout:11083: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:16.628 INFO:tasks.rados.rados.0.smithi107.stdout:11083: finishing write tid 2 to smithi10729407-8 2020-06-12T02:39:16.629 INFO:tasks.rados.rados.0.smithi107.stdout:11082: finishing write tid 3 to smithi10729407-18 2020-06-12T02:39:16.629 INFO:tasks.rados.rados.0.smithi107.stdout:11083: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:16.629 INFO:tasks.rados.rados.0.smithi107.stdout:11085: done (2 left) 2020-06-12T02:39:16.629 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:39:16.629 INFO:tasks.rados.rados.0.smithi107.stdout:11086: seq_num 11085 ranges {688128=458752,1146880=401408} 2020-06-12T02:39:16.630 INFO:tasks.rados.rados.0.smithi107.stdout:11086: writing smithi10729407-13 from 688128 to 1146880 tid 1 2020-06-12T02:39:16.630 INFO:tasks.rados.rados.0.smithi107.stdout:11086: writing smithi10729407-13 from 1146880 to 1548288 tid 2 2020-06-12T02:39:16.630 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:39:16.630 INFO:tasks.rados.rados.0.smithi107.stdout:11087: seq_num 11086 ranges {0=475136,475136=401408} 2020-06-12T02:39:16.631 INFO:tasks.rados.rados.0.smithi107.stdout:11087: writing smithi10729407-10 from 0 to 475136 tid 1 2020-06-12T02:39:16.631 INFO:tasks.rados.rados.0.smithi107.stdout:11087: writing smithi10729407-10 from 475136 to 876544 tid 2 2020-06-12T02:39:16.631 INFO:tasks.rados.rados.0.smithi107.stdout:11088: delete oid 6 current snap is 0 2020-06-12T02:39:16.631 INFO:tasks.rados.rados.0.smithi107.stdout:11082: finishing write tid 4 to smithi10729407-18 2020-06-12T02:39:16.631 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1884 (ObjNum 11081 snap 0 seq_num 11081) dirty exists 2020-06-12T02:39:16.632 INFO:tasks.rados.rados.0.smithi107.stdout:11082: left oid 18 (ObjNum 11081 snap 0 seq_num 11081) 2020-06-12T02:39:16.632 INFO:tasks.rados.rados.0.smithi107.stdout:11083: finishing write tid 5 to smithi10729407-8 2020-06-12T02:39:16.632 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 2058 (ObjNum 11082 snap 0 seq_num 11082) dirty exists 2020-06-12T02:39:16.632 INFO:tasks.rados.rados.0.smithi107.stdout:11083: left oid 8 (ObjNum 11082 snap 0 seq_num 11082) 2020-06-12T02:39:16.632 INFO:tasks.rados.rados.0.smithi107.stdout:11086: finishing write tid 1 to smithi10729407-13 2020-06-12T02:39:16.633 INFO:tasks.rados.rados.0.smithi107.stdout:11086: finishing write tid 2 to smithi10729407-13 2020-06-12T02:39:16.633 INFO:tasks.rados.rados.0.smithi107.stdout:11086: finishing write tid 4 to smithi10729407-13 2020-06-12T02:39:16.633 INFO:tasks.rados.rados.0.smithi107.stdout:11087: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:16.633 INFO:tasks.rados.rados.0.smithi107.stdout:11082: done (4 left) 2020-06-12T02:39:16.633 INFO:tasks.rados.rados.0.smithi107.stdout:11083: done (3 left) 2020-06-12T02:39:16.633 INFO:tasks.rados.rados.0.smithi107.stdout:11088: done (2 left) 2020-06-12T02:39:16.634 INFO:tasks.rados.rados.0.smithi107.stdout:11089: delete oid 1 current snap is 0 2020-06-12T02:39:16.634 INFO:tasks.rados.rados.0.smithi107.stdout:11086: finishing write tid 5 to smithi10729407-13 2020-06-12T02:39:16.634 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1789 (ObjNum 11085 snap 0 seq_num 11085) dirty exists 2020-06-12T02:39:16.634 INFO:tasks.rados.rados.0.smithi107.stdout:11086: left oid 13 (ObjNum 11085 snap 0 seq_num 11085) 2020-06-12T02:39:16.634 INFO:tasks.rados.rados.0.smithi107.stdout:11087: finishing write tid 2 to smithi10729407-10 2020-06-12T02:39:16.635 INFO:tasks.rados.rados.0.smithi107.stdout:11086: done (2 left) 2020-06-12T02:39:16.635 INFO:tasks.rados.rados.0.smithi107.stdout:11089: done (1 left) 2020-06-12T02:39:16.635 INFO:tasks.rados.rados.0.smithi107.stdout:11090: delete oid 8 current snap is 0 2020-06-12T02:39:16.635 INFO:tasks.rados.rados.0.smithi107.stdout:11087: finishing write tid 4 to smithi10729407-10 2020-06-12T02:39:16.635 INFO:tasks.rados.rados.0.smithi107.stdout:11087: finishing write tid 5 to smithi10729407-10 2020-06-12T02:39:16.635 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 819 (ObjNum 11086 snap 0 seq_num 11086) dirty exists 2020-06-12T02:39:16.636 INFO:tasks.rados.rados.0.smithi107.stdout:11087: left oid 10 (ObjNum 11086 snap 0 seq_num 11086) 2020-06-12T02:39:16.636 INFO:tasks.rados.rados.0.smithi107.stdout:11087: done (1 left) 2020-06-12T02:39:16.636 INFO:tasks.rados.rados.0.smithi107.stdout:11090: done (0 left) 2020-06-12T02:39:16.636 INFO:tasks.rados.rados.0.smithi107.stdout:11091: delete oid 8 current snap is 0 2020-06-12T02:39:16.636 INFO:tasks.rados.rados.0.smithi107.stdout:11091: done (0 left) 2020-06-12T02:39:16.637 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:39:16.637 INFO:tasks.rados.rados.0.smithi107.stdout:11092: seq_num 11091 ranges {0=557056,557056=401408} 2020-06-12T02:39:16.637 INFO:tasks.rados.rados.0.smithi107.stdout:11092: writing smithi10729407-7 from 0 to 557056 tid 1 2020-06-12T02:39:16.637 INFO:tasks.rados.rados.0.smithi107.stdout:11092: writing smithi10729407-7 from 557056 to 958464 tid 2 2020-06-12T02:39:16.637 INFO:tasks.rados.rados.0.smithi107.stdout:11093: delete oid 9 current snap is 0 2020-06-12T02:39:16.638 INFO:tasks.rados.rados.0.smithi107.stdout:11093: done (1 left) 2020-06-12T02:39:16.638 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:39:16.638 INFO:tasks.rados.rados.0.smithi107.stdout:11094: seq_num 11093 ranges {0=589824,589824=401408} 2020-06-12T02:39:16.638 INFO:tasks.rados.rados.0.smithi107.stdout:11094: writing smithi10729407-12 from 0 to 589824 tid 1 2020-06-12T02:39:16.638 INFO:tasks.rados.rados.0.smithi107.stdout:11094: writing smithi10729407-12 from 589824 to 991232 tid 2 2020-06-12T02:39:16.638 INFO:tasks.rados.rados.0.smithi107.stdout:11095: delete oid 4 current snap is 0 2020-06-12T02:39:16.639 INFO:tasks.rados.rados.0.smithi107.stdout:11092: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:16.639 INFO:tasks.rados.rados.0.smithi107.stdout:11092: finishing write tid 2 to smithi10729407-7 2020-06-12T02:39:16.639 INFO:tasks.rados.rados.0.smithi107.stdout:11092: finishing write tid 4 to smithi10729407-7 2020-06-12T02:39:16.639 INFO:tasks.rados.rados.0.smithi107.stdout:11092: finishing write tid 5 to smithi10729407-7 2020-06-12T02:39:16.639 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1805 (ObjNum 11091 snap 0 seq_num 11091) dirty exists 2020-06-12T02:39:16.639 INFO:tasks.rados.rados.0.smithi107.stdout:11092: left oid 7 (ObjNum 11091 snap 0 seq_num 11091) 2020-06-12T02:39:16.640 INFO:tasks.rados.rados.0.smithi107.stdout:11092: done (2 left) 2020-06-12T02:39:16.640 INFO:tasks.rados.rados.0.smithi107.stdout:11095: done (1 left) 2020-06-12T02:39:16.640 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:39:16.640 INFO:tasks.rados.rados.0.smithi107.stdout:11096: seq_num 11095 ranges {0=622592,622592=401408} 2020-06-12T02:39:16.640 INFO:tasks.rados.rados.0.smithi107.stdout:11096: writing smithi10729407-6 from 0 to 622592 tid 1 2020-06-12T02:39:16.641 INFO:tasks.rados.rados.0.smithi107.stdout:11096: writing smithi10729407-6 from 622592 to 1024000 tid 2 2020-06-12T02:39:16.641 INFO:tasks.rados.rados.0.smithi107.stdout:11097: delete oid 20 current snap is 0 2020-06-12T02:39:16.641 INFO:tasks.rados.rados.0.smithi107.stdout:11094: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:16.641 INFO:tasks.rados.rados.0.smithi107.stdout:11094: finishing write tid 2 to smithi10729407-12 2020-06-12T02:39:16.641 INFO:tasks.rados.rados.0.smithi107.stdout:11094: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:16.642 INFO:tasks.rados.rados.0.smithi107.stdout:11097: done (2 left) 2020-06-12T02:39:16.642 INFO:tasks.rados.rados.0.smithi107.stdout:11098: delete oid 15 current snap is 0 2020-06-12T02:39:16.642 INFO:tasks.rados.rados.0.smithi107.stdout:11094: finishing write tid 5 to smithi10729407-12 2020-06-12T02:39:16.642 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 2058 (ObjNum 11093 snap 0 seq_num 11093) dirty exists 2020-06-12T02:39:16.642 INFO:tasks.rados.rados.0.smithi107.stdout:11094: left oid 12 (ObjNum 11093 snap 0 seq_num 11093) 2020-06-12T02:39:16.642 INFO:tasks.rados.rados.0.smithi107.stdout:11096: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:16.643 INFO:tasks.rados.rados.0.smithi107.stdout:11094: done (2 left) 2020-06-12T02:39:16.643 INFO:tasks.rados.rados.0.smithi107.stdout:11098: done (1 left) 2020-06-12T02:39:16.643 INFO:tasks.rados.rados.0.smithi107.stdout:11099: delete oid 16 current snap is 0 2020-06-12T02:39:16.643 INFO:tasks.rados.rados.0.smithi107.stdout:11099: done (1 left) 2020-06-12T02:39:16.643 INFO:tasks.rados.rados.0.smithi107.stdout:11100: delete oid 1 current snap is 0 2020-06-12T02:39:16.643 INFO:tasks.rados.rados.0.smithi107.stdout:11096: finishing write tid 2 to smithi10729407-6 2020-06-12T02:39:16.644 INFO:tasks.rados.rados.0.smithi107.stdout:11096: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:16.644 INFO:tasks.rados.rados.0.smithi107.stdout:11096: finishing write tid 5 to smithi10729407-6 2020-06-12T02:39:16.644 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 2002 (ObjNum 11095 snap 0 seq_num 11095) dirty exists 2020-06-12T02:39:16.644 INFO:tasks.rados.rados.0.smithi107.stdout:11096: left oid 6 (ObjNum 11095 snap 0 seq_num 11095) 2020-06-12T02:39:16.644 INFO:tasks.rados.rados.0.smithi107.stdout:11096: done (1 left) 2020-06-12T02:39:16.645 INFO:tasks.rados.rados.0.smithi107.stdout:11100: done (0 left) 2020-06-12T02:39:16.645 INFO:tasks.rados.rados.0.smithi107.stdout:11101: delete oid 15 current snap is 0 2020-06-12T02:39:16.645 INFO:tasks.rados.rados.0.smithi107.stdout:11101: done (0 left) 2020-06-12T02:39:16.645 INFO:tasks.rados.rados.0.smithi107.stdout:11102: delete oid 11 current snap is 0 2020-06-12T02:39:16.645 INFO:tasks.rados.rados.0.smithi107.stdout:11102: done (0 left) 2020-06-12T02:39:16.645 INFO:tasks.rados.rados.0.smithi107.stdout:11103: delete oid 12 current snap is 0 2020-06-12T02:39:16.646 INFO:tasks.rados.rados.0.smithi107.stdout:11103: done (0 left) 2020-06-12T02:39:16.646 INFO:tasks.rados.rados.0.smithi107.stdout:11104: delete oid 12 current snap is 0 2020-06-12T02:39:16.646 INFO:tasks.rados.rados.0.smithi107.stdout:11104: done (0 left) 2020-06-12T02:39:16.646 INFO:tasks.rados.rados.0.smithi107.stdout:11105: delete oid 18 current snap is 0 2020-06-12T02:39:16.646 INFO:tasks.rados.rados.0.smithi107.stdout:11105: done (0 left) 2020-06-12T02:39:16.647 INFO:tasks.rados.rados.0.smithi107.stdout:11106: delete oid 15 current snap is 0 2020-06-12T02:39:16.647 INFO:tasks.rados.rados.0.smithi107.stdout:11106: done (0 left) 2020-06-12T02:39:16.647 INFO:tasks.rados.rados.0.smithi107.stdout:11107: delete oid 2 current snap is 0 2020-06-12T02:39:16.647 INFO:tasks.rados.rados.0.smithi107.stdout:11107: done (0 left) 2020-06-12T02:39:16.647 INFO:tasks.rados.rados.0.smithi107.stdout:11108: delete oid 4 current snap is 0 2020-06-12T02:39:16.647 INFO:tasks.rados.rados.0.smithi107.stdout:11108: done (0 left) 2020-06-12T02:39:16.648 INFO:tasks.rados.rados.0.smithi107.stdout:11109: delete oid 20 current snap is 0 2020-06-12T02:39:16.648 INFO:tasks.rados.rados.0.smithi107.stdout:11109: done (0 left) 2020-06-12T02:39:16.648 INFO:tasks.rados.rados.0.smithi107.stdout:11110: delete oid 1 current snap is 0 2020-06-12T02:39:16.648 INFO:teuthology.orchestra.run.smithi188.stdout:3.es0 2020-06-12T02:39:16.649 INFO:teuthology.orchestra.run.smithi188.stdout:3.9s2 2020-06-12T02:39:16.649 INFO:teuthology.orchestra.run.smithi188.stdout:3.fs0 2020-06-12T02:39:16.649 INFO:teuthology.orchestra.run.smithi188.stdout:3.8s2 2020-06-12T02:39:16.649 INFO:teuthology.orchestra.run.smithi188.stdout:3.7s2 2020-06-12T02:39:16.649 INFO:teuthology.orchestra.run.smithi188.stdout:3.16s1 2020-06-12T02:39:16.650 INFO:teuthology.orchestra.run.smithi188.stdout:3.15s0 2020-06-12T02:39:16.650 INFO:teuthology.orchestra.run.smithi188.stdout:3.14s1 2020-06-12T02:39:16.650 INFO:teuthology.orchestra.run.smithi188.stdout:3.13s0 2020-06-12T02:39:16.650 INFO:teuthology.orchestra.run.smithi188.stdout:3.ds0 2020-06-12T02:39:16.650 INFO:teuthology.orchestra.run.smithi188.stdout:3.12s1 2020-06-12T02:39:16.650 INFO:teuthology.orchestra.run.smithi188.stdout:3.11s0 2020-06-12T02:39:16.651 INFO:teuthology.orchestra.run.smithi188.stdout:2.7 2020-06-12T02:39:16.651 INFO:teuthology.orchestra.run.smithi188.stdout:2.0 2020-06-12T02:39:16.651 INFO:teuthology.orchestra.run.smithi188.stdout:2.5 2020-06-12T02:39:16.651 INFO:teuthology.orchestra.run.smithi188.stdout:1.0 2020-06-12T02:39:16.651 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:39:16.605+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:39:16.652 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-06-12T02:39:16.658 INFO:tasks.rados.rados.0.smithi107.stdout:11110: done (0 left) 2020-06-12T02:39:16.659 INFO:tasks.rados.rados.0.smithi107.stdout:11111: delete oid 20 current snap is 0 2020-06-12T02:39:16.659 INFO:tasks.rados.rados.0.smithi107.stdout:11111: done (0 left) 2020-06-12T02:39:16.659 INFO:tasks.rados.rados.0.smithi107.stdout:11112: delete oid 11 current snap is 0 2020-06-12T02:39:16.659 INFO:tasks.rados.rados.0.smithi107.stdout:11112: done (0 left) 2020-06-12T02:39:16.659 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:39:16.660 INFO:tasks.rados.rados.0.smithi107.stdout:11113: seq_num 11112 ranges {876544=507904,1384448=483328,1867776=319488} 2020-06-12T02:39:16.660 INFO:tasks.rados.rados.0.smithi107.stdout:11113: writing smithi10729407-10 from 876544 to 1384448 tid 1 2020-06-12T02:39:16.660 INFO:tasks.rados.rados.0.smithi107.stdout:11113: writing smithi10729407-10 from 1384448 to 1867776 tid 2 2020-06-12T02:39:16.660 INFO:tasks.rados.rados.0.smithi107.stdout:11113: writing smithi10729407-10 from 1867776 to 2187264 tid 3 2020-06-12T02:39:16.660 INFO:tasks.rados.rados.0.smithi107.stdout:11114: delete oid 1 current snap is 0 2020-06-12T02:39:16.660 INFO:tasks.rados.rados.0.smithi107.stdout:11113: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:16.661 INFO:tasks.rados.rados.0.smithi107.stdout:11114: done (1 left) 2020-06-12T02:39:16.661 INFO:tasks.rados.rados.0.smithi107.stdout:11115: delete oid 6 current snap is 0 2020-06-12T02:39:16.661 INFO:tasks.rados.rados.0.smithi107.stdout:11113: finishing write tid 2 to smithi10729407-10 2020-06-12T02:39:16.661 INFO:tasks.rados.rados.0.smithi107.stdout:11115: done (1 left) 2020-06-12T02:39:16.662 INFO:tasks.rados.rados.0.smithi107.stdout:11116: delete oid 11 current snap is 0 2020-06-12T02:39:16.662 INFO:tasks.rados.rados.0.smithi107.stdout:11113: finishing write tid 3 to smithi10729407-10 2020-06-12T02:39:16.662 INFO:tasks.rados.rados.0.smithi107.stdout:11113: finishing write tid 5 to smithi10729407-10 2020-06-12T02:39:16.662 INFO:tasks.rados.rados.0.smithi107.stdout:11116: done (1 left) 2020-06-12T02:39:16.662 INFO:tasks.rados.rados.0.smithi107.stdout:11117: delete oid 12 current snap is 0 2020-06-12T02:39:16.663 INFO:tasks.rados.rados.0.smithi107.stdout:11113: finishing write tid 6 to smithi10729407-10 2020-06-12T02:39:16.664 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 823 (ObjNum 11112 snap 0 seq_num 11112) dirty exists 2020-06-12T02:39:16.664 INFO:tasks.rados.rados.0.smithi107.stdout:11113: left oid 10 (ObjNum 11112 snap 0 seq_num 11112) 2020-06-12T02:39:16.664 INFO:tasks.rados.rados.0.smithi107.stdout:11113: done (1 left) 2020-06-12T02:39:16.664 INFO:tasks.rados.rados.0.smithi107.stdout:11117: done (0 left) 2020-06-12T02:39:16.664 INFO:tasks.rados.rados.0.smithi107.stdout:11118: delete oid 10 current snap is 0 2020-06-12T02:39:16.665 INFO:tasks.rados.rados.0.smithi107.stdout:11118: done (0 left) 2020-06-12T02:39:16.665 INFO:tasks.rados.rados.0.smithi107.stdout:11119: delete oid 8 current snap is 0 2020-06-12T02:39:16.665 INFO:tasks.rados.rados.0.smithi107.stdout:11119: done (0 left) 2020-06-12T02:39:16.665 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:39:16.666 INFO:tasks.rados.rados.0.smithi107.stdout:11120: seq_num 11119 ranges {0=630784,630784=524288,1155072=278528} 2020-06-12T02:39:16.666 INFO:tasks.rados.rados.0.smithi107.stdout:11120: writing smithi10729407-18 from 0 to 630784 tid 1 2020-06-12T02:39:16.667 INFO:tasks.rados.rados.0.smithi107.stdout:11120: writing smithi10729407-18 from 630784 to 1155072 tid 2 2020-06-12T02:39:16.667 INFO:tasks.rados.rados.0.smithi107.stdout:11120: writing smithi10729407-18 from 1155072 to 1433600 tid 3 2020-06-12T02:39:16.668 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:39:16.668 INFO:tasks.rados.rados.0.smithi107.stdout:11121: seq_num 11120 ranges {0=647168,647168=409600,1056768=393216} 2020-06-12T02:39:16.668 INFO:tasks.rados.rados.0.smithi107.stdout:11121: writing smithi10729407-14 from 0 to 647168 tid 1 2020-06-12T02:39:16.668 INFO:tasks.rados.rados.0.smithi107.stdout:11121: writing smithi10729407-14 from 647168 to 1056768 tid 2 2020-06-12T02:39:16.669 INFO:tasks.rados.rados.0.smithi107.stdout:11121: writing smithi10729407-14 from 1056768 to 1449984 tid 3 2020-06-12T02:39:16.669 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:39:16.670 INFO:tasks.rados.rados.0.smithi107.stdout:11122: seq_num 11121 ranges {0=663552,663552=696320,1359872=106496} 2020-06-12T02:39:16.672 INFO:tasks.rados.rados.0.smithi107.stdout:11122: writing smithi10729407-19 from 0 to 663552 tid 1 2020-06-12T02:39:16.677 INFO:tasks.rados.rados.0.smithi107.stdout:11122: writing smithi10729407-19 from 663552 to 1359872 tid 2 2020-06-12T02:39:16.678 INFO:tasks.rados.rados.0.smithi107.stdout:11122: writing smithi10729407-19 from 1359872 to 1466368 tid 3 2020-06-12T02:39:16.678 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:39:16.678 INFO:tasks.rados.rados.0.smithi107.stdout:11123: seq_num 11122 ranges {0=679936,679936=581632,1261568=221184} 2020-06-12T02:39:16.682 INFO:tasks.rados.rados.0.smithi107.stdout:11123: writing smithi10729407-9 from 0 to 679936 tid 1 2020-06-12T02:39:16.685 INFO:tasks.rados.rados.0.smithi107.stdout:11123: writing smithi10729407-9 from 679936 to 1261568 tid 2 2020-06-12T02:39:16.686 INFO:tasks.rados.rados.0.smithi107.stdout:11123: writing smithi10729407-9 from 1261568 to 1482752 tid 3 2020-06-12T02:39:16.687 INFO:tasks.rados.rados.0.smithi107.stdout:11124: delete oid 3 current snap is 0 2020-06-12T02:39:16.687 INFO:tasks.rados.rados.0.smithi107.stdout:11120: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:16.687 INFO:tasks.rados.rados.0.smithi107.stdout:11120: finishing write tid 2 to smithi10729407-18 2020-06-12T02:39:16.687 INFO:tasks.rados.rados.0.smithi107.stdout:11120: finishing write tid 3 to smithi10729407-18 2020-06-12T02:39:16.687 INFO:tasks.rados.rados.0.smithi107.stdout:11120: finishing write tid 5 to smithi10729407-18 2020-06-12T02:39:16.688 INFO:tasks.rados.rados.0.smithi107.stdout:11120: finishing write tid 6 to smithi10729407-18 2020-06-12T02:39:16.688 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1889 (ObjNum 11119 snap 0 seq_num 11119) dirty exists 2020-06-12T02:39:16.688 INFO:tasks.rados.rados.0.smithi107.stdout:11120: left oid 18 (ObjNum 11119 snap 0 seq_num 11119) 2020-06-12T02:39:16.688 INFO:tasks.rados.rados.0.smithi107.stdout:11121: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:16.688 INFO:tasks.rados.rados.0.smithi107.stdout:11121: finishing write tid 2 to smithi10729407-14 2020-06-12T02:39:16.689 INFO:tasks.rados.rados.0.smithi107.stdout:11121: finishing write tid 3 to smithi10729407-14 2020-06-12T02:39:16.689 INFO:tasks.rados.rados.0.smithi107.stdout:11121: finishing write tid 5 to smithi10729407-14 2020-06-12T02:39:16.689 INFO:tasks.rados.rados.0.smithi107.stdout:11121: finishing write tid 6 to smithi10729407-14 2020-06-12T02:39:16.689 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1744 (ObjNum 11120 snap 0 seq_num 11120) dirty exists 2020-06-12T02:39:16.689 INFO:tasks.rados.rados.0.smithi107.stdout:11121: left oid 14 (ObjNum 11120 snap 0 seq_num 11120) 2020-06-12T02:39:16.690 INFO:tasks.rados.rados.0.smithi107.stdout:11122: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:16.690 INFO:tasks.rados.rados.0.smithi107.stdout:11122: finishing write tid 2 to smithi10729407-19 2020-06-12T02:39:16.690 INFO:tasks.rados.rados.0.smithi107.stdout:11122: finishing write tid 3 to smithi10729407-19 2020-06-12T02:39:16.692 INFO:tasks.rados.rados.0.smithi107.stdout:11122: finishing write tid 5 to smithi10729407-19 2020-06-12T02:39:16.694 INFO:tasks.rados.rados.0.smithi107.stdout:11120: done (4 left) 2020-06-12T02:39:16.694 INFO:tasks.rados.rados.0.smithi107.stdout:11121: done (3 left) 2020-06-12T02:39:16.694 INFO:tasks.rados.rados.0.smithi107.stdout:11124: done (2 left) 2020-06-12T02:39:16.695 INFO:tasks.rados.rados.0.smithi107.stdout:11125: delete oid 5 current snap is 0 2020-06-12T02:39:16.695 INFO:tasks.rados.rados.0.smithi107.stdout:11123: finishing write tid 1 to smithi10729407-9 2020-06-12T02:39:16.695 INFO:tasks.rados.rados.0.smithi107.stdout:11123: finishing write tid 2 to smithi10729407-9 2020-06-12T02:39:16.696 INFO:tasks.rados.rados.0.smithi107.stdout:11123: finishing write tid 3 to smithi10729407-9 2020-06-12T02:39:16.696 INFO:tasks.rados.rados.0.smithi107.stdout:11123: finishing write tid 5 to smithi10729407-9 2020-06-12T02:39:16.697 INFO:tasks.rados.rados.0.smithi107.stdout:11122: finishing write tid 6 to smithi10729407-19 2020-06-12T02:39:16.698 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 2065 (ObjNum 11121 snap 0 seq_num 11121) dirty exists 2020-06-12T02:39:16.698 INFO:tasks.rados.rados.0.smithi107.stdout:11122: left oid 19 (ObjNum 11121 snap 0 seq_num 11121) 2020-06-12T02:39:16.698 INFO:tasks.rados.rados.0.smithi107.stdout:11122: done (2 left) 2020-06-12T02:39:16.698 INFO:tasks.rados.rados.0.smithi107.stdout:11125: done (1 left) 2020-06-12T02:39:16.698 INFO:tasks.rados.rados.0.smithi107.stdout:11126: delete oid 3 current snap is 0 2020-06-12T02:39:16.699 INFO:tasks.rados.rados.0.smithi107.stdout:11123: finishing write tid 6 to smithi10729407-9 2020-06-12T02:39:16.699 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1855 (ObjNum 11122 snap 0 seq_num 11122) dirty exists 2020-06-12T02:39:16.700 INFO:tasks.rados.rados.0.smithi107.stdout:11123: left oid 9 (ObjNum 11122 snap 0 seq_num 11122) 2020-06-12T02:39:16.700 INFO:tasks.rados.rados.0.smithi107.stdout:11123: done (1 left) 2020-06-12T02:39:16.700 INFO:tasks.rados.rados.0.smithi107.stdout:11126: done (0 left) 2020-06-12T02:39:16.700 INFO:tasks.rados.rados.0.smithi107.stdout:11127: delete oid 3 current snap is 0 2020-06-12T02:39:16.702 INFO:tasks.rados.rados.0.smithi107.stdout:11127: done (0 left) 2020-06-12T02:39:16.702 INFO:tasks.rados.rados.0.smithi107.stdout:11128: delete oid 3 current snap is 0 2020-06-12T02:39:16.704 INFO:tasks.rados.rados.0.smithi107.stdout:11128: done (0 left) 2020-06-12T02:39:16.705 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:39:16.705 INFO:tasks.rados.rados.0.smithi107.stdout:11129: seq_num 11128 ranges {0=778240,778240=737280,1515520=65536} 2020-06-12T02:39:16.708 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:39:16.705+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:39:16.709 INFO:tasks.rados.rados.0.smithi107.stdout:11129: writing smithi10729407-11 from 0 to 778240 tid 1 2020-06-12T02:39:16.714 INFO:tasks.rados.rados.0.smithi107.stdout:11129: writing smithi10729407-11 from 778240 to 1515520 tid 2 2020-06-12T02:39:16.715 INFO:tasks.rados.rados.0.smithi107.stdout:11129: writing smithi10729407-11 from 1515520 to 1581056 tid 3 2020-06-12T02:39:16.715 INFO:tasks.rados.rados.0.smithi107.stdout:11130: delete oid 5 current snap is 0 2020-06-12T02:39:16.718 INFO:tasks.rados.rados.0.smithi107.stdout:11130: done (1 left) 2020-06-12T02:39:16.718 INFO:tasks.rados.rados.0.smithi107.stdout:11131: delete oid 10 current snap is 0 2020-06-12T02:39:16.720 INFO:tasks.rados.rados.0.smithi107.stdout:11131: done (1 left) 2020-06-12T02:39:16.720 INFO:tasks.rados.rados.0.smithi107.stdout:11132: delete oid 10 current snap is 0 2020-06-12T02:39:16.722 INFO:tasks.rados.rados.0.smithi107.stdout:11129: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:16.723 INFO:tasks.rados.rados.0.smithi107.stdout:11132: done (1 left) 2020-06-12T02:39:16.723 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:39:16.723 INFO:tasks.rados.rados.0.smithi107.stdout:11133: seq_num 11132 ranges {1449984=442368,1892352=720896,2613248=483328} 2020-06-12T02:39:16.734 INFO:tasks.rados.rados.0.smithi107.stdout:11133: writing smithi10729407-14 from 1449984 to 1892352 tid 1 2020-06-12T02:39:16.742 INFO:tasks.rados.rados.0.smithi107.stdout:11133: writing smithi10729407-14 from 1892352 to 2613248 tid 2 2020-06-12T02:39:16.742 INFO:tasks.rados.rados.0.smithi107.stdout:11133: writing smithi10729407-14 from 2613248 to 3096576 tid 3 2020-06-12T02:39:16.742 INFO:tasks.rados.rados.0.smithi107.stdout:11134: delete oid 2 current snap is 0 2020-06-12T02:39:16.743 INFO:tasks.rados.rados.0.smithi107.stdout:11129: finishing write tid 2 to smithi10729407-11 2020-06-12T02:39:16.743 INFO:tasks.rados.rados.0.smithi107.stdout:11129: finishing write tid 3 to smithi10729407-11 2020-06-12T02:39:16.743 INFO:tasks.rados.rados.0.smithi107.stdout:11129: finishing write tid 5 to smithi10729407-11 2020-06-12T02:39:16.743 INFO:tasks.rados.rados.0.smithi107.stdout:11129: finishing write tid 6 to smithi10729407-11 2020-06-12T02:39:16.743 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1775 (ObjNum 11128 snap 0 seq_num 11128) dirty exists 2020-06-12T02:39:16.744 INFO:tasks.rados.rados.0.smithi107.stdout:11129: left oid 11 (ObjNum 11128 snap 0 seq_num 11128) 2020-06-12T02:39:16.744 INFO:tasks.rados.rados.0.smithi107.stdout:11129: done (2 left) 2020-06-12T02:39:16.745 INFO:tasks.rados.rados.0.smithi107.stdout:11134: done (1 left) 2020-06-12T02:39:16.745 INFO:tasks.rados.rados.0.smithi107.stdout:11135: delete oid 20 current snap is 0 2020-06-12T02:39:16.745 INFO:tasks.rados.rados.0.smithi107.stdout:11133: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:16.747 INFO:tasks.rados.rados.0.smithi107.stdout:11135: done (1 left) 2020-06-12T02:39:16.748 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:39:16.748 INFO:tasks.rados.rados.0.smithi107.stdout:11136: seq_num 11135 ranges {1433600=491520,1925120=770048,2695168=434176} 2020-06-12T02:39:16.758 INFO:tasks.rados.rados.0.smithi107.stdout:11136: writing smithi10729407-18 from 1433600 to 1925120 tid 1 2020-06-12T02:39:16.762 INFO:tasks.rados.rados.0.smithi107.stdout:11136: writing smithi10729407-18 from 1925120 to 2695168 tid 2 2020-06-12T02:39:16.766 INFO:tasks.rados.rados.0.smithi107.stdout:11136: writing smithi10729407-18 from 2695168 to 3129344 tid 3 2020-06-12T02:39:16.766 INFO:tasks.rados.rados.0.smithi107.stdout:11137: delete oid 20 current snap is 0 2020-06-12T02:39:16.766 INFO:tasks.rados.rados.0.smithi107.stdout:11133: finishing write tid 2 to smithi10729407-14 2020-06-12T02:39:16.766 INFO:tasks.rados.rados.0.smithi107.stdout:11133: finishing write tid 3 to smithi10729407-14 2020-06-12T02:39:16.767 INFO:tasks.rados.rados.0.smithi107.stdout:11133: finishing write tid 5 to smithi10729407-14 2020-06-12T02:39:16.767 INFO:tasks.rados.rados.0.smithi107.stdout:11133: finishing write tid 6 to smithi10729407-14 2020-06-12T02:39:16.767 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1748 (ObjNum 11132 snap 0 seq_num 11132) dirty exists 2020-06-12T02:39:16.767 INFO:tasks.rados.rados.0.smithi107.stdout:11133: left oid 14 (ObjNum 11132 snap 0 seq_num 11132) 2020-06-12T02:39:16.767 INFO:tasks.rados.rados.0.smithi107.stdout:11136: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:16.768 INFO:tasks.rados.rados.0.smithi107.stdout:11133: done (2 left) 2020-06-12T02:39:16.768 INFO:tasks.rados.rados.0.smithi107.stdout:11137: done (1 left) 2020-06-12T02:39:16.769 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:39:16.769 INFO:tasks.rados.rados.0.smithi107.stdout:11138: seq_num 11137 ranges {0=532480,532480=540672,1073152=450560,1523712=212992} 2020-06-12T02:39:16.772 INFO:tasks.rados.rados.0.smithi107.stdout:11138: writing smithi10729407-2 from 0 to 532480 tid 1 2020-06-12T02:39:16.776 INFO:tasks.rados.rados.0.smithi107.stdout:11138: writing smithi10729407-2 from 532480 to 1073152 tid 2 2020-06-12T02:39:16.778 INFO:tasks.rados.rados.0.smithi107.stdout:11138: writing smithi10729407-2 from 1073152 to 1523712 tid 3 2020-06-12T02:39:16.780 INFO:tasks.rados.rados.0.smithi107.stdout:11138: writing smithi10729407-2 from 1523712 to 1736704 tid 4 2020-06-12T02:39:16.780 INFO:tasks.rados.rados.0.smithi107.stdout:11139: delete oid 12 current snap is 0 2020-06-12T02:39:16.781 INFO:tasks.rados.rados.0.smithi107.stdout:11136: finishing write tid 2 to smithi10729407-18 2020-06-12T02:39:16.781 INFO:tasks.rados.rados.0.smithi107.stdout:11136: finishing write tid 3 to smithi10729407-18 2020-06-12T02:39:16.781 INFO:tasks.rados.rados.0.smithi107.stdout:11136: finishing write tid 5 to smithi10729407-18 2020-06-12T02:39:16.781 INFO:tasks.rados.rados.0.smithi107.stdout:11136: finishing write tid 6 to smithi10729407-18 2020-06-12T02:39:16.781 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1893 (ObjNum 11135 snap 0 seq_num 11135) dirty exists 2020-06-12T02:39:16.782 INFO:tasks.rados.rados.0.smithi107.stdout:11136: left oid 18 (ObjNum 11135 snap 0 seq_num 11135) 2020-06-12T02:39:16.782 INFO:tasks.rados.rados.0.smithi107.stdout:11138: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:16.785 INFO:tasks.rados.rados.0.smithi107.stdout:11136: done (2 left) 2020-06-12T02:39:16.785 INFO:tasks.rados.rados.0.smithi107.stdout:11139: done (1 left) 2020-06-12T02:39:16.786 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:39:16.786 INFO:tasks.rados.rados.0.smithi107.stdout:11140: seq_num 11139 ranges {3096576=565248,3661824=761856,4423680=417792,4841472=24576} 2020-06-12T02:39:16.806 INFO:tasks.rados.rados.0.smithi107.stdout:11140: writing smithi10729407-14 from 3096576 to 3661824 tid 1 2020-06-12T02:39:16.809 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:39:16.806+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:39:16.810 INFO:tasks.rados.rados.0.smithi107.stdout:11140: writing smithi10729407-14 from 3661824 to 4423680 tid 2 2020-06-12T02:39:16.812 INFO:tasks.rados.rados.0.smithi107.stdout:11140: writing smithi10729407-14 from 4423680 to 4841472 tid 3 2020-06-12T02:39:16.813 INFO:tasks.rados.rados.0.smithi107.stdout:11140: writing smithi10729407-14 from 4841472 to 4866048 tid 4 2020-06-12T02:39:16.813 INFO:tasks.rados.rados.0.smithi107.stdout:11141: delete oid 13 current snap is 0 2020-06-12T02:39:16.813 INFO:tasks.rados.rados.0.smithi107.stdout:11138: finishing write tid 2 to smithi10729407-2 2020-06-12T02:39:16.813 INFO:tasks.rados.rados.0.smithi107.stdout:11138: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:16.813 INFO:tasks.rados.rados.0.smithi107.stdout:11138: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:16.814 INFO:tasks.rados.rados.0.smithi107.stdout:11138: finishing write tid 6 to smithi10729407-2 2020-06-12T02:39:16.814 INFO:tasks.rados.rados.0.smithi107.stdout:11138: finishing write tid 7 to smithi10729407-2 2020-06-12T02:39:16.814 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 1050 (ObjNum 11137 snap 0 seq_num 11137) dirty exists 2020-06-12T02:39:16.814 INFO:tasks.rados.rados.0.smithi107.stdout:11138: left oid 2 (ObjNum 11137 snap 0 seq_num 11137) 2020-06-12T02:39:16.815 INFO:tasks.rados.rados.0.smithi107.stdout:11138: done (2 left) 2020-06-12T02:39:16.815 INFO:tasks.rados.rados.0.smithi107.stdout:11141: done (1 left) 2020-06-12T02:39:16.815 INFO:tasks.rados.rados.0.smithi107.stdout:11142: delete oid 4 current snap is 0 2020-06-12T02:39:16.817 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-objectstore-tool --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op export --pgid 2.5 --file /var/log/ceph/exp.2.5.7 2020-06-12T02:39:16.819 INFO:tasks.rados.rados.0.smithi107.stdout:11142: done (1 left) 2020-06-12T02:39:16.819 INFO:tasks.rados.rados.0.smithi107.stdout:11143: delete oid 10 current snap is 0 2020-06-12T02:39:16.820 INFO:tasks.rados.rados.0.smithi107.stdout:11140: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:16.822 INFO:tasks.rados.rados.0.smithi107.stdout:11140: finishing write tid 2 to smithi10729407-14 2020-06-12T02:39:16.822 INFO:tasks.rados.rados.0.smithi107.stdout:11140: finishing write tid 3 to smithi10729407-14 2020-06-12T02:39:16.822 INFO:tasks.rados.rados.0.smithi107.stdout:11140: finishing write tid 4 to smithi10729407-14 2020-06-12T02:39:16.822 INFO:tasks.rados.rados.0.smithi107.stdout:11140: finishing write tid 6 to smithi10729407-14 2020-06-12T02:39:16.822 INFO:tasks.rados.rados.0.smithi107.stdout:11143: done (1 left) 2020-06-12T02:39:16.823 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:39:16.823 INFO:tasks.rados.rados.0.smithi107.stdout:11144: seq_num 11143 ranges {958464=630784,1589248=696320,2285568=507904} 2020-06-12T02:39:16.830 INFO:tasks.rados.rados.0.smithi107.stdout:11144: writing smithi10729407-7 from 958464 to 1589248 tid 1 2020-06-12T02:39:16.830 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:39:16.830 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:39:16.831 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2020-06-12T02:39:16.831 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:39:16.834 INFO:tasks.rados.rados.0.smithi107.stdout:11144: writing smithi10729407-7 from 1589248 to 2285568 tid 2 2020-06-12T02:39:16.842 INFO:tasks.rados.rados.0.smithi107.stdout:11144: writing smithi10729407-7 from 2285568 to 2793472 tid 3 2020-06-12T02:39:16.842 INFO:tasks.rados.rados.0.smithi107.stdout:11145: delete oid 20 current snap is 0 2020-06-12T02:39:16.842 INFO:tasks.rados.rados.0.smithi107.stdout:11140: finishing write tid 7 to smithi10729407-14 2020-06-12T02:39:16.843 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1753 (ObjNum 11139 snap 0 seq_num 11139) dirty exists 2020-06-12T02:39:16.843 INFO:tasks.rados.rados.0.smithi107.stdout:11140: left oid 14 (ObjNum 11139 snap 0 seq_num 11139) 2020-06-12T02:39:16.843 INFO:tasks.rados.rados.0.smithi107.stdout:11140: done (2 left) 2020-06-12T02:39:16.843 INFO:tasks.rados.rados.0.smithi107.stdout:11145: done (1 left) 2020-06-12T02:39:16.843 INFO:tasks.rados.rados.0.smithi107.stdout:11146: delete oid 13 current snap is 0 2020-06-12T02:39:16.844 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-06-12T02:39:16.845 INFO:tasks.rados.rados.0.smithi107.stdout:11146: done (1 left) 2020-06-12T02:39:16.845 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:39:16.845 INFO:tasks.rados.rados.0.smithi107.stdout:11147: seq_num 11146 ranges {0=679936,679936=802816,1482752=401408} 2020-06-12T02:39:16.848 INFO:tasks.rados.rados.0.smithi107.stdout:11147: writing smithi10729407-4 from 0 to 679936 tid 1 2020-06-12T02:39:16.852 INFO:tasks.rados.rados.0.smithi107.stdout:11147: writing smithi10729407-4 from 679936 to 1482752 tid 2 2020-06-12T02:39:16.867 INFO:tasks.rados.rados.0.smithi107.stdout:11147: writing smithi10729407-4 from 1482752 to 1884160 tid 3 2020-06-12T02:39:16.867 INFO:tasks.rados.rados.0.smithi107.stdout:11148: delete oid 8 current snap is 0 2020-06-12T02:39:16.867 INFO:tasks.rados.rados.0.smithi107.stdout:11144: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:16.868 INFO:tasks.rados.rados.0.smithi107.stdout:11144: finishing write tid 2 to smithi10729407-7 2020-06-12T02:39:16.868 INFO:tasks.rados.rados.0.smithi107.stdout:11144: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:16.868 INFO:tasks.rados.rados.0.smithi107.stdout:11144: finishing write tid 5 to smithi10729407-7 2020-06-12T02:39:16.868 INFO:tasks.rados.rados.0.smithi107.stdout:11144: finishing write tid 6 to smithi10729407-7 2020-06-12T02:39:16.868 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1809 (ObjNum 11143 snap 0 seq_num 11143) dirty exists 2020-06-12T02:39:16.868 INFO:tasks.rados.rados.0.smithi107.stdout:11144: left oid 7 (ObjNum 11143 snap 0 seq_num 11143) 2020-06-12T02:39:16.869 INFO:tasks.rados.rados.0.smithi107.stdout:11147: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:16.869 INFO:tasks.rados.rados.0.smithi107.stdout:11144: done (2 left) 2020-06-12T02:39:16.869 INFO:tasks.rados.rados.0.smithi107.stdout:11148: done (1 left) 2020-06-12T02:39:16.869 INFO:tasks.rados.rados.0.smithi107.stdout:11149: delete oid 13 current snap is 0 2020-06-12T02:39:16.869 INFO:tasks.rados.rados.0.smithi107.stdout:11149: done (1 left) 2020-06-12T02:39:16.870 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:39:16.870 INFO:tasks.rados.rados.0.smithi107.stdout:11150: seq_num 11149 ranges {3129344=729088,3858432=450560,4308992=753664} 2020-06-12T02:39:16.882 INFO:tasks.rados.rados.0.smithi107.stdout:11150: writing smithi10729407-18 from 3129344 to 3858432 tid 1 2020-06-12T02:39:16.885 INFO:tasks.rados.rados.0.smithi107.stdout:11150: writing smithi10729407-18 from 3858432 to 4308992 tid 2 2020-06-12T02:39:16.891 INFO:tasks.rados.rados.0.smithi107.stdout:11150: writing smithi10729407-18 from 4308992 to 5062656 tid 3 2020-06-12T02:39:16.891 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:39:16.891 INFO:tasks.rados.rados.0.smithi107.stdout:11151: seq_num 11150 ranges {0=745472,745472=737280,1482752=466944} 2020-06-12T02:39:16.896 INFO:tasks.rados.rados.0.smithi107.stdout:11151: writing smithi10729407-1 from 0 to 745472 tid 1 2020-06-12T02:39:16.901 INFO:tasks.rados.rados.0.smithi107.stdout:11151: writing smithi10729407-1 from 745472 to 1482752 tid 2 2020-06-12T02:39:16.904 INFO:tasks.rados.rados.0.smithi107.stdout:11151: writing smithi10729407-1 from 1482752 to 1949696 tid 3 2020-06-12T02:39:16.904 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:39:16.904 INFO:tasks.rados.rados.0.smithi107.stdout:11152: seq_num 11151 ranges {0=761856,761856=622592,1384448=581632} 2020-06-12T02:39:16.909 INFO:tasks.rados.rados.0.smithi107.stdout:11152: writing smithi10729407-3 from 0 to 761856 tid 1 2020-06-12T02:39:16.910 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:39:16.907+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:39:16.912 INFO:tasks.rados.rados.0.smithi107.stdout:11152: writing smithi10729407-3 from 761856 to 1384448 tid 2 2020-06-12T02:39:16.916 INFO:tasks.rados.rados.0.smithi107.stdout:11152: writing smithi10729407-3 from 1384448 to 1966080 tid 3 2020-06-12T02:39:16.916 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:39:16.916 INFO:tasks.rados.rados.0.smithi107.stdout:11153: seq_num 11152 ranges {1736704=778240,2514944=507904,3022848=565248,3588096=131072} 2020-06-12T02:39:16.930 INFO:tasks.rados.rados.0.smithi107.stdout:11153: writing smithi10729407-2 from 1736704 to 2514944 tid 1 2020-06-12T02:39:16.934 INFO:tasks.rados.rados.0.smithi107.stdout:11153: writing smithi10729407-2 from 2514944 to 3022848 tid 2 2020-06-12T02:39:16.938 INFO:tasks.rados.rados.0.smithi107.stdout:11153: writing smithi10729407-2 from 3022848 to 3588096 tid 3 2020-06-12T02:39:16.939 INFO:tasks.rados.rados.0.smithi107.stdout:11153: writing smithi10729407-2 from 3588096 to 3719168 tid 4 2020-06-12T02:39:16.939 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:39:16.939 INFO:tasks.rados.rados.0.smithi107.stdout:11154: seq_num 11153 ranges {1581056=794624,2375680=442368,2818048=761856} 2020-06-12T02:39:16.953 INFO:tasks.rados.rados.0.smithi107.stdout:11154: writing smithi10729407-11 from 1581056 to 2375680 tid 1 2020-06-12T02:39:16.955 INFO:tasks.rados.rados.0.smithi107.stdout:11154: writing smithi10729407-11 from 2375680 to 2818048 tid 2 2020-06-12T02:39:16.959 INFO:tasks.rados.rados.0.smithi107.stdout:11154: writing smithi10729407-11 from 2818048 to 3579904 tid 3 2020-06-12T02:39:16.960 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:39:16.960 INFO:tasks.rados.rados.0.smithi107.stdout:11155: seq_num 11154 ranges {4866048=409600,5275648=720896,5996544=581632,6578176=303104} 2020-06-12T02:39:16.989 INFO:tasks.rados.rados.0.smithi107.stdout:11155: writing smithi10729407-14 from 4866048 to 5275648 tid 1 2020-06-12T02:39:16.990 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:39:16.991 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:39:16.991 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2020-06-12T02:39:16.991 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2020-06-12T02:39:16.992 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:39:16.993 INFO:tasks.rados.rados.0.smithi107.stdout:11155: writing smithi10729407-14 from 5275648 to 5996544 tid 2 2020-06-12T02:39:16.996 INFO:tasks.rados.rados.0.smithi107.stdout:11155: writing smithi10729407-14 from 5996544 to 6578176 tid 3 2020-06-12T02:39:16.998 INFO:tasks.rados.rados.0.smithi107.stdout:11155: writing smithi10729407-14 from 6578176 to 6881280 tid 4 2020-06-12T02:39:16.999 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:39:16.999 INFO:tasks.rados.rados.0.smithi107.stdout:11156: seq_num 11155 ranges {0=425984,425984=606208,1032192=737280,1769472=262144} 2020-06-12T02:39:17.000 INFO:tasks.rados.rados.0.smithi107.stdout:11156: writing smithi10729407-8 from 0 to 425984 tid 1 2020-06-12T02:39:17.002 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-06-12T02:39:17.004 INFO:tasks.rados.rados.0.smithi107.stdout:11156: writing smithi10729407-8 from 425984 to 1032192 tid 2 2020-06-12T02:39:17.009 INFO:tasks.rados.rados.0.smithi107.stdout:11156: writing smithi10729407-8 from 1032192 to 1769472 tid 3 2020-06-12T02:39:17.020 INFO:tasks.rados.rados.0.smithi107.stdout:11156: writing smithi10729407-8 from 1769472 to 2031616 tid 4 2020-06-12T02:39:17.020 INFO:tasks.rados.rados.0.smithi107.stdout: waiting on 8 2020-06-12T02:39:17.020 INFO:tasks.rados.rados.0.smithi107.stdout:11147: finishing write tid 2 to smithi10729407-4 2020-06-12T02:39:17.020 INFO:tasks.rados.rados.0.smithi107.stdout:11147: finishing write tid 3 to smithi10729407-4 2020-06-12T02:39:17.021 INFO:tasks.rados.rados.0.smithi107.stdout:11147: finishing write tid 5 to smithi10729407-4 2020-06-12T02:39:17.021 INFO:tasks.rados.rados.0.smithi107.stdout:11147: finishing write tid 6 to smithi10729407-4 2020-06-12T02:39:17.021 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 2087 (ObjNum 11146 snap 0 seq_num 11146) dirty exists 2020-06-12T02:39:17.022 INFO:tasks.rados.rados.0.smithi107.stdout:11147: left oid 4 (ObjNum 11146 snap 0 seq_num 11146) 2020-06-12T02:39:17.022 INFO:tasks.rados.rados.0.smithi107.stdout:11150: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:17.022 INFO:tasks.rados.rados.0.smithi107.stdout:11147: done (7 left) 2020-06-12T02:39:17.022 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:39:17.022 INFO:tasks.rados.rados.0.smithi107.stdout:11157: seq_num 11156 ranges {0=450560,450560=491520,942080=548864,1490944=565248} 2020-06-12T02:39:17.023 INFO:tasks.rados.rados.0.smithi107.stdout:11157: writing smithi10729407-20 from 0 to 450560 tid 1 2020-06-12T02:39:17.023 INFO:tasks.rados.rados.0.smithi107.stdout:11157: writing smithi10729407-20 from 450560 to 942080 tid 2 2020-06-12T02:39:17.023 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:39:17.016+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:39:17.024 INFO:tasks.rados.rados.0.smithi107.stdout:11157: writing smithi10729407-20 from 942080 to 1490944 tid 3 2020-06-12T02:39:17.026 INFO:tasks.rados.rados.0.smithi107.stdout:11157: writing smithi10729407-20 from 1490944 to 2056192 tid 4 2020-06-12T02:39:17.026 INFO:tasks.rados.rados.0.smithi107.stdout: waiting on 8 2020-06-12T02:39:17.026 INFO:tasks.rados.rados.0.smithi107.stdout:11150: finishing write tid 2 to smithi10729407-18 2020-06-12T02:39:17.026 INFO:tasks.rados.rados.0.smithi107.stdout:11150: finishing write tid 3 to smithi10729407-18 2020-06-12T02:39:17.027 INFO:tasks.rados.rados.0.smithi107.stdout:11150: finishing write tid 5 to smithi10729407-18 2020-06-12T02:39:17.027 INFO:tasks.rados.rados.0.smithi107.stdout:11150: finishing write tid 6 to smithi10729407-18 2020-06-12T02:39:17.027 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1897 (ObjNum 11149 snap 0 seq_num 11149) dirty exists 2020-06-12T02:39:17.027 INFO:tasks.rados.rados.0.smithi107.stdout:11150: left oid 18 (ObjNum 11149 snap 0 seq_num 11149) 2020-06-12T02:39:17.029 INFO:tasks.rados.rados.0.smithi107.stdout:11151: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:17.030 INFO:tasks.rados.rados.0.smithi107.stdout:11151: finishing write tid 2 to smithi10729407-1 2020-06-12T02:39:17.030 INFO:tasks.rados.rados.0.smithi107.stdout:11151: finishing write tid 3 to smithi10729407-1 2020-06-12T02:39:17.030 INFO:tasks.rados.rados.0.smithi107.stdout:11151: finishing write tid 5 to smithi10729407-1 2020-06-12T02:39:17.030 INFO:tasks.rados.rados.0.smithi107.stdout:11151: finishing write tid 6 to smithi10729407-1 2020-06-12T02:39:17.031 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 2010 (ObjNum 11150 snap 0 seq_num 11150) dirty exists 2020-06-12T02:39:17.031 INFO:tasks.rados.rados.0.smithi107.stdout:11151: left oid 1 (ObjNum 11150 snap 0 seq_num 11150) 2020-06-12T02:39:17.031 INFO:tasks.rados.rados.0.smithi107.stdout:11152: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:17.031 INFO:tasks.rados.rados.0.smithi107.stdout:11152: finishing write tid 2 to smithi10729407-3 2020-06-12T02:39:17.031 INFO:tasks.rados.rados.0.smithi107.stdout:11152: finishing write tid 3 to smithi10729407-3 2020-06-12T02:39:17.032 INFO:tasks.rados.rados.0.smithi107.stdout:11152: finishing write tid 5 to smithi10729407-3 2020-06-12T02:39:17.033 INFO:tasks.rados.rados.0.smithi107.stdout:11152: finishing write tid 6 to smithi10729407-3 2020-06-12T02:39:17.033 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1966 (ObjNum 11151 snap 0 seq_num 11151) dirty exists 2020-06-12T02:39:17.033 INFO:tasks.rados.rados.0.smithi107.stdout:11152: left oid 3 (ObjNum 11151 snap 0 seq_num 11151) 2020-06-12T02:39:17.033 INFO:tasks.rados.rados.0.smithi107.stdout:11153: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:17.033 INFO:tasks.rados.rados.0.smithi107.stdout:11153: finishing write tid 2 to smithi10729407-2 2020-06-12T02:39:17.034 INFO:tasks.rados.rados.0.smithi107.stdout:11153: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:17.034 INFO:tasks.rados.rados.0.smithi107.stdout:11153: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:17.034 INFO:tasks.rados.rados.0.smithi107.stdout:11153: finishing write tid 6 to smithi10729407-2 2020-06-12T02:39:17.034 INFO:tasks.rados.rados.0.smithi107.stdout:11153: finishing write tid 7 to smithi10729407-2 2020-06-12T02:39:17.034 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 1055 (ObjNum 11152 snap 0 seq_num 11152) dirty exists 2020-06-12T02:39:17.034 INFO:tasks.rados.rados.0.smithi107.stdout:11153: left oid 2 (ObjNum 11152 snap 0 seq_num 11152) 2020-06-12T02:39:17.035 INFO:tasks.rados.rados.0.smithi107.stdout:11154: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:17.035 INFO:tasks.rados.rados.0.smithi107.stdout:11154: finishing write tid 2 to smithi10729407-11 2020-06-12T02:39:17.035 INFO:tasks.rados.rados.0.smithi107.stdout:11154: finishing write tid 3 to smithi10729407-11 2020-06-12T02:39:17.036 INFO:tasks.rados.rados.0.smithi107.stdout:11154: finishing write tid 5 to smithi10729407-11 2020-06-12T02:39:17.036 INFO:tasks.rados.rados.0.smithi107.stdout:11154: finishing write tid 6 to smithi10729407-11 2020-06-12T02:39:17.036 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1779 (ObjNum 11153 snap 0 seq_num 11153) dirty exists 2020-06-12T02:39:17.036 INFO:tasks.rados.rados.0.smithi107.stdout:11154: left oid 11 (ObjNum 11153 snap 0 seq_num 11153) 2020-06-12T02:39:17.036 INFO:tasks.rados.rados.0.smithi107.stdout:11155: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:17.037 INFO:tasks.rados.rados.0.smithi107.stdout:11155: finishing write tid 2 to smithi10729407-14 2020-06-12T02:39:17.037 INFO:tasks.rados.rados.0.smithi107.stdout:11155: finishing write tid 3 to smithi10729407-14 2020-06-12T02:39:17.037 INFO:tasks.rados.rados.0.smithi107.stdout:11150: done (7 left) 2020-06-12T02:39:17.037 INFO:tasks.rados.rados.0.smithi107.stdout:11151: done (6 left) 2020-06-12T02:39:17.038 INFO:tasks.rados.rados.0.smithi107.stdout:11152: done (5 left) 2020-06-12T02:39:17.038 INFO:tasks.rados.rados.0.smithi107.stdout:11153: done (4 left) 2020-06-12T02:39:17.038 INFO:tasks.rados.rados.0.smithi107.stdout:11154: done (3 left) 2020-06-12T02:39:17.038 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:39:17.038 INFO:tasks.rados.rados.0.smithi107.stdout:11155: finishing write tid 4 to smithi10729407-14 2020-06-12T02:39:17.038 INFO:tasks.rados.rados.0.smithi107.stdout:11155: finishing write tid 6 to smithi10729407-14 2020-06-12T02:39:17.039 INFO:tasks.rados.rados.0.smithi107.stdout:11155: finishing write tid 7 to smithi10729407-14 2020-06-12T02:39:17.039 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1758 (ObjNum 11154 snap 0 seq_num 11154) dirty exists 2020-06-12T02:39:17.039 INFO:tasks.rados.rados.0.smithi107.stdout:11155: left oid 14 (ObjNum 11154 snap 0 seq_num 11154) 2020-06-12T02:39:17.041 INFO:tasks.rados.rados.0.smithi107.stdout:11156: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:17.041 INFO:tasks.rados.rados.0.smithi107.stdout:11158: seq_num 11157 ranges {0=466944,466944=778240,1245184=753664,1998848=73728} 2020-06-12T02:39:17.043 INFO:tasks.rados.rados.0.smithi107.stdout:11158: writing smithi10729407-13 from 0 to 466944 tid 1 2020-06-12T02:39:17.043 INFO:tasks.rados.rados.0.smithi107.stdout:11158: writing smithi10729407-13 from 466944 to 1245184 tid 2 2020-06-12T02:39:17.043 INFO:tasks.rados.rados.0.smithi107.stdout:11158: writing smithi10729407-13 from 1245184 to 1998848 tid 3 2020-06-12T02:39:17.043 INFO:tasks.rados.rados.0.smithi107.stdout:11158: writing smithi10729407-13 from 1998848 to 2072576 tid 4 2020-06-12T02:39:17.043 INFO:tasks.rados.rados.0.smithi107.stdout:11155: done (3 left) 2020-06-12T02:39:17.044 INFO:tasks.rados.rados.0.smithi107.stdout:11159: delete oid 7 current snap is 0 2020-06-12T02:39:17.044 INFO:tasks.rados.rados.0.smithi107.stdout:11156: finishing write tid 2 to smithi10729407-8 2020-06-12T02:39:17.044 INFO:tasks.rados.rados.0.smithi107.stdout:11156: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:17.044 INFO:tasks.rados.rados.0.smithi107.stdout:11156: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:17.045 INFO:tasks.rados.rados.0.smithi107.stdout:11156: finishing write tid 6 to smithi10729407-8 2020-06-12T02:39:17.045 INFO:tasks.rados.rados.0.smithi107.stdout:11156: finishing write tid 7 to smithi10729407-8 2020-06-12T02:39:17.045 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 2070 (ObjNum 11155 snap 0 seq_num 11155) dirty exists 2020-06-12T02:39:17.045 INFO:tasks.rados.rados.0.smithi107.stdout:11156: left oid 8 (ObjNum 11155 snap 0 seq_num 11155) 2020-06-12T02:39:17.045 INFO:tasks.rados.rados.0.smithi107.stdout:11157: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:17.046 INFO:tasks.rados.rados.0.smithi107.stdout:11157: finishing write tid 2 to smithi10729407-20 2020-06-12T02:39:17.046 INFO:tasks.rados.rados.0.smithi107.stdout:11157: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:17.046 INFO:tasks.rados.rados.0.smithi107.stdout:11157: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:17.046 INFO:tasks.rados.rados.0.smithi107.stdout:11157: finishing write tid 6 to smithi10729407-20 2020-06-12T02:39:17.046 INFO:tasks.rados.rados.0.smithi107.stdout:11158: finishing write tid 1 to smithi10729407-13 2020-06-12T02:39:17.047 INFO:tasks.rados.rados.0.smithi107.stdout:11157: finishing write tid 7 to smithi10729407-20 2020-06-12T02:39:17.047 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 863 (ObjNum 11156 snap 0 seq_num 11156) dirty exists 2020-06-12T02:39:17.047 INFO:tasks.rados.rados.0.smithi107.stdout:11157: left oid 20 (ObjNum 11156 snap 0 seq_num 11156) 2020-06-12T02:39:17.048 INFO:tasks.rados.rados.0.smithi107.stdout:11158: finishing write tid 2 to smithi10729407-13 2020-06-12T02:39:17.048 INFO:tasks.rados.rados.0.smithi107.stdout:11156: done (3 left) 2020-06-12T02:39:17.049 INFO:tasks.rados.rados.0.smithi107.stdout:11157: done (2 left) 2020-06-12T02:39:17.049 INFO:tasks.rados.rados.0.smithi107.stdout:11159: done (1 left) 2020-06-12T02:39:17.049 INFO:tasks.rados.rados.0.smithi107.stdout:11160: delete oid 14 current snap is 0 2020-06-12T02:39:17.049 INFO:tasks.rados.rados.0.smithi107.stdout:11158: finishing write tid 3 to smithi10729407-13 2020-06-12T02:39:17.049 INFO:tasks.rados.rados.0.smithi107.stdout:11158: finishing write tid 4 to smithi10729407-13 2020-06-12T02:39:17.050 INFO:tasks.rados.rados.0.smithi107.stdout:11158: finishing write tid 6 to smithi10729407-13 2020-06-12T02:39:17.050 INFO:tasks.rados.rados.0.smithi107.stdout:11160: done (1 left) 2020-06-12T02:39:17.051 INFO:tasks.rados.rados.0.smithi107.stdout:11161: delete oid 16 current snap is 0 2020-06-12T02:39:17.052 INFO:tasks.rados.rados.0.smithi107.stdout:11158: finishing write tid 7 to smithi10729407-13 2020-06-12T02:39:17.052 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1797 (ObjNum 11157 snap 0 seq_num 11157) dirty exists 2020-06-12T02:39:17.052 INFO:tasks.rados.rados.0.smithi107.stdout:11158: left oid 13 (ObjNum 11157 snap 0 seq_num 11157) 2020-06-12T02:39:17.052 INFO:tasks.rados.rados.0.smithi107.stdout:11158: done (1 left) 2020-06-12T02:39:17.052 INFO:tasks.rados.rados.0.smithi107.stdout:11161: done (0 left) 2020-06-12T02:39:17.053 INFO:tasks.rados.rados.0.smithi107.stdout:11162: delete oid 10 current snap is 0 2020-06-12T02:39:17.056 INFO:tasks.rados.rados.0.smithi107.stdout:11162: done (0 left) 2020-06-12T02:39:17.056 INFO:tasks.rados.rados.0.smithi107.stdout:11163: delete oid 11 current snap is 0 2020-06-12T02:39:17.059 INFO:tasks.rados.rados.0.smithi107.stdout:11163: done (0 left) 2020-06-12T02:39:17.059 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:39:17.060 INFO:tasks.rados.rados.0.smithi107.stdout:11164: seq_num 11163 ranges {2056192=565248,2621440=532480,3153920=696320,3850240=376832} 2020-06-12T02:39:17.075 INFO:tasks.rados.rados.0.smithi107.stdout:11164: writing smithi10729407-20 from 2056192 to 2621440 tid 1 2020-06-12T02:39:17.078 INFO:tasks.rados.rados.0.smithi107.stdout:11164: writing smithi10729407-20 from 2621440 to 3153920 tid 2 2020-06-12T02:39:17.083 INFO:tasks.rados.rados.0.smithi107.stdout:11164: writing smithi10729407-20 from 3153920 to 3850240 tid 3 2020-06-12T02:39:17.086 INFO:tasks.rados.rados.0.smithi107.stdout:11164: writing smithi10729407-20 from 3850240 to 4227072 tid 4 2020-06-12T02:39:17.087 INFO:tasks.rados.rados.0.smithi107.stdout:11165: delete oid 2 current snap is 0 2020-06-12T02:39:17.087 INFO:tasks.rados.rados.0.smithi107.stdout:11164: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:17.087 INFO:tasks.rados.rados.0.smithi107.stdout:11164: finishing write tid 2 to smithi10729407-20 2020-06-12T02:39:17.091 INFO:tasks.rados.rados.0.smithi107.stdout:11165: done (1 left) 2020-06-12T02:39:17.092 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:39:17.092 INFO:tasks.rados.rados.0.smithi107.stdout:11166: seq_num 11165 ranges {0=598016,598016=704512,1302528=712704,2015232=188416} 2020-06-12T02:39:17.095 INFO:tasks.rados.rados.0.smithi107.stdout:11166: writing smithi10729407-10 from 0 to 598016 tid 1 2020-06-12T02:39:17.101 INFO:tasks.rados.rados.0.smithi107.stdout:11166: writing smithi10729407-10 from 598016 to 1302528 tid 2 2020-06-12T02:39:17.105 INFO:tasks.rados.rados.0.smithi107.stdout:11166: writing smithi10729407-10 from 1302528 to 2015232 tid 3 2020-06-12T02:39:17.106 INFO:tasks.rados.rados.0.smithi107.stdout:11166: writing smithi10729407-10 from 2015232 to 2203648 tid 4 2020-06-12T02:39:17.107 INFO:tasks.rados.rados.0.smithi107.stdout:11167: delete oid 8 current snap is 0 2020-06-12T02:39:17.107 INFO:tasks.rados.rados.0.smithi107.stdout:11164: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:17.107 INFO:tasks.rados.rados.0.smithi107.stdout:11164: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:17.107 INFO:tasks.rados.rados.0.smithi107.stdout:11164: finishing write tid 6 to smithi10729407-20 2020-06-12T02:39:17.107 INFO:tasks.rados.rados.0.smithi107.stdout:11164: finishing write tid 7 to smithi10729407-20 2020-06-12T02:39:17.108 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 868 (ObjNum 11163 snap 0 seq_num 11163) dirty exists 2020-06-12T02:39:17.108 INFO:tasks.rados.rados.0.smithi107.stdout:11164: left oid 20 (ObjNum 11163 snap 0 seq_num 11163) 2020-06-12T02:39:17.108 INFO:tasks.rados.rados.0.smithi107.stdout:11166: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:17.110 INFO:tasks.rados.rados.0.smithi107.stdout:11164: done (2 left) 2020-06-12T02:39:17.110 INFO:tasks.rados.rados.0.smithi107.stdout:11167: done (1 left) 2020-06-12T02:39:17.111 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:39:17.111 INFO:tasks.rados.rados.0.smithi107.stdout:11168: seq_num 11167 ranges {0=630784,630784=466944,1097728=679936,1777664=458752} 2020-06-12T02:39:17.114 INFO:tasks.rados.rados.0.smithi107.stdout:11168: writing smithi10729407-12 from 0 to 630784 tid 1 2020-06-12T02:39:17.117 INFO:tasks.rados.rados.0.smithi107.stdout:11168: writing smithi10729407-12 from 630784 to 1097728 tid 2 2020-06-12T02:39:17.119 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:39:17.116+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:39:17.121 INFO:tasks.rados.rados.0.smithi107.stdout:11168: writing smithi10729407-12 from 1097728 to 1777664 tid 3 2020-06-12T02:39:17.124 INFO:tasks.rados.rados.0.smithi107.stdout:11168: writing smithi10729407-12 from 1777664 to 2236416 tid 4 2020-06-12T02:39:17.124 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:39:17.124 INFO:tasks.rados.rados.0.smithi107.stdout:11169: seq_num 11168 ranges {4227072=647168,4874240=802816,5677056=483328,6160384=319488} 2020-06-12T02:39:17.151 INFO:tasks.rados.rados.0.smithi107.stdout:11169: writing smithi10729407-20 from 4227072 to 4874240 tid 1 2020-06-12T02:39:17.156 INFO:tasks.rados.rados.0.smithi107.stdout:11169: writing smithi10729407-20 from 4874240 to 5677056 tid 2 2020-06-12T02:39:17.159 INFO:tasks.rados.rados.0.smithi107.stdout:11169: writing smithi10729407-20 from 5677056 to 6160384 tid 3 2020-06-12T02:39:17.161 INFO:tasks.rados.rados.0.smithi107.stdout:11169: writing smithi10729407-20 from 6160384 to 6479872 tid 4 2020-06-12T02:39:17.161 INFO:tasks.rados.rados.0.smithi107.stdout:11170: delete oid 16 current snap is 0 2020-06-12T02:39:17.162 INFO:tasks.rados.rados.0.smithi107.stdout:11166: finishing write tid 2 to smithi10729407-10 2020-06-12T02:39:17.162 INFO:tasks.rados.rados.0.smithi107.stdout:11166: finishing write tid 3 to smithi10729407-10 2020-06-12T02:39:17.162 INFO:tasks.rados.rados.0.smithi107.stdout:11166: finishing write tid 4 to smithi10729407-10 2020-06-12T02:39:17.163 INFO:tasks.rados.rados.0.smithi107.stdout:11166: finishing write tid 6 to smithi10729407-10 2020-06-12T02:39:17.163 INFO:tasks.rados.rados.0.smithi107.stdout:11166: finishing write tid 7 to smithi10729407-10 2020-06-12T02:39:17.163 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 833 (ObjNum 11165 snap 0 seq_num 11165) dirty exists 2020-06-12T02:39:17.164 INFO:tasks.rados.rados.0.smithi107.stdout:11166: left oid 10 (ObjNum 11165 snap 0 seq_num 11165) 2020-06-12T02:39:17.164 INFO:tasks.rados.rados.0.smithi107.stdout:11168: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:17.164 INFO:tasks.rados.rados.0.smithi107.stdout:11168: finishing write tid 2 to smithi10729407-12 2020-06-12T02:39:17.164 INFO:tasks.rados.rados.0.smithi107.stdout:11168: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:17.165 INFO:tasks.rados.rados.0.smithi107.stdout:11168: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:17.165 INFO:tasks.rados.rados.0.smithi107.stdout:11168: finishing write tid 6 to smithi10729407-12 2020-06-12T02:39:17.165 INFO:tasks.rados.rados.0.smithi107.stdout:11168: finishing write tid 7 to smithi10729407-12 2020-06-12T02:39:17.165 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 2071 (ObjNum 11167 snap 0 seq_num 11167) dirty exists 2020-06-12T02:39:17.165 INFO:tasks.rados.rados.0.smithi107.stdout:11168: left oid 12 (ObjNum 11167 snap 0 seq_num 11167) 2020-06-12T02:39:17.166 INFO:tasks.rados.rados.0.smithi107.stdout:11169: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:17.166 INFO:tasks.rados.rados.0.smithi107.stdout:11166: done (3 left) 2020-06-12T02:39:17.166 INFO:tasks.rados.rados.0.smithi107.stdout:11168: done (2 left) 2020-06-12T02:39:17.167 INFO:tasks.rados.rados.0.smithi107.stdout:11170: done (1 left) 2020-06-12T02:39:17.167 INFO:tasks.rados.rados.0.smithi107.stdout:11171: delete oid 15 current snap is 0 2020-06-12T02:39:17.169 INFO:tasks.rados.rados.0.smithi107.stdout:11171: done (1 left) 2020-06-12T02:39:17.169 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:39:17.169 INFO:tasks.rados.rados.0.smithi107.stdout:11172: seq_num 11171 ranges {0=696320,696320=458752,1155072=663552,1818624=483328} 2020-06-12T02:39:17.172 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:39:17.172 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2020-06-12T02:39:17.172 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2020-06-12T02:39:17.172 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2020-06-12T02:39:17.172 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.173 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14657 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~606208] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:39:17.173 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.209225+0000", 2020-06-12T02:39:17.173 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 45.957964885999999, 2020-06-12T02:39:17.173 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016128905999999998, 2020-06-12T02:39:17.173 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:17.174 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:17.174 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:17.174 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:17.174 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:17.174 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14657 2020-06-12T02:39:17.175 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.175 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:17.175 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.175 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:17.175 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209225+0000", 2020-06-12T02:39:17.176 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:17.176 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.176 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.176 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:17.176 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209225+0000", 2020-06-12T02:39:17.176 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018386699999999999 2020-06-12T02:39:17.177 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.177 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.177 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:17.177 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209409+0000", 2020-06-12T02:39:17.177 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8160000000000001e-06 2020-06-12T02:39:17.178 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.178 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.178 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:17.178 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209414+0000", 2020-06-12T02:39:17.178 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0230000000000004e-06 2020-06-12T02:39:17.179 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.179 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.179 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:17.179 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209421+0000", 2020-06-12T02:39:17.179 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0139999999999999e-06 2020-06-12T02:39:17.179 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.180 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.180 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:17.180 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209424+0000", 2020-06-12T02:39:17.180 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0054798169999999997 2020-06-12T02:39:17.180 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.181 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.181 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:17.181 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.214904+0000", 2020-06-12T02:39:17.181 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123063 2020-06-12T02:39:17.181 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.182 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.182 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:17.182 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215027+0000", 2020-06-12T02:39:17.182 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001159923 2020-06-12T02:39:17.182 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.183 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.183 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:17.184 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.216187+0000", 2020-06-12T02:39:17.184 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2835000000000002e-05 2020-06-12T02:39:17.184 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.184 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.185 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:17.185 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.216270+0000", 2020-06-12T02:39:17.185 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073575799999999998 2020-06-12T02:39:17.186 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.186 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.186 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:17.186 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.217005+0000", 2020-06-12T02:39:17.186 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0082434659999999996 2020-06-12T02:39:17.187 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.187 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.187 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:17.187 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225249+0000", 2020-06-12T02:39:17.187 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0145999999999997e-05 2020-06-12T02:39:17.187 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.188 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.188 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:39:17.188 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225299+0000", 2020-06-12T02:39:17.188 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5177999999999999e-05 2020-06-12T02:39:17.188 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.189 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.189 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:17.189 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225354+0000", 2020-06-12T02:39:17.189 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016128905999999998 2020-06-12T02:39:17.189 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:17.190 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:17.190 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:17.190 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.190 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.190 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14658 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~598016] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:39:17.190 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.209455+0000", 2020-06-12T02:39:17.191 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 45.957735132000003, 2020-06-12T02:39:17.191 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016447966000000001, 2020-06-12T02:39:17.191 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:17.191 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:17.191 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:17.192 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:17.192 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:17.192 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14658 2020-06-12T02:39:17.192 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.192 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:17.192 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.193 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:17.193 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209455+0000", 2020-06-12T02:39:17.193 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:17.193 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.193 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.194 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:17.194 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209455+0000", 2020-06-12T02:39:17.194 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037921599999999998 2020-06-12T02:39:17.194 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.194 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.195 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:17.195 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209834+0000", 2020-06-12T02:39:17.195 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8339999999999999e-06 2020-06-12T02:39:17.195 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.195 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.195 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:17.196 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209842+0000", 2020-06-12T02:39:17.196 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2564e-05 2020-06-12T02:39:17.196 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.196 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.196 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:17.197 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209855+0000", 2020-06-12T02:39:17.197 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.135e-06 2020-06-12T02:39:17.197 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.197 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.197 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:17.198 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209860+0000", 2020-06-12T02:39:17.198 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0052530290000000002 2020-06-12T02:39:17.198 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.198 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.198 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:17.198 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215113+0000", 2020-06-12T02:39:17.199 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6229000000000003e-05 2020-06-12T02:39:17.199 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.199 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.199 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:17.199 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215209+0000", 2020-06-12T02:39:17.200 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045765409999999999 2020-06-12T02:39:17.200 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.200 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.200 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:17.200 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.219785+0000", 2020-06-12T02:39:17.200 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000102846 2020-06-12T02:39:17.201 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.201 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.201 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:17.201 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.219888+0000", 2020-06-12T02:39:17.202 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00077088700000000005 2020-06-12T02:39:17.202 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.202 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.203 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:17.204 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.220659+0000", 2020-06-12T02:39:17.204 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046979739999999997 2020-06-12T02:39:17.204 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.204 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.204 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:17.205 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225357+0000", 2020-06-12T02:39:17.205 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00046069600000000002 2020-06-12T02:39:17.205 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.205 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.206 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:39:17.206 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225818+0000", 2020-06-12T02:39:17.206 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5014999999999998e-05 2020-06-12T02:39:17.206 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.206 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.207 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:17.207 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225903+0000", 2020-06-12T02:39:17.207 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016447966000000001 2020-06-12T02:39:17.208 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:17.208 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:17.208 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:17.208 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.208 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.209 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14659 3.0s0 3:0dddb690:::smithi10729407-10:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:39:17.209 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.209909+0000", 2020-06-12T02:39:17.209 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 45.957280924000003, 2020-06-12T02:39:17.209 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016618236000000002, 2020-06-12T02:39:17.210 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:17.210 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:17.210 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:17.210 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:17.211 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:17.211 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14659 2020-06-12T02:39:17.211 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.211 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:17.211 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.212 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:17.212 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209909+0000", 2020-06-12T02:39:17.212 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:17.212 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.212 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.213 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:17.213 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209909+0000", 2020-06-12T02:39:17.213 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8085999999999999e-05 2020-06-12T02:39:17.214 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.214 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.214 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:17.214 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209967+0000", 2020-06-12T02:39:17.215 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3169999999999998e-06 2020-06-12T02:39:17.215 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.215 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.215 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:17.215 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209974+0000", 2020-06-12T02:39:17.216 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1246999999999999e-05 2020-06-12T02:39:17.216 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.216 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.216 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:17.216 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209985+0000", 2020-06-12T02:39:17.217 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1439999999999996e-06 2020-06-12T02:39:17.217 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.217 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.217 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:17.218 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209989+0000", 2020-06-12T02:39:17.218 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0052537900000000004 2020-06-12T02:39:17.218 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.218 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.219 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:17.219 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215243+0000", 2020-06-12T02:39:17.219 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6023000000000002e-05 2020-06-12T02:39:17.219 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.219 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.220 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:17.220 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215299+0000", 2020-06-12T02:39:17.220 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0087846910000000007 2020-06-12T02:39:17.220 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.220 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.220 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:17.221 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.224083+0000", 2020-06-12T02:39:17.221 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8504000000000004e-05 2020-06-12T02:39:17.221 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.221 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.221 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:17.222 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.224182+0000", 2020-06-12T02:39:17.222 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044251400000000003 2020-06-12T02:39:17.222 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.222 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.223 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:17.223 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.224624+0000", 2020-06-12T02:39:17.223 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013383290000000001 2020-06-12T02:39:17.223 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.224 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.224 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:17.224 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225963+0000", 2020-06-12T02:39:17.224 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047942699999999999 2020-06-12T02:39:17.224 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.225 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.225 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:39:17.225 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.226442+0000", 2020-06-12T02:39:17.225 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5164e-05 2020-06-12T02:39:17.226 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.226 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.226 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:17.226 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.226527+0000", 2020-06-12T02:39:17.226 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016618236000000002 2020-06-12T02:39:17.227 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:17.227 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:17.227 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:17.227 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.227 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.228 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14660 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e59)", 2020-06-12T02:39:17.228 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.210020+0000", 2020-06-12T02:39:17.228 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 45.957169946, 2020-06-12T02:39:17.228 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017984777, 2020-06-12T02:39:17.228 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:17.229 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:39:17.229 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:17.229 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:17.229 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:17.229 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14660 2020-06-12T02:39:17.230 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.230 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:17.230 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.230 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:17.230 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210020+0000", 2020-06-12T02:39:17.231 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:17.231 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.231 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.231 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:17.231 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210020+0000", 2020-06-12T02:39:17.232 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2875999999999999e-05 2020-06-12T02:39:17.232 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.232 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.232 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:17.233 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210043+0000", 2020-06-12T02:39:17.233 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2200000000000001e-06 2020-06-12T02:39:17.233 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.233 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.233 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:17.234 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210046+0000", 2020-06-12T02:39:17.234 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7410000000000003e-06 2020-06-12T02:39:17.234 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.234 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.235 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:17.235 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210053+0000", 2020-06-12T02:39:17.235 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5610000000000001e-06 2020-06-12T02:39:17.235 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.235 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.236 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:17.236 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.210055+0000", 2020-06-12T02:39:17.236 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0052596409999999998 2020-06-12T02:39:17.236 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.236 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.236 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:17.237 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215315+0000", 2020-06-12T02:39:17.237 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5627999999999997e-05 2020-06-12T02:39:17.237 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.237 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.237 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:17.238 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.215371+0000", 2020-06-12T02:39:17.238 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0098069549999999991 2020-06-12T02:39:17.238 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.238 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.238 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:17.239 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225178+0000", 2020-06-12T02:39:17.239 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4530000000000001e-05 2020-06-12T02:39:17.239 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.239 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.239 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:17.240 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.225232+0000", 2020-06-12T02:39:17.240 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017497350000000001 2020-06-12T02:39:17.240 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.240 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.240 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:17.241 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.226982+0000", 2020-06-12T02:39:17.241 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8154999999999998e-05 2020-06-12T02:39:17.241 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.241 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.241 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:17.242 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.227080+0000", 2020-06-12T02:39:17.242 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00092473499999999999 2020-06-12T02:39:17.242 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.242 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.242 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:17.242 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.228005+0000", 2020-06-12T02:39:17.243 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017984777 2020-06-12T02:39:17.243 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:17.243 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:17.243 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:17.243 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.244 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.244 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.3s2 59/53 ECSubWrite(tid=5970, reqid=client.4512.0:15226, at_version=59'1316, trim_to=0'0, roll_forward_to=59'1315))", 2020-06-12T02:39:17.244 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.829184+0000", 2020-06-12T02:39:17.244 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 44.338005729999999, 2020-06-12T02:39:17.245 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.081295588000000002, 2020-06-12T02:39:17.245 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:17.245 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:17.245 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:17.245 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.246 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:17.246 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829184+0000", 2020-06-12T02:39:17.246 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:17.246 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.247 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.247 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:17.247 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829184+0000", 2020-06-12T02:39:17.247 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00023814999999999999 2020-06-12T02:39:17.247 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.248 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.248 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:17.248 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829422+0000", 2020-06-12T02:39:17.248 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9768000000000001e-05 2020-06-12T02:39:17.248 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.249 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.249 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:17.249 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829442+0000", 2020-06-12T02:39:17.249 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1022000000000001e-05 2020-06-12T02:39:17.249 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.250 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.250 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:17.250 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829453+0000", 2020-06-12T02:39:17.250 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2759999999999997e-06 2020-06-12T02:39:17.250 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.251 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.251 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:17.251 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829460+0000", 2020-06-12T02:39:17.251 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7284000000000002e-05 2020-06-12T02:39:17.251 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.252 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.252 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:17.252 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829517+0000", 2020-06-12T02:39:17.252 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7028999999999999e-05 2020-06-12T02:39:17.252 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.253 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.253 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:17.253 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.829534+0000", 2020-06-12T02:39:17.253 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.080899873999999997 2020-06-12T02:39:17.253 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.254 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.254 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:17.254 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.910434+0000", 2020-06-12T02:39:17.254 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6184999999999999e-05 2020-06-12T02:39:17.255 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.255 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.255 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:17.255 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.910480+0000", 2020-06-12T02:39:17.255 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.081295588000000002 2020-06-12T02:39:17.256 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:17.256 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:17.256 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:17.256 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.257 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.257 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15277 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~720896] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:39:17.257 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:33.067109+0000", 2020-06-12T02:39:17.257 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 44.100081127000003, 2020-06-12T02:39:17.257 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015909191999999999, 2020-06-12T02:39:17.258 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:17.258 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:17.258 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:17.258 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:17.258 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:17.259 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15277 2020-06-12T02:39:17.259 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.259 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:17.259 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.259 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:17.260 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067109+0000", 2020-06-12T02:39:17.260 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:17.261 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.261 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.261 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:17.261 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067109+0000", 2020-06-12T02:39:17.262 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018431400000000001 2020-06-12T02:39:17.262 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.262 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.262 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:17.262 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067293+0000", 2020-06-12T02:39:17.262 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1790000000000002e-06 2020-06-12T02:39:17.263 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.263 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.263 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:17.263 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067300+0000", 2020-06-12T02:39:17.263 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5259999999999999e-06 2020-06-12T02:39:17.264 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.264 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.264 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:17.264 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067310+0000", 2020-06-12T02:39:17.265 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8179999999999997e-06 2020-06-12T02:39:17.265 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.265 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.265 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:17.265 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067314+0000", 2020-06-12T02:39:17.266 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6529999999999997e-05 2020-06-12T02:39:17.266 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.266 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.266 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:17.267 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067360+0000", 2020-06-12T02:39:17.267 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012724399999999999 2020-06-12T02:39:17.267 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.267 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.267 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:17.268 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.067488+0000", 2020-06-12T02:39:17.268 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0010983989999999999 2020-06-12T02:39:17.268 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.268 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.268 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:17.271 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.068586+0000", 2020-06-12T02:39:17.271 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5165000000000001e-05 2020-06-12T02:39:17.271 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.272 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.272 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:17.272 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.068681+0000", 2020-06-12T02:39:17.272 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086510899999999997 2020-06-12T02:39:17.272 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.272 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.273 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:17.273 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.069546+0000", 2020-06-12T02:39:17.273 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013284429 2020-06-12T02:39:17.273 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.273 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.274 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:17.274 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.082831+0000", 2020-06-12T02:39:17.274 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4197000000000006e-05 2020-06-12T02:39:17.274 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.275 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.275 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:39:17.275 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.082925+0000", 2020-06-12T02:39:17.275 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3282000000000006e-05 2020-06-12T02:39:17.276 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.276 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.276 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:17.276 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.083018+0000", 2020-06-12T02:39:17.276 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015909191999999999 2020-06-12T02:39:17.276 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:17.277 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:17.277 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:17.277 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.277 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.277 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15331 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~688128] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:39:17.278 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:33.276748+0000", 2020-06-12T02:39:17.278 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 43.890441615, 2020-06-12T02:39:17.278 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016825066999999999, 2020-06-12T02:39:17.278 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:17.278 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:17.279 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:17.279 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:17.279 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:17.279 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15331 2020-06-12T02:39:17.279 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.279 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:17.280 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.280 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:17.280 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276748+0000", 2020-06-12T02:39:17.280 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:17.280 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.281 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.281 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:17.281 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276748+0000", 2020-06-12T02:39:17.281 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016936699999999999 2020-06-12T02:39:17.281 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.282 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.282 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:17.282 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276918+0000", 2020-06-12T02:39:17.282 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6389999999999998e-06 2020-06-12T02:39:17.282 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.282 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.283 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:17.283 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276926+0000", 2020-06-12T02:39:17.283 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3591000000000001e-05 2020-06-12T02:39:17.283 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.283 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.284 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:17.284 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276940+0000", 2020-06-12T02:39:17.284 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0429999999999996e-06 2020-06-12T02:39:17.284 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.284 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.284 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:17.285 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276949+0000", 2020-06-12T02:39:17.285 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8825000000000001e-05 2020-06-12T02:39:17.285 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.286 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.286 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:17.286 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.276998+0000", 2020-06-12T02:39:17.286 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6483000000000005e-05 2020-06-12T02:39:17.286 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.287 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.287 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:17.287 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.277084+0000", 2020-06-12T02:39:17.287 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00057545800000000005 2020-06-12T02:39:17.287 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.287 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.288 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:17.288 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.277660+0000", 2020-06-12T02:39:17.288 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015617126 2020-06-12T02:39:17.288 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.288 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.289 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:17.289 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.293277+0000", 2020-06-12T02:39:17.289 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00021931699999999999 2020-06-12T02:39:17.289 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.289 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.289 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:39:17.290 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.293496+0000", 2020-06-12T02:39:17.290 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7218000000000005e-05 2020-06-12T02:39:17.290 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.290 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.290 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:17.291 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.293573+0000", 2020-06-12T02:39:17.291 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016825066999999999 2020-06-12T02:39:17.291 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:17.291 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:17.291 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:17.292 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.292 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.292 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.7s1 60/37 ECSubWrite(tid=8504, reqid=client.4512.0:16225, at_version=60'1427, trim_to=0'0, roll_forward_to=60'1425))", 2020-06-12T02:39:17.292 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:36.033154+0000", 2020-06-12T02:39:17.292 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 41.134036252999998, 2020-06-12T02:39:17.292 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016768244000000002, 2020-06-12T02:39:17.293 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:17.293 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:17.293 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:17.293 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.293 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:17.294 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033154+0000", 2020-06-12T02:39:17.294 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:17.294 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.294 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.294 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:17.295 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033154+0000", 2020-06-12T02:39:17.295 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000154444 2020-06-12T02:39:17.295 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.295 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.296 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:17.296 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033308+0000", 2020-06-12T02:39:17.296 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8772000000000002e-05 2020-06-12T02:39:17.296 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.296 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.297 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:17.297 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033337+0000", 2020-06-12T02:39:17.297 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6704000000000001e-05 2020-06-12T02:39:17.297 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.297 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.298 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:17.298 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033354+0000", 2020-06-12T02:39:17.298 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6100000000000002e-06 2020-06-12T02:39:17.298 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.298 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.298 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:17.299 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033360+0000", 2020-06-12T02:39:17.299 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000173893 2020-06-12T02:39:17.299 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.299 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.299 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:17.300 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033534+0000", 2020-06-12T02:39:17.300 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1929999999999999e-05 2020-06-12T02:39:17.300 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.300 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.300 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:17.300 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033546+0000", 2020-06-12T02:39:17.301 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016272066000000002 2020-06-12T02:39:17.301 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.301 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.301 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:17.301 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.049818+0000", 2020-06-12T02:39:17.302 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103825 2020-06-12T02:39:17.302 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.302 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.302 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:17.302 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.049922+0000", 2020-06-12T02:39:17.303 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016768244000000002 2020-06-12T02:39:17.303 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:17.303 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:17.303 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:17.303 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.303 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.304 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:17977 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:39:17.304 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:41.103849+0000", 2020-06-12T02:39:17.304 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 36.063341071000004, 2020-06-12T02:39:17.304 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015465273, 2020-06-12T02:39:17.304 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:17.305 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:39:17.305 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:17.305 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:17.305 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:17.305 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 17977 2020-06-12T02:39:17.306 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.306 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:17.306 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.306 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:17.307 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.103849+0000", 2020-06-12T02:39:17.307 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:17.307 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.307 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.307 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:17.308 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.103849+0000", 2020-06-12T02:39:17.308 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1254e-05 2020-06-12T02:39:17.308 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.308 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.308 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:17.309 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.103870+0000", 2020-06-12T02:39:17.309 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9869999999999999e-06 2020-06-12T02:39:17.309 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.309 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.309 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:17.309 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.103873+0000", 2020-06-12T02:39:17.310 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6159999999999998e-06 2020-06-12T02:39:17.310 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.310 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.310 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:17.310 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.103880+0000", 2020-06-12T02:39:17.311 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0899999999999999e-06 2020-06-12T02:39:17.311 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.311 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.311 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:17.311 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.103882+0000", 2020-06-12T02:39:17.311 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0066572259999999996 2020-06-12T02:39:17.312 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.312 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.312 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:17.312 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.110539+0000", 2020-06-12T02:39:17.312 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5937000000000003e-05 2020-06-12T02:39:17.313 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.313 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.313 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:17.313 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.110595+0000", 2020-06-12T02:39:17.313 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0066983820000000001 2020-06-12T02:39:17.314 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.314 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.314 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:17.314 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.117293+0000", 2020-06-12T02:39:17.314 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103828 2020-06-12T02:39:17.314 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.315 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.315 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:17.315 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.117397+0000", 2020-06-12T02:39:17.315 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0019169530000000001 2020-06-12T02:39:17.315 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.316 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.316 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:17.316 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.119314+0000", 2020-06-12T02:39:17.316 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015465273 2020-06-12T02:39:17.317 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:17.317 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:17.317 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:17.317 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.317 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.318 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.fs2 62/37 ECSubWrite(tid=3343, reqid=client.4512.0:18307, at_version=62'1623, trim_to=0'0, roll_forward_to=62'1622))", 2020-06-12T02:39:17.318 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:42.062050+0000", 2020-06-12T02:39:17.318 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 35.105139719, 2020-06-12T02:39:17.318 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019528429999999999, 2020-06-12T02:39:17.319 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:17.319 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:17.319 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:17.319 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.319 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:17.320 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062050+0000", 2020-06-12T02:39:17.320 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:17.320 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.320 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.320 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:17.321 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062050+0000", 2020-06-12T02:39:17.321 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00022828799999999999 2020-06-12T02:39:17.321 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.321 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.321 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:17.321 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062279+0000", 2020-06-12T02:39:17.322 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2115000000000001e-05 2020-06-12T02:39:17.322 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.322 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.322 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:17.322 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062301+0000", 2020-06-12T02:39:17.323 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9301e-05 2020-06-12T02:39:17.323 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.323 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.323 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:17.323 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062320+0000", 2020-06-12T02:39:17.323 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2811e-05 2020-06-12T02:39:17.324 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.324 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.324 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:17.324 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062333+0000", 2020-06-12T02:39:17.324 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8856e-05 2020-06-12T02:39:17.325 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.325 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.325 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:17.325 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062392+0000", 2020-06-12T02:39:17.325 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1246999999999999e-05 2020-06-12T02:39:17.325 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.326 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.326 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:17.326 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.062403+0000", 2020-06-12T02:39:17.327 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019095523 2020-06-12T02:39:17.327 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.327 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.327 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:17.327 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.081498+0000", 2020-06-12T02:39:17.327 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0289000000000003e-05 2020-06-12T02:39:17.328 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.328 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.328 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:17.328 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.081579+0000", 2020-06-12T02:39:17.328 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019528429999999999 2020-06-12T02:39:17.329 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:17.329 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:17.329 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:17.329 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.329 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.330 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.fs2 62/37 ECSubWrite(tid=3344, reqid=client.4512.0:18308, at_version=62'1624, trim_to=0'0, roll_forward_to=62'1622))", 2020-06-12T02:39:17.330 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:42.064848+0000", 2020-06-12T02:39:17.330 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 35.102341733999999, 2020-06-12T02:39:17.330 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01677273, 2020-06-12T02:39:17.330 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:17.330 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:17.331 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:17.331 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.331 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:17.331 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.064848+0000", 2020-06-12T02:39:17.331 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:17.332 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.332 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.332 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:17.332 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.064848+0000", 2020-06-12T02:39:17.332 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000383655 2020-06-12T02:39:17.333 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.333 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.333 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:17.333 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.065232+0000", 2020-06-12T02:39:17.333 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3065999999999999e-05 2020-06-12T02:39:17.333 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.334 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.334 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:17.334 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.065255+0000", 2020-06-12T02:39:17.334 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7142999999999998e-05 2020-06-12T02:39:17.334 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.335 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.335 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:17.335 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.065272+0000", 2020-06-12T02:39:17.335 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7750000000000004e-06 2020-06-12T02:39:17.335 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.336 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.336 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:17.336 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.065279+0000", 2020-06-12T02:39:17.336 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00091758699999999998 2020-06-12T02:39:17.336 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.337 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.337 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:17.337 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.066197+0000", 2020-06-12T02:39:17.337 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4158999999999999e-05 2020-06-12T02:39:17.338 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.338 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.338 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:17.338 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.066211+0000", 2020-06-12T02:39:17.338 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015373370000000001 2020-06-12T02:39:17.338 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.339 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.339 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:17.339 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.081584+0000", 2020-06-12T02:39:17.339 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6974999999999998e-05 2020-06-12T02:39:17.339 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.340 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.340 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:17.340 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.081621+0000", 2020-06-12T02:39:17.340 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01677273 2020-06-12T02:39:17.340 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:17.341 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:17.341 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:17.341 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.341 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.341 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:20350 3.17s0 3:ea271007:::smithi10729407-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e79)", 2020-06-12T02:39:17.341 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:39:05.652726+0000", 2020-06-12T02:39:17.342 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 11.514464028000001, 2020-06-12T02:39:17.342 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016846652, 2020-06-12T02:39:17.342 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:17.342 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:39:17.342 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:17.343 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:17.343 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:17.343 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 20350 2020-06-12T02:39:17.343 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.343 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:17.343 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.344 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:17.344 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:05.652726+0000", 2020-06-12T02:39:17.344 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:17.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.345 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.345 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:17.345 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:05.652726+0000", 2020-06-12T02:39:17.345 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1288999999999999e-05 2020-06-12T02:39:17.346 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.346 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.346 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:17.346 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:05.652747+0000", 2020-06-12T02:39:17.346 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.084e-06 2020-06-12T02:39:17.347 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.347 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.347 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:17.347 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:05.652750+0000", 2020-06-12T02:39:17.348 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.28e-06 2020-06-12T02:39:17.348 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.348 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.348 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:17.348 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:05.652757+0000", 2020-06-12T02:39:17.349 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2019999999999998e-06 2020-06-12T02:39:17.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.349 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.349 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:17.349 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:05.652759+0000", 2020-06-12T02:39:17.350 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010900821999999999 2020-06-12T02:39:17.350 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.350 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.350 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:17.350 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:05.663660+0000", 2020-06-12T02:39:17.350 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7930000000000003e-05 2020-06-12T02:39:17.351 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.351 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.351 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:17.351 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:05.663718+0000", 2020-06-12T02:39:17.351 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0038938850000000001 2020-06-12T02:39:17.352 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.352 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.352 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:17.352 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:05.667612+0000", 2020-06-12T02:39:17.352 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2225999999999998e-05 2020-06-12T02:39:17.352 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.353 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.353 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:17.353 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:05.667664+0000", 2020-06-12T02:39:17.353 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0019089339999999999 2020-06-12T02:39:17.353 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.354 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.354 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:17.354 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:05.669573+0000", 2020-06-12T02:39:17.354 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016846652 2020-06-12T02:39:17.354 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:17.355 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:17.355 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:17.355 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.355 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.355 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.fs2 81/37 ECSubWrite(tid=4725, reqid=client.4512.0:21809, at_version=0'0, trim_to=0'0, roll_forward_to=81'1925))", 2020-06-12T02:39:17.355 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:39:09.705341+0000", 2020-06-12T02:39:17.356 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 7.4618486659999999, 2020-06-12T02:39:17.356 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020075065999999999, 2020-06-12T02:39:17.356 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:17.356 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:17.356 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:17.357 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.357 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:17.357 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:09.705341+0000", 2020-06-12T02:39:17.357 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:17.358 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.358 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.358 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:17.358 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:09.705341+0000", 2020-06-12T02:39:17.358 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1474000000000002e-05 2020-06-12T02:39:17.359 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.359 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.359 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:17.359 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:09.705393+0000", 2020-06-12T02:39:17.359 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.313e-05 2020-06-12T02:39:17.359 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.360 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.360 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:17.360 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:09.705406+0000", 2020-06-12T02:39:17.360 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5523999999999999e-05 2020-06-12T02:39:17.360 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.361 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.361 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:17.361 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:09.705422+0000", 2020-06-12T02:39:17.361 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9789999999999998e-06 2020-06-12T02:39:17.361 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.362 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.362 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:17.362 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:09.705427+0000", 2020-06-12T02:39:17.362 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6386e-05 2020-06-12T02:39:17.362 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.362 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.363 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:17.363 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:09.705464+0000", 2020-06-12T02:39:17.363 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8689999999999998e-06 2020-06-12T02:39:17.363 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.363 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.364 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:17.364 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:09.705471+0000", 2020-06-12T02:39:17.364 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019902108000000002 2020-06-12T02:39:17.364 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.364 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.364 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:17.365 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:09.725373+0000", 2020-06-12T02:39:17.365 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3596000000000001e-05 2020-06-12T02:39:17.365 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.365 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.365 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:17.366 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:09.725416+0000", 2020-06-12T02:39:17.366 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020075065999999999 2020-06-12T02:39:17.366 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:17.366 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:17.366 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:17.366 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.367 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.367 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:21814 3.12s0 3:4afffc6f:::smithi10729407-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e81)", 2020-06-12T02:39:17.367 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:39:09.725599+0000", 2020-06-12T02:39:17.367 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 7.441591206, 2020-06-12T02:39:17.368 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016506874000000001, 2020-06-12T02:39:17.368 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:17.368 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:39:17.368 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:17.368 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:17.369 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:17.369 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 21814 2020-06-12T02:39:17.369 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.369 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:17.369 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.370 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:17.370 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:09.725599+0000", 2020-06-12T02:39:17.370 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:17.370 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.370 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.371 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:17.371 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:09.725599+0000", 2020-06-12T02:39:17.371 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9135000000000003e-05 2020-06-12T02:39:17.371 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.371 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.371 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:17.372 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:09.725648+0000", 2020-06-12T02:39:17.372 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7729999999999996e-06 2020-06-12T02:39:17.372 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.372 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.372 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:17.373 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:09.725654+0000", 2020-06-12T02:39:17.373 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2133999999999999e-05 2020-06-12T02:39:17.373 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.373 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.373 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:17.374 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:09.725666+0000", 2020-06-12T02:39:17.374 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6600000000000003e-06 2020-06-12T02:39:17.374 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.374 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.374 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:17.374 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:09.725671+0000", 2020-06-12T02:39:17.375 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013024987 2020-06-12T02:39:17.375 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.375 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.375 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:17.375 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:09.738696+0000", 2020-06-12T02:39:17.376 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3449000000000006e-05 2020-06-12T02:39:17.376 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.376 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.376 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:17.376 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:09.738789+0000", 2020-06-12T02:39:17.376 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001518159 2020-06-12T02:39:17.377 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.377 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.377 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:17.377 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:09.740307+0000", 2020-06-12T02:39:17.377 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7070000000000004e-05 2020-06-12T02:39:17.378 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.378 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.378 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:17.378 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:09.740404+0000", 2020-06-12T02:39:17.379 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017015070000000001 2020-06-12T02:39:17.379 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.379 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.379 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:17.379 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:09.742106+0000", 2020-06-12T02:39:17.380 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016506874000000001 2020-06-12T02:39:17.380 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:17.380 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:17.380 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:17.380 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.380 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.381 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:22447 3.12s0 3:4afffc6f:::smithi10729407-20:head [append 0~770048] snapc 0=[] ondisk+write+known_if_redirected e83)", 2020-06-12T02:39:17.381 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:39:11.552422+0000", 2020-06-12T02:39:17.381 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 5.6147679349999997, 2020-06-12T02:39:17.381 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017463409999999999, 2020-06-12T02:39:17.381 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:17.382 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:17.382 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:17.382 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:17.382 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:17.382 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 22447 2020-06-12T02:39:17.383 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.383 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:17.383 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.383 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:17.383 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.552422+0000", 2020-06-12T02:39:17.383 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:17.384 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.384 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.384 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:17.384 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.552422+0000", 2020-06-12T02:39:17.384 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000468479 2020-06-12T02:39:17.385 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.385 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.385 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:17.385 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.552891+0000", 2020-06-12T02:39:17.385 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2530000000000004e-06 2020-06-12T02:39:17.385 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.386 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.386 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:17.386 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.552899+0000", 2020-06-12T02:39:17.386 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7618e-05 2020-06-12T02:39:17.386 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.387 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.387 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:17.387 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.552916+0000", 2020-06-12T02:39:17.387 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8750000000000002e-06 2020-06-12T02:39:17.387 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.387 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.388 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:17.388 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.552923+0000", 2020-06-12T02:39:17.388 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00083629699999999995 2020-06-12T02:39:17.389 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.389 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.389 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:17.389 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.553760+0000", 2020-06-12T02:39:17.389 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000142299 2020-06-12T02:39:17.390 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.390 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.391 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:17.391 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.553902+0000", 2020-06-12T02:39:17.392 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028476780000000002 2020-06-12T02:39:17.392 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.392 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.392 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:17.392 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.556750+0000", 2020-06-12T02:39:17.392 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8752000000000006e-05 2020-06-12T02:39:17.393 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.393 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.393 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:17.393 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.556848+0000", 2020-06-12T02:39:17.393 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00076719199999999998 2020-06-12T02:39:17.394 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.394 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.394 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:17.394 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.557616+0000", 2020-06-12T02:39:17.394 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012051117 2020-06-12T02:39:17.395 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.395 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.395 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:17.395 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.569667+0000", 2020-06-12T02:39:17.395 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010488999999999999 2020-06-12T02:39:17.396 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.396 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.396 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:39:17.396 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.569772+0000", 2020-06-12T02:39:17.396 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011396 2020-06-12T02:39:17.397 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.397 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.397 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:17.397 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.569886+0000", 2020-06-12T02:39:17.397 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017463409999999999 2020-06-12T02:39:17.398 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:17.398 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:17.398 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:17.398 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.399 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.400 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:22449 3.12s0 3:4afffc6f:::smithi10729407-20:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e83)", 2020-06-12T02:39:17.400 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:39:11.555104+0000", 2020-06-12T02:39:17.400 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 5.6120858460000003, 2020-06-12T02:39:17.401 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016242128000000002, 2020-06-12T02:39:17.401 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:17.401 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:17.401 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:17.401 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:17.401 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:17.402 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 22449 2020-06-12T02:39:17.402 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.402 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:17.402 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.402 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:17.403 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.555104+0000", 2020-06-12T02:39:17.403 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:17.403 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.403 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.403 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:17.404 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.555104+0000", 2020-06-12T02:39:17.404 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7029000000000001e-05 2020-06-12T02:39:17.404 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.404 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.404 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:17.405 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.555141+0000", 2020-06-12T02:39:17.405 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4000000000000001e-06 2020-06-12T02:39:17.405 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.405 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.405 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:17.406 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.555145+0000", 2020-06-12T02:39:17.406 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2609999999999999e-06 2020-06-12T02:39:17.406 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.406 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.406 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:17.407 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.555151+0000", 2020-06-12T02:39:17.407 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.4739999999999999e-06 2020-06-12T02:39:17.407 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.408 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.408 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:17.408 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.555153+0000", 2020-06-12T02:39:17.408 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1388e-05 2020-06-12T02:39:17.408 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.409 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.409 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:17.409 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.555245+0000", 2020-06-12T02:39:17.409 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4327999999999998e-05 2020-06-12T02:39:17.410 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.410 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.410 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:17.410 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.555309+0000", 2020-06-12T02:39:17.411 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011102341999999999 2020-06-12T02:39:17.411 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.411 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.411 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:17.411 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.566411+0000", 2020-06-12T02:39:17.412 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010966 2020-06-12T02:39:17.412 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.412 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.412 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:17.412 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.566521+0000", 2020-06-12T02:39:17.412 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00043209500000000002 2020-06-12T02:39:17.413 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.413 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.413 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:17.413 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.566953+0000", 2020-06-12T02:39:17.413 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.003115633 2020-06-12T02:39:17.414 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.414 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.415 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:17.415 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.570069+0000", 2020-06-12T02:39:17.415 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011631829999999999 2020-06-12T02:39:17.415 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.415 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.416 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:39:17.416 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.571232+0000", 2020-06-12T02:39:17.416 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114335 2020-06-12T02:39:17.416 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.416 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.417 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:17.417 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.571346+0000", 2020-06-12T02:39:17.417 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016242128000000002 2020-06-12T02:39:17.417 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:17.417 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:17.418 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:17.419 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.419 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.419 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:22450 3.12s0 3:4afffc6f:::smithi10729407-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e83)", 2020-06-12T02:39:17.419 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:39:11.555177+0000", 2020-06-12T02:39:17.420 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 5.6120127350000004, 2020-06-12T02:39:17.420 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018302539, 2020-06-12T02:39:17.420 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:17.420 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:39:17.420 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:17.421 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:17.421 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:17.421 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 22450 2020-06-12T02:39:17.421 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.422 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:17.422 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.422 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:17.423 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.555177+0000", 2020-06-12T02:39:17.423 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:17.423 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.423 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.423 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:17.424 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.555177+0000", 2020-06-12T02:39:17.424 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2076e-05 2020-06-12T02:39:17.424 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.424 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.424 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:17.425 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.555199+0000", 2020-06-12T02:39:17.425 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.1609999999999999e-06 2020-06-12T02:39:17.425 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.425 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.425 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:17.426 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.555203+0000", 2020-06-12T02:39:17.426 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4749999999999998e-06 2020-06-12T02:39:17.426 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.426 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.426 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:17.427 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.555209+0000", 2020-06-12T02:39:17.427 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.306e-06 2020-06-12T02:39:17.427 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.427 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.428 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:17.428 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.555211+0000", 2020-06-12T02:39:17.428 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012595600000000001 2020-06-12T02:39:17.428 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.428 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.429 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:17.429 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.555337+0000", 2020-06-12T02:39:17.429 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9788000000000002e-05 2020-06-12T02:39:17.429 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.429 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.430 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:17.430 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.555427+0000", 2020-06-12T02:39:17.430 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012648456000000001 2020-06-12T02:39:17.430 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.431 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.431 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:17.431 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.568076+0000", 2020-06-12T02:39:17.431 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6217000000000007e-05 2020-06-12T02:39:17.432 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.432 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.432 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:17.432 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.568142+0000", 2020-06-12T02:39:17.433 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0039074139999999997 2020-06-12T02:39:17.433 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.433 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.433 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:17.433 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.572049+0000", 2020-06-12T02:39:17.434 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010529 2020-06-12T02:39:17.434 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.434 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.434 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:17.434 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.572154+0000", 2020-06-12T02:39:17.435 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013254 2020-06-12T02:39:17.435 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.435 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.435 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:17.436 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.573480+0000", 2020-06-12T02:39:17.436 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018302539 2020-06-12T02:39:17.436 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:17.436 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:17.436 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:17.437 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.437 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.437 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:23516 3.12s0 3:4afffc6f:::smithi10729407-20:head [append 0~622592] snapc 0=[] ondisk+write+known_if_redirected e84)", 2020-06-12T02:39:17.437 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:39:14.615949+0000", 2020-06-12T02:39:17.437 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 2.5512414290000001, 2020-06-12T02:39:17.438 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.83379187, 2020-06-12T02:39:17.438 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:17.438 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:17.438 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:17.438 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:17.439 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:17.439 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 23516 2020-06-12T02:39:17.439 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.439 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:17.439 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.440 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:17.440 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:14.615949+0000", 2020-06-12T02:39:17.440 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:17.440 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.441 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.441 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:17.441 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:14.615949+0000", 2020-06-12T02:39:17.441 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000317437 2020-06-12T02:39:17.442 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.442 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.442 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:17.442 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:14.616266+0000", 2020-06-12T02:39:17.442 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2809999999999997e-06 2020-06-12T02:39:17.443 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.443 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.443 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:17.443 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:14.616272+0000", 2020-06-12T02:39:17.443 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0375e-05 2020-06-12T02:39:17.444 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.444 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.444 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:17.444 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:14.616283+0000", 2020-06-12T02:39:17.444 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4049999999999997e-06 2020-06-12T02:39:17.445 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.445 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.445 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:17.445 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:14.616289+0000", 2020-06-12T02:39:17.445 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2094000000000001e-05 2020-06-12T02:39:17.446 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.447 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.447 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:17.447 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:14.616331+0000", 2020-06-12T02:39:17.447 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3955000000000006e-05 2020-06-12T02:39:17.447 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.448 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.448 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:17.448 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:14.616405+0000", 2020-06-12T02:39:17.448 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051945699999999997 2020-06-12T02:39:17.449 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.449 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.449 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:17.449 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:14.616925+0000", 2020-06-12T02:39:17.450 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.003606096 2020-06-12T02:39:17.450 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.450 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.450 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:17.450 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:14.620531+0000", 2020-06-12T02:39:17.451 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.81343419299999997 2020-06-12T02:39:17.451 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.452 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.453 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:17.453 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:15.433965+0000", 2020-06-12T02:39:17.453 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4960000000000001e-06 2020-06-12T02:39:17.453 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.453 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.454 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2020-06-12T02:39:17.454 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:15.433968+0000", 2020-06-12T02:39:17.454 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0156454049999999 2020-06-12T02:39:17.454 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.454 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.455 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:17.455 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:16.449614+0000", 2020-06-12T02:39:17.455 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012667599999999999 2020-06-12T02:39:17.455 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.455 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.456 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:17.456 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:16.449740+0000", 2020-06-12T02:39:17.456 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.83379187 2020-06-12T02:39:17.456 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:17.457 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:17.457 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:17.458 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.458 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.458 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:23517 3.12s0 3:4afffc6f:::smithi10729407-20:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e84)", 2020-06-12T02:39:17.458 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:39:14.616321+0000", 2020-06-12T02:39:17.459 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 2.550868747, 2020-06-12T02:39:17.459 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8338269519999999, 2020-06-12T02:39:17.459 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:17.461 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:17.461 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:17.461 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:17.461 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:17.462 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 23517 2020-06-12T02:39:17.462 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.462 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:17.462 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.462 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:17.463 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:14.616321+0000", 2020-06-12T02:39:17.463 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:17.463 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.463 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.463 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:17.464 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:14.616321+0000", 2020-06-12T02:39:17.464 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6791e-05 2020-06-12T02:39:17.464 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.464 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.464 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:17.465 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:14.616368+0000", 2020-06-12T02:39:17.465 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7100000000000001e-06 2020-06-12T02:39:17.465 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.465 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.465 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:17.465 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:14.616372+0000", 2020-06-12T02:39:17.466 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4189999999999999e-06 2020-06-12T02:39:17.466 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.466 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.467 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:17.467 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:14.616378+0000", 2020-06-12T02:39:17.467 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.88e-06 2020-06-12T02:39:17.467 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.467 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.468 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:17.468 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:14.616381+0000", 2020-06-12T02:39:17.468 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0034247129999999998 2020-06-12T02:39:17.468 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.468 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.468 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:17.469 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:14.619806+0000", 2020-06-12T02:39:17.469 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000108756 2020-06-12T02:39:17.469 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.470 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.475 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:17.476 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:14.619915+0000", 2020-06-12T02:39:17.476 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00032439300000000002 2020-06-12T02:39:17.476 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.476 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.477 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:17.477 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:14.620239+0000", 2020-06-12T02:39:17.477 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001242925 2020-06-12T02:39:17.477 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.477 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.478 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:17.478 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:14.621482+0000", 2020-06-12T02:39:17.478 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.81251964300000001 2020-06-12T02:39:17.478 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.479 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.479 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:17.479 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:15.434002+0000", 2020-06-12T02:39:17.479 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.164e-06 2020-06-12T02:39:17.479 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.480 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.480 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2020-06-12T02:39:17.480 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:15.434003+0000", 2020-06-12T02:39:17.480 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.01607811 2020-06-12T02:39:17.480 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.481 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.481 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:17.481 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:16.450081+0000", 2020-06-12T02:39:17.481 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7447999999999995e-05 2020-06-12T02:39:17.481 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.482 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.482 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:17.482 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:16.450148+0000", 2020-06-12T02:39:17.482 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8338269519999999 2020-06-12T02:39:17.482 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:17.483 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:17.483 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:17.483 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.483 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.483 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:23518 3.12s0 3:4afffc6f:::smithi10729407-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e84)", 2020-06-12T02:39:17.484 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:39:14.616406+0000", 2020-06-12T02:39:17.484 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 2.5507838610000002, 2020-06-12T02:39:17.484 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.835862729, 2020-06-12T02:39:17.484 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:17.485 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:39:17.485 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:17.485 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:17.485 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:17.486 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 23518 2020-06-12T02:39:17.486 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.486 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:17.486 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.486 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:17.486 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:14.616406+0000", 2020-06-12T02:39:17.487 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:17.487 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.487 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.487 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:17.488 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:14.616406+0000", 2020-06-12T02:39:17.488 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2855999999999999e-05 2020-06-12T02:39:17.488 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.488 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.488 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:17.489 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:14.616429+0000", 2020-06-12T02:39:17.489 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2179999999999999e-06 2020-06-12T02:39:17.489 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.489 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.490 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:17.490 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:14.616432+0000", 2020-06-12T02:39:17.490 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.64e-06 2020-06-12T02:39:17.491 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.491 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.491 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:17.492 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:14.616439+0000", 2020-06-12T02:39:17.492 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3460000000000001e-06 2020-06-12T02:39:17.492 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.492 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.493 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:17.493 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:14.616441+0000", 2020-06-12T02:39:17.493 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041215360000000003 2020-06-12T02:39:17.494 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.494 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.494 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:17.494 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:14.620563+0000", 2020-06-12T02:39:17.494 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8405000000000005e-05 2020-06-12T02:39:17.495 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.495 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.495 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:17.495 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:14.620651+0000", 2020-06-12T02:39:17.496 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.81338101200000001 2020-06-12T02:39:17.496 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.496 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.496 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:17.496 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:15.434032+0000", 2020-06-12T02:39:17.497 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0240000000000001e-06 2020-06-12T02:39:17.497 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.497 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.497 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2020-06-12T02:39:17.497 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:15.434033+0000", 2020-06-12T02:39:17.498 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0161461730000001 2020-06-12T02:39:17.498 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.499 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.499 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:17.499 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:16.450179+0000", 2020-06-12T02:39:17.499 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000197431 2020-06-12T02:39:17.500 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.500 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.500 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:17.500 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:16.450377+0000", 2020-06-12T02:39:17.500 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0018920880000000001 2020-06-12T02:39:17.500 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:17.501 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:17.501 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:17.501 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:16.452269+0000", 2020-06-12T02:39:17.501 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.835862729 2020-06-12T02:39:17.501 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:17.502 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:17.502 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:17.502 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:17.502 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:17.502 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:39:17.519 INFO:tasks.rados.rados.0.smithi107.stdout:11172: writing smithi10729407-5 from 0 to 696320 tid 1 2020-06-12T02:39:17.519 INFO:tasks.rados.rados.0.smithi107.stdout:11172: writing smithi10729407-5 from 696320 to 1155072 tid 2 2020-06-12T02:39:17.520 INFO:tasks.rados.rados.0.smithi107.stdout:11172: writing smithi10729407-5 from 1155072 to 1818624 tid 3 2020-06-12T02:39:17.520 INFO:tasks.rados.rados.0.smithi107.stdout:11172: writing smithi10729407-5 from 1818624 to 2301952 tid 4 2020-06-12T02:39:17.520 INFO:tasks.rados.rados.0.smithi107.stdout:11173: delete oid 2 current snap is 0 2020-06-12T02:39:17.520 INFO:tasks.rados.rados.0.smithi107.stdout:11169: finishing write tid 2 to smithi10729407-20 2020-06-12T02:39:17.520 INFO:tasks.rados.rados.0.smithi107.stdout:11169: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:17.521 INFO:tasks.rados.rados.0.smithi107.stdout:11169: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:17.521 INFO:tasks.rados.rados.0.smithi107.stdout:11169: finishing write tid 6 to smithi10729407-20 2020-06-12T02:39:17.521 INFO:tasks.rados.rados.0.smithi107.stdout:11169: finishing write tid 7 to smithi10729407-20 2020-06-12T02:39:17.521 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 873 (ObjNum 11168 snap 0 seq_num 11168) dirty exists 2020-06-12T02:39:17.521 INFO:tasks.rados.rados.0.smithi107.stdout:11169: left oid 20 (ObjNum 11168 snap 0 seq_num 11168) 2020-06-12T02:39:17.522 INFO:tasks.rados.rados.0.smithi107.stdout:11172: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:17.522 INFO:tasks.rados.rados.0.smithi107.stdout:11169: done (2 left) 2020-06-12T02:39:17.522 INFO:tasks.rados.rados.0.smithi107.stdout:11173: done (1 left) 2020-06-12T02:39:17.522 INFO:tasks.rados.rados.0.smithi107.stdout:11174: delete oid 13 current snap is 0 2020-06-12T02:39:17.522 INFO:tasks.rados.rados.0.smithi107.stdout:11172: finishing write tid 2 to smithi10729407-5 2020-06-12T02:39:17.523 INFO:tasks.rados.rados.0.smithi107.stdout:11174: done (1 left) 2020-06-12T02:39:17.523 INFO:tasks.rados.rados.0.smithi107.stdout:11175: delete oid 16 current snap is 0 2020-06-12T02:39:17.523 INFO:tasks.rados.rados.0.smithi107.stdout:11172: finishing write tid 3 to smithi10729407-5 2020-06-12T02:39:17.523 INFO:tasks.rados.rados.0.smithi107.stdout:11172: finishing write tid 4 to smithi10729407-5 2020-06-12T02:39:17.523 INFO:tasks.rados.rados.0.smithi107.stdout:11172: finishing write tid 6 to smithi10729407-5 2020-06-12T02:39:17.524 INFO:tasks.rados.rados.0.smithi107.stdout:11175: done (1 left) 2020-06-12T02:39:17.524 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:39:17.524 INFO:tasks.rados.rados.0.smithi107.stdout:11176: seq_num 11175 ranges {0=770048,770048=794624,1564672=638976,2203648=172032} 2020-06-12T02:39:17.524 INFO:tasks.rados.rados.0.smithi107.stdout:11176: writing smithi10729407-7 from 0 to 770048 tid 1 2020-06-12T02:39:17.524 INFO:tasks.rados.rados.0.smithi107.stdout:11176: writing smithi10729407-7 from 770048 to 1564672 tid 2 2020-06-12T02:39:17.524 INFO:tasks.rados.rados.0.smithi107.stdout:11176: writing smithi10729407-7 from 1564672 to 2203648 tid 3 2020-06-12T02:39:17.525 INFO:tasks.rados.rados.0.smithi107.stdout:11176: writing smithi10729407-7 from 2203648 to 2375680 tid 4 2020-06-12T02:39:17.525 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:39:17.525 INFO:tasks.rados.rados.0.smithi107.stdout:11177: seq_num 11176 ranges {0=786432,786432=729088,1515520=450560,1966080=425984} 2020-06-12T02:39:17.525 INFO:tasks.rados.rados.0.smithi107.stdout:11177: writing smithi10729407-16 from 0 to 786432 tid 1 2020-06-12T02:39:17.525 INFO:tasks.rados.rados.0.smithi107.stdout:11177: writing smithi10729407-16 from 786432 to 1515520 tid 2 2020-06-12T02:39:17.526 INFO:tasks.rados.rados.0.smithi107.stdout:11177: writing smithi10729407-16 from 1515520 to 1966080 tid 3 2020-06-12T02:39:17.526 INFO:tasks.rados.rados.0.smithi107.stdout:11177: writing smithi10729407-16 from 1966080 to 2392064 tid 4 2020-06-12T02:39:17.526 INFO:tasks.rados.rados.0.smithi107.stdout:11178: delete oid 20 current snap is 0 2020-06-12T02:39:17.526 INFO:tasks.rados.rados.0.smithi107.stdout:11172: finishing write tid 7 to smithi10729407-5 2020-06-12T02:39:17.526 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 1082 (ObjNum 11171 snap 0 seq_num 11171) dirty exists 2020-06-12T02:39:17.527 INFO:tasks.rados.rados.0.smithi107.stdout:11172: left oid 5 (ObjNum 11171 snap 0 seq_num 11171) 2020-06-12T02:39:17.527 INFO:tasks.rados.rados.0.smithi107.stdout:11176: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:17.527 INFO:tasks.rados.rados.0.smithi107.stdout:11176: finishing write tid 2 to smithi10729407-7 2020-06-12T02:39:17.527 INFO:tasks.rados.rados.0.smithi107.stdout:11176: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:17.528 INFO:tasks.rados.rados.0.smithi107.stdout:11176: finishing write tid 4 to smithi10729407-7 2020-06-12T02:39:17.528 INFO:tasks.rados.rados.0.smithi107.stdout:11176: finishing write tid 6 to smithi10729407-7 2020-06-12T02:39:17.528 INFO:tasks.rados.rados.0.smithi107.stdout:11177: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:17.528 INFO:tasks.rados.rados.0.smithi107.stdout:11176: finishing write tid 7 to smithi10729407-7 2020-06-12T02:39:17.528 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1815 (ObjNum 11175 snap 0 seq_num 11175) dirty exists 2020-06-12T02:39:17.529 INFO:tasks.rados.rados.0.smithi107.stdout:11176: left oid 7 (ObjNum 11175 snap 0 seq_num 11175) 2020-06-12T02:39:17.529 INFO:tasks.rados.rados.0.smithi107.stdout:11172: done (3 left) 2020-06-12T02:39:17.529 INFO:tasks.rados.rados.0.smithi107.stdout:11176: done (2 left) 2020-06-12T02:39:17.529 INFO:tasks.rados.rados.0.smithi107.stdout:11178: done (1 left) 2020-06-12T02:39:17.529 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:39:17.530 INFO:tasks.rados.rados.0.smithi107.stdout:11179: seq_num 11178 ranges {0=16384} 2020-06-12T02:39:17.530 INFO:tasks.rados.rados.0.smithi107.stdout:11179: writing smithi10729407-20 from 0 to 16384 tid 1 2020-06-12T02:39:17.530 INFO:tasks.rados.rados.0.smithi107.stdout:11180: delete oid 15 current snap is 0 2020-06-12T02:39:17.530 INFO:tasks.rados.rados.0.smithi107.stdout:11180: done (2 left) 2020-06-12T02:39:17.530 INFO:tasks.rados.rados.0.smithi107.stdout:11181: delete oid 11 current snap is 0 2020-06-12T02:39:17.530 INFO:tasks.rados.rados.0.smithi107.stdout:11177: finishing write tid 2 to smithi10729407-16 2020-06-12T02:39:17.531 INFO:tasks.rados.rados.0.smithi107.stdout:11181: done (2 left) 2020-06-12T02:39:17.531 INFO:tasks.rados.rados.0.smithi107.stdout:11182: delete oid 2 current snap is 0 2020-06-12T02:39:17.531 INFO:tasks.rados.rados.0.smithi107.stdout:11177: finishing write tid 3 to smithi10729407-16 2020-06-12T02:39:17.531 INFO:tasks.rados.rados.0.smithi107.stdout:11179: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:17.531 INFO:tasks.rados.rados.0.smithi107.stdout:11182: done (2 left) 2020-06-12T02:39:17.532 INFO:tasks.rados.rados.0.smithi107.stdout:11183: delete oid 18 current snap is 0 2020-06-12T02:39:17.532 INFO:tasks.rados.rados.0.smithi107.stdout:11177: finishing write tid 4 to smithi10729407-16 2020-06-12T02:39:17.532 INFO:tasks.rados.rados.0.smithi107.stdout:11177: finishing write tid 6 to smithi10729407-16 2020-06-12T02:39:17.532 INFO:tasks.rados.rados.0.smithi107.stdout:11177: finishing write tid 7 to smithi10729407-16 2020-06-12T02:39:17.535 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 1974 (ObjNum 11176 snap 0 seq_num 11176) dirty exists 2020-06-12T02:39:17.536 INFO:tasks.rados.rados.0.smithi107.stdout:11177: left oid 16 (ObjNum 11176 snap 0 seq_num 11176) 2020-06-12T02:39:17.536 INFO:tasks.rados.rados.0.smithi107.stdout:11177: done (2 left) 2020-06-12T02:39:17.536 INFO:tasks.rados.rados.0.smithi107.stdout:11183: done (1 left) 2020-06-12T02:39:17.537 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:39:17.537 INFO:tasks.rados.rados.0.smithi107.stdout:11184: seq_num 11183 ranges {1949696=98304} 2020-06-12T02:39:17.537 INFO:tasks.rados.rados.0.smithi107.stdout:11184: writing smithi10729407-1 from 1949696 to 2048000 tid 1 2020-06-12T02:39:17.538 INFO:tasks.rados.rados.0.smithi107.stdout:11185: delete oid 6 current snap is 0 2020-06-12T02:39:17.538 INFO:tasks.rados.rados.0.smithi107.stdout:11179: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:17.538 INFO:tasks.rados.rados.0.smithi107.stdout:11179: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:17.539 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 876 (ObjNum 11178 snap 0 seq_num 11178) dirty exists 2020-06-12T02:39:17.539 INFO:tasks.rados.rados.0.smithi107.stdout:11179: left oid 20 (ObjNum 11178 snap 0 seq_num 11178) 2020-06-12T02:39:17.540 INFO:tasks.rados.rados.0.smithi107.stdout:11184: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:17.540 INFO:tasks.rados.rados.0.smithi107.stdout:11179: done (2 left) 2020-06-12T02:39:17.540 INFO:tasks.rados.rados.0.smithi107.stdout:11185: done (1 left) 2020-06-12T02:39:17.541 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:39:17.541 INFO:tasks.rados.rados.0.smithi107.stdout:11186: seq_num 11185 ranges {0=131072} 2020-06-12T02:39:17.541 INFO:tasks.rados.rados.0.smithi107.stdout:11186: writing smithi10729407-6 from 0 to 131072 tid 1 2020-06-12T02:39:17.542 INFO:tasks.rados.rados.0.smithi107.stdout:11187: delete oid 10 current snap is 0 2020-06-12T02:39:17.543 INFO:tasks.rados.rados.0.smithi107.stdout:11184: finishing write tid 3 to smithi10729407-1 2020-06-12T02:39:17.543 INFO:tasks.rados.rados.0.smithi107.stdout:11187: done (2 left) 2020-06-12T02:39:17.544 INFO:tasks.rados.rados.0.smithi107.stdout:11188: delete oid 17 current snap is 0 2020-06-12T02:39:17.544 INFO:tasks.rados.rados.0.smithi107.stdout:11184: finishing write tid 4 to smithi10729407-1 2020-06-12T02:39:17.544 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 2013 (ObjNum 11183 snap 0 seq_num 11183) dirty exists 2020-06-12T02:39:17.545 INFO:tasks.rados.rados.0.smithi107.stdout:11184: left oid 1 (ObjNum 11183 snap 0 seq_num 11183) 2020-06-12T02:39:17.545 INFO:tasks.rados.rados.0.smithi107.stdout:11186: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:17.545 INFO:tasks.rados.rados.0.smithi107.stdout:11186: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:17.546 INFO:tasks.rados.rados.0.smithi107.stdout:11184: done (2 left) 2020-06-12T02:39:17.546 INFO:tasks.rados.rados.0.smithi107.stdout:11188: done (1 left) 2020-06-12T02:39:17.546 INFO:tasks.rados.rados.0.smithi107.stdout:11189: delete oid 16 current snap is 0 2020-06-12T02:39:17.547 INFO:tasks.rados.rados.0.smithi107.stdout:11186: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:17.547 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 2015 (ObjNum 11185 snap 0 seq_num 11185) dirty exists 2020-06-12T02:39:17.547 INFO:tasks.rados.rados.0.smithi107.stdout:11186: left oid 6 (ObjNum 11185 snap 0 seq_num 11185) 2020-06-12T02:39:17.547 INFO:tasks.rados.rados.0.smithi107.stdout:11186: done (1 left) 2020-06-12T02:39:17.547 INFO:tasks.rados.rados.0.smithi107.stdout:11189: done (0 left) 2020-06-12T02:39:17.547 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:39:17.548 INFO:tasks.rados.rados.0.smithi107.stdout:11190: seq_num 11189 ranges {2301952=196608} 2020-06-12T02:39:17.548 INFO:tasks.rados.rados.0.smithi107.stdout:11190: writing smithi10729407-5 from 2301952 to 2498560 tid 1 2020-06-12T02:39:17.548 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:39:17.548 INFO:tasks.rados.rados.0.smithi107.stdout:11191: seq_num 11190 ranges {0=212992} 2020-06-12T02:39:17.548 INFO:tasks.rados.rados.0.smithi107.stdout:11191: writing smithi10729407-8 from 0 to 212992 tid 1 2020-06-12T02:39:17.549 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:39:17.549 INFO:tasks.rados.rados.0.smithi107.stdout:11192: seq_num 11191 ranges {1884160=229376} 2020-06-12T02:39:17.549 INFO:tasks.rados.rados.0.smithi107.stdout:11192: writing smithi10729407-4 from 1884160 to 2113536 tid 1 2020-06-12T02:39:17.549 INFO:tasks.rados.rados.0.smithi107.stdout:11193: delete oid 14 current snap is 0 2020-06-12T02:39:17.550 INFO:tasks.rados.rados.0.smithi107.stdout:11190: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:17.550 INFO:tasks.rados.rados.0.smithi107.stdout:11190: finishing write tid 3 to smithi10729407-5 2020-06-12T02:39:17.551 INFO:tasks.rados.rados.0.smithi107.stdout:11191: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:17.551 INFO:tasks.rados.rados.0.smithi107.stdout:11191: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:17.551 INFO:tasks.rados.rados.0.smithi107.stdout:11190: finishing write tid 4 to smithi10729407-5 2020-06-12T02:39:17.552 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 1084 (ObjNum 11189 snap 0 seq_num 11189) dirty exists 2020-06-12T02:39:17.552 INFO:tasks.rados.rados.0.smithi107.stdout:11190: left oid 5 (ObjNum 11189 snap 0 seq_num 11189) 2020-06-12T02:39:17.552 INFO:tasks.rados.rados.0.smithi107.stdout:11191: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:17.554 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 2075 (ObjNum 11190 snap 0 seq_num 11190) dirty exists 2020-06-12T02:39:17.554 INFO:tasks.rados.rados.0.smithi107.stdout:11191: left oid 8 (ObjNum 11190 snap 0 seq_num 11190) 2020-06-12T02:39:17.554 INFO:tasks.rados.rados.0.smithi107.stdout:11190: done (3 left) 2020-06-12T02:39:17.554 INFO:tasks.rados.rados.0.smithi107.stdout:11191: done (2 left) 2020-06-12T02:39:17.555 INFO:tasks.rados.rados.0.smithi107.stdout:11193: done (1 left) 2020-06-12T02:39:17.555 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:39:17.555 INFO:tasks.rados.rados.0.smithi107.stdout:11194: seq_num 11193 ranges {2236416=262144} 2020-06-12T02:39:17.555 INFO:tasks.rados.rados.0.smithi107.stdout:11194: writing smithi10729407-12 from 2236416 to 2498560 tid 1 2020-06-12T02:39:17.555 INFO:tasks.rados.rados.0.smithi107.stdout:11195: delete oid 16 current snap is 0 2020-06-12T02:39:17.556 INFO:tasks.rados.rados.0.smithi107.stdout:11192: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:17.556 INFO:tasks.rados.rados.0.smithi107.stdout:11192: finishing write tid 3 to smithi10729407-4 2020-06-12T02:39:17.556 INFO:tasks.rados.rados.0.smithi107.stdout:11192: finishing write tid 4 to smithi10729407-4 2020-06-12T02:39:17.556 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 2090 (ObjNum 11191 snap 0 seq_num 11191) dirty exists 2020-06-12T02:39:17.556 INFO:tasks.rados.rados.0.smithi107.stdout:11192: left oid 4 (ObjNum 11191 snap 0 seq_num 11191) 2020-06-12T02:39:17.557 INFO:tasks.rados.rados.0.smithi107.stdout:11192: done (2 left) 2020-06-12T02:39:17.557 INFO:tasks.rados.rados.0.smithi107.stdout:11195: done (1 left) 2020-06-12T02:39:17.557 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:39:17.557 INFO:tasks.rados.rados.0.smithi107.stdout:11196: seq_num 11195 ranges {1482752=303104} 2020-06-12T02:39:17.557 INFO:tasks.rados.rados.0.smithi107.stdout:11196: writing smithi10729407-9 from 1482752 to 1785856 tid 1 2020-06-12T02:39:17.558 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:39:17.558 INFO:tasks.rados.rados.0.smithi107.stdout:11197: seq_num 11196 ranges {212992=319488} 2020-06-12T02:39:17.558 INFO:tasks.rados.rados.0.smithi107.stdout:11197: writing smithi10729407-8 from 212992 to 532480 tid 1 2020-06-12T02:39:17.558 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:39:17.558 INFO:tasks.rados.rados.0.smithi107.stdout:11198: seq_num 11197 ranges {131072=335872} 2020-06-12T02:39:17.558 INFO:tasks.rados.rados.0.smithi107.stdout:11198: writing smithi10729407-6 from 131072 to 466944 tid 1 2020-06-12T02:39:17.559 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:39:17.559 INFO:tasks.rados.rados.0.smithi107.stdout:11199: seq_num 11198 ranges {1966080=352256} 2020-06-12T02:39:17.559 INFO:tasks.rados.rados.0.smithi107.stdout:11199: writing smithi10729407-3 from 1966080 to 2318336 tid 1 2020-06-12T02:39:17.559 INFO:tasks.rados.rados.0.smithi107.stdout:11200: delete oid 11 current snap is 0 2020-06-12T02:39:17.559 INFO:tasks.rados.rados.0.smithi107.stdout:11194: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:17.560 INFO:tasks.rados.rados.0.smithi107.stdout:11194: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:17.560 INFO:tasks.rados.rados.0.smithi107.stdout:11194: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:17.560 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 2073 (ObjNum 11193 snap 0 seq_num 11193) dirty exists 2020-06-12T02:39:17.560 INFO:tasks.rados.rados.0.smithi107.stdout:11194: left oid 12 (ObjNum 11193 snap 0 seq_num 11193) 2020-06-12T02:39:17.560 INFO:tasks.rados.rados.0.smithi107.stdout:11196: finishing write tid 1 to smithi10729407-9 2020-06-12T02:39:17.561 INFO:tasks.rados.rados.0.smithi107.stdout:11197: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:17.561 INFO:tasks.rados.rados.0.smithi107.stdout:11197: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:17.561 INFO:tasks.rados.rados.0.smithi107.stdout:11196: finishing write tid 3 to smithi10729407-9 2020-06-12T02:39:17.561 INFO:tasks.rados.rados.0.smithi107.stdout:11198: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:17.561 INFO:tasks.rados.rados.0.smithi107.stdout:11197: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:17.562 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 2077 (ObjNum 11196 snap 0 seq_num 11196) dirty exists 2020-06-12T02:39:17.562 INFO:tasks.rados.rados.0.smithi107.stdout:11197: left oid 8 (ObjNum 11196 snap 0 seq_num 11196) 2020-06-12T02:39:17.563 INFO:tasks.rados.rados.0.smithi107.stdout:11196: finishing write tid 4 to smithi10729407-9 2020-06-12T02:39:17.564 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1857 (ObjNum 11195 snap 0 seq_num 11195) dirty exists 2020-06-12T02:39:17.564 INFO:tasks.rados.rados.0.smithi107.stdout:11196: left oid 9 (ObjNum 11195 snap 0 seq_num 11195) 2020-06-12T02:39:17.564 INFO:tasks.rados.rados.0.smithi107.stdout:11198: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:17.564 INFO:tasks.rados.rados.0.smithi107.stdout:11198: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:17.564 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 2017 (ObjNum 11197 snap 0 seq_num 11197) dirty exists 2020-06-12T02:39:17.565 INFO:tasks.rados.rados.0.smithi107.stdout:11198: left oid 6 (ObjNum 11197 snap 0 seq_num 11197) 2020-06-12T02:39:17.565 INFO:tasks.rados.rados.0.smithi107.stdout:11194: done (5 left) 2020-06-12T02:39:17.565 INFO:tasks.rados.rados.0.smithi107.stdout:11196: done (4 left) 2020-06-12T02:39:17.565 INFO:tasks.rados.rados.0.smithi107.stdout:11197: done (3 left) 2020-06-12T02:39:17.565 INFO:tasks.rados.rados.0.smithi107.stdout:11198: done (2 left) 2020-06-12T02:39:17.566 INFO:tasks.rados.rados.0.smithi107.stdout:11200: done (1 left) 2020-06-12T02:39:17.566 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:39:17.566 INFO:tasks.rados.rados.0.smithi107.stdout:11201: seq_num 11200 ranges {0=385024} 2020-06-12T02:39:17.566 INFO:tasks.rados.rados.0.smithi107.stdout:11201: writing smithi10729407-10 from 0 to 385024 tid 1 2020-06-12T02:39:17.566 INFO:tasks.rados.rados.0.smithi107.stdout:11202: delete oid 20 current snap is 0 2020-06-12T02:39:17.567 INFO:tasks.rados.rados.0.smithi107.stdout:11199: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:17.567 INFO:tasks.rados.rados.0.smithi107.stdout:11202: done (2 left) 2020-06-12T02:39:17.567 INFO:tasks.rados.rados.0.smithi107.stdout:11203: delete oid 1 current snap is 0 2020-06-12T02:39:17.567 INFO:tasks.rados.rados.0.smithi107.stdout:11199: finishing write tid 3 to smithi10729407-3 2020-06-12T02:39:17.567 INFO:tasks.rados.rados.0.smithi107.stdout:11199: finishing write tid 4 to smithi10729407-3 2020-06-12T02:39:17.568 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1978 (ObjNum 11198 snap 0 seq_num 11198) dirty exists 2020-06-12T02:39:17.568 INFO:tasks.rados.rados.0.smithi107.stdout:11199: left oid 3 (ObjNum 11198 snap 0 seq_num 11198) 2020-06-12T02:39:17.568 INFO:tasks.rados.rados.0.smithi107.stdout:11201: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:17.568 INFO:tasks.rados.rados.0.smithi107.stdout:11199: done (2 left) 2020-06-12T02:39:17.568 INFO:tasks.rados.rados.0.smithi107.stdout:11203: done (1 left) 2020-06-12T02:39:17.569 INFO:tasks.rados.rados.0.smithi107.stdout:11204: delete oid 13 current snap is 0 2020-06-12T02:39:17.569 INFO:tasks.rados.rados.0.smithi107.stdout:11204: done (1 left) 2020-06-12T02:39:17.569 INFO:tasks.rados.rados.0.smithi107.stdout:11205: delete oid 20 current snap is 0 2020-06-12T02:39:17.569 INFO:tasks.rados.rados.0.smithi107.stdout:11201: finishing write tid 3 to smithi10729407-10 2020-06-12T02:39:17.569 INFO:tasks.rados.rados.0.smithi107.stdout:11201: finishing write tid 4 to smithi10729407-10 2020-06-12T02:39:17.570 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 836 (ObjNum 11200 snap 0 seq_num 11200) dirty exists 2020-06-12T02:39:17.570 INFO:tasks.rados.rados.0.smithi107.stdout:11201: left oid 10 (ObjNum 11200 snap 0 seq_num 11200) 2020-06-12T02:39:17.570 INFO:tasks.rados.rados.0.smithi107.stdout:11201: done (1 left) 2020-06-12T02:39:17.570 INFO:tasks.rados.rados.0.smithi107.stdout:11205: done (0 left) 2020-06-12T02:39:17.571 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:39:17.571 INFO:tasks.rados.rados.0.smithi107.stdout:11206: seq_num 11205 ranges {0=466944} 2020-06-12T02:39:17.571 INFO:tasks.rados.rados.0.smithi107.stdout:11206: writing smithi10729407-14 from 0 to 466944 tid 1 2020-06-12T02:39:17.571 INFO:tasks.rados.rados.0.smithi107.stdout:11207: delete oid 8 current snap is 0 2020-06-12T02:39:17.571 INFO:tasks.rados.rados.0.smithi107.stdout:11207: done (1 left) 2020-06-12T02:39:17.571 INFO:tasks.rados.rados.0.smithi107.stdout:11208: delete oid 2 current snap is 0 2020-06-12T02:39:17.572 INFO:tasks.rados.rados.0.smithi107.stdout:11206: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:17.572 INFO:tasks.rados.rados.0.smithi107.stdout:11208: done (1 left) 2020-06-12T02:39:17.572 INFO:tasks.rados.rados.0.smithi107.stdout:11209: delete oid 16 current snap is 0 2020-06-12T02:39:17.572 INFO:tasks.rados.rados.0.smithi107.stdout:11209: done (1 left) 2020-06-12T02:39:17.572 INFO:tasks.rados.rados.0.smithi107.stdout:11210: delete oid 11 current snap is 0 2020-06-12T02:39:17.573 INFO:tasks.rados.rados.0.smithi107.stdout:11206: finishing write tid 3 to smithi10729407-14 2020-06-12T02:39:17.573 INFO:tasks.rados.rados.0.smithi107.stdout:11210: done (1 left) 2020-06-12T02:39:17.573 INFO:tasks.rados.rados.0.smithi107.stdout:11211: delete oid 13 current snap is 0 2020-06-12T02:39:17.573 INFO:tasks.rados.rados.0.smithi107.stdout:11206: finishing write tid 4 to smithi10729407-14 2020-06-12T02:39:17.573 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1762 (ObjNum 11205 snap 0 seq_num 11205) dirty exists 2020-06-12T02:39:17.574 INFO:tasks.rados.rados.0.smithi107.stdout:11206: left oid 14 (ObjNum 11205 snap 0 seq_num 11205) 2020-06-12T02:39:17.574 INFO:tasks.rados.rados.0.smithi107.stdout:11206: done (1 left) 2020-06-12T02:39:17.574 INFO:tasks.rados.rados.0.smithi107.stdout:11211: done (0 left) 2020-06-12T02:39:17.574 INFO:tasks.rados.rados.0.smithi107.stdout:11212: delete oid 18 current snap is 0 2020-06-12T02:39:17.574 INFO:tasks.rados.rados.0.smithi107.stdout:11212: done (0 left) 2020-06-12T02:39:17.574 INFO:tasks.rados.rados.0.smithi107.stdout:11213: delete oid 16 current snap is 0 2020-06-12T02:39:17.575 INFO:tasks.rados.rados.0.smithi107.stdout:11213: done (0 left) 2020-06-12T02:39:17.575 INFO:tasks.rados.rados.0.smithi107.stdout:11214: delete oid 7 current snap is 0 2020-06-12T02:39:17.575 INFO:tasks.rados.rados.0.smithi107.stdout:11214: done (0 left) 2020-06-12T02:39:17.575 INFO:tasks.rados.rados.0.smithi107.stdout:11215: delete oid 3 current snap is 0 2020-06-12T02:39:17.575 INFO:tasks.rados.rados.0.smithi107.stdout:11215: done (0 left) 2020-06-12T02:39:17.576 INFO:tasks.rados.rados.0.smithi107.stdout:11216: delete oid 16 current snap is 0 2020-06-12T02:39:17.576 INFO:tasks.rados.rados.0.smithi107.stdout:11216: done (0 left) 2020-06-12T02:39:17.576 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:39:17.576 INFO:tasks.rados.rados.0.smithi107.stdout:11217: seq_num 11216 ranges {1785856=655360} 2020-06-12T02:39:17.576 INFO:tasks.rados.rados.0.smithi107.stdout:11217: writing smithi10729407-9 from 1785856 to 2441216 tid 1 2020-06-12T02:39:17.577 INFO:tasks.rados.rados.0.smithi107.stdout:11218: delete oid 10 current snap is 0 2020-06-12T02:39:17.577 INFO:tasks.rados.rados.0.smithi107.stdout:11218: done (1 left) 2020-06-12T02:39:17.577 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:39:17.577 INFO:tasks.rados.rados.0.smithi107.stdout:11219: seq_num 11218 ranges {0=688128} 2020-06-12T02:39:17.577 INFO:tasks.rados.rados.0.smithi107.stdout:11219: writing smithi10729407-20 from 0 to 688128 tid 1 2020-06-12T02:39:17.577 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:39:17.578 INFO:tasks.rados.rados.0.smithi107.stdout:11220: seq_num 11219 ranges {0=704512} 2020-06-12T02:39:17.578 INFO:tasks.rados.rados.0.smithi107.stdout:11220: writing smithi10729407-7 from 0 to 704512 tid 1 2020-06-12T02:39:17.578 INFO:tasks.rados.rados.0.smithi107.stdout:11221: delete oid 19 current snap is 0 2020-06-12T02:39:17.578 INFO:tasks.rados.rados.0.smithi107.stdout:11217: finishing write tid 1 to smithi10729407-9 2020-06-12T02:39:17.578 INFO:tasks.rados.rados.0.smithi107.stdout:11219: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:17.579 INFO:tasks.rados.rados.0.smithi107.stdout:11217: finishing write tid 3 to smithi10729407-9 2020-06-12T02:39:17.579 INFO:tasks.rados.rados.0.smithi107.stdout:11221: done (3 left) 2020-06-12T02:39:17.579 INFO:tasks.rados.rados.0.smithi107.stdout:11222: delete oid 5 current snap is 0 2020-06-12T02:39:17.579 INFO:tasks.rados.rados.0.smithi107.stdout:11217: finishing write tid 4 to smithi10729407-9 2020-06-12T02:39:17.579 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1859 (ObjNum 11216 snap 0 seq_num 11216) dirty exists 2020-06-12T02:39:17.579 INFO:tasks.rados.rados.0.smithi107.stdout:11217: left oid 9 (ObjNum 11216 snap 0 seq_num 11216) 2020-06-12T02:39:17.580 INFO:tasks.rados.rados.0.smithi107.stdout:11217: done (3 left) 2020-06-12T02:39:17.580 INFO:tasks.rados.rados.0.smithi107.stdout:11222: done (2 left) 2020-06-12T02:39:17.580 INFO:tasks.rados.rados.0.smithi107.stdout:11223: delete oid 1 current snap is 0 2020-06-12T02:39:17.580 INFO:tasks.rados.rados.0.smithi107.stdout:11219: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:17.580 INFO:tasks.rados.rados.0.smithi107.stdout:11223: done (2 left) 2020-06-12T02:39:17.581 INFO:tasks.rados.rados.0.smithi107.stdout:11224: delete oid 5 current snap is 0 2020-06-12T02:39:17.581 INFO:tasks.rados.rados.0.smithi107.stdout:11220: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:17.581 INFO:tasks.rados.rados.0.smithi107.stdout:11220: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:17.581 INFO:tasks.rados.rados.0.smithi107.stdout:11224: done (2 left) 2020-06-12T02:39:17.581 INFO:tasks.rados.rados.0.smithi107.stdout:11225: delete oid 12 current snap is 0 2020-06-12T02:39:17.581 INFO:tasks.rados.rados.0.smithi107.stdout:11219: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:17.582 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 880 (ObjNum 11218 snap 0 seq_num 11218) dirty exists 2020-06-12T02:39:17.582 INFO:tasks.rados.rados.0.smithi107.stdout:11219: left oid 20 (ObjNum 11218 snap 0 seq_num 11218) 2020-06-12T02:39:17.582 INFO:tasks.rados.rados.0.smithi107.stdout:11220: finishing write tid 4 to smithi10729407-7 2020-06-12T02:39:17.582 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1818 (ObjNum 11219 snap 0 seq_num 11219) dirty exists 2020-06-12T02:39:17.582 INFO:tasks.rados.rados.0.smithi107.stdout:11220: left oid 7 (ObjNum 11219 snap 0 seq_num 11219) 2020-06-12T02:39:17.583 INFO:tasks.rados.rados.0.smithi107.stdout:11219: done (2 left) 2020-06-12T02:39:17.583 INFO:tasks.rados.rados.0.smithi107.stdout:11220: done (1 left) 2020-06-12T02:39:17.583 INFO:tasks.rados.rados.0.smithi107.stdout:11225: done (0 left) 2020-06-12T02:39:17.583 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:39:17.583 INFO:tasks.rados.rados.0.smithi107.stdout:11226: seq_num 11225 ranges {0=802816} 2020-06-12T02:39:17.584 INFO:tasks.rados.rados.0.smithi107.stdout:11226: writing smithi10729407-15 from 0 to 802816 tid 1 2020-06-12T02:39:17.584 INFO:tasks.rados.rados.0.smithi107.stdout:11227: delete oid 10 current snap is 0 2020-06-12T02:39:17.584 INFO:tasks.rados.rados.0.smithi107.stdout:11227: done (1 left) 2020-06-12T02:39:17.584 INFO:tasks.rados.rados.0.smithi107.stdout:11228: delete oid 17 current snap is 0 2020-06-12T02:39:17.584 INFO:tasks.rados.rados.0.smithi107.stdout:11226: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:17.585 INFO:tasks.rados.rados.0.smithi107.stdout:11228: done (1 left) 2020-06-12T02:39:17.585 INFO:tasks.rados.rados.0.smithi107.stdout:11229: delete oid 11 current snap is 0 2020-06-12T02:39:17.585 INFO:tasks.rados.rados.0.smithi107.stdout:11226: finishing write tid 3 to smithi10729407-15 2020-06-12T02:39:17.585 INFO:tasks.rados.rados.0.smithi107.stdout:11229: done (1 left) 2020-06-12T02:39:17.585 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:39:17.586 INFO:tasks.rados.rados.0.smithi107.stdout:11230: seq_num 11229 ranges {0=466944,466944=401408} 2020-06-12T02:39:17.586 INFO:tasks.rados.rados.0.smithi107.stdout:11230: writing smithi10729407-8 from 0 to 466944 tid 1 2020-06-12T02:39:17.586 INFO:tasks.rados.rados.0.smithi107.stdout:11230: writing smithi10729407-8 from 466944 to 868352 tid 2 2020-06-12T02:39:17.586 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:39:17.586 INFO:tasks.rados.rados.0.smithi107.stdout:11231: seq_num 11230 ranges {2113536=483328,2596864=401408} 2020-06-12T02:39:17.587 INFO:tasks.rados.rados.0.smithi107.stdout:11231: writing smithi10729407-4 from 2113536 to 2596864 tid 1 2020-06-12T02:39:17.587 INFO:tasks.rados.rados.0.smithi107.stdout:11231: writing smithi10729407-4 from 2596864 to 2998272 tid 2 2020-06-12T02:39:17.587 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:39:17.587 INFO:tasks.rados.rados.0.smithi107.stdout:11232: seq_num 11231 ranges {0=499712,499712=401408} 2020-06-12T02:39:17.587 INFO:tasks.rados.rados.0.smithi107.stdout:11232: writing smithi10729407-2 from 0 to 499712 tid 1 2020-06-12T02:39:17.587 INFO:tasks.rados.rados.0.smithi107.stdout:11232: writing smithi10729407-2 from 499712 to 901120 tid 2 2020-06-12T02:39:17.588 INFO:tasks.rados.rados.0.smithi107.stdout:11233: delete oid 1 current snap is 0 2020-06-12T02:39:17.588 INFO:tasks.rados.rados.0.smithi107.stdout:11226: finishing write tid 4 to smithi10729407-15 2020-06-12T02:39:17.588 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 2080 (ObjNum 11225 snap 0 seq_num 11225) dirty exists 2020-06-12T02:39:17.588 INFO:tasks.rados.rados.0.smithi107.stdout:11226: left oid 15 (ObjNum 11225 snap 0 seq_num 11225) 2020-06-12T02:39:17.588 INFO:tasks.rados.rados.0.smithi107.stdout:11230: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:17.589 INFO:tasks.rados.rados.0.smithi107.stdout:11230: finishing write tid 2 to smithi10729407-8 2020-06-12T02:39:17.589 INFO:tasks.rados.rados.0.smithi107.stdout:11230: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:17.589 INFO:tasks.rados.rados.0.smithi107.stdout:11230: finishing write tid 5 to smithi10729407-8 2020-06-12T02:39:17.589 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 2083 (ObjNum 11229 snap 0 seq_num 11229) dirty exists 2020-06-12T02:39:17.589 INFO:tasks.rados.rados.0.smithi107.stdout:11230: left oid 8 (ObjNum 11229 snap 0 seq_num 11229) 2020-06-12T02:39:17.589 INFO:tasks.rados.rados.0.smithi107.stdout:11231: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:17.590 INFO:tasks.rados.rados.0.smithi107.stdout:11226: done (4 left) 2020-06-12T02:39:17.590 INFO:tasks.rados.rados.0.smithi107.stdout:11230: done (3 left) 2020-06-12T02:39:17.590 INFO:tasks.rados.rados.0.smithi107.stdout:11233: done (2 left) 2020-06-12T02:39:17.590 INFO:tasks.rados.rados.0.smithi107.stdout:11234: delete oid 13 current snap is 0 2020-06-12T02:39:17.590 INFO:tasks.rados.rados.0.smithi107.stdout:11231: finishing write tid 2 to smithi10729407-4 2020-06-12T02:39:17.591 INFO:tasks.rados.rados.0.smithi107.stdout:11231: finishing write tid 4 to smithi10729407-4 2020-06-12T02:39:17.591 INFO:tasks.rados.rados.0.smithi107.stdout:11232: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:17.591 INFO:tasks.rados.rados.0.smithi107.stdout:11234: done (2 left) 2020-06-12T02:39:17.591 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:39:17.591 INFO:tasks.rados.rados.0.smithi107.stdout:11235: seq_num 11234 ranges {802816=557056,1359872=401408} 2020-06-12T02:39:17.591 INFO:tasks.rados.rados.0.smithi107.stdout:11235: writing smithi10729407-15 from 802816 to 1359872 tid 1 2020-06-12T02:39:17.592 INFO:tasks.rados.rados.0.smithi107.stdout:11235: writing smithi10729407-15 from 1359872 to 1761280 tid 2 2020-06-12T02:39:17.592 INFO:tasks.rados.rados.0.smithi107.stdout:11236: delete oid 3 current snap is 0 2020-06-12T02:39:17.592 INFO:tasks.rados.rados.0.smithi107.stdout:11232: finishing write tid 2 to smithi10729407-2 2020-06-12T02:39:17.592 INFO:tasks.rados.rados.0.smithi107.stdout:11232: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:17.592 INFO:tasks.rados.rados.0.smithi107.stdout:11231: finishing write tid 5 to smithi10729407-4 2020-06-12T02:39:17.593 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 2094 (ObjNum 11230 snap 0 seq_num 11230) dirty exists 2020-06-12T02:39:17.593 INFO:tasks.rados.rados.0.smithi107.stdout:11231: left oid 4 (ObjNum 11230 snap 0 seq_num 11230) 2020-06-12T02:39:17.593 INFO:tasks.rados.rados.0.smithi107.stdout:11232: finishing write tid 5 to smithi10729407-2 2020-06-12T02:39:17.593 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 1062 (ObjNum 11231 snap 0 seq_num 11231) dirty exists 2020-06-12T02:39:17.593 INFO:tasks.rados.rados.0.smithi107.stdout:11232: left oid 2 (ObjNum 11231 snap 0 seq_num 11231) 2020-06-12T02:39:17.593 INFO:tasks.rados.rados.0.smithi107.stdout:11231: done (3 left) 2020-06-12T02:39:17.594 INFO:tasks.rados.rados.0.smithi107.stdout:11232: done (2 left) 2020-06-12T02:39:17.594 INFO:tasks.rados.rados.0.smithi107.stdout:11236: done (1 left) 2020-06-12T02:39:17.594 INFO:tasks.rados.rados.0.smithi107.stdout:11237: delete oid 19 current snap is 0 2020-06-12T02:39:17.594 INFO:tasks.rados.rados.0.smithi107.stdout:11237: done (1 left) 2020-06-12T02:39:17.595 INFO:tasks.rados.rados.0.smithi107.stdout:11238: delete oid 11 current snap is 0 2020-06-12T02:39:17.595 INFO:tasks.rados.rados.0.smithi107.stdout:11235: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:17.595 INFO:tasks.rados.rados.0.smithi107.stdout:11235: finishing write tid 2 to smithi10729407-15 2020-06-12T02:39:17.595 INFO:tasks.rados.rados.0.smithi107.stdout:11235: finishing write tid 4 to smithi10729407-15 2020-06-12T02:39:17.595 INFO:tasks.rados.rados.0.smithi107.stdout:11238: done (1 left) 2020-06-12T02:39:17.596 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:39:17.596 INFO:tasks.rados.rados.0.smithi107.stdout:11239: seq_num 11238 ranges {2998272=622592,3620864=401408} 2020-06-12T02:39:17.596 INFO:tasks.rados.rados.0.smithi107.stdout:11239: writing smithi10729407-4 from 2998272 to 3620864 tid 1 2020-06-12T02:39:17.596 INFO:tasks.rados.rados.0.smithi107.stdout:11239: writing smithi10729407-4 from 3620864 to 4022272 tid 2 2020-06-12T02:39:17.596 INFO:tasks.rados.rados.0.smithi107.stdout:11240: delete oid 20 current snap is 0 2020-06-12T02:39:17.596 INFO:tasks.rados.rados.0.smithi107.stdout:11235: finishing write tid 5 to smithi10729407-15 2020-06-12T02:39:17.597 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 2086 (ObjNum 11234 snap 0 seq_num 11234) dirty exists 2020-06-12T02:39:17.597 INFO:tasks.rados.rados.0.smithi107.stdout:11235: left oid 15 (ObjNum 11234 snap 0 seq_num 11234) 2020-06-12T02:39:17.597 INFO:tasks.rados.rados.0.smithi107.stdout:11235: done (2 left) 2020-06-12T02:39:17.597 INFO:tasks.rados.rados.0.smithi107.stdout:11240: done (1 left) 2020-06-12T02:39:17.597 INFO:tasks.rados.rados.0.smithi107.stdout:11241: delete oid 13 current snap is 0 2020-06-12T02:39:17.598 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-06-12T02:39:17.599 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:39:17.502+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:39:17.603 INFO:tasks.rados.rados.0.smithi107.stdout:11241: done (1 left) 2020-06-12T02:39:17.603 INFO:tasks.rados.rados.0.smithi107.stdout:11242: delete oid 15 current snap is 0 2020-06-12T02:39:17.604 INFO:tasks.rados.rados.0.smithi107.stdout:11239: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:17.604 INFO:tasks.rados.rados.0.smithi107.stdout:11242: done (1 left) 2020-06-12T02:39:17.604 INFO:tasks.rados.rados.0.smithi107.stdout:11243: delete oid 18 current snap is 0 2020-06-12T02:39:17.604 INFO:tasks.rados.rados.0.smithi107.stdout:11243: done (1 left) 2020-06-12T02:39:17.604 INFO:tasks.rados.rados.0.smithi107.stdout:11244: delete oid 17 current snap is 0 2020-06-12T02:39:17.604 INFO:tasks.rados.rados.0.smithi107.stdout:11239: finishing write tid 2 to smithi10729407-4 2020-06-12T02:39:17.605 INFO:tasks.rados.rados.0.smithi107.stdout:11239: finishing write tid 4 to smithi10729407-4 2020-06-12T02:39:17.605 INFO:tasks.rados.rados.0.smithi107.stdout:11244: done (1 left) 2020-06-12T02:39:17.605 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:39:17.610 INFO:tasks.rados.rados.0.smithi107.stdout:11245: seq_num 11244 ranges {0=720896,720896=401408} 2020-06-12T02:39:17.611 INFO:tasks.rados.rados.0.smithi107.stdout:11245: writing smithi10729407-19 from 0 to 720896 tid 1 2020-06-12T02:39:17.611 INFO:tasks.rados.rados.0.smithi107.stdout:11245: writing smithi10729407-19 from 720896 to 1122304 tid 2 2020-06-12T02:39:17.611 INFO:tasks.rados.rados.0.smithi107.stdout:11246: delete oid 9 current snap is 0 2020-06-12T02:39:17.611 INFO:tasks.rados.rados.0.smithi107.stdout:11239: finishing write tid 5 to smithi10729407-4 2020-06-12T02:39:17.611 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 2097 (ObjNum 11238 snap 0 seq_num 11238) dirty exists 2020-06-12T02:39:17.612 INFO:tasks.rados.rados.0.smithi107.stdout:11239: left oid 4 (ObjNum 11238 snap 0 seq_num 11238) 2020-06-12T02:39:17.612 INFO:tasks.rados.rados.0.smithi107.stdout:11239: done (2 left) 2020-06-12T02:39:17.612 INFO:tasks.rados.rados.0.smithi107.stdout:11246: done (1 left) 2020-06-12T02:39:17.612 INFO:tasks.rados.rados.0.smithi107.stdout:11247: delete oid 18 current snap is 0 2020-06-12T02:39:17.612 INFO:tasks.rados.rados.0.smithi107.stdout:11247: done (1 left) 2020-06-12T02:39:17.612 INFO:tasks.rados.rados.0.smithi107.stdout:11248: delete oid 8 current snap is 0 2020-06-12T02:39:17.613 INFO:tasks.rados.rados.0.smithi107.stdout:11245: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:17.613 INFO:tasks.rados.rados.0.smithi107.stdout:11248: done (1 left) 2020-06-12T02:39:17.613 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:39:17.613 INFO:tasks.rados.rados.0.smithi107.stdout:11249: seq_num 11248 ranges {0=786432,786432=401408} 2020-06-12T02:39:17.615 INFO:tasks.rados.rados.0.smithi107.stdout:11249: writing smithi10729407-10 from 0 to 786432 tid 1 2020-06-12T02:39:17.616 INFO:tasks.rados.rados.0.smithi107.stdout:11249: writing smithi10729407-10 from 786432 to 1187840 tid 2 2020-06-12T02:39:17.616 INFO:tasks.rados.rados.0.smithi107.stdout:11250: delete oid 4 current snap is 0 2020-06-12T02:39:17.616 INFO:tasks.rados.rados.0.smithi107.stdout:11245: finishing write tid 2 to smithi10729407-19 2020-06-12T02:39:17.616 INFO:tasks.rados.rados.0.smithi107.stdout:11245: finishing write tid 4 to smithi10729407-19 2020-06-12T02:39:17.617 INFO:tasks.rados.rados.0.smithi107.stdout:11245: finishing write tid 5 to smithi10729407-19 2020-06-12T02:39:17.617 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 2079 (ObjNum 11244 snap 0 seq_num 11244) dirty exists 2020-06-12T02:39:17.617 INFO:tasks.rados.rados.0.smithi107.stdout:11245: left oid 19 (ObjNum 11244 snap 0 seq_num 11244) 2020-06-12T02:39:17.617 INFO:tasks.rados.rados.0.smithi107.stdout:11245: done (2 left) 2020-06-12T02:39:17.617 INFO:tasks.rados.rados.0.smithi107.stdout:11250: done (1 left) 2020-06-12T02:39:17.618 INFO:tasks.rados.rados.0.smithi107.stdout:11251: delete oid 15 current snap is 0 2020-06-12T02:39:17.618 INFO:tasks.rados.rados.0.smithi107.stdout:11249: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:17.618 INFO:tasks.rados.rados.0.smithi107.stdout:11251: done (1 left) 2020-06-12T02:39:17.618 INFO:tasks.rados.rados.0.smithi107.stdout:11252: delete oid 9 current snap is 0 2020-06-12T02:39:17.618 INFO:tasks.rados.rados.0.smithi107.stdout:11249: finishing write tid 2 to smithi10729407-10 2020-06-12T02:39:17.619 INFO:tasks.rados.rados.0.smithi107.stdout:11249: finishing write tid 4 to smithi10729407-10 2020-06-12T02:39:17.619 INFO:tasks.rados.rados.0.smithi107.stdout:11249: finishing write tid 5 to smithi10729407-10 2020-06-12T02:39:17.619 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 841 (ObjNum 11248 snap 0 seq_num 11248) dirty exists 2020-06-12T02:39:17.619 INFO:tasks.rados.rados.0.smithi107.stdout:11249: left oid 10 (ObjNum 11248 snap 0 seq_num 11248) 2020-06-12T02:39:17.619 INFO:tasks.rados.rados.0.smithi107.stdout:11249: done (1 left) 2020-06-12T02:39:17.619 INFO:tasks.rados.rados.0.smithi107.stdout:11252: done (0 left) 2020-06-12T02:39:17.620 INFO:tasks.rados.rados.0.smithi107.stdout:11253: delete oid 2 current snap is 0 2020-06-12T02:39:17.620 INFO:tasks.rados.rados.0.smithi107.stdout:11253: done (0 left) 2020-06-12T02:39:17.620 INFO:tasks.rados.rados.0.smithi107.stdout:11254: delete oid 15 current snap is 0 2020-06-12T02:39:17.620 INFO:tasks.rados.rados.0.smithi107.stdout:11254: done (0 left) 2020-06-12T02:39:17.620 INFO:tasks.rados.rados.0.smithi107.stdout:11255: delete oid 18 current snap is 0 2020-06-12T02:39:17.621 INFO:tasks.rados.rados.0.smithi107.stdout:11255: done (0 left) 2020-06-12T02:39:17.621 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:39:17.621 INFO:tasks.rados.rados.0.smithi107.stdout:11256: seq_num 11255 ranges {466944=507904,974848=483328,1458176=319488} 2020-06-12T02:39:17.621 INFO:tasks.rados.rados.0.smithi107.stdout:11256: writing smithi10729407-14 from 466944 to 974848 tid 1 2020-06-12T02:39:17.621 INFO:tasks.rados.rados.0.smithi107.stdout:11256: writing smithi10729407-14 from 974848 to 1458176 tid 2 2020-06-12T02:39:17.621 INFO:tasks.rados.rados.0.smithi107.stdout:11256: writing smithi10729407-14 from 1458176 to 1777664 tid 3 2020-06-12T02:39:17.622 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:39:17.622 INFO:tasks.rados.rados.0.smithi107.stdout:11257: seq_num 11256 ranges {466944=524288,991232=770048,1761280=32768} 2020-06-12T02:39:17.622 INFO:tasks.rados.rados.0.smithi107.stdout:11257: writing smithi10729407-6 from 466944 to 991232 tid 1 2020-06-12T02:39:17.622 INFO:tasks.rados.rados.0.smithi107.stdout:11257: writing smithi10729407-6 from 991232 to 1761280 tid 2 2020-06-12T02:39:17.622 INFO:tasks.rados.rados.0.smithi107.stdout:11257: writing smithi10729407-6 from 1761280 to 1794048 tid 3 2020-06-12T02:39:17.623 INFO:tasks.rados.rados.0.smithi107.stdout:11258: delete oid 19 current snap is 0 2020-06-12T02:39:17.623 INFO:tasks.rados.rados.0.smithi107.stdout:11256: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:17.623 INFO:tasks.rados.rados.0.smithi107.stdout:11256: finishing write tid 2 to smithi10729407-14 2020-06-12T02:39:17.623 INFO:tasks.rados.rados.0.smithi107.stdout:11256: finishing write tid 3 to smithi10729407-14 2020-06-12T02:39:17.623 INFO:tasks.rados.rados.0.smithi107.stdout:11256: finishing write tid 5 to smithi10729407-14 2020-06-12T02:39:17.623 INFO:tasks.rados.rados.0.smithi107.stdout:11256: finishing write tid 6 to smithi10729407-14 2020-06-12T02:39:17.624 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1766 (ObjNum 11255 snap 0 seq_num 11255) dirty exists 2020-06-12T02:39:17.624 INFO:tasks.rados.rados.0.smithi107.stdout:11256: left oid 14 (ObjNum 11255 snap 0 seq_num 11255) 2020-06-12T02:39:17.624 INFO:tasks.rados.rados.0.smithi107.stdout:11257: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:17.624 INFO:tasks.rados.rados.0.smithi107.stdout:11256: done (2 left) 2020-06-12T02:39:17.624 INFO:tasks.rados.rados.0.smithi107.stdout:11258: done (1 left) 2020-06-12T02:39:17.625 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:39:17.625 INFO:tasks.rados.rados.0.smithi107.stdout:11259: seq_num 11258 ranges {0=557056,557056=540672,1097728=262144} 2020-06-12T02:39:17.625 INFO:tasks.rados.rados.0.smithi107.stdout:11259: writing smithi10729407-3 from 0 to 557056 tid 1 2020-06-12T02:39:17.628 INFO:tasks.rados.rados.0.smithi107.stdout:11259: writing smithi10729407-3 from 557056 to 1097728 tid 2 2020-06-12T02:39:17.628 INFO:tasks.rados.rados.0.smithi107.stdout:11259: writing smithi10729407-3 from 1097728 to 1359872 tid 3 2020-06-12T02:39:17.629 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:39:17.629 INFO:tasks.rados.rados.0.smithi107.stdout:11260: seq_num 11259 ranges {0=573440,573440=425984,999424=376832} 2020-06-12T02:39:17.629 INFO:tasks.rados.rados.0.smithi107.stdout:11260: writing smithi10729407-19 from 0 to 573440 tid 1 2020-06-12T02:39:17.629 INFO:tasks.rados.rados.0.smithi107.stdout:11260: writing smithi10729407-19 from 573440 to 999424 tid 2 2020-06-12T02:39:17.630 INFO:tasks.rados.rados.0.smithi107.stdout:11260: writing smithi10729407-19 from 999424 to 1376256 tid 3 2020-06-12T02:39:17.630 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:39:17.630 INFO:tasks.rados.rados.0.smithi107.stdout:11261: seq_num 11260 ranges {1187840=589824,1777664=761856,2539520=40960} 2020-06-12T02:39:17.630 INFO:tasks.rados.rados.0.smithi107.stdout:11261: writing smithi10729407-10 from 1187840 to 1777664 tid 1 2020-06-12T02:39:17.631 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:39:17.624+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:39:17.632 INFO:tasks.rados.rados.0.smithi107.stdout:11261: writing smithi10729407-10 from 1777664 to 2539520 tid 2 2020-06-12T02:39:17.632 INFO:tasks.rados.rados.0.smithi107.stdout:11261: writing smithi10729407-10 from 2539520 to 2580480 tid 3 2020-06-12T02:39:17.632 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:39:17.632 INFO:tasks.rados.rados.0.smithi107.stdout:11262: seq_num 11261 ranges {0=606208,606208=647168,1253376=155648} 2020-06-12T02:39:17.637 INFO:tasks.rados.rados.0.smithi107.stdout:11262: writing smithi10729407-12 from 0 to 606208 tid 1 2020-06-12T02:39:17.639 INFO:tasks.rados.rados.0.smithi107.stdout:11262: writing smithi10729407-12 from 606208 to 1253376 tid 2 2020-06-12T02:39:17.642 INFO:tasks.rados.rados.0.smithi107.stdout:11262: writing smithi10729407-12 from 1253376 to 1409024 tid 3 2020-06-12T02:39:17.642 INFO:tasks.rados.rados.0.smithi107.stdout:11263: delete oid 20 current snap is 0 2020-06-12T02:39:17.642 INFO:tasks.rados.rados.0.smithi107.stdout:11257: finishing write tid 2 to smithi10729407-6 2020-06-12T02:39:17.642 INFO:tasks.rados.rados.0.smithi107.stdout:11257: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:17.643 INFO:tasks.rados.rados.0.smithi107.stdout:11257: finishing write tid 5 to smithi10729407-6 2020-06-12T02:39:17.643 INFO:tasks.rados.rados.0.smithi107.stdout:11257: finishing write tid 6 to smithi10729407-6 2020-06-12T02:39:17.643 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 2024 (ObjNum 11256 snap 0 seq_num 11256) dirty exists 2020-06-12T02:39:17.643 INFO:tasks.rados.rados.0.smithi107.stdout:11257: left oid 6 (ObjNum 11256 snap 0 seq_num 11256) 2020-06-12T02:39:17.644 INFO:tasks.rados.rados.0.smithi107.stdout:11259: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:17.644 INFO:tasks.rados.rados.0.smithi107.stdout:11259: finishing write tid 2 to smithi10729407-3 2020-06-12T02:39:17.644 INFO:tasks.rados.rados.0.smithi107.stdout:11259: finishing write tid 3 to smithi10729407-3 2020-06-12T02:39:17.644 INFO:tasks.rados.rados.0.smithi107.stdout:11259: finishing write tid 5 to smithi10729407-3 2020-06-12T02:39:17.644 INFO:tasks.rados.rados.0.smithi107.stdout:11259: finishing write tid 6 to smithi10729407-3 2020-06-12T02:39:17.645 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1987 (ObjNum 11258 snap 0 seq_num 11258) dirty exists 2020-06-12T02:39:17.645 INFO:tasks.rados.rados.0.smithi107.stdout:11259: left oid 3 (ObjNum 11258 snap 0 seq_num 11258) 2020-06-12T02:39:17.645 INFO:tasks.rados.rados.0.smithi107.stdout:11260: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:17.645 INFO:tasks.rados.rados.0.smithi107.stdout:11260: finishing write tid 2 to smithi10729407-19 2020-06-12T02:39:17.645 INFO:tasks.rados.rados.0.smithi107.stdout:11260: finishing write tid 3 to smithi10729407-19 2020-06-12T02:39:17.645 INFO:tasks.rados.rados.0.smithi107.stdout:11260: finishing write tid 5 to smithi10729407-19 2020-06-12T02:39:17.646 INFO:tasks.rados.rados.0.smithi107.stdout:11260: finishing write tid 6 to smithi10729407-19 2020-06-12T02:39:17.646 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 2084 (ObjNum 11259 snap 0 seq_num 11259) dirty exists 2020-06-12T02:39:17.646 INFO:tasks.rados.rados.0.smithi107.stdout:11260: left oid 19 (ObjNum 11259 snap 0 seq_num 11259) 2020-06-12T02:39:17.646 INFO:tasks.rados.rados.0.smithi107.stdout:11261: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:17.646 INFO:tasks.rados.rados.0.smithi107.stdout:11261: finishing write tid 2 to smithi10729407-10 2020-06-12T02:39:17.647 INFO:tasks.rados.rados.0.smithi107.stdout:11261: finishing write tid 3 to smithi10729407-10 2020-06-12T02:39:17.647 INFO:tasks.rados.rados.0.smithi107.stdout:11261: finishing write tid 5 to smithi10729407-10 2020-06-12T02:39:17.648 INFO:tasks.rados.rados.0.smithi107.stdout:11261: finishing write tid 6 to smithi10729407-10 2020-06-12T02:39:17.648 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 845 (ObjNum 11260 snap 0 seq_num 11260) dirty exists 2020-06-12T02:39:17.648 INFO:tasks.rados.rados.0.smithi107.stdout:11261: left oid 10 (ObjNum 11260 snap 0 seq_num 11260) 2020-06-12T02:39:17.648 INFO:tasks.rados.rados.0.smithi107.stdout:11257: done (5 left) 2020-06-12T02:39:17.649 INFO:tasks.rados.rados.0.smithi107.stdout:11259: done (4 left) 2020-06-12T02:39:17.649 INFO:tasks.rados.rados.0.smithi107.stdout:11260: done (3 left) 2020-06-12T02:39:17.649 INFO:tasks.rados.rados.0.smithi107.stdout:11261: done (2 left) 2020-06-12T02:39:17.649 INFO:tasks.rados.rados.0.smithi107.stdout:11263: done (1 left) 2020-06-12T02:39:17.649 INFO:tasks.rados.rados.0.smithi107.stdout:11264: delete oid 8 current snap is 0 2020-06-12T02:39:17.649 INFO:tasks.rados.rados.0.smithi107.stdout:11262: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:17.650 INFO:tasks.rados.rados.0.smithi107.stdout:11264: done (1 left) 2020-06-12T02:39:17.650 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:39:17.650 INFO:tasks.rados.rados.0.smithi107.stdout:11265: seq_num 11264 ranges {0=655360,655360=696320,1351680=106496} 2020-06-12T02:39:17.650 INFO:tasks.rados.rados.0.smithi107.stdout:11265: writing smithi10729407-15 from 0 to 655360 tid 1 2020-06-12T02:39:17.654 INFO:tasks.rados.rados.0.smithi107.stdout:11265: writing smithi10729407-15 from 655360 to 1351680 tid 2 2020-06-12T02:39:17.654 INFO:tasks.rados.rados.0.smithi107.stdout:11265: writing smithi10729407-15 from 1351680 to 1458176 tid 3 2020-06-12T02:39:17.655 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:39:17.655 INFO:tasks.rados.rados.0.smithi107.stdout:11266: seq_num 11265 ranges {2580480=671744,3252224=581632,3833856=221184} 2020-06-12T02:39:17.663 INFO:teuthology.orchestra.run.smithi188.stderr:Exporting 2.5 info 2.5( empty local-lis/les=34/35 n=0 ec=16/16 lis/c=34/34 les/c/f=35/35/0 sis=34 pruub=11.334724061s) 2020-06-12T02:39:17.666 INFO:teuthology.orchestra.run.smithi188.stderr:Export successful 2020-06-12T02:39:17.673 INFO:tasks.rados.rados.0.smithi107.stdout:11266: writing smithi10729407-10 from 2580480 to 3252224 tid 1 2020-06-12T02:39:17.679 INFO:tasks.rados.rados.0.smithi107.stdout:11266: writing smithi10729407-10 from 3252224 to 3833856 tid 2 2020-06-12T02:39:17.680 INFO:tasks.rados.rados.0.smithi107.stdout:11266: writing smithi10729407-10 from 3833856 to 4055040 tid 3 2020-06-12T02:39:17.680 INFO:tasks.rados.rados.0.smithi107.stdout:11267: delete oid 5 current snap is 0 2020-06-12T02:39:17.680 INFO:tasks.rados.rados.0.smithi107.stdout:11262: finishing write tid 2 to smithi10729407-12 2020-06-12T02:39:17.680 INFO:tasks.rados.rados.0.smithi107.stdout:11262: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:17.680 INFO:tasks.rados.rados.0.smithi107.stdout:11262: finishing write tid 5 to smithi10729407-12 2020-06-12T02:39:17.681 INFO:tasks.rados.rados.0.smithi107.stdout:11262: finishing write tid 6 to smithi10729407-12 2020-06-12T02:39:17.681 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 2088 (ObjNum 11261 snap 0 seq_num 11261) dirty exists 2020-06-12T02:39:17.681 INFO:tasks.rados.rados.0.smithi107.stdout:11262: left oid 12 (ObjNum 11261 snap 0 seq_num 11261) 2020-06-12T02:39:17.681 INFO:tasks.rados.rados.0.smithi107.stdout:11265: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:17.681 INFO:tasks.rados.rados.0.smithi107.stdout:11265: finishing write tid 2 to smithi10729407-15 2020-06-12T02:39:17.682 INFO:tasks.rados.rados.0.smithi107.stdout:11265: finishing write tid 3 to smithi10729407-15 2020-06-12T02:39:17.682 INFO:tasks.rados.rados.0.smithi107.stdout:11265: finishing write tid 5 to smithi10729407-15 2020-06-12T02:39:17.682 INFO:tasks.rados.rados.0.smithi107.stdout:11265: finishing write tid 6 to smithi10729407-15 2020-06-12T02:39:17.683 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 2095 (ObjNum 11264 snap 0 seq_num 11264) dirty exists 2020-06-12T02:39:17.683 INFO:tasks.rados.rados.0.smithi107.stdout:11265: left oid 15 (ObjNum 11264 snap 0 seq_num 11264) 2020-06-12T02:39:17.683 INFO:tasks.rados.rados.0.smithi107.stdout:11262: done (3 left) 2020-06-12T02:39:17.684 INFO:tasks.rados.rados.0.smithi107.stdout:11265: done (2 left) 2020-06-12T02:39:17.684 INFO:tasks.rados.rados.0.smithi107.stdout:11267: done (1 left) 2020-06-12T02:39:17.684 INFO:tasks.rados.rados.0.smithi107.stdout:11268: delete oid 4 current snap is 0 2020-06-12T02:39:17.684 INFO:tasks.rados.rados.0.smithi107.stdout:11266: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:17.685 INFO:tasks.rados.rados.0.smithi107.stdout:11268: done (1 left) 2020-06-12T02:39:17.685 INFO:tasks.rados.rados.0.smithi107.stdout:11269: delete oid 13 current snap is 0 2020-06-12T02:39:17.688 INFO:tasks.rados.rados.0.smithi107.stdout:11269: done (1 left) 2020-06-12T02:39:17.688 INFO:tasks.rados.rados.0.smithi107.stdout:11270: delete oid 14 current snap is 0 2020-06-12T02:39:17.690 INFO:tasks.rados.rados.0.smithi107.stdout:11266: finishing write tid 2 to smithi10729407-10 2020-06-12T02:39:17.691 INFO:tasks.rados.rados.0.smithi107.stdout:11266: finishing write tid 3 to smithi10729407-10 2020-06-12T02:39:17.691 INFO:tasks.rados.rados.0.smithi107.stdout:11270: done (1 left) 2020-06-12T02:39:17.691 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:39:17.691 INFO:tasks.rados.rados.0.smithi107.stdout:11271: seq_num 11270 ranges {0=753664,753664=450560,1204224=352256} 2020-06-12T02:39:17.695 INFO:tasks.rados.rados.0.smithi107.stdout:11271: writing smithi10729407-17 from 0 to 753664 tid 1 2020-06-12T02:39:17.697 INFO:tasks.rados.rados.0.smithi107.stdout:11271: writing smithi10729407-17 from 753664 to 1204224 tid 2 2020-06-12T02:39:17.699 INFO:tasks.rados.rados.0.smithi107.stdout:11271: writing smithi10729407-17 from 1204224 to 1556480 tid 3 2020-06-12T02:39:17.700 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:39:17.700 INFO:tasks.rados.rados.0.smithi107.stdout:11272: seq_num 11271 ranges {0=770048,770048=737280,1507328=65536} 2020-06-12T02:39:17.704 INFO:tasks.rados.rados.0.smithi107.stdout:11272: writing smithi10729407-9 from 0 to 770048 tid 1 2020-06-12T02:39:17.709 INFO:tasks.rados.rados.0.smithi107.stdout:11272: writing smithi10729407-9 from 770048 to 1507328 tid 2 2020-06-12T02:39:17.709 INFO:tasks.rados.rados.0.smithi107.stdout:11272: writing smithi10729407-9 from 1507328 to 1572864 tid 3 2020-06-12T02:39:17.710 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:39:17.710 INFO:tasks.rados.rados.0.smithi107.stdout:11273: seq_num 11272 ranges {1794048=794624,2588672=622592,3211264=180224} 2020-06-12T02:39:17.724 INFO:tasks.rados.rados.0.smithi107.stdout:11273: writing smithi10729407-6 from 1794048 to 2588672 tid 1 2020-06-12T02:39:17.727 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:39:17.725+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:39:17.728 INFO:tasks.rados.rados.0.smithi107.stdout:11273: writing smithi10729407-6 from 2588672 to 3211264 tid 2 2020-06-12T02:39:17.729 INFO:tasks.rados.rados.0.smithi107.stdout:11273: writing smithi10729407-6 from 3211264 to 3391488 tid 3 2020-06-12T02:39:17.730 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:39:17.730 INFO:tasks.rados.rados.0.smithi107.stdout:11274: seq_num 11273 ranges {1376256=409600,1785856=507904,2293760=589824,2883584=106496} 2020-06-12T02:39:17.740 INFO:tasks.rados.rados.0.smithi107.stdout:11274: writing smithi10729407-19 from 1376256 to 1785856 tid 1 2020-06-12T02:39:17.743 INFO:tasks.rados.rados.0.smithi107.stdout:11274: writing smithi10729407-19 from 1785856 to 2293760 tid 2 2020-06-12T02:39:17.746 INFO:tasks.rados.rados.0.smithi107.stdout:11274: writing smithi10729407-19 from 2293760 to 2883584 tid 3 2020-06-12T02:39:17.748 INFO:tasks.rados.rados.0.smithi107.stdout:11274: writing smithi10729407-19 from 2883584 to 2990080 tid 4 2020-06-12T02:39:17.748 INFO:tasks.rados.rados.0.smithi107.stdout:11275: delete oid 16 current snap is 0 2020-06-12T02:39:17.748 INFO:tasks.rados.rados.0.smithi107.stdout:11266: finishing write tid 5 to smithi10729407-10 2020-06-12T02:39:17.748 INFO:tasks.rados.rados.0.smithi107.stdout:11266: finishing write tid 6 to smithi10729407-10 2020-06-12T02:39:17.749 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 849 (ObjNum 11265 snap 0 seq_num 11265) dirty exists 2020-06-12T02:39:17.749 INFO:tasks.rados.rados.0.smithi107.stdout:11266: left oid 10 (ObjNum 11265 snap 0 seq_num 11265) 2020-06-12T02:39:17.749 INFO:tasks.rados.rados.0.smithi107.stdout:11271: finishing write tid 1 to smithi10729407-17 2020-06-12T02:39:17.749 INFO:tasks.rados.rados.0.smithi107.stdout:11271: finishing write tid 2 to smithi10729407-17 2020-06-12T02:39:17.750 INFO:tasks.rados.rados.0.smithi107.stdout:11271: finishing write tid 3 to smithi10729407-17 2020-06-12T02:39:17.750 INFO:tasks.rados.rados.0.smithi107.stdout:11271: finishing write tid 5 to smithi10729407-17 2020-06-12T02:39:17.750 INFO:tasks.rados.rados.0.smithi107.stdout:11272: finishing write tid 1 to smithi10729407-9 2020-06-12T02:39:17.750 INFO:tasks.rados.rados.0.smithi107.stdout:11272: finishing write tid 2 to smithi10729407-9 2020-06-12T02:39:17.750 INFO:tasks.rados.rados.0.smithi107.stdout:11272: finishing write tid 3 to smithi10729407-9 2020-06-12T02:39:17.751 INFO:tasks.rados.rados.0.smithi107.stdout:11271: finishing write tid 6 to smithi10729407-17 2020-06-12T02:39:17.751 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 2104 (ObjNum 11270 snap 0 seq_num 11270) dirty exists 2020-06-12T02:39:17.751 INFO:tasks.rados.rados.0.smithi107.stdout:11271: left oid 17 (ObjNum 11270 snap 0 seq_num 11270) 2020-06-12T02:39:17.751 INFO:tasks.rados.rados.0.smithi107.stdout:11272: finishing write tid 5 to smithi10729407-9 2020-06-12T02:39:17.751 INFO:tasks.rados.rados.0.smithi107.stdout:11272: finishing write tid 6 to smithi10729407-9 2020-06-12T02:39:17.752 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1865 (ObjNum 11271 snap 0 seq_num 11271) dirty exists 2020-06-12T02:39:17.752 INFO:tasks.rados.rados.0.smithi107.stdout:11272: left oid 9 (ObjNum 11271 snap 0 seq_num 11271) 2020-06-12T02:39:17.752 INFO:tasks.rados.rados.0.smithi107.stdout:11273: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:17.752 INFO:tasks.rados.rados.0.smithi107.stdout:11273: finishing write tid 2 to smithi10729407-6 2020-06-12T02:39:17.752 INFO:tasks.rados.rados.0.smithi107.stdout:11273: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:17.753 INFO:tasks.rados.rados.0.smithi107.stdout:11273: finishing write tid 5 to smithi10729407-6 2020-06-12T02:39:17.753 INFO:tasks.rados.rados.0.smithi107.stdout:11273: finishing write tid 6 to smithi10729407-6 2020-06-12T02:39:17.753 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 2028 (ObjNum 11272 snap 0 seq_num 11272) dirty exists 2020-06-12T02:39:17.753 INFO:tasks.rados.rados.0.smithi107.stdout:11273: left oid 6 (ObjNum 11272 snap 0 seq_num 11272) 2020-06-12T02:39:17.754 INFO:tasks.rados.rados.0.smithi107.stdout:11274: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:17.754 INFO:tasks.rados.rados.0.smithi107.stdout:11266: done (5 left) 2020-06-12T02:39:17.755 INFO:tasks.rados.rados.0.smithi107.stdout:11271: done (4 left) 2020-06-12T02:39:17.755 INFO:tasks.rados.rados.0.smithi107.stdout:11272: done (3 left) 2020-06-12T02:39:17.755 INFO:tasks.rados.rados.0.smithi107.stdout:11273: done (2 left) 2020-06-12T02:39:17.755 INFO:tasks.rados.rados.0.smithi107.stdout:11275: done (1 left) 2020-06-12T02:39:17.755 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:39:17.756 INFO:tasks.rados.rados.0.smithi107.stdout:11276: seq_num 11275 ranges {704512=442368,1146880=720896,1867776=483328} 2020-06-12T02:39:17.758 INFO:tasks.rados.rados.0.smithi107.stdout:11276: writing smithi10729407-7 from 704512 to 1146880 tid 1 2020-06-12T02:39:17.762 INFO:tasks.rados.rados.0.smithi107.stdout:11276: writing smithi10729407-7 from 1146880 to 1867776 tid 2 2020-06-12T02:39:17.763 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:39:17.764 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:39:17.764 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2020-06-12T02:39:17.764 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:39:17.765 INFO:tasks.rados.rados.0.smithi107.stdout:11276: writing smithi10729407-7 from 1867776 to 2351104 tid 3 2020-06-12T02:39:17.765 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:39:17.765 INFO:tasks.rados.rados.0.smithi107.stdout:11277: seq_num 11276 ranges {4055040=458752,4513792=606208,5120000=409600,5529600=188416} 2020-06-12T02:39:17.770 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd unset noscrub 2020-06-12T02:39:17.774 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-06-12T02:39:17.790 INFO:tasks.rados.rados.0.smithi107.stdout:11277: writing smithi10729407-10 from 4055040 to 4513792 tid 1 2020-06-12T02:39:17.793 INFO:tasks.rados.rados.0.smithi107.stdout:11277: writing smithi10729407-10 from 4513792 to 5120000 tid 2 2020-06-12T02:39:17.796 INFO:tasks.rados.rados.0.smithi107.stdout:11277: writing smithi10729407-10 from 5120000 to 5529600 tid 3 2020-06-12T02:39:17.797 INFO:tasks.rados.rados.0.smithi107.stdout:11277: writing smithi10729407-10 from 5529600 to 5718016 tid 4 2020-06-12T02:39:17.797 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:39:17.798 INFO:tasks.rados.rados.0.smithi107.stdout:11278: seq_num 11277 ranges {3391488=475136,3866624=491520,4358144=565248,4923392=147456} 2020-06-12T02:39:17.820 INFO:tasks.rados.rados.0.smithi107.stdout:11278: writing smithi10729407-6 from 3391488 to 3866624 tid 1 2020-06-12T02:39:17.823 INFO:tasks.rados.rados.0.smithi107.stdout:11278: writing smithi10729407-6 from 3866624 to 4358144 tid 2 2020-06-12T02:39:17.828 INFO:tasks.rados.rados.0.smithi107.stdout:11278: writing smithi10729407-6 from 4358144 to 4923392 tid 3 2020-06-12T02:39:17.829 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:39:17.825+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:39:17.829 INFO:tasks.rados.rados.0.smithi107.stdout:11278: writing smithi10729407-6 from 4923392 to 5070848 tid 4 2020-06-12T02:39:17.829 INFO:tasks.rados.rados.0.smithi107.stdout:11279: delete oid 11 current snap is 0 2020-06-12T02:39:17.830 INFO:tasks.rados.rados.0.smithi107.stdout:11274: finishing write tid 2 to smithi10729407-19 2020-06-12T02:39:17.830 INFO:tasks.rados.rados.0.smithi107.stdout:11274: finishing write tid 3 to smithi10729407-19 2020-06-12T02:39:17.830 INFO:tasks.rados.rados.0.smithi107.stdout:11274: finishing write tid 4 to smithi10729407-19 2020-06-12T02:39:17.830 INFO:tasks.rados.rados.0.smithi107.stdout:11274: finishing write tid 6 to smithi10729407-19 2020-06-12T02:39:17.830 INFO:tasks.rados.rados.0.smithi107.stdout:11274: finishing write tid 7 to smithi10729407-19 2020-06-12T02:39:17.831 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 2093 (ObjNum 11273 snap 0 seq_num 11273) dirty exists 2020-06-12T02:39:17.831 INFO:tasks.rados.rados.0.smithi107.stdout:11274: left oid 19 (ObjNum 11273 snap 0 seq_num 11273) 2020-06-12T02:39:17.831 INFO:tasks.rados.rados.0.smithi107.stdout:11276: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:17.831 INFO:tasks.rados.rados.0.smithi107.stdout:11276: finishing write tid 2 to smithi10729407-7 2020-06-12T02:39:17.831 INFO:tasks.rados.rados.0.smithi107.stdout:11276: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:17.831 INFO:tasks.rados.rados.0.smithi107.stdout:11276: finishing write tid 5 to smithi10729407-7 2020-06-12T02:39:17.832 INFO:tasks.rados.rados.0.smithi107.stdout:11276: finishing write tid 6 to smithi10729407-7 2020-06-12T02:39:17.832 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1822 (ObjNum 11275 snap 0 seq_num 11275) dirty exists 2020-06-12T02:39:17.832 INFO:tasks.rados.rados.0.smithi107.stdout:11276: left oid 7 (ObjNum 11275 snap 0 seq_num 11275) 2020-06-12T02:39:17.832 INFO:tasks.rados.rados.0.smithi107.stdout:11277: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:17.832 INFO:tasks.rados.rados.0.smithi107.stdout:11277: finishing write tid 2 to smithi10729407-10 2020-06-12T02:39:17.833 INFO:tasks.rados.rados.0.smithi107.stdout:11277: finishing write tid 3 to smithi10729407-10 2020-06-12T02:39:17.833 INFO:tasks.rados.rados.0.smithi107.stdout:11277: finishing write tid 4 to smithi10729407-10 2020-06-12T02:39:17.833 INFO:tasks.rados.rados.0.smithi107.stdout:11277: finishing write tid 6 to smithi10729407-10 2020-06-12T02:39:17.842 INFO:tasks.rados.rados.0.smithi107.stdout:11277: finishing write tid 7 to smithi10729407-10 2020-06-12T02:39:17.842 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 854 (ObjNum 11276 snap 0 seq_num 11276) dirty exists 2020-06-12T02:39:17.842 INFO:tasks.rados.rados.0.smithi107.stdout:11277: left oid 10 (ObjNum 11276 snap 0 seq_num 11276) 2020-06-12T02:39:17.843 INFO:tasks.rados.rados.0.smithi107.stdout:11278: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:17.844 INFO:tasks.rados.rados.0.smithi107.stdout:11278: finishing write tid 2 to smithi10729407-6 2020-06-12T02:39:17.844 INFO:tasks.rados.rados.0.smithi107.stdout:11278: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:17.844 INFO:tasks.rados.rados.0.smithi107.stdout:11278: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:17.845 INFO:tasks.rados.rados.0.smithi107.stdout:11278: finishing write tid 6 to smithi10729407-6 2020-06-12T02:39:17.845 INFO:tasks.rados.rados.0.smithi107.stdout:11274: done (4 left) 2020-06-12T02:39:17.845 INFO:tasks.rados.rados.0.smithi107.stdout:11276: done (3 left) 2020-06-12T02:39:17.845 INFO:tasks.rados.rados.0.smithi107.stdout:11277: done (2 left) 2020-06-12T02:39:17.845 INFO:tasks.rados.rados.0.smithi107.stdout:11279: done (1 left) 2020-06-12T02:39:17.846 INFO:tasks.rados.rados.0.smithi107.stdout:11280: delete oid 18 current snap is 0 2020-06-12T02:39:17.846 INFO:tasks.rados.rados.0.smithi107.stdout:11278: finishing write tid 7 to smithi10729407-6 2020-06-12T02:39:17.846 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 2033 (ObjNum 11277 snap 0 seq_num 11277) dirty exists 2020-06-12T02:39:17.846 INFO:tasks.rados.rados.0.smithi107.stdout:11278: left oid 6 (ObjNum 11277 snap 0 seq_num 11277) 2020-06-12T02:39:17.846 INFO:tasks.rados.rados.0.smithi107.stdout:11278: done (1 left) 2020-06-12T02:39:17.846 INFO:tasks.rados.rados.0.smithi107.stdout:11280: done (0 left) 2020-06-12T02:39:17.847 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:39:17.847 INFO:tasks.rados.rados.0.smithi107.stdout:11281: seq_num 11280 ranges {2351104=524288,2875392=548864,3424256=655360} 2020-06-12T02:39:17.865 INFO:tasks.rados.rados.0.smithi107.stdout:11281: writing smithi10729407-7 from 2351104 to 2875392 tid 1 2020-06-12T02:39:17.866 INFO:tasks.rados.rados.0.smithi107.stdout:11281: writing smithi10729407-7 from 2875392 to 3424256 tid 2 2020-06-12T02:39:17.866 INFO:tasks.rados.rados.0.smithi107.stdout:11281: writing smithi10729407-7 from 3424256 to 4079616 tid 3 2020-06-12T02:39:17.866 INFO:tasks.rados.rados.0.smithi107.stdout:11282: delete oid 19 current snap is 0 2020-06-12T02:39:17.866 INFO:tasks.rados.rados.0.smithi107.stdout:11281: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:17.867 INFO:tasks.rados.rados.0.smithi107.stdout:11282: done (1 left) 2020-06-12T02:39:17.867 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:39:17.867 INFO:tasks.rados.rados.0.smithi107.stdout:11283: seq_num 11282 ranges {0=557056,557056=761856,1318912=442368} 2020-06-12T02:39:17.874 INFO:tasks.rados.rados.0.smithi107.stdout:11283: writing smithi10729407-1 from 0 to 557056 tid 1 2020-06-12T02:39:17.875 INFO:tasks.rados.rados.0.smithi107.stdout:11283: writing smithi10729407-1 from 557056 to 1318912 tid 2 2020-06-12T02:39:17.877 INFO:tasks.rados.rados.0.smithi107.stdout:11283: writing smithi10729407-1 from 1318912 to 1761280 tid 3 2020-06-12T02:39:17.877 INFO:tasks.rados.rados.0.smithi107.stdout:11284: delete oid 12 current snap is 0 2020-06-12T02:39:17.878 INFO:tasks.rados.rados.0.smithi107.stdout:11281: finishing write tid 2 to smithi10729407-7 2020-06-12T02:39:17.878 INFO:tasks.rados.rados.0.smithi107.stdout:11281: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:17.878 INFO:tasks.rados.rados.0.smithi107.stdout:11281: finishing write tid 5 to smithi10729407-7 2020-06-12T02:39:17.878 INFO:tasks.rados.rados.0.smithi107.stdout:11281: finishing write tid 6 to smithi10729407-7 2020-06-12T02:39:17.878 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1826 (ObjNum 11280 snap 0 seq_num 11280) dirty exists 2020-06-12T02:39:17.879 INFO:tasks.rados.rados.0.smithi107.stdout:11281: left oid 7 (ObjNum 11280 snap 0 seq_num 11280) 2020-06-12T02:39:17.880 INFO:tasks.rados.rados.0.smithi107.stdout:11281: done (2 left) 2020-06-12T02:39:17.880 INFO:tasks.rados.rados.0.smithi107.stdout:11284: done (1 left) 2020-06-12T02:39:17.881 INFO:tasks.rados.rados.0.smithi107.stdout:11285: delete oid 7 current snap is 0 2020-06-12T02:39:17.890 INFO:tasks.rados.rados.0.smithi107.stdout:11285: done (1 left) 2020-06-12T02:39:17.891 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:39:17.891 INFO:tasks.rados.rados.0.smithi107.stdout:11286: seq_num 11285 ranges {5718016=606208,6324224=417792,6742016=573440,7315456=212992} 2020-06-12T02:39:17.918 INFO:tasks.rados.rados.0.smithi107.stdout:11286: writing smithi10729407-10 from 5718016 to 6324224 tid 1 2020-06-12T02:39:17.919 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:39:17.919 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:39:17.919 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2020-06-12T02:39:17.920 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2020-06-12T02:39:17.920 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:39:17.920 INFO:tasks.rados.rados.0.smithi107.stdout:11286: writing smithi10729407-10 from 6324224 to 6742016 tid 2 2020-06-12T02:39:17.924 INFO:tasks.rados.rados.0.smithi107.stdout:11286: writing smithi10729407-10 from 6742016 to 7315456 tid 3 2020-06-12T02:39:17.925 INFO:tasks.rados.rados.0.smithi107.stdout:11286: writing smithi10729407-10 from 7315456 to 7528448 tid 4 2020-06-12T02:39:17.926 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:39:17.926 INFO:tasks.rados.rados.0.smithi107.stdout:11287: seq_num 11286 ranges {1556480=622592,2179072=704512,2883584=499712} 2020-06-12T02:39:17.929 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:39:17.926+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:39:17.930 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-06-12T02:39:17.938 INFO:tasks.rados.rados.0.smithi107.stdout:11287: writing smithi10729407-17 from 1556480 to 2179072 tid 1 2020-06-12T02:39:17.942 INFO:tasks.rados.rados.0.smithi107.stdout:11287: writing smithi10729407-17 from 2179072 to 2883584 tid 2 2020-06-12T02:39:17.945 INFO:tasks.rados.rados.0.smithi107.stdout:11287: writing smithi10729407-17 from 2883584 to 3383296 tid 3 2020-06-12T02:39:17.945 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:39:17.946 INFO:tasks.rados.rados.0.smithi107.stdout:11288: seq_num 11287 ranges {0=638976,638976=581632,1220608=540672,1761280=81920} 2020-06-12T02:39:17.949 INFO:tasks.rados.rados.0.smithi107.stdout:11288: writing smithi10729407-8 from 0 to 638976 tid 1 2020-06-12T02:39:17.952 INFO:tasks.rados.rados.0.smithi107.stdout:11288: writing smithi10729407-8 from 638976 to 1220608 tid 2 2020-06-12T02:39:17.956 INFO:tasks.rados.rados.0.smithi107.stdout:11288: writing smithi10729407-8 from 1220608 to 1761280 tid 3 2020-06-12T02:39:17.957 INFO:tasks.rados.rados.0.smithi107.stdout:11288: writing smithi10729407-8 from 1761280 to 1843200 tid 4 2020-06-12T02:39:17.957 INFO:tasks.rados.rados.0.smithi107.stdout:11289: delete oid 11 current snap is 0 2020-06-12T02:39:17.958 INFO:tasks.rados.rados.0.smithi107.stdout:11283: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:17.958 INFO:tasks.rados.rados.0.smithi107.stdout:11283: finishing write tid 2 to smithi10729407-1 2020-06-12T02:39:17.958 INFO:tasks.rados.rados.0.smithi107.stdout:11283: finishing write tid 3 to smithi10729407-1 2020-06-12T02:39:17.958 INFO:tasks.rados.rados.0.smithi107.stdout:11283: finishing write tid 5 to smithi10729407-1 2020-06-12T02:39:17.958 INFO:tasks.rados.rados.0.smithi107.stdout:11283: finishing write tid 6 to smithi10729407-1 2020-06-12T02:39:17.958 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 2037 (ObjNum 11282 snap 0 seq_num 11282) dirty exists 2020-06-12T02:39:17.959 INFO:tasks.rados.rados.0.smithi107.stdout:11283: left oid 1 (ObjNum 11282 snap 0 seq_num 11282) 2020-06-12T02:39:17.959 INFO:tasks.rados.rados.0.smithi107.stdout:11286: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:17.959 INFO:tasks.rados.rados.0.smithi107.stdout:11286: finishing write tid 2 to smithi10729407-10 2020-06-12T02:39:17.959 INFO:tasks.rados.rados.0.smithi107.stdout:11286: finishing write tid 3 to smithi10729407-10 2020-06-12T02:39:17.959 INFO:tasks.rados.rados.0.smithi107.stdout:11286: finishing write tid 4 to smithi10729407-10 2020-06-12T02:39:17.960 INFO:tasks.rados.rados.0.smithi107.stdout:11286: finishing write tid 6 to smithi10729407-10 2020-06-12T02:39:17.960 INFO:tasks.rados.rados.0.smithi107.stdout:11286: finishing write tid 7 to smithi10729407-10 2020-06-12T02:39:17.960 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 859 (ObjNum 11285 snap 0 seq_num 11285) dirty exists 2020-06-12T02:39:17.960 INFO:tasks.rados.rados.0.smithi107.stdout:11286: left oid 10 (ObjNum 11285 snap 0 seq_num 11285) 2020-06-12T02:39:17.960 INFO:tasks.rados.rados.0.smithi107.stdout:11287: finishing write tid 1 to smithi10729407-17 2020-06-12T02:39:17.961 INFO:tasks.rados.rados.0.smithi107.stdout:11287: finishing write tid 2 to smithi10729407-17 2020-06-12T02:39:17.961 INFO:tasks.rados.rados.0.smithi107.stdout:11287: finishing write tid 3 to smithi10729407-17 2020-06-12T02:39:17.962 INFO:tasks.rados.rados.0.smithi107.stdout:11287: finishing write tid 5 to smithi10729407-17 2020-06-12T02:39:17.962 INFO:tasks.rados.rados.0.smithi107.stdout:11288: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:17.962 INFO:tasks.rados.rados.0.smithi107.stdout:11283: done (4 left) 2020-06-12T02:39:17.962 INFO:tasks.rados.rados.0.smithi107.stdout:11286: done (3 left) 2020-06-12T02:39:17.963 INFO:tasks.rados.rados.0.smithi107.stdout:11289: done (2 left) 2020-06-12T02:39:17.963 INFO:tasks.rados.rados.0.smithi107.stdout:11290: delete oid 12 current snap is 0 2020-06-12T02:39:17.963 INFO:tasks.rados.rados.0.smithi107.stdout:11287: finishing write tid 6 to smithi10729407-17 2020-06-12T02:39:17.964 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 2108 (ObjNum 11286 snap 0 seq_num 11286) dirty exists 2020-06-12T02:39:17.964 INFO:tasks.rados.rados.0.smithi107.stdout:11287: left oid 17 (ObjNum 11286 snap 0 seq_num 11286) 2020-06-12T02:39:17.965 INFO:tasks.rados.rados.0.smithi107.stdout:11287: done (2 left) 2020-06-12T02:39:17.965 INFO:tasks.rados.rados.0.smithi107.stdout:11290: done (1 left) 2020-06-12T02:39:17.965 INFO:tasks.rados.rados.0.smithi107.stdout:11291: delete oid 14 current snap is 0 2020-06-12T02:39:17.965 INFO:tasks.rados.rados.0.smithi107.stdout:11291: done (1 left) 2020-06-12T02:39:17.966 INFO:tasks.rados.rados.0.smithi107.stdout:11292: delete oid 18 current snap is 0 2020-06-12T02:39:17.967 INFO:tasks.rados.rados.0.smithi107.stdout:11288: finishing write tid 2 to smithi10729407-8 2020-06-12T02:39:17.969 INFO:tasks.rados.rados.0.smithi107.stdout:11292: done (1 left) 2020-06-12T02:39:17.969 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:39:17.970 INFO:tasks.rados.rados.0.smithi107.stdout:11293: seq_num 11292 ranges {0=729088,729088=458752,1187840=729088,1916928=16384} 2020-06-12T02:39:17.972 INFO:tasks.rados.rados.0.smithi107.stdout:11293: writing smithi10729407-19 from 0 to 729088 tid 1 2020-06-12T02:39:17.975 INFO:tasks.rados.rados.0.smithi107.stdout:11293: writing smithi10729407-19 from 729088 to 1187840 tid 2 2020-06-12T02:39:17.980 INFO:tasks.rados.rados.0.smithi107.stdout:11293: writing smithi10729407-19 from 1187840 to 1916928 tid 3 2020-06-12T02:39:17.981 INFO:tasks.rados.rados.0.smithi107.stdout:11293: writing smithi10729407-19 from 1916928 to 1933312 tid 4 2020-06-12T02:39:17.981 INFO:tasks.rados.rados.0.smithi107.stdout:11294: delete oid 5 current snap is 0 2020-06-12T02:39:17.981 INFO:tasks.rados.rados.0.smithi107.stdout:11288: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:17.981 INFO:tasks.rados.rados.0.smithi107.stdout:11288: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:17.981 INFO:tasks.rados.rados.0.smithi107.stdout:11288: finishing write tid 6 to smithi10729407-8 2020-06-12T02:39:17.982 INFO:tasks.rados.rados.0.smithi107.stdout:11288: finishing write tid 7 to smithi10729407-8 2020-06-12T02:39:17.982 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 2100 (ObjNum 11287 snap 0 seq_num 11287) dirty exists 2020-06-12T02:39:17.982 INFO:tasks.rados.rados.0.smithi107.stdout:11288: left oid 8 (ObjNum 11287 snap 0 seq_num 11287) 2020-06-12T02:39:17.984 INFO:tasks.rados.rados.0.smithi107.stdout:11288: done (2 left) 2020-06-12T02:39:17.984 INFO:tasks.rados.rados.0.smithi107.stdout:11294: done (1 left) 2020-06-12T02:39:17.984 INFO:tasks.rados.rados.0.smithi107.stdout:11295: delete oid 9 current snap is 0 2020-06-12T02:39:17.986 INFO:tasks.rados.rados.0.smithi107.stdout:11293: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:17.987 INFO:tasks.rados.rados.0.smithi107.stdout:11293: finishing write tid 2 to smithi10729407-19 2020-06-12T02:39:17.987 INFO:tasks.rados.rados.0.smithi107.stdout:11295: done (1 left) 2020-06-12T02:39:17.987 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:39:17.987 INFO:tasks.rados.rados.0.smithi107.stdout:11296: seq_num 11295 ranges {5070848=778240,5849088=507904,6356992=499712,6856704=196608} 2020-06-12T02:39:18.019 INFO:tasks.rados.rados.0.smithi107.stdout:11296: writing smithi10729407-6 from 5070848 to 5849088 tid 1 2020-06-12T02:39:18.022 INFO:tasks.rados.rados.0.smithi107.stdout:11296: writing smithi10729407-6 from 5849088 to 6356992 tid 2 2020-06-12T02:39:18.026 INFO:tasks.rados.rados.0.smithi107.stdout:11296: writing smithi10729407-6 from 6356992 to 6856704 tid 3 2020-06-12T02:39:18.028 INFO:tasks.rados.rados.0.smithi107.stdout:11296: writing smithi10729407-6 from 6856704 to 7053312 tid 4 2020-06-12T02:39:18.028 INFO:tasks.rados.rados.0.smithi107.stdout:11297: delete oid 7 current snap is 0 2020-06-12T02:39:18.028 INFO:tasks.rados.rados.0.smithi107.stdout:11293: finishing write tid 3 to smithi10729407-19 2020-06-12T02:39:18.028 INFO:tasks.rados.rados.0.smithi107.stdout:11293: finishing write tid 4 to smithi10729407-19 2020-06-12T02:39:18.029 INFO:tasks.rados.rados.0.smithi107.stdout:11293: finishing write tid 6 to smithi10729407-19 2020-06-12T02:39:18.029 INFO:tasks.rados.rados.0.smithi107.stdout:11293: finishing write tid 7 to smithi10729407-19 2020-06-12T02:39:18.029 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 2101 (ObjNum 11292 snap 0 seq_num 11292) dirty exists 2020-06-12T02:39:18.029 INFO:tasks.rados.rados.0.smithi107.stdout:11293: left oid 19 (ObjNum 11292 snap 0 seq_num 11292) 2020-06-12T02:39:18.030 INFO:tasks.rados.rados.0.smithi107.stdout:11296: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:18.030 INFO:tasks.rados.rados.0.smithi107.stdout:11293: done (2 left) 2020-06-12T02:39:18.030 INFO:tasks.rados.rados.0.smithi107.stdout:11297: done (1 left) 2020-06-12T02:39:18.031 INFO:tasks.rados.rados.0.smithi107.stdout:11298: delete oid 3 current snap is 0 2020-06-12T02:39:18.031 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:39:18.028+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:39:18.033 INFO:tasks.rados.rados.0.smithi107.stdout:11298: done (1 left) 2020-06-12T02:39:18.033 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:39:18.033 INFO:tasks.rados.rados.0.smithi107.stdout:11299: seq_num 11298 ranges {0=425984,425984=614400,1040384=679936,1720320=311296} 2020-06-12T02:39:18.036 INFO:tasks.rados.rados.0.smithi107.stdout:11299: writing smithi10729407-9 from 0 to 425984 tid 1 2020-06-12T02:39:18.040 INFO:tasks.rados.rados.0.smithi107.stdout:11299: writing smithi10729407-9 from 425984 to 1040384 tid 2 2020-06-12T02:39:18.045 INFO:tasks.rados.rados.0.smithi107.stdout:11299: writing smithi10729407-9 from 1040384 to 1720320 tid 3 2020-06-12T02:39:18.047 INFO:tasks.rados.rados.0.smithi107.stdout:11299: writing smithi10729407-9 from 1720320 to 2031616 tid 4 2020-06-12T02:39:18.047 INFO:tasks.rados.rados.0.smithi107.stdout:11300: delete oid 19 current snap is 0 2020-06-12T02:39:18.050 INFO:tasks.rados.rados.0.smithi107.stdout:11296: finishing write tid 2 to smithi10729407-6 2020-06-12T02:39:18.050 INFO:tasks.rados.rados.0.smithi107.stdout:11296: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:18.050 INFO:tasks.rados.rados.0.smithi107.stdout:11296: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:18.050 INFO:tasks.rados.rados.0.smithi107.stdout:11296: finishing write tid 6 to smithi10729407-6 2020-06-12T02:39:18.051 INFO:tasks.rados.rados.0.smithi107.stdout:11299: finishing write tid 1 to smithi10729407-9 2020-06-12T02:39:18.051 INFO:tasks.rados.rados.0.smithi107.stdout:11296: finishing write tid 7 to smithi10729407-6 2020-06-12T02:39:18.051 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 2042 (ObjNum 11295 snap 0 seq_num 11295) dirty exists 2020-06-12T02:39:18.051 INFO:tasks.rados.rados.0.smithi107.stdout:11296: left oid 6 (ObjNum 11295 snap 0 seq_num 11295) 2020-06-12T02:39:18.052 INFO:tasks.rados.rados.0.smithi107.stdout:11296: done (2 left) 2020-06-12T02:39:18.053 INFO:tasks.rados.rados.0.smithi107.stdout:11300: done (1 left) 2020-06-12T02:39:18.053 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:39:18.053 INFO:tasks.rados.rados.0.smithi107.stdout:11301: seq_num 11300 ranges {7053312=458752,7512064=778240,8290304=688128,8978432=139264} 2020-06-12T02:39:18.094 INFO:tasks.rados.rados.0.smithi107.stdout:11301: writing smithi10729407-6 from 7053312 to 7512064 tid 1 2020-06-12T02:39:18.099 INFO:tasks.rados.rados.0.smithi107.stdout:11301: writing smithi10729407-6 from 7512064 to 8290304 tid 2 2020-06-12T02:39:18.102 INFO:tasks.rados.rados.0.smithi107.stdout:11301: writing smithi10729407-6 from 8290304 to 8978432 tid 3 2020-06-12T02:39:18.103 INFO:tasks.rados.rados.0.smithi107.stdout:11301: writing smithi10729407-6 from 8978432 to 9117696 tid 4 2020-06-12T02:39:18.103 INFO:tasks.rados.rados.0.smithi107.stdout:11302: delete oid 19 current snap is 0 2020-06-12T02:39:18.104 INFO:tasks.rados.rados.0.smithi107.stdout:11299: finishing write tid 2 to smithi10729407-9 2020-06-12T02:39:18.104 INFO:tasks.rados.rados.0.smithi107.stdout:11299: finishing write tid 3 to smithi10729407-9 2020-06-12T02:39:18.104 INFO:tasks.rados.rados.0.smithi107.stdout:11299: finishing write tid 4 to smithi10729407-9 2020-06-12T02:39:18.104 INFO:tasks.rados.rados.0.smithi107.stdout:11299: finishing write tid 6 to smithi10729407-9 2020-06-12T02:39:18.104 INFO:tasks.rados.rados.0.smithi107.stdout:11299: finishing write tid 7 to smithi10729407-9 2020-06-12T02:39:18.104 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1871 (ObjNum 11298 snap 0 seq_num 11298) dirty exists 2020-06-12T02:39:18.105 INFO:tasks.rados.rados.0.smithi107.stdout:11299: left oid 9 (ObjNum 11298 snap 0 seq_num 11298) 2020-06-12T02:39:18.105 INFO:tasks.rados.rados.0.smithi107.stdout:11301: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:18.107 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:39:18.107 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2020-06-12T02:39:18.107 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2020-06-12T02:39:18.107 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2020-06-12T02:39:18.107 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.108 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:2782 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~696320] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:39:18.108 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:36.808170+0000", 2020-06-12T02:39:18.108 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 101.294329636, 2020-06-12T02:39:18.108 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016972839999999999, 2020-06-12T02:39:18.108 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:18.109 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:18.109 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:18.109 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:18.109 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:18.109 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2782 2020-06-12T02:39:18.110 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.110 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:18.110 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.110 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:18.110 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.808170+0000", 2020-06-12T02:39:18.111 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:18.113 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.114 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.114 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:18.114 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.808170+0000", 2020-06-12T02:39:18.114 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000261778 2020-06-12T02:39:18.114 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.115 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.115 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:18.115 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.808432+0000", 2020-06-12T02:39:18.115 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0920000000000003e-06 2020-06-12T02:39:18.115 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.116 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.116 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:18.116 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.808438+0000", 2020-06-12T02:39:18.117 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2539e-05 2020-06-12T02:39:18.117 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.117 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.117 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:18.118 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.808450+0000", 2020-06-12T02:39:18.118 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1859999999999997e-06 2020-06-12T02:39:18.118 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.118 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.118 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:18.119 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.808457+0000", 2020-06-12T02:39:18.119 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0988000000000001e-05 2020-06-12T02:39:18.120 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.120 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.121 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:18.121 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.808498+0000", 2020-06-12T02:39:18.121 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011723 2020-06-12T02:39:18.122 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.122 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.122 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:18.123 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.808615+0000", 2020-06-12T02:39:18.123 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0015648109999999999 2020-06-12T02:39:18.123 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.123 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.123 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:18.124 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.810180+0000", 2020-06-12T02:39:18.124 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1718999999999994e-05 2020-06-12T02:39:18.125 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.125 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.125 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:18.125 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.810271+0000", 2020-06-12T02:39:18.125 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00082076299999999996 2020-06-12T02:39:18.126 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.126 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.126 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:18.127 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.811092+0000", 2020-06-12T02:39:18.127 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01210689 2020-06-12T02:39:18.127 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.127 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.127 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:18.128 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.823199+0000", 2020-06-12T02:39:18.128 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017554300000000001 2020-06-12T02:39:18.128 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.128 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.128 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:39:18.128 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.824954+0000", 2020-06-12T02:39:18.129 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000188414 2020-06-12T02:39:18.129 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.129 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.129 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:18.129 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.825143+0000", 2020-06-12T02:39:18.130 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016972839999999999 2020-06-12T02:39:18.130 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.130 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:18.130 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.130 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.131 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.131 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:2785 3.0s0 3:0dddb690:::smithi10729407-10:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:39:18.131 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:36.813804+0000", 2020-06-12T02:39:18.131 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 101.288695366, 2020-06-12T02:39:18.131 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014100781999999999, 2020-06-12T02:39:18.132 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:18.132 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:18.132 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:18.132 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:18.133 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:18.133 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2785 2020-06-12T02:39:18.133 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.133 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:18.134 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.134 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:18.134 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813804+0000", 2020-06-12T02:39:18.134 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:18.134 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.135 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.135 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:18.135 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813804+0000", 2020-06-12T02:39:18.135 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1301999999999999e-05 2020-06-12T02:39:18.135 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.136 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.136 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:18.136 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813846+0000", 2020-06-12T02:39:18.136 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4530000000000004e-06 2020-06-12T02:39:18.136 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.136 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.137 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:18.137 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813850+0000", 2020-06-12T02:39:18.137 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7940000000000003e-06 2020-06-12T02:39:18.138 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.138 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.138 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:18.138 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813858+0000", 2020-06-12T02:39:18.138 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9450000000000002e-06 2020-06-12T02:39:18.138 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.139 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.139 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:18.139 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813861+0000", 2020-06-12T02:39:18.139 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0084749969999999997 2020-06-12T02:39:18.139 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.140 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.140 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:18.140 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.822336+0000", 2020-06-12T02:39:18.140 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013055400000000001 2020-06-12T02:39:18.141 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.141 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.141 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:18.141 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.822466+0000", 2020-06-12T02:39:18.141 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047301400000000001 2020-06-12T02:39:18.142 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.142 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.142 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:18.142 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.822939+0000", 2020-06-12T02:39:18.143 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086569499999999998 2020-06-12T02:39:18.143 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.143 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.144 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:18.144 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.823805+0000", 2020-06-12T02:39:18.144 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0039911460000000001 2020-06-12T02:39:18.144 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.144 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.144 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:39:18.145 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.827796+0000", 2020-06-12T02:39:18.145 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010888200000000001 2020-06-12T02:39:18.145 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.145 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.145 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:18.146 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.827905+0000", 2020-06-12T02:39:18.146 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014100781999999999 2020-06-12T02:39:18.146 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.146 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:18.146 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.146 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.147 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.148 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:2786 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:39:18.148 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:36.813885+0000", 2020-06-12T02:39:18.148 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 101.288614549, 2020-06-12T02:39:18.149 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015757724000000001, 2020-06-12T02:39:18.149 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:18.149 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:39:18.149 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:18.149 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:18.150 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:18.150 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 2786 2020-06-12T02:39:18.150 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.150 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:18.150 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.151 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:18.151 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813885+0000", 2020-06-12T02:39:18.151 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:18.152 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.152 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.152 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:18.152 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813885+0000", 2020-06-12T02:39:18.152 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2388999999999999e-05 2020-06-12T02:39:18.153 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.153 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.153 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:18.153 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813907+0000", 2020-06-12T02:39:18.153 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9660000000000001e-06 2020-06-12T02:39:18.154 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.154 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.154 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:18.154 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813910+0000", 2020-06-12T02:39:18.154 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7649999999999993e-06 2020-06-12T02:39:18.155 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.155 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.155 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:18.155 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813918+0000", 2020-06-12T02:39:18.155 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0870000000000001e-06 2020-06-12T02:39:18.156 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.156 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.156 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:18.156 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.813920+0000", 2020-06-12T02:39:18.156 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093684819999999992 2020-06-12T02:39:18.156 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.157 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.157 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:18.157 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.823289+0000", 2020-06-12T02:39:18.157 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103643 2020-06-12T02:39:18.157 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.158 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.158 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:18.158 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.823392+0000", 2020-06-12T02:39:18.158 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0050514449999999999 2020-06-12T02:39:18.158 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.159 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.159 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:18.159 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.828444+0000", 2020-06-12T02:39:18.159 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000102824 2020-06-12T02:39:18.159 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.159 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.160 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:18.160 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.828547+0000", 2020-06-12T02:39:18.160 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0010961230000000001 2020-06-12T02:39:18.160 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.160 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.161 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:18.161 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.829643+0000", 2020-06-12T02:39:18.161 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015757724000000001 2020-06-12T02:39:18.161 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.161 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:18.162 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.162 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.162 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.162 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:3330 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~630784] snapc 0=[] ondisk+write+known_if_redirected e27)", 2020-06-12T02:39:18.162 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:38.603794+0000", 2020-06-12T02:39:18.162 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 99.498705857999994, 2020-06-12T02:39:18.163 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016093917999999999, 2020-06-12T02:39:18.163 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:18.163 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:18.163 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:18.163 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:18.164 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:18.164 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 3330 2020-06-12T02:39:18.164 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.164 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:18.164 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.164 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:18.165 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.603794+0000", 2020-06-12T02:39:18.165 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:18.165 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.165 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.165 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:18.166 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.603794+0000", 2020-06-12T02:39:18.166 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00042303700000000002 2020-06-12T02:39:18.166 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.166 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.166 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:18.166 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.604217+0000", 2020-06-12T02:39:18.167 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9350000000000006e-06 2020-06-12T02:39:18.167 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.167 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.167 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:18.167 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.604227+0000", 2020-06-12T02:39:18.168 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9343000000000001e-05 2020-06-12T02:39:18.168 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.168 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.168 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:18.168 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.604246+0000", 2020-06-12T02:39:18.168 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2369999999999998e-06 2020-06-12T02:39:18.169 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.169 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.169 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:18.169 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.604255+0000", 2020-06-12T02:39:18.169 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1530000000000003e-05 2020-06-12T02:39:18.170 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.170 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.170 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:18.170 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.604307+0000", 2020-06-12T02:39:18.170 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103756 2020-06-12T02:39:18.171 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.171 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.171 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:18.171 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.604411+0000", 2020-06-12T02:39:18.171 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00073281400000000003 2020-06-12T02:39:18.172 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.172 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.172 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:18.172 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.605143+0000", 2020-06-12T02:39:18.172 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071483730000000004 2020-06-12T02:39:18.172 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.173 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.173 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:18.173 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.612292+0000", 2020-06-12T02:39:18.173 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0075185879999999997 2020-06-12T02:39:18.173 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.174 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.174 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:39:18.175 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.619810+0000", 2020-06-12T02:39:18.175 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7305000000000006e-05 2020-06-12T02:39:18.175 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.175 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.176 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:18.176 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.619888+0000", 2020-06-12T02:39:18.176 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016093917999999999 2020-06-12T02:39:18.176 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.176 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:18.177 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.177 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.177 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.177 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:3335 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e27)", 2020-06-12T02:39:18.177 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:38.616040+0000", 2020-06-12T02:39:18.177 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 99.486459119000003, 2020-06-12T02:39:18.178 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015075565000000001, 2020-06-12T02:39:18.178 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:18.178 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:39:18.178 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:18.178 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:18.179 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:18.179 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 3335 2020-06-12T02:39:18.179 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.179 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:18.179 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.179 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:18.180 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.616040+0000", 2020-06-12T02:39:18.180 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:18.180 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.180 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.180 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:18.181 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.616040+0000", 2020-06-12T02:39:18.181 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1950000000000002e-05 2020-06-12T02:39:18.181 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.181 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.181 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:18.182 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.616062+0000", 2020-06-12T02:39:18.182 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.1370000000000002e-06 2020-06-12T02:39:18.182 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.182 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.182 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:18.184 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.616066+0000", 2020-06-12T02:39:18.184 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5019999999999998e-06 2020-06-12T02:39:18.184 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.185 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.185 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:18.185 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.616072+0000", 2020-06-12T02:39:18.185 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3470000000000001e-06 2020-06-12T02:39:18.186 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.186 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.186 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:18.186 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.616074+0000", 2020-06-12T02:39:18.186 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0067702309999999998 2020-06-12T02:39:18.187 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.187 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.187 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:18.187 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.622845+0000", 2020-06-12T02:39:18.187 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7738000000000005e-05 2020-06-12T02:39:18.187 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.188 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.188 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:18.189 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.622942+0000", 2020-06-12T02:39:18.189 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0050881010000000003 2020-06-12T02:39:18.190 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.190 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.191 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:18.191 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.628031+0000", 2020-06-12T02:39:18.192 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000108171 2020-06-12T02:39:18.192 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.192 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.192 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:18.193 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.628139+0000", 2020-06-12T02:39:18.193 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0029773880000000001 2020-06-12T02:39:18.193 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.193 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.193 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:18.194 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.631116+0000", 2020-06-12T02:39:18.194 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015075565000000001 2020-06-12T02:39:18.194 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.194 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:18.194 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.195 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.195 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.195 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:4969 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~729088,stat] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-06-12T02:39:18.195 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:43.713430+0000", 2020-06-12T02:39:18.195 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 94.389069655, 2020-06-12T02:39:18.196 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.18980771900000001, 2020-06-12T02:39:18.196 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:18.196 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:18.196 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:18.196 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:18.197 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:18.197 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 4969 2020-06-12T02:39:18.197 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.197 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:18.197 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.197 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:18.198 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.713430+0000", 2020-06-12T02:39:18.198 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:18.198 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.198 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.199 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:18.199 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.713430+0000", 2020-06-12T02:39:18.199 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018522800000000001 2020-06-12T02:39:18.199 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.199 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.199 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:18.200 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.713615+0000", 2020-06-12T02:39:18.200 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3590000000000002e-06 2020-06-12T02:39:18.200 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.200 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.200 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:18.201 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.713623+0000", 2020-06-12T02:39:18.201 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0939e-05 2020-06-12T02:39:18.201 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.201 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.201 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:18.202 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.713633+0000", 2020-06-12T02:39:18.202 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3229999999999999e-06 2020-06-12T02:39:18.202 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.202 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.202 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:18.202 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.713638+0000", 2020-06-12T02:39:18.203 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7401000000000002e-05 2020-06-12T02:39:18.203 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.203 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.203 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:18.203 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.713685+0000", 2020-06-12T02:39:18.204 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011488 2020-06-12T02:39:18.204 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.204 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.204 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:18.204 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.713800+0000", 2020-06-12T02:39:18.204 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000662541 2020-06-12T02:39:18.205 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.205 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.205 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:18.205 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.714463+0000", 2020-06-12T02:39:18.205 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043485090000000004 2020-06-12T02:39:18.206 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.206 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.206 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:18.206 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.718811+0000", 2020-06-12T02:39:18.206 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.18428694000000001 2020-06-12T02:39:18.207 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.207 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.207 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:39:18.207 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.903098+0000", 2020-06-12T02:39:18.207 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013959899999999999 2020-06-12T02:39:18.207 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.208 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.208 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:18.208 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.903238+0000", 2020-06-12T02:39:18.208 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.18980771900000001 2020-06-12T02:39:18.208 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.209 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:18.209 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.210 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.210 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.210 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:4970 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~401408] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-06-12T02:39:18.210 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:43.716006+0000", 2020-06-12T02:39:18.211 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 94.386493541999997, 2020-06-12T02:39:18.211 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.19216903699999999, 2020-06-12T02:39:18.211 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:18.211 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:18.211 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:18.212 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:18.212 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:18.212 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 4970 2020-06-12T02:39:18.212 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.212 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:18.212 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.213 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:18.213 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716006+0000", 2020-06-12T02:39:18.213 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:18.213 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.213 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.214 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:18.214 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716006+0000", 2020-06-12T02:39:18.214 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000115757 2020-06-12T02:39:18.214 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.214 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.215 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:18.215 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716122+0000", 2020-06-12T02:39:18.215 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1989999999999997e-06 2020-06-12T02:39:18.215 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.215 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.215 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:18.216 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716127+0000", 2020-06-12T02:39:18.216 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2121e-05 2020-06-12T02:39:18.216 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.216 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.216 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:18.217 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716139+0000", 2020-06-12T02:39:18.217 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.113e-06 2020-06-12T02:39:18.217 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.217 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.217 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:18.217 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716144+0000", 2020-06-12T02:39:18.218 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014413900000000001 2020-06-12T02:39:18.218 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.218 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.218 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:18.218 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.717586+0000", 2020-06-12T02:39:18.219 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0257000000000002e-05 2020-06-12T02:39:18.219 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.219 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.219 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:18.219 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.717656+0000", 2020-06-12T02:39:18.219 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.18596705899999999 2020-06-12T02:39:18.220 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.220 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.220 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:18.220 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.903623+0000", 2020-06-12T02:39:18.220 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7858999999999999e-05 2020-06-12T02:39:18.221 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.221 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.221 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:18.221 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.903721+0000", 2020-06-12T02:39:18.222 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000683536 2020-06-12T02:39:18.222 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.222 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.222 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:18.222 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.904404+0000", 2020-06-12T02:39:18.223 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.003364438 2020-06-12T02:39:18.223 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.223 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.223 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:18.223 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.907769+0000", 2020-06-12T02:39:18.223 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00033094500000000002 2020-06-12T02:39:18.224 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.224 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.224 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:39:18.224 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.908100+0000", 2020-06-12T02:39:18.224 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5363000000000003e-05 2020-06-12T02:39:18.225 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.225 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.225 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:18.225 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.908175+0000", 2020-06-12T02:39:18.225 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.19216903699999999 2020-06-12T02:39:18.226 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.226 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:18.226 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.226 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.226 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.226 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:4971 3.0s0 3:0dddb690:::smithi10729407-10:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-06-12T02:39:18.227 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:43.716190+0000", 2020-06-12T02:39:18.227 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 94.386309683999997, 2020-06-12T02:39:18.227 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.19259003999999999, 2020-06-12T02:39:18.227 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:18.227 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:18.228 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:18.228 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:18.228 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:18.228 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 4971 2020-06-12T02:39:18.228 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.228 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:18.229 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.229 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:18.229 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716190+0000", 2020-06-12T02:39:18.229 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:18.229 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.230 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.230 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:18.230 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716190+0000", 2020-06-12T02:39:18.230 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3192999999999998e-05 2020-06-12T02:39:18.230 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.230 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.231 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:18.231 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716233+0000", 2020-06-12T02:39:18.231 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6279999999999998e-06 2020-06-12T02:39:18.231 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.231 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.232 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:18.232 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716237+0000", 2020-06-12T02:39:18.232 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6619999999999999e-06 2020-06-12T02:39:18.232 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.232 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.232 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:18.233 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716243+0000", 2020-06-12T02:39:18.233 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.495e-06 2020-06-12T02:39:18.233 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.233 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.233 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:18.234 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716247+0000", 2020-06-12T02:39:18.234 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014483740000000001 2020-06-12T02:39:18.234 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.234 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.234 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:18.235 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.717695+0000", 2020-06-12T02:39:18.235 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000108108 2020-06-12T02:39:18.235 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.235 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.236 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:18.236 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.717803+0000", 2020-06-12T02:39:18.236 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.18873436800000001 2020-06-12T02:39:18.236 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.236 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.237 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:18.237 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.906538+0000", 2020-06-12T02:39:18.237 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9384999999999998e-05 2020-06-12T02:39:18.237 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.237 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.237 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:18.238 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.906637+0000", 2020-06-12T02:39:18.238 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00042339299999999999 2020-06-12T02:39:18.238 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.238 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.238 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:18.239 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.907061+0000", 2020-06-12T02:39:18.239 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00071097900000000004 2020-06-12T02:39:18.239 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.239 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.240 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:18.240 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.907772+0000", 2020-06-12T02:39:18.240 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00089641299999999998 2020-06-12T02:39:18.240 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.240 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.241 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:39:18.241 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.908668+0000", 2020-06-12T02:39:18.241 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112042 2020-06-12T02:39:18.241 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.241 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.242 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:18.242 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.908780+0000", 2020-06-12T02:39:18.242 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.19259003999999999 2020-06-12T02:39:18.242 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.242 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:18.243 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.243 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.243 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.243 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:4972 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-06-12T02:39:18.243 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:43.716274+0000", 2020-06-12T02:39:18.244 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 94.386225914999997, 2020-06-12T02:39:18.244 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.193959876, 2020-06-12T02:39:18.244 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:18.244 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:39:18.244 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:18.245 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:18.245 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:18.245 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 4972 2020-06-12T02:39:18.245 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.245 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:18.245 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.246 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:18.246 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716274+0000", 2020-06-12T02:39:18.246 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:18.246 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.246 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.247 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:18.247 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716274+0000", 2020-06-12T02:39:18.247 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6852999999999999e-05 2020-06-12T02:39:18.247 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.247 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.248 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:18.248 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716301+0000", 2020-06-12T02:39:18.248 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3069999999999998e-06 2020-06-12T02:39:18.248 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.248 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.248 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:18.249 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716304+0000", 2020-06-12T02:39:18.249 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6080000000000001e-06 2020-06-12T02:39:18.249 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.249 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.249 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:18.250 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716310+0000", 2020-06-12T02:39:18.250 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6110000000000001e-06 2020-06-12T02:39:18.250 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.250 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.250 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:18.250 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.716313+0000", 2020-06-12T02:39:18.251 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001512918 2020-06-12T02:39:18.251 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.251 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.251 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:18.251 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.717826+0000", 2020-06-12T02:39:18.252 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3773000000000002e-05 2020-06-12T02:39:18.252 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.252 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.252 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:18.253 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.717880+0000", 2020-06-12T02:39:18.254 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.18955132599999999 2020-06-12T02:39:18.254 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.254 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.254 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:18.254 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.907431+0000", 2020-06-12T02:39:18.255 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101158 2020-06-12T02:39:18.255 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.255 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.255 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:18.255 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.907532+0000", 2020-06-12T02:39:18.255 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00160593 2020-06-12T02:39:18.256 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.256 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.256 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:18.256 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.909138+0000", 2020-06-12T02:39:18.256 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8590000000000006e-05 2020-06-12T02:39:18.257 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.257 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.257 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:18.257 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.909207+0000", 2020-06-12T02:39:18.257 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0010268020000000001 2020-06-12T02:39:18.258 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.258 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.258 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:18.258 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.910234+0000", 2020-06-12T02:39:18.258 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.193959876 2020-06-12T02:39:18.258 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.259 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:18.259 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.259 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.259 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.259 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:5006 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-06-12T02:39:18.260 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:43.975526+0000", 2020-06-12T02:39:18.260 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 94.126973864999997, 2020-06-12T02:39:18.260 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014759296, 2020-06-12T02:39:18.260 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:18.260 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:39:18.260 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:18.261 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:18.261 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:18.261 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 5006 2020-06-12T02:39:18.261 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.261 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:18.262 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.262 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:18.262 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.975526+0000", 2020-06-12T02:39:18.262 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:18.262 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.262 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.263 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:18.263 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.975526+0000", 2020-06-12T02:39:18.263 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5868999999999997e-05 2020-06-12T02:39:18.263 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.263 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.264 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:18.264 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.975572+0000", 2020-06-12T02:39:18.264 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1210000000000003e-06 2020-06-12T02:39:18.264 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.264 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.264 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:18.265 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.975578+0000", 2020-06-12T02:39:18.265 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2204999999999999e-05 2020-06-12T02:39:18.265 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.265 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.265 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:18.266 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.975590+0000", 2020-06-12T02:39:18.266 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1289999999999998e-06 2020-06-12T02:39:18.266 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.266 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.266 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:18.267 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.975594+0000", 2020-06-12T02:39:18.267 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.7705e-05 2020-06-12T02:39:18.267 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.267 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.267 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:18.268 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.975622+0000", 2020-06-12T02:39:18.268 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9941999999999997e-05 2020-06-12T02:39:18.268 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.268 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.268 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:18.269 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.975672+0000", 2020-06-12T02:39:18.269 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0057881060000000003 2020-06-12T02:39:18.269 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.269 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.269 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:18.269 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.981460+0000", 2020-06-12T02:39:18.270 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0473999999999997e-05 2020-06-12T02:39:18.270 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.270 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.270 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:18.270 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.981520+0000", 2020-06-12T02:39:18.271 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.007714294 2020-06-12T02:39:18.271 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.271 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.271 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:18.271 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.989235+0000", 2020-06-12T02:39:18.272 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010107600000000001 2020-06-12T02:39:18.272 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.272 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.272 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:18.272 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.989336+0000", 2020-06-12T02:39:18.272 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00094937500000000002 2020-06-12T02:39:18.273 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.273 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.273 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:18.273 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:43.990285+0000", 2020-06-12T02:39:18.273 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014759296 2020-06-12T02:39:18.274 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.274 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:18.274 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.274 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.274 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.274 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:7841 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~565248] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-06-12T02:39:18.275 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:51.996595+0000", 2020-06-12T02:39:18.275 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 86.105904375999998, 2020-06-12T02:39:18.275 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015060283000000001, 2020-06-12T02:39:18.275 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:18.275 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:18.276 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:18.276 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:18.276 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:18.276 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 7841 2020-06-12T02:39:18.276 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.276 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:18.277 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.277 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:18.277 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.996595+0000", 2020-06-12T02:39:18.277 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:18.277 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.278 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.278 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:18.278 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.996595+0000", 2020-06-12T02:39:18.278 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015120699999999999 2020-06-12T02:39:18.278 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.278 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.279 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:18.280 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.996746+0000", 2020-06-12T02:39:18.280 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4369999999999998e-06 2020-06-12T02:39:18.280 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.280 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.281 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:18.281 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.996752+0000", 2020-06-12T02:39:18.281 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0850000000000003e-06 2020-06-12T02:39:18.281 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.281 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.282 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:18.282 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.996761+0000", 2020-06-12T02:39:18.282 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2370000000000003e-06 2020-06-12T02:39:18.282 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.282 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.283 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:18.283 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.996765+0000", 2020-06-12T02:39:18.283 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5185999999999998e-05 2020-06-12T02:39:18.283 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.284 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.284 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:18.284 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.996800+0000", 2020-06-12T02:39:18.284 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7265000000000001e-05 2020-06-12T02:39:18.284 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.284 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.285 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:18.285 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.996868+0000", 2020-06-12T02:39:18.285 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0024047539999999998 2020-06-12T02:39:18.285 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.285 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.286 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:18.286 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.999272+0000", 2020-06-12T02:39:18.286 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3034999999999994e-05 2020-06-12T02:39:18.286 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.286 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.286 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:18.287 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.999355+0000", 2020-06-12T02:39:18.287 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049334499999999996 2020-06-12T02:39:18.287 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.287 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.287 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:18.288 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:51.999849+0000", 2020-06-12T02:39:18.288 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011708226 2020-06-12T02:39:18.288 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.288 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.288 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:18.288 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.011557+0000", 2020-06-12T02:39:18.289 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1449999999999997e-05 2020-06-12T02:39:18.289 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.289 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.289 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:39:18.289 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.011608+0000", 2020-06-12T02:39:18.290 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7055999999999998e-05 2020-06-12T02:39:18.290 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.290 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.290 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:18.290 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.011656+0000", 2020-06-12T02:39:18.290 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015060283000000001 2020-06-12T02:39:18.291 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.291 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:18.291 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.291 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.291 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.292 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:7845 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e33)", 2020-06-12T02:39:18.292 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:52.002017+0000", 2020-06-12T02:39:18.292 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 86.100482145000001, 2020-06-12T02:39:18.292 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017208558999999998, 2020-06-12T02:39:18.292 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:18.292 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:39:18.293 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:18.293 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:18.293 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:18.293 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 7845 2020-06-12T02:39:18.293 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.293 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:18.294 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.294 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:18.294 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.002017+0000", 2020-06-12T02:39:18.294 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:18.294 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.295 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.295 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:18.299 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.002017+0000", 2020-06-12T02:39:18.299 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2636000000000003e-05 2020-06-12T02:39:18.299 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.299 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.299 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:18.300 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.002060+0000", 2020-06-12T02:39:18.300 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5080000000000002e-06 2020-06-12T02:39:18.300 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.300 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.300 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:18.301 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.002067+0000", 2020-06-12T02:39:18.301 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3125000000000001e-05 2020-06-12T02:39:18.301 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.301 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.301 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:18.301 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.002080+0000", 2020-06-12T02:39:18.302 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5560000000000001e-06 2020-06-12T02:39:18.302 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.302 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.302 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:18.302 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.002084+0000", 2020-06-12T02:39:18.303 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0088536750000000001 2020-06-12T02:39:18.303 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.303 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.303 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:18.303 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.010938+0000", 2020-06-12T02:39:18.304 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3282999999999995e-05 2020-06-12T02:39:18.304 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.304 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.304 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:18.304 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.011021+0000", 2020-06-12T02:39:18.304 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0052873970000000001 2020-06-12T02:39:18.305 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.305 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.305 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:18.305 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.016309+0000", 2020-06-12T02:39:18.305 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010365399999999999 2020-06-12T02:39:18.306 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.306 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.306 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:18.306 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.016412+0000", 2020-06-12T02:39:18.306 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002813725 2020-06-12T02:39:18.307 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.307 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.307 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:18.307 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:52.019226+0000", 2020-06-12T02:39:18.307 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017208558999999998 2020-06-12T02:39:18.307 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.308 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:18.308 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.308 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.308 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.308 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8809 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~606208] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:39:18.308 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.707225+0000", 2020-06-12T02:39:18.309 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 66.395274708000002, 2020-06-12T02:39:18.309 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016771206, 2020-06-12T02:39:18.309 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:18.309 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:18.309 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:18.310 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:18.310 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:18.310 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8809 2020-06-12T02:39:18.310 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.310 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:18.311 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.311 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:18.311 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.707225+0000", 2020-06-12T02:39:18.311 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:18.311 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.311 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.312 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:18.312 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.707225+0000", 2020-06-12T02:39:18.312 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035342099999999998 2020-06-12T02:39:18.312 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.312 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.313 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:18.313 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.707578+0000", 2020-06-12T02:39:18.313 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2309999999999996e-06 2020-06-12T02:39:18.313 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.313 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.313 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:18.314 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.707587+0000", 2020-06-12T02:39:18.314 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4343e-05 2020-06-12T02:39:18.314 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.314 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.314 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:18.315 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.707601+0000", 2020-06-12T02:39:18.315 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8460000000000002e-06 2020-06-12T02:39:18.315 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.315 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.315 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:18.315 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.707608+0000", 2020-06-12T02:39:18.316 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7374999999999998e-05 2020-06-12T02:39:18.316 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.316 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.316 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:18.316 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.707675+0000", 2020-06-12T02:39:18.317 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111599 2020-06-12T02:39:18.317 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.318 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.318 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:18.318 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.707787+0000", 2020-06-12T02:39:18.318 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00079845100000000004 2020-06-12T02:39:18.318 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.319 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.319 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:18.319 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.708585+0000", 2020-06-12T02:39:18.319 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010899008999999999 2020-06-12T02:39:18.319 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.320 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.320 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:18.320 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.719484+0000", 2020-06-12T02:39:18.320 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044306450000000004 2020-06-12T02:39:18.320 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.321 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.321 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:39:18.321 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.723915+0000", 2020-06-12T02:39:18.321 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1285999999999994e-05 2020-06-12T02:39:18.321 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.321 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.322 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:18.322 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.723996+0000", 2020-06-12T02:39:18.322 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016771206 2020-06-12T02:39:18.322 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.322 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:18.323 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.323 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.323 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.323 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.3s1 49/21 ECSubWrite(tid=4109, reqid=client.4512.0:8841, at_version=49'773, trim_to=0'0, roll_forward_to=49'771))", 2020-06-12T02:39:18.323 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.822295+0000", 2020-06-12T02:39:18.323 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 66.280204136999998, 2020-06-12T02:39:18.324 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01733517, 2020-06-12T02:39:18.324 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:18.324 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:18.324 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:18.324 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.325 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:18.325 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822295+0000", 2020-06-12T02:39:18.325 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:18.325 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.325 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.325 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:18.326 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.822295+0000", 2020-06-12T02:39:18.326 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00089525499999999997 2020-06-12T02:39:18.326 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.326 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.326 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:18.327 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.823191+0000", 2020-06-12T02:39:18.327 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3315999999999998e-05 2020-06-12T02:39:18.327 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.327 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.327 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:18.327 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.823214+0000", 2020-06-12T02:39:18.328 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7096e-05 2020-06-12T02:39:18.328 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.328 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.328 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:18.328 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.823231+0000", 2020-06-12T02:39:18.329 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6890000000000004e-06 2020-06-12T02:39:18.329 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.329 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.329 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:18.329 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.823239+0000", 2020-06-12T02:39:18.329 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00061031599999999996 2020-06-12T02:39:18.330 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.330 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.330 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:18.330 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.823849+0000", 2020-06-12T02:39:18.330 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4205999999999999e-05 2020-06-12T02:39:18.330 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.331 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.331 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:18.331 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.823863+0000", 2020-06-12T02:39:18.331 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015707789999999999 2020-06-12T02:39:18.331 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.332 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.332 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:18.332 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.839571+0000", 2020-06-12T02:39:18.332 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9502000000000001e-05 2020-06-12T02:39:18.332 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.332 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.333 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:18.333 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.839631+0000", 2020-06-12T02:39:18.333 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01733517 2020-06-12T02:39:18.333 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.333 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:18.334 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.334 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.334 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.334 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8854 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~409600] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:39:18.334 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.854691+0000", 2020-06-12T02:39:18.334 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 66.247808254999995, 2020-06-12T02:39:18.335 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01967967, 2020-06-12T02:39:18.335 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:18.335 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:18.335 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:18.335 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:18.336 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:18.336 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8854 2020-06-12T02:39:18.336 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.336 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:18.336 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.336 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:18.337 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.854691+0000", 2020-06-12T02:39:18.337 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:18.337 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.337 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.337 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:18.338 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.854691+0000", 2020-06-12T02:39:18.338 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017767799999999999 2020-06-12T02:39:18.338 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.338 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.338 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:18.338 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.854869+0000", 2020-06-12T02:39:18.339 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8049999999999996e-06 2020-06-12T02:39:18.339 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.339 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.339 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:18.339 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.854878+0000", 2020-06-12T02:39:18.339 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5644000000000001e-05 2020-06-12T02:39:18.340 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.340 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.340 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:18.340 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.854893+0000", 2020-06-12T02:39:18.340 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.584e-06 2020-06-12T02:39:18.341 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.341 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.341 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:18.341 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.854900+0000", 2020-06-12T02:39:18.341 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002416769 2020-06-12T02:39:18.342 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.342 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.342 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:18.342 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.857317+0000", 2020-06-12T02:39:18.342 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000100385 2020-06-12T02:39:18.342 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.343 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.343 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:18.343 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.857417+0000", 2020-06-12T02:39:18.343 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0047283969999999996 2020-06-12T02:39:18.343 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.344 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.344 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:18.344 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.862146+0000", 2020-06-12T02:39:18.344 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1930999999999998e-05 2020-06-12T02:39:18.344 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.344 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.345 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:18.345 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.862238+0000", 2020-06-12T02:39:18.345 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00071479400000000004 2020-06-12T02:39:18.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.345 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.345 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:18.346 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.862952+0000", 2020-06-12T02:39:18.346 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011219316999999999 2020-06-12T02:39:18.346 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.346 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.346 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:18.347 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.874172+0000", 2020-06-12T02:39:18.347 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000104137 2020-06-12T02:39:18.347 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.347 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.347 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:39:18.348 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.874276+0000", 2020-06-12T02:39:18.348 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5229000000000006e-05 2020-06-12T02:39:18.348 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.348 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.348 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:18.348 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.874371+0000", 2020-06-12T02:39:18.349 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01967967 2020-06-12T02:39:18.349 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.349 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:18.349 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.350 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.350 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8855 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~516096] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:39:18.350 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.858551+0000", 2020-06-12T02:39:18.350 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 66.243948396999997, 2020-06-12T02:39:18.350 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020039164000000002, 2020-06-12T02:39:18.350 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:18.351 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:18.351 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:18.351 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:18.351 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:18.351 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8855 2020-06-12T02:39:18.352 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.352 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:18.352 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.352 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:18.352 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.858551+0000", 2020-06-12T02:39:18.352 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:18.353 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.353 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.353 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:18.353 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.858551+0000", 2020-06-12T02:39:18.353 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00020537100000000001 2020-06-12T02:39:18.354 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.354 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.354 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:18.354 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.858757+0000", 2020-06-12T02:39:18.354 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4870000000000002e-06 2020-06-12T02:39:18.354 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.355 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.355 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:18.355 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.858765+0000", 2020-06-12T02:39:18.355 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4358e-05 2020-06-12T02:39:18.355 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.356 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.356 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:18.356 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.858779+0000", 2020-06-12T02:39:18.356 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3879999999999997e-06 2020-06-12T02:39:18.356 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.356 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.357 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:18.357 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.858786+0000", 2020-06-12T02:39:18.357 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034089100000000001 2020-06-12T02:39:18.357 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.357 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.358 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:18.358 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.859127+0000", 2020-06-12T02:39:18.358 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0877000000000005e-05 2020-06-12T02:39:18.358 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.358 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.358 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:18.359 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.859208+0000", 2020-06-12T02:39:18.359 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0059661269999999999 2020-06-12T02:39:18.359 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.359 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.359 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:18.360 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.865174+0000", 2020-06-12T02:39:18.360 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000102761 2020-06-12T02:39:18.360 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.360 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.360 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:18.360 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.865276+0000", 2020-06-12T02:39:18.361 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00079971700000000003 2020-06-12T02:39:18.361 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.361 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.361 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:18.361 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.866076+0000", 2020-06-12T02:39:18.362 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0083001380000000003 2020-06-12T02:39:18.362 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.362 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.362 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:18.362 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.874376+0000", 2020-06-12T02:39:18.362 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041190660000000002 2020-06-12T02:39:18.363 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.363 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.363 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:39:18.363 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.878495+0000", 2020-06-12T02:39:18.363 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4982999999999995e-05 2020-06-12T02:39:18.364 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.364 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.364 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:18.364 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.878590+0000", 2020-06-12T02:39:18.364 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020039164000000002 2020-06-12T02:39:18.364 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.365 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:18.365 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.365 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.365 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.365 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8856 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~442368] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:39:18.366 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.861853+0000", 2020-06-12T02:39:18.366 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 66.240646948000006, 2020-06-12T02:39:18.366 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022102294000000001, 2020-06-12T02:39:18.366 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:18.366 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:18.366 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:18.367 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:18.367 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:18.367 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8856 2020-06-12T02:39:18.367 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.367 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:18.368 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.368 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:18.368 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.861853+0000", 2020-06-12T02:39:18.368 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:18.368 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.368 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.369 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:18.369 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.861853+0000", 2020-06-12T02:39:18.369 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017153099999999999 2020-06-12T02:39:18.369 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.369 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.370 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:18.370 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.862024+0000", 2020-06-12T02:39:18.370 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0889999999999999e-06 2020-06-12T02:39:18.370 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.370 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.370 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:18.371 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.862029+0000", 2020-06-12T02:39:18.371 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3033999999999999e-05 2020-06-12T02:39:18.371 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.371 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.371 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:18.371 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.862042+0000", 2020-06-12T02:39:18.372 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1349999999999997e-06 2020-06-12T02:39:18.372 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.372 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.372 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:18.372 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.862048+0000", 2020-06-12T02:39:18.373 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0081330410000000006 2020-06-12T02:39:18.373 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.373 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.373 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:18.373 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.870181+0000", 2020-06-12T02:39:18.373 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012252500000000001 2020-06-12T02:39:18.374 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.374 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.374 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:18.374 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.870304+0000", 2020-06-12T02:39:18.374 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000740999 2020-06-12T02:39:18.375 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.375 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.375 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:18.375 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.871045+0000", 2020-06-12T02:39:18.375 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012634975 2020-06-12T02:39:18.375 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.376 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.376 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:18.376 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.883680+0000", 2020-06-12T02:39:18.376 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017667900000000001 2020-06-12T02:39:18.376 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.377 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.377 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:39:18.377 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.883857+0000", 2020-06-12T02:39:18.377 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8286000000000001e-05 2020-06-12T02:39:18.377 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.377 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.378 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:18.378 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.883955+0000", 2020-06-12T02:39:18.378 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022102294000000001 2020-06-12T02:39:18.378 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.378 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:18.378 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.379 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.379 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.379 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8857 3.0s0 3:0dddb690:::smithi10729407-10:head [append 0~237568] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:39:18.379 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.863733+0000", 2020-06-12T02:39:18.379 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 66.238766964999996, 2020-06-12T02:39:18.380 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020470258000000002, 2020-06-12T02:39:18.380 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:18.380 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:18.380 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:18.380 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:18.380 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:18.381 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8857 2020-06-12T02:39:18.381 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.381 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:18.381 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.381 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:18.382 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.863733+0000", 2020-06-12T02:39:18.382 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:18.382 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.382 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.382 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:18.382 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.863733+0000", 2020-06-12T02:39:18.383 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014776000000000001 2020-06-12T02:39:18.383 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.383 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.383 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:18.383 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.863880+0000", 2020-06-12T02:39:18.384 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5790000000000004e-06 2020-06-12T02:39:18.384 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.384 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.384 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:18.384 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.863889+0000", 2020-06-12T02:39:18.384 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6960999999999999e-05 2020-06-12T02:39:18.385 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.385 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.385 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:18.385 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.863906+0000", 2020-06-12T02:39:18.385 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5509999999999996e-06 2020-06-12T02:39:18.386 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.386 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.386 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:18.386 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.863913+0000", 2020-06-12T02:39:18.386 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010569258 2020-06-12T02:39:18.386 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.387 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.387 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:18.387 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.874482+0000", 2020-06-12T02:39:18.387 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011260399999999999 2020-06-12T02:39:18.387 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.388 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.388 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:18.388 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.874594+0000", 2020-06-12T02:39:18.388 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014913820000000001 2020-06-12T02:39:18.388 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.388 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.389 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:18.389 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.876086+0000", 2020-06-12T02:39:18.389 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0078744260000000003 2020-06-12T02:39:18.389 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.389 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.390 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:18.390 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.883960+0000", 2020-06-12T02:39:18.390 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015276999999999999 2020-06-12T02:39:18.390 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.390 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.390 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:39:18.391 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.884113+0000", 2020-06-12T02:39:18.391 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9967000000000001e-05 2020-06-12T02:39:18.391 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.391 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.391 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:18.392 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.884203+0000", 2020-06-12T02:39:18.392 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020470258000000002 2020-06-12T02:39:18.392 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.392 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:18.392 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.392 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.393 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.393 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8858 3.0s0 3:0dddb690:::smithi10729407-10:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e49)", 2020-06-12T02:39:18.393 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.863968+0000", 2020-06-12T02:39:18.393 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 66.238531672999997, 2020-06-12T02:39:18.393 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022225373999999999, 2020-06-12T02:39:18.394 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:18.394 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:18.394 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:18.394 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:18.394 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:18.394 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8858 2020-06-12T02:39:18.395 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.395 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:18.395 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.395 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:18.395 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.863968+0000", 2020-06-12T02:39:18.396 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:18.396 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.396 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.396 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:18.396 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.863968+0000", 2020-06-12T02:39:18.396 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5352000000000006e-05 2020-06-12T02:39:18.397 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.397 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.397 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:18.397 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864043+0000", 2020-06-12T02:39:18.397 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9110000000000004e-06 2020-06-12T02:39:18.398 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.398 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.398 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:18.398 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864050+0000", 2020-06-12T02:39:18.398 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2505e-05 2020-06-12T02:39:18.398 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.399 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.399 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:18.399 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864063+0000", 2020-06-12T02:39:18.399 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0900000000000004e-06 2020-06-12T02:39:18.399 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.399 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.400 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:18.400 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864068+0000", 2020-06-12T02:39:18.400 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014563938 2020-06-12T02:39:18.400 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.400 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.401 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:18.401 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.878632+0000", 2020-06-12T02:39:18.401 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000106852 2020-06-12T02:39:18.401 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.401 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.401 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:18.402 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.878739+0000", 2020-06-12T02:39:18.402 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002931796 2020-06-12T02:39:18.402 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.402 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.402 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:18.403 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.881670+0000", 2020-06-12T02:39:18.403 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025365280000000001 2020-06-12T02:39:18.403 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.403 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.403 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:18.403 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.884207+0000", 2020-06-12T02:39:18.404 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001895224 2020-06-12T02:39:18.404 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.404 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.404 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:39:18.404 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.886102+0000", 2020-06-12T02:39:18.404 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1178e-05 2020-06-12T02:39:18.405 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.405 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.405 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:18.405 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.886193+0000", 2020-06-12T02:39:18.405 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022225373999999999 2020-06-12T02:39:18.406 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.406 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:18.406 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.406 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.406 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.406 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:8859 3.0s0 3:0dddb690:::smithi10729407-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e49)", 2020-06-12T02:39:18.407 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.864115+0000", 2020-06-12T02:39:18.407 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 66.238384772000003, 2020-06-12T02:39:18.407 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.03325032, 2020-06-12T02:39:18.407 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:18.407 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:39:18.408 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:18.408 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:18.408 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:18.408 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 8859 2020-06-12T02:39:18.408 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.408 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:18.409 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.409 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:18.409 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864115+0000", 2020-06-12T02:39:18.409 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:18.409 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.410 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.410 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:18.410 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864115+0000", 2020-06-12T02:39:18.410 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6795999999999997e-05 2020-06-12T02:39:18.410 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.410 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.411 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:18.411 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864162+0000", 2020-06-12T02:39:18.411 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5599999999999999e-06 2020-06-12T02:39:18.411 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.411 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.411 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:18.412 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864168+0000", 2020-06-12T02:39:18.412 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5658000000000002e-05 2020-06-12T02:39:18.412 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.412 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.412 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:18.413 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864184+0000", 2020-06-12T02:39:18.413 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8380000000000001e-06 2020-06-12T02:39:18.413 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.413 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.413 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:18.413 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.864189+0000", 2020-06-12T02:39:18.414 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019365752 2020-06-12T02:39:18.414 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.414 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.414 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:18.414 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.883554+0000", 2020-06-12T02:39:18.415 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000118819 2020-06-12T02:39:18.415 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.415 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.415 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:18.415 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.883673+0000", 2020-06-12T02:39:18.415 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0089243440000000007 2020-06-12T02:39:18.416 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.416 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.416 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:18.416 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.892598+0000", 2020-06-12T02:39:18.416 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7429999999999994e-05 2020-06-12T02:39:18.417 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.417 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.417 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:18.417 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.892695+0000", 2020-06-12T02:39:18.417 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004670123 2020-06-12T02:39:18.417 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.418 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.418 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:18.418 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.897365+0000", 2020-06-12T02:39:18.418 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.03325032 2020-06-12T02:39:18.418 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.418 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:18.419 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.419 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.419 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:18.419 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:39:18.419 INFO:tasks.rados.rados.0.smithi107.stdout:11299: done (2 left) 2020-06-12T02:39:18.420 INFO:tasks.rados.rados.0.smithi107.stdout:11302: done (1 left) 2020-06-12T02:39:18.420 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:39:18.420 INFO:tasks.rados.rados.0.smithi107.stdout:11303: seq_num 11302 ranges {1458176=491520,1949696=548864,2498560=655360,3153920=401408} 2020-06-12T02:39:18.433 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-objectstore-tool --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --force --op remove --pgid 2.5 2020-06-12T02:39:18.434 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-06-12T02:39:18.435 INFO:tasks.rados.rados.0.smithi107.stdout:11303: writing smithi10729407-15 from 1458176 to 1949696 tid 1 2020-06-12T02:39:18.435 INFO:tasks.rados.rados.0.smithi107.stdout:11303: writing smithi10729407-15 from 1949696 to 2498560 tid 2 2020-06-12T02:39:18.435 INFO:tasks.rados.rados.0.smithi107.stdout:11303: writing smithi10729407-15 from 2498560 to 3153920 tid 3 2020-06-12T02:39:18.435 INFO:tasks.rados.rados.0.smithi107.stdout:11303: writing smithi10729407-15 from 3153920 to 3555328 tid 4 2020-06-12T02:39:18.435 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:39:18.436 INFO:tasks.rados.rados.0.smithi107.stdout:11304: seq_num 11303 ranges {0=507904,507904=434176,942080=466944,1409024=425984,1835008=278528} 2020-06-12T02:39:18.436 INFO:tasks.rados.rados.0.smithi107.stdout:11304: writing smithi10729407-2 from 0 to 507904 tid 1 2020-06-12T02:39:18.436 INFO:tasks.rados.rados.0.smithi107.stdout:11304: writing smithi10729407-2 from 507904 to 942080 tid 2 2020-06-12T02:39:18.436 INFO:tasks.rados.rados.0.smithi107.stdout:11304: writing smithi10729407-2 from 942080 to 1409024 tid 3 2020-06-12T02:39:18.436 INFO:tasks.rados.rados.0.smithi107.stdout:11304: writing smithi10729407-2 from 1409024 to 1835008 tid 4 2020-06-12T02:39:18.437 INFO:tasks.rados.rados.0.smithi107.stdout:11304: writing smithi10729407-2 from 1835008 to 2113536 tid 5 2020-06-12T02:39:18.437 INFO:tasks.rados.rados.0.smithi107.stdout:11305: delete oid 20 current snap is 0 2020-06-12T02:39:18.437 INFO:tasks.rados.rados.0.smithi107.stdout:11301: finishing write tid 2 to smithi10729407-6 2020-06-12T02:39:18.437 INFO:tasks.rados.rados.0.smithi107.stdout:11301: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:18.437 INFO:tasks.rados.rados.0.smithi107.stdout:11301: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:18.437 INFO:tasks.rados.rados.0.smithi107.stdout:11301: finishing write tid 6 to smithi10729407-6 2020-06-12T02:39:18.438 INFO:tasks.rados.rados.0.smithi107.stdout:11301: finishing write tid 7 to smithi10729407-6 2020-06-12T02:39:18.438 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 2047 (ObjNum 11300 snap 0 seq_num 11300) dirty exists 2020-06-12T02:39:18.438 INFO:tasks.rados.rados.0.smithi107.stdout:11301: left oid 6 (ObjNum 11300 snap 0 seq_num 11300) 2020-06-12T02:39:18.438 INFO:tasks.rados.rados.0.smithi107.stdout:11303: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:18.438 INFO:tasks.rados.rados.0.smithi107.stdout:11303: finishing write tid 2 to smithi10729407-15 2020-06-12T02:39:18.439 INFO:tasks.rados.rados.0.smithi107.stdout:11303: finishing write tid 3 to smithi10729407-15 2020-06-12T02:39:18.439 INFO:tasks.rados.rados.0.smithi107.stdout:11303: finishing write tid 4 to smithi10729407-15 2020-06-12T02:39:18.439 INFO:tasks.rados.rados.0.smithi107.stdout:11303: finishing write tid 6 to smithi10729407-15 2020-06-12T02:39:18.439 INFO:tasks.rados.rados.0.smithi107.stdout:11303: finishing write tid 7 to smithi10729407-15 2020-06-12T02:39:18.439 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 2105 (ObjNum 11302 snap 0 seq_num 11302) dirty exists 2020-06-12T02:39:18.439 INFO:tasks.rados.rados.0.smithi107.stdout:11303: left oid 15 (ObjNum 11302 snap 0 seq_num 11302) 2020-06-12T02:39:18.440 INFO:tasks.rados.rados.0.smithi107.stdout:11304: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:18.440 INFO:tasks.rados.rados.0.smithi107.stdout:11304: finishing write tid 2 to smithi10729407-2 2020-06-12T02:39:18.440 INFO:tasks.rados.rados.0.smithi107.stdout:11304: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:18.440 INFO:tasks.rados.rados.0.smithi107.stdout:11301: done (3 left) 2020-06-12T02:39:18.440 INFO:tasks.rados.rados.0.smithi107.stdout:11303: done (2 left) 2020-06-12T02:39:18.441 INFO:tasks.rados.rados.0.smithi107.stdout:11305: done (1 left) 2020-06-12T02:39:18.441 INFO:tasks.rados.rados.0.smithi107.stdout:11306: delete oid 12 current snap is 0 2020-06-12T02:39:18.441 INFO:tasks.rados.rados.0.smithi107.stdout:11304: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:18.441 INFO:tasks.rados.rados.0.smithi107.stdout:11304: finishing write tid 5 to smithi10729407-2 2020-06-12T02:39:18.441 INFO:tasks.rados.rados.0.smithi107.stdout:11304: finishing write tid 7 to smithi10729407-2 2020-06-12T02:39:18.442 INFO:tasks.rados.rados.0.smithi107.stdout:11306: done (1 left) 2020-06-12T02:39:18.442 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:39:18.442 INFO:tasks.rados.rados.0.smithi107.stdout:11307: seq_num 11306 ranges {1843200=557056,2400256=532480,2932736=638976,3571712=434176} 2020-06-12T02:39:18.442 INFO:tasks.rados.rados.0.smithi107.stdout:11307: writing smithi10729407-8 from 1843200 to 2400256 tid 1 2020-06-12T02:39:18.442 INFO:tasks.rados.rados.0.smithi107.stdout:11307: writing smithi10729407-8 from 2400256 to 2932736 tid 2 2020-06-12T02:39:18.442 INFO:tasks.rados.rados.0.smithi107.stdout:11307: writing smithi10729407-8 from 2932736 to 3571712 tid 3 2020-06-12T02:39:18.443 INFO:tasks.rados.rados.0.smithi107.stdout:11307: writing smithi10729407-8 from 3571712 to 4005888 tid 4 2020-06-12T02:39:18.443 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:39:18.443 INFO:tasks.rados.rados.0.smithi107.stdout:11308: seq_num 11307 ranges {3383296=573440,3956736=417792,4374528=442368,4816896=425984,5242880=319488} 2020-06-12T02:39:18.443 INFO:tasks.rados.rados.0.smithi107.stdout:11308: writing smithi10729407-17 from 3383296 to 3956736 tid 1 2020-06-12T02:39:18.443 INFO:tasks.rados.rados.0.smithi107.stdout:11308: writing smithi10729407-17 from 3956736 to 4374528 tid 2 2020-06-12T02:39:18.444 INFO:tasks.rados.rados.0.smithi107.stdout:11308: writing smithi10729407-17 from 4374528 to 4816896 tid 3 2020-06-12T02:39:18.444 INFO:tasks.rados.rados.0.smithi107.stdout:11308: writing smithi10729407-17 from 4816896 to 5242880 tid 4 2020-06-12T02:39:18.444 INFO:tasks.rados.rados.0.smithi107.stdout:11308: writing smithi10729407-17 from 5242880 to 5562368 tid 5 2020-06-12T02:39:18.444 INFO:tasks.rados.rados.0.smithi107.stdout:11309: delete oid 11 current snap is 0 2020-06-12T02:39:18.444 INFO:tasks.rados.rados.0.smithi107.stdout:11304: finishing write tid 8 to smithi10729407-2 2020-06-12T02:39:18.444 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 1069 (ObjNum 11303 snap 0 seq_num 11303) dirty exists 2020-06-12T02:39:18.445 INFO:tasks.rados.rados.0.smithi107.stdout:11304: left oid 2 (ObjNum 11303 snap 0 seq_num 11303) 2020-06-12T02:39:18.445 INFO:tasks.rados.rados.0.smithi107.stdout:11307: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:18.445 INFO:tasks.rados.rados.0.smithi107.stdout:11307: finishing write tid 2 to smithi10729407-8 2020-06-12T02:39:18.445 INFO:tasks.rados.rados.0.smithi107.stdout:11307: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:18.445 INFO:tasks.rados.rados.0.smithi107.stdout:11307: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:18.446 INFO:tasks.rados.rados.0.smithi107.stdout:11307: finishing write tid 6 to smithi10729407-8 2020-06-12T02:39:18.446 INFO:tasks.rados.rados.0.smithi107.stdout:11307: finishing write tid 7 to smithi10729407-8 2020-06-12T02:39:18.446 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 2110 (ObjNum 11306 snap 0 seq_num 11306) dirty exists 2020-06-12T02:39:18.446 INFO:tasks.rados.rados.0.smithi107.stdout:11307: left oid 8 (ObjNum 11306 snap 0 seq_num 11306) 2020-06-12T02:39:18.446 INFO:tasks.rados.rados.0.smithi107.stdout:11308: finishing write tid 1 to smithi10729407-17 2020-06-12T02:39:18.446 INFO:tasks.rados.rados.0.smithi107.stdout:11304: done (3 left) 2020-06-12T02:39:18.447 INFO:tasks.rados.rados.0.smithi107.stdout:11307: done (2 left) 2020-06-12T02:39:18.447 INFO:tasks.rados.rados.0.smithi107.stdout:11309: done (1 left) 2020-06-12T02:39:18.447 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:39:18.447 INFO:tasks.rados.rados.0.smithi107.stdout:11310: seq_num 11309 ranges {0=606208,606208=589824,1196032=458752,1654784=557056} 2020-06-12T02:39:18.447 INFO:tasks.rados.rados.0.smithi107.stdout:11310: writing smithi10729407-13 from 0 to 606208 tid 1 2020-06-12T02:39:18.448 INFO:tasks.rados.rados.0.smithi107.stdout:11310: writing smithi10729407-13 from 606208 to 1196032 tid 2 2020-06-12T02:39:18.448 INFO:tasks.rados.rados.0.smithi107.stdout:11310: writing smithi10729407-13 from 1196032 to 1654784 tid 3 2020-06-12T02:39:18.448 INFO:tasks.rados.rados.0.smithi107.stdout:11310: writing smithi10729407-13 from 1654784 to 2211840 tid 4 2020-06-12T02:39:18.448 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:39:18.448 INFO:tasks.rados.rados.0.smithi107.stdout:11311: seq_num 11310 ranges {0=630784,630784=475136,1105920=614400,1720320=516096} 2020-06-12T02:39:18.448 INFO:tasks.rados.rados.0.smithi107.stdout:11311: writing smithi10729407-4 from 0 to 630784 tid 1 2020-06-12T02:39:18.449 INFO:tasks.rados.rados.0.smithi107.stdout:11311: writing smithi10729407-4 from 630784 to 1105920 tid 2 2020-06-12T02:39:18.449 INFO:tasks.rados.rados.0.smithi107.stdout:11311: writing smithi10729407-4 from 1105920 to 1720320 tid 3 2020-06-12T02:39:18.449 INFO:tasks.rados.rados.0.smithi107.stdout:11311: writing smithi10729407-4 from 1720320 to 2236416 tid 4 2020-06-12T02:39:18.449 INFO:tasks.rados.rados.0.smithi107.stdout:11312: delete oid 9 current snap is 0 2020-06-12T02:39:18.449 INFO:tasks.rados.rados.0.smithi107.stdout:11308: finishing write tid 2 to smithi10729407-17 2020-06-12T02:39:18.450 INFO:tasks.rados.rados.0.smithi107.stdout:11308: finishing write tid 3 to smithi10729407-17 2020-06-12T02:39:18.450 INFO:tasks.rados.rados.0.smithi107.stdout:11308: finishing write tid 4 to smithi10729407-17 2020-06-12T02:39:18.450 INFO:tasks.rados.rados.0.smithi107.stdout:11308: finishing write tid 5 to smithi10729407-17 2020-06-12T02:39:18.450 INFO:tasks.rados.rados.0.smithi107.stdout:11308: finishing write tid 7 to smithi10729407-17 2020-06-12T02:39:18.450 INFO:tasks.rados.rados.0.smithi107.stdout:11308: finishing write tid 8 to smithi10729407-17 2020-06-12T02:39:18.450 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 2114 (ObjNum 11307 snap 0 seq_num 11307) dirty exists 2020-06-12T02:39:18.451 INFO:tasks.rados.rados.0.smithi107.stdout:11308: left oid 17 (ObjNum 11307 snap 0 seq_num 11307) 2020-06-12T02:39:18.451 INFO:tasks.rados.rados.0.smithi107.stdout:11310: finishing write tid 1 to smithi10729407-13 2020-06-12T02:39:18.451 INFO:tasks.rados.rados.0.smithi107.stdout:11310: finishing write tid 2 to smithi10729407-13 2020-06-12T02:39:18.451 INFO:tasks.rados.rados.0.smithi107.stdout:11310: finishing write tid 3 to smithi10729407-13 2020-06-12T02:39:18.451 INFO:tasks.rados.rados.0.smithi107.stdout:11310: finishing write tid 4 to smithi10729407-13 2020-06-12T02:39:18.451 INFO:tasks.rados.rados.0.smithi107.stdout:11310: finishing write tid 6 to smithi10729407-13 2020-06-12T02:39:18.452 INFO:tasks.rados.rados.0.smithi107.stdout:11311: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:18.452 INFO:tasks.rados.rados.0.smithi107.stdout:11310: finishing write tid 7 to smithi10729407-13 2020-06-12T02:39:18.452 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1808 (ObjNum 11309 snap 0 seq_num 11309) dirty exists 2020-06-12T02:39:18.452 INFO:tasks.rados.rados.0.smithi107.stdout:11310: left oid 13 (ObjNum 11309 snap 0 seq_num 11309) 2020-06-12T02:39:18.452 INFO:tasks.rados.rados.0.smithi107.stdout:11308: done (3 left) 2020-06-12T02:39:18.453 INFO:tasks.rados.rados.0.smithi107.stdout:11310: done (2 left) 2020-06-12T02:39:18.453 INFO:tasks.rados.rados.0.smithi107.stdout:11312: done (1 left) 2020-06-12T02:39:18.453 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:39:18.453 INFO:tasks.rados.rados.0.smithi107.stdout:11313: seq_num 11312 ranges {0=663552,663552=638976,1302528=630784,1933312=335872} 2020-06-12T02:39:18.453 INFO:tasks.rados.rados.0.smithi107.stdout:11313: writing smithi10729407-7 from 0 to 663552 tid 1 2020-06-12T02:39:18.453 INFO:tasks.rados.rados.0.smithi107.stdout:11313: writing smithi10729407-7 from 663552 to 1302528 tid 2 2020-06-12T02:39:18.454 INFO:tasks.rados.rados.0.smithi107.stdout:11313: writing smithi10729407-7 from 1302528 to 1933312 tid 3 2020-06-12T02:39:18.454 INFO:tasks.rados.rados.0.smithi107.stdout:11313: writing smithi10729407-7 from 1933312 to 2269184 tid 4 2020-06-12T02:39:18.454 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:39:18.454 INFO:tasks.rados.rados.0.smithi107.stdout:11314: seq_num 11313 ranges {2211840=679936,2891776=573440,3465216=442368,3907584=589824} 2020-06-12T02:39:18.454 INFO:tasks.rados.rados.0.smithi107.stdout:11314: writing smithi10729407-13 from 2211840 to 2891776 tid 1 2020-06-12T02:39:18.455 INFO:tasks.rados.rados.0.smithi107.stdout:11314: writing smithi10729407-13 from 2891776 to 3465216 tid 2 2020-06-12T02:39:18.455 INFO:tasks.rados.rados.0.smithi107.stdout:11314: writing smithi10729407-13 from 3465216 to 3907584 tid 3 2020-06-12T02:39:18.455 INFO:tasks.rados.rados.0.smithi107.stdout:11314: writing smithi10729407-13 from 3907584 to 4497408 tid 4 2020-06-12T02:39:18.455 INFO:tasks.rados.rados.0.smithi107.stdout:11315: delete oid 20 current snap is 0 2020-06-12T02:39:18.455 INFO:tasks.rados.rados.0.smithi107.stdout:11311: finishing write tid 2 to smithi10729407-4 2020-06-12T02:39:18.455 INFO:tasks.rados.rados.0.smithi107.stdout:11311: finishing write tid 3 to smithi10729407-4 2020-06-12T02:39:18.456 INFO:tasks.rados.rados.0.smithi107.stdout:11311: finishing write tid 4 to smithi10729407-4 2020-06-12T02:39:18.456 INFO:tasks.rados.rados.0.smithi107.stdout:11313: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:18.456 INFO:tasks.rados.rados.0.smithi107.stdout:11311: finishing write tid 6 to smithi10729407-4 2020-06-12T02:39:18.456 INFO:tasks.rados.rados.0.smithi107.stdout:11311: finishing write tid 7 to smithi10729407-4 2020-06-12T02:39:18.456 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 2119 (ObjNum 11310 snap 0 seq_num 11310) dirty exists 2020-06-12T02:39:18.457 INFO:tasks.rados.rados.0.smithi107.stdout:11311: left oid 4 (ObjNum 11310 snap 0 seq_num 11310) 2020-06-12T02:39:18.457 INFO:tasks.rados.rados.0.smithi107.stdout:11313: finishing write tid 2 to smithi10729407-7 2020-06-12T02:39:18.457 INFO:tasks.rados.rados.0.smithi107.stdout:11313: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:18.457 INFO:tasks.rados.rados.0.smithi107.stdout:11313: finishing write tid 4 to smithi10729407-7 2020-06-12T02:39:18.457 INFO:tasks.rados.rados.0.smithi107.stdout:11313: finishing write tid 6 to smithi10729407-7 2020-06-12T02:39:18.457 INFO:tasks.rados.rados.0.smithi107.stdout:11313: finishing write tid 7 to smithi10729407-7 2020-06-12T02:39:18.458 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1833 (ObjNum 11312 snap 0 seq_num 11312) dirty exists 2020-06-12T02:39:18.458 INFO:tasks.rados.rados.0.smithi107.stdout:11313: left oid 7 (ObjNum 11312 snap 0 seq_num 11312) 2020-06-12T02:39:18.458 INFO:tasks.rados.rados.0.smithi107.stdout:11314: finishing write tid 1 to smithi10729407-13 2020-06-12T02:39:18.458 INFO:tasks.rados.rados.0.smithi107.stdout:11311: done (3 left) 2020-06-12T02:39:18.458 INFO:tasks.rados.rados.0.smithi107.stdout:11313: done (2 left) 2020-06-12T02:39:18.459 INFO:tasks.rados.rados.0.smithi107.stdout:11315: done (1 left) 2020-06-12T02:39:18.459 INFO:tasks.rados.rados.0.smithi107.stdout:11316: delete oid 9 current snap is 0 2020-06-12T02:39:18.459 INFO:tasks.rados.rados.0.smithi107.stdout:11316: done (1 left) 2020-06-12T02:39:18.459 INFO:tasks.rados.rados.0.smithi107.stdout:11317: delete oid 6 current snap is 0 2020-06-12T02:39:18.459 INFO:tasks.rados.rados.0.smithi107.stdout:11314: finishing write tid 2 to smithi10729407-13 2020-06-12T02:39:18.459 INFO:tasks.rados.rados.0.smithi107.stdout:11314: finishing write tid 3 to smithi10729407-13 2020-06-12T02:39:18.460 INFO:tasks.rados.rados.0.smithi107.stdout:11317: done (1 left) 2020-06-12T02:39:18.460 INFO:tasks.rados.rados.0.smithi107.stdout:11318: delete oid 16 current snap is 0 2020-06-12T02:39:18.460 INFO:tasks.rados.rados.0.smithi107.stdout:11314: finishing write tid 4 to smithi10729407-13 2020-06-12T02:39:18.460 INFO:tasks.rados.rados.0.smithi107.stdout:11314: finishing write tid 6 to smithi10729407-13 2020-06-12T02:39:18.460 INFO:tasks.rados.rados.0.smithi107.stdout:11318: done (1 left) 2020-06-12T02:39:18.461 INFO:tasks.rados.rados.0.smithi107.stdout:11319: delete oid 20 current snap is 0 2020-06-12T02:39:18.461 INFO:tasks.rados.rados.0.smithi107.stdout:11314: finishing write tid 7 to smithi10729407-13 2020-06-12T02:39:18.461 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1813 (ObjNum 11313 snap 0 seq_num 11313) dirty exists 2020-06-12T02:39:18.461 INFO:tasks.rados.rados.0.smithi107.stdout:11314: left oid 13 (ObjNum 11313 snap 0 seq_num 11313) 2020-06-12T02:39:18.461 INFO:tasks.rados.rados.0.smithi107.stdout:11314: done (1 left) 2020-06-12T02:39:18.461 INFO:tasks.rados.rados.0.smithi107.stdout:11319: done (0 left) 2020-06-12T02:39:18.462 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:39:18.462 INFO:tasks.rados.rados.0.smithi107.stdout:11320: seq_num 11319 ranges {0=778240,778240=679936,1458176=786432,2244608=139264} 2020-06-12T02:39:18.462 INFO:tasks.rados.rados.0.smithi107.stdout:11320: writing smithi10729407-3 from 0 to 778240 tid 1 2020-06-12T02:39:18.462 INFO:tasks.rados.rados.0.smithi107.stdout:11320: writing smithi10729407-3 from 778240 to 1458176 tid 2 2020-06-12T02:39:18.462 INFO:tasks.rados.rados.0.smithi107.stdout:11320: writing smithi10729407-3 from 1458176 to 2244608 tid 3 2020-06-12T02:39:18.462 INFO:tasks.rados.rados.0.smithi107.stdout:11320: writing smithi10729407-3 from 2244608 to 2383872 tid 4 2020-06-12T02:39:18.463 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:39:18.463 INFO:tasks.rados.rados.0.smithi107.stdout:11321: seq_num 11320 ranges {4005888=794624,4800512=614400,5414912=589824,6004736=401408} 2020-06-12T02:39:18.463 INFO:tasks.rados.rados.0.smithi107.stdout:11321: writing smithi10729407-8 from 4005888 to 4800512 tid 1 2020-06-12T02:39:18.463 INFO:tasks.rados.rados.0.smithi107.stdout:11321: writing smithi10729407-8 from 4800512 to 5414912 tid 2 2020-06-12T02:39:18.463 INFO:tasks.rados.rados.0.smithi107.stdout:11321: writing smithi10729407-8 from 5414912 to 6004736 tid 3 2020-06-12T02:39:18.464 INFO:tasks.rados.rados.0.smithi107.stdout:11321: writing smithi10729407-8 from 6004736 to 6406144 tid 4 2020-06-12T02:39:18.464 INFO:tasks.rados.rados.0.smithi107.stdout:11322: delete oid 1 current snap is 0 2020-06-12T02:39:18.464 INFO:tasks.rados.rados.0.smithi107.stdout:11320: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:18.464 INFO:tasks.rados.rados.0.smithi107.stdout:11320: finishing write tid 2 to smithi10729407-3 2020-06-12T02:39:18.464 INFO:tasks.rados.rados.0.smithi107.stdout:11320: finishing write tid 3 to smithi10729407-3 2020-06-12T02:39:18.465 INFO:tasks.rados.rados.0.smithi107.stdout:11320: finishing write tid 4 to smithi10729407-3 2020-06-12T02:39:18.465 INFO:tasks.rados.rados.0.smithi107.stdout:11320: finishing write tid 6 to smithi10729407-3 2020-06-12T02:39:18.465 INFO:tasks.rados.rados.0.smithi107.stdout:11320: finishing write tid 7 to smithi10729407-3 2020-06-12T02:39:18.465 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1995 (ObjNum 11319 snap 0 seq_num 11319) dirty exists 2020-06-12T02:39:18.465 INFO:tasks.rados.rados.0.smithi107.stdout:11320: left oid 3 (ObjNum 11319 snap 0 seq_num 11319) 2020-06-12T02:39:18.465 INFO:tasks.rados.rados.0.smithi107.stdout:11320: done (2 left) 2020-06-12T02:39:18.466 INFO:tasks.rados.rados.0.smithi107.stdout:11322: done (1 left) 2020-06-12T02:39:18.466 INFO:tasks.rados.rados.0.smithi107.stdout:11323: delete oid 19 current snap is 0 2020-06-12T02:39:18.466 INFO:tasks.rados.rados.0.smithi107.stdout:11321: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:18.466 INFO:tasks.rados.rados.0.smithi107.stdout:11321: finishing write tid 2 to smithi10729407-8 2020-06-12T02:39:18.466 INFO:tasks.rados.rados.0.smithi107.stdout:11323: done (1 left) 2020-06-12T02:39:18.467 INFO:tasks.rados.rados.0.smithi107.stdout:11324: delete oid 3 current snap is 0 2020-06-12T02:39:18.467 INFO:tasks.rados.rados.0.smithi107.stdout:11324: done (1 left) 2020-06-12T02:39:18.467 INFO:tasks.rados.rados.0.smithi107.stdout:11325: delete oid 9 current snap is 0 2020-06-12T02:39:18.467 INFO:tasks.rados.rados.0.smithi107.stdout:11325: done (1 left) 2020-06-12T02:39:18.467 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:39:18.467 INFO:tasks.rados.rados.0.smithi107.stdout:11326: seq_num 11325 ranges {0=73728} 2020-06-12T02:39:18.468 INFO:tasks.rados.rados.0.smithi107.stdout:11326: writing smithi10729407-11 from 0 to 73728 tid 1 2020-06-12T02:39:18.468 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:39:18.468 INFO:tasks.rados.rados.0.smithi107.stdout:11327: seq_num 11326 ranges {0=90112} 2020-06-12T02:39:18.468 INFO:tasks.rados.rados.0.smithi107.stdout:11327: writing smithi10729407-12 from 0 to 90112 tid 1 2020-06-12T02:39:18.468 INFO:tasks.rados.rados.0.smithi107.stdout:11328: delete oid 19 current snap is 0 2020-06-12T02:39:18.468 INFO:tasks.rados.rados.0.smithi107.stdout:11321: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:18.469 INFO:tasks.rados.rados.0.smithi107.stdout:11321: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:18.469 INFO:tasks.rados.rados.0.smithi107.stdout:11321: finishing write tid 6 to smithi10729407-8 2020-06-12T02:39:18.469 INFO:tasks.rados.rados.0.smithi107.stdout:11321: finishing write tid 7 to smithi10729407-8 2020-06-12T02:39:18.469 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 2115 (ObjNum 11320 snap 0 seq_num 11320) dirty exists 2020-06-12T02:39:18.469 INFO:tasks.rados.rados.0.smithi107.stdout:11321: left oid 8 (ObjNum 11320 snap 0 seq_num 11320) 2020-06-12T02:39:18.470 INFO:tasks.rados.rados.0.smithi107.stdout:11326: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:18.470 INFO:tasks.rados.rados.0.smithi107.stdout:11326: finishing write tid 3 to smithi10729407-11 2020-06-12T02:39:18.470 INFO:tasks.rados.rados.0.smithi107.stdout:11326: finishing write tid 4 to smithi10729407-11 2020-06-12T02:39:18.470 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1790 (ObjNum 11325 snap 0 seq_num 11325) dirty exists 2020-06-12T02:39:18.470 INFO:tasks.rados.rados.0.smithi107.stdout:11326: left oid 11 (ObjNum 11325 snap 0 seq_num 11325) 2020-06-12T02:39:18.470 INFO:tasks.rados.rados.0.smithi107.stdout:11327: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:18.471 INFO:tasks.rados.rados.0.smithi107.stdout:11327: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:18.471 INFO:tasks.rados.rados.0.smithi107.stdout:11321: done (3 left) 2020-06-12T02:39:18.471 INFO:tasks.rados.rados.0.smithi107.stdout:11326: done (2 left) 2020-06-12T02:39:18.471 INFO:tasks.rados.rados.0.smithi107.stdout:11328: done (1 left) 2020-06-12T02:39:18.471 INFO:tasks.rados.rados.0.smithi107.stdout:11329: delete oid 3 current snap is 0 2020-06-12T02:39:18.472 INFO:tasks.rados.rados.0.smithi107.stdout:11327: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:18.472 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 2107 (ObjNum 11326 snap 0 seq_num 11326) dirty exists 2020-06-12T02:39:18.472 INFO:tasks.rados.rados.0.smithi107.stdout:11327: left oid 12 (ObjNum 11326 snap 0 seq_num 11326) 2020-06-12T02:39:18.472 INFO:tasks.rados.rados.0.smithi107.stdout:11327: done (1 left) 2020-06-12T02:39:18.472 INFO:tasks.rados.rados.0.smithi107.stdout:11329: done (0 left) 2020-06-12T02:39:18.472 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:39:18.473 INFO:tasks.rados.rados.0.smithi107.stdout:11330: seq_num 11329 ranges {0=139264} 2020-06-12T02:39:18.473 INFO:tasks.rados.rados.0.smithi107.stdout:11330: writing smithi10729407-9 from 0 to 139264 tid 1 2020-06-12T02:39:18.473 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:39:18.473 INFO:tasks.rados.rados.0.smithi107.stdout:11331: seq_num 11330 ranges {0=163840} 2020-06-12T02:39:18.473 INFO:tasks.rados.rados.0.smithi107.stdout:11331: writing smithi10729407-18 from 0 to 163840 tid 1 2020-06-12T02:39:18.474 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:39:18.474 INFO:tasks.rados.rados.0.smithi107.stdout:11332: seq_num 11331 ranges {0=180224} 2020-06-12T02:39:18.474 INFO:tasks.rados.rados.0.smithi107.stdout:11332: writing smithi10729407-6 from 0 to 180224 tid 1 2020-06-12T02:39:18.474 INFO:tasks.rados.rados.0.smithi107.stdout:11333: delete oid 8 current snap is 0 2020-06-12T02:39:18.474 INFO:tasks.rados.rados.0.smithi107.stdout:11330: finishing write tid 1 to smithi10729407-9 2020-06-12T02:39:18.474 INFO:tasks.rados.rados.0.smithi107.stdout:11333: done (3 left) 2020-06-12T02:39:18.475 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:39:18.475 INFO:tasks.rados.rados.0.smithi107.stdout:11334: seq_num 11333 ranges {2236416=212992} 2020-06-12T02:39:18.475 INFO:tasks.rados.rados.0.smithi107.stdout:11334: writing smithi10729407-4 from 2236416 to 2449408 tid 1 2020-06-12T02:39:18.475 INFO:tasks.rados.rados.0.smithi107.stdout:11335: delete oid 15 current snap is 0 2020-06-12T02:39:18.475 INFO:tasks.rados.rados.0.smithi107.stdout:11331: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:18.475 INFO:tasks.rados.rados.0.smithi107.stdout:11332: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:18.476 INFO:tasks.rados.rados.0.smithi107.stdout:11332: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:18.476 INFO:tasks.rados.rados.0.smithi107.stdout:11330: finishing write tid 3 to smithi10729407-9 2020-06-12T02:39:18.476 INFO:tasks.rados.rados.0.smithi107.stdout:11332: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:18.476 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 2051 (ObjNum 11331 snap 0 seq_num 11331) dirty exists 2020-06-12T02:39:18.476 INFO:tasks.rados.rados.0.smithi107.stdout:11332: left oid 6 (ObjNum 11331 snap 0 seq_num 11331) 2020-06-12T02:39:18.477 INFO:tasks.rados.rados.0.smithi107.stdout:11331: finishing write tid 3 to smithi10729407-18 2020-06-12T02:39:18.477 INFO:tasks.rados.rados.0.smithi107.stdout:11330: finishing write tid 4 to smithi10729407-9 2020-06-12T02:39:18.477 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1876 (ObjNum 11329 snap 0 seq_num 11329) dirty exists 2020-06-12T02:39:18.477 INFO:tasks.rados.rados.0.smithi107.stdout:11330: left oid 9 (ObjNum 11329 snap 0 seq_num 11329) 2020-06-12T02:39:18.477 INFO:tasks.rados.rados.0.smithi107.stdout:11331: finishing write tid 4 to smithi10729407-18 2020-06-12T02:39:18.477 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1906 (ObjNum 11330 snap 0 seq_num 11330) dirty exists 2020-06-12T02:39:18.478 INFO:tasks.rados.rados.0.smithi107.stdout:11331: left oid 18 (ObjNum 11330 snap 0 seq_num 11330) 2020-06-12T02:39:18.478 INFO:tasks.rados.rados.0.smithi107.stdout:11330: done (4 left) 2020-06-12T02:39:18.478 INFO:tasks.rados.rados.0.smithi107.stdout:11331: done (3 left) 2020-06-12T02:39:18.478 INFO:tasks.rados.rados.0.smithi107.stdout:11332: done (2 left) 2020-06-12T02:39:18.478 INFO:tasks.rados.rados.0.smithi107.stdout:11335: done (1 left) 2020-06-12T02:39:18.479 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:39:18.479 INFO:tasks.rados.rados.0.smithi107.stdout:11336: seq_num 11335 ranges {180224=245760} 2020-06-12T02:39:18.479 INFO:tasks.rados.rados.0.smithi107.stdout:11336: writing smithi10729407-6 from 180224 to 425984 tid 1 2020-06-12T02:39:18.479 INFO:tasks.rados.rados.0.smithi107.stdout:11337: delete oid 15 current snap is 0 2020-06-12T02:39:18.479 INFO:tasks.rados.rados.0.smithi107.stdout:11334: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:18.479 INFO:tasks.rados.rados.0.smithi107.stdout:11334: finishing write tid 3 to smithi10729407-4 2020-06-12T02:39:18.480 INFO:tasks.rados.rados.0.smithi107.stdout:11334: finishing write tid 4 to smithi10729407-4 2020-06-12T02:39:18.480 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 2121 (ObjNum 11333 snap 0 seq_num 11333) dirty exists 2020-06-12T02:39:18.480 INFO:tasks.rados.rados.0.smithi107.stdout:11334: left oid 4 (ObjNum 11333 snap 0 seq_num 11333) 2020-06-12T02:39:18.480 INFO:tasks.rados.rados.0.smithi107.stdout:11336: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:18.480 INFO:tasks.rados.rados.0.smithi107.stdout:11334: done (2 left) 2020-06-12T02:39:18.480 INFO:tasks.rados.rados.0.smithi107.stdout:11337: done (1 left) 2020-06-12T02:39:18.481 INFO:tasks.rados.rados.0.smithi107.stdout:11338: delete oid 9 current snap is 0 2020-06-12T02:39:18.481 INFO:tasks.rados.rados.0.smithi107.stdout:11336: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:18.481 INFO:tasks.rados.rados.0.smithi107.stdout:11338: done (1 left) 2020-06-12T02:39:18.481 INFO:tasks.rados.rados.0.smithi107.stdout:11339: delete oid 14 current snap is 0 2020-06-12T02:39:18.481 INFO:tasks.rados.rados.0.smithi107.stdout:11336: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:18.482 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 2053 (ObjNum 11335 snap 0 seq_num 11335) dirty exists 2020-06-12T02:39:18.482 INFO:tasks.rados.rados.0.smithi107.stdout:11336: left oid 6 (ObjNum 11335 snap 0 seq_num 11335) 2020-06-12T02:39:18.482 INFO:tasks.rados.rados.0.smithi107.stdout:11336: done (1 left) 2020-06-12T02:39:18.482 INFO:tasks.rados.rados.0.smithi107.stdout:11339: done (0 left) 2020-06-12T02:39:18.482 INFO:tasks.rados.rados.0.smithi107.stdout:11340: delete oid 8 current snap is 0 2020-06-12T02:39:18.482 INFO:tasks.rados.rados.0.smithi107.stdout:11340: done (0 left) 2020-06-12T02:39:18.483 INFO:tasks.rados.rados.0.smithi107.stdout:11341: delete oid 8 current snap is 0 2020-06-12T02:39:18.483 INFO:tasks.rados.rados.0.smithi107.stdout:11341: done (0 left) 2020-06-12T02:39:18.483 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:39:18.483 INFO:tasks.rados.rados.0.smithi107.stdout:11342: seq_num 11341 ranges {4497408=344064} 2020-06-12T02:39:18.483 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:39:18.418+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:39:18.486 INFO:tasks.rados.rados.0.smithi107.stdout:11342: writing smithi10729407-13 from 4497408 to 4841472 tid 1 2020-06-12T02:39:18.486 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:39:18.486 INFO:tasks.rados.rados.0.smithi107.stdout:11343: seq_num 11342 ranges {163840=360448} 2020-06-12T02:39:18.486 INFO:tasks.rados.rados.0.smithi107.stdout:11343: writing smithi10729407-18 from 163840 to 524288 tid 1 2020-06-12T02:39:18.487 INFO:tasks.rados.rados.0.smithi107.stdout:11344: delete oid 2 current snap is 0 2020-06-12T02:39:18.487 INFO:tasks.rados.rados.0.smithi107.stdout:11342: finishing write tid 1 to smithi10729407-13 2020-06-12T02:39:18.487 INFO:tasks.rados.rados.0.smithi107.stdout:11342: finishing write tid 3 to smithi10729407-13 2020-06-12T02:39:18.487 INFO:tasks.rados.rados.0.smithi107.stdout:11342: finishing write tid 4 to smithi10729407-13 2020-06-12T02:39:18.487 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1815 (ObjNum 11341 snap 0 seq_num 11341) dirty exists 2020-06-12T02:39:18.488 INFO:tasks.rados.rados.0.smithi107.stdout:11342: left oid 13 (ObjNum 11341 snap 0 seq_num 11341) 2020-06-12T02:39:18.488 INFO:tasks.rados.rados.0.smithi107.stdout:11342: done (2 left) 2020-06-12T02:39:18.488 INFO:tasks.rados.rados.0.smithi107.stdout:11344: done (1 left) 2020-06-12T02:39:18.488 INFO:tasks.rados.rados.0.smithi107.stdout:11345: delete oid 8 current snap is 0 2020-06-12T02:39:18.488 INFO:tasks.rados.rados.0.smithi107.stdout:11343: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:18.488 INFO:tasks.rados.rados.0.smithi107.stdout:11343: finishing write tid 3 to smithi10729407-18 2020-06-12T02:39:18.489 INFO:tasks.rados.rados.0.smithi107.stdout:11345: done (1 left) 2020-06-12T02:39:18.489 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:39:18.489 INFO:tasks.rados.rados.0.smithi107.stdout:11346: seq_num 11345 ranges {7528448=409600} 2020-06-12T02:39:18.489 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2020-06-12T02:39:18.505 INFO:tasks.rados.rados.0.smithi107.stdout:11346: writing smithi10729407-10 from 7528448 to 7938048 tid 1 2020-06-12T02:39:18.505 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:39:18.506 INFO:tasks.rados.rados.0.smithi107.stdout:11343: finishing write tid 4 to smithi10729407-18 2020-06-12T02:39:18.506 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1908 (ObjNum 11342 snap 0 seq_num 11342) dirty exists 2020-06-12T02:39:18.506 INFO:tasks.rados.rados.0.smithi107.stdout:11343: left oid 18 (ObjNum 11342 snap 0 seq_num 11342) 2020-06-12T02:39:18.506 INFO:tasks.rados.rados.0.smithi107.stdout:11347: seq_num 11346 ranges {2269184=425984} 2020-06-12T02:39:18.520 INFO:tasks.rados.rados.0.smithi107.stdout:11347: writing smithi10729407-7 from 2269184 to 2695168 tid 1 2020-06-12T02:39:18.520 INFO:tasks.rados.rados.0.smithi107.stdout:11343: done (2 left) 2020-06-12T02:39:18.520 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:39:18.521 INFO:tasks.rados.rados.0.smithi107.stdout:11348: seq_num 11347 ranges {90112=442368} 2020-06-12T02:39:18.523 INFO:tasks.rados.rados.0.smithi107.stdout:11348: writing smithi10729407-12 from 90112 to 532480 tid 1 2020-06-12T02:39:18.523 INFO:tasks.rados.rados.0.smithi107.stdout:11349: delete oid 2 current snap is 0 2020-06-12T02:39:18.524 INFO:tasks.rados.rados.0.smithi107.stdout:11346: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:18.524 INFO:tasks.rados.rados.0.smithi107.stdout:11346: finishing write tid 3 to smithi10729407-10 2020-06-12T02:39:18.524 INFO:tasks.rados.rados.0.smithi107.stdout:11346: finishing write tid 4 to smithi10729407-10 2020-06-12T02:39:18.524 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 861 (ObjNum 11345 snap 0 seq_num 11345) dirty exists 2020-06-12T02:39:18.524 INFO:tasks.rados.rados.0.smithi107.stdout:11346: left oid 10 (ObjNum 11345 snap 0 seq_num 11345) 2020-06-12T02:39:18.525 INFO:tasks.rados.rados.0.smithi107.stdout:11347: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:18.525 INFO:tasks.rados.rados.0.smithi107.stdout:11346: done (3 left) 2020-06-12T02:39:18.525 INFO:tasks.rados.rados.0.smithi107.stdout:11349: done (2 left) 2020-06-12T02:39:18.526 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:39:18.526 INFO:tasks.rados.rados.0.smithi107.stdout:11350: seq_num 11349 ranges {0=483328} 2020-06-12T02:39:18.528 INFO:tasks.rados.rados.0.smithi107.stdout:11350: writing smithi10729407-8 from 0 to 483328 tid 1 2020-06-12T02:39:18.528 INFO:tasks.rados.rados.0.smithi107.stdout:11351: delete oid 14 current snap is 0 2020-06-12T02:39:18.528 INFO:tasks.rados.rados.0.smithi107.stdout:11347: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:18.529 INFO:tasks.rados.rados.0.smithi107.stdout:11347: finishing write tid 4 to smithi10729407-7 2020-06-12T02:39:18.529 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1835 (ObjNum 11346 snap 0 seq_num 11346) dirty exists 2020-06-12T02:39:18.529 INFO:tasks.rados.rados.0.smithi107.stdout:11347: left oid 7 (ObjNum 11346 snap 0 seq_num 11346) 2020-06-12T02:39:18.530 INFO:tasks.rados.rados.0.smithi107.stdout:11348: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:18.530 INFO:tasks.rados.rados.0.smithi107.stdout:11347: done (3 left) 2020-06-12T02:39:18.531 INFO:tasks.rados.rados.0.smithi107.stdout:11351: done (2 left) 2020-06-12T02:39:18.531 INFO:tasks.rados.rados.0.smithi107.stdout:11352: delete oid 13 current snap is 0 2020-06-12T02:39:18.532 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:39:18.530+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:39:18.534 INFO:tasks.rados.rados.0.smithi107.stdout:11352: done (2 left) 2020-06-12T02:39:18.535 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:39:18.535 INFO:tasks.rados.rados.0.smithi107.stdout:11353: seq_num 11352 ranges {0=532480} 2020-06-12T02:39:18.538 INFO:tasks.rados.rados.0.smithi107.stdout:11353: writing smithi10729407-3 from 0 to 532480 tid 1 2020-06-12T02:39:18.539 INFO:tasks.rados.rados.0.smithi107.stdout:11354: delete oid 17 current snap is 0 2020-06-12T02:39:18.539 INFO:tasks.rados.rados.0.smithi107.stdout:11348: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:18.539 INFO:tasks.rados.rados.0.smithi107.stdout:11350: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:18.539 INFO:tasks.rados.rados.0.smithi107.stdout:11350: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:18.542 INFO:tasks.rados.rados.0.smithi107.stdout:11350: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:18.542 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 2123 (ObjNum 11349 snap 0 seq_num 11349) dirty exists 2020-06-12T02:39:18.542 INFO:tasks.rados.rados.0.smithi107.stdout:11350: left oid 8 (ObjNum 11349 snap 0 seq_num 11349) 2020-06-12T02:39:18.543 INFO:tasks.rados.rados.0.smithi107.stdout:11348: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:18.543 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 2110 (ObjNum 11347 snap 0 seq_num 11347) dirty exists 2020-06-12T02:39:18.543 INFO:tasks.rados.rados.0.smithi107.stdout:11348: left oid 12 (ObjNum 11347 snap 0 seq_num 11347) 2020-06-12T02:39:18.543 INFO:tasks.rados.rados.0.smithi107.stdout:11348: done (3 left) 2020-06-12T02:39:18.544 INFO:tasks.rados.rados.0.smithi107.stdout:11350: done (2 left) 2020-06-12T02:39:18.544 INFO:tasks.rados.rados.0.smithi107.stdout:11354: done (1 left) 2020-06-12T02:39:18.544 INFO:tasks.rados.rados.0.smithi107.stdout:11355: delete oid 18 current snap is 0 2020-06-12T02:39:18.546 INFO:tasks.rados.rados.0.smithi107.stdout:11355: done (1 left) 2020-06-12T02:39:18.547 INFO:tasks.rados.rados.0.smithi107.stdout:11356: delete oid 6 current snap is 0 2020-06-12T02:39:18.547 INFO:tasks.rados.rados.0.smithi107.stdout:11353: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:18.549 INFO:tasks.rados.rados.0.smithi107.stdout:11353: finishing write tid 3 to smithi10729407-3 2020-06-12T02:39:18.550 INFO:tasks.rados.rados.0.smithi107.stdout:11356: done (1 left) 2020-06-12T02:39:18.550 INFO:tasks.rados.rados.0.smithi107.stdout:11357: delete oid 4 current snap is 0 2020-06-12T02:39:18.551 INFO:tasks.rados.rados.0.smithi107.stdout:11353: finishing write tid 4 to smithi10729407-3 2020-06-12T02:39:18.551 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 1999 (ObjNum 11352 snap 0 seq_num 11352) dirty exists 2020-06-12T02:39:18.551 INFO:tasks.rados.rados.0.smithi107.stdout:11353: left oid 3 (ObjNum 11352 snap 0 seq_num 11352) 2020-06-12T02:39:18.552 INFO:tasks.rados.rados.0.smithi107.stdout:11353: done (1 left) 2020-06-12T02:39:18.552 INFO:tasks.rados.rados.0.smithi107.stdout:11357: done (0 left) 2020-06-12T02:39:18.553 INFO:tasks.rados.rados.0.smithi107.stdout:11358: delete oid 6 current snap is 0 2020-06-12T02:39:18.557 INFO:tasks.rados.rados.0.smithi107.stdout:11358: done (0 left) 2020-06-12T02:39:18.557 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:39:18.557 INFO:tasks.rados.rados.0.smithi107.stdout:11359: seq_num 11358 ranges {0=630784} 2020-06-12T02:39:18.560 INFO:tasks.rados.rados.0.smithi107.stdout:11359: writing smithi10729407-20 from 0 to 630784 tid 1 2020-06-12T02:39:18.560 INFO:tasks.rados.rados.0.smithi107.stdout:11360: delete oid 9 current snap is 0 2020-06-12T02:39:18.562 INFO:tasks.rados.rados.0.smithi107.stdout:11360: done (1 left) 2020-06-12T02:39:18.562 INFO:tasks.rados.rados.0.smithi107.stdout:11361: delete oid 3 current snap is 0 2020-06-12T02:39:18.566 INFO:tasks.rados.rados.0.smithi107.stdout:11359: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:18.567 INFO:tasks.rados.rados.0.smithi107.stdout:11361: done (1 left) 2020-06-12T02:39:18.567 INFO:tasks.rados.rados.0.smithi107.stdout:11362: delete oid 11 current snap is 0 2020-06-12T02:39:18.572 INFO:tasks.rados.rados.0.smithi107.stdout:11359: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:18.573 INFO:tasks.rados.rados.0.smithi107.stdout:11362: done (1 left) 2020-06-12T02:39:18.573 INFO:tasks.rados.rados.0.smithi107.stdout:11363: delete oid 16 current snap is 0 2020-06-12T02:39:18.575 INFO:tasks.rados.rados.0.smithi107.stdout:11359: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:18.575 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 887 (ObjNum 11358 snap 0 seq_num 11358) dirty exists 2020-06-12T02:39:18.575 INFO:tasks.rados.rados.0.smithi107.stdout:11359: left oid 20 (ObjNum 11358 snap 0 seq_num 11358) 2020-06-12T02:39:18.576 INFO:tasks.rados.rados.0.smithi107.stdout:11359: done (1 left) 2020-06-12T02:39:18.576 INFO:tasks.rados.rados.0.smithi107.stdout:11363: done (0 left) 2020-06-12T02:39:18.576 INFO:tasks.rados.rados.0.smithi107.stdout:11364: delete oid 11 current snap is 0 2020-06-12T02:39:18.578 INFO:tasks.rados.rados.0.smithi107.stdout:11364: done (0 left) 2020-06-12T02:39:18.578 INFO:tasks.rados.rados.0.smithi107.stdout:11365: delete oid 2 current snap is 0 2020-06-12T02:39:18.581 INFO:tasks.rados.rados.0.smithi107.stdout:11365: done (0 left) 2020-06-12T02:39:18.581 INFO:tasks.rados.rados.0.smithi107.stdout:11366: delete oid 9 current snap is 0 2020-06-12T02:39:18.583 INFO:tasks.rados.rados.0.smithi107.stdout:11366: done (0 left) 2020-06-12T02:39:18.583 INFO:tasks.rados.rados.0.smithi107.stdout:11367: delete oid 3 current snap is 0 2020-06-12T02:39:18.586 INFO:tasks.rados.rados.0.smithi107.stdout:11367: done (0 left) 2020-06-12T02:39:18.586 INFO:tasks.rados.rados.0.smithi107.stdout:11368: delete oid 9 current snap is 0 2020-06-12T02:39:18.589 INFO:tasks.rados.rados.0.smithi107.stdout:11368: done (0 left) 2020-06-12T02:39:18.589 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:39:18.590 INFO:tasks.rados.rados.0.smithi107.stdout:11369: seq_num 11368 ranges {0=794624} 2020-06-12T02:39:18.594 INFO:tasks.rados.rados.0.smithi107.stdout:11369: writing smithi10729407-3 from 0 to 794624 tid 1 2020-06-12T02:39:18.594 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:39:18.594 INFO:tasks.rados.rados.0.smithi107.stdout:11370: seq_num 11369 ranges {2695168=417792,3112960=401408} 2020-06-12T02:39:18.611 INFO:tasks.rados.rados.0.smithi107.stdout:11370: writing smithi10729407-7 from 2695168 to 3112960 tid 1 2020-06-12T02:39:18.614 INFO:tasks.rados.rados.0.smithi107.stdout:11370: writing smithi10729407-7 from 3112960 to 3514368 tid 2 2020-06-12T02:39:18.614 INFO:tasks.rados.rados.0.smithi107.stdout:11371: delete oid 13 current snap is 0 2020-06-12T02:39:18.614 INFO:tasks.rados.rados.0.smithi107.stdout:11369: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:18.614 INFO:tasks.rados.rados.0.smithi107.stdout:11369: finishing write tid 3 to smithi10729407-3 2020-06-12T02:39:18.614 INFO:tasks.rados.rados.0.smithi107.stdout:11369: finishing write tid 4 to smithi10729407-3 2020-06-12T02:39:18.615 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 2004 (ObjNum 11368 snap 0 seq_num 11368) dirty exists 2020-06-12T02:39:18.615 INFO:tasks.rados.rados.0.smithi107.stdout:11369: left oid 3 (ObjNum 11368 snap 0 seq_num 11368) 2020-06-12T02:39:18.617 INFO:tasks.rados.rados.0.smithi107.stdout:11369: done (2 left) 2020-06-12T02:39:18.618 INFO:tasks.rados.rados.0.smithi107.stdout:11371: done (1 left) 2020-06-12T02:39:18.618 INFO:tasks.rados.rados.0.smithi107.stdout:11372: delete oid 14 current snap is 0 2020-06-12T02:39:18.618 INFO:tasks.rados.rados.0.smithi107.stdout:11370: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:18.619 INFO:tasks.rados.rados.0.smithi107.stdout:11370: finishing write tid 2 to smithi10729407-7 2020-06-12T02:39:18.619 INFO:tasks.rados.rados.0.smithi107.stdout:11370: finishing write tid 4 to smithi10729407-7 2020-06-12T02:39:18.620 INFO:tasks.rados.rados.0.smithi107.stdout:11372: done (1 left) 2020-06-12T02:39:18.620 INFO:tasks.rados.rados.0.smithi107.stdout:11373: delete oid 15 current snap is 0 2020-06-12T02:39:18.621 INFO:tasks.rados.rados.0.smithi107.stdout:11370: finishing write tid 5 to smithi10729407-7 2020-06-12T02:39:18.621 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1838 (ObjNum 11369 snap 0 seq_num 11369) dirty exists 2020-06-12T02:39:18.621 INFO:tasks.rados.rados.0.smithi107.stdout:11370: left oid 7 (ObjNum 11369 snap 0 seq_num 11369) 2020-06-12T02:39:18.624 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:39:18.621+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:39:18.624 INFO:tasks.rados.rados.0.smithi107.stdout:11370: done (1 left) 2020-06-12T02:39:18.624 INFO:tasks.rados.rados.0.smithi107.stdout:11373: done (0 left) 2020-06-12T02:39:18.625 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:39:18.625 INFO:tasks.rados.rados.0.smithi107.stdout:11374: seq_num 11373 ranges {0=483328,483328=401408} 2020-06-12T02:39:18.627 INFO:tasks.rados.rados.0.smithi107.stdout:11374: writing smithi10729407-1 from 0 to 483328 tid 1 2020-06-12T02:39:18.629 INFO:tasks.rados.rados.0.smithi107.stdout:11374: writing smithi10729407-1 from 483328 to 884736 tid 2 2020-06-12T02:39:18.629 INFO:tasks.rados.rados.0.smithi107.stdout:11375: delete oid 9 current snap is 0 2020-06-12T02:39:18.631 INFO:tasks.rados.rados.0.smithi107.stdout:11375: done (1 left) 2020-06-12T02:39:18.632 INFO:tasks.rados.rados.0.smithi107.stdout:11376: delete oid 4 current snap is 0 2020-06-12T02:39:18.635 INFO:tasks.rados.rados.0.smithi107.stdout:11376: done (1 left) 2020-06-12T02:39:18.635 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:39:18.635 INFO:tasks.rados.rados.0.smithi107.stdout:11377: seq_num 11376 ranges {0=532480,532480=401408} 2020-06-12T02:39:18.638 INFO:tasks.rados.rados.0.smithi107.stdout:11377: writing smithi10729407-6 from 0 to 532480 tid 1 2020-06-12T02:39:18.640 INFO:tasks.rados.rados.0.smithi107.stdout:11377: writing smithi10729407-6 from 532480 to 933888 tid 2 2020-06-12T02:39:18.640 INFO:tasks.rados.rados.0.smithi107.stdout:11378: delete oid 8 current snap is 0 2020-06-12T02:39:18.641 INFO:tasks.rados.rados.0.smithi107.stdout:11374: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:18.641 INFO:tasks.rados.rados.0.smithi107.stdout:11374: finishing write tid 2 to smithi10729407-1 2020-06-12T02:39:18.641 INFO:tasks.rados.rados.0.smithi107.stdout:11374: finishing write tid 4 to smithi10729407-1 2020-06-12T02:39:18.641 INFO:tasks.rados.rados.0.smithi107.stdout:11374: finishing write tid 5 to smithi10729407-1 2020-06-12T02:39:18.642 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 2058 (ObjNum 11373 snap 0 seq_num 11373) dirty exists 2020-06-12T02:39:18.642 INFO:tasks.rados.rados.0.smithi107.stdout:11374: left oid 1 (ObjNum 11373 snap 0 seq_num 11373) 2020-06-12T02:39:18.643 INFO:tasks.rados.rados.0.smithi107.stdout:11374: done (2 left) 2020-06-12T02:39:18.643 INFO:tasks.rados.rados.0.smithi107.stdout:11378: done (1 left) 2020-06-12T02:39:18.643 INFO:tasks.rados.rados.0.smithi107.stdout:11379: delete oid 14 current snap is 0 2020-06-12T02:39:18.644 INFO:tasks.rados.rados.0.smithi107.stdout:11377: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:18.646 INFO:tasks.rados.rados.0.smithi107.stdout:11379: done (1 left) 2020-06-12T02:39:18.646 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:39:18.646 INFO:tasks.rados.rados.0.smithi107.stdout:11380: seq_num 11379 ranges {0=581632,581632=401408} 2020-06-12T02:39:18.648 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:39:18.648 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:39:18.648 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2020-06-12T02:39:18.649 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:39:18.649 INFO:tasks.rados.rados.0.smithi107.stdout:11380: writing smithi10729407-4 from 0 to 581632 tid 1 2020-06-12T02:39:18.651 INFO:tasks.rados.rados.0.smithi107.stdout:11380: writing smithi10729407-4 from 581632 to 983040 tid 2 2020-06-12T02:39:18.651 INFO:tasks.rados.rados.0.smithi107.stdout:11381: delete oid 12 current snap is 0 2020-06-12T02:39:18.652 INFO:tasks.rados.rados.0.smithi107.stdout:11377: finishing write tid 2 to smithi10729407-6 2020-06-12T02:39:18.652 INFO:tasks.rados.rados.0.smithi107.stdout:11377: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:18.654 INFO:tasks.rados.rados.0.smithi107.stdout:11377: finishing write tid 5 to smithi10729407-6 2020-06-12T02:39:18.654 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 2061 (ObjNum 11376 snap 0 seq_num 11376) dirty exists 2020-06-12T02:39:18.654 INFO:tasks.rados.rados.0.smithi107.stdout:11377: left oid 6 (ObjNum 11376 snap 0 seq_num 11376) 2020-06-12T02:39:18.655 INFO:tasks.rados.rados.0.smithi107.stdout:11380: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:18.655 INFO:tasks.rados.rados.0.smithi107.stdout:11377: done (2 left) 2020-06-12T02:39:18.656 INFO:tasks.rados.rados.0.smithi107.stdout:11381: done (1 left) 2020-06-12T02:39:18.656 INFO:tasks.rados.rados.0.smithi107.stdout:11382: delete oid 11 current snap is 0 2020-06-12T02:39:18.658 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-06-12T02:39:18.658 INFO:tasks.rados.rados.0.smithi107.stdout:11382: done (1 left) 2020-06-12T02:39:18.659 INFO:tasks.rados.rados.0.smithi107.stdout:11383: delete oid 8 current snap is 0 2020-06-12T02:39:18.660 INFO:tasks.rados.rados.0.smithi107.stdout:11380: finishing write tid 2 to smithi10729407-4 2020-06-12T02:39:18.662 INFO:tasks.rados.rados.0.smithi107.stdout:11380: finishing write tid 4 to smithi10729407-4 2020-06-12T02:39:18.665 INFO:tasks.rados.rados.0.smithi107.stdout:11380: finishing write tid 5 to smithi10729407-4 2020-06-12T02:39:18.666 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 2127 (ObjNum 11379 snap 0 seq_num 11379) dirty exists 2020-06-12T02:39:18.666 INFO:tasks.rados.rados.0.smithi107.stdout:11380: left oid 4 (ObjNum 11379 snap 0 seq_num 11379) 2020-06-12T02:39:18.666 INFO:tasks.rados.rados.0.smithi107.stdout:11380: done (1 left) 2020-06-12T02:39:18.666 INFO:tasks.rados.rados.0.smithi107.stdout:11383: done (0 left) 2020-06-12T02:39:18.666 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:39:18.667 INFO:tasks.rados.rados.0.smithi107.stdout:11384: seq_num 11383 ranges {0=647168,647168=401408} 2020-06-12T02:39:18.669 INFO:tasks.rados.rados.0.smithi107.stdout:11384: writing smithi10729407-13 from 0 to 647168 tid 1 2020-06-12T02:39:18.671 INFO:tasks.rados.rados.0.smithi107.stdout:11384: writing smithi10729407-13 from 647168 to 1048576 tid 2 2020-06-12T02:39:18.671 INFO:tasks.rados.rados.0.smithi107.stdout:11385: delete oid 7 current snap is 0 2020-06-12T02:39:18.674 INFO:tasks.rados.rados.0.smithi107.stdout:11384: finishing write tid 1 to smithi10729407-13 2020-06-12T02:39:18.679 INFO:tasks.rados.rados.0.smithi107.stdout:11385: done (1 left) 2020-06-12T02:39:18.679 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:39:18.679 INFO:tasks.rados.rados.0.smithi107.stdout:11386: seq_num 11385 ranges {0=679936,679936=401408} 2020-06-12T02:39:18.683 INFO:tasks.rados.rados.0.smithi107.stdout:11386: writing smithi10729407-9 from 0 to 679936 tid 1 2020-06-12T02:39:18.686 INFO:tasks.rados.rados.0.smithi107.stdout:11386: writing smithi10729407-9 from 679936 to 1081344 tid 2 2020-06-12T02:39:18.686 INFO:tasks.rados.rados.0.smithi107.stdout:11387: delete oid 3 current snap is 0 2020-06-12T02:39:18.686 INFO:tasks.rados.rados.0.smithi107.stdout:11384: finishing write tid 2 to smithi10729407-13 2020-06-12T02:39:18.687 INFO:tasks.rados.rados.0.smithi107.stdout:11384: finishing write tid 4 to smithi10729407-13 2020-06-12T02:39:18.687 INFO:tasks.rados.rados.0.smithi107.stdout:11384: finishing write tid 5 to smithi10729407-13 2020-06-12T02:39:18.687 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1820 (ObjNum 11383 snap 0 seq_num 11383) dirty exists 2020-06-12T02:39:18.687 INFO:tasks.rados.rados.0.smithi107.stdout:11384: left oid 13 (ObjNum 11383 snap 0 seq_num 11383) 2020-06-12T02:39:18.689 INFO:tasks.rados.rados.0.smithi107.stdout:11384: done (2 left) 2020-06-12T02:39:18.689 INFO:tasks.rados.rados.0.smithi107.stdout:11387: done (1 left) 2020-06-12T02:39:18.690 INFO:tasks.rados.rados.0.smithi107.stdout:11388: delete oid 18 current snap is 0 2020-06-12T02:39:18.690 INFO:tasks.rados.rados.0.smithi107.stdout:11386: finishing write tid 1 to smithi10729407-9 2020-06-12T02:39:18.695 INFO:tasks.rados.rados.0.smithi107.stdout:11388: done (1 left) 2020-06-12T02:39:18.695 INFO:tasks.rados.rados.0.smithi107.stdout:11389: delete oid 11 current snap is 0 2020-06-12T02:39:18.696 INFO:tasks.rados.rados.0.smithi107.stdout:11386: finishing write tid 2 to smithi10729407-9 2020-06-12T02:39:18.697 INFO:tasks.rados.rados.0.smithi107.stdout:11386: finishing write tid 4 to smithi10729407-9 2020-06-12T02:39:18.699 INFO:tasks.rados.rados.0.smithi107.stdout:11389: done (1 left) 2020-06-12T02:39:18.699 INFO:tasks.rados.rados.0.smithi107.stdout:11390: delete oid 10 current snap is 0 2020-06-12T02:39:18.700 INFO:tasks.rados.rados.0.smithi107.stdout:11386: finishing write tid 5 to smithi10729407-9 2020-06-12T02:39:18.700 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1884 (ObjNum 11385 snap 0 seq_num 11385) dirty exists 2020-06-12T02:39:18.700 INFO:tasks.rados.rados.0.smithi107.stdout:11386: left oid 9 (ObjNum 11385 snap 0 seq_num 11385) 2020-06-12T02:39:18.702 INFO:tasks.rados.rados.0.smithi107.stdout:11386: done (1 left) 2020-06-12T02:39:18.710 INFO:tasks.rados.rados.0.smithi107.stdout:11390: done (0 left) 2020-06-12T02:39:18.711 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:39:18.711 INFO:tasks.rados.rados.0.smithi107.stdout:11391: seq_num 11390 ranges {0=770048,770048=401408} 2020-06-12T02:39:18.714 INFO:tasks.rados.rados.0.smithi107.stdout:11391: writing smithi10729407-16 from 0 to 770048 tid 1 2020-06-12T02:39:18.715 INFO:tasks.rados.rados.0.smithi107.stdout:11391: writing smithi10729407-16 from 770048 to 1171456 tid 2 2020-06-12T02:39:18.715 INFO:tasks.rados.rados.0.smithi107.stdout:11392: delete oid 13 current snap is 0 2020-06-12T02:39:18.715 INFO:tasks.rados.rados.0.smithi107.stdout:11392: done (1 left) 2020-06-12T02:39:18.715 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:39:18.716 INFO:tasks.rados.rados.0.smithi107.stdout:11393: seq_num 11392 ranges {0=802816,802816=401408} 2020-06-12T02:39:18.716 INFO:tasks.rados.rados.0.smithi107.stdout:11393: writing smithi10729407-11 from 0 to 802816 tid 1 2020-06-12T02:39:18.719 INFO:tasks.rados.rados.0.smithi107.stdout:11393: writing smithi10729407-11 from 802816 to 1204224 tid 2 2020-06-12T02:39:18.719 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:39:18.719 INFO:tasks.rados.rados.0.smithi107.stdout:11394: seq_num 11393 ranges {0=417792,417792=622592,1040384=180224} 2020-06-12T02:39:18.721 INFO:tasks.rados.rados.0.smithi107.stdout:11394: writing smithi10729407-19 from 0 to 417792 tid 1 2020-06-12T02:39:18.727 INFO:tasks.rados.rados.0.smithi107.stdout:11394: writing smithi10729407-19 from 417792 to 1040384 tid 2 2020-06-12T02:39:18.727 INFO:tasks.rados.rados.0.smithi107.stdout:11394: writing smithi10729407-19 from 1040384 to 1220608 tid 3 2020-06-12T02:39:18.727 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:39:18.727 INFO:tasks.rados.rados.0.smithi107.stdout:11395: seq_num 11394 ranges {0=434176,434176=507904,942080=294912} 2020-06-12T02:39:18.727 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:39:18.724+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:39:18.731 INFO:tasks.rados.rados.0.smithi107.stdout:11395: writing smithi10729407-14 from 0 to 434176 tid 1 2020-06-12T02:39:18.733 INFO:tasks.rados.rados.0.smithi107.stdout:11395: writing smithi10729407-14 from 434176 to 942080 tid 2 2020-06-12T02:39:18.735 INFO:tasks.rados.rados.0.smithi107.stdout:11395: writing smithi10729407-14 from 942080 to 1236992 tid 3 2020-06-12T02:39:18.735 INFO:tasks.rados.rados.0.smithi107.stdout:11396: delete oid 4 current snap is 0 2020-06-12T02:39:18.735 INFO:tasks.rados.rados.0.smithi107.stdout:11391: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:18.736 INFO:tasks.rados.rados.0.smithi107.stdout:11393: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:18.736 INFO:tasks.rados.rados.0.smithi107.stdout:11394: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:18.736 INFO:tasks.rados.rados.0.smithi107.stdout:11394: finishing write tid 2 to smithi10729407-19 2020-06-12T02:39:18.737 INFO:tasks.rados.rados.0.smithi107.stdout:11394: finishing write tid 3 to smithi10729407-19 2020-06-12T02:39:18.737 INFO:tasks.rados.rados.0.smithi107.stdout:11394: finishing write tid 5 to smithi10729407-19 2020-06-12T02:39:18.737 INFO:tasks.rados.rados.0.smithi107.stdout:11391: finishing write tid 2 to smithi10729407-16 2020-06-12T02:39:18.738 INFO:tasks.rados.rados.0.smithi107.stdout:11396: done (4 left) 2020-06-12T02:39:18.738 INFO:tasks.rados.rados.0.smithi107.stdout:11397: delete oid 20 current snap is 0 2020-06-12T02:39:18.738 INFO:tasks.rados.rados.0.smithi107.stdout:11391: finishing write tid 4 to smithi10729407-16 2020-06-12T02:39:18.739 INFO:tasks.rados.rados.0.smithi107.stdout:11394: finishing write tid 6 to smithi10729407-19 2020-06-12T02:39:18.739 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 2115 (ObjNum 11393 snap 0 seq_num 11393) dirty exists 2020-06-12T02:39:18.739 INFO:tasks.rados.rados.0.smithi107.stdout:11394: left oid 19 (ObjNum 11393 snap 0 seq_num 11393) 2020-06-12T02:39:18.740 INFO:tasks.rados.rados.0.smithi107.stdout:11393: finishing write tid 2 to smithi10729407-11 2020-06-12T02:39:18.740 INFO:tasks.rados.rados.0.smithi107.stdout:11393: finishing write tid 4 to smithi10729407-11 2020-06-12T02:39:18.740 INFO:tasks.rados.rados.0.smithi107.stdout:11395: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:18.742 INFO:tasks.rados.rados.0.smithi107.stdout:11393: finishing write tid 5 to smithi10729407-11 2020-06-12T02:39:18.742 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1797 (ObjNum 11392 snap 0 seq_num 11392) dirty exists 2020-06-12T02:39:18.742 INFO:tasks.rados.rados.0.smithi107.stdout:11393: left oid 11 (ObjNum 11392 snap 0 seq_num 11392) 2020-06-12T02:39:18.743 INFO:tasks.rados.rados.0.smithi107.stdout:11391: finishing write tid 5 to smithi10729407-16 2020-06-12T02:39:18.743 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 2008 (ObjNum 11390 snap 0 seq_num 11390) dirty exists 2020-06-12T02:39:18.743 INFO:tasks.rados.rados.0.smithi107.stdout:11391: left oid 16 (ObjNum 11390 snap 0 seq_num 11390) 2020-06-12T02:39:18.744 INFO:tasks.rados.rados.0.smithi107.stdout:11391: done (4 left) 2020-06-12T02:39:18.744 INFO:tasks.rados.rados.0.smithi107.stdout:11393: done (3 left) 2020-06-12T02:39:18.744 INFO:tasks.rados.rados.0.smithi107.stdout:11394: done (2 left) 2020-06-12T02:39:18.745 INFO:tasks.rados.rados.0.smithi107.stdout:11397: done (1 left) 2020-06-12T02:39:18.745 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:39:18.745 INFO:tasks.rados.rados.0.smithi107.stdout:11398: seq_num 11397 ranges {0=483328,483328=606208,1089536=196608} 2020-06-12T02:39:18.746 INFO:tasks.rados.rados.0.smithi107.stdout:11398: writing smithi10729407-10 from 0 to 483328 tid 1 2020-06-12T02:39:18.750 INFO:tasks.rados.rados.0.smithi107.stdout:11398: writing smithi10729407-10 from 483328 to 1089536 tid 2 2020-06-12T02:39:18.752 INFO:tasks.rados.rados.0.smithi107.stdout:11398: writing smithi10729407-10 from 1089536 to 1286144 tid 3 2020-06-12T02:39:18.752 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:39:18.753 INFO:tasks.rados.rados.0.smithi107.stdout:11399: seq_num 11398 ranges {0=499712,499712=491520,991232=311296} 2020-06-12T02:39:18.754 INFO:tasks.rados.rados.0.smithi107.stdout:11399: writing smithi10729407-7 from 0 to 499712 tid 1 2020-06-12T02:39:18.757 INFO:tasks.rados.rados.0.smithi107.stdout:11399: writing smithi10729407-7 from 499712 to 991232 tid 2 2020-06-12T02:39:18.760 INFO:tasks.rados.rados.0.smithi107.stdout:11399: writing smithi10729407-7 from 991232 to 1302528 tid 3 2020-06-12T02:39:18.760 INFO:tasks.rados.rados.0.smithi107.stdout:11400: delete oid 18 current snap is 0 2020-06-12T02:39:18.760 INFO:tasks.rados.rados.0.smithi107.stdout:11395: finishing write tid 2 to smithi10729407-14 2020-06-12T02:39:18.761 INFO:tasks.rados.rados.0.smithi107.stdout:11395: finishing write tid 3 to smithi10729407-14 2020-06-12T02:39:18.761 INFO:tasks.rados.rados.0.smithi107.stdout:11395: finishing write tid 5 to smithi10729407-14 2020-06-12T02:39:18.761 INFO:tasks.rados.rados.0.smithi107.stdout:11398: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:18.761 INFO:tasks.rados.rados.0.smithi107.stdout:11395: finishing write tid 6 to smithi10729407-14 2020-06-12T02:39:18.762 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1776 (ObjNum 11394 snap 0 seq_num 11394) dirty exists 2020-06-12T02:39:18.770 INFO:tasks.rados.rados.0.smithi107.stdout:11395: left oid 14 (ObjNum 11394 snap 0 seq_num 11394) 2020-06-12T02:39:18.770 INFO:tasks.rados.rados.0.smithi107.stdout:11398: finishing write tid 2 to smithi10729407-10 2020-06-12T02:39:18.772 INFO:tasks.rados.rados.0.smithi107.stdout:11399: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:18.772 INFO:tasks.rados.rados.0.smithi107.stdout:11395: done (3 left) 2020-06-12T02:39:18.773 INFO:tasks.rados.rados.0.smithi107.stdout:11400: done (2 left) 2020-06-12T02:39:18.773 INFO:tasks.rados.rados.0.smithi107.stdout:11401: delete oid 12 current snap is 0 2020-06-12T02:39:18.773 INFO:tasks.rados.rados.0.smithi107.stdout:11398: finishing write tid 3 to smithi10729407-10 2020-06-12T02:39:18.773 INFO:tasks.rados.rados.0.smithi107.stdout:11398: finishing write tid 5 to smithi10729407-10 2020-06-12T02:39:18.773 INFO:tasks.rados.rados.0.smithi107.stdout:11398: finishing write tid 6 to smithi10729407-10 2020-06-12T02:39:18.774 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 866 (ObjNum 11397 snap 0 seq_num 11397) dirty exists 2020-06-12T02:39:18.774 INFO:tasks.rados.rados.0.smithi107.stdout:11398: left oid 10 (ObjNum 11397 snap 0 seq_num 11397) 2020-06-12T02:39:18.774 INFO:tasks.rados.rados.0.smithi107.stdout:11398: done (2 left) 2020-06-12T02:39:18.774 INFO:tasks.rados.rados.0.smithi107.stdout:11401: done (1 left) 2020-06-12T02:39:18.774 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:39:18.775 INFO:tasks.rados.rados.0.smithi107.stdout:11402: seq_num 11401 ranges {0=548864,548864=540672,1089536=262144} 2020-06-12T02:39:18.775 INFO:tasks.rados.rados.0.smithi107.stdout:11402: writing smithi10729407-2 from 0 to 548864 tid 1 2020-06-12T02:39:18.776 INFO:tasks.rados.rados.0.smithi107.stdout:11402: writing smithi10729407-2 from 548864 to 1089536 tid 2 2020-06-12T02:39:18.776 INFO:tasks.rados.rados.0.smithi107.stdout:11402: writing smithi10729407-2 from 1089536 to 1351680 tid 3 2020-06-12T02:39:18.776 INFO:tasks.rados.rados.0.smithi107.stdout:11403: delete oid 13 current snap is 0 2020-06-12T02:39:18.777 INFO:tasks.rados.rados.0.smithi107.stdout:11399: finishing write tid 2 to smithi10729407-7 2020-06-12T02:39:18.777 INFO:tasks.rados.rados.0.smithi107.stdout:11399: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:18.778 INFO:tasks.rados.rados.0.smithi107.stdout:11399: finishing write tid 5 to smithi10729407-7 2020-06-12T02:39:18.778 INFO:tasks.rados.rados.0.smithi107.stdout:11399: finishing write tid 6 to smithi10729407-7 2020-06-12T02:39:18.778 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1843 (ObjNum 11398 snap 0 seq_num 11398) dirty exists 2020-06-12T02:39:18.778 INFO:tasks.rados.rados.0.smithi107.stdout:11399: left oid 7 (ObjNum 11398 snap 0 seq_num 11398) 2020-06-12T02:39:18.779 INFO:tasks.rados.rados.0.smithi107.stdout:11402: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:18.779 INFO:tasks.rados.rados.0.smithi107.stdout:11399: done (2 left) 2020-06-12T02:39:18.780 INFO:tasks.rados.rados.0.smithi107.stdout:11403: done (1 left) 2020-06-12T02:39:18.780 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:39:18.780 INFO:tasks.rados.rados.0.smithi107.stdout:11404: seq_num 11403 ranges {1302528=581632,1884160=712704,2596864=90112} 2020-06-12T02:39:18.789 INFO:tasks.rados.rados.0.smithi107.stdout:11404: writing smithi10729407-7 from 1302528 to 1884160 tid 1 2020-06-12T02:39:18.793 INFO:tasks.rados.rados.0.smithi107.stdout:11404: writing smithi10729407-7 from 1884160 to 2596864 tid 2 2020-06-12T02:39:18.794 INFO:tasks.rados.rados.0.smithi107.stdout:11404: writing smithi10729407-7 from 2596864 to 2686976 tid 3 2020-06-12T02:39:18.794 INFO:tasks.rados.rados.0.smithi107.stdout:11405: delete oid 19 current snap is 0 2020-06-12T02:39:18.795 INFO:tasks.rados.rados.0.smithi107.stdout:11402: finishing write tid 2 to smithi10729407-2 2020-06-12T02:39:18.795 INFO:tasks.rados.rados.0.smithi107.stdout:11402: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:18.795 INFO:tasks.rados.rados.0.smithi107.stdout:11402: finishing write tid 5 to smithi10729407-2 2020-06-12T02:39:18.796 INFO:tasks.rados.rados.0.smithi107.stdout:11402: finishing write tid 6 to smithi10729407-2 2020-06-12T02:39:18.796 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 1076 (ObjNum 11401 snap 0 seq_num 11401) dirty exists 2020-06-12T02:39:18.796 INFO:tasks.rados.rados.0.smithi107.stdout:11402: left oid 2 (ObjNum 11401 snap 0 seq_num 11401) 2020-06-12T02:39:18.797 INFO:tasks.rados.rados.0.smithi107.stdout:11402: done (2 left) 2020-06-12T02:39:18.797 INFO:tasks.rados.rados.0.smithi107.stdout:11405: done (1 left) 2020-06-12T02:39:18.797 INFO:tasks.rados.rados.0.smithi107.stdout:11406: delete oid 10 current snap is 0 2020-06-12T02:39:18.799 INFO:tasks.rados.rados.0.smithi107.stdout:11406: done (1 left) 2020-06-12T02:39:18.799 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:39:18.800 INFO:tasks.rados.rados.0.smithi107.stdout:11407: seq_num 11406 ranges {0=630784,630784=417792,1048576=385024} 2020-06-12T02:39:18.803 INFO:tasks.rados.rados.0.smithi107.stdout:11407: writing smithi10729407-12 from 0 to 630784 tid 1 2020-06-12T02:39:18.805 INFO:tasks.rados.rados.0.smithi107.stdout:11407: writing smithi10729407-12 from 630784 to 1048576 tid 2 2020-06-12T02:39:18.808 INFO:tasks.rados.rados.0.smithi107.stdout:11407: writing smithi10729407-12 from 1048576 to 1433600 tid 3 2020-06-12T02:39:18.808 INFO:tasks.rados.rados.0.smithi107.stdout:11408: delete oid 4 current snap is 0 2020-06-12T02:39:18.808 INFO:tasks.rados.rados.0.smithi107.stdout:11404: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:18.808 INFO:tasks.rados.rados.0.smithi107.stdout:11404: finishing write tid 2 to smithi10729407-7 2020-06-12T02:39:18.809 INFO:tasks.rados.rados.0.smithi107.stdout:11404: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:18.809 INFO:tasks.rados.rados.0.smithi107.stdout:11404: finishing write tid 5 to smithi10729407-7 2020-06-12T02:39:18.809 INFO:tasks.rados.rados.0.smithi107.stdout:11404: finishing write tid 6 to smithi10729407-7 2020-06-12T02:39:18.810 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1847 (ObjNum 11403 snap 0 seq_num 11403) dirty exists 2020-06-12T02:39:18.810 INFO:tasks.rados.rados.0.smithi107.stdout:11404: left oid 7 (ObjNum 11403 snap 0 seq_num 11403) 2020-06-12T02:39:18.810 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:39:18.811 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:39:18.811 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2020-06-12T02:39:18.811 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2020-06-12T02:39:18.812 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:39:18.812 INFO:tasks.rados.rados.0.smithi107.stdout:11407: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:18.813 INFO:tasks.rados.rados.0.smithi107.stdout:11404: done (2 left) 2020-06-12T02:39:18.813 INFO:tasks.rados.rados.0.smithi107.stdout:11408: done (1 left) 2020-06-12T02:39:18.813 INFO:tasks.rados.rados.0.smithi107.stdout:11409: delete oid 7 current snap is 0 2020-06-12T02:39:18.815 INFO:tasks.rados.rados.0.smithi107.stdout:11407: finishing write tid 2 to smithi10729407-12 2020-06-12T02:39:18.816 INFO:tasks.rados.rados.0.smithi107.stdout:11409: done (1 left) 2020-06-12T02:39:18.816 INFO:tasks.rados.rados.0.smithi107.stdout:11410: delete oid 9 current snap is 0 2020-06-12T02:39:18.816 INFO:tasks.rados.rados.0.smithi107.stdout:11407: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:18.817 INFO:tasks.rados.rados.0.smithi107.stdout:11407: finishing write tid 5 to smithi10729407-12 2020-06-12T02:39:18.819 INFO:tasks.rados.rados.0.smithi107.stdout:11410: done (1 left) 2020-06-12T02:39:18.819 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:39:18.819 INFO:tasks.rados.rados.0.smithi107.stdout:11411: seq_num 11410 ranges {0=704512,704512=753664,1458176=49152} 2020-06-12T02:39:18.820 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-06-12T02:39:18.825 INFO:tasks.rados.rados.0.smithi107.stdout:11411: writing smithi10729407-15 from 0 to 704512 tid 1 2020-06-12T02:39:18.826 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:39:18.823+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:39:18.827 INFO:tasks.rados.rados.0.smithi107.stdout:11411: writing smithi10729407-15 from 704512 to 1458176 tid 2 2020-06-12T02:39:18.828 INFO:tasks.rados.rados.0.smithi107.stdout:11411: writing smithi10729407-15 from 1458176 to 1507328 tid 3 2020-06-12T02:39:18.828 INFO:tasks.rados.rados.0.smithi107.stdout:11412: delete oid 13 current snap is 0 2020-06-12T02:39:18.828 INFO:tasks.rados.rados.0.smithi107.stdout:11407: finishing write tid 6 to smithi10729407-12 2020-06-12T02:39:18.829 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 2121 (ObjNum 11406 snap 0 seq_num 11406) dirty exists 2020-06-12T02:39:18.829 INFO:tasks.rados.rados.0.smithi107.stdout:11407: left oid 12 (ObjNum 11406 snap 0 seq_num 11406) 2020-06-12T02:39:18.830 INFO:tasks.rados.rados.0.smithi107.stdout:11407: done (2 left) 2020-06-12T02:39:18.830 INFO:tasks.rados.rados.0.smithi107.stdout:11412: done (1 left) 2020-06-12T02:39:18.831 INFO:tasks.rados.rados.0.smithi107.stdout:11413: delete oid 12 current snap is 0 2020-06-12T02:39:18.833 INFO:tasks.rados.rados.0.smithi107.stdout:11413: done (1 left) 2020-06-12T02:39:18.834 INFO:tasks.rados.rados.0.smithi107.stdout:11414: delete oid 19 current snap is 0 2020-06-12T02:39:18.836 INFO:tasks.rados.rados.0.smithi107.stdout:11411: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:18.836 INFO:tasks.rados.rados.0.smithi107.stdout:11411: finishing write tid 2 to smithi10729407-15 2020-06-12T02:39:18.837 INFO:tasks.rados.rados.0.smithi107.stdout:11411: finishing write tid 3 to smithi10729407-15 2020-06-12T02:39:18.837 INFO:tasks.rados.rados.0.smithi107.stdout:11414: done (1 left) 2020-06-12T02:39:18.837 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:39:18.837 INFO:tasks.rados.rados.0.smithi107.stdout:11415: seq_num 11414 ranges {0=770048,770048=737280,1507328=65536} 2020-06-12T02:39:18.842 INFO:tasks.rados.rados.0.smithi107.stdout:11415: writing smithi10729407-8 from 0 to 770048 tid 1 2020-06-12T02:39:18.847 INFO:tasks.rados.rados.0.smithi107.stdout:11415: writing smithi10729407-8 from 770048 to 1507328 tid 2 2020-06-12T02:39:18.847 INFO:tasks.rados.rados.0.smithi107.stdout:11415: writing smithi10729407-8 from 1507328 to 1572864 tid 3 2020-06-12T02:39:18.848 INFO:tasks.rados.rados.0.smithi107.stdout:11416: delete oid 18 current snap is 0 2020-06-12T02:39:18.848 INFO:tasks.rados.rados.0.smithi107.stdout:11411: finishing write tid 5 to smithi10729407-15 2020-06-12T02:39:18.848 INFO:tasks.rados.rados.0.smithi107.stdout:11411: finishing write tid 6 to smithi10729407-15 2020-06-12T02:39:18.849 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 2130 (ObjNum 11410 snap 0 seq_num 11410) dirty exists 2020-06-12T02:39:18.849 INFO:tasks.rados.rados.0.smithi107.stdout:11411: left oid 15 (ObjNum 11410 snap 0 seq_num 11410) 2020-06-12T02:39:18.850 INFO:tasks.rados.rados.0.smithi107.stdout:11411: done (2 left) 2020-06-12T02:39:18.850 INFO:tasks.rados.rados.0.smithi107.stdout:11416: done (1 left) 2020-06-12T02:39:18.851 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:39:18.851 INFO:tasks.rados.rados.0.smithi107.stdout:11417: seq_num 11416 ranges {1507328=802816,2310144=507904,2818048=294912} 2020-06-12T02:39:18.862 INFO:tasks.rados.rados.0.smithi107.stdout:11417: writing smithi10729407-15 from 1507328 to 2310144 tid 1 2020-06-12T02:39:18.865 INFO:tasks.rados.rados.0.smithi107.stdout:11417: writing smithi10729407-15 from 2310144 to 2818048 tid 2 2020-06-12T02:39:18.867 INFO:tasks.rados.rados.0.smithi107.stdout:11417: writing smithi10729407-15 from 2818048 to 3112960 tid 3 2020-06-12T02:39:18.868 INFO:tasks.rados.rados.0.smithi107.stdout:11418: delete oid 6 current snap is 0 2020-06-12T02:39:18.868 INFO:tasks.rados.rados.0.smithi107.stdout:11415: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:18.868 INFO:tasks.rados.rados.0.smithi107.stdout:11415: finishing write tid 2 to smithi10729407-8 2020-06-12T02:39:18.868 INFO:tasks.rados.rados.0.smithi107.stdout:11415: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:18.869 INFO:tasks.rados.rados.0.smithi107.stdout:11415: finishing write tid 5 to smithi10729407-8 2020-06-12T02:39:18.869 INFO:tasks.rados.rados.0.smithi107.stdout:11415: finishing write tid 6 to smithi10729407-8 2020-06-12T02:39:18.869 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 2134 (ObjNum 11414 snap 0 seq_num 11414) dirty exists 2020-06-12T02:39:18.869 INFO:tasks.rados.rados.0.smithi107.stdout:11415: left oid 8 (ObjNum 11414 snap 0 seq_num 11414) 2020-06-12T02:39:18.870 INFO:tasks.rados.rados.0.smithi107.stdout:11415: done (2 left) 2020-06-12T02:39:18.870 INFO:tasks.rados.rados.0.smithi107.stdout:11418: done (1 left) 2020-06-12T02:39:18.870 INFO:tasks.rados.rados.0.smithi107.stdout:11419: delete oid 18 current snap is 0 2020-06-12T02:39:18.874 INFO:tasks.rados.rados.0.smithi107.stdout:11419: done (1 left) 2020-06-12T02:39:18.874 INFO:tasks.rados.rados.0.smithi107.stdout:11420: delete oid 6 current snap is 0 2020-06-12T02:39:18.875 INFO:tasks.rados.rados.0.smithi107.stdout:11417: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:18.875 INFO:tasks.rados.rados.0.smithi107.stdout:11417: finishing write tid 2 to smithi10729407-15 2020-06-12T02:39:18.877 INFO:tasks.rados.rados.0.smithi107.stdout:11417: finishing write tid 3 to smithi10729407-15 2020-06-12T02:39:18.877 INFO:tasks.rados.rados.0.smithi107.stdout:11420: done (1 left) 2020-06-12T02:39:18.878 INFO:tasks.rados.rados.0.smithi107.stdout:11421: delete oid 6 current snap is 0 2020-06-12T02:39:18.878 INFO:tasks.rados.rados.0.smithi107.stdout:11417: finishing write tid 5 to smithi10729407-15 2020-06-12T02:39:18.880 INFO:tasks.rados.rados.0.smithi107.stdout:11421: done (1 left) 2020-06-12T02:39:18.880 INFO:tasks.rados.rados.0.smithi107.stdout:11422: delete oid 9 current snap is 0 2020-06-12T02:39:18.880 INFO:tasks.rados.rados.0.smithi107.stdout:11417: finishing write tid 6 to smithi10729407-15 2020-06-12T02:39:18.880 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 2138 (ObjNum 11416 snap 0 seq_num 11416) dirty exists 2020-06-12T02:39:18.881 INFO:tasks.rados.rados.0.smithi107.stdout:11417: left oid 15 (ObjNum 11416 snap 0 seq_num 11416) 2020-06-12T02:39:18.882 INFO:tasks.rados.rados.0.smithi107.stdout:11417: done (1 left) 2020-06-12T02:39:18.882 INFO:tasks.rados.rados.0.smithi107.stdout:11422: done (0 left) 2020-06-12T02:39:18.882 INFO:tasks.rados.rados.0.smithi107.stdout:11423: delete oid 14 current snap is 0 2020-06-12T02:39:18.884 INFO:tasks.rados.rados.0.smithi107.stdout:11423: done (0 left) 2020-06-12T02:39:18.885 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:39:18.885 INFO:tasks.rados.rados.0.smithi107.stdout:11424: seq_num 11423 ranges {0=516096,516096=548864,1064960=655360} 2020-06-12T02:39:18.888 INFO:tasks.rados.rados.0.smithi107.stdout:11424: writing smithi10729407-19 from 0 to 516096 tid 1 2020-06-12T02:39:18.891 INFO:tasks.rados.rados.0.smithi107.stdout:11424: writing smithi10729407-19 from 516096 to 1064960 tid 2 2020-06-12T02:39:18.895 INFO:tasks.rados.rados.0.smithi107.stdout:11424: writing smithi10729407-19 from 1064960 to 1720320 tid 3 2020-06-12T02:39:18.895 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:39:18.895 INFO:tasks.rados.rados.0.smithi107.stdout:11425: seq_num 11424 ranges {3112960=532480,3645440=434176,4079616=532480,4612096=237568} 2020-06-12T02:39:18.915 INFO:tasks.rados.rados.0.smithi107.stdout:11425: writing smithi10729407-15 from 3112960 to 3645440 tid 1 2020-06-12T02:39:18.917 INFO:tasks.rados.rados.0.smithi107.stdout:11425: writing smithi10729407-15 from 3645440 to 4079616 tid 2 2020-06-12T02:39:18.921 INFO:tasks.rados.rados.0.smithi107.stdout:11425: writing smithi10729407-15 from 4079616 to 4612096 tid 3 2020-06-12T02:39:18.930 INFO:tasks.rados.rados.0.smithi107.stdout:11425: writing smithi10729407-15 from 4612096 to 4849664 tid 4 2020-06-12T02:39:18.930 INFO:tasks.rados.rados.0.smithi107.stdout:11426: delete oid 7 current snap is 0 2020-06-12T02:39:18.930 INFO:tasks.rados.rados.0.smithi107.stdout:11424: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:18.931 INFO:tasks.rados.rados.0.smithi107.stdout:11424: finishing write tid 2 to smithi10729407-19 2020-06-12T02:39:18.931 INFO:tasks.rados.rados.0.smithi107.stdout:11424: finishing write tid 3 to smithi10729407-19 2020-06-12T02:39:18.931 INFO:tasks.rados.rados.0.smithi107.stdout:11424: finishing write tid 5 to smithi10729407-19 2020-06-12T02:39:18.931 INFO:tasks.rados.rados.0.smithi107.stdout:11424: finishing write tid 6 to smithi10729407-19 2020-06-12T02:39:18.931 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 2127 (ObjNum 11423 snap 0 seq_num 11423) dirty exists 2020-06-12T02:39:18.932 INFO:tasks.rados.rados.0.smithi107.stdout:11424: left oid 19 (ObjNum 11423 snap 0 seq_num 11423) 2020-06-12T02:39:18.932 INFO:tasks.rados.rados.0.smithi107.stdout:11425: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:18.932 INFO:tasks.rados.rados.0.smithi107.stdout:11424: done (2 left) 2020-06-12T02:39:18.932 INFO:tasks.rados.rados.0.smithi107.stdout:11426: done (1 left) 2020-06-12T02:39:18.932 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:39:18.932 INFO:tasks.rados.rados.0.smithi107.stdout:11427: seq_num 11426 ranges {0=565248,565248=598016,1163264=499712,1662976=106496} 2020-06-12T02:39:18.933 INFO:tasks.rados.rados.0.smithi107.stdout:11427: writing smithi10729407-12 from 0 to 565248 tid 1 2020-06-12T02:39:18.933 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:39:18.927+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:39:18.933 INFO:tasks.rados.rados.0.smithi107.stdout:11427: writing smithi10729407-12 from 565248 to 1163264 tid 2 2020-06-12T02:39:18.936 INFO:tasks.rados.rados.0.smithi107.stdout:11427: writing smithi10729407-12 from 1163264 to 1662976 tid 3 2020-06-12T02:39:18.937 INFO:tasks.rados.rados.0.smithi107.stdout:11427: writing smithi10729407-12 from 1662976 to 1769472 tid 4 2020-06-12T02:39:18.937 INFO:tasks.rados.rados.0.smithi107.stdout:11428: delete oid 4 current snap is 0 2020-06-12T02:39:18.938 INFO:tasks.rados.rados.0.smithi107.stdout:11425: finishing write tid 2 to smithi10729407-15 2020-06-12T02:39:18.938 INFO:tasks.rados.rados.0.smithi107.stdout:11425: finishing write tid 3 to smithi10729407-15 2020-06-12T02:39:18.938 INFO:tasks.rados.rados.0.smithi107.stdout:11425: finishing write tid 4 to smithi10729407-15 2020-06-12T02:39:18.939 INFO:tasks.rados.rados.0.smithi107.stdout:11425: finishing write tid 6 to smithi10729407-15 2020-06-12T02:39:18.941 INFO:tasks.rados.rados.0.smithi107.stdout:11425: finishing write tid 7 to smithi10729407-15 2020-06-12T02:39:18.941 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 2143 (ObjNum 11424 snap 0 seq_num 11424) dirty exists 2020-06-12T02:39:18.942 INFO:tasks.rados.rados.0.smithi107.stdout:11425: left oid 15 (ObjNum 11424 snap 0 seq_num 11424) 2020-06-12T02:39:18.942 INFO:tasks.rados.rados.0.smithi107.stdout:11425: done (2 left) 2020-06-12T02:39:18.942 INFO:tasks.rados.rados.0.smithi107.stdout:11428: done (1 left) 2020-06-12T02:39:18.942 INFO:tasks.rados.rados.0.smithi107.stdout:11429: delete oid 6 current snap is 0 2020-06-12T02:39:18.943 INFO:tasks.rados.rados.0.smithi107.stdout:11429: done (1 left) 2020-06-12T02:39:18.943 INFO:tasks.rados.rados.0.smithi107.stdout:11430: delete oid 14 current snap is 0 2020-06-12T02:39:18.944 INFO:tasks.rados.rados.0.smithi107.stdout:11427: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:18.945 INFO:tasks.rados.rados.0.smithi107.stdout:11427: finishing write tid 2 to smithi10729407-12 2020-06-12T02:39:18.945 INFO:tasks.rados.rados.0.smithi107.stdout:11427: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:18.946 INFO:tasks.rados.rados.0.smithi107.stdout:11427: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:18.946 INFO:tasks.rados.rados.0.smithi107.stdout:11430: done (1 left) 2020-06-12T02:39:18.946 INFO:tasks.rados.rados.0.smithi107.stdout:11431: delete oid 15 current snap is 0 2020-06-12T02:39:18.948 INFO:tasks.rados.rados.0.smithi107.stdout:11431: done (1 left) 2020-06-12T02:39:18.949 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:39:18.949 INFO:tasks.rados.rados.0.smithi107.stdout:11432: seq_num 11431 ranges {0=655360,655360=475136,1130496=688128,1818624=40960} 2020-06-12T02:39:18.952 INFO:tasks.rados.rados.0.smithi107.stdout:11432: writing smithi10729407-6 from 0 to 655360 tid 1 2020-06-12T02:39:18.955 INFO:tasks.rados.rados.0.smithi107.stdout:11432: writing smithi10729407-6 from 655360 to 1130496 tid 2 2020-06-12T02:39:18.960 INFO:tasks.rados.rados.0.smithi107.stdout:11432: writing smithi10729407-6 from 1130496 to 1818624 tid 3 2020-06-12T02:39:18.961 INFO:tasks.rados.rados.0.smithi107.stdout:11432: writing smithi10729407-6 from 1818624 to 1859584 tid 4 2020-06-12T02:39:18.962 INFO:tasks.rados.rados.0.smithi107.stdout:11433: delete oid 1 current snap is 0 2020-06-12T02:39:18.962 INFO:tasks.rados.rados.0.smithi107.stdout:11427: finishing write tid 6 to smithi10729407-12 2020-06-12T02:39:18.962 INFO:tasks.rados.rados.0.smithi107.stdout:11427: finishing write tid 7 to smithi10729407-12 2020-06-12T02:39:18.962 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 2132 (ObjNum 11426 snap 0 seq_num 11426) dirty exists 2020-06-12T02:39:18.963 INFO:tasks.rados.rados.0.smithi107.stdout:11427: left oid 12 (ObjNum 11426 snap 0 seq_num 11426) 2020-06-12T02:39:18.963 INFO:tasks.rados.rados.0.smithi107.stdout:11432: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:18.970 INFO:tasks.rados.rados.0.smithi107.stdout:11432: finishing write tid 2 to smithi10729407-6 2020-06-12T02:39:18.970 INFO:tasks.rados.rados.0.smithi107.stdout:11432: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:18.971 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:39:18.972 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2020-06-12T02:39:18.972 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2020-06-12T02:39:18.972 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2020-06-12T02:39:18.973 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.973 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s2 26/21 ECSubWrite(tid=409, reqid=client.4512.0:1573, at_version=26'100, trim_to=0'0, roll_forward_to=26'98))", 2020-06-12T02:39:18.973 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.892374+0000", 2020-06-12T02:39:18.973 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 105.075613082, 2020-06-12T02:39:18.973 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014071322000000001, 2020-06-12T02:39:18.974 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:18.974 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:18.974 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:18.974 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.975 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:18.975 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.892374+0000", 2020-06-12T02:39:18.975 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:18.975 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.975 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.976 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:18.976 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.892374+0000", 2020-06-12T02:39:18.976 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047864199999999999 2020-06-12T02:39:18.976 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.977 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.977 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:18.977 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.892853+0000", 2020-06-12T02:39:18.977 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5139000000000001e-05 2020-06-12T02:39:18.978 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.978 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.978 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:18.978 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.892878+0000", 2020-06-12T02:39:18.978 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1543999999999999e-05 2020-06-12T02:39:18.979 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.979 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.979 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:18.979 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.892899+0000", 2020-06-12T02:39:18.980 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8989999999999996e-06 2020-06-12T02:39:18.980 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.980 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.981 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:18.981 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.892909+0000", 2020-06-12T02:39:18.981 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00023009199999999999 2020-06-12T02:39:18.981 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.982 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.982 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:18.982 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.893139+0000", 2020-06-12T02:39:18.982 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0044000000000001e-05 2020-06-12T02:39:18.983 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.983 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.983 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:18.983 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.893149+0000", 2020-06-12T02:39:18.983 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013233405 2020-06-12T02:39:18.984 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.984 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.984 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:18.984 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.906383+0000", 2020-06-12T02:39:18.985 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2557000000000004e-05 2020-06-12T02:39:18.985 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.985 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.985 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:18.985 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.906445+0000", 2020-06-12T02:39:18.986 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014071322000000001 2020-06-12T02:39:18.986 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.986 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:18.986 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:18.986 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.987 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.987 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s2 26/21 ECSubWrite(tid=410, reqid=client.4512.0:1574, at_version=26'101, trim_to=0'0, roll_forward_to=26'98))", 2020-06-12T02:39:18.987 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:33.895487+0000", 2020-06-12T02:39:18.987 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 105.072500287, 2020-06-12T02:39:18.987 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012904807000000001, 2020-06-12T02:39:18.988 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:18.988 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:18.989 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:18.989 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.989 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:18.989 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895487+0000", 2020-06-12T02:39:18.990 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:18.990 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.990 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.990 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:18.990 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895487+0000", 2020-06-12T02:39:18.991 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8478000000000002e-05 2020-06-12T02:39:18.991 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.991 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.991 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:18.992 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895575+0000", 2020-06-12T02:39:18.992 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8672e-05 2020-06-12T02:39:18.992 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.992 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.992 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:18.993 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895594+0000", 2020-06-12T02:39:18.993 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8028000000000001e-05 2020-06-12T02:39:18.993 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.993 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.994 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:18.994 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895612+0000", 2020-06-12T02:39:18.996 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2689999999999999e-06 2020-06-12T02:39:18.996 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.996 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.996 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:18.996 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.895620+0000", 2020-06-12T02:39:18.997 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0038940289999999998 2020-06-12T02:39:18.997 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.997 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.997 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:18.997 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.899514+0000", 2020-06-12T02:39:18.998 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4370000000000001e-06 2020-06-12T02:39:18.998 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.998 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.998 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:18.998 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.899522+0000", 2020-06-12T02:39:18.999 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0088184019999999995 2020-06-12T02:39:18.999 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:18.999 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:18.999 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:18.999 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.908340+0000", 2020-06-12T02:39:18.999 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1492000000000001e-05 2020-06-12T02:39:19.000 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.000 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.000 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:19.000 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:33.908392+0000", 2020-06-12T02:39:19.000 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012904807000000001 2020-06-12T02:39:19.001 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.001 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:19.001 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.001 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.001 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.002 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 27/21 ECSubWrite(tid=180, reqid=client.4512.0:2782, at_version=27'116, trim_to=0'0, roll_forward_to=27'115))", 2020-06-12T02:39:19.002 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:36.811166+0000", 2020-06-12T02:39:19.002 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 102.15682058199999, 2020-06-12T02:39:19.002 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013435703, 2020-06-12T02:39:19.002 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:19.003 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:19.003 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:19.003 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.003 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:19.003 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.811166+0000", 2020-06-12T02:39:19.003 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:19.004 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.004 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.004 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:19.004 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.811166+0000", 2020-06-12T02:39:19.005 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015779300000000001 2020-06-12T02:39:19.005 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.005 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.005 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:19.006 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.811324+0000", 2020-06-12T02:39:19.006 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8149000000000001e-05 2020-06-12T02:39:19.006 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.006 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.006 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:19.006 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.811352+0000", 2020-06-12T02:39:19.007 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6880000000000001e-05 2020-06-12T02:39:19.007 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.007 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.007 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:19.007 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.811369+0000", 2020-06-12T02:39:19.008 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1099999999999999e-06 2020-06-12T02:39:19.008 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.008 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.008 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:19.008 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.811375+0000", 2020-06-12T02:39:19.009 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0038906349999999999 2020-06-12T02:39:19.009 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.009 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.009 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:19.009 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.815266+0000", 2020-06-12T02:39:19.010 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0536e-05 2020-06-12T02:39:19.010 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.010 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.010 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:19.010 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.815277+0000", 2020-06-12T02:39:19.011 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0092955610000000008 2020-06-12T02:39:19.011 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.011 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.011 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:19.011 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.824572+0000", 2020-06-12T02:39:19.012 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0039000000000002e-05 2020-06-12T02:39:19.012 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.012 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.012 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:19.013 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:36.824602+0000", 2020-06-12T02:39:19.013 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013435703 2020-06-12T02:39:19.014 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.014 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:19.014 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.014 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.015 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.015 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 27/21 ECSubWrite(tid=226, reqid=client.4512.0:3334, at_version=0'0, trim_to=0'0, roll_forward_to=27'147))", 2020-06-12T02:39:19.015 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:37:38.627654+0000", 2020-06-12T02:39:19.015 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 100.34033282999999, 2020-06-12T02:39:19.015 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01294059, 2020-06-12T02:39:19.016 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:19.016 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:19.016 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:19.016 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.016 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:19.017 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.627654+0000", 2020-06-12T02:39:19.017 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:19.017 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.017 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.017 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:19.018 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.627654+0000", 2020-06-12T02:39:19.018 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6134000000000003e-05 2020-06-12T02:39:19.018 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.018 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.018 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:19.018 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.627690+0000", 2020-06-12T02:39:19.019 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1439999999999992e-06 2020-06-12T02:39:19.019 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.019 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.019 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:19.019 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.627699+0000", 2020-06-12T02:39:19.020 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.59e-05 2020-06-12T02:39:19.020 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.020 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.020 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:19.020 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.627715+0000", 2020-06-12T02:39:19.021 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.602e-06 2020-06-12T02:39:19.021 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.021 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.021 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:19.022 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.627724+0000", 2020-06-12T02:39:19.022 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0059999999999998e-05 2020-06-12T02:39:19.022 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.022 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.022 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:19.023 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.627784+0000", 2020-06-12T02:39:19.023 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1885999999999999e-05 2020-06-12T02:39:19.023 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.023 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.023 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:19.024 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.627796+0000", 2020-06-12T02:39:19.024 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012765789 2020-06-12T02:39:19.024 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.024 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.025 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:19.025 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.640562+0000", 2020-06-12T02:39:19.025 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3074999999999998e-05 2020-06-12T02:39:19.025 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.026 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.026 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:19.026 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:37:38.640595+0000", 2020-06-12T02:39:19.026 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01294059 2020-06-12T02:39:19.026 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.027 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:19.027 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.027 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.027 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.027 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 49/21 ECSubWrite(tid=652, reqid=client.4512.0:8809, at_version=49'402, trim_to=0'0, roll_forward_to=36'401))", 2020-06-12T02:39:19.028 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.708711+0000", 2020-06-12T02:39:19.028 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 67.259276267999994, 2020-06-12T02:39:19.028 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011533180000000001, 2020-06-12T02:39:19.028 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:19.029 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:19.030 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:19.030 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.030 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:19.030 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.708711+0000", 2020-06-12T02:39:19.031 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:19.031 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.031 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.031 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:19.031 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.708711+0000", 2020-06-12T02:39:19.032 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00023433800000000001 2020-06-12T02:39:19.032 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.032 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.032 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:19.032 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.708945+0000", 2020-06-12T02:39:19.033 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5338999999999999e-05 2020-06-12T02:39:19.033 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.033 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.033 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:19.033 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.708970+0000", 2020-06-12T02:39:19.034 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7054e-05 2020-06-12T02:39:19.034 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.034 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.034 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:19.034 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.708987+0000", 2020-06-12T02:39:19.035 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8599999999999993e-06 2020-06-12T02:39:19.035 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.035 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.035 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:19.036 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.708995+0000", 2020-06-12T02:39:19.036 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3531e-05 2020-06-12T02:39:19.036 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.036 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.036 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:19.036 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.709049+0000", 2020-06-12T02:39:19.037 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2737e-05 2020-06-12T02:39:19.037 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.037 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.037 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:19.037 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.709062+0000", 2020-06-12T02:39:19.038 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011127211999999999 2020-06-12T02:39:19.038 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.038 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.038 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:19.039 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.720189+0000", 2020-06-12T02:39:19.039 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5108999999999997e-05 2020-06-12T02:39:19.039 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.040 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.040 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:19.040 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.720244+0000", 2020-06-12T02:39:19.041 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011533180000000001 2020-06-12T02:39:19.041 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.042 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:19.042 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.042 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.042 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.042 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 49/21 ECSubWrite(tid=663, reqid=client.4512.0:8855, at_version=49'409, trim_to=0'0, roll_forward_to=49'407))", 2020-06-12T02:39:19.043 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.866635+0000", 2020-06-12T02:39:19.043 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 67.101351941999994, 2020-06-12T02:39:19.043 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01099082, 2020-06-12T02:39:19.044 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:19.044 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:19.044 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:19.044 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.045 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:19.045 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.866635+0000", 2020-06-12T02:39:19.045 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:19.046 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.046 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.046 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:19.046 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.866635+0000", 2020-06-12T02:39:19.046 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000193139 2020-06-12T02:39:19.047 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.047 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.047 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:19.047 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.866828+0000", 2020-06-12T02:39:19.047 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7207999999999999e-05 2020-06-12T02:39:19.048 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.048 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.048 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:19.048 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.866845+0000", 2020-06-12T02:39:19.048 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4484999999999999e-05 2020-06-12T02:39:19.048 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.049 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.049 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:19.049 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.866860+0000", 2020-06-12T02:39:19.049 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6740000000000002e-06 2020-06-12T02:39:19.049 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.050 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.050 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:19.050 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.866866+0000", 2020-06-12T02:39:19.050 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8099999999999997e-05 2020-06-12T02:39:19.050 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.051 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.051 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:19.051 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.866914+0000", 2020-06-12T02:39:19.051 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1518999999999999e-05 2020-06-12T02:39:19.051 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.051 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.052 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:19.052 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.866925+0000", 2020-06-12T02:39:19.052 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010641576 2020-06-12T02:39:19.052 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.052 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.053 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:19.053 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.877567+0000", 2020-06-12T02:39:19.053 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9119000000000001e-05 2020-06-12T02:39:19.053 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.053 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.053 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:19.054 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.877626+0000", 2020-06-12T02:39:19.054 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01099082 2020-06-12T02:39:19.054 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.054 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:19.054 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.055 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.055 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.055 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.9s1 49/37 ECSubWrite(tid=4126, reqid=client.4512.0:8876, at_version=49'700, trim_to=0'0, roll_forward_to=49'697))", 2020-06-12T02:39:19.055 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:11.918553+0000", 2020-06-12T02:39:19.056 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 67.049433871000005, 2020-06-12T02:39:19.056 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011086824, 2020-06-12T02:39:19.056 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:19.056 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:19.056 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:19.057 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.057 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:19.057 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.918553+0000", 2020-06-12T02:39:19.057 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:19.057 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.058 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.058 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:19.058 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.918553+0000", 2020-06-12T02:39:19.058 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00024818900000000001 2020-06-12T02:39:19.058 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.058 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.059 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:19.059 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.918801+0000", 2020-06-12T02:39:19.059 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3191e-05 2020-06-12T02:39:19.059 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.059 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.060 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:19.060 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.918815+0000", 2020-06-12T02:39:19.060 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0339999999999999e-05 2020-06-12T02:39:19.060 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.060 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.061 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:19.061 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.918825+0000", 2020-06-12T02:39:19.061 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2440000000000004e-06 2020-06-12T02:39:19.061 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.061 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.061 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:19.062 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.918829+0000", 2020-06-12T02:39:19.062 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4657000000000003e-05 2020-06-12T02:39:19.062 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.062 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.062 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:19.063 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.918874+0000", 2020-06-12T02:39:19.063 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4171e-05 2020-06-12T02:39:19.063 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.063 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.063 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:19.064 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.918888+0000", 2020-06-12T02:39:19.064 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010692896 2020-06-12T02:39:19.064 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.064 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.064 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:19.064 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.929581+0000", 2020-06-12T02:39:19.065 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9135999999999998e-05 2020-06-12T02:39:19.065 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.065 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.065 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:19.066 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:11.929640+0000", 2020-06-12T02:39:19.066 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011086824 2020-06-12T02:39:19.066 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.066 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:19.066 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.067 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.067 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.067 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.9s1 49/37 ECSubWrite(tid=4209, reqid=client.4512.0:9129, at_version=49'718, trim_to=0'0, roll_forward_to=49'716))", 2020-06-12T02:39:19.067 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:12.647408+0000", 2020-06-12T02:39:19.068 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 66.320579096000003, 2020-06-12T02:39:19.068 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01161733, 2020-06-12T02:39:19.068 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:19.068 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:19.068 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:19.068 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.069 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:19.069 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.647408+0000", 2020-06-12T02:39:19.069 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:19.069 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.069 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.070 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:19.070 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.647408+0000", 2020-06-12T02:39:19.070 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00022105000000000001 2020-06-12T02:39:19.070 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.070 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.071 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:19.071 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.647629+0000", 2020-06-12T02:39:19.071 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2779e-05 2020-06-12T02:39:19.071 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.071 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.072 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:19.072 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.647652+0000", 2020-06-12T02:39:19.072 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0165000000000001e-05 2020-06-12T02:39:19.072 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.072 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.073 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:19.073 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.647672+0000", 2020-06-12T02:39:19.073 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2609999999999993e-06 2020-06-12T02:39:19.073 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.073 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.074 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:19.074 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.647680+0000", 2020-06-12T02:39:19.075 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2991999999999997e-05 2020-06-12T02:39:19.075 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.075 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.075 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:19.075 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.647733+0000", 2020-06-12T02:39:19.076 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4059e-05 2020-06-12T02:39:19.076 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.077 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.077 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:19.078 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.647747+0000", 2020-06-12T02:39:19.078 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011169695 2020-06-12T02:39:19.078 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.078 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.079 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:19.079 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.658917+0000", 2020-06-12T02:39:19.079 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000108329 2020-06-12T02:39:19.079 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.079 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.080 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:19.080 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:12.659025+0000", 2020-06-12T02:39:19.080 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01161733 2020-06-12T02:39:19.080 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.080 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:19.081 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.081 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.081 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.081 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.9s1 54/37 ECSubWrite(tid=4461, reqid=client.4512.0:10001, at_version=54'808, trim_to=0'0, roll_forward_to=54'806))", 2020-06-12T02:39:19.081 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:17.943598+0000", 2020-06-12T02:39:19.081 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 61.024389460000002, 2020-06-12T02:39:19.082 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011839257000000001, 2020-06-12T02:39:19.082 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:19.082 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:19.082 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:19.082 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.083 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:19.083 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.943598+0000", 2020-06-12T02:39:19.083 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:19.083 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.083 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.084 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:19.084 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.943598+0000", 2020-06-12T02:39:19.084 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00028123299999999998 2020-06-12T02:39:19.084 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.084 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.085 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:19.085 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.943879+0000", 2020-06-12T02:39:19.085 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6650999999999999e-05 2020-06-12T02:39:19.085 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.085 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.085 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:19.086 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.943905+0000", 2020-06-12T02:39:19.087 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0922000000000001e-05 2020-06-12T02:39:19.087 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.088 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.088 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:19.088 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.943926+0000", 2020-06-12T02:39:19.089 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8770000000000005e-06 2020-06-12T02:39:19.089 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.089 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.090 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:19.090 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.943936+0000", 2020-06-12T02:39:19.090 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4051e-05 2020-06-12T02:39:19.091 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.092 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.092 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:19.092 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.944010+0000", 2020-06-12T02:39:19.093 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3377000000000001e-05 2020-06-12T02:39:19.093 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.093 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.094 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:19.094 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.944024+0000", 2020-06-12T02:39:19.095 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011355466 2020-06-12T02:39:19.095 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.095 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.095 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:19.096 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.955379+0000", 2020-06-12T02:39:19.096 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7679999999999997e-05 2020-06-12T02:39:19.097 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.097 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.097 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:19.098 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:17.955437+0000", 2020-06-12T02:39:19.098 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011839257000000001 2020-06-12T02:39:19.098 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.098 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:19.099 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.099 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.099 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.100 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 59/53 ECSubWrite(tid=44, reqid=client.4512.0:14904, at_version=0'0, trim_to=0'0, roll_forward_to=59'463))", 2020-06-12T02:39:19.100 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.874670+0000", 2020-06-12T02:39:19.100 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 47.093316692000002, 2020-06-12T02:39:19.100 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054588419999999999, 2020-06-12T02:39:19.101 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:19.101 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:19.104 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:19.105 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.105 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:19.105 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.874670+0000", 2020-06-12T02:39:19.105 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:19.106 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.106 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.106 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:19.106 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.874670+0000", 2020-06-12T02:39:19.106 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.7840000000000001e-05 2020-06-12T02:39:19.107 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.107 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.107 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:19.107 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.874698+0000", 2020-06-12T02:39:19.107 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3350000000000007e-06 2020-06-12T02:39:19.108 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.108 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.109 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:19.109 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.874707+0000", 2020-06-12T02:39:19.109 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5979999999999997e-06 2020-06-12T02:39:19.109 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.109 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.110 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:19.110 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.874715+0000", 2020-06-12T02:39:19.110 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1420000000000003e-06 2020-06-12T02:39:19.110 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.115 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.116 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:19.116 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.874719+0000", 2020-06-12T02:39:19.116 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2641000000000001e-05 2020-06-12T02:39:19.116 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.116 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.116 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:19.117 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.874752+0000", 2020-06-12T02:39:19.117 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6160000000000002e-06 2020-06-12T02:39:19.117 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.117 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.117 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:19.118 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.874761+0000", 2020-06-12T02:39:19.118 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054451052 2020-06-12T02:39:19.119 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.119 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.119 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:19.119 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.929212+0000", 2020-06-12T02:39:19.120 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7196e-05 2020-06-12T02:39:19.120 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.120 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.120 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:19.120 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.929259+0000", 2020-06-12T02:39:19.120 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054588419999999999 2020-06-12T02:39:19.121 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.121 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:19.121 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.121 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.121 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.122 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s2 59/29 ECSubWrite(tid=7891, reqid=client.4512.0:15240, at_version=59'1239, trim_to=0'0, roll_forward_to=59'1238))", 2020-06-12T02:39:19.122 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.947393+0000", 2020-06-12T02:39:19.122 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 46.020593615999999, 2020-06-12T02:39:19.122 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013284153, 2020-06-12T02:39:19.123 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:19.123 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:19.123 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:19.123 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.123 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:19.124 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.947393+0000", 2020-06-12T02:39:19.124 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:19.124 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.124 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.124 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:19.124 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.947393+0000", 2020-06-12T02:39:19.125 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119331 2020-06-12T02:39:19.125 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.125 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.125 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:19.125 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.947513+0000", 2020-06-12T02:39:19.126 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9915000000000002e-05 2020-06-12T02:39:19.126 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.126 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.127 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:19.127 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.947533+0000", 2020-06-12T02:39:19.127 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5891000000000001e-05 2020-06-12T02:39:19.127 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.127 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.128 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:19.128 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.947549+0000", 2020-06-12T02:39:19.128 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6919999999999996e-06 2020-06-12T02:39:19.129 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.129 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.129 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:19.129 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.947555+0000", 2020-06-12T02:39:19.130 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0016197220000000001 2020-06-12T02:39:19.130 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.130 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.130 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:19.130 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.949175+0000", 2020-06-12T02:39:19.131 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4054e-05 2020-06-12T02:39:19.131 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.131 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.131 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:19.131 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.949189+0000", 2020-06-12T02:39:19.132 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011443251999999999 2020-06-12T02:39:19.132 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.132 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.132 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:19.132 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.960632+0000", 2020-06-12T02:39:19.133 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5296000000000001e-05 2020-06-12T02:39:19.134 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.134 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.134 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:19.134 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.960678+0000", 2020-06-12T02:39:19.135 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013284153 2020-06-12T02:39:19.135 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.135 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:19.135 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.135 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.136 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.136 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s2 59/29 ECSubWrite(tid=7893, reqid=client.4512.0:15241, at_version=59'1240, trim_to=0'0, roll_forward_to=59'1238))", 2020-06-12T02:39:19.136 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.951130+0000", 2020-06-12T02:39:19.136 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 46.016856826999998, 2020-06-12T02:39:19.137 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017482992999999999, 2020-06-12T02:39:19.137 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:19.137 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:19.137 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:19.137 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.138 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:19.138 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.951130+0000", 2020-06-12T02:39:19.138 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:19.138 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.138 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.139 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:19.139 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.951130+0000", 2020-06-12T02:39:19.139 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000113396 2020-06-12T02:39:19.139 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.139 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.140 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:19.140 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.951244+0000", 2020-06-12T02:39:19.140 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0119000000000002e-05 2020-06-12T02:39:19.140 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.140 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.141 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:19.141 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.951264+0000", 2020-06-12T02:39:19.141 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5778999999999999e-05 2020-06-12T02:39:19.141 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.141 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.142 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:19.142 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.951279+0000", 2020-06-12T02:39:19.142 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.127e-06 2020-06-12T02:39:19.142 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.142 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.143 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:19.143 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.951286+0000", 2020-06-12T02:39:19.143 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00086118200000000005 2020-06-12T02:39:19.143 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.143 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.144 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:19.144 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.952147+0000", 2020-06-12T02:39:19.144 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7280000000000007e-06 2020-06-12T02:39:19.144 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.144 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.145 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:19.145 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.952157+0000", 2020-06-12T02:39:19.145 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016405067999999998 2020-06-12T02:39:19.146 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.146 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.146 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:19.146 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.968562+0000", 2020-06-12T02:39:19.146 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1594000000000001e-05 2020-06-12T02:39:19.147 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.147 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.147 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:19.148 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.968613+0000", 2020-06-12T02:39:19.148 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017482992999999999 2020-06-12T02:39:19.148 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.148 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:19.148 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.149 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.149 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.149 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s2 59/29 ECSubWrite(tid=7895, reqid=client.4512.0:15242, at_version=59'1241, trim_to=0'0, roll_forward_to=59'1238))", 2020-06-12T02:39:19.149 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.954847+0000", 2020-06-12T02:39:19.149 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 46.013140155000002, 2020-06-12T02:39:19.150 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01380811, 2020-06-12T02:39:19.150 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:19.150 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:19.150 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:19.150 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.151 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:19.151 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954847+0000", 2020-06-12T02:39:19.151 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:19.151 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.151 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.152 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:19.152 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954847+0000", 2020-06-12T02:39:19.152 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1149000000000005e-05 2020-06-12T02:39:19.152 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.152 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.153 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:19.153 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954928+0000", 2020-06-12T02:39:19.153 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2370000000000001e-05 2020-06-12T02:39:19.153 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.154 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.154 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:19.154 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954950+0000", 2020-06-12T02:39:19.154 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6290000000000002e-05 2020-06-12T02:39:19.155 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.155 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.155 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:19.155 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954967+0000", 2020-06-12T02:39:19.156 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1500000000000004e-06 2020-06-12T02:39:19.156 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.156 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.156 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:19.156 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.954973+0000", 2020-06-12T02:39:19.157 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0057361110000000003 2020-06-12T02:39:19.157 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.157 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.157 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:19.157 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.960709+0000", 2020-06-12T02:39:19.158 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3400000000000003e-06 2020-06-12T02:39:19.158 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.158 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.158 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:19.158 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.960715+0000", 2020-06-12T02:39:19.158 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0079014319999999999 2020-06-12T02:39:19.159 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.159 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.159 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:19.159 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.968617+0000", 2020-06-12T02:39:19.159 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8268e-05 2020-06-12T02:39:19.161 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.161 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.162 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:19.162 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.968655+0000", 2020-06-12T02:39:19.162 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01380811 2020-06-12T02:39:19.163 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.163 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:19.163 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.164 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.164 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.164 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s2 59/29 ECSubWrite(tid=7896, reqid=client.4512.0:15243, at_version=59'1242, trim_to=0'0, roll_forward_to=59'1238))", 2020-06-12T02:39:19.164 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.957094+0000", 2020-06-12T02:39:19.165 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 46.010893451000001, 2020-06-12T02:39:19.165 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011598476, 2020-06-12T02:39:19.165 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:19.166 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:19.166 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:19.166 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.167 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:19.167 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.957094+0000", 2020-06-12T02:39:19.167 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:19.168 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.168 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.168 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:19.169 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.957094+0000", 2020-06-12T02:39:19.169 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0612000000000002e-05 2020-06-12T02:39:19.170 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.170 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.170 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:19.170 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.957154+0000", 2020-06-12T02:39:19.171 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9766e-05 2020-06-12T02:39:19.171 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.171 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.172 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:19.172 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.957174+0000", 2020-06-12T02:39:19.173 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5478999999999999e-05 2020-06-12T02:39:19.173 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.173 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.174 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:19.174 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.957189+0000", 2020-06-12T02:39:19.174 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6460000000000004e-06 2020-06-12T02:39:19.175 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.175 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.175 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:19.176 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.957196+0000", 2020-06-12T02:39:19.176 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0047543350000000002 2020-06-12T02:39:19.176 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.177 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.177 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:19.178 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.961950+0000", 2020-06-12T02:39:19.178 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2198000000000001e-05 2020-06-12T02:39:19.178 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.179 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.179 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:19.179 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.961963+0000", 2020-06-12T02:39:19.180 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0066945809999999998 2020-06-12T02:39:19.180 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.180 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.181 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:19.181 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.968657+0000", 2020-06-12T02:39:19.181 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4858999999999999e-05 2020-06-12T02:39:19.182 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.182 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.182 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:19.182 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.968692+0000", 2020-06-12T02:39:19.183 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011598476 2020-06-12T02:39:19.183 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.183 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:19.184 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.184 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.184 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.185 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.9s1 59/37 ECSubWrite(tid=5986, reqid=client.4512.0:15292, at_version=59'1321, trim_to=0'0, roll_forward_to=59'1318))", 2020-06-12T02:39:19.185 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:33.119863+0000", 2020-06-12T02:39:19.185 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 45.848123837999999, 2020-06-12T02:39:19.186 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011994942999999999, 2020-06-12T02:39:19.186 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:19.186 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:19.187 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:19.187 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.187 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:19.188 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.119863+0000", 2020-06-12T02:39:19.188 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:19.188 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.189 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.189 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:19.189 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.119863+0000", 2020-06-12T02:39:19.190 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00029855799999999998 2020-06-12T02:39:19.190 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.190 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.190 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:19.191 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.120162+0000", 2020-06-12T02:39:19.191 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0029000000000001e-05 2020-06-12T02:39:19.191 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.192 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.192 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:19.192 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.120182+0000", 2020-06-12T02:39:19.192 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7958e-05 2020-06-12T02:39:19.193 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.193 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.194 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:19.194 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.120200+0000", 2020-06-12T02:39:19.194 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0509999999999999e-06 2020-06-12T02:39:19.195 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.195 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.195 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:19.195 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.120207+0000", 2020-06-12T02:39:19.195 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000110938 2020-06-12T02:39:19.196 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.196 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.196 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:19.196 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.120318+0000", 2020-06-12T02:39:19.196 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8726999999999999e-05 2020-06-12T02:39:19.196 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.197 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.197 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:19.197 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.120336+0000", 2020-06-12T02:39:19.197 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011476471 2020-06-12T02:39:19.198 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.198 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.198 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:19.198 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.131813+0000", 2020-06-12T02:39:19.198 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5210999999999998e-05 2020-06-12T02:39:19.199 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.199 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.199 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:19.199 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.131858+0000", 2020-06-12T02:39:19.199 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011994942999999999 2020-06-12T02:39:19.199 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.200 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:19.200 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.200 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.200 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.200 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.0s1 59/53 ECSubWrite(tid=72, reqid=client.4512.0:15331, at_version=59'481, trim_to=0'0, roll_forward_to=59'480))", 2020-06-12T02:39:19.201 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:33.277833+0000", 2020-06-12T02:39:19.201 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 45.690153576999997, 2020-06-12T02:39:19.201 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011615464000000001, 2020-06-12T02:39:19.201 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:19.201 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:19.201 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:19.202 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.202 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:19.202 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.277833+0000", 2020-06-12T02:39:19.202 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:19.202 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.203 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.203 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:19.203 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.277833+0000", 2020-06-12T02:39:19.203 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015142500000000001 2020-06-12T02:39:19.203 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.203 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.204 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:19.204 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.277985+0000", 2020-06-12T02:39:19.204 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9533000000000001e-05 2020-06-12T02:39:19.204 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.204 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.205 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:19.205 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.278004+0000", 2020-06-12T02:39:19.205 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1656e-05 2020-06-12T02:39:19.205 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.205 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.205 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:19.206 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.278016+0000", 2020-06-12T02:39:19.206 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.874e-06 2020-06-12T02:39:19.206 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.206 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.206 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:19.207 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.278022+0000", 2020-06-12T02:39:19.207 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6635000000000002e-05 2020-06-12T02:39:19.207 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.207 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.207 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:19.207 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.278079+0000", 2020-06-12T02:39:19.208 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5098e-05 2020-06-12T02:39:19.208 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.208 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.208 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:19.209 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.278094+0000", 2020-06-12T02:39:19.209 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011301432 2020-06-12T02:39:19.209 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.210 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.210 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:19.210 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.289395+0000", 2020-06-12T02:39:19.210 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3810999999999997e-05 2020-06-12T02:39:19.210 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.211 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.211 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:19.211 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:33.289449+0000", 2020-06-12T02:39:19.211 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011615464000000001 2020-06-12T02:39:19.211 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.212 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:19.212 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.212 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.212 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.212 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s2 61/29 ECSubWrite(tid=8614, reqid=client.4512.0:16426, at_version=61'1366, trim_to=0'0, roll_forward_to=61'1364))", 2020-06-12T02:39:19.212 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:36.535477+0000", 2020-06-12T02:39:19.213 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 42.432509852000003, 2020-06-12T02:39:19.213 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013175393000000001, 2020-06-12T02:39:19.213 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:19.213 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:19.213 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:19.214 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.214 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:19.214 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.535477+0000", 2020-06-12T02:39:19.214 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:19.214 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.214 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.215 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:19.215 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.535477+0000", 2020-06-12T02:39:19.215 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000108959 2020-06-12T02:39:19.215 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.215 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.216 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:19.216 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.535586+0000", 2020-06-12T02:39:19.216 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6935999999999998e-05 2020-06-12T02:39:19.216 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.217 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.217 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:19.217 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.535603+0000", 2020-06-12T02:39:19.217 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1772e-05 2020-06-12T02:39:19.217 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.218 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.218 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:19.218 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.535615+0000", 2020-06-12T02:39:19.218 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6449999999999999e-06 2020-06-12T02:39:19.218 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.219 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.219 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:19.219 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.535621+0000", 2020-06-12T02:39:19.219 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5775000000000001e-05 2020-06-12T02:39:19.219 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.219 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.220 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:19.220 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.535667+0000", 2020-06-12T02:39:19.220 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2561999999999999e-05 2020-06-12T02:39:19.220 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.220 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.220 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:19.221 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.535680+0000", 2020-06-12T02:39:19.221 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012913482 2020-06-12T02:39:19.221 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.221 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.221 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:19.222 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.548593+0000", 2020-06-12T02:39:19.222 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9262000000000003e-05 2020-06-12T02:39:19.222 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.222 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.222 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:19.223 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.548653+0000", 2020-06-12T02:39:19.223 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013175393000000001 2020-06-12T02:39:19.223 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.223 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:19.223 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.223 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.224 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.224 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s2 62/29 ECSubWrite(tid=9525, reqid=client.4512.0:18291, at_version=62'1502, trim_to=0'0, roll_forward_to=62'1500))", 2020-06-12T02:39:19.224 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:41.996668+0000", 2020-06-12T02:39:19.224 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 36.971318846000003, 2020-06-12T02:39:19.224 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011313457000000001, 2020-06-12T02:39:19.225 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:19.225 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:19.225 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:19.225 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.226 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:19.226 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.996668+0000", 2020-06-12T02:39:19.226 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:19.226 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.226 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.226 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:19.227 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.996668+0000", 2020-06-12T02:39:19.227 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011186799999999999 2020-06-12T02:39:19.227 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.227 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.227 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:19.228 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.996780+0000", 2020-06-12T02:39:19.228 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2617e-05 2020-06-12T02:39:19.228 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.228 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.228 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:19.228 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.996803+0000", 2020-06-12T02:39:19.229 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9653999999999999e-05 2020-06-12T02:39:19.229 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.229 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.229 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:19.229 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.996822+0000", 2020-06-12T02:39:19.230 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4410000000000004e-06 2020-06-12T02:39:19.230 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.230 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.230 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:19.230 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.996830+0000", 2020-06-12T02:39:19.231 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001336519 2020-06-12T02:39:19.231 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.231 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.231 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:19.231 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.998166+0000", 2020-06-12T02:39:19.231 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0524e-05 2020-06-12T02:39:19.232 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.232 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.232 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:19.232 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:41.998207+0000", 2020-06-12T02:39:19.232 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097440600000000006 2020-06-12T02:39:19.232 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.233 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.233 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:19.233 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.007951+0000", 2020-06-12T02:39:19.233 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0774e-05 2020-06-12T02:39:19.233 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.234 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.234 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:19.234 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:42.007982+0000", 2020-06-12T02:39:19.234 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011313457000000001 2020-06-12T02:39:19.234 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.234 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:19.235 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.235 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.235 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.235 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s2 83/76 ECSubWrite(tid=1495, reqid=client.4512.0:22459, at_version=83'1715, trim_to=0'0, roll_forward_to=83'1713))", 2020-06-12T02:39:19.235 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:39:11.589372+0000", 2020-06-12T02:39:19.236 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 7.3786154000000002, 2020-06-12T02:39:19.236 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011037343, 2020-06-12T02:39:19.236 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:19.236 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:19.236 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:19.236 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.237 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:19.237 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.589372+0000", 2020-06-12T02:39:19.237 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:19.237 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.237 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.238 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:19.238 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.589372+0000", 2020-06-12T02:39:19.238 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00046881800000000002 2020-06-12T02:39:19.238 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.238 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.239 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:19.239 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.589840+0000", 2020-06-12T02:39:19.239 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8403999999999999e-05 2020-06-12T02:39:19.239 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.239 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.239 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:19.240 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.589859+0000", 2020-06-12T02:39:19.240 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4915e-05 2020-06-12T02:39:19.240 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.240 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.240 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:19.241 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.589874+0000", 2020-06-12T02:39:19.241 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1809999999999998e-06 2020-06-12T02:39:19.241 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.241 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.242 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:19.242 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.589880+0000", 2020-06-12T02:39:19.242 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017840799999999999 2020-06-12T02:39:19.242 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.242 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.242 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:19.243 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.590058+0000", 2020-06-12T02:39:19.243 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1355999999999999e-05 2020-06-12T02:39:19.243 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.243 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.243 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:19.244 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.590070+0000", 2020-06-12T02:39:19.244 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010282136000000001 2020-06-12T02:39:19.244 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.244 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.244 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:19.245 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.600352+0000", 2020-06-12T02:39:19.245 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7124999999999999e-05 2020-06-12T02:39:19.245 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.245 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.245 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:19.246 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.600409+0000", 2020-06-12T02:39:19.246 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011037343 2020-06-12T02:39:19.246 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.246 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:19.246 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.247 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.247 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.247 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWrite(3.5s2 83/76 ECSubWrite(tid=1496, reqid=client.4512.0:22460, at_version=83'1716, trim_to=0'0, roll_forward_to=83'1713))", 2020-06-12T02:39:19.247 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:39:11.592750+0000", 2020-06-12T02:39:19.247 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 7.3752365649999998, 2020-06-12T02:39:19.248 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010961372, 2020-06-12T02:39:19.248 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:19.248 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:19.248 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:19.248 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.248 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:19.249 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.592750+0000", 2020-06-12T02:39:19.249 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:19.249 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.249 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.249 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:19.250 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.592750+0000", 2020-06-12T02:39:19.250 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00050692000000000001 2020-06-12T02:39:19.250 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.250 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.250 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:19.250 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.593257+0000", 2020-06-12T02:39:19.251 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6143e-05 2020-06-12T02:39:19.251 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.251 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.251 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:19.251 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.593284+0000", 2020-06-12T02:39:19.252 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8719000000000002e-05 2020-06-12T02:39:19.252 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.252 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.252 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:19.252 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.593302+0000", 2020-06-12T02:39:19.252 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.109e-06 2020-06-12T02:39:19.253 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.253 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.253 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:19.253 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.593309+0000", 2020-06-12T02:39:19.253 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3928000000000003e-05 2020-06-12T02:39:19.254 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.254 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.254 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:19.254 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.593363+0000", 2020-06-12T02:39:19.254 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2108e-05 2020-06-12T02:39:19.254 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.255 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.255 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:19.255 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.593375+0000", 2020-06-12T02:39:19.255 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010300729 2020-06-12T02:39:19.255 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.256 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.256 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:19.256 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.603676+0000", 2020-06-12T02:39:19.256 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5716000000000002e-05 2020-06-12T02:39:19.257 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.257 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.257 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:19.257 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:11.603712+0000", 2020-06-12T02:39:19.257 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010961372 2020-06-12T02:39:19.257 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.258 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:19.258 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.258 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.258 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:19.259 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:39:19.259 INFO:tasks.rados.rados.0.smithi107.stdout:11432: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:19.259 INFO:tasks.rados.rados.0.smithi107.stdout:11432: finishing write tid 6 to smithi10729407-6 2020-06-12T02:39:19.259 INFO:tasks.rados.rados.0.smithi107.stdout:11427: done (2 left) 2020-06-12T02:39:19.259 INFO:tasks.rados.rados.0.smithi107.stdout:11433: done (1 left) 2020-06-12T02:39:19.260 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:39:19.260 INFO:tasks.rados.rados.0.smithi107.stdout:11434: seq_num 11433 ranges {1204224=688128,1892352=638976,2531328=565248} 2020-06-12T02:39:19.271 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-06-12T02:39:19.272 INFO:tasks.rados.rados.0.smithi107.stdout:11434: writing smithi10729407-11 from 1204224 to 1892352 tid 1 2020-06-12T02:39:19.273 INFO:tasks.rados.rados.0.smithi107.stdout:11434: writing smithi10729407-11 from 1892352 to 2531328 tid 2 2020-06-12T02:39:19.273 INFO:tasks.rados.rados.0.smithi107.stdout:11434: writing smithi10729407-11 from 2531328 to 3096576 tid 3 2020-06-12T02:39:19.273 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:39:19.273 INFO:tasks.rados.rados.0.smithi107.stdout:11435: seq_num 11434 ranges {0=704512,704512=573440,1277952=458752,1736704=172032} 2020-06-12T02:39:19.274 INFO:tasks.rados.rados.0.smithi107.stdout:11435: writing smithi10729407-10 from 0 to 704512 tid 1 2020-06-12T02:39:19.274 INFO:tasks.rados.rados.0.smithi107.stdout:11435: writing smithi10729407-10 from 704512 to 1277952 tid 2 2020-06-12T02:39:19.274 INFO:tasks.rados.rados.0.smithi107.stdout:11435: writing smithi10729407-10 from 1277952 to 1736704 tid 3 2020-06-12T02:39:19.274 INFO:tasks.rados.rados.0.smithi107.stdout:11435: writing smithi10729407-10 from 1736704 to 1908736 tid 4 2020-06-12T02:39:19.274 INFO:tasks.rados.rados.0.smithi107.stdout:11436: delete oid 1 current snap is 0 2020-06-12T02:39:19.275 INFO:tasks.rados.rados.0.smithi107.stdout:11432: finishing write tid 7 to smithi10729407-6 2020-06-12T02:39:19.275 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 2070 (ObjNum 11431 snap 0 seq_num 11431) dirty exists 2020-06-12T02:39:19.275 INFO:tasks.rados.rados.0.smithi107.stdout:11432: left oid 6 (ObjNum 11431 snap 0 seq_num 11431) 2020-06-12T02:39:19.275 INFO:tasks.rados.rados.0.smithi107.stdout:11434: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:19.275 INFO:tasks.rados.rados.0.smithi107.stdout:11434: finishing write tid 2 to smithi10729407-11 2020-06-12T02:39:19.275 INFO:tasks.rados.rados.0.smithi107.stdout:11434: finishing write tid 3 to smithi10729407-11 2020-06-12T02:39:19.276 INFO:tasks.rados.rados.0.smithi107.stdout:11434: finishing write tid 5 to smithi10729407-11 2020-06-12T02:39:19.276 INFO:tasks.rados.rados.0.smithi107.stdout:11434: finishing write tid 6 to smithi10729407-11 2020-06-12T02:39:19.276 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1801 (ObjNum 11433 snap 0 seq_num 11433) dirty exists 2020-06-12T02:39:19.276 INFO:tasks.rados.rados.0.smithi107.stdout:11434: left oid 11 (ObjNum 11433 snap 0 seq_num 11433) 2020-06-12T02:39:19.276 INFO:tasks.rados.rados.0.smithi107.stdout:11435: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:19.277 INFO:tasks.rados.rados.0.smithi107.stdout:11432: done (3 left) 2020-06-12T02:39:19.277 INFO:tasks.rados.rados.0.smithi107.stdout:11434: done (2 left) 2020-06-12T02:39:19.277 INFO:tasks.rados.rados.0.smithi107.stdout:11436: done (1 left) 2020-06-12T02:39:19.277 INFO:tasks.rados.rados.0.smithi107.stdout:11437: delete oid 4 current snap is 0 2020-06-12T02:39:19.277 INFO:tasks.rados.rados.0.smithi107.stdout:11435: finishing write tid 2 to smithi10729407-10 2020-06-12T02:39:19.278 INFO:tasks.rados.rados.0.smithi107.stdout:11435: finishing write tid 3 to smithi10729407-10 2020-06-12T02:39:19.278 INFO:tasks.rados.rados.0.smithi107.stdout:11435: finishing write tid 4 to smithi10729407-10 2020-06-12T02:39:19.278 INFO:tasks.rados.rados.0.smithi107.stdout:11437: done (1 left) 2020-06-12T02:39:19.278 INFO:tasks.rados.rados.0.smithi107.stdout:11438: delete oid 11 current snap is 0 2020-06-12T02:39:19.278 INFO:tasks.rados.rados.0.smithi107.stdout:11435: finishing write tid 6 to smithi10729407-10 2020-06-12T02:39:19.278 INFO:tasks.rados.rados.0.smithi107.stdout:11435: finishing write tid 7 to smithi10729407-10 2020-06-12T02:39:19.279 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 872 (ObjNum 11434 snap 0 seq_num 11434) dirty exists 2020-06-12T02:39:19.279 INFO:tasks.rados.rados.0.smithi107.stdout:11435: left oid 10 (ObjNum 11434 snap 0 seq_num 11434) 2020-06-12T02:39:19.279 INFO:tasks.rados.rados.0.smithi107.stdout:11435: done (1 left) 2020-06-12T02:39:19.279 INFO:tasks.rados.rados.0.smithi107.stdout:11438: done (0 left) 2020-06-12T02:39:19.279 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:39:19.280 INFO:tasks.rados.rados.0.smithi107.stdout:11439: seq_num 11438 ranges {1859584=770048,2629632=516096,3145728=442368,3588096=245760} 2020-06-12T02:39:19.280 INFO:tasks.rados.rados.0.smithi107.stdout:11439: writing smithi10729407-6 from 1859584 to 2629632 tid 1 2020-06-12T02:39:19.280 INFO:tasks.rados.rados.0.smithi107.stdout:11439: writing smithi10729407-6 from 2629632 to 3145728 tid 2 2020-06-12T02:39:19.280 INFO:tasks.rados.rados.0.smithi107.stdout:11439: writing smithi10729407-6 from 3145728 to 3588096 tid 3 2020-06-12T02:39:19.280 INFO:tasks.rados.rados.0.smithi107.stdout:11439: writing smithi10729407-6 from 3588096 to 3833856 tid 4 2020-06-12T02:39:19.280 INFO:tasks.rados.rados.0.smithi107.stdout:11440: delete oid 10 current snap is 0 2020-06-12T02:39:19.281 INFO:tasks.rados.rados.0.smithi107.stdout:11439: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:19.281 INFO:tasks.rados.rados.0.smithi107.stdout:11440: done (1 left) 2020-06-12T02:39:19.281 INFO:tasks.rados.rados.0.smithi107.stdout:11441: delete oid 1 current snap is 0 2020-06-12T02:39:19.281 INFO:tasks.rados.rados.0.smithi107.stdout:11439: finishing write tid 2 to smithi10729407-6 2020-06-12T02:39:19.281 INFO:tasks.rados.rados.0.smithi107.stdout:11439: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:19.282 INFO:tasks.rados.rados.0.smithi107.stdout:11439: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:19.282 INFO:tasks.rados.rados.0.smithi107.stdout:11439: finishing write tid 6 to smithi10729407-6 2020-06-12T02:39:19.282 INFO:tasks.rados.rados.0.smithi107.stdout:11441: done (1 left) 2020-06-12T02:39:19.282 INFO:tasks.rados.rados.0.smithi107.stdout:11442: delete oid 13 current snap is 0 2020-06-12T02:39:19.282 INFO:tasks.rados.rados.0.smithi107.stdout:11439: finishing write tid 7 to smithi10729407-6 2020-06-12T02:39:19.282 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 2077 (ObjNum 11438 snap 0 seq_num 11438) dirty exists 2020-06-12T02:39:19.283 INFO:tasks.rados.rados.0.smithi107.stdout:11439: left oid 6 (ObjNum 11438 snap 0 seq_num 11438) 2020-06-12T02:39:19.283 INFO:tasks.rados.rados.0.smithi107.stdout:11439: done (1 left) 2020-06-12T02:39:19.283 INFO:tasks.rados.rados.0.smithi107.stdout:11442: done (0 left) 2020-06-12T02:39:19.283 INFO:tasks.rados.rados.0.smithi107.stdout:11443: delete oid 3 current snap is 0 2020-06-12T02:39:19.283 INFO:tasks.rados.rados.0.smithi107.stdout:11443: done (0 left) 2020-06-12T02:39:19.284 INFO:tasks.rados.rados.0.smithi107.stdout:11444: delete oid 20 current snap is 0 2020-06-12T02:39:19.284 INFO:tasks.rados.rados.0.smithi107.stdout:11444: done (0 left) 2020-06-12T02:39:19.284 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:39:19.284 INFO:tasks.rados.rados.0.smithi107.stdout:11445: seq_num 11444 ranges {0=466944,466944=671744,1138688=434176,1572864=499712} 2020-06-12T02:39:19.284 INFO:tasks.rados.rados.0.smithi107.stdout:11445: writing smithi10729407-1 from 0 to 466944 tid 1 2020-06-12T02:39:19.284 INFO:tasks.rados.rados.0.smithi107.stdout:11445: writing smithi10729407-1 from 466944 to 1138688 tid 2 2020-06-12T02:39:19.285 INFO:tasks.rados.rados.0.smithi107.stdout:11445: writing smithi10729407-1 from 1138688 to 1572864 tid 3 2020-06-12T02:39:19.285 INFO:tasks.rados.rados.0.smithi107.stdout:11445: writing smithi10729407-1 from 1572864 to 2072576 tid 4 2020-06-12T02:39:19.285 INFO:tasks.rados.rados.0.smithi107.stdout:11446: delete oid 15 current snap is 0 2020-06-12T02:39:19.285 INFO:tasks.rados.rados.0.smithi107.stdout:11445: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:19.285 INFO:tasks.rados.rados.0.smithi107.stdout:11446: done (1 left) 2020-06-12T02:39:19.286 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:39:19.286 INFO:tasks.rados.rados.0.smithi107.stdout:11447: seq_num 11446 ranges {3833856=507904,4341760=434176,4775936=802816,5578752=368640} 2020-06-12T02:39:19.286 INFO:tasks.rados.rados.0.smithi107.stdout:11447: writing smithi10729407-6 from 3833856 to 4341760 tid 1 2020-06-12T02:39:19.286 INFO:tasks.rados.rados.0.smithi107.stdout:11447: writing smithi10729407-6 from 4341760 to 4775936 tid 2 2020-06-12T02:39:19.286 INFO:tasks.rados.rados.0.smithi107.stdout:11447: writing smithi10729407-6 from 4775936 to 5578752 tid 3 2020-06-12T02:39:19.286 INFO:tasks.rados.rados.0.smithi107.stdout:11447: writing smithi10729407-6 from 5578752 to 5947392 tid 4 2020-06-12T02:39:19.287 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:39:19.287 INFO:tasks.rados.rados.0.smithi107.stdout:11448: seq_num 11447 ranges {0=524288,524288=720896,1245184=606208,1851392=278528} 2020-06-12T02:39:19.287 INFO:tasks.rados.rados.0.smithi107.stdout:11448: writing smithi10729407-11 from 0 to 524288 tid 1 2020-06-12T02:39:19.287 INFO:tasks.rados.rados.0.smithi107.stdout:11448: writing smithi10729407-11 from 524288 to 1245184 tid 2 2020-06-12T02:39:19.287 INFO:tasks.rados.rados.0.smithi107.stdout:11448: writing smithi10729407-11 from 1245184 to 1851392 tid 3 2020-06-12T02:39:19.288 INFO:tasks.rados.rados.0.smithi107.stdout:11448: writing smithi10729407-11 from 1851392 to 2129920 tid 4 2020-06-12T02:39:19.288 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:39:19.288 INFO:tasks.rados.rados.0.smithi107.stdout:11449: seq_num 11448 ranges {0=540672,540672=606208,1146880=417792,1564672=581632} 2020-06-12T02:39:19.288 INFO:tasks.rados.rados.0.smithi107.stdout:11449: writing smithi10729407-14 from 0 to 540672 tid 1 2020-06-12T02:39:19.288 INFO:tasks.rados.rados.0.smithi107.stdout:11449: writing smithi10729407-14 from 540672 to 1146880 tid 2 2020-06-12T02:39:19.289 INFO:tasks.rados.rados.0.smithi107.stdout:11449: writing smithi10729407-14 from 1146880 to 1564672 tid 3 2020-06-12T02:39:19.289 INFO:tasks.rados.rados.0.smithi107.stdout:11449: writing smithi10729407-14 from 1564672 to 2146304 tid 4 2020-06-12T02:39:19.289 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:39:19.289 INFO:tasks.rados.rados.0.smithi107.stdout:11450: seq_num 11449 ranges {0=557056,557056=491520,1048576=573440,1622016=540672} 2020-06-12T02:39:19.289 INFO:tasks.rados.rados.0.smithi107.stdout:11450: writing smithi10729407-20 from 0 to 557056 tid 1 2020-06-12T02:39:19.289 INFO:tasks.rados.rados.0.smithi107.stdout:11450: writing smithi10729407-20 from 557056 to 1048576 tid 2 2020-06-12T02:39:19.290 INFO:tasks.rados.rados.0.smithi107.stdout:11450: writing smithi10729407-20 from 1048576 to 1622016 tid 3 2020-06-12T02:39:19.290 INFO:tasks.rados.rados.0.smithi107.stdout:11450: writing smithi10729407-20 from 1622016 to 2162688 tid 4 2020-06-12T02:39:19.290 INFO:tasks.rados.rados.0.smithi107.stdout:11451: delete oid 10 current snap is 0 2020-06-12T02:39:19.290 INFO:tasks.rados.rados.0.smithi107.stdout:11445: finishing write tid 2 to smithi10729407-1 2020-06-12T02:39:19.290 INFO:tasks.rados.rados.0.smithi107.stdout:11445: finishing write tid 3 to smithi10729407-1 2020-06-12T02:39:19.291 INFO:tasks.rados.rados.0.smithi107.stdout:11445: finishing write tid 4 to smithi10729407-1 2020-06-12T02:39:19.291 INFO:tasks.rados.rados.0.smithi107.stdout:11445: finishing write tid 6 to smithi10729407-1 2020-06-12T02:39:19.291 INFO:tasks.rados.rados.0.smithi107.stdout:11445: finishing write tid 7 to smithi10729407-1 2020-06-12T02:39:19.292 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 2083 (ObjNum 11444 snap 0 seq_num 11444) dirty exists 2020-06-12T02:39:19.292 INFO:tasks.rados.rados.0.smithi107.stdout:11445: left oid 1 (ObjNum 11444 snap 0 seq_num 11444) 2020-06-12T02:39:19.292 INFO:tasks.rados.rados.0.smithi107.stdout:11447: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:19.293 INFO:tasks.rados.rados.0.smithi107.stdout:11447: finishing write tid 2 to smithi10729407-6 2020-06-12T02:39:19.293 INFO:tasks.rados.rados.0.smithi107.stdout:11447: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:19.293 INFO:tasks.rados.rados.0.smithi107.stdout:11447: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:19.293 INFO:tasks.rados.rados.0.smithi107.stdout:11447: finishing write tid 6 to smithi10729407-6 2020-06-12T02:39:19.293 INFO:tasks.rados.rados.0.smithi107.stdout:11447: finishing write tid 7 to smithi10729407-6 2020-06-12T02:39:19.294 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 2088 (ObjNum 11446 snap 0 seq_num 11446) dirty exists 2020-06-12T02:39:19.294 INFO:tasks.rados.rados.0.smithi107.stdout:11447: left oid 6 (ObjNum 11446 snap 0 seq_num 11446) 2020-06-12T02:39:19.294 INFO:tasks.rados.rados.0.smithi107.stdout:11448: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:19.294 INFO:tasks.rados.rados.0.smithi107.stdout:11448: finishing write tid 2 to smithi10729407-11 2020-06-12T02:39:19.294 INFO:tasks.rados.rados.0.smithi107.stdout:11448: finishing write tid 3 to smithi10729407-11 2020-06-12T02:39:19.295 INFO:tasks.rados.rados.0.smithi107.stdout:11448: finishing write tid 4 to smithi10729407-11 2020-06-12T02:39:19.295 INFO:tasks.rados.rados.0.smithi107.stdout:11448: finishing write tid 6 to smithi10729407-11 2020-06-12T02:39:19.295 INFO:tasks.rados.rados.0.smithi107.stdout:11448: finishing write tid 7 to smithi10729407-11 2020-06-12T02:39:19.295 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1807 (ObjNum 11447 snap 0 seq_num 11447) dirty exists 2020-06-12T02:39:19.296 INFO:tasks.rados.rados.0.smithi107.stdout:11448: left oid 11 (ObjNum 11447 snap 0 seq_num 11447) 2020-06-12T02:39:19.296 INFO:tasks.rados.rados.0.smithi107.stdout:11449: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:19.296 INFO:tasks.rados.rados.0.smithi107.stdout:11449: finishing write tid 2 to smithi10729407-14 2020-06-12T02:39:19.296 INFO:tasks.rados.rados.0.smithi107.stdout:11449: finishing write tid 3 to smithi10729407-14 2020-06-12T02:39:19.296 INFO:tasks.rados.rados.0.smithi107.stdout:11449: finishing write tid 4 to smithi10729407-14 2020-06-12T02:39:19.296 INFO:tasks.rados.rados.0.smithi107.stdout:11450: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:19.297 INFO:tasks.rados.rados.0.smithi107.stdout:11449: finishing write tid 6 to smithi10729407-14 2020-06-12T02:39:19.297 INFO:tasks.rados.rados.0.smithi107.stdout:11449: finishing write tid 7 to smithi10729407-14 2020-06-12T02:39:19.297 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1783 (ObjNum 11448 snap 0 seq_num 11448) dirty exists 2020-06-12T02:39:19.297 INFO:tasks.rados.rados.0.smithi107.stdout:11449: left oid 14 (ObjNum 11448 snap 0 seq_num 11448) 2020-06-12T02:39:19.297 INFO:tasks.rados.rados.0.smithi107.stdout:11445: done (5 left) 2020-06-12T02:39:19.298 INFO:tasks.rados.rados.0.smithi107.stdout:11447: done (4 left) 2020-06-12T02:39:19.298 INFO:tasks.rados.rados.0.smithi107.stdout:11448: done (3 left) 2020-06-12T02:39:19.298 INFO:tasks.rados.rados.0.smithi107.stdout:11449: done (2 left) 2020-06-12T02:39:19.298 INFO:tasks.rados.rados.0.smithi107.stdout:11451: done (1 left) 2020-06-12T02:39:19.298 INFO:tasks.rados.rados.0.smithi107.stdout:11452: delete oid 2 current snap is 0 2020-06-12T02:39:19.298 INFO:tasks.rados.rados.0.smithi107.stdout:11452: done (1 left) 2020-06-12T02:39:19.299 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:39:19.299 INFO:tasks.rados.rados.0.smithi107.stdout:11453: seq_num 11452 ranges {0=606208,606208=589824,1196032=794624,1990656=221184} 2020-06-12T02:39:19.299 INFO:tasks.rados.rados.0.smithi107.stdout:11453: writing smithi10729407-2 from 0 to 606208 tid 1 2020-06-12T02:39:19.299 INFO:tasks.rados.rados.0.smithi107.stdout:11453: writing smithi10729407-2 from 606208 to 1196032 tid 2 2020-06-12T02:39:19.299 INFO:tasks.rados.rados.0.smithi107.stdout:11453: writing smithi10729407-2 from 1196032 to 1990656 tid 3 2020-06-12T02:39:19.300 INFO:tasks.rados.rados.0.smithi107.stdout:11453: writing smithi10729407-2 from 1990656 to 2211840 tid 4 2020-06-12T02:39:19.300 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:39:19.300 INFO:tasks.rados.rados.0.smithi107.stdout:11454: seq_num 11453 ranges {0=622592,622592=475136,1097728=557056,1654784=573440} 2020-06-12T02:39:19.300 INFO:tasks.rados.rados.0.smithi107.stdout:11454: writing smithi10729407-3 from 0 to 622592 tid 1 2020-06-12T02:39:19.300 INFO:tasks.rados.rados.0.smithi107.stdout:11454: writing smithi10729407-3 from 622592 to 1097728 tid 2 2020-06-12T02:39:19.300 INFO:tasks.rados.rados.0.smithi107.stdout:11454: writing smithi10729407-3 from 1097728 to 1654784 tid 3 2020-06-12T02:39:19.301 INFO:tasks.rados.rados.0.smithi107.stdout:11454: writing smithi10729407-3 from 1654784 to 2228224 tid 4 2020-06-12T02:39:19.301 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:39:19.301 INFO:tasks.rados.rados.0.smithi107.stdout:11455: seq_num 11454 ranges {0=638976,638976=761856,1400832=761856,2162688=81920} 2020-06-12T02:39:19.301 INFO:tasks.rados.rados.0.smithi107.stdout:11455: writing smithi10729407-4 from 0 to 638976 tid 1 2020-06-12T02:39:19.301 INFO:tasks.rados.rados.0.smithi107.stdout:11455: writing smithi10729407-4 from 638976 to 1400832 tid 2 2020-06-12T02:39:19.302 INFO:tasks.rados.rados.0.smithi107.stdout:11455: writing smithi10729407-4 from 1400832 to 2162688 tid 3 2020-06-12T02:39:19.302 INFO:tasks.rados.rados.0.smithi107.stdout:11455: writing smithi10729407-4 from 2162688 to 2244608 tid 4 2020-06-12T02:39:19.302 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:39:19.302 INFO:tasks.rados.rados.0.smithi107.stdout:11456: seq_num 11455 ranges {0=655360,655360=647168,1302528=573440,1875968=385024} 2020-06-12T02:39:19.302 INFO:tasks.rados.rados.0.smithi107.stdout:11456: writing smithi10729407-18 from 0 to 655360 tid 1 2020-06-12T02:39:19.303 INFO:tasks.rados.rados.0.smithi107.stdout:11456: writing smithi10729407-18 from 655360 to 1302528 tid 2 2020-06-12T02:39:19.303 INFO:tasks.rados.rados.0.smithi107.stdout:11456: writing smithi10729407-18 from 1302528 to 1875968 tid 3 2020-06-12T02:39:19.303 INFO:tasks.rados.rados.0.smithi107.stdout:11456: writing smithi10729407-18 from 1875968 to 2260992 tid 4 2020-06-12T02:39:19.303 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:39:19.303 INFO:tasks.rados.rados.0.smithi107.stdout:11457: seq_num 11456 ranges {1572864=671744,2244608=532480,2777088=778240,3555328=294912} 2020-06-12T02:39:19.303 INFO:tasks.rados.rados.0.smithi107.stdout:11457: writing smithi10729407-8 from 1572864 to 2244608 tid 1 2020-06-12T02:39:19.304 INFO:tasks.rados.rados.0.smithi107.stdout:11457: writing smithi10729407-8 from 2244608 to 2777088 tid 2 2020-06-12T02:39:19.304 INFO:tasks.rados.rados.0.smithi107.stdout:11457: writing smithi10729407-8 from 2777088 to 3555328 tid 3 2020-06-12T02:39:19.304 INFO:tasks.rados.rados.0.smithi107.stdout:11457: writing smithi10729407-8 from 3555328 to 3850240 tid 4 2020-06-12T02:39:19.304 INFO:tasks.rados.rados.0.smithi107.stdout:11458: delete oid 13 current snap is 0 2020-06-12T02:39:19.304 INFO:tasks.rados.rados.0.smithi107.stdout:11450: finishing write tid 2 to smithi10729407-20 2020-06-12T02:39:19.305 INFO:tasks.rados.rados.0.smithi107.stdout:11450: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:19.305 INFO:tasks.rados.rados.0.smithi107.stdout:11450: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:19.305 INFO:tasks.rados.rados.0.smithi107.stdout:11450: finishing write tid 6 to smithi10729407-20 2020-06-12T02:39:19.305 INFO:tasks.rados.rados.0.smithi107.stdout:11450: finishing write tid 7 to smithi10729407-20 2020-06-12T02:39:19.305 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 894 (ObjNum 11449 snap 0 seq_num 11449) dirty exists 2020-06-12T02:39:19.305 INFO:tasks.rados.rados.0.smithi107.stdout:11450: left oid 20 (ObjNum 11449 snap 0 seq_num 11449) 2020-06-12T02:39:19.306 INFO:tasks.rados.rados.0.smithi107.stdout:11453: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:19.306 INFO:tasks.rados.rados.0.smithi107.stdout:11453: finishing write tid 2 to smithi10729407-2 2020-06-12T02:39:19.306 INFO:tasks.rados.rados.0.smithi107.stdout:11453: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:19.306 INFO:tasks.rados.rados.0.smithi107.stdout:11453: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:19.306 INFO:tasks.rados.rados.0.smithi107.stdout:11453: finishing write tid 6 to smithi10729407-2 2020-06-12T02:39:19.307 INFO:tasks.rados.rados.0.smithi107.stdout:11453: finishing write tid 7 to smithi10729407-2 2020-06-12T02:39:19.307 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 1082 (ObjNum 11452 snap 0 seq_num 11452) dirty exists 2020-06-12T02:39:19.307 INFO:tasks.rados.rados.0.smithi107.stdout:11453: left oid 2 (ObjNum 11452 snap 0 seq_num 11452) 2020-06-12T02:39:19.307 INFO:tasks.rados.rados.0.smithi107.stdout:11454: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:19.307 INFO:tasks.rados.rados.0.smithi107.stdout:11454: finishing write tid 2 to smithi10729407-3 2020-06-12T02:39:19.307 INFO:tasks.rados.rados.0.smithi107.stdout:11454: finishing write tid 3 to smithi10729407-3 2020-06-12T02:39:19.308 INFO:tasks.rados.rados.0.smithi107.stdout:11454: finishing write tid 4 to smithi10729407-3 2020-06-12T02:39:19.308 INFO:tasks.rados.rados.0.smithi107.stdout:11454: finishing write tid 6 to smithi10729407-3 2020-06-12T02:39:19.308 INFO:tasks.rados.rados.0.smithi107.stdout:11455: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:19.308 INFO:tasks.rados.rados.0.smithi107.stdout:11454: finishing write tid 7 to smithi10729407-3 2020-06-12T02:39:19.308 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 2014 (ObjNum 11453 snap 0 seq_num 11453) dirty exists 2020-06-12T02:39:19.309 INFO:tasks.rados.rados.0.smithi107.stdout:11454: left oid 3 (ObjNum 11453 snap 0 seq_num 11453) 2020-06-12T02:39:19.309 INFO:tasks.rados.rados.0.smithi107.stdout:11455: finishing write tid 2 to smithi10729407-4 2020-06-12T02:39:19.309 INFO:tasks.rados.rados.0.smithi107.stdout:11455: finishing write tid 3 to smithi10729407-4 2020-06-12T02:39:19.309 INFO:tasks.rados.rados.0.smithi107.stdout:11455: finishing write tid 4 to smithi10729407-4 2020-06-12T02:39:19.309 INFO:tasks.rados.rados.0.smithi107.stdout:11455: finishing write tid 6 to smithi10729407-4 2020-06-12T02:39:19.309 INFO:tasks.rados.rados.0.smithi107.stdout:11455: finishing write tid 7 to smithi10729407-4 2020-06-12T02:39:19.310 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 2136 (ObjNum 11454 snap 0 seq_num 11454) dirty exists 2020-06-12T02:39:19.310 INFO:tasks.rados.rados.0.smithi107.stdout:11455: left oid 4 (ObjNum 11454 snap 0 seq_num 11454) 2020-06-12T02:39:19.310 INFO:tasks.rados.rados.0.smithi107.stdout:11456: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:19.310 INFO:tasks.rados.rados.0.smithi107.stdout:11456: finishing write tid 2 to smithi10729407-18 2020-06-12T02:39:19.310 INFO:tasks.rados.rados.0.smithi107.stdout:11456: finishing write tid 3 to smithi10729407-18 2020-06-12T02:39:19.311 INFO:tasks.rados.rados.0.smithi107.stdout:11456: finishing write tid 4 to smithi10729407-18 2020-06-12T02:39:19.311 INFO:tasks.rados.rados.0.smithi107.stdout:11456: finishing write tid 6 to smithi10729407-18 2020-06-12T02:39:19.311 INFO:tasks.rados.rados.0.smithi107.stdout:11456: finishing write tid 7 to smithi10729407-18 2020-06-12T02:39:19.311 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1918 (ObjNum 11455 snap 0 seq_num 11455) dirty exists 2020-06-12T02:39:19.311 INFO:tasks.rados.rados.0.smithi107.stdout:11456: left oid 18 (ObjNum 11455 snap 0 seq_num 11455) 2020-06-12T02:39:19.311 INFO:tasks.rados.rados.0.smithi107.stdout:11457: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:19.312 INFO:tasks.rados.rados.0.smithi107.stdout:11450: done (6 left) 2020-06-12T02:39:19.312 INFO:tasks.rados.rados.0.smithi107.stdout:11453: done (5 left) 2020-06-12T02:39:19.312 INFO:tasks.rados.rados.0.smithi107.stdout:11454: done (4 left) 2020-06-12T02:39:19.312 INFO:tasks.rados.rados.0.smithi107.stdout:11455: done (3 left) 2020-06-12T02:39:19.312 INFO:tasks.rados.rados.0.smithi107.stdout:11456: done (2 left) 2020-06-12T02:39:19.313 INFO:tasks.rados.rados.0.smithi107.stdout:11458: done (1 left) 2020-06-12T02:39:19.313 INFO:tasks.rados.rados.0.smithi107.stdout:11459: delete oid 11 current snap is 0 2020-06-12T02:39:19.313 INFO:tasks.rados.rados.0.smithi107.stdout:11459: done (1 left) 2020-06-12T02:39:19.313 INFO:tasks.rados.rados.0.smithi107.stdout:11460: delete oid 10 current snap is 0 2020-06-12T02:39:19.313 INFO:tasks.rados.rados.0.smithi107.stdout:11460: done (1 left) 2020-06-12T02:39:19.313 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:39:19.314 INFO:tasks.rados.rados.0.smithi107.stdout:11461: seq_num 11460 ranges {0=737280,737280=516096,1253376=761856,2015232=327680} 2020-06-12T02:39:19.314 INFO:tasks.rados.rados.0.smithi107.stdout:11461: writing smithi10729407-15 from 0 to 737280 tid 1 2020-06-12T02:39:19.314 INFO:tasks.rados.rados.0.smithi107.stdout:11461: writing smithi10729407-15 from 737280 to 1253376 tid 2 2020-06-12T02:39:19.314 INFO:tasks.rados.rados.0.smithi107.stdout:11461: writing smithi10729407-15 from 1253376 to 2015232 tid 3 2020-06-12T02:39:19.320 INFO:tasks.rados.rados.0.smithi107.stdout:11461: writing smithi10729407-15 from 2015232 to 2342912 tid 4 2020-06-12T02:39:19.320 INFO:tasks.rados.rados.0.smithi107.stdout:11462: delete oid 7 current snap is 0 2020-06-12T02:39:19.320 INFO:tasks.rados.rados.0.smithi107.stdout:11457: finishing write tid 2 to smithi10729407-8 2020-06-12T02:39:19.320 INFO:tasks.rados.rados.0.smithi107.stdout:11457: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:19.320 INFO:tasks.rados.rados.0.smithi107.stdout:11457: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:19.321 INFO:tasks.rados.rados.0.smithi107.stdout:11457: finishing write tid 6 to smithi10729407-8 2020-06-12T02:39:19.321 INFO:tasks.rados.rados.0.smithi107.stdout:11457: finishing write tid 7 to smithi10729407-8 2020-06-12T02:39:19.321 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 2150 (ObjNum 11456 snap 0 seq_num 11456) dirty exists 2020-06-12T02:39:19.321 INFO:tasks.rados.rados.0.smithi107.stdout:11457: left oid 8 (ObjNum 11456 snap 0 seq_num 11456) 2020-06-12T02:39:19.322 INFO:tasks.rados.rados.0.smithi107.stdout:11461: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:19.322 INFO:tasks.rados.rados.0.smithi107.stdout:11457: done (2 left) 2020-06-12T02:39:19.322 INFO:tasks.rados.rados.0.smithi107.stdout:11462: done (1 left) 2020-06-12T02:39:19.322 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:39:19.322 INFO:tasks.rados.rados.0.smithi107.stdout:11463: seq_num 11462 ranges {2072576=770048,2842624=688128,3530752=720896,4251648=196608} 2020-06-12T02:39:19.323 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:39:19.259+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:39:19.326 INFO:teuthology.orchestra.run.smithi188.stdout: marking collection for removal 2020-06-12T02:39:19.326 INFO:teuthology.orchestra.run.smithi188.stdout:setting '_remove' omap key 2020-06-12T02:39:19.326 INFO:teuthology.orchestra.run.smithi188.stdout:finish_remove_pgs 2.5_head removing 2.5 2020-06-12T02:39:19.327 INFO:teuthology.orchestra.run.smithi188.stdout:Remove successful 2020-06-12T02:39:19.327 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-06-12T02:39:19.327 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-06-12T02:39:19.328 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-06-12T02:39:19.328 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-06-12T02:39:19.328 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-06-12T02:39:19.328 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-06-12T02:39:19.328 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-06-12T02:39:19.328 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.7 2020-06-12T02:39:19.329 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: problem getting command descriptions from osd.7 2020-06-12T02:39:19.329 INFO:tasks.rados.rados.0.smithi107.stdout:11463: writing smithi10729407-1 from 2072576 to 2842624 tid 1 2020-06-12T02:39:19.329 INFO:tasks.rados.rados.0.smithi107.stdout:11463: writing smithi10729407-1 from 2842624 to 3530752 tid 2 2020-06-12T02:39:19.329 INFO:tasks.rados.rados.0.smithi107.stdout:11463: writing smithi10729407-1 from 3530752 to 4251648 tid 3 2020-06-12T02:39:19.329 INFO:tasks.rados.rados.0.smithi107.stdout:11463: writing smithi10729407-1 from 4251648 to 4448256 tid 4 2020-06-12T02:39:19.330 INFO:tasks.rados.rados.0.smithi107.stdout:11464: delete oid 6 current snap is 0 2020-06-12T02:39:19.330 INFO:tasks.rados.rados.0.smithi107.stdout:11461: finishing write tid 2 to smithi10729407-15 2020-06-12T02:39:19.330 INFO:tasks.rados.rados.0.smithi107.stdout:11461: finishing write tid 3 to smithi10729407-15 2020-06-12T02:39:19.330 INFO:tasks.rados.rados.0.smithi107.stdout:11461: finishing write tid 4 to smithi10729407-15 2020-06-12T02:39:19.330 INFO:tasks.rados.rados.0.smithi107.stdout:11461: finishing write tid 6 to smithi10729407-15 2020-06-12T02:39:19.331 INFO:tasks.rados.rados.0.smithi107.stdout:11461: finishing write tid 7 to smithi10729407-15 2020-06-12T02:39:19.331 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 2155 (ObjNum 11460 snap 0 seq_num 11460) dirty exists 2020-06-12T02:39:19.331 INFO:tasks.rados.rados.0.smithi107.stdout:11461: left oid 15 (ObjNum 11460 snap 0 seq_num 11460) 2020-06-12T02:39:19.331 INFO:tasks.rados.rados.0.smithi107.stdout:11463: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:19.331 INFO:tasks.rados.rados.0.smithi107.stdout:11463: finishing write tid 2 to smithi10729407-1 2020-06-12T02:39:19.331 INFO:tasks.rados.rados.0.smithi107.stdout:11463: finishing write tid 3 to smithi10729407-1 2020-06-12T02:39:19.332 INFO:tasks.rados.rados.0.smithi107.stdout:11463: finishing write tid 4 to smithi10729407-1 2020-06-12T02:39:19.332 INFO:tasks.rados.rados.0.smithi107.stdout:11463: finishing write tid 6 to smithi10729407-1 2020-06-12T02:39:19.332 INFO:tasks.rados.rados.0.smithi107.stdout:11463: finishing write tid 7 to smithi10729407-1 2020-06-12T02:39:19.332 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 2093 (ObjNum 11462 snap 0 seq_num 11462) dirty exists 2020-06-12T02:39:19.332 INFO:tasks.rados.rados.0.smithi107.stdout:11463: left oid 1 (ObjNum 11462 snap 0 seq_num 11462) 2020-06-12T02:39:19.333 INFO:tasks.rados.rados.0.smithi107.stdout:11461: done (2 left) 2020-06-12T02:39:19.333 INFO:tasks.rados.rados.0.smithi107.stdout:11463: done (1 left) 2020-06-12T02:39:19.333 INFO:tasks.rados.rados.0.smithi107.stdout:11464: done (0 left) 2020-06-12T02:39:19.333 INFO:tasks.rados.rados.0.smithi107.stdout:11465: delete oid 19 current snap is 0 2020-06-12T02:39:19.333 INFO:tasks.rados.rados.0.smithi107.stdout:11465: done (0 left) 2020-06-12T02:39:19.333 INFO:tasks.rados.rados.0.smithi107.stdout:11466: delete oid 19 current snap is 0 2020-06-12T02:39:19.334 INFO:tasks.rados.rados.0.smithi107.stdout:11466: done (0 left) 2020-06-12T02:39:19.334 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:39:19.334 INFO:tasks.rados.rados.0.smithi107.stdout:11467: seq_num 11466 ranges {3850240=40960} 2020-06-12T02:39:19.334 INFO:tasks.rados.rados.0.smithi107.stdout:11467: writing smithi10729407-8 from 3850240 to 3891200 tid 1 2020-06-12T02:39:19.334 INFO:tasks.rados.rados.0.smithi107.stdout:11468: delete oid 15 current snap is 0 2020-06-12T02:39:19.335 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-06-12T02:39:19.335 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-06-12T02:39:19.336 INFO:tasks.rados.rados.0.smithi107.stdout:11467: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:19.337 INFO:tasks.rados.rados.0.smithi107.stdout:11468: done (1 left) 2020-06-12T02:39:19.337 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:39:19.337 INFO:tasks.rados.rados.0.smithi107.stdout:11469: seq_num 11468 ranges {0=73728} 2020-06-12T02:39:19.337 INFO:tasks.rados.rados.0.smithi107.stdout:11469: writing smithi10729407-19 from 0 to 73728 tid 1 2020-06-12T02:39:19.338 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:39:19.338 INFO:tasks.rados.rados.0.smithi107.stdout:11470: seq_num 11469 ranges {2244608=90112} 2020-06-12T02:39:19.343 INFO:tasks.rados.rados.0.smithi107.stdout:11470: writing smithi10729407-4 from 2244608 to 2334720 tid 1 2020-06-12T02:39:19.343 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:39:19.343 INFO:tasks.rados.rados.0.smithi107.stdout:11471: seq_num 11470 ranges {1769472=106496} 2020-06-12T02:39:19.354 INFO:tasks.rados.rados.0.smithi107.stdout:11471: writing smithi10729407-12 from 1769472 to 1875968 tid 1 2020-06-12T02:39:19.355 INFO:tasks.rados.rados.0.smithi107.stdout:11472: delete oid 10 current snap is 0 2020-06-12T02:39:19.355 INFO:tasks.rados.rados.0.smithi107.stdout:11467: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:19.355 INFO:tasks.rados.rados.0.smithi107.stdout:11467: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:19.355 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 2158 (ObjNum 11466 snap 0 seq_num 11466) dirty exists 2020-06-12T02:39:19.355 INFO:tasks.rados.rados.0.smithi107.stdout:11467: left oid 8 (ObjNum 11466 snap 0 seq_num 11466) 2020-06-12T02:39:19.356 INFO:tasks.rados.rados.0.smithi107.stdout:11469: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:19.356 INFO:tasks.rados.rados.0.smithi107.stdout:11469: finishing write tid 3 to smithi10729407-19 2020-06-12T02:39:19.356 INFO:tasks.rados.rados.0.smithi107.stdout:11469: finishing write tid 4 to smithi10729407-19 2020-06-12T02:39:19.356 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 2136 (ObjNum 11468 snap 0 seq_num 11468) dirty exists 2020-06-12T02:39:19.356 INFO:tasks.rados.rados.0.smithi107.stdout:11469: left oid 19 (ObjNum 11468 snap 0 seq_num 11468) 2020-06-12T02:39:19.356 INFO:tasks.rados.rados.0.smithi107.stdout:11470: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:19.357 INFO:tasks.rados.rados.0.smithi107.stdout:11470: finishing write tid 3 to smithi10729407-4 2020-06-12T02:39:19.357 INFO:tasks.rados.rados.0.smithi107.stdout:11470: finishing write tid 4 to smithi10729407-4 2020-06-12T02:39:19.357 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 2138 (ObjNum 11469 snap 0 seq_num 11469) dirty exists 2020-06-12T02:39:19.357 INFO:tasks.rados.rados.0.smithi107.stdout:11470: left oid 4 (ObjNum 11469 snap 0 seq_num 11469) 2020-06-12T02:39:19.358 INFO:tasks.rados.rados.0.smithi107.stdout:11467: done (4 left) 2020-06-12T02:39:19.358 INFO:tasks.rados.rados.0.smithi107.stdout:11469: done (3 left) 2020-06-12T02:39:19.358 INFO:tasks.rados.rados.0.smithi107.stdout:11470: done (2 left) 2020-06-12T02:39:19.359 INFO:tasks.rados.rados.0.smithi107.stdout:11472: done (1 left) 2020-06-12T02:39:19.359 INFO:tasks.rados.rados.0.smithi107.stdout:11473: delete oid 3 current snap is 0 2020-06-12T02:39:19.359 INFO:tasks.rados.rados.0.smithi107.stdout:11471: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:19.359 INFO:tasks.rados.rados.0.smithi107.stdout:11473: done (1 left) 2020-06-12T02:39:19.360 INFO:tasks.rados.rados.0.smithi107.stdout:11474: delete oid 14 current snap is 0 2020-06-12T02:39:19.362 INFO:tasks.rados.rados.0.smithi107.stdout:11471: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:19.362 INFO:tasks.rados.rados.0.smithi107.stdout:11474: done (1 left) 2020-06-12T02:39:19.363 INFO:tasks.rados.rados.0.smithi107.stdout:11475: delete oid 14 current snap is 0 2020-06-12T02:39:19.364 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:39:19.360+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:39:19.365 INFO:tasks.rados.rados.0.smithi107.stdout:11471: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:19.365 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 2138 (ObjNum 11470 snap 0 seq_num 11470) dirty exists 2020-06-12T02:39:19.366 INFO:tasks.rados.rados.0.smithi107.stdout:11471: left oid 12 (ObjNum 11470 snap 0 seq_num 11470) 2020-06-12T02:39:19.367 INFO:tasks.rados.rados.0.smithi107.stdout:11471: done (1 left) 2020-06-12T02:39:19.368 INFO:tasks.rados.rados.0.smithi107.stdout:11475: done (0 left) 2020-06-12T02:39:19.368 INFO:tasks.rados.rados.0.smithi107.stdout:11476: delete oid 4 current snap is 0 2020-06-12T02:39:19.369 INFO:tasks.rados.rados.0.smithi107.stdout:11476: done (0 left) 2020-06-12T02:39:19.369 INFO:tasks.rados.rados.0.smithi107.stdout:11477: delete oid 8 current snap is 0 2020-06-12T02:39:19.372 INFO:tasks.rados.rados.0.smithi107.stdout:11477: done (0 left) 2020-06-12T02:39:19.372 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:39:19.372 INFO:tasks.rados.rados.0.smithi107.stdout:11478: seq_num 11477 ranges {73728=221184} 2020-06-12T02:39:19.374 INFO:tasks.rados.rados.0.smithi107.stdout:11478: writing smithi10729407-19 from 73728 to 294912 tid 1 2020-06-12T02:39:19.374 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:39:19.374 INFO:tasks.rados.rados.0.smithi107.stdout:11479: seq_num 11478 ranges {0=237568} 2020-06-12T02:39:19.375 INFO:tasks.rados.rados.0.smithi107.stdout:11479: writing smithi10729407-10 from 0 to 237568 tid 1 2020-06-12T02:39:19.376 INFO:tasks.rados.rados.0.smithi107.stdout:11480: delete oid 15 current snap is 0 2020-06-12T02:39:19.378 INFO:tasks.rados.rados.0.smithi107.stdout:11478: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:19.379 INFO:tasks.rados.rados.0.smithi107.stdout:11479: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:19.381 INFO:tasks.rados.rados.0.smithi107.stdout:11480: done (2 left) 2020-06-12T02:39:19.381 INFO:tasks.rados.rados.0.smithi107.stdout:11481: delete oid 15 current snap is 0 2020-06-12T02:39:19.382 INFO:tasks.rados.rados.0.smithi107.stdout:11479: finishing write tid 3 to smithi10729407-10 2020-06-12T02:39:19.382 INFO:tasks.rados.rados.0.smithi107.stdout:11478: finishing write tid 3 to smithi10729407-19 2020-06-12T02:39:19.384 INFO:tasks.rados.rados.0.smithi107.stdout:11479: finishing write tid 4 to smithi10729407-10 2020-06-12T02:39:19.384 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 878 (ObjNum 11478 snap 0 seq_num 11478) dirty exists 2020-06-12T02:39:19.385 INFO:tasks.rados.rados.0.smithi107.stdout:11479: left oid 10 (ObjNum 11478 snap 0 seq_num 11478) 2020-06-12T02:39:19.385 INFO:tasks.rados.rados.0.smithi107.stdout:11478: finishing write tid 4 to smithi10729407-19 2020-06-12T02:39:19.386 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 2140 (ObjNum 11477 snap 0 seq_num 11477) dirty exists 2020-06-12T02:39:19.386 INFO:tasks.rados.rados.0.smithi107.stdout:11478: left oid 19 (ObjNum 11477 snap 0 seq_num 11477) 2020-06-12T02:39:19.386 INFO:tasks.rados.rados.0.smithi107.stdout:11478: done (2 left) 2020-06-12T02:39:19.386 INFO:tasks.rados.rados.0.smithi107.stdout:11479: done (1 left) 2020-06-12T02:39:19.386 INFO:tasks.rados.rados.0.smithi107.stdout:11481: done (0 left) 2020-06-12T02:39:19.387 INFO:tasks.rados.rados.0.smithi107.stdout:11482: delete oid 12 current snap is 0 2020-06-12T02:39:19.387 INFO:tasks.rados.rados.0.smithi107.stdout:11482: done (0 left) 2020-06-12T02:39:19.387 INFO:tasks.rados.rados.0.smithi107.stdout:11483: delete oid 2 current snap is 0 2020-06-12T02:39:19.391 INFO:tasks.rados.rados.0.smithi107.stdout:11483: done (0 left) 2020-06-12T02:39:19.392 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:39:19.392 INFO:tasks.rados.rados.0.smithi107.stdout:11484: seq_num 11483 ranges {0=319488} 2020-06-12T02:39:19.394 INFO:tasks.rados.rados.0.smithi107.stdout:11484: writing smithi10729407-4 from 0 to 319488 tid 1 2020-06-12T02:39:19.394 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:39:19.394 INFO:tasks.rados.rados.0.smithi107.stdout:11485: seq_num 11484 ranges {0=335872} 2020-06-12T02:39:19.396 INFO:tasks.rados.rados.0.smithi107.stdout:11485: writing smithi10729407-5 from 0 to 335872 tid 1 2020-06-12T02:39:19.396 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:39:19.396 INFO:tasks.rados.rados.0.smithi107.stdout:11486: seq_num 11485 ranges {2260992=360448} 2020-06-12T02:39:19.430 INFO:tasks.rados.rados.0.smithi107.stdout:11486: writing smithi10729407-18 from 2260992 to 2621440 tid 1 2020-06-12T02:39:19.431 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:39:19.431 INFO:tasks.rados.rados.0.smithi107.stdout:11487: seq_num 11486 ranges {0=376832} 2020-06-12T02:39:19.431 INFO:tasks.rados.rados.0.smithi107.stdout:11487: writing smithi10729407-14 from 0 to 376832 tid 1 2020-06-12T02:39:19.431 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:39:19.431 INFO:tasks.rados.rados.0.smithi107.stdout:11488: seq_num 11487 ranges {0=393216} 2020-06-12T02:39:19.432 INFO:tasks.rados.rados.0.smithi107.stdout:11488: writing smithi10729407-8 from 0 to 393216 tid 1 2020-06-12T02:39:19.432 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:39:19.432 INFO:tasks.rados.rados.0.smithi107.stdout:11489: seq_num 11488 ranges {0=409600} 2020-06-12T02:39:19.432 INFO:tasks.rados.rados.0.smithi107.stdout:11489: writing smithi10729407-7 from 0 to 409600 tid 1 2020-06-12T02:39:19.433 INFO:tasks.rados.rados.0.smithi107.stdout:11490: delete oid 12 current snap is 0 2020-06-12T02:39:19.433 INFO:tasks.rados.rados.0.smithi107.stdout:11484: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:19.433 INFO:tasks.rados.rados.0.smithi107.stdout:11484: finishing write tid 3 to smithi10729407-4 2020-06-12T02:39:19.433 INFO:tasks.rados.rados.0.smithi107.stdout:11485: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:19.433 INFO:tasks.rados.rados.0.smithi107.stdout:11484: finishing write tid 4 to smithi10729407-4 2020-06-12T02:39:19.434 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 2141 (ObjNum 11483 snap 0 seq_num 11483) dirty exists 2020-06-12T02:39:19.434 INFO:tasks.rados.rados.0.smithi107.stdout:11484: left oid 4 (ObjNum 11483 snap 0 seq_num 11483) 2020-06-12T02:39:19.434 INFO:tasks.rados.rados.0.smithi107.stdout:11485: finishing write tid 3 to smithi10729407-5 2020-06-12T02:39:19.435 INFO:tasks.rados.rados.0.smithi107.stdout:11485: finishing write tid 4 to smithi10729407-5 2020-06-12T02:39:19.435 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 1090 (ObjNum 11484 snap 0 seq_num 11484) dirty exists 2020-06-12T02:39:19.435 INFO:tasks.rados.rados.0.smithi107.stdout:11485: left oid 5 (ObjNum 11484 snap 0 seq_num 11484) 2020-06-12T02:39:19.435 INFO:tasks.rados.rados.0.smithi107.stdout:11486: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:19.436 INFO:tasks.rados.rados.0.smithi107.stdout:11486: finishing write tid 3 to smithi10729407-18 2020-06-12T02:39:19.436 INFO:tasks.rados.rados.0.smithi107.stdout:11486: finishing write tid 4 to smithi10729407-18 2020-06-12T02:39:19.436 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1920 (ObjNum 11485 snap 0 seq_num 11485) dirty exists 2020-06-12T02:39:19.436 INFO:tasks.rados.rados.0.smithi107.stdout:11486: left oid 18 (ObjNum 11485 snap 0 seq_num 11485) 2020-06-12T02:39:19.436 INFO:tasks.rados.rados.0.smithi107.stdout:11487: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:19.437 INFO:tasks.rados.rados.0.smithi107.stdout:11487: finishing write tid 3 to smithi10729407-14 2020-06-12T02:39:19.437 INFO:tasks.rados.rados.0.smithi107.stdout:11488: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:19.437 INFO:tasks.rados.rados.0.smithi107.stdout:11484: done (6 left) 2020-06-12T02:39:19.437 INFO:tasks.rados.rados.0.smithi107.stdout:11485: done (5 left) 2020-06-12T02:39:19.437 INFO:tasks.rados.rados.0.smithi107.stdout:11486: done (4 left) 2020-06-12T02:39:19.438 INFO:tasks.rados.rados.0.smithi107.stdout:11490: done (3 left) 2020-06-12T02:39:19.438 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:39:19.438 INFO:tasks.rados.rados.0.smithi107.stdout:11491: seq_num 11490 ranges {0=442368} 2020-06-12T02:39:19.438 INFO:tasks.rados.rados.0.smithi107.stdout:11491: writing smithi10729407-9 from 0 to 442368 tid 1 2020-06-12T02:39:19.438 INFO:tasks.rados.rados.0.smithi107.stdout:11492: delete oid 12 current snap is 0 2020-06-12T02:39:19.439 INFO:tasks.rados.rados.0.smithi107.stdout:11487: finishing write tid 4 to smithi10729407-14 2020-06-12T02:39:19.439 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1787 (ObjNum 11486 snap 0 seq_num 11486) dirty exists 2020-06-12T02:39:19.439 INFO:tasks.rados.rados.0.smithi107.stdout:11487: left oid 14 (ObjNum 11486 snap 0 seq_num 11486) 2020-06-12T02:39:19.439 INFO:tasks.rados.rados.0.smithi107.stdout:11488: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:19.439 INFO:tasks.rados.rados.0.smithi107.stdout:11489: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:19.439 INFO:tasks.rados.rados.0.smithi107.stdout:11489: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:19.440 INFO:tasks.rados.rados.0.smithi107.stdout:11488: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:19.440 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 2163 (ObjNum 11487 snap 0 seq_num 11487) dirty exists 2020-06-12T02:39:19.441 INFO:tasks.rados.rados.0.smithi107.stdout:11488: left oid 8 (ObjNum 11487 snap 0 seq_num 11487) 2020-06-12T02:39:19.441 INFO:tasks.rados.rados.0.smithi107.stdout:11489: finishing write tid 4 to smithi10729407-7 2020-06-12T02:39:19.442 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1852 (ObjNum 11488 snap 0 seq_num 11488) dirty exists 2020-06-12T02:39:19.442 INFO:tasks.rados.rados.0.smithi107.stdout:11489: left oid 7 (ObjNum 11488 snap 0 seq_num 11488) 2020-06-12T02:39:19.442 INFO:tasks.rados.rados.0.smithi107.stdout:11487: done (4 left) 2020-06-12T02:39:19.443 INFO:tasks.rados.rados.0.smithi107.stdout:11488: done (3 left) 2020-06-12T02:39:19.443 INFO:tasks.rados.rados.0.smithi107.stdout:11489: done (2 left) 2020-06-12T02:39:19.444 INFO:tasks.rados.rados.0.smithi107.stdout:11492: done (1 left) 2020-06-12T02:39:19.444 INFO:tasks.rados.rados.0.smithi107.stdout:11493: delete oid 5 current snap is 0 2020-06-12T02:39:19.445 INFO:tasks.rados.rados.0.smithi107.stdout:11491: finishing write tid 1 to smithi10729407-9 2020-06-12T02:39:19.445 INFO:tasks.rados.rados.0.smithi107.stdout:11491: finishing write tid 3 to smithi10729407-9 2020-06-12T02:39:19.446 INFO:tasks.rados.rados.0.smithi107.stdout:11493: done (1 left) 2020-06-12T02:39:19.447 INFO:tasks.rados.rados.0.smithi107.stdout:11494: delete oid 2 current snap is 0 2020-06-12T02:39:19.447 INFO:tasks.rados.rados.0.smithi107.stdout:11491: finishing write tid 4 to smithi10729407-9 2020-06-12T02:39:19.447 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1888 (ObjNum 11490 snap 0 seq_num 11490) dirty exists 2020-06-12T02:39:19.448 INFO:tasks.rados.rados.0.smithi107.stdout:11491: left oid 9 (ObjNum 11490 snap 0 seq_num 11490) 2020-06-12T02:39:19.448 INFO:tasks.rados.rados.0.smithi107.stdout:11491: done (1 left) 2020-06-12T02:39:19.448 INFO:tasks.rados.rados.0.smithi107.stdout:11494: done (0 left) 2020-06-12T02:39:19.449 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:39:19.449 INFO:tasks.rados.rados.0.smithi107.stdout:11495: seq_num 11494 ranges {2621440=507904} 2020-06-12T02:39:19.452 INFO:tasks.rados.rados.0.smithi107.stdout:11495: writing smithi10729407-18 from 2621440 to 3129344 tid 1 2020-06-12T02:39:19.452 INFO:tasks.rados.rados.0.smithi107.stdout:11496: delete oid 9 current snap is 0 2020-06-12T02:39:19.457 INFO:tasks.rados.rados.0.smithi107.stdout:11496: done (1 left) 2020-06-12T02:39:19.457 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:39:19.457 INFO:tasks.rados.rados.0.smithi107.stdout:11497: seq_num 11496 ranges {409600=540672} 2020-06-12T02:39:19.463 INFO:tasks.rados.rados.0.smithi107.stdout:11497: writing smithi10729407-7 from 409600 to 950272 tid 1 2020-06-12T02:39:19.463 INFO:tasks.rados.rados.0.smithi107.stdout:11498: delete oid 15 current snap is 0 2020-06-12T02:39:19.464 INFO:tasks.rados.rados.0.smithi107.stdout:11495: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:19.464 INFO:tasks.rados.rados.0.smithi107.stdout:11495: finishing write tid 3 to smithi10729407-18 2020-06-12T02:39:19.464 INFO:tasks.rados.rados.0.smithi107.stdout:11495: finishing write tid 4 to smithi10729407-18 2020-06-12T02:39:19.465 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1922 (ObjNum 11494 snap 0 seq_num 11494) dirty exists 2020-06-12T02:39:19.465 INFO:tasks.rados.rados.0.smithi107.stdout:11495: left oid 18 (ObjNum 11494 snap 0 seq_num 11494) 2020-06-12T02:39:19.465 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:39:19.460+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:39:19.467 INFO:tasks.rados.rados.0.smithi107.stdout:11495: done (2 left) 2020-06-12T02:39:19.468 INFO:tasks.rados.rados.0.smithi107.stdout:11498: done (1 left) 2020-06-12T02:39:19.468 INFO:tasks.rados.rados.0.smithi107.stdout:11499: delete oid 13 current snap is 0 2020-06-12T02:39:19.469 INFO:tasks.rados.rados.0.smithi107.stdout:11497: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:19.469 INFO:tasks.rados.rados.0.smithi107.stdout:11497: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:19.470 INFO:tasks.rados.rados.0.smithi107.stdout:11499: done (1 left) 2020-06-12T02:39:19.470 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:39:19.471 INFO:tasks.rados.rados.0.smithi107.stdout:11500: seq_num 11499 ranges {0=589824} 2020-06-12T02:39:19.473 INFO:tasks.rados.rados.0.smithi107.stdout:11500: writing smithi10729407-3 from 0 to 589824 tid 1 2020-06-12T02:39:19.474 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:39:19.474 INFO:tasks.rados.rados.0.smithi107.stdout:11497: finishing write tid 4 to smithi10729407-7 2020-06-12T02:39:19.474 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1854 (ObjNum 11496 snap 0 seq_num 11496) dirty exists 2020-06-12T02:39:19.474 INFO:tasks.rados.rados.0.smithi107.stdout:11497: left oid 7 (ObjNum 11496 snap 0 seq_num 11496) 2020-06-12T02:39:19.474 INFO:tasks.rados.rados.0.smithi107.stdout:11501: seq_num 11500 ranges {4448256=606208} 2020-06-12T02:39:19.475 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:39:19.475 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2020-06-12T02:39:19.475 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.476 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:24529 3.bs0 3:d0ebae7d:::smithi10729407-3:head [append 0~589824 in=589824b] snapc 0=[] ondisk+write+known_if_redirected e87)", 2020-06-12T02:39:19.476 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:39:19.471885+0000", 2020-06-12T02:39:19.476 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.000565271, 2020-06-12T02:39:19.476 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00058332499999999999, 2020-06-12T02:39:19.476 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:19.477 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:39:19.477 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:19.477 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:19.477 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:19.477 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 24529 2020-06-12T02:39:19.478 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.478 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:19.478 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.478 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:19.478 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:19.471885+0000", 2020-06-12T02:39:19.479 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:19.479 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.479 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.479 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:19.480 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:19.471885+0000", 2020-06-12T02:39:19.480 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000274259 2020-06-12T02:39:19.480 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.480 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.480 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:19.481 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:19.472159+0000", 2020-06-12T02:39:19.481 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8799999999999996e-06 2020-06-12T02:39:19.481 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.481 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.481 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:19.482 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:19.472165+0000", 2020-06-12T02:39:19.482 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1908e-05 2020-06-12T02:39:19.482 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.482 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.482 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:19.483 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:19.472177+0000", 2020-06-12T02:39:19.483 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9739999999999999e-06 2020-06-12T02:39:19.483 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.483 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.483 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:19.483 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:19.472183+0000", 2020-06-12T02:39:19.484 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7716000000000002e-05 2020-06-12T02:39:19.484 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.484 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.484 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:19.484 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:19.472231+0000", 2020-06-12T02:39:19.485 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011000100000000001 2020-06-12T02:39:19.485 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.485 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.485 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:19.485 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:19.472341+0000", 2020-06-12T02:39:19.486 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00045573800000000002 2020-06-12T02:39:19.486 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.486 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:19.486 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.486 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.486 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.487 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:24530 3.bs0 3.be75d70b (undecoded) ondisk+write+known_if_redirected e87)", 2020-06-12T02:39:19.487 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:39:19.472216+0000", 2020-06-12T02:39:19.487 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.000234443, 2020-06-12T02:39:19.487 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000300921, 2020-06-12T02:39:19.488 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:19.488 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "queued for pg", 2020-06-12T02:39:19.489 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:19.489 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:19.489 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:19.489 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 24530 2020-06-12T02:39:19.492 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.492 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:19.492 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.493 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:19.493 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:19.472216+0000", 2020-06-12T02:39:19.493 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:19.494 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.494 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.494 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:19.494 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:19.472216+0000", 2020-06-12T02:39:19.494 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7625e-05 2020-06-12T02:39:19.495 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.495 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.495 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:19.495 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:19.472253+0000", 2020-06-12T02:39:19.495 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4630000000000001e-06 2020-06-12T02:39:19.496 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.497 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.497 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:19.497 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:19.472257+0000", 2020-06-12T02:39:19.497 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0519999999999998e-06 2020-06-12T02:39:19.498 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.498 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.498 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:19.498 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:19.472263+0000", 2020-06-12T02:39:19.499 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5280000000000002e-06 2020-06-12T02:39:19.499 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.499 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.499 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:19.499 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:19.472265+0000", 2020-06-12T02:39:19.500 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9667999999999999e-05 2020-06-12T02:39:19.500 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.500 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:19.500 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.500 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.501 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.501 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:24531 3.bs0 3.be75d70b (undecoded) ondisk+read+rwordered+known_if_redirected e87)", 2020-06-12T02:39:19.501 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:39:19.472289+0000", 2020-06-12T02:39:19.501 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.00016130300000000001, 2020-06-12T02:39:19.501 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00026221399999999998, 2020-06-12T02:39:19.502 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:19.502 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "queued for pg", 2020-06-12T02:39:19.502 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:19.502 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:19.502 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:19.503 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 24531 2020-06-12T02:39:19.503 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.503 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:19.503 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.503 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:19.504 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:19.472289+0000", 2020-06-12T02:39:19.504 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:19.504 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.505 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.505 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:19.505 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:19.472289+0000", 2020-06-12T02:39:19.506 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2717999999999998e-05 2020-06-12T02:39:19.506 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.506 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.506 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:19.507 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:19.472311+0000", 2020-06-12T02:39:19.507 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5219999999999999e-06 2020-06-12T02:39:19.507 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.507 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.508 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:19.508 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:19.472315+0000", 2020-06-12T02:39:19.508 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4370000000000004e-06 2020-06-12T02:39:19.508 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.509 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.509 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:19.509 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:19.472321+0000", 2020-06-12T02:39:19.509 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.368e-06 2020-06-12T02:39:19.509 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.509 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.510 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:19.510 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:19.472324+0000", 2020-06-12T02:39:19.510 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5045000000000001e-05 2020-06-12T02:39:19.510 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.510 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:19.511 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.511 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.511 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2020-06-12T02:39:19.511 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 3 2020-06-12T02:39:19.511 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:39:19.514 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-06-12T02:39:19.514 INFO:tasks.rados.rados.0.smithi107.stdout:11501: writing smithi10729407-1 from 4448256 to 5054464 tid 1 2020-06-12T02:39:19.514 INFO:tasks.rados.rados.0.smithi107.stdout:11497: done (2 left) 2020-06-12T02:39:19.515 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:39:19.515 INFO:tasks.rados.rados.0.smithi107.stdout:11502: seq_num 11501 ranges {2162688=622592} 2020-06-12T02:39:19.516 INFO:tasks.rados.rados.0.smithi107.stdout:11502: writing smithi10729407-20 from 2162688 to 2785280 tid 1 2020-06-12T02:39:19.516 INFO:tasks.rados.rados.0.smithi107.stdout:11503: delete oid 14 current snap is 0 2020-06-12T02:39:19.517 INFO:tasks.rados.rados.0.smithi107.stdout:11500: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:19.517 INFO:tasks.rados.rados.0.smithi107.stdout:11500: finishing write tid 3 to smithi10729407-3 2020-06-12T02:39:19.517 INFO:tasks.rados.rados.0.smithi107.stdout:11500: finishing write tid 4 to smithi10729407-3 2020-06-12T02:39:19.518 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 2017 (ObjNum 11499 snap 0 seq_num 11499) dirty exists 2020-06-12T02:39:19.524 INFO:tasks.rados.rados.0.smithi107.stdout:11500: left oid 3 (ObjNum 11499 snap 0 seq_num 11499) 2020-06-12T02:39:19.524 INFO:tasks.rados.rados.0.smithi107.stdout:11501: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:19.524 INFO:tasks.rados.rados.0.smithi107.stdout:11501: finishing write tid 3 to smithi10729407-1 2020-06-12T02:39:19.524 INFO:tasks.rados.rados.0.smithi107.stdout:11501: finishing write tid 4 to smithi10729407-1 2020-06-12T02:39:19.525 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 2096 (ObjNum 11500 snap 0 seq_num 11500) dirty exists 2020-06-12T02:39:19.525 INFO:tasks.rados.rados.0.smithi107.stdout:11501: left oid 1 (ObjNum 11500 snap 0 seq_num 11500) 2020-06-12T02:39:19.526 INFO:tasks.rados.rados.0.smithi107.stdout:11500: done (3 left) 2020-06-12T02:39:19.527 INFO:tasks.rados.rados.0.smithi107.stdout:11501: done (2 left) 2020-06-12T02:39:19.527 INFO:tasks.rados.rados.0.smithi107.stdout:11503: done (1 left) 2020-06-12T02:39:19.527 INFO:tasks.rados.rados.0.smithi107.stdout:11504: delete oid 9 current snap is 0 2020-06-12T02:39:19.527 INFO:tasks.rados.rados.0.smithi107.stdout:11504: done (1 left) 2020-06-12T02:39:19.527 INFO:tasks.rados.rados.0.smithi107.stdout:11505: delete oid 3 current snap is 0 2020-06-12T02:39:19.528 INFO:tasks.rados.rados.0.smithi107.stdout:11502: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:19.528 INFO:tasks.rados.rados.0.smithi107.stdout:11505: done (1 left) 2020-06-12T02:39:19.528 INFO:tasks.rados.rados.0.smithi107.stdout:11506: delete oid 5 current snap is 0 2020-06-12T02:39:19.528 INFO:tasks.rados.rados.0.smithi107.stdout:11506: done (1 left) 2020-06-12T02:39:19.528 INFO:tasks.rados.rados.0.smithi107.stdout:11507: delete oid 3 current snap is 0 2020-06-12T02:39:19.529 INFO:tasks.rados.rados.0.smithi107.stdout:11502: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:19.530 INFO:tasks.rados.rados.0.smithi107.stdout:11502: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:19.530 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 896 (ObjNum 11501 snap 0 seq_num 11501) dirty exists 2020-06-12T02:39:19.530 INFO:tasks.rados.rados.0.smithi107.stdout:11502: left oid 20 (ObjNum 11501 snap 0 seq_num 11501) 2020-06-12T02:39:19.530 INFO:tasks.rados.rados.0.smithi107.stdout:11502: done (1 left) 2020-06-12T02:39:19.531 INFO:tasks.rados.rados.0.smithi107.stdout:11507: done (0 left) 2020-06-12T02:39:19.531 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:39:19.531 INFO:tasks.rados.rados.0.smithi107.stdout:11508: seq_num 11507 ranges {5054464=729088} 2020-06-12T02:39:19.561 INFO:tasks.rados.rados.0.smithi107.stdout:11508: writing smithi10729407-1 from 5054464 to 5783552 tid 1 2020-06-12T02:39:19.562 INFO:tasks.rados.rados.0.smithi107.stdout:11509: delete oid 15 current snap is 0 2020-06-12T02:39:19.565 INFO:tasks.rados.rados.0.smithi107.stdout:11509: done (1 left) 2020-06-12T02:39:19.566 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:39:19.566 INFO:tasks.rados.rados.0.smithi107.stdout:11510: seq_num 11509 ranges {1171456=761856} 2020-06-12T02:39:19.566 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:39:19.561+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:39:19.581 INFO:tasks.rados.rados.0.smithi107.stdout:11510: writing smithi10729407-16 from 1171456 to 1933312 tid 1 2020-06-12T02:39:19.582 INFO:tasks.rados.rados.0.smithi107.stdout:11511: delete oid 19 current snap is 0 2020-06-12T02:39:19.582 INFO:tasks.rados.rados.0.smithi107.stdout:11508: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:19.582 INFO:tasks.rados.rados.0.smithi107.stdout:11508: finishing write tid 3 to smithi10729407-1 2020-06-12T02:39:19.582 INFO:tasks.rados.rados.0.smithi107.stdout:11508: finishing write tid 4 to smithi10729407-1 2020-06-12T02:39:19.583 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 2098 (ObjNum 11507 snap 0 seq_num 11507) dirty exists 2020-06-12T02:39:19.583 INFO:tasks.rados.rados.0.smithi107.stdout:11508: left oid 1 (ObjNum 11507 snap 0 seq_num 11507) 2020-06-12T02:39:19.583 INFO:tasks.rados.rados.0.smithi107.stdout:11508: done (2 left) 2020-06-12T02:39:19.583 INFO:tasks.rados.rados.0.smithi107.stdout:11511: done (1 left) 2020-06-12T02:39:19.583 INFO:tasks.rados.rados.0.smithi107.stdout:11512: delete oid 19 current snap is 0 2020-06-12T02:39:19.583 INFO:tasks.rados.rados.0.smithi107.stdout:11512: done (1 left) 2020-06-12T02:39:19.584 INFO:tasks.rados.rados.0.smithi107.stdout:11513: delete oid 12 current snap is 0 2020-06-12T02:39:19.584 INFO:tasks.rados.rados.0.smithi107.stdout:11510: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:19.585 INFO:tasks.rados.rados.0.smithi107.stdout:11513: done (1 left) 2020-06-12T02:39:19.585 INFO:tasks.rados.rados.0.smithi107.stdout:11514: delete oid 1 current snap is 0 2020-06-12T02:39:19.586 INFO:tasks.rados.rados.0.smithi107.stdout:11510: finishing write tid 3 to smithi10729407-16 2020-06-12T02:39:19.586 INFO:tasks.rados.rados.0.smithi107.stdout:11514: done (1 left) 2020-06-12T02:39:19.587 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:39:19.587 INFO:tasks.rados.rados.0.smithi107.stdout:11515: seq_num 11514 ranges {0=442368,442368=401408} 2020-06-12T02:39:19.589 INFO:tasks.rados.rados.0.smithi107.stdout:11515: writing smithi10729407-12 from 0 to 442368 tid 1 2020-06-12T02:39:19.592 INFO:tasks.rados.rados.0.smithi107.stdout:11515: writing smithi10729407-12 from 442368 to 843776 tid 2 2020-06-12T02:39:19.592 INFO:tasks.rados.rados.0.smithi107.stdout:11516: delete oid 2 current snap is 0 2020-06-12T02:39:19.592 INFO:tasks.rados.rados.0.smithi107.stdout:11510: finishing write tid 4 to smithi10729407-16 2020-06-12T02:39:19.593 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 2021 (ObjNum 11509 snap 0 seq_num 11509) dirty exists 2020-06-12T02:39:19.593 INFO:tasks.rados.rados.0.smithi107.stdout:11510: left oid 16 (ObjNum 11509 snap 0 seq_num 11509) 2020-06-12T02:39:19.594 INFO:tasks.rados.rados.0.smithi107.stdout:11510: done (2 left) 2020-06-12T02:39:19.594 INFO:tasks.rados.rados.0.smithi107.stdout:11516: done (1 left) 2020-06-12T02:39:19.594 INFO:tasks.rados.rados.0.smithi107.stdout:11517: delete oid 16 current snap is 0 2020-06-12T02:39:19.595 INFO:tasks.rados.rados.0.smithi107.stdout:11515: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:19.596 INFO:tasks.rados.rados.0.smithi107.stdout:11517: done (1 left) 2020-06-12T02:39:19.597 INFO:tasks.rados.rados.0.smithi107.stdout:11518: delete oid 10 current snap is 0 2020-06-12T02:39:19.607 INFO:tasks.rados.rados.0.smithi107.stdout:11518: done (1 left) 2020-06-12T02:39:19.607 INFO:tasks.rados.rados.0.smithi107.stdout:11519: delete oid 19 current snap is 0 2020-06-12T02:39:19.609 INFO:tasks.rados.rados.0.smithi107.stdout:11515: finishing write tid 2 to smithi10729407-12 2020-06-12T02:39:19.609 INFO:tasks.rados.rados.0.smithi107.stdout:11515: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:19.609 INFO:tasks.rados.rados.0.smithi107.stdout:11519: done (1 left) 2020-06-12T02:39:19.609 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:39:19.610 INFO:tasks.rados.rados.0.smithi107.stdout:11520: seq_num 11519 ranges {0=524288,524288=401408} 2020-06-12T02:39:19.610 INFO:tasks.rados.rados.0.smithi107.stdout:11520: writing smithi10729407-6 from 0 to 524288 tid 1 2020-06-12T02:39:19.610 INFO:tasks.rados.rados.0.smithi107.stdout:11520: writing smithi10729407-6 from 524288 to 925696 tid 2 2020-06-12T02:39:19.610 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:39:19.611 INFO:tasks.rados.rados.0.smithi107.stdout:11521: seq_num 11520 ranges {950272=540672,1490944=401408} 2020-06-12T02:39:19.616 INFO:tasks.rados.rados.0.smithi107.stdout:11521: writing smithi10729407-7 from 950272 to 1490944 tid 1 2020-06-12T02:39:19.618 INFO:tasks.rados.rados.0.smithi107.stdout:11521: writing smithi10729407-7 from 1490944 to 1892352 tid 2 2020-06-12T02:39:19.619 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:39:19.619 INFO:tasks.rados.rados.0.smithi107.stdout:11522: seq_num 11521 ranges {0=557056,557056=401408} 2020-06-12T02:39:19.621 INFO:tasks.rados.rados.0.smithi107.stdout:11522: writing smithi10729407-17 from 0 to 557056 tid 1 2020-06-12T02:39:19.624 INFO:tasks.rados.rados.0.smithi107.stdout:11522: writing smithi10729407-17 from 557056 to 958464 tid 2 2020-06-12T02:39:19.624 INFO:tasks.rados.rados.0.smithi107.stdout:11523: delete oid 9 current snap is 0 2020-06-12T02:39:19.625 INFO:tasks.rados.rados.0.smithi107.stdout:11515: finishing write tid 5 to smithi10729407-12 2020-06-12T02:39:19.625 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 2149 (ObjNum 11514 snap 0 seq_num 11514) dirty exists 2020-06-12T02:39:19.625 INFO:tasks.rados.rados.0.smithi107.stdout:11515: left oid 12 (ObjNum 11514 snap 0 seq_num 11514) 2020-06-12T02:39:19.625 INFO:tasks.rados.rados.0.smithi107.stdout:11520: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:19.626 INFO:tasks.rados.rados.0.smithi107.stdout:11520: finishing write tid 2 to smithi10729407-6 2020-06-12T02:39:19.626 INFO:tasks.rados.rados.0.smithi107.stdout:11520: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:19.626 INFO:tasks.rados.rados.0.smithi107.stdout:11520: finishing write tid 5 to smithi10729407-6 2020-06-12T02:39:19.627 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 2102 (ObjNum 11519 snap 0 seq_num 11519) dirty exists 2020-06-12T02:39:19.627 INFO:tasks.rados.rados.0.smithi107.stdout:11520: left oid 6 (ObjNum 11519 snap 0 seq_num 11519) 2020-06-12T02:39:19.627 INFO:tasks.rados.rados.0.smithi107.stdout:11521: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:19.628 INFO:tasks.rados.rados.0.smithi107.stdout:11521: finishing write tid 2 to smithi10729407-7 2020-06-12T02:39:19.628 INFO:tasks.rados.rados.0.smithi107.stdout:11521: finishing write tid 4 to smithi10729407-7 2020-06-12T02:39:19.628 INFO:tasks.rados.rados.0.smithi107.stdout:11515: done (4 left) 2020-06-12T02:39:19.629 INFO:tasks.rados.rados.0.smithi107.stdout:11520: done (3 left) 2020-06-12T02:39:19.629 INFO:tasks.rados.rados.0.smithi107.stdout:11523: done (2 left) 2020-06-12T02:39:19.638 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:39:19.638 INFO:tasks.rados.rados.0.smithi107.stdout:11524: seq_num 11523 ranges {843776=589824,1433600=401408} 2020-06-12T02:39:19.639 INFO:tasks.rados.rados.0.smithi107.stdout:11524: writing smithi10729407-12 from 843776 to 1433600 tid 1 2020-06-12T02:39:19.639 INFO:tasks.rados.rados.0.smithi107.stdout:11524: writing smithi10729407-12 from 1433600 to 1835008 tid 2 2020-06-12T02:39:19.639 INFO:tasks.rados.rados.0.smithi107.stdout:11525: delete oid 14 current snap is 0 2020-06-12T02:39:19.639 INFO:tasks.rados.rados.0.smithi107.stdout:11521: finishing write tid 5 to smithi10729407-7 2020-06-12T02:39:19.640 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1857 (ObjNum 11520 snap 0 seq_num 11520) dirty exists 2020-06-12T02:39:19.640 INFO:tasks.rados.rados.0.smithi107.stdout:11521: left oid 7 (ObjNum 11520 snap 0 seq_num 11520) 2020-06-12T02:39:19.640 INFO:tasks.rados.rados.0.smithi107.stdout:11522: finishing write tid 1 to smithi10729407-17 2020-06-12T02:39:19.641 INFO:tasks.rados.rados.0.smithi107.stdout:11522: finishing write tid 2 to smithi10729407-17 2020-06-12T02:39:19.641 INFO:tasks.rados.rados.0.smithi107.stdout:11522: finishing write tid 4 to smithi10729407-17 2020-06-12T02:39:19.641 INFO:tasks.rados.rados.0.smithi107.stdout:11522: finishing write tid 5 to smithi10729407-17 2020-06-12T02:39:19.641 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 2144 (ObjNum 11521 snap 0 seq_num 11521) dirty exists 2020-06-12T02:39:19.641 INFO:tasks.rados.rados.0.smithi107.stdout:11522: left oid 17 (ObjNum 11521 snap 0 seq_num 11521) 2020-06-12T02:39:19.642 INFO:tasks.rados.rados.0.smithi107.stdout:11521: done (3 left) 2020-06-12T02:39:19.642 INFO:tasks.rados.rados.0.smithi107.stdout:11522: done (2 left) 2020-06-12T02:39:19.642 INFO:tasks.rados.rados.0.smithi107.stdout:11525: done (1 left) 2020-06-12T02:39:19.643 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:39:19.643 INFO:tasks.rados.rados.0.smithi107.stdout:11526: seq_num 11525 ranges {925696=630784,1556480=401408} 2020-06-12T02:39:19.648 INFO:tasks.rados.rados.0.smithi107.stdout:11526: writing smithi10729407-6 from 925696 to 1556480 tid 1 2020-06-12T02:39:19.651 INFO:tasks.rados.rados.0.smithi107.stdout:11526: writing smithi10729407-6 from 1556480 to 1957888 tid 2 2020-06-12T02:39:19.651 INFO:tasks.rados.rados.0.smithi107.stdout:11527: delete oid 7 current snap is 0 2020-06-12T02:39:19.652 INFO:tasks.rados.rados.0.smithi107.stdout:11524: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:19.652 INFO:tasks.rados.rados.0.smithi107.stdout:11524: finishing write tid 2 to smithi10729407-12 2020-06-12T02:39:19.652 INFO:tasks.rados.rados.0.smithi107.stdout:11524: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:19.653 INFO:tasks.rados.rados.0.smithi107.stdout:11524: finishing write tid 5 to smithi10729407-12 2020-06-12T02:39:19.653 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 2153 (ObjNum 11523 snap 0 seq_num 11523) dirty exists 2020-06-12T02:39:19.653 INFO:tasks.rados.rados.0.smithi107.stdout:11524: left oid 12 (ObjNum 11523 snap 0 seq_num 11523) 2020-06-12T02:39:19.654 INFO:tasks.rados.rados.0.smithi107.stdout:11524: done (2 left) 2020-06-12T02:39:19.654 INFO:tasks.rados.rados.0.smithi107.stdout:11527: done (1 left) 2020-06-12T02:39:19.655 INFO:tasks.rados.rados.0.smithi107.stdout:11528: delete oid 15 current snap is 0 2020-06-12T02:39:19.659 INFO:tasks.rados.rados.0.smithi107.stdout:11526: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:19.659 INFO:tasks.rados.rados.0.smithi107.stdout:11528: done (1 left) 2020-06-12T02:39:19.660 INFO:tasks.rados.rados.0.smithi107.stdout:11529: delete oid 14 current snap is 0 2020-06-12T02:39:19.660 INFO:tasks.rados.rados.0.smithi107.stdout:11526: finishing write tid 2 to smithi10729407-6 2020-06-12T02:39:19.661 INFO:tasks.rados.rados.0.smithi107.stdout:11526: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:19.662 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:39:19.662 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2020-06-12T02:39:19.662 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2020-06-12T02:39:19.663 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2020-06-12T02:39:19.663 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:39:19.663 INFO:tasks.rados.rados.0.smithi107.stdout:11526: finishing write tid 5 to smithi10729407-6 2020-06-12T02:39:19.664 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 2105 (ObjNum 11525 snap 0 seq_num 11525) dirty exists 2020-06-12T02:39:19.664 INFO:tasks.rados.rados.0.smithi107.stdout:11526: left oid 6 (ObjNum 11525 snap 0 seq_num 11525) 2020-06-12T02:39:19.664 INFO:tasks.rados.rados.0.smithi107.stdout:11526: done (1 left) 2020-06-12T02:39:19.664 INFO:tasks.rados.rados.0.smithi107.stdout:11529: done (0 left) 2020-06-12T02:39:19.664 INFO:tasks.rados.rados.0.smithi107.stdout:11530: delete oid 18 current snap is 0 2020-06-12T02:39:19.666 INFO:tasks.rados.rados.0.smithi107.stdout:11530: done (0 left) 2020-06-12T02:39:19.666 INFO:tasks.rados.rados.0.smithi107.stdout:11531: delete oid 5 current snap is 0 2020-06-12T02:39:19.666 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:39:19.664+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:39:19.668 INFO:tasks.rados.rados.0.smithi107.stdout:11531: done (0 left) 2020-06-12T02:39:19.668 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:39:19.669 INFO:tasks.rados.rados.0.smithi107.stdout:11532: seq_num 11531 ranges {0=729088,729088=401408} 2020-06-12T02:39:19.673 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-06-12T02:39:19.674 INFO:tasks.rados.rados.0.smithi107.stdout:11532: writing smithi10729407-15 from 0 to 729088 tid 1 2020-06-12T02:39:19.675 INFO:tasks.rados.rados.0.smithi107.stdout:11532: writing smithi10729407-15 from 729088 to 1130496 tid 2 2020-06-12T02:39:19.675 INFO:tasks.rados.rados.0.smithi107.stdout:11533: delete oid 18 current snap is 0 2020-06-12T02:39:19.677 INFO:tasks.rados.rados.0.smithi107.stdout:11533: done (1 left) 2020-06-12T02:39:19.677 INFO:tasks.rados.rados.0.smithi107.stdout:11534: delete oid 2 current snap is 0 2020-06-12T02:39:19.679 INFO:tasks.rados.rados.0.smithi107.stdout:11532: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:19.680 INFO:tasks.rados.rados.0.smithi107.stdout:11534: done (1 left) 2020-06-12T02:39:19.680 INFO:tasks.rados.rados.0.smithi107.stdout:11535: delete oid 11 current snap is 0 2020-06-12T02:39:19.682 INFO:tasks.rados.rados.0.smithi107.stdout:11535: done (1 left) 2020-06-12T02:39:19.682 INFO:tasks.rados.rados.0.smithi107.stdout:11536: delete oid 8 current snap is 0 2020-06-12T02:39:19.686 INFO:tasks.rados.rados.0.smithi107.stdout:11532: finishing write tid 2 to smithi10729407-15 2020-06-12T02:39:19.686 INFO:tasks.rados.rados.0.smithi107.stdout:11532: finishing write tid 4 to smithi10729407-15 2020-06-12T02:39:19.687 INFO:tasks.rados.rados.0.smithi107.stdout:11532: finishing write tid 5 to smithi10729407-15 2020-06-12T02:39:19.687 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 2169 (ObjNum 11531 snap 0 seq_num 11531) dirty exists 2020-06-12T02:39:19.687 INFO:tasks.rados.rados.0.smithi107.stdout:11532: left oid 15 (ObjNum 11531 snap 0 seq_num 11531) 2020-06-12T02:39:19.688 INFO:tasks.rados.rados.0.smithi107.stdout:11532: done (1 left) 2020-06-12T02:39:19.688 INFO:tasks.rados.rados.0.smithi107.stdout:11536: done (0 left) 2020-06-12T02:39:19.688 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:39:19.688 INFO:tasks.rados.rados.0.smithi107.stdout:11537: seq_num 11536 ranges {1835008=409600,2244608=622592,2867200=180224} 2020-06-12T02:39:19.700 INFO:tasks.rados.rados.0.smithi107.stdout:11537: writing smithi10729407-12 from 1835008 to 2244608 tid 1 2020-06-12T02:39:19.703 INFO:tasks.rados.rados.0.smithi107.stdout:11537: writing smithi10729407-12 from 2244608 to 2867200 tid 2 2020-06-12T02:39:19.705 INFO:tasks.rados.rados.0.smithi107.stdout:11537: writing smithi10729407-12 from 2867200 to 3047424 tid 3 2020-06-12T02:39:19.705 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:39:19.705 INFO:tasks.rados.rados.0.smithi107.stdout:11538: seq_num 11537 ranges {0=425984,425984=507904,933888=294912} 2020-06-12T02:39:19.707 INFO:tasks.rados.rados.0.smithi107.stdout:11538: writing smithi10729407-5 from 0 to 425984 tid 1 2020-06-12T02:39:19.710 INFO:tasks.rados.rados.0.smithi107.stdout:11538: writing smithi10729407-5 from 425984 to 933888 tid 2 2020-06-12T02:39:19.712 INFO:tasks.rados.rados.0.smithi107.stdout:11538: writing smithi10729407-5 from 933888 to 1228800 tid 3 2020-06-12T02:39:19.712 INFO:tasks.rados.rados.0.smithi107.stdout:11539: delete oid 15 current snap is 0 2020-06-12T02:39:19.713 INFO:tasks.rados.rados.0.smithi107.stdout:11537: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:19.714 INFO:tasks.rados.rados.0.smithi107.stdout:11537: finishing write tid 2 to smithi10729407-12 2020-06-12T02:39:19.714 INFO:tasks.rados.rados.0.smithi107.stdout:11537: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:19.715 INFO:tasks.rados.rados.0.smithi107.stdout:11539: done (2 left) 2020-06-12T02:39:19.715 INFO:tasks.rados.rados.0.smithi107.stdout:11540: delete oid 1 current snap is 0 2020-06-12T02:39:19.715 INFO:tasks.rados.rados.0.smithi107.stdout:11537: finishing write tid 5 to smithi10729407-12 2020-06-12T02:39:19.717 INFO:tasks.rados.rados.0.smithi107.stdout:11537: finishing write tid 6 to smithi10729407-12 2020-06-12T02:39:19.718 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 2157 (ObjNum 11536 snap 0 seq_num 11536) dirty exists 2020-06-12T02:39:19.718 INFO:tasks.rados.rados.0.smithi107.stdout:11537: left oid 12 (ObjNum 11536 snap 0 seq_num 11536) 2020-06-12T02:39:19.718 INFO:tasks.rados.rados.0.smithi107.stdout:11537: done (2 left) 2020-06-12T02:39:19.719 INFO:tasks.rados.rados.0.smithi107.stdout:11540: done (1 left) 2020-06-12T02:39:19.719 INFO:tasks.rados.rados.0.smithi107.stdout:11541: delete oid 1 current snap is 0 2020-06-12T02:39:19.719 INFO:tasks.rados.rados.0.smithi107.stdout:11538: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:19.721 INFO:tasks.rados.rados.0.smithi107.stdout:11541: done (1 left) 2020-06-12T02:39:19.721 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:39:19.722 INFO:tasks.rados.rados.0.smithi107.stdout:11542: seq_num 11541 ranges {0=491520,491520=491520,983040=311296} 2020-06-12T02:39:19.734 INFO:tasks.rados.rados.0.smithi107.stdout:11542: writing smithi10729407-8 from 0 to 491520 tid 1 2020-06-12T02:39:19.734 INFO:tasks.rados.rados.0.smithi107.stdout:11542: writing smithi10729407-8 from 491520 to 983040 tid 2 2020-06-12T02:39:19.734 INFO:tasks.rados.rados.0.smithi107.stdout:11542: writing smithi10729407-8 from 983040 to 1294336 tid 3 2020-06-12T02:39:19.735 INFO:tasks.rados.rados.0.smithi107.stdout:11543: delete oid 19 current snap is 0 2020-06-12T02:39:19.735 INFO:tasks.rados.rados.0.smithi107.stdout:11538: finishing write tid 2 to smithi10729407-5 2020-06-12T02:39:19.735 INFO:tasks.rados.rados.0.smithi107.stdout:11538: finishing write tid 3 to smithi10729407-5 2020-06-12T02:39:19.735 INFO:tasks.rados.rados.0.smithi107.stdout:11538: finishing write tid 5 to smithi10729407-5 2020-06-12T02:39:19.735 INFO:tasks.rados.rados.0.smithi107.stdout:11538: finishing write tid 6 to smithi10729407-5 2020-06-12T02:39:19.736 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 1097 (ObjNum 11537 snap 0 seq_num 11537) dirty exists 2020-06-12T02:39:19.736 INFO:tasks.rados.rados.0.smithi107.stdout:11538: left oid 5 (ObjNum 11537 snap 0 seq_num 11537) 2020-06-12T02:39:19.736 INFO:tasks.rados.rados.0.smithi107.stdout:11538: done (2 left) 2020-06-12T02:39:19.736 INFO:tasks.rados.rados.0.smithi107.stdout:11543: done (1 left) 2020-06-12T02:39:19.736 INFO:tasks.rados.rados.0.smithi107.stdout:11544: delete oid 11 current snap is 0 2020-06-12T02:39:19.737 INFO:tasks.rados.rados.0.smithi107.stdout:11544: done (1 left) 2020-06-12T02:39:19.737 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:39:19.737 INFO:tasks.rados.rados.0.smithi107.stdout:11545: seq_num 11544 ranges {319488=548864,868352=548864,1417216=253952} 2020-06-12T02:39:19.740 INFO:tasks.rados.rados.0.smithi107.stdout:11545: writing smithi10729407-4 from 319488 to 868352 tid 1 2020-06-12T02:39:19.743 INFO:tasks.rados.rados.0.smithi107.stdout:11545: writing smithi10729407-4 from 868352 to 1417216 tid 2 2020-06-12T02:39:19.745 INFO:tasks.rados.rados.0.smithi107.stdout:11545: writing smithi10729407-4 from 1417216 to 1671168 tid 3 2020-06-12T02:39:19.745 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:39:19.745 INFO:tasks.rados.rados.0.smithi107.stdout:11546: seq_num 11545 ranges {1228800=565248,1794048=434176,2228224=368640} 2020-06-12T02:39:19.755 INFO:tasks.rados.rados.0.smithi107.stdout:11546: writing smithi10729407-5 from 1228800 to 1794048 tid 1 2020-06-12T02:39:19.759 INFO:tasks.rados.rados.0.smithi107.stdout:11546: writing smithi10729407-5 from 1794048 to 2228224 tid 2 2020-06-12T02:39:19.760 INFO:tasks.rados.rados.0.smithi107.stdout:11546: writing smithi10729407-5 from 2228224 to 2596864 tid 3 2020-06-12T02:39:19.760 INFO:tasks.rados.rados.0.smithi107.stdout:11547: delete oid 20 current snap is 0 2020-06-12T02:39:19.760 INFO:tasks.rados.rados.0.smithi107.stdout:11542: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:19.761 INFO:tasks.rados.rados.0.smithi107.stdout:11542: finishing write tid 2 to smithi10729407-8 2020-06-12T02:39:19.761 INFO:tasks.rados.rados.0.smithi107.stdout:11542: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:19.761 INFO:tasks.rados.rados.0.smithi107.stdout:11542: finishing write tid 5 to smithi10729407-8 2020-06-12T02:39:19.761 INFO:tasks.rados.rados.0.smithi107.stdout:11542: finishing write tid 6 to smithi10729407-8 2020-06-12T02:39:19.762 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 2175 (ObjNum 11541 snap 0 seq_num 11541) dirty exists 2020-06-12T02:39:19.762 INFO:tasks.rados.rados.0.smithi107.stdout:11542: left oid 8 (ObjNum 11541 snap 0 seq_num 11541) 2020-06-12T02:39:19.762 INFO:tasks.rados.rados.0.smithi107.stdout:11545: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:19.762 INFO:tasks.rados.rados.0.smithi107.stdout:11545: finishing write tid 2 to smithi10729407-4 2020-06-12T02:39:19.762 INFO:tasks.rados.rados.0.smithi107.stdout:11545: finishing write tid 3 to smithi10729407-4 2020-06-12T02:39:19.763 INFO:tasks.rados.rados.0.smithi107.stdout:11545: finishing write tid 5 to smithi10729407-4 2020-06-12T02:39:19.763 INFO:tasks.rados.rados.0.smithi107.stdout:11545: finishing write tid 6 to smithi10729407-4 2020-06-12T02:39:19.763 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 2148 (ObjNum 11544 snap 0 seq_num 11544) dirty exists 2020-06-12T02:39:19.763 INFO:tasks.rados.rados.0.smithi107.stdout:11545: left oid 4 (ObjNum 11544 snap 0 seq_num 11544) 2020-06-12T02:39:19.764 INFO:tasks.rados.rados.0.smithi107.stdout:11542: done (3 left) 2020-06-12T02:39:19.764 INFO:tasks.rados.rados.0.smithi107.stdout:11545: done (2 left) 2020-06-12T02:39:19.764 INFO:tasks.rados.rados.0.smithi107.stdout:11547: done (1 left) 2020-06-12T02:39:19.765 INFO:tasks.rados.rados.0.smithi107.stdout:11548: delete oid 9 current snap is 0 2020-06-12T02:39:19.766 INFO:tasks.rados.rados.0.smithi107.stdout:11546: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:19.766 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:39:19.764+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:39:19.767 INFO:tasks.rados.rados.0.smithi107.stdout:11546: finishing write tid 2 to smithi10729407-5 2020-06-12T02:39:19.769 INFO:tasks.rados.rados.0.smithi107.stdout:11546: finishing write tid 3 to smithi10729407-5 2020-06-12T02:39:19.774 INFO:tasks.rados.rados.0.smithi107.stdout:11548: done (1 left) 2020-06-12T02:39:19.775 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:39:19.775 INFO:tasks.rados.rados.0.smithi107.stdout:11549: seq_num 11548 ranges {0=614400,614400=483328,1097728=319488} 2020-06-12T02:39:19.775 INFO:tasks.rados.rados.0.smithi107.stdout:11549: writing smithi10729407-1 from 0 to 614400 tid 1 2020-06-12T02:39:19.777 INFO:tasks.rados.rados.0.smithi107.stdout:11549: writing smithi10729407-1 from 614400 to 1097728 tid 2 2020-06-12T02:39:19.779 INFO:tasks.rados.rados.0.smithi107.stdout:11549: writing smithi10729407-1 from 1097728 to 1417216 tid 3 2020-06-12T02:39:19.780 INFO:tasks.rados.rados.0.smithi107.stdout:11550: delete oid 6 current snap is 0 2020-06-12T02:39:19.780 INFO:tasks.rados.rados.0.smithi107.stdout:11546: finishing write tid 5 to smithi10729407-5 2020-06-12T02:39:19.780 INFO:tasks.rados.rados.0.smithi107.stdout:11546: finishing write tid 6 to smithi10729407-5 2020-06-12T02:39:19.780 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 1101 (ObjNum 11545 snap 0 seq_num 11545) dirty exists 2020-06-12T02:39:19.780 INFO:tasks.rados.rados.0.smithi107.stdout:11546: left oid 5 (ObjNum 11545 snap 0 seq_num 11545) 2020-06-12T02:39:19.785 INFO:tasks.rados.rados.0.smithi107.stdout:11549: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:19.789 INFO:tasks.rados.rados.0.smithi107.stdout:11549: finishing write tid 2 to smithi10729407-1 2020-06-12T02:39:19.789 INFO:tasks.rados.rados.0.smithi107.stdout:11549: finishing write tid 3 to smithi10729407-1 2020-06-12T02:39:19.790 INFO:tasks.rados.rados.0.smithi107.stdout:11549: finishing write tid 5 to smithi10729407-1 2020-06-12T02:39:19.800 INFO:tasks.rados.rados.0.smithi107.stdout:11546: done (2 left) 2020-06-12T02:39:19.800 INFO:tasks.rados.rados.0.smithi107.stdout:11550: done (1 left) 2020-06-12T02:39:19.801 INFO:tasks.rados.rados.0.smithi107.stdout:11551: delete oid 4 current snap is 0 2020-06-12T02:39:19.802 INFO:tasks.rados.rados.0.smithi107.stdout:11549: finishing write tid 6 to smithi10729407-1 2020-06-12T02:39:19.802 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 2111 (ObjNum 11548 snap 0 seq_num 11548) dirty exists 2020-06-12T02:39:19.802 INFO:tasks.rados.rados.0.smithi107.stdout:11549: left oid 1 (ObjNum 11548 snap 0 seq_num 11548) 2020-06-12T02:39:19.802 INFO:tasks.rados.rados.0.smithi107.stdout:11549: done (1 left) 2020-06-12T02:39:19.803 INFO:tasks.rados.rados.0.smithi107.stdout:11551: done (0 left) 2020-06-12T02:39:19.803 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:39:19.803 INFO:tasks.rados.rados.0.smithi107.stdout:11552: seq_num 11551 ranges {0=663552,663552=589824,1253376=212992} 2020-06-12T02:39:19.803 INFO:tasks.rados.rados.0.smithi107.stdout:11552: writing smithi10729407-6 from 0 to 663552 tid 1 2020-06-12T02:39:19.803 INFO:tasks.rados.rados.0.smithi107.stdout:11552: writing smithi10729407-6 from 663552 to 1253376 tid 2 2020-06-12T02:39:19.804 INFO:tasks.rados.rados.0.smithi107.stdout:11552: writing smithi10729407-6 from 1253376 to 1466368 tid 3 2020-06-12T02:39:19.804 INFO:tasks.rados.rados.0.smithi107.stdout:11553: delete oid 8 current snap is 0 2020-06-12T02:39:19.810 INFO:tasks.rados.rados.0.smithi107.stdout:11553: done (1 left) 2020-06-12T02:39:19.810 INFO:tasks.rados.rados.0.smithi107.stdout:11554: delete oid 2 current snap is 0 2020-06-12T02:39:19.811 INFO:tasks.rados.rados.0.smithi107.stdout:11552: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:19.811 INFO:tasks.rados.rados.0.smithi107.stdout:11554: done (1 left) 2020-06-12T02:39:19.811 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:39:19.811 INFO:tasks.rados.rados.0.smithi107.stdout:11555: seq_num 11554 ranges {0=712704,712704=638976,1351680=163840} 2020-06-12T02:39:19.811 INFO:tasks.rados.rados.0.smithi107.stdout:11555: writing smithi10729407-2 from 0 to 712704 tid 1 2020-06-12T02:39:19.814 INFO:tasks.rados.rados.0.smithi107.stdout:11555: writing smithi10729407-2 from 712704 to 1351680 tid 2 2020-06-12T02:39:19.815 INFO:tasks.rados.rados.0.smithi107.stdout:11555: writing smithi10729407-2 from 1351680 to 1515520 tid 3 2020-06-12T02:39:19.815 INFO:tasks.rados.rados.0.smithi107.stdout:11556: delete oid 3 current snap is 0 2020-06-12T02:39:19.816 INFO:tasks.rados.rados.0.smithi107.stdout:11552: finishing write tid 2 to smithi10729407-6 2020-06-12T02:39:19.816 INFO:tasks.rados.rados.0.smithi107.stdout:11552: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:19.816 INFO:tasks.rados.rados.0.smithi107.stdout:11552: finishing write tid 5 to smithi10729407-6 2020-06-12T02:39:19.816 INFO:tasks.rados.rados.0.smithi107.stdout:11552: finishing write tid 6 to smithi10729407-6 2020-06-12T02:39:19.817 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 2116 (ObjNum 11551 snap 0 seq_num 11551) dirty exists 2020-06-12T02:39:19.817 INFO:tasks.rados.rados.0.smithi107.stdout:11552: left oid 6 (ObjNum 11551 snap 0 seq_num 11551) 2020-06-12T02:39:19.820 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:39:19.821 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2020-06-12T02:39:19.821 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2020-06-12T02:39:19.821 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2020-06-12T02:39:19.821 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.823 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:13722 3.2s0 3:4afffc6f:::smithi10729407-20:head [stat,delete] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-06-12T02:39:19.823 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:28.421613+0000", 2020-06-12T02:39:19.823 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 51.393801185000001, 2020-06-12T02:39:19.823 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046123156999999998, 2020-06-12T02:39:19.823 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:19.824 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:39:19.824 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:19.824 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:19.824 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:19.824 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 13722 2020-06-12T02:39:19.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.825 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:19.825 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.825 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:19.825 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:28.421613+0000", 2020-06-12T02:39:19.825 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:19.826 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.826 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.826 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:19.826 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:28.421613+0000", 2020-06-12T02:39:19.827 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.533e-05 2020-06-12T02:39:19.827 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.827 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.827 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:19.827 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:28.421638+0000", 2020-06-12T02:39:19.828 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4389999999999999e-06 2020-06-12T02:39:19.828 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.828 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.829 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:19.829 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:28.421643+0000", 2020-06-12T02:39:19.829 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1055e-05 2020-06-12T02:39:19.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.829 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.830 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:19.830 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:28.421654+0000", 2020-06-12T02:39:19.830 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3290000000000004e-06 2020-06-12T02:39:19.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.830 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.831 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:19.831 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:28.421659+0000", 2020-06-12T02:39:19.831 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3794000000000002e-05 2020-06-12T02:39:19.831 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.831 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.832 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:19.832 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:28.421692+0000", 2020-06-12T02:39:19.832 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2979999999999995e-05 2020-06-12T02:39:19.832 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.832 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.832 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:19.833 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:28.421775+0000", 2020-06-12T02:39:19.833 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.045960229999999998 2020-06-12T02:39:19.833 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.833 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.833 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:19.834 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:28.467736+0000", 2020-06-12T02:39:19.834 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046123156999999998 2020-06-12T02:39:19.834 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.834 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:19.834 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.835 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.835 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.835 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14661 3.bs0 3:d09543d2:::smithi10729407-16:head [stat,delete] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:39:19.835 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.152199+0000", 2020-06-12T02:39:19.835 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 48.663214920999998, 2020-06-12T02:39:19.835 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.057759869999999998, 2020-06-12T02:39:19.836 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:19.836 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:39:19.836 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:19.836 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:19.836 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:19.837 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14661 2020-06-12T02:39:19.837 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.837 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:19.837 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.837 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:19.838 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.152199+0000", 2020-06-12T02:39:19.838 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:19.838 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.838 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.838 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:19.839 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.152199+0000", 2020-06-12T02:39:19.839 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6761999999999999e-05 2020-06-12T02:39:19.839 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.839 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.839 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:19.839 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.152226+0000", 2020-06-12T02:39:19.840 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1790000000000008e-06 2020-06-12T02:39:19.840 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.840 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.840 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:19.840 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.152234+0000", 2020-06-12T02:39:19.841 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1841999999999999e-05 2020-06-12T02:39:19.841 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.841 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.841 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:19.841 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.152246+0000", 2020-06-12T02:39:19.841 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4660000000000002e-06 2020-06-12T02:39:19.842 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.842 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.842 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:19.842 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.152253+0000", 2020-06-12T02:39:19.842 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9688000000000003e-05 2020-06-12T02:39:19.843 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.843 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.843 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:19.843 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.152303+0000", 2020-06-12T02:39:19.843 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4331000000000004e-05 2020-06-12T02:39:19.843 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.844 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.844 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:19.844 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.152377+0000", 2020-06-12T02:39:19.844 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.057581602000000003 2020-06-12T02:39:19.844 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.845 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.845 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:19.845 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.209959+0000", 2020-06-12T02:39:19.845 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.057759869999999998 2020-06-12T02:39:19.845 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.845 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:19.846 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.846 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.846 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.846 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14912 3.5s1 3:a446366d:::smithi10729407-11:head [append 0~524288,stat] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:39:19.846 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.893048+0000", 2020-06-12T02:39:19.847 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 47.922365356999997, 2020-06-12T02:39:19.847 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038906464000000002, 2020-06-12T02:39:19.847 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:19.847 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:19.847 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:19.847 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:19.848 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:19.848 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14912 2020-06-12T02:39:19.848 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.848 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:19.848 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.849 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:19.849 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.893048+0000", 2020-06-12T02:39:19.849 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:19.849 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.849 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.849 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:19.850 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.893048+0000", 2020-06-12T02:39:19.850 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000144361 2020-06-12T02:39:19.850 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.850 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.850 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:19.851 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.893193+0000", 2020-06-12T02:39:19.851 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7450000000000001e-06 2020-06-12T02:39:19.851 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.851 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.851 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:19.851 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.893201+0000", 2020-06-12T02:39:19.852 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9619e-05 2020-06-12T02:39:19.852 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.852 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.852 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:19.852 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.893221+0000", 2020-06-12T02:39:19.853 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9449999999999993e-06 2020-06-12T02:39:19.853 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.853 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.853 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:19.853 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.893229+0000", 2020-06-12T02:39:19.853 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3192000000000003e-05 2020-06-12T02:39:19.854 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.854 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.854 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:19.854 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.893272+0000", 2020-06-12T02:39:19.854 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6910000000000006e-05 2020-06-12T02:39:19.855 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.855 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.855 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:19.855 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.893359+0000", 2020-06-12T02:39:19.855 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000519824 2020-06-12T02:39:19.855 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.856 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.856 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:19.856 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.893879+0000", 2020-06-12T02:39:19.856 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0031381719999999998 2020-06-12T02:39:19.856 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.857 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.857 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:19.857 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.897017+0000", 2020-06-12T02:39:19.857 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034756033999999998 2020-06-12T02:39:19.857 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.858 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.858 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:39:19.858 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.931773+0000", 2020-06-12T02:39:19.858 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000181662 2020-06-12T02:39:19.858 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.858 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.859 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:19.859 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.931955+0000", 2020-06-12T02:39:19.859 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.038906464000000002 2020-06-12T02:39:19.860 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.860 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:19.860 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.860 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.860 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.860 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14913 3.5s1 3:a446366d:::smithi10729407-11:head [append 0~532480] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:39:19.861 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.896342+0000", 2020-06-12T02:39:19.861 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 47.919072083000003, 2020-06-12T02:39:19.861 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.045786648999999999, 2020-06-12T02:39:19.861 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:19.861 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:19.862 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:19.862 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:19.862 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:19.862 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14913 2020-06-12T02:39:19.862 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.863 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:19.863 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.863 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:19.863 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.896342+0000", 2020-06-12T02:39:19.863 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:19.863 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.864 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.864 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:19.864 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.896342+0000", 2020-06-12T02:39:19.864 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000140648 2020-06-12T02:39:19.864 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.865 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.865 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:19.865 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.896482+0000", 2020-06-12T02:39:19.865 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5550000000000002e-06 2020-06-12T02:39:19.865 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.865 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.866 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:19.866 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.896488+0000", 2020-06-12T02:39:19.866 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3759999999999997e-06 2020-06-12T02:39:19.866 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.866 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.867 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:19.867 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.896497+0000", 2020-06-12T02:39:19.867 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6890000000000001e-06 2020-06-12T02:39:19.867 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.867 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.867 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:19.868 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.896503+0000", 2020-06-12T02:39:19.868 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0099999999999999e-05 2020-06-12T02:39:19.868 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.868 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.868 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:19.869 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.896543+0000", 2020-06-12T02:39:19.869 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109474 2020-06-12T02:39:19.869 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.869 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.869 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:19.870 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.896652+0000", 2020-06-12T02:39:19.870 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.036111747 2020-06-12T02:39:19.870 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.870 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.870 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:19.871 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.932764+0000", 2020-06-12T02:39:19.871 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7319999999999998e-05 2020-06-12T02:39:19.871 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.871 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.871 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:19.871 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.932842+0000", 2020-06-12T02:39:19.872 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00062652699999999999 2020-06-12T02:39:19.872 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.872 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.872 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:19.872 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.933468+0000", 2020-06-12T02:39:19.873 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0084385180000000008 2020-06-12T02:39:19.873 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.873 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.873 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:19.873 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.941907+0000", 2020-06-12T02:39:19.873 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012994699999999999 2020-06-12T02:39:19.874 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.874 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.874 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:39:19.874 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.942037+0000", 2020-06-12T02:39:19.874 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1748000000000003e-05 2020-06-12T02:39:19.875 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.875 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.875 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:19.875 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.942128+0000", 2020-06-12T02:39:19.875 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.045786648999999999 2020-06-12T02:39:19.875 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.876 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:19.876 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.876 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.876 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.876 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14914 3.5s1 3:a446366d:::smithi10729407-11:head [append 0~671744] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:39:19.877 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.900206+0000", 2020-06-12T02:39:19.877 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 47.915207727999999, 2020-06-12T02:39:19.877 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046728764999999998, 2020-06-12T02:39:19.877 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:19.877 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:19.878 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:19.878 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:19.878 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:19.878 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14914 2020-06-12T02:39:19.878 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.878 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:19.879 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.879 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:19.879 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900206+0000", 2020-06-12T02:39:19.879 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:19.880 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.880 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.880 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:19.880 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900206+0000", 2020-06-12T02:39:19.880 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000241518 2020-06-12T02:39:19.881 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.881 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.881 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:19.881 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900447+0000", 2020-06-12T02:39:19.881 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0219e-05 2020-06-12T02:39:19.882 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.882 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.882 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:19.882 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900458+0000", 2020-06-12T02:39:19.882 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7387000000000001e-05 2020-06-12T02:39:19.882 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.883 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:19.883 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900475+0000", 2020-06-12T02:39:19.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.661e-06 2020-06-12T02:39:19.883 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.884 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:19.884 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900483+0000", 2020-06-12T02:39:19.884 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4412000000000002e-05 2020-06-12T02:39:19.884 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.885 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:19.885 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900517+0000", 2020-06-12T02:39:19.885 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7199000000000005e-05 2020-06-12T02:39:19.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.885 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.886 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:19.886 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900584+0000", 2020-06-12T02:39:19.886 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.035314096000000003 2020-06-12T02:39:19.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.886 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.886 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:19.887 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.935898+0000", 2020-06-12T02:39:19.887 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000117197 2020-06-12T02:39:19.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:19.888 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.936016+0000", 2020-06-12T02:39:19.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012090289999999999 2020-06-12T02:39:19.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.888 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:19.888 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.937225+0000", 2020-06-12T02:39:19.889 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0095344799999999997 2020-06-12T02:39:19.889 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.889 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.889 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:19.889 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.946759+0000", 2020-06-12T02:39:19.890 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4289999999999993e-05 2020-06-12T02:39:19.890 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.890 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.890 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:39:19.890 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.946853+0000", 2020-06-12T02:39:19.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1277000000000001e-05 2020-06-12T02:39:19.891 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.891 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.891 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:19.891 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.946935+0000", 2020-06-12T02:39:19.892 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.046728764999999998 2020-06-12T02:39:19.892 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.892 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:19.892 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.892 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.892 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.893 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14915 3.5s1 3:a446366d:::smithi10729407-11:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:39:19.893 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.900556+0000", 2020-06-12T02:39:19.893 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 47.914857965000003, 2020-06-12T02:39:19.893 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.047233896999999997, 2020-06-12T02:39:19.893 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:19.894 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:19.894 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:19.894 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:19.894 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:19.894 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14915 2020-06-12T02:39:19.895 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.895 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:19.895 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.895 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:19.895 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900556+0000", 2020-06-12T02:39:19.895 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:19.896 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.896 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.896 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:19.896 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900556+0000", 2020-06-12T02:39:19.896 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3131000000000001e-05 2020-06-12T02:39:19.897 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.897 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.897 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:19.897 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900639+0000", 2020-06-12T02:39:19.897 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5459999999999992e-06 2020-06-12T02:39:19.898 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.898 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.898 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:19.898 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900647+0000", 2020-06-12T02:39:19.899 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5846e-05 2020-06-12T02:39:19.899 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.899 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.899 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:19.899 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900663+0000", 2020-06-12T02:39:19.900 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6760000000000001e-06 2020-06-12T02:39:19.900 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.900 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.900 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:19.900 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900670+0000", 2020-06-12T02:39:19.900 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2813e-05 2020-06-12T02:39:19.901 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.901 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.901 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:19.901 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900693+0000", 2020-06-12T02:39:19.901 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8223999999999998e-05 2020-06-12T02:39:19.902 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.902 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.902 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:19.902 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900741+0000", 2020-06-12T02:39:19.902 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.041466733999999998 2020-06-12T02:39:19.902 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.903 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.903 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:19.903 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.942208+0000", 2020-06-12T02:39:19.903 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010308 2020-06-12T02:39:19.903 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.904 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.904 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:19.904 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.942311+0000", 2020-06-12T02:39:19.904 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00032335099999999999 2020-06-12T02:39:19.904 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.905 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.905 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:19.905 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.942634+0000", 2020-06-12T02:39:19.905 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043047110000000001 2020-06-12T02:39:19.905 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.905 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.906 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:19.906 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.946939+0000", 2020-06-12T02:39:19.906 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6283999999999999e-05 2020-06-12T02:39:19.906 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.906 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.907 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:39:19.907 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.946985+0000", 2020-06-12T02:39:19.907 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000804501 2020-06-12T02:39:19.907 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.907 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.907 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:19.908 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.947790+0000", 2020-06-12T02:39:19.908 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.047233896999999997 2020-06-12T02:39:19.908 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.908 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:19.908 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.909 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.909 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.909 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:14916 3.5s1 3:a446366d:::smithi10729407-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e59)", 2020-06-12T02:39:19.909 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:31.900719+0000", 2020-06-12T02:39:19.909 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 47.914694384000001, 2020-06-12T02:39:19.910 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049202672000000003, 2020-06-12T02:39:19.910 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:19.910 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:39:19.910 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:19.910 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:19.910 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:19.911 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 14916 2020-06-12T02:39:19.911 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.911 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:19.911 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.911 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:19.912 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900719+0000", 2020-06-12T02:39:19.912 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:19.912 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.912 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.912 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:19.912 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900719+0000", 2020-06-12T02:39:19.913 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2898000000000002e-05 2020-06-12T02:39:19.913 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.913 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.913 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:19.913 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900752+0000", 2020-06-12T02:39:19.914 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8879999999999999e-06 2020-06-12T02:39:19.914 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.914 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.914 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:19.914 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900756+0000", 2020-06-12T02:39:19.914 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.011e-06 2020-06-12T02:39:19.915 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.915 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.915 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:19.915 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900764+0000", 2020-06-12T02:39:19.915 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2880000000000001e-06 2020-06-12T02:39:19.916 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.916 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.916 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:19.916 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900767+0000", 2020-06-12T02:39:19.916 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.145e-05 2020-06-12T02:39:19.917 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.917 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.917 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:19.917 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900789+0000", 2020-06-12T02:39:19.917 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7469000000000002e-05 2020-06-12T02:39:19.918 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.918 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.918 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:19.918 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.900836+0000", 2020-06-12T02:39:19.918 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.042561179999999997 2020-06-12T02:39:19.919 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.919 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.919 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:19.919 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.943397+0000", 2020-06-12T02:39:19.919 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000106377 2020-06-12T02:39:19.919 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.920 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.920 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:19.920 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.943504+0000", 2020-06-12T02:39:19.920 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043186759999999996 2020-06-12T02:39:19.920 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.921 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.921 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:19.921 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.947823+0000", 2020-06-12T02:39:19.921 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4439999999999997e-05 2020-06-12T02:39:19.921 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.922 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.922 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:19.922 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.947917+0000", 2020-06-12T02:39:19.922 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0020049949999999999 2020-06-12T02:39:19.922 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.923 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.923 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:19.923 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:31.949922+0000", 2020-06-12T02:39:19.923 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.049202672000000003 2020-06-12T02:39:19.923 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.923 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:19.924 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.924 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.924 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.924 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15220 3.7s0 3:e653d00f:::smithi10729407-18:head [append 0~532480 in=532480b] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:39:19.924 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.817582+0000", 2020-06-12T02:39:19.925 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 46.997831990999998, 2020-06-12T02:39:19.925 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.084332243000000001, 2020-06-12T02:39:19.925 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:19.925 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:19.925 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:19.926 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:19.926 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:19.926 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15220 2020-06-12T02:39:19.926 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.926 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:19.926 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.927 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:19.927 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.817582+0000", 2020-06-12T02:39:19.927 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:19.927 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.927 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.928 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:19.928 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.817582+0000", 2020-06-12T02:39:19.928 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014160799999999999 2020-06-12T02:39:19.928 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.928 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.928 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:19.929 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.817723+0000", 2020-06-12T02:39:19.929 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6889999999999993e-06 2020-06-12T02:39:19.929 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.929 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.929 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:19.930 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.817732+0000", 2020-06-12T02:39:19.930 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2214e-05 2020-06-12T02:39:19.930 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.930 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.930 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:19.930 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.817744+0000", 2020-06-12T02:39:19.931 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9570000000000002e-06 2020-06-12T02:39:19.931 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.931 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.931 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:19.932 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.817752+0000", 2020-06-12T02:39:19.932 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4947e-05 2020-06-12T02:39:19.932 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.932 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.932 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:19.933 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.817817+0000", 2020-06-12T02:39:19.933 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012504899999999999 2020-06-12T02:39:19.933 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.933 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.933 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:19.933 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.817942+0000", 2020-06-12T02:39:19.934 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00081927200000000001 2020-06-12T02:39:19.934 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.934 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.934 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:19.934 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.818761+0000", 2020-06-12T02:39:19.935 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.082918138000000002 2020-06-12T02:39:19.935 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.935 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.935 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:19.935 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901680+0000", 2020-06-12T02:39:19.936 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000150111 2020-06-12T02:39:19.936 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.936 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.936 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:39:19.936 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901830+0000", 2020-06-12T02:39:19.936 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4258000000000005e-05 2020-06-12T02:39:19.937 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.937 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.937 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:19.937 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901914+0000", 2020-06-12T02:39:19.937 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.084332243000000001 2020-06-12T02:39:19.938 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.938 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:19.938 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.938 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.938 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.938 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15221 3.7s0 3:e653d00f:::smithi10729407-18:head [append 0~417792 in=417792b] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:39:19.939 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.819933+0000", 2020-06-12T02:39:19.939 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 46.995480426999997, 2020-06-12T02:39:19.939 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091280863000000004, 2020-06-12T02:39:19.939 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:19.940 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:19.940 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:19.940 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:19.940 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:19.941 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15221 2020-06-12T02:39:19.941 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.941 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:19.941 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.941 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:19.941 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.819933+0000", 2020-06-12T02:39:19.942 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:19.942 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.942 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.942 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:19.942 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.819933+0000", 2020-06-12T02:39:19.943 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000302949 2020-06-12T02:39:19.943 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.943 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.943 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:19.943 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.820236+0000", 2020-06-12T02:39:19.943 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0984e-05 2020-06-12T02:39:19.944 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.944 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.944 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:19.944 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.820247+0000", 2020-06-12T02:39:19.944 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4894e-05 2020-06-12T02:39:19.945 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.945 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.945 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:19.945 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.820262+0000", 2020-06-12T02:39:19.945 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2409999999999998e-06 2020-06-12T02:39:19.945 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.946 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.946 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:19.946 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.820269+0000", 2020-06-12T02:39:19.946 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0024275859999999998 2020-06-12T02:39:19.946 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.947 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.947 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:19.947 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822697+0000", 2020-06-12T02:39:19.947 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011567599999999999 2020-06-12T02:39:19.947 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.948 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.948 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:19.948 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822813+0000", 2020-06-12T02:39:19.948 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000483714 2020-06-12T02:39:19.948 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.949 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.949 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:19.949 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.823296+0000", 2020-06-12T02:39:19.949 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.087735321000000005 2020-06-12T02:39:19.950 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.950 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.950 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:19.950 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.911032+0000", 2020-06-12T02:39:19.950 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3463999999999998e-05 2020-06-12T02:39:19.950 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.951 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.951 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:39:19.951 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.911125+0000", 2020-06-12T02:39:19.951 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9034000000000002e-05 2020-06-12T02:39:19.951 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.952 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.952 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:19.952 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.911214+0000", 2020-06-12T02:39:19.952 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091280863000000004 2020-06-12T02:39:19.952 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.952 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:19.953 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.953 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.953 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.953 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 59/37 ECSubWriteReply(tid=7875, last_complete=59'1325, committed=1, applied=1))", 2020-06-12T02:39:19.953 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.822130+0000", 2020-06-12T02:39:19.954 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 46.993283198, 2020-06-12T02:39:19.954 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.079550689999999993, 2020-06-12T02:39:19.954 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:19.954 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:19.954 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:19.955 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.955 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:19.955 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822130+0000", 2020-06-12T02:39:19.955 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:19.955 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.956 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.956 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:19.956 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822130+0000", 2020-06-12T02:39:19.956 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8309e-05 2020-06-12T02:39:19.956 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.957 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.957 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:19.957 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822189+0000", 2020-06-12T02:39:19.957 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6299999999999993e-06 2020-06-12T02:39:19.957 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.957 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.958 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:19.958 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822198+0000", 2020-06-12T02:39:19.958 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4632e-05 2020-06-12T02:39:19.958 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.959 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.959 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:19.959 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822213+0000", 2020-06-12T02:39:19.959 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8600000000000004e-06 2020-06-12T02:39:19.960 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.960 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.960 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:19.960 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822220+0000", 2020-06-12T02:39:19.960 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.079445632000000002 2020-06-12T02:39:19.961 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.961 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.961 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:19.961 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901666+0000", 2020-06-12T02:39:19.961 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5627000000000001e-05 2020-06-12T02:39:19.961 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.962 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.962 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:19.962 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901681+0000", 2020-06-12T02:39:19.962 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.079550689999999993 2020-06-12T02:39:19.962 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.963 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:19.963 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.963 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.963 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.964 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 59/37 ECSubWriteReply(tid=7875, last_complete=59'1325, committed=1, applied=1))", 2020-06-12T02:39:19.964 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.822633+0000", 2020-06-12T02:39:19.964 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 46.992780846999999, 2020-06-12T02:39:19.964 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.079294854999999997, 2020-06-12T02:39:19.964 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:19.964 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:19.965 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:19.965 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.965 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:19.965 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822633+0000", 2020-06-12T02:39:19.965 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:19.966 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.966 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.966 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:19.966 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822633+0000", 2020-06-12T02:39:19.967 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9357000000000001e-05 2020-06-12T02:39:19.967 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.967 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.967 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:19.967 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822692+0000", 2020-06-12T02:39:19.967 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5350000000000004e-06 2020-06-12T02:39:19.968 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.968 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.968 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:19.968 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822701+0000", 2020-06-12T02:39:19.968 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.243e-05 2020-06-12T02:39:19.969 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.969 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.969 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:19.969 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822713+0000", 2020-06-12T02:39:19.969 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.857e-06 2020-06-12T02:39:19.969 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.970 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.970 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:19.970 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822719+0000", 2020-06-12T02:39:19.970 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.079006642000000002 2020-06-12T02:39:19.970 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.971 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.971 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:19.971 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901726+0000", 2020-06-12T02:39:19.971 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000202034 2020-06-12T02:39:19.971 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.972 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.972 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:19.972 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901928+0000", 2020-06-12T02:39:19.972 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.079294854999999997 2020-06-12T02:39:19.973 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.973 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:19.973 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.973 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.973 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.974 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15222 3.7s0 3:e653d00f:::smithi10729407-18:head [append 0~524288] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:39:19.974 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.822885+0000", 2020-06-12T02:39:19.974 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 46.992528401000001, 2020-06-12T02:39:19.974 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.088458414999999999, 2020-06-12T02:39:19.974 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:19.975 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:19.975 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:19.975 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:19.975 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:19.975 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15222 2020-06-12T02:39:19.975 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.976 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:19.976 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.976 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:19.976 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822885+0000", 2020-06-12T02:39:19.976 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:19.976 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.977 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.977 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:19.977 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.822885+0000", 2020-06-12T02:39:19.977 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019930499999999999 2020-06-12T02:39:19.977 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.978 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.978 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:19.978 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.823085+0000", 2020-06-12T02:39:19.978 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4039999999999995e-06 2020-06-12T02:39:19.978 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.979 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.979 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:19.979 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.823093+0000", 2020-06-12T02:39:19.979 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5169e-05 2020-06-12T02:39:19.979 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.980 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.980 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:19.980 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.823108+0000", 2020-06-12T02:39:19.981 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6379999999999998e-06 2020-06-12T02:39:19.981 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.981 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.981 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:19.981 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.823115+0000", 2020-06-12T02:39:19.982 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.078878105000000004 2020-06-12T02:39:19.982 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.982 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.982 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:19.982 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.901993+0000", 2020-06-12T02:39:19.983 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000116167 2020-06-12T02:39:19.983 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.983 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.983 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:19.983 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.902109+0000", 2020-06-12T02:39:19.983 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00071338200000000004 2020-06-12T02:39:19.984 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.984 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.984 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:19.984 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.902822+0000", 2020-06-12T02:39:19.984 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0083967999999999994 2020-06-12T02:39:19.985 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.985 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.985 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:19.985 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.911219+0000", 2020-06-12T02:39:19.985 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8220000000000002e-05 2020-06-12T02:39:19.985 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.986 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.986 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:39:19.986 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.911267+0000", 2020-06-12T02:39:19.986 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6224999999999996e-05 2020-06-12T02:39:19.986 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.987 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.987 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:19.987 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.911344+0000", 2020-06-12T02:39:19.987 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.088458414999999999 2020-06-12T02:39:19.987 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.987 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:19.988 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:19.988 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.988 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.988 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15223 3.7s0 3:e653d00f:::smithi10729407-18:head [append 0~262144] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:39:19.988 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.824390+0000", 2020-06-12T02:39:19.989 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 46.991023230000003, 2020-06-12T02:39:19.989 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.088985642000000004, 2020-06-12T02:39:19.989 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:19.989 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:19.989 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:19.990 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:19.990 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:19.990 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15223 2020-06-12T02:39:19.990 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.990 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:19.990 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.991 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:19.991 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824390+0000", 2020-06-12T02:39:19.991 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:19.991 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.991 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.991 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:19.992 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824390+0000", 2020-06-12T02:39:19.992 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014672400000000001 2020-06-12T02:39:19.992 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.992 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.992 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:19.993 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824537+0000", 2020-06-12T02:39:19.993 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2020000000000004e-06 2020-06-12T02:39:19.993 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.993 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.993 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:19.993 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824545+0000", 2020-06-12T02:39:19.994 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5106000000000001e-05 2020-06-12T02:39:19.994 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.994 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.994 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:19.994 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824560+0000", 2020-06-12T02:39:19.995 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2539999999999998e-06 2020-06-12T02:39:19.995 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.995 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.995 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:19.995 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824567+0000", 2020-06-12T02:39:19.995 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.082220249999999995 2020-06-12T02:39:19.996 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.996 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.996 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:19.996 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.906787+0000", 2020-06-12T02:39:19.996 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109362 2020-06-12T02:39:19.997 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.997 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.997 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:19.997 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.906896+0000", 2020-06-12T02:39:19.997 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00074103899999999998 2020-06-12T02:39:19.998 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.998 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.998 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:19.998 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.907637+0000", 2020-06-12T02:39:19.998 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0054187489999999996 2020-06-12T02:39:19.999 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:19.999 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:19.999 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:19.999 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.913056+0000", 2020-06-12T02:39:20.000 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00026505799999999998 2020-06-12T02:39:20.000 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.000 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.000 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:39:20.000 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.913321+0000", 2020-06-12T02:39:20.000 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4898000000000002e-05 2020-06-12T02:39:20.001 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.001 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.001 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:20.001 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.913376+0000", 2020-06-12T02:39:20.001 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.088985642000000004 2020-06-12T02:39:20.002 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:20.002 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:20.002 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:20.002 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.002 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.003 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15224 3.7s0 3:e653d00f:::smithi10729407-18:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:39:20.003 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.824617+0000", 2020-06-12T02:39:20.003 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 46.990796385000003, 2020-06-12T02:39:20.003 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.09133223, 2020-06-12T02:39:20.003 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:20.003 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:20.004 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:20.004 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:20.004 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:20.004 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15224 2020-06-12T02:39:20.004 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.005 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:20.005 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.005 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:20.005 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824617+0000", 2020-06-12T02:39:20.005 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:20.005 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.006 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.006 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:20.006 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824617+0000", 2020-06-12T02:39:20.006 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3214999999999996e-05 2020-06-12T02:39:20.006 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.007 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.007 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:20.007 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824691+0000", 2020-06-12T02:39:20.007 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4949999999999996e-06 2020-06-12T02:39:20.007 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.008 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.008 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:20.008 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824697+0000", 2020-06-12T02:39:20.008 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1935e-05 2020-06-12T02:39:20.008 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.009 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.009 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:20.009 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824709+0000", 2020-06-12T02:39:20.009 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1459999999999998e-06 2020-06-12T02:39:20.009 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.010 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.010 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:20.010 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824713+0000", 2020-06-12T02:39:20.010 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.086691905 2020-06-12T02:39:20.010 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.010 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.011 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:20.012 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.911405+0000", 2020-06-12T02:39:20.012 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3385000000000003e-05 2020-06-12T02:39:20.012 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.012 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.012 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:20.013 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.911478+0000", 2020-06-12T02:39:20.013 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00033631999999999998 2020-06-12T02:39:20.014 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.014 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.014 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:20.015 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.911815+0000", 2020-06-12T02:39:20.015 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0037284330000000002 2020-06-12T02:39:20.015 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.016 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.016 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:20.016 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.915543+0000", 2020-06-12T02:39:20.017 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035071100000000002 2020-06-12T02:39:20.017 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.017 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.018 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:39:20.018 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.915894+0000", 2020-06-12T02:39:20.018 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5685e-05 2020-06-12T02:39:20.019 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.019 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.019 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:20.020 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.915950+0000", 2020-06-12T02:39:20.020 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.09133223 2020-06-12T02:39:20.020 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:20.020 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:20.020 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:20.021 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.021 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.021 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:15225 3.7s0 3:e653d00f:::smithi10729407-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e59)", 2020-06-12T02:39:20.022 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:32.824754+0000", 2020-06-12T02:39:20.022 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 46.990659856000001, 2020-06-12T02:39:20.022 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093329925999999994, 2020-06-12T02:39:20.023 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:20.023 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:39:20.023 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:20.024 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:20.024 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:20.024 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15225 2020-06-12T02:39:20.025 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.025 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:20.025 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.025 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:20.026 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824754+0000", 2020-06-12T02:39:20.026 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:20.026 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.027 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.027 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:20.027 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824754+0000", 2020-06-12T02:39:20.028 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3890999999999997e-05 2020-06-12T02:39:20.028 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.028 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.029 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:20.029 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824798+0000", 2020-06-12T02:39:20.029 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6840000000000001e-06 2020-06-12T02:39:20.030 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.030 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.030 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:20.031 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824803+0000", 2020-06-12T02:39:20.031 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1881e-05 2020-06-12T02:39:20.031 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.032 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.033 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:20.033 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824815+0000", 2020-06-12T02:39:20.033 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1400000000000002e-06 2020-06-12T02:39:20.034 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.034 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.034 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:20.035 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.824819+0000", 2020-06-12T02:39:20.035 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.087909733000000004 2020-06-12T02:39:20.035 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.036 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.036 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:20.036 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.912729+0000", 2020-06-12T02:39:20.037 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2892999999999996e-05 2020-06-12T02:39:20.037 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.037 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.038 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:20.038 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.912792+0000", 2020-06-12T02:39:20.038 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0040055159999999998 2020-06-12T02:39:20.039 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.039 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.039 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:20.040 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.916798+0000", 2020-06-12T02:39:20.040 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9858000000000006e-05 2020-06-12T02:39:20.040 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.040 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.041 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:20.041 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.916887+0000", 2020-06-12T02:39:20.041 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00119633 2020-06-12T02:39:20.042 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.042 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.042 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:20.043 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:32.918084+0000", 2020-06-12T02:39:20.043 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093329925999999994 2020-06-12T02:39:20.043 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:20.044 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:20.044 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:20.044 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.045 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.045 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:16220 3.bs0 3:d09543d2:::smithi10729407-16:head [append 0~401408] snapc 0=[] ondisk+write+known_if_redirected e60)", 2020-06-12T02:39:20.045 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:36.011355+0000", 2020-06-12T02:39:20.046 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 43.804058216999998, 2020-06-12T02:39:20.046 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034124793, 2020-06-12T02:39:20.046 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:20.047 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:20.047 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:20.047 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:20.048 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:20.048 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 16220 2020-06-12T02:39:20.048 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.048 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:20.049 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.049 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:20.049 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.011355+0000", 2020-06-12T02:39:20.050 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:20.050 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.050 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.051 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:20.051 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.011355+0000", 2020-06-12T02:39:20.052 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012013 2020-06-12T02:39:20.052 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.052 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.053 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:20.053 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.011476+0000", 2020-06-12T02:39:20.053 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.49e-06 2020-06-12T02:39:20.054 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.054 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.054 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:20.055 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.011481+0000", 2020-06-12T02:39:20.056 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8449999999999998e-06 2020-06-12T02:39:20.056 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.056 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.056 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:20.056 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.011491+0000", 2020-06-12T02:39:20.057 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0640000000000001e-06 2020-06-12T02:39:20.057 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.057 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.057 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:20.057 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.011496+0000", 2020-06-12T02:39:20.058 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4215999999999999e-05 2020-06-12T02:39:20.058 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.058 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.058 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:20.058 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.011530+0000", 2020-06-12T02:39:20.059 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8854999999999998e-05 2020-06-12T02:39:20.059 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.059 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.059 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:20.059 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.011589+0000", 2020-06-12T02:39:20.059 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022149404000000001 2020-06-12T02:39:20.060 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.060 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.060 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:20.060 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033738+0000", 2020-06-12T02:39:20.060 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4819000000000002e-05 2020-06-12T02:39:20.061 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.061 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.061 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:20.061 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.033823+0000", 2020-06-12T02:39:20.061 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00062732599999999999 2020-06-12T02:39:20.061 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.062 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.062 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:20.062 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.034451+0000", 2020-06-12T02:39:20.062 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0092556410000000002 2020-06-12T02:39:20.062 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.063 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.063 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:20.063 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.043706+0000", 2020-06-12T02:39:20.063 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001695925 2020-06-12T02:39:20.063 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.064 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.064 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:39:20.064 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.045402+0000", 2020-06-12T02:39:20.064 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8077999999999993e-05 2020-06-12T02:39:20.064 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.064 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.065 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:20.065 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.045480+0000", 2020-06-12T02:39:20.065 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.034124793 2020-06-12T02:39:20.065 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:20.065 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:20.066 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:20.066 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.066 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.066 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:16221 3.bs0 3:d09543d2:::smithi10729407-16:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e60)", 2020-06-12T02:39:20.066 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:36.011530+0000", 2020-06-12T02:39:20.067 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 43.803884091999997, 2020-06-12T02:39:20.067 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.037974159, 2020-06-12T02:39:20.067 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:20.067 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:20.067 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:20.067 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:20.068 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:20.068 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 16221 2020-06-12T02:39:20.068 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.068 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:20.068 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.069 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:20.069 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.011530+0000", 2020-06-12T02:39:20.069 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:20.069 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.069 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.070 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:20.070 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.011530+0000", 2020-06-12T02:39:20.070 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.9193000000000002e-05 2020-06-12T02:39:20.070 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.070 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.070 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:20.071 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.011569+0000", 2020-06-12T02:39:20.071 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.473e-06 2020-06-12T02:39:20.071 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.071 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.071 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:20.072 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.011572+0000", 2020-06-12T02:39:20.072 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2190000000000001e-06 2020-06-12T02:39:20.072 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.072 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.072 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:20.072 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.011578+0000", 2020-06-12T02:39:20.073 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6510000000000001e-06 2020-06-12T02:39:20.073 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.073 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.073 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:20.073 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.011581+0000", 2020-06-12T02:39:20.074 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8065999999999998e-05 2020-06-12T02:39:20.074 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.074 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.074 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:20.074 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.011619+0000", 2020-06-12T02:39:20.074 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103662 2020-06-12T02:39:20.075 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.075 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.075 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:20.075 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.011723+0000", 2020-06-12T02:39:20.075 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.026879159 2020-06-12T02:39:20.076 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.076 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.076 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:20.076 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.038602+0000", 2020-06-12T02:39:20.076 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9561000000000004e-05 2020-06-12T02:39:20.076 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.077 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.077 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:20.077 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.038702+0000", 2020-06-12T02:39:20.077 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00032271099999999999 2020-06-12T02:39:20.077 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.078 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.078 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_started", 2020-06-12T02:39:20.078 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.039024+0000", 2020-06-12T02:39:20.078 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0048470220000000003 2020-06-12T02:39:20.078 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.079 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.079 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_committed", 2020-06-12T02:39:20.079 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.043871+0000", 2020-06-12T02:39:20.079 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0055483970000000001 2020-06-12T02:39:20.079 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.079 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.080 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2020-06-12T02:39:20.080 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.049420+0000", 2020-06-12T02:39:20.080 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4044999999999999e-05 2020-06-12T02:39:20.080 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.080 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.081 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:20.081 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.049504+0000", 2020-06-12T02:39:20.081 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.037974159 2020-06-12T02:39:20.081 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:20.082 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:20.082 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:20.082 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.082 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.082 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:16222 3.bs0 3:d09543d2:::smithi10729407-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e60)", 2020-06-12T02:39:20.082 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:38:36.011625+0000", 2020-06-12T02:39:20.083 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 43.803788593, 2020-06-12T02:39:20.083 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040910490000000001, 2020-06-12T02:39:20.083 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:20.083 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:39:20.083 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:20.084 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:20.084 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:20.084 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 16222 2020-06-12T02:39:20.084 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.084 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:20.085 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.085 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:20.085 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.011625+0000", 2020-06-12T02:39:20.086 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:20.086 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.086 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.086 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:20.086 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.011625+0000", 2020-06-12T02:39:20.086 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5862999999999997e-05 2020-06-12T02:39:20.087 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.087 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.087 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:20.087 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.011681+0000", 2020-06-12T02:39:20.087 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5590000000000002e-06 2020-06-12T02:39:20.088 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.088 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.088 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:20.088 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.011688+0000", 2020-06-12T02:39:20.088 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7836e-05 2020-06-12T02:39:20.088 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.089 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.089 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:20.089 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.011705+0000", 2020-06-12T02:39:20.089 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3040000000000001e-06 2020-06-12T02:39:20.089 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.090 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.090 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:20.090 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.011712+0000", 2020-06-12T02:39:20.090 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8677999999999999e-05 2020-06-12T02:39:20.090 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.090 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.091 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:20.091 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.011740+0000", 2020-06-12T02:39:20.091 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5043000000000001e-05 2020-06-12T02:39:20.091 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.091 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.092 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:20.092 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.011795+0000", 2020-06-12T02:39:20.092 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.028051098999999999 2020-06-12T02:39:20.092 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.092 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.092 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:20.093 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.039846+0000", 2020-06-12T02:39:20.093 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7810000000000001e-05 2020-06-12T02:39:20.093 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.093 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.093 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:20.094 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.039934+0000", 2020-06-12T02:39:20.094 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010413906000000001 2020-06-12T02:39:20.094 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.094 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.094 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:20.094 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.050348+0000", 2020-06-12T02:39:20.095 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000127952 2020-06-12T02:39:20.095 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.095 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.095 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:20.095 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.050476+0000", 2020-06-12T02:39:20.096 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0020594400000000001 2020-06-12T02:39:20.096 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.096 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.096 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:20.096 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:38:36.052536+0000", 2020-06-12T02:39:20.096 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.040910490000000001 2020-06-12T02:39:20.097 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:20.097 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:20.097 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:20.097 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.097 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.098 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 82/21 ECSubWriteReply(tid=10980, last_complete=82'1864, committed=1, applied=1))", 2020-06-12T02:39:20.098 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:39:10.611205+0000", 2020-06-12T02:39:20.098 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 9.2042082300000008, 2020-06-12T02:39:20.098 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.048961863000000001, 2020-06-12T02:39:20.098 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:20.098 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:20.099 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:20.099 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.099 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:20.099 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:10.611205+0000", 2020-06-12T02:39:20.099 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:20.100 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.100 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.100 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:20.100 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:10.611205+0000", 2020-06-12T02:39:20.100 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.7093e-05 2020-06-12T02:39:20.100 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.101 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.101 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:20.101 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:10.611233+0000", 2020-06-12T02:39:20.101 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9150000000000004e-06 2020-06-12T02:39:20.101 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.102 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.102 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:20.102 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:10.611237+0000", 2020-06-12T02:39:20.102 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1489999999999997e-06 2020-06-12T02:39:20.102 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.102 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.103 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:20.103 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:10.611245+0000", 2020-06-12T02:39:20.103 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.151e-06 2020-06-12T02:39:20.103 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.103 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.104 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:20.104 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:10.611248+0000", 2020-06-12T02:39:20.104 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7561999999999997e-05 2020-06-12T02:39:20.104 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.104 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.104 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:20.105 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:10.611285+0000", 2020-06-12T02:39:20.105 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.048881992999999999 2020-06-12T02:39:20.105 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.105 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.105 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:20.106 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:10.660167+0000", 2020-06-12T02:39:20.106 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.048961863000000001 2020-06-12T02:39:20.106 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:20.106 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:20.106 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:20.106 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.107 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.107 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:24254 3.bs0 3:d09543d2:::smithi10729407-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e87)", 2020-06-12T02:39:20.107 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:39:18.707934+0000", 2020-06-12T02:39:20.107 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 1.1074799310000001, 2020-06-12T02:39:20.107 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032758715000000001, 2020-06-12T02:39:20.108 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:20.108 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:39:20.108 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:20.108 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:20.108 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:20.109 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 24254 2020-06-12T02:39:20.109 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.109 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:20.109 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.110 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:20.110 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:18.707934+0000", 2020-06-12T02:39:20.110 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:20.110 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.110 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.110 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:20.111 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:18.707934+0000", 2020-06-12T02:39:20.111 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3965999999999999e-05 2020-06-12T02:39:20.111 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.111 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.111 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:20.112 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:18.707978+0000", 2020-06-12T02:39:20.112 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4529999999999999e-06 2020-06-12T02:39:20.112 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.112 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.112 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:20.112 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:18.707984+0000", 2020-06-12T02:39:20.113 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1691999999999999e-05 2020-06-12T02:39:20.113 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.113 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.113 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:20.113 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:18.707996+0000", 2020-06-12T02:39:20.114 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5279999999999997e-06 2020-06-12T02:39:20.114 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.114 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.114 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:20.114 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:18.708000+0000", 2020-06-12T02:39:20.114 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0035848970000000001 2020-06-12T02:39:20.115 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.115 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.115 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:20.115 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:18.711585+0000", 2020-06-12T02:39:20.115 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2578999999999997e-05 2020-06-12T02:39:20.116 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.116 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.116 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:20.116 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:18.711668+0000", 2020-06-12T02:39:20.116 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018406756 2020-06-12T02:39:20.116 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.117 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.117 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:20.117 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:18.730075+0000", 2020-06-12T02:39:20.117 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041761400000000002 2020-06-12T02:39:20.117 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.118 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.118 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:20.118 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:18.730492+0000", 2020-06-12T02:39:20.118 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0079498150000000007 2020-06-12T02:39:20.118 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.118 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.119 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:20.119 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:18.738442+0000", 2020-06-12T02:39:20.119 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6366999999999994e-05 2020-06-12T02:39:20.119 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.119 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.120 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:20.120 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:18.738538+0000", 2020-06-12T02:39:20.120 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002154048 2020-06-12T02:39:20.120 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:20.120 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:20.120 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2020-06-12T02:39:20.121 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:18.740692+0000", 2020-06-12T02:39:20.121 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.032758715000000001 2020-06-12T02:39:20.121 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:20.121 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:20.121 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:20.121 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:20.122 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:20.122 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:39:20.142 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.2.5.7 2020-06-12T02:39:20.143 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:39:20.121+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:39:20.143 INFO:tasks.rados.rados.0.smithi107.stdout:11555: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:20.144 INFO:tasks.rados.rados.0.smithi107.stdout:11555: finishing write tid 2 to smithi10729407-2 2020-06-12T02:39:20.144 INFO:tasks.rados.rados.0.smithi107.stdout:11552: done (2 left) 2020-06-12T02:39:20.144 INFO:tasks.rados.rados.0.smithi107.stdout:11556: done (1 left) 2020-06-12T02:39:20.144 INFO:tasks.rados.rados.0.smithi107.stdout:11557: delete oid 1 current snap is 0 2020-06-12T02:39:20.144 INFO:tasks.rados.rados.0.smithi107.stdout:11555: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:20.145 INFO:tasks.rados.rados.0.smithi107.stdout:11555: finishing write tid 5 to smithi10729407-2 2020-06-12T02:39:20.146 INFO:tasks.rados.rados.0.smithi107.stdout:11557: done (1 left) 2020-06-12T02:39:20.146 INFO:tasks.rados.rados.0.smithi107.stdout:11558: delete oid 4 current snap is 0 2020-06-12T02:39:20.146 INFO:tasks.rados.rados.0.smithi107.stdout:11558: done (1 left) 2020-06-12T02:39:20.146 INFO:tasks.rados.rados.0.smithi107.stdout:11559: delete oid 1 current snap is 0 2020-06-12T02:39:20.147 INFO:tasks.rados.rados.0.smithi107.stdout:11555: finishing write tid 6 to smithi10729407-2 2020-06-12T02:39:20.147 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 1091 (ObjNum 11554 snap 0 seq_num 11554) dirty exists 2020-06-12T02:39:20.147 INFO:tasks.rados.rados.0.smithi107.stdout:11555: left oid 2 (ObjNum 11554 snap 0 seq_num 11554) 2020-06-12T02:39:20.147 INFO:tasks.rados.rados.0.smithi107.stdout:11555: done (1 left) 2020-06-12T02:39:20.147 INFO:tasks.rados.rados.0.smithi107.stdout:11559: done (0 left) 2020-06-12T02:39:20.148 INFO:tasks.rados.rados.0.smithi107.stdout:11560: delete oid 15 current snap is 0 2020-06-12T02:39:20.148 INFO:tasks.rados.rados.0.smithi107.stdout:11560: done (0 left) 2020-06-12T02:39:20.148 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:39:20.148 INFO:tasks.rados.rados.0.smithi107.stdout:11561: seq_num 11560 ranges {0=409600,409600=794624,1204224=409600} 2020-06-12T02:39:20.148 INFO:tasks.rados.rados.0.smithi107.stdout:11561: writing smithi10729407-16 from 0 to 409600 tid 1 2020-06-12T02:39:20.149 INFO:tasks.rados.rados.0.smithi107.stdout:11561: writing smithi10729407-16 from 409600 to 1204224 tid 2 2020-06-12T02:39:20.149 INFO:tasks.rados.rados.0.smithi107.stdout:11561: writing smithi10729407-16 from 1204224 to 1613824 tid 3 2020-06-12T02:39:20.149 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:39:20.149 INFO:tasks.rados.rados.0.smithi107.stdout:11562: seq_num 11561 ranges {0=425984,425984=679936,1105920=475136,1581056=49152} 2020-06-12T02:39:20.149 INFO:tasks.rados.rados.0.smithi107.stdout:11562: writing smithi10729407-15 from 0 to 425984 tid 1 2020-06-12T02:39:20.150 INFO:tasks.rados.rados.0.smithi107.stdout:11562: writing smithi10729407-15 from 425984 to 1105920 tid 2 2020-06-12T02:39:20.150 INFO:tasks.rados.rados.0.smithi107.stdout:11562: writing smithi10729407-15 from 1105920 to 1581056 tid 3 2020-06-12T02:39:20.150 INFO:tasks.rados.rados.0.smithi107.stdout:11562: writing smithi10729407-15 from 1581056 to 1630208 tid 4 2020-06-12T02:39:20.150 INFO:tasks.rados.rados.0.smithi107.stdout:11563: delete oid 19 current snap is 0 2020-06-12T02:39:20.150 INFO:tasks.rados.rados.0.smithi107.stdout:11561: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:20.151 INFO:tasks.rados.rados.0.smithi107.stdout:11561: finishing write tid 2 to smithi10729407-16 2020-06-12T02:39:20.151 INFO:tasks.rados.rados.0.smithi107.stdout:11561: finishing write tid 3 to smithi10729407-16 2020-06-12T02:39:20.151 INFO:tasks.rados.rados.0.smithi107.stdout:11561: finishing write tid 5 to smithi10729407-16 2020-06-12T02:39:20.151 INFO:tasks.rados.rados.0.smithi107.stdout:11561: finishing write tid 6 to smithi10729407-16 2020-06-12T02:39:20.152 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 2027 (ObjNum 11560 snap 0 seq_num 11560) dirty exists 2020-06-12T02:39:20.152 INFO:tasks.rados.rados.0.smithi107.stdout:11561: left oid 16 (ObjNum 11560 snap 0 seq_num 11560) 2020-06-12T02:39:20.152 INFO:tasks.rados.rados.0.smithi107.stdout:11562: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:20.152 INFO:tasks.rados.rados.0.smithi107.stdout:11561: done (2 left) 2020-06-12T02:39:20.152 INFO:tasks.rados.rados.0.smithi107.stdout:11563: done (1 left) 2020-06-12T02:39:20.153 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:39:20.153 INFO:tasks.rados.rados.0.smithi107.stdout:11564: seq_num 11563 ranges {0=466944,466944=450560,917504=491520,1409024=262144} 2020-06-12T02:39:20.153 INFO:tasks.rados.rados.0.smithi107.stdout:11564: writing smithi10729407-9 from 0 to 466944 tid 1 2020-06-12T02:39:20.153 INFO:tasks.rados.rados.0.smithi107.stdout:11564: writing smithi10729407-9 from 466944 to 917504 tid 2 2020-06-12T02:39:20.154 INFO:tasks.rados.rados.0.smithi107.stdout:11564: writing smithi10729407-9 from 917504 to 1409024 tid 3 2020-06-12T02:39:20.155 INFO:tasks.rados.rados.0.smithi107.stdout:11564: writing smithi10729407-9 from 1409024 to 1671168 tid 4 2020-06-12T02:39:20.155 INFO:tasks.rados.rados.0.smithi107.stdout:11565: delete oid 10 current snap is 0 2020-06-12T02:39:20.155 INFO:tasks.rados.rados.0.smithi107.stdout:11562: finishing write tid 2 to smithi10729407-15 2020-06-12T02:39:20.156 INFO:tasks.rados.rados.0.smithi107.stdout:11562: finishing write tid 3 to smithi10729407-15 2020-06-12T02:39:20.156 INFO:tasks.rados.rados.0.smithi107.stdout:11562: finishing write tid 4 to smithi10729407-15 2020-06-12T02:39:20.156 INFO:tasks.rados.rados.0.smithi107.stdout:11562: finishing write tid 6 to smithi10729407-15 2020-06-12T02:39:20.156 INFO:tasks.rados.rados.0.smithi107.stdout:11564: finishing write tid 1 to smithi10729407-9 2020-06-12T02:39:20.156 INFO:tasks.rados.rados.0.smithi107.stdout:11562: finishing write tid 7 to smithi10729407-15 2020-06-12T02:39:20.157 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 2182 (ObjNum 11561 snap 0 seq_num 11561) dirty exists 2020-06-12T02:39:20.157 INFO:tasks.rados.rados.0.smithi107.stdout:11562: left oid 15 (ObjNum 11561 snap 0 seq_num 11561) 2020-06-12T02:39:20.157 INFO:tasks.rados.rados.0.smithi107.stdout:11562: done (2 left) 2020-06-12T02:39:20.157 INFO:tasks.rados.rados.0.smithi107.stdout:11565: done (1 left) 2020-06-12T02:39:20.157 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:39:20.158 INFO:tasks.rados.rados.0.smithi107.stdout:11566: seq_num 11565 ranges {3047424=499712,3547136=663552,4210688=458752,4669440=81920} 2020-06-12T02:39:20.158 INFO:tasks.rados.rados.0.smithi107.stdout:11566: writing smithi10729407-12 from 3047424 to 3547136 tid 1 2020-06-12T02:39:20.158 INFO:tasks.rados.rados.0.smithi107.stdout:11566: writing smithi10729407-12 from 3547136 to 4210688 tid 2 2020-06-12T02:39:20.158 INFO:tasks.rados.rados.0.smithi107.stdout:11566: writing smithi10729407-12 from 4210688 to 4669440 tid 3 2020-06-12T02:39:20.158 INFO:tasks.rados.rados.0.smithi107.stdout:11566: writing smithi10729407-12 from 4669440 to 4751360 tid 4 2020-06-12T02:39:20.159 INFO:tasks.rados.rados.0.smithi107.stdout:11567: delete oid 10 current snap is 0 2020-06-12T02:39:20.159 INFO:tasks.rados.rados.0.smithi107.stdout:11564: finishing write tid 2 to smithi10729407-9 2020-06-12T02:39:20.159 INFO:tasks.rados.rados.0.smithi107.stdout:11564: finishing write tid 3 to smithi10729407-9 2020-06-12T02:39:20.159 INFO:tasks.rados.rados.0.smithi107.stdout:11564: finishing write tid 4 to smithi10729407-9 2020-06-12T02:39:20.159 INFO:tasks.rados.rados.0.smithi107.stdout:11564: finishing write tid 6 to smithi10729407-9 2020-06-12T02:39:20.160 INFO:tasks.rados.rados.0.smithi107.stdout:11564: finishing write tid 7 to smithi10729407-9 2020-06-12T02:39:20.160 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1897 (ObjNum 11563 snap 0 seq_num 11563) dirty exists 2020-06-12T02:39:20.160 INFO:tasks.rados.rados.0.smithi107.stdout:11564: left oid 9 (ObjNum 11563 snap 0 seq_num 11563) 2020-06-12T02:39:20.160 INFO:tasks.rados.rados.0.smithi107.stdout:11566: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:20.160 INFO:tasks.rados.rados.0.smithi107.stdout:11564: done (2 left) 2020-06-12T02:39:20.161 INFO:tasks.rados.rados.0.smithi107.stdout:11567: done (1 left) 2020-06-12T02:39:20.161 INFO:tasks.rados.rados.0.smithi107.stdout:11568: delete oid 16 current snap is 0 2020-06-12T02:39:20.161 INFO:tasks.rados.rados.0.smithi107.stdout:11568: done (1 left) 2020-06-12T02:39:20.161 INFO:tasks.rados.rados.0.smithi107.stdout:11569: delete oid 5 current snap is 0 2020-06-12T02:39:20.161 INFO:tasks.rados.rados.0.smithi107.stdout:11566: finishing write tid 2 to smithi10729407-12 2020-06-12T02:39:20.161 INFO:tasks.rados.rados.0.smithi107.stdout:11566: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:20.162 INFO:tasks.rados.rados.0.smithi107.stdout:11566: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:20.162 INFO:tasks.rados.rados.0.smithi107.stdout:11569: done (1 left) 2020-06-12T02:39:20.162 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:39:20.162 INFO:tasks.rados.rados.0.smithi107.stdout:11570: seq_num 11569 ranges {1630208=565248,2195456=606208,2801664=434176,3235840=163840} 2020-06-12T02:39:20.162 INFO:tasks.rados.rados.0.smithi107.stdout:11570: writing smithi10729407-15 from 1630208 to 2195456 tid 1 2020-06-12T02:39:20.163 INFO:tasks.rados.rados.0.smithi107.stdout:11570: writing smithi10729407-15 from 2195456 to 2801664 tid 2 2020-06-12T02:39:20.163 INFO:tasks.rados.rados.0.smithi107.stdout:11570: writing smithi10729407-15 from 2801664 to 3235840 tid 3 2020-06-12T02:39:20.163 INFO:tasks.rados.rados.0.smithi107.stdout:11570: writing smithi10729407-15 from 3235840 to 3399680 tid 4 2020-06-12T02:39:20.163 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:39:20.163 INFO:tasks.rados.rados.0.smithi107.stdout:11571: seq_num 11570 ranges {0=581632,581632=491520,1073152=647168,1720320=65536} 2020-06-12T02:39:20.163 INFO:tasks.rados.rados.0.smithi107.stdout:11571: writing smithi10729407-7 from 0 to 581632 tid 1 2020-06-12T02:39:20.164 INFO:tasks.rados.rados.0.smithi107.stdout:11571: writing smithi10729407-7 from 581632 to 1073152 tid 2 2020-06-12T02:39:20.164 INFO:tasks.rados.rados.0.smithi107.stdout:11571: writing smithi10729407-7 from 1073152 to 1720320 tid 3 2020-06-12T02:39:20.164 INFO:tasks.rados.rados.0.smithi107.stdout:11571: writing smithi10729407-7 from 1720320 to 1785856 tid 4 2020-06-12T02:39:20.164 INFO:tasks.rados.rados.0.smithi107.stdout:11572: delete oid 3 current snap is 0 2020-06-12T02:39:20.164 INFO:tasks.rados.rados.0.smithi107.stdout:11566: finishing write tid 6 to smithi10729407-12 2020-06-12T02:39:20.165 INFO:tasks.rados.rados.0.smithi107.stdout:11566: finishing write tid 7 to smithi10729407-12 2020-06-12T02:39:20.165 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 2164 (ObjNum 11565 snap 0 seq_num 11565) dirty exists 2020-06-12T02:39:20.165 INFO:tasks.rados.rados.0.smithi107.stdout:11566: left oid 12 (ObjNum 11565 snap 0 seq_num 11565) 2020-06-12T02:39:20.165 INFO:tasks.rados.rados.0.smithi107.stdout:11570: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:20.165 INFO:tasks.rados.rados.0.smithi107.stdout:11570: finishing write tid 2 to smithi10729407-15 2020-06-12T02:39:20.165 INFO:tasks.rados.rados.0.smithi107.stdout:11570: finishing write tid 3 to smithi10729407-15 2020-06-12T02:39:20.166 INFO:tasks.rados.rados.0.smithi107.stdout:11570: finishing write tid 4 to smithi10729407-15 2020-06-12T02:39:20.166 INFO:tasks.rados.rados.0.smithi107.stdout:11570: finishing write tid 6 to smithi10729407-15 2020-06-12T02:39:20.166 INFO:tasks.rados.rados.0.smithi107.stdout:11566: done (3 left) 2020-06-12T02:39:20.166 INFO:tasks.rados.rados.0.smithi107.stdout:11572: done (2 left) 2020-06-12T02:39:20.166 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:39:20.167 INFO:tasks.rados.rados.0.smithi107.stdout:11573: seq_num 11572 ranges {0=614400,614400=704512,1318912=499712} 2020-06-12T02:39:20.167 INFO:tasks.rados.rados.0.smithi107.stdout:11573: writing smithi10729407-10 from 0 to 614400 tid 1 2020-06-12T02:39:20.167 INFO:tasks.rados.rados.0.smithi107.stdout:11573: writing smithi10729407-10 from 614400 to 1318912 tid 2 2020-06-12T02:39:20.167 INFO:tasks.rados.rados.0.smithi107.stdout:11573: writing smithi10729407-10 from 1318912 to 1818624 tid 3 2020-06-12T02:39:20.168 INFO:tasks.rados.rados.0.smithi107.stdout:11574: delete oid 2 current snap is 0 2020-06-12T02:39:20.168 INFO:tasks.rados.rados.0.smithi107.stdout:11570: finishing write tid 7 to smithi10729407-15 2020-06-12T02:39:20.168 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 2187 (ObjNum 11569 snap 0 seq_num 11569) dirty exists 2020-06-12T02:39:20.168 INFO:tasks.rados.rados.0.smithi107.stdout:11570: left oid 15 (ObjNum 11569 snap 0 seq_num 11569) 2020-06-12T02:39:20.168 INFO:tasks.rados.rados.0.smithi107.stdout:11571: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:20.169 INFO:tasks.rados.rados.0.smithi107.stdout:11571: finishing write tid 2 to smithi10729407-7 2020-06-12T02:39:20.169 INFO:tasks.rados.rados.0.smithi107.stdout:11571: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:20.169 INFO:tasks.rados.rados.0.smithi107.stdout:11571: finishing write tid 4 to smithi10729407-7 2020-06-12T02:39:20.170 INFO:tasks.rados.rados.0.smithi107.stdout:11571: finishing write tid 6 to smithi10729407-7 2020-06-12T02:39:20.170 INFO:tasks.rados.rados.0.smithi107.stdout:11573: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:20.170 INFO:tasks.rados.rados.0.smithi107.stdout:11571: finishing write tid 7 to smithi10729407-7 2020-06-12T02:39:20.170 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1863 (ObjNum 11570 snap 0 seq_num 11570) dirty exists 2020-06-12T02:39:20.171 INFO:tasks.rados.rados.0.smithi107.stdout:11571: left oid 7 (ObjNum 11570 snap 0 seq_num 11570) 2020-06-12T02:39:20.171 INFO:tasks.rados.rados.0.smithi107.stdout:11570: done (3 left) 2020-06-12T02:39:20.171 INFO:tasks.rados.rados.0.smithi107.stdout:11571: done (2 left) 2020-06-12T02:39:20.171 INFO:tasks.rados.rados.0.smithi107.stdout:11574: done (1 left) 2020-06-12T02:39:20.171 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:39:20.172 INFO:tasks.rados.rados.0.smithi107.stdout:11575: seq_num 11574 ranges {0=647168,647168=475136,1122304=622592,1744896=106496} 2020-06-12T02:39:20.172 INFO:tasks.rados.rados.0.smithi107.stdout:11575: writing smithi10729407-14 from 0 to 647168 tid 1 2020-06-12T02:39:20.172 INFO:tasks.rados.rados.0.smithi107.stdout:11575: writing smithi10729407-14 from 647168 to 1122304 tid 2 2020-06-12T02:39:20.172 INFO:tasks.rados.rados.0.smithi107.stdout:11575: writing smithi10729407-14 from 1122304 to 1744896 tid 3 2020-06-12T02:39:20.172 INFO:tasks.rados.rados.0.smithi107.stdout:11575: writing smithi10729407-14 from 1744896 to 1851392 tid 4 2020-06-12T02:39:20.173 INFO:tasks.rados.rados.0.smithi107.stdout:11576: delete oid 15 current snap is 0 2020-06-12T02:39:20.173 INFO:tasks.rados.rados.0.smithi107.stdout:11573: finishing write tid 2 to smithi10729407-10 2020-06-12T02:39:20.173 INFO:tasks.rados.rados.0.smithi107.stdout:11573: finishing write tid 3 to smithi10729407-10 2020-06-12T02:39:20.173 INFO:tasks.rados.rados.0.smithi107.stdout:11573: finishing write tid 5 to smithi10729407-10 2020-06-12T02:39:20.173 INFO:tasks.rados.rados.0.smithi107.stdout:11573: finishing write tid 6 to smithi10729407-10 2020-06-12T02:39:20.174 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 885 (ObjNum 11572 snap 0 seq_num 11572) dirty exists 2020-06-12T02:39:20.174 INFO:tasks.rados.rados.0.smithi107.stdout:11573: left oid 10 (ObjNum 11572 snap 0 seq_num 11572) 2020-06-12T02:39:20.174 INFO:tasks.rados.rados.0.smithi107.stdout:11575: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:20.174 INFO:tasks.rados.rados.0.smithi107.stdout:11573: done (2 left) 2020-06-12T02:39:20.175 INFO:tasks.rados.rados.0.smithi107.stdout:11576: done (1 left) 2020-06-12T02:39:20.175 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:39:20.175 INFO:tasks.rados.rados.0.smithi107.stdout:11577: seq_num 11576 ranges {0=679936,679936=647168,1327104=557056} 2020-06-12T02:39:20.175 INFO:tasks.rados.rados.0.smithi107.stdout:11577: writing smithi10729407-5 from 0 to 679936 tid 1 2020-06-12T02:39:20.175 INFO:tasks.rados.rados.0.smithi107.stdout:11577: writing smithi10729407-5 from 679936 to 1327104 tid 2 2020-06-12T02:39:20.176 INFO:tasks.rados.rados.0.smithi107.stdout:11577: writing smithi10729407-5 from 1327104 to 1884160 tid 3 2020-06-12T02:39:20.176 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:39:20.176 INFO:tasks.rados.rados.0.smithi107.stdout:11578: seq_num 11577 ranges {1785856=696320,2482176=532480,3014656=671744} 2020-06-12T02:39:20.176 INFO:tasks.rados.rados.0.smithi107.stdout:11578: writing smithi10729407-7 from 1785856 to 2482176 tid 1 2020-06-12T02:39:20.176 INFO:tasks.rados.rados.0.smithi107.stdout:11578: writing smithi10729407-7 from 2482176 to 3014656 tid 2 2020-06-12T02:39:20.177 INFO:tasks.rados.rados.0.smithi107.stdout:11578: writing smithi10729407-7 from 3014656 to 3686400 tid 3 2020-06-12T02:39:20.177 INFO:tasks.rados.rados.0.smithi107.stdout:11579: delete oid 9 current snap is 0 2020-06-12T02:39:20.177 INFO:tasks.rados.rados.0.smithi107.stdout:11575: finishing write tid 2 to smithi10729407-14 2020-06-12T02:39:20.177 INFO:tasks.rados.rados.0.smithi107.stdout:11575: finishing write tid 3 to smithi10729407-14 2020-06-12T02:39:20.177 INFO:tasks.rados.rados.0.smithi107.stdout:11575: finishing write tid 4 to smithi10729407-14 2020-06-12T02:39:20.178 INFO:tasks.rados.rados.0.smithi107.stdout:11575: finishing write tid 6 to smithi10729407-14 2020-06-12T02:39:20.178 INFO:tasks.rados.rados.0.smithi107.stdout:11575: finishing write tid 7 to smithi10729407-14 2020-06-12T02:39:20.178 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1795 (ObjNum 11574 snap 0 seq_num 11574) dirty exists 2020-06-12T02:39:20.178 INFO:tasks.rados.rados.0.smithi107.stdout:11575: left oid 14 (ObjNum 11574 snap 0 seq_num 11574) 2020-06-12T02:39:20.178 INFO:tasks.rados.rados.0.smithi107.stdout:11577: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:20.179 INFO:tasks.rados.rados.0.smithi107.stdout:11577: finishing write tid 2 to smithi10729407-5 2020-06-12T02:39:20.179 INFO:tasks.rados.rados.0.smithi107.stdout:11577: finishing write tid 3 to smithi10729407-5 2020-06-12T02:39:20.179 INFO:tasks.rados.rados.0.smithi107.stdout:11577: finishing write tid 5 to smithi10729407-5 2020-06-12T02:39:20.179 INFO:tasks.rados.rados.0.smithi107.stdout:11577: finishing write tid 6 to smithi10729407-5 2020-06-12T02:39:20.179 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 1106 (ObjNum 11576 snap 0 seq_num 11576) dirty exists 2020-06-12T02:39:20.180 INFO:tasks.rados.rados.0.smithi107.stdout:11577: left oid 5 (ObjNum 11576 snap 0 seq_num 11576) 2020-06-12T02:39:20.180 INFO:tasks.rados.rados.0.smithi107.stdout:11578: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:20.180 INFO:tasks.rados.rados.0.smithi107.stdout:11575: done (3 left) 2020-06-12T02:39:20.180 INFO:tasks.rados.rados.0.smithi107.stdout:11577: done (2 left) 2020-06-12T02:39:20.180 INFO:tasks.rados.rados.0.smithi107.stdout:11579: done (1 left) 2020-06-12T02:39:20.181 INFO:tasks.rados.rados.0.smithi107.stdout:11580: delete oid 1 current snap is 0 2020-06-12T02:39:20.181 INFO:tasks.rados.rados.0.smithi107.stdout:11580: done (1 left) 2020-06-12T02:39:20.181 INFO:tasks.rados.rados.0.smithi107.stdout:11581: delete oid 1 current snap is 0 2020-06-12T02:39:20.181 INFO:tasks.rados.rados.0.smithi107.stdout:11578: finishing write tid 2 to smithi10729407-7 2020-06-12T02:39:20.181 INFO:tasks.rados.rados.0.smithi107.stdout:11578: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:20.182 INFO:tasks.rados.rados.0.smithi107.stdout:11581: done (1 left) 2020-06-12T02:39:20.182 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:39:20.182 INFO:tasks.rados.rados.0.smithi107.stdout:11582: seq_num 11581 ranges {0=761856,761856=516096,1277952=688128} 2020-06-12T02:39:20.182 INFO:tasks.rados.rados.0.smithi107.stdout:11582: writing smithi10729407-8 from 0 to 761856 tid 1 2020-06-12T02:39:20.182 INFO:tasks.rados.rados.0.smithi107.stdout:11582: writing smithi10729407-8 from 761856 to 1277952 tid 2 2020-06-12T02:39:20.183 INFO:tasks.rados.rados.0.smithi107.stdout:11582: writing smithi10729407-8 from 1277952 to 1966080 tid 3 2020-06-12T02:39:20.183 INFO:tasks.rados.rados.0.smithi107.stdout:11583: delete oid 6 current snap is 0 2020-06-12T02:39:20.183 INFO:tasks.rados.rados.0.smithi107.stdout:11578: finishing write tid 5 to smithi10729407-7 2020-06-12T02:39:20.183 INFO:tasks.rados.rados.0.smithi107.stdout:11578: finishing write tid 6 to smithi10729407-7 2020-06-12T02:39:20.183 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1867 (ObjNum 11577 snap 0 seq_num 11577) dirty exists 2020-06-12T02:39:20.184 INFO:tasks.rados.rados.0.smithi107.stdout:11578: left oid 7 (ObjNum 11577 snap 0 seq_num 11577) 2020-06-12T02:39:20.184 INFO:tasks.rados.rados.0.smithi107.stdout:11582: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:20.184 INFO:tasks.rados.rados.0.smithi107.stdout:11578: done (2 left) 2020-06-12T02:39:20.184 INFO:tasks.rados.rados.0.smithi107.stdout:11583: done (1 left) 2020-06-12T02:39:20.184 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:39:20.185 INFO:tasks.rados.rados.0.smithi107.stdout:11584: seq_num 11583 ranges {0=802816,802816=688128,1490944=516096} 2020-06-12T02:39:20.185 INFO:tasks.rados.rados.0.smithi107.stdout:11584: writing smithi10729407-18 from 0 to 802816 tid 1 2020-06-12T02:39:20.185 INFO:tasks.rados.rados.0.smithi107.stdout:11584: writing smithi10729407-18 from 802816 to 1490944 tid 2 2020-06-12T02:39:20.185 INFO:tasks.rados.rados.0.smithi107.stdout:11584: writing smithi10729407-18 from 1490944 to 2007040 tid 3 2020-06-12T02:39:20.185 INFO:tasks.rados.rados.0.smithi107.stdout:11585: delete oid 4 current snap is 0 2020-06-12T02:39:20.185 INFO:tasks.rados.rados.0.smithi107.stdout:11582: finishing write tid 2 to smithi10729407-8 2020-06-12T02:39:20.186 INFO:tasks.rados.rados.0.smithi107.stdout:11582: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:20.186 INFO:tasks.rados.rados.0.smithi107.stdout:11582: finishing write tid 5 to smithi10729407-8 2020-06-12T02:39:20.186 INFO:tasks.rados.rados.0.smithi107.stdout:11582: finishing write tid 6 to smithi10729407-8 2020-06-12T02:39:20.186 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 2192 (ObjNum 11581 snap 0 seq_num 11581) dirty exists 2020-06-12T02:39:20.186 INFO:tasks.rados.rados.0.smithi107.stdout:11582: left oid 8 (ObjNum 11581 snap 0 seq_num 11581) 2020-06-12T02:39:20.187 INFO:tasks.rados.rados.0.smithi107.stdout:11582: done (2 left) 2020-06-12T02:39:20.187 INFO:tasks.rados.rados.0.smithi107.stdout:11585: done (1 left) 2020-06-12T02:39:20.187 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:39:20.187 INFO:tasks.rados.rados.0.smithi107.stdout:11586: seq_num 11585 ranges {0=434176,434176=450560,884736=761856,1646592=393216} 2020-06-12T02:39:20.187 INFO:tasks.rados.rados.0.smithi107.stdout:11586: writing smithi10729407-2 from 0 to 434176 tid 1 2020-06-12T02:39:20.188 INFO:tasks.rados.rados.0.smithi107.stdout:11586: writing smithi10729407-2 from 434176 to 884736 tid 2 2020-06-12T02:39:20.188 INFO:tasks.rados.rados.0.smithi107.stdout:11586: writing smithi10729407-2 from 884736 to 1646592 tid 3 2020-06-12T02:39:20.188 INFO:tasks.rados.rados.0.smithi107.stdout:11586: writing smithi10729407-2 from 1646592 to 2039808 tid 4 2020-06-12T02:39:20.188 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:39:20.188 INFO:tasks.rados.rados.0.smithi107.stdout:11587: seq_num 11586 ranges {1851392=450560,2301952=737280,3039232=565248,3604480=303104} 2020-06-12T02:39:20.189 INFO:tasks.rados.rados.0.smithi107.stdout:11587: writing smithi10729407-14 from 1851392 to 2301952 tid 1 2020-06-12T02:39:20.189 INFO:tasks.rados.rados.0.smithi107.stdout:11587: writing smithi10729407-14 from 2301952 to 3039232 tid 2 2020-06-12T02:39:20.189 INFO:tasks.rados.rados.0.smithi107.stdout:11587: writing smithi10729407-14 from 3039232 to 3604480 tid 3 2020-06-12T02:39:20.189 INFO:tasks.rados.rados.0.smithi107.stdout:11587: writing smithi10729407-14 from 3604480 to 3907584 tid 4 2020-06-12T02:39:20.190 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:39:20.190 INFO:tasks.rados.rados.0.smithi107.stdout:11588: seq_num 11587 ranges {0=466944,466944=671744,1138688=778240,1916928=155648} 2020-06-12T02:39:20.190 INFO:tasks.rados.rados.0.smithi107.stdout:11588: writing smithi10729407-4 from 0 to 466944 tid 1 2020-06-12T02:39:20.190 INFO:tasks.rados.rados.0.smithi107.stdout:11588: writing smithi10729407-4 from 466944 to 1138688 tid 2 2020-06-12T02:39:20.191 INFO:tasks.rados.rados.0.smithi107.stdout:11588: writing smithi10729407-4 from 1138688 to 1916928 tid 3 2020-06-12T02:39:20.191 INFO:tasks.rados.rados.0.smithi107.stdout:11588: writing smithi10729407-4 from 1916928 to 2072576 tid 4 2020-06-12T02:39:20.191 INFO:tasks.rados.rados.0.smithi107.stdout:11589: delete oid 19 current snap is 0 2020-06-12T02:39:20.192 INFO:tasks.rados.rados.0.smithi107.stdout:11584: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:20.192 INFO:tasks.rados.rados.0.smithi107.stdout:11584: finishing write tid 2 to smithi10729407-18 2020-06-12T02:39:20.192 INFO:tasks.rados.rados.0.smithi107.stdout:11584: finishing write tid 3 to smithi10729407-18 2020-06-12T02:39:20.192 INFO:tasks.rados.rados.0.smithi107.stdout:11584: finishing write tid 5 to smithi10729407-18 2020-06-12T02:39:20.192 INFO:tasks.rados.rados.0.smithi107.stdout:11584: finishing write tid 6 to smithi10729407-18 2020-06-12T02:39:20.193 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1928 (ObjNum 11583 snap 0 seq_num 11583) dirty exists 2020-06-12T02:39:20.193 INFO:tasks.rados.rados.0.smithi107.stdout:11584: left oid 18 (ObjNum 11583 snap 0 seq_num 11583) 2020-06-12T02:39:20.193 INFO:tasks.rados.rados.0.smithi107.stdout:11586: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:20.193 INFO:tasks.rados.rados.0.smithi107.stdout:11586: finishing write tid 2 to smithi10729407-2 2020-06-12T02:39:20.193 INFO:tasks.rados.rados.0.smithi107.stdout:11586: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:20.194 INFO:tasks.rados.rados.0.smithi107.stdout:11586: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:20.194 INFO:tasks.rados.rados.0.smithi107.stdout:11586: finishing write tid 6 to smithi10729407-2 2020-06-12T02:39:20.194 INFO:tasks.rados.rados.0.smithi107.stdout:11586: finishing write tid 7 to smithi10729407-2 2020-06-12T02:39:20.195 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 1097 (ObjNum 11585 snap 0 seq_num 11585) dirty exists 2020-06-12T02:39:20.195 INFO:tasks.rados.rados.0.smithi107.stdout:11586: left oid 2 (ObjNum 11585 snap 0 seq_num 11585) 2020-06-12T02:39:20.195 INFO:tasks.rados.rados.0.smithi107.stdout:11587: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:20.195 INFO:tasks.rados.rados.0.smithi107.stdout:11587: finishing write tid 2 to smithi10729407-14 2020-06-12T02:39:20.196 INFO:tasks.rados.rados.0.smithi107.stdout:11587: finishing write tid 3 to smithi10729407-14 2020-06-12T02:39:20.196 INFO:tasks.rados.rados.0.smithi107.stdout:11587: finishing write tid 4 to smithi10729407-14 2020-06-12T02:39:20.196 INFO:tasks.rados.rados.0.smithi107.stdout:11587: finishing write tid 6 to smithi10729407-14 2020-06-12T02:39:20.197 INFO:tasks.rados.rados.0.smithi107.stdout:11587: finishing write tid 7 to smithi10729407-14 2020-06-12T02:39:20.197 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1800 (ObjNum 11586 snap 0 seq_num 11586) dirty exists 2020-06-12T02:39:20.197 INFO:tasks.rados.rados.0.smithi107.stdout:11587: left oid 14 (ObjNum 11586 snap 0 seq_num 11586) 2020-06-12T02:39:20.198 INFO:tasks.rados.rados.0.smithi107.stdout:11588: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:20.198 INFO:tasks.rados.rados.0.smithi107.stdout:11584: done (4 left) 2020-06-12T02:39:20.198 INFO:tasks.rados.rados.0.smithi107.stdout:11586: done (3 left) 2020-06-12T02:39:20.199 INFO:tasks.rados.rados.0.smithi107.stdout:11587: done (2 left) 2020-06-12T02:39:20.199 INFO:tasks.rados.rados.0.smithi107.stdout:11589: done (1 left) 2020-06-12T02:39:20.200 INFO:tasks.rados.rados.0.smithi107.stdout:11590: delete oid 17 current snap is 0 2020-06-12T02:39:20.200 INFO:tasks.rados.rados.0.smithi107.stdout:11588: finishing write tid 2 to smithi10729407-4 2020-06-12T02:39:20.200 INFO:tasks.rados.rados.0.smithi107.stdout:11588: finishing write tid 3 to smithi10729407-4 2020-06-12T02:39:20.201 INFO:tasks.rados.rados.0.smithi107.stdout:11588: finishing write tid 4 to smithi10729407-4 2020-06-12T02:39:20.201 INFO:tasks.rados.rados.0.smithi107.stdout:11588: finishing write tid 6 to smithi10729407-4 2020-06-12T02:39:20.201 INFO:tasks.rados.rados.0.smithi107.stdout:11588: finishing write tid 7 to smithi10729407-4 2020-06-12T02:39:20.202 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 2156 (ObjNum 11587 snap 0 seq_num 11587) dirty exists 2020-06-12T02:39:20.202 INFO:tasks.rados.rados.0.smithi107.stdout:11588: left oid 4 (ObjNum 11587 snap 0 seq_num 11587) 2020-06-12T02:39:20.202 INFO:tasks.rados.rados.0.smithi107.stdout:11588: done (1 left) 2020-06-12T02:39:20.203 INFO:tasks.rados.rados.0.smithi107.stdout:11590: done (0 left) 2020-06-12T02:39:20.203 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:39:20.203 INFO:tasks.rados.rados.0.smithi107.stdout:11591: seq_num 11590 ranges {0=516096,516096=729088,1245184=548864,1794048=327680} 2020-06-12T02:39:20.204 INFO:tasks.rados.rados.0.smithi107.stdout:11591: writing smithi10729407-3 from 0 to 516096 tid 1 2020-06-12T02:39:20.204 INFO:tasks.rados.rados.0.smithi107.stdout:11591: writing smithi10729407-3 from 516096 to 1245184 tid 2 2020-06-12T02:39:20.204 INFO:tasks.rados.rados.0.smithi107.stdout:11591: writing smithi10729407-3 from 1245184 to 1794048 tid 3 2020-06-12T02:39:20.205 INFO:tasks.rados.rados.0.smithi107.stdout:11591: writing smithi10729407-3 from 1794048 to 2121728 tid 4 2020-06-12T02:39:20.205 INFO:tasks.rados.rados.0.smithi107.stdout:11592: delete oid 20 current snap is 0 2020-06-12T02:39:20.205 INFO:tasks.rados.rados.0.smithi107.stdout:11591: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:20.206 INFO:tasks.rados.rados.0.smithi107.stdout:11592: done (1 left) 2020-06-12T02:39:20.206 INFO:tasks.rados.rados.0.smithi107.stdout:11593: delete oid 5 current snap is 0 2020-06-12T02:39:20.206 INFO:tasks.rados.rados.0.smithi107.stdout:11593: done (1 left) 2020-06-12T02:39:20.206 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:39:20.206 INFO:tasks.rados.rados.0.smithi107.stdout:11594: seq_num 11593 ranges {0=565248,565248=778240,1343488=720896,2064384=106496} 2020-06-12T02:39:20.207 INFO:tasks.rados.rados.0.smithi107.stdout:11594: writing smithi10729407-11 from 0 to 565248 tid 1 2020-06-12T02:39:20.207 INFO:tasks.rados.rados.0.smithi107.stdout:11594: writing smithi10729407-11 from 565248 to 1343488 tid 2 2020-06-12T02:39:20.207 INFO:tasks.rados.rados.0.smithi107.stdout:11594: writing smithi10729407-11 from 1343488 to 2064384 tid 3 2020-06-12T02:39:20.207 INFO:tasks.rados.rados.0.smithi107.stdout:11594: writing smithi10729407-11 from 2064384 to 2170880 tid 4 2020-06-12T02:39:20.207 INFO:tasks.rados.rados.0.smithi107.stdout:11595: delete oid 13 current snap is 0 2020-06-12T02:39:20.208 INFO:tasks.rados.rados.0.smithi107.stdout:11591: finishing write tid 2 to smithi10729407-3 2020-06-12T02:39:20.208 INFO:tasks.rados.rados.0.smithi107.stdout:11591: finishing write tid 3 to smithi10729407-3 2020-06-12T02:39:20.209 INFO:tasks.rados.rados.0.smithi107.stdout:11591: finishing write tid 4 to smithi10729407-3 2020-06-12T02:39:20.209 INFO:tasks.rados.rados.0.smithi107.stdout:11591: finishing write tid 6 to smithi10729407-3 2020-06-12T02:39:20.209 INFO:tasks.rados.rados.0.smithi107.stdout:11591: finishing write tid 7 to smithi10729407-3 2020-06-12T02:39:20.210 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 2034 (ObjNum 11590 snap 0 seq_num 11590) dirty exists 2020-06-12T02:39:20.210 INFO:tasks.rados.rados.0.smithi107.stdout:11591: left oid 3 (ObjNum 11590 snap 0 seq_num 11590) 2020-06-12T02:39:20.210 INFO:tasks.rados.rados.0.smithi107.stdout:11594: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:20.210 INFO:tasks.rados.rados.0.smithi107.stdout:11591: done (2 left) 2020-06-12T02:39:20.210 INFO:tasks.rados.rados.0.smithi107.stdout:11595: done (1 left) 2020-06-12T02:39:20.210 INFO:tasks.rados.rados.0.smithi107.stdout:11596: delete oid 15 current snap is 0 2020-06-12T02:39:20.211 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-06-12T02:39:20.216 INFO:tasks.rados.rados.0.smithi107.stdout:11596: done (1 left) 2020-06-12T02:39:20.217 INFO:tasks.rados.rados.0.smithi107.stdout:11597: delete oid 13 current snap is 0 2020-06-12T02:39:20.217 INFO:tasks.rados.rados.0.smithi107.stdout:11597: done (1 left) 2020-06-12T02:39:20.217 INFO:tasks.rados.rados.0.smithi107.stdout:11598: delete oid 18 current snap is 0 2020-06-12T02:39:20.217 INFO:tasks.rados.rados.0.smithi107.stdout:11598: done (1 left) 2020-06-12T02:39:20.218 INFO:tasks.rados.rados.0.smithi107.stdout:11599: delete oid 7 current snap is 0 2020-06-12T02:39:20.218 INFO:tasks.rados.rados.0.smithi107.stdout:11594: finishing write tid 2 to smithi10729407-11 2020-06-12T02:39:20.218 INFO:tasks.rados.rados.0.smithi107.stdout:11594: finishing write tid 3 to smithi10729407-11 2020-06-12T02:39:20.218 INFO:tasks.rados.rados.0.smithi107.stdout:11594: finishing write tid 4 to smithi10729407-11 2020-06-12T02:39:20.218 INFO:tasks.rados.rados.0.smithi107.stdout:11594: finishing write tid 6 to smithi10729407-11 2020-06-12T02:39:20.219 INFO:tasks.rados.rados.0.smithi107.stdout:11599: done (1 left) 2020-06-12T02:39:20.219 INFO:tasks.rados.rados.0.smithi107.stdout:11600: delete oid 3 current snap is 0 2020-06-12T02:39:20.219 INFO:tasks.rados.rados.0.smithi107.stdout:11594: finishing write tid 7 to smithi10729407-11 2020-06-12T02:39:20.219 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1815 (ObjNum 11593 snap 0 seq_num 11593) dirty exists 2020-06-12T02:39:20.219 INFO:tasks.rados.rados.0.smithi107.stdout:11594: left oid 11 (ObjNum 11593 snap 0 seq_num 11593) 2020-06-12T02:39:20.220 INFO:tasks.rados.rados.0.smithi107.stdout:11594: done (1 left) 2020-06-12T02:39:20.220 INFO:tasks.rados.rados.0.smithi107.stdout:11600: done (0 left) 2020-06-12T02:39:20.220 INFO:tasks.rados.rados.0.smithi107.stdout:11601: delete oid 11 current snap is 0 2020-06-12T02:39:20.220 INFO:tasks.rados.rados.0.smithi107.stdout:11601: done (0 left) 2020-06-12T02:39:20.220 INFO:tasks.rados.rados.0.smithi107.stdout:11602: delete oid 6 current snap is 0 2020-06-12T02:39:20.220 INFO:tasks.rados.rados.0.smithi107.stdout:11602: done (0 left) 2020-06-12T02:39:20.221 INFO:tasks.rados.rados.0.smithi107.stdout:11603: delete oid 18 current snap is 0 2020-06-12T02:39:20.221 INFO:tasks.rados.rados.0.smithi107.stdout:11603: done (0 left) 2020-06-12T02:39:20.221 INFO:tasks.rados.rados.0.smithi107.stdout:11604: delete oid 12 current snap is 0 2020-06-12T02:39:20.221 INFO:tasks.rados.rados.0.smithi107.stdout:11604: done (0 left) 2020-06-12T02:39:20.221 INFO:tasks.rados.rados.0.smithi107.stdout:11605: delete oid 18 current snap is 0 2020-06-12T02:39:20.222 INFO:tasks.rados.rados.0.smithi107.stdout:11605: done (0 left) 2020-06-12T02:39:20.222 INFO:tasks.rados.rados.0.smithi107.stdout:11606: delete oid 4 current snap is 0 2020-06-12T02:39:20.222 INFO:tasks.rados.rados.0.smithi107.stdout:11606: done (0 left) 2020-06-12T02:39:20.222 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:39:20.222 INFO:tasks.rados.rados.0.smithi107.stdout:11607: seq_num 11606 ranges {0=786432,786432=573440,1359872=466944,1826816=565248} 2020-06-12T02:39:20.223 INFO:tasks.rados.rados.0.smithi107.stdout:11607: writing smithi10729407-16 from 0 to 786432 tid 1 2020-06-12T02:39:20.223 INFO:tasks.rados.rados.0.smithi107.stdout:11607: writing smithi10729407-16 from 786432 to 1359872 tid 2 2020-06-12T02:39:20.223 INFO:tasks.rados.rados.0.smithi107.stdout:11607: writing smithi10729407-16 from 1359872 to 1826816 tid 3 2020-06-12T02:39:20.223 INFO:tasks.rados.rados.0.smithi107.stdout:11607: writing smithi10729407-16 from 1826816 to 2392064 tid 4 2020-06-12T02:39:20.223 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:39:20.223 INFO:tasks.rados.rados.0.smithi107.stdout:11608: seq_num 11607 ranges {0=802816,802816=458752,1261568=679936,1941504=466944} 2020-06-12T02:39:20.224 INFO:tasks.rados.rados.0.smithi107.stdout:11608: writing smithi10729407-13 from 0 to 802816 tid 1 2020-06-12T02:39:20.224 INFO:tasks.rados.rados.0.smithi107.stdout:11608: writing smithi10729407-13 from 802816 to 1261568 tid 2 2020-06-12T02:39:20.224 INFO:tasks.rados.rados.0.smithi107.stdout:11608: writing smithi10729407-13 from 1261568 to 1941504 tid 3 2020-06-12T02:39:20.224 INFO:tasks.rados.rados.0.smithi107.stdout:11608: writing smithi10729407-13 from 1941504 to 2408448 tid 4 2020-06-12T02:39:20.224 INFO:tasks.rados.rados.0.smithi107.stdout:11609: delete oid 10 current snap is 0 2020-06-12T02:39:20.225 INFO:tasks.rados.rados.0.smithi107.stdout:11607: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:20.225 INFO:tasks.rados.rados.0.smithi107.stdout:11607: finishing write tid 2 to smithi10729407-16 2020-06-12T02:39:20.225 INFO:tasks.rados.rados.0.smithi107.stdout:11607: finishing write tid 3 to smithi10729407-16 2020-06-12T02:39:20.225 INFO:tasks.rados.rados.0.smithi107.stdout:11607: finishing write tid 4 to smithi10729407-16 2020-06-12T02:39:20.225 INFO:tasks.rados.rados.0.smithi107.stdout:11607: finishing write tid 6 to smithi10729407-16 2020-06-12T02:39:20.226 INFO:tasks.rados.rados.0.smithi107.stdout:11608: finishing write tid 1 to smithi10729407-13 2020-06-12T02:39:20.226 INFO:tasks.rados.rados.0.smithi107.stdout:11607: finishing write tid 7 to smithi10729407-16 2020-06-12T02:39:20.226 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 2040 (ObjNum 11606 snap 0 seq_num 11606) dirty exists 2020-06-12T02:39:20.226 INFO:tasks.rados.rados.0.smithi107.stdout:11607: left oid 16 (ObjNum 11606 snap 0 seq_num 11606) 2020-06-12T02:39:20.226 INFO:tasks.rados.rados.0.smithi107.stdout:11607: done (2 left) 2020-06-12T02:39:20.227 INFO:tasks.rados.rados.0.smithi107.stdout:11609: done (1 left) 2020-06-12T02:39:20.227 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:39:20.227 INFO:tasks.rados.rados.0.smithi107.stdout:11610: seq_num 11609 ranges {0=32768} 2020-06-12T02:39:20.227 INFO:tasks.rados.rados.0.smithi107.stdout:11610: writing smithi10729407-6 from 0 to 32768 tid 1 2020-06-12T02:39:20.227 INFO:tasks.rados.rados.0.smithi107.stdout:11611: delete oid 18 current snap is 0 2020-06-12T02:39:20.227 INFO:tasks.rados.rados.0.smithi107.stdout:11611: done (2 left) 2020-06-12T02:39:20.228 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:39:20.228 INFO:tasks.rados.rados.0.smithi107.stdout:11612: seq_num 11611 ranges {0=65536} 2020-06-12T02:39:20.228 INFO:tasks.rados.rados.0.smithi107.stdout:11612: writing smithi10729407-7 from 0 to 65536 tid 1 2020-06-12T02:39:20.228 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:39:20.228 INFO:tasks.rados.rados.0.smithi107.stdout:11613: seq_num 11612 ranges {0=81920} 2020-06-12T02:39:20.229 INFO:tasks.rados.rados.0.smithi107.stdout:11613: writing smithi10729407-18 from 0 to 81920 tid 1 2020-06-12T02:39:20.229 INFO:tasks.rados.rados.0.smithi107.stdout:11614: delete oid 14 current snap is 0 2020-06-12T02:39:20.229 INFO:tasks.rados.rados.0.smithi107.stdout:11610: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:20.229 INFO:tasks.rados.rados.0.smithi107.stdout:11610: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:20.229 INFO:tasks.rados.rados.0.smithi107.stdout:11610: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:20.230 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 2124 (ObjNum 11609 snap 0 seq_num 11609) dirty exists 2020-06-12T02:39:20.230 INFO:tasks.rados.rados.0.smithi107.stdout:11610: left oid 6 (ObjNum 11609 snap 0 seq_num 11609) 2020-06-12T02:39:20.230 INFO:tasks.rados.rados.0.smithi107.stdout:11613: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:20.230 INFO:tasks.rados.rados.0.smithi107.stdout:11612: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:20.230 INFO:tasks.rados.rados.0.smithi107.stdout:11612: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:20.231 INFO:tasks.rados.rados.0.smithi107.stdout:11610: done (4 left) 2020-06-12T02:39:20.231 INFO:tasks.rados.rados.0.smithi107.stdout:11614: done (3 left) 2020-06-12T02:39:20.231 INFO:tasks.rados.rados.0.smithi107.stdout:11615: delete oid 8 current snap is 0 2020-06-12T02:39:20.231 INFO:tasks.rados.rados.0.smithi107.stdout:11613: finishing write tid 3 to smithi10729407-18 2020-06-12T02:39:20.231 INFO:tasks.rados.rados.0.smithi107.stdout:11608: finishing write tid 2 to smithi10729407-13 2020-06-12T02:39:20.231 INFO:tasks.rados.rados.0.smithi107.stdout:11608: finishing write tid 3 to smithi10729407-13 2020-06-12T02:39:20.232 INFO:tasks.rados.rados.0.smithi107.stdout:11608: finishing write tid 4 to smithi10729407-13 2020-06-12T02:39:20.232 INFO:tasks.rados.rados.0.smithi107.stdout:11608: finishing write tid 6 to smithi10729407-13 2020-06-12T02:39:20.234 INFO:tasks.rados.rados.0.smithi107.stdout:11612: finishing write tid 4 to smithi10729407-7 2020-06-12T02:39:20.235 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1870 (ObjNum 11611 snap 0 seq_num 11611) dirty exists 2020-06-12T02:39:20.235 INFO:tasks.rados.rados.0.smithi107.stdout:11612: left oid 7 (ObjNum 11611 snap 0 seq_num 11611) 2020-06-12T02:39:20.235 INFO:tasks.rados.rados.0.smithi107.stdout:11613: finishing write tid 4 to smithi10729407-18 2020-06-12T02:39:20.236 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1934 (ObjNum 11612 snap 0 seq_num 11612) dirty exists 2020-06-12T02:39:20.236 INFO:tasks.rados.rados.0.smithi107.stdout:11613: left oid 18 (ObjNum 11612 snap 0 seq_num 11612) 2020-06-12T02:39:20.236 INFO:tasks.rados.rados.0.smithi107.stdout:11612: done (3 left) 2020-06-12T02:39:20.236 INFO:tasks.rados.rados.0.smithi107.stdout:11613: done (2 left) 2020-06-12T02:39:20.236 INFO:tasks.rados.rados.0.smithi107.stdout:11615: done (1 left) 2020-06-12T02:39:20.237 INFO:tasks.rados.rados.0.smithi107.stdout:11616: delete oid 9 current snap is 0 2020-06-12T02:39:20.237 INFO:tasks.rados.rados.0.smithi107.stdout:11608: finishing write tid 7 to smithi10729407-13 2020-06-12T02:39:20.237 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1833 (ObjNum 11607 snap 0 seq_num 11607) dirty exists 2020-06-12T02:39:20.237 INFO:tasks.rados.rados.0.smithi107.stdout:11608: left oid 13 (ObjNum 11607 snap 0 seq_num 11607) 2020-06-12T02:39:20.237 INFO:tasks.rados.rados.0.smithi107.stdout:11608: done (1 left) 2020-06-12T02:39:20.238 INFO:tasks.rados.rados.0.smithi107.stdout:11616: done (0 left) 2020-06-12T02:39:20.238 INFO:tasks.rados.rados.0.smithi107.stdout:11617: delete oid 13 current snap is 0 2020-06-12T02:39:20.239 INFO:tasks.rados.rados.0.smithi107.stdout:11617: done (0 left) 2020-06-12T02:39:20.239 INFO:tasks.rados.rados.0.smithi107.stdout:11618: delete oid 7 current snap is 0 2020-06-12T02:39:20.239 INFO:tasks.rados.rados.0.smithi107.stdout:11618: done (0 left) 2020-06-12T02:39:20.239 INFO:tasks.rados.rados.0.smithi107.stdout:11619: delete oid 4 current snap is 0 2020-06-12T02:39:20.239 INFO:tasks.rados.rados.0.smithi107.stdout:11619: done (0 left) 2020-06-12T02:39:20.240 INFO:tasks.rados.rados.0.smithi107.stdout:11620: delete oid 5 current snap is 0 2020-06-12T02:39:20.240 INFO:tasks.rados.rados.0.smithi107.stdout:11620: done (0 left) 2020-06-12T02:39:20.240 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:39:20.240 INFO:tasks.rados.rados.0.smithi107.stdout:11621: seq_num 11620 ranges {0=221184} 2020-06-12T02:39:20.240 INFO:tasks.rados.rados.0.smithi107.stdout:11621: writing smithi10729407-14 from 0 to 221184 tid 1 2020-06-12T02:39:20.240 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:39:20.241 INFO:tasks.rados.rados.0.smithi107.stdout:11622: seq_num 11621 ranges {0=237568} 2020-06-12T02:39:20.241 INFO:tasks.rados.rados.0.smithi107.stdout:11622: writing smithi10729407-10 from 0 to 237568 tid 1 2020-06-12T02:39:20.241 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:39:20.241 INFO:tasks.rados.rados.0.smithi107.stdout:11623: seq_num 11622 ranges {0=253952} 2020-06-12T02:39:20.241 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:39:20.230+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:39:20.243 INFO:tasks.rados.rados.0.smithi107.stdout:11623: writing smithi10729407-11 from 0 to 253952 tid 1 2020-06-12T02:39:20.243 INFO:tasks.rados.rados.0.smithi107.stdout:11624: delete oid 1 current snap is 0 2020-06-12T02:39:20.243 INFO:tasks.rados.rados.0.smithi107.stdout:11621: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:20.243 INFO:tasks.rados.rados.0.smithi107.stdout:11621: finishing write tid 3 to smithi10729407-14 2020-06-12T02:39:20.243 INFO:tasks.rados.rados.0.smithi107.stdout:11622: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:20.244 INFO:tasks.rados.rados.0.smithi107.stdout:11621: finishing write tid 4 to smithi10729407-14 2020-06-12T02:39:20.244 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1803 (ObjNum 11620 snap 0 seq_num 11620) dirty exists 2020-06-12T02:39:20.244 INFO:tasks.rados.rados.0.smithi107.stdout:11621: left oid 14 (ObjNum 11620 snap 0 seq_num 11620) 2020-06-12T02:39:20.245 INFO:tasks.rados.rados.0.smithi107.stdout:11623: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:20.245 INFO:tasks.rados.rados.0.smithi107.stdout:11622: finishing write tid 3 to smithi10729407-10 2020-06-12T02:39:20.245 INFO:tasks.rados.rados.0.smithi107.stdout:11623: finishing write tid 3 to smithi10729407-11 2020-06-12T02:39:20.245 INFO:tasks.rados.rados.0.smithi107.stdout:11621: done (3 left) 2020-06-12T02:39:20.246 INFO:tasks.rados.rados.0.smithi107.stdout:11624: done (2 left) 2020-06-12T02:39:20.246 INFO:tasks.rados.rados.0.smithi107.stdout:11625: delete oid 12 current snap is 0 2020-06-12T02:39:20.246 INFO:tasks.rados.rados.0.smithi107.stdout:11623: finishing write tid 4 to smithi10729407-11 2020-06-12T02:39:20.247 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1818 (ObjNum 11622 snap 0 seq_num 11622) dirty exists 2020-06-12T02:39:20.247 INFO:tasks.rados.rados.0.smithi107.stdout:11623: left oid 11 (ObjNum 11622 snap 0 seq_num 11622) 2020-06-12T02:39:20.247 INFO:tasks.rados.rados.0.smithi107.stdout:11622: finishing write tid 4 to smithi10729407-10 2020-06-12T02:39:20.248 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 888 (ObjNum 11621 snap 0 seq_num 11621) dirty exists 2020-06-12T02:39:20.248 INFO:tasks.rados.rados.0.smithi107.stdout:11622: left oid 10 (ObjNum 11621 snap 0 seq_num 11621) 2020-06-12T02:39:20.249 INFO:tasks.rados.rados.0.smithi107.stdout:11622: done (2 left) 2020-06-12T02:39:20.250 INFO:tasks.rados.rados.0.smithi107.stdout:11623: done (1 left) 2020-06-12T02:39:20.250 INFO:tasks.rados.rados.0.smithi107.stdout:11625: done (0 left) 2020-06-12T02:39:20.250 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:39:20.250 INFO:tasks.rados.rados.0.smithi107.stdout:11626: seq_num 11625 ranges {0=303104} 2020-06-12T02:39:20.251 INFO:tasks.rados.rados.0.smithi107.stdout:11626: writing smithi10729407-3 from 0 to 303104 tid 1 2020-06-12T02:39:20.251 INFO:tasks.rados.rados.0.smithi107.stdout:11627: delete oid 10 current snap is 0 2020-06-12T02:39:20.256 INFO:tasks.rados.rados.0.smithi107.stdout:11627: done (1 left) 2020-06-12T02:39:20.256 INFO:tasks.rados.rados.0.smithi107.stdout:11628: delete oid 1 current snap is 0 2020-06-12T02:39:20.257 INFO:tasks.rados.rados.0.smithi107.stdout:11626: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:20.258 INFO:tasks.rados.rados.0.smithi107.stdout:11626: finishing write tid 3 to smithi10729407-3 2020-06-12T02:39:20.259 INFO:tasks.rados.rados.0.smithi107.stdout:11628: done (1 left) 2020-06-12T02:39:20.259 INFO:tasks.rados.rados.0.smithi107.stdout:11629: delete oid 4 current snap is 0 2020-06-12T02:39:20.260 INFO:tasks.rados.rados.0.smithi107.stdout:11626: finishing write tid 4 to smithi10729407-3 2020-06-12T02:39:20.260 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 2042 (ObjNum 11625 snap 0 seq_num 11625) dirty exists 2020-06-12T02:39:20.260 INFO:tasks.rados.rados.0.smithi107.stdout:11626: left oid 3 (ObjNum 11625 snap 0 seq_num 11625) 2020-06-12T02:39:20.262 INFO:tasks.rados.rados.0.smithi107.stdout:11626: done (1 left) 2020-06-12T02:39:20.262 INFO:tasks.rados.rados.0.smithi107.stdout:11629: done (0 left) 2020-06-12T02:39:20.262 INFO:tasks.rados.rados.0.smithi107.stdout:11630: delete oid 10 current snap is 0 2020-06-12T02:39:20.264 INFO:tasks.rados.rados.0.smithi107.stdout:11630: done (0 left) 2020-06-12T02:39:20.265 INFO:tasks.rados.rados.0.smithi107.stdout:11631: delete oid 4 current snap is 0 2020-06-12T02:39:20.267 INFO:tasks.rados.rados.0.smithi107.stdout:11631: done (0 left) 2020-06-12T02:39:20.267 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:39:20.267 INFO:tasks.rados.rados.0.smithi107.stdout:11632: seq_num 11631 ranges {32768=401408} 2020-06-12T02:39:20.269 INFO:tasks.rados.rados.0.smithi107.stdout:11632: writing smithi10729407-6 from 32768 to 434176 tid 1 2020-06-12T02:39:20.270 INFO:tasks.rados.rados.0.smithi107.stdout:11633: delete oid 3 current snap is 0 2020-06-12T02:39:20.272 INFO:tasks.rados.rados.0.smithi107.stdout:11633: done (1 left) 2020-06-12T02:39:20.273 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:39:20.273 INFO:tasks.rados.rados.0.smithi107.stdout:11634: seq_num 11633 ranges {0=434176} 2020-06-12T02:39:20.275 INFO:tasks.rados.rados.0.smithi107.stdout:11634: writing smithi10729407-20 from 0 to 434176 tid 1 2020-06-12T02:39:20.275 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:39:20.276 INFO:tasks.rados.rados.0.smithi107.stdout:11635: seq_num 11634 ranges {81920=450560} 2020-06-12T02:39:20.279 INFO:tasks.rados.rados.0.smithi107.stdout:11635: writing smithi10729407-18 from 81920 to 532480 tid 1 2020-06-12T02:39:20.279 INFO:tasks.rados.rados.0.smithi107.stdout:11636: delete oid 5 current snap is 0 2020-06-12T02:39:20.280 INFO:tasks.rados.rados.0.smithi107.stdout:11632: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:20.280 INFO:tasks.rados.rados.0.smithi107.stdout:11632: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:20.280 INFO:tasks.rados.rados.0.smithi107.stdout:11634: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:20.282 INFO:tasks.rados.rados.0.smithi107.stdout:11636: done (3 left) 2020-06-12T02:39:20.283 INFO:tasks.rados.rados.0.smithi107.stdout:11637: delete oid 11 current snap is 0 2020-06-12T02:39:20.283 INFO:tasks.rados.rados.0.smithi107.stdout:11632: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:20.283 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 2128 (ObjNum 11631 snap 0 seq_num 11631) dirty exists 2020-06-12T02:39:20.283 INFO:tasks.rados.rados.0.smithi107.stdout:11632: left oid 6 (ObjNum 11631 snap 0 seq_num 11631) 2020-06-12T02:39:20.285 INFO:tasks.rados.rados.0.smithi107.stdout:11634: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:20.289 INFO:tasks.rados.rados.0.smithi107.stdout:11635: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:20.290 INFO:tasks.rados.rados.0.smithi107.stdout:11632: done (3 left) 2020-06-12T02:39:20.290 INFO:tasks.rados.rados.0.smithi107.stdout:11637: done (2 left) 2020-06-12T02:39:20.290 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:39:20.290 INFO:tasks.rados.rados.0.smithi107.stdout:11638: seq_num 11637 ranges {0=499712} 2020-06-12T02:39:20.291 INFO:tasks.rados.rados.0.smithi107.stdout:11638: writing smithi10729407-5 from 0 to 499712 tid 1 2020-06-12T02:39:20.291 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:39:20.291 INFO:tasks.rados.rados.0.smithi107.stdout:11639: seq_num 11638 ranges {0=516096} 2020-06-12T02:39:20.292 INFO:tasks.rados.rados.0.smithi107.stdout:11639: writing smithi10729407-1 from 0 to 516096 tid 1 2020-06-12T02:39:20.292 INFO:tasks.rados.rados.0.smithi107.stdout:11640: delete oid 7 current snap is 0 2020-06-12T02:39:20.293 INFO:tasks.rados.rados.0.smithi107.stdout:11635: finishing write tid 3 to smithi10729407-18 2020-06-12T02:39:20.294 INFO:tasks.rados.rados.0.smithi107.stdout:11634: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:20.294 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 900 (ObjNum 11633 snap 0 seq_num 11633) dirty exists 2020-06-12T02:39:20.294 INFO:tasks.rados.rados.0.smithi107.stdout:11634: left oid 20 (ObjNum 11633 snap 0 seq_num 11633) 2020-06-12T02:39:20.294 INFO:tasks.rados.rados.0.smithi107.stdout:11635: finishing write tid 4 to smithi10729407-18 2020-06-12T02:39:20.294 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1936 (ObjNum 11634 snap 0 seq_num 11634) dirty exists 2020-06-12T02:39:20.295 INFO:tasks.rados.rados.0.smithi107.stdout:11635: left oid 18 (ObjNum 11634 snap 0 seq_num 11634) 2020-06-12T02:39:20.296 INFO:tasks.rados.rados.0.smithi107.stdout:11638: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:20.296 INFO:tasks.rados.rados.0.smithi107.stdout:11634: done (4 left) 2020-06-12T02:39:20.296 INFO:tasks.rados.rados.0.smithi107.stdout:11635: done (3 left) 2020-06-12T02:39:20.296 INFO:tasks.rados.rados.0.smithi107.stdout:11640: done (2 left) 2020-06-12T02:39:20.297 INFO:tasks.rados.rados.0.smithi107.stdout:11641: delete oid 14 current snap is 0 2020-06-12T02:39:20.297 INFO:tasks.rados.rados.0.smithi107.stdout:11638: finishing write tid 3 to smithi10729407-5 2020-06-12T02:39:20.298 INFO:tasks.rados.rados.0.smithi107.stdout:11638: finishing write tid 4 to smithi10729407-5 2020-06-12T02:39:20.298 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 1111 (ObjNum 11637 snap 0 seq_num 11637) dirty exists 2020-06-12T02:39:20.298 INFO:tasks.rados.rados.0.smithi107.stdout:11638: left oid 5 (ObjNum 11637 snap 0 seq_num 11637) 2020-06-12T02:39:20.299 INFO:tasks.rados.rados.0.smithi107.stdout:11638: done (2 left) 2020-06-12T02:39:20.299 INFO:tasks.rados.rados.0.smithi107.stdout:11641: done (1 left) 2020-06-12T02:39:20.299 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:39:20.300 INFO:tasks.rados.rados.0.smithi107.stdout:11642: seq_num 11641 ranges {0=573440} 2020-06-12T02:39:20.302 INFO:tasks.rados.rados.0.smithi107.stdout:11642: writing smithi10729407-15 from 0 to 573440 tid 1 2020-06-12T02:39:20.302 INFO:tasks.rados.rados.0.smithi107.stdout:11643: delete oid 16 current snap is 0 2020-06-12T02:39:20.302 INFO:tasks.rados.rados.0.smithi107.stdout:11639: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:20.303 INFO:tasks.rados.rados.0.smithi107.stdout:11639: finishing write tid 3 to smithi10729407-1 2020-06-12T02:39:20.303 INFO:tasks.rados.rados.0.smithi107.stdout:11639: finishing write tid 4 to smithi10729407-1 2020-06-12T02:39:20.303 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 2130 (ObjNum 11638 snap 0 seq_num 11638) dirty exists 2020-06-12T02:39:20.304 INFO:tasks.rados.rados.0.smithi107.stdout:11639: left oid 1 (ObjNum 11638 snap 0 seq_num 11638) 2020-06-12T02:39:20.305 INFO:tasks.rados.rados.0.smithi107.stdout:11639: done (2 left) 2020-06-12T02:39:20.305 INFO:tasks.rados.rados.0.smithi107.stdout:11643: done (1 left) 2020-06-12T02:39:20.305 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:39:20.306 INFO:tasks.rados.rados.0.smithi107.stdout:11644: seq_num 11643 ranges {434176=606208} 2020-06-12T02:39:20.311 INFO:tasks.rados.rados.0.smithi107.stdout:11644: writing smithi10729407-20 from 434176 to 1040384 tid 1 2020-06-12T02:39:20.311 INFO:tasks.rados.rados.0.smithi107.stdout:11645: delete oid 14 current snap is 0 2020-06-12T02:39:20.313 INFO:tasks.rados.rados.0.smithi107.stdout:11642: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:20.314 INFO:tasks.rados.rados.0.smithi107.stdout:11645: done (2 left) 2020-06-12T02:39:20.314 INFO:tasks.rados.rados.0.smithi107.stdout:11646: delete oid 11 current snap is 0 2020-06-12T02:39:20.314 INFO:tasks.rados.rados.0.smithi107.stdout:11642: finishing write tid 3 to smithi10729407-15 2020-06-12T02:39:20.316 INFO:tasks.rados.rados.0.smithi107.stdout:11642: finishing write tid 4 to smithi10729407-15 2020-06-12T02:39:20.317 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 2196 (ObjNum 11641 snap 0 seq_num 11641) dirty exists 2020-06-12T02:39:20.317 INFO:tasks.rados.rados.0.smithi107.stdout:11642: left oid 15 (ObjNum 11641 snap 0 seq_num 11641) 2020-06-12T02:39:20.318 INFO:tasks.rados.rados.0.smithi107.stdout:11644: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:20.318 INFO:tasks.rados.rados.0.smithi107.stdout:11642: done (2 left) 2020-06-12T02:39:20.318 INFO:tasks.rados.rados.0.smithi107.stdout:11646: done (1 left) 2020-06-12T02:39:20.318 INFO:tasks.rados.rados.0.smithi107.stdout:11647: delete oid 11 current snap is 0 2020-06-12T02:39:20.319 INFO:tasks.rados.rados.0.smithi107.stdout:11644: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:20.319 INFO:tasks.rados.rados.0.smithi107.stdout:11644: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:20.319 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 902 (ObjNum 11643 snap 0 seq_num 11643) dirty exists 2020-06-12T02:39:20.319 INFO:tasks.rados.rados.0.smithi107.stdout:11644: left oid 20 (ObjNum 11643 snap 0 seq_num 11643) 2020-06-12T02:39:20.320 INFO:tasks.rados.rados.0.smithi107.stdout:11644: done (1 left) 2020-06-12T02:39:20.320 INFO:tasks.rados.rados.0.smithi107.stdout:11647: done (0 left) 2020-06-12T02:39:20.320 INFO:tasks.rados.rados.0.smithi107.stdout:11648: delete oid 10 current snap is 0 2020-06-12T02:39:20.323 INFO:tasks.rados.rados.0.smithi107.stdout:11648: done (0 left) 2020-06-12T02:39:20.323 INFO:tasks.rados.rados.0.smithi107.stdout:11649: delete oid 12 current snap is 0 2020-06-12T02:39:20.326 INFO:tasks.rados.rados.0.smithi107.stdout:11649: done (0 left) 2020-06-12T02:39:20.326 INFO:tasks.rados.rados.0.smithi107.stdout:11650: delete oid 15 current snap is 0 2020-06-12T02:39:20.329 INFO:tasks.rados.rados.0.smithi107.stdout:11650: done (0 left) 2020-06-12T02:39:20.329 INFO:tasks.rados.rados.0.smithi107.stdout:11651: delete oid 14 current snap is 0 2020-06-12T02:39:20.331 INFO:tasks.rados.rados.0.smithi107.stdout:11651: done (0 left) 2020-06-12T02:39:20.331 INFO:tasks.rados.rados.0.smithi107.stdout:11652: delete oid 9 current snap is 0 2020-06-12T02:39:20.333 INFO:tasks.rados.rados.0.smithi107.stdout:11652: done (0 left) 2020-06-12T02:39:20.333 INFO:tasks.rados.rados.0.smithi107.stdout:11653: delete oid 3 current snap is 0 2020-06-12T02:39:20.336 INFO:tasks.rados.rados.0.smithi107.stdout:11653: done (0 left) 2020-06-12T02:39:20.336 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:39:20.336 INFO:tasks.rados.rados.0.smithi107.stdout:11654: seq_num 11653 ranges {0=770048} 2020-06-12T02:39:20.336 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:39:20.333+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:39:20.340 INFO:tasks.rados.rados.0.smithi107.stdout:11654: writing smithi10729407-16 from 0 to 770048 tid 1 2020-06-12T02:39:20.340 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:39:20.340 INFO:tasks.rados.rados.0.smithi107.stdout:11655: seq_num 11654 ranges {516096=786432} 2020-06-12T02:39:20.348 INFO:tasks.rados.rados.0.smithi107.stdout:11655: writing smithi10729407-1 from 516096 to 1302528 tid 1 2020-06-12T02:39:20.348 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:39:20.348 INFO:tasks.rados.rados.0.smithi107.stdout:11656: seq_num 11655 ranges {0=802816} 2020-06-12T02:39:20.353 INFO:tasks.rados.rados.0.smithi107.stdout:11656: writing smithi10729407-7 from 0 to 802816 tid 1 2020-06-12T02:39:20.353 INFO:tasks.rados.rados.0.smithi107.stdout:11657: delete oid 10 current snap is 0 2020-06-12T02:39:20.353 INFO:tasks.rados.rados.0.smithi107.stdout:11654: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:20.353 INFO:tasks.rados.rados.0.smithi107.stdout:11654: finishing write tid 3 to smithi10729407-16 2020-06-12T02:39:20.354 INFO:tasks.rados.rados.0.smithi107.stdout:11654: finishing write tid 4 to smithi10729407-16 2020-06-12T02:39:20.354 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 2047 (ObjNum 11653 snap 0 seq_num 11653) dirty exists 2020-06-12T02:39:20.354 INFO:tasks.rados.rados.0.smithi107.stdout:11654: left oid 16 (ObjNum 11653 snap 0 seq_num 11653) 2020-06-12T02:39:20.355 INFO:tasks.rados.rados.0.smithi107.stdout:11655: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:20.355 INFO:tasks.rados.rados.0.smithi107.stdout:11655: finishing write tid 3 to smithi10729407-1 2020-06-12T02:39:20.360 INFO:tasks.rados.rados.0.smithi107.stdout:11654: done (3 left) 2020-06-12T02:39:20.360 INFO:tasks.rados.rados.0.smithi107.stdout:11657: done (2 left) 2020-06-12T02:39:20.361 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:39:20.361 INFO:tasks.rados.rados.0.smithi107.stdout:11658: seq_num 11657 ranges {0=434176,434176=401408} 2020-06-12T02:39:20.362 INFO:tasks.rados.rados.0.smithi107.stdout:11658: writing smithi10729407-15 from 0 to 434176 tid 1 2020-06-12T02:39:20.364 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:39:20.365 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:39:20.365 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.365 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "MOSDECSubOpWrite(3.13s2 87/85 ECSubWrite(tid=3971, reqid=client.4512.0:24870, at_version=87'1874, trim_to=0'0, roll_forward_to=87'1873))", 2020-06-12T02:39:20.366 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:39:20.360976+0000", 2020-06-12T02:39:20.366 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.0015357909999999999, 2020-06-12T02:39:20.366 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0015580030000000001, 2020-06-12T02:39:20.366 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:20.366 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:39:20.367 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:20.367 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.367 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:20.367 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:20.360976+0000", 2020-06-12T02:39:20.367 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:20.368 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.368 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.368 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:20.368 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:20.360976+0000", 2020-06-12T02:39:20.368 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.7148000000000002e-05 2020-06-12T02:39:20.369 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.369 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.369 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:20.369 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:20.361043+0000", 2020-06-12T02:39:20.369 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2700999999999997e-05 2020-06-12T02:39:20.370 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.370 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.378 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:20.379 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:20.361076+0000", 2020-06-12T02:39:20.379 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.0288999999999998e-05 2020-06-12T02:39:20.379 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.379 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.379 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:20.380 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:20.361096+0000", 2020-06-12T02:39:20.380 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1749999999999999e-06 2020-06-12T02:39:20.380 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.389 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.389 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:20.389 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:20.361103+0000", 2020-06-12T02:39:20.389 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4530000000000002e-05 2020-06-12T02:39:20.390 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.390 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.390 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:20.390 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:20.361148+0000", 2020-06-12T02:39:20.390 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.098e-05 2020-06-12T02:39:20.391 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.391 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.391 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:20.391 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:20.361159+0000", 2020-06-12T02:39:20.391 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00018282300000000001 2020-06-12T02:39:20.392 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:20.392 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:20.392 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:20.393 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:20.399 INFO:teuthology.orchestra.run.smithi188.stdout: ], 2020-06-12T02:39:20.399 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 1 2020-06-12T02:39:20.399 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:39:20.403 INFO:tasks.rados.rados.0.smithi107.stdout:11658: writing smithi10729407-15 from 434176 to 835584 tid 2 2020-06-12T02:39:20.403 INFO:tasks.rados.rados.0.smithi107.stdout:11659: delete oid 2 current snap is 0 2020-06-12T02:39:20.403 INFO:tasks.rados.rados.0.smithi107.stdout:11655: finishing write tid 4 to smithi10729407-1 2020-06-12T02:39:20.403 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 2132 (ObjNum 11654 snap 0 seq_num 11654) dirty exists 2020-06-12T02:39:20.403 INFO:tasks.rados.rados.0.smithi107.stdout:11655: left oid 1 (ObjNum 11654 snap 0 seq_num 11654) 2020-06-12T02:39:20.404 INFO:tasks.rados.rados.0.smithi107.stdout:11656: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:20.404 INFO:tasks.rados.rados.0.smithi107.stdout:11656: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:20.404 INFO:tasks.rados.rados.0.smithi107.stdout:11655: done (3 left) 2020-06-12T02:39:20.404 INFO:tasks.rados.rados.0.smithi107.stdout:11659: done (2 left) 2020-06-12T02:39:20.404 INFO:tasks.rados.rados.0.smithi107.stdout:11660: delete oid 6 current snap is 0 2020-06-12T02:39:20.405 INFO:tasks.rados.rados.0.smithi107.stdout:11658: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:20.405 INFO:tasks.rados.rados.0.smithi107.stdout:11656: finishing write tid 4 to smithi10729407-7 2020-06-12T02:39:20.405 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1874 (ObjNum 11655 snap 0 seq_num 11655) dirty exists 2020-06-12T02:39:20.405 INFO:tasks.rados.rados.0.smithi107.stdout:11656: left oid 7 (ObjNum 11655 snap 0 seq_num 11655) 2020-06-12T02:39:20.405 INFO:tasks.rados.rados.0.smithi107.stdout:11656: done (2 left) 2020-06-12T02:39:20.406 INFO:tasks.rados.rados.0.smithi107.stdout:11660: done (1 left) 2020-06-12T02:39:20.406 INFO:tasks.rados.rados.0.smithi107.stdout:11661: delete oid 16 current snap is 0 2020-06-12T02:39:20.406 INFO:tasks.rados.rados.0.smithi107.stdout:11658: finishing write tid 2 to smithi10729407-15 2020-06-12T02:39:20.406 INFO:tasks.rados.rados.0.smithi107.stdout:11658: finishing write tid 4 to smithi10729407-15 2020-06-12T02:39:20.406 INFO:tasks.rados.rados.0.smithi107.stdout:11661: done (1 left) 2020-06-12T02:39:20.407 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:39:20.407 INFO:tasks.rados.rados.0.smithi107.stdout:11662: seq_num 11661 ranges {0=507904,507904=401408} 2020-06-12T02:39:20.407 INFO:tasks.rados.rados.0.smithi107.stdout:11662: writing smithi10729407-3 from 0 to 507904 tid 1 2020-06-12T02:39:20.407 INFO:tasks.rados.rados.0.smithi107.stdout:11662: writing smithi10729407-3 from 507904 to 909312 tid 2 2020-06-12T02:39:20.407 INFO:tasks.rados.rados.0.smithi107.stdout:11663: delete oid 14 current snap is 0 2020-06-12T02:39:20.408 INFO:tasks.rados.rados.0.smithi107.stdout:11658: finishing write tid 5 to smithi10729407-15 2020-06-12T02:39:20.408 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 2200 (ObjNum 11657 snap 0 seq_num 11657) dirty exists 2020-06-12T02:39:20.408 INFO:tasks.rados.rados.0.smithi107.stdout:11658: left oid 15 (ObjNum 11657 snap 0 seq_num 11657) 2020-06-12T02:39:20.408 INFO:tasks.rados.rados.0.smithi107.stdout:11658: done (2 left) 2020-06-12T02:39:20.408 INFO:tasks.rados.rados.0.smithi107.stdout:11663: done (1 left) 2020-06-12T02:39:20.409 INFO:tasks.rados.rados.0.smithi107.stdout:11664: delete oid 4 current snap is 0 2020-06-12T02:39:20.409 INFO:tasks.rados.rados.0.smithi107.stdout:11662: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:20.409 INFO:tasks.rados.rados.0.smithi107.stdout:11662: finishing write tid 2 to smithi10729407-3 2020-06-12T02:39:20.410 INFO:tasks.rados.rados.0.smithi107.stdout:11664: done (1 left) 2020-06-12T02:39:20.410 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:39:20.410 INFO:tasks.rados.rados.0.smithi107.stdout:11665: seq_num 11664 ranges {0=557056,557056=401408} 2020-06-12T02:39:20.410 INFO:tasks.rados.rados.0.smithi107.stdout:11665: writing smithi10729407-2 from 0 to 557056 tid 1 2020-06-12T02:39:20.410 INFO:tasks.rados.rados.0.smithi107.stdout:11665: writing smithi10729407-2 from 557056 to 958464 tid 2 2020-06-12T02:39:20.411 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:39:20.411 INFO:tasks.rados.rados.0.smithi107.stdout:11666: seq_num 11665 ranges {0=573440,573440=401408} 2020-06-12T02:39:20.411 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-06-12T02:39:20.413 INFO:tasks.rados.rados.0.smithi107.stdout:11666: writing smithi10729407-11 from 0 to 573440 tid 1 2020-06-12T02:39:20.413 INFO:tasks.rados.rados.0.smithi107.stdout:11666: writing smithi10729407-11 from 573440 to 974848 tid 2 2020-06-12T02:39:20.414 INFO:tasks.rados.rados.0.smithi107.stdout:11667: delete oid 8 current snap is 0 2020-06-12T02:39:20.414 INFO:tasks.rados.rados.0.smithi107.stdout:11662: finishing write tid 4 to smithi10729407-3 2020-06-12T02:39:20.414 INFO:tasks.rados.rados.0.smithi107.stdout:11662: finishing write tid 5 to smithi10729407-3 2020-06-12T02:39:20.414 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 2051 (ObjNum 11661 snap 0 seq_num 11661) dirty exists 2020-06-12T02:39:20.415 INFO:tasks.rados.rados.0.smithi107.stdout:11662: left oid 3 (ObjNum 11661 snap 0 seq_num 11661) 2020-06-12T02:39:20.415 INFO:tasks.rados.rados.0.smithi107.stdout:11665: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:20.415 INFO:tasks.rados.rados.0.smithi107.stdout:11665: finishing write tid 2 to smithi10729407-2 2020-06-12T02:39:20.415 INFO:tasks.rados.rados.0.smithi107.stdout:11665: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:20.415 INFO:tasks.rados.rados.0.smithi107.stdout:11666: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:20.415 INFO:tasks.rados.rados.0.smithi107.stdout:11662: done (3 left) 2020-06-12T02:39:20.416 INFO:tasks.rados.rados.0.smithi107.stdout:11667: done (2 left) 2020-06-12T02:39:20.416 INFO:tasks.rados.rados.0.smithi107.stdout:11668: delete oid 16 current snap is 0 2020-06-12T02:39:20.416 INFO:tasks.rados.rados.0.smithi107.stdout:11665: finishing write tid 5 to smithi10729407-2 2020-06-12T02:39:20.416 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 1101 (ObjNum 11664 snap 0 seq_num 11664) dirty exists 2020-06-12T02:39:20.416 INFO:tasks.rados.rados.0.smithi107.stdout:11665: left oid 2 (ObjNum 11664 snap 0 seq_num 11664) 2020-06-12T02:39:20.417 INFO:tasks.rados.rados.0.smithi107.stdout:11665: done (2 left) 2020-06-12T02:39:20.417 INFO:tasks.rados.rados.0.smithi107.stdout:11668: done (1 left) 2020-06-12T02:39:20.418 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:39:20.418 INFO:tasks.rados.rados.0.smithi107.stdout:11669: seq_num 11668 ranges {0=622592,622592=401408} 2020-06-12T02:39:20.418 INFO:tasks.rados.rados.0.smithi107.stdout:11669: writing smithi10729407-17 from 0 to 622592 tid 1 2020-06-12T02:39:20.420 INFO:tasks.rados.rados.0.smithi107.stdout:11669: writing smithi10729407-17 from 622592 to 1024000 tid 2 2020-06-12T02:39:20.421 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:39:20.421 INFO:tasks.rados.rados.0.smithi107.stdout:11670: seq_num 11669 ranges {0=638976,638976=401408} 2020-06-12T02:39:20.424 INFO:tasks.rados.rados.0.smithi107.stdout:11670: writing smithi10729407-12 from 0 to 638976 tid 1 2020-06-12T02:39:20.426 INFO:tasks.rados.rados.0.smithi107.stdout:11670: writing smithi10729407-12 from 638976 to 1040384 tid 2 2020-06-12T02:39:20.427 INFO:tasks.rados.rados.0.smithi107.stdout:11671: delete oid 4 current snap is 0 2020-06-12T02:39:20.427 INFO:tasks.rados.rados.0.smithi107.stdout:11666: finishing write tid 2 to smithi10729407-11 2020-06-12T02:39:20.427 INFO:tasks.rados.rados.0.smithi107.stdout:11666: finishing write tid 4 to smithi10729407-11 2020-06-12T02:39:20.427 INFO:tasks.rados.rados.0.smithi107.stdout:11666: finishing write tid 5 to smithi10729407-11 2020-06-12T02:39:20.427 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1824 (ObjNum 11665 snap 0 seq_num 11665) dirty exists 2020-06-12T02:39:20.428 INFO:tasks.rados.rados.0.smithi107.stdout:11666: left oid 11 (ObjNum 11665 snap 0 seq_num 11665) 2020-06-12T02:39:20.428 INFO:tasks.rados.rados.0.smithi107.stdout:11669: finishing write tid 1 to smithi10729407-17 2020-06-12T02:39:20.428 INFO:tasks.rados.rados.0.smithi107.stdout:11669: finishing write tid 2 to smithi10729407-17 2020-06-12T02:39:20.429 INFO:tasks.rados.rados.0.smithi107.stdout:11669: finishing write tid 4 to smithi10729407-17 2020-06-12T02:39:20.429 INFO:tasks.rados.rados.0.smithi107.stdout:11669: finishing write tid 5 to smithi10729407-17 2020-06-12T02:39:20.429 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 2165 (ObjNum 11668 snap 0 seq_num 11668) dirty exists 2020-06-12T02:39:20.429 INFO:tasks.rados.rados.0.smithi107.stdout:11669: left oid 17 (ObjNum 11668 snap 0 seq_num 11668) 2020-06-12T02:39:20.430 INFO:tasks.rados.rados.0.smithi107.stdout:11666: done (3 left) 2020-06-12T02:39:20.431 INFO:tasks.rados.rados.0.smithi107.stdout:11669: done (2 left) 2020-06-12T02:39:20.431 INFO:tasks.rados.rados.0.smithi107.stdout:11671: done (1 left) 2020-06-12T02:39:20.431 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:39:20.432 INFO:tasks.rados.rados.0.smithi107.stdout:11672: seq_num 11671 ranges {0=671744,671744=401408} 2020-06-12T02:39:20.433 INFO:tasks.rados.rados.0.smithi107.stdout:11672: writing smithi10729407-19 from 0 to 671744 tid 1 2020-06-12T02:39:20.436 INFO:tasks.rados.rados.0.smithi107.stdout:11672: writing smithi10729407-19 from 671744 to 1073152 tid 2 2020-06-12T02:39:20.436 INFO:tasks.rados.rados.0.smithi107.stdout:11673: delete oid 9 current snap is 0 2020-06-12T02:39:20.436 INFO:tasks.rados.rados.0.smithi107.stdout:11670: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:20.436 INFO:tasks.rados.rados.0.smithi107.stdout:11670: finishing write tid 2 to smithi10729407-12 2020-06-12T02:39:20.437 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:39:20.433+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:39:20.438 INFO:tasks.rados.rados.0.smithi107.stdout:11670: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:20.438 INFO:tasks.rados.rados.0.smithi107.stdout:11673: done (2 left) 2020-06-12T02:39:20.438 INFO:tasks.rados.rados.0.smithi107.stdout:11674: delete oid 4 current snap is 0 2020-06-12T02:39:20.441 INFO:tasks.rados.rados.0.smithi107.stdout:11674: done (2 left) 2020-06-12T02:39:20.441 INFO:tasks.rados.rados.0.smithi107.stdout:11675: delete oid 5 current snap is 0 2020-06-12T02:39:20.442 INFO:tasks.rados.rados.0.smithi107.stdout:11672: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:20.443 INFO:tasks.rados.rados.0.smithi107.stdout:11672: finishing write tid 2 to smithi10729407-19 2020-06-12T02:39:20.443 INFO:tasks.rados.rados.0.smithi107.stdout:11670: finishing write tid 5 to smithi10729407-12 2020-06-12T02:39:20.443 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 2171 (ObjNum 11669 snap 0 seq_num 11669) dirty exists 2020-06-12T02:39:20.444 INFO:tasks.rados.rados.0.smithi107.stdout:11670: left oid 12 (ObjNum 11669 snap 0 seq_num 11669) 2020-06-12T02:39:20.444 INFO:tasks.rados.rados.0.smithi107.stdout:11670: done (2 left) 2020-06-12T02:39:20.445 INFO:tasks.rados.rados.0.smithi107.stdout:11675: done (1 left) 2020-06-12T02:39:20.445 INFO:tasks.rados.rados.0.smithi107.stdout:11676: delete oid 17 current snap is 0 2020-06-12T02:39:20.447 INFO:tasks.rados.rados.0.smithi107.stdout:11672: finishing write tid 4 to smithi10729407-19 2020-06-12T02:39:20.447 INFO:tasks.rados.rados.0.smithi107.stdout:11676: done (1 left) 2020-06-12T02:39:20.447 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:39:20.448 INFO:tasks.rados.rados.0.smithi107.stdout:11677: seq_num 11676 ranges {532480=753664,1286144=401408} 2020-06-12T02:39:20.454 INFO:tasks.rados.rados.0.smithi107.stdout:11677: writing smithi10729407-18 from 532480 to 1286144 tid 1 2020-06-12T02:39:20.457 INFO:tasks.rados.rados.0.smithi107.stdout:11677: writing smithi10729407-18 from 1286144 to 1687552 tid 2 2020-06-12T02:39:20.458 INFO:tasks.rados.rados.0.smithi107.stdout:11678: delete oid 15 current snap is 0 2020-06-12T02:39:20.458 INFO:tasks.rados.rados.0.smithi107.stdout:11672: finishing write tid 5 to smithi10729407-19 2020-06-12T02:39:20.458 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 2174 (ObjNum 11671 snap 0 seq_num 11671) dirty exists 2020-06-12T02:39:20.459 INFO:tasks.rados.rados.0.smithi107.stdout:11672: left oid 19 (ObjNum 11671 snap 0 seq_num 11671) 2020-06-12T02:39:20.460 INFO:tasks.rados.rados.0.smithi107.stdout:11672: done (2 left) 2020-06-12T02:39:20.460 INFO:tasks.rados.rados.0.smithi107.stdout:11678: done (1 left) 2020-06-12T02:39:20.460 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:39:20.460 INFO:tasks.rados.rados.0.smithi107.stdout:11679: seq_num 11678 ranges {0=794624,794624=401408} 2020-06-12T02:39:20.465 INFO:tasks.rados.rados.0.smithi107.stdout:11679: writing smithi10729407-14 from 0 to 794624 tid 1 2020-06-12T02:39:20.469 INFO:tasks.rados.rados.0.smithi107.stdout:11679: writing smithi10729407-14 from 794624 to 1196032 tid 2 2020-06-12T02:39:20.469 INFO:tasks.rados.rados.0.smithi107.stdout:11680: delete oid 5 current snap is 0 2020-06-12T02:39:20.470 INFO:tasks.rados.rados.0.smithi107.stdout:11677: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:20.470 INFO:tasks.rados.rados.0.smithi107.stdout:11677: finishing write tid 2 to smithi10729407-18 2020-06-12T02:39:20.470 INFO:tasks.rados.rados.0.smithi107.stdout:11677: finishing write tid 4 to smithi10729407-18 2020-06-12T02:39:20.471 INFO:tasks.rados.rados.0.smithi107.stdout:11680: done (2 left) 2020-06-12T02:39:20.471 INFO:tasks.rados.rados.0.smithi107.stdout:11681: delete oid 8 current snap is 0 2020-06-12T02:39:20.471 INFO:tasks.rados.rados.0.smithi107.stdout:11677: finishing write tid 5 to smithi10729407-18 2020-06-12T02:39:20.471 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1939 (ObjNum 11676 snap 0 seq_num 11676) dirty exists 2020-06-12T02:39:20.472 INFO:tasks.rados.rados.0.smithi107.stdout:11677: left oid 18 (ObjNum 11676 snap 0 seq_num 11676) 2020-06-12T02:39:20.474 INFO:tasks.rados.rados.0.smithi107.stdout:11679: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:20.475 INFO:tasks.rados.rados.0.smithi107.stdout:11679: finishing write tid 2 to smithi10729407-14 2020-06-12T02:39:20.475 INFO:tasks.rados.rados.0.smithi107.stdout:11677: done (2 left) 2020-06-12T02:39:20.475 INFO:tasks.rados.rados.0.smithi107.stdout:11681: done (1 left) 2020-06-12T02:39:20.475 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:39:20.475 INFO:tasks.rados.rados.0.smithi107.stdout:11682: seq_num 11681 ranges {0=442368,442368=794624,1236992=8192} 2020-06-12T02:39:20.477 INFO:tasks.rados.rados.0.smithi107.stdout:11682: writing smithi10729407-9 from 0 to 442368 tid 1 2020-06-12T02:39:20.482 INFO:tasks.rados.rados.0.smithi107.stdout:11682: writing smithi10729407-9 from 442368 to 1236992 tid 2 2020-06-12T02:39:20.482 INFO:tasks.rados.rados.0.smithi107.stdout:11682: writing smithi10729407-9 from 1236992 to 1245184 tid 3 2020-06-12T02:39:20.483 INFO:tasks.rados.rados.0.smithi107.stdout:11683: delete oid 17 current snap is 0 2020-06-12T02:39:20.483 INFO:tasks.rados.rados.0.smithi107.stdout:11679: finishing write tid 4 to smithi10729407-14 2020-06-12T02:39:20.483 INFO:tasks.rados.rados.0.smithi107.stdout:11679: finishing write tid 5 to smithi10729407-14 2020-06-12T02:39:20.483 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1810 (ObjNum 11678 snap 0 seq_num 11678) dirty exists 2020-06-12T02:39:20.483 INFO:tasks.rados.rados.0.smithi107.stdout:11679: left oid 14 (ObjNum 11678 snap 0 seq_num 11678) 2020-06-12T02:39:20.484 INFO:tasks.rados.rados.0.smithi107.stdout:11682: finishing write tid 1 to smithi10729407-9 2020-06-12T02:39:20.488 INFO:tasks.rados.rados.0.smithi107.stdout:11679: done (2 left) 2020-06-12T02:39:20.488 INFO:tasks.rados.rados.0.smithi107.stdout:11683: done (1 left) 2020-06-12T02:39:20.488 INFO:tasks.rados.rados.0.smithi107.stdout:11684: delete oid 19 current snap is 0 2020-06-12T02:39:20.489 INFO:tasks.rados.rados.0.smithi107.stdout:11682: finishing write tid 2 to smithi10729407-9 2020-06-12T02:39:20.490 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd unset nodeep-scrub 2020-06-12T02:39:20.491 INFO:tasks.rados.rados.0.smithi107.stdout:11682: finishing write tid 3 to smithi10729407-9 2020-06-12T02:39:20.491 INFO:tasks.rados.rados.0.smithi107.stdout:11682: finishing write tid 5 to smithi10729407-9 2020-06-12T02:39:20.492 INFO:tasks.rados.rados.0.smithi107.stdout:11684: done (1 left) 2020-06-12T02:39:20.492 INFO:tasks.rados.rados.0.smithi107.stdout:11685: delete oid 19 current snap is 0 2020-06-12T02:39:20.493 INFO:tasks.rados.rados.0.smithi107.stdout:11682: finishing write tid 6 to smithi10729407-9 2020-06-12T02:39:20.493 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1905 (ObjNum 11681 snap 0 seq_num 11681) dirty exists 2020-06-12T02:39:20.493 INFO:tasks.rados.rados.0.smithi107.stdout:11682: left oid 9 (ObjNum 11681 snap 0 seq_num 11681) 2020-06-12T02:39:20.494 INFO:tasks.rados.rados.0.smithi107.stdout:11682: done (1 left) 2020-06-12T02:39:20.495 INFO:tasks.rados.rados.0.smithi107.stdout:11685: done (0 left) 2020-06-12T02:39:20.495 INFO:tasks.rados.rados.0.smithi107.stdout:11686: delete oid 2 current snap is 0 2020-06-12T02:39:20.497 INFO:tasks.rados.rados.0.smithi107.stdout:11686: done (0 left) 2020-06-12T02:39:20.497 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:39:20.497 INFO:tasks.rados.rados.0.smithi107.stdout:11687: seq_num 11686 ranges {0=524288,524288=663552,1187840=139264} 2020-06-12T02:39:20.500 INFO:tasks.rados.rados.0.smithi107.stdout:11687: writing smithi10729407-17 from 0 to 524288 tid 1 2020-06-12T02:39:20.504 INFO:tasks.rados.rados.0.smithi107.stdout:11687: writing smithi10729407-17 from 524288 to 1187840 tid 2 2020-06-12T02:39:20.505 INFO:tasks.rados.rados.0.smithi107.stdout:11687: writing smithi10729407-17 from 1187840 to 1327104 tid 3 2020-06-12T02:39:20.505 INFO:tasks.rados.rados.0.smithi107.stdout:11688: delete oid 4 current snap is 0 2020-06-12T02:39:20.507 INFO:tasks.rados.rados.0.smithi107.stdout:11687: finishing write tid 1 to smithi10729407-17 2020-06-12T02:39:20.512 INFO:tasks.rados.rados.0.smithi107.stdout:11688: done (1 left) 2020-06-12T02:39:20.513 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:39:20.513 INFO:tasks.rados.rados.0.smithi107.stdout:11689: seq_num 11688 ranges {0=557056,557056=434176,991232=368640} 2020-06-12T02:39:20.515 INFO:tasks.rados.rados.0.smithi107.stdout:11689: writing smithi10729407-4 from 0 to 557056 tid 1 2020-06-12T02:39:20.519 INFO:tasks.rados.rados.0.smithi107.stdout:11689: writing smithi10729407-4 from 557056 to 991232 tid 2 2020-06-12T02:39:20.521 INFO:tasks.rados.rados.0.smithi107.stdout:11689: writing smithi10729407-4 from 991232 to 1359872 tid 3 2020-06-12T02:39:20.521 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:39:20.521 INFO:tasks.rados.rados.0.smithi107.stdout:11687: finishing write tid 2 to smithi10729407-17 2020-06-12T02:39:20.521 INFO:tasks.rados.rados.0.smithi107.stdout:11687: finishing write tid 3 to smithi10729407-17 2020-06-12T02:39:20.521 INFO:tasks.rados.rados.0.smithi107.stdout:11687: finishing write tid 5 to smithi10729407-17 2020-06-12T02:39:20.522 INFO:tasks.rados.rados.0.smithi107.stdout:11687: finishing write tid 6 to smithi10729407-17 2020-06-12T02:39:20.522 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 2174 (ObjNum 11686 snap 0 seq_num 11686) dirty exists 2020-06-12T02:39:20.522 INFO:tasks.rados.rados.0.smithi107.stdout:11687: left oid 17 (ObjNum 11686 snap 0 seq_num 11686) 2020-06-12T02:39:20.523 INFO:tasks.rados.rados.0.smithi107.stdout:11690: seq_num 11689 ranges {1302528=573440,1875968=720896,2596864=81920} 2020-06-12T02:39:20.531 INFO:tasks.rados.rados.0.smithi107.stdout:11690: writing smithi10729407-1 from 1302528 to 1875968 tid 1 2020-06-12T02:39:20.535 INFO:tasks.rados.rados.0.smithi107.stdout:11690: writing smithi10729407-1 from 1875968 to 2596864 tid 2 2020-06-12T02:39:20.537 INFO:tasks.rados.rados.0.smithi107.stdout:11690: writing smithi10729407-1 from 2596864 to 2678784 tid 3 2020-06-12T02:39:20.537 INFO:tasks.rados.rados.0.smithi107.stdout:11687: done (2 left) 2020-06-12T02:39:20.537 INFO:tasks.rados.rados.0.smithi107.stdout:11691: delete oid 9 current snap is 0 2020-06-12T02:39:20.537 INFO:tasks.rados.rados.0.smithi107.stdout:11689: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:20.538 INFO:tasks.rados.rados.0.smithi107.stdout:11689: finishing write tid 2 to smithi10729407-4 2020-06-12T02:39:20.538 INFO:tasks.rados.rados.0.smithi107.stdout:11689: finishing write tid 3 to smithi10729407-4 2020-06-12T02:39:20.538 INFO:tasks.rados.rados.0.smithi107.stdout:11689: finishing write tid 5 to smithi10729407-4 2020-06-12T02:39:20.538 INFO:tasks.rados.rados.0.smithi107.stdout:11689: finishing write tid 6 to smithi10729407-4 2020-06-12T02:39:20.538 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 2178 (ObjNum 11688 snap 0 seq_num 11688) dirty exists 2020-06-12T02:39:20.539 INFO:tasks.rados.rados.0.smithi107.stdout:11689: left oid 4 (ObjNum 11688 snap 0 seq_num 11688) 2020-06-12T02:39:20.539 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:39:20.534+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:39:20.539 INFO:tasks.rados.rados.0.smithi107.stdout:11690: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:20.540 INFO:tasks.rados.rados.0.smithi107.stdout:11689: done (2 left) 2020-06-12T02:39:20.540 INFO:tasks.rados.rados.0.smithi107.stdout:11691: done (1 left) 2020-06-12T02:39:20.540 INFO:tasks.rados.rados.0.smithi107.stdout:11692: delete oid 15 current snap is 0 2020-06-12T02:39:20.548 INFO:tasks.rados.rados.0.smithi107.stdout:11692: done (1 left) 2020-06-12T02:39:20.549 INFO:tasks.rados.rados.0.smithi107.stdout:11693: delete oid 3 current snap is 0 2020-06-12T02:39:20.549 INFO:tasks.rados.rados.0.smithi107.stdout:11693: done (1 left) 2020-06-12T02:39:20.549 INFO:tasks.rados.rados.0.smithi107.stdout:11694: delete oid 7 current snap is 0 2020-06-12T02:39:20.549 INFO:tasks.rados.rados.0.smithi107.stdout:11690: finishing write tid 2 to smithi10729407-1 2020-06-12T02:39:20.550 INFO:tasks.rados.rados.0.smithi107.stdout:11694: done (1 left) 2020-06-12T02:39:20.550 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:39:20.550 INFO:tasks.rados.rados.0.smithi107.stdout:11695: seq_num 11694 ranges {1196032=655360,1851392=589824,2441216=212992} 2020-06-12T02:39:20.560 INFO:tasks.rados.rados.0.smithi107.stdout:11695: writing smithi10729407-14 from 1196032 to 1851392 tid 1 2020-06-12T02:39:20.564 INFO:tasks.rados.rados.0.smithi107.stdout:11695: writing smithi10729407-14 from 1851392 to 2441216 tid 2 2020-06-12T02:39:20.566 INFO:tasks.rados.rados.0.smithi107.stdout:11695: writing smithi10729407-14 from 2441216 to 2654208 tid 3 2020-06-12T02:39:20.566 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:39:20.567 INFO:tasks.rados.rados.0.smithi107.stdout:11696: seq_num 11695 ranges {0=671744,671744=475136,1146880=327680} 2020-06-12T02:39:20.570 INFO:tasks.rados.rados.0.smithi107.stdout:11696: writing smithi10729407-16 from 0 to 671744 tid 1 2020-06-12T02:39:20.573 INFO:tasks.rados.rados.0.smithi107.stdout:11696: writing smithi10729407-16 from 671744 to 1146880 tid 2 2020-06-12T02:39:20.576 INFO:tasks.rados.rados.0.smithi107.stdout:11696: writing smithi10729407-16 from 1146880 to 1474560 tid 3 2020-06-12T02:39:20.576 INFO:tasks.rados.rados.0.smithi107.stdout:11697: delete oid 2 current snap is 0 2020-06-12T02:39:20.577 INFO:tasks.rados.rados.0.smithi107.stdout:11690: finishing write tid 3 to smithi10729407-1 2020-06-12T02:39:20.577 INFO:tasks.rados.rados.0.smithi107.stdout:11690: finishing write tid 5 to smithi10729407-1 2020-06-12T02:39:20.577 INFO:tasks.rados.rados.0.smithi107.stdout:11690: finishing write tid 6 to smithi10729407-1 2020-06-12T02:39:20.577 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 2137 (ObjNum 11689 snap 0 seq_num 11689) dirty exists 2020-06-12T02:39:20.577 INFO:tasks.rados.rados.0.smithi107.stdout:11690: left oid 1 (ObjNum 11689 snap 0 seq_num 11689) 2020-06-12T02:39:20.578 INFO:tasks.rados.rados.0.smithi107.stdout:11695: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:20.578 INFO:tasks.rados.rados.0.smithi107.stdout:11695: finishing write tid 2 to smithi10729407-14 2020-06-12T02:39:20.579 INFO:tasks.rados.rados.0.smithi107.stdout:11695: finishing write tid 3 to smithi10729407-14 2020-06-12T02:39:20.579 INFO:tasks.rados.rados.0.smithi107.stdout:11695: finishing write tid 5 to smithi10729407-14 2020-06-12T02:39:20.579 INFO:tasks.rados.rados.0.smithi107.stdout:11696: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:20.580 INFO:tasks.rados.rados.0.smithi107.stdout:11690: done (3 left) 2020-06-12T02:39:20.580 INFO:tasks.rados.rados.0.smithi107.stdout:11697: done (2 left) 2020-06-12T02:39:20.581 INFO:tasks.rados.rados.0.smithi107.stdout:11698: delete oid 2 current snap is 0 2020-06-12T02:39:20.581 INFO:tasks.rados.rados.0.smithi107.stdout:11695: finishing write tid 6 to smithi10729407-14 2020-06-12T02:39:20.581 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1814 (ObjNum 11694 snap 0 seq_num 11694) dirty exists 2020-06-12T02:39:20.582 INFO:tasks.rados.rados.0.smithi107.stdout:11695: left oid 14 (ObjNum 11694 snap 0 seq_num 11694) 2020-06-12T02:39:20.582 INFO:tasks.rados.rados.0.smithi107.stdout:11695: done (2 left) 2020-06-12T02:39:20.583 INFO:tasks.rados.rados.0.smithi107.stdout:11698: done (1 left) 2020-06-12T02:39:20.583 INFO:tasks.rados.rados.0.smithi107.stdout:11699: delete oid 1 current snap is 0 2020-06-12T02:39:20.585 INFO:tasks.rados.rados.0.smithi107.stdout:11699: done (1 left) 2020-06-12T02:39:20.586 INFO:tasks.rados.rados.0.smithi107.stdout:11700: delete oid 7 current snap is 0 2020-06-12T02:39:20.587 INFO:tasks.rados.rados.0.smithi107.stdout:11700: done (1 left) 2020-06-12T02:39:20.587 INFO:tasks.rados.rados.0.smithi107.stdout:11701: delete oid 7 current snap is 0 2020-06-12T02:39:20.588 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:39:20.588 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:39:20.588 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:39:20.589 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:39:20.589 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:39:20.590 INFO:tasks.rados.rados.0.smithi107.stdout:11696: finishing write tid 2 to smithi10729407-16 2020-06-12T02:39:20.590 INFO:tasks.rados.rados.0.smithi107.stdout:11696: finishing write tid 3 to smithi10729407-16 2020-06-12T02:39:20.590 INFO:tasks.rados.rados.0.smithi107.stdout:11701: done (1 left) 2020-06-12T02:39:20.590 INFO:tasks.rados.rados.0.smithi107.stdout:11702: delete oid 19 current snap is 0 2020-06-12T02:39:20.590 INFO:tasks.rados.rados.0.smithi107.stdout:11696: finishing write tid 5 to smithi10729407-16 2020-06-12T02:39:20.592 INFO:tasks.rados.rados.0.smithi107.stdout:11696: finishing write tid 6 to smithi10729407-16 2020-06-12T02:39:20.592 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 2057 (ObjNum 11695 snap 0 seq_num 11695) dirty exists 2020-06-12T02:39:20.592 INFO:tasks.rados.rados.0.smithi107.stdout:11696: left oid 16 (ObjNum 11695 snap 0 seq_num 11695) 2020-06-12T02:39:20.593 INFO:tasks.rados.rados.0.smithi107.stdout:11696: done (1 left) 2020-06-12T02:39:20.593 INFO:tasks.rados.rados.0.smithi107.stdout:11702: done (0 left) 2020-06-12T02:39:20.593 INFO:tasks.rados.rados.0.smithi107.stdout:11703: delete oid 10 current snap is 0 2020-06-12T02:39:20.595 INFO:tasks.rados.rados.0.smithi107.stdout:11703: done (0 left) 2020-06-12T02:39:20.595 INFO:tasks.rados.rados.0.smithi107.stdout:11704: delete oid 1 current snap is 0 2020-06-12T02:39:20.599 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-06-12T02:39:20.600 INFO:tasks.rados.rados.0.smithi107.stdout:11704: done (0 left) 2020-06-12T02:39:20.600 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:39:20.600 INFO:tasks.rados.rados.0.smithi107.stdout:11705: seq_num 11704 ranges {0=425984,425984=688128,1114112=417792,1531904=98304} 2020-06-12T02:39:20.602 INFO:tasks.rados.rados.0.smithi107.stdout:11705: writing smithi10729407-13 from 0 to 425984 tid 1 2020-06-12T02:39:20.605 INFO:tasks.rados.rados.0.smithi107.stdout:11705: writing smithi10729407-13 from 425984 to 1114112 tid 2 2020-06-12T02:39:20.608 INFO:tasks.rados.rados.0.smithi107.stdout:11705: writing smithi10729407-13 from 1114112 to 1531904 tid 3 2020-06-12T02:39:20.609 INFO:tasks.rados.rados.0.smithi107.stdout:11705: writing smithi10729407-13 from 1531904 to 1630208 tid 4 2020-06-12T02:39:20.609 INFO:tasks.rados.rados.0.smithi107.stdout:11706: delete oid 16 current snap is 0 2020-06-12T02:39:20.609 INFO:tasks.rados.rados.0.smithi107.stdout:11705: finishing write tid 1 to smithi10729407-13 2020-06-12T02:39:20.621 INFO:tasks.rados.rados.0.smithi107.stdout:11705: finishing write tid 2 to smithi10729407-13 2020-06-12T02:39:20.621 INFO:tasks.rados.rados.0.smithi107.stdout:11706: done (1 left) 2020-06-12T02:39:20.621 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:39:20.621 INFO:tasks.rados.rados.0.smithi107.stdout:11707: seq_num 11706 ranges {0=458752,458752=450560,909312=425984,1335296=327680} 2020-06-12T02:39:20.622 INFO:tasks.rados.rados.0.smithi107.stdout:11707: writing smithi10729407-3 from 0 to 458752 tid 1 2020-06-12T02:39:20.622 INFO:tasks.rados.rados.0.smithi107.stdout:11707: writing smithi10729407-3 from 458752 to 909312 tid 2 2020-06-12T02:39:20.625 INFO:tasks.rados.rados.0.smithi107.stdout:11707: writing smithi10729407-3 from 909312 to 1335296 tid 3 2020-06-12T02:39:20.627 INFO:tasks.rados.rados.0.smithi107.stdout:11707: writing smithi10729407-3 from 1335296 to 1662976 tid 4 2020-06-12T02:39:20.628 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:39:20.628 INFO:tasks.rados.rados.0.smithi107.stdout:11708: seq_num 11707 ranges {0=475136,475136=737280,1212416=466944} 2020-06-12T02:39:20.631 INFO:tasks.rados.rados.0.smithi107.stdout:11708: writing smithi10729407-15 from 0 to 475136 tid 1 2020-06-12T02:39:20.636 INFO:tasks.rados.rados.0.smithi107.stdout:11708: writing smithi10729407-15 from 475136 to 1212416 tid 2 2020-06-12T02:39:20.639 INFO:tasks.rados.rados.0.smithi107.stdout:11708: writing smithi10729407-15 from 1212416 to 1679360 tid 3 2020-06-12T02:39:20.639 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:39:20.639 INFO:tasks.rados.rados.0.smithi107.stdout:11709: seq_num 11708 ranges {2654208=491520,3145728=671744,3817472=532480} 2020-06-12T02:39:20.640 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:39:20.635+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:39:20.656 INFO:tasks.rados.rados.0.smithi107.stdout:11709: writing smithi10729407-14 from 2654208 to 3145728 tid 1 2020-06-12T02:39:20.660 INFO:tasks.rados.rados.0.smithi107.stdout:11709: writing smithi10729407-14 from 3145728 to 3817472 tid 2 2020-06-12T02:39:20.664 INFO:tasks.rados.rados.0.smithi107.stdout:11709: writing smithi10729407-14 from 3817472 to 4349952 tid 3 2020-06-12T02:39:20.665 INFO:tasks.rados.rados.0.smithi107.stdout:11710: delete oid 19 current snap is 0 2020-06-12T02:39:20.665 INFO:tasks.rados.rados.0.smithi107.stdout:11705: finishing write tid 3 to smithi10729407-13 2020-06-12T02:39:20.665 INFO:tasks.rados.rados.0.smithi107.stdout:11705: finishing write tid 4 to smithi10729407-13 2020-06-12T02:39:20.665 INFO:tasks.rados.rados.0.smithi107.stdout:11705: finishing write tid 6 to smithi10729407-13 2020-06-12T02:39:20.665 INFO:tasks.rados.rados.0.smithi107.stdout:11705: finishing write tid 7 to smithi10729407-13 2020-06-12T02:39:20.666 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1839 (ObjNum 11704 snap 0 seq_num 11704) dirty exists 2020-06-12T02:39:20.666 INFO:tasks.rados.rados.0.smithi107.stdout:11705: left oid 13 (ObjNum 11704 snap 0 seq_num 11704) 2020-06-12T02:39:20.666 INFO:tasks.rados.rados.0.smithi107.stdout:11707: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:20.666 INFO:tasks.rados.rados.0.smithi107.stdout:11707: finishing write tid 2 to smithi10729407-3 2020-06-12T02:39:20.666 INFO:tasks.rados.rados.0.smithi107.stdout:11707: finishing write tid 3 to smithi10729407-3 2020-06-12T02:39:20.667 INFO:tasks.rados.rados.0.smithi107.stdout:11707: finishing write tid 4 to smithi10729407-3 2020-06-12T02:39:20.667 INFO:tasks.rados.rados.0.smithi107.stdout:11708: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:20.668 INFO:tasks.rados.rados.0.smithi107.stdout:11707: finishing write tid 6 to smithi10729407-3 2020-06-12T02:39:20.668 INFO:tasks.rados.rados.0.smithi107.stdout:11707: finishing write tid 7 to smithi10729407-3 2020-06-12T02:39:20.668 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 2063 (ObjNum 11706 snap 0 seq_num 11706) dirty exists 2020-06-12T02:39:20.668 INFO:tasks.rados.rados.0.smithi107.stdout:11707: left oid 3 (ObjNum 11706 snap 0 seq_num 11706) 2020-06-12T02:39:20.668 INFO:tasks.rados.rados.0.smithi107.stdout:11708: finishing write tid 2 to smithi10729407-15 2020-06-12T02:39:20.669 INFO:tasks.rados.rados.0.smithi107.stdout:11708: finishing write tid 3 to smithi10729407-15 2020-06-12T02:39:20.669 INFO:tasks.rados.rados.0.smithi107.stdout:11708: finishing write tid 5 to smithi10729407-15 2020-06-12T02:39:20.669 INFO:tasks.rados.rados.0.smithi107.stdout:11708: finishing write tid 6 to smithi10729407-15 2020-06-12T02:39:20.669 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 2208 (ObjNum 11707 snap 0 seq_num 11707) dirty exists 2020-06-12T02:39:20.669 INFO:tasks.rados.rados.0.smithi107.stdout:11708: left oid 15 (ObjNum 11707 snap 0 seq_num 11707) 2020-06-12T02:39:20.670 INFO:tasks.rados.rados.0.smithi107.stdout:11709: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:20.670 INFO:tasks.rados.rados.0.smithi107.stdout:11705: done (4 left) 2020-06-12T02:39:20.671 INFO:tasks.rados.rados.0.smithi107.stdout:11707: done (3 left) 2020-06-12T02:39:20.671 INFO:tasks.rados.rados.0.smithi107.stdout:11708: done (2 left) 2020-06-12T02:39:20.671 INFO:tasks.rados.rados.0.smithi107.stdout:11710: done (1 left) 2020-06-12T02:39:20.671 INFO:tasks.rados.rados.0.smithi107.stdout:11711: delete oid 11 current snap is 0 2020-06-12T02:39:20.672 INFO:tasks.rados.rados.0.smithi107.stdout:11709: finishing write tid 2 to smithi10729407-14 2020-06-12T02:39:20.672 INFO:tasks.rados.rados.0.smithi107.stdout:11709: finishing write tid 3 to smithi10729407-14 2020-06-12T02:39:20.672 INFO:tasks.rados.rados.0.smithi107.stdout:11709: finishing write tid 5 to smithi10729407-14 2020-06-12T02:39:20.672 INFO:tasks.rados.rados.0.smithi107.stdout:11711: done (1 left) 2020-06-12T02:39:20.672 INFO:tasks.rados.rados.0.smithi107.stdout:11712: delete oid 20 current snap is 0 2020-06-12T02:39:20.673 INFO:tasks.rados.rados.0.smithi107.stdout:11709: finishing write tid 6 to smithi10729407-14 2020-06-12T02:39:20.673 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1818 (ObjNum 11708 snap 0 seq_num 11708) dirty exists 2020-06-12T02:39:20.674 INFO:tasks.rados.rados.0.smithi107.stdout:11709: left oid 14 (ObjNum 11708 snap 0 seq_num 11708) 2020-06-12T02:39:20.674 INFO:tasks.rados.rados.0.smithi107.stdout:11709: done (1 left) 2020-06-12T02:39:20.674 INFO:tasks.rados.rados.0.smithi107.stdout:11712: done (0 left) 2020-06-12T02:39:20.674 INFO:tasks.rados.rados.0.smithi107.stdout:11713: delete oid 4 current snap is 0 2020-06-12T02:39:20.675 INFO:tasks.rados.rados.0.smithi107.stdout:11713: done (0 left) 2020-06-12T02:39:20.675 INFO:tasks.rados.rados.0.smithi107.stdout:11714: delete oid 1 current snap is 0 2020-06-12T02:39:20.678 INFO:tasks.rados.rados.0.smithi107.stdout:11714: done (0 left) 2020-06-12T02:39:20.678 INFO:tasks.rados.rados.0.smithi107.stdout:11715: delete oid 6 current snap is 0 2020-06-12T02:39:20.681 INFO:tasks.rados.rados.0.smithi107.stdout:11715: done (0 left) 2020-06-12T02:39:20.681 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:39:20.682 INFO:tasks.rados.rados.0.smithi107.stdout:11716: seq_num 11715 ranges {0=606208,606208=663552,1269760=540672} 2020-06-12T02:39:20.685 INFO:tasks.rados.rados.0.smithi107.stdout:11716: writing smithi10729407-16 from 0 to 606208 tid 1 2020-06-12T02:39:20.688 INFO:tasks.rados.rados.0.smithi107.stdout:11716: writing smithi10729407-16 from 606208 to 1269760 tid 2 2020-06-12T02:39:20.692 INFO:tasks.rados.rados.0.smithi107.stdout:11716: writing smithi10729407-16 from 1269760 to 1810432 tid 3 2020-06-12T02:39:20.693 INFO:tasks.rados.rados.0.smithi107.stdout:11717: delete oid 13 current snap is 0 2020-06-12T02:39:20.693 INFO:tasks.rados.rados.0.smithi107.stdout:11716: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:20.709 INFO:tasks.rados.rados.0.smithi107.stdout:11717: done (1 left) 2020-06-12T02:39:20.710 INFO:tasks.rados.rados.0.smithi107.stdout:11718: delete oid 10 current snap is 0 2020-06-12T02:39:20.710 INFO:tasks.rados.rados.0.smithi107.stdout:11718: done (1 left) 2020-06-12T02:39:20.710 INFO:tasks.rados.rados.0.smithi107.stdout:11719: delete oid 19 current snap is 0 2020-06-12T02:39:20.710 INFO:tasks.rados.rados.0.smithi107.stdout:11716: finishing write tid 2 to smithi10729407-16 2020-06-12T02:39:20.710 INFO:tasks.rados.rados.0.smithi107.stdout:11719: done (1 left) 2020-06-12T02:39:20.711 INFO:tasks.rados.rados.0.smithi107.stdout:11720: delete oid 20 current snap is 0 2020-06-12T02:39:20.711 INFO:tasks.rados.rados.0.smithi107.stdout:11716: finishing write tid 3 to smithi10729407-16 2020-06-12T02:39:20.711 INFO:tasks.rados.rados.0.smithi107.stdout:11716: finishing write tid 5 to smithi10729407-16 2020-06-12T02:39:20.711 INFO:tasks.rados.rados.0.smithi107.stdout:11720: done (1 left) 2020-06-12T02:39:20.711 INFO:tasks.rados.rados.0.smithi107.stdout:11721: delete oid 18 current snap is 0 2020-06-12T02:39:20.711 INFO:tasks.rados.rados.0.smithi107.stdout:11721: done (1 left) 2020-06-12T02:39:20.712 INFO:tasks.rados.rados.0.smithi107.stdout:11722: delete oid 15 current snap is 0 2020-06-12T02:39:20.712 INFO:tasks.rados.rados.0.smithi107.stdout:11716: finishing write tid 6 to smithi10729407-16 2020-06-12T02:39:20.712 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 2067 (ObjNum 11715 snap 0 seq_num 11715) dirty exists 2020-06-12T02:39:20.712 INFO:tasks.rados.rados.0.smithi107.stdout:11716: left oid 16 (ObjNum 11715 snap 0 seq_num 11715) 2020-06-12T02:39:20.713 INFO:tasks.rados.rados.0.smithi107.stdout:11716: done (1 left) 2020-06-12T02:39:20.714 INFO:tasks.rados.rados.0.smithi107.stdout:11722: done (0 left) 2020-06-12T02:39:20.714 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:39:20.714 INFO:tasks.rados.rados.0.smithi107.stdout:11723: seq_num 11722 ranges {0=729088,729088=704512,1433600=499712} 2020-06-12T02:39:20.716 INFO:tasks.rados.rados.0.smithi107.stdout:11723: writing smithi10729407-18 from 0 to 729088 tid 1 2020-06-12T02:39:20.721 INFO:tasks.rados.rados.0.smithi107.stdout:11723: writing smithi10729407-18 from 729088 to 1433600 tid 2 2020-06-12T02:39:20.725 INFO:tasks.rados.rados.0.smithi107.stdout:11723: writing smithi10729407-18 from 1433600 to 1933312 tid 3 2020-06-12T02:39:20.725 INFO:tasks.rados.rados.0.smithi107.stdout:11724: delete oid 9 current snap is 0 2020-06-12T02:39:20.732 INFO:tasks.rados.rados.0.smithi107.stdout:11723: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:20.732 INFO:tasks.rados.rados.0.smithi107.stdout:11723: finishing write tid 2 to smithi10729407-18 2020-06-12T02:39:20.732 INFO:tasks.rados.rados.0.smithi107.stdout:11724: done (1 left) 2020-06-12T02:39:20.733 INFO:tasks.rados.rados.0.smithi107.stdout:11725: delete oid 13 current snap is 0 2020-06-12T02:39:20.734 INFO:tasks.rados.rados.0.smithi107.stdout:11723: finishing write tid 3 to smithi10729407-18 2020-06-12T02:39:20.734 INFO:tasks.rados.rados.0.smithi107.stdout:11723: finishing write tid 5 to smithi10729407-18 2020-06-12T02:39:20.735 INFO:tasks.rados.rados.0.smithi107.stdout:11725: done (1 left) 2020-06-12T02:39:20.736 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:39:20.736 INFO:tasks.rados.rados.0.smithi107.stdout:11726: seq_num 11725 ranges {0=778240,778240=802816,1581056=401408} 2020-06-12T02:39:20.739 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:39:20.736+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:39:20.741 INFO:tasks.rados.rados.0.smithi107.stdout:11726: writing smithi10729407-2 from 0 to 778240 tid 1 2020-06-12T02:39:20.743 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:39:20.743 INFO:teuthology.orchestra.run.smithi188.stdout: "size": 20, 2020-06-12T02:39:20.744 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 600, 2020-06-12T02:39:20.744 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [ 2020-06-12T02:39:20.745 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.745 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11604 3.3s0 3:ce45c70a:::smithi10729407-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e56)", 2020-06-12T02:39:20.745 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:22.436304+0000", 2020-06-12T02:39:20.745 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 58.301842125999997, 2020-06-12T02:39:20.746 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026739028000000001, 2020-06-12T02:39:20.746 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:20.746 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:20.746 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:20.746 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:20.747 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:20.747 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11604 2020-06-12T02:39:20.747 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.747 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:20.747 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.748 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:20.748 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.436304+0000", 2020-06-12T02:39:20.748 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:20.748 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.749 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.749 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:20.749 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.436304+0000", 2020-06-12T02:39:20.749 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0182e-05 2020-06-12T02:39:20.749 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.750 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.750 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:20.750 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.436335+0000", 2020-06-12T02:39:20.750 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2780000000000002e-06 2020-06-12T02:39:20.750 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.750 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.751 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:20.751 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.436338+0000", 2020-06-12T02:39:20.751 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5050000000000004e-06 2020-06-12T02:39:20.751 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.751 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.752 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:20.752 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.436344+0000", 2020-06-12T02:39:20.752 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.6189999999999998e-06 2020-06-12T02:39:20.752 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.752 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.752 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:20.753 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.436347+0000", 2020-06-12T02:39:20.753 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1742000000000002e-05 2020-06-12T02:39:20.753 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.753 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.753 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:20.754 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.436379+0000", 2020-06-12T02:39:20.754 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5516000000000001e-05 2020-06-12T02:39:20.754 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.755 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.755 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:20.755 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.436444+0000", 2020-06-12T02:39:20.755 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012671909 2020-06-12T02:39:20.756 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.756 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.756 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:20.756 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.449116+0000", 2020-06-12T02:39:20.756 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5255999999999998e-05 2020-06-12T02:39:20.756 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.757 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.757 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:20.757 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.449201+0000", 2020-06-12T02:39:20.757 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0065621330000000004 2020-06-12T02:39:20.757 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.758 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.758 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:20.758 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.455764+0000", 2020-06-12T02:39:20.758 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000107519 2020-06-12T02:39:20.758 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.759 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.759 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:20.759 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.455871+0000", 2020-06-12T02:39:20.759 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0071723689999999996 2020-06-12T02:39:20.759 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.759 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.760 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:20.760 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.463043+0000", 2020-06-12T02:39:20.760 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.026739028000000001 2020-06-12T02:39:20.760 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:20.760 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:20.761 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:20.761 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.761 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.761 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:11605 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~712704] snapc 0=[] ondisk+write+known_if_redirected e56)", 2020-06-12T02:39:20.761 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:22.440426+0000", 2020-06-12T02:39:20.762 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 58.297720443000003, 2020-06-12T02:39:20.762 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022109515, 2020-06-12T02:39:20.762 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:20.762 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:20.762 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:20.762 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:20.763 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:20.763 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 11605 2020-06-12T02:39:20.763 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.763 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:20.763 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.764 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:20.764 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.440426+0000", 2020-06-12T02:39:20.764 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:20.764 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.765 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.765 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:20.765 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.440426+0000", 2020-06-12T02:39:20.766 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00086067700000000001 2020-06-12T02:39:20.766 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.766 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.766 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:20.766 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.441287+0000", 2020-06-12T02:39:20.766 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1980000000000007e-06 2020-06-12T02:39:20.767 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.767 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.767 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:20.767 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.441296+0000", 2020-06-12T02:39:20.767 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.6762999999999999e-05 2020-06-12T02:39:20.768 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.768 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.768 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:20.768 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.441313+0000", 2020-06-12T02:39:20.769 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2409999999999998e-06 2020-06-12T02:39:20.769 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.769 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.769 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:20.769 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.441320+0000", 2020-06-12T02:39:20.770 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0079076760000000006 2020-06-12T02:39:20.770 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.770 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.770 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:20.770 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.449228+0000", 2020-06-12T02:39:20.771 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.2406999999999998e-05 2020-06-12T02:39:20.771 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.771 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.771 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:20.771 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.449320+0000", 2020-06-12T02:39:20.772 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00067351200000000003 2020-06-12T02:39:20.772 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.772 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.772 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:20.773 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.449994+0000", 2020-06-12T02:39:20.773 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0061543329999999997 2020-06-12T02:39:20.773 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.773 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.774 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:20.774 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.456148+0000", 2020-06-12T02:39:20.774 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0063118419999999998 2020-06-12T02:39:20.774 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.774 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.774 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:20.775 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.462460+0000", 2020-06-12T02:39:20.775 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5865999999999995e-05 2020-06-12T02:39:20.776 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.776 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.776 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:20.776 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:22.462536+0000", 2020-06-12T02:39:20.776 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022109515 2020-06-12T02:39:20.777 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:20.777 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:20.777 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:20.777 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.777 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.778 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:14918 3.9s0 3:93ea0cad:::smithi10729407-19:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e59)", 2020-06-12T02:39:20.778 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:31.902764+0000", 2020-06-12T02:39:20.778 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 48.835383049999997, 2020-06-12T02:39:20.778 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.027517911999999999, 2020-06-12T02:39:20.778 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:20.779 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:20.779 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:20.779 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:20.779 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:20.779 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 14918 2020-06-12T02:39:20.780 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.780 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:20.780 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.780 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:20.780 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:31.902764+0000", 2020-06-12T02:39:20.781 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:20.781 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.781 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.781 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:20.781 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:31.902764+0000", 2020-06-12T02:39:20.781 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.8561e-05 2020-06-12T02:39:20.782 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.782 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.782 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:20.782 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:31.902792+0000", 2020-06-12T02:39:20.782 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.8930000000000004e-06 2020-06-12T02:39:20.783 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.783 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.783 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:20.783 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:31.902797+0000", 2020-06-12T02:39:20.783 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.2152e-05 2020-06-12T02:39:20.784 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.784 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.784 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:20.784 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:31.902809+0000", 2020-06-12T02:39:20.784 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.4589999999999997e-06 2020-06-12T02:39:20.784 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.785 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.785 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:20.785 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:31.902815+0000", 2020-06-12T02:39:20.786 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6838999999999997e-05 2020-06-12T02:39:20.786 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.786 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.786 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:20.787 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:31.902851+0000", 2020-06-12T02:39:20.787 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7704000000000002e-05 2020-06-12T02:39:20.787 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.787 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.787 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:20.788 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:31.902909+0000", 2020-06-12T02:39:20.788 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.027372304 2020-06-12T02:39:20.788 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.788 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.788 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:20.789 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:31.930281+0000", 2020-06-12T02:39:20.789 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.027517911999999999 2020-06-12T02:39:20.789 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:20.789 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:20.789 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:20.790 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.790 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.790 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:15173 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~573440] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:39:20.790 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:32.691770+0000", 2020-06-12T02:39:20.791 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 48.046376782000003, 2020-06-12T02:39:20.791 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022201624999999999, 2020-06-12T02:39:20.791 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:20.791 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:20.791 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:20.792 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:20.792 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:20.792 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 15173 2020-06-12T02:39:20.792 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.792 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:20.792 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.793 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:20.793 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.691770+0000", 2020-06-12T02:39:20.793 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:20.793 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.793 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.794 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:20.794 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.691770+0000", 2020-06-12T02:39:20.794 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00055849100000000004 2020-06-12T02:39:20.794 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.794 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.795 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:20.795 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.692328+0000", 2020-06-12T02:39:20.795 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.6500000000000002e-06 2020-06-12T02:39:20.795 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.796 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.796 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:20.796 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.692337+0000", 2020-06-12T02:39:20.797 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.467e-05 2020-06-12T02:39:20.797 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.797 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.798 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:20.798 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.692352+0000", 2020-06-12T02:39:20.798 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.6599999999999998e-06 2020-06-12T02:39:20.799 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.799 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.799 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:20.799 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.692358+0000", 2020-06-12T02:39:20.800 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0023564900000000001 2020-06-12T02:39:20.800 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.800 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.800 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:20.801 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.694715+0000", 2020-06-12T02:39:20.801 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.8064000000000002e-05 2020-06-12T02:39:20.801 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.801 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.801 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:20.802 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.694813+0000", 2020-06-12T02:39:20.802 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0044087759999999997 2020-06-12T02:39:20.802 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.802 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.803 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:20.803 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.699222+0000", 2020-06-12T02:39:20.803 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9463999999999998e-05 2020-06-12T02:39:20.803 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.803 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.804 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:20.804 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.699291+0000", 2020-06-12T02:39:20.804 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00097042899999999997 2020-06-12T02:39:20.804 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.805 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.805 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:20.805 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.700261+0000", 2020-06-12T02:39:20.805 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0059618650000000002 2020-06-12T02:39:20.805 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.806 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.806 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:20.806 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.706223+0000", 2020-06-12T02:39:20.806 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.007674775 2020-06-12T02:39:20.806 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.807 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.807 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:20.807 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.713898+0000", 2020-06-12T02:39:20.807 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3291e-05 2020-06-12T02:39:20.807 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.807 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.808 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:20.808 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.713971+0000", 2020-06-12T02:39:20.808 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.022201624999999999 2020-06-12T02:39:20.808 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:20.808 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:20.809 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:20.809 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.809 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.809 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:15174 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~229376] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:39:20.809 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:32.692967+0000", 2020-06-12T02:39:20.810 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 48.045179937, 2020-06-12T02:39:20.810 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021149365999999999, 2020-06-12T02:39:20.810 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:20.810 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:20.811 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:20.811 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:20.811 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:20.811 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 15174 2020-06-12T02:39:20.811 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.812 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:20.812 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.812 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:20.812 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.692967+0000", 2020-06-12T02:39:20.812 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:20.813 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.813 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.813 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:20.813 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.692967+0000", 2020-06-12T02:39:20.813 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00034681000000000002 2020-06-12T02:39:20.813 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.814 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.814 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:20.814 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.693313+0000", 2020-06-12T02:39:20.814 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2090000000000002e-06 2020-06-12T02:39:20.814 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.815 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.815 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:20.815 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.693320+0000", 2020-06-12T02:39:20.815 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.164e-05 2020-06-12T02:39:20.815 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.816 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.816 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:20.816 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.693331+0000", 2020-06-12T02:39:20.816 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.357e-06 2020-06-12T02:39:20.816 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.816 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.817 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:20.817 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.693336+0000", 2020-06-12T02:39:20.817 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0015410910000000001 2020-06-12T02:39:20.817 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.817 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.818 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:20.818 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.694877+0000", 2020-06-12T02:39:20.818 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.7024000000000002e-05 2020-06-12T02:39:20.818 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.818 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.819 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:20.819 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.694934+0000", 2020-06-12T02:39:20.819 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0088149830000000002 2020-06-12T02:39:20.819 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.819 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.819 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:20.820 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.703749+0000", 2020-06-12T02:39:20.820 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014228600000000001 2020-06-12T02:39:20.820 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.820 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.821 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:20.821 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.703891+0000", 2020-06-12T02:39:20.821 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00069168500000000004 2020-06-12T02:39:20.821 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.821 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.822 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:20.822 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.704583+0000", 2020-06-12T02:39:20.822 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0034036829999999998 2020-06-12T02:39:20.822 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.822 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.823 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:20.823 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.707986+0000", 2020-06-12T02:39:20.823 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0060828699999999998 2020-06-12T02:39:20.823 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.823 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.824 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:20.824 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.714069+0000", 2020-06-12T02:39:20.824 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.6727999999999997e-05 2020-06-12T02:39:20.824 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.825 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.825 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:20.825 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.714116+0000", 2020-06-12T02:39:20.825 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021149365999999999 2020-06-12T02:39:20.825 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:20.826 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:20.826 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:20.826 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.826 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.826 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:15175 3.3s0 3:c2ca77da:::smithi10729407-13:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:39:20.827 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:32.693372+0000", 2020-06-12T02:39:20.827 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 48.044774324000002, 2020-06-12T02:39:20.827 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.023275062999999999, 2020-06-12T02:39:20.827 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:20.827 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:20.828 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:20.828 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:20.828 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:20.828 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 15175 2020-06-12T02:39:20.828 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.828 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:20.829 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.829 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:20.829 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.693372+0000", 2020-06-12T02:39:20.829 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:20.829 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.830 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.830 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:20.830 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.693372+0000", 2020-06-12T02:39:20.830 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.5139999999999998e-05 2020-06-12T02:39:20.830 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.831 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.831 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:20.831 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.693417+0000", 2020-06-12T02:39:20.831 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.8190000000000002e-06 2020-06-12T02:39:20.832 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.832 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.832 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:20.832 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.693421+0000", 2020-06-12T02:39:20.833 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0336000000000001e-05 2020-06-12T02:39:20.833 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.833 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.833 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:20.833 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.693432+0000", 2020-06-12T02:39:20.834 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4650000000000001e-06 2020-06-12T02:39:20.834 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.834 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.834 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:20.834 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.693434+0000", 2020-06-12T02:39:20.835 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001515984 2020-06-12T02:39:20.835 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.835 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.835 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:20.835 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.694950+0000", 2020-06-12T02:39:20.835 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1379999999999998e-05 2020-06-12T02:39:20.836 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.836 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.836 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:20.836 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.695011+0000", 2020-06-12T02:39:20.836 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011265878999999999 2020-06-12T02:39:20.837 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.837 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.837 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:20.837 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.706277+0000", 2020-06-12T02:39:20.837 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2454999999999997e-05 2020-06-12T02:39:20.838 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.838 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.838 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:20.838 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.706340+0000", 2020-06-12T02:39:20.839 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00059463099999999998 2020-06-12T02:39:20.839 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.839 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.839 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:20.840 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.706934+0000", 2020-06-12T02:39:20.840 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0019727239999999999 2020-06-12T02:39:20.840 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.840 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.841 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:20.841 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.708907+0000", 2020-06-12T02:39:20.842 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0076545620000000002 2020-06-12T02:39:20.842 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.842 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.842 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:20.843 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.716562+0000", 2020-06-12T02:39:20.843 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5687999999999997e-05 2020-06-12T02:39:20.843 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.844 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.844 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:20.844 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.716647+0000", 2020-06-12T02:39:20.844 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.023275062999999999 2020-06-12T02:39:20.845 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:20.845 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:20.846 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:20.846 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.846 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.847 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:15176 3.3s0 3:c2ca77da:::smithi10729407-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e59)", 2020-06-12T02:39:20.847 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:32.693459+0000", 2020-06-12T02:39:20.847 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 48.044687242000002, 2020-06-12T02:39:20.848 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025713850999999999, 2020-06-12T02:39:20.848 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:20.848 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:20.848 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:20.849 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:20.849 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:20.849 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 15176 2020-06-12T02:39:20.850 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.850 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:20.850 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.851 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:20.851 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.693459+0000", 2020-06-12T02:39:20.851 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:20.851 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.852 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.852 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:20.852 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.693459+0000", 2020-06-12T02:39:20.852 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.7639000000000001e-05 2020-06-12T02:39:20.853 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.853 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.853 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:20.853 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.693487+0000", 2020-06-12T02:39:20.853 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2799999999999999e-06 2020-06-12T02:39:20.854 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.854 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.854 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:20.854 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.693490+0000", 2020-06-12T02:39:20.855 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.5519999999999999e-06 2020-06-12T02:39:20.855 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.855 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.856 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:20.856 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.693498+0000", 2020-06-12T02:39:20.856 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2759999999999999e-06 2020-06-12T02:39:20.857 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.857 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.857 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:20.858 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.693500+0000", 2020-06-12T02:39:20.858 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0015322269999999999 2020-06-12T02:39:20.858 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.859 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.859 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:20.859 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.695032+0000", 2020-06-12T02:39:20.860 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.9116999999999996e-05 2020-06-12T02:39:20.860 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.860 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.860 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:20.861 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.695101+0000", 2020-06-12T02:39:20.861 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.012542372 2020-06-12T02:39:20.862 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.862 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.863 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:20.863 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.707644+0000", 2020-06-12T02:39:20.863 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.9744000000000001e-05 2020-06-12T02:39:20.864 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.864 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.864 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:20.865 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.707724+0000", 2020-06-12T02:39:20.865 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0097828480000000002 2020-06-12T02:39:20.865 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.866 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.866 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:20.867 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.717506+0000", 2020-06-12T02:39:20.867 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1787999999999996e-05 2020-06-12T02:39:20.867 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.868 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.868 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:20.869 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.717598+0000", 2020-06-12T02:39:20.870 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0015750079999999999 2020-06-12T02:39:20.871 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.871 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.871 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:20.871 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.719173+0000", 2020-06-12T02:39:20.871 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.025713850999999999 2020-06-12T02:39:20.872 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:20.872 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:20.872 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:20.873 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.873 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.873 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:15226 3.3s0 3:ce45c70a:::smithi10729407-7:head [append 0~548864] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:39:20.873 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:32.827734+0000", 2020-06-12T02:39:20.874 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 47.910412368999999, 2020-06-12T02:39:20.874 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.083147429999999994, 2020-06-12T02:39:20.874 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:20.874 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:20.874 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:20.875 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:20.875 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:20.875 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 15226 2020-06-12T02:39:20.875 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.875 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:20.876 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.876 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:20.876 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.827734+0000", 2020-06-12T02:39:20.876 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:20.876 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.877 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.877 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:20.877 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.827734+0000", 2020-06-12T02:39:20.877 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000451643 2020-06-12T02:39:20.877 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.878 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.878 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:20.878 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.828186+0000", 2020-06-12T02:39:20.878 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2770000000000002e-06 2020-06-12T02:39:20.878 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.879 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.879 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:20.879 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.828192+0000", 2020-06-12T02:39:20.879 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.1668000000000001e-05 2020-06-12T02:39:20.879 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.880 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.880 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:20.880 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.828204+0000", 2020-06-12T02:39:20.880 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5350000000000004e-06 2020-06-12T02:39:20.881 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.881 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.881 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:20.882 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.828209+0000", 2020-06-12T02:39:20.882 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7537000000000002e-05 2020-06-12T02:39:20.882 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.883 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.883 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:20.884 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.828257+0000", 2020-06-12T02:39:20.885 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000100552 2020-06-12T02:39:20.885 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.886 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.886 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:20.886 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.828357+0000", 2020-06-12T02:39:20.886 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00070831400000000004 2020-06-12T02:39:20.887 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.887 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.887 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:20.887 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.829066+0000", 2020-06-12T02:39:20.887 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00321163 2020-06-12T02:39:20.888 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.888 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.888 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:20.888 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.832277+0000", 2020-06-12T02:39:20.888 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.078530193999999998 2020-06-12T02:39:20.889 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.889 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.889 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:20.889 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.910808+0000", 2020-06-12T02:39:20.889 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.4079999999999995e-05 2020-06-12T02:39:20.890 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.890 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.890 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:20.890 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.910882+0000", 2020-06-12T02:39:20.890 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.083147429999999994 2020-06-12T02:39:20.891 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:20.891 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:20.891 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:20.891 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.891 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.892 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:15227 3.3s0 3:ce45c70a:::smithi10729407-7:head [append 0~704512] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:39:20.892 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:32.831798+0000", 2020-06-12T02:39:20.892 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 47.906348637999997, 2020-06-12T02:39:20.892 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.079221454999999996, 2020-06-12T02:39:20.892 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:20.893 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:20.893 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:20.893 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:20.893 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:20.893 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 15227 2020-06-12T02:39:20.894 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.894 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:20.894 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.894 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:20.894 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.831798+0000", 2020-06-12T02:39:20.895 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:20.895 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.895 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.896 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:20.896 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.831798+0000", 2020-06-12T02:39:20.896 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00061004699999999995 2020-06-12T02:39:20.896 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.896 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.897 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:20.897 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.832408+0000", 2020-06-12T02:39:20.897 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.0799999999999995e-06 2020-06-12T02:39:20.897 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.897 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.898 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:20.898 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.832417+0000", 2020-06-12T02:39:20.898 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3973e-05 2020-06-12T02:39:20.898 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.898 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.899 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:20.899 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.832431+0000", 2020-06-12T02:39:20.899 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.3990000000000001e-06 2020-06-12T02:39:20.899 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.900 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.900 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:20.900 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.832437+0000", 2020-06-12T02:39:20.900 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0188000000000002e-05 2020-06-12T02:39:20.900 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.901 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.901 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:20.901 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.832478+0000", 2020-06-12T02:39:20.901 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000106051 2020-06-12T02:39:20.901 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.902 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.902 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:20.902 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.832584+0000", 2020-06-12T02:39:20.902 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00067036799999999996 2020-06-12T02:39:20.902 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.903 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.903 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:20.903 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.833254+0000", 2020-06-12T02:39:20.903 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0076593140000000004 2020-06-12T02:39:20.903 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.904 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.904 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:20.904 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.840913+0000", 2020-06-12T02:39:20.904 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.070062532999999996 2020-06-12T02:39:20.904 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.905 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.905 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:20.905 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.910976+0000", 2020-06-12T02:39:20.905 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3501999999999998e-05 2020-06-12T02:39:20.905 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.906 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.906 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:20.906 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.911019+0000", 2020-06-12T02:39:20.906 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.079221454999999996 2020-06-12T02:39:20.906 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:20.907 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:20.907 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:20.907 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.907 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.907 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:15228 3.3s0 3:ce45c70a:::smithi10729407-7:head [append 0~499712] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:39:20.908 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:32.834638+0000", 2020-06-12T02:39:20.908 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 47.903508881999997, 2020-06-12T02:39:20.908 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.07850733, 2020-06-12T02:39:20.908 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:20.908 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:20.909 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:20.909 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:20.909 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:20.909 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 15228 2020-06-12T02:39:20.909 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.910 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:20.911 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.911 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:20.911 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.834638+0000", 2020-06-12T02:39:20.911 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:20.911 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.911 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.912 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:20.912 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.834638+0000", 2020-06-12T02:39:20.912 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00041470899999999998 2020-06-12T02:39:20.912 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.912 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.913 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:20.913 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835052+0000", 2020-06-12T02:39:20.913 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.1869999999999996e-06 2020-06-12T02:39:20.913 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.913 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.914 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:20.914 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835058+0000", 2020-06-12T02:39:20.914 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8429999999999999e-06 2020-06-12T02:39:20.915 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.915 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.915 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:20.916 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835066+0000", 2020-06-12T02:39:20.916 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1910000000000002e-06 2020-06-12T02:39:20.916 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.916 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.916 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:20.916 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835071+0000", 2020-06-12T02:39:20.917 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001686218 2020-06-12T02:39:20.917 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.917 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.917 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:20.918 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.836757+0000", 2020-06-12T02:39:20.919 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000126668 2020-06-12T02:39:20.919 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.919 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.919 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:20.919 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.836884+0000", 2020-06-12T02:39:20.920 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00067049399999999999 2020-06-12T02:39:20.920 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.920 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.920 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:20.920 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.837554+0000", 2020-06-12T02:39:20.921 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0034814799999999999 2020-06-12T02:39:20.921 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.922 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.922 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:20.922 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.841035+0000", 2020-06-12T02:39:20.923 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.072068473999999993 2020-06-12T02:39:20.923 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.923 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.923 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:20.923 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.913104+0000", 2020-06-12T02:39:20.923 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1066000000000003e-05 2020-06-12T02:39:20.924 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.924 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.924 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:20.924 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.913145+0000", 2020-06-12T02:39:20.925 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.07850733 2020-06-12T02:39:20.925 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:20.925 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:20.925 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:20.925 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.926 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.926 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:15229 3.3s0 3:ce45c70a:::smithi10729407-7:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:39:20.926 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:32.835098+0000", 2020-06-12T02:39:20.926 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 47.903048214999998, 2020-06-12T02:39:20.926 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.080709318000000002, 2020-06-12T02:39:20.927 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:20.927 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:20.927 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:20.927 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:20.927 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:20.928 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 15229 2020-06-12T02:39:20.928 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.928 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:20.928 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.928 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:20.928 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835098+0000", 2020-06-12T02:39:20.929 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:20.929 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.929 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.929 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:20.929 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835098+0000", 2020-06-12T02:39:20.930 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.6427999999999997e-05 2020-06-12T02:39:20.930 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.930 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.930 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:20.930 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835135+0000", 2020-06-12T02:39:20.931 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.2619999999999999e-06 2020-06-12T02:39:20.931 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.931 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.931 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:20.931 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835138+0000", 2020-06-12T02:39:20.932 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.0209999999999996e-06 2020-06-12T02:39:20.932 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.932 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.932 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:20.933 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835144+0000", 2020-06-12T02:39:20.933 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.464e-06 2020-06-12T02:39:20.933 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.933 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.933 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:20.934 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835147+0000", 2020-06-12T02:39:20.934 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.004835799 2020-06-12T02:39:20.934 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.934 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.934 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:20.935 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.839982+0000", 2020-06-12T02:39:20.935 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7294999999999998e-05 2020-06-12T02:39:20.935 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.935 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.935 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:20.936 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.840060+0000", 2020-06-12T02:39:20.936 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00028398300000000002 2020-06-12T02:39:20.936 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.936 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.936 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:20.937 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.840344+0000", 2020-06-12T02:39:20.937 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0017394960000000001 2020-06-12T02:39:20.937 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.937 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.937 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:20.938 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.842083+0000", 2020-06-12T02:39:20.938 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.073668073000000001 2020-06-12T02:39:20.938 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.938 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.938 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:20.939 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.915751+0000", 2020-06-12T02:39:20.939 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.6496999999999998e-05 2020-06-12T02:39:20.939 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.939 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.939 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:20.939 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.915808+0000", 2020-06-12T02:39:20.940 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.080709318000000002 2020-06-12T02:39:20.940 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:20.940 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:20.940 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:20.940 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.941 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.941 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:15230 3.3s0 3:ce45c70a:::smithi10729407-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e59)", 2020-06-12T02:39:20.941 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:32.835168+0000", 2020-06-12T02:39:20.941 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 47.902978586000003, 2020-06-12T02:39:20.941 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.083254480000000006, 2020-06-12T02:39:20.941 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:20.942 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:20.942 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:20.942 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:20.942 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:20.942 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 15230 2020-06-12T02:39:20.943 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.943 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:20.943 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.943 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:20.943 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835168+0000", 2020-06-12T02:39:20.944 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:20.944 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.944 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.944 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:20.945 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835168+0000", 2020-06-12T02:39:20.945 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1753999999999999e-05 2020-06-12T02:39:20.945 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.945 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.946 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:20.946 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835190+0000", 2020-06-12T02:39:20.946 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9560000000000002e-06 2020-06-12T02:39:20.946 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.947 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.947 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:20.947 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835193+0000", 2020-06-12T02:39:20.947 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2999999999999998e-06 2020-06-12T02:39:20.947 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.947 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.948 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:20.948 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835199+0000", 2020-06-12T02:39:20.948 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.3300000000000001e-06 2020-06-12T02:39:20.948 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.949 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.949 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:20.949 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835201+0000", 2020-06-12T02:39:20.949 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0057324740000000004 2020-06-12T02:39:20.949 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.950 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.950 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:20.955 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.840934+0000", 2020-06-12T02:39:20.956 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4407000000000006e-05 2020-06-12T02:39:20.957 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.957 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.957 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:20.958 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.841028+0000", 2020-06-12T02:39:20.958 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.074811723999999996 2020-06-12T02:39:20.958 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.958 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.959 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:20.959 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.915840+0000", 2020-06-12T02:39:20.959 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.0995999999999996e-05 2020-06-12T02:39:20.960 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.960 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.960 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:20.961 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.915911+0000", 2020-06-12T02:39:20.961 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025115390000000001 2020-06-12T02:39:20.961 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.962 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.962 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:20.962 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.918422+0000", 2020-06-12T02:39:20.963 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.083254480000000006 2020-06-12T02:39:20.963 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:20.964 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:20.964 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:20.964 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.964 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.965 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:15231 3.3s0 3:c2ca77da:::smithi10729407-13:head [stat,delete] snapc 0=[] ondisk+write+known_if_redirected e59)", 2020-06-12T02:39:20.965 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:32.835222+0000", 2020-06-12T02:39:20.965 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 47.902924427999999, 2020-06-12T02:39:20.966 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.081008726000000003, 2020-06-12T02:39:20.966 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:20.966 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:20.967 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:20.967 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:20.967 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:20.967 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 15231 2020-06-12T02:39:20.968 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.968 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:20.968 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.968 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:20.969 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835222+0000", 2020-06-12T02:39:20.969 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:20.969 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.969 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.969 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:20.970 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835222+0000", 2020-06-12T02:39:20.970 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1980999999999998e-05 2020-06-12T02:39:20.970 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.970 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.970 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:20.971 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835244+0000", 2020-06-12T02:39:20.971 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.9179999999999998e-06 2020-06-12T02:39:20.971 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.971 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.971 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:20.972 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835247+0000", 2020-06-12T02:39:20.972 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.8449999999999999e-06 2020-06-12T02:39:20.972 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.973 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.974 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:20.974 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835253+0000", 2020-06-12T02:39:20.974 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2809999999999998e-06 2020-06-12T02:39:20.974 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.974 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.975 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:20.975 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.835255+0000", 2020-06-12T02:39:20.975 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0058042370000000003 2020-06-12T02:39:20.975 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.975 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.975 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:20.976 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.841059+0000", 2020-06-12T02:39:20.976 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5568000000000004e-05 2020-06-12T02:39:20.976 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.976 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.976 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:20.977 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.841155+0000", 2020-06-12T02:39:20.977 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00045573899999999998 2020-06-12T02:39:20.977 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.977 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.977 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:20.978 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.841611+0000", 2020-06-12T02:39:20.978 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0022931090000000002 2020-06-12T02:39:20.978 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.978 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.978 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:20.979 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.843904+0000", 2020-06-12T02:39:20.979 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.072291870999999994 2020-06-12T02:39:20.979 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.979 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.980 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:20.980 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.916196+0000", 2020-06-12T02:39:20.980 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.5176999999999999e-05 2020-06-12T02:39:20.980 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.980 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.981 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:20.981 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:32.916231+0000", 2020-06-12T02:39:20.981 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.081008726000000003 2020-06-12T02:39:20.981 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:20.981 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:20.982 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:20.982 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.982 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.982 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18323 3.3s0 3:ce45c70a:::smithi10729407-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e62)", 2020-06-12T02:39:20.984 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.095481+0000", 2020-06-12T02:39:20.984 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 38.642665221999998, 2020-06-12T02:39:20.985 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021801823000000001, 2020-06-12T02:39:20.985 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:20.985 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:20.985 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:20.985 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:20.986 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:20.986 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18323 2020-06-12T02:39:20.986 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.986 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:20.986 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.987 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:20.987 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.095481+0000", 2020-06-12T02:39:20.987 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:20.987 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.987 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.988 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:20.988 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.095481+0000", 2020-06-12T02:39:20.988 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.3367e-05 2020-06-12T02:39:20.988 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.988 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.989 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:20.989 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.095525+0000", 2020-06-12T02:39:20.989 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.4779999999999996e-06 2020-06-12T02:39:20.989 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.989 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.990 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:20.990 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.095531+0000", 2020-06-12T02:39:20.990 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3572e-05 2020-06-12T02:39:20.990 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.990 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.990 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:20.991 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.095545+0000", 2020-06-12T02:39:20.991 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.391e-06 2020-06-12T02:39:20.991 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.992 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.992 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:20.992 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.095549+0000", 2020-06-12T02:39:20.992 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0087690979999999995 2020-06-12T02:39:20.992 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.993 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.993 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:20.993 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.104318+0000", 2020-06-12T02:39:20.993 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.4034000000000001e-05 2020-06-12T02:39:20.993 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.994 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.994 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:20.994 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.104412+0000", 2020-06-12T02:39:20.994 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.011331465000000001 2020-06-12T02:39:20.994 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.995 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.995 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:20.995 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.115744+0000", 2020-06-12T02:39:20.995 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.9160000000000006e-05 2020-06-12T02:39:20.995 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.995 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.997 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:20.997 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.115843+0000", 2020-06-12T02:39:20.997 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001440258 2020-06-12T02:39:20.998 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.998 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:20.998 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:20.998 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.117283+0000", 2020-06-12T02:39:20.998 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021801823000000001 2020-06-12T02:39:20.999 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:20.999 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:20.999 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:20.999 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:20.999 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.000 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:18324 3.9s0 3:977e4b9d:::smithi10729407-12:head [append 0~720896] snapc 0=[] ondisk+write+known_if_redirected e62)", 2020-06-12T02:39:21.000 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:38:42.097735+0000", 2020-06-12T02:39:21.000 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 38.640411872999998, 2020-06-12T02:39:21.000 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021826338000000001, 2020-06-12T02:39:21.000 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:21.001 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:21.001 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:21.002 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:21.002 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:21.002 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 18324 2020-06-12T02:39:21.002 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.003 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:21.003 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.003 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:21.003 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.097735+0000", 2020-06-12T02:39:21.004 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:21.004 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.004 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.004 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:21.004 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.097735+0000", 2020-06-12T02:39:21.005 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0022114930000000001 2020-06-12T02:39:21.005 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.005 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.005 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:21.005 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.099946+0000", 2020-06-12T02:39:21.006 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.2009999999999999e-06 2020-06-12T02:39:21.006 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.006 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.006 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:21.006 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.099954+0000", 2020-06-12T02:39:21.007 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3570999999999999e-05 2020-06-12T02:39:21.008 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.008 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.008 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:21.008 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.099968+0000", 2020-06-12T02:39:21.008 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.2600000000000002e-06 2020-06-12T02:39:21.009 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.009 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.009 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:21.009 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.099974+0000", 2020-06-12T02:39:21.009 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.1767999999999997e-05 2020-06-12T02:39:21.010 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.010 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.010 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:21.010 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.100016+0000", 2020-06-12T02:39:21.010 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.5339000000000004e-05 2020-06-12T02:39:21.011 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.011 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.011 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:21.011 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.100101+0000", 2020-06-12T02:39:21.011 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000738665 2020-06-12T02:39:21.012 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.012 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.012 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:21.012 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.100840+0000", 2020-06-12T02:39:21.012 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.018628308999999999 2020-06-12T02:39:21.012 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.013 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.013 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:21.013 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.119468+0000", 2020-06-12T02:39:21.013 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7867000000000001e-05 2020-06-12T02:39:21.013 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.014 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.014 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:21.014 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.119516+0000", 2020-06-12T02:39:21.014 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.4864999999999998e-05 2020-06-12T02:39:21.014 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.015 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.015 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:21.015 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:38:42.119561+0000", 2020-06-12T02:39:21.015 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021826338000000001 2020-06-12T02:39:21.015 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:21.016 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:21.016 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:21.016 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.016 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.017 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:22473 3.9s0 3:93ea0cad:::smithi10729407-19:head [append 0~532480] snapc 0=[] ondisk+write+known_if_redirected e83)", 2020-06-12T02:39:21.017 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:39:11.624591+0000", 2020-06-12T02:39:21.017 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.1135559920000002, 2020-06-12T02:39:21.017 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021165435999999999, 2020-06-12T02:39:21.017 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:21.018 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:21.018 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:21.018 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:21.018 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:21.018 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 22473 2020-06-12T02:39:21.019 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.020 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:21.020 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.020 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:21.020 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.624591+0000", 2020-06-12T02:39:21.020 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:21.021 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.021 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.021 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:21.021 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.624591+0000", 2020-06-12T02:39:21.021 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000582954 2020-06-12T02:39:21.022 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.022 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.022 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:21.022 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.625174+0000", 2020-06-12T02:39:21.022 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.0399999999999993e-06 2020-06-12T02:39:21.023 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.023 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.023 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:21.023 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.625182+0000", 2020-06-12T02:39:21.023 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.3997e-05 2020-06-12T02:39:21.024 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.024 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.024 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:21.024 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.625196+0000", 2020-06-12T02:39:21.024 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.1809999999999998e-06 2020-06-12T02:39:21.025 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.025 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.025 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:21.025 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.625202+0000", 2020-06-12T02:39:21.025 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00075445600000000003 2020-06-12T02:39:21.025 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.026 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.026 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:21.026 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.625956+0000", 2020-06-12T02:39:21.026 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000109559 2020-06-12T02:39:21.027 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.027 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.027 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:21.027 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.626066+0000", 2020-06-12T02:39:21.027 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0044162940000000003 2020-06-12T02:39:21.027 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.028 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.028 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:21.028 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.630482+0000", 2020-06-12T02:39:21.028 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.2767000000000003e-05 2020-06-12T02:39:21.028 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.029 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.029 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:21.029 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.630555+0000", 2020-06-12T02:39:21.029 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00068772099999999997 2020-06-12T02:39:21.029 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.030 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.030 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:21.030 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.631243+0000", 2020-06-12T02:39:21.030 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0085930759999999998 2020-06-12T02:39:21.030 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.030 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.031 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:21.031 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.639836+0000", 2020-06-12T02:39:21.031 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0058491230000000003 2020-06-12T02:39:21.031 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.031 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.032 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:21.032 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.645685+0000", 2020-06-12T02:39:21.032 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.1267999999999996e-05 2020-06-12T02:39:21.032 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.032 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.032 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:21.033 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.645756+0000", 2020-06-12T02:39:21.033 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021165435999999999 2020-06-12T02:39:21.033 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:21.033 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:21.033 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:21.034 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.034 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.034 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:22477 3.9s0 3:93ea0cad:::smithi10729407-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e83)", 2020-06-12T02:39:21.034 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:39:11.631206+0000", 2020-06-12T02:39:21.034 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 9.1069401600000006, 2020-06-12T02:39:21.035 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024571365000000001, 2020-06-12T02:39:21.035 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:21.035 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:21.035 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:21.035 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:21.035 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:21.036 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 22477 2020-06-12T02:39:21.036 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.036 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:21.036 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.036 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:21.037 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.631206+0000", 2020-06-12T02:39:21.037 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:21.037 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.037 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.037 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:21.037 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.631206+0000", 2020-06-12T02:39:21.038 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4255000000000001e-05 2020-06-12T02:39:21.038 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.038 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.038 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:21.038 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.631231+0000", 2020-06-12T02:39:21.039 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.0460000000000001e-06 2020-06-12T02:39:21.039 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.039 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.039 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:21.039 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.631234+0000", 2020-06-12T02:39:21.040 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.5699999999999998e-06 2020-06-12T02:39:21.040 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.040 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.040 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:21.040 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.631240+0000", 2020-06-12T02:39:21.040 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.1739999999999999e-06 2020-06-12T02:39:21.041 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.041 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.041 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:21.041 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.631242+0000", 2020-06-12T02:39:21.041 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.014604799 2020-06-12T02:39:21.042 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.042 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.042 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:21.042 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.645847+0000", 2020-06-12T02:39:21.042 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.1564999999999996e-05 2020-06-12T02:39:21.043 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.043 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.043 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:21.044 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.645929+0000", 2020-06-12T02:39:21.044 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0061438930000000001 2020-06-12T02:39:21.044 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.044 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.044 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:21.044 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.652073+0000", 2020-06-12T02:39:21.045 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010995800000000001 2020-06-12T02:39:21.045 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.045 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.045 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:21.045 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.652183+0000", 2020-06-12T02:39:21.046 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0035951049999999999 2020-06-12T02:39:21.046 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.046 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.046 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:21.046 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:11.655778+0000", 2020-06-12T02:39:21.047 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.024571365000000001 2020-06-12T02:39:21.047 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:21.047 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:21.047 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:21.047 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.048 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.048 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:23512 3.9s0 3:977e4b9d:::smithi10729407-12:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected e84)", 2020-06-12T02:39:21.048 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:39:14.607276+0000", 2020-06-12T02:39:21.048 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 6.1308708860000003, 2020-06-12T02:39:21.048 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.845355595, 2020-06-12T02:39:21.049 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:21.049 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "reached pg", 2020-06-12T02:39:21.049 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:21.049 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:21.049 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:21.049 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 23512 2020-06-12T02:39:21.050 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.050 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:21.050 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.050 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:21.050 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:14.607276+0000", 2020-06-12T02:39:21.051 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:21.051 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.051 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.051 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:21.051 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:14.607276+0000", 2020-06-12T02:39:21.051 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.0021999999999997e-05 2020-06-12T02:39:21.052 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.052 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.052 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:21.052 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:14.607316+0000", 2020-06-12T02:39:21.052 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.4920000000000002e-06 2020-06-12T02:39:21.053 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.053 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.053 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:21.053 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:14.607319+0000", 2020-06-12T02:39:21.053 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 6.195e-06 2020-06-12T02:39:21.054 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.054 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.054 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:21.054 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:14.607325+0000", 2020-06-12T02:39:21.054 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.4940000000000002e-06 2020-06-12T02:39:21.054 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.055 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.055 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:21.055 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:14.607328+0000", 2020-06-12T02:39:21.055 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0046396120000000004 2020-06-12T02:39:21.055 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.056 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.056 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:21.056 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:14.611968+0000", 2020-06-12T02:39:21.056 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.5278000000000005e-05 2020-06-12T02:39:21.056 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.056 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.057 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:21.057 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:14.612063+0000", 2020-06-12T02:39:21.057 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0015834550000000001 2020-06-12T02:39:21.057 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.057 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.058 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:21.058 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:14.613646+0000", 2020-06-12T02:39:21.058 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.1162999999999994e-05 2020-06-12T02:39:21.058 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.058 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.059 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:21.059 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:14.613737+0000", 2020-06-12T02:39:21.059 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00032367799999999999 2020-06-12T02:39:21.059 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.059 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.059 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:21.060 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:14.614061+0000", 2020-06-12T02:39:21.060 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0025969740000000002 2020-06-12T02:39:21.060 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.060 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.060 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:21.061 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:14.616658+0000", 2020-06-12T02:39:21.061 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.81916990199999995 2020-06-12T02:39:21.061 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.061 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.061 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:21.061 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:15.435828+0000", 2020-06-12T02:39:21.062 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.1389999999999999e-06 2020-06-12T02:39:21.062 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.062 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.062 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2020-06-12T02:39:21.062 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:15.435831+0000", 2020-06-12T02:39:21.063 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0167041139999999 2020-06-12T02:39:21.063 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.063 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.063 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:21.063 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:16.452535+0000", 2020-06-12T02:39:21.063 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.6076999999999995e-05 2020-06-12T02:39:21.064 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.064 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.064 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:21.064 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:16.452631+0000", 2020-06-12T02:39:21.064 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.845355595 2020-06-12T02:39:21.065 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:21.065 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:21.065 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:21.065 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.065 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.066 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:23513 3.9s0 3:977e4b9d:::smithi10729407-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e84)", 2020-06-12T02:39:21.066 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:39:14.607360+0000", 2020-06-12T02:39:21.066 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 6.1307863679999999, 2020-06-12T02:39:21.066 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.847083005, 2020-06-12T02:39:21.066 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:21.066 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "started", 2020-06-12T02:39:21.067 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:21.067 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:21.067 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:21.067 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 23513 2020-06-12T02:39:21.067 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.068 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:21.068 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.068 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:21.068 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:14.607360+0000", 2020-06-12T02:39:21.068 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:21.068 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.069 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.069 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:21.069 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:14.607360+0000", 2020-06-12T02:39:21.069 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.2575e-05 2020-06-12T02:39:21.069 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.070 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.070 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:21.070 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:14.607383+0000", 2020-06-12T02:39:21.070 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.7680000000000002e-06 2020-06-12T02:39:21.070 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.070 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.071 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:21.071 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:14.607387+0000", 2020-06-12T02:39:21.071 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.3730000000000004e-06 2020-06-12T02:39:21.071 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.071 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.072 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:21.072 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:14.607394+0000", 2020-06-12T02:39:21.072 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 2.486e-06 2020-06-12T02:39:21.072 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.072 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.072 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:21.073 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:14.607396+0000", 2020-06-12T02:39:21.073 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0047899980000000002 2020-06-12T02:39:21.073 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.073 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.073 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:21.074 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:14.612186+0000", 2020-06-12T02:39:21.074 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00010095200000000001 2020-06-12T02:39:21.074 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.074 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.074 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:21.075 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:14.612287+0000", 2020-06-12T02:39:21.075 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.002412675 2020-06-12T02:39:21.075 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.075 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.075 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:21.075 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:14.614700+0000", 2020-06-12T02:39:21.076 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 9.7948000000000005e-05 2020-06-12T02:39:21.076 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.076 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.076 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:21.076 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:14.614798+0000", 2020-06-12T02:39:21.076 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.82104995599999997 2020-06-12T02:39:21.077 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.077 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.077 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:21.077 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:15.435848+0000", 2020-06-12T02:39:21.077 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 3.84e-07 2020-06-12T02:39:21.078 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.078 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.078 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for peered", 2020-06-12T02:39:21.078 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:15.435848+0000", 2020-06-12T02:39:21.078 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.0174151870000001 2020-06-12T02:39:21.079 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.079 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.079 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:21.079 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:16.453264+0000", 2020-06-12T02:39:21.079 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00014553799999999999 2020-06-12T02:39:21.079 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.080 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.080 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:21.080 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:16.453409+0000", 2020-06-12T02:39:21.080 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0010341650000000001 2020-06-12T02:39:21.080 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.081 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.081 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:21.081 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:16.454443+0000", 2020-06-12T02:39:21.081 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 1.847083005 2020-06-12T02:39:21.081 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:21.081 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:21.082 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:21.082 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.082 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.082 INFO:teuthology.orchestra.run.smithi188.stdout: "description": "osd_op(client.4512.0:24785 3.3s0 3:c2ca77da:::smithi10729407-13:head [append 0~458752] snapc 0=[] ondisk+write+known_if_redirected e87)", 2020-06-12T02:39:21.082 INFO:teuthology.orchestra.run.smithi188.stdout: "initiated_at": "2020-06-12T02:39:20.193282+0000", 2020-06-12T02:39:21.083 INFO:teuthology.orchestra.run.smithi188.stdout: "age": 0.54486411400000001, 2020-06-12T02:39:21.083 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021421272000000002, 2020-06-12T02:39:21.083 INFO:teuthology.orchestra.run.smithi188.stdout: "type_data": { 2020-06-12T02:39:21.083 INFO:teuthology.orchestra.run.smithi188.stdout: "flag_point": "commit sent; apply or cleanup", 2020-06-12T02:39:21.083 INFO:teuthology.orchestra.run.smithi188.stdout: "client_info": { 2020-06-12T02:39:21.083 INFO:teuthology.orchestra.run.smithi188.stdout: "client": "client.4512", 2020-06-12T02:39:21.084 INFO:teuthology.orchestra.run.smithi188.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:21.084 INFO:teuthology.orchestra.run.smithi188.stdout: "tid": 24785 2020-06-12T02:39:21.084 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.084 INFO:teuthology.orchestra.run.smithi188.stdout: "events": [ 2020-06-12T02:39:21.084 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.085 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "initiated", 2020-06-12T02:39:21.085 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:20.193282+0000", 2020-06-12T02:39:21.085 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0 2020-06-12T02:39:21.085 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.085 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.085 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "throttled", 2020-06-12T02:39:21.086 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:20.193282+0000", 2020-06-12T02:39:21.086 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00068734299999999998 2020-06-12T02:39:21.086 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.086 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.086 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "header_read", 2020-06-12T02:39:21.087 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:20.193970+0000", 2020-06-12T02:39:21.087 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 4.7249999999999997e-06 2020-06-12T02:39:21.087 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.087 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.087 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "all_read", 2020-06-12T02:39:21.087 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:20.193975+0000", 2020-06-12T02:39:21.088 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 7.7389999999999999e-06 2020-06-12T02:39:21.088 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.088 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.088 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "dispatched", 2020-06-12T02:39:21.088 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:20.193982+0000", 2020-06-12T02:39:21.089 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.0189999999999997e-06 2020-06-12T02:39:21.089 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.089 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.089 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "queued_for_pg", 2020-06-12T02:39:21.089 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:20.193987+0000", 2020-06-12T02:39:21.090 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.001634103 2020-06-12T02:39:21.090 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.090 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.090 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:21.090 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:20.195621+0000", 2020-06-12T02:39:21.090 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.000106067 2020-06-12T02:39:21.091 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.091 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.091 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:21.091 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:20.195727+0000", 2020-06-12T02:39:21.091 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0038180029999999999 2020-06-12T02:39:21.092 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.092 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.092 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "reached_pg", 2020-06-12T02:39:21.092 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:20.199545+0000", 2020-06-12T02:39:21.092 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 8.8374000000000005e-05 2020-06-12T02:39:21.092 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.093 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.093 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "started", 2020-06-12T02:39:21.093 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:20.199634+0000", 2020-06-12T02:39:21.093 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.00048978200000000002 2020-06-12T02:39:21.093 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.094 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.094 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_started", 2020-06-12T02:39:21.094 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:20.200124+0000", 2020-06-12T02:39:21.094 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0091135850000000004 2020-06-12T02:39:21.094 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.094 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.095 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "sub_op_committed", 2020-06-12T02:39:21.095 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:20.209237+0000", 2020-06-12T02:39:21.095 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.0054110879999999997 2020-06-12T02:39:21.095 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.095 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.096 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "commit_sent", 2020-06-12T02:39:21.096 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:20.214648+0000", 2020-06-12T02:39:21.096 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 5.5444e-05 2020-06-12T02:39:21.096 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2020-06-12T02:39:21.096 INFO:teuthology.orchestra.run.smithi188.stdout: { 2020-06-12T02:39:21.096 INFO:teuthology.orchestra.run.smithi188.stdout: "event": "done", 2020-06-12T02:39:21.097 INFO:teuthology.orchestra.run.smithi188.stdout: "time": "2020-06-12T02:39:20.214704+0000", 2020-06-12T02:39:21.097 INFO:teuthology.orchestra.run.smithi188.stdout: "duration": 0.021421272000000002 2020-06-12T02:39:21.097 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:21.097 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:21.097 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:21.098 INFO:teuthology.orchestra.run.smithi188.stdout: } 2020-06-12T02:39:21.098 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2020-06-12T02:39:21.098 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:39:21.116 INFO:tasks.rados.rados.0.smithi107.stdout:11726: writing smithi10729407-2 from 778240 to 1581056 tid 2 2020-06-12T02:39:21.117 INFO:tasks.rados.rados.0.smithi107.stdout:11726: writing smithi10729407-2 from 1581056 to 1982464 tid 3 2020-06-12T02:39:21.117 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:39:21.117 INFO:tasks.rados.rados.0.smithi107.stdout:11727: seq_num 11726 ranges {1662976=794624,2457600=688128,3145728=516096} 2020-06-12T02:39:21.117 INFO:tasks.rados.rados.0.smithi107.stdout:11727: writing smithi10729407-3 from 1662976 to 2457600 tid 1 2020-06-12T02:39:21.117 INFO:tasks.rados.rados.0.smithi107.stdout:11727: writing smithi10729407-3 from 2457600 to 3145728 tid 2 2020-06-12T02:39:21.118 INFO:tasks.rados.rados.0.smithi107.stdout:11727: writing smithi10729407-3 from 3145728 to 3661824 tid 3 2020-06-12T02:39:21.118 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:39:21.118 INFO:tasks.rados.rados.0.smithi107.stdout:11728: seq_num 11727 ranges {0=409600,409600=573440,983040=491520,1474560=540672} 2020-06-12T02:39:21.118 INFO:tasks.rados.rados.0.smithi107.stdout:11728: writing smithi10729407-1 from 0 to 409600 tid 1 2020-06-12T02:39:21.118 INFO:tasks.rados.rados.0.smithi107.stdout:11728: writing smithi10729407-1 from 409600 to 983040 tid 2 2020-06-12T02:39:21.119 INFO:tasks.rados.rados.0.smithi107.stdout:11728: writing smithi10729407-1 from 983040 to 1474560 tid 3 2020-06-12T02:39:21.119 INFO:tasks.rados.rados.0.smithi107.stdout:11728: writing smithi10729407-1 from 1474560 to 2015232 tid 4 2020-06-12T02:39:21.119 INFO:tasks.rados.rados.0.smithi107.stdout:11729: delete oid 14 current snap is 0 2020-06-12T02:39:21.119 INFO:tasks.rados.rados.0.smithi107.stdout:11723: finishing write tid 6 to smithi10729407-18 2020-06-12T02:39:21.119 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1944 (ObjNum 11722 snap 0 seq_num 11722) dirty exists 2020-06-12T02:39:21.120 INFO:tasks.rados.rados.0.smithi107.stdout:11723: left oid 18 (ObjNum 11722 snap 0 seq_num 11722) 2020-06-12T02:39:21.120 INFO:tasks.rados.rados.0.smithi107.stdout:11726: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:21.120 INFO:tasks.rados.rados.0.smithi107.stdout:11726: finishing write tid 2 to smithi10729407-2 2020-06-12T02:39:21.120 INFO:tasks.rados.rados.0.smithi107.stdout:11726: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:21.120 INFO:tasks.rados.rados.0.smithi107.stdout:11726: finishing write tid 5 to smithi10729407-2 2020-06-12T02:39:21.120 INFO:tasks.rados.rados.0.smithi107.stdout:11726: finishing write tid 6 to smithi10729407-2 2020-06-12T02:39:21.121 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 1108 (ObjNum 11725 snap 0 seq_num 11725) dirty exists 2020-06-12T02:39:21.121 INFO:tasks.rados.rados.0.smithi107.stdout:11726: left oid 2 (ObjNum 11725 snap 0 seq_num 11725) 2020-06-12T02:39:21.121 INFO:tasks.rados.rados.0.smithi107.stdout:11728: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:21.121 INFO:tasks.rados.rados.0.smithi107.stdout:11727: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:21.121 INFO:tasks.rados.rados.0.smithi107.stdout:11727: finishing write tid 2 to smithi10729407-3 2020-06-12T02:39:21.122 INFO:tasks.rados.rados.0.smithi107.stdout:11727: finishing write tid 3 to smithi10729407-3 2020-06-12T02:39:21.122 INFO:tasks.rados.rados.0.smithi107.stdout:11727: finishing write tid 5 to smithi10729407-3 2020-06-12T02:39:21.122 INFO:tasks.rados.rados.0.smithi107.stdout:11727: finishing write tid 6 to smithi10729407-3 2020-06-12T02:39:21.122 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 2071 (ObjNum 11726 snap 0 seq_num 11726) dirty exists 2020-06-12T02:39:21.122 INFO:tasks.rados.rados.0.smithi107.stdout:11727: left oid 3 (ObjNum 11726 snap 0 seq_num 11726) 2020-06-12T02:39:21.122 INFO:tasks.rados.rados.0.smithi107.stdout:11723: done (4 left) 2020-06-12T02:39:21.123 INFO:tasks.rados.rados.0.smithi107.stdout:11726: done (3 left) 2020-06-12T02:39:21.123 INFO:tasks.rados.rados.0.smithi107.stdout:11727: done (2 left) 2020-06-12T02:39:21.123 INFO:tasks.rados.rados.0.smithi107.stdout:11729: done (1 left) 2020-06-12T02:39:21.123 INFO:tasks.rados.rados.0.smithi107.stdout:11730: delete oid 16 current snap is 0 2020-06-12T02:39:21.123 INFO:tasks.rados.rados.0.smithi107.stdout:11730: done (1 left) 2020-06-12T02:39:21.124 INFO:tasks.rados.rados.0.smithi107.stdout:11731: delete oid 12 current snap is 0 2020-06-12T02:39:21.124 INFO:tasks.rados.rados.0.smithi107.stdout:11728: finishing write tid 2 to smithi10729407-1 2020-06-12T02:39:21.124 INFO:tasks.rados.rados.0.smithi107.stdout:11728: finishing write tid 3 to smithi10729407-1 2020-06-12T02:39:21.124 INFO:tasks.rados.rados.0.smithi107.stdout:11728: finishing write tid 4 to smithi10729407-1 2020-06-12T02:39:21.124 INFO:tasks.rados.rados.0.smithi107.stdout:11731: done (1 left) 2020-06-12T02:39:21.124 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 17 current snap is 0 2020-06-12T02:39:21.125 INFO:tasks.rados.rados.0.smithi107.stdout:11732: seq_num 11731 ranges {1327104=475136,1802240=557056,2359296=516096,2875392=434176,3309568=98304} 2020-06-12T02:39:21.125 INFO:tasks.rados.rados.0.smithi107.stdout:11732: writing smithi10729407-17 from 1327104 to 1802240 tid 1 2020-06-12T02:39:21.125 INFO:tasks.rados.rados.0.smithi107.stdout:11732: writing smithi10729407-17 from 1802240 to 2359296 tid 2 2020-06-12T02:39:21.125 INFO:tasks.rados.rados.0.smithi107.stdout:11732: writing smithi10729407-17 from 2359296 to 2875392 tid 3 2020-06-12T02:39:21.125 INFO:tasks.rados.rados.0.smithi107.stdout:11732: writing smithi10729407-17 from 2875392 to 3309568 tid 4 2020-06-12T02:39:21.126 INFO:tasks.rados.rados.0.smithi107.stdout:11732: writing smithi10729407-17 from 3309568 to 3407872 tid 5 2020-06-12T02:39:21.126 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:39:21.126 INFO:tasks.rados.rados.0.smithi107.stdout:11733: seq_num 11732 ranges {0=491520,491520=442368,933888=679936,1613824=483328} 2020-06-12T02:39:21.126 INFO:tasks.rados.rados.0.smithi107.stdout:11733: writing smithi10729407-14 from 0 to 491520 tid 1 2020-06-12T02:39:21.126 INFO:tasks.rados.rados.0.smithi107.stdout:11733: writing smithi10729407-14 from 491520 to 933888 tid 2 2020-06-12T02:39:21.127 INFO:tasks.rados.rados.0.smithi107.stdout:11733: writing smithi10729407-14 from 933888 to 1613824 tid 3 2020-06-12T02:39:21.127 INFO:tasks.rados.rados.0.smithi107.stdout:11733: writing smithi10729407-14 from 1613824 to 2097152 tid 4 2020-06-12T02:39:21.127 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:39:21.127 INFO:tasks.rados.rados.0.smithi107.stdout:11734: seq_num 11733 ranges {0=507904,507904=729088,1236992=483328,1720320=393216} 2020-06-12T02:39:21.127 INFO:tasks.rados.rados.0.smithi107.stdout:11734: writing smithi10729407-20 from 0 to 507904 tid 1 2020-06-12T02:39:21.127 INFO:tasks.rados.rados.0.smithi107.stdout:11734: writing smithi10729407-20 from 507904 to 1236992 tid 2 2020-06-12T02:39:21.128 INFO:tasks.rados.rados.0.smithi107.stdout:11734: writing smithi10729407-20 from 1236992 to 1720320 tid 3 2020-06-12T02:39:21.128 INFO:tasks.rados.rados.0.smithi107.stdout:11734: writing smithi10729407-20 from 1720320 to 2113536 tid 4 2020-06-12T02:39:21.128 INFO:tasks.rados.rados.0.smithi107.stdout:11735: delete oid 10 current snap is 0 2020-06-12T02:39:21.128 INFO:tasks.rados.rados.0.smithi107.stdout:11728: finishing write tid 6 to smithi10729407-1 2020-06-12T02:39:21.128 INFO:tasks.rados.rados.0.smithi107.stdout:11728: finishing write tid 7 to smithi10729407-1 2020-06-12T02:39:21.129 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 2146 (ObjNum 11727 snap 0 seq_num 11727) dirty exists 2020-06-12T02:39:21.129 INFO:tasks.rados.rados.0.smithi107.stdout:11728: left oid 1 (ObjNum 11727 snap 0 seq_num 11727) 2020-06-12T02:39:21.129 INFO:tasks.rados.rados.0.smithi107.stdout:11732: finishing write tid 1 to smithi10729407-17 2020-06-12T02:39:21.129 INFO:tasks.rados.rados.0.smithi107.stdout:11732: finishing write tid 2 to smithi10729407-17 2020-06-12T02:39:21.129 INFO:tasks.rados.rados.0.smithi107.stdout:11732: finishing write tid 3 to smithi10729407-17 2020-06-12T02:39:21.129 INFO:tasks.rados.rados.0.smithi107.stdout:11732: finishing write tid 4 to smithi10729407-17 2020-06-12T02:39:21.130 INFO:tasks.rados.rados.0.smithi107.stdout:11732: finishing write tid 5 to smithi10729407-17 2020-06-12T02:39:21.130 INFO:tasks.rados.rados.0.smithi107.stdout:11732: finishing write tid 7 to smithi10729407-17 2020-06-12T02:39:21.130 INFO:tasks.rados.rados.0.smithi107.stdout:11733: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:21.130 INFO:tasks.rados.rados.0.smithi107.stdout:11732: finishing write tid 8 to smithi10729407-17 2020-06-12T02:39:21.130 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 2185 (ObjNum 11731 snap 0 seq_num 11731) dirty exists 2020-06-12T02:39:21.131 INFO:tasks.rados.rados.0.smithi107.stdout:11732: left oid 17 (ObjNum 11731 snap 0 seq_num 11731) 2020-06-12T02:39:21.131 INFO:tasks.rados.rados.0.smithi107.stdout:11733: finishing write tid 2 to smithi10729407-14 2020-06-12T02:39:21.131 INFO:tasks.rados.rados.0.smithi107.stdout:11733: finishing write tid 3 to smithi10729407-14 2020-06-12T02:39:21.131 INFO:tasks.rados.rados.0.smithi107.stdout:11733: finishing write tid 4 to smithi10729407-14 2020-06-12T02:39:21.131 INFO:tasks.rados.rados.0.smithi107.stdout:11733: finishing write tid 6 to smithi10729407-14 2020-06-12T02:39:21.131 INFO:tasks.rados.rados.0.smithi107.stdout:11734: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:21.132 INFO:tasks.rados.rados.0.smithi107.stdout:11733: finishing write tid 7 to smithi10729407-14 2020-06-12T02:39:21.132 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1824 (ObjNum 11732 snap 0 seq_num 11732) dirty exists 2020-06-12T02:39:21.132 INFO:tasks.rados.rados.0.smithi107.stdout:11733: left oid 14 (ObjNum 11732 snap 0 seq_num 11732) 2020-06-12T02:39:21.132 INFO:tasks.rados.rados.0.smithi107.stdout:11728: done (4 left) 2020-06-12T02:39:21.132 INFO:tasks.rados.rados.0.smithi107.stdout:11732: done (3 left) 2020-06-12T02:39:21.133 INFO:tasks.rados.rados.0.smithi107.stdout:11733: done (2 left) 2020-06-12T02:39:21.133 INFO:tasks.rados.rados.0.smithi107.stdout:11735: done (1 left) 2020-06-12T02:39:21.133 INFO:tasks.rados.rados.0.smithi107.stdout:11736: delete oid 15 current snap is 0 2020-06-12T02:39:21.133 INFO:tasks.rados.rados.0.smithi107.stdout:11736: done (1 left) 2020-06-12T02:39:21.133 INFO:tasks.rados.rados.0.smithi107.stdout:11737: delete oid 4 current snap is 0 2020-06-12T02:39:21.133 INFO:tasks.rados.rados.0.smithi107.stdout:11734: finishing write tid 2 to smithi10729407-20 2020-06-12T02:39:21.134 INFO:tasks.rados.rados.0.smithi107.stdout:11734: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:21.134 INFO:tasks.rados.rados.0.smithi107.stdout:11734: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:21.134 INFO:tasks.rados.rados.0.smithi107.stdout:11737: done (1 left) 2020-06-12T02:39:21.134 INFO:tasks.rados.rados.0.smithi107.stdout:11738: delete oid 10 current snap is 0 2020-06-12T02:39:21.134 INFO:tasks.rados.rados.0.smithi107.stdout:11738: done (1 left) 2020-06-12T02:39:21.135 INFO:tasks.rados.rados.0.smithi107.stdout:11739: delete oid 15 current snap is 0 2020-06-12T02:39:21.135 INFO:tasks.rados.rados.0.smithi107.stdout:11734: finishing write tid 6 to smithi10729407-20 2020-06-12T02:39:21.135 INFO:tasks.rados.rados.0.smithi107.stdout:11739: done (1 left) 2020-06-12T02:39:21.135 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:39:21.135 INFO:tasks.rados.rados.0.smithi107.stdout:11740: seq_num 11739 ranges {0=614400,614400=483328,1097728=483328,1581056=540672,2121728=98304} 2020-06-12T02:39:21.135 INFO:tasks.rados.rados.0.smithi107.stdout:11740: writing smithi10729407-5 from 0 to 614400 tid 1 2020-06-12T02:39:21.136 INFO:tasks.rados.rados.0.smithi107.stdout:11740: writing smithi10729407-5 from 614400 to 1097728 tid 2 2020-06-12T02:39:21.136 INFO:tasks.rados.rados.0.smithi107.stdout:11740: writing smithi10729407-5 from 1097728 to 1581056 tid 3 2020-06-12T02:39:21.136 INFO:tasks.rados.rados.0.smithi107.stdout:11740: writing smithi10729407-5 from 1581056 to 2121728 tid 4 2020-06-12T02:39:21.136 INFO:tasks.rados.rados.0.smithi107.stdout:11740: writing smithi10729407-5 from 2121728 to 2220032 tid 5 2020-06-12T02:39:21.136 INFO:tasks.rados.rados.0.smithi107.stdout:11741: delete oid 12 current snap is 0 2020-06-12T02:39:21.137 INFO:tasks.rados.rados.0.smithi107.stdout:11734: finishing write tid 7 to smithi10729407-20 2020-06-12T02:39:21.137 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 909 (ObjNum 11733 snap 0 seq_num 11733) dirty exists 2020-06-12T02:39:21.137 INFO:tasks.rados.rados.0.smithi107.stdout:11734: left oid 20 (ObjNum 11733 snap 0 seq_num 11733) 2020-06-12T02:39:21.137 INFO:tasks.rados.rados.0.smithi107.stdout:11740: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:21.137 INFO:tasks.rados.rados.0.smithi107.stdout:11740: finishing write tid 2 to smithi10729407-5 2020-06-12T02:39:21.138 INFO:tasks.rados.rados.0.smithi107.stdout:11740: finishing write tid 3 to smithi10729407-5 2020-06-12T02:39:21.138 INFO:tasks.rados.rados.0.smithi107.stdout:11734: done (2 left) 2020-06-12T02:39:21.138 INFO:tasks.rados.rados.0.smithi107.stdout:11741: done (1 left) 2020-06-12T02:39:21.138 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:39:21.138 INFO:tasks.rados.rados.0.smithi107.stdout:11742: seq_num 11741 ranges {0=647168,647168=655360,1302528=442368,1744896=507904} 2020-06-12T02:39:21.138 INFO:tasks.rados.rados.0.smithi107.stdout:11742: writing smithi10729407-15 from 0 to 647168 tid 1 2020-06-12T02:39:21.139 INFO:tasks.rados.rados.0.smithi107.stdout:11742: writing smithi10729407-15 from 647168 to 1302528 tid 2 2020-06-12T02:39:21.139 INFO:tasks.rados.rados.0.smithi107.stdout:11742: writing smithi10729407-15 from 1302528 to 1744896 tid 3 2020-06-12T02:39:21.139 INFO:tasks.rados.rados.0.smithi107.stdout:11742: writing smithi10729407-15 from 1744896 to 2252800 tid 4 2020-06-12T02:39:21.139 INFO:tasks.rados.rados.0.smithi107.stdout:11743: delete oid 4 current snap is 0 2020-06-12T02:39:21.139 INFO:tasks.rados.rados.0.smithi107.stdout:11740: finishing write tid 4 to smithi10729407-5 2020-06-12T02:39:21.139 INFO:tasks.rados.rados.0.smithi107.stdout:11740: finishing write tid 5 to smithi10729407-5 2020-06-12T02:39:21.140 INFO:tasks.rados.rados.0.smithi107.stdout:11740: finishing write tid 7 to smithi10729407-5 2020-06-12T02:39:21.140 INFO:tasks.rados.rados.0.smithi107.stdout:11740: finishing write tid 8 to smithi10729407-5 2020-06-12T02:39:21.140 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 1119 (ObjNum 11739 snap 0 seq_num 11739) dirty exists 2020-06-12T02:39:21.140 INFO:tasks.rados.rados.0.smithi107.stdout:11740: left oid 5 (ObjNum 11739 snap 0 seq_num 11739) 2020-06-12T02:39:21.140 INFO:tasks.rados.rados.0.smithi107.stdout:11742: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:21.141 INFO:tasks.rados.rados.0.smithi107.stdout:11740: done (2 left) 2020-06-12T02:39:21.141 INFO:tasks.rados.rados.0.smithi107.stdout:11743: done (1 left) 2020-06-12T02:39:21.141 INFO:tasks.rados.rados.0.smithi107.stdout:11744: delete oid 20 current snap is 0 2020-06-12T02:39:21.141 INFO:tasks.rados.rados.0.smithi107.stdout:11744: done (1 left) 2020-06-12T02:39:21.141 INFO:tasks.rados.rados.0.smithi107.stdout:11745: delete oid 10 current snap is 0 2020-06-12T02:39:21.141 INFO:tasks.rados.rados.0.smithi107.stdout:11745: done (1 left) 2020-06-12T02:39:21.142 INFO:tasks.rados.rados.0.smithi107.stdout:11746: delete oid 1 current snap is 0 2020-06-12T02:39:21.142 INFO:tasks.rados.rados.0.smithi107.stdout:11742: finishing write tid 2 to smithi10729407-15 2020-06-12T02:39:21.142 INFO:tasks.rados.rados.0.smithi107.stdout:11742: finishing write tid 3 to smithi10729407-15 2020-06-12T02:39:21.142 INFO:tasks.rados.rados.0.smithi107.stdout:11746: done (1 left) 2020-06-12T02:39:21.142 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:39:21.143 INFO:tasks.rados.rados.0.smithi107.stdout:11747: seq_num 11746 ranges {3661824=729088,4390912=524288,4915200=630784,5545984=450560} 2020-06-12T02:39:21.143 INFO:tasks.rados.rados.0.smithi107.stdout:11747: writing smithi10729407-3 from 3661824 to 4390912 tid 1 2020-06-12T02:39:21.143 INFO:tasks.rados.rados.0.smithi107.stdout:11747: writing smithi10729407-3 from 4390912 to 4915200 tid 2 2020-06-12T02:39:21.143 INFO:tasks.rados.rados.0.smithi107.stdout:11747: writing smithi10729407-3 from 4915200 to 5545984 tid 3 2020-06-12T02:39:21.143 INFO:tasks.rados.rados.0.smithi107.stdout:11747: writing smithi10729407-3 from 5545984 to 5996544 tid 4 2020-06-12T02:39:21.143 INFO:tasks.rados.rados.0.smithi107.stdout:11748: delete oid 14 current snap is 0 2020-06-12T02:39:21.144 INFO:tasks.rados.rados.0.smithi107.stdout:11742: finishing write tid 4 to smithi10729407-15 2020-06-12T02:39:21.144 INFO:tasks.rados.rados.0.smithi107.stdout:11742: finishing write tid 6 to smithi10729407-15 2020-06-12T02:39:21.144 INFO:tasks.rados.rados.0.smithi107.stdout:11742: finishing write tid 7 to smithi10729407-15 2020-06-12T02:39:21.144 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 2216 (ObjNum 11741 snap 0 seq_num 11741) dirty exists 2020-06-12T02:39:21.144 INFO:tasks.rados.rados.0.smithi107.stdout:11742: left oid 15 (ObjNum 11741 snap 0 seq_num 11741) 2020-06-12T02:39:21.145 INFO:tasks.rados.rados.0.smithi107.stdout:11747: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:21.145 INFO:tasks.rados.rados.0.smithi107.stdout:11742: done (2 left) 2020-06-12T02:39:21.145 INFO:tasks.rados.rados.0.smithi107.stdout:11748: done (1 left) 2020-06-12T02:39:21.145 INFO:tasks.rados.rados.0.smithi107.stdout:11749: delete oid 7 current snap is 0 2020-06-12T02:39:21.145 INFO:tasks.rados.rados.0.smithi107.stdout:11747: finishing write tid 2 to smithi10729407-3 2020-06-12T02:39:21.145 INFO:tasks.rados.rados.0.smithi107.stdout:11749: done (1 left) 2020-06-12T02:39:21.146 INFO:tasks.rados.rados.0.smithi107.stdout:11750: delete oid 15 current snap is 0 2020-06-12T02:39:21.146 INFO:tasks.rados.rados.0.smithi107.stdout:11750: done (1 left) 2020-06-12T02:39:21.146 INFO:tasks.rados.rados.0.smithi107.stdout:11751: delete oid 16 current snap is 0 2020-06-12T02:39:21.146 INFO:tasks.rados.rados.0.smithi107.stdout:11747: finishing write tid 3 to smithi10729407-3 2020-06-12T02:39:21.146 INFO:tasks.rados.rados.0.smithi107.stdout:11747: finishing write tid 4 to smithi10729407-3 2020-06-12T02:39:21.147 INFO:tasks.rados.rados.0.smithi107.stdout:11747: finishing write tid 6 to smithi10729407-3 2020-06-12T02:39:21.147 INFO:tasks.rados.rados.0.smithi107.stdout:11751: done (1 left) 2020-06-12T02:39:21.147 INFO:tasks.rados.rados.0.smithi107.stdout:11752: delete oid 5 current snap is 0 2020-06-12T02:39:21.147 INFO:tasks.rados.rados.0.smithi107.stdout:11747: finishing write tid 7 to smithi10729407-3 2020-06-12T02:39:21.147 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 2077 (ObjNum 11746 snap 0 seq_num 11746) dirty exists 2020-06-12T02:39:21.148 INFO:tasks.rados.rados.0.smithi107.stdout:11747: left oid 3 (ObjNum 11746 snap 0 seq_num 11746) 2020-06-12T02:39:21.148 INFO:tasks.rados.rados.0.smithi107.stdout:11747: done (1 left) 2020-06-12T02:39:21.148 INFO:tasks.rados.rados.0.smithi107.stdout:11752: done (0 left) 2020-06-12T02:39:21.148 INFO:tasks.rados.rados.0.smithi107.stdout:11753: delete oid 19 current snap is 0 2020-06-12T02:39:21.148 INFO:tasks.rados.rados.0.smithi107.stdout:11753: done (0 left) 2020-06-12T02:39:21.148 INFO:tasks.rados.rados.0.smithi107.stdout:11754: delete oid 4 current snap is 0 2020-06-12T02:39:21.149 INFO:tasks.rados.rados.0.smithi107.stdout:11754: done (0 left) 2020-06-12T02:39:21.149 INFO:tasks.rados.rados.0.smithi107.stdout:11755: delete oid 9 current snap is 0 2020-06-12T02:39:21.149 INFO:tasks.rados.rados.0.smithi107.stdout:11755: done (0 left) 2020-06-12T02:39:21.149 INFO:tasks.rados.rados.0.smithi107.stdout:11756: delete oid 6 current snap is 0 2020-06-12T02:39:21.149 INFO:tasks.rados.rados.0.smithi107.stdout:11756: done (0 left) 2020-06-12T02:39:21.150 INFO:tasks.rados.rados.0.smithi107.stdout:11757: delete oid 1 current snap is 0 2020-06-12T02:39:21.150 INFO:tasks.rados.rados.0.smithi107.stdout:11757: done (0 left) 2020-06-12T02:39:21.150 INFO:tasks.rados.rados.0.smithi107.stdout:11758: delete oid 4 current snap is 0 2020-06-12T02:39:21.150 INFO:tasks.rados.rados.0.smithi107.stdout:11758: done (0 left) 2020-06-12T02:39:21.150 INFO:tasks.rados.rados.0.smithi107.stdout:11759: delete oid 12 current snap is 0 2020-06-12T02:39:21.150 INFO:tasks.rados.rados.0.smithi107.stdout:11759: done (0 left) 2020-06-12T02:39:21.151 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:39:21.151 INFO:tasks.rados.rados.0.smithi107.stdout:11760: seq_num 11759 ranges {0=147456} 2020-06-12T02:39:21.151 INFO:tasks.rados.rados.0.smithi107.stdout:11760: writing smithi10729407-12 from 0 to 147456 tid 1 2020-06-12T02:39:21.151 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:39:21.151 INFO:tasks.rados.rados.0.smithi107.stdout:11761: seq_num 11760 ranges {0=163840} 2020-06-12T02:39:21.152 INFO:tasks.rados.rados.0.smithi107.stdout:11761: writing smithi10729407-8 from 0 to 163840 tid 1 2020-06-12T02:39:21.152 INFO:tasks.rados.rados.0.smithi107.stdout:11762: delete oid 19 current snap is 0 2020-06-12T02:39:21.152 INFO:tasks.rados.rados.0.smithi107.stdout:11760: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:21.152 INFO:tasks.rados.rados.0.smithi107.stdout:11760: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:21.152 INFO:tasks.rados.rados.0.smithi107.stdout:11762: done (2 left) 2020-06-12T02:39:21.152 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:39:21.153 INFO:tasks.rados.rados.0.smithi107.stdout:11763: seq_num 11762 ranges {0=196608} 2020-06-12T02:39:21.153 INFO:tasks.rados.rados.0.smithi107.stdout:11763: writing smithi10729407-13 from 0 to 196608 tid 1 2020-06-12T02:39:21.153 INFO:tasks.rados.rados.0.smithi107.stdout:11764: delete oid 4 current snap is 0 2020-06-12T02:39:21.153 INFO:tasks.rados.rados.0.smithi107.stdout:11760: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:21.153 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 2185 (ObjNum 11759 snap 0 seq_num 11759) dirty exists 2020-06-12T02:39:21.154 INFO:tasks.rados.rados.0.smithi107.stdout:11760: left oid 12 (ObjNum 11759 snap 0 seq_num 11759) 2020-06-12T02:39:21.154 INFO:tasks.rados.rados.0.smithi107.stdout:11761: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:21.154 INFO:tasks.rados.rados.0.smithi107.stdout:11761: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:21.154 INFO:tasks.rados.rados.0.smithi107.stdout:11763: finishing write tid 1 to smithi10729407-13 2020-06-12T02:39:21.154 INFO:tasks.rados.rados.0.smithi107.stdout:11761: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:21.154 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 2219 (ObjNum 11760 snap 0 seq_num 11760) dirty exists 2020-06-12T02:39:21.155 INFO:tasks.rados.rados.0.smithi107.stdout:11761: left oid 8 (ObjNum 11760 snap 0 seq_num 11760) 2020-06-12T02:39:21.155 INFO:tasks.rados.rados.0.smithi107.stdout:11760: done (3 left) 2020-06-12T02:39:21.155 INFO:tasks.rados.rados.0.smithi107.stdout:11761: done (2 left) 2020-06-12T02:39:21.155 INFO:tasks.rados.rados.0.smithi107.stdout:11764: done (1 left) 2020-06-12T02:39:21.155 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:39:21.156 INFO:tasks.rados.rados.0.smithi107.stdout:11765: seq_num 11764 ranges {0=229376} 2020-06-12T02:39:21.156 INFO:tasks.rados.rados.0.smithi107.stdout:11765: writing smithi10729407-1 from 0 to 229376 tid 1 2020-06-12T02:39:21.156 INFO:tasks.rados.rados.0.smithi107.stdout:11766: delete oid 12 current snap is 0 2020-06-12T02:39:21.156 INFO:tasks.rados.rados.0.smithi107.stdout:11766: done (2 left) 2020-06-12T02:39:21.156 INFO:tasks.rados.rados.0.smithi107.stdout:11767: delete oid 19 current snap is 0 2020-06-12T02:39:21.156 INFO:tasks.rados.rados.0.smithi107.stdout:11763: finishing write tid 3 to smithi10729407-13 2020-06-12T02:39:21.157 INFO:tasks.rados.rados.0.smithi107.stdout:11763: finishing write tid 4 to smithi10729407-13 2020-06-12T02:39:21.157 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1843 (ObjNum 11762 snap 0 seq_num 11762) dirty exists 2020-06-12T02:39:21.157 INFO:tasks.rados.rados.0.smithi107.stdout:11763: left oid 13 (ObjNum 11762 snap 0 seq_num 11762) 2020-06-12T02:39:21.157 INFO:tasks.rados.rados.0.smithi107.stdout:11765: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:21.157 INFO:tasks.rados.rados.0.smithi107.stdout:11763: done (2 left) 2020-06-12T02:39:21.158 INFO:tasks.rados.rados.0.smithi107.stdout:11767: done (1 left) 2020-06-12T02:39:21.158 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:39:21.158 INFO:tasks.rados.rados.0.smithi107.stdout:11768: seq_num 11767 ranges {0=278528} 2020-06-12T02:39:21.158 INFO:tasks.rados.rados.0.smithi107.stdout:11768: writing smithi10729407-11 from 0 to 278528 tid 1 2020-06-12T02:39:21.158 INFO:tasks.rados.rados.0.smithi107.stdout:11769: delete oid 2 current snap is 0 2020-06-12T02:39:21.158 INFO:tasks.rados.rados.0.smithi107.stdout:11765: finishing write tid 3 to smithi10729407-1 2020-06-12T02:39:21.159 INFO:tasks.rados.rados.0.smithi107.stdout:11769: done (2 left) 2020-06-12T02:39:21.159 INFO:tasks.rados.rados.0.smithi107.stdout:11770: delete oid 12 current snap is 0 2020-06-12T02:39:21.159 INFO:tasks.rados.rados.0.smithi107.stdout:11765: finishing write tid 4 to smithi10729407-1 2020-06-12T02:39:21.159 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 2151 (ObjNum 11764 snap 0 seq_num 11764) dirty exists 2020-06-12T02:39:21.163 INFO:tasks.rados.rados.0.smithi107.stdout:11765: left oid 1 (ObjNum 11764 snap 0 seq_num 11764) 2020-06-12T02:39:21.163 INFO:tasks.rados.rados.0.smithi107.stdout:11768: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:21.163 INFO:tasks.rados.rados.0.smithi107.stdout:11768: finishing write tid 3 to smithi10729407-11 2020-06-12T02:39:21.163 INFO:tasks.rados.rados.0.smithi107.stdout:11765: done (2 left) 2020-06-12T02:39:21.164 INFO:tasks.rados.rados.0.smithi107.stdout:11770: done (1 left) 2020-06-12T02:39:21.164 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:39:21.164 INFO:tasks.rados.rados.0.smithi107.stdout:11771: seq_num 11770 ranges {1933312=327680} 2020-06-12T02:39:21.164 INFO:tasks.rados.rados.0.smithi107.stdout:11771: writing smithi10729407-18 from 1933312 to 2260992 tid 1 2020-06-12T02:39:21.164 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:39:21.165 INFO:tasks.rados.rados.0.smithi107.stdout:11772: seq_num 11771 ranges {229376=344064} 2020-06-12T02:39:21.165 INFO:tasks.rados.rados.0.smithi107.stdout:11772: writing smithi10729407-1 from 229376 to 573440 tid 1 2020-06-12T02:39:21.165 INFO:tasks.rados.rados.0.smithi107.stdout:11773: delete oid 14 current snap is 0 2020-06-12T02:39:21.165 INFO:tasks.rados.rados.0.smithi107.stdout:11768: finishing write tid 4 to smithi10729407-11 2020-06-12T02:39:21.166 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1827 (ObjNum 11767 snap 0 seq_num 11767) dirty exists 2020-06-12T02:39:21.166 INFO:tasks.rados.rados.0.smithi107.stdout:11768: left oid 11 (ObjNum 11767 snap 0 seq_num 11767) 2020-06-12T02:39:21.166 INFO:tasks.rados.rados.0.smithi107.stdout:11771: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:21.166 INFO:tasks.rados.rados.0.smithi107.stdout:11771: finishing write tid 3 to smithi10729407-18 2020-06-12T02:39:21.167 INFO:tasks.rados.rados.0.smithi107.stdout:11768: done (3 left) 2020-06-12T02:39:21.167 INFO:tasks.rados.rados.0.smithi107.stdout:11773: done (2 left) 2020-06-12T02:39:21.167 INFO:tasks.rados.rados.0.smithi107.stdout:11774: delete oid 5 current snap is 0 2020-06-12T02:39:21.167 INFO:tasks.rados.rados.0.smithi107.stdout:11771: finishing write tid 4 to smithi10729407-18 2020-06-12T02:39:21.167 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1946 (ObjNum 11770 snap 0 seq_num 11770) dirty exists 2020-06-12T02:39:21.168 INFO:tasks.rados.rados.0.smithi107.stdout:11771: left oid 18 (ObjNum 11770 snap 0 seq_num 11770) 2020-06-12T02:39:21.168 INFO:tasks.rados.rados.0.smithi107.stdout:11771: done (2 left) 2020-06-12T02:39:21.168 INFO:tasks.rados.rados.0.smithi107.stdout:11774: done (1 left) 2020-06-12T02:39:21.168 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:39:21.168 INFO:tasks.rados.rados.0.smithi107.stdout:11775: seq_num 11774 ranges {0=393216} 2020-06-12T02:39:21.169 INFO:tasks.rados.rados.0.smithi107.stdout:11775: writing smithi10729407-20 from 0 to 393216 tid 1 2020-06-12T02:39:21.169 INFO:tasks.rados.rados.0.smithi107.stdout:11776: delete oid 9 current snap is 0 2020-06-12T02:39:21.169 INFO:tasks.rados.rados.0.smithi107.stdout:11772: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:21.169 INFO:tasks.rados.rados.0.smithi107.stdout:11772: finishing write tid 3 to smithi10729407-1 2020-06-12T02:39:21.169 INFO:tasks.rados.rados.0.smithi107.stdout:11772: finishing write tid 4 to smithi10729407-1 2020-06-12T02:39:21.170 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 2153 (ObjNum 11771 snap 0 seq_num 11771) dirty exists 2020-06-12T02:39:21.170 INFO:tasks.rados.rados.0.smithi107.stdout:11772: left oid 1 (ObjNum 11771 snap 0 seq_num 11771) 2020-06-12T02:39:21.170 INFO:tasks.rados.rados.0.smithi107.stdout:11772: done (2 left) 2020-06-12T02:39:21.170 INFO:tasks.rados.rados.0.smithi107.stdout:11776: done (1 left) 2020-06-12T02:39:21.170 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:39:21.171 INFO:tasks.rados.rados.0.smithi107.stdout:11777: seq_num 11776 ranges {2260992=434176} 2020-06-12T02:39:21.171 INFO:tasks.rados.rados.0.smithi107.stdout:11777: writing smithi10729407-18 from 2260992 to 2695168 tid 1 2020-06-12T02:39:21.171 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:39:21.171 INFO:tasks.rados.rados.0.smithi107.stdout:11778: seq_num 11777 ranges {0=450560} 2020-06-12T02:39:21.171 INFO:tasks.rados.rados.0.smithi107.stdout:11778: writing smithi10729407-9 from 0 to 450560 tid 1 2020-06-12T02:39:21.171 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:39:21.172 INFO:tasks.rados.rados.0.smithi107.stdout:11779: seq_num 11778 ranges {196608=466944} 2020-06-12T02:39:21.172 INFO:tasks.rados.rados.0.smithi107.stdout:11779: writing smithi10729407-13 from 196608 to 663552 tid 1 2020-06-12T02:39:21.172 INFO:tasks.rados.rados.0.smithi107.stdout:11780: delete oid 4 current snap is 0 2020-06-12T02:39:21.172 INFO:tasks.rados.rados.0.smithi107.stdout:11775: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:21.173 INFO:tasks.rados.rados.0.smithi107.stdout:11775: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:21.173 INFO:tasks.rados.rados.0.smithi107.stdout:11775: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:21.173 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 912 (ObjNum 11774 snap 0 seq_num 11774) dirty exists 2020-06-12T02:39:21.173 INFO:tasks.rados.rados.0.smithi107.stdout:11775: left oid 20 (ObjNum 11774 snap 0 seq_num 11774) 2020-06-12T02:39:21.173 INFO:tasks.rados.rados.0.smithi107.stdout:11778: finishing write tid 1 to smithi10729407-9 2020-06-12T02:39:21.174 INFO:tasks.rados.rados.0.smithi107.stdout:11777: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:21.174 INFO:tasks.rados.rados.0.smithi107.stdout:11778: finishing write tid 3 to smithi10729407-9 2020-06-12T02:39:21.174 INFO:tasks.rados.rados.0.smithi107.stdout:11779: finishing write tid 1 to smithi10729407-13 2020-06-12T02:39:21.174 INFO:tasks.rados.rados.0.smithi107.stdout:11775: done (4 left) 2020-06-12T02:39:21.175 INFO:tasks.rados.rados.0.smithi107.stdout:11780: done (3 left) 2020-06-12T02:39:21.175 INFO:tasks.rados.rados.0.smithi107.stdout:11781: delete oid 4 current snap is 0 2020-06-12T02:39:21.175 INFO:tasks.rados.rados.0.smithi107.stdout:11777: finishing write tid 3 to smithi10729407-18 2020-06-12T02:39:21.176 INFO:tasks.rados.rados.0.smithi107.stdout:11779: finishing write tid 3 to smithi10729407-13 2020-06-12T02:39:21.176 INFO:tasks.rados.rados.0.smithi107.stdout:11778: finishing write tid 4 to smithi10729407-9 2020-06-12T02:39:21.176 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1911 (ObjNum 11777 snap 0 seq_num 11777) dirty exists 2020-06-12T02:39:21.176 INFO:tasks.rados.rados.0.smithi107.stdout:11778: left oid 9 (ObjNum 11777 snap 0 seq_num 11777) 2020-06-12T02:39:21.177 INFO:tasks.rados.rados.0.smithi107.stdout:11779: finishing write tid 4 to smithi10729407-13 2020-06-12T02:39:21.177 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1845 (ObjNum 11778 snap 0 seq_num 11778) dirty exists 2020-06-12T02:39:21.177 INFO:tasks.rados.rados.0.smithi107.stdout:11779: left oid 13 (ObjNum 11778 snap 0 seq_num 11778) 2020-06-12T02:39:21.178 INFO:tasks.rados.rados.0.smithi107.stdout:11777: finishing write tid 4 to smithi10729407-18 2020-06-12T02:39:21.178 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1948 (ObjNum 11776 snap 0 seq_num 11776) dirty exists 2020-06-12T02:39:21.178 INFO:tasks.rados.rados.0.smithi107.stdout:11777: left oid 18 (ObjNum 11776 snap 0 seq_num 11776) 2020-06-12T02:39:21.179 INFO:tasks.rados.rados.0.smithi107.stdout:11777: done (3 left) 2020-06-12T02:39:21.179 INFO:tasks.rados.rados.0.smithi107.stdout:11778: done (2 left) 2020-06-12T02:39:21.179 INFO:tasks.rados.rados.0.smithi107.stdout:11779: done (1 left) 2020-06-12T02:39:21.180 INFO:tasks.rados.rados.0.smithi107.stdout:11781: done (0 left) 2020-06-12T02:39:21.180 INFO:tasks.rados.rados.0.smithi107.stdout:11782: delete oid 19 current snap is 0 2020-06-12T02:39:21.180 INFO:tasks.rados.rados.0.smithi107.stdout:11782: done (0 left) 2020-06-12T02:39:21.181 INFO:tasks.rados.rados.0.smithi107.stdout:11783: delete oid 7 current snap is 0 2020-06-12T02:39:21.181 INFO:tasks.rados.rados.0.smithi107.stdout:11783: done (0 left) 2020-06-12T02:39:21.181 INFO:tasks.rados.rados.0.smithi107.stdout:11784: delete oid 7 current snap is 0 2020-06-12T02:39:21.181 INFO:tasks.rados.rados.0.smithi107.stdout:11784: done (0 left) 2020-06-12T02:39:21.182 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:39:21.182 INFO:tasks.rados.rados.0.smithi107.stdout:11785: seq_num 11784 ranges {663552=565248} 2020-06-12T02:39:21.182 INFO:tasks.rados.rados.0.smithi107.stdout:11785: writing smithi10729407-13 from 663552 to 1228800 tid 1 2020-06-12T02:39:21.182 INFO:tasks.rados.rados.0.smithi107.stdout:11786: delete oid 14 current snap is 0 2020-06-12T02:39:21.182 INFO:tasks.rados.rados.0.smithi107.stdout:11786: done (1 left) 2020-06-12T02:39:21.183 INFO:tasks.rados.rados.0.smithi107.stdout:11787: delete oid 2 current snap is 0 2020-06-12T02:39:21.183 INFO:tasks.rados.rados.0.smithi107.stdout:11787: done (1 left) 2020-06-12T02:39:21.183 INFO:tasks.rados.rados.0.smithi107.stdout:11788: delete oid 14 current snap is 0 2020-06-12T02:39:21.183 INFO:tasks.rados.rados.0.smithi107.stdout:11785: finishing write tid 1 to smithi10729407-13 2020-06-12T02:39:21.184 INFO:tasks.rados.rados.0.smithi107.stdout:11788: done (1 left) 2020-06-12T02:39:21.184 INFO:tasks.rados.rados.0.smithi107.stdout:11789: delete oid 9 current snap is 0 2020-06-12T02:39:21.184 INFO:tasks.rados.rados.0.smithi107.stdout:11785: finishing write tid 3 to smithi10729407-13 2020-06-12T02:39:21.184 INFO:tasks.rados.rados.0.smithi107.stdout:11785: finishing write tid 4 to smithi10729407-13 2020-06-12T02:39:21.185 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1847 (ObjNum 11784 snap 0 seq_num 11784) dirty exists 2020-06-12T02:39:21.185 INFO:tasks.rados.rados.0.smithi107.stdout:11785: left oid 13 (ObjNum 11784 snap 0 seq_num 11784) 2020-06-12T02:39:21.185 INFO:tasks.rados.rados.0.smithi107.stdout:11785: done (1 left) 2020-06-12T02:39:21.185 INFO:tasks.rados.rados.0.smithi107.stdout:11789: done (0 left) 2020-06-12T02:39:21.186 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:39:21.186 INFO:tasks.rados.rados.0.smithi107.stdout:11790: seq_num 11789 ranges {0=647168} 2020-06-12T02:39:21.186 INFO:tasks.rados.rados.0.smithi107.stdout:11790: writing smithi10729407-9 from 0 to 647168 tid 1 2020-06-12T02:39:21.186 INFO:tasks.rados.rados.0.smithi107.stdout:11791: delete oid 16 current snap is 0 2020-06-12T02:39:21.187 INFO:tasks.rados.rados.0.smithi107.stdout:11791: done (1 left) 2020-06-12T02:39:21.187 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:39:21.187 INFO:tasks.rados.rados.0.smithi107.stdout:11792: seq_num 11791 ranges {0=679936} 2020-06-12T02:39:21.187 INFO:tasks.rados.rados.0.smithi107.stdout:11792: writing smithi10729407-12 from 0 to 679936 tid 1 2020-06-12T02:39:21.188 INFO:tasks.rados.rados.0.smithi107.stdout:11793: delete oid 20 current snap is 0 2020-06-12T02:39:21.188 INFO:tasks.rados.rados.0.smithi107.stdout:11790: finishing write tid 1 to smithi10729407-9 2020-06-12T02:39:21.188 INFO:tasks.rados.rados.0.smithi107.stdout:11790: finishing write tid 3 to smithi10729407-9 2020-06-12T02:39:21.188 INFO:tasks.rados.rados.0.smithi107.stdout:11790: finishing write tid 4 to smithi10729407-9 2020-06-12T02:39:21.188 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1914 (ObjNum 11789 snap 0 seq_num 11789) dirty exists 2020-06-12T02:39:21.189 INFO:tasks.rados.rados.0.smithi107.stdout:11790: left oid 9 (ObjNum 11789 snap 0 seq_num 11789) 2020-06-12T02:39:21.189 INFO:tasks.rados.rados.0.smithi107.stdout:11790: done (2 left) 2020-06-12T02:39:21.189 INFO:tasks.rados.rados.0.smithi107.stdout:11793: done (1 left) 2020-06-12T02:39:21.189 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:39:21.190 INFO:tasks.rados.rados.0.smithi107.stdout:11794: seq_num 11793 ranges {0=712704} 2020-06-12T02:39:21.190 INFO:tasks.rados.rados.0.smithi107.stdout:11794: writing smithi10729407-14 from 0 to 712704 tid 1 2020-06-12T02:39:21.190 INFO:tasks.rados.rados.0.smithi107.stdout:11795: delete oid 10 current snap is 0 2020-06-12T02:39:21.190 INFO:tasks.rados.rados.0.smithi107.stdout:11792: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:21.190 INFO:tasks.rados.rados.0.smithi107.stdout:11795: done (2 left) 2020-06-12T02:39:21.191 INFO:tasks.rados.rados.0.smithi107.stdout:11796: delete oid 7 current snap is 0 2020-06-12T02:39:21.191 INFO:tasks.rados.rados.0.smithi107.stdout:11792: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:21.191 INFO:tasks.rados.rados.0.smithi107.stdout:11792: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:21.191 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 2192 (ObjNum 11791 snap 0 seq_num 11791) dirty exists 2020-06-12T02:39:21.192 INFO:tasks.rados.rados.0.smithi107.stdout:11792: left oid 12 (ObjNum 11791 snap 0 seq_num 11791) 2020-06-12T02:39:21.192 INFO:tasks.rados.rados.0.smithi107.stdout:11792: done (2 left) 2020-06-12T02:39:21.192 INFO:tasks.rados.rados.0.smithi107.stdout:11796: done (1 left) 2020-06-12T02:39:21.193 INFO:tasks.rados.rados.0.smithi107.stdout:11797: delete oid 5 current snap is 0 2020-06-12T02:39:21.193 INFO:tasks.rados.rados.0.smithi107.stdout:11794: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:21.193 INFO:tasks.rados.rados.0.smithi107.stdout:11794: finishing write tid 3 to smithi10729407-14 2020-06-12T02:39:21.194 INFO:tasks.rados.rados.0.smithi107.stdout:11797: done (1 left) 2020-06-12T02:39:21.194 INFO:tasks.rados.rados.0.smithi107.stdout:11798: delete oid 13 current snap is 0 2020-06-12T02:39:21.194 INFO:tasks.rados.rados.0.smithi107.stdout:11794: finishing write tid 4 to smithi10729407-14 2020-06-12T02:39:21.194 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1830 (ObjNum 11793 snap 0 seq_num 11793) dirty exists 2020-06-12T02:39:21.195 INFO:tasks.rados.rados.0.smithi107.stdout:11794: left oid 14 (ObjNum 11793 snap 0 seq_num 11793) 2020-06-12T02:39:21.195 INFO:tasks.rados.rados.0.smithi107.stdout:11794: done (1 left) 2020-06-12T02:39:21.195 INFO:tasks.rados.rados.0.smithi107.stdout:11798: done (0 left) 2020-06-12T02:39:21.196 INFO:tasks.rados.rados.0.smithi107.stdout:11799: delete oid 11 current snap is 0 2020-06-12T02:39:21.196 INFO:teuthology.orchestra.run.smithi188.stdout:get_pg_num_history pg_num_history pg_num_history(e84 pg_nums {1={3=1},2={16=8},3={21=16,64=26}} deleted_pools ) 2020-06-12T02:39:21.197 INFO:teuthology.orchestra.run.smithi188.stdout:Importing pgid 2.5 2020-06-12T02:39:21.197 INFO:teuthology.orchestra.run.smithi188.stdout:write_pg epoch 76 info 2.5( empty local-lis/les=34/35 n=0 ec=16/16 lis/c=34/34 les/c/f=35/35/0 sis=34 pruub=11.334724061s) 2020-06-12T02:39:21.197 INFO:teuthology.orchestra.run.smithi188.stdout:Import successful 2020-06-12T02:39:21.198 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:39:21.097+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:39:21.198 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-06-12T02:39:21.207 INFO:tasks.rados.rados.0.smithi107.stdout:11799: done (0 left) 2020-06-12T02:39:21.208 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:39:21.208 INFO:tasks.rados.rados.0.smithi107.stdout:11800: seq_num 11799 ranges {0=417792,417792=401408} 2020-06-12T02:39:21.208 INFO:tasks.rados.rados.0.smithi107.stdout:11800: writing smithi10729407-19 from 0 to 417792 tid 1 2020-06-12T02:39:21.208 INFO:tasks.rados.rados.0.smithi107.stdout:11800: writing smithi10729407-19 from 417792 to 819200 tid 2 2020-06-12T02:39:21.209 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:39:21.209 INFO:tasks.rados.rados.0.smithi107.stdout:11801: seq_num 11800 ranges {0=434176,434176=401408} 2020-06-12T02:39:21.209 INFO:tasks.rados.rados.0.smithi107.stdout:11801: writing smithi10729407-20 from 0 to 434176 tid 1 2020-06-12T02:39:21.210 INFO:tasks.rados.rados.0.smithi107.stdout:11801: writing smithi10729407-20 from 434176 to 835584 tid 2 2020-06-12T02:39:21.210 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:39:21.210 INFO:tasks.rados.rados.0.smithi107.stdout:11802: seq_num 11801 ranges {163840=450560,614400=401408} 2020-06-12T02:39:21.211 INFO:tasks.rados.rados.0.smithi107.stdout:11802: writing smithi10729407-8 from 163840 to 614400 tid 1 2020-06-12T02:39:21.211 INFO:tasks.rados.rados.0.smithi107.stdout:11802: writing smithi10729407-8 from 614400 to 1015808 tid 2 2020-06-12T02:39:21.211 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:39:21.211 INFO:tasks.rados.rados.0.smithi107.stdout:11803: seq_num 11802 ranges {712704=466944,1179648=401408} 2020-06-12T02:39:21.212 INFO:tasks.rados.rados.0.smithi107.stdout:11803: writing smithi10729407-14 from 712704 to 1179648 tid 1 2020-06-12T02:39:21.212 INFO:tasks.rados.rados.0.smithi107.stdout:11803: writing smithi10729407-14 from 1179648 to 1581056 tid 2 2020-06-12T02:39:21.212 INFO:tasks.rados.rados.0.smithi107.stdout:11804: delete oid 17 current snap is 0 2020-06-12T02:39:21.213 INFO:tasks.rados.rados.0.smithi107.stdout:11800: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:21.213 INFO:tasks.rados.rados.0.smithi107.stdout:11800: finishing write tid 2 to smithi10729407-19 2020-06-12T02:39:21.213 INFO:tasks.rados.rados.0.smithi107.stdout:11800: finishing write tid 4 to smithi10729407-19 2020-06-12T02:39:21.213 INFO:tasks.rados.rados.0.smithi107.stdout:11800: finishing write tid 5 to smithi10729407-19 2020-06-12T02:39:21.214 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 2195 (ObjNum 11799 snap 0 seq_num 11799) dirty exists 2020-06-12T02:39:21.214 INFO:tasks.rados.rados.0.smithi107.stdout:11800: left oid 19 (ObjNum 11799 snap 0 seq_num 11799) 2020-06-12T02:39:21.214 INFO:tasks.rados.rados.0.smithi107.stdout:11801: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:21.214 INFO:tasks.rados.rados.0.smithi107.stdout:11801: finishing write tid 2 to smithi10729407-20 2020-06-12T02:39:21.215 INFO:tasks.rados.rados.0.smithi107.stdout:11801: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:21.215 INFO:tasks.rados.rados.0.smithi107.stdout:11802: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:21.215 INFO:tasks.rados.rados.0.smithi107.stdout:11801: finishing write tid 5 to smithi10729407-20 2020-06-12T02:39:21.215 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 916 (ObjNum 11800 snap 0 seq_num 11800) dirty exists 2020-06-12T02:39:21.216 INFO:tasks.rados.rados.0.smithi107.stdout:11801: left oid 20 (ObjNum 11800 snap 0 seq_num 11800) 2020-06-12T02:39:21.216 INFO:tasks.rados.rados.0.smithi107.stdout:11802: finishing write tid 2 to smithi10729407-8 2020-06-12T02:39:21.216 INFO:tasks.rados.rados.0.smithi107.stdout:11802: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:21.216 INFO:tasks.rados.rados.0.smithi107.stdout:11802: finishing write tid 5 to smithi10729407-8 2020-06-12T02:39:21.217 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 2222 (ObjNum 11801 snap 0 seq_num 11801) dirty exists 2020-06-12T02:39:21.217 INFO:tasks.rados.rados.0.smithi107.stdout:11802: left oid 8 (ObjNum 11801 snap 0 seq_num 11801) 2020-06-12T02:39:21.217 INFO:tasks.rados.rados.0.smithi107.stdout:11803: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:21.217 INFO:tasks.rados.rados.0.smithi107.stdout:11800: done (4 left) 2020-06-12T02:39:21.218 INFO:tasks.rados.rados.0.smithi107.stdout:11801: done (3 left) 2020-06-12T02:39:21.218 INFO:tasks.rados.rados.0.smithi107.stdout:11802: done (2 left) 2020-06-12T02:39:21.218 INFO:tasks.rados.rados.0.smithi107.stdout:11804: done (1 left) 2020-06-12T02:39:21.219 INFO:tasks.rados.rados.0.smithi107.stdout:11805: delete oid 4 current snap is 0 2020-06-12T02:39:21.219 INFO:tasks.rados.rados.0.smithi107.stdout:11803: finishing write tid 2 to smithi10729407-14 2020-06-12T02:39:21.219 INFO:tasks.rados.rados.0.smithi107.stdout:11803: finishing write tid 4 to smithi10729407-14 2020-06-12T02:39:21.220 INFO:tasks.rados.rados.0.smithi107.stdout:11805: done (1 left) 2020-06-12T02:39:21.220 INFO:tasks.rados.rados.0.smithi107.stdout:11806: delete oid 16 current snap is 0 2020-06-12T02:39:21.220 INFO:tasks.rados.rados.0.smithi107.stdout:11803: finishing write tid 5 to smithi10729407-14 2020-06-12T02:39:21.221 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1833 (ObjNum 11802 snap 0 seq_num 11802) dirty exists 2020-06-12T02:39:21.221 INFO:tasks.rados.rados.0.smithi107.stdout:11803: left oid 14 (ObjNum 11802 snap 0 seq_num 11802) 2020-06-12T02:39:21.221 INFO:tasks.rados.rados.0.smithi107.stdout:11803: done (1 left) 2020-06-12T02:39:21.222 INFO:tasks.rados.rados.0.smithi107.stdout:11806: done (0 left) 2020-06-12T02:39:21.222 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:39:21.222 INFO:tasks.rados.rados.0.smithi107.stdout:11807: seq_num 11806 ranges {0=532480,532480=401408} 2020-06-12T02:39:21.222 INFO:tasks.rados.rados.0.smithi107.stdout:11807: writing smithi10729407-16 from 0 to 532480 tid 1 2020-06-12T02:39:21.223 INFO:tasks.rados.rados.0.smithi107.stdout:11807: writing smithi10729407-16 from 532480 to 933888 tid 2 2020-06-12T02:39:21.223 INFO:tasks.rados.rados.0.smithi107.stdout:11808: delete oid 17 current snap is 0 2020-06-12T02:39:21.223 INFO:tasks.rados.rados.0.smithi107.stdout:11807: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:21.223 INFO:tasks.rados.rados.0.smithi107.stdout:11808: done (1 left) 2020-06-12T02:39:21.224 INFO:tasks.rados.rados.0.smithi107.stdout:11809: delete oid 14 current snap is 0 2020-06-12T02:39:21.224 INFO:tasks.rados.rados.0.smithi107.stdout:11809: done (1 left) 2020-06-12T02:39:21.224 INFO:tasks.rados.rados.0.smithi107.stdout:11810: delete oid 20 current snap is 0 2020-06-12T02:39:21.225 INFO:tasks.rados.rados.0.smithi107.stdout:11810: done (1 left) 2020-06-12T02:39:21.225 INFO:tasks.rados.rados.0.smithi107.stdout:11811: delete oid 13 current snap is 0 2020-06-12T02:39:21.225 INFO:tasks.rados.rados.0.smithi107.stdout:11807: finishing write tid 2 to smithi10729407-16 2020-06-12T02:39:21.226 INFO:tasks.rados.rados.0.smithi107.stdout:11811: done (1 left) 2020-06-12T02:39:21.226 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:39:21.226 INFO:tasks.rados.rados.0.smithi107.stdout:11812: seq_num 11811 ranges {679936=614400,1294336=401408} 2020-06-12T02:39:21.226 INFO:tasks.rados.rados.0.smithi107.stdout:11812: writing smithi10729407-12 from 679936 to 1294336 tid 1 2020-06-12T02:39:21.226 INFO:tasks.rados.rados.0.smithi107.stdout:11812: writing smithi10729407-12 from 1294336 to 1695744 tid 2 2020-06-12T02:39:21.227 INFO:tasks.rados.rados.0.smithi107.stdout:11813: delete oid 7 current snap is 0 2020-06-12T02:39:21.227 INFO:tasks.rados.rados.0.smithi107.stdout:11807: finishing write tid 4 to smithi10729407-16 2020-06-12T02:39:21.227 INFO:tasks.rados.rados.0.smithi107.stdout:11807: finishing write tid 5 to smithi10729407-16 2020-06-12T02:39:21.227 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 2083 (ObjNum 11806 snap 0 seq_num 11806) dirty exists 2020-06-12T02:39:21.227 INFO:tasks.rados.rados.0.smithi107.stdout:11807: left oid 16 (ObjNum 11806 snap 0 seq_num 11806) 2020-06-12T02:39:21.228 INFO:tasks.rados.rados.0.smithi107.stdout:11807: done (2 left) 2020-06-12T02:39:21.228 INFO:tasks.rados.rados.0.smithi107.stdout:11813: done (1 left) 2020-06-12T02:39:21.228 INFO:tasks.rados.rados.0.smithi107.stdout:11814: delete oid 13 current snap is 0 2020-06-12T02:39:21.228 INFO:tasks.rados.rados.0.smithi107.stdout:11814: done (1 left) 2020-06-12T02:39:21.228 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:39:21.228 INFO:tasks.rados.rados.0.smithi107.stdout:11815: seq_num 11814 ranges {0=671744,671744=401408} 2020-06-12T02:39:21.229 INFO:tasks.rados.rados.0.smithi107.stdout:11815: writing smithi10729407-20 from 0 to 671744 tid 1 2020-06-12T02:39:21.229 INFO:tasks.rados.rados.0.smithi107.stdout:11815: writing smithi10729407-20 from 671744 to 1073152 tid 2 2020-06-12T02:39:21.229 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:39:21.229 INFO:tasks.rados.rados.0.smithi107.stdout:11816: seq_num 11815 ranges {2695168=688128,3383296=401408} 2020-06-12T02:39:21.229 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:39:21.197+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:39:21.232 INFO:tasks.rados.rados.0.smithi107.stdout:11816: writing smithi10729407-18 from 2695168 to 3383296 tid 1 2020-06-12T02:39:21.232 INFO:tasks.rados.rados.0.smithi107.stdout:11816: writing smithi10729407-18 from 3383296 to 3784704 tid 2 2020-06-12T02:39:21.232 INFO:tasks.rados.rados.0.smithi107.stdout:11817: delete oid 13 current snap is 0 2020-06-12T02:39:21.233 INFO:tasks.rados.rados.0.smithi107.stdout:11812: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:21.233 INFO:tasks.rados.rados.0.smithi107.stdout:11812: finishing write tid 2 to smithi10729407-12 2020-06-12T02:39:21.233 INFO:tasks.rados.rados.0.smithi107.stdout:11812: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:21.233 INFO:tasks.rados.rados.0.smithi107.stdout:11812: finishing write tid 5 to smithi10729407-12 2020-06-12T02:39:21.233 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 2198 (ObjNum 11811 snap 0 seq_num 11811) dirty exists 2020-06-12T02:39:21.234 INFO:tasks.rados.rados.0.smithi107.stdout:11812: left oid 12 (ObjNum 11811 snap 0 seq_num 11811) 2020-06-12T02:39:21.234 INFO:tasks.rados.rados.0.smithi107.stdout:11815: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:21.234 INFO:tasks.rados.rados.0.smithi107.stdout:11815: finishing write tid 2 to smithi10729407-20 2020-06-12T02:39:21.234 INFO:tasks.rados.rados.0.smithi107.stdout:11815: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:21.234 INFO:tasks.rados.rados.0.smithi107.stdout:11815: finishing write tid 5 to smithi10729407-20 2020-06-12T02:39:21.235 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 920 (ObjNum 11814 snap 0 seq_num 11814) dirty exists 2020-06-12T02:39:21.235 INFO:tasks.rados.rados.0.smithi107.stdout:11815: left oid 20 (ObjNum 11814 snap 0 seq_num 11814) 2020-06-12T02:39:21.235 INFO:tasks.rados.rados.0.smithi107.stdout:11812: done (3 left) 2020-06-12T02:39:21.235 INFO:tasks.rados.rados.0.smithi107.stdout:11815: done (2 left) 2020-06-12T02:39:21.235 INFO:tasks.rados.rados.0.smithi107.stdout:11817: done (1 left) 2020-06-12T02:39:21.235 INFO:tasks.rados.rados.0.smithi107.stdout:11818: delete oid 5 current snap is 0 2020-06-12T02:39:21.236 INFO:tasks.rados.rados.0.smithi107.stdout:11816: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:21.236 INFO:tasks.rados.rados.0.smithi107.stdout:11818: done (1 left) 2020-06-12T02:39:21.236 INFO:tasks.rados.rados.0.smithi107.stdout:11819: delete oid 3 current snap is 0 2020-06-12T02:39:21.236 INFO:tasks.rados.rados.0.smithi107.stdout:11816: finishing write tid 2 to smithi10729407-18 2020-06-12T02:39:21.236 INFO:tasks.rados.rados.0.smithi107.stdout:11816: finishing write tid 4 to smithi10729407-18 2020-06-12T02:39:21.237 INFO:tasks.rados.rados.0.smithi107.stdout:11816: finishing write tid 5 to smithi10729407-18 2020-06-12T02:39:21.237 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1951 (ObjNum 11815 snap 0 seq_num 11815) dirty exists 2020-06-12T02:39:21.237 INFO:tasks.rados.rados.0.smithi107.stdout:11816: left oid 18 (ObjNum 11815 snap 0 seq_num 11815) 2020-06-12T02:39:21.237 INFO:tasks.rados.rados.0.smithi107.stdout:11816: done (1 left) 2020-06-12T02:39:21.237 INFO:tasks.rados.rados.0.smithi107.stdout:11819: done (0 left) 2020-06-12T02:39:21.237 INFO:tasks.rados.rados.0.smithi107.stdout:11820: delete oid 14 current snap is 0 2020-06-12T02:39:21.238 INFO:tasks.rados.rados.0.smithi107.stdout:11820: done (0 left) 2020-06-12T02:39:21.238 INFO:tasks.rados.rados.0.smithi107.stdout:11821: delete oid 9 current snap is 0 2020-06-12T02:39:21.238 INFO:tasks.rados.rados.0.smithi107.stdout:11821: done (0 left) 2020-06-12T02:39:21.238 INFO:tasks.rados.rados.0.smithi107.stdout:11822: delete oid 9 current snap is 0 2020-06-12T02:39:21.240 INFO:tasks.rados.rados.0.smithi107.stdout:11822: done (0 left) 2020-06-12T02:39:21.240 INFO:tasks.rados.rados.0.smithi107.stdout:11823: delete oid 13 current snap is 0 2020-06-12T02:39:21.240 INFO:tasks.rados.rados.0.smithi107.stdout:11823: done (0 left) 2020-06-12T02:39:21.240 INFO:tasks.rados.rados.0.smithi107.stdout:11824: delete oid 15 current snap is 0 2020-06-12T02:39:21.240 INFO:tasks.rados.rados.0.smithi107.stdout:11824: done (0 left) 2020-06-12T02:39:21.241 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:39:21.241 INFO:tasks.rados.rados.0.smithi107.stdout:11825: seq_num 11824 ranges {0=434176,434176=802816} 2020-06-12T02:39:21.241 INFO:tasks.rados.rados.0.smithi107.stdout:11825: writing smithi10729407-13 from 0 to 434176 tid 1 2020-06-12T02:39:21.245 INFO:tasks.rados.rados.0.smithi107.stdout:11825: writing smithi10729407-13 from 434176 to 1236992 tid 2 2020-06-12T02:39:21.246 INFO:tasks.rados.rados.0.smithi107.stdout:11826: delete oid 19 current snap is 0 2020-06-12T02:39:21.246 INFO:tasks.rados.rados.0.smithi107.stdout:11825: finishing write tid 1 to smithi10729407-13 2020-06-12T02:39:21.249 INFO:tasks.rados.rados.0.smithi107.stdout:11826: done (1 left) 2020-06-12T02:39:21.249 INFO:tasks.rados.rados.0.smithi107.stdout:11827: delete oid 12 current snap is 0 2020-06-12T02:39:21.254 INFO:tasks.rados.rados.0.smithi107.stdout:11827: done (1 left) 2020-06-12T02:39:21.254 INFO:tasks.rados.rados.0.smithi107.stdout:11828: delete oid 12 current snap is 0 2020-06-12T02:39:21.255 INFO:tasks.rados.rados.0.smithi107.stdout:11825: finishing write tid 2 to smithi10729407-13 2020-06-12T02:39:21.255 INFO:tasks.rados.rados.0.smithi107.stdout:11825: finishing write tid 4 to smithi10729407-13 2020-06-12T02:39:21.257 INFO:tasks.rados.rados.0.smithi107.stdout:11825: finishing write tid 5 to smithi10729407-13 2020-06-12T02:39:21.257 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1855 (ObjNum 11824 snap 0 seq_num 11824) dirty exists 2020-06-12T02:39:21.258 INFO:tasks.rados.rados.0.smithi107.stdout:11825: left oid 13 (ObjNum 11824 snap 0 seq_num 11824) 2020-06-12T02:39:21.258 INFO:tasks.rados.rados.0.smithi107.stdout:11825: done (1 left) 2020-06-12T02:39:21.258 INFO:tasks.rados.rados.0.smithi107.stdout:11828: done (0 left) 2020-06-12T02:39:21.259 INFO:tasks.rados.rados.0.smithi107.stdout:11829: delete oid 4 current snap is 0 2020-06-12T02:39:21.260 INFO:tasks.rados.rados.0.smithi107.stdout:11829: done (0 left) 2020-06-12T02:39:21.260 INFO:tasks.rados.rados.0.smithi107.stdout:11830: delete oid 9 current snap is 0 2020-06-12T02:39:21.262 INFO:tasks.rados.rados.0.smithi107.stdout:11830: done (0 left) 2020-06-12T02:39:21.262 INFO:tasks.rados.rados.0.smithi107.stdout:11831: delete oid 3 current snap is 0 2020-06-12T02:39:21.265 INFO:tasks.rados.rados.0.smithi107.stdout:11831: done (0 left) 2020-06-12T02:39:21.265 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:39:21.265 INFO:tasks.rados.rados.0.smithi107.stdout:11832: seq_num 11831 ranges {0=548864,548864=434176,983040=368640} 2020-06-12T02:39:21.268 INFO:tasks.rados.rados.0.smithi107.stdout:11832: writing smithi10729407-12 from 0 to 548864 tid 1 2020-06-12T02:39:21.271 INFO:tasks.rados.rados.0.smithi107.stdout:11832: writing smithi10729407-12 from 548864 to 983040 tid 2 2020-06-12T02:39:21.273 INFO:tasks.rados.rados.0.smithi107.stdout:11832: writing smithi10729407-12 from 983040 to 1351680 tid 3 2020-06-12T02:39:21.274 INFO:tasks.rados.rados.0.smithi107.stdout:11833: delete oid 16 current snap is 0 2020-06-12T02:39:21.274 INFO:tasks.rados.rados.0.smithi107.stdout:11832: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:21.275 INFO:tasks.rados.rados.0.smithi107.stdout:11833: done (1 left) 2020-06-12T02:39:21.276 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:39:21.276 INFO:tasks.rados.rados.0.smithi107.stdout:11834: seq_num 11833 ranges {573440=589824,1163264=606208,1769472=196608} 2020-06-12T02:39:21.282 INFO:tasks.rados.rados.0.smithi107.stdout:11834: writing smithi10729407-1 from 573440 to 1163264 tid 1 2020-06-12T02:39:21.286 INFO:tasks.rados.rados.0.smithi107.stdout:11834: writing smithi10729407-1 from 1163264 to 1769472 tid 2 2020-06-12T02:39:21.287 INFO:tasks.rados.rados.0.smithi107.stdout:11834: writing smithi10729407-1 from 1769472 to 1966080 tid 3 2020-06-12T02:39:21.287 INFO:tasks.rados.rados.0.smithi107.stdout:11835: delete oid 8 current snap is 0 2020-06-12T02:39:21.287 INFO:tasks.rados.rados.0.smithi107.stdout:11832: finishing write tid 2 to smithi10729407-12 2020-06-12T02:39:21.288 INFO:tasks.rados.rados.0.smithi107.stdout:11832: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:21.288 INFO:tasks.rados.rados.0.smithi107.stdout:11832: finishing write tid 5 to smithi10729407-12 2020-06-12T02:39:21.288 INFO:tasks.rados.rados.0.smithi107.stdout:11832: finishing write tid 6 to smithi10729407-12 2020-06-12T02:39:21.288 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 2205 (ObjNum 11831 snap 0 seq_num 11831) dirty exists 2020-06-12T02:39:21.288 INFO:tasks.rados.rados.0.smithi107.stdout:11832: left oid 12 (ObjNum 11831 snap 0 seq_num 11831) 2020-06-12T02:39:21.289 INFO:tasks.rados.rados.0.smithi107.stdout:11834: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:21.290 INFO:tasks.rados.rados.0.smithi107.stdout:11832: done (2 left) 2020-06-12T02:39:21.290 INFO:tasks.rados.rados.0.smithi107.stdout:11835: done (1 left) 2020-06-12T02:39:21.290 INFO:tasks.rados.rados.0.smithi107.stdout:11836: delete oid 2 current snap is 0 2020-06-12T02:39:21.292 INFO:tasks.rados.rados.0.smithi107.stdout:11836: done (1 left) 2020-06-12T02:39:21.292 INFO:tasks.rados.rados.0.smithi107.stdout:11837: delete oid 17 current snap is 0 2020-06-12T02:39:21.296 INFO:tasks.rados.rados.0.smithi107.stdout:11837: done (1 left) 2020-06-12T02:39:21.296 INFO:tasks.rados.rados.0.smithi107.stdout:11838: delete oid 11 current snap is 0 2020-06-12T02:39:21.297 INFO:tasks.rados.rados.0.smithi107.stdout:11834: finishing write tid 2 to smithi10729407-1 2020-06-12T02:39:21.299 INFO:tasks.rados.rados.0.smithi107.stdout:11834: finishing write tid 3 to smithi10729407-1 2020-06-12T02:39:21.300 INFO:tasks.rados.rados.0.smithi107.stdout:11834: finishing write tid 5 to smithi10729407-1 2020-06-12T02:39:21.301 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:39:21.298+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:39:21.302 INFO:tasks.rados.rados.0.smithi107.stdout:11838: done (1 left) 2020-06-12T02:39:21.303 INFO:tasks.rados.rados.0.smithi107.stdout:11839: delete oid 14 current snap is 0 2020-06-12T02:39:21.303 INFO:tasks.rados.rados.0.smithi107.stdout:11834: finishing write tid 6 to smithi10729407-1 2020-06-12T02:39:21.304 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 2157 (ObjNum 11833 snap 0 seq_num 11833) dirty exists 2020-06-12T02:39:21.304 INFO:tasks.rados.rados.0.smithi107.stdout:11834: left oid 1 (ObjNum 11833 snap 0 seq_num 11833) 2020-06-12T02:39:21.304 INFO:tasks.rados.rados.0.smithi107.stdout:11834: done (1 left) 2020-06-12T02:39:21.305 INFO:tasks.rados.rados.0.smithi107.stdout:11839: done (0 left) 2020-06-12T02:39:21.305 INFO:tasks.rados.rados.0.smithi107.stdout:11840: delete oid 7 current snap is 0 2020-06-12T02:39:21.306 INFO:tasks.rados.rados.0.smithi107.stdout:11840: done (0 left) 2020-06-12T02:39:21.306 INFO:tasks.rados.rados.0.smithi107.stdout:11841: delete oid 9 current snap is 0 2020-06-12T02:39:21.308 INFO:tasks.rados.rados.0.smithi107.stdout:11841: done (0 left) 2020-06-12T02:39:21.308 INFO:tasks.rados.rados.0.smithi107.stdout:11842: delete oid 8 current snap is 0 2020-06-12T02:39:21.311 INFO:tasks.rados.rados.0.smithi107.stdout:11842: done (0 left) 2020-06-12T02:39:21.311 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:39:21.311 INFO:tasks.rados.rados.0.smithi107.stdout:11843: seq_num 11842 ranges {0=737280,737280=417792,1155072=385024} 2020-06-12T02:39:21.316 INFO:tasks.rados.rados.0.smithi107.stdout:11843: writing smithi10729407-16 from 0 to 737280 tid 1 2020-06-12T02:39:21.318 INFO:tasks.rados.rados.0.smithi107.stdout:11843: writing smithi10729407-16 from 737280 to 1155072 tid 2 2020-06-12T02:39:21.320 INFO:tasks.rados.rados.0.smithi107.stdout:11843: writing smithi10729407-16 from 1155072 to 1540096 tid 3 2020-06-12T02:39:21.320 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:39:21.321 INFO:tasks.rados.rados.0.smithi107.stdout:11844: seq_num 11843 ranges {0=753664,753664=704512,1458176=98304} 2020-06-12T02:39:21.325 INFO:tasks.rados.rados.0.smithi107.stdout:11844: writing smithi10729407-17 from 0 to 753664 tid 1 2020-06-12T02:39:21.329 INFO:tasks.rados.rados.0.smithi107.stdout:11844: writing smithi10729407-17 from 753664 to 1458176 tid 2 2020-06-12T02:39:21.331 INFO:tasks.rados.rados.0.smithi107.stdout:11844: writing smithi10729407-17 from 1458176 to 1556480 tid 3 2020-06-12T02:39:21.331 INFO:tasks.rados.rados.0.smithi107.stdout:11845: delete oid 6 current snap is 0 2020-06-12T02:39:21.331 INFO:tasks.rados.rados.0.smithi107.stdout:11843: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:21.331 INFO:tasks.rados.rados.0.smithi107.stdout:11843: finishing write tid 2 to smithi10729407-16 2020-06-12T02:39:21.331 INFO:tasks.rados.rados.0.smithi107.stdout:11843: finishing write tid 3 to smithi10729407-16 2020-06-12T02:39:21.332 INFO:tasks.rados.rados.0.smithi107.stdout:11843: finishing write tid 5 to smithi10729407-16 2020-06-12T02:39:21.333 INFO:tasks.rados.rados.0.smithi107.stdout:11843: finishing write tid 6 to smithi10729407-16 2020-06-12T02:39:21.333 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 2090 (ObjNum 11842 snap 0 seq_num 11842) dirty exists 2020-06-12T02:39:21.333 INFO:tasks.rados.rados.0.smithi107.stdout:11843: left oid 16 (ObjNum 11842 snap 0 seq_num 11842) 2020-06-12T02:39:21.337 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-06-12T02:39:21.338 INFO:tasks.rados.rados.0.smithi107.stdout:11843: done (2 left) 2020-06-12T02:39:21.338 INFO:tasks.rados.rados.0.smithi107.stdout:11845: done (1 left) 2020-06-12T02:39:21.339 INFO:tasks.rados.rados.0.smithi107.stdout:11846: delete oid 6 current snap is 0 2020-06-12T02:39:21.339 INFO:tasks.rados.rados.0.smithi107.stdout:11844: finishing write tid 1 to smithi10729407-17 2020-06-12T02:39:21.339 INFO:tasks.rados.rados.0.smithi107.stdout:11846: done (1 left) 2020-06-12T02:39:21.340 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:39:21.340 INFO:tasks.rados.rados.0.smithi107.stdout:11847: seq_num 11846 ranges {0=802816,802816=802816} 2020-06-12T02:39:21.342 INFO:tasks.rados.rados.0.smithi107.stdout:11847: writing smithi10729407-14 from 0 to 802816 tid 1 2020-06-12T02:39:21.348 INFO:tasks.rados.rados.0.smithi107.stdout:11847: writing smithi10729407-14 from 802816 to 1605632 tid 2 2020-06-12T02:39:21.348 INFO:tasks.rados.rados.0.smithi107.stdout:11848: delete oid 2 current snap is 0 2020-06-12T02:39:21.348 INFO:tasks.rados.rados.0.smithi107.stdout:11844: finishing write tid 2 to smithi10729407-17 2020-06-12T02:39:21.349 INFO:tasks.rados.rados.0.smithi107.stdout:11844: finishing write tid 3 to smithi10729407-17 2020-06-12T02:39:21.349 INFO:tasks.rados.rados.0.smithi107.stdout:11844: finishing write tid 5 to smithi10729407-17 2020-06-12T02:39:21.349 INFO:tasks.rados.rados.0.smithi107.stdout:11844: finishing write tid 6 to smithi10729407-17 2020-06-12T02:39:21.350 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 2201 (ObjNum 11843 snap 0 seq_num 11843) dirty exists 2020-06-12T02:39:21.350 INFO:tasks.rados.rados.0.smithi107.stdout:11844: left oid 17 (ObjNum 11843 snap 0 seq_num 11843) 2020-06-12T02:39:21.351 INFO:tasks.rados.rados.0.smithi107.stdout:11847: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:21.352 INFO:tasks.rados.rados.0.smithi107.stdout:11844: done (2 left) 2020-06-12T02:39:21.352 INFO:tasks.rados.rados.0.smithi107.stdout:11848: done (1 left) 2020-06-12T02:39:21.352 INFO:tasks.rados.rados.0.smithi107.stdout:11849: delete oid 12 current snap is 0 2020-06-12T02:39:21.354 INFO:tasks.rados.rados.0.smithi107.stdout:11849: done (1 left) 2020-06-12T02:39:21.354 INFO:tasks.rados.rados.0.smithi107.stdout:11850: delete oid 8 current snap is 0 2020-06-12T02:39:21.359 INFO:tasks.rados.rados.0.smithi107.stdout:11850: done (1 left) 2020-06-12T02:39:21.359 INFO:tasks.rados.rados.0.smithi107.stdout:11851: delete oid 5 current snap is 0 2020-06-12T02:39:21.360 INFO:tasks.rados.rados.0.smithi107.stdout:11847: finishing write tid 2 to smithi10729407-14 2020-06-12T02:39:21.360 INFO:tasks.rados.rados.0.smithi107.stdout:11847: finishing write tid 4 to smithi10729407-14 2020-06-12T02:39:21.361 INFO:tasks.rados.rados.0.smithi107.stdout:11847: finishing write tid 5 to smithi10729407-14 2020-06-12T02:39:21.361 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1839 (ObjNum 11846 snap 0 seq_num 11846) dirty exists 2020-06-12T02:39:21.362 INFO:tasks.rados.rados.0.smithi107.stdout:11847: left oid 14 (ObjNum 11846 snap 0 seq_num 11846) 2020-06-12T02:39:21.369 INFO:tasks.rados.rados.0.smithi107.stdout:11847: done (1 left) 2020-06-12T02:39:21.369 INFO:tasks.rados.rados.0.smithi107.stdout:11851: done (0 left) 2020-06-12T02:39:21.369 INFO:tasks.rados.rados.0.smithi107.stdout:11852: delete oid 12 current snap is 0 2020-06-12T02:39:21.370 INFO:tasks.rados.rados.0.smithi107.stdout:11852: done (0 left) 2020-06-12T02:39:21.370 INFO:tasks.rados.rados.0.smithi107.stdout:11853: delete oid 12 current snap is 0 2020-06-12T02:39:21.370 INFO:tasks.rados.rados.0.smithi107.stdout:11853: done (0 left) 2020-06-12T02:39:21.370 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:39:21.370 INFO:tasks.rados.rados.0.smithi107.stdout:11854: seq_num 11853 ranges {0=524288,524288=442368,966656=745472,1712128=16384} 2020-06-12T02:39:21.371 INFO:tasks.rados.rados.0.smithi107.stdout:11854: writing smithi10729407-2 from 0 to 524288 tid 1 2020-06-12T02:39:21.372 INFO:tasks.rados.rados.0.smithi107.stdout:11854: writing smithi10729407-2 from 524288 to 966656 tid 2 2020-06-12T02:39:21.383 INFO:tasks.rados.rados.0.smithi107.stdout:11854: writing smithi10729407-2 from 966656 to 1712128 tid 3 2020-06-12T02:39:21.383 INFO:tasks.rados.rados.0.smithi107.stdout:11854: writing smithi10729407-2 from 1712128 to 1728512 tid 4 2020-06-12T02:39:21.383 INFO:tasks.rados.rados.0.smithi107.stdout:11855: delete oid 13 current snap is 0 2020-06-12T02:39:21.383 INFO:tasks.rados.rados.0.smithi107.stdout:11854: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:21.384 INFO:tasks.rados.rados.0.smithi107.stdout:11855: done (1 left) 2020-06-12T02:39:21.384 INFO:tasks.rados.rados.0.smithi107.stdout:11856: delete oid 15 current snap is 0 2020-06-12T02:39:21.384 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:39:21.384 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:39:21.384 INFO:teuthology.orchestra.run.smithi188.stdout: "num_ops": 0 2020-06-12T02:39:21.385 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:39:21.385 INFO:tasks.rados.rados.0.smithi107.stdout:11856: done (1 left) 2020-06-12T02:39:21.386 INFO:tasks.rados.rados.0.smithi107.stdout:11857: delete oid 13 current snap is 0 2020-06-12T02:39:21.386 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-06-12T02:39:21.387 INFO:tasks.rados.rados.0.smithi107.stdout:11854: finishing write tid 2 to smithi10729407-2 2020-06-12T02:39:21.389 INFO:tasks.rados.rados.0.smithi107.stdout:11854: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:21.392 INFO:tasks.rados.rados.0.smithi107.stdout:11854: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:21.393 INFO:tasks.rados.rados.0.smithi107.stdout:11857: done (1 left) 2020-06-12T02:39:21.394 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:39:21.394 INFO:tasks.rados.rados.0.smithi107.stdout:11858: seq_num 11857 ranges {0=589824,589824=778240,1368064=425984} 2020-06-12T02:39:21.395 INFO:tasks.rados.rados.0.smithi107.stdout:11858: writing smithi10729407-12 from 0 to 589824 tid 1 2020-06-12T02:39:21.396 INFO:tasks.rados.rados.0.smithi107.stdout:11858: writing smithi10729407-12 from 589824 to 1368064 tid 2 2020-06-12T02:39:21.399 INFO:tasks.rados.rados.0.smithi107.stdout:11858: writing smithi10729407-12 from 1368064 to 1794048 tid 3 2020-06-12T02:39:21.399 INFO:tasks.rados.rados.0.smithi107.stdout:11859: delete oid 7 current snap is 0 2020-06-12T02:39:21.399 INFO:tasks.rados.rados.0.smithi107.stdout:11854: finishing write tid 6 to smithi10729407-2 2020-06-12T02:39:21.400 INFO:tasks.rados.rados.0.smithi107.stdout:11854: finishing write tid 7 to smithi10729407-2 2020-06-12T02:39:21.400 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 1117 (ObjNum 11853 snap 0 seq_num 11853) dirty exists 2020-06-12T02:39:21.400 INFO:tasks.rados.rados.0.smithi107.stdout:11854: left oid 2 (ObjNum 11853 snap 0 seq_num 11853) 2020-06-12T02:39:21.401 INFO:tasks.rados.rados.0.smithi107.stdout:11858: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:21.402 INFO:tasks.rados.rados.0.smithi107.stdout:11854: done (2 left) 2020-06-12T02:39:21.402 INFO:tasks.rados.rados.0.smithi107.stdout:11859: done (1 left) 2020-06-12T02:39:21.402 INFO:tasks.rados.rados.0.smithi107.stdout:11860: delete oid 20 current snap is 0 2020-06-12T02:39:21.403 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:39:21.399+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:39:21.404 INFO:tasks.rados.rados.0.smithi107.stdout:11860: done (1 left) 2020-06-12T02:39:21.404 INFO:tasks.rados.rados.0.smithi107.stdout:11861: delete oid 19 current snap is 0 2020-06-12T02:39:21.407 INFO:tasks.rados.rados.0.smithi107.stdout:11858: finishing write tid 2 to smithi10729407-12 2020-06-12T02:39:21.408 INFO:tasks.rados.rados.0.smithi107.stdout:11858: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:21.409 INFO:tasks.rados.rados.0.smithi107.stdout:11858: finishing write tid 5 to smithi10729407-12 2020-06-12T02:39:21.410 INFO:tasks.rados.rados.0.smithi107.stdout:11861: done (1 left) 2020-06-12T02:39:21.410 INFO:tasks.rados.rados.0.smithi107.stdout:11862: delete oid 17 current snap is 0 2020-06-12T02:39:21.411 INFO:tasks.rados.rados.0.smithi107.stdout:11858: finishing write tid 6 to smithi10729407-12 2020-06-12T02:39:21.411 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 2212 (ObjNum 11857 snap 0 seq_num 11857) dirty exists 2020-06-12T02:39:21.411 INFO:tasks.rados.rados.0.smithi107.stdout:11858: left oid 12 (ObjNum 11857 snap 0 seq_num 11857) 2020-06-12T02:39:21.412 INFO:tasks.rados.rados.0.smithi107.stdout:11858: done (1 left) 2020-06-12T02:39:21.412 INFO:tasks.rados.rados.0.smithi107.stdout:11862: done (0 left) 2020-06-12T02:39:21.413 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:39:21.413 INFO:tasks.rados.rados.0.smithi107.stdout:11863: seq_num 11862 ranges {1540096=671744,2211840=655360,2867200=466944,3334144=81920} 2020-06-12T02:39:21.424 INFO:tasks.rados.rados.0.smithi107.stdout:11863: writing smithi10729407-16 from 1540096 to 2211840 tid 1 2020-06-12T02:39:21.433 INFO:tasks.rados.rados.0.smithi107.stdout:11863: writing smithi10729407-16 from 2211840 to 2867200 tid 2 2020-06-12T02:39:21.433 INFO:tasks.rados.rados.0.smithi107.stdout:11863: writing smithi10729407-16 from 2867200 to 3334144 tid 3 2020-06-12T02:39:21.434 INFO:tasks.rados.rados.0.smithi107.stdout:11863: writing smithi10729407-16 from 3334144 to 3416064 tid 4 2020-06-12T02:39:21.434 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:39:21.434 INFO:tasks.rados.rados.0.smithi107.stdout:11864: seq_num 11863 ranges {1794048=688128,2482176=532480,3014656=671744} 2020-06-12T02:39:21.445 INFO:tasks.rados.rados.0.smithi107.stdout:11864: writing smithi10729407-12 from 1794048 to 2482176 tid 1 2020-06-12T02:39:21.449 INFO:tasks.rados.rados.0.smithi107.stdout:11864: writing smithi10729407-12 from 2482176 to 3014656 tid 2 2020-06-12T02:39:21.452 INFO:tasks.rados.rados.0.smithi107.stdout:11864: writing smithi10729407-12 from 3014656 to 3686400 tid 3 2020-06-12T02:39:21.453 INFO:tasks.rados.rados.0.smithi107.stdout:11865: delete oid 18 current snap is 0 2020-06-12T02:39:21.453 INFO:tasks.rados.rados.0.smithi107.stdout:11863: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:21.453 INFO:tasks.rados.rados.0.smithi107.stdout:11863: finishing write tid 2 to smithi10729407-16 2020-06-12T02:39:21.453 INFO:tasks.rados.rados.0.smithi107.stdout:11863: finishing write tid 3 to smithi10729407-16 2020-06-12T02:39:21.453 INFO:tasks.rados.rados.0.smithi107.stdout:11863: finishing write tid 4 to smithi10729407-16 2020-06-12T02:39:21.454 INFO:tasks.rados.rados.0.smithi107.stdout:11863: finishing write tid 6 to smithi10729407-16 2020-06-12T02:39:21.454 INFO:tasks.rados.rados.0.smithi107.stdout:11863: finishing write tid 7 to smithi10729407-16 2020-06-12T02:39:21.454 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 2095 (ObjNum 11862 snap 0 seq_num 11862) dirty exists 2020-06-12T02:39:21.454 INFO:tasks.rados.rados.0.smithi107.stdout:11863: left oid 16 (ObjNum 11862 snap 0 seq_num 11862) 2020-06-12T02:39:21.456 INFO:tasks.rados.rados.0.smithi107.stdout:11863: done (2 left) 2020-06-12T02:39:21.456 INFO:tasks.rados.rados.0.smithi107.stdout:11865: done (1 left) 2020-06-12T02:39:21.456 INFO:tasks.rados.rados.0.smithi107.stdout:11866: delete oid 4 current snap is 0 2020-06-12T02:39:21.458 INFO:tasks.rados.rados.0.smithi107.stdout:11864: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:21.461 INFO:tasks.rados.rados.0.smithi107.stdout:11864: finishing write tid 2 to smithi10729407-12 2020-06-12T02:39:21.461 INFO:tasks.rados.rados.0.smithi107.stdout:11866: done (1 left) 2020-06-12T02:39:21.461 INFO:tasks.rados.rados.0.smithi107.stdout:11867: delete oid 1 current snap is 0 2020-06-12T02:39:21.465 INFO:tasks.rados.rados.0.smithi107.stdout:11867: done (1 left) 2020-06-12T02:39:21.465 INFO:tasks.rados.rados.0.smithi107.stdout:11868: delete oid 3 current snap is 0 2020-06-12T02:39:21.467 INFO:tasks.rados.rados.0.smithi107.stdout:11864: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:21.467 INFO:tasks.rados.rados.0.smithi107.stdout:11864: finishing write tid 5 to smithi10729407-12 2020-06-12T02:39:21.467 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2020-06-12T02:39:21.469 INFO:tasks.rados.rados.0.smithi107.stdout:11864: finishing write tid 6 to smithi10729407-12 2020-06-12T02:39:21.469 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 2217 (ObjNum 11863 snap 0 seq_num 11863) dirty exists 2020-06-12T02:39:21.469 INFO:tasks.rados.rados.0.smithi107.stdout:11864: left oid 12 (ObjNum 11863 snap 0 seq_num 11863) 2020-06-12T02:39:21.473 INFO:tasks.rados.rados.0.smithi107.stdout:11864: done (1 left) 2020-06-12T02:39:21.473 INFO:tasks.rados.rados.0.smithi107.stdout:11868: done (0 left) 2020-06-12T02:39:21.473 INFO:tasks.rados.rados.0.smithi107.stdout:11869: delete oid 20 current snap is 0 2020-06-12T02:39:21.475 INFO:tasks.rados.rados.0.smithi107.stdout:11869: done (0 left) 2020-06-12T02:39:21.476 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:39:21.476 INFO:tasks.rados.rados.0.smithi107.stdout:11870: seq_num 11869 ranges {0=786432,786432=688128,1474560=516096} 2020-06-12T02:39:21.481 INFO:tasks.rados.rados.0.smithi107.stdout:11870: writing smithi10729407-9 from 0 to 786432 tid 1 2020-06-12T02:39:21.485 INFO:tasks.rados.rados.0.smithi107.stdout:11870: writing smithi10729407-9 from 786432 to 1474560 tid 2 2020-06-12T02:39:21.488 INFO:tasks.rados.rados.0.smithi107.stdout:11870: writing smithi10729407-9 from 1474560 to 1990656 tid 3 2020-06-12T02:39:21.488 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:39:21.489 INFO:tasks.rados.rados.0.smithi107.stdout:11871: seq_num 11870 ranges {0=802816,802816=573440,1376256=475136,1851392=155648} 2020-06-12T02:39:21.493 INFO:tasks.rados.rados.0.smithi107.stdout:11871: writing smithi10729407-4 from 0 to 802816 tid 1 2020-06-12T02:39:21.497 INFO:tasks.rados.rados.0.smithi107.stdout:11871: writing smithi10729407-4 from 802816 to 1376256 tid 2 2020-06-12T02:39:21.500 INFO:tasks.rados.rados.0.smithi107.stdout:11871: writing smithi10729407-4 from 1376256 to 1851392 tid 3 2020-06-12T02:39:21.501 INFO:tasks.rados.rados.0.smithi107.stdout:11871: writing smithi10729407-4 from 1851392 to 2007040 tid 4 2020-06-12T02:39:21.501 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:39:21.501 INFO:tasks.rados.rados.0.smithi107.stdout:11872: seq_num 11871 ranges {0=417792,417792=458752,876544=638976,1515520=409600,1925120=98304} 2020-06-12T02:39:21.503 INFO:tasks.rados.rados.0.smithi107.stdout:11872: writing smithi10729407-13 from 0 to 417792 tid 1 2020-06-12T02:39:21.504 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:39:21.501+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:39:21.507 INFO:tasks.rados.rados.0.smithi107.stdout:11872: writing smithi10729407-13 from 417792 to 876544 tid 2 2020-06-12T02:39:21.512 INFO:tasks.rados.rados.0.smithi107.stdout:11872: writing smithi10729407-13 from 876544 to 1515520 tid 3 2020-06-12T02:39:21.514 INFO:tasks.rados.rados.0.smithi107.stdout:11872: writing smithi10729407-13 from 1515520 to 1925120 tid 4 2020-06-12T02:39:21.514 INFO:tasks.rados.rados.0.smithi107.stdout:11872: writing smithi10729407-13 from 1925120 to 2023424 tid 5 2020-06-12T02:39:21.515 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:39:21.515 INFO:tasks.rados.rados.0.smithi107.stdout:11873: seq_num 11872 ranges {0=442368,442368=745472,1187840=442368,1630208=417792} 2020-06-12T02:39:21.515 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:39:21.518 INFO:tasks.rados.rados.0.smithi107.stdout:11873: writing smithi10729407-7 from 0 to 442368 tid 1 2020-06-12T02:39:21.522 INFO:tasks.rados.rados.0.smithi107.stdout:11873: writing smithi10729407-7 from 442368 to 1187840 tid 2 2020-06-12T02:39:21.525 INFO:tasks.rados.rados.0.smithi107.stdout:11873: writing smithi10729407-7 from 1187840 to 1630208 tid 3 2020-06-12T02:39:21.528 INFO:tasks.rados.rados.0.smithi107.stdout:11873: writing smithi10729407-7 from 1630208 to 2048000 tid 4 2020-06-12T02:39:21.528 INFO:tasks.rados.rados.0.smithi107.stdout:11874: delete oid 3 current snap is 0 2020-06-12T02:39:21.528 INFO:tasks.rados.rados.0.smithi107.stdout:11870: finishing write tid 1 to smithi10729407-9 2020-06-12T02:39:21.528 INFO:tasks.rados.rados.0.smithi107.stdout:11870: finishing write tid 2 to smithi10729407-9 2020-06-12T02:39:21.529 INFO:tasks.rados.rados.0.smithi107.stdout:11870: finishing write tid 3 to smithi10729407-9 2020-06-12T02:39:21.529 INFO:tasks.rados.rados.0.smithi107.stdout:11870: finishing write tid 5 to smithi10729407-9 2020-06-12T02:39:21.529 INFO:tasks.rados.rados.0.smithi107.stdout:11871: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:21.529 INFO:tasks.rados.rados.0.smithi107.stdout:11870: finishing write tid 6 to smithi10729407-9 2020-06-12T02:39:21.530 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1922 (ObjNum 11869 snap 0 seq_num 11869) dirty exists 2020-06-12T02:39:21.530 INFO:tasks.rados.rados.0.smithi107.stdout:11870: left oid 9 (ObjNum 11869 snap 0 seq_num 11869) 2020-06-12T02:39:21.530 INFO:tasks.rados.rados.0.smithi107.stdout:11872: finishing write tid 1 to smithi10729407-13 2020-06-12T02:39:21.530 INFO:tasks.rados.rados.0.smithi107.stdout:11872: finishing write tid 2 to smithi10729407-13 2020-06-12T02:39:21.530 INFO:tasks.rados.rados.0.smithi107.stdout:11871: finishing write tid 2 to smithi10729407-4 2020-06-12T02:39:21.530 INFO:tasks.rados.rados.0.smithi107.stdout:11871: finishing write tid 3 to smithi10729407-4 2020-06-12T02:39:21.531 INFO:tasks.rados.rados.0.smithi107.stdout:11872: finishing write tid 3 to smithi10729407-13 2020-06-12T02:39:21.531 INFO:tasks.rados.rados.0.smithi107.stdout:11871: finishing write tid 4 to smithi10729407-4 2020-06-12T02:39:21.531 INFO:tasks.rados.rados.0.smithi107.stdout:11872: finishing write tid 4 to smithi10729407-13 2020-06-12T02:39:21.531 INFO:tasks.rados.rados.0.smithi107.stdout:11872: finishing write tid 5 to smithi10729407-13 2020-06-12T02:39:21.531 INFO:tasks.rados.rados.0.smithi107.stdout:11871: finishing write tid 6 to smithi10729407-4 2020-06-12T02:39:21.532 INFO:tasks.rados.rados.0.smithi107.stdout:11872: finishing write tid 7 to smithi10729407-13 2020-06-12T02:39:21.532 INFO:tasks.rados.rados.0.smithi107.stdout:11873: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:21.532 INFO:tasks.rados.rados.0.smithi107.stdout:11872: finishing write tid 8 to smithi10729407-13 2020-06-12T02:39:21.533 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1863 (ObjNum 11871 snap 0 seq_num 11871) dirty exists 2020-06-12T02:39:21.533 INFO:tasks.rados.rados.0.smithi107.stdout:11872: left oid 13 (ObjNum 11871 snap 0 seq_num 11871) 2020-06-12T02:39:21.533 INFO:tasks.rados.rados.0.smithi107.stdout:11871: finishing write tid 7 to smithi10729407-4 2020-06-12T02:39:21.533 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 2208 (ObjNum 11870 snap 0 seq_num 11870) dirty exists 2020-06-12T02:39:21.533 INFO:tasks.rados.rados.0.smithi107.stdout:11871: left oid 4 (ObjNum 11870 snap 0 seq_num 11870) 2020-06-12T02:39:21.534 INFO:tasks.rados.rados.0.smithi107.stdout:11870: done (4 left) 2020-06-12T02:39:21.534 INFO:tasks.rados.rados.0.smithi107.stdout:11871: done (3 left) 2020-06-12T02:39:21.534 INFO:tasks.rados.rados.0.smithi107.stdout:11872: done (2 left) 2020-06-12T02:39:21.535 INFO:tasks.rados.rados.0.smithi107.stdout:11874: done (1 left) 2020-06-12T02:39:21.535 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 14 current snap is 0 2020-06-12T02:39:21.535 INFO:tasks.rados.rados.0.smithi107.stdout:11875: seq_num 11874 ranges {1605632=475136,2080768=516096,2596864=458752,3055616=573440,3629056=57344} 2020-06-12T02:39:21.542 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:39:21.544 INFO:tasks.rados.rados.0.smithi107.stdout:11875: writing smithi10729407-14 from 1605632 to 2080768 tid 1 2020-06-12T02:39:21.547 INFO:tasks.rados.rados.0.smithi107.stdout:11875: writing smithi10729407-14 from 2080768 to 2596864 tid 2 2020-06-12T02:39:21.550 INFO:tasks.rados.rados.0.smithi107.stdout:11875: writing smithi10729407-14 from 2596864 to 3055616 tid 3 2020-06-12T02:39:21.553 INFO:tasks.rados.rados.0.smithi107.stdout:11875: writing smithi10729407-14 from 3055616 to 3629056 tid 4 2020-06-12T02:39:21.554 INFO:tasks.rados.rados.0.smithi107.stdout:11875: writing smithi10729407-14 from 3629056 to 3686400 tid 5 2020-06-12T02:39:21.554 INFO:tasks.rados.rados.0.smithi107.stdout:11876: delete oid 17 current snap is 0 2020-06-12T02:39:21.555 INFO:tasks.rados.rados.0.smithi107.stdout:11873: finishing write tid 2 to smithi10729407-7 2020-06-12T02:39:21.555 INFO:tasks.rados.rados.0.smithi107.stdout:11873: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:21.555 INFO:tasks.rados.rados.0.smithi107.stdout:11873: finishing write tid 4 to smithi10729407-7 2020-06-12T02:39:21.556 INFO:tasks.rados.rados.0.smithi107.stdout:11873: finishing write tid 6 to smithi10729407-7 2020-06-12T02:39:21.556 INFO:tasks.rados.rados.0.smithi107.stdout:11873: finishing write tid 7 to smithi10729407-7 2020-06-12T02:39:21.556 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1889 (ObjNum 11872 snap 0 seq_num 11872) dirty exists 2020-06-12T02:39:21.556 INFO:tasks.rados.rados.0.smithi107.stdout:11873: left oid 7 (ObjNum 11872 snap 0 seq_num 11872) 2020-06-12T02:39:21.556 INFO:tasks.rados.rados.0.smithi107.stdout:11875: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:21.558 INFO:tasks.rados.rados.0.smithi107.stdout:11873: done (2 left) 2020-06-12T02:39:21.558 INFO:tasks.rados.rados.0.smithi107.stdout:11876: done (1 left) 2020-06-12T02:39:21.558 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:39:21.558 INFO:tasks.rados.rados.0.smithi107.stdout:11877: seq_num 11876 ranges {2048000=507904,2555904=729088,3284992=425984,3710976=450560} 2020-06-12T02:39:21.571 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:39:21.572 INFO:tasks.rados.rados.0.smithi107.stdout:11877: writing smithi10729407-7 from 2048000 to 2555904 tid 1 2020-06-12T02:39:21.576 INFO:tasks.rados.rados.0.smithi107.stdout:11877: writing smithi10729407-7 from 2555904 to 3284992 tid 2 2020-06-12T02:39:21.578 INFO:tasks.rados.rados.0.smithi107.stdout:11877: writing smithi10729407-7 from 3284992 to 3710976 tid 3 2020-06-12T02:39:21.581 INFO:tasks.rados.rados.0.smithi107.stdout:11877: writing smithi10729407-7 from 3710976 to 4161536 tid 4 2020-06-12T02:39:21.582 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:39:21.582 INFO:tasks.rados.rados.0.smithi107.stdout:11878: seq_num 11877 ranges {0=524288,524288=614400,1138688=630784,1769472=360448} 2020-06-12T02:39:21.591 INFO:tasks.rados.rados.0.smithi107.stdout:11878: writing smithi10729407-1 from 0 to 524288 tid 1 2020-06-12T02:39:21.591 INFO:tasks.rados.rados.0.smithi107.stdout:11878: writing smithi10729407-1 from 524288 to 1138688 tid 2 2020-06-12T02:39:21.591 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:39:21.592 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2020-06-12T02:39:21.592 INFO:teuthology.orchestra.run.smithi188.stdout: "ops": [], 2020-06-12T02:39:21.592 INFO:teuthology.orchestra.run.smithi188.stdout: "complaint_time": 30, 2020-06-12T02:39:21.592 INFO:teuthology.orchestra.run.smithi188.stdout: "num_blocked_ops": 0 2020-06-12T02:39:21.593 INFO:teuthology.orchestra.run.smithi188.stdout:} 2020-06-12T02:39:21.593 INFO:tasks.rados.rados.0.smithi107.stdout:11878: writing smithi10729407-1 from 1138688 to 1769472 tid 3 2020-06-12T02:39:21.595 INFO:tasks.rados.rados.0.smithi107.stdout:11878: writing smithi10729407-1 from 1769472 to 2129920 tid 4 2020-06-12T02:39:21.595 INFO:tasks.rados.rados.0.smithi107.stdout:11879: delete oid 12 current snap is 0 2020-06-12T02:39:21.596 INFO:tasks.rados.rados.0.smithi107.stdout:11875: finishing write tid 2 to smithi10729407-14 2020-06-12T02:39:21.596 INFO:tasks.rados.rados.0.smithi107.stdout:11875: finishing write tid 3 to smithi10729407-14 2020-06-12T02:39:21.596 INFO:tasks.rados.rados.0.smithi107.stdout:11875: finishing write tid 4 to smithi10729407-14 2020-06-12T02:39:21.596 INFO:tasks.rados.rados.0.smithi107.stdout:11875: finishing write tid 5 to smithi10729407-14 2020-06-12T02:39:21.596 INFO:tasks.rados.rados.0.smithi107.stdout:11875: finishing write tid 7 to smithi10729407-14 2020-06-12T02:39:21.597 INFO:tasks.rados.rados.0.smithi107.stdout:11875: finishing write tid 8 to smithi10729407-14 2020-06-12T02:39:21.597 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1845 (ObjNum 11874 snap 0 seq_num 11874) dirty exists 2020-06-12T02:39:21.597 INFO:tasks.rados.rados.0.smithi107.stdout:11875: left oid 14 (ObjNum 11874 snap 0 seq_num 11874) 2020-06-12T02:39:21.597 INFO:tasks.rados.rados.0.smithi107.stdout:11877: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:21.597 INFO:tasks.rados.rados.0.smithi107.stdout:11877: finishing write tid 2 to smithi10729407-7 2020-06-12T02:39:21.598 INFO:tasks.rados.rados.0.smithi107.stdout:11877: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:21.598 INFO:tasks.rados.rados.0.smithi107.stdout:11877: finishing write tid 4 to smithi10729407-7 2020-06-12T02:39:21.598 INFO:tasks.rados.rados.0.smithi107.stdout:11877: finishing write tid 6 to smithi10729407-7 2020-06-12T02:39:21.598 INFO:tasks.rados.rados.0.smithi107.stdout:11877: finishing write tid 7 to smithi10729407-7 2020-06-12T02:39:21.598 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1894 (ObjNum 11876 snap 0 seq_num 11876) dirty exists 2020-06-12T02:39:21.598 INFO:tasks.rados.rados.0.smithi107.stdout:11877: left oid 7 (ObjNum 11876 snap 0 seq_num 11876) 2020-06-12T02:39:21.600 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-06-12T02:39:21.601 INFO:teuthology.orchestra.run.smithi188:> sudo rm -f /var/log/ceph/exp.2.5.7 2020-06-12T02:39:21.601 INFO:tasks.rados.rados.0.smithi107.stdout:11875: done (3 left) 2020-06-12T02:39:21.601 INFO:tasks.rados.rados.0.smithi107.stdout:11877: done (2 left) 2020-06-12T02:39:21.602 INFO:tasks.rados.rados.0.smithi107.stdout:11879: done (1 left) 2020-06-12T02:39:21.602 INFO:tasks.rados.rados.0.smithi107.stdout:11880: delete oid 18 current snap is 0 2020-06-12T02:39:21.602 INFO:tasks.rados.rados.0.smithi107.stdout:11880: done (1 left) 2020-06-12T02:39:21.609 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:39:21.609 INFO:tasks.rados.rados.0.smithi107.stdout:11881: seq_num 11880 ranges {2007040=573440,2580480=671744,3252224=802816,4055040=131072} 2020-06-12T02:39:21.612 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:39:21.610+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:39:21.614 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:39:21.614 INFO:tasks.rados.rados.0.smithi107.stdout:11881: writing smithi10729407-4 from 2007040 to 2580480 tid 1 2020-06-12T02:39:21.618 INFO:tasks.rados.rados.0.smithi107.stdout:11881: writing smithi10729407-4 from 2580480 to 3252224 tid 2 2020-06-12T02:39:21.624 INFO:tasks.rados.rados.0.smithi107.stdout:11881: writing smithi10729407-4 from 3252224 to 4055040 tid 3 2020-06-12T02:39:21.625 INFO:tasks.rados.rados.0.smithi107.stdout:11881: writing smithi10729407-4 from 4055040 to 4186112 tid 4 2020-06-12T02:39:21.625 INFO:tasks.rados.rados.0.smithi107.stdout:11882: delete oid 19 current snap is 0 2020-06-12T02:39:21.625 INFO:tasks.rados.rados.0.smithi107.stdout:11878: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:21.625 INFO:tasks.rados.rados.0.smithi107.stdout:11878: finishing write tid 2 to smithi10729407-1 2020-06-12T02:39:21.626 INFO:tasks.rados.rados.0.smithi107.stdout:11878: finishing write tid 3 to smithi10729407-1 2020-06-12T02:39:21.626 INFO:tasks.rados.rados.0.smithi107.stdout:11878: finishing write tid 4 to smithi10729407-1 2020-06-12T02:39:21.626 INFO:tasks.rados.rados.0.smithi107.stdout:11878: finishing write tid 6 to smithi10729407-1 2020-06-12T02:39:21.626 INFO:tasks.rados.rados.0.smithi107.stdout:11878: finishing write tid 7 to smithi10729407-1 2020-06-12T02:39:21.626 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 2165 (ObjNum 11877 snap 0 seq_num 11877) dirty exists 2020-06-12T02:39:21.627 INFO:tasks.rados.rados.0.smithi107.stdout:11878: left oid 1 (ObjNum 11877 snap 0 seq_num 11877) 2020-06-12T02:39:21.627 INFO:tasks.rados.rados.0.smithi107.stdout:11881: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:21.627 INFO:tasks.rados.rados.0.smithi107.stdout:11878: done (2 left) 2020-06-12T02:39:21.628 INFO:tasks.rados.rados.0.smithi107.stdout:11882: done (1 left) 2020-06-12T02:39:21.628 INFO:tasks.rados.rados.0.smithi107.stdout:11883: delete oid 6 current snap is 0 2020-06-12T02:39:21.631 INFO:tasks.rados.rados.0.smithi107.stdout:11883: done (1 left) 2020-06-12T02:39:21.631 INFO:tasks.rados.rados.0.smithi107.stdout:11884: delete oid 5 current snap is 0 2020-06-12T02:39:21.632 INFO:tasks.rados.rados.0.smithi107.stdout:11881: finishing write tid 2 to smithi10729407-4 2020-06-12T02:39:21.632 INFO:tasks.rados.rados.0.smithi107.stdout:11881: finishing write tid 3 to smithi10729407-4 2020-06-12T02:39:21.633 INFO:tasks.rados.rados.0.smithi107.stdout:11881: finishing write tid 4 to smithi10729407-4 2020-06-12T02:39:21.634 INFO:tasks.rados.rados.0.smithi107.stdout:11884: done (1 left) 2020-06-12T02:39:21.634 INFO:tasks.rados.rados.0.smithi107.stdout:11885: delete oid 1 current snap is 0 2020-06-12T02:39:21.634 INFO:tasks.rados.rados.0.smithi107.stdout:11881: finishing write tid 6 to smithi10729407-4 2020-06-12T02:39:21.636 INFO:tasks.rados.rados.0.smithi107.stdout:11881: finishing write tid 7 to smithi10729407-4 2020-06-12T02:39:21.636 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 2214 (ObjNum 11880 snap 0 seq_num 11880) dirty exists 2020-06-12T02:39:21.636 INFO:tasks.rados.rados.0.smithi107.stdout:11881: left oid 4 (ObjNum 11880 snap 0 seq_num 11880) 2020-06-12T02:39:21.636 INFO:tasks.rados.rados.0.smithi107.stdout:11881: done (1 left) 2020-06-12T02:39:21.637 INFO:tasks.rados.rados.0.smithi107.stdout:11885: done (0 left) 2020-06-12T02:39:21.637 INFO:tasks.rados.rados.0.smithi107.stdout:11886: delete oid 9 current snap is 0 2020-06-12T02:39:21.637 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:39:21.638 INFO:tasks.rados.rados.0.smithi107.stdout:11886: done (0 left) 2020-06-12T02:39:21.638 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:39:21.638 INFO:tasks.rados.rados.0.smithi107.stdout:11887: seq_num 11886 ranges {0=671744,671744=425984,1097728=802816,1900544=376832} 2020-06-12T02:39:21.642 INFO:tasks.rados.rados.0.smithi107.stdout:11887: writing smithi10729407-9 from 0 to 671744 tid 1 2020-06-12T02:39:21.644 INFO:tasks.rados.rados.0.smithi107.stdout:11887: writing smithi10729407-9 from 671744 to 1097728 tid 2 2020-06-12T02:39:21.649 INFO:tasks.rados.rados.0.smithi107.stdout:11887: writing smithi10729407-9 from 1097728 to 1900544 tid 3 2020-06-12T02:39:21.652 INFO:tasks.rados.rados.0.smithi107.stdout:11887: writing smithi10729407-9 from 1900544 to 2277376 tid 4 2020-06-12T02:39:21.652 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:39:21.652 INFO:tasks.rados.rados.0.smithi107.stdout:11888: seq_num 11887 ranges {0=688128,688128=712704,1400832=557056,1957888=335872} 2020-06-12T02:39:21.657 INFO:tasks.rados.rados.0.smithi107.stdout:11888: writing smithi10729407-6 from 0 to 688128 tid 1 2020-06-12T02:39:21.661 INFO:tasks.rados.rados.0.smithi107.stdout:11888: writing smithi10729407-6 from 688128 to 1400832 tid 2 2020-06-12T02:39:21.665 INFO:tasks.rados.rados.0.smithi107.stdout:11888: writing smithi10729407-6 from 1400832 to 1957888 tid 3 2020-06-12T02:39:21.667 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:39:21.668 INFO:tasks.rados.rados.0.smithi107.stdout:11888: writing smithi10729407-6 from 1957888 to 2293760 tid 4 2020-06-12T02:39:21.668 INFO:tasks.rados.rados.0.smithi107.stdout:11889: delete oid 8 current snap is 0 2020-06-12T02:39:21.668 INFO:tasks.rados.rados.0.smithi107.stdout:11887: finishing write tid 1 to smithi10729407-9 2020-06-12T02:39:21.668 INFO:tasks.rados.rados.0.smithi107.stdout:11887: finishing write tid 2 to smithi10729407-9 2020-06-12T02:39:21.669 INFO:tasks.rados.rados.0.smithi107.stdout:11887: finishing write tid 3 to smithi10729407-9 2020-06-12T02:39:21.669 INFO:tasks.rados.rados.0.smithi107.stdout:11887: finishing write tid 4 to smithi10729407-9 2020-06-12T02:39:21.669 INFO:tasks.rados.rados.0.smithi107.stdout:11887: finishing write tid 6 to smithi10729407-9 2020-06-12T02:39:21.669 INFO:tasks.rados.rados.0.smithi107.stdout:11887: finishing write tid 7 to smithi10729407-9 2020-06-12T02:39:21.669 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1928 (ObjNum 11886 snap 0 seq_num 11886) dirty exists 2020-06-12T02:39:21.670 INFO:tasks.rados.rados.0.smithi107.stdout:11887: left oid 9 (ObjNum 11886 snap 0 seq_num 11886) 2020-06-12T02:39:21.670 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-06-12T02:39:21.670 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-06-12T02:39:21.670 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-06-12T02:39:21.670 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-06-12T02:39:21.671 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-06-12T02:39:21.671 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-06-12T02:39:21.671 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-06-12T02:39:21.671 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.7 2020-06-12T02:39:21.671 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: problem getting command descriptions from osd.7 2020-06-12T02:39:21.672 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-06-12T02:39:21.673 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-06-12T02:39:21.673 INFO:tasks.rados.rados.0.smithi107.stdout:11887: done (2 left) 2020-06-12T02:39:21.673 INFO:tasks.rados.rados.0.smithi107.stdout:11889: done (1 left) 2020-06-12T02:39:21.673 INFO:tasks.rados.rados.0.smithi107.stdout:11890: delete oid 4 current snap is 0 2020-06-12T02:39:21.674 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-06-12T02:39:21.675 INFO:tasks.rados.rados.0.smithi107.stdout:11890: done (1 left) 2020-06-12T02:39:21.675 INFO:tasks.rados.rados.0.smithi107.stdout:11891: delete oid 11 current snap is 0 2020-06-12T02:39:21.677 INFO:tasks.rados.rados.0.smithi107.stdout:11888: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:21.678 INFO:tasks.rados.rados.0.smithi107.stdout:11888: finishing write tid 2 to smithi10729407-6 2020-06-12T02:39:21.678 INFO:tasks.rados.rados.0.smithi107.stdout:11888: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:21.679 INFO:tasks.rados.rados.0.smithi107.stdout:11888: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:21.679 INFO:tasks.rados.rados.0.smithi107.stdout:11888: finishing write tid 6 to smithi10729407-6 2020-06-12T02:39:21.680 INFO:tasks.rados.rados.0.smithi107.stdout:11888: finishing write tid 7 to smithi10729407-6 2020-06-12T02:39:21.681 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 2172 (ObjNum 11887 snap 0 seq_num 11887) dirty exists 2020-06-12T02:39:21.681 INFO:tasks.rados.rados.0.smithi107.stdout:11888: left oid 6 (ObjNum 11887 snap 0 seq_num 11887) 2020-06-12T02:39:21.682 INFO:tasks.rados.rados.0.smithi107.stdout:11888: done (1 left) 2020-06-12T02:39:21.683 INFO:tasks.rados.rados.0.smithi107.stdout:11891: done (0 left) 2020-06-12T02:39:21.683 INFO:tasks.rados.rados.0.smithi107.stdout:11892: delete oid 10 current snap is 0 2020-06-12T02:39:21.685 INFO:tasks.rados.rados.0.smithi107.stdout:11892: done (0 left) 2020-06-12T02:39:21.686 INFO:tasks.rados.rados.0.smithi107.stdout:11893: delete oid 8 current snap is 0 2020-06-12T02:39:21.688 INFO:tasks.rados.rados.0.smithi107.stdout:11893: done (0 left) 2020-06-12T02:39:21.688 INFO:tasks.rados.rados.0.smithi107.stdout:11894: delete oid 5 current snap is 0 2020-06-12T02:39:21.690 INFO:tasks.rados.rados.0.smithi107.stdout:11894: done (0 left) 2020-06-12T02:39:21.691 INFO:tasks.rados.rados.0.smithi107.stdout:11895: delete oid 15 current snap is 0 2020-06-12T02:39:21.693 INFO:tasks.rados.rados.0.smithi107.stdout:11895: done (0 left) 2020-06-12T02:39:21.693 INFO:tasks.rados.rados.0.smithi107.stdout:11896: delete oid 1 current snap is 0 2020-06-12T02:39:21.696 INFO:tasks.rados.rados.0.smithi107.stdout:11896: done (0 left) 2020-06-12T02:39:21.696 INFO:tasks.rados.rados.0.smithi107.stdout:11897: delete oid 16 current snap is 0 2020-06-12T02:39:21.699 INFO:tasks.rados.rados.0.smithi107.stdout:11897: done (0 left) 2020-06-12T02:39:21.699 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:39:21.699 INFO:tasks.rados.rados.0.smithi107.stdout:11898: seq_num 11897 ranges {0=57344} 2020-06-12T02:39:21.700 INFO:tasks.rados.rados.0.smithi107.stdout:11898: writing smithi10729407-16 from 0 to 57344 tid 1 2020-06-12T02:39:21.700 INFO:tasks.rados.rados.0.smithi107.stdout:11899: delete oid 4 current snap is 0 2020-06-12T02:39:21.703 INFO:tasks.rados.rados.0.smithi107.stdout:11899: done (1 left) 2020-06-12T02:39:21.703 INFO:tasks.rados.rados.0.smithi107.stdout:11900: delete oid 2 current snap is 0 2020-06-12T02:39:21.707 INFO:tasks.rados.rados.0.smithi107.stdout:11898: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:21.708 INFO:tasks.rados.rados.0.smithi107.stdout:11900: done (1 left) 2020-06-12T02:39:21.708 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:39:21.708 INFO:tasks.rados.rados.0.smithi107.stdout:11901: seq_num 11900 ranges {0=106496} 2020-06-12T02:39:21.709 INFO:tasks.rados.rados.0.smithi107.stdout:11901: writing smithi10729407-8 from 0 to 106496 tid 1 2020-06-12T02:39:21.709 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:39:21.709 INFO:tasks.rados.rados.0.smithi107.stdout:11902: seq_num 11901 ranges {0=122880} 2020-06-12T02:39:21.710 INFO:tasks.rados.rados.0.smithi107.stdout:11902: writing smithi10729407-4 from 0 to 122880 tid 1 2020-06-12T02:39:21.710 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:39:21.710 INFO:tasks.rados.rados.0.smithi107.stdout:11903: seq_num 11902 ranges {0=139264} 2020-06-12T02:39:21.711 INFO:tasks.rados.rados.0.smithi107.stdout:11903: writing smithi10729407-12 from 0 to 139264 tid 1 2020-06-12T02:39:21.711 INFO:tasks.rados.rados.0.smithi107.stdout:11904: delete oid 20 current snap is 0 2020-06-12T02:39:21.711 INFO:tasks.rados.rados.0.smithi107.stdout:11898: finishing write tid 3 to smithi10729407-16 2020-06-12T02:39:21.713 INFO:tasks.rados.rados.0.smithi107.stdout:11898: finishing write tid 4 to smithi10729407-16 2020-06-12T02:39:21.714 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 2100 (ObjNum 11897 snap 0 seq_num 11897) dirty exists 2020-06-12T02:39:21.715 INFO:tasks.rados.rados.0.smithi107.stdout:11898: left oid 16 (ObjNum 11897 snap 0 seq_num 11897) 2020-06-12T02:39:21.715 INFO:tasks.rados.rados.0.smithi107.stdout:11901: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:21.715 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:39:21.710+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:39:21.716 INFO:tasks.rados.rados.0.smithi107.stdout:11898: done (4 left) 2020-06-12T02:39:21.716 INFO:tasks.rados.rados.0.smithi107.stdout:11904: done (3 left) 2020-06-12T02:39:21.716 INFO:tasks.rados.rados.0.smithi107.stdout:11905: delete oid 19 current snap is 0 2020-06-12T02:39:21.717 INFO:tasks.rados.rados.0.smithi107.stdout:11902: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:21.717 INFO:tasks.rados.rados.0.smithi107.stdout:11903: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:21.717 INFO:tasks.rados.rados.0.smithi107.stdout:11903: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:21.718 INFO:tasks.rados.rados.0.smithi107.stdout:11901: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:21.718 INFO:tasks.rados.rados.0.smithi107.stdout:11902: finishing write tid 3 to smithi10729407-4 2020-06-12T02:39:21.719 INFO:tasks.rados.rados.0.smithi107.stdout:11905: done (3 left) 2020-06-12T02:39:21.719 INFO:tasks.rados.rados.0.smithi107.stdout:11906: delete oid 19 current snap is 0 2020-06-12T02:39:21.719 INFO:tasks.rados.rados.0.smithi107.stdout:11903: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:21.720 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 2221 (ObjNum 11902 snap 0 seq_num 11902) dirty exists 2020-06-12T02:39:21.720 INFO:tasks.rados.rados.0.smithi107.stdout:11903: left oid 12 (ObjNum 11902 snap 0 seq_num 11902) 2020-06-12T02:39:21.720 INFO:tasks.rados.rados.0.smithi107.stdout:11901: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:21.720 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 2232 (ObjNum 11900 snap 0 seq_num 11900) dirty exists 2020-06-12T02:39:21.721 INFO:tasks.rados.rados.0.smithi107.stdout:11901: left oid 8 (ObjNum 11900 snap 0 seq_num 11900) 2020-06-12T02:39:21.721 INFO:tasks.rados.rados.0.smithi107.stdout:11902: finishing write tid 4 to smithi10729407-4 2020-06-12T02:39:21.721 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 2218 (ObjNum 11901 snap 0 seq_num 11901) dirty exists 2020-06-12T02:39:21.722 INFO:tasks.rados.rados.0.smithi107.stdout:11902: left oid 4 (ObjNum 11901 snap 0 seq_num 11901) 2020-06-12T02:39:21.722 INFO:tasks.rados.rados.0.smithi107.stdout:11901: done (3 left) 2020-06-12T02:39:21.722 INFO:tasks.rados.rados.0.smithi107.stdout:11902: done (2 left) 2020-06-12T02:39:21.722 INFO:tasks.rados.rados.0.smithi107.stdout:11903: done (1 left) 2020-06-12T02:39:21.723 INFO:tasks.rados.rados.0.smithi107.stdout:11906: done (0 left) 2020-06-12T02:39:21.723 INFO:tasks.rados.rados.0.smithi107.stdout:11907: delete oid 4 current snap is 0 2020-06-12T02:39:21.724 INFO:tasks.rados.rados.0.smithi107.stdout:11907: done (0 left) 2020-06-12T02:39:21.724 INFO:tasks.rados.rados.0.smithi107.stdout:11908: delete oid 9 current snap is 0 2020-06-12T02:39:21.726 INFO:tasks.rados.rados.0.smithi107.stdout:11908: done (0 left) 2020-06-12T02:39:21.726 INFO:tasks.rados.rados.0.smithi107.stdout:11909: delete oid 3 current snap is 0 2020-06-12T02:39:21.731 INFO:tasks.rados.rados.0.smithi107.stdout:11909: done (0 left) 2020-06-12T02:39:21.731 INFO:tasks.rados.rados.0.smithi107.stdout:11910: delete oid 17 current snap is 0 2020-06-12T02:39:21.734 INFO:tasks.rados.rados.0.smithi107.stdout:11910: done (0 left) 2020-06-12T02:39:21.734 INFO:tasks.rados.rados.0.smithi107.stdout:11911: delete oid 2 current snap is 0 2020-06-12T02:39:21.742 INFO:tasks.rados.rados.0.smithi107.stdout:11911: done (0 left) 2020-06-12T02:39:21.743 INFO:tasks.rados.rados.0.smithi107.stdout:11912: delete oid 4 current snap is 0 2020-06-12T02:39:21.743 INFO:tasks.rados.rados.0.smithi107.stdout:11912: done (0 left) 2020-06-12T02:39:21.743 INFO:tasks.rados.rados.0.smithi107.stdout:11913: delete oid 4 current snap is 0 2020-06-12T02:39:21.743 INFO:tasks.rados.rados.0.smithi107.stdout:11913: done (0 left) 2020-06-12T02:39:21.744 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:39:21.744 INFO:tasks.rados.rados.0.smithi107.stdout:11914: seq_num 11913 ranges {2293760=327680} 2020-06-12T02:39:21.756 INFO:tasks.rados.rados.0.smithi107.stdout:11914: writing smithi10729407-6 from 2293760 to 2621440 tid 1 2020-06-12T02:39:21.757 INFO:tasks.rados.rados.0.smithi107.stdout:11915: delete oid 10 current snap is 0 2020-06-12T02:39:21.758 INFO:tasks.rados.rados.0.smithi107.stdout:11915: done (1 left) 2020-06-12T02:39:21.758 INFO:tasks.rados.rados.0.smithi107.stdout:11916: delete oid 11 current snap is 0 2020-06-12T02:39:21.765 INFO:tasks.rados.rados.0.smithi107.stdout:11914: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:21.765 INFO:tasks.rados.rados.0.smithi107.stdout:11914: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:21.765 INFO:tasks.rados.rados.0.smithi107.stdout:11916: done (1 left) 2020-06-12T02:39:21.766 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:39:21.766 INFO:tasks.rados.rados.0.smithi107.stdout:11917: seq_num 11916 ranges {4161536=376832} 2020-06-12T02:39:21.780 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:21.781 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:21.787 INFO:tasks.rados.rados.0.smithi107.stdout:11917: writing smithi10729407-7 from 4161536 to 4538368 tid 1 2020-06-12T02:39:21.788 INFO:tasks.rados.rados.0.smithi107.stdout:11918: delete oid 2 current snap is 0 2020-06-12T02:39:21.788 INFO:tasks.rados.rados.0.smithi107.stdout:11914: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:21.788 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 2175 (ObjNum 11913 snap 0 seq_num 11913) dirty exists 2020-06-12T02:39:21.788 INFO:tasks.rados.rados.0.smithi107.stdout:11914: left oid 6 (ObjNum 11913 snap 0 seq_num 11913) 2020-06-12T02:39:21.790 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-06-12T02:39:21.791 INFO:tasks.rados.rados.0.smithi107.stdout:11914: done (2 left) 2020-06-12T02:39:21.791 INFO:tasks.rados.rados.0.smithi107.stdout:11918: done (1 left) 2020-06-12T02:39:21.792 INFO:tasks.rados.rados.0.smithi107.stdout:11919: delete oid 2 current snap is 0 2020-06-12T02:39:21.792 INFO:tasks.rados.rados.0.smithi107.stdout:11919: done (1 left) 2020-06-12T02:39:21.793 INFO:tasks.rados.rados.0.smithi107.stdout:11920: delete oid 10 current snap is 0 2020-06-12T02:39:21.794 INFO:tasks.rados.rados.0.smithi107.stdout:11917: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:21.798 INFO:tasks.rados.rados.0.smithi107.stdout:11917: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:21.798 INFO:tasks.rados.rados.0.smithi107.stdout:11920: done (1 left) 2020-06-12T02:39:21.799 INFO:tasks.rados.rados.0.smithi107.stdout:11921: delete oid 6 current snap is 0 2020-06-12T02:39:21.799 INFO:tasks.rados.rados.0.smithi107.stdout:11917: finishing write tid 4 to smithi10729407-7 2020-06-12T02:39:21.799 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1896 (ObjNum 11916 snap 0 seq_num 11916) dirty exists 2020-06-12T02:39:21.800 INFO:tasks.rados.rados.0.smithi107.stdout:11917: left oid 7 (ObjNum 11916 snap 0 seq_num 11916) 2020-06-12T02:39:21.800 INFO:tasks.rados.rados.0.smithi107.stdout:11917: done (1 left) 2020-06-12T02:39:21.800 INFO:tasks.rados.rados.0.smithi107.stdout:11921: done (0 left) 2020-06-12T02:39:21.800 INFO:tasks.rados.rados.0.smithi107.stdout:11922: delete oid 2 current snap is 0 2020-06-12T02:39:21.801 INFO:tasks.rados.rados.0.smithi107.stdout:11922: done (0 left) 2020-06-12T02:39:21.801 INFO:tasks.rados.rados.0.smithi107.stdout:11923: delete oid 10 current snap is 0 2020-06-12T02:39:21.809 INFO:tasks.rados.rados.0.smithi107.stdout:11923: done (0 left) 2020-06-12T02:39:21.810 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:39:21.810 INFO:tasks.rados.rados.0.smithi107.stdout:11924: seq_num 11923 ranges {0=491520} 2020-06-12T02:39:21.810 INFO:tasks.rados.rados.0.smithi107.stdout:11924: writing smithi10729407-18 from 0 to 491520 tid 1 2020-06-12T02:39:21.810 INFO:tasks.rados.rados.0.smithi107.stdout:11925: delete oid 14 current snap is 0 2020-06-12T02:39:21.811 INFO:tasks.rados.rados.0.smithi107.stdout:11925: done (1 left) 2020-06-12T02:39:21.811 INFO:tasks.rados.rados.0.smithi107.stdout:11926: delete oid 1 current snap is 0 2020-06-12T02:39:21.814 INFO:tasks.rados.rados.0.smithi107.stdout:11924: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:21.814 INFO:tasks.rados.rados.0.smithi107.stdout:11926: done (1 left) 2020-06-12T02:39:21.815 INFO:tasks.rados.rados.0.smithi107.stdout:11927: delete oid 6 current snap is 0 2020-06-12T02:39:21.818 INFO:tasks.rados.rados.0.smithi107.stdout:11924: finishing write tid 3 to smithi10729407-18 2020-06-12T02:39:21.819 INFO:tasks.rados.rados.0.smithi107.stdout:11927: done (1 left) 2020-06-12T02:39:21.819 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:39:21.819 INFO:tasks.rados.rados.0.smithi107.stdout:11928: seq_num 11927 ranges {0=557056} 2020-06-12T02:39:21.821 INFO:tasks.rados.rados.0.smithi107.stdout:11928: writing smithi10729407-10 from 0 to 557056 tid 1 2020-06-12T02:39:21.822 INFO:tasks.rados.rados.0.smithi107.stdout:11929: delete oid 3 current snap is 0 2020-06-12T02:39:21.822 INFO:tasks.rados.rados.0.smithi107.stdout:11924: finishing write tid 4 to smithi10729407-18 2020-06-12T02:39:21.822 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1955 (ObjNum 11923 snap 0 seq_num 11923) dirty exists 2020-06-12T02:39:21.823 INFO:tasks.rados.rados.0.smithi107.stdout:11924: left oid 18 (ObjNum 11923 snap 0 seq_num 11923) 2020-06-12T02:39:21.825 INFO:tasks.rados.rados.0.smithi107.stdout:11924: done (2 left) 2020-06-12T02:39:21.825 INFO:tasks.rados.rados.0.smithi107.stdout:11929: done (1 left) 2020-06-12T02:39:21.825 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:39:21.826 INFO:tasks.rados.rados.0.smithi107.stdout:11930: seq_num 11929 ranges {0=589824} 2020-06-12T02:39:21.829 INFO:tasks.rados.rados.0.smithi107.stdout:11930: writing smithi10729407-2 from 0 to 589824 tid 1 2020-06-12T02:39:21.830 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:39:21.830 INFO:tasks.rados.rados.0.smithi107.stdout:11931: seq_num 11930 ranges {0=614400} 2020-06-12T02:39:21.834 INFO:tasks.rados.rados.0.smithi107.stdout:11931: writing smithi10729407-15 from 0 to 614400 tid 1 2020-06-12T02:39:21.835 INFO:tasks.rados.rados.0.smithi107.stdout:11932: delete oid 1 current snap is 0 2020-06-12T02:39:21.835 INFO:tasks.rados.rados.0.smithi107.stdout:11928: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:21.835 INFO:tasks.rados.rados.0.smithi107.stdout:11928: finishing write tid 3 to smithi10729407-10 2020-06-12T02:39:21.835 INFO:tasks.rados.rados.0.smithi107.stdout:11928: finishing write tid 4 to smithi10729407-10 2020-06-12T02:39:21.835 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 904 (ObjNum 11927 snap 0 seq_num 11927) dirty exists 2020-06-12T02:39:21.836 INFO:tasks.rados.rados.0.smithi107.stdout:11928: left oid 10 (ObjNum 11927 snap 0 seq_num 11927) 2020-06-12T02:39:21.836 INFO:tasks.rados.rados.0.smithi107.stdout:11930: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:21.836 INFO:tasks.rados.rados.0.smithi107.stdout:11930: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:21.837 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:39:21.833+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:39:21.837 INFO:tasks.rados.rados.0.smithi107.stdout:11930: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:21.838 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 1124 (ObjNum 11929 snap 0 seq_num 11929) dirty exists 2020-06-12T02:39:21.838 INFO:tasks.rados.rados.0.smithi107.stdout:11930: left oid 2 (ObjNum 11929 snap 0 seq_num 11929) 2020-06-12T02:39:21.839 INFO:tasks.rados.rados.0.smithi107.stdout:11928: done (3 left) 2020-06-12T02:39:21.839 INFO:tasks.rados.rados.0.smithi107.stdout:11930: done (2 left) 2020-06-12T02:39:21.839 INFO:tasks.rados.rados.0.smithi107.stdout:11932: done (1 left) 2020-06-12T02:39:21.839 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:39:21.839 INFO:tasks.rados.rados.0.smithi107.stdout:11933: seq_num 11932 ranges {0=647168} 2020-06-12T02:39:21.842 INFO:tasks.rados.rados.0.smithi107.stdout:11933: writing smithi10729407-3 from 0 to 647168 tid 1 2020-06-12T02:39:21.843 INFO:tasks.rados.rados.0.smithi107.stdout:11934: delete oid 10 current snap is 0 2020-06-12T02:39:21.843 INFO:tasks.rados.rados.0.smithi107.stdout:11931: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:21.845 INFO:tasks.rados.rados.0.smithi107.stdout:11931: finishing write tid 3 to smithi10729407-15 2020-06-12T02:39:21.845 INFO:tasks.rados.rados.0.smithi107.stdout:11934: done (2 left) 2020-06-12T02:39:21.845 INFO:tasks.rados.rados.0.smithi107.stdout:11935: delete oid 19 current snap is 0 2020-06-12T02:39:21.848 INFO:tasks.rados.rados.0.smithi107.stdout:11931: finishing write tid 4 to smithi10729407-15 2020-06-12T02:39:21.848 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 2234 (ObjNum 11930 snap 0 seq_num 11930) dirty exists 2020-06-12T02:39:21.848 INFO:tasks.rados.rados.0.smithi107.stdout:11931: left oid 15 (ObjNum 11930 snap 0 seq_num 11930) 2020-06-12T02:39:21.852 INFO:tasks.rados.rados.0.smithi107.stdout:11931: done (2 left) 2020-06-12T02:39:21.853 INFO:tasks.rados.rados.0.smithi107.stdout:11935: done (1 left) 2020-06-12T02:39:21.853 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:39:21.853 INFO:tasks.rados.rados.0.smithi107.stdout:11936: seq_num 11935 ranges {614400=696320} 2020-06-12T02:39:21.860 INFO:tasks.rados.rados.0.smithi107.stdout:11936: writing smithi10729407-15 from 614400 to 1310720 tid 1 2020-06-12T02:39:21.860 INFO:tasks.rados.rados.0.smithi107.stdout:11937: delete oid 8 current snap is 0 2020-06-12T02:39:21.861 INFO:tasks.rados.rados.0.smithi107.stdout:11933: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:21.861 INFO:tasks.rados.rados.0.smithi107.stdout:11933: finishing write tid 3 to smithi10729407-3 2020-06-12T02:39:21.861 INFO:tasks.rados.rados.0.smithi107.stdout:11933: finishing write tid 4 to smithi10729407-3 2020-06-12T02:39:21.862 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 2104 (ObjNum 11932 snap 0 seq_num 11932) dirty exists 2020-06-12T02:39:21.863 INFO:tasks.rados.rados.0.smithi107.stdout:11933: left oid 3 (ObjNum 11932 snap 0 seq_num 11932) 2020-06-12T02:39:21.868 INFO:tasks.rados.rados.0.smithi107.stdout:11936: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:21.869 INFO:tasks.rados.rados.0.smithi107.stdout:11933: done (2 left) 2020-06-12T02:39:21.869 INFO:tasks.rados.rados.0.smithi107.stdout:11937: done (1 left) 2020-06-12T02:39:21.870 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:39:21.870 INFO:tasks.rados.rados.0.smithi107.stdout:11938: seq_num 11937 ranges {0=729088} 2020-06-12T02:39:21.875 INFO:tasks.rados.rados.0.smithi107.stdout:11938: writing smithi10729407-19 from 0 to 729088 tid 1 2020-06-12T02:39:21.875 INFO:tasks.rados.rados.0.smithi107.stdout:11939: delete oid 11 current snap is 0 2020-06-12T02:39:21.875 INFO:tasks.rados.rados.0.smithi107.stdout:11936: finishing write tid 3 to smithi10729407-15 2020-06-12T02:39:21.876 INFO:tasks.rados.rados.0.smithi107.stdout:11936: finishing write tid 4 to smithi10729407-15 2020-06-12T02:39:21.876 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 2237 (ObjNum 11935 snap 0 seq_num 11935) dirty exists 2020-06-12T02:39:21.876 INFO:tasks.rados.rados.0.smithi107.stdout:11936: left oid 15 (ObjNum 11935 snap 0 seq_num 11935) 2020-06-12T02:39:21.877 INFO:tasks.rados.rados.0.smithi107.stdout:11936: done (2 left) 2020-06-12T02:39:21.877 INFO:tasks.rados.rados.0.smithi107.stdout:11939: done (1 left) 2020-06-12T02:39:21.877 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:39:21.878 INFO:tasks.rados.rados.0.smithi107.stdout:11940: seq_num 11939 ranges {647168=761856} 2020-06-12T02:39:21.885 INFO:tasks.rados.rados.0.smithi107.stdout:11940: writing smithi10729407-3 from 647168 to 1409024 tid 1 2020-06-12T02:39:21.885 INFO:tasks.rados.rados.0.smithi107.stdout:11941: delete oid 4 current snap is 0 2020-06-12T02:39:21.885 INFO:tasks.rados.rados.0.smithi107.stdout:11938: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:21.885 INFO:tasks.rados.rados.0.smithi107.stdout:11938: finishing write tid 3 to smithi10729407-19 2020-06-12T02:39:21.886 INFO:tasks.rados.rados.0.smithi107.stdout:11938: finishing write tid 4 to smithi10729407-19 2020-06-12T02:39:21.886 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 2226 (ObjNum 11937 snap 0 seq_num 11937) dirty exists 2020-06-12T02:39:21.886 INFO:tasks.rados.rados.0.smithi107.stdout:11938: left oid 19 (ObjNum 11937 snap 0 seq_num 11937) 2020-06-12T02:39:21.888 INFO:tasks.rados.rados.0.smithi107.stdout:11938: done (2 left) 2020-06-12T02:39:21.888 INFO:tasks.rados.rados.0.smithi107.stdout:11941: done (1 left) 2020-06-12T02:39:21.888 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:39:21.888 INFO:tasks.rados.rados.0.smithi107.stdout:11942: seq_num 11941 ranges {729088=794624} 2020-06-12T02:39:21.900 INFO:tasks.rados.rados.0.smithi107.stdout:11942: writing smithi10729407-19 from 729088 to 1523712 tid 1 2020-06-12T02:39:21.900 INFO:tasks.rados.rados.0.smithi107.stdout:11943: delete oid 1 current snap is 0 2020-06-12T02:39:21.900 INFO:tasks.rados.rados.0.smithi107.stdout:11940: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:21.900 INFO:tasks.rados.rados.0.smithi107.stdout:11940: finishing write tid 3 to smithi10729407-3 2020-06-12T02:39:21.901 INFO:tasks.rados.rados.0.smithi107.stdout:11940: finishing write tid 4 to smithi10729407-3 2020-06-12T02:39:21.901 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 2106 (ObjNum 11939 snap 0 seq_num 11939) dirty exists 2020-06-12T02:39:21.901 INFO:tasks.rados.rados.0.smithi107.stdout:11940: left oid 3 (ObjNum 11939 snap 0 seq_num 11939) 2020-06-12T02:39:21.902 INFO:tasks.rados.rados.0.smithi107.stdout:11940: done (2 left) 2020-06-12T02:39:21.902 INFO:tasks.rados.rados.0.smithi107.stdout:11943: done (1 left) 2020-06-12T02:39:21.902 INFO:tasks.rados.rados.0.smithi107.stdout:11944: delete oid 14 current snap is 0 2020-06-12T02:39:21.903 INFO:tasks.rados.rados.0.smithi107.stdout:11944: done (1 left) 2020-06-12T02:39:21.904 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:39:21.904 INFO:tasks.rados.rados.0.smithi107.stdout:11945: seq_num 11944 ranges {0=442368,442368=401408} 2020-06-12T02:39:21.906 INFO:tasks.rados.rados.0.smithi107.stdout:11945: writing smithi10729407-8 from 0 to 442368 tid 1 2020-06-12T02:39:21.909 INFO:tasks.rados.rados.0.smithi107.stdout:11945: writing smithi10729407-8 from 442368 to 843776 tid 2 2020-06-12T02:39:21.909 INFO:tasks.rados.rados.0.smithi107.stdout:11946: delete oid 15 current snap is 0 2020-06-12T02:39:21.909 INFO:tasks.rados.rados.0.smithi107.stdout:11942: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:21.909 INFO:tasks.rados.rados.0.smithi107.stdout:11942: finishing write tid 3 to smithi10729407-19 2020-06-12T02:39:21.910 INFO:tasks.rados.rados.0.smithi107.stdout:11942: finishing write tid 4 to smithi10729407-19 2020-06-12T02:39:21.910 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 2228 (ObjNum 11941 snap 0 seq_num 11941) dirty exists 2020-06-12T02:39:21.910 INFO:tasks.rados.rados.0.smithi107.stdout:11942: left oid 19 (ObjNum 11941 snap 0 seq_num 11941) 2020-06-12T02:39:21.912 INFO:tasks.rados.rados.0.smithi107.stdout:11945: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:21.915 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:39:21.913+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:39:21.916 INFO:tasks.rados.rados.0.smithi107.stdout:11942: done (2 left) 2020-06-12T02:39:21.917 INFO:tasks.rados.rados.0.smithi107.stdout:11946: done (1 left) 2020-06-12T02:39:21.917 INFO:tasks.rados.rados.0.smithi107.stdout:11947: delete oid 9 current snap is 0 2020-06-12T02:39:21.917 INFO:tasks.rados.rados.0.smithi107.stdout:11945: finishing write tid 2 to smithi10729407-8 2020-06-12T02:39:21.919 INFO:tasks.rados.rados.0.smithi107.stdout:11947: done (1 left) 2020-06-12T02:39:21.919 INFO:tasks.rados.rados.0.smithi107.stdout:11948: delete oid 4 current snap is 0 2020-06-12T02:39:21.919 INFO:tasks.rados.rados.0.smithi107.stdout:11945: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:21.921 INFO:tasks.rados.rados.0.smithi107.stdout:11945: finishing write tid 5 to smithi10729407-8 2020-06-12T02:39:21.921 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 2241 (ObjNum 11944 snap 0 seq_num 11944) dirty exists 2020-06-12T02:39:21.922 INFO:tasks.rados.rados.0.smithi107.stdout:11945: left oid 8 (ObjNum 11944 snap 0 seq_num 11944) 2020-06-12T02:39:21.923 INFO:tasks.rados.rados.0.smithi107.stdout:11945: done (1 left) 2020-06-12T02:39:21.924 INFO:tasks.rados.rados.0.smithi107.stdout:11948: done (0 left) 2020-06-12T02:39:21.924 INFO:tasks.rados.rados.0.smithi107.stdout:11949: delete oid 9 current snap is 0 2020-06-12T02:39:21.926 INFO:tasks.rados.rados.0.smithi107.stdout:11949: done (0 left) 2020-06-12T02:39:21.926 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:39:21.927 INFO:tasks.rados.rados.0.smithi107.stdout:11950: seq_num 11949 ranges {1409024=524288,1933312=401408} 2020-06-12T02:39:21.930 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:21.931 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:21.936 INFO:tasks.rados.rados.0.smithi107.stdout:11950: writing smithi10729407-3 from 1409024 to 1933312 tid 1 2020-06-12T02:39:21.938 INFO:tasks.rados.rados.0.smithi107.stdout:11950: writing smithi10729407-3 from 1933312 to 2334720 tid 2 2020-06-12T02:39:21.939 INFO:tasks.rados.rados.0.smithi107.stdout:11951: delete oid 18 current snap is 0 2020-06-12T02:39:21.940 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-06-12T02:39:21.941 INFO:tasks.rados.rados.0.smithi107.stdout:11951: done (1 left) 2020-06-12T02:39:21.941 INFO:tasks.rados.rados.0.smithi107.stdout:11952: delete oid 16 current snap is 0 2020-06-12T02:39:21.946 INFO:tasks.rados.rados.0.smithi107.stdout:11950: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:21.948 INFO:tasks.rados.rados.0.smithi107.stdout:11950: finishing write tid 2 to smithi10729407-3 2020-06-12T02:39:21.948 INFO:tasks.rados.rados.0.smithi107.stdout:11950: finishing write tid 4 to smithi10729407-3 2020-06-12T02:39:21.949 INFO:tasks.rados.rados.0.smithi107.stdout:11950: finishing write tid 5 to smithi10729407-3 2020-06-12T02:39:21.949 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 2109 (ObjNum 11949 snap 0 seq_num 11949) dirty exists 2020-06-12T02:39:21.949 INFO:tasks.rados.rados.0.smithi107.stdout:11950: left oid 3 (ObjNum 11949 snap 0 seq_num 11949) 2020-06-12T02:39:21.950 INFO:tasks.rados.rados.0.smithi107.stdout:11950: done (1 left) 2020-06-12T02:39:21.950 INFO:tasks.rados.rados.0.smithi107.stdout:11952: done (0 left) 2020-06-12T02:39:21.950 INFO:tasks.rados.rados.0.smithi107.stdout:11953: delete oid 14 current snap is 0 2020-06-12T02:39:21.952 INFO:tasks.rados.rados.0.smithi107.stdout:11953: done (0 left) 2020-06-12T02:39:21.953 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:39:21.953 INFO:tasks.rados.rados.0.smithi107.stdout:11954: seq_num 11953 ranges {0=598016,598016=401408} 2020-06-12T02:39:21.956 INFO:tasks.rados.rados.0.smithi107.stdout:11954: writing smithi10729407-15 from 0 to 598016 tid 1 2020-06-12T02:39:21.958 INFO:tasks.rados.rados.0.smithi107.stdout:11954: writing smithi10729407-15 from 598016 to 999424 tid 2 2020-06-12T02:39:21.959 INFO:tasks.rados.rados.0.smithi107.stdout:11955: delete oid 4 current snap is 0 2020-06-12T02:39:21.963 INFO:tasks.rados.rados.0.smithi107.stdout:11955: done (1 left) 2020-06-12T02:39:21.964 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:39:21.965 INFO:tasks.rados.rados.0.smithi107.stdout:11956: seq_num 11955 ranges {0=630784,630784=401408} 2020-06-12T02:39:21.967 INFO:tasks.rados.rados.0.smithi107.stdout:11956: writing smithi10729407-18 from 0 to 630784 tid 1 2020-06-12T02:39:21.971 INFO:tasks.rados.rados.0.smithi107.stdout:11956: writing smithi10729407-18 from 630784 to 1032192 tid 2 2020-06-12T02:39:21.972 INFO:tasks.rados.rados.0.smithi107.stdout:11957: delete oid 7 current snap is 0 2020-06-12T02:39:21.973 INFO:tasks.rados.rados.0.smithi107.stdout:11954: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:21.973 INFO:tasks.rados.rados.0.smithi107.stdout:11954: finishing write tid 2 to smithi10729407-15 2020-06-12T02:39:21.973 INFO:tasks.rados.rados.0.smithi107.stdout:11954: finishing write tid 4 to smithi10729407-15 2020-06-12T02:39:21.973 INFO:tasks.rados.rados.0.smithi107.stdout:11954: finishing write tid 5 to smithi10729407-15 2020-06-12T02:39:21.973 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 2244 (ObjNum 11953 snap 0 seq_num 11953) dirty exists 2020-06-12T02:39:21.974 INFO:tasks.rados.rados.0.smithi107.stdout:11954: left oid 15 (ObjNum 11953 snap 0 seq_num 11953) 2020-06-12T02:39:21.974 INFO:tasks.rados.rados.0.smithi107.stdout:11954: done (2 left) 2020-06-12T02:39:21.974 INFO:tasks.rados.rados.0.smithi107.stdout:11957: done (1 left) 2020-06-12T02:39:21.974 INFO:tasks.rados.rados.0.smithi107.stdout:11958: delete oid 16 current snap is 0 2020-06-12T02:39:21.975 INFO:tasks.rados.rados.0.smithi107.stdout:11958: done (1 left) 2020-06-12T02:39:21.975 INFO:tasks.rados.rados.0.smithi107.stdout:11959: delete oid 5 current snap is 0 2020-06-12T02:39:21.976 INFO:tasks.rados.rados.0.smithi107.stdout:11956: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:21.976 INFO:tasks.rados.rados.0.smithi107.stdout:11956: finishing write tid 2 to smithi10729407-18 2020-06-12T02:39:21.977 INFO:tasks.rados.rados.0.smithi107.stdout:11956: finishing write tid 4 to smithi10729407-18 2020-06-12T02:39:21.978 INFO:tasks.rados.rados.0.smithi107.stdout:11959: done (1 left) 2020-06-12T02:39:21.978 INFO:tasks.rados.rados.0.smithi107.stdout:11960: delete oid 6 current snap is 0 2020-06-12T02:39:21.978 INFO:tasks.rados.rados.0.smithi107.stdout:11956: finishing write tid 5 to smithi10729407-18 2020-06-12T02:39:21.979 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1959 (ObjNum 11955 snap 0 seq_num 11955) dirty exists 2020-06-12T02:39:21.979 INFO:tasks.rados.rados.0.smithi107.stdout:11956: left oid 18 (ObjNum 11955 snap 0 seq_num 11955) 2020-06-12T02:39:21.982 INFO:tasks.rados.rados.0.smithi107.stdout:11956: done (1 left) 2020-06-12T02:39:21.982 INFO:tasks.rados.rados.0.smithi107.stdout:11960: done (0 left) 2020-06-12T02:39:21.982 INFO:tasks.rados.rados.0.smithi107.stdout:11961: delete oid 20 current snap is 0 2020-06-12T02:39:21.984 INFO:tasks.rados.rados.0.smithi107.stdout:11961: done (0 left) 2020-06-12T02:39:21.984 INFO:tasks.rados.rados.0.smithi107.stdout:11962: delete oid 13 current snap is 0 2020-06-12T02:39:21.985 INFO:tasks.rados.rados.0.smithi107.stdout:11962: done (0 left) 2020-06-12T02:39:21.986 INFO:tasks.rados.rados.0.smithi107.stdout:11963: delete oid 11 current snap is 0 2020-06-12T02:39:21.987 INFO:tasks.rados.rados.0.smithi107.stdout:11963: done (0 left) 2020-06-12T02:39:21.988 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:39:21.988 INFO:tasks.rados.rados.0.smithi107.stdout:11964: seq_num 11963 ranges {843776=761856,1605632=401408} 2020-06-12T02:39:21.996 INFO:tasks.rados.rados.0.smithi107.stdout:11964: writing smithi10729407-8 from 843776 to 1605632 tid 1 2020-06-12T02:39:21.997 INFO:teuthology.orchestra.run.smithi107.stdout: 2020-06-12T02:39:21.998 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":88,"fsid":"2f5030a1-3206-43c9-8471-d10429688d36","created":"2020-06-12T02:37:00.578493+0000","modified":"2020-06-12T02:39:21.451241+0000","last_up_change":"2020-06-12T02:39:15.419817+0000","last_in_change":"2020-06-12T02:37:10.705891+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-06-12T02:37:03.610773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-06-12T02:37:14.831188+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2020-06-12T02:37:25.893619+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":26,"pg_placement_num":26,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":26,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"64","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":true,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"ad4b4814-83e7-4a2e-9b1f-d10406174f7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":26614},{"type":"v1","addr":"172.21.15.107:6817","nonce":26614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6819","nonce":26614},{"type":"v1","addr":"172.21.15.107:6821","nonce":26614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6827","nonce":26614},{"type":"v1","addr":"172.21.15.107:6829","nonce":26614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":26614},{"type":"v1","addr":"172.21.15.107:6825","nonce":26614}]},"public_addr":"172.21.15.107:6817/26614","cluster_addr":"172.21.15.107:6821/26614","heartbeat_back_addr":"172.21.15.107:6829/26614","heartbeat_front_addr":"172.21.15.107:6825/26614","state":["exists","up"]},{"osd":1,"uuid":"046adf0d-b9e9-41ad-a7c9-3841f8b8f790","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":26611},{"type":"v1","addr":"172.21.15.107:6801","nonce":26611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":26611},{"type":"v1","addr":"172.21.15.107:6803","nonce":26611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":26611},{"type":"v1","addr":"172.21.15.107:6807","nonce":26611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":26611},{"type":"v1","addr":"172.21.15.107:6805","nonce":26611}]},"public_addr":"172.21.15.107:6801/26611","cluster_addr":"172.21.15.107:6803/26611","heartbeat_back_addr":"172.21.15.107:6807/26611","heartbeat_front_addr":"172.21.15.107:6805/26611","state":["exists","up"]},{"osd":2,"uuid":"e15da6b7-dc79-4185-a5f9-0f443d8b52b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":26613},{"type":"v1","addr":"172.21.15.107:6820","nonce":26613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":26613},{"type":"v1","addr":"172.21.15.107:6824","nonce":26613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":26613},{"type":"v1","addr":"172.21.15.107:6831","nonce":26613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":26613},{"type":"v1","addr":"172.21.15.107:6828","nonce":26613}]},"public_addr":"172.21.15.107:6820/26613","cluster_addr":"172.21.15.107:6824/26613","heartbeat_back_addr":"172.21.15.107:6831/26613","heartbeat_front_addr":"172.21.15.107:6828/26613","state":["exists","up"]},{"osd":3,"uuid":"843e0dd3-5ce4-46b9-9004-69849048d2bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":26612},{"type":"v1","addr":"172.21.15.107:6809","nonce":26612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":26612},{"type":"v1","addr":"172.21.15.107:6811","nonce":26612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":26612},{"type":"v1","addr":"172.21.15.107:6815","nonce":26612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":26612},{"type":"v1","addr":"172.21.15.107:6813","nonce":26612}]},"public_addr":"172.21.15.107:6809/26612","cluster_addr":"172.21.15.107:6811/26612","heartbeat_back_addr":"172.21.15.107:6815/26612","heartbeat_front_addr":"172.21.15.107:6813/26612","state":["exists","up"]},{"osd":4,"uuid":"c066ab09-e031-4b49-9df3-d54cd9f9b580","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":26685},{"type":"v1","addr":"172.21.15.188:6820","nonce":26685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":26685},{"type":"v1","addr":"172.21.15.188:6823","nonce":26685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":26685},{"type":"v1","addr":"172.21.15.188:6832","nonce":26685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6825","nonce":26685},{"type":"v1","addr":"172.21.15.188:6828","nonce":26685}]},"public_addr":"172.21.15.188:6820/26685","cluster_addr":"172.21.15.188:6823/26685","heartbeat_back_addr":"172.21.15.188:6832/26685","heartbeat_front_addr":"172.21.15.188:6828/26685","state":["exists","up"]},{"osd":5,"uuid":"56a24e45-1b1b-4a65-b4b4-57cfab296f65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6819","nonce":26684},{"type":"v1","addr":"172.21.15.188:6821","nonce":26684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":26684},{"type":"v1","addr":"172.21.15.188:6826","nonce":26684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6831","nonce":26684},{"type":"v1","addr":"172.21.15.188:6833","nonce":26684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6827","nonce":26684},{"type":"v1","addr":"172.21.15.188:6829","nonce":26684}]},"public_addr":"172.21.15.188:6821/26684","cluster_addr":"172.21.15.188:6826/26684","heartbeat_back_addr":"172.21.15.188:6833/26684","heartbeat_front_addr":"172.21.15.188:6829/26684","state":["exists","up"]},{"osd":6,"uuid":"f207a2a8-b3c7-4fd8-ae90-4c2d7a5c1c38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":26682},{"type":"v1","addr":"172.21.15.188:6811","nonce":26682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":26682},{"type":"v1","addr":"172.21.15.188:6813","nonce":26682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":26682},{"type":"v1","addr":"172.21.15.188:6817","nonce":26682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":26682},{"type":"v1","addr":"172.21.15.188:6815","nonce":26682}]},"public_addr":"172.21.15.188:6811/26682","cluster_addr":"172.21.15.188:6813/26682","heartbeat_back_addr":"172.21.15.188:6817/26682","heartbeat_front_addr":"172.21.15.188:6815/26682","state":["exists","up"]},{"osd":7,"uuid":"4e5cc44f-af17-47c9-973f-8e87d1bd98db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":18,"up_from":34,"up_thru":69,"down_at":85,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":29046},{"type":"v1","addr":"172.21.15.188:6803","nonce":29046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":29046},{"type":"v1","addr":"172.21.15.188:6805","nonce":29046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":29046},{"type":"v1","addr":"172.21.15.188:6809","nonce":29046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":29046},{"type":"v1","addr":"172.21.15.188:6807","nonce":29046}]},"public_addr":"172.21.15.188:6803/29046","cluster_addr":"172.21.15.188:6805/29046","heartbeat_back_addr":"172.21.15.188:6809/29046","heartbeat_front_addr":"172.21.15.188:6807/29046","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:39:10.003202+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:39:09.540687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:39:09.584675+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:39:10.075640+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:39:09.801034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:39:09.186391+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:39:10.882758+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-06-12T02:39:10.052990+0000","dead_epoch":85}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-06-12T02:39:21.999 INFO:tasks.rados.rados.0.smithi107.stdout:11964: writing smithi10729407-8 from 1605632 to 2007040 tid 2 2020-06-12T02:39:21.999 INFO:tasks.rados.rados.0.smithi107.stdout:11965: delete oid 7 current snap is 0 2020-06-12T02:39:22.006 INFO:tasks.rados.rados.0.smithi107.stdout:11965: done (1 left) 2020-06-12T02:39:22.006 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:39:22.006 INFO:tasks.rados.rados.0.smithi107.stdout:11966: seq_num 11965 ranges {0=794624,794624=401408} 2020-06-12T02:39:22.010 INFO:tasks.ceph.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-06-12T02:37:03.610773+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-06-12T02:37:14.831188+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'unique_pool_0', 'create_time': '2020-06-12T02:37:25.893619+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 3, 'size': 3, 'min_size': 2, 'crush_rule': 1, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '64', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'teuthologyprofile', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': True, 'options': {}, 'application_metadata': {'rados': {}}}] 2020-06-12T02:39:22.010 INFO:teuthology.orchestra.run.smithi188:> CEPH_ARGS='--filestore-merge-threshold 1 --filestore-split-multiple 1' sudo -E ceph-objectstore-tool --no-mon-config --log-file=/var/log/ceph/objectstore_tool.$pid.log --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op apply-layout-settings --pool device_health_metrics 2020-06-12T02:39:22.011 INFO:tasks.rados.rados.0.smithi107.stdout:11966: writing smithi10729407-13 from 0 to 794624 tid 1 2020-06-12T02:39:22.013 INFO:tasks.rados.rados.0.smithi107.stdout:11966: writing smithi10729407-13 from 794624 to 1196032 tid 2 2020-06-12T02:39:22.014 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 6 current snap is 0 2020-06-12T02:39:22.014 INFO:tasks.rados.rados.0.smithi107.stdout:11967: seq_num 11966 ranges {0=409600,409600=466944,876544=335872} 2020-06-12T02:39:22.016 INFO:tasks.rados.rados.0.smithi107.stdout:11967: writing smithi10729407-6 from 0 to 409600 tid 1 2020-06-12T02:39:22.016 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:39:22.013+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:39:22.019 INFO:tasks.rados.rados.0.smithi107.stdout:11967: writing smithi10729407-6 from 409600 to 876544 tid 2 2020-06-12T02:39:22.022 INFO:tasks.rados.rados.0.smithi107.stdout:11967: writing smithi10729407-6 from 876544 to 1212416 tid 3 2020-06-12T02:39:22.022 INFO:tasks.rados.rados.0.smithi107.stdout:11968: delete oid 4 current snap is 0 2020-06-12T02:39:22.023 INFO:tasks.rados.rados.0.smithi107.stdout:11964: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:22.023 INFO:tasks.rados.rados.0.smithi107.stdout:11964: finishing write tid 2 to smithi10729407-8 2020-06-12T02:39:22.023 INFO:tasks.rados.rados.0.smithi107.stdout:11964: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:22.024 INFO:tasks.rados.rados.0.smithi107.stdout:11964: finishing write tid 5 to smithi10729407-8 2020-06-12T02:39:22.024 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 2247 (ObjNum 11963 snap 0 seq_num 11963) dirty exists 2020-06-12T02:39:22.024 INFO:tasks.rados.rados.0.smithi107.stdout:11964: left oid 8 (ObjNum 11963 snap 0 seq_num 11963) 2020-06-12T02:39:22.024 INFO:tasks.rados.rados.0.smithi107.stdout:11967: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:22.025 INFO:tasks.rados.rados.0.smithi107.stdout:11966: finishing write tid 1 to smithi10729407-13 2020-06-12T02:39:22.025 INFO:tasks.rados.rados.0.smithi107.stdout:11966: finishing write tid 2 to smithi10729407-13 2020-06-12T02:39:22.025 INFO:tasks.rados.rados.0.smithi107.stdout:11966: finishing write tid 4 to smithi10729407-13 2020-06-12T02:39:22.026 INFO:tasks.rados.rados.0.smithi107.stdout:11966: finishing write tid 5 to smithi10729407-13 2020-06-12T02:39:22.026 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1867 (ObjNum 11965 snap 0 seq_num 11965) dirty exists 2020-06-12T02:39:22.026 INFO:tasks.rados.rados.0.smithi107.stdout:11966: left oid 13 (ObjNum 11965 snap 0 seq_num 11965) 2020-06-12T02:39:22.026 INFO:tasks.rados.rados.0.smithi107.stdout:11964: done (3 left) 2020-06-12T02:39:22.026 INFO:tasks.rados.rados.0.smithi107.stdout:11966: done (2 left) 2020-06-12T02:39:22.027 INFO:tasks.rados.rados.0.smithi107.stdout:11968: done (1 left) 2020-06-12T02:39:22.027 INFO:tasks.rados.rados.0.smithi107.stdout:11969: delete oid 18 current snap is 0 2020-06-12T02:39:22.028 INFO:tasks.rados.rados.0.smithi107.stdout:11969: done (1 left) 2020-06-12T02:39:22.028 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:39:22.028 INFO:tasks.rados.rados.0.smithi107.stdout:11970: seq_num 11969 ranges {589824=466944,1056768=573440,1630208=229376} 2020-06-12T02:39:22.033 INFO:tasks.rados.rados.0.smithi107.stdout:11970: writing smithi10729407-2 from 589824 to 1056768 tid 1 2020-06-12T02:39:22.037 INFO:tasks.rados.rados.0.smithi107.stdout:11970: writing smithi10729407-2 from 1056768 to 1630208 tid 2 2020-06-12T02:39:22.039 INFO:tasks.rados.rados.0.smithi107.stdout:11970: writing smithi10729407-2 from 1630208 to 1859584 tid 3 2020-06-12T02:39:22.039 INFO:tasks.rados.rados.0.smithi107.stdout:11971: delete oid 20 current snap is 0 2020-06-12T02:39:22.039 INFO:tasks.rados.rados.0.smithi107.stdout:11967: finishing write tid 2 to smithi10729407-6 2020-06-12T02:39:22.040 INFO:tasks.rados.rados.0.smithi107.stdout:11967: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:22.040 INFO:tasks.rados.rados.0.smithi107.stdout:11967: finishing write tid 5 to smithi10729407-6 2020-06-12T02:39:22.040 INFO:tasks.rados.rados.0.smithi107.stdout:11967: finishing write tid 6 to smithi10729407-6 2020-06-12T02:39:22.041 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 2185 (ObjNum 11966 snap 0 seq_num 11966) dirty exists 2020-06-12T02:39:22.041 INFO:tasks.rados.rados.0.smithi107.stdout:11967: left oid 6 (ObjNum 11966 snap 0 seq_num 11966) 2020-06-12T02:39:22.041 INFO:tasks.rados.rados.0.smithi107.stdout:11970: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:22.042 INFO:tasks.rados.rados.0.smithi107.stdout:11967: done (2 left) 2020-06-12T02:39:22.043 INFO:tasks.rados.rados.0.smithi107.stdout:11971: done (1 left) 2020-06-12T02:39:22.043 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:39:22.043 INFO:tasks.rados.rados.0.smithi107.stdout:11972: seq_num 11971 ranges {0=499712,499712=737280,1236992=65536} 2020-06-12T02:39:22.045 INFO:tasks.rados.rados.0.smithi107.stdout:11972: writing smithi10729407-1 from 0 to 499712 tid 1 2020-06-12T02:39:22.053 INFO:tasks.rados.rados.0.smithi107.stdout:11972: writing smithi10729407-1 from 499712 to 1236992 tid 2 2020-06-12T02:39:22.053 INFO:tasks.rados.rados.0.smithi107.stdout:11972: writing smithi10729407-1 from 1236992 to 1302528 tid 3 2020-06-12T02:39:22.053 INFO:tasks.rados.rados.0.smithi107.stdout:11973: delete oid 12 current snap is 0 2020-06-12T02:39:22.054 INFO:tasks.rados.rados.0.smithi107.stdout:11970: finishing write tid 2 to smithi10729407-2 2020-06-12T02:39:22.054 INFO:tasks.rados.rados.0.smithi107.stdout:11970: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:22.054 INFO:tasks.rados.rados.0.smithi107.stdout:11970: finishing write tid 5 to smithi10729407-2 2020-06-12T02:39:22.054 INFO:tasks.rados.rados.0.smithi107.stdout:11970: finishing write tid 6 to smithi10729407-2 2020-06-12T02:39:22.054 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 1128 (ObjNum 11969 snap 0 seq_num 11969) dirty exists 2020-06-12T02:39:22.055 INFO:tasks.rados.rados.0.smithi107.stdout:11970: left oid 2 (ObjNum 11969 snap 0 seq_num 11969) 2020-06-12T02:39:22.055 INFO:tasks.rados.rados.0.smithi107.stdout:11970: done (2 left) 2020-06-12T02:39:22.056 INFO:tasks.rados.rados.0.smithi107.stdout:11973: done (1 left) 2020-06-12T02:39:22.056 INFO:tasks.rados.rados.0.smithi107.stdout:11974: delete oid 14 current snap is 0 2020-06-12T02:39:22.056 INFO:tasks.rados.rados.0.smithi107.stdout:11974: done (1 left) 2020-06-12T02:39:22.056 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:39:22.056 INFO:tasks.rados.rados.0.smithi107.stdout:11975: seq_num 11974 ranges {0=548864,548864=442368,991232=360448} 2020-06-12T02:39:22.058 INFO:tasks.rados.rados.0.smithi107.stdout:11975: writing smithi10729407-16 from 0 to 548864 tid 1 2020-06-12T02:39:22.061 INFO:tasks.rados.rados.0.smithi107.stdout:11975: writing smithi10729407-16 from 548864 to 991232 tid 2 2020-06-12T02:39:22.077 INFO:tasks.rados.rados.0.smithi107.stdout:11975: writing smithi10729407-16 from 991232 to 1351680 tid 3 2020-06-12T02:39:22.077 INFO:tasks.rados.rados.0.smithi107.stdout:11976: delete oid 4 current snap is 0 2020-06-12T02:39:22.078 INFO:tasks.rados.rados.0.smithi107.stdout:11972: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:22.078 INFO:tasks.rados.rados.0.smithi107.stdout:11972: finishing write tid 2 to smithi10729407-1 2020-06-12T02:39:22.078 INFO:tasks.rados.rados.0.smithi107.stdout:11972: finishing write tid 3 to smithi10729407-1 2020-06-12T02:39:22.078 INFO:tasks.rados.rados.0.smithi107.stdout:11972: finishing write tid 5 to smithi10729407-1 2020-06-12T02:39:22.078 INFO:tasks.rados.rados.0.smithi107.stdout:11972: finishing write tid 6 to smithi10729407-1 2020-06-12T02:39:22.079 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 2189 (ObjNum 11971 snap 0 seq_num 11971) dirty exists 2020-06-12T02:39:22.079 INFO:tasks.rados.rados.0.smithi107.stdout:11972: left oid 1 (ObjNum 11971 snap 0 seq_num 11971) 2020-06-12T02:39:22.079 INFO:tasks.rados.rados.0.smithi107.stdout:11975: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:22.079 INFO:tasks.rados.rados.0.smithi107.stdout:11972: done (2 left) 2020-06-12T02:39:22.079 INFO:tasks.rados.rados.0.smithi107.stdout:11976: done (1 left) 2020-06-12T02:39:22.079 INFO:tasks.rados.rados.0.smithi107.stdout:11977: delete oid 17 current snap is 0 2020-06-12T02:39:22.080 INFO:tasks.rados.rados.0.smithi107.stdout:11977: done (1 left) 2020-06-12T02:39:22.080 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:39:22.080 INFO:tasks.rados.rados.0.smithi107.stdout:11978: seq_num 11977 ranges {0=598016,598016=491520,1089536=311296} 2020-06-12T02:39:22.080 INFO:tasks.rados.rados.0.smithi107.stdout:11978: writing smithi10729407-4 from 0 to 598016 tid 1 2020-06-12T02:39:22.081 INFO:tasks.rados.rados.0.smithi107.stdout:11978: writing smithi10729407-4 from 598016 to 1089536 tid 2 2020-06-12T02:39:22.082 INFO:tasks.rados.rados.0.smithi107.stdout:11978: writing smithi10729407-4 from 1089536 to 1400832 tid 3 2020-06-12T02:39:22.082 INFO:tasks.rados.rados.0.smithi107.stdout:11979: delete oid 12 current snap is 0 2020-06-12T02:39:22.082 INFO:tasks.rados.rados.0.smithi107.stdout:11975: finishing write tid 2 to smithi10729407-16 2020-06-12T02:39:22.082 INFO:tasks.rados.rados.0.smithi107.stdout:11975: finishing write tid 3 to smithi10729407-16 2020-06-12T02:39:22.082 INFO:tasks.rados.rados.0.smithi107.stdout:11975: finishing write tid 5 to smithi10729407-16 2020-06-12T02:39:22.083 INFO:tasks.rados.rados.0.smithi107.stdout:11975: finishing write tid 6 to smithi10729407-16 2020-06-12T02:39:22.083 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 2115 (ObjNum 11974 snap 0 seq_num 11974) dirty exists 2020-06-12T02:39:22.083 INFO:tasks.rados.rados.0.smithi107.stdout:11975: left oid 16 (ObjNum 11974 snap 0 seq_num 11974) 2020-06-12T02:39:22.083 INFO:tasks.rados.rados.0.smithi107.stdout:11978: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:22.084 INFO:tasks.rados.rados.0.smithi107.stdout:11975: done (2 left) 2020-06-12T02:39:22.084 INFO:tasks.rados.rados.0.smithi107.stdout:11979: done (1 left) 2020-06-12T02:39:22.084 INFO:tasks.rados.rados.0.smithi107.stdout:11980: delete oid 20 current snap is 0 2020-06-12T02:39:22.085 INFO:tasks.rados.rados.0.smithi107.stdout:11980: done (1 left) 2020-06-12T02:39:22.085 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:39:22.085 INFO:tasks.rados.rados.0.smithi107.stdout:11981: seq_num 11980 ranges {0=647168,647168=548864,1196032=253952} 2020-06-12T02:39:22.088 INFO:tasks.rados.rados.0.smithi107.stdout:11981: writing smithi10729407-5 from 0 to 647168 tid 1 2020-06-12T02:39:22.091 INFO:tasks.rados.rados.0.smithi107.stdout:11981: writing smithi10729407-5 from 647168 to 1196032 tid 2 2020-06-12T02:39:22.093 INFO:tasks.rados.rados.0.smithi107.stdout:11981: writing smithi10729407-5 from 1196032 to 1449984 tid 3 2020-06-12T02:39:22.093 INFO:tasks.rados.rados.0.smithi107.stdout:11982: delete oid 18 current snap is 0 2020-06-12T02:39:22.094 INFO:tasks.rados.rados.0.smithi107.stdout:11978: finishing write tid 2 to smithi10729407-4 2020-06-12T02:39:22.094 INFO:tasks.rados.rados.0.smithi107.stdout:11978: finishing write tid 3 to smithi10729407-4 2020-06-12T02:39:22.094 INFO:tasks.rados.rados.0.smithi107.stdout:11978: finishing write tid 5 to smithi10729407-4 2020-06-12T02:39:22.094 INFO:tasks.rados.rados.0.smithi107.stdout:11978: finishing write tid 6 to smithi10729407-4 2020-06-12T02:39:22.095 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 2232 (ObjNum 11977 snap 0 seq_num 11977) dirty exists 2020-06-12T02:39:22.095 INFO:tasks.rados.rados.0.smithi107.stdout:11978: left oid 4 (ObjNum 11977 snap 0 seq_num 11977) 2020-06-12T02:39:22.104 INFO:tasks.rados.rados.0.smithi107.stdout:11978: done (2 left) 2020-06-12T02:39:22.105 INFO:tasks.rados.rados.0.smithi107.stdout:11982: done (1 left) 2020-06-12T02:39:22.105 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:39:22.105 INFO:tasks.rados.rados.0.smithi107.stdout:11983: seq_num 11982 ranges {2007040=679936,2686976=770048,3457024=32768} 2020-06-12T02:39:22.110 INFO:tasks.rados.rados.0.smithi107.stdout:11983: writing smithi10729407-8 from 2007040 to 2686976 tid 1 2020-06-12T02:39:22.115 INFO:tasks.rados.rados.0.smithi107.stdout:11983: writing smithi10729407-8 from 2686976 to 3457024 tid 2 2020-06-12T02:39:22.115 INFO:tasks.rados.rados.0.smithi107.stdout:11983: writing smithi10729407-8 from 3457024 to 3489792 tid 3 2020-06-12T02:39:22.115 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:39:22.115 INFO:tasks.rados.rados.0.smithi107.stdout:11984: seq_num 11983 ranges {1302528=696320,1998848=647168,2646016=155648} 2020-06-12T02:39:22.117 INFO:tasks.ceph.osd.6.smithi188.stderr:2020-06-12T02:39:22.115+0000 7f87c1d08700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26678) UID: 0 2020-06-12T02:39:22.126 INFO:tasks.rados.rados.0.smithi107.stdout:11984: writing smithi10729407-1 from 1302528 to 1998848 tid 1 2020-06-12T02:39:22.130 INFO:tasks.rados.rados.0.smithi107.stdout:11984: writing smithi10729407-1 from 1998848 to 2646016 tid 2 2020-06-12T02:39:22.130 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:22.130 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:22.131 INFO:tasks.rados.rados.0.smithi107.stdout:11984: writing smithi10729407-1 from 2646016 to 2801664 tid 3 2020-06-12T02:39:22.132 INFO:tasks.rados.rados.0.smithi107.stdout:11985: delete oid 16 current snap is 0 2020-06-12T02:39:22.132 INFO:tasks.rados.rados.0.smithi107.stdout:11981: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:22.132 INFO:tasks.rados.rados.0.smithi107.stdout:11981: finishing write tid 2 to smithi10729407-5 2020-06-12T02:39:22.132 INFO:tasks.rados.rados.0.smithi107.stdout:11981: finishing write tid 3 to smithi10729407-5 2020-06-12T02:39:22.133 INFO:tasks.rados.rados.0.smithi107.stdout:11981: finishing write tid 5 to smithi10729407-5 2020-06-12T02:39:22.133 INFO:tasks.rados.rados.0.smithi107.stdout:11981: finishing write tid 6 to smithi10729407-5 2020-06-12T02:39:22.133 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 1131 (ObjNum 11980 snap 0 seq_num 11980) dirty exists 2020-06-12T02:39:22.133 INFO:tasks.rados.rados.0.smithi107.stdout:11981: left oid 5 (ObjNum 11980 snap 0 seq_num 11980) 2020-06-12T02:39:22.133 INFO:tasks.rados.rados.0.smithi107.stdout:11983: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:22.133 INFO:tasks.rados.rados.0.smithi107.stdout:11983: finishing write tid 2 to smithi10729407-8 2020-06-12T02:39:22.134 INFO:tasks.rados.rados.0.smithi107.stdout:11983: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:22.134 INFO:tasks.rados.rados.0.smithi107.stdout:11983: finishing write tid 5 to smithi10729407-8 2020-06-12T02:39:22.134 INFO:tasks.rados.rados.0.smithi107.stdout:11983: finishing write tid 6 to smithi10729407-8 2020-06-12T02:39:22.134 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 2251 (ObjNum 11982 snap 0 seq_num 11982) dirty exists 2020-06-12T02:39:22.134 INFO:tasks.rados.rados.0.smithi107.stdout:11983: left oid 8 (ObjNum 11982 snap 0 seq_num 11982) 2020-06-12T02:39:22.135 INFO:tasks.rados.rados.0.smithi107.stdout:11981: done (3 left) 2020-06-12T02:39:22.135 INFO:tasks.rados.rados.0.smithi107.stdout:11983: done (2 left) 2020-06-12T02:39:22.135 INFO:tasks.rados.rados.0.smithi107.stdout:11985: done (1 left) 2020-06-12T02:39:22.135 INFO:tasks.rados.rados.0.smithi107.stdout:11986: delete oid 15 current snap is 0 2020-06-12T02:39:22.137 INFO:tasks.rados.rados.0.smithi107.stdout:11986: done (1 left) 2020-06-12T02:39:22.138 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:39:22.138 INFO:tasks.rados.rados.0.smithi107.stdout:11987: seq_num 11986 ranges {0=745472,745472=704512,1449984=98304} 2020-06-12T02:39:22.142 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-06-12T02:39:22.143 INFO:tasks.rados.rados.0.smithi107.stdout:11987: writing smithi10729407-10 from 0 to 745472 tid 1 2020-06-12T02:39:22.146 INFO:tasks.rados.rados.0.smithi107.stdout:11987: writing smithi10729407-10 from 745472 to 1449984 tid 2 2020-06-12T02:39:22.147 INFO:tasks.rados.rados.0.smithi107.stdout:11987: writing smithi10729407-10 from 1449984 to 1548288 tid 3 2020-06-12T02:39:22.147 INFO:tasks.rados.rados.0.smithi107.stdout:11988: delete oid 7 current snap is 0 2020-06-12T02:39:22.147 INFO:tasks.rados.rados.0.smithi107.stdout:11984: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:22.147 INFO:tasks.rados.rados.0.smithi107.stdout:11984: finishing write tid 2 to smithi10729407-1 2020-06-12T02:39:22.148 INFO:tasks.rados.rados.0.smithi107.stdout:11984: finishing write tid 3 to smithi10729407-1 2020-06-12T02:39:22.148 INFO:tasks.rados.rados.0.smithi107.stdout:11984: finishing write tid 5 to smithi10729407-1 2020-06-12T02:39:22.148 INFO:tasks.rados.rados.0.smithi107.stdout:11984: finishing write tid 6 to smithi10729407-1 2020-06-12T02:39:22.148 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 2193 (ObjNum 11983 snap 0 seq_num 11983) dirty exists 2020-06-12T02:39:22.148 INFO:tasks.rados.rados.0.smithi107.stdout:11984: left oid 1 (ObjNum 11983 snap 0 seq_num 11983) 2020-06-12T02:39:22.151 INFO:tasks.rados.rados.0.smithi107.stdout:11984: done (2 left) 2020-06-12T02:39:22.151 INFO:tasks.rados.rados.0.smithi107.stdout:11988: done (1 left) 2020-06-12T02:39:22.151 INFO:tasks.rados.rados.0.smithi107.stdout:11989: delete oid 9 current snap is 0 2020-06-12T02:39:22.153 INFO:tasks.rados.rados.0.smithi107.stdout:11987: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:22.154 INFO:tasks.rados.rados.0.smithi107.stdout:11989: done (1 left) 2020-06-12T02:39:22.155 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:39:22.155 INFO:tasks.rados.rados.0.smithi107.stdout:11990: seq_num 11989 ranges {1859584=802816,2662400=802816} 2020-06-12T02:39:22.169 INFO:tasks.rados.rados.0.smithi107.stdout:11990: writing smithi10729407-2 from 1859584 to 2662400 tid 1 2020-06-12T02:39:22.174 INFO:tasks.rados.rados.0.smithi107.stdout:11990: writing smithi10729407-2 from 2662400 to 3465216 tid 2 2020-06-12T02:39:22.174 INFO:tasks.rados.rados.0.smithi107.stdout:11991: delete oid 13 current snap is 0 2020-06-12T02:39:22.174 INFO:tasks.rados.rados.0.smithi107.stdout:11987: finishing write tid 2 to smithi10729407-10 2020-06-12T02:39:22.174 INFO:tasks.rados.rados.0.smithi107.stdout:11987: finishing write tid 3 to smithi10729407-10 2020-06-12T02:39:22.175 INFO:tasks.rados.rados.0.smithi107.stdout:11987: finishing write tid 5 to smithi10729407-10 2020-06-12T02:39:22.175 INFO:tasks.rados.rados.0.smithi107.stdout:11987: finishing write tid 6 to smithi10729407-10 2020-06-12T02:39:22.175 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 909 (ObjNum 11986 snap 0 seq_num 11986) dirty exists 2020-06-12T02:39:22.175 INFO:tasks.rados.rados.0.smithi107.stdout:11987: left oid 10 (ObjNum 11986 snap 0 seq_num 11986) 2020-06-12T02:39:22.176 INFO:tasks.rados.rados.0.smithi107.stdout:11987: done (2 left) 2020-06-12T02:39:22.176 INFO:tasks.rados.rados.0.smithi107.stdout:11991: done (1 left) 2020-06-12T02:39:22.176 INFO:tasks.rados.rados.0.smithi107.stdout:11992: delete oid 19 current snap is 0 2020-06-12T02:39:22.179 INFO:tasks.rados.rados.0.smithi107.stdout:11992: done (1 left) 2020-06-12T02:39:22.179 INFO:tasks.rados.rados.0.smithi107.stdout:11993: delete oid 19 current snap is 0 2020-06-12T02:39:22.180 INFO:tasks.rados.rados.0.smithi107.stdout:11990: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:22.188 INFO:tasks.rados.rados.0.smithi107.stdout:11990: finishing write tid 2 to smithi10729407-2 2020-06-12T02:39:22.188 INFO:tasks.rados.rados.0.smithi107.stdout:11990: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:22.188 INFO:tasks.rados.rados.0.smithi107.stdout:11993: done (1 left) 2020-06-12T02:39:22.188 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:39:22.189 INFO:tasks.rados.rados.0.smithi107.stdout:11994: seq_num 11993 ranges {1400832=466944,1867776=745472,2613248=458752} 2020-06-12T02:39:22.194 INFO:tasks.rados.rados.0.smithi107.stdout:11994: writing smithi10729407-4 from 1400832 to 1867776 tid 1 2020-06-12T02:39:22.198 INFO:tasks.rados.rados.0.smithi107.stdout:11994: writing smithi10729407-4 from 1867776 to 2613248 tid 2 2020-06-12T02:39:22.202 INFO:tasks.rados.rados.0.smithi107.stdout:11994: writing smithi10729407-4 from 2613248 to 3072000 tid 3 2020-06-12T02:39:22.202 INFO:tasks.rados.rados.0.smithi107.stdout:11995: delete oid 1 current snap is 0 2020-06-12T02:39:22.202 INFO:tasks.rados.rados.0.smithi107.stdout:11990: finishing write tid 5 to smithi10729407-2 2020-06-12T02:39:22.202 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 1131 (ObjNum 11989 snap 0 seq_num 11989) dirty exists 2020-06-12T02:39:22.203 INFO:tasks.rados.rados.0.smithi107.stdout:11990: left oid 2 (ObjNum 11989 snap 0 seq_num 11989) 2020-06-12T02:39:22.204 INFO:tasks.rados.rados.0.smithi107.stdout:11994: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:22.205 INFO:tasks.rados.rados.0.smithi107.stdout:11990: done (2 left) 2020-06-12T02:39:22.205 INFO:tasks.rados.rados.0.smithi107.stdout:11995: done (1 left) 2020-06-12T02:39:22.205 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:39:22.205 INFO:tasks.rados.rados.0.smithi107.stdout:11996: seq_num 11995 ranges {0=499712,499712=516096,1015808=475136,1490944=212992} 2020-06-12T02:39:22.208 INFO:tasks.rados.rados.0.smithi107.stdout:11996: writing smithi10729407-9 from 0 to 499712 tid 1 2020-06-12T02:39:22.211 INFO:tasks.rados.rados.0.smithi107.stdout:11996: writing smithi10729407-9 from 499712 to 1015808 tid 2 2020-06-12T02:39:22.214 INFO:tasks.rados.rados.0.smithi107.stdout:11996: writing smithi10729407-9 from 1015808 to 1490944 tid 3 2020-06-12T02:39:22.216 INFO:tasks.rados.rados.0.smithi107.stdout:11996: writing smithi10729407-9 from 1490944 to 1703936 tid 4 2020-06-12T02:39:22.216 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 11 current snap is 0 2020-06-12T02:39:22.216 INFO:tasks.rados.rados.0.smithi107.stdout:11997: seq_num 11996 ranges {0=516096,516096=794624,1310720=409600} 2020-06-12T02:39:22.218 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:39:22.215+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:39:22.219 INFO:tasks.rados.rados.0.smithi107.stdout:11997: writing smithi10729407-11 from 0 to 516096 tid 1 2020-06-12T02:39:22.224 INFO:tasks.rados.rados.0.smithi107.stdout:11997: writing smithi10729407-11 from 516096 to 1310720 tid 2 2020-06-12T02:39:22.228 INFO:tasks.rados.rados.0.smithi107.stdout:11997: writing smithi10729407-11 from 1310720 to 1720320 tid 3 2020-06-12T02:39:22.228 INFO:tasks.rados.rados.0.smithi107.stdout:11998: delete oid 1 current snap is 0 2020-06-12T02:39:22.228 INFO:tasks.rados.rados.0.smithi107.stdout:11994: finishing write tid 2 to smithi10729407-4 2020-06-12T02:39:22.228 INFO:tasks.rados.rados.0.smithi107.stdout:11994: finishing write tid 3 to smithi10729407-4 2020-06-12T02:39:22.228 INFO:tasks.rados.rados.0.smithi107.stdout:11994: finishing write tid 5 to smithi10729407-4 2020-06-12T02:39:22.229 INFO:tasks.rados.rados.0.smithi107.stdout:11994: finishing write tid 6 to smithi10729407-4 2020-06-12T02:39:22.229 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 2236 (ObjNum 11993 snap 0 seq_num 11993) dirty exists 2020-06-12T02:39:22.229 INFO:tasks.rados.rados.0.smithi107.stdout:11994: left oid 4 (ObjNum 11993 snap 0 seq_num 11993) 2020-06-12T02:39:22.229 INFO:tasks.rados.rados.0.smithi107.stdout:11996: finishing write tid 1 to smithi10729407-9 2020-06-12T02:39:22.230 INFO:tasks.rados.rados.0.smithi107.stdout:11996: finishing write tid 2 to smithi10729407-9 2020-06-12T02:39:22.230 INFO:tasks.rados.rados.0.smithi107.stdout:11996: finishing write tid 3 to smithi10729407-9 2020-06-12T02:39:22.230 INFO:tasks.rados.rados.0.smithi107.stdout:11996: finishing write tid 4 to smithi10729407-9 2020-06-12T02:39:22.230 INFO:tasks.rados.rados.0.smithi107.stdout:11996: finishing write tid 6 to smithi10729407-9 2020-06-12T02:39:22.230 INFO:tasks.rados.rados.0.smithi107.stdout:11996: finishing write tid 7 to smithi10729407-9 2020-06-12T02:39:22.231 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1937 (ObjNum 11995 snap 0 seq_num 11995) dirty exists 2020-06-12T02:39:22.231 INFO:tasks.rados.rados.0.smithi107.stdout:11996: left oid 9 (ObjNum 11995 snap 0 seq_num 11995) 2020-06-12T02:39:22.232 INFO:tasks.rados.rados.0.smithi107.stdout:11997: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:22.236 INFO:tasks.rados.rados.0.smithi107.stdout:11997: finishing write tid 2 to smithi10729407-11 2020-06-12T02:39:22.237 INFO:tasks.rados.rados.0.smithi107.stdout:11997: finishing write tid 3 to smithi10729407-11 2020-06-12T02:39:22.237 INFO:tasks.rados.rados.0.smithi107.stdout:11997: finishing write tid 5 to smithi10729407-11 2020-06-12T02:39:22.238 INFO:tasks.rados.rados.0.smithi107.stdout:11997: finishing write tid 6 to smithi10729407-11 2020-06-12T02:39:22.238 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1837 (ObjNum 11996 snap 0 seq_num 11996) dirty exists 2020-06-12T02:39:22.239 INFO:tasks.rados.rados.0.smithi107.stdout:11997: left oid 11 (ObjNum 11996 snap 0 seq_num 11996) 2020-06-12T02:39:22.240 INFO:tasks.rados.rados.0.smithi107.stdout:11994: done (3 left) 2020-06-12T02:39:22.240 INFO:tasks.rados.rados.0.smithi107.stdout:11996: done (2 left) 2020-06-12T02:39:22.240 INFO:tasks.rados.rados.0.smithi107.stdout:11997: done (1 left) 2020-06-12T02:39:22.240 INFO:tasks.rados.rados.0.smithi107.stdout:11998: done (0 left) 2020-06-12T02:39:22.240 INFO:tasks.rados.rados.0.smithi107.stdout:11999: delete oid 11 current snap is 0 2020-06-12T02:39:22.241 INFO:tasks.rados.rados.0.smithi107.stdout:11999: done (0 left) 2020-06-12T02:39:22.242 INFO:tasks.rados.rados.0.smithi107.stdout:12000: delete oid 6 current snap is 0 2020-06-12T02:39:22.244 INFO:tasks.rados.rados.0.smithi107.stdout:12000: done (0 left) 2020-06-12T02:39:22.244 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:39:22.245 INFO:tasks.rados.rados.0.smithi107.stdout:12001: seq_num 12000 ranges {3489792=581632,4071424=786432,4857856=417792} 2020-06-12T02:39:22.266 INFO:tasks.rados.rados.0.smithi107.stdout:12001: writing smithi10729407-8 from 3489792 to 4071424 tid 1 2020-06-12T02:39:22.270 INFO:tasks.rados.rados.0.smithi107.stdout:12001: writing smithi10729407-8 from 4071424 to 4857856 tid 2 2020-06-12T02:39:22.273 INFO:tasks.rados.rados.0.smithi107.stdout:12001: writing smithi10729407-8 from 4857856 to 5275648 tid 3 2020-06-12T02:39:22.273 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:39:22.273 INFO:tasks.rados.rados.0.smithi107.stdout:12002: seq_num 12001 ranges {1548288=598016,2146304=671744,2818048=475136,3293184=57344} 2020-06-12T02:39:22.284 INFO:tasks.rados.rados.0.smithi107.stdout:12002: writing smithi10729407-10 from 1548288 to 2146304 tid 1 2020-06-12T02:39:22.288 INFO:tasks.rados.rados.0.smithi107.stdout:12002: writing smithi10729407-10 from 2146304 to 2818048 tid 2 2020-06-12T02:39:22.291 INFO:tasks.rados.rados.0.smithi107.stdout:12002: writing smithi10729407-10 from 2818048 to 3293184 tid 3 2020-06-12T02:39:22.292 INFO:tasks.rados.rados.0.smithi107.stdout:12002: writing smithi10729407-10 from 3293184 to 3350528 tid 4 2020-06-12T02:39:22.292 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:39:22.293 INFO:tasks.rados.rados.0.smithi107.stdout:12003: seq_num 12002 ranges {2334720=614400,2949120=548864,3497984=630784,4128768=24576} 2020-06-12T02:39:22.308 INFO:tasks.rados.rados.0.smithi107.stdout:12003: writing smithi10729407-3 from 2334720 to 2949120 tid 1 2020-06-12T02:39:22.311 INFO:tasks.rados.rados.0.smithi107.stdout:12003: writing smithi10729407-3 from 2949120 to 3497984 tid 2 2020-06-12T02:39:22.315 INFO:tasks.rados.rados.0.smithi107.stdout:12003: writing smithi10729407-3 from 3497984 to 4128768 tid 3 2020-06-12T02:39:22.316 INFO:tasks.rados.rados.0.smithi107.stdout:12003: writing smithi10729407-3 from 4128768 to 4153344 tid 4 2020-06-12T02:39:22.316 INFO:tasks.rados.rados.0.smithi107.stdout:12004: delete oid 20 current snap is 0 2020-06-12T02:39:22.316 INFO:tasks.rados.rados.0.smithi107.stdout:12001: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:22.316 INFO:tasks.rados.rados.0.smithi107.stdout:12001: finishing write tid 2 to smithi10729407-8 2020-06-12T02:39:22.317 INFO:tasks.rados.rados.0.smithi107.stdout:12001: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:22.317 INFO:tasks.rados.rados.0.smithi107.stdout:12001: finishing write tid 5 to smithi10729407-8 2020-06-12T02:39:22.317 INFO:tasks.rados.rados.0.smithi107.stdout:12001: finishing write tid 6 to smithi10729407-8 2020-06-12T02:39:22.318 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 2256 (ObjNum 12000 snap 0 seq_num 12000) dirty exists 2020-06-12T02:39:22.318 INFO:tasks.rados.rados.0.smithi107.stdout:12001: left oid 8 (ObjNum 12000 snap 0 seq_num 12000) 2020-06-12T02:39:22.318 INFO:tasks.rados.rados.0.smithi107.stdout:12002: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:22.318 INFO:tasks.rados.rados.0.smithi107.stdout:12002: finishing write tid 2 to smithi10729407-10 2020-06-12T02:39:22.319 INFO:tasks.rados.rados.0.smithi107.stdout:12002: finishing write tid 3 to smithi10729407-10 2020-06-12T02:39:22.319 INFO:tasks.rados.rados.0.smithi107.stdout:12002: finishing write tid 4 to smithi10729407-10 2020-06-12T02:39:22.319 INFO:tasks.rados.rados.0.smithi107.stdout:12002: finishing write tid 6 to smithi10729407-10 2020-06-12T02:39:22.320 INFO:tasks.rados.rados.0.smithi107.stdout:12002: finishing write tid 7 to smithi10729407-10 2020-06-12T02:39:22.320 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 914 (ObjNum 12001 snap 0 seq_num 12001) dirty exists 2020-06-12T02:39:22.320 INFO:tasks.rados.rados.0.smithi107.stdout:12002: left oid 10 (ObjNum 12001 snap 0 seq_num 12001) 2020-06-12T02:39:22.325 INFO:tasks.rados.rados.0.smithi107.stdout:12001: done (3 left) 2020-06-12T02:39:22.325 INFO:tasks.rados.rados.0.smithi107.stdout:12002: done (2 left) 2020-06-12T02:39:22.326 INFO:tasks.rados.rados.0.smithi107.stdout:12004: done (1 left) 2020-06-12T02:39:22.326 INFO:tasks.rados.rados.0.smithi107.stdout:12005: delete oid 7 current snap is 0 2020-06-12T02:39:22.326 INFO:tasks.rados.rados.0.smithi107.stdout:12005: done (1 left) 2020-06-12T02:39:22.327 INFO:tasks.rados.rados.0.smithi107.stdout:12006: delete oid 15 current snap is 0 2020-06-12T02:39:22.327 INFO:tasks.rados.rados.0.smithi107.stdout:12003: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:22.327 INFO:tasks.rados.rados.0.smithi107.stdout:12003: finishing write tid 2 to smithi10729407-3 2020-06-12T02:39:22.328 INFO:tasks.rados.rados.0.smithi107.stdout:12003: finishing write tid 3 to smithi10729407-3 2020-06-12T02:39:22.328 INFO:tasks.rados.rados.0.smithi107.stdout:12006: done (1 left) 2020-06-12T02:39:22.328 INFO:tasks.rados.rados.0.smithi107.stdout:12007: delete oid 1 current snap is 0 2020-06-12T02:39:22.329 INFO:tasks.rados.rados.0.smithi107.stdout:12003: finishing write tid 4 to smithi10729407-3 2020-06-12T02:39:22.329 INFO:tasks.rados.rados.0.smithi107.stdout:12003: finishing write tid 6 to smithi10729407-3 2020-06-12T02:39:22.331 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:39:22.320+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:39:22.331 INFO:tasks.rados.rados.0.smithi107.stdout:12003: finishing write tid 7 to smithi10729407-3 2020-06-12T02:39:22.332 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 2121 (ObjNum 12002 snap 0 seq_num 12002) dirty exists 2020-06-12T02:39:22.333 INFO:tasks.rados.rados.0.smithi107.stdout:12003: left oid 3 (ObjNum 12002 snap 0 seq_num 12002) 2020-06-12T02:39:22.333 INFO:tasks.rados.rados.0.smithi107.stdout:12003: done (1 left) 2020-06-12T02:39:22.333 INFO:tasks.rados.rados.0.smithi107.stdout:12007: done (0 left) 2020-06-12T02:39:22.334 INFO:tasks.rados.rados.0.smithi107.stdout:12008: delete oid 9 current snap is 0 2020-06-12T02:39:22.334 INFO:tasks.rados.rados.0.smithi107.stdout:12008: done (0 left) 2020-06-12T02:39:22.335 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:39:22.335 INFO:tasks.rados.rados.0.smithi107.stdout:12009: seq_num 12008 ranges {0=720896,720896=704512,1425408=499712} 2020-06-12T02:39:22.336 INFO:teuthology.orchestra.run.smithi188.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:22.337 INFO:teuthology.orchestra.run.smithi188.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:22.337 INFO:tasks.rados.rados.0.smithi107.stdout:12009: writing smithi10729407-6 from 0 to 720896 tid 1 2020-06-12T02:39:22.346 INFO:tasks.rados.rados.0.smithi107.stdout:12009: writing smithi10729407-6 from 720896 to 1425408 tid 2 2020-06-12T02:39:22.347 INFO:tasks.rados.rados.0.smithi107.stdout:12009: writing smithi10729407-6 from 1425408 to 1925120 tid 3 2020-06-12T02:39:22.347 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:39:22.347 INFO:tasks.rados.rados.0.smithi107.stdout:12010: seq_num 12009 ranges {0=737280,737280=589824,1327104=434176,1761280=180224} 2020-06-12T02:39:22.348 INFO:tasks.rados.rados.0.smithi107.stdout:12010: writing smithi10729407-16 from 0 to 737280 tid 1 2020-06-12T02:39:22.348 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-06-12T02:39:22.350 INFO:tasks.rados.rados.0.smithi107.stdout:12010: writing smithi10729407-16 from 737280 to 1327104 tid 2 2020-06-12T02:39:22.352 INFO:tasks.rados.rados.0.smithi107.stdout:12010: writing smithi10729407-16 from 1327104 to 1761280 tid 3 2020-06-12T02:39:22.354 INFO:tasks.rados.rados.0.smithi107.stdout:12010: writing smithi10729407-16 from 1761280 to 1941504 tid 4 2020-06-12T02:39:22.355 INFO:tasks.rados.rados.0.smithi107.stdout:12011: delete oid 18 current snap is 0 2020-06-12T02:39:22.355 INFO:tasks.rados.rados.0.smithi107.stdout:12009: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:22.355 INFO:tasks.rados.rados.0.smithi107.stdout:12009: finishing write tid 2 to smithi10729407-6 2020-06-12T02:39:22.356 INFO:tasks.rados.rados.0.smithi107.stdout:12009: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:22.356 INFO:tasks.rados.rados.0.smithi107.stdout:12009: finishing write tid 5 to smithi10729407-6 2020-06-12T02:39:22.356 INFO:tasks.rados.rados.0.smithi107.stdout:12009: finishing write tid 6 to smithi10729407-6 2020-06-12T02:39:22.356 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 2201 (ObjNum 12008 snap 0 seq_num 12008) dirty exists 2020-06-12T02:39:22.356 INFO:tasks.rados.rados.0.smithi107.stdout:12009: left oid 6 (ObjNum 12008 snap 0 seq_num 12008) 2020-06-12T02:39:22.357 INFO:tasks.rados.rados.0.smithi107.stdout:12010: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:22.357 INFO:tasks.rados.rados.0.smithi107.stdout:12009: done (2 left) 2020-06-12T02:39:22.357 INFO:tasks.rados.rados.0.smithi107.stdout:12011: done (1 left) 2020-06-12T02:39:22.357 INFO:tasks.rados.rados.0.smithi107.stdout:12012: delete oid 17 current snap is 0 2020-06-12T02:39:22.360 INFO:tasks.rados.rados.0.smithi107.stdout:12012: done (1 left) 2020-06-12T02:39:22.360 INFO:tasks.rados.rados.0.smithi107.stdout:12013: delete oid 1 current snap is 0 2020-06-12T02:39:22.362 INFO:tasks.rados.rados.0.smithi107.stdout:12010: finishing write tid 2 to smithi10729407-16 2020-06-12T02:39:22.363 INFO:tasks.rados.rados.0.smithi107.stdout:12013: done (1 left) 2020-06-12T02:39:22.364 INFO:tasks.rados.rados.0.smithi107.stdout:12014: delete oid 4 current snap is 0 2020-06-12T02:39:22.366 INFO:tasks.rados.rados.0.smithi107.stdout:12014: done (1 left) 2020-06-12T02:39:22.366 INFO:tasks.rados.rados.0.smithi107.stdout:12015: delete oid 17 current snap is 0 2020-06-12T02:39:22.367 INFO:tasks.rados.rados.0.smithi107.stdout:12010: finishing write tid 3 to smithi10729407-16 2020-06-12T02:39:22.368 INFO:tasks.rados.rados.0.smithi107.stdout:12010: finishing write tid 4 to smithi10729407-16 2020-06-12T02:39:22.371 INFO:tasks.rados.rados.0.smithi107.stdout:12010: finishing write tid 6 to smithi10729407-16 2020-06-12T02:39:22.372 INFO:tasks.rados.rados.0.smithi107.stdout:12010: finishing write tid 7 to smithi10729407-16 2020-06-12T02:39:22.373 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 2126 (ObjNum 12009 snap 0 seq_num 12009) dirty exists 2020-06-12T02:39:22.373 INFO:tasks.rados.rados.0.smithi107.stdout:12010: left oid 16 (ObjNum 12009 snap 0 seq_num 12009) 2020-06-12T02:39:22.374 INFO:tasks.rados.rados.0.smithi107.stdout:12010: done (1 left) 2020-06-12T02:39:22.374 INFO:tasks.rados.rados.0.smithi107.stdout:12015: done (0 left) 2020-06-12T02:39:22.374 INFO:tasks.rados.rados.0.smithi107.stdout:12016: delete oid 4 current snap is 0 2020-06-12T02:39:22.378 INFO:tasks.rados.rados.0.smithi107.stdout:12016: done (0 left) 2020-06-12T02:39:22.379 INFO:tasks.rados.rados.0.smithi107.stdout:12017: delete oid 17 current snap is 0 2020-06-12T02:39:22.381 INFO:tasks.rados.rados.0.smithi107.stdout:12017: done (0 left) 2020-06-12T02:39:22.382 INFO:tasks.rados.rados.0.smithi107.stdout:12018: delete oid 9 current snap is 0 2020-06-12T02:39:22.383 INFO:tasks.rados.rados.0.smithi107.stdout:12018: done (0 left) 2020-06-12T02:39:22.384 INFO:tasks.rados.rados.0.smithi107.stdout:12019: delete oid 13 current snap is 0 2020-06-12T02:39:22.386 INFO:tasks.rados.rados.0.smithi107.stdout:12019: done (0 left) 2020-06-12T02:39:22.386 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:39:22.387 INFO:tasks.rados.rados.0.smithi107.stdout:12020: seq_num 12019 ranges {0=499712,499712=688128,1187840=761856,1949696=155648} 2020-06-12T02:39:22.389 INFO:tasks.rados.rados.0.smithi107.stdout:12020: writing smithi10729407-19 from 0 to 499712 tid 1 2020-06-12T02:39:22.393 INFO:tasks.rados.rados.0.smithi107.stdout:12020: writing smithi10729407-19 from 499712 to 1187840 tid 2 2020-06-12T02:39:22.397 INFO:tasks.rados.rados.0.smithi107.stdout:12020: writing smithi10729407-19 from 1187840 to 1949696 tid 3 2020-06-12T02:39:22.398 INFO:tasks.rados.rados.0.smithi107.stdout:12020: writing smithi10729407-19 from 1949696 to 2105344 tid 4 2020-06-12T02:39:22.399 INFO:tasks.rados.rados.0.smithi107.stdout:12021: delete oid 13 current snap is 0 2020-06-12T02:39:22.399 INFO:tasks.rados.rados.0.smithi107.stdout:12020: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:22.401 INFO:tasks.rados.rados.0.smithi107.stdout:12021: done (1 left) 2020-06-12T02:39:22.401 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:39:22.402 INFO:tasks.rados.rados.0.smithi107.stdout:12022: seq_num 12021 ranges {0=532480,532480=499712,1032192=778240,1810432=327680} 2020-06-12T02:39:22.404 INFO:tasks.rados.rados.0.smithi107.stdout:12022: writing smithi10729407-20 from 0 to 532480 tid 1 2020-06-12T02:39:22.408 INFO:tasks.rados.rados.0.smithi107.stdout:12022: writing smithi10729407-20 from 532480 to 1032192 tid 2 2020-06-12T02:39:22.415 INFO:tasks.rados.rados.0.smithi107.stdout:12022: writing smithi10729407-20 from 1032192 to 1810432 tid 3 2020-06-12T02:39:22.416 INFO:tasks.rados.rados.0.smithi107.stdout:12022: writing smithi10729407-20 from 1810432 to 2138112 tid 4 2020-06-12T02:39:22.417 INFO:tasks.rados.rados.0.smithi107.stdout:12023: delete oid 7 current snap is 0 2020-06-12T02:39:22.418 INFO:tasks.rados.rados.0.smithi107.stdout:12020: finishing write tid 2 to smithi10729407-19 2020-06-12T02:39:22.418 INFO:tasks.rados.rados.0.smithi107.stdout:12020: finishing write tid 3 to smithi10729407-19 2020-06-12T02:39:22.418 INFO:tasks.rados.rados.0.smithi107.stdout:12020: finishing write tid 4 to smithi10729407-19 2020-06-12T02:39:22.419 INFO:tasks.rados.rados.0.smithi107.stdout:12020: finishing write tid 6 to smithi10729407-19 2020-06-12T02:39:22.419 INFO:tasks.rados.rados.0.smithi107.stdout:12020: finishing write tid 7 to smithi10729407-19 2020-06-12T02:39:22.419 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 2237 (ObjNum 12019 snap 0 seq_num 12019) dirty exists 2020-06-12T02:39:22.420 INFO:tasks.rados.rados.0.smithi107.stdout:12020: left oid 19 (ObjNum 12019 snap 0 seq_num 12019) 2020-06-12T02:39:22.420 INFO:tasks.rados.rados.0.smithi107.stdout:12022: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:22.423 INFO:tasks.rados.rados.0.smithi107.stdout:12020: done (2 left) 2020-06-12T02:39:22.423 INFO:tasks.rados.rados.0.smithi107.stdout:12023: done (1 left) 2020-06-12T02:39:22.424 INFO:tasks.rados.rados.0.smithi107.stdout:12024: delete oid 7 current snap is 0 2020-06-12T02:39:22.424 INFO:tasks.rados.rados.0.smithi107.stdout:12024: done (1 left) 2020-06-12T02:39:22.424 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:39:22.425 INFO:tasks.rados.rados.0.smithi107.stdout:12025: seq_num 12024 ranges {1925120=581632,2506752=557056,3063808=548864,3612672=425984,4038656=73728} 2020-06-12T02:39:22.428 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:39:22.426+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:39:22.434 INFO:tasks.rados.rados.0.smithi107.stdout:12025: writing smithi10729407-6 from 1925120 to 2506752 tid 1 2020-06-12T02:39:22.437 INFO:tasks.rados.rados.0.smithi107.stdout:12025: writing smithi10729407-6 from 2506752 to 3063808 tid 2 2020-06-12T02:39:22.441 INFO:tasks.rados.rados.0.smithi107.stdout:12025: writing smithi10729407-6 from 3063808 to 3612672 tid 3 2020-06-12T02:39:22.443 INFO:tasks.rados.rados.0.smithi107.stdout:12025: writing smithi10729407-6 from 3612672 to 4038656 tid 4 2020-06-12T02:39:22.444 INFO:tasks.rados.rados.0.smithi107.stdout:12025: writing smithi10729407-6 from 4038656 to 4112384 tid 5 2020-06-12T02:39:22.444 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:39:22.444 INFO:tasks.rados.rados.0.smithi107.stdout:12026: seq_num 12025 ranges {0=598016,598016=442368,1040384=761856,1802240=401408} 2020-06-12T02:39:22.447 INFO:tasks.rados.rados.0.smithi107.stdout:12026: writing smithi10729407-15 from 0 to 598016 tid 1 2020-06-12T02:39:22.450 INFO:tasks.rados.rados.0.smithi107.stdout:12026: writing smithi10729407-15 from 598016 to 1040384 tid 2 2020-06-12T02:39:22.454 INFO:tasks.rados.rados.0.smithi107.stdout:12026: writing smithi10729407-15 from 1040384 to 1802240 tid 3 2020-06-12T02:39:22.457 INFO:tasks.rados.rados.0.smithi107.stdout:12026: writing smithi10729407-15 from 1802240 to 2203648 tid 4 2020-06-12T02:39:22.457 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 13 current snap is 0 2020-06-12T02:39:22.457 INFO:tasks.rados.rados.0.smithi107.stdout:12027: seq_num 12026 ranges {0=614400,614400=729088,1343488=516096,1859584=360448} 2020-06-12T02:39:22.460 INFO:tasks.rados.rados.0.smithi107.stdout:12027: writing smithi10729407-13 from 0 to 614400 tid 1 2020-06-12T02:39:22.465 INFO:tasks.rados.rados.0.smithi107.stdout:12027: writing smithi10729407-13 from 614400 to 1343488 tid 2 2020-06-12T02:39:22.468 INFO:tasks.rados.rados.0.smithi107.stdout:12027: writing smithi10729407-13 from 1343488 to 1859584 tid 3 2020-06-12T02:39:22.479 INFO:tasks.rados.rados.0.smithi107.stdout:12027: writing smithi10729407-13 from 1859584 to 2220032 tid 4 2020-06-12T02:39:22.479 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:39:22.482 INFO:tasks.rados.rados.0.smithi107.stdout:12028: seq_num 12027 ranges {0=638976,638976=655360,1294336=720896,2015232=229376} 2020-06-12T02:39:22.482 INFO:tasks.rados.rados.0.smithi107.stdout:12028: writing smithi10729407-11 from 0 to 638976 tid 1 2020-06-12T02:39:22.483 INFO:tasks.rados.rados.0.smithi107.stdout:12028: writing smithi10729407-11 from 638976 to 1294336 tid 2 2020-06-12T02:39:22.488 INFO:tasks.rados.rados.0.smithi107.stdout:12028: writing smithi10729407-11 from 1294336 to 2015232 tid 3 2020-06-12T02:39:22.488 INFO:tasks.rados.rados.0.smithi107.stdout:12028: writing smithi10729407-11 from 2015232 to 2244608 tid 4 2020-06-12T02:39:22.489 INFO:tasks.rados.rados.0.smithi107.stdout:12029: delete oid 2 current snap is 0 2020-06-12T02:39:22.489 INFO:tasks.rados.rados.0.smithi107.stdout:12022: finishing write tid 2 to smithi10729407-20 2020-06-12T02:39:22.489 INFO:tasks.rados.rados.0.smithi107.stdout:12022: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:22.489 INFO:tasks.rados.rados.0.smithi107.stdout:12022: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:22.489 INFO:tasks.rados.rados.0.smithi107.stdout:12022: finishing write tid 6 to smithi10729407-20 2020-06-12T02:39:22.490 INFO:tasks.rados.rados.0.smithi107.stdout:12022: finishing write tid 7 to smithi10729407-20 2020-06-12T02:39:22.497 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 932 (ObjNum 12021 snap 0 seq_num 12021) dirty exists 2020-06-12T02:39:22.498 INFO:tasks.rados.rados.0.smithi107.stdout:12022: left oid 20 (ObjNum 12021 snap 0 seq_num 12021) 2020-06-12T02:39:22.498 INFO:tasks.rados.rados.0.smithi107.stdout:12025: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:22.498 INFO:tasks.rados.rados.0.smithi107.stdout:12025: finishing write tid 2 to smithi10729407-6 2020-06-12T02:39:22.498 INFO:tasks.rados.rados.0.smithi107.stdout:12025: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:22.498 INFO:tasks.rados.rados.0.smithi107.stdout:12025: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:22.499 INFO:tasks.rados.rados.0.smithi107.stdout:12025: finishing write tid 5 to smithi10729407-6 2020-06-12T02:39:22.499 INFO:tasks.rados.rados.0.smithi107.stdout:12025: finishing write tid 7 to smithi10729407-6 2020-06-12T02:39:22.499 INFO:tasks.rados.rados.0.smithi107.stdout:12025: finishing write tid 8 to smithi10729407-6 2020-06-12T02:39:22.499 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 2208 (ObjNum 12024 snap 0 seq_num 12024) dirty exists 2020-06-12T02:39:22.499 INFO:tasks.rados.rados.0.smithi107.stdout:12025: left oid 6 (ObjNum 12024 snap 0 seq_num 12024) 2020-06-12T02:39:22.500 INFO:tasks.rados.rados.0.smithi107.stdout:12026: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:22.500 INFO:tasks.rados.rados.0.smithi107.stdout:12026: finishing write tid 2 to smithi10729407-15 2020-06-12T02:39:22.501 INFO:tasks.rados.rados.0.smithi107.stdout:12026: finishing write tid 3 to smithi10729407-15 2020-06-12T02:39:22.501 INFO:tasks.rados.rados.0.smithi107.stdout:12026: finishing write tid 4 to smithi10729407-15 2020-06-12T02:39:22.501 INFO:tasks.rados.rados.0.smithi107.stdout:12026: finishing write tid 6 to smithi10729407-15 2020-06-12T02:39:22.501 INFO:tasks.rados.rados.0.smithi107.stdout:12026: finishing write tid 7 to smithi10729407-15 2020-06-12T02:39:22.502 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 2262 (ObjNum 12025 snap 0 seq_num 12025) dirty exists 2020-06-12T02:39:22.502 INFO:tasks.rados.rados.0.smithi107.stdout:12026: left oid 15 (ObjNum 12025 snap 0 seq_num 12025) 2020-06-12T02:39:22.502 INFO:tasks.rados.rados.0.smithi107.stdout:12027: finishing write tid 1 to smithi10729407-13 2020-06-12T02:39:22.502 INFO:tasks.rados.rados.0.smithi107.stdout:12027: finishing write tid 2 to smithi10729407-13 2020-06-12T02:39:22.503 INFO:tasks.rados.rados.0.smithi107.stdout:12027: finishing write tid 3 to smithi10729407-13 2020-06-12T02:39:22.503 INFO:tasks.rados.rados.0.smithi107.stdout:12027: finishing write tid 4 to smithi10729407-13 2020-06-12T02:39:22.503 INFO:tasks.rados.rados.0.smithi107.stdout:12027: finishing write tid 6 to smithi10729407-13 2020-06-12T02:39:22.503 INFO:tasks.rados.rados.0.smithi107.stdout:12028: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:22.503 INFO:tasks.rados.rados.0.smithi107.stdout:12027: finishing write tid 7 to smithi10729407-13 2020-06-12T02:39:22.503 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1875 (ObjNum 12026 snap 0 seq_num 12026) dirty exists 2020-06-12T02:39:22.504 INFO:tasks.rados.rados.0.smithi107.stdout:12027: left oid 13 (ObjNum 12026 snap 0 seq_num 12026) 2020-06-12T02:39:22.505 INFO:tasks.rados.rados.0.smithi107.stdout:12022: done (5 left) 2020-06-12T02:39:22.505 INFO:tasks.rados.rados.0.smithi107.stdout:12025: done (4 left) 2020-06-12T02:39:22.506 INFO:tasks.rados.rados.0.smithi107.stdout:12026: done (3 left) 2020-06-12T02:39:22.506 INFO:tasks.rados.rados.0.smithi107.stdout:12027: done (2 left) 2020-06-12T02:39:22.506 INFO:tasks.rados.rados.0.smithi107.stdout:12029: done (1 left) 2020-06-12T02:39:22.506 INFO:tasks.rados.rados.0.smithi107.stdout:12030: delete oid 8 current snap is 0 2020-06-12T02:39:22.506 INFO:tasks.rados.rados.0.smithi107.stdout:12030: done (1 left) 2020-06-12T02:39:22.506 INFO:tasks.rados.rados.0.smithi107.stdout:12031: delete oid 9 current snap is 0 2020-06-12T02:39:22.507 INFO:tasks.rados.rados.0.smithi107.stdout:12028: finishing write tid 2 to smithi10729407-11 2020-06-12T02:39:22.507 INFO:tasks.rados.rados.0.smithi107.stdout:12028: finishing write tid 3 to smithi10729407-11 2020-06-12T02:39:22.507 INFO:tasks.rados.rados.0.smithi107.stdout:12028: finishing write tid 4 to smithi10729407-11 2020-06-12T02:39:22.508 INFO:tasks.rados.rados.0.smithi107.stdout:12031: done (1 left) 2020-06-12T02:39:22.508 INFO:tasks.rados.rados.0.smithi107.stdout:12032: delete oid 18 current snap is 0 2020-06-12T02:39:22.508 INFO:tasks.rados.rados.0.smithi107.stdout:12028: finishing write tid 6 to smithi10729407-11 2020-06-12T02:39:22.508 INFO:tasks.rados.rados.0.smithi107.stdout:12028: finishing write tid 7 to smithi10729407-11 2020-06-12T02:39:22.508 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1843 (ObjNum 12027 snap 0 seq_num 12027) dirty exists 2020-06-12T02:39:22.508 INFO:tasks.rados.rados.0.smithi107.stdout:12028: left oid 11 (ObjNum 12027 snap 0 seq_num 12027) 2020-06-12T02:39:22.509 INFO:tasks.rados.rados.0.smithi107.stdout:12028: done (1 left) 2020-06-12T02:39:22.509 INFO:tasks.rados.rados.0.smithi107.stdout:12032: done (0 left) 2020-06-12T02:39:22.509 INFO:tasks.rados.rados.0.smithi107.stdout:12033: delete oid 10 current snap is 0 2020-06-12T02:39:22.509 INFO:tasks.rados.rados.0.smithi107.stdout:12033: done (0 left) 2020-06-12T02:39:22.509 INFO:tasks.rados.rados.0.smithi107.stdout:12034: delete oid 9 current snap is 0 2020-06-12T02:39:22.510 INFO:tasks.rados.rados.0.smithi107.stdout:12034: done (0 left) 2020-06-12T02:39:22.510 INFO:tasks.rados.rados.0.smithi107.stdout:12035: delete oid 7 current snap is 0 2020-06-12T02:39:22.511 INFO:tasks.rados.rados.0.smithi107.stdout:12035: done (0 left) 2020-06-12T02:39:22.511 INFO:tasks.rados.rados.0.smithi107.stdout:12036: delete oid 3 current snap is 0 2020-06-12T02:39:22.511 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:22.512 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:22.512 INFO:tasks.rados.rados.0.smithi107.stdout:12036: done (0 left) 2020-06-12T02:39:22.513 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:39:22.513 INFO:tasks.rados.rados.0.smithi107.stdout:12037: seq_num 12036 ranges {2220032=786432,3006464=466944,3473408=491520,3964928=581632,4546560=65536} 2020-06-12T02:39:22.521 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-06-12T02:39:22.541 INFO:tasks.rados.rados.0.smithi107.stdout:12037: writing smithi10729407-13 from 2220032 to 3006464 tid 1 2020-06-12T02:39:22.541 INFO:tasks.rados.rados.0.smithi107.stdout:12037: writing smithi10729407-13 from 3006464 to 3473408 tid 2 2020-06-12T02:39:22.541 INFO:tasks.rados.rados.0.smithi107.stdout:12037: writing smithi10729407-13 from 3473408 to 3964928 tid 3 2020-06-12T02:39:22.542 INFO:tasks.rados.rados.0.smithi107.stdout:12037: writing smithi10729407-13 from 3964928 to 4546560 tid 4 2020-06-12T02:39:22.542 INFO:tasks.rados.rados.0.smithi107.stdout:12037: writing smithi10729407-13 from 4546560 to 4612096 tid 5 2020-06-12T02:39:22.542 INFO:tasks.rados.rados.0.smithi107.stdout:12037: finishing write tid 1 to smithi10729407-13 2020-06-12T02:39:22.542 INFO:tasks.rados.rados.0.smithi107.stdout:12038: delete oid 17 current snap is 0 2020-06-12T02:39:22.543 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:39:22.529+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:39:22.549 INFO:tasks.rados.rados.0.smithi107.stdout:12037: finishing write tid 2 to smithi10729407-13 2020-06-12T02:39:22.549 INFO:tasks.rados.rados.0.smithi107.stdout:12038: done (1 left) 2020-06-12T02:39:22.551 INFO:tasks.rados.rados.0.smithi107.stdout:12039: delete oid 10 current snap is 0 2020-06-12T02:39:22.551 INFO:tasks.rados.rados.0.smithi107.stdout:12037: finishing write tid 3 to smithi10729407-13 2020-06-12T02:39:22.555 INFO:tasks.rados.rados.0.smithi107.stdout:12037: finishing write tid 4 to smithi10729407-13 2020-06-12T02:39:22.556 INFO:tasks.rados.rados.0.smithi107.stdout:12037: finishing write tid 5 to smithi10729407-13 2020-06-12T02:39:22.557 INFO:tasks.rados.rados.0.smithi107.stdout:12037: finishing write tid 7 to smithi10729407-13 2020-06-12T02:39:22.557 INFO:tasks.rados.rados.0.smithi107.stdout:12039: done (1 left) 2020-06-12T02:39:22.557 INFO:tasks.rados.rados.0.smithi107.stdout:12040: delete oid 12 current snap is 0 2020-06-12T02:39:22.557 INFO:tasks.rados.rados.0.smithi107.stdout:12037: finishing write tid 8 to smithi10729407-13 2020-06-12T02:39:22.557 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1881 (ObjNum 12036 snap 0 seq_num 12036) dirty exists 2020-06-12T02:39:22.558 INFO:tasks.rados.rados.0.smithi107.stdout:12037: left oid 13 (ObjNum 12036 snap 0 seq_num 12036) 2020-06-12T02:39:22.558 INFO:tasks.rados.rados.0.smithi107.stdout:12037: done (1 left) 2020-06-12T02:39:22.558 INFO:tasks.rados.rados.0.smithi107.stdout:12040: done (0 left) 2020-06-12T02:39:22.558 INFO:tasks.rados.rados.0.smithi107.stdout:12041: delete oid 14 current snap is 0 2020-06-12T02:39:22.558 INFO:tasks.rados.rados.0.smithi107.stdout:12041: done (0 left) 2020-06-12T02:39:22.559 INFO:tasks.rados.rados.0.smithi107.stdout:12042: delete oid 11 current snap is 0 2020-06-12T02:39:22.560 INFO:tasks.rados.rados.0.smithi107.stdout:12042: done (0 left) 2020-06-12T02:39:22.560 INFO:tasks.rados.rados.0.smithi107.stdout:12043: delete oid 12 current snap is 0 2020-06-12T02:39:22.561 INFO:tasks.rados.rados.0.smithi107.stdout:12043: done (0 left) 2020-06-12T02:39:22.561 INFO:tasks.rados.rados.0.smithi107.stdout:12044: delete oid 1 current snap is 0 2020-06-12T02:39:22.564 INFO:tasks.rados.rados.0.smithi107.stdout:12044: done (0 left) 2020-06-12T02:39:22.564 INFO:tasks.rados.rados.0.smithi107.stdout:12045: delete oid 5 current snap is 0 2020-06-12T02:39:22.566 INFO:tasks.rados.rados.0.smithi107.stdout:12045: done (0 left) 2020-06-12T02:39:22.566 INFO:tasks.rados.rados.0.smithi107.stdout:12046: delete oid 4 current snap is 0 2020-06-12T02:39:22.570 INFO:tasks.rados.rados.0.smithi107.stdout:12046: done (0 left) 2020-06-12T02:39:22.570 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:39:22.570 INFO:tasks.rados.rados.0.smithi107.stdout:12047: seq_num 12046 ranges {1941504=155648} 2020-06-12T02:39:22.581 INFO:tasks.rados.rados.0.smithi107.stdout:12047: writing smithi10729407-16 from 1941504 to 2097152 tid 1 2020-06-12T02:39:22.581 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:39:22.582 INFO:tasks.rados.rados.0.smithi107.stdout:12048: seq_num 12047 ranges {0=172032} 2020-06-12T02:39:22.582 INFO:tasks.rados.rados.0.smithi107.stdout:12048: writing smithi10729407-18 from 0 to 172032 tid 1 2020-06-12T02:39:22.583 INFO:tasks.rados.rados.0.smithi107.stdout:12049: delete oid 17 current snap is 0 2020-06-12T02:39:22.585 INFO:tasks.rados.rados.0.smithi107.stdout:12047: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:22.588 INFO:tasks.rados.rados.0.smithi107.stdout:12049: done (2 left) 2020-06-12T02:39:22.588 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:39:22.588 INFO:tasks.rados.rados.0.smithi107.stdout:12050: seq_num 12049 ranges {0=204800} 2020-06-12T02:39:22.589 INFO:tasks.rados.rados.0.smithi107.stdout:12050: writing smithi10729407-2 from 0 to 204800 tid 1 2020-06-12T02:39:22.589 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:39:22.596 INFO:tasks.rados.rados.0.smithi107.stdout:12051: seq_num 12050 ranges {0=221184} 2020-06-12T02:39:22.597 INFO:tasks.rados.rados.0.smithi107.stdout:12051: writing smithi10729407-11 from 0 to 221184 tid 1 2020-06-12T02:39:22.597 INFO:tasks.rados.rados.0.smithi107.stdout:12052: delete oid 15 current snap is 0 2020-06-12T02:39:22.597 INFO:tasks.rados.rados.0.smithi107.stdout:12048: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:22.598 INFO:tasks.rados.rados.0.smithi107.stdout:12048: finishing write tid 3 to smithi10729407-18 2020-06-12T02:39:22.598 INFO:tasks.rados.rados.0.smithi107.stdout:12050: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:22.598 INFO:tasks.rados.rados.0.smithi107.stdout:12048: finishing write tid 4 to smithi10729407-18 2020-06-12T02:39:22.598 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1965 (ObjNum 12047 snap 0 seq_num 12047) dirty exists 2020-06-12T02:39:22.598 INFO:tasks.rados.rados.0.smithi107.stdout:12048: left oid 18 (ObjNum 12047 snap 0 seq_num 12047) 2020-06-12T02:39:22.599 INFO:tasks.rados.rados.0.smithi107.stdout:12050: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:22.599 INFO:tasks.rados.rados.0.smithi107.stdout:12048: done (4 left) 2020-06-12T02:39:22.599 INFO:tasks.rados.rados.0.smithi107.stdout:12052: done (3 left) 2020-06-12T02:39:22.599 INFO:tasks.rados.rados.0.smithi107.stdout:12053: delete oid 20 current snap is 0 2020-06-12T02:39:22.599 INFO:tasks.rados.rados.0.smithi107.stdout:12051: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:22.600 INFO:tasks.rados.rados.0.smithi107.stdout:12047: finishing write tid 3 to smithi10729407-16 2020-06-12T02:39:22.601 INFO:tasks.rados.rados.0.smithi107.stdout:12053: done (3 left) 2020-06-12T02:39:22.601 INFO:tasks.rados.rados.0.smithi107.stdout:12054: delete oid 20 current snap is 0 2020-06-12T02:39:22.601 INFO:tasks.rados.rados.0.smithi107.stdout:12050: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:22.601 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 1134 (ObjNum 12049 snap 0 seq_num 12049) dirty exists 2020-06-12T02:39:22.602 INFO:tasks.rados.rados.0.smithi107.stdout:12050: left oid 2 (ObjNum 12049 snap 0 seq_num 12049) 2020-06-12T02:39:22.602 INFO:tasks.rados.rados.0.smithi107.stdout:12047: finishing write tid 4 to smithi10729407-16 2020-06-12T02:39:22.602 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 2129 (ObjNum 12046 snap 0 seq_num 12046) dirty exists 2020-06-12T02:39:22.602 INFO:tasks.rados.rados.0.smithi107.stdout:12047: left oid 16 (ObjNum 12046 snap 0 seq_num 12046) 2020-06-12T02:39:22.603 INFO:tasks.rados.rados.0.smithi107.stdout:12051: finishing write tid 3 to smithi10729407-11 2020-06-12T02:39:22.604 INFO:tasks.rados.rados.0.smithi107.stdout:12051: finishing write tid 4 to smithi10729407-11 2020-06-12T02:39:22.604 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1846 (ObjNum 12050 snap 0 seq_num 12050) dirty exists 2020-06-12T02:39:22.605 INFO:tasks.rados.rados.0.smithi107.stdout:12051: left oid 11 (ObjNum 12050 snap 0 seq_num 12050) 2020-06-12T02:39:22.606 INFO:tasks.rados.rados.0.smithi107.stdout:12047: done (3 left) 2020-06-12T02:39:22.606 INFO:tasks.rados.rados.0.smithi107.stdout:12050: done (2 left) 2020-06-12T02:39:22.606 INFO:tasks.rados.rados.0.smithi107.stdout:12051: done (1 left) 2020-06-12T02:39:22.607 INFO:tasks.rados.rados.0.smithi107.stdout:12054: done (0 left) 2020-06-12T02:39:22.607 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:39:22.607 INFO:tasks.rados.rados.0.smithi107.stdout:12055: seq_num 12054 ranges {4612096=286720} 2020-06-12T02:39:22.631 INFO:tasks.rados.rados.0.smithi107.stdout:12055: writing smithi10729407-13 from 4612096 to 4898816 tid 1 2020-06-12T02:39:22.631 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:39:22.632 INFO:tasks.rados.rados.0.smithi107.stdout:12056: seq_num 12055 ranges {0=303104} 2020-06-12T02:39:22.633 INFO:tasks.rados.rados.0.smithi107.stdout:12056: writing smithi10729407-7 from 0 to 303104 tid 1 2020-06-12T02:39:22.633 INFO:tasks.rados.rados.0.smithi107.stdout:12057: delete oid 3 current snap is 0 2020-06-12T02:39:22.634 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:39:22.631+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:39:22.635 INFO:tasks.rados.rados.0.smithi107.stdout:12055: finishing write tid 1 to smithi10729407-13 2020-06-12T02:39:22.642 INFO:tasks.rados.rados.0.smithi107.stdout:12056: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:22.642 INFO:tasks.rados.rados.0.smithi107.stdout:12057: done (2 left) 2020-06-12T02:39:22.643 INFO:tasks.rados.rados.0.smithi107.stdout:12058: delete oid 9 current snap is 0 2020-06-12T02:39:22.643 INFO:tasks.rados.rados.0.smithi107.stdout:12055: finishing write tid 3 to smithi10729407-13 2020-06-12T02:39:22.643 INFO:tasks.rados.rados.0.smithi107.stdout:12055: finishing write tid 4 to smithi10729407-13 2020-06-12T02:39:22.643 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1883 (ObjNum 12054 snap 0 seq_num 12054) dirty exists 2020-06-12T02:39:22.644 INFO:tasks.rados.rados.0.smithi107.stdout:12055: left oid 13 (ObjNum 12054 snap 0 seq_num 12054) 2020-06-12T02:39:22.644 INFO:tasks.rados.rados.0.smithi107.stdout:12056: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:22.644 INFO:tasks.rados.rados.0.smithi107.stdout:12055: done (2 left) 2020-06-12T02:39:22.645 INFO:tasks.rados.rados.0.smithi107.stdout:12058: done (1 left) 2020-06-12T02:39:22.646 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:39:22.646 INFO:tasks.rados.rados.0.smithi107.stdout:12059: seq_num 12058 ranges {0=352256} 2020-06-12T02:39:22.647 INFO:tasks.rados.rados.0.smithi107.stdout:12059: writing smithi10729407-12 from 0 to 352256 tid 1 2020-06-12T02:39:22.647 INFO:tasks.rados.rados.0.smithi107.stdout:12060: delete oid 3 current snap is 0 2020-06-12T02:39:22.647 INFO:tasks.rados.rados.0.smithi107.stdout:12056: finishing write tid 4 to smithi10729407-7 2020-06-12T02:39:22.648 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1905 (ObjNum 12055 snap 0 seq_num 12055) dirty exists 2020-06-12T02:39:22.648 INFO:tasks.rados.rados.0.smithi107.stdout:12056: left oid 7 (ObjNum 12055 snap 0 seq_num 12055) 2020-06-12T02:39:22.649 INFO:tasks.rados.rados.0.smithi107.stdout:12059: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:22.650 INFO:tasks.rados.rados.0.smithi107.stdout:12056: done (2 left) 2020-06-12T02:39:22.650 INFO:tasks.rados.rados.0.smithi107.stdout:12060: done (1 left) 2020-06-12T02:39:22.650 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 1 current snap is 0 2020-06-12T02:39:22.650 INFO:tasks.rados.rados.0.smithi107.stdout:12061: seq_num 12060 ranges {0=385024} 2020-06-12T02:39:22.651 INFO:tasks.rados.rados.0.smithi107.stdout:12061: writing smithi10729407-1 from 0 to 385024 tid 1 2020-06-12T02:39:22.651 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 8 current snap is 0 2020-06-12T02:39:22.651 INFO:tasks.rados.rados.0.smithi107.stdout:12062: seq_num 12061 ranges {0=401408} 2020-06-12T02:39:22.653 INFO:tasks.rados.rados.0.smithi107.stdout:12062: writing smithi10729407-8 from 0 to 401408 tid 1 2020-06-12T02:39:22.654 INFO:tasks.rados.rados.0.smithi107.stdout:12063: delete oid 18 current snap is 0 2020-06-12T02:39:22.655 INFO:tasks.rados.rados.0.smithi107.stdout:12059: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:22.658 INFO:tasks.rados.rados.0.smithi107.stdout:12061: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:22.659 INFO:tasks.rados.rados.0.smithi107.stdout:12063: done (3 left) 2020-06-12T02:39:22.659 INFO:tasks.rados.rados.0.smithi107.stdout:12064: delete oid 11 current snap is 0 2020-06-12T02:39:22.659 INFO:tasks.rados.rados.0.smithi107.stdout:12061: finishing write tid 3 to smithi10729407-1 2020-06-12T02:39:22.660 INFO:tasks.rados.rados.0.smithi107.stdout:12059: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:22.660 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 2241 (ObjNum 12058 snap 0 seq_num 12058) dirty exists 2020-06-12T02:39:22.660 INFO:tasks.rados.rados.0.smithi107.stdout:12059: left oid 12 (ObjNum 12058 snap 0 seq_num 12058) 2020-06-12T02:39:22.660 INFO:tasks.rados.rados.0.smithi107.stdout:12061: finishing write tid 4 to smithi10729407-1 2020-06-12T02:39:22.661 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 2211 (ObjNum 12060 snap 0 seq_num 12060) dirty exists 2020-06-12T02:39:22.661 INFO:tasks.rados.rados.0.smithi107.stdout:12061: left oid 1 (ObjNum 12060 snap 0 seq_num 12060) 2020-06-12T02:39:22.662 INFO:tasks.rados.rados.0.smithi107.stdout:12059: done (3 left) 2020-06-12T02:39:22.662 INFO:tasks.rados.rados.0.smithi107.stdout:12061: done (2 left) 2020-06-12T02:39:22.662 INFO:tasks.rados.rados.0.smithi107.stdout:12064: done (1 left) 2020-06-12T02:39:22.663 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:39:22.663 INFO:tasks.rados.rados.0.smithi107.stdout:12065: seq_num 12064 ranges {2097152=450560} 2020-06-12T02:39:22.675 INFO:tasks.rados.rados.0.smithi107.stdout:12065: writing smithi10729407-16 from 2097152 to 2547712 tid 1 2020-06-12T02:39:22.676 INFO:tasks.rados.rados.0.smithi107.stdout:12066: delete oid 18 current snap is 0 2020-06-12T02:39:22.676 INFO:tasks.rados.rados.0.smithi107.stdout:12062: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:22.676 INFO:tasks.rados.rados.0.smithi107.stdout:12062: finishing write tid 3 to smithi10729407-8 2020-06-12T02:39:22.676 INFO:tasks.rados.rados.0.smithi107.stdout:12062: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:22.677 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 2266 (ObjNum 12061 snap 0 seq_num 12061) dirty exists 2020-06-12T02:39:22.677 INFO:tasks.rados.rados.0.smithi107.stdout:12062: left oid 8 (ObjNum 12061 snap 0 seq_num 12061) 2020-06-12T02:39:22.678 INFO:tasks.rados.rados.0.smithi107.stdout:12062: done (2 left) 2020-06-12T02:39:22.678 INFO:tasks.rados.rados.0.smithi107.stdout:12066: done (1 left) 2020-06-12T02:39:22.679 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:39:22.679 INFO:tasks.rados.rados.0.smithi107.stdout:12067: seq_num 12066 ranges {0=491520} 2020-06-12T02:39:22.682 INFO:tasks.rados.rados.0.smithi107.stdout:12067: writing smithi10729407-4 from 0 to 491520 tid 1 2020-06-12T02:39:22.682 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:39:22.682 INFO:tasks.rados.rados.0.smithi107.stdout:12068: seq_num 12067 ranges {0=507904} 2020-06-12T02:39:22.685 INFO:tasks.rados.rados.0.smithi107.stdout:12068: writing smithi10729407-5 from 0 to 507904 tid 1 2020-06-12T02:39:22.685 INFO:tasks.rados.rados.0.smithi107.stdout:12069: delete oid 17 current snap is 0 2020-06-12T02:39:22.685 INFO:tasks.rados.rados.0.smithi107.stdout:12065: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:22.685 INFO:tasks.rados.rados.0.smithi107.stdout:12065: finishing write tid 3 to smithi10729407-16 2020-06-12T02:39:22.686 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:22.686 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:22.687 INFO:tasks.rados.rados.0.smithi107.stdout:12065: finishing write tid 4 to smithi10729407-16 2020-06-12T02:39:22.687 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 2133 (ObjNum 12064 snap 0 seq_num 12064) dirty exists 2020-06-12T02:39:22.687 INFO:tasks.rados.rados.0.smithi107.stdout:12065: left oid 16 (ObjNum 12064 snap 0 seq_num 12064) 2020-06-12T02:39:22.689 INFO:tasks.rados.rados.0.smithi107.stdout:12067: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:22.689 INFO:tasks.rados.rados.0.smithi107.stdout:12067: finishing write tid 3 to smithi10729407-4 2020-06-12T02:39:22.690 INFO:tasks.rados.rados.0.smithi107.stdout:12068: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:22.691 INFO:tasks.rados.rados.0.smithi107.stdout:12065: done (3 left) 2020-06-12T02:39:22.691 INFO:tasks.rados.rados.0.smithi107.stdout:12069: done (2 left) 2020-06-12T02:39:22.691 INFO:tasks.rados.rados.0.smithi107.stdout:12070: delete oid 3 current snap is 0 2020-06-12T02:39:22.692 INFO:tasks.rados.rados.0.smithi107.stdout:12067: finishing write tid 4 to smithi10729407-4 2020-06-12T02:39:22.692 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 2246 (ObjNum 12066 snap 0 seq_num 12066) dirty exists 2020-06-12T02:39:22.692 INFO:tasks.rados.rados.0.smithi107.stdout:12067: left oid 4 (ObjNum 12066 snap 0 seq_num 12066) 2020-06-12T02:39:22.693 INFO:tasks.rados.rados.0.smithi107.stdout:12068: finishing write tid 3 to smithi10729407-5 2020-06-12T02:39:22.702 INFO:tasks.rados.rados.0.smithi107.stdout:12067: done (2 left) 2020-06-12T02:39:22.702 INFO:tasks.rados.rados.0.smithi107.stdout:12070: done (1 left) 2020-06-12T02:39:22.703 INFO:tasks.rados.rados.0.smithi107.stdout:12071: delete oid 12 current snap is 0 2020-06-12T02:39:22.703 INFO:tasks.rados.rados.0.smithi107.stdout:12068: finishing write tid 4 to smithi10729407-5 2020-06-12T02:39:22.703 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 1134 (ObjNum 12067 snap 0 seq_num 12067) dirty exists 2020-06-12T02:39:22.703 INFO:tasks.rados.rados.0.smithi107.stdout:12068: left oid 5 (ObjNum 12067 snap 0 seq_num 12067) 2020-06-12T02:39:22.703 INFO:tasks.rados.rados.0.smithi107.stdout:12068: done (1 left) 2020-06-12T02:39:22.703 INFO:tasks.rados.rados.0.smithi107.stdout:12071: done (0 left) 2020-06-12T02:39:22.704 INFO:tasks.rados.rados.0.smithi107.stdout:12072: delete oid 7 current snap is 0 2020-06-12T02:39:22.704 INFO:tasks.rados.rados.0.smithi107.stdout:12072: done (0 left) 2020-06-12T02:39:22.704 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:39:22.704 INFO:tasks.rados.rados.0.smithi107.stdout:12073: seq_num 12072 ranges {0=589824} 2020-06-12T02:39:22.705 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-06-12T02:39:22.706 INFO:tasks.rados.rados.0.smithi107.stdout:12073: writing smithi10729407-9 from 0 to 589824 tid 1 2020-06-12T02:39:22.706 INFO:tasks.rados.rados.0.smithi107.stdout:12074: delete oid 14 current snap is 0 2020-06-12T02:39:22.706 INFO:tasks.rados.rados.0.smithi107.stdout:12074: done (1 left) 2020-06-12T02:39:22.707 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:39:22.707 INFO:tasks.rados.rados.0.smithi107.stdout:12075: seq_num 12074 ranges {0=622592} 2020-06-12T02:39:22.709 INFO:tasks.rados.rados.0.smithi107.stdout:12075: writing smithi10729407-12 from 0 to 622592 tid 1 2020-06-12T02:39:22.710 INFO:tasks.rados.rados.0.smithi107.stdout:12076: delete oid 7 current snap is 0 2020-06-12T02:39:22.710 INFO:tasks.rados.rados.0.smithi107.stdout:12073: finishing write tid 1 to smithi10729407-9 2020-06-12T02:39:22.710 INFO:tasks.rados.rados.0.smithi107.stdout:12073: finishing write tid 3 to smithi10729407-9 2020-06-12T02:39:22.712 INFO:tasks.rados.rados.0.smithi107.stdout:12076: done (2 left) 2020-06-12T02:39:22.712 INFO:tasks.rados.rados.0.smithi107.stdout:12077: delete oid 20 current snap is 0 2020-06-12T02:39:22.712 INFO:tasks.rados.rados.0.smithi107.stdout:12073: finishing write tid 4 to smithi10729407-9 2020-06-12T02:39:22.713 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1944 (ObjNum 12072 snap 0 seq_num 12072) dirty exists 2020-06-12T02:39:22.713 INFO:tasks.rados.rados.0.smithi107.stdout:12073: left oid 9 (ObjNum 12072 snap 0 seq_num 12072) 2020-06-12T02:39:22.714 INFO:tasks.rados.rados.0.smithi107.stdout:12073: done (2 left) 2020-06-12T02:39:22.714 INFO:tasks.rados.rados.0.smithi107.stdout:12077: done (1 left) 2020-06-12T02:39:22.714 INFO:tasks.rados.rados.0.smithi107.stdout:12078: delete oid 17 current snap is 0 2020-06-12T02:39:22.716 INFO:tasks.rados.rados.0.smithi107.stdout:12075: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:22.717 INFO:tasks.rados.rados.0.smithi107.stdout:12075: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:22.718 INFO:tasks.rados.rados.0.smithi107.stdout:12078: done (1 left) 2020-06-12T02:39:22.718 INFO:tasks.rados.rados.0.smithi107.stdout:12079: delete oid 20 current snap is 0 2020-06-12T02:39:22.719 INFO:tasks.rados.rados.0.smithi107.stdout:12075: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:22.719 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 2244 (ObjNum 12074 snap 0 seq_num 12074) dirty exists 2020-06-12T02:39:22.720 INFO:tasks.rados.rados.0.smithi107.stdout:12075: left oid 12 (ObjNum 12074 snap 0 seq_num 12074) 2020-06-12T02:39:22.720 INFO:tasks.rados.rados.0.smithi107.stdout:12075: done (1 left) 2020-06-12T02:39:22.721 INFO:tasks.rados.rados.0.smithi107.stdout:12079: done (0 left) 2020-06-12T02:39:22.721 INFO:tasks.rados.rados.0.smithi107.stdout:12080: delete oid 8 current snap is 0 2020-06-12T02:39:22.722 INFO:tasks.rados.rados.0.smithi107.stdout:12080: done (0 left) 2020-06-12T02:39:22.722 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:39:22.723 INFO:tasks.rados.rados.0.smithi107.stdout:12081: seq_num 12080 ranges {0=720896} 2020-06-12T02:39:22.728 INFO:tasks.rados.rados.0.smithi107.stdout:12081: writing smithi10729407-7 from 0 to 720896 tid 1 2020-06-12T02:39:22.728 INFO:tasks.rados.rados.0.smithi107.stdout:12082: delete oid 2 current snap is 0 2020-06-12T02:39:22.729 INFO:tasks.rados.rados.0.smithi107.stdout:12082: done (1 left) 2020-06-12T02:39:22.729 INFO:tasks.rados.rados.0.smithi107.stdout:12083: delete oid 13 current snap is 0 2020-06-12T02:39:22.732 INFO:tasks.rados.rados.0.smithi107.stdout:12083: done (1 left) 2020-06-12T02:39:22.732 INFO:tasks.rados.rados.0.smithi107.stdout:12084: delete oid 10 current snap is 0 2020-06-12T02:39:22.733 INFO:tasks.rados.rados.0.smithi107.stdout:12081: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:22.734 INFO:tasks.rados.rados.0.smithi107.stdout:12081: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:22.734 INFO:tasks.rados.rados.0.smithi107.stdout:12084: done (1 left) 2020-06-12T02:39:22.735 INFO:tasks.rados.rados.0.smithi107.stdout:12085: delete oid 10 current snap is 0 2020-06-12T02:39:22.735 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:39:22.732+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:39:22.737 INFO:tasks.rados.rados.0.smithi107.stdout:12081: finishing write tid 4 to smithi10729407-7 2020-06-12T02:39:22.737 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1909 (ObjNum 12080 snap 0 seq_num 12080) dirty exists 2020-06-12T02:39:22.737 INFO:tasks.rados.rados.0.smithi107.stdout:12081: left oid 7 (ObjNum 12080 snap 0 seq_num 12080) 2020-06-12T02:39:22.737 INFO:tasks.rados.rados.0.smithi107.stdout:12081: done (1 left) 2020-06-12T02:39:22.738 INFO:tasks.rados.rados.0.smithi107.stdout:12085: done (0 left) 2020-06-12T02:39:22.738 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:39:22.738 INFO:tasks.rados.rados.0.smithi107.stdout:12086: seq_num 12085 ranges {0=409600,409600=401408} 2020-06-12T02:39:22.739 INFO:tasks.rados.rados.0.smithi107.stdout:12086: writing smithi10729407-2 from 0 to 409600 tid 1 2020-06-12T02:39:22.741 INFO:tasks.rados.rados.0.smithi107.stdout:12086: writing smithi10729407-2 from 409600 to 811008 tid 2 2020-06-12T02:39:22.742 INFO:tasks.rados.rados.0.smithi107.stdout:12087: delete oid 19 current snap is 0 2020-06-12T02:39:22.744 INFO:tasks.rados.rados.0.smithi107.stdout:12087: done (1 left) 2020-06-12T02:39:22.744 INFO:tasks.rados.rados.0.smithi107.stdout:12088: delete oid 6 current snap is 0 2020-06-12T02:39:22.748 INFO:tasks.rados.rados.0.smithi107.stdout:12088: done (1 left) 2020-06-12T02:39:22.748 INFO:tasks.rados.rados.0.smithi107.stdout:12089: delete oid 3 current snap is 0 2020-06-12T02:39:22.749 INFO:tasks.rados.rados.0.smithi107.stdout:12086: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:22.749 INFO:tasks.rados.rados.0.smithi107.stdout:12086: finishing write tid 2 to smithi10729407-2 2020-06-12T02:39:22.752 INFO:tasks.rados.rados.0.smithi107.stdout:12086: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:22.752 INFO:tasks.rados.rados.0.smithi107.stdout:12089: done (1 left) 2020-06-12T02:39:22.752 INFO:tasks.rados.rados.0.smithi107.stdout:12090: delete oid 7 current snap is 0 2020-06-12T02:39:22.753 INFO:tasks.rados.rados.0.smithi107.stdout:12086: finishing write tid 5 to smithi10729407-2 2020-06-12T02:39:22.753 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 1138 (ObjNum 12085 snap 0 seq_num 12085) dirty exists 2020-06-12T02:39:22.753 INFO:tasks.rados.rados.0.smithi107.stdout:12086: left oid 2 (ObjNum 12085 snap 0 seq_num 12085) 2020-06-12T02:39:22.754 INFO:tasks.rados.rados.0.smithi107.stdout:12086: done (1 left) 2020-06-12T02:39:22.754 INFO:tasks.rados.rados.0.smithi107.stdout:12090: done (0 left) 2020-06-12T02:39:22.754 INFO:tasks.rados.rados.0.smithi107.stdout:12091: delete oid 3 current snap is 0 2020-06-12T02:39:22.757 INFO:tasks.rados.rados.0.smithi107.stdout:12091: done (0 left) 2020-06-12T02:39:22.757 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:39:22.757 INFO:tasks.rados.rados.0.smithi107.stdout:12092: seq_num 12091 ranges {0=507904,507904=401408} 2020-06-12T02:39:22.760 INFO:tasks.rados.rados.0.smithi107.stdout:12092: writing smithi10729407-10 from 0 to 507904 tid 1 2020-06-12T02:39:22.762 INFO:tasks.rados.rados.0.smithi107.stdout:12092: writing smithi10729407-10 from 507904 to 909312 tid 2 2020-06-12T02:39:22.762 INFO:tasks.rados.rados.0.smithi107.stdout:12093: delete oid 19 current snap is 0 2020-06-12T02:39:22.766 INFO:tasks.rados.rados.0.smithi107.stdout:12093: done (1 left) 2020-06-12T02:39:22.766 INFO:tasks.rados.rados.0.smithi107.stdout:12094: delete oid 6 current snap is 0 2020-06-12T02:39:22.768 INFO:tasks.rados.rados.0.smithi107.stdout:12092: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:22.768 INFO:tasks.rados.rados.0.smithi107.stdout:12092: finishing write tid 2 to smithi10729407-10 2020-06-12T02:39:22.770 INFO:tasks.rados.rados.0.smithi107.stdout:12092: finishing write tid 4 to smithi10729407-10 2020-06-12T02:39:22.770 INFO:tasks.rados.rados.0.smithi107.stdout:12094: done (1 left) 2020-06-12T02:39:22.771 INFO:tasks.rados.rados.0.smithi107.stdout:12095: delete oid 18 current snap is 0 2020-06-12T02:39:22.772 INFO:tasks.rados.rados.0.smithi107.stdout:12092: finishing write tid 5 to smithi10729407-10 2020-06-12T02:39:22.772 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 921 (ObjNum 12091 snap 0 seq_num 12091) dirty exists 2020-06-12T02:39:22.772 INFO:tasks.rados.rados.0.smithi107.stdout:12092: left oid 10 (ObjNum 12091 snap 0 seq_num 12091) 2020-06-12T02:39:22.775 INFO:tasks.rados.rados.0.smithi107.stdout:12092: done (1 left) 2020-06-12T02:39:22.776 INFO:tasks.rados.rados.0.smithi107.stdout:12095: done (0 left) 2020-06-12T02:39:22.776 INFO:tasks.rados.rados.0.smithi107.stdout:12096: delete oid 13 current snap is 0 2020-06-12T02:39:22.778 INFO:tasks.rados.rados.0.smithi107.stdout:12096: done (0 left) 2020-06-12T02:39:22.778 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:39:22.779 INFO:tasks.rados.rados.0.smithi107.stdout:12097: seq_num 12096 ranges {0=589824,589824=401408} 2020-06-12T02:39:22.781 INFO:tasks.rados.rados.0.smithi107.stdout:12097: writing smithi10729407-17 from 0 to 589824 tid 1 2020-06-12T02:39:22.784 INFO:tasks.rados.rados.0.smithi107.stdout:12097: writing smithi10729407-17 from 589824 to 991232 tid 2 2020-06-12T02:39:22.784 INFO:tasks.rados.rados.0.smithi107.stdout:12098: delete oid 13 current snap is 0 2020-06-12T02:39:22.789 INFO:tasks.rados.rados.0.smithi107.stdout:12098: done (1 left) 2020-06-12T02:39:22.789 INFO:tasks.rados.rados.0.smithi107.stdout:12099: delete oid 10 current snap is 0 2020-06-12T02:39:22.790 INFO:tasks.rados.rados.0.smithi107.stdout:12097: finishing write tid 1 to smithi10729407-17 2020-06-12T02:39:22.791 INFO:tasks.rados.rados.0.smithi107.stdout:12099: done (1 left) 2020-06-12T02:39:22.791 INFO:tasks.rados.rados.0.smithi107.stdout:12100: delete oid 19 current snap is 0 2020-06-12T02:39:22.801 INFO:tasks.rados.rados.0.smithi107.stdout:12100: done (1 left) 2020-06-12T02:39:22.801 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 8 current snap is 0 2020-06-12T02:39:22.801 INFO:tasks.rados.rados.0.smithi107.stdout:12101: seq_num 12100 ranges {0=655360,655360=401408} 2020-06-12T02:39:22.801 INFO:tasks.rados.rados.0.smithi107.stdout:12101: writing smithi10729407-8 from 0 to 655360 tid 1 2020-06-12T02:39:22.802 INFO:tasks.rados.rados.0.smithi107.stdout:12101: writing smithi10729407-8 from 655360 to 1056768 tid 2 2020-06-12T02:39:22.802 INFO:tasks.rados.rados.0.smithi107.stdout:12102: delete oid 16 current snap is 0 2020-06-12T02:39:22.802 INFO:tasks.rados.rados.0.smithi107.stdout:12097: finishing write tid 2 to smithi10729407-17 2020-06-12T02:39:22.802 INFO:tasks.rados.rados.0.smithi107.stdout:12097: finishing write tid 4 to smithi10729407-17 2020-06-12T02:39:22.803 INFO:tasks.rados.rados.0.smithi107.stdout:12097: finishing write tid 5 to smithi10729407-17 2020-06-12T02:39:22.803 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 2251 (ObjNum 12096 snap 0 seq_num 12096) dirty exists 2020-06-12T02:39:22.803 INFO:tasks.rados.rados.0.smithi107.stdout:12097: left oid 17 (ObjNum 12096 snap 0 seq_num 12096) 2020-06-12T02:39:22.805 INFO:tasks.rados.rados.0.smithi107.stdout:12101: finishing write tid 1 to smithi10729407-8 2020-06-12T02:39:22.807 INFO:tasks.rados.rados.0.smithi107.stdout:12097: done (2 left) 2020-06-12T02:39:22.807 INFO:tasks.rados.rados.0.smithi107.stdout:12102: done (1 left) 2020-06-12T02:39:22.808 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:39:22.808 INFO:tasks.rados.rados.0.smithi107.stdout:12103: seq_num 12102 ranges {0=688128,688128=401408} 2020-06-12T02:39:22.812 INFO:tasks.rados.rados.0.smithi107.stdout:12103: writing smithi10729407-7 from 0 to 688128 tid 1 2020-06-12T02:39:22.815 INFO:tasks.rados.rados.0.smithi107.stdout:12103: writing smithi10729407-7 from 688128 to 1089536 tid 2 2020-06-12T02:39:22.815 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:39:22.815 INFO:tasks.rados.rados.0.smithi107.stdout:12104: seq_num 12103 ranges {507904=704512,1212416=401408} 2020-06-12T02:39:22.823 INFO:tasks.rados.rados.0.smithi107.stdout:12104: writing smithi10729407-5 from 507904 to 1212416 tid 1 2020-06-12T02:39:22.826 INFO:tasks.rados.rados.0.smithi107.stdout:12104: writing smithi10729407-5 from 1212416 to 1613824 tid 2 2020-06-12T02:39:22.826 INFO:tasks.rados.rados.0.smithi107.stdout:12105: delete oid 6 current snap is 0 2020-06-12T02:39:22.826 INFO:tasks.rados.rados.0.smithi107.stdout:12101: finishing write tid 2 to smithi10729407-8 2020-06-12T02:39:22.827 INFO:tasks.rados.rados.0.smithi107.stdout:12101: finishing write tid 4 to smithi10729407-8 2020-06-12T02:39:22.827 INFO:tasks.rados.rados.0.smithi107.stdout:12101: finishing write tid 5 to smithi10729407-8 2020-06-12T02:39:22.827 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 8 v 2270 (ObjNum 12100 snap 0 seq_num 12100) dirty exists 2020-06-12T02:39:22.827 INFO:tasks.rados.rados.0.smithi107.stdout:12101: left oid 8 (ObjNum 12100 snap 0 seq_num 12100) 2020-06-12T02:39:22.827 INFO:tasks.rados.rados.0.smithi107.stdout:12103: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:22.827 INFO:tasks.rados.rados.0.smithi107.stdout:12103: finishing write tid 2 to smithi10729407-7 2020-06-12T02:39:22.828 INFO:tasks.rados.rados.0.smithi107.stdout:12103: finishing write tid 4 to smithi10729407-7 2020-06-12T02:39:22.828 INFO:tasks.rados.rados.0.smithi107.stdout:12103: finishing write tid 5 to smithi10729407-7 2020-06-12T02:39:22.828 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1913 (ObjNum 12102 snap 0 seq_num 12102) dirty exists 2020-06-12T02:39:22.828 INFO:tasks.rados.rados.0.smithi107.stdout:12103: left oid 7 (ObjNum 12102 snap 0 seq_num 12102) 2020-06-12T02:39:22.829 INFO:tasks.rados.rados.0.smithi107.stdout:12104: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:22.829 INFO:tasks.rados.rados.0.smithi107.stdout:12101: done (3 left) 2020-06-12T02:39:22.829 INFO:tasks.rados.rados.0.smithi107.stdout:12103: done (2 left) 2020-06-12T02:39:22.830 INFO:tasks.rados.rados.0.smithi107.stdout:12105: done (1 left) 2020-06-12T02:39:22.830 INFO:tasks.rados.rados.0.smithi107.stdout:12106: delete oid 15 current snap is 0 2020-06-12T02:39:22.834 INFO:tasks.rados.rados.0.smithi107.stdout:12106: done (1 left) 2020-06-12T02:39:22.834 INFO:tasks.rados.rados.0.smithi107.stdout:12107: delete oid 16 current snap is 0 2020-06-12T02:39:22.835 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:39:22.831+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:39:22.836 INFO:tasks.rados.rados.0.smithi107.stdout:12104: finishing write tid 2 to smithi10729407-5 2020-06-12T02:39:22.836 INFO:tasks.rados.rados.0.smithi107.stdout:12104: finishing write tid 4 to smithi10729407-5 2020-06-12T02:39:22.837 INFO:tasks.rados.rados.0.smithi107.stdout:12107: done (1 left) 2020-06-12T02:39:22.837 INFO:tasks.rados.rados.0.smithi107.stdout:12108: delete oid 9 current snap is 0 2020-06-12T02:39:22.838 INFO:tasks.rados.rados.0.smithi107.stdout:12104: finishing write tid 5 to smithi10729407-5 2020-06-12T02:39:22.838 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 1137 (ObjNum 12103 snap 0 seq_num 12103) dirty exists 2020-06-12T02:39:22.838 INFO:tasks.rados.rados.0.smithi107.stdout:12104: left oid 5 (ObjNum 12103 snap 0 seq_num 12103) 2020-06-12T02:39:22.839 INFO:tasks.rados.rados.0.smithi107.stdout:12104: done (1 left) 2020-06-12T02:39:22.839 INFO:tasks.rados.rados.0.smithi107.stdout:12108: done (0 left) 2020-06-12T02:39:22.839 INFO:tasks.rados.rados.0.smithi107.stdout:12109: delete oid 9 current snap is 0 2020-06-12T02:39:22.841 INFO:tasks.rados.rados.0.smithi107.stdout:12109: done (0 left) 2020-06-12T02:39:22.842 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:39:22.842 INFO:tasks.rados.rados.0.smithi107.stdout:12110: seq_num 12109 ranges {0=409600,409600=466944,876544=335872} 2020-06-12T02:39:22.844 INFO:tasks.rados.rados.0.smithi107.stdout:12110: writing smithi10729407-20 from 0 to 409600 tid 1 2020-06-12T02:39:22.846 INFO:tasks.rados.rados.0.smithi107.stdout:12110: writing smithi10729407-20 from 409600 to 876544 tid 2 2020-06-12T02:39:22.848 INFO:tasks.rados.rados.0.smithi107.stdout:12110: writing smithi10729407-20 from 876544 to 1212416 tid 3 2020-06-12T02:39:22.849 INFO:tasks.rados.rados.0.smithi107.stdout:12111: delete oid 4 current snap is 0 2020-06-12T02:39:22.852 INFO:tasks.rados.rados.0.smithi107.stdout:12111: done (1 left) 2020-06-12T02:39:22.852 INFO:tasks.rados.rados.0.smithi107.stdout:12112: delete oid 6 current snap is 0 2020-06-12T02:39:22.855 INFO:tasks.rados.rados.0.smithi107.stdout:12112: done (1 left) 2020-06-12T02:39:22.855 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:39:22.855 INFO:tasks.rados.rados.0.smithi107.stdout:12113: seq_num 12112 ranges {0=458752,458752=573440,1032192=229376} 2020-06-12T02:39:22.856 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:22.856 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:22.857 INFO:tasks.rados.rados.0.smithi107.stdout:12113: writing smithi10729407-18 from 0 to 458752 tid 1 2020-06-12T02:39:22.861 INFO:tasks.rados.rados.0.smithi107.stdout:12113: writing smithi10729407-18 from 458752 to 1032192 tid 2 2020-06-12T02:39:22.863 INFO:tasks.rados.rados.0.smithi107.stdout:12113: writing smithi10729407-18 from 1032192 to 1261568 tid 3 2020-06-12T02:39:22.863 INFO:tasks.rados.rados.0.smithi107.stdout:12114: delete oid 2 current snap is 0 2020-06-12T02:39:22.863 INFO:tasks.rados.rados.0.smithi107.stdout:12110: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:22.863 INFO:tasks.rados.rados.0.smithi107.stdout:12110: finishing write tid 2 to smithi10729407-20 2020-06-12T02:39:22.863 INFO:tasks.rados.rados.0.smithi107.stdout:12110: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:22.864 INFO:tasks.rados.rados.0.smithi107.stdout:12110: finishing write tid 5 to smithi10729407-20 2020-06-12T02:39:22.864 INFO:tasks.rados.rados.0.smithi107.stdout:12110: finishing write tid 6 to smithi10729407-20 2020-06-12T02:39:22.864 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 940 (ObjNum 12109 snap 0 seq_num 12109) dirty exists 2020-06-12T02:39:22.864 INFO:tasks.rados.rados.0.smithi107.stdout:12110: left oid 20 (ObjNum 12109 snap 0 seq_num 12109) 2020-06-12T02:39:22.867 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-06-12T02:39:22.868 INFO:tasks.rados.rados.0.smithi107.stdout:12110: done (2 left) 2020-06-12T02:39:22.868 INFO:tasks.rados.rados.0.smithi107.stdout:12114: done (1 left) 2020-06-12T02:39:22.868 INFO:tasks.rados.rados.0.smithi107.stdout:12115: delete oid 7 current snap is 0 2020-06-12T02:39:22.869 INFO:tasks.rados.rados.0.smithi107.stdout:12113: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:22.869 INFO:tasks.rados.rados.0.smithi107.stdout:12115: done (1 left) 2020-06-12T02:39:22.869 INFO:tasks.rados.rados.0.smithi107.stdout:12116: delete oid 19 current snap is 0 2020-06-12T02:39:22.870 INFO:tasks.rados.rados.0.smithi107.stdout:12113: finishing write tid 2 to smithi10729407-18 2020-06-12T02:39:22.871 INFO:tasks.rados.rados.0.smithi107.stdout:12113: finishing write tid 3 to smithi10729407-18 2020-06-12T02:39:22.871 INFO:tasks.rados.rados.0.smithi107.stdout:12113: finishing write tid 5 to smithi10729407-18 2020-06-12T02:39:22.873 INFO:tasks.rados.rados.0.smithi107.stdout:12113: finishing write tid 6 to smithi10729407-18 2020-06-12T02:39:22.873 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1972 (ObjNum 12112 snap 0 seq_num 12112) dirty exists 2020-06-12T02:39:22.873 INFO:tasks.rados.rados.0.smithi107.stdout:12113: left oid 18 (ObjNum 12112 snap 0 seq_num 12112) 2020-06-12T02:39:22.873 INFO:tasks.rados.rados.0.smithi107.stdout:12113: done (1 left) 2020-06-12T02:39:22.873 INFO:tasks.rados.rados.0.smithi107.stdout:12116: done (0 left) 2020-06-12T02:39:22.874 INFO:tasks.rados.rados.0.smithi107.stdout:12117: delete oid 12 current snap is 0 2020-06-12T02:39:22.877 INFO:tasks.rados.rados.0.smithi107.stdout:12117: done (0 left) 2020-06-12T02:39:22.877 INFO:tasks.rados.rados.0.smithi107.stdout:12118: delete oid 13 current snap is 0 2020-06-12T02:39:22.878 INFO:teuthology.orchestra.run.smithi188.stderr:Nothing to do for non-filestore backend 2020-06-12T02:39:22.878 INFO:tasks.rados.rados.0.smithi107.stdout:12118: done (0 left) 2020-06-12T02:39:22.878 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:39:22.879 INFO:tasks.rados.rados.0.smithi107.stdout:12119: seq_num 12118 ranges {0=557056,557056=679936,1236992=122880} 2020-06-12T02:39:22.881 INFO:tasks.rados.rados.0.smithi107.stdout:12119: writing smithi10729407-12 from 0 to 557056 tid 1 2020-06-12T02:39:22.885 INFO:tasks.rados.rados.0.smithi107.stdout:12119: writing smithi10729407-12 from 557056 to 1236992 tid 2 2020-06-12T02:39:22.886 INFO:tasks.rados.rados.0.smithi107.stdout:12119: writing smithi10729407-12 from 1236992 to 1359872 tid 3 2020-06-12T02:39:22.887 INFO:tasks.rados.rados.0.smithi107.stdout:12120: delete oid 19 current snap is 0 2020-06-12T02:39:22.889 INFO:tasks.rados.rados.0.smithi107.stdout:12119: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:22.894 INFO:tasks.rados.rados.0.smithi107.stdout:12119: finishing write tid 2 to smithi10729407-12 2020-06-12T02:39:22.894 INFO:tasks.rados.rados.0.smithi107.stdout:12119: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:22.896 INFO:tasks.rados.rados.0.smithi107.stdout:12119: finishing write tid 5 to smithi10729407-12 2020-06-12T02:39:22.899 INFO:tasks.rados.rados.0.smithi107.stdout:12119: finishing write tid 6 to smithi10729407-12 2020-06-12T02:39:22.900 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 2253 (ObjNum 12118 snap 0 seq_num 12118) dirty exists 2020-06-12T02:39:22.900 INFO:tasks.rados.rados.0.smithi107.stdout:12119: left oid 12 (ObjNum 12118 snap 0 seq_num 12118) 2020-06-12T02:39:22.900 INFO:tasks.rados.rados.0.smithi107.stdout:12119: done (1 left) 2020-06-12T02:39:22.900 INFO:tasks.rados.rados.0.smithi107.stdout:12120: done (0 left) 2020-06-12T02:39:22.900 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:39:22.901 INFO:tasks.rados.rados.0.smithi107.stdout:12121: seq_num 12120 ranges {1261568=589824,1851392=499712,2351104=303104} 2020-06-12T02:39:22.911 INFO:tasks.rados.rados.0.smithi107.stdout:12121: writing smithi10729407-18 from 1261568 to 1851392 tid 1 2020-06-12T02:39:22.913 INFO:tasks.rados.rados.0.smithi107.stdout:12121: writing smithi10729407-18 from 1851392 to 2351104 tid 2 2020-06-12T02:39:22.914 INFO:tasks.rados.rados.0.smithi107.stdout:12121: writing smithi10729407-18 from 2351104 to 2654208 tid 3 2020-06-12T02:39:22.915 INFO:tasks.rados.rados.0.smithi107.stdout:12122: delete oid 13 current snap is 0 2020-06-12T02:39:22.916 INFO:tasks.rados.rados.0.smithi107.stdout:12121: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:22.917 INFO:tasks.rados.rados.0.smithi107.stdout:12122: done (1 left) 2020-06-12T02:39:22.917 INFO:tasks.rados.rados.0.smithi107.stdout:12123: delete oid 19 current snap is 0 2020-06-12T02:39:22.920 INFO:tasks.rados.rados.0.smithi107.stdout:12123: done (1 left) 2020-06-12T02:39:22.920 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:39:22.921 INFO:tasks.rados.rados.0.smithi107.stdout:12124: seq_num 12123 ranges {1212416=638976,1851392=548864,2400256=253952} 2020-06-12T02:39:22.931 INFO:tasks.rados.rados.0.smithi107.stdout:12124: writing smithi10729407-20 from 1212416 to 1851392 tid 1 2020-06-12T02:39:22.935 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:39:22.932+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:39:22.936 INFO:tasks.rados.rados.0.smithi107.stdout:12124: writing smithi10729407-20 from 1851392 to 2400256 tid 2 2020-06-12T02:39:22.937 INFO:tasks.rados.rados.0.smithi107.stdout:12124: writing smithi10729407-20 from 2400256 to 2654208 tid 3 2020-06-12T02:39:22.938 INFO:tasks.rados.rados.0.smithi107.stdout:12125: delete oid 2 current snap is 0 2020-06-12T02:39:22.938 INFO:tasks.rados.rados.0.smithi107.stdout:12121: finishing write tid 2 to smithi10729407-18 2020-06-12T02:39:22.938 INFO:tasks.rados.rados.0.smithi107.stdout:12121: finishing write tid 3 to smithi10729407-18 2020-06-12T02:39:22.939 INFO:tasks.rados.rados.0.smithi107.stdout:12121: finishing write tid 5 to smithi10729407-18 2020-06-12T02:39:22.939 INFO:tasks.rados.rados.0.smithi107.stdout:12121: finishing write tid 6 to smithi10729407-18 2020-06-12T02:39:22.939 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1976 (ObjNum 12120 snap 0 seq_num 12120) dirty exists 2020-06-12T02:39:22.940 INFO:tasks.rados.rados.0.smithi107.stdout:12121: left oid 18 (ObjNum 12120 snap 0 seq_num 12120) 2020-06-12T02:39:22.940 INFO:tasks.rados.rados.0.smithi107.stdout:12124: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:22.941 INFO:tasks.rados.rados.0.smithi107.stdout:12121: done (2 left) 2020-06-12T02:39:22.941 INFO:tasks.rados.rados.0.smithi107.stdout:12125: done (1 left) 2020-06-12T02:39:22.941 INFO:tasks.rados.rados.0.smithi107.stdout:12126: delete oid 11 current snap is 0 2020-06-12T02:39:22.945 INFO:tasks.rados.rados.0.smithi107.stdout:12126: done (1 left) 2020-06-12T02:39:22.945 INFO:tasks.rados.rados.0.smithi107.stdout:12127: delete oid 2 current snap is 0 2020-06-12T02:39:22.945 INFO:tasks.rados.rados.0.smithi107.stdout:12124: finishing write tid 2 to smithi10729407-20 2020-06-12T02:39:22.947 INFO:tasks.rados.rados.0.smithi107.stdout:12124: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:22.958 INFO:tasks.rados.rados.0.smithi107.stdout:12127: done (1 left) 2020-06-12T02:39:22.959 INFO:tasks.rados.rados.0.smithi107.stdout:12128: delete oid 19 current snap is 0 2020-06-12T02:39:22.959 INFO:tasks.rados.rados.0.smithi107.stdout:12124: finishing write tid 5 to smithi10729407-20 2020-06-12T02:39:22.959 INFO:tasks.rados.rados.0.smithi107.stdout:12124: finishing write tid 6 to smithi10729407-20 2020-06-12T02:39:22.960 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 944 (ObjNum 12123 snap 0 seq_num 12123) dirty exists 2020-06-12T02:39:22.960 INFO:tasks.rados.rados.0.smithi107.stdout:12124: left oid 20 (ObjNum 12123 snap 0 seq_num 12123) 2020-06-12T02:39:22.960 INFO:tasks.rados.rados.0.smithi107.stdout:12124: done (1 left) 2020-06-12T02:39:22.961 INFO:tasks.rados.rados.0.smithi107.stdout:12128: done (0 left) 2020-06-12T02:39:22.961 INFO:tasks.rados.rados.0.smithi107.stdout:12129: delete oid 13 current snap is 0 2020-06-12T02:39:22.972 INFO:tasks.rados.rados.0.smithi107.stdout:12129: done (0 left) 2020-06-12T02:39:22.972 INFO:tasks.rados.rados.0.smithi107.stdout:12130: delete oid 12 current snap is 0 2020-06-12T02:39:22.972 INFO:tasks.rados.rados.0.smithi107.stdout:12130: done (0 left) 2020-06-12T02:39:22.973 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:39:22.973 INFO:tasks.rados.rados.0.smithi107.stdout:12131: seq_num 12130 ranges {1613824=761856,2375680=589824,2965504=212992} 2020-06-12T02:39:22.974 INFO:tasks.rados.rados.0.smithi107.stdout:12131: writing smithi10729407-5 from 1613824 to 2375680 tid 1 2020-06-12T02:39:22.975 INFO:tasks.rados.rados.0.smithi107.stdout:12131: writing smithi10729407-5 from 2375680 to 2965504 tid 2 2020-06-12T02:39:22.975 INFO:tasks.rados.rados.0.smithi107.stdout:12131: writing smithi10729407-5 from 2965504 to 3178496 tid 3 2020-06-12T02:39:22.975 INFO:tasks.rados.rados.0.smithi107.stdout:12132: delete oid 1 current snap is 0 2020-06-12T02:39:22.977 INFO:tasks.rados.rados.0.smithi107.stdout:12132: done (1 left) 2020-06-12T02:39:22.981 INFO:tasks.rados.rados.0.smithi107.stdout:12133: delete oid 16 current snap is 0 2020-06-12T02:39:22.984 INFO:tasks.rados.rados.0.smithi107.stdout:12131: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:22.984 INFO:tasks.rados.rados.0.smithi107.stdout:12133: done (1 left) 2020-06-12T02:39:22.984 INFO:tasks.rados.rados.0.smithi107.stdout:12134: delete oid 3 current snap is 0 2020-06-12T02:39:22.985 INFO:tasks.rados.rados.0.smithi107.stdout:12131: finishing write tid 2 to smithi10729407-5 2020-06-12T02:39:22.985 INFO:tasks.rados.rados.0.smithi107.stdout:12131: finishing write tid 3 to smithi10729407-5 2020-06-12T02:39:22.985 INFO:tasks.rados.rados.0.smithi107.stdout:12131: finishing write tid 5 to smithi10729407-5 2020-06-12T02:39:22.986 INFO:tasks.rados.rados.0.smithi107.stdout:12131: finishing write tid 6 to smithi10729407-5 2020-06-12T02:39:22.986 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 1141 (ObjNum 12130 snap 0 seq_num 12130) dirty exists 2020-06-12T02:39:22.986 INFO:tasks.rados.rados.0.smithi107.stdout:12131: left oid 5 (ObjNum 12130 snap 0 seq_num 12130) 2020-06-12T02:39:22.987 INFO:tasks.rados.rados.0.smithi107.stdout:12131: done (1 left) 2020-06-12T02:39:22.988 INFO:tasks.rados.rados.0.smithi107.stdout:12134: done (0 left) 2020-06-12T02:39:22.988 INFO:tasks.rados.rados.0.smithi107.stdout:12135: delete oid 1 current snap is 0 2020-06-12T02:39:22.988 INFO:tasks.rados.rados.0.smithi107.stdout:12135: done (0 left) 2020-06-12T02:39:22.989 INFO:tasks.rados.rados.0.smithi107.stdout:12136: delete oid 7 current snap is 0 2020-06-12T02:39:22.989 INFO:tasks.rados.rados.0.smithi107.stdout:12136: done (0 left) 2020-06-12T02:39:22.990 INFO:tasks.rados.rados.0.smithi107.stdout:12137: delete oid 3 current snap is 0 2020-06-12T02:39:22.991 INFO:tasks.rados.rados.0.smithi107.stdout:12137: done (0 left) 2020-06-12T02:39:22.992 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 5 current snap is 0 2020-06-12T02:39:22.992 INFO:tasks.rados.rados.0.smithi107.stdout:12138: seq_num 12137 ranges {3178496=475136,3653632=630784,4284416=573440} 2020-06-12T02:39:23.012 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:23.013 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:23.013 INFO:tasks.rados.rados.0.smithi107.stdout:12138: writing smithi10729407-5 from 3178496 to 3653632 tid 1 2020-06-12T02:39:23.015 INFO:tasks.rados.rados.0.smithi107.stdout:12138: writing smithi10729407-5 from 3653632 to 4284416 tid 2 2020-06-12T02:39:23.020 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-06-12T02:39:23.021 INFO:tasks.rados.rados.0.smithi107.stdout:12138: writing smithi10729407-5 from 4284416 to 4857856 tid 3 2020-06-12T02:39:23.021 INFO:tasks.rados.rados.0.smithi107.stdout:12139: delete oid 16 current snap is 0 2020-06-12T02:39:23.022 INFO:tasks.rados.rados.0.smithi107.stdout:12139: done (1 left) 2020-06-12T02:39:23.022 INFO:tasks.rados.rados.0.smithi107.stdout:12140: delete oid 7 current snap is 0 2020-06-12T02:39:23.024 INFO:tasks.rados.rados.0.smithi107.stdout:12138: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:23.024 INFO:tasks.rados.rados.0.smithi107.stdout:12138: finishing write tid 2 to smithi10729407-5 2020-06-12T02:39:23.025 INFO:tasks.rados.rados.0.smithi107.stdout:12138: finishing write tid 3 to smithi10729407-5 2020-06-12T02:39:23.029 INFO:tasks.rados.rados.0.smithi107.stdout:12140: done (1 left) 2020-06-12T02:39:23.029 INFO:tasks.rados.rados.0.smithi107.stdout:12141: delete oid 15 current snap is 0 2020-06-12T02:39:23.030 INFO:tasks.rados.rados.0.smithi107.stdout:12138: finishing write tid 5 to smithi10729407-5 2020-06-12T02:39:23.030 INFO:tasks.rados.rados.0.smithi107.stdout:12138: finishing write tid 6 to smithi10729407-5 2020-06-12T02:39:23.030 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 1145 (ObjNum 12137 snap 0 seq_num 12137) dirty exists 2020-06-12T02:39:23.030 INFO:tasks.rados.rados.0.smithi107.stdout:12138: left oid 5 (ObjNum 12137 snap 0 seq_num 12137) 2020-06-12T02:39:23.030 INFO:tasks.rados.rados.0.smithi107.stdout:12138: done (1 left) 2020-06-12T02:39:23.031 INFO:tasks.rados.rados.0.smithi107.stdout:12141: done (0 left) 2020-06-12T02:39:23.031 INFO:tasks.rados.rados.0.smithi107.stdout:12142: delete oid 19 current snap is 0 2020-06-12T02:39:23.031 INFO:tasks.rados.rados.0.smithi107.stdout:12142: done (0 left) 2020-06-12T02:39:23.032 INFO:tasks.rados.rados.0.smithi107.stdout:12143: delete oid 3 current snap is 0 2020-06-12T02:39:23.033 INFO:tasks.rados.rados.0.smithi107.stdout:12143: done (0 left) 2020-06-12T02:39:23.033 INFO:tasks.rados.rados.0.smithi107.stdout:12144: delete oid 7 current snap is 0 2020-06-12T02:39:23.035 INFO:tasks.rados.rados.0.smithi107.stdout:12144: done (0 left) 2020-06-12T02:39:23.035 INFO:tasks.rados.rados.0.smithi107.stdout:12145: delete oid 19 current snap is 0 2020-06-12T02:39:23.036 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:39:23.033+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:39:23.038 INFO:tasks.rados.rados.0.smithi107.stdout:12145: done (0 left) 2020-06-12T02:39:23.038 INFO:tasks.rados.rados.0.smithi107.stdout:12146: delete oid 11 current snap is 0 2020-06-12T02:39:23.040 INFO:tasks.rados.rados.0.smithi107.stdout:12146: done (0 left) 2020-06-12T02:39:23.040 INFO:tasks.rados.rados.0.smithi107.stdout:12147: delete oid 8 current snap is 0 2020-06-12T02:39:23.043 INFO:tasks.rados.rados.0.smithi107.stdout:12147: done (0 left) 2020-06-12T02:39:23.043 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:39:23.043 INFO:tasks.rados.rados.0.smithi107.stdout:12148: seq_num 12147 ranges {0=647168,647168=720896,1368064=483328} 2020-06-12T02:39:23.047 INFO:tasks.rados.rados.0.smithi107.stdout:12148: writing smithi10729407-7 from 0 to 647168 tid 1 2020-06-12T02:39:23.051 INFO:tasks.rados.rados.0.smithi107.stdout:12148: writing smithi10729407-7 from 647168 to 1368064 tid 2 2020-06-12T02:39:23.055 INFO:tasks.rados.rados.0.smithi107.stdout:12148: writing smithi10729407-7 from 1368064 to 1851392 tid 3 2020-06-12T02:39:23.055 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:39:23.055 INFO:tasks.rados.rados.0.smithi107.stdout:12149: seq_num 12148 ranges {0=663552,663552=606208,1269760=598016} 2020-06-12T02:39:23.061 INFO:tasks.rados.rados.0.smithi107.stdout:12149: writing smithi10729407-19 from 0 to 663552 tid 1 2020-06-12T02:39:23.064 INFO:tasks.rados.rados.0.smithi107.stdout:12149: writing smithi10729407-19 from 663552 to 1269760 tid 2 2020-06-12T02:39:23.069 INFO:tasks.rados.rados.0.smithi107.stdout:12149: writing smithi10729407-19 from 1269760 to 1867776 tid 3 2020-06-12T02:39:23.069 INFO:tasks.rados.rados.0.smithi107.stdout:12150: delete oid 16 current snap is 0 2020-06-12T02:39:23.069 INFO:tasks.rados.rados.0.smithi107.stdout:12148: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:23.069 INFO:tasks.rados.rados.0.smithi107.stdout:12148: finishing write tid 2 to smithi10729407-7 2020-06-12T02:39:23.070 INFO:tasks.rados.rados.0.smithi107.stdout:12148: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:23.070 INFO:tasks.rados.rados.0.smithi107.stdout:12148: finishing write tid 5 to smithi10729407-7 2020-06-12T02:39:23.070 INFO:tasks.rados.rados.0.smithi107.stdout:12148: finishing write tid 6 to smithi10729407-7 2020-06-12T02:39:23.070 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1921 (ObjNum 12147 snap 0 seq_num 12147) dirty exists 2020-06-12T02:39:23.070 INFO:tasks.rados.rados.0.smithi107.stdout:12148: left oid 7 (ObjNum 12147 snap 0 seq_num 12147) 2020-06-12T02:39:23.073 INFO:tasks.rados.rados.0.smithi107.stdout:12148: done (2 left) 2020-06-12T02:39:23.073 INFO:tasks.rados.rados.0.smithi107.stdout:12150: done (1 left) 2020-06-12T02:39:23.073 INFO:tasks.rados.rados.0.smithi107.stdout:12151: delete oid 11 current snap is 0 2020-06-12T02:39:23.074 INFO:tasks.rados.rados.0.smithi107.stdout:12149: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:23.075 INFO:tasks.rados.rados.0.smithi107.stdout:12149: finishing write tid 2 to smithi10729407-19 2020-06-12T02:39:23.076 INFO:tasks.rados.rados.0.smithi107.stdout:12151: done (1 left) 2020-06-12T02:39:23.076 INFO:tasks.rados.rados.0.smithi107.stdout:12152: delete oid 17 current snap is 0 2020-06-12T02:39:23.076 INFO:tasks.rados.rados.0.smithi107.stdout:12149: finishing write tid 3 to smithi10729407-19 2020-06-12T02:39:23.077 INFO:tasks.rados.rados.0.smithi107.stdout:12149: finishing write tid 5 to smithi10729407-19 2020-06-12T02:39:23.077 INFO:tasks.rados.rados.0.smithi107.stdout:12149: finishing write tid 6 to smithi10729407-19 2020-06-12T02:39:23.078 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 2263 (ObjNum 12148 snap 0 seq_num 12148) dirty exists 2020-06-12T02:39:23.078 INFO:tasks.rados.rados.0.smithi107.stdout:12149: left oid 19 (ObjNum 12148 snap 0 seq_num 12148) 2020-06-12T02:39:23.079 INFO:tasks.rados.rados.0.smithi107.stdout:12149: done (1 left) 2020-06-12T02:39:23.079 INFO:tasks.rados.rados.0.smithi107.stdout:12152: done (0 left) 2020-06-12T02:39:23.080 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:39:23.081 INFO:tasks.rados.rados.0.smithi107.stdout:12153: seq_num 12152 ranges {0=729088,729088=598016,1327104=606208} 2020-06-12T02:39:23.082 INFO:tasks.rados.rados.0.smithi107.stdout:12153: writing smithi10729407-9 from 0 to 729088 tid 1 2020-06-12T02:39:23.087 INFO:tasks.rados.rados.0.smithi107.stdout:12153: writing smithi10729407-9 from 729088 to 1327104 tid 2 2020-06-12T02:39:23.090 INFO:tasks.rados.rados.0.smithi107.stdout:12153: writing smithi10729407-9 from 1327104 to 1933312 tid 3 2020-06-12T02:39:23.090 INFO:tasks.rados.rados.0.smithi107.stdout:12154: delete oid 4 current snap is 0 2020-06-12T02:39:23.093 INFO:tasks.rados.rados.0.smithi107.stdout:12153: finishing write tid 1 to smithi10729407-9 2020-06-12T02:39:23.097 INFO:tasks.rados.rados.0.smithi107.stdout:12153: finishing write tid 2 to smithi10729407-9 2020-06-12T02:39:23.097 INFO:tasks.rados.rados.0.smithi107.stdout:12154: done (1 left) 2020-06-12T02:39:23.097 INFO:tasks.rados.rados.0.smithi107.stdout:12155: delete oid 1 current snap is 0 2020-06-12T02:39:23.099 INFO:tasks.rados.rados.0.smithi107.stdout:12153: finishing write tid 3 to smithi10729407-9 2020-06-12T02:39:23.099 INFO:tasks.rados.rados.0.smithi107.stdout:12153: finishing write tid 5 to smithi10729407-9 2020-06-12T02:39:23.109 INFO:tasks.rados.rados.0.smithi107.stdout:12155: done (1 left) 2020-06-12T02:39:23.110 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 19 current snap is 0 2020-06-12T02:39:23.110 INFO:tasks.rados.rados.0.smithi107.stdout:12156: seq_num 12155 ranges {1867776=778240,2646016=647168,3293184=548864,3842048=8192} 2020-06-12T02:39:23.116 INFO:tasks.rados.rados.0.smithi107.stdout:12156: writing smithi10729407-19 from 1867776 to 2646016 tid 1 2020-06-12T02:39:23.119 INFO:tasks.rados.rados.0.smithi107.stdout:12156: writing smithi10729407-19 from 2646016 to 3293184 tid 2 2020-06-12T02:39:23.122 INFO:tasks.rados.rados.0.smithi107.stdout:12156: writing smithi10729407-19 from 3293184 to 3842048 tid 3 2020-06-12T02:39:23.123 INFO:tasks.rados.rados.0.smithi107.stdout:12156: writing smithi10729407-19 from 3842048 to 3850240 tid 4 2020-06-12T02:39:23.123 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:39:23.123 INFO:tasks.rados.rados.0.smithi107.stdout:12157: seq_num 12156 ranges {0=794624,794624=532480,1327104=671744} 2020-06-12T02:39:23.127 INFO:tasks.rados.rados.0.smithi107.stdout:12157: writing smithi10729407-14 from 0 to 794624 tid 1 2020-06-12T02:39:23.132 INFO:tasks.rados.rados.0.smithi107.stdout:12157: writing smithi10729407-14 from 794624 to 1327104 tid 2 2020-06-12T02:39:23.137 INFO:tasks.rados.rados.0.smithi107.stdout:12157: writing smithi10729407-14 from 1327104 to 1998848 tid 3 2020-06-12T02:39:23.137 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:39:23.137 INFO:tasks.rados.rados.0.smithi107.stdout:12158: seq_num 12157 ranges {2654208=409600,3063808=466944,3530752=516096,4046848=622592} 2020-06-12T02:39:23.138 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:39:23.134+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:39:23.153 INFO:tasks.rados.rados.0.smithi107.stdout:12158: writing smithi10729407-20 from 2654208 to 3063808 tid 1 2020-06-12T02:39:23.155 INFO:tasks.rados.rados.0.smithi107.stdout:12158: writing smithi10729407-20 from 3063808 to 3530752 tid 2 2020-06-12T02:39:23.158 INFO:tasks.rados.rados.0.smithi107.stdout:12158: writing smithi10729407-20 from 3530752 to 4046848 tid 3 2020-06-12T02:39:23.163 INFO:tasks.rados.rados.0.smithi107.stdout:12158: writing smithi10729407-20 from 4046848 to 4669440 tid 4 2020-06-12T02:39:23.163 INFO:tasks.rados.rados.0.smithi107.stdout:12159: delete oid 5 current snap is 0 2020-06-12T02:39:23.163 INFO:tasks.rados.rados.0.smithi107.stdout:12153: finishing write tid 6 to smithi10729407-9 2020-06-12T02:39:23.163 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1950 (ObjNum 12152 snap 0 seq_num 12152) dirty exists 2020-06-12T02:39:23.163 INFO:tasks.rados.rados.0.smithi107.stdout:12153: left oid 9 (ObjNum 12152 snap 0 seq_num 12152) 2020-06-12T02:39:23.164 INFO:tasks.rados.rados.0.smithi107.stdout:12156: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:23.164 INFO:tasks.rados.rados.0.smithi107.stdout:12157: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:23.164 INFO:tasks.rados.rados.0.smithi107.stdout:12156: finishing write tid 2 to smithi10729407-19 2020-06-12T02:39:23.164 INFO:tasks.rados.rados.0.smithi107.stdout:12156: finishing write tid 3 to smithi10729407-19 2020-06-12T02:39:23.165 INFO:tasks.rados.rados.0.smithi107.stdout:12156: finishing write tid 4 to smithi10729407-19 2020-06-12T02:39:23.165 INFO:tasks.rados.rados.0.smithi107.stdout:12156: finishing write tid 6 to smithi10729407-19 2020-06-12T02:39:23.165 INFO:tasks.rados.rados.0.smithi107.stdout:12156: finishing write tid 7 to smithi10729407-19 2020-06-12T02:39:23.165 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 2268 (ObjNum 12155 snap 0 seq_num 12155) dirty exists 2020-06-12T02:39:23.165 INFO:tasks.rados.rados.0.smithi107.stdout:12156: left oid 19 (ObjNum 12155 snap 0 seq_num 12155) 2020-06-12T02:39:23.166 INFO:tasks.rados.rados.0.smithi107.stdout:12157: finishing write tid 2 to smithi10729407-14 2020-06-12T02:39:23.166 INFO:tasks.rados.rados.0.smithi107.stdout:12157: finishing write tid 3 to smithi10729407-14 2020-06-12T02:39:23.166 INFO:tasks.rados.rados.0.smithi107.stdout:12157: finishing write tid 5 to smithi10729407-14 2020-06-12T02:39:23.166 INFO:tasks.rados.rados.0.smithi107.stdout:12157: finishing write tid 6 to smithi10729407-14 2020-06-12T02:39:23.166 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1855 (ObjNum 12156 snap 0 seq_num 12156) dirty exists 2020-06-12T02:39:23.167 INFO:tasks.rados.rados.0.smithi107.stdout:12157: left oid 14 (ObjNum 12156 snap 0 seq_num 12156) 2020-06-12T02:39:23.167 INFO:tasks.rados.rados.0.smithi107.stdout:12158: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:23.167 INFO:tasks.rados.rados.0.smithi107.stdout:12158: finishing write tid 2 to smithi10729407-20 2020-06-12T02:39:23.167 INFO:tasks.rados.rados.0.smithi107.stdout:12158: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:23.168 INFO:tasks.rados.rados.0.smithi107.stdout:12153: done (4 left) 2020-06-12T02:39:23.168 INFO:tasks.rados.rados.0.smithi107.stdout:12156: done (3 left) 2020-06-12T02:39:23.168 INFO:tasks.rados.rados.0.smithi107.stdout:12157: done (2 left) 2020-06-12T02:39:23.168 INFO:tasks.rados.rados.0.smithi107.stdout:12159: done (1 left) 2020-06-12T02:39:23.168 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 18 current snap is 0 2020-06-12T02:39:23.169 INFO:tasks.rados.rados.0.smithi107.stdout:12160: seq_num 12159 ranges {2654208=442368,3096576=638976,3735552=524288,4259840=442368} 2020-06-12T02:39:23.170 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:23.170 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:23.180 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-06-12T02:39:23.182 INFO:tasks.rados.rados.0.smithi107.stdout:12160: writing smithi10729407-18 from 2654208 to 3096576 tid 1 2020-06-12T02:39:23.186 INFO:tasks.rados.rados.0.smithi107.stdout:12160: writing smithi10729407-18 from 3096576 to 3735552 tid 2 2020-06-12T02:39:23.189 INFO:tasks.rados.rados.0.smithi107.stdout:12160: writing smithi10729407-18 from 3735552 to 4259840 tid 3 2020-06-12T02:39:23.192 INFO:tasks.rados.rados.0.smithi107.stdout:12160: writing smithi10729407-18 from 4259840 to 4702208 tid 4 2020-06-12T02:39:23.192 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:39:23.193 INFO:tasks.rados.rados.0.smithi107.stdout:12161: seq_num 12160 ranges {0=458752,458752=516096,974848=737280,1712128=352256} 2020-06-12T02:39:23.195 INFO:tasks.rados.rados.0.smithi107.stdout:12161: writing smithi10729407-10 from 0 to 458752 tid 1 2020-06-12T02:39:23.198 INFO:tasks.rados.rados.0.smithi107.stdout:12161: writing smithi10729407-10 from 458752 to 974848 tid 2 2020-06-12T02:39:23.203 INFO:tasks.rados.rados.0.smithi107.stdout:12161: writing smithi10729407-10 from 974848 to 1712128 tid 3 2020-06-12T02:39:23.206 INFO:tasks.rados.rados.0.smithi107.stdout:12161: writing smithi10729407-10 from 1712128 to 2064384 tid 4 2020-06-12T02:39:23.207 INFO:tasks.rados.rados.0.smithi107.stdout:12162: delete oid 4 current snap is 0 2020-06-12T02:39:23.207 INFO:tasks.rados.rados.0.smithi107.stdout:12158: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:23.207 INFO:tasks.rados.rados.0.smithi107.stdout:12158: finishing write tid 6 to smithi10729407-20 2020-06-12T02:39:23.208 INFO:tasks.rados.rados.0.smithi107.stdout:12158: finishing write tid 7 to smithi10729407-20 2020-06-12T02:39:23.208 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 949 (ObjNum 12157 snap 0 seq_num 12157) dirty exists 2020-06-12T02:39:23.208 INFO:tasks.rados.rados.0.smithi107.stdout:12158: left oid 20 (ObjNum 12157 snap 0 seq_num 12157) 2020-06-12T02:39:23.208 INFO:tasks.rados.rados.0.smithi107.stdout:12160: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:23.209 INFO:tasks.rados.rados.0.smithi107.stdout:12160: finishing write tid 2 to smithi10729407-18 2020-06-12T02:39:23.209 INFO:tasks.rados.rados.0.smithi107.stdout:12161: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:23.209 INFO:tasks.rados.rados.0.smithi107.stdout:12160: finishing write tid 3 to smithi10729407-18 2020-06-12T02:39:23.210 INFO:tasks.rados.rados.0.smithi107.stdout:12160: finishing write tid 4 to smithi10729407-18 2020-06-12T02:39:23.210 INFO:tasks.rados.rados.0.smithi107.stdout:12160: finishing write tid 6 to smithi10729407-18 2020-06-12T02:39:23.210 INFO:tasks.rados.rados.0.smithi107.stdout:12160: finishing write tid 7 to smithi10729407-18 2020-06-12T02:39:23.210 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1981 (ObjNum 12159 snap 0 seq_num 12159) dirty exists 2020-06-12T02:39:23.210 INFO:tasks.rados.rados.0.smithi107.stdout:12160: left oid 18 (ObjNum 12159 snap 0 seq_num 12159) 2020-06-12T02:39:23.221 INFO:tasks.rados.rados.0.smithi107.stdout:12161: finishing write tid 2 to smithi10729407-10 2020-06-12T02:39:23.222 INFO:tasks.rados.rados.0.smithi107.stdout:12158: done (3 left) 2020-06-12T02:39:23.222 INFO:tasks.rados.rados.0.smithi107.stdout:12160: done (2 left) 2020-06-12T02:39:23.222 INFO:tasks.rados.rados.0.smithi107.stdout:12162: done (1 left) 2020-06-12T02:39:23.222 INFO:tasks.rados.rados.0.smithi107.stdout:12163: delete oid 9 current snap is 0 2020-06-12T02:39:23.223 INFO:tasks.rados.rados.0.smithi107.stdout:12163: done (1 left) 2020-06-12T02:39:23.223 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:39:23.223 INFO:tasks.rados.rados.0.smithi107.stdout:12164: seq_num 12163 ranges {0=516096,516096=573440,1089536=507904,1597440=507904,2105344=16384} 2020-06-12T02:39:23.223 INFO:tasks.rados.rados.0.smithi107.stdout:12164: writing smithi10729407-9 from 0 to 516096 tid 1 2020-06-12T02:39:23.223 INFO:tasks.rados.rados.0.smithi107.stdout:12164: writing smithi10729407-9 from 516096 to 1089536 tid 2 2020-06-12T02:39:23.224 INFO:tasks.rados.rados.0.smithi107.stdout:12164: writing smithi10729407-9 from 1089536 to 1597440 tid 3 2020-06-12T02:39:23.227 INFO:tasks.rados.rados.0.smithi107.stdout:12164: writing smithi10729407-9 from 1597440 to 2105344 tid 4 2020-06-12T02:39:23.228 INFO:tasks.rados.rados.0.smithi107.stdout:12164: writing smithi10729407-9 from 2105344 to 2121728 tid 5 2020-06-12T02:39:23.228 INFO:tasks.rados.rados.0.smithi107.stdout:12165: delete oid 19 current snap is 0 2020-06-12T02:39:23.228 INFO:tasks.rados.rados.0.smithi107.stdout:12161: finishing write tid 3 to smithi10729407-10 2020-06-12T02:39:23.228 INFO:tasks.rados.rados.0.smithi107.stdout:12161: finishing write tid 4 to smithi10729407-10 2020-06-12T02:39:23.228 INFO:tasks.rados.rados.0.smithi107.stdout:12161: finishing write tid 6 to smithi10729407-10 2020-06-12T02:39:23.229 INFO:tasks.rados.rados.0.smithi107.stdout:12161: finishing write tid 7 to smithi10729407-10 2020-06-12T02:39:23.229 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 927 (ObjNum 12160 snap 0 seq_num 12160) dirty exists 2020-06-12T02:39:23.229 INFO:tasks.rados.rados.0.smithi107.stdout:12161: left oid 10 (ObjNum 12160 snap 0 seq_num 12160) 2020-06-12T02:39:23.230 INFO:tasks.rados.rados.0.smithi107.stdout:12161: done (2 left) 2020-06-12T02:39:23.230 INFO:tasks.rados.rados.0.smithi107.stdout:12165: done (1 left) 2020-06-12T02:39:23.230 INFO:tasks.rados.rados.0.smithi107.stdout:12166: delete oid 6 current snap is 0 2020-06-12T02:39:23.231 INFO:tasks.rados.rados.0.smithi107.stdout:12164: finishing write tid 1 to smithi10729407-9 2020-06-12T02:39:23.231 INFO:tasks.rados.rados.0.smithi107.stdout:12164: finishing write tid 2 to smithi10729407-9 2020-06-12T02:39:23.234 INFO:tasks.rados.rados.0.smithi107.stdout:12166: done (1 left) 2020-06-12T02:39:23.234 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:39:23.234 INFO:tasks.rados.rados.0.smithi107.stdout:12167: seq_num 12166 ranges {0=565248,565248=671744,1236992=679936,1916928=253952} 2020-06-12T02:39:23.237 INFO:tasks.rados.rados.0.smithi107.stdout:12167: writing smithi10729407-19 from 0 to 565248 tid 1 2020-06-12T02:39:23.238 INFO:tasks.ceph.osd.3.smithi107.stderr:2020-06-12T02:39:23.235+0000 7fa5efc5c700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26609) UID: 0 2020-06-12T02:39:23.241 INFO:tasks.rados.rados.0.smithi107.stdout:12167: writing smithi10729407-19 from 565248 to 1236992 tid 2 2020-06-12T02:39:23.245 INFO:tasks.rados.rados.0.smithi107.stdout:12167: writing smithi10729407-19 from 1236992 to 1916928 tid 3 2020-06-12T02:39:23.247 INFO:tasks.rados.rados.0.smithi107.stdout:12167: writing smithi10729407-19 from 1916928 to 2170880 tid 4 2020-06-12T02:39:23.247 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:39:23.247 INFO:tasks.rados.rados.0.smithi107.stdout:12168: seq_num 12167 ranges {0=581632,581632=557056,1138688=491520,1630208=557056} 2020-06-12T02:39:23.258 INFO:tasks.rados.rados.0.smithi107.stdout:12168: writing smithi10729407-3 from 0 to 581632 tid 1 2020-06-12T02:39:23.259 INFO:tasks.rados.rados.0.smithi107.stdout:12168: writing smithi10729407-3 from 581632 to 1138688 tid 2 2020-06-12T02:39:23.259 INFO:tasks.rados.rados.0.smithi107.stdout:12168: writing smithi10729407-3 from 1138688 to 1630208 tid 3 2020-06-12T02:39:23.262 INFO:tasks.rados.rados.0.smithi107.stdout:12168: writing smithi10729407-3 from 1630208 to 2187264 tid 4 2020-06-12T02:39:23.262 INFO:tasks.rados.rados.0.smithi107.stdout:12169: delete oid 15 current snap is 0 2020-06-12T02:39:23.262 INFO:tasks.rados.rados.0.smithi107.stdout:12164: finishing write tid 3 to smithi10729407-9 2020-06-12T02:39:23.263 INFO:tasks.rados.rados.0.smithi107.stdout:12164: finishing write tid 4 to smithi10729407-9 2020-06-12T02:39:23.263 INFO:tasks.rados.rados.0.smithi107.stdout:12164: finishing write tid 5 to smithi10729407-9 2020-06-12T02:39:23.263 INFO:tasks.rados.rados.0.smithi107.stdout:12164: finishing write tid 7 to smithi10729407-9 2020-06-12T02:39:23.263 INFO:tasks.rados.rados.0.smithi107.stdout:12164: finishing write tid 8 to smithi10729407-9 2020-06-12T02:39:23.263 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1957 (ObjNum 12163 snap 0 seq_num 12163) dirty exists 2020-06-12T02:39:23.264 INFO:tasks.rados.rados.0.smithi107.stdout:12164: left oid 9 (ObjNum 12163 snap 0 seq_num 12163) 2020-06-12T02:39:23.264 INFO:tasks.rados.rados.0.smithi107.stdout:12167: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:23.264 INFO:tasks.rados.rados.0.smithi107.stdout:12167: finishing write tid 2 to smithi10729407-19 2020-06-12T02:39:23.264 INFO:tasks.rados.rados.0.smithi107.stdout:12167: finishing write tid 3 to smithi10729407-19 2020-06-12T02:39:23.264 INFO:tasks.rados.rados.0.smithi107.stdout:12167: finishing write tid 4 to smithi10729407-19 2020-06-12T02:39:23.265 INFO:tasks.rados.rados.0.smithi107.stdout:12167: finishing write tid 6 to smithi10729407-19 2020-06-12T02:39:23.265 INFO:tasks.rados.rados.0.smithi107.stdout:12167: finishing write tid 7 to smithi10729407-19 2020-06-12T02:39:23.266 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 2274 (ObjNum 12166 snap 0 seq_num 12166) dirty exists 2020-06-12T02:39:23.266 INFO:tasks.rados.rados.0.smithi107.stdout:12167: left oid 19 (ObjNum 12166 snap 0 seq_num 12166) 2020-06-12T02:39:23.266 INFO:tasks.rados.rados.0.smithi107.stdout:12164: done (3 left) 2020-06-12T02:39:23.267 INFO:tasks.rados.rados.0.smithi107.stdout:12167: done (2 left) 2020-06-12T02:39:23.267 INFO:tasks.rados.rados.0.smithi107.stdout:12169: done (1 left) 2020-06-12T02:39:23.267 INFO:tasks.rados.rados.0.smithi107.stdout:12170: delete oid 13 current snap is 0 2020-06-12T02:39:23.271 INFO:tasks.rados.rados.0.smithi107.stdout:12170: done (1 left) 2020-06-12T02:39:23.271 INFO:tasks.rados.rados.0.smithi107.stdout:12171: delete oid 15 current snap is 0 2020-06-12T02:39:23.272 INFO:tasks.rados.rados.0.smithi107.stdout:12168: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:23.272 INFO:tasks.rados.rados.0.smithi107.stdout:12168: finishing write tid 2 to smithi10729407-3 2020-06-12T02:39:23.272 INFO:tasks.rados.rados.0.smithi107.stdout:12168: finishing write tid 3 to smithi10729407-3 2020-06-12T02:39:23.273 INFO:tasks.rados.rados.0.smithi107.stdout:12168: finishing write tid 4 to smithi10729407-3 2020-06-12T02:39:23.273 INFO:tasks.rados.rados.0.smithi107.stdout:12168: finishing write tid 6 to smithi10729407-3 2020-06-12T02:39:23.274 INFO:tasks.rados.rados.0.smithi107.stdout:12168: finishing write tid 7 to smithi10729407-3 2020-06-12T02:39:23.274 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 2149 (ObjNum 12167 snap 0 seq_num 12167) dirty exists 2020-06-12T02:39:23.275 INFO:tasks.rados.rados.0.smithi107.stdout:12168: left oid 3 (ObjNum 12167 snap 0 seq_num 12167) 2020-06-12T02:39:23.275 INFO:tasks.rados.rados.0.smithi107.stdout:12168: done (1 left) 2020-06-12T02:39:23.275 INFO:tasks.rados.rados.0.smithi107.stdout:12171: done (0 left) 2020-06-12T02:39:23.275 INFO:tasks.rados.rados.0.smithi107.stdout:12172: delete oid 13 current snap is 0 2020-06-12T02:39:23.277 INFO:tasks.rados.rados.0.smithi107.stdout:12172: done (0 left) 2020-06-12T02:39:23.278 INFO:tasks.rados.rados.0.smithi107.stdout:12173: delete oid 15 current snap is 0 2020-06-12T02:39:23.279 INFO:tasks.rados.rados.0.smithi107.stdout:12173: done (0 left) 2020-06-12T02:39:23.280 INFO:tasks.rados.rados.0.smithi107.stdout:12174: delete oid 2 current snap is 0 2020-06-12T02:39:23.283 INFO:tasks.rados.rados.0.smithi107.stdout:12174: done (0 left) 2020-06-12T02:39:23.283 INFO:tasks.rados.rados.0.smithi107.stdout:12175: delete oid 3 current snap is 0 2020-06-12T02:39:23.284 INFO:tasks.rados.rados.0.smithi107.stdout:12175: done (0 left) 2020-06-12T02:39:23.284 INFO:tasks.rados.rados.0.smithi107.stdout:12176: delete oid 1 current snap is 0 2020-06-12T02:39:23.287 INFO:tasks.rados.rados.0.smithi107.stdout:12176: done (0 left) 2020-06-12T02:39:23.288 INFO:tasks.rados.rados.0.smithi107.stdout:12177: delete oid 20 current snap is 0 2020-06-12T02:39:23.291 INFO:tasks.rados.rados.0.smithi107.stdout:12177: done (0 left) 2020-06-12T02:39:23.291 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 10 current snap is 0 2020-06-12T02:39:23.291 INFO:tasks.rados.rados.0.smithi107.stdout:12178: seq_num 12177 ranges {2064384=745472,2809856=655360,3465216=466944,3932160=442368,4374528=40960} 2020-06-12T02:39:23.305 INFO:tasks.rados.rados.0.smithi107.stdout:12178: writing smithi10729407-10 from 2064384 to 2809856 tid 1 2020-06-12T02:39:23.309 INFO:tasks.rados.rados.0.smithi107.stdout:12178: writing smithi10729407-10 from 2809856 to 3465216 tid 2 2020-06-12T02:39:23.312 INFO:tasks.rados.rados.0.smithi107.stdout:12178: writing smithi10729407-10 from 3465216 to 3932160 tid 3 2020-06-12T02:39:23.315 INFO:tasks.rados.rados.0.smithi107.stdout:12178: writing smithi10729407-10 from 3932160 to 4374528 tid 4 2020-06-12T02:39:23.315 INFO:tasks.rados.rados.0.smithi107.stdout:12178: writing smithi10729407-10 from 4374528 to 4415488 tid 5 2020-06-12T02:39:23.316 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:39:23.316 INFO:tasks.rados.rados.0.smithi107.stdout:12179: seq_num 12178 ranges {0=761856,761856=540672,1302528=622592,1925120=442368} 2020-06-12T02:39:23.320 INFO:tasks.rados.rados.0.smithi107.stdout:12179: writing smithi10729407-6 from 0 to 761856 tid 1 2020-06-12T02:39:23.323 INFO:tasks.rados.rados.0.smithi107.stdout:12179: writing smithi10729407-6 from 761856 to 1302528 tid 2 2020-06-12T02:39:23.327 INFO:tasks.rados.rados.0.smithi107.stdout:12179: writing smithi10729407-6 from 1302528 to 1925120 tid 3 2020-06-12T02:39:23.330 INFO:tasks.rados.rados.0.smithi107.stdout:12179: writing smithi10729407-6 from 1925120 to 2367488 tid 4 2020-06-12T02:39:23.331 INFO:tasks.rados.rados.0.smithi107.stdout:12180: delete oid 9 current snap is 0 2020-06-12T02:39:23.331 INFO:tasks.rados.rados.0.smithi107.stdout:12178: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:23.331 INFO:tasks.rados.rados.0.smithi107.stdout:12178: finishing write tid 2 to smithi10729407-10 2020-06-12T02:39:23.331 INFO:tasks.rados.rados.0.smithi107.stdout:12178: finishing write tid 3 to smithi10729407-10 2020-06-12T02:39:23.331 INFO:tasks.rados.rados.0.smithi107.stdout:12178: finishing write tid 4 to smithi10729407-10 2020-06-12T02:39:23.332 INFO:tasks.rados.rados.0.smithi107.stdout:12178: finishing write tid 5 to smithi10729407-10 2020-06-12T02:39:23.332 INFO:tasks.rados.rados.0.smithi107.stdout:12178: finishing write tid 7 to smithi10729407-10 2020-06-12T02:39:23.332 INFO:tasks.rados.rados.0.smithi107.stdout:12178: finishing write tid 8 to smithi10729407-10 2020-06-12T02:39:23.332 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 933 (ObjNum 12177 snap 0 seq_num 12177) dirty exists 2020-06-12T02:39:23.333 INFO:tasks.rados.rados.0.smithi107.stdout:12178: left oid 10 (ObjNum 12177 snap 0 seq_num 12177) 2020-06-12T02:39:23.333 INFO:tasks.rados.rados.0.smithi107.stdout:12178: done (2 left) 2020-06-12T02:39:23.333 INFO:tasks.rados.rados.0.smithi107.stdout:12180: done (1 left) 2020-06-12T02:39:23.334 INFO:tasks.rados.rados.0.smithi107.stdout:12181: delete oid 15 current snap is 0 2020-06-12T02:39:23.337 INFO:tasks.rados.rados.0.smithi107.stdout:12181: done (1 left) 2020-06-12T02:39:23.337 INFO:tasks.rados.rados.0.smithi107.stdout:12182: delete oid 13 current snap is 0 2020-06-12T02:39:23.337 INFO:tasks.rados.rados.0.smithi107.stdout:12179: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:23.339 INFO:tasks.rados.rados.0.smithi107.stdout:12182: done (1 left) 2020-06-12T02:39:23.339 INFO:tasks.rados.rados.0.smithi107.stdout:12183: delete oid 3 current snap is 0 2020-06-12T02:39:23.339 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:39:23.336+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:39:23.340 INFO:tasks.rados.rados.0.smithi107.stdout:12179: finishing write tid 2 to smithi10729407-6 2020-06-12T02:39:23.340 INFO:tasks.rados.rados.0.smithi107.stdout:12179: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:23.342 INFO:tasks.rados.rados.0.smithi107.stdout:12179: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:23.343 INFO:tasks.rados.rados.0.smithi107.stdout:12183: done (1 left) 2020-06-12T02:39:23.343 INFO:tasks.rados.rados.0.smithi107.stdout:12184: delete oid 7 current snap is 0 2020-06-12T02:39:23.344 INFO:tasks.rados.rados.0.smithi107.stdout:12179: finishing write tid 6 to smithi10729407-6 2020-06-12T02:39:23.346 INFO:tasks.rados.rados.0.smithi107.stdout:12179: finishing write tid 7 to smithi10729407-6 2020-06-12T02:39:23.347 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 2225 (ObjNum 12178 snap 0 seq_num 12178) dirty exists 2020-06-12T02:39:23.347 INFO:tasks.rados.rados.0.smithi107.stdout:12179: left oid 6 (ObjNum 12178 snap 0 seq_num 12178) 2020-06-12T02:39:23.347 INFO:tasks.rados.rados.0.smithi107.stdout:12179: done (1 left) 2020-06-12T02:39:23.347 INFO:tasks.rados.rados.0.smithi107.stdout:12184: done (0 left) 2020-06-12T02:39:23.347 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 4 current snap is 0 2020-06-12T02:39:23.348 INFO:tasks.rados.rados.0.smithi107.stdout:12185: seq_num 12184 ranges {0=65536} 2020-06-12T02:39:23.348 INFO:tasks.rados.rados.0.smithi107.stdout:12185: writing smithi10729407-4 from 0 to 65536 tid 1 2020-06-12T02:39:23.348 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 16 current snap is 0 2020-06-12T02:39:23.348 INFO:tasks.rados.rados.0.smithi107.stdout:12186: seq_num 12185 ranges {0=81920} 2020-06-12T02:39:23.348 INFO:tasks.rados.rados.0.smithi107.stdout:12186: writing smithi10729407-16 from 0 to 81920 tid 1 2020-06-12T02:39:23.348 INFO:tasks.rados.rados.0.smithi107.stdout:12187: delete oid 18 current snap is 0 2020-06-12T02:39:23.349 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:23.349 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:23.349 INFO:tasks.rados.rados.0.smithi107.stdout:12185: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:23.352 INFO:tasks.rados.rados.0.smithi107.stdout:12186: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:23.353 INFO:tasks.rados.rados.0.smithi107.stdout:12187: done (2 left) 2020-06-12T02:39:23.353 INFO:tasks.rados.rados.0.smithi107.stdout:12188: delete oid 18 current snap is 0 2020-06-12T02:39:23.353 INFO:tasks.rados.rados.0.smithi107.stdout:12185: finishing write tid 3 to smithi10729407-4 2020-06-12T02:39:23.357 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-06-12T02:39:23.358 INFO:tasks.rados.rados.0.smithi107.stdout:12186: finishing write tid 3 to smithi10729407-16 2020-06-12T02:39:23.358 INFO:tasks.rados.rados.0.smithi107.stdout:12185: finishing write tid 4 to smithi10729407-4 2020-06-12T02:39:23.359 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 2257 (ObjNum 12184 snap 0 seq_num 12184) dirty exists 2020-06-12T02:39:23.359 INFO:tasks.rados.rados.0.smithi107.stdout:12185: left oid 4 (ObjNum 12184 snap 0 seq_num 12184) 2020-06-12T02:39:23.360 INFO:tasks.rados.rados.0.smithi107.stdout:12185: done (2 left) 2020-06-12T02:39:23.360 INFO:tasks.rados.rados.0.smithi107.stdout:12188: done (1 left) 2020-06-12T02:39:23.360 INFO:tasks.rados.rados.0.smithi107.stdout:12189: delete oid 9 current snap is 0 2020-06-12T02:39:23.361 INFO:tasks.rados.rados.0.smithi107.stdout:12186: finishing write tid 4 to smithi10729407-16 2020-06-12T02:39:23.361 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 2153 (ObjNum 12185 snap 0 seq_num 12185) dirty exists 2020-06-12T02:39:23.361 INFO:tasks.rados.rados.0.smithi107.stdout:12186: left oid 16 (ObjNum 12185 snap 0 seq_num 12185) 2020-06-12T02:39:23.362 INFO:tasks.rados.rados.0.smithi107.stdout:12186: done (1 left) 2020-06-12T02:39:23.362 INFO:tasks.rados.rados.0.smithi107.stdout:12189: done (0 left) 2020-06-12T02:39:23.362 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:39:23.363 INFO:tasks.rados.rados.0.smithi107.stdout:12190: seq_num 12189 ranges {0=147456} 2020-06-12T02:39:23.371 INFO:tasks.rados.rados.0.smithi107.stdout:12190: writing smithi10729407-20 from 0 to 147456 tid 1 2020-06-12T02:39:23.372 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 6 current snap is 0 2020-06-12T02:39:23.372 INFO:tasks.rados.rados.0.smithi107.stdout:12191: seq_num 12190 ranges {2367488=163840} 2020-06-12T02:39:23.377 INFO:tasks.rados.rados.0.smithi107.stdout:12191: writing smithi10729407-6 from 2367488 to 2531328 tid 1 2020-06-12T02:39:23.378 INFO:tasks.rados.rados.0.smithi107.stdout:12192: delete oid 9 current snap is 0 2020-06-12T02:39:23.378 INFO:tasks.rados.rados.0.smithi107.stdout:12190: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:23.378 INFO:tasks.rados.rados.0.smithi107.stdout:12190: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:23.378 INFO:tasks.rados.rados.0.smithi107.stdout:12190: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:23.378 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 952 (ObjNum 12189 snap 0 seq_num 12189) dirty exists 2020-06-12T02:39:23.379 INFO:tasks.rados.rados.0.smithi107.stdout:12190: left oid 20 (ObjNum 12189 snap 0 seq_num 12189) 2020-06-12T02:39:23.380 INFO:tasks.rados.rados.0.smithi107.stdout:12190: done (2 left) 2020-06-12T02:39:23.380 INFO:tasks.rados.rados.0.smithi107.stdout:12192: done (1 left) 2020-06-12T02:39:23.380 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 9 current snap is 0 2020-06-12T02:39:23.381 INFO:tasks.rados.rados.0.smithi107.stdout:12193: seq_num 12192 ranges {0=196608} 2020-06-12T02:39:23.381 INFO:tasks.rados.rados.0.smithi107.stdout:12193: writing smithi10729407-9 from 0 to 196608 tid 1 2020-06-12T02:39:23.381 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:39:23.382 INFO:tasks.rados.rados.0.smithi107.stdout:12194: seq_num 12193 ranges {0=212992} 2020-06-12T02:39:23.383 INFO:tasks.rados.rados.0.smithi107.stdout:12194: writing smithi10729407-15 from 0 to 212992 tid 1 2020-06-12T02:39:23.383 INFO:tasks.rados.rados.0.smithi107.stdout:12195: delete oid 11 current snap is 0 2020-06-12T02:39:23.383 INFO:tasks.rados.rados.0.smithi107.stdout:12191: finishing write tid 1 to smithi10729407-6 2020-06-12T02:39:23.383 INFO:tasks.rados.rados.0.smithi107.stdout:12191: finishing write tid 3 to smithi10729407-6 2020-06-12T02:39:23.384 INFO:tasks.rados.rados.0.smithi107.stdout:12191: finishing write tid 4 to smithi10729407-6 2020-06-12T02:39:23.384 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 6 v 2227 (ObjNum 12190 snap 0 seq_num 12190) dirty exists 2020-06-12T02:39:23.385 INFO:tasks.rados.rados.0.smithi107.stdout:12191: left oid 6 (ObjNum 12190 snap 0 seq_num 12190) 2020-06-12T02:39:23.385 INFO:tasks.rados.rados.0.smithi107.stdout:12193: finishing write tid 1 to smithi10729407-9 2020-06-12T02:39:23.386 INFO:tasks.rados.rados.0.smithi107.stdout:12193: finishing write tid 3 to smithi10729407-9 2020-06-12T02:39:23.387 INFO:tasks.rados.rados.0.smithi107.stdout:12191: done (3 left) 2020-06-12T02:39:23.387 INFO:tasks.rados.rados.0.smithi107.stdout:12195: done (2 left) 2020-06-12T02:39:23.387 INFO:tasks.rados.rados.0.smithi107.stdout:12196: delete oid 1 current snap is 0 2020-06-12T02:39:23.388 INFO:tasks.rados.rados.0.smithi107.stdout:12194: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:23.389 INFO:tasks.rados.rados.0.smithi107.stdout:12193: finishing write tid 4 to smithi10729407-9 2020-06-12T02:39:23.389 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1962 (ObjNum 12192 snap 0 seq_num 12192) dirty exists 2020-06-12T02:39:23.389 INFO:tasks.rados.rados.0.smithi107.stdout:12193: left oid 9 (ObjNum 12192 snap 0 seq_num 12192) 2020-06-12T02:39:23.390 INFO:tasks.rados.rados.0.smithi107.stdout:12193: done (2 left) 2020-06-12T02:39:23.390 INFO:tasks.rados.rados.0.smithi107.stdout:12196: done (1 left) 2020-06-12T02:39:23.391 INFO:tasks.rados.rados.0.smithi107.stdout:12197: delete oid 5 current snap is 0 2020-06-12T02:39:23.391 INFO:tasks.rados.rados.0.smithi107.stdout:12194: finishing write tid 3 to smithi10729407-15 2020-06-12T02:39:23.393 INFO:tasks.rados.rados.0.smithi107.stdout:12194: finishing write tid 4 to smithi10729407-15 2020-06-12T02:39:23.393 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 2279 (ObjNum 12193 snap 0 seq_num 12193) dirty exists 2020-06-12T02:39:23.393 INFO:tasks.rados.rados.0.smithi107.stdout:12194: left oid 15 (ObjNum 12193 snap 0 seq_num 12193) 2020-06-12T02:39:23.394 INFO:tasks.rados.rados.0.smithi107.stdout:12194: done (1 left) 2020-06-12T02:39:23.394 INFO:tasks.rados.rados.0.smithi107.stdout:12197: done (0 left) 2020-06-12T02:39:23.394 INFO:tasks.rados.rados.0.smithi107.stdout:12198: delete oid 13 current snap is 0 2020-06-12T02:39:23.395 INFO:tasks.rados.rados.0.smithi107.stdout:12198: done (0 left) 2020-06-12T02:39:23.395 INFO:tasks.rados.rados.0.smithi107.stdout:12199: delete oid 1 current snap is 0 2020-06-12T02:39:23.398 INFO:tasks.rados.rados.0.smithi107.stdout:12199: done (0 left) 2020-06-12T02:39:23.399 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 16 current snap is 0 2020-06-12T02:39:23.399 INFO:tasks.rados.rados.0.smithi107.stdout:12200: seq_num 12199 ranges {81920=311296} 2020-06-12T02:39:23.401 INFO:tasks.rados.rados.0.smithi107.stdout:12200: writing smithi10729407-16 from 81920 to 393216 tid 1 2020-06-12T02:39:23.401 INFO:tasks.rados.rados.0.smithi107.stdout:12201: delete oid 20 current snap is 0 2020-06-12T02:39:23.403 INFO:teuthology.orchestra.run.smithi188:> CEPH_ARGS='--filestore-merge-threshold 1 --filestore-split-multiple 1' sudo -E ceph-objectstore-tool --no-mon-config --log-file=/var/log/ceph/objectstore_tool.$pid.log --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op apply-layout-settings --pool rbd 2020-06-12T02:39:23.404 INFO:tasks.rados.rados.0.smithi107.stdout:12201: done (1 left) 2020-06-12T02:39:23.404 INFO:tasks.rados.rados.0.smithi107.stdout:12202: delete oid 6 current snap is 0 2020-06-12T02:39:23.406 INFO:tasks.rados.rados.0.smithi107.stdout:12202: done (1 left) 2020-06-12T02:39:23.406 INFO:tasks.rados.rados.0.smithi107.stdout:12203: delete oid 4 current snap is 0 2020-06-12T02:39:23.407 INFO:tasks.rados.rados.0.smithi107.stdout:12200: finishing write tid 1 to smithi10729407-16 2020-06-12T02:39:23.407 INFO:tasks.rados.rados.0.smithi107.stdout:12200: finishing write tid 3 to smithi10729407-16 2020-06-12T02:39:23.409 INFO:tasks.rados.rados.0.smithi107.stdout:12200: finishing write tid 4 to smithi10729407-16 2020-06-12T02:39:23.409 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 16 v 2155 (ObjNum 12199 snap 0 seq_num 12199) dirty exists 2020-06-12T02:39:23.409 INFO:tasks.rados.rados.0.smithi107.stdout:12200: left oid 16 (ObjNum 12199 snap 0 seq_num 12199) 2020-06-12T02:39:23.410 INFO:tasks.rados.rados.0.smithi107.stdout:12200: done (1 left) 2020-06-12T02:39:23.410 INFO:tasks.rados.rados.0.smithi107.stdout:12203: done (0 left) 2020-06-12T02:39:23.410 INFO:tasks.rados.rados.0.smithi107.stdout:12204: delete oid 1 current snap is 0 2020-06-12T02:39:23.415 INFO:tasks.rados.rados.0.smithi107.stdout:12204: done (0 left) 2020-06-12T02:39:23.416 INFO:tasks.rados.rados.0.smithi107.stdout:12205: delete oid 9 current snap is 0 2020-06-12T02:39:23.417 INFO:tasks.rados.rados.0.smithi107.stdout:12205: done (0 left) 2020-06-12T02:39:23.417 INFO:tasks.rados.rados.0.smithi107.stdout:12206: delete oid 17 current snap is 0 2020-06-12T02:39:23.417 INFO:tasks.rados.rados.0.smithi107.stdout:12206: done (0 left) 2020-06-12T02:39:23.417 INFO:tasks.rados.rados.0.smithi107.stdout:12207: delete oid 9 current snap is 0 2020-06-12T02:39:23.418 INFO:tasks.rados.rados.0.smithi107.stdout:12207: done (0 left) 2020-06-12T02:39:23.419 INFO:tasks.rados.rados.0.smithi107.stdout:12208: delete oid 15 current snap is 0 2020-06-12T02:39:23.421 INFO:tasks.rados.rados.0.smithi107.stdout:12208: done (0 left) 2020-06-12T02:39:23.421 INFO:tasks.rados.rados.0.smithi107.stdout:12209: delete oid 5 current snap is 0 2020-06-12T02:39:23.423 INFO:tasks.rados.rados.0.smithi107.stdout:12209: done (0 left) 2020-06-12T02:39:23.423 INFO:tasks.rados.rados.0.smithi107.stdout:12210: delete oid 4 current snap is 0 2020-06-12T02:39:23.425 INFO:tasks.rados.rados.0.smithi107.stdout:12210: done (0 left) 2020-06-12T02:39:23.426 INFO:tasks.rados.rados.0.smithi107.stdout:12211: delete oid 15 current snap is 0 2020-06-12T02:39:23.428 INFO:tasks.rados.rados.0.smithi107.stdout:12211: done (0 left) 2020-06-12T02:39:23.428 INFO:tasks.rados.rados.0.smithi107.stdout:12212: delete oid 6 current snap is 0 2020-06-12T02:39:23.431 INFO:tasks.rados.rados.0.smithi107.stdout:12212: done (0 left) 2020-06-12T02:39:23.431 INFO:tasks.rados.rados.0.smithi107.stdout:12213: delete oid 14 current snap is 0 2020-06-12T02:39:23.433 INFO:tasks.rados.rados.0.smithi107.stdout:12213: done (0 left) 2020-06-12T02:39:23.433 INFO:tasks.rados.rados.0.smithi107.stdout:12214: delete oid 4 current snap is 0 2020-06-12T02:39:23.435 INFO:tasks.rados.rados.0.smithi107.stdout:12214: done (0 left) 2020-06-12T02:39:23.436 INFO:tasks.rados.rados.0.smithi107.stdout:12215: delete oid 4 current snap is 0 2020-06-12T02:39:23.438 INFO:tasks.rados.rados.0.smithi107.stdout:12215: done (0 left) 2020-06-12T02:39:23.438 INFO:tasks.rados.rados.0.smithi107.stdout:12216: delete oid 9 current snap is 0 2020-06-12T02:39:23.443 INFO:tasks.rados.rados.0.smithi107.stdout:12216: done (0 left) 2020-06-12T02:39:23.445 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 12 current snap is 0 2020-06-12T02:39:23.445 INFO:tasks.rados.rados.0.smithi107.stdout:12217: seq_num 12216 ranges {0=598016} 2020-06-12T02:39:23.446 INFO:tasks.ceph.osd.4.smithi188.stderr:2020-06-12T02:39:23.443+0000 7fd21518e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26672) UID: 0 2020-06-12T02:39:23.446 INFO:tasks.rados.rados.0.smithi107.stdout:12217: writing smithi10729407-12 from 0 to 598016 tid 1 2020-06-12T02:39:23.447 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:39:23.447 INFO:tasks.rados.rados.0.smithi107.stdout:12218: seq_num 12217 ranges {0=614400} 2020-06-12T02:39:23.448 INFO:tasks.rados.rados.0.smithi107.stdout:12218: writing smithi10729407-1 from 0 to 614400 tid 1 2020-06-12T02:39:23.448 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 14 current snap is 0 2020-06-12T02:39:23.448 INFO:tasks.rados.rados.0.smithi107.stdout:12219: seq_num 12218 ranges {0=630784} 2020-06-12T02:39:23.452 INFO:tasks.rados.rados.0.smithi107.stdout:12219: writing smithi10729407-14 from 0 to 630784 tid 1 2020-06-12T02:39:23.452 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:39:23.452 INFO:tasks.rados.rados.0.smithi107.stdout:12220: seq_num 12219 ranges {0=647168} 2020-06-12T02:39:23.457 INFO:tasks.rados.rados.0.smithi107.stdout:12220: writing smithi10729407-2 from 0 to 647168 tid 1 2020-06-12T02:39:23.457 INFO:tasks.rados.rados.0.smithi107.stdout:12221: delete oid 3 current snap is 0 2020-06-12T02:39:23.457 INFO:tasks.rados.rados.0.smithi107.stdout:12217: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:23.457 INFO:tasks.rados.rados.0.smithi107.stdout:12217: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:23.457 INFO:tasks.rados.rados.0.smithi107.stdout:12217: finishing write tid 4 to smithi10729407-12 2020-06-12T02:39:23.458 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 2276 (ObjNum 12216 snap 0 seq_num 12216) dirty exists 2020-06-12T02:39:23.458 INFO:tasks.rados.rados.0.smithi107.stdout:12217: left oid 12 (ObjNum 12216 snap 0 seq_num 12216) 2020-06-12T02:39:23.459 INFO:tasks.rados.rados.0.smithi107.stdout:12218: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:23.459 INFO:tasks.rados.rados.0.smithi107.stdout:12219: finishing write tid 1 to smithi10729407-14 2020-06-12T02:39:23.459 INFO:tasks.rados.rados.0.smithi107.stdout:12219: finishing write tid 3 to smithi10729407-14 2020-06-12T02:39:23.460 INFO:tasks.rados.rados.0.smithi107.stdout:12219: finishing write tid 4 to smithi10729407-14 2020-06-12T02:39:23.460 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 14 v 1858 (ObjNum 12218 snap 0 seq_num 12218) dirty exists 2020-06-12T02:39:23.461 INFO:tasks.rados.rados.0.smithi107.stdout:12219: left oid 14 (ObjNum 12218 snap 0 seq_num 12218) 2020-06-12T02:39:23.465 INFO:tasks.rados.rados.0.smithi107.stdout:12217: done (4 left) 2020-06-12T02:39:23.465 INFO:tasks.rados.rados.0.smithi107.stdout:12219: done (3 left) 2020-06-12T02:39:23.466 INFO:tasks.rados.rados.0.smithi107.stdout:12221: done (2 left) 2020-06-12T02:39:23.466 INFO:tasks.rados.rados.0.smithi107.stdout:12222: delete oid 15 current snap is 0 2020-06-12T02:39:23.467 INFO:tasks.rados.rados.0.smithi107.stdout:12218: finishing write tid 3 to smithi10729407-1 2020-06-12T02:39:23.467 INFO:tasks.rados.rados.0.smithi107.stdout:12220: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:23.468 INFO:tasks.rados.rados.0.smithi107.stdout:12222: done (2 left) 2020-06-12T02:39:23.469 INFO:tasks.rados.rados.0.smithi107.stdout:12223: delete oid 8 current snap is 0 2020-06-12T02:39:23.469 INFO:tasks.rados.rados.0.smithi107.stdout:12218: finishing write tid 4 to smithi10729407-1 2020-06-12T02:39:23.469 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 2234 (ObjNum 12217 snap 0 seq_num 12217) dirty exists 2020-06-12T02:39:23.469 INFO:tasks.rados.rados.0.smithi107.stdout:12218: left oid 1 (ObjNum 12217 snap 0 seq_num 12217) 2020-06-12T02:39:23.471 INFO:tasks.rados.rados.0.smithi107.stdout:12220: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:23.471 INFO:tasks.rados.rados.0.smithi107.stdout:12218: done (2 left) 2020-06-12T02:39:23.472 INFO:tasks.rados.rados.0.smithi107.stdout:12223: done (1 left) 2020-06-12T02:39:23.472 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:39:23.473 INFO:tasks.rados.rados.0.smithi107.stdout:12224: seq_num 12223 ranges {0=720896} 2020-06-12T02:39:23.476 INFO:tasks.rados.rados.0.smithi107.stdout:12224: writing smithi10729407-11 from 0 to 720896 tid 1 2020-06-12T02:39:23.476 INFO:tasks.rados.rados.0.smithi107.stdout:12225: delete oid 12 current snap is 0 2020-06-12T02:39:23.477 INFO:tasks.rados.rados.0.smithi107.stdout:12220: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:23.477 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 1144 (ObjNum 12219 snap 0 seq_num 12219) dirty exists 2020-06-12T02:39:23.477 INFO:tasks.rados.rados.0.smithi107.stdout:12220: left oid 2 (ObjNum 12219 snap 0 seq_num 12219) 2020-06-12T02:39:23.478 INFO:tasks.rados.rados.0.smithi107.stdout:12220: done (2 left) 2020-06-12T02:39:23.479 INFO:tasks.rados.rados.0.smithi107.stdout:12225: done (1 left) 2020-06-12T02:39:23.479 INFO:tasks.rados.rados.0.smithi107.stdout:12226: delete oid 8 current snap is 0 2020-06-12T02:39:23.481 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd set noscrub 2020-06-12T02:39:23.483 INFO:tasks.rados.rados.0.smithi107.stdout:12226: done (1 left) 2020-06-12T02:39:23.483 INFO:tasks.rados.rados.0.smithi107.stdout:12227: delete oid 10 current snap is 0 2020-06-12T02:39:23.484 INFO:tasks.rados.rados.0.smithi107.stdout:12224: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:23.486 INFO:tasks.rados.rados.0.smithi107.stdout:12227: done (1 left) 2020-06-12T02:39:23.486 INFO:tasks.rados.rados.0.smithi107.stdout:12228: delete oid 7 current snap is 0 2020-06-12T02:39:23.488 INFO:tasks.rados.rados.0.smithi107.stdout:12228: done (1 left) 2020-06-12T02:39:23.489 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 15 current snap is 0 2020-06-12T02:39:23.489 INFO:tasks.rados.rados.0.smithi107.stdout:12229: seq_num 12228 ranges {0=802816} 2020-06-12T02:39:23.494 INFO:tasks.rados.rados.0.smithi107.stdout:12229: writing smithi10729407-15 from 0 to 802816 tid 1 2020-06-12T02:39:23.494 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 20 current snap is 0 2020-06-12T02:39:23.494 INFO:tasks.rados.rados.0.smithi107.stdout:12230: seq_num 12229 ranges {0=417792,417792=401408} 2020-06-12T02:39:23.496 INFO:tasks.rados.rados.0.smithi107.stdout:12230: writing smithi10729407-20 from 0 to 417792 tid 1 2020-06-12T02:39:23.498 INFO:tasks.rados.rados.0.smithi107.stdout:12230: writing smithi10729407-20 from 417792 to 819200 tid 2 2020-06-12T02:39:23.499 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 9 current snap is 0 2020-06-12T02:39:23.499 INFO:tasks.rados.rados.0.smithi107.stdout:12231: seq_num 12230 ranges {0=434176,434176=401408} 2020-06-12T02:39:23.501 INFO:tasks.rados.rados.0.smithi107.stdout:12231: writing smithi10729407-9 from 0 to 434176 tid 1 2020-06-12T02:39:23.505 INFO:tasks.rados.rados.0.smithi107.stdout:12231: writing smithi10729407-9 from 434176 to 835584 tid 2 2020-06-12T02:39:23.506 INFO:tasks.rados.rados.0.smithi107.stdout:12232: delete oid 19 current snap is 0 2020-06-12T02:39:23.506 INFO:tasks.rados.rados.0.smithi107.stdout:12224: finishing write tid 3 to smithi10729407-11 2020-06-12T02:39:23.507 INFO:tasks.rados.rados.0.smithi107.stdout:12224: finishing write tid 4 to smithi10729407-11 2020-06-12T02:39:23.509 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1853 (ObjNum 12223 snap 0 seq_num 12223) dirty exists 2020-06-12T02:39:23.509 INFO:tasks.rados.rados.0.smithi107.stdout:12224: left oid 11 (ObjNum 12223 snap 0 seq_num 12223) 2020-06-12T02:39:23.510 INFO:tasks.rados.rados.0.smithi107.stdout:12230: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:23.511 INFO:tasks.rados.rados.0.smithi107.stdout:12229: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:23.511 INFO:tasks.rados.rados.0.smithi107.stdout:12229: finishing write tid 3 to smithi10729407-15 2020-06-12T02:39:23.511 INFO:tasks.rados.rados.0.smithi107.stdout:12230: finishing write tid 2 to smithi10729407-20 2020-06-12T02:39:23.511 INFO:tasks.rados.rados.0.smithi107.stdout:12230: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:23.512 INFO:tasks.rados.rados.0.smithi107.stdout:12224: done (4 left) 2020-06-12T02:39:23.512 INFO:tasks.rados.rados.0.smithi107.stdout:12232: done (3 left) 2020-06-12T02:39:23.512 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:39:23.513 INFO:tasks.rados.rados.0.smithi107.stdout:12233: seq_num 12232 ranges {647168=466944,1114112=401408} 2020-06-12T02:39:23.513 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:23.513 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:23.515 INFO:tasks.rados.rados.0.smithi107.stdout:12233: writing smithi10729407-2 from 647168 to 1114112 tid 1 2020-06-12T02:39:23.518 INFO:tasks.rados.rados.0.smithi107.stdout:12233: writing smithi10729407-2 from 1114112 to 1515520 tid 2 2020-06-12T02:39:23.518 INFO:tasks.rados.rados.0.smithi107.stdout:12229: finishing write tid 4 to smithi10729407-15 2020-06-12T02:39:23.519 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 2286 (ObjNum 12228 snap 0 seq_num 12228) dirty exists 2020-06-12T02:39:23.519 INFO:tasks.rados.rados.0.smithi107.stdout:12229: left oid 15 (ObjNum 12228 snap 0 seq_num 12228) 2020-06-12T02:39:23.519 INFO:tasks.rados.rados.0.smithi107.stdout:12230: finishing write tid 5 to smithi10729407-20 2020-06-12T02:39:23.519 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 956 (ObjNum 12229 snap 0 seq_num 12229) dirty exists 2020-06-12T02:39:23.519 INFO:tasks.rados.rados.0.smithi107.stdout:12230: left oid 20 (ObjNum 12229 snap 0 seq_num 12229) 2020-06-12T02:39:23.520 INFO:tasks.rados.rados.0.smithi107.stdout:12231: finishing write tid 1 to smithi10729407-9 2020-06-12T02:39:23.520 INFO:tasks.rados.rados.0.smithi107.stdout:12229: done (3 left) 2020-06-12T02:39:23.520 INFO:tasks.rados.rados.0.smithi107.stdout:12230: done (2 left) 2020-06-12T02:39:23.520 INFO:tasks.rados.rados.0.smithi107.stdout:12234: delete oid 6 current snap is 0 2020-06-12T02:39:23.520 INFO:tasks.rados.rados.0.smithi107.stdout:12231: finishing write tid 2 to smithi10729407-9 2020-06-12T02:39:23.521 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-06-12T02:39:23.522 INFO:tasks.rados.rados.0.smithi107.stdout:12231: finishing write tid 4 to smithi10729407-9 2020-06-12T02:39:23.523 INFO:tasks.rados.rados.0.smithi107.stdout:12234: done (2 left) 2020-06-12T02:39:23.523 INFO:tasks.rados.rados.0.smithi107.stdout:12235: delete oid 3 current snap is 0 2020-06-12T02:39:23.526 INFO:tasks.rados.rados.0.smithi107.stdout:12233: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:23.527 INFO:tasks.rados.rados.0.smithi107.stdout:12231: finishing write tid 5 to smithi10729407-9 2020-06-12T02:39:23.527 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 9 v 1968 (ObjNum 12230 snap 0 seq_num 12230) dirty exists 2020-06-12T02:39:23.527 INFO:tasks.rados.rados.0.smithi107.stdout:12231: left oid 9 (ObjNum 12230 snap 0 seq_num 12230) 2020-06-12T02:39:23.535 INFO:tasks.rados.rados.0.smithi107.stdout:12231: done (2 left) 2020-06-12T02:39:23.535 INFO:tasks.rados.rados.0.smithi107.stdout:12235: done (1 left) 2020-06-12T02:39:23.535 INFO:tasks.rados.rados.0.smithi107.stdout:12236: delete oid 14 current snap is 0 2020-06-12T02:39:23.535 INFO:tasks.rados.rados.0.smithi107.stdout:12233: finishing write tid 2 to smithi10729407-2 2020-06-12T02:39:23.536 INFO:tasks.rados.rados.0.smithi107.stdout:12236: done (1 left) 2020-06-12T02:39:23.536 INFO:tasks.rados.rados.0.smithi107.stdout:12237: delete oid 9 current snap is 0 2020-06-12T02:39:23.536 INFO:tasks.rados.rados.0.smithi107.stdout:12233: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:23.536 INFO:tasks.rados.rados.0.smithi107.stdout:12233: finishing write tid 5 to smithi10729407-2 2020-06-12T02:39:23.537 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 1147 (ObjNum 12232 snap 0 seq_num 12232) dirty exists 2020-06-12T02:39:23.537 INFO:tasks.rados.rados.0.smithi107.stdout:12233: left oid 2 (ObjNum 12232 snap 0 seq_num 12232) 2020-06-12T02:39:23.538 INFO:tasks.rados.rados.0.smithi107.stdout:12233: done (1 left) 2020-06-12T02:39:23.538 INFO:tasks.rados.rados.0.smithi107.stdout:12237: done (0 left) 2020-06-12T02:39:23.538 INFO:tasks.rados.rados.0.smithi107.stdout:12238: delete oid 16 current snap is 0 2020-06-12T02:39:23.541 INFO:tasks.rados.rados.0.smithi107.stdout:12238: done (0 left) 2020-06-12T02:39:23.541 INFO:tasks.rados.rados.0.smithi107.stdout:12239: delete oid 2 current snap is 0 2020-06-12T02:39:23.541 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:39:23.538+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:39:23.546 INFO:tasks.rados.rados.0.smithi107.stdout:12239: done (0 left) 2020-06-12T02:39:23.546 INFO:tasks.rados.rados.0.smithi107.stdout:12240: delete oid 10 current snap is 0 2020-06-12T02:39:23.548 INFO:tasks.rados.rados.0.smithi107.stdout:12240: done (0 left) 2020-06-12T02:39:23.548 INFO:tasks.rados.rados.0.smithi107.stdout:12241: delete oid 20 current snap is 0 2020-06-12T02:39:23.550 INFO:tasks.rados.rados.0.smithi107.stdout:12241: done (0 left) 2020-06-12T02:39:23.551 INFO:tasks.rados.rados.0.smithi107.stdout:12242: delete oid 12 current snap is 0 2020-06-12T02:39:23.553 INFO:tasks.rados.rados.0.smithi107.stdout:12242: done (0 left) 2020-06-12T02:39:23.554 INFO:tasks.rados.rados.0.smithi107.stdout:12243: delete oid 6 current snap is 0 2020-06-12T02:39:23.557 INFO:tasks.rados.rados.0.smithi107.stdout:12243: done (0 left) 2020-06-12T02:39:23.557 INFO:tasks.rados.rados.0.smithi107.stdout:12244: delete oid 17 current snap is 0 2020-06-12T02:39:23.560 INFO:tasks.rados.rados.0.smithi107.stdout:12244: done (0 left) 2020-06-12T02:39:23.560 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 13 current snap is 0 2020-06-12T02:39:23.560 INFO:tasks.rados.rados.0.smithi107.stdout:12245: seq_num 12244 ranges {0=671744,671744=401408} 2020-06-12T02:39:23.564 INFO:tasks.rados.rados.0.smithi107.stdout:12245: writing smithi10729407-13 from 0 to 671744 tid 1 2020-06-12T02:39:23.566 INFO:tasks.rados.rados.0.smithi107.stdout:12245: writing smithi10729407-13 from 671744 to 1073152 tid 2 2020-06-12T02:39:23.566 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 5 current snap is 0 2020-06-12T02:39:23.567 INFO:tasks.rados.rados.0.smithi107.stdout:12246: seq_num 12245 ranges {0=688128,688128=401408} 2020-06-12T02:39:23.570 INFO:tasks.rados.rados.0.smithi107.stdout:12246: writing smithi10729407-5 from 0 to 688128 tid 1 2020-06-12T02:39:23.573 INFO:tasks.rados.rados.0.smithi107.stdout:12246: writing smithi10729407-5 from 688128 to 1089536 tid 2 2020-06-12T02:39:23.573 INFO:tasks.rados.rados.0.smithi107.stdout:12247: delete oid 7 current snap is 0 2020-06-12T02:39:23.575 INFO:tasks.rados.rados.0.smithi107.stdout:12245: finishing write tid 1 to smithi10729407-13 2020-06-12T02:39:23.578 INFO:tasks.rados.rados.0.smithi107.stdout:12246: finishing write tid 1 to smithi10729407-5 2020-06-12T02:39:23.578 INFO:tasks.rados.rados.0.smithi107.stdout:12247: done (2 left) 2020-06-12T02:39:23.579 INFO:tasks.rados.rados.0.smithi107.stdout:12248: delete oid 20 current snap is 0 2020-06-12T02:39:23.582 INFO:tasks.rados.rados.0.smithi107.stdout:12248: done (2 left) 2020-06-12T02:39:23.582 INFO:tasks.rados.rados.0.smithi107.stdout:12249: delete oid 1 current snap is 0 2020-06-12T02:39:23.590 INFO:tasks.rados.rados.0.smithi107.stdout:12245: finishing write tid 2 to smithi10729407-13 2020-06-12T02:39:23.590 INFO:tasks.rados.rados.0.smithi107.stdout:12245: finishing write tid 4 to smithi10729407-13 2020-06-12T02:39:23.591 INFO:tasks.rados.rados.0.smithi107.stdout:12246: finishing write tid 2 to smithi10729407-5 2020-06-12T02:39:23.591 INFO:tasks.rados.rados.0.smithi107.stdout:12249: done (2 left) 2020-06-12T02:39:23.591 INFO:tasks.rados.rados.0.smithi107.stdout:12250: delete oid 14 current snap is 0 2020-06-12T02:39:23.592 INFO:tasks.rados.rados.0.smithi107.stdout:12246: finishing write tid 4 to smithi10729407-5 2020-06-12T02:39:23.592 INFO:tasks.rados.rados.0.smithi107.stdout:12245: finishing write tid 5 to smithi10729407-13 2020-06-12T02:39:23.592 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 13 v 1896 (ObjNum 12244 snap 0 seq_num 12244) dirty exists 2020-06-12T02:39:23.593 INFO:tasks.rados.rados.0.smithi107.stdout:12245: left oid 13 (ObjNum 12244 snap 0 seq_num 12244) 2020-06-12T02:39:23.593 INFO:tasks.rados.rados.0.smithi107.stdout:12246: finishing write tid 5 to smithi10729407-5 2020-06-12T02:39:23.593 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 5 v 1151 (ObjNum 12245 snap 0 seq_num 12245) dirty exists 2020-06-12T02:39:23.594 INFO:tasks.rados.rados.0.smithi107.stdout:12246: left oid 5 (ObjNum 12245 snap 0 seq_num 12245) 2020-06-12T02:39:23.594 INFO:tasks.rados.rados.0.smithi107.stdout:12245: done (2 left) 2020-06-12T02:39:23.594 INFO:tasks.rados.rados.0.smithi107.stdout:12246: done (1 left) 2020-06-12T02:39:23.595 INFO:tasks.rados.rados.0.smithi107.stdout:12250: done (0 left) 2020-06-12T02:39:23.595 INFO:tasks.rados.rados.0.smithi107.stdout:12251: delete oid 9 current snap is 0 2020-06-12T02:39:23.596 INFO:tasks.rados.rados.0.smithi107.stdout:12251: done (0 left) 2020-06-12T02:39:23.596 INFO:tasks.rados.rados.0.smithi107.stdout:12252: delete oid 10 current snap is 0 2020-06-12T02:39:23.597 INFO:tasks.rados.rados.0.smithi107.stdout:12252: done (0 left) 2020-06-12T02:39:23.597 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:39:23.597 INFO:tasks.rados.rados.0.smithi107.stdout:12253: seq_num 12252 ranges {0=802816,802816=401408} 2020-06-12T02:39:23.598 INFO:tasks.rados.rados.0.smithi107.stdout:12253: writing smithi10729407-20 from 0 to 802816 tid 1 2020-06-12T02:39:23.602 INFO:tasks.rados.rados.0.smithi107.stdout:12253: writing smithi10729407-20 from 802816 to 1204224 tid 2 2020-06-12T02:39:23.602 INFO:tasks.rados.rados.0.smithi107.stdout:12254: delete oid 15 current snap is 0 2020-06-12T02:39:23.603 INFO:tasks.rados.rados.0.smithi107.stdout:12254: done (1 left) 2020-06-12T02:39:23.603 INFO:tasks.rados.rados.0.smithi107.stdout:12255: delete oid 1 current snap is 0 2020-06-12T02:39:23.605 INFO:tasks.rados.rados.0.smithi107.stdout:12253: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:23.607 INFO:tasks.rados.rados.0.smithi107.stdout:12255: done (1 left) 2020-06-12T02:39:23.608 INFO:tasks.rados.rados.0.smithi107.stdout:12256: delete oid 8 current snap is 0 2020-06-12T02:39:23.611 INFO:tasks.rados.rados.0.smithi107.stdout:12256: done (1 left) 2020-06-12T02:39:23.611 INFO:tasks.rados.rados.0.smithi107.stdout:12257: delete oid 6 current snap is 0 2020-06-12T02:39:23.611 INFO:tasks.rados.rados.0.smithi107.stdout:12253: finishing write tid 2 to smithi10729407-20 2020-06-12T02:39:23.613 INFO:tasks.rados.rados.0.smithi107.stdout:12257: done (1 left) 2020-06-12T02:39:23.614 INFO:tasks.rados.rados.0.smithi107.stdout:12258: delete oid 11 current snap is 0 2020-06-12T02:39:23.615 INFO:tasks.rados.rados.0.smithi107.stdout:12253: finishing write tid 4 to smithi10729407-20 2020-06-12T02:39:23.615 INFO:tasks.rados.rados.0.smithi107.stdout:12258: done (1 left) 2020-06-12T02:39:23.616 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:39:23.616 INFO:tasks.rados.rados.0.smithi107.stdout:12259: seq_num 12258 ranges {0=499712,499712=630784,1130496=172032} 2020-06-12T02:39:23.619 INFO:tasks.rados.rados.0.smithi107.stdout:12259: writing smithi10729407-4 from 0 to 499712 tid 1 2020-06-12T02:39:23.623 INFO:tasks.rados.rados.0.smithi107.stdout:12259: writing smithi10729407-4 from 499712 to 1130496 tid 2 2020-06-12T02:39:23.624 INFO:tasks.rados.rados.0.smithi107.stdout:12259: writing smithi10729407-4 from 1130496 to 1302528 tid 3 2020-06-12T02:39:23.624 INFO:tasks.rados.rados.0.smithi107.stdout:12260: delete oid 5 current snap is 0 2020-06-12T02:39:23.625 INFO:tasks.rados.rados.0.smithi107.stdout:12253: finishing write tid 5 to smithi10729407-20 2020-06-12T02:39:23.625 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 961 (ObjNum 12252 snap 0 seq_num 12252) dirty exists 2020-06-12T02:39:23.625 INFO:tasks.rados.rados.0.smithi107.stdout:12253: left oid 20 (ObjNum 12252 snap 0 seq_num 12252) 2020-06-12T02:39:23.627 INFO:tasks.rados.rados.0.smithi107.stdout:12253: done (2 left) 2020-06-12T02:39:23.627 INFO:tasks.rados.rados.0.smithi107.stdout:12260: done (1 left) 2020-06-12T02:39:23.628 INFO:tasks.rados.rados.0.smithi107.stdout:12261: delete oid 12 current snap is 0 2020-06-12T02:39:23.628 INFO:tasks.rados.rados.0.smithi107.stdout:12259: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:23.630 INFO:tasks.rados.rados.0.smithi107.stdout:12261: done (1 left) 2020-06-12T02:39:23.630 INFO:tasks.rados.rados.0.smithi107.stdout:12262: delete oid 5 current snap is 0 2020-06-12T02:39:23.631 INFO:tasks.rados.rados.0.smithi107.stdout:12259: finishing write tid 2 to smithi10729407-4 2020-06-12T02:39:23.632 INFO:tasks.rados.rados.0.smithi107.stdout:12259: finishing write tid 3 to smithi10729407-4 2020-06-12T02:39:23.633 INFO:tasks.rados.rados.0.smithi107.stdout:12262: done (1 left) 2020-06-12T02:39:23.633 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 20 current snap is 0 2020-06-12T02:39:23.633 INFO:tasks.rados.rados.0.smithi107.stdout:12263: seq_num 12262 ranges {1204224=573440,1777664=565248,2342912=237568} 2020-06-12T02:39:23.643 INFO:tasks.rados.rados.0.smithi107.stdout:12263: writing smithi10729407-20 from 1204224 to 1777664 tid 1 2020-06-12T02:39:23.644 INFO:tasks.ceph.osd.5.smithi188.stderr:2020-06-12T02:39:23.641+0000 7f4646242700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26674) UID: 0 2020-06-12T02:39:23.646 INFO:tasks.rados.rados.0.smithi107.stdout:12263: writing smithi10729407-20 from 1777664 to 2342912 tid 2 2020-06-12T02:39:23.647 INFO:tasks.rados.rados.0.smithi107.stdout:12263: writing smithi10729407-20 from 2342912 to 2580480 tid 3 2020-06-12T02:39:23.648 INFO:tasks.rados.rados.0.smithi107.stdout:12264: delete oid 15 current snap is 0 2020-06-12T02:39:23.648 INFO:tasks.rados.rados.0.smithi107.stdout:12259: finishing write tid 5 to smithi10729407-4 2020-06-12T02:39:23.648 INFO:tasks.rados.rados.0.smithi107.stdout:12259: finishing write tid 6 to smithi10729407-4 2020-06-12T02:39:23.648 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 2267 (ObjNum 12258 snap 0 seq_num 12258) dirty exists 2020-06-12T02:39:23.649 INFO:tasks.rados.rados.0.smithi107.stdout:12259: left oid 4 (ObjNum 12258 snap 0 seq_num 12258) 2020-06-12T02:39:23.649 INFO:tasks.rados.rados.0.smithi107.stdout:12263: finishing write tid 1 to smithi10729407-20 2020-06-12T02:39:23.652 INFO:tasks.rados.rados.0.smithi107.stdout:12259: done (2 left) 2020-06-12T02:39:23.652 INFO:tasks.rados.rados.0.smithi107.stdout:12264: done (1 left) 2020-06-12T02:39:23.653 INFO:tasks.rados.rados.0.smithi107.stdout:12265: delete oid 1 current snap is 0 2020-06-12T02:39:23.655 INFO:tasks.rados.rados.0.smithi107.stdout:12265: done (1 left) 2020-06-12T02:39:23.655 INFO:tasks.rados.rados.0.smithi107.stdout:12266: delete oid 18 current snap is 0 2020-06-12T02:39:23.658 INFO:tasks.rados.rados.0.smithi107.stdout:12263: finishing write tid 2 to smithi10729407-20 2020-06-12T02:39:23.659 INFO:tasks.rados.rados.0.smithi107.stdout:12266: done (1 left) 2020-06-12T02:39:23.659 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 15 current snap is 0 2020-06-12T02:39:23.659 INFO:tasks.rados.rados.0.smithi107.stdout:12267: seq_num 12266 ranges {0=638976,638976=557056,1196032=245760} 2020-06-12T02:39:23.664 INFO:tasks.rados.rados.0.smithi107.stdout:12267: writing smithi10729407-15 from 0 to 638976 tid 1 2020-06-12T02:39:23.667 INFO:tasks.rados.rados.0.smithi107.stdout:12267: writing smithi10729407-15 from 638976 to 1196032 tid 2 2020-06-12T02:39:23.669 INFO:tasks.rados.rados.0.smithi107.stdout:12267: writing smithi10729407-15 from 1196032 to 1441792 tid 3 2020-06-12T02:39:23.669 INFO:tasks.rados.rados.0.smithi107.stdout:12268: delete oid 3 current snap is 0 2020-06-12T02:39:23.669 INFO:tasks.rados.rados.0.smithi107.stdout:12263: finishing write tid 3 to smithi10729407-20 2020-06-12T02:39:23.669 INFO:tasks.rados.rados.0.smithi107.stdout:12263: finishing write tid 5 to smithi10729407-20 2020-06-12T02:39:23.670 INFO:tasks.rados.rados.0.smithi107.stdout:12263: finishing write tid 6 to smithi10729407-20 2020-06-12T02:39:23.670 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 20 v 965 (ObjNum 12262 snap 0 seq_num 12262) dirty exists 2020-06-12T02:39:23.670 INFO:tasks.rados.rados.0.smithi107.stdout:12263: left oid 20 (ObjNum 12262 snap 0 seq_num 12262) 2020-06-12T02:39:23.670 INFO:tasks.rados.rados.0.smithi107.stdout:12267: finishing write tid 1 to smithi10729407-15 2020-06-12T02:39:23.674 INFO:tasks.rados.rados.0.smithi107.stdout:12263: done (2 left) 2020-06-12T02:39:23.674 INFO:tasks.rados.rados.0.smithi107.stdout:12268: done (1 left) 2020-06-12T02:39:23.674 INFO:tasks.rados.rados.0.smithi107.stdout:12269: delete oid 13 current snap is 0 2020-06-12T02:39:23.676 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-06-12T02:39:23.676 INFO:tasks.rados.rados.0.smithi107.stdout:12269: done (1 left) 2020-06-12T02:39:23.677 INFO:tasks.rados.rados.0.smithi107.stdout:12270: delete oid 5 current snap is 0 2020-06-12T02:39:23.681 INFO:tasks.rados.rados.0.smithi107.stdout:12270: done (1 left) 2020-06-12T02:39:23.681 INFO:tasks.rados.rados.0.smithi107.stdout:12271: delete oid 19 current snap is 0 2020-06-12T02:39:23.682 INFO:tasks.rados.rados.0.smithi107.stdout:12267: finishing write tid 2 to smithi10729407-15 2020-06-12T02:39:23.682 INFO:tasks.rados.rados.0.smithi107.stdout:12267: finishing write tid 3 to smithi10729407-15 2020-06-12T02:39:23.682 INFO:tasks.rados.rados.0.smithi107.stdout:12267: finishing write tid 5 to smithi10729407-15 2020-06-12T02:39:23.683 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:23.683 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:23.684 INFO:tasks.rados.rados.0.smithi107.stdout:12271: done (1 left) 2020-06-12T02:39:23.684 INFO:tasks.rados.rados.0.smithi107.stdout:12272: delete oid 19 current snap is 0 2020-06-12T02:39:23.686 INFO:tasks.rados.rados.0.smithi107.stdout:12267: finishing write tid 6 to smithi10729407-15 2020-06-12T02:39:23.687 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 15 v 2293 (ObjNum 12266 snap 0 seq_num 12266) dirty exists 2020-06-12T02:39:23.687 INFO:tasks.rados.rados.0.smithi107.stdout:12267: left oid 15 (ObjNum 12266 snap 0 seq_num 12266) 2020-06-12T02:39:23.687 INFO:tasks.rados.rados.0.smithi107.stdout:12267: done (1 left) 2020-06-12T02:39:23.687 INFO:tasks.rados.rados.0.smithi107.stdout:12272: done (0 left) 2020-06-12T02:39:23.688 INFO:tasks.rados.rados.0.smithi107.stdout:12273: delete oid 12 current snap is 0 2020-06-12T02:39:23.691 INFO:tasks.rados.rados.0.smithi107.stdout:12273: done (0 left) 2020-06-12T02:39:23.691 INFO:tasks.rados.rados.0.smithi107.stdout:12274: delete oid 13 current snap is 0 2020-06-12T02:39:23.697 INFO:tasks.rados.rados.0.smithi107.stdout:12274: done (0 left) 2020-06-12T02:39:23.697 INFO:tasks.rados.rados.0.smithi107.stdout:12275: delete oid 13 current snap is 0 2020-06-12T02:39:23.697 INFO:tasks.rados.rados.0.smithi107.stdout:12275: done (0 left) 2020-06-12T02:39:23.697 INFO:tasks.rados.rados.0.smithi107.stdout:12276: delete oid 17 current snap is 0 2020-06-12T02:39:23.698 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-06-12T02:39:23.699 INFO:tasks.rados.rados.0.smithi107.stdout:12276: done (0 left) 2020-06-12T02:39:23.699 INFO:tasks.rados.rados.0.smithi107.stdout:12277: delete oid 7 current snap is 0 2020-06-12T02:39:23.701 INFO:tasks.rados.rados.0.smithi107.stdout:12277: done (0 left) 2020-06-12T02:39:23.701 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:39:23.701 INFO:tasks.rados.rados.0.smithi107.stdout:12278: seq_num 12277 ranges {0=417792,417792=532480,950272=671744} 2020-06-12T02:39:23.703 INFO:tasks.rados.rados.0.smithi107.stdout:12278: writing smithi10729407-12 from 0 to 417792 tid 1 2020-06-12T02:39:23.706 INFO:tasks.rados.rados.0.smithi107.stdout:12278: writing smithi10729407-12 from 417792 to 950272 tid 2 2020-06-12T02:39:23.711 INFO:tasks.rados.rados.0.smithi107.stdout:12278: writing smithi10729407-12 from 950272 to 1622016 tid 3 2020-06-12T02:39:23.711 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:39:23.711 INFO:tasks.rados.rados.0.smithi107.stdout:12279: seq_num 12278 ranges {0=434176,434176=466944,901120=581632,1482752=155648} 2020-06-12T02:39:23.713 INFO:tasks.rados.rados.0.smithi107.stdout:12279: writing smithi10729407-7 from 0 to 434176 tid 1 2020-06-12T02:39:23.718 INFO:tasks.rados.rados.0.smithi107.stdout:12279: writing smithi10729407-7 from 434176 to 901120 tid 2 2020-06-12T02:39:23.720 INFO:tasks.rados.rados.0.smithi107.stdout:12279: writing smithi10729407-7 from 901120 to 1482752 tid 3 2020-06-12T02:39:23.721 INFO:tasks.rados.rados.0.smithi107.stdout:12279: writing smithi10729407-7 from 1482752 to 1638400 tid 4 2020-06-12T02:39:23.721 INFO:tasks.rados.rados.0.smithi107.stdout:12280: delete oid 17 current snap is 0 2020-06-12T02:39:23.721 INFO:tasks.rados.rados.0.smithi107.stdout:12278: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:23.721 INFO:tasks.rados.rados.0.smithi107.stdout:12278: finishing write tid 2 to smithi10729407-12 2020-06-12T02:39:23.723 INFO:tasks.rados.rados.0.smithi107.stdout:12278: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:23.724 INFO:tasks.rados.rados.0.smithi107.stdout:12278: finishing write tid 5 to smithi10729407-12 2020-06-12T02:39:23.725 INFO:tasks.rados.rados.0.smithi107.stdout:12280: done (2 left) 2020-06-12T02:39:23.725 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 17 current snap is 0 2020-06-12T02:39:23.725 INFO:tasks.rados.rados.0.smithi107.stdout:12281: seq_num 12280 ranges {0=475136,475136=630784,1105920=548864,1654784=24576} 2020-06-12T02:39:23.727 INFO:tasks.rados.rados.0.smithi107.stdout:12281: writing smithi10729407-17 from 0 to 475136 tid 1 2020-06-12T02:39:23.731 INFO:tasks.rados.rados.0.smithi107.stdout:12281: writing smithi10729407-17 from 475136 to 1105920 tid 2 2020-06-12T02:39:23.735 INFO:tasks.rados.rados.0.smithi107.stdout:12281: writing smithi10729407-17 from 1105920 to 1654784 tid 3 2020-06-12T02:39:23.736 INFO:tasks.rados.rados.0.smithi107.stdout:12281: writing smithi10729407-17 from 1654784 to 1679360 tid 4 2020-06-12T02:39:23.736 INFO:tasks.rados.rados.0.smithi107.stdout:12282: delete oid 19 current snap is 0 2020-06-12T02:39:23.737 INFO:tasks.rados.rados.0.smithi107.stdout:12279: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:23.737 INFO:tasks.rados.rados.0.smithi107.stdout:12279: finishing write tid 2 to smithi10729407-7 2020-06-12T02:39:23.738 INFO:tasks.rados.rados.0.smithi107.stdout:12279: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:23.738 INFO:tasks.rados.rados.0.smithi107.stdout:12278: finishing write tid 6 to smithi10729407-12 2020-06-12T02:39:23.739 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 2287 (ObjNum 12277 snap 0 seq_num 12277) dirty exists 2020-06-12T02:39:23.739 INFO:tasks.rados.rados.0.smithi107.stdout:12278: left oid 12 (ObjNum 12277 snap 0 seq_num 12277) 2020-06-12T02:39:23.739 INFO:tasks.rados.rados.0.smithi107.stdout:12279: finishing write tid 4 to smithi10729407-7 2020-06-12T02:39:23.740 INFO:tasks.rados.rados.0.smithi107.stdout:12279: finishing write tid 6 to smithi10729407-7 2020-06-12T02:39:23.740 INFO:tasks.rados.rados.0.smithi107.stdout:12279: finishing write tid 7 to smithi10729407-7 2020-06-12T02:39:23.740 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1930 (ObjNum 12278 snap 0 seq_num 12278) dirty exists 2020-06-12T02:39:23.741 INFO:tasks.rados.rados.0.smithi107.stdout:12279: left oid 7 (ObjNum 12278 snap 0 seq_num 12278) 2020-06-12T02:39:23.741 INFO:tasks.rados.rados.0.smithi107.stdout:12281: finishing write tid 1 to smithi10729407-17 2020-06-12T02:39:23.742 INFO:tasks.rados.rados.0.smithi107.stdout:12278: done (3 left) 2020-06-12T02:39:23.742 INFO:tasks.rados.rados.0.smithi107.stdout:12279: done (2 left) 2020-06-12T02:39:23.742 INFO:tasks.rados.rados.0.smithi107.stdout:12282: done (1 left) 2020-06-12T02:39:23.743 INFO:tasks.rados.rados.0.smithi107.stdout:12283: delete oid 12 current snap is 0 2020-06-12T02:39:23.743 INFO:tasks.rados.rados.0.smithi107.stdout:12283: done (1 left) 2020-06-12T02:39:23.743 INFO:tasks.rados.rados.0.smithi107.stdout:12284: delete oid 5 current snap is 0 2020-06-12T02:39:23.746 INFO:tasks.ceph.osd.2.smithi107.stderr:2020-06-12T02:39:23.743+0000 7f238f67b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26607) UID: 0 2020-06-12T02:39:23.746 INFO:tasks.rados.rados.0.smithi107.stdout:12281: finishing write tid 2 to smithi10729407-17 2020-06-12T02:39:23.748 INFO:tasks.rados.rados.0.smithi107.stdout:12281: finishing write tid 3 to smithi10729407-17 2020-06-12T02:39:23.748 INFO:tasks.rados.rados.0.smithi107.stdout:12281: finishing write tid 4 to smithi10729407-17 2020-06-12T02:39:23.749 INFO:tasks.rados.rados.0.smithi107.stdout:12281: finishing write tid 6 to smithi10729407-17 2020-06-12T02:39:23.750 INFO:tasks.rados.rados.0.smithi107.stdout:12284: done (1 left) 2020-06-12T02:39:23.750 INFO:tasks.rados.rados.0.smithi107.stdout:12285: delete oid 11 current snap is 0 2020-06-12T02:39:23.751 INFO:tasks.rados.rados.0.smithi107.stdout:12285: done (1 left) 2020-06-12T02:39:23.752 INFO:tasks.rados.rados.0.smithi107.stdout:12286: delete oid 14 current snap is 0 2020-06-12T02:39:23.752 INFO:tasks.rados.rados.0.smithi107.stdout:12281: finishing write tid 7 to smithi10729407-17 2020-06-12T02:39:23.752 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 17 v 2274 (ObjNum 12280 snap 0 seq_num 12280) dirty exists 2020-06-12T02:39:23.753 INFO:tasks.rados.rados.0.smithi107.stdout:12281: left oid 17 (ObjNum 12280 snap 0 seq_num 12280) 2020-06-12T02:39:23.754 INFO:tasks.rados.rados.0.smithi107.stdout:12281: done (1 left) 2020-06-12T02:39:23.754 INFO:tasks.rados.rados.0.smithi107.stdout:12286: done (0 left) 2020-06-12T02:39:23.754 INFO:tasks.rados.rados.0.smithi107.stdout:12287: delete oid 19 current snap is 0 2020-06-12T02:39:23.757 INFO:tasks.rados.rados.0.smithi107.stdout:12287: done (0 left) 2020-06-12T02:39:23.757 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 3 current snap is 0 2020-06-12T02:39:23.758 INFO:tasks.rados.rados.0.smithi107.stdout:12288: seq_num 12287 ranges {0=589824,589824=671744,1261568=532480} 2020-06-12T02:39:23.760 INFO:tasks.rados.rados.0.smithi107.stdout:12288: writing smithi10729407-3 from 0 to 589824 tid 1 2020-06-12T02:39:23.764 INFO:tasks.rados.rados.0.smithi107.stdout:12288: writing smithi10729407-3 from 589824 to 1261568 tid 2 2020-06-12T02:39:23.770 INFO:tasks.rados.rados.0.smithi107.stdout:12288: writing smithi10729407-3 from 1261568 to 1794048 tid 3 2020-06-12T02:39:23.770 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 1 current snap is 0 2020-06-12T02:39:23.770 INFO:tasks.rados.rados.0.smithi107.stdout:12289: seq_num 12288 ranges {0=606208,606208=557056,1163264=507904,1671168=139264} 2020-06-12T02:39:23.771 INFO:tasks.rados.rados.0.smithi107.stdout:12289: writing smithi10729407-1 from 0 to 606208 tid 1 2020-06-12T02:39:23.775 INFO:tasks.rados.rados.0.smithi107.stdout:12289: writing smithi10729407-1 from 606208 to 1163264 tid 2 2020-06-12T02:39:23.779 INFO:tasks.rados.rados.0.smithi107.stdout:12289: writing smithi10729407-1 from 1163264 to 1671168 tid 3 2020-06-12T02:39:23.779 INFO:tasks.rados.rados.0.smithi107.stdout:12289: writing smithi10729407-1 from 1671168 to 1810432 tid 4 2020-06-12T02:39:23.780 INFO:tasks.rados.rados.0.smithi107.stdout:12290: delete oid 17 current snap is 0 2020-06-12T02:39:23.780 INFO:tasks.rados.rados.0.smithi107.stdout:12288: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:23.781 INFO:tasks.rados.rados.0.smithi107.stdout:12289: finishing write tid 1 to smithi10729407-1 2020-06-12T02:39:23.786 INFO:tasks.rados.rados.0.smithi107.stdout:12290: done (2 left) 2020-06-12T02:39:23.786 INFO:tasks.rados.rados.0.smithi107.stdout:12291: delete oid 6 current snap is 0 2020-06-12T02:39:23.786 INFO:tasks.rados.rados.0.smithi107.stdout:12288: finishing write tid 2 to smithi10729407-3 2020-06-12T02:39:23.786 INFO:tasks.rados.rados.0.smithi107.stdout:12288: finishing write tid 3 to smithi10729407-3 2020-06-12T02:39:23.787 INFO:tasks.rados.rados.0.smithi107.stdout:12288: finishing write tid 5 to smithi10729407-3 2020-06-12T02:39:23.791 INFO:tasks.rados.rados.0.smithi107.stdout:12288: finishing write tid 6 to smithi10729407-3 2020-06-12T02:39:23.792 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 2163 (ObjNum 12287 snap 0 seq_num 12287) dirty exists 2020-06-12T02:39:23.792 INFO:tasks.rados.rados.0.smithi107.stdout:12288: left oid 3 (ObjNum 12287 snap 0 seq_num 12287) 2020-06-12T02:39:23.796 INFO:tasks.rados.rados.0.smithi107.stdout:12289: finishing write tid 2 to smithi10729407-1 2020-06-12T02:39:23.796 INFO:tasks.rados.rados.0.smithi107.stdout:12289: finishing write tid 3 to smithi10729407-1 2020-06-12T02:39:23.796 INFO:tasks.rados.rados.0.smithi107.stdout:12289: finishing write tid 4 to smithi10729407-1 2020-06-12T02:39:23.796 INFO:tasks.rados.rados.0.smithi107.stdout:12289: finishing write tid 6 to smithi10729407-1 2020-06-12T02:39:23.800 INFO:tasks.rados.rados.0.smithi107.stdout:12288: done (2 left) 2020-06-12T02:39:23.800 INFO:tasks.rados.rados.0.smithi107.stdout:12291: done (1 left) 2020-06-12T02:39:23.801 INFO:tasks.rados.rados.0.smithi107.stdout:12292: delete oid 5 current snap is 0 2020-06-12T02:39:23.801 INFO:tasks.rados.rados.0.smithi107.stdout:12289: finishing write tid 7 to smithi10729407-1 2020-06-12T02:39:23.801 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 1 v 2245 (ObjNum 12288 snap 0 seq_num 12288) dirty exists 2020-06-12T02:39:23.801 INFO:tasks.rados.rados.0.smithi107.stdout:12289: left oid 1 (ObjNum 12288 snap 0 seq_num 12288) 2020-06-12T02:39:23.803 INFO:tasks.rados.rados.0.smithi107.stdout:12289: done (1 left) 2020-06-12T02:39:23.803 INFO:tasks.rados.rados.0.smithi107.stdout:12292: done (0 left) 2020-06-12T02:39:23.803 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 10 current snap is 0 2020-06-12T02:39:23.803 INFO:tasks.rados.rados.0.smithi107.stdout:12293: seq_num 12292 ranges {0=671744,671744=499712,1171456=491520,1662976=212992} 2020-06-12T02:39:23.807 INFO:tasks.rados.rados.0.smithi107.stdout:12293: writing smithi10729407-10 from 0 to 671744 tid 1 2020-06-12T02:39:23.811 INFO:tasks.rados.rados.0.smithi107.stdout:12293: writing smithi10729407-10 from 671744 to 1171456 tid 2 2020-06-12T02:39:23.815 INFO:tasks.rados.rados.0.smithi107.stdout:12293: writing smithi10729407-10 from 1171456 to 1662976 tid 3 2020-06-12T02:39:23.817 INFO:tasks.rados.rados.0.smithi107.stdout:12293: writing smithi10729407-10 from 1662976 to 1875968 tid 4 2020-06-12T02:39:23.817 INFO:tasks.rados.rados.0.smithi107.stdout:12293: finishing write tid 1 to smithi10729407-10 2020-06-12T02:39:23.817 INFO:tasks.rados.rados.0.smithi107.stdout:12294: delete oid 11 current snap is 0 2020-06-12T02:39:23.820 INFO:tasks.rados.rados.0.smithi107.stdout:12294: done (1 left) 2020-06-12T02:39:23.820 INFO:tasks.rados.rados.0.smithi107.stdout:12295: delete oid 18 current snap is 0 2020-06-12T02:39:23.830 INFO:tasks.rados.rados.0.smithi107.stdout:12293: finishing write tid 2 to smithi10729407-10 2020-06-12T02:39:23.831 INFO:tasks.rados.rados.0.smithi107.stdout:12295: done (1 left) 2020-06-12T02:39:23.831 INFO:tasks.rados.rados.0.smithi107.stdout:12296: delete oid 3 current snap is 0 2020-06-12T02:39:23.831 INFO:tasks.rados.rados.0.smithi107.stdout:12293: finishing write tid 3 to smithi10729407-10 2020-06-12T02:39:23.832 INFO:tasks.rados.rados.0.smithi107.stdout:12296: done (1 left) 2020-06-12T02:39:23.832 INFO:tasks.rados.rados.0.smithi107.stdout:12297: delete oid 15 current snap is 0 2020-06-12T02:39:23.833 INFO:tasks.rados.rados.0.smithi107.stdout:12293: finishing write tid 4 to smithi10729407-10 2020-06-12T02:39:23.833 INFO:tasks.rados.rados.0.smithi107.stdout:12293: finishing write tid 6 to smithi10729407-10 2020-06-12T02:39:23.835 INFO:tasks.rados.rados.0.smithi107.stdout:12293: finishing write tid 7 to smithi10729407-10 2020-06-12T02:39:23.835 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 10 v 941 (ObjNum 12292 snap 0 seq_num 12292) dirty exists 2020-06-12T02:39:23.836 INFO:tasks.rados.rados.0.smithi107.stdout:12293: left oid 10 (ObjNum 12292 snap 0 seq_num 12292) 2020-06-12T02:39:23.836 INFO:tasks.rados.rados.0.smithi107.stdout:12293: done (1 left) 2020-06-12T02:39:23.837 INFO:tasks.rados.rados.0.smithi107.stdout:12297: done (0 left) 2020-06-12T02:39:23.837 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 12 current snap is 0 2020-06-12T02:39:23.837 INFO:tasks.rados.rados.0.smithi107.stdout:12298: seq_num 12297 ranges {0=753664,753664=770048,1523712=434176} 2020-06-12T02:39:23.845 INFO:tasks.rados.rados.0.smithi107.stdout:12298: writing smithi10729407-12 from 0 to 753664 tid 1 2020-06-12T02:39:23.848 INFO:tasks.rados.rados.0.smithi107.stdout:12298: writing smithi10729407-12 from 753664 to 1523712 tid 2 2020-06-12T02:39:23.849 INFO:tasks.ceph.osd.0.smithi107.stderr:2020-06-12T02:39:23.845+0000 7fa06edbb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26602) UID: 0 2020-06-12T02:39:23.852 INFO:tasks.rados.rados.0.smithi107.stdout:12298: writing smithi10729407-12 from 1523712 to 1957888 tid 3 2020-06-12T02:39:23.852 INFO:tasks.rados.rados.0.smithi107.stdout:12299: delete oid 7 current snap is 0 2020-06-12T02:39:23.852 INFO:tasks.rados.rados.0.smithi107.stdout:12298: finishing write tid 1 to smithi10729407-12 2020-06-12T02:39:23.853 INFO:tasks.rados.rados.0.smithi107.stdout:12299: done (1 left) 2020-06-12T02:39:23.854 INFO:tasks.rados.rados.0.smithi107.stdout:12300: delete oid 11 current snap is 0 2020-06-12T02:39:23.856 INFO:tasks.rados.rados.0.smithi107.stdout:12300: done (1 left) 2020-06-12T02:39:23.856 INFO:tasks.rados.rados.0.smithi107.stdout:12301: delete oid 15 current snap is 0 2020-06-12T02:39:23.862 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2020-06-12T02:39:23.862 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-06-12T02:39:23.863 INFO:tasks.rados.rados.0.smithi107.stdout:12301: done (1 left) 2020-06-12T02:39:23.863 INFO:tasks.rados.rados.0.smithi107.stdout:12302: delete oid 3 current snap is 0 2020-06-12T02:39:23.863 INFO:tasks.rados.rados.0.smithi107.stdout:12298: finishing write tid 2 to smithi10729407-12 2020-06-12T02:39:23.864 INFO:tasks.rados.rados.0.smithi107.stdout:12298: finishing write tid 3 to smithi10729407-12 2020-06-12T02:39:23.864 INFO:tasks.rados.rados.0.smithi107.stdout:12298: finishing write tid 5 to smithi10729407-12 2020-06-12T02:39:23.865 INFO:tasks.rados.rados.0.smithi107.stdout:12298: finishing write tid 6 to smithi10729407-12 2020-06-12T02:39:23.866 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 12 v 2294 (ObjNum 12297 snap 0 seq_num 12297) dirty exists 2020-06-12T02:39:23.866 INFO:tasks.rados.rados.0.smithi107.stdout:12298: left oid 12 (ObjNum 12297 snap 0 seq_num 12297) 2020-06-12T02:39:23.866 INFO:tasks.rados.rados.0.smithi107.stdout:12298: done (1 left) 2020-06-12T02:39:23.866 INFO:tasks.rados.rados.0.smithi107.stdout:12302: done (0 left) 2020-06-12T02:39:23.866 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:39:23.867 INFO:tasks.rados.rados.0.smithi107.stdout:12303: seq_num 12302 ranges {0=442368,442368=638976,1081344=466944,1548288=499712} 2020-06-12T02:39:23.867 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2020-06-12T02:39:23.868 INFO:tasks.rados.rados.0.smithi107.stdout:12303: writing smithi10729407-11 from 0 to 442368 tid 1 2020-06-12T02:39:23.871 INFO:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-06-12T02:39:23.873 INFO:tasks.rados.rados.0.smithi107.stdout:12303: writing smithi10729407-11 from 442368 to 1081344 tid 2 2020-06-12T02:39:23.875 INFO:tasks.rados.rados.0.smithi107.stdout:12303: writing smithi10729407-11 from 1081344 to 1548288 tid 3 2020-06-12T02:39:23.879 INFO:tasks.rados.rados.0.smithi107.stdout:12303: writing smithi10729407-11 from 1548288 to 2048000 tid 4 2020-06-12T02:39:23.880 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 7 current snap is 0 2020-06-12T02:39:23.880 INFO:tasks.rados.rados.0.smithi107.stdout:12304: seq_num 12303 ranges {0=458752,458752=524288,983040=671744,1654784=409600} 2020-06-12T02:39:23.883 INFO:tasks.rados.rados.0.smithi107.stdout:12304: writing smithi10729407-7 from 0 to 458752 tid 1 2020-06-12T02:39:23.886 INFO:tasks.rados.rados.0.smithi107.stdout:12304: writing smithi10729407-7 from 458752 to 983040 tid 2 2020-06-12T02:39:23.890 INFO:tasks.rados.rados.0.smithi107.stdout:12304: writing smithi10729407-7 from 983040 to 1654784 tid 3 2020-06-12T02:39:23.893 INFO:tasks.rados.rados.0.smithi107.stdout:12304: writing smithi10729407-7 from 1654784 to 2064384 tid 4 2020-06-12T02:39:23.894 INFO:tasks.rados.rados.0.smithi107.stdout:12305: delete oid 20 current snap is 0 2020-06-12T02:39:23.894 INFO:tasks.rados.rados.0.smithi107.stdout:12303: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:23.894 INFO:tasks.rados.rados.0.smithi107.stdout:12303: finishing write tid 2 to smithi10729407-11 2020-06-12T02:39:23.894 INFO:tasks.rados.rados.0.smithi107.stdout:12303: finishing write tid 3 to smithi10729407-11 2020-06-12T02:39:23.895 INFO:tasks.rados.rados.0.smithi107.stdout:12303: finishing write tid 4 to smithi10729407-11 2020-06-12T02:39:23.895 INFO:tasks.rados.rados.0.smithi107.stdout:12303: finishing write tid 6 to smithi10729407-11 2020-06-12T02:39:23.895 INFO:tasks.rados.rados.0.smithi107.stdout:12303: finishing write tid 7 to smithi10729407-11 2020-06-12T02:39:23.895 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1862 (ObjNum 12302 snap 0 seq_num 12302) dirty exists 2020-06-12T02:39:23.896 INFO:tasks.rados.rados.0.smithi107.stdout:12303: left oid 11 (ObjNum 12302 snap 0 seq_num 12302) 2020-06-12T02:39:23.896 INFO:tasks.rados.rados.0.smithi107.stdout:12304: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:23.897 INFO:tasks.rados.rados.0.smithi107.stdout:12303: done (2 left) 2020-06-12T02:39:23.897 INFO:tasks.rados.rados.0.smithi107.stdout:12305: done (1 left) 2020-06-12T02:39:23.897 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 3 current snap is 0 2020-06-12T02:39:23.897 INFO:tasks.rados.rados.0.smithi107.stdout:12306: seq_num 12305 ranges {0=491520,491520=696320,1187840=638976,1826816=270336} 2020-06-12T02:39:23.898 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2020-06-12T02:39:23.898 INFO:tasks.rados.rados.0.smithi107.stdout:12306: writing smithi10729407-3 from 0 to 491520 tid 1 2020-06-12T02:39:23.902 INFO:tasks.rados.rados.0.smithi107.stdout:12306: writing smithi10729407-3 from 491520 to 1187840 tid 2 2020-06-12T02:39:23.906 INFO:tasks.rados.rados.0.smithi107.stdout:12306: writing smithi10729407-3 from 1187840 to 1826816 tid 3 2020-06-12T02:39:23.909 INFO:tasks.rados.rados.0.smithi107.stdout:12306: writing smithi10729407-3 from 1826816 to 2097152 tid 4 2020-06-12T02:39:23.909 INFO:tasks.rados.rados.0.smithi107.stdout:12307: delete oid 17 current snap is 0 2020-06-12T02:39:23.909 INFO:tasks.rados.rados.0.smithi107.stdout:12304: finishing write tid 2 to smithi10729407-7 2020-06-12T02:39:23.910 INFO:tasks.rados.rados.0.smithi107.stdout:12304: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:23.910 INFO:tasks.rados.rados.0.smithi107.stdout:12304: finishing write tid 4 to smithi10729407-7 2020-06-12T02:39:23.910 INFO:tasks.rados.rados.0.smithi107.stdout:12304: finishing write tid 6 to smithi10729407-7 2020-06-12T02:39:23.911 INFO:tasks.rados.rados.0.smithi107.stdout:12306: finishing write tid 1 to smithi10729407-3 2020-06-12T02:39:23.911 INFO:tasks.rados.rados.0.smithi107.stdout:12304: finishing write tid 7 to smithi10729407-7 2020-06-12T02:39:23.911 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 7 v 1936 (ObjNum 12303 snap 0 seq_num 12303) dirty exists 2020-06-12T02:39:23.911 INFO:tasks.rados.rados.0.smithi107.stdout:12304: left oid 7 (ObjNum 12303 snap 0 seq_num 12303) 2020-06-12T02:39:23.913 INFO:tasks.rados.rados.0.smithi107.stdout:12304: done (2 left) 2020-06-12T02:39:23.913 INFO:tasks.rados.rados.0.smithi107.stdout:12307: done (1 left) 2020-06-12T02:39:23.913 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 11 current snap is 0 2020-06-12T02:39:23.914 INFO:tasks.rados.rados.0.smithi107.stdout:12308: seq_num 12307 ranges {2048000=524288,2572288=458752,3031040=655360,3686400=491520} 2020-06-12T02:39:23.927 INFO:tasks.rados.rados.0.smithi107.stdout:12308: writing smithi10729407-11 from 2048000 to 2572288 tid 1 2020-06-12T02:39:23.928 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2020-06-12T02:39:23.934 INFO:tasks.rados.rados.0.smithi107.stdout:12308: writing smithi10729407-11 from 2572288 to 3031040 tid 2 2020-06-12T02:39:23.934 INFO:tasks.rados.rados.0.smithi107.stdout:12308: writing smithi10729407-11 from 3031040 to 3686400 tid 3 2020-06-12T02:39:23.937 INFO:tasks.rados.rados.0.smithi107.stdout:12308: writing smithi10729407-11 from 3686400 to 4177920 tid 4 2020-06-12T02:39:23.938 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 18 current snap is 0 2020-06-12T02:39:23.938 INFO:tasks.rados.rados.0.smithi107.stdout:12309: seq_num 12308 ranges {0=540672,540672=745472,1286144=409600,1695744=450560} 2020-06-12T02:39:23.941 INFO:tasks.rados.rados.0.smithi107.stdout:12309: writing smithi10729407-18 from 0 to 540672 tid 1 2020-06-12T02:39:23.946 INFO:tasks.rados.rados.0.smithi107.stdout:12309: writing smithi10729407-18 from 540672 to 1286144 tid 2 2020-06-12T02:39:23.949 INFO:tasks.rados.rados.0.smithi107.stdout:12309: writing smithi10729407-18 from 1286144 to 1695744 tid 3 2020-06-12T02:39:23.951 INFO:tasks.rados.rados.0.smithi107.stdout:12309: writing smithi10729407-18 from 1695744 to 2146304 tid 4 2020-06-12T02:39:23.951 INFO:tasks.rados.rados.0.smithi107.stdout:12310: delete oid 17 current snap is 0 2020-06-12T02:39:23.951 INFO:tasks.rados.rados.0.smithi107.stdout:12306: finishing write tid 2 to smithi10729407-3 2020-06-12T02:39:23.952 INFO:tasks.rados.rados.0.smithi107.stdout:12306: finishing write tid 3 to smithi10729407-3 2020-06-12T02:39:23.952 INFO:tasks.rados.rados.0.smithi107.stdout:12306: finishing write tid 4 to smithi10729407-3 2020-06-12T02:39:23.952 INFO:tasks.rados.rados.0.smithi107.stdout:12306: finishing write tid 6 to smithi10729407-3 2020-06-12T02:39:23.953 INFO:tasks.rados.rados.0.smithi107.stdout:12306: finishing write tid 7 to smithi10729407-3 2020-06-12T02:39:23.953 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 3 v 2170 (ObjNum 12305 snap 0 seq_num 12305) dirty exists 2020-06-12T02:39:23.953 INFO:tasks.rados.rados.0.smithi107.stdout:12306: left oid 3 (ObjNum 12305 snap 0 seq_num 12305) 2020-06-12T02:39:23.953 INFO:tasks.rados.rados.0.smithi107.stdout:12308: finishing write tid 1 to smithi10729407-11 2020-06-12T02:39:23.953 INFO:tasks.rados.rados.0.smithi107.stdout:12308: finishing write tid 2 to smithi10729407-11 2020-06-12T02:39:23.954 INFO:tasks.ceph.osd.1.smithi107.stderr:2020-06-12T02:39:23.948+0000 7f76a68b7700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26605) UID: 0 2020-06-12T02:39:23.954 INFO:tasks.rados.rados.0.smithi107.stdout:12308: finishing write tid 3 to smithi10729407-11 2020-06-12T02:39:23.955 INFO:tasks.rados.rados.0.smithi107.stdout:12308: finishing write tid 4 to smithi10729407-11 2020-06-12T02:39:23.955 INFO:tasks.rados.rados.0.smithi107.stdout:12308: finishing write tid 6 to smithi10729407-11 2020-06-12T02:39:23.955 INFO:tasks.rados.rados.0.smithi107.stdout:12308: finishing write tid 7 to smithi10729407-11 2020-06-12T02:39:23.955 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 11 v 1867 (ObjNum 12307 snap 0 seq_num 12307) dirty exists 2020-06-12T02:39:23.955 INFO:tasks.rados.rados.0.smithi107.stdout:12308: left oid 11 (ObjNum 12307 snap 0 seq_num 12307) 2020-06-12T02:39:23.961 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2020-06-12T02:39:23.961 INFO:tasks.rados.rados.0.smithi107.stdout:12306: done (3 left) 2020-06-12T02:39:23.961 INFO:tasks.rados.rados.0.smithi107.stdout:12308: done (2 left) 2020-06-12T02:39:23.961 INFO:tasks.rados.rados.0.smithi107.stdout:12310: done (1 left) 2020-06-12T02:39:23.962 INFO:tasks.rados.rados.0.smithi107.stdout:12311: delete oid 7 current snap is 0 2020-06-12T02:39:23.963 INFO:tasks.rados.rados.0.smithi107.stdout:12309: finishing write tid 1 to smithi10729407-18 2020-06-12T02:39:23.963 INFO:tasks.rados.rados.0.smithi107.stdout:12309: finishing write tid 2 to smithi10729407-18 2020-06-12T02:39:23.964 INFO:tasks.rados.rados.0.smithi107.stdout:12311: done (1 left) 2020-06-12T02:39:23.964 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 19 current snap is 0 2020-06-12T02:39:23.964 INFO:tasks.rados.rados.0.smithi107.stdout:12312: seq_num 12311 ranges {0=589824,589824=450560,1040384=630784,1671168=524288} 2020-06-12T02:39:23.970 INFO:tasks.rados.rados.0.smithi107.stdout:12312: writing smithi10729407-19 from 0 to 589824 tid 1 2020-06-12T02:39:23.970 INFO:tasks.rados.rados.0.smithi107.stdout:12312: writing smithi10729407-19 from 589824 to 1040384 tid 2 2020-06-12T02:39:23.973 INFO:tasks.rados.rados.0.smithi107.stdout:12312: writing smithi10729407-19 from 1040384 to 1671168 tid 3 2020-06-12T02:39:23.976 INFO:tasks.rados.rados.0.smithi107.stdout:12312: writing smithi10729407-19 from 1671168 to 2195456 tid 4 2020-06-12T02:39:23.976 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 4 current snap is 0 2020-06-12T02:39:23.976 INFO:tasks.rados.rados.0.smithi107.stdout:12313: seq_num 12312 ranges {1302528=606208,1908736=729088,2637824=442368,3080192=434176} 2020-06-12T02:39:23.985 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2020-06-12T02:39:23.986 INFO:tasks.rados.rados.0.smithi107.stdout:12313: writing smithi10729407-4 from 1302528 to 1908736 tid 1 2020-06-12T02:39:23.991 INFO:tasks.rados.rados.0.smithi107.stdout:12313: writing smithi10729407-4 from 1908736 to 2637824 tid 2 2020-06-12T02:39:23.994 INFO:tasks.rados.rados.0.smithi107.stdout:12313: writing smithi10729407-4 from 2637824 to 3080192 tid 3 2020-06-12T02:39:23.996 INFO:tasks.rados.rados.0.smithi107.stdout:12313: writing smithi10729407-4 from 3080192 to 3514368 tid 4 2020-06-12T02:39:23.997 INFO:tasks.rados.rados.0.smithi107.stdout:12314: delete oid 20 current snap is 0 2020-06-12T02:39:23.997 INFO:tasks.rados.rados.0.smithi107.stdout:12309: finishing write tid 3 to smithi10729407-18 2020-06-12T02:39:23.997 INFO:tasks.rados.rados.0.smithi107.stdout:12309: finishing write tid 4 to smithi10729407-18 2020-06-12T02:39:23.998 INFO:tasks.rados.rados.0.smithi107.stdout:12309: finishing write tid 6 to smithi10729407-18 2020-06-12T02:39:23.998 INFO:tasks.rados.rados.0.smithi107.stdout:12309: finishing write tid 7 to smithi10729407-18 2020-06-12T02:39:23.998 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 18 v 1990 (ObjNum 12308 snap 0 seq_num 12308) dirty exists 2020-06-12T02:39:23.999 INFO:tasks.rados.rados.0.smithi107.stdout:12309: left oid 18 (ObjNum 12308 snap 0 seq_num 12308) 2020-06-12T02:39:23.999 INFO:tasks.rados.rados.0.smithi107.stdout:12312: finishing write tid 1 to smithi10729407-19 2020-06-12T02:39:23.999 INFO:tasks.rados.rados.0.smithi107.stdout:12312: finishing write tid 2 to smithi10729407-19 2020-06-12T02:39:24.000 INFO:tasks.rados.rados.0.smithi107.stdout:12312: finishing write tid 3 to smithi10729407-19 2020-06-12T02:39:24.000 INFO:tasks.rados.rados.0.smithi107.stdout:12312: finishing write tid 4 to smithi10729407-19 2020-06-12T02:39:24.000 INFO:tasks.rados.rados.0.smithi107.stdout:12312: finishing write tid 6 to smithi10729407-19 2020-06-12T02:39:24.001 INFO:tasks.rados.rados.0.smithi107.stdout:12312: finishing write tid 7 to smithi10729407-19 2020-06-12T02:39:24.001 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 19 v 2299 (ObjNum 12311 snap 0 seq_num 12311) dirty exists 2020-06-12T02:39:24.001 INFO:tasks.rados.rados.0.smithi107.stdout:12312: left oid 19 (ObjNum 12311 snap 0 seq_num 12311) 2020-06-12T02:39:24.002 INFO:tasks.rados.rados.0.smithi107.stdout:12313: finishing write tid 1 to smithi10729407-4 2020-06-12T02:39:24.003 INFO:tasks.rados.rados.0.smithi107.stdout:12309: done (3 left) 2020-06-12T02:39:24.003 INFO:tasks.rados.rados.0.smithi107.stdout:12312: done (2 left) 2020-06-12T02:39:24.003 INFO:tasks.rados.rados.0.smithi107.stdout:12314: done (1 left) 2020-06-12T02:39:24.003 INFO:tasks.rados.rados.0.smithi107.stdout:12315: delete oid 17 current snap is 0 2020-06-12T02:39:24.004 INFO:tasks.rados.rados.0.smithi107.stdout:12313: finishing write tid 2 to smithi10729407-4 2020-06-12T02:39:24.005 INFO:tasks.rados.rados.0.smithi107.stdout:12313: finishing write tid 3 to smithi10729407-4 2020-06-12T02:39:24.007 INFO:tasks.rados.rados.0.smithi107.stdout:12313: finishing write tid 4 to smithi10729407-4 2020-06-12T02:39:24.007 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2020-06-12T02:39:24.010 INFO:tasks.rados.rados.0.smithi107.stdout:12313: finishing write tid 6 to smithi10729407-4 2020-06-12T02:39:24.011 INFO:tasks.rados.rados.0.smithi107.stdout:12315: done (1 left) 2020-06-12T02:39:24.012 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 2 current snap is 0 2020-06-12T02:39:24.012 INFO:tasks.rados.rados.0.smithi107.stdout:12316: seq_num 12315 ranges {0=655360,655360=786432,1441792=614400,2056192=204800} 2020-06-12T02:39:24.015 INFO:tasks.rados.rados.0.smithi107.stdout:12316: writing smithi10729407-2 from 0 to 655360 tid 1 2020-06-12T02:39:24.021 INFO:tasks.rados.rados.0.smithi107.stdout:12316: writing smithi10729407-2 from 655360 to 1441792 tid 2 2020-06-12T02:39:24.025 INFO:tasks.rados.rados.0.smithi107.stdout:12316: writing smithi10729407-2 from 1441792 to 2056192 tid 3 2020-06-12T02:39:24.032 INFO:tasks.rados.rados.0.smithi107.stdout:12316: writing smithi10729407-2 from 2056192 to 2260992 tid 4 2020-06-12T02:39:24.032 INFO:tasks.rados.rados.0.smithi107.stdout:12317: delete oid 1 current snap is 0 2020-06-12T02:39:24.033 INFO:tasks.rados.rados.0.smithi107.stdout:12313: finishing write tid 7 to smithi10729407-4 2020-06-12T02:39:24.033 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 4 v 2282 (ObjNum 12312 snap 0 seq_num 12312) dirty exists 2020-06-12T02:39:24.033 INFO:tasks.rados.rados.0.smithi107.stdout:12313: left oid 4 (ObjNum 12312 snap 0 seq_num 12312) 2020-06-12T02:39:24.034 INFO:tasks.rados.rados.0.smithi107.stdout:12316: finishing write tid 1 to smithi10729407-2 2020-06-12T02:39:24.034 INFO:tasks.rados.rados.0.smithi107.stdout:12313: done (2 left) 2020-06-12T02:39:24.034 INFO:tasks.rados.rados.0.smithi107.stdout:12317: done (1 left) 2020-06-12T02:39:24.035 INFO:tasks.rados.rados.0.smithi107.stdout:append oid 7 current snap is 0 2020-06-12T02:39:24.035 INFO:tasks.rados.rados.0.smithi107.stdout:12318: seq_num 12317 ranges {0=688128,688128=606208,1294336=581632,1875968=417792} 2020-06-12T02:39:24.036 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2020-06-12T02:39:24.036 INFO:tasks.rados.rados.0.smithi107.stdout:12318: writing smithi10729407-7 from 0 to 688128 tid 1 2020-06-12T02:39:24.037 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-06-12T02:39:24.037 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-06-12T02:39:24.038 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-06-12T02:39:24.038 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-06-12T02:39:24.038 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-06-12T02:39:24.038 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-06-12T02:39:24.039 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-06-12T02:39:24.039 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.7 2020-06-12T02:39:24.039 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: problem getting command descriptions from osd.7 2020-06-12T02:39:24.040 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-06-12T02:39:24.041 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-06-12T02:39:24.041 INFO:tasks.rados.rados.0.smithi107.stdout:12318: writing smithi10729407-7 from 688128 to 1294336 tid 2 2020-06-12T02:39:24.042 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2020-06-12T02:39:24.042 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2020-06-12T02:39:24.043 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:24.043 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.4512.0:26225 3.8s0 3:14244450:::smithi10729407-2:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e88)", 2020-06-12T02:39:24.043 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:39:24.028320+0000", 2020-06-12T02:39:24.043 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.012276646, 2020-06-12T02:39:24.043 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012299236999999999, 2020-06-12T02:39:24.043 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:24.044 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2020-06-12T02:39:24.044 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2020-06-12T02:39:24.044 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.4512", 2020-06-12T02:39:24.044 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3215137305", 2020-06-12T02:39:24.044 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 26225 2020-06-12T02:39:24.045 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:24.045 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:24.045 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:24.045 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:24.045 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:24.028320+0000", 2020-06-12T02:39:24.046 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:24.046 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:24.046 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:24.047 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:24.047 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:24.028320+0000", 2020-06-12T02:39:24.047 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.986e-05 2020-06-12T02:39:24.048 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:24.048 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:24.048 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:24.049 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:24.028370+0000", 2020-06-12T02:39:24.049 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0979999999999999e-06 2020-06-12T02:39:24.049 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:24.049 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:24.050 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:24.050 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:24.028376+0000", 2020-06-12T02:39:24.050 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2237e-05 2020-06-12T02:39:24.051 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:24.051 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:24.051 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:24.052 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:24.028389+0000", 2020-06-12T02:39:24.052 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5469999999999998e-06 2020-06-12T02:39:24.052 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:24.053 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:24.053 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:24.053 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:24.028393+0000", 2020-06-12T02:39:24.054 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0092804270000000008 2020-06-12T02:39:24.054 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:24.054 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:24.054 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:24.055 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:24.037674+0000", 2020-06-12T02:39:24.055 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000102099 2020-06-12T02:39:24.055 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:24.055 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:24.056 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for rw locks", 2020-06-12T02:39:24.056 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:24.037776+0000", 2020-06-12T02:39:24.056 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023896920000000001 2020-06-12T02:39:24.056 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:24.057 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:24.057 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:24.057 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:24.040165+0000", 2020-06-12T02:39:24.058 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010835100000000001 2020-06-12T02:39:24.058 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:24.058 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:24.059 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2020-06-12T02:39:24.059 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:24.040274+0000", 2020-06-12T02:39:24.059 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011953310999999999 2020-06-12T02:39:24.059 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:24.060 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:24.060 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:24.060 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:24.061 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:24.061 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDECSubOpRead(3.8s0 88/85 ECSubRead(tid=12191, to_read={3:14244450:::smithi10729407-2:head=0,4096,0}, subchunks={3:14244450:::smithi10729407-2:head=[0,1]}, attrs_to_read=))", 2020-06-12T02:39:24.061 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2020-06-12T02:39:24.040493+0000", 2020-06-12T02:39:24.062 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.000103786, 2020-06-12T02:39:24.062 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000187553, 2020-06-12T02:39:24.062 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2020-06-12T02:39:24.063 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2020-06-12T02:39:24.063 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2020-06-12T02:39:24.063 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:24.064 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2020-06-12T02:39:24.064 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:24.040493+0000", 2020-06-12T02:39:24.064 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:24.065 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:24.065 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:24.065 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2020-06-12T02:39:24.066 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:24.040493+0000", 2020-06-12T02:39:24.066 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:24.066 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:24.066 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:24.067 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2020-06-12T02:39:24.067 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:24.040493+0000", 2020-06-12T02:39:24.067 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2020-06-12T02:39:24.068 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:24.068 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:24.068 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2020-06-12T02:39:24.069 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:24.040493+0000", 2020-06-12T02:39:24.069 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0080000000000001e-05 2020-06-12T02:39:24.069 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:24.070 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:24.070 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2020-06-12T02:39:24.070 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:24.040513+0000", 2020-06-12T02:39:24.071 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2670000000000006e-06 2020-06-12T02:39:24.071 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:24.071 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:24.072 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2020-06-12T02:39:24.072 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:24.040522+0000", 2020-06-12T02:39:24.072 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6780999999999998e-05 2020-06-12T02:39:24.072 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2020-06-12T02:39:24.073 INFO:teuthology.orchestra.run.smithi107.stdout: { 2020-06-12T02:39:24.073 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2020-06-12T02:39:24.073 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2020-06-12T02:39:24.040568+0000", 2020-06-12T02:39:24.073 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5128000000000003e-05 2020-06-12T02:39:24.074 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:24.074 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2020-06-12T02:39:24.074 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:24.074 INFO:teuthology.orchestra.run.smithi107.stdout: } 2020-06-12T02:39:24.075 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2020-06-12T02:39:24.075 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 2 2020-06-12T02:39:24.075 INFO:teuthology.orchestra.run.smithi107.stdout:} 2020-06-12T02:39:24.079 INFO:tasks.rados.rados.0.smithi107.stdout:12318: writing smithi10729407-7 from 1294336 to 1875968 tid 3 2020-06-12T02:39:24.079 INFO:tasks.rados.rados.0.smithi107.stdout:12318: writing smithi10729407-7 from 1875968 to 2293760 tid 4 2020-06-12T02:39:24.080 INFO:tasks.rados.rados.0.smithi107.stdout:12319: delete oid 1 current snap is 0 2020-06-12T02:39:24.080 INFO:tasks.rados.rados.0.smithi107.stdout:12316: finishing write tid 2 to smithi10729407-2 2020-06-12T02:39:24.080 INFO:tasks.rados.rados.0.smithi107.stdout:12316: finishing write tid 3 to smithi10729407-2 2020-06-12T02:39:24.081 INFO:tasks.rados.rados.0.smithi107.stdout:12316: finishing write tid 4 to smithi10729407-2 2020-06-12T02:39:24.081 INFO:tasks.rados.rados.0.smithi107.stdout:12316: finishing write tid 6 to smithi10729407-2 2020-06-12T02:39:24.081 INFO:tasks.rados.rados.0.smithi107.stdout:12316: finishing write tid 7 to smithi10729407-2 2020-06-12T02:39:24.081 INFO:tasks.rados.rados.0.smithi107.stdout:update_object_version oid 2 v 1153 (ObjNum 12315 snap 0 seq_num 12315) dirty exists 2020-06-12T02:39:24.082 INFO:tasks.rados.rados.0.smithi107.stdout:12316: left oid 2 (ObjNum 12315 snap 0 seq_num 12315) 2020-06-12T02:39:24.082 INFO:tasks.rados.rados.0.smithi107.stdout:12318: finishing write tid 1 to smithi10729407-7 2020-06-12T02:39:24.082 INFO:tasks.rados.rados.0.smithi107.stdout:12318: finishing write tid 2 to smithi10729407-7 2020-06-12T02:39:24.082 INFO:tasks.rados.rados.0.smithi107.stdout:12316: done (2 left) 2020-06-12T02:39:24.083 INFO:tasks.rados.rados.0.smithi107.stdout:12319: done (1 left) 2020-06-12T02:39:24.083 INFO:tasks.rados.rados.0.smithi107.stdout:12320: delete oid 3 current snap is 0 2020-06-12T02:39:24.083 INFO:tasks.rados.rados.0.smithi107.stdout:12318: finishing write tid 3 to smithi10729407-7 2020-06-12T02:39:24.083 INFO:tasks.rados.rados.0.smithi107.stdout:12320: done (1 left) 2020-06-12T02:39:24.084 INFO:tasks.rados.rados.0.smithi107.stdout:append oid (excl) 2 current snap is 0 2020-06-12T02:39:24.084 INFO:tasks.rados.rados.0.smithi107.stdout:12321: seq_num 12320 ranges {2260992=745472,3006464=655360,3661824=802816,4464640=147456} 2020-06-12T02:39:24.084 INFO:tasks.rados.rados.0.smithi107.stdout:12321: writing smithi10729407-2 from 2260992 to 3006464 tid 1 2020-06-12T02:39:24.085 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2020-06-12T02:39:24.087 DEBUG:teuthology.task.console_log:Killing console logger for smithi107 2020-06-12T02:39:24.088 DEBUG:teuthology.task.console_log:Killing console logger for smithi188 2020-06-12T02:39:24.088 DEBUG:teuthology.task.console_log:Killing console logger for smithi107 2020-06-12T02:39:24.089 DEBUG:teuthology.task.console_log:Killing console logger for smithi188 2020-06-12T02:39:24.090 DEBUG:teuthology.exit:Finished running handlers